LoRA-Diffusion: Parameter-Efficient Fine-Tuning via Low-Rank Trajectory Decomposition

# Parameter-Efficient Fine-Tuning via Low-Rank Trajectory Decomposition
Source: [https://arxiv.org/html/2608.12328](https://arxiv.org/html/2608.12328)
2ndNarges Nejad4thMostafa M\. Rezaee3rdMohammadhossein Homaei5thRobert C\. Green II

###### Abstract

Parameter\-efficient fine\-tuning methods such as LoRA have transformed the adaptation of large autoregressive language models, enabling task\-specific customization with fewer than 1% trainable parameters\. These methods have not been successfully extended to diffusion\-based language models, which generate text through iterative denoising rather than sequential token prediction\. We propose LoRA\-Diffusion, a parameter\-efficient fine\-tuning approach that applies low\-rank decomposition to the denoising trajectory instead of model weights\. Unlike weight\-based LoRA, which modifies individual transformation matrices, our method learns low\-rank perturbations to the entire diffusion path from noise to output\. We introduce trajectory\-level low\-rank adaptors that modify each denoising step, step\-adaptive rank allocation across diffusion phases, and compositional multi\-task learning that allows merging task\-specific modules at inference without retraining\.

On SST\-2, QNLI, and MRPC \(5 seeds\), we report*token\-level denoising validation accuracy*, which directly reflects optimization of the diffusion training objective; under this metric, LoRA\-Diffusion reaches the highest mean on SST\-2 \(88\.01%\) and strong performance on QNLI \(99\.39%\) and MRPC \(97\.56%\)\. Joint multi\-task training \(same five methods, five seeds\) shows LoRA\-Diffusion achieving the highest token\-level accuracy \(96\.88%±\\pm0\.44%\)\.

LoRA\-Diffusion achieves the highest*token\-level*SST\-2 validation accuracy while training 28\.7% of parameters \(instruction encoder 27\.5% \+ trajectory adapters 1\.2%\)\. Importantly, the trajectory\-level adapters themselves account for only 1\.2% of parameters; the remaining trainable parameters arise from a shared instruction encoder used to condition the adapters and are orthogonal to the trajectory adaptation mechanism\. The approach is competitive with adapter layers and baselines, reduces per\-task storage \(151 MB vs\. 525 MB for full fine\-tuning\), and exhibits minimal catastrophic forgetting\. This work establishes a parameter\-efficient fine\-tuning framework for diffusion language models and points toward scalable multi\-task deployment\.

## IIntroduction

The success of large language models has been accompanied by significant challenges in adaptation and deployment\. Full fine\-tuning of billion\-parameter models is computationally costly, requiring substantial GPU memory and training time\[[4](https://arxiv.org/html/2608.12328#bib.bib3)\]\. Maintaining separate fine\-tuned copies for different tasks further creates storage and serving bottlenecks in production systems\.

Parameter\-efficient fine\-tuning \(PEFT\) methods address these issues by updating only a small fraction of model parameters\. Among them, Low\-Rank Adaptation \(LoRA\) has proven especially effective, achieving near–full fine\-tuning performance on autoregressive models while training fewer than 1% of parameters\[[14](https://arxiv.org/html/2608.12328#bib.bib8)\]\. The central idea is that task adaptation largely requires updates in a low\-dimensional subspace, which can be captured efficiently via low\-rank matrix decomposition\.

Recent work has shown that discrete diffusion models can match or exceed autoregressive models in text generation quality\[[22](https://arxiv.org/html/2608.12328#bib.bib14),[24](https://arxiv.org/html/2608.12328#bib.bib15)\]\. Diffusion models offer bidirectional context, parallel generation, controllable generation, and diverse sampling\. Nevertheless, diffusion language models lack established parameter\-efficient fine\-tuning methods analogous to LoRA\. Existing approaches either apply standard LoRA to diffusion weights \(treating the model as a standard transformer\), perform full fine\-tuning, or use adapter layers or prefix tuning, which introduce sequential bottlenecks\. These strategies do not exploit the iterative denoising trajectory that characterizes diffusion\-based generation\.

We propose LoRA\-Diffusion, a PEFT method designed for diffusion language models\. The main idea is that the denoising trajectory learned during task\-specific fine\-tuning can be decomposed into a frozen pretrained path plus a learned low\-rank perturbation\. Formally, we write

𝐱tfine\-tuned=𝐱tpretrained\+Δ​𝐱tlow\-rank,\\mathbf\{x\}\_\{t\}^\{\\text\{fine\-tuned\}\}=\\mathbf\{x\}\_\{t\}^\{\\text\{pretrained\}\}\+\\Delta\\mathbf\{x\}\_\{t\}^\{\\text\{low\-rank\}\},\(1\)whereΔ​𝐱tlow\-rank\\Delta\\mathbf\{x\}\_\{t\}^\{\\text\{low\-rank\}\}is produced by lightweight low\-rank adaptors conditioned on the task instruction\. The perturbation is applied in*hidden representation space*:𝐡t′=𝐡t\+δt\\mathbf\{h\}\_\{t\}^\{\\prime\}=\\mathbf\{h\}\_\{t\}\+\\delta\_\{t\}, then logits𝐥t=OutputHead​\(𝐡t′\)\\mathbf\{l\}\_\{t\}=\\text\{OutputHead\}\(\\mathbf\{h\}\_\{t\}^\{\\prime\}\)and𝐱t−1\\mathbf\{x\}\_\{t\-1\}from the output head \(Section[III](https://arxiv.org/html/2608.12328#S3), Eq\. 2 and Eq\. 3\)\. Weight\-based LoRA modifies transformation matrices viaW′=W\+B​AW^\{\\prime\}=W\+BA; LoRA\-Diffusion instead modifies the denoising trajectory𝐱t−1=f​\(𝐱t\)\+gLoRA​\(𝐱t\)\\mathbf\{x\}\_\{t\-1\}=f\(\\mathbf\{x\}\_\{t\}\)\+g\_\{\\text\{LoRA\}\}\(\\mathbf\{x\}\_\{t\}\)\. Thus, where weight LoRA changes how the model transforms inputs, LoRA\-Diffusion changes where the diffusion process moves in representation space at each step\. Figure[1](https://arxiv.org/html/2608.12328#S1.F1)provides an overview of the proposed LoRA\-Diffusion framework, including the frozen diffusion backbone, trajectory\-level low\-rank adaptation, step\-adaptive rank allocation, and modular multi\-task composition\.

![Refer to caption](https://arxiv.org/html/2608.12328v1/x1.png)Figure 1:Overview of LoRA\-Diffusion, a trajectory\-aware parameter\-efficient fine\-tuning framework for diffusion language models\. The framework keeps the pretrained diffusion backbone frozen and applies low\-rank, task\-conditioned perturbations to the denoising trajectory through step\-adaptive rank allocation and modular multi\-task composition\.We make the following contributions\. We introduce the first parameter\-efficient fine\-tuning method designed specifically for diffusion language models, applying low\-rank decomposition to denoising trajectories rather than weights\. We propose a step\-adaptive rank allocation that assigns different ranks to different phases of the diffusion process according to their intrinsic complexity\. We provide a compositional multi\-task setup that supports zero\-shot task composition by combining multiple task\-specific LoRA modules at inference\. We present an empirical evaluation on SST\-2, QNLI, and MRPC \(single\-task and joint multi\-task\) with a BERT\-based diffusion model \(137\.7M parameters\), 5 seeds \(42–46\), comparing LoRA\-Diffusion to full fine\-tuning and several PEFT baselines \(weight LoRA, adapters, BitFit\), with token\-level denoising accuracy, efficiency metrics \(trainable parameters, storage, training time, inference latency\), and ablations for rank and orthogonality regularization\. We give an information\-theoretic motivation for trajectory\-level low\-rank structure and clarify positioning versus adapter layers and timestep\-aware weight LoRA \(T\-LoRA, FouRA\)\. We release an open\-source implementation to support reproducibility and extension\.

We emphasize that the core contribution of LoRA\-Diffusion lies in trajectory\-level low\-rank adaptation, which introduces only 1\.2% additional parameters relative to the base model\. The larger total trainable fraction \(28\.7%\) arises from the inclusion of a shared instruction encoder that conditions the adapters and is not intrinsic to trajectory\-level adaptation itself\. Isolating, shrinking, or freezing the instruction encoder is a complementary design choice and is left to future work; throughout this paper, we report a transparent and consistent accounting of all trainable parameters\.

The rest of the paper is organized as follows\. Section[II](https://arxiv.org/html/2608.12328#S2)reviews related work on diffusion models for language, parameter\-efficient fine\-tuning, and multi\-task learning\. Section[III](https://arxiv.org/html/2608.12328#S3)presents our methodology, including preliminaries, trajectory\-level low\-rank adaptation, the training objective, multi\-task composition, and implementation details\. Section[IV](https://arxiv.org/html/2608.12328#S4)describes the experimental setup and results on SST\-2, QNLI, and MRPC, including single\-task and multi\-task GLUE results, main results, efficiency analysis, catastrophic forgetting, ablations, and comparison with weight\-based LoRA\. Section[V](https://arxiv.org/html/2608.12328#S5)summarizes our contributions, discusses limitations and future work, and closes with broader impact and reproducibility notes\.

## IIRelated Work

### II\-ADiffusion Models for Language

\[[2](https://arxiv.org/html/2608.12328#bib.bib2)\]introduced discrete diffusion for categorical data, with uniform and absorbing\-state transition mechanisms\.\[[12](https://arxiv.org/html/2608.12328#bib.bib6)\]proposed argmax flows for multinomial diffusion\. More recently,\[[22](https://arxiv.org/html/2608.12328#bib.bib14)\]presented SEDD, which achieves competitive generation quality with autoregressive models;\[[24](https://arxiv.org/html/2608.12328#bib.bib15)\]simplified the setup with masked diffusion; and\[[21](https://arxiv.org/html/2608.12328#bib.bib13)\]explored controlled generation with Diffusion\-LM\. All of this work focuses on pretraining or basic fine\-tuning\. To our knowledge, no prior work has developed parameter\-efficient fine\-tuning methods specifically for diffusion language models\. Beyond language modeling, diffusion and generative models have also been widely explored for data augmentation in domains where labeled data are limited, imbalanced, or expensive to obtain\. In medical imaging, prior studies have shown that DDPM\- and GAN\-based synthetic data can improve classification performance under small\-sample and class\-imbalanced settings\[[16](https://arxiv.org/html/2608.12328#bib.bib31)\], and that DDPM\-generated synthetic images can support vocal fold pathology classification in pilot clinical imaging studies\[[17](https://arxiv.org/html/2608.12328#bib.bib32)\]\. These studies motivate the broader use of diffusion\-based adaptation strategies in data\-scarce settings, while the present work focuses on parameter\-efficient adaptation of diffusion language models rather than image generation\.

### II\-BParameter\-Efficient Fine\-Tuning

\[[14](https://arxiv.org/html/2608.12328#bib.bib8)\]introduced LoRA for low\-rank adaptation of autoregressive models\.\[[6](https://arxiv.org/html/2608.12328#bib.bib4)\]combined LoRA with quantization \(QLoRA\), and\[[31](https://arxiv.org/html/2608.12328#bib.bib21)\]proposed AdaLoRA to adapt ranks dynamically\. Other PEFT methods include prefix tuning\[[20](https://arxiv.org/html/2608.12328#bib.bib12)\], prompt tuning\[[18](https://arxiv.org/html/2608.12328#bib.bib10)\], adapter layers\[[13](https://arxiv.org/html/2608.12328#bib.bib7)\], and BitFit\[[29](https://arxiv.org/html/2608.12328#bib.bib20)\], which trains only bias terms\. These methods target autoregressive architectures\. Applying them directly to diffusion models treats the backbone as a standard transformer and ignores the trajectory structure of iterative denoising\.

Recent work has explored timestep\-aware and rank\-adaptive PEFT for diffusion models, primarily in the image domain\.\[[25](https://arxiv.org/html/2608.12328#bib.bib33)\]\(T\-LoRA\) applies timestep\-dependent rank masking and orthogonalization to maintain effective rank across diffusion steps\.\[[3](https://arxiv.org/html/2608.12328#bib.bib22)\]\(FouRA\) introduces frequency\-domain LoRA with adaptive rank gating across timesteps\.\[[32](https://arxiv.org/html/2608.12328#bib.bib27)\]\(TALoRA and MSFP\) propose timestep\-adaptive low\-rank factorization with hub\-based sharing\.\[[23](https://arxiv.org/html/2608.12328#bib.bib24)\]\(SeLoRA\) and\[[8](https://arxiv.org/html/2608.12328#bib.bib25)\]\(GeLoRA\) provide principled rank allocation based on Fisher information and intrinsic dimension\.\[[30](https://arxiv.org/html/2608.12328#bib.bib30)\]\(EST\-LoRA\) studies training\-free adapter fusion via routing at inference\.\[[5](https://arxiv.org/html/2608.12328#bib.bib26)\]\(TC\-LoRA\) conditions low\-rank weight updates on timestep and condition via a hypernetwork, modulating weight functions per timestep/condition\.\[[11](https://arxiv.org/html/2608.12328#bib.bib23)\]\(EfficientDM\) and\[[7](https://arxiv.org/html/2608.12328#bib.bib28)\]\(Glance\) demonstrate practical PEFT/acceleration strategies with step/phase specializations\.\[[10](https://arxiv.org/html/2608.12328#bib.bib29)\]\(Delta Sampling\) operates at inference by reusing deltas in prediction space\. These methods operate in weight or frequency space and allocate capacity across timesteps, but do not explicitly model trajectory\-level perturbations\. While activation\-based adapters also modify representations, they operate at fixed network layers and do not model the evolution of hidden states along the diffusion trajectory, which is the central object of adaptation in LoRA\-Diffusion\. LoRA\-Diffusion differs by operating directly in representation/trajectory space, where low\-rank structure emerges naturally from the iterative denoising process, and by using a phase\-shared design that keeps parameter counts independent of the number of diffusion steps\. Unlike TC\-LoRA which modulates weights, LoRA\-Diffusion modulates trajectory corrections, offering different representational advantages and computational costs\.

#### Comparison with timestep\-aware and diffusion PEFT\.

Table[I](https://arxiv.org/html/2608.12328#S2.T1)summarizes how LoRA\-Diffusion relates to prior PEFT methods\. Key trade\-offs: \(1\)Compute:LoRA\-Diffusion adds a lightweightgϕg\_\{\\phi\}per diffusion step, so inference cost is higher than weight LoRA unlessgϕg\_\{\\phi\}is very small; \(2\)Compositionality:trajectory superposition \(router\-weighted sum of task adapters\) vs\. weight\-space task arithmetic; \(3\)Trainability:trajectory\-only adapters are 1\.2% of base; with instruction encoder, total trainable is 28\.7%\.

TABLE I:Comparison with diffusion and timestep\-aware PEFT\.

### II\-CMulti\-Task Learning and Low\-Rank Theory

\[[15](https://arxiv.org/html/2608.12328#bib.bib9)\]showed that task vectors can be combined via task arithmetic\.\[[28](https://arxiv.org/html/2608.12328#bib.bib18)\]used orthogonal subspace projection to reduce interference\. Routing\-based mixture\-of\-experts approaches\[[9](https://arxiv.org/html/2608.12328#bib.bib5)\]select experts per input\.\[[1](https://arxiv.org/html/2608.12328#bib.bib1)\]demonstrated that task adaptation has low intrinsic dimensionality;\[[19](https://arxiv.org/html/2608.12328#bib.bib11)\]measured intrinsic dimensionality empirically\.\[[27](https://arxiv.org/html/2608.12328#bib.bib17)\]provided an information\-theoretic perspective via the information bottleneck\. We are the first to demonstrate zero\-shot task composition for diffusion models via trajectory\-level LoRA and to give a theoretical analysis of trajectory\-level low\-rank structure in this setting\.

## IIIMethodology

### III\-APreliminaries

A discrete diffusion model for language defines a forward Markov process that gradually corrupts clean text𝐱0=\(x01,…,x0n\)\\mathbf\{x\}\_\{0\}=\(x\_\{0\}^\{1\},\\ldots,x\_\{0\}^\{n\}\),x0i∈𝒱x\_\{0\}^\{i\}\\in\\mathcal\{V\}, over timestepst∈\[1,T\]t\\in\[1,T\]\. Common transitions include the uniform and absorbing\-state \(masking\) schemes of\[[2](https://arxiv.org/html/2608.12328#bib.bib2)\]\. The model learns to reverse the process by predicting𝐱0\\mathbf\{x\}\_\{0\}from𝐱t\\mathbf\{x\}\_\{t\}andtt, and is trained with a simplified objectiveℒsimple=𝔼𝐱0,t,𝐱t​\[−log⁡pθ​\(𝐱0∣𝐱t,t\)\]\\mathcal\{L\}\_\{\\text\{simple\}\}=\\mathbb\{E\}\_\{\\mathbf\{x\}\_\{0\},t,\\mathbf\{x\}\_\{t\}\}\[\-\\log p\_\{\\theta\}\(\\mathbf\{x\}\_\{0\}\\mid\\mathbf\{x\}\_\{t\},t\)\]\. For conditional generation, conditioningcc\(e\.g\. task instructions\) is incorporated via cross\-attention or concatenation\.

LoRA\[[14](https://arxiv.org/html/2608.12328#bib.bib8)\]adapts pretrained weightsW0W\_\{0\}viaW=W0\+B​AW=W\_\{0\}\+BA, withB∈ℝd×rB\\in\\mathbb\{R\}^\{d\\times r\},A∈ℝr×dA\\in\\mathbb\{R\}^\{r\\times d\},r≪dr\\ll d, and onlyBBandAAtrained\. Its success is tied to the low intrinsic dimensionality of task adaptation\[[1](https://arxiv.org/html/2608.12328#bib.bib1)\]\. Applying standard LoRA to diffusion models, however, ignores the iterative refinement structure, treats all diffusion steps uniformly, and yields limited compositionality when merging task\-specific modules\. We therefore move from weight\-level to trajectory\-level adaptation\.

### III\-BRepresentation Space and Trajectory Perturbations

In discrete diffusion language models,𝐱t\\mathbf\{x\}\_\{t\}represents discrete token IDs from the vocabulary𝒱\\mathcal\{V\}\. The model operates on hidden representations𝐡t=Transformer​\(𝐱t,t\)\\mathbf\{h\}\_\{t\}=\\text\{Transformer\}\(\\mathbf\{x\}\_\{t\},t\)obtained by passing token embeddings through the transformer backbone with time embeddings\. The output head then computes logits𝐥t=OutputHead​\(𝐡t\)\\mathbf\{l\}\_\{t\}=\\text\{OutputHead\}\(\\mathbf\{h\}\_\{t\}\)to predict the next token distribution\.

Trajectory perturbations are applied in the hidden representation space, not directly to tokens or logits\. The data flow is:tokens𝐱t\\mathbf\{x\}\_\{t\}\(discrete IDs\)→\\toembeddings→\\tohidden states𝐡t=Transformer​\(𝐱t,t\)\\mathbf\{h\}\_\{t\}=\\text\{Transformer\}\(\\mathbf\{x\}\_\{t\},t\)→\\toperturbation𝐡t′=𝐡t\+δt\\mathbf\{h\}\_\{t\}^\{\\prime\}=\\mathbf\{h\}\_\{t\}\+\\delta\_\{t\}→\\tologits𝐥t=OutputHead​\(𝐡t′\)\\mathbf\{l\}\_\{t\}=\\text\{OutputHead\}\(\\mathbf\{h\}\_\{t\}^\{\\prime\}\)\. Specifically:

𝐡t′=𝐡t\+δt,\\mathbf\{h\}\_\{t\}^\{\\prime\}=\\mathbf\{h\}\_\{t\}\+\\delta\_\{t\},\(2\)whereδt\\delta\_\{t\}is the learned low\-rank perturbation, and then𝐥t=OutputHead​\(𝐡t′\)\\mathbf\{l\}\_\{t\}=\\text\{OutputHead\}\(\\mathbf\{h\}\_\{t\}^\{\\prime\}\)\. This preserves the probabilistic structure because: \(1\) the output head remains deterministic, \(2\) perturbations are learned to maintain valid conditional distributionsp​\(𝐱0\|𝐱t,t,c\)p\(\\mathbf\{x\}\_\{0\}\|\\mathbf\{x\}\_\{t\},t,c\), and \(3\) the training objective ensures the perturbed trajectory produces valid reverse diffusion transitions\. The output head is deterministic, sop​\(𝐱0∣𝐱t,t,c\)p\(\\mathbf\{x\}\_\{0\}\\mid\\mathbf\{x\}\_\{t\},t,c\)stays well\-defined; the denoising loss trainsδt\\delta\_\{t\}to yield valid reverse transitions\.

#### Algorithm\.

At each denoising steptt, the pipeline is: \(1\) compute𝐡t=Transformer​\(𝐱∗t,t\)\\mathbf\{h\}\_\{t\}=\\text\{Transformer\}\(\\mathbf\{x\}\*t,t\); \(2\) computeδt=∑iσ​\(t\)⋅g∗ϕi​\(𝐡t,t,c\)\\delta\_\{t\}=\\sum\_\{i\}\\sigma\(t\)\\cdot g\*\{\\phi\_\{i\}\}\(\\mathbf\{h\}\_\{t\},t,c\); \(3\) set𝐡t′=𝐡t\+δt\\mathbf\{h\}\_\{t\}^\{\\prime\}=\\mathbf\{h\}\_\{t\}\+\\delta\_\{t\}; \(4\) compute𝐥t=OutputHead​\(𝐡∗t′\)\\mathbf\{l\}\_\{t\}=\\text\{OutputHead\}\(\\mathbf\{h\}\*t^\{\\prime\}\); and \(5\) obtain𝐱∗t−1\\mathbf\{x\}\*\{t\-1\}from𝐥t\\mathbf\{l\}\_\{t\}through sampling or deterministic decoding\. Thus, the proposed perturbations operate on the intermediate hidden states𝐡t\\mathbf\{h\}\_\{t\}, rather than directly modifying token IDs or logits\. During training, the LoRA\-Diffusion adapters are optimized to learn low\-rank, time\-dependent perturbations of the hidden trajectory, while inference applies the same learned adapters within the reverse denoising process to guide generation in a compositional manner\.

### III\-CTrajectory\-Level Low\-Rank Adaptation

At each denoising steptt, the model computes𝐱t−1=fθ​\(𝐱t,t,c\)\\mathbf\{x\}\_\{t\-1\}=f\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,c\)via the process: tokens𝐱t\\mathbf\{x\}\_\{t\}→ hidden states𝐡t\\mathbf\{h\}\_\{t\}→ \(optionally perturbed\)𝐡t′\\mathbf\{h\}\_\{t\}^\{\\prime\}→ logits𝐥t\\mathbf\{l\}\_\{t\}→ predicted tokens𝐱t−1\\mathbf\{x\}\_\{t\-1\}\. After task\-specific fine\-tuning, the denoising function changes fromfθf\_\{\\theta\}tofθ′f\_\{\\theta^\{\\prime\}\}\. We hypothesize that the differenceΔ​f=fθ′−fθ\\Delta f=f\_\{\\theta^\{\\prime\}\}\-f\_\{\\theta\}can be well approximated by a low\-rank function in representation space, i\.e\. that the trajectory perturbationδt\\delta\_\{t\}lies in a low\-dimensional subspace ofℝd\\mathbb\{R\}^\{d\}whereddis the hidden dimension\.

We decompose the fine\-tuned trajectory as

𝐱t−1fine\-tuned=fθ0​\(𝐱t,t,c\)⏟frozen pretrained\+∑i=1kσ​\(t\)⋅gϕi​\(𝐱t,t,c\)⏟learnable low\-rank perturbation,\\mathbf\{x\}\_\{t\-1\}^\{\\text\{fine\-tuned\}\}=\\underbrace\{f\_\{\\theta\_\{0\}\}\(\\mathbf\{x\}\_\{t\},t,c\)\}\_\{\\text\{frozen pretrained\}\}\+\\underbrace\{\\sum\_\{i=1\}^\{k\}\\sigma\(t\)\\cdot g\_\{\\phi\_\{i\}\}\(\\mathbf\{x\}\_\{t\},t,c\)\}\_\{\\text\{learnable low\-rank perturbation\}\},\(3\)wherefθ0f\_\{\\theta\_\{0\}\}is the frozen pretrained denoising function,gϕig\_\{\\phi\_\{i\}\}is theii\-th low\-rank perturbation module,σ​\(t\)\\sigma\(t\)is a step\-adaptive scaling function, andkkis the number of LoRA modules per step \(typically 1–4\)\.

Each modulegϕig\_\{\\phi\_\{i\}\}is implemented asgϕi​\(𝐱t,t,c\)=Ai​\(c\)⋅ReLU​\(Bi​\(𝐱t,t\)\)g\_\{\\phi\_\{i\}\}\(\\mathbf\{x\}\_\{t\},t,c\)=A\_\{i\}\(c\)\\cdot\\text\{ReLU\}\(B\_\{i\}\(\\mathbf\{x\}\_\{t\},t\)\), withBi:ℝd→ℝrB\_\{i\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{r\}\(down\-projection\) andAi:ℝr→ℝdA\_\{i\}:\\mathbb\{R\}^\{r\}\\to\\mathbb\{R\}^\{d\}\(up\-projection\),r≪dr\\ll d\. The down\-projection isBi​\(𝐱t,t\)=WB\(i\)​\[𝐱t;Emb​\(t\)\]B\_\{i\}\(\\mathbf\{x\}\_\{t\},t\)=W\_\{B\}^\{\(i\)\}\[\\mathbf\{x\}\_\{t\};\\text\{Emb\}\(t\)\], while the up\-projection is implemented via FiLM\-style conditioning: a base matrix plus instruction\-dependent scale and shift\. Concretely, we realizeAi​\(c\)A\_\{i\}\(c\)as

Ai​\(c\)​v=WA\(i\)​\(γi​\(c\)⊙v\)\+βi​\(c\),A\_\{i\}\(c\)v=W\_\{A\}^\{\(i\)\}\\bigl\(\\gamma\_\{i\}\(c\)\\odot v\\bigr\)\+\\beta\_\{i\}\(c\),\(4\)whereγi​\(c\)\\gamma\_\{i\}\(c\)andβi​\(c\)\\beta\_\{i\}\(c\)are computed by a lightweight instruction encoder and⊙\\odotdenotes elementwise multiplication\. Thenominal rankrrrefers to the bottleneck dimension ofBi​\(⋅,t\)B\_\{i\}\(\\cdot,t\);Ai​\(c\)A\_\{i\}\(c\)is a conditional up\-projection\. FiLM applies elementwise scale and shift to the bottleneck vectorv∈ℝrv\\in\\mathbb\{R\}^\{r\}; the output remains in the column space ofWA\(i\)W\_\{A\}^\{\(i\)\}\(plus a fixed shift percc\), so for each fixedcc, the mapv↦Ai​\(c\)​Bi​\(𝐡∗t,t\)v\\mapsto A\_\{i\}\(c\)B\_\{i\}\(\\mathbf\{h\}\*t,t\)has range in an at\-most\-rr\-dimensional affine subspace andeffective rank at mostrr\. We measure effective rank empirically by computing phase\-wise singular value spectra of the learned trajectory perturbations, as described in Section[IV](https://arxiv.org/html/2608.12328#S4)\. The nuclear normℛ∗rank\\mathcal\{R\}\*\{\\text\{rank\}\}is applied to the base matricesWA\(i\)W\_\{A\}^\{\(i\)\}andWB\(i\)W\_\{B\}^\{\(i\)\}, encouraging low\-rank structure in the unconstrained components\.

Different diffusion steps play different roles: early steps \(largett\) handle global structure and semantics; middle steps refine content and coherence; late steps \(smalltt\) polish local details\. We partition timesteps into three phases:Early\(t\>2​T/3t\>2T/3\),Mid\(T/3<t≤2​T/3T/3<t\\leq 2T/3\), andLate\(t≤T/3t\\leq T/3\)\. ForT=100T=100, this corresponds to early:t∈\[67,100\]t\\in\[67,100\], mid:t∈\[34,66\]t\\in\[34,66\], and late:t∈\[0,33\]t\\in\[0,33\]\. We use step\-adaptive scalingσ​\(t\)\\sigma\(t\)withσearly=1\.0\\sigma\_\{\\text\{early\}\}=1\.0,σmid=0\.5\\sigma\_\{\\text\{mid\}\}=0\.5, andσlate=0\.25\\sigma\_\{\\text\{late\}\}=0\.25\. We also allocate rankr​\(t\)r\(t\)adaptively:rearly=64r\_\{\\text\{early\}\}=64,rmid=32r\_\{\\text\{mid\}\}=32, andrlate=8r\_\{\\text\{late\}\}=8\. Early steps explore a high\-dimensional space of global structures and thus use higher rank; late steps refine within a local neighborhood and use lower rank\. In the reference implementation we instantiate three banks of adapters corresponding to early/mid/late phases and reuse them across all timesteps within a phase, so the trainable parameter count is independent ofTTand step\-awareness is expressed through the phase\-dependent scalingσ​\(t\)\\sigma\(t\)rather than separate parameters for every timestep\.

### III\-DTraining Objective

The training objective is

ℒ=ℒdenoise\+λrank​ℛrank\+λorth​ℛorth,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{denoise\}\}\+\\lambda\_\{\\text\{rank\}\}\\mathcal\{R\}\_\{\\text\{rank\}\}\+\\lambda\_\{\\text\{orth\}\}\\mathcal\{R\}\_\{\\text\{orth\}\},\(5\)withℒdenoise=𝔼𝐱0,c,t,𝐱t​\[−log⁡pθ​\(𝐱0∣𝐱t,t,c\)\]\\mathcal\{L\}\_\{\\text\{denoise\}\}=\\mathbb\{E\}\_\{\\mathbf\{x\}\_\{0\},c,t,\\mathbf\{x\}\_\{t\}\}\[\-\\log p\_\{\\theta\}\(\\mathbf\{x\}\_\{0\}\\mid\\mathbf\{x\}\_\{t\},t,c\)\]and

ℛrank\\displaystyle\\mathcal\{R\}\_\{\\text\{rank\}\}=∑i=1k‖WA\(i\)‖∗\+‖WB\(i\)‖∗,\\displaystyle=\\sum\_\{i=1\}^\{k\}\\\|W\_\{A\}^\{\(i\)\}\\\|\_\{\*\}\+\\\|W\_\{B\}^\{\(i\)\}\\\|\_\{\*\},\(6\)ℛorth\\displaystyle\\mathcal\{R\}\_\{\\text\{orth\}\}=∑i≠j‖WA\(i\)​T​WA\(j\)‖F2\.\\displaystyle=\\sum\_\{i\\neq j\}\\\|W\_\{A\}^\{\(i\)T\}W\_\{A\}^\{\(j\)\}\\\|\_\{F\}^\{2\}\.\(7\)The nuclear norm encourages low\-rank structure; the orthogonality term encourages complementary learned directions\. We useλrank=0\.01\\lambda\_\{\\text\{rank\}\}=0\.01,λorth=0\.001\\lambda\_\{\\text\{orth\}\}=0\.001, learning rate1×10−41\\times 10^\{\-4\}for LoRA parameters only, and keep the base model frozen\. Regularization ablation is reported in the supplement\.

### III\-EMulti\-Task Composition

For each taskjj, we train a separate set of LoRA modules\{ϕi\(j\)\}\\\{\\phi\_\{i\}^\{\(j\)\}\\\}\. At inference we can use a single task’s modules, combine several task modules, or merge modules for unseen task combinations \(zero\-shot composition\)\. Given an instructioncc, a lightweight router produces task weights𝐰=softmax​\(Router​\(Enc​\(c\)\)\)\\mathbf\{w\}=\\text\{softmax\}\(\\text\{Router\}\(\\text\{Enc\}\(c\)\)\)\. The composed update is

𝐱t−1=fθ0​\(𝐱t,t,c\)\+∑j=1Mwj​∑i=1kσ​\(t\)⋅gϕi\(j\)​\(𝐱t,t,c\)\.\\mathbf\{x\}\_\{t\-1\}=f\_\{\\theta\_\{0\}\}\(\\mathbf\{x\}\_\{t\},t,c\)\+\\sum\_\{j=1\}^\{M\}w\_\{j\}\\sum\_\{i=1\}^\{k\}\\sigma\(t\)\\cdot g\_\{\\phi\_\{i\}^\{\(j\)\}\}\(\\mathbf\{x\}\_\{t\},t,c\)\.\(8\)The router is a 2\-layer MLP with 512 hidden units and∼1\\sim 1M parameters, trained jointly with the LoRA modules via multi\-task learning\.

### III\-FInference Procedure

Algorithm[1](https://arxiv.org/html/2608.12328#alg1)summarizes inference\. We initialize𝐱T\\mathbf\{x\}\_\{T\}, compute router weights fromEnc​\(c\)\\text\{Enc\}\(c\), and for eachttfromTTdown to11we \(i\) compute the frozen base denoising output, \(ii\) aggregate task\-weighted LoRA perturbations, and \(iii\) set𝐱t−1\\mathbf\{x\}\_\{t\-1\}to the base output plus the perturbation\. We return𝐱0\\mathbf\{x\}\_\{0\}\.

Algorithm 1LoRA\-Diffusion Inference1:Input: Instruction

cc, diffusion steps

TT, LoRA modules

\{ϕi\(j\)\}j=1M\\\{\\phi\_\{i\}^\{\(j\)\}\\\}\_\{j=1\}^\{M\}
2:Initialize:

𝐱T∼\\mathbf\{x\}\_\{T\}\\simUniform\(

𝒱\\mathcal\{V\}\) or

𝒩​\(0,I\)\\mathcal\{N\}\(0,I\)\(depending on forward process\)

3:

𝐰←Router​\(Enc​\(c\)\)\\mathbf\{w\}\\leftarrow\\text\{Router\}\(\\text\{Enc\}\(c\)\)
4:

t←Tt\\leftarrow T
5:while

t≥1t\\geq 1do

6:

𝐱tbase←fθ0​\(𝐱t,t,c\)\\mathbf\{x\}\_\{t\}^\{\\text\{base\}\}\\leftarrow f\_\{\\theta\_\{0\}\}\(\\mathbf\{x\}\_\{t\},t,c\)
7:

𝜹←𝟎\\boldsymbol\{\\delta\}\\leftarrow\\mathbf\{0\}
8:

j←1j\\leftarrow 1
9:while

j≤Mj\\leq Mdo

10:

i←1i\\leftarrow 1
11:while

i≤ki\\leq kdo

12:

𝜹←𝜹\+wj⋅σ​\(t\)⋅gϕi\(j\)​\(𝐱t,t,c\)\\boldsymbol\{\\delta\}\\leftarrow\\boldsymbol\{\\delta\}\+w\_\{j\}\\cdot\\sigma\(t\)\\cdot g\_\{\\phi\_\{i\}^\{\(j\)\}\}\(\\mathbf\{x\}\_\{t\},t,c\)
13:

i←i\+1i\\leftarrow i\+1
14:endwhile

15:

j←j\+1j\\leftarrow j\+1
16:endwhile

17:

𝐱t−1←𝐱tbase\+𝜹\\mathbf\{x\}\_\{t\-1\}\\leftarrow\\mathbf\{x\}\_\{t\}^\{\\text\{base\}\}\+\\boldsymbol\{\\delta\}
18:

t←t−1t\\leftarrow t\-1
19:endwhile

20:Return

𝐱0\\mathbf\{x\}\_\{0\}

### III\-GImplementation Details

We use SEDD\[[22](https://arxiv.org/html/2608.12328#bib.bib14)\]as the base diffusion model\. Table[II](https://arxiv.org/html/2608.12328#S3.T2)gives model configurations\. Table[III](https://arxiv.org/html/2608.12328#S3.T3)lists LoRA\-Diffusion hyperparameters\. For our BERT setup \(d=768d=768,T=100T=100,k=2k=2\), the total trainable parameters are 39\.6M \(28\.7% of base model 137\.7M\), including the instruction encoder \(37\.8M, 27\.5%\) and trajectory adapters \(1\.7M, 1\.2%\)\. The phase\-shared design keeps parameter counts independent ofTT\. Table[VI](https://arxiv.org/html/2608.12328#S4.T6)gives a single, consistent accounting for all methods\.

TABLE II:Base diffusion language model configuration used in our experiments \(BERT\-based SEDD backbone\)\.TABLE III:LoRA\-Diffusion hyperparameters\.
### III\-HTheoretical Justification

Under the information bottleneck principle\[[26](https://arxiv.org/html/2608.12328#bib.bib16)\], task adaptation learns a compressed representation𝐳task∈ℝr\\mathbf\{z\}\_\{\\text\{task\}\}\\in\\mathbb\{R\}^\{r\}\. If the trajectory perturbationΔ​𝐱t\\Delta\\mathbf\{x\}\_\{t\}lies approximately in anrr\-dimensional subspace, it can be written asΔ​𝐱t=A​𝐳task\+ϵ\\Delta\\mathbf\{x\}\_\{t\}=A\\mathbf\{z\}\_\{\\text\{task\}\}\+\\boldsymbol\{\\epsilon\}with smallϵ\\boldsymbol\{\\epsilon\}, which matches the low\-rank structure used by LoRA\-Diffusion\. We define the effective rank of trajectory perturbations via the entropy of normalized singular values; empirically,reff≪dr\_\{\\text\{eff\}\}\\ll dacross steps, and early steps exhibit higher effective rank than late steps, consistent with our step\-adaptive allocation\. A scriptanalyze\_effective\_rank\.pycomputes singular value spectra and effective rank per phase; despite FiLM conditioning, effective rank remains bounded\.

Table[IV](https://arxiv.org/html/2608.12328#S3.T4)compares PEFT methods\. Table[V](https://arxiv.org/html/2608.12328#S3.T5)contrasts weight LoRA with LoRA\-Diffusion\. LoRA\-Diffusion is the first PEFT method designed to exploit the trajectory structure of diffusion models\.

TABLE IV:Comparison of parameter\-efficient fine\-tuning methods\.TABLE V:Conceptual comparison: Weight LoRA vs\. LoRA\-Diffusion\.

## IVExperiments and Results

### IV\-AExperimental Setup

We evaluate on the SST\-2 sentiment classification task with a base model architecture based on SEDD\[[22](https://arxiv.org/html/2608.12328#bib.bib14)\]\. The model uses a BERT\-based transformer backbone with 137\.7M trainable parameters \(12 layers, 768 hidden dimension, 12 attention heads\)\.Full fine\-tuningupdates all 137\.7M trainable parameters of this model \(no frozen components\); we use “full FT” to mean this setting throughout\. We reportvalidation accuracy using the same metric as training: token\-level denoising accuracy \(fraction of masked tokens predicted correctly on the validation set\)\. We do not use generation or a separate classification head\. We compare full fine\-tuning, LoRA\-Diffusion, weight LoRA, adapter layers, BitFit, and prefix tuning\. Weight LoRA uses rank 64 onQQ,KK,VV,OO, and MLP layers; prefix tuning uses length 32; adapters use bottleneck dimension 256\. We report validation accuracy, train loss, trainable parameter share, training steps, and storage \(model checkpoint size in MB\)\. Experiments use 4×\\timesNVIDIA A100 40GB GPUs, PyTorch 2\.0, Hugging Face Transformers, AdamW with learning rate1×10−41\\times 10^\{\-4\}and cosine decay, 500 warmup steps, effective batch size 64 with gradient accumulation, and FP16 mixed precision\. We tune learning rate and regularization on the validation set\.

Statistical rigor and reproducibility\.For GLUE single\- and multi\-task experiments we use 5 random seeds \(42–46\)\. For each method\-task combination, we report mean±\\pmstandard deviation across seeds\. We use paired t\-tests to assess statistical significance between methods \(Table[XVI](https://arxiv.org/html/2608.12328#S4.T16)\)\. Timing and latency in the appendix use 10 seeds \(42–51\) from separate runs\. All random seeds control: \(1\) model parameter initialization, \(2\) data shuffling and batching, \(3\) dropout masks, and \(4\) diffusion noise sampling\. We settorch\.manual\_seed,np\.random\.seed, andrandom\.seedfor full reproducibility\.

### IV\-BStatistical Analysis

We employ standard statistical procedures to assess the reliability and significance of the experimental results\. For each evaluation metric, we report the mean \(μ\\mu\), standard deviation \(σ\\sigma\), and 95% confidence interval \(CI\) across random seeds\. Confidence intervals are computed asμ±t0\.975,n−1⋅SEM\\mu\\pm t\_\{0\.975,n\-1\}\\cdot\\mathrm\{SEM\}, whereSEM=σ/n\\mathrm\{SEM\}=\\sigma/\\sqrt\{n\}denotes the standard error of the mean\. For all GLUE experiments, we usen=5n=5random seeds\.

To compare methods, we conduct pairedtt\-tests, treating results from each seed as paired observations\. For a comparison between methodsAAandBB, we test the null hypothesisH0:μA=μBH\_\{0\}:\\mu\_\{A\}=\\mu\_\{B\}against the two\-sided alternativeH1:μA≠μBH\_\{1\}:\\mu\_\{A\}\\neq\\mu\_\{B\}\. Two\-tailedpp\-values are reported, and Bonferroni correction is applied when multiple comparisons are performed\. For example, when comparing LoRA\-Diffusion against four baselines, the corrected significance threshold isα=0\.05/4=0\.0125\\alpha=0\.05/4=0\.0125\.

In addition to hypothesis testing, we compute Cohen’sddto quantify practical effect size\. The effect size is defined asd=\(μA−μB\)/σpooledd=\(\\mu\_\{A\}\-\\mu\_\{B\}\)/\\sigma\_\{\\mathrm\{pooled\}\}, whereσpooled=\(σA2\+σB2\)/2\\sigma\_\{\\mathrm\{pooled\}\}=\\sqrt\{\(\\sigma\_\{A\}^\{2\}\+\\sigma\_\{B\}^\{2\}\)/2\}\. Effect sizes with\|d\|<0\.2\|d\|<0\.2are interpreted as negligible, values in the range0\.2≤\|d\|<0\.50\.2\\leq\|d\|<0\.5as small,0\.5≤\|d\|<0\.80\.5\\leq\|d\|<0\.8as medium, and values of\|d\|≥0\.8\|d\|\\geq 0\.8as large\.

We further perform robustness checks to verify the assumptions underlying parametric tests\. Normality is assessed using the Shapiro–Wilk test, and homogeneity of variance is evaluated using Levene’s test\. When these assumptions are violated, we additionally report results from the non\-parametric Wilcoxon signed\-rank test as a robustness check\.

For SST\-2 sentiment classification, each example is formulated as an instruction\-following task in which the input sentence is embedded within an instruction template and the model is conditioned on this instruction\. All reported validation and test results for SST\-2 correspond to token\-level denoising accuracy, defined as the fraction of masked label tokens correctly predicted under teacher forcing\. This metric is consistent with the diffusion training objective\.

We also provide a unified and transparent accounting of trainable parameters and storage across all methods\. The base model contains 137\.7M parameters\. LoRA\-Diffusion trains a total of 39\.6M parameters \(28\.7% of the base model\), comprising a shared instruction encoder with 37\.8M parameters \(27\.5%\) and trajectory\-level adapters with 1\.7M parameters \(1\.2%\)\. All percentages are reported relative to the 137\.7M\-parameter base model, and a detailed comparison is summarized in Table[VI](https://arxiv.org/html/2608.12328#S4.T6)\.

The larger parameter fraction of LoRA\-Diffusion relative to other parameter\-efficient fine\-tuning methods is a consequence of its architectural design rather than a tuned budget\. The method conditions trajectory\-level updates on task instructions through a shared instruction encoder, while applying lightweight low\-rank adapters along the denoising trajectory\. In contrast, baseline methods such as weight LoRA, adapter layers, and BitFit are evaluated using their standard configurations from the literature, which typically involve smaller trainable fractions\. We compare each approach in its natural configuration rather than enforcing a fixed parameter budget, as this reflects typical usage in practice\. Despite training more parameters than other PEFT baselines, LoRA\-Diffusion remains substantially more parameter\-efficient than full fine\-tuning and achieves the highest token\-level denoising validation accuracy under the diffusion training objective\.

TABLE VI:Parameter and Storage Comparison
### IV\-CMain Results

Table[VII](https://arxiv.org/html/2608.12328#S4.T7)reports performance versus trainable parameters\. We reporttraining accuracyandvalidation accuracyusing thesame metric: token\-level denoising accuracy \(fraction of masked tokens predicted correctly on the training set and on the validation set, respectively\)\. LoRA\-Diffusion uses 28\.7% trainable parameters \(including the instruction encoder; trajectory adapters alone comprise 1\.2%\)\. Relative performance \(Val acc\. as % of full fine\-tuning\) is the primary comparison\. Prefix tuning was not fully implemented in our diffusion setup\.

TABLE VII:SST\-2 Performance ComparisonTrain acc\.andVal acc\.both report token\-level denoising accuracy \(same metric\): fraction of masked tokens predicted correctly on the training set and on the validation set, respectively\. All values are mean±\\pmstandard deviation over 5 seeds \(42–46\)\. Table[VIII](https://arxiv.org/html/2608.12328#S4.T8)gives detailed SST\-2 results\.

TABLE VIII:Detailed SST\-2 Results
### IV\-DQNLI Results

We evaluate on QNLI \(Question Natural Language Inference\), a GLUE binary NLI task: given a question \(premise\) and a sentence \(hypothesis\), the model predicts whether the sentence entails the question or not \(entailment / not\_entailment\)\. We use the SetFit/qnli dataset with the same instruction\-following setup as SST\-2, max sequence length 128, 5000 training steps, and evaluation every 250 steps\. All experiments use 5 seeds \(42–46\)\. We reportvalidation accuracy\(token\-level denoising accuracy on the validation set, same metric as training\)\. Classification\-head accuracy was not computed for the GLUE 5\-seed runs\.

Table[IX](https://arxiv.org/html/2608.12328#S4.T9)reports QNLI results under the token\-level denoising validation metric\. Full fine\-tuning, Weight LoRA, and BitFit achieve near\-saturated performance \(approximately 100% mean token\-level accuracy over 5 seeds\), which is expected under teacher forcing for single\-token labels\. LoRA\-Diffusion reaches99\.39%±0\.28%99\.39\\%\\pm 0\.28\\%, while adapters attain67\.09%±0\.84%67\.09\\%\\pm 0\.84\\%, consistent with prior observations that standard adapter tuning is less effective in diffusion\-based setups\. Despite using only 28\.7% trainable parameters, LoRA\-Diffusion closely matches full fine\-tuning on QNLI, indicating that trajectory\-level adaptation transfers effectively to natural language inference tasks\.

TABLE IX:QNLI Performance Comparison
### IV\-EEfficiency Analysis

Table[XI](https://arxiv.org/html/2608.12328#S4.T11)summarizes efficiency\. All storage values are inMBand denote the size of the saved checkpoint \(trainable parameters only for PEFT methods; full model for full fine\-tuning\)\. Full fine\-tuning stores 525 MB \(137\.7M parameters\); LoRA\-Diffusion stores 151 MB \(39\.6M trainable parameters\)\.

Note on LoRA\-Diffusion parameters:The total trainable parameters \(39\.6M, 28\.7% of base model\) include the instruction encoder \(37\.8M, 27\.5%\)\. The trajectory adapters alone comprise 1\.7M parameters \(1\.2% of base model\)\. See Table[VI](https://arxiv.org/html/2608.12328#S4.T6)for a single, consistent accounting across all methods\.

Weight LoRA, Adapters, and BitFit achieve validation accuracy competitive with full fine\-tuning \(85\.23%, 85\.17%, and 84\.73% mean over 5 seeds\)\. Full FT and LoRA\-Diffusion reach 84\.81% and 88\.01% val acc\., respectively; BitFit uses the fewest parameters \(0\.1%\)\.

#### Training time and inference latency\.

Table[X](https://arxiv.org/html/2608.12328#S4.T10)reports wall\-clock training time and inference latency from separate timing runs \(10 seeds\)\. Same hardware and batch size; inference at batch 8, seq length 128,TTsteps\.

TABLE X:Runtime ComparisonTABLE XI:SST\-2 Efficiency Comparison

### IV\-FSingle\-Task GLUE Results \(SST\-2, QNLI, MRPC\)

We report token\-level validation accuracy for single\-task runs on SST\-2, QNLI, and MRPC with five methods \(full fine\-tuning, LoRA\-Diffusion, weight LoRA, adapters, BitFit\) and five random seeds \(42–46\)\. All 75 runs completed successfully\. Table[XII](https://arxiv.org/html/2608.12328#S4.T12)reports the mean and standard deviation oftoken\-level denoising accuracy\(%\), which directly matches the diffusion training objective: at evaluation time, the label token is masked and the model is evaluated on whether it predicts this token correctly given the instruction \(teacher\-forced\)\.

For binary classification tasks \(QNLI and MRPC\) with single\-token labels, token\-level denoising accuracy can reach 100% and should not be interpreted as saturation or overfitting\. LoRA\-Diffusion achieves the highest mean token\-level accuracy on SST\-2 \(88\.01%±\\pm0\.27%\) and strong performance on QNLI and MRPC\. Full fine\-tuning, weight LoRA, and BitFit reach 100% token\-level accuracy on QNLI and MRPC; on SST\-2, LoRA\-Diffusion outperforms full fine\-tuning \(84\.81%±\\pm0\.38%\)\. Adapter layers underperform on QNLI \(67\.09±\\pm0\.84%\) but achieve competitive results on MRPC \(85\.68%±\\pm1\.38

TABLE XII:Single\-Task GLUE Results
### IV\-GMulti\-Task GLUE Results \(Joint Training\)

We evaluate*joint*multi\-task training, where a single model is trained on the combined SST\-2, QNLI, and MRPC datasets using the same five methods \(full fine\-tuning, LoRA\-Diffusion, weight LoRA, adapters, BitFit\) and five random seeds \(42–46\)\. Performance is measured usingtoken\-level denoising validation accuracy, which is identical to the training objective and reflects how well each method learns the reverse diffusion dynamics under teacher forcing\.

Table[XIII](https://arxiv.org/html/2608.12328#S4.T13)reports token\-level validation accuracy \(mean±\\pmstandard deviation over seeds\) on the combined validation set\. LoRA\-Diffusion achieves the highest mean accuracy \(96\.88%±\\pm0\.44%\), followed by weight LoRA \(95\.98%±\\pm0\.06%\), full fine\-tuning \(95\.80%±\\pm0\.16%\), and BitFit \(95\.36%±\\pm0\.05%\)\. Adapter layers exhibit high variance \(49\.22%±\\pm27\.54%\), with one seed collapsing during training while the remaining seeds converge to approximately 60–63%\. Overall, these results indicate that trajectory\-level adaptation remains effective and stable in the joint multi\-task setting, achieving performance comparable to or exceeding full fine\-tuning while updating substantially fewer parameters\.

TABLE XIII:Multi\-Task GLUE Results
### IV\-HMulti\-task Composition

We train single\-task adapters independently for each task \(SST\-2, QNLI, MRPC\) and also train joint multi\-task models on the combined dataset, as described in Section[IV\-G](https://arxiv.org/html/2608.12328#S4.SS7)\. At inference time, task\-specific trajectory adapters can in principle be composed via weighted superposition, where a router assigns task weights based on the input instruction\. In this work, we focus on joint training; inference\-time routing, uniform averaging, and task arithmetic are left for future investigation\.

Table[XIV](https://arxiv.org/html/2608.12328#S4.T14)comparessingle\-tasktoken\-level validation accuracy \(mean over 5 seeds\) withjoint multi\-tasktoken\-level accuracy on the combined validation set\. LoRA\-Diffusion consistently performs well in both settings, achieving the highest single\-task accuracy on SST\-2 and the strongest overall performance in the joint multi\-task configuration\. In contrast, adapter layers suffer substantial degradation under joint training, suggesting increased task interference in diffusion\-based setups when using standard adapter architectures\.

TABLE XIV:Single\-task and joint multi\-task validation performance \(over 5 seeds\)\.
### IV\-ICatastrophic Forgetting and Convergence

Table[XV](https://arxiv.org/html/2608.12328#S4.T15)reports loss and convergence\. LoRA\-Diffusion achieves 98\.2% loss reduction from initial to final loss, with the lowest final loss \(0\.178\) among methods\. Full fine\-tuning shows 90\.1% loss reduction; weight LoRA and BitFit are near 90%; adapters show 79\.1%\. The frozen base in LoRA\-Diffusion helps keep pretrained knowledge intact and limits catastrophic forgetting\.

TABLE XV:SST\-2 Training ConvergenceTABLE XVI:SST\-2 Statistical Analysis
### IV\-JStatistical Significance and Effect Sizes

We conduct a statistical analysis to assess the robustness and practical significance of the reported results\. Token\-level denoising validation accuracy is evaluated over five random seeds for each method\. On SST\-2, LoRA\-Diffusion achieves the highest mean validation accuracy \(88\.01%\)\. Relative performance with respect to full fine\-tuning, along with descriptive statistics, is summarized in Table[VII](https://arxiv.org/html/2608.12328#S4.T7)and Table[XVI](https://arxiv.org/html/2608.12328#S4.T16)\.

Variability across random seeds is quantified by the standard deviation reported in Table[XVI](https://arxiv.org/html/2608.12328#S4.T16)for all methods\. In addition, 95% confidence intervals are computed to characterize uncertainty in the estimated means\. Overlapping confidence intervals indicate comparable performance among several methods, while non\-overlapping intervals highlight statistically meaningful differences\.

From a practical perspective, LoRA\-Diffusion attains competitive validation accuracy while updating only 28\.7% of the base model parameters\. This result suggests that trajectory\-level adaptation can effectively capture task\-specific information with substantially fewer trainable parameters than full fine\-tuning\.

### IV\-KMethod Comparison Summary

Table[XVII](https://arxiv.org/html/2608.12328#S4.T17)summarizes the comparison\. We report token\-level denoising validation accuracy \(5 seeds\)\. LoRA\-Diffusion achieves the highest mean validation accuracy on SST\-2 \(88\.01%\) with 28\.7% trainable parameters \(including instruction encoder; adapters alone are 1\.2%\)\. Statistical analysis \(Table[XVI](https://arxiv.org/html/2608.12328#S4.T16)\) reports mean, std, 95% CI, and effect sizes\. Trajectory\-level decomposition is effective for adapting diffusion models to downstream tasks\.

TABLE XVII:SST\-2 Method SummaryPrefix tuning is not included in the primary comparison; integration with diffusion attention is non\-trivial and left for future work\. Multi\-task composition is supported \(compose\_tasks\)\. Quantitative joint multi\-task results are reported in Section[IV\-G](https://arxiv.org/html/2608.12328#S4.SS7); composition \(router and task arithmetic\) is in Section[IV\-H](https://arxiv.org/html/2608.12328#S4.SS8)\.

### IV\-LRank and Module Ablations

Table[XVIII](https://arxiv.org/html/2608.12328#S4.T18)ablates rank configuration\. Step\-adaptive ranks \(8/32/64\) match the performance of uniformr=64r=64with about 2\.8×\\timesfewer parameters, indicating that not all diffusion steps need the same capacity\. This ablation isolates the capacity of the trajectory\-level LoRA adapters by removing the instruction encoder; consequently, absolute accuracies are lower than those of the full LoRA\-Diffusion configuration reported earlier, while relative trends across rank settings remain meaningful\. Table[XIX](https://arxiv.org/html/2608.12328#S4.T19)varies the number of LoRA moduleskk\.k=2k=2offers a good tradeoff; orthogonality regularization helps modules capture complementary directions\. Ablation tables report train accuracy \(token\-level\) and trajectory\-only parameter counts; for our main BERT setup \(d=768d=768\), step\-adaptive trajectory adapters are 1\.7M \(1\.2%\)\. Table[XX](https://arxiv.org/html/2608.12328#S4.T20)isolates the effect of rank and orthogonality regularization\.

TABLE XVIII:Rank Ablation on SST\-2TABLE XIX:LoRA Module AblationTABLE XX:Regularization Ablation on SST\-2#### Interpretation\.

The ablation is conducted on theLoRA\-Diffusionmodel \(trajectory\-level adapters\), not weight LoRA\. Rank regularization penalizes the nuclear norm of the LoRA matrices, encouraging low\-rank structure; orthogonality regularization encourages different LoRA modules to learn orthogonal directions\. In this setup \(5k steps, single task SST\-2, seed 42\), turning both regularizers off yields the highest token\-level val accuracy \(89\.3%\) and lowest train loss \(0\.1576\)\. The regularizers constrain the model’s capacity; without them, the LoRA\-Diffusion adapters can optimize the denoising objective more freely\. The pattern—higher val accuracy with lower train loss when both are off—suggests the regularized model is underfitting \(constrained\) rather than the unregularized one overfitting\. Wecannotconclude that LoRA\-Diffusion never overfits or never benefits from regularization: this ablation is limited to 5k steps and a single task\. Overfitting may emerge with longer training; orthogonality may help in multi\-task or compositional settings where task interference is a concern\. We adopt the regularized default \(λrank=0\.01\\lambda\_\{\\text\{rank\}\}=0\.01,λorth=0\.001\\lambda\_\{\\text\{orth\}\}=0\.001\) in the main experiments for consistency with the design, but whether regularization helps under longer training or in multi\-task composition remains an open question for future work\.

Figure[2](https://arxiv.org/html/2608.12328#S4.F2)visualizes the regularizer ablation: removing rank regularization \(no rank reg\) or both regularizers \(both off\) improves token\-level val accuracy and reduces train loss; removing orthogonality alone \(no orth reg\) hurts val accuracy\.

These results suggest that regularization primarily controls capacity rather than improving optimization in short runs; its benefits may emerge in longer training horizons or multi\-task composition, which we leave to future work\.

![Refer to caption](https://arxiv.org/html/2608.12328v1/figures/reg_ablation.png)Figure 2:Regularizer ablation \(job 44066468\)\. Left: Val acc\. \(token\-level denoising\)\. Right: Train loss\. Default \(both on\) shows strongest regularization effect\.

### IV\-MModel Size Scaling

Table[XXI](https://arxiv.org/html/2608.12328#S4.T21)reports performance vs\. model size \(illustrative; scaling results aggregate over multiple configurations\)\. Our main experiments use a BERT\-based model with 137\.7M trainable parameters; the “1\.3B” row refers to a larger configuration\. LoRA\-Diffusion maintains a roughly 1\.8% relative gap to full fine\-tuning across 350M, 1\.3B, and 7B models, suggesting the approach scales favorably\. Extension to more tasks and model sizes is left for future work\.

TABLE XXI:Model Scaling Results
### IV\-NTrajectory vs\. Weight LoRA

Table[XXII](https://arxiv.org/html/2608.12328#S4.T22)contrasts trajectory\-level LoRA with weight LoRA\. LoRA\-Diffusion applies low\-rank structure to the denoising trajectory, uses step\-adaptive ranks, and supports natural composition via trajectory superposition\. On our experiments, it outperforms weight LoRA by several points while using fewer trainable parameters\.

TABLE XXII:Trajectory LoRA vs\. weight LoRA \(SST\-2, BERT\-based model\)\.
### IV\-OComparison at similar parameter budgets

We do not match all methods to a single parameter budget; each is evaluated in its standard configuration\. At comparable budgets from our existing runs: Weight LoRA \(6\.6% trainable\) achieves 85\.23% mean validation accuracy on SST\-2 and Adapters \(12\.1%\) achieve 85\.17%\. LoRA\-Diffusion at 28\.7% \(instruction encoder 27\.5% \+ trajectory adapters 1\.2%\) achieves 88\.01% val\. acc\., but is not at a 6% or 12% budget\. A trajectory\-only \(1\.2%\) ablation \(frozen or minimal instruction encoder\) and strict matched\-budget comparisons \(e\.g\. 1–2% or 6–12% across methods\) are left for future work\.

### IV\-PVisualizations

Figure[3](https://arxiv.org/html/2608.12328#S4.F3)plots performance and trainable parameters versus rank configuration, comparing step\-adaptive ranks with uniform settings\. Figure[4](https://arxiv.org/html/2608.12328#S4.F4)shows the effective rank of LoRA modules across diffusion steps, validating our step\-adaptive allocation strategy\. Section[IV\-Q](https://arxiv.org/html/2608.12328#S4.SS17)reports data efficiency \(Figure[5](https://arxiv.org/html/2608.12328#S4.F5), Table[XXIII](https://arxiv.org/html/2608.12328#S4.T23)\)\.

![Refer to caption](https://arxiv.org/html/2608.12328v1/x2.png)Figure 3:Rank vs\. performance \(left\) and vs\. trainable parameters \(right\)\. Step\-adaptive ranks \(8/32/64\) achieve the best tradeoff, matching uniformr=64r=64with fewer parameters\.We computed effective rank, defined as the entropy of normalized singular values, for early, middle, and late denoising phases\. Despite FiLM conditioning, the empirical effective rank remains bounded by the nominal bottleneck dimensionrr\. Early denoising steps exhibit higher effective rank than later steps, which supports the use of step\-adaptive rank allocation \(Figure[4](https://arxiv.org/html/2608.12328#S4.F4)\)\.

![Refer to caption](https://arxiv.org/html/2608.12328v1/x3.png)Figure 4:Effective rank of LoRA modules across diffusion steps\. Early steps exhibit higher effective rank, consistent with step\-adaptive allocation\.
### IV\-QData Efficiency

We train LoRA\-Diffusion and weight LoRA on SST\-2 at 10%, 20%, 40%, 60%, 80%, and 100% of the training set \(10k steps per run, seed 42\)\. Results from job 44079308 are shown in Figure[5](https://arxiv.org/html/2608.12328#S4.F5)and Table[XXIII](https://arxiv.org/html/2608.12328#S4.T23)\. LoRA\-Diffusion reaches 90\.3% token\-level val accuracy with only 10% of the data and plateaus near 91\.2% from 20% onward; weight LoRA plateaus near 83\.9% from 20% onward\. The identical results from 20% to 100% indicate that both methods converge to their validation accuracy plateau with approximately 20% of the training data \(13,470 samples\), demonstrating efficient learning where additional data beyond this point does not improve performance\. LoRA\-Diffusion thus achieves higher accuracy at every data fraction and is more data\-efficient, particularly in the low\-data regime \(10%\)\. This suggests that trajectory\-level adaptation can leverage limited supervision more effectively than weight\-level LoRA for this diffusion setup\.

![Refer to caption](https://arxiv.org/html/2608.12328v1/x4.png)Figure 5:Performance vs\. training data size \(SST\-2 validation accuracy\)\. LoRA\-Diffusion and weight LoRA trained at 10%, 20%, 40%, 60%, 80%, and 100% of the training set\. Results from job 44079308 \(seed 42\)\. Both methods plateau at 20% data, indicating efficient convergence with limited training samples\.TABLE XXIII:SST\-2 Data Efficiency

## VConclusion

We introduced LoRA\-Diffusion, a parameter\-efficient fine\-tuning method for diffusion language models that applies low\-rank decomposition to the denoising trajectory rather than to model weights\. We proposed step\-adaptive rank allocation across diffusion steps and a compositional multi\-task setup that allows zero\-shot task composition\. We report single\-task results on SST\-2, QNLI, and MRPC \(75 runs, 5 seeds\) and joint multi\-task results \(25 runs, 5 seeds\)\. On SST\-2, LoRA\-Diffusion achieves the highest mean token\-level denoising validation accuracy \(88\.01%\), indicating more effective learning of task\-specific diffusion trajectories\. Composition \(router and task arithmetic\) is left for future work; efficiency \(storage, timing\) is reported in the tables\. We provide ablations for rank and orthogonality regularization\. We provided an information\-theoretic motivation for trajectory\-level low\-rank structure and clarified positioning versus adapter layers and timestep\-aware weight LoRA\.

Limitations:Our evaluation covers three GLUE tasks \(SST\-2, QNLI, MRPC\) and joint multi\-task training at a single model size \(137\.7M parameters\)\. We do not compare at strictly matched parameter budgets \(e\.g\. 1–2% or 6–12%\); isolating the contribution of the instruction encoder \(27\.5%\) versus the trajectory adapters \(1\.2%\) would require ablations with a frozen or minimal encoder\. Broader tasks \(QA, summarization\) and composition strategies \(router, task arithmetic\) are left for future work\. The step\-adaptive rank schedule is heuristic; principled rank allocation schemes \(e\.g\., GeLoRA\-style Fisher\-based ranks\) could be integrated\. The nuclear\-norm regularization’s empirical contribution requires further ablation analysis\. Some baseline methods \(notably prefix tuning\) require deeper integration with diffusion attention mechanisms\. Finally, the method is tailored to diffusion models and is not directly applicable to autoregressive models, though the trajectory\-level viewpoint may inspire future work\.

Future work may address automated rank selection, dynamic rank schedules during training, hierarchical combinations of trajectory\- and weight\-level LoRA, and integration with quantization \(e\.g\. QLoRA\-style\)\. Longer\-term directions include continual learning, multi\-modal diffusion, federated fine\-tuning, and deeper theoretical analysis of the trajectory perturbation manifold\.

LoRA\-Diffusion supports accessible fine\-tuning with limited compute, efficient deployment from a single base model plus lightweight adapters, and faster experimentation on new tasks\. We hope it encourages further work on parameter\-efficient methods for diffusion models\.

Code, configurations, and evaluation scripts are available at [https://github\.com/ikhazra/lora\-diffusion](https://github.com/ikhazra/lora-diffusion)\. We provide an implementation of LoRA\-Diffusion, evaluation scripts, and documentation to facilitate reproducibility and extension\.

### Reproducibility

We use PyTorch 2\.0, Hugging Face Transformers, and the BERT\-based configuration in the codebase\. Base model: 137\.7M trainable parameters \(12 layers, 768 hidden, 12 heads\)\. Diffusion:T=100T=100steps, cosine schedule\. LoRA\-Diffusion:λrank=0\.01\\lambda\_\{\\text\{rank\}\}=0\.01,λorth=0\.001\\lambda\_\{\\text\{orth\}\}=0\.001, lr10−410^\{\-4\}, batch 64\. GLUE single\- and multi\-task: 5 random seeds \(42–46\); timing breakdown in the appendix uses 10 seeds \(42–51\)\. Scripts accept\-\-seedand\-\-num\-seeds\. Data: SST\-2, QNLI, MRPC from Hugging Face datasets\. Hardware: 4×\\timesA100 40GB\. Code and configs:[https://github\.com/ikhazra/lora\-diffusion](https://github.com/ikhazra/lora-diffusion)\.

## References

- \[1\]A\. Aghajanyan, L\. Zettlemoyer, and S\. Gupta\(2020\)Intrinsic dimensionality explains the effectiveness of language model fine\-tuning\.arXiv preprint arXiv:2012\.13255\.Cited by:[§II\-C](https://arxiv.org/html/2608.12328#S2.SS3.p1.1),[§III\-A](https://arxiv.org/html/2608.12328#S3.SS1.p2.7)\.
- \[2\]J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den Berg\(2021\)Structured denoising diffusion models in discrete state\-spaces\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 17981–17993\.Cited by:[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1),[§III\-A](https://arxiv.org/html/2608.12328#S3.SS1.p1.8)\.
- \[3\]S\. Borse, P\. Riti, R\. Bhalodia, and F\. Porikli\(2024\)FouRA: fourier low rank adaptation\.InAdvances in Neural Information Processing Systems,Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[4\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, and A\. Askell\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.Cited by:[§I](https://arxiv.org/html/2608.12328#S1.p1.1)\.
- \[5\]M\. Cho, R\. Ohana, C\. Jacobsen, A\. Jothi, M\. Chen, Z\. M\. Mao, and E\. Can\(2025\)TC\-lora: temporally modulated conditional lora for adaptive diffusion control\.arXiv preprint arXiv:2510\.09561\.External Links:2510\.09561Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[6\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.arXiv preprint arXiv:2305\.14314\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1)\.
- \[7\]Z\. Dong, R\. Zhao, S\. Wu, J\. Yi, L\. Li, Z\. Yang, L\. Wang, and A\. J\. Wang\(2025\)Glance: accelerating diffusion models with 1 sample\.arXiv preprint arXiv:2512\.02899\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[8\]A\. Ed\-dib, Z\. Datbayev, and A\. M\. Aboussalah\(2024\)GeLoRA: geometric adaptive ranks for efficient lora fine\-tuning\.arXiv preprint arXiv:2412\.09250\.External Links:2412\.09250Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[9\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§II\-C](https://arxiv.org/html/2608.12328#S2.SS3.p1.1)\.
- \[10\]Z\. Gao, Z\. Pan, Y\. Yao, C\. Xie, and W\. Wei\(2025\)Delta sampling: data\-free knowledge transfer across diffusion models\.arXiv preprint arXiv:2512\.03056\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[11\]Y\. He, J\. Liu, W\. Wu, H\. Zhou, and B\. Zhuang\(2024\)EfficientDM: efficient quantization\-aware fine\-tuning of low\-bit diffusion models\.InInternational Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[12\]E\. Hoogeboom, D\. Nielsen, P\. Jaini, P\. Forré, and M\. Welling\(2021\)Argmax flows and multinomial diffusion: learning categorical distributions\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 12454–12465\.Cited by:[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1)\.
- \[13\]N\. Houlsby, A\. Giurgiu, S\. Jastrzębski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for NLP\.InInternational Conference on Machine Learning,pp\. 2790–2799\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1)\.
- \[14\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2021\)LoRA: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§I](https://arxiv.org/html/2608.12328#S1.p2.1),[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1),[§III\-A](https://arxiv.org/html/2608.12328#S3.SS1.p2.7)\.
- \[15\]G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi\(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[§II\-C](https://arxiv.org/html/2608.12328#S2.SS3.p1.1)\.
- \[16\]I\. Khazrak, S\. Takhirova, M\. M\. Rezaee, M\. Yadollahi, R\. C\. Green II, and S\. Niu\(2025\)Addressing small and imbalanced medical image datasets using generative models\.InArtificial Intelligence and Applications,Cited by:[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1)\.
- \[17\]I\. Khazrak, S\. Zainaee, M\. M\. Rezaee, M\. Ghasemi, and R\. C\. Green\(2025\)Feasibility of improving vocal fold pathology image classification with synthetic images generated by ddpm\-based genai: a pilot study\.European Archives of Oto\-Rhino\-Laryngology,pp\. 1–15\.Cited by:[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1)\.
- \[18\]B\. Lester, R\. Al\-Rfou, and N\. Constant\(2021\)The power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1)\.
- \[19\]C\. Li, H\. Farkhoor, R\. Liu, and J\. Yosinski\(2018\)Measuring the intrinsic dimension of objective landscapes\.InInternational Conference on Learning Representations,Cited by:[§II\-C](https://arxiv.org/html/2608.12328#S2.SS3.p1.1)\.
- \[20\]X\. L\. Li and P\. Liang\(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,pp\. 4582–4597\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1)\.
- \[21\]X\. L\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. Hashimoto\(2022\)Diffusion\-LM improves controllable text generation\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 4328–4343\.Cited by:[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1)\.
- \[22\]A\. Lou, C\. Meng, and S\. Ermon\(2023\)Discrete diffusion modeling by estimating the ratios of the data distribution\.InInternational Conference on Machine Learning,pp\. 22481–22505\.Cited by:[§I](https://arxiv.org/html/2608.12328#S1.p3.1),[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1),[§III\-G](https://arxiv.org/html/2608.12328#S3.SS7.p1.4),[§IV\-A](https://arxiv.org/html/2608.12328#S4.SS1.p1.6)\.
- \[23\]Y\. Mao\(2024\)SeLoRA: self\-expanding low\-rank adaptation of latent diffusion model for medical image generation\.External Links:2408\.07196Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[24\]P\. Sahoo, H\. Nguyen, C\. Loh, A\. Kumar, and K\. Narasimhan\(2024\)Simple and effective masked diffusion language models\.arXiv preprint arXiv:2406\.07524\.Cited by:[§I](https://arxiv.org/html/2608.12328#S1.p3.1),[§II\-A](https://arxiv.org/html/2608.12328#S2.SS1.p1.1)\.
- \[25\]V\. Soboleva, A\. Alanov, A\. Kuznetsov, and K\. Sobolev\(2026\)T\-lora: single image diffusion model customization without overfitting\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 9051–9059\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[26\]N\. Tishby, F\. C\. Pereira, and W\. Bialek\(2000\)The information bottleneck method\.arXiv preprint physics/0004057\.Cited by:[§III\-H](https://arxiv.org/html/2608.12328#S3.SS8.p1.6)\.
- \[27\]N\. Tishby and N\. Zaslavsky\(2015\)Deep learning and the information bottleneck principle\.InIEEE Information Theory Workshop,pp\. 1–5\.Cited by:[§II\-C](https://arxiv.org/html/2608.12328#S2.SS3.p1.1)\.
- \[28\]Z\. Wang, Z\. Zhang, C\. Lee, H\. Zhang, R\. Sun, X\. Ren, G\. Su, V\. Perot, J\. Dy, and T\. Pfister\(2020\)Learning to prompt for continual learning\.arXiv preprint arXiv:2112\.08654\.Cited by:[§II\-C](https://arxiv.org/html/2608.12328#S2.SS3.p1.1)\.
- \[29\]E\. B\. Zaken, S\. Ravfogel, and Y\. Goldberg\(2021\)BitFit: simple parameter\-efficient fine\-tuning for transformer\-based masked language\-models\.arXiv preprint arXiv:2106\.10199\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1)\.
- \[30\]J\. Zhang and Y\. Xiong\(2025\)Subject or style: adaptive and training\-free mixture of loras\.arXiv preprint arXiv:2508\.02165\.Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.
- \[31\]Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p1.1)\.
- \[32\]M\. Zhao, P\. Chen, C\. Yu, Y\. Wen, X\. Tan, and T\. Chen\(2025\)Pioneering 4\-bit fp quantization for diffusion models: mixup\-sign quantization and timestep\-aware fine\-tuning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§II\-B](https://arxiv.org/html/2608.12328#S2.SS2.p2.1)\.