FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models

# FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models
Source: [https://arxiv.org/html/2607.05711](https://arxiv.org/html/2607.05711)
Zihan MinXingyang LiZhekai ZhangHaocheng XiLvmin ZhangManeesh AgrawalaJun\-Yan ZhuSong HanYujun LinMuyang Li

###### Abstract

Diffusion models have become a dominant paradigm for high\-quality generative modeling, while post\-training is essential for adapting them to diverse downstream applications\. However, post\-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter\-efficient fine\-tuning methods only partially address\. To overcome these limitations, we proposeFourTune, an efficient post\-training framework for diffusion models based on an end\-to\-endW4A4G4paradigm\. FourTune introduces a triple\-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization\-sensitive outliers, enabling stable training under native 4\-bit computation\. In addition, FourTune employs hardware\-efficient block\-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead\. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full\-precision fine\-tuning\. On FLUX\.1\-dev \(12B\), FourTune reduces memory overhead by2\.25×\\timesand increases end\-to\-end training throughput by2\.27×\\timescompared to BF16 LoRA\.

![[Uncaptioned image]](https://arxiv.org/html/2607.05711v1/x1.png)

Figure 1:Qualitative and quantitative comparison of FourTune against baselines\.Top:Visual comparisons across three diverse post\-training tasks: Customization, Reinforcement Learning, and Distillation\. Despite extremely low\-bit quantization \(W4A4G4\), FourTune produces high\-fidelity images visually indistinguishable from the full\-precision BF16 LoRA baseline\.Bottom:Efficiency benchmarks performed on FLUX\.1\-dev \(1024×10241024\\times 1024\)\.\(a\)FourTune reduces GPU memory consumption by2\.25×\\timescompared to BF16 LoRA, achieving a compact footprint comparable to NF4 QLoRA\.\(b\)In terms of training speed, FourTune significantly outperforms existing methods, achieving2\.27×\\timesand2\.79×\\timesspeedups over BF16 LoRA and NF4 QLoRA, respectively, effectively breaking the memory\-speed trade\-off in large model post\-training\.
## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.05711v1/x2.png)Figure 2:Comparison of forward and backward pipelines across different PEFT methods\.\(a\)LoRA follows the standard high\-precision training pipeline, where weights \(WW\), activations \(AA\), and gradients \(GG\) are all stored and computed in high precision\.\(b\)QLoRA reduces the memory footprint by storing weights in 4\-bit precision, but still requires on\-the\-fly dequantization to 16\-bit for computation \(W4A16G16\)\.\(c\)Our FourTune quantizes weights, activations, and gradients all to 4\-bit \(W4A4G4\), enabling low\-bit computation for both the forward and backward passes while preserving memory efficiency\.Generative models have demonstrated remarkable capabilities in synthesizing high\-fidelity and semantically complex content\(Black Forest Labs,[2024](https://arxiv.org/html/2607.05711#bib.bib7); Qwen Team,[2025](https://arxiv.org/html/2607.05711#bib.bib8)\)\. Driven by the pursuit of higher generation quality, the community has scaled up model sizes to unlock greater potential, witnessing explosive parameter growth\. Model sizes have increased from the 860M\-parameter SD1\.5\(Rombachet al\.,[2022](https://arxiv.org/html/2607.05711#bib.bib1)\), to the 12B DiT\-based FLUX\.1\(Black Forest Labs,[2024](https://arxiv.org/html/2607.05711#bib.bib7)\), and more recently to the 20B\-parameter Qwen\-Image\(Qwen Team,[2025](https://arxiv.org/html/2607.05711#bib.bib8)\)\. However, this rapid scaling has substantially raised the computational requirements for both training and deployment, making it increasingly difficult to run such models on consumer\-grade GPUs\. Although advanced quantization methods\(Liet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib58),[2023](https://arxiv.org/html/2607.05711#bib.bib56); Shanget al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib57)\)have lowered the barrier for inference via low\-bit quantization, the resource bottleneck in post\-training remains largely unaddressed\.

Crucially, post\-training serves as a vital pathway toward model practicality and personalization\. Specifically, through customization, models can learn specific characters, styles, or concepts\(Kumariet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib16); Ruizet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib17)\); via Reinforcement Learning, models can be aligned with human aesthetic preferences\(Feiet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib24); Shenet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib23)\); and through distillation, the inference steps of diffusion models can be significantly reduced to save deployment costs\(Chenet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib36); Franset al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib35)\)\. However, the computation and memory requirements of the diffusion model post\-training are still prohibitively high\. Consequently, achieving computationally efficient and memory\-friendly post\-training has emerged as a critical issue\.

Recent parameter\-efficient fine\-tuning \(PEFT\) methods have gained increasing popularity by substantially reducing the number of trainable parameters and computational cost, while achieving performance comparable to full\-parameter fine\-tuning\(Schulman and Thinking Machines Lab,[2025](https://arxiv.org/html/2607.05711#bib.bib55)\)\. However, despite these advances, existing PEFT approaches have not yet translated into further practical efficiency gains, as they remain fundamentally constrained by a persistent memory–speed trade\-off\. For instance, LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.05711#bib.bib50)\)\([Figure2](https://arxiv.org/html/2607.05711#S1.F2)\(a\)\) reduces training overhead by minimizing the number of trainable parameters\. Nevertheless, the full\-precision backbone weights still occupy a substantial amount of GPU memory\. To alleviate this bottleneck, QLoRA\(Dettmerset al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib61)\)\([Figure2](https://arxiv.org/html/2607.05711#S1.F2)\(b\)\) reduces memory usage by quantizing the base model weights\. However, it does not reduce computational cost, and the frequent online dequantization can even incur additional computational overhead\. In summary, a natural question arises:Is it possible to further accelerate post\-training beyond LoRA?

To address this challenge, we proposeFourTune, a fully 4\-bit post\-training framework designed to break existing efficiency bottlenecks\. Unlike methods like QLoRA that only quantize weights, FourTune adopts a fully quantized backbone, using 4\-bit precision for weights, activations, and gradients \(W4A4G4\) \([Figure2](https://arxiv.org/html/2607.05711#S1.F2)\(c\)\)\. To the best of our knowledge, FourTune is thefirstfully 4\-bit post\-training framework for weight, activation, and gradient for large generative models\. We introduce a Numerical Stabilizer on top of the standard LoRA fine\-tuning framework to ensure stable and convergent training under extremely low\-bit quantization regimes\. We also employ block\-wise quantization to enable efficient on\-the\-fly transposition for direct 4\-bit backpropagation, eliminating transposition and dequantization overhead\. Moreover, we propose customized kernel fusion on the LoRA module and the MLP module to minimize memory access overhead\. This synergistic design reduces both computational load and memory footprint, achieving memory usage comparable to QLoRA while surpassing the training speed of full\-precision LoRA fine\-tuning\.

We conduct extensive experiments across customization, reinforcement learning, and distillation tasks, showing thatFourTuneenables efficient post\-training of diffusion models while matching the generation quality of full\-precision fine\-tuning\. Compared to BF16 LoRA training, FourTune reduces memory footprint by up to 2\.25×\\timesand accelerates training by up to 2\.27×\\times\. Overall, FourTune bridges the gap between high efficiency and high performance for diffusion\-model post\-training\.

## 2Related Work

### 2\.1Post\-Training of Diffusion Models

As diffusion models scale up\(Podellet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib2); Black Forest Labs,[2024](https://arxiv.org/html/2607.05711#bib.bib7); Qwen Team,[2025](https://arxiv.org/html/2607.05711#bib.bib8); Wan Team,[2025](https://arxiv.org/html/2607.05711#bib.bib12)\), post\-training has become essential for downstream adaptation of large diffusion models\. This phase typically includes customization, reinforcement learning, and distillation\.

Customization enables models to learn new concepts from specific identities, styles, or subjects, while preserving the original generative fidelity\(Kumariet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib16); Ruizet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib17); Galet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib18)\)\. Reinforcement Learning incorporates human preferences into training through online RL\(Clarket al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib19); Prabhudesaiet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib20),[2024](https://arxiv.org/html/2607.05711#bib.bib21)\), which performs direct gradient updates on differentiable rewards, and policy\-based approaches\(Blacket al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib25); Fan and Lee,[2023](https://arxiv.org/html/2607.05711#bib.bib26); Fanet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib27)\)\. Distillation compresses multi\-step denoising into efficient few\-step generation, using methods like trajectory regression\(Luhman and Luhman,[2021](https://arxiv.org/html/2607.05711#bib.bib31); Salimanset al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib32); Liuet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib33)\), consistency modeling\(Songet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib37); Guet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib38); Kimet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib39)\), and distribution matching\(Yinet al\.,[2024b](https://arxiv.org/html/2607.05711#bib.bib43),[a](https://arxiv.org/html/2607.05711#bib.bib44); Saueret al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib45)\)\.

### 2\.2Parameter\-Efficient Adaptation

Parameter\-efficient fine\-tuning \(PEFT\) has become a standard alternative to full fine\-tuning, motivated by the observation that adaptation in large models often lies in a low intrinsic dimension\(Aghajanyanet al\.,[2021](https://arxiv.org/html/2607.05711#bib.bib79)\)\. Among PEFT methods, Low\-Rank Adaptation \(LoRA\) and its variants\(Huet al\.,[2022](https://arxiv.org/html/2607.05711#bib.bib50); Sobolevaet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib51); Zhanget al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib52); Liuet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib54)\)are particularly effective in both language models and diffusion models \([Schulman et al\., 2025](https://arxiv.org/html/2607.05711#bib.bib55);YEHet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib76); Chenet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib36)\), as they introduce low\-rank update matrices while keeping pretrained weights frozen, thereby avoiding additional inference latency incurred by adapter\-based methods\(Houlsbyet al\.,[2019](https://arxiv.org/html/2607.05711#bib.bib80)\)\.

To further improve fine\-tuning efficiency, QLoRA\(Dettmerset al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib61)\)performs LoRA training with a 4\-bit quantized backbone, substantially reducing memory consumption at the cost of increased computational overhead due to online dequantization\. Building on this, subsequent works explore improved accuracy–efficiency trade\-offs\. For instance, LoftQ\(Liet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib62)\)tackles this through LoRA\-aware quantization techniques, while QA\-LoRA\(Xuet al\.,[2024](https://arxiv.org/html/2607.05711#bib.bib63)\)reduces auxiliary overhead by integrating additional weight to the quantized base model\.

### 2\.3Model Quantization

Quantization reduces memory and computation by mapping full\-precision weights𝑾\\boldsymbol\{W\}to low\-bit representations, typically via𝑾~=Round​\(𝑾/s\)\\tilde\{\\boldsymbol\{W\}\}=\\mathrm\{Round\}\(\\boldsymbol\{W\}/s\)with scaling factorss\. Modern formats such as NF4 and hardware\-aware floating\-point schemes \(e\.g\., MXFP4 and NVFP4\) adopt group\-wise scaling to better preserve numerical fidelity on contemporary accelerators\(Dettmerset al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib61); Rouhaniet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib75); NVIDIA,[2024](https://arxiv.org/html/2607.05711#bib.bib74)\)\.

Prior work has shown that diffusion models can be effectively compressed to 8\-bit or 4\-bit precision for efficient inference\(Liet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib56); Shanget al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib57); Liet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib58)\)\. More recent studies extend quantization to training, including pretraining large language models directly in 4\-bit precision \(NVFP4\)\(NVIDIA,[2025](https://arxiv.org/html/2607.05711#bib.bib59)\)\. Quantization has also been combined with parameter\-efficient adaptation\. For example, QeRL\(Huanget al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib60)\)integrates NVFP4 with LoRA and demonstrates improved exploration in reinforcement learning settings\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x3.png)Figure 3:Overview of the FourTune framework\.\(a\)Weight Quantization:High\-precision weights are decomposed via SVD into a quantization\-friendly 4\-bit residual and a high\-precision stabilizer\. \(b\)Triple\-Branch Pipeline:The architecture consists of a frozen 4\-bit backbone for arithmetic efficiency, a frozen stabilizer to ensure numerical precision, and a trainable LoRA branch for task adaptation\.

## 3Method

We propose FourTune, a post\-training framework that establishes a native 4\-bit pipeline for large generative models \([Figure3](https://arxiv.org/html/2607.05711#S2.F3)\)\. Our method quantizes weights, activations, and gradients to 4\-bit and executes backbone matrix multiplications directly in 4\-bit\. This design maximizes arithmetic efficiency while significantly reducing memory footprint\. In this section, we describe the overall 4\-bit training pipeline backed by a small full\-precision stabilizer branch, a block\-wise quantization strategy that enables efficient transposition during backpropagation, and customized kernel fusions that improve memory\-bandwidth utilization\.

### 3\.1Triple\-Branch Hybrid\-Precision Pipeline

Directly performing forward and backpropagation on aggressively quantized \(e\.g\., 4\-bit\) weights is often numerically unstable, due to the limited dynamic range and the presence of high\-magnitude outliers in weights, activations, and gradients\. To address this challenge, we propose a*triple\-branch hybrid\-precision pipeline*that explicitly decouples numerical stabilization from task adaptation\.

Our method features an additional*low\-rank, full\-precision stabilizer branch*to explicitly handle quantization\-sensitive outliers and improve numerical stability during training\. Concretely, after isolating the stabilizer component, the remaining frozen backbone weights are quantized using a hardware\-friendly low\-bit floating\-point format \(NVFP4\) to leverage the powerful 4\-bit computing units on GPUs\.

Formally, building upon the spectral decomposition from SVDQuant\(Liet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib58)\), we decompose the pre\-trained weight matrix𝑾∈ℝm×n\\boldsymbol\{W\}\\in\\mathbb\{R\}^\{m\\times n\}into a quantization\-friendly residualRRand a low\-rank outlier component𝑳stab\\boldsymbol\{L\}\_\{\\text\{stab\}\}:

𝑾≈𝑹\+𝑳stab,where​𝑳stab=𝑳𝟏​𝑳𝟐\.\\boldsymbol\{W\}\\approx\\boldsymbol\{R\}\+\\boldsymbol\{L\}\_\{\\text\{stab\}\},\\quad\\text\{where \}\\boldsymbol\{L\}\_\{\\text\{stab\}\}=\\boldsymbol\{L\_\{1\}L\_\{2\}\}\.\(1\)Here,𝑳stab\\boldsymbol\{L\}\_\{\\text\{stab\}\}captures the high\-magnitude outliers that are sensitive to quantization, while the residual exhibits a compressed value range suitable for 4\-bit representation\.

Distinct from previous approaches that utilize this decomposition solely for inference acceleration, we propose to utilize𝑳stab\\boldsymbol\{L\}\_\{\\text\{stab\}\}as a frozen “numerical stabilizer” during training\. This allows us to maintain the majority of parameters \(the residual𝑹\\boldsymbol\{R\}\) in the 4\-bit domain for computationally efficient matrix multiplication \(GEMM4bit\\text\{GEMM\}\_\{\\text\{4bit\}\}\), while isolating numerical risks in the high\-precision branch\. To adapt the model to downstream tasks, we introduce a third, trainable low\-rank branch𝑳l​o​r​a\\boldsymbol\{L\}\_\{lora\}\. The forward propagation for an input𝑿\\boldsymbol\{X\}is thus formulated as a triple\-branch computation:

𝒀\\displaystyle\\boldsymbol\{Y\}=GEMM4bit​\(Q​\(𝑿\),Q​\(𝑹\)\)\\displaystyle=\\text\{GEMM\}\_\{\\text\{4bit\}\}\(Q\(\\boldsymbol\{X\}\),Q\(\\boldsymbol\{R\}\)\)1Frozen 4\-Bit Backbone\(2\)\+𝑿⋅𝑳stab\\displaystyle\\quad\+\\boldsymbol\{X\}\\cdot\\boldsymbol\{L\}\_\{\\text\{stab\}\}2Frozen Stabilizer\+𝑿⋅\(𝑨​𝑩\)\\displaystyle\\quad\+\\boldsymbol\{X\}\\cdot\(\\boldsymbol\{AB\}\)3Trainable AdapterwhereQ​\(⋅\)Q\(\\cdot\)denotes the 4\-bit quantization, applied offline for the residual weights𝑹\\boldsymbol\{R\}and dynamically for activations𝑿\\boldsymbol\{X\}\. In this pipeline, while error signals propagate through all branches to ensure correct gradient flow to preceding layers, weight gradients are computed exclusively for the adapter branch \(𝑨,𝑩\\boldsymbol\{A\},\\boldsymbol\{B\}\)\. Branches1and2function as frozen, differentiable pathways that facilitate backward propagation without requiring parameter updates\. This design enables native 4\-bit computation for the backbone, significantly reducing memory bandwidth and computational latency\.

### 3\.24\-Bit Backward Pass via Block\-Quantization

In the backward pass, our pipeline minimizes computational redundancy by distinguishing between gradient propagation \(required for all branches\) and parameter updates \(exclusive to the adapter\)\. Let𝑮=∂ℒ∂𝒀\\boldsymbol\{G\}=\\frac\{\\partial\\mathcal\{\{L\}\}\}\{\\partial\\boldsymbol\{Y\}\}denote the incoming gradient\.

Propagating gradients to the input requires computing∂ℒ∂𝑿=𝑮​𝑾⊤\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{X\}\}=\\boldsymbol\{GW\}^\{\\top\}\. Leveraging the native 4\-bit capability of our backbone, this operation is formulated as:

∂ℒ∂𝑿\\displaystyle\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{X\}\}=GEMM4bit​\(Q​\(𝑮\),Q​\(𝑹\)⊤\)\\displaystyle=\\text\{GEMM\}\_\{\\text\{4bit\}\}\(Q\(\\boldsymbol\{G\}\),Q\(\\boldsymbol\{R\}\)^\{\\top\}\)Backbone Pathway\(3\)\+𝑮⋅𝑳stab⊤\\displaystyle\\quad\+\\boldsymbol\{G\}\\cdot\\boldsymbol\{L\}\_\{\\text\{stab\}\}^\{\\top\}Stabilizer Pathway\+𝑮⋅\(𝑨​𝑩\)⊤\\displaystyle\\quad\+\\boldsymbol\{G\}\\cdot\(\\boldsymbol\{AB\}\)^\{\\top\}Adapter PathwayThe critical challenge lies in the termQ​\(𝑹\)⊤Q\(\\boldsymbol\{R\}\)^\{\\top\}, which requires performing matrix multiplication with a transposed 4\-bit weight matrix\. Standard per\-output\-channel quantization is ill\-suited for this operation because the scaling factors, originally aligned with the output dimension, become aligned with the reduction dimension \(accumulation axis\) after transposition\. This misalignment prevents the decoupling of scales from the inner product loop, making direct computation inefficient\.

To overcome this, we employ a block\-wise quantization strategy\. By partitioning weights into small, independent blocks, quantization parameters are encapsulated locally\. Crucially, this structure facilitates efficient online transposition: since scales are bound to local data blocks rather than global rows or columns, the computation kernel can fetch a 4\-bit block and its corresponding scale simultaneously, regardless of whether the access pattern is transposed or non\-transposed\. This allows us to computeQ​\(𝑮\)​Q​\(𝑹\)⊤Q\(\\boldsymbol\{G\}\)Q\(\\boldsymbol\{R\}\)^\{\\top\}directly without dequantizing and re\-quantizing the weights, maintaining the high throughput of 4\-bit arithmetic\.

Concurrently, we compute weight gradients exclusively for the trainable adapter branch\. The gradients are derived using standard backpropagation chain rules:

∂ℒ∂𝑩=\(𝑿​𝑨\)⊤​𝑮,∂ℒ∂𝑨=𝑿⊤​\(𝑮​𝑩⊤\)\.\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{B\}\}=\(\\boldsymbol\{X\}\\boldsymbol\{A\}\)^\{\\top\}\\boldsymbol\{G\},\\quad\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{A\}\}=\\boldsymbol\{X\}^\{\\top\}\(\\boldsymbol\{G\}\\boldsymbol\{B\}^\{\\top\}\)\.

### 3\.3Kernel Fusion for Efficiency

#### LoRA Fusion\.

While the stabilizer branch and the trainable low\-rank branch introduce negligible additional computation, executing them as independent pathways can incur significant latency due to increased memory access overhead\. In this regime, the system bottleneck shifts from computation to memory bandwidth\. To mitigate this, we leverage the fact that all three branches share the same input during the quantization and down\-projection phases, as illustrated in[Figure3](https://arxiv.org/html/2607.05711#S2.F3)\(b\)\. Accordingly, we fuse these operations into a single kernel\. Furthermore, since the outputs of these branches must be accumulated, we fuse the 4\-bit backbone computation with the up\-projection operations to eliminate redundant memory traffic\. This strategy substantially reduces the inference latency overhead typically introduced by the LoRA architecture\.

#### MLP Fusion\.

In MLP blocks, intermediate activation tensors often possess high dimensionality \(e\.g\.,3072×122883072\\times 12288\)\. Applying activation functions such as GeLU on these tensors becomes memory\-bound when data is repeatedly written to and read from global memory\. To address this, we fuse the first fully connected \(FC1\) GEMM with the subsequent FC2 quantization kernel\. This approach allows intermediate results to be consumed immediately without materializing large tensors in global memory, thereby optimizing bandwidth utilization and accelerating end\-to\-end execution\.

Table 1:Quantitative results on Customization tasks\.TaskMethodPrecisionFLUX\.1\-devQwen\-ImageSimilarityImageQualityDiversityPromptFollowingSimilarityImageQualityDiversityPromptFollowingIdentityBF16 LoRAW16A16G160\.77133\.490\.5770\.9410\.60124\.540\.7420\.998FP8 LoRAW8A8G80\.78330\.700\.5320\.9120\.59721\.660\.7670\.998NF4 QLoRAW4A16G160\.78032\.270\.5300\.9290\.59725\.630\.6850\.994\\rowcolorourscolor\!40\\cellcolorwhiteOursW4A4G40\.78334\.770\.5700\.9130\.61224\.400\.6950\.995StyleBF16 LoRAW16A16G160\.80632\.020\.5200\.8640\.69627\.340\.6830\.999FP8 LoRAW8A8G80\.82130\.050\.6050\.8930\.69227\.290\.6760\.999NF4 QLoRAW4A16G160\.80632\.190\.5290\.8830\.71033\.720\.6460\.928\\rowcolorourscolor\!40\\cellcolorwhiteOursW4A4G40\.81234\.870\.5470\.8740\.70129\.880\.6840\.999SubjectBF16 LoRAW16A16G160\.73116\.570\.4150\.9460\.61816\.920\.7070\.999FP8 LoRAW8A8G80\.73315\.720\.4600\.9430\.55818\.020\.7140\.999NF4 QLoRAW4A16G160\.72415\.300\.4180\.8960\.69515\.060\.6390\.992\\rowcolorourscolor\!40\\cellcolorwhiteOursW4A4G40\.73316\.220\.4150\.9200\.70326\.920\.6890\.983

## 4Experiments

To evaluate the performance and versatility ofFourTuneacross different computational tiers, we conduct all experiments on platforms powered by the NVIDIA Blackwell architecture, including 8×\\timesNVIDIA RTX Pro 6000 GPUs and 8×\\timesNVIDIA RTX 5090 consumer GPUs\. Leveraging Blackwell’s 4\-bit Tensor Cores, we enable a realistic evaluation of both training and inference efficiency\.

### 4\.1Experimental Settings

We evaluate FourTune across three core post\-training paradigms: Customization, Reinforcement Learning, and Distillation\. Please refer to our appendix for more details\.

#### Customization Setup\.

We evaluate FourTune on a hybrid customization benchmark built from Custom Diffusion\(Kumariet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib16)\)and web\-collected data, covering three tasks: Human Identity, Artistic Style, and General Subject\. Experiments are conducted on FLUX\.1\-dev \(12B\)\(Black Forest Labs,[2024](https://arxiv.org/html/2607.05711#bib.bib7)\)and Qwen\-Image \(20B\)\(Qwen Team,[2025](https://arxiv.org/html/2607.05711#bib.bib8)\)using LoRA with rankr=64r=64and identical hyperparameters\. Evaluation uses task\-specific feature metrics: AntelopeV2 for Human Identity\(InsightFace Contributors,[2022](https://arxiv.org/html/2607.05711#bib.bib64); Denget al\.,[2019](https://arxiv.org/html/2607.05711#bib.bib65)\), CLIP for Artistic Style\(Radfordet al\.,[2021](https://arxiv.org/html/2607.05711#bib.bib66)\), and DINOv3 for General Subject\(Siméoniet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib67)\), with PyIQA for image quality\(Chen and Mo,[2022](https://arxiv.org/html/2607.05711#bib.bib77)\)and BLIP for prompt following\(Liet al\.,[2022](https://arxiv.org/html/2607.05711#bib.bib78)\)\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x4.png)Figure 4:Validation curves for customization tasks, showing performance on par with full\-precision training and with substantial acceleration\.Table 2:Quantitative results for model distillation using theπ\\pi\-Flow algorithm\. We evaluate the 4\-step student model on HPSv2 and COCO\-10k benchmarks\. Performance is measured in terms of teacher alignment \(FID\), prompt alignment \(CLIP\), and preference alignment \(HPSv2\.1\)\. Our W4A4G4 method maintains generation quality comparable to the 16\-bit LoRA baseline\.MethodPrecisionHPSv2 promptsCOCO\-10k promptsTeacher align\.Prompt align\.Pref\. align\.Teacher align\.Prompt align\.Pref\. align\.FID↓\\downarrowCLIP↑\\uparrowHPSv2\.1↑\\uparrowFID↓\\downarrowCLIP↑\\uparrowHPSv2\.1↑\\uparrowBF16 LoRAW16A16G1615\.500\.2880\.3165\.900\.2690\.311SparseLoRAW16A16G16202\.510\.2610\.24575\.450\.2640\.238NF4 QLoRAW4A16G1615\.510\.2870\.3106\.300\.2690\.307\\rowcolorourscolor\!40OursW4A4G415\.500\.2830\.3176\.700\.2660\.310
#### Reinforcement Learning Setup\.

We evaluate FourTune in a reinforcement learning setting using SRPO\(Feiet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib24)\), applying preference gradients only to LoRA adapters for parameter\-efficient optimization\. Experiments follow the original SRPO protocol with FLUX\.1\-dev as the backbone, trained on HPDv2 and guided by the HPSv2\.1 reward model\(Wuet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib68)\)\. Evaluation is performed using Aesthetic Score v2\.5\(discus0434,[2024](https://arxiv.org/html/2607.05711#bib.bib69)\), PickScore\(Kirstainet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib70)\), ImageReward\(Xuet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib71)\), and SGP\-HPS\(Feiet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib24)\)\.

#### Distillation Setup\.

We evaluate FourTune on model distillation usingπ\\pi\-Flow\(Chenet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib36)\), comparing against BF16 LoRA, NF4 QLoRA, and SparseLoRA\(Khakiet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib82)\)\. Following the originalπ\\pi\-Flow protocol, we distill FLUX\.1\-dev\(Black Forest Labs,[2024](https://arxiv.org/html/2607.05711#bib.bib7)\)into a 4\-NFE student in a data\-free setting\. Evaluation is conducted on COCO\-10k and HPSv2 prompts\(Wuet al\.,[2023](https://arxiv.org/html/2607.05711#bib.bib68)\), reporting Teacher\-FID for quality, CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2607.05711#bib.bib66)\)for prompt alignment, and HPSv2\.1 for preference alignment\.

### 4\.2Main Results

Table 3:Quantitative results on Reinforcement Learning tasks using the SRPO algorithm\.MethodAesPickScoreIRSGP\-HPSBase model6\.01350\.23001\.11890\.0015BF16 Finetune6\.34470\.23161\.02090\.0029BF16 LoRA6\.18320\.23071\.07350\.0020NF4 QLoRA6\.21890\.23181\.09310\.0020\\rowcolorourscolor\!40Ours6\.31190\.23081\.01520\.0027#### Accuracy Results\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x5.png)Figure 5:Qualitative comparison on customization\. FourTune matches the generation quality of full\-precision LoRA\.ForCustomization\(Table[1](https://arxiv.org/html/2607.05711#S3.T1)and Figure[5](https://arxiv.org/html/2607.05711#S4.F5)\), FourTune scales post\-training to models with up to 20B parameters\. It matches the generation quality of full\-precision LoRA across human identity preservation, artistic style transfer, and general subject reconstruction, validating its ability to capture fine\-grained visual details\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x6.png)Figure 6:Qualitative comparison on RL\. FourTune even matches the performance of full\-precision, full\-parameter finetuning\.ForReinforcement Learning\(Table[3](https://arxiv.org/html/2607.05711#S4.T3)and Figure[6](https://arxiv.org/html/2607.05711#S4.F6)\), FourTune achieves performance comparable to even the full\-precision, full\-parameter fine\-tuning\. This indicates that the 4\-bit training pipeline retains numerical fidelity to support the fine\-grained updates in RL training\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x7.png)Figure 7:Qualitative comparison on distillation\. FourTune closely matches the visual quality of BF16 LoRA and NF4 QLoRA, while SparseLoRA exhibits noticeable artifacts\.ForDistillation\(Table[2](https://arxiv.org/html/2607.05711#S4.T2)and Figure[7](https://arxiv.org/html/2607.05711#S4.F7)\), FourTune can match the quality of the original 16\-bitπ\\pi\-Flow implementation\. Meanwhile, SparseLoRA underperforms FourTune even at 50% sparsity, likely because timestep distillation is highly accuracy\-sensitive, whereas SparseLoRA imposes only coarse\-grained column sparsity\. These results suggest that FourTune remains stable even for distillation, improving efficiency without compromising generation quality\.

#### Efficiency Results\.

We first evaluate the efficiency of FourTune on the DiT component of FLUX\.1\-dev\. By quantizing weights to 4\-bit, the weight memory consumption is 7\.71 GB, compared to 22\.59 GB for BF16 LoRA, achieving a2\.93×\\timesreduction\. For a single DiT training step, FourTune achieves a total latency of 612\.4 ms whereas BF16 LoRA requires 1541\.0 ms, corresponding to a2\.52×\\timesspeedup\.

We further measure end\-to\-end training efficiency on the customization task\. FourTune reduces the overall memory consumption to 14\.51 GB, compared to 29\.35 GB required by BF16 LoRA, corresponding to a2\.02×\\timesreduction\. On NVIDIA RTX 5090, FourTune achieves a single\-step training time of 0\.86 s, corresponding to a2\.27×\\timesspeedup over BF16 LoRA and a2\.79×\\timesspeedup over NF4 QLoRA\. On NVIDIA RTX Pro 6000, FourTune achieves a single\-step training time of 0\.66 s, yielding a2\.18×\\timesspeedup compared to BF16 LoRA and a2\.85×\\timesspeedup over NF4 QLoRA\.

These results demonstrate that FourTune not only reduces memory consumption substantially, but also consistently delivers practical end\-to\-end training acceleration\.

### 4\.3Ablation Study

To rigorously evaluate the effectiveness of our proposed method, we conducted ablation studies focusing on four key aspects: training precision configurations, the impact of the Stabilizer, the impact of our block\-level quantization, and the efficiency gains from kernel fusion optimizations\.

#### Impact of Precision Configurations\.

We introduce a W4A4G4 training framework that achieves performance comparable to full\-precision training\. To investigate how the precision of different components affects final training outcomes, we conducted experiments across four different precision settings \(W4A4G4, W4A16G4, W4A4G16, and W4A16G16\) on RL tasks\. Specifically, we performed complete training on a subset of 500 prompts from the HPDv2 Benchmark\. It is important to note that among these configurations, only W4A4G4 fully benefits from the acceleration provided by 4\-bit computation\.

As demonstrated in Table[4](https://arxiv.org/html/2607.05711#S4.T4), variations in component precision do not lead to performance degradation\. Notably, our W4A4G4 configuration enables the model to leverage significant hardware acceleration while maintaining performance alignment with full\-precision baselines\.

Table 4:Ablation study on different precision configurations\. Note that W4A4G4 is the only setting that fully utilizes 4\-bit acceleration\.MethodAesPickScoreIRSGP\-HPSW4A16G166\.08160\.23521\.22310\.0002W4A4G166\.03080\.23571\.19820\.0005W4A16G46\.13860\.23541\.21960\.0001\\rowcolorourscolor\!40W4A4G4 \(Ours\)6\.17790\.23531\.17120\.0008![Refer to caption](https://arxiv.org/html/2607.05711v1/x8.png)Figure 8:Effect of the Stabilizer on numerical stability\.Comparison of gradient norms with and without the proposed Stabilizer\. The results show that our method prevents the gradient explosion observed in the baseline setting \(red line\), ensuring stable convergence\.
#### Effectiveness of the Stabilizer\.

During native 4\-bit training, the accumulation of quantization errors occasionally triggers gradient explosion, rendering the training process ineffective\. To investigate this, we monitored the gradient norms over training steps in knowledge distillation experiments, comparing settings with and without the Stabilizer\. As illustrated in Figure[8](https://arxiv.org/html/2607.05711#S4.F8), the naive W4A4G4 pipeline suffers from gradient explosion, while our FourTune’s gradient norms remain stable\. These results demonstrate that our Stabilizer effectively maintains numerical stability, thereby preventing gradient explosion and ensuring consistent convergence\.

#### Impact of Quantization Granularity\.

As detailed in Section[3](https://arxiv.org/html/2607.05711#S3), we adopt block\-wise quantization with a16×1616\\times 16granularity to resolve the non\-contiguous memory access issues inherent to standard group\-wise quantization \(1×161\\times 16\) during backpropagation\. To verify that this coarser granularity does not compromise representation accuracy, we explicitly compare the performance of block\-wise versus group\-wise strategies on FLUX\.1\-dev\. We measure reconstruction fidelity on 5,000 samples from the MJHQ\-30K dataset against the BF16 baseline\. As presented in Table[5](https://arxiv.org/html/2607.05711#S4.T5), our block\-wise strategy yields LPIPS and PSNR metrics comparable to the fine\-grained group\-wise approach\. These results confirm that our design secures significant efficiency gains in the backward pass without incurring any noticeable degradation in generation quality\.

Table 5:Quantitative comparison of quantization strategies on FLUX\.1\-dev\. We evaluate reconstruction quality on 5,000 samples from the MJHQ\-30K dataset compared to the BF16 baseline\. Block\-wise quantization \(16×1616\\times 16\) maintains high fidelity comparable to the finer\-grained group\-wise \(1×161\\times 16\) strategy\.Quantization StrategyGranularityLPIPS↓\\downarrowPSNR↑\\uparrowgroup\-wise1×161\\times 160\.20321\.5\\rowcolorourscolor\!40block\-wise \(Ours\)16×1616\\times 160\.22720\.4
#### Kernel Fusion Breakdown\.

To analyze the sources of efficiency gains from Kernel Fusion, we decompose the contributions of individual fusion strategies, as shown in Figure[9](https://arxiv.org/html/2607.05711#S4.F9)\. Fusing LoRA\-related kernels yields a 1\.82×\\timesspeedup over the unfused 4\-bit baseline, while further fusing the MLP components provides an additional 1\.10×\\timesacceleration\. When all kernels are jointly fused, the combined effect results in an overall 2\.52×\\timesspeedup compared to standard LoRA, empirically demonstrating the effectiveness and complementarity of kernel fusion\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x9.png)Figure 9:Breakdown of efficiency gains from different kernel fusion strategies\. The combined optimization results in a 2\.52×\\timesspeedup relative to 16\-bit LoRA fine\-tuning\.

## 5Conclusion

We have presented FourTune, a fully 4\-bit post\-training framework that enables end\-to\-end W4A4G4 optimization for large diffusion models by using a frozen stabilizer branch, block\-wise quantization for efficient backward computation, and low\-bit fused kernels\. Extensive experiments across customization, reinforcement learning, and distillation demonstrate that FourTune matches full\-precision baselines while substantially reducing memory footprint and improving throughput\. This work contributes to efficient post\-training of diffusion models\.

## Impact Statement

This work improves the efficiency of post\-training large diffusion models through native 4\-bit optimization, reducing computational and energy costs and making model adaptation more accessible\. However, easier adaptation may also lower the barrier to misuse, such as generating deceptive or harmful content\. In our release, we follow the licenses of the base models and datasets, and we encourage downstream users to apply appropriate safety filters and responsible deployment practices\.

## Acknowledgements

We thank MIT\-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Program, National Science Foundation, Packard Foundation, Dell, LG, Hyundai, and Samsung for supporting this research\. We thank NVIDIA for donating the DGX server\.

## References

- A\. Aghajanyan, S\. Gupta, and L\. Zettlemoyer \(2021\)Intrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InInternational Joint Conference on Natural Language Processing,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- Black Forest Labs \(2024\)FLUX\.Note:[https://github\.com/black\-forest\-labs/flux](https://github.com/black-forest-labs/flux)Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px3.p1.2)\.
- K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. Levine \(2024\)Training diffusion models with reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- C\. Chen and J\. Mo \(2022\)IQA\-PyTorch: pytorch toolbox for image quality assessment\.Note:\[Online\]\. Available:[https://github\.com/chaofengc/IQA\-PyTorch](https://github.com/chaofengc/IQA-PyTorch)Cited by:[§A\.1](https://arxiv.org/html/2607.05711#A1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- H\. Chen, K\. Zhang, H\. Tan, L\. J\. Guibas, G\. Wetzstein, and S\. Bi \(2025\)Pi\-flow: policy\-based few\-step generation via imitation distillation\.ArXiv preprint\.Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px3.p1.2)\.
- K\. Clark, P\. Vicol, K\. Swersky, and D\. J\. Fleet \(2024\)Directly fine\-tuning diffusion models on differentiable rewards\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- J\. Deng, J\. Guo, N\. Xue, and S\. Zafeiriou \(2019\)ArcFace: additive angular margin loss for deep face recognition\.InConference on Computer Vision and Pattern Recognition,Cited by:[§A\.1](https://arxiv.org/html/2607.05711#A1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized llms\.InConference and Workshop on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p1.3)\.
- discus0434 \(2024\)Aesthetic predictor v2\.5\.Note:[https://github\.com/discus0434/aesthetic\-predictor\-v2\-5](https://github.com/discus0434/aesthetic-predictor-v2-5)GitHub repository, accessed 2026\-01\-21Cited by:[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Fan and K\. Lee \(2023\)Optimizing DDPM sampling with shortcut fine\-tuning\.InInternational Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- Y\. Fan, O\. Watkins, Y\. Du, H\. Liu, M\. Ryu, C\. Boutilier, P\. Abbeel, M\. Ghavamzadeh, K\. Lee, and K\. Lee \(2023\)Reinforcement learning for fine\-tuning text\-to\-image diffusion models\.InConference and Workshop on Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- S\. Fei, S\. Wang, L\. Ji, A\. Li, S\. Zhang, L\. Liu, J\. Hou, J\. Gong, X\. Zhao, and X\. Qiu \(2025\)SRPO: self\-referential policy optimization for vision\-language\-action models\.ArXiv preprint\.Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px2.p1.1)\.
- K\. Frans, D\. Hafner, S\. Levine, and P\. Abbeel \(2025\)One step diffusion via shortcut models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p2.1)\.
- R\. Gal, Y\. Alaluf, Y\. Atzmon, O\. Patashnik, A\. H\. Bermano, G\. Chechik, and D\. Cohen\-Or \(2023\)An image is worth one word: personalizing text\-to\-image generation using textual inversion\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- J\. Gu, C\. Wang, S\. Zhai, Y\. Zhang, L\. Liu, and J\. M\. Susskind \(2024\)Data\-free distillation of diffusion models with bootstrapping\.InInternational Conference on Machine Learning,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly \(2019\)Parameter\-efficient transfer learning for NLP\.InInternational Conference on Machine Learning,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- W\. Huang, Y\. Ge, S\. Yang, Y\. Xiao, H\. Mao, Y\. Lin, H\. Ye, S\. Liu, K\. C\. Cheung, H\. Yin, Y\. Lu, X\. Qi, S\. Han, and Y\. Chen \(2025\)QeRL: beyond efficiency \- quantization\-enhanced reinforcement learning for llms\.ArXiv preprint\.Cited by:[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p2.1)\.
- InsightFace Contributors \(2022\)InsightFace python library \(model zoo: antelopev2\)\.Note:[https://pypi\.org/project/insightface/0\.7/](https://pypi.org/project/insightface/0.7/)Released Nov 28, 2022\. Model pack antelopev2: SCRFD\-10GF \+ ResNet100@Glint360K\. Accessed 2026\-01\-21Cited by:[§A\.1](https://arxiv.org/html/2607.05711#A1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- S\. Khaki, X\. Li, J\. Guo, L\. Zhu, K\. N\. Plataniotis, A\. Yazdanbakhsh, K\. Keutzer, S\. Han, and Z\. Liu \(2025\)SparseLoRA: accelerating LLM fine\-tuning with contextual sparsity\.InInternational Conference on Machine Learning,Cited by:[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px3.p1.2)\.
- D\. Kim, C\. Lai, W\. Liao, N\. Murata, Y\. Takida, T\. Uesaka, Y\. He, Y\. Mitsufuji, and S\. Ermon \(2024\)Consistency trajectory models: learning probability flow ODE trajectory of diffusion\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. Levy \(2023\)Pick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.InConference and Workshop on Neural Information Processing Systems,Cited by:[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px2.p1.1)\.
- N\. Kumari, B\. Zhang, R\. Zhang, E\. Shechtman, and J\. Zhu \(2023\)Multi\-concept customization of text\-to\-image diffusion\.InConference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Li, D\. Li, C\. Xiong, and S\. C\. H\. Hoi \(2022\)BLIP: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InInternational Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvári, G\. Niu, and S\. Sabato \(Eds\.\),Cited by:[§A\.1](https://arxiv.org/html/2607.05711#A1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Li, Y\. Lin, Z\. Zhang, T\. Cai, X\. Li, J\. Guo, E\. Xie, C\. Meng, J\. Zhu, and S\. Han \(2025\)SVDQuant: absorbing outliers by low\-rank component for 4\-bit diffusion models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p2.1),[§3\.1](https://arxiv.org/html/2607.05711#S3.SS1.p3.3)\.
- X\. Li, Y\. Liu, L\. Lian, H\. Yang, Z\. Dong, D\. Kang, S\. Zhang, and K\. Keutzer \(2023\)Q\-diffusion: quantizing diffusion models\.InInternational Conference on Computer Vision,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p2.1)\.
- Y\. Li, Y\. Yu, C\. Liang, N\. Karampatziakis, P\. He, W\. Chen, and T\. Zhao \(2024\)LoftQ: lora\-fine\-tuning\-aware quantization for large language models\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p2.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024\)DoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- X\. Liu, C\. Gong, and Q\. Liu \(2023\)Flow straight and fast: learning to generate and transfer data with rectified flow\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- E\. Luhman and T\. Luhman \(2021\)Knowledge distillation in iterative generative models for improved sampling speed\.ArXiv preprint\.Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- NVIDIA \(2024\)NVIDIA blackwell architecture technical brief\.External Links:[Link](https://resources.nvidia.com/en-us-blackwell-architecture)Cited by:[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p1.3)\.
- NVIDIA \(2025\)Pretraining large language models with NVFP4\.ArXiv preprint\.Cited by:[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p2.1)\.
- D\. Podell, Z\. English, K\. Lacey, A\. Blattmann, T\. Dockhorn, J\. Müller, J\. Penna, and R\. Rombach \(2024\)SDXL: improving latent diffusion models for high\-resolution image synthesis\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p1.1)\.
- M\. Prabhudesai, A\. Goyal, D\. Pathak, and K\. Fragkiadaki \(2023\)Aligning text\-to\-image diffusion models with reward backpropagation\.ArXiv preprint\.Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- M\. Prabhudesai, R\. Mendonca, Z\. Qin, K\. Fragkiadaki, and D\. Pathak \(2024\)Video diffusion alignment via reward gradients\.ArXiv preprint\.Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- Qwen Team \(2025\)Qwen\-image technical report\.ArXiv preprint\.Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,Cited by:[§A\.1](https://arxiv.org/html/2607.05711#A1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px3.p1.2)\.
- R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer \(2022\)High\-resolution image synthesis with latent diffusion models\.InConference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p1.1)\.
- B\. D\. Rouhani, R\. Zhao, A\. More, M\. Hall, A\. Khodamoradi, S\. Deng, D\. Choudhary, M\. Cornea, E\. Dellinger, K\. Denolf, D\. Stosic, V\. Elango, M\. Golub, A\. Heinecke, P\. James\-Roxby, D\. Jani, G\. Kolhe, M\. Langhammer, A\. Li, L\. Melnick, M\. Mesmakhosroshahi, A\. Rodriguez, M\. Schulte, R\. Shafipour, L\. Shao, M\. Y\. Siu, P\. Dubey, P\. Micikevicius, M\. Naumov, C\. Verilli, R\. Wittig, D\. Burger, and E\. S\. Chung \(2023\)Microscaling data formats for deep learning\.ArXiv preprint\.Cited by:[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p1.3)\.
- N\. Ruiz, Y\. Li, V\. Jampani, Y\. Pritch, M\. Rubinstein, and K\. Aberman \(2023\)DreamBooth: fine tuning text\-to\-image diffusion models for subject\-driven generation\.InConference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- T\. Salimans, T\. Mensink, J\. Heek, and E\. Hoogeboom \(2024\)Multistep distillation of diffusion models via moment matching\.InConference and Workshop on Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- A\. Sauer, D\. Lorenz, A\. Blattmann, and R\. Rombach \(2024\)Adversarial diffusion distillation\.InEuropean Conference on Computer Vision,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- J\. Schulman and Thinking Machines Lab \(2025\)LoRA without regret\.External Links:[Link](https://thinkingmachines.ai/blog/lora/)Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- Y\. Shang, Z\. Yuan, B\. Xie, B\. Wu, and Y\. Yan \(2023\)Post\-training quantization on diffusion models\.InConference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.05711#S2.SS3.p2.1)\.
- X\. Shen, Z\. Li, Z\. Yang, S\. Zhang, Y\. Zhang, D\. Li, C\. Wang, Q\. Lu, and Y\. Tang \(2025\)Directly aligning the full diffusion trajectory with fine\-grained human preference\.ArXiv preprint\.Cited by:[§1](https://arxiv.org/html/2607.05711#S1.p2.1)\.
- O\. Siméoni, H\. V\. Vo, M\. Seitzer, F\. Baldassarre, M\. Oquab, C\. Jose, V\. Khalidov, M\. Szafraniec, S\. E\. Yi, M\. Ramamonjisoa, F\. Massa, D\. Haziza, L\. Wehrstedt, J\. Wang, T\. Darcet, T\. Moutakanni, L\. Sentana, C\. Roberts, A\. Vedaldi, J\. Tolan, J\. Brandt, C\. Couprie, J\. Mairal, H\. Jégou, P\. Labatut, and P\. Bojanowski \(2025\)DINOv3\.ArXiv preprint\.Cited by:[§A\.1](https://arxiv.org/html/2607.05711#A1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px1.p1.1)\.
- V\. Soboleva, A\. Alanov, A\. Kuznetsov, and K\. Sobolev \(2025\)T\-lora: single image diffusion model customization without overfitting\.ArXiv preprint\.Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- Y\. Song, P\. Dhariwal, M\. Chen, and I\. Sutskever \(2023\)Consistency models\.InInternational Conference on Machine Learning,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- Wan Team \(2025\)Wan: open and advanced large\-scale video generative models\.ArXiv preprint\.Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p1.1)\.
- X\. Wu, Y\. Hao, K\. Sun, Y\. Chen, F\. Zhu, R\. Zhao, and H\. Li \(2023\)Human preference score v2: A solid benchmark for evaluating human preferences of text\-to\-image synthesis\.ArXiv preprint\.Cited by:[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px3.p1.2)\.
- J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. Dong \(2023\)ImageReward: learning and evaluating human preferences for text\-to\-image generation\.InConference and Workshop on Neural Information Processing Systems,Cited by:[§4\.1](https://arxiv.org/html/2607.05711#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Xu, L\. Xie, X\. Gu, X\. Chen, H\. Chang, H\. Zhang, Z\. Chen, X\. Zhang, and Q\. Tian \(2024\)QA\-lora: quantization\-aware low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p2.1)\.
- S\. YEH, Y\. Hsieh, Z\. Gao, B\. B\. W\. Yang, G\. Oh, and Y\. Gong \(2024\)Navigating text\-to\-image customization: from lyCORIS fine\-tuning to model evaluation\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.
- T\. Yin, M\. Gharbi, T\. Park, R\. Zhang, E\. Shechtman, F\. Durand, and B\. Freeman \(2024a\)Improved distribution matching distillation for fast image synthesis\.InConference and Workshop on Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- T\. Yin, M\. Gharbi, R\. Zhang, E\. Shechtman, F\. Durand, W\. T\. Freeman, and T\. Park \(2024b\)One\-step diffusion with distribution matching distillation\.InConference on Computer Vision and Pattern Recognition,Cited by:[§2\.1](https://arxiv.org/html/2607.05711#S2.SS1.p2.1)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023\)Adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.05711#S2.SS2.p1.1)\.

## Appendix ADetailed Experimental Settings

### A\.1Customization Evaluation Pipeline

To ensure a comprehensive evaluation, we select five distinct concepts within each category, each exhibiting substantial visual variance\. Following standard few\-shot customization protocols, we curate five independent prompt–image pairs per concept\. This high\-variance construction is designed to probe the model’s generalization capability and training stability across a diverse set of visual concepts\.

During evaluation, we construct five sets of diverse test prompts for each concept\. As discussed in Section[4](https://arxiv.org/html/2607.05711#S4), customization quality is quantified using task\-specific feature\-space metrics\. Specifically, for*Human Identity*, we employ AntelopeV2 to extract facial embeddings and compute cosine similarity with the training samples\(InsightFace Contributors,[2022](https://arxiv.org/html/2607.05711#bib.bib64); Denget al\.,[2019](https://arxiv.org/html/2607.05711#bib.bib65)\); for*Artistic Style*, we use CLIP to measure stylistic alignment between generated and reference images\(Radfordet al\.,[2021](https://arxiv.org/html/2607.05711#bib.bib66)\); and for*General Subject*, we adopt DINOv3 to extract fine\-grained visual features for evaluating structural and textural consistency\(Siméoniet al\.,[2025](https://arxiv.org/html/2607.05711#bib.bib67)\)\. In addition, we assess image quality using PyIQA\(Chen and Mo,[2022](https://arxiv.org/html/2607.05711#bib.bib77)\), measure diversity by averaging pairwise DINOv3 embedding distances among generated images, and evaluate prompt\-following behavior using BLIP\(Liet al\.,[2022](https://arxiv.org/html/2607.05711#bib.bib78)\)\.

### A\.2Customization Dataset

Figure[10](https://arxiv.org/html/2607.05711#A1.F10)shows the complete dataset used on customization tasks\. The dataset consists of three categories: Human Identity, which focuses on preserving personal identity while allowing variations in appearance or attributes\. Artistic Style, which targets adapting artistic or visual styles while keeping the underlying content unchanged\. General Subject, which emphasizes generating a specific subject consistently across different contexts\.

![Refer to caption](https://arxiv.org/html/2607.05711v1/x10.png)\(a\) Human Identity
![Refer to caption](https://arxiv.org/html/2607.05711v1/x11.png)\(b\) Artistic Style
![Refer to caption](https://arxiv.org/html/2607.05711v1/x12.png)\(c\) General Subject

Figure 10:Hybrid customization dataset covering three categories: Human Identity, Artistic Style, and General Subject\.
### A\.3Customization Metrics

We evaluate customization behavior using two complementary metrics:*diversity*and*Prompt Following*\. Diversity is measured using DINO image embeddings to capture visual variation, while Prompt Following is measured using BLIP to assess prompt\-image alignment\.

#### Diversity\.

Diversity measures the visual variation among generated images for the same customization task\. Given a set of generated imagesℐ=\{i1,…,in\}\\mathcal\{I\}=\\\{i\_\{1\},\\ldots,i\_\{n\}\\\}, we extract DINO image embeddingsd​\(i\)d\(i\)and compute the average pairwise cosine similarity\. The diversity score is defined as

D=1−1\(n2\)​∑j<kcos​\(d​\(ij\),d​\(ik\)\),D=1\-\\frac\{1\}\{\\binom\{n\}\{2\}\}\\sum\_\{j<k\}\\mathrm\{cos\}\\big\(d\(i\_\{j\}\),d\(i\_\{k\}\)\\big\),where larger values indicate higher visual diversity\.

#### Prompt Following\.

Prompt Following evaluates how well generated images follow the input prompts\. Given paired prompts and images\{\(pi,ii\)\}i=1n\\\{\(p\_\{i\},i\_\{i\}\)\\\}\_\{i=1\}^\{n\}, we compute cosine similarity between BLIP text and image embeddings\. The instruction fidelity score is defined as

F=1n​∑i=1ncos​\(e​\(pi\),e​\(ii\)\),F=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathrm\{cos\}\\big\(e\(p\_\{i\}\),e\(i\_\{i\}\)\\big\),wheree​\(⋅\)e\(\\cdot\)denotes BLIP embeddings and larger values indicate better prompt\-image alignment\.

## Appendix BGeneralization

Although our main experiments focus on large\-scale DiT backbones, i\.e\., FLUX\.1\-dev and Qwen\-Image, we further evaluate whether FourTune generalizes beyond the main experimental setup\. We consider two complementary settings: transferring to a conventional latent diffusion backbone, SDXL, and running FourTune with an INT4 implementation on an NVIDIA RTX 4090 GPU\.

Table 6:Generalization to SDXL on identity customization\. Our W4A4G4 method maintains generation quality comparable to the 16\-bit LoRA baseline\.MethodPrecisionSimilarityImageQualityDiversityPromptFollowingBF16 LoRAW16A16G160\.45411\.170\.6580\.935\\rowcolorourscolor\!40OursW4A4G40\.45315\.330\.6640\.976As shown in Table[6](https://arxiv.org/html/2607.05711#A2.T6), FourTune transfers well to SDXL under the same W4A4G4 training pipeline\. Its overall customization quality remains competitive with the BF16 LoRA baseline, suggesting that the proposed 4\-bit training scheme is not restricted to DiT architectures\.

Table 7:Generalization across quantization formats and hardware/software stacks on FLUX\.1\-dev identity customization\. The INT4 variant is evaluated on an RTX 4090 GPU and achieves a2\.6×2\.6\\timesspeedup over QLoRA\.MethodPrecisionSimilarityImageQualityDiversityPromptFollowingBF16 LoRAW16A16G160\.77133\.490\.5770\.941\\rowcolorourscolor\!40Ours\(NVFP4\)W4A4G40\.78334\.770\.5700\.913\\rowcolorourscolor\!40Ours\(INT4\)W4A4G40\.77731\.640\.5450\.962FourTune also remains effective under a separate INT4 runtime on the Ada architecture\. As shown in Table[7](https://arxiv.org/html/2607.05711#A2.T7), both NVFP4 and INT4 variants achieve competitive generation quality compared with BF16 LoRA, while the INT4 implementation delivers a2\.6×2\.6\\timesspeedup over QLoRA\. These results suggest that FourTune generalizes across model architectures, low\-bit formats, and hardware/software stacks\.