Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning

# Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
Source: [https://arxiv.org/html/2605.24058](https://arxiv.org/html/2605.24058)
Yoshihiko Fujisawa Fujitsu Limited, Institute of Science Tokyo fujisawa\.y\.5bdc@m\.isct\.ac\.jp &Yuma Ichikawa Fujitsu Limited, RIKEN Center for AIP ichikawa\.yuma@fujitsu\.com&Yudai Fujimoto Fujitsu Limited, Institute of Science Tokyo fujimoto\.y\.7de0@m\.isct\.ac\.jp&Akira Sakai Fujitsu Limited, Tokai University akira\.sakai@fujitsu\.com&Katsuki Fujisawa Institute of Science Tokyo fujisawa\.k\.2110@m\.isct\.ac\.jp

###### Abstract

On\-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task\-specific LoRA adapter\. In the unmerged adapter\-mode setting, however, the adapter is more than a compact storage module; it introduces an additional dense floating\-point branch, maintains a trainable state for local updates, and acts as a unit of communication and hot\-swapping\. We introduce LoRDBA, a LoRA\-compatible adapter that replaces both low\-rank factors with binary sign carriers while representing magnitudes through lightweight, channel\-wise scales, converting the dense adapter branch into two sign\-accumulation matrix multiplications interleaved with channel\-wise scaling\. A finite\-sample analysis shows that reconstruction quality is governed by the residual\-to\-magnitude ratio of the original LoRA factors\. In adapter\-mode experiments, LoRDBA outperforms low\-bit baselines at matched model sizes while matching fp16 LoRA quality in selected regimes\. The unmerged adapter incurs at most8%8\\%prefill latency overhead at matched rankr=16r\{=\}16despite an over10×10\\timesreduction in adapter footprint, with moderate training memory overhead of approximately1\.6×1\.6\\timesthat of fp16 LoRA\.

## 1Introduction

Large language models are increasingly adapted using parameter\-efficient updates rather than full fine\-tuning\. LoRA enables this paradigm by freezing pretrained weights and injecting low\-rank adapter matrices into Transformer projections, reducing the number of trainable parameters\(Hu et al\.,[2022](https://arxiv.org/html/2605.24058#bib.bib15)\)\. QLoRA extends this approach to low\-bit settings by backpropagating through a frozen quantized base model into LoRA adapters\(Dettmers et al\.,[2023](https://arxiv.org/html/2605.24058#bib.bib11)\)\. This base\-plus\-adapter design is attractive for on\-device personalization, privacy\-preserving adaptation, and communication\-efficient update distribution; a compact low\-bit base can remain fixed while task\-specific adapters are trained, exchanged, or selected\. Recent multi\-adapter serving systems, including S\-LoRA and Punica, demonstrate that practical deployment often treats adapters as swappable runtime objects rather than as weights that are always merged into the base model\(Sheng et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib32); Chen et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib9)\)\.

In practical low\-bit deployments, merging a trained adapter into a quantized base requires materializing full\-precision weights or requantizing per task, forfeiting hot\-swap capability; it is therefore preferable to serve the adapter as an unmerged side branch\(Frantar et al\.,[2022](https://arxiv.org/html/2605.24058#bib.bib12); Lin et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib25)\)\. This unmerged branch creates a distinct bottleneck: the base projection runs on fused low\-bit kernels while the adapter uses dense fp16 factors\. An adapter compressor for this regime must preserve accuracy, reduce unmerged computation cost, and avoid inflating the local training state\.

![Refer to caption](https://arxiv.org/html/2605.24058v1/x1.png)Figure 1:LoRDBA \(red\) Pareto\-dominates FP16 LoRA and LoRAQuant at matched adapter sizes onLLaMA\-2\-7B; cf\. Table[1](https://arxiv.org/html/2605.24058#S5.T1)\.![Refer to caption](https://arxiv.org/html/2605.24058v1/x2.png)Figure 2:Adapter\-mode LoRA versus LoRDBA\. The base model is stored in a low\-bit codebook and executed with a fused dequantization–matrix multiplication kernel, while the adapter is served in an unmerged form\. \(a\) Standard LoRA introduces two dense fp16 factors and requires a separate fp16 matrix multiplication branch\. \(b\) LoRDBA replaces both factors with sign matrices coupled by three channel\-wise scale vectors, reducing the adapter forward pass to two sign\-accumulation matrix multiplications\.We introduce LoRDBA, a low\-rank double\-binary adapter designed for unmerged adapter\-mode deployment\. LoRDBA replaces both LoRA factors with binary sign carriers and augments them with a compact scale envelope along the input, rank, and output axes\. Binary signs encode directional information while lightweight channel\-wise scales capture magnitudes; the quantized base model remains frozen and unchanged\. LoRDBA is trained end\-to\-end using a smooth\-sign straight\-through estimator and exported as a near\-binary adapter\. At inference, it replaces the dense adapter branch with binary\-weight accumulation interleaved with scaling\. We analyze when this binary reconstruction is accurate, showing that the key quantity is the residual\-to\-magnitude ratio of the floating\-point LoRA factors\. Unlike BitDelta\(Liu et al\.,[2024a](https://arxiv.org/html/2605.24058#bib.bib26)\), which binarizes the fullΘ​\(N​M\)\\Theta\(NM\)delta discarding low\-rank structure, and LoRAQuant\(Mirzaei et al\.,[2025](https://arxiv.org/html/2605.24058#bib.bib30)\), which retains mixed\-precision SVD components, LoRDBA commits*both*rank\-RRfactors to±1\\pm 1and trains end\-to\-end, preserving the low\-rank adapter format\.

We evaluate LoRDBA under the adapter\-mode protocol—adapter never merged; storage, latency, and training memory measured separately\. Across LLaMA\-based math reasoning and summarization benchmarks, LoRDBA improves the accuracy–size trade\-off over all evaluated low\-bit baselines as previewed in Figure[1](https://arxiv.org/html/2605.24058#S1.F1)\.

## 2Preliminaries

#### Notation\.

Uppercase Roman letters denote matrices; lowercase boldface denotes vectors\.\{±1\}p×q\\\{\\pm 1\\\}^\{p\\times q\}is the set of sign matrices\.∥⋅∥F\\\|\\cdot\\\|\_\{F\}and∥⋅∥op\\\|\\cdot\\\|\_\{\\text\{op\}\}are the Frobenius and operator norms\.diag⁡\(𝒗\)\\operatorname\{diag\}\(\{\\bm\{v\}\}\)constructs a diagonal matrix;sign⁡\(U\)\\operatorname\{sign\}\(U\)is entry\-wise sign withsign⁡\(0\)=\+1\\operatorname\{sign\}\(0\)\{=\}\{\+\}1\. The sub\-Gaussian Orlicz norm is‖X‖ψ2≔inf\{t\>0:𝔼​\[exp⁡\(X2/t2\)\]≤2\}\\\|X\\\|\_\{\\psi\_\{2\}\}\\coloneqq\\inf\\\{t\>0:\{\\mathbb\{E\}\}\[\\exp\(X^\{2\}/t^\{2\}\)\]\\leq 2\\\}\(Vershynin,[2018](https://arxiv.org/html/2605.24058#bib.bib34)\)\.

#### LoRA\.

Given a frozen effective weight matrixW0∈ℝN×MW\_\{0\}\\in\{\\mathbb\{R\}\}^\{N\\times M\}, LoRA reparameterizes the fine\-tuning increment as

Δ​W=A​B⊤∈ℝN×M,A∈ℝN×r,B∈ℝM×r,\\Delta W=AB^\{\\top\}\\in\{\\mathbb\{R\}\}^\{N\\times M\},~~A\\in\{\\mathbb\{R\}\}^\{N\\times r\},~B\\in\{\\mathbb\{R\}\}^\{M\\times r\},\(1\)wherer≪min⁡\(N,M\)r\\ll\\min\(N,M\)is the adapter rank\. Following the column\-vector convention, the adapted forward pass for an input𝒙∈ℝN\{\\bm\{x\}\}\\in\{\\mathbb\{R\}\}^\{N\}is𝒚=W0⊤​𝒙\+B​\(A⊤​𝒙\)∈ℝM\{\\bm\{y\}\}=W\_\{0\}^\{\\top\}\{\\bm\{x\}\}\+B\(A^\{\\top\}\{\\bm\{x\}\}\)\\in\{\\mathbb\{R\}\}^\{M\}\. Thus, each adapted projection incurs two adapter GEMMs with an aggregate arithmetic complexity of𝒪​\(r​\(N\+M\)\)\\mathcal\{O\}\(r\(N\+M\)\)per token and stores16​r​\(N\+M\)16r\(N\+M\)bits when both factors are kept in fp16\. The base weights remain fixed throughout adaptation\.

#### Adapter\-mode deployment\.

We assess LoRDBA in a QLoRA\-style on\-device regime\(Dettmers et al\.,[2023](https://arxiv.org/html/2605.24058#bib.bib11); Sheng et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib32); Chen et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib9)\)\. LetWfpW\_\{\\mathrm\{fp\}\}denote the original full\-precision pretrained weight, and letWQW\_\{Q\}be its stored low\-bit codebook representation\. The effective deployed base weight is

W~0≔dequant​\(WQ\),\\widetilde\{W\}\_\{0\}\\coloneqq\\mathrm\{dequant\}\(W\_\{Q\}\),which generally differs fromWfpW\_\{\\mathrm\{fp\}\}due to quantization error\. In this setting, the base model is never materialized in fp16, and the trained adapter is served in an unmerged state\. The adapted forward pass is

𝒚=W~0⊤​𝒙\+Δ​W⊤​𝒙=dequant​\(WQ\)⊤​𝒙\+Δ​W⊤​𝒙\.\{\\bm\{y\}\}=\\widetilde\{W\}\_\{0\}^\{\\top\}\{\\bm\{x\}\}\+\\Delta W^\{\\top\}\{\\bm\{x\}\}~=~\\mathrm\{dequant\}\(W\_\{Q\}\)^\{\\top\}\{\\bm\{x\}\}\+\\Delta W^\{\\top\}\{\\bm\{x\}\}\.\(2\)This factorizes the operation into a base GEMM—executed by a fused dequantize\-matmul kernel when the base is quantized—and an independent adapter side branch\. The device\-side workload is characterized by the following four conditions:

1. \(A1\)Frozen base:the base model is held fixed and is not updated\. In typical on\-device deployments it is stored in a low\-bit format such as NF4 or GPTQ, but the conditions themselves do not mandate a specific base precision\.
2. \(A2\)Adapter\-only training:optimizer states for the base weights are never materialized\.
3. \(A3\)unmerged serving:the adapter is evaluated as a side branch rather than being merged into a task\-specific base weight\.
4. \(A4\)Hot\-swappability:multiple task\-specific adapters can share the same on\-device base\(Sheng et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib32); Chen et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib9)\)\.

The unmerged regime decouples base and adapter kernels: the base GEMM runs at the codebook’s native precision while the adapter GEMM runs at its own bit width\. Reducing the adapter bit width is thus the primary mechanism for improving the bandwidth\-compute envelope without violating the no\-merging constraint\.

#### Bits per weight \(BPW\)\.

Following standard practice\(Mirzaei et al\.,[2025](https://arxiv.org/html/2605.24058#bib.bib30)\), we quantify adapter compression in*bits per weight*\(BPW\) relative to the reference LoRA parameter countr0​\(N\+M\)r\_\{0\}\(N\+M\)\. An fp16 LoRA adapter of rankrrthus has BPW=16​r/r0=16r/r\_\{0\}, and the full\-size reference at rankr0r\_\{0\}has BPW=16=16\. The LoRDBA\-specific BPW metrics are defined in Section[3\.1](https://arxiv.org/html/2605.24058#S3.SS1)once the adapter parameterization is introduced\.

## 3Method

### 3\.1Low Rank Double Binary Adapter

LoRDBA parameterizes the LoRA delta with two binary factors and a small collection of channel\-wise fp16 scale vectors across the input, rank, and output axes\. This design follows the multi\-envelope principle introduced for full\-weight extreme quantization in MDBF\(Ichikawa et al\.,[2025b](https://arxiv.org/html/2605.24058#bib.bib19)\), but applies it to the much smaller low\-rank LoRA delta\. Since a single set of scales is a fragile one\-dimensional summary of the rank axis, we allow the adapter to contain multiple envelopes that share the same binary carriers, with the number of envelopes controlled by an integerℓ≥1\\ell\\geq 1\.

###### Definition 3\.1\(Low Rank Double Binary Adapter\)\.

Fix a reference fp16 LoRA rankr0≥1r\_\{0\}\\geq 1used only for normalizing storage\. A*LoRDBA adapter*of binary carrier rankR∈ℕR\\in\\mathbb\{N\}and envelope rankℓ≥1\\ell\\geq 1for a host projection of shapeN×MN\\times Mis represented as the tupleθ=\(B1,B2,\(𝛂\(i\),𝛃\(i\),𝛄\(i\)\)i=1ℓ\)\\theta=\\bigl\(B\_\{1\},B\_\{2\},\(\{\\bm\{\\alpha\}\}^\{\(i\)\},\{\\bm\{\\beta\}\}^\{\(i\)\},\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\_\{i=1\}^\{\\ell\}\\bigr\)with

B1∈\{±1\}N×R,B2∈\{±1\}R×M,𝜶\(i\)∈ℝN,𝜷\(i\)∈ℝR,𝜸\(i\)∈ℝM,B\_\{1\}\\in\\\{\\pm 1\\\}^\{N\\times R\},~B\_\{2\}\\in\\\{\\pm 1\\\}^\{R\\times M\},~\{\\bm\{\\alpha\}\}^\{\(i\)\}\\in\{\\mathbb\{R\}\}^\{N\},~\{\\bm\{\\beta\}\}^\{\(i\)\}\\in\{\\mathbb\{R\}\}^\{R\},~\{\\bm\{\\gamma\}\}^\{\(i\)\}\\in\{\\mathbb\{R\}\}^\{M\},\(3\)whose induced weight update is

Δ​W​\(θ\)=∑i=1ℓdiag⁡\(𝜶\(i\)\)​B1​diag⁡\(𝜷\(i\)\)​B2​diag⁡\(𝜸\(i\)\)∈ℝN×M\.\\Delta W\(\\theta\)=\\sum\_\{i=1\}^\{\\ell\}\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}^\{\(i\)\}\)B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}^\{\(i\)\}\)B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\in\{\\mathbb\{R\}\}^\{N\\times M\}\.\(4\)The adapted forward pass is𝐲=W0⊤​𝐱\+Δ​W​\(θ\)⊤​𝐱\{\\bm\{y\}\}=W\_\{0\}^\{\\top\}\{\\bm\{x\}\}\+\\Delta W\(\\theta\)^\{\\top\}\{\\bm\{x\}\}and the exact storage cost is

bits​\(θ\)=R​\(N\+M\)\+16​ℓ​\(N\+R\+M\)\.\\mathrm\{bits\}\(\\theta\)=R\(N\+M\)\+16\\ell\(N\+R\+M\)\.\(5\)We refer toRRas the*binary carrier rank*\(the inner dimension of the binary matmul\), reservingr0r\_\{0\}for the fixed reference LoRA rank used only in BPW normalization\.

The exact storage cost of Eq\. \([5](https://arxiv.org/html/2605.24058#S3.E5)\) induces two BPW metrics:

BPWbc≔Rr0,BPWtot≔R​\(N\+M\)\+16​ℓ​\(N\+R\+M\)r0​\(N\+M\)\.\\mathrm\{BPW\}\_\{\\mathrm\{bc\}\}~\\coloneqq~\\frac\{R\}\{r\_\{0\}\},\\qquad\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}~\\coloneqq~\\frac\{R\(N\+M\)\+16\\ell\(N\+R\+M\)\}\{r\_\{0\}\(N\+M\)\}\.\(6\)BPWbc\\mathrm\{BPW\}\_\{\\mathrm\{bc\}\}counts only the binary carriers;BPWtot\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}includes all fp16 scale vectors\. At the default operating pointR=r0,ℓ=1R\{=\}r\_\{0\},\\ell\{=\}1, we haveBPWtot→1\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}\\to 1asr0,N,M→∞r\_\{0\},N,M\\to\\infty\. All tables reportBPWtot\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}and actual storage in MB\.

The canonicalℓ=1\\ell=1,R=r0R=r\_\{0\}case reduces toΔ​W=diag⁡\(𝜶\)​B1​diag⁡\(𝜷\)​B2​diag⁡\(𝜸\)\\Delta W=\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\(Figure[2](https://arxiv.org/html/2605.24058#S1.F2)\)\. Forℓ≥2\\ell\\geq 2, Eq\. \([4](https://arxiv.org/html/2605.24058#S3.E4)\) sumsℓ\\ellrank\-RRterms; inference cost grows linearly inℓ\\ell, while binary\-carrier storage is unchanged\.

### 3\.2LoRDBA Training

Under \(A1\)–\(A4\), LoRDBA is trained end\-to\-end on the downstream task with the baseW~0=dequant​\(WQ\)\\widetilde\{W\}\_\{0\}=\\mathrm\{dequant\}\(W\_\{Q\}\)frozen\. The procedure has three components: a forward pass with binarized carriers, a backward pass with a smooth\-sign STE, and an SVD\-based initialization\. We call this pipelineQAT Full; all main results use it\.

#### Forward and backward pass\.

We maintain real\-valued latent carriersH1∈ℝN×RH\_\{1\}\\in\{\\mathbb\{R\}\}^\{N\\times R\},H2∈ℝR×MH\_\{2\}\\in\{\\mathbb\{R\}\}^\{R\\times M\}and binarize them in the forward pass:B1=sign⁡\(H1\)B\_\{1\}=\\operatorname\{sign\}\(H\_\{1\}\),B2=sign⁡\(H2\)B\_\{2\}=\\operatorname\{sign\}\(H\_\{2\}\)\. The weight updateΔ​W​\(θ\)\\Delta W\(\\theta\)is computed via Eq\. \([4](https://arxiv.org/html/2605.24058#S3.E4)\)\. On the backward pass, the non\-differentiablesign⁡\(⋅\)\\operatorname\{sign\}\(\\cdot\)is replaced by a smooth\-sign straight\-through estimator \(STE\)\(Leng et al\.,[2018](https://arxiv.org/html/2605.24058#bib.bib23); Liu et al\.,[2020](https://arxiv.org/html/2605.24058#bib.bib28); Boža and Macko,[2025a](https://arxiv.org/html/2605.24058#bib.bib7); Ichikawa and Arai,[2025](https://arxiv.org/html/2605.24058#bib.bib17); Ichikawa,[2024](https://arxiv.org/html/2605.24058#bib.bib16)\)with a temperature schedule \(Appendix[F](https://arxiv.org/html/2605.24058#A6)\)\. Recent high\-dimensional analyses of quantized\-model training further indicate that STE dynamics can be strongly shaped by quantization hyperparameters such as bit width and quantization range\(Ichikawa et al\.,[2025c](https://arxiv.org/html/2605.24058#bib.bib20)\)\. Channel\-wise scales\(𝜶,𝜷,𝜸\)\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)are trained in fp32 and cast to fp16 at export\. The rest of the pipeline \(AdamW optimizer, data pipeline, learning rate schedule\) is identical to a standard LoRA training loop\.

#### Initialization\.

The latent carriers are initialized from the rank\-RRthin SVD of a short,10%10\\%\-budget fp16 LoRA warm\-up\. GivenΔ​Wwarm≈UR​SR​VR⊤\\Delta W\_\{\\mathrm\{warm\}\}\\approx U\_\{R\}S\_\{R\}V\_\{R\}^\{\\top\}:

H1\(0\)=UR,H2\(0\)=VR⊤,B1\(0\)=sign⁡\(H1\(0\)\),B2\(0\)=sign⁡\(H2\(0\)\),𝜷\(0\)=diag⁡\(SR\),H\_\{1\}^\{\(0\)\}=U\_\{R\},~~H\_\{2\}^\{\(0\)\}=V\_\{R\}^\{\\top\},~~B\_\{1\}^\{\(0\)\}=\\operatorname\{sign\}\(H\_\{1\}^\{\(0\)\}\),~~B\_\{2\}^\{\(0\)\}=\\operatorname\{sign\}\(H\_\{2\}^\{\(0\)\}\),~~\{\\bm\{\\beta\}\}^\{\(0\)\}=\\operatorname\{diag\}\(S\_\{R\}\),\(7\)with𝜶\(0\),𝜸\(0\)\{\\bm\{\\alpha\}\}^\{\(0\)\},\{\\bm\{\\gamma\}\}^\{\(0\)\}recovered by one closed\-form per\-axis least\-squares sweep, cf\. Appendix[I\.1](https://arxiv.org/html/2605.24058#A9.SS1)\. Forℓ≥2\\ell\\geq 2, additional envelopes are zero\-initialized as𝜷\(i\)=𝟎\{\\bm\{\\beta\}\}^\{\(i\)\}\{=\}\{\\bm\{0\}\}, so the initial function equals theℓ=1\\ell\{=\}1warm\-start\. This initialization adds negligible overhead relative to the full QAT budget\.

#### Parameter and storage cost\.

The trainable parameter count isR​\(N\+M\)\+ℓ​\(N\+R\+M\)R\(N\{\+\}M\)\+\\ell\(N\{\+\}R\{\+\}M\)—the same leading term as a rank\-RRLoRA adapter plus the envelope overhead\. At export, onlysign⁡\(H1\)\\operatorname\{sign\}\(H\_\{1\}\),sign⁡\(H2\)\\operatorname\{sign\}\(H\_\{2\}\), and the fp16 scales are stored, yielding the storage of Eq\. \([5](https://arxiv.org/html/2605.24058#S3.E5)\)\.

#### Optional training variants\.

LoRDBA admits alternative training modes that share the same exported adapter format and inference kernel\. These are evaluated as ablations in Appendix[G](https://arxiv.org/html/2605.24058#A7):

- •QAT Freeze: freeze the binary carriersB1,B2B\_\{1\},B\_\{2\}from the PTQ initialization and train only the scale vectors\(𝜶,𝜷,𝜸\)\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\), reducing the trainable count toℓ​\(N\+R\+M\)\\ell\(N\{\+\}R\{\+\}M\)per projection\. Wall\-clock time is comparable to QAT Full due to the full forward–backward pass through the frozen base\.
- •PTQ\-LoRDBA: a training\-data\-free scaled\-consensus ADMM that fits a LoRDBA adapter to an existing fp16 LoRAΔ​W⋆\\Delta W^\{\\star\}in Frobenius norm \(Appendix[D](https://arxiv.org/html/2605.24058#A4)\)\. It also serves as the initialization for QAT Full\.

### 3\.3Adapter\-Mode Inference Kernel

For an input batchX∈ℝT×NX\\in\{\\mathbb\{R\}\}^\{T\\times N\}and a canonicalℓ=1\\ell\{=\}1LoRDBA adapter, the evaluation order is

X​Δ​W​\(θ\)=\(\(\(\(X​D𝜶\)​B1\)​D𝜷\)​B2\)​D𝜸,X\\Delta W\(\\theta\)=\\Bigl\(\\Bigl\(\\Bigl\(\\bigl\(XD\_\{\{\\bm\{\\alpha\}\}\}\\bigr\)B\_\{1\}\\Bigr\)D\_\{\{\\bm\{\\beta\}\}\}\\Bigr\)B\_\{2\}\\Bigr\)D\_\{\{\\bm\{\\gamma\}\}\},\(8\)whereD𝒗≔diag⁡\(𝒗\)D\_\{\{\\bm\{v\}\}\}\\coloneqq\\operatorname\{diag\}\(\{\\bm\{v\}\}\)\. The diagonal products are element\-wise fp16 rescalings; the inner matmuls\(⋅\)​B1\(\\cdot\)B\_\{1\}and\(⋅\)​B2\(\\cdot\)B\_\{2\}are binary\-weight GEMMs\. Each column reduces to sign\-accumulation:

\(X​B\)t​j=∑i:Bi​j=\+1Xt​i−∑i:Bi​j=−1Xt​i,\(XB\)\_\{tj\}=\\sum\_\{i:B\_\{ij\}=\+1\}X\_\{ti\}\-\\sum\_\{i:B\_\{ij\}=\-1\}X\_\{ti\},\(9\)which eliminates multiplications in the inner product\. We implement the binary GEMMs viagemlite\(Badri and Shaji,[2023](https://arxiv.org/html/2605.24058#bib.bib4)\); the base codebook GEMM is unchanged\. The bit\-packed\{B1,B2\}\\\{B\_\{1\},B\_\{2\}\\\}is stored once per adapter; forℓ≥2\\ell\\geq 2the same carriers are shared across envelopes\. Pseudocode and microbenchmarks are in Appendices[J](https://arxiv.org/html/2605.24058#A10)and[5\.2](https://arxiv.org/html/2605.24058#S5.SS2)\.

## 4Theoretical Analysis

We explain why replacing both fp16 LoRA factors by binary sign carriers can preserve the LoRA update when factor magnitudes are not too dispersed\. The formal model and proof are in Appendix[C](https://arxiv.org/html/2605.24058#A3); here we state the guarantee in the compact form used to interpret the method\.

Intuition\.If the entries of each LoRA factor are well\-separated from zero \(i\.e\., dominated by their signs\), then replacing the factors by their element\-wise signs introduces a relative Frobenius error proportional to the residual\-to\-magnitude ratioζ/μ\\zeta/\\mu\.

###### Theorem 4\.1\(Informal LoRDBA expressivity\)\.

LetA∈ℝN×rA\\in\\mathbb\{R\}^\{N\\times r\}andB∈ℝM×rB\\in\\mathbb\{R\}^\{M\\times r\}be fp16 LoRA factors\. Suppose that, after fixing the usual positive column\-rescaling gauge of LoRA, their entries can be written as

Ai​k=μA​σi​kA\+ξi​kA,Bj​k=μB​σj​kB\+ξj​kB,A\_\{ik\}=\\mu\_\{A\}\\sigma^\{A\}\_\{ik\}\+\\xi^\{A\}\_\{ik\},\\qquad B\_\{jk\}=\\mu\_\{B\}\\sigma^\{B\}\_\{jk\}\+\\xi^\{B\}\_\{jk\},whereμA,μB\>0\\mu\_\{A\},\\mu\_\{B\}\>0, the sign arrays are in\{±1\}\\\{\\pm 1\\\}, and the residuals are independent, mean\-zero, sub\-Gaussian with scale at mostζ\\zeta, and independent of the signs\. For the relative bound, assume the signs are jointly i\.i\.d\. Rademacher,N​M\>8NM\>8,δ∈\(0,1−8/\(N​M\)\)\\delta\\in\(0,1\-8/\(NM\)\),ζ≤max⁡\(μA,μB\)\\zeta\\leq\\max\(\\mu\_\{A\},\\mu\_\{B\}\), andlog⁡\(6​N​M/δ\)≤c1​r\\log\(6NM/\\delta\)\\leq c\_\{1\}rfor a universal constantc1\>0c\_\{1\}\>0\.

Letθ⋆\\theta^\{\\star\}be the canonical single\-envelope LoRDBA adapter with binary carriersσA\\sigma^\{A\}and\(σB\)⊤\(\\sigma^\{B\}\)^\{\\top\}, unit input/output scales, and rank\-axis scaleμA​μB​𝟏r\\mu\_\{A\}\\mu\_\{B\}\\mathbf\{1\}\_\{r\}\. Then, for a universal constantC′\>0C^\{\\prime\}\>0, with probability at least1−δ−8/\(N​M\)1\-\\delta\-8/\(NM\),

‖A​B⊤−Δ​W​\(θ⋆\)‖F‖Δ​W​\(θ⋆\)‖F≤C′​ζmin⁡\(μA,μB\)​log⁡\(2​N​M/δ\)\.\\frac\{\\\|AB^\{\\top\}\-\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\}\{\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\}\\leq C^\{\\prime\}\\frac\{\\zeta\}\{\\min\(\\mu\_\{A\},\\mu\_\{B\}\)\}\\sqrt\{\\log\(2NM/\\delta\)\}\.\(10\)The full statement, including the absolute Frobenius bound, the observed\-sign version usingsign⁡\(A\)\\operatorname\{sign\}\(A\)andsign⁡\(B\)\\operatorname\{sign\}\(B\), and the monotonic extension to envelope rankℓ≥1\\ell\\geq 1, is given in Appendix[C](https://arxiv.org/html/2605.24058#A3)\.

The bound is controlled byζ/μ\\zeta/\\mu: the residual\-to\-magnitude ratio after separating signs from factor entries\. In the low\-noise regime, binary carriers preserve the Frobenius signal up to logarithmic factors, predicting a performance plateau near the binary\-carrier limit\. Appendix[C\.6](https://arxiv.org/html/2605.24058#A3.SS6)reports the plug\-in diagnostic\.

#### Role of the bound in the QAT pipeline\.

Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)directly characterises PTQ reconstruction quality: given fp16 LoRA factors whose entries are well\-separated from zero, the binary sign carriers preserve most of the Frobenius signal\. In the QAT Full pipeline \(Section[3\.2](https://arxiv.org/html/2605.24058#S3.SS2)\), this PTQ solution serves as the initialization; the subsequent smooth\-sign training refines both the binary carriers and the scales on the downstream loss, which explains why QAT Full consistently outperforms PTQ\-LoRDBA \(Appendix[G](https://arxiv.org/html/2605.24058#A7)\)\. The theorem thus plays two complementary roles: \(i\) it identifies the residual\-to\-magnitude ratioζ/min⁡\(μA,μB\)\\zeta/\\min\(\\mu\_\{A\},\\mu\_\{B\}\)as the diagnostic governing initialization quality, and \(ii\) it predicts a diminishing\-returns plateau near the binary\-carrier limit, beyond which spending more bits on dense factors yields little improvement—a prediction confirmed by the BPW sweep of Section[5\.4](https://arxiv.org/html/2605.24058#S5.SS4)\.

## 5Experiments

Under the adapter\-mode conditions \(A1\)–\(A4\), we evaluate LoRDBA along the three axes that jointly determine deployment viability:

- RQ1Accuracy\(Section[5\.1](https://arxiv.org/html/2605.24058#S5.SS1)\): Does LoRDBA match or exceed fp16 LoRA at matched adapter size?
- RQ2Inference speed\(Section[5\.2](https://arxiv.org/html/2605.24058#S5.SS2)\): Does the binary\-weight kernel reduce unmerged adapter latency?
- RQ3Training cost\(Section[5\.3](https://arxiv.org/html/2605.24058#S5.SS3)\): What is the peak memory and wall\-clock overhead relative to the corresponding fp16 LoRA loop?

#### Evaluation protocol\.

We follow the adapter\-mode plug\-and\-play protocol\(Liu et al\.,[2024a](https://arxiv.org/html/2605.24058#bib.bib26); Mirzaei et al\.,[2025](https://arxiv.org/html/2605.24058#bib.bib30)\)\. A reference fp16 LoRA of rankr0r\_\{0\}is first fine\-tuned on the downstream task atop the frozen base\. The candidate method then emits an adapter at the target BPW under \(A1\)–\(A4\), and the adapter is evaluated*without merging*\. Accuracy is compared at matched adapter size, latency on the unmerged forward pass, and peak memory on the same training device\. LoRDBA trains end\-to\-end rather than compressing a pre\-trained LoRA; the optional ADMM refinement is in Appendix[D](https://arxiv.org/html/2605.24058#A4)\.

#### Setup

All experiments instantiate \(A1\)–\(A4\): the base is frozen in bf16, only the adapter is trainable, and the adapter is served unmerged\. We use bf16 rather than a quantized base to isolate adapter\-compression effects from base\-quantization noise; LoRDBA is orthogonal to the base format and composes with base\-side PTQ pipelines, including layer\-wise error\-propagation and submodule\-level quantization methods\(Arai and Ichikawa,[2025](https://arxiv.org/html/2605.24058#bib.bib1); Ichikawa et al\.,[2025a](https://arxiv.org/html/2605.24058#bib.bib18)\), since it only modifies the adapter branch\.

Models\.LLaMA\-2\-7Bis the primary base;LLaMA\-3\.2\-3Bserves as an ablation base\. The reference LoRA targets all seven Transformer projections\{q,k,v,o,gate,up,down\}proj\\\{q,k,v,o,\\mathrm\{gate\},\\mathrm\{up\},\\mathrm\{down\}\\\}\_\{\\mathrm\{proj\}\}withα=16\\alpha\{=\}16and rankr0=16r\_\{0\}\{=\}16; in ablationsr0∈\{1,2,4,64\}r\_\{0\}\\in\\\{1,2,4,64\\\}\.

Tasks and metrics\.We fine\-tune task\-specific adapters onMetaMathQA,Magicoder\-Evol\-Instruct\-110K, andEdinburghNLP/xsum, and evaluate onGSM8Kwith88\-shot,Minerva Mathwith44\-shot, andXSumvia ROUGE\-L F1, all throughlm\-evaluation\-harness\(Gao et al\.,[2023](https://arxiv.org/html/2605.24058#bib.bib13)\)\.

Baselines\.LoRA fp16at several ranks,LoRAQuant\(Mirzaei et al\.,[2025](https://arxiv.org/html/2605.24058#bib.bib30)\), andBitDelta\(Liu et al\.,[2024a](https://arxiv.org/html/2605.24058#bib.bib26)\)\(scalar, column, column\+\+distill, and row variants\)\. All post\-hoc compressors operate on the same task\-specific fp16 adapter\.

Hardware\.Training and evaluation use NVIDIA H100 \(80 GB\) and B200 GPUs\. The latency benchmark \(Table[2](https://arxiv.org/html/2605.24058#S5.T2)\) runs single\-stream on a single H100 NVL\. Full hyper\-parameter and hardware details are in Appendix[K](https://arxiv.org/html/2605.24058#A11)\.

Compute budget\.Candidates are matched at equal adapter storage, not equal training compute\. QAT Full requires1\.51\.5–1\.8×1\.8\\timestotal GPU\-hours vs\. the fp16 LoRA baseline \(one\-time cost; details in Appendix[K](https://arxiv.org/html/2605.24058#A11)\)\. QAT Freeze offers near\-equivalent accuracy at comparable wall\-clock cost \(Appendix[G](https://arxiv.org/html/2605.24058#A7)\)\.

### 5\.1Adapter Accuracy \(RQ1\)

Table 1:LLaMA\-2\-7B adapter\-mode accuracy at matched size\.*Rank*denotes the binary carrier rankRR, cf\. Definition[3\.1](https://arxiv.org/html/2605.24058#S3.Thmtheorem1); for uncompressed LoRA rows, it coincides with the standard LoRA rank\. BPW isBPWtot\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}of Eq\.[6](https://arxiv.org/html/2605.24058#S3.E6); Avg averages GSM8K, Minerva Math, and XSum\. The full\-size LoRA FP16 reference atr0=16r\_\{0\}\{=\}16, 76\.3 MB, BPW=16\{\}=16achieves Avg28\.8128\.81\. Optional training variants are compared in Appendix[G](https://arxiv.org/html/2605.24058#A7)\.Table[1](https://arxiv.org/html/2605.24058#S5.T1)compares QAT Full LoRDBA against baselines at three matched adapter\-size tiers on LLaMA\-2\-7B\.

#### Result\.

LoRDBA outperforms all baselines at every size tier, where Avg≔\\coloneqq\(GSM8K \+ Minerva \+ XSum\) / 3:

- •∼7\{\\sim\}7MB tier\.QAT Full @1bpw, achieving Avg24\.1424\.14at7\.27\.2MB, Pareto\-dominates both LoRAQuantρ=0\.9\\rho\{=\}0\.9with Avg23\.1723\.17at8\.08\.0MB and LoRA FP16r=2r\{=\}2with Avg23\.2123\.21at9\.59\.5MB\.
- •∼9\.5\{\\sim\}9\.5MB tier\.QAT Full @2bpw with Avg25\.5925\.59exceeds FP16r=2r\{=\}2by\+2\.4\+2\.4pt at the same storage\.
- •∼19\{\\sim\}19MB tier\.QAT Fullr=64r\{=\}64@1bpw with Avg28\.0828\.08attains97\.5%97\.5\\%of the full\-size FP16 reference Avg of28\.8128\.81at76\.376\.3MB, i\.e\.4×4\\timescompression\.111The recovery ratio is the absolute Avg ratio28\.08/28\.8128\.08/28\.81; it does not measure improvement over a no\-adapter baseline\.Notably,r=64r\{=\}64@1bpw andr=16r\{=\}16@4bpw achieve near\-identical Avg—28\.0828\.08vs\.28\.0728\.07—at the same budget, consistent with the plateau predicted by Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)\.

The improvement is concentrated on math reasoning, namely GSM8K and Minerva; among the methods evaluated on XSum in Table[1](https://arxiv.org/html/2605.24058#S5.T1), scores are relatively flat at1717–1818ROUGE\-L, so the Avg differences are primarily driven by the math benchmarks\. The matched comparison on LLaMA\-3\.2\-3B in Appendix[H](https://arxiv.org/html/2605.24058#A8)preserves the same ordering\.

### 5\.2unmerged Adapter Inference Speed \(RQ2\)

We ask whether the binary adapter branch introduces latency overhead relative to fp16 LoRA in the unmerged regime\.

Table 2:Prefill latency on a single NVIDIA H100 NVL withLLaMA\-2\-7B, prompt length=56=56tokens\. Unmerged rows execute the adapter as a side branch over a frozen44\-bit base\. The LoRDBA kernel atℓ=1\\ell\{=\}1is a fused WGMMA CUDA kernel for sign\-accumulation matmuls\.Table[2](https://arxiv.org/html/2605.24058#S5.T2)shows that unmerged LoRDBA adds only2\.62\.6ms, or8%8\\%, to the prefill latency of unmerged fp16 LoRA atr=16r\{=\}16—36\.136\.1ms vs\.33\.533\.5ms—while compressing the adapter by10\.6×10\.6\\times, from76\.276\.2MB to7\.27\.2MB\. Even compared with the closest\-size fp16 baseline, LoRAr=1r\{=\}1at4\.84\.8MB, LoRDBA adds only3\.43\.4ms or10%10\\%at1\.5×1\.5\\timesthe adapter size\. The binary kernel does not achieve a net speed\-up at prefill batch sizeT=56T\{=\}56because the base\-model GEMM dominates total latency; the adapter bandwidth saving becomes more pronounced when multiple adapters are served concurrently or at larger adapter ranks\.

#### Analysis\.

The theoretical adapter\-bandwidth ratio atℓ=1,R=r0\\ell\{=\}1,R\{=\}r\_\{0\}is

16​r0​\(N\+M\)r0​\(N\+M\)\+16​\(N\+r0\+M\)≈8\.0×\(r0=16\),12\.8×\(r0=64\),→16×\(r0→∞\),\\frac\{16r\_\{0\}\(N\{\+\}M\)\}\{r\_\{0\}\(N\{\+\}M\)\+16\(N\{\+\}r\_\{0\}\{\+\}M\)\}~\\approx~8\.0\\times~\(r\_\{0\}\{=\}16\),\\quad 12\.8\\times~\(r\_\{0\}\{=\}64\),\\quad\\to 16\\times~\(r\_\{0\}\\to\\infty\),\(11\)
which upper\-bounds the wall\-clock speedup when the adapter matmul is bandwidth\-bound\. The end\-to\-end measurement confirms that, even outside that regime, the fused LoRDBA kernel preserves unmerged fp16 LoRA latency at10×10\\timessmaller adapter size\.

### 5\.3On\-Device Training Memory \(RQ3\)

We compare LoRDBA QAT Full to standard LoRA fp16 training in peak GPU memory and wall\-clock time\. Table[3](https://arxiv.org/html/2605.24058#S5.T3)reports measurements on a single NVIDIA B200\.

#### Result\.

The overhead is moderate and practical:

- •Peak memory:58\.058\.0GB \(@4BPW\) and55\.655\.6GB \(@1BPW\) vs\.36\.336\.3GB for LoRA fp16, i\.e\. approximately1\.51\.5–1\.6×1\.6\\times\. The overhead is due to the smooth\-sign STE keeping both latent and binarized copies of the binary carriers plus their Adam states; @4BPW is slightly larger due to higher carrier rank\.
- •Wall\-clock time:718718–744744min total, approximately1\.9×1\.9\\timesthe385385min LoRA baseline\. Per\-step time of804804–899899ms is1\.71\.7–1\.9×1\.9\\timesthat of LoRA fp16 at468468ms\.

When the training budget is limited, QAT Freeze offers an effective trade\-off between accuracy and efficiency\. By fixing the binary signs obtained from PTQ and updating only the channel\-wise scales, it matches QAT Full within0\.020\.02average accuracy points at 2 BPW, while reducing optimizer\-state memory through a substantially smaller set of trainable parameters\. Detailed wall\-clock and memory comparisons across all training modes are reported in Appendix[G](https://arxiv.org/html/2605.24058#A7)\.

Table 3:On\-device training peak memory, step time, and total wall\-clock time onLLaMA\-2\-7B\+MetaMathQA\(single NVIDIA B200\)\. LoRA fp16 trains for22epochs; QAT Full adds11\-epoch STE refinement after PTQ initialization\. All rows freeze the bf16 base and update the adapter only; LoRDBA uses the smooth\-sign STE\.*Time*= total wall\-clock: LoRA pre\-training\+\+PTQ init\+\+QAT fine\-tuning\.

### 5\.4Ablations

#### BPW sweep\.

Figure[3](https://arxiv.org/html/2605.24058#A12.F3)sweeps the adapter bit budget on theLLaMA\-3\.2\-3Bablation base\. Since the carrier termR/r0R/r\_\{0\}dominatesBPWtot\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}\(Eq\.[6](https://arxiv.org/html/2605.24058#S3.E6)\) at practical dimensions, the binary\-carrier limitBPWbc=1\\mathrm\{BPW\}\_\{\\mathrm\{bc\}\}\{=\}1controls the curve’s knee: trimming the carrier rank costs≈1\{\\approx\}1pt GSM8K, while spending more bits beyondBPWtot≈1\.5\\mathrm\{BPW\}\_\{\\mathrm\{tot\}\}\{\\approx\}1\.5yields no further gain\.

#### Envelope rankℓ≥2\\ell\\geq 2\.

Table[9](https://arxiv.org/html/2605.24058#A8.T9)in the Appendix reportsℓ=2\\ell=2ablations\. Atℓ=2,R=32\\ell\{=\}2,R\{=\}32, corresponding to19\.119\.1MB, GSM8K reaches47\.84%47\.84\\%with only25\.025\.0GB peak memory—roughly half theℓ=1\\ell\{=\}1variant—at a5\.65\.6pt accuracy cost vs\.ℓ=1\\ell\{=\}1@1bpwR=64R\{=\}64at53\.60%53\.60\\%\. Theℓ=2,R=4\\ell\{=\}2,R\{=\}4configuration at7\.27\.2MB and24\.524\.5GB retains44\.35%44\.35\\%GSM8K, competitive withℓ=1\\ell\{=\}1@1bpw while halving training memory, demonstrating a practical memory–accuracy dial for constrained devices\.

#### Training mode variants\.

Appendix[G](https://arxiv.org/html/2605.24058#A7)\(Table[6](https://arxiv.org/html/2605.24058#A7.T6)\) compares QAT Freeze, QAT Scratch, and PTQ\-LoRDBA against QAT Full\. QAT Freeze matches QAT Full within0\.020\.02Avg at 2 BPW\. QAT Scratch trails by44–99pt on GSM8K due to missing task\-informed initialization\. PTQ\-LoRDBA degrades at≤2\\leq 2BPW on math but retains summarization quality\.

## 6Related Work

#### Low\-rank adaptation\.

LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2605.24058#bib.bib15)\)reparameterizes fine\-tuning updates as a low\-rank product of dense fp16 factors\. Extensions improve rank structure\(Liu et al\.,[2024b](https://arxiv.org/html/2605.24058#bib.bib27); Kopiczko et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib22)\), initialization\(Meng et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib29); Hayou et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib14)\), or training, but all keep full\-precision factors\. QLoRA\(Dettmers et al\.,[2023](https://arxiv.org/html/2605.24058#bib.bib11)\)freezes a44\-bit base and trains fp16 LoRA; S\-LoRA\(Sheng et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib32)\)and Punica\(Chen et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib9)\)operationalize multi\-tenant adapter serving\. LoRDBA compresses the*trained*adapter and composes with any LoRA variant\.

#### Base\-model quantization\.

LoftQ\(Li et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib24)\)and QA\-LoRA\(Xu et al\.,[2024a](https://arxiv.org/html/2605.24058#bib.bib37)\)quantize the frozen base while initializing adapters to compensate for quantization error; adapters remain floating\-point\. Base\-model PTQ methods\(Frantar et al\.,[2022](https://arxiv.org/html/2605.24058#bib.bib12); Lin et al\.,[2024](https://arxiv.org/html/2605.24058#bib.bib25)\)are orthogonal to LoRDBA\. Recent PTQ methods further improve the base\-side quantization objective by explicitly propagating quantization errors across layers\(Arai and Ichikawa,[2025](https://arxiv.org/html/2605.24058#bib.bib1)\)or by extending the optimization unit from individual layers to larger submodules\(Ichikawa et al\.,[2025a](https://arxiv.org/html/2605.24058#bib.bib18)\)\. LoRDBA instead compresses the*adapter*while reusing the base\-side codebook kernel without modification\.

#### Adapter compression\.

BitDelta\(Liu et al\.,[2024a](https://arxiv.org/html/2605.24058#bib.bib26)\)binarizes the full fine\-tuning deltaΔ​W\\Delta Wwith per\-column scales; it discards the low\-rank structure, producing aΘ​\(N​M\)\\Theta\(NM\)\-bit delta much larger than a LoRA adapter at small rank\. LoRAQuant\(Mirzaei et al\.,[2025](https://arxiv.org/html/2605.24058#bib.bib30)\)re\-decomposesA​B⊤AB^\{\\top\}via SVD into mixed\-bit sub\-LoRAs with gradient optimization, retaining floating\-point components\. LoRDBA differs from both by committing both factors to±1\\pm 1and training end\-to\-end\.

#### Binary matrix factorization\.

The±1\\pm 1parameterization connects to binary networks\(Courbariaux et al\.,[2016](https://arxiv.org/html/2605.24058#bib.bib10); Rastegari et al\.,[2016](https://arxiv.org/html/2605.24058#bib.bib31); Liu et al\.,[2020](https://arxiv.org/html/2605.24058#bib.bib28)\)and full\-LLM binarization \(BitNet\(Wang et al\.,[2023](https://arxiv.org/html/2605.24058#bib.bib35)\), OneBit\(Xu et al\.,[2024b](https://arxiv.org/html/2605.24058#bib.bib38)\)\)\. DBF\(Boža and Macko,[2025b](https://arxiv.org/html/2605.24058#bib.bib8)\)factorizes a dense weight as two sign matrices with channel\-wise scales; MDBF\(Ichikawa et al\.,[2025b](https://arxiv.org/html/2605.24058#bib.bib19)\)extends this idea to rank\-ℓ\\ellmulti\-envelope scaling for full\-weight extreme quantization\. LoRDBA applies the factorization to the much smaller LoRA delta, adds an expressivity guarantee \(Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)\), and introduces task\-specific QAT via the smooth\-sign STE\(Leng et al\.,[2018](https://arxiv.org/html/2605.24058#bib.bib23); Boža and Macko,[2025a](https://arxiv.org/html/2605.24058#bib.bib7)\)\. Complementary to algorithmic compression methods, OneComp\(Ichikawa et al\.,[2026](https://arxiv.org/html/2605.24058#bib.bib21)\)packages heterogeneous post\-training compression techniques into a reproducible, hardware\-aware pipeline, whereas LoRDBA focuses on the adapter representation and its unmerged inference primitive\.

## 7Conclusion

LoRDBA shows that both factors of a low\-rank adapter can be binarized to±1\\pm 1, provided that a compact set of channel\-wise scales preserves their row, column, and joint\-spectrum magnitudes\. Under the QLoRA\-style adapter\-mode conditions \(A1\)–\(A4\), QAT Full LoRDBA achieves accuracy that matches or exceeds fp16 LoRA at the same adapter size, while introducing at most8%8\\%prefill\-latency overhead, and compressing the adapter by more than10×10\\times\. The additional training cost is moderate: LoRDBA requires approximately1\.6×1\.6\\timeshigher peak memory and1\.8×1\.8\\timeslonger step time due to the smooth\-sign STE\. Across LLaMA\-2\-7B math reasoning and summarization benchmarks, as well as LLaMA\-3\.2\-3B ablations, QAT Full LoRDBA Pareto\-dominates all evaluated adapter\-compression methods at every tested size tier\.

#### Limitations\.

Our expressivity guarantee in Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)relies on the sub\-Gaussian sign\-noise decomposition in Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\. As a result, the guarantee may weaken when the factor magnitudes are heavy\-tailed or bimodal\. Our experiments are also limited to LLaMA\-2\-7B and LLaMA\-3\.2\-3B on math reasoning and summarization tasks, leaving validation on larger models, alternative architectures, and additional domains to future work\. Finally, LoRDBA targets the unmerged adapter\-mode setting\. Adapters that must be merged into the base model before inference are outside the scope of the current framework\.

#### Broader impact\.

By compressing LoRA adapters to≤4\{\\leq\}4bits per weight, LoRDBA reduces the communication and storage costs of distributing task\-specific models\. This makes on\-device personalization more practical, as adapters can be deployed locally without requiring users to upload private data to the cloud, providing a direct privacy benefit\. At the same time, LoRDBA is both model\- and task\-agnostic, and could therefore also be used to compress adapters fine\-tuned for harmful applications\. We recommend pairing LoRDBA with the same safety guardrails used for the underlying base model\.

## Acknowledgments and Disclosure of Funding

This work was supported by the Council for Science, Technology and Innovation \(CSTI\), Cross\-ministerial Strategic Innovation Promotion Program \(SIP\), “Promotion of Application of Advanced Quantum Technology Infrastructure to Social Issues” \(Funding agency: QST\), JST BOOST \(Grant No\. JPMJBY24D0\), and was carried out using the TSUBAME4\.0 supercomputer at Institute of Science Tokyo\.

## References

- Arai and Ichikawa \[2025\]Yamato Arai and Yuma Ichikawa\.Quantization error propagation: Revisiting layer\-wise post\-training quantization\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025\.
- Attouch et al\. \[2010\]H\. Attouch, J\. Bolte, P\. Redont, and A\. Soubeyran\.Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka–łojasiewicz inequality\.*Mathematics of Operations Research*, 35\(2\):438–457, 2010\.
- Attouch et al\. \[2013\]H\. Attouch, J\. Bolte, and B\. F\. Svaiter\.Convergence of descent methods for semi\-algebraic and tame problems: Proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods\.*Mathematical Programming*, 137\(1–2\):91–129, 2013\.
- Badri and Shaji \[2023\]Hicham Badri and Appu Shaji\.gemlite: Cuda kernels for low\-bit matrix multiplication, 2023\.[https://github\.com/mobiusml/gemlite](https://github.com/mobiusml/gemlite)\.
- Bolte et al\. \[2007\]J\. Bolte, A\. Daniilidis, and A\. Lewis\.The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems\.*SIAM Journal on Optimization*, 17\(4\):1205–1223, 2007\.
- Boyd et al\. \[2011\]Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein\.Distributed optimization and statistical learning via the alternating direction method of multipliers\.*Foundations and Trends in Machine Learning*, 3\(1\):1–122, 2011\.
- Boža and Macko \[2025a\]Vladimír Boža and Vladimír Macko\.Two sparse matrices are better than one: Sparsifying neural networks with double sparse factorization\.In*International Conference on Learning Representations \(ICLR\)*, 2025a\.
- Boža and Macko \[2025b\]Vladimír Boža and Vladimír Macko\.Addition is almost all you need: Compressing large language models with double binary factorization\.In*International Conference on Machine Learning \(ICML\)*, 2025b\.
- Chen et al\. \[2024\]Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy\.Punica: Multi\-tenant LoRA serving\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024\.
- Courbariaux et al\. \[2016\]Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El\-Yaniv, and Yoshua Bengio\.Binarized neural networks: Training deep neural networks with weights and activations constrained to\+1\+1or−1\-1\.*arXiv preprint arXiv:1602\.02830*, 2016\.
- Dettmers et al\. \[2023\]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\.QLoRA: Efficient finetuning of quantized llms\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Frantar et al\. \[2022\]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.GPTQ: Accurate post\-training quantization for generative pre\-trained transformers\.*arXiv preprint arXiv:2210\.17323*, 2022\.
- Gao et al\. \[2023\]Leo Gao, Jonathan Tow, Stella Biderman, et al\.A framework for few\-shot language model evaluation\.[https://github\.com/EleutherAI/lm\-evaluation\-harness](https://github.com/EleutherAI/lm-evaluation-harness), 2023\.
- Hayou et al\. \[2024\]Soufiane Hayou, Nikhil Ghosh, and Bin Yu\.LoRA\+: Efficient low rank adaptation of large models\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Hu et al\. \[2022\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.Lora: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Ichikawa \[2024\]Yuma Ichikawa\.Controlling continuous relaxation for combinatorial optimization\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.
- Ichikawa and Arai \[2025\]Yuma Ichikawa and Yamato Arai\.Optimization by parallel quasi\-quantum annealing with gradient\-based sampling\.In*The Thirteenth International Conference on Learning Representations*, 2025\.
- Ichikawa et al\. \[2025a\]Yuma Ichikawa, Yudai Fujimoto, and Akira Sakai\.LPCD: Unified framework from layer\-wise to submodule quantization\.*arXiv preprint arXiv:2512\.01546*, 2025a\.
- Ichikawa et al\. \[2025b\]Yuma Ichikawa, Yoshihiko Fujisawa, Yudai Fujimoto, Akira Sakai, and Katsuki Fujisawa\.More than bits: Multi\-envelope double binary factorization for extreme quantization\.*arXiv preprint arXiv:2512\.24545*, 2025b\.
- Ichikawa et al\. \[2025c\]Yuma Ichikawa, Shuhei Kashiwamura, and Ayaka Sakata\.High\-dimensional learning dynamics of quantized models with straight\-through estimator\.*arXiv preprint arXiv:2510\.10693*, 2025c\.
- Ichikawa et al\. \[2026\]Yuma Ichikawa, Keiji Kimura, Akihiro Yoshida, Yudai Fujimoto, Hiroki Tokura, Yamato Arai, Yoshiyuki Ishii, Yusei Kawakami, Genki Shikada, Achille Jacquemond, Yoshihiko Fujisawa, Katsuki Fujisawa, Takumi Honda, and Akira Sakai\.OneComp: One\-line revolution for generative AI model compression\.*arXiv preprint arXiv:2603\.28845*, 2026\.
- Kopiczko et al\. \[2024\]Dawid J\. Kopiczko, Tijmen Blankevoort, and Yuki M\. Asano\.VeRA: Vector\-based random matrix adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Leng et al\. \[2018\]Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin\.Extremely low bit neural network: Squeeze the last bit out with ADMM\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 32, 2018\.
- Li et al\. \[2024\]Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao\.LoftQ: LoRA\-fine\-tuning\-aware quantization for large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Lin et al\. \[2024\]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.*Proceedings of Machine Learning and Systems*, 6, 2024\.
- Liu et al\. \[2024a\]James Liu, Guangxuan Xiao, Kai Li, Jason D\. Lee, Song Han, Tri Dao, and Tianle Cai\.BitDelta: Your fine\-tune may only be worth one bit\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024a\.
- Liu et al\. \[2024b\]Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\.DoRA: Weight\-decomposed low\-rank adaptation\.*International Conference on Machine Learning \(ICML\)*, 2024b\.
- Liu et al\. \[2020\]Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang\-Ting Cheng\.ReActNet: Towards precise binary neural network with generalized activation functions\.In*European Conference on Computer Vision \(ECCV\)*, 2020\.
- Meng et al\. \[2024\]Fanxu Meng, Zhaohui Wang, and Muhan Zhang\.PiSSA: Principal singular values and singular vectors adaptation of large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Mirzaei et al\. \[2025\]Amir Reza Mirzaei, Yuqiao Wen, Yanshuai Cao, and Lili Mou\.LoRAQuant: Mixed\-precision quantization of LoRA to ultra\-low bits\.*arXiv preprint arXiv:2510\.26690*, 2025\.
- Rastegari et al\. \[2016\]Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi\.XNOR\-Net: Imagenet classification using binary convolutional neural networks\.In*European Conference on Computer Vision \(ECCV\)*, 2016\.
- Sheng et al\. \[2024\]Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph Gonzalez, and Ion Stoica\.S\-LoRA: Serving thousands of concurrent LoRA adapters\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024\.
- Themelis and Patrinos \[2020\]Andreas Themelis and Panagiotis Patrinos\.Douglas–Rachford splitting and ADMM for nonconvex optimization: Tight convergence results\.*SIAM Journal on Optimization*, 30\(1\):149–181, 2020\.
- Vershynin \[2018\]Roman Vershynin\.*High\-Dimensional Probability: An Introduction with Applications in Data Science*\.Cambridge University Press, 2018\.
- Wang et al\. \[2023\]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei\.BitNet: Scaling 1\-bit transformers for large language models\.*arXiv preprint arXiv:2310\.11453*, 2023\.
- Wang et al\. \[2019\]Yu Wang, Wotao Yin, and Jinshan Zeng\.Global convergence of ADMM in nonconvex nonsmooth optimization\.*Journal of Scientific Computing*, 78\(1\):29–63, 2019\.
- Xu et al\. \[2024a\]Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian\.QA\-LoRA: Quantization\-aware low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.
- Xu et al\. \[2024b\]Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che\.OneBit: Towards extremely low\-bit large language models\.*arXiv preprint arXiv:2402\.11295*, 2024b\.

## Appendix Contents

## Appendix ASymbol Glossary

Table[4](https://arxiv.org/html/2605.24058#A1.T4)collects the principal symbols used throughout the paper\.

Table 4:Glossary of the principal symbols used in the paper\.
## Appendix BSelf\-contained probabilistic background

This appendix collects the elementary concentration facts used by the expressivity proof\. Apart from Markov’s inequality, every estimate below is proved from scratch using only Cauchy–Schwarz and series manipulations, so the proof of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)does not depend on any external reference\. Throughout we work with real\-valued random variables defined on a common probability space, writeC,c,c1,c2,…C,c,c\_\{1\},c\_\{2\},\\ldotsfor finite, strictly positive numerical constants whose value may change from line to line and which are independent of all model parameters, and denote

‖X‖ψ2≔inf\{t\>0:𝔼​exp⁡\(X2/t2\)≤2\},‖X‖ψ1≔inf\{s\>0:𝔼​exp⁡\(\|X\|/s\)≤2\}\.\\\|X\\\|\_\{\\psi\_\{2\}\}\\coloneqq\\inf\\bigl\\\{t\>0:\{\\mathbb\{E\}\}\\exp\(X^\{2\}/t^\{2\}\)\\leq 2\\bigr\\\},\\qquad\\\|X\\\|\_\{\\psi\_\{1\}\}\\coloneqq\\inf\\bigl\\\{s\>0:\{\\mathbb\{E\}\}\\exp\(\|X\|/s\)\\leq 2\\bigr\\\}\.\(12\)
###### Lemma B\.1\(Sub\-Gaussian tail and second moment\)\.

LetXXbe a real random variable with‖X‖ψ2≤K\\\|X\\\|\_\{\\psi\_\{2\}\}\\leq K\. Then

𝔼​\[X2\]≤K2,Pr⁡\[\|X\|≥t\]≤2​exp⁡\(−t2/K2\),t≥0,\{\\mathbb\{E\}\}\[X^\{2\}\]\\;\\leq\\;K^\{2\},\\qquad\\Pr\[\|X\|\\geq t\]\\;\\leq\\;2\\exp\(\-t^\{2\}/K^\{2\}\),~t\\geq 0,\(13\)and for every integerp≥1p\\geq 1,\(𝔼​\|X\|p\)1/p≤C​K​p\(\{\\mathbb\{E\}\}\|X\|^\{p\}\)^\{1/p\}\\leq CK\\sqrt\{p\}with a numerical constantCC\.

###### Proof\.

The inequality1\+u≤eu1\+u\\leq e^\{u\}atu=X2/K2u=X^\{2\}/K^\{2\}and the definition of∥⋅∥ψ2\\\|\\cdot\\\|\_\{\\psi\_\{2\}\}give1\+𝔼​\[X2\]/K2≤𝔼​exp⁡\(X2/K2\)≤21\+\{\\mathbb\{E\}\}\[X^\{2\}\]/K^\{2\}\\leq\{\\mathbb\{E\}\}\\exp\(X^\{2\}/K^\{2\}\)\\leq 2, hence𝔼​\[X2\]≤K2\{\\mathbb\{E\}\}\[X^\{2\}\]\\leq K^\{2\}\. Markov’s inequality applied toexp⁡\(X2/K2\)\\exp\(X^\{2\}/K^\{2\}\)at levelexp⁡\(t2/K2\)\\exp\(t^\{2\}/K^\{2\}\)yields the tail bound\. For the moment bound, integrate the tail:

𝔼​\|X\|p=p​∫0∞tp−1​Pr⁡\[\|X\|≥t\]​𝑑t≤2​p​∫0∞tp−1​e−t2/K2​𝑑t=p​Kp​Γ​\(p/2\)≤\(C​K\)p​pp/2,\{\\mathbb\{E\}\}\|X\|^\{p\}=p\\int\_\{0\}^\{\\infty\}t^\{p\-1\}\\Pr\[\|X\|\\geq t\]dt\\leq 2p\\int\_\{0\}^\{\\infty\}t^\{p\-1\}e^\{\-t^\{2\}/K^\{2\}\}dt=pK^\{p\}\\Gamma\(p/2\)\\leq\(CK\)^\{p\}p^\{p/2\},where the last step uses Stirling\. Taking thepp\-th root proves the moment bound\. ∎

###### Lemma B\.2\(MGF bound and sub\-Gaussian sum\)\.

If𝔼​X=0\{\\mathbb\{E\}\}X=0and‖X‖ψ2≤K\\\|X\\\|\_\{\\psi\_\{2\}\}\\leq K, then𝔼​exp⁡\(λ​X\)≤exp⁡\(C​λ2​K2\)\{\\mathbb\{E\}\}\\exp\(\\lambda X\)\\leq\\exp\(C\\lambda^\{2\}K^\{2\}\)for everyλ∈ℝ\\lambda\\in\{\\mathbb\{R\}\}\. If furthermoreX1,…,XnX\_\{1\},\\ldots,X\_\{n\}are mutually independent, mean\-zero with‖Xi‖ψ2≤K\\\|X\_\{i\}\\\|\_\{\\psi\_\{2\}\}\\leq K, anda1,…,an∈ℝa\_\{1\},\\ldots,a\_\{n\}\\in\{\\mathbb\{R\}\}are deterministic, then for everyu≥0u\\geq 0,

Pr⁡\[\|∑i=1nai​Xi\|≥C​K​‖a‖2​u\]≤2​e−u\.\\Pr\\left\[\\Bigl\|\\sum\_\{i=1\}^\{n\}a\_\{i\}X\_\{i\}\\Bigr\|\\geq CK\\\|a\\\|\_\{2\}\\sqrt\{u\}\\right\]\\;\\leq\\;2e^\{\-u\}\.\(14\)

###### Proof\.

Expanding the exponential as a Taylor series and using𝔼​X=0\{\\mathbb\{E\}\}X=0gives

𝔼​eλ​X=1\+∑p≥2λp​𝔼​Xpp\!\.\{\\mathbb\{E\}\}e^\{\\lambda X\}=1\+\\sum\_\{p\\geq 2\}\\frac\{\\lambda^\{p\}\{\\mathbb\{E\}\}X^\{p\}\}\{p\!\}\.By Lemma[B\.1](https://arxiv.org/html/2605.24058#A2.Thmtheorem1),\|𝔼​Xp\|≤𝔼​\|X\|p≤\(C​K​p\)p\|\{\\mathbb\{E\}\}X^\{p\}\|\\leq\{\\mathbb\{E\}\}\|X\|^\{p\}\\leq\(CK\\sqrt\{p\}\)^\{p\}\. Hence

∑p≥2\|λ\|p​\|𝔼​Xp\|p\!≤∑p≥2\(C​\|λ\|​K​p\)pp\!≤C1​λ2​K2​exp⁡\(C2​λ2​K2\),\\sum\_\{p\\geq 2\}\\frac\{\|\\lambda\|^\{p\}\|\{\\mathbb\{E\}\}X^\{p\}\|\}\{p\!\}\\leq\\sum\_\{p\\geq 2\}\\frac\{\(C\|\\lambda\|K\\sqrt\{p\}\)^\{p\}\}\{p\!\}\\leq C\_\{1\}\\lambda^\{2\}K^\{2\}\\exp\(C\_\{2\}\\lambda^\{2\}K^\{2\}\),where the last inequality is the standard consequence ofp\!≥\(p/e\)pp\!\\geq\(p/e\)^\{p\}and splitting the series atp≃λ2​K2p\\simeq\\lambda^\{2\}K^\{2\}\. Since1\+s​exp⁡\(s′\)≤exp⁡\(C​\(s\+s′\)\)1\+s\\exp\(s^\{\\prime\}\)\\leq\\exp\(C\(s\+s^\{\\prime\}\)\)fors,s′≥0s,s^\{\\prime\}\\geq 0, this yields𝔼​eλ​X≤exp⁡\(C′′′​λ2​K2\)\{\\mathbb\{E\}\}e^\{\\lambda X\}\\leq\\exp\(C^\{\\prime\\prime\\prime\}\\lambda^\{2\}K^\{2\}\)for a universal constantC′′′C^\{\\prime\\prime\\prime\}\. By independence,𝔼​exp⁡\(λ​∑iai​Xi\)=∏i𝔼​eλ​ai​Xi≤exp⁡\(C​λ2​K2​‖a‖22\)\{\\mathbb\{E\}\}\\exp\(\\lambda\\sum\_\{i\}a\_\{i\}X\_\{i\}\)=\\prod\_\{i\}\{\\mathbb\{E\}\}e^\{\\lambda a\_\{i\}X\_\{i\}\}\\leq\\exp\(C\\lambda^\{2\}K^\{2\}\\\|a\\\|\_\{2\}^\{2\}\)\. Chernoff’s inequalityPr⁡\[∑ai​Xi\>t\]≤infλ\>0e−λ​t​𝔼​eλ​∑ai​Xi\\Pr\[\\sum a\_\{i\}X\_\{i\}\>t\]\\leq\\inf\_\{\\lambda\>0\}e^\{\-\\lambda t\}\{\\mathbb\{E\}\}e^\{\\lambda\\sum a\_\{i\}X\_\{i\}\}optimised atλ=t/\(2​C​K2​‖a‖22\)\\lambda=t/\(2CK^\{2\}\\\|a\\\|\_\{2\}^\{2\}\)givesPr⁡\[∑ai​Xi\>t\]≤exp⁡\(−c​t2/\(K2​‖a‖22\)\)\\Pr\[\\sum a\_\{i\}X\_\{i\}\>t\]\\leq\\exp\(\-ct^\{2\}/\(K^\{2\}\\\|a\\\|\_\{2\}^\{2\}\)\)\. A symmetric bound on the negative tail and adjusting constants yields Eq\. \([14](https://arxiv.org/html/2605.24058#A2.E14)\)\. ∎

###### Lemma B\.3\(Sub\-exponential product and Bernstein bound\)\.

IfX,YX,Yare independent real random variables with‖X‖ψ2≤K\\\|X\\\|\_\{\\psi\_\{2\}\}\\leq Kand‖Y‖ψ2≤L\\\|Y\\\|\_\{\\psi\_\{2\}\}\\leq L, then𝔼​\[X​Y\]=𝔼​X​𝔼​Y\{\\mathbb\{E\}\}\[XY\]=\{\\mathbb\{E\}\}X\{\\mathbb\{E\}\}Yand the productX​YXYis sub\-exponential with‖X​Y‖ψ1≤K​L\\\|XY\\\|\_\{\\psi\_\{1\}\}\\leq KL\. Moreover, ifW1,…,WnW\_\{1\},\\ldots,W\_\{n\}are mutually independent, mean\-zero, and‖Wi‖ψ1≤L0\\\|W\_\{i\}\\\|\_\{\\psi\_\{1\}\}\\leq L\_\{0\}, then for everyu≥0u\\geq 0,

Pr⁡\[\|∑i=1nWi\|≥t\]≤2​exp⁡\(−c​min⁡\(t2n​L02,tL0\)\),t≥0\.\\Pr\\left\[\\Bigl\|\\sum\_\{i=1\}^\{n\}W\_\{i\}\\Bigr\|\\geq t\\right\]\\;\\leq\\;2\\exp\\Bigl\(\-c\\min\\Bigl\(\\frac\{t^\{2\}\}\{nL\_\{0\}^\{2\}\},\\frac\{t\}\{L\_\{0\}\}\\Bigr\)\\Bigr\),\\quad t\\geq 0\.\(15\)

###### Proof\.

For the product part, the Young inequality\|x​y\|/\(K​L\)≤x2/\(2​K2\)\+y2/\(2​L2\)\|xy\|/\(KL\)\\leq x^\{2\}/\(2K^\{2\}\)\+y^\{2\}/\(2L^\{2\}\)combined with the AM–GM and Cauchy–Schwarz inequalities yields

𝔼​exp⁡\(\|X​Y\|/\(K​L\)\)≤𝔼​exp⁡\(X2/\(2​K2\)\+Y2/\(2​L2\)\)=𝔼​exp⁡\(X2/\(2​K2\)\)​𝔼​exp⁡\(Y2/\(2​L2\)\)≤2,\{\\mathbb\{E\}\}\\exp\(\|XY\|/\(KL\)\)\\leq\{\\mathbb\{E\}\}\\exp\(X^\{2\}/\(2K^\{2\}\)\+Y^\{2\}/\(2L^\{2\}\)\)=\{\\mathbb\{E\}\}\\exp\(X^\{2\}/\(2K^\{2\}\)\)\{\\mathbb\{E\}\}\\exp\(Y^\{2\}/\(2L^\{2\}\)\)\\leq 2,where we used independence in the equality and theψ2\\psi\_\{2\}\-defining bound twice in the last step \(after noting thatexp⁡\(X2/\(2​K2\)\)≤exp⁡\(X2/K2\)\\exp\(X^\{2\}/\(2K^\{2\}\)\)\\leq\\sqrt\{\\exp\(X^\{2\}/K^\{2\}\)\}, so its expectation is at most2\\sqrt\{2\}\)\. Hence‖X​Y‖ψ1≤K​L\\\|XY\\\|\_\{\\psi\_\{1\}\}\\leq KL\. For the mean,𝔼​\|X​Y\|≤𝔼​X2​𝔼​Y2<∞\{\\mathbb\{E\}\}\|XY\|\\leq\\sqrt\{\{\\mathbb\{E\}\}X^\{2\}\{\\mathbb\{E\}\}Y^\{2\}\}<\\inftyby Cauchy–Schwarz and Lemma[B\.1](https://arxiv.org/html/2605.24058#A2.Thmtheorem1), so by Fubini𝔼​\[X​Y\]=𝔼​X​𝔼​Y\{\\mathbb\{E\}\}\[XY\]=\{\\mathbb\{E\}\}X\{\\mathbb\{E\}\}Y\.

For the Bernstein bound, an argument identical to that of Lemma[B\.2](https://arxiv.org/html/2605.24058#A2.Thmtheorem2)\(now using\(𝔼​\|W\|p\)1/p≤C​L0​p\(\{\\mathbb\{E\}\}\|W\|^\{p\}\)^\{1/p\}\\leq CL\_\{0\}pinstead ofC​K​pCK\\sqrt\{p\}, which is the sub\-exponential moment bound proved by integrating the tailPr⁡\[\|W\|≥t\]≤2​e−t/L0\\Pr\[\|W\|\\geq t\]\\leq 2e^\{\-t/L\_\{0\}\}\) yields, for\|λ\|≤c/L0\|\\lambda\|\\leq c/L\_\{0\},

𝔼​exp⁡\(λ​∑i=1nWi\)≤exp⁡\(C​n​λ2​L02\)\.\{\\mathbb\{E\}\}\\exp\\Bigl\(\\lambda\\sum\_\{i=1\}^\{n\}W\_\{i\}\\Bigr\)\\leq\\exp\(Cn\\lambda^\{2\}L\_\{0\}^\{2\}\)\.Optimising the resulting Chernoff bound atλ=min⁡\(t/\(2​C​n​L02\),c/\(2​L0\)\)\\lambda=\\min\\bigl\(t/\(2CnL\_\{0\}^\{2\}\),c/\(2L\_\{0\}\)\\bigr\)gives the two\-regime bound Eq\. \([15](https://arxiv.org/html/2605.24058#A2.E15)\)\. ∎

###### Lemma B\.4\(Sign\-flip bound from aψ2\\psi\_\{2\}residual\)\.

If‖X‖ψ2≤K\\\|X\\\|\_\{\\psi\_\{2\}\}\\leq Kandμ\>0\\mu\>0, thenPr⁡\[\|X\|≥μ\]≤2​exp⁡\(−μ2/K2\)\\Pr\[\|X\|\\geq\\mu\]\\leq 2\\exp\(\-\\mu^\{2\}/K^\{2\}\)\.

###### Proof\.

Markov’s inequality applied toexp⁡\(X2/K2\)\\exp\(X^\{2\}/K^\{2\}\)at levelexp⁡\(μ2/K2\)\\exp\(\\mu^\{2\}/K^\{2\}\)givesPr⁡\[\|X\|≥μ\]=Pr⁡\[exp⁡\(X2/K2\)≥exp⁡\(μ2/K2\)\]≤2​exp⁡\(−μ2/K2\)\\Pr\[\|X\|\\geq\\mu\]=\\Pr\[\\exp\(X^\{2\}/K^\{2\}\)\\geq\\exp\(\\mu^\{2\}/K^\{2\}\)\]\\leq 2\\exp\(\-\\mu^\{2\}/K^\{2\}\)\. ∎

## Appendix CExpressivity of LoRDBA: Proof of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)

This appendix gives a full, self\-contained proof of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)from Section[4](https://arxiv.org/html/2605.24058#S4)\. The argument is entirely a matter of entry\-wise sub\-Gaussian and sub\-exponential concentration, and no sketching, Rademacher compression, or Johnson–Lindenstrauss reduction is used\. All underlying probabilistic facts are proved from scratch in Appendix[B](https://arxiv.org/html/2605.24058#A2)\(sub\-Gaussian sum tail Lemma[B\.2](https://arxiv.org/html/2605.24058#A2.Thmtheorem2), sub\-exponential product and Bernstein inequality Lemma[B\.3](https://arxiv.org/html/2605.24058#A2.Thmtheorem3), and the sign\-flip bound Lemma[B\.4](https://arxiv.org/html/2605.24058#A2.Thmtheorem4)\)\. In particular, the sub\-Gaussian Orlicz norm and the second\-moment bound𝔼​\[X2\]≤‖X‖ψ22\{\\mathbb\{E\}\}\[X^\{2\}\]\\leq\\\|X\\\|\_\{\\psi\_\{2\}\}^\{2\}used freely below are recorded in Lemma[B\.1](https://arxiv.org/html/2605.24058#A2.Thmtheorem1)\.

###### Assumption C\.1\(Sign\-plus\-sub\-Gaussian\-noise factor model\)\.

The LoRA factorsA∈ℝN×rA\\in\{\\mathbb\{R\}\}^\{N\\times r\}andB∈ℝM×rB\\in\{\\mathbb\{R\}\}^\{M\\times r\}are random matrices defined on a common probability space\(Ω,𝒜,Pr\)\(\\Omega,\{\\mathcal\{A\}\},\\Pr\)and admit the entry\-wise decomposition

Ai​k=μA​σi​kA\+ξi​kA,Bj​k=μB​σj​kB\+ξj​kB,i∈\[N\],j∈\[M\],k∈\[r\],A\_\{ik\}=\\mu\_\{A\}\\sigma^\{A\}\_\{ik\}\+\\xi^\{A\}\_\{ik\},~~B\_\{jk\}=\\mu\_\{B\}\\sigma^\{B\}\_\{jk\}\+\\xi^\{B\}\_\{jk\},~~i\\in\[N\],~j\\in\[M\],~k\\in\[r\],\(16\)whereμA,μB\>0\\mu\_\{A\},\\mu\_\{B\}\>0andζ\>0\\zeta\>0are deterministic scalars and the sign arraysσA∈\{±1\}N×r\\sigma^\{A\}\\in\\\{\\pm 1\\\}^\{N\\times r\},σB∈\{±1\}M×r\\sigma^\{B\}\\in\\\{\\pm 1\\\}^\{M\\times r\}and residual arraysξA∈ℝN×r\\xi^\{A\}\\in\{\\mathbb\{R\}\}^\{N\\times r\},ξB∈ℝM×r\\xi^\{B\}\\in\{\\mathbb\{R\}\}^\{M\\times r\}satisfy:

1. 1\.The families\{ξi​kA\}\(i,k\)∈\[N\]×\[r\]\\\{\\xi^\{A\}\_\{ik\}\\\}\_\{\(i,k\)\\in\[N\]\\times\[r\]\}and\{ξj​kB\}\(j,k\)∈\[M\]×\[r\]\\\{\\xi^\{B\}\_\{jk\}\\\}\_\{\(j,k\)\\in\[M\]\\times\[r\]\}consist of mutually independent random variables, the two families are mutually independent, and each entry has zero mean\.
2. 2\.The residuals are uniformly sub\-Gaussian: max⁡\(max\(i,k\)∈\[N\]×\[r\]⁡‖ξi​kA‖ψ2,max\(j,k\)∈\[M\]×\[r\]⁡‖ξj​kB‖ψ2\)≤ζ\.\\max\\Bigl\(\\max\_\{\(i,k\)\\in\[N\]\\times\[r\]\}\\\|\\xi^\{A\}\_\{ik\}\\\|\_\{\\psi\_\{2\}\},\\max\_\{\(j,k\)\\in\[M\]\\times\[r\]\}\\\|\\xi^\{B\}\_\{jk\}\\\|\_\{\\psi\_\{2\}\}\\Bigr\)\\leq\\zeta\.
3. 3\.The signσ\\sigma\-algebraℱ≔σ​\(σA,σB\)\{\\mathcal\{F\}\}\\coloneqq\\sigma\(\\sigma^\{A\},\\sigma^\{B\}\)is independent ofσ​\(ξA,ξB\)\\sigma\(\\xi^\{A\},\\xi^\{B\}\)\.

We allow the latent sign arrays to be either fixed or random\. WhenσA,σB\\sigma^\{A\},\\sigma^\{B\}are fixed, all probability statements are taken only over the residual arrays\(ξA,ξB\)\(\\xi^\{A\},\\xi^\{B\}\); in this case Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(iii\) is vacuous\. For the relative bound in Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1), the entries ofσA\\sigma^\{A\}andσB\\sigma^\{B\}are additionally assumed to be jointly i\.i\.d\. Rademacher\.

The canonical reconstruction used in Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)is

θ⋆≔\(σA,\(σB\)⊤,𝟏N,μA​μB​𝟏r,𝟏M\),\\theta^\{\\star\}\\coloneqq\\bigl\(\\sigma^\{A\},\(\\sigma^\{B\}\)^\{\\top\},\{\\bm\{1\}\}\_\{N\},\\mu\_\{A\}\\mu\_\{B\}\{\\bm\{1\}\}\_\{r\},\{\\bm\{1\}\}\_\{M\}\\bigr\),\(17\)which is a single\-envelope LoRDBA adapter with the latent signs as the binary carriers\. For each envelope rankℓ≥1\\ell\\geq 1, letΘℓ\\Theta\_\{\\ell\}denote the corresponding LoRDBA class of Definition[3\.1](https://arxiv.org/html/2605.24058#S3.Thmtheorem1)\. The zero\-padding embedding

ιℓ:Θℓ−1↪Θℓ,\(B1,B2,\{\(𝜶\(i\),𝜷\(i\),𝜸\(i\)\)\}i=1ℓ−1\)↦\(B1,B2,\{\(𝜶\(i\),𝜷\(i\),𝜸\(i\)\)\}i=1ℓ−1,\(𝟎,𝟎,𝟎\)\)\\iota\_\{\\ell\}:\\Theta\_\{\\ell\-1\}\\hookrightarrow\\Theta\_\{\\ell\},~~\\bigl\(B\_\{1\},B\_\{2\},\\\{\(\{\\bm\{\\alpha\}\}^\{\(i\)\},\{\\bm\{\\beta\}\}^\{\(i\)\},\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\ell\-1\}\\bigr\)\\mapsto\\bigl\(B\_\{1\},B\_\{2\},\\\{\(\{\\bm\{\\alpha\}\}^\{\(i\)\},\{\\bm\{\\beta\}\}^\{\(i\)\},\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\ell\-1\},\(\{\\bm\{0\}\},\{\\bm\{0\}\},\{\\bm\{0\}\}\)\\bigr\)\(18\)preservesΔ​W\\Delta W, so the best achievable reconstruction error is non\-increasing in the envelope rank\.

LetE≔A​B⊤−Δ​W​\(θ⋆\)∈ℝN×ME\\coloneqq AB^\{\\top\}\-\\Delta W\(\\theta^\{\\star\}\)\\in\{\\mathbb\{R\}\}^\{N\\times M\}denote the reconstruction residual\. We first expandEEentry\-wise \(Lemma[C\.2](https://arxiv.org/html/2605.24058#A3.Thmtheorem2)\), then bound each entry by sub\-exponential concentration \(Lemma[C\.3](https://arxiv.org/html/2605.24058#A3.Thmtheorem3)\), aggregate by a union bound to obtain \([27](https://arxiv.org/html/2605.24058#A3.E27)\), and finally combine the Frobenius upper bound with a Chebyshev lower bound on‖Δ​W​\(θ⋆\)‖F2\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}\(Lemma[C\.4](https://arxiv.org/html/2605.24058#A3.Thmtheorem4)\) to obtain \([10](https://arxiv.org/html/2605.24058#S4.E10)\)\.

For the absolute bound \([27](https://arxiv.org/html/2605.24058#A3.E27)\) the signsσA,σB\\sigma^\{A\},\\sigma^\{B\}are deterministic, so all probability statements below are taken with respect to the residuals\(ξA,ξB\)\(\\xi^\{A\},\\xi^\{B\}\)\. For the relative bound \([10](https://arxiv.org/html/2605.24058#S4.E10)\) the signs are additionally i\.i\.d\. Rademacher and jointly independent of the residuals; every statement proved for deterministic signs therefore also holds*conditionally*on\(σA,σB\)\(\\sigma^\{A\},\\sigma^\{B\}\), and by Fubini the residual event of probability at least1−δ1\-\\deltaholds on the product space\.

### C\.1Entry\-wise expansion

The heart of the argument is the following entry\-wise expansion of the residual into three pieces with distinct concentration behaviour\. Its pairwiseL2L^\{2\}\-orthogonality keeps the final variance computation additive\.

###### Lemma C\.2\(Residual expansion\)\.

Under Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1), for everyi∈\[N\]i\\in\[N\]andj∈\[M\]j\\in\[M\],

Ei​j=μA​∑k=1rσi​kA​ξj​kB⏟=⁣:Ti​j\(1\)\+μB​∑k=1rξi​kA​σj​kB⏟=⁣:Ti​j\(2\)\+∑k=1rξi​kA​ξj​kB⏟=⁣:Ti​j\(3\)\.E\_\{ij\}\\;=\\;\\underbrace\{\\mu\_\{A\}\\sum\_\{k=1\}^\{r\}\\sigma^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\}\_\{=:T^\{\(1\)\}\_\{ij\}\}\\;\+\\;\\underbrace\{\\mu\_\{B\}\\sum\_\{k=1\}^\{r\}\\xi^\{A\}\_\{ik\}\\sigma^\{B\}\_\{jk\}\}\_\{=:T^\{\(2\)\}\_\{ij\}\}\\;\+\\;\\underbrace\{\\sum\_\{k=1\}^\{r\}\\xi^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\}\_\{=:T^\{\(3\)\}\_\{ij\}\}\.\(19\)Letℱ≔σ​\(σA,σB\)\{\\mathcal\{F\}\}\\coloneqq\\sigma\(\\sigma^\{A\},\\sigma^\{B\}\)denote theσ\\sigma\-algebra generated by the signs \(which reduces to the trivialσ\\sigma\-algebra when the signs are deterministic\)\. Then the three summands are pairwiseL2L^\{2\}\-orthogonal conditionally onℱ\{\\mathcal\{F\}\}:

𝔼​\[Ti​j\(a\)​Ti​j\(b\)\|ℱ\]=0for every distinct​a,b∈\{1,2,3\}\.\{\\mathbb\{E\}\}\\bigl\[T^\{\(a\)\}\_\{ij\}T^\{\(b\)\}\_\{ij\}\\big\|\{\\mathcal\{F\}\}\\bigr\]\\;=\\;0\\quad\\text\{for every distinct \}a,b\\in\\\{1,2,3\\\}\.\(20\)In particular, taking total expectation, the same orthogonality holds unconditionally\.

###### Proof\.

*Step 1: entry\-wise identity\.*Substituting Eq\. \([16](https://arxiv.org/html/2605.24058#A3.E16)\) into\(A​B⊤\)i​j=∑k=1rAi​k​Bj​k\(AB^\{\\top\}\)\_\{ij\}=\\sum\_\{k=1\}^\{r\}A\_\{ik\}B\_\{jk\}and expanding the scalar product\(μA​σi​kA\+ξi​kA\)​\(μB​σj​kB\+ξj​kB\)\(\\mu\_\{A\}\\sigma^\{A\}\_\{ik\}\+\\xi^\{A\}\_\{ik\}\)\(\\mu\_\{B\}\\sigma^\{B\}\_\{jk\}\+\\xi^\{B\}\_\{jk\}\)termwise,

\(A​B⊤\)i​j=μA​μB​∑k=1rσi​kA​σj​kB⏟=\[Δ​W​\(θ⋆\)\]i​j\+Ti​j\(1\)\+Ti​j\(2\)\+Ti​j\(3\),\(AB^\{\\top\}\)\_\{ij\}\\;=\\;\\underbrace\{\\mu\_\{A\}\\mu\_\{B\}\\sum\_\{k=1\}^\{r\}\\sigma^\{A\}\_\{ik\}\\sigma^\{B\}\_\{jk\}\}\_\{=\[\\Delta W\(\\theta^\{\\star\}\)\]\_\{ij\}\}\\;\+\\;T^\{\(1\)\}\_\{ij\}\+T^\{\(2\)\}\_\{ij\}\+T^\{\(3\)\}\_\{ij\},where the identification of the leading term with\[Δ​W​\(θ⋆\)\]i​j\[\\Delta W\(\\theta^\{\\star\}\)\]\_\{ij\}follows from Eq\. \([17](https://arxiv.org/html/2605.24058#A3.E17)\) together with\[diag⁡\(𝟏N\)​σA​diag⁡\(μA​μB​𝟏r\)​\(σB\)⊤​diag⁡\(𝟏M\)\]i​j=μA​μB​∑kσi​kA​σj​kB\[\\operatorname\{diag\}\(\{\\bm\{1\}\}\_\{N\}\)\\sigma^\{A\}\\operatorname\{diag\}\(\\mu\_\{A\}\\mu\_\{B\}\{\\bm\{1\}\}\_\{r\}\)\(\\sigma^\{B\}\)^\{\\top\}\\operatorname\{diag\}\(\{\\bm\{1\}\}\_\{M\}\)\]\_\{ij\}=\\mu\_\{A\}\\mu\_\{B\}\\sum\_\{k\}\\sigma^\{A\}\_\{ik\}\\sigma^\{B\}\_\{jk\}\. Subtracting this leading term yields Eq\. \([19](https://arxiv.org/html/2605.24058#A3.E19)\)\.

*Step 2: conditionalL2L^\{2\}\-orthogonality\.*By Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(iii\) the signσ\\sigma\-algebraℱ=σ​\(σA,σB\)\{\\mathcal\{F\}\}=\\sigma\(\\sigma^\{A\},\\sigma^\{B\}\)is independent ofσ​\(ξA,ξB\)\\sigma\(\\xi^\{A\},\\xi^\{B\}\), which implies that the conditional law of\(ξA,ξB\)\(\\xi^\{A\},\\xi^\{B\}\)givenℱ\{\\mathcal\{F\}\}equals its unconditional law; all residual moments appearing below therefore equal their unconditional counterparts\. Fix\(i,j\)\(i,j\)and recallTi​j\(1\)​Ti​j\(2\)=μA​μB​∑k,lσi​kA​σj​lB​ξj​kB​ξi​lAT^\{\(1\)\}\_\{ij\}T^\{\(2\)\}\_\{ij\}=\\mu\_\{A\}\\mu\_\{B\}\\sum\_\{k,l\}\\sigma^\{A\}\_\{ik\}\\sigma^\{B\}\_\{jl\}\\xi^\{B\}\_\{jk\}\\xi^\{A\}\_\{il\}\. The sign factors areℱ\{\\mathcal\{F\}\}\-measurable and can be pulled out of the conditional expectation; Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(i\) ensures thatξi​lA\\xi^\{A\}\_\{il\}andξj​kB\\xi^\{B\}\_\{jk\}are independent \(they belong to different independent families\) with𝔼​\[ξi​lA\]=𝔼​\[ξj​kB\]=0\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{il\}\]=\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jk\}\]=0, so

𝔼​\[Ti​j\(1\)​Ti​j\(2\)∣ℱ\]=μA​μB​∑k,l=1rσi​kA​σj​lB​𝔼​\[ξi​lA\]​𝔼​\[ξj​kB\]=0\.\{\\mathbb\{E\}\}\[T^\{\(1\)\}\_\{ij\}T^\{\(2\)\}\_\{ij\}\\mid\{\\mathcal\{F\}\}\]=\\mu\_\{A\}\\mu\_\{B\}\\sum\_\{k,l=1\}^\{r\}\\sigma^\{A\}\_\{ik\}\\sigma^\{B\}\_\{jl\}\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{il\}\]\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jk\}\]=0\.ForTi​j\(1\)​Ti​j\(3\)=μA​∑k,lσi​kA​ξj​kB​ξi​lA​ξj​lBT^\{\(1\)\}\_\{ij\}T^\{\(3\)\}\_\{ij\}=\\mu\_\{A\}\\sum\_\{k,l\}\\sigma^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\\xi^\{A\}\_\{il\}\\xi^\{B\}\_\{jl\}, the factorξi​lA\\xi^\{A\}\_\{il\}is independent of\(ξj​kB,ξj​lB\)\(\\xi^\{B\}\_\{jk\},\\xi^\{B\}\_\{jl\}\)\(different families, cf\. \(i\)\), so its expectation factorises and vanishes:

𝔼​\[Ti​j\(1\)​Ti​j\(3\)∣ℱ\]=μA​∑k,l=1rσi​kA​𝔼​\[ξi​lA\]​𝔼​\[ξj​kB​ξj​lB\]=0\.\{\\mathbb\{E\}\}\[T^\{\(1\)\}\_\{ij\}T^\{\(3\)\}\_\{ij\}\\mid\{\\mathcal\{F\}\}\]=\\mu\_\{A\}\\sum\_\{k,l=1\}^\{r\}\\sigma^\{A\}\_\{ik\}\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{il\}\]\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jk\}\\xi^\{B\}\_\{jl\}\]=0\.The identity𝔼​\[Ti​j\(2\)​Ti​j\(3\)∣ℱ\]=0\{\\mathbb\{E\}\}\[T^\{\(2\)\}\_\{ij\}T^\{\(3\)\}\_\{ij\}\\mid\{\\mathcal\{F\}\}\]=0is symmetric, replacing the roles ofAAandBB:

𝔼​\[Ti​j\(2\)​Ti​j\(3\)∣ℱ\]=μB​∑k,l=1rσj​kB​𝔼​\[ξj​lB\]​𝔼​\[ξi​kA​ξi​lA\]=0\.\{\\mathbb\{E\}\}\[T^\{\(2\)\}\_\{ij\}T^\{\(3\)\}\_\{ij\}\\mid\{\\mathcal\{F\}\}\]=\\mu\_\{B\}\\sum\_\{k,l=1\}^\{r\}\\sigma^\{B\}\_\{jk\}\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jl\}\]\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{ik\}\\xi^\{A\}\_\{il\}\]=0\.This establishes Eq\. \([20](https://arxiv.org/html/2605.24058#A3.E20)\)\. Taking expectations and using the tower property yields the unconditional orthogonality\. ∎

### C\.2Sub\-exponential tail of each entry

With Lemma[C\.2](https://arxiv.org/html/2605.24058#A3.Thmtheorem2)in hand, each summand in Eq\. \([19](https://arxiv.org/html/2605.24058#A3.E19)\) is either a sum of independent sub\-Gaussians or a sum of independent products of sub\-Gaussians; the latter is sub\-exponential\. A union bound over the three summands therefore yields a sub\-exponential tail forEi​jE\_\{ij\}\.

###### Lemma C\.3\(Sub\-exponential bound onEi​jE\_\{ij\}\)\.

LetV≔r​ζ2​\(μA2\+μB2\+ζ2\)V\\coloneqq r\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\. There exists a universal constantc1\>0c\_\{1\}\>0such that, for every\(i,j\)∈\[N\]×\[M\]\(i,j\)\\in\[N\]\\times\[M\]and everyt\>0t\>0,

Pr⁡\[\|Ei​j\|\>t\]≤6​exp⁡\(−c1​min⁡\(t2V,tζ2\)\)\.\\Pr\\bigl\[\|E\_\{ij\}\|\>t\\bigr\]\\;\\leq\\;6\\exp\\Bigl\(\-c\_\{1\}\\min\\Bigl\(\\frac\{t^\{2\}\}\{V\},\\frac\{t\}\{\\zeta^\{2\}\}\\Bigr\)\\Bigr\)\.\(21\)Moreover𝔼​\[Ei​j2\]≤V\{\\mathbb\{E\}\}\[E\_\{ij\}^\{2\}\]\\leq V, hence𝔼​‖E‖F2≤N​M​V\{\\mathbb\{E\}\}\\\|E\\\|\_\{F\}^\{2\}\\leq NMV\.

###### Proof\.

Fix\(i,j\)∈\[N\]×\[M\]\(i,j\)\\in\[N\]\\times\[M\]and condition on theσ\\sigma\-algebraℱ=σ​\(σA,σB\)\{\\mathcal\{F\}\}=\\sigma\(\\sigma^\{A\},\\sigma^\{B\}\)\. By Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(iii\) the residuals\(ξA,ξB\)\(\\xi^\{A\},\\xi^\{B\}\)are independent ofℱ\{\\mathcal\{F\}\}, so the conditional joint law of the residuals givenℱ\{\\mathcal\{F\}\}coincides with their unconditional law\. All probability statements in parts \(a\)–\(c\) below are understood conditionally onℱ\{\\mathcal\{F\}\}; the bounds are uniform over realisations ofℱ\{\\mathcal\{F\}\}and therefore also hold unconditionally\. Throughout we writeca,cb\>0c\_\{a\},c\_\{b\}\>0for the universal constants produced by Lemma[B\.2](https://arxiv.org/html/2605.24058#A2.Thmtheorem2)\(sub\-Gaussian sum tail\) and Lemma[B\.3](https://arxiv.org/html/2605.24058#A2.Thmtheorem3)\(Bernstein\) of Appendix[B](https://arxiv.org/html/2605.24058#A2)\.

*\(a\)Ti​j\(1\)T^\{\(1\)\}\_\{ij\}is sub\-Gaussian\.*For eachk∈\[r\]k\\in\[r\],\|σi​kA\|=1\|\\sigma^\{A\}\_\{ik\}\|=1and the positive\-homogeneity of∥⋅∥ψ2\\\|\\cdot\\\|\_\{\\psi\_\{2\}\}give‖σi​kA​ξj​kB‖ψ2=‖ξj​kB‖ψ2≤ζ\\\|\\sigma^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\\\|\_\{\\psi\_\{2\}\}=\\\|\\xi^\{B\}\_\{jk\}\\\|\_\{\\psi\_\{2\}\}\\leq\\zeta\. By Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(i\) the family\{ξj​kB\}k=1r\\\{\\xi^\{B\}\_\{jk\}\\\}\_\{k=1\}^\{r\}is mutually independent and theℱ\{\\mathcal\{F\}\}\-measurable signsσi​kA\\sigma^\{A\}\_\{ik\}preserve \(conditional\) independence and the zero mean\. Applying Lemma[B\.2](https://arxiv.org/html/2605.24058#A2.Thmtheorem2)conditionally onℱ\{\\mathcal\{F\}\}to\{Xk=σi​kA​ξj​kB\}k=1r\\\{X\_\{k\}=\\sigma^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\\\}\_\{k=1\}^\{r\}withak≡1a\_\{k\}\\equiv 1,K=ζK=\\zeta, scaled by the deterministic constantμA\\mu\_\{A\}, yields

Pr⁡\[\|Ti​j\(1\)\|\>t\|ℱ\]≤2​exp⁡\(−ca​t2/\(μA2​r​ζ2\)\)∀t≥0\.\\Pr\\bigl\[\|T^\{\(1\)\}\_\{ij\}\|\>t\\big\|\{\\mathcal\{F\}\}\\bigr\]\\;\\leq\\;2\\exp\\bigl\(\-c\_\{a\}t^\{2\}/\(\\mu\_\{A\}^\{2\}r\\zeta^\{2\}\)\\bigr\)\\qquad\\forall t\\geq 0\.\(22\)
*\(b\)Ti​j\(2\)T^\{\(2\)\}\_\{ij\}is sub\-Gaussian\.*Exchanging the roles of the two families in \(a\),

Pr⁡\[\|Ti​j\(2\)\|\>t\|ℱ\]≤2​exp⁡\(−ca​t2/\(μB2​r​ζ2\)\)∀t≥0\.\\Pr\\bigl\[\|T^\{\(2\)\}\_\{ij\}\|\>t\\big\|\{\\mathcal\{F\}\}\\bigr\]\\;\\leq\\;2\\exp\\bigl\(\-c\_\{a\}t^\{2\}/\(\\mu\_\{B\}^\{2\}r\\zeta^\{2\}\)\\bigr\)\\qquad\\forall t\\geq 0\.\(23\)
*\(c\)Ti​j\(3\)T^\{\(3\)\}\_\{ij\}is sub\-exponential\.*By Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(i\),ξi​kA\\xi^\{A\}\_\{ik\}andξj​kB\\xi^\{B\}\_\{jk\}are independent with zero mean, so Lemma[B\.3](https://arxiv.org/html/2605.24058#A2.Thmtheorem3)gives𝔼​\[ξi​kA​ξj​kB∣ℱ\]=𝔼​\[ξi​kA\]​𝔼​\[ξj​kB\]=0\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\\mid\{\\mathcal\{F\}\}\]=\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{ik\}\]\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jk\}\]=0and‖ξi​kA​ξj​kB‖ψ1≤‖ξi​kA‖ψ2​‖ξj​kB‖ψ2≤ζ2\\\|\\xi^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\\\|\_\{\\psi\_\{1\}\}\\leq\\\|\\xi^\{A\}\_\{ik\}\\\|\_\{\\psi\_\{2\}\}\\\|\\xi^\{B\}\_\{jk\}\\\|\_\{\\psi\_\{2\}\}\\leq\\zeta^\{2\}\. The summands\{ξi​kA​ξj​kB\}k=1r\\\{\\xi^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\\\}\_\{k=1\}^\{r\}are independent acrosskk: indeed,σ​\(ξi​kA​ξj​kB\)⊆σ​\(ξi​kA,ξj​kB\)\\sigma\(\\xi^\{A\}\_\{ik\}\\xi^\{B\}\_\{jk\}\)\\subseteq\\sigma\(\\xi^\{A\}\_\{ik\},\\xi^\{B\}\_\{jk\}\), and the pairs\(ξi​kA,ξj​kB\)\(\\xi^\{A\}\_\{ik\},\\xi^\{B\}\_\{jk\}\)for distinctk∈\[r\]k\\in\[r\]involve disjoint index sets in the mutually independent families\{ξ⋅⋅A\},\{ξ⋅⋅B\}\\\{\\xi^\{A\}\_\{\\cdot\\cdot\}\\\},\\\{\\xi^\{B\}\_\{\\cdot\\cdot\}\\\}\. The Bernstein bound Eq\. \([15](https://arxiv.org/html/2605.24058#A2.E15)\) of Lemma[B\.3](https://arxiv.org/html/2605.24058#A2.Thmtheorem3)therefore yields, withcb\>0c\_\{b\}\>0the universal constant of that lemma,

Pr⁡\[\|Ti​j\(3\)\|\>t\|ℱ\]≤2​exp⁡\(−cb​min⁡\(t2/\(r​ζ4\),t/ζ2\)\)∀t≥0\.\\Pr\\bigl\[\|T^\{\(3\)\}\_\{ij\}\|\>t\\big\|\{\\mathcal\{F\}\}\\bigr\]\\;\\leq\\;2\\exp\\Bigl\(\-c\_\{b\}\\min\\bigl\(t^\{2\}/\(r\\zeta^\{4\}\),t/\\zeta^\{2\}\\bigr\)\\Bigr\)\\qquad\\forall t\\geq 0\.\(24\)
*Combination\.*Triangle inequality gives\|Ei​j\|≤\|Ti​j\(1\)\|\+\|Ti​j\(2\)\|\+\|Ti​j\(3\)\|\|E\_\{ij\}\|\\leq\|T^\{\(1\)\}\_\{ij\}\|\+\|T^\{\(2\)\}\_\{ij\}\|\+\|T^\{\(3\)\}\_\{ij\}\|, so

\{\|Ei​j\|\>t\}⊆\{\|Ti​j\(1\)\|\>t/3\}∪\{\|Ti​j\(2\)\|\>t/3\}∪\{\|Ti​j\(3\)\|\>t/3\}\.\\\{\|E\_\{ij\}\|\>t\\\}\\;\\subseteq\\;\\\{\|T^\{\(1\)\}\_\{ij\}\|\>t/3\\\}\\cup\\\{\|T^\{\(2\)\}\_\{ij\}\|\>t/3\\\}\\cup\\\{\|T^\{\(3\)\}\_\{ij\}\|\>t/3\\\}\.SinceV=r​ζ2​\(μA2\+μB2\+ζ2\)V=r\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)dominates each ofμA2​r​ζ2\\mu\_\{A\}^\{2\}r\\zeta^\{2\},μB2​r​ζ2\\mu\_\{B\}^\{2\}r\\zeta^\{2\}, andr​ζ4r\\zeta^\{4\}, Eqs\. \([22](https://arxiv.org/html/2605.24058#A3.E22)\)–\([23](https://arxiv.org/html/2605.24058#A3.E23)\) att/3t/3are bounded by2​exp⁡\(−ca​t2/\(9​V\)\)2\\exp\(\-c\_\{a\}t^\{2\}/\(9V\)\), and Eq\. \([24](https://arxiv.org/html/2605.24058#A3.E24)\) att/3t/3is bounded by2​exp⁡\(−cb​min⁡\(t2/\(9​V\),t/\(3​ζ2\)\)\)2\\exp\(\-c\_\{b\}\\min\(t^\{2\}/\(9V\),t/\(3\\zeta^\{2\}\)\)\)\. Defining

c1≔19​min⁡\(ca,cb\),c\_\{1\}\\;\\coloneqq\\;\\frac\{1\}\{9\}\\min\(c\_\{a\},c\_\{b\}\),\(25\)each of the three bounds is dominated by2​exp⁡\(−c1​min⁡\(t2/V,t/ζ2\)\)2\\exp\(\-c\_\{1\}\\min\(t^\{2\}/V,t/\\zeta^\{2\}\)\): indeedca/9≥c1c\_\{a\}/9\\geq c\_\{1\},cb/9≥c1c\_\{b\}/9\\geq c\_\{1\}, andcb/3≥c1c\_\{b\}/3\\geq c\_\{1\}\. Taking the union over the three events yields Eq\. \([21](https://arxiv.org/html/2605.24058#A3.E21)\) conditionally onℱ\{\\mathcal\{F\}\}; uniformity overℱ\{\\mathcal\{F\}\}gives the unconditional bound\.

*Second moment\.*Lemma[B\.1](https://arxiv.org/html/2605.24058#A2.Thmtheorem1)gives𝔼​\[\(ξi​kA\)2\]≤ζ2\{\\mathbb\{E\}\}\[\(\\xi^\{A\}\_\{ik\}\)^\{2\}\]\\leq\\zeta^\{2\}and𝔼​\[\(ξj​kB\)2\]≤ζ2\{\\mathbb\{E\}\}\[\(\\xi^\{B\}\_\{jk\}\)^\{2\}\]\\leq\\zeta^\{2\}\. Using the mutual independence of Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(i\), zero means, and\(σ⋅⋅⋅\)2=1\(\\sigma^\{\\cdot\}\_\{\\cdot\\cdot\}\)^\{2\}=1,

𝔼​\[\(Ti​j\(1\)\)2∣ℱ\]\\displaystyle\{\\mathbb\{E\}\}\[\(T^\{\(1\)\}\_\{ij\}\)^\{2\}\\mid\{\\mathcal\{F\}\}\]=μA2​∑k,l=1rσi​kA​σi​lA​𝔼​\[ξj​kB​ξj​lB\]=μA2​∑k=1r𝔼​\[\(ξj​kB\)2\]≤μA2​r​ζ2,\\displaystyle=\\mu\_\{A\}^\{2\}\\sum\_\{k,l=1\}^\{r\}\\sigma^\{A\}\_\{ik\}\\sigma^\{A\}\_\{il\}\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jk\}\\xi^\{B\}\_\{jl\}\]=\\mu\_\{A\}^\{2\}\\sum\_\{k=1\}^\{r\}\{\\mathbb\{E\}\}\[\(\\xi^\{B\}\_\{jk\}\)^\{2\}\]\\;\\leq\\;\\mu\_\{A\}^\{2\}r\\zeta^\{2\},𝔼​\[\(Ti​j\(2\)\)2∣ℱ\]\\displaystyle\{\\mathbb\{E\}\}\[\(T^\{\(2\)\}\_\{ij\}\)^\{2\}\\mid\{\\mathcal\{F\}\}\]≤μB2​r​ζ2,\\displaystyle\\leq\\;\\mu\_\{B\}^\{2\}r\\zeta^\{2\},𝔼​\[\(Ti​j\(3\)\)2∣ℱ\]\\displaystyle\{\\mathbb\{E\}\}\[\(T^\{\(3\)\}\_\{ij\}\)^\{2\}\\mid\{\\mathcal\{F\}\}\]=∑k,l=1r𝔼​\[ξi​kA​ξi​lA\]​𝔼​\[ξj​kB​ξj​lB\]=∑k=1r𝔼​\[\(ξi​kA\)2\]​𝔼​\[\(ξj​kB\)2\]≤r​ζ4,\\displaystyle=\\sum\_\{k,l=1\}^\{r\}\{\\mathbb\{E\}\}\[\\xi^\{A\}\_\{ik\}\\xi^\{A\}\_\{il\}\]\{\\mathbb\{E\}\}\[\\xi^\{B\}\_\{jk\}\\xi^\{B\}\_\{jl\}\]=\\sum\_\{k=1\}^\{r\}\{\\mathbb\{E\}\}\[\(\\xi^\{A\}\_\{ik\}\)^\{2\}\]\{\\mathbb\{E\}\}\[\(\\xi^\{B\}\_\{jk\}\)^\{2\}\]\\;\\leq\\;r\\zeta^\{4\},where the off\-diagonal \(k≠lk\\neq l\) cross terms vanish because the residuals within each family are mean zero and mutually independent\. Combined with the conditional orthogonality of Lemma[C\.2](https://arxiv.org/html/2605.24058#A3.Thmtheorem2),𝔼​\[Ei​j2∣ℱ\]=∑a=13𝔼​\[\(Ti​j\(a\)\)2∣ℱ\]≤V\{\\mathbb\{E\}\}\[E\_\{ij\}^\{2\}\\mid\{\\mathcal\{F\}\}\]=\\sum\_\{a=1\}^\{3\}\{\\mathbb\{E\}\}\[\(T^\{\(a\)\}\_\{ij\}\)^\{2\}\\mid\{\\mathcal\{F\}\}\]\\leq V\. Taking total expectation and summing over\(i,j\)∈\[N\]×\[M\]\(i,j\)\\in\[N\]\\times\[M\]yields𝔼​‖E‖F2≤N​M​V\{\\mathbb\{E\}\}\\\|E\\\|\_\{F\}^\{2\}\\leq NMV\. ∎

### C\.3Frobenius bound by union

The per\-entry bound of Lemma[C\.3](https://arxiv.org/html/2605.24058#A3.Thmtheorem3)transforms into a Frobenius bound by a union argument over theN​MNMentries ofEE\.

###### Proof of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1), Eq\. \([27](https://arxiv.org/html/2605.24058#A3.E27)\)\.

FixN,M≥2N,M\\geq 2,r≥1r\\geq 1, andδ∈\(0,1\)\\delta\\in\(0,1\)withu≔log⁡\(6​N​M/δ\)≤c1​ru\\coloneqq\\log\(6NM/\\delta\)\\leq c\_\{1\}r, wherec1c\_\{1\}is the constant defined in Eq\. \([25](https://arxiv.org/html/2605.24058#A3.E25)\)\. SinceN​M≥4NM\\geq 4andδ<1\\delta<1,u≥log⁡24\>0u\\geq\\log 24\>0\. Sett⋆≔V​u/c1\>0t\_\{\\star\}\\coloneqq\\sqrt\{Vu/c\_\{1\}\}\>0\.

*Step 1: the sub\-Gaussian branch binds att=t⋆t=t\_\{\\star\}\.*In Eq\. \([21](https://arxiv.org/html/2605.24058#A3.E21)\) the inner minimum satisfiesmin⁡\(c1​t2/V,c1​t/ζ2\)=c1​t2/V\\min\(c\_\{1\}t^\{2\}/V,c\_\{1\}t/\\zeta^\{2\}\)=c\_\{1\}t^\{2\}/Vifft2/V≤t/ζ2t^\{2\}/V\\leq t/\\zeta^\{2\}, ifft≤V/ζ2t\\leq V/\\zeta^\{2\}\. Att=t⋆t=t\_\{\\star\}this becomesV​u/c1≤V/ζ2\\sqrt\{Vu/c\_\{1\}\}\\leq V/\\zeta^\{2\}, equivalentlyu≤c1​V/ζ4u\\leq c\_\{1\}V/\\zeta^\{4\}\. FromV=r​ζ2​\(μA2\+μB2\+ζ2\)V=r\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)andζ\>0\\zeta\>0,

Vζ4=r​\(μA2ζ2\+μB2ζ2\+1\)≥r,\\frac\{V\}\{\\zeta^\{4\}\}\\;=\\;r\\Bigl\(\\frac\{\\mu\_\{A\}^\{2\}\}\{\\zeta^\{2\}\}\+\\frac\{\\mu\_\{B\}^\{2\}\}\{\\zeta^\{2\}\}\+1\\Bigr\)\\;\\geq\\;r,so the standing hypothesisu≤c1​ru\\leq c\_\{1\}rentailsu≤c1​V/ζ4u\\leq c\_\{1\}V/\\zeta^\{4\}, and the sub\-Gaussian branch binds:

c1​min⁡\(t⋆2/V,t⋆/ζ2\)=c1​t⋆2/V=u\.c\_\{1\}\\min\(t\_\{\\star\}^\{2\}/V,t\_\{\\star\}/\\zeta^\{2\}\)\\;=\\;c\_\{1\}t\_\{\\star\}^\{2\}/V\\;=\\;u\.
*Step 2: per\-entry bound\.*Substitutingt=t⋆t=t\_\{\\star\}into Eq\. \([21](https://arxiv.org/html/2605.24058#A3.E21)\) and using Step 1,

Pr⁡\[Ei​j2\>V​u/c1\]=Pr⁡\[\|Ei​j\|\>t⋆\]≤6​e−u=δN​M,\(i,j\)∈\[N\]×\[M\]\.\\Pr\\bigl\[E\_\{ij\}^\{2\}\>Vu/c\_\{1\}\\bigr\]\\;=\\;\\Pr\\bigl\[\|E\_\{ij\}\|\>t\_\{\\star\}\\bigr\]\\;\\leq\\;6e^\{\-u\}\\;=\\;\\frac\{\\delta\}\{NM\},\\qquad\(i,j\)\\in\[N\]\\times\[M\]\.
*Step 3: union bound\.*Ifmax\(i,j\)⁡Ei​j2≤V​u/c1\\max\_\{\(i,j\)\}E\_\{ij\}^\{2\}\\leq Vu/c\_\{1\}, then‖E‖F2=∑i,jEi​j2≤N​M⋅V​u/c1\\\|E\\\|\_\{F\}^\{2\}=\\sum\_\{i,j\}E\_\{ij\}^\{2\}\\leq NM\\cdot Vu/c\_\{1\}\. Contrapositively,\{‖E‖F2\>N​M​V​u/c1\}⊆⋃i,j\{Ei​j2\>V​u/c1\}\\\{\\\|E\\\|\_\{F\}^\{2\}\>NMVu/c\_\{1\}\\\}\\subseteq\\bigcup\_\{i,j\}\\\{E\_\{ij\}^\{2\}\>Vu/c\_\{1\}\\\}\. A union bound over theN​MNMentries givesPr⁡\[‖E‖F2\>N​M​V​u/c1\]≤N​M⋅δ/\(N​M\)=δ\\Pr\[\\\|E\\\|\_\{F\}^\{2\}\>NMVu/c\_\{1\}\]\\leq NM\\cdot\\delta/\(NM\)=\\delta\. Hence, with probability at least1−δ1\-\\delta,

‖E‖F2≤N​M​r​ζ2​\(μA2\+μB2\+ζ2\)c1​log⁡\(6​N​M/δ\)\.\\\|E\\\|\_\{F\}^\{2\}\\;\\leq\\;\\frac\{NMr\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\}\{c\_\{1\}\}\\log\(6NM/\\delta\)\.\(26\)
*Step 4: conversionlog⁡\(6​N​M/δ\)→log⁡\(2​N​M/δ\)\\log\(6NM/\\delta\)\\to\\log\(2NM/\\delta\)\.*ForN,M≥2N,M\\geq 2andδ∈\(0,1\)\\delta\\in\(0,1\),2​N​M/δ≥82NM/\\delta\\geq 8, solog⁡\(2​N​M/δ\)≥log⁡8\>log⁡3\\log\(2NM/\\delta\)\\geq\\log 8\>\\log 3\. Therefore

log⁡\(6​N​M/δ\)=log⁡3\+log⁡\(2​N​M/δ\)≤log⁡\(2​N​M/δ\)\+log⁡\(2​N​M/δ\)=2​log⁡\(2​N​M/δ\)\.\\log\(6NM/\\delta\)\\;=\\;\\log 3\+\\log\(2NM/\\delta\)\\;\\leq\\;\\log\(2NM/\\delta\)\+\\log\(2NM/\\delta\)\\;=\\;2\\log\(2NM/\\delta\)\.Substituting into Eq\. \([26](https://arxiv.org/html/2605.24058#A3.E26)\) and settingC≔2/c1\>0C\\coloneqq 2/c\_\{1\}\>0yields the absolute bound

‖A​B⊤−Δ​W​\(θ⋆\)‖F2≤C⋅N​M​r​ζ2​\(μA2\+μB2\+ζ2\)​log⁡\(2​N​M/δ\)\.\\big\\\|AB^\{\\top\}\-\\Delta W\(\\theta^\{\\star\}\)\\big\\\|\_\{F\}^\{2\}\\;\\leq\\;C\\cdot NMr\\zeta^\{2\}\\bigl\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\\bigr\)\\log\(2NM/\\delta\)\.\(27\)The constantCCdepends only on the universal constantsca,cbc\_\{a\},c\_\{b\}produced by Lemma[B\.2](https://arxiv.org/html/2605.24058#A2.Thmtheorem2)and Lemma[B\.3](https://arxiv.org/html/2605.24058#A2.Thmtheorem3)of Appendix[B](https://arxiv.org/html/2605.24058#A2)through Eq\. \([25](https://arxiv.org/html/2605.24058#A3.E25)\), and is otherwise independent of all model parameters\. ∎

### C\.4Chebyshev lower bound on the signal

The relative bound requires a lower bound on‖Δ​W​\(θ⋆\)‖F2\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}\. This is supplied by a second\-moment argument on the Rademacher bilinear form of the signs\.

###### Lemma C\.4\(Lower bound on‖Δ​W​\(θ⋆\)‖F2\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}\)\.

Assume\{σi​kA\}∪\{σj​kB\}\\\{\\sigma^\{A\}\_\{ik\}\\\}\\cup\\\{\\sigma^\{B\}\_\{jk\}\\\}are i\.i\.d\. Rademacher withσA⟂σB\\sigma^\{A\}\\perp\\sigma^\{B\}\. LetWi​j≔∑k=1rσi​kA​σj​kBW\_\{ij\}\\coloneqq\\sum\_\{k=1\}^\{r\}\\sigma^\{A\}\_\{ik\}\\sigma^\{B\}\_\{jk\}\. Then for everyN,M≥1N,M\\geq 1,r≥1r\\geq 1,

Pr⁡\[∑i,jWi​j2≥12​N​M​r\]≥1−8N​M\.\\Pr\\Bigl\[\\textstyle\\sum\_\{i,j\}W\_\{ij\}^\{2\}\\geq\\frac\{1\}\{2\}NMr\\Bigr\]\\;\\geq\\;1\-\\frac\{8\}\{NM\}\.\(28\)In particular, on this event‖Δ​W​\(θ⋆\)‖F2=μA2​μB2​∑i​jWi​j2≥12​μA2​μB2​N​M​r\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}=\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}\\sum\_\{ij\}W\_\{ij\}^\{2\}\\geq\\frac\{1\}\{2\}\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}NMr\.

###### Proof\.

DefineZ≔∑i,jWi​j2Z\\coloneqq\\sum\_\{i,j\}W\_\{ij\}^\{2\}and, fork,l∈\[r\]k,l\\in\[r\],

Gk​lA≔∑i=1Nσi​kA​σi​lA,Gk​lB≔∑j=1Mσj​kB​σj​lB,G^\{A\}\_\{kl\}\\;\\coloneqq\\;\\sum\_\{i=1\}^\{N\}\\sigma^\{A\}\_\{ik\}\\sigma^\{A\}\_\{il\},\\qquad G^\{B\}\_\{kl\}\\;\\coloneqq\\;\\sum\_\{j=1\}^\{M\}\\sigma^\{B\}\_\{jk\}\\sigma^\{B\}\_\{jl\},so thatGk​lA=Gl​kAG^\{A\}\_\{kl\}=G^\{A\}\_\{lk\}andGk​lB=Gl​kBG^\{B\}\_\{kl\}=G^\{B\}\_\{lk\}\. ExpandingWi​j2=∑k,lσi​kA​σi​lA​σj​kB​σj​lBW\_\{ij\}^\{2\}=\\sum\_\{k,l\}\\sigma^\{A\}\_\{ik\}\\sigma^\{A\}\_\{il\}\\sigma^\{B\}\_\{jk\}\\sigma^\{B\}\_\{jl\}and interchanging the finite sums over\(i,j\)\(i,j\)and\(k,l\)\(k,l\),

Z=∑k,l=1rGk​lA​Gk​lB=∑k=1rGk​kA​Gk​kB⏟=r​N​M\+∑k≠lGk​lA​Gk​lB,Z\\;=\\;\\sum\_\{k,l=1\}^\{r\}G^\{A\}\_\{kl\}G^\{B\}\_\{kl\}\\;=\\;\\underbrace\{\\sum\_\{k=1\}^\{r\}G^\{A\}\_\{kk\}G^\{B\}\_\{kk\}\}\_\{=rNM\}\\;\+\\;\\sum\_\{k\\neq l\}G^\{A\}\_\{kl\}G^\{B\}\_\{kl\},\(29\)where the first sum is deterministic becauseGk​kA=∑i\(σi​kA\)2=NG^\{A\}\_\{kk\}=\\sum\_\{i\}\(\\sigma^\{A\}\_\{ik\}\)^\{2\}=NandGk​kB=MG^\{B\}\_\{kk\}=M\. Ifr=1r=1the off\-diagonal sum is empty,Z=N​M=r​N​M≥r​N​M/2Z=NM=rNM\\geq rNM/2deterministically, and Eq\. \([28](https://arxiv.org/html/2605.24058#A3.E28)\) is trivial\. We henceforth assumer≥2r\\geq 2\.

*Mean\.*Fork≠lk\\neq land fixedi∈\[N\]i\\in\[N\],σi​kA\\sigma^\{A\}\_\{ik\}andσi​lA\\sigma^\{A\}\_\{il\}are independent Rademachers, hence𝔼​\[σi​kA​σi​lA\]=𝔼​\[σi​kA\]​𝔼​\[σi​lA\]=0\{\\mathbb\{E\}\}\[\\sigma^\{A\}\_\{ik\}\\sigma^\{A\}\_\{il\}\]=\{\\mathbb\{E\}\}\[\\sigma^\{A\}\_\{ik\}\]\{\\mathbb\{E\}\}\[\\sigma^\{A\}\_\{il\}\]=0, so𝔼​\[Gk​lA\]=0\{\\mathbb\{E\}\}\[G^\{A\}\_\{kl\}\]=0and similarly𝔼​\[Gk​lB\]=0\{\\mathbb\{E\}\}\[G^\{B\}\_\{kl\}\]=0\. ByσA⟂σB\\sigma^\{A\}\\perp\\sigma^\{B\},𝔼​\[Gk​lA​Gk​lB\]=𝔼​\[Gk​lA\]​𝔼​\[Gk​lB\]=0\{\\mathbb\{E\}\}\[G^\{A\}\_\{kl\}G^\{B\}\_\{kl\}\]=\{\\mathbb\{E\}\}\[G^\{A\}\_\{kl\}\]\{\\mathbb\{E\}\}\[G^\{B\}\_\{kl\}\]=0\. Hence𝔼​\[Z\]=r​N​M\{\\mathbb\{E\}\}\[Z\]=rNM\.

*Variance\.*By symmetry,∑k≠lGk​lA​Gk​lB=2​S\\sum\_\{k\\neq l\}G^\{A\}\_\{kl\}G^\{B\}\_\{kl\}=2SwithS≔∑1≤k<l≤rGk​lA​Gk​lBS\\coloneqq\\sum\_\{1\\leq k<l\\leq r\}G^\{A\}\_\{kl\}G^\{B\}\_\{kl\}\. We claim

𝔼​\[Gk​lA​Gk′​l′A\]=N⋅𝟏​\{\(k,l\)=\(k′,l′\)\}for​k<l,k′<l′\.\{\\mathbb\{E\}\}\\bigl\[G^\{A\}\_\{kl\}G^\{A\}\_\{k^\{\\prime\}l^\{\\prime\}\}\\bigr\]\\;=\\;N\\cdot\\mathbf\{1\}\\\{\(k,l\)=\(k^\{\\prime\},l^\{\\prime\}\)\\\}\\qquad\\text\{for \}k<l,~k^\{\\prime\}<l^\{\\prime\}\.\(30\)Indeed,Gk​lA​Gk′​l′A=∑i,i′=1Nσi​kA​σi​lA​σi′​k′A​σi′​l′AG^\{A\}\_\{kl\}G^\{A\}\_\{k^\{\\prime\}l^\{\\prime\}\}=\\sum\_\{i,i^\{\\prime\}=1\}^\{N\}\\sigma^\{A\}\_\{ik\}\\sigma^\{A\}\_\{il\}\\sigma^\{A\}\_\{i^\{\\prime\}k^\{\\prime\}\}\\sigma^\{A\}\_\{i^\{\\prime\}l^\{\\prime\}\}\. By the joint i\.i\.d\. Rademacher hypothesis, all\{σp​qA\}\(p,q\)∈\[N\]×\[r\]\\\{\\sigma^\{A\}\_\{pq\}\\\}\_\{\(p,q\)\\in\[N\]\\times\[r\]\}are mutually independent, so the expectation of any finite product factorises across distinct\(p,q\)\(p,q\)pairs\. We split by the coincidence pattern ofiiandi′i^\{\\prime\}:

- •*i≠i′i\\neq i^\{\\prime\}\.*The four indices\(i,k\),\(i,l\),\(i′,k′\),\(i′,l′\)\(i,k\),\(i,l\),\(i^\{\\prime\},k^\{\\prime\}\),\(i^\{\\prime\},l^\{\\prime\}\)are pairwise distinct \(ask≠lk\\neq landk′≠l′k^\{\\prime\}\\neq l^\{\\prime\}andi≠i′i\\neq i^\{\\prime\}\), so the expectation factorises into four separate factors𝔼​\[σi​kA\]​⋯​𝔼​\[σi′​l′A\]=0\{\\mathbb\{E\}\}\[\\sigma^\{A\}\_\{ik\}\]\\cdots\{\\mathbb\{E\}\}\[\\sigma^\{A\}\_\{i^\{\\prime\}l^\{\\prime\}\}\]=0\.
- •*i=i′i=i^\{\\prime\}\.*The product becomesσi​kA​σi​lA​σi​k′A​σi​l′A\\sigma^\{A\}\_\{ik\}\\sigma^\{A\}\_\{il\}\\sigma^\{A\}\_\{ik^\{\\prime\}\}\\sigma^\{A\}\_\{il^\{\\prime\}\}, whose expectation equals11iff each column indexq∈\{k,l,k′,l′\}q\\in\\\{k,l,k^\{\\prime\},l^\{\\prime\}\\\}appears with even multiplicity in the multiset and equals0otherwise\. Underk<lk<landk′<l′k^\{\\prime\}<l^\{\\prime\}, even multiplicity forces\{k,l\}=\{k′,l′\}\\\{k,l\\\}=\\\{k^\{\\prime\},l^\{\\prime\}\\\}as multisets, hence\(k,l\)=\(k′,l′\)\(k,l\)=\(k^\{\\prime\},l^\{\\prime\}\)\. In the diagonal case each index appears exactly twice,\(σi⁣⋅A\)2=1\(\\sigma^\{A\}\_\{i\\cdot\}\)^\{2\}=1, and the expectation is11; summing overi∈\[N\]i\\in\[N\]contributesNN\.

This proves Eq\. \([30](https://arxiv.org/html/2605.24058#A3.E30)\); the identical argument withBBin place ofAAgives𝔼​\[Gk​lB​Gk′​l′B\]=M⋅𝟏​\{\(k,l\)=\(k′,l′\)\}\{\\mathbb\{E\}\}\[G^\{B\}\_\{kl\}G^\{B\}\_\{k^\{\\prime\}l^\{\\prime\}\}\]=M\\cdot\\mathbf\{1\}\\\{\(k,l\)=\(k^\{\\prime\},l^\{\\prime\}\)\\\}fork<lk<l,k′<l′k^\{\\prime\}<l^\{\\prime\}\. ByσA⟂σB\\sigma^\{A\}\\perp\\sigma^\{B\},

𝔼​\[Gk​lA​Gk​lB​Gk′​l′A​Gk′​l′B\]=𝔼​\[Gk​lA​Gk′​l′A\]​𝔼​\[Gk​lB​Gk′​l′B\]=N​M⋅𝟏​\{\(k,l\)=\(k′,l′\)\}\.\{\\mathbb\{E\}\}\\bigl\[G^\{A\}\_\{kl\}G^\{B\}\_\{kl\}G^\{A\}\_\{k^\{\\prime\}l^\{\\prime\}\}G^\{B\}\_\{k^\{\\prime\}l^\{\\prime\}\}\\bigr\]=\{\\mathbb\{E\}\}\\bigl\[G^\{A\}\_\{kl\}G^\{A\}\_\{k^\{\\prime\}l^\{\\prime\}\}\\bigr\]\{\\mathbb\{E\}\}\\bigl\[G^\{B\}\_\{kl\}G^\{B\}\_\{k^\{\\prime\}l^\{\\prime\}\}\\bigr\]=NM\\cdot\\mathbf\{1\}\\\{\(k,l\)=\(k^\{\\prime\},l^\{\\prime\}\)\\\}\.Summing over the\(r2\)\\binom\{r\}\{2\}pairs withk<lk<l,

𝔼​\[S2\]=∑k<l,k′<l′𝔼​\[Gk​lA​Gk​lB​Gk′​l′A​Gk′​l′B\]=\(r2\)​N​M\.\{\\mathbb\{E\}\}\[S^\{2\}\]\\;=\\;\\sum\_\{\\begin\{subarray\}\{c\}k<l,\\\\ k^\{\\prime\}<l^\{\\prime\}\\end\{subarray\}\}\{\\mathbb\{E\}\}\\bigl\[G^\{A\}\_\{kl\}G^\{B\}\_\{kl\}G^\{A\}\_\{k^\{\\prime\}l^\{\\prime\}\}G^\{B\}\_\{k^\{\\prime\}l^\{\\prime\}\}\\bigr\]\\;=\\;\\binom\{r\}\{2\}NM\.Combined with Eq\. \([29](https://arxiv.org/html/2605.24058#A3.E29)\) andZ−r​N​M=2​SZ\-rNM=2S,

Var​\(Z\)=𝔼​\[\(Z−𝔼​\[Z\]\)2\]=4​𝔼​\[S2\]=2​r​\(r−1\)​N​M≤2​r2​N​M\.\\mathrm\{Var\}\(Z\)\\;=\\;\{\\mathbb\{E\}\}\[\(Z\-\{\\mathbb\{E\}\}\[Z\]\)^\{2\}\]\\;=\\;4\{\\mathbb\{E\}\}\[S^\{2\}\]\\;=\\;2r\(r\-1\)NM\\;\\leq\\;2r^\{2\}NM\.
*Chebyshev\.*SinceZ≥0Z\\geq 0,\{Z<r​N​M/2\}=\{r​N​M−Z\>r​N​M/2\}⊆\{\|Z−r​N​M\|\>r​N​M/2\}\\\{Z<rNM/2\\\}=\\\{rNM\-Z\>rNM/2\\\}\\subseteq\\\{\|Z\-rNM\|\>rNM/2\\\}\. Chebyshev’s inequality therefore gives

Pr⁡\[Z<r​N​M/2\]≤Pr⁡\[\|Z−r​N​M\|\>r​N​M/2\]≤Var​\(Z\)\(r​N​M/2\)2≤2​r2​N​Mr2​N2​M2/4=8N​M,\\Pr\\bigl\[Z<rNM/2\\bigr\]\\;\\leq\\;\\Pr\\bigl\[\|Z\-rNM\|\>rNM/2\\bigr\]\\;\\leq\\;\\frac\{\\mathrm\{Var\}\(Z\)\}\{\(rNM/2\)^\{2\}\}\\;\\leq\\;\\frac\{2r^\{2\}NM\}\{r^\{2\}N^\{2\}M^\{2\}/4\}\\;=\\;\\frac\{8\}\{NM\},which is the complementary form of Eq\. \([28](https://arxiv.org/html/2605.24058#A3.E28)\)\. ∎

###### Proof of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1), Eq\. \([10](https://arxiv.org/html/2605.24058#S4.E10)\)\.

*Transfer of the absolute bound to the joint space\.*In the relative statement of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)we assumeN​M\>8NM\>8andδ∈\(0,1−8/\(N​M\)\)\\delta\\in\(0,1\-8/\(NM\)\), so the final probability lower bound is positive\. Under the Rademacher hypothesis on the signs, the arguments in Lemmas[C\.2](https://arxiv.org/html/2605.24058#A3.Thmtheorem2)–[C\.3](https://arxiv.org/html/2605.24058#A3.Thmtheorem3)and in the preceding proof were performed conditionally onℱ=σ​\(σA,σB\)\{\\mathcal\{F\}\}=\\sigma\(\\sigma^\{A\},\\sigma^\{B\}\)\. In particular, for every realisation\(sA,sB\)\(s^\{A\},s^\{B\}\)of\(σA,σB\)\(\\sigma^\{A\},\\sigma^\{B\}\)the conditional probability

Pr⁡\[ℰ1\|σA=sA,σB=sB\]≥1−δ,ℰ1≔\{‖E‖F2≤C​N​M​r​ζ2​\(μA2\+μB2\+ζ2\)​log⁡\(2​N​M/δ\)\}\.\\Pr\\bigl\[\{\\mathcal\{E\}\}\_\{1\}\\bigl\|\\sigma^\{A\}=s^\{A\},\\sigma^\{B\}=s^\{B\}\\bigr\]\\;\\geq\\;1\-\\delta,\\qquad\{\\mathcal\{E\}\}\_\{1\}\\;\\coloneqq\\;\\bigl\\\{\\\|E\\\|\_\{F\}^\{2\}\\leq CNMr\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\\log\(2NM/\\delta\)\\bigr\\\}\.Taking expectation over the signs \(tower property\) givesPr⁡\[ℰ1\]≥1−δ\\Pr\[\{\\mathcal\{E\}\}\_\{1\}\]\\geq 1\-\\deltaon the joint space\. Letℰ2≔\{∑i,jWi​j2≥N​M​r/2\}\{\\mathcal\{E\}\}\_\{2\}\\coloneqq\\\{\\sum\_\{i,j\}W\_\{ij\}^\{2\}\\geq NMr/2\\\}be the signal event of Lemma[C\.4](https://arxiv.org/html/2605.24058#A3.Thmtheorem4); it depends only on the signs and hasPr⁡\[ℰ2\]≥1−8/\(N​M\)\\Pr\[\{\\mathcal\{E\}\}\_\{2\}\]\\geq 1\-8/\(NM\)\. The union boundPr⁡\[ℰ1c∪ℰ2c\]≤δ\+8/\(N​M\)\\Pr\[\{\\mathcal\{E\}\}\_\{1\}^\{c\}\\cup\{\\mathcal\{E\}\}\_\{2\}^\{c\}\]\\leq\\delta\+8/\(NM\)yields

Pr⁡\[ℰ1∩ℰ2\]≥1−δ−8N​M\.\\Pr\[\{\\mathcal\{E\}\}\_\{1\}\\cap\{\\mathcal\{E\}\}\_\{2\}\]\\;\\geq\\;1\-\\delta\-\\frac\{8\}\{NM\}\.
*Ratio onℰ1∩ℰ2\{\\mathcal\{E\}\}\_\{1\}\\cap\{\\mathcal\{E\}\}\_\{2\}\.*By definition ofℰ2\{\\mathcal\{E\}\}\_\{2\},

‖Δ​W​\(θ⋆\)‖F2=μA2​μB2​∑i,jWi​j2≥12​μA2​μB2​N​M​r\.\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}=\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}\\sum\_\{i,j\}W\_\{ij\}^\{2\}\\;\\geq\\;\\tfrac\{1\}\{2\}\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}NMr\.Combining with the formal upper bound on‖E‖F2\\\|E\\\|\_\{F\}^\{2\}provided byℰ1\{\\mathcal\{E\}\}\_\{1\},

‖E‖F2‖Δ​W​\(θ⋆\)‖F2≤C​N​M​r​ζ2​\(μA2\+μB2\+ζ2\)​log⁡\(2​N​M/δ\)12​μA2​μB2​N​M​r=2​C⋅ζ2​\(μA2\+μB2\+ζ2\)μA2​μB2​log⁡\(2​N​M/δ\)\.\\frac\{\\\|E\\\|\_\{F\}^\{2\}\}\{\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}\}\\;\\leq\\;\\frac\{CNMr\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\\log\(2NM/\\delta\)\}\{\\tfrac\{1\}\{2\}\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}NMr\}\\;=\\;2C\\cdot\\frac\{\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\}\{\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}\}\\log\(2NM/\\delta\)\.\(31\)Letμmax≔max⁡\(μA,μB\)\\mu\_\{\\max\}\\coloneqq\\max\(\\mu\_\{A\},\\mu\_\{B\}\)andμmin≔min⁡\(μA,μB\)\\mu\_\{\\min\}\\coloneqq\\min\(\\mu\_\{A\},\\mu\_\{B\}\); the hypothesisζ≤μmax\\zeta\\leq\\mu\_\{\\max\}yieldsμA2\+μB2\+ζ2≤3​μmax2\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\\leq 3\\mu\_\{\\max\}^\{2\}\. UsingμA​μB=μmax​μmin\\mu\_\{A\}\\mu\_\{B\}=\\mu\_\{\\max\}\\mu\_\{\\min\},

ζ2​\(μA2\+μB2\+ζ2\)μA2​μB2≤3​ζ2​μmax2μmax2​μmin2=3​ζ2μmin2\.\\frac\{\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\}\{\\mu\_\{A\}^\{2\}\\mu\_\{B\}^\{2\}\}\\;\\leq\\;\\frac\{3\\zeta^\{2\}\\mu\_\{\\max\}^\{2\}\}\{\\mu\_\{\\max\}^\{2\}\\mu\_\{\\min\}^\{2\}\}\\;=\\;\\frac\{3\\zeta^\{2\}\}\{\\mu\_\{\\min\}^\{2\}\}\.Substituting into Eq\. \([31](https://arxiv.org/html/2605.24058#A3.E31)\) yields

‖E‖F2‖Δ​W​\(θ⋆\)‖F2≤6​C​ζ2μmin2​log⁡\(2​N​M/δ\)\.\\frac\{\\\|E\\\|\_\{F\}^\{2\}\}\{\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}^\{2\}\}\\;\\leq\\;\\frac\{6C\\zeta^\{2\}\}\{\\mu\_\{\\min\}^\{2\}\}\\log\(2NM/\\delta\)\.Taking square roots \(both sides are non\-negative\) and settingC′≔6​C\>0C^\{\\prime\}\\coloneqq\\sqrt\{6C\}\>0proves Eq\. \([10](https://arxiv.org/html/2605.24058#S4.E10)\) on the eventℰ≔ℰ1∩ℰ2\{\\mathcal\{E\}\}\\coloneqq\{\\mathcal\{E\}\}\_\{1\}\\cap\{\\mathcal\{E\}\}\_\{2\}, whose probability is at least1−δ−8/\(N​M\)1\-\\delta\-8/\(NM\)\.

*Small\-noise transfer\.*Putτ≔C′​ζ​log⁡\(2​N​M/δ\)/μmin\\tau\\coloneqq C^\{\\prime\}\\zeta\\sqrt\{\\log\(2NM/\\delta\)\}/\\mu\_\{\\min\}and assumeτ≤1/2\\tau\\leq 1/2\. Onℰ\{\\mathcal\{E\}\}, Eq\. \([10](https://arxiv.org/html/2605.24058#S4.E10)\) gives‖E‖F≤τ​‖Δ​W​\(θ⋆\)‖F\\\|E\\\|\_\{F\}\\leq\\tau\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}, and the reverse triangle inequality\|‖A​B⊤‖F−‖Δ​W​\(θ⋆\)‖F\|≤‖A​B⊤−Δ​W​\(θ⋆\)‖F=‖E‖F\\bigl\|\\\|AB^\{\\top\}\\\|\_\{F\}\-\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\\bigr\|\\leq\\\|AB^\{\\top\}\-\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}=\\\|E\\\|\_\{F\}implies

‖A​B⊤‖F≥‖Δ​W​\(θ⋆\)‖F−‖E‖F≥\(1−τ\)​‖Δ​W​\(θ⋆\)‖F≥12​‖Δ​W​\(θ⋆\)‖F\.\\\|AB^\{\\top\}\\\|\_\{F\}\\;\\geq\\;\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\-\\\|E\\\|\_\{F\}\\;\\geq\\;\(1\-\\tau\)\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\\;\\geq\\;\\tfrac\{1\}\{2\}\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\.Dividing the bound Eq\. \([10](https://arxiv.org/html/2605.24058#S4.E10)\) by a denominator at least12​‖Δ​W​\(θ⋆\)‖F\\tfrac\{1\}\{2\}\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}costs at most a factor of22, which establishes the transferred bound‖E‖F/‖A​B⊤‖F≤2​τ\\\|E\\\|\_\{F\}/\\\|AB^\{\\top\}\\\|\_\{F\}\\leq 2\\tauonℰ\{\\mathcal\{E\}\}\. ∎

The theorem is*honest*in the sense that it tracks the actual quantity controlling LoRDBA error—the residual\-to\-mean ratioζ/min⁡\(μA,μB\)\\zeta/\\min\(\\mu\_\{A\},\\mu\_\{B\}\)\. A naive “11\-bit is lossless toε\\varepsilon” claim is false for arbitrary fp16 adapters \(sign quantisation losesΘ​\(1\)\\Theta\(1\)of the Frobenius signal whenever the entries are concentrated on a sphere rather than near±μ\\pm\\mu\); the bound above degrades gracefully in the relevant regime and becomes vacuous precisely where sign quantisation should fail\. The ratio is directly observable on any trained adapter, as we verify next\.

### C\.5Sign consistency and observed\-sign reconstruction

The canonical reconstructionθ⋆\\theta^\{\\star\}uses the latent sign arraysσA,σB\\sigma^\{A\},\\sigma^\{B\}, which are not in general equal to the observed entrywise signssign⁡\(A\),sign⁡\(B\)\\operatorname\{sign\}\(A\),\\operatorname\{sign\}\(B\)\. The next lemma shows that the two coincide with high probability whenever the residual is small relative to the latent mean, so the observed\-sign LoRDBA adapter inherits the same Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)guarantees up to an explicit additional failure probability\.

###### Lemma C\.5\(Sign consistency of the canonical reconstruction\)\.

Under Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1), the latent sign arraysσA,σB\\sigma^\{A\},\\sigma^\{B\}and the observed entrywise sign patternssign⁡\(A\),sign⁡\(B\)\\operatorname\{sign\}\(A\),\\operatorname\{sign\}\(B\)agree simultaneously with probability at least

1−2​N​r​exp⁡\(−μA2/ζ2\)−2​M​r​exp⁡\(−μB2/ζ2\)\.1\\;\-\\;2Nr\\exp\\bigl\(\-\\mu\_\{A\}^\{2\}/\\zeta^\{2\}\\bigr\)\\;\-\\;2Mr\\exp\\bigl\(\-\\mu\_\{B\}^\{2\}/\\zeta^\{2\}\\bigr\)\.\(32\)Equivalently, on the event of probability \([32](https://arxiv.org/html/2605.24058#A3.E32)\),

sign⁡\(A\)=σA,sign⁡\(B\)=σB\.\\operatorname\{sign\}\(A\)\\;=\\;\\sigma^\{A\},\\qquad\\operatorname\{sign\}\(B\)\\;=\\;\\sigma^\{B\}\.\(33\)Consequently, on the same event, the observed\-sign choiceB1=sign⁡\(A\)B\_\{1\}=\\operatorname\{sign\}\(A\),B2=sign⁡\(B⊤\)B\_\{2\}=\\operatorname\{sign\}\(B^\{\\top\}\)equals the latent\-sign binary factors of the canonical reconstruction in Eq\. \([17](https://arxiv.org/html/2605.24058#A3.E17)\)\.

###### Proof\.

Fix an entry\(i,k\)∈\[N\]×\[r\]\(i,k\)\\in\[N\]\\times\[r\]ofAA\. By Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)\(i\)–\(ii\),

Ai​k=μA​σi​kA\+ξi​kA,σi​kA∈\{±1\},μA\>0,‖ξi​kA‖ψ2≤ζ\.A\_\{ik\}\\;=\\;\\mu\_\{A\}\\sigma^\{A\}\_\{ik\}\+\\xi^\{A\}\_\{ik\},\\qquad\\sigma^\{A\}\_\{ik\}\\in\\\{\\pm 1\\\},\\qquad\\mu\_\{A\}\>0,\\qquad\\\|\\xi^\{A\}\_\{ik\}\\\|\_\{\\psi\_\{2\}\}\\leq\\zeta\.We claim the implication

sign⁡\(Ai​k\)≠σi​kA⟹\|ξi​kA\|≥μA\.\\operatorname\{sign\}\(A\_\{ik\}\)\\neq\\sigma^\{A\}\_\{ik\}\\;\\Longrightarrow\\;\|\\xi^\{A\}\_\{ik\}\|\\geq\\mu\_\{A\}\.\(34\)Indeed, multiplyingAi​kA\_\{ik\}byσi​kA∈\{±1\}\\sigma^\{A\}\_\{ik\}\\in\\\{\\pm 1\\\}gives

σi​kA​Ai​k=μA\+σi​kA​ξi​kA\.\\sigma^\{A\}\_\{ik\}A\_\{ik\}\\;=\\;\\mu\_\{A\}\+\\sigma^\{A\}\_\{ik\}\\xi^\{A\}\_\{ik\}\.Under the conventionsign⁡\(0\)=\+1\\operatorname\{sign\}\(0\)=\+1, a mismatch impliesσi​kA​Ai​k≤0\\sigma^\{A\}\_\{ik\}A\_\{ik\}\\leq 0: ifσi​kA=\+1\\sigma^\{A\}\_\{ik\}=\+1then the mismatch forcesAi​k<0A\_\{ik\}<0, while ifσi​kA=−1\\sigma^\{A\}\_\{ik\}=\-1it forcesAi​k≥0A\_\{ik\}\\geq 0\. Therefore

σi​kA​Ai​k=μA\+σi​kA​ξi​kA≤0,\\sigma^\{A\}\_\{ik\}A\_\{ik\}=\\mu\_\{A\}\+\\sigma^\{A\}\_\{ik\}\\xi^\{A\}\_\{ik\}\\leq 0,which impliesσi​kA​ξi​kA≤−μA\\sigma^\{A\}\_\{ik\}\\xi^\{A\}\_\{ik\}\\leq\-\\mu\_\{A\}and hence\|ξi​kA\|≥μA\|\\xi^\{A\}\_\{ik\}\|\\geq\\mu\_\{A\}\. This proves Eq\. \([34](https://arxiv.org/html/2605.24058#A3.E34)\)\.

By Eq\. \([34](https://arxiv.org/html/2605.24058#A3.E34)\) and Lemma[B\.4](https://arxiv.org/html/2605.24058#A2.Thmtheorem4)of Appendix[B](https://arxiv.org/html/2605.24058#A2)applied toξi​kA\\xi^\{A\}\_\{ik\}withK=ζK=\\zetaandμ=μA\\mu=\\mu\_\{A\},

Pr⁡\[sign⁡\(Ai​k\)≠σi​kA\]≤Pr⁡\[\|ξi​kA\|≥μA\]≤2​exp⁡\(−μA2/ζ2\)\.\\Pr\\bigl\[\\operatorname\{sign\}\(A\_\{ik\}\)\\neq\\sigma^\{A\}\_\{ik\}\\bigr\]\\;\\leq\\;\\Pr\\bigl\[\|\\xi^\{A\}\_\{ik\}\|\\geq\\mu\_\{A\}\\bigr\]\\;\\leq\\;2\\exp\\bigl\(\-\\mu\_\{A\}^\{2\}/\\zeta^\{2\}\\bigr\)\.\(35\)The identical argument withAAreplaced byBBandμA\\mu\_\{A\}byμB\\mu\_\{B\}gives, for every\(j,k\)∈\[M\]×\[r\]\(j,k\)\\in\[M\]\\times\[r\],

Pr⁡\[sign⁡\(Bj​k\)≠σj​kB\]≤2​exp⁡\(−μB2/ζ2\)\.\\Pr\\bigl\[\\operatorname\{sign\}\(B\_\{jk\}\)\\neq\\sigma^\{B\}\_\{jk\}\\bigr\]\\;\\leq\\;2\\exp\\bigl\(\-\\mu\_\{B\}^\{2\}/\\zeta^\{2\}\\bigr\)\.\(36\)
A union bound over theN​rNrentries ofAAand theM​rMrentries ofBB, using Eqs\. \([35](https://arxiv.org/html/2605.24058#A3.E35)\)–\([36](https://arxiv.org/html/2605.24058#A3.E36)\), yields

Pr⁡\[sign⁡\(A\)≠σA​or​sign⁡\(B\)≠σB\]≤2​N​r​exp⁡\(−μA2/ζ2\)\+2​M​r​exp⁡\(−μB2/ζ2\)\.\\Pr\\bigl\[\\operatorname\{sign\}\(A\)\\neq\\sigma^\{A\}\\ \\text\{or\}\\ \\operatorname\{sign\}\(B\)\\neq\\sigma^\{B\}\\bigr\]\\;\\leq\\;2Nr\\exp\\bigl\(\-\\mu\_\{A\}^\{2\}/\\zeta^\{2\}\\bigr\)\+2Mr\\exp\\bigl\(\-\\mu\_\{B\}^\{2\}/\\zeta^\{2\}\\bigr\)\.\(37\)The complement is exactly Eq\. \([33](https://arxiv.org/html/2605.24058#A3.E33)\) and has probability at least the value displayed in Eq\. \([32](https://arxiv.org/html/2605.24058#A3.E32)\)\.

On the event Eq\. \([33](https://arxiv.org/html/2605.24058#A3.E33)\), the observed\-sign binary factors satisfy

B1=sign\(A\)=σA,B2=sign\(B⊤\)=sign\(B\)⊤=\(σB\)⊤,B\_\{1\}\\;=\\;\\operatorname\{sign\}\(A\)\\;=\\;\\sigma^\{A\},\\qquad B\_\{2\}\\;=\\;\\operatorname\{sign\}\(B^\{\\top\}\)\\;=\\;\\operatorname\{sign\}\(B\)^\{\\top\}\\;=\\;\(\\sigma^\{B\}\)^\{\\top\},becausesign\\operatorname\{sign\}is applied entrywise and is therefore preserved under transposition\. These coincide with the binary factors of the canonical reconstruction in Eq\. \([17](https://arxiv.org/html/2605.24058#A3.E17)\), completing the proof\. ∎

###### Corollary C\.6\(LoRDBA envelope\-rank monotonicity\)\.

Under the hypotheses of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1), for everyℓ≥1\\ell\\geq 1,infθ∈Θℓ‖A​B⊤−Δ​W​\(θ\)‖F\\inf\_\{\\theta\\in\\Theta\_\{\\ell\}\}\\\|AB^\{\\top\}\-\\Delta W\(\\theta\)\\\|\_\{F\}is bounded by the absolute bound in Eq\. \([27](https://arxiv.org/html/2605.24058#A3.E27)\) and, in the small\-noise regimeτ≤1/2\\tau\\leq 1/2, by a factor of two times the right\-hand side of Eq\. \([10](https://arxiv.org/html/2605.24058#S4.E10)\) relative to‖A​B⊤‖F\\\|AB^\{\\top\}\\\|\_\{F\}\.

###### Proof\.

Applyιℓ\\iota\_\{\\ell\}iteratively toθ⋆∈Θ1\\theta^\{\\star\}\\in\\Theta\_\{1\}to obtain a feasible point with the sameΔ​W\\Delta WinΘℓ\\Theta\_\{\\ell\}, then invoke Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1); the small\-noise transfer follows from‖A​B⊤‖F≥12​‖Δ​W​\(θ⋆\)‖F\\\|AB^\{\\top\}\\\|\_\{F\}\\geq\\tfrac\{1\}\{2\}\\\|\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}under the standing conditionτ≤1/2\\tau\\leq 1/2\. ∎

###### Corollary C\.7\(Observed\-sign reconstruction\)\.

Assume the hypotheses of Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)and define the observed\-sign LoRDBA adapter

θ^≔\(sign\(A\),sign\(B\)⊤,𝟏N,μAμB𝟏r,𝟏M\)\.\\widehat\{\\theta\}\\;\\coloneqq\\;\\bigl\(\\operatorname\{sign\}\(A\),\\operatorname\{sign\}\(B\)^\{\\top\},\{\\bm\{1\}\}\_\{N\},\\mu\_\{A\}\\mu\_\{B\}\{\\bm\{1\}\}\_\{r\},\{\\bm\{1\}\}\_\{M\}\\bigr\)\.\(38\)On the sign\-consistency event of Lemma[C\.5](https://arxiv.org/html/2605.24058#A3.Thmtheorem5),

Δ​W​\(θ^\)=Δ​W​\(θ⋆\)\.\\Delta W\(\\widehat\{\\theta\}\)\\;=\\;\\Delta W\(\\theta^\{\\star\}\)\.\(39\)Consequently, every bound in Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)and Corollary[C\.6](https://arxiv.org/html/2605.24058#A3.Thmtheorem6)applies toθ^\\widehat\{\\theta\}, with the failure probability of the corresponding event increased by at most

pflip≔2​N​r​exp⁡\(−μA2/ζ2\)\+2​M​r​exp⁡\(−μB2/ζ2\)\.p\_\{\\mathrm\{flip\}\}\\;\\coloneqq\\;2Nr\\exp\\bigl\(\-\\mu\_\{A\}^\{2\}/\\zeta^\{2\}\\bigr\)\+2Mr\\exp\\bigl\(\-\\mu\_\{B\}^\{2\}/\\zeta^\{2\}\\bigr\)\.\(40\)

###### Proof\.

By Lemma[C\.5](https://arxiv.org/html/2605.24058#A3.Thmtheorem5), on the event Eq\. \([33](https://arxiv.org/html/2605.24058#A3.E33)\) the binary factors ofθ^\\widehat\{\\theta\}andθ⋆\\theta^\{\\star\}coincide,

sign\(A\)=σA,sign\(B\)⊤=\(σB\)⊤\.\\operatorname\{sign\}\(A\)\\;=\\;\\sigma^\{A\},\\qquad\\operatorname\{sign\}\(B\)^\{\\top\}\\;=\\;\(\\sigma^\{B\}\)^\{\\top\}\.The scale vectors ofθ^\\widehat\{\\theta\}are the same as those ofθ⋆\\theta^\{\\star\}, namely𝟏N,μA​μB​𝟏r,𝟏M\{\\bm\{1\}\}\_\{N\},\\mu\_\{A\}\\mu\_\{B\}\{\\bm\{1\}\}\_\{r\},\{\\bm\{1\}\}\_\{M\}in Eq\. \([17](https://arxiv.org/html/2605.24058#A3.E17)\)\. Substituting both equalities into Eq\. \([4](https://arxiv.org/html/2605.24058#S3.E4)\) yieldsΔ​W​\(θ^\)=Δ​W​\(θ⋆\)\\Delta W\(\\widehat\{\\theta\}\)=\\Delta W\(\\theta^\{\\star\}\)on this event, proving Eq\. \([39](https://arxiv.org/html/2605.24058#A3.E39)\)\.

For each eventℰ′\{\\mathcal\{E\}\}^\{\\prime\}produced by Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)or Corollary[C\.6](https://arxiv.org/html/2605.24058#A3.Thmtheorem6), the corresponding inequality bounds the residual‖A​B⊤−Δ​W​\(θ⋆\)‖F\\\|AB^\{\\top\}\-\\Delta W\(\\theta^\{\\star\}\)\\\|\_\{F\}\. On the intersectionℰ′∩\{sign⁡\(A\)=σA,sign⁡\(B\)=σB\}\{\\mathcal\{E\}\}^\{\\prime\}\\cap\\\{\\operatorname\{sign\}\(A\)=\\sigma^\{A\},\\operatorname\{sign\}\(B\)=\\sigma^\{B\}\\\}, this residual equals‖A​B⊤−Δ​W​\(θ^\)‖F\\\|AB^\{\\top\}\-\\Delta W\(\\widehat\{\\theta\}\)\\\|\_\{F\}by Eq\. \([39](https://arxiv.org/html/2605.24058#A3.E39)\), so the inequality also bounds the observed\-sign residual\. The complement bound follows from

Pr⁡\[\(ℰ′\)c∪\{sign⁡\(A\)≠σA​or​sign⁡\(B\)≠σB\}\]≤Pr⁡\[\(ℰ′\)c\]\+pflip,\\Pr\\bigl\[\(\{\\mathcal\{E\}\}^\{\\prime\}\)^\{c\}\\cup\\\{\\operatorname\{sign\}\(A\)\\neq\\sigma^\{A\}\\ \\text\{or\}\\ \\operatorname\{sign\}\(B\)\\neq\\sigma^\{B\}\\\}\\bigr\]\\;\\leq\\;\\Pr\[\(\{\\mathcal\{E\}\}^\{\\prime\}\)^\{c\}\]\\;\+\\;p\_\{\\mathrm\{flip\}\},which adds at mostpflipp\_\{\\mathrm\{flip\}\}of Eq\. \([40](https://arxiv.org/html/2605.24058#A3.E40)\) to the original failure probability ofℰ′\{\\mathcal\{E\}\}^\{\\prime\}\. ∎

###### Corollary C\.8\(Operator\-norm consequence\)\.

Under the hypotheses used for the absolute bound in Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1),E≔A​B⊤−Δ​W​\(θ⋆\)E\\coloneqq AB^\{\\top\}\-\\Delta W\(\\theta^\{\\star\}\)satisfies, with probability at least1−δ1\-\\deltaover the residuals,

‖E‖op≤‖E‖F≤C​N​M​r​ζ2​\(μA2\+μB2\+ζ2\)​log⁡\(2​N​M/δ\),\\\|E\\\|\_\{\\mathrm\{op\}\}\\;\\leq\\;\\\|E\\\|\_\{F\}\\;\\leq\\;\\sqrt\{CNMr\\zeta^\{2\}\(\\mu\_\{A\}^\{2\}\+\\mu\_\{B\}^\{2\}\+\\zeta^\{2\}\)\\log\(2NM/\\delta\)\},\(41\)whereC\>0C\>0is the universal constant of Eq\. \([27](https://arxiv.org/html/2605.24058#A3.E27)\)\.

###### Proof\.

For everyX∈ℝN×MX\\in\{\\mathbb\{R\}\}^\{N\\times M\}, the singular value decomposition gives

‖X‖op2=σ1​\(X\)2≤∑k=1min⁡\(N,M\)σk​\(X\)2=‖X‖F2,\\\|X\\\|\_\{\\mathrm\{op\}\}^\{2\}\\;=\\;\\sigma\_\{1\}\(X\)^\{2\}\\;\\leq\\;\\sum\_\{k=1\}^\{\\min\(N,M\)\}\\sigma\_\{k\}\(X\)^\{2\}\\;=\\;\\\|X\\\|\_\{F\}^\{2\},whereσ1​\(X\)≥σ2​\(X\)≥⋯≥0\\sigma\_\{1\}\(X\)\\geq\\sigma\_\{2\}\(X\)\\geq\\cdots\\geq 0are the singular values ofXX\. Hence‖E‖op≤‖E‖F\\\|E\\\|\_\{\\mathrm\{op\}\}\\leq\\\|E\\\|\_\{F\}deterministically\. The second inequality of Eq\. \([41](https://arxiv.org/html/2605.24058#A3.E41)\) is exactly the square root of Eq\. \([27](https://arxiv.org/html/2605.24058#A3.E27)\), which holds with probability at least1−δ1\-\\delta\. ∎

### C\.6Gauge fixing and empirical identifiability of Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)

#### LoRA factor gauge\.

The productA​B⊤AB^\{\\top\}is invariant under the per\-column positive rescaling

A:k↦dk​A:k,B:k↦dk−1​B:k,dk\>0,k∈\[r0\],A\_\{:k\}\\;\\mapsto\\;d\_\{k\}A\_\{:k\},\\qquad B\_\{:k\}\\;\\mapsto\\;d\_\{k\}^\{\-1\}B\_\{:k\},\\qquad d\_\{k\}\>0,~k\\in\[r\_\{0\}\],\(42\)since\(dk​A:k\)​\(dk−1​B:k\)⊤=A:k​B:k⊤\(d\_\{k\}A\_\{:k\}\)\(d\_\{k\}^\{\-1\}B\_\{:k\}\)^\{\\top\}=A\_\{:k\}B\_\{:k\}^\{\\top\}\. The natural plug\-in proxies for\(μA,μB,ζ\)\(\\mu\_\{A\},\\mu\_\{B\},\\zeta\)depend on the particular gauge\. We therefore compute every diagnostic in the following*column\-balancing gauge*: for every nonzero column pair\(A:k,B:k\)\(A\_\{:k\},B\_\{:k\}\), set

dk=\(‖B:k‖2/‖A:k‖2\)1/2,A:k←dk​A:k,B:k←dk−1​B:k;d\_\{k\}\\;=\\;\\bigl\(\\\|B\_\{:k\}\\\|\_\{2\}/\\\|A\_\{:k\}\\\|\_\{2\}\\bigr\)^\{1/2\},\\qquad A\_\{:k\}\\leftarrow d\_\{k\}A\_\{:k\},\\qquad B\_\{:k\}\\leftarrow d\_\{k\}^\{\-1\}B\_\{:k\};\(43\)zero\-contribution column pairs \(if any\) are left unchanged and excluded from the diagnostic\. By construction this transformation preservesA​B⊤AB^\{\\top\}and the entrywise sign patterns of both columns \(sincedk\>0d\_\{k\}\>0\), and equalises the column norms‖A:k‖2=‖B:k‖2=‖A:kold‖2​‖B:kold‖2\\\|A\_\{:k\}\\\|\_\{2\}=\\\|B\_\{:k\}\\\|\_\{2\}=\\sqrt\{\\\|A\_\{:k\}^\{\\mathrm\{old\}\}\\\|\_\{2\}\\\|B\_\{:k\}^\{\\mathrm\{old\}\}\\\|\_\{2\}\}for every nonzero pair\. Thus the LoRA update and the canonical LoRDBA sign factors are unchanged, while the plug\-in magnitude diagnostics are computed in a fixed, reproducible gauge\. Eq\. \([43](https://arxiv.org/html/2605.24058#A3.E43)\) is the gauge used by every empirical statement of the main paper and by every diagnostic in this appendix\.

#### Plug\-in estimators\.

For any trained fp16 LoRA factorsA,BA,Bin the gauge of Eq\. \([43](https://arxiv.org/html/2605.24058#A3.E43)\), the quantities in Assumption[C\.1](https://arxiv.org/html/2605.24058#A3.Thmtheorem1)are empirically identifiable by the plug\-in estimators

μ^A≔1N​r0​∑i,k\|Ai​k\|,μ^B≔1M​r0​∑j,k\|Bj​k\|,ζ^A≔\(1N​r0​∑i,k\(\|Ai​k\|−μ^A\)2\)1/2,\\widehat\{\\mu\}\_\{A\}\\coloneqq\\tfrac\{1\}\{Nr\_\{0\}\}\\sum\_\{i,k\}\|A\_\{ik\}\|,~~\\widehat\{\\mu\}\_\{B\}\\coloneqq\\tfrac\{1\}\{Mr\_\{0\}\}\\sum\_\{j,k\}\|B\_\{jk\}\|,~~\\widehat\{\\zeta\}\_\{A\}\\coloneqq\\bigl\(\\tfrac\{1\}\{Nr\_\{0\}\}\\sum\_\{i,k\}\(\|A\_\{ik\}\|\-\\widehat\{\\mu\}\_\{A\}\)^\{2\}\\bigr\)^\{1/2\},ζ^B\\widehat\{\\zeta\}\_\{B\}analogous, andζ^≔max⁡\(ζ^A,ζ^B\)\\widehat\{\\zeta\}\\coloneqq\\max\(\\widehat\{\\zeta\}\_\{A\},\\widehat\{\\zeta\}\_\{B\}\)\. The reported residual\-to\-mean ratio isζ^/min⁡\(μ^A,μ^B\)\\widehat\{\\zeta\}/\\min\(\\widehat\{\\mu\}\_\{A\},\\widehat\{\\mu\}\_\{B\}\)\. Table[5](https://arxiv.org/html/2605.24058#A3.T5)reports this quantity on the LoRA adapters used in our experiments\. The observed0\.180\.18–0\.270\.27window justifies the small\-noise regime and, together with Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1), predicts the≈20%\\approx 20\\%relative reconstruction error we measure in Table[10](https://arxiv.org/html/2605.24058#A8.T10)\.

Table 5:Empirical residual\-to\-mean ratio of fp16 LoRA factors \(adapter rankr0=64r\_\{0\}=64, averaged over all attention and MLP projections\), computed in the column\-balancing gauge of Eq\. \([43](https://arxiv.org/html/2605.24058#A3.E43)\)\.

## Appendix DOptional Refinement: PTQ\-LoRDBA via Consensus ADMM

This appendix records the optional, training\-data\-free route that refines a pre\-trained fp16 LoRA adapter into a LoRDBA adapter without running the end\-to\-end smooth\-sign trainer of Section[3\.2](https://arxiv.org/html/2605.24058#S3.SS2)\. It is included for completeness: the LoRDBA adapter of Definition[3\.1](https://arxiv.org/html/2605.24058#S3.Thmtheorem1)is unchanged, the inference kernel of Section[3\.3](https://arxiv.org/html/2605.24058#S3.SS3)is reused unchanged, and empirical comparisons in Appendix[H](https://arxiv.org/html/2605.24058#A8)–[L](https://arxiv.org/html/2605.24058#A12)confirm that the end\-to\-end trainer is the preferred route under the adapter\-mode conditions \(A1\)–\(A4\) of Section[2](https://arxiv.org/html/2605.24058#S2)\. PTQ becomes attractive only when an fp16 LoRA is already available and the device cannot spare a training pass\.

#### Setup\.

Let an fp16 LoRA adapterΔ​W⋆=A⋆​\(B⋆\)⊤\\Delta W^\{\\star\}=A^\{\\star\}\(B^\{\\star\}\)^\{\\top\}of rankr0r\_\{0\}be derived using any standard fine\-tuning recipe, and fix a target carrier rankR∈ℕR\\in\\mathbb\{N\}\. Theℓ=1\\ell=1PTQ\-LoRDBA problem is the Frobenius least\-squares fitting problem

minθ=\(B1,B2,𝜶,𝜷,𝜸\)⁡12​‖Δ​W⋆−Δ​W​\(θ\)‖F2​s\.t\.​B1∈\{±1\}N×R,B2∈\{±1\}R×M\.\\min\_\{\\theta=\(B\_\{1\},B\_\{2\},\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)\}\\frac\{1\}\{2\}\\bigl\\\|\\Delta W^\{\\star\}\-\\Delta W\(\\theta\)\\bigr\\\|\_\{F\}^\{2\}~\\text\{s\.t\.\}~B\_\{1\}\\in\\\{\\pm 1\\\}^\{N\\times R\},B\_\{2\}\\in\\\{\\pm 1\\\}^\{R\\times M\}\.\(44\)The generalℓ≥2\\ell\\geq 2case replaces the single scale triple by a sum ofℓ\\ellscale triples; all updates below generalise block\-coordinate\-wise by viewing\(𝜶,𝜷,𝜸\)\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)as stacked matrices \(Algorithm[3](https://arxiv.org/html/2605.24058#alg3)\)\. Eq\. \([44](https://arxiv.org/html/2605.24058#A4.E44)\) is non\-convex, non\-smooth, and combinatorial in\(B1,B2\)\(B\_\{1\},B\_\{2\}\)but decouples along the three continuous variables for fixed binaries and along each binary for fixed other variables\.

#### Consensus ADMM\.

To exploit this separability while respecting the discrete constraint, we introduce continuous relaxationsU1∈ℝN×RU\_\{1\}\\in\{\\mathbb\{R\}\}^\{N\\times R\},U2∈ℝR×MU\_\{2\}\\in\{\\mathbb\{R\}\}^\{R\\times M\}of\(B1,B2\)\(B\_\{1\},B\_\{2\}\)together with binary copiesM1,M2M\_\{1\},M\_\{2\}, and minimise

f​\(U;𝜶,𝜷,𝜸\)\+I\{±1\}​\(M1\)\+I\{±1\}​\(M2\),s\.t\.​Uk−Mk=0,k∈\{1,2\},f\(U;\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)\+I\_\{\\\{\\pm 1\\\}\}\(M\_\{1\}\)\+I\_\{\\\{\\pm 1\\\}\}\(M\_\{2\}\),~\\text\{s\.t\.\}~U\_\{k\}\-M\_\{k\}=0,k\\in\\\{1,2\\\},\(45\)withf​\(U;𝜶,𝜷,𝜸\)=12​‖Δ​W⋆−diag⁡\(𝜶\)​U1​diag⁡\(𝜷\)​U2​diag⁡\(𝜸\)‖F2f\(U;\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)=\\frac\{1\}\{2\}\\\|\\Delta W^\{\\star\}\-\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)U\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)U\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\\|\_\{F\}^\{2\}andI\{±1\}I\_\{\\\{\\pm 1\\\}\}as the indicator of the binary set\. Using per\-block normalisersn1=N​Rn\_\{1\}=NR,n2=R​Mn\_\{2\}=RM, scaled penaltiesρ~k=ρ/nk\\widetilde\{\\rho\}\_\{k\}=\\rho/n\_\{k\}, and scaled dualsY1,Y2Y\_\{1\},Y\_\{2\}, the \(Gauss–Seidel\) sweep over the consensus\-scaled augmented Lagrangian\[Boyd et al\.,[2011](https://arxiv.org/html/2605.24058#bib.bib6), Themelis and Patrinos,[2020](https://arxiv.org/html/2605.24058#bib.bib33)\]produces the following ordered updates, whereU^\(t,1\)≔\(U1,U2\(t\)\)\\widehat\{U\}^\{\(t,1\)\}\\coloneqq\(U\_\{1\},U\_\{2\}^\{\(t\)\}\)andU^\(t,2\)≔\(U1\(t\+1\),U2\)\\widehat\{U\}^\{\(t,2\)\}\\coloneqq\(U\_\{1\}^\{\(t\+1\)\},U\_\{2\}\)make the within\-sweep dependency explicit:

U1\(t\+1\)\\displaystyle U\_\{1\}^\{\(t\+1\)\}=arg​minU1⁡f​\(U1,U2\(t\);𝜶\(t\),𝜷\(t\),𝜸\(t\)\)\+ρ~12​‖U1−M1\(t\)\+Y1\(t\)‖F2,\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{U\_\{1\}\}f\\bigl\(U\_\{1\},U\_\{2\}^\{\(t\)\};\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\\bigr\)\+\\frac\{\\widetilde\{\\rho\}\_\{1\}\}\{2\}\\\|U\_\{1\}\-M\_\{1\}^\{\(t\)\}\+Y\_\{1\}^\{\(t\)\}\\\|\_\{F\}^\{2\},\(46\)U2\(t\+1\)\\displaystyle U\_\{2\}^\{\(t\+1\)\}=arg​minU2⁡f​\(U1\(t\+1\),U2;𝜶\(t\),𝜷\(t\),𝜸\(t\)\)\+ρ~22​‖U2−M2\(t\)\+Y2\(t\)‖F2,\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{U\_\{2\}\}f\\bigl\(U\_\{1\}^\{\(t\+1\)\},U\_\{2\};\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\\bigr\)\+\\frac\{\\widetilde\{\\rho\}\_\{2\}\}\{2\}\\\|U\_\{2\}\-M\_\{2\}^\{\(t\)\}\+Y\_\{2\}^\{\(t\)\}\\\|\_\{F\}^\{2\},\(47\)\(𝜶,𝜷,𝜸\)\(t\+1\)\\displaystyle\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)^\{\(t\+1\)\}←one block\-coordinate sweep of​arg​min𝜶,𝜷,𝜸⁡f​\(U\(t\+1\);𝜶,𝜷,𝜸\),\\displaystyle\\leftarrow\\text\{one block\-coordinate sweep of \}\\operatorname\*\{arg\\,min\}\_\{\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\}f\\bigl\(U^\{\(t\+1\)\};\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\\bigr\),\(48\)Mk\(t\+1\)\\displaystyle M\_\{k\}^\{\(t\+1\)\}=sign⁡\(Uk\(t\+1\)\+Yk\(t\)\),Yk\(t\+1\)=Yk\(t\)\+Uk\(t\+1\)−Mk\(t\+1\),k∈\{1,2\}\.\\displaystyle=\\operatorname\{sign\}\\bigl\(U\_\{k\}^\{\(t\+1\)\}\+Y\_\{k\}^\{\(t\)\}\\bigr\),~Y\_\{k\}^\{\(t\+1\)\}=Y\_\{k\}^\{\(t\)\}\+U\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\+1\)\},~k\\in\\\{1,2\\\}\.\(49\)We collectively refer to Eqs\. \([46](https://arxiv.org/html/2605.24058#A4.E46)\)–\([47](https://arxiv.org/html/2605.24058#A4.E47)\) as the*UU\-step*\. EachUkU\_\{k\}\-subproblem is a strongly convex Tikhonov least\-squares problem inUkU\_\{k\}\(its Hessian is the sum of a positive\-semidefinite normal\-equation Gram matrix andρ~k​I\\widetilde\{\\rho\}\_\{k\}I\), and hence admits a unique minimiser regardless of the conditioning of the cross\-covariance\. Eq\. \([48](https://arxiv.org/html/2605.24058#A4.E48)\) is a block\-coordinate sweep in the three scale variables: with the other two fixed, the subproblem in𝜶\{\\bm\{\\alpha\}\}decouples intoNNscalar least\-squares \(one per coordinate\), the subproblem in𝜸\{\\bm\{\\gamma\}\}intoMMscalar least\-squares, and the subproblem in𝜷\{\\bm\{\\beta\}\}is anRR\-dimensional positive semidefinite least\-squares system whose normal matrix is the Hadamard product of two Gram matrices and is hence positive semidefinite by the Schur product theorem; the exact normal equations and coordinate\-wise minimisers are recorded in Appendix[I\.1](https://arxiv.org/html/2605.24058#A9.SS1)\. Eq\. \([49](https://arxiv.org/html/2605.24058#A4.E49)\) is the sign\-projection step modulated by the dualYkY\_\{k\}, which biases the projection by the accumulated discretisation residual\.

We warm\-start the iteration from the rank\-RRthin SVD ofΔ​W⋆\\Delta W^\{\\star\}exactly as in Eq\. \([7](https://arxiv.org/html/2605.24058#S3.E7)\) of the main text, with\(M\(0\),Y\(0\)\)=\(U\(0\),0\)\(M^\{\(0\)\},Y^\{\(0\)\}\)=\(U^\{\(0\)\},0\)\. A Boyd\-style residual\-balancing schedule onρ\\rho, with the default\(τ,μ\)=\(2,10\)\(\\tau,\\mu\)=\(2,10\), is truncated attmaxρ=K/2t\_\{\\max\}^\{\\rho\}=K/2, ensuring the second half of the run is a fixed\-penalty tail, exactly the regime addressed by the rigorous stabilisation theorem of Appendix[E](https://arxiv.org/html/2605.24058#A5)\.

#### Finite stabilisation on a fixed\-penalty tail\.

Define the sign\-test matricesZk\(t\)≔Uk\(t\+1\)\+Yk\(t\)Z\_\{k\}^\{\(t\)\}\\coloneqq U\_\{k\}^\{\(t\+1\)\}\+Y\_\{k\}^\{\(t\)\}so that the binary update is preciselyMk\(t\+1\)=sign⁡\(Zk\(t\)\)M\_\{k\}^\{\(t\+1\)\}=\\operatorname\{sign\}\(Z\_\{k\}^\{\(t\)\}\)\. Once a fixed\-penalty tail has converged to a single limit and the limit remains at a positive distance from the sign boundary0, the signs ofZk\(t\)Z\_\{k\}^\{\(t\)\}can no longer change, and the binary iterate freezes\. The precise penalty\-agnostic statement is Theorem[E\.3](https://arxiv.org/html/2605.24058#A5.Thmtheorem3)in Appendix[E](https://arxiv.org/html/2605.24058#A5), and its convergence hypothesis is delivered in turn by Theorem[E\.6](https://arxiv.org/html/2605.24058#A5.Thmtheorem6)of the same appendix, which combines a sufficient\-decrease Lyapunov inequality with the Kurdyka–Łojasiewicz inequality on the semi\-algebraic Lyapunov function to guarantee full convergence of the iterate sequence to a single limit\. Empirically, the freeze occurs withinK⋆≲50K^\{\\star\}\\lesssim 50sweeps for every model and rank tested\.

## Appendix EConvergence of PTQ\-LoRDBA

Throughout this appendix we adopt the shorthands

n1≔N​R,n2≔R​M,ρ~k≔ρ/nk,k∈\{1,2\},n\_\{1\}\\;\\coloneqq\\;NR,\\qquad n\_\{2\}\\;\\coloneqq\\;RM,\\qquad\\widetilde\{\\rho\}\_\{k\}\\;\\coloneqq\\;\\rho/n\_\{k\},\\qquad k\\in\\\{1,2\\\},\(50\)and recall the PTQ\-LoRDBA objective

f​\(U;𝜶,𝜷,𝜸\)≔12​‖Δ​W⋆−diag⁡\(𝜶\)​U1​diag⁡\(𝜷\)​U2​diag⁡\(𝜸\)‖F2\.f\(U;\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)\\;\\coloneqq\\;\\tfrac\{1\}\{2\}\\bigl\\\|\\Delta W^\{\\star\}\-\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)U\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)U\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\bigr\\\|\_\{F\}^\{2\}\.\(51\)For fixed scale vectors and one of the twoUU\-blocks,ffis a convex quadratic because the map from that block to the reconstructed matrix is linear\. The binary projection step, however, is discontinuous at zero sign\-test entries\. Consequently, boundedness and Lipschitz smoothness alone do not imply finite binary stabilisation\. We therefore state the convergence facts in the exact form used by the paper: a penalty\-agnostic sign\-margin theorem, and a standard KL\-tail theorem whose sufficient\-decrease and relative\-error hypotheses must hold on the fixed\-penalty tail\.

###### Assumption E\.1\(Bounded fixed\-penalty tail\)\.

There exists an iterationTρT\_\{\\rho\}after which the penalty is fixed,ρ\(t\)≡ρ\>0\\rho^\{\(t\)\}\\equiv\\rho\>0\. On this tail, all iterates\(U\(t\),M\(t\),Y\(t\),𝛂\(t\),𝛃\(t\),𝛄\(t\)\)\(U^\{\(t\)\},M^\{\(t\)\},Y^\{\(t\)\},\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\)belong to a compact set𝒞\\mathcal\{C\}\. TheU1U\_\{1\}\- andU2U\_\{2\}\-subproblems in Eqs\. \([46](https://arxiv.org/html/2605.24058#A4.E46)\)–\([47](https://arxiv.org/html/2605.24058#A4.E47)\) are solved exactly, the scale update is well defined, and the partial gradients∇U1f\\nabla\_\{U\_\{1\}\}fand∇U2f\\nabla\_\{U\_\{2\}\}fare Lipschitz on𝒞\\mathcal\{C\}with respect to all their arguments\.

This assumption is the bounded\-tail regularity used in the convergence statements below\. It is not a consequence of discreteness alone; in an implementation it is enforced by the usual safeguards on the continuous least\-squares and scale variables\.

###### Lemma E\.2\(Gradient–dual identity on a fixed\-penalty tail\)\.

Fort≥Tρt\\geq T\_\{\\rho\}define the Gauss–Seidel arguments

U^\(t,1\)≔\(U1\(t\+1\),U2\(t\)\),U^\(t,2\)≔\(U1\(t\+1\),U2\(t\+1\)\)\.\\widehat\{U\}^\{\(t,1\)\}\\;\\coloneqq\\;\(U\_\{1\}^\{\(t\+1\)\},U\_\{2\}^\{\(t\)\}\),\\qquad\\widehat\{U\}^\{\(t,2\)\}\\;\\coloneqq\\;\(U\_\{1\}^\{\(t\+1\)\},U\_\{2\}^\{\(t\+1\)\}\)\.\(52\)Then, for everyk∈\{1,2\}k\\in\\\{1,2\\\},

ρ​Yk\(t\+1\)\+nk​∇Ukf​\(U^\(t,k\);𝜶\(t\),𝜷\(t\),𝜸\(t\)\)\+ρ​\(Mk\(t\+1\)−Mk\(t\)\)=0\.\\rho Y\_\{k\}^\{\(t\+1\)\}\\;\+\\;n\_\{k\}\\nabla\_\{U\_\{k\}\}f\\bigl\(\\widehat\{U\}^\{\(t,k\)\};\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\\bigr\)\\;\+\\;\\rho\\bigl\(M\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}\\bigr\)\\;=\\;0\.\(53\)

###### Proof\.

The first\-order condition of the exactUkU\_\{k\}\-subproblem is

∇Ukf​\(U^\(t,k\);𝜶\(t\),𝜷\(t\),𝜸\(t\)\)\+ρ~k​\(Uk\(t\+1\)−Mk\(t\)\+Yk\(t\)\)=0\.\\nabla\_\{U\_\{k\}\}f\\bigl\(\\widehat\{U\}^\{\(t,k\)\};\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\\bigr\)\+\\widetilde\{\\rho\}\_\{k\}\\bigl\(U\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}\+Y\_\{k\}^\{\(t\)\}\\bigr\)=0\.The dual update gives

Uk\(t\+1\)−Mk\(t\)\+Yk\(t\)=Yk\(t\+1\)\+\(Mk\(t\+1\)−Mk\(t\)\)\.U\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}\+Y\_\{k\}^\{\(t\)\}=Y\_\{k\}^\{\(t\+1\)\}\+\\bigl\(M\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}\\bigr\)\.Multiplying bynkn\_\{k\}and usingnk​ρ~k=ρn\_\{k\}\\widetilde\{\\rho\}\_\{k\}=\\rhoyields Eq\. \([53](https://arxiv.org/html/2605.24058#A5.E53)\)\. ∎

### E\.1Finite stabilisation under a sign margin

The next theorem is the stabilisation statement invoked in the main text\. It assumes convergence of the fixed\-penalty tail and proves that a positive limiting sign margin forces the binary iterate to freeze\.

###### Theorem E\.3\(Finite stabilisation on a fixed\-penalty tail\)\.

Fix the penalty parametersρ~1,ρ~2\>0\\widetilde\{\\rho\}\_\{1\},\\widetilde\{\\rho\}\_\{2\}\>0on the tailt≥Tρt\\geq T\_\{\\rho\}, and assume that the resulting PTQ\-LoRDBA iterates\(Uk\(t\),Mk\(t\),Yk\(t\),𝛂\(t\),𝛃\(t\),𝛄\(t\)\)\(U\_\{k\}^\{\(t\)\},M\_\{k\}^\{\(t\)\},Y\_\{k\}^\{\(t\)\},\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\)converge to a limit\(Uk∞,Mk∞,Yk∞,𝛂∞,𝛃∞,𝛄∞\)\(U\_\{k\}^\{\\infty\},M\_\{k\}^\{\\infty\},Y\_\{k\}^\{\\infty\},\{\\bm\{\\alpha\}\}^\{\\infty\},\{\\bm\{\\beta\}\}^\{\\infty\},\{\\bm\{\\gamma\}\}^\{\\infty\}\)for everyk∈\{1,2\}k\\in\\\{1,2\\\}\. Define

Zk\(t\)≔Uk\(t\+1\)\+Yk\(t\),Zk∞≔Uk∞\+Yk∞,k∈\{1,2\},Z\_\{k\}^\{\(t\)\}\\;\\coloneqq\\;U\_\{k\}^\{\(t\+1\)\}\+Y\_\{k\}^\{\(t\)\},\\qquad Z\_\{k\}^\{\\infty\}\\;\\coloneqq\\;U\_\{k\}^\{\\infty\}\+Y\_\{k\}^\{\\infty\},\\qquad k\\in\\\{1,2\\\},\(54\)and suppose the strict sign\-margin condition

η≔mink∈\{1,2\}⁡mina,b⁡\|\(Zk∞\)a​b\|\>0\.\\eta\\;\\coloneqq\\;\\min\_\{k\\in\\\{1,2\\\}\}\\min\_\{a,b\}\\bigl\|\(Z\_\{k\}^\{\\infty\}\)\_\{ab\}\\bigr\|\\;\>\\;0\.\(55\)Then:

1. \(i\)There exists a finite iterationT⋆≥TρT\_\{\\star\}\\geq T\_\{\\rho\}such that, for everyt≥T⋆t\\geq T\_\{\\star\}andk∈\{1,2\}k\\in\\\{1,2\\\}, Mk\(t\+1\)=sign⁡\(Zk∞\)≕Mk∞\.M\_\{k\}^\{\(t\+1\)\}=\\operatorname\{sign\}\(Z\_\{k\}^\{\\infty\}\)\\eqqcolon M\_\{k\}^\{\\infty\}\.\(56\)
2. \(ii\)If theUkU\_\{k\}\-subproblems are solved exactly andffis continuously differentiable inUU, the limit satisfies Uk∞=Mk∞,Mk∞=sign⁡\(Mk∞\+Yk∞\),ρ​Yk∞\+nk​∇Ukf​\(U∞;𝜶∞,𝜷∞,𝜸∞\)=0\.U\_\{k\}^\{\\infty\}=M\_\{k\}^\{\\infty\},\\qquad M\_\{k\}^\{\\infty\}=\\operatorname\{sign\}\(M\_\{k\}^\{\\infty\}\+Y\_\{k\}^\{\\infty\}\),\\qquad\\rho Y\_\{k\}^\{\\infty\}\+n\_\{k\}\\nabla\_\{U\_\{k\}\}f\(U^\{\\infty\};\{\\bm\{\\alpha\}\}^\{\\infty\},\{\\bm\{\\beta\}\}^\{\\infty\},\{\\bm\{\\gamma\}\}^\{\\infty\}\)=0\.\(57\)
3. \(iii\)Equivalently, Mk∞=sign⁡\(Mk∞−1ρ~k​∇Ukf​\(U∞;𝜶∞,𝜷∞,𝜸∞\)\)\.M\_\{k\}^\{\\infty\}=\\operatorname\{sign\}\\Bigl\(M\_\{k\}^\{\\infty\}\-\\tfrac\{1\}\{\\widetilde\{\\rho\}\_\{k\}\}\\nabla\_\{U\_\{k\}\}f\(U^\{\\infty\};\{\\bm\{\\alpha\}\}^\{\\infty\},\{\\bm\{\\beta\}\}^\{\\infty\},\{\\bm\{\\gamma\}\}^\{\\infty\}\)\\Bigr\)\.\(58\)

###### Proof\.

SinceUk\(t\+1\)→Uk∞U\_\{k\}^\{\(t\+1\)\}\\to U\_\{k\}^\{\\infty\}andYk\(t\)→Yk∞Y\_\{k\}^\{\(t\)\}\\to Y\_\{k\}^\{\\infty\}, we haveZk\(t\)→Zk∞Z\_\{k\}^\{\(t\)\}\\to Z\_\{k\}^\{\\infty\}entrywise\. ChooseT⋆T\_\{\\star\}so that‖Zk\(t\)−Zk∞‖max<η/2\\\|Z\_\{k\}^\{\(t\)\}\-Z\_\{k\}^\{\\infty\}\\\|\_\{\\max\}<\\eta/2for bothkkand allt≥T⋆t\\geq T\_\{\\star\}\. Then every entry ofZk\(t\)Z\_\{k\}^\{\(t\)\}has the same sign as the corresponding entry ofZk∞Z\_\{k\}^\{\\infty\}, proving Eq\. \([56](https://arxiv.org/html/2605.24058#A5.E56)\) fromMk\(t\+1\)=sign⁡\(Zk\(t\)\)M\_\{k\}^\{\(t\+1\)\}=\\operatorname\{sign\}\(Z\_\{k\}^\{\(t\)\}\)\.

Fort≥T⋆t\\geq T\_\{\\star\}the dual update becomesYk\(t\+1\)−Yk\(t\)=Uk\(t\+1\)−Mk∞Y\_\{k\}^\{\(t\+1\)\}\-Y\_\{k\}^\{\(t\)\}=U\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\\infty\}\. Passing to the limit givesUk∞=Mk∞U\_\{k\}^\{\\infty\}=M\_\{k\}^\{\\infty\}\. Substituting this identity intoMk∞=sign⁡\(Zk∞\)M\_\{k\}^\{\\infty\}=\\operatorname\{sign\}\(Z\_\{k\}^\{\\infty\}\)gives the second relation in Eq\. \([57](https://arxiv.org/html/2605.24058#A5.E57)\)\. Finally, fort≥T⋆\+1t\\geq T\_\{\\star\}\+1, the termMk\(t\+1\)−Mk\(t\)M\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}in Eq\. \([53](https://arxiv.org/html/2605.24058#A5.E53)\) is zero; passing to the limit in Eq\. \([53](https://arxiv.org/html/2605.24058#A5.E53)\) yields the gradient–dual stationarity relation\. Solving it forYk∞Y\_\{k\}^\{\\infty\}and usingnk/ρ=1/ρ~kn\_\{k\}/\\rho=1/\\widetilde\{\\rho\}\_\{k\}gives Eq\. \([58](https://arxiv.org/html/2605.24058#A5.E58)\)\. ∎

### E\.2KL convergence of a regular fixed\-penalty tail

The following theorem is the rigorous convergence statement for the ADMM tail\. It is deliberately conditional: the nonconvex binary projection does not yield an unconditional descent estimate without an additional margin or regularisation condition\. The assumptions below are the standard sufficient\-decrease and relative\-error hypotheses used in KL analyses of nonconvex ADMM and block\-coordinate methods\.

Let

𝒳\(t\)≔\(U\(t\),M\(t\),Y\(t\),𝜶\(t\),𝜷\(t\),𝜸\(t\),M\(t−1\)\),t≥Tρ\+1,\\mathcal\{X\}^\{\(t\)\}\\coloneqq\\bigl\(U^\{\(t\)\},M^\{\(t\)\},Y^\{\(t\)\},\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\},M^\{\(t\-1\)\}\\bigr\),\\qquad t\\geq T\_\{\\rho\}\+1,\(59\)where the last component records the one\-step memory needed by the usual ADMM Lyapunov functions\.

###### Assumption E\.5\(KL\-regular ADMM tail\)\.

On the fixed\-penalty tail of Assumption[E\.1](https://arxiv.org/html/2605.24058#A5.Thmtheorem1), there exist a proper lower\-semicontinuous semi\-algebraic Lyapunov functionΦ\\Phion the state variable𝒳\\mathcal\{X\}, constantsa,b\>0a,b\>0, and an iterationT≥Tρ\+1T\\geq T\_\{\\rho\}\+1such that, for everyt≥Tt\\geq T,

Φ​\(𝒳\(t\)\)−Φ​\(𝒳\(t\+1\)\)\\displaystyle\\Phi\(\\mathcal\{X\}^\{\(t\)\}\)\-\\Phi\(\\mathcal\{X\}^\{\(t\+1\)\}\)≥a​‖𝒳\(t\+1\)−𝒳\(t\)‖2,\\displaystyle\\geq a\\\|\\mathcal\{X\}^\{\(t\+1\)\}\-\\mathcal\{X\}^\{\(t\)\}\\\|^\{2\},\(60\)dist⁡\(0,∂Φ​\(𝒳\(t\+1\)\)\)\\displaystyle\\operatorname\{dist\}\\bigl\(0,\\partial\\Phi\(\\mathcal\{X\}^\{\(t\+1\)\}\)\\bigr\)≤b​‖𝒳\(t\+1\)−𝒳\(t\)‖\.\\displaystyle\\leq b\\\|\\mathcal\{X\}^\{\(t\+1\)\}\-\\mathcal\{X\}^\{\(t\)\}\\\|\.\(61\)The sequence\{Φ​\(𝒳\(t\)\)\}t≥T\\\{\\Phi\(\\mathcal\{X\}^\{\(t\)\}\)\\\}\_\{t\\geq T\}is bounded below\.

The semi\-algebraicity requirement is natural here: after introducing the finite binary indicators, all remaining terms in the augmented Lagrangian are polynomial functions of the continuous variables\. The substantive algorithm\-dependent requirements are the two inequalities \([60](https://arxiv.org/html/2605.24058#A5.E60)\)–\([61](https://arxiv.org/html/2605.24058#A5.E61)\); they are the precise conditions under which the KL convergence conclusion is valid\.

###### Theorem E\.6\(KL convergence of a regular fixed\-penalty tail\)\.

Suppose Assumptions[E\.1](https://arxiv.org/html/2605.24058#A5.Thmtheorem1)and[E\.5](https://arxiv.org/html/2605.24058#A5.Thmtheorem5)hold\. Then the PTQ\-LoRDBA fixed\-penalty tail has finite length and converges to a single limit

\(U\(t\),M\(t\),Y\(t\),𝜶\(t\),𝜷\(t\),𝜸\(t\)\)⟶\(U∞,M⋆,Y∞,𝜶∞,𝜷∞,𝜸∞\)\.\\bigl\(U^\{\(t\)\},M^\{\(t\)\},Y^\{\(t\)\},\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\\bigr\)\\longrightarrow\\bigl\(U^\{\\infty\},M^\{\\star\},Y^\{\\infty\},\{\\bm\{\\alpha\}\}^\{\\infty\},\{\\bm\{\\beta\}\}^\{\\infty\},\{\\bm\{\\gamma\}\}^\{\\infty\}\\bigr\)\.\(62\)Moreover:

1. \(i\)The increments of all tail variables are summable, hence square\-summable\.
2. \(ii\)The binary iterate terminates in finitely many steps: there is a finiteK⋆K^\{\\star\}such thatM\(t\)=M⋆M^\{\(t\)\}=M^\{\\star\}for everyt≥K⋆t\\geq K^\{\\star\}\.
3. \(iii\)The limit satisfies, for eachk∈\{1,2\}k\\in\\\{1,2\\\}, Uk∞=Mk⋆,Mk⋆∈arg​minm∈\{±1\}nk⁡‖m−\(Uk∞\+Yk∞\)‖F2,ρ​Yk∞\+nk​∇Ukf​\(U∞;𝜶∞,𝜷∞,𝜸∞\)=0\.U\_\{k\}^\{\\infty\}=M\_\{k\}^\{\\star\},\\qquad M\_\{k\}^\{\\star\}\\in\\operatorname\*\{arg\\,min\}\_\{m\\in\\\{\\pm 1\\\}^\{n\_\{k\}\}\}\\\|m\-\(U\_\{k\}^\{\\infty\}\+Y\_\{k\}^\{\\infty\}\)\\\|\_\{F\}^\{2\},\\qquad\\rho Y\_\{k\}^\{\\infty\}\+n\_\{k\}\\nabla\_\{U\_\{k\}\}f\(U^\{\\infty\};\{\\bm\{\\alpha\}\}^\{\\infty\},\{\\bm\{\\beta\}\}^\{\\infty\},\{\\bm\{\\gamma\}\}^\{\\infty\}\)=0\.\(63\)If every entry ofUk∞\+Yk∞U\_\{k\}^\{\\infty\}\+Y\_\{k\}^\{\\infty\}is nonzero, the argmin is the singleton\{sign⁡\(Uk∞\+Yk∞\)\}\\\{\\operatorname\{sign\}\(U\_\{k\}^\{\\infty\}\+Y\_\{k\}^\{\\infty\}\)\\\}\.

###### Proof\.

By Assumption[E\.1](https://arxiv.org/html/2605.24058#A5.Thmtheorem1), the tail sequence has a cluster point\. SinceΦ\\Phiis semi\-algebraic, it satisfies the Kurdyka–Łojasiewicz inequality at every point of its domain\[Bolte et al\.,[2007](https://arxiv.org/html/2605.24058#bib.bib5), Attouch et al\.,[2010](https://arxiv.org/html/2605.24058#bib.bib2)\]\. Combining the KL inequality at a cluster point with the sufficient\-decrease and relative\-error estimates \([60](https://arxiv.org/html/2605.24058#A5.E60)\)–\([61](https://arxiv.org/html/2605.24058#A5.E61)\) gives the standard finite\-length conclusion for descent methods\[Attouch et al\.,[2013](https://arxiv.org/html/2605.24058#bib.bib3), Wang et al\.,[2019](https://arxiv.org/html/2605.24058#bib.bib36)\]:

∑t≥T‖𝒳\(t\+1\)−𝒳\(t\)‖<∞\.\\sum\_\{t\\geq T\}\\\|\\mathcal\{X\}^\{\(t\+1\)\}\-\\mathcal\{X\}^\{\(t\)\}\\\|<\\infty\.\(64\)Hence\{𝒳\(t\)\}t≥T\\\{\\mathcal\{X\}^\{\(t\)\}\\\}\_\{t\\geq T\}is Cauchy and converges to a single limit\. Projecting this convergence onto the variables\(U,M,Y,𝜶,𝜷,𝜸\)\(U,M,Y,\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)proves Eq\. \([62](https://arxiv.org/html/2605.24058#A5.E62)\), and claim \(i\) follows immediately from Eq\. \([64](https://arxiv.org/html/2605.24058#A5.E64)\)\.

For claim \(ii\), each entry ofΔ​Mk\(t\)=Mk\(t\+1\)−Mk\(t\)\\Delta M\_\{k\}^\{\(t\)\}=M\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}belongs to\{−2,0,2\}\\\{\-2,0,2\\\}\. Therefore, wheneverΔ​Mk\(t\)≠0\\Delta M\_\{k\}^\{\(t\)\}\\neq 0,‖Δ​Mk\(t\)‖F≥2\\\|\\Delta M\_\{k\}^\{\(t\)\}\\\|\_\{F\}\\geq 2\. The summability in claim \(i\) implies that only finitely many such nonzero jumps can occur\. Thus the two binary blocks are eventually constant; denote the common tail value byM⋆M^\{\\star\}\.

It remains to verify stationarity\. The dual update givesYk\(t\+1\)−Yk\(t\)=Uk\(t\+1\)−Mk\(t\+1\)Y\_\{k\}^\{\(t\+1\)\}\-Y\_\{k\}^\{\(t\)\}=U\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\+1\)\}\. Since the left\-hand side tends to zero by claim \(i\), passing to the limit givesUk∞=Mk⋆U\_\{k\}^\{\\infty\}=M\_\{k\}^\{\\star\}\. The binary update is the exact entrywise projection

Mk\(t\+1\)∈arg​minm∈\{±1\}nk⁡‖m−\(Uk\(t\+1\)\+Yk\(t\)\)‖F2\.M\_\{k\}^\{\(t\+1\)\}\\in\\operatorname\*\{arg\\,min\}\_\{m\\in\\\{\\pm 1\\\}^\{n\_\{k\}\}\}\\\|m\-\(U\_\{k\}^\{\(t\+1\)\}\+Y\_\{k\}^\{\(t\)\}\)\\\|\_\{F\}^\{2\}\.Taking limits in this finite\-dimensional inequality for everym∈\{±1\}nkm\\in\\\{\\pm 1\\\}^\{n\_\{k\}\}yields the argmin condition in Eq\. \([63](https://arxiv.org/html/2605.24058#A5.E63)\)\. Finally, by claim \(ii\) the termMk\(t\+1\)−Mk\(t\)M\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\)\}in Eq\. \([53](https://arxiv.org/html/2605.24058#A5.E53)\) is zero for all sufficiently largett\. Passing to the limit in Eq\. \([53](https://arxiv.org/html/2605.24058#A5.E53)\), using the continuity of∇Ukf\\nabla\_\{U\_\{k\}\}fon𝒞\\mathcal\{C\}, gives the gradient–dual stationarity equation in Eq\. \([63](https://arxiv.org/html/2605.24058#A5.E63)\)\. IfUk∞\+Yk∞U\_\{k\}^\{\\infty\}\+Y\_\{k\}^\{\\infty\}has no zero entries, the binary projection decouples entrywise and is uniquely given by the sign of that vector, proving the final assertion\. ∎

## Appendix FQAT\-LoRDBA: Smooth\-Sign STE

QAT\-LoRDBA optimizes real\-valued latent carrier matricesU1,U2U\_\{1\},U\_\{2\}together with the scale vectors\(𝜶,𝜷,𝜸\)\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)\. On the forward pass it exports the binary carriersB1=sign⁡\(U1\)B\_\{1\}=\\operatorname\{sign\}\(U\_\{1\}\)andB2=sign⁡\(U2\)B\_\{2\}=\\operatorname\{sign\}\(U\_\{2\}\)insideΔ​W​\(θ\)\\Delta W\(\\theta\)as in Eq\. \([4](https://arxiv.org/html/2605.24058#S3.E4)\); on the backward pass we use the*smooth\-sign*straight\-through estimator\[Leng et al\.,[2018](https://arxiv.org/html/2605.24058#bib.bib23), Liu et al\.,[2020](https://arxiv.org/html/2605.24058#bib.bib28)\]for the non\-differentiable sign maps:

∂∂u​sign⁡\(u\)⟵∂∂u​tanh⁡\(κ​u\)=κ​\(1−tanh2⁡\(κ​u\)\),\\frac\{\\partial\}\{\\partial u\}\\operatorname\{sign\}\(u\)\\;\\longleftarrow\\;\\frac\{\\partial\}\{\\partial u\}\\tanh\(\\kappa u\)\\;=\\;\\kappa\\bigl\(1\-\\tanh^\{2\}\(\\kappa u\)\\bigr\),\(65\)withκ=100\\kappa=100by default\. The scales\(𝜶,𝜷,𝜸\)\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)are trained in fp32 and do not require an STE\. We initialise the latent carriersU1,U2U\_\{1\},U\_\{2\}via the SVD warm start of Eq\. \([7](https://arxiv.org/html/2605.24058#S3.E7)\) from a10%10\\%\-budget warm\-up fp16 LoRA run, which we find substantially stabilises optimisation at carrier rankR≥64R\\geq 64\. Algorithm[2](https://arxiv.org/html/2605.24058#alg2)summarises the training loop\.

## Appendix GTraining Mode Comparison

Table[6](https://arxiv.org/html/2605.24058#A7.T6)compares the four LoRDBA training modes at each size tier onLLaMA\-2\-7B\. Several patterns emerge: \(i\) QAT Full consistently leads, followed closely by QAT Freeze at matched BPW\. \(ii\) QAT Scratch, which does not require a pre\-trained FP16 adapter, trails QAT Full by44–99pt on GSM8K, confirming that the PTQ warm\-start is decisive for task\-specific adapters\. \(iii\) PTQ\-LoRDBA—training\-free and requiring only∼2\{\\sim\}2min—degrades sharply at≤2\\leq 2BPW on math reasoning but retains reasonable summarization quality, achieving XSum17\.1817\.18vs\. FP1618\.8718\.87at 1 BPW, revealing task\-dependent quantization sensitivity that QAT closes\.

Table 6:LLaMA\-2\-7B: LoRDBA training mode comparison \(Avg==\(GSM8K \+ Minerva \+ XSum\)/3\)\.“–” = configuration not evaluated at that tier\. QAT Freeze @2bpw matches QAT Full within0\.020\.02Avg\.

#### QAT Freeze training time\.

Despite training only scale vectors—l​\(N\+R\+M\)l\(N\{\+\}R\{\+\}M\)parameters per layer—QAT Freeze requires wall\-clock time comparable to QAT Full, approximately1111–1515h at 2 epochs on H100, because the dominant cost is the full forward–backward pass through the frozen44\-bit base model, which is identical across all QAT modes\. The Adam optimizer state is smaller due to the absence of latent carriers, so peak memory is marginally lower, but each training step still performs the complete loss computation and gradient propagation through all Transformer layers\. The parameter update itself is a negligible fraction of total step time\.

#### BitDelta comparison\.

BitDelta\[Liu et al\.,[2024a](https://arxiv.org/html/2605.24058#bib.bib26)\]operates at4\.84\.8MB in its column\+\+distill variant, a smaller size tier than LoRDBA’s primary operating points of77–1919MB\. At this size, BitDelta achieves GSM8K31\.9231\.92, Minerva6\.466\.46, XSum18\.7018\.70, yielding Avg19\.0319\.03, substantially below QAT Full @1bpw with Avg24\.1424\.14at7\.27\.2MB\. The XSum result of18\.7018\.70is competitive, suggesting that for summarization at the∼5\{\\sim\}5MB tier a full\-rank binary delta may be viable; however, the low\-rank LoRDBA structure dominates at all tiers where both methods can operate\.

## Appendix HExtended Numerical Results

This appendix reports adapter\-mode results that complement Section[5\.1](https://arxiv.org/html/2605.24058#S5.SS1): a comprehensive all\-methods comparison onLLaMA\-2\-7Bacross three tasks \(Table[7](https://arxiv.org/html/2605.24058#A8.T7)\); aLLaMA\-3\.2\-3Bablation \(Table[8](https://arxiv.org/html/2605.24058#A8.T8)\); and the matched training\-time accounting \(Table[9](https://arxiv.org/html/2605.24058#A8.T9)\)\. All rows are evaluated under the adapter\-mode protocol of Appendix[K](https://arxiv.org/html/2605.24058#A11)\. Each task uses a*task\-specific*LoRA FP16r=16r\{=\}16adapter; all post\-hoc methods compress the same adapter per task\.

Table 7:Llama\-2\-7B, rank 16: All LoRA adapter compression methods across three tasks\. Each task uses a*task\-specific*LoRA FP16r=16r\{=\}16adapter\. GSM8K \(8\-shot exact match %\), Minerva Math \(4\-shot %\), XSum \(ROUGE\-L F1\)\. All post\-hoc methods compress the same LoRA FP16 per task\.“–” = not evaluated\.Bold= best among compressed methods at≤19\\leq 19MB at matching size tier\. ‡Time = additional cost after LoRA FP16 training: Math∼\\sim5h, Summ∼\\sim3–6h on H100\. QAT Scratch does not require a pre\-trained FP16 adapter\. All runs use a single NVIDIA H100 80 GB or B200\.

Table 8:LLaMA\-3\.2\-3B, rank 16: all LoRA adapter compression methods across three tasks\. Each task uses a*task\-specific*LoRA FP16r=16r\{=\}16adapter\. GSM8K \(8\-shot exact match %\), Minerva Math \(4\-shot %\), XSum \(ROUGE\-L F1\)\. All post\-hoc methods compress the same LoRA FP16 per task\.“–” = not evaluated\.Bold= best among compressed methods at the corresponding size tier\. ‡Time = additional cost after LoRA FP16 training,∼\\sim5h on H100\. QAT Scratch does not require a pre\-trained FP16 adapter\. All runs use a single NVIDIA H100 80 GB or B200\.

Table 9:Adapter\-mode QAT Full training statistics onLLaMA\-2\-7B\(MetaMathQA\)\. LoRA pre\-trained for22epochs; QAT for11epoch \(bf16 base, single NVIDIA B200\)\. “Peak” is the device\-side QAT training peak memory; “ms/step” is the average wall\-clock per QAT training step\.#### Reconstruction error\.

Table[10](https://arxiv.org/html/2605.24058#A8.T10)records the ADMM\-converged relative reconstruction error ofΔ​W⋆\\Delta W^\{\\star\}by PTQ\-LoRDBA as a function of BPW\. The monotonic decay with rank is qualitatively consistent with Theorem[4\.1](https://arxiv.org/html/2605.24058#S4.Thmtheorem1)and with the empirical residual\-to\-mean ratios of Table[5](https://arxiv.org/html/2605.24058#A3.T5)\.

Table 10:ADMM\-converged per\-element relative reconstruction error‖Δ​W⋆−Δ​W​\(θ\)‖F/‖Δ​W⋆‖F\\\|\\Delta W^\{\\star\}\-\\Delta W\(\\theta\)\\\|\_\{F\}/\\\|\\Delta W^\{\\star\}\\\|\_\{F\}onLLaMA\-2\-7B\(averaged across projections\)\.

## Appendix IPseudocode

### I\.1Closed\-form scale updates for PTQ\-LoRDBA

This subsection derives the per\-axis least\-squares updates used by Eq\. \([48](https://arxiv.org/html/2605.24058#A4.E48)\)\. Throughout, letRRdenote the LoRDBA binary carrier rank of Definition[3\.1](https://arxiv.org/html/2605.24058#S3.Thmtheorem1)\(so thatB1∈\{±1\}N×RB\_\{1\}\\in\\\{\\pm 1\\\}^\{N\\times R\}andB2∈\{±1\}R×MB\_\{2\}\\in\\\{\\pm 1\\\}^\{R\\times M\}\); to avoid the notation clash, we use the symbolTTfor the regression target, defined below\. We treat the canonical single\-envelope caseℓ=1\\ell=1; the multi\-envelope case applies the same updates block\-coordinate\-wise to each envelope while holding the remaining envelopes fixed \(see the residual definition at the end of this subsection\)\.

#### Setup\.

Fix the binary carriersB1,B2B\_\{1\},B\_\{2\}and define the regression target

T≔Δ​W⋆∈ℝN×M\.T\\;\\coloneqq\\;\\Delta W^\{\\star\}\\;\\in\\;\{\\mathbb\{R\}\}^\{N\\times M\}\.\(66\)The PTQ\-LoRDBA scale subproblem is the joint minimisation

min𝜶∈ℝN,𝜷∈ℝR,𝜸∈ℝM⁡12​‖T−diag⁡\(𝜶\)​B1​diag⁡\(𝜷\)​B2​diag⁡\(𝜸\)‖F2,\\min\_\{\{\\bm\{\\alpha\}\}\\in\{\\mathbb\{R\}\}^\{N\},\{\\bm\{\\beta\}\}\\in\{\\mathbb\{R\}\}^\{R\},\{\\bm\{\\gamma\}\}\\in\{\\mathbb\{R\}\}^\{M\}\}\\;\\frac\{1\}\{2\}\\bigl\\\|T\-\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\bigr\\\|\_\{F\}^\{2\},\(67\)which we solve by per\-axis block\-coordinate descent\. Each per\-axis subproblem admits a closed\-form least\-squares solution, derived next\.

#### 𝜶\{\\bm\{\\alpha\}\}\-update \(rowwise\)\.

For fixedB1,B2,𝜷,𝜸B\_\{1\},B\_\{2\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}, set

Dα≔B1​diag⁡\(𝜷\)​B2​diag⁡\(𝜸\)∈ℝN×M\.D\_\{\\alpha\}\\;\\coloneqq\\;B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\;\\in\\;\{\\mathbb\{R\}\}^\{N\\times M\}\.\(68\)Substituting Eq\. \([68](https://arxiv.org/html/2605.24058#A9.E68)\) into Eq\. \([67](https://arxiv.org/html/2605.24058#A9.E67)\) and expanding the Frobenius norm rowwise,

12​‖T−diag⁡\(𝜶\)​Dα‖F2=∑i=1N12​‖Ti,:−αi​\(Dα\)i,:‖22,\\frac\{1\}\{2\}\\bigl\\\|T\-\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)D\_\{\\alpha\}\\bigr\\\|\_\{F\}^\{2\}\\;=\\;\\sum\_\{i=1\}^\{N\}\\frac\{1\}\{2\}\\bigl\\\|T\_\{i,:\}\-\\alpha\_\{i\}\(D\_\{\\alpha\}\)\_\{i,:\}\\bigr\\\|\_\{2\}^\{2\},which decouples over the row indexii\. Each scalar problem

minαi∈ℝ⁡12​‖Ti,:−αi​\(Dα\)i,:‖22\\min\_\{\\alpha\_\{i\}\\in\{\\mathbb\{R\}\}\}\\;\\frac\{1\}\{2\}\\bigl\\\|T\_\{i,:\}\-\\alpha\_\{i\}\(D\_\{\\alpha\}\)\_\{i,:\}\\bigr\\\|\_\{2\}^\{2\}\(69\)has the unique minimum\-norm least\-squares solution

αi=\{⟨Ti,:,\(Dα\)i,:⟩‖\(Dα\)i,:‖22,‖\(Dα\)i,:‖2\>0,0,‖\(Dα\)i,:‖2=0,i∈\[N\]\.\\alpha\_\{i\}\\;=\\;\\begin\{cases\}\\dfrac\{\\langle T\_\{i,:\},\(D\_\{\\alpha\}\)\_\{i,:\}\\rangle\}\{\\\|\(D\_\{\\alpha\}\)\_\{i,:\}\\\|\_\{2\}^\{2\}\},&\\\|\(D\_\{\\alpha\}\)\_\{i,:\}\\\|\_\{2\}\>0,\\\\\[8\.00003pt\] 0,&\\\|\(D\_\{\\alpha\}\)\_\{i,:\}\\\|\_\{2\}=0,\\end\{cases\}\\qquad i\\in\[N\]\.\(70\)

#### 𝜸\{\\bm\{\\gamma\}\}\-update \(columnwise\)\.

For fixedB1,B2,𝜶,𝜷B\_\{1\},B\_\{2\},\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\}, set

Dγ≔diag⁡\(𝜶\)​B1​diag⁡\(𝜷\)​B2∈ℝN×M\.D\_\{\\gamma\}\\;\\coloneqq\\;\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)B\_\{2\}\\;\\in\\;\{\\mathbb\{R\}\}^\{N\\times M\}\.\(71\)The objective Eq\. \([67](https://arxiv.org/html/2605.24058#A9.E67)\) expands columnwise as

12​‖T−Dγ​diag⁡\(𝜸\)‖F2=∑j=1M12​‖T:,j−γj​\(Dγ\):,j‖22,\\frac\{1\}\{2\}\\bigl\\\|T\-D\_\{\\gamma\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\bigr\\\|\_\{F\}^\{2\}\\;=\\;\\sum\_\{j=1\}^\{M\}\\frac\{1\}\{2\}\\bigl\\\|T\_\{:,j\}\-\\gamma\_\{j\}\(D\_\{\\gamma\}\)\_\{:,j\}\\bigr\\\|\_\{2\}^\{2\},and analogously decouples overj∈\[M\]j\\in\[M\]\. The unique minimum\-norm least\-squares solution is

γj=\{⟨T:,j,\(Dγ\):,j⟩‖\(Dγ\):,j‖22,‖\(Dγ\):,j‖2\>0,0,‖\(Dγ\):,j‖2=0,j∈\[M\]\.\\gamma\_\{j\}\\;=\\;\\begin\{cases\}\\dfrac\{\\langle T\_\{:,j\},\(D\_\{\\gamma\}\)\_\{:,j\}\\rangle\}\{\\\|\(D\_\{\\gamma\}\)\_\{:,j\}\\\|\_\{2\}^\{2\}\},&\\\|\(D\_\{\\gamma\}\)\_\{:,j\}\\\|\_\{2\}\>0,\\\\\[8\.00003pt\] 0,&\\\|\(D\_\{\\gamma\}\)\_\{:,j\}\\\|\_\{2\}=0,\\end\{cases\}\\qquad j\\in\[M\]\.\(72\)

#### 𝜷\{\\bm\{\\beta\}\}\-update \(R\-dimensional\)\.

The𝜷\{\\bm\{\\beta\}\}\-update is anRR\-dimensional least\-squares problem\. For fixedB1,B2,𝜶,𝜸B\_\{1\},B\_\{2\},\{\\bm\{\\alpha\}\},\{\\bm\{\\gamma\}\}, define

P≔diag⁡\(𝜶\)​B1∈ℝN×R,Q≔B2​diag⁡\(𝜸\)∈ℝR×M\.P\\;\\coloneqq\\;\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)B\_\{1\}\\;\\in\\;\{\\mathbb\{R\}\}^\{N\\times R\},\\qquad Q\\;\\coloneqq\\;B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\;\\in\\;\{\\mathbb\{R\}\}^\{R\\times M\}\.\(73\)Expanding the bilinear identity

diag⁡\(𝜶\)​B1​diag⁡\(𝜷\)​B2​diag⁡\(𝜸\)=P​diag⁡\(𝜷\)​Q=∑k=1Rβk​P:,k​Qk,:,\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)\\;=\\;P\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)Q\\;=\\;\\sum\_\{k=1\}^\{R\}\\beta\_\{k\}P\_\{:,k\}Q\_\{k,:\},\(74\)where the second equality uses the column–row decompositiondiag⁡\(𝜷\)=∑kβk​𝒆k​𝒆k⊤\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)=\\sum\_\{k\}\\beta\_\{k\}\{\\bm\{e\}\}\_\{k\}\{\\bm\{e\}\}\_\{k\}^\{\\top\}andP​𝒆k​𝒆k⊤​Q=P:,k​Qk,:P\{\\bm\{e\}\}\_\{k\}\{\\bm\{e\}\}\_\{k\}^\{\\top\}Q=P\_\{:,k\}Q\_\{k,:\}\. Vectorising Eq\. \([74](https://arxiv.org/html/2605.24058#A9.E74)\) givesvec⁡\(P​diag⁡\(𝜷\)​Q\)=H​𝜷\\operatorname\{vec\}\(P\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)Q\)=H\{\\bm\{\\beta\}\}withH∈ℝN​M×RH\\in\{\\mathbb\{R\}\}^\{NM\\times R\}defined by

H:,k≔vec⁡\(P:,k​Qk,:\),k∈\[R\]\.H\_\{:,k\}\\;\\coloneqq\\;\\operatorname\{vec\}\(P\_\{:,k\}Q\_\{k,:\}\),\\qquad k\\in\[R\]\.\(75\)Hence Eq\. \([67](https://arxiv.org/html/2605.24058#A9.E67)\) reduces, in the𝜷\{\\bm\{\\beta\}\}\-direction, to the standard linear least\-squares problem

min𝜷∈ℝR⁡12​‖vec⁡\(T\)−H​𝜷‖22,\\min\_\{\{\\bm\{\\beta\}\}\\in\{\\mathbb\{R\}\}^\{R\}\}\\;\\frac\{1\}\{2\}\\bigl\\\|\\operatorname\{vec\}\(T\)\-H\{\\bm\{\\beta\}\}\\bigr\\\|\_\{2\}^\{2\},\(76\)with normal equationsH⊤​H​𝜷=H⊤​vec⁡\(T\)H^\{\\top\}H\{\\bm\{\\beta\}\}=H^\{\\top\}\\operatorname\{vec\}\(T\)\. We record these in our notation as

Gβ​𝜷=𝒉β,G\_\{\\beta\}\{\\bm\{\\beta\}\}\\;=\\;\{\\bm\{h\}\}\_\{\\beta\},\(77\)where, fork,l∈\[R\]k,l\\in\[R\],

\(Gβ\)k​l=⟨P:,k,P:,l⟩​⟨Qk,:,Ql,:⟩,\(𝒉β\)k=⟨T,P:,k​Qk,:⟩F\.\(G\_\{\\beta\}\)\_\{kl\}\\;=\\;\\langle P\_\{:,k\},P\_\{:,l\}\\rangle\\langle Q\_\{k,:\},Q\_\{l,:\}\\rangle,\\qquad\(\{\\bm\{h\}\}\_\{\\beta\}\)\_\{k\}\\;=\\;\\langle T,P\_\{:,k\}Q\_\{k,:\}\\rangle\_\{F\}\.\(78\)Equivalently, in matrix form,

Gβ=\(P⊤​P\)⊙\(Q​Q⊤\),G\_\{\\beta\}\\;=\\;\(P^\{\\top\}P\)\\odot\(QQ^\{\\top\}\),\(79\)where⊙\\odotis the Hadamard product\. SinceP⊤​PP^\{\\top\}PandQ​Q⊤QQ^\{\\top\}are positive semidefinite,GβG\_\{\\beta\}is positive semidefinite by the Schur product theorem\. WhenGβG\_\{\\beta\}is nonsingular, the update is𝜷=Gβ−1​𝒉β\{\\bm\{\\beta\}\}=G\_\{\\beta\}^\{\-1\}\{\\bm\{h\}\}\_\{\\beta\}; when singular, we use the unique minimum\-norm least\-squares solution

𝜷=Gβ†​𝒉β,\{\\bm\{\\beta\}\}\\;=\\;G\_\{\\beta\}^\{\\dagger\}\{\\bm\{h\}\}\_\{\\beta\},\(80\)whereGβ†G\_\{\\beta\}^\{\\dagger\}denotes the Moore–Penrose pseudoinverse\.

#### Generalisation to envelope rankℓ≥2\\ell\\geq 2\.

For envelope rankℓ≥2\\ell\\geq 2, the same per\-axis updates Eqs\. \([70](https://arxiv.org/html/2605.24058#A9.E70)\), \([72](https://arxiv.org/html/2605.24058#A9.E72)\), and \([80](https://arxiv.org/html/2605.24058#A9.E80)\) apply to one envelopei∈\[ℓ\]i\\in\[\\ell\]at a time after replacing the targetTTby the per\-envelope residual

T\(i\)≔Δ​W⋆−∑m≠idiag⁡\(𝜶\(m\)\)​B1​diag⁡\(𝜷\(m\)\)​B2​diag⁡\(𝜸\(m\)\),i∈\[ℓ\]\.T^\{\(i\)\}\\;\\coloneqq\\;\\Delta W^\{\\star\}\-\\sum\_\{m\\neq i\}\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}^\{\(m\)\}\)B\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}^\{\(m\)\}\)B\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}^\{\(m\)\}\),\\qquad i\\in\[\\ell\]\.\(81\)Cycling the envelope indexi∈\[ℓ\]i\\in\[\\ell\]in any order produces a block\-coordinate descent on Eq\. \([67](https://arxiv.org/html/2605.24058#A9.E67)\) extended toℓ\\ellenvelopes, and each per\-envelope update inherits the closed form above withTTreplaced byT\(i\)T^\{\(i\)\}\.

Algorithm 1PTQ\-LoRDBA atℓ=1\\ell=1\(scaled consensus ADMM\)\.1:fp16 LoRA factors

A⋆,B⋆A^\{\\star\},B^\{\\star\}, carrier rank

RR, outer iterations

KK, penalty schedule

\{ρ\(t\)\}\\\{\\rho^\{\(t\)\}\\\}
2:LoRDBA adapter

θ⋆=\(B1⋆,B2⋆,𝜶⋆,𝜷⋆,𝜸⋆\)\\theta^\{\\star\}=\(B\_\{1\}^\{\\star\},B\_\{2\}^\{\\star\},\{\\bm\{\\alpha\}\}^\{\\star\},\{\\bm\{\\beta\}\}^\{\\star\},\{\\bm\{\\gamma\}\}^\{\\star\}\)
3:

\(UΣ,SΣ,VΣ\)←thinSVD​\(A⋆​\(B⋆\)⊤,rank=R\)\(U\_\{\\Sigma\},S\_\{\\Sigma\},V\_\{\\Sigma\}\)\\leftarrow\\mathrm\{thinSVD\}\(A^\{\\star\}\(B^\{\\star\}\)^\{\\top\},\\text\{rank\}=R\)⊳\\trianglerightEq\. \([7](https://arxiv.org/html/2605.24058#S3.E7)\)

4:

U1\(0\)←sign⁡\(UΣ\)U\_\{1\}^\{\(0\)\}\\leftarrow\\operatorname\{sign\}\(U\_\{\\Sigma\}\),

U2\(0\)←sign⁡\(VΣ⊤\)U\_\{2\}^\{\(0\)\}\\leftarrow\\operatorname\{sign\}\(V\_\{\\Sigma\}^\{\\top\}\),

𝜷\(0\)←diag⁡\(SΣ\)\{\\bm\{\\beta\}\}^\{\(0\)\}\\leftarrow\\operatorname\{diag\}\(S\_\{\\Sigma\}\)
5:

Mk\(0\)←Uk\(0\)M\_\{k\}^\{\(0\)\}\\leftarrow U\_\{k\}^\{\(0\)\},

Yk\(0\)←0Y\_\{k\}^\{\(0\)\}\\leftarrow 0for

k∈\{1,2\}k\\in\\\{1,2\\\}
6:Initialise

\(𝜶\(0\),𝜸\(0\)\)\(\{\\bm\{\\alpha\}\}^\{\(0\)\},\{\\bm\{\\gamma\}\}^\{\(0\)\}\)by one block\-coordinate sweep of Eq\. \([48](https://arxiv.org/html/2605.24058#A4.E48)\) at

\(U\(0\),𝜷\(0\)\)\(U^\{\(0\)\},\{\\bm\{\\beta\}\}^\{\(0\)\}\)
7:for

t=0,1,…,K−1t=0,1,\\dots,K\-1do

8:Update

U1\(t\+1\)U\_\{1\}^\{\(t\+1\)\},

U2\(t\+1\)U\_\{2\}^\{\(t\+1\)\}by Tikhonov least\-squares \(Eqs\. \([46](https://arxiv.org/html/2605.24058#A4.E46)\)–\([47](https://arxiv.org/html/2605.24058#A4.E47)\)\)

9:Update

\(𝜶\(t\+1\),𝜷\(t\+1\),𝜸\(t\+1\)\)\(\{\\bm\{\\alpha\}\}^\{\(t\+1\)\},\{\\bm\{\\beta\}\}^\{\(t\+1\)\},\{\\bm\{\\gamma\}\}^\{\(t\+1\)\}\)by one block\-coordinate sweep of Eq\. \([48](https://arxiv.org/html/2605.24058#A4.E48)\) \(per\-axis closed\-form least\-squares\)

10:

Mk\(t\+1\)←sign⁡\(Uk\(t\+1\)\+Yk\(t\)\)M\_\{k\}^\{\(t\+1\)\}\\leftarrow\\operatorname\{sign\}\(U\_\{k\}^\{\(t\+1\)\}\+Y\_\{k\}^\{\(t\)\}\)for

k∈\{1,2\}k\\in\\\{1,2\\\}
11:

Yk\(t\+1\)←Yk\(t\)\+Uk\(t\+1\)−Mk\(t\+1\)Y\_\{k\}^\{\(t\+1\)\}\\leftarrow Y\_\{k\}^\{\(t\)\}\+U\_\{k\}^\{\(t\+1\)\}\-M\_\{k\}^\{\(t\+1\)\}for

k∈\{1,2\}k\\in\\\{1,2\\\}
12:Residual\-balance

ρ\(t\+1\)\\rho^\{\(t\+1\)\}\(Boyd, truncated at

K/2K/2\)

13:if

M\(t\+1\)=M\(t\)M^\{\(t\+1\)\}=M^\{\(t\)\}thenbreak⊳\\trianglerightdiscrete freeze; Theorem[E\.6](https://arxiv.org/html/2605.24058#A5.Thmtheorem6)\(ii\)

14:endif

15:endfor

16:return

\(M1\(t\),M2\(t\),𝜶\(t\),𝜷\(t\),𝜸\(t\)\)\(M\_\{1\}^\{\(t\)\},M\_\{2\}^\{\(t\)\},\{\\bm\{\\alpha\}\}^\{\(t\)\},\{\\bm\{\\beta\}\}^\{\(t\)\},\{\\bm\{\\gamma\}\}^\{\(t\)\}\)

Algorithm 2QAT\-LoRDBA \(smooth\-sign STE\)\.1:base model

W0W\_\{0\}, stream

\{\(𝒙t,𝒚t\)\}\\\{\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{y\}\}\_\{t\}\)\\\}, carrier rank

RR, STE temperature

κ\\kappa, optimiser

𝗈𝗉𝗍\\mathsf\{opt\}
2:Initialise

\(U1,U2\)\(U\_\{1\},U\_\{2\}\)via Eq\. \([7](https://arxiv.org/html/2605.24058#S3.E7)\) from a

10%10\\%\-budget fp16 LoRA warm\-up

3:Initialise scales

\(𝜶,𝜷,𝜸\)\(\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)by one block\-coordinate sweep of Eq\. \([48](https://arxiv.org/html/2605.24058#A4.E48)\) \(per\-axis closed\-form least\-squares\)

4:foreach training step

ttdo

5:Forward:

Δ​W←diag⁡\(𝜶\)​sign⁡\(U1\)​diag⁡\(𝜷\)​sign⁡\(U2\)​diag⁡\(𝜸\)\\Delta W\\leftarrow\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}\)\\operatorname\{sign\}\(U\_\{1\}\)\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}\)\\operatorname\{sign\}\(U\_\{2\}\)\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}\)
6:Compute task loss

ℓt​\(W0\+Δ​W;𝒙t,𝒚t\)\\ell\_\{t\}\(W\_\{0\}\+\\Delta W;\{\\bm\{x\}\}\_\{t\},\{\\bm\{y\}\}\_\{t\}\)
7:Backward: replace

∂sign⁡\(U\)/∂U\\partial\\operatorname\{sign\}\(U\)/\\partial Uby

κ​\(1−tanh2⁡\(κ​U\)\)\\kappa\(1\-\\tanh^\{2\}\(\\kappa U\)\)in autograd

8:Step

\(U1,U2,𝜶,𝜷,𝜸\)←𝗈𝗉𝗍​\(⋅,∇ℓt\)\(U\_\{1\},U\_\{2\},\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)\\leftarrow\\mathsf\{opt\}\(\\cdot,\\nabla\\ell\_\{t\}\)
9:endfor

10:return

\(B1,B2,𝜶,𝜷,𝜸\)←\(sign⁡\(U1\),sign⁡\(U2\),𝜶,𝜷,𝜸\)\(B\_\{1\},B\_\{2\},\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)\\leftarrow\(\\operatorname\{sign\}\(U\_\{1\}\),\\operatorname\{sign\}\(U\_\{2\}\),\{\\bm\{\\alpha\}\},\{\\bm\{\\beta\}\},\{\\bm\{\\gamma\}\}\)

Algorithm 3PTQ\-LoRDBA at general envelope rankℓ≥1\\ell\\geq 1\.1:fp16 LoRA factors

A⋆,B⋆A^\{\\star\},B^\{\\star\}, carrier rank

RR, envelope rank

ℓ≥1\\ell\\geq 1, outer iterations

KK
2:

\(UΣ,SΣ,VΣ\)←thinSVD​\(A⋆​\(B⋆\)⊤,rank=R\)\(U\_\{\\Sigma\},S\_\{\\Sigma\},V\_\{\\Sigma\}\)\\leftarrow\\mathrm\{thinSVD\}\(A^\{\\star\}\(B^\{\\star\}\)^\{\\top\},\\text\{rank\}=R\)
3:Split singular spectrum into

ℓ\\elldisjoint index sets

ℐ1,…,ℐℓ⊆\[R\]\\mathcal\{I\}\_\{1\},\\dots,\\mathcal\{I\}\_\{\\ell\}\\subseteq\[R\]; initialise

𝜷\(i\)\{\\bm\{\\beta\}\}^\{\(i\)\}from

\{\(SΣ\)j​j\}j∈ℐi\\\{\(S\_\{\\Sigma\}\)\_\{jj\}\\\}\_\{j\\in\\mathcal\{I\}\_\{i\}\}\(extended by zeros to

ℝR\{\\mathbb\{R\}\}^\{R\}\)

4:Initialise

\(B1,B2\)\(B\_\{1\},B\_\{2\}\)by Eq\. \([7](https://arxiv.org/html/2605.24058#S3.E7)\), and

\{\(𝜶\(i\),𝜸\(i\)\)\}i=1ℓ\\\{\(\{\\bm\{\\alpha\}\}^\{\(i\)\},\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\ell\}by solving the per\-envelope scale step

5:for

t=0,1,…,K−1t=0,1,\\dots,K\-1do

6:

\(U1,U2\)\(U\_\{1\},U\_\{2\}\)\-update: outer\-loop sign\-ADMM step \(Algorithm[1](https://arxiv.org/html/2605.24058#alg1), lines 6–10\) with

ffreplaced by the envelope\-sum objective

12​‖Δ​W⋆−∑idiag⁡\(𝜶\(i\)\)​U1​diag⁡\(𝜷\(i\)\)​U2​diag⁡\(𝜸\(i\)\)‖F2\\frac\{1\}\{2\}\\\|\\Delta W^\{\\star\}\-\\sum\_\{i\}\\operatorname\{diag\}\(\{\\bm\{\\alpha\}\}^\{\(i\)\}\)U\_\{1\}\\operatorname\{diag\}\(\{\\bm\{\\beta\}\}^\{\(i\)\}\)U\_\{2\}\\operatorname\{diag\}\(\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\\|\_\{F\}^\{2\}
7:Update

\{\(𝜶\(i\),𝜷\(i\),𝜸\(i\)\)\}i=1ℓ\\\{\(\{\\bm\{\\alpha\}\}^\{\(i\)\},\{\\bm\{\\beta\}\}^\{\(i\)\},\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\ell\}by block\-coordinate closed\-form least\-squares, cycling over envelope index

ii
8:endfor

9:return

\(B1,B2,\{\(𝜶\(i\),𝜷\(i\),𝜸\(i\)\)\}i=1ℓ\)\\bigl\(B\_\{1\},B\_\{2\},\\\{\(\{\\bm\{\\alpha\}\}^\{\(i\)\},\{\\bm\{\\beta\}\}^\{\(i\)\},\{\\bm\{\\gamma\}\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\ell\}\\bigr\)

## Appendix JInference\-Kernel Details

Equation \([8](https://arxiv.org/html/2605.24058#S3.E8)\) expresses the LoRDBA forward pass as three element\-wise scale fusions and two binary matmuls\. Each binary matmul𝒛=B⊤​𝒘\{\\bm\{z\}\}=B^\{\\top\}\{\\bm\{w\}\}withB∈\{±1\}p×qB\\in\\\{\\pm 1\\\}^\{p\\times q\}and fp16 input𝒘∈ℝp\{\\bm\{w\}\}\\in\{\\mathbb\{R\}\}^\{p\}computes, for everyj∈\[q\]j\\in\[q\], the exact sign\-accumulation

zj=∑i=1pBi​j​wi=∑i:Bi​j=\+1wi−∑i:Bi​j=−1wi,z\_\{j\}\\;=\\;\\sum\_\{i=1\}^\{p\}B\_\{ij\}w\_\{i\}\\;=\\;\\sum\_\{i:B\_\{ij\}=\+1\}w\_\{i\}\\;\-\\;\\sum\_\{i:B\_\{ij\}=\-1\}w\_\{i\},\(82\)i\.e\. a signed sum of fp16 values with no floating\-point multiplications\. On contemporary GPUs this primitive is realised by thegemlitelibrary\[Badri and Shaji,[2023](https://arxiv.org/html/2605.24058#bib.bib4)\]using warp\-level predicated fp16 reductions \(the same kernel used by DBF\[Boža and Macko,[2025b](https://arxiv.org/html/2605.24058#bib.bib8)\], BitNet\[Wang et al\.,[2023](https://arxiv.org/html/2605.24058#bib.bib35)\], and OneBit\[Xu et al\.,[2024b](https://arxiv.org/html/2605.24058#bib.bib38)\]\), giving a measured≈2×\\approx 2\\timesspeedup over an INT4 GEMM at typical adapter shapes\. Table[11](https://arxiv.org/html/2605.24058#A10.T11)summarises the arithmetic intensity\.

Table 11:Arithmetic\-intensity breakdown of the adapter matmul on a host projection of shapeN×MN\\times M\. The fp16/INT4 LoRA baselines use rankr0r\_\{0\}; LoRDBA uses binary carrier rankRRand envelope rankℓ\\ell\(the table is forℓ=1\\ell=1\)\. One fp16 multiplication and one fp16 addition count as two separate ops\. LoRDBA trades two fp16 GEMMs for two sign\-accumulation matmuls \(zero floating\-point multiplications on the inner edges\) plus three channel\-wise vector scales that fuse into the residual\-stream epilogue with the scale traffic already accounted for in the bytes column\.
## Appendix KImplementation Details and Compute

#### Hyper\-parameters\.

PTQ\-LoRDBA\.K=100K=100outer iterations; Boyd residual\-balancing with\(τ,μ\)=\(2,10\)\(\\tau,\\mu\)=\(2,10\); scale\-matched warm penaltyρ\(0\)=‖Δ​W⋆‖F2/\(N​R\+R​M\)\\rho^\{\(0\)\}=\\\|\\Delta W^\{\\star\}\\\|\_\{F\}^\{2\}/\(NR\+RM\)\. The SVD warm start of Eq\. \([7](https://arxiv.org/html/2605.24058#S3.E7)\) is used throughout\.QAT\-LoRDBA\.LoRA warm\-up: two epochs, AdamW,lr=2×10−4\\mathrm\{lr\}=2\\times 10^\{\-4\}; QAT refinement: one epoch,lr=5×10−5\\mathrm\{lr\}=5\\times 10^\{\-5\}\. Both use linear warm\-up over5%5\\%of steps, cosine to zero,κ=100\\kappa=100\.Envelope rank\.Identical schedule forℓ≥2\\ell\\geq 2, with the split\-spectrum initialisation of Algorithm[3](https://arxiv.org/html/2605.24058#alg3)\.

#### Compute budget\.

On a single NVIDIA H100 \(80 GB\), PTQ\-LoRDBA completes in∼2\{\\sim\}2minutes per model, while QAT Full requires∼11\{\\sim\}11h \(1 epochs\) forLLaMA\-2\-7Bon MetaMathQA\. Full\-run numbers are in Table[12](https://arxiv.org/html/2605.24058#A11.T12)\.

Table 12:Wall\-clock budget for adapter production on a single NVIDIA H100 \(80GB\)\.

## Appendix LAdditional Figures

![Refer to caption](https://arxiv.org/html/2605.24058v1/x3.png)Figure 3:BPW sweep onLLaMA\-3\.2\-3B\+MetaMathQA\(GSM8K,r=16r\{=\}16\)\. Four training modes are compared; PTQ saturates below22BPW while QAT variants close the gap to FP16 LoRA\.