Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning

# Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning
Source: [https://arxiv.org/html/2608.05253](https://arxiv.org/html/2608.05253)
###### Abstract

Quantized orthogonal fine\-tuning \(QOFT\) enables parameter\-efficient adaptation of low\-bit language models by learning structured activation rotations before frozen quantized weights\. However, its task\-specific updates remain constrained to linear orthogonal transformations, limiting input\-dependent nonlinear corrections\. We introduce AuroOFT, which keeps QOFT as a stable quantization\-compatible branch while attaching a zero\-start gated low\-rank nonlinear residual to each adapted linear layer\. AuroOFT maps activations into an RMS\-normalized compact latent space and uses adaptive nonlinear bases with bounded or token\-dependent gating\. The zero\-initialized up projection makes AuroOFT functionally identical to QOFT at initialization, while orthogonality remains a branch\-level stability property rather than a property of the combined nonlinear layer\. Under matched data, optimization, decoding, and parser protocols, AuroOFT improves Macro\-6 over matched QOFT by 1\.30–2\.70 points on the 1\.5B/3B Qwen2\.5 settings, exceeds QLoRA by 6\.52–10\.62 points, and saves 32\.3–44\.7% trainable parameters relative to QLoRA in representative scales\. The small exam\-style multiple\-choice math set is treated only as a protocol\-sensitivity diagnostic\. Our code is available at the anonymous repositoryhttps://anonymous\.4open\.science/r/AuroOFT\-F3FD\.

## Introduction

Parameter\-efficient fine\-tuning has become the standard route for adapting large models under realistic compute and storage budgets\(Lialinet al\.[2023](https://arxiv.org/html/2608.05253#bib.bib26)\)\. For low\-bit reasoning models, however, an effective adapter should behave like a*small but faithful correction layer*\. It must fit beside a frozen quantized foundation model, preserve useful pretrained geometry, and still express the input\-dependent corrections required by mathematical reasoning\. Figure[1](https://arxiv.org/html/2608.05253#Sx1.F1)summarizes this tension\. Additive adapters can be flexible but need not preserve geometry\. Orthogonal adapters can be stable but may restrict the local functions available for adaptation\. The central question is therefore not only whether an adapter can be made small, but what kind of correction a small adapter is allowed to express\.

![Refer to caption](https://arxiv.org/html/2608.05253v1/Figures/research_problem_ppt_latest_cropped.png)Figure 1:Research problem and challenge\. Low\-bit reasoning adapters must balance quantized stability with expressive correction capacity\.This issue is especially visible in mathematical reasoning\. A model may preserve broad pretrained knowledge and still fail because a local symbolic step, choice elimination, or numerical comparison requires an input\-dependent correction\. LoRA and QLoRA obtain efficiency by constraining updates to compact low\-rank linear subspaces\(Huet al\.[2022](https://arxiv.org/html/2608.05253#bib.bib1); Dettmerset al\.[2023](https://arxiv.org/html/2608.05253#bib.bib2)\)\. OFT, OFTv2, and QOFT instead use structure\-preserving orthogonal transformations, with QOFT applying an input\-side Cayley\-Neumann rotation before frozen NF4 weights\(Qiuet al\.[2023](https://arxiv.org/html/2608.05253#bib.bib3),[2025](https://arxiv.org/html/2608.05253#bib.bib5)\)\. This gives a stable quantized path, but the task\-specific change remains an orthogonal linear transformation\. Nonlinear low\-rank methods such as LoRAN and AuroRA show that compact adapters need not remain linear\(Liet al\.[2024a](https://arxiv.org/html/2608.05253#bib.bib10); Donget al\.[2025](https://arxiv.org/html/2608.05253#bib.bib15)\), yet they are designed around LoRA\-style additive updates rather than the geometric backbone that motivates QOFT\.

We call the resulting gap the*orthogonal expressivity bottleneck*\. QOFT is attractive because its rotations are structured, quantization\-compatible, and conservative around frozen low\-bit weights\. The same design, however, asks each adapted layer to express task change primarily through an input\-side orthogonal linear family\. For reasoning, this scope can be too narrow\. The desired correction may depend on the current token state, the latent arithmetic context, or a nonlinear interaction that is hard to represent by rotation alone\. Simply inserting a nonlinear module into the rotation path would blur the geometric meaning of QOFT, while replacing QOFT would discard the stability that made it useful\.

*Can we keep QOFT’s quantized orthogonal path as the stable backbone while giving each layer expressive nonlinear corrections?*Our answer is AuroOFT, a zero\-start nonlinear residual augmentation for quantized orthogonal fine\-tuning\. Each adapted layer keeps the QOFT branch unchanged\. Activations are rotated by the learned Cayley\-Neumann adapter and then passed through the frozen NF4 weight\. In parallel, a lower branch applies a down projection, an ANL, a zero\-initialized up projection, and a gate\. The branches meet by residual addition \(Figure[2](https://arxiv.org/html/2608.05253#Sx2.F2)\)\.

This decomposition gives AuroOFT its central principle\. It treats*orthogonality as a stable quantized backbone, nonlinearity as a zero\-start residual correction*\. The nonlinear branch expands the layer\-wise function family without modifying the quantized base weight or inserting nonlinear operations inside the Cayley\-Neumann rotation\. Zero initialization of the up projection makes the initial model exactly function\-equivalent to QOFT, while the input\-dependent residual deliberately turns the combined layer into a nonlinear mapping\. Thus, only the QOFT branch preserves orthogonal structure\. AuroOFT therefore borrows the nonlinear\-low\-rank principle behind AuroRA, but changes the augmented object\. The nonlinear map is placed beside a quantized orthogonal QOFT layer, not in place of it\.

This distinction also shapes how the method should be evaluated\. Improvements over QLoRA test whether the parameter budget is used more effectively than a standard low\-rank additive adapter\. Improvements over matched QOFT test whether the nonlinear residual supplies useful expressivity beyond the stable orthogonal carrier\. We therefore emphasize comparisons under fixed data, optimizer, decoding, and parser conditions, and treat small or protocol\-sensitive benchmarks as diagnostic rather than as standalone evidence for a structural claim\.

The broader point is that AuroOFT is not a request to abandon orthogonality, nor a claim that nonlinear adapters are universally better\. It is a more specific proposal about where expressivity should enter a quantized orthogonal adapter\. The orthogonal branch remains responsible for stable low\-bit adaptation\. The residual branch handles local, input\-dependent deviations that the rotation family may not capture\. This separation lets the method preserve a clean baseline computation at initialization, expose a controlled set of AuroOFT\-specific variables for ablation, and keep the scientific comparison focused on the added correction family\. In this sense, the method is deliberately conservative in its starting point, but less restrictive in the corrections it can learn during optimization\.

This paper contributes a compact framework for turning the above diagnosis into a testable adaptation object\.

- •A diagnosis\.The*orthogonal expressivity bottleneck*isolates the central limitation\. Input\-side rotations give QOFT a stable quantized path, yet restrict task updates to a structured linear family\.
- •An adaptation object\.AuroOFT instantiates this diagnosis as a zero\-start gated nonlinear low\-rank residual branch parallel to QOFT, with tanh, normalized spline, and enhanced dual\-ANL variants\.
- •A boundary analysis\.The formulation makes explicit what is preserved and what is not\. It covers exact initialization equivalence to QOFT, containment of the QOFT family, branch\-limited orthogonality, and non\-mergeable residual cost during inference\.
- •A matched evaluation\.Mathematical reasoning experiments align data, optimization, decoding, and parsers, while treating the small exam\-style multiple\-choice math set only as a protocol diagnostic\. The protocol uses Macro\-6 to separate structural gains over QOFT from parameter\-efficiency comparisons with QLoRA\.

## Related Work

#### Stable and quantized adaptation\.

PEFT methods differ in the adaptation object they choose\. These include bottleneck modules\(Houlsbyet al\.[2019](https://arxiv.org/html/2608.05253#bib.bib16)\), prompts\(Li and Liang[2021](https://arxiv.org/html/2608.05253#bib.bib17); Liuet al\.[2022b](https://arxiv.org/html/2608.05253#bib.bib24)\), activation scalings\(Liuet al\.[2022a](https://arxiv.org/html/2608.05253#bib.bib25)\), and low\-rank additive matrices\(Huet al\.[2022](https://arxiv.org/html/2608.05253#bib.bib1)\)\. Quantized adaptation makes this choice sharper\. GPTQ, SmoothQuant, and AWQ reduce the frozen model footprint\(Frantaret al\.[2023](https://arxiv.org/html/2608.05253#bib.bib18); Xiaoet al\.[2023](https://arxiv.org/html/2608.05253#bib.bib19); Linet al\.[2024](https://arxiv.org/html/2608.05253#bib.bib20)\), while QLoRA, LoftQ, and QA\-LoRA attach LoRA\-style updates to low\-bit weights\(Dettmerset al\.[2023](https://arxiv.org/html/2608.05253#bib.bib2); Liet al\.[2024b](https://arxiv.org/html/2608.05253#bib.bib21); Xuet al\.[2024](https://arxiv.org/html/2608.05253#bib.bib22)\)\.

Orthogonal PEFT changes the prior by learning geometry\-preserving rotations\. OFT introduces orthogonal fine\-tuning\(Qiuet al\.[2023](https://arxiv.org/html/2608.05253#bib.bib3)\), BOFT compresses rotations with butterfly factors\(Liuet al\.[2024b](https://arxiv.org/html/2608.05253#bib.bib4)\), and OFTv2/QOFT scales the idea through input\-centric Cayley\-Neumann rotations before frozen quantized weights\(Qiuet al\.[2025](https://arxiv.org/html/2608.05253#bib.bib5)\)\. These lines emphasize efficiency, low\-bit compatibility, and stability, but still leave a common gap for reasoning\. The task correction is usually a linear additive map or an input\-side orthogonal rotation\. AuroOFT keeps the quantized orthogonal carrier while adding a separate nonlinear correction object\.

#### Expressive residual adaptation\.

Expressive PEFT asks what a compact adapter can represent, not only how many parameters it adds\. LoRA analyses and variants such as AdaLoRA, DyLoRA, DoRA, VeRA, FourierFT, MiLoRA, MoSLoRA, and PeriodicLoRA enrich low\-rank updates through rank allocation, shared bases, magnitude\-direction decomposition, spectral coefficients, mixed subspaces, or staged rank growth\(Zeng and Lee[2023](https://arxiv.org/html/2608.05253#bib.bib12); Zhanget al\.[2023](https://arxiv.org/html/2608.05253#bib.bib6); Valipouret al\.[2023](https://arxiv.org/html/2608.05253#bib.bib7); Liuet al\.[2024a](https://arxiv.org/html/2608.05253#bib.bib8); Kopiczkoet al\.[2024](https://arxiv.org/html/2608.05253#bib.bib23); Gaoet al\.[2024](https://arxiv.org/html/2608.05253#bib.bib9); Wanget al\.[2025](https://arxiv.org/html/2608.05253#bib.bib14); Wuet al\.[2024](https://arxiv.org/html/2608.05253#bib.bib11); Menget al\.[2024](https://arxiv.org/html/2608.05253#bib.bib13)\)\. LoRAN and AuroRA further insert nonlinear hidden mappings, including tanh and B\-spline components, into the low\-rank path\(Liet al\.[2024a](https://arxiv.org/html/2608.05253#bib.bib10); Donget al\.[2025](https://arxiv.org/html/2608.05253#bib.bib15)\)\. These works show that compact adaptation need not be purely linear, but they are designed around LoRA\-style additive updates rather than a quantized orthogonal main branch\.

AuroOFT uses their expressivity lesson in a different role\. The nonlinear map is zero\-started, gated, and residual beside QOFT, so ablations can vary only the correction family while the orthogonal carrier remains fixed\. This distinction is important for attribution\. The proposed residual is not an alternative backbone, but a controlled way to test whether nonlinear corrections can expand QOFT without changing its quantized orthogonal carrier\. The comparison therefore follows a narrow principle\. Preserve the stable branch, vary the residual branch, and measure whether the extra expressivity improves adaptation under matched conditions\. This principle bridges prior adapter design to the AuroOFT layer formalized next\.

![Refer to caption](https://arxiv.org/html/2608.05253v1/Figures/methodology_ppt_latest_cropped.png)Figure 2:AuroOFT layer\. A stable QOFT branch is augmented by a zero\-start gated nonlinear low\-rank residual\.

## Preliminaries

#### QOFT\.

Letx∈ℝdinx\\in\\mathbb\{R\}^\{d\_\{\\rm in\}\}be the input activation of a linear layer with frozen quantized weightWq,lW\_\{q,l\}\. QOFT applies an orthogonal transformation to the activation side before the quantized linear map\.

hlQ=Dequant\(Wq,l\)⊤Rl⊤x\.h\_\{l\}^\{Q\}=\\operatorname\{Dequant\}\(W\_\{q,l\}\)^\{\\top\}R\_\{l\}^\{\\top\}x\.\(1\)The rotationRlR\_\{l\}is parameterized by a skew\-symmetric matrix through a Cayley transform\. OFTv2 replaces the inverse in the Cayley transform with a truncated Neumann series, and the implementation used in our experiments follows the common setting of block size3232and five Neumann terms\. The important point for AuroOFT is that the orthogonal adapter acts before the frozen NF4 weight\. The quantized base weight is not rotated or modified\.

#### AuroRA\-style nonlinear low\-rank mapping\.

LoRA uses a two\-projector linear update\.

h=W0​x\+B​A​x,h=W\_\{0\}x\+BAx,\(2\)which is still linear inxxfor fixedAAandBB\. AuroRA inserts an adaptive nonlinear layerσ​\(⋅\)\\sigma\(\\cdot\)between the two projectors\.

h=W0​x\+B​σ​\(A​x\)\.h=W\_\{0\}x\+B\\sigma\(Ax\)\.\(3\)Its ANL combines fixed nonlinear activations and learnable B\-spline components\. AuroOFT adopts this nonlinear\-low\-rank principle, but it does not replace a LoRA branch\. It uses the nonlinear mapping as a residual beside QOFT\.

## Method

### AuroOFT Layer

Figure[2](https://arxiv.org/html/2608.05253#Sx2.F2)shows the layer\-level construction, and Algorithm 1 summarizes the corresponding forward computation\. AuroOFT wraps an already adapted OFT/QOFT linear module rather than replacing it\. The upper branch is the stable quantized orthogonal path inherited from QOFT\. The lower branch is a nonlinear low\-rank residual evaluated on the same activation and added only at the output\.

For an adapted layerll, AuroOFT computes the following branch input and residual output\.

zl=Al​x,z~l=𝒩l​\(zl\),z\_\{l\}=A\_\{l\}x,\\qquad\\tilde\{z\}\_\{l\}=\\mathcal\{N\}\_\{l\}\(z\_\{l\}\),\(4\)hl\\displaystyle h\_\{l\}=hlQ\+Δ​hl,\\displaystyle=h\_\{l\}^\{Q\}\+\\Delta h\_\{l\},\(5\)Δ​hl\\displaystyle\\Delta h\_\{l\}=Gl​\(z~l\)​αr​Bl​Drop⁡\(ϕl​\(z~l\)\),\\displaystyle=G\_\{l\}\(\\tilde\{z\}\_\{l\}\)\\frac\{\\alpha\}\{r\}B\_\{l\}\\operatorname\{Drop\}\\\!\\left\(\\phi\_\{l\}\(\\tilde\{z\}\_\{l\}\)\\right\),wherehlQh\_\{l\}^\{Q\}is the QOFT output in Eq\.[1](https://arxiv.org/html/2608.05253#Sx3.E1),Al∈ℝr×dinA\_\{l\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\rm in\}\}is the down projection,Bl∈ℝdout×rB\_\{l\}\\in\\mathbb\{R\}^\{d\_\{\\rm out\}\\times r\}is the up projection,𝒩l\\mathcal\{N\}\_\{l\}is either the identity map or FP32 RMS normalization,ϕl\\phi\_\{l\}is an adaptive nonlinear layer in the compact rank\-rrspace,α/r\\alpha/ris the residual scaling factor, andGlG\_\{l\}is a scalar, bounded scalar, or token\-dependent gate\. This matches the implementation\. An already adapted OFT/QOFT module is wrapped, the nonlinear branch is evaluated on the same input activation, and its gated residual is added to the wrapped module output\.

This parallel placement is the defining design choice\. The nonlinear branch is not inside the Cayley\-Neumann rotation and does not change the orthogonal parameterization\. It provides an additional input\-dependent correction after the QOFT feature has been computed\. The zero\-initialized up projection ensures that this additional path starts inactive, while the gate andα/r\\alpha/rscaling regulate how the residual enters the layer during training\.

Algorithm 1AuroOFT Layer Forward Pass

### Lite ANL

The lightweight variant uses a two\-tanh hidden transformation plus a learnable per\-dimension residual scale\.

ϕlLite​\(z\)=tanh⁡\(Hl​tanh⁡\(z\)\)\+sl⊙tanh⁡\(z\),\\phi\_\{l\}^\{\\rm Lite\}\(z\)=\\tanh\(H\_\{l\}\\tanh\(z\)\)\+s\_\{l\}\\odot\\tanh\(z\),\(6\)wherez=z~lz=\\tilde\{z\}\_\{l\},Hl∈ℝr×rH\_\{l\}\\in\\mathbb\{R\}^\{r\\times r\}, andsl∈ℝrs\_\{l\}\\in\\mathbb\{R\}^\{r\}\. The code initializesHlH\_\{l\}as the identity andsls\_\{l\}to a small positive value\. This variant is computationally simple and isolates whether nonlinear hidden mixing alone is useful beyond QOFT\.

### SplineNorm ANL

The normalized spline variant follows the AuroRA motivation more closely while adapting it to the AuroOFT branch\.

z¯=tanh⁡\(RMSNormFP32⁡\(z\)\),\\bar\{z\}=\\tanh\(\\operatorname\{RMSNorm\}\_\{\\rm FP32\}\(z\)\),\(7\)ϕlSplineNorm​\(z\)=tanh⁡\(Hl​tanh⁡\(z¯\)\)\+Cl​Ψ​\(z¯\),\\phi\_\{l\}^\{\\rm SplineNorm\}\(z\)=\\tanh\(H\_\{l\}\\tanh\(\\bar\{z\}\)\)\+C\_\{l\}\\Psi\(\\bar\{z\}\),\(8\)whereℬj​\(⋅\)\\mathcal\{B\}\_\{j\}\(\\cdot\)denotes thejj\-th B\-spline basis response on the bounded interval\[−1,1\]\[\-1,1\],Ψj​\(z¯\)=∑i=1rℬj​\(z¯i\)\\Psi\_\{j\}\(\\bar\{z\}\)=\\sum\_\{i=1\}^\{r\}\\mathcal\{B\}\_\{j\}\(\\bar\{z\}\_\{i\}\)is the implementation’s rank\-dimension aggregated spline feature, andCl∈ℝr×\(M\+p\)C\_\{l\}\\in\\mathbb\{R\}^\{r\\times\(M\+p\)\}is the learnable spline coefficient matrix for grid sizeMMand orderpp\. The FP32 RMS normalization and tanh bounding are used to keep low\-dimensional activations within the spline grid\. This is an adaptation of the AuroRA ANL idea, not a direct transplantation of its LoRA theory\.

### Enhanced ANL and Gating

The enhanced branch uses an FP32\-normalized compact representation

z=RMSNormFP32⁡\(Al​x\),z=\\operatorname\{RMSNorm\}\_\{\\rm FP32\}\(A\_\{l\}x\),\(9\)and combines two nonlinear bases\.

ut\\displaystyle u\_\{t\}=tanh⁡\(Pt,l​tanh⁡z\),\\displaystyle=\\tanh\(P\_\{t,l\}\\tanh z\),\(10\)ug\\displaystyle u\_\{g\}=SiLU⁡\(Pg,l​z\)⊙tanh⁡\(Pv,l​z\)\.\\displaystyle=\\operatorname\{SiLU\}\(P\_\{g,l\}z\)\\odot\\tanh\(P\_\{v,l\}z\)\.\(11\)For the dual ANL, the final hidden output is

ϕlDual​\(z\)\\displaystyle\\phi\_\{l\}^\{\\rm Dual\}\(z\)=πt,l​ut\+πg,l​ug,\\displaystyle=\\pi\_\{t,l\}u\_\{t\}\+\\pi\_\{g,l\}u\_\{g\},\(12\)\[πt,l,πg,l\]\\displaystyle\[\\pi\_\{t,l\},\\pi\_\{g,l\}\]=softmax⁡\(ml\)\.\\displaystyle=\\operatorname\{softmax\}\(m\_\{l\}\)\.\(13\)The code supports scalar, bounded, and token\-dependent gates\.

Gl​\(z\)\\displaystyle G\_\{l\}\(z\)=ηl,\\displaystyle=\\eta\_\{l\},\(14\)Gl​\(z\)\\displaystyle G\_\{l\}\(z\)=2​σ​\(γl\),\\displaystyle=2\\sigma\(\\gamma\_\{l\}\),\(15\)Gl​\(z\)\\displaystyle G\_\{l\}\(z\)=2​σ​\(Wg,l​z\),\\displaystyle=2\\sigma\(W\_\{g,l\}z\),\(16\)corresponding respectively to the unconstrained scalar, bounded scalar, and token\-dependent gate\. In the enhanced implementation,ηl\\eta\_\{l\}is initialized to one, whileγl=0\\gamma\_\{l\}=0andWg,l=0W\_\{g,l\}=0make the bounded and token gates initialize to one\. This avoids the small\-gate start that can suppress early gradients through the zero\-initialized up projection\.

### Layer Injection and Optimization

AuroOFT supports all\-layer injection or selective targets such as query/value and query/key/value plus MLP up/down projections\. The QOFT branch and nonlinear residual use separate parameter groups, with branch\-specific learning rate and gradient clipping\. Parameters are identified by module identity rather than name, avoiding accidental capture of unrelated QOFT modules\. Disabling AuroOFT recovers the same QOFT training interface\.

### Structural Properties

###### Proposition 1\(Zero\-start equivalence\)\.

IfBl=0B\_\{l\}=0for every nonlinear branch, then the initialized AuroOFT model computes exactly the same function as the corresponding QOFT model on every input sequence\.

###### Proof\.

For each adapted layer,Bl=0B\_\{l\}=0impliesΔ​hl=0\\Delta h\_\{l\}=0in Eq\.[5](https://arxiv.org/html/2608.05253#Sx4.E5), regardless ofAlA\_\{l\},ϕl\\phi\_\{l\}, dropout, or gate values\. Hencehl=hlQh\_\{l\}=h\_\{l\}^\{Q\}at every adapted layer, and the layer\-wise equality composes through the network\. ∎

###### Proposition 2\(QOFT containment\)\.

The function family induced by AuroOFT contains the corresponding QOFT family when the nonlinear residual is inactive\.

###### Proof\.

Any QOFT model can be represented by setting the nonlinear residual to zero, for example throughBl=0B\_\{l\}=0at all adapted layers\. Thus QOFT is a subfamily of AuroOFT\. We use the up\-projection condition because it holds for all implemented gate types, including bounded and token\-dependent gates whose finite\-parameter values are positive\. ∎

#### Orthogonality boundary\.

The main branch retains QOFT’s approximate orthogonal structure, but the full AuroOFT layer adds an input\-dependent nonlinear residual and is not an orthogonal transformation\. We therefore do not claim global angle, norm, or forgetting guarantees for the combined mapping\.

#### Non\-mergeability and cost\.

BecauseΔ​hl\\Delta h\_\{l\}is nonlinear in the current activation, the residual cannot generally be merged into a static post\-training weight matrix\. Unlike linear LoRA\-style adapters, AuroOFT therefore adds inference cost from projection, nonlinear computation, gating, and up projection\.

Table 1:Pass@1 mathematical reasoning results across Qwen2\.5 model scales\.Table 2:AuroOFT ablation results on Qwen2\.5\-3B\. Only AuroOFT\-specific residual\-branch components are varied\.
#### Parameter cost\.

For the Lite variant, each adapted layer adds the following trainable parameters beyond the QOFT branch in that layer\.

r​din\+r2\+r​dout\+r\+κgrd\_\{\\rm in\}\+r^\{2\}\+rd\_\{\\rm out\}\+r\+\\kappa\_\{g\}\(17\)parameters, corresponding toAlA\_\{l\},HlH\_\{l\},BlB\_\{l\}, the Lite residual scalesls\_\{l\}, and the chosen gate\. SplineNorm replacessls\_\{l\}with the spline coefficients and therefore adds

r​din\+r2\+r​dout\+r​\(M\+p\)\+κgrd\_\{\\rm in\}\+r^\{2\}\+rd\_\{\\rm out\}\+r\(M\+p\)\+\\kappa\_\{g\}\(18\)parameters for grid sizeMM, orderpp, and the chosen gate\. For the enhanced dual\-ANL variant, the branch adds

r​din\+r​dout\+3​r2\+2\+κgrd\_\{\\rm in\}\+rd\_\{\\rm out\}\+3r^\{2\}\+2\+\\kappa\_\{g\}\(19\)parameters, where the threer2r^\{2\}terms are the tanh, SwiGLU gate, and SwiGLU value projections, the two additional parameters are the dual mixing logits, andκg\\kappa\_\{g\}is11for scalar or bounded scalar gating andrrfor token\-dependent gating\. In Eq\.[18](https://arxiv.org/html/2608.05253#Sx4.E18),r​\(M\+p\)r\(M\+p\)corresponds to the learnable spline coefficient matrix with output rankrrandM\+pM\+pbasis responses\. In all variants, the dominant additional computation scales asO​\(r​din\+r2\+r​dout\)O\(rd\_\{\\rm in\}\+r^\{2\}\+rd\_\{\\rm out\}\)per token and adapted layer, with a small spline\-basis overhead for SplineNorm\.

![Refer to caption](https://arxiv.org/html/2608.05253v1/x1.png)

Figure 4:Per\-benchmark Pass@1 differences between AuroOFT and QOFT\. Positive bars indicate gains from the nonlinear residual branch\.Table 3:Trainable\-parameter savings against QLoRA\.
![Refer to caption](https://arxiv.org/html/2608.05253v1/x2.png)
Figure 5:Proxy\-development diagnostics\.

## Experiments

### Experimental Setup

Evaluation Principle\.The key question is whether the nonlinear residual improves over QOFT under matched adaptation conditions\. Unless otherwise stated, methods share the same model, data, sample order, optimizer, length, quantization, adapter placement, decoding, and parser, so differences isolate the residual branch\.

Training Setup\.We use Qwen2\.5 models with NF4, OpenR1\-Math data, effective batch size 32, OFT block size 32, five Cayley\-Neumann terms, and length 16,384 when feasible\. A matched manifest fixes shared QOFT fields\. Only residual\-branch variables vary\.

Benchmarks and Metrics\.We evaluate AMC23, AQUA, CMATH, GaoKao, Minerva Math, OlympiadBench, and SAT Math\. Macro\-6 averages the six non\-SAT benchmarks and serves as the main aggregate\. SAT Math is retained as a protocol\-sensitive diagnostic\. We report official accuracy plus sample counts, formatting failures, timeouts, parameters, memory, time, and throughput\.

Selection Protocol\.Model selection uses a leakage\-audited proxy set drawn from OpenR1 samples outside the 50k training subset and deduplicated against final tests by normalized text hashes\. Test benchmarks do not choose rank, gates, ANL type, branch learning rate, or targets\. After freezing a configuration, matched QOFT and AuroOFT runs use the same seed policy, decoding setup, and parser\.

### Main Experimental Results

Table[1](https://arxiv.org/html/2608.05253#Sx4.T1)reports Pass@1 across Qwen2\.5 scales for the frozen baseline, QLoRA, QOFT, and AuroOFT under the same benchmark columns\.

AuroOFT shows its clearest structural gains in the 1\.5B and 3B settings\. Macro\-6 improves over matched QOFT by 1\.30, 1\.83, 0\.38, and 2\.70 points on 1\.5B\-it, 1\.5B, 3B\-it, and 3B, with non\-degraded metrics on 7/7, 6/7, 5/7, and 7/7 benchmarks\. Relative to QLoRA, AuroOFT gains 6\.52–10\.62 Macro\-6 points while using fewer trainable parameters\. Larger models remain task\-dependent, with AQUA and SAT especially sensitive to scale and protocol\.

### Further Analysis

\(I\) Ablation Study\.Table[2](https://arxiv.org/html/2608.05253#Sx4.T2)isolates AuroOFT\-specific choices while fixing the base model, quantization, QOFT branch, data, optimizer, decoding, and parser\. The rows compare no branch, Lite tanh, bounded SplineNorm, and enhanced dual tanh/SwiGLU variants\. Rank/gate rows test added residual capacity under the same evaluation protocol\.

\(II\) Diagnostic Bar\-Chart Analysis\.Figure[4](https://arxiv.org/html/2608.05253#Sx4.F4)shows per\-benchmark AuroOFT\-QOFT differences\. Gains concentrate in the 1\.5B and 3B settings, while 7B\-it is mixed and SAT remains diagnostic\. Figure[5](https://arxiv.org/html/2608.05253#Sx4.F5)reports proxy\-dev screening, where token\-gated rank\-2 tanh gives the highest proxy accuracy and Dual\-ANL balances capacity with reliability without using final tests for selection\.

\(III\) Parameter Efficiency Analysis\.AuroOFT also uses adapter parameters more efficiently than QLoRA\. Table[3](https://arxiv.org/html/2608.05253#Sx4.T3)shows savings of 8\.26M, 7\.99M, and 17\.77M parameters on 1\.5B\-it, 3B\-it, and 7B\-it, or 44\.7%, 32\.3%, and 44\.0%\. The saved budget can support more task\-specialized adapters while keeping correction capacity compact\.

\(IV\) Training Dynamics Analysis\.Figure[6](https://arxiv.org/html/2608.05253#Sx5.F6)checks optimization stability\. AuroOFT starts from the same loss as QOFT and follows a comparable matched\-training trajectory\. The 120\-step variants are diagnostics, not accuracy evidence\.

![[Uncaptioned image]](https://arxiv.org/html/2608.05253v1/x3.png)

Figure 6:Training dynamics under matched adaptation\.
![Refer to caption](https://arxiv.org/html/2608.05253v1/Figures/case_study_update_tight.png)Figure 7:Qualitative case study on a GSM8K\-style multiple\-choice problem\. The example illustrates local correction by the zero\-start nonlinear residual\.\(V\) Case Study\.Figure[7](https://arxiv.org/html/2608.05253#Sx5.F7)gives a qualitative GSM8K\-style multiple\-choice example\. The QOFT branch supplies the stable quantized backbone, while the zero\-start gated residual corrects a local algebraic error and shifts probability toward the correct option in the illustrated choice set\.

## Discussion

Orthogonality as a Backbone, Not a Constraint

AuroOFT treats orthogonality as a stable low\-bit backbone, not a global constraint\. The QOFT branch preserves the quantization\-compatible activation rotation, while the nonlinear residual handles corrections that a rotation may express inefficiently\. Only the main branch is orthogonal\. The full mapping is not\.

Why a Parallel Nonlinear Residual?

AuroRA and LoRAN motivate nonlinear low\-rank mappings for LoRA\-style updates\. AuroOFT places the nonlinear map beside QOFT\. The wrapped OFT/QOFT module gives the backbone output, while a zero\-start gated residual adds correction without disturbing the Cayley\-Neumann rotation or frozen NF4 path\. Its effect is isolated by initialization, gating, and ablation\.

Implementation Choices for Stable Expressivity

The implementation is built around controlled expressivity\. Residual parameters are separated from QOFT, injection can be selective, and Lite, SplineNorm, and Enhanced ANL variants share one wrapper\. RMS\-normalized activations, bounded/token gates, and target choices regulate capacity, while ablations vary basis, normalization, rank, and gating with the carrier fixed\.

How to Interpret the Empirical Evidence

Matched evidence keeps the interpretation grounded\. Comparisons with QOFT isolate structural capacity beyond the orthogonal carrier, while comparisons with QLoRA test parameter efficiency under the same model scale\. Macro\-6 and per\-benchmark matched results support method\-level interpretation\. SAT Math is retained only to expose protocol sensitivity, not to serve as a standalone structural claim\.

## Limitations

Structural boundary\.AuroOFT uses orthogonality as a stable carrier, but only the QOFT branch remains orthogonal\. The input\-dependent residual cannot generally be merged into frozen quantized weights, so AuroOFT should be viewed as a controlled expressivity extension rather than a zero\-overhead replacement for QOFT\.

Evidence boundary\.Claims for AuroOFT require matched base models, data order, training recipes, decoding, and parsers\. SAT Math is useful but small and prompt\-sensitive\. Broader validation should test more domains, scales, seeds, and latency\-sensitive deployments\.

## Conclusion

We presented AuroOFT, a zero\-start nonlinear residual augmentation for quantized orthogonal fine\-tuning\. It keeps QOFT as the stable low\-bit backbone and adds gated low\-rank nonlinear correction through a parallel residual path\. This design preserves the QOFT starting function, keeps the orthogonal carrier fixed, and makes added capacity attributable\. Matched results show improved adaptation over QOFT in the 1\.5B and 3B settings, and stronger Macro\-6 accuracy than QLoRA with fewer trainable parameters\.

## References

- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p2.1),[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- H\. Dong, W\. Zhu, G\. Song, and L\. Wang \(2025\)AuroRA: breaking low\-rank bottleneck of LoRA with nonlinear mapping\.InAdvances in Neural Information Processing Systems,Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p2.1),[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2023\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.InInternational Conference on Learning Representations,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- Z\. Gao, Q\. Wang, A\. Chen, Z\. Liu, B\. Wu, L\. Chen, and J\. Li \(2024\)Parameter\-efficient fine\-tuning with discrete fourier transform\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2405.03003)Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly \(2019\)Parameter\-efficient transfer learning for NLP\.InInternational Conference on Machine Learning,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p2.1),[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- D\. J\. Kopiczko, T\. Blankevoort, and Y\. M\. Asano \(2024\)VeRA: vector\-based random matrix adaptation\.InInternational Conference on Learning Representations,Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- X\. L\. Li and P\. Liang \(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, L\. Song, and H\. Hou \(2024a\)LoRAN: improved low\-rank adaptation by a non\-linear transformation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,External Links:[Link](https://aclanthology.org/2024.findings-emnlp.177/)Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p2.1),[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, Y\. Yu, C\. Liang, P\. He, N\. Karampatziakis, W\. Chen, and T\. Zhao \(2024b\)LoftQ: LoRA\-fine\-tuning\-aware quantization for large language models\.InInternational Conference on Learning Representations,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- V\. Lialin, V\. Deshpande, X\. Yao, and A\. Rumshisky \(2023\)Scaling down to scale up: a guide to parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.15647\.Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)AWQ: activation\-aware weight quantization for LLM compression and acceleration\.InProceedings of Machine Learning and Systems,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- H\. Liu, D\. Tam, M\. Muqeeth, J\. Mohta, T\. Huang, M\. Bansal, and C\. Raffel \(2022a\)Few\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.InAdvances in Neural Information Processing Systems,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024a\)DoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning,Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- W\. Liu, Z\. Qiu, Y\. Feng, Y\. Xiu, Y\. Xue, L\. Yu, H\. Feng, Z\. Liu, J\. Heo, S\. Peng, Y\. Wen, M\. J\. Black, A\. Weller, and B\. Schölkopf \(2024b\)Parameter\-efficient orthogonal finetuning via butterfly factorization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=7NzgkEdGyr)Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p2.1)\.
- X\. Liu, K\. Ji, Y\. Fu, W\. L\. Tam, Z\. Du, Z\. Yang, and J\. Tang \(2022b\)P\-tuning v2: prompt tuning can be comparable to fine\-tuning across scales and tasks\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- X\. Meng, D\. Dai, W\. Luo, Z\. Yang, S\. Wu, X\. Wang, P\. Wang, Q\. Dong, L\. Chen, and Z\. Sui \(2024\)PeriodicLoRA: breaking the low\-rank bottleneck in LoRA optimization\.arXiv preprint arXiv:2402\.16141\.External Links:[Link](https://arxiv.org/abs/2402.16141)Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- Z\. Qiu, W\. Liu, H\. Feng, Y\. Xue, Y\. Feng, Z\. Liu, D\. Zhang, A\. Weller, and B\. Schölkopf \(2023\)Controlling text\-to\-image diffusion by orthogonal finetuning\.InAdvances in Neural Information Processing Systems,Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p2.1),[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p2.1)\.
- Z\. Qiu, W\. Liu, A\. Weller, and B\. Schölkopf \(2025\)Orthogonal finetuning made scalable\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://aclanthology.org/2025.emnlp-main.1627/)Cited by:[Introduction](https://arxiv.org/html/2608.05253#Sx1.p2.1),[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p2.1)\.
- M\. Valipour, M\. Rezagholizadeh, I\. Kobyzev, and A\. Ghodsi \(2023\)DyLoRA: parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- H\. Wang, Y\. Li, S\. Wang, G\. Chen, and Y\. Chen \(2025\)MiLoRA: harnessing minor singular components for parameter\-efficient LLM finetuning\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies,External Links:[Link](https://aclanthology.org/2025.naacl-long.248/)Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- T\. Wu, J\. Wang, Z\. Zhao, and N\. Wong \(2024\)Mixture\-of\-subspaces in low\-rank adaptation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://aclanthology.org/2024.emnlp-main.450/)Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.InInternational Conference on Machine Learning,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- Y\. Xu, L\. Xie, X\. Gu, X\. Chen, H\. Chang, H\. Zhang, Z\. Chen, X\. Zhang, and Q\. Tian \(2024\)QA\-LoRA: quantization\-aware low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[Stable and quantized adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px1.p1.1)\.
- Y\. Zeng and K\. Lee \(2023\)The expressive power of low\-rank adaptation\.InWorkshop on Optimization for Machine Learning,External Links:[Link](https://openreview.net/forum?id=likXVjmh3E)Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[Expressive residual adaptation\.](https://arxiv.org/html/2608.05253#Sx2.SS0.SSS0.Px2.p1.1)\.