通过ℓp正则化在大语言模型中实现低秩适应的自动秩分配

# Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via ℓ_𝑝 Regularization
Source: [https://arxiv.org/html/2609.28998](https://arxiv.org/html/2609.28998)
Chuanyang ZhengYik\-Chung WuYihang Gao††thanks:\*Corresponding author: Yihang Gao \(gaoyihang@hkbu\.edu\.hk\)\.

###### Abstract

Low\-rank adaptation \(LoRA\) has become a popular parameter\-efficient fine\-tuning method for large language models\. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency\. Existing adaptive\-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective\. In this work, we proposeℓp\\ell\_\{p\}\-LoRA, a principled rank\-allocation method based onℓp\\ell\_\{p\}regularization with0<p<10<p<1, which is a classical sparsity\-inducing technique in signal processing and statistics\. Specifically, we regularize the energy of each rank\-one LoRA component, encouraging redundant components to vanish while preserving important ones\. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two\-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components\. Experiments on natural language understanding and question\-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines\.

###### Index Terms:

Low\-rank adaptation, Adaptive rank allocation,

ℓp\\ell\_\{p\}regularization, Implicit thresholding

††address:1National University of Singapore, Singapore2Huawei, Hong Kong
3The University of Hong Kong, Hong Kong4Hong Kong Baptist University, Hong Kong
## 1Introduction

Large language models \(LLMs\) have achieved remarkable performance across a wide range of downstream tasks\[[1](https://arxiv.org/html/2609.28998#bib.bib11)\]\. However, as model sizes continue to grow, fully fine\-tuning all model parameters becomes increasingly expensive in both computation and memory\. Parameter\-efficient fine\-tuning \(PEFT\) has therefore emerged as an effective approach for adapting pretrained models to downstream tasks with substantially fewer trainable parameters\[[9](https://arxiv.org/html/2609.28998#bib.bib4)\]\. Among various PEFT methods, low\-rank adaptation \(LoRA\)\[[10](https://arxiv.org/html/2609.28998#bib.bib1)\]has become one of the most widely adopted approaches\. LoRA freezes the pretrained weights and represents each trainable weight update by a low\-rank factorization, thereby significantly reducing the number of trainable parameters\.

An important question in LoRA is how to determine the rank of each adaptation matrix\[[18](https://arxiv.org/html/2609.28998#bib.bib2)\]\. Standard LoRA typically assigns a pre\-specified rank uniformly across different layers and modules\. However, different components of a pretrained model may require different adaptation capacities for a given downstream task, making a uniform rank allocation potentially inefficient\. From a resource\-allocation perspective, given a total rank budget determined by computational or memory constraints, the problem is therefore how to distribute the available ranks across different LoRA matrices\.

Existing adaptive\-rank methods have primarily addressed this problem by designing importance measures for individual LoRA components\. The pioneering work AdaLoRA\[[18](https://arxiv.org/html/2609.28998#bib.bib2)\], for example, dynamically allocates the rank budget according to importance scores constructed from information collected during training\. Subsequent methods further refine such criteria by incorporating additional gradient or layer\-wise information\[[6](https://arxiv.org/html/2609.28998#bib.bib12),[13](https://arxiv.org/html/2609.28998#bib.bib13)\]\. These approaches determine rank allocation through designed importance scores rather than deriving the pruning criterion directly from an explicit optimization formulation\. Another line of work considers probabilistic or Bayesian formulations for adaptive rank determination\[[16](https://arxiv.org/html/2609.28998#bib.bib14),[7](https://arxiv.org/html/2609.28998#bib.bib15)\]\. Such approaches provide an alternative perspective, but typically introduce additional modeling and inference mechanisms beyond standard LoRA training\.

In this work, we seek a principled, mathematically motivated rank\-allocation method for LoRA\. We observe that adaptive rank allocation can be naturally interpreted as a structured sparsity problem: each rank\-one LoRA component can be viewed as a candidate component to be either retained or removed\. Motivated by the sparsity\-inducing properties ofℓp\\ell\_\{p\}regularization with0<p<10<p<1, which has been widely studied in signal processing and statistical modeling\[[3](https://arxiv.org/html/2609.28998#bib.bib3)\], we formulate rank selection by imposing anℓp\\ell\_\{p\}penalty on the energy of rank\-one LoRA components\. Under this formulation, redundant components are encouraged to vanish, while informative components tend to be retained\.

## 2Background

### 2\.1Low\-Rank Adaptation

Consider LoRA\-based fine\-tuning\[[10](https://arxiv.org/html/2609.28998#bib.bib1)\]for a multi\-layer neural networkϕ⁡\(⋅,Θ\)\\phi\(\\cdot;\\Theta\), such as a transformer\-based LLM\[[14](https://arxiv.org/html/2609.28998#bib.bib5)\]\. LetΘ:=\{𝑾\(1\),𝑾\(2\),⋯,𝑾\(L\)\}\\Theta:=\\\{\\bm\{W\}^\{\(1\)\},\\bm\{W\}^\{\(2\)\},\\cdots,\\bm\{W\}^\{\(L\)\}\\\}denote the collection of parameter matrices equipped with LoRA\. For each parameter matrix, the fine\-tuned parameter is represented as

𝑾ft=𝑾pt\+Δ​𝑾:=𝑾pt\+𝑩​𝑨,\\bm\{W\}\_\{\\text\{ft\}\}=\\bm\{W\}\_\{\\text\{pt\}\}\+\\Delta\\bm\{W\}:=\\bm\{W\}\_\{\\text\{pt\}\}\+\\bm\{B\}\\bm\{A\},\(1\)where𝑾pt∈ℝm×n\\bm\{W\}\_\{\\mathrm\{pt\}\}\\in\\mathbb\{R\}^\{m\\times n\}and𝑾ft∈ℝm×n\\bm\{W\}\_\{\\mathrm\{ft\}\}\\in\\mathbb\{R\}^\{m\\times n\}denote the pretrained and fine\-tuned parameter matrices, respectively, andΔ​𝑾∈ℝm×n\\Delta\\bm\{W\}\\in\\mathbb\{R\}^\{m\\times n\}is the corresponding adaptation matrix\. We take the trainable LoRA factors𝑨∈ℝr×n\\bm\{A\}\\in\\mathbb\{R\}^\{r\\times n\}and𝑩∈ℝm×r\\bm\{B\}\\in\\mathbb\{R\}^\{m\\times r\}, wherer≪min⁡\{m,n\}r\\ll\\min\\\{m,n\\\}\. For notational simplicity, throughout the paper we present the formulation and derivation for a single LoRA\-equipped parameter matrix and suppress the layer or module index\. The same construction can be applied to all parameter matrices equipped with LoRA\.

The rankrrdetermines the number of rank\-one components available for adapting each parameter matrix\. Since different layers and modules can exhibit different levels of relevance to a downstream task, assigning an identical rank to all LoRA modules may lead to inefficient use of the available rank budget\[[18](https://arxiv.org/html/2609.28998#bib.bib2)\]\. Some parameter matrices may be sufficiently adapted using only a few rank\-one components, whereas others may require a larger adaptation capacity\. At the same time, unnecessarily large ranks increase the number of trainable parameters and may introduce redundant degrees of freedom\. These observations motivate adaptive rank allocation, where different parameter matrices are assigned different effective ranks according to their contribution to the downstream task\[[13](https://arxiv.org/html/2609.28998#bib.bib13),[6](https://arxiv.org/html/2609.28998#bib.bib12)\]\.

### 2\.2ℓp\\ell\_\{p\}Regularization

Sparsity\-promoting regularization has been widely studied in signal processing, sparse recovery, and statistical estimation\[[3](https://arxiv.org/html/2609.28998#bib.bib3)\]\. A representative formulation is

min𝒙∈ℝd⁡f⁡\(𝒙\)\+λ​‖𝒙‖pp=f⁡\(𝒙\)\+λ​∑i=1d\|xi\|p,0<p≤1,\\min\_\{\\bm\{x\}\\in\\mathbb\{R\}^\{d\}\}f\(\\bm\{x\}\)\+\\lambda\\\|\\bm\{x\}\\\|\_\{p\}^\{p\}=f\(\\bm\{x\}\)\+\\lambda\\sum\_\{i=1\}^\{d\}\|x\_\{i\}\|^\{p\},\\quad 0<p\\leq 1,\(2\)whereffdenotes the data\-fitting objective andλ\>0\\lambda\>0controls the regularization parameter\. The classicalℓ1\\ell\_\{1\}penalty provides a convex surrogate for theℓ0\\ell\_\{0\}sparsity measure, while the nonconvexℓp\\ell\_\{p\}penalty with0<p<10<p<1more closely approximatesℓ0\\ell\_\{0\}and generally induces stronger sparsity\[[2](https://arxiv.org/html/2609.28998#bib.bib16),[3](https://arxiv.org/html/2609.28998#bib.bib3)\]\.

A key property ofℓp\\ell\_\{p\}regularization is its thresholding behavior\. Given a reference point𝒙~\\tilde\{\\bm\{x\}\}, the corresponding proximal subproblem takes the form

min𝒙⁡12​‖𝒙−𝒙~‖22\+λ​‖𝒙‖pp\.\\min\_\{\\bm\{x\}\}\\frac\{1\}\{2\}\\\|\\bm\{x\}\-\\tilde\{\\bm\{x\}\}\\\|\_\{2\}^\{2\}\+\\lambda\\\|\\bm\{x\}\\\|\_\{p\}^\{p\}\.\(3\)For each coordinate, the problem reduces to a scalar minimization in which sufficiently small entries of𝒙~\\tilde\{\\bm\{x\}\}are mapped exactly to zero, whereas sufficiently large entries are retained as nonzero\. This zero–nonzero transition acts as a thresholding rule and provides a natural mechanism for variable selection and sparse recovery\.

## 3Method

### 3\.1Motivation and Problem Formulation

As shown in \([1](https://arxiv.org/html/2609.28998#S2.E1)\), the trainable parameters in LoRA fine\-tuning are the factor matrices𝑨,𝑩\{\\bm\{A\},\\bm\{B\}\}\. Accordingly, the training problem can be written abstractly as

min𝑨,𝑩⁡ℒ⁡\(𝑩​𝑨\),\\min\_\{\\bm\{A\},\\bm\{B\}\}\\mathcal\{L\}\(\\bm\{B\}\\bm\{A\}\),\(4\)whereℒ\\mathcal\{L\}denotes the training loss\. Its specific form depends on the underlying model, downstream task, and training data\.

The LoRA update admits the rank\-one decomposition

𝑩𝑨=∑i=1r𝑩:,i𝑨i,:\.\\bm\{B\}\\bm\{A\}=\\sum\_\{i=1\}^\{r\}\\bm\{B\}\_\{:,i\}\\bm\{A\}\_\{i,:\}\.\(5\)We define the energy of theii\-th rank\-one component by

ei:=∥𝑩:,i𝑨i,:∥F=∥𝑩:,i∥2∥𝑨i,:∥2,e\_\{i\}:=\\\|\\bm\{B\}\_\{:,i\}\\bm\{A\}\_\{i,:\}\\\|\_\{\\mathrm\{F\}\}=\\\|\\bm\{B\}\_\{:,i\}\\\|\_\{2\}\\\|\\bm\{A\}\_\{i,:\}\\\|\_\{2\},\(6\)and collect the component energies into

Φ⁡\(𝑨,𝑩\):=\[e1,e2,…,er\]⊤∈ℝr\.\\Phi\(\\bm\{A\},\\bm\{B\}\):=\[e\_\{1\},e\_\{2\},\\ldots,e\_\{r\}\]^\{\\top\}\\in\\mathbb\{R\}^\{r\}\.\(7\)A component is active wheneverei≠0e\_\{i\}\\neq 0\. Hence,‖Φ⁡\(𝑨,𝑩\)‖0\\\|\\Phi\(\\bm\{A\},\\bm\{B\}\)\\\|\_\{0\}counts the number of active rank\-one components and provides a natural notion of the effective LoRA rank\. In particular,rank⁡\(𝑩​𝑨\)≤‖Φ⁡\(𝑨,𝑩\)‖0\\operatorname\{rank\}\(\\bm\{B\}\\bm\{A\}\)\\leq\\\|\\Phi\(\\bm\{A\},\\bm\{B\}\)\\\|\_\{0\}\. Given a rank budgetbb, rank allocation can therefore be formulated as the cardinality\-constrained problem

min𝑨,𝑩ℒ⁡\(𝑩​𝑨\),s\.t\.‖Φ⁡\(𝑨,𝑩\)‖0≤b\.\\min\_\{\\bm\{A\},\\bm\{B\}\}\\quad\\mathcal\{L\}\(\\bm\{B\}\\bm\{A\}\),\\quad\\mathrm\{s\.t\.\}\\quad\\\|\\Phi\(\\bm\{A\},\\bm\{B\}\)\\\|\_\{0\}\\leq b\.\(8\)For multiple LoRA\-equipped parameter matrices,Φ\\Phican be understood as the concatenation of their component\-energy vectors, and the same formulation applies to a global rank budget\.

Problem \([8](https://arxiv.org/html/2609.28998#S3.E8)\) is difficult to optimize directly due to the combinatorial and discontinuous property of theℓ0\\ell\_\{0\}constraint\. A natural alternative is to replace the hard cardinality constraint by a sparsity\-promoting penalty\. Moreover, for any fixed vector𝒙\\bm\{x\},limp→0\+‖𝒙‖pp=‖𝒙‖0,\\lim\_\{p\\rightarrow 0^\{\+\}\}\\\|\\bm\{x\}\\\|\_\{p\}^\{p\}=\\\|\\bm\{x\}\\\|\_\{0\},which motivates the use of the nonconvexℓp\\ell\_\{p\}penalty with0<p<10<p<1as a continuous surrogate for theℓ0\\ell\_\{0\}measure\[[3](https://arxiv.org/html/2609.28998#bib.bib3)\]\. We therefore consider the followingℓp\\ell\_\{p\}\-regularized LoRA problem:

min𝑨,𝑩ℒ⁡\(𝑩​𝑨\)\+λ​‖Φ⁡\(𝑨,𝑩\)‖pp,\\min\_\{\\bm\{A\},\\bm\{B\}\}\\quad\\mathcal\{L\}\(\\bm\{B\}\\bm\{A\}\)\+\\lambda\\\|\\Phi\(\\bm\{A\},\\bm\{B\}\)\\\|\_\{p\}^\{p\},\(9\)or equivalently,

min𝑨,𝑩ℒ\(𝑩𝑨\)\+λ∑i=1r\(∥𝑩:,i∥2∥𝑨i,:∥2\)p,\\min\_\{\\bm\{A\},\\bm\{B\}\}\\quad\\mathcal\{L\}\(\\bm\{B\}\\bm\{A\}\)\+\\lambda\\sum\_\{i=1\}^\{r\}\\left\(\\\|\\bm\{B\}\_\{:,i\}\\\|\_\{2\}\\\|\\bm\{A\}\_\{i,:\}\\\|\_\{2\}\\right\)^\{p\},\(10\)whereλ\>0\\lambda\>0is the regularization parameter and0<p<10<p<1\. The resulting penalty acts directly on entire rank\-one components rather than individual entries of𝑨\\bm\{A\}and𝑩\\bm\{B\}, thereby promoting sparsity at the rank\-component level\.

### 3\.2Algorithm Design: Implicit Thresholding

We next develop a tractable procedure based on theℓp\\ell\_\{p\}\-regularized formulation in \([10](https://arxiv.org/html/2609.28998#S3.E10)\)\. Let\(𝑨k,𝑩k\)\(\\bm\{A\}\_\{k\},\\bm\{B\}\_\{k\}\)denote the current LoRA factors at iterationkk\. Directly handling the training lossℒ⁡\(𝑩​𝑨\)\\mathcal\{L\}\(\\bm\{B\}\\bm\{A\}\)together with the nonconvexℓp\\ell\_\{p\}penalty is generally difficult\. We therefore construct a local quadratic model of the training loss around\(𝑨k,𝑩k\)\(\\bm\{A\}\_\{k\},\\bm\{B\}\_\{k\}\)\. The resulting local model is

ℒ⁡\(𝑩​𝑨\)≈ℒ⁡\(𝑩k​𝑨k\)\+⟨∇𝑨ℒk,𝑨−𝑨k⟩\+⟨∇𝑩ℒk,𝑩−𝑩k⟩\+M2​‖𝑨−𝑨k‖F2\+M2​‖𝑩−𝑩k‖F2,\\begin\{split\}\\mathcal\{L\}\(\\bm\{B\}\\bm\{A\}\)&\\approx\\mathcal\{L\}\(\\bm\{B\}\_\{k\}\\bm\{A\}\_\{k\}\)\+\\left\\langle\\nabla\_\{\\bm\{A\}\}\\mathcal\{L\}\_\{k\},\\bm\{A\}\-\\bm\{A\}\_\{k\}\\right\\rangle\\\\ &\\quad\+\\left\\langle\\nabla\_\{\\bm\{B\}\}\\mathcal\{L\}\_\{k\},\\bm\{B\}\-\\bm\{B\}\_\{k\}\\right\\rangle\\\\ &\\quad\+\\frac\{M\}\{2\}\\left\\\|\\bm\{A\}\-\\bm\{A\}\_\{k\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\frac\{M\}\{2\}\\left\\\|\\bm\{B\}\-\\bm\{B\}\_\{k\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},\\end\{split\}\(11\)where∇𝑨ℒk:=∇𝑨ℒ​\(𝑩k​𝑨k\),∇𝑩ℒk:=∇𝑩ℒ​\(𝑩k​𝑨k\)\\nabla\_\{\\bm\{A\}\}\\mathcal\{L\}\_\{k\}:=\\nabla\_\{\\bm\{A\}\}\\mathcal\{L\}\(\\bm\{B\}\_\{k\}\\bm\{A\}\_\{k\}\),~\\nabla\_\{\\bm\{B\}\}\\mathcal\{L\}\_\{k\}:=\\nabla\_\{\\bm\{B\}\}\\mathcal\{L\}\(\\bm\{B\}\_\{k\}\\bm\{A\}\_\{k\}\)denote the gradients of the objective with respect to two factors\. Substituting \([11](https://arxiv.org/html/2609.28998#S3.E11)\) into \([10](https://arxiv.org/html/2609.28998#S3.E10)\) gives the local regularized subproblem

min𝑨,𝑩\\displaystyle\\min\_\{\\bm\{A\},\\bm\{B\}\}⟨∇𝑨ℒk,𝑨−𝑨k⟩\+⟨∇𝑩ℒk,𝑩−𝑩k⟩\+M2​‖𝑨−𝑨k‖F2\\displaystyle\\left\\langle\\nabla\_\{\\bm\{A\}\}\\mathcal\{L\}\_\{k\},\\bm\{A\}\-\\bm\{A\}\_\{k\}\\right\\rangle\+\\left\\langle\\nabla\_\{\\bm\{B\}\}\\mathcal\{L\}\_\{k\},\\bm\{B\}\-\\bm\{B\}\_\{k\}\\right\\rangle\+\\frac\{M\}\{2\}\\left\\\|\\bm\{A\}\-\\bm\{A\}\_\{k\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\(12\)\+M2‖𝑩−𝑩k‖F2\+λ∑i=1r\(∥𝑩:,i∥2∥𝑨i,:∥2\)p\.\\displaystyle\+\\frac\{M\}\{2\}\\left\\\|\\bm\{B\}\-\\bm\{B\}\_\{k\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\lambda\\sum\_\{i=1\}^\{r\}\\left\(\\\|\\bm\{B\}\_\{:,i\}\\\|\_\{2\}\\\|\\bm\{A\}\_\{i,:\}\\\|\_\{2\}\\right\)^\{p\}\.
Define the nominal gradient steps

𝑨~:=𝑨k−1M​∇𝑨ℒk,𝑩~:=𝑩k−1M​∇𝑩ℒk\.\\widetilde\{\\bm\{A\}\}:=\\bm\{A\}\_\{k\}\-\\frac\{1\}\{M\}\\nabla\_\{\\bm\{A\}\}\\mathcal\{L\}\_\{k\},\\qquad\\widetilde\{\\bm\{B\}\}:=\\bm\{B\}\_\{k\}\-\\frac\{1\}\{M\}\\nabla\_\{\\bm\{B\}\}\\mathcal\{L\}\_\{k\}\.\(13\)By completing the square and removing terms independent of\(𝑨,𝑩\)\(\\bm\{A\},\\bm\{B\}\), problem \([12](https://arxiv.org/html/2609.28998#S3.E12)\) is equivalently written as

min𝑨,𝑩M2\(‖𝑨−𝑨~‖F2\+‖𝑩−𝑩~‖F2\)\+λ∑i=1r\(∥𝑩:,i∥2∥𝑨i,:∥2\)p\.\\min\_\{\\bm\{A\},\\bm\{B\}\}\\frac\{M\}\{2\}\\left\(\\left\\\|\\bm\{A\}\-\\widetilde\{\\bm\{A\}\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\left\\\|\\bm\{B\}\-\\widetilde\{\\bm\{B\}\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\)\+\\lambda\\sum\_\{i=1\}^\{r\}\\left\(\\\|\\bm\{B\}\_\{:,i\}\\\|\_\{2\}\\\|\\bm\{A\}\_\{i,:\}\\\|\_\{2\}\\right\)^\{p\}\.\(14\)Different from the classicalℓp\\ell\_\{p\}proximal problem in \([3](https://arxiv.org/html/2609.28998#S2.E3)\), the regularization term in \([14](https://arxiv.org/html/2609.28998#S3.E14)\) is imposed on the energy of each rank\-one component,∥𝑩:,i∥2∥𝑨i,:∥2\\\|\\bm\{B\}\_\{:,i\}\\\|\_\{2\}\\\|\\bm\{A\}\_\{i,:\}\\\|\_\{2\}\. Consequently, the two LoRA factors are coupled within each component, and the standard coordinate\-wise thresholding rules forℓp\\ell\_\{p\}regularization cannot be directly applied\. Nevertheless, we show below that this coupled problem still admits a tractable reduction and gives rise to an implicit thresholding rule for rank\-one LoRA components\.

The key observation is that problem \([14](https://arxiv.org/html/2609.28998#S3.E14)\) is separable across the rank\-one components\. For eachi∈\{1,…,r\}i\\in\\\{1,\\ldots,r\\\}, define𝒂i:=𝑨i,:,𝒃i:=𝑩:,i,\\bm\{a\}\_\{i\}:=\\bm\{A\}\_\{i,:\},~\\bm\{b\}\_\{i\}:=\\bm\{B\}\_\{:,i\},and similarly𝒂~i:=𝑨~i,:,𝒃~i:=𝑩~:,i\.\\widetilde\{\\bm\{a\}\}\_\{i\}:=\\widetilde\{\\bm\{A\}\}\_\{i,:\},~\\widetilde\{\\bm\{b\}\}\_\{i\}:=\\widetilde\{\\bm\{B\}\}\_\{:,i\}\.Then \([14](https://arxiv.org/html/2609.28998#S3.E14)\) can be decomposed intorrindependent subproblems of the form

min𝒂i,𝒃i⁡M2​\(‖𝒂i−𝒂~i‖22\+‖𝒃i−𝒃~i‖22\)\+λ​\(‖𝒂i‖2​‖𝒃i‖2\)p\.\\min\_\{\\bm\{a\}\_\{i\},\\bm\{b\}\_\{i\}\}\\frac\{M\}\{2\}\\left\(\\left\\\|\\bm\{a\}\_\{i\}\-\\widetilde\{\\bm\{a\}\}\_\{i\}\\right\\\|\_\{2\}^\{2\}\+\\left\\\|\\bm\{b\}\_\{i\}\-\\widetilde\{\\bm\{b\}\}\_\{i\}\\right\\\|\_\{2\}^\{2\}\\right\)\+\\lambda\\left\(\\left\\\|\\bm\{a\}\_\{i\}\\right\\\|\_\{2\}\\left\\\|\\bm\{b\}\_\{i\}\\right\\\|\_\{2\}\\right\)^\{p\}\.\(15\)
Solving \([15](https://arxiv.org/html/2609.28998#S3.E15)\) directly is still challenging, we observe that its dimensionality can be further substantially reduced and the resulting two\-dimensional problem becomes solvable\. In particular, for any fixed magnitudes‖𝒂i‖2\\\|\\bm\{a\}\_\{i\}\\\|\_\{2\}and‖𝒃i‖2\\\|\\bm\{b\}\_\{i\}\\\|\_\{2\}, the quadratic terms are minimized when𝒂i\\bm\{a\}\_\{i\}and𝒃i\\bm\{b\}\_\{i\}are aligned with their corresponding nominal vectors\. Hence, an optimal solution must be written as

𝒂i=αi​𝒂~i,𝒃i=βi​𝒃~i,0≤αi,βi≤1\.\\bm\{a\}\_\{i\}=\\alpha\_\{i\}\\widetilde\{\\bm\{a\}\}\_\{i\},\\quad\\bm\{b\}\_\{i\}=\\beta\_\{i\}\\widetilde\{\\bm\{b\}\}\_\{i\},\\quad 0\\leq\\alpha\_\{i\},\\beta\_\{i\}\\leq 1\.\(16\)Letsi:=‖𝒂~i‖2,ti:=‖𝒃~i‖2\.s\_\{i\}:=\\\|\\widetilde\{\\bm\{a\}\}\_\{i\}\\\|\_\{2\},~t\_\{i\}:=\\\|\\widetilde\{\\bm\{b\}\}\_\{i\}\\\|\_\{2\}\.Substituting \([16](https://arxiv.org/html/2609.28998#S3.E16)\) into \([15](https://arxiv.org/html/2609.28998#S3.E15)\) reduces the vector optimization problem to the following two\-dimensional problem:

min0≤α,β≤1⁡fi​\(α,β\),\\min\_\{0\\leq\\alpha,\\beta\\leq 1\}f\_\{i\}\(\\alpha,\\beta\),\(17\)where

fi​\(α,β\):=M2​\[si2​\(α−1\)2\+ti2​\(β−1\)2\]\+λ​\(si​ti​α​β\)p\.f\_\{i\}\(\\alpha,\\beta\):=\\frac\{M\}\{2\}\\left\[s\_\{i\}^\{2\}\(\\alpha\-1\)^\{2\}\+t\_\{i\}^\{2\}\(\\beta\-1\)^\{2\}\\right\]\+\\lambda\(s\_\{i\}t\_\{i\}\\alpha\\beta\)^\{p\}\.\(18\)Therefore, the thresholding decision with\(αi,βi\)\(\\alpha\_\{i\},\\beta\_\{i\}\)for each rank\-one component is obtained by solving the t wo\-dimensional problem \([17](https://arxiv.org/html/2609.28998#S3.E17)\)\.

We next briefly characterize the solution of \([17](https://arxiv.org/html/2609.28998#S3.E17)\)\. For a nonzero stationary point withα,β\>0\\alpha,\\beta\>0, the first\-order optimality conditions are

M⁡\(α−1\)​si2\+p​λ​\(β​si​ti\)p​αp−1=0,M⁡\(β−1\)​ti2\+p​λ​\(α​si​ti\)p​βp−1=0\.\\begin\{split\}M\(\\alpha\-1\)s\_\{i\}^\{2\}\+p\\lambda\(\\beta s\_\{i\}t\_\{i\}\)^\{p\}\\alpha^\{p\-1\}=0,\\\\ M\(\\beta\-1\)t\_\{i\}^\{2\}\+p\\lambda\(\\alpha s\_\{i\}t\_\{i\}\)^\{p\}\\beta^\{p\-1\}=0\.\\end\{split\}\(19\)Eliminating one variable from the above system gives

β=1±1−4​α​\(1−α\)​si2/ti22\.\\beta=\\frac\{1\\pm\\sqrt\{1\-4\\alpha\(1\-\\alpha\)s\_\{i\}^\{2\}/t\_\{i\}^\{2\}\}\}\{2\}\.\(20\)Substituting \([20](https://arxiv.org/html/2609.28998#S3.E20)\) back into either first\-order condition reduces the original two\-variable system to a one\-dimensional root\-finding problem inα\\alpha, which can be efficiently solved using standard numerical methods, such as the bisection method and Newton’s method\.

The above characterization only covers nonzero stationary points\. Since theℓp\\ell\_\{p\}penalty with0<p<10<p<1is non\-differentiable whenα​β=0\\alpha\\beta=0, we additionally evaluate the corresponding boundary solutions\. Let𝒮i\\mathcal\{S\}\_\{i\}denote the set of nonzero stationary solutions obtained from \([19](https://arxiv.org/html/2609.28998#S3.E19)\) andℬ=\{\(0,1\),\(1,0\)\}\\mathcal\{B\}=\\\{\(0,1\),\(1,0\)\\\}the set of boundary candidates\. The solution of \([17](https://arxiv.org/html/2609.28998#S3.E17)\) is then obtained among all candidate stationary and boundary points:

\(αi,βi\)∈arg⁡min\(α,β\)∈𝒮i∪ℬ​fi​\(α,β\)\.\(\\alpha\_\{i\},\\beta\_\{i\}\)\\in\\arg\\min\_\{\(\\alpha,\\beta\)\\in\\mathcal\{S\}\_\{i\}\\cup\\mathcal\{B\}\}f\_\{i\}\(\\alpha,\\beta\)\.\(21\)
In contrast to the classicalℓp\\ell\_\{p\}proximal problem in \([3](https://arxiv.org/html/2609.28998#S2.E3)\), our thresholding decision is obtained implicitly by solving the two\-dimensional problem \([17](https://arxiv.org/html/2609.28998#S3.E17)\)\. Specifically, if its solution satisfiesαi​βi=0\\alpha\_\{i\}\\beta\_\{i\}=0, the corresponding rank\-one component is identified as a pruning candidate; otherwise, it is retained\. We refer to this optimization\-induced zero–nonzero decision as an*implicit thresholding rule*for LoRA rank allocation\.

### 3\.3LoRA Rank Allocation Based on Thresholding

We now incorporate the proposed implicit thresholding rule into standard LoRA training\. Importantly, the thresholding procedure does not replace the original optimizer used for fine\-tuning\. The LoRA factors are updated following standard training, while the proposed criterion is periodically invoked to identify redundant rank\-one components\. We refer to the resulting method asℓp\\ell\_\{p\}\-LoRA, as its rank allocation is driven byℓp\\ell\_\{p\}regularization\. The detailed algorithm is presented in Algorithm[1](https://arxiv.org/html/2609.28998#alg1)\.

Algorithm 1Rank Allocation via Thresholding \(ℓp\\ell\_\{p\}\-LoRA\)0:LoRA factors

𝑨,𝑩\\bm\{A\},\\bm\{B\}, regularization parameter

λ\\lambda, curvature parameter

MM
1:foreach training iteration

kkdo

2:Update

𝑨,𝑩\\bm\{A\},\\bm\{B\}using an optimizer \(e\.g\., AdamW\)

3:Aggregate gradients

𝑮¯A\\overline\{\\bm\{G\}\}\_\{A\}and

𝑮¯B\\overline\{\\bm\{G\}\}\_\{B\}as approximations to

∇𝑨ℒk\\nabla\_\{\\bm\{A\}\}\\mathcal\{L\}\_\{k\}and

∇𝑩ℒk\\nabla\_\{\\bm\{B\}\}\\mathcal\{L\}\_\{k\}
4:if

kkis a pruning stepthen

5:

𝑨~←𝑨−M−1​𝑮¯A\\widetilde\{\\bm\{A\}\}\\leftarrow\\bm\{A\}\-M^\{\-1\}\\overline\{\\bm\{G\}\}\_\{A\},

𝑩~←𝑩−M−1​𝑮¯B\\widetilde\{\\bm\{B\}\}\\leftarrow\\bm\{B\}\-M^\{\-1\}\\overline\{\\bm\{G\}\}\_\{B\}
6:for

i=1,…,ri=1,\\ldots,rdo

7:Solve \([17](https://arxiv.org/html/2609.28998#S3.E17)\) for

\(αi⋆,βi⋆\)\(\\alpha\_\{i\}^\{\\star\},\\beta\_\{i\}^\{\\star\}\)
8:if

αi⋆​βi⋆=0\\alpha\_\{i\}^\{\\star\}\\beta\_\{i\}^\{\\star\}=0then

9:Prune the

ii\-th rank\-one component

10:endif

11:endfor

12:endif

13:endfor

## 4Experiment

### 4\.1Natural Language Understanding

We first evaluate the proposed method on natural language understanding tasks using DeBERTaV3\-base\[[8](https://arxiv.org/html/2609.28998#bib.bib17)\]\. Experiments are conducted on four tasks from the GLUE benchmark\[[15](https://arxiv.org/html/2609.28998#bib.bib6)\], including CoLA, RTE, MRPC, and STS\-B\. Following the standard evaluation protocol, we report Matthews correlation for CoLA, accuracy for RTE, MRPC, and Pearson correlation for STS\-B\. We compare our method with LoRA\[[10](https://arxiv.org/html/2609.28998#bib.bib1)\], AdaLoRA\[[18](https://arxiv.org/html/2609.28998#bib.bib2)\], and IGU\-LoRA\[[6](https://arxiv.org/html/2609.28998#bib.bib12)\]under the same training schedule and a final rank budget equal to half of initial rank\. For our method, we setp=0\.5p=0\.5andM=10M=10, with the remaining pruning hyperparameters kept fixed across all tasks\.

Table 1:Results on NLU tasks from the GLUE benchmark using DeBERTaV3\-base\. Higher values indicate better performance\. Results are reported as the mean over five random seeds, with the empirical standard deviation shown in the subscript\. The best result is shown in bold\.Table[1](https://arxiv.org/html/2609.28998#S4.T1)summarizes the results\.ℓp\\ell\_\{p\}\-LoRA achieves the best performance on two out of the four tasks and remains competitive on the remaining tasks\. In particular, the consistent performance across tasks with different evaluation metrics suggests that the proposedℓp\\ell\_\{p\}\-based implicit thresholding criterion can effectively identify redundant LoRA components while maintaining the adaptation capability of the model\. Overall, these results demonstrate that rank allocation based onℓp\\ell\_\{p\}regularization provides a competitive alternative to existing fixed\-rank and adaptive\-rank LoRA methods\.

### 4\.2Question Answering

We further evaluate the proposed method on question\-answering tasks using Qwen2\.5\-7B\[[17](https://arxiv.org/html/2609.28998#bib.bib18)\]to examine its effectiveness in a larger\-scale language model and reasoning tasks\. Experiments are conducted on BoolQ\[[4](https://arxiv.org/html/2609.28998#bib.bib7)\], ARC\-Easy\[[5](https://arxiv.org/html/2609.28998#bib.bib8)\], OpenBookQA\[[11](https://arxiv.org/html/2609.28998#bib.bib9)\], and CommonsenseQA\[[12](https://arxiv.org/html/2609.28998#bib.bib10)\]\. All experiments are conducted under the same training setting and a final rank budget equal to half of the initial rank, and report accuracy averaged over five random seeds\.

Table 2:Results on question answering tasks using Qwen2\.5\-7B\. We report accuracy as the mean over five random seeds, with the empirical standard deviation shown in the subscript\. The best result is shown in bold\.As shown in Table[2](https://arxiv.org/html/2609.28998#S4.T2), our method achieves the best performance on two of the four tasks, including BoolQ and CommonsenseQA, while remaining competitive on ARC\-Easy and OpenBookQA\. In particular, the improvement on BoolQ is more pronounced, while the result on CommonsenseQA is comparable to the strongest baseline\. These results indicate that the proposed thresholding\-based rank\-allocation criterion remains effective when applied to a larger language model across different question\-answering tasks\.

## 5Conclusion

In this work, we studied adaptive rank allocation for LoRA from a structured sparsity perspective\. By imposing anℓp\\ell\_\{p\}penalty on the energy of rank\-one LoRA components, we formulated rank allocation as anℓp\\ell\_\{p\}\-regularized optimization problem\. We derived a tractable component\-wise subproblem and reduced the coupled optimization to a two\-dimensional scalar problem, leading to an implicit thresholding criterion for pruning redundant components while preserving standard LoRA training for the remaining ones\. Experiments on natural language understanding and question\-answering tasks demonstrate that the proposed method achieves competitive performance with existing fixed\-rank and adaptive\-rank LoRA methods under the same rank budget\.

## References

- \[1\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in Neural Information Processing Systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p1.1)\.
- \[2\]E\. J\. Candès, J\. Romberg, and T\. Tao\(2006\)Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information\.IEEE Transactions on Information Theory52\(2\),pp\. 489–509\.Cited by:[§2\.2](https://arxiv.org/html/2609.28998#S2.SS2.p1.2)\.
- \[3\]R\. Chartrand\(2007\)Exact reconstruction of sparse signals via nonconvex minimization\.IEEE Signal Processing Letters14\(10\),pp\. 707–710\.Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p4.1),[§2\.2](https://arxiv.org/html/2609.28998#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2609.28998#S2.SS2.p1.2),[§3\.1](https://arxiv.org/html/2609.28998#S3.SS1.p3.1)\.
- \[4\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)Boolq: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2924–2936\.Cited by:[§4\.2](https://arxiv.org/html/2609.28998#S4.SS2.p1.1)\.
- \[5\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.2](https://arxiv.org/html/2609.28998#S4.SS2.p1.1)\.
- \[6\]X\. Cui, H\. Li, R\. Zeng, Y\. Zhao, J\. Qian, W\. Duan, B\. Liu, and Z\. Zhou\(2026\)IGU\-loRA: adaptive rank allocation via integrated gradients and uncertainty\-aware scoring\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=MnToYQx9My)Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.28998#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2609.28998#S4.SS1.p1.1)\.
- \[7\]Z\. Duan, Y\. Wang, J\. Fu, Z\. Yue, B\. Chen, and Z\. Xu\(2026\)BaRA: bayesian adaptive rank allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2606\.29184\.Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p3.1)\.
- \[8\]P\. He, J\. Gao, and W\. Chen\(2023\)DeBERTaV3: improving DeBERTa using electra\-style pre\-training with gradient\-disentangled embedding sharing\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sE7-XhLxHA)Cited by:[§4\.1](https://arxiv.org/html/2609.28998#S4.SS1.p1.1)\.
- \[9\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for NLP\.InInternational Conference on Machine Learning,pp\. 2790–2799\.Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p1.1)\.
- \[10\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.28998#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.28998#S4.SS1.p1.1)\.
- \[11\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[§4\.2](https://arxiv.org/html/2609.28998#S4.SS2.p1.1)\.
- \[12\]A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant\(2019\)Commonsenseqa: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.Cited by:[§4\.2](https://arxiv.org/html/2609.28998#S4.SS2.p1.1)\.
- \[13\]M\. Valipour, M\. Rezagholizadeh, I\. Kobyzev, and A\. Ghodsi\(2023\)DyLoRA: parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 3274–3287\.Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.28998#S2.SS1.p2.1)\.
- \[14\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[§2\.1](https://arxiv.org/html/2609.28998#S2.SS1.p1.1)\.
- \[15\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§4\.1](https://arxiv.org/html/2609.28998#S4.SS1.p1.1)\.
- \[16\]A\. Yang, M\. Robeyns, X\. Wang, and L\. Aitchison\(2024\)Bayesian low\-rank adaptation for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 1812–1842\.Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p3.1)\.
- \[17\]A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei,et al\.\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§4\.2](https://arxiv.org/html/2609.28998#S4.SS2.p1.1)\.
- \[18\]Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)Adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lq62uWRJjiY)Cited by:[§1](https://arxiv.org/html/2609.28998#S1.p2.1),[§1](https://arxiv.org/html/2609.28998#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.28998#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2609.28998#S4.SS1.p1.1)\.