Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study

# Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study
Source: [https://arxiv.org/html/2605.07366](https://arxiv.org/html/2605.07366)
###### Abstract

Adaptive rank allocation forLoRA— allocating more parameters to important layers and fewer to unimportant ones — consistently improves efficiency under supervised fine\-tuning \(SFT\)\. We investigate whether this success transfers to reinforcement learning, specifically Group Relative Policy Optimization \(GRPO\)\. Using gradient\-magnitude profiling on Qwen 2\.5 1\.5B with GSM8K, we find that it does not: proportional rank allocationdegradesaccuracy by 4\.5 points compared to uniform allocation \(70\.0% vs\. 74\.5%\), despite using identical parameter budgets\. We identify two mechanisms behind this failure\. First, the gradient landscape underGRPOis fundamentally flatter than under SFT — the max\-to\-min layer importance ratio is only 2\.17×\\times, compared to\>\>10×\\timesreported in SFT literature\. All layers carry meaningful gradient signal; none are truly idle\. Second, we discover agradient amplification effect: non\-uniform allocation widens the importance spread from 2\.17×\\timesto 3\.00×\\times, creating a positive feedback loop where high\-rank layers absorb more gradient while low\-rank layers are progressively silenced\. Our results suggest that gradient importance does not predict capacity requirements under RL, and that naïve transfer of SFT\-era rank allocation to alignment training should be avoided\.

## 1Introduction

Parameter\-efficient fine\-tuning via Low\-Rank Adaptation\(Hu et al\.,[2022](https://arxiv.org/html/2605.07366#bib.bib4)\)has become the standard approach for adapting large language models\.LoRAdecomposes weight updates into low\-rank matricesΔ​W=B​A\\Delta W=BAwith uniform rankrracross all layers\.

Recent work has challenged this uniform assumption\. AdaLoRA\(Zhang et al\.,[2023](https://arxiv.org/html/2605.07366#bib.bib9)\)dynamically prunes singular values during training based on importance scores\. GoRA\(He et al\.,[2025](https://arxiv.org/html/2605.07366#bib.bib3)\)allocates rank proportionally to gradient\-weight products at initialization\. Aletheia\(Saket,[2026](https://arxiv.org/html/2605.07366#bib.bib5)\)selects layers via lightweight gradient probes\. ILA\(Shi et al\.,[2024](https://arxiv.org/html/2605.07366#bib.bib7)\)shows that only 10–30% of layers are significant for alignment under SFT\. These methods achieve meaningful efficiency gains, establishing thatgradient importance correlates with capacity requirements under supervised objectives\.

A natural question follows: does this correlation hold under reinforcement learning? RL\-based alignment methods likeGRPO\(Shao et al\.,[2024](https://arxiv.org/html/2605.07366#bib.bib6)\)optimize a fundamentally different objective — advantage\-weighted policy gradients with sparse, binary reward signals rather than dense per\-token cross\-entropy loss\. Theoretical work suggests that RL gradients concentrate differently than SFT gradients\(Young,[2026](https://arxiv.org/html/2605.07366#bib.bib8)\), motivating the hypothesis that rank allocation strategies should differ\.

We test this hypothesis by applying gradient\-based rank profiling toGRPOtraining and find a surprising result:adaptive rank allocation hurts performance under RL\. Our investigation reveals three findings:

1. 1\.Flat gradient landscape:UnderGRPO, layer importance is distributed far more uniformly than under SFT \(2\.17×\\timesmax/min ratio vs\.\>\>10×\\times\)\. All layers are load\-bearing\.
2. 2\.Gradient amplification:Non\-uniform allocation creates a positive feedback loop — high\-rank layers absorb more gradient while low\-rank layers are silenced, widening the spread from 2\.17×\\timesto 3\.00×\\times\.
3. 3\.Generalization gap:Models train equally well \(identical reward curves\) but generalize differently — the damage from rank reallocation appears only at evaluation time\.

## 2Method

### 2\.1GRPO with LoRA

GRPOgeneratesKKcompletions per prompt, computes rewards, and normalizes advantages within the group:

A^i=ri−μgroupσgroup\\hat\{A\}\_\{i\}=\\frac\{r\_\{i\}\-\\mu\_\{\\text\{group\}\}\}\{\\sigma\_\{\\text\{group\}\}\}\(1\)When combined withLoRA, gradients flow through adapter parametersB\(l\),A\(l\)B^\{\(l\)\},A^\{\(l\)\}at each layerll\.

### 2\.2Reward Sensitivity Profiling

We define the reward sensitivity score for layerllas the mean gradient norm overTTtraining steps:

S​\(l\)=1T​∑t=1T∑m∈ℳ‖∇θm\(l\)ℒt‖2S\(l\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\sum\_\{m\\in\\mathcal\{M\}\}\\left\\\|\\nabla\_\{\\theta\_\{m\}^\{\(l\)\}\}\\mathcal\{L\}\_\{t\}\\right\\\|\_\{2\}\(2\)whereℳ\\mathcal\{M\}is the set of target modules \(q/k/v/o/up/down/gate projections\)\. This score captures how much each layer’s parameters respond to theGRPOreward signal\.

### 2\.3Rank Allocation

Given total rank budgetRtotal=L×runiformR\_\{\\text\{total\}\}=L\\times r\_\{\\text\{uniform\}\}, we allocate per\-layer rank:

r\(l\)=clip​\(round​\(S​\(l\)∑l′S​\(l′\)×Rtotal\),rmin,rmax\)r^\{\(l\)\}=\\text\{clip\}\\left\(\\text\{round\}\\left\(\\frac\{S\(l\)\}\{\\sum\_\{l^\{\\prime\}\}S\(l^\{\\prime\}\)\}\\times R\_\{\\text\{total\}\}\\right\),r\_\{\\min\},r\_\{\\max\}\\right\)\(3\)withrmin=4r\_\{\\min\}=4andrmax=64r\_\{\\max\}=64, rounded to multiples of 4\. We also evaluate random allocation as a control\.

## 3Experiments

### 3\.1Setup

Model:Qwen/Qwen2\.5\-1\.5B\-Instruct \(28 transformer layers\)\.Dataset:GSM8K\(Cobbe et al\.,[2021](https://arxiv.org/html/2605.07366#bib.bib1)\)with structured XML output format\.Rewards:Format compliance \(1\.0 for correct<think\>/<answer\>tags\) and answer correctness \(1\.0 for correct numerical answer\)\.LoRA:Applied to all 7 projection modules per layer \(196 adapters total\)\. Uniform baseline:r=32r=32\.GRPO:K=4K=4generations,β=0\.05\\beta=0\.05KL coefficient, 1000 training steps, learning rate10−510^\{\-5\}, vLLM colocate generation\.

### 3\.2Gradient Landscape Under GRPO

![Refer to caption](https://arxiv.org/html/2605.07366v1/x1.png)Figure 1:Per\-layer gradient magnitude duringGRPOtraining \(1000 steps, 28 layers\)\. The distribution is notably flat compared to SFT\-era findings\.Figure[1](https://arxiv.org/html/2605.07366#S3.F1)shows the reward sensitivity map\. Key observations:

- •Flat distribution:Max/min importance ratio is 2\.17×\\times\(Layer 15 hottest at 4\.68%, Layer 26 coldest at 2\.15%\)\. Even the coldest layer carries 46% of the hottest layer’s gradient signal\.
- •Middle\-layer concentration:Layers 9–18 carry 43\.0% of gradient, but early \(29\.8%\) and late \(27\.2%\) layers remain meaningful — unlike SFT where ILA\(Shi et al\.,[2024](https://arxiv.org/html/2605.07366#bib.bib7)\)reports\>\>80% concentration in the top 30%\.
- •Temporal stability:Early\-vs\-late training correlation is 0\.962, indicating stable structural patterns, not transient noise\.
- •Module importance:Attention \(52\.9%\) and FFN \(47\.1%\) contribute roughly equally\. Theup\_projmodule is most reward\-sensitive \(21\.4%\)\.

### 3\.3Rank Allocation Results

Table 1:GSM8K accuracy under different rank allocations \(nn=200 test samples\)\. All “same budget” methods use total rank 896 \(28×3228\\times 32\)\. Confidence intervals are Wilson score intervals\.Table[1](https://arxiv.org/html/2605.07366#S3.T1)shows our main result:uniform allocation outperforms all non\-uniform variants, including gradient\-aware proportional allocation\. Proportional allocation with identical parameter budget scores 4\.5 points below uniform\. Random allocation — a control with non\-uniform ranks but no gradient guidance — scores even lower at 67\.5%\. Reduced\-budget allocation performs below the untrained base model\.

Two observations stand out\. First,gradient\-aware allocation outperforms random\(70\.0% vs\. 67\.5%\), confirming that the importance signal is meaningful — it identifies genuinely important layers\. However, both lose to uniform, suggesting that while the signal is directionally correct, any deviation from uniform allocation damages performance underGRPO\.

Second, training reward curves are nearly identical across all configurations — correctness reward reaches 0\.74–0\.77 for all methods by step 1000\. The performance gap appearsonly at evaluation time, indicating that rank reallocation damages generalization rather than training dynamics\. Models learn equally well on the training distribution but differ in their ability to transfer to unseen problems\.

### 3\.4The Gradient Amplification Effect

![Refer to caption](https://arxiv.org/html/2605.07366v1/x2.png)Figure 2:Normalized layer importance under uniform vs\. proportional allocation\. Non\-uniform allocation amplifies the original importance spread from 2\.17×\\timesto 3\.00×\\times\.We profile gradients duringalltraining runs, not just the uniform baseline\. Figure[2](https://arxiv.org/html/2605.07366#S3.F2)reveals a striking effect: the gradient importance spreadwidensunder non\-uniform allocation\.

- •Uniform: max/min ratio 2\.17×\\times
- •Proportional \(same budget\): 3\.00×\\times\(\+38%\)
- •Reduced 70%: 3\.57×\\times\(\+64%\)

Layers given higher rank \(e\.g\., Layer 18:rr=40\) see their gradient shareincreasefrom 4\.53% to 5\.12%\. Conversely, layers given lower rank \(e\.g\., Layer 24:rr=20\) see their sharedecreasefrom 2\.23% to 1\.71%\. The allocation creates a positive feedback loop: more capacity→\\rightarrowmore gradient→\\rightarrowappears even more “important\.”

Crucially, this effect iscausal, not correlative\. The random allocation experiment — where ranks bear no relation to gradient importance — shows equally strong amplification\. The correlation between allocated rank and resulting gradient shift is 0\.972 for random allocation and 0\.946 for proportional\. Under random allocation, Layer 1 \(normally 3\.09% of gradient\) receives rank 48 and jumps to 4\.21%; Layer 10 \(normally 3\.93%\) receives rank 16 and drops to 2\.85%\.Rank determines gradient importance, not the other way around\.

This means gradient profiling under one allocation cannot reliably inform a different allocation — the “profile then retrain” paradigm is fundamentally flawed forGRPO\. One might suspect this is trivially explained by more parameters yielding larger aggregate gradient norms\. However, our reward sensitivity score \(Eq\. 2\) normalizes by module count, and the amplification persists even when examining individual modules at identical dimensions \(e\.g\.,q\_projat 1536×\\times1536 across all layers\)\. The effect reflects genuine changes in how the training signal distributes across the network, not merely a parameter\-counting artifact\.

This amplification effect has not been reported in prior work, likely because SFT\-based methods show the opposite pattern — AdaLoRA’s dynamic pruning stabilizes importance distributions by adjusting continuously during training rather than committing to a fixed allocation\.111We also discovered that PEFT’srank\_patternwildcard matching \(e\.g\.,model\.layers\.\*\.q\_proj\) silently fails for Qwen models, falling back to default rank\. Exact module paths \(e\.g\.,model\.layers\.N\.self\_attn\.q\_proj\) are required\.

### 3\.5Why SFT Methods Fail Under RL

The fundamental difference lies in gradient distribution:

Under SFT, the loss is per\-token cross\-entropy where a small number of layers dominate the gradient landscape\. ILA\(Shi et al\.,[2024](https://arxiv.org/html/2605.07366#bib.bib7)\)shows the top 30% of layers carry\>\>80% of gradient signal, and freezing the rest improves performance\. This concentrated structure makes adaptive allocation effective — there are genuinely idle layers whose capacity can be safely redistributed\.

UnderGRPO, the loss is advantage\-weighted policy gradient with sparse, binary reward\. Our profiling reveals a fundamentally flatter landscape where the top 30% carry only 35\.7% of signal\. Even the “coldest” layers contribute meaningfully \(2\.15% of total, or 46% of the hottest\)\. Reducing their capacity — even modestly, fromrr=32 torr=20 — damages evaluation accuracy while leaving training reward unchanged\.

We hypothesize that low\-gradient layers handle essential structural functions under RL: output formatting, numerical precision, and coherence maintenance\. These functions generate small gradients because they are already well\-handled by the pretrained model, but they become bottlenecks when capacity is reduced\.

This hypothesis is supported by our module\-level analysis: attention and FFN modules contribute 52\.9% and 47\.1% respectively, a near\-even split unlike SFT where FFN layers in the top half of the network dominate\(Zhang et al\.,[2023](https://arxiv.org/html/2605.07366#bib.bib9)\)\. Theup\_projmodule is most reward\-sensitive \(21\.4%\), whileq\_projcontributes only 9\.5% — yet both are essential for correct mathematical reasoning\.

Our findings also reveal that the gradient importance map underGRPOis temporally stable \(early\-vs\-late training correlation: 0\.962\), ruling out the possibility that a different profiling window would yield a more useful allocation\. The flat landscape is a structural property of howGRPOdistributes learning, not an artifact of averaging over noisy training phases\.

## 4Related Work

Adaptive rank for SFT\.AdaLoRA\(Zhang et al\.,[2023](https://arxiv.org/html/2605.07366#bib.bib9)\)prunes singular values during training\. GoRA\(He et al\.,[2025](https://arxiv.org/html/2605.07366#bib.bib3)\)uses gradient\-weight products for initialization\-time allocation\. IGU\-LoRA\(Cui et al\.,[2026](https://arxiv.org/html/2605.07366#bib.bib2)\)applies integrated gradients with uncertainty\-aware scoring\. Aletheia\(Saket,[2026](https://arxiv.org/html/2605.07366#bib.bib5)\)selects layers via gradient probes\.All operate exclusively under supervised objectives\.

Layer importance in alignment\.ILA\(Shi et al\.,[2024](https://arxiv.org/html/2605.07366#bib.bib7)\)learns binary layer masks showing 10–30% of layers suffice for SFT alignment\.Young \([2026](https://arxiv.org/html/2605.07366#bib.bib8)\)prove that RLHF gradients concentrate at specific positions\. Our work extends this line by showing that RL’s gradient concentration isinsufficientfor effective rank allocation\.

GRPO and LoRA\.DeepSeekMath\(Shao et al\.,[2024](https://arxiv.org/html/2605.07366#bib.bib6)\)introducedGRPOfor mathematical reasoning\. To our knowledge, no prior work has investigated adaptive rank allocation specifically under RL\-based alignment methods\.

## 5Limitations

Our findings are derived from a single model \(Qwen 2\.5 1\.5B\), a single dataset \(GSM8K\), and a single RL algorithm \(GRPO\)\. Whether these results generalize to larger models, other domains \(code, safety alignment\), or other RL methods \(PPO, DPO\) remains an open question\. Our evaluation usesnn=200 test samples with single\-seed runs; the confidence intervals in Table[1](https://arxiv.org/html/2605.07366#S3.T1)overlap, and we encourage replication at larger scale with multiple seeds to establish statistical significance\. We compare to SFT gradient distributions by citing prior work rather than running a direct SFT baseline on the same model and data — a head\-to\-head comparison would strengthen the contrast\. Finally, our hypothesis that cold layers handle structural functions \(formatting, numerical precision\) is untested; per\-reward gradient decomposition could partially verify this\.

## 6Conclusion

We investigated whether gradient\-based rank allocation — a proven technique for SFT — transfers to reinforcement learning alignment viaGRPO\. Our experiments show that it does not: proportional allocation degrades accuracy by 4\.5 points despite identical parameter budgets\. We identify the flat gradient landscape underGRPOand the gradient amplification effect as key mechanisms behind this failure\.

These findings have practical implications: practitioners should not naïvely apply SFT\-era rank allocation strategies to RL training\. A random control confirms that gradient\-aware allocation \(70\.0%\) outperforms uninformed allocation \(67\.5%\), validating that the importance signal is real — but insufficient to overcome the fundamental need for uniform capacity underGRPO\.

Future work should exploredynamicrank adaptation methods that adjust continuously during training \(avoiding the “profile then retrain” problem\) and investigate whether the amplification effect — where rank causally determines gradient importance withr\>0\.97r\>0\.97correlation — can be exploited for targeted capacity expansion rather than reallocation\.

## References

- Cobbe et al\. \(2021\)Cobbe, K\., Kosaraju, V\., Bavarian, M\., Chen, M\., Jun, H\., Kaiser, L\., Plappert, M\., Tworek, J\., Hilton, J\., Nakano, R\., Hesse, C\., and Schulman, J\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Cui et al\. \(2026\)Cui, X\., Li, H\., Zeng, R\., Zhao, Y\., Qian, J\., Duan, W\., Liu, B\., and Zhou, Z\.IGU\-LoRA: Adaptive rank allocation via integrated gradients and uncertainty\-aware scoring\.*arXiv preprint arXiv:2603\.13792*, 2026\.
- He et al\. \(2025\)He, H\., Ye, P\., Ren, Y\., Yuan, Y\., Zhou, L\., Ju, S\., and Chen, L\.GoRA: Gradient\-driven adaptive low rank adaptation\.In*Advances in Neural Information Processing Systems*, 2025\.
- Hu et al\. \(2022\)Hu, E\. J\., Shen, Y\., Wallis, P\., Allen\-Zhu, Z\., Li, Y\., Wang, S\., Wang, L\., and Chen, W\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations*, 2022\.
- Saket \(2026\)Saket, A\.Aletheia: Gradient\-guided layer selection for efficient LoRA fine\-tuning across architectures\.*arXiv preprint arXiv:2604\.15351*, 2026\.
- Shao et al\. \(2024\)Shao, Z\., Wang, P\., Zhu, Q\., Xu, R\., Song, J\., Zhang, M\., Li, Y\. K\., Wu, Y\., and Guo, D\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shi et al\. \(2024\)Shi, G\., Lu, Z\., Dong, X\., Zhang, W\., Zhang, X\., Feng, Y\., and Wu, X\.\-M\.Understanding layer significance in LLM alignment\.*arXiv preprint arXiv:2410\.17875*, 2024\.
- Young \(2026\)Young, R\.Why is RLHF alignment shallow? A gradient analysis\.*arXiv preprint arXiv:2603\.04851*, 2026\.
- Zhang et al\. \(2023\)Zhang, Q\., Chen, M\., Bukharin, A\., He, P\., Cheng, Y\., Chen, W\., and Zhao, T\.AdaLoRA: Adaptive budget allocation for parameter\-efficient fine\-tuning\.In*International Conference on Learning Representations*, 2023\.