GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

# GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Source: [https://arxiv.org/html/2608.11674](https://arxiv.org/html/2608.11674)
Jingwei XuWanyu WangKai\-Yuan GuoZhenbo Yu\\correspondingYi Wang\\correspondingThanks:Contributions made while affiliated with Shanghai AI Lab\.Yu Qiao\\corresponding

###### Abstract

On\-policy rollout methods such as GRPO are central to post\-training of large language models\. Yet, they frequently suffer from training instabilities, cross\-task capability degradation, and response\-length inflation\. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear\. We introduce*Principal\-Subspace Overlap*, a dimension\-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights\. Despite low average overlap, transient spikes often precede performance degradation\. To address this, we proposeGCPO\(GeometricallyConstrainedPolicyOptimization\), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction\. Across mathematical reasoning, code generation, and tool\-use tasks on Qwen3\-8B and GLM4\-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27\.69 and 2\.37 points, respectively\. Furthermore, GCPO preserves general capabilities, eliminates response\-length inflation, and stabilizes policy entropy\. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post\-training\.

Code & Datasets—https://github\.com/Icarus1411/GCPO

## 1Introduction

Rollout\-based reinforcement learning \(RL\) has become a widely used post\-training mechanism for improving large language models \(LLMs\) on mathematical reasoning, coding, and tool use\([21](https://arxiv.org/html/2608.11674#bib.bib1);[28](https://arxiv.org/html/2608.11674#bib.bib25);[6](https://arxiv.org/html/2608.11674#bib.bib8)\)\. Its optimization data, however, are generated by the evolving policy itself: each update changes both the policy and the rollout distribution used to construct the next update\. This feedback loop can yield unstable training, degradation on capabilities outside the optimized task, and response\-length inflation\([11](https://arxiv.org/html/2608.11674#bib.bib30);[18](https://arxiv.org/html/2608.11674#bib.bib9)\)\.

Most existing remedies stabilize RL through objective\- or policy\-level controls\. KL regularization constrains divergence from a reference policy, clipping limits large likelihood\-ratio changes, and reward design modifies the optimization signal\([34](https://arxiv.org/html/2608.11674#bib.bib13);[32](https://arxiv.org/html/2608.11674#bib.bib12);[33](https://arxiv.org/html/2608.11674#bib.bib11)\)\. While effective, these methods may treat different updates as similar if they have similar policy\-level statistics, even when their directions in parameter space differ\. Because updates of similar magnitude may interact differently with the structured parameter space of the pretrained model, we ask whether their stepwise directions provide a complementary diagnostic of training dynamics\.

We focus on the dominant singular subspaces of each pretrained linear weight matrix\. The top right and left singular vectors identify the dominant input and output directions, respectively\. Together, they define the optimal rank\-kkapproximation of the pretrained transformation\([8](https://arxiv.org/html/2608.11674#bib.bib28)\)\. Updates overlapping these subspaces can therefore alter structurally prominent components of the pretrained transformation\. Rather than interpreting these directions as a literal decomposition of semantic knowledge, we treat them as a functionally distinguished yet tractable structural reference\. Prior analyses further show that RL updates are predominantly off\-principal when averaged over training\([23](https://arxiv.org/html/2608.11674#bib.bib3);[31](https://arxiv.org/html/2608.11674#bib.bib20);[19](https://arxiv.org/html/2608.11674#bib.bib34)\)\. This aggregate regularity leaves open an important question: do individual updates exhibit transient principal\-subspace overlap, and how is such overlap related to training performance?

Our stepwise analysis reveals transient entry of individual updates into the pretrained principal subspaces\. We decompose each realized update into four blocks according to its left\- and right\-side overlap with these subspaces, and subtract the overlap expected under an isotropic null\. Although most update energy lies in the doubly orthogonal block, aggregate statistics obscure intermittent spikes in*excess principal\-subspace overlap*\. Across the runs we examine, sustained or repeated spikes coincide with, and often precede, declines in validation accuracy\. We therefore treat overlap as a geometric correlate of unstable phases, motivating updates that preserve the selected pretrained mappings by remaining in their bilateral orthogonal complement\.

We instantiate this hypothesis inGCPO\(GeometricallyConstrainedPolicyOptimization\), which augments rollout\-based policy optimization with a hard geometric constraint on the effective policy update\. At each rollout iteration, GCPO optimizes the underlying policy objective while requiring every adapted layer update to lie in the bilateral orthogonal complement of the selected pretrained principal subspaces\. GCPO thus changes the admissible directions of policy improvement, rather than introducing another policy\-level penalty\. The constraint is complementary to KL regularization: KL controls policy change in output space, whereas GCPO controls the feasible directions of parameter change\. It also has a precise layer\-level guarantee: for inputs in a selected principal input subspace, the adapted layer’s response is unchanged\. The remaining complement is large, retaining a high\-dimensional feasible complement for task adaptation\.

Comprehensive evaluations on Qwen3\-8B and GLM4\-9B cover mathematical reasoning, code generation, and tool use\. Across all tasks, GCPO achieves the best accuracy, outperforming the strongest baseline by 1\.02–2\.37 points \(±std over 3 seeds\) and the corresponding instruction\-tuned model by 7\.09–27\.69 points\. When trained on mathematics and evaluated on other tasks, it attains the strongest worst\-case retention\. It also shows smoother accuracy and policy\-entropy trajectories and suppresses response\-length inflation\.

In summary, our main contributions are threefold:

- •We introduce a dimension\-corrected, stepwise measure of principal\-subspace overlap and show that elevated overlap is a warning signal associated with declining validation performance in rollout RL\.
- •We formulate rollout RL as a constrained policy optimization problem and propose GCPO to enforce bilateral orthogonality on every effective policy update, which preserves the principal pretrained mappings while retaining a large complementary update space\.
- •Across two model families and three task domains, GCPO outperforms all evaluated baselines, better preserves cross\-task capabilities, and exhibits more stable accuracy, policy\-entropy, and response\-length dynamics\.

## 2Related Work

#### Rollout\-based RL for LLM Alignment\.

Rollout\-based reinforcement learning has emerged as the dominant paradigm for enhancing LLM reasoning capabilities\. Building upon PPO\([21](https://arxiv.org/html/2608.11674#bib.bib1)\), recent advances like GRPO\([22](https://arxiv.org/html/2608.11674#bib.bib2)\), GSPO\([34](https://arxiv.org/html/2608.11674#bib.bib13)\), GMPO\([33](https://arxiv.org/html/2608.11674#bib.bib11)\), and DAPO\([32](https://arxiv.org/html/2608.11674#bib.bib12)\)have significantly improved advantage estimation and scaling efficiency\. Despite these innovations, the dynamic nature of self\-generated rollouts frequently drives policies toward optimization instabilities, response\-length inflation, and general capability degradation\([30](https://arxiv.org/html/2608.11674#bib.bib14)\)\. Crucially, existing methods primarily intervene through objectives or observable policy behavior, leaving the geometry of realized parameter updates less explored\. This highlights the need to diagnose the roots of these instabilities from a novel scope rather than symptomatic output\-space patching\.

#### Regularization and Stability in RL\.

RL post\-training is commonly stabilized through objective\- or policy\-level controls\. KL penalties discourage deviation from a reference policy, clipping constrains large likelihood\-ratio changes, and reward shaping adjusts the scalar optimization signal to reduce undesirable behaviors such as reward hacking or length inflation\([24](https://arxiv.org/html/2608.11674#bib.bib10);[15](https://arxiv.org/html/2608.11674#bib.bib29)\)\. These techniques are effective, but they remain soft controls: they discourage unstable updates rather than ruling them out\. Their effectiveness often depends on carefully tuned penalty coefficients and can be weakened by the high\-variance feedback of rollout\-based RL\([1](https://arxiv.org/html/2608.11674#bib.bib32);[9](https://arxiv.org/html/2608.11674#bib.bib33)\)\. These limitations motivate looking beyond objective\-level controls and designing a complementary approach that constrains the feasible update space directly\.

#### Geometric Analysis of Policy Optimization\.

Recent studies suggest that the geometry of parameter updates plays a critical role in RL\-based post\-training\. Rather than treating policy updates as unstructured perturbations, these studies show that RL\-induced updates exhibit systematic geometric patterns relative to the pretrained weights\. In particular, recent analyses find that successful RL updates are, on average, more concentrated outside the dominant singular subspaces of the pretrained operators\([23](https://arxiv.org/html/2608.11674#bib.bib3);[2](https://arxiv.org/html/2608.11674#bib.bib4)\)\. This off\-principal tendency suggests that effective policy adaptation may preferentially exploit directions that interfere less with the dominant structures learned during pre\-training, which provides a foundation for understanding the importance of parameter\-space geometry in rollout\-based RL\. However, existing analyses mainly characterize aggregate update behavior, leaving open how stepwise deviations from this geometry emerge during training and whether they are related to optimization instability\.

## 3Principal\-Subspace Overlap and Instability

![Refer to caption](https://arxiv.org/html/2608.11674v1/principal_subspace_intrusion_gsm8k_math500_side_by_side.png)Figure 1:Stepwise update overlap and validation performance\. The red curve is the 5\-step moving average of excess principal\-subspace overlap; the blue curve is validation accuracy\. In both runs, episodes of elevated overlap accompany subsequent gradual degradation\. This observation is correlational\.Prior work finds that RL updates, when aggregated over training, predominantly lie outside the dominant singular subspaces of pretrained weights\([23](https://arxiv.org/html/2608.11674#bib.bib3);[2](https://arxiv.org/html/2608.11674#bib.bib4)\)\. Nevertheless, aggregate statistics may obscure intermittent spikes, termed*excess principal\-subspace overlap*, which often precede validation degradation\. Controlled interventions further show that amplifying this overlap causes dose\-dependent accuracy drops, motivating complementary\-subspace constraints\.

### 3\.1Preliminaries

For a pretrained weight matrixWr​e​f∈ℝdo​u​t×di​nW\_\{ref\}\\in\\mathbb\{R\}^\{d\_\{out\}\\times d\_\{in\}\}, letWr​e​f=Φ​Σ​Ψ⊤W\_\{ref\}=\\Phi\\Sigma\\Psi^\{\\top\}be its SVD\. We call the spans of the top\-kkleft and right singular vectors,Φk\\Phi\_\{k\}andΨk\\Psi\_\{k\}, its*principal subspaces*\. They capture high\-energy input–output directions of the pretrained linear operator; they are a structural proxy, not a literal partition of semantic knowledge\. LetΠΦ=Φk​Φk⊤,ΠΨ=Ψk​Ψk⊤\\Pi\_\{\\Phi\}=\\Phi\_\{k\}\\Phi\_\{k\}^\{\\top\},\\Pi\_\{\\Psi\}=\\Psi\_\{k\}\\Psi\_\{k\}^\{\\top\}be the associated projectors, andΠΦ⟂=I−ΠΦ,ΠΨ⟂=I−ΠΨ\\Pi\_\{\\Phi\}^\{\\perp\}=I\-\\Pi\_\{\\Phi\},\\Pi\_\{\\Psi\}^\{\\perp\}=I\-\\Pi\_\{\\Psi\}denote the projectors onto their orthogonal complements\.

We analyze the realized updateδ\(t\)​W=Wt−Wt−1\\delta^\{\(t\)\}W=W\_\{t\}\-W\_\{t\-1\}, rather than the raw gradient, so the measurement includes the optimizer, learning rate, and momentum\. Its energy can be decomposed into four mutually orthogonal blocks:

δ\(t\)​W=ΠΦ​δ\(t\)​W​ΠΨ⏟δ\(t\)​WP​P\+ΠΦ​δ\(t\)​W​ΠΨ⟂⏟δ\(t\)​WP​O\\displaystyle\\delta^\{\(t\)\}W=\\underbrace\{\\Pi\_\{\\Phi\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\}\}\_\{\\delta^\{\(t\)\}W^\{PP\}\}\+\\underbrace\{\\Pi\_\{\\Phi\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\}^\{\\perp\}\}\_\{\\delta^\{\(t\)\}W^\{PO\}\}\(1\)\+ΠΦ⟂​δ\(t\)​W​ΠΨ⏟δ\(t\)​WO​P\+ΠΦ⟂​δ\(t\)​W​ΠΨ⟂⏟δ\(t\)​WO​O\.\\displaystyle\+\\underbrace\{\\Pi\_\{\\Phi\}^\{\\perp\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\}\}\_\{\\delta^\{\(t\)\}W^\{OP\}\}\+\\underbrace\{\\Pi\_\{\\Phi\}^\{\\perp\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\}^\{\\perp\}\}\_\{\\delta^\{\(t\)\}W^\{OO\}\}\.LetEi​j=‖δ\(t\)​Wi​j‖F2E\_\{ij\}=\\\|\\delta^\{\(t\)\}W^\{ij\}\\\|\_\{F\}^\{2\}andEtotal=‖δ\(t\)​W‖F2E\_\{\\mathrm\{total\}\}=\\\|\\delta^\{\(t\)\}W\\\|\_\{F\}^\{2\}\. TheO​OOOblock is orthogonal to the principal subspace on both sides; the other three blocks have overlap on at least one side\. We summarize this overlap by

Ot=EP​P\+EP​O\+EO​PEtotal=1−EO​OEtotal\.O\_\{t\}=\\frac\{E\_\{PP\}\+E\_\{PO\}\+E\_\{OP\}\}\{E\_\{\\mathrm\{total\}\}\}=1\-\\frac\{E\_\{OO\}\}\{E\_\{\\mathrm\{total\}\}\}\.\(2\)

### 3\.2Observed Excess Principal\-subspace Overlap

This raw ratio depends partly on dimensionality: an isotropic update has nonzero overlap simply because the principal subspaces have dimensionkk\. For adout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}matrix, its expected overlap is

Onull=1−\(dout−k\)​\(din−k\)dout​din\.O\_\{\\mathrm\{null\}\}=1\-\\frac\{\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\)\}\{d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}\}\.\(3\)We therefore report the dimension\-corrected quantity

Otexcess=Ot−Onull\.O\_\{t\}^\{\\mathrm\{excess\}\}=O\_\{t\}\-O\_\{\\mathrm\{null\}\}\.\(4\)PositiveOtexcessO\_\{t\}^\{\\mathrm\{excess\}\}means that an update is more aligned with the principal subspaces than an isotropic update of the same size\. We refer to this quantity as*excess principal\-subspace overlap*\. It is an observable alignment statistic, not a claim that every principal direction is harmful\.

Figure[1](https://arxiv.org/html/2608.11674#S3.F1)shows two consistent findings\. First, most update energy remains inO​OOO\(93\.8% for Qwen3\-8B and 97\.7% for GLM4\-9B on average\), agreeing with the aggregate off\-principal trend\. Second, this average masks intermittent positive spikes inOtexcessO\_\{t\}^\{\\mathrm\{excess\}\}\. In the displayed runs, sustained or repeated spikes coincide with, and precede, declining validation accuracy; the effect appears as gradual deterioration in one setting and a sharp drop in another\. Layer\-wise measurements further show that these spikes are concentrated rather than uniform, often in intermediate and upper layers\. More results are provided in Appendix[E](https://arxiv.org/html/2608.11674#A5.SSx1)\.

#### Controlled intervention\.

To move beyond correlation, we intervene on a GRPO update of Qwen3\-8B on ToolAlpaca\. We rescale its principal\-overlapping component byη\\etawhile preserving the Frobenius norm of every layer\-wise update\. Details are provided in Appendix[D](https://arxiv.org/html/2608.11674#A4)\. As shown in Figure[2](https://arxiv.org/html/2608.11674#S3.F2), increasing overlap produces a clear dose\-dependent accuracy drop\. Relative to the original update \(η=1\\eta=1, 56\.89%\), removing the overlapping component improves accuracy, whereas principal\-subspace injection substantially reduces accuracy\. A matched random\-subspace intervention causes substantially less degradation\. These results provide local intervention evidence that increasing principal\-subspace overlap can directly harm model performance, beyond the effects of update magnitude or arbitrary directional perturbation\. Additional interventions across models, tasks, and checkpoints show the same dose\-dependent trend \(Appendix[E](https://arxiv.org/html/2608.11674#A5.SSx2)\)\.

![Refer to caption](https://arxiv.org/html/2608.11674v1/principal_intervention_combined.png)Figure 2:Controlled intervention on the step\-150 GRPO update of Qwen3\-8B on ToolAlpaca\. \(a\) Increasing the principal\-overlapping component under layer\-wise norm matching produces a dose\-dependent accuracy drop\. \(b\) Orthogonalization improves accuracy, whereas matched principal\-subspace injection is substantially more harmful than a random\-subspace intervention\.Together, the observational and intervention results motivate preserving the dominant input and output directions while optimizing in their bilateral orthogonal complement\. We implement this restriction next\.

## 4Methodology

Our diagnostic suggests a simple intervention: rather than modifying the rollout objective, restrict each policy update to the bilateral orthogonal complement of the dominant singular subspaces of the pretrained weights\. GCPO realizes this intervention within GRPO through a projected low\-rank parameterization\.

### 4\.1Problem Formulation

Letθ0\\theta\_\{0\}be the pretrained policy andℳ\\mathcal\{M\}the set of adapted linear layers\. For eachℓ∈ℳ\\ell\\in\\mathcal\{M\}, we compute once the rank\-kksingular subspacesΦk\(ℓ\)\\Phi\_\{k\}^\{\(\\ell\)\}andΨk\(ℓ\)\\Psi\_\{k\}^\{\(\\ell\)\}ofWref\(ℓ\)W\_\{\\mathrm\{ref\}\}^\{\(\\ell\)\}\. We seek a policy update that maximizes the usual rollout objective while having zero principal\-subspace overlap at every adapted layer\.

maxΔ​θ\\displaystyle\\max\_\{\\Delta\\theta\}\\quad𝒥rollout\(πθ0\+Δ​θ\)−β𝔻KL\(πθ0\+Δ​θ\|\|πref\),\\displaystyle\\mathcal\{J\}\_\{\\text\{rollout\}\}\(\\pi\_\{\\theta\_\{0\}\+\\Delta\\theta\}\)\-\\beta\\mathbb\{D\}\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\_\{0\}\+\\Delta\\theta\}\\,\|\|\\,\\pi\_\{\\text\{ref\}\}\),\(5\)s\.t\.Φk\(ℓ\)⊤​δ\(t\)​W\(ℓ\)=0,\\displaystyle\\Phi\_\{k\}^\{\(\\ell\)\\top\}\\delta^\{\(t\)\}W^\{\(\\ell\)\}=0,\(6\)δ\(t\)​W\(ℓ\)​Ψk\(ℓ\)=0,∀ℓ∈ℳ,\\displaystyle\\delta^\{\(t\)\}W^\{\(\\ell\)\}\\Psi\_\{k\}^\{\(\\ell\)\}=0,\\quad\\forall\\ell\\in\\mathcal\{M\},\(7\)The two constraints are equivalent to retaining only theO​OOOblock of each update\. They complement, rather than replace, KL regularization: KL controls policy change in output space, whereas the constraints control where the parameter update can lie\. A penalty could encourage these constraints, but would require trading them against the rollout objective\. We instead enforce them exactly by parameterization\.

### 4\.2GCPO: Geometrically Constrained Policy Optimization

For each adapted layer, with fixed orthogonal\-complement projectorsΠΦ⟂,ΠΨ⟂\\Pi\_\{\\Phi\}^\{\\perp\},\\Pi\_\{\\Psi\}^\{\\perp\}, we can parameterize its update as

δ\(t\)​W\(ℓ\)=α​ΠΦ⟂​L\(ℓ\)​R\(ℓ\)​ΠΨ⟂,\\delta^\{\(t\)\}W^\{\(\\ell\)\}=\\alpha\\Pi\_\{\\Phi\}^\{\\perp\}L^\{\(\\ell\)\}R^\{\(\\ell\)\}\\Pi\_\{\\Psi\}^\{\\perp\},\(8\)whereL\(ℓ\)L^\{\(\\ell\)\}andR\(ℓ\)R^\{\(\\ell\)\}are trainable low\-rank factors andα\\alphais the standard scaling constant\. The right projector removes components acting on dominant input directions; the left projector removes components written into dominant output directions\.

This parameterization satisfies both constraints for every value of the trainable factors in Eq\. \([6](https://arxiv.org/html/2608.11674#S4.E6)\) and Eq\. \([7](https://arxiv.org/html/2608.11674#S4.E7)\)\. Thus, even if rollout gradients contain principal\-subspace components, those components cannot change the effective layer update\. This is the central property of GCPO: it converts the observed preference for off\-principal updates into an exact feasible set, while leaving optimization within that set unchanged\.

### 4\.3Theoretical Properties and Comparison

#### Exact subspace preservation\.

GCPO satisfies the bilateral constraints at every adapted layer and every optimization step\. Consequently, the pretrained layer mapping is unchanged for inputs in the selected principal input subspace, with an analogous guarantee for the principal output subspace\. Formal proofs are provided in Appendix[B](https://arxiv.org/html/2608.11674#A2)\.

#### Capacity and cost\.

The feasible update space has dimension\(dout−k\)​\(din−k\)\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\), and every feasible rank\-rrupdate can be represented by the projected low\-rank parameterization\. Thus, forkksmall relative to the layer width, GCPO retains a large space for task adaptation\. Formal proofs are provided in Appendix[B](https://arxiv.org/html/2608.11674#A2)\.

#### Distinction from Low\-Rank Adaptation\.

Although GCPO uses low\-rank factorization for efficient implementation, it solves a different constrained optimization problem from conventional low\-rank adaptation\. LoRA primarily restricts update rank for parameter efficiency, while some geometric variants shape update directions to prevent catastrophic forgetting typically under fixed\-data supervised fine\-tuning\([20](https://arxiv.org/html/2608.11674#bib.bib18);[26](https://arxiv.org/html/2608.11674#bib.bib5);[4](https://arxiv.org/html/2608.11674#bib.bib6);[27](https://arxiv.org/html/2608.11674#bib.bib31)\)\. In contrast, GCPO is motivated by transient*principal\-subspace overlap*under on\-policy rollout feedback in RL, where each policy update changes the subsequent training distribution\. It therefore constrains update direction through bilateral orthogonality to stabilize stepwise RL dynamics and preserve pretrained mappings, while low rank serves mainly as an efficient parameterization of this feasible space\.

## 5Experiments

In this section, we evaluate GCPO in terms of general\-task performance, cross\-task capability retention, training stability and efficiency, and design effectiveness through ablation and structural analyses\.

#### Models and Tasks\.

We adopt two widely used instruction\-tuned LLMs as our backbone models: Qwen3\-8B\([30](https://arxiv.org/html/2608.11674#bib.bib14)\)and GLM4\-9B\([7](https://arxiv.org/html/2608.11674#bib.bib15)\), and evaluate them across three representative domains using established benchmarks and consistent protocols across methods:

- •Mathematical Reasoning:Evaluated on the MATH500\([16](https://arxiv.org/html/2608.11674#bib.bib17)\)\. The reward is computed via exact matching of the final boxed answer using rule\-based parsing\.
- •Code Generation:Evaluated on HumanEval\+\([17](https://arxiv.org/html/2608.11674#bib.bib21)\)for Python function synthesis\. The reward is granted based on the execution pass rate across unit tests\.
- •Tool Use:Evaluated on ToolAlpaca\([25](https://arxiv.org/html/2608.11674#bib.bib16)\), which requires generating the appropriate API call given a tool specification and a user request\. The reward is based on exact function\-name matching, argument\-key consistency, and normalized argument\-value matching\.

#### Baselines\.

We extensively compare GCPO against a comprehensive suite of optimization strategies\. These include GRPO\([22](https://arxiv.org/html/2608.11674#bib.bib2)\), objective\-level variants including GSPO\([34](https://arxiv.org/html/2608.11674#bib.bib13)\), DAPO\([32](https://arxiv.org/html/2608.11674#bib.bib12)\), and GMPO\([33](https://arxiv.org/html/2608.11674#bib.bib11)\), which modify objectives or clip probability ratios\. To compare with parameter\-space constraints, we include GRPO\-LoRA\([12](https://arxiv.org/html/2608.11674#bib.bib7)\), which uses unconstrained low\-rank adaptation without the hard bilateral orthogonality enforced by GCPO\. To isolate the effect of the directional constraint from low\-rank parameterization, GRPO\-LoRA and GCPO use the same adaptation rank and scaling configuration\.

#### Implementation Details\.

For all GRPO\-based methods, we sampleK=16K=16rollouts per prompt during training\. We adopt the same hyperparameters to ensure equity\. For GCPO, we precompute the top\-k=8k=8singular subspaces and constrain updates to their bilateral orthogonal complements\. For evaluation, we report bootstrap\-estimated majority@16 accuracy from 16 test\-time responses per example\. More details on seeds, data splits, evaluation, and hyperparameters are provided in Appendix[D](https://arxiv.org/html/2608.11674#A4)\.

### 5\.1Main Results

Table[1](https://arxiv.org/html/2608.11674#S5.T1)reports the mean and standard deviation over three independent training seeds across three task domains and two backbone models\. GCPO achieves the best accuracy in all six model–task settings\. Relative to the corresponding instruction\-tuned base models, it improves accuracy by7\.097\.09–27\.6927\.69percentage points, and surpasses the strongest competing method on each benchmark by1\.021\.02–2\.372\.37points\. Moreover, GCPO exhibits the lowest standard deviation in all six settings, indicating reduced sensitivity to training stochasticity\. The margins are particularly pronounced on GLM4\-9B, where GCPO outperforms the strongest baselines by2\.152\.15–2\.372\.37points, demonstrating consistent effectiveness across backbones with different optimization behaviors\.

Beyond individual benchmarks, GCPO also achieves the strongest average performance\. On Qwen3\-8B, it obtains an average accuracy of78\.6378\.63, exceeding the strongest baseline average of77\.4877\.48by1\.151\.15points\. On GLM4\-9B, it improves the strongest baseline average from73\.7373\.73to76\.1276\.12\. Averaged across all six model–task settings, GCPO reaches77\.3877\.38, compared with75\.6175\.61for the strongest baseline, which suggests effectiveness and generalizability\.

Crucially, we emphasize that these gains are not merely artifacts of low\-rank parameter reduction: as shown in Table[1](https://arxiv.org/html/2608.11674#S5.T1), GRPO\-LoRA with the same rank achieves consistently lower accuracy than GCPO across all evaluations\. This confirms that the explicit directional orthogonal projection is the primary driver of our method’s superiority\.

Table 1:Main results across tasks\. We report bootstrap\-estimated majority@16 accuracy as mean±\\pmstandard deviation over three training seeds, in percentage points\.
### 5\.2Cross\-Task Capability Preservation

Single\-domain RL post\-training may improve the target task while degrading other capabilities\. To examine whether our geometric constraint better preserves such general capabilities, we post\-train models on MATH500 and evaluate their cross\-task capabilities on code generation \(HumanEval\+\) and tool\-use \(ToolAlpaca\) benchmarks\.

#### Existing RL variants poorly retain tool\-use capability\.

As shown in Table[2](https://arxiv.org/html/2608.11674#S5.T2), GRPO and its variants generally exhibit substantial degradation on ToolAlpaca after MATH500 post\-training\. This drop is most severe for GRPO, with a large decline of−14\.97\-14\.97on GLM4\-9B\. These results suggest that conventional RL strategies are insufficient to preserve general capabilities beyond the reward\-optimized domain\.

#### Mathematical reasoning can transfer to code generation\.

In contrast, RL training on math tasks has a milder effect on coding capabilities\. Several variants even improve code generation performance, indicating positive transfer from mathematical reasoning to coding, likely because both tasks rely on step\-by\-step reasoning and symbolic problem solving\.

#### GCPO best preserves general capabilities\.

GCPO obtains the best WorstΔ\\Deltascores on both Qwen3\-8B and GLM4\-9B, with gains of\+1\.03\+1\.03and\+0\.91\+0\.91, while also improving HumanEval\+ by\+3\.99\+3\.99and\+5\.88\+5\.88, respectively\. These results indicate that constraining updates away from dominant pretrained subspaces better balances task\-specific RL adaptation with the preservation of general capabilities\.

MethodHumanEval\+ToolAlpacaWorstΔ\\Delta↑\\uparrowQwen3\-8BBase \(Instruct\)73\.5856\.530\.00GRPO64\.71\(−8\.87\)\(\-8\.87\)50\.72\(−5\.81\)\(\-5\.81\)−8\.87\-8\.87GSPO76\.82\(\+3\.24\)\(\+3\.24\)53\.41\(−3\.12\)\(\-3\.12\)−3\.12\-3\.12DAPO75\.54\(\+1\.96\)\(\+1\.96\)53\.09\(−3\.44\)\(\-3\.44\)−3\.44\-3\.44GMPO74\.31\(\+0\.73\)\(\+0\.73\)52\.90\(−3\.63\)\(\-3\.63\)−3\.63\-3\.63GRPO\-LoRA74\.63\(\+1\.05\)\(\+1\.05\)56\.69\(\+0\.16\)\(\+0\.16\)\+0\.16\+0\.16GCPO77\.57\(\+3\.99\)\(\+3\.99\)57\.56\(\+1\.03\)\(\+1\.03\)\+1\.03\+1\.03GLM4\-9BBase \(Instruct\)76\.5542\.470\.00GRPO76\.91\(\+0\.36\)\(\+0\.36\)27\.50\(−14\.97\)\(\-14\.97\)−14\.97\-14\.97GSPO81\.58\(\+5\.03\)\(\+5\.03\)41\.29\(−1\.18\)\(\-1\.18\)−1\.18\-1\.18DAPO81\.39\(\+4\.84\)\(\+4\.84\)37\.68\(−4\.79\)\(\-4\.79\)−4\.79\-4\.79GMPO79\.76\(\+3\.21\)\(\+3\.21\)41\.33\(−1\.14\)\(\-1\.14\)−1\.14\-1\.14GRPO\-LoRA81\.00\(\+4\.45\)\(\+4\.45\)43\.14\(\+0\.67\)\(\+0\.67\)\+0\.67\+0\.67GCPO82\.43\(\+5\.88\)\(\+5\.88\)43\.38\(\+0\.91\)\(\+0\.91\)\+0\.91\+0\.91Table 2:Cross\-task capability retention after training on MATH500 and evaluated on HumanEval\+ and ToolAlpaca benchmarks\. Values in parentheses denote absolute accuracy changes relative to the base model\. WorstΔ\\Deltadenotes the smaller change across the two evaluation tasks\.

### 5\.3Optimization Dynamics of GCPO

We investigate the underlying optimization dynamics of GCPO, explicitly focusing on its ability to stabilize the training trajectory, suppress shortcut learning behaviors, and maintain hardware efficiency\.

#### Robust Training Stability\.

Figure[3](https://arxiv.org/html/2608.11674#S5.F3)illustrates the accuracy trajectories evaluated every 5 steps during training on the ToolAlpaca for Qwen3\-8B\. GRPO exhibits severe performance oscillations and high\-variance fluctuations, whereas GCPO yields a smooth, continuously increasing accuracy curve\. This is consistent with our finding that our bilateral orthogonal constraint effectively acts as a structural barrier, preventing variance\-heavy gradients from intruding into the principal subspace and ensuring stable, directional policy improvement\.

Figure 3:Training stability and accuracy trajectory on ToolAlpaca with Qwen3\-8B, evaluated every 5 training steps\.
#### Stable Policy Entropy Dynamics\.

We further examine the policy entropy during RL post\-training to characterize the exploration–exploitation behavior of different methods\. A desirable optimization trajectory should gradually reduce entropy, while avoiding both entropy oscillations and premature entropy collapse\([10](https://arxiv.org/html/2608.11674#bib.bib26);[3](https://arxiv.org/html/2608.11674#bib.bib27)\)\. As shown in Figure[4](https://arxiv.org/html/2608.11674#S5.F4), GRPO exhibits large entropy oscillations, while most baselines exhibit rapid entropy decay, suggesting premature overconfident exploitation and reduced policy diversity\. GCPO instead maintains a smooth, gradual decay, suggesting more controlled policy specialization without rapid loss of diversity\.

Figure 4:Policy entropy dynamics on MATH500 with GLM4\-9B\. GCPO maintains a smooth and gradual entropy decay, avoiding both the severe oscillations of GRPO and the premature entropy collapse in other baselines\.
#### Resistance to Response\-Length Inflation\.

A notorious failure mode in rollout\-based RL is*response\-length inflation*, where the policy learns a shortcut to hack rewards by generating excessively long, redundant tokens rather than improving genuine reasoning\. As shown in Figure[5](https://arxiv.org/html/2608.11674#S5.F5), GRPO suffers from aggressive and uncontrolled output length expansion on MATH500\. Conversely, GCPO substantially reduces response length, maintaining concise and stable generation lengths\. We hypothesize that shielding dominant pretrained directions—including those encoding length priors—from high\-variance gradients reduces the policy’s tendency to exploit verbosity as a reward\-hacking shortcut\.

Figure 5:Response length on MATH500 with Qwen3\-8B\. GCPO most effectively mitigates response\-length inflation\.
#### Superior Memory Efficiency\.

Figure[6](https://arxiv.org/html/2608.11674#S5.F6)compares peak GPU memory under matched GRPO training configurations\. Full\-parameter GRPO requires substantially more memory because optimizer states and gradients are maintained for all adapted dense weights\. Both GRPO\-LoRA and GCPO reduce this cost through low\-rank adaptation\. Importantly, GCPO achieves comparable peak memory to LoRA, showing that the bilateral geometric constraint does not sacrifice the memory efficiency of parameter\-efficient training\. The principal singular vectors are frozen and require no optimizer states, while the learned update can be merged into the pretrained weights for inference\.

Figure 6:Peak per\-GPU training memory on MATH500 with GLM4\-9B, where G\-LoRA abbreviates GRPO\-LoRA\.

### 5\.4Ablation Studies

#### Orthogonal Constraints Prevent Capability Collapse\.

Table[3](https://arxiv.org/html/2608.11674#S5.T3)reveals that the target subspace strongly affects optimization performance\. Forcing updates into the*Principal*subspace triggers a severe performance collapse, indicating that altering dominant singular directions disrupts task\-relevant capabilities\. Conversely, the*Orthogonal*complement achieves peak accuracy, substantially outperforming*Random*projections\. By structurally routing updates away from the core manifold, it safely resolves the conflict between aggressive task adaptation and knowledge retention\.

#### Bilateral Projections Seal Collapse Pathways\.

Unilateral constraints on solely the*left*\(output\) or*right*\(input\) singular spaces yield only marginal gains over the unconstrained baseline\. Geometrically, securing just one side leaves the opposite subspace vulnerable to dominant task gradients, permitting partial overlap\. The*bilateral*projection is mathematically necessary to seal both pathways simultaneously, ensuring zero principal\-subspace overlap and optimal accuracy\.

#### Hard Constraints Survive Iterative Rollout Dynamics\.

Both KL divergence and explicit soft orthogonality penalties \(‖Φk⊤​δ​W‖F2\+‖δ​W​Ψk‖F2\\\|\\Phi\_\{k\}^\{\\top\}\\delta W\\\|\_\{F\}^\{2\}\+\\\|\\delta W\\Psi\_\{k\}\\\|\_\{F\}^\{2\}\) underperform our approach\. Under the iterative feedback loops of RL, dominant task gradients inevitably overwhelm soft loss terms, allowing overlap to persist\. By enforcing orthogonality by construction, our*Hard*constraint robustly shields the pretrained manifold\.

Table 3:Geometric design ablations on MATH500 benchmark with GLM4\-9B, considering different projection directions, subspace selections, and constraint mechanisms\.
#### The Protected Dimension Balances Protection and Adaptation\.

Performance varies substantially withkkand peaks atk=8k=8on both tasks \(Figure[7](https://arxiv.org/html/2608.11674#S5.F7)\)\. Smallerkkprovides insufficient protection, whereas largerkkoverly restricts the feasible update space, highlighting the importance ofkkin balancing subspace protection and adaptation capacity\.

Figure 7:Effect ofkkon MATH500 and ToolAlpaca with Qwen3\-8B, reported as mean±\\pmstandard deviation\.

### 5\.5Limitations

Our study focuses on the parameter\-space dynamics of on\-policy RL, and it remains unclear whether the observed geometric patterns also extend to other post\-training paradigms, such as DPO\([29](https://arxiv.org/html/2608.11674#bib.bib22)\), KTO\([5](https://arxiv.org/html/2608.11674#bib.bib23)\), and OPD\([14](https://arxiv.org/html/2608.11674#bib.bib24)\)\. Although principal\-subspace overlap is consistently associated with performance degradation, its causal relationship with model instability, rollout behaviors such as length inflation and reward hacking is not yet fully understood\. Moreover, future work should examine broader alignment objectives, model scales, and training settings, while adaptive, layer\-wise selection ofkk, to assess the generality of this principal\-subspace overlap and develop more stable and capability\-preserving post\-training\.

## 6Conclusion

We study rollout\-based RL instability through the geometry of step\-wise parameter updates\. We identify transient principal\-subspace overlap as a warning signal associated with performance degradation, with controlled interventions providing local evidence of its harmful effect\. Based on this diagnosis, we propose GCPO, which restricts effective updates to the bilateral orthogonal complements of pretrained principal subspaces\. Across two backbones and three task domains, GCPO achieves the highest mean performance overall, while improving capability retention, response\-length control, and policy\-entropy stability\. These results highlight parameter\-space geometry as a useful diagnostic and design perspective for stable LLM reinforcement learning\.

## Acknowledgement

We gratefully acknowledge NovaCore for the valuable guidance and resources that supported this research\. Additionally, We thank the open\-source community for releasing code and datasets that made this research possible\.

## References

- Achiamet al\.\(2017\)J\. Achiam, D\. Held, A\. Tamar, and P\. AbbeelConstrained policy optimization\.InInternational conference on machine learning,pp\. 22–31\.Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px2.p1.1)\.
- Caiet al\.\(2026\)Y\. Cai, D\. Cao, L\. Lin, C\. Luo, X\. Xu, K\. Yang, W\. Liu, S\. Yang, T\. Zhao, G\. Sun, G\. Liu, and J\. FangLearning to foresee: unveiling the unlocking efficiency of on\-policy distillation\.External Links:2605\.11739,[Link](https://arxiv.org/abs/2605.11739)Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2608.11674#S3.p1.1)\.
- Cuiet al\.\(2025\)G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen,et al\.The entropy mechanism of reinforcement learning for reasoning language models\.arXiv preprint arXiv:2505\.22617\.Cited by:[§5\.3](https://arxiv.org/html/2608.11674#S5.SS3.SSS0.Px2.p1.1)\.
- Dragoiet al\.\(2026\)M\. Dragoi, I\. Pintilie, A\. Dragomir, A\. Barbalau, and F\. BradTailLoR: protecting principal components in parameter\-efficient continual learning\.External Links:2606\.06494,[Link](https://arxiv.org/abs/2606.06494)Cited by:[§4\.3](https://arxiv.org/html/2608.11674#S4.SS3.SSS0.Px3.p1.1)\.
- Ethayarajhet al\.\(2024\)K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. KielaKto: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[§5\.5](https://arxiv.org/html/2608.11674#S5.SS5.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p1.1)\.
- GLMet al\.\(2024\)T\. GLM, A\. Zeng, B\. Xu, B\. Wang, C\. Zhang, D\. Yin, D\. Rojas, G\. Feng, H\. Zhao, H\. Lai, H\. Yu, H\. Wang, J\. Sun, J\. Zhang, J\. Cheng, J\. Gui, J\. Tang, J\. Zhang, J\. Li, L\. Zhao, L\. Wu, L\. Zhong, M\. Liu, M\. Huang, P\. Zhang, Q\. Zheng, R\. Lu, S\. Duan, S\. Zhang, S\. Cao, S\. Yang, W\. L\. Tam, W\. Zhao, X\. Liu, X\. Xia, X\. Zhang, X\. Gu, X\. Lv, X\. Liu, X\. Liu, X\. Yang, X\. Song, X\. Zhang, Y\. An, Y\. Xu, Y\. Niu, Y\. Yang, Y\. Li, Y\. Bai, Y\. Dong, Z\. Qi, Z\. Wang, Z\. Yang, Z\. Du, Z\. Hou, and Z\. WangChatGLM: a family of large language models from glm\-130b to glm\-4 all tools\.External Links:2406\.12793Cited by:[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px1.p1.1)\.
- Golubet al\.\(1987\)G\. H\. Golub, A\. Hoffman, and G\. W\. StewartA generalization of the eckart\-young\-mirsky matrix approximation theorem\.Linear Algebra and its applications88,pp\. 317–327\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p3.1)\.
- Grontaset al\.\(2025\)P\. D\. Grontas, A\. Terpin, E\. C\. Balta, R\. D’Andrea, and J\. LygerosPinet: optimizing hard\-constrained neural networks with orthogonal projection layers\.arXiv preprint arXiv:2508\.10480\.Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px2.p1.1)\.
- Haoet al\.\(2026\)Z\. Hao, H\. Wang, H\. Liu, J\. Luo, J\. Yu, H\. Dong, Q\. Lin, C\. Wang, and J\. ChenRethinking entropy interventions in RLVR: an entropy change perspective\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 31105–31133\.External Links:[Link](https://aclanthology.org/2026.acl-long.1436/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1436),ISBN 979\-8\-89176\-390\-6Cited by:[§5\.3](https://arxiv.org/html/2608.11674#S5.SS3.SSS0.Px2.p1.1)\.
- Harmonet al\.\(2025\)J\. Harmon, A\. Hochlehnert, M\. Bethge, and A\. PrabhuMapping post\-training forgetting in language models at scale\.arXiv preprint arXiv:2510\.17776\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px2.p1.1)\.
- Hübotteret al\.\(2026\)J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin, and A\. KrauseReinforcement learning via self\-distillation\.External Links:2601\.20802,[Link](https://arxiv.org/abs/2601.20802)Cited by:[Appendix D](https://arxiv.org/html/2608.11674#A4.SSx1.p1.1)\.
- Kumaret al\.\(2025\)K\. Kumar, T\. Ashraf, O\. Thawakar, R\. M\. Anwer, H\. Cholakkal, M\. Shah, M\. Yang, P\. H\. Torr, F\. S\. Khan, and S\. KhanLlm post\-training: a deep dive into reasoning large language models\.arXiv preprint arXiv:2502\.21321\.Cited by:[§5\.5](https://arxiv.org/html/2608.11674#S5.SS5.p1.1)\.
- Liet al\.\(2026\)Z\. Li, J\. Lou, F\. Dong, Z\. Fan, M\. Ren, H\. Lin, X\. Han, D\. Zhang, L\. Sun, Y\. Lu,et al\.Tackling length inflation without trade\-offs: group relative reward rescaling for reinforcement learning\.arXiv preprint arXiv:2603\.10535\.Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px2.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[Appendix D](https://arxiv.org/html/2608.11674#A4.SSx1.p2.1),[1st item](https://arxiv.org/html/2608.11674#S5.I1.i1.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=1qvx610Cu7)Cited by:[Appendix D](https://arxiv.org/html/2608.11674#A4.SSx1.p2.1),[2nd item](https://arxiv.org/html/2608.11674#S5.I1.i2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p1.1)\.
- Schneideret al\.\(2024\)J\. Schneider, P\. Schumacher, S\. Guist, L\. Chen, D\. Häufle, B\. Schölkopf, and D\. BüchlerIdentifying policy gradient subspaces\.External Links:2401\.06604,[Link](https://arxiv.org/abs/2401.06604)Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p3.1)\.
- Schotthöferet al\.\(2024\)S\. Schotthöfer, E\. Zangrando, G\. Ceruti, F\. Tudisco, and J\. KuschGeoLoRA: geometric integration for parameter efficient fine\-tuning\.External Links:2410\.18720,[Link](https://arxiv.org/abs/2410.18720)Cited by:[§4\.3](https://arxiv.org/html/2608.11674#S4.SS3.SSS0.Px3.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p1.1),[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px2.p1.1)\.
- Shenet al\.\(2026\)Z\. Shen, Y\. Li, Q\. Yin, C\. T\. Leong, Z\. Wang, Y\. Chen, R\. Han, S\. Lee, and Y\. R\. FungOn the geometry of on\-policy distillation\.External Links:2606\.07082,[Link](https://arxiv.org/abs/2606.07082)Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p3.1),[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2608.11674#S3.p1.1)\.
- Singhalet al\.\(2023\)P\. Singhal, T\. Goyal, J\. Xu, and G\. DurrettA long way to go: investigating length correlations in rlhf\.arXiv preprint arXiv:2310\.03716\.Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px2.p1.1)\.
- Tanget al\.\(2023\)Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. SunToolalpaca: generalized tool learning for language models with 3000 simulated cases\.arXiv preprint arXiv:2306\.05301\.Cited by:[Appendix D](https://arxiv.org/html/2608.11674#A4.SSx1.p3.1),[3rd item](https://arxiv.org/html/2608.11674#S5.I1.i3.p1.1)\.
- Wanget al\.\(2025\)H\. Wang, Y\. Li, S\. Wang, G\. Chen, and Y\. ChenMiLoRA: harnessing minor singular components for parameter\-efficient LLM finetuning\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4823–4836\.External Links:[Link](https://aclanthology.org/2025.naacl-long.248/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.248),ISBN 979\-8\-89176\-189\-6Cited by:[§4\.3](https://arxiv.org/html/2608.11674#S4.SS3.SSS0.Px3.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. HuangOrthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10658–10671\.Cited by:[§4\.3](https://arxiv.org/html/2608.11674#S4.SS3.SSS0.Px3.p1.1)\.
- Wanget al\.\(2022\)X\. Wang, S\. Wang, X\. Liang, D\. Zhao, J\. Huang, X\. Xu, B\. Dai, and Q\. MiaoDeep reinforcement learning: a survey\.IEEE Transactions on Neural Networks and Learning Systems35\(4\),pp\. 5064–5078\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p1.1)\.
- Xuet al\.\(2024\)S\. Xu, W\. Fu, J\. Gao, W\. Ye, W\. Liu, Z\. Mei, G\. Wang, C\. Yu, and Y\. WuIs dpo superior to ppo for llm alignment? a comprehensive study\.External Links:2404\.10719,[Link](https://arxiv.org/abs/2404.10719)Cited by:[§5\.5](https://arxiv.org/html/2608.11674#S5.SS5.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2026a\)G\. Yu, W\. Liu, Y\. Hu, H\. Ma, J\. Jiang, and H\. YeDense supervision, sparse updates: on the sparsity and geometry of on\-policy distillation\.External Links:2606\.13657,[Link](https://arxiv.org/abs/2606.13657)Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p3.1)\.
- Yuet al\.\(2026b\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p2.1),[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2025\)Y\. Zhao, Y\. Liu, J\. Liu, J\. Chen, X\. Wu, Y\. Hao, T\. Lv, S\. Huang, L\. Cui, Q\. Ye,et al\.Geometric\-mean policy optimization\.arXiv preprint arXiv:2507\.20673\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p2.1),[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[§1](https://arxiv.org/html/2608.11674#S1.p2.1),[§2](https://arxiv.org/html/2608.11674#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.11674#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix ANotations

Table[4](https://arxiv.org/html/2608.11674#A1.T4)summarizes the main notation used in our geometric analysis and method\. The superscriptℓ\\elldenotes a linear layer, andttdenotes an optimization step\. We omit the layer index when it is clear from context\.

Table 4:Summary of the main notation\.
## Appendix BProofs

We provide proofs for the geometric properties used in the diagnostic measure and GCPO formulation\. For clarity, we omit the layer indexℓ\\ellunless necessary\.

### Orthogonality of the Update Decomposition

###### Lemma B\.1\.

The four blocksδ\(t\)​WP​P\\delta^\{\(t\)\}W^\{PP\},δ\(t\)​WP​O\\delta^\{\(t\)\}W^\{PO\},δ\(t\)​WO​P\\delta^\{\(t\)\}W^\{OP\}, andδ\(t\)​WO​O\\delta^\{\(t\)\}W^\{OO\}are mutually orthogonal under the Frobenius inner product\. Consequently,

‖δ\(t\)​W‖F2=EP​P\+EP​O\+EO​P\+EO​O\.\\\|\\delta^\{\(t\)\}W\\\|\_\{F\}^\{2\}=E\_\{PP\}\+E\_\{PO\}\+E\_\{OP\}\+E\_\{OO\}\.\(9\)

###### Proof\.

The principal and complementary projectors satisfy

ΠΦ​ΠΦ⟂=0,ΠΨ​ΠΨ⟂=0\.\\Pi\_\{\\Phi\}\\Pi\_\{\\Phi\}^\{\\perp\}=0,\\qquad\\Pi\_\{\\Psi\}\\Pi\_\{\\Psi\}^\{\\perp\}=0\.\(10\)Consider, for example, theP​PPPandP​OPOblocks\. Their Frobenius inner product is

⟨ΠΦ​δ\(t\)​W​ΠΨ,ΠΦ​δ\(t\)​W​ΠΨ⟂⟩F\\displaystyle\\left\\langle\\Pi\_\{\\Phi\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\},\\Pi\_\{\\Phi\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\}^\{\\perp\}\\right\\rangle\_\{F\}=tr⁡\(ΠΨ​δ\(t\)​W⊤​ΠΦ​δ\(t\)​W​ΠΨ⟂\)=0,\\displaystyle=\\operatorname\{tr\}\\left\(\\Pi\_\{\\Psi\}\\delta^\{\(t\)\}W^\{\\top\}\\Pi\_\{\\Phi\}\\delta^\{\(t\)\}W\\Pi\_\{\\Psi\}^\{\\perp\}\\right\)=0,\(11\)where the last equality follows from cyclic invariance of the trace andΠΨ⟂​ΠΨ=0\\Pi\_\{\\Psi\}^\{\\perp\}\\Pi\_\{\\Psi\}=0\. The remaining pairs follow analogously from orthogonality of the left or right projectors\. The squared norm of their sum is therefore the sum of their squared norms\. ∎

It follows immediately that the fraction of energy overlapping at least one principal subspace is

Ot=EP​P\+EP​O\+EO​PEtotal=1−EO​OEtotal\.O\_\{t\}=\\frac\{E\_\{PP\}\+E\_\{PO\}\+E\_\{OP\}\}\{E\_\{\\mathrm\{total\}\}\}=1\-\\frac\{E\_\{OO\}\}\{E\_\{\\mathrm\{total\}\}\}\.\(12\)

### Expected Overlap of an Isotropic Update

###### Proposition B\.2\.

Letδ​W∈ℝdout×din\\delta W\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}have a uniformly random direction under the Frobenius norm\. Then its expected principal\-subspace overlap is

𝔼⁡\[O\]=1−\(dout−k\)​\(din−k\)dout​din\.\\mathbb\{E\}\[O\]=1\-\\frac\{\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\)\}\{d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}\}\.\(13\)

###### Proof\.

Let

z=vec⁡\(δ​W\)∈ℝD,D=dout​din\.z=\\operatorname\{vec\}\(\\delta W\)\\in\\mathbb\{R\}^\{D\},\\qquad D=d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}\.\(14\)Using the vectorization identity, the doubly orthogonal component can be written as

vec⁡\(ΠΦ⟂​δ​W​ΠΨ⟂\)=Q​z,Q=ΠΨ⟂⊗ΠΦ⟂\.\\operatorname\{vec\}\\left\(\\Pi\_\{\\Phi\}^\{\\perp\}\\delta W\\Pi\_\{\\Psi\}^\{\\perp\}\\right\)=Qz,\\qquad Q=\\Pi\_\{\\Psi\}^\{\\perp\}\\otimes\\Pi\_\{\\Phi\}^\{\\perp\}\.\(15\)The matrixQQis an orthogonal projector with rank

rank⁡\(Q\)=\(dout−k\)​\(din−k\)\.\\operatorname\{rank\}\(Q\)=\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\)\.\(16\)For a uniformly random direction,

𝔼⁡\[z​z⊤‖z‖22\]=IDD\.\\mathbb\{E\}\\left\[\\frac\{zz^\{\\top\}\}\{\\\|z\\\|\_\{2\}^\{2\}\}\\right\]=\\frac\{I\_\{D\}\}\{D\}\.\(17\)Therefore,

𝔼⁡\[‖Q​z‖22‖z‖22\]\\displaystyle\\mathbb\{E\}\\left\[\\frac\{\\\|Qz\\\|\_\{2\}^\{2\}\}\{\\\|z\\\|\_\{2\}^\{2\}\}\\right\]=tr⁡\(Q​𝔼​\[z​z⊤‖z‖22\]\)\\displaystyle=\\operatorname\{tr\}\\left\(Q\\,\\mathbb\{E\}\\left\[\\frac\{zz^\{\\top\}\}\{\\\|z\\\|\_\{2\}^\{2\}\}\\right\]\\right\)\(18\)=rank⁡\(Q\)D\\displaystyle=\\frac\{\\operatorname\{rank\}\(Q\)\}\{D\}\(19\)=\(dout−k\)​\(din−k\)dout​din\.\\displaystyle=\\frac\{\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\)\}\{d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}\}\.\(20\)SinceO=1−‖Q​z‖22/‖z‖22O=1\-\\\|Qz\\\|\_\{2\}^\{2\}/\\\|z\\\|\_\{2\}^\{2\}, the stated result follows\. ∎

This result gives the dimensionality\-dependent null valueOnullO\_\{\\mathrm\{null\}\}used in the excess\-overlap statistic\.

### Exact Feasibility and Subspace Preservation

###### Proposition B\.3\.

Suppose the cumulative adaptation of layerℓ\\ellis parameterized as

Δ​Wt\(ℓ\)=s​ΠΦ\(ℓ\)⟂​Lt\(ℓ\)​Rt\(ℓ\)​ΠΨ\(ℓ\)⟂\.\\Delta W\_\{t\}^\{\(\\ell\)\}=s\\Pi\_\{\\Phi\}^\{\(\\ell\)\\perp\}L\_\{t\}^\{\(\\ell\)\}R\_\{t\}^\{\(\\ell\)\}\\Pi\_\{\\Psi\}^\{\(\\ell\)\\perp\}\.\(21\)Then both the cumulative adaptation and every realized update satisfy the bilateral constraints:

Φk\(ℓ\)⊤​Δ​Wt\(ℓ\)=0,Δ​Wt\(ℓ\)​Ψk\(ℓ\)=0,\\Phi\_\{k\}^\{\(\\ell\)\\top\}\\Delta W\_\{t\}^\{\(\\ell\)\}=0,\\qquad\\Delta W\_\{t\}^\{\(\\ell\)\}\\Psi\_\{k\}^\{\(\\ell\)\}=0,\(22\)and

Φk\(ℓ\)⊤​δ\(t\)​W\(ℓ\)=0,δ\(t\)​W\(ℓ\)​Ψk\(ℓ\)=0\.\\Phi\_\{k\}^\{\(\\ell\)\\top\}\\delta^\{\(t\)\}W^\{\(\\ell\)\}=0,\\qquad\\delta^\{\(t\)\}W^\{\(\\ell\)\}\\Psi\_\{k\}^\{\(\\ell\)\}=0\.\(23\)

###### Proof\.

By construction,

Φk\(ℓ\)⊤​ΠΦ\(ℓ\)⟂=0,ΠΨ\(ℓ\)⟂​Ψk\(ℓ\)=0\.\\Phi\_\{k\}^\{\(\\ell\)\\top\}\\Pi\_\{\\Phi\}^\{\(\\ell\)\\perp\}=0,\\qquad\\Pi\_\{\\Psi\}^\{\(\\ell\)\\perp\}\\Psi\_\{k\}^\{\(\\ell\)\}=0\.\(24\)Hence,

Φk\(ℓ\)⊤​Δ​Wt\(ℓ\)\\displaystyle\\Phi\_\{k\}^\{\(\\ell\)\\top\}\\Delta W\_\{t\}^\{\(\\ell\)\}=s​Φk\(ℓ\)⊤​ΠΦ\(ℓ\)⟂​Lt\(ℓ\)​Rt\(ℓ\)​ΠΨ\(ℓ\)⟂=0,\\displaystyle=s\\Phi\_\{k\}^\{\(\\ell\)\\top\}\\Pi\_\{\\Phi\}^\{\(\\ell\)\\perp\}L\_\{t\}^\{\(\\ell\)\}R\_\{t\}^\{\(\\ell\)\}\\Pi\_\{\\Psi\}^\{\(\\ell\)\\perp\}=0,\(25\)Δ​Wt\(ℓ\)​Ψk\(ℓ\)\\displaystyle\\Delta W\_\{t\}^\{\(\\ell\)\}\\Psi\_\{k\}^\{\(\\ell\)\}=s​ΠΦ\(ℓ\)⟂​Lt\(ℓ\)​Rt\(ℓ\)​ΠΨ\(ℓ\)⟂​Ψk\(ℓ\)=0\.\\displaystyle=s\\Pi\_\{\\Phi\}^\{\(\\ell\)\\perp\}L\_\{t\}^\{\(\\ell\)\}R\_\{t\}^\{\(\\ell\)\}\\Pi\_\{\\Psi\}^\{\(\\ell\)\\perp\}\\Psi\_\{k\}^\{\(\\ell\)\}=0\.\(26\)
The realized update is

δ\(t\)​W\(ℓ\)=Δ​Wt\(ℓ\)−Δ​Wt−1\(ℓ\)\.\\delta^\{\(t\)\}W^\{\(\\ell\)\}=\\Delta W\_\{t\}^\{\(\\ell\)\}\-\\Delta W\_\{t\-1\}^\{\(\\ell\)\}\.\(27\)Because the constraints are linear and both cumulative adaptations satisfy them, their difference satisfies them as well\. ∎

###### Corollary B\.4\.

For everyx∈span⁡\(Ψk\(ℓ\)\)x\\in\\operatorname\{span\}\(\\Psi\_\{k\}^\{\(\\ell\)\}\)andy∈span⁡\(Φk\(ℓ\)\)y\\in\\operatorname\{span\}\(\\Phi\_\{k\}^\{\(\\ell\)\}\),

Wt\(ℓ\)​x=Wref\(ℓ\)​x,y⊤​Wt\(ℓ\)=y⊤​Wref\(ℓ\)\.W\_\{t\}^\{\(\\ell\)\}x=W\_\{\\mathrm\{ref\}\}^\{\(\\ell\)\}x,\\qquad y^\{\\top\}W\_\{t\}^\{\(\\ell\)\}=y^\{\\top\}W\_\{\\mathrm\{ref\}\}^\{\(\\ell\)\}\.\(28\)

###### Proof\.

Writex=Ψk\(ℓ\)​ax=\\Psi\_\{k\}^\{\(\\ell\)\}aandy=Φk\(ℓ\)​by=\\Phi\_\{k\}^\{\(\\ell\)\}b\. The bilateral constraints implyΔ​Wt\(ℓ\)​x=0\\Delta W\_\{t\}^\{\(\\ell\)\}x=0andy⊤​Δ​Wt\(ℓ\)=0y^\{\\top\}\\Delta W\_\{t\}^\{\(\\ell\)\}=0\. SubstitutingWt\(ℓ\)=Wref\(ℓ\)\+Δ​Wt\(ℓ\)W\_\{t\}^\{\(\\ell\)\}=W\_\{\\mathrm\{ref\}\}^\{\(\\ell\)\}\+\\Delta W\_\{t\}^\{\(\\ell\)\}proves the result\. ∎

The corollary is a layer\-level algebraic guarantee for the selected subspaces; it does not imply unconditional preservation of all model\-level capabilities\.

### Dimension and Expressivity of the Feasible Space

###### Proposition B\.5\.

Define the bilateral feasible space

𝒮=\{A:Φk⊤A=0,AΨk=0\}\.\\mathcal\{S\}=\\left\\\{A:\\Phi\_\{k\}^\{\\top\}A=0,\\;A\\Psi\_\{k\}=0\\right\\\}\.\(29\)Then

𝒮=\{ΠΦ⟂​Z​ΠΨ⟂:Z∈ℝdout×din\},\\mathcal\{S\}=\\left\\\{\\Pi\_\{\\Phi\}^\{\\perp\}Z\\Pi\_\{\\Psi\}^\{\\perp\}:Z\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}\\right\\\},\(30\)and

dim\(𝒮\)=\(dout−k\)​\(din−k\)\.\\dim\(\\mathcal\{S\}\)=\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\)\.\(31\)Moreover, every matrix in𝒮\\mathcal\{S\}with rank at mostrrcan be represented by the GCPO low\-rank parameterization\.

###### Proof\.

IfA∈𝒮A\\in\\mathcal\{S\}, then its columns lie inspan⁡\(Φk\)⟂\\operatorname\{span\}\(\\Phi\_\{k\}\)^\{\\perp\}, implyingΠΦ⟂​A=A\\Pi\_\{\\Phi\}^\{\\perp\}A=A\. Similarly,A​Ψk=0A\\Psi\_\{k\}=0impliesA​ΠΨ⟂=AA\\Pi\_\{\\Psi\}^\{\\perp\}=A\. Thus,

A=ΠΦ⟂​A​ΠΨ⟂\.A=\\Pi\_\{\\Phi\}^\{\\perp\}A\\Pi\_\{\\Psi\}^\{\\perp\}\.\(32\)
LetΦ⟂\\Phi\_\{\\perp\}andΨ⟂\\Psi\_\{\\perp\}be orthonormal bases for the corresponding complementary subspaces\. Every feasible matrix has a unique representation

A=Φ⟂​B​Ψ⟂⊤,B∈ℝ\(dout−k\)×\(din−k\)\.A=\\Phi\_\{\\perp\}B\\Psi\_\{\\perp\}^\{\\top\},\\qquad B\\in\\mathbb\{R\}^\{\(d\_\{\\mathrm\{out\}\}\-k\)\\times\(d\_\{\\mathrm\{in\}\}\-k\)\}\.\(33\)Therefore, the dimension of𝒮\\mathcal\{S\}is\(dout−k\)​\(din−k\)\(d\_\{\\mathrm\{out\}\}\-k\)\(d\_\{\\mathrm\{in\}\}\-k\)\.

Ifrank⁡\(A\)≤r\\operatorname\{rank\}\(A\)\\leq r, thenrank⁡\(B\)≤r\\operatorname\{rank\}\(B\)\\leq r, soBBadmits a factorizationB=U​VB=UVwith at mostrrlatent dimensions\. Taking

L=Φ⟂​U,R=V​Ψ⟂⊤L=\\Phi\_\{\\perp\}U,\\qquad R=V\\Psi\_\{\\perp\}^\{\\top\}\(34\)gives

A=ΠΦ⟂​L​R​ΠΨ⟂,A=\\Pi\_\{\\Phi\}^\{\\perp\}LR\\Pi\_\{\\Psi\}^\{\\perp\},\(35\)which is exactly the GCPO parameterization up to its scaling coefficient\. ∎

## Appendix CMethod Details

### Global Overlap Aggregation and Robustness

#### Global aggregation\.

For each adapted matrixℓ∈ℳ\\ell\\in\\mathcal\{M\}, we first compute the layer\-wise dimension\-corrected overlap

Ot,ℓexcess=\(1−EO​O\(t,ℓ\)Etotal\(t,ℓ\)\)−\[1−\(dout\(ℓ\)−kℓ\)​\(din\(ℓ\)−kℓ\)dout\(ℓ\)​din\(ℓ\)\]\.O\_\{t,\\ell\}^\{\\mathrm\{excess\}\}=\\left\(1\-\\frac\{E\_\{OO\}^\{\(t,\\ell\)\}\}\{E\_\{\\mathrm\{total\}\}^\{\(t,\\ell\)\}\}\\right\)\-\\left\[1\-\\frac\{\(d\_\{\\mathrm\{out\}\}^\{\(\\ell\)\}\-k\_\{\\ell\}\)\(d\_\{\\mathrm\{in\}\}^\{\(\\ell\)\}\-k\_\{\\ell\}\)\}\{d\_\{\\mathrm\{out\}\}^\{\(\\ell\)\}d\_\{\\mathrm\{in\}\}^\{\(\\ell\)\}\}\\right\]\.\(36\)The global statistic shown in Figure[1](https://arxiv.org/html/2608.11674#S3.F1)is the unweighted macro\-average

Ot,globalexcess=1\|ℳ\|​∑ℓ∈ℳOt,ℓexcess\.O\_\{t,\\mathrm\{global\}\}^\{\\mathrm\{excess\}\}=\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{\\ell\\in\\mathcal\{M\}\}O\_\{t,\\ell\}^\{\\mathrm\{excess\}\}\.\(37\)Normalization and null correction are therefore performed within each matrix before aggregation\. Consequently, each adapted matrix contributes equally, preventing matrices with larger dimensions or update norms from mechanically dominating the global statistic\. The displayed curve applies a five\-step moving average only after this layer\-wise aggregation\.

### Algorithm Details

Algorithm[1](https://arxiv.org/html/2608.11674#alg1)summarizes the implementation of GCPO\. The principal subspaces are computed once before training, while the projected low\-rank factors are used for both policy optimization and rollout generation\.

Algorithm 1Geometrically Constrained Policy OptimizationInput: Frozen policyθ0\\theta\_\{0\}and adapted layersℳ\\mathcal\{M\} Parameter: Protected rankkk, adaptation rankrr, scalingα\\alpha, and training stepsTT Output: Trained factors\{L\(ℓ\),R\(ℓ\)\}ℓ∈ℳ\\\{L^\{\(\\ell\)\},R^\{\(\\ell\)\}\\\}\_\{\\ell\\in\\mathcal\{M\}\}

1:Set

s←α/rs\\leftarrow\\alpha/r\.

2:foreach layer

ℓ∈ℳ\\ell\\in\\mathcal\{M\}do

3:Compute the top\-

kksingular vectors

Φk\(ℓ\)\\Phi\_\{k\}^\{\(\\ell\)\}and

Ψk\(ℓ\)\\Psi\_\{k\}^\{\(\\ell\)\}of

Wref\(ℓ\)W\_\{\\mathrm\{ref\}\}^\{\(\\ell\)\}\.

4:Initialize

L\(ℓ\)L^\{\(\\ell\)\}to zero and

R\(ℓ\)R^\{\(\\ell\)\}with the standard LoRA initialization\.

5:endfor

6:for

t=1,…,Tt=1,\\ldots,Tdo

7:foreach layer

ℓ∈ℳ\\ell\\in\\mathcal\{M\}do

8:

L¯\(ℓ\)←L\(ℓ\)−Φk\(ℓ\)​\(Φk\(ℓ\)⊤​L\(ℓ\)\)\\bar\{L\}^\{\(\\ell\)\}\\leftarrow L^\{\(\\ell\)\}\-\\Phi\_\{k\}^\{\(\\ell\)\}\\bigl\(\\Phi\_\{k\}^\{\(\\ell\)\\top\}L^\{\(\\ell\)\}\\bigr\)\.

9:

R¯\(ℓ\)←R\(ℓ\)−\(R\(ℓ\)​Ψk\(ℓ\)\)​Ψk\(ℓ\)⊤\\bar\{R\}^\{\(\\ell\)\}\\leftarrow R^\{\(\\ell\)\}\-\\bigl\(R^\{\(\\ell\)\}\\Psi\_\{k\}^\{\(\\ell\)\}\\bigr\)\\Psi\_\{k\}^\{\(\\ell\)\\top\}\.

10:Set

Wt\(ℓ\)←Wref\(ℓ\)\+s​L¯\(ℓ\)​R¯\(ℓ\)W\_\{t\}^\{\(\\ell\)\}\\leftarrow W\_\{\\mathrm\{ref\}\}^\{\(\\ell\)\}\+s\\bar\{L\}^\{\(\\ell\)\}\\bar\{R\}^\{\(\\ell\)\}\.

11:endfor

12:Synchronize the projected factors to the rollout engine\.

13:Generate rollouts and compute the policy\-optimization loss\.

14:Update the raw factors

\{L\(ℓ\),R\(ℓ\)\}ℓ∈ℳ\\\{L^\{\(\\ell\)\},R^\{\(\\ell\)\}\\\}\_\{\\ell\\in\\mathcal\{M\}\}\.

15:endfor

16:return

\{L\(ℓ\),R\(ℓ\)\}ℓ∈ℳ\\\{L^\{\(\\ell\)\},R^\{\(\\ell\)\}\\\}\_\{\\ell\\in\\mathcal\{M\}\}\.

## Appendix DExperimental Details

### Datasets and Splits

Table[5](https://arxiv.org/html/2608.11674#A4.T5)summarizes the datasets used for RL training and held\-out evaluation\. MATH500 and HumanEval\+ are established benchmarks for mathematical reasoning and code generation, respectively\. Following the common practice of constructing fixed train–test splits for controlled RL evaluation\([13](https://arxiv.org/html/2608.11674#bib.bib19)\), we build task\-specific splits from their official evaluation sets and keep them identical across all methods\. Consequently, the corresponding results measure performance on our held\-out subsets and should not be interpreted as official full\-benchmark scores\.

Table 5:Dataset splits used for RL training and held\-out evaluation\.For MATH500\([16](https://arxiv.org/html/2608.11674#bib.bib17)\)and HumanEval\+\([17](https://arxiv.org/html/2608.11674#bib.bib21)\), no problem identifier appears in both the training and evaluation splits\. Each HumanEval\+ problem retains its complete EvalPlus functional\-test suite; tests associated with the same programming problem are never divided between the two splits\.

For ToolAlpaca, constructed from ToolAlpaca\([25](https://arxiv.org/html/2608.11674#bib.bib16)\), the held\-out set contains examples from 10 API families that do not appear during training: Axolotl, Auth0, A Bíblia Digital, Apache Superset, Améthyste, Abstract Public Holidays, AbuseIPDB, 1Forge, Lob\.com, and AniAPI\. This protocol evaluates generalization to previously unseen tool families rather than to new examples from APIs observed during training\.

The MATH500 and HumanEval\+ splits are generated once with seed 42 and kept fixed across all methods, backbone models, and training runs\. This data\-splitting seed is independent of the three random seeds used for repeated training\. Therefore, the reported variation across runs reflects training and sampling stochasticity, rather than changes in the train–test partition\.

### Training and Evaluation Settings

Table[6](https://arxiv.org/html/2608.11674#A4.T6)summarizes the settings shared across tasks and methods unless overridden by a method\-specific configuration\. All main model–task–method configurations are independently trained with three random seeds\. For each configuration, we report the mean and standard deviation of the three resulting task\-level scores\.

Table 6:Shared training, rollout, and evaluation settings\.For each training seed, we generate 16 responses per held\-out example and report majority@16 to reduce sensitivity to individual stochastic generations and evaluate the model’s consistency across multiple samples\. We estimate majority@16 using 1,000 bootstrap trials, each of which resamples 16 responses with replacement and scores the majority\-voted answer\. Correctness is averaged over bootstrap trials and examples to obtain one score per training run, and final results are reported as the mean and standard deviation over three independent runs\.

#### Effective Training Horizons\.

The common training loop usesdrop\_last=Trueand a default limit of 30 epochs\. As a result, the effective number of optimization steps depends on the size of the corresponding training split\. Table[7](https://arxiv.org/html/2608.11674#A4.T7)summarizes the resulting training horizons\.

Table 7:Effective training horizons\. All runs are configured for at most 300 steps, with a batch size of 32,drop\_last=True, and at most 30 epochs\.MATH500 and ToolAlpaca can reach the configured limit of 300 optimization steps\. HumanEval\+ contains 147 training problems, yielding four complete batches per epoch and therefore at most 120 steps under the default 30\-epoch limit\. For cross\-task evaluation, we use the same task\-specific checkpointfor every method and seed: step 250 for MATH500, step 100 forHumanEval\+, and step 150 for ToolAlpaca\. We do not select a separate best\-validation checkpoint for each seed\.

### Method\-specific Hyperparameters

We retain method\-specific optimization settings rather than forcing all algorithms to share a common rollout count, learning rate, or clipping rule\. Table[8](https://arxiv.org/html/2608.11674#A4.T8)lists the principal hyperparameters\. The same configuration is used across the three independent runs of each method\.

Table 8:Method\-specific rollout and optimization hyperparameters\.The parameter\-efficient configurations are summarized in Table[9](https://arxiv.org/html/2608.11674#A4.T9)\. LoRA and GCPO adapt all linear layers\. For matched comparisons, the full\-parameter, LoRA, and GCPO variants use identical data splits, batch sizes, rollout settings, and evaluation protocols; they differ only in the parameterization of the policy update\.

Table 9:Parameter\-efficient adaptation settings used in the main experiments\.The main GCPO configuration uses scaling parameterα=16\\alpha=16, and protects the topk=8k=8left and right singular directions of each adapted weight matrix\. The ablation studies additionally considerk∈\{4,8,16,32,64\}k\\in\\\{4,8,16,32,64\\\}and adaptation ranksr∈\{16,32,64\}r\\in\\\{16,32,64\\\}\.

### Reward and Evaluation Protocols

All three tasks use binary response\-level rewards\. Table[10](https://arxiv.org/html/2608.11674#A4.T10)summarizes the task\-specific prediction extraction and correctness criteria\. A response receives a binary reward of 1 if it satisfies the task\-specific criterion and 0 otherwise\.

Table 10:Task\-specific response extraction and scoring\.For MATH500, the model is prompted to provide its final answer inside\\boxed\{\}\. For HumanEval\+, each problem retains its full functional\-test suite, and a generated program is considered correct only when it passes every test\. The 5\-second execution limit is used by the evaluator, regardless of the shorter time limit stated in the textual prompt\.

The ToolAlpaca evaluator measures static structural correctness rather than actual API execution\. It does not evaluate the order of multiple actions, and arguments from multiple predicted actions are merged before comparison\. Accordingly, we describe this metric as action\-name and argument matching rather than execution correctness\.

For every training seed, each held\-out example is evaluated using 16 sampled responses under the decoding configuration in Table[6](https://arxiv.org/html/2608.11674#A4.T6)\. The resulting responses are aggregated into a single task\-level score for that seed\. Final results are reported as the mean and standard deviation across the three independently trained seeds\.

### Computational Resources

All experiments were conducted on a single compute node equipped with four NVIDIA A100 GPUs\. Table[11](https://arxiv.org/html/2608.11674#A4.T11)summarizes the hardware configuration and distributed training setup\. The 48\-hour allocation denotes the maximum wall\-clock budget assigned to each job, rather than the actual runtime of every training run\.

Table 11:Computational resources and distributed training configuration used in our experiments\.For the matched GRPO comparison, full\-parameter GRPO, GRPO\-LoRA, and GCPO use identical data splits, batch sizes, rollout configurations, and evaluation protocols\. They differ only in the parameterization and geometric constraint of the policy update\.

The effective number of optimization steps additionally depends on the dataset size because the common trainer usesdrop\_last=Trueand terminates after 30 epochs\. Table[7](https://arxiv.org/html/2608.11674#A4.T7)reports the resulting training horizon for each task\.

For MATH500 and ToolAlpaca, the epoch limit permits the trainer to reach the configured maximum of 300 steps\. For HumanEval\+, the 147\-example training split produces only four complete batches per epoch, resulting in at most 120 optimization steps\. Since checkpoints are saved every 50 steps, we use the step\-100 checkpoint for cross\-task evaluation\. DAPO uses a separate epoch budget of 3,000 and is therefore not subject to the default 120\-step limit on HumanEval\+\.

## Appendix EAdditional Experiments

### Additional Overlap Trajectories

Figure[8](https://arxiv.org/html/2608.11674#A5.F8)complements the representative cases in Figure[1](https://arxiv.org/html/2608.11674#S3.F1)with additional trajectories on HumanEval\+ and ToolAlpaca\. We use the same excess\-overlap measure, 5\-step moving average, and validation protocol as in Section[3\.2](https://arxiv.org/html/2608.11674#S3.SS2)\. Elevated overlap again tends to accompany or precede validation degradation, although its timing and magnitude vary across tasks\. To avoid redundant per\-seed plots, we show one fixed\-seed trajectory for each remaining task\.

![Refer to caption](https://arxiv.org/html/2608.11674v1/principal_subspace_intrusion_side_by_side_additional.png)Figure 8:Stepwise update overlap and validation performance\. The red curve is the 5\-step moving average of excess principal\-subspace overlap; the blue curve is validation accuracy\. In both runs, episodes of elevated overlap accompany subsequent gradual degradation\. This observation is correlational\.
### Additional Controlled Interventions

We repeat the layer\-wise norm\-matched intervention on three additional model–task–checkpoint configurations\. As shown in Figure[9](https://arxiv.org/html/2608.11674#A5.F9)and[10](https://arxiv.org/html/2608.11674#A5.F10), increasing the principal\-overlapping component consistently produces a dose\-dependent accuracy drop across settings\. These results support the robustness of the local intervention effect beyond the main configuration\.

![Refer to caption](https://arxiv.org/html/2608.11674v1/principal_intervention_combined_Qwen_Math.png)Figure 9:Additional controlled intervention on the step\-125 GRPO update of Qwen3\-8B on MATH500\. \(a\) Increasing the principal\-overlapping component under layer\-wise norm matching produces a dose\-dependent accuracy drop\. \(b\) Orthogonalization improves accuracy, whereas matched principal\-subspace injection is substantially more harmful than a random\-subspace intervention\.![Refer to caption](https://arxiv.org/html/2608.11674v1/principal_intervention_combined_GLM_Humaneval.png)Figure 10:Additional controlled intervention on the step\-70 GRPO update of GLM on Humaneval\. \(a\) Increasing the principal\-overlapping component under layer\-wise norm matching produces a dose\-dependent accuracy drop\. \(b\) Orthogonalization improves accuracy, whereas matched principal\-subspace injection is substantially more harmful than a random\-subspace intervention\.