Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning

# From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning
Source: [https://arxiv.org/html/2606.31092](https://arxiv.org/html/2606.31092)
###### Abstract

Full fine\-tuning adapts large language models to new tasks but can erode capabilities they already possess\. Existing remedies protect through*proxies*—parameter distances, importance penalties, output matching, or the dominant singular directions of the weights—none of which answers the operative question:*which activation directions does the preserved capability rely on?*We argue a capability is characterized more faithfully by the activation subspace it induces than by the singular geometry of the weight matrix, and on this view developfunction\-space protection, instantiated asFora\(*Function\-space Orthogonal Residual Adaptation*\)\. From label\-free calibration inputs we estimate, per layer, the principal directions𝐐\\mathbf\{Q\}of the input\-activation covariance and form a right projectorP𝐐=𝐈−𝐐𝐐⊤P\_\{\\\!\\mathbf\{Q\}\}=\\mathbf\{I\}\-\\mathbf\{Q\}\\mathbf\{Q\}^\{\\top\}; paired with a left projectorP𝐔P\_\{\\\!\\mathbf\{U\}\}from the weight’s SVD, theForaupdate isΔ​𝐖=P𝐔​𝐌​P𝐐\+𝐔2​𝐃δ​𝐕2⊤\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\+\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}—a high\-capacity branch structurally barred from reading the capability’s function directions, plus a narrow spectral channel for controlled plasticity\. The construction extends to parameter\-efficient adaptation via𝐌→αr​𝐁​𝒜\\mathbf\{M\}\\to\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}\. Across three settings on Qwen3\-1\.7B \(COGS and GSM8K learned while preserving translation, translation learned while preserving math\), it consistently outperforms weight\-space projection and standard regularization on preservation, with only a small new\-task trade\-off in the math\-preservation setting\. A controlled ablation isolating the projection*source*shows the advantage comes not from projection itself, but from projecting onto capability\-derived rather than weight\-derived directions\. Code is available at[https://github\.com/zrui239/FORA](https://github.com/zrui239/FORA)\.

## 1Introduction

Full fine\-tuning—updating all parameters of a pre\-trained model\(Vaswaniet al\.,[2017](https://arxiv.org/html/2606.31092#bib.bib20)\)under unconstrained gradients—remains the most expressive way to adapt a large language model to a new task\(Hanet al\.,[2024](https://arxiv.org/html/2606.31092#bib.bib32)\)\. This expressiveness is also a liability: unconstrained gradients are free to overwrite the functional pathways of capabilities the model already possesses\. A model expected to retain translation quality while learning to parse semantics, or to retain arithmetic ability while learning a new language pair, must avoid this drift, yet the standard fine\-tuning objective does nothing to discourage it\.

A long line of work mitigates such forgetting\(McCloskey and Cohen,[1989](https://arxiv.org/html/2606.31092#bib.bib21); French,[1999](https://arxiv.org/html/2606.31092#bib.bib22); Parisiet al\.,[2019](https://arxiv.org/html/2606.31092#bib.bib23); Shiet al\.,[2025](https://arxiv.org/html/2606.31092#bib.bib31)\)\. Regularization methods anchor the weights through importance\-weighted\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.31092#bib.bib1)\)or plainℓ2\\ell\_\{2\}\(Liet al\.,[2018](https://arxiv.org/html/2606.31092#bib.bib2)\)penalties; distillation methods keep the output distribution close to the original model’s\(Li and Hoiem,[2018](https://arxiv.org/html/2606.31092#bib.bib3)\); gradient\-projection methods steer updates away from stored task subspaces\(Sahaet al\.,[2021](https://arxiv.org/html/2606.31092#bib.bib4); Xiong and Xie,[2026](https://arxiv.org/html/2606.31092#bib.bib5)\); and weight\-decomposition methods restrict adaptation to chosen singular components of the pre\-trained weight\(Menget al\.,[2024](https://arxiv.org/html/2606.31092#bib.bib7); Liuet al\.,[2024a](https://arxiv.org/html/2606.31092#bib.bib8)\)\. Each of these protects a capability through a*proxy*: parameter importance, output behavior, gradient history, or weight geometry\. None of them directly encodes the set of activation directions that the capability uses during inference\.

#### Our claim\.

A preserved capability is better characterized by the activation subspace it induces in each layer than by the dominant singular directions of the weight matrix\.The top right\-singular vectors𝐕1\\mathbf\{V\}\_\{1\}of a weight𝐖0\\mathbf\{W\}\_\{0\}summarize where that weight is*large*—an aggregate over everything the model has ever learned\. They need not align with where a*specific*capability is functionally active: translation may rely on activation patterns spread across many moderate singular components, or oriented diagonally with respect to the SVD basis\. Protecting𝐕1\\mathbf\{V\}\_\{1\}therefore constrains a subspace that is largely misaligned with the capability\.

#### Method\.

We address this by deriving the protected subspace from the capability itself\. Forwarding label\-free calibration inputs of the preserved capability, we collect per\-layer input activations, take the top\-kkeigenvectors𝐐\\mathbf\{Q\}of their covariance, and form a*function\-space*right projectorP𝐐=𝐈−𝐐𝐐⊤P\_\{\\\!\\mathbf\{Q\}\}=\\mathbf\{I\}\-\\mathbf\{Q\}\\mathbf\{Q\}^\{\\top\}\. The full\-rank update is

Δ​𝐖=P𝐔​𝐌​P𝐐⏟high\-capacity, function\-protected\+𝐔2​𝐃δ​𝐕2⊤⏟controlled spectral channel,\\Delta\\\!\\mathbf\{W\}\\;=\\;\\underbrace\{P\_\{\\\!\\mathbf\{U\}\}\\,\\mathbf\{M\}\\,P\_\{\\\!\\mathbf\{Q\}\}\}\_\{\\text\{high\-capacity, function\-protected\}\}\\;\+\\;\\underbrace\{\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\}\_\{\\text\{controlled spectral channel\}\},whereP𝐔=𝐈−𝐔1​𝐔1⊤P\_\{\\\!\\mathbf\{U\}\}=\\mathbf\{I\}\-\\mathbf\{U\}\_\{1\}\\mathbf\{U\}\_\{1\}^\{\\top\}prevents the high\-capacity branch from writing the principal output directions of𝐖0\\mathbf\{W\}\_\{0\}, andP𝐐P\_\{\\\!\\mathbf\{Q\}\}prevents it from reading the capability’s function directions\. The two channels play complementary roles: on inputs aligned with the protected capability the dense branch contributes nothing and the response is determined solely by the narrow spectral channel—a small, calibrated adjustment rather than an outright freeze—while on all other inputs the dense branch retains full capacity\. The same parameterization extends to low\-rank adaptation by replacing𝐌\\mathbf\{M\}withαr​𝐁​𝒜\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}\.

#### Findings\.

We instantiate the method on Qwen3\-1\.7B across three full\-rank settings \(Section[4](https://arxiv.org/html/2606.31092#S4)\)\. Learning COGS while preserving translation, we hold perplexity at 4\.39—within 0\.04 of the 4\.35 baseline—at 98\.6% COGS exact match, while weight\-space projection degrades perplexity to 4\.86 at the same accuracy and is no better than unprotected fine\-tuning\. Learning GSM8K while preserving translation, we match the baseline perplexity exactly \(4\.35\)\. Reversing the roles—learning translation while preserving math—we retain GSM8K within 0\.9 points of the original model where full fine\-tuning loses 2\.3, confirming𝐐\\mathbf\{Q\}is not a translation\-specific artifact\. A controlled ablation swapping onlyP𝐕P\_\{\\\!\\mathbf\{V\}\}forP𝐐P\_\{\\\!\\mathbf\{Q\}\}shows the gain comes from*what*is protected, not from projecting at all\. Figure[1](https://arxiv.org/html/2606.31092#S1.F1)summarizes all three settings on the preservation–adaptation plane\.

![Refer to caption](https://arxiv.org/html/2606.31092v1/x1.png)Figure 1:Preservation–adaptation trade\-off across all three settings\(visualizing Tables[2](https://arxiv.org/html/2606.31092#S4.T2),[3](https://arxiv.org/html/2606.31092#S4.T3), and[4](https://arxiv.org/html/2606.31092#S4.T4)\)\. Each panel plots a new\-task metric \(horizontal\) against the preserved\-capability metric \(vertical\); the dashed line marks the preserved\-capability target and the shaded band the favorable region\. \(a\) Trans→\\toCOGS and \(b\) Trans→\\toGSM8K plot new\-task EM against translation PPL \(lower better, axis inverted\); \(c\) Math→\\toTrans plots translation PPL against retained GSM8K EM \(higher better\)\. In all three,FProj/Fora\(large outlined markers\) achieve the strongest preservation while remaining competitive on the new task, whereas the weight\-space and regularization baselines sacrifice preservation\.
#### Contributions\.

\(i\) We reformulate capability preservation as*function\-space protection*: the protected object is the activation subspace induced by the retained capability, rather than parameter distance or the singular geometry of𝐖0\\mathbf\{W\}\_\{0\}\. \(ii\) We proposeFora\(*Function\-space Orthogonal Residual Adaptation*\), a structurally constrained full\-rank update

Δ​𝐖=P𝐔​𝐌​P𝐐\+𝐔2​𝐃δ​𝐕2⊤,\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\+\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\},whereP𝐐P\_\{\\\!\\mathbf\{Q\}\}blocks reads from capability\-sensitive directions,P𝐔P\_\{\\\!\\mathbf\{U\}\}blocks writes to dominant output directions, and the spectral channel provides bounded plasticity\. \(iii\) We characterize this parameterization through three operator\-level properties and validate the mechanism with controlledP𝐕P\_\{\\\!\\mathbf\{V\}\}\-versus\-P𝐐P\_\{\\\!\\mathbf\{Q\}\}ablations, showing that the key factor is the*source*of the protected subspace\. Section[3](https://arxiv.org/html/2606.31092#S3)develops the method, Section[4](https://arxiv.org/html/2606.31092#S4)evaluates the full\-rank setting, and later sections present ablations and the low\-rank extension\.

## 2Related Work

#### Regularization and distillation\.

Classical continual\-learning methods bound how far parameters or outputs may move\. Importance\-based regularizers such as EWC\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.31092#bib.bib1)\), SI\(Zenkeet al\.,[2017](https://arxiv.org/html/2606.31092#bib.bib25)\), and MAS\(Aljundiet al\.,[2018](https://arxiv.org/html/2606.31092#bib.bib24)\)estimate which parameters should remain stable; L2\-SP\(Liet al\.,[2018](https://arxiv.org/html/2606.31092#bib.bib2)\)penalizes theℓ2\\ell\_\{2\}distance to the pre\-trained weights; LwF\(Li and Hoiem,[2018](https://arxiv.org/html/2606.31092#bib.bib3)\)distills the frozen model’s output distribution on new\-task inputs\. All three rely on*indirect*signals—parameter importance, parameter proximity, output consistency—and none constrains which input directions an update may act on, leaving the capability’s functional pathway exposed\. Under full\-rank fine\-tuning they degrade translation almost as much as no protection at all \(Section[4\.2](https://arxiv.org/html/2606.31092#S4.SS2)\)\.

#### Subspace and gradient projection\.

A second family constrains gradients\. Gradient Projection Memory\(Sahaet al\.,[2021](https://arxiv.org/html/2606.31092#bib.bib4)\)projects new gradients onto the orthogonal complement of stored per\-task subspaces; memory\-based methods such as GEM\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2606.31092#bib.bib26)\)and A\-GEM\(Chaudhryet al\.,[2019](https://arxiv.org/html/2606.31092#bib.bib27)\)constrain gradients using episodic replay buffers, providing strong baselines when preserved\-capability data can be stored\. O\-LoRA\(Wanget al\.,[2023](https://arxiv.org/html/2606.31092#bib.bib6)\)learns sequential tasks in distinct orthogonal LoRA subspaces to minimize interference, and OPLoRA\(Xiong and Xie,[2026](https://arxiv.org/html/2606.31092#bib.bib5)\)provides the analogous projection in the PEFT setting\. These act at the*optimization*level, modifying gradients during training; we act at the*parameterization*level, constraining the structure ofΔ​𝐖\\Delta\\\!\\mathbf\{W\}so protection holds regardless of optimizer state or training trajectory\.

#### Weight\-space constrained fine\-tuning\.

A third line decomposes𝐖0\\mathbf\{W\}\_\{0\}by SVD and restricts adaptation to chosen spectral components: PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2606.31092#bib.bib7)\)fine\-tunes the residual after initializing from principal singular components, MiLoRA\(Wanget al\.,[2025](https://arxiv.org/html/2606.31092#bib.bib9)\)instead initializes from minor singular components, AdaLoRA\(Zhanget al\.,[2023](https://arxiv.org/html/2606.31092#bib.bib10)\)adaptively allocates rank across layers by pruning singular values during training, and DoRA\(Liuet al\.,[2024a](https://arxiv.org/html/2606.31092#bib.bib8)\)splits weights into magnitude and direction\. Orthogonal fine\-tuning methods such as BOFT\(Liuet al\.,[2024b](https://arxiv.org/html/2606.31092#bib.bib28)\)parameterize update transformations to preserve orthogonality, but do not derive the protected subspace from a specific preserved capability’s activations\. Memory\-efficient PEFT variants like QLoRA\(Dettmerset al\.,[2023](https://arxiv.org/html/2606.31092#bib.bib33)\)focus on the computational cost of adaptation, whereas we address capability preservation under structural projection\. All assume the weight’s*singular directions*define what to protect—but the top\-rrright singular vectors𝐕1\\mathbf\{V\}\_\{1\}mark where the weight is large, not where a capability is active\. Section[4\.5](https://arxiv.org/html/2606.31092#S4.SS5)shows this mismatch directly: protecting𝐕1\\mathbf\{V\}\_\{1\}fails where the capability\-derived𝐐\\mathbf\{Q\}succeeds\.

#### Activation\-aware adaptation\.

Recent adapters use activation statistics: CorDA\(Yanget al\.,[2024](https://arxiv.org/html/2606.31092#bib.bib11)\)orients low\-rank initialization by input\-activation covariance, and EVA\(Paischeret al\.,[2025](https://arxiv.org/html/2606.31092#bib.bib12)\)guides placement by activation variance\. At the predictive\-function level, function\-space regularization\(Rudneret al\.,[2023](https://arxiv.org/html/2606.31092#bib.bib29)\)constrains the outputs a model may produce, while we constrain the*internal*activation subspace through a hard projectorP𝐐P\_\{\\\!\\mathbf\{Q\}\}on per\-layer weight updates\. Mechanistic studies further suggest that specific behaviors such as refusal can be mediated by low\-dimensional activation directions\(Arditiet al\.,[2024](https://arxiv.org/html/2606.31092#bib.bib30)\), supporting the premise that activation subspaces are meaningful objects of protection\. We share the premise that activations encode capabilities but differ in two ways: we use the activation subspace as a*hard right projector*P𝐐P\_\{\\\!\\mathbf\{Q\}\}on the update operator rather than a soft initialization, and we establish the principle in*full\-rank*fine\-tuning before specializing to low rank\. To our knowledge this is the first controlled comparison of weight\-space \(P𝐕P\_\{\\\!\\mathbf\{V\}\}\) against function\-space \(P𝐐P\_\{\\\!\\mathbf\{Q\}\}\) projection under full\-rank fine\-tuning, identifying the projection*source*as the decisive factor\.

## 3Method

![Refer to caption](https://arxiv.org/html/2606.31092v1/x2.png)Figure 2:From weight\-space projection to function\-space protection\.\(a\) Unconstrained full fine\-tuning can overwrite any direction, offering maximal plasticity but no protection\. \(b\) Weight\-space projection masks the principal singular directions of𝐖0\\mathbf\{W\}\_\{0\}; these mark where the weight is large, which need not coincide with the capability’s functional footprint\. \(c\)Forareplaces the weight\-derived mask with a capability\-derived projectorP𝐐P\_\{\\\!\\mathbf\{Q\}\}, so the high\-capacity branch avoids the activation directions the preserved capability uses, while a narrow spectral channel𝐔2​𝐃δ​𝐕2⊤\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}supplies controlled calibration\.We develop the method as a sequence of parameterizations, each resolving a limitation of the preceding one \(Figure[2](https://arxiv.org/html/2606.31092#S3.F2)\)\. The endpoint is a full\-rank update that protects a capability through the activation subspace it induces, and the derivation shows that each design choice follows from the limitation it addresses rather than being assumed\.

### 3\.1Setup: the need for a constraint

Consider a pre\-trained weight𝐖0∈ℝdout×din\\mathbf\{W\}\_\{0\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}in any linear layer of a transformer\. We are given a preserved capability𝒟pres\\mathcal\{D\}\_\{\\text\{pres\}\}\(inputs only—no labels\) and a new task𝒟new\\mathcal\{D\}\_\{\\text\{new\}\}, and seek an updateΔ​𝐖\\Delta\\\!\\mathbf\{W\}such that𝐖^=𝐖0\+Δ​𝐖\\widehat\{\\mathbf\{W\}\}=\\mathbf\{W\}\_\{0\}\+\\Delta\\\!\\mathbf\{W\}learns𝒟new\\mathcal\{D\}\_\{\\text\{new\}\}while retaining performance on𝒟pres\\mathcal\{D\}\_\{\\text\{pres\}\}\. Plain full fine\-tuning sets

Δ​𝐖=𝐌,𝐌∈ℝdout×din,\\Delta\\\!\\mathbf\{W\}=\\mathbf\{M\},\\qquad\\mathbf\{M\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\},\(1\)a dense matrix optimized on𝒟new\\mathcal\{D\}\_\{\\text\{new\}\}\. Its capacity is maximal, but so is its reach:𝐌\\mathbf\{M\}may read from and write to every input and output direction, including those on which the preserved capability depends\. A constraint is therefore required, together with a principled basis from which to derive it\.

#### SVD coordinates\.

A natural starting point is the structure already in𝐖0\\mathbf\{W\}\_\{0\}\. Write𝐖0=𝐔​Σ​𝐕⊤\\mathbf\{W\}\_\{0\}=\\mathbf\{U\}\\Sigma\\mathbf\{V\}^\{\\top\}and split the singular triplets into a principal block \(top\-rr\) and a residual block:

𝐔=\[𝐔1​𝐔2\],𝐕=\[𝐕1​𝐕2\],Σ=\[Σ100Σ2\],\\mathbf\{U\}=\[\\mathbf\{U\}\_\{1\}\\;\\mathbf\{U\}\_\{2\}\],\\quad\\mathbf\{V\}=\[\\mathbf\{V\}\_\{1\}\\;\\mathbf\{V\}\_\{2\}\],\\quad\\Sigma=\\begin\{bmatrix\}\\Sigma\_\{1\}&0\\\\ 0&\\Sigma\_\{2\}\\end\{bmatrix\},\(2\)with𝐔1∈ℝdout×r\\mathbf\{U\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times r\},𝐕1∈ℝdin×r\\mathbf\{V\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\\times r\}\. Define projectors onto the residual subspaces,

P𝐔=𝐈−𝐔1​𝐔1⊤,P𝐕=𝐈−𝐕1​𝐕1⊤\.P\_\{\\\!\\mathbf\{U\}\}=\\mathbf\{I\}\-\\mathbf\{U\}\_\{1\}\\mathbf\{U\}\_\{1\}^\{\\top\},\\qquad P\_\{\\\!\\mathbf\{V\}\}=\\mathbf\{I\}\-\\mathbf\{V\}\_\{1\}\\mathbf\{V\}\_\{1\}^\{\\top\}\.\(3\)These give a vocabulary for restrictingΔ​𝐖\\Delta\\\!\\mathbf\{W\}to the part of𝐖0\\mathbf\{W\}\_\{0\}not dominated by its largest singular modes\. It remains to choose the form this restriction should take\.

### 3\.2Spectral calibration and residual\-space expansion

#### A conservative start\.

The most cautious update confined to the residual subspace is a diagonal rescaling of the residual singular values:

Δ​𝐖=𝐔2​𝐃δ​𝐕2⊤,𝐃δ=diag⁡\(𝜹\),\\Delta\\\!\\mathbf\{W\}=\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\},\\qquad\\mathbf\{D\}\_\{\\delta\}=\\operatorname\{diag\}\(\\bm\{\\delta\}\),\(4\)withdres=min⁡\(dout,din\)−rd\_\{\\text\{res\}\}=\\min\(d\_\{\\text\{out\}\},d\_\{\\text\{in\}\}\)\-rfree scalars\. The update lies entirely within𝐔2​\(⋅\)​𝐕2⊤\\mathbf\{U\}\_\{2\}\(\\cdot\)\\mathbf\{V\}\_\{2\}^\{\\top\}and cannot affect the principal directions of𝐖0\\mathbf\{W\}\_\{0\}\. It is safe, and—as we retain throughout—it constitutes a genuine, low\-degree\-of\-freedom adjustment channel \(Appendix[D](https://arxiv.org/html/2606.31092#A4)quantifies what this channel achieves in isolation\)\. Its limitation is expressiveness: a diagonal map scales each residual mode independently and cannot represent interactions among them, so by itself it lacks the capacity to learn a demanding new task\.

#### Adding mixing capacity\.

Relaxing the diagonal constraint restores expressiveness:

Δ​𝐖=𝐔2​\(𝐃δ\+𝐒res\)​𝐕2⊤,\\Delta\\\!\\mathbf\{W\}=\\mathbf\{U\}\_\{2\}\(\\mathbf\{D\}\_\{\\delta\}\+\\mathbf\{S\}\_\{\\mathrm\{res\}\}\)\\mathbf\{V\}\_\{2\}^\{\\top\},\(5\)where𝐒res∈ℝdres×dres\\mathbf\{S\}\_\{\\mathrm\{res\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{res\}\}\\times d\_\{\\text\{res\}\}\}mixes residual directions and𝐃δ\\mathbf\{D\}\_\{\\delta\}is retained as an explicit per\-mode term\. This form has genuine learning capacity while remaining within the residual subspace\. However,𝐒res\\mathbf\{S\}\_\{\\mathrm\{res\}\}is expressed in the abstract SVD coordinates\(𝐔2,𝐕2\)\(\\mathbf\{U\}\_\{2\},\\mathbf\{V\}\_\{2\}\), in which it is difficult to reason about*which input directions*the update can access—precisely the reasoning required to protect a capability\.

### 3\.3From residual coordinates to function\-space projection

#### Exposing the right projector\.

A standard identity rewrites the residual\-coordinate form as a two\-sided projection\. For any𝐌\\mathbf\{M\},

𝐔2​\(𝐔2⊤​𝐌𝐕2\)​𝐕2⊤=\(𝐈−𝐔1​𝐔1⊤\)​𝐌​\(𝐈−𝐕1​𝐕1⊤\)=P𝐔​𝐌​P𝐕\.\\mathbf\{U\}\_\{2\}\(\\mathbf\{U\}\_\{2\}^\{\\top\}\\mathbf\{M\}\\mathbf\{V\}\_\{2\}\)\\mathbf\{V\}\_\{2\}^\{\\top\}=\(\\mathbf\{I\}\-\\mathbf\{U\}\_\{1\}\\mathbf\{U\}\_\{1\}^\{\\top\}\)\\,\\mathbf\{M\}\\,\(\\mathbf\{I\}\-\\mathbf\{V\}\_\{1\}\\mathbf\{V\}\_\{1\}^\{\\top\}\)=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{V\}\}\.\(6\)Setting𝐒res=𝐔2⊤​𝐌𝐕2\\mathbf\{S\}\_\{\\mathrm\{res\}\}=\\mathbf\{U\}\_\{2\}^\{\\top\}\\mathbf\{M\}\\mathbf\{V\}\_\{2\}turns Eq\.[5](https://arxiv.org/html/2606.31092#S3.E5)into

𝐔2​\(𝐃δ\+𝐒res\)​𝐕2⊤=𝐔2​𝐃δ​𝐕2⊤\+P𝐔​𝐌​P𝐕\.\\mathbf\{U\}\_\{2\}\(\\mathbf\{D\}\_\{\\delta\}\+\\mathbf\{S\}\_\{\\mathrm\{res\}\}\)\\mathbf\{V\}\_\{2\}^\{\\top\}=\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\+P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{V\}\}\.\(7\)This does not change the set of representable updates—an unrestricted𝐒res\\mathbf\{S\}\_\{\\mathrm\{res\}\}and an unrestricted𝐌\\mathbf\{M\}span the same residual matrices—but it changes what is*visible*: the termP𝐔​𝐌​P𝐕P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{V\}\}displays the right projectorP𝐕P\_\{\\\!\\mathbf\{V\}\}explicitly\. The constraint is no longer the opaque “live in𝐔2​\(⋅\)​𝐕2⊤\\mathbf\{U\}\_\{2\}\(\\cdot\)\\mathbf\{V\}\_\{2\}^\{\\top\}” but the functional “do not read fromspan⁡\(𝐕1\)\\operatorname\{span\}\(\\mathbf\{V\}\_\{1\}\)\.” This exposes the question the SVD form concealed: whetherspan⁡\(𝐕1\)\\operatorname\{span\}\(\\mathbf\{V\}\_\{1\}\)is in fact the right subspace to protect\.

#### Limitation ofP𝐕P\_\{\\\!\\mathbf\{V\}\}as the protected subspace\.

𝐕1\\mathbf\{V\}\_\{1\}holds the input directions in which𝐖0\\mathbf\{W\}\_\{0\}has its largest singular values—an aggregate compressed across every capability the model carries through a single factorization\. A specific capability such as English–Chinese translation may instead activate patterns distributed over many moderate singular components, or oriented diagonally with respect to the SVD basis\.𝐕1\\mathbf\{V\}\_\{1\}thus identifies where the*weight*is large, not where the*capability*is active\.

#### Constructing a capability\-specific projector\.

We therefore derive the protected subspace from the capability rather than the weight\. Forwarding label\-free calibration inputs from𝒟pres\\mathcal\{D\}\_\{\\text\{pres\}\}, we collect the input activations at each target layer and stack them as𝐗pres∈ℝN×d\\mathbf\{X\}\_\{\\text\{pres\}\}\\in\\mathbb\{R\}^\{N\\times d\}overNNcalibration tokens\. We form the activation covariance and take its top\-kkeigenvectors,

𝐂pres=1N​𝐗pres⊤​𝐗pres,𝐐=\[𝒒1,…,𝒒k\],\\mathbf\{C\}\_\{\\text\{pres\}\}=\\tfrac\{1\}\{N\}\\mathbf\{X\}\_\{\\text\{pres\}\}^\{\\top\}\\mathbf\{X\}\_\{\\text\{pres\}\},\\qquad\\mathbf\{Q\}=\[\\bm\{q\}\_\{1\},\\ldots,\\bm\{q\}\_\{k\}\],\(8\)ordered by decreasing eigenvalue—the directions of largest activation variance, i\.e\. the capability’s functional signature in that layer\. Thefunction\-space right projectoris

P𝐐=𝐈−𝐐𝐐⊤\.P\_\{\\\!\\mathbf\{Q\}\}=\\mathbf\{I\}\-\\mathbf\{Q\}\\mathbf\{Q\}^\{\\top\}\.\(9\)Three properties are worth noting: the construction requires*no labels*\(inputs of𝒟pres\\mathcal\{D\}\_\{\\text\{pres\}\}suffice\); it is performed*per layer*; and it is*capability\-specific*, capturing the activation structure of the particular capability rather than an aggregate\.

#### Function\-space orthogonal residual adaptation \(Fora\)\.

ReplacingP𝐕P\_\{\\\!\\mathbf\{V\}\}byP𝐐P\_\{\\\!\\mathbf\{Q\}\}in Eq\.[7](https://arxiv.org/html/2606.31092#S3.E7)givesFora\(*Function\-space Orthogonal Residual Adaptation*\), our full\-rank update:

Δ​𝐖Fora=P𝐔​𝐌​P𝐐⏟function\-protected adaptation\+𝐔2​𝐃δ​𝐕2⊤⏟spectral calibration\\boxed\{\\;\\Delta\\\!\\mathbf\{W\}\_\{\\textsf\{Fora\}\}=\\underbrace\{P\_\{\\\!\\mathbf\{U\}\}\\,\\mathbf\{M\}\\,P\_\{\\\!\\mathbf\{Q\}\}\}\_\{\\text\{function\-protected adaptation\}\}\\;\+\\;\\underbrace\{\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\}\_\{\\text\{spectral calibration\}\}\\;\}\(10\)P𝐔​𝐌​P𝐐P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}is the high\-capacity branch: a dense matrix with full expressive power, blocked from reading the capability’s function directions \(viaP𝐐P\_\{\\\!\\mathbf\{Q\}\}\) and from writing𝐖0\\mathbf\{W\}\_\{0\}’s principal output directions \(viaP𝐔P\_\{\\\!\\mathbf\{U\}\}\)\.𝐔2​𝐃δ​𝐕2⊤\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}is the spectral channel from Eq\.[4](https://arxiv.org/html/2606.31092#S3.E4)\. The projection\-only variant without this channel,Δ​𝐖=P𝐔​𝐌​P𝐐\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}, we denoteFProj\. Table[1](https://arxiv.org/html/2606.31092#S3.T1)summarizes the derivation; Figure[3](https://arxiv.org/html/2606.31092#S3.F3)depicts the construction of𝐐\\mathbf\{Q\}and the resulting protected forward computation\.

![Refer to caption](https://arxiv.org/html/2606.31092v1/x3.png)Figure 3:Construction and forward computation ofFora\.\(a\)*Function\-space source\.*From label\-free preserved\-capability inputs we collect per\-layer activations𝐗pres\\mathbf\{X\}\_\{\\text\{pres\}\}, form the covariance𝐂=1N​𝐗pres⊤​𝐗pres\\mathbf\{C\}=\\tfrac\{1\}\{N\}\\mathbf\{X\}\_\{\\text\{pres\}\}^\{\\top\}\\mathbf\{X\}\_\{\\text\{pres\}\}, and take its top\-kfk\_\{f\}eigenvectors𝐐\\mathbf\{Q\}to build the right projectorP𝐐=𝐈−𝐐𝐐⊤P\_\{\\\!\\mathbf\{Q\}\}=\\mathbf\{I\}\-\\mathbf\{Q\}\\mathbf\{Q\}^\{\\top\}\. \(b\)*Protected forward pass\.*The output combines the frozen weight𝐖\\mathbf\{W\}, the function\-protected high\-capacity branchP𝐔​𝐌​P𝐐P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}, and the spectral calibration branch𝐔2​𝐃δ​𝐕2⊤\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}; only𝐌\\mathbf\{M\}and𝜹\\bm\{\\delta\}are trained\. \(c\)*Spectral source\.*The left projectorP𝐔=𝐈−𝐔1​𝐔1⊤P\_\{\\\!\\mathbf\{U\}\}=\\mathbf\{I\}\-\\mathbf\{U\}\_\{1\}\\mathbf\{U\}\_\{1\}^\{\\top\}and the calibration branch are both derived from the SVD of𝐖0\\mathbf\{W\}\_\{0\}, with𝐔1,𝐕1\\mathbf\{U\}\_\{1\},\\mathbf\{V\}\_\{1\}the principal block and𝐔2,𝐕2\\mathbf\{U\}\_\{2\},\\mathbf\{V\}\_\{2\}the residual block\.
#### Structural properties\.

Three facts follow immediately from the projector definitions\.

###### Proposition 1\(Output\-side protection\)\.

𝐔1⊤​P𝐔​𝐌​P𝐐=𝟎\\mathbf\{U\}\_\{1\}^\{\\top\}\\,P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}=\\bm\{0\}: the high\-capacity branch contributes nothing to the principal output directions of𝐖0\\mathbf\{W\}\_\{0\}\.

###### Proposition 2\(No read on capability directions\)\.

For any𝐱∈span⁡\(𝐐\)\\bm\{x\}\\in\\operatorname\{span\}\(\\mathbf\{Q\}\),P𝐔​𝐌​P𝐐​𝐱=𝟎\\;P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\\,\\bm\{x\}=\\bm\{0\}: the high\-capacity branch is blind to inputs aligned with the preserved capability’s function directions\.

###### Proposition 3\(Controlled response via the spectral channel\)\.

Under the full update, for𝐱∈span⁡\(𝐐\)\\bm\{x\}\\in\\operatorname\{span\}\(\\mathbf\{Q\}\),

Δ​𝐖Fora​𝒙=𝐔2​𝐃δ​𝐕2⊤​𝒙\.\\Delta\\\!\\mathbf\{W\}\_\{\\textsf\{Fora\}\}\\,\\bm\{x\}=\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\\,\\bm\{x\}\.\(11\)On protected directions the response is governed entirely by the spectral channel\. The capability is not frozen: the model retains a low\-degree\-of\-freedom, per\-mode adjustment through𝛅\\bm\{\\delta\}, while the dense branch is barred from these directions\.

This statement makes the design intent precise: rather than hard\-freezing the preserved capability, the method routes any adaptation along its directions through a narrow, controllable channel, while leaving the high\-capacity branch unconstrained elsewhere\.

### 3\.4Two channels over complementary subspaces

The final update decomposes the input space into two channels\. Thefunction\-protected channelP𝐔​𝐌​P𝐐P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}carries the majority of the learning capacity: between its two projectors the dense matrix𝐌\\mathbf\{M\}retains full expressive power for every direction outsidespan⁡\(𝐔1\)\\operatorname\{span\}\(\\mathbf\{U\}\_\{1\}\)andspan⁡\(𝐐\)\\operatorname\{span\}\(\\mathbf\{Q\}\), withP𝐔P\_\{\\\!\\mathbf\{U\}\}enforcing a weight\-structure prior \(preserving𝐖0\\mathbf\{W\}\_\{0\}’s dominant output space\) andP𝐐P\_\{\\\!\\mathbf\{Q\}\}enforcing the capability\-specific constraint that distinguishes the method from weight\-space projection\. Thespectral channel𝐔2​𝐃δ​𝐕2⊤\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}contributes onlydresd\_\{\\text\{res\}\}scalars, but by Proposition[3](https://arxiv.org/html/2606.31092#Thmproposition3)it is the only component able to respond to inputs inspan⁡\(𝐐\)\\operatorname\{span\}\(\\mathbf\{Q\}\), granting the protected capability a small, bounded degree of plasticity precisely where the dense branch is inactive, rather than freezing it\. This yields a clear separation of roles: for𝒙∈span⁡\(𝐐\)\\bm\{x\}\\in\\operatorname\{span\}\(\\mathbf\{Q\}\)only the spectral channel responds; for𝒙⟂span⁡\(𝐐\)\\bm\{x\}\\perp\\operatorname\{span\}\(\\mathbf\{Q\}\)the dense branch provides full capacity and the spectral channel adds residual calibration; and neither channel writes tospan⁡\(𝐔1\)\\operatorname\{span\}\(\\mathbf\{U\}\_\{1\}\)\. Protection is therefore a property of the parameterization rather than of the optimization trajectory\.

### 3\.5Low\-rank extension

The protection structure in Eq\.[10](https://arxiv.org/html/2606.31092#S3.E10)—P𝐔P\_\{\\\!\\mathbf\{U\}\}on the left,P𝐐P\_\{\\\!\\mathbf\{Q\}\}on the right—does not depend on𝐌\\mathbf\{M\}being dense\. Substituting a low\-rank product,

𝐌⟶αr​𝐁​𝒜,𝐁∈ℝdout×rlora,𝒜∈ℝrlora×din,rlora≪min⁡\(dout,din\),\\mathbf\{M\}\\;\\longrightarrow\\;\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\},\\qquad\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times r\_\{\\text\{lora\}\}\},\\ \\mathcal\{A\}\\in\\mathbb\{R\}^\{r\_\{\\text\{lora\}\}\\times d\_\{\\text\{in\}\}\},\\ r\_\{\\text\{lora\}\}\\ll\\min\(d\_\{\\text\{out\}\},d\_\{\\text\{in\}\}\),\(12\)yields the parameter\-efficient variantFora\-LR \(andFProj\-LR without the spectral channel\)

Δ​𝐖Fora−L​R=P𝐔​\(αr​𝐁​𝒜\)​P𝐐\+𝐔2​𝐃δ​𝐕2⊤\\boxed\{\\;\\Delta\\\!\\mathbf\{W\}\_\{\\textsf\{Fora\}\-LR\}=P\_\{\\\!\\mathbf\{U\}\}\\\!\\left\(\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}\\right\)\\\!P\_\{\\\!\\mathbf\{Q\}\}\\;\+\\;\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\\;\}\(13\)This is a substitution, not a new method: the principle is established in the full\-rank setting and the low\-rank form inherits it unchanged\. We treat it as evidence of generality \(Section[4\.6](https://arxiv.org/html/2606.31092#S4.SS6)\), not the primary contribution\.

Table 1:Stages of the derivation\. Each stage addresses a limitation of the previous one; the function\-space update is the destination, with the low\-rank form as a substitution\.StageUpdate ruleLimitation motivating the next stageFull FT𝐌\\mathbf\{M\}no protectionSpectral calibration𝐔2​𝐃δ​𝐕2⊤\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}diagonal: no mixing across residual modesResidual expansion𝐔2​\(𝐃δ\+𝐒res\)​𝐕2⊤\\mathbf\{U\}\_\{2\}\(\\mathbf\{D\}\_\{\\delta\}\+\\mathbf\{S\}\_\{\\mathrm\{res\}\}\)\\mathbf\{V\}\_\{2\}^\{\\top\}input/output structure hidden in SVD coordsWeight\-space projection𝐔2​𝐃δ​𝐕2⊤\+P𝐔​𝐌​P𝐕\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\+P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{V\}\}P𝐕P\_\{\\\!\\mathbf\{V\}\}protects weight geometry, not capabilityFora\(ours\)𝐔2​𝐃δ​𝐕2⊤\+P𝐔​𝐌​P𝐐\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\+P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}—Fora\-LR \(low\-rank\)𝐔2​𝐃δ​𝐕2⊤\+P𝐔​αr​𝐁​𝒜​P𝐐\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\+P\_\{\\\!\\mathbf\{U\}\}\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}\\,P\_\{\\\!\\mathbf\{Q\}\}\(parameter\-efficient variant\)

## 4Experiments

We evaluate function\-space protection across three full\-rank settings, isolate the projection source by ablation, and verify the principle transfers to low\-rank adaptation\. The common question is whether protecting the*capability\-derived*subspace𝐐\\mathbf\{Q\}outperforms protecting the*weight\-derived*𝐕1\\mathbf\{V\}\_\{1\}while learning a new task; Figure[1](https://arxiv.org/html/2606.31092#S1.F1)previews the headline result, with our methods occupying the favorable corner of every preservation–adaptation plane\.

### 4\.1Setup

#### Model and protocol\.

All experiments use Qwen3\-1\.7B\(Yanget al\.,[2025](https://arxiv.org/html/2606.31092#bib.bib18)\)\(decoder\-only\) in bf16, applying every method to all attention projections \(Q, K, V, O\) and feed\-forward linear layers in each block\. The function subspace𝐐\\mathbf\{Q\}is built per layer from the topkf=16k\_\{f\}=16eigenvectors of the input\-activation covariance, estimated from label\-free preserved\-capability inputs in one forward pass; the SVD rank forP𝐔/P𝐕P\_\{\\\!\\mathbf\{U\}\}/P\_\{\\\!\\mathbf\{V\}\}isr=100r=100\. We optimize with AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2606.31092#bib.bib19)\)\(lr1×10−51\\\!\\times\\\!10^\{\-5\}, batch 8, grad\-accum 8\) and early stopping \(patience 5\), selecting the best\-validation\-loss checkpoint\. All full\-rank methods train the same parameter count \(∼1\.7\\sim\\\!1\.7B\)\. Results are mean±\\pmstd over 5 seeds unless noted\. Full hyperparameters and dataset splits are given in Appendix[A](https://arxiv.org/html/2606.31092#A1)\(Table[8](https://arxiv.org/html/2606.31092#A1.T8)\), and per\-seed results for all full\-rank settings in Appendix[B](https://arxiv.org/html/2606.31092#A2)\.

#### Starting models\.

The starting weights𝐖0\\mathbf\{W\}\_\{0\}are*not*raw Qwen3\-1\.7B\. To create a well\-defined capability to preserve, we first LoRA\-fine\-tune the base model on 5,000 examples \(translation or GSM8K\) and merge the adapter into the weights; this capability\-specialized checkpoint is𝐖0\\mathbf\{W\}\_\{0\}, and its merged capability is what we protect while learning the new task\. The baseline preserved\-capability numbers \(translation chrF=0\.413=0\.413, PPL=4\.35=4\.35; GSM8K EM=49\.7=49\.7\) are measured on these checkpoints\.

#### Tasks and baselines\.

COGS\(Kim and Linzen,[2020](https://arxiv.org/html/2606.31092#bib.bib14)\)\(compositional semantic parsing; 1000/300/500\), GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.31092#bib.bib15)\)\(grade\-school math; 1000/300/1319\), and bidirectional English–Chinese translation \(OPUS\(Tiedemann,[2012](https://arxiv.org/html/2606.31092#bib.bib17)\); 1000/300/500\), reporting COGS/GSM8K exact match \(EM\), translation chrF, and translation perplexity \(PPL, lower better\)\. Baselines: Full FT\(Δ​𝐖=𝐌\\Delta\\\!\\mathbf\{W\}=\\mathbf\{M\}\);WProj\(Δ​𝐖=P𝐔​𝐌​P𝐕\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{V\}\}\); EWC\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.31092#bib.bib1)\); L2\-SP\(Liet al\.,[2018](https://arxiv.org/html/2606.31092#bib.bib2)\); LwF\(Li and Hoiem,[2018](https://arxiv.org/html/2606.31092#bib.bib3)\); and ours,FProj\(Δ​𝐖=P𝐔​𝐌​P𝐐\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\) andFora\(Δ​𝐖=P𝐔​𝐌​P𝐐\+𝐔2​𝐃δ​𝐕2⊤\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\+\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\)\.

### 4\.2COGS Fine\-Tuning with Translation Retention

Our main setting starts from the translation\-specialized𝐖0\\mathbf\{W\}\_\{0\}, builds𝐐\\mathbf\{Q\}from translation calibration inputs, and fine\-tunes all linear layers on COGS\. Table[2](https://arxiv.org/html/2606.31092#S4.T2)reports the result\.

Table 2:COGS Fine\-Tuning with Translation Retention\.All methods full\-rank \(∼1\.7\\sim\\\!1\.7B params\), 5 seeds; the starting model is the translation\-specialized𝐖0\\mathbf\{W\}\_\{0\}\.FProj\(Δ​𝐖=P𝐔​𝐌​P𝐐\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\) is the projection\-only ablation;Fora\(Δ​𝐖=P𝐔​𝐌​P𝐐\+𝐔2​𝐃δ​𝐕2⊤\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\+\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\) is the full method\. Bold rows are ours\.CategoryMethodCOGS EM↑\\uparrowchrF↑\\uparrowΔ\\DeltachrFPPL↓\\downarrowBest EpBaselineTranslation—0\.413±0\.0100\.413\_\{\\pm 0\.010\}—4\.35±0\.194\.35\_\{\\pm 0\.19\}—No protectionFull FT99\.00±0\.4599\.00\_\{\\pm 0\.45\}0\.405±0\.0060\.405\_\{\\pm 0\.006\}−0\.01\-0\.014\.74±0\.244\.74\_\{\\pm 0\.24\}6\.0Weight\-spaceWProj98\.92±0\.5898\.92\_\{\\pm 0\.58\}0\.405±0\.0080\.405\_\{\\pm 0\.008\}−0\.01\-0\.014\.86±0\.224\.86\_\{\\pm 0\.22\}4\.4RegularizationEWC98\.84±0\.6198\.84\_\{\\pm 0\.61\}0\.404±0\.0070\.404\_\{\\pm 0\.007\}−0\.01\-0\.014\.77±0\.214\.77\_\{\\pm 0\.21\}6\.0RegularizationL2\-SP98\.48±0\.8698\.48\_\{\\pm 0\.86\}0\.405±0\.0060\.405\_\{\\pm 0\.006\}−0\.01\-0\.014\.75±0\.214\.75\_\{\\pm 0\.21\}5\.0RegularizationLwF92\.12±1\.1592\.12\_\{\\pm 1\.15\}0\.406±0\.0070\.406\_\{\\pm 0\.007\}−0\.01\-0\.014\.39±0\.184\.39\_\{\\pm 0\.18\}8\.2OursFProj98\.64±0\.99\\mathbf\{98\.64\_\{\\pm 0\.99\}\}0\.414±0\.010\\mathbf\{0\.414\_\{\\pm 0\.010\}\}\+0\.00\\mathbf\{\+0\.00\}4\.39±0\.19\\mathbf\{4\.39\_\{\\pm 0\.19\}\}5\.0Fora98\.68±0\.81\\mathbf\{98\.68\_\{\\pm 0\.81\}\}0\.414±0\.011\\mathbf\{0\.414\_\{\\pm 0\.011\}\}\+0\.00\\mathbf\{\+0\.00\}4\.39±0\.19\\mathbf\{4\.39\_\{\\pm 0\.19\}\}4\.8We highlight three findings\.\(1\)P𝐐P\_\{\\\!\\mathbf\{Q\}\}outperformsP𝐕P\_\{\\\!\\mathbf\{V\}\}\.Weight\-space projection degrades translation PPL to 4\.86—higher than even unprotected Full FT\(4\.74\)—whereas function\-space projection maintains PPL at 4\.39, within 0\.04 of baseline, at 98\.64% COGS EM \(only 0\.36 below Full FT\)\.\(2\)P𝐐P\_\{\\\!\\mathbf\{Q\}\}outperforms regularization\.EWC and L2\-SP reach PPL 4\.75–4\.77, statistically indistinguishable from unprotected fine\-tuning\.\(3\)P𝐐P\_\{\\\!\\mathbf\{Q\}\}avoids the distillation trade\-off\.LwF attains PPL 4\.39 but at a COGS EM of 92\.12% \(versus\>98\.5%\>98\.5\\%for all other methods\), reflecting the stability–plasticity tension\. Function\-space protection is the only method that achieves both strong preservation and strong new\-task learning\.

### 4\.3GSM8K Fine\-Tuning with Translation Retention

We repeat the setting with GSM8K as the new task\. GSM8K is far harder than COGS for a 1\.7B model trained on 1000 examples \(best EM∼47%\\sim\\\!47\\%vs∼99%\\sim\\\!99\\%\), testing whether protection still helps when new\-task headroom is narrow\.

Table 3:GSM8K Fine\-Tuning with Translation Retention\.Same translation\-specialized𝐖0\\mathbf\{W\}\_\{0\}as Table[2](https://arxiv.org/html/2606.31092#S4.T2)\(chrF0\.4130\.413, PPL4\.354\.35\); the new task is GSM8K\. 5 seeds\.MethodGSM8K EM↑\\uparrowchrF↑\\uparrowΔ\\DeltachrFPPL↓\\downarrowBest EpTranslation baseline—0\.413±0\.0100\.413\_\{\\pm 0\.010\}—4\.35±0\.194\.35\_\{\\pm 0\.19\}—Full FT46\.13±1\.0846\.13\_\{\\pm 1\.08\}0\.406±0\.0100\.406\_\{\\pm 0\.010\}−0\.01\-0\.014\.40±0\.174\.40\_\{\\pm 0\.17\}1\.8WProj46\.46±0\.9646\.46\_\{\\pm 0\.96\}0\.406±0\.0100\.406\_\{\\pm 0\.010\}−0\.01\-0\.014\.42±0\.184\.42\_\{\\pm 0\.18\}1\.8EWC46\.60±1\.1846\.60\_\{\\pm 1\.18\}0\.405±0\.0100\.405\_\{\\pm 0\.010\}−0\.01\-0\.014\.40±0\.174\.40\_\{\\pm 0\.17\}1\.8L2\-SP46\.69±0\.6046\.69\_\{\\pm 0\.60\}0\.405±0\.0100\.405\_\{\\pm 0\.010\}−0\.01\-0\.014\.40±0\.174\.40\_\{\\pm 0\.17\}1\.8LwF46\.07±0\.7846\.07\_\{\\pm 0\.78\}0\.407±0\.0100\.407\_\{\\pm 0\.010\}−0\.01\-0\.014\.34±0\.174\.34\_\{\\pm 0\.17\}2\.8FProj46\.66±0\.91\\mathbf\{46\.66\_\{\\pm 0\.91\}\}0\.411±0\.010\\mathbf\{0\.411\_\{\\pm 0\.010\}\}−0\.00\\mathbf\{\-0\.00\}4\.35±0\.17\\mathbf\{4\.35\_\{\\pm 0\.17\}\}1\.8Fora46\.87±1\.01\\mathbf\{46\.87\_\{\\pm 1\.01\}\}0\.411±0\.010\\mathbf\{0\.411\_\{\\pm 0\.010\}\}−0\.00\\mathbf\{\-0\.00\}4\.35±0\.17\\mathbf\{4\.35\_\{\\pm 0\.17\}\}1\.8GSM8K EM spans only 0\.8 points across all methods \(46\.07–46\.87\), reflecting the task’s difficulty at this scale; the discriminating axis is therefore preservation\.FProjandForamaintain chrF at 0\.411 and PPL at 4\.35—indistinguishable from baseline—while Full FT,WProj, EWC, and L2\-SPall drift \(chrF 0\.405–0\.406, PPL 4\.40–4\.42\)\.Foraadditionally attains the highest GSM8K EM \(46\.87\): when new\-task headroom is limited, the spectral channel’s modest additional plasticity yields a small improvement without disturbing preservation\. As Figure[1](https://arxiv.org/html/2606.31092#S1.F1)\(b\) shows, our methods occupy the favorable region in this setting as well\.

### 4\.4Translation Fine\-Tuning with Math Reasoning Retention

To show that𝐐\\mathbf\{Q\}is not specific to translation, we reverse the roles: beginning from a GSM8K\-fine\-tuned model, we learn translation as the new task while preserving math reasoning, with𝐐=𝐐math\\mathbf\{Q\}=\\mathbf\{Q\}\_\{\\text\{math\}\}constructed from GSM8K calibration inputs\. This is the more demanding direction and the one most likely to reveal forgetting\.

Table 4:Translation Fine\-Tuning with Math Reasoning Retention\.Starting model: GSM8K\-tuned Qwen3\-1\.7B;𝐐=𝐐math\\mathbf\{Q\}=\\mathbf\{Q\}\_\{\\text\{math\}\}from GSM8K inputs; 5 seeds\.Δ\\DeltaGSM8K is measured against the GSM8K\-tuned starting model \(49\.7\)\.MethodchrF↑\\uparrowPPL↓\\downarrowGSM8K EM↑\\uparrowΔ\\DeltaGSM8KBest EpStarting model——49\.70——Full FT0\.396±0\.0080\.396\_\{\\pm 0\.008\}4\.79±0\.224\.79\_\{\\pm 0\.22\}47\.43±0\.6847\.43\_\{\\pm 0\.68\}−2\.27\-2\.271\.0WProj0\.395±0\.0080\.395\_\{\\pm 0\.008\}4\.79±0\.224\.79\_\{\\pm 0\.22\}47\.05±0\.8447\.05\_\{\\pm 0\.84\}−2\.65\-2\.651\.0EWC0\.395±0\.0090\.395\_\{\\pm 0\.009\}4\.79±0\.224\.79\_\{\\pm 0\.22\}47\.26±0\.8047\.26\_\{\\pm 0\.80\}−2\.44\-2\.441\.0L2\-SP0\.394±0\.0070\.394\_\{\\pm 0\.007\}4\.79±0\.224\.79\_\{\\pm 0\.22\}47\.13±0\.4747\.13\_\{\\pm 0\.47\}−2\.57\-2\.571\.0LwF0\.395±0\.0070\.395\_\{\\pm 0\.007\}5\.03±0\.225\.03\_\{\\pm 0\.22\}48\.28±0\.6848\.28\_\{\\pm 0\.68\}−1\.42\-1\.422\.2FProj0\.390±0\.0080\.390\_\{\\pm 0\.008\}4\.80±0\.224\.80\_\{\\pm 0\.22\}48\.84±0\.36\\mathbf\{48\.84\_\{\\pm 0\.36\}\}−0\.86\\mathbf\{\-0\.86\}1\.0Fora0\.390±0\.009\\mathbf\{0\.390\_\{\\pm 0\.009\}\}4\.80±0\.224\.80\_\{\\pm 0\.22\}48\.84±0\.42\\mathbf\{48\.84\_\{\\pm 0\.42\}\}−0\.86\\mathbf\{\-0\.86\}1\.0Table[4](https://arxiv.org/html/2606.31092#S4.T4)confirms that the pattern extends to math\. Our methods retain GSM8K at 48\.84% \(Δ=−0\.86\\Delta=\-0\.86\), versus−2\.27\-2\.27for Full FTand−2\.4\-2\.4to−2\.7\-2\.7for the weight\-space and regularization baselines—approximately a threefold reduction in forgetting\. LwF is intermediate \(Δ=−1\.42\\Delta=\-1\.42\) but degrades translation \(PPL 5\.03 vs\. 4\.79–4\.80\)\. The sole trade\-off is a small reduction in chrF forFProj/Fora\(0\.390 vs\. 0\.396\): when the preserved and new capabilities share activation structure, protecting𝐐math\\mathbf\{Q\}\_\{\\text\{math\}\}withholds a portion of new\-task capacity, a modest and deliberate cost given the substantial retention gain\. Importantly, constructing𝐐\\mathbf\{Q\}from math and preserving math excludes the alternative explanation that𝐐\\mathbf\{Q\}merely encodes translation\.

### 4\.5Ablation: The Projection Source Is What Matters

#### P𝐕P\_\{\\\!\\mathbf\{V\}\}versusP𝐐P\_\{\\\!\\mathbf\{Q\}\}\.

The central mechanistic claim is that projecting is not enough—*what*subspace is projected onto decides preservation\. Table[5](https://arxiv.org/html/2606.31092#S4.T5)isolates this by swapping only the right projector \(P𝐕→P𝐐P\_\{\\\!\\mathbf\{V\}\}\\\!\\to\\\!P\_\{\\\!\\mathbf\{Q\}\}\) while holdingP𝐔P\_\{\\\!\\mathbf\{U\}\}and the high\-capacity branch𝐌\\mathbf\{M\}fixed, across all three settings\.

Table 5:Projection Source Ablation: Weight\-Space vs\. Function\-Space\.Swapping only the right projectorP𝐕→P𝐐P\_\{\\\!\\mathbf\{V\}\}\\\!\\to\\\!P\_\{\\\!\\mathbf\{Q\}\}while holdingP𝐔P\_\{\\\!\\mathbf\{U\}\}and𝐌\\mathbf\{M\}fixed\.P𝐐P\_\{\\\!\\mathbf\{Q\}\}wins on the preserved metric in every setting\.SettingRight projectorNew task↑\\uparrowPreservedTrans→\\toCOGSP𝐕P\_\{\\\!\\mathbf\{V\}\}\(weight\)COGS EM98\.9298\.92PPL4\.864\.86P𝐐P\_\{\\\!\\mathbf\{Q\}\}\(function\)COGS EM98\.6498\.64PPL4\.39\\mathbf\{4\.39\}Trans→\\toGSM8KP𝐕P\_\{\\\!\\mathbf\{V\}\}\(weight\)GSM8K EM46\.4646\.46PPL4\.424\.42P𝐐P\_\{\\\!\\mathbf\{Q\}\}\(function\)GSM8K EM46\.6646\.66PPL4\.35\\mathbf\{4\.35\}Math→\\toTransP𝐕P\_\{\\\!\\mathbf\{V\}\}\(weight\)chrF0\.3950\.395GSM8K EM47\.0547\.05P𝐐P\_\{\\\!\\mathbf\{Q\}\}\(function\)chrF0\.3900\.390GSM8K EM48\.84\\mathbf\{48\.84\}In every settingP𝐐P\_\{\\\!\\mathbf\{Q\}\}improves the preserved metric overP𝐕P\_\{\\\!\\mathbf\{V\}\}: PPL decreases by 0\.47 \(COGS\) and 0\.07 \(GSM8K\), and GSM8K retention increases by 1\.79 points \(Math→\\toTrans\); new\-task performance is comparable or marginally lower, the expected consequence of protecting capability\-relevant directions more precisely \(Figure[4](https://arxiv.org/html/2606.31092#A3.F4), Appendix[C](https://arxiv.org/html/2606.31092#A3)\)\. We conclude thatprojection alone is insufficient; the projection*source*—weight geometry versus capability function—is the decisive factor\.

#### Contribution of the spectral channel\.

Table[6](https://arxiv.org/html/2606.31092#S4.T6)contrastsFProj\(without𝜹\\bm\{\\delta\}\) andFora\(with𝜹\\bm\{\\delta\}\)\. Adding the channel never degrades a preserved metric, and where the new task admits additional plasticity it produces a small, consistent improvement—most evident as\+0\.21\+0\.21GSM8K EM in the headroom\-limited Trans→\\toGSM8K setting and\+0\.04\+0\.04on COGS\. This matches the behavior characterized by Proposition[3](https://arxiv.org/html/2606.31092#Thmproposition3): a bounded calibration channel that the optimizer can exploit when beneficial and that otherwise remains inactive, leaving the protection guarantee intact\. We therefore retain𝜹\\bm\{\\delta\}as a low\-cost component of the full method\.

Table 6:Spectral Channel Ablation:FProjvs\.Fora\.FProj:Δ​𝐖=P𝐔​𝐌​P𝐐\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\};Fora:Δ​𝐖=P𝐔​𝐌​P𝐐\+𝐔2​𝐃δ​𝐕2⊤\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\+\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\. The channel adds a small new\-task margin where headroom exists and never harms preservation\.Setting / metricFProjForaΔ\\DeltaCOGS EM98\.6498\.6498\.6898\.68\+0\.04\+0\.04COGS translation PPL4\.394\.394\.394\.390\.00\\;\\,0\.00GSM8K EM46\.6646\.6646\.8746\.87\+0\.21\+0\.21GSM8K translation PPL4\.354\.354\.354\.350\.00\\;\\,0\.00Math→\\toTrans chrF0\.38970\.38970\.39040\.3904\+0\.0007\+0\.0007Math→\\toTrans GSM8K EM48\.8448\.8448\.8448\.840\.00\\;\\,0\.00
#### Subspace dimensionsrrandkfk\_\{f\}\.

The method has two structural dimensions: the SVD frozen rankrrinP𝐔P\_\{\\\!\\mathbf\{U\}\}\(default 100\) and the function\-subspace sizekfk\_\{f\}inP𝐐P\_\{\\\!\\mathbf\{Q\}\}\(default 16\)\. Sweeping each independently on the Math→\\toTrans setting \(5 seeds per configuration\), GSM8K retention stays within a0\.40\.4\-point band asrrranges over2020–400400andkfk\_\{f\}over44–6464, with translation chrF/PPL unchanged—every configuration lies within one standard deviation of the default, so\(r,kf\)=\(100,16\)\(r,k\_\{f\}\)=\(100,16\)is not a tuned operating point\. This robustness concerns*how many*directions are protected, not*which*subspace they come from; the latter remains decisive \(Table[5](https://arxiv.org/html/2606.31092#S4.T5)\)\. Full results are in Appendix[E\.1](https://arxiv.org/html/2606.31092#A5.SS1)\(Table[14](https://arxiv.org/html/2606.31092#A5.T14)\)\.

### 4\.6Low\-Rank Extension

Finally, we verify that the principle transfers to parameter\-efficient fine\-tuning\. Table[7](https://arxiv.org/html/2606.31092#S4.T7)compares methods at matched parameter count \(∼17\\sim\\\!17M\) under COGS fine\-tuning with translation retention\.

Table 7:Low\-Rank Extension under COGS Fine\-Tuning\.Replacing the full\-rank matrix𝐌\\mathbf\{M\}with a LoRA\(Huet al\.,[2022](https://arxiv.org/html/2606.31092#bib.bib13)\)low\-rank productαr​𝐁​𝒜\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}; all methods∼17\\sim\\\!17M trainable parameters, 5 seeds\.FProj\-LR:Δ​𝐖=P𝐔​αr​𝐁​𝒜​P𝐐\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}\\,P\_\{\\\!\\mathbf\{Q\}\};Fora\-LR adds the spectral channel𝐔2​𝐃δ​𝐕2⊤\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\. This table evaluates generality, not the main full\-rank claim\. Baseline PPL 4\.35\.MethodCOGS EM↑\\uparrowchrF↑\\uparrowPPL↓\\downarrowLoRA\-r1695\.44±1\.7595\.44\_\{\\pm 1\.75\}0\.414±0\.0100\.414\_\{\\pm 0\.010\}4\.54±0\.204\.54\_\{\\pm 0\.20\}LoRA\-r16 \+ replay 1%95\.52±1\.7895\.52\_\{\\pm 1\.78\}0\.414±0\.0090\.414\_\{\\pm 0\.009\}4\.50±0\.194\.50\_\{\\pm 0\.19\}LoRA\-r16 \+ replay 10%95\.84±1\.77\\mathbf\{95\.84\_\{\\pm 1\.77\}\}0\.405±0\.0100\.405\_\{\\pm 0\.010\}4\.55±0\.174\.55\_\{\\pm 0\.17\}OPLoRA\-r1694\.44±2\.5594\.44\_\{\\pm 2\.55\}0\.414±0\.0100\.414\_\{\\pm 0\.010\}4\.54±0\.184\.54\_\{\\pm 0\.18\}CorDA\-k16\-r1694\.16±1\.4094\.16\_\{\\pm 1\.40\}0\.414±0\.0090\.414\_\{\\pm 0\.009\}4\.38±0\.19\\mathbf\{4\.38\_\{\\pm 0\.19\}\}FProj\-LR94\.60±2\.2994\.60\_\{\\pm 2\.29\}0\.413±0\.0100\.413\_\{\\pm 0\.010\}4\.38±0\.18\\mathbf\{4\.38\_\{\\pm 0\.18\}\}Fora\-LR94\.48±2\.2194\.48\_\{\\pm 2\.21\}0\.413±0\.0090\.413\_\{\\pm 0\.009\}4\.38±0\.19\\mathbf\{4\.38\_\{\\pm 0\.19\}\}The protection mechanism behaves identically at low rank\.Fora\-LR and its projection\-only variantFProj\-LR match CorDA on translation PPL \(4\.38, closest to baseline\) and exceed OPLoRA’s weight\-space projection on COGS EM \(94\.60 vs\. 94\.44\) at lower PPL\. Replay\-based LoRA attains the highest COGS EM \(95\.84% at 10% replay\), but at higher PPL \(4\.50–4\.55\) and with a requirement for preserved\-capability labels and mixed training that function\-space protection does not impose\. We present this not as a contribution to PEFT but as evidence that the sameP𝐔​\(⋅\)​P𝐐P\_\{\\\!\\mathbf\{U\}\}\(\\cdot\)P\_\{\\\!\\mathbf\{Q\}\}structure preserves capability whether the adapter is dense or low\-rank; the full\-rank results \(Tables[2](https://arxiv.org/html/2606.31092#S4.T2)–[4](https://arxiv.org/html/2606.31092#S4.T4)\) remain the primary contribution\.

## 5Discussion

#### Why weight\-space projection fails\.

The SVD of𝐖0\\mathbf\{W\}\_\{0\}captures the weight’s dominant linear modes—an aggregate over every capability the model holds—so a single capability may reside along moderate\-singular\-value directions that the top\-rrblock excludes\. Removingspan⁡\(𝐕1\)\\operatorname\{span\}\(\\mathbf\{V\}\_\{1\}\)then has two adverse effects: it leaves capability\-critical directions outside the block unprotected, and it constrains in\-block directions that the capability does not use\.P𝐐P\_\{\\\!\\mathbf\{Q\}\}avoids both by acting on the input space the capability*actually activates*, which accounts for its retaining the preserved capability across all three settings whereP𝐕P\_\{\\\!\\mathbf\{V\}\}does not\.

#### Two complementary mechanisms\.

Protection and plasticity are assigned to separate components: the projectors\(P𝐔,P𝐐\)\(P\_\{\\\!\\mathbf\{U\}\},P\_\{\\\!\\mathbf\{Q\}\}\)provide structural protection, while the spectral channel𝜹\\bm\{\\delta\}—the only component permitted to act on the protected directions \(Proposition[3](https://arxiv.org/html/2606.31092#Thmproposition3)\)—provides a bounded degree of plasticity there\. Empirically, this channel yields a small, consistent new\-task improvement where the task admits headroom and remains inactive otherwise, never reducing preservation \(Table[6](https://arxiv.org/html/2606.31092#S4.T6)\); Appendix[D](https://arxiv.org/html/2606.31092#A4)confirms that it carries real but limited plasticity in isolation\. The two roles are thus complementary rather than competing, so the method gains capacity without compromising the protection guarantee\.

#### Scope and generality\.

Full\-rank methods reach 98\.5–99\.0% COGS EM, against 95\.8% for the best low\-rank method \(Tables[2](https://arxiv.org/html/2606.31092#S4.T2),[7](https://arxiv.org/html/2606.31092#S4.T7)\)—a∼3\\sim\\\!3\-point plasticity gap under an identical protection mechanism, leading us to recommend full\-rank fine\-tuning where resources permit and the low\-rank form otherwise\. Finally,𝐐\\mathbf\{Q\}is not specific to translation: constructed from GSM8K inputs,𝐐math\\mathbf\{Q\}\_\{\\text\{math\}\}preserves math reasoning during translation fine\-tuning \(Table[4](https://arxiv.org/html/2606.31092#S4.T4)\)\. As the construction requires only label\-free inputs, function\-space protection applies to any preservation target for which representative calibration data are available\.

## 6Limitations and Future Work

Single\-capability protection\.The current formulation protects one capability through a single𝐐\\mathbf\{Q\}; preserving several simultaneously requires a union projectorP𝐐=𝐈−\[𝐐1​𝐐2\]​\[𝐐1​𝐐2\]⊤P\_\{\\\!\\mathbf\{Q\}\}=\\mathbf\{I\}\-\[\\mathbf\{Q\}\_\{1\}\\;\\mathbf\{Q\}\_\{2\}\]\[\\mathbf\{Q\}\_\{1\}\\;\\mathbf\{Q\}\_\{2\}\]^\{\\top\}, which raises the effective dimension and may reduce the high\-capacity branch’s adaptive room\.Subspace dimensions\.Forais robust to both structural dimensions on the setting we sweep \(Table[14](https://arxiv.org/html/2606.31092#A5.T14)\); whether this insensitivity holds across tasks, and whether an automatic eigenvalue\-decay criterion can selectkfk\_\{f\}, remains open\.Calibration data\.𝐐\\mathbf\{Q\}depends on the calibration inputs; its sensitivity to their number \(we use 500–1000\) and to domain shift is not yet characterized\.Replay\.Replay remains strong at low rank \(Table[7](https://arxiv.org/html/2606.31092#S4.T7)\); whether data\-level and structural protection are complementary is a natural next step\.Scale and architecture\.All experiments use Qwen3\-1\.7B; larger models, other architectures, and additional capabilities would strengthen the generality claims\. None of these issues bears on the central claim—that activation\-derived function subspaces protect capabilities more faithfully than weight\-space projections—but each identifies a direction for further study\.

## 7Conclusion

We have argued that capability preservation in fine\-tuning is more appropriately formulated asfunction\-space protectionthan as weight\-space protection\. The activation\-derived subspace𝐐\\mathbf\{Q\}—constructed from preserved\-capability inputs without labels—is a more faithful protection target than the weight\-derived𝐕1\\mathbf\{V\}\_\{1\}, as it captures the directions a capability uses rather than those in which the weight is large\. Instantiated asΔ​𝐖=P𝐔​𝐌​P𝐐\+𝐔2​𝐃δ​𝐕2⊤\\Delta\\\!\\mathbf\{W\}=P\_\{\\\!\\mathbf\{U\}\}\\mathbf\{M\}P\_\{\\\!\\mathbf\{Q\}\}\+\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}, the method preserves capabilities more effectively than weight\-space projection and standard regularization across three settings, at the cost of only a small new\-task trade\-off in the math\-preservation setting, and a controlled ablation attributes the improvement to the projection*source*rather than to projection itself\. Established in full\-rank fine\-tuning, the principle transfers without modification to low\-rank adaptation via𝐌→αr​𝐁​𝒜\\mathbf\{M\}\\to\\tfrac\{\\alpha\}\{r\}\\mathbf\{B\}\\mathcal\{A\}\.

## References

- R\. Aljundi, F\. Babiloni, M\. Elhoseiny, M\. Rohrbach, and T\. Tuytelaars \(2018\)Memory aware synapses: learning what \(not\) to forget\.InEuropean Conference on Computer Vision \(ECCV\),External Links:[Document](https://dx.doi.org/10.1007/978-3-030-01219-9%5F9)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. Nanda \(2024\)Refusal in language models is mediated by a single direction\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/f545448535dfde4f9786555403ab7c49-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px4.p1.4)\.
- A\. Chaudhry, M\. Ranzato, M\. Rohrbach, and M\. Elhoseiny \(2019\)Efficient lifelong learning with A\-GEM\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Hkf2_sC5FX)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px3.p1.4)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://neurips.cc/virtual/2023/oral/73855)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px3.p1.5)\.
- R\. M\. French \(1999\)Catastrophic forgetting in connectionist networks\.Trends in Cognitive Sciences3\(4\),pp\. 128–135\.External Links:[Document](https://dx.doi.org/10.1016/S1364-6613%2899%2901294-2)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1)\.
- Z\. Han, C\. Gao, J\. Liu, J\. Zhang, and S\. Q\. Zhang \(2024\)Parameter\-efficient fine\-tuning for large models: a comprehensive survey\.Transactions on Machine Learning Research \(TMLR\)\.External Links:[Link](https://arxiv.org/abs/2403.14608)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Table 7](https://arxiv.org/html/2606.31092#S4.T7)\.
- N\. Kim and T\. Linzen \(2020\)COGS: a compositional generalization challenge based on semantic interpretation\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.731)Cited by:[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px3.p1.4)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska, D\. Hassabis, C\. Clopath, D\. Kumaran, and R\. Hadsell \(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1611835114)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px3.p1.4)\.
- X\. Li, Y\. Grandvalet, and F\. Davoine \(2018\)Explicit inductive bias for transfer learning with convolutional networks\.InInternational Conference on Machine Learning \(ICML\),pp\. 2825–2834\.External Links:[Link](https://proceedings.mlr.press/v80/li18a.html)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px3.p1.4)\.
- Z\. Li and D\. Hoiem \(2018\)Learning without forgetting\.IEEE Transactions on Pattern Analysis and Machine Intelligence40\(12\),pp\. 2935–2947\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2017.2773081)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px3.p1.4)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024a\)DoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://proceedings.mlr.press/v235/liu24p.html)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px3.p1.5)\.
- W\. Liu, Z\. Qiu, Y\. Feng, Y\. Xiu, Y\. Xue, L\. Yu, H\. Feng, Z\. Liu, J\. Heo, S\. Peng, Y\. Wen, M\. J\. Black, A\. Weller, and B\. Schölkopf \(2024b\)Parameter\-efficient orthogonal finetuning via butterfly factorization\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=7NzgkEdGyr)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px3.p1.5)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 6467–6476\.Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[1st item](https://arxiv.org/html/2606.31092#A1.I2.i1.p1.1),[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px1.p1.7)\.
- M\. McCloskey and N\. J\. Cohen \(1989\)Catastrophic interference in connectionist networks: the sequential learning problem\.InPsychology of Learning and Motivation,Vol\.24,pp\. 109–165\.External Links:[Document](https://dx.doi.org/10.1016/S0079-7421%2808%2960536-8)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1)\.
- F\. Meng, Z\. Wang, and M\. Zhang \(2024\)PiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://papers.nips.cc/paper_files/paper/2024/hash/db36f4d603cc9e3a2a5e10b93e6428f2-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px3.p1.5)\.
- F\. Paischer, L\. Hauzenberger, T\. Schmied, B\. Alkin, M\. P\. Deisenroth, and S\. Hochreiter \(2025\)One initialization to rule them all: fine\-tuning via explained variance adaptation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://neurips.cc/virtual/2025/loc/san-diego/poster/116183)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px4.p1.4)\.
- G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. Wermter \(2019\)Continual lifelong learning with neural networks: a review\.Neural Networks113,pp\. 54–71\.External Links:[Document](https://dx.doi.org/10.1016/j.neunet.2019.01.012)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are NLP models really able to solve simple math word problems?\.InAnnual Conference of the Nations of the Americas Chapter of the ACL \(NAACL\),External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.168)Cited by:[Appendix D](https://arxiv.org/html/2606.31092#A4.p1.4)\.
- T\. G\. J\. Rudner, S\. Kapoor, S\. Qiu, and A\. G\. Wilson \(2023\)Function\-space regularization in neural networks: a probabilistic perspective\.InInternational Conference on Machine Learning \(ICML\),pp\. 29275–29290\.External Links:[Link](https://proceedings.mlr.press/v202/rudner23a.html)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px4.p1.4)\.
- G\. Saha, I\. Garg, and K\. Roy \(2021\)Gradient projection memory for continual learning\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=3AOj0RCNC2)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Shi, Z\. Xu, H\. Wang, W\. Qin, W\. Wang, Y\. Wang, Z\. Wang, S\. Ebrahimi, and H\. Wang \(2025\)Continual learning of large language models: a comprehensive survey\.ACM Computing Surveys\.External Links:[Document](https://dx.doi.org/10.1145/3735633)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1)\.
- J\. Tiedemann \(2012\)Parallel data, tools and interfaces in OPUS\.InInternational Conference on Language Resources and Evaluation \(LREC\),External Links:[Link](https://aclanthology.org/L12-1246/)Cited by:[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px3.p1.4)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p1.1)\.
- H\. Wang, Z\. Xiao, Y\. Li, S\. Wang, G\. Chen, and Y\. Chen \(2025\)MiLoRA: harnessing minor singular components for parameter\-efficient llm finetuning\.InAnnual Conference of the Nations of the Americas Chapter of the ACL \(NAACL\),External Links:[Link](https://aclanthology.org/2025.naacl-long.248/)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px3.p1.5)\.
- X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. Huang \(2023\)Orthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.715)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Xiong and X\. Xie \(2026\)OPLoRA: orthogonal projection lora prevents catastrophic forgetting during parameter\-efficient fine\-tuning\.InAAAI Conference on Artificial Intelligence,External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i40.40703)Cited by:[§1](https://arxiv.org/html/2606.31092#S1.p2.1),[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Lu, J\. Jiao, J\. Tu, J\. Zhang, J\. Yang,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[1st item](https://arxiv.org/html/2606.31092#A1.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2606.31092#S4.SS1.SSS0.Px1.p1.7)\.
- Y\. Yang, X\. Li, Z\. Zhou, S\. L\. Song, J\. Wu, L\. Nie, and B\. Ghanem \(2024\)CorDA: context\-oriented decomposition adaptation of large language models for task\-aware parameter\-efficient fine\-tuning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://neurips.cc/virtual/2024/poster/95886)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px4.p1.4)\.
- F\. Zenke, B\. Poole, and S\. Ganguli \(2017\)Continual learning through synaptic intelligence\.InInternational Conference on Machine Learning \(ICML\),pp\. 3987–3995\.External Links:[Link](http://proceedings.mlr.press/v70/zenke17a.html)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=lq62uWRJji)Cited by:[§2](https://arxiv.org/html/2606.31092#S2.SS0.SSS0.Px3.p1.5)\.

## Appendix AImplementation Details

### A\.1Model Configuration

- •Base model: Qwen3\-1\.7B\(Yanget al\.,[2025](https://arxiv.org/html/2606.31092#bib.bib18)\)\(decoder\-only transformer\)
- •Starting checkpoints𝐖0\\mathbf\{W\}\_\{0\}: capability\-specialized models obtained by LoRA fine\-tuning the base model on 5,000 examples \(translation or GSM8K\) and merging the adapter into the weights\. The merged capability is the one preserved during subsequent new\-task fine\-tuning\.
- •Precision: bf16 \(mixed\-precision training and inference\)
- •Target modules: All linear layers in attention \(Q, K, V, O\) and feed\-forward networks \(gate, up, down projections\) across all 28 transformer blocks
- •Frozen rank:r=100r=100for the SVD\-based left projectorP𝐔P\_\{\\\!\\mathbf\{U\}\}\(andP𝐕P\_\{\\\!\\mathbf\{V\}\}in the weight\-space baseline\)
- •Function subspace dimension:kf=16k\_\{f\}=16for𝐐\\mathbf\{Q\}
- •Calibration samples: 500–1000 preserved\-capability inputs \(no labels\)

### A\.2Training Configuration

- •Optimizer: AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2606.31092#bib.bib19)\)
- •Learning rate: 1e\-5 \(full\-rank\), 1e\-4 \(low\-rank\)
- •Batch size: 8 per GPU
- •Gradient accumulation: 8 \(effective batch size = 64\)
- •Max epochs: 30 \(full\-rank\); early stopping patience = 5
- •LoRA configuration:r=16r=16,α=128\\alpha=128; target modules = all linear
- •Seeds: 42, 101, 102, 123, 456 \(5 seeds for most experiments\)

### A\.3Dataset Splits

Table 8:Dataset splits used in all experiments\.DatasetTrainValidTestCOGS1000300500GSM8K10003001319 \(full official\)Translation1000300500 \(bidirectional\)
### A\.4Evaluation Protocol

- •Checkpoint selection: Best validation loss on the new task\.
- •COGS EM: Exact string match between generated and reference output, evaluated on 500 test examples\.
- •GSM8K EM: Exact match on final answer number, evaluated on 1319 test examples with greedy decoding\.
- •Translation chrF/PPL: chrF score and perplexity on 500 bidirectional test examples; PPL computed on response tokens only\.
- •Per\-seed results: Reported as mean±\\pmstandard deviation across seeds\.

## Appendix BFull Per\-Seed Results

This appendix reports the individual\-seed measurements underlying the aggregated means and standard deviations in Section[4](https://arxiv.org/html/2606.31092#S4): Table[9](https://arxiv.org/html/2606.31092#A2.T9)for COGS fine\-tuning \(summarized in Table[2](https://arxiv.org/html/2606.31092#S4.T2)\), Table[10](https://arxiv.org/html/2606.31092#A2.T10)for GSM8K fine\-tuning \(Table[3](https://arxiv.org/html/2606.31092#S4.T3)\), and Table[11](https://arxiv.org/html/2606.31092#A2.T11)for the reverse translation\-with\-math\-preservation direction \(Table[4](https://arxiv.org/html/2606.31092#S4.T4)\)\.

### B\.1COGS Full Fine\-Tuning \(5 Seeds\)

Table 9:Per\-seed results for COGS full fine\-tuning with translation preservation\.MethodSeedCOGS EMchrFPPLVal LossBest EpFull FT4298\.800\.40264\.6000\.0025610198\.400\.41414\.5050\.0032810299\.800\.40864\.8640\.0025612399\.000\.40175\.0290\.0019545699\.000\.39674\.7140\.00255WProj4299\.200\.40134\.6170\.0029510199\.400\.41384\.6600\.0037510298\.400\.41355\.2430\.0022312399\.000\.39614\.9540\.0021545698\.600\.40104\.8280\.00304FProj4298\.600\.41244\.2190\.0024510198\.800\.42424\.1530\.0031610297\.400\.42184\.7210\.0022412399\.400\.40174\.4780\.0019545699\.000\.41014\.3940\.00215Fora4298\.600\.41244\.2100\.0024510198\.000\.42714\.1540\.0030510298\.000\.42194\.7210\.0024412399\.400\.40104\.4780\.0019545699\.400\.40984\.3860\.00215
### B\.2GSM8K Full Fine\-Tuning \(5 Seeds\)

Table 10:Per\-seed results for GSM8K full fine\-tuning with translation preservation\.MethodSeedGSM8K EMchrFPPLVal LossBest EpFull FT10147\.540\.40794\.2700\.432110247\.230\.41024\.2200\.433212345\.720\.41794\.2910\.44524245\.410\.39944\.5680\.422245644\.730\.39284\.6470\.4472FProj10147\.230\.41734\.2310\.433110247\.230\.41504\.1740\.433212345\.640\.41894\.2330\.44724247\.690\.40304\.5020\.421245645\.490\.40004\.5980\.4492Fora10147\.230\.41784\.2350\.433110244\.960\.41524\.1750\.435212347\.920\.42004\.2330\.44324246\.850\.40294\.5010\.421245647\.380\.39884\.5980\.4482
### B\.3Translation Fine\-Tuning with Math Preservation \(5 Seeds\)

Table[11](https://arxiv.org/html/2606.31092#A2.T11)reports the reverse direction: a GSM8K\-tuned starting model, translation as the new task, and math reasoning preserved through𝐐math\\mathbf\{Q\}\_\{\\text\{math\}\}\.

Table 11:Per\-seed results for translation fine\-tuning with math\-reasoning preservation\. GSM8K is the preserved capability; chrF/PPL are the new translation task\.MethodSeedGSM8K EMchrFPPLVal LossBest EpFull FT10146\.550\.40354\.621\.705110247\.010\.39964\.571\.803112348\.140\.40004\.711\.81414248\.070\.38554\.961\.696145647\.380\.39055\.071\.7721WProj10145\.720\.40104\.621\.705110246\.850\.40084\.571\.803112347\.160\.39834\.711\.81314247\.690\.38314\.961\.696145647\.840\.39095\.081\.7721EWC10146\.250\.40214\.621\.704110246\.930\.39924\.571\.804112347\.310\.40094\.711\.81514248\.450\.38324\.961\.697145647\.380\.38725\.071\.7731L2\-SP10146\.550\.40024\.621\.705110246\.930\.39774\.571\.804112347\.230\.39924\.711\.81414247\.840\.38264\.961\.697145647\.080\.39145\.071\.7731LwF10147\.690\.40034\.861\.772310248\.140\.39984\.801\.866212347\.610\.39764\.971\.87924248\.820\.39005\.211\.751245649\.130\.38545\.321\.8292FProj10148\.600\.39814\.651\.712110248\.750\.39614\.581\.811112349\.360\.39264\.731\.82014248\.450\.37944\.961\.703145649\.050\.38225\.091\.7811Fora10148\.220\.39764\.651\.711110249\.130\.39564\.581\.811112349\.280\.39654\.731\.82114248\.900\.37844\.961\.703145648\.670\.38415\.091\.7821

## Appendix CFull Low\-Rank \(PEFT\) Results

Table[12](https://arxiv.org/html/2606.31092#A3.T12)provides the complete PEFT results for the COGS with translation preservation setting, including methods at different parameter scales\.

Table 12:Complete PEFT results on COGS with translation preservation\. Methods are grouped by trainable parameter count\. 5 seeds where available;†denotes 3 seeds\. Translation baseline: chrF =0\.413±0\.0100\.413\\pm 0\.010, PPL =4\.35±0\.194\.35\\pm 0\.19\.MethodParamsCOGS EM↑\\uparrowchrF↑\\uparrowPPL↓\\downarrowBest Ep∼\\sim3M parametersPiSSA\-r3†3\.27M98\.60±0\.8098\.60\_\{\\pm 0\.80\}0\.390±0\.0090\.390\_\{\\pm 0\.009\}6\.31±0\.406\.31\_\{\\pm 0\.40\}5\.7DoRA\-r3†3\.84M98\.00±1\.2598\.00\_\{\\pm 1\.25\}0\.408±0\.0160\.408\_\{\\pm 0\.016\}4\.93±0\.204\.93\_\{\\pm 0\.20\}5\.7LoRA\-r3†3\.27M97\.53±1\.2197\.53\_\{\\pm 1\.21\}0\.406±0\.0150\.406\_\{\\pm 0\.015\}4\.95±0\.204\.95\_\{\\pm 0\.20\}6\.3∼\\sim17M parameters \(main comparison\)LoRA\-r16 replay1017\.43M95\.84±1\.77\\mathbf\{95\.84\_\{\\pm 1\.77\}\}0\.405±0\.0100\.405\_\{\\pm 0\.010\}4\.55±0\.174\.55\_\{\\pm 0\.17\}10\.0LoRA\-r16 replay117\.43M95\.52±1\.7895\.52\_\{\\pm 1\.78\}0\.414±0\.0090\.414\_\{\\pm 0\.009\}4\.50±0\.194\.50\_\{\\pm 0\.19\}9\.6LoRA\-r1617\.43M95\.44±1\.7595\.44\_\{\\pm 1\.75\}0\.414±0\.0100\.414\_\{\\pm 0\.010\}4\.54±0\.204\.54\_\{\\pm 0\.20\}9\.6LoRA\-r16 replay517\.43M95\.40±1\.6595\.40\_\{\\pm 1\.65\}0\.410±0\.0080\.410\_\{\\pm 0\.008\}4\.45±0\.204\.45\_\{\\pm 0\.20\}9\.0FProj\-LR†17\.43M94\.60±2\.2994\.60\_\{\\pm 2\.29\}0\.413±0\.0100\.413\_\{\\pm 0\.010\}4\.38±0\.18\\mathbf\{4\.38\_\{\\pm 0\.18\}\}10\.8Fora\-LR†17\.76M94\.48±2\.2194\.48\_\{\\pm 2\.21\}0\.413±0\.0090\.413\_\{\\pm 0\.009\}4\.38±0\.19\\mathbf\{4\.38\_\{\\pm 0\.19\}\}11\.0OPLoRA\-r1617\.43M94\.44±2\.5594\.44\_\{\\pm 2\.55\}0\.414±0\.0100\.414\_\{\\pm 0\.010\}4\.54±0\.184\.54\_\{\\pm 0\.18\}10\.8CorDA\-k16\-r1617\.43M94\.16±1\.4094\.16\_\{\\pm 1\.40\}0\.414±0\.0090\.414\_\{\\pm 0\.009\}4\.38±0\.19\\mathbf\{4\.38\_\{\\pm 0\.19\}\}8\.8Larger parameter scalesLoRA\-r6469\.73M97\.64±0\.9597\.64\_\{\\pm 0\.95\}0\.410±0\.0110\.410\_\{\\pm 0\.011\}4\.82±0\.214\.82\_\{\\pm 0\.21\}5\.6DoRA\-r16†34\.09M96\.90±0\.4296\.90\_\{\\pm 0\.42\}0\.418±0\.0010\.418\_\{\\pm 0\.001\}4\.38±0\.064\.38\_\{\\pm 0\.06\}10\.0Key observations from the full PEFT results:

1. 1\.Replay dominates COGS EMat matched parameters \(top\-3 methods are replay\-based\), but at the cost of weaker PPL \(4\.454\.45–4\.554\.55\)\.
2. 2\.Function\-space methods achieve best PPL\(4\.384\.38, matching the translation baseline of4\.354\.35\), indicating superior capability preservation\.
3. 3\.CorDA and our method converge on similar PPL\(4\.384\.38\), supporting the shared insight that activation\-derived subspaces protect capabilities\.
4. 4\.Increasing rank does not monotonically improve performance: LoRA\-r64 \(69\.7369\.73M\) achieves97\.64%97\.64\\%COGS but with much worse PPL \(4\.824\.82\) than 17M\-parameter methods\.

![Refer to caption](https://arxiv.org/html/2606.31092v1/x4.png)Figure 4:Projection source across three settings\(appendix visualization of Table[5](https://arxiv.org/html/2606.31092#S4.T5)\)\. \(a\) Translation PPL:P𝐐P\_\{\\\!\\mathbf\{Q\}\}\(function\-space\) is consistently closer to baseline thanP𝐕P\_\{\\\!\\mathbf\{V\}\}\(weight\-space\)\. \(b\) New\-task or preserved metric:P𝐐P\_\{\\\!\\mathbf\{Q\}\}is comparable or better\. The systematic advantage ofP𝐐P\_\{\\\!\\mathbf\{Q\}\}confirms that the protected subspace’s*source*is what matters\.
## Appendix DSpectral\-Only Baseline

To clarify the role of the spectral calibration channel𝜹\\bm\{\\delta\}, we report an experiment where*only*the diagonal calibration branch is trained:Δ​𝐖=𝐔2​𝐃δ​𝐕2⊤\\Delta\\\!\\mathbf\{W\}=\\mathbf\{U\}\_\{2\}\\mathbf\{D\}\_\{\\delta\}\\mathbf\{V\}\_\{2\}^\{\\top\}\(no𝐌\\mathbf\{M\}or𝐁​𝒜\\mathbf\{B\}\\mathcal\{A\}branch\)\. This isolates what the channel can do on its own—establishing it as a genuine, low\-degree\-of\-freedom plasticity channel rather than a no\-op—and shows why it is*paired with*, not substituted for, the high\-capacity protected branch\. It is compared against no fine\-tuning and standard LoRA\-r16 on the SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2606.31092#bib.bib16)\)mathematical reasoning task, with GSM8K retention as the preserved capability\.

Table 13:Spectral\-only baseline on SVAMP fine\-tuning with GSM8K retention\. Model: Qwen3\-1\.7B \(GSM8K LoRA merged\)\.MethodSVAMP↑\\uparrowGSM8K Retention↑\\uparrowEpochVal LossNo fine\-tuning70\.749\.7—2\.87Diagonal\-only72\.7±4\.772\.7\_\{\\pm 4\.7\}23\.7±0\.623\.7\_\{\\pm 0\.6\}250\.057LoRA\-r1670\.7±2\.470\.7\_\{\\pm 2\.4\}20\.4±2\.420\.4\_\{\\pm 2\.4\}80\.044The diagonal\-only configuration \(Table[13](https://arxiv.org/html/2606.31092#A4.T13)\) achieves modest new\-task learning \(SVAMP72\.7%72\.7\\%vs\. baseline70\.7%70\.7\\%\) and some GSM8K retention \(23\.7%23\.7\\%\)—better than LoRA\-r16 \(20\.4%20\.4\\%\), but far below the baseline \(49\.7%49\.7\\%\)\. This shows that:

1. 1\.The spectral calibration channel alone carries*non\-zero*, real plasticity—enough for minor adaptation, but not enough to learn a full task by itself\.
2. 2\.It is not designed to preserve a capability on its own: without the projectors, GSM8K retention still drops from49\.7%49\.7\\%to23\.7%23\.7\\%\.
3. 3\.In the full method the projectors \(P𝐔P\_\{\\\!\\mathbf\{U\}\},P𝐐P\_\{\\\!\\mathbf\{Q\}\}\) supply the structural protection while𝜹\\bm\{\\delta\}contributes the bounded, controlled plasticity that Proposition[3](https://arxiv.org/html/2606.31092#Thmproposition3)reserves for the protected directions—two roles that compose rather than compete\.

## Appendix ESensitivity to Subspace Dimensions, and Planned Extensions

### E\.1Sensitivity to the subspace dimensionsrrandkfk\_\{f\}

Forahas two structural hyperparameters: the SVD frozen rankrrinP𝐔P\_\{\\\!\\mathbf\{U\}\}\(default100100\) and the function\-subspace sizekfk\_\{f\}inP𝐐P\_\{\\\!\\mathbf\{Q\}\}\(default1616\)\. We sweep each independently on the Math→\\toTrans setting \(preserving GSM8K reasoning while learning translation\), holding the other at its default and running55seeds per configuration\. The default\(r,kf\)=\(100,16\)\(r,k\_\{f\}\)=\(100,16\)row is theForaresult already reported in Table[4](https://arxiv.org/html/2606.31092#S4.T4)\. Table[14](https://arxiv.org/html/2606.31092#A5.T14)collects both sweeps\.

Table 14:Sensitivity to the subspace dimensionson Math→\\toTrans \(preserve GSM8K, learn translation\); 5 seeds per row\. Left: vary the SVD frozen rankrratkf=16k\_\{f\}=16\. Right: vary the function\-subspace sizekfk\_\{f\}atr=100r=100\. The\(r,kf\)=\(100,16\)\(r,k\_\{f\}\)=\(100,16\)row is the defaultForaconfiguration \(Table[4](https://arxiv.org/html/2606.31092#S4.T4)\)\. Across both sweeps GSM8K retention stays within a0\.40\.4\-point band and translation chrF/PPL are essentially unchanged\.Frozen rankrr\(kf=16k\_\{f\}=16\)Function sizekfk\_\{f\}\(r=100r=100\)rrGSM8KchrFPPLkfk\_\{f\}GSM8KchrFPPL202048\.61±0\.6448\.61\_\{\\pm 0\.64\}0\.3900\.3904\.804\.804448\.63±0\.2548\.63\_\{\\pm 0\.25\}0\.3890\.3894\.804\.80505048\.82±0\.3348\.82\_\{\\pm 0\.33\}0\.3900\.3904\.804\.808849\.01±0\.3549\.01\_\{\\pm 0\.35\}0\.3890\.3894\.804\.80100100\(def\.\)48\.84±0\.4248\.84\_\{\\pm 0\.42\}0\.3900\.3904\.804\.801616\(def\.\)48\.84±0\.4248\.84\_\{\\pm 0\.42\}0\.3900\.3904\.804\.8020020048\.79±0\.3448\.79\_\{\\pm 0\.34\}0\.3900\.3904\.804\.80323248\.66±0\.3748\.66\_\{\\pm 0\.37\}0\.3900\.3904\.804\.8040040048\.95±0\.3948\.95\_\{\\pm 0\.39\}0\.3900\.3904\.814\.81646448\.95±0\.2548\.95\_\{\\pm 0\.25\}0\.3900\.3904\.804\.80Both hyperparameters are inconsequential within the ranges tested\. Varyingrrover a twentyfold range \(20→40020\\\!\\to\\\!400\) moves GSM8K retention by0\.340\.34points \(48\.6148\.61–48\.9548\.95\) and leaves translation chrF and PPL unchanged; varyingkfk\_\{f\}over a sixteenfold range \(4→644\\\!\\to\\\!64\) moves it by0\.380\.38points \(48\.6348\.63–49\.0149\.01\)\. Every configuration lies within one standard deviation of the default, so the differences are not statistically meaningful and the default\(100,16\)\(100,16\)—fixed a priori—is not a tuned operating point\. The behavior is consistent with the structural account of the method: once𝐐\\mathbf\{Q\}spans the dominant activation directions of the preserved capability, enlarging it adds near\-degenerate directions that the high\-capacity branch was already largely avoiding, and enlargingrrsimilarly saturates the output\-side prior\.

We emphasize that this robustness concerns*how many*directions are protected, not*which subspace*they are drawn from\. The latter is the load\-bearing design choice, and the projection\-source ablation of Table[5](https://arxiv.org/html/2606.31092#S4.T5)shows it is decisive: replacing the capability\-derivedP𝐐P\_\{\\\!\\mathbf\{Q\}\}by the weight\-derivedP𝐕P\_\{\\\!\\mathbf\{V\}\}degrades the preserved metric in every setting, even though both project away a subspace of comparable rank\. Insensitivity tokfk\_\{f\}therefore makesForaeasy to deploy without per\-task tuning; it does not weaken the central claim that the projection source is what matters\.

### E\.2Planned Extensions for Future Versions

We outline additional analyses planned for future versions of this manuscript\.

#### RandomQQbaseline\.

To further verify that the source of𝐐\\mathbf\{Q\}matters, we plan to compare the learned𝐐\\mathbf\{Q\}\(from preserved\-capability activations\) against a random orthogonal projector of the same rank\. If random𝐐\\mathbf\{Q\}fails to protect capabilities, it strengthens the claim that activation\-derived subspaces carry capability\-specific information\.

#### Calibration sample size sensitivity\.

We plan to evaluate preservation quality as a function of calibration sample countN∈\{50,100,250,500,1000\}N\\in\\\{50,100,250,500,1000\\\}, measuring whether𝐐\\mathbf\{Q\}stabilizes with modest data and whether domain shift in calibration inputs affects protection quality\.

#### Multi\-capability union projection\.

For simultaneous protection of multiple capabilities \(e\.g\., translation\+\+math\+\+code\), we plan to construct𝐐union=\[𝐐1​𝐐2​𝐐3\]\\mathbf\{Q\}\_\{\\text\{union\}\}=\[\\mathbf\{Q\}\_\{1\}\\;\\mathbf\{Q\}\_\{2\}\\;\\mathbf\{Q\}\_\{3\}\]and evaluate the resulting trade\-off between protection breadth and adaptive capacity\.

#### Full\-rank replay comparison\.

A direct comparison of our full\-rank method against full fine\-tuning with data replay \(mixing1%1\\%,5%5\\%,10%10\\%of preserved\-capability data\) would quantify the complementarity \(or redundancy\) of structural versus data\-level protection\.

#### Larger\-model validation\.

Scaling experiments to Qwen3\-4B and Qwen3\-8B, and to models from other families \(e\.g\., LLaMA\), would test whether the relative advantage of function\-space over weight\-space protection grows, shrinks, or remains constant with model scale\.