Preference Tuning as Spectral Update Reorganization

# Preference Tuning as Spectral Update Reorganization
Source: [https://arxiv.org/html/2607.20438](https://arxiv.org/html/2607.20438)
Haibo JinUniversity of Illinois at Urbana\-ChampaignLiying KangHong Kong Polytechnic UniversityHaohan WangUniversity of Illinois at Urbana\-Champaign

###### Abstract

Preference\-based post\-training is usually understood through endpoint behavior, yet the learned update that produces this behavior remains largely opaque\. We study RLHF and related preference optimization through the spectral structure of their induced parameter updates\. By decomposing effective LoRA updates and reloading their spectral components as plug\-in modules, we turn preference\-induced updates into objects that can be isolated, recomposed, and directly intervened on\. Across model families, optimization algorithms, and supervision regimes, these updates consistently develop a spectral head–tail organization\. A compact head emerges early and carries the dominant endpoint shift, while a heterogeneous residual tail remains\. The split is functional rather than merely descriptive\. Plug\-in intervention shows that the head accounts for the visible behavioral departure from the base model, while the tail is weak in isolation\. Cross\-run recomposition further shows that mixed adapters follow the source of the head, indicating that the head carries run\-level solver bias\. This endpoint dominance does not imply learning sufficiency\. Head\-only learning is non\-vacuous but fails to recover the full solution, especially on out\-of\-distribution behavior\. Tail\-only learning yields little visible gain, yet the full solution is not recovered without the tail\. These findings recast preference post\-training as structured update reorganization rather than a monolithic behavioral correction, and suggest that alignment gain and coverage loss are tied to how the learned update itself is organized\.

## 1Introduction

Preference\-based post\-training has become a standard mechanism for shaping the behavior of modern large language models \(LLMs\)\(Christianoet al\.,[2017](https://arxiv.org/html/2607.20438#bib.bib10); Ouyanget al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib8); Schulmanet al\.,[2017](https://arxiv.org/html/2607.20438#bib.bib11); Rafailovet al\.,[2023](https://arxiv.org/html/2607.20438#bib.bib2)\)\. It is typically judged by endpoint behavior: whether the tuned model follows instructions more reliably, produces preferred responses, or improves on reasoning\-oriented tasks\(Jaechet al\.,[2024](https://arxiv.org/html/2607.20438#bib.bib13); Guoet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib14); Teamet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib17)\)\. Yet this endpoint view leaves the learned update itself largely opaque\. Post\-training does not merely change model outputs; it induces a parameter update whose internal organization may determine which behaviors are strengthened, preserved, or lost\. This distinction matters because preference tuning often improves broadly shared alignment while weakening more conditional, specialized, or long\-tail capabilities\(Liuet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib41)\)\. Explaining this tension, therefore, requires looking beyond whether the final model improves, and asking how the induced update is structured inside the model\.

Prior work frames this issue as a debate about endpoint behavior: whether preference tuning merely reweights capabilities already present in the base model, or whether it creates new reasoning and decision patterns\(Mukherjeeet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib18); Yueet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib19); Zhanget al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib25); Chenet al\.,[2025a](https://arxiv.org/html/2607.20438#bib.bib23); Wanget al\.,[2025b](https://arxiv.org/html/2607.20438#bib.bib26)\)\. This framing misses a more basic structural question\. A preference\-tuned model differs from its initialization by a learned update, yet that update is usually treated as a monolithic consequence of optimization\. Consequently, it remains unclear whether alignment gains, solver biases, and coverage losses arise from the same update directions or from separable components with different roles\.

We therefore make the learned update the unit of analysis\. For each preference\-tuned checkpoint, we treat its difference from the pre\-tuning initialization as an additive effective update, instantiated in our main experiments by LoRA adapters\(Huet al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib16)\)\. We decompose each module\-level update with singular value decomposition \(SVD\)\(Wallet al\.,[2003](https://arxiv.org/html/2607.20438#bib.bib9)\), yielding a leading*spectral head*and a complementary*residual tail*\. These components are not only descriptive: they can be converted back into plug\-in adapters, enabling controlled deletion, isolation, cross\-run recomposition, and training\-time restriction\. This turns preference\-induced updates from opaque optimization endpoints into manipulable experimental objects\.

This update\-centered view leads to a different picture of preference post\-training\. Rather than acting as a single monolithic behavioral correction, preference tuning reorganizes the learned update into structurally distinct components\. A dominant spectral head consolidates the main behavioral shift, while a heterogeneous residual tail remains weak in isolation but is tied to the recovery of broader and more conditional behavior\. Under this view, the tension between alignment gain and coverage loss is not merely an external side effect of post\-training, but a property of how the update itself is organized\.

We substantiate this view through three findings\.

- •Preference\-induced updates develop a stable spectral head–tail structure\.Across model families, optimization algorithms, and supervision regimes, the update does not remain diffuse or collapse into a purely low\-rank object\. A coherent spectral head emerges early, while a heterogeneous residual tail remains throughout training\.
- •The spectral split is functional, not merely descriptive\.Source\-controlled plug\-in intervention shows that the spectral head and residual tail induce different endpoint behaviors when isolated\. The residual tail remains weak in isolation, whereas the spectral head carries the dominant but run\-dependent behavioral effect\.
- •Functional separation has mechanistic signatures across recomposition, training, and supervision\.Cross\-run recomposition shows that mixed adapters follow the source of the spectral head, indicating that the head carries run\-level solver bias\. Training\-time projection shows that head\-only learning is non\-vacuous but fails to recover the full solution, especially on out\-of\-distribution behavior\. Supervision corruption further suggests that coherent prompt–preference alignment helps determine whether coherent update structure can form\.

Together, these findings frame preference post\-training as structured update reorganization rather than a single behavioral shift\. The spectral head explains the dominant endpoint behavior, but the residual tail and the supervision conditions under which the structure forms are essential to understanding breadth, learnability, and coverage loss\.

## 2Related Work

Preference post\-training and behavioral accounts\.Preference\-based post\-training has become a central mechanism for shaping the behavior of large language models\. Early work studied reward learning from human preference comparisons\(Christianoet al\.,[2017](https://arxiv.org/html/2607.20438#bib.bib10)\)\. RLHF later became a standard recipe for instruction alignment\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib8); Baiet al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib12)\)\. Direct preference optimization simplified this pipeline by optimizing directly from preference pairs without an explicit reward\-model reinforcement\-learning loop\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.20438#bib.bib2)\)\. Recent reasoning\-oriented post\-training extends reward\-based optimization to mathematical, coding, and complex reasoning tasks\. Representative examples include GRPO\-style training in DeepSeekMath\(Shaoet al\.,[2024](https://arxiv.org/html/2607.20438#bib.bib7)\)and later reasoning models trained with large\-scale reinforcement learning\(Guoet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib14)\)\. These works define the class of post\-training processes we study, but they are primarily evaluated through endpoint behavior, such as instruction following, preferred response generation, and reasoning accuracy\. We do not propose a new preference objective\. We analyze the parameter update induced by such objectives and ask how this update is organized inside the model\.

What RLHF changes in language models\.Recent studies have begun to question what endpoint improvement reflects\. One line of work asks whether RLVR truly expands the base model’s reasoning capacity or mainly improves the sampling efficiency of high\-reward paths already present in the base distribution\(Yueet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib19)\)\. Other work analyzes whether the reasoning boundary shrinks, expands, or changes in a stage\-dependent manner during training\(Yaoet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib28)\)\. Capability\-boundary collapse in RLVR has also motivated hybrid\-policy and exploration\-based methods that aim to counteract such narrowing\(Donget al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib33)\)\.

Another line localizes RL\-induced changes at finer granularity\. Some studies analyze how RL reshapes reasoning patterns and token\-level dynamics\(Chenet al\.,[2025b](https://arxiv.org/html/2607.20438#bib.bib24)\)\. Others show that high\-entropy or high\-significance decision tokens play a disproportionate role in RL\-based reasoning\(Wanget al\.,[2025a](https://arxiv.org/html/2607.20438#bib.bib30); Liuet al\.,[2026](https://arxiv.org/html/2607.20438#bib.bib31)\)\. Trajectory\-level information peaks have also been used to identify key thinking tokens in reasoning traces\(Qianet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib32)\)\. At the parameter level, recent work finds that RL fine\-tuning can modify sparse subnetworks rather than the full model\(Mukherjeeet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib18); Balashov,[2025](https://arxiv.org/html/2607.20438#bib.bib22)\), and that random sparse subnetworks may suffice for effective RLVR\(Adewuyiet al\.,[2026](https://arxiv.org/html/2607.20438#bib.bib20)\)\. Work on RLVR update directions further argues that update magnitude or sparsity alone is insufficient for understanding RL\-induced change\(Huanget al\.,[2026](https://arxiv.org/html/2607.20438#bib.bib21)\)\. Other studies examine base\-model compatibility, mid\-training conditions, and scaling behavior in RL post\-training\(Wanget al\.,[2025b](https://arxiv.org/html/2607.20438#bib.bib26); Tanet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib29)\)\. Post\-training trade\-offs such as forgetting and retention have also been studied, with evidence that online RL can preserve prior capabilities differently from supervised fine\-tuning\(Chenet al\.,[2025a](https://arxiv.org/html/2607.20438#bib.bib23); Shenfeldet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib27)\)\.

These studies suggest that RL effects are not uniform endpoint\-score improvements\. They appear across reasoning trajectories, token\-level decision points, parameter footprints, update directions, and capability coverage\. Most analyses, however, still operate at the level of behavioral trajectories, token dynamics, sparse subnetworks, or coarse update effects\. We instead study whether a preference\-induced update contains separable spectral components with distinct endpoint and learning roles\.

Parameter\-space views of model behavior\.A complementary line of work treats the difference between a fine\-tuned model and its pretrained initialization as a manipulable object in parameter space\. Task arithmetic defines such differences as task vectors and composes them to steer model behavior\(Ilharcoet al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib40)\)\. Model soups average fine\-tuned weights to improve accuracy without increasing inference cost\(Wortsmanet al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib39)\)\. Later merging methods study how to reduce interference when combining task\-specific models\(Yadavet al\.,[2023](https://arxiv.org/html/2607.20438#bib.bib38)\)\. Recent work further develops task\-vector bases for efficient model editing\(Zenget al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib36)\), layer\-aware task arithmetic for disentangling task\-specific knowledge from instruction\-following behavior\(Chenet al\.,[2025c](https://arxiv.org/html/2607.20438#bib.bib35)\), and task singular vectors for reducing task interference in model merging\(Gargiuloet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib37)\)\.

We share the view that fine\-tuning differences are meaningful objects for analysis and intervention, but prior work typically manipulates the update as a whole for model editing or merging\. Closest in spirit, task singular vector methods also use SVD to analyze task matrices\(Gargiuloet al\.,[2025](https://arxiv.org/html/2607.20438#bib.bib37)\), but their goal is to reduce interference in model merging\. Our spectral components are used instead as diagnostic and interventional objects for functional separation within a preference\-induced update, allowing us to distinguish endpoint salience from learning sufficiency\.

## 3Preliminaries

Endpoint behavior collapses the effect of preference tuning into a final model response\. This view is useful for evaluation, but it cannot reveal whether the same learned directions control the visible behavioral shift, support optimization, or preserve more conditional capabilities\. We therefore use the additive update from the pre\-tuning model to the tuned checkpoint as the basic object of analysis\. Once written as a collection of module\-level patches, this update can be decomposed, removed, substituted, and recomposed without changing the base model\.

### 3\.1Effective Updates

Module\-level linear projections provide a natural unit for this analysis, since the learned change to each projection can be expressed as an additive patch on the corresponding base weight\. LetWℓ,j\(0\)W\_\{\\ell,j\}^\{\(0\)\}denote the pre\-tuning weight of modulejjin Transformer blockℓ\\ell, and letWℓ,j\(t\)W\_\{\\ell,j\}^\{\(t\)\}denote the corresponding weight at checkpointtt\. The effective updateΔ​Wℓ,j\(t\)\\Delta W\_\{\\ell,j\}^\{\(t\)\}is defined by

Wℓ,j\(t\)=Wℓ,j\(0\)\+Δ​Wℓ,j\(t\)\.W\_\{\\ell,j\}^\{\(t\)\}=W\_\{\\ell,j\}^\{\(0\)\}\+\\Delta W\_\{\\ell,j\}^\{\(t\)\}\.For dense finetuning,Δ​Wℓ,j\(t\)=Wℓ,j\(t\)−Wℓ,j\(0\)\\Delta W\_\{\\ell,j\}^\{\(t\)\}=W\_\{\\ell,j\}^\{\(t\)\}\-W\_\{\\ell,j\}^\{\(0\)\}\. In our main LoRA setting\(Huet al\.,[2022](https://arxiv.org/html/2607.20438#bib.bib16)\), the backbone is frozen and the effective update is given by the scaled adapter product

Δ​Wℓ,j\(t\)=sℓ,j\(t\)​Bℓ,j\(t\)​Aℓ,j\(t\),\\Delta W\_\{\\ell,j\}^\{\(t\)\}=s\_\{\\ell,j\}^\{\(t\)\}B\_\{\\ell,j\}^\{\(t\)\}A\_\{\\ell,j\}^\{\(t\)\},whereAℓ,j\(t\)A\_\{\\ell,j\}^\{\(t\)\}andBℓ,j\(t\)B\_\{\\ell,j\}^\{\(t\)\}are the LoRA factors andsℓ,j\(t\)s\_\{\\ell,j\}^\{\(t\)\}absorbs implementation\-specific scaling\.

The checkpoint\-level update is the collection of these module\-level patches over the targeted attention and MLP projections\. Full module lists, LoRA dimensions, and scaling conventions are given in Appendix[A](https://arxiv.org/html/2607.20438#A1)\. Because the update is additive, component deletion, isolation, cross\-run substitution, and recomposition can all be implemented as operations onΔ​W\\Delta W\. When no ambiguity arises, we omit the checkpoint superscripttt\.

### 3\.2Spectral Partition of Updates

A useful partition must separate dominant update directions while preserving exact recomposability\. For each module\-level updateΔ​Wℓ,j\\Delta W\_\{\\ell,j\}, we compute

Δ​Wℓ,j=Uℓ,j​Σℓ,j​Vℓ,j⊤,\\Delta W\_\{\\ell,j\}=U\_\{\\ell,j\}\\Sigma\_\{\\ell,j\}V\_\{\\ell,j\}^\{\\top\},with singular values sorted in decreasing order\.

Given a split rankrr, the spectral head is the leading truncated component

Δ​Wℓ,jhead​\(r\)=Uℓ,j\(:,1:r\)​Σℓ,j\(1:r,1:r\)​\(Vℓ,j\(:,1:r\)\)⊤,\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)=U\_\{\\ell,j\}^\{\(:,1:r\)\}\\Sigma\_\{\\ell,j\}^\{\(1:r,1:r\)\}\\bigl\(V\_\{\\ell,j\}^\{\(:,1:r\)\}\\bigr\)^\{\\top\},and the residual tail is its complement

Δ​Wℓ,jtail​\(r\)=Δ​Wℓ,j−Δ​Wℓ,jhead​\(r\)\.\\Delta W\_\{\\ell,j\}^\{\\mathrm\{tail\}\}\(r\)=\\Delta W\_\{\\ell,j\}\-\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)\.Thus,

Δ​Wℓ,j=Δ​Wℓ,jhead​\(r\)\+Δ​Wℓ,jtail​\(r\),\\Delta W\_\{\\ell,j\}=\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)\+\\Delta W\_\{\\ell,j\}^\{\\mathrm\{tail\}\}\(r\),which enables exact deletion, isolation, and recomposition at the update level\.

The names head and tail are purely spectral at definition time\. We do not assume that they correspond to fixed behavioral categories\. Their roles are tested empirically through plug\-in intervention, cross\-run recomposition, and training\-time projection\.

### 3\.3Tracking Spectral Concentration

To study whether preference tuning forms a concentrated update structure over training, we track the singular\-value spectra of eachΔ​Wℓ,j\(t\)\\Delta W\_\{\\ell,j\}^\{\(t\)\}across checkpoints and summarize concentration with entropy effective rank\. For singular values\{σi\}\\\{\\sigma\_\{i\}\\\}, define

pi=σi∑kσk,reff=exp⁡\(−∑ipi​log⁡\(pi\+ε\)\),p\_\{i\}=\\frac\{\\sigma\_\{i\}\}\{\\sum\_\{k\}\\sigma\_\{k\}\},\\qquad r\_\{\\mathrm\{eff\}\}=\\exp\\left\(\-\\sum\_\{i\}p\_\{i\}\\log\(p\_\{i\}\+\\varepsilon\)\\right\),whereε\\varepsilonis a small numerical constant\.

Smaller effective rank indicates that update mass is concentrated in fewer dominant directions, while larger effective rank indicates a flatter and more dispersed spectrum\. Unless otherwise stated, checkpoint\-level summaries report the medianreffr\_\{\\mathrm\{eff\}\}over targeted modules\. Numerical safeguards and additional rank statistics are provided in Appendix[A](https://arxiv.org/html/2607.20438#A1)\.

## 4Spectral Separation

A head–tail partition should not be assumed to be meaningful from the decomposition alone\. The learned update could remain broadly diffuse, with no stable leading component, or collapse into a few dominant directions, leaving little residual structure to analyze\. Preference\-induced updates follow neither pattern\. During training, update mass concentrates into a compact spectral head, while a non\-negligible residual tail remains\.

We first examine a controlled reference trajectory to trace how this structure forms over training, then test whether the same pattern persists across model families and supervision regimes\.

### 4\.1Formation in a Reference Trajectory

We start from a controlled reference trajectory,Qwen2\.5\-0\.5Bunder synthetic preference supervision, to isolate the formation of spectral structure before testing its generality\. The backbone is frozen and preference tuning is performed with LoRA adapters of rankr0=64r\_\{0\}=64\. For each checkpoint, we track the module\-level effective updates defined in Section[3\.1](https://arxiv.org/html/2607.20438#S3.SS1)through two complementary views: singular\-value spectra, which show the shape of each update operator, and entropy effective rank, which summarizes spectral concentration\. Full training details, target modules, and checkpoint schedules are given in Appendix[B](https://arxiv.org/html/2607.20438#A2)\.

![Refer to caption](https://arxiv.org/html/2607.20438v1/x1.png)\(a\)Effective rank
![Refer to caption](https://arxiv.org/html/2607.20438v1/x2.png)\(b\)Singular spectra

Figure 1:Formation of spectral head–tail organization in the reference trajectory\. The effective rank contracts early, partially rebounds, and then stabilizes, while the singular spectra show a growing dominant head and a persistent residual tail\.Figure[1](https://arxiv.org/html/2607.20438#S4.F1)shows that spectral organization forms early but does not continue toward rank collapse\. The effective rank first decreases, indicating that update mass concentrates into fewer leading directions\. It then partially rebounds and stabilizes, showing that preference tuning does not keep compressing the update into an increasingly smaller subspace\.

The singular spectra explain this non\-monotonic trajectory\. Early in training, the spectrum is relatively flat and weakly structured\. As training proceeds, the leading singular values separate from the rest of the spectrum and form a compact head\. At the same time, the remaining singular directions do not disappear\. The residual tail stays visible, accounting for the rebound and stabilization in effective rank\. Preference tuning therefore reorganizes the update into a head–tail spectrum rather than leaving it diffuse or reducing it to a purely low\-rank object\.

![Refer to caption](https://arxiv.org/html/2607.20438v1/x3.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x4.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x5.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x6.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x7.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x8.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x9.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x10.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x11.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x12.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x13.png)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x14.png)
Gray dashed: early checkpoint Red solid: final checkpoint

Figure 2:Spectral generality across model families, optimization algorithms, and supervision regimes\. Across settings, preference tuning sharpens a leading spectral head while retaining a residual tail\.
### 4\.2Generality Across Models and Training Regimes

A single controlled trajectory could still reflect the idiosyncrasy of one model, optimizer, or supervision source\. We therefore compare early and final singular spectra across 12 settings spanning model families, scales, optimization algorithms, and supervision regimes\. The grid in Figure[2](https://arxiv.org/html/2607.20438#S4.F2)includes Qwen and Llama models, DPO and GRPO training, and both synthetic and benchmark\-derived preference data\.

Across all settings, training sharpens a small leading head while retaining a visible residual tail\. The steepness of the final spectrum varies with model and regime, but the qualitative transition is stable: the final update is more organized than the early update, yet it does not collapse into only a few singular directions\.

This recurrence across settings makes the spectral split a stable property of preference\-induced updates rather than a visual artifact of the reference run\. Preference tuning consistently reorganizes the update into a dominant head and a retained tail\.

## 5Functional Separation

A spectral split does not by itself imply behavioral specialization\. The leading directions may simply contain more update energy, while the residual directions may be too weak to matter\. To test whether the head–tail structure has functional content, we convert each component back into a plug\-in adapter and inject it into the same base model\.

### 5\.1Source\-Controlled Plug\-in

Programmatically verifiable tasks let us control training bias and measure solver behavior without ambiguity\. We train two DPO adapters from the same base model under matched optimization settings but different training\-distribution biases\.Run\-Directemphasizes short\-horizon extraction, local transformation, and shallow computation, whereasRun\-Deliberateplaces more mass on tasks requiring explicit decomposition and multi\-step reasoning\.

For each final adapter, we split the effective update into aspectral headand aresidual tailusing the partition in Section[3\.2](https://arxiv.org/html/2607.20438#S3.SS2), with split rankk=16k=16\. Each component is refactorized into LoRA form and loaded back into the same base model\. Since head and tail come from the same checkpoint and exactly reconstruct the full update when summed, the comparison keeps the backbone, training run, and optimization history fixed\. Only the injected spectral component changes\.

This yields four variants on the same base model:base,full,head\-only, andtail\-only\. All variants are evaluated on ID, OOD, and TRAP splits under the same final\-answer metric\. Full task construction, surface\-form variants, decoding settings, and scoring details are given in Appendix[C](https://arxiv.org/html/2607.20438#A3)\.

![Refer to caption](https://arxiv.org/html/2607.20438v1/x15.png)\(a\)Run\-Direct
![Refer to caption](https://arxiv.org/html/2607.20438v1/x16.png)\(b\)Run\-Deliberate

Figure 3:Source\-controlled plug\-in intervention at the final checkpoint with split rankk=16k=16\. Each panel comparesbase,full,head\-only, andtail\-onlyon ID, OOD, and TRAP evaluations\.
### 5\.2The Head Dominates Plug\-in Behavior

Figure[3](https://arxiv.org/html/2607.20438#S5.F3)shows a clear behavioral asymmetry\. Across both runs and evaluation families,tail\-onlyremains close tobase, whereashead\-onlycarries the visible departure from the base model\. This departure is not uniformly beneficial: inRun\-Direct, the head is brittle and fails to recover strong OOD behavior, while inRun\-Deliberate, it preserves a distinct profile, including stronger OOD behavior than bothbaseandfull, while remaining close tofullon TRAP\. The spectral head is therefore not merely the largest\-energy fragment of the adapter; it carries the dominant endpoint effect induced by the training distribution\. The scoring protocol and exact plug\-in values are provided in Appendix[C\.5](https://arxiv.org/html/2607.20438#A3.SS5)\. Plug\-in isolation therefore shows that the spectral split is functional rather than merely descriptive\.

## 6Deep Analysis

Same\-run plug\-in isolation shows that the spectral head and residual tail express different endpoint behaviors\. This asymmetry leaves three questions unresolved\. First, whether the head carries the behavioral identity of a training run, rather than only a larger share of update magnitude\. Second, whether endpoint dominance makes the head sufficient during learning\. Third, whether coherent preference supervision is an upstream condition for stable update formation\. We address these questions through cross\-run recomposition, training\-time projection, and supervision corruption\.

### 6\.1Cross\-run Composition

If the spectral head carries run\-level solver bias, recomposing components across runs should preserve the solver profile of the head source more than that of the tail source\. We test this by exchanging head and tail components between the two controlled runs\.

LetAAdenoteRun\-DirectandBBdenoteRun\-Deliberate\. Starting from the final checkpoints, we split each learned adapter into its spectral head and residual tail, and construct two mixed adapters:

head​\(A\)\+tail​\(B\),head​\(B\)\+tail​\(A\)\.\\textbf\{head\}\(A\)\+\\textbf\{tail\}\(B\),\\qquad\\textbf\{head\}\(B\)\+\\textbf\{tail\}\(A\)\.Recomposition is exact at the effective\-update level via rank concatenation, as shown in Appendix[A\.6](https://arxiv.org/html/2607.20438#A1.SS6)\.

![Refer to caption](https://arxiv.org/html/2607.20438v1/x17.png)\(a\)head​\(B\)\+tail​\(A\)\\textbf\{head\}\(B\)\+\\textbf\{tail\}\(A\)
![Refer to caption](https://arxiv.org/html/2607.20438v1/x18.png)\(b\)head​\(A\)\+tail​\(B\)\\textbf\{head\}\(A\)\+\\textbf\{tail\}\(B\)

Figure 4:Cross\-run composition measured by mean absolute difference \(MAD\)\. Each panel compares one mixed adapter against candidate reference variants\. Smaller values indicate greater behavioral similarity\.Figure[4](https://arxiv.org/html/2607.20438#S6.F4)shows that the mixed adapters are closest to variants derived from their head\-source runs\. The adapterhead​\(B\)\+tail​\(A\)\\textbf\{head\}\(B\)\+\\textbf\{tail\}\(A\)is closest toRun\-Deliberate, whereashead​\(A\)\+tail​\(B\)\\textbf\{head\}\(A\)\+\\textbf\{tail\}\(B\)is closest toRun\-Direct; matching the tail source alone does not determine the dominant behavior\. This strengthens the plug\-in result: the spectral head acts as the principal carrier of run\-level solver bias, while the residual tail contributes less to the dominant similarity pattern\. The recomposition procedure and MAD summaries are provided in Appendix[D\.1](https://arxiv.org/html/2607.20438#A4.SS1)\.

### 6\.2Training\-time Spectral Projection

Endpoint salience and learning sufficiency are different claims\. A component can dominate the behavior of a final plug\-in adapter without being sufficient to recover the full solution during optimization\. To separate these notions, we intervene during training by projecting each update step into one of three conditions:full,head\-only, ortail\-only\.

After each optimizer step, every module\-level LoRA update is projected into the selected spectral subspace\.Fullapplies no projection\.Head\-onlyretains only the top\-kksingular directions, whereastail\-onlyremoves them and keeps the residual component\. All variants use the same base model, data, optimizer, and evaluation protocol\. Implementation details are given in Appendix[D\.2](https://arxiv.org/html/2607.20438#A4.SS2)\.

![Refer to caption](https://arxiv.org/html/2607.20438v1/x19.png)\(a\)Run\-Direct
![Refer to caption](https://arxiv.org/html/2607.20438v1/x20.png)\(b\)Run\-Deliberate

Figure 5:Training\-time spectral projection\. For each evaluation family, gray and black squares markbaseandfull, and the vertical segment indicates the full\-training span\. Red and blue markers denotehead\-onlyandtail\-onlytraining\.Figure[5](https://arxiv.org/html/2607.20438#S6.F5)shows that the projected training regimes do not recover the full solution\. In both runs,tail\-onlyremains close tobase, whilehead\-onlydeparts frombaseon selected ID and TRAP evaluations but remains far fromfull, especially on OOD behavior\. The spectral head is therefore learning\-relevant but not learning\-sufficient\. The residual tail is weak alone, yet the full solution is not recovered without it, suggesting a support role in preserving breadth and completing learning\. The projection procedure and family\-level summaries are provided in Appendix[D\.2](https://arxiv.org/html/2607.20438#A4.SS2)\.

### 6\.3Supervision Corruption

A coherent spectral structure may also depend on the coherence of the supervision that drives the update\. We test this upstream condition by corrupting prompt–preference alignment while keeping the prompts fixed\. Specifically, we corrupt a fractionρ∈\{0\.25,0\.50,1\.00\}\\rho\\in\\\{0\.25,0\.50,1\.00\\\}of the training set withshuffle\_pairs, replacing each corrupted sample’s preference pair with that of another example\. Details of the corruption procedure are given in Appendix[D\.3](https://arxiv.org/html/2607.20438#A4.SS3)\.

![Refer to caption](https://arxiv.org/html/2607.20438v1/x21.png)\(a\)Run\-Direct
![Refer to caption](https://arxiv.org/html/2607.20438v1/x22.png)\(b\)Run\-Deliberate

Figure 6:Supervision corruption undershuffle\_pairs\. Each panel reports family\-level mean accuracy across corruption ratiosρ\\rho, whereρ=0\\rho=0denotes uncorrupted training\. Dashed lines mark base\-model performance\.Figure[6](https://arxiv.org/html/2607.20438#S6.F6)shows that full corruption sharply degrades both runs, especially on ID and OOD behavior\. Stable preference learning thus depends on preserving prompt–preference consistency\. Partial corruption is not a simple linear noise response\. InRun\-Direct, moderate corruption already causes substantial degradation, withρ=0\.50\\rho=0\.50approaching collapse across evaluations\. InRun\-Deliberate, intermediate corruption levels retain substantial ID and TRAP gains overbase, even as OOD transfer weakens\.

These results suggest that prompt–preference consistency is not merely a label\-quality detail\. It is a plausible upstream condition for coherent functional separation\. The evidence, however, is behavioral rather than spectrally closed\. It supports the supervision\-consistency hypothesis, but does not establish how corruption changes head strength, tail thickness, or spectral trajectories\. The corruption procedure and family\-level summaries are provided in Appendix[D\.3](https://arxiv.org/html/2607.20438#A4.SS3)\.

## 7Discussion and Limitations

### 7\.1From Spectral Organization to Post\-Training Trade\-offs

Signal coherence shapes update geometry\.The head–tail organization suggests that preference tuning does not distribute update mass uniformly\. A plausible explanation is that globally coherent directions, shared across prompts, minibatches, and preference comparisons, accumulate into the spectral head\. More conditional, weakly reinforced, or locally conflicting directions align less globally and remain dispersed in the residual tail\. The supervision\-corruption results are consistent with this view: breaking prompt–preference alignment injects incompatible pressures and weakens stable preference learning\. We do not claim a closed spectral mechanism, but the evidence supports a link between coherent supervision and coherent update geometry\.

Endpoint salience is not learning sufficiency\.A component that controls final plug\-in behavior need not be sufficient for learning\. The spectral head dominates endpoint behavior and cross\-run recomposition, yet head\-only training fails to recover the full solution, especially on OOD behavior\. The residual tail is weak alone, but the full solution is not recovered without it\. This separates behavioral payload from optimization support: some directions matter less by visibly determining the endpoint, and more by helping preserve breadth and complete learning\.

Alignment gain and coverage loss may share a structural source\.Preference tuning consolidates coherent, reward\-consistent directions into the spectral head\. The same process can underrepresent behavior that is conditional, long\-tail, or inconsistently reinforced\. From this perspective, alignment gain and coverage loss are not only consequences of reward design or data distribution\. They can also arise from how the learned update organizes behavioral change\.

### 7\.2Limitations

Update\-level analysis\.Our analysis identifies structure in learned updates, not the circuits that implement the resulting behavior\. It shows that spectral components have different behavioral roles, but does not locate the specific attention heads, MLP features, or computational pathways involved\.

Incomplete supervision\-to\-spectrum link\.The corruption experiments show that prompt–preference consistency affects stable preference learning, but they do not directly track how corruption changes head strength, tail thickness, subspace stability, or effective\-rank trajectories\.

Controlled intervention setting\.Our interventions rely on LoRA\-style updates with a frozen backbone, where effective updates are exactly decomposable and can be refactorized into plug\-in adapters\. This enables controlled causal tests, but leaves open how fully the same structure transfers to full\-parameter finetuning, larger RLHF pipelines, proprietary systems, and more open\-ended instruction\-following settings\.

## 8Conclusion

We studied preference post\-training as structured update reorganization\. Preference\-induced updates consistently form a spectral head–tail organization, where a compact head carries dominant endpoint behavior and run\-level solver bias, while a retained residual tail is weak alone but appears necessary for learning completeness and behavioral breadth\. This perspective reframes alignment gain and coverage loss as consequences of how the update itself is organized\. Preference tuning consolidates coherent behavioral directions, but can leave conditional or weakly reinforced behavior fragile\. Future post\-training should therefore monitor update geometry alongside endpoint rewards\. Spectrum\-aware diagnostics may help improve alignment without unnecessarily narrowing behavioral coverage\.

## References

- I\. Adewuyi, S\. Okibe, and V\. Ivanov \(2026\)The multiple ticket hypothesis: random sparse subnetworks suffice for rlvr\.arXiv preprint arXiv:2602\.01599\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p1.1)\.
- A\. Balashov \(2025\)Reinforcement learning fine\-tunes a sparse subnetwork in large language models\.arXiv preprint arXiv:2507\.17107\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- H\. Chen, N\. Razin, K\. Narasimhan, and D\. Chen \(2025a\)Retaining by doing: the role of on\-policy data in mitigating forgetting\.arXiv preprint arXiv:2510\.18874\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p2.1),[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- X\. Chen, T\. Li, and D\. Zou \(2025b\)Reshaping reasoning in llms: a theoretical analysis of rl training dynamics through pattern selection\.arXiv preprint arXiv:2506\.04695\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- Y\. Chen, Y\. Wei, C\. Hsu, C\. Yu, C\. Huang, Y\. Lin, Y\. Wu, and W\. Lee \(2025c\)Layer\-aware task arithmetic: disentangling task\-specific and instruction\-following knowledge\.arXiv preprint arXiv:2502\.201863\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p5.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1),[§2](https://arxiv.org/html/2607.20438#S2.p1.1)\.
- Y\. Dong, X\. Jiang, Y\. Tao, H\. Liu, K\. Zhang, L\. Mou, R\. Cao, Y\. Ma, J\. Chen, B\. Li,et al\.\(2025\)Rl\-plus: countering capability boundary collapse of llms in reinforcement learning with hybrid\-policy optimization\.arXiv preprint arXiv:2508\.00222\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p2.1)\.
- A\. A\. Gargiulo, D\. Crisostomi, M\. S\. Bucarelli, S\. Scardapane, F\. Silvestri, and E\. Rodola \(2025\)Task singular vectors: reducing task interference in model merging\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 18695–18705\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p5.1),[§2](https://arxiv.org/html/2607.20438#S2.p6.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1),[§2](https://arxiv.org/html/2607.20438#S2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.20438#S3.SS1.p1.7)\.
- K\. Huang, H\. Meng, J\. Wu, J\. Lu, C\. Ma, Z\. Chen, X\. Wang, B\. Ding, J\. Wu, X\. Wang,et al\.\(2026\)On the direction of rlvr updates for llm reasoning: identification and exploitation\.arXiv preprint arXiv:2603\.22117\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p5.1)\.
- A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.\(2024\)Openai o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1)\.
- H\. Liu, L\. Cao, Y\. Ren, M\. Zhou, H\. Dong, X\. Ma, S\. Han, and D\. Zhang \(2026\)Not all tokens matter: towards efficient llm reasoning via token significance in reinforcement learning\.External Links:2506\.08125,[Link](https://arxiv.org/abs/2506.08125)Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- K\. Liu, D\. Yang, Z\. Qian, W\. Yin, Y\. Wang, H\. Li, J\. Liu, P\. Zhai, Y\. Liu, and L\. Zhang \(2025\)Reinforcement learning meets large language models: a survey of advancements and applications across the llm lifecycle\.arXiv preprint arXiv:2509\.16679\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1)\.
- S\. Mukherjee, L\. Yuan, D\. Hakkani\-Tur, and H\. Peng \(2025\)Reinforcement learning finetunes small subnetworks in large language models\.arXiv preprint arXiv:2505\.11711\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p2.1),[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1),[§2](https://arxiv.org/html/2607.20438#S2.p1.1)\.
- C\. Qian, D\. Liu, H\. Wen, Z\. Bai, Y\. Liu, and J\. Shao \(2025\)Demystifying reasoning dynamics with mutual information: thinking tokens are information peaks in llm reasoning\.arXiv preprint arXiv:2506\.02867\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1),[§2](https://arxiv.org/html/2607.20438#S2.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p1.1)\.
- I\. Shenfeld, J\. Pari, and P\. Agrawal \(2025\)Rl’s razor: why online reinforcement learning forgets less\.arXiv preprint arXiv:2509\.04259\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- Z\. Tan, H\. Geng, X\. Yu, M\. Zhang, G\. Wan, Y\. Zhou, Q\. He, X\. Xue, H\. Zhou, Y\. Fan,et al\.\(2025\)Scaling behaviors of llm reinforcement learning post\-training: an empirical study in mathematical reasoning\.arXiv preprint arXiv:2509\.25300\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao,et al\.\(2025\)Kimi k1\. 5: scaling reinforcement learning with llms\.arXiv preprint arXiv:2501\.12599\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p1.1)\.
- M\. E\. Wall, A\. Rechtsteiner, and L\. M\. Rocha \(2003\)Singular value decomposition and principal component analysis\.InA practical approach to microarray data analysis,pp\. 91–109\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p3.1)\.
- S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang,et al\.\(2025a\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.arXiv preprint arXiv:2506\.01939\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- Z\. Wang, F\. Zhou, X\. Li, and P\. Liu \(2025b\)Octothinker: mid\-training incentivizes reinforcement learning scaling\.arXiv preprint arXiv:2506\.20512\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p2.1),[§2](https://arxiv.org/html/2607.20438#S2.p3.1)\.
- M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith,et al\.\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational conference on machine learning,pp\. 23965–23998\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p5.1)\.
- P\. Yadav, D\. Tam, L\. Choshen, C\. A\. Raffel, and M\. Bansal \(2023\)Ties\-merging: resolving interference when merging models\.Advances in neural information processing systems36,pp\. 7093–7115\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p5.1)\.
- X\. Yao, L\. Yu, X\. Hu, F\. Teng, Q\. Cui, J\. Zhou, and Y\. Liu \(2025\)The debate on rlvr reasoning capability boundary: shrinkage, expansion, or both? a two\-stage dynamic view\.arXiv preprint arXiv:2510\.04028\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p2.1)\.
- Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, S\. Song, and G\. Huang \(2025\)Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\.arXiv preprint arXiv:2504\.13837\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p2.1),[§2](https://arxiv.org/html/2607.20438#S2.p2.1)\.
- S\. Zeng, Y\. He, W\. You, Y\. Hao, Y\. H\. Tsai, M\. Yamada, and H\. Zhao \(2025\)Efficient model editing with task vector bases: a theoretical framework and scalable approach\.arXiv e\-prints,pp\. arXiv–2502\.Cited by:[§2](https://arxiv.org/html/2607.20438#S2.p5.1)\.
- R\. Zhang, M\. Kaniselvan, and N\. Mireshghallah \(2025\)Reinforcement learning improves traversal of hierarchical knowledge in llms\.arXiv preprint arXiv:2511\.05933\.Cited by:[§1](https://arxiv.org/html/2607.20438#S1.p2.1)\.

## Appendix AUpdate Construction and Spectral Operations

All spectral operations in the paper are defined on the scaled effective update, not on the LoRA factors separately\. This convention makes the decomposition independent of a particular adapter parameterization and makes component isolation, recomposition, and projection exact at the effective\-update level\. This appendix gives the implementation details for constructing these updates, splitting them spectrally, converting components back into adapter form, and computing spectral statistics\.

### A\.1Targeted Modules and Update Units

We analyze preference\-induced updates at the level of module\-wise linear projections inside Transformer blocks\. For the decoder\-only architectures studied in the paper, the targeted module set is

ℳ=\{q\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj,down\_proj\}\.\\mathcal\{M\}=\\\{\\texttt\{q\\\_proj\},\\texttt\{k\\\_proj\},\\texttt\{v\\\_proj\},\\texttt\{o\\\_proj\},\\texttt\{gate\\\_proj\},\\texttt\{up\\\_proj\},\\texttt\{down\\\_proj\}\\\}\.For a model withLLTransformer blocks, the checkpoint\-level update is represented as

Δ​θ\(t\)=\{Δ​Wℓ,j\(t\)\}ℓ=1,…,L,j∈ℳ\.\\Delta\\theta^\{\(t\)\}=\\\{\\Delta W\_\{\\ell,j\}^\{\(t\)\}\\\}\_\{\\ell=1,\\dots,L,\\;j\\in\\mathcal\{M\}\}\.We also use the block\-restricted collection

Δ​θℓ\(t\)=\{Δ​Wℓ,j\(t\)\}j∈ℳ\\Delta\\theta\_\{\\ell\}^\{\(t\)\}=\\\{\\Delta W\_\{\\ell,j\}^\{\(t\)\}\\\}\_\{j\\in\\mathcal\{M\}\}when summarizing updates within a Transformer block\. Unless otherwise stated, all operations below are applied independently to each module\-level updateΔ​Wℓ,j\(t\)\\Delta W\_\{\\ell,j\}^\{\(t\)\}\.

### A\.2Effective Updates under Dense Tuning and LoRA

LetWℓ,j\(0\)W\_\{\\ell,j\}^\{\(0\)\}denote the pre\-tuning weight of modulejjin blockℓ\\ell, and letWℓ,j\(t\)W\_\{\\ell,j\}^\{\(t\)\}denote the corresponding weight at checkpointtt\. The effective update is the additive patch satisfying

Wℓ,j\(t\)=Wℓ,j\(0\)\+Δ​Wℓ,j\(t\)\.W\_\{\\ell,j\}^\{\(t\)\}=W\_\{\\ell,j\}^\{\(0\)\}\+\\Delta W\_\{\\ell,j\}^\{\(t\)\}\.For dense finetuning, this gives

Δ​Wℓ,j\(t\)=Wℓ,j\(t\)−Wℓ,j\(0\)\.\\Delta W\_\{\\ell,j\}^\{\(t\)\}=W\_\{\\ell,j\}^\{\(t\)\}\-W\_\{\\ell,j\}^\{\(0\)\}\.
In the LoRA setting, the backbone is frozen and the learned change is represented by low\-rank adapter factors\. For a target module with input dimensiondind\_\{\\mathrm\{in\}\}, output dimensiondoutd\_\{\\mathrm\{out\}\}, and LoRA rankr0r\_\{0\}, we write

Aℓ,j\(t\)∈ℝr0×din,Bℓ,j\(t\)∈ℝdout×r0\.A\_\{\\ell,j\}^\{\(t\)\}\\in\\mathbb\{R\}^\{r\_\{0\}\\times d\_\{\\mathrm\{in\}\}\},\\qquad B\_\{\\ell,j\}^\{\(t\)\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{0\}\}\.The scaled effective update is

Δ​Wℓ,j\(t\)=sℓ,j\(t\)​Bℓ,j\(t\)​Aℓ,j\(t\),\\Delta W\_\{\\ell,j\}^\{\(t\)\}=s\_\{\\ell,j\}^\{\(t\)\}B\_\{\\ell,j\}^\{\(t\)\}A\_\{\\ell,j\}^\{\(t\)\},wheresℓ,j\(t\)s\_\{\\ell,j\}^\{\(t\)\}absorbs implementation\-specific scaling conventions, including the usual LoRA factorα/r0\\alpha/r\_\{0\}\. All spectral decompositions, statistics, and interventions are applied to this scaled matrix\. We do not decomposeAℓ,j\(t\)A\_\{\\ell,j\}^\{\(t\)\}orBℓ,j\(t\)B\_\{\\ell,j\}^\{\(t\)\}separately\.

Sincerank⁡\(Δ​Wℓ,j\(t\)\)≤r0\\operatorname\{rank\}\(\\Delta W\_\{\\ell,j\}^\{\(t\)\}\)\\leq r\_\{0\}, each LoRA update has at mostr0r\_\{0\}nonzero singular values\. Unless otherwise stated, the main experiments user0=64r\_\{0\}=64\.

### A\.3Spectral Decomposition

For each module\-level effective update, we compute

Δ​Wℓ,j\(t\)=Uℓ,j\(t\)​Σℓ,j\(t\)​\(Vℓ,j\(t\)\)⊤,\\Delta W\_\{\\ell,j\}^\{\(t\)\}=U\_\{\\ell,j\}^\{\(t\)\}\\Sigma\_\{\\ell,j\}^\{\(t\)\}\\left\(V\_\{\\ell,j\}^\{\(t\)\}\\right\)^\{\\top\},with singular values sorted in non\-increasing order\. The decomposition can be obtained either by materializingΔ​Wℓ,j\(t\)\\Delta W\_\{\\ell,j\}^\{\(t\)\}and applying an economy SVD, or by using an equivalent compact low\-rank computation\. Both procedures define the same spectral components at the level of the scaled effective update\.

All reported spectra and effective\-rank statistics are computed after applying the LoRA scaling factorsℓ,j\(t\)s\_\{\\ell,j\}^\{\(t\)\}\.

### A\.4Spectral Head and Residual Tail

Given a split rankrr, the spectral head is the leading truncated component

Δ​Wℓ,jhead​\(r\)=Uℓ,j\(:,1:r\)​Σℓ,j\(1:r,1:r\)​\(Vℓ,j\(:,1:r\)\)⊤\.\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)=U\_\{\\ell,j\}^\{\(:,1:r\)\}\\Sigma\_\{\\ell,j\}^\{\(1:r,1:r\)\}\\left\(V\_\{\\ell,j\}^\{\(:,1:r\)\}\\right\)^\{\\top\}\.The residual tail is the complementary component

Δ​Wℓ,jtail​\(r\)=Δ​Wℓ,j−Δ​Wℓ,jhead​\(r\)\.\\Delta W\_\{\\ell,j\}^\{\\mathrm\{tail\}\}\(r\)=\\Delta W\_\{\\ell,j\}\-\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)\.Thus,

Δ​Wℓ,j=Δ​Wℓ,jhead​\(r\)\+Δ​Wℓ,jtail​\(r\)\.\\Delta W\_\{\\ell,j\}=\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)\+\\Delta W\_\{\\ell,j\}^\{\\mathrm\{tail\}\}\(r\)\.Unless otherwise stated, the plug\-in, recomposition, and projection experiments use split rankr=16r=16with LoRA rankr0=64r\_\{0\}=64\.

### A\.5Refactorizing Spectral Components into LoRA Form

Intervention requires each spectral component to be loaded as an adapter\. Given any rank\-qqupdate component

Δ​Wc=Uc​Σc​Vc⊤,\\Delta W\_\{c\}=U\_\{c\}\\Sigma\_\{c\}V\_\{c\}^\{\\top\},we construct LoRA\-style factors

Bc=Uc​Σc1/2,Ac=Σc1/2​Vc⊤,B\_\{c\}=U\_\{c\}\\Sigma\_\{c\}^\{1/2\},\\qquad A\_\{c\}=\\Sigma\_\{c\}^\{1/2\}V\_\{c\}^\{\\top\},so that

Bc​Ac=Δ​Wc\.B\_\{c\}A\_\{c\}=\\Delta W\_\{c\}\.The refactorized adapter uses scaling factor11, since the original LoRA scaling has already been absorbed intoΔ​Wc\\Delta W\_\{c\}\.

If an implementation requires a fixed adapter rank, components with rank below the required size are padded with zero columns inBcB\_\{c\}and zero rows inAcA\_\{c\}\. This padding preserves the effective update exactly\.

### A\.6Isolation and Recomposition

The additive update view makes component isolation and recomposition exact at the effective\-update level\.

#### Isolation\.

For a targeted module, isolating a component means loading only that component on top of the frozen base weight:

Wℓ,j=Wℓ,j\(0\)\+Δ​Wℓ,jhead​\(r\)W\_\{\\ell,j\}=W\_\{\\ell,j\}^\{\(0\)\}\+\\Delta W\_\{\\ell,j\}^\{\\mathrm\{head\}\}\(r\)or

Wℓ,j=Wℓ,j\(0\)\+Δ​Wℓ,jtail​\(r\)\.W\_\{\\ell,j\}=W\_\{\\ell,j\}^\{\(0\)\}\+\\Delta W\_\{\\ell,j\}^\{\\mathrm\{tail\}\}\(r\)\.The same operation is applied independently to every targeted module\.

#### Cross\-run recomposition\.

LetAAandBBdenote two independently trained runs\. A mixed update such as

Δ​Wmix=Δ​WAhead​\(r\)\+Δ​WBtail​\(r\)\\Delta W\_\{\\mathrm\{mix\}\}=\\Delta W\_\{A\}^\{\\mathrm\{head\}\}\(r\)\+\\Delta W\_\{B\}^\{\\mathrm\{tail\}\}\(r\)can be represented exactly by rank concatenation\. If

Δ​WAhead=BAh​AAh,Δ​WBtail=BBt​ABt,\\Delta W\_\{A\}^\{\\mathrm\{head\}\}=B\_\{A\}^\{h\}A\_\{A\}^\{h\},\\qquad\\Delta W\_\{B\}^\{\\mathrm\{tail\}\}=B\_\{B\}^\{t\}A\_\{B\}^\{t\},then

Δ​Wmix=\[BAhBBt\]​\[AAhABt\]\.\\Delta W\_\{\\mathrm\{mix\}\}=\\begin\{bmatrix\}B\_\{A\}^\{h\}&B\_\{B\}^\{t\}\\end\{bmatrix\}\\begin\{bmatrix\}A\_\{A\}^\{h\}\\\\ A\_\{B\}^\{t\}\\end\{bmatrix\}\.This construction preserves the exact sum of the two effective\-update components\.

### A\.7Training\-Time Spectral Projection

For training\-time projection, the effective update is constrained after each optimizer step\. LetΔ​W\(τ\)\\Delta W^\{\(\\tau\)\}denote the current module\-level update at optimizer stepτ\\tau\. We replace it with one of

Πfull​\(Δ​W\(τ\)\)=Δ​W\(τ\),\\Pi\_\{\\mathrm\{full\}\}\(\\Delta W^\{\(\\tau\)\}\)=\\Delta W^\{\(\\tau\)\},Πhead​\(Δ​W\(τ\)\)=Δ​Whead​\(r\),\\Pi\_\{\\mathrm\{head\}\}\(\\Delta W^\{\(\\tau\)\}\)=\\Delta W^\{\\mathrm\{head\}\}\(r\),Πtail​\(Δ​W\(τ\)\)=Δ​Wtail​\(r\)\.\\Pi\_\{\\mathrm\{tail\}\}\(\\Delta W^\{\(\\tau\)\}\)=\\Delta W^\{\\mathrm\{tail\}\}\(r\)\.The projected update is refactorized into LoRA form using Appendix[A\.5](https://arxiv.org/html/2607.20438#A1.SS5)and written back to the adapter parameters before the next forward pass\. Projection is applied independently to each targeted module\.

### A\.8Entropy Effective Rank

For singular values\{σi\}\\\{\\sigma\_\{i\}\\\}, define

pi=σi∑kσk\.p\_\{i\}=\\frac\{\\sigma\_\{i\}\}\{\\sum\_\{k\}\\sigma\_\{k\}\}\.The spectral entropy is

H=−∑ipi​log⁡\(pi\+ε\),H=\-\\sum\_\{i\}p\_\{i\}\\log\(p\_\{i\}\+\\varepsilon\),and the effective rank is

reff=exp⁡\(H\)\.r\_\{\\mathrm\{eff\}\}=\\exp\(H\)\.Smallerreffr\_\{\\mathrm\{eff\}\}indicates concentration in fewer dominant singular directions, while largerreffr\_\{\\mathrm\{eff\}\}indicates a flatter spectrum\. Unless otherwise stated, checkpoint\-level summaries report the median effective rank over all targeted modules:

r~eff\(t\)=medianℓ,j⁡reff,ℓ,j\(t\)\.\\widetilde\{r\}\_\{\\mathrm\{eff\}\}^\{\(t\)\}=\\operatorname\{median\}\_\{\\ell,j\}\\;r\_\{\\mathrm\{eff\},\\ell,j\}^\{\(t\)\}\.

### A\.9Numerical Safeguards

All spectra are computed after forming the scaled effective update\. Singular values below a fixed numerical tolerance are treated as negligible when computing quantities that depend on rank support\. For entropy effective rank, the normalizationpi=σi/∑kσkp\_\{i\}=\\sigma\_\{i\}/\\sum\_\{k\}\\sigma\_\{k\}is applied only when the denominator is nonzero, and a small constantε\\varepsilonis used inside the logarithm\. The same tolerance and entropy convention are used across all compared runs\.

## Appendix BDetails for Spectral Formation

This appendix records the experimental grid and numerical summaries used for the spectral\-formation analysis in Section[4](https://arxiv.org/html/2607.20438#S4)\. All spectra are computed from the scaled effective updates defined in Appendix[A\.2](https://arxiv.org/html/2607.20438#A1.SS2), and checkpoint\-level summaries aggregate module\-level quantities over the targeted projections in Appendix[A\.1](https://arxiv.org/html/2607.20438#A1.SS1)\.

### B\.1Experimental Setup

Table 1:Experimental design for tracking spectral formation over training\.For each saved checkpointtt, we construct the module\-level effective updateΔ​Wℓ,j\(t\)\\Delta W\_\{\\ell,j\}^\{\(t\)\}, compute its singular\-value spectrum, and summarize spectral concentration with entropy effective rank\. Unless otherwise stated, run\-level effective\-rank summaries are reported as the median over all targeted modules\.

### B\.2Reference\-Trajectory Effective Rank

In the reference trajectory, the median effective rank follows a non\-monotonic pattern\. It decreases from43\.5743\.57at step 31 to41\.4841\.48at step 93, indicating early concentration of update mass into fewer leading directions\. It then rebounds to42\.6042\.60by step 217 and ends at42\.0342\.03by step 837\.

The rebound after the early contraction is the relevant feature\. A monotonic low\-rank\-collapse account would predict continued decrease as training proceeds\. Instead, the trajectory shows early concentration followed by stabilization, consistent with the coexistence of a growing spectral head and a retained residual tail\.

### B\.3Reference\-Trajectory Singular Spectra

The raw singular spectra show the same formation process directly\. At step 31, the largest singular values are approximately0\.00140\.0014,0\.00120\.0012, and0\.00070\.0007, while the remaining singular values lie near the10−410^\{\-4\}scale\. By step 837, the leading singular values increase to approximately0\.01700\.0170,0\.00690\.0069, and0\.00590\.0059\.

Thus, training substantially amplifies the leading directions, producing a clearer spectral head\. At the same time, the residual spectrum remains visible rather than disappearing\. This combination explains why effective rank contracts early but does not continue collapsing throughout training\.

### B\.4Generality Grid

The generality grid uses the same early\-versus\-final comparison protocol across 12 model–algorithm–supervision settings\. Each panel compares the singular spectrum of an early checkpoint with that of the final checkpoint for one setting\.

Across the grid, the final spectrum consistently exhibits a more pronounced leading head than the early spectrum, while retaining a visible residual tail\. The slope and separation strength vary across models and regimes, but the qualitative transition remains the same\. This supports the interpretation that head–tail organization is not tied to the reference trajectory alone\.

## Appendix CDetails for Functional Separation

The plug\-in experiments isolate the behavioral effect of spectral components under a fixed source\. The base model, training run, checkpoint, decoding protocol, and scoring rule are held fixed\. Only the loaded update component changes\. This appendix specifies the controlled task setting, the two training\-distribution biases, the plug\-in construction, and the exact values used in Figure[3](https://arxiv.org/html/2607.20438#S5.F3)\.

### C\.1Controlled Task Setting

We use synthetic, programmatically verifiable tasks as a controlled intervention testbed\. Their role is to provide an environment where training pressure, solver bias, and evaluation targets can be specified precisely\. This makes component\-level intervention easier to interpret than in open\-ended generation tasks\.

All tasks follow the same output protocol\. The model is prompted to solve the task and produce a final answer after a fixed markerFINAL:\. Only the content following this marker is used for scoring\.

### C\.2Training Runs

The plug\-in comparison uses two controlled DPO runs\. Both runs start from the same base model and share the same formatting protocol, optimizer configuration, decoding setup, and evaluation procedure\. They differ in training\-distribution bias, which creates two distinct solver pressures under otherwise matched conditions\.

#### Run\-Direct\.

Run\-Directis biased toward short\-horizon tasks solved by direct extraction, local transformation, or shallow computation\.

#### Run\-Deliberate\.

Run\-Deliberateassigns more probability mass to tasks requiring explicit intermediate decomposition and multi\-step reasoning\.

### C\.3Plug\-in Construction

For each final adapter, we split every module\-level effective update into a spectral head and residual tail using Appendix[A\.4](https://arxiv.org/html/2607.20438#A1.SS4)\. Unless otherwise stated, the split rank isk=16k=16\.

For each run, we evaluate four variants:

- •base: the original base model with no adapter loaded;
- •full: the full learned LoRA adapter;
- •head\-only: only the spectral head is loaded;
- •tail\-only: only the residual tail is loaded\.

Thehead\-onlyandtail\-onlyadapters are obtained by the refactorization procedure in Appendix[A\.5](https://arxiv.org/html/2607.20438#A1.SS5)\. Since the head and tail come from the same checkpoint and exactly reconstruct the full effective update when summed, the comparison controls for the backbone, training run, checkpoint, and optimization history\. Only the injected spectral component differs\.

### C\.4Evaluation and Scoring

We evaluate each variant on ID, OOD, and TRAP splits\. ID follows the same distributional bias as the corresponding training run\. OOD contains tasks requiring stronger transfer beyond the dominant training bias\. TRAP is designed to expose shortcutting, brittle pattern matching, or over\-deliberate solving\.

Each split is evaluated under both Canonical and Perturbed surface forms\. Canonical follows the standard task wording, while Perturbed changes superficial phrasing without changing the answer\.

All comparisons use strict final\-answer exact match\. Only the text afterFINAL:is extracted and compared with the programmatically computed ground truth\. Decoding is greedy and identical across variants\.

### C\.5Exact Plug\-in Results

Table[2](https://arxiv.org/html/2607.20438#A3.T2)reports the exact plug\-in values corresponding to Figure[3](https://arxiv.org/html/2607.20438#S5.F3)\. These values support the main\-text observation thattail\-onlyremains close tobase, whilehead\-onlycarries the main visible departure from the base model, with a run\-dependent behavioral profile\.

Table 2:Source\-controlled plug\-in results at the final checkpoint with split rankk=16k=16\. Metric is strict final\-answer accuracy under greedy decoding\. ID\-C/P, OOD\-C/P, and TRAP\-C/P denote the Canonical and Perturbed surface forms of the corresponding evaluation splits\.

## Appendix DDetails for Deep Analysis

This appendix provides the details omitted from Section[6](https://arxiv.org/html/2607.20438#S6)\. It covers cross\-run composition, training\-time spectral projection, and supervision corruption\.

### D\.1Cross\-run Composition

LetAAdenoteRun\-DirectandBBdenoteRun\-Deliberate\. Starting from the final checkpoints, we split each learned adapter into a spectral head and residual tail, then construct

head​\(A\)\+tail​\(B\),head​\(B\)\+tail​\(A\)\.\\mathrm\{head\}\(A\)\+\\mathrm\{tail\}\(B\),\\qquad\\mathrm\{head\}\(B\)\+\\mathrm\{tail\}\(A\)\.The recomposition is exact at the effective\-update level using the rank\-concatenation procedure in Appendix[A\.6](https://arxiv.org/html/2607.20438#A1.SS6)\.

To quantify behavioral similarity, we use the mean absolute difference \(MAD\) across the six evaluation metrics:

MAD​\(u,v\)=16​∑m∈\{ID​\-​C,ID​\-​P,OOD​\-​C,OOD​\-​P,TRAP​\-​C,TRAP​\-​P\}\|um−vm\|\.\\mathrm\{MAD\}\(u,v\)=\\frac\{1\}\{6\}\\sum\_\{m\\in\\\{\\mathrm\{ID\\text\{\-\}C,ID\\text\{\-\}P,OOD\\text\{\-\}C,OOD\\text\{\-\}P,TRAP\\text\{\-\}C,TRAP\\text\{\-\}P\}\\\}\}\|u\_\{m\}\-v\_\{m\}\|\.Smaller MAD indicates greater behavioral similarity\.

Table 3:MAD\-based similarity summary for within\-run and cross\-run comparisons\. Smaller values indicate greater behavioral similarity\.ComparisonCompared toMADWithin\-runRun\-Direct: full vs head\-only0\.2047Within\-runRun\-Direct: tail\-only vs base0\.0045Within\-runRun\-Deliberate: full vs head\-only0\.1248Within\-runRun\-Deliberate: tail\-only vs base0\.0087Cross\-runhead\(B\)\+tail\(A\) vs Run\-Deliberate head\-only0\.0320Cross\-runhead\(B\)\+tail\(A\) vs Run\-Deliberate full0\.1080Cross\-runhead\(B\)\+tail\(A\) vs Run\-Direct head\-only0\.1480Cross\-runhead\(B\)\+tail\(A\) vs Run\-Direct full0\.1330Cross\-runhead\(B\)\+tail\(A\) vs Run\-Deliberate tail\-only0\.1190Cross\-runhead\(B\)\+tail\(A\) vs Run\-Direct tail\-only0\.1260Cross\-runhead\(A\)\+tail\(B\) vs Run\-Direct head\-only0\.0340Cross\-runhead\(A\)\+tail\(B\) vs Run\-Direct full0\.1830Cross\-runhead\(A\)\+tail\(B\) vs Run\-Deliberate head\-only0\.1450Cross\-runhead\(A\)\+tail\(B\) vs Run\-Deliberate full0\.1870Cross\-runhead\(A\)\+tail\(B\) vs Run\-Direct tail\-only0\.1040Cross\-runhead\(A\)\+tail\(B\) vs Run\-Deliberate tail\-only0\.1120
### D\.2Training\-time Spectral Projection

To test whether the spectral head is sufficient during learning, we intervene after each optimizer step by projecting every targeted module into one of three variants:full,head\-only, ortail\-only\. Thefullcondition applies no projection\. Thehead\-onlycondition retains only the top\-kksingular directions\. Thetail\-onlycondition removes them and retains only the residual component\.

In implementation, we compute an equivalent economy decomposition inside the LoRA parameterization using QR factorization followed by SVD on the resultingr×rr\\times rmatrix\. This yields an exact projection of the current update into the chosen spectral subspace while keeping the LoRA parameter shapes fixed throughout training\.

Table 4:Family\-level summary for training\-time projection experiments withk=16k=16\.
### D\.3Supervision Corruption

To test whether stable preference learning depends on prompt–preference consistency, we corrupt a fractionρ∈\{0\.25,0\.50,1\.00\}\\rho\\in\\\{0\.25,0\.50,1\.00\\\}of the training set usingshuffle\_pairs\. For each corrupted example, the original prompt is kept fixed, but its preference pair\(y\+,y−\)\(y^\{\+\},y^\{\-\}\)is replaced by the pair from another randomly selected training example\. The pair is replaced as a whole; chosen and rejected responses are not shuffled independently\.

Corruption is applied only to the training set\. All evaluation splits remain clean\. In Figure[6](https://arxiv.org/html/2607.20438#S6.F6), the point atρ=0\\rho=0corresponds to the uncorrupted full\-training result\.

Table 5:Family\-level summary for supervision\-corruption experiments using training\-timeshuffle\_pairs\.The supervision\-corruption experiment is an upstream behavioral probe\. It does not, by itself, provide a complete spectral account of how corruption changes the strength of the spectral head, the thickness of the residual tail, or the trajectory of effective rank\.