CORAM: Coherent Orthogonal Rotation for Model Merging

# CORAM: Coherent Orthogonal Rotation for Model Merging
Source: [https://arxiv.org/html/2608.17366](https://arxiv.org/html/2608.17366)
###### Abstract

Merging finetuned models combines specialized capabilities without joint training or access to the original data\. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update\. Orthogonal Model Merging \(OrthoMerge\) uses a single orthogonal transform for each weight matrix, but such a transform cannot change singular values\. We propose CORAM, which partitions each target matrix into row slices, represents every expert slice by its singular value decomposition in the corresponding base\-model SVD frame, and merges the task\-specific factors on their corresponding manifolds\. Because manifold averaging contracts the merged update, CORAM applies an amplification coefficientλ=κ​c^\\lambda=\\kappa\\hat\{c\}\. The scalec^\\hat\{c\}is estimated from the expert and merged update norms and is approximatelyN\\sqrt\{N\}forNNexperts with comparable update magnitudes\. The restoration strengthκ\\kappais selected from the dispersion of expert updates without evaluating candidate merged models\. This rule remains within0\.720\.72points of the best swept value on all evaluated suites\. CORAM also includes spread slicing to distribute highly updated rows across slices and a residual pathway for non\-target layers\. Across four suites covering three model families, 3B to 9B scales, and language and vision\-language experts, CORAM improves over OrthoMerge by0\.250\.25to1\.351\.35points and matches or exceeds the strongest weight\-space baselines\.

## Introduction

Model merging provides a training\-free mechanism for consolidating independently finetuned capabilities, such as coding, mathematical reasoning, and multilingual understanding, into a single model without joint training or access to the original training data\([34](https://arxiv.org/html/2608.17366#bib.bib7);[14](https://arxiv.org/html/2608.17366#bib.bib5)\)\. Most existing methods represent each finetuned model as a task vector in Euclidean weight space and combine these vectors through linear arithmetic\([14](https://arxiv.org/html/2608.17366#bib.bib5);[35](https://arxiv.org/html/2608.17366#bib.bib4);[40](https://arxiv.org/html/2608.17366#bib.bib6)\)\. Although simple and effective, this formulation ignores the geometric structure of weight updates\. In particular, finetuning a weight matrix changes both its singular subspaces and its singular\-value spectrum, which are not naturally represented by direct Euclidean addition\.

Recent work addresses this limitation by performing model merging on structured geometric spaces\. Orthogonal Model Merging \(OrthoMerge\)\([38](https://arxiv.org/html/2608.17366#bib.bib3)\), for example, aligns finetuned weights using orthogonal transformations\. However, it estimates a single transformation for an entire weight matrix\. This matrix\-level granularity can be restrictive because fine\-tuning updates often concentrate in low\-dimensional and heterogeneous subspaces\([2](https://arxiv.org/html/2608.17366#bib.bib40);[13](https://arxiv.org/html/2608.17366#bib.bib39)\)\. A single matrix\-wide transformation may therefore mix strongly updated directions with largely unchanged ones\.

![Refer to caption](https://arxiv.org/html/2608.17366v1/AuthorKit27/Figures/f1_28.png)Figure 1:Overview of CORAM\. Each expert slice is factored in the base frame into a rotation, a spectral shift𝑑𝑆i=Si​S0−1−I\\mathit\{dS\}\_\{i\}=S\_\{i\}S\_\{0\}^\{\-1\}\-I, and a right factor\. Each factor is averaged in its own space\. The coefficientλ\\lambdais given by Eq\. \([3](https://arxiv.org/html/2608.17366#Sx3.E3)\)\.A finer representation is provided by CORA\([32](https://arxiv.org/html/2608.17366#bib.bib1)\), which parameterizes model adaptation using row\-level weight slices\. However, CORA addresses adaptation of an individual model rather than merging multiple independently finetuned models\. Extending this representation to model merging raises two additional problems\. First, the task\-specific factors of each slice lie on different curved spaces and must be combined using geometry\-compatible operations\. Second, averaging on these spaces can substantially reduce the magnitude of the resulting update\.

We introduceCORAM, a slice\-level manifold method for model merging, illustrated in Fig\.[1](https://arxiv.org/html/2608.17366#Sx1.F1)\. CORAM partitions each target weight matrix into row slices and represents each expert slice through its SVD relative to the corresponding base\-model slice\. It merges three factors on their natural spaces: rotations are combined using a log\-Euclidean mean on the special orthogonal group\([25](https://arxiv.org/html/2608.17366#bib.bib9);[4](https://arxiv.org/html/2608.17366#bib.bib8)\), relative spectral shifts are combined through linear averaging, and right factors are aggregated using a polar mean on the Stiefel manifold\([6](https://arxiv.org/html/2608.17366#bib.bib10);[17](https://arxiv.org/html/2608.17366#bib.bib21)\)\. We additionally introduce a conflict\-aware variant that masks, within each slice, neuron columns on which the experts exhibit inconsistent updates before aggregation\. This adapts the conflict\-handling mechanism of[38](https://arxiv.org/html/2608.17366#bib.bib3)to the slice level\.

A central challenge is that manifold averaging contracts the merged update toward the base model\([18](https://arxiv.org/html/2608.17366#bib.bib42)\)\. Without correction, this contraction can remove a substantial portion of the task\-specific signal\. CORAM compensates for the contraction using an amplification coefficientλ=κ​c^\\lambda=\\kappa\\hat\{c\}, wherec^\\hat\{c\}estimates the contraction scale andκ\\kappasets restoration strength\. Importantly, neither term requires evaluating candidate merged models\. The contraction scalec^\\hat\{c\}is computed from the norms of the expert and merged updates, and is approximated byN\\sqrt\{N\}when theNNexperts contribute updates of comparable magnitude, as established in Proposition[1](https://arxiv.org/html/2608.17366#Thmproposition1)\.

The appropriate restoration strengthκ\\kappadepends on how evenly the experts modify the base model\. When expert updates have comparable magnitudes, near\-full restoration is effective\. When the magnitudes differ substantially, as observed for the strongest base model in our study, partial restoration performs better\. CORAM distinguishes these two cases using a dispersion statistic computed from the expert updates without evaluating candidate merged models\. The resulting rule selects the amplification coefficient without a sweep overλ\\lambdaand remains within 0\.72 score points of the best swept coefficient across all evaluated suites\.

The slice\-level formulation motivates two complementary refinements\. First,*spread slicing*orders rows according to update magnitude and distributes them across slices, reducing the concentration of highly updated rows in a small number of slices\. Second, a*residual pathway*restores fine\-tuning updates from layers outside the sliced merging targets, adapting the residual\-decoupling principle of[38](https://arxiv.org/html/2608.17366#bib.bib3)to the layer\-level residuals\. Their effectiveness also depends on the distribution of update magnitudes: spread slicing is beneficial by itself when expert updates are relatively uniform, whereas with strongly uneven updates it is most effective when combined with the residual pathway\.

We evaluate CORAM on four heterogeneous model\-merging suites spanning three model families, model sizes from 3B to 9B parameters, and both language and vision\-language experts\. Our main contributions are as follows:

- •Search\-free amplification\.We introduce the ruleλ=κ​c^\\lambda=\\kappa\\hat\{c\}, where the contraction scalec^\\hat\{c\}is estimated from expert and merged update norms and restoration strengthκ\\kappais selected using a zero\-evaluation dispersion statistic\. The predicted coefficient remains within 0\.72 score points of the swept optimum on every evaluated suite\.
- •Slice\-level manifold merging\.We merge per\-slice SVD factors using geometry\-compatible operations on the special orthogonal group, Euclidean spectral space, and the Stiefel manifold\.
- •Geometry\-motivated refinement\.We introduce spread slicing and show how its effectiveness changes with the dispersion of expert\-update magnitudes\.
- •Evaluation across heterogeneous settings\.Across four suites covering three model families, 3B–9B parameter scales, and language and vision\-language tasks, CORAM improves over OrthoMerge by 0\.25–1\.35 score points under a common evaluation protocol and matches or exceeds the strongest evaluated weight\-space baselines\.

## Related Work

#### Euclidean weight\-space merging\.

Most model\-merging methods represent each finetuned expert as a task vector and combine the vectors in Euclidean weight space\. Uniform averaging\([34](https://arxiv.org/html/2608.17366#bib.bib7)\), Fisher\- or regression\-weighted merging\([24](https://arxiv.org/html/2608.17366#bib.bib12);[16](https://arxiv.org/html/2608.17366#bib.bib13)\), and task arithmetic\([14](https://arxiv.org/html/2608.17366#bib.bib5)\)differ mainly in how the updates are weighted\. TIES, DARE, and Localize\-and\-Stitch further reduce interference through sign agreement, sparsification, or parameter localization\([35](https://arxiv.org/html/2608.17366#bib.bib4);[40](https://arxiv.org/html/2608.17366#bib.bib6);[9](https://arxiv.org/html/2608.17366#bib.bib15)\)\. These methods are simple and strong, but they treat a weight matrix as a flat vector and do not explicitly model its rotational and spectral structure\.

#### Geometric and subspace merging\.

To move beyond direct weight arithmetic, existing methods align model units\([3](https://arxiv.org/html/2608.17366#bib.bib16);[29](https://arxiv.org/html/2608.17366#bib.bib17)\), merge in tangent spaces\([26](https://arxiv.org/html/2608.17366#bib.bib38)\), or construct subspaces from the SVD of task updates\([7](https://arxiv.org/html/2608.17366#bib.bib11);[30](https://arxiv.org/html/2608.17366#bib.bib18);[23](https://arxiv.org/html/2608.17366#bib.bib19)\)\. Closest to CORAM, OrthoMerge averages one orthogonal transform per weight matrix and merges the remaining residual conventionally\([38](https://arxiv.org/html/2608.17366#bib.bib3)\)\. An orthogonal transform cannot change singular values, so the spectral part of the update remains in the residual\. CORA provides a finer per\-slice SVD representation, but studies model adaptation rather than multi\-expert merging\([32](https://arxiv.org/html/2608.17366#bib.bib1)\)\. CORAM uses this per\-slice representation for model merging and averages the rotation, relative spectral shift, and right factor in their spaces\. Its conflict\-aware variants and residual pathway adapt the conflict handling and residual decoupling of OrthoMerge to slices and non\-target layers\.

#### Coefficient selection in model merging\.

Merging on these structured spaces raises a further question: how to set the scale of the merged update\. Task arithmetic uses a fixed constant\([14](https://arxiv.org/html/2608.17366#bib.bib5)\), and benchmark protocols select one value per suite\([10](https://arxiv.org/html/2608.17366#bib.bib2)\)\. AdaMerging instead optimizes per\-layer coefficients at test time\([37](https://arxiv.org/html/2608.17366#bib.bib14)\), while Model Stock derives a geometric rule for models finetuned on the same task\([15](https://arxiv.org/html/2608.17366#bib.bib43)\)\. CORAM relates the coefficient to the contraction of the geometric merge\. The contraction scale is measured from the expert and merged update norms, and the restoration strength is selected from the dispersion of expert\-update magnitudes\. This choice is motivated by prior observations that merging behavior changes with the base model\([10](https://arxiv.org/html/2608.17366#bib.bib2);[36](https://arxiv.org/html/2608.17366#bib.bib41)\), and avoids a per\-benchmark sweep overλ\\lambda\.

## Method

### Slice Representation in the Base Frame

Following CORA\([32](https://arxiv.org/html/2608.17366#bib.bib1)\), we partition each target weight matrixW∈ℝdout×dinW\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}intoG=dout/hG=d\_\{\\mathrm\{out\}\}/hrow slices of heighthh,W\(g\)=W\[gh:\(g\+1\)h,:\]W^\{\(g\)\}=W\[gh\{:\}\(g\{\+\}1\)h,\\,:\], and we work in the base frame\. Each base slice is factored once asW0\(g\)=U0​S0​V0⊤W\_\{0\}^\{\(g\)\}=U\_\{0\}\\,S\_\{0\}\\,V\_\{0\}^\{\\top\}\. For each experti∈\{1,…,N\}i\\in\\\{1,\\dots,N\\\}and each slice, we express the expert weights relative to this frame using three quantities: a rotationQi∈S​O​\(h\)Q\_\{i\}\\in SO\(h\)aligning the expert left singular basis withU0U\_\{0\}, obtained by the polar factor ofU0⊤​UiU\_\{0\}^\{\\top\}U\_\{i\}; a relative spectral shift𝑑𝑆i=Si​S0−1−I\\mathit\{dS\}\_\{i\}=S\_\{i\}S\_\{0\}^\{\-1\}\-I; and the right factorViV\_\{i\}\. Before formingQiQ\_\{i\}, we resolve the sign ambiguity of each singular direction jointly against the base pair\(U0,V0\)\(U\_\{0\},V\_\{0\}\): each\(u,v\)\(u,v\)pair is flipped together, leavingWiW\_\{i\}unchanged\. The triple\(Qi,𝑑𝑆i,Vi\)\(Q\_\{i\},\\mathit\{dS\}\_\{i\},V\_\{i\}\)lies on a product of structured spaces: the special orthogonal groupS​O​\(h\)SO\(h\), the relative spectral shifts, and the Stiefel manifoldSt⁡\(din,h\)\\mathrm\{St\}\(d\_\{\\mathrm\{in\}\},h\)\([1](https://arxiv.org/html/2608.17366#bib.bib20)\), and this structure dictates how the factors should be averaged\.

### Per\-Slice Manifold Merging

Given the per\-expert slice triples, CORAM merges each factor by the mean native to its space \(Fig\.[1](https://arxiv.org/html/2608.17366#Sx1.F1), panel 2\):

*Rotations\.*We average\{Qi\}\\\{Q\_\{i\}\\\}by the log\-Euclidean mean onS​O​\(h\)SO\(h\)\([25](https://arxiv.org/html/2608.17366#bib.bib9);[4](https://arxiv.org/html/2608.17366#bib.bib8)\), mapping each rotation to the Lie algebra, averaging, and mapping back,

Q¯=exp⁡\(1N​∑ilog⁡Qi\)\.\\bar\{Q\}\\;=\\;\\exp\\\!\\Big\(\\tfrac\{1\}\{N\}\\textstyle\\sum\_\{i\}\\log Q\_\{i\}\\Big\)\.\(1\)Because𝔰​𝔬​\(h\)\\mathfrak\{so\}\(h\)is a linear space, the operation inside the exponential is an ordinary mean, while the nonlinearity enters only through the logarithmic and exponential maps\.

*Spectral shifts\.*The relative spectral shifts are averaged linearly,𝑑𝑆¯=∑i1N​𝑑𝑆i\\overline\{\\mathit\{dS\}\}=\\sum\_\{i\}\\tfrac\{1\}\{N\}\\mathit\{dS\}\_\{i\}\.

*Right factors\.*We average\{Vi\}\\\{V\_\{i\}\\\}by the polar mean on the Stiefel manifold\([6](https://arxiv.org/html/2608.17366#bib.bib10);[17](https://arxiv.org/html/2608.17366#bib.bib21)\), forming the weighted Euclidean sum and projecting back onto the manifold via the polar decompositionV¯=polar⁡\(∑i1N​Vi\)\\bar\{V\}=\\mathrm\{polar\}\\\!\\big\(\\textstyle\\sum\_\{i\}\\tfrac\{1\}\{N\}V\_\{i\}\\big\)\.

The merged slice is reconstructed in the base frame as

Wmerge,g=\(U0​Q¯\)​S0​\(I\+𝑑𝑆¯\)​V¯⊤\+η¯\(g\),W\_\{\\mathrm\{merge\},g\}\\;=\\;\\big\(U\_\{0\}\\,\\bar\{Q\}\\big\)\\,S\_\{0\}\\,\\big\(I\+\\overline\{\\mathit\{dS\}\}\\big\)\\,\\bar\{V\}^\{\\top\}\\;\+\\;\\bar\{\\eta\}^\{\(g\)\},\(2\)whereη¯\(g\)\\bar\{\\eta\}^\{\(g\)\}is a conflict residual, defined in the conflict\-aware variants below, and is zero when no masking is applied\. Stacking the merged slices yieldsWmergeW\_\{\\mathrm\{merge\}\}for the matrix\. Applying the same procedure to all target matrices produces the geometric merge of the model\.

### Conflict\-Aware Variants

Experts can disagree sharply on individual neurons, and averaging across such disagreements dilutes every expert’s contribution\. Adapting the conflict\-handling idea of Orthogonal Model Merging\([38](https://arxiv.org/html/2608.17366#bib.bib3)\)to the slice level, our conflict\-aware variants detect, per slice and per column, the experts whose update direction opposes the consensus: withτi=Wi\(g\)−W0\(g\)\\tau\_\{i\}=W\_\{i\}^\{\(g\)\}\-W\_\{0\}^\{\(g\)\}andτ¯=1N​∑iτi\\bar\{\\tau\}=\\tfrac\{1\}\{N\}\\sum\_\{i\}\\tau\_\{i\}, columnjjof expertiiis flagged whencos\(τi\[:,j\],τ¯\[:,j\]\)\\cos\\\!\\big\(\\tau\_\{i\}\[:,j\],\\,\\bar\{\\tau\}\[:,j\]\\big\)is negative\. The flagged columns split each expert update in two parts: the part that enters the manifold means above, and the complement, which is either discarded or averaged across experts and added back to Eq\. \([2](https://arxiv.org/html/2608.17366#Sx3.E2)\) asη¯\(g\)\\bar\{\\eta\}^\{\(g\)\}\. Combining these choices yields the three variants used in our experiments:

- •The non\-flagged columns are merged on the manifolds, and the flagged ones are discarded\.
- •The non\-flagged columns are merged on the manifolds, and the flagged ones return asη¯\(g\)\\bar\{\\eta\}^\{\(g\)\}\.
- •The flagged columns are merged on the manifolds, and the non\-flagged ones return asη¯\(g\)\\bar\{\\eta\}^\{\(g\)\}\.

The fourth combination, merging only the flagged columns and discarding the rest, would throw away the signal on which the experts agree, and we do not use it\. For the first two variants, flagged columns are masked out of that expert’s contribution before the slice is factored, so the per\-slice manifold means above are taken over non\-conflicting evidence only\. We refer to the plain method as CORAM and collectively denote the conflict\-aware variants by CORAM\-C\. Both method families are evaluated throughout\.

### Amplitude Restoration

Merging on curved spaces introduces an effect absent from Euclidean merging\. The manifold mean*contracts*the merged update toward the base, i\.e\.,‖Wmerge−W0‖\\\|W\_\{\\mathrm\{merge\}\}\-W\_\{0\}\\\|is substantially smaller than the typical expert update‖Wi−W0‖\\\|W\_\{i\}\-W\_\{0\}\\\|\. The merged direction is preserved, but its magnitude is reduced\. CORAM therefore applies a single global amplification,

W∗=W0\+λ⁡\(Wmerge−W0\),λ=κ​c^,W^\{\*\}\\;=\\;W\_\{0\}\+\\lambda\\,\\big\(W\_\{\\mathrm\{merge\}\}\-W\_\{0\}\\big\),\\qquad\\lambda=\\kappa\\,\\hat\{c\},\(3\)where the scalec^\\hat\{c\}and strengthκ\\kappaare determined without evaluating candidate merged models\.

*Contraction scale\.*The contraction is directly measurable from quantities already computed during merging:

c^=cRMS=RMSi​‖Δ​Wi‖‖Δ​Wmerge‖,Δ​W≡W−W0,\\hat\{c\}\\;=\\;c\_\{\\mathrm\{RMS\}\}\\;=\\;\\frac\{\\mathrm\{RMS\}\_\{i\}\\,\\\|\\Delta W\_\{i\}\\\|\}\{\\\|\\Delta W\_\{\\mathrm\{merge\}\}\\\|\},\\qquad\\Delta W\\equiv W\-W\_\{0\},\(4\)the ratio between the typical expert update magnitude and the merged one\. No task evaluation is involved\. In the ideal case, this rate has a closed form:

###### Proposition 1\(Cancellation scale\)\.

Letτ1,…,τN\\tau\_\{1\},\\dots,\\tau\_\{N\}be slice updates with𝔼⁡⟨τi,τj⟩=0\\mathbb\{E\}\\langle\\tau\_\{i\},\\tau\_\{j\}\\rangle=0fori≠ji\\neq jand‖τi‖=‖τ‖\\\|\\tau\_\{i\}\\\|=\\\|\\tau\\\|for allii\. Then𝔼​‖1N​∑iτi‖2=‖τ‖2/N\\mathbb\{E\}\\,\\big\\\|\\tfrac\{1\}\{N\}\\sum\_\{i\}\\tau\_\{i\}\\big\\\|^\{2\}=\\\|\\tau\\\|^\{2\}/N, so restoring the mean to the typical expert magnitude requires amplification byN\\sqrt\{N\}\.

The proposition follows from the random\-vector cancellation law\. It applies directly to the rotation average in the linear space𝔰​𝔬​\(h\)\\mathfrak\{so\}\(h\)and to the Euclidean average of the relative spectral shifts \(supplementary material\)\. The additional effect of the Stiefel mean is included incRMSc\_\{\\mathrm\{RMS\}\}, which extends theN\\sqrt\{N\}scale to correlated or uneven updates\. Across four low\-dispersion settings,cRMS=2\.09/2\.03/1\.72/2\.09c\_\{\\mathrm\{RMS\}\}=2\.09/2\.03/1\.72/2\.09, compared withN=2\.24/2\.24/1\.73/2\.24\\sqrt\{N\}=2\.24/2\.24/1\.73/2\.24, with a difference below10%10\\%in each case\. We therefore useλ=κ​N\\lambda=\\kappa\\sqrt\{N\}as the default prescription and retaincRMSc\_\{\\mathrm\{RMS\}\}as the direct measurement of contraction\.

*Restoration strength\.*The remaining constantκ\\kappais determined empirically\. Sweeps on every base place its optimum in one of two cases: a low\-dispersion case, with comparable expert update magnitudes and near\-full restoration \(κ≈1\.15\\kappa\\approx 1\.15\), and a high\-dispersion case, with strongly uneven update magnitudes and partial restoration \(κ≈0\.5\\kappa\\approx 0\.5\)\. We distinguish these cases*before task evaluation*using the dispersion statistic,

D=maxi⁡rimini⁡ri,ri=‖Δ​Wi‖‖W0‖,D=\\frac\{\\max\_\{i\}r\_\{i\}\}\{\\min\_\{i\}r\_\{i\}\},\\quad r\_\{i\}=\\frac\{\\\|\\Delta W\_\{i\}\\\|\}\{\\\|W\_\{0\}\\\|\},\(5\)computed from the same cached quantities as Eq\. \([4](https://arxiv.org/html/2608.17366#Sx3.E4)\)\. SmallDDselects the low\-dispersion case, whereas largeDDselects the high\-dispersion case\. The amplification coefficient is therefore set from Proposition[1](https://arxiv.org/html/2608.17366#Thmproposition1)andDD, without a per\-benchmark sweep overλ\\lambda\. We also tested alternative closed\-form choices overλ∈\[1,4\]\\lambda\\in\[1,4\], as reported in the supplementary material\.

## Geometry\-Derived Refinements

The slice\-level formulation supports two additional refinements\. Both are optional and are evaluated separately and together\. Their effects depend on the distribution of expert\-update magnitudes\. Spread slicing is effective by itself when the update magnitudes are comparable\. When the update magnitudes differ substantially, it works better together with the residual pathway\.

### Spread Slicing

Contiguous slicing follows the row order of the pretrained matrix\. If heavily updated rows cluster in a small number of slices, these slices may have poorly conditioned SVDs while the remaining slices contain little update signal\. Spread slicing distributes the important rows more evenly\. Before slicing, we score every rowrrof every expert by its relative update magnitude,si\(r\)=∥ΔWi\[r,:\]∥/∥W0\[r,:\]∥s\_\{i\}\(r\)=\\\|\\Delta W\_\{i\}\[r,:\]\\\|/\\\|W\_\{0\}\[r,:\]\\\|, and we summarize the scores of a row across the experts by their largest values\(1\)s\_\{\(1\)\}, the second largests\(2\)s\_\{\(2\)\}, their mean, and their variance\. A priority is formed from these quantities, and the rows are then handed out to slices in descending priority\. Four settings are available:

- •Mean\.The priority is the mean of the scores, which weights all experts equally\.
- •Energy\.The priority iss\(1\)s\_\{\(1\)\}, so the rows that some expert updates most strongly are spread first\.
- •Variance\.The priority isVari⁡\[si​\(r\)\]​s\(1\)\\operatorname\{Var\}\_\{i\}\[s\_\{i\}\(r\)\]\\,s\_\{\(1\)\}\. The variance is large when experts disagree on how much to update the row, whiles\(1\)s\_\{\(1\)\}discounts rows with little update signal\.
- •Owner balance\.The priority is\(s\(1\)−s\(2\)\)​s\(1\)\(s\_\{\(1\)\}\-s\_\{\(2\)\}\)\\,s\_\{\(1\)\}, which is large when an expert dominates the row and that row also changes a lot\.

The first three settings assign rows to slices in round\-robin order\. Owner balance assigns each row to the slice with the smallest accumulated priority, and breaks ties to prevent rows dominated by the same expert from concentrating in one slice\. After merging, the permutation is inverted to restore the original row order\. This permutation changes only how rows are grouped into slices and does not change the function computed by any expert\. The same permutation is applied to all experts, so the merge remains well defined\. We denote this configuration by CORAM\+\+SS\.

### Layer\-Level Residual Pathway

The slice geometry covers the target linear layers, including the attention and MLP projections\. Embeddings and normalization parameters are not included, so a purely geometric merge omits their fine\-tuning updates\. Following the orthogonal and residual decoupling idea of[38](https://arxiv.org/html/2608.17366#bib.bib3), we merge the non\-target residualsρi=Wi−W0\\rho\_\{i\}=W\_\{i\}\-W\_\{0\}using a conventional weight\-space method and add the result as a separate patch\.

We use task arithmetic\([14](https://arxiv.org/html/2608.17366#bib.bib5)\), TIES\([35](https://arxiv.org/html/2608.17366#bib.bib4)\), or Task Singular Vectors\([7](https://arxiv.org/html/2608.17366#bib.bib11)\)for the residual merge\. For Task Singular Vectors, we follow the distributed\-TSVM implementation of[38](https://arxiv.org/html/2608.17366#bib.bib3)\. We denote this configuration by CORAM\+\+RP\. The final model combines the amplified geometric merge with the unscaled residual patch,

W∗=W0\+\{λ⁡\(Wmerge−W0\)target layers,ρ¯non\-target,W^\{\*\}\\;=\\;W\_\{0\}\+\\begin\{cases\}\\lambda\\,\(W\_\{\\mathrm\{merge\}\}\-W\_\{0\}\)&\\text\{target layers\},\\\\ \\bar\{\\rho\}&\\text\{non\-target\},\\end\{cases\}\(6\)whereλ\\lambdafollows the amplitude\-restoration rule: the amplification corrects the contraction of the*manifold*mean only, while the residual merger is Euclidean and needs no correction\. When both refinements are enabled, CORAM\+\+SS\+\+RP applies spread slicing before geometric merging and adds the residual patch afterward\.

## Experiments

In\-domainIn\-domain AvgOut\-of\-domainMethodMATH500HEval\+\+SciQACSQASIQAκ\\kappa\-rulebestλ\\lambdaM\-ARCAGIEvalOOD AvgLlama\-3\.1\-8B17\.8021\.5271\.4070\.6048\.1145\.8933\.0230\.2731\.65Task\-specific FT19\.0038\.5491\.8282\.4756\.8157\.73–––Linear23\.4032\.2683\.5976\.8251\.8453\.5834\.4132\.2733\.34Task Arithmetic24\.8037\.2086\.2979\.5253\.3356\.2334\.9732\.6833\.83TIES21\.4040\.4382\.8778\.3054\.0455\.4134\.4032\.5333\.47DARE\-TIES19\.2040\.3082\.2478\.5454\.4554\.9533\.5032\.2432\.87OrthoMerge \(OFT\)24\.2037\.8787\.7280\.7555\.1257\.1335\.1533\.1334\.14CORAM21\.8040\.4388\.3181\.2456\.8157\.7257\.98 \(h=16h\{=\}16,λ=2\.40\\lambda\{=\}2\.40\)35\.2132\.5333\.87CORAM\-C20\.6038\.4188\.4481\.2456\.8157\.1057\.74 \(λ=2\.25\\lambda\{=\}2\.25\)35\.1532\.6133\.88CORAM\-C\+\+SS21\.4039\.8888\.4081\.0856\.7057\.4958\.21\(λ=2\.30\\lambda\{=\}2\.30\)35\.1232\.1733\.65
In\-domainIn\-domain AvgOut\-of\-domainMethodInstr\.MathCodingMulti\.Safetyκ\\kappa\-rulebestλ\\lambdaMMLU†AGIEvalOOD AvgLlama\-3\.2\-3B7\.5828\.5127\.4440\.7231\.4127\.1856\.7924\.0140\.40Task\-specific FT39\.5669\.8344\.3341\.7380\.4655\.18–––Linear9\.8040\.8637\.1542\.2240\.2134\.0557\.5425\.8841\.71Task Arithmetic18\.3045\.4940\.1842\.4744\.9038\.2757\.5026\.8242\.16TIES22\.1850\.0439\.4042\.0341\.9339\.1256\.7627\.0841\.92DARE\-TIES31\.6156\.9439\.0141\.2248\.5043\.4555\.9425\.5540\.74OrthoMerge \(TSV\-M\+C\)20\.1555\.5741\.7642\.2850\.9042\.1357\.4226\.8742\.14CORAM\+\+SS\+\+RP31\.2452\.9942\.1241\.7248\.4043\.3043\.30 \(λ=2\.60\\lambda\{=\}2\.60\)56\.0225\.6040\.81CORAM\-C\+\+RP31\.7953\.8340\.7941\.5648\.6543\.3243\.32 \(λ=2\.60\\lambda\{=\}2\.60\)56\.1825\.6040\.89CORAM\-C\+\+SS\+\+RP30\.5052\.8442\.4041\.7049\.9443\.4843\.48\(λ=2\.60\\lambda\{=\}2\.60\)56\.1725\.6740\.92

Table 1:Top–T1: merging five orthogonally finetuned Llama\-3\.1\-8B experts\.Bottom–T2: merging five fully finetuned Llama\-3\.2\-3B experts from MergeBench\. Higher is better\. Per column, the best result among merging methods is in bold \(base\-model and task\-specific\-FT reference rows excluded\)\. CORAM\-C denotes the conflict\-aware variants \(masking neurons where experts disagree\)\.\+\+SS adds spread slicing \(a task\-informed row permutation\) and\+\+RP the residual pathway \(conventional merge of the non\-target layers\)\. CORAM rows evaluate the search\-freeκ\\kappa\-rule checkpoint \(gridλ=2\.60\\lambda\{=\}2\.60for both suites\)\. “bestλ\\lambda” is the sweep optimum, for reference \(see Setup\)\.### Setup

#### Suites\.

We evaluate CORAM on four suites covering three model families, model sizes from 3B to 9B, and both language and vision\-language experts\.

T1contains five orthogonally finetuned Llama\-3\.1\-8B experts\([27](https://arxiv.org/html/2608.17366#bib.bib37)\)released by[38](https://arxiv.org/html/2608.17366#bib.bib3)\. We evaluate them on MATH500\([12](https://arxiv.org/html/2608.17366#bib.bib22)\), HumanEval\+\+\([19](https://arxiv.org/html/2608.17366#bib.bib23)\), ScienceQA\([22](https://arxiv.org/html/2608.17366#bib.bib24)\), CommonsenseQA\([31](https://arxiv.org/html/2608.17366#bib.bib25)\), and Social\-IQA\([28](https://arxiv.org/html/2608.17366#bib.bib26)\)\.

T2andT4each contain five fully finetuned experts for instruction following, mathematics, coding, multilingual tasks, and safety\. The experts are based on Llama\-3\.2\-3B and Gemma\-2\-9B and are obtained from MergeBench\([10](https://arxiv.org/html/2608.17366#bib.bib2)\)\. We follow the task suites, evaluation protocol, and hyperparameter settings of MergeBench\.

T3contains three Qwen2\.5\-VL\-7B\-Instruct experts for spatial reasoning, OCR, and medical multimodal question answering\. We follow the vision\-language setup of[38](https://arxiv.org/html/2608.17366#bib.bib3)and evaluate on MMSI\-Bench\([39](https://arxiv.org/html/2608.17366#bib.bib31)\), EmbSpatial\([5](https://arxiv.org/html/2608.17366#bib.bib32)\), MMMU\-Med\([41](https://arxiv.org/html/2608.17366#bib.bib36)\), PathVQA\([8](https://arxiv.org/html/2608.17366#bib.bib33)\), OCRBench\([21](https://arxiv.org/html/2608.17366#bib.bib34)\), and CharXiv\([33](https://arxiv.org/html/2608.17366#bib.bib35)\)\. We use the multiple\-choice subset of MMMU\-Med\. CharXiv is scored using its official GPT judge, gpt\-4o\-2024\-05\-13\. Our evaluation pipeline obtains 69\.50 for the best variant of[38](https://arxiv.org/html/2608.17366#bib.bib3), compared with the reported score of 69\.90\.

#### Cost\.

CORAM uses slice SVDs that are computed once and cached for subsequent merges\. The geometric merge requires 0\.5 to 0\.8 GPU\-hours for each configuration\. The evaluation cost of a sweep overλ\\lambdais one to two orders of magnitude larger than the merge cost\. Detailed storage and runtime statistics are provided in the supplementary material\.

#### Out\-of\-domain evaluation\.

We evaluate the retention of general capabilities using the out\-of\-domain \(OOD\) tasks adopted by[38](https://arxiv.org/html/2608.17366#bib.bib3)\. For T1, we use M\-ARC and AGIEval\([42](https://arxiv.org/html/2608.17366#bib.bib27)\)\. For T2 and T4, we use MMLU†\([11](https://arxiv.org/html/2608.17366#bib.bib28)\)and AGIEval, with the mathematics and coding subsets removed from MMLU\. For T3, we use IFEval\([43](https://arxiv.org/html/2608.17366#bib.bib29)\)and MMBench\([20](https://arxiv.org/html/2608.17366#bib.bib30)\)\. All OOD tasks are evaluated zero\-shot using the same evaluation harness\. The specific differences are provided in the supplementary material\.

In\-domainIn\-domain AvgOut\-of\-domainMethodMMSIEmbSp\.MMMUMedPathVQAOCRB\.CharXivκ\\kappa\-rulebestλ\\lambdaIFEvalMMB\.OOD AvgQwen2\.5\-VL\-7B\-It\.27\.8069\.9753\.1066\.3084\.7067\.2061\.5163\.0383\.9373\.48Task\-specific FT32\.6070\.5855\.1766\.8185\.0072\.5063\.78–––Linear29\.2071\.2955\.1768\.4784\.8067\.3062\.7158\.2384\.1971\.21Task Arithmetic29\.1071\.0755\.8668\.3884\.6066\.1062\.5259\.3384\.1971\.76TIES32\.1071\.5457\.9368\.4482\.8069\.4063\.7054\.5384\.0269\.27DARE\-TIES32\.1071\.7658\.6266\.9880\.8069\.6063\.3151\.0282\.8266\.92OrthoMerge \(TIES\+C\)32\.3071\.7656\.5568\.1483\.1069\.5063\.5654\.5383\.6869\.10CORAM33\.2071\.5960\.0066\.6985\.4067\.0063\.9864\.10 \(λ=1\.60\\lambda\{=\}1\.60\)53\.4283\.2568\.33CORAM\+\+RP33\.2072\.1760\.6967\.2284\.9068\.1064\.3864\.38\(λ=2\.00\\lambda\{=\}2\.00\)54\.5382\.9968\.76CORAM\-C\+\+RP33\.7072\.0359\.3167\.3784\.9067\.9064\.2064\.26 \(λ=1\.75\\lambda\{=\}1\.75\)52\.6883\.1667\.92Table 2:T3: merging three Qwen2\.5\-VL\-7B\-Instruct vision–language experts\. Layout and bolding as in Table[1](https://arxiv.org/html/2608.17366#Sx5.T1)\.κ\\kappa\-rule grid pointλ=2\.00\\lambda\{=\}2\.00\. CORAM\-C denotes the conflict\-aware variants \(masking neurons where experts disagree\)\.\+\+RP adds the residual pathway \(conventional merge of the non\-target layers\)\. The OrthoMerge out\-of\-domain entries are obtained from our zero\-shot re\-evaluation\.In\-domainIn\-domain AvgOut\-of\-domainMethodInstr\.MathCodingMulti\.Safetyκ\\kappa\-rulebestλ\\lambdaMMLU†AGIEvalOOD AvgGemma\-2\-9B14\.2369\.8343\.3754\.6334\.3943\.2970\.4138\.0154\.21Task\-specific FT65\.0679\.7658\.5155\.9176\.0067\.05–––Linear27\.1781\.0551\.5253\.7959\.3154\.5767\.8837\.7552\.81Task Arithmetic27\.5482\.3451\.6551\.3154\.2653\.4264\.4435\.7050\.07TIES24\.0382\.9446\.6045\.6452\.8450\.4160\.6034\.9247\.76DARE\-TIES18\.1176\.5720\.2234\.2349\.1239\.6542\.1827\.1034\.64OrthoMerge \(TA\+C\)26\.4381\.2752\.8353\.7859\.3754\.7467\.8037\.8552\.83CORAM\-C27\.3681\.2751\.6253\.5257\.8354\.3254\.32 \(λ=1\.10\\lambda\{=\}1\.10\)67\.2238\.0352\.63CORAM\+\+RP26\.0681\.5851\.8453\.5959\.7354\.5654\.84 \(λ=1\.25\\lambda\{=\}1\.25\)67\.3037\.2852\.29CORAM\+\+SS\+\+RP26\.2581\.8053\.1053\.6658\.2854\.6254\.99\(λ=1\.25\\lambda\{=\}1\.25\)67\.5937\.6252\.60Table 3:T4: merging five fully finetuned Gemma\-2\-9B experts from MergeBench with uneven update magnitudes \(D≈16D\{\\approx\}16\)\. The rule selectsκ=0\.5\\kappa\{=\}0\.5, corresponding toλ=1\.10\\lambda\{=\}1\.10\. Layout and bolding follow Table[1](https://arxiv.org/html/2608.17366#Sx5.T1)\. CORAM\-C masks neurons where experts disagree\.\+\+SS adds a task\-informed row permutation, and\+\+RP merges the non\-target layers\.
#### Baselines and CORAM configurations\.

We implement linear averaging\([34](https://arxiv.org/html/2608.17366#bib.bib7)\), task arithmetic\([14](https://arxiv.org/html/2608.17366#bib.bib5)\), TIES\([35](https://arxiv.org/html/2608.17366#bib.bib4)\), and DARE\-TIES\([40](https://arxiv.org/html/2608.17366#bib.bib6);[35](https://arxiv.org/html/2608.17366#bib.bib4)\)as per\-tensor algorithms following their original definitions\. We use the MergeBench coefficient settings and the baseline set of[38](https://arxiv.org/html/2608.17366#bib.bib3)\. For Orthogonal Model Merging\([38](https://arxiv.org/html/2608.17366#bib.bib3)\), we evaluate all released variants under our protocol and report the strongest result for each suite\. On T2, our reproduction obtains 42\.13, compared with the published 42\.07\. All table comparisons use the same protocol\. Reproduction details are provided in the supplementary material\.

The base\-model and task\-specific finetuning results for T1, T2, and T3 are taken from[38](https://arxiv.org/html/2608.17366#bib.bib3)\. The corresponding results for T4 are measured using our evaluation harness\. CORAM denotes the plain method, and CORAM\-C denotes the conflict\-aware variants\. CORAM\+\+SS includes spread slicing, while CORAM\+\+RP includes the residual pathway\. The main tables report three representative configurations for each suite\. The complete component combinations are reported in the supplementary material\.

We use a slice height ofh=8h\\\!=\\\!8andh=16h\\\!=\\\!16in experiments\. Each CORAM row evaluates one checkpoint selected without a sweep overλ\\lambda\. The task scores, OOD scores, andκ\\kappa\-rule average are obtained from the model merged usingλ=κ​N\\lambda\\\!=\\\!\\kappa\\sqrt\{N\}\.κ\\kappais selected using the dispersion statisticDDin Eq\. \([5](https://arxiv.org/html/2608.17366#Sx3.E5)\)\. The bestλ\\lambdacolumn is the only result obtained from a sweep\. It reports the in\-domain average at the best value for each suite and measures the difference between the selected coefficient and the sweep optimum\. The detailed configuration corresponding to each CORAM row is reported in the supplementary material\.

### Main Results

Tables[1](https://arxiv.org/html/2608.17366#Sx5.T1)–[3](https://arxiv.org/html/2608.17366#Sx5.T3)report the main comparison\.*First*, CORAM outperforms every OrthoMerge variant on all four suites, by\+1\.08\+1\.08\(T1\),\+1\.35\+1\.35\(T2\),\+0\.82\+0\.82\(T3\), and\+0\.25\+0\.25\(T4\) at the respective best configurations, all measured under our single harness against our reproduction of OrthoMerge \(see Setup\)\.*Second*, against the strongest weight\-space baselines CORAM leads on T1, T3, and T4, and matches DARE\-TIES on T2 \(43\.48 vs\. 43\.45\)\. No baseline is consistently competitive: the method strongest on any one suite trails CORAM by 1\.1–15\.3 points on the others\. Retention tells the same story: on T4 the aggressive DARE\-TIES baseline collapses out\-of\-domain \(34\.64 vs\. our 52\.60\), while CORAM’s OOD averages stay within the band of the mildest mergers on every suite\.*Third*, the search\-freeκ\\kappa\-rule recovers near\-peak accuracy throughout: across all suites and configurations, its gap to the swept\-λ\\lambdaoptimum is at most 0\.72 points, including on T4, where the rule selects the*high\-dispersion*case \(κ≈0\.5\\kappa\{\\approx\}0\.5\) purely from the zero\-evaluation dispersionDD\. It removes the need for a per\-suiteλ\\lambdasweep of ten or more grid points, each requiring a full\-suite evaluation\.

### The Amplification Dichotomy

#### Two cases\.

T4 evaluates the rule on a different model family and the largest model considered\. The value ofDDis computed before task evaluation\. Across the five bases \(see supplementary material\),DDseparates the models into low\- and high\-dispersion cases\. Qwen2\.5\-VL\-7B, Gemma\-2\-2B, Llama\-3\.2\-3B, and Llama\-3\.1\-8B haveD∈\[1\.3,3\.5\]D\\in\[1\.3,3\.5\], while Gemma\-2\-9B hasD≈16D\\approx 16\. The optimalκ\\kappa, measured byλ\\lambdasweeps on every base, follows the same division\. It lies between approximately0\.90\.9and1\.21\.2in the low\-dispersion case and near0\.50\.5in the high\-dispersion case\.

We therefore adopt two shared constants,κ=1\.15\\kappa\{=\}1\.15andκ=0\.5\\kappa\{=\}0\.5, fixed once across all suites rather than fitted per suite\. Because the sweep optima are flat, this choice differs from the per\-suite optimum by at most0\.720\.72points\. Usingκ=1\.10\\kappa=1\.10does not change any comparison in the tables\. The supplementary material reports the per\-configuration ranges ofκopt\\kappa\_\{\\mathrm\{opt\}\}, the relation betweenκ\\kappaandDD, and a multi\-seed evaluation of the main\-table checkpoints\. The value used in the high\-dispersion case is calibrated on the only available base with a large value ofDD, as discussed in the limitations\. Our results support these two cases but do not establish howκ\\kappabehaves for intermediate values ofDD\.

#### Relation to base model strength\.

The two cases are not explained by model family or fine\-tuning method\. Within Gemma\-2, the 2B base hasD=1\.67D=1\.67, while the 9B base hasD=16D=16, so the selectedκ\\kappachanges with scale within the same architecture\. The 9B math expert is trained with GRPO, while the others use SFT\. Excluding the math expert givesD=16\.0D=16\.0for Gemma\-2\-9B andD=1\.67D=1\.67for Gemma\-2\-2B under matched fine\-tuning methods\. The difference is therefore more closely associated with base\-model strength, consistent with observations that merging behaves differently on stronger bases\([10](https://arxiv.org/html/2608.17366#bib.bib2);[36](https://arxiv.org/html/2608.17366#bib.bib41)\)\. The statisticDDmeasures this difference before merging\.

#### Explanation\.

Proposition[1](https://arxiv.org/html/2608.17366#Thmproposition1)assumes nearly orthogonal updates with comparable norms\. A small value ofDDindicates that the update norms are comparable, so near\-full restoration is appropriate\. AtD≈16D\\approx 16, the update magnitudes differ substantially and the equal\-norm assumption no longer holds\. Full restoration then amplifies the merged update too strongly, and a value nearκ=0\.5\\kappa=0\.5performs better\. The T4 results in Table[3](https://arxiv.org/html/2608.17366#Sx5.T3)show thatDDdistinguishes the two cases before task evaluation\.

Table 4:Best in\-domain average reached at theκ\\kappa\-rule point \(best configuration per method and suite\)\. RP is inapplicable on T1 \(no non\-target residual\)\.

### Component Effects in the Two Cases

Table[4](https://arxiv.org/html/2608.17366#Sx5.T4)evaluates each component under theκ\\kappa\-rule for CORAM in the top half and CORAM\-C in the bottom half\. CORAM\-C performs better than plain CORAM on T2 by\+0\.21\+0\.21and on T3 by\+0\.07\+0\.07\. Combining spread slicing and the residual pathway gives the best value at the selectedκ\\kappaon two of the four suites, with 43\.48 on T2, and 54\.71 on T4\. On T1, CORAM\-C\+\+SS gives the best result under theκ\\kappa\-rule at 57\.98, while CORAM\+\+RP gives the best result on T3 at 64\.38\. Sweepingλ\\lambdafurther improves the T1 result to 58\.21 atλ=2\.30\\lambda=2\.30and the T4 result to 54\.99 atλ=1\.25\\lambda=1\.25\. These gains quantify the remaining gap between theκ\\kappa\-rule point and the swept optimum\.

#### Effect of spread slicing\.

The sweep results below are reported in the supplementary material\. When the expert\-update magnitudes are comparable, spread slicing improves the swept optimum by\+0\.34\+0\.34on T1,\+0\.41\+0\.41on T2, and\+0\.10\+0\.10on T3 relative to the best contiguous configuration on the same branch\. When the update magnitudes differ substantially, spread slicing alone does not help\. Every spread\-only configuration on T4 is0\.580\.58to0\.860\.86points below its contiguous counterpart\.

The reduction is concentrated in the safety task\. Four of the five T4 domains change by less than±1\\pm 1point, while safety decreases by4\.74\.7points\. The supplementary analysis links this drop to less stable open\-ended generation rather than a general capability loss\. Adding the residual pathway restores the omitted embedding and normalization updates\. The combined configuration outperforms the residual pathway alone, with 54\.62 vs\. 54\.56 at the selectedκ\\kappaand 54\.99 vs\. 54\.84 at the swept optimum in Table[3](https://arxiv.org/html/2608.17366#Sx5.T3)\. Spread slicing works by itself when update magnitudes are comparable, while uneven updates benefit from the residual pathway\.

## Limitations

#### Intermediate values ofDD\.

Four bases have lowDD, while one has highDD\. We do not extrapolate to intermediate values, which are not represented in current public expert suites\.

#### Scale and strength\.

Proposition[1](https://arxiv.org/html/2608.17366#Thmproposition1)gives the ideal scale, andcRMSc\_\{\\mathrm\{RMS\}\}extends it to correlated or uneven updates\. It remains within10%10\\%ofN\\sqrt\{N\}when update magnitudes are comparable\. However,κ≈1\.15\\kappa\\approx 1\.15andκ≈0\.5\\kappa\\approx 0\.5are empirical values selected byDD\. The latter is calibrated on a single high\-DDbase\. A full contraction analysis of the composed slice map remains open\.

#### Behavioral evidence\.

The failure of spread slicing under uneven updates is observed at the domain and generation levels and is corrected by the residual pathway\. We do not provide a theoretical explanation\.

#### Scope\.

We follow the expert suites and protocols of MergeBench and[38](https://arxiv.org/html/2608.17366#bib.bib3)\. Larger numbers of experts, other architectures such as mixture\-of\-experts models, and experts derived from different base checkpoints are not evaluated\.

## Conclusion

We presented CORAM, a slice\-level method for merging finetuned experts\. Each expert slice is represented by its SVD in the base frame, and the task\-specific factors are averaged on their corresponding manifolds\. CORAM compensates for the resulting contraction using an amplification scale estimated from the norms of the expert and merged updates and a strength selected from update dispersion without evaluating candidate merged models\. Spread slicing helps when update magnitudes are comparable, while uneven updates benefit from the residual pathway\. Across four suites covering three model families and both language and vision\-language experts, CORAM improves over Orthogonal Model Merging and matches or exceeds the strongest weight\-space baselines without a per\-benchmark sweep overλ\\lambda\. Future work will study intermediate update dispersion and contraction under manifold averaging\.

## References

- Absilet al\.\(2008\)P\.\-A\. Absil, R\. Mahony, and R\. SepulchreOptimization algorithms on matrix manifolds\.Princeton University Press\.External Links:ISBN 9780691132983,[Link](http://www.jstor.org/stable/j.ctt7smmk)Cited by:[Slice Representation in the Base Frame](https://arxiv.org/html/2608.17366#Sx3.SSx1.p1.1)\.
- Aghajanyanet al\.\(2021\)A\. Aghajanyan, S\. Gupta, and L\. ZettlemoyerIntrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 7319–7328\.External Links:[Link](https://aclanthology.org/2021.acl-long.568)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p2.1)\.
- Ainsworthet al\.\(2023\)S\. Ainsworth, J\. Hayase, and S\. SrinivasaGit re\-basin: merging models modulo permutation symmetries\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CQsmMYmlP5T)Cited by:[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1)\.
- Arsignyet al\.\(2006\)V\. Arsigny, P\. Fillard, X\. Pennec, and N\. AyacheLog\-euclidean metrics for fast and simple calculus on diffusion tensors\.Magnetic Resonance in Medicine56\(2\),pp\. 411–421\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/mrm.20965),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.20965),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/mrm\.20965Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p4.1),[Per\-Slice Manifold Merging](https://arxiv.org/html/2608.17366#Sx3.SSx2.p2.1)\.
- Duet al\.\(2024\)M\. Du, B\. Wu, Z\. Li, X\. Huang, and Z\. WeiEmbSpatial\-bench: benchmarking spatial understanding for embodied tasks with large vision\-language models\.InACL \(Short Papers\),pp\. 346–355\.External Links:[Link](https://aclanthology.org/2024.acl-short.33)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1)\.
- Edelmanet al\.\(1998\)A\. Edelman, T\. A\. Arias, and S\. T\. SmithThe geometry of algorithms with orthogonality constraints\.SIAM Journal on Matrix Analysis and Applications20\(2\),pp\. 303–353\.External Links:[Document](https://dx.doi.org/10.1137/S0895479895290954)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p4.1),[Per\-Slice Manifold Merging](https://arxiv.org/html/2608.17366#Sx3.SSx2.p4.1)\.
- Gargiuloet al\.\(2025\)A\. A\. Gargiulo, D\. Crisostomi, M\. S\. Bucarelli, S\. Scardapane, F\. Silvestri, and E\. RodolàTask singular vectors: reducing task interference in model merging\.InCVPR,pp\. 18695–18705\.Cited by:[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1),[Layer\-Level Residual Pathway](https://arxiv.org/html/2608.17366#Sx4.SSx2.p2.1)\.
- Heet al\.\(2020\)X\. He, Y\. Zhang, L\. Mou, E\. Xing, and P\. XiePathVQA: 30000\+ questions for medical visual question answering\.External Links:2003\.10286,[Link](https://arxiv.org/abs/2003.10286)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1)\.
- Heet al\.\(2024\)Y\. He, Y\. Hu, Y\. Lin, T\. Zhang, and H\. ZhaoLocalize\-and\-stitch: efficient model merging via sparse task arithmetic\.CoRRabs/2408\.13656\.External Links:[Link](https://doi.org/10.48550/arXiv.2408.13656)Cited by:[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2025\)Y\. He, S\. Zeng, Y\. Hu, R\. Yang, T\. Zhang, and H\. ZhaoMergeBench: a benchmark for merging domain\-specialized LLMs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=rw50iUoyLu)Cited by:[Coefficient selection in model merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px3.p1.1),[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p3.1),[Relation to base model strength\.](https://arxiv.org/html/2608.17366#Sx5.SSx3.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InICLR,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[Out\-of\-domain evaluation\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px3.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.InNeurIPS Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2103.03874)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p2.1)\.
- Ilharcoet al\.\(2023\)G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, L\. Schmidt, H\. Hajishirzi, and A\. FarhadiEditing models with task arithmetic\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6t0Kwf8-jrj)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p1.1),[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1),[Coefficient selection in model merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px3.p1.1),[Layer\-Level Residual Pathway](https://arxiv.org/html/2608.17366#Sx4.SSx2.p2.1),[Baselines and CORAM configurations\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px4.p1.1)\.
- Janget al\.\(2024\)D\. Jang, S\. Yun, and D\. HanModel stock: all we need is just a few fine\-tuned models\.InEuropean Conference on Computer Vision \(ECCV\),External Links:[Link](https://arxiv.org/abs/2403.19522)Cited by:[Coefficient selection in model merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px3.p1.1)\.
- Jinet al\.\(2023\)X\. Jin, X\. Ren, D\. Preotiuc\-Pietro, and P\. ChengDataless knowledge fusion by merging weights of language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FCnohuR6AnM)Cited by:[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1)\.
- Kanekoet al\.\(2013\)T\. Kaneko, S\. Fiori, and T\. TanakaEmpirical arithmetic averaging over the compact stiefel manifold\.IEEE Transactions on Signal Processing61\(4\),pp\. 883–894\.External Links:[Document](https://dx.doi.org/10.1109/TSP.2012.2226167)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p4.1),[Per\-Slice Manifold Merging](https://arxiv.org/html/2608.17366#Sx3.SSx2.p4.1)\.
- Karcher \(2014\)H\. KarcherRiemannian center of mass and so called karcher mean\.External Links:1407\.2087,[Link](https://arxiv.org/abs/1407.2087)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p5.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.InNeurIPS,External Links:[Link](https://arxiv.org/abs/2305.01210)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1)\.
- Liuet al\.\(2024a\)Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu, K\. Chen, and D\. LinMMBench: is your multi\-modal model an all\-around player?\.InEuropean Conference on Computer Vision \(ECCV\),External Links:[Link](https://arxiv.org/abs/2307.06281)Cited by:[Out\-of\-domain evaluation\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px3.p1.1)\.
- Liuet al\.\(2024b\)Y\. Liu, Z\. Li, M\. Huang, B\. Yang, W\. Yu, C\. Li, X\. Yin, C\. Liu, L\. Jin, and X\. BaiOCRBench: on the hidden mystery of ocr in large multimodal models\.Science China Information Sciences67\(12\)\.External Links:ISSN 1869\-1919,[Link](http://dx.doi.org/10.1007/s11432-024-4235-6),[Document](https://dx.doi.org/10.1007/s11432-024-4235-6)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1)\.
- Luet al\.\(2022\)P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. KalyanLearn to explain: multimodal reasoning via thought chains for science question answering\.InNeurIPS,External Links:[Link](https://arxiv.org/abs/2209.09513)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1)\.
- Marczaket al\.\(2025\)D\. Marczak, S\. Magistri, S\. Cygert, B\. Twardowski, A\. D\. Bagdanov, and J\. van de WeijerNo task left behind: isotropic model merging with common and task\-specific subspaces\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=RBZpAa27ls)Cited by:[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1)\.
- Matena and Raffel \(2022\)M\. Matena and C\. RaffelMerging models with fisher\-weighted averaging\.InNeurIPS,External Links:[Link](http://papers.nips.cc/paper_files/paper/2022/hash/70c26937fbf3d4600b69a129031b66ec-Abstract-Conference.html)Cited by:[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1)\.
- Moakher \(2002\)M\. MoakherMeans and averaging in the group of rotations\.SIAM Journal on Matrix Analysis and Applications24\(1\),pp\. 1–16\.External Links:[Document](https://dx.doi.org/10.1137/S0895479801383877)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p4.1),[Per\-Slice Manifold Merging](https://arxiv.org/html/2608.17366#Sx3.SSx2.p2.1)\.
- Ortiz\-Jiménezet al\.\(2023\)G\. Ortiz\-Jiménez, A\. Favero, and P\. FrossardTask arithmetic in the tangent space: improved editing of pre\-trained models\.InNeurIPS,External Links:[Link](https://arxiv.org/abs/2305.12827)Cited by:[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1)\.
- Qiuet al\.\(2023\)Z\. Qiu, W\. Liu, H\. Feng, Y\. Xue, Y\. Feng, Z\. Liu, D\. Zhang, A\. Weller, and B\. SchölkopfControlling text\-to\-image diffusion by orthogonal finetuning\.InNeurIPS,External Links:[Link](https://arxiv.org/abs/2306.07280)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. L\. Bras, and Y\. ChoiSocial iqa: commonsense reasoning about social interactions\.InEMNLP/IJCNLP \(1\),pp\. 4462–4472\.External Links:[Link](https://doi.org/10.18653/v1/D19-1454)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1)\.
- Stoicaet al\.\(2024\)G\. Stoica, D\. Bolya, J\. Bjorner, T\. Hearn, and J\. HoffmanZipIt\! merging models from different tasks without training\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=LEYUkvdUhq)Cited by:[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1)\.
- Stoicaet al\.\(2025\)G\. Stoica, P\. Ramesh, B\. Ecsedi, L\. Choshen, and J\. HoffmanModel merging with SVD to tie the knots\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=67X93aZHII)Cited by:[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1)\.
- Talmoret al\.\(2019\)A\. Talmor, J\. Herzig, N\. Lourie, and J\. BerantCommonsenseQA: a question answering challenge targeting commonsense knowledge\.InNAACL\-HLT,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1)\.
- Wanget al\.\(2026\)P\. Wang, Z\. Liu, W\. Wang, and W\. JiangCORA: per\-slice coherent orthogonal rotation for svd\-based low\-rank adaptation\.External Links:2607\.02576,[Link](https://arxiv.org/abs/2607.02576)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p3.1),[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1),[Slice Representation in the Base Frame](https://arxiv.org/html/2608.17366#Sx3.SSx1.p1.1)\.
- Wanget al\.\(2024\)Z\. Wang, M\. Xia, L\. He, H\. Chen, Y\. Liu, R\. Zhu, K\. Liang, X\. Wu, H\. Liu, S\. Malladi, A\. Chevalier, S\. Arora, and D\. ChenCharXiv: charting gaps in realistic chart understanding in multimodal llms\.InNeurIPS Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2406.18521)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1)\.
- Wortsmanet al\.\(2022\)M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. G\. Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith, and L\. SchmidtModel soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://proceedings.mlr.press/v162/wortsman22a.html)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p1.1),[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1),[Baselines and CORAM configurations\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px4.p1.1)\.
- Yadavet al\.\(2023\)P\. Yadav, D\. Tam, L\. Choshen, C\. A\. Raffel, and M\. BansalTIES\-merging: resolving interference when merging models\.InNeurIPS,External Links:[Link](http://papers.nips.cc/paper_files/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p1.1),[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1),[Layer\-Level Residual Pathway](https://arxiv.org/html/2608.17366#Sx4.SSx2.p2.1),[Baselines and CORAM configurations\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px4.p1.1)\.
- Yadavet al\.\(2025\)P\. Yadav, T\. Vu, J\. Lai, A\. Chronopoulou, M\. Faruqui, M\. Bansal, and T\. MunkhdalaiWhat matters for model merging at scale?\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=9sbetmvNpW)Cited by:[Coefficient selection in model merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px3.p1.1),[Relation to base model strength\.](https://arxiv.org/html/2608.17366#Sx5.SSx3.SSS0.Px2.p1.1)\.
- Yanget al\.\(2024\)E\. Yang, Z\. Wang, L\. Shen, S\. Liu, G\. Guo, X\. Wang, and D\. TaoAdaMerging: adaptive model merging for multi\-task learning\.InICLR,External Links:[Link](https://openreview.net/forum?id=nZP6NgD3QY)Cited by:[Coefficient selection in model merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2026\)S\. Yang, K\. Shi, and W\. LiuOrthogonal model merging\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=pCaeMsqpCa)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p2.1),[Introduction](https://arxiv.org/html/2608.17366#Sx1.p4.1),[Introduction](https://arxiv.org/html/2608.17366#Sx1.p7.1),[Geometric and subspace merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px2.p1.1),[Conflict\-Aware Variants](https://arxiv.org/html/2608.17366#Sx3.SSx3.p1.1),[Layer\-Level Residual Pathway](https://arxiv.org/html/2608.17366#Sx4.SSx2.p1.1),[Layer\-Level Residual Pathway](https://arxiv.org/html/2608.17366#Sx4.SSx2.p2.1),[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p2.1),[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1),[Out\-of\-domain evaluation\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px3.p1.1),[Baselines and CORAM configurations\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px4.p1.1),[Baselines and CORAM configurations\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px4.p2.1),[Scope\.](https://arxiv.org/html/2608.17366#Sx6.SSx4.SSS0.Px4.p1.1)\.
- Yanget al\.\(2025\)S\. Yang, R\. Xu, Y\. Xie, S\. Yang, M\. Li, J\. Lin, C\. Zhu, X\. Chen, H\. Duan, X\. Yue, D\. Lin, T\. Wang, and J\. PangMMSI\-bench: a benchmark for multi\-image spatial intelligence\.CoRRabs/2505\.23764\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.23764)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1)\.
- Yuet al\.\(2024\)L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. LiLanguage models are super mario: absorbing abilities from homologous models as a free lunch\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=fq0NaiU8Ex)Cited by:[Introduction](https://arxiv.org/html/2608.17366#Sx1.p1.1),[Euclidean weight\-space merging\.](https://arxiv.org/html/2608.17366#Sx2.SS0.SSS0.Px1.p1.1),[Baselines and CORAM configurations\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px4.p1.1)\.
- Yueet al\.\(2024\)X\. Yue, Y\. Ni, T\. Zheng, K\. Zhang, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. ChenMMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InCVPR,pp\. 9556–9567\.External Links:[Link](https://doi.org/10.1109/CVPR52733.2024.00913)Cited by:[Suites\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px1.p4.1)\.
- Zhonget al\.\(2024\)W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. DuanAGIEval: a human\-centric benchmark for evaluating foundation models\.InFindings of NAACL,External Links:[Link](https://aclanthology.org/2024.findings-naacl.149)Cited by:[Out\-of\-domain evaluation\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[Out\-of\-domain evaluation\.](https://arxiv.org/html/2608.17366#Sx5.SSx1.SSS0.Px3.p1.1)\.