Bayesian Model Merging

# Bayesian Model Merging
Source: [https://arxiv.org/html/2605.12843](https://arxiv.org/html/2605.12843)
Kaiyang Li1Shaobo Han2Qing Su1Shihao Ji1 1School of Computing, University of Connecticut, Storrs, CT 06269 \{kaiyang\.li, qing\.2\.su, shihao\.ji\}@uconn\.edu 2Optical Networking and Sensing, NEC Labs America, Princeton, NJ 08540 shaobo@nec\-labs\.com

###### Abstract

Model merging aims to combine multiple task\-specific expert models into a single model without joint retraining, offering a practical alternative to multi\-task learning when data access or computational budget is limited\. Existing methods, however, face two key limitations: \(1\) they overlook the valuable inductive bias of strong anchor models and estimate the merged weights from scratch, and \(2\) they rely on a shared hyperparameter setting across different modules of the network, lacking a global optimization strategy\. This paper introducesBayesian Model Merging \(BMM\), a*plug\-and\-play*bi\-level optimization framework, where the inner level formulates the model merging as an activation\-based Bayesian regression under a strong prior induced by an anchor model, yielding an efficient closed\-form solution; and the outer level leverages a Bayesian optimization procedure to search module\-specific hyperparameters globally based on a small validation set\. Furthermore, we reveal a key alignment between activation statistics and task vectors, enabling us to derive a data\-free variant of BMM that estimates the Gram matrix for regression without any auxiliary data\. Across extensive benchmarks, including up to 20\-task merging in vision and 5\-task merging in language, BMM consistently outperforms allplug\-and\-playanchor baselines \(e\.g\., TA, WUDI\-Merging, and TSV\)\. In particular, on the ViT\-L/14 benchmark for 8\-task merging, a single merged model reaches95\.1, closely matching the average performance of eight task\-specific experts \(95\.8\)\.

## 1Introduction

Adapting foundation models to downstream tasks via fine\-tuning has become standard practice, but maintaining a separate expert model for each task introduces substantial storage, deployment, and operational overhead\. While multi\-task learning offers a unified alternative\[[1](https://arxiv.org/html/2605.12843#bib.bib1)\], it necessitates joint data access and incurs prohibitive computational costs, making it often infeasible under data silos, privacy constraints, or limited computing budgets\. Model merging\[[2](https://arxiv.org/html/2605.12843#bib.bib2),[3](https://arxiv.org/html/2605.12843#bib.bib3),[4](https://arxiv.org/html/2605.12843#bib.bib4)\]offers a practical solution\. By directly combining multiple expert models into a single architecture, it enables unified inference without revisiting the original training data or incurring the joint retraining costs\. This paradigm has become increasingly relevant with the flourishing ecosystem of open\-source models on platforms such as Hugging Face\[[5](https://arxiv.org/html/2605.12843#bib.bib5)\], which offers an abundant supply of task\-specific experts for integration\.

A central challenge in model merging is how to effectively combine expert models when auxiliary data is limited or unavailable\. Existing methods fall into two broad regimes based on their use of auxiliary data\.*Data\-assisted*methods, such as Fisher Merging\[[2](https://arxiv.org/html/2605.12843#bib.bib2)\]and RegMean\[[6](https://arxiv.org/html/2605.12843#bib.bib6)\], rely on a small calibration set to estimate empirical statistics for merging\.*Data\-free*methods, including Task Arithmetic \(TA\)\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], TIES\[[4](https://arxiv.org/html/2605.12843#bib.bib4)\], WUDI\-Merging\[[7](https://arxiv.org/html/2605.12843#bib.bib7)\], TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\], and ISO\-CTS\[[9](https://arxiv.org/html/2605.12843#bib.bib9)\], avoid auxiliary data for estimating merged parameters, though some still use a held\-out validation set for lightweight hyperparameter selection\. Despite their differences, both regimes suffer from two important limitations\. First, they typically estimate merged weights from scratch without exploiting prior knowledge from strong anchor models, leaving valuable inductive bias underused\. Second, many advanced methods operate primarily through module\-wise merging in isolation, and adopt a shared hyperparameter setting across different modules for tuning convenience, overlooking module heterogeneity and lacking a global optimization strategy for hyperparameter search\.

To address these limitations, we proposeBayesian Model Merging \(BMM\), a*plug\-and\-play*bi\-level optimization framework that is built on two key ideas\. First, instead of estimating merged weights from scratch, BMM leverages prior knowledge from strong anchor models \(e\.g\., TA\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], RegMean\[[6](https://arxiv.org/html/2605.12843#bib.bib6)\], or TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\]\), leading to an activation\-based Bayesian regression with an efficient closed\-form solution for module\-wise merging\. Second, rather than relying on a shared hyperparameter setting across all modules, BMM performs globally coordinated architecture\-aware Bayesian optimization to adapt regularization strengths across heterogeneous modules of the network\. Moreover, we theoretically and empirically reveal a key alignment between activation statistics and module\-level task vectors, enabling us to derive a data\-free variant of BMM while preserving the closed\-form solution\.

Empirically, BMM is effective across both vision and language model merging\. Across four backbones from two model families and seven*plug\-and\-play*anchor models, BMM consistently improves over every anchor in both data\-assisted and data\-free settings, with relative gains of up to 27% on weaker anchors while still improving the strongest ones\. In particular, on the standard ViT\-L/14 benchmark, a single merged model reaches95\.1%, closely matching the average performance of eight task\-specific experts \(95\.8%\)\. Extensive ablation studies are conducted to further validate the main components and design choices of BMM\.

## 2Related Work

Training\-free model mergingefficiently combines expert checkpoints without full data access or costly joint retraining\. Existing methods generally fall into two broad regimes based on their reliance on auxiliary data:data\-assistedanddata\-free\.Data\-assisted methodsleverage a small calibration set or activation statistics to guide the merge\. For example, Fisher Merging\[[2](https://arxiv.org/html/2605.12843#bib.bib2)\]performs Fisher\-weighted average of task\-specific models, while RegMean\[[6](https://arxiv.org/html/2605.12843#bib.bib6)\]casts linear\-layer merging as regression with a closed\-form solution\.Data\-free methodseliminate the need of auxiliary data and rely entirely on the weights of the expert models\. In this regime, Task Arithmetic \(TA\)\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\]serves as the seminal baseline\. Subsequent works mainly mitigate interference among models through sparsification, including TIES\[[4](https://arxiv.org/html/2605.12843#bib.bib4)\], DARE\[[10](https://arxiv.org/html/2605.12843#bib.bib10)\], PCB\-Merging\[[11](https://arxiv.org/html/2605.12843#bib.bib11)\], and Localize\-and\-Stitch\[[12](https://arxiv.org/html/2605.12843#bib.bib12)\]\. Another line of work \(e\.g\., TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\]and ISO\-CTS\[[9](https://arxiv.org/html/2605.12843#bib.bib9)\]\) leverages structured low\-rank subspaces to isolate task\-specific features or align task\-relevant subspaces\. More recently, methods such as WUDI\-Merging\[[7](https://arxiv.org/html/2605.12843#bib.bib7)\]and DOGE\[[13](https://arxiv.org/html/2605.12843#bib.bib13)\]optimize explicit data\-free merging objectives, but do not exploit prior knowledge from strong anchor models and learn a uniform hyperparameter setting for different modules of the network\. Our BMM is complementary to these approaches: it leverages existing merged solutions as anchors, admits an efficient closed\-form solution for module\-level merging, and exploits bi\-level optimization for hyperparameter search across different module groups\.

#### Representation geometry, neural collapse, and alignment after fine\-tuning\.

A growing literature investigates the emergence of structured geometry in trained neural networks\. Neural collapse\[[14](https://arxiv.org/html/2605.12843#bib.bib14)\]shows that during the terminal phase of training, the last\-layer features collapse to their class means, and classifier weights align with the same simplex geometry\. Neural Feature Ansatz\[[15](https://arxiv.org/html/2605.12843#bib.bib15)\]and Deep RFM\[[16](https://arxiv.org/html/2605.12843#bib.bib16)\]broaden this perspective to network\-wise feature learning and connect the layer\-wise weight structure to the average gradient outer products\. A recent analysis by Liu et al\.\[[17](https://arxiv.org/html/2605.12843#bib.bib17)\]further shows that latent representations, network weights, and gradients become mutually aligned across hidden layers\. Several recent works\[[18](https://arxiv.org/html/2605.12843#bib.bib18),[19](https://arxiv.org/html/2605.12843#bib.bib19),[20](https://arxiv.org/html/2605.12843#bib.bib20)\]also connect neural collapse to transfer learning and fine\-tuning, including collapse\-inspired fine\-tuning, transferability estimation, and analyses of downstream geometric complexity, but these studies mainly focus on last\-layer collapse, transferability, or downstream fine\-tuning behavior\. In contrast, we study model merging from task\-specific checkpoints, each of which is fine\-tuned on a specific downstream task until convergence\. We establish a module\-wise alignment relation between the Gram matrix of the task vectors and the corresponding second\-moment statistics of the activations, and leverage it to derive a data\-free variant of BMM with a closed\-form solution\.

## 3Problem Definition

#### Notation\.

Letθpre∈ℝd\\theta\_\{\\mathrm\{pre\}\}\\in\\mathbb\{R\}^\{d\}denote the parameters of a pretrained base model, and\{θ\(t\)\}t=1T\\\{\\theta^\{\(t\)\}\\\}\_\{t=1\}^\{T\}represent the parameters ofTTdistinct models fine\-tuned fromθpre\\theta\_\{\\mathrm\{pre\}\}on different downstream tasks\. All models share the same architecture and operate within the same parameter spaceℝd\\mathbb\{R\}^\{d\}\.

#### Task Vectors\.

Following the framework of Task Arithmetic \(TA\)\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], we use task vectors to represent task\-specific parameter offsets\. Specifically, the task vector for thett\-th model is defined asτ\(t\)=θ\(t\)−θpre\\tau^\{\(t\)\}=\\theta^\{\(t\)\}\-\\theta\_\{\\mathrm\{pre\}\}, fort∈\{1,⋯,T\}t\\in\\\{1,\\cdots,T\\\}\.

Objective\.Given a pretrained modelθpre\\theta\_\{\\mathrm\{pre\}\}and task vectors\{τ\(t\)\}t=1T\\\{\\tau^\{\(t\)\}\\\}\_\{t=1\}^\{T\}, our goal is to design an aggregation strategy𝒜\\mathcal\{A\}that combinesTTtask vectors into a single merged model, parameterized by:

θmerged=θpre\+𝒜​\(τ\(1\),⋯,τ\(T\)\),\\theta\_\{\\mathrm\{merged\}\}=\\theta\_\{\\mathrm\{pre\}\}\+\\mathcal\{A\}\\big\(\\tau^\{\(1\)\},\\cdots,\\tau^\{\(T\)\}\\big\),\(1\)such that the merged model preserves the task\-specific capabilities of all the fine\-tuned models as much as possible\. We consider model merging without access to the original task\-specific training data, and thus multi\-task learning isn’t applicable\. Instead, we study two standard model merging settings: \(i\) data\-assisted, where a small*unlabeled*calibration set is available to guide the merge, and \(ii\) data\-free, where the merging procedure itself uses no auxiliary data\. Following the seminal work of TA\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], the data\-free merging implies no auxiliary data is used to estimate activation statistics, while a small validation set is still used strictly for hyperparameter tuning\.

## 4Methodology

### 4\.1Model Merging as Bayesian Linear Regression

#### Module\-wise Decomposition\.

While Eq\. \([1](https://arxiv.org/html/2605.12843#S3.E1)\) defines the merging objective at the full parameter spaceℝd\\mathbb\{R\}^\{d\}, deep neural networks are practically composed of multiple distinct modules \(e\.g\., linear projections in self\-attention or MLP blocks\)\. For computational tractability, we decompose the full parameter spaceℝd\\mathbb\{R\}^\{d\}into module\-wise parameter partitions, indexed bym=\{1,⋯,M\}m=\\\{1,\\cdots,M\\\}\. Following the approach of WUDI\-Merging\[[7](https://arxiv.org/html/2605.12843#bib.bib7)\], our aggregation strategy focuses specifically on 2D weight matrices, while keeping all the weights for biases intact\. In the following, we will present our merging method at the module level, and the same method applies to allMMmodules of the network equally\.

Let𝐖pre∈ℝdout×din\\mathbf\{W\}\_\{\\mathrm\{pre\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}be one of theMMpretrained 2D weight matrices\. We define its module\-wise task vector for thett\-th task as the parameter offset𝐔\(t\)=𝐖\(t\)−𝐖pre\\mathbf\{U\}^\{\(t\)\}=\\mathbf\{W\}^\{\(t\)\}\-\\mathbf\{W\}\_\{\\mathrm\{pre\}\}\. Our core objective is to aggregate these task\-specific updates\{𝐔\(t\)\}t=1T\\\{\\mathbf\{U\}^\{\(t\)\}\\\}\_\{t=1\}^\{T\}into a single merged task vector𝐔\\mathbf\{U\}, such that the final merged module is parameterized by

𝐖merged=𝐖pre\+s⋅𝐔,\\mathbf\{W\}\_\{\\text\{merged\}\}=\\mathbf\{W\}\_\{\\text\{pre\}\}\+s\\cdot\\mathbf\{U\},\(2\)wheressis a scaling hyperparameter that is tuned on a validation set\.

#### Activation\-based Regression\.

We frame the estimation of the merged module\-wise task vector𝐔\\mathbf\{U\}as an activation\-based regression problem\. In the data\-assisted setting, we collectNNrepresentative activations for thett\-th task, denoted as\{𝐱n\(t\)\}n=1N\\\{\\mathbf\{x\}\_\{n\}^\{\(t\)\}\\\}\_\{n=1\}^\{N\}, where𝐱n\(t\)∈ℝdin\\mathbf\{x\}\_\{n\}^\{\(t\)\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\}\. They are collected by passing the task\-specific unlabeled calibration data through the corresponding fine\-tuned model\. To preserve the task\-specific capabilities, we aim to align the residual output induced by the merged task vector𝐔\\mathbf\{U\}with that induced by the original task\-specific task vectors\{𝐔\(t\)\}t=1T\\\{\\mathbf\{U\}^\{\(t\)\}\\\}\_\{t=1\}^\{T\}\. Specifically, we define the residual output as𝐲n\(t\)=𝐔\(t\)​𝐱n\(t\)=\(𝐖\(t\)−𝐖pre\)​𝐱n\(t\)\\mathbf\{y\}\_\{n\}^\{\(t\)\}=\\mathbf\{U\}^\{\(t\)\}\\mathbf\{x\}\_\{n\}^\{\(t\)\}=\(\\mathbf\{W\}^\{\(t\)\}\-\\mathbf\{W\}\_\{\\mathrm\{pre\}\}\)\\mathbf\{x\}\_\{n\}^\{\(t\)\}\. We concatenate all activation vectors column\-wise across allTTtasks to construct the activation matrix𝐗\\mathbf\{X\}, and similarly construct the residual output matrix𝐘\\mathbf\{Y\}:

𝐗=\[𝐱1\(1\),…,𝐱N\(1\),…,𝐱1\(T\),…,𝐱N\(T\)\],𝐘=\[𝐲1\(1\),…,𝐲N\(1\),…,𝐲1\(T\),…,𝐲N\(T\)\]\.\\mathbf\{X\}=\\left\[\\mathbf\{x\}\_\{1\}^\{\(1\)\},\\dots,\\mathbf\{x\}\_\{N\}^\{\(1\)\},\\dots,\\mathbf\{x\}\_\{1\}^\{\(T\)\},\\dots,\\mathbf\{x\}\_\{N\}^\{\(T\)\}\\right\],\\quad\\mathbf\{Y\}=\\left\[\\mathbf\{y\}\_\{1\}^\{\(1\)\},\\dots,\\mathbf\{y\}\_\{N\}^\{\(1\)\},\\dots,\\mathbf\{y\}\_\{1\}^\{\(T\)\},\\dots,\\mathbf\{y\}\_\{N\}^\{\(T\)\}\\right\]\.\(3\)We assume a linear observation model where the residual outputs𝐘\\mathbf\{Y\}are generated by the merged module\-wise task vector𝐔\\mathbf\{U\}acting on𝐗\\mathbf\{X\}, corrupted by Gaussian noise𝐄\\mathbf\{E\}:

𝐘=𝐔𝐗\+𝐄,\\mathbf\{Y\}=\\mathbf\{U\}\\mathbf\{X\}\+\\mathbf\{E\},\(4\)where the noise matrix satisfies𝐄:,j∼𝒩​\(0,β−1​𝐈\)\\mathbf\{E\}\_\{:,j\}\\sim\\mathcal\{N\}\(0,\\beta^\{\-1\}\\mathbf\{I\}\)independently for each column indexjj\. Under this linear observation model, the likelihood function of observing𝐘\\mathbf\{Y\}given𝐔\\mathbf\{U\}can be expressed as:

p​\(𝐘∣𝐔,𝐗,β\)∝exp⁡\(−β2​‖𝐘−𝐔𝐗‖F2\)\.p\(\\mathbf\{Y\}\\mid\\mathbf\{U\},\\mathbf\{X\},\\beta\)\\propto\\exp\\left\(\-\\frac\{\\beta\}\{2\}\\\|\\mathbf\{Y\}\-\\mathbf\{U\}\\mathbf\{X\}\\\|\_\{F\}^\{2\}\\right\)\.\(5\)

#### Regularization by Anchors\.

Purely data\-driven estimation of𝐔\\mathbf\{U\}on limited calibration data is not only prone to overfitting but also leaves valuable inductive bias of anchor models \(e\.g\., existing merging solutions\) unused\. To regularize the solution and inject such priors, we introduce a module\-wise*anchor*task vector, defined as𝐔\(0\)=𝐖anchor−𝐖pre\\mathbf\{U\}^\{\(0\)\}=\\mathbf\{W\}\_\{\\mathrm\{anchor\}\}\-\\mathbf\{W\}\_\{\\mathrm\{pre\}\}, where𝐖anchor\\mathbf\{W\}\_\{\\mathrm\{anchor\}\}denotes the module weights obtained from an existing merging solution \(e\.g\., TA\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], TIES\[[4](https://arxiv.org/html/2605.12843#bib.bib4)\], or TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\]\)\. We then impose an element\-wise independent Gaussian prior on𝐔\\mathbf\{U\}centered at anchor task vector𝐔\(0\)\\mathbf\{U\}^\{\(0\)\}:

p​\(𝐔\|𝐔\(0\),α\)∝exp⁡\(−α2​‖𝐔−𝐔\(0\)‖F2\),p\(\\mathbf\{U\}\|\\mathbf\{U\}^\{\(0\)\},\\alpha\)\\propto\\exp\\left\(\-\\frac\{\\alpha\}\{2\}\\\|\\mathbf\{U\}\-\\mathbf\{U\}^\{\(0\)\}\\\|\_\{F\}^\{2\}\\right\),\(6\)whereα\\alphacontrols the precision of this anchor\-induced prior\. Given these formulations, the*maximum a posteriori*\(MAP\) estimate of𝐔\\mathbf\{U\}is obtained by maximizing the posteriorp​\(𝐔\|𝐘,𝐗\)∝p​\(𝐘\|𝐔,𝐗,β\)​p​\(𝐔\|𝐔\(0\),α\)p\(\\mathbf\{U\}\|\\mathbf\{Y\},\\mathbf\{X\}\)\\propto p\(\\mathbf\{Y\}\|\\mathbf\{U\},\\mathbf\{X\},\\beta\)\\,p\(\\mathbf\{U\}\|\\mathbf\{U\}^\{\(0\)\},\\alpha\)\. Taking the negative logarithm transforms posterior maximization into a minimization problem\. Let the regularization weightλ=αβ≥0\\lambda\\\!=\\\!\\frac\{\\alpha\}\{\\beta\}\\geq 0, the MAP objective simplifies to:

𝐔MAP=arg⁡min𝐔⁡\(‖𝐘−𝐔𝐗‖F2\+λ​‖𝐔−𝐔\(0\)‖F2\),\\displaystyle\\mathbf\{U\}\_\{\\text\{MAP\}\}=\\arg\\min\_\{\\mathbf\{U\}\}\\left\(\\\|\\mathbf\{Y\}\-\\mathbf\{U\}\\mathbf\{X\}\\\|\_\{F\}^\{2\}\+\\lambda\\\|\\mathbf\{U\}\-\\mathbf\{U\}^\{\(0\)\}\\\|\_\{F\}^\{2\}\\right\),\(7\)whereλ\\lambdabalances the empirical fit on the task\-specific activations \(the first term\) with the prior knowledge encapsulated by the anchor model \(the second term\)\. Let the derivative of the objective above w\.r\.t\.𝐔\\mathbf\{U\}to be zero, we derive a closed\-form solution for the optimal module\-wise task vector:

𝐔MAP=\(𝐘𝐗⊤\+λ​𝐔\(0\)\)​\(𝐗𝐗⊤\+λ​𝐈\)−1,\\mathbf\{U\}\_\{\\text\{MAP\}\}=\\left\(\\mathbf\{Y\}\\mathbf\{X\}^\{\\top\}\+\\lambda\\mathbf\{U\}^\{\(0\)\}\\right\)\\left\(\\mathbf\{X\}\\mathbf\{X\}^\{\\top\}\+\\lambda\\mathbf\{I\}\\right\)^\{\-1\},\(8\)where𝐈∈ℝdin×din\\mathbf\{I\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times d\_\{\\mathrm\{in\}\}\}is the identity matrix\. Given aλ≥0\\lambda\\geq 0, this closed\-form solution allows us to compute the optimal merged task vectors for allMMmodules efficiently111The MAP point estimate of𝐔\\mathbf\{U\}is used mainly for the purpose of computational efficiency\. Note that the covariance of the Gaussian posterior also has a closed\-form solution, enabling us to sample from the posterior effectively\. The corresponding sampling\-based BMM is presented in Appendix[B](https://arxiv.org/html/2605.12843#A2), which demonstrates improved uncertainty calibration\.\.

𝝀\\bm\{\\lambda\}𝐔m\\mathbf\{U\}\_\{m\}𝐔m\(0\)\\mathbf\{U\}\_\{m\}^\{\(0\)\}𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}𝐘m\\mathbf\{Y\}\_\{m\}𝐗m\\mathbf\{X\}\_\{m\}\{𝐔m\(t\)\}\\\{\\mathbf\{U\}\_\{m\}^\{\(t\)\}\\\}m=\{1,⋯,M\}m=\\\{1,\\cdots,M\\\}data\-assisteddata\-freePredictive Evidence& BO update

![[Uncaptioned image]](https://arxiv.org/html/2605.12843v1/figures/vit_llama_task_variation_fancy.png)

Figure 1:Probabilistic formulation of BMM\.The framework can adopt different observation sources: empirical activations \(data\-assisted\) or expert weight\-induced surrogates \(data\-free\)\. The inner MAP estimate is solved in closed\-form, while the outer loop optimizes𝝀\\bm\{\\lambda\}by BO\.Figure 2:Alignment between activation statistics and task\-vectorsby empirical verification of Theorem[1](https://arxiv.org/html/2605.12843#Thmtheorem1)\. The consistent positive cosine similarity scores across vision and language backbones corroborates the proposed alignment between activation statistics and task\-vectors\.

### 4\.2From Predictive Evidence to Bayesian Optimization

Eq\. \([8](https://arxiv.org/html/2605.12843#S4.E8)\) treats allMMmodules of the network in isolation for module\-wise linear regression\. As neural network modules are highly coupled, we need jointly coordinate module\-wise regularization strengths \(i\.e\.,λ\\lambda’s\) to maximize the end\-to\-end performance of the full network\. We cast this global coordination as a Bayesian model selection problem\. Specifically, we evaluate the predictive evidence of merged model on a held\-out validation set𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}222This is the same validation set used by other merging methods \(e\.g\., TA\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], RegMean\[[6](https://arxiv.org/html/2605.12843#bib.bib6)\], and TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\]\) for hyperparameter selection\. Therefore, BMM uses no extra data for model merging and represents a fair comparison\., conditioned on the local activation data𝒟act=\{𝐗m,𝐘m\}m=1M\\mathcal\{D\}\_\{\\text\{act\}\}=\\\{\\mathbf\{X\}\_\{m\},\\mathbf\{Y\}\_\{m\}\\\}\_\{m=1\}^\{M\}for data\-assisted merging\. In the framework of Bayesian model selection, a natural objective is to find𝝀⋆\\bm\{\\lambda\}^\{\\star\}that maximizes the held\-out predictive evidence:

𝝀⋆=arg⁡max𝝀≥𝟎⁡p​\(𝒟val∣𝒟act,𝒰\(0\),𝝀\)\.\\bm\{\\lambda\}^\{\\star\}=\\arg\\max\_\{\\bm\{\\lambda\}\\geq\\mathbf\{0\}\}p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid\\mathcal\{D\}\_\{\\mathrm\{act\}\},\\mathcal\{U\}^\{\(0\)\},\\bm\{\\lambda\}\)\.\(9\)Let𝒰=\{𝐔m\}m=1M\\mathcal\{U\}=\\\{\\mathbf\{U\}\_\{m\}\\\}\_\{m=1\}^\{M\}be latent variables, the predictive evidence can be expressed as

p​\(𝒟val∣𝒟act,𝒰\(0\),𝝀\)=∫p​\(𝒟val∣𝒰\)​p​\(𝒰∣𝒟act,𝒰\(0\),𝝀\)​𝑑𝒰\.p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid\\mathcal\{D\}\_\{\\mathrm\{act\}\},\\mathcal\{U\}^\{\(0\)\},\\bm\{\\lambda\}\)=\\int p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid\\mathcal\{U\}\)\\,p\(\\mathcal\{U\}\\mid\\mathcal\{D\}\_\{\\mathrm\{act\}\},\\mathcal\{U\}^\{\(0\)\},\\bm\{\\lambda\}\)\\,d\\mathcal\{U\}\.\(10\)Eqs\. \([9](https://arxiv.org/html/2605.12843#S4.E9)\) and \([10](https://arxiv.org/html/2605.12843#S4.E10)\) formalizes a global optimization strategy: the optimal𝝀⋆\\bm\{\\lambda\}^\{\\star\}is determined by maximizing the predictive evidence, effectively accounting for the uncertainty of latent task vectors𝒰\\mathcal\{U\}through marginalization\.

Algorithm 1Bayesian Bi\-level Optimization for Model Merging1:Anchors

𝒰\(0\)=\{𝐔m\(0\)\}m=1M\\mathcal\{U\}^\{\(0\)\}=\\\{\\mathbf\{U\}\_\{m\}^\{\(0\)\}\\\}\_\{m=1\}^\{M\}, Activation data

\{𝐗m,𝐘m\}m=1M\\\{\\mathbf\{X\}\_\{m\},\\mathbf\{Y\}\_\{m\}\\\}\_\{m=1\}^\{M\}, Val set

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}, \#Iters

KK
2:Merged module\-wise task vectors

𝒰⋆\\mathcal\{U\}^\{\\star\}
3:Initialize evaluation history

ℋ←∅\\mathcal\{H\}\\leftarrow\\emptysetand

𝒢​𝒫\\mathcal\{GP\}surrogate

ℳ\\mathcal\{M\}
4:for

k=1,2,…,Kk=1,2,\\dots,Kdo

5:\# Outer Loop: Surrogate\-Guided Proposal

6:

𝝀\(k\)←arg⁡max𝝀⁡EI​\(𝝀;ℳ\)\\bm\{\\lambda\}^\{\(k\)\}\\leftarrow\\arg\\max\_\{\\bm\{\\lambda\}\}\\text\{EI\}\(\\bm\{\\lambda\};\\mathcal\{M\}\)⊳\\trianglerightMaximize acquisition function

7:\# Inner Loop: Closed\-form MAP solution

8:foreach module

m∈\{1,…,M\}m\\in\\\{1,\\dots,M\\\}do

9:

𝐔m⋆←\(𝐘m​𝐗m⊤\+λm\(k\)​𝐔m\(0\)\)​\(𝐗m​𝐗m⊤\+λm\(k\)​𝐈\)−1\\mathbf\{U\}^\{\\star\}\_\{m\}\\leftarrow\\left\(\\mathbf\{Y\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}\+\\lambda\_\{m\}^\{\(k\)\}\\mathbf\{U\}\_\{m\}^\{\(0\)\}\\right\)\\left\(\\mathbf\{X\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}\+\\lambda\_\{m\}^\{\(k\)\}\\mathbf\{I\}\\right\)^\{\-1\}⊳\\trianglerightSolve inner MAP via Eq\. \([8](https://arxiv.org/html/2605.12843#S4.E8)\)

10:endfor

11:Assemble full merged model

𝒰\(k\)⋆←\{𝐔m⋆\}m=1M\\mathcal\{U\}^\{\\star\}\_\{\(k\)\}\\leftarrow\\\{\\mathbf\{U\}^\{\\star\}\_\{m\}\\\}\_\{m=1\}^\{M\}
12:\# Outer Loop: Validation evaluation

13:

f\(k\)←Score​\(𝒰\(k\)⋆,𝒟val\)f^\{\(k\)\}\\leftarrow\\mathrm\{Score\}\(\\mathcal\{U\}^\{\\star\}\_\{\(k\)\},\\mathcal\{D\}\_\{\\text\{val\}\}\)
14:

ℋ←ℋ∪\{\(𝝀\(k\),f\(k\)\)\}\\mathcal\{H\}\\leftarrow\\mathcal\{H\}\\cup\\\{\(\\bm\{\\lambda\}^\{\(k\)\},f^\{\(k\)\}\)\\\}; Update surrogate

ℳ\\mathcal\{M\}
15:endfor

16:

𝝀⋆←arg⁡max\(𝝀,f\)∈ℋ⁡f\\bm\{\\lambda\}^\{\\star\}\\leftarrow\\arg\\max\_\{\(\\bm\{\\lambda\},f\)\\in\\mathcal\{H\}\}\\;f⊳\\trianglerightExtract global optimal configuration

17:return

𝒰⋆​\(𝝀⋆\)\\mathcal\{U\}^\{\\star\}\(\\bm\{\\lambda\}^\{\\star\}\)

To reconcile this Bayesian model selection with practical constraints, we introduce two key refinements\. First, to align the objective with the downstream task performances, we replace the intractable likelihoodp​\(𝒟val∣𝒰\)p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid\\mathcal\{U\}\)with a validation utilityScore​\(⋅\)\\text\{Score\}\(\\cdot\), which is the average accuracy of merged model on validation sets acrossTTtasks:

p​\(𝒟val∣𝒰\)∝Score​\(𝒰,𝒟val\)=1T​∑t=1TAcct​\(θmerged​\(𝒰\),𝒟val\(t\)\),p\(\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\mid\\mathcal\{U\}\)\\propto\\text\{Score\}\(\\mathcal\{U\},\\mathcal\{D\}\_\{\\text\{val\}\}\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\text\{Acc\}\_\{t\}\(\\theta\_\{\\text\{merged\}\}\(\\mathcal\{U\}\),\\mathcal\{D\}\_\{\\text\{val\}\}^\{\(t\)\}\),\(11\)whereθmerged​\(𝒰\)\\theta\_\{\\text\{merged\}\}\(\\mathcal\{U\}\)reconstructs the weights of merged model via Eq\. \([2](https://arxiv.org/html/2605.12843#S4.E2)\)\. Second, we adopt an empirical\-Bayes approximation to bypass the integration in Eq\. \([10](https://arxiv.org/html/2605.12843#S4.E10)\), which is in general prohibitive for deep networks\. Specifically, by substituting the full posteriorp​\(𝒰∣𝒟act,𝒰\(0\),𝝀\)p\(\\mathcal\{U\}\\mid\\mathcal\{D\}\_\{\\mathrm\{act\}\},\\mathcal\{U\}^\{\(0\)\},\\bm\{\\lambda\}\)with a point estimate𝒰⋆​\(𝝀\)\\mathcal\{U\}^\{\\star\}\(\\bm\{\\lambda\}\), we transform the marginalization into a bi\-level optimization problem:

𝝀⋆=arg⁡max𝝀≥𝟎⁡Score​\(𝒰⋆​\(𝝀\),𝒟val\),\\bm\{\\lambda\}^\{\\star\}=\\arg\\max\_\{\\bm\{\\lambda\}\\geq\\mathbf\{0\}\}\\mathrm\{Score\}\\big\(\\mathcal\{U\}^\{\\star\}\(\\bm\{\\lambda\}\),\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\big\),\(12\)subject to

𝒰⋆​\(𝝀\)=arg⁡max𝒰⁡p​\(𝒰∣𝒟act,𝒰\(0\),𝝀\)\.\\mathcal\{U\}^\{\\star\}\(\\bm\{\\lambda\}\)=\\arg\\max\_\{\\mathcal\{U\}\}p\(\\mathcal\{U\}\\mid\\mathcal\{D\}\_\{\\mathrm\{act\}\},\\mathcal\{U\}^\{\(0\)\},\\bm\{\\lambda\}\)\.\(13\)The inner optimization in Eq\. \([13](https://arxiv.org/html/2605.12843#S4.E13)\) admits a closed\-form solution via Eq\. \([8](https://arxiv.org/html/2605.12843#S4.E8)\)\. Therefore, the remaining task is to optimize the outer objective in Eq\. \([12](https://arxiv.org/html/2605.12843#S4.E12)\)\. We treat this outer objective as a black\-box function of𝝀\\bm\{\\lambda\}, where each evaluation requires reconstructing a full merged model and measuring its validation performance\. For optimization efficiency, we adopt a Gaussian process \(𝒢​𝒫\\mathcal\{GP\}\)\-based Bayesian Optimization \(BO\)\[[21](https://arxiv.org/html/2605.12843#bib.bib21)\]\. At iterationkk, given the evaluation historyℋk−1=\{\(𝝀\(i\),f\(i\)\)\}i=1k−1\\mathcal\{H\}\_\{k\-1\}=\\\{\(\\bm\{\\lambda\}^\{\(i\)\},f^\{\(i\)\}\)\\\}\_\{i=1\}^\{k\-1\}, wheref\(i\)=Score​\(U∗​\(𝝀\(i\)\),𝒟val\)f^\{\(i\)\}=\\text\{Score\}\(U^\{\*\}\(\\bm\{\\lambda\}^\{\(i\)\}\),\\mathcal\{D\}\_\{\\text\{val\}\}\), we fit a𝒢​𝒫\\mathcal\{GP\}surrogate over𝝀\\bm\{\\lambda\}\. We then select the next candidate𝝀\(k\)\\bm\{\\lambda\}^\{\(k\)\}by maximizing an acquisition function, e\.g\., Expected Improvement \(EI\), which balances exploitation \(high predicted utility\) and exploration \(high uncertainty\)\. For the selected candidate𝝀\(k\)\\bm\{\\lambda\}^\{\(k\)\}, we solve the corresponding inner MAP problem, evaluate its validation scoref\(k\)f^\{\(k\)\}, and update the evaluation history toℋk\\mathcal\{H\}\_\{k\}\. The best configuration observed during the search is returned as𝝀∗\\bm\{\\lambda\}^\{\*\}\. This Bayesian bi\-level optimization procedure is summarized in Algorithm[1](https://arxiv.org/html/2605.12843#alg1)\.

The naïve decomposition of full model parameters intoMMindependent weight matrices poses a severe computational bottleneck for𝒢​𝒫\\mathcal\{GP\}\-based BO \(e\.g\.,M=96M\\\!=\\\!96for ViT\-L/14 andM=196M\\\!=\\\!196for Llama\-3\.1\-8B\)\. To reconcile search efficiency with architectural expressiveness, we propose a block\-wise parameter tying strategy\. Specifically, we partition the network’s consecutive Transformer layers intoBBsequential blocks\. Within each block, modules sharing identical functional roles are tied into four module groups: attention\-in \(Q/K/V\), attention\-out, MLP\-in, and MLP\-out\. Combined with a block\-specific scaling factorss\(Eq\.[2](https://arxiv.org/html/2605.12843#S4.E2)\), this reduces the parameterization to a compact 5\-dimensional search subspace per block, yielding a total BO search space of5​B5Bdimensions\. This architecture\-aware module decomposition ensures that the merging run\-time of BMM remains competitive with state\-of\-the\-art methods, such as TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\], WUDI\-Merging\[[7](https://arxiv.org/html/2605.12843#bib.bib7)\], and ISO\-CTS\[[9](https://arxiv.org/html/2605.12843#bib.bib9)\]\.

### 4\.3From Data\-Assisted to Data\-Free

The MAP estimate of𝐔\\mathbf\{U\}in Eq\. \([8](https://arxiv.org/html/2605.12843#S4.E8)\) performs effectively in the data\-assisted setting, where a small unlabeled calibration set is available to assess𝐗\\mathbf\{X\}and𝐘\\mathbf\{Y\}for merging\. However, in many practical scenarios, such a calibration set is often unavailable due to privacy or storage constraints\. To support this data\-free setting, we investigate and reveal a key alignment between activation statistics and task vectors based on recent work of representation geometry and neural collapse\[[14](https://arxiv.org/html/2605.12843#bib.bib14),[18](https://arxiv.org/html/2605.12843#bib.bib18),[17](https://arxiv.org/html/2605.12843#bib.bib17)\], which allows us to derive a data\-free variant of BMM\. In the following, we will present our analysis at the module level, and the same analysis applies to allMMmodules of the network equally\.

#### Alignment between Activation Statistics and Task Vectors\.

Let𝐱\\mathbf\{x\}denote an input activation to module𝐖\(t\)\\mathbf\{W\}^\{\(t\)\}and𝐲=𝐔\(t\)​𝐱=\(𝐖\(t\)−𝐖pre\)​𝐱\\mathbf\{y\}=\\mathbf\{U\}^\{\(t\)\}\\mathbf\{x\}=\(\\mathbf\{W\}^\{\(t\)\}\-\\mathbf\{W\}\_\{\\mathrm\{pre\}\}\)\\mathbf\{x\}the corresponding residual output\. Under a set of mild assumptions, the following theorem holds, with the proof provided in Appendix[A](https://arxiv.org/html/2605.12843#A1)\.

###### Theorem 1\.

Under Assumption 1, the Gram matrix of input activations and the Gram matrix of task\-vectors have a positively correlated alignment:

cosF⁡\(𝔼​\[𝐱𝐱⊤\],\(𝐔\(t\)\)⊤​𝐔\(t\)\)\>αt,\\operatorname\{cos\}\_\{F\}\\\!\\left\(\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\],\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\\right\)\>\\alpha\_\{t\},\(14\)whereαt∈\(0,1\]\\alpha\_\{t\}\\in\(0,1\]is the alignment constant in Assumption 1, andcosF⁡\(𝐀,𝐁\)=Tr⁡\(𝐀⊤​𝐁\)/\(‖𝐀‖F​‖𝐁‖F\)\\operatorname\{cos\}\_\{F\}\(\\mathbf\{A\},\\mathbf\{B\}\)=\\operatorname\{Tr\}\(\\mathbf\{A\}^\{\\top\}\\mathbf\{B\}\)/\(\\\|\\mathbf\{A\}\\\|\_\{F\}\\\|\\mathbf\{B\}\\\|\_\{F\}\)is the Frobenius cosine\-similarity\.

To empirically validate Theorem[1](https://arxiv.org/html/2605.12843#Thmtheorem1), we first estimate𝔼​\[𝐱𝐱⊤\]\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\]by its empirical mean,𝔼​\[𝐱𝐱⊤\]≈1N​𝐗\(t\)​\(𝐗\(t\)\)⊤\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\]\\approx\\frac\{1\}\{N\}\\mathbf\{X\}^\{\(t\)\}\(\\\!\\mathbf\{X\}^\{\(t\)\}\\\!\)^\{\\top\}, where𝐗\(t\)=\[𝐱1\(t\),…,𝐱N\(t\)\]\\mathbf\{X\}^\{\(t\)\}\\\!=\\\!\\left\[\\\!\\mathbf\{x\}\_\{1\}^\{\(t\)\}\\\!,\\dots,\\mathbf\{x\}\_\{N\}^\{\(t\)\}\\\!\\right\]is collected the same as in Eq\.[3](https://arxiv.org/html/2605.12843#S4.E3)fromNNcalibration samples\. We quantify the correlation between𝐀=1N​𝐗\(t\)​\(𝐗\(t\)\)⊤\\mathbf\{A\}\\\!=\\\!\\frac\{1\}\{N\}\\mathbf\{X\}^\{\(t\)\}\(\\\!\\mathbf\{X\}^\{\(t\)\}\\\!\)^\{\\top\}and𝐁=\(𝐔\(t\)\)⊤​𝐔\(t\)\\mathbf\{B\}=\(\\\!\\mathbf\{U\}^\{\(t\)\}\\\!\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}by using the Frobenius cosine\-similarity:cosF⁡\(𝐀,𝐁\)\\cos\_\{F\}\(\\mathbf\{A\},\\mathbf\{B\}\)\. Figure[2](https://arxiv.org/html/2605.12843#S4.F2)reports the task\-level cosine similarity scores and its means on the 8\-task vision and 5\-task language benchmarks across four different backbone architectures\. The positive correlation scores \(\>0\.26\) consistently across four benchmarks empirically corroborate our theoretical analysis\.

Data\-Free MAP Estimate of𝐔\\mathbf\{U\}\.According to Theorem[1](https://arxiv.org/html/2605.12843#Thmtheorem1), we further approximate𝐗\(t\)​\(𝐗\(t\)\)⊤≈c​\(𝐔\(t\)\)⊤​𝐔\(t\)\\mathbf\{X\}^\{\(t\)\}\(\\mathbf\{X\}^\{\(t\)\}\)^\{\\top\}\\approx c\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}, wherec\>0c\>0is a positive scalar thanks to the positive correlation between activation statistics and task vectors\. Along with the forward relation𝐘\(t\)=𝐔\(t\)​𝐗\(t\)\\mathbf\{Y\}^\{\(t\)\}=\\mathbf\{U\}^\{\(t\)\}\\mathbf\{X\}^\{\(t\)\}, we can derive a data\-free MAP estimate of𝐔\\mathbf\{U\}by inserting them into Eq\. \([8](https://arxiv.org/html/2605.12843#S4.E8)\), which yields:

𝐔MAP∗​\(λ~\)=\(∑t=1T𝐔\(t\)​\(𝐔\(t\)\)⊤​𝐔\(t\)\+λ~​𝐔\(0\)\)​\(∑t=1T\(𝐔\(t\)\)⊤​𝐔\(t\)\+λ~​𝐈\)−1,\\mathbf\{U\}\_\{\\text\{MAP\}\}^\{\*\}\(\\tilde\{\\lambda\}\)=\\left\(\\sum\_\{t=1\}^\{T\}\\mathbf\{U\}^\{\(t\)\}\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\+\\tilde\{\\lambda\}\\mathbf\{U\}^\{\(0\)\}\\right\)\\left\(\\sum\_\{t=1\}^\{T\}\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\+\\tilde\{\\lambda\}\\mathbf\{I\}\\right\)^\{\-1\},\(15\)withλ~=λ/c\\tilde\{\\lambda\}=\\lambda/c\. This estimate bypasses the need of a calibration set to assess the local activation data\{𝐗,𝐘\}\\\{\\mathbf\{X\},\\mathbf\{Y\}\\\}for model merging, and the merged task vector𝐔\\mathbf\{U\}depends merely on the task\-specific task vectors\{𝐔\(t\)\}t=1T\\\{\\mathbf\{U\}^\{\(t\)\}\\\}\_\{t=1\}^\{T\}\. Similarly,λ~\\tilde\{\\lambda\}can be included in𝝀\\bm\{\\lambda\}and tuned by BO as shown in Algorithm[1](https://arxiv.org/html/2605.12843#alg1)\.

## 5Experiments

We evaluate BMM for both vision tasks and language tasks across four backbone architectures with seven different anchor models, which are also the baseline models for performance comparison\. Ablation studies are conducted to further validate the main components and design choices of BMM\. All our experiments are conducted on a server equipped with 8 NVIDIA RTX\-6000 48GB GPUs\.

### 5\.1Experimental Settings

#### Datasets and Models\.

Forvisiontasks, we follow the scalability evaluation settings of\[[8](https://arxiv.org/html/2605.12843#bib.bib8),[9](https://arxiv.org/html/2605.12843#bib.bib9)\], covering 8\-, 14\-, and 20\-task merging scenarios with ViT\-B/32 and ViT\-L/14 backbones\. Forlanguagetasks, following\[[22](https://arxiv.org/html/2605.12843#bib.bib22)\], we evaluate BMM for 5\-task merging based on Llama\-3\.2\-3B and Llama\-3\.1\-8B\. To avoid data leakage, we keep the training, validation, and test splits disjoint across tasks\. Validation data are used only for hyperparameter search, and test data are used exclusively for final evaluation\. Full benchmark descriptions, metrics, and asset/license info are provided in Appendix[C](https://arxiv.org/html/2605.12843#A3)\.

#### Baselines\.

For performance comparison, we choose six model merging methods, including RegMean\[[6](https://arxiv.org/html/2605.12843#bib.bib6)\], a data\-assisted method that relies on an unlabeled calibration set to guide the merging, and five data\-free merging methods: TA\[[3](https://arxiv.org/html/2605.12843#bib.bib3)\], TIES\[[4](https://arxiv.org/html/2605.12843#bib.bib4)\], TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\], WUDI\-Merging\[[7](https://arxiv.org/html/2605.12843#bib.bib7)\], and ISO\-CTS\[[9](https://arxiv.org/html/2605.12843#bib.bib9)\]\. The generated models are also used as the anchors that serve as the prior knowledge for BMM\. Additionally, we consider an anchor model that is the original pre\-trained backbone, such that𝐔\(0\)=𝟎\\mathbf\{U\}^\{\(0\)\}=\\mathbf\{0\}, indicating no prior knowledge is used for BMM\. We compare BMM with all seven aforementioned anchors as well as the individual fine\-tuned models for performance comparison\.

#### Experiment Details\.

Following RegMean\[[6](https://arxiv.org/html/2605.12843#bib.bib6)\], our data\-assisted BMM collects local activation data𝒟act=\{𝐗m,𝐘m\}m=1M\\mathcal\{D\}\_\{\\text\{act\}\}=\\\{\\mathbf\{X\}\_\{m\},\\mathbf\{Y\}\_\{m\}\\\}\_\{m=1\}^\{M\}using 128 and 1,000 samples per task for ViT and Llama, respectively\. We partition the networks intoB=3B=3\(ViT\) andB=1B=1\(Llama\) sequential blocks\. Each block has 5 hyperparameters to tune: one scales∈\[1\.0,1\.3\]s\\in\[1\.0,1\.3\]\(Eq\.[2](https://arxiv.org/html/2605.12843#S4.E2)\) and four group\-wise regularization strengthsλ\\lambda’s \(attention\-in/out and MLP\-in/out\)\. We sample eachλ\\lambdain the log\-scale uniformly from\[10−4,1\]\[10^\{\-4\},1\]for ViT models and\[10−3,100\]\[10^\{\-3\},100\]for Llama models\. With BO budgets of merelyK=200K\\\!=\\\!200\(ViT\) andK=100K\\\!=\\\!100\(Llama\) trials, optimizing the full 15D/5D spaces requires fewer validation evaluations than a uniform15×1515\\times 15grid search in 2D space \(225 trials\), ensuring computational efficiency\. Additional complexity analysis and wall\-clock runtime breakdowns are provided in Appendix[D](https://arxiv.org/html/2605.12843#A4)\.

Table 1:Model merging on vision tasks with ViT architectures\. The blue subscript values indicate therelative percentage improvementcompared to the correspondinganchor\. \(Indiv\.: Individual\)ModelTasksSettingIndiv\.PretrainedTATIESRegMeanTSVWUDIISO\-CTSViT\-B/328anchor92\.847\.770\.475\.782\.385\.987\.086\.4data\-assisted\-84\.8\+77\.885\.0\+20\.785\.7\+13\.185\.2\+3\.588\.9\+3\.589\.4\+2\.890\.2\+4\.3data\-free\-87\.9\+84\.487\.0\+23\.687\.0\+14\.987\.9\+6\.887\.8\+2\.287\.6\+0\.789\.0\+3\.014anchor90\.956\.965\.268\.276\.679\.980\.581\.5data\-assisted\-78\.9\+38\.879\.2\+21\.579\.7\+16\.879\.1\+3\.284\.3\+5\.584\.6\+5\.185\.5\+4\.9data\-free\-82\.7\+45\.482\.4\+26\.482\.3\+20\.682\.9\+8\.282\.8\+3\.781\.7\+1\.584\.3\+3\.520anchor91\.355\.760\.464\.072\.276\.976\.177\.6data\-assisted\-75\.6\+35\.775\.5\+25\.076\.3\+19\.275\.9\+5\.082\.0\+6\.782\.2\+8\.082\.8\+6\.6data\-free\-80\.0\+43\.578\.4\+29\.879\.2\+23\.779\.7\+10\.479\.7\+3\.778\.0\+2\.681\.5\+5\.0ViT\-L/148anchor95\.865\.184\.887\.090\.093\.094\.094\.8data\-assisted\-92\.3\+41\.792\.8\+9\.493\.1\+7\.192\.7\+2\.994\.4\+1\.694\.7\+0\.795\.1\+0\.3data\-free\-94\.4\+44\.994\.2\+11\.094\.2\+8\.394\.0\+4\.594\.4\+1\.594\.3\+0\.395\.0\+0\.214anchor94\.368\.579\.480\.285\.589\.190\.591\.0data\-assisted\-88\.0\+28\.588\.3\+11\.288\.3\+10\.188\.2\+3\.291\.6\+2\.791\.7\+1\.392\.2\+1\.4data\-free\-90\.9\+32\.890\.9\+14\.590\.8\+13\.390\.6\+6\.091\.3\+2\.491\.0\+0\.592\.1\+1\.220anchor94\.765\.474\.076\.782\.887\.788\.490\.1data\-assisted\-86\.0\+31\.586\.2\+16\.586\.4\+12\.686\.2\+4\.190\.9\+3\.690\.8\+2\.791\.5\+1\.5data\-free\-89\.8\+37\.389\.6\+21\.189\.5\+16\.689\.4\+7\.990\.3\+3\.089\.7\+1\.591\.1\+1\.1data\-assisted\-\+42\.3\+17\.4\+13\.1\+3\.7\+3\.9\+3\.4\+3\.2Avg\. Improvementdata\-free\-\+48\.0\+21\.1\+16\.2\+7\.3\+2\.7\+1\.2\+2\.3

Table 2:Model merging on language tasks with Llama architectures\. The blue subscript values indicate therelative percentage improvementcompared to the correspondinganchor\.ModelSettingIndiv\.PretrainedTATIESRegMeanTSVWUDIISO\-CTSLlama\-3\.2\-3Banchor0\.4990\.3010\.4380\.4350\.3770\.4710\.4610\.441data\-assisted\-0\.302\+0\.30\.448\+2\.30\.447\+2\.80\.406\+7\.70\.478\+1\.50\.479\+3\.90\.454\+2\.9data\-free\-0\.303\+0\.70\.455\+3\.90\.468\+7\.60\.456\+21\.00\.486\+3\.20\.478\+3\.70\.457\+3\.6Llama\-3\.1\-8Banchor0\.6300\.3850\.5410\.5540\.5260\.5570\.5600\.556data\-assisted\-0\.387\+0\.50\.568\+5\.00\.579\+4\.50\.530\+0\.80\.574\+3\.10\.564\+0\.70\.576\+3\.6data\-free\-0\.387\+0\.50\.568\+5\.00\.558\+0\.70\.528\+0\.40\.573\+2\.90\.561\+0\.20\.564\+1\.4data\-assisted\-\+0\.4\+3\.6\+3\.6\+4\.2\+2\.3\+2\.3\+3\.3Avg\. Improvementdata\-free\-\+0\.6\+4\.4\+4\.2\+10\.7\+3\.0\+1\.9\+2\.5

### 5\.2Main Results

#### Vision Benchmarks \(ViT\)\.

Table[1](https://arxiv.org/html/2605.12843#S5.T1)reports the performance of BMM on vision tasks with ViT architectures\. As can be seen, BMM is an effective*plug\-and\-play*merging method\. Across all evaluated anchors, model capacities, and task scales, BMM yields noticeable improvements in both data\-assisted and data\-free settings\. The gains are especially large over weaker anchors, such as TA and TIES, with average relative improvements of21\.1%21\.1\\%and16\.2%16\.2\\%, respectively\. At the same time, BMM also consistently outperforms strong recent anchors such as WUDI\-Merging and ISO\-CTS\.

In addition, BMM remains effective as the number of merged tasks increases\. For example, in the 20\-task setting with ViT\-B/32, BMM with ISO\-CTS as anchor improves the merging performance from77\.6%77\.6\\%to82\.8%82\.8\\%in the data\-assisted setting and to81\.5%81\.5\\%in the data\-free setting\. Similar gains are also observed on ViT\-L/14, suggesting that BMM can effectively alleviate task interference in larger\-scale merging settings\.

Finally, BMM brings the performance of model merging close to that of individual task\-specific experts\. On the ViT\-L/14 8\-task benchmark, BMM with ISO\-CTS as anchor achieves95\.1%95\.1\\%in the data\-assisted setting and95\.0%95\.0\\%in the data\-free setting, closely matching the average performance \(95\.8%95\.8\\%\) of the eight individual fine\-tuned models\. Detailed experimental results of mean±\\pmstd over five random seeds and the per\-task breakdowns are provided in Appendices[E\.1](https://arxiv.org/html/2605.12843#A5.SS1)and[E\.2](https://arxiv.org/html/2605.12843#A5.SS2), respectively\.

#### Language Benchmarks \(Llama\)\.

Similar to the vision benchmarks, Table[2](https://arxiv.org/html/2605.12843#S5.T2)shows that BMM consistently improves performance across all anchor models on the 5\-task language benchmarks\. The gains are especially pronounced for weaker anchors, with the largest improvement reaching21\.0%21\.0\\%for RegMean on Llama\-3\.2\-3B in the data\-free setting\. Importantly, BMM also pushes already competitive anchors to stronger results\. For instance, in the data\-free setting, BMM improves the TSV anchor from0\.4710\.471to0\.4860\.486on Llama\-3\.2\-3B, achieving the best 3B result among all the merging methods\. On Llama\-3\.1\-8B, BMM improves the TSV anchor from0\.5570\.557to0\.5730\.573, while the data\-assisted BMM with TIES reaches the overall best 8B performance of0\.5790\.579\. Interestingly, the data\-free BMM is particularly strong on the 3B model and often outperforms its data\-assisted counterpart, suggesting that the proposed alignment\-based approximation can generalize effectively even without auxiliary calibration data\. Category\-level language breakdowns are provided in Appendix[E\.2](https://arxiv.org/html/2605.12843#A5.SS2)\.

Table 3:Different hyperparameter search methods \(shared\-λ\\lambda,random search, andBO\) and their impacts on 8\-, 14\-, and 20\-task merging with ViT\-B/32 architecture\. Results for data\-assisted and data\-free variants are reported as mean±\\pmstd over seeds 0–4\.SettingVariant8 Tasks14 Tasks20 TasksTSVWUDIISO\-CTSTSVWUDIISO\-CTSTSVWUDIISO\-CTSanchor\-85\.987\.086\.479\.980\.581\.576\.976\.177\.6data\-assistedshared\-λ\\lambda88\.1±0\.0188\.1\_\{\\pm 0\.01\}89\.0±0\.0189\.0\_\{\\pm 0\.01\}89\.3±0\.0189\.3\_\{\\pm 0\.01\}83\.2±0\.0083\.2\_\{\\pm 0\.00\}83\.5±0\.0083\.5\_\{\\pm 0\.00\}83\.9±0\.0083\.9\_\{\\pm 0\.00\}80\.1±0\.0180\.1\_\{\\pm 0\.01\}80\.6±0\.0180\.6\_\{\\pm 0\.01\}80\.3±0\.0280\.3\_\{\\pm 0\.02\}random88\.6±0\.0988\.6\_\{\\pm 0\.09\}89\.2±0\.0289\.2\_\{\\pm 0\.02\}89\.7±0\.2489\.7\_\{\\pm 0\.24\}83\.7±0\.1183\.7\_\{\\pm 0\.11\}83\.9±0\.3083\.9\_\{\\pm 0\.30\}84\.9±0\.0784\.9\_\{\\pm 0\.07\}80\.6±0\.1880\.6\_\{\\pm 0\.18\}81\.2±0\.3281\.2\_\{\\pm 0\.32\}81\.7±0\.3781\.7\_\{\\pm 0\.37\}BO88\.9±0\.05\\mathbf\{88\.9\}\_\{\\pm\\mathbf\{0\.05\}\}89\.4±0\.02\\mathbf\{89\.4\}\_\{\\pm\\mathbf\{0\.02\}\}90\.2±0\.03\\mathbf\{90\.2\}\_\{\\pm\\mathbf\{0\.03\}\}84\.3±0\.04\\mathbf\{84\.3\}\_\{\\pm\\mathbf\{0\.04\}\}84\.6±0\.04\\mathbf\{84\.6\}\_\{\\pm\\mathbf\{0\.04\}\}85\.5±0\.06\\mathbf\{85\.5\}\_\{\\pm\\mathbf\{0\.06\}\}82\.0±0\.04\\mathbf\{82\.0\}\_\{\\pm\\mathbf\{0\.04\}\}82\.2±0\.06\\mathbf\{82\.2\}\_\{\\pm\\mathbf\{0\.06\}\}82\.8±0\.05\\mathbf\{82\.8\}\_\{\\pm\\mathbf\{0\.05\}\}data\-freeshared\-λ\\lambda87\.1±0\.0087\.1\_\{\\pm 0\.00\}87\.1±0\.0287\.1\_\{\\pm 0\.02\}88\.0±0\.0088\.0\_\{\\pm 0\.00\}81\.5±0\.0081\.5\_\{\\pm 0\.00\}80\.7±0\.0080\.7\_\{\\pm 0\.00\}82\.7±0\.0082\.7\_\{\\pm 0\.00\}77\.9±0\.0077\.9\_\{\\pm 0\.00\}76\.7±0\.0176\.7\_\{\\pm 0\.01\}78\.8±0\.0078\.8\_\{\\pm 0\.00\}random87\.4±0\.2087\.4\_\{\\pm 0\.20\}87\.4±0\.1387\.4\_\{\\pm 0\.13\}88\.1±0\.2388\.1\_\{\\pm 0\.23\}82\.2±0\.2482\.2\_\{\\pm 0\.24\}81\.2±0\.0881\.2\_\{\\pm 0\.08\}83\.3±0\.1883\.3\_\{\\pm 0\.18\}79\.0±0\.1879\.0\_\{\\pm 0\.18\}77\.3±0\.2077\.3\_\{\\pm 0\.20\}80\.0±0\.3380\.0\_\{\\pm 0\.33\}BO87\.8±0\.04\\mathbf\{87\.8\}\_\{\\pm\\mathbf\{0\.04\}\}87\.6±0\.04\\mathbf\{87\.6\}\_\{\\pm\\mathbf\{0\.04\}\}89\.0±0\.03\\mathbf\{89\.0\}\_\{\\pm\\mathbf\{0\.03\}\}82\.8±0\.03\\mathbf\{82\.8\}\_\{\\pm\\mathbf\{0\.03\}\}81\.7±0\.06\\mathbf\{81\.7\}\_\{\\pm\\mathbf\{0\.06\}\}84\.3±0\.05\\mathbf\{84\.3\}\_\{\\pm\\mathbf\{0\.05\}\}79\.7±0\.05\\mathbf\{79\.7\}\_\{\\pm\\mathbf\{0\.05\}\}78\.0±0\.02\\mathbf\{78\.0\}\_\{\\pm\\mathbf\{0\.02\}\}81\.5±0\.04\\mathbf\{81\.5\}\_\{\\pm\\mathbf\{0\.04\}\}

### 5\.3Ablation Study

#### Effectiveness of BO\-based global coordination\.

To investigate the effectiveness of BO\-based global coordination, Table[3](https://arxiv.org/html/2605.12843#S5.T3)compares BMM \(w/BO\) against two hyperparameter\-tuning baselines:shared\-λ\\lambdaandrandom search\. Whileshared\-λ\\lambdaassigns a single regularization strengthλ\\lambdato all four module groups and optimizes it via an extensive grid search,random searchrelaxes this constraint by allowing module\-specific hyperparameters, but tunes them using 200 trials of random search\.

As can be seen from Table[3](https://arxiv.org/html/2605.12843#S5.T3),shared\-λ\\lambdaalready yields substantial improvements over the anchor baselines, confirming that the closed\-form MAP estimator is the primary source of performance gains\. Relaxing the shared constraint to module\-specific hyperparameters withrandom searchfurther improves the performance, validating the importance of heterogeneous regularization across different module groups\. Finally, BMM \(w/BO\) achieves the best overall results by effectively coordinating the module\-specific hyperparameters with guided search\. Its advantage is most pronounced in the challenging 20\-task data\-free setting\. On the TSV anchor, BMM \(w/BO\) reaches an accuracy of79\.7%79\.7\\%, compared with79\.0%79\.0\\%forrandom searchand77\.9%77\.9\\%forshared\-λ\\lambdatuning\. This suggests that BO becomes particularly useful when the search space is more complex and the validation signal needs to balance task interference effectively in large\-scale merging settings\.

#### Sensitivity to the Fraction of Validation Set\.

Figure[3](https://arxiv.org/html/2605.12843#S5.F3)\(left\) reports the final test performance as the fraction of validation set used for BO evolves\. Across both TSV and ISO\-CTS anchors, and in data\-assisted and data\-free settings, BMM consistently outperforms the strongest anchor baseline \(ISO\-CTS\) even when a small fraction of the validation set is used\. The performance curves are relatively stable from10%10\\%to100%100\\%of the validation set, indicating that the outer\-loop optimization is not overly sensitive to the validation\-set size\. This suggests that BMM can use a small validation subset to reduce the optimization overhead while preserving most of the gains from global hyperparameter search\.

#### Sensitivity to BO BudgetKK\.

Figure[3](https://arxiv.org/html/2605.12843#S5.F3)\(right\) reports the evolution of final test performance as the number of BO trials \(KK\) increases\. BMM consistently outperforms the strongest anchor baseline \(ISO\-CTS\) under a small BO budget \(KK\)\. The performance curves reach a plateau after roughly40−6040\-60trials, while increasing the budget further yields only a marginal return\. These results indicate that BO is efficient in global hyperparameter search, reaching stable results with a modest budget\. Additional runtime\-performance comparisons against other methods are reported in Appendix[D\.3](https://arxiv.org/html/2605.12843#A4.SS3)\.

![Refer to caption](https://arxiv.org/html/2605.12843v1/x1.png)Figure 3:Ablation study of BMM on 20\-task merging \(ViT\-B/32\)\.\(Left\) Test accuracy as a function of the validation set fraction used for Bayesian Optimization \(BO\)\. \(Right\) Test accuracy vs\. the number of BO search trials \(KK\)\. All curves report mean±\\pmstd across 5 seeds\. Solid and circle\-dashed lines represent data\-assisted and data\-free BMM\. Blue/green colors indicate ISO\-CTS/TSV anchors\. The horizontal gray dot\-dashed line establishes the ISO\-CTS baseline\.

## 6Conclusion

This paper introducesBayesian Model Merging \(BMM\), a*plug\-and\-play*framework for model merging\. BMM leverages existing merging solutions as informative priors and formulates module\-wise merging as an anchor\-regularized Bayesian linear regression with a closed\-form solution\. To coordinate local merging decisions with end\-to\-end performance, BMM further adopts a𝒢​𝒫\\mathcal\{GP\}\-based Bayesian optimization to search for regularization strengths globally\. We also derive a data\-free variant of BMM based on a key alignment between activation statistics and task vectors, while preserving the closed\-form solution\. Across vision and language benchmarks, BMM consistently improves diverse anchor baselines, scales to challenging 20\-task settings, and closely approaches the performance of individual fine\-tuned experts on high\-capacity ViT architectures\. Our results show that combining informative anchors with global hyperparameter coordination provides a practical path towards scalable model merging when auxiliary calibration data is limited or unavailable\.

#### Limitations and Broader Impacts\.

BMM does not exhibit major limitations in the evaluated settings, though our experiments are currently limited to models up to the 8B scale due to computational constraints\. Regarding the broader impacts, BMM enables efficient aggregation of task\-specific experts without access to original training data, reducing the cost of foundation\-model customization\. We do not foresee any societal risks beyond those generally associated with large vision and language models, although merged models may inherit biases, safety risks, or domain\-specific failures from their underlying experts\.

## References

- \[1\]Sebastian Ruder\.An overview of multi\-task learning in deep neural networks\.arXiv preprint arXiv:1706\.05098, 2017\.
- \[2\]Michael S\. Matena and Colin A\. Raffel\.Merging models with fisher\-weighted averaging\.Advances in Neural Information Processing Systems, 35:17703–17716, 2022\.
- \[3\]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi\.Editing models with task arithmetic\.InThe Eleventh International Conference on Learning Representations, 2023\.
- \[4\]Prateek Yadav, Derek Tam, Leshem Choshen, Colin A\. Raffel, and Mohit Bansal\.TIES\-merging: Resolving interference when merging models\.InThirty\-seventh Conference on Neural Information Processing Systems, 2023\.
- \[5\]Hugging Face\.The hugging face hub\.[https://huggingface\.co](https://huggingface.co/), 2026\.Accessed: 2026\-05\-04\.
- \[6\]Xisen Jin, Xiang Ren, Daniel Preotiuc\-Pietro, and Pengxiang Cheng\.Dataless knowledge fusion by merging weights of language models\.InThe Eleventh International Conference on Learning Representations, 2023\.
- \[7\]Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan\.Whoever started the interference should end it: Guiding data\-free model merging via task vectors\.InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 10121–10143\. PMLR, 2025\.
- \[8\]Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà\.Task singular vectors: Reducing task interference in model merging\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18695–18705, 2025\.
- \[9\]Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D\. Bagdanov, and Joost van de Weijer\.No task left behind: Isotropic model merging with common and task\-specific subspaces\.InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 43177–43199\. PMLR, 2025\.
- \[10\]Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li\.Language models are super mario: Absorbing abilities from homologous models as a free lunch\.InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 57755–57775\. PMLR, 2024\.
- \[11\]Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho\-Kin Tang, Daojing He, and Min Zhang\.Parameter competition balancing for model merging\.Advances in Neural Information Processing Systems, 37, 2024\.
- \[12\]Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao\.Localize\-and\-stitch: Efficient model merging via sparse task arithmetic\.Transactions on Machine Learning Research, 2025\.Accepted to TMLR\.
- \[13\]Yongxian Wei, Anke Tang, Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao\.Modeling multi\-task model merging as adaptive projective gradient descent\.InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 66178–66193\. PMLR, 2025\.
- \[14\]Vardan Papyan, X\. Y\. Han, and David L\. Donoho\.Prevalence of neural collapse during the terminal phase of deep learning training\.Proceedings of the National Academy of Sciences, 117\(40\):24652–24663, 2020\.
- \[15\]Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin\.Mechanism for feature learning in neural networks and kernel machines\.Science, 383\(6690\):1461–1467, 2024\.
- \[16\]Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin\.Average gradient outer product as a mechanism for deep neural collapse\.InAdvances in Neural Information Processing Systems, 2024\.
- \[17\]Liu Ziyin, Isaac Chuang, Tomer Galanti, and Tomaso Poggio\.Formation of representations in neural networks\.InThe Thirteenth International Conference on Learning Representations \(ICLR 2025\), 2025\.Spotlight\.
- \[18\]Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez\-Granda, Zhihui Zhu, and Qing Qu\.Understanding and improving transfer learning of deep models via neural collapse\.arXiv preprint arXiv:2212\.12206, 2022\.
- \[19\]Yuhe Ding, Bo Jiang, Lijun Sheng, Aihua Zheng, and Jian Liang\.Unleashing the power of neural collapse for transferability estimation\.arXiv preprint arXiv:2310\.05754, 2023\.
- \[20\]Michael Munn, Benoit Dherin, and Javier Gonzalvo\.The impact of geometric complexity on neural collapse in transfer learning\.InAdvances in Neural Information Processing Systems, 2024\.
- \[21\]Peter I Frazier\.Bayesian optimization\.InRecent advances in optimization and modeling of contemporary problems, pages 255–278\. Informs, 2018\.
- \[22\]Yifei He, Siqi Zeng, Yuzheng Hu, Rui Yang, Tong Zhang, and Han Zhao\.Mergebench: A benchmark for merging domain\-specialized llms\.arXiv preprint arXiv:2505\.10833, 2025\.
- \[23\]Xuhong Li, Yves Grandvalet, and Franck Davoine\.Explicit inductive bias for transfer learning with convolutional networks\.InProceedings of the 35th International Conference on Machine Learning, pages 2825–2834\. PMLR, 2018\.
- \[24\]Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma\.The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects\.InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 7654–7663\. PMLR, 2019\.
- \[25\]Jingfeng Wu, Difan Wang, and Weijie J\. Su\.The alignment property of SGD noise and how it helps select flat minima: A stability analysis\.InAdvances in Neural Information Processing Systems, volume 35, pages 4680–4693, 2022\.
- \[26\]James Martens and Roger Grosse\.Optimizing neural networks with kronecker\-factored approximate curvature\.InProceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 2408–2417\. PMLR, 2015\.
- \[27\]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q\. Weinberger\.On calibration of modern neural networks\.InInternational Conference on Machine Learning, 2017\.
- \[28\]Agustinus Kristiadi, Matthias Hein, and Philipp Hennig\.Being bayesian, even just a bit, fixes overconfidence in relu networks\.InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 5436–5446\. PMLR, 2020\.
- \[29\]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei\-Fei\.3d object representations for fine\-grained categorization\.InICCV Workshops, 2013\.
- \[30\]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi\.Describing textures in the wild\.InCVPR, 2014\.
- \[31\]Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth\.Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification\.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019\.
- \[32\]Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel\.The german traffic sign recognition benchmark: A multi\-class classification competition\.InIJCNN, 2011\.
- \[33\]Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner\.Gradient\-based learning applied to document recognition\.Proceedings of the IEEE, 1998\.
- \[34\]Gong Cheng, Junwei Han, and Xiaoqiang Lu\.Remote sensing image scene classification: Benchmark and state of the art\.Proceedings of the IEEE, 2017\.
- \[35\]Jianxiong Xiao, James Hays, Krista A\. Ehinger, Aude Oliva, and Antonio Torralba\.Sun database: Exploring a large collection of scene categories\.InIJCV, 2016\.
- \[36\]Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y\. Ng\.Reading digits in natural images with unsupervised feature learning\.InNeurIPS Workshops, 2011\.
- \[37\]Alex Krizhevsky and Geoffrey Hinton\.Learning multiple layers of features from tiny images\.Technical report, University of Toronto, 2009\.
- \[38\]Adam Coates, Andrew Ng, and Honglak Lee\.An analysis of single\-layer networks in unsupervised feature learning\.InAISTATS, 2011\.
- \[39\]Maria\-Elena Nilsback and Andrew Zisserman\.Automated flower classification over a large number of classes\.InICVGIP, 2008\.
- \[40\]Omkar M\. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C\. V\. Jawahar\.Cats and dogs\.InCVPR, 2012\.
- \[41\]Bastiaan S\. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling\.Rotation equivariant cnns for digital pathology\.InMICCAI, 2018\.
- \[42\]Ian J\. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong\-Hyun Lee, et al\.Challenges in representation learning: A report on three machine learning contests\.arXiv preprint arXiv:1307\.0414, 2013\.
- \[43\]Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre van Schaik\.Emnist: Extending mnist to handwritten letters\.InIJCNN, 2017\.
- \[44\]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool\.Food\-101: Mining discriminative components with random forests\.InECCV, 2014\.
- \[45\]Han Xiao, Kashif Rasul, and Roland Vollgraf\.Fashion\-mnist: A novel image dataset for benchmarking machine learning algorithms\.arXiv preprint arXiv:1708\.07747, 2017\.
- \[46\]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D\. Manning, Andrew Y\. Ng, and Christopher Potts\.Recursive deep models for semantic compositionality over a sentiment treebank\.InEMNLP, 2013\.
- \[47\]Tarin Clanuwat, Mikel Bober\-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha\.Deep learning for classical japanese literature\.arXiv preprint arXiv:1812\.01718, 2018\.
- \[48\]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al\.Tülu 3: Pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124, 2024\.
- \[49\]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou\.Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911, 2023\.
- \[50\]Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He\.Dart\-math: Difficulty\-aware rejection tuning for mathematical problem\-solving\.Advances in Neural Information Processing Systems, 37:7821–7846, 2024\.
- \[51\]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al\.Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions\.Hugging Face repository, 2024\.
- \[52\]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168, 2021\.
- \[53\]Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei\-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al\.Aya dataset: An open\-access collection for multilingual instruction tuning\.arXiv preprint arXiv:2402\.06619, 2024\.
- \[54\]Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen\.Okapi: Instruction\-tuned large language models in multiple languages with reinforcement learning from human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, 2023\.
- \[55\]Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang\.Magicoder: Empowering code generation with oss\-instruct\.arXiv preprint arXiv:2312\.02120, 2023\.
- \[56\]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732, 2021\.
- \[57\]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang\.Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation\.InThirty\-seventh Conference on Neural Information Processing Systems, 2023\.
- \[58\]Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri\.Wildguard: Open one\-stop moderation tools for safety risks, jailbreaks, and refusals of llms\.arXiv preprint arXiv:2406\.18495, 2024\.
- \[59\]Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al\.Wildteaming at scale: From in\-the\-wild jailbreaks to \(adversarially\) safer language models\.Advances in Neural Information Processing Systems, 37:47094–47165, 2024\.
- \[60\]Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al\.Harmbench: A standardized evaluation framework for automated red teaming and robust refusal\.arXiv preprint arXiv:2402\.04249, 2024\.
- \[61\]Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy\.Xstest: A test suite for identifying exaggerated safety behaviours in large language models\.arXiv preprint arXiv:2308\.01263, 2023\.
- \[62\]Gene H\. Golub and Charles F\. Van Loan\.Matrix Computations\.Johns Hopkins University Press, 4 edition, 2013\.
- \[63\]Carl Edward Rasmussen and Christopher K\. I\. Williams\.Gaussian Processes for Machine Learning\.MIT Press, 2006\.

## Appendix AProof of Theorem[1](https://arxiv.org/html/2605.12843#Thmtheorem1)

Let𝐱\\mathbf\{x\}denote an input activation to module𝐖\(t\)\\mathbf\{W\}^\{\(t\)\}and𝐲=𝐔\(t\)​𝐱=\(𝐖\(t\)−𝐖pre\)​𝐱\\mathbf\{y\}=\\mathbf\{U\}^\{\(t\)\}\\mathbf\{x\}=\(\\mathbf\{W\}^\{\(t\)\}\-\\mathbf\{W\}\_\{\\mathrm\{pre\}\}\)\\mathbf\{x\}the corresponding residual output\. Using standard Stochastic Gradient Descent \(SGD\) with theL2L\_\{2\}regularization, we define𝐠𝐲=−∇𝐲ℓ\\mathbf\{g\}\_\{\\mathbf\{y\}\}=\-\\nabla\_\{\\mathbf\{y\}\}\\ellas the backpropagated descent\-direction signal induced by the fine\-tune lossℓ\\ell\. The single\-step parameter update for the task vector𝐔\(t\)\\mathbf\{U\}^\{\(t\)\}is governed by:

δ​\(𝐔\(t\)\)=η​\(𝐠𝐲​𝐱⊤−ρ​𝐔\(t\)\),\\delta\(\\mathbf\{U\}^\{\(t\)\}\)=\\eta\\left\(\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\mathbf\{x\}^\{\\top\}\-\\rho\\mathbf\{U\}^\{\(t\)\}\\right\),\(16\)whereρ\>0\\rho\>0is the weight decay coefficient, andη\>0\\eta\>0is the learning rate\. By defining𝐃\(t\)=𝐠𝐲​𝐱⊤\\mathbf\{D\}^\{\(t\)\}=\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\mathbf\{x\}^\{\\top\}as the per\-sample descent matrix, we have

δ​\(𝐔\(t\)\)=η​\(𝐃\(t\)−ρ​𝐔\(t\)\)\.\\delta\(\\mathbf\{U\}^\{\(t\)\}\)=\\eta\\big\(\\mathbf\{D\}^\{\(t\)\}\-\\rho\\mathbf\{U\}^\{\(t\)\}\\big\)\.\(17\)In order to connect the fine\-tuned checkpoints with the*unobserved*activations, we study the activation statistics at the terminal phase of fine\-tuning\. Since model merging operates on fully fine\-tuned checkpoints rather than the intermediate ones, our goal here is not to characterize the full stochastic training trajectory, but to capture the geometric structure of representations from the final converged solutions\. Specifically, we assume the fine\-tuning process reaches a state that satisfies the following assumption\.

Assumption 1\.*The fine\-tuned checkpoint is assumed to converge on tasktt, and satisfies the following three conditions:*

1. 1\.The expected per\-sample descent matrix is aligned with the task vector:𝔼​\[𝐃\(t\)\]=ρ​𝐔\(t\)\\mathbb\{E\}\[\\mathbf\{D\}^\{\(t\)\}\]=\\rho\\mathbf\{U\}^\{\(t\)\}, or equivalently,𝔼​\[δ​\(𝐔\(t\)\)\]=𝟎\\mathbb\{E\}\[\\delta\(\\mathbf\{U\}^\{\(t\)\}\)\]=\\mathbf\{0\}\.
2. 2\.At convergence, the centered descent matrix fluctuations are assumed to retain a positive Frobenius overlap with the Gram matrix of the mean descent matrix signal\. That is, let 𝐃¯\(t\)=𝔼​\[𝐃\(t\)\],𝐂t=𝔼​\[\(𝐃\(t\)−𝐃¯\(t\)\)⊤​\(𝐃\(t\)−𝐃¯\(t\)\)\],𝐌t=\(𝐃¯\(t\)\)⊤​𝐃¯\(t\)\.\\overline\{\\mathbf\{D\}\}^\{\(t\)\}=\\mathbb\{E\}\[\\mathbf\{D\}^\{\(t\)\}\],\\quad\\mathbf\{C\}\_\{t\}=\\mathbb\{E\}\\\!\\left\[\(\\mathbf\{D\}^\{\(t\)\}\-\\overline\{\\mathbf\{D\}\}^\{\(t\)\}\)^\{\\top\}\(\\mathbf\{D\}^\{\(t\)\}\-\\overline\{\\mathbf\{D\}\}^\{\(t\)\}\)\\right\],\\quad\\mathbf\{M\}\_\{t\}=\(\\overline\{\\mathbf\{D\}\}^\{\(t\)\}\)^\{\\top\}\\overline\{\\mathbf\{D\}\}^\{\(t\)\}\.\(18\)We havecosF⁡\(𝐂t,𝐌t\)\>αt\\operatorname\{cos\}\_\{F\}\(\\mathbf\{C\}\_\{t\},\\mathbf\{M\}\_\{t\}\)\>\\alpha\_\{t\},0<αt≤10<\\alpha\_\{t\}\\leq 1, wherecosF⁡\(𝐀,𝐁\)=Tr⁡\(𝐀⊤​𝐁\)/\(‖𝐀‖F​‖𝐁‖F\)\\operatorname\{cos\}\_\{F\}\(\\mathbf\{A\},\\mathbf\{B\}\)=\\operatorname\{Tr\}\(\\mathbf\{A\}^\{\\top\}\\mathbf\{B\}\)/\(\\\|\\mathbf\{A\}\\\|\_\{F\}\\\|\\mathbf\{B\}\\\|\_\{F\}\)is the Frobenius cosine\-similarity\.
3. 3\.The gradient energy factorizes from the second\-moment of input activation: 𝔼​\[‖𝐠𝐲‖22​𝐱𝐱⊤\]=𝔼​\[‖𝐠𝐲‖22\]​𝔼​\[𝐱𝐱⊤\]\.\\mathbb\{E\}\\\!\\left\[\\\|\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\\|\_\{2\}^\{2\}\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\\right\]=\\mathbb\{E\}\[\\\|\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\\|\_\{2\}^\{2\}\]\\,\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\]\.\(19\)

*All expectations are w\.r\.t\. the stochasticity induced by mini\-batch sampling while conditioning on the fine\-tuned checkpoint\.*

Assumption 1 reflects a local quasi\-stationary basin in which the mean of update\-drift becomes negligible and the weight norm remains stable\. Condition \(1\) states that the averaged task\-specific descent direction is balanced by the learned task vector\. This resembles shrinkage around a pretrained solution inL2L\_\{2\}\-style transfer learning\[[23](https://arxiv.org/html/2605.12843#bib.bib23)\]and is consistent with terminal representation geometry, where features, weights, and gradients align along task\-relevant directions\[[14](https://arxiv.org/html/2605.12843#bib.bib14),[17](https://arxiv.org/html/2605.12843#bib.bib17)\]\. Condition \(2\) weakens exact proportionality to a cosine\-similarity alignment\. It allows nonzero minibatch fluctuations, but assumes that their dominant input\-side second\-moment geometry remains aligned with the mean task\-descent geometry\. This is motivated by evidence that SGD noise is anisotropic and geometry\-aware rather than isotropic\[[24](https://arxiv.org/html/2605.12843#bib.bib24),[25](https://arxiv.org/html/2605.12843#bib.bib25)\]; in particular, Wu et al\.\[[25](https://arxiv.org/html/2605.12843#bib.bib25)\]quantify alignment between SGD noise covariance and Fisher geometry using Frobenius\-type alignment factors\. Condition \(3\) states residual error signals become less structured, motivating a coarse decoupling between gradient energy and input activations in the spirit of K\-FAC\-style layer\-wise curvature factorizations\[[26](https://arxiv.org/html/2605.12843#bib.bib26)\]and representation/gradient geometry views such as AGOP/NFA\[[15](https://arxiv.org/html/2605.12843#bib.bib15),[16](https://arxiv.org/html/2605.12843#bib.bib16)\]\.

Theorem[1](https://arxiv.org/html/2605.12843#Thmtheorem1)Restated\.Under Assumption 1, the Gram matrix of input activations and the Gram matrix of task\-vectors have a positively correlated alignment:

cosF⁡\(𝔼​\[𝐱𝐱⊤\],\(𝐔\(t\)\)⊤​𝐔\(t\)\)\>αt\.\\operatorname\{cos\}\_\{F\}\\\!\\left\(\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\],\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\\right\)\>\\alpha\_\{t\}\.\(20\)
*Proof\.*For notational brevity, we omit the task superscript and write

𝐃¯=𝔼​\[𝐃\],𝐌=𝐃¯⊤​𝐃¯,𝐂=𝔼​\[\(𝐃−𝐃¯\)⊤​\(𝐃−𝐃¯\)\]\.\\overline\{\\mathbf\{D\}\}=\\mathbb\{E\}\[\\mathbf\{D\}\],\\qquad\\mathbf\{M\}=\\overline\{\\mathbf\{D\}\}^\{\\top\}\\overline\{\\mathbf\{D\}\},\\qquad\\mathbf\{C\}=\\mathbb\{E\}\\\!\\left\[\(\\mathbf\{D\}\-\\overline\{\\mathbf\{D\}\}\)^\{\\top\}\(\\mathbf\{D\}\-\\overline\{\\mathbf\{D\}\}\)\\right\]\.By the second\-moment decomposition,

𝔼​\[𝐃⊤​𝐃\]=𝐌\+𝐂\.\\mathbb\{E\}\[\\mathbf\{D\}^\{\\top\}\\mathbf\{D\}\]=\\mathbf\{M\}\+\\mathbf\{C\}\.\(21\)Assumption 1\.1 gives

𝐌=ρ2​\(𝐔\(t\)\)⊤​𝐔\(t\)\.\\mathbf\{M\}=\\rho^\{2\}\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\.\(22\)Since the positive scaling byρ2\\rho^\{2\}does not affect cosine similarity, it suffices to lower\-bound

cosF⁡\(𝐌\+𝐂,𝐌\)\.\\operatorname\{cos\}\_\{F\}\(\\mathbf\{M\}\+\\mathbf\{C\},\\mathbf\{M\}\)\.If𝐂=𝟎\\mathbf\{C\}=\\mathbf\{0\}, this cosine is11\>αt\\alpha\_\{t\}\. Otherwise, let

r=‖𝐂‖F‖𝐌‖F,γ=cosF⁡\(𝐂,𝐌\)\.r=\\frac\{\\\|\\mathbf\{C\}\\\|\_\{F\}\}\{\\\|\\mathbf\{M\}\\\|\_\{F\}\},\\qquad\\gamma=\\operatorname\{cos\}\_\{F\}\(\\mathbf\{C\},\\mathbf\{M\}\)\.\(23\)By Assumption 1\.2,γ\>0\\gamma\>0\. From the definitions ofrrandγ\\gamma,

⟨𝐂,𝐌⟩F=γ​‖𝐂‖F​‖𝐌‖F=r​γ​‖𝐌‖F2\.\\langle\\mathbf\{C\},\\mathbf\{M\}\\rangle\_\{F\}=\\gamma\\\|\\mathbf\{C\}\\\|\_\{F\}\\\|\\mathbf\{M\}\\\|\_\{F\}=r\\gamma\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\.\(24\)Moreover,

‖𝐌\+𝐂‖F2=‖𝐌‖F2\+‖𝐂‖F2\+2​⟨𝐂,𝐌⟩F=‖𝐌‖F2​\(1\+r2\+2​r​γ\)\.\\\|\\mathbf\{M\}\+\\mathbf\{C\}\\\|\_\{F\}^\{2\}=\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\+\\\|\\mathbf\{C\}\\\|\_\{F\}^\{2\}\+2\\langle\\mathbf\{C\},\\mathbf\{M\}\\rangle\_\{F\}=\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\(1\+r^\{2\}\+2r\\gamma\)\.\(25\)Using Eqs\. \([24](https://arxiv.org/html/2605.12843#A1.E24)\) and \([25](https://arxiv.org/html/2605.12843#A1.E25)\), the Frobenius cosine\-similarity becomes

cosF⁡\(𝐌\+𝐂,𝐌\)\\displaystyle\\operatorname\{cos\}\_\{F\}\(\\mathbf\{M\}\+\\mathbf\{C\},\\mathbf\{M\}\)=⟨𝐌\+𝐂,𝐌⟩F‖𝐌\+𝐂‖F​‖𝐌‖F\\displaystyle=\\frac\{\\langle\\mathbf\{M\}\+\\mathbf\{C\},\\mathbf\{M\}\\rangle\_\{F\}\}\{\\\|\\mathbf\{M\}\+\\mathbf\{C\}\\\|\_\{F\}\\\|\\mathbf\{M\}\\\|\_\{F\}\}\(26\)=‖𝐌‖F2\+⟨𝐂,𝐌⟩F‖𝐌‖F2​1\+r2\+2​r​γ=1\+r​γ1\+r2\+2​r​γ\.\\displaystyle=\\frac\{\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\+\\langle\\mathbf\{C\},\\mathbf\{M\}\\rangle\_\{F\}\}\{\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\\sqrt\{1\+r^\{2\}\+2r\\gamma\}\}=\\frac\{1\+r\\gamma\}\{\\sqrt\{1\+r^\{2\}\+2r\\gamma\}\}\.Forr≥0r\\geq 0andγ∈\(0,1\]\\gamma\\in\(0,1\], we have

1\+r​γ1\+r2\+2​r​γ≥γ,\\frac\{1\+r\\gamma\}\{\\sqrt\{1\+r^\{2\}\+2r\\gamma\}\}\\geq\\gamma,since both sides are nonnegative and the squared inequality reduces to

\(1−γ2\)​\(1\+2​r​γ\)≥0\.\(1\-\\gamma^\{2\}\)\(1\+2r\\gamma\)\\geq 0\.Therefore,

cosF⁡\(𝐌\+𝐂,𝐌\)≥γ\>αt\.\\operatorname\{cos\}\_\{F\}\(\\mathbf\{M\}\+\\mathbf\{C\},\\mathbf\{M\}\)\\geq\\gamma\>\\alpha\_\{t\}\.\(27\)Finally, by Eq\. \([21](https://arxiv.org/html/2605.12843#A1.E21)\),

𝔼​\[𝐃⊤​𝐃\]=𝐌\+𝐂,\\mathbb\{E\}\[\\mathbf\{D\}^\{\\top\}\\mathbf\{D\}\]=\\mathbf\{M\}\+\\mathbf\{C\},and by Eq\. \([22](https://arxiv.org/html/2605.12843#A1.E22)\),

𝐌=ρ2​\(𝐔\(t\)\)⊤​𝐔\(t\)\.\\mathbf\{M\}=\\rho^\{2\}\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\.Sinceρ2\>0\\rho^\{2\}\>0, this positive scalar does not change Frobenius cosine\-similarity\. Therefore Eq\. \([27](https://arxiv.org/html/2605.12843#A1.E27)\) implies

cosF⁡\(𝔼​\[𝐃⊤​𝐃\],\(𝐔\(t\)\)⊤​𝐔\(t\)\)\>αt\.\\operatorname\{cos\}\_\{F\}\\\!\\left\(\\mathbb\{E\}\[\\mathbf\{D\}^\{\\top\}\\mathbf\{D\}\],\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\\right\)\>\\alpha\_\{t\}\.\(28\)
It remains to connect𝔼​\[𝐃⊤​𝐃\]\\mathbb\{E\}\[\\mathbf\{D\}^\{\\top\}\\mathbf\{D\}\]to the activation Gram matrix\. Since𝐃=𝐠𝐲​𝐱⊤\\mathbf\{D\}=\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\mathbf\{x\}^\{\\top\},

𝐃⊤​𝐃=‖𝐠𝐲‖22​𝐱𝐱⊤\.\\mathbf\{D\}^\{\\top\}\\mathbf\{D\}=\\\|\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\\|\_\{2\}^\{2\}\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\.\(29\)Taking expectations and applying Assumption 1\.3,

𝔼​\[𝐃⊤​𝐃\]=𝔼​\[‖𝐠𝐲‖22\]​𝔼​\[𝐱𝐱⊤\],\\mathbb\{E\}\[\\mathbf\{D\}^\{\\top\}\\mathbf\{D\}\]=\\mathbb\{E\}\[\\\|\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\\|\_\{2\}^\{2\}\]\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\],\(30\)where𝔼​\[‖𝐠𝐲‖22\]\\mathbb\{E\}\[\\\|\\mathbf\{g\}\_\{\\mathbf\{y\}\}\\\|\_\{2\}^\{2\}\]does not change cosine similarity\. Combining this with Eq\. \([28](https://arxiv.org/html/2605.12843#A1.E28)\), we have

cosF⁡\(𝔼​\[𝐱𝐱⊤\],\(𝐔\(t\)\)⊤​𝐔\(t\)\)\>αt\.\\operatorname\{cos\}\_\{F\}\\\!\\left\(\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\],\(\\mathbf\{U\}^\{\(t\)\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\\right\)\>\\alpha\_\{t\}\.\(31\)This completes the proof\.■\\blacksquare

## Appendix BSampling\-based BMM for Uncertainty Calibration

#### Sampling\-based BMM\.

A MAP point estimate of𝐔\\mathbf\{U\}is used in Algorithm[1](https://arxiv.org/html/2605.12843#alg1)mainly for the purpose of computational efficiency\. Note that the Bayesian linear regression formulation in Sec\.[4\.1](https://arxiv.org/html/2605.12843#S4.SS1)leads to a Gaussian posterior𝒩​\(𝐔MAP,𝚺row\)\\mathcal\{N\}\(\\mathbf\{U\}\_\{\\text\{MAP\}\},\\mathbf\{\\Sigma\}\_\{\\text\{row\}\}\)with

𝐔MAP=\(𝐘𝐗⊤\+λ​𝐔\(0\)\)​\(𝐗𝐗⊤\+λ​𝐈\)−1,𝚺row=β−1​\(𝐗𝐗⊤\+λ​𝐈\)−1,\\mathbf\{U\}\_\{\\text\{MAP\}\}=\\left\(\\mathbf\{Y\}\\mathbf\{X\}^\{\\top\}\+\\lambda\\mathbf\{U\}^\{\(0\)\}\\right\)\\left\(\\mathbf\{X\}\\mathbf\{X\}^\{\\top\}\+\\lambda\\mathbf\{I\}\\right\)^\{\-1\},\\quad\\quad\\mathbf\{\\Sigma\}\_\{\\text\{row\}\}=\\beta^\{\-1\}\\left\(\\mathbf\{X\}\\mathbf\{X\}^\{\\top\}\+\\lambda\\mathbf\{I\}\\right\)^\{\-1\},\(32\)where𝐈∈ℝdin×din\\mathbf\{I\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times d\_\{\\mathrm\{in\}\}\}is the identity matrix,𝚺row\\mathbf\{\\Sigma\}\_\{\\text\{row\}\}is the covariance matrix for each row of𝐔\\mathbf\{U\}, andβ\\betais the precision of Gaussian noise, which can be tuned on a validation set\. Once𝝀⋆\\bm\{\\lambda\}^\{\\star\}is determined by Algorithm[1](https://arxiv.org/html/2605.12843#alg1), we can sample multiple merged models for uncertainty calibration\[[27](https://arxiv.org/html/2605.12843#bib.bib27)\]\.

![Refer to caption](https://arxiv.org/html/2605.12843v1/x2.png)\(a\)8\-task benchmark
![Refer to caption](https://arxiv.org/html/2605.12843v1/x3.png)\(b\)20\-task benchmark

Figure 4:Pareto frontiers of sampling\-based BMM vs\. MAP\-perturbed BMM on ViT\-B/32 benchmarks\. Each dot is generated by varying eitherβ\\betaorσiid\\sigma\_\{\\text\{iid\}\}, and the black dot is the performance of BMM \(MAP\), corresponding toβ=∞\\beta=\\inftyorσiid=0\\sigma\_\{\\text\{iid\}\}=0\.Specifically, let\{θmerged\(r\)\}r=1S\\\{\\theta\_\{\\text\{merged\}\}^\{\(r\)\}\\\}\_\{r=1\}^\{S\}beSSmerged models sampled from the Gaussian posteriors \(given𝝀⋆\\bm\{\\lambda\}^\{\\star\}andβ\\beta\), and𝐳​\(𝐱;θ\)\\mathbf\{z\}\(\\mathbf\{x\};\\theta\)denote the logit vector of modelθ\\theta\. The prediction of the model ensemble is the average probability acrossSSmerged models:

p​\(𝐲\|𝐱\)=1S​∑r=1Ssoftmax⁡\(𝐳​\(𝐱;θmerged\(r\)\)\)\.p\(\\mathbf\{y\}\|\\mathbf\{x\}\)=\\frac\{1\}\{S\}\\sum\_\{r=1\}^\{S\}\\operatorname\{softmax\}\\\!\\left\(\\mathbf\{z\}\(\\mathbf\{x\};\\theta^\{\(r\)\}\_\{\\mathrm\{merged\}\}\)\\right\)\.\(33\)As we will see, this model ensemble improves the uncertainty calibration as measured by ECE, a standard metric for this purpose\.

#### Expected Calibration Error\.

ECE is a commonly adopted metric to measure the calibration of a model\. First, it computes the confidence of the model,maxy⁡p​\(y\|𝐱i\)\\max\_\{y\}p\(y\|\\mathbf\{x\}\_\{i\}\), for each𝐱i\\mathbf\{x\}\_\{i\}in the dataset\. Then it groups the predictions into equally spaced buckets\{B1,B2,⋯,Bκ\}\\\{B\_\{1\},B\_\{2\},\\cdots,B\_\{\\kappa\}\\\}based on their confidence scores\. For example, ifκ\\kappa= 20, thenB1B\_\{1\}would represent all examples for which the model’s confidence scores were between 0 and 0\.05\. Then ECE is calculated as

ECE=∑b=1κ\|Bb\|n​\|acc⁡\(Bb\)−conf⁡\(Bb\)\|,\\mathrm\{ECE\}=\\sum\_\{b=1\}^\{\\kappa\}\\frac\{\\left\|B\_\{b\}\\right\|\}\{n\}\\left\|\\operatorname\{acc\}\\left\(B\_\{b\}\\right\)\-\\operatorname\{conf\}\\left\(B\_\{b\}\\right\)\\right\|,\(34\)wherennis the number of examples in the dataset, acc\(BbB\_\{b\}\) is the average accuracy of the model on all the examples inBbB\_\{b\}and conf\(BbB\_\{b\}\) is the average confidence on all the examples inBbB\_\{b\}\. In our experiments, we setκ\\kappa= 20\. For a perfectly calibrated model, the ECE will be 0 for anyκ\\kappa\.

#### Experimental Setup\.

We evaluate the uncertainty calibration of model ensemble, constructed from sampling\-based BMM \(controlled byβ\\beta\), on the ViT\-B/32 8\-task and 20\-task benchmarks in the data\-assisted setting, where ISO\-CTS serves as the anchor model\. Preliminary experiments show that sampling only for the last MLP output matrix leads to the most prominent ECE performance – similar observations have also been made by\[[28](https://arxiv.org/html/2605.12843#bib.bib28)\]\. As a baseline, we also perturb the MAP solution of the last MLP output matrix with IID Gaussian noise \(controlled byσiid\\sigma\_\{\\text\{iid\}\}\)\. Both methods generateS=10S=10samples of merged models, and use the average prediction of Eq\.[33](https://arxiv.org/html/2605.12843#A2.E33)for final classification\. By varyingβ\\betaandσiid\\sigma\_\{\\text\{iid\}\}, each model ensemble trades\-off between classification accuracies and ECE scores, and the Pareto frontiers of both methods are reported in Figure[4](https://arxiv.org/html/2605.12843#A2.F4)\.

#### Results and Analysis\.

As can be seen, the Pareto frontier of sampling\-based BMM dominates that of MAP\-perturbed BMM, indicating a better trade\-off between classification accuracies and ECE scores for the former\. In addition, the gap between two Pareto frontiers is more pronounced on the 8\-task benchmark, indicating that merging on 20\-task benchmark is more challenging and the flexibility for the tradeoff is limited\.

## Appendix CExperimental Protocol and Assets

### C\.1Vision Benchmarks

For the*vision*experiments, to ensure a rigorous and fair comparison, we strictly adhere to the benchmarks established by TSV\[[8](https://arxiv.org/html/2605.12843#bib.bib8)\]and ISO\-CTS\[[9](https://arxiv.org/html/2605.12843#bib.bib9)\], adopting their exact datasets and identical training, validation, and test splits\. Specifically, our evaluation spans three benchmarks with increasing task diversity\. The initial 8\-task benchmark comprises Stanford Cars\[[29](https://arxiv.org/html/2605.12843#bib.bib29)\], DTD\[[30](https://arxiv.org/html/2605.12843#bib.bib30)\], EuroSAT\[[31](https://arxiv.org/html/2605.12843#bib.bib31)\], GTSRB\[[32](https://arxiv.org/html/2605.12843#bib.bib32)\], MNIST\[[33](https://arxiv.org/html/2605.12843#bib.bib33)\], RESISC45\[[34](https://arxiv.org/html/2605.12843#bib.bib34)\], SUN397\[[35](https://arxiv.org/html/2605.12843#bib.bib35)\], and SVHN\[[36](https://arxiv.org/html/2605.12843#bib.bib36)\]\. The 14\-task benchmark extends this setting by adding CIFAR\-100\[[37](https://arxiv.org/html/2605.12843#bib.bib37)\], STL\-10\[[38](https://arxiv.org/html/2605.12843#bib.bib38)\], Flowers102\[[39](https://arxiv.org/html/2605.12843#bib.bib39)\], Oxford\-IIIT Pets\[[40](https://arxiv.org/html/2605.12843#bib.bib40)\], PCAM\[[41](https://arxiv.org/html/2605.12843#bib.bib41)\], and FER2013\[[42](https://arxiv.org/html/2605.12843#bib.bib42)\]\. Finally, the largest benchmark encompasses 20 tasks by further incorporating EMNIST\[[43](https://arxiv.org/html/2605.12843#bib.bib43)\], CIFAR\-10\[[37](https://arxiv.org/html/2605.12843#bib.bib37)\], Food101\[[44](https://arxiv.org/html/2605.12843#bib.bib44)\], Fashion\-MNIST\[[45](https://arxiv.org/html/2605.12843#bib.bib45)\], Rendered SST\-2\[[46](https://arxiv.org/html/2605.12843#bib.bib46)\], and KMNIST\[[47](https://arxiv.org/html/2605.12843#bib.bib47)\]\. Table[1](https://arxiv.org/html/2605.12843#S5.T1)in the main paper reports the performance of the merged models averaged across the 8, 14, and 20 tasks, while the “mean±\\pmstd” results and the per\-task breakdown radar charts are provided in Appendices[E\.1](https://arxiv.org/html/2605.12843#A5.SS1)and[E\.2](https://arxiv.org/html/2605.12843#A5.SS2), respectively\.

### C\.2Language Benchmarks

Table 4:Summary of training and evaluation datasets for natural language processing experiments\. Official standard splits are adopted for all benchmarks, with the exception of the GSM8k validation set \(500 samples from the training set\) and datasets marked with \* \(custom 1/3\-2/3 splits\)\.CategoryTrainingValidationTestMetricInstructionTULU\-3 Persona\[[48](https://arxiv.org/html/2605.12843#bib.bib48)\]IFEval\*\[[49](https://arxiv.org/html/2605.12843#bib.bib49)\]IFEval\*\[[49](https://arxiv.org/html/2605.12843#bib.bib49)\]Prompt Acc\.MathematicsDART\[[50](https://arxiv.org/html/2605.12843#bib.bib50)\], NuminaMath\[[51](https://arxiv.org/html/2605.12843#bib.bib51)\]GSM8k\[[52](https://arxiv.org/html/2605.12843#bib.bib52)\]GSM8k\[[52](https://arxiv.org/html/2605.12843#bib.bib52)\]EM \(8\-shot\)MultilingualAya\[[53](https://arxiv.org/html/2605.12843#bib.bib53)\]M\_MMLU\[[54](https://arxiv.org/html/2605.12843#bib.bib54)\]\(fr, es, de, ru\)M\_MMLU\[[54](https://arxiv.org/html/2605.12843#bib.bib54)\]\(fr, es, de, ru\)AccuracyCodingMagicoder\[[55](https://arxiv.org/html/2605.12843#bib.bib55)\]MBPP\[[56](https://arxiv.org/html/2605.12843#bib.bib56)\]HumanEval\+\[[57](https://arxiv.org/html/2605.12843#bib.bib57)\], MBPP\+\[[57](https://arxiv.org/html/2605.12843#bib.bib57)\]Pass@1SafetyWildGuard\[[58](https://arxiv.org/html/2605.12843#bib.bib58)\], WildJailbreak\[[59](https://arxiv.org/html/2605.12843#bib.bib59)\]HarmBench\*\[[60](https://arxiv.org/html/2605.12843#bib.bib60)\], XSTest\*\[[61](https://arxiv.org/html/2605.12843#bib.bib61)\]HarmBench\*\[[60](https://arxiv.org/html/2605.12843#bib.bib60)\], XSTest\*\[[61](https://arxiv.org/html/2605.12843#bib.bib61)\]RTA / Acc\.For the NLP experiments, Table[4](https://arxiv.org/html/2605.12843#A3.T4)summarizes the training, validation, and test splits, together with the corresponding evaluation metrics\. Our evaluation covers five task categories\. For categories involving multiple test sets, we first average the individual test scores within each category to obtain a category\-level score\. Throughout the main text, we report the macro\-average over the five category\-level scores as the overall performance, while category\-level language breakdowns are provided in Appendix[E\.2](https://arxiv.org/html/2605.12843#A5.SS2)\.

### C\.3Licenses and Asset Usage

We use existing public assets, including pretrained backbones, task\-specific checkpoints, benchmark datasets, evaluation suites, and baseline implementations\. These assets are described above, and their original creators are credited through the corresponding references\.

For vision experiments, we follow public model\-merging benchmark settings based on ViT backbones and task\-specific expert checkpoints\. For language experiments, we use Llama\-3\.2\-3B and Llama\-3\.1\-8B backbones under the corresponding Meta Llama Community License Agreements and acceptable\-use policy\. We do not redistribute third\-party checkpoints or model weights except where permitted by their original licenses\.

All datasets and evaluation suites are used only for research, calibration, validation, and evaluation\. Their source URLs, versions when applicable, and license or source\-term information are provided in the supplementary asset manifest\. Safety benchmarks may contain harmful or adversarial prompts and are used only for safety research and evaluation\.

Baseline methods and third\-party libraries are used according to their original papers, public repositories, and licenses\. We retain required copyright notices, citations, and license files where applicable\. We do not introduce new scraped datasets or human\-subject data\. All copyrights remain with the original asset owners, and all external assets are used in compliance with their respective licenses and terms of use\.

## Appendix DComputational Costs and Runtime

### D\.1Complexity Analysis

We briefly analyze the costs of the closed\-form estimators \(Eqs\.[8](https://arxiv.org/html/2605.12843#S4.E8),[15](https://arxiv.org/html/2605.12843#S4.E15)\) and the BO search \([12](https://arxiv.org/html/2605.12843#S4.E12)\)\. LetTTbe the number of tasks,MMbe the number of merged 2D modules, and𝐔m∈ℝdout,m×din,m\\mathbf\{U\}\_\{m\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\},m\}\\times d\_\{\\mathrm\{in\},m\}\}\. In the data\-assisted setting, withNNcalibration samples per task,𝐗m∈ℝdin,m×Nact\\mathbf\{X\}\_\{m\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\},m\}\\times N\_\{\\mathrm\{act\}\}\}and𝐘m∈ℝdout,m×Nact\\mathbf\{Y\}\_\{m\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\},m\}\\times N\_\{\\mathrm\{act\}\}\}, whereNact=N​TN\_\{\\mathrm\{act\}\}=NT\. We omit the one\-time forward cost for collecting activations\.

#### Closed\-form estimators\.

The data\-assisted estimator in Eq\. \([8](https://arxiv.org/html/2605.12843#S4.E8)\) first caches𝐗m​𝐗m⊤\\mathbf\{X\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}and𝐘m​𝐗m⊤\\mathbf\{Y\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}, which costs𝒪​\(Nact​din,m2\+Nact​dout,m​din,m\)\.\\mathcal\{O\}\\\!\\left\(N\_\{\\mathrm\{act\}\}d\_\{\\mathrm\{in\},m\}^\{2\}\+N\_\{\\mathrm\{act\}\}d\_\{\\mathrm\{out\},m\}d\_\{\\mathrm\{in\},m\}\\right\)\.Then, given a module\-wiseλm\\lambda\_\{m\}, solving

\(𝐗m​𝐗m⊤\+λm​𝐈\)​𝐔m⊤=\(𝐘m​𝐗m⊤\+λm​𝐔m\(0\)\)⊤,\\left\(\\mathbf\{X\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}\+\\lambda\_\{m\}\\mathbf\{I\}\\right\)\\mathbf\{U\}\_\{m\}^\{\\top\}=\\left\(\\mathbf\{Y\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}\+\\lambda\_\{m\}\\mathbf\{U\}\_\{m\}^\{\(0\)\}\\right\)^\{\\top\},costs𝒪​\(din,m3\+dout,m​din,m2\)\\mathcal\{O\}\\\!\\left\(d\_\{\\mathrm\{in\},m\}^\{3\}\+d\_\{\\mathrm\{out\},m\}d\_\{\\mathrm\{in\},m\}^\{2\}\\right\)by using a dense Cholesky solver\[[62](https://arxiv.org/html/2605.12843#bib.bib62)\]\. Therefore, assuming a square 2D module for simplicity, the total cost is𝒪​\(d3\)\\mathcal\{O\}\(d^\{3\}\)\. Similarly, the data\-free estimator in Eq\. \([15](https://arxiv.org/html/2605.12843#S4.E15)\) replaces activation statistics with the Gram\-matrix of task vectors, whose construction costs𝒪​\(T​dout,m​din,m2\)\\mathcal\{O\}\(Td\_\{\\mathrm\{out\},m\}d\_\{\\mathrm\{in\},m\}^\{2\}\)per module, followed by the same Cholesky solver, and leads to the same total cost of𝒪​\(d3\)\\mathcal\{O\}\(d^\{3\}\)\. Typically,d=1,024d=1,024in our benchmarks\. Thus, the cost of the closed\-form estimators is insignificant on modern high\-performance GPUs\.

#### BO search overhead\.

Algorithm[1](https://arxiv.org/html/2605.12843#alg1)performsKKBO trials\. The exact𝒢​𝒫\\mathcal\{GP\}update costs𝒪​\(K3\)\\mathcal\{O\}\(K^\{3\}\)due to the Cholesky factorization of the𝒢​𝒫\\mathcal\{GP\}kernel matrix\[[63](https://arxiv.org/html/2605.12843#bib.bib63),[21](https://arxiv.org/html/2605.12843#bib.bib21)\]\. In practice, this overhead is very small compared with repeated validation\-set evaluations\. Hence, the practical runtime of BO is dominated by evaluating candidate merged models on the validation set\.

### D\.2Runtime Breakdown

Table 5:Runtime breakdown of BMM on vision and language tasks\.The ViT timings are measured on a single NVIDIA A6000 GPU, while the Llama timings are reported per worker under an 8\-way parallel BO execution\.*Gram*denotes the one\-time Gram\-matrix construction cost in the data\-assisted setting\.*Closed\-form*and*GP*denote the cumulative costs of closed\-form solution and GP surrogate updates, respectively\.*Search*includes Gram construction, closed\-form solution, GP updates, model assembly, checkpoint loading/book\-keeping, and rounding\.*Val\. Cost*reports the time spent on validation\-set evaluations\.ModelTasksSettingGramClosed\-formGPSearchVal\. CostTotalViT\-B/328data\-assisted0\.4 min2\.5 min0\.4 min3\.3 min17\.3 min20\.6 minViT\-B/328data\-free–2\.2 min0\.4 min2\.9 min17\.3 min20\.2 minViT\-B/3214data\-assisted0\.6 min4\.5 min0\.4 min6\.2 min22\.3 min28\.5 minViT\-B/3214data\-free–5\.2 min0\.4 min5\.6 min22\.3 min27\.9 minViT\-B/3220data\-assisted0\.8 min4\.6 min0\.4 min8\.0 min41\.6 min49\.6 minViT\-B/3220data\-free–6\.9 min0\.4 min7\.3 min41\.6 min48\.8 minViT\-L/148data\-assisted2\.8 min8\.2 min0\.4 min17\.1 min168\.4 min185\.5 minViT\-L/148data\-free–13\.9 min0\.4 min14\.3 min168\.4 min182\.7 minViT\-L/1414data\-assisted5\.8 min12\.1 min0\.4 min18\.6 min292\.1 min310\.7 minViT\-L/1414data\-free–11\.3 min0\.4 min12\.8 min292\.1 min304\.9 minViT\-L/1420data\-assisted9\.8 min17\.0 min0\.4 min31\.7 min539\.0 min570\.7 minViT\-L/1420data\-free–21\.5 min0\.4 min21\.9 min539\.0 min560\.9 minLlama\-3\.2\-3B5data\-assisted1\.0 min12\.9 min0\.1 min14\.5 min126\.9 min141\.4 minLlama\-3\.2\-3B5data\-free–13\.4 min0\.1 min13\.5 min126\.9 min140\.3 minLlama\-3\.1\-8B5data\-assisted2\.8 min38\.4 min0\.1 min43\.4 min190\.7 min234\.1 minLlama\-3\.1\-8B5data\-free–39\.9 min0\.1 min39\.9 min190\.7 min230\.6 minTable[5](https://arxiv.org/html/2605.12843#A4.T5)reports the wall\-clock runtime of the BMM pipeline\. The vision experiments withK=200K\\\!=\\\!200BO trials are measured on a single NVIDIA RTX A6000 48GB, with validation tensors cached locally to reduce I/O overhead\. The language experiments withK=100K\\\!=\\\!100BO trials report per\-worker timings under an 8\-GPU parallel setup\. Across both vision and language benchmarks, the algorithmic overhead of BMM remains small compared with validation\-set forward passes, indicating that the main runtime cost comes from model evaluation rather than from the BMM update itself\.

### D\.3Runtime–Performance Trade\-offs

Table 6:Runtime and performance comparison of BMM and state\-of\-the\-art data\-free merging methods\. For ViT\-B/32, BMM uses ISO\-CTS as anchor; for Llama\-3\.2\-3B, BMM uses TSV as achor\. Baselines use their full grid\-search budgets, whereas BMM is evaluated with BO budgets ofK=20K=20andK=60K=60\. The ViT runtimes are measured on a single NVIDIA A6000 GPU, and the Llama runtimes report 8\-GPU parallel wall\-clock time\. Scores denote test accuracies for ViT\-B/32 and aggregate benchmark scores for Llama\-3\.2\-3B\.ModelTasksMethodSearch Dim\.\#TrialsRuntimeScoreViT\-B/3220TSV1308\.4 min76\.9ViT\-B/3220WUDI\-Merging224116\.1 min76\.1ViT\-B/3220ISO\-CTS322591\.2 min77\.6ViT\-B/3220BMM \(K=20K=20\)15205\.60 min80\.0ViT\-B/3220BMM \(K=60K=60\)156015\.55 min81\.5Llama\-3\.2\-3B5TSV13073\.4 min0\.471Llama\-3\.2\-3B5WUDI\-Merging224418\.8 min0\.461Llama\-3\.2\-3B5ISO\-CTS3225318\.0 min0\.441Llama\-3\.2\-3B5BMM \(K=20K=20\)52028\.6 min0\.465Llama\-3\.2\-3B5BMM \(K=60K=60\)56084\.8 min0\.473As shown in Figure[3](https://arxiv.org/html/2605.12843#S5.F3)\(right\) and Table[6](https://arxiv.org/html/2605.12843#A4.T6), BMM provides favorable runtime–performance trade\-offs in the data\-free setting, even under constrained BO budgets\. On the 20\-task ViT\-B/32 benchmark, BMM with onlyK=20K=20trials outperforms all grid\-search anchors while requiring substantially less wall\-clock time\. Increasing the budget toK=60K=60further improves the score to81\.52%81\.52\\%\. This efficiency stems from BMM’s closed\-form updates and BO\-based search, which avoid the main bottlenecks of baseline methods: WUDI\-Merging relies on costly iterative SGD optimization, whereas ISO\-CTS requires exhaustive grid search over225225configurations\. For Llama\-3\.2\-3B, BMM also substantially reduces search cost\. WithK=20K=20, it reaches a score close to the strongest TSV baseline in less than half of TSV’s wall\-clock time and far less time than WUDI\-Merging or ISO\-CTS\. These results suggest that BMM’s closed\-form solution combined with BO\-based search provides an efficient model merging method across both vision and language benchmarks\.

## Appendix EAdditional Experimental Results

### E\.1Detailed results on vision tasks with standard deviations

Table 7:Model merging on vision tasks with ViT architectures\. Values represent the mean accuracy over five independent runs \(seeds 0–4\)\. Black subscripts indicate the standard deviation, which are omitted in Table[1](https://arxiv.org/html/2605.12843#S5.T1)due to lack of space\. \(Indiv\.: Individual\)ModelTasksSettingIndiv\.PretrainedTATIESRegMeanTSVWUDIISO\-CTSViT\-B/328anchor92\.847\.770\.475\.782\.385\.987\.086\.4data\-assisted\-84\.8±0\.02\\pm 0\.0285\.0±0\.05\\pm 0\.0585\.7±0\.03\\pm 0\.0385\.2±0\.05\\pm 0\.0588\.9±0\.05\\pm 0\.0589\.4±0\.02\\pm 0\.0290\.2±0\.03\\pm 0\.03data\-free\-87\.9±0\.06\\pm 0\.0687\.0±0\.05\\pm 0\.0587\.0±0\.06\\pm 0\.0687\.9±0\.09\\pm 0\.0987\.8±0\.04\\pm 0\.0487\.6±0\.04\\pm 0\.0489\.0±0\.03\\pm 0\.0314anchor90\.956\.965\.268\.276\.679\.980\.581\.5data\-assisted\-78\.9±0\.03\\pm 0\.0379\.2±0\.04\\pm 0\.0479\.7±0\.04\\pm 0\.0479\.1±0\.02\\pm 0\.0284\.3±0\.04\\pm 0\.0484\.6±0\.04\\pm 0\.0485\.5±0\.06\\pm 0\.06data\-free\-82\.7±0\.06\\pm 0\.0682\.4±0\.03\\pm 0\.0382\.3±0\.10\\pm 0\.1082\.9±0\.09\\pm 0\.0982\.8±0\.03\\pm 0\.0381\.7±0\.06\\pm 0\.0684\.3±0\.05\\pm 0\.0520anchor91\.355\.760\.464\.072\.276\.976\.177\.6data\-assisted\-75\.6±0\.05\\pm 0\.0575\.5±0\.04\\pm 0\.0476\.3±0\.03\\pm 0\.0375\.9±0\.06\\pm 0\.0682\.0±0\.04\\pm 0\.0482\.2±0\.06\\pm 0\.0682\.8±0\.05\\pm 0\.05data\-free\-80\.0±0\.06\\pm 0\.0678\.4±0\.06\\pm 0\.0679\.2±0\.05\\pm 0\.0579\.7±0\.03\\pm 0\.0379\.7±0\.05\\pm 0\.0578\.0±0\.02\\pm 0\.0281\.5±0\.04\\pm 0\.04ViT\-L/148anchor95\.865\.184\.887\.090\.093\.094\.094\.8data\-assisted\-92\.3±0\.01\\pm 0\.0192\.8±0\.05\\pm 0\.0593\.1±0\.03\\pm 0\.0392\.7±0\.03\\pm 0\.0394\.4±0\.03\\pm 0\.0394\.7±0\.02\\pm 0\.0295\.1±0\.03\\pm 0\.03data\-free\-94\.4±0\.02\\pm 0\.0294\.2±0\.01\\pm 0\.0194\.2±0\.03\\pm 0\.0394\.0±0\.02\\pm 0\.0294\.4±0\.01\\pm 0\.0194\.3±0\.04\\pm 0\.0495\.0±0\.02\\pm 0\.0214anchor94\.368\.579\.480\.285\.589\.190\.591\.0data\-assisted\-88\.0±0\.01\\pm 0\.0188\.3±0\.03\\pm 0\.0388\.3±0\.03\\pm 0\.0388\.2±0\.02\\pm 0\.0291\.6±0\.01\\pm 0\.0191\.7±0\.03\\pm 0\.0392\.2±0\.07\\pm 0\.07data\-free\-90\.9±0\.08\\pm 0\.0890\.9±0\.04\\pm 0\.0490\.8±0\.02\\pm 0\.0290\.6±0\.05\\pm 0\.0591\.3±0\.05\\pm 0\.0591\.0±0\.06\\pm 0\.0692\.1±0\.06\\pm 0\.0620anchor94\.765\.474\.076\.782\.887\.788\.490\.1data\-assisted\-86\.0±0\.02\\pm 0\.0286\.2±0\.03\\pm 0\.0386\.4±0\.01\\pm 0\.0186\.2±0\.01\\pm 0\.0190\.9±0\.03\\pm 0\.0390\.8±0\.02\\pm 0\.0291\.5±0\.01\\pm 0\.01data\-free\-89\.8±0\.02\\pm 0\.0289\.6±0\.01\\pm 0\.0189\.5±0\.01\\pm 0\.0189\.4±0\.04\\pm 0\.0490\.3±0\.02\\pm 0\.0289\.7±0\.02\\pm 0\.0291\.1±0\.11\\pm 0\.11

Table[7](https://arxiv.org/html/2605.12843#A5.T7)provides the comprehensive results corresponding to Table[1](https://arxiv.org/html/2605.12843#S5.T1), including standard deviations across five random seeds \(0–4\)\. The results demonstrate minimal variance across all settings, with standard deviations predominantly ranging from±0\.01\\pm 0\.01to±0\.06\\pm 0\.06\. The low variance confirms that the performance improvements achieved by BMM are stable and robust, rather than artifacts of specific seed initializations\.

### E\.2Radar Charts: Per\-Task Breakdowns

Tables[1](https://arxiv.org/html/2605.12843#S5.T1)and[2](https://arxiv.org/html/2605.12843#S5.T2)report the performance of merged models averaged across multiple tasks\. However, such summaries may hide task\-specific trade\-offs\. To address this issue, Figures[5](https://arxiv.org/html/2605.12843#A5.F5)–[8](https://arxiv.org/html/2605.12843#A5.F8)provide vision per\-task radar charts for all evaluated ViT settings\. Similarly, Figures[9](https://arxiv.org/html/2605.12843#A5.F9)–[10](https://arxiv.org/html/2605.12843#A5.F10)provide language per\-task radar charts for all evaluated Llama settings\.

#### Key Observations\.

In most evaluated settings, BMM exhibits a Pareto dominance over the corresponding anchor baselines, improving every task without reducing performance on any individual task\. This indicates that the average gains reported in Tables[1](https://arxiv.org/html/2605.12843#S5.T1)and[2](https://arxiv.org/html/2605.12843#S5.T2)generally reflect a broad multi\-task expansion rather than a redistribution of performance across tasks\. The strong overlap between the data\-assisted and data\-free curves further suggests that our data\-free surrogate captures much of the task geometry revealed by the calibration\-based estimate, enabling competitive data\-free model merging without any auxiliary calibration data\.

AnchorData\-assisted BMMData\-free BMM

8 Tasks14 Tasks20 TasksPretrained\(0–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x4.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x5.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x6.png)TA\(10–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x7.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x8.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x9.png)TIES\(10–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x10.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x11.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x12.png)RegMean\(20–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x13.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x14.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x15.png)TSV\(50–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x16.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x17.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x18.png)

Figure 5:Radar charts: ViT\-B/32 per\-task breakdowns\(corresponding to Table[1](https://arxiv.org/html/2605.12843#S5.T1)\)\. The radial axis limits \(min–max\) are annotated on the left\.AnchorData\-assisted BMMData\-free BMM

8 Tasks14 Tasks20 TasksWUDI\(50–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x19.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x20.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x21.png)ISO\-CTS\(50–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x22.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x23.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x24.png)

Figure 6:Radar charts: ViT\-B/32 per\-task breakdowns\(corresponding to Table[1](https://arxiv.org/html/2605.12843#S5.T1)\)\. Continued from Figure[5](https://arxiv.org/html/2605.12843#A5.F5)on the remaining anchor models: WUDI and ISO\-CTS\.8 Tasks14 Tasks20 TasksPretrained\(10–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x25.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x26.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x27.png)TA\(10–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x28.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x29.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x30.png)
Figure 7:Radar charts: ViT\-L/14 per\-task breakdowns\(corresponding to Table[1](https://arxiv.org/html/2605.12843#S5.T1)\)\. Layout, legend, and row\-specific axis scaling follow Figure[5](https://arxiv.org/html/2605.12843#A5.F5)\.AnchorData\-assisted BMMData\-free BMM

8 Tasks14 Tasks20 TasksTIES\(30–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x31.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x32.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x33.png)RegMean\(40–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x34.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x35.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x36.png)TSV\(60–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x37.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x38.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x39.png)WUDI\(60–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x40.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x41.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x42.png)ISO\-CTS\(60–100\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x43.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x44.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x45.png)

Figure 8:Radar charts: ViT\-L/14 per\-task breakdowns\(corresponding to Table[1](https://arxiv.org/html/2605.12843#S5.T1)\)\. Continued from Figure[7](https://arxiv.org/html/2605.12843#A5.F7)on the remaining anchor models\.AnchorData\-assisted BMMData\-free BMM

Llama\-3\.2\-3BLlama\-3\.1\-8BPretrained\(0–60\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x46.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x47.png)TA\(20–80\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x48.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x49.png)TIES\(20–80\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x50.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x51.png)RegMean\(10–80\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x52.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x53.png)

Figure 9:Radar charts: Llama per\-task breakdowns\(corresponding to Table[2](https://arxiv.org/html/2605.12843#S5.T2)\)\. The radial axis limits \(min\-max\) are annotated on the left\. “IF” denotes Instruction Following\.AnchorData\-assisted BMMData\-free BMM

Llama\-3\.2\-3BLlama\-3\.1\-8BTSV\(30–80\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x54.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x55.png)WUDI\(20–80\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x56.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x57.png)ISO\-CTS\(20–80\)![Refer to caption](https://arxiv.org/html/2605.12843v1/x58.png)![Refer to caption](https://arxiv.org/html/2605.12843v1/x59.png)

Figure 10:Radar charts: Llama per\-task breakdowns\(corresponding to Table[2](https://arxiv.org/html/2605.12843#S5.T2)\)\. Continued from Figure[9](https://arxiv.org/html/2605.12843#A5.F9)on the remaining anchor models\. “IF” denotes Instruction Following\.

## Appendix FA Hybrid Estimate of Gram Matrix in Few\-Shot Regimes

Theorem[1](https://arxiv.org/html/2605.12843#Thmtheorem1)indicates that the Gram matrix𝔼​\[𝐱𝐱⊤\]\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\]can be estimated empirically from few\-shot calibration samples \(data\-assisted\) or approximated by the expert\-weight surrogate𝐔⊤​𝐔\\mathbf\{U\}^\{\\top\}\\mathbf\{U\}\(data\-free\)\. Therefore, a hybrid estimate of Gram matrix may be worthy of investigation\. We consider amixvariant that interpolates between the few\-shot empirical estimate and the data\-free surrogate:

𝐆mmix​\(ϵ\)=ϵ​𝐆mfew\+\(1−ϵ\)​𝐆mdf,ϵ∈\[0,1\],\\mathbf\{G\}^\{\\mathrm\{mix\}\}\_\{m\}\(\\epsilon\)=\\epsilon\\,\\mathbf\{G\}^\{\\mathrm\{few\}\}\_\{m\}\+\(1\-\\epsilon\)\\,\\mathbf\{G\}^\{\\mathrm\{df\}\}\_\{m\},\\quad\\epsilon\\in\[0,1\],\(35\)where𝐆mfew=𝐗m​𝐗m⊤\\mathbf\{G\}^\{\\mathrm\{few\}\}\_\{m\}=\\mathbf\{X\}\_\{m\}\\mathbf\{X\}\_\{m\}^\{\\top\}is estimated from few\-shot calibration samples,𝐆mdf=∑t=1T\(𝐔m\(t\)\)⊤​𝐔m\(t\)\\mathbf\{G\}^\{\\mathrm\{df\}\}\_\{m\}=\\sum\_\{t=1\}^\{T\}\(\\mathbf\{U\}^\{\(t\)\}\_\{m\}\)^\{\\top\}\\mathbf\{U\}^\{\(t\)\}\_\{m\}is the data\-free estimate, and the mixing weightϵ\\epsilonis optimized by BO\.

Table[8](https://arxiv.org/html/2605.12843#A6.T8)shows thismixvariant consistently improves upon the pure data\-free variant and often enhances few\-shot results, particularly for TSV and ISO\-CTS\. This is likely because the 1\-shot or 5\-shot empirical estimates are often noisy and rank\-deficient, and the data\-free surrogate provides a reliable estimate based on the weights of expert models\. As can be seen,1\-shot mixboosts 8\-task TSV accuracy from 87\.8% to 88\.9%, and5\-shot miximproves 20\-task ISO\-CTS from 82\.2% to 83\.7%\. Although themixvariant is very close to pure few\-shot estimates for WUDI in certain cases, its overall robustness confirms that blending limited empirical statistics with data\-free surrogate is highly effective in few\-shot regimes\.

Table 8:Different Gram matrix estimation strategies and their impacts on 8\-, 14\-, and 20\-task merging with ViT\-B/32 architecture\. Themixvariant interpolates between data\-assisted Gram matrix estimate \(1\-shot or 5\-shot\) and data\-free estimate by a BO\-optimized weight\. Results for data\-free and data\-assisted variants are reported as mean±\\pmstd over seeds 0–4\.Setting8 Tasks14 Tasks20 TasksTSVWUDIISO\-CTSTSVWUDIISO\-CTSTSVWUDIISO\-CTSanchor85\.987\.086\.479\.980\.581\.576\.976\.177\.6data\-free87\.8±0\.0487\.8\_\{\\pm 0\.04\}87\.6±0\.0487\.6\_\{\\pm 0\.04\}89\.0±0\.0389\.0\_\{\\pm 0\.03\}82\.8±0\.0382\.8\_\{\\pm 0\.03\}81\.7±0\.0681\.7\_\{\\pm 0\.06\}84\.3±0\.0584\.3\_\{\\pm 0\.05\}79\.7±0\.0579\.7\_\{\\pm 0\.05\}78\.0±0\.0278\.0\_\{\\pm 0\.02\}81\.5±0\.0481\.5\_\{\\pm 0\.04\}1\-shot87\.8±0\.0687\.8\_\{\\pm 0\.06\}89\.0±0\.02\\mathbf\{89\.0\}\_\{\\pm\\mathbf\{0\.02\}\}88\.7±0\.0388\.7\_\{\\pm 0\.03\}83\.2±0\.1583\.2\_\{\\pm 0\.15\}84\.1±0\.06\\mathbf\{84\.1\}\_\{\\pm\\mathbf\{0\.06\}\}84\.2±0\.0784\.2\_\{\\pm 0\.07\}80\.6±0\.0680\.6\_\{\\pm 0\.06\}81\.1±0\.04\\mathbf\{81\.1\}\_\{\\pm\\mathbf\{0\.04\}\}80\.5±0\.0680\.5\_\{\\pm 0\.06\}1\-shot mix88\.9±0\.05\\mathbf\{88\.9\}\_\{\\pm\\mathbf\{0\.05\}\}89\.0±0\.08\\mathbf\{89\.0\}\_\{\\pm\\mathbf\{0\.08\}\}89\.7±0\.05\\mathbf\{89\.7\}\_\{\\pm\\mathbf\{0\.05\}\}84\.3±0\.07\\mathbf\{84\.3\}\_\{\\pm\\mathbf\{0\.07\}\}84\.1±0\.05\\mathbf\{84\.1\}\_\{\\pm\\mathbf\{0\.05\}\}85\.7±0\.04\\mathbf\{85\.7\}\_\{\\pm\\mathbf\{0\.04\}\}81\.5±0\.15\\mathbf\{81\.5\}\_\{\\pm\\mathbf\{0\.15\}\}81\.1±0\.03\\mathbf\{81\.1\}\_\{\\pm\\mathbf\{0\.03\}\}82\.8±0\.03\\mathbf\{82\.8\}\_\{\\pm\\mathbf\{0\.03\}\}5\-shot88\.5±0\.1188\.5\_\{\\pm 0\.11\}89\.3±0\.06\\mathbf\{89\.3\}\_\{\\pm\\mathbf\{0\.06\}\}89\.5±0\.0389\.5\_\{\\pm 0\.03\}84\.1±0\.0484\.1\_\{\\pm 0\.04\}84\.6±0\.07\\mathbf\{84\.6\}\_\{\\pm\\mathbf\{0\.07\}\}85\.3±0\.0485\.3\_\{\\pm 0\.04\}81\.7±0\.0281\.7\_\{\\pm 0\.02\}81\.9±0\.04\\mathbf\{81\.9\}\_\{\\pm\\mathbf\{0\.04\}\}82\.2±0\.0282\.2\_\{\\pm 0\.02\}5\-shot mix89\.4±0\.11\\mathbf\{89\.4\}\_\{\\pm\\mathbf\{0\.11\}\}89\.3±0\.07\\mathbf\{89\.3\}\_\{\\pm\\mathbf\{0\.07\}\}90\.2±0\.02\\mathbf\{90\.2\}\_\{\\pm\\mathbf\{0\.02\}\}84\.9±0\.11\\mathbf\{84\.9\}\_\{\\pm\\mathbf\{0\.11\}\}84\.6±0\.08\\mathbf\{84\.6\}\_\{\\pm\\mathbf\{0\.08\}\}86\.1±0\.06\\mathbf\{86\.1\}\_\{\\pm\\mathbf\{0\.06\}\}82\.4±0\.03\\mathbf\{82\.4\}\_\{\\pm\\mathbf\{0\.03\}\}81\.9±0\.03\\mathbf\{81\.9\}\_\{\\pm\\mathbf\{0\.03\}\}83\.7±0\.02\\mathbf\{83\.7\}\_\{\\pm\\mathbf\{0\.02\}\}