Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

# Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
Source: [https://arxiv.org/html/2607.11997](https://arxiv.org/html/2607.11997)
###### Abstract

Multi\-task model merging combines separately trained expert models into a single model that handles all tasks without co\-training\. Standard practice merges experts at their optimal validation loss\. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model\. We fine\-tune experts on five domains \(Math, Code, Instruction Following, Multilingual, and Safety\) across three model sizes \(Qwen 3\.5 0\.8B, 2B, and 4B\), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration\. Our findings reveal a striking method\-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification\-based methods achieve their best performance well past the validation optimum\. We formalize this through bias\-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high\-variance individual learners\. These results suggest that training duration and merging method should be chosen jointly rather than independently\.

## 1Introduction

Model merging has emerged as a practical approach for combining expert models trained separately into a single multi\-task model\(Wortsmanet al\.,[2022](https://arxiv.org/html/2607.11997#bib.bib1); Ilharcoet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib2)\)\. By operating directly in weight space, merging avoids the computational overhead of multi\-task joint training and the data\-sharing requirements that come with it\. Unlike model ensembling or mixture\-of\-experts approaches, weight\-space merging does not increase inference latency or memory footprint, since all individual experts are merged into a single model\. These benefits make model merging particularly attractive for Large Language Models \(LLMs\), where training costs are substantial, and inference latency must remain low\.

A common assumption in the model merging literature is that domain experts should be trained to optimal validation loss before merging\. This convention is implicit in major benchmarks and method comparisons: experts are fine\-tuned with standard early\-stopping or fixed\-epoch protocols and then frozen at their best validation checkpoint before being combined\(He and others,[2025](https://arxiv.org/html/2607.11997#bib.bib5); Yadavet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib3); Yuet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib4)\)\. The implicit assumption is that each expert contributes its best possible knowledge to the merged model when frozen at its individual optimum\. Recent work has begun to question this assumption:Horoiet al\.\([2025](https://arxiv.org/html/2607.11997#bib.bib6)\)show, for vision \(CLIP\) and encoder\-decoder \(T5\) models, that experts trained well past their individual optimum can in fact*hurt*the merged model\. We revisit the question for decoder\-only LLMs fine\-tuned with parameter\-efficient adapters and find a more nuanced picture in which the right training duration depends strongly on the merging method\.

This paper challenges this assumption\. Drawing an analogy from ensemble learning theory, where random forests deliberately grow each decision tree to high depth before averaging predictions\(Breiman,[2001](https://arxiv.org/html/2607.11997#bib.bib20)\), we hypothesize that the optimal training duration for merging may differ from the optimal duration for individual model performance\. We investigate whether a similar mechanism operates in weight space when merging fine\-tuned LLM experts: do some merging methods behave like the averaging step in a random forest and benefit from intentionally overtrained experts?

We focus on small open\-weight LLMs because of their prevalence in industrial fine\-tuning pipelines\. Enterprises commonly fine\-tune models below 8B to match or exceed the task\-specific quality of larger general\-purpose models at a fraction of cost and latency\(Belcaket al\.,[2025](https://arxiv.org/html/2607.11997#bib.bib22); Abdinet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib24)\)\. Model merging is especially relevant in this regime, since deploying multiple specialized models is typically more expensive than one merged model with comparable capabilities\. We conduct a systematic study across 3 model sizes \(Qwen 3\.5 0\.8B, 2B, 4B\), 5 task domains \(Math, Code, Instruction Following, Multilingual, Safety\) and 5 merging methods \(Simple Averaging, Task Arithmetic, TIES\-Merging, DARE\+TIES, Greedy Soup\)\.

Our contributions are as follows\. First, we present a systematic study of how training duration of domain experts affects the quality of merged decoder\-only LLMs, spanning different model sizes, domains, and merging methods\. Second, we show that the optimal training duration is method\-dependent: Simple Averaging peaks with undertrained experts, Task Arithmetic and Greedy Soup occupy an intermediate regime, while sparsification\-based methods \(TIES, DARE\+TIES\) peak with overfitted experts\. We explain these results through bias\-variance\-covariance decomposition and mode connectivity analysis, showing that overfitted experts provide higher diversity that benefits methods with interference\-resolution mechanisms, analogous to how variance reduction enables overtrained trees in random forests\. Together, these findings translate into practical guidance: with sparsification\-based merging, train each expert past the validation optimum rather than early\-stopping\.

## 2Related Work

Weight\-space model merging combines separately fine\-tuned models without retraining\. Model Soups\(Wortsmanet al\.,[2022](https://arxiv.org/html/2607.11997#bib.bib1)\)demonstrated that averaging weights of models fine\-tuned with different hyperparameters improves accuracy and robustness\. Recent work has proposed merging methods that go beyond simple averaging\. Task Arithmetic\(Ilharcoet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib2)\)formalizes task vectors \(weight deltas from pre\-training\) that can be added to compose multi\-task models\. TIES\-Merging\(Yadavet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib3)\)addresses interference between task vectors through trimming, sign election, and disjoint merging\. DARE\(Yuet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib4)\)randomly drops and rescales delta parameters before merging\. The recent MergeBench suite\(He and others,[2025](https://arxiv.org/html/2607.11997#bib.bib5)\)provides a comprehensive evaluation of 8 merging methods across 5 domains, which we adopt for our experiments\. Most of these works fix the merging recipe and produce experts through a standard fine\-tuning protocol with early stopping or a fixed budget chosen for individual model quality, treating the choice of training duration as orthogonal to the merging step\. ATM\(Zhouet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib7)\)instead interleaves tuning and merging, reinterpreting task arithmetic as a single noisy gradient step toward a joint objective, which similarly couples the optimization trajectory to the merge\. We instead hold the merging methods fixed and vary expert training duration explicitly, and show that this dimension is in fact tightly coupled to the choice of merging method\.

A separate line of work studies weight averaging along a single training trajectory\. Stochastic Weight Averaging \(SWA\)\(Izmailovet al\.,[2018](https://arxiv.org/html/2607.11997#bib.bib9)\)averages checkpoints from late training to reach better\-generalizing minima, and LAWA\(Kaddour,[2022](https://arxiv.org/html/2607.11997#bib.bib12)\)extends this to maintain a FIFO queue of recent checkpoints, which yields gains even when individual checkpoints are past validation optimum\. Most directly related to our motivation, Post\-Hoc Reversal\(Ranjanet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib11)\)demonstrates validation\-optimal single model checkpoints are not the validation\-optimal checkpoints for an ensemble of those models, suggesting that selection criteria designed for individual models can be misleading once aggregation is involved\. All of these works study*single\-trajectory temporal*averaging within one task\. We instead study*cross\-task*merging where experts are trained on disjoint data, and find the stopping rule for the merged model\.

Most directly related to our work,Horoiet al\.\([2025](https://arxiv.org/html/2607.11997#bib.bib6)\)also vary expert training duration across merging methods, and find that overtrained experts degrade merging for CLIP and T5 models under full fine\-tuning and LoRA, tracing the effect to the memorization of difficult examples that merging discards\. Our study examines the same question for substantially larger decoder\-only LLMs fine\-tuned with quantized low\-rank adapters, and finds that the picture is method\-dependent in this regime; we discuss the relationship to their findings in Section[4](https://arxiv.org/html/2607.11997#S4)\.

DiWA\(Rameet al\.,[2022](https://arxiv.org/html/2607.11997#bib.bib10)\)provides a bias\-variance\-covariance decomposition for weight\-averaged models, showing that averaging succeeds when variance dominates the error budget\.Tranet al\.\([2026](https://arxiv.org/html/2607.11997#bib.bib17)\)explicitly analyze model soups through this decomposition\. We apply this framework to understand why overfitted experts may benefit sparsification\-based model merging methods\.

## 3Experimental Setup

We use the Qwen 3\.5 model family\(Qwen Team,[2026](https://arxiv.org/html/2607.11997#bib.bib25)\)at three scales: 0\.8B, 2B, and 4B total parameters\. All experts are fine\-tuned with QLoRA\(Dettmerset al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib18)\), i\.e\. low\-rank adapters at rankr=16r=16on top of a 4\-bit quantized base model, targeting all attention and MLP projection modules\. We use a constant learning rate of2×10−42\\times 10^\{\-4\}with a 100\-step linear warmup, an effective batch size of 8, and the AdamW optimizer\. The constant schedule with no decay ensures the learning rate remains active past the optimal validation loss checkpoint, allowing each expert to overfit\.

Following the setup of MergeBench\(He and others,[2025](https://arxiv.org/html/2607.11997#bib.bib5)\), we adopt 5 task domains, each with a dedicated fine\-tuning dataset and evaluation benchmark\. For each dataset, we fine\-tune a Qwen 3\.5 expert using only the training data from that domain\. All individual experts are then merged using all 5 domain LoRA adapters\. We aggregate metrics over the 4 domains with accuracy metrics; the Safety domain measures the refuse\-to\-answer rate \(higher is better\) and is reported in Section[4\.2](https://arxiv.org/html/2607.11997#S4.SS2)\. For each domain and model size, we identify the validation optimum T\* independently, defined as the training step with minimum loss on that domain’s held\-out validation split \(1K examples\)\. Because T\* is selected separately per domain and size, the absolute number of steps it corresponds to varies across experts, so all training durations are expressed as multiples of each expert’s own T\* rather than as absolute step counts\. We continue training up to 5×\\timesT\* and save LoRA adapters at 8 checkpoints: \{0\.25, 0\.5, 0\.75, 1\.0, 1\.5, 2\.0, 3\.0, 5\.0\}×\\timesT\*, spanning from heavily undertrained to extremely overfitted experts\.

DomainTraining DataEvaluation BenchmarkMetricMathematicsDART\-Math\(Wei and others,[2024](https://arxiv.org/html/2607.11997#bib.bib31)\)GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.11997#bib.bib27)\)Exact\-match accuracyCodeMagicoder\(Luoet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib32)\)HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2607.11997#bib.bib28)\)Pass@1InstructionTULU\-3\(Lambertet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib33)\)IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib29)\)Prompt\-level accuracyMultilingualAya\(Singhet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib35)\)ARC\(Clarket al\.,[2018](https://arxiv.org/html/2607.11997#bib.bib30)\)Normalized accuracySafetyBeaverTails\(Jiet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib34)\)HarmBench\(Mazeikaet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib36)\)Refuse\-to\-answer rateTable 1:Task domains, datasets, and benchmarks\. Training datasets are subsampled to 10K training and 1K validation examples\.We evaluate 5 merging methods spanning different families:Simple Averaging\(Wortsmanet al\.,[2022](https://arxiv.org/html/2607.11997#bib.bib1)\): equal\-weight linear combination of experts \(wi=1/Nw\_\{i\}=1/N\),Task Arithmetic\(Ilharcoet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib2)\): scales task vectors byλ\\lambdabefore combining withλ∈\{0\.5,1\.0,1\.5\}\\lambda\\in\\\{0\.5,1\.0,1\.5\\\}selected via grid search,TIES\(Yadavet al\.,[2023](https://arxiv.org/html/2607.11997#bib.bib3)\): trims small\-magnitude parameters, resolves sign conflicts, merging values with densityk=0\.5k=0\.5,DARE\+TIES\(Yuet al\.,[2024](https://arxiv.org/html/2607.11997#bib.bib4)\): randomly drops and rescales delta parameters before TIES merging with densityk=0\.5k=0\.5andGreedy Soup\(Wortsmanet al\.,[2022](https://arxiv.org/html/2607.11997#bib.bib1)\): iteratively adds experts to the merge, keeping each only if it improves the held\-out score of the merged model\.

## 4Results

Here we first show results for the average multi\-domain accuracy of merged models as a function of training duration, broken down by merging method and model size\. We aggregate the quality scores over the four domains: Math, Code, Instruction, Multilingual\. Full results are provided in Appendix[A](https://arxiv.org/html/2607.11997#A1)\. Then, we proceed to show results on refuse\-to\-answer\(%\) on HarmBench across training durations\.

### 4\.1Training duration vs\. merged model quality

![Refer to caption](https://arxiv.org/html/2607.11997v1/x1.png)Figure 1:Effect of training duration on merged model performance\. Each line represents one merging method; the x\-axis shows training steps as a multiple of T\* \(optimal validation checkpoint per expert\)\. Green shading indicates the undertrained regime \(<<T\*\); red indicates the overfitted regime \(\>\>T\*\)\. Results are averaged over the four accuracy\-based domains\. Optimal T steps is shown as method\-specific\.The results in Figure[1](https://arxiv.org/html/2607.11997#S4.F1)reveal a striking method\-dependent pattern\. Simple Averaging achieves its best performance in the undertrained regime \(0\.25–0\.75×\\timesT\*\) and degrades with longer training, losing 14–23 percentage points on average by 5×\\timesT\* \(and up to 45 points on individual domains such as Math\)\. In contrast, TIES\-Merging peaks well past T\* \(at 1\.5–3×\\timesT\* for 0\.8B and 2B, and 3–5×\\timesT\* for 4B\), exceeding its T\* score by 2\.5–8\.7 percentage points\. DARE\+TIES shows a similar but flatter profile, remaining stable across training durations with a slight preference for overfitted experts\. Task Arithmetic occupies an intermediate regime: it degrades on some domains \(especially Math, where it peaks at 0\.25×\\timesT\*\) but tolerates moderate overfitting on others\. Greedy Soup, which selectively includes experts based on validation improvement, tends to favor undertrained experts for smaller LLM sizes but shows a more balanced selection at 4B\. Its curve is notably less smooth than the other methods because the greedy selection includes different subsets of experts at each training duration\.

We attribute this discrepancy to how each method handles inter\-expert interference\. Simple Averaging treats every parameter of every expert with equal weight, so any noisy or specialized parameter in an overtrained expert directly pollutes the merged adapter\. Task Arithmetic applies a global scaleλ\\lambdato the task vectors before combining, which partially attenuates overfitting\-induced noise but cannot resolve sign conflicts or selectively suppress interfering parameters, placing it between Simple Averaging and sparsification methods\. TIES and DARE\+TIES, by contrast, include explicit interference\-resolution steps: TIES trims small\-magnitude updates and resolves sign conflicts, while DARE first stochastically prunes deltas and rescales the survivors\. These steps act as a variance\-reduction mechanism over the experts: when overfitting introduces idiosyncratic, low\-magnitude or sign\-conflicting components, sparsification removes them before the average is taken\. The result is the same intuition that motivates random forests\(Breiman,[2001](https://arxiv.org/html/2607.11997#bib.bib20)\): deeper, higher\-variance individual learners are tolerated when the aggregation step explicitly suppresses the noise they introduce\. Greedy Soup occupies a middle ground because its selection criterion is the held\-out validation score of the merge, not of individual experts, so it implicitly co\-optimizes training duration and merging composition, but only across a small discrete set of choices\.

These results partly agree with and partly diverge fromHoroiet al\.\([2025](https://arxiv.org/html/2607.11997#bib.bib6)\), who study the same question on smaller CLIP and T5 models\. We agree that averaging\-style methods \(Simple Averaging, Greedy Soup\) favor undertrained experts, but diverge for the sparsification\-based methods: where they report that TIES and DARE degrade with overtraining, we find that TIES and DARE\+TIES improve\. This divergence is consistent with their own observation that a higher LoRA rank attenuates the overtraining penalty, since we use a larger rank \(r=16r=16vs\.r=8r=8\) on substantially larger decoder\-only models with quantized adapters and a constant learning rate\.

### 4\.2Training duration vs\. merged model safety

The results in Table[2](https://arxiv.org/html/2607.11997#S4.T2)echo the accuracy pattern with one notable shift\. Simple Averaging diffuses the safety signal across all parameters and peaks near T\* \(1×\\timesfor 2B and 4B, 2×\\timesfor 0\.8B\), leaving the merged model at or below 25% refusal even at 4B\. Task Arithmetic preserves more safety than Simple Averaging and shows a clear preference for longer training at larger scales: it peaks at 2×\\timesfor 0\.8B \(10%\) but reaches 30\.5% at 3×\\timesfor 2B and 50% at 3×\\timesfor 4B\. TIES and DARE\+TIES retain the most refusal behavior and, like in the accuracy setting, prefer overtrained experts, with the optimum drifting later as model size grows\. DARE\+TIES peaks at 1\.5×\\timesfor 0\.8B and 5×\\timesfor both 2B and 4B\. The 4B TIES merge reaches 68\.5% refusal at 3×\\times, more than 2\.5×\\timeshigher than Simple Averaging at any duration\.

Method0\.25×\\times0\.5×\\times0\.75×\\times1\.0×\\times1\.5×\\times2\.0×\\times3\.0×\\times5\.0×\\timesQwen3\.5\-0\.8BSimple Avg\.2\.56\.54\.57\.05\.08\.53\.54\.5Task Arithm\.5\.05\.54\.59\.05\.010\.08\.07\.0TIES22\.521\.519\.020\.024\.015\.517\.521\.0DARE\+TIES21\.022\.020\.519\.023\.522\.521\.019\.5Greedy Soup2\.55\.07\.06\.04\.09\.05\.52\.0Qwen3\.5\-2BSimple Avg\.5\.08\.510\.012\.511\.010\.08\.53\.0Task Arithm\.9\.513\.018\.030\.026\.529\.530\.530\.0TIES35\.035\.033\.536\.027\.020\.516\.523\.5DARE\+TIES34\.034\.037\.037\.541\.036\.040\.545\.5Greedy Soup3\.59\.011\.012\.510\.09\.59\.02\.5Qwen3\.5\-4BSimple Avg\.24\.020\.521\.525\.522\.014\.017\.06\.0Task Arithm\.49\.044\.048\.542\.046\.039\.050\.044\.0TIES51\.551\.057\.553\.560\.564\.068\.564\.0DARE\+TIES50\.049\.053\.053\.555\.055\.060\.061\.5Greedy Soup22\.520\.022\.523\.021\.022\.517\.07\.0

Table 2:Refuse\-to\-answer rate \(%\) on HarmBench across training durations and model sizes\. Best per row in bold\.Greedy Soup behaves similarly to Simple Averaging on safety: peaking near T\* \(1×\\timesfor 2B and 4B at 12\.5% and 23% respectively, 2×\\timesfor 0\.8B at 9%\) and retains little refusal at extreme durations\. This is because greedy selection optimizes for accuracy\-based validation scores, which may actively exclude the safety expert when it hurts accuracy, removing the refusal signal\. This is consistent with the interpretation that the safety expert produces a localized parameter update \(refusal\), and sparsification preserves this signal even when other experts are overtrained, so the same recipe which helps accuracy also strengthens safety\.

## 5Analysis

### 5\.1Bias\-variance decomposition

To understand why overfitted experts benefit sparsification\-based merging, we apply the bias\-variance\-covariance decomposition from DiWA\(Rameet al\.,[2022](https://arxiv.org/html/2607.11997#bib.bib10)\)to our setting\. For each training duration, we decompose the merged model’s expected error into three components:Bias: how far the average expert prediction is from ground truth which decreases with training,Variance: how much individual expert predictions differ which increases with overfitting andCovariance: the correlation of experts’ errors\.

Figure[2](https://arxiv.org/html/2607.11997#S5.F2)shows the decomposition across all three model sizes, which clarifies the mechanism behind the method\-dependent pattern\. As training progresses past T\*, bias drops \(experts capture task\-specific patterns\) while variance rises \(experts diverge\)\. Simple averaging passes increased variance and covariance in the merged model, leading to degraded performance\. Sparsification methods explicitly filter out small or sign\-conflicting parameter values, reducing variance and covariance while preserving the low\-bias signal\. Sparsification acts as a variance\-reduction mechanism analogous to the averaging step in random forests, where the aggregation step suppresses idiosyncrasies of deeper, higher\-variance trees\.

![Refer to caption](https://arxiv.org/html/2607.11997v1/x2.png)Figure 2:Bias\-variance\-covariance decomposition of the merged model error against training duration, shown across all three model sizes\. As training increases past T\*, bias drops while variance and covariance rise\.
### 5\.2Mode connectivity

We evaluate linear mode connectivity between expert pairs to assess whether overfitted experts remain in the same loss basin\. For each pair\(θA,θB\)\(\\theta\_\{A\},\\theta\_\{B\}\), we interpolateθ​\(α\)=α​θA\+\(1−α\)​θB\\theta\(\\alpha\)=\\alpha\\theta\_\{A\}\+\(1\-\\alpha\)\\theta\_\{B\}forα∈\[0,1\]\\alpha\\in\[0,1\]and measure the loss barrier \(maximum loss along the path minus the average of the endpoint losses\)\.

![Refer to caption](https://arxiv.org/html/2607.11997v1/figures/fig6_mode_connectivity.png)Figure 3:Mode connectivity: average loss barrier between expert pairs vs\. training duration\. The barrier grows monotonically with training duration but remains moderate even at 5×\\timesT\*\.Figure[3](https://arxiv.org/html/2607.11997#S5.F3)shows that the loss barrier increases with training duration, indicating that overfitted experts are progressively less linearly mode\-connected\. However, the barrier remains moderate \(below 0\.35\) even at 5×\\timesT\*, suggesting that LoRA’s low\-rank constraint prevents experts from leaving the pre\-trained basin\. This explains why weight\-space merging remains viable even for heavily overfitted experts: the limiting factor is not basin escape but the merging method’s ability to suppress expert\-specific noise within the basin\. In other words, LoRA gives us a wide “safe zone” where the mode connectivity breakdown is unlikely\.

## 6Conclusion

We conduct a systematic study of how training duration of domain experts affects the quality of merged multi\-task LLMs\. Our experiments across three model sizes, five domains and five merging methods reveal that the optimal training duration for merging is fundamentally method\-dependent\. Simple Averaging benefits from undertrained experts \(0\.25–1×\\timesT\*\), Task Arithmetic and Greedy Soup occupy an intermediate regime \(0\.25–3×\\timesT\* depending on the domain\), while sparsification\-based methods \(TIES, DARE\+TIES\) achieve peak performance with deliberately overfitted experts \(2–5×\\timesT\*\)\. This finding directly challenges the widespread assumption that domain experts should be merged at their individual validation optima\.

Through bias\-variance decomposition, we show that overfitted experts provide higher diversity \(lower bias, higher variance\), which benefits methods equipped with interference\-resolution mechanisms\. The sparsification step in TIES and DARE acts as a variance\-reduction mechanism analogous to the aggregation step in random forests, filtering overfitting\-induced noise while preserving task\-specific knowledge\.

Our results have direct practical implications: when using sparsification\-based merging methods, practitioners should deliberately train experts past their validation optimum\. For the Qwen 3\.5 family, training to 2–5×\\timesT\* with TIES or DARE\+TIES yields the best merged models across all tested sizes, recovering most of the gap to single\-domain experts while retaining all capabilities in a single adapter\. Our study has several limitations\. We use a single open\-weight model family \(Qwen 3\.5\) and a single fine\-tuning recipe: QLoRA adapters at a fixed rank \(r=16r=16\) on a 4\-bit quantized base, trained with a constant learning rate\. Each of these is potentially load\-bearing: prior work finds that full fine\-tuning and LoRA respond differently to training budget\(Horoiet al\.,[2025](https://arxiv.org/html/2607.11997#bib.bib6)\), and learning\-rate decay regulates how sharply a model overtrains\(Rofinet al\.,[2026](https://arxiv.org/html/2607.11997#bib.bib8)\), so the constant schedule may amplify the regime we study\. Our non\-deterministic methods \(Greedy Soup and DARE’s stochastic pruning\) are run with a single seed, and the bias\-variance\-covariance account in Section[5](https://arxiv.org/html/2607.11997#S5)is an explanatory hypothesis rather than a proven mechanism\. Future work should test whether the pattern persists under full fine\-tuning, learning\-rate decay, larger base models, and other PEFT methods\.

## References

- M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, J\. Bao, H\. Behl,et al\.\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p4.1)\.
- P\. Belcak, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. C\. Lin, and P\. Molchanov \(2025\)Small language models are the future of agentic AI\.arXiv preprint arXiv:2506\.02153\.Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p4.1)\.
- L\. Breiman \(2001\)Random forests\.Machine Learning45\(1\),pp\. 5–32\.Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p3.1),[§4\.1](https://arxiv.org/html/2607.11997#S4.SS1.p2.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.3.3.1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.5.3.1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.2.3.1.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2024\)QLoRA: efficient finetuning of quantized language models\.Advances in Neural Information Processing Systems\.Cited by:[§3](https://arxiv.org/html/2607.11997#S3.p1.2)\.
- T\. Heet al\.\(2025\)MergeBench: a comprehensive benchmark for merging in foundation models\.arXiv preprint arXiv:2505\.10833\.Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p2.1),[§2](https://arxiv.org/html/2607.11997#S2.p1.1),[§3](https://arxiv.org/html/2607.11997#S3.p2.2)\.
- S\. Horoi, G\. Wolf, E\. Belilovsky, and G\. K\. Dziugaite \(2025\)Less is more: undertraining experts improves model upcycling\.arXiv preprint arXiv:2506\.14126\.Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p2.1),[§2](https://arxiv.org/html/2607.11997#S2.p3.1),[§4\.1](https://arxiv.org/html/2607.11997#S4.SS1.p3.2),[§6](https://arxiv.org/html/2607.11997#S6.p3.2)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2023\)Editing models with task arithmetic\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p1.1),[§2](https://arxiv.org/html/2607.11997#S2.p1.1),[§3](https://arxiv.org/html/2607.11997#S3.p3.5)\.
- P\. Izmailov, D\. Podoprikhin, T\. Garipov, D\. Vetrov, and A\. G\. Wilson \(2018\)Averaging weights leads to wider optima and better generalization\.InUncertainty in Artificial Intelligence,Cited by:[§2](https://arxiv.org/html/2607.11997#S2.p2.1)\.
- J\. Ji, M\. Liu, J\. Dai, X\. Pan, C\. Zhang, C\. Bian, B\. Chen, R\. Sun, Y\. Wang, and Y\. Yang \(2024\)BeaverTails: towards improved safety alignment of LLM via a human\-preference dataset\.Advances in Neural Information Processing Systems\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.6.2.1.1)\.
- J\. Kaddour \(2022\)Stop wasting my time\! saving days of ImageNet and BERT training with latest weight averaging\.arXiv preprint\.Cited by:[§2](https://arxiv.org/html/2607.11997#S2.p2.1)\.
- N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda,et al\.\(2024\)TÜLU 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.4.2.1.1)\.
- Y\. Luo, C\. Xu, P\. Zhao, R\. Sun, J\. Zhu, T\. Xiao,et al\.\(2024\)Magicoder: empowering code generation with OSS\-instruct\.arXiv preprint arXiv:2312\.02120\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.3.2.1.1)\.
- M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li,et al\.\(2024\)HarmBench: a standardized evaluation framework for automated red teaming and robust refusal\.arXiv preprint arXiv:2402\.04249\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.6.3.1.1)\.
- Qwen Team \(2026\)Qwen 3\.5 model collection\.Note:[https://huggingface\.co/collections/Qwen/qwen35](https://huggingface.co/collections/Qwen/qwen35)Cited by:[§3](https://arxiv.org/html/2607.11997#S3.p1.2)\.
- A\. Rame, M\. Kirchmeyer, T\. Rahier, A\. Rakotomamonjy, P\. Gallinari, and M\. Cord \(2022\)Diverse weight averaging for out\-of\-distribution generalization\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.11997#S2.p4.1),[§5\.1](https://arxiv.org/html/2607.11997#S5.SS1.p1.1)\.
- R\. Ranjan, S\. Garg, M\. Raman, C\. Guestrin, and Z\. C\. Lipton \(2024\)Post\-hoc reversal: are we selecting models prematurely?\.arXiv preprint arXiv:2404\.07815\.Cited by:[§2](https://arxiv.org/html/2607.11997#S2.p2.1)\.
- M\. Rofin, A\. Varre, and N\. Flammarion \(2026\)\(How\) learning rates regulate catastrophic overtraining\.arXiv preprint arXiv:2604\.13627\.Cited by:[§6](https://arxiv.org/html/2607.11997#S6.p3.2)\.
- S\. Singh, F\. Vargus, D\. Dsouza,et al\.\(2024\)Aya dataset: an open\-access collection for multilingual instruction tuning\.arXiv preprint arXiv:2402\.06619\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.5.2.1.1)\.
- H\. Tran, M\. Nguyen, and Q\. Pham \(2026\)Leveraging model soups to classify ICH images from the Mekong Delta\.arXiv preprint arXiv:2603\.02181\.Cited by:[§2](https://arxiv.org/html/2607.11997#S2.p4.1)\.
- Y\. Weiet al\.\(2024\)DART\-Math: difficulty\-aware rejection tuning for mathematical problem\-solving\.arXiv preprint arXiv:2407\.13690\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.2.2.1.1)\.
- M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carber, S\. Kornblith,et al\.\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p1.1),[§2](https://arxiv.org/html/2607.11997#S2.p1.1),[§3](https://arxiv.org/html/2607.11997#S3.p3.5)\.
- P\. Yadav, D\. Tam, L\. Choshen, C\. Raffel, and M\. Bansal \(2023\)TIES\-merging: resolving interference when merging models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p2.1),[§2](https://arxiv.org/html/2607.11997#S2.p1.1),[§3](https://arxiv.org/html/2607.11997#S3.p3.5)\.
- L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. Li \(2024\)Language models are super mario: absorbing abilities from homologous models as a free lunch\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.11997#S1.p2.1),[§2](https://arxiv.org/html/2607.11997#S2.p1.1),[§3](https://arxiv.org/html/2607.11997#S3.p3.5)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[Table 1](https://arxiv.org/html/2607.11997#S3.T1.2.4.3.1.1)\.
- L\. Zhou, D\. Solombrino, D\. Crisostomi, M\. S\. Bucarelli, F\. Silvestri, and E\. Rodolà \(2024\)ATM: improving model merging by alternating tuning and merging\.arXiv preprint arXiv:2411\.03055\.Cited by:[§2](https://arxiv.org/html/2607.11997#S2.p1.1)\.

## Appendix AAdditional Results

### A\.1Main results tables

Tables[3](https://arxiv.org/html/2607.11997#A1.T3)–[5](https://arxiv.org/html/2607.11997#A1.T5)report per\-domain accuracy at T\* \(standard practice\) and the best score across all training durations for each merging method and model size\. For every method and size, the best training duration outperforms T\*\.

ModelMathematicsCodeInstructionMultilingualAverageBase \(no FT\)10\.523\.233\.637\.226\.1Expert @ T\*38\.223\.229\.039\.732\.5Simple Avg @ T\*8\.017\.118\.539\.120\.6Best Simple Averaging16\.817\.718\.541\.323\.6Task Arithmetic @ T\*24\.324\.423\.142\.828\.7Best Task Arithmetic28\.624\.427\.043\.630\.9TIES @ T\*18\.222\.033\.640\.628\.6Best TIES22\.924\.435\.941\.131\.1DARE\+TIES @ T\*16\.521\.335\.139\.228\.1Best DARE\+TIES18\.023\.235\.539\.829\.1Greedy Soup @ T\*7\.920\.129\.639\.624\.3Best Greedy Soup12\.624\.431\.241\.027\.3Table 3:Qwen3\.5\-0\.8B: per\-domain accuracy at T\* and best across all training durations \(bold\)\.ModelMathematicsCodeInstructionMultilingualAverageBase \(no FT\)21\.839\.030\.141\.733\.2Expert @ T\*52\.936\.632\.047\.342\.2Simple Averaging @ T\*23\.532\.327\.950\.333\.5Best Simple Averaging43\.636\.630\.551\.540\.5Task Arithmetic @ T\*32\.939\.035\.550\.939\.6Best Task Arithmetic43\.143\.935\.552\.343\.7TIES @ T\*28\.640\.232\.045\.436\.5Best TIES30\.743\.335\.748\.439\.5DARE\+TIES @ T\*27\.737\.230\.743\.334\.7Best DARE\+TIES30\.040\.931\.245\.937\.0Greedy Soup @ T\*28\.939\.033\.545\.036\.6Best Greedy Soup50\.039\.635\.950\.944\.1Table 4:Qwen3\.5\-2B: per\-domain accuracy at T\* and best across all training durations \(bold\)\.ModelMathematicsCodeInstructionMultilingualAverageBase \(no FT\)61\.6–30\.954\.148\.8Expert @ T\*78\.059\.844\.054\.959\.2Simple Averaging @ T\*45\.652\.433\.860\.448\.1Best Simple Averaging64\.856\.137\.061\.154\.7Task Arithmetic @ T\*66\.459\.146\.460\.758\.2Best Task Arithmetic74\.165\.946\.462\.862\.3TIES @ T\*54\.956\.736\.656\.451\.1Best TIES66\.761\.651\.659\.459\.8DARE\+TIES @ T\*60\.356\.134\.255\.651\.5Best DARE\+TIES63\.661\.638\.157\.155\.1Greedy Soup @ T\*65\.457\.345\.157\.356\.3Best Greedy Soup66\.064\.655\.658\.361\.1Table 5:Qwen3\.5\-4B: per\-domain accuracy at T\* and best across all training durations \(bold\)\.
### A\.2Per\-domain breakdown

Figure[4](https://arxiv.org/html/2607.11997#A1.F4)decomposes the aggregate trends from Figure[1](https://arxiv.org/html/2607.11997#S4.F1)by domain and model size\. The method\-dependent pattern from the main results holds consistently across domains\.

![Refer to caption](https://arxiv.org/html/2607.11997v1/figures/fig2_grid_perdomain.png)Figure 4:Per\-domain performance across merging methods, model sizes, and domains\. The method\-dependent pattern from Figure[1](https://arxiv.org/html/2607.11997#S4.F1)holds across domains, though the magnitude varies\.Mathematics shows the sharpest divergence between methods: at the 4B scale, Simple Averaging drops from 64\.8% at 0\.25×\\timesT\* to 20\.3% at 5×\\timesT\*, while TIES improves from 60\.3% at 0\.25×\\timesT\* to a peak of 66\.7% at 3×\\timesT\*\. Code generation shows similar trends but with smaller gaps, since HumanEval is more saturated\. Instruction Following and Multilingual tasks are more stable across methods, suggesting that the sensitivity to training duration scales with the difficulty of the underlying task: tasks where individual experts gain a lot from extra training are also tasks where merging method choice matters most\.

### A\.3Base model scale dependence

The interaction between training duration and merging quality varies with model size in a way consistent with our bias\-variance results\. For Simple Averaging, 0\.8B models degrade fastest with overfitting \(the merged 0\.8B model loses essentially all of its math performance by 3×\\timesT\*\), 2B models are intermediate, and 4B models show the most gradual decline \(e\.g\., from 64\.8% at 0\.25×\\timesto 20\.3% at 5×\\timeson Math\)\. For TIES, the optimal overfitting level shifts rightward with model size: 0\.8B peaks at 1\.5–3×\\timesT\*, 2B peaks at 2–3×\\timesT\*, and 4B peaks at 3–5×\\timesT\*\. Table[6](https://arxiv.org/html/2607.11997#A1.T6)summarizes the range of optimal training durations for each method across the four accuracy domains\.

MethodQwen 3\.5\-0\.8BQwen 3\.5\-2BQwen 3\.5\-4BSimple Averaging0\.25–1\.0×\\times0\.25–0\.5×\\times0\.25–0\.75×\\timesTask Arithmetic0\.25–3\.0×\\times0\.25–3\.0×\\times0\.25–2\.0×\\timesTIES1\.5–3\.0×\\times2\.0–3\.0×\\times3\.0–5\.0×\\timesDARE\+TIES2\.0–5\.0×\\times0\.25–5\.0×\\times3\.0–5\.0×\\timesGreedy Soup0\.25–0\.75×\\times0\.25–2\.0×\\times0\.25–3\.0×\\times

Table 6:Range of optimal training durations \(multiples of T\*\) per method across the four accuracy domains, by model size\. The optimum for sparsification\-based methods shifts rightward as model size grows, while Simple Averaging stays anchored to the undertrained regime regardless of scale\.This is consistent with the overparameterization view of fine\-tuning\. Larger models have more capacity to absorb task\-specific knowledge into low\-magnitude parameter subsets that are spread across the network, rather than concentrated in a few salient directions\. As a result, the additional variance introduced by overtraining is also more diffuse, which makes it easier for sparsification\-based methods to filter it out: the noisy directions are still individually small and sign\-conflicting after the optimum, just as they need to be for TIES and DARE to discard them\. Conversely, in smaller models, additional training pushes a relatively small set of parameters further from the shared basin, and even sparsification cannot fully recover the merged signal once those salient directions disagree across experts\. The net effect is that the band of training durations for which sparsification\-based merging is helpful both widens and shifts later as model size grows, which provides a useful pattern for practitioners: larger backbones reward more aggressive overtraining of each expert\.

### A\.4Optimal training duration

Tables[7](https://arxiv.org/html/2607.11997#A1.T7)–[9](https://arxiv.org/html/2607.11997#A1.T9)report the optimal training duration per domain for each merging method, separately for the 0\.8B, 2B, and 4B Qwen3\.5 models\. They show that the same dichotomy holds across model sizes: averaging\-style methods peak undertrained and sparsification\-based methods peak overtrained\. The Safety column reports the training duration at which each method achieves its highest refuse\-to\-answer rate\.

MethodMathematicsCodeInstructionMultilingualSafetySimple Averaging0\.25×\\times0\.5×\\times1\.0×\\times0\.25×\\times2\.0×\\timesTask Arithmetic0\.25×\\times0\.75×\\times3\.0×\\times0\.25×\\times2\.0×\\timesTIES2\.0×\\times3\.0×\\times3\.0×\\times1\.5×\\times1\.5×\\timesDARE\+TIES3\.0×\\times2\.0×\\times5\.0×\\times5\.0×\\times1\.5×\\timesGreedy Soup0\.25×\\times0\.25×\\times0\.25×\\times0\.75×\\times2\.0×\\times

Table 7:Optimal training duration per domain for Qwen3\.5\-0\.8B\.MethodMathematicsCodeInstructionMultilingualSafetySimple Averaging0\.25×\\times0\.25×\\times0\.25×\\times0\.5×\\times1\.0×\\timesTask Arithmetic0\.25×\\times3\.0×\\times1\.0×\\times1\.5×\\times3\.0×\\timesTIES3\.0×\\times3\.0×\\times3\.0×\\times2\.0×\\times1\.0×\\timesDARE\+TIES2\.0×\\times5\.0×\\times0\.25×\\times3\.0×\\times5\.0×\\timesGreedy Soup2\.0×\\times0\.75×\\times0\.25×\\times0\.5×\\times1\.0×\\times

Table 8:Optimal training duration per domain for Qwen3\.5\-2B\.MethodMathematicsCodeInstructionMultilingualSafetySimple Averaging0\.25×\\times0\.75×\\times0\.5×\\times0\.25×\\times1\.0×\\timesTask Arithmetic0\.25×\\times1\.5×\\times1\.0×\\times2\.0×\\times3\.0×\\timesTIES3\.0×\\times3\.0×\\times5\.0×\\times5\.0×\\times3\.0×\\timesDARE\+TIES3\.0×\\times5\.0×\\times3\.0×\\times5\.0×\\times5\.0×\\timesGreedy Soup2\.0×\\times3\.0×\\times0\.25×\\times2\.0×\\times1\.0×\\times

Table 9:Optimal training duration per domain for Qwen3\.5\-4B\.