跨LLM训练阶段切换优化器的短暂代价

# The Ephemeral Cost of Switching OptimizersAcross LLM Training Stages
Source: [https://arxiv.org/html/2610.22650](https://arxiv.org/html/2610.22650)
Xinyu Zhao††thanks:Work done during an internship at MIT\-IBM Watson AI Lab\. Correspondence to:lxz@mit\.eduDavis WertheimerAffiliation:MIT\-IBM Watson AI LabAshutosh SharmaAffiliation:MIT\-IBM Watson AI LabRameswar PandaAffiliation:MIT\-IBM Watson AI LabPaul Pu LiangAffiliation:MIT

###### Abstract

Recent work suggests that switching optimizers between pretraining and subsequent training stages incurs a performance cost, but systematic investigation of this phenomenon is lacking\. We study the mismatch between AdamW and Muon optimizers by pretraining 1\.2B Llama\-3 models from scratch while varying the proportion of updates taken with each optimizer, and similarly interpolating between the two optimizers during continued pretraining and supervised fine\-tuning \(SFT\)\. Surprisingly, we find little evidence of a meaningful optimizer mismatch penalty: final performance is largely explained by the independent effects of the pretraining and post\-training optimizers, with downstream performance generally improving as the proportion of Muon updates during SFT increases\. Although matching the pretraining optimizer can reduce forgetting during SFT, this does not translate into improved final benchmark performance\. We further evaluate AdamW and Muon for task\-general SFT of open\-weight models, across the Llama and Qwen families up to 14B parameters, where Muon remains competitive with AdamW, despite these models having been pretrained with AdamW\. Together, our results suggest that the choice of optimizer for post\-training can be made largely independently of the optimizer used during pretraining\. Our code is released[here](https://anonymous.4open.science/r/sft-optimizer-BB44)\.

## 1Introduction

AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.22650#bib.bib1)\)has been the de facto optimizer for training most large language models\. However, recent results have shown that Muon\([Jordan et al\., 2024](https://arxiv.org/html/2610.22650#bib.bib6)\)can outperform AdamW at scale in LLM pretraining\([Liu et al\., 2025](https://arxiv.org/html/2610.22650#bib.bib5);[Khona et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib7)\)\. Yet it remains an open question whether Muon is also a better optimizer for supervised post\-training stages, and whether switching optimizers between training stages is detrimental to final performance\. Prior work has suggested such a penalty for using a different optimizer for SFT\([Qu et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib2)\), due to differing regularization effects\([Liu et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib3)\)or implicit biases\([Qu et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib2)\)\. It has been unclear whether this effect is limited to greater forgetting during task\-specific fine\-tuning \(“forgetting cost"\), or also extends to degraded final benchmark performance \(“performance cost"\) following the larger\-scale, multi\-task SFT stage typical in foundation model training\.

In this work, we examine whether there is a cost to switching between AdamW and Muon across LLM training stages\. We train our own 1\.2B Llama\-3 models from scratch while varying the proportion of AdamW and Muon updates used during pretraining, continued pretraining, and task\-general SFT, allowing us to separate the effect of optimizer mismatch from each optimizer’s independent effect, as well as that of the underlying data shift\. We additionally evaluate AdamW and Muon for task\-general SFT of released open\-weight models up to 14B parameters\.

Our main findings are as follows\. First, after accounting for the marginal effects of the pretraining and post\-training optimizers, we find little evidence of an additional performance cost from switching optimizers: interaction effects are small during SFT, and apparent switching costs during continued pretraining largely disappear after tuning the learning rate for each setting\. Second, optimizer consistency does reduce forgetting of the pretraining distribution during SFT, but this does not translate into improved downstream benchmark performance\. Instead, increasing the proportion of Muon updates during SFT generally improves performance regardless of the pretraining optimizer\. Finally, across AdamW\-pretrained open\-weight models from the Llama and Qwen families up to 14B parameters, Muon remains competitive with AdamW for task\-general SFT\.

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/avg6_residual_heatmap_allopt_anova_flip.png)

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/forget_residual_heatmap_uniform_anova.png)

Figure 1:SFT benchmark performance and forgetting residuals\.Rows index the proportion of Muon updates for 1\.2B Llama\-3 models during pretraining \(0\.0=AdamW0\.0=\\mathrm\{AdamW\},1\.0=Muon1\.0=\\mathrm\{Muon\}\); columns index the fraction during SFT\. Average of three independent SFT runs per cell\.Left:Average of six downstream benchmarks \(higher is better\), using the best learning rate \(Table[4](https://arxiv.org/html/2610.22650#A1.T4)\) for each setting\. Optimizer interaction effects are weak and inconsistent\.Right:change in pretraining validation loss after SFT \(lower is better\), at learning rate4×10−44\\times 10^\{\-4\}, showing a diagonal matched\-optimizer preference pattern\. Heatmap cells show the residual after accounting for row/column marginal averages, shown along the sides of each heatmap\. Raw benchmark averages can be found in Table[7](https://arxiv.org/html/2610.22650#A2.T7)\.
## 2Experimental Setup

For our 1\.2B\-parameter experiments, we pre\-train Llama\-3 transformer models with 32 layers, 24 attention heads with full multi\-head\-attention, and a hidden dimension of 1536, following the hyperparameter recommendations for this configuration given by[Wen et al\. \(2026\)](https://arxiv.org/html/2610.22650#bib.bib4)\. Full architecture details can be found in Appendix[A\.1](https://arxiv.org/html/2610.22650#A1.SS1), and training hyperparameters are detailed in Appendix[A\.3](https://arxiv.org/html/2610.22650#A1.SS3)\.

##### Optimizers

We focus on comparing two different optimization algorithms: AdamW and Muon\. We adopt the decoupled weight decay version of AdamW with gradient norm clipping, and we detail the algorithm in Algorithm[1](https://arxiv.org/html/2610.22650#alg1)\.[Wen et al\. \(2026\)](https://arxiv.org/html/2610.22650#bib.bib4)performed extensive hyperparameter search to find the best hyperparameters for pre\-training, and we adopt these for our1×1\\timesand8×8\\timesChinchilla models\. Our Muon algorithm is presented in Algorithm[2](https://arxiv.org/html/2610.22650#alg2)and uses the 0\.2 RMS update scaling factor from[Liu et al\. \(2025\)](https://arxiv.org/html/2610.22650#bib.bib5), and the same base learning rate as our AdamW pretraining\.

Recent work has explored interleaving updates from different optimizers during pretraining\([Pang et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib16)\)\. Following this approach, we construct mixed AdamW\-Muon optimizers by independently selecting a Muon update at each step with probabilityp∈\[0\.0,0\.1,0\.25,0\.5,0\.75,0\.9,1\.0\]p\\in\[0\.0,0\.1,0\.25,0\.5,0\.75,0\.9,1\.0\], and an AdamW update otherwise\. Since we useβ1=0\.9\\beta\_\{1\}=0\.9for our AdamW training, the expected update RMS according to[Su \(2025\)](https://arxiv.org/html/2610.22650#bib.bib17)is\(1−β1\)/\(1\+β1\)≈0\.22\\sqrt\{\(1\-\\beta\_\{1\}\)/\(1\+\\beta\_\{1\}\)\}\\approx 0\.22\. We use this to unify update scales between optimizers, admitting a common base learning rate within and across mixture settings\.

##### Pretraining

Using the same data mixture as[Wen et al\. \(2026\)](https://arxiv.org/html/2610.22650#bib.bib4), we pre\-train our 1\.2B\-parameter Llama3 models at1×1\\timesand8×8\\timesChinchilla scales\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.22650#bib.bib15)\), using approximately 24B and 193B tokens respectively\.

##### SFT

We use the Tulu\-3 SFT mixture\([Lambert et al\., 2025](https://arxiv.org/html/2610.22650#bib.bib8)\)and train for two epochs\. We select the best learning rate based on validation loss on a held\-out set, and perform final validation on MMLU\([Hendrycks et al\., 2021a](https://arxiv.org/html/2610.22650#bib.bib9)\), BBH\([Suzgun et al\., 2023](https://arxiv.org/html/2610.22650#bib.bib14)\), GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.22650#bib.bib12)\), MATH\([Hendrycks et al\., 2021b](https://arxiv.org/html/2610.22650#bib.bib13)\), HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.22650#bib.bib10)\), and MBPP\([Austin et al\., 2021](https://arxiv.org/html/2610.22650#bib.bib11)\)\. We pack the SFT sequences to 4096 tokens for our 1\.2B models and Llama\-2\-7B, and pack sequences to 8192 tokens for Qwen models\. Specific hyperparameters for each setting can be found in Appendix[A\.3](https://arxiv.org/html/2610.22650#A1.SS3)\.

## 3Cost of Optimizer Mismatch

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/cpt_grid_residual_heatmap_5e5_anova.png)

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/cpt_grid_residual_heatmap_anova.png)

Figure 2:Continued pretraining \(CPT\) performance residuals\.Formatting follows Fig\.[1](https://arxiv.org/html/2610.22650#S1.F1): rows index the proportion of Muon updates during pretraining; columns index the fraction for CPT\.Left:validation loss with a fixed learning rate of5×10−55\\times 10^\{\-5\}for all settings\.Right:validation loss using the best learning rate \(Table[5](https://arxiv.org/html/2610.22650#A1.T5)\) for each setting\. Matched\-optimizer preference vanishes once learning rate is accounted for\.To isolate the cost of switching optimizers across training stages, we relax the AdamW/Muon optimizer assignment from a binary choice to a continuous degree of mismatch\. Specifically, we treat AdamW and Muon as defining two distinct update distributions, and construct intermediate optimizers by randomly selecting an AdamW/Muon update at each step\. This allows us to isolate and quantify switching cost in greater detail, by smoothly annealing the degree of mismatch across stages\.

We implement this by pre\-training our 1\.2B models at 1×\\timesChinchilla while selecting a Muon update at each step with probabilityp∈\[0\.0,0\.1,0\.25,0\.5,0\.75,0\.9,1\.0\]p\\in\[0\.0,0\.1,0\.25,0\.5,0\.75,0\.9,1\.0\], wherep=0\.0p=0\.0corresponds to full\-AdamW andp=1\.0p=1\.0corresponds to full\-Muon\. Then, during the SFT stage, we independently vary the optimizer mixture over the same values ofpp\. We measure average performance on the six benchmarks across three random seeds for each PT\-SFT optimizer pairing\.

##### Isolating the optimizer mismatch effect

To separate optimizer mismatch from the independent effects of the optimizer in either stage, we decompose the score grid into row effects, column effects, and their interaction\. For global meanμ\\mu, the performance for each cellXi​jX\_\{ij\}decomposes as

Xi​j=μ\+αi⏟PT optimizer effect\+βj⏟SFT optimizer effect\+γi​j⏟PT\-SFT interactionX\_\{ij\}=\\mu\+\\underbrace\{\\alpha\_\{i\}\}\_\{\\text\{PT optimizer effect\}\}\+\\underbrace\{\\beta\_\{j\}\}\_\{\\text\{SFT optimizer effect\}\}\+\\underbrace\{\\gamma\_\{ij\}\}\_\{\\text\{PT\-SFT interaction\}\}\(1\)whereαi\\alpha\_\{i\}andβj\\beta\_\{j\}are the centered row and column marginal effects respectively, andγi​j\\gamma\_\{ij\}is the remaining unexplained quantity:

αi=X¯i,:−μ,βj=X¯:,j−μ,γi​j=Xi​j−X¯i,:−X¯:,j\+μ\\alpha\_\{i\}=\\bar\{X\}\_\{i,:\}\-\\mu,\\qquad\\beta\_\{j\}=\\bar\{X\}\_\{:,j\}\-\\mu,\\qquad\\gamma\_\{ij\}=X\_\{ij\}\-\\bar\{X\}\_\{i,:\}\-\\bar\{X\}\_\{:,j\}\+\\mu\(2\)
By examining the trends inγi​j\\gamma\_\{ij\}across the score grid, we can identify interaction effects between optimizers, independent from the performance of each optimizer in general\.

##### Results

If matching optimizers across stages provides a performance benefit, then we would expect better residuals along the diagonalpPT=pSFTp\_\{\\text\{PT\}\}=p\_\{\\text\{SFT\}\}, particularly at the fully matched AdamW/AdamW and Muon/Muon corners, and unfavorable residuals as the degree of mismatch increases\. Figure[1](https://arxiv.org/html/2610.22650#S1.F1)\(left\) displays the observed residuals in each heatmap cell: contrary to the optimizer\-matching hypothesis, interaction effects between the PT and SFT optimizers are mild, with no consistent overall pattern\. The matched\-optimizer cells are not meaningfully better than predicted from the independent row\- and column\-wise effects: AdamW\-PT/AdamW\-SFT has a negative residual of−0\.2\-0\.2while Muon\-PT/Muon\-SFT has positive residual\+0\.2\+0\.2, with other diagonal interactions ranging from−0\.6\-0\.6to\+0\.3\+0\.3\. These interactions are small relative to the SFT\-optimizer main effect \(−0\.9\-0\.9to\+0\.7\+0\.7\), and are generally comparable to seed noise in the underlying benchmark averages \(Table[7](https://arxiv.org/html/2610.22650#A2.T7)\)\. Thus, we find little evidence that matching the optimizer across stages provides meaningful performance benefit\.

Final performance is instead dominated by the independent effects of the optimizer used in each stage\. The marginal SFT optimizer effect increases monotonically with the proportion of Muon updates, indicating that Muon during SFT generally improves downstream performance regardless of pretraining optimizer\. Models pretrained with at least some Muon updates also perform better on average across SFT optimizers, although this primarily appears as a pretraining artifact rather than increased compatibility with Muon SFT; these models already achieve lower pretraining validation loss than AdamW\-PT \(Table[3](https://arxiv.org/html/2610.22650#A1.T3)\)\.

##### Effect of SFT distribution

The shift in training data introduced by SFT could potentially obscure a true underlying cost of switching optimizers across training stages\. To confirm the lack of broad interaction trends in Figure[1](https://arxiv.org/html/2610.22650#S1.F1)\(left\), we instead continue training on the original pre\-training data distribution\. The results of this continued pretraining stage are shown in Figure[2](https://arxiv.org/html/2610.22650#S3.F2)\. The marginal averages show that final performance improves as more Muon steps are taken, for both pre\-training and post\-training\. When using the same learning rate of 5e\-5 across settings, matched CPT optimizers do show the expected favorable interactions \(left\), reflected by negative residuals, but these effects disappear once the learning rate is tuned for all settings \(right\)\.111For SFT, we do not observe significant interaction effects at either a fixed LR \(Figure[4](https://arxiv.org/html/2610.22650#A2.F4)\) or at the best LR \(Figure 1, left\)This suggests that optimizer\-switching costs, when they appear, may stem from hyperparameter mismatch, rather than the optimizersper se\.

##### Matched optimizer mitigates forgetting

We next examine whether optimizer consistency affects forgetting of the pretraining distribution, despite limited interaction effect on final downstream performance\. We quantify forgetting as the increase in held\-out validation loss after SFT, shown in Figure[1](https://arxiv.org/html/2610.22650#S1.F1)\(right\)\. The column\-wise averages show that using more Muon updates during SFT generally results in greater forgetting, while the row\-wise averages suggest that using Muon during pretraining mitigates SFT forgetting\. Beyond these effects, the residuals show that matched optimizers further reduce forgetting relative to mismatched settings\. Thus optimizer switching costs may also manifest as forgetting, rather than as a straightforward reduction in learning capability\.

## 4Does model scale change the preferred SFT optimizer?

Table 1:SFT benchmark performance for our 1\.2B pretrained models at1×1\\timesand8×8\\timesChinchilla data ratios\.AdamW\-PT and Muon\-PT denote the pretraining optimizer; AdamW and Muon denote the SFT optimizer\. Mean and standard deviation are reported over three SFT seeds\.Prior work has reported that for large open\-weight LLMs, using the matched optimizer for task\-specific SFT may have a slight edge\([Qu et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib2);[Liu et al\., 2026](https://arxiv.org/html/2610.22650#bib.bib3)\), while comparisons for task\-general SFT have been limited to a few models, and generally inconclusive\([Liu et al\., 2025](https://arxiv.org/html/2610.22650#bib.bib5)\)\. In Section[3](https://arxiv.org/html/2610.22650#S3), our experiments at the 1\.2B 1×\\timesChinchilla scale suggest that Muon may have the edge in task\-general SFT regardless of the pretraining optimizer\. In this section, we examine whether this phenomenon holds at scale, by increasing the pretraining tokens for our 1\.2B model, and by fine\-tuning open\-weight models from various model families up to 14B parameters\.

##### Data scaling

We compare optimizer settings for our 1\.2B models at8×8\\timesChinchilla data ratio in Table[1](https://arxiv.org/html/2610.22650#S4.T1)\. While Muon is clearly the better SFT optimizer for our AdamW\-pretrained models at1×1\\times, there is a negligible performance gap at8×8\\times\. On the other hand, Muon clearly remains the better SFT optimizer for Muon\-pretrained models at both1×1\\timesand8×8\\timesChinchilla data ratios\.

##### Open weight models

Table 2:AdamW vs\. Muon SFT results for Llama2\-7B, Qwen2\.5\-7B, and Qwen3 family\. Mean and stdev\. reported over three trials\. Muon achieves higher benchmark average in four of five settings\.\(a\)Llama2\-7B
\(b\)Qwen2\.5\-7B
\(c\)Qwen3 family at 4B, 8B, and 14B model sizes\.

We select an array of AdamW\-pretrained open\-weight models to SFT using AdamW and Muon, and report average performance over 3 seeds for each setting\. Table[2](https://arxiv.org/html/2610.22650#S4.T2)reports results on Llama2\-7B, Qwen2\.5\-7B, and the 4B, 8B, and 14B parameter models from Qwen3\. Across all models surveyed, we find that Muon\-SFT produces a higher mean benchmark average in 4 of 5 settings, although all gaps \(except perhaps Llama2\-7B\) are small relative to run\-to\-run variation\.

## 5Conclusion

We study whether switching optimizers between pretraining and subsequent training stages incurs a performance cost, focusing on AdamW and Muon\. Across controlled experiments interpolating between AdamW and Muon updates, we find little evidence of a meaningful optimizer mismatch penalty: final performance is largely determined by the independent marginal effects of the pretraining and post\-training optimizers, rather than by whether they match\. Optimizer consistency can reduce forgetting of the pretraining distribution \(which may be important in continual\-learning settings, where retaining performance is itself an objective\), and ease hyperparameter transfer\. But this does not translate to improved final performance under task\-general SFT\. Instead, at the 1\.2B scale, increasing the proportion of Muon updates during SFT generally improves downstream performance regardless of pretraining optimizer\. Experiments on open\-weight models across the Llama and Qwen families up to 14B further show that Muon remains competitive with AdamW for SFT of AdamW\-pretrained models\. Together, these results suggest that the choice of optimizer for post\-training can be made largely independently of the optimizer used during pretraining, and that Muon is a viable post\-training optimizer even for models originally trained with AdamW\.

## References

- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=7Bywt2mQsCe)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. SifreTraining compute\-optimal large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30016–30030\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px2.p1.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2610.22650#S1.p1.1)\.
- Khonaet al\.\(2026\)M\. Khona, A\. Vavre, B\. Wang, D\. Fu, H\. Wu, M\. Chrzanowski, B\. Catanzaro, D\. Mudigere, J\. Pool, M\. Lightstone, M\. Shoeybi, M\. Patwary, N\. Tajbakhsh, and T\. BlankevoortSOAP, muon, and beyond: pushing llm pretraining scales\.External Links:2607\.20548,[Link](https://arxiv.org/abs/2607.20548)Cited by:[§1](https://arxiv.org/html/2610.22650#S1.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, X\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. HajishirziTulu 3: pushing frontiers in open language model post\-training\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=i1uGbfHHpH)Cited by:[§A\.3\.2](https://arxiv.org/html/2610.22650#A1.SS3.SSS2.p1.1),[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. YangMuon is scalable for llm training\.External Links:2502\.16982,[Link](https://arxiv.org/abs/2502.16982)Cited by:[§A\.2](https://arxiv.org/html/2610.22650#A1.SS2.p1.1),[§A\.3\.2](https://arxiv.org/html/2610.22650#A1.SS3.SSS2.Px2.p2.1),[§A\.3\.2](https://arxiv.org/html/2610.22650#A1.SS3.SSS2.Px2.p3.1),[§1](https://arxiv.org/html/2610.22650#S1.p1.1),[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.22650#S4.p1.1)\.
- Liuet al\.\(2026\)Y\. Liu, J\. Wang, and T\. ZhangOptimizer\-model consistency: full finetuning with the same optimizer as pretraining forgets less\.External Links:2605\.06654,[Link](https://arxiv.org/abs/2605.06654)Cited by:[§1](https://arxiv.org/html/2610.22650#S1.p1.1),[§4](https://arxiv.org/html/2610.22650#S4.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§1](https://arxiv.org/html/2610.22650#S1.p1.1)\.
- Panget al\.\(2026\)T\. Pang, Y\. Fang, Z\. Liu, S\. Deng, L\. Hsiung, S\. Yu, and Y\. YangHTMuon: improving muon via heavy\-tailed spectral correction\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 36504–36535\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1819/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1819),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px1.p2.1)\.
- Quet al\.\(2026\)X\. Qu, P\. Huang, and S\. HorvathCan Muon fine\-tune Adam\-pretrained models?\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.22650#S1.p1.1),[§4](https://arxiv.org/html/2610.22650#S4.p1.1)\.
- Su \(2025\)J\. SuWhy is Adam’s update RMS 0\.2?\.Note:Blog post, Scientific SpacesExternal Links:[Link](https://kexue.fm/archives/11267)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px1.p2.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. WeiChallenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px3.p1.1)\.
- Wenet al\.\(2026\)K\. Wen, D\. L\. W\. Hall, T\. Ma, and P\. LiangFantastic pretraining optimizers and where to find them\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2J51qUZ0iG)Cited by:[§A\.1](https://arxiv.org/html/2610.22650#A1.SS1.p1.1),[§A\.1](https://arxiv.org/html/2610.22650#A1.SS1.p2.1),[§A\.2](https://arxiv.org/html/2610.22650#A1.SS2.p1.1),[§A\.3\.1](https://arxiv.org/html/2610.22650#A1.SS3.SSS1.p1.1),[§A\.3\.1](https://arxiv.org/html/2610.22650#A1.SS3.SSS1.p3.1),[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.22650#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.22650#S2.p1.1)\.

## Appendix AExperimental Details

### A\.1Model Architecture

For pretraining, we use the Llama\-3 architecture and tokenizer\. Following the 1\.2B configuration of[Wen et al\. \[2026\]](https://arxiv.org/html/2610.22650#bib.bib4), our models have 32 transformer layers, a hidden dimension of 1536, an intermediate dimension of 6144, and 24 attention heads\. We use full multi\-head attention with 24 key\-value heads\. The context length is 4096 tokens and the vocabulary size is 128,256\. We use SwiGLU activations and rotary positional embeddings \(RoPE\) withθ=500,000\\theta=500\{,\}000\. The input embedding and language\-model head are untied\.

This configuration contains approximately 1\.60B parameters in total, of which approximately 1\.21B are non\-embedding parameters\. Following[Wen et al\. \[2026\]](https://arxiv.org/html/2610.22650#bib.bib4), we refer to the model by its non\-embedding parameter count as a 1\.2B model\. During training, model parameters are stored in FP32 and activations are computed in BF16\.

### A\.2Optimizers

Algorithm 1AdamW1:Gradient

GtG\_\{t\}, parameters

Wt−1W\_\{t\-1\}, first moment

mt−1m\_\{t\-1\}, second moment

vt−1v\_\{t\-1\}, learning rate

η\\eta, weight decay

λ\\lambda, coefficients

β1,β2\\beta\_\{1\},\\beta\_\{2\}, constant

ϵ\\epsilon
2:

mt←β1​mt−1\+\(1−β1\)​Gtm\_\{t\}\\leftarrow\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)G\_\{t\}
3:

m^t←mt/\(1−β1t\)\\hat\{m\}\_\{t\}\\leftarrow m\_\{t\}/\(1\-\\beta\_\{1\}^\{t\}\)
4:

vt←β2​vt−1\+\(1−β2\)​Gt⊙2v\_\{t\}\\leftarrow\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)G\_\{t\}^\{\\odot 2\}
5:

v^t←vt/\(1−β2t\)\\hat\{v\}\_\{t\}\\leftarrow v\_\{t\}/\(1\-\\beta\_\{2\}^\{t\}\)
6:

Ut←m^t⊘\(v^t\+ϵ\)U\_\{t\}\\leftarrow\\hat\{m\}\_\{t\}\\oslash\\left\(\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\epsilon\\right\)
7:

Wt←\(1−η​λ\)​Wt−1−η​UtW\_\{t\}\\leftarrow\(1\-\\eta\\lambda\)W\_\{t\-1\}\-\\eta U\_\{t\}
8:return

Wt,mt,vtW\_\{t\},m\_\{t\},v\_\{t\}

Algorithm 2Muon1:Gradient

Gt∈ℝm×nG\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}, parameters

Wt−1W\_\{t\-1\}, momentum

mt−1m\_\{t\-1\}, learning rate

η\\eta, weight decay

λ\\lambda, momentum coefficient

μ\\mu
2:

mt←μ​mt−1\+Gtm\_\{t\}\\leftarrow\\mu m\_\{t\-1\}\+G\_\{t\}
3:

Bt←Gt\+μ​mtB\_\{t\}\\leftarrow G\_\{t\}\+\\mu m\_\{t\}
4:

Ot←NS5​\(Bt\)O\_\{t\}\\leftarrow\\mathrm\{NS\}\_\{5\}\(B\_\{t\}\)
5:

Ut←0\.2​max⁡\(m,n\)​OtU\_\{t\}\\leftarrow 0\.2\\sqrt\{\\max\(m,n\)\}\\,O\_\{t\}
6:

Wt←\(1−η​λ\)​Wt−1−η​UtW\_\{t\}\\leftarrow\(1\-\\eta\\lambda\)W\_\{t\-1\}\-\\eta U\_\{t\}
7:return

Wt,mtW\_\{t\},m\_\{t\}

Following[Liu et al\. \[2025\]](https://arxiv.org/html/2610.22650#bib.bib5)and[Wen et al\. \[2026\]](https://arxiv.org/html/2610.22650#bib.bib4), we perform orthogonalization on the concatenated attention head matrices\.

### A\.3Hyperparameters

#### A\.3\.1Pretraining

We pretrain our 1\.2B models using the same data mixture as[Wen et al\. \[2026\]](https://arxiv.org/html/2610.22650#bib.bib4), consisting of DCLM\-baseline, StarCoderData, and ProofPile\-2\. All data are tokenized using the Llama\-3 tokenizer\. We train at both1×1\\timesand8×8\\timesChinchilla data ratios, corresponding to approximately 24\.16B and 193\.29B tokens, respectively\. With a global batch size of 256 sequences and a sequence length of 4096, each optimization step contains 1,048,576 tokens\. We therefore train for 23,042 steps at1×1\\timesChinchilla and 184,336 steps at8×8\\timesChinchilla\.

We use a linear warmup followed by linear learning\-rate decay to zero\. For the1×1\\timesChinchilla runs, AdamW uses a peak learning rate of2×10−32\\times 10^\{\-3\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.98\\beta\_\{2\}=0\.98,ϵ=10−10\\epsilon=10^\{\-10\}, weight decay0\.20\.2, gradient\-norm clipping at2\.02\.0, and 2,000 warmup steps\. Muon uses a peak matrix learning rate of2×10−32\\times 10^\{\-3\}, momentumμ=0\.98\\mu=0\.98, weight decay0\.10\.1, five Newton–Schulz iterations, and no learning\-rate warmup\. Parameters not optimized by Muon are optimized with AdamW using a learning rate of1\.2×10−31\.2\\times 10^\{\-3\},\(β1,β2\)=\(0\.8,0\.98\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.8,0\.98\),ϵ=10−15\\epsilon=10^\{\-15\}, and weight decay0\.10\.1\.

For the8×8\\timesChinchilla runs, we use the budget\-specific hyperparameters from[Wen et al\. \[2026\]](https://arxiv.org/html/2610.22650#bib.bib4)\. AdamW uses a peak learning rate of2×10−32\\times 10^\{\-3\}, weight decay0\.20\.2, gradient\-norm clipping at1\.01\.0, and 1,000 warmup steps\. Muon uses a matched peak matrix learning rate of2×10−32\\times 10^\{\-3\}, weight decay0\.10\.1, gradient\-norm clipping at2\.02\.0, and no warmup; its remaining hyperparameters and the AdamW parameter group are unchanged from the1×1\\timessetting\.

For mixed\-optimizer pretraining runs, at each optimization step we independently select a Muon update with probabilityppand an AdamW update otherwise\. We use 2,000 learning rate warmup steps\. As described in Section 2, we use a common learning rate and weight decay for the two update branches so that their update magnitudes remain comparable\.

Final validation loss on C4\-EN for our pretrained models is shown in Table[3](https://arxiv.org/html/2610.22650#A1.T3)\.

Table 3:C4\-EN validation loss for our 1\.2B Llama3 models\.
#### A\.3\.2SFT

We perform supervised fine\-tuning using the Tulu\-3 SFT mixture\[[Lambert et al\., 2025](https://arxiv.org/html/2610.22650#bib.bib8)\]for two epochs for both our 1\.2B models and open weight models\.

##### 1\.2B Pretrained Models

For our 1\.2B models, training examples are greedily packed into sequences of length 4096 without splitting individual conversations across sequence boundaries\. Labels corresponding to prompt and user tokens are masked from the loss\. For both AdamW and Muon, we use a linear warmup for 3% of the total SFT steps, followed by linear decay to zero\. We sweep the base learning rate separately for each optimizer and experimental setting and select the learning rate with the lowest held\-out validation loss\. When mixing optimizers, a Muon update is independently selected at each step with probabilityppand an AdamW update is selected otherwise\.

The optimizer hyperparameters largely follow the corresponding pretraining configurations described in Appendix[A\.3\.1](https://arxiv.org/html/2610.22650#A1.SS3.SSS1), with several differences for SFT\. For AdamW, we use\(β1,β2\)=\(0\.9,0\.98\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.98\),ϵ=10−10\\epsilon=10^\{\-10\}, weight decay0\.20\.2, and global gradient\-norm clipping at1\.01\.0\. For Muon, we use Nesterov momentumμ=0\.98\\mu=0\.98, weight decay0\.10\.1, and five Newton–Schulz iterations withϵNS=10−5\\epsilon\_\{\\mathrm\{NS\}\}=10^\{\-5\}\. As during pretraining, Muon is applied to matrix\-valued parameters, while embeddings, the language\-model head, and normalization parameters are optimized by a separate AdamW parameter group with\(β1,β2\)=\(0\.8,0\.98\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.8,0\.98\),ϵ=10−15\\epsilon=10^\{\-15\}, weight decay0\.10\.1, and learning rate5×10−55\\times 10^\{\-5\}\. Global gradient\-norm clipping at1\.01\.0is also used for Muon runs\.

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/sft_lr_2x2.png)Figure 3:SFT learning rate sweep for AdamW and Muon on 1\.2B pretrained Llama3 models\.Table 4:Best SFT learning rate for our 1\.2B1×1\\timesChinchilla Llama3 models\.Table 5:Best CPT learning rate for our 1\.2B1×1\\timesChinchilla Llama3 models\.The results of our SFT learning rate sweeps can be found in Figure[3](https://arxiv.org/html/2610.22650#A1.F3)and Table[4](https://arxiv.org/html/2610.22650#A1.T4)\. For the SFT grid reported in Figure[1](https://arxiv.org/html/2610.22650#S1.F1), optimal learning rates are found by first bracketing the best learning rates for the first and last column as well as the matched\-optimizer diagonal\. Since these sweep results show that the optimal learning rate is in\{4×10−4,8×10−4\}\\\{4\\times 10^\{\-4\},8\\times 10^\{\-4\}\\\}, we only sweep these results for the rest of the table\.

We perform the same procedure for our CPT learning rate sweep and report the best learning rates for each setting in Table[5](https://arxiv.org/html/2610.22650#A1.T5)\.

Across both SFT and CPT, we observe that the optimal post\-training learning rate depends on the pretraining optimizer as well as the post\-training optimizer mixture\. In particular, models pretrained with a larger proportion of Muon updates tend to prefer a higher learning rate when post\-trained with AdamW\-heavy mixtures, while increasing the proportion of Muon updates during post\-training generally reduces this difference\. This dependence motivates selecting the learning rate separately for each PT–post\-training optimizer pairing rather than using a single learning rate across the grid\.

##### Open Weight Models

For open\-weight models, we use the same Tulu\-3 mixture and assistant\-only loss, and greedily pack conversations without splitting them across sequence boundaries\. Packed conversations are isolated using a document\-level attention mask with position IDs reset at each conversation boundary\. We pack sequences to a maximum length of 8192 tokens for Qwen2\.5 and Qwen3 models and 4096 tokens for Llama\-2, whose context length is limited to 4096 tokens\.

For AdamW, we use\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\),ϵ=10−8\\epsilon=10^\{\-8\}, zero weight decay, and global gradient\-norm clipping at1\.01\.0\. For Muon, we use the RMS\-matched update scaling of[Liu et al\. \[2025\]](https://arxiv.org/html/2610.22650#bib.bib5),0\.2​max⁡\(m,n\)0\.2\\sqrt\{\\max\(m,n\)\}, with Nesterov momentumμ=0\.95\\mu=0\.95and five Newton–Schulz iterations\. Muon is applied to two\-dimensional parameter matrices, while embeddings, the language\-model head, and normalization parameters are optimized with AdamW at the same learning rate\. Both AdamW and Muon use the same learning rate within a given model and learning\-rate sweep\.

Table 6:Learning rate sweep results for open weight models\.We use3%3\\%learning\-rate warmup and then follow[Liu et al\. \[2025\]](https://arxiv.org/html/2610.22650#bib.bib5)in using a cosine decay schedule to0\.10\.1times the peak learning rate\. We use a global batch size of 524,288 tokens per optimization step, corresponding to 64 sequences of length 8192 for Qwen models and 128 sequences of length 4096 for Llama\-2\. We sweep the peak learning rate separately for each model and select the value with the lowest held\-out validation loss\. Our selected learning rates are shown in Table[6](https://arxiv.org/html/2610.22650#A1.T6)\.

## Appendix BAdditional Results

### B\.1SFT benchmark performance for 1\.2B models

Table 7:SFT benchmark performance at the best learning rate for each setting\.Average and standard deviation are reported over 3 random SFT seeds\.Table[7](https://arxiv.org/html/2610.22650#A2.T7)reports the average and standard deviation for benchmark averages across three random seeds for the SFT experiments in Section[3](https://arxiv.org/html/2610.22650#S3)\. These are the results used to calculate row\-wise averages, column\-wise averages, and residuals in Figure[1](https://arxiv.org/html/2610.22650#S1.F1)\(left\)\.

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/avg6_residual_heatmap_anova.png)Figure 4:SFT benchmark performance at uniform LR 4e\-4\.Three independent SFT runs are averaged for each cell\.
### B\.2SFT benchmark performance at uniform LR for 1\.2B models

As a companion to Figure[1](https://arxiv.org/html/2610.22650#S1.F1), we recreate the same heatmap using models trained at the same LR \(4e\-4\) for all settings, which is the majority\-optimal setting, and present the results in Figure[4](https://arxiv.org/html/2610.22650#A2.F4)\. These results largely reflect the trends seen in Figure[1](https://arxiv.org/html/2610.22650#S1.F1): we see that increasing the proportion of Muon updates used in the SFT stage generally increases final benchmark performance\. In the residual cells, we observe only mild interaction effects caused by optimizer consistency across training stages\.

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/math_residual_heatmap_allopt_anova.png)

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/code_residual_heatmap_allopt_anova.png)

![Refer to caption](https://arxiv.org/html/2610.22650v1/figures/anova/mmlu_bbh_residual_heatmap_allopt_anova.png)

Figure 5:SFT benchmark performance by task type at the best LR for each setting\.Breakdown of the SFT results in Figure 1 by downstream task category for our 1\.2B Llama\-3 models\. Rows index the proportion of Muon updates used during pretraining \(0\.0=AdamW0\.0=\\mathrm\{AdamW\},1\.0=Muon1\.0=\\mathrm\{Muon\}\), and columns index the proportion used during SFT\. Results show the same qualitative trends as the aggregate benchmark performance in Figure 1: increasing the proportion of Muon updates during SFT generally improves performance across task types, with relatively small interaction effects between the pretraining and SFT optimizer choices\.
### B\.3SFT benchmark performance by task type for 1\.2B models

Figure[5](https://arxiv.org/html/2610.22650#A2.F5)breaks down the SFT results from Figure[1](https://arxiv.org/html/2610.22650#S1.F1)by benchmark type rather than averaging across all six downstream tasks\. Across task categories, we observe the same qualitative trends as in the aggregate results: performance is largely explained by the independent effects of the pretraining and SFT optimizer choices, with relatively small interaction effects from matching the optimizer across stages\. In particular, increasing the proportion of Muon updates during SFT generally improves performance across task types, regardless of the optimizer distribution used during pretraining\.

### B\.4Post\-SFT forgetting for 1\.2B models

Table 8:Forgetting after SFT @ LR 4e\-4, as measured by change in pretraining validation loss\.Average and standard deviation are reported over 3 random seeds\. These values are used to calculate in Figure[1](https://arxiv.org/html/2610.22650#S1.F1)\(right\)\. A smaller value indicates less forgetting and indicates that the model has retained more pretraining knowledge\.Table[8](https://arxiv.org/html/2610.22650#A2.T8)reports the average and standard deviation for change in pretraining validation loss across three random seeds after SFT on our 1\.2B Llama\-3 models\. These are the results used to calculate marginal and residual effects in Figure[1](https://arxiv.org/html/2610.22650#S1.F1)\(right\)\.

### B\.5CPT performance for 1\.2B models

Table 9:Continued pretraining \(CPT\) performance at LR 5e\-5\.Average and standard deviation are reported over 3 random seeds\. These values are used to calculate the residuals in Figure[2](https://arxiv.org/html/2610.22650#S3.F2)\(left\)\.Table 10:Continued pretraining \(CPT\) performance at best LR for each setting\.Average and standard deviation are reported over 3 random seeds\. These values are used to calculate the residuals in Figure[2](https://arxiv.org/html/2610.22650#S3.F2)\(right\)\.Table[9](https://arxiv.org/html/2610.22650#A2.T9)\(uniform LR\) and Table[10](https://arxiv.org/html/2610.22650#A2.T10)\(best LRs\) report the average and standard deviation for CPT final loss across three random seeds for the CPT experiments in Section[3](https://arxiv.org/html/2610.22650#S3)\. These are the results used to calculate row\-wise averages, column\-wise averages, and residuals in Figure[2](https://arxiv.org/html/2610.22650#S3.F2)\.