Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

# Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging
Source: [https://arxiv.org/html/2608.05541](https://arxiv.org/html/2608.05541)
Yu Gu3,1,∗,‡Zhi Zheng2,∗,†Yunpeng Ba1Xialiang Tong4Mingxuan Yuan4Zhenkun Wang1,† 1School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, China 2School of Computing, National University of Singapore, Singapore 3School of Intelligence Science and Technology, Nanjing University, China 4Noah’s Ark Lab, Huawei Technologies Ltd\., China ‡Work done while interning at Southern University of Science and Technology\. ∗Equal contribution\.†Co\-corresponding authors

###### Abstract

Evolution Strategy \(ES\) is a promising alternative to gradient\-based fine\-tuning for resource\-constrained Large Language Model \(LLM\) reasoning\. However, directly applying ES to billion\-parameter LLMs ishighly ineffective\. In such high\-dimensional parameter spaces, most random perturbations are nearly orthogonal to useful update directions, leading to unstable optimization\. We proposeHyper\-ES, a subspace\-based ES framework that avoids the weakness of ES in full\-parameter search whileexploiting its strength in low\-dimensional optimization\. Instead of asking ES to discover useful directions from random perturbations in the LLM parameter space,Hyper\-ESfirst performs a small number of inexpensive gradient\-based fine\-tuning runs to obtain descent directions\. Although each direction may provide only a limited improvement on its own, their span forms a compact adaptation subspace that captures useful reasoning updates\.Hyper\-ESthen applies CMA\-ES to optimize layer\-wise DARE–TIES merging coefficients within this subspace, allowing ES to search over combinations of meaningful descent directions rather than over arbitrary full\-model perturbations\. We evaluateHyper\-ESon three Qwen2\.5\-Instruct and DeepSeek\-R1\-Distill backbones across six mathematical reasoning datasets\. Results show thatHyper\-ESconsistently outperforms GRPO\-LoRA by 1% while requiring 10% fewer space\-consuming gradient updates\.111Code at[https://github\.com/kuangrepi/Hyper\-ES](https://github.com/kuangrepi/Hyper-ES)\.

Hyper\-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

## 1Introduction

Large Language Models \(LLMs\) can demonstrate incredible capabilities in mathematical reasoningLiet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib9)\), especially when equipped with the chain\-of\-thought \(CoT\) prompting techniquesWeiet al\.\([2022](https://arxiv.org/html/2608.05541#bib.bib32)\)and further refined with gradient\-based supervised fine\-tuning \(SFT\)Zheng and Lee \([2025](https://arxiv.org/html/2608.05541#bib.bib34)\)or reinforcement learning \(RL\) methodsShaoet al\.\([2024](https://arxiv.org/html/2608.05541#bib.bib13)\)\. However, although these methods achieve significant improvements in reasoning ability, the gradient backpropagation process within them leads to significantly higher time and memory consumptionLiuet al\.\([2025b](https://arxiv.org/html/2608.05541#bib.bib33)\), which hinders the fine\-tuning of LLMs for reasoning in resource\-constrained scenariosParket al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib35)\)\.

![Refer to caption](https://arxiv.org/html/2608.05541v1/x1.png)\(a\)Gradient\-based GRPO for LLM Reasoning
![Refer to caption](https://arxiv.org/html/2608.05541v1/x2.png)\(b\)Gradient\-free ES for LLM Reasoning
![Refer to caption](https://arxiv.org/html/2608.05541v1/x3.png)\(c\)Hyper\-ES \(Ours\) for ES over Merging Descent Directions

Figure 1:\(a\) GRPO updates model parameters with policy gradients, which is usually resource\-consuming\. \(b\) ES offers a gradient\-free method for LLM reasoning, suitable for resource\-constrained scenarios, but will lead to significant inefficiency & out\-of\-control random walk\. \(c\) Hyper\-ES replaces full\-parameter exploration with low\-dimensional search over coefficients of fast\-obtained descent directions, leading to Higher LLM Reasoning Performance with Fewer Gradient Updates\.Recent Evolution Strategy \(ES\) methods have emerged as a promising alternative to gradient\-based fine\-tuning for improving LLM reasoning abilityQiuet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib12)\); Sarkaret al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib30)\); Sunet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib29)\)\. Unlike gradient\-based methods \(e\.g\., Group Relative Policy Optimization \(GRPO\), shown in Figure[1](https://arxiv.org/html/2608.05541#S1.F1)\(a\)\) that backpropagate through the model, as shown in Figure[1](https://arxiv.org/html/2608.05541#S1.F1)\(b\), ES only perturbs parameters, evaluates each candidate with a verifiable reward, and updates the parameters toward better\-performing regions, demonstrating10×\\timesspace\-efficiency\(Sunet al\.,[2026](https://arxiv.org/html/2608.05541#bib.bib29)\)\. However, its direct application to LLMs is severely limited by the dimensionality\. In billion\-parameter spaces, almost all randomly sampled perturbations lie in directions that are irrelevant to task improvement\. With a limited population size, ES therefore receives extremely weak directional signals, making full\-parameter search sample\-inefficient and unstableHoyet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib18)\); Abdiet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib27)\)\.

To keep the efficiency of ES while avoiding its drawback in high\-dimensional direction discovery, we decouple the exploration of update directions from derivative\-free optimization\. Rather than asking ES to find useful descent directions through random perturbations in the full LLM parameter space,Hyper\-ESfirst constructsNNfew\-shot LoRA directions from a shared base model by running fewer than ten GRPO update steps on different data subsets\. Although these preliminary updates are cheap and individually limited, their LoRA deltas provide task\-relevant descent directions that better align with reasoning improvement than isotropic random noise\.Hyper\-ESthen treats these deltas as basis directions and applies the Covariance Matrix Adaptation Evolution Strategy \(CMA\-ES\)Hansen and Ostermeier \([2001](https://arxiv.org/html/2608.05541#bib.bib42)\)to optimize layer\-wise DARE–TIES merging coefficients over their combinations\.Hyper\-ESreformulates full\-parameter adaptation as a low\-dimensional, structured coefficient search with only hundreds of parameters, allowing ES to preserve its memory\-light and parallelizable nature while operating in the regime where it is most effective\. Empirically, we implementHyper\-ESon Qwen2\.5\-0\.5B\-Instruct, Qwen2\.5\-1\.5B\-Instruct, and DeepSeek\-R1\-Distill\-1\.5B\. Across four arithmetic reasoning benchmarks, including GSM8K, GSM\-Hard, SVAMP, and MultiArith, as well as two more challenging mathematical reasoning benchmarks, AMC23 and MATH\-500,Hyper\-ESimproves over model\-merging baselines and slightly but consistently outperforms single\-stage GRPO\. In particular, it achieves up to a 1% performance gain while requiring 10% fewer expensive backpropagation steps\. Our contributions are as follows:

- •We proposeHyper\-ES, a descent\-direction\-assisted ES framework that converts full\-parameter search into a compact layer\-wise coefficient search over task\-relevant LoRA updates\.
- •Hyper\-ESuses few\-shot GRPO updates to construct informative descent directions, allowing ES to avoid high\-dimensional random exploration while retaining its efficiency\.
- •Acrosssixmath reasoning benchmarks andthreeLLM backbones,Hyper\-ESconsistently improves over model\-merging baselines and surpasses single\-stage GRPO with fewer expensive backpropagation steps\.

## 2Preliminaries

### 2\.1LLM Reasoning

The language reasoning process addresses a given question𝑸=\(q1,…,q\|𝑸\|\)\\boldsymbol\{Q\}=\(q\_\{1\},\\ldots,q\_\{\|\\boldsymbol\{Q\}\|\}\)by first generating a series of CoT language reasoning tokens𝑹=\(r1,…,r\|𝑹\|\)\\boldsymbol\{R\}=\(r\_\{1\},\\ldots,r\_\{\|\\boldsymbol\{R\}\|\}\), followed by answer tokens𝑨=\(a1,…,a\|𝑨\|\)\\boldsymbol\{A\}=\(a\_\{1\},\\ldots,a\_\{\|\\boldsymbol\{A\}\|\}\)\. Both reasoning and answer tokens are produced according to the next\-token prediction policyπθ\\pi\_\{\\theta\}of LLMs as follows:

p​\(𝑹,𝑨\|𝑸\)=\\displaystyle p\(\\boldsymbol\{R\},\\boldsymbol\{A\}\|\\boldsymbol\{Q\}\)=∏t=1\|𝑹\|πθ​\(rt\|\[𝑸,𝒓1:t−1\]\)\\displaystyle\\prod\_\{t=1\}^\{\|\\boldsymbol\{R\}\|\}\\pi\_\{\\theta\}\(r\_\{t\}\|\[\\boldsymbol\{Q\},\\boldsymbol\{r\}\_\{1:t\-1\}\]\)∏t=1\|𝑨\|πθ​\(at\|\[𝑸,𝑹,𝒂1:t−1\]\),\\displaystyle\\qquad\\prod\_\{t=1\}^\{\|\\boldsymbol\{A\}\|\}\\pi\_\{\\theta\}\(a\_\{t\}\|\[\\boldsymbol\{Q\},\\boldsymbol\{R\},\\boldsymbol\{a\}\_\{1:t\-1\}\]\),where𝒓1:t−1=\(r1,…,rt−1\)\\boldsymbol\{r\}\_\{1:t\-1\}=\(r\_\{1\},\\ldots,r\_\{t\-1\}\)and𝒂1:t−1=\(a1,…,at−1\)\\boldsymbol\{a\}\_\{1:t\-1\}=\(a\_\{1\},\\ldots,a\_\{t\-1\}\);\[⋅,⋅\]\[\\cdot,\\cdot\],\[⋅,⋅,⋅\]\[\\cdot,\\cdot,\\cdot\]denote concatenation\.

𝒥GRPO\(θ\)=1G𝔼\{𝑹,𝑨\}g=1G∼p\(⋅,⋅\|𝑸\)\[∑g=1G1\|𝑹g\|\+\|𝑨g\|∑t=1\|𝑹g\|\+\|𝑨g\|\(min\(pg,tA^g,clip\(pg,t,1−ϵ,1\+ϵ\)A^g\)\]\\displaystyle\\mathcal\{J\}\_\{\\text\{GRPO\}\}\(\\theta\)=\\frac\{1\}\{G\}\\mathbb\{E\}\_\{\\\{\\boldsymbol\{R\},\\boldsymbol\{A\}\\\}\_\{g=1\}^\{G\}\\sim p\(\\cdot,\\cdot\|\\boldsymbol\{Q\}\)\}\\Bigg\[\\sum\_\{g=1\}^\{G\}\\frac\{1\}\{\\left\|\\boldsymbol\{R\}\_\{g\}\\right\|\+\\left\|\\boldsymbol\{A\}\_\{g\}\\right\|\}\\sum\_\{t=1\}^\{\\left\|\\boldsymbol\{R\}\_\{g\}\\right\|\+\\left\|\\boldsymbol\{A\}\_\{g\}\\right\|\}\\Big\(\\min\\left\(p\_\{g,t\}\\hat\{A\}\_\{g\},\\text\{clip\}\(p\_\{g,t\},1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{g\}\\right\)\\Bigg\]\(1\)𝑨^g=f​\(𝑨g\)−mean​\(f​\(𝑨\)\)g=1Gstd​\(f​\(𝑨\)\)g=1G,pg,t=\{πθ​\(ag,t\|\[𝑸,𝒓g,\(ag,1,…,ag,t−1\)\]\)πθold​\(ag,t\|\[𝑸,𝒓,\(ag,1,…,ag,t−1\)\]\)if​t\>\|𝑹g\|πθ​\(𝒓g,t\|\[𝑸,\(𝒓g,1,…,𝒓g,t−1\)\]\)πθold​\(𝒓g,t\|\[𝑸,\(𝒓g,1,…,𝒓g,t−1\)\]\)if​t≤\|𝑹g\|\.\\displaystyle\\quad\\hat\{\\boldsymbol\{A\}\}\_\{g\}=\\frac\{f\(\\boldsymbol\{A\}\_\{g\}\)\-\\text\{mean\}\(f\(\\boldsymbol\{A\}\)\)\_\{g=1\}^\{G\}\}\{\\text\{std\}\(f\(\\boldsymbol\{A\}\)\)\_\{g=1\}^\{G\}\},\\qquad p\_\{g,t\}=##### RL Fine\-tuning for LLM Reasoning

RL methods—such as Group Relative Policy Optimization \(GRPO\)Shaoet al\.\([2024](https://arxiv.org/html/2608.05541#bib.bib13)\), Dr\. GRPOLiuet al\.\([2025c](https://arxiv.org/html/2608.05541#bib.bib5)\), DAPOYuet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib22)\)—sample multiple candidate CoTs\[𝑹,𝑨\]\[\\boldsymbol\{R\},\\boldsymbol\{A\}\]per question and assess each with a reward reflecting the answer quality𝑨\\boldsymbol\{A\}\. For example, in standard GRPOShaoet al\.\([2024](https://arxiv.org/html/2608.05541#bib.bib13)\)\(shown in Eq\. \([1](https://arxiv.org/html/2608.05541#S2.E1)\)\),GGcandidate CoTs\{𝑹,𝑨\}g=1G\\\{\\boldsymbol\{R\},\\boldsymbol\{A\}\\\}\_\{g=1\}^\{G\}are generated for each𝑸\\boldsymbol\{Q\}, and the objective is updated according to the relative advantage𝑨^g\\hat\{\\boldsymbol\{A\}\}\_\{g\}within theseGGsamples\. These RL\-based approaches often significantly surpass Supervised Fine\-tuning \(SFT\) in mathematical reasoning tasks, making them the top choice of LLM Reasoning\. However, due to the long rollout trajectory \(usually thousands of tokens\), updating the parameter through backpropagation consumes a huge amount of time and space, which is unaffordable for resource\-constrained application scenarios\.

### 2\.2Evolution Strategy for LLM Reasoning

##### ES Fine\-tuning for LLM Reasoning

Direct ES starts from the pretrained parametersθ∈ℝd\\theta\\in\\mathbb\{R\}^\{d\}and repeatedly builds a population of nearby models\. At iterationtt, the current model isθt\\theta\_\{t\},θ0=θ\\theta\_\{0\}=\\theta\. ES samplesGGrandom perturbations from the Gaussian distribution and forms a population as follows:

θt,k=θt\+\\displaystyle\\theta\_\{t,k\}=\\theta\_\{t\}\+ϵt,k,ϵt,k∼𝒩​\(0,σ2​Id\),\\displaystyle\\epsilon\_\{t,k\},\\quad\\epsilon\_\{t,k\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\),\(2\)k=1,…,G,\\displaystyle k=1,\\ldots,G,whereϵt,k\\epsilon\_\{t,k\}is a random direction in the parameter space andσ\\sigmacontrols the degree of the perturbation\. Each candidate modelθt,k\\theta\_\{t,k\}is evaluated by a scalar fitnessf​\(θt,k\)f\(\\theta\_\{t,k\}\), such as a verifiable reward or validation accuracy \(e\.g\., the answer\-based reward𝑨^g\\hat\{\\boldsymbol\{A\}\}\_\{g\}in Eq\. \([1](https://arxiv.org/html/2608.05541#S2.E1)\)\)\. These fitness values are then used to estimate an update direction:

g^ES​\(θt\)=1G​∑k=1Gf​\(θt,k\)​ϵt,k\.\\widehat\{g\}\_\{\\mathrm\{ES\}\}\(\\theta\_\{t\}\)=\\frac\{1\}\{G\}\\sum\_\{k=1\}^\{G\}f\(\\theta\_\{t,k\}\)\\epsilon\_\{t,k\}\.The current parameters are updated as followsSalimanset al\.\([2017](https://arxiv.org/html/2608.05541#bib.bib28)\); Qiuet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib12)\); Sunet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib29)\):

θt\+1=θt\+ηt​g^ES​\(θt\)\.\\theta\_\{t\+1\}=\\theta\_\{t\}\+\\eta\_\{t\}\\widehat\{g\}\_\{\\mathrm\{ES\}\}\(\\theta\_\{t\}\)\.Hereηt\\eta\_\{t\}is the ES learning rate\. Due to ES only requiring model forward rollout and evaluations, instead of backpropagation, ES\-based LLM fine\-tuning methods usually consume less time and space compared to gradient\-based methods \(e\.g\., GRPO\)\. As mentioned inSunet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib29)\), ES consumes10×10\\timesless GPU memory usage, making it suitable for resource\-constrained scenarios\.

However, whenθt\\theta\_\{t\}is of thousands, even billions of dimensions, everyϵt,k\\epsilon\_\{t,k\}is a random vector in a billion\-dimensional space\.

##### Problem 1: directional discovery failure\.

The first drawback of direct ES is that it can hardly discover descent directions by random search\. In high dimensions, a random perturbation is likely to be nearly orthogonal to any useful descent direction\. This remains true considering ES populations, because the population size grows far more slowly than the ambient parameter dimension\.

###### Lemma 1\(High\-dimensional angles concentrate near orthogonality\)\.

Letg∗=−∇θℒ​\(θt\)g^\{\*\}=\-\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{t\}\)be any fixed nonzero descent direction\. Letu1,…,uGu\_\{1\},\\ldots,u\_\{G\}beG=G​\(d\)G=G\(d\)independently sampled unit perturbation directions inℝd\\mathbb\{R\}^\{d\}, and define the acute angle

αk=arccos⁡\(\|⟨uk,g∗‖g∗‖⟩\|\)∈\[0,π/2\]\.\\alpha\_\{k\}=\\arccos\\left\(\\left\|\\left\\langle u\_\{k\},\\frac\{g^\{\*\}\}\{\\\|g^\{\*\}\\\|\}\\right\\rangle\\right\|\\right\)\\in\[0,\\pi/2\]\.\(3\)Given a limited population \(i\.e\.,log⁡G=o​\(d\)\\log G=o\(d\)\),

max1≤k≤G⁡\|⟨uk,g∗‖g∗‖⟩\|→d→∞𝑝0,\\max\_\{1\\leq k\\leq G\}\\left\|\\left\\langle u\_\{k\},\\frac\{g^\{\*\}\}\{\\\|g^\{\*\}\\\|\}\\right\\rangle\\right\|\\xrightarrow\[d\\to\\infty\]\{p\}0,\(4\)or equivalently,

min1≤k≤G⁡αk→d→∞𝑝π2\.\\min\_\{1\\leq k\\leq G\}\\alpha\_\{k\}\\xrightarrow\[d\\to\\infty\]\{p\}\\frac\{\\pi\}\{2\}\.\(5\)

The proof is provided in Appendix[C\.1](https://arxiv.org/html/2608.05541#A3.SS1)\. For billion\-parameter LLMs, Lemma[1](https://arxiv.org/html/2608.05541#Thmlemma1)implies that a practical ES population is dominated by perturbations whose angles to the loss\-reducing direction are close to90∘90^\{\\circ\}\. The population can contain models with different rewards, but those reward differences do not imply that ES has found a direction close to the optimal update\. Instead, when the useful projection ontog∗g^\{\*\}is tiny, the update induced by population reweighting is largely formed from components orthogonal tog∗g^\{\*\}\.

##### Problem 2: orthogonal random\-walk\.

The second drawback is the accumulation of these orthogonal updates\. Even if each individual orthogonal component is uninformative for loss reduction, repeated ES updates can accumulate in the parameter space and cause the model to drift away from the pretrained initialization\.

###### Lemma 2\(Irrelevant ES updates accumulate parameter drift\)\.

Letrt=g^ES​\(θt\)−Projg∗​g^ES​\(θt\)r\_\{t\}=\\widehat\{g\}\_\{\\mathrm\{ES\}\}\(\\theta\_\{t\}\)\-\\mathrm\{Proj\}\_\{g^\{\*\}\}\\widehat\{g\}\_\{\\mathrm\{ES\}\}\(\\theta\_\{t\}\)denote the component of the ES update estimate orthogonal tog∗g^\{\*\}\. Suppose the useful component is negligible and the orthogonal components have a nonzero second moment,𝔼​‖rt‖22=ρ2\>0\\mathbb\{E\}\\\|r\_\{t\}\\\|\_\{2\}^\{2\}=\\rho^\{2\}\>0\. Then, afterTTES steps, the accumulated orthogonal parameter displacement satisfies

𝔼​\[‖∑t=0T−1ηt​rt‖22\]≈ρ2​∑t=0T−1ηt2\.\\mathbb\{E\}\\\!\\left\[\\left\\\|\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}r\_\{t\}\\right\\\|\_\{2\}^\{2\}\\right\]\\approx\\rho^\{2\}\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{2\}\.\(6\)Thus, ES can induce a large parameter\-space drift even when it makes little progress along the optimal direction, leading to unstable optimization\.

![Refer to caption](https://arxiv.org/html/2608.05541v1/x4.png)\(a\)ES leads to wrong directions
![Refer to caption](https://arxiv.org/html/2608.05541v1/x5.png)\(b\)ES leads to random walk

Figure 2:Evidence for Lemmas[1](https://arxiv.org/html/2608.05541#Thmlemma1)and[2](https://arxiv.org/html/2608.05541#Thmlemma2)\. We show the detailed raw data for this figure in Appendix[C\.2](https://arxiv.org/html/2608.05541#A3.SS2)\.We show evidence for these two in Figure[2](https://arxiv.org/html/2608.05541#S2.F2)and the proof is provided in Appendix[C\.1](https://arxiv.org/html/2608.05541#A3.SS1)\. Together, Lemmas[1](https://arxiv.org/html/2608.05541#Thmlemma1)and[2](https://arxiv.org/html/2608.05541#Thmlemma2)explain why direct ES becomesineffective and unstableat the LLM scale\. The population is likely to consist of perturbations nearly orthogonal to the optimal descent direction, while repeated updates along unrelated directions can still move the model far from the pretrained parameters\. This geometric mismatch is consistent with the observation ofHoyet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib18)\)\.

This motivates two requirements for ES\-based LLM reasoning fine\-tuning\.1\)The search dimension must be small enough for population\-based adaptation to be meaningful, and2\)the reduced search space must contain task\-relevant directions\.Hyper\-ESsatisfies both requirements by constructing a low\-dimensional search space from descent directions\.

## 3Method: Hyper\-ES

Hyper\-ESfollows the pipeline in Figure[1](https://arxiv.org/html/2608.05541#S1.F1)\(c\): it first prepares a small set of task\-relevant descent directions, and then runs ES over combinations of these directions rather than over the full model parameters\. The role of ES is therefore changed from direction discovery to direction selection\. Figure[3](https://arxiv.org/html/2608.05541#S3.F3)gives the implementation details behind this design: the left panel shows how the descent directions are obtained, and the right panel shows how a search variable is decoded into a merged model\.

![Refer to caption](https://arxiv.org/html/2608.05541v1/x6.png)\(a\)Descent Directions from few\-shot Gradient\-based Updates
![Refer to caption](https://arxiv.org/html/2608.05541v1/x7.png)\(b\)Model Merging Process with ES\-sampled coefficients𝒛\\boldsymbol\{z\}

Figure 3:Implementation details ofHyper\-ES\. \(a\) A small number of few\-shot GRPO updates extract task\-relevant descent directions from different data subsets\. \(b\) A search variable𝒛\\boldsymbol\{z\}is decoded into layer\-wise DARE–TIES merging coefficients that combine these directions into a final model\.### 3\.1Start\-up: Preparing Descent Directions

Hyper\-ESstarts from a pretrained model with parameters𝜽\\boldsymbol\{\\theta\}\. As illustrated in Figure[3](https://arxiv.org/html/2608.05541#S3.F3)\(a\), we partition the training data intoNNsubsets\{𝒟i\}i=1N\\\{\\mathcal\{D\}\_\{i\}\\\}\_\{i=1\}^\{N\}and run one short LoRA\-based GRPO update on each subset, typically 7 update steps\. These short runs are not meant to produce strong, standalone models\. Their purpose is to cheaply reveal descent directions that already contain a task signal, so that the later ES stage does not need to find such directions through random perturbations in the original parameter space\.

Let𝜽i\\boldsymbol\{\\theta\}\_\{i\}denote the parameters obtained after theii\-th short GRPO run\. For LLMs withLLlayer\-wise modules \(which means, attention or MLP\), indexed byℓ∈\{1,…,L\}\\ell\\in\\\{1,\\ldots,L\\\}, we define the descent direction as follows:

Δ​𝜽i\(ℓ\)=𝜽i\(ℓ\)−𝜽0\(ℓ\),\\Delta\\boldsymbol\{\\theta\}\_\{i\}^\{\(\\ell\)\}=\\boldsymbol\{\\theta\}\_\{i\}^\{\(\\ell\)\}\-\\boldsymbol\{\\theta\}\_\{0\}^\{\(\\ell\)\},where𝜽0\(ℓ\)\\boldsymbol\{\\theta\}\_\{0\}^\{\(\\ell\)\}and𝜽i\(ℓ\)\\boldsymbol\{\\theta\}\_\{i\}^\{\(\\ell\)\}are the base parameters and the parameters after theii\-th short run for layer\-wise groupℓ\\ell\. When the group corresponds to a LoRA\-updated weight matrix with rankrrand scaleα\\alpha, this delta is computed without materializing a full fine\-tuned model as follows:

Δ​𝜽i\(ℓ\)=αr​𝑩i\(ℓ\)​𝑨i\(ℓ\)\.\\displaystyle\\qquad\\Delta\\boldsymbol\{\\theta\}\_\{i\}^\{\(\\ell\)\}=\\frac\{\\alpha\}\{r\}\\boldsymbol\{B\}\_\{i\}^\{\(\\ell\)\}\\boldsymbol\{A\}\_\{i\}^\{\(\\ell\)\}\.𝒫=\\displaystyle\\mathcal\{P\}=\{Δ​𝜽i\(ℓ\):i=1,…,N,ℓ=1,…,L\}\\displaystyle\\left\\\{\\Delta\\boldsymbol\{\\theta\}\_\{i\}^\{\(\\ell\)\}:i=1,\\ldots,N,\\;\\ell=1,\\ldots,L\\right\\\}Then𝒫\\mathcal\{P\}is the direction pool, which can provide descent directions for the following merging search of Hyper\-ES\.

### 3\.2Hyper\-ES Search over Directions

After the start\-up stage,Hyper\-ESkeeps the pretrained model and the direction pool fixed\. It then searches over a compact variable𝒛\\boldsymbol\{z\}that specifies how to combine the cached directions\. For each layer group,𝒛\\boldsymbol\{z\}contains two types of variables: drop\-rate logits controlling DAREYuet al\.\([2024](https://arxiv.org/html/2608.05541#bib.bib10)\)sparsification, and mixing\-weight logits controlling how strongly each direction contributes\. For

𝒛=\{𝒛ℓ\}ℓ=1L∈ℝL×2​N,𝒛ℓ=\(𝒂ℓ,1:N,𝒃ℓ,1:N\),\\boldsymbol\{z\}=\\\{\\boldsymbol\{z\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}\\in\\mathbb\{R\}^\{L\\times 2N\},\\quad\\boldsymbol\{z\}\_\{\\ell\}=\(\\boldsymbol\{a\}\_\{\\ell,1:N\},\\boldsymbol\{b\}\_\{\\ell,1:N\}\),whereaℓ,ia\_\{\\ell,i\}is a drop\-rate logit andbℓ,ib\_\{\\ell,i\}is a mixing\-weight logit for directioniiin the layer\-wise groupℓ\\ell\. These logits are decoded as follows:

pi\(ℓ\)=σ​\(aℓ,i\),ωi\(ℓ\)=exp⁡\(bℓ,i\),p\_\{i\}^\{\(\\ell\)\}=\\sigma\(a\_\{\\ell,i\}\),\\qquad\\omega\_\{i\}^\{\(\\ell\)\}=\\exp\(b\_\{\\ell,i\}\),wherepi\(ℓ\)∈\(0,1\)p\_\{i\}^\{\(\\ell\)\}\\in\(0,1\)is the DARE drop rate andωi\(ℓ\)\>0\\omega\_\{i\}^\{\(\\ell\)\}\>0is the non\-negative mixing weight\.

##### From search variables to a model\.

As shown in Figure[3](https://arxiv.org/html/2608.05541#S3.F3)\(b\), given𝒛\\boldsymbol\{z\},Hyper\-ESdecodespi\(ℓ\)p\_\{i\}^\{\(\\ell\)\}andωi\(ℓ\)\\omega\_\{i\}^\{\(\\ell\)\}, applies DARE\-style sparsification to the directions, resolves sign conflicts with weighted TIESYadavet al\.\([2023](https://arxiv.org/html/2608.05541#bib.bib4)\), and obtains a merged updateΔ​𝜽¯\(ℓ\)​\(𝒛ℓ\)\\overline\{\\Delta\\boldsymbol\{\\theta\}\}^\{\(\\ell\)\}\(\\boldsymbol\{z\}\_\{\\ell\}\)for each group\. The final model is then assembled as follows:

𝜽∗\(ℓ\)​\(𝒛\)=𝜽0\(ℓ\)\+γ​Δ​𝜽¯\(ℓ\)​\(𝒛ℓ\),ℓ=1,…,L,\\boldsymbol\{\\theta\}\_\{\*\}^\{\(\\ell\)\}\(\\boldsymbol\{z\}\)=\\boldsymbol\{\\theta\}\_\{0\}^\{\(\\ell\)\}\+\\gamma\\overline\{\\Delta\\boldsymbol\{\\theta\}\}^\{\(\\ell\)\}\(\\boldsymbol\{z\}\_\{\\ell\}\),\\quad\\ell=1,\\ldots,L,whereγ\>0\\gamma\>0is a global scale selected by validation search\. Figure[3](https://arxiv.org/html/2608.05541#S3.F3)\(b\) illustrates this decoding path\. Full DARE–TIES equations are provided in Appendix[D\.1](https://arxiv.org/html/2608.05541#A4.SS1)\. Equivalently, with the base parameters𝜽0\\boldsymbol\{\\theta\}\_\{0\}and direction pool𝒫\\mathcal\{P\}fixed, the merging operator is

ℳ𝜽0,𝒫:𝒛∈ℝL×2​N↦𝜽∗​\(𝒛\),\\mathcal\{M\}\_\{\\boldsymbol\{\\theta\}\_\{0\},\\mathcal\{P\}\}:\\boldsymbol\{z\}\\in\\mathbb\{R\}^\{L\\times 2N\}\\mapsto\\boldsymbol\{\\theta\}\_\{\*\}\(\\boldsymbol\{z\}\),where𝜽∗​\(𝒛\)=ℳ𝜽0,𝒫​\(𝒛\)\\boldsymbol\{\\theta\}\_\{\*\}\(\\boldsymbol\{z\}\)=\\mathcal\{M\}\_\{\\boldsymbol\{\\theta\}\_\{0\},\\mathcal\{P\}\}\(\\boldsymbol\{z\}\)denotes the full model assembled from\{𝜽∗\(ℓ\)​\(𝒛\)\}ℓ=1L\\\{\\boldsymbol\{\\theta\}\_\{\*\}^\{\(\\ell\)\}\(\\boldsymbol\{z\}\)\\\}\_\{\\ell=1\}^\{L\}and the unchanged base parameters\. This construction changes the optimization problem from searching over millions or billions of weights to searching overL×2​NL\\times 2Nmerging variables\. In our settings, the search dimension is960960for the 0\.5B backbones \(withL=48L=48layer groups\)\. The objective is to maximize accuracy on a 600\-size validation dataset after merging:

𝒛∗=arg⁡max𝒛∈ℝL×2​N⁡f​\(𝜽∗​\(𝒛\)\),\\boldsymbol\{z\}^\{\*\}=\\arg\\max\_\{\\boldsymbol\{z\}\\in\\mathbb\{R\}^\{L\\times 2N\}\}f\\\!\\left\(\\boldsymbol\{\\theta\}\_\{\*\}\(\\boldsymbol\{z\}\)\\right\),\(7\)whereffdenotes accuracy on a validation set\.

##### CMA\-ES optimization\.

We optimize Eq\. \([7](https://arxiv.org/html/2608.05541#S3.E7)\) with CMA\-ES\. Following the DARE–TIES merging procedure, we first perform a grid search over shared merging hyperparameters to find a stable starting point\. CMA\-ES is then initialized from this point and directly searches the full vector𝒛\\boldsymbol\{z\}, allowing each layer group to choose direction\-specific drop rates and mixing weights\. It maintains a Gaussian search distribution over theL×2​NL\\times 2N\-dimensional coefficient space,

𝒛k\(t\)∼𝒩​\(𝒛\(t\),\(σ𝒛\(t\)\)2​𝑪𝒛\(t\)\),k=1,…,G\.\\boldsymbol\{z\}\_\{k\}^\{\(t\)\}\\sim\\mathcal\{N\}\\\!\\left\(\\boldsymbol\{z\}^\{\(t\)\},\(\\sigma\_\{\\boldsymbol\{z\}\}^\{\(t\)\}\)^\{2\}\\boldsymbol\{C\}\_\{\\boldsymbol\{z\}\}^\{\(t\)\}\\right\),\\quad k=1,\\ldots,G\.\(8\)Each sampled candidate is decoded into a merged model𝜽∗​\(𝒛k\(t\)\)\\boldsymbol\{\\theta\}\_\{\*\}\(\\boldsymbol\{z\}\_\{k\}^\{\(t\)\}\)and evaluated by validation accuracy\. After ranking candidates by their validation fitnessf​\(𝜽∗​\(𝒛\)\)f\\\!\\left\(\\boldsymbol\{\\theta\}\_\{\*\}\(\\boldsymbol\{z\}\)\\right\), the top\-μ\\mu\(we haveμ=G2\\mu=\\frac\{G\}\{2\}in our experiments\) candidates update the search mean as follows:

𝒛\(t\+1\)=∑k=1μ𝒛k:G\(t\),\\displaystyle\\qquad\\boldsymbol\{z\}^\{\(t\+1\)\}=\\sum\_\{k=1\}^\{\\mu\}\\boldsymbol\{z\}\_\{k:G\}^\{\(t\)\},\(9\)where𝒛k:G\(t\)\\boldsymbol\{z\}\_\{k:G\}^\{\(t\)\}denotes thekk\-th ranked candidate in generationtt\. We use the final distribution mean as𝒛∗\\boldsymbol\{z\}^\{\*\}and return the merged model𝜽∗​\(𝒛∗\)\\boldsymbol\{\\theta\}\_\{\*\}\(\\boldsymbol\{z\}^\{\*\}\)\.

## 4Experiments

In this section, we implement the proposedHyper\-ESalgorithm to reinforce the mathematical reasoning capabilities of three LLMs, includingQwen2\.5\-0\.5B\-Instruct,Qwen2\.5\-1\.5B\-Instruct, andDeepSeek\-R1\-Distill\-Qwen\-1\.5B\.

### 4\.1Implementation Details

##### Training & Testing Settings\.

This paper adopts two training datasets to incorporate reasoning step\-count information, i\.e\., GSM8K\-Aug\(Denget al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib48)\)and the DeepScaler dataset\(Luoet al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib20)\)\. We involve four arithmetic reasoning benchmarks \(GSM8K, SVAMP, MultiArith, and GSM\-Hard\) and two high\-difficulty mathematics benchmarks \(MATH\-500 and AMC2023\) for testing\. Test answers are verified using Math\-Verify\(Kydlíček,[2025](https://arxiv.org/html/2608.05541#bib.bib17)\)after standard answer normalization\. For the Qwen2\.5 backbones, the maximum generation length is capped at 1024 tokens during both training and evaluation\. For the DeepSeek\-R1\-Distill\-Qwen\-1\.5B model, the generation length is limited to 6144 tokens during training and evaluation\.

In preparing eachΔ​θ\\Delta\\thetafor Hyper\-ES, we use the inherent number of reasoning hops in GSM8K\-Aug and categories in DeepScaler to build subsets\. Each LoRA direction is produced by only 7 GRPO steps with a batch size of 256, consuming 1\.8k samples per direction\. We then buildN=10N=10directions for Qwen2\.5\-Instruct andN=7N=7directions for DeepSeek\-R1\-Distill, which limits the total training data and the gradient calculation of the proposed Hyper\-ES to 17,920 and 12,544 samples, respectively\. In contrast, the GRPO/CMA\-ES baseline is trained on the full set of direction\-generation subsets \(utilizing approximately 20,000 samples for GSM8K\-Aug and 14,000 samples for DeepScaler\), which requires more training steps and samples in total\. The specific LoRA configurations, including rankr=32r=32and scalingα=64\\alpha=64, are provided in Appendix[E\.1\.2](https://arxiv.org/html/2608.05541#A5.SS1.SSS2)\. In implementing the baseline reinforcement learning training, we utilize the verl\-0\.4\.x framework for RLVR\. Please refer to Appendix[D\.2](https://arxiv.org/html/2608.05541#A4.SS2)for more details in the ES stage\.

##### Baselines\.

We compare the proposedHyper\-ESagainst several baselines: \(i\) theBasemodel without fine\-tuning, \(ii\)GRPO\+LoRA\(Shaoet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib13)\)trained on the full dataset with equivalent gradient update steps, \(iii\)CMA\-ES\+LoRAdirectly applied over the original LoRA parameter space, and \(iv\)Average Merge\(Ilharcoet al\.,[2022](https://arxiv.org/html/2608.05541#bib.bib1)\), which averages all LoRA task vectors uniformly\. For all evaluated models, the Qwen2\.5 arithmetic benchmarks are evaluated under greedy decoding \(temperatureτ=0\\tau=0\)\. For the DeepSeek\-R1\-Distill\-Qwen\-1\.5B model, we use sampling decoding under standard evaluation configurations, setting the temperatureτ=0\.6\\tau=0\.6and top\-p to0\.950\.95\.

##### Metrics\.

For the Qwen2\.5 LLMs, we evaluate using standard accuracy \(%\) under greedy decoding \(T=0T=0\)\. For the DeepSeek\-R1\-Distill\-Qwen\-1\.5B, we report Mean@32—defined as the average Pass@1 accuracy over 32 independent sampling runs—to effectively reduce the variance of the generations on competitive mathematics datasets\.

### 4\.2Performance on Arithmetic Reasoning Benchmarks

As shown in Table[1](https://arxiv.org/html/2608.05541#S4.T1), we present the evaluation ofHyper\-ESand baselines on arithmetic reasoning datasets\.DARE\+CMA\-ESrepresents applying CMA\-ES for coefficients over the vanilla DARE merging methodYuet al\.\([2024](https://arxiv.org/html/2608.05541#bib.bib10)\)\. Overall,Hyper\-ESoutperforms direct weight\-space evolutionary search and matches or slightly exceeds the GRPO\+LoRA baseline\.

On the Qwen2\.5\-0\.5B\-Instruct base LLM,Hyper\-ESachieves an average accuracy of 57\.13%, slightly outperforming the GRPO\+LoRA baseline \(56\.23%\) and yielding a substantial margin over CMA\-ES\+LoRA \(52\.76%\)\. On Qwen2\.5\-1\.5B\-Instruct,Hyper\-ESachieves an average score of 74\.26% \(compared to 73\.51% for GRPO\+LoRA\)\. Compared to Average Merge, which yields 72\.36%,Hyper\-ESdemonstrates the advantage of optimizing layer\-wise weighting coefficients instead of executing uniform parameter mixing\.

Table 1:Arithmetic reasoning results on Qwen2\.5 models\. Accuracy \(%\) is reported\. Best result per benchmark within each model group isunderlined; best average isboldfaced; second\-best average isshaded\.Table 2:Results on DeepSeek\-R1\-Distill\-Qwen\-1\.5B trained with the DeepScaler competition curriculum\. All scores are Mean@32 \(%\)—the average of 32 independent Pass@1 evaluations, multiplied by 100\. Best result per benchmark isunderlined; best average isboldfaced; second\-best average isshaded\.Table 3:Ablation study on Qwen2\.5\-0\.5B\-Instruct\. All scores are test\-set accuracies \(%\)\.Avg\.= mean of GSM8K, SVAMP, MultiArith, and GSM\-Hard\. TheΔ\\Deltacolumn reports average\-score change\.Table 4:Efficiency\-oriented ablations on Qwen2\.5\-0\.5B\-Instruct\. All scores are test\-set accuracies \(%\)\.VariantGradient SamplesGSM8KSVAMPMultiArithGSM\-HardAvg\.GRPO\+LoRA20,00050\.5761\.3395\.5617\.4456\.23Hyper\-ES\- \(7 gradients each direction\)17,92051\.1865\.3393\.8918\.1257\.13Hyper\-ES\- \(4 gradients each direction\)10,24049\.8160\.6795\.0017\.3655\.71Hyper\-ES\- Pure\-random directionsΔ​θ\\Delta\\theta047\.0148\.3377\.7817\.2147\.58Hyper\-ES\- CMA\-ES directionsΔ​θ\\Delta\\theta045\.1957\.0087\.2214\.4850\.97
### 4\.3Performance on Mathematical Reasoning Capabilities

To evaluate the generalizability ofHyper\-ES, we conduct experiments on DeepSeek\-R1\-Distill\-Qwen\-1\.5B across three mathematical reasoning benchmarks, including GSM8K, MATH\-500, and AMC2023\. As shown in Table[2](https://arxiv.org/html/2608.05541#S4.T2),Hyper\-ESreaches an average score of 70\.97%, demonstrating performance advantages over both GRPO\+LoRA \(70\.39%\) and CMA\-ES\+LoRA \(69\.63%\)\.

Specifically,Hyper\-ESprovides measurable gains on MATH\-500, increasing the score to 76\.97% \(compared to 75\.48% for GRPO\+LoRA\), and GSM8K, increasing the score to 76\.49% \(compared to 76\.07% for GRPO\+LoRA\)\. On AMC2023,Hyper\-ESremains competitive with GRPO\+LoRA\. These results demonstrate that searching over combinations of task\-relevant descent directions generalizes well across diverse mathematical reasoning tasks\. Synthesizing the results on 3 LLMs and 6 benchmarks,Hyper\-ESis the most effective resource\-constrained fine\-tuning method for LLM Reasoning\. More results, including multi\-seed evaluation, additional baselines, code\-generation evaluation, and detailed execution time, are provided in Appendix[E\.4](https://arxiv.org/html/2608.05541#A5.SS4)\.

## 5Discussion and Analysis

### 5\.1Ablation Study

To evaluate the contribution of each component, we perform ablations on Qwen2\.5\-0\.5B\-Instruct\. Table[3](https://arxiv.org/html/2608.05541#S4.T3)uses the fullHyper\-ESresults as references\.

##### w/oCMA\-ES\.

Replacing CMA\-ES with the grid\-only variant drops the average from 57\.13% to 55\.92% \(−1\.21\-1\.21\)\. This shows that the coarse grid search can identify a useful global drop\-rate and scale, but the layer\-wise CMA\-ES search still contributes by adapting the merge coefficients to different parameter groups\.

##### w/oGrouping\.

Replacing difficulty\-aware grouping with random grouping causes the largest drop among these ablations, from 57\.13% to 53\.62% \(−3\.51\-3\.51\)\. This supports the role of structured direction\-pool construction\.

### 5\.2Discussion on Efficiency

##### Memory and wall\-clock efficiency\.

The main extra cost ofHyper\-ESis constructing several short GRPO\-induced directions\. Since these runs are independent, the peak GPU memory is comparable to a single GRPO\+LoRA run and does not grow with the number of directions\. The later merging and CMA\-ES stages use stored task vectors and validation evaluations, avoiding gradient\-memory overhead\. As shown in Appendix[E\.2](https://arxiv.org/html/2608.05541#A5.SS2), the direction runs can also be parallelized across GPU nodes without communication, reducing wall\-clock time by 1 hour on Qwen\-0\.5B\.

##### Training\-sample budget\.

Table[4](https://arxiv.org/html/2608.05541#S4.T4)shows that fewer gradient steps are sufficient to construct useful directions\. The full setting uses 7 updates per direction, totaling 17,920 samples, and reaches 57\.13% average accuracy\. Reducing this to 4 updates uses only 10,240 samples while still achieving 55\.71%\. Thus,Hyper\-EScan also save time without parallelism by shortening direction construction\.

##### Task\-aligned directions are necessary\.

The gains are not from low\-dimensional search alone\. Random directions drop to 47\.58%, close to the base model, and CMA\-ES without task\-aligned directions also fails to matchHyper\-ES\. This supports our claim that the key is searching over meaningful GRPO\-induced descent directions, not arbitrary subspaces\.

### 5\.3The Effectiveness of Hyper\-ES Mainly due to the CMA\-ES Stage

Hyper\-ESis effective because it searches over combinations of task\-aligned directions\. The complete GSM8K results in Appendix[E\.3](https://arxiv.org/html/2608.05541#A5.SS3)show that individual short LoRA directions are weak, but merging them is beneficial: on Qwen2\.5\-0\.5B, grid\-onlyHyper\-ESimproves over GRPO\+LoRA from 50\.57% to 50\.72%, and CMA\-ES further improves to 51\.18%\. On Qwen2\.5\-1\.5B, the same comparison is 74\.22%→\\rightarrow75\.21%→\\rightarrow75\.51%\.

This suggests that the gains mainly come from the search stage\. The grid search finds a useful global merge configuration, while CMA\-ES provides additional layer\-wise coefficient adaptation\.

10\.24k17\.92k20\.6k52525454565658586060Samples with gradients for trainingAverage accuracy \(%\)CMA\-ES\+LoRAGRPO\+LoRAHyper\-ESFigure 4:Sample\-efficiency curve on Qwen2\.5\-0\.5B\-Instruct arithmetic benchmarks\.Hyper\-ESachieves higher average accuracy than GRPO\+LoRA with fewer training samples, while CMA\-ES\+LoRA remains lower\.

## 6Conclusion

This paper presentsHyper\-ES, an ES framework that searches over descent direction merging hyperparameters instead of directly perturbing LLM weights\. The method addresses the high\-dimensional random\-walk failure mode of direct ES by constructing a compact, task\-aligned search space from brief direction\-generation updates\. Empirically,Hyper\-ESimproves over average merging and grid\-only merging and slightly improves over single\-stage GRPO on the reported Qwen2\.5 arithmetic benchmarks and DeepSeek\-R1\-Distill\-Qwen\-1\.5B hard math benchmarks\. The results suggest that short gradient\-based runs can be used not only to train models directly, but also to define useful low\-dimensional spaces for later derivative\-free ES optimization\.

## Limitations

This paper only considers GRPO as an effective method for fine\-tuning large models for reasoning\. Considering OPD\-based methods and SFT on certain specific datasets are also suitable as initial gradients\. In the future, we will consider further demonstrating whether the effectiveness of Hyper\-ES is limited to GRPO\.

We will also consider online dynamic methods as future work, obtaining the best direction through model iteration and performing ES search\.

## References

- Evolutionary strategies lead to catastrophic forgetting in llms\.arXiv preprint arXiv:2601\.20861\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.05541#S1.p2.1)\.
- T\. Akiba, M\. Shing, Y\. Tang, Q\. Sun, and D\. Ha \(2025\)Evolutionary optimization of model merging recipes\.Nature Machine Intelligence7\(2\),pp\. 195–204\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§E\.4\.3](https://arxiv.org/html/2608.05541#A5.SS4.SSS3.p1.1),[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§E\.4\.3](https://arxiv.org/html/2608.05541#A5.SS4.SSS3.p1.1),[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- S\. Chen, J\. Zhang, T\. Zhu, W\. Liu, S\. Gao, M\. Xiong, M\. Li, and J\. He \(2025\)Bring reason to vision: understanding perception and reasoning through model merging\.arXiv preprint arXiv:2505\.05464\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1)\.
- R\. Cheng, F\. Xiong, Y\. Wei, W\. Zhu, and C\. Yuan \(2025\)Whoever started the interference should end it: guiding data\-free model merging via task vectors\.arXiv preprint arXiv:2503\.08099\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1),[§E\.4\.2](https://arxiv.org/html/2608.05541#A5.SS4.SSS2.p1.1),[Table 12](https://arxiv.org/html/2608.05541#A5.T12.1.4.3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- Y\. Deng, Y\. Choi, and S\. Shieber \(2024\)From explicit cot to implicit cot: learning to internalize cot step by step\.arXiv preprint arXiv:2405\.14838\.Cited by:[§4\.1](https://arxiv.org/html/2608.05541#S4.SS1.SSS0.Px1.p1.1)\.
- Z\. Fu, H\. Wang, J\. Kang, K\. Kawaguchi, and J\. Wu \(2026\)Reasoning resides in layers: restoring temporal reasoning in video\-language models with layer\-selective merging\.arXiv preprint arXiv:2604\.11399\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1)\.
- Y\. Gan and P\. Isola \(2026\)Neural thickets: diverse task experts are dense around pretrained weights\.arXiv preprint arXiv:2603\.12228\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Pal: program\-aided language models\.InInternational conference on machine learning,pp\. 10764–10799\.Cited by:[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1),[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.p1.1)\.
- N\. Hansen and A\. Ostermeier \(2001\)Completely derandomized self\-adaptation in evolution strategies\.Evolutionary computation9\(2\),pp\. 159–195\.Cited by:[§D\.3](https://arxiv.org/html/2608.05541#A4.SS3.SSSx1.p1.1),[§1](https://arxiv.org/html/2608.05541#S1.p3.1)\.
- W\. Hoy, B\. Wang, and X\. Pan \(2026\)Matching accuracy, different geometry: evolution strategies vs grpo in llm post\-training\.arXiv preprint arXiv:2604\.01499\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.05541#S2.SS2.SSS0.Px3.p2.1)\.
- C\. Huang, Q\. Liu, B\. Y\. Lin, T\. Pang, C\. Du, and M\. Lin \(2023\)Lorahub: efficient cross\-task generalization via dynamic lora composition\.arXiv preprint arXiv:2307\.13269\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1),[§D\.4](https://arxiv.org/html/2608.05541#A4.SS4.SSS0.Px1.p1.1),[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.05541#S4.SS1.SSS0.Px2.p1.3)\.
- D\. Korotyshova, B\. Shaposhnikov, A\. Malakhov, A\. Khokhulin, N\. Surnachev, K\. Ovcharenko, G\. Bredis, A\. Gorbatovski, V\. Sinii, and D\. Gavrilov \(2025\)ESSA: evolutionary strategies for scalable alignment\.arXiv preprint arXiv:2507\.04453\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1)\.
- H\. Kydlíček \(2025\)Math\-Verify: Math Verification Library\.External Links:[Link](https://github.com/huggingface/math-verify)Cited by:[§4\.1](https://arxiv.org/html/2608.05541#S4.SS1.SSS0.Px1.p1.1)\.
- Z\. Li, D\. Zhang, M\. Zhang, J\. Zhang, Z\. Liu, Y\. Yao, H\. Xu, J\. Zheng, P\. Wang, X\. Chen,et al\.\(2025\)From system 1 to system 2: a survey of reasoning large language models\.arXiv preprint arXiv:2502\.17419\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- W\. Liu, S\. Song, H\. Hao, and A\. Zhou \(2025a\)EA4LLM: a gradient\-free approach to large language model optimization via evolutionary algorithms\.arXiv preprint arXiv:2510\.10603\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1)\.
- Y\. Liu, J\. Wu, Y\. He, R\. Gong, J\. Xia, L\. Li, H\. Gao, H\. Chen, B\. Bi, J\. Zhang,et al\.\(2025b\)Efficient inference for large reasoning models: a survey\.arXiv preprint arXiv:2503\.23077\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025c\)Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2608.05541#S2.SS1.SSS0.Px1.p1.7)\.
- Z\. Liu, J\. Liu, Y\. He, W\. Wang, J\. Liu, L\. Pan, X\. Hu, S\. Xiong, J\. Huang, J\. Hu,et al\.\(2025d\)Part i: tricks or traps? a deep dive into rl for llm reasoning\.arXiv preprint arXiv:2508\.08221\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1)\.
- M\. Luo, S\. Tan, J\. Wong, X\. Shi, W\. Y\. Tang, M\. Roongta, C\. Cai, J\. Luo, T\. Zhang, L\. E\. Li,et al\.\(2025\)Deepscaler: surpassing o1\-preview with a 1\.5 b model by scaling rl\.Notion Blog3\(5\)\.Cited by:[§4\.1](https://arxiv.org/html/2608.05541#S4.SS1.SSS0.Px1.p1.1)\.
- Z\. Ma, W\. Huang, G\. Song, H\. Guo, S\. Ma, Z\. Cao, and Y\. Gong \(2025\)Evolutionary system 2 reasoning: an empirical proof\.arXiv preprint arXiv:2512\.05760\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1)\.
- Open\-Thought \(2025\)tiny\-grpo: Minimal Hackable GRPO Implementation\.External Links:[Link](https://github.com/open-thought/tiny-grpo)Cited by:[§E\.4\.2](https://arxiv.org/html/2608.05541#A5.SS4.SSS2.p1.1),[Table 12](https://arxiv.org/html/2608.05541#A5.T12.1.2.1.1)\.
- T\. Park, G\. Lee, and M\. Kim \(2025\)MobileRAG: a fast, memory\-efficient, and energy\-efficient method for on\-device rag\.arXiv preprint arXiv:2507\.01079\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p1.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are nlp models really able to solve simple math word problems?\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 2080–2094\.Cited by:[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- X\. Qiu, Y\. Gan, C\. F\. Hayes, Q\. Liang, Y\. Xu, R\. Dailey, E\. Meyerson, B\. Hodjat, and R\. Miikkulainen \(2025\)Evolution strategies at scale: llm fine\-tuning beyond reinforcement learning\.arXiv preprint arXiv:2509\.24372\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.05541#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.05541#S2.SS2.SSS0.Px1.p1.13)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.p1.1)\.
- S\. Roy and D\. Roth \(2015\)Solving general arithmetic word problems\.InProceedings of the 2015 conference on empirical methods in natural language processing,pp\. 1743–1752\.Cited by:[§F\.2](https://arxiv.org/html/2608.05541#A6.SS2.p1.1)\.
- T\. Salimans, J\. Ho, X\. Chen, S\. Sidor, and I\. Sutskever \(2017\)Evolution strategies as a scalable alternative to reinforcement learning\.arXiv preprint arXiv:1703\.03864\.Cited by:[§2\.2](https://arxiv.org/html/2608.05541#S2.SS2.SSS0.Px1.p1.13)\.
- B\. Sarkar, M\. Fellows, J\. A\. Duque, A\. Letcher, A\. L\. Villares, A\. Sims, C\. Wibault, D\. Samsonov, D\. Cope, J\. Liesen,et al\.\(2025\)Evolution strategies at the hyperscale\.arXiv preprint arXiv:2511\.16652\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p2.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.05541#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.05541#S2.SS1.SSS0.Px1.p1.7),[§4\.1](https://arxiv.org/html/2608.05541#S4.SS1.SSS0.Px2.p1.3)\.
- Z\. Sun, S\. Dang, G\. Dai, and H\. Ye \(2026\)ESSAM: a novel competitive evolution strategies approach to reinforcement learning for memory efficient llms fine\-tuning\.arXiv preprint arXiv:2602\.01003\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.05541#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.05541#S2.SS2.SSS0.Px1.p1.12),[§2\.2](https://arxiv.org/html/2608.05541#S2.SS2.SSS0.Px1.p1.13)\.
- F\. Tajwar, G\. Zeng, Y\. Zhou, Y\. Song, D\. Arora, Y\. Jiang, J\. Schneider, R\. Salakhutdinov, H\. Feng, and A\. Zanette \(2026\)Maximum likelihood reinforcement learning\.arXiv preprint arXiv:2602\.02710\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p1.1)\.
- P\. Yadav, D\. Tam, L\. Choshen, C\. A\. Raffel, and M\. Bansal \(2023\)Ties\-merging: resolving interference when merging models\.Advances in neural information processing systems36,pp\. 7093–7115\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1),[§D\.4](https://arxiv.org/html/2608.05541#A4.SS4.SSS0.Px3.p1.1),[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.SSS0.Px2.p1.1),[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.05541#S3.SS2.SSS0.Px1.p1.4)\.
- E\. Yang, L\. Shen, G\. Guo, X\. Wang, X\. Cao, J\. Zhang, and D\. Tao \(2026\)Model merging in llms, mllms, and beyond: methods, theories, applications, and opportunities\.ACM Computing Surveys58\(8\),pp\. 1–41\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1)\.
- Z\. Yang, B\. Qi, H\. Sun, W\. Long, R\. Zhao, and X\. Gao \(2025\)Cabs: conflict\-aware and balanced sparsification for enhancing model merging\.arXiv preprint arXiv:2503\.01874\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1),[§E\.4\.2](https://arxiv.org/html/2608.05541#A5.SS4.SSS2.p1.1),[Table 12](https://arxiv.org/html/2608.05541#A5.T12.1.3.2.1)\.
- Q\. Yin, Y\. Wu, Z\. Shen, S\. Li, Z\. Wang, Y\. Li, C\. T\. Leong, J\. Kang, and J\. Gu \(2025a\)Evaluating parameter efficient methods for rlvr\.arXiv preprint arXiv:2512\.23165\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1)\.
- Q\. Yin, H\. Yao, J\. Chen, J\. Huang, Z\. Zhao, and F\. Su \(2025b\)Towards efficient multimodal unified reasoning model via model merging\.arXiv preprint arXiv:2510\.08987\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1)\.
- L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. Li \(2024\)Language models are super mario: absorbing abilities from homologous models as a free lunch\.InForty\-first International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px3.p1.1),[§D\.4](https://arxiv.org/html/2608.05541#A4.SS4.SSS0.Px2.p1.3),[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.SSS0.Px2.p1.1),[§F\.1](https://arxiv.org/html/2608.05541#A6.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.05541#S3.SS2.p1.2),[§4\.2](https://arxiv.org/html/2608.05541#S4.SS2.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025\)Dapo: an open\-source llm reinforcement learning system at scale, 2025\.URL https://arxiv\. org/abs/2503\.144761,pp\. 2\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2608.05541#S2.SS1.SSS0.Px1.p1.7)\.
- Z\. Zheng, Y\. Gu, W\. Liu, Y\. W\. Teh, and W\. S\. Lee \(2025\)Soft\-grpo: surpassing discrete\-token llm reinforcement learning via gumbel\-reparameterized soft\-thinking policy optimization\.arXiv preprint arXiv:2511\.06411\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1)\.
- Z\. Zheng and W\. S\. Lee \(2025\)Reasoning\-cv: fine\-tuning powerful reasoning llms for knowledge\-assisted claim verification\.arXiv preprint arXiv:2505\.12348\.Cited by:[§1](https://arxiv.org/html/2608.05541#S1.p1.1)\.
- Z\. Zheng and W\. S\. Lee \(2026\)Beyond imitation: reinforcement learning for active latent planning\.arXiv preprint arXiv:2601\.21598\.Cited by:[Appendix A](https://arxiv.org/html/2608.05541#A1.SS0.SSS0.Px2.p1.1)\.

## Appendix ARelated Work

##### Evolution Strategies for LLM Fine\-Tuning\.

As advanced LLMs are equipped with larger and larger parameters, fine\-tuning them in devices with moderate scales becomes unaffordable, and ES have emerged as efficient gradient\-free optimizers for LLMs\.Qiuet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib12)\)demonstrated that ES can scale to full\-parameter LLM fine\-tuning, matching or exceeding GRPO in sample efficiency and training stability\. This momentum has driven ES\-based methods into pre\-trainingLiuet al\.\([2025a](https://arxiv.org/html/2608.05541#bib.bib6)\), System 2 reasoning\(Maet al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib44)\), few\-shot adaptationGan and Isola \([2026](https://arxiv.org/html/2608.05541#bib.bib7)\); Korotyshovaet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib45)\), and memory\-efficient tuning via sharpness\-aware mechanisms\(Sunet al\.,[2026](https://arxiv.org/html/2608.05541#bib.bib29)\)\. However, directly applying ES to the full LLM parameter space induces an isotropic random walk in the high\-dimensional weight space, simultaneously slowing convergence and causing catastrophic forgetting of the pre\-trained manifold\(Abdiet al\.,[2026](https://arxiv.org/html/2608.05541#bib.bib27)\)\.Hyper\-ESresolves this fundamental limitation by restricting CMA\-ES to the low\-dimensional manifold of GRPO\-derived task\-vector merging coefficients, replacing unconstrained weight perturbations with task\-aligned descent directions\.

##### Reinforcement Learning for LLM Reasoning\.

GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib13); Guoet al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib19)\)and its successorsLiuet al\.\([2025c](https://arxiv.org/html/2608.05541#bib.bib5)\); Yuet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib22)\); Liuet al\.\([2025d](https://arxiv.org/html/2608.05541#bib.bib21)\); Tajwaret al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib23)\); Zhenget al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib3)\); Zheng and Lee \([2026](https://arxiv.org/html/2608.05541#bib.bib2)\)have become the standard RLVR approach for reasoning, leveraging group\-relative verifiable rewards to discover effective descent directions in the loss landscape\. The standard RLVR pipeline, however, requires simultaneously maintaining actor, reference, and reward models in GPU memory, creating prohibitive overhead for resource\-constrained deployments\.Hyper\-ESreframes GRPO as a*direction provider*: short GRPO runs expose task\-specific descent directions as LoRA task vectors, which are reused as the search basis for a subsequent memory\-light CMA\-ES stage\. As shown inYinet al\.\([2025a](https://arxiv.org/html/2608.05541#bib.bib24)\), GRPO\+LoRA can still demonstrate acceptable performance\.

##### Model Merging in LLM\.

Model merging composes capabilities by linearly manipulating stored task vectors\(Yanget al\.,[2026](https://arxiv.org/html/2608.05541#bib.bib46); Chenet al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib25); Yinet al\.,[2025b](https://arxiv.org/html/2608.05541#bib.bib26)\)\. Task Arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2608.05541#bib.bib1)\)establishes the additive framework; DARE\(Yuet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib10)\)reduces interference via random sparsification, and TIES\-Merging\(Yadavet al\.,[2023](https://arxiv.org/html/2608.05541#bib.bib4)\)via sign consensus\. To further mitigate conflicts, CABS\(Yanget al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib49)\)introduces conflict\-aware and balanced sparsification to eliminate parameter overlap, while WUDI\-Merging\(Chenget al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib50)\)guides model merging by modeling task vectors as approximate linear subspaces of linear layer inputs to eliminate interference\. Some previous work applied evolutionary black\-box search to post\-trained weight merging \(Akibaet al\.\([2025](https://arxiv.org/html/2608.05541#bib.bib11)\); Fuet al\.\([2026](https://arxiv.org/html/2608.05541#bib.bib8)\); Huanget al\.\([2023](https://arxiv.org/html/2608.05541#bib.bib47)\)\) and used CMA\-ES to optimize layer\-wise DARE/TIES parameters to fuse models fine\-tuned for different specializations of the same base, thus merging LoRA adapters across multi\-task tasks\. Unlike these approaches, Hyper\-ES considers providing different feasible coarse directions for the same task and model merging is used to implement the ES search rather than to integrate multiple capabilities\. Furthermore, Hyper\-ES pioneers a paradigm shift: instead of merging different pre\-trained specialized models, it reuses DARE\-TIES parameterization as a generalized dimensionality\-reduced manifold with arbitrary parameter increments \(Δ​W\\Delta W\)\. This elevates model merging from a post\-trained merging tool to a core optimization mechanism—directly replacing full\-space reinforcement learning updates and fundamentally solving the high\-dimensional ES random walk problem\.

## Appendix BPrompt

We use the same prompt format during rollout generation and evaluation\.

Prompt for Mathematical Reasoning\.systemLet’s think step by step and output the final answer after “\#\#\#\#”\.user\{Question\}

##### Reward Function

Rewards are binary: 1\.0 for a correct answer and 0\.0 otherwise\. Answer extraction follows a priority order: \(1\)\#\#\#\# <number\>; \(2\)`\\boxed\{\.\.\.\}`; \(3\) natural\-language patterns such as “the answer isXX”; and \(4\) the last number in the response as a fallback\. Answers are normalized by stripping commas and casting whole\-valued floats to integers before comparison\.

## Appendix CDetailed Proof

### C\.1Proofs for the ES Random\-Walk Lemmas

###### Proof of Lemma[1](https://arxiv.org/html/2608.05541#Thmlemma1)\.

By rotational symmetry of the unit sphere, we may setg∗/‖g∗‖=e1g^\{\*\}/\\\|g^\{\*\}\\\|=e\_\{1\}without loss of generality\. For a uniformly sampled unit vectoru∈ℝdu\\in\\mathbb\{R\}^\{d\}, standard concentration on the sphere gives

Pr⁡\(\|⟨u,e1⟩\|≥τ\)≤2​exp⁡\(−d​τ22\)\.\\Pr\\left\(\|\\langle u,e\_\{1\}\\rangle\|\\geq\\tau\\right\)\\leq 2\\exp\\\!\\left\(\-\\frac\{d\\tau^\{2\}\}\{2\}\\right\)\.\(10\)Applying a union bound overGGindependently sampled perturbation directionsu1,…,uGu\_\{1\},\\ldots,u\_\{G\}yields

Pr⁡\(max1≤k≤G⁡\|⟨uk,e1⟩\|≥τ\)≤2​G​exp⁡\(−d​τ22\)\.\\Pr\\\!\\left\(\\max\_\{1\\leq k\\leq G\}\|\\langle u\_\{k\},e\_\{1\}\\rangle\|\\geq\\tau\\right\)\\leq 2G\\exp\\\!\\left\(\-\\frac\{d\\tau^\{2\}\}\{2\}\\right\)\.\(11\)For any fixedτ\>0\\tau\>0, iflog⁡G=o​\(d\)\\log G=o\(d\), then2​G​exp⁡\(−d​τ2/2\)→02G\\exp\(\-d\\tau^\{2\}/2\)\\to 0\. Hence

max1≤k≤G⁡\|⟨uk,e1⟩\|→d→∞𝑝0\.\\max\_\{1\\leq k\\leq G\}\|\\langle u\_\{k\},e\_\{1\}\\rangle\|\\xrightarrow\[d\\to\\infty\]\{p\}0\.\(12\)Sinceαk=arccos⁡\(\|⟨uk,e1⟩\|\)\\alpha\_\{k\}=\\arccos\(\|\\langle u\_\{k\},e\_\{1\}\\rangle\|\)andarccos⁡\(⋅\)\\arccos\(\\cdot\)is continuous at0, this is equivalent tomink⁡αk→𝑝π/2\\min\_\{k\}\\alpha\_\{k\}\\xrightarrow\{p\}\\pi/2\. ∎

###### Proof of Lemma[2](https://arxiv.org/html/2608.05541#Thmlemma2)\.

Write the ES update as the sum of a useful component parallel tog∗g^\{\*\}and an orthogonal componentrtr\_\{t\}\. When the useful component is negligible, the parameter displacement afterTTsteps is dominated by

RT=∑t=0T−1ηt​rt\.R\_\{T\}=\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}r\_\{t\}\.\(13\)Assuming the orthogonal components have mean\-zero cross terms across iterations,𝔼​⟨rt,rs⟩≈0\\mathbb\{E\}\\langle r\_\{t\},r\_\{s\}\\rangle\\approx 0fort≠st\\neq s, we obtain

𝔼​‖RT‖22\\displaystyle\\mathbb\{E\}\\\|R\_\{T\}\\\|\_\{2\}^\{2\}=𝔼​‖∑t=0T−1ηt​rt‖22\\displaystyle=\\mathbb\{E\}\\left\\\|\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}r\_\{t\}\\right\\\|\_\{2\}^\{2\}\(14\)=∑t=0T−1ηt2​𝔼​‖rt‖22\+2​∑s<tηs​ηt​𝔼​⟨rs,rt⟩\\displaystyle=\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{2\}\\mathbb\{E\}\\\|r\_\{t\}\\\|\_\{2\}^\{2\}\+2\\sum\_\{s<t\}\\eta\_\{s\}\\eta\_\{t\}\\mathbb\{E\}\\langle r\_\{s\},r\_\{t\}\\rangle≈ρ2​∑t=0T−1ηt2\.\\displaystyle\\approx\\rho^\{2\}\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{2\}\.Thus the orthogonal updates do not cancel in squared norm; they accumulate as random\-walk drift in parameter space\. ∎

###### Proposition 1\(Norm bound for merged updates\)\.

LetΔ​θ¯\(ℓ\)\\overline\{\\Delta\\theta\}^\{\(\\ell\)\}denote the merged update for groupℓ\\ell\. If DARE sparsification is applied without rescaling and the TIES disjoint merge uses non\-negative mixing weights, then

‖γ​Δ​θ¯\(ℓ\)‖F≤γ​maxi⁡‖Δ​θi\(ℓ\)‖F\.\\left\\\|\\gamma\\overline\{\\Delta\\theta\}^\{\(\\ell\)\}\\right\\\|\_\{F\}\\leq\\gamma\\max\_\{i\}\\left\\\|\\Delta\\theta\_\{i\}^\{\(\\ell\)\}\\right\\\|\_\{F\}\.\(15\)

###### Proof sketch\.

Sparsification cannot increase the Frobenius norm\. The TIES disjoint merge computes, coordinate\-wise, a weighted average over sign\-agreeing entries with non\-negative weights\. Thus the unscaled merged update is bounded by the largest sparsified task\-vector norm\. Multiplication byγ\\gammagives the result\. ∎

###### Proposition 2\(Alignment in the task\-vector subspace\)\.

Let𝒮=span​\{Δ​θ1,…,Δ​θN\}\\mathcal\{S\}=\\mathrm\{span\}\\\{\\Delta\\theta\_\{1\},\\ldots,\\Delta\\theta\_\{N\}\\\}and letg∗g^\{\*\}be a target descent direction\. If the projection ofg∗g^\{\*\}onto𝒮\\mathcal\{S\}has cosinec\>0c\>0, then an isotropic perturbation in𝒮\\mathcal\{S\}has expected alignmentΘ​\(c/dim𝒮\)\\Theta\(c/\\sqrt\{\\dim\\mathcal\{S\}\}\)withg∗g^\{\*\}, whereas an isotropic perturbation in the full space has expected alignmentO​\(1/d\)O\(1/\\sqrt\{d\}\)\.

This statement formalizes why the low\-dimensional subspace alone is insufficient: it must also be task\-aligned\. GRPO\-derived task vectors are intended to provide this alignment\.

Table 5:Pairwise cosine similarity between GRPO\+LoRA and CMA\-ES\+LoRA task vectors on Qwen2\.5\-0\.5B, each trained for 10 steps on GSM8K\. The near\-zero off\-diagonal value supports the directional\-mismatch evidence in Figure[2](https://arxiv.org/html/2608.05541#S2.F2)\.Table 6:Total adapterℓ2\\ell\_\{2\}norm‖Δ​W‖F\\\|\\Delta W\\\|\_\{F\}summed over all expanded LoRA modules \(scalingα/r=2\.0\\alpha/r=2\.0\)\. CMA\-ES\+LoRA produces adapters significantly larger than GRPO\+LoRA andHyper\-ES\.
### C\.2Additional Evidence for ES Drift

This appendix provides numerical diagnostics that correspond to the direct\-ES failure modes illustrated in Figure[2](https://arxiv.org/html/2608.05541#S2.F2)\. Table[5](https://arxiv.org/html/2608.05541#A3.T5)shows that CMA\-ES\+LoRA moves in a direction almost orthogonal to the GRPO\+LoRA update, while Table[6](https://arxiv.org/html/2608.05541#A3.T6)shows that CMA\-ES\+LoRA also produces much larger adapter norms\.

## Appendix DDetailed Methods

### D\.1Detailed Merging Operator

This appendix gives the full parameterization of the merging operator used in Section[3](https://arxiv.org/html/2608.05541#S3)\. For each groupℓ∈\{1,…,L\}\\ell\\in\\\{1,\\ldots,L\\\}, the search variable𝒛ℓ=\(𝒂ℓ,1:N,𝒃ℓ,1:N\)\\boldsymbol\{z\}\_\{\\ell\}=\(\\boldsymbol\{a\}\_\{\\ell,1:N\},\\boldsymbol\{b\}\_\{\\ell,1:N\}\)contains a drop\-rate logit and a mixing\-weight logit for each directioni∈\{1,…,N\}i\\in\\\{1,\\ldots,N\\\}\. They are decoded as

pi\(ℓ\)=σ​\(aℓ,i\),ωi\(ℓ\)=exp⁡\(bℓ,i\)\.p\_\{i\}^\{\(\\ell\)\}=\\sigma\(a\_\{\\ell,i\}\),\\qquad\\omega\_\{i\}^\{\(\\ell\)\}=\\exp\(b\_\{\\ell,i\}\)\.\(16\)
##### LoRA task vectors\.

For a LoRA adapter with rankrrand scalingα\\alpha, the task vector is

Δ​θi\(ℓ\)=αr​Bi\(ℓ\)​Ai\(ℓ\)\.\\Delta\\theta\_\{i\}^\{\(\\ell\)\}=\\frac\{\\alpha\}\{r\}B\_\{i\}^\{\(\\ell\)\}A\_\{i\}^\{\(\\ell\)\}\.\(17\)These deltas are precomputed and cached before the search\.

##### DARE sparsification\.

Given the decoded drop rate, we sample a Bernoulli mask and sparsify the task vector:

Δ​θ~i\(ℓ\)\\displaystyle\\widetilde\{\\Delta\\theta\}\_\{i\}^\{\(\\ell\)\}=Δ​θi\(ℓ\)⊙Mi\(ℓ\),\\displaystyle=\\Delta\\theta\_\{i\}^\{\(\\ell\)\}\\odot M\_\{i\}^\{\(\\ell\)\},\(18\)Mi\(ℓ\)\\displaystyle M\_\{i\}^\{\(\\ell\)\}∼Bernoulli​\(1−pi\(ℓ\)\)\.\\displaystyle\\sim\\mathrm\{Bernoulli\}\(1\-p\_\{i\}^\{\(\\ell\)\}\)\.Unlike canonical DARE, we omit the1/\(1−p\)1/\(1\-p\)rescaling factor and instead control magnitude through a separate global scale\.

##### TIES sign election and disjoint merge\.

For each scalar parameterjj, the elected sign is

s^j\(ℓ\)=sign​\(∑i=1Nωi\(ℓ\)​Δ​θ~i,j\(ℓ\)\)\.\\hat\{s\}\_\{j\}^\{\(\\ell\)\}=\\mathrm\{sign\}\\\!\\left\(\\sum\_\{i=1\}^\{N\}\\omega\_\{i\}^\{\(\\ell\)\}\\widetilde\{\\Delta\\theta\}\_\{i,j\}^\{\(\\ell\)\}\\right\)\.\(19\)Only sign\-agreeing entries contribute to the merged update:

Δ​θ¯j\(ℓ\)=∑iωi\(ℓ\)​Δ​θ~i,j\(ℓ\)​1​\[sgn​\(Δ​θ~i,j\(ℓ\)\)=s^j\(ℓ\)\]∑iωi\(ℓ\)​𝟏​\[sgn​\(Δ​θ~i,j\(ℓ\)\)=s^j\(ℓ\)\]\.\\overline\{\\Delta\\theta\}\_\{j\}^\{\(\\ell\)\}=\\frac\{\\sum\_\{i\}\\omega\_\{i\}^\{\(\\ell\)\}\\widetilde\{\\Delta\\theta\}\_\{i,j\}^\{\(\\ell\)\}\\,\\mathbf\{1\}\[\\mathrm\{sgn\}\(\\widetilde\{\\Delta\\theta\}\_\{i,j\}^\{\(\\ell\)\}\)=\\hat\{s\}\_\{j\}^\{\(\\ell\)\}\]\}\{\\sum\_\{i\}\\omega\_\{i\}^\{\(\\ell\)\}\\mathbf\{1\}\[\\mathrm\{sgn\}\(\\widetilde\{\\Delta\\theta\}\_\{i,j\}^\{\(\\ell\)\}\)=\\hat\{s\}\_\{j\}^\{\(\\ell\)\}\]\}\.\(20\)The final update is scaled and applied as

θ∗\(ℓ\)=θ0\(ℓ\)\+γ​Δ​θ¯\(ℓ\),γ=scale×N\.\\theta\_\{\*\}^\{\(\\ell\)\}=\\theta\_\{0\}^\{\(\\ell\)\}\+\\gamma\\overline\{\\Delta\\theta\}^\{\(\\ell\)\},\\qquad\\gamma=\\texttt\{scale\}\\times N\.\(21\)

### D\.2Search Details

##### Grid warm start\.

The warm\-start stage restricts all groups to share the same drop rate and scale:

pi\(ℓ\)=p,ωi\(ℓ\)=1,γ=scale×N,∀i,ℓ\.p\_\{i\}^\{\(\\ell\)\}=p,\\quad\\omega\_\{i\}^\{\(\\ell\)\}=1,\\quad\\gamma=\\texttt\{scale\}\\times N,\\qquad\\forall i,\\ell\.\(22\)We sweepscale∈\{1\.0,1\.1,…,3\.0\}\\in\\\{1\.0,1\.1,\\ldots,3\.0\\\}andp∈\{0\.3,0\.5,0\.7,0\.9\}p\\in\\\{0\.3,0\.5,0\.7,0\.9\\\}across DARE/TIES flag combinations\. In our experiments, the optimal global configurations are identified asscale=2\.7\\texttt\{scale\}=2\.7andp=0\.9p=0\.9for Qwen2\.5\-0\.5B,scale=2\.6\\texttt\{scale\}=2\.6andp=0\.7p=0\.7for Qwen2\.5\-1\.5B, andscale=2\.9\\texttt\{scale\}=2\.9andp=0\.5p=0\.5for DeepSeek\-R1\-Distill\-Qwen\-1\.5B, all of which serve as the initialization for the subsequent CMA\-ES stage\.

##### CMA\-ES\.

CMA\-ES operates in theL×2​NL\\times 2N\-dimensional merging space\. At generationtt, it samplesGGcandidates

zk\(t\)∼𝒩​\(mz\(t\),\(σz\(t\)\)2​Cz\(t\)\),k=1,…,G\.z\_\{k\}^\{\(t\)\}\\sim\\mathcal\{N\}\\\!\\left\(m\_\{z\}^\{\(t\)\},\(\\sigma\_\{z\}^\{\(t\)\}\)^\{2\}C\_\{z\}^\{\(t\)\}\\right\),\\quad k=1,\\ldots,G\.\(23\)Each candidate is decoded into a merged model and evaluated on the validation set:

fk\(t\)=𝒜​\(θ∗​\(zk\(t\)\)\)\.f\_\{k\}^\{\(t\)\}=\\mathcal\{A\}\\\!\\left\(\\theta\_\{\*\}\(z\_\{k\}^\{\(t\)\}\)\\right\)\.\(24\)Before ranking candidates, we normalize fitness values within each generation:

f^k\(t\)=fk\(t\)−meanj​\(fj\(t\)\)stdj​\(fj\(t\)\)\+ε\.\\widehat\{f\}\_\{k\}^\{\(t\)\}=\\frac\{f\_\{k\}^\{\(t\)\}\-\\mathrm\{mean\}\_\{j\}\(f\_\{j\}^\{\(t\)\}\)\}\{\\mathrm\{std\}\_\{j\}\(f\_\{j\}^\{\(t\)\}\)\+\\varepsilon\}\.\(25\)The top\-μ\\mucandidates under the normalized fitness update the mean:

mz\(t\+1\)=∑k=1μck​zk:G\(t\),ck\>0,∑k=1μck=1\.m\_\{z\}^\{\(t\+1\)\}=\\sum\_\{k=1\}^\{\\mu\}c\_\{k\}z\_\{k:G\}^\{\(t\)\},\\quad c\_\{k\}\>0,\\quad\\sum\_\{k=1\}^\{\\mu\}c\_\{k\}=1\.\(26\)We initializemz\(0\)m\_\{z\}^\{\(0\)\}from the best grid configuration, useσz\(0\)=0\.3\\sigma\_\{z\}^\{\(0\)\}=0\.3, and setCz\(0\)=IC\_\{z\}^\{\(0\)\}=I\. The default CMA\-ES population size grows logarithmically with dimension,

G=4\+⌊3​ln⁡\(L×2​N\)⌋,G=4\+\\lfloor 3\\ln\(L\\times 2N\)\\rfloor,\(27\)which is about2424in our mainL×2​N=960L\\times 2N=960search space\. The final solution is taken from the converged distribution mean rather than the historical best candidate\.

### D\.3Detailed Introduction to Evolution Strategies

#### CMA\-ES update rules

CMA\-ES\(Hansen and Ostermeier,[2001](https://arxiv.org/html/2608.05541#bib.bib42)\)maintains a Gaussian search distribution𝒩​\(m\(t\),\(σ\(t\)\)2​C\(t\)\)\\mathcal\{N\}\(m^\{\(t\)\},\(\\sigma^\{\(t\)\}\)^\{2\}C^\{\(t\)\}\)and updates it through mean recombination, cumulative step\-size adaptation, and covariance adaptation\.

##### Mean update\.

After sortingGGoffspring by fitness, the top\-μ\\mucandidates define the next mean:

m\(t\+1\)=∑k=1μck​xk:G\(t\),ck\>0,∑k=1μck=1\.m^\{\(t\+1\)\}=\\sum\_\{k=1\}^\{\\mu\}c\_\{k\}x\_\{k:G\}^\{\(t\)\},\\qquad c\_\{k\}\>0,\\quad\\sum\_\{k=1\}^\{\\mu\}c\_\{k\}=1\.\(28\)

##### Step\-size adaptation\.

The global step size is adapted with a cumulative evolution path:

pσ\(t\+1\)\\displaystyle p\_\{\\sigma\}^\{\(t\+1\)\}=\(1−cσ\)​pσ\(t\)\\displaystyle=\(1\-c\_\{\\sigma\}\)p\_\{\\sigma\}^\{\(t\)\}\+cσ​\(2−cσ\)​μw​\(C\(t\)\)−1/2\\displaystyle\\quad\+\\sqrt\{c\_\{\\sigma\}\(2\-c\_\{\\sigma\}\)\\mu\_\{w\}\}\\,\(C^\{\(t\)\}\)^\{\-1/2\}×m\(t\+1\)−m\(t\)σ\(t\),\\displaystyle\\quad\\times\\frac\{m^\{\(t\+1\)\}\-m^\{\(t\)\}\}\{\\sigma^\{\(t\)\}\},\(29\)σ\(t\+1\)\\displaystyle\\sigma^\{\(t\+1\)\}=σ\(t\)​exp⁡\(cσdσ​\(‖pσ\(t\+1\)‖χn−1\)\),\\displaystyle=\\sigma^\{\(t\)\}\\exp\\\!\\left\(\\frac\{c\_\{\\sigma\}\}\{d\_\{\\sigma\}\}\\left\(\\frac\{\\\|p\_\{\\sigma\}^\{\(t\+1\)\}\\\|\}\{\\chi\_\{n\}\}\-1\\right\)\\right\),\(30\)
whereχn=𝔼​‖𝒩​\(0,In\)‖\\chi\_\{n\}=\\mathbb\{E\}\\\|\\mathcal\{N\}\(0,I\_\{n\}\)\\\|is the expected norm of a standard Gaussian innndimensions\.

##### Covariance update\.

The covariance matrix is updated with rank\-one and rank\-μ\\muterms:

C\(t\+1\)\\displaystyle C^\{\(t\+1\)\}=\(1−c1−cμ\)​C\(t\)\+c1​pc\(t\+1\)​\(pc\(t\+1\)\)⊤\\displaystyle=\(1\-c\_\{1\}\-c\_\{\\mu\}\)C^\{\(t\)\}\+c\_\{1\}p\_\{c\}^\{\(t\+1\)\}\(p\_\{c\}^\{\(t\+1\)\}\)^\{\\top\}\(31\)\+cμ​∑k=1μck​yk:G\(t\)​\(yk:G\(t\)\)⊤,\\displaystyle\\quad\+c\_\{\\mu\}\\sum\_\{k=1\}^\{\\mu\}c\_\{k\}y\_\{k:G\}^\{\(t\)\}\(y\_\{k:G\}^\{\(t\)\}\)^\{\\top\},whereyk:G\(t\)=\(xk:G\(t\)−m\(t\)\)/σ\(t\)y\_\{k:G\}^\{\(t\)\}=\(x\_\{k:G\}^\{\(t\)\}\-m^\{\(t\)\}\)/\\sigma^\{\(t\)\}andpc\(t\)p\_\{c\}^\{\(t\)\}is the covariance evolution path\. These updates explain why CMA\-ES benefits from a compact search space: covariance estimation is meaningful only when the population provides enough signal relative to the dimension\.

### D\.4Detailed Introduction to Model Merging

##### Task arithmetic\.

Task arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2608.05541#bib.bib1)\)computes task vectorsΔ​Wi=Wi−W0\\Delta W\_\{i\}=W\_\{i\}\-W\_\{0\}and combines fine\-tuned models as

W∗=W0\+∑iλi​Δ​Wi\.W^\{\*\}=W\_\{0\}\+\\sum\_\{i\}\\lambda\_\{i\}\\Delta W\_\{i\}\.\(32\)It is simple and efficient, but it can suffer from interference when task vectors have conflicting parameter signs\.

##### DARE\.

DARE\(Yuet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib10)\)reduces interference by randomly sparsifying task vectors:

Δ​W~i\\displaystyle\\widetilde\{\\Delta W\}\_\{i\}=11−p​Δ​Wi⊙mi,\\displaystyle=\\frac\{1\}\{1\-p\}\\Delta W\_\{i\}\\odot m\_\{i\},\(33\)mi\\displaystyle m\_\{i\}∼Bernoulli​\(1−p\)\.\\displaystyle\\sim\\mathrm\{Bernoulli\}\(1\-p\)\.The canonical1/\(1−p\)1/\(1\-p\)factor preserves expected magnitude\.Hyper\-ESomits this fixed factor in the main merging operator and instead controls magnitude with the validation\-selected global scaleγ\\gamma\.

##### TIES\-Merging\.

TIES\-Merging\(Yadavet al\.,[2023](https://arxiv.org/html/2608.05541#bib.bib4)\)mitigates sign conflicts by electing an aggregate sign and averaging only sign\-consistent entries\.Hyper\-ESapplies this sign election after DARE sparsification and makes the election weight\-aware through the CMA\-ES\-decoded mixing weights\.

## Appendix EDetailed Experiment

### E\.1Training Details

Table 7:GRPO training hyperparameters\.Table 8:LoRA adapter configuration used across all models\.Table 9:Time efficiency breakdown ofHyper\-ESacross different stages\. Execution times are measured primarily on 8 NVIDIA GeForce RTX 3090 \(24GB\) GPUs\.#### E\.1\.1GRPO Hyperparameters

The detailed hyperparameter configurations used during the GRPO training phase across different model backbones are summarized in Table[7](https://arxiv.org/html/2608.05541#A5.T7)\.

#### E\.1\.2LoRA Configuration

The specific structural parameters and training configurations applied to the LoRA adapters across all evaluated backbones are detailed in Table[8](https://arxiv.org/html/2608.05541#A5.T8)\.

### E\.2Time Efficiency

Experiments in this paper are mainly conducted on 8 Nvidia 3090 24GB GPUs, and we also involve A100 and H100 GPUs for larger settings\. Table[9](https://arxiv.org/html/2608.05541#A5.T9)provides a detailed breakdown of the execution time consumed during each phase ofHyper\-EScompared to the GRPO\-LoRA baseline\.

While maintaining an equivalent memory footprint to GRPO\-LoRA,Hyper\-ESsignificantly reduces the wall\-clock execution time by bypassing the heavy communication overhead typical of distributed reinforcement learning\. Taking our Qwen2\.5\-0\.5B\-Instruct experiment with 10 directions as a representative case, because the training of multiple few\-shot LoRAs is entirely independent, they can be launched concurrently in an embarrassingly parallel fashion\. This compresses the direction\-generation phase to just1,7691,769s \(29\.529\.5min\) for the longest single run\. Combined with the subsequent decoupled grid search and low\-dimensional CMA\-ES, the end\-to\-end wall\-clock time is only3\.233\.23h\. This represents an approximate24%24\\%reduction in total training time compared to standard GRPO\+LoRA \(4\.254\.25h\), demonstrating both the parallelizability and the overall efficiency of our framework\.

### E\.3Performances over Stages

Table[10](https://arxiv.org/html/2608.05541#A5.T10)reports validation and GSM8K test accuracy for individual difficulty\-level adapters\. Individual LoRA adapters each train on a single difficulty bucket; their results show that the benefit ofHyper\-EScomes from merging the direction pool rather than selecting one direction\.

Table 10:Complete per\-method GSM8K results\. Val\. and Test denote held\-out validation and full GSM8K test accuracy \(%\) under greedy decoding \(T=0T=0\)\.
### E\.4Additional Results

#### E\.4\.1Multi\-seed Results

We conduct three independent runs on Qwen2\.5\-0\.5B\-Instruct by using different random seeds for dataset splitting, short GRPO training, direction\-pool construction, and CMA\-ES search\. Therefore, a new direction pool is constructed in each run\.

Table 11:Multi\-seed results on Qwen2\.5\-0\.5B\-Instruct\. We report mean±\\pmstandard deviation over three independent runs\.Compared with GRPO\+LoRA,Hyper\-ESimproves the average score from56\.04±0\.2956\.04\\pm 0\.29to56\.62±0\.4456\.62\\pm 0\.44\. The modest standard deviation across three independent runs, further demonstrates the stability ofHyper\-ESacross different direction\-pool constructions\.

#### E\.4\.2Additional Baselines

We further compareHyper\-ESwith tinyGRPO\(Open\-Thought,[2025](https://arxiv.org/html/2608.05541#bib.bib61)\), CABS\(Yanget al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib49)\), and WUDI\-Merging\(Chenget al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib50)\)\. We apply the model\-merging baselines to the same direction pool used byHyper\-ES\.

Table 12:Comparison with additional baselines on Qwen2\.5\-0\.5B\-Instruct\.Hyper\-ESachieves the best result on all four benchmarks\. It improves the average score by 6\.92 points over tinyGRPO and by 1\.56 points over the strongest model\-merging baseline\.

#### E\.4\.3Out\-of\-domain Code Generation

To evaluate the generalizability ofHyper\-ESto code\-generation tasks, we conduct experiments on MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2608.05541#bib.bib60)\)and HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2608.05541#bib.bib62)\)\. The evaluation covers the MBPP test set with 500 problems and the HumanEval test set with 164 problems, using greedy decoding withT=0T=0and a maximum output length of 1,024 tokens\.

Table 13:Out\-of\-domain generalization results on code\-generation tasks\.Hyper\-ESachieves higher average code\-generation performance on both models\. On Qwen2\.5\-1\.5B, the average score increases from 38\.44 to 46\.45\. These results demonstrate thatHyper\-ESgeneralizes effectively to code\-generation tasks\.

## Appendix FBaselines & Datasets & License

### F\.1Baselines

Based onQwen2\.5\-0\.5B/1\.5B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib16)\)andDeepSeek\-R1\-Distill\-Qwen\-1\.5B\(Guoet al\.,[2025](https://arxiv.org/html/2608.05541#bib.bib19)\), we include a fine\-tuning baseline \(GRPO\+LoRA\), direct CMA\-ES\+LoRA, model\-merging baselines such as Average Merge\(Ilharcoet al\.,[2022](https://arxiv.org/html/2608.05541#bib.bib1)\)and DARE\+TIES\(Yuet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib10); Yadavet al\.,[2023](https://arxiv.org/html/2608.05541#bib.bib4)\), andHyper\-ES\.

##### Fine\-tuning Baseline

GRPO\+LoRA trains a single LoRA adapter on the full training corpus for the same total number of gradient steps as the combined direction\-generation budget\. We implement it using VERL\. The training prompt is given in Appendix[B](https://arxiv.org/html/2608.05541#A2)\. This baseline controls for the total GRPO compute invested byHyper\-ESand isolates the effect of the evolutionary merging stage\.

##### Model Merging Baselines

Average Merge uniformly averages allNNLoRA task vectors\. DARE\+TIES applies random sparsification\(Yuet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib10)\)followed by sign\-conflict resolution\(Yadavet al\.,[2023](https://arxiv.org/html/2608.05541#bib.bib4)\)at fixed drop rates\. These baselines share the same direction pool asHyper\-ESbut use fixed or grid\-searched hyperparameters rather than CMA\-ES\-optimized per\-layer coefficients\.

### F\.2Datasets

We adopt GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.05541#bib.bib15)\)for in\-domain evaluation, and GSM\-Hard\(Gaoet al\.,[2023](https://arxiv.org/html/2608.05541#bib.bib41)\), SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2608.05541#bib.bib39)\), and MultiArith\(Roy and Roth,[2015](https://arxiv.org/html/2608.05541#bib.bib40)\)for out\-of\-domain evaluation on Qwen2\.5 models\. For DeepSeek\-R1\-Distill\-Qwen\-1\.5B, we additionally report MATH\-500\(Lightmanet al\.,[2024](https://arxiv.org/html/2608.05541#bib.bib38)\), AMC23\. For code\-generation evaluation, we use the MBPP test set\(Austinet al\.,[2021](https://arxiv.org/html/2608.05541#bib.bib60)\)with 500 problems and HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2608.05541#bib.bib62)\)with 164 problems\.

### F\.3License

For base LLMs, datasets, and frameworks, we list their licenses in Table[14](https://arxiv.org/html/2608.05541#A6.T14)\.

Table 14:A summary of licenses\.
### F\.4LLM Usage

This paper employs Claude Code for experiments and writing, but we are responsible for the contents\.