Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Summary
This paper investigates Evolution Strategies (ES) as a post-training paradigm for LLM reasoning, showing that ES provides broader reasoning coverage and better Pass@K performance than GRPO through sparse functional updates and population diversity.
View Cached Full Text
Cached at: 08/28/26, 03:24 AM
Paper page - Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Source: https://huggingface.co/papers/2608.27351
Abstract
Evolution strategies improve reasoning diversity and Pass@K over GRPO through sparse functional updates and population diversity, supporting a hybrid training approach.
Evolution Strategies(ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g.,Group Relative Policy Optimization(GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES overGRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show thatverifier-projected Jensen-Shannon diversityacross the ES population is helpful to higherPass@Kperformances. Empirically, unlikeGRPO, which exhibitsentropy collapse, ES improves Pass@1 while attaining higherPass@KthanGRPO. We further develop a sequentialGRPO-ES training strategy that combinesGRPO’s strength in Pass@1 with ES’s gains inPass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. Thisfunctional sparsitysuggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead tocatastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smallerpopulation sizein a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative toGRPO.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.27351
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.27351 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.27351 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.27351 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging
This paper proposes Hyper-ES, a subspace-based evolution strategy framework for LLM reasoning that obtains descent directions via lightweight gradient-based fine-tuning and then uses CMA-ES to merge layer-wise DARE-TIES coefficients, consistently outperforming GRPO-LoRA while requiring fewer gradient updates.
Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies
This paper proposes using Evolution Strategies (ES) instead of Reinforcement Learning for post-training LLMs, showing that ES improves solution coverage (pass@k) and achieves better results on math benchmarks.
@Kevin_GuoweiXu: How should LLMs sample on hard reasoning problems during post-training and inference where direct rollouts rarely produ…
Introduces BES (Bidirectional Evolutionary Search), a search framework for LLMs that combines forward candidate evolution with backward goal decomposition to improve sampling on hard reasoning problems during post-training and inference.
When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
This paper investigates when chain-of-thought reasoning is beneficial for LLMs, showing that early-stage entropy dynamics reliably indicate reasoning utility, and introduces EDRM, a lightweight, training-free framework that adaptively selects inference strategies to achieve significant token savings while maintaining or improving accuracy.
What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search
Large-scale study of 15 LLMs across 8 tasks reveals that optimization success hinges on maintaining localized search trajectories rather than initial problem-solving ability or solution novelty.