@Kevin_GuoweiXu: How should LLMs sample on hard reasoning problems during post-training and inference where direct rollouts rarely produ…
Summary
Introduces BES (Bidirectional Evolutionary Search), a search framework for LLMs that combines forward candidate evolution with backward goal decomposition to improve sampling on hard reasoning problems during post-training and inference.
Similar Articles
Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
This paper investigates Evolution Strategies (ES) as a post-training paradigm for LLM reasoning, showing that ES provides broader reasoning coverage and better Pass@K performance than GRPO through sparse functional updates and population diversity.
Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies
This paper proposes using Evolution Strategies (ES) instead of Reinforcement Learning for post-training LLMs, showing that ES improves solution coverage (pass@k) and achieves better results on math benchmarks.
Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging
This paper proposes Hyper-ES, a subspace-based evolution strategy framework for LLM reasoning that obtains descent directions via lightweight gradient-based fine-tuning and then uses CMA-ES to merge layer-wise DARE-TIES coefficients, consistently outperforming GRPO-LoRA while requiring fewer gradient updates.
@lateinteraction: very cool work !!
Guowei Xu discusses limitations of Best-of-N and tree search methods for LLMs on hard reasoning problems, noting sparse verification signals and that candidates remain within the model's distribution.
Amortizing Sequential Experimental Design in Reasoning LLMs
This paper proposes amortizing sequential Bayesian experimental design into LLMs by training the reasoning that guides action selection with information-gain rewards. Trained on Twenty Questions, the resulting policies outperform training- and inference-time BED baselines at lower cost, generalize to unseen categories, and transfer to the Medi-Q clinical dialogue benchmark.