Self-Play Search Distillation for Large Language Model Reasoning
Summary
The paper introduces Self-Play Search Distillation (SPSD), a framework that uses self-play of MuZero-like networks to generate superhuman synthetic data for training LLMs, significantly improving reasoning performance on mathematics benchmarks.
View Cached Full Text
Cached at: 09/28/26, 09:52 AM
# Self-Play Search Distillation for Large Language Model Reasoning Source: [https://arxiv.org/abs/2609.30936](https://arxiv.org/abs/2609.30936) [View PDF](https://arxiv.org/pdf/2609.30936) > Abstract:Improving reasoning abilities in Large Language Models \(LLMs\) requires high\-quality data that exposes difficult decisions, competing alternatives, and their consequences\. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling\. We introduce Self\-Play Search Distillation \(SPSD\), a framework for generating superhuman synthetic data via self\-play of MuZero\-like networks trained on board games\. SPSD uses executable environments to turn search into structured reasoning problems\. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates\. By converting the self\-play search records into superhuman chains\-of\-thought, we train LLMs with environment\-grounded supervision\. Although trained only on self\-play search records, SPSD transfers to unseen mathematics\. On Qwen3\-4B\-Base, it raises the mean over six mathematics benchmarks from 24\.1 to 36\.6 while increasing the held\-out\-game win rate from 15% to 45%\. SPSD offers an annotation\-efficient way to create high\-quality synthetic data for improving LLM performance in reasoning tasks\. ## Submission history From: Lorenzo Molfetta \[[view email](https://arxiv.org/show-email/eb362dbf/2609.30936)\] **\[v1\]**Fri, 25 Sep 2026 07:53:21 UTC \(2,137 KB\)
Similar Articles
Self-Play Search Distillation for Large Language Model Reasoning
This paper introduces Self-Play Search Distillation (SPSD), a framework that uses self-play in board games to generate synthetic data for improving large language model reasoning, with demonstrated improvements on mathematical benchmarks.
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Introduces Least Square Policy Distillation (LSPD), an RL-based method for sample-efficient distillation in large language models, improving performance on mathematical reasoning benchmarks.
Self-play helped AI achieve superhuman performance in Go, so why hasn’t it done the same for LLMs? Researchers have found a solution.
Researchers introduce Self-Guided Self-Play (SGS), a self-play algorithm for LLMs that prevents reward hacking by using a Guide role to score synthetic problems. Applied to theorem proving in Lean4, SGS surpasses RL baselines and allows a 7B model to outperform a 671B model.
MindZero: Learning Online Mental Reasoning With Zero Annotations
MindZero introduces a self-supervised reinforcement learning framework that trains multimodal large language models for efficient and robust online mental reasoning without requiring mental state annotations, outperforming model-based methods in accuracy and efficiency.
Google's Self-Improvements in LLMs (18 minute read)
Google researchers propose a data-free on-policy self-distillation framework where LLMs synthesize their own curriculum of queries and privileged reasoning traces from coarse topic descriptions, achieving self-improvement across text and multimodal benchmarks without external supervision.