Self-Play Search Distillation for Large Language Model Reasoning

arXiv cs.AI Papers

Summary

The paper introduces Self-Play Search Distillation (SPSD), a framework that uses self-play of MuZero-like networks to generate superhuman synthetic data for training LLMs, significantly improving reasoning performance on mathematics benchmarks.

arXiv:2609.30936v1 Announce Type: new Abstract: Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.
Original Article
View Cached Full Text

Cached at: 09/28/26, 09:52 AM

# Self-Play Search Distillation for Large Language Model Reasoning
Source: [https://arxiv.org/abs/2609.30936](https://arxiv.org/abs/2609.30936)
[View PDF](https://arxiv.org/pdf/2609.30936)

> Abstract:Improving reasoning abilities in Large Language Models \(LLMs\) requires high\-quality data that exposes difficult decisions, competing alternatives, and their consequences\. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling\. We introduce Self\-Play Search Distillation \(SPSD\), a framework for generating superhuman synthetic data via self\-play of MuZero\-like networks trained on board games\. SPSD uses executable environments to turn search into structured reasoning problems\. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates\. By converting the self\-play search records into superhuman chains\-of\-thought, we train LLMs with environment\-grounded supervision\. Although trained only on self\-play search records, SPSD transfers to unseen mathematics\. On Qwen3\-4B\-Base, it raises the mean over six mathematics benchmarks from 24\.1 to 36\.6 while increasing the held\-out\-game win rate from 15% to 45%\. SPSD offers an annotation\-efficient way to create high\-quality synthetic data for improving LLM performance in reasoning tasks\.

## Submission history

From: Lorenzo Molfetta \[[view email](https://arxiv.org/show-email/eb362dbf/2609.30936)\] **\[v1\]**Fri, 25 Sep 2026 07:53:21 UTC \(2,137 KB\)

Similar Articles

Self-Play Search Distillation for Large Language Model Reasoning

Hugging Face Daily Papers

This paper introduces Self-Play Search Distillation (SPSD), a framework that uses self-play in board games to generate synthetic data for improving large language model reasoning, with demonstrated improvements on mathematical benchmarks.

MindZero: Learning Online Mental Reasoning With Zero Annotations

arXiv cs.AI

MindZero introduces a self-supervised reinforcement learning framework that trains multimodal large language models for efficient and robust online mental reasoning without requiring mental state annotations, outperforming model-based methods in accuracy and efficiency.

Google's Self-Improvements in LLMs (18 minute read)

TLDR AI

Google researchers propose a data-free on-policy self-distillation framework where LLMs synthesize their own curriculum of queries and privileged reasoning traces from coarse topic descriptions, achieving self-improvement across text and multimodal benchmarks without external supervision.