SPADE: Self-Play in Adaptive Synthetic Executable Environments

Hugging Face Daily Papers Papers
self-play reinforcement-learning language-model synthetic-environments reasoning tool-use

Summary

SPADE introduces a self-play reinforcement learning framework for language models that generates adaptive executable training environments to enhance reasoning and tool-use capabilities, demonstrating significant performance gains across multiple benchmarks.

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.
Original Article
View Cached Full Text

Cached at: 08/20/26, 04:03 AM

Paper page - SPADE: Self-Play in Adaptive Synthetic Executable Environments

Source: https://huggingface.co/papers/2608.19197 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

SPADE is a self-play reinforcement learning framework where a language model designs adaptive executable training environments and learns to solve them, improving reasoning and tool-use performance through regret-based environment targeting.

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduceSPADE(Self-Play in Adaptive Synthetic Executable Environments), aself-play RLframework in which a singleLLMplays two roles: anEnvironment Designerthat writes complete, long-horizon training environments as executable code with anOpenAI Gym-stylereset()/step()interface, and aReasoning Agentthat learns to act in them. Each is a stateful, multi-turn environment (state transitions,reward functions, andverification code), so one interface spans reasoning problems and multi-step agentic tool use. TheReasoning Agent’sregretis estimated using the gap between its reward with and withoutprivileged hints; in optimizing thisregretsignal theEnvironment Designerlearns to target environments at the edge of the agent’s capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding theEnvironment Designeron documents sampled from a large pretraining corpus, and giving it anaccumulated environment memory. Scaling to 30B-parameter models,SPADEimproves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component,SPADEtakes a concrete step toward open-ended self-improvement.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.19197

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.19197 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.19197 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.19197 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Self-Programmed Execution for Language-Model Agents

arXiv cs.AI

This paper introduces Self-Programmed Execution (SPE), an agent architecture where the language model generates its own orchestration program rather than relying on a fixed external harness. It presents 'Spell', a Lisp-based language enabling this self-editing and re-evaluation, demonstrating that frontier models can successfully perform agentic tasks using this method.