Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalization

Hugging Face Daily Papers Papers
reinforcement-learning large-language-models reasoning recursive-composition verifiable-environments llm-training scalability

Summary

The paper introduces RACES, a recursive automated composition framework that treats verifiable environments as composable building blocks to scale reinforcement learning for LLMs, enabling efficient reasoning generalization through compositional operators.

Reinforcement Learning (RL) with verifiable environments has emerged as a powerful approach for enhancing the reasoning capabilities of Large Language Models (LLMs). While prior research demonstrates that scaling environment quantity improves RL performance, existing manual or individual construction methods suffer from linear scaling limits, thereby hindering scalable reasoning generalization. This paper introduces RACES (Recursive Automated Composition for Environment Scaling), a framework that conceptualizes verifiable environments as composable building blocks that can be recursively assembled. The key insight is that when the codomain (output type) of one environment matches the domain (input type) of another, they can be automatically fused into a new verifiable environment, enabling recursive composition. RACES is implemented with 300 individual environments and defines a set of composition operators (SEQUENTIAL, PARALLEL, SORT, and SELECT) that induce diverse reasoning patterns. Extensive experiments show that RL training on these composite environments consistently enhances reasoning generalization. Specifically, RACES improves DeepSeek-R1-Distill-Qwen-14B by an average of 3.1 points (from 48.2 to 51.3) and boosts Qwen3-14B performance from 58.8 to 61.1 on six benchmarks, which are unseen during the construction of training environments. Moreover, RACES achieves performance comparable to training on 300 individual environments using only 50 base environments, demonstrating significant efficiency in environment utilization.
Original Article
View Cached Full Text

Cached at: 06/11/26, 01:39 PM

Paper page - Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalization

Source: https://huggingface.co/papers/2606.12373 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Recursive automated composition framework enables scalable reinforcement learning for language models by automatically combining verifiable environments through compositional operators.

Reinforcement Learning(RL) withverifiable environmentshas emerged as a powerful approach for enhancing the reasoning capabilities ofLarge Language Models(LLMs). While prior research demonstrates that scaling environment quantity improves RL performance, existing manual or individual construction methods suffer from linear scaling limits, thereby hindering scalable reasoning generalization. This paper introduces RACES (Recursive Automated Composition for Environment Scaling), a framework that conceptualizesverifiable environmentsas composable building blocks that can be recursively assembled. The key insight is that when the codomain (output type) of one environment matches the domain (input type) of another, they can be automatically fused into a new verifiable environment, enablingrecursive composition. RACES is implemented with 300 individual environments and defines a set ofcomposition operators(SEQUENTIAL,PARALLEL,SORT, andSELECT) that induce diverse reasoning patterns. Extensive experiments show that RL training on these composite environments consistently enhances reasoning generalization. Specifically, RACES improves DeepSeek-R1-Distill-Qwen-14B by an average of 3.1 points (from 48.2 to 51.3) and boosts Qwen3-14B performance from 58.8 to 61.1 on six benchmarks, which are unseen during the construction of training environments. Moreover, RACES achieves performance comparable to training on 300 individual environments using only 50 base environments, demonstrating significant efficiency in environment utilization.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.12373 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.12373 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.12373 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Learning to Reason with Curriculum II: Compositional Generalization

arXiv cs.LG

This paper theoretically analyzes how curriculum learning, by decomposing complex problems into simpler sub-problems and composing solutions, can dramatically reduce the sample complexity of learning to simulate sequential computations (semiautomata) compared to direct methods, achieving subpolynomial supervision requirements in supervised fine-tuning and exponentially weaker coverage conditions in reinforcement learning with verifiable rewards.

RL Post-Training Builds Compositional Reasoning Strategies

arXiv cs.CL

This paper investigates whether reinforcement learning post-training can compose primitive skills into higher-level compositional strategies, using a fully observable rewrite-grammar environment. The authors find that RL reorganizes primitive competence through phased compositional mechanisms, while rejection fine-tuning plateaus due to producing many invalid shortcut-like rewrites.