Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
Summary
This paper introduces VHD-Play, a pipeline that generates diverse agentic reinforcement learning environments by first solving mathematical models, significantly improving training for language-model agents like Qwen3.6-35B-A3B at low cost and extending to external benchmarks.
View Cached Full Text
Cached at: 09/24/26, 03:38 AM
Paper page - Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
Source: https://huggingface.co/papers/2609.27321
Abstract
Language-modelagentsincreasinglyfacelong-horizontaskswithevolvingstate,interdependentdecisions,anddelayedoutcomes.Scalingtheirtrainingrequiresdiverseagenticenvironments,dependableoutcomesignals,andlowextensioncost.Existinggenerationpipelinescommonlyconstructanenvironmentbeforedefiningitsoutcomeruleorannotatingitstrajectories,leavingdynamicsandevaluationtobealignedposthoc.VHD-Playreversesthisdependencybysamplingandsolvingamathematicalmodelbeforeacorpus-groundedsetterrendersitsdecisionprocessasstatefultools.Theexecutabledynamicsandtrajectory-scoringreferenceareinheritedfromthesamesolvedmodel.Thepipelineproduces3,300diverseagenticenvironmentsatacostofafewcentseach.TrainingQwen3.6-35B-A3Bonthreefamiliesraisesitsmeanagenticscorefrom0.204to0.815inafive-familydiagnostic.Gainsalsoappearonheld-outinstancesfromallthreetrainingfamiliesandeightunseenmechanismfamilies,thenextendbeyondthegeneratedsubstratetoexternalbenchmarksforgeneralfunctioncalling,travelplanning,and365-daye-commerce.OnE-CommerceBench,thetrainedcheckpointcompleteseveryrunwithoutbankruptcyandexceedsQwen3.7-Max.Wecomparewritten-outproblemswithstatefulversionsthatrevealorhidetheirparameters.Thecomparisonshowsthatmostofthelearnablegapliesinstatefulinteractionratherthanunderlyingproblemsolving.Afrozen35Bsetterrealizeslargerenvironments,andscale-matchedtrainingretainsgainsasmechanismsizeandhorizongrow,indicatingthepotentialforanevolvingtrainingsubstrate.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.27321
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.27321 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.27321 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.27321 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models
This paper proposes using game engines as a verifiable trajectory data engine for scaling world models, introducing RLHEV to combine dense engine signals with human feedback for reinforcement learning post-training.
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
AgentV-RL introduces an Agentic Verifier framework that enhances reward modeling through bidirectional verification with forward and backward agents augmented with tools, achieving 25.2% improvement over state-of-the-art ORMs. The approach addresses error propagation and grounding issues in verifiers for complex reasoning tasks through multi-turn deliberative processes combined with reinforcement learning.
Beyond Static Evaluation: Building Simulation Environments for Scalable Agentic Reinforcement Learning
Introduces AgenticAI-Supervisor, an API and UI-driven simulation environment for scalable reinforcement learning of LLM agents, using verifiable execution outcomes and reward shaping to mitigate reward hacking.
Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis
This paper proposes EvoEnv, a method where language models construct verifiable Python environments for self-improvement through reinforcement learning, achieving a 3.3% relative gain on Qwen3-4B-Thinking.
@dair_ai: Highly-recommended read from MIT on the part of RL with verifiable rewards that everyone keeps hitting. RLVR only optim…
This paper from MIT proposes an adversarial generator-discriminator framework that combines verifiable rewards with a learned signal from human demonstrations to address issues like diversity collapse, unnatural responses, and reward hacking in RLVR training of language models.