EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Hugging Face Daily Papers Papers
agentic-rl world-model reinforcement-learning llm-agents world-rehearsal

Summary

EnvACE introduces world rehearsal, an agentic reinforcement learning method that replaces external environment interaction by having the policy rehearse environment responses internally, achieving strong performance across multiple benchmarks.

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:55 AM

Paper page - EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Source: https://huggingface.co/papers/2608.06197 Published on Aug 6

·

Submitted byhttps://huggingface.co/xuzishan

3on Aug 7

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Traininglargelanguagemodelagentsforlong-horizontoolusetypicallyreliesoninteractionswithrealorsynthesizedexecutableenvironments,whoseconstructionandverificationarecostly,oronexternalsimulatorsthataredifficulttoground.WeintroduceEnvACE,anagenticreinforcementlearningmethodthatreplacesexternalenvironmentinteractionduringtrainingwithworldrehearsal.Thepolicyalternatesbetweenactingandrehearsal:itfirstgeneratesatoolcall,thenplaystheroleoftheenvironmenttoproducetheresponseinducedbythataction,andconditionssubsequentdecisionsontherehearsedresponse.Bothrolesarejointlyoptimizedend-to-endusingtask-successrewards.Throughworldrehearsal,thepolicyinternalizestherelationshipbetweenactionsandtheirenvironmentresponsesinitsparameters,yieldinganagentworldmodelthatdirectlysupportsdecisionmaking.AcrossBFCL-v4,tau^2-Bench,VitaBench,andFinMCP-Bench,EnvACEachievesstrongandtransferableperformance,outperformingenvironment-scalingbaselinesintheoverallevaluation.Controlledstudiesfurthershowthatworldrehearsalconsistentlyimprovespolicylearningacrossmodelscales.Attesttime,theinternalizedworldmodelenablesprivaterehearsalbeforecommittedexecution,yieldingfurthergainsunderamoderaterehearsalbudgetwithoutadditionalexternalinteraction.OurfindingsestablishworldrehearsalasanewpathtowardscalingLLMagenttrainingbeyondtheconstraintsofexternalenvironments.Ourcodeispubliclyavailableathttps://github.com/Within-yao/EnvACE.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2608\.06197

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.06197 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.06197 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.06197 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

Hugging Face Daily Papers

This paper introduces AEM, a supervision-free method for agentic reinforcement learning that adapts entropy dynamics at the response level to improve exploration-exploitation trade-offs. It demonstrates performance gains on benchmarks like ALFWorld and SWE-bench by aligning uncertainty estimation with action granularity.

Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents

Hugging Face Blog

Huggingface introduces EcomRLVE-GYM, a framework providing eight verifiable environments for training reinforcement learning agents on complex e-commerce tasks. The tool features adaptive difficulty curricula and algorithmic rewards to improve task completion in shopping assistants, demonstrated by training a Qwen 3 8B model.