Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Summary
This paper proposes Feedback-Enriched Environments (FEEs) to bootstrap self-evolving agents in long-horizon tasks by enriching feedback for reinforcement learning, showing improved stability and performance on benchmarks using Qwen3 models and RL algorithms.
View Cached Full Text
Cached at: 09/09/26, 04:29 AM
Paper page - Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Source: https://huggingface.co/papers/2609.08404
Abstract
Feedback-Enriched Environments adapt task settings to provide observation-level guidance, improving reinforcement learning stability and exploration for long-horizon agent tasks.
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents throughReinforcement Learning(RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional agent-side warming up viasupervised fine-tuning(SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to environment-side adaptation by constructingFeedback-Enriched Environments(FEEs). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such asGRPO,GSPO, andDAPOdemonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs (1) stabilizes training dynamics by reducingentropy volatility, (2) facilitates proactivestate-space explorationin difficult tasks, (3) ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and (4) identifies intra-group feedback consistency as a critical boundary for stable optimization.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2609\.08404
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.08404 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.08404 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.08404 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis
This paper proposes EvoEnv, a method where language models construct verifiable Python environments for self-improvement through reinforcement learning, achieving a 3.3% relative gain on Qwen3-4B-Thinking.
EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
EnvFactory automates the creation of executable tool environments and natural multi-turn trajectories for training LLMs with agentic reinforcement learning, achieving superior performance on benchmarks like BFCLv3 and MCP-Atlas with fewer environments than prior work.
Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning
This paper introduces EFCA, a multi-timescale credit assignment method for agentic reinforcement learning that uses short-term feedback and medium-term state-history signals from environment interaction to improve task success and quality on ALFWorld and WebShop.
Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL
Envs-FORGE is a prompting policy for synthesizing training environments for reinforcement learning agents, converting verifier rewards into per-seed actions to improve performance on benchmarks like SWE-bench and Terminal-Bench.
From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning
This paper proposes the LLM-as-Environment-Engineer framework, where a policy model analyzes failures to automatically redesign the training environment for reinforcement learning, and introduces MAPF-FrozenLake as a controllable testbed. The framework, using Qwen3-4B, outperforms larger models like GPT and Gemini, showing that policy learning improves the model's ability to diagnose weaknesses.