ECHO: Terminal Agents Learn World Models for Free

Hugging Face Daily Papers Papers
reinforcement-learning cli-agents world-models policy-gradient self-improvement

Summary

ECHO introduces a hybrid objective that combines policy-gradient loss with environment observation prediction to provide dense supervision from terminal feedback, doubling performance on TerminalBench-2.0 for Qwen3 models.

CLI agents are the closest thing language models have to an embodied setting: the model emits commands, the terminal executes them, and the returned stream -- stdout, errors, files, logs, and traces -- records the consequences. We argue that this stream is a supervision signal, but standard agent RL discards it: GRPO-style training updates action tokens with sparse outcome-level rewards while ignoring environment responses already in the rollout. Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds. We introduce ECHO (Environment Cross-entropy Hybrid Objective), a hybrid objective that combines the standard policy-gradient loss on action tokens with an auxiliary loss that trains the policy to predict environment observation tokens resulting from its own actions. ECHO reuses the same forward pass as GRPO, requires no additional rollouts, and turns terminal feedback into dense supervision for all rollouts. ECHO doubles GRPO pass@1 on TerminalBench-2.0: Qwen3-8B improves from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%. ECHO also produces policies that better predict terminal dynamics, even on trajectories they did not generate: across held-out rollouts, it sharply reduces environment-token cross-entropy while GRPO alone barely changes it. From base Qwen3-8B, ECHO matches expert-SFT-then-GRPO performance on held-out terminal tasks without expert demonstrations, and recovers roughly half of the expert-SFT initialization benefit on TerminalBench-2.0. In some settings, the environment prediction loss alone enables verifier-free self-improvement, allowing policies to improve on unseen OOD tasks by learning only from environment interactions. Together, these results suggest that environment observations are not merely context for future actions, but a dense, on-policy supervision signal already present in every rollout.
Original Article
View Cached Full Text

Cached at: 05/26/26, 06:45 PM

Paper page - ECHO: Terminal Agents Learn World Models for Free

Source: https://huggingface.co/papers/2605.24517

Abstract

Environment cross-entropy hybrid objective combines policy-gradient loss with auxiliary environment observation prediction to provide dense supervision from terminal feedback, improving agent performance and self-improvement capabilities.

CLI agentsare the closest thinglanguage modelshave to an embodied setting: the model emits commands, the terminal executes them, and the returned stream -- stdout, errors, files, logs, and traces -- records the consequences. We argue that this stream is a supervision signal, but standard agent RL discards it:GRPO-style training updatesaction tokenswith sparse outcome-level rewards while ignoringenvironment responsesalready in the rollout. Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds. We introduceECHO(Environment Cross-entropy Hybrid Objective), a hybrid objective that combines the standardpolicy-gradient lossonaction tokenswith an auxiliary loss that trains the policy to predictenvironment observation tokensresulting from its own actions.ECHOreuses the same forward pass asGRPO, requires no additional rollouts, and turnsterminal feedbackintodense supervisionfor all rollouts.ECHOdoublesGRPOpass@1 on TerminalBench-2.0: Qwen3-8B improves from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%.ECHOalso produces policies that better predict terminal dynamics, even on trajectories they did not generate: across held-out rollouts, it sharply reduces environment-token cross-entropy whileGRPOalone barely changes it. From base Qwen3-8B,ECHOmatches expert-SFT-then-GRPOperformance on held-out terminal tasks without expert demonstrations, and recovers roughly half of the expert-SFT initialization benefit on TerminalBench-2.0. In some settings, theenvironment prediction lossalone enables verifier-freeself-improvement, allowing policies to improve on unseen OOD tasks by learning only from environment interactions. Together, these results suggest that environment observations are not merely context for future actions, but a dense,on-policy supervisionsignal already present in every rollout.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.24517

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.24517 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.24517 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.24517 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Terminal-World: Scaling Terminal-Agent Environments via Agent Skills

arXiv cs.CL

Terminal-World introduces a fully automated pipeline that uses agent skills to synthesize high-quality training data for terminal agents, enabling models to outperform baselines with only 1.2% of the training data. The method co-derives task instructions, environments, and teacher trajectories from skill primitives.

@MSFTResearch: Computer-use AI agents struggle with multi-step workflows like email and customer support. Echoverse trains agents in r…

X AI KOLs Timeline

Microsoft Research introduces Echoverse, a set of deep, evolving environments for training computer-use agents. A 9B model trained on these environments nearly doubles its baseline score, coming within 14 points of GPT-5.4, demonstrating that high-fidelity simulation and co-evolution of model, world, and verifier significantly improve agent performance on multi-step workflows.