EmbodiedCLBench：评估自我进化具身智能体的持续学习能力

# EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents
Source: [https://arxiv.org/html/2610.13021](https://arxiv.org/html/2610.13021)
Jie HuangAffiliation:School of Intelligence Science and Technology, Peking UniversityAffiliation:State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, ChinaRuixun LiuAffiliation:School of Intelligence Science and Technology, Peking UniversityKaichen HeAffiliation:School of Intelligence Science and Technology, Peking UniversityXinyi HeAffiliation:School of Intelligence Science and Technology, Peking UniversityRui HuangAffiliation:University of Hong KongZilong ZhengAffiliation:State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, ChinaZhenliang ZhangAffiliation:State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, ChinaYiwu Zhong⋆Equal contribution†Corresponding authorAffiliation:School of Intelligence Science and Technology, Peking UniversityAffiliation:State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

###### Abstract

Continual learning requires models to accumulate past experience and transfer it to new tasks\. Despite substantial research on continual learning, existing work remains largely confined to non\-embodied settings, without extending to embodied environments\. To address this gap, we introduceEmbodiedCLBench, the first benchmark in embodied simulation for evaluating the continual learning capability of self\-evolving agents\. With compositional generalization as the core evaluation principle, our benchmark evaluates whether agents can first learn from basic tasks and then recombine the acquired skills to solve complex, unseen advanced tasks\. Based onEmbodiedCLBench, we conduct extensive experiments across multiple agent harnesses and learning methods, revealing consistent limitations and distinctive behaviors\. Our results show that while advanced tasks can be hardly solved under zero\-shot setting, the experience from basic tasks enables consistent improvement\. Such benefit from the experience remains stable even if the task complexity increases\. However, this improvement does not scale accordingly with the number of learning samples, and current agents cannot extract meaningful information from additional experience\. Collectively, our findings highlight continual learning as a fundamental yet underexplored capability, and our benchmark offers valuable resources for designing effective self\-evolving agents in embodied environments\. Our code and dataset are available at[https://pku\-value\-lab\.github\.io/EmbodiedCLBench\-homepage](https://pku-value-lab.github.io/EmbodiedCLBench-homepage)\.

## 1Introduction

Self\-evolving agents\([Gao et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib2);[Fang et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib3)\)can accumulate experience from past interactions and progressively enhance their task performance without additional human supervision\. Such agents are capable of retaining and transferring skills across tasks over time, a capability broadly termed*continual learning*\([Zheng et al\., 2025b](https://arxiv.org/html/2610.13021#bib.bib4)\)\. This capability is particularly important for embodied intelligence\([Liu et al\., 2025a](https://arxiv.org/html/2610.13021#bib.bib14);[Yifan et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib15);[Liu et al\., 2025b](https://arxiv.org/html/2610.13021#bib.bib16)\), where agents are expected to operate in complex, ever\-changing physical environments that cannot be fully anticipated at training time\. It is therefore important to develop a principled approach for evaluating the capability to learn from prior interactions and transfers the acquired skills to new tasks\.

Existing benchmarks in continual learning and embodied intelligence partially address this evaluation goal\. Research on continual learning has largely focused on non\-embodied settings, including knowledge\-intensive tasks\([Gao et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib5)\), interactive digital environments\([Jiang et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib8)\), and long\-term simulated worlds\([Cai et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib10)\)\. These platforms do not capture the spatial complexity and physical constraints inherent in embodied tasks\. Meanwhile, embodied benchmarks evaluate capabilities such as scene perception\([Wang et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib17)\), visual navigation\([Anderson et al\., 2018](https://arxiv.org/html/2610.13021#bib.bib21)\), robotic manipulation\([Gu et al\., 2023](https://arxiv.org/html/2610.13021#bib.bib28)\), and long\-horizon task planning\([Shridhar et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib31)\), but are not specifically designed to assess continual learning\. Therefore, there remains a lack of a dedicated benchmark for evaluating whether self\-evolving agents can accumulate experience and transfer it to solve novel tasks in embodied environments\.

To address this gap, we introduceEmbodiedCLBench, the first benchmark in embodied simulation for evaluating the continual learning capability of self\-evolving agents\. As illustrated in Figure[1](https://arxiv.org/html/2610.13021#S1.F1), our design stems from compositional generalization\([Furuta et al\., 2023](https://arxiv.org/html/2610.13021#bib.bib41);[Guo et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib40)\)\. Instead of learning every new task from scratch, a self\-evolving agent should recombine previously acquired skill elements to solve novel, unseen tasks\. We thus consider compositional generalization as a principled way to measure agent continual learning capability\. Specifically, we first assign a set of basic tasks for the agent to execute and learn the skills\. We then test whether the agent can effectively leverage such experience to solve unseen advanced tasks that require recombining the learned skills\. From this perspective, basic tasks serve as test\-time learning\([Sun et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib55);[Liu et al\., 2021](https://arxiv.org/html/2610.13021#bib.bib56)\)data for the agent to build experience before attempting advanced tasks\. To this end,EmbodiedCLBenchorganizes tasks into three compositional levels: \(1\) atomic actions from TongSim\([Sun et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib46)\), representing basic object–action interactions; \(2\) basic tasks, with each composing a small number of atomic actions into practical household operations across multiple categories; and \(3\) advanced tasks, with each combining elements from multiple basic\-task categories\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/tongbench_self_evolution_intro.png)Figure 1:Compositional generalization is leveraged as a principled way to measure continual learning capability\.An agent that has experience from basic tasks should recombine previously acquired skills to solve novel, unseen advanced tasks, rather than learning each new task from scratch\.Based onEmbodiedCLBench, we conduct extensive evaluation across multiple agent harnesses \(e\.g\.,OpenClawandCodex\) and learning methods \(In\-Context Learning and Skill Learning\)\. Our results reveal several notable and, in some cases, unexpected findings\.First, while advanced tasks can hardly be solved in the zero\-shot setting, agents with experience from related basic tasks achieve substantially higher success rates\. This validates that experience from simpler tasks can be effectively transferred to unseen, harder tasks\.Second, the performance gain over zero\-shot agents remains at a stable level as task complexity increases\. This suggests that continual learning capability is not sensitive to task difficulty\.Third, the improvement from learning does not scale accordingly with the number of learning samples\. A single basic task significantly improves performance, and additional samples yield no consistent further improvement\.Fourth, past experience facilitates the decision behavior of agents by selecting more meaningful actions with fewer unexecutable ones\.

Our contributions are summarized as follows:

- •We introduceEmbodiedCLBench,the first benchmarkin embodied simulation for evaluating the continual learning capability of self\-evolving agents\.
- •Our key innovationlies in leveraging compositional generalization as a principled way to construct tasks, design evaluation protocols, and evaluate continual learning capability\.
- •We conduct acomprehensive evaluationacross multiple agent harnesses and learning methods, revealing limitations of current agents and providing insights for future design\.

## 2Related Work

#### Continual Learning Benchmarks\.

Evaluating continual learning capabilities has become an increasingly active topic for LLMs and agents\([Gao et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib2);[Fang et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib3);[Zheng et al\., 2025b](https://arxiv.org/html/2610.13021#bib.bib4)\)\. Existing benchmarks span several distinct domains\. In knowledge\-intensive tasks, prior work evaluates whether agents can effectively transfer procedural experience across coding, reasoning, and question\-answering tasks\([Gao et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib5);[Wu et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib6);[Shu et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib7)\)\. In interactive digital environments, benchmarks test continual learning capabilities in web automation, operating systems, and knowledge graphs\([Jiang et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib8);[Zheng et al\., 2025a](https://arxiv.org/html/2610.13021#bib.bib9)\)\. In long\-term simulated environments, benchmarks model persistent virtual worlds where past behaviors continuously shape future states and decisions\([Cai et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib10)\)\. Another major direction focuses on skill\-centric evolution, studying whether agents can abstract reusable skills from experience and transfer them across tasks\([Zhong et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib11);[Zhang et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib12);[Guan et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib13);[Lei et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib57)\)\. Despite this progress, these benchmarks primarily operate in digital or symbolic environments, leaving continual learning in embodied scenarios largely unexplored\.

#### Embodied AI Benchmarks\.

Embodied AI benchmarks evaluate the capabilities ranging from perception to long\-horizon task execution\([Liu et al\., 2025a](https://arxiv.org/html/2610.13021#bib.bib14);[Yifan et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib15);[Liu et al\., 2025b](https://arxiv.org/html/2610.13021#bib.bib16)\)\. At the perception level, benchmarks test 3D scene understanding\([Wang et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib17)\), spatial reasoning\([Du et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib18)\), physical world understanding\([Chow et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib19)\), and scene dynamics\([Wang et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib20)\)\. For navigation, the field has progressed from language\-conditioned navigation\([Anderson et al\., 2018](https://arxiv.org/html/2610.13021#bib.bib21);[Ku et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib22)\)to sound\-guided navigation[Liu et al\. \(2026\)](https://arxiv.org/html/2610.13021#bib.bib1), object grounding\([Qi et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib23)\), multi\-modal lifelong navigation\([Khanna et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib24)\), and active exploration for question answering\([Majumdar et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib25)\)\. In manipulation, benchmarks evaluate multi\-task generalization\([Yu et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib26);[James et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib27)\), object\-level skill transfer\([Gu et al\., 2023](https://arxiv.org/html/2610.13021#bib.bib28)\), lifelong knowledge transfer\([Liu et al\., 2023](https://arxiv.org/html/2610.13021#bib.bib29)\), and everyday task scalability\([Nasiriany et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib30)\)\. For long\-horizon planning, benchmarks require agents to follow compositional instructions\([Shridhar et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib31)\), interact through dialogue\([Padmakumar et al\., 2021](https://arxiv.org/html/2610.13021#bib.bib32)\), compose language\-conditioned skills\([Mees et al\., 2022](https://arxiv.org/html/2610.13021#bib.bib33)\), handle diverse household activities\([Li et al\., 2023](https://arxiv.org/html/2610.13021#bib.bib34)\), and adapt plans based on visual feedback\([Tupini et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib35)\)\. These benchmarks comprehensively cover perception, navigation, manipulation, and planning\. Recent works further explore general\-purpose embodied agent evaluation and realistic simulation environments\([Peng et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib58);[Ren et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib59)\)\. However, none is designed for evaluating the continual learning capability of skills in long\-horizon tasks\.

#### Compositional Generalization\.

Compositional generalization, the ability to recombine learned elements for solving novel problems, has been studied across multiple domains\. In digital environments, benchmarks test whether mobile and web agents can compose atomic operations into multi\-step tasks\([Guo et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib40);[Furuta et al\., 2023](https://arxiv.org/html/2610.13021#bib.bib41)\)\. In video understanding, benchmarks study whether models can recognize unseen action–object combinations from separately learned components\([Li et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib44);[Ahn et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib45)\)\. For reasoning tasks, benchmarks show that models proficient in individual abilities still fail when these abilities must be combined\([Alazraki et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib42);[Li et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib43)\)\. In embodied manipulation, benchmarks evaluate whether robot policies can generalize to unseen skill compositions after training on atomic skills\([Wu et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib36);[Chen et al\., 2025a](https://arxiv.org/html/2610.13021#bib.bib37);[Chen et al\., 2025b](https://arxiv.org/html/2610.13021#bib.bib38);[Haresh et al\., 2024](https://arxiv.org/html/2610.13021#bib.bib39)\)\. In contrast,EmbodiedCLBenchevaluates the continual learning capability in embodied simulation without any benchmark\-specific training\.

## 3Benchmark Construction

As illustrated in Figure[2](https://arxiv.org/html/2610.13021#S3.F2), the construction ofEmbodiedCLBenchconsists of four stages: \(1\) environment setup, \(2\) task generation, \(3\) graph construction, and \(4\) evaluation protocol\. We introduce each stage in the following subsections\.

### 3\.1Environment Setup: Embodied Simulation

Evaluating compositional generalization requires a large combinatorial space of object–action pairs\. We thus adopt TongSim as the simulation platform\([Sun et al\., 2025](https://arxiv.org/html/2610.13021#bib.bib46)\)\. It provides a controllable environment that supports rich physical interactions and scales to diverse task configurations\. It was also used in recent embodied agent evaluation studies\([He et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib47);[Zhao et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib48)\)\.

Specifically, TongSim is a simulation with household scenarios and supports a wide range of interactiveobjects\(such as cups, plates, cabinets, lamps, and appliances\) and executableactions\(such as walk to, look at, pick up, put down, open, close, and switch on, etc\.\)\. This simulation offers three key advantages: \(1\) the platform supports a large number of object types and action primitives, providing sufficient combinatorial diversity; \(2\) all scenes are indoor domestic environments, so that the tasks are grounded in meaningful, real\-world household activities; and \(3\) TongSim has not been widely used as training data for existing foundation models, making it suitable for evaluating the generalization capability of agents with reduced risk of data contamination\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/tongbench_pipeline_overview_lrx_refine.png)Figure 2:Overview of the construction process ofEmbodiedCLBench\.During task generation, basic tasks connect a few atomic actions into simple household operations, and advanced tasks combine elements from multiple basic tasks into compositional tasks\. For each task, a state\-transition graph is built, with each graph node as an intermediate state and paired with action options and a first\-person\-view image\. During evaluation, the agent observes the current state and selects from candidate actions\. After each selection, the agent transitions to the next node on the graph\.
### 3\.2Task Generation: Compositional Generalization

Our goal is to evaluate the capability of continual learning\. However, it is an abstract concept that is challenging to measure directly\. We operationalize it through compositional generalization: if an agent has learned from basic tasks, it should recombine previously mastered skill elements to solve novel tasks rather than learning each new task from scratch\. This leads to a natural evaluation protocol—let the agent first complete simpler tasks to accumulate experience, then test whether it can leverage that experience to solve more complex, unseen tasks\.

We design two levels of tasks based on atomic actions available in TongSim\. Specifically, anatomic actionis a single interaction with an object \(e\.g\., “pick up the cup”\)\. It consists of a sequence ofsteps\(e\.g\., walking to the cup, looking at it, and picking it up\), each of which is one executable action\. Building on it, abasic taskcomposes a small number of atomic actions into a practical household operation \(e\.g\., “put the cup on the table”, which consists of “pick up the cup” and “put down the cup on the table”\)\. Further, anadvanced taskcomposes more atomic actions than a basic task \(e\.g\., “tidy up the room”, which requires picking up the book, putting the book on the shelf, closing the cabinet, and switching off the lamp\)\. We provide more details in the benchmark overview \(Section[4\.1](https://arxiv.org/html/2610.13021#S4.SS1)\)\. All task descriptions are generated by Qwen3\.7\-Plus and manually verified for correctness \(see Appendix[B](https://arxiv.org/html/2610.13021#A2)\)\.

### 3\.3Graph Construction: Action Trajectory Annotation

For TongSim simulation, we chose its closed\-source version, which provides more diverse actions than the open\-source version\. As a result, online simulation is not practical for the benchmark evaluation\. Therefore, we construct an offline, reproducible evaluation process in the form of a state\-transition graph, as shown in Figure[2](https://arxiv.org/html/2610.13021#S3.F2)\. A graph structure naturally represents the task procedure: at each state, an agent selects an action and transitions to the next state\. This graph structure offers three advantages: \(1\) offline evaluation can be run in parallel without running the simulator; \(2\) flexible control over the task complexity through graph branching and depth; and \(3\) fine\-grained analysis of agent behavior by checking each decision against the pre\-computed graph\.

Specifically, each graph node represents a unique environment state \(e\.g\., “near the desk and looking at the lamp”\)\. It is paired with an image captured from TongSim and a set of action candidates presented to the agent during evaluation\. These candidates are generated by Qwen3\.7\-Plus and manually verified for correctness\. Each executable candidate corresponds to an edge that deterministically transitions to the next state\. Given a task, we first define its initial and goal states and then use breadth\-first search to find the shortest path between them\. Since each state supports only a bounded set of objects and actions, the reachable state space is compact enough for reliable graph search\. We construct the distractor paths by expanding alternative executable actions from nodes along the shortest path\. These paths are executable but deviate from the shortest path\. The detailed construction pipeline and task examples are provided in Appendices[B](https://arxiv.org/html/2610.13021#A2)and[C](https://arxiv.org/html/2610.13021#A3)\.

### 3\.4Evaluation Protocol

With the task graphs built, we define how agents are evaluated on them\. The evaluation consists of a learning phase and a testing phase\. In the learning phase, the agent completes basic tasks to accumulate experience\. In the testing phase, the agent attempts unseen advanced tasks with this experience\. In this setup, basic tasks serve as the memory\([Xu et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib60);[Huang et al\., 2026](https://arxiv.org/html/2610.13021#bib.bib61)\)and test\-time learning data\([Sun et al\., 2020](https://arxiv.org/html/2610.13021#bib.bib55);[Liu et al\., 2021](https://arxiv.org/html/2610.13021#bib.bib56)\)for the self\-evolving agents\.

For each task, the evaluation follows an interactive, multi\-turn format: \(1\) the agent receives a task description and an image of the initial state; \(2\) at each step, the agent selects one action from a set of action candidates; \(3\) based on the action decision, the agent transitions to the next state and receives a new image with updated objects and action candidates; \(4\) this repeats until the agent reaches the goal \(success\) or exhausts the step budget \(failure\)\. The action candidates at each step are designed into three types: optimal actions that follow the shortest path, suboptimal actions that are executable but deviate from the shortest path, and invalid actions that cannot be executed in the current state\. This interactive format preserves the sequential decision\-making structure of embodied tasks while enabling fully offline, deterministic, and reproducible evaluation\.

## 4Main Experiments

In this section, we first describe the experimental setup, including the benchmark overview, evaluation settings, agent configurations, and metrics\. We then report and discuss the main results\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/tongbench_benchmark_statistics.png)Figure 3:Benchmark statistics ofEmbodiedCLBench: environment configuration \(a\), task composition across six categories \(b\), and task complexity at two levels \(c\)\.### 4\.1Experimental Setup

#### Benchmark overview\.

Figure[3](https://arxiv.org/html/2610.13021#S4.F3)summarizes the overall statistics ofEmbodiedCLBench\. The environment covers diverse scenes, object categories, and atomic actions, providing a broad combinatorial space for task construction\. The benchmark contains 268 tasks: 168 basic tasks and 100 advanced tasks\. Basic tasks span six categories, with each grounded in a distinct type of daily household activity\. Advanced tasks require substantially more atomic actions and shortest\-path steps\. The atomic actions of each advanced task come from two to four categories of basic tasks\. Each atomic action consists of 3 steps\. Therefore, the shortest path has 6 steps for a basic task \(2 atomic actions\) and 12 steps for an advanced task \(4 atomic actions\)\. With this scale, real\-world grounding, and structured difficulty,EmbodiedCLBenchcan reveal meaningful differences in the continual learning capability of agents\. Examples of basic and advanced tasks are provided in Appendix[C](https://arxiv.org/html/2610.13021#A3)\.

#### Evaluation settings\.

We evaluate agents under three settings: \(1\)Zero\-shot:the agent performs basic and advanced tasks directly, without a learning phase\. \(2\)In\-Context Learning \(ICL\):in the learning phase, the agent completes two related basic tasks for each advanced task\. In the testing phase, the full interaction trajectories of these basic tasks are retained in context\. \(3\)Skill Learning:in the learning phase, the agent completes the same basic tasks as in ICL\. It then writes a concise skill summary of its experience in Markdown format\. In the testing phase, only this skill summary is provided in context, instead of the interaction trajectories\.

The step budget of each task is the length of its shortest path plus extra steps\. In the testing phase, we allow 3 extra steps as a small tolerance for suboptimal decisions\. In the learning phase, we allow 6 extra steps for more exploration\.

#### Agent configurations\.

We evaluate all tasks under each of four agent harnesses:OpenClaw\([OpenClaw, 2026](https://arxiv.org/html/2610.13021#bib.bib49)\),Claude Code\([Anthropic, 2026](https://arxiv.org/html/2610.13021#bib.bib50)\),Codex\([OpenAI, 2026b](https://arxiv.org/html/2610.13021#bib.bib51)\), andHermes\([Nous Research, 2026](https://arxiv.org/html/2610.13021#bib.bib52)\)\. We pair each harness with two backbone models through their official APIs: Qwen3\.8\-Flash\([Qwen, 2026](https://arxiv.org/html/2610.13021#bib.bib53)\)and GPT5\.6\-Terra\([OpenAI, 2026a](https://arxiv.org/html/2610.13021#bib.bib54)\)\.

Table 1:Main results onEmbodiedCLBench\.For In\-Context Learning and Skill Learning, agents first learn from two basic tasks and then attempt one advanced task\.
#### Metrics\.

We report Success Rate \(SR\), which measures whether an agent can complete a task within the allotted step budget:

SR=1N​∑i=1Nci,ci∈\{0,1\}​indicates whether task​i​is completed\.\\mathrm\{SR\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}c\_\{i\}\\,,\\quad c\_\{i\}\\in\\\{0,1\\\}\\text\{ indicates whether task \}i\\text\{ is completed\.\}\(1\)Success rate directly reflects continual learning capability—whether accumulated experience enables the agent to solve novel, compositionally harder tasks\.

### 4\.2Results

Table[1](https://arxiv.org/html/2610.13021#S4.T1)presents the main results\. Agents achieve a moderate zero\-shot success rate on basic tasks \(around 50%\), indicating that current agents can execute simple household tasks\. Zero\-shot success rate on advanced tasks drops to around 10%, validating that advanced tasks are genuinely difficult\. Both ICL and Skill Learning substantially improve advanced task success rate, raising it from around 10% to above 30%\. This improvement is consistent across all harnesses, suggesting that experience from basic tasks can effectively transfer to unseen advanced tasks\.

Finding 1: While advanced tasks are hardly solvable under zero\-shot setting, experience from basic tasks enables consistent improvement across different harnesses and models\.

#### Discussion\.

Between ICL and Skill Learning, there is no consistent advantage of one over the other\. We conjecture that both methods can extract meaningful information from the agent experience\. Besides, the performance across harnesses is close yet far from saturation, suggesting great potential for improvement\. Designing harnesses for continual learning can be a future direction\.

## 5Analysis

### 5\.1Effects of Compositional Complexity

#### Motivation and Setting\.

Compositional generalization is the core approach for evaluating continual learning inEmbodiedCLBench\. A natural question follows: does more complex composition make continual learning harder? In the main experiment, advanced tasks consist of 4 atomic actions\. Here, we extend our evaluation data to advanced tasks with 3 to 6 atomic actions to control compositional complexity\. We measure both the raw success rate at each level and the gain over zero shot, defined as the success rate of In\-Context Learning or Skill Learning minus the zero\-shot success rate\. We useHermeswith Qwen3\.8\-Flash for this analysis\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/experiment/L_N/L_N_complexity.png)Figure 4:Effect of compositional complexity on continual learning\.Thexx\-axis varies the number of atomic actions per advanced task from 3 to 6\. \(a\) reports the raw success rate achieved by each setting\. \(b\) reports the gain over zero shot, defined as the success rate of ICL or Skill Learning minus that of zero shot, isolating the improvement attributed to the learning phase\.
#### Results and Analysis\.

As shown in Figure[4](https://arxiv.org/html/2610.13021#S5.F4)\(a\), success rate decreases across all three settings as atomic actions increase from 3 to 6\. The largest drop occurs between 3 and 4 atomic actions \(e\.g\., ICL falls from 72\.0% to 42\.0%\), after which the success rate levels off\. In the zero\-shot setting, success rate decreases sharply \(e\.g\., from 28\.0% to 4\.0%\)\. In contrast, ICL and Skill Learning remain more stable in relative terms\. Specifically, Figure[4](https://arxiv.org/html/2610.13021#S5.F4)\(b\) shows that the gain over zero shot drops initially but then stabilizes at around 30%\. This indicates that experience from basic tasks remains helpful as compositional complexity increases\. At the highest complexity \(6 atomic actions\), zero\-shot success rate drops to 4\.0%, while ICL and Skill Learning still achieve 32\.0% and 26\.0%, respectively\. This shows that experience from basic tasks still brings a substantial gain over zero shot, even when zero\-shot agents can barely solve the tasks\.

Finding 2: In zero\-shot setting, success rate decreases sharply as complexity increases\. In contrast, the gain over zero\-shot setting remains stable, indicating that the benefit of continual learning is less sensitive to task difficulty\.

### 5\.2Scaling with the Number of Learning Samples

#### Motivation and Setting\.

Continual learning assumes that accumulated experience improves an agent’s performance on harder tasks\. If this holds, more experience should lead to a higher success rate on advanced tasks\. In the main experiment, agents learn from 2 basic tasks before attempting advanced tasks\. Here, we increase the number of learning samples \(i\.e\., basic tasks in the learning phase\) fromk=0k\{=\}0\(zero shot\) tok=6k\{=\}6\. We measure the success rate on the same set of advanced tasks under both In\-Context Learning and Skill Learning settings\. We evaluate two agent harnesses,HermesandOpenClaw, both using Qwen3\.8\-Flash as the backbone model\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/experiment/n_sample/n_sample_extra_steps_3.png)Figure 5:Success rate versus the number of learning samples\.Each learning sample is a basic task that the agent completes before being tested on advanced tasks\.Results and Analysis\.As shown in Figure[5](https://arxiv.org/html/2610.13021#S5.F5), the most substantial improvement occurs betweenk=0k\{=\}0andk=1k\{=\}1\. A single learning sample raises the success rate from 8\.0%–10\.0% to 28\.0%–40\.0% across both harnesses and learning methods\. This initial jump accounts for most of the total improvement across thekkrange\. Even minimal experience \(completing just one basic task\) provides sufficient new information for the agent to substantially improve on unseen advanced tasks\. Beyondk=1k\{=\}1, however, success rate does not continue to rise but instead fluctuates without a clear upward trend\. This pattern remains across ICL and Skill Learning, and across both harnesses\. None of the four curves shows consistent improvement with additional learning samples\. This suggests that current agents cannot extract sufficiently distinct information from additional learning samples\. Therefore, scalable continual learning, where success rate keeps improving as learning samples increase, remains an open challenge for embodied agents\.

Finding 3: Success rate does not scale accordingly with the number of learning samples\. A single learning sample significantly improves performance, while additional samples yield no consistent further gain\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/experiment/action_analysis/fig_action_analysis_delta.png)Figure 6:Error analysis of decision behavior\.Each group of bars shows the change in selection rate relative to the zero\-shot setting for three action types:*optimal actions*follow the shortest path,*suboptimal actions*are executable but deviate from the shortest path, and*invalid actions*cannot be executed in the current state\.

### 5\.3Error Analysis

#### Motivation and Setting\.

The main results show that ICL and Skill Learning improve success rate, but do not reveal what agents actually learn from experience\. As defined in Section[3\.4](https://arxiv.org/html/2610.13021#S3.SS4), the action candidates at each step fall into three types: optimal actions that follow the shortest path, suboptimal actions that are executable but deviate from the shortest path, and invalid actions that cannot be executed in the current state\. We analyze how agents choose among these three action types\. Instead of running new experiments, we use the selected actions recorded at each decision step in the main experiment \(all four harnesses with Qwen3\.8\-Flash\)\. Specifically, we compare the selection rate of each action type under ICL and Skill Learning with that in the zero\-shot setting\.

Results and Analysis\.As shown in Figure[6](https://arxiv.org/html/2610.13021#S5.F6), experience sharply reduces invalid actions across all harnesses \(by 13%–20%\)\. Fewer invalid actions allow agents to stay on executable trajectories and reach the goal within the step budget\. This suggests that the reduction in invalid actions contributes to the higher success rates in Table[1](https://arxiv.org/html/2610.13021#S4.T1)\. Meanwhile, optimal actions increase substantially\. Suboptimal actions also increase slightly \(by 3%–8%\)\. Overall, experience shifts the selections of agents from invalid actions to optimal and suboptimal actions\. ICL and Skill Learning show the same direction of change, but Skill Learning consistently produces smaller shifts across all harnesses\. In contrast, Table[1](https://arxiv.org/html/2610.13021#S4.T1)shows no consistent advantage between ICL and Skill Learning in success rate\. This may reflect a difference in how experience is represented\. Interaction trajectories may induce larger changes in decision behavior, while skill summaries may induce smaller but more focused changes\.

Finding 4: Continual learning alters the decision behavior of agents, resulting in an increase in optimal actions and a decrease in infeasible actions\.

### 5\.4Step Budget Sensitivity

#### Motivation and Setting\.

Each task has a shortest path to the goal, but agents often make suboptimal decisions along the way\. As described in Section[4\.1](https://arxiv.org/html/2610.13021#S4.SS1), agents are allowed 3 extra steps by default as a small tolerance for suboptimal decisions\. Here, we vary the number of extra steps from 0 to 6\. We examine how the step budget affects success rate under different evaluation settings\. We useHermeswith Qwen3\.8\-Flash for this analysis\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/experiment/step_budget/step_budget_sensitivity.png)Figure 7:Step budget sensitivity\.Thexx\-axis shows the number of extra steps allowed beyond the shortest path\. By default, agents are allowed 3 extra steps in the testing phase\.Results and Analysis\.As shown in Figure[7](https://arxiv.org/html/2610.13021#S5.F7), all three settings show a monotonically increasing success rate as the budget grows from 0 to 6 extra steps\. This is expected, since a larger budget gives agents more room for suboptimal decisions\. Across all budget values, the ranking of the three settings remains consistent\. ICL achieves the highest success rate, followed by Skill Learning and then zero shot\. This indicates that the gain over zero shot holds across different budgets\. At the strictest budget \(0 extra steps\), zero\-shot success rate is 0\.0%, while ICL and Skill Learning still achieve 15\.0% and 7\.0%, respectively\. Even under this strictest budget, learning from experience remains effective\. Notably, compared to the zero\-shot setting, ICL and Skill Learning exhibit a steeper increase in success rate as the budget grows\. This indicates that the agents with prior experience can better benefit from additional tolerance\. Meanwhile, the gap between experienced agents and zero\-shot agents remains substantial even at the most generous budget \(6 extra steps\)\. This suggests that additional tolerance cannot replace the role of prior experience\.

Finding 5: As the number of extra steps increases, success rate improves across all settings, and continual learning methods yield a steeper increase compared to the zero\-shot setting\.

## 6Conclusion

We introduceEmbodiedCLBench, the first embodied simulation benchmark for evaluating the continual learning capability of self\-evolving agents\. Our key idea is to formulate continual learning through compositional generalization: rather than solving each new task from scratch, an agent should accumulate experience from basic tasks and recombine previously acquired skills to solve unseen, more complex tasks\. Following this principle,EmbodiedCLBenchorganizes tasks into atomic actions, basic tasks, and advanced tasks, providing a structured protocol for measuring whether embodied agents can acquire, retain, and transfer experience over time\.

Extensive evaluations across multiple agent harnesses and learning methods reveal both promising progress and clear limitations: \(1\) Experience from basic tasks substantially improves performance on unseen advanced tasks over zero\-shot execution, showing that current agents can transfer previously acquired knowledge to new compositions\. \(2\) However, the performance of existing agents is far from saturation\. Designing effective continual learning methods for embodied agents is a promising direction to explore\. \(3\) Further, the benefit of learning does not consistently increase with the amount of experience\. How to enable the agents to accumulate and exploit additional interactions is still an open problem\. Overall,EmbodiedCLBenchprovides a principled testbed for studying self\-evolution in embodied agents and highlights the need for stronger mechanisms that can continuously consolidate, generalize, and reuse experience across tasks\.

## References

- Ahnet al\.\(2026\)G\. Ahn, I\. Lee, T\. Kim, M\. Shim, D\. Wee, and J\. ChoiWhy Can’t I Open My Drawer? Mitigating Object\-Driven Shortcuts in Zero\-Shot Compositional Action Recognition\.arXiv\(en\-US\)\.Note:arXiv:2601\.16211 \[cs\.CV\]Comment: Project page: https://ahngeo\.github\.io/assets/html/RCORE\.htmlExternal Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.16211)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Alazrakiet al\.\(2026\)L\. Alazraki, L\. Chen, A\. Brassard, J\. Stacey, H\. A\. Rahmani, and M\. ReiAgentcoma: a compositional benchmark mixing commonsense and mathematical reasoning in real\-world scenarios\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8383–8410\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Andersonet al\.\(2018\)P\. Anderson, Q\. Wu, D\. Teney, J\. Bruce, M\. Johnson, N\. Sünderhauf, I\. Reid, S\. Gould, and A\. van den HengelVision\-and\-Language Navigation: Interpreting Visually\-Grounded Navigation Instructions in Real Environments\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Anthropic \(2026\)AnthropicClaude Code\.External Links:[Link](https://claude.com/product/claude-code)Cited by:[§4\.1](https://arxiv.org/html/2610.13021#S4.SS1.SSS0.Px3.p1.1)\.
- Caiet al\.\(2025\)Y\. Cai, Y\. Hao, J\. Zhou, H\. Yan, Z\. Lei, R\. Zhen, Z\. Han, Y\. Yang, J\. Li, Q\. Pan, T\. Huai, Q\. Chen, X\. Li, K\. Chen, B\. Zhang, X\. Qiu, and L\. HeBuilding Self\-Evolving Agents via Experience\-Driven Lifelong Learning: A Framework and Benchmark\.arXiv preprint arXiv:2508\.19005\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2025a\)Y\. Chen, Z\. Chen, N\. T\. Chan, J\. Chen, J\. Yin, J\. Shi, Y\. Gao, Y\. Li, and J\. HuoRoboHiMan: A Hierarchical Evaluation Paradigm for Compositional Generalization in Long\-Horizon Manipulation\.arXiv preprint arXiv:2510\.13149\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2025b\)Z\. Chen, J\. Yin, Y\. Chen, J\. Huo, P\. Tian, J\. Shi, Y\. Hou, Y\. Li, and Y\. GaoDeCo: Task Decomposition and Skill Composition for Zero\-Shot Generalization in Long\-Horizon 3D Manipulation\.arXiv preprint arXiv:2505\.00527\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Chowet al\.\(2025\)W\. Chow, J\. Mao, B\. Li, D\. Seita, V\. Guizilini, and Y\. WangPhysBench: Benchmarking and Enhancing Vision\-Language Models for Physical World Understanding\.arXiv preprint arXiv:2501\.16411\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Duet al\.\(2024\)M\. Du, B\. Wu, Z\. Li, X\. Huang, and Z\. WeiEmbSpatial\-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision\-Language Models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 346–355\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-short.33)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Fanget al\.\(2025\)J\. Fang, Y\. Peng, X\. Zhang, Y\. Wang, X\. Yi, G\. Zhang, Y\. Xu, B\. Wu, S\. Liu, Z\. Li, Z\. Ren, N\. Aletras, X\. Wang, H\. Zhou, and Z\. MengA Comprehensive Survey of Self\-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems\.arXiv preprint arXiv:2508\.07407\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p1.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Furutaet al\.\(2023\)H\. Furuta, Y\. Matsuo, A\. Faust, and I\. GurExposing Limitations of Language Model Agents in Sequential\-Task Compositions on the Web\.arXiv preprint arXiv:2311\.18751\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p3.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2025\)H\. Gao, J\. Geng, W\. Hua, M\. Hu, X\. Juan, H\. Liu, S\. Liu, J\. Qiu, X\. Qi, Y\. Wu, H\. Wang, H\. Xiao, Y\. Zhou, S\. Zhang, J\. Zhang, J\. Xiang, Y\. Fang, Q\. Zhao, D\. Liu, Q\. Ren, C\. Qian, Z\. Wang, M\. Hu, H\. Wang, Q\. Wu, H\. Ji, and M\. WangA Survey of Self\-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence\.arXiv preprint arXiv:2507\.21046\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p1.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2026\)X\. Gao, C\. Hu, H\. Chen, P\. Yao, Z\. Wang, Y\. Bai, Z\. Wu, Y\. Han, X\. Cong, J\. Gui, Y\. Deng, and T\. LiEvoAgentBench: Benchmarking Agent Self\-Evolution via Ability Transfer\.arXiv preprint arXiv:2607\.05202\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2023\)J\. Gu, F\. Xiang, X\. Li, Z\. Ling, X\. Liu, T\. Mu, Y\. Tang, S\. Tao, X\. Wei, Y\. Yao, X\. Yuan, P\. Xie, Z\. Huang, R\. Chen, and H\. SuManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Guanet al\.\(2026\)T\. Guan, Y\. Wang, H\. Yang, S\. Cao, S\. Liu, Y\. Hu, J\. Li, and M\. ZhangContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?\.arXiv preprint arXiv:2608\.03874\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2025\)Y\. Guo, T\. Miao, Z\. Wu, P\. Cheng, M\. Zhou, and Z\. ZhangAtomic\-to\-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System\.arXiv preprint arXiv:2506\.08972\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p3.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Hareshet al\.\(2024\)S\. Haresh, D\. Dijkman, A\. Bhattacharyya, and R\. MemisevicClevrSkills: Compositional Language And Visual Reasoning in Robotics\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 38235–38266\.External Links:[Document](https://dx.doi.org/10.52202/079017-1209)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Heet al\.\(2026\)X\. He, Y\. Yang, C\. Fu, S\. Guo, S\. Zhu, L\. Fan, Z\. Zhang, and Y\. PengAutomatic Cognitive Task Generation for In\-Situ Evaluation of Embodied Agents\.arXiv preprint arXiv:2602\.05249\.Cited by:[§3\.1](https://arxiv.org/html/2610.13021#S3.SS1.p1.1)\.
- Huanget al\.\(2026\)J\. Huang, R\. Liu, S\. Sun, X\. Yang, Y\. Li, Y\. Zhu, and Y\. ZhongM3\{\}^\{3\}eval: multi\-modal memory evaluation through cognitively\-grounded video tasks\.arXiv preprint arXiv:2606\.05008\.Cited by:[§3\.4](https://arxiv.org/html/2610.13021#S3.SS4.p1.1)\.
- Jameset al\.\(2020\)S\. James, Z\. Ma, D\. R\. Arrojo, and A\. J\. DavisonRLBench: The Robot Learning Benchmark & Learning Environment\.IEEE Robotics and Automation Letters5\(2\),pp\. 3019–3026\.External Links:[Document](https://dx.doi.org/10.1109/lra.2020.2974707)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Jianget al\.\(2026\)S\. Jiang, L\. Ma, Z\. Hong, K\. Wang, Z\. Lu, T\. Wang, S\. Chen, J\. Zhang, T\. Pan, W\. Li, J\. Liang, and Y\. XiaoSEA\-Eval: A Benchmark for Evaluating Self\-Evolving Agents Beyond Episodic Assessment\.arXiv preprint arXiv:2604\.08988\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Khannaet al\.\(2024\)M\. Khanna, R\. Ramrakhya, G\. Chhablani, S\. Yenamandra, T\. Gervet, M\. Chang, Z\. Kira, D\. S\. Chaplot, D\. Batra, and R\. MottaghiGOAT\-Bench: A Benchmark for Multi\-Modal Lifelong Navigation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 16373–16383\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Kuet al\.\(2020\)A\. Ku, P\. Anderson, R\. Patel, E\. Ie, and J\. BaldridgeRoom\-Across\-Room: Multilingual Vision\-and\-Language Navigation with Dense Spatiotemporal Grounding\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 4392–4412\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.356)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Leiet al\.\(2026\)Y\. Lei, Z\. Wan, J\. Zhang, S\. Alam, Z\. Zhong, P\. Huang, X\. Wang, J\. Zhang, D\. Zhou, Y\. Hsieh,et al\.SkillEvolBench: benchmarking the evolution from episodic experience to procedural skills\.arXiv preprint arXiv:2605\.24117\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)C\. Li, R\. Zhang, J\. Wong, C\. Gokmen, S\. Srivastava, R\. Martín\-Martín, C\. Wang, G\. Levine, M\. Lingelbach, J\. Sun, M\. Anvari, M\. Hwang, M\. Sharma, A\. Aydin, D\. Bansal, S\. Hunter, K\. Kim, A\. Lou, C\. R\. Matthews, I\. Villa\-Renteria, J\. H\. Tang, C\. Tang, F\. Xia, S\. Savarese, H\. Gweon, K\. Liu, J\. Wu, and L\. Fei\-FeiBEHAVIOR\-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation\.InProceedings of The 6th Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.205,pp\. 80–93\.Note:Month: 14–18 DecCited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)R\. Li, Z\. Feng, T\. Xu, L\. Li, X\. Wu, M\. Awais, S\. Atito, and J\. KittlerC2C: Component\-to\-Composition Learning for Zero\-Shot Compositional Action Recognition\.arXiv preprint arXiv:2407\.06113\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2025\)T\. Li, J\. Zhang, Y\. Rao, and Y\. ChengUnveiling the Compositional Ability Gap in Vision\-Language Reasoning Model\.InAdvances in Neural Information Processing Systems,Vol\.38, Main Conference,pp\. 126574–126592\.External Links:[Document](https://dx.doi.org/10.52202/085713-4222)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2023\)B\. Liu, Y\. Zhu, C\. Gao, Y\. Feng, Q\. Liu, Y\. Zhu, and P\. StoneLIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning\.arXiv preprint arXiv:2306\.03310\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2025a\)H\. Liu, D\. Guo, and A\. CangelosiEmbodied intelligence: a synergy of morphology, action, perception and learning\.ACM Computing Surveys57\(7\),pp\. 1–36\.External Links:ISSN 1557\-7341,[Document](https://dx.doi.org/10.1145/3717059)Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p1.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)R\. Liu, Y\. Wang, J\. Xie, Y\. You, D\. Cai, J\. Lin, X\. Chen, Z\. Guo, Y\. Chu, Q\. Yang, X\. Cheng, J\. Xu, and Y\. ZhongOmniEcho: audio\-visual spatial understanding for omni\-modal embodied agents\.arXiv preprint arXiv:2609\.23407\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2025b\)Y\. Liu, W\. Chen, Y\. Bai, X\. Liang, G\. Li, W\. Gao, and L\. LinAligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI\.IEEE/ASME Transactions on Mechatronics\.External Links:[Document](https://dx.doi.org/10.1109/tmech.2025.3574943)Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p1.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2021\)Y\. Liu, P\. Kothari, B\. Van Delft, B\. Bellot\-Gurlet, T\. Mordan, and A\. AlahiTtt\+\+: when does self\-supervised test\-time training fail or thrive?\.Advances in Neural Information Processing Systems34,pp\. 21808–21820\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p3.1),[§3\.4](https://arxiv.org/html/2610.13021#S3.SS4.p1.1)\.
- Majumdaret al\.\(2024\)A\. Majumdar, A\. Ajay, X\. Zhang, P\. Putta, S\. Yenamandra, M\. Henaff, S\. Silwal, P\. Mcvay, O\. Maksymets, S\. Arnaud, K\. Yadav, Q\. Li, B\. Newman, M\. Sharma, V\. Berges, S\. Zhang, P\. Agrawal, Y\. Bisk, D\. Batra, M\. Kalakrishnan, F\. Meier, C\. Paxton, A\. Sax, and A\. RajeswaranOpenEQA: Embodied Question Answering in the Era of Foundation Models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 16488–16498\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Meeset al\.\(2022\)O\. Mees, L\. Hermann, E\. Rosete\-Beas, and W\. BurgardCALVIN: A Benchmark for Language\-Conditioned Policy Learning for Long\-Horizon Robot Manipulation Tasks\.IEEE Robotics and Automation Letters7\(3\),pp\. 7327–7334\.External Links:[Document](https://dx.doi.org/10.1109/lra.2022.3180108)Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Nasirianyet al\.\(2024\)S\. Nasiriany, A\. Maddukuri, L\. Zhang, A\. Parikh, A\. Lo, A\. Joshi, A\. Mandlekar, and Y\. ZhuRoboCasa: Large\-Scale Simulation of Everyday Tasks for Generalist Robots\.arXiv preprint arXiv:2406\.02523\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Nous Research \(2026\)Nous ResearchHermes Agent\.External Links:[Link](https://github.com/NousResearch/hermes-agent)Cited by:[§4\.1](https://arxiv.org/html/2610.13021#S4.SS1.SSS0.Px3.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.6 Terra Model \| OpenAI API\.External Links:[Link](https://developers.openai.com/api/docs/models/gpt-5.6-terra)Cited by:[§4\.1](https://arxiv.org/html/2610.13021#S4.SS1.SSS0.Px3.p1.1)\.
- OpenAI \(2026b\)OpenAIOpenAI Codex CLI \- Getting Started\.External Links:[Link](https://help.openai.com/en/articles/11096431)Cited by:[§4\.1](https://arxiv.org/html/2610.13021#S4.SS1.SSS0.Px3.p1.1)\.
- OpenClaw \(2026\)OpenClawOpenClaw — open\-source AI assistant\.External Links:[Link](https://openclaw.ai/)Cited by:[§4\.1](https://arxiv.org/html/2610.13021#S4.SS1.SSS0.Px3.p1.1)\.
- Padmakumaret al\.\(2021\)A\. Padmakumar, J\. Thomason, A\. Shrivastava, P\. Lange, A\. Narayan\-Chen, S\. Gella, R\. Piramuthu, G\. Tur, and D\. Hakkani\-TurTEACh: Task\-driven Embodied Agents that Chat\.arXiv preprint arXiv:2110\.00534\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Penget al\.\(2024\)Y\. Peng, J\. Han, Z\. Zhang, L\. Fan, T\. Liu, S\. Qi, X\. Feng, Y\. Ma, Y\. Wang, and S\. ZhuThe tong test: evaluating artificial general intelligence through dynamic embodied physical and social interactions\.Engineering34,pp\. 12–22\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Qiet al\.\(2020\)Y\. Qi, Q\. Wu, P\. Anderson, X\. Wang, W\. Y\. Wang, C\. Shen, and A\. v\. d\. HengelREVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Qwen \(2026\)QwenQwen3\.8\-Flash\.External Links:[Link](https://www.qwencloud.com/models/qwen3.8-flash)Cited by:[§4\.1](https://arxiv.org/html/2610.13021#S4.SS1.SSS0.Px3.p1.1)\.
- Renet al\.\(2025\)J\. Ren, Y\. Zhuang, X\. Ye, L\. Mao, X\. He, J\. Shen, M\. Dogra, Y\. Liang, R\. Zhang, T\. Yue,et al\.Simworld: an open\-ended realistic simulator for autonomous agents in physical and social worlds\.arXiv preprint arXiv:2512\.01078\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Shridharet al\.\(2020\)M\. Shridhar, J\. Thomason, D\. Gordon, Y\. Bisk, W\. Han, R\. Mottaghi, L\. Zettlemoyer, and D\. FoxALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Shuet al\.\(2026\)Y\. Shu, B\. J\. Gutiérrez, S\. P\. Jonnalagedda, Y\. Yao, H\. Sun, and Y\. SuAgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents\.arXiv preprint arXiv:2606\.02461\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Sunet al\.\(2020\)Y\. Sun, X\. Wang, Z\. Liu, J\. Miller, A\. Efros, and M\. HardtTest\-time training with self\-supervision for generalization under distribution shifts\.InInternational conference on machine learning,pp\. 9229–9248\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p3.1),[§3\.4](https://arxiv.org/html/2610.13021#S3.SS4.p1.1)\.
- Sunet al\.\(2025\)Z\. Sun, K\. Wu, C\. Fu, Z\. Song, L\. Shi, Z\. Xue, B\. Jing, Y\. Yang, X\. Gao, A\. Li, T\. Guo, H\. Li, X\. Yang, R\. Liu, X\. He, Y\. Wang, Y\. Li, M\. Liu, Y\. Lu, H\. Xie, S\. Zhao, B\. Dai, W\. Wang, T\. Yuan, S\. Zhu, Y\. Peng, and Z\. ZhangTongSIM: A General Platform for Simulating Intelligent Machines\.arXiv preprint arXiv:2512\.20206\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.13021#S3.SS1.p1.1)\.
- Tupiniet al\.\(2026\)A\. Tupini, L\. Liden, R\. Tan, Y\. Wang, and J\. GaoAsgardBench – Evaluating Visually Grounded Interactive Planning Under Minimal Feedback\.arXiv preprint arXiv:2603\.15888\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)T\. Wang, X\. Mao, C\. Zhu, R\. Xu, R\. Lyu, P\. Li, X\. Chen, W\. Zhang, K\. Chen, T\. Xue, X\. Liu, C\. Lu, D\. Lin, and J\. PangEmbodiedScan: A Holistic Multi\-Modal 3D Perception Suite Towards Embodied AI\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 19757–19767\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p2.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)W\. Wang, C\. Duan, Z\. Peng, Y\. Liu, and B\. ZhouEmbodied Scene Understanding for Vision Language Models via MetaVQA\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 22453–22464\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024\)C\. Wu, Z\. R\. Tam, C\. Lin, Y\. Chen, and H\. LeeStreamBench: Towards Benchmarking Continuous Improvement of Language Agents\.arXiv preprint arXiv:2406\.08747\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026\)Z\. Wu, B\. Wei, L\. Liu, Z\. He, X\. Wang, J\. Liu, Z\. Li, G\. Yao, J\. Zheng, X\. Yang, and Y\. WangATOM\-Bench: A Real\-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies\.arXiv preprint arXiv:2606\.16826\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px3.p1.1)\.
- Xuet al\.\(2026\)B\. Xu, X\. Dai, and K\. ZhangContextual agentic memory is a memo, not true memory\.arXiv preprint arXiv:2604\.27707\.Cited by:[§3\.4](https://arxiv.org/html/2610.13021#S3.SS4.p1.1)\.
- Yifanet al\.\(2025\)C\. Yifan, M\. Wei, X\. Wang, Y\. Liu, J\. Wang, H\. Song, L\. Ma, D\. Di, C\. Sun, K\. Liu, L\. Qi, J\. Yu, X\. Tian, S\. Liang, C\. Duan, Z\. Hong, W\. Zhang, and T\. LiuEmbodied AI: A Survey on the Evolution from Perceptive to Behavioral Intelligence\.SmartBot\.External Links:[Document](https://dx.doi.org/10.1002/smb2.70003)Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p1.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2020\)T\. Yu, D\. Quillen, Z\. He, R\. Julian, K\. Hausman, C\. Finn, and S\. LevineMeta\-World: A Benchmark and Evaluation for Multi\-Task and Meta Reinforcement Learning\.InProceedings of the Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.100,pp\. 1094–1100\.Note:Month: 30 Oct–01 NovCited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, K\. Shi, S\. Huang, A\. Nie, Y\. Zeng, Y\. Zhao, Z\. Fang, Q\. Su, H\. Qiu, W\. Yang, Q\. Ren, S\. Zou, W\. Huang, L\. Chen, Z\. Chen, and F\. ZhaoSkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents\.arXiv preprint arXiv:2604\.17308\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2026\)S\. Zhao, X\. Song, T\. Guo, X\. Gao, M\. Liu, X\. Han, Y\. Zhang, Z\. Zhang, X\. Feng, and B\. DaiNormAct: Benchmarking Embodied Agents’ Proactive Compliance with Unspoken Social Norms\.arXiv preprint arXiv:2606\.27826\.Cited by:[§3\.1](https://arxiv.org/html/2610.13021#S3.SS1.p1.1)\.
- Zhenget al\.\(2025a\)J\. Zheng, X\. Cai, Q\. Li, D\. Zhang, Z\. Li, Y\. Zhang, L\. Song, and Q\. MaLifelongAgentBench: Evaluating LLM Agents as Lifelong Learners\.arXiv preprint arXiv:2505\.11942\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2025b\)J\. Zheng, C\. Shi, X\. Cai, Q\. Li, D\. Zhang, C\. Li, D\. Yu, and Q\. MaLifelong Learning of Large Language Model based Agents: A Roadmap\.arXiv preprint arXiv:2501\.07278\.Cited by:[§1](https://arxiv.org/html/2610.13021#S1.p1.1),[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.
- Zhonget al\.\(2026\)S\. Zhong, Y\. Lu, J\. Ning, Y\. Wan, L\. Feng, Y\. Ao, L\. F\. R\. Ribeiro, M\. Dreyer, S\. Ammirati, and C\. XiongSkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real\-World Tasks\.arXiv preprint arXiv:2604\.20087\.Cited by:[§2](https://arxiv.org/html/2610.13021#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ABasic Task Categories

As described in Section[3\.2](https://arxiv.org/html/2610.13021#S3.SS2), basic tasks inEmbodiedCLBenchspan six capability categories\. Table[2](https://arxiv.org/html/2610.13021#A1.T2)lists each category and its core evaluation question\.

Table 2:Six capability categories of basic tasks inEmbodiedCLBench\.
## Appendix BGraph Construction Pipeline

As described in Sections[3\.2](https://arxiv.org/html/2610.13021#S3.SS2)and[3\.4](https://arxiv.org/html/2610.13021#S3.SS4),EmbodiedCLBenchbuilds tasks from atomic actions available in TongSim\. Given the atomic actions for each task, we first construct the state\-transition graph\. We then generate two types of textual content: task descriptions and per\-state action options\.

#### LLM Generation\.

Both task descriptions and action options are produced by a three\-stage pipeline powered by Qwen3\.7\-Plus\. The pipeline serves two purposes: converting low\-level atomic actions into natural household scenarios, and ensuring that action options do not give away the correct answer through shortcuts\. The key prompt specifications for all three stages are provided in Section[B\.1](https://arxiv.org/html/2610.13021#A2.SS1)\.

#### Human Verification\.

Three annotators independently verify each generated sample\. The verification covers three aspects: captured images, task descriptions, and per\-state action options\. For each aspect, annotators check for factual errors, logical inconsistencies, and whether the content reflects realistic household activities\. A sample is retained only if all three annotators approve it\.

### B\.1Prompt Specifications

Stage 1: Task Description Rewrite\# Stage 1: Task Description Rewrite\#\# ObjectiveRewrite a baseline task description into a natural, scene\-based household request\. The rewritten description is what the tested model actually sees\. You do not change the underlying task structure \-\- only the surface expression\.\#\# Task Hierarchy\- L1 \(atomic goal\): the smallest unit of household goal, e\.g\., "turn off the TV", "open the cabinet", "place the cup on the table"\.\- L2 \(basic task\): composed of 2 L1 goals forming a coherent household activity\.\- L4 \(advanced task\): composed of 4 L1 goals combining multiple capability categories\.\#\# Input Fields\- level: L2 or L4\.\- ability\_categories: capability categories involved \(e\.g\., appliance operation, spatial placement\)\.\- l1\_goals: list of constituent L1 atomic goals with descriptions, action types, and target objects\.\- baseline\_task\_description: a direct concatenation of all L1 goals \(e\.g\., "Turn off the TV, open the lamp, close the cabinet door"\)\.\#\# Rewrite Principles1\. Merge multiple L1 goals into a single coherent living scenario \(e\.g\., "A guest is arriving soon \-\- make the living room presentable and quiet"\)\. The model should need to reason about what to do from the scene image, not read the steps directly from the text\.2\. Do NOT produce a checklist, enumerate target objects by name, expose execution order, or translate L1 goals one\-by\-one with synonyms\.3\. Replace specific object names with functional or contextual references \(e\.g\., TV \-\> "noise source"; table lamp \-\> "reading light"; cabinet \-\> "storage area"\)\.4\. Add a plausible living scenario \(guest arriving, bedtime routine, post\-cooking cleanup, etc\.\) that makes all L1 goals co\-exist naturally\.\#\# OutputA single JSON object with fields: description\_highlevel \(the rewritten task description\), rationale \(why the rewrite is natural and leak\-free\), and anti\_leak\_check \(boolean flags for exposes\_order, is\_checklist, mentions\_specific\_objects\)\.

Stage 2: Per\-State Option Rewrite\# Stage 2: Per\-State Option Rewrite\#\# ObjectiveRewrite the four hard\-coded action options at a single graph state into natural language\. After rewriting, all four options should look like plausible next steps, forcing the model to rely on the scene image and current state rather than linguistic cues\.\#\# Underlying Execution Pattern \(must be hidden\)Each household goal typically requires three phases:1\. Observe: look at the target object \(look\_at\)\.2\. Approach: move to the target object \(walk\_to\)\.3\. Operate: execute the action \(open / close / switch\_on / etc\.\)\.If option wording reveals which phase it belongs to \(e\.g\., observe options always say "check the status of X"\), the model can guess the answer without seeing the image\. The core challenge is to make all options sound like purposeful household actions regardless of their underlying phase\.\#\# Option Roles\- success\_path: the optimal next action\. Must not stand out among the four options\.\- valid\_detour: executable and task\-relevant, but not optimal\. Must read like active task progress, not like a detour\.\- invalid: cannot be executed in the current state\. Must read like a plausible next step; its appeal comes from semantic relevance to the task, not from low\-quality wording\.\#\# Rewrite Rules1\. Each option: 1 sentence, 10\-25 English words, naming the involved object or area and connecting to the high\-level task scenario\.2\. Two\-layer fit: every option \(including distractors\) must fit both the current sub\-goal and the high\-level task description\.3\. Hide phases: do not let observe options read as "status checks", approach options as "moving closer to operate", or operate options as the only ones with a sense of completion\.4\. Uniform style: all four options should be similar in length, detail, and directness\. The correct option must not be the only one containing task keywords\.5\. Invalid options must not contain words like "wrong", "invalid", or "not needed", and must not explain why the action cannot be executed\.\#\# OutputA single JSON object containing: state\_id, options \(each with id, text, role, design\_note\), and self\_check \(correct\_stands\_out, current\_goal\_fit, highlevel\_task\_fit, phase\_exposed, has\_implementation\_terms\)\.

Stage 3: Cross\-State Option Hardening\# Stage 3: Cross\-State Option Hardening\#\# ObjectiveGiven a batch of states from the same task graph, detect and eliminate cross\-state linguistic patterns that the tested model could exploit\. The model sees its full interaction history, so any recurring pattern across consecutive steps can be discovered and used to bypass visual reasoning\.\#\# ContextEach household goal takes three phases \(observe \-\> approach \-\> operate\)\. A basic task has 2 such goals, producing a repeating 2\-beat rhythm on the shortest path:Goal 1: observe \-\> approach \-\> operateGoal 2: observe \-\> approach \-\> operateStage 2 made each state’s options internally deceptive but could not address cross\-state patterns\. This stage fixes that\.\#\# InputA batch of states \(either the full shortest path or a branch cluster\), each containing: state\_id, state\_context, belongs\_to\_l1 \(which sub\-goal this state belongs to\), and options \(with baseline\_text, rewritten\_text, role, action\_type, object\)\.\#\# Patterns to Detect and Eliminate1\. Three\-beat rhythm: correct options \(success\_path\) across consecutive states show a cyclic "attention/focus \-\> head/move \-\> execute/switch" pattern that repeats per sub\-goal\.2\. Style bias: correct options are systematically shorter, more direct, or contain more task keywords than distractors\.3\. Expression repetition: the same phrasing or sentence structure appears across multiple states\.\#\# Hardening StrategiesA\. Blur phase boundaries: make observe\-phase correct options sound like preparation or positioning rather than looking\. Make approach\-phase correct options sound like executing part of an action rather than walking\.B\. Mix phase styles in distractors: in an observe state, give some distractors an operate\-style wording; in an operate state, give some distractors an observe\-style wording\.C\. Break repetition across sub\-goals: if sub\-goal 1’s correct options follow pattern X, sub\-goal 2’s correct options must use a different pattern\.D\. Vary detail level: some correct options should be short and direct, others should include contextual detail, matching the variation in distractors\.E\. Vary verbs and sentence structures: the same action type appearing in different states must use different verbs and phrasings\.\#\# Constraints\- Modify rewritten\_text only \(not state\_id, id, role, action\_type, or baseline\_text\)\.\- Edits are adjustments to Stage 2 output, not full rewrites from baseline\_text\.\- Options must remain specific, natural, and connected to the high\-level task scenario\.\- Do not use implementation terms \(graph, state, precondition, phase\)\.\#\# OutputA single JSON object containing: states \(each with state\_id, modified flag, options, modification\_note\) and pattern\_report \(documenting which patterns were found and how they were fixed\)\.

## Appendix CTask Examples

We provide task examples at both compositional levels to illustrate the evaluation format\. The agent first receives a natural\-language task description\. At each step, it observes the current\-state image and selects one action from multiple candidates\. Specifically, we include two basic\-task examples \(Figures[8](https://arxiv.org/html/2610.13021#A3.F8)and[9](https://arxiv.org/html/2610.13021#A3.F9); Tables[3](https://arxiv.org/html/2610.13021#A3.T3)and[4](https://arxiv.org/html/2610.13021#A3.T4)\) and one advanced\-task example \(Figure[10](https://arxiv.org/html/2610.13021#A3.F10); Table[5](https://arxiv.org/html/2610.13021#A3.T5)\)\.

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state0.jpg)

State 0

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state1.jpg)

State 1

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state2.jpg)

State 2

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state3.jpg)

State 3

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state4.jpg)

State 4

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state5.jpg)

State 5

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0015_state6.jpg)

State 6

Figure 8:Visual states along the shortest path for a basic\-task example\.Table 3:Basic\-task example: action options and hidden graph roles at each state\.Task:I’m about to watch a movie in the living room\. Turn off the overhead light so the screen isn’t washed out, and open up the media cabinet so everything’s within reach\.StateOptionText shown to the agentHidden roleState 0AHead over to the ceiling light area to get ready to turn it off\.Invalid in current stateState 0BSwitch on the overhead light to brighten the living room before the movie\.Invalid in current stateState 0CShut off the ceiling light right away to set the mood for the movie\.Invalid in current stateState 0DThe overhead light is still on — that’s the first thing to deal with before the movie\.Correct shortest\-path actionState 1ACheck the ceiling light to see if it’s already been switched off\.Invalid in current stateState 1BSwitch off the ceiling light to darken the room for movie viewing\.Invalid in current stateState 1CTurn on the ceiling light to brighten the space before settling in for the movie\.Invalid in current stateState 1DWalk over to the ceiling light so you can reach the switch\.Correct shortest\-path actionState 2ASwitch off the overhead light — the room needs to be dark for the movie\.Correct shortest\-path actionState 2BStep closer to the ceiling fixture to get a better grip on the switch\.Executable detourState 2CGlance up at the ceiling light to confirm how bright it still is\.Invalid in current stateState 2DCheck the television area to see if the media setup is ready\.Executable detourState 3AHead over to the television area to get the media setup going\.Invalid in current stateState 3BWalk toward the ceiling light to start working on the room’s ambiance\.Executable detourState 3CSwitch on the overhead light to brighten things up before the movie\.Executable detourState 3DWith the light off, the media cabinet by the TV is the next thing to handle\.Correct shortest\-path actionState 4APower down the television to eliminate any screen glare during the movie\.Invalid in current stateState 4BWalk up to the TV unit — the cabinet is right ahead\.Correct shortest\-path actionState 4CFocus on the television to assess whether the screen needs attention\.Invalid in current stateState 4DSwitch on the television so you can test the viewing angle for the movie\.Invalid in current stateState 5AShift your focus to the ceiling light to confirm it’s off for movie time\.Executable detourState 5BOpen the media cabinet so everything is within reach for the movie\.Correct shortest\-path actionState 5CTake a closer look at the television area to plan the next move for the media setup\.Invalid in current stateState 5DStep over to the television unit to get a better position for the entertainment area\.Executable detour![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state0.jpg)

State 0

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state1.jpg)

State 1

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state2.jpg)

State 2

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state3.jpg)

State 3

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state4.jpg)

State 4

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state5.jpg)

State 5

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l2_task_L2_HL_v5_adult_0022_state6.jpg)

State 6

Figure 9:Visual states along the shortest path for a basic\-task example\.Table 4:Basic\-task example: action options and hidden graph roles at each state\.Task:Get the living room ready for a relaxed evening — turn on some background entertainment and open up the storage cabinet so everything’s within reach\.StateOptionText shown to the agentHidden roleState 0AStart by checking the TV area — that’s where the background entertainment will come from\.Correct shortest\-path actionState 0BShut off the TV screen since it’s not needed for the living room ambiance\.Invalid in current stateState 0CHead over to the TV wall to get closer to the entertainment controls\.Invalid in current stateState 0DPull open the storage cabinet to make everything accessible for the evening\.Invalid in current stateState 1AShut the storage cabinet to keep the living room looking tidy\.Invalid in current stateState 1BShift your focus to the cabinet area near the TV to assess what’s needed\.Invalid in current stateState 1CGet to the entertainment setup — the background noise needs to start soon\.Correct shortest\-path actionState 1DPull open the storage compartment so things can be reached easily for the evening\.Invalid in current stateState 2ASwitch on the screen to bring some background entertainment into the living room\.Correct shortest\-path actionState 2BHead over to the screen area to get a better position for the setup\.Executable detourState 2CFocus on the storage cabinet to make sure it’s ready for easy access\.Executable detourState 2DCheck the screen to see if it’s already running before proceeding\.Invalid in current stateState 3AThe storage cabinet still needs opening — that’s the next thing to handle\.Correct shortest\-path actionState 3BHead over to the storage unit to get it ready for the evening\.Invalid in current stateState 3CSwitch off the ambient background element near the TV to set a cleaner mood\.Executable detourState 3DWalk toward the entertainment area to adjust the atmosphere\.Executable detourState 4AShut the storage cabinet to keep the living room looking tidy\.Invalid in current stateState 4BCheck the cabinet area to see what’s currently inside\.Invalid in current stateState 4CPull open the storage cabinet to make everything accessible for the evening\.Invalid in current stateState 4DMove into position near the storage cabinet — it needs to be opened\.Correct shortest\-path actionState 5AStep closer to the storage cabinet to get a better grip on the handle\.Executable detourState 5BScan the cabinet area to confirm everything is in order\.Invalid in current stateState 5CGlance toward the TV wall to take in the overall setup before finishing up\.Executable detourState 5DOpen up the storage cabinet so everything is within reach for the evening\.Correct shortest\-path action![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state0.jpg)

State 0

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state1.jpg)

State 1

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state2.jpg)

State 2

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state3.jpg)

State 3

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state4.jpg)

State 4

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state5.jpg)

State 5

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state6.jpg)

State 6

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state7.jpg)

State 7

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state8.jpg)

State 8

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state9.jpg)

State 9

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state10.jpg)

State 10

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state11.jpg)

State 11

![Refer to caption](https://arxiv.org/html/2610.13021v1/figure/tongbench_generation_example/l4_task_L4_HL_v5_adult_0044_state12.jpg)

State 12

Figure 10:Visual states along the shortest path for an advanced\-task example, including the final state\.Table 5:Advanced\-task example: action options and hidden graph roles\.Task:I’m done for the night and heading to bed — dim the bedroom lamp, open up the study desk so it’s ready for tomorrow, shut the bedside drawer, and toss the old remote into the bin\.StateOptionText shown to the agentHidden roleState 0ACheck the trash can area to see what needs to be tossed before bed\.Executable detourState 0BSwitch on the table lamp to brighten the bedside area for cleanup\.Invalid in current stateState 0CThe lamp’s still glowing — that’s the first thing to sort out before the room can rest\.Correct shortest\-path actionState 0DHead over to the table lamp to adjust the lighting for bedtime\.Invalid in current stateState 1ASwitch off the table lamp to dim the room for bedtime\.Invalid in current stateState 1BGet closer to the lamp — you’ll need to be right there to kill the light\.Correct shortest\-path actionState 1CBrighten the table lamp so you can see what’s left to tidy\.Invalid in current stateState 1DFocus on the table lamp area to assess what needs adjusting for the night\.Invalid in current stateState 2ABrighten the lamp a bit so you can see what’s left on the nightstand before closing it up\.Invalid in current stateState 2BKill the lamp light to start dimming the room for the night\.Correct shortest\-path actionState 2CShift over to the lamp side to get the lighting sorted before settling in\.Invalid in current stateState 2DFocus on the lamp area to assess what still needs adjusting before bed\.Invalid in current stateState 3AGlance over at the trash can to see what needs tossing before bed\.Executable detourState 3BThe study desk still has stuff on it — that’s the next area to sort out\.Correct shortest\-path actionState 3CHead over to the study desk to get in position for clearing it off\.Invalid in current stateState 3DFlip open the study desk to start organizing items for tomorrow’s setup\.Invalid in current stateState 4AGlance over at the desk to see what’s still out on the surface\.Invalid in current stateState 4BFlip open the desk to start organizing what’s inside for tomorrow\.Invalid in current stateState 4CShut the desk drawers to keep the room tidy for the night\.Invalid in current stateState 4DStep up to the desk so you can reach across and clear it off properly\.Correct shortest\-path actionState 5AShift your focus to the desk to assess what still needs clearing\.Invalid in current stateState 5BOpen the desk cabinet so you can stow tonight’s items inside\.Correct shortest\-path actionState 5CGlance over at the table lamp to check whether the room is dim enough\.Executable detourState 5DTake a moment to review the bedside cabinet before closing it up for the night\.Executable detourState 6AShift your focus to the table lamp to see if it still needs attention for the night\.Executable detourState 6BGlance over at the trash can to check if anything else needs tossing before bed\.Executable detourState 6CTake a look at the remote control in your hand to decide where it should go next\.Invalid in current stateState 6DThe bedroom nightstand is still open — head back and deal with that next\.Correct shortest\-path actionState 7AShut the bedside cabinet to keep the room tidy for the night\.Invalid in current stateState 7BPull the bedside cabinet open to check if anything needs to be put away\.Invalid in current stateState 7CGlance over at the bedside cabinet to see what’s still out\.Invalid in current stateState 7DMove in close to the nightstand — the cabinet needs shutting and you need to be right there\.Correct shortest\-path actionState 8AShift focus to the desk area to see what needs clearing for tomorrow\.Executable detourState 8BShut the bedside cabinet to keep the nightstand tidy before sleep\.Correct shortest\-path actionState 8CCheck the table lamp status to see if it needs dimming for bed\.Executable detourState 8DGlance at the bedside cabinet to confirm what’s inside before closing\.Invalid in current stateState 9AShift focus to the table lamp area to continue dimming the room for rest\.Executable detourState 9BThe remote control still needs to go — spot the trash can so you know where it’s headed\.Correct shortest\-path actionState 9CGlance over at the desk surface to assess what else still needs to be cleared away\.Executable detourState 9DCheck the remote control in your hand to decide whether it should be put away\.Invalid in current stateState 10ASeal up the trash can so nothing spills out overnight\.Invalid in current stateState 10BGlance at the trash can to make sure it’s ready to receive items\.Invalid in current stateState 10CPop open the trash can lid so you can drop things in\.Invalid in current stateState 10DGet to the trash can — the remote control in your hand needs to go there\.Correct shortest\-path actionState 11ASet the remote control down near the trash can to keep the nightstand clear\.Correct shortest\-path actionState 11BCheck the bedside cabinet to make sure it’s properly closed for the night\.Executable detourState 11CTake a closer look at the remote control to decide where it should go\.Invalid in current stateState 11DTurn your attention to the desk area to see what still needs clearing\.Executable detour