Object-Centric Environment Modeling for Agentic Tasks

# Object-Centric Environment Modeling for Agentic Tasks
Source: [https://arxiv.org/html/2607.02846](https://arxiv.org/html/2607.02846)
Yiyang Li1,Tianyi Ma1,Zehong Wang1,Yijun Ma1,Yanfang Ye1† 1University of Notre Dame†Corresponding Author \{yli62, yye7\}@nd\.edu[Code](https://github.com/Yiyang-Ian-Li/OCM)

###### Abstract

Large language model \(LLM\) agents can improve through accumulated experience, but free\-form textual memories become difficult to maintain, validate, and reuse as interactions grow\. Recent symbolic approaches learn executable skills or programmatic world models, yet often store local procedures or assume simplified dynamics\. We proposeObject\-Centric EnvironmentModeling \(OCM\), which organizes experience into an executable object\-centric environment model\. OCM maintains two connected code bases: object knowledge, which defines environment entities and mechanisms as Python classes, and procedure knowledge, which records reusable interaction patterns that must import and use the object model\. OCM works in an online setting: after each episode, OCM reflects on the trajectory, updates both knowledge bases, and verifies that all procedures execute against the updated object model\. During future interaction, the agent uses progressive knowledge disclosure to inspect compact code signatures first and read source code only when needed\. Experiments show that OCM achieves the best average rank across benchmarks and reduces invalid actions, demonstrating that agents can benefit from building object\-centric environment models\.

Object\-Centric Environment Modeling for Agentic Tasks

Yiyang Li1, Tianyi Ma1, Zehong Wang1, Yijun Ma1, Yanfang Ye1†1University of Notre Dame†Corresponding Author\{yli62, yye7\}@nd\.edu[Code](https://github.com/Yiyang-Ian-Li/OCM)

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.02846v1/x1.png)Figure 1:Upper left: Isolated skills resemble scattered manuals whose assumed tools and parts are potentially inconsistent\.Upper right: Monolithic symbolic world models resemble an overcomplicated machine that attempts to model all transitions\.Bottom: OCM standardizes objects and mechanisms first, then grounds procedural knowledge in this shared object model, making experience easier to organize, verify, and reuse\.As large language model \(LLM\) agents are increasingly deployed in interactive environments, their ability to learn from past experiences has become critical for solving long\-horizon and open\-ended tasksZhanget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib6)\); Huet al\.\([2025b](https://arxiv.org/html/2607.02846#bib.bib7)\); Yeet al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib31)\); Chenet al\.\([2025a](https://arxiv.org/html/2607.02846#bib.bib32),[b](https://arxiv.org/html/2607.02846#bib.bib33)\)\. To enable agents to improve over time, prior work has introduced memory mechanisms that store and reuse previous experience as textual memory streams, verbal reflections, or reusable workflowsZhaoet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib2)\); Wanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib3)\); Zhonget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib4)\)\. Despite their effectiveness, textual knowledge becomes increasingly difficult to manage as agents accumulate more interactions\. Newly written knowledge may be redundant or inconsistent with previous ones, while large collections of textual knowledge are hard to validate or auditHuet al\.\([2025a](https://arxiv.org/html/2607.02846#bib.bib16)\); Chaoet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib8)\)\. These limitations can make agent behavior less controllable and less stable, especially in complex environments where errors in memory may propagate across future decisionsXionget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib5)\); Lamet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib17)\)\.

To make accumulated experience easier to inspect and reuse, a growing line of work represents agent knowledge in symbolic or executable formsNinget al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib9)\)\. These methods learn symbolic rules, code\-based skills, or programmatic world models, improving interpretability and enabling execution\-based validationWanget al\.\([2023](https://arxiv.org/html/2607.02846#bib.bib10),[2025](https://arxiv.org/html/2607.02846#bib.bib11)\); Zhenget al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib14)\)\. However, existing approaches typically capture only one side of the desired structure\. Code\-based skill learning stores reusable procedures, but these procedures often remain local, without a shared account of the objects, states, affordances, and preconditions that make them validWanget al\.\([2023](https://arxiv.org/html/2607.02846#bib.bib10)\); Zhouet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib12)\); Wanget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib11)\)\. Programmatic world\-modeling methods instead aim to model environment dynamics explicitly, but are usually studied in settings with smaller action spaces and well\-specified statesTanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib13)\); Khanet al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib18)\); Piriyakulkijet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib15)\)\. This leaves a missing middle ground: agents need symbolic knowledge that is more coherent than isolated skills, but less demanding than a monolithic transition model\. Figure[1](https://arxiv.org/html/2607.02846#S1.F1)summarizes this tension\. The key challenge is to organize experience around persistent environment entities and mechanisms, so that procedural knowledge can be reused while remaining consistent with a shared object\-level model\.

To address this gap, we proposeObject\-Centric EnvironmentModeling \(OCM\), a framework that enables an agent to construct and maintain an object\-centric model of its environment through interaction\. OCM represents the agent’s knowledge with two mutually connected code bases\. The first,*object knowledge*, defines Python classes for the objects in the environment; class attributes and methods encode object properties and interaction mechanisms\. The second,*procedure knowledge*, stores interaction experience as executable Python files\. Each procedure file describes a concrete piece of knowledge or experience, and must import and use the classes defined in object knowledge\. After each interaction episode, the agent updates both forms of knowledge and executes the procedure files to check whether the accumulated knowledge remains mutually consistent, turning past experience into an auditable, executable, and object\-grounded environment model\. During future interactions, the OCM agent can inspect the signatures of object classes and procedure functions and selectively read their source code\. OCM encourages the agent to organize knowledge around persistent entities and their interaction mechanisms, enabling more structured knowledge accumulation, self\-consistency checking, and more controllable reuse of prior experience in complex environments\.

We evaluate OCM on three interactive benchmarks: ScienceWorld, ALFWorld, and PlanCraft\. OCM achieves the best average rank among compared methods and reduces invalid actions\. These results indicate that grounding procedural experience in a shared object model helps agents accumulate reusable environment knowledge and act more reliably in complex interactive tasks\. Our main contributions are as follows:

- •We proposeobject\-centric environment modeling, a framework that represents agent experience as an executable environment model composed of object knowledge and object\-grounded procedure knowledge\.
- •We introduce a continual learning loop that reflects on completed trajectories, updates the object\-centric code base, and verifies executable consistency before committing new knowledge\.
- •We validate OCM on three interactive benchmarks and analyze how its learned knowledge improves performance and action feasibility\.

## 2Related Work

### 2\.1Memory\-Augmented Agents

Memory mechanisms enable LLM agents to reuse past experience across long\-horizon interactionsKagayaet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib28)\); Chenet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib29)\); Zhanget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib6)\); Huet al\.\([2025b](https://arxiv.org/html/2607.02846#bib.bib7)\)\. Most existing methods store experience primarily as natural\-language memories: Reflexion stores verbal feedback from previous trials to improve later attemptsShinnet al\.\([2023](https://arxiv.org/html/2607.02846#bib.bib1)\); ExpeL extracts reusable lessons from successful and failed trajectoriesZhaoet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib2)\); and Agent Workflow Memory abstracts trajectories into reusable workflowsWanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib3)\)\. These methods show that experience reuse can substantially improve agent behavior\. But free\-form textual memory is difficult to maintain as it grows, learned memories may become stale, redundant, or mutually inconsistent, and large memory collections are hard to validate or auditHuet al\.\([2025a](https://arxiv.org/html/2607.02846#bib.bib16)\); Chaoet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib8)\); Xionget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib5)\); Lamet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib17)\)\. OCM shares the goal of continual experience accumulation, but represents experience as executable object\-grounded code rather than only textual memory\.

### 2\.2Symbolic World Modeling

Symbolic and executable representations provide a more structured substrate for agent knowledge\. Prior work in object\-oriented modeling represents environments through objects, attributes, relations, and transition mechanisms, enabling abstraction and transferDiuket al\.\([2008](https://arxiv.org/html/2607.02846#bib.bib23)\); Kanskyet al\.\([2017](https://arxiv.org/html/2607.02846#bib.bib24)\)\. Recent LLM agent methods similarly use symbolic programs to store reusable knowledgeDaineseet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib27)\); Shaoet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib25)\); Zhaoet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib26)\)\. For example, Voyager, Wall\-E, and ASI accumulate executable skills in codeWanget al\.\([2023](https://arxiv.org/html/2607.02846#bib.bib10)\); Zhouet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib12)\); Wanget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib11)\); WorldCoder and PoE\-World learn Python world models for planningTanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib13)\); Piriyakulkijet al\.\([2026](https://arxiv.org/html/2607.02846#bib.bib15)\)\. These approaches improve interpretability and enable execution\-based verification, but they typically focus on either local skills or world modeling in structured environments\. In contrast, OCM couples procedural experience with a shared object\-centric code model that defines persistent entities and mechanisms\. This encourages a unified, auditable, and self\-consistent environment model that can be incrementally updated through interaction\.

![Refer to caption](https://arxiv.org/html/2607.02846v1/x2.png)Figure 2:Overview of OCM\.\(a\) During an episode, the agent uses progressive knowledge disclosure: it first observes signatures of learned object and procedure codes, then selectively inspects relevant source code before choosing an environment action\. \(b\) After each episode, OCM converts the trajectory into an object\-centric update plan, generates new candidate object and procedure code, and commits the update only after verification, where all procedure files must execute successfully against the updated object code\. By grounding procedure knowledge in shared object knowledge, OCM maintains an executable and auditable environment model for future interactions\.

## 3Object\-Centric Environment Modeling

### 3\.1Preliminary

#### Problem Formulation\.

We consider a partially observable interactive environment with observation space𝒪\\mathcal\{O\}and action space𝒜\\mathcal\{A\}\. An agent continually interacts with the environment over a sequence ofKKtask episodes\{τk\}k=1K\\\{\\tau\_\{k\}\\\}\_\{k=1\}^\{K\}\. At stepttin episodekk, the agent observesotk∈𝒪o\_\{t\}^\{k\}\\in\\mathcal\{O\}and selects an actionatk∈𝒜a\_\{t\}^\{k\}\\in\\mathcal\{A\}\. The trajectory of episodekkis denoted asHk=\(o0k,a1k,o1k,…,aTk,oTk\),H^\{k\}=\(o\_\{0\}^\{k\},a\_\{1\}^\{k\},o\_\{1\}^\{k\},\\ldots,a\_\{T\}^\{k\},o\_\{T\}^\{k\}\),whereo0ko\_\{0\}^\{k\}includes the initial task description\. After the episode terminates, the agent receives a final rewardRk∈\[0,1\]R^\{k\}\\in\[0,1\]\. The goal is to maximize task success and cumulative reward∑k=1KRk\\sum\_\{k=1\}^\{K\}R^\{k\}through continual interaction and knowledge accumulation\.

#### Knowledge Representation\.

OCM maintains an executable environment model consisting of two connected knowledge bases\. The first is*object knowledge*, denotedKoK\_\{o\}, implemented as a Python module that defines object classes and helper functions\. These classes encode reusable entities in the environment, including their observable states, affordances, constraints, relations, and transition mechanisms\. The second is*procedure knowledge*, denotedKpK\_\{p\}, implemented as a set of Python files\. Each procedural file records one focused interaction pattern or experiential rule, such as when a mechanism applies, why an action succeeds or fails, or how feedback should be interpreted\. A central constraint of OCM is that procedure knowledge must be grounded in object knowledge\. A procedural file is not allowed to stand as an isolated trajectory summary; it must import and use classes or functions fromKoK\_\{o\}\. Thus, episodic experience is linked to persistent object\-level abstractions rather than being stored as disconnected textual memories or action scripts\. Together,\(Ko,Kp\)\(K\_\{o\},K\_\{p\}\)forms the agent’s object\-centric environment model\.

#### Overview\.

During an episodekk, if knowledge has been accumulated, the agent acts using the current environment model\. The agent is given compact interfaces ofKokK\_\{o\}^\{k\}andKpkK\_\{p\}^\{k\}: signatures of object classes and method functions, together with an index of procedural files and their public functions\. These summaries expose the structure of the learned model without placing all accumulated source code into the prompt\.

After an episode ends, OCM updates its environment model through two stages\. First, an object\-centric reflection stage converts the trajectory into a structured learning plan, identifying which reusable object mechanisms or procedural patterns may need to be created or revised\. Second, a knowledge update stage generates concrete Python code for the planned changes and validates the resulting candidate knowledge base\. An update is committed only if the candidate code is executable against the current object model\.

As illustrated in Figure[2](https://arxiv.org/html/2607.02846#S2.F2), this yields a closed learning loop: future decisions are made from the current object\-centric model; new feedback is reflected into the object\-grounded model; and only executable updates that remain consistent with prior knowledge are incorporated\.

### 3\.2Object\-Centric Knowledge Reflection

The reflection stage converts a completed episode into a structured learning plan\. Instead of directly rewriting the knowledge base from the raw trajectory, OCM first decides*what*should be learned\. This separation prevents incidental episode details, such as transient locations, repeated invalid actions, or task\-specific action sequences, from being stored as long\-term knowledge\.

Given the trajectoryHkH^\{k\}, task resultsRkR^\{k\}, the object signaturesSig​\(Kok\)\\mathrm\{Sig\}\(K\_\{o\}^\{k\}\), and the procedural indexInd​\(Kpk\)\\mathrm\{Ind\}\(K\_\{p\}^\{k\}\), the reflection model produces

Pk\\displaystyle P^\{k\}=Reflect​\(Hk,Rk,Sig​\(Kok\),Ind​\(Kpk\)\)\\displaystyle=\\mathrm\{Reflect\}\\\!\\left\(H^\{k\},R^\{k\},\\mathrm\{Sig\}\(K\_\{o\}^\{k\}\),\\mathrm\{Ind\}\(K\_\{p\}^\{k\}\)\\right\)=\(Pok,Ppk,Iok,Ipk\)\.\\displaystyle=\\left\(P\_\{o\}^\{k\},P\_\{p\}^\{k\},I\_\{o\}^\{k\},I\_\{p\}^\{k\}\\right\)\.Here,PokP\_\{o\}^\{k\}andPpkP\_\{p\}^\{k\}specify planned updates to object knowledge and procedure knowledge, whileIokI\_\{o\}^\{k\}andIpkI\_\{p\}^\{k\}specify object symbols and procedural files whose source code should be inspected before editing\. The plan is constrained to contain only reusable mechanisms or decision patterns supported by the episode\. Object updates should describe recurring environment mechanisms, such as object states, affordances, preconditions, effects, or relations\. For example, the agent should learn that closed containers must be opened before their contents are accessible, rather than memorize that a particular cabinet was closed in one episode\. Procedure updates, in turn, must be explained through the object model: a procedural pattern should identify which object classes or methods determine when the pattern applies and why it succeeds or fails\.

The inspection fields connect new evidence to prior knowledge\. If the episode appears to revise an existing class, function, or procedural pattern, OCM retrieves the corresponding source code before generating an update\. The agent therefore sees both the new trajectory evidence and the relevant old implementation, reducing accidental duplication and making revisions local\. Reflection thus acts as a routing mechanism: it filters noisy experience into a small set of object\-grounded learning targets and identifies the prior code that those targets must remain compatible with\.

### 3\.3Object\-Centric Knowledge Update

Given the reflection plan, OCM generates concrete Python code\. The agent receives the planPkP^\{k\}, the trajectoryHkH^\{k\}, task resultsRkR^\{k\}, and the inspected source codeSk=Src​\(Iok,Ipk\)S^\{k\}=\\mathrm\{Src\}\(I\_\{o\}^\{k\},I\_\{p\}^\{k\}\)\. It outputs the candidate object and procedure code:

\(Δ​Cok,Δ​Cpk\)=Update​\(Pk,Hk,Rk,Sk\)\.\(\\Delta C\_\{o\}^\{k\},\\Delta C\_\{p\}^\{k\}\)=\\mathrm\{Update\}\(P^\{k\},H^\{k\},R^\{k\},S^\{k\}\)\.
Object updatesΔ​Cok\\Delta C\_\{o\}^\{k\}are merged intoKokK\_\{o\}^\{k\}\. New top\-level definitions are appended, while updated definitions replace existing definitions with the same name\. This allows the object model to evolve as new evidence accumulates while keeping a single shared namespace for environment entities and mechanisms\. The resulting object module is then compiled as part of validation before it can affect future decisions\.

Procedural updatesΔ​Cpk\\Delta C\_\{p\}^\{k\}are stored as separate Python files\. Each procedural file must encode one focused interaction pattern and must import and use relevant symbols fromKokK\_\{o\}^\{k\}\. A procedure may define workflows, rules, failure patterns, heuristics, or recovery principles\. It must not define a complete action controller or replay an entire trajectory\. This restriction distinguishes procedure knowledge from executable policies: procedures explain reusable mechanisms and decision patterns, but they do not directly control the agent\.

Before committing a candidate update, OCM verifies that the full procedural knowledge remains executable against the candidate object knowledge:

Verify​\(K~o,K~p\)=⋀p∈K~pExec​\(p∣K~o\)\.\\mathrm\{Verify\}\(\\widetilde\{K\}\_\{o\},\\widetilde\{K\}\_\{p\}\)=\\bigwedge\_\{p\\in\\widetilde\{K\}\_\{p\}\}\\mathrm\{Exec\}\(p\\mid\\widetilde\{K\}\_\{o\}\)\.Here,Exec​\(p∣K~o\)\\mathrm\{Exec\}\(p\\mid\\widetilde\{K\}\_\{o\}\)returns true only if procedureppexecutes successfully\. Every procedure file in the current candidate knowledge base\(K~o,K~p\)\(\\widetilde\{K\}\_\{o\},\\widetilde\{K\}\_\{p\}\)must execute successfully using the updated object knowledge\. If verification fails, the error is returned to the update model for another attempt; otherwise, the candidate is committed\. This verification step does not prove that the learned model is semantically complete or always correct\. Instead, it enforces a practical invariant: every committed procedure remains executable against the current object knowledge base\. As a result, local update errors are less likely to silently corrupt future decisions, and the accumulated environment model remains auditable as ordinary source code\.

### 3\.4Progressive Knowledge Disclosure

The same object\-centric structure used for learning also controls how knowledge is exposed during future decision\-making\. Placing all accumulated code into every prompt would be expensive and distracting\. OCM instead exposes a compact interface first, and reveals source code only when needed\.

At stepttof episodekk, the decision context is

Ct\(0\)=\[ℐ,otk,H<tk,Sig​\(Kok\),Ind​\(Kpk\)\],C\_\{t\}^\{\(0\)\}=\\left\[\\mathcal\{I\},o\_\{t\}^\{k\},H\_\{<t\}^\{k\},\\mathrm\{Sig\}\(K\_\{o\}^\{k\}\),\\mathrm\{Ind\}\(K\_\{p\}^\{k\}\)\\right\],whereℐ\\mathcal\{I\}is the benchmark instruction,otko\_\{t\}^\{k\}is the new observation, andH<tkH\_\{<t\}^\{k\}is the trajectory so far\. The object signatures expose public classes, methods, functions, and docstrings fromKokK\_\{o\}^\{k\}; the procedural index lists procedural files and their public functions\. These summaries provide enough information for the agent to recognize potentially relevant knowledge without loading the entire knowledge base into context\.

When the agent needs implementation details, it may issue an internal inspection requestqqof the formInspect\[\.\.\.\]instead of an environment action\. OCM retrieves the matching object definitions or procedural files and expands the context:

Ct\(1\)=Ct\(0\)⊕Src​\(Retrieve​\(q;Kok,Kpk\)\)\.C\_\{t\}^\{\(1\)\}=C\_\{t\}^\{\(0\)\}\\oplus\\mathrm\{Src\}\\\!\\left\(\\mathrm\{Retrieve\}\(q;K\_\{o\}^\{k\},K\_\{p\}^\{k\}\)\\right\)\.Inspection is internal and bounded: if no matching code is found, the agent is informed that inspection failed; if the inspection budget is exhausted, it must produce an environment action\.

Progressive disclosure prevents the growing knowledge model from overfilling the context window\. Reflection and update organize experience into object classes and focused procedural files; signatures and indexes expose the high\-level structure of that model; inspection reveals implementation details only when they are relevant to the present decision\. Thus, OCM can reuse an expanding body of learned knowledge while preserving the scalability, auditability, and consistency benefits of an executable object\-centric environment model\.

## 4Experiments

Table 1:Performance and Average Ranking Comparison\.Results are reported on ScienceWorld, ALFWorld, and PlanCraft\.SRmeans Success Rate\. The best and second\-best performances are highlighted inBoldandUnderline\.![Refer to caption](https://arxiv.org/html/2607.02846v1/x3.png)Figure 3:Distribution of invalid actions across ScienceWorld and PlanCraft\.Each violin shows the per\-task distribution of invalid actions for one method, with the x\-axis plotted on alog⁡\(1\+x\)\\log\(1\+x\)scale to reduce the effect of long\-tailed failures\. Black dots mark medians, horizontal black bars mark interquartile ranges, and numeric annotations report mean invalid actions per task in the original scale\. OCM yields the lowest mean invalid\-action count on both benchmarks, suggesting that object\-centric environment modeling improves action feasibility\.### 4\.1Experimental Setting

#### Benchmarks\.

We evaluate OCM on three interactive agent benchmarks: ScienceWorldWanget al\.\([2022](https://arxiv.org/html/2607.02846#bib.bib20)\), ALFWorldShridharet al\.\([2020](https://arxiv.org/html/2607.02846#bib.bib22)\), and PlanCraftDaganet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib21)\), which respectively capture challenges in grounded scientific reasoning, embodied household interaction, and structured crafting dynamics\. To ensure a fair comparison and control computational resources, we sampled five tasks from each of 30 ScienceWorld task categories as test data\. Meanwhile, we used the unseen split for ALFWorld and the test \(small\) split for PlanCraft\. More details are provided in Appendix[A\.1](https://arxiv.org/html/2607.02846#A1.SS1)\.

#### Baselines\.

We compare OCM with standard interactive agents, including ReActYaoet al\.\([2022](https://arxiv.org/html/2607.02846#bib.bib19)\)and ReflexionShinnet al\.\([2023](https://arxiv.org/html/2607.02846#bib.bib1)\); memory\-augmented agents, including ExpelZhaoet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib2)\)and AWMWanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib3)\); and symbolic\-learning or world\-modeling agents, including ASIWanget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib11)\), WorldCoderTanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib13)\), and Wall\-EZhouet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib12)\)\. All methods are evaluated in the same online setting: agents process episodes sequentially and may update their memory or learned knowledge after each episode, but receive no offline training data or benchmark\-specific demonstrations\. We use GPT\-4\.1\-mini as the backbone LLMOpenAI \([2025](https://arxiv.org/html/2607.02846#bib.bib30)\)\. More implementation details are provided in Appendix[A\.2](https://arxiv.org/html/2607.02846#A1.SS2)\.

#### Metrics\.

We report success rate \(SR\), average reward, average environment steps, and average rank \(A\.R\.\) across benchmark metrics\. For PlanCraft, we additionally report results by difficulty level\. To analyze interaction quality, we measure the number of invalid actions rejected by the environment\.

![Refer to caption](https://arxiv.org/html/2607.02846v1/x4.png)Figure 4:Analysis of OCM across ScienceWorld and PlanCraft\.\(a\) Cumulative performance over the online evaluation sequence, measured by reward on ScienceWorld and success rate on PlanCraft\. \(b\) Paired win/loss comparisons between OCM and each baseline on the same task instances\. Bars to the right indicate tasks solved by OCM but not the baseline, while bars to the left indicate the opposite\. \(c\) Growth of OCM’s environment knowledge over time\. Object and procedural entries both expand throughout evaluation, showing that OCM accumulates reusable environment structure rather than relying only on per\-task reflection\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2607.02846#S4.T1)shows that OCM achieves the best average rank across the three benchmarks, indicating consistent gains across different environment families\. The improvement is greatest on ScienceWorld and PlanCraft, where success depends heavily on reusable object mechanisms and state transformations\. On ScienceWorld, OCM obtains the highest success rate and reward while using a competitive number of steps\. This suggests that object\-centric knowledge helps the agent reuse recurring mechanisms such as containment, heating, cooling, and material transformation, rather than relying only on task\-specific experience\. PlanCraft provides a complementary test of structured transformation knowledge\. OCM achieves the best overall success rate, with especially clear gains on medium tasks\. Easy tasks can often be solved by direct reasoning or short\-horizon search, but harder recipes require tracking intermediate dependencies and avoiding invalid operations\. The stronger gains in these settings are consistent with the design of OCM, which can represent materials, tools, and transformations as reusable object\-centered knowledge\. Results on ALFWorld are more mixed: OCM is competitive but does not outperform all baselines\. This suggests that the benefits of object\-centric modeling are most pronounced when transfer requires explicit mechanisms\. In benchmarks dominated by high\-level household routines, textual experiential lessons may already provide strong guidance\.

### 4\.3Learning Dynamics

Figure[4](https://arxiv.org/html/2607.02846#S4.F4)examines how performance evolves over the online evaluation process\. On ScienceWorld and PlanCraft, OCM maintains a stronger cumulative performance curve over time, suggesting that its learned knowledge remains useful as more episodes are processed\. This is important for online agents, where accumulated memory can easily become noisy, redundant, or overly specific\. The paired win\-loss comparison compares methods on the same task instances\. Across both benchmarks, OCM solves more tasks that each baseline fails to solve than vice versa\. This indicates that the aggregate gains are not driven by a small number of outlier tasks, but reflect broader improvements across the evaluation set\. The knowledge\-growth curves further show that both object knowledge and procedure knowledge continue to expand during evaluation\. This supports the intended division of labor: object knowledge captures reusable entities and mechanisms, while procedure knowledge records focused ways of applying them\.

### 4\.4Action Validity Analysis

Final success alone does not fully characterize agent behavior, since an agent may eventually solve a task after many invalid or unproductive actions\. Figure[3](https://arxiv.org/html/2607.02846#S4.F3)therefore measures invalid actions at the task level\. OCM has the lowest mean invalid\-action count on both ScienceWorld and PlanCraft, indicating that its learned knowledge improves not only task completion but also action feasibility\.

This result is closely aligned with the method design\. Workflow\-based memories can still fail when their implicit preconditions do not match the current state\. OCM reduces this risk by grounding procedural knowledge in shared object definitions and exposing object signatures during action selection\. As a result, the agent can better reason about affordances and preconditions before acting\.

### 4\.5Ablation Study

![Refer to caption](https://arxiv.org/html/2607.02846v1/x5.png)Figure 5:Example of Object\-Centric Knowledge Update\.From a successful PlanCraft trajectory for craftingpink\_glazed\_terracotta, OCM extracts object knowledge for the intermediatepink\_terracottarecipe\.Table[2](https://arxiv.org/html/2607.02846#S4.T2)isolates the contribution of the main components of OCM on PlanCraft\.Removing object knowledgereduces the success rate, showing that procedural experience alone does not fully capture the reusable structure of crafting tasks\. Without a shared object model, learned procedures are less reliably grounded in environment constraints\.Removing procedure knowledgealso hurts performance and increases invalid actions\. This indicates that object knowledge alone is not sufficient: object classes encode affordances and mechanisms, but the agent still benefits from procedure files that summarize recurring ways of applying them\. The two knowledge bases therefore play complementary roles\. Theno\-verification variantsuffers the largest drop in success rate\. During online interactions, unverified updates can persist across future episodes and mislead later decisions\. Verification helps prevent this by requiring newly written procedures to execute successfully against the current object knowledge before they are committed\.

Table 2:Ablation results of OCM on PlanCraft\.
### 4\.6Case Study

We further examine how OCM turns a successful interaction into reusable knowledge on a PlanCraft task whose goal is to craftpink\_glazed\_terracotta\. The initial inventory containsterracottaandpink\_dye, but the target item is not directly craftable from these ingredients\. The successful trajectory therefore has two stages: the agent first fills the crafting grid withterracottain every slot except the center slotB2, placespink\_dyeinB2, obtainspink\_terracottafrom the output slot, and then smeltspink\_terracottaintopink\_glazed\_terracotta\. This trajectory is useful because it contains both a recipe and a state transformation, rather than a single isolated action\.

After the episode, OCM writes object knowledge that makes the intermediate object explicit\. As illustrated in Figure[5](https://arxiv.org/html/2607.02846#S4.F5), the learnedPinkTerracottaclass definescrafting\_requirements\(\), which records the exact slot\-level recipe, andcan\_be\_crafted\(\), which checks whether the inventory and crafting grid satisfy those requirements without extra misplaced items\. It also writesPinkGlazedTerracotta, which records thatpink\_terracottais the smeltable precursor\. These object definitions separate stable environment mechanisms from the particular episode in which they were observed\.

OCM also writes procedure knowledge grounded in those objects\. Thecraft\_pink\_terracottaprocedure importsPinkTerracotta, uses its requirements to detect missing slots, wrong item types, insufficient quantities, and unexpected extra items, and records recovery principles for repairing the grid\. Thesmelt\_pink\_terracottaprocedure importsPinkGlazedTerracottaand encodes the precursor\-to\-output transformation\. Both procedures include verifier tests, such as checking that a correct grid passescan\_be\_crafted\(\), and that a grid with an extra wrong item fails\. This case illustrates the central benefit of object\-centric modeling: the learned knowledge is not merely “craft pink terracotta before smelting,” but an executable account of the object constraints, failure modes, and transformation preconditions that make the experience reusable\.

## 5Conclusion

We presented OCM, a framework for organizing interactive agent experience as an executable object\-centric environment model\. By grounding procedural knowledge in shared object definitions and verifying updates before committing them, OCM supports more structured, reusable, and auditable experience accumulation than free\-form memories or isolated skills\. Experiments across three benchmarks show that OCM improves overall performance and reduces invalid actions, suggesting that object\-centric executable modeling is an effective substrate for online agent learning\.

## Acknowledgements

This work was partially supported by the NSF under grants IIS\-2533550, IIS\-2321504, IIS\-2217239, CNS\-2426514, and CMMI\-2146076, Notre Dame Strategic Framework Research Grant \(2025\), and Notre Dame Poverty Research Package \(2025\)\. Any expressed opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily reflect the views of the sponsors\.

## Limitations

OCM has several limitations that suggest directions for future work\. First, our experiments focus on text\-based interactive environments with relatively stable object types, action interfaces, and transition mechanisms\. This setting is well aligned with object\-centric modeling, but future work could study how to extend OCM to more open\-ended, multimodal, or dynamically changing environments where object boundaries and affordances are less explicit\.

Second, OCM verifies executable consistency rather than providing a formal guarantee of semantic correctness\. The current verification step ensures that procedural knowledge remains runnable against the object knowledge base, which is useful for maintaining an auditable code model, but it does not prove that every learned mechanism is complete or universally correct\. Future work could incorporate stronger automatically generated tests, counterexample search, or environment\-grounded validation to further improve semantic reliability\.

Third, OCM introduces additional computation for reflection, knowledge updating, inspection, and verification\. Although progressive knowledge disclosure reduces the need to place the full accumulated code base into every prompt, the framework still requires extra LLM calls after episodes and occasional source\-code inspection during interaction\. Improving the efficiency of OCM is an important practical direction\.

Finally, the quality of the learned object and procedure knowledge depends on the backbone model’s ability to abstract reusable mechanisms from trajectories and write coherent code\. Future work could evaluate how OCM scales across model families, model sizes, and more constrained code\-generation settings\.

## Ethical Considerations

This work evaluates LLM agents in controlled simulated environments and does not involve human subjects, user studies, personal data, or sensitive real\-world applications\. The proposed framework concerns the organization and verification of accumulated agent knowledge rather than deployment in high\-stakes settings\. We therefore do not identify direct ethical concerns beyond standard risks of LLM\-based agents, such as imperfect reasoning or incorrect actions, which are limited here to benchmark environments\.

## References

- STALE: can llm agents know when their memories are no longer valid?\.arXiv preprint arXiv:2605\.06527\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- C\. Chen, Z\. Zhang, B\. Guo, S\. Ma, I\. Khalilov, S\. Gebreegziabher, Y\. Ye, Z\. Xiao, Y\. Yao, T\. Li,et al\.\(2025a\)The obvious invisible threat: llm\-powered gui agents’ vulnerability to fine\-print injections\.InSoups,Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1)\.
- C\. Chen, D\. Zhou, Y\. Ye, T\. J\. Li, and Y\. Yao \(2025b\)Clear: towards contextual llm\-empowered privacy policy analysis and risk generation for large language model applications\.InProceedings of the 30th International Conference on Intelligent User Interfaces,pp\. 277–297\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1)\.
- M\. Chen, Y\. Li, Y\. Yang, S\. Yu, B\. Lin, and X\. He \(2024\)Automanual: constructing instruction manuals by llm agents via interactive environmental learning\.Advances in Neural Information Processing Systems37,pp\. 589–631\.Cited by:[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- G\. Dagan, F\. Keller, and A\. Lascarides \(2024\)Plancraft: an evaluation dataset for planning with llm agents\.arXiv preprint arXiv:2412\.21033\.Cited by:[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px1.p1.1)\.
- N\. Dainese, M\. Merler, M\. Alakuijala, and P\. Marttinen \(2024\)Generating code world models with large language models guided by monte carlo tree search\.Advances in Neural Information Processing Systems37,pp\. 60429–60474\.Cited by:[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- C\. Diuk, A\. Cohen, and M\. L\. Littman \(2008\)An object\-oriented representation for efficient reinforcement learning\.InProceedings of the 25th international conference on Machine learning,pp\. 240–247\.Cited by:[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- Y\. Hu, Y\. Wang, and J\. McAuley \(2025a\)Evaluating memory in llm agents via incremental multi\-turn interactions\.arXiv preprint arXiv:2507\.05257\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- Y\. Hu, S\. Liu, Y\. Yue, G\. Zhang, B\. Liu, F\. Zhu, J\. Lin, H\. Guo, S\. Dou, Z\. Xi,et al\.\(2025b\)Memory in the age of ai agents\.arXiv preprint arXiv:2512\.13564\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- T\. Kagaya, T\. J\. Yuan, Y\. Lou, J\. Karlekar, S\. Pranata, A\. Kinose, K\. Oguri, F\. Wick, and Y\. You \(2024\)Rap: retrieval\-augmented planning with contextual memory for multimodal llm agents\.arXiv preprint arXiv:2402\.03610\.Cited by:[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- K\. Kansky, T\. Silver, D\. A\. Mély, M\. Eldawy, M\. Lázaro\-Gredilla, X\. Lou, N\. Dorfman, S\. Sidor, S\. Phoenix, and D\. George \(2017\)Schema networks: zero\-shot transfer with a generative causal model of intuitive physics\.InInternational conference on machine learning,pp\. 1809–1818\.Cited by:[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- Z\. Khan, A\. Prasad, E\. Stengel\-Eskin, J\. Cho, and M\. Bansal \(2025\)One life to learn: inferring symbolic world models for stochastic environments from unguided exploration\.arXiv preprint arXiv:2510\.12088\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p2.1)\.
- C\. Lam, J\. Li, L\. Zhang, and K\. Zhao \(2026\)Governing evolving memory in llm agents: risks, mechanisms, and the stability and safety governed memory \(ssgm\) framework\.arXiv preprint arXiv:2603\.11768\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- X\. Ning, K\. Tieu, D\. Fu, T\. Wei, Z\. Li, Y\. Bei, J\. Zou, M\. Ai, Z\. Liu, T\. Li,et al\.\(2026\)Code as agent harness\.arXiv preprint arXiv:2605\.18747\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p2.1)\.
- OpenAI \(2025\)Introducing GPT\-4\.1 in the API\.Note:Blog postAccessed: 2026\-05\-14External Links:[Link](https://openai.com/index/gpt-4-1/)Cited by:[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- T\. Piriyakulkij, Y\. Liang, H\. Tang, A\. Weller, M\. Kryven, and K\. Ellis \(2026\)Poe\-world: compositional world modeling with products of programmatic experts\.Advances in Neural Information Processing Systems38,pp\. 26609–26638\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- J\. Shao, H\. Yin, Y\. Lyu, X\. Yu, L\. Guo, I\. Tsang, J\. Kwok, and Y\. Li \(2026\)Lifting traces to logic: programmatic skill induction with neuro\-symbolic learning for long\-horizon agentic tasks\.arXiv preprint arXiv:2605\.01293\.Cited by:[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2020\)Alfworld: aligning text and embodied environments for interactive learning\.arXiv preprint arXiv:2010\.03768\.Cited by:[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px1.p1.1)\.
- H\. Tang, D\. Key, and K\. Ellis \(2024\)Worldcoder, a model\-based llm agent: building world models by writing code and interacting with the environment\.Advances in Neural Information Processing Systems37,pp\. 70148–70212\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§1](https://arxiv.org/html/2607.02846#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- R\. Wang, P\. Jansen, M\. Côté, and P\. Ammanabrolu \(2022\)Scienceworld: is your agent smarter than a 5th grader?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 11279–11298\.Cited by:[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px1.p1.1)\.
- Z\. Z\. Wang, A\. Gandhi, G\. Neubig, and D\. Fried \(2025\)Inducing programmatic skills for agentic tasks\.arXiv preprint arXiv:2504\.06821\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§1](https://arxiv.org/html/2607.02846#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig \(2024\)Agent workflow memory\.arXiv preprint arXiv:2409\.07429\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- Z\. Xiong, Y\. Lin, W\. Xie, P\. He, Z\. Liu, J\. Tang, H\. Lakkaraju, and Z\. Xiang \(2025\)How memory management impacts llm agents: an empirical study of experience\-following behavior\.arXiv preprint arXiv:2505\.16067\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Ye, Z\. Zhang, T\. Ma, Z\. Wang, Y\. Li, S\. Hou, W\. Sun, K\. Shi, Y\. Ma, W\. Song,et al\.\(2025\)Llms4all: a review of large language models across academic disciplines\.arXiv preprint arXiv:2509\.19580\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1)\.
- Z\. Zhang, Q\. Dai, X\. Bo, C\. Ma, R\. Li, X\. Chen, J\. Zhu, Z\. Dong, and J\. Wen \(2025\)A survey on the memory mechanism of large language model\-based agents\.ACM Transactions on Information Systems43\(6\),pp\. 1–47\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024\)Expel: llm agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§1](https://arxiv.org/html/2607.02846#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02846#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.
- H\. Zhao, S\. Zhou, H\. Yang, Z\. Qin, and T\. Zhou \(2026\)Neuro\-symbolic synergy for interactive world modeling\.arXiv preprint arXiv:2602\.10480\.Cited by:[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1)\.
- Y\. Zheng, L\. Zhong, Y\. Wang, R\. Dai, K\. Liu, X\. Chu, L\. Lv, P\. Torr, and K\. Q\. Lin \(2026\)Code2world: a gui world model via renderable code generation\.arXiv preprint arXiv:2602\.09856\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p2.1)\.
- W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang \(2024\)Memorybank: enhancing large language models with long\-term memory\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 19724–19731\.Cited by:[§1](https://arxiv.org/html/2607.02846#S1.p1.1)\.
- S\. Zhou, T\. Zhou, Y\. Yang, G\. Long, D\. Ye, J\. Jiang, and C\. Zhang \(2024\)Wall\-e: world alignment by rule learning improves world model\-based llm agents\.arXiv preprint arXiv:2410\.07484\.Cited by:[§A\.2](https://arxiv.org/html/2607.02846#A1.SS2.p1.1),[§1](https://arxiv.org/html/2607.02846#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.02846#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.02846#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix AExperimental Details

### A\.1Benchmark Details

Table[3](https://arxiv.org/html/2607.02846#A1.T3)summarizes the evaluation splits used in our experiments\. ScienceWorld contains 30 task categories, and we sample up to five test instances from each category\. Theidentify\-life\-stages\-2category contains only four available instances under our filtering, resulting in 149 ScienceWorld tasks in total\. For ALFWorld, we use the out\-of\-distribution evaluation split\. For PlanCraft, we use the test\-small split, which preserves the benchmark’s difficulty structure while keeping online evaluation computationally tractable\. In all benchmarks, episodes are evaluated in the benchmark’s default task order without reordering, curriculum design, or task\-order tuning\.

Table 3:Evaluation splits used in our experiments\.PlanCraft overall results include easy, medium, hard, and impossible tasks\. Impossible tasks are instances where the target item cannot be synthesized from the available inventory and recipes; on these tasks, the episode is counted as successful if the agent correctly uses theimpossibleaction\. PlanCraft exposes five high\-level action types:move,smelt,think,search, andimpossible\. Themoveaction transfers item stacks among inventory, crafting\-grid, and output slots;smeltapplies furnace\-like transformations;searchretrieves recipe information; andimpossiblelets the agent declare that the target item cannot be crafted from the available resources\. Thethinkaction is handled specially: it is parsed by the runner as an internal reasoning action rather than submitted to the environment, so it is not counted as an environment step\.

### A\.2Baseline Details

ReActYaoet al\.\([2022](https://arxiv.org/html/2607.02846#bib.bib19)\)is a standard interactive baseline that alternates between a brief reasoning trace and a primitive environment action\. ReflexionShinnet al\.\([2023](https://arxiv.org/html/2607.02846#bib.bib1)\)augments this loop with a self\-reflection after failed attempts\. Since Reflexion does not maintain cross\-task memory in our setting, we allow it one retry within the same task episode and keep at most three within\-task reflections in context\. ExpeLZhaoet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib2)\)maintains cross\-task textual rules induced from prior trajectories; because this memory is reused across tasks, we evaluate ExpeL without an additional retry\. ExpeL retrieves the two most similar prior experiences, keeps at most 20 learned rules, keeps the three most recent reflections in context, updates rules in batches of eight trajectories, critiques eight successful trajectories per update, and uses task\-similarity retrieval with theall\-mpnet\-base\-v2embedder\. AWMWanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib3)\)stores successful action workflows and retrieves one workflow for later tasks, with one induction pass after each episode\. ASIWanget al\.\([2025](https://arxiv.org/html/2607.02846#bib.bib11)\)induces symbolic task knowledge from experience\. Wall\-EZhouet al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib12)\)and WorldCoderTanget al\.\([2024](https://arxiv.org/html/2607.02846#bib.bib13)\)are included as symbolic\-learning or world\-modeling baselines\. Wall\-E uses one candidate action, mines rules every 100 trajectories, and allows up to three internal retries during rule generation\. WorldCoder uses bandit planning with exploration rate 0\.05, bandit constant 5\.0, and at most three LLM planning steps\. These hyperparameters match the important defaults used by the corresponding open\-source implementations whenever applicable\. For OCM, we allow up to three attempts when parsing reflection plans or repairing generated knowledge updates\.

The original WorldCoder formulation was not directly usable in our environments\. These benchmarks are complex and partially observable, and the agent could not reliably learn effective transition and reward functions from raw interaction alone during decision making\. To obtain a runnable and competitive adaptation, we let the agent synthesize a transition function and a reward function from the trajectory at the end of each episode\. In later episodes, these learned functions are included in the context and used as auxiliary world\-model information for action selection\.

For fairness, all methods use the same backbone model and are evaluated in the same online order\. We do not include benchmark examples, task\-specific shortcuts, or hand\-written benchmark\-specific heuristics in any prompt\. Prompts contain only generic interaction instructions and the action\-format information needed to communicate with each environment\.

### A\.3Prompt Templates

The following boxes reproduce the OCM prompt templates used in our implementation, with runtime values represented as placeholders\.

`OCM Action Prompt OCM Reflection Prompt OCM Knowledge\-Update Prompt`

`Stage Evidence OCM knowledge Effect on 2\-3\_225 Prior episodes Related melting\-point tasks expose repeated partial progress and failures on thermometer use, nested substance access, temperature/state reading, and threshold\-based box selection\. OCM creates MeltSort, a reusable procedure with accessibility checks, temperature/state retrieval, threshold comparison, and conditional focus \(Figures 6–7\)\. The relevant procedure is already available before the later unknown\-substance task begins\. Target episode The task asks whether solid unknown substance Y has melting point above 200\.0 degrees Celsius\. All seven baselines fail on the same instance\. During action selection, OCM calls and inspects MeltSort, exposing the decision skeleton in Figure 7 rather than a fixed action script\. The agent decomposes the inspected source into legal actions, heats and re\-measures the substance, observes 390 degrees Celsius, and focuses on the blue box\. Table 4: Evidence that OCM reuses previously learned procedural knowledge on a later ScienceWorld task where all baselines fail\.``A\.4 Additional Case Study: Reusing Learned Procedural Knowledge We present an additional ScienceWorld case that illustrates how OCM reuses knowledge accumulated from earlier episodes\. Before episode 2\-3\_225, the agent had interacted with several related melting\-point tasks\. Those earlier trajectories did not all end in success, but they provided reusable evidence about the task structure: the agent must focus on a thermometer, access the target substance even when it is nested in a container, measure or induce its temperature, compare the result against a threshold, and then focus on the correct destination box\. From these experiences, OCM wrote a procedural entry for measuring and sorting a substance by melting point; we refer to this entry as MeltSort below\. Figures 6 and 7 show the object interface and procedural excerpt used by this case\. The later task 2\-3\_225 asks the agent to measure the melting point of solid unknown substance Y and focus on the blue box if the measured temperature is above 200\.0 degrees Celsius, otherwise on the orange box\. This task is difficult for baselines: ReAct, Reflexion, ExpeL, AWM, ASI, WorldCoder, and Wall\-E all fail on the same instance, while OCM succeeds with reward 100\. During decision making, OCM calls the learned melting\-point procedure and inspects its source\. The procedure itself is not an environment action, so the agent then converts the inspected procedure into valid ScienceWorld actions: it focuses on the thermometer and substance, picks up the thermometer, uses it on the substance, opens the oven, moves the substance into the oven, activates the oven, waits, measures again, observes a temperature of 390 degrees Celsius, and focuses on the blue box\. The case shows that the learned procedure is useful not as a script to copy verbatim, but as an executable and inspectable model of the hidden task logic\. Object Knowledge Interface Used by the Procedure Figure 6: Excerpt of the object knowledge used by the melting\-point procedure\. The full object\_knowledge\.py contains additional unrelated methods; here we retain the state, temperature, focus, inventory, and container\-accessibility interfaces needed for this case\. Learned Procedural Knowledge Figure 7: Excerpt of the learned procedural code used by OCM for measuring and sorting a substance by melting point\. The full procedure contains longer docstrings and additional comments; this excerpt preserves the executable control flow and calls into the object interface in Figure 6\. In 2\-3\_225, the procedure arguments bind to the thermometer, solid unknown substance Y, the blue box, the orange box, and threshold 200\.0\. The inspected code therefore gives the agent the correct decision skeleton, while the online trajectory supplies the concrete environment actions needed to realize it\.`