MemPrism: Task-Conditioned Relational Memory Views for Long-Horizon Agents

# MemPrism: Task-Conditioned Relational Memory Views for Long-Horizon Agents
Source: [https://arxiv.org/html/2608.06745](https://arxiv.org/html/2608.06745)
\\setheadertext

Preprint\\setheadertitleMemPrism: Task\-Conditioned Relational Memory Views for Long\-Horizon Agents\\correspondingemail∗\*Equal Contribution†\\daggerCorresponding Author\. \\emailiconzhisheng\.researcher@gmail\.com\\githublinkhttps://github\.com/Feld\-maxiu/MemPrism

Bingfan Zeng2,\*Bangde Cao3,\*Zhengwei Xie4Yuxuan Li1Jinhan Li4Zheng Lu5Xiangchen Guan5Zikai Xiao6Rui Qian7,†\\daggerJingwei Song8,†\\dagger 1Nanyang Technological University2South China University of Technology3Beijing University of Posts and Telecommunications 4University of Science and Technology of China5Peking University6Zhejiang University7Fudan University8Shanghai Jiao Tong University

###### Abstract

Long\-horizon agents rely on memory to reuse experiences, yet existing memory systems often assume that evidence can be directly consumed through a fixed representation\. This leads torepresentation mismatch, where relevant information is available but not organized for the current decision\. To this end, we proposeMemPrism, a task\-conditioned relational memory framework that separates persistent experience storage from decision\-time working memory\. MemPrism records interactions as the event stream and dynamically constructs relational views according to the current task context\. A lightweight view policy selects the relation structure, evidence range, outcome condition, and granularity, while a deterministic composer and render transform historical facts into a temporary optical working\-memory view for a frozen task policy\. Experiments on long\-horizon embodied and web\-agent benchmarks show that MemPrism consistently improves the task performance, especially as trajectories become longer, while reducing memory token consumption\. Furthermore, the learned view policy transfers across different VLMs without additional adaptation, demonstrating the effectiveness of task\-conditioned relational views as a general memory interface for agents\.

## Introduction

For long\-horizon agents, past experience becomes useful memory only when it is organized to support the current decision\. This challenge is becoming increasingly important as Large language model \(LLM\) and vision\-language model \(VLM\) based agents tackle web navigationzhou2024webarena, software engineeringyang2024swe, tool useqin2023toolllm, embodied interaction, and open\-world explorationwang2023voyager, where trajectories often span tens or even hundreds of steps\. At this scale, success depends not only on local reasoning, but also on how effectively the agent can structure and use its growing interaction history\.

![Refer to caption](https://arxiv.org/html/2608.06745v1/Figures/Difference.jpg)Figure 1:Conceptual comparison of different memory paradigms for long\-horizon agents\. Existing approaches improve memory organization, control, or representation medium, but generally provide history in a fixed form for downstream decisions\. In contrast, MemPrism preserves stable interaction facts as an event stream and dynamically constructs task\-conditioned relational views, allowing the same history to support different decision requirements\.Recent work has advanced agent memory along three complementary directions\. One line organizes raw trajectories into persistent structures, such as hierarchical summarieslee2024human, knowledge graphsanokhin2024arigraph, and fact storesxu2026mem\. Another makes memory control itself learnable by treating writing, updating, deletion, and retrieval as part of the agent policyyu2026agentic\. A third revisits the representation medium, optical memory renders long interaction histories as compact images to reduce token costfeng2026reimagining\. These directions improve how memory is stored, managed, and accessed\. However, they often retain a common assumption: once relevant evidence has been preserved and retrieved, it can be delivered to the policy through a fixed representationyao2022react;park2023generative;wang2023augmenting\.

However, a unique representation is not equally useful for every decisionzeng2024structural\. The same history may need to be organized in different ways for different tasks\. Loop detection benefits from aligning repeated actions with their outcomes\. State tracking requires changes to the same entity to be grouped over time\. Failure analysis links actions to environment feedback, while dependency analysis connects related events that may be far apart in the trajectory\. The timeline, state table, action\-effect view, and dependency trace may all be built from the same facts, yet each highlights a different type of evidence\.

As a result, an agent may still fail even when the correct evidence has been stored and retrievedhe2026memoryarena\. We call this failure*representation mismatch*\. It occurs when the available evidence is not organized in the relation form needed by the current subtask\. This failure is different from storage failure and retrieval failure\. It lies at the read\-time interface between retrieved evidence and the task policy and also reveals a key observation in long\-horizon memory:

Does access to the right evidence guarantee that it will be useful for the current decision?

This observation suggests a principle: persistent memory should preserve stable and reusable interaction factshu2025hiagent, while working memory should be constructed for the current decision\. Rather than relying on a fixed representation, the agent should reorganize the same history according to the current observation, task goal, and execution state\. Therefore, effective memory requires learning not only what to store and retrieve but how to present retrieved evidence to the policy\.

To this end, we proposeMemPrism, a task\-conditioned relational working\-memory framework\. MemPrism records real interactions in a unified event stream\. At each step, a lightweight view policy selects the relation type, evidence range, outcome filter, and level of detail\. A deterministic pipeline then constructs a task\-conditioned view from the event stream, which a frozen VLM uses to produce the next action\. This view is temporary and serves only the current decision; it is never written back to persistent memory\. Only executed actions and the resulting environment feedback are appended to the event stream\.

MemPrism uses optical views as a unified output format, not only to compress history but also to make relations explicit\. Its two\-dimensional layout encodes structure directly: spatial proximity groups related actions, row and column alignment supports state comparison, arrows indicate change or dependency, and highlights draw attention to failures and task\-relevant evidence\. Thus, optical memory serves not merely as a compact representation but as a structured interface for task\-conditioned relations\.

Our main contributions are as follows:

- •We formulate post\-retrieval representation mismatch\.We distinguish this failure from missing storage or failed retrieval: the required evidence is available, but its representation does not expose the relations needed for the current decision\. We therefore formulate the construction of task\-specific relational views as an explicit working\-memory decision problem\.
- •We decouple persistent history from temporary working views\.MemPrism stores interactions in a unified and reusable event stream, while constructing a new working view on demand at each decision step\. This separation avoids irreversible distortion caused by fixed summaries, repeated compression, or previously generated views\.
- •We introduce a relational optical view space\.Each view is jointly defined by its relation type, evidence range, outcome condition, and level of detail\. Two\-dimensional layouts explicitly represent temporal order, action\-outcome relations, entity\-state changes, and local dependencies\.
- •We introduce a task\-conditioned view policy\.We initialize the policy with action\-conditioned soft supervision and further optimize it with trajectory\-level grouped GRPO, which isolates the effect of view selection and allows performance gains to be more directly attributed to the memory representation interface\.

## Related Work

### Structured Memory for Long\-Horizon Agents

To reduce the cost of growing interaction histories, many methods convert raw trajectories into compact persistent structurespacker2023memgpt;lee2024human;zhong2024memorybank\. HiAgent builds hierarchical working memory around subgoals\. AriGraph organizes episodic events and semantic knowledge in a dynamic graphanokhin2024arigraph\. A\-MEM builds an evolving memory network with indices, tags, and linksxu2026mem\. These methods improve history organization and accessjiang2026magma\. However, their relation structures are mainly formed during memory writing or maintenance, and are then shared across later decisionszeng2024structural;park2023generative;wang2023augmenting\. In contrast, MemPrism stores normalized events and builds relation views only when they are needed\.

### Learnable and State\-Adaptive Memory Control

Recent work treats memory control as a learnable policyzhao2024expel;shinn2023reflexion\. Memory\-R1 uses reinforcement learning to optimize memory writing, updating, deletion, and useyan2026memory\. AgeMem represents long\-term and short\-term memory operations as callable actions, and learns when to store, retrieve, summarize, or forgetyu2026agentic\. Other methods make memory access depend on the current agent state\. MemCompiler selects cross\-task experience from a structured state and converts it into text and latent guidance\. SAM keeps compact cues to raw trajectories and restores distant history based on the current intentji2026memory\. These methods learn which memory action to take, which evidence to access, or which guidance to produce\. MemPrism studies a different stage: how the same evidence should be organized before it is given to the task policy\.

### Optical and Visual Memory Representation

Optical memory converts long text histories into two\-dimensional imagesli2026ocr\. AgentOCR renders observation\-action traces as compact images and learns a dynamic compression ratefeng2026reimagining\. MemOCR further assigns visual space based on layout and information importanceshi2026memocr\. These methods focus on storing or compressing history more efficiently\. MemPrism instead uses two\-dimensional layout to express relation semantics\. Spatial proximity, alignment, arrows, and highlights expose action effects, entity states, temporal change, and local dependencies\. Thus, optical memory in MemPrism is not only a compact medium\. It is the output space of a task\-conditioned relational memory interface\.

## Method

![Refer to caption](https://arxiv.org/html/2608.06745v1/Figures/Memprism.jpg)Figure 2:Overview of MemPrism, a task\-conditioned relational working\-memory framework\. MemPrism separates persistent history from decision\-time representations: Recorder maintains a unified event stream, Router selects the required relation view, Composer constructs a task\-specific organization from the same facts, and Render produces a temporary optical working memory for the frozen task VLM\. Adapter further learns view selection through action\-conditioned supervision and grouped GRPO while keeping the remaining modules frozen\.### Overview

MemPrism addresses post\-retrieval representation mismatch in long\-horizon agents\. It separates persistent interaction records from the working memory used at each decision step\. The framework contains five modules:Recorder,Router,Composer,Render, andAdapter\. The first four modules form the inference pipeline\. Adapter is used only during training\.

Given a task goalgg, the agent receives an observationoto\_\{t\}at steptt\. Recorder stores all interactions as an event stream

ℰ<t=\(e~1,…,e~t−1\)\.\\mathcal\{E\}\_\{<t\}=\(\\tilde\{e\}\_\{1\},\\ldots,\\tilde\{e\}\_\{t\-1\}\)\.\(1\)Router selects a view action from the current decision state\. Composer selects events and builds a relational structure\. Render converts this structure into an optical working\-memory viewVtV\_\{t\}\. A frozen task policy reads the current input andVtV\_\{t\}to produce an environment action\. Only the executed action and its feedback are appended to the event stream\. The temporary view is discarded after the current step\.

### Recorder: Persistent Event Memory

Recorder stores each real interaction in raw and structured forms\. A raw event is

et=\(ot−,ut,yt\+,rt,mt\),e\_\{t\}=\(o\_\{t\}^\{\-\},u\_\{t\},y\_\{t\}^\{\+\},r\_\{t\},m\_\{t\}\),\(2\)whereot−o\_\{t\}^\{\-\}is the observation before action execution,utu\_\{t\}is the executed action,yt\+y\_\{t\}^\{\+\}is the feedback,rtr\_\{t\}is the reward or step label, andmtm\_\{t\}contains task and environment metadata\.

An event extractorΦ\\Phi, together with rule\-based checks, maps the raw event to

e~t=\(t,etent,etact,etout,Δ​st,ξt\),\\tilde\{e\}\_\{t\}=\(t,e\_\{t\}^\{\\mathrm\{ent\}\},e\_\{t\}^\{\\mathrm\{act\}\},e\_\{t\}^\{\\mathrm\{out\}\},\\Delta s\_\{t\},\\xi\_\{t\}\),\(3\)whereetente\_\{t\}^\{\\mathrm\{ent\}\},etacte\_\{t\}^\{\\mathrm\{act\}\}, andetoute\_\{t\}^\{\\mathrm\{out\}\}denote the main entity, normalized action type, and outcome type\.Δ​st\\Delta s\_\{t\}is a sparse set of state updates\.ξt\\xi\_\{t\}groups optional evidence, such as tags, images, boxes, and entity crops\. The outcome type is one of exception, state update, or no observed change\.

Recorder keeps the latest observed value of each state key\. It does not require the extractor to infer old values\. After the environment returns the result ofutu\_\{t\}, the event stream is updated by

ℰ≤t=ℰ<t⊕e~t,\\mathcal\{E\}\_\{\\leq t\}=\\mathcal\{E\}\_\{<t\}\\oplus\\tilde\{e\}\_\{t\},\(4\)where⊕\\oplusdenotes temporal append\. Relational structures and rendered views are never written back to the event stream\.

### Router: Decision\-Conditioned View Routing

Router decides how the agent should view its history\. It does not produce an environment action\. Its input contains the current observation, the task goal, recent\-event features, and global history statistics\. These inputs are encoded as

zt=Encθ⁡\(ot,g,Summary⁡\(ℰ<t\)\)\.z\_\{t\}=\\operatorname\{Enc\}\_\{\\theta\}\\bigl\(o\_\{t\},g,\\operatorname\{Summary\}\(\\mathcal\{E\}\_\{<t\}\)\\bigr\)\.\(5\)
Router selects a joint view action

atv=\(τt,wt,ct,γt\)∈𝒜v,a\_\{t\}^\{v\}=\(\\tau\_\{t\},w\_\{t\},c\_\{t\},\\gamma\_\{t\}\)\\in\\mathcal\{A\}^\{v\},\(6\)whereτt\\tau\_\{t\},wtw\_\{t\},ctc\_\{t\}, andγt\\gamma\_\{t\}denote the relation type, temporal range, outcome filter, and rendering granularity\.

FactorOptionsRelationτ\\tauTemporal, Effect, State, DependencyRangewwShort, Long, AllOutcomeccAll, Exception, Update, No\-changeGranularityγ\\gammaCoarse, Medium, FineTable 1:Four factors of the joint view action\.The two recent ranges contain the latest 6 and 10 events\. The joint action space is

𝒜v=𝒯×𝒲×𝒪×𝒢,\|𝒜v\|=144\.\\mathcal\{A\}^\{v\}=\\mathcal\{T\}\\times\\mathcal\{W\}\\times\\mathcal\{O\}\\times\\mathcal\{G\},\\qquad\|\\mathcal\{A\}^\{v\}\|=144\.\(7\)
Router predicts one distribution over all 144 actions

ℓt=fθ\(zt\),πθv\(a∣zt\)=softmax\(ℓt\)a\.\\ell\_\{t\}=f\_\{\\theta\}\(z\_\{t\}\),\\qquad\\pi\_\{\\theta\}^\{v\}\(a\\mid z\_\{t\}\)=\\operatorname\{softmax\}\(\\ell\_\{t\}\)\_\{a\}\.\(8\)A joint head keeps dependencies between the four factors\. During training, actions are sampled from the policy\. During evaluation, Router uses

atv=arg⁡maxa∈𝒜v⁡πθv​\(a∣zt\)\.a\_\{t\}^\{v\}=\\arg\\max\_\{a\\in\\mathcal\{A\}^\{v\}\}\\pi\_\{\\theta\}^\{v\}\(a\\mid z\_\{t\}\)\.\(9\)

### Composer: Relational View Construction

Composer converts the selected view action into a relational structure\. It first selects events by the temporal range, outcome filter, and task goal

ℰ^t=Select⁡\(ℰ<t,wt,ct,g\)\.\\widehat\{\\mathcal\{E\}\}\_\{t\}=\\operatorname\{Select\}\(\\mathcal\{E\}\_\{<t\},w\_\{t\},c\_\{t\},g\)\.\(10\)The selection step also restores task\-related events that may be removed by a strict filter\. For the dependency view, it keeps the required recent context\.

Composer then builds a relation\-specific structure

St=𝒞τt,γt​\(ℰ^t\)\.S\_\{t\}=\\mathcal\{C\}\_\{\\tau\_\{t\},\\gamma\_\{t\}\}\(\\widehat\{\\mathcal\{E\}\}\_\{t\}\)\.\(11\)
##### TemporalTrace\.

Events are ordered by time\. This view shows action order and task progress\.

##### ActionEffect\.

Events are grouped by action type and entity\. This view aligns repeated actions with their outcomes and makes failed loops easier to detect\.

##### EntityState\.

State updates are grouped by entity and state key\. This view shows how an entity state changes over time\.

##### DependencyChain\.

The latest event is used as an anchor\. Composer retrieves events that share entities, state keys, or nearby time steps\. This view gives a local dependency trace\. It is not treated as a verified causal graph\.

The granularityγt\\gamma\_\{t\}controls the number of entries and the amount of detail\. It does not change the stored events\.

### Render: Optical Working Memory

Render maps the relational structure to an optical working\-memory view

Vt=ℛτt,γt​\(St\)\.V\_\{t\}=\\mathcal\{R\}\_\{\\tau\_\{t\},\\gamma\_\{t\}\}\(S\_\{t\}\)\.\(12\)Render uses a fixed visual grammar\. Spatial proximity denotes event groups\. Rows and columns support state comparison\. Cards align actions and outcomes\. Arrows show state changes or local dependencies\. Highlights mark failures and task\-related evidence\. In visual environments, event images, boxes, and crops can also be added\.

Render is deterministic\. The same events, view action, and rendering settings always produce the same view\. The frozen task policy then produces an environment action as

ut∼πtask\(⋅∣Xt,g,Htshort,Vt\),u\_\{t\}\\sim\\pi\_\{\\mathrm\{task\}\}\\bigl\(\\cdot\\mid X\_\{t\},g,H\_\{t\}^\{\\mathrm\{short\}\},V\_\{t\}\\bigr\),\(13\)whereXtX\_\{t\}is the current environment input andHtshortH\_\{t\}^\{\\mathrm\{short\}\}is the shared short\-term context\.

### Adapter: View Policy Learning

Adapter trains Router in two stages\. The task policy, event extractor, Composer, and Render remain frozen in both stages\.

##### Decision\-conditioned initialization\.

For an offline decision state, a view teacher receives the current observation, task goal, event history, and reference next actionutrefu\_\{t\}^\{\\mathrm\{ref\}\}\. It returns a soft distribution over view actions

qT​\(a∣t\)=qT​\(a∣ot,g,ℰ<t,utref\)\.q\_\{T\}\(a\\mid t\)=q\_\{T\}\(a\\mid o\_\{t\},g,\\mathcal\{E\}\_\{<t\},u\_\{t\}^\{\\mathrm\{ref\}\}\)\.\(14\)Router is initialized with

ℒSFT=\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=KL\(qT\(⋅∣t\)∥πθv\(⋅∣zt\)\)\\displaystyle\\;\\operatorname\{KL\}\\bigl\(q\_\{T\}\(\\cdot\\mid t\)\\,\\\|\\,\\pi\_\{\\theta\}^\{v\}\(\\cdot\\mid z\_\{t\}\)\\bigr\)\(15\)\+λC​∑a∈𝒜vπθv​\(a∣zt\)​C^t​\(a\),\\displaystyle\+\\lambda\_\{C\}\\sum\_\{a\\in\\mathcal\{A\}^\{v\}\}\\pi\_\{\\theta\}^\{v\}\(a\\mid z\_\{t\}\)\\widehat\{C\}\_\{t\}\(a\),whereC^t​\(a\)\\widehat\{C\}\_\{t\}\(a\)is the structural cost of a view action\. The cost term discourages large ranges or fine views when they are not needed\.

##### Outcome\-guided optimization\.

Teacher supervision is based on reference trajectories and may not cover states produced by the deployed task policy\. We therefore refine Router with grouped GRPO in online environments\.

For each task, we sampleGGtrajectories\. LetRiR\_\{i\}be the return of trajectoryii, and letμg\\mu\_\{g\}andσg\\sigma\_\{g\}be the mean and standard deviation within the task group\. The trajectory advantage is

Ai=\{Ri−μgσg\+ε,σg\>ε,0,otherwise\.A\_\{i\}=\\begin\{cases\}\\dfrac\{R\_\{i\}\-\\mu\_\{g\}\}\{\\sigma\_\{g\}\+\\varepsilon\},&\\sigma\_\{g\}\>\\varepsilon,\\\\\[4\.0pt\] 0,&\\text\{otherwise\}\.\\end\{cases\}\(16\)All view decisions in one trajectory shareAiA\_\{i\}\. Letπoldv\\pi\_\{\\mathrm\{old\}\}^\{v\}be the rollout policy and define

ρi,t=πθv​\(ai,tv∣zi,t\)πoldv​\(ai,tv∣zi,t\)\.\\rho\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}^\{v\}\(a\_\{i,t\}^\{v\}\\mid z\_\{i,t\}\)\}\{\\pi\_\{\\mathrm\{old\}\}^\{v\}\(a\_\{i,t\}^\{v\}\\mid z\_\{i,t\}\)\}\.\(17\)
Adapter updates Router with

ℒGRPO=\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}=−𝔼i,t​\[min⁡\(ρi,t​Ai,clip⁡\(ρi,t,1−ϵ,1\+ϵ\)​Ai\)\]\\displaystyle\-\\mathbb\{E\}\_\{i,t\}\\left\[\\min\\left\(\\rho\_\{i,t\}A\_\{i\},\\operatorname\{clip\}\(\\rho\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)A\_\{i\}\\right\)\\right\]\(18\)\+βKL𝔼i,t\[KL\(πrefv\(⋅∣zi,t\)∥πθv\(⋅∣zi,t\)\)\]\\displaystyle\+\\beta\_\{\\mathrm\{KL\}\}\\mathbb\{E\}\_\{i,t\}\\left\[\\operatorname\{KL\}\\bigl\(\\pi\_\{\\mathrm\{ref\}\}^\{v\}\(\\cdot\\mid z\_\{i,t\}\)\\,\\\|\\,\\pi\_\{\\theta\}^\{v\}\(\\cdot\\mid z\_\{i,t\}\)\\bigr\)\\right\]−βH𝔼i,t\[ℋ\(πθv\(⋅∣zi,t\)\)\]\.\\displaystyle\-\\beta\_\{H\}\\mathbb\{E\}\_\{i,t\}\\left\[\\mathcal\{H\}\\bigl\(\\pi\_\{\\theta\}^\{v\}\(\\cdot\\mid z\_\{i,t\}\)\\bigr\)\\right\]\.Here,πrefv\\pi\_\{\\mathrm\{ref\}\}^\{v\}is the policy after supervised initialization\. The KL term limits large policy changes, and the entropy term keeps enough exploration\. Only Router is updated\. Thus, trajectory rewards are used to learn how history should be presented without changing the task policy\.

## Experiments

### Experiment Setup

##### Benchmarks

We evaluate MemPrism on three benchmarks\.ALFWorldshridhar2020alfredis a text\-based embodied environment spanning six household activity categories, with 3,827 training instances and 140 evaluation games; we report task success rate \(SR\)\.EB\-ALFREDis a visual embodied benchmark built on ALFREDshridhar2020alfredand evaluated through the EmbodiedBenchyang2025embodiedbenchwrapper, providing RGB and text observations across 300 tasks in six subsets \(Base, Common, Complex, Visual, Spatial, Long; 50 each\); we report per\-subset success rate and average SR\. Results use EmbodiedBench rather than the official ALFRED leaderboard and should not be compared directly\.Mind2Webdeng2023mind2webis an offline web\-agent benchmark with 1,009 training tasks across 137 websites and 31 domains, with three test splits \(Cross\-Task / 252, Cross\-Website / 177, Cross\-Domain / 912\) measuring generalization; the model predicts CLICK, TYPE, or SELECT under teacher\-forced replay and the next state always follows the human trajectory\. We report step\-level action accuracy and exact episode success\.

##### Baselines

We compare MemPrism against the following methods\.No MemoryandFull Text Historyset lower and upper text\-memory bounds\.LangMemmaintains working and long\-term memory via episodic observation\-thought\-action chains\.A\-Memxu2026memuses Zettelkasten\-style atomic notes with dynamic linking and memory evolution\.Mem0andMem0gchhikara2025mem0extract facts via LLM calls and retrieve top\-kksimilar memories \(Mem0gadditionally employs a Neo4j entity\-relation graph\)\.SFT on CFG Datayang2025embodiedbenchfine\-tunes the task VLM on CFG\-Bench, a fine\-grained action understanding benchmark spanning physical interaction, temporal\-causal relation, intentional understanding, and evaluative judgment\.MemPrism\-SFTuses supervised view distillation;MemPrism\-SFT\+GRPOadds online grouped GRPO on ALFWorld and EB\-ALFRED\. For mechanism analysis, we compare Recent\-4 Text, Full Text History, four fixed relation views \(ActionEffect, DependencyChain, EntityState, and TemporalTrace\), dynamically selected text and optical views, and planners with one action\-space factor fixed; these variants are not treated as external baselines\.

##### Implementation

The task VLM is Qwen2\.5\-VL\-7B\-Instructwu2025qwen, kept frozen across all experiments\. The textual event extractor \(Phi\) uses Qwen3\.5\-2B across all three benchmarks\.

The view planner encodes current observation, task goal, window statistics \(last 8 events\), and global statistics \(last 64 events\) through a frozen all\-MiniLM\-L6\-v2 encoder, projected to a shared dimension, and fed into a lightweight Transformer followed by a 3\-layer residual MLP head that outputs 144 joint view logits\. The action space spans 4 view types×\\times3 temporal windows×\\times4 outcome filters×\\times3 granularities\. Evaluation uses greedy view selection\.

For all benchmarks, training uses 8 task groups per iteration with 16 trajectories each \(128 total\)\. ALFWorld allows a maximum of 50 environment steps with binary terminal success as reward\. EB\-ALFRED uses a maximum of 30 steps with a reward of success\+\+0\.2×\\timesgoal\-condition progress\. Both use clippingϵ=0\.2\\epsilon=0\.2, reference KL coefficientβKL=0\.2\\beta\_\{\\text\{KL\}\}=0\.2, learning rate3×10−63\\times 10^\{\-6\}, and one update epoch per iteration; ALFWorld additionally uses an entropy bonusβH=0\.003\\beta\_\{H\}=0\.003\. Mind2Web uses exact action match as the reward signal\.

### Main Results

We compare MemPrism with existing memory systems\. Recent\-4 Text, Full Text History, fixed views, and constrained planner variants are reserved for the analysis in Section 4\.3\.

#### ALFWorld and Mind2Web

ALFWorldMind2WebMethodOverall SRTest\-Task Act\. Acc\.Test\-Website Act\. Acc\.Test\-Domain Act\. Acc\.Overall Act\. Acc\.Full history31\.4312\.577\.957\.918\.79LangMem38\.2713\.208\.858\.429\.45A\-Mem34\.6810\.857\.628\.088\.35Mem037\.4911\.427\.357\.288\.12Mem0g35\.3411\.087\.127\.057\.92MemPrism\(SFT\)34\.2912\.9510\.0510\.8510\.81MemPrism\(SFT\+GRPO\)40\.7114\.2311\.1312\.8312\.87

Table 2:Main results on ALFWorld and Mind2Web\. SR denotes exact episode success and Act\. Acc\. denotes step\-level action accuracy\.Table[2](https://arxiv.org/html/2608.06745#Sx4.T2)reports results on ALFWorld and Mind2Web\. On ALFWorld, MemPrism\-SFT\+GRPO achieves 40\.71% ,outperforming all compared methods and exceeding the strongest baseline LangMem \(38\.27%\) by 2\.44 points; GRPO improves over SFT by 6\.42 points\. On Mind2Web, MemPrism\-SFT\+GRPO attains 12\.87% overall action accuracy, 4\.08 points above Full History \(8\.79%\)\. The four dialogue\-oriented memory methods perform comparably to Full History \(7\.92%–9\.45%\), indicating that their fact extraction and vector retrieval mechanisms do not confer additional gains on web interaction tasks, further underscoring the necessity of MemPrism’s relation\-aware organization in this setting\.

MethodBaseCommonComplexVisualSpatialLongAvg\. SRNo Memory10\.08\.06\.02\.00\.02\.04\.7Full history16\.010\.014\.00\.010\.012\.010\.3SFT on CFG Data16\.016\.08\.08\.04\.06\.09\.7A\-Mem8\.010\.012\.00\.06\.06\.07\.0Mem016\.016\.014\.00\.010\.012\.011\.7Mem0g14\.016\.016\.00\.08\.012\.011\.0MemPrism\(SFT\)20\.020\.022\.00\.016\.016\.015\.7MemPrism\(SFT\+GRPO\)24\.022\.026\.00\.016\.018\.017\.7

Table 3:Main results on EB\-ALFRED\.Table[3](https://arxiv.org/html/2608.06745#Sx4.T3)reports results on EB\-ALFRED\. No Memory yields only 4\.7% average SR, confirming the difficulty of visual embodied tasks without memory\. SFT on CFG Data achieves 9\.7% SR without memory, nearly doubling No Memory, indicating that fine\-grained action understanding fine\-tuning strengthens the VLM’s task comprehension\. Full History raises SR to 10\.3%\. Among the three external memory baselines, A\-Mem underperforms at 7\.0% SR, while Mem0 reaches 11\.7% and Mem0greaches 11\.0%, both comparable to Full History but below MemPrism\. MemPrism\-SFT achieves 15\.7% SR, already exceeding all baselines\. GRPO further improves this to 17\.7%, yielding a gain of 7\.4 SR points over Full History\.

### Where Does the Gain Come From?

![Refer to caption](https://arxiv.org/html/2608.06745v1/x1.png)Figure 3:ALFWorld mechanism analysis\. Top row: memory scope and organization×\\timespresentation ablation—Fixed Text and Fixed Optical share the same configuration and differ only in presentation medium; Dynamic Text and Dynamic Optical apply the same contrast under dynamic organization\. Bottom row: constrained planner factors \(each variant fixes one factor while keeping the others selectable\) and effect of planner learning and optimization \(frequency\-matched random→\\rightarrowSFT→\\rightarrowSFT\+GRPO\)\.Figure[3](https://arxiv.org/html/2608.06745#Sx4.F3)consolidates four ablations that answer a single question: which design choices produce the final gain?

Memory scope and relation structure\.The four fixed relation views use identical evidence windows \(recent long \+ medium \+ all\), yet SR varies from 30\.00% to 35\.71%, with TemporalTrace as the strongest at 35\.71%\. Dynamic MemPrism further adds 5\.00 points to reach 40\.71%\. The 5\.71\-point maximum gap under identical evidence directly validates the representation mismatch hypothesis: relevant information is already present in the history, but it is not organized in a way that serves the current decision—different relation structures expose information of different quality to the policy, ultimately leading to different success rates\.

Dynamic organization and optical presentation\.Fixed Text and Fixed Optical share the same TemporalTrace \+ recent long \+ medium \+ all configuration, differing only in presentation medium: 30\.00% vs\. 35\.71% \(\+5\.71\)\. Introducing dynamic organization on text raises SR to 37\.14% \(\+7\.14\), and further adding optical presentation reaches 40\.71%\. Optical presentation independently contributes 3\.57 points, dynamic organization contributes 5\.00 points, and their complementary combination yields the best result\.

Planner factors\.Each constrained variant fixes exactly one factor to a specified value: View Type to TemporalTrace, Outcome Filter to all, Granularity to fine, or Temporal Range to recent long; the remaining factors are still selected by the planner\. The full planner reaches 40\.70% in this export\. Fixing fine granularity causes the largest observed reduction, to 32\.86% \(\-7\.84 points\), followed by fixing TemporalTrace as the view type at 35\.71% \(\-4\.99\), recent long as the temporal range at 38\.57% \(\-2\.13\), and all as the outcome filter at 40\.00% \(\-0\.70\)\. This ranking indicates that adaptive granularity is the most consequential factor, while a fixed all\-outcome filter is comparatively close to the full policy\.

Planner learning and optimization\.Frequency\-Matched Random samples view configurations according to the learned planner’s overall configuration frequencies, but does not condition its choices on the current decision context, achieving 30\.71% SR\. The context\-conditioned SFT planner improves SR to 34\.29%, indicating that the gain cannot be explained solely by the marginal usage frequencies of different configurations\. Matching the view type, temporal range, granularity, and outcome filter to the current task state helps the frozen task policy access more appropriate historical evidence for each decision\. GRPO further increases SR to 40\.71%\. These results indicate that context\-adaptive selection is effective: supervised distillation provides the planner with an initial context\-conditioned selection capability, while online reinforcement learning further aligns view selection with downstream task success through task\-level feedback\.

### Long\-Horizon Behavior

![Refer to caption](https://arxiv.org/html/2608.06745v1/x2.png)Figure 4:ALFWorld success rate and average prompt tokens over cumulative trajectory\-length thresholds\. Solid circles report SR \(left axis\) and dashed triangles report prompt tokens \(right axis\)\.Figure[4](https://arxiv.org/html/2608.06745#Sx4.F4)demonstrate two clear advantages of MemPrism that become pronounced as trajectory length increases\.

Success rate advantage widens with longer trajectories\.Full History holds a narrow lead at the shortest thresholds \(≤\\leq5 and≤\\leq10 steps, 0\.6–1\.3 points\), but MemPrism overtakes it at≤\\leq15 steps \(24\.3% vs\. 16\.0%\) and never trails thereafter\. The gap continues to widen, reaching 9\.3 points at≤\\leq40 and≤\\leq50 steps \(40\.7% vs\. 31\.4%\)\. Notably, Full History plateaus after≤\\leq30 steps \(gaining only 5\.0 points from≤\\leq30 to≤\\leq50\), while MemPrism gains an additional 9\.3 points over the same span, demonstrating that its advantage compounds as more history accumulates\.

Prompt tokens stabilize while Full History costs escalate\.Full History prompt tokens grow nearly linearly with trajectory length, from 574 at≤\\leq5 steps to 1,201 at≤\\leq50 steps—a 2\.1×\\timesincrease\. In contrast, MemPrism token usage remains tightly bounded between 642 and 852 across all thresholds, with no upward trend beyond≤\\leq15 steps\. At the≤\\leq50\-step threshold, MemPrism achieves a 33\.6% token reduction \(797 vs\. 1,201\) while delivering a 9\.3\-point SR improvement, showing that compiled views compress history effectively without losing decision\-relevant information\.

### Cross\-Model Transfer and Prompt Compression

![Refer to caption](https://arxiv.org/html/2608.06745v1/x3.png)Figure 5:Full History and MemPrism SR across task VLMs, with the within\-model prompt\-token reduction overlaid\. Qwen2\.5\-VL\-7B is the planner’s training\-time task VLM; Qwen3\-VL\-4B, Qwen2\.5\-VL\-3B, and MiniCPM\-V\-2\.6 use the same planner checkpoint without any per\-model adaptation\. Token counts are comparable only within a model\.Figure[5](https://arxiv.org/html/2608.06745#Sx4.F5)establish that MemPrism’s learned view policy transfers effectively to unseen task VLMs, yielding consistent improvements in both success rate and prompt efficiency\.

Universal success rate improvement\.On its training VLM \(Qwen2\.5\-VL\-7B\), MemPrism raises SR from 31\.43% to 40\.71% \(\+9\.28 pp\)\. The same planner, applied zero\-shot to three unseen VLMs, improves every model: Qwen3\-VL\-4B from 39\.17% to 41\.43% \(\+2\.26\), Qwen2\.5\-VL\-3B from 16\.30% to 21\.42% \(\+5\.12\), and MiniCPM\-V\-2\.6 from 14\.17% to 23\.57% \(\+9\.40\)\. The gains are not concentrated at a particular model scale or architecture\.

Substantial prompt compression across all models\.Token reduction ranges from 30\.64% \(Qwen3\-VL\-4B\) to 48\.15% \(MiniCPM\-V\-2\.6\), with the training\-domain model achieving 33\.65%\. In every case, higher SR is achieved with substantially fewer tokens\. MiniCPM\-V\-2\.6 exemplifies this interplay: its extreme visual token density \(1\.8M pixels→\\rightarrowonly 640 tokens\) combined with strong OCR makes full\-history inputs disproportionately verbose, yielding the largest compression margin \(48\.15%\) and absolute SR gain \(\+9\.40 pp\) among all models\. The co\-occurrence of SR gains and token reduction across four models indicates that compiled representations are useful across model families\.

## Conclusion

We presented MemPrism, a task\-conditioned relational memory framework for long\-horizon agents\. Our central finding is that retrieving the correct evidence is not sufficient: how that evidence is organized determines whether it can support the current decision\. Even when the historical facts remain unchanged, different relational views can lead to different outcomes, with dynamic organization and optical presentation contributing complementary benefits\. Motivated by this observation, MemPrism decouples stable event storage from the construction of temporary, decision\-specific working\-memory views\. Experiments further show that its advantages become more pronounced as trajectories grow longer, and that the learned view policy transfers across different VLMs without additional adaptation\. Thus, these results identify post\-retrieval representation mismatch as a fundamental bottleneck in long\-horizon agents and establish task\-conditioned memory organization for effective memory use\.

## References

APPENDIX

## Appendix AReproducibility Statement

### A\.1Artifact and result provenance

The released artifact contains the source code for Recorder, Router, Composer, Render, benchmark adapters, training launchers, and evaluation scripts\. Each reported result is accompanied by a machine\-readable run manifest containing the code revision, working\-tree status, data split, ordered episode list, random seed, model identifiers, checkpoint hashes, prompt version, decoding settings, and output paths\. Rendered working\-memory views are derived artifacts; the persistent event stream and environment trajectory remain the sources of truth\.

### A\.2Models, checkpoints, and common settings

All three benchmarks use the same Router architecture and tensor interface\. Its text backbone is a frozenall\-MiniLM\-L6\-v2\. The trainable checkpoint is aViewPolicyModelwithd\_model=512=512, eight attention heads, four Transformer encoder layers, a 1,024\-dimensional feed\-forward sublayer, and dropout0\.10\.1\. Final evaluation takes the argmax of the 144\-way Router distribution and uses greedy task\-policy decoding\. The run manifest binds each reported result to the exact task\-policy, Phi, MiniLM, and Router model IDs and SHA\-256 checkpoint hashes rather than to local filesystem paths\.

At decision steptt, the benchmark adapter serializes the current state as textxtx\_\{t\}and supplies the task goalgg\. MiniLM tokenizes each string to at most 256 tokens, applies attention\-mask\-aware mean pooling to the final hidden states, andL2L\_\{2\}\-normalizes the result, producingeto,eg∈ℝ384e\_\{t\}^\{o\},e^\{g\}\\in\\mathbb\{R\}^\{384\}\. In parallel,EventWindowSummarizerconverts the most recent eight raw/structured event pairs intout∈ℝ128u\_\{t\}\\in\\mathbb\{R\}^\{128\}\. Its first eight coordinates encode theaction\_frequencyprofile used by all three benchmarks\. Raw actions are canonicalized from fields such asop,action\_type, ortype, and the eight largest normalized signature frequencies are stored in descending order\. The next three coordinates are the exception, state\-update, and no\-observed\-change ratios\. Seven further coordinates record the normalized number and coverage of entities, the delta ratio, the maximum exception and no\-change streaks, and recency\-weighted state\-change and exception densities\. The remaining positions are zero padded to form the fixed 128\-dimensional interface\. This channel summarizes repetition and concentration without assuming a benchmark\-specific structured action taxonomy\. A separate vectorst∈ℝ8s\_\{t\}\\in\\mathbb\{R\}^\{8\}, computed over at most the latest 64 events, contains the exception ratio, state\-update ratio, normalized unique\-entity count, normalized history length, a deterministic unit hash of the most recent entity, normalized terminal exception and no\-change streaks, and recent state\-update density\.

The encoder does not concatenate these heterogeneous features into one flat input\. Four independent linear layers projectetoe\_\{t\}^\{o\},ege^\{g\},utu\_\{t\}, andsts\_\{t\}to 512 dimensions; a learned 512\-dimensional\[CLS\]token is prepended to form the five\-token sequence\[CLS,o,g,w,s\]\[\\mathrm\{CLS\},o,g,w,s\]\. Sinusoidal positional encodings and dropout are applied before the four\-layer pre\-norm Transformer, whose feed\-forward blocks use GELU\. The final\[CLS\]state passes through a512→512→512512\\\!\\rightarrow\\\!512\\\!\\rightarrow\\\!512GELU MLP and LayerNorm to yieldzt∈ℝ512z\_\{t\}\\in\\mathbb\{R\}^\{512\}\. The current training and evaluationViewPolicyModel\.encodepath does not pass the optional CLIP image embedding, so Router selection uses the five tokens above; environment images and compiled optical views are consumed downstream by the frozen task policy\. Finally, three residual MLP blocks with 1,024\-dimensional hidden layers mapztz\_\{t\}to a single joint distribution over all 144 view actions\. The reportedτ\\tau, window/filter, andγ\\gammamarginals are derived from these joint logits rather than trained by separate heads\.

### A\.3Training dataset setup

We use only the official training partitions associated with the three evaluation settings\. ALFWorld distillation and online rollouts draw from its training games after the launcher removes unsupported task variants\. The ALFRED path uses the official ALFRED training annotations for learning and reserves EB\-ALFRED episodes for evaluation\. Mind2Web uses its official training split to construct teacher\-forced human trajectories and action\-conditioned view\-teacher records\. No evaluation episode is used to optimize the Router\.

##### Cold\-start distillation\.

We first train the encoder and joint view policy from the teacher distributionqtq\_\{t\}, before any online policy optimization\. For each decision state, the loss combines distribution matching with the expected structural cost of a view action,KL​\(qt∥πθ\)\+λcost​𝔼πθ​\[C​\(a\)\]\\mathrm\{KL\}\(q\_\{t\}\\\|\\pi\_\{\\theta\}\)\+\\lambda\_\{\\mathrm\{cost\}\}\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\[C\(a\)\]\. The 144\-way action head is trained jointly with the encoder and the three residual Planner blocks; the unused compatibility\-only entity\-hint head remains frozen\. The resulting D1 checkpoint supplies both the online initialization and the fixed reference policy used by the GRPO KL term\.

##### Grouped GRPO training\.

For ALFWorld and the ALFRED training path, trajectories for the same task form a group and their terminal task signals are normalized within that group\. For Mind2Web, each iteration contains eight offline\-replay task groups with 16 trajectories per group, and the binary reward indicates whether the predicted action is correct\. During this phase, the encoder, Phi, compiler, renderer, frozen task policy, and D1 reference policy remain fixed\. Optimization updates only the shared three\-block Planner and its 144\-way joint action head\.

### A\.4Parameter configurations

The common architecture is fixed as described in Section[A\.2](https://arxiv.org/html/2608.06745#A1.SS2); Table[A\.1](https://arxiv.org/html/2608.06745#A1.T1)therefore reports only stage\-specific optimization settings\.

Table A\.1:Training configurations for the shared Router\. The architecture and input tensor shapes are identical across benchmarks; rows below record only stage\-specific optimization settings\.SettingHyperparametersD1, sharedAdamW; learning rate10−410^\{\-4\}; weight decay0\.010\.01; batch size 256; gradient\-norm cap1\.01\.0; seed 7\.D1, ALFWorld50 epochs; cosine schedule; minimum learning\-rate ratio0\.010\.01;λcost=0\.02\\lambda\_\{\\mathrm\{cost\}\}=0\.02; BF16\.D1, ALFRED path20 epochs; cosine schedule; minimum learning\-rate ratio0\.010\.01;λcost=0\\lambda\_\{\\mathrm\{cost\}\}=0; no automatic mixed precision\.D1, Mind2Web50 epochs; cosine schedule; minimum learning\-rate ratio0\.010\.01;λcost=0\.02\\lambda\_\{\\mathrm\{cost\}\}=0\.02; BF16\.GRPO, ALFWorld/ALFRED sharedPlanner train mode; AdamW; learning rate3×10−63\\times 10^\{\-6\}; zero weight decay; clipping coefficient0\.20\.2; KL coefficient0\.20\.2; one update epoch; update batch size 512; gradient\-norm cap1\.01\.0; constant learning rate; rollout temperature1\.01\.0\.GRPO, ALFWorld100 iterations; 8 task groups per iteration; 16 trajectories per group; maximum 50 environment steps; entropy coefficient0\.0030\.003\.GRPO, ALFRED path100 iterations; 8 task groups per iteration; 16 trajectories per group; maximum 30 environment steps; progress\-reward coefficient0\.20\.2; entropy coefficient0\.GRPO, Mind2WebPlanner train mode; learning rate10−610^\{\-6\}; 8 task groups per iteration; 16 trajectories per group;ϵlow=0\.1\\epsilon\_\{\\mathrm\{low\}\}=0\.1,ϵhigh=0\.15\\epsilon\_\{\\mathrm\{high\}\}=0\.15;βKL=0\.01\\beta\_\{\\mathrm\{KL\}\}=0\.01; rollout temperatureT=1\.0T=1\.0; top\-p=0\.85=0\.85; reward𝟏​\[action correct\]\\mathbf\{1\}\[\\text\{action correct\}\]; 150 training steps\.##### Hardware, software, and checkpoints\.

All final runs used one node with eight NVIDIA H200 GPUs; wall\-clock training was 24 h for ALFWorld, 48 h for EB\-ALFRED, and 24 h for Mind2Web\. The peak per\-GPU memory footprints were 127\.0 GB for ALFWorld, 77\.6 GB for EB\-ALFRED, and 127\.0 GB for Mind2Web\. ALFWorld and Mind2Web Planner training use Python 3\.10 and Transformers 4\.51\.1 with a CUDA\-matched PyTorch, and ALFWorld model serving uses vLLM 0\.8\.5–0\.11\.0\. EB\-ALFRED uses Python 3\.9\.20, Torch 2\.4\.0, AI2\-THOR 2\.1\.0, NumPy 1\.26\.4, and Transformers 4\.57\.6 for simulation and Planner updates, together with Python 3\.12, Torch 2\.11\.0, SGLang 0\.5\.14, Transformers 5\.8\.1, and NVIDIA driver 580\.126\.20 for model serving\. Epoch and GRPO checkpoints store the full model, encoder, Planner, optimizer state, stage, epoch or iteration, metrics, and configuration; exported reference or final weight\-only checkpoints may omit optimizer state by design\.

## Appendix BEB\-ALFRED Case Studies

The following cards preserve the prompt–response order while avoiding a second full copy of the invariant EB\-ALFRED policy prompt already given in Section[E](https://arxiv.org/html/2608.06745#A5)\. Each card shows its decision\-relevant prompt excerpt, the two visual inputs delivered to the frozen task policy, the logged model output, and the resulting progress update\. The implementation concatenates the shared policy prefix, decision\-specific suffix, current camera image, and compiled view into one multimodalrole=usermessage\.

Case 1SYSTEMPolicy promptTask\.Put a clean pan on the counter\.
The EB\-ALFRED task\-policy prompt in Section[E](https://arxiv.org/html/2608.06745#A5)requires reasoning inside<think\>tags followed by exactly one admissible action inside<action\>tags\.↓\\downarrowUSERDecision\-specific promptCurrent state\.At Step 22,CounterTop,SinkBasin, andFaucetare visible, and the robot is holding thePan\.
Recent text history\.A previous placement attempt was invalid because the robot was not holding an object; the subsequent action successfully picked up the pan\.
Attachments\.The first image is the current camera observation; the second is the compiledTemporalTraceview\.↓\\downarrowVISIONVisual inputs to the frozen task policy![Refer to caption](https://arxiv.org/html/2608.06745v1/Figures/case1_current_observation.png)Current observation![Refer to caption](https://arxiv.org/html/2608.06745v1/Figures/case1_temporal_trace.png)CompiledTemporalTrace↓\\downarrowVLMAssistant output<think\>The task is to put a clean pan on the counter\. The robot is holding a pan and a counter is visible\. Since the faucet is turned on, the pan has been cleaned; the next step is to place it on the counter\.</think\>
<action\>put down the object in hand</action\>↓\\downarrowRESULTRecorded outcomeLast action executed successfully\.Progress:0→1/30\\rightarrow 1/3\.State updates:held\_object \-\> none;pan \-\> on counter\.Figure B\.1:Case study ofMemPrismon EB\-ALFRED \(part I\)\.Case 2SYSTEMPolicy promptTask\.Move the two books from the bed to the desk\.
The same EB\-ALFRED task\-policy prompt in Section[E](https://arxiv.org/html/2608.06745#A5)requires reasoning inside<think\>tags followed by exactly one admissible action inside<action\>tags\.↓\\downarrowUSERDecision\-specific promptCurrent state\.At Step 21, the desk is visible and the robot is holding a book\.
Recent text history\.Step 18 successfully found a book; Step 19 successfully found the desk\.
Attachments\.The first image is the current camera observation; the second is the compiledEntityStateview\.↓\\downarrowVISIONVisual inputs to the frozen task policy![Refer to caption](https://arxiv.org/html/2608.06745v1/Figures/case2_current_observation.png)Current observation![Refer to caption](https://arxiv.org/html/2608.06745v1/Figures/case2_entity_state.png)CompiledEntityState↓\\downarrowVLMAssistant output<think\>I need to place the book on the desk as part of the task\. The book is currently being held, and a visible desk has been located\. The next step is to put down the book on the desk\.</think\>
<action\>put down the object in hand</action\>↓\\downarrowRESULTRecorded outcomeLast action executed successfully\.Progress:0→1/20\\rightarrow 1/2\.State updates:book \-\> on desk;held\_object \-\> none\.Figure B\.2:Case study ofMemPrismon EB\-ALFRED \(part II\)\.
## Appendix CEvaluation Protocols, Baseline Alignment, and Extra Results

### C\.1Common controlled protocol

All memory\-interface comparisons use the same ordered ALFWorld episode list, initial environment states, observation and admissible\-action interfaces, short\-term textual\-context rule, Qwen2\.5\-VL\-7B\-Instruct task backbone, maximum 50\-step horizon, action parser, and decoding settings\. Once online policies choose different actions, their subsequent observations naturally differ; fairness is therefore enforced at the initial\-task and interface levels, not by forcing counterfactual per\-decision observations to match\. Each method may transform only the historical evidence generated by its own trajectory through the memory interface\. The experiment record specifies whether a method trains the task backbone, a separate memory policy, a writer, or no parameters\.

The ALFWorld unit is an evaluation episode, not necessarily a unique source game\. The run record contains the 140 ordered episode identifiers, the number of unique source\-game identifiers, and the deterministic sampling or repetition rule used to construct the list\.

The primary effectiveness metric is task success rate\. Efficiency metrics are mean and maximum prompt tokens per decision, visual tokens attributable to memory, mean rendered pixels, rendering latency, task\-model latency, and stored bytes\. Each archived result contains the raw success count and denominator together with a Wilson 95% interval\. For conditions evaluated on the same ordered tasks, the archived comparison additionally contains a paired bootstrap interval for the SR difference, obtained by resampling the common episode identifiers while preserving within\-episode pairing\.

### C\.2External baselines

We build every external baseline from its official public implementation rather than from a common reimplementation\. LangMem, A\-Mem, Mem0, andMem0g\\text\{\{Mem0\}\}^\{\\scriptscriptstyle g\}are evaluated as memory interfaces with the frozen common task policy\. CFG\-Bench instead supplies external fine\-grained\-action supervision and is therefore reported as a separately labeled task\-policy baseline, not as a same\-reader memory comparison\. Each memory adapter receives the same serialized sequence of completed environment transitions, and retrieved content is placed in the same memory slot of the task\-policy prompt\. We retain the defaults of the evaluated release unless a change is required by this shared agent interface\. The run manifest records the exact release or commit, writer model, embedding model, retrieval depth, update schedule, and insertion position where applicable\. All auxiliary model calls and their token or API costs are included in the efficiency results, and no baseline is given observations or environment state unavailable toMemPrism\.

##### LangMem\.

We use LangMem from its official public codebase\.Official implementation:[https://github\.com/langchain\-ai/langmem](https://github.com/langchain-ai/langmem);documentation:[https://langchain\-ai\.github\.io/langmem/](https://langchain-ai.github.io/langmem/)\. Completed transitions are processed through its stateful memory manager and stored as a searchable collection; at decision time, the task goal and current observation form the retrieval query\. We use the documented collection\-management and search behavior of the pinned release, including its standard consolidation logic and query limit, while keeping the memory namespace local to each episode to prevent cross\-task leakage\.

##### A\-Mem\.

Our A\-Mem baseline follows the released system\.Official implementation:[https://github\.com/WujiangXu/A\-mem\-sys/](https://github.com/WujiangXu/A-mem-sys/)\. Each completed transition is introduced as a note, after which A\-Mem generates its structured semantic attributes and connects it to related notes following its Zettelkasten\-inspired organization\. Retrieval uses the repository’s combined content\-and\-metadata representation and linked memory structure\. We preserve the recommended note\-construction, linking, and memory\-evolution settings of the pinned release\.

##### Mem0 andMem0g\\text\{\{Mem0\}\}^\{\\scriptscriptstyle g\}\.

Both variants are instantiated from the released Mem0 system\.Official implementation:[https://github\.com/mem0ai/mem0](https://github.com/mem0ai/mem0)\. Mem0 uses the release’s standard extraction and memory\-update pipeline over the serialized transitions\.Mem0g\\text\{\{Mem0\}\}^\{\\scriptscriptstyle g\}uses the same writer and retrieval interface but enables the graph\-memory configuration so that extracted entities and relations can participate in storage and retrieval; its graph construction parameters are left at the recommended values for the pinned release\. Reporting the pinned commit is important here because defaults on the actively maintained main branch may change after the evaluation\.

##### CFG\-Bench transfer baseline\.

CFG\-Bench organizes fine\-grained embodied\-action knowledge into physical interaction, temporal–causal relation, intentional understanding, and evaluative judgment\.Project page:[https://cfg\-bench\.github\.io/](https://cfg-bench.github.io/);official implementation:[https://github\.com/CFG\-Bench/CFG\-Bench](https://github.com/CFG-Bench/CFG-Bench)\. We use the released data format, inference code, and judging pipeline without modification\. For the downstream EB\-ALFRED reference, we reproduce the paper’s transfer recipe by applying LoRA SFT to Qwen2\.5\-VL\-7B on the CFG data for 800 epochs with AdamW, a learning rate of10−610^\{\-6\}, batch size 8, and eight NVIDIA H200 GPUs\. The reported 90\.3%/9\.7% train/validation split is sampled with tier balancing\. Because this condition changes the task\-policy parameters using external supervision, we label it “CFG\-SFT \(reported\-recipe reproduction\),” pin its resulting checkpoint and repository commit, and exclude it from controlled same\-reader memory\-interface claims\.

### C\.3Optical baseline scope

##### Included baseline\.

We compare against the Qwen2\.5\-VL\-7B\-Instruct optical\-history condition that the AgentOCR paper denotes “OCR \(w/o RL\)\.” We refer to it as “AgentOCR \(w/o RL\)” for clarity\. This condition renders the accumulated history as an image but does not apply RL to the task\-policy model, making it the closest reported AgentOCR setting to our frozen\-policy evaluation\.official implementation:[https://github\.com/langfengQ/AgentOCR](https://github.com/langfengQ/AgentOCR)\.

##### Excluded optical systems\.

The RL\-trained AgentOCR system jointly learns environment actions and an adaptive compression rate using a compression\-aware reward\. We exclude it because its ALFWorld score therefore reflects task\-policy RL in addition to the optical memory representation\. MemOCR jointly trains its memory\-drafting and memory\-reading behaviors with budget\-aware RL and full\-parameter updates to Qwen2\.5\-VL\-7B\-Instruct on long\-context QA data\. It has no reported ALFWorld setting, and a valid transfer would require training an ALFWorld policy model rather than inserting a fixed memory interface into our frozen policy\. We therefore do not report a MemOCR number\.official implementation:[https://github\.com/meituan/MemOCR](https://github.com/meituan/MemOCR)\. OCR\-Memory proposes a learned optical locate\-and\-transcribe retriever, but no official implementation was publicly linked when our artifacts were frozen; without executable code, we cannot reproduce it under the same ALFWorld protocol\.

### C\.4ALFWorld comparison with AgentOCR \(w/o RL\)

Table[C\.1](https://arxiv.org/html/2608.06745#A3.T1)compares the two non\-RL task\-policy conditions on ALFWorld with the same Qwen2\.5\-VL\-7B\-Instruct backbone\. AgentOCR reports 25\.30% SR with 0\.47k average memory\-context tokens per step, whereasMemPrismachieves 40\.71% SR with 797 average prompt tokens, or 0\.80k after rounding\. This is an absolute improvement of 15\.41 percentage points in SR\. The token column retains the accounting available from each artifact: AgentOCR reports memory\-context tokens, whileMemPrismreports the complete average prompt length\. We therefore use it to describe the observed input footprint rather than claim an exactly budget\-matched comparison\.

Table C\.1:ALFWorld comparison with the 7B no\-RL optical\-history setting reported by AgentOCR\. MemPrism keeps its task policy frozen while training only the view Router\.MethodTask\-policy backboneTask\-policy RLSR \(%\)Avg\. tokens/stepAgentOCR \(w/o RL\)Qwen2\.5\-VL\-7B\-InstructNo25\.300\.47kMemPrismQwen2\.5\-VL\-7B\-InstructNo40\.710\.80k \(797\)
### C\.5Optical\-resolution sensitivity

We evaluate the finalMemPrismcheckpoint on ALFWorld with maximum image long sides of 448, 672, and 896 pixels\. The checkpoint, compiler, rendering rules, prompt template, ordered episode list, and decoding settings are fixed; the maximum\-long\-side cap is the only configured factor\. Online trajectories, including later Router inputs and selected views, are allowed to diverge naturally after an action differs\.

Table C\.2:ALFWorld resolution sensitivity under three maximum\-long\-side limits\.Max side \(px\)SR \(%\)Avg\. prompt tok\.Invalid action \(%\)44838\.577030\.04367240\.717970\.06489640\.009170\.097The 672\-pixel setting obtains the highest SR at40\.71%40\.71\\%, with 797 average prompt tokens and a0\.064%0\.064\\%invalid\-action rate\. Reducing the longest side to 448 pixels lowers the average prompt length to 703 tokens and the invalid\-action rate to0\.043%0\.043\\%, while SR decreases to38\.57%38\.57\\%\. Increasing the longest side to 896 pixels raises the average prompt length to 917 tokens and the invalid\-action rate to0\.097%0\.097\\%, without improving over the 672\-pixel SR\. We therefore retain 672 pixels as the default setting\.

### C\.6View\-selection distributions across benchmarks

Table C\.3:Marginal view\-selection distributions \(%\) across the three benchmarks\.View choiceALFWorldEB\-ALFREDMind2WebView typeTemporalTrace55\.1542\.2842\.26ActionEffect31\.2421\.987\.62EntityState13\.6120\.8618\.48DependencyChain0\.0014\.8831\.64Window / outcome filterrecent\_short / all47\.0420\.3731\.70recent\_short / exception3\.731\.861\.17recent\_short / state\_update24\.8427\.3717\.80recent\_short / no\_observed\_change0\.043\.332\.30recent\_long / all0\.167\.1422\.10recent\_long / exception0\.001\.160\.69recent\_long / state\_update0\.0012\.2910\.60recent\_long / no\_observed\_change0\.001\.121\.35all / all0\.008\.308\.62all / exception0\.001\.120\.14all / state\_update4\.5414\.213\.16all / no\_observed\_change19\.651\.750\.37Granularitycoarse38\.5626\.8122\.76medium0\.0035\.3956\.85fine61\.4437\.8020\.39The zeroDependencyChainfrequency on ALFWorld reflects policy concentration rather than removal of this action from the view space\.DependencyChainaccounts for only about3%3\\%of the marginal probability mass in the ALFWorld teacher distribution and therefore enters GRPO as a comparatively rare view type\. GRPO further concentrates the policy on view types that receive more consistent task\-level reward, soDependencyChainis never the argmax under final greedy selection and appears as0\.00%0\.00\\%in Table[C\.3](https://arxiv.org/html/2608.06745#A3.T3)\. Its compiler and action remain available even though they are not selected in this evaluation\.

## Appendix DAdditional Method and Implementation Details

### D\.1Raw and structured events

At decision steptt, the persistent history is an ordered event streamℰ<t=\(e~1,…,e~t−1\)\\mathcal\{E\}\_\{<t\}=\(\\widetilde\{e\}\_\{1\},\\ldots,\\widetilde\{e\}\_\{t\-1\}\)\. The environment transition is first preserved as a raw tuple

et=\(ot−,ut,yt\+,rt,mt\),e\_\{t\}=\(o\_\{t\}^\{\-\},u\_\{t\},y\_\{t\}^\{\+\},r\_\{t\},m\_\{t\}\),\(D\.1\)whereot−o\_\{t\}^\{\-\}is the pre\-action observation,utu\_\{t\}is the action that was actually executed \(or the recorded human action in Mind2Web\),yt\+y\_\{t\}^\{\+\}is the resulting feedback,rtr\_\{t\}is the task signal, andmtm\_\{t\}contains the step index and benchmark metadata\. Raw events are never replaced by model summaries\.

Phi and deterministic validators derive

e~t=\(t,etent,etact,etout,Δ​st,ξt\),\\widetilde\{e\}\_\{t\}=\(t,e^\{\\mathrm\{ent\}\}\_\{t\},e^\{\\mathrm\{act\}\}\_\{t\},e^\{\\mathrm\{out\}\}\_\{t\},\\Delta s\_\{t\},\\xi\_\{t\}\),\(D\.2\)whereΔ​st=\{\(kj,vjnew\)\}j=1nt\\Delta s\_\{t\}=\\\{\(k\_\{j\},v^\{\\mathrm\{new\}\}\_\{j\}\)\\\}\_\{j=1\}^\{n\_\{t\}\}is a sparse set of observed new values\. The extractor is not required to hallucinate old values\. Validation drops empty keys, unknown values, and exact duplicates within one extractor response, but it retains accepted deltas on repeated events\. The event stream tracks the latest value per key to classify whether an update changes known state; individual view extractors may suppress unchanged values in their displayed output\. The raw event is preserved as a fallback\. The outcome is one ofexception,state\_update, orno\_observed\_change\. Optional evidenceξt\\xi\_\{t\}may contain tags, an action\-result image, entity boxes, and crops\.

### D\.2Benchmark\-specific Phi adapters

##### ALFWorld\.

The adapter receives the pre\-action textual observation, the selected admissible command, and the environment feedback\. Rules provide hard evidence for the normalized action and explicit failure language, while Phi extracts the primary entity and sparse semantic state updates\. There is no image\-grounding stage\.

##### EB\-ALFRED\.

The adapter records action metadata and a synchronized post\-action camera frame; audit metadata may additionally contain before/post frame paths and hashes\. Stage 1 extracts the primary entity, normalized action, and sparse state updates\. If the simulator explicitly reports failure, the outcome is forced toexceptionand the state delta is cleared\. Stage 2 uses the post\-action image to localize at most three validated, visible candidate entities\. Invalid or missing boxes remove only visual evidence; they cannot alter Stage 1 semantics\. A stale\-frame flag prevents unsynchronized pixels from being treated as grounded evidence\.

##### Mind2Web\.

The adapter usescleaned\_htmlnear candidate backend\-node IDs when available, falling back to compact HTML snippets\. The hard action metadata is normalized into operation, target, and value\. Phi may emit onlypage\_state,overlay\_state,field\_value,choice\_value,result\_state, orworkflow\_state\. Because replay follows a valid human trajectory, an event may remain astate\_updateeven when no member of this restricted semantic key set is extracted\.

### D\.3Composer and rendering invariants

Composer first applies the chosen time and outcome scope, then restores events whose normalized terms directly match the task goal\. DependencyChain also retains the most recent context required to define its anchor\. Thus, scope is a principal selection rule rather than an irreversible deletion:

ℰ^t=Merge⁡\(σwt,ct​\(ℰ<t\),TaskRelevant⁡\(ℰ<t,g\),RecentContext⁡\(ℰ<t,τt\)\)\.\\widehat\{\\mathcal\{E\}\}\_\{t\}=\\operatorname\{Merge\}\\\!\\left\(\\sigma\_\{w\_\{t\},c\_\{t\}\}\(\\mathcal\{E\}\_\{<t\}\),\\operatorname\{TaskRelevant\}\(\\mathcal\{E\}\_\{<t\},g\),\\operatorname\{RecentContext\}\(\\mathcal\{E\}\_\{<t\},\\tau\_\{t\}\)\\right\)\.\(D\.3\)The four compilers then expose complementary relations:

- •TemporalTracepreserves event order and displays action, outcome, and sparse changes\.
- •ActionEffectgroups repeated action–entity pairs and aligns attempts with observed outcomes\.
- •EntityStategroups accepted new values by entity and state key, forming an ordered state chain without requiring inferred old values\.
- •DependencyChainanchors on the latest available event and retrieves recent events sharing an entity or state key\. It is a local dependency trace, not a verified causal graph\.

Granularity changes the number and detail of displayed entries but never changes the persistent event stream\. Render uses fixed spatial semantics: proximity indicates grouping, rows and columns support comparison, arrows indicate change or dependency, and highlights mark failures or task\-relevant evidence\. The same event stream, action, goal, and renderer configuration produce the same image\. Compiled views are discarded after the current decision and are never appended to persistent memory\.

Decision\-time compilation and environment update

1. 1\.Input:current observationoto\_\{t\}, goalgg, persistent event streamℰ<t\\mathcal\{E\}\_\{<t\}, frozen task policy, and view policy\.
2. 2\.Encodeoto\_\{t\},gg, the last\-eight\-event window summary, and last\-64\-event global statistics; selectatviewa\_\{t\}^\{\\mathrm\{view\}\}\.
3. 3\.Apply the selected scope, restore task\-matched evidence, and retain required recent dependency context\.
4. 4\.Build the relation\-specific intermediate structure and render the temporary optical viewVtV\_\{t\}\.
5. 5\.Query the frozen task policy with the current benchmark input,gg, shared short text context, andVtV\_\{t\}; obtain environment actionutu\_\{t\}\.
6. 6\.Online benchmarks executeutu\_\{t\}\. Mind2Web scores the prediction but advances with the human action and recorded next state\.
7. 7\.Preserve the raw transition, derive and validatee~t\\widetilde\{e\}\_\{t\}, and append it once to obtainℰ≤t\\mathcal\{E\}\_\{\\leq t\}\. DiscardVtV\_\{t\}\.

Figure D\.1:Complete decision\-time data flow\. The temporary view influences the current action but cannot overwrite recorded history\.
### D\.4Router architecture and optimization

Router does not read the full rendered history\. It consumes a 384\-dimensional embedding of the current textual observation, a 384\-dimensional goal embedding, a 128\-dimensional summary of the most recent eight events, and eight global statistics computed from at most 64 events\. The frozen text embeddings are projected into a 512\-dimensional shared space and processed by a four\-layer Transformer encoder with eight attention heads, a 1024\-dimensional feed\-forward sublayer, and dropout0\.10\.1\. The planner head contains three residual MLP blocks with hidden dimension 1024, followed by layer normalization and one joint 144\-way output layer\. A 64\-dimensional entity\-hint head is retained for compatibility with the older rendering interface but is frozen by the current D1/D2 objective\. A joint action head is used instead of four independent heads so that dependencies among relation, range, outcome, and granularity are retained\. All three benchmarks use theaction\_frequencyevent\-window channel in both training and evaluation\. It canonicalizes each raw action signature and stores the eight largest normalized frequencies in descending order, thereby exposing whether the recent history is concentrated on one repeated operation or distributed across several operations without relying on a fixed action taxonomy\. The remaining active coordinates encode outcome ratios, entity and delta statistics, maximum exception and no\-change streaks, and recency\-weighted state\-change and exception densities; unused coordinates are zero padded to 128 dimensions\.

For supervised initialization, a teacher distributionqT​\(a∣t\)q\_\{T\}\(a\\mid t\)is conditioned on the decision state and a reference next action\. Router minimizes

ℒSFT=KL\(qT\(⋅∣t\)∥πθv\(⋅∣zt\)\)\+λC∑a∈𝒜viewπθv\(a∣zt\)C^t\(a\),\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\\operatorname\{KL\}\\\!\\left\(q\_\{T\}\(\\cdot\\mid t\)\\,\\\|\\,\\pi\_\{\\theta\}^\{v\}\(\\cdot\\mid z\_\{t\}\)\\right\)\+\\lambda\_\{C\}\\sum\_\{a\\in\\mathcal\{A\}^\{\\mathrm\{view\}\}\}\\pi\_\{\\theta\}^\{v\}\(a\\mid z\_\{t\}\)\\widehat\{C\}\_\{t\}\(a\),\(D\.4\)whereC^t\\widehat\{C\}\_\{t\}discourages unnecessarily large or fine views\.

For GRPO refinement, grouped trajectories from the same task share the normalized trajectory advantage

Ai=\{\(Ri−μg\)/\(σg\+ϵ\),σg\>ϵ,0,otherwise\.A\_\{i\}=\\begin\{cases\}\(R\_\{i\}\-\\mu\_\{g\}\)/\(\\sigma\_\{g\}\+\\epsilon\),&\\sigma\_\{g\}\>\\epsilon,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(D\.5\)Only Router is updated\. The task policy, event extractor, compiler, renderer, and reference view policy remain frozen\. The clipped objective includes a reference\-policy KL term and, when enabled by the benchmark configuration, an entropy bonus\. Mind2Web constructs groups from offline\-replay tasks and uses the binary rewardRi=𝟏​\[action correct\]R\_\{i\}=\\mathbf\{1\}\[\\text\{action correct\}\]\. The complete benchmark\-specific settings are reported in Table[A\.1](https://arxiv.org/html/2608.06745#A1.T1)\.

### D\.5Computational and storage complexity

Appending a raw and structured event is amortizedO​\(1\)O\(1\), excluding the Phi model call\. LetNt=\|ℰ<t\|N\_\{t\}=\|\\mathcal\{E\}\_\{<t\}\|, letDtD\_\{t\}be the number of accepted state updates in scope, and letPtP\_\{t\}be the number of rendered pixels\. Scope filtering and goal\-term restoration requireO​\(Nt\)O\(N\_\{t\}\)time\. TemporalTrace and ActionEffect construction areO​\(Nt\)O\(N\_\{t\}\); EntityState isO​\(Nt\+Dt\)O\(N\_\{t\}\+D\_\{t\}\); and the current backward dependency scan isO​\(Nt\)O\(N\_\{t\}\)\. Rendering isO​\(Pt\)O\(P\_\{t\}\)\. Router inference is constant inNtN\_\{t\}after its fixed\-size summaries are constructed\. Persistent semantic storage isO​\(Nt\+Dt\)O\(N\_\{t\}\+D\_\{t\}\), plus any benchmark\-specific images retained as raw evidence\.

These asymptotic costs exclude model inference, which dominates wall\-clock time\. Efficiency accounting therefore records Phi calls, task\-policy calls, rendered pixels, visual tokens, prompt tokens, renderer latency, model latency, and peak stored bytes as separate fields rather than combining them into one opaque “memory cost” number\.

## Appendix EPrompts, Schemas, and Action Definitions

### E\.1Phi event\-extraction prompts

`Prompt E\.0: ALFWorld Phi Event Extraction Prompt E\.0: EB\-ALFRED Phi Semantic Stage Prompt E\.0: EB\-ALFRED Phi Grounding Stage Prompt E\.0: Mind2Web Phi Event Extraction`

`E\.2 View\-teacher prompt and target schema Prompt E\.0: Generic View Teacher Prompt E\.0: Mind2Web View Teacher E\.3 Task\-policy prompts Prompt E\.0: ALFWorld Task Policy Prompt E\.0: EB\-ALFRED Task Policy—System Message Prompt E\.0: EB\-ALFRED Task Policy—User Message Prompt E\.0: Mind2Web Task Policy E\.4 View and environment action definitions Router selects atview=\(τt,wt,ct,γt\)∈𝒯×𝒲×𝒪×𝒢,a\_\{t\}^\{\\mathrm\{view\}\}=\(\\tau\_\{t\},w\_\{t\},c\_\{t\},\\gamma\_\{t\}\)\\in\\mathcal\{T\}\\times\\mathcal\{W\}\\times\\mathcal\{O\}\\times\\mathcal\{G\}, \(E\.1\) where 𝒯=\\displaystyle\\mathcal\{T\}=\{\} \{TemporalTrace,ActionEffect,EntityState,DependencyChain\},\\displaystyle\\\{\\textsc\{TemporalTrace\},\\textsc\{ActionEffect\},\\textsc\{EntityState\},\\textsc\{DependencyChain\}\\\}, \(E\.2\) 𝒲=\\displaystyle\\mathcal\{W\}=\{\} \{recent\_short​\(6\),recent\_long​\(10\),all\},\\displaystyle\\\{\\texttt\{recent\\\_short\}\(6\),\\texttt\{recent\\\_long\}\(10\),\\texttt\{all\}\\\}, \(E\.3\) 𝒪=\\displaystyle\\mathcal\{O\}=\{\} \{all,exception,state\_update,no\_observed\_change\},\\displaystyle\\\{\\texttt\{all\},\\texttt\{exception\},\\texttt\{state\\\_update\},\\texttt\{no\\\_observed\\\_change\}\\\}, \(E\.4\) 𝒢=\\displaystyle\\mathcal\{G\}=\{\} \{coarse,medium,fine\}\.\\displaystyle\\\{\\texttt\{coarse\},\\texttt\{medium\},\\texttt\{fine\}\\\}\. \(E\.5\) The implementation enumerates view type, time window, outcome filter, and granularity in that order\. The 4×3×4×3=1444\\times 3\\times 4\\times 3=144 actions use the stable zero\-based index i​\(a\)=\(\(iτ⋅3\+iw\)⋅4\+ic\)⋅3\+iγ\.i\(a\)=\\bigl\(\(i\_\{\\tau\}\\cdot 3\+i\_\{w\}\)\\cdot 4\+i\_\{c\}\\bigr\)\\cdot 3\+i\_\{\\gamma\}\. \(E\.6\) The serialized action object is \{ "tau": "TemporalTrace\|ActionEffect\|EntityState\| DependencyChain", "window": "recent\_short\|recent\_long\|all", "filter": "all\|exception\|state\_update\| no\_observed\_change", "gamma": "coarse\|medium\|fine" \} Legacy names such as timeline, action\_outcome, state\_table, and causal\_chain are accepted only as read\-time aliases and serialize back to the four canonical names\. Environment actions use separate benchmark contracts\. ALFWorld returns one exact member of the current textual admissible\-action set\. EB\-ALFRED returns one exact name from the scene\-filtered skill list; the parser additionally canonicalizes any put down \.\.\. form to put down the object in hand when that action is available\. Mind2Web returns one of CLICK, TYPE, or SELECT with a current element ref; TYPE and SELECT also require value\. E\.5 Parsing and validation Phi accepts a mapping or attempts to parse the substring from the first opening brace to the last closing brace in a text response\. ALFWorld extracts the text enclosed by <action\> tags and applies the fixed admissible\-action policy\. EB\-ALFRED matches in order by exact string, case\-insensitive string, and normalized whitespace\. Mind2Web accepts exactly one tagged JSON action, rejects unknown operations or references absent from the current snapshot, and scores the normalized operation, target reference, and value against the recorded human action\.`