Learning to Learn from Multimodal Experience

# Learning to Learn from Multimodal Experience
Source: [https://arxiv.org/html/2605.16857](https://arxiv.org/html/2605.16857)
Xingyu Sui1, Weixiang Zhao1, Yongxin Tang1 Yanyan Zhao1,Yang Wu2,Dandan Tu2,Bing Qin1 1Harbin Institute of Technology,2Huawei Technologies Co\., Ltd \{xysui, wxzhao, yyzhao\}@ir\.hit\.edu\.cn

###### Abstract

Multimodal agents increasingly interact with real\-world\-like environments and generate rich trajectories spanning perception, reasoning, and action\. A key research focus is how to transform these raw multimodal trajectories into reusable experience that can improve future task performance\. Existing experience\-driven learning methods are mostly developed in textual settings and typically rely on manually designed memory schemas, making them less effective for multimodal experience that requires composing heterogeneous signals across modalities, temporal scales, and abstraction levels\. We proposeAutoMMemo, a general framework that enables agents to automatically design memory mechanisms for learning from multimodal experience\.AutoMMemorepresents each memory mechanism as an executable*memo program*, which defines how past episodes are stored, organized, and retrieved\. It discovers effective memo programs through an iterative process of update\-then\-retrieve evaluation, reflection\-guided mutation, and budget\-aware tree search\. Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show thatAutoMMemoconsistently improves agent performance over no\-memory agents and fixed memory baselines, while maintaining reasonable token and interaction costs\. Transfer and search\-process analyses further demonstrate that the learned memory designs capture reusable experience structures and progressively refine retrieval relevance\. These results highlight adaptive memory design as a key optimization target for multimodal experience\-driven learning\.

## 1Introduction

Recent advances in multimodal large language models \(MLLMs\) have enabled agents to actively explore and interact within environments that more closely resemble the real world\(Yaoet al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib3); Duranteet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib1); Liet al\.,[2025b](https://arxiv.org/html/2605.16857#bib.bib2)\)\. Such interactions naturally generate large volumes of trajectories, capturing rich signals across perception, reasoning, and action\. A central research focus, therefore, is how to transform these raw interaction trajectories into reusable experience that can continuously improve agent performance\(Shinnet al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib4); Silver and Sutton,[2025](https://arxiv.org/html/2605.16857#bib.bib5)\)\. In this context, experience\-driven learning via memory mechanisms has emerged as a promising paradigm, enabling agents to accumulate, organize, and reuse experience across tasks\(Gaoet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib6); Fanget al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib22)\)\. As a result, the effectiveness of an agent is no longer solely determined by its underlying model, but increasingly by how it represents and utilizes its experience\(Douet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib19)\)\.

However, existing experience\-driven learning paradigms have been predominantly developed in textual settings\(Zhaoet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib13); Zhanget al\.,[2025a](https://arxiv.org/html/2605.16857#bib.bib44); Xiaet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib15)\), leaving their extension to multimodal environments largely underexplored\. This creates a fundamental mismatch: in real\-world scenarios, experience is inherently multimodal, arising from interactions that involve diverse signals such as vision, language, and action dynamics\(Driesset al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib8); Sarchet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib9)\)\. These signals differ in modality, temporal scope, and semantic abstraction, forming a complex mixture of structured and unstructured information\. Thus, designing an effective memory system for experience learning requires making a series of tightly coupled decisions, such as what to store, how to represent it, how to organize it, and when to retrieve it\. For multimodal experience, these decisions become particularly challenging, as useful information may be distributed across visual observations, language\-based reasoning, and action trajectories at different temporal scales and abstraction levels\(Huet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib10)\)\.

Despite this complexity, existing approaches largely rely on manually designed memory schemas, with fixed and predefined formats for experience representation and retrieval\. While effective in controlled settings, such designs inherently depend on human priors and lack the flexibility to adapt across tasks, modalities, and environments\. As a result, they often lead to sub\-optimal experience utilization: inappropriate experience representations may either discard critical signals, such as fine\-grained visual cues\(Boet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib16); Jianget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib45)\), or introduce excessive noise\(Liet al\.,[2025a](https://arxiv.org/html/2605.16857#bib.bib17); Zhuet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib18)\), ultimately hindering learning efficiency and generalization\.

These limitations point to a deeper issue: the optimal way to structure and utilize multimodal experience is inherently task\-dependent and evolves over time as both the environment and the agent itself change\. Therefore, memory design in multimodal agents should not be treated as a one\-shot engineering decision, but rather as a continually evolving process\. Building on this insight, we argue that effective multimodal experience\-driven learning requires not only accumulating or retrieving experience, but learning how to structure and utilize it adaptively\. In contrast to existing approaches that operate on fixed memory designs, this shifts the focus toward a higher\-level objective: enabling agents to adapt their own memory strategies for improved learning\. This view is consistent with human learning, where individuals continuously refine how they organize, abstract, and recall experience based on context and goals\(Cohen and Eichenbaum,[1993](https://arxiv.org/html/2605.16857#bib.bib20); Eichenbaum,[2017](https://arxiv.org/html/2605.16857#bib.bib21)\)\.

To this end, we proposeAutoMMemo, a general framework that enables agents to automatically design memory for learning from multimodal experience\. Rather than relying on fixed memory schemas,AutoMMemorepresents each memory mechanism as an executable*memo program*, which specifies how past multimodal episodes are stored, organized, and retrieved for future tasks\. This abstraction provides a unified memory interface while keeping the internal design space open\-ended, allowing agents to explore diverse strategies for structuring and utilizing experience\.AutoMMemodiscovers effective memo programs through iterative evaluation, reflection, mutation, and selection\. Each candidate is assessed under an update\-then\-retrieve protocol that measures whether its constructed memory improves performance on held\-out tasks\. The resulting trajectories and retrieved memories are then used by a meta agent to diagnose weaknesses and generate improved descendants\. To make this search efficient under limited evaluation budgets,AutoMMemouses a budget\-aware tree search strategy that balances re\-evaluating promising programs with exploring new memory designs\.

Empirically, our experiments show that adaptively learned memory mechanisms consistently improve agent performance across both GUI/Web navigation and multimodal visual reasoning benchmarks\. Compared with no\-memory agents and manually fixed memory designs, the memo programs discovered byAutoMMemolead to stronger task performance while maintaining reasonable interaction and token costs\. Further transfer analyses show that effective memory strategies are not merely overfitted to a single benchmark or model, but can provide measurable gains across different environments and execution agents\. In addition, our search\-process analysis reveals thatAutoMMemoprogressively refines memory designs by retaining useful experience patterns, removing noisy or redundant information, and improving retrieval relevance over iterations\. Together, these results demonstrate that memory design itself is a key object of optimization for multimodal agents, and that learning how to structure and retrieve experience can substantially improve experience\-driven learning beyond fixed memory schemas\.

## 2Related Works

Learning from Experience with Memory\.Memory\-augmented experience learning has emerged as a central paradigm for enabling agents to improve from past interactions\(Gaoet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib6); Fanget al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib22)\)\. Existing approaches primarily focus on extracting and representing experience from interaction trajectories, and largely rely on manually designed memory schemas\.

In textual settings, experience is typically represented in predefined forms, such as insights\(Shinnet al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib4); Zhaoet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib13); Fuet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib23); Zhanget al\.,[2025a](https://arxiv.org/html/2605.16857#bib.bib44)\), skills\(Wanget al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib11); Chenet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib24); Xiaet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib15)\), or structured workflows\(Wanget al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib14)\)\. These representations enable experience reuse but are fixed by design and require human specification\.

Recent work extends this paradigm to multimodal settings by incorporating additional sensory signals\(Sarchet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib9); Chhikaraet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib28); Yuanet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib26); Chenet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib27)\)\. For example,Boet al\.\([2025](https://arxiv.org/html/2605.16857#bib.bib16)\); Jianget al\.\([2026](https://arxiv.org/html/2605.16857#bib.bib45)\)augment experience extraction with visual information, whileLiuet al\.\([2025](https://arxiv.org/html/2605.16857#bib.bib25)\)models cross\-modal dependencies via graph\-structured memory\. Despite these extensions, they still rely on predefined memory formats and manually designed representation structures\.

Automatic Memory Design\.To overcome the limitations of manually designed memory schemas, recent works explore automatic memory design, where agents learn to optimize their memory\.

In textual settings, existing approaches can be broadly categorized based on their design spaces\. The first line of works operate within constrained design spaces, where memory is defined using predefined components\. These components may correspond to a single form of experience\(Zhouet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib32); Zhanget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib33); Huanget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib34)\)or combinations of multiple predefined memory structures\(Zhanget al\.,[2025b](https://arxiv.org/html/2605.16857#bib.bib29)\), over which optimization or selection is performed\. The second line of works explore more open\-ended design spaces, where memory is defined and optimized in the form of executable programs or policies\(Xionget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib47); Panet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib30)\)\.

Despite recent progress, existing approaches remain largely confined to purely textual environments and struggle to generalize to multimodal experience\. A concurrent study extends automatic memory design to multimodal settings\(Liuet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib31)\); however, it is mainly tailored to question\-answering tasks, leaving the broader challenge of learning generalizable and flexible memory mechanisms for multimodal experience largely unaddressed\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x1.png)Figure 1:Overview ofAutoMMemo\. It searches over executable memo programs that specify how multimodal experience is stored and retrieved\. Each candidate is evaluated with an update\-then\-retrieve protocol on held\-out tasks, and the resulting trajectories and scores are used to derive reflection\-based improvement insights for program mutation\. A budget\-aware tree search maintains a memo\-structure tree and decides whether to re\-evaluate an existing node or generate a new descendant\.
## 3Method

We proposeAutoMMemo, a general framework for learning adaptive memory mechanisms from multimodal experience\. As illustrated in Figure[1](https://arxiv.org/html/2605.16857#S2.F1),AutoMMemoconsists of four stages\. First, we define a unified memo\-program interface that keeps evaluation comparable while leaving the internal memory mechanism open\-ended \(Section[3\.1](https://arxiv.org/html/2605.16857#S3.SS1)\)\. Second, we evaluate each candidate using an update\-then\-retrieve protocol that measures whether the program can convert past episodes into useful context for held\-out tasks \(Section[3\.2](https://arxiv.org/html/2605.16857#S3.SS2)\)\. Third, we use execution evidence to diagnose memory failures and mutate programs into improved descendants \(Section[3\.3](https://arxiv.org/html/2605.16857#S3.SS3)\)\. Finally, we allocate the limited evaluation budget with a tree\-search strategy that balances re\-testing existing programs and exploring new memory designs \(Section[3\.4](https://arxiv.org/html/2605.16857#S3.SS4)\)\.

### 3\.1Executable Memo Programs

A memo programmmis an executable Python program with two operations,m\.update​\(e\)m\.\\mathrm\{update\}\(e\)andm\.retrieve​\(e\)m\.\\mathrm\{retrieve\}\(e\), whereeedenotes an episode recorder\. The update operation incorporates past experience into an internal memory state, while the retrieve operation returns a memory payload for a new task\. This interface fixes the external contract of memory, but leaves the internal mechanism open\-ended\. The episode recorder provides a unified representation of multimodal experience\. It may include task instructions, visual observations, environment states, action histories, model messages, task\-specific metadata, and final outcomes\. The retrieved memory payload may contain textual summaries, visual references, structured metadata, or multimodal embeddings\. Textual fields are injected into the agent prompt, while visual references can be provided as additional visual inputs\.

Under this abstraction, a memo program can implement different memory strategies: what to store, how to summarize trajectories, how to index textual and visual observations, how to associate states with actions and outcomes, and how to rank retrieved memories\. The program can use a controlled set of general primitives, such as embedding models, multimodal model calls, vector stores, graph utilities, and lightweight databases\. During search, the execution agent, environment, and evaluator are fixed; only the memo program is optimized\. Thus, each candidate is judged by downstream reward under the same evaluation protocol\.

### 3\.2Update\-then\-Retrieve Evaluation

Given a candidate memo programmm,AutoMMemoevaluates whether it can transform past multimodal experience into useful context for future tasks\. Let𝒟u\\mathcal\{D\}\_\{u\}be a set of update episodes and𝒟r\\mathcal\{D\}\_\{r\}be a disjoint set of retrieve\-time tasks\. The update episodes are collected once using the fixed execution agent and shared across all candidates\.

For each update episodeτ∈𝒟u\\tau\\in\\mathcal\{D\}\_\{u\}, we obtain an episode recordereτe\_\{\\tau\}and callm\.update​\(eτ\)m\.\\mathrm\{update\}\(e\_\{\\tau\}\)to build the program’s memory state\. After all update episodes are processed, this memory state is fixed\. For each held\-out taskx∈𝒟rx\\in\\mathcal\{D\}\_\{r\}, the program receives a partial recorderexe\_\{x\}containing the task input and initial observations, and returns a memory payloadpx=m\.retrieve​\(ex\)p\_\{x\}=m\.\\mathrm\{retrieve\}\(e\_\{x\}\)\. The payload is provided to the fixed execution agent, which completes the task and receives a scalar rewardRx∈\[0,1\]R\_\{x\}\\in\[0,1\]\.

The validation score ofmmis the average reward over retrieve\-time tasks:

s​\(m\)=1\|𝒟r\|​∑x∈𝒟rRx\.s\(m\)=\\frac\{1\}\{\|\\mathcal\{D\}\_\{r\}\|\}\\sum\_\{x\\in\\mathcal\{D\}\_\{r\}\}R\_\{x\}\.\(1\)Because the update episodes, execution agent, environment, and evaluator are shared across candidates, differences ins​\(m\)s\(m\)reflect the quality of the memo program\.

To ensure comparability, all candidates follow the same retrieval budget, including limits on payload length and the number of returned visual references\. This prevents trivial gains from excessive context injection and encourages concise, task\-relevant multimodal memory\. The retrieved payloads and resulting trajectories are recorded as evidence for subsequent mutation\.

### 3\.3Reflection\-Guided Program Mutation

After evaluating a memo program,AutoMMemouses execution evidence to generate improved descendants\. For a candidatemim\_\{i\}, the meta agent receives its source code, validation score, retrieved payloads, and representative retrieve\-time trajectories\. This evidence allows the meta agent to diagnose how the current memory mechanism affects downstream behavior\.

The diagnosis identifies whether retrieved memories are helpful, missing, irrelevant, redundant, or misleading\. For multimodal tasks, it also examines whether visual memories help ground observations, recognize relevant states, select actions, or avoid previous failures\. The diagnosis may reveal structural issues such as weak text\-vision alignment, unstable retrieval keys, or excessive truncation\.

Conditioned on this diagnosis, a code generation model mutates the parent program into a child memo program\. The mutation may revise trajectory summarization, textual or visual indexing, state\-action association, memory ranking, or payload formatting\. The child must preserve the samem\.updatem\.\\mathrm\{update\}andm\.retrievem\.\\mathrm\{retrieve\}interface, so it can be evaluated under the same protocol\.

Before full evaluation, the child undergoes a quick examination that checks imports, interface implementation, execution on sample episodes, payload schema validity, and retrieval\-budget compliance\. If the child fails, a repair model attempts to fix it for a limited number of attempts\. Only valid programs are inserted into the generation tree for full evaluation\. We provide the full prompts used by the meta agent, including diagnosis, mutation, quick examination, and repair prompts, in Appendix[K](https://arxiv.org/html/2605.16857#A11)\.

### 3\.4Budget\-Aware Tree Search

Evaluating memo programs is expensive, soAutoMMemomust decide whether to re\-evaluate existing programs or generate new ones\. We maintain a generation tree𝒯\\mathcal\{T\}, where each nodeiiis a memo programmim\_\{i\}, and each edgei→ji\\rightarrow jindicates thatmjm\_\{j\}is generated by mutatingmim\_\{i\}\.

At each round,AutoMMemoconsiders two actions\. The first action,Evaluate​\(i\)\\textsc\{Evaluate\}\(i\), re\-evaluates an existing program to reduce uncertainty\. Its score is

UCBieval=μ^i\+ce​log⁡Nni,\\mathrm\{UCB\}^\{\\mathrm\{eval\}\}\_\{i\}=\\hat\{\\mu\}\_\{i\}\+c\_\{e\}\\sqrt\{\\frac\{\\log N\}\{n\_\{i\}\}\},\(2\)whereμ^i\\hat\{\\mu\}\_\{i\}is the empirical score of nodeii,nin\_\{i\}is its number of full evaluations,NNis the total number of evaluations consumed, andcec\_\{e\}controls exploration\.

The second action,Generate​\(i\)\\textsc\{Generate\}\(i\), expands nodeiiby reflection\-guided mutation\. A good parent should both perform well and have potential to produce better descendants\. For each evaluated childjjofii, we measure its positive improvement by

Δi→j=max⁡\(0,μ^j−μ^isnap\),\\Delta\_\{i\\rightarrow j\}=\\max\\left\(0,\\hat\{\\mu\}\_\{j\}\-\\hat\{\\mu\}^\{\\mathrm\{snap\}\}\_\{i\}\\right\),\(3\)whereμ^isnap\\hat\{\\mu\}^\{\\mathrm\{snap\}\}\_\{i\}is the parent’s score whenjjis generated\. This avoids changing the improvement estimate when the parent is later re\-evaluated\.

For nodeii, letKiK\_\{i\}be the number of evaluated children andSiS\_\{i\}be their cumulative positive improvement\. We estimate its local improvement potential as

Δ^i=β​ρ​max⁡\(0,μ^i−μ^0\)\+Siβ\+Ki,\\hat\{\\Delta\}\_\{i\}=\\frac\{\\beta\\rho\\max\(0,\\hat\{\\mu\}\_\{i\}\-\\hat\{\\mu\}\_\{0\}\)\+S\_\{i\}\}\{\\beta\+K\_\{i\}\},\(4\)wherem0m\_\{0\}is the no\-memory root,ρ\\rhocontrols the strength of a root\-relative prior, andβ\\betais a pseudo\-count\. The generation action is scored by

UCBigen=μ^i\+Δ^i\+cg​log⁡Nβ\+Ki,\\mathrm\{UCB\}^\{\\mathrm\{gen\}\}\_\{i\}=\\hat\{\\mu\}\_\{i\}\+\\hat\{\\Delta\}\_\{i\}\+c\_\{g\}\\sqrt\{\\frac\{\\log N\}\{\\beta\+K\_\{i\}\}\},\(5\)wherecgc\_\{g\}controls exploration over parent nodes\.

To avoid collapsing into a single lineage,AutoMMemouses a minimum\-width constraint\. The root is always expandable, while a non\-root node becomes expandable only after its parent has generated at leastBBchildren:

ℰ=\{0\}∪\{i∈𝒯∖\{0\}:\|Children​\(Parent​\(i\)\)\|≥B\}\.\\mathcal\{E\}=\\\{0\\\}\\cup\\left\\\{i\\in\\mathcal\{T\}\\setminus\\\{0\\\}:\|\\mathrm\{Children\}\(\\mathrm\{Parent\}\(i\)\)\|\\geq B\\right\\\}\.\(6\)At each round,AutoMMemoselects the highest\-scoring action among all evaluation actions and eligible generation actions\. It then either re\-evaluates an existing node or generates, repairs, evaluates, and inserts a new child\.

After the search budget is exhausted,AutoMMemoselects a robust final program using a lower confidence bound:

m∗=arg⁡maxi∈𝒯⁡\(μ^i−ce​log⁡Nni\)\.m^\{\*\}=\\arg\\max\_\{i\\in\\mathcal\{T\}\}\\left\(\\hat\{\\mu\}\_\{i\}\-c\_\{e\}\\sqrt\{\\frac\{\\log N\}\{n\_\{i\}\}\}\\right\)\.\(7\)This favors programs that are both high\-performing and sufficiently verified, yielding a robust memory strategy for learning from multimodal experience\.

## 4Experiments

### 4\.1Experimental Setup

Benchmarks\.We evaluateAutoMMemoon four benchmarks covering GUI/Web navigation and multimodal visual reasoning: WebVoyager\(Heet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib35)\), Mind2Web\(Denget al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib37)\), AgentVista\(Suet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib36)\), and MMSearch\-Plus\(Taoet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib38)\)\. Detailed per\-benchmark statistics and dataset split details are provided in Appendix[D](https://arxiv.org/html/2605.16857#A4)\.

Models\.We instantiate the execution agent with both open\-source and closed\-source backbones, including Qwen3\-VL\-32B\(Baiet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib39)\), GPT\-5\.4\-nano\(Singhet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib40)\), and Qwen3\.5\-Plus\(Baiet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib39)\)\. Unless otherwise specified, the meta agent is GPT\-5\(Singhet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib40)\)\. We use GPT\-5\.4\-mini\(Singhet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib40)\)as the LLM judge for all benchmarks\. The two GUI/Web navigation benchmarks share the same evaluation prompt, while the two visual reasoning/search benchmarks share another prompt tailored to answer correctness and evidence consistency\.

Baselines\.We compare with three groups of memory baselines\. Text\-based baselines include Trajectory Retrieval\(Parket al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib41); Xuet al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib42)\), ReasoningBank\(Ouyanget al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib43)\), and G\-Memory\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.16857#bib.bib44)\)\. Multimodal baselines include XSkill\(Jianget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib45)\)and M2\(Yanet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib46)\)\. We also compare with ALMA\(Xionget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib47)\), which we adapt as a text\-based automatic memory\-design baseline under our evaluation protocol\. For a fair comparison under the same sequential search budget, both ALMA andAutoMMemoare allocated 20 search steps; for ALMA, we sample one candidate design per step to match our sequential design\-search protocol\. Detailed baseline implementations and adaptations are provided in Appendix[E](https://arxiv.org/html/2605.16857#A5)\.

Evaluation protocol\.All methods start from an empty memory for each benchmark\. Memory design/search and memory construction are performed independently on the training split of each benchmark, without mixing episodes across benchmarks\. The resulting memory system is evaluated on the held\-out test split of the same benchmark without further updates, resulting in an offline evaluation protocol\. Each method is evaluated three times on each benchmark, and we report the mean score\. Task failures are counted as zero, while infrastructure\-level invalid runs, such as browser crashes, page loading failures, or API errors, are excluded and rerun when possible\. We report success rate for GUI/Web navigation benchmarks and judge\-based accuracy for visual reasoning benchmarks\. Following the evaluation protocols of WebVoyager\(Heet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib35)\)and AgentVista\(Suet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib36)\), we use benchmark\-family\-specific LLM\-as\-judge prompts for GUI/Web navigation and visual reasoning tasks, respectively\. The evaluator receives the task instruction, trajectory, final answer or state, and a bounded number of screenshots, and outputs a binary correctness label\. Full prompts are provided in Appendix[K](https://arxiv.org/html/2605.16857#A11)\. AVG\. GUI and AVG\. VR are computed as simple macro\-averages over the corresponding benchmarks\.

Implementation details\.The search budget is set to 20 steps for bothAutoMMemoand ALMA\. During judging, the evaluator can access the task, the agent trajectory, the final answer or state, and up to five screenshots\. During execution, we allow up to eight images in the agent context\. Retrieved memories are capped at 50k characters and two images\. Unless otherwise specified, all methods use the same execution budget, retrieval budget, and evaluation protocol\. Additional hyperparameters, concurrency settings, model identifiers, system details, and all prompt templates are provided in Appendix[F](https://arxiv.org/html/2605.16857#A6)and Appendix[K](https://arxiv.org/html/2605.16857#A11)\.

Table 1:Main results on GUI\-agent and visual\-reasoning benchmarks across different execution models\. Numbers in parentheses indicate absolute performance changes over the corresponding NoMemory baseline, and bold denotes the best result within each execution\-model block\.
### 4\.2Main Results

Table[1](https://arxiv.org/html/2605.16857#S4.T1)comparesAutoMMemowith text\-based, multimodal\-based, and automatic memory design baselines across GUI\-agent and visual\-reasoning benchmarks\. We report the main results on Qwen3\-VL\-32B and GPT\-5\.4\-nano in the main text, and provide the results on Qwen3\.5\-Plus in Appendix[G](https://arxiv.org/html/2605.16857#A7)\.

AutoMMemoconsistently improves multimodal agents\.Across both execution agents,AutoMMemoachieves the best average performance on GUI navigation and visual reasoning tasks\. Compared with the NoMemory baseline,AutoMMemoimproves AVG\. GUI by 19\.23 and 5\.21 points on Qwen3\-VL\-32B and GPT\-5\.4\-nano, respectively, and improves AVG\. VR by 4\.60 and 2\.55 points\. These results show that automatically discovered memory mechanisms can effectively transform past multimodal trajectories into useful task context\.

Automatic memory design is more robust than fixed memory schemas\.Existing memory methods often improve certain benchmarks but degrade others, indicating that manually designed memory formats are sensitive to task type and execution model\. For example, several text\-based and multimodal\-based baselines improve individual GUI tasks but fail to consistently benefit visual reasoning\. In contrast,AutoMMemoachieves stable gains across both task categories, demonstrating the advantage of adapting the memory mechanism itself rather than relying on fixed experience representations or retrieval rules\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x2.png)Figure 2:Cross\-task transferability of the searched memory under Qwen3\-VL\-32B as the execution agent\. Each subplot corresponds to a target benchmark, and each bar reports the performance obtained by applying the memory design searched on a source benchmark to that target benchmark\.
### 4\.3Transfer Analysis

We further examine whether the searched memory transfer across benchmarks and execution models\. As shown in Figures[2](https://arxiv.org/html/2605.16857#S4.F2)and[3](https://arxiv.org/html/2605.16857#S4.F3), the learned memory mechanisms exhibit clear transferability, but the strongest performance is typically achieved when the memory is searched under the target setting\.

Cross\-benchmark transfer\.Figure[2](https://arxiv.org/html/2605.16857#S4.F2)evaluates memory designs searched on one benchmark and applied to other benchmarks, using Qwen3\-VL\-32B as the execution agent\. The results show that transferred memory designs often remain competitive and can outperform both NoMemory and the best fixed baseline\. However, the best results generally appear on the diagonal, where the search and evaluation benchmarks match\. This indicates that the discovered memory mechanisms capture reusable multimodal experience structures, but still need to adapt to benchmark\-specific requirements, such as GUI interaction patterns or visual\-evidence organization\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x3.png)Figure 3:Cross\-model memory transfer on the WebVoyager benchmark\. Rows indicate evaluation models and columns indicate the models used to search memory designs\.Cross\-model transfer\.Figure[3](https://arxiv.org/html/2605.16857#S4.F3)evaluates transfer across execution models on WebVoyager\. Each cell reports task performance, with the improvement over the corresponding NoMemory baseline in parentheses\. Off\-diagonal results evaluate whether a searched memory design can generalize across execution models\. All off\-diagonal entries improve over the corresponding NoMemory baselines, showing that searched memory designs are not tied to a single backbone\. Nevertheless, diagonal entries remain the strongest for all models, suggesting that different execution agents benefit from different memory behaviors due to variations in visual grounding, instruction following, and retrieval usage\. Together, these results show thatAutoMMemolearns memory designs with both reusable and target\-adaptive components\.

### 4\.4Cost Analysis

Table[2](https://arxiv.org/html/2605.16857#S4.T2)analyzes the inference\-time cost of different memory settings on Qwen3\-VL\-32B, including task performance, token consumption, and interaction steps\. The results show that the gains ofAutoMMemodo not come from using more context or longer trajectories\.

AutoMMemoimproves the performance–cost trade\-off\.AutoMMemoachieves the best performance on all four benchmarks while keeping token usage and interaction steps among the lowest\. For example, on WebVoyager, it improves success rate from 36\.32 to 51\.84, while reducing token consumption from 307\.0K to 98\.0K and interaction steps from 11\.31 to 9\.66\. Similar trends appear on Mind2Web, AgentVista, and MMSearch\-Plus, indicating thatAutoMMemoimproves execution efficiency by retrieving more task\-relevant memory rather than injecting more context\.

Fixed memory designs often introduce inefficient context\.Prior memory methods do not consistently improve the performance–cost trade\-off\. Several baselines consume more tokens than NoMemory but yield limited gains or even performance drops, suggesting that fixed memory schemas may introduce noisy or redundant context\. Conversely, lower cost alone is also insufficient: M2often uses very few tokens or steps, but its performance remains belowAutoMMemo\. Overall,AutoMMemoachieves a stronger Pareto trade\-off between accuracy and inference efficiency\.

Search\-time overhead\.The above analysis focuses on inference\-time cost\. We report offline search\-time overhead in Appendix[H](https://arxiv.org/html/2605.16857#A8); under the same 20\-step sequential search budget as ALMA,AutoMMemouses fewer total search\-time tokens and less wall\-clock time, showing that its gains are not obtained from a larger offline design budget\.

Table 2:Cost and efficiency analysis under Qwen3\-VL\-32B\. Perf\. denotes the final task performance, Tok\. reports the average number of tokens per episode in thousands, and \#Steps reports the average number of agent interaction steps\. Bold and underline indicate the best and second\-best results within each benchmark and metric, respectively\.![Refer to caption](https://arxiv.org/html/2605.16857v1/x4.png)Figure 4:Memory\-design search trajectory for theWebVoyagersearch run underQwen3\-VL\-32B\. Each point represents a memory design evaluated or re\-evaluated at a search round, and the node labels correspond to the search tree in Appendix[I](https://arxiv.org/html/2605.16857#A9)\. The dashed curve tracks the best validation performance found so far\. Failed rounds denote candidates rejected by quick examination before full evaluation\. The final memory program selected by the LCB rule is N11\.
### 4\.5Search Process and Memory Evolution

We further examine howAutoMMemoimproves memory designs during search\. Figure[4](https://arxiv.org/html/2605.16857#S4.F4)shows a representative search trajectory onWebVoyagerwithQwen3\-VL\-32B, where node labels correspond to the search tree in Appendix[I](https://arxiv.org/html/2605.16857#A9); Appendix[J](https://arxiv.org/html/2605.16857#A10)visualizes the final selected memo programs across benchmarks\. Starting from NoMemory,AutoMMemoquickly discovers stronger memory programs, and the best validation performance continues to improve as search proceeds\. The final program selected by the LCB rule is N11\.

The trajectory also shows that search is not monotonic: some generated candidates underperform the current best program, and one candidate fails the quick examination before full evaluation\. This indicates thatAutoMMemoexplores diverse memory designs rather than merely applying greedy local refinements\. Through evaluation feedback and reflection\-guided mutation, the search gradually identifies more task\-conditioned memory mechanisms that organize relevant experience, filter noisy context, and provide useful guidance to the execution agent\.

## 5Conclusion

In this paper, we studied how multimodal agents can better learn from their own experience by treating memory design as an adaptive optimization problem rather than a fixed human\-designed component\. We introducedAutoMMemo, which represents memory mechanisms as executable memo programs and searches for effective update and retrieval strategies through evaluation, reflection, mutation, and budget\-aware tree search\. Experiments on GUI/web navigation and multimodal visual reasoning benchmarks show thatAutoMMemoconsistently outperforms no\-memory agents, manually designed memory baselines, and text\-only automatic memory\-design methods\. Further analyses demonstrate that the learned memory designs improve not only task performance, but also inference efficiency, often reducing token usage and interaction steps\. Transfer and search\-process results further indicate thatAutoMMemodiscovers reusable yet task\-adaptive memory structures\. Overall, our findings highlight memory design as a key optimization target for multimodal agents\. Beyond improving foundation models or enlarging experience buffers, effective agent learning requires adaptive mechanisms for deciding how experience should be represented, organized, and reused\.

## References

- Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§F\.5](https://arxiv.org/html/2605.16857#A6.SS5.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p2.1)\.
- W\. Bo, S\. Zhang, Y\. Sun, J\. Wu, Q\. Xie, X\. Tan, K\. Chen, W\. He, X\. Li, N\. Zhao,et al\.\(2025\)Agentic learner with grow\-and\-refine multimodal semantic memory\.arXiv preprint arXiv:2511\.21678\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p3.1),[§2](https://arxiv.org/html/2605.16857#S2.p3.1)\.
- M\. Chen, Y\. Li, Y\. Yang, S\. Yu, B\. Lin, and X\. He \(2024\)Automanual: constructing instruction manuals by llm agents via interactive environmental learning\.Advances in Neural Information Processing Systems37,pp\. 589–631\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- Z\. Chen, T\. Wu, Z\. Zhou, Z\. Xie, Z\. Weng, and Y\. Zhang \(2026\)PolarMem: a training\-free polarized latent graph memory for verifiable multimodal agents\.arXiv preprint arXiv:2602\.00415\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p3.1)\.
- P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav \(2025\)Mem0: building production\-ready ai agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p3.1)\.
- N\. J\. Cohen and H\. Eichenbaum \(1993\)Memory, amnesia, and the hippocampal system\.MIT press\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p4.1)\.
- X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su \(2023\)Mind2web: towards a generalist agent for the web\.Advances in Neural Information Processing Systems36,pp\. 28091–28114\.Cited by:[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p1.1)\.
- S\. Dou, M\. Zhang, C\. Huang, J\. Chen, F\. Chen, S\. Liu, Y\. Liu, C\. Liu, C\. ZHONG, Z\. Zhang,et al\.\(2025\)EvaLearn: quantifying the learning capability and efficiency of llms via sequential problem solving\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1)\.
- D\. Driess, F\. Xia, M\. S\. Sajjadi, C\. Lynch, A\. Chowdhery, B\. Ichter, A\. Wahid, J\. Tompson, Q\. Vuong, T\. Yu,et al\.\(2023\)PaLM\-e: an embodied multimodal language model\.InProceedings of the 40th International Conference on Machine Learning,pp\. 8469–8488\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p2.1)\.
- Z\. Durante, Q\. Huang, N\. Wake, R\. Gong, J\. S\. Park, B\. Sarkar, R\. Taori, Y\. Noda, D\. Terzopoulos, Y\. Choi,et al\.\(2024\)Agent ai: surveying the horizons of multimodal interaction\.arXiv preprint arXiv:2401\.03568\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1)\.
- H\. Eichenbaum \(2017\)Memory: organization and control\.Annual review of psychology68,pp\. 19–45\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p4.1)\.
- J\. Fang, Y\. Peng, X\. Zhang, Y\. Wang, X\. Yi, G\. Zhang, Y\. Xu, B\. Wu, S\. Liu, Z\. Li,et al\.\(2025\)A comprehensive survey of self\-evolving ai agents: a new paradigm bridging foundation models and lifelong agentic systems\.arXiv preprint arXiv:2508\.07407\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1),[§2](https://arxiv.org/html/2605.16857#S2.p1.1)\.
- Y\. Fu, D\. Kim, J\. Kim, S\. Sohn, L\. Logeswaran, K\. Bae, and H\. Lee \(2024\)Autoguide: automated generation and selection of context\-aware guidelines for large language model agents\.Advances in Neural Information Processing Systems37,pp\. 119919–119948\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- H\. Gao, J\. Geng, W\. Hua, M\. Hu, X\. Juan, H\. Liu, S\. Liu, J\. Qiu, X\. Qi, Y\. Wu,et al\.\(2025\)A survey of self\-evolving agents: on path to artificial super intelligence\.arXiv preprint arXiv:2507\.21046\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1),[§2](https://arxiv.org/html/2605.16857#S2.p1.1)\.
- H\. He, W\. Yao, K\. Ma, W\. Yu, Y\. Dai, H\. Zhang, Z\. Lan, and D\. Yu \(2024\)Webvoyager: building an end\-to\-end web agent with large multimodal models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6864–6890\.Cited by:[§K\.3](https://arxiv.org/html/2605.16857#A11.SS3.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p4.1)\.
- Y\. Hu, S\. Liu, Y\. Yue, G\. Zhang, B\. Liu, F\. Zhu, J\. Lin, H\. Guo, S\. Dou, Z\. Xi,et al\.\(2025\)Memory in the age of ai agents\.arXiv preprint arXiv:2512\.13564\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p2.1)\.
- C\. Huang, H\. Zhang, J\. Xu, Z\. Zheng, and Y\. Lin \(2026\)Bilevel optimization of agent skills via monte carlo tree search\.arXiv preprint arXiv:2604\.15709\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p5.1)\.
- G\. Jiang, Z\. Su, X\. Qu, and Y\. R\. Fung \(2026\)Xskill: continual learning from experience and skills in multimodal agents\.arXiv preprint arXiv:2603\.12056\.Cited by:[§E\.2](https://arxiv.org/html/2605.16857#A5.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.16857#S1.p3.1),[§2](https://arxiv.org/html/2605.16857#S2.p3.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§F\.5](https://arxiv.org/html/2605.16857#A6.SS5.p2.1)\.
- R\. Li, Y\. Zhai, B\. Xu, L\. Xu, N\. Shi, W\. Zhang, R\. Lin, and L\. Wang \(2025a\)EchoTrail\-gui: building actionable memory for gui agents via critic\-guided self\-exploration\.arXiv preprint arXiv:2512\.19396\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p3.1)\.
- Y\. Li, Z\. Liu, Z\. Li, X\. Zhang, Z\. Xu, X\. Chen, H\. Shi, S\. Jiang, X\. Wang, J\. Wang,et al\.\(2025b\)Perception, reason, think, and plan: a survey on large multimodal reasoning models\.arXiv preprint arXiv:2505\.04921\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1)\.
- J\. Liu, Z\. Ling, S\. Qiu, Y\. Liu, S\. Han, P\. Xia, H\. Tu, Z\. Zheng, C\. Xie, C\. Fleming,et al\.\(2026\)Omni\-simplemem: autoresearch\-guided discovery of lifelong multimodal agent memory\.arXiv e\-prints,pp\. arXiv–2604\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p6.1)\.
- J\. Liu, Y\. Sun, W\. Cheng, H\. Lei, Y\. Chen, L\. Wen, X\. Yang, D\. Fu, P\. Cai, N\. Deng,et al\.\(2025\)Memverse: multimodal memory for lifelong learning agents\.arXiv preprint arXiv:2512\.03627\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p3.1)\.
- S\. Ouyang, J\. Yan, I\. Hsu, Y\. Chen, K\. Jiang, Z\. Wang, R\. Han, L\. T\. Le, S\. Daruki, X\. Tang,et al\.\(2025\)Reasoningbank: scaling agent self\-evolving with reasoning memory\.arXiv preprint arXiv:2509\.25140\.Cited by:[§E\.1](https://arxiv.org/html/2605.16857#A5.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- W\. Pan, S\. Liu, X\. Zhou, S\. Zhang, W\. Shi, M\. Xu, and X\. Jia \(2026\)M\-star: every task deserves its own memory harness\.arXiv preprint arXiv:2604\.11811\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p5.1)\.
- J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th annual acm symposium on user interface software and technology,pp\. 1–22\.Cited by:[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga,et al\.\(2019\)Pytorch: an imperative style, high\-performance deep learning library\.Advances in neural information processing systems32\.Cited by:[§F\.5](https://arxiv.org/html/2605.16857#A6.SS5.p2.1)\.
- G\. Sarch, L\. Jang, M\. J\. Tarr, W\. W\. Cohen, K\. Marino, and K\. Fragkiadaki \(2024\)Vlm agents generate their own memories: distilling experience into embodied programs of thought\.Advances in Neural Information Processing Systems37,pp\. 75942–75985\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p2.1),[§2](https://arxiv.org/html/2605.16857#S2.p3.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1),[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- D\. Silver and R\. S\. Sutton \(2025\)Welcome to the era of experience\.Google AI1\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§F\.5](https://arxiv.org/html/2605.16857#A6.SS5.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p2.1)\.
- Z\. Su, J\. Gao, H\. Guo, Z\. Liu, L\. Zhang, X\. Geng, S\. Huang, P\. Xia, G\. Jiang, C\. Wang,et al\.\(2026\)Agentvista: evaluating multimodal agents in ultra\-challenging realistic visual scenarios\.arXiv preprint arXiv:2602\.23166\.Cited by:[§K\.3](https://arxiv.org/html/2605.16857#A11.SS3.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p4.1)\.
- X\. Tao, Y\. Teng, X\. Su, X\. Fu, J\. Wu, C\. Tao, Z\. Liu, H\. Bai, R\. Liu, and L\. Kong \(2025\)Mmsearch\-plus: benchmarking provenance\-aware search for multimodal browsing agents\.arXiv preprint arXiv:2508\.21475\.Cited by:[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig \(2025\)Agent workflow memory\.InInternational Conference on Machine Learning,pp\. 63897–63911\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- P\. Xia, J\. Chen, H\. Wang, J\. Liu, K\. Zeng, Y\. Wang, S\. Han, Y\. Zhou, X\. Zhao, H\. Chen,et al\.\(2026\)SkillRL: evolving agents via recursive skill\-augmented reinforcement learning\.InICLR 2026 Workshop on Memory for LLM\-Based Agentic Systems,Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p2.1),[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- Y\. Xiong, S\. Hu, and J\. Clune \(2026\)Learning to continually learn via meta\-learning agentic memory designs\.arXiv preprint arXiv:2602\.07755\.Cited by:[§E\.3](https://arxiv.org/html/2605.16857#A5.SS3.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.16857#S2.p5.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- P\. Xu, W\. Ping, X\. Wu, L\. McAfee, C\. Zhu, Z\. Liu, S\. Subramanian, E\. Bakhturina, M\. Shoeybi, and B\. Catanzaro \(2023\)Retrieval meets long context large language models\.InThe Twelfth international conference on learning representations,Cited by:[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- D\. Yan, H\. Zhang, G\. Huzhang, Y\. Li, Y\. Wang, Q\. Chen, Z\. Xu, W\. Luo, Y\. Li, W\. Dong, and C\. Shen \(2026\)M2: dual\-memory augmentation for long\-horizon web agents via trajectory summarization and insight retrieval\.External Links:2603\.00503,[Link](https://arxiv.org/abs/2603.00503)Cited by:[§E\.2](https://arxiv.org/html/2605.16857#A5.SS2.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p1.1)\.
- S\. Yuan, J\. Wang, Z\. Liu, M\. Yuan, H\. Peng, J\. Zhao, B\. Wang, and H\. Wang \(2026\)TED: training\-free experience distillation for multimodal reasoning\.arXiv preprint arXiv:2603\.26778\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p3.1)\.
- G\. Zhang, M\. Fu, G\. Wan, M\. Yu, K\. Wang, and S\. Yan \(2025a\)G\-memory: tracing hierarchical memory for multi\-agent systems\.arXiv preprint arXiv:2506\.07398\.Cited by:[§E\.1](https://arxiv.org/html/2605.16857#A5.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.16857#S1.p2.1),[§2](https://arxiv.org/html/2605.16857#S2.p2.1),[§4\.1](https://arxiv.org/html/2605.16857#S4.SS1.p3.1)\.
- G\. Zhang, H\. Ren, C\. Zhan, Z\. Zhou, J\. Wang, H\. Zhu, W\. Zhou, and S\. Yan \(2025b\)Memevolve: meta\-evolution of agent memory systems\.arXiv preprint arXiv:2512\.18746\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p5.1)\.
- H\. Zhang, S\. Fan, H\. P\. Zou, Y\. Chen, Z\. Wang, J\. Zhou, C\. Li, W\. Huang, Y\. Yao, K\. Zheng,et al\.\(2026\)EvoSkills: self\-evolving agent skills via co\-evolutionary verification\.arXiv preprint arXiv:2604\.01687\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p5.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024\)Expel: llm agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p2.1),[§2](https://arxiv.org/html/2605.16857#S2.p2.1)\.
- H\. Zhou, S\. Guo, A\. Liu, Z\. Yu, Z\. Gong, B\. Zhao, Z\. Chen, M\. Zhang, Y\. Chen, J\. Li,et al\.\(2026\)Memento\-skills: let agents design agents\.arXiv preprint arXiv:2603\.18743\.Cited by:[§2](https://arxiv.org/html/2605.16857#S2.p5.1)\.
- S\. Zhu, W\. Wu, K\. Zhou, S\. Wang, and B\. Huang \(2026\)Hybrid self\-evolving structured memory for gui agents\.arXiv preprint arXiv:2603\.10291\.Cited by:[§1](https://arxiv.org/html/2605.16857#S1.p3.1)\.

## Appendix ALimitations

AlthoughAutoMMemodemonstrates consistent gains across GUI/web navigation and multimodal visual reasoning benchmarks, several limitations remain\.

First, our experiments are conducted on a finite set of benchmark environments\. While these tasks cover diverse multimodal interaction patterns, they do not fully represent more open\-ended deployment scenarios, such as long\-horizon embodied interaction, multi\-user environments, or continuously changing task distributions\. Evaluating whether the searched memo programs remain effective in such non\-stationary settings is an important direction for future work\.

Second, our current protocol optimizes memory design in an offline setting\. For each benchmark, memory construction and search are performed on the training split, and the resulting memory system is then evaluated on held\-out tasks without further adaptation\. This controlled setup enables fair comparison across candidate memory designs, but it does not fully capture continual deployment settings where new experiences arrive over time\. ExtendingAutoMMemoto support stable online memory evolution remains an open challenge\.

Third, although the final learned memory systems can reduce inference\-time token usage and interaction steps, the search process itself still requires multiple evaluations of candidate memo programs\. This offline cost may become significant when environment interaction is expensive or when much larger execution agents are used\. Future work could improve search efficiency through surrogate evaluation, partial trajectory reuse, adaptive early stopping, or stronger candidate filtering\.

Finally, multimodal memory introduces privacy and safety considerations\. Stored trajectories may contain screenshots, user\-provided content, web pages, or other sensitive information\. Moreover, because memo programs are executable, practical deployment should include sandboxing, permission control, resource limits, and mechanisms for inspecting, filtering, or deleting stored memories\.

## Appendix BImpact Statement

This work aims to improve how multimodal agents store, organize, and retrieve past experience\. Its potential positive impact is to make agents more reliable and efficient by reducing their dependence on manually engineered memory schemas\. Adaptive memory mechanisms may help agents reuse prior experience, avoid repeated mistakes, and reduce unnecessary context consumption in applications such as web navigation assistance, visual search, accessibility tools, and interactive tutoring\.

At the same time, stronger memory\-augmented agents introduce potential risks\. If multimodal trajectories are stored without sufficient safeguards, they may preserve sensitive user information, private visual content, or misleading historical context\. Retrieved memories could also bias future behavior if they are stale, irrelevant, or improperly filtered\. In addition, more persistent agent memory may increase the capability of autonomous systems in web environments, which could be misused for harmful automation\.

Responsible deployment therefore requires privacy\-preserving storage, user control over memory retention and deletion, careful filtering of sensitive content, and transparent logging of memory usage\. BecauseAutoMMemosearches over executable memo programs, deployment should also require sandboxed execution, restricted permissions, and validation before a searched memory mechanism is used outside controlled benchmark settings\.

## Appendix CAdditional Algorithm Details

This appendix provides the full training procedure ofAutoMMemo\. The main text introduces the memo\-program abstraction, the update\-then\-retrieve evaluation protocol, reflection\-guided mutation, and the budget\-aware tree search rule\. Here, we give the complete pseudocode and further implementation details\. In our experiments, both the update batch𝒟u\\mathcal\{D\}\_\{u\}and the retrieve batch𝒟r\\mathcal\{D\}\_\{r\}used during memory\-design search are drawn from the training split; the held\-out test split is used only for final offline evaluation\.

### C\.1Full Training Procedure

Algorithm[1](https://arxiv.org/html/2605.16857#alg1)summarizes the overall search procedure\. The search tree stores candidate memo programs, where each node corresponds to one executable memo program and each edge corresponds to a reflection\-guided mutation from a parent program\. At each search round,AutoMMemoeither re\-evaluates an existing node to reduce score uncertainty or generates a new child from an expandable parent\. Failed generations that do not pass quick examination are discarded and do not consume full\-evaluation budget\.

Algorithm 1AutoMMemo1:Update batch

𝒟u\\mathcal\{D\}\_\{u\}, retrieve batch

𝒟r\\mathcal\{D\}\_\{r\}, root memo program

m0m\_\{0\}, search budget

TT, constants

ce,cg,ρ,β,Bc\_\{e\},c\_\{g\},\\rho,\\beta,B, repair budget

LL
2:Initialize search tree

𝒯←\{0\}\\mathcal\{T\}\\leftarrow\\\{0\\\}
3:Set

Parent​\(0\)←∅\\mathrm\{Parent\}\(0\)\\leftarrow\\varnothing,

Children​\(0\)←∅\\mathrm\{Children\}\(0\)\\leftarrow\\varnothing
4:Set

K0←0K\_\{0\}\\leftarrow 0,

S0←0S\_\{0\}\\leftarrow 0
5:

r0←FullEval​\(m0,𝒟u,𝒟r\)r\_\{0\}\\leftarrow\\textsc\{FullEval\}\(m\_\{0\},\\mathcal\{D\}\_\{u\},\\mathcal\{D\}\_\{r\}\)
6:Set

μ^0←r0\\hat\{\\mu\}\_\{0\}\\leftarrow r\_\{0\},

n0←1n\_\{0\}\\leftarrow 1,

N←1N\\leftarrow 1
7:for

t=1,…,Tt=1,\\ldots,Tdo

8:Compute expandable set

ℰ\\mathcal\{E\}using Eq\. \([15](https://arxiv.org/html/2605.16857#A3.E15)\)

9:Initialize action set

𝒜t←∅\\mathcal\{A\}\_\{t\}\\leftarrow\\varnothing
10:foreach node

i∈𝒯i\\in\\mathcal\{T\}do

11:Compute

UCBieval\\mathrm\{UCB\}^\{\\mathrm\{eval\}\}\_\{i\}using Eq\. \([13](https://arxiv.org/html/2605.16857#A3.E13)\)

12:Add

\(Evaluate,i,UCBieval\)\(\\textsc\{Evaluate\},i,\\mathrm\{UCB\}^\{\\mathrm\{eval\}\}\_\{i\}\)to

𝒜t\\mathcal\{A\}\_\{t\}
13:endfor

14:foreach node

i∈ℰi\\in\\mathcal\{E\}do

15:Compute

Δ^i\\hat\{\\Delta\}\_\{i\}using Eq\. \([12](https://arxiv.org/html/2605.16857#A3.E12)\)

16:Compute

UCBigen\\mathrm\{UCB\}^\{\\mathrm\{gen\}\}\_\{i\}using Eq\. \([14](https://arxiv.org/html/2605.16857#A3.E14)\)

17:Add

\(Generate,i,UCBigen\)\(\\textsc\{Generate\},i,\\mathrm\{UCB\}^\{\\mathrm\{gen\}\}\_\{i\}\)to

𝒜t\\mathcal\{A\}\_\{t\}
18:endfor

19:Select

\(at,it,⋅\)←arg⁡max\(a,i,u\)∈𝒜t⁡u\(a\_\{t\},i\_\{t\},\\cdot\)\\leftarrow\\arg\\max\_\{\(a,i,u\)\\in\\mathcal\{A\}\_\{t\}\}u
20:if

at=Evaluatea\_\{t\}=\\textsc\{Evaluate\}then

21:

r←FullEval​\(mit,𝒟u,𝒟r\)r\\leftarrow\\textsc\{FullEval\}\(m\_\{i\_\{t\}\},\\mathcal\{D\}\_\{u\},\\mathcal\{D\}\_\{r\}\)
22:

μ^it←\(nit​μ^it\+r\)/\(nit\+1\)\\hat\{\\mu\}\_\{i\_\{t\}\}\\leftarrow\(n\_\{i\_\{t\}\}\\hat\{\\mu\}\_\{i\_\{t\}\}\+r\)/\(n\_\{i\_\{t\}\}\+1\)
23:

nit←nit\+1n\_\{i\_\{t\}\}\\leftarrow n\_\{i\_\{t\}\}\+1,

N←N\+1N\\leftarrow N\+1
24:else

25:

μ^itsnap←μ^it\\hat\{\\mu\}^\{\\mathrm\{snap\}\}\_\{i\_\{t\}\}\\leftarrow\\hat\{\\mu\}\_\{i\_\{t\}\}
26:

fit←Reflect​\(mit\)f\_\{i\_\{t\}\}\\leftarrow\\textsc\{Reflect\}\(m\_\{i\_\{t\}\}\)
27:

mj←MutateAndRepair​\(mit,fit,L\)m\_\{j\}\\leftarrow\\textsc\{MutateAndRepair\}\(m\_\{i\_\{t\}\},f\_\{i\_\{t\}\},L\)
28:if

mjm\_\{j\}passes quick examinationthen

29:

rj←FullEval​\(mj,𝒟u,𝒟r\)r\_\{j\}\\leftarrow\\textsc\{FullEval\}\(m\_\{j\},\\mathcal\{D\}\_\{u\},\\mathcal\{D\}\_\{r\}\)
30:Add a new node

jjto

𝒯\\mathcal\{T\}
31:

Parent​\(j\)←it\\mathrm\{Parent\}\(j\)\\leftarrow i\_\{t\}
32:

Children​\(it\)←Children​\(it\)∪\{j\}\\mathrm\{Children\}\(i\_\{t\}\)\\leftarrow\\mathrm\{Children\}\(i\_\{t\}\)\\cup\\\{j\\\}
33:Initialize

Children​\(j\)←∅\\mathrm\{Children\}\(j\)\\leftarrow\\varnothing
34:Set

μ^j←rj\\hat\{\\mu\}\_\{j\}\\leftarrow r\_\{j\},

nj←1n\_\{j\}\\leftarrow 1,

Kj←0K\_\{j\}\\leftarrow 0,

Sj←0S\_\{j\}\\leftarrow 0
35:

Δit→j←max⁡\(0,μ^j−μ^itsnap\)\\Delta\_\{i\_\{t\}\\rightarrow j\}\\leftarrow\\max\(0,\\hat\{\\mu\}\_\{j\}\-\\hat\{\\mu\}^\{\\mathrm\{snap\}\}\_\{i\_\{t\}\}\)
36:

Sit←Sit\+Δit→jS\_\{i\_\{t\}\}\\leftarrow S\_\{i\_\{t\}\}\+\\Delta\_\{i\_\{t\}\\rightarrow j\}
37:

Kit←Kit\+1K\_\{i\_\{t\}\}\\leftarrow K\_\{i\_\{t\}\}\+1,

N←N\+1N\\leftarrow N\+1
38:endif

39:endif

40:endfor

41:return

m∗=arg⁡maxi∈𝒯⁡LCBievalm^\{\*\}=\\arg\\max\_\{i\\in\\mathcal\{T\}\}\\mathrm\{LCB\}^\{\\mathrm\{eval\}\}\_\{i\}

### C\.2Full Evaluation

The procedureFullEval​\(m,𝒟u,𝒟r\)\\textsc\{FullEval\}\(m,\\mathcal\{D\}\_\{u\},\\mathcal\{D\}\_\{r\}\)evaluates a memo program under the update\-then\-retrieve protocol\. Before evaluation, the internal memory state ofmmis reset\. The program then processes the update batch by callingm\.update​\(eτ\)m\.\\mathrm\{update\}\(e\_\{\\tau\}\)for each update episode recordereτ∈𝒟ue\_\{\\tau\}\\in\\mathcal\{D\}\_\{u\}\. After all update episodes have been processed, the memory state is frozen\.

For each retrieve\-time taskx∈𝒟rx\\in\\mathcal\{D\}\_\{r\}, the memo program receives a partial recorderexe\_\{x\}containing the task input and initial observations\. It returns a memory payloadpx=m\.retrieve​\(ex\)p\_\{x\}=m\.\\mathrm\{retrieve\}\(e\_\{x\}\)\. The payload is truncated according to the fixed retrieval budget and then injected into the execution agent\. The evaluator returns a scalar rewardRx∈\[0,1\]R\_\{x\}\\in\[0,1\]\. The full\-evaluation score is

FullEval​\(m,𝒟u,𝒟r\)=1\|𝒟r\|​∑x∈𝒟rRx\.\\textsc\{FullEval\}\(m,\\mathcal\{D\}\_\{u\},\\mathcal\{D\}\_\{r\}\)=\\frac\{1\}\{\|\\mathcal\{D\}\_\{r\}\|\}\\sum\_\{x\\in\\mathcal\{D\}\_\{r\}\}R\_\{x\}\.\(8\)
During full evaluation,AutoMMemorecords the retrieved payload, retrieved images, execution trajectory, screenshots, intermediate observations, errors, and final outcome\. These records are used by the reflection step to diagnose how the current memory mechanism affects downstream agent behavior\.

### C\.3Reflection, Mutation, and Repair

The reflection step analyzes an evaluated memo program using both its source code and its execution evidence\. Given a nodeii,Reflect​\(mi\)\\textsc\{Reflect\}\(m\_\{i\}\)receives the memo program, its empirical score, retrieved memory payloads, representative task trajectories, screenshots, retrieved images, and error cases\. It outputs a structured feedback objectfif\_\{i\}\. This feedback identifies failure modes such as missing memories, irrelevant retrievals, redundant payloads, misleading examples, weak text\-image alignment, unstable retrieval keys, insufficient visual summarization, or excessive truncation\.

The mutation step then produces a child program conditioned on the parent program and the feedback:

mj∼Mutate​\(mi,fi\)\.m\_\{j\}\\sim\\textsc\{Mutate\}\(m\_\{i\},f\_\{i\}\)\.\(9\)The generated child may change the memory schema, trajectory summarization rule, text\-image embedding strategy, indexing structure, retrieval ranking function, metadata filtering rule, or payload formatting strategy\. However, it must preserve the same update and retrieve interface as all other memo programs\.

Before full evaluation, the child program is checked by quick examination\. If the program fails, a repair model is invoked with the quick\-examination report and the failed test case\. The repair process is repeated for at mostLLattempts\. A child that still fails after repair is discarded and is not added to the search tree\.

Algorithm 2MutateAndRepair1:Parent memo program

mim\_\{i\}, reflection feedback

fif\_\{i\}, repair budget

LL
2:

m←Mutate​\(mi,fi\)m\\leftarrow\\textsc\{Mutate\}\(m\_\{i\},f\_\{i\}\)
3:for

ℓ=0,…,L\\ell=0,\\ldots,Ldo

4:

q←QuickExam​\(m\)q\\leftarrow\\textsc\{QuickExam\}\(m\)
5:if

q=Passq=\\textsc\{Pass\}then

6:return

mm
7:endif

8:if

ℓ<L\\ell<Lthen

9:

m←Repair​\(m,q\)m\\leftarrow\\textsc\{Repair\}\(m,q\)
10:endif

11:endfor

12:returninvalid program

### C\.4Quick Examination

Quick examination is a lightweight validation step that prevents invalid programs from consuming full\-evaluation budget\. It checks whether the generated program:

- •imports successfully;
- •implements the required update and retrieve interface;
- •runs on sample update episodes without errors;
- •runs on sample retrieve\-time inputs without errors;
- •returns a schema\-valid memory payload;
- •respects the maximum payload length and maximum number of retrieved images\.

Only programs that pass quick examination are eligible for full evaluation and insertion into the generation tree\.

### C\.5Generation Statistics

For each nodeii,AutoMMemomaintainsKiK\_\{i\}, the number of evaluated children generated fromii, andSiS\_\{i\}, the cumulative positive improvement obtained by these children\. When a generated childjjreceives scoreμ^j\\hat\{\\mu\}\_\{j\}, its positive improvement over the parent snapshot is

Δi→j=max⁡\(0,μ^j−μ^isnap\),\\Delta\_\{i\\rightarrow j\}=\\max\\left\(0,\\hat\{\\mu\}\_\{j\}\-\\hat\{\\mu\}^\{\\mathrm\{snap\}\}\_\{i\}\\right\),\(10\)whereμ^isnap\\hat\{\\mu\}^\{\\mathrm\{snap\}\}\_\{i\}denotes the empirical score of the parent at the time the child is generated\. The snapshot is used so that a child improvement remains fixed even if the parent is later re\-evaluated\.

For rarely expanded nodes, we use a root\-relative prior:

Δiprior=ρ​max⁡\(0,μ^i−μ^0\),\\Delta\_\{i\}^\{\\mathrm\{prior\}\}=\\rho\\max\\left\(0,\\hat\{\\mu\}\_\{i\}\-\\hat\{\\mu\}\_\{0\}\\right\),\(11\)wherem0m\_\{0\}is the no\-memory root andρ\\rhocontrols the prior strength\. The estimated local improvement potential of nodeiiis

Δ^i=β​Δiprior\+Siβ\+Ki\.\\hat\{\\Delta\}\_\{i\}=\\frac\{\\beta\\Delta\_\{i\}^\{\\mathrm\{prior\}\}\+S\_\{i\}\}\{\\beta\+K\_\{i\}\}\.\(12\)Here,β\\betais a pseudo\-count\. WhenKiK\_\{i\}is small, the estimate is influenced by the root\-relative prior; as more children are generated, it becomes dominated by observed improvements\.

### C\.6Action Scores and Expansion Constraint

For an existing nodeii, the evaluation action is scored by

UCBieval=μ^i\+ce​log⁡Nni,\\mathrm\{UCB\}^\{\\mathrm\{eval\}\}\_\{i\}=\\hat\{\\mu\}\_\{i\}\+c\_\{e\}\\sqrt\{\\frac\{\\log N\}\{n\_\{i\}\}\},\(13\)whereNNis the number of full evaluations consumed so far\. This action reduces uncertainty in the empirical quality of existing candidates\.

For an expandable nodeii, the generation action is scored by

UCBigen=μ^i\+Δ^i\+cg​log⁡Nβ\+Ki\.\\mathrm\{UCB\}^\{\\mathrm\{gen\}\}\_\{i\}=\\hat\{\\mu\}\_\{i\}\+\\hat\{\\Delta\}\_\{i\}\+c\_\{g\}\\sqrt\{\\frac\{\\log N\}\{\\beta\+K\_\{i\}\}\}\.\(14\)This score combines the current quality of the parent, its estimated local improvement potential, and an exploration bonus for under\-expanded parents\.

The set of expandable nodes is

ℰ=\{0\}∪\{i∈𝒯∖\{0\}:\|Children​\(Parent​\(i\)\)\|≥B\}\.\\mathcal\{E\}=\\\{0\\\}\\cup\\left\\\{i\\in\\mathcal\{T\}\\setminus\\\{0\\\}:\|\\mathrm\{Children\}\(\\mathrm\{Parent\}\(i\)\)\|\\geq B\\right\\\}\.\(15\)Thus, the root is always expandable, while a non\-root node becomes expandable only after its parent has generated at leastBBchildren\. This minimum\-width constraint encourages sibling diversity before the search grows deeper\.

At each round, the selected action is

\(at,it\)=arg⁡max\(a,i,u\)∈𝒜t⁡u,\(a\_\{t\},i\_\{t\}\)=\\arg\\max\_\{\(a,i,u\)\\in\\mathcal\{A\}\_\{t\}\}u,\(16\)where𝒜t\\mathcal\{A\}\_\{t\}contains all evaluation actions and all eligible generation actions\.

### C\.7Robust Final Selection

After the search budget is exhausted,AutoMMemoselects the final memo program using a lower confidence bound rather than the raw empirical score\. For each nodeii, define

LCBieval=μ^i−ce​log⁡Nni\.\\mathrm\{LCB\}^\{\\mathrm\{eval\}\}\_\{i\}=\\hat\{\\mu\}\_\{i\}\-c\_\{e\}\\sqrt\{\\frac\{\\log N\}\{n\_\{i\}\}\}\.\(17\)The returned program is

m∗=arg⁡maxi∈𝒯⁡LCBieval\.m^\{\*\}=\\arg\\max\_\{i\\in\\mathcal\{T\}\}\\mathrm\{LCB\}^\{\\mathrm\{eval\}\}\_\{i\}\.\(18\)This criterion reduces the chance of selecting a candidate whose empirical score is high only because it has been evaluated too few times\.

## Appendix DDataset Details

For benchmarks without an official training split, we construct train/test splits with an approximately 1:4 ratio while preserving the distribution of task categories, domains, or websites within each benchmark\. For Mind2Web, whose original scale is substantially larger than the other benchmarks, we sample websites to match the overall evaluation budget\. Specifically, we select cross\-task examples from the same websites as the sampled training split, and additionally sample websites from the cross\-website and cross\-domain splits\. Across the four benchmarks, the resulting splits contain 304 training tasks and 1,231 held\-out test tasks in total\.

For all benchmarks, the training split is used for both memory construction and memory\-design search, while the held\-out test split is used only for final offline evaluation\. During memory\-design search, the training split is further divided into an update batch𝒟u\\mathcal\{D\}\_\{u\}and a search\-validation retrieve batch𝒟r\\mathcal\{D\}\_\{r\}\. Candidate memo programs are evaluated on𝒟r\\mathcal\{D\}\_\{r\}only for search\-time model selection, and no held\-out test examples are used for memory construction, memory\-design search, or candidate selection\. All statistics reported below are computed from the final train/test split files used in our experiments\. Aggregate files and scratch files are excluded from the reported statistics\.

### D\.1Overall Statistics

Table[3](https://arxiv.org/html/2605.16857#A4.T3)summarizes the resulting train/test splits\. Across the four benchmarks, our splits contain 304 training tasks and 1,231 held\-out test tasks\. The number of domains or sites is computed from the union of the train and test files used for each benchmark\.

Table 3:Benchmark statistics\. The train split is used for memory construction and design search, while the held\-out test split is used only for final offline evaluation\.
### D\.2Split Construction

For AgentVista, MMSearch\-Plus, and WebVoyager, we group tasks by their task category, domain, or website, and sample approximately 20% of the tasks within each group for training\. The remaining tasks are used for held\-out testing\. This stratified procedure keeps the train/test distributions comparable while ensuring that final evaluation is performed on tasks that are not directly used during memory construction or design search\.

Mind2Web provides three evaluation settings: cross\-task, cross\-website, and cross\-domain\. Since its original scale is substantially larger than the other benchmarks, we sample websites to match the overall evaluation budget\. We include cross\-task examples from websites that also appear in the sampled training split, and additionally sample websites from the cross\-website and cross\-domain settings\. This preserves the intended generalization structure of Mind2Web while keeping the final evaluation scale comparable to the other benchmarks\.

### D\.3Mind2Web Evaluation Settings

Table[4](https://arxiv.org/html/2605.16857#A4.T4)reports the sampled Mind2Web sites under the three evaluation settings\. The cross\-task setting evaluates tasks from websites that are also observed in the training split, while the cross\-website and cross\-domain settings evaluate generalization to unseen websites or domains\.

Table 4:Mind2Web sampled sites by evaluation setting\.
### D\.4Domain and Site Distributions

Table[5](https://arxiv.org/html/2605.16857#A4.T5)reports the main domain or website distributions after combining the train and test splits used in our experiments\. For AgentVista and MMSearch\-Plus, the entries correspond to high\-level domains\. For Mind2Web and WebVoyager, the entries correspond to websites, and we list the top websites by number of tasks\. These distributions are used only for split construction and dataset analysis; they are not provided as task inputs to the agent\.

Table 5:Domain and site distributions of the sampled benchmark splits\.DatasetDomain/SiteTasksDatasetDomain/SiteTasksAgentVistacommerce42MMSearch\-PlusGeography64geography39Sports54entertainment39Academic Research50technology34Film & TV40society25Technology36academics15Video Games31culture15Vlog19––Music17Mind2Webbudget34WebVoyagerWolfram Alpha46reddit33Allrecipes45babycenter28ESPN44newegg28Apple43ticketcenter26Cambridge Dictionary43thumbtack24Huggingface43spothero24ArXiv43drugs23Coursera42finance\.yahoo23BBC News42healthline23Amazon41

### D\.5Fine\-Grained Annotations

AgentVista and MMSearch\-Plus additionally contain fine\-grained subdomain annotations\. We use these annotations only for stratified split construction and dataset analysis\. They are not provided as task inputs to the agent or to the memory system during training or evaluation\.

## Appendix EBaseline Details

This section provides implementation details for the baselines used in our experiments\. For fair comparison, all memory baselines are instantiated under the same update\-then\-retrieve protocol asAutoMMemo\. For each benchmark, a baseline first processes the training episodes to construct its memory, and is then evaluated on the held\-out test split without further memory updates\. Unless otherwise specified, all baselines use the same execution agent, train/test split, evaluation protocol, retrieval budget, and context budget asAutoMMemo\. In particular, retrieved memory contents are truncated to the same maximum payload length, and the number of returned visual references is restricted under the same image budget when the baseline supports multimodal memory\.

#### NoMemory\.

NoMemory is the base execution agent without any external memory\. The agent receives only the current task input, environment observations, and its own interaction history during execution\. This baseline measures the performance of the underlying execution agent before introducing any experience\-driven memory mechanism\.

### E\.1Text\-based Baselines

Text\-based baselines convert previous episodes into textual memories and retrieve relevant textual contents at test time\. These methods do not directly return visual references to the execution agent\. When the original method is not defined for our multimodal agent setting, we adapt it by extracting textual information from the episode recorder, including the task instruction, textual observations, action history, model messages, and final outcome\.

#### Trajectory Retrieval\.

Trajectory Retrieval stores previous interaction trajectories as retrievable examples\. During the update phase, each training episode is converted into a textual trajectory record, including the task instruction, important observations, executed actions, intermediate responses, and final outcome\. During retrieval, the current task is used as the query to select the most relevant trajectory records\. The retrieved records are formatted as demonstrations or reference cases and injected into the execution agent\. To ensure a fair comparison, retrieved trajectories are truncated to satisfy the same memory payload budget used byAutoMMemo\.

#### ReasoningBank\.

ReasoningBank\[Ouyanget al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib43)\]stores reusable reasoning memories distilled from past successful and failed experiences\. During the update phase, each episode is converted into generalizable reasoning knowledge, such as task\-solving strategies, failure patterns, corrective lessons, and reusable decision rules\. These memories are indexed by task description and episode metadata\. During retrieval, the current task is used to select relevant reasoning memories, which are then provided to the execution agent as textual guidance\. Compared with Trajectory Retrieval, this baseline uses more abstract and compressed reasoning memories rather than raw trajectory records\.

#### G\-Memory\.

G\-Memory\[Zhanget al\.,[2025a](https://arxiv.org/html/2605.16857#bib.bib44)\]organizes experience using a hierarchical graph memory\. Following its original design, we instantiate three types of textual memory structures corresponding to high\-level insights, task/query\-level records, and fine\-grained interaction traces\. During the update phase, training episodes are decomposed into these memory units and inserted into the graph structure\. During retrieval, the current task is matched against the graph memory, and relevant high\-level insights and condensed interaction records are retrieved\. The retrieved graph contents are linearized into text before being injected into the execution agent\.

### E\.2Multimodal Baselines

Multimodal baselines make use of visual information in addition to textual episode contents\. These methods are adapted to our episode\-recorder format, which may contain task instructions, screenshots, visual observations, action histories, model messages, and final outcomes\. When a baseline supports visual memory, we allow it to return visual references under the same maximum\-image budget asAutoMMemo\.

#### XSkill\.

XSkill\[Jianget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib45)\]is a dual\-stream continual learning framework that accumulates both experiences and skills from multimodal agent trajectories\. During the update phase, each training episode is used to extract action\-level experience and task\-level skill memories, grounded in the visual observations\. During retrieval, XSkill selects relevant experiences and skills conditioned on the current task and visual context\. The retrieved textual knowledge and visual references are then provided to the execution agent under the same retrieval budget asAutoMMemo\.

#### M2\\mathrm\{M\}^\{2\}\.

M2\\mathrm\{M\}^\{2\}\[Yanet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib46)\]is a dual\-memory framework for long\-horizon web agents that combines trajectory summarization and insight retrieval\. We adapt it to our offline update\-then\-retrieve evaluation protocol\. During the update phase, training episodes are converted into trajectory\-level summaries and higher\-level insights\. During retrieval, the current task and initial visual context are used to retrieve relevant summaries and insights from the constructed memory\. The retrieved textual contents and visual references are then provided to the execution agent at the beginning of the episode\.

Under this protocol, we use the long\-term memory retrieval component ofM2\\mathrm\{M\}^\{2\}, while disabling test\-time dynamic trajectory summarization\. This adaptation ensures that all baselines provide memory only through the same initial retrieval interface\.

### E\.3Automatic Memory Design Baselines

Automatic memory design baselines search for memory mechanisms rather than relying on a single hand\-written memory schema\. These baselines are evaluated under the same sequential search budget asAutoMMemo, so that performance differences reflect the quality of the searched memory design rather than additional search cost\.

#### ALMA\.

ALMA\[Xionget al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib47)\]is an automatic memory\-design baseline that uses a meta agent to search over memory designs expressed as executable code\. We adapt ALMA to our update\-then\-retrieve evaluation protocol\. Each candidate memory design is evaluated by first updating memory on the training episodes and then retrieving memory for held\-out tasks\. To match the sequential search setting ofAutoMMemo, ALMA is allocated the same number of search steps, and one candidate memory design is generated and evaluated at each step\. The final ALMA memory design is selected according to validation performance and then evaluated on the test split under the same execution and retrieval budgets\.

### E\.4Summary of Baseline Implementations

Table[6](https://arxiv.org/html/2605.16857#A5.T6)summarizes the main differences among the baseline implementations\. All methods are evaluated using the same benchmark splits, execution agents, and evaluation protocol\.

Table 6:Summary of baseline implementations under our unified update\-then\-retrieve protocol\. All methods use the same train/test splits, execution agents, evaluation protocol, and retrieval budget\.

## Appendix FImplementation Details

### F\.1Evaluation Protocol and Metrics

We follow the offline update\-then\-retrieve protocol described in Section[4](https://arxiv.org/html/2605.16857#S4)\. This appendix provides additional implementation\-level details, including interaction limits, invalid\-run handling, search hyperparameters, final\-evaluation settings, and system configurations\.

Each episode is limited to at most 15 interaction steps\. Task failures are counted as zero\. Infrastructure\-level invalid runs, such as browser crashes, page loading failures, API errors, or tool execution failures, are excluded from the aggregate results and rerun when possible\. During final evaluation, retrieved memory is injected only once before the first user turn, and no additional memory updates are performed on the held\-out test split\.

For GUI/web navigation benchmarks, we report success rate\. For visual reasoning/search benchmarks, we report judge\-based accuracy\. AVG\. GUI is computed as the simple macro\-average over WebVoyager and Mind2Web, while AVG\. VR is computed as the simple macro\-average over AgentVista and MMSearch\-Plus\. We do not use dataset\-size\-weighted averaging unless otherwise specified\.

All baselines are adapted to the same memory interface and worker\-side injection budget used byAutoMMemo\. Detailed baseline adaptations and retrieval settings are provided in Appendix[E](https://arxiv.org/html/2605.16857#A5)\.

### F\.2Memory\-Design Search

We run memory\-design search for 20 steps\. At each search step, a candidate memo program is either generated by mutating an existing node or selected for re\-evaluation according to the search algorithm described in Section[3](https://arxiv.org/html/2605.16857#S3)\. Before full evaluation, each generated memo program is checked by a quick test on 5 randomly sampled training tasks to filter out invalid implementations and obvious runtime failures\.

If the quick test fails, we regenerate or repair the candidate for up to three attempts\. If all attempts fail, the current search step is marked as failed, and no child memo program is saved or sent to full evaluation\. Otherwise, the candidate is evaluated under the update\-then\-retrieve protocol on the training split of the corresponding benchmark\. After the search budget is exhausted, the final memo program is selected using the confidence\-aware selection rule described in Section[3](https://arxiv.org/html/2605.16857#S3)\.

The same 20\-step sequential search budget is used forAutoMMemoand ALMA\. Baseline\-specific adaptations, including the one\-candidate\-per\-step ALMA protocol, are described in Appendix[E](https://arxiv.org/html/2605.16857#A5)\.

### F\.3Search Hyperparameters

Table[7](https://arxiv.org/html/2605.16857#A6.T7)lists the hyperparameters used during memory\-design search\.

Table 7:Hyperparameters used during memory\-design search\.HyperparameterValueSearch steps20Quick\-test tasks per candidate5Maximum regeneration / repair attempts3Full evaluation tasks per valid candidateTraining retrieve setMaximum interaction steps15Evaluation confidence coefficientcevalc\_\{\\mathrm\{eval\}\}0\.2Generation confidence coefficientcgenc\_\{\\mathrm\{gen\}\}0\.2Generation prior strengthρ\\rho0\.5Generation prior pseudo\-countβ\\beta1\.0Minimum widthBB2Max evaluation screenshots5Max browser concurrency8Max update batch concurrency8Max retrieve batch concurrency8Max meta observation summary length50,000Max meta images per episode4Max retrieved memory characters50,000Max retrieved memory images2Max meta retrieved\-memory summary length20,000Max meta retrieved\-memory images per episode2Success trajectories for meta analysis2Failure trajectories for meta analysis2
### F\.4Final Evaluation Hyperparameters

Table[8](https://arxiv.org/html/2605.16857#A6.T8)lists the hyperparameters used during final held\-out evaluation\. All methods use the same execution budget, context budget, retrieval budget, and evaluation protocol\. During final evaluation, retrieved memories are injected before the first user turn using the same worker\-side text and image budget for all methods\.

Table 8:Hyperparameters used during final held\-out evaluation\.HyperparameterValueJudge modelGPT\-5\.4\-miniMaximum interaction steps15Max evaluation screenshots5Max browser concurrency16Max update batch concurrency16Max retrieve batch concurrency16Max images in agent context8Retrieved memory character budget50,000Retrieved memory image budget2Independent evaluation runs3
### F\.5Model and System Details

We instantiate the execution agent with Qwen3\-VL\-32B\[Baiet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib39)\], GPT\-5\.4\-nano\[Singhet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib40)\], and Qwen3\.5\-Plus\[Baiet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib39)\]\. Unless otherwise specified, the meta agent uses GPT\-5\[Singhet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib40)\], and the LLM judge uses GPT\-5\.4\-mini\[Singhet al\.,[2025](https://arxiv.org/html/2605.16857#bib.bib40)\]\. Closed\-source models are accessed through their official API endpoints, while Qwen3\-VL\-32B is served locally\.

All experiments are conducted using PyTorch\[Paszkeet al\.,[2019](https://arxiv.org/html/2605.16857#bib.bib48)\]on 8 NVIDIA Tesla A100 GPUs, with open\-source models launched via vLLM\[Kwonet al\.,[2023](https://arxiv.org/html/2605.16857#bib.bib49)\]to enable efficient inference\. During memory\-design search, we set the maximum browser concurrency to 8 and the maximum concurrent update and retrieval batches to 8\. During final evaluation, we increase these values to 16 to improve throughput\. These concurrency settings affect only wall\-clock efficiency and are kept fixed across methods within the same evaluation stage\.

Table 9:Main results on GUI\-agent and visual\-reasoning benchmarks across different execution models\. Numbers in parentheses indicate absolute performance changes over the corresponding NoMemory baseline, and bold denotes the best result within each execution\-model block\.

## Appendix GAdditional Results on Qwen3\.5\-Plus

Table[9](https://arxiv.org/html/2605.16857#A6.T9)also reports the results on Qwen3\.5\-Plus\. The observations are consistent with the main results\.AutoMMemoachieves the best performance on all four benchmarks and obtains the highest average scores for both GUI navigation and visual reasoning\. Compared with the NoMemory baseline,AutoMMemoimproves AVG\. GUI from 44\.17 to 55\.36, yielding an absolute gain of 11\.19 points, and improves AVG\. VR from 21\.01 to 30\.71, yielding a gain of 9\.70 points\.

The improvement is especially notable on Mind2Web and AgentVista\. On Mind2Web,AutoMMemoimproves from 30\.11 to 45\.68, substantially outperforming both text\-based and multimodal\-based memory baselines\. On AgentVista,AutoMMemoimproves from 17\.87 to 28\.81, showing that the learned memory mechanism is also effective for visual reasoning\. These results further confirm thatAutoMMemogeneralizes beyond a specific execution model and provides consistent benefits across heterogeneous multimodal tasks\.

## Appendix HSearch\-Time Cost

In the main text, we analyze the inference\-time cost of the final learned memo program\. Here we report the offline search\-time overhead incurred during memory\-design optimization\. All numbers exclude the initial root evaluation, which is shared by the search procedure and is not counted as part of the 20\-step search budget\.

Table[10](https://arxiv.org/html/2605.16857#A8.T10)summarizes the search process ofAutoMMemounder Qwen3\-VL\-32B\. Across the four benchmarks,AutoMMemodynamically allocates the fixed 20\-step search budget between generating new memo programs and re\-evaluating existing nodes\. Only 43 out of 80 search steps are generation actions, while the remaining 37 steps re\-evaluate existing programs to reduce score uncertainty\. This behavior improves search efficiency because re\-evaluation steps do not invoke the meta agent for reflection, mutation, or repair\. Most generated candidates pass quick examination, with only one failed generation on WebVoyager, indicating that the quick examination and repair stage effectively prevents invalid programs from consuming full\-evaluation budget\.

Table[11](https://arxiv.org/html/2605.16857#A8.T11)compares the search\-time token and wall\-clock overhead ofAutoMMemoand ALMA\. Meta tokens include reflection, mutation, and repair calls forAutoMMemo, and candidate memory\-design generation calls for ALMA under our adapted sequential search protocol\. Evaluation tokens include execution\-agent and judge\-model tokens consumed by full evaluations during search\. Across all four benchmarks,AutoMMemouses fewer meta tokens, fewer evaluation tokens, fewer total search\-time tokens, and less wall\-clock time than ALMA\. Aggregated over the four benchmarks,AutoMMemoreduces meta\-search tokens by 45\.7%, evaluation tokens by 22\.3%, total search\-time tokens by 22\.6%, and wall\-clock time by 26\.6%\. These results show that the performance gains ofAutoMMemoare not obtained by using a larger offline design budget\.

Table 10:Search\-time process statistics under Qwen3\-VL\-32B\. We report the number of generation actions, re\-evaluation actions, valid generated programs, candidates rejected by quick examination, and full evaluations consumed during the 20\-step search\. All numbers exclude the initial root evaluation\.Table 11:Search\-time token and wall\-clock overhead under Qwen3\-VL\-32B\. Both methods use the same 20\-step sequential search budget, and all numbers exclude the initial root evaluation\. Meta tokens include reflection, mutation, and repair calls forAutoMMemo, and candidate memory\-design generation calls for ALMA\. Evaluation tokens include execution\-agent and judge\-model tokens consumed by search\-time full evaluations\. Numbers in parentheses denote relative reductions over ALMA\.
## Appendix IDetailed Memory Evolution

This appendix provides a node\-level analysis of the memory evolution process discussed in Section[4](https://arxiv.org/html/2605.16857#S4)\. While Figure[4](https://arxiv.org/html/2605.16857#S4.F4)shows the overall search trajectory, Figure[5](https://arxiv.org/html/2605.16857#A9.F5)visualizes how individual memory candidates are generated, refined, and evaluated during the search process\. Specifically, the figure corresponds to theWebVoyagersearch run underQwen3\-VL\-32Bas the execution agent\. Each node corresponds to a candidate memory design, annotated with its validation score and a short description of the main design change\. The scores in this figure are used for search\-time model selection and are not directly comparable to the held\-out test results in the main table\. The final selected memory program is node N11, which is chosen by the lower\-confidence\-bound \(LCB\) rule described in Section 3\.4\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x5.png)Figure 5:Visualization of the memory\-design search tree for theWebVoyagersearch run withQwen3\-VL\-32B\. Each node denotes a candidate memory design, where the score reports its validation performance\. The root node is the NoMemory baseline, and each child node summarizes the main modification made to its parent design\. The final selected memory program is N11, chosen by the LCB rule after search\. The search gradually evolves from simple domain\-intent memories to more task\-conditioned and recovery\-aware designs, revealing how stronger memory mechanisms emerge through iterative exploration\.#### Evolution along the high\-performing path\.

Starting from the NoMemory root,AutoMMemoquickly discovers that adding structured task experience is beneficial\. An early candidate introduces domain\-intent playbooks, which substantially improves over the root by organizing experience around website domains and task intents\. Later candidates further refine this idea by adding common web intents, task\-conditioned filters, and cleaner task briefs\. The strongest node in the tree emphasizes filtering noisy UI text and preserving task\-relevant information, suggesting that effective memory is not simply a larger context store, but a mechanism for selecting and presenting useful experience to the agent\.

#### Why broader memory is not always better\.

Several generated candidates underperform despite adding more memory content or broader fallback rules\. For example, candidates that rely on persistent storage of complex playbooks or generic fallback rules obtain much lower validation scores than more task\-conditioned candidates\. This indicates that excessive or weakly targeted memory can introduce irrelevant context, distract the execution agent, and reduce the effectiveness of retrieval\. The search process therefore needs to optimize not only what to remember, but also how memory is abstracted, indexed, filtered, and injected\.

#### Search reveals reusable but task\-sensitive design principles\.

The evolution tree also shows that some alternative branches achieve competitive scores by specializing memory toward particular domains or interaction patterns, such as local\-business hints, topic\-conditioned retrieval, or task extraction recovery\. These branches suggest that useful memory designs often share reusable principles, including intent abstraction, domain\-aware organization, and observation filtering\. However, their uneven performance also confirms that memory mechanisms must be adapted to the target task distribution rather than manually fixed in advance\.

## Appendix JLearned Memo Programs Across Benchmarks

To better illustrate whatAutoMMemodiscovers, we visualize the final memo programs selected after search on each benchmark under Qwen3\-VL\-32B\. While Appendix[I](https://arxiv.org/html/2605.16857#A9)analyzes how memo programs evolve during search, this section focuses on the internal structure of the final selected programs\. Figures[6](https://arxiv.org/html/2605.16857#A10.F6)–[9](https://arxiv.org/html/2605.16857#A10.F9)show implementation\-level schematics of the learned memo programs for WebVoyager, Mind2Web, AgentVista, and MMSearch\-Plus, respectively\. All four programs follow the same externalupdate/retrieveinterface, but their internal memory states, indexing rules, retrieval strategies, and returned payloads differ substantially across benchmarks\.

#### WebVoyager\.

Figure[6](https://arxiv.org/html/2605.16857#A10.F6)shows the final memo program selected for WebVoyager\. The learned design uses a domain\-intent memory structure that combines static site\-intent playbooks, dynamic success shards, anti\-stuck flags, and a lightweight episode counter\. During retrieval, the program extracts a clean task brief, predicts the target domain and intent, collects candidates from static playbooks and dynamic success memories, and optionally adds anti\-stuck nudges when recent failures suggest repeated actions, waiting, or interface blockage\. The retrieved payload is intentionally small, returning at most two concise hints with metadata\. This design reflects the website\-specific nature of WebVoyager, where reusable navigation checklists and recovery hints are often more useful than long trajectory recall\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x6.png)Figure 6:Learned memo program for WebVoyager under Qwen3\-VL\-32B\. The selected program uses a domain\-intent memory design\. During initialization, it builds static site\-intent playbooks and an in\-memory state containing success shards, anti\-stuck flags, and an episode counter\. Duringretrieve, it extracts a clean task brief, predicts the domain and intent, adds task\-conditioned and static playbook candidates, optionally includes anti\-stuck nudges or dynamic success memories, and returns at most two concise hints\. Duringupdate, it detects repeated actions, waiting, cookie overlays, or other stuck patterns, updates anti\-stuck flags, and stores successful trajectories into domain\-intent shards\.
#### Mind2Web\.

Figure[7](https://arxiv.org/html/2605.16857#A10.F7)shows the selected memo program for Mind2Web\. Unlike the WebVoyager design, this program is organized around host\-aware playbooks and task blueprints\. It maintains static host playbooks and micro\-skills for recurring website widgets, while also updating trap counters and success recipes from interaction outcomes\. At retrieval time, it builds a fused text\-image query from the task description and initial screenshots, ranks memories using multimodal similarity, domain match, trap signals, and success bonuses, and returns the task blueprint together with the top ranked memory items\. This structure matches Mind2Web, where tasks frequently require handling website\-specific forms, dropdowns, date pickers, and recurring interaction traps\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x7.png)Figure 7:Learned memo program for Mind2Web under Qwen3\-VL\-32B\. The selected program uses a host\-aware playbook memory\. It initializes static host playbooks and micro\-skills, while maintaining dynamic trap counters and success recipes\. Duringretrieve, it detects the target domain, constructs a task blueprint from the initial description, builds a fused text\-image query using the task and up to two initial images, ranks candidate memories by multimodal similarity, domain match, trap signals, and success bonuses, and returns the task blueprint together with the top ranked memories\. Duringupdate, it detects UI traps from failed trajectories and summarizes successful episodes into reusable recipes\.
#### AgentVista\.

Figure[8](https://arxiv.org/html/2605.16857#A10.F8)illustrates a more compact procedural memory learned for AgentVista\. The program initializes a library of visual\-reasoning task families, where each family contains keywords, anchors, and concise hint items\. Retrieval routes the current question to a task family using token and ngram matches, anchor shortcuts, adaptive tag biases, and historical success priors\. If no family is confidently matched, the program falls back to a general image\-first protocol that encourages the agent to inspect visible labels, numbers, legends, and on\-screen text before using external tools\. During update, the program logs outcomes and adjusts routing biases based on success or failure\. This suggests that AgentVista benefits more from task\-family\-level visual reasoning procedures than from large episodic stores\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x8.png)Figure 8:Learned memo program for AgentVista under Qwen3\-VL\-32B\. The selected program uses a task\-family procedural memory\. It initializes a family library containing curated visual\-reasoning task families, keywords, anchors, and concise hint items\. Duringretrieve, it extracts the initial question, scores task families using tokens, ngrams, anchors, adaptive tag biases, and historical success priors, and returns one or two procedure\-level hints for the predicted family\. If no confident family is found, it falls back to a general image\-first protocol\. Duringupdate, it logs the episode outcome, updates family\-level success/failure statistics, reinforces useful routing tags, and decays misleading associations\.
#### MMSearch\-Plus\.

Figure[9](https://arxiv.org/html/2605.16857#A10.F9)shows the learned memo program for MMSearch\-Plus\. The selected program combines a domain\-tuned skillbook with cue\-indexed episodic memory\. It extracts high\-signal cues from task text and images, including OCR\-derived text, quoted phrases, capitalized terms, numbers, years, and multilingual tokens\. These cues are used to detect a coarse domain and retrieve relevant structured episodes from persistent JSONL storage\. The returned payload contains at most a few episodic matches together with one domain\-specific skillbook tip\. During update, the program records successful and failed search behavior, including tools, queries, final answers, and failure notes\. This design is well suited to MMSearch\-Plus, where effective search often depends on identifying distinctive visual or textual evidence and converting it into targeted search queries\.

![Refer to caption](https://arxiv.org/html/2605.16857v1/x9.png)Figure 9:Learned memo program for MMSearch\-Plus under Qwen3\-VL\-32B\. The selected program combines a domain\-tuned skillbook with cue\-indexed episodic memory\. Duringretrieve, it extracts text and image cues from the initial input, optionally uses OCR with caching, detects a coarse domain, scores episodic candidates by cue overlap, success bonus, age decay, and seed matching, and returns up to two episodic matches together with one skillbook tip\. Duringupdate, it extracts cues, tools, search queries, final answers, and failure notes, then appends a structured episodic entry to persistent JSONL storage\.
#### Comparison across learned programs\.

Figures[6](https://arxiv.org/html/2605.16857#A10.F6)–[9](https://arxiv.org/html/2605.16857#A10.F9)show thatAutoMMemodoes not simply discover a single universal memory template\. Instead, it adapts the internal organization of memory to the structure of each benchmark\. WebVoyager emphasizes domain\-intent navigation playbooks and anti\-stuck recovery; Mind2Web emphasizes host\-aware task blueprints, widget skills, traps, and success recipes; AgentVista emphasizes task\-family procedural routing; and MMSearch\-Plus emphasizes cue extraction, episodic search memory, and domain\-specific search skills\. Despite these differences, the learned programs share a common principle: they compress raw multimodal trajectories into compact, task\-facing memory payloads rather than injecting full histories\. This supports the main observation that effective memory design requires selecting, abstracting, and presenting experience in a form matched to the target task distribution\.

## Appendix KPrompt Details

This appendix reports the prompt templates used in our experiments\. We group them according to their roles in the system: meta\-agent prompts, execution\-agent prompts, evaluation prompts, and baseline\-specific prompts\. When a prompt template is shared across multiple benchmarks or methods, we report it once and instantiate it with the corresponding task description, trajectory records, retrieved memories, or benchmark\-specific metadata\.

### K\.1Meta\-Agent Prompts

The meta agent uses four prompt templates during memory\-design search\. First, the analysis prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)diagnoses the current memo program by inspecting its source code, sampled successful and failed trajectories, retrieved memories, benchmark scores, and previous improvement examples\. It outputs structured feedback, including memory relevance labels, content\-quality issues, structural weaknesses, and prioritized code\-level suggestions\. Second, the code\-generation prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)rewrites an existing memo program based on the analysis results and the observed performance\. Third, at the beginning of search or when no previous source is available, the initialization prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)generates a new memo program directly from the task description and the memo\-program interface\. Finally, when a generated program fails static checks or runtime execution, the repair prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)asks the meta agent to preserve the intended memory design while fixing implementation errors\.

These prompts jointly define the mutation process in our design\-search loop\. They expose the same memo\-program contract to the meta agent: each candidate must implementretrieveandupdate, return a valid retrieved\-memory payload, and respect the text and image budgets used by the execution agent\. The prompts also specify the available implementation utilities, including multimodal chat calls, embedding clients, vector search, and graph operations\.

### K\.2Execution\-Agent and Memory\-Injection Prompts

The execution agent uses benchmark\-family\-specific system prompts\. For GUI/web navigation tasks, we use the WebVoyager\-style execution prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)\. For multimodal visual reasoning and tool\-use tasks, we use the AgentVista\-style execution prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)\. These execution prompts define the agent’s role, action format, observation usage, and termination behavior\. All methods and baselines use the same execution prompt under the same benchmark and execution\-model setting, so performance differences come from the memory mechanism rather than from prompt tuning\.

Retrieved memories are injected through a shared prompt fragment shown in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)\. This fragment serializes theRetrievedMemoryPayloadreturned byretrieve, including textual guidance, optional image references, and structured metadata\. When visual observations are included in the retrieved memory, they are attached subject to the same image\-budget constraints used throughout the experiments\. For AgentVista\-style tasks that require summarizing information returned by the visit tool, we use the tool\-summary prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3)\.

### K\.3Evaluation Prompts

We use LLM\-as\-judge evaluation with benchmark\-family\-specific prompts, following the evaluation protocols of WebVoyager\[Heet al\.,[2024](https://arxiv.org/html/2605.16857#bib.bib35)\]for GUI/web navigation and AgentVista\[Suet al\.,[2026](https://arxiv.org/html/2605.16857#bib.bib36)\]for visual reasoning\. The GUI/web navigation evaluator uses the prompt in Prompt 6, which judges whether the final browser state or answer satisfies the user task\. The visual reasoning evaluator uses the prompt in Prompt 8, which checks answer correctness and consistency with the available evidence\. Both evaluators receive the task instruction, trajectory information, final answer or final state, and a bounded number of screenshots\. The evaluator outputs a binary correctness label, which is used as success rate for GUI/web navigation tasks and judge\-based accuracy for visual reasoning tasks\. The GUI/web navigation evaluator uses the prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3), which judges whether the final browser state or answer satisfies the user task\. The visual reasoning evaluator uses the prompt in Prompt[K\.3](https://arxiv.org/html/2605.16857#A11.SS3), which checks answer correctness and consistency with the available evidence\. Both evaluators receive the task instruction, trajectory information, final answer or final state, and a bounded number of screenshots\. The evaluator outputs a binary correctness label, which is used as success rate for GUI/web navigation tasks and judge\-based accuracy for visual reasoning tasks\.

Prompt 1: Meta Agent Analysis Prompt

Meta Agent Analysis Prompt\[SYSTEM\]You are a \*\*Senior Agent Construction Engineer\*\* responsible for provide suggestions for a memory structure written by a entry level engineer, to make the memory structure better for downstream agent to finish tasks\.\#\#\# Memo Information Overview1\. \*\*source\_code\*\*\- A ‘MemoStructure‘ implementation coordinates memory; use private helpers or small inner abstractions for clarity\.\- \*\*retrieve\*\* / \*\*update\*\* are the only hooks the runtime calls\.\- code usage: Your memory structure will be used in the agent workflow:\- ‘retrieve\(recorder\)‘: used \*\*before\*\* executing the task\. It returns a “RetrievedMemoryPayload“ \(see schema below\)\. The worker injects a JSON text summary of that payload plus any referenced images into the execution agent’s first user message\.\- ‘update\(recorder\)‘: used \*\*after\*\* task is finished, to update the trajectory, reward, or other information\.\#\#\# RetrievedMemoryPayload \(return type of “MemoStructure\.retrieve“\)“retrieve“ must return a \*\*JSON\-serializable\*\* “RetrievedMemoryPayload“ \(import from “meta\_self\_evolve\.contracts\.types“\), i\.e\. a dict with:\- “items“: a \*\*flat\*\* list of memory dicts; each dict may include:\- “text“ \(“str“, optional\): guidance for the execution agent\.\- “images“ \(optional\): list of image refs with the same JSON shape as trajectory “ImageRef“: “\{"kind": "path"\|"url", "value": "<relative path or URL\>", "mime": "<optional\>"\}“\. When “kind“ is “"path"“, “value“ is resolved relative to the episode \*\*artifact root\*\*\.\- “metadata“ \(“dict“, optional\): any structured fields \(e\.g\. “memory\_id“, “score“, “source“, layer names\)\.\- “metadata“ \(“dict“\): episode\-level retrieval notes \(use “\{\}“ if none\)\.\*\*Empty retrieval:\*\* return “\{"items": \[\], "metadata": \{\}\}“ \(or use “empty\_retrieved\_memory\_payload\(\)“ from “meta\_self\_evolve\.common\.retrieved\_memory“\)\.\*\*Image / text budgets \(truncation is expected\):\*\*\- Put \*\*only a small number\*\* of images in “items\[\*\]\.images“ \(typically 0–2\)\. Extra images are \*\*dropped\*\*; do not rely on the model seeing every path you emit\.\- The JSON text of the payload is also \*\*truncated\*\* to “RolloutLimits\.max\_retrieved\_memory\_chars“ in the execution agent’s first user turn\.\- \*\*Execution agent:\*\* at most “RolloutLimits\.max\_retrieved\_memory\_images“ images are loaded from the payload \(in “items“ order\) on the first user turn; this is \*\*separate\*\* from the per\-step screenshot budget “RolloutLimits\.max\_images\_per\_step“ for environment observations\.\- \*\*Meta analysis:\*\* after the truncated JSON block, at most “MetaTrajectoryLimits\.meta\_retrieved\_memory\_images\_per\_episode“ payload images are attached per sampled episode \(again in order\); this is \*\*separate\*\* from “MetaTrajectoryLimits\.max\_meta\_images\_per\_episode“ for trajectory screenshots\.\*\*Runtime:\*\* the worker injects this into the first user turn as \(1\) a text block \(header \+ pretty\-printed JSON of the payload\), then \(2\) “image\_url“ parts for each referenced image, subject to the limits above\.2\. \*\*examples\*\*\- \*\*examples\*\*: sampled retrieve trajectories, split into \*\*FAILED TRAJECTORIES\*\* and \*\*SUCCESSFUL TRAJECTORIES\*\*\.\- In each trajectory section below, every episode may include a text block starting with ‘retrieved\_memory \(JSON from retrieve before this episode\):‘ — that is the serialized “RetrievedMemoryPayload“ from “retrieve“ \(truncated in logs\); use it together with trajectory screenshots to judge whether retrieval helped\.3\. \*\*benchmark\_eval\_score\*\*\- performance\(success rate\) of current memory stucture \+ general agent system\. Need to use the score to analyze the performance and bottleneck of current memory structure\.\#\#\# Your Task:You will analyze past suggestion examples\(including past source code, suggestions, and the improve score it led to\) and the current retrieved trajectories and memory source code, then produce concrete, prioritized suggestions to improve the memory structure\.Follow the numbered procedure below and produce the requested structured outputs\.Step 1 — Learn from past suggestions & the improve score1\. Look at the provided improve\_score \(positive → improvement, negative → degradation\) and the single suggestion\_example that produced that score\.2\. Explain why that suggestion led to improvement or degradation:\- What pattern in the change made it succeed or fail?\- Which behaviors, assumptions, or shortcuts in that suggestion were helpful? Which were harmful?\- From these concrete cases, extract 2–5 general principles to adopt and 2–5 pitfalls to avoid when creating future suggestions\.Step 2 — Inspect sampled trajectories and benchmark performance and decide which memories are useful1\. Review the sampled FAILED TRAJECTORIES first, then the SUCCESSFUL TRAJECTORIES\. For each episode, use the ‘retrieved\_memory‘ JSON block when present \(it is omitted when empty\)\.2\. Compare the two groups: identify which retrieved memories or retrieval patterns appear in successful runs, and which missing / noisy / misleading memories correlate with failed runs\.3\. For each retrieved memory item \(or memory group\) returned for the trajectory, label it as one of:\- Useful & Relevant — clearly applies to the current situation and can guide action;\- Potentially Useful — has value but needs reformatting, summarization, or indexing to be helpful;\- Irrelevant / Confusing — not related to this trajectory or misleading;\- Empty / Badly Formatted — blank, placeholder, or not parseable\.4\. For each memory you mark Useful/Potentially Useful, say how it would help \(e\.g\., provides a repeated subgoal, highlights a trap, identifies key object interactions\)\.5\. For Irrelevant/Empty items, explain why they failed retrieval combine with the memory source code \(e\.g\., wrong keying, over\-specific content, missing summarization\)\.Step 3 — Inspect memory source and produce concrete suggestions1\. Review the memory source code \(retrieval keys, indexing, storage format, layers\)\. Using Step 1 principles and Step 2 labels, propose specific changes to the memory system that address the observed issues\.2\. Combined the memory source code with your analysis in step 2, giving suggestions\. For each suggested change, include:\- What to change \(code\-level or pipeline change, e\.g\., add summarization layer, change indexing key, normalize objects to noun\-phrases\)\.\- Why it will help \(link back to a principle or a concrete failing you observed\)\.3\. Prioritize suggestions: label them High / Medium / Low priority and give an implementation order\.4\. Link Analysis to Benchmark Performance\- Use benchmark\_eval\_score to identify which structural weaknesses correlate with poor performance\.Extra checks \(quality & coherence\)1\. Flag obvious content issues: duplicates, empty entries, raw dumps, mis\-typed fields, or numeric types that break JSON serialization\.2\. Check layer interaction: do layers pass structured outputs to each other, or only dump free\-form text?3\. If retrieval returns empty lists or dicts, emphasize structural fixes \(keying, ensure type consistency, avoid over\-relying on try/except fallbacks\)\.Goal: Combine reflection on past improvement signals with current system diagnosis to produce actionable, high\-level suggestions that strengthen memory structure quality\.\#\#\# Benchmark Information:\{task\\\_description\}\#\#\# Required Output:Return a JSON object followed below json schema:\{"learned\_from\_suggestion\_example": \{"type": "string","description": "Findings derived from the provided suggestion\_example and improve\_score\. Bullet list of concrete factors \(patterns\) that made the suggestion succeed or fail\. And principles to adopt when making future suggestions\."\},"trajectory\_score\_assessment": \{"type": "array","description": "analysis each retrieved module information based on current trajectories sampled and the benchmark scores\.","items": \{"type": "object","properties": \{"label": \{"type": "string","enum": \["Useful","Potentially Useful","Irrelevant","Empty/BadFormat"\],"description": "Categorization of the memory item’s relevance based on whether the retrieved content actually helps the agent\."\},"how\_it\_can\_help": \{"type": "string","description": "If Useful/Potentially Useful: short note how it could guide actions \(subgoal, trap, object use…\)\. If Irrelevant/Empty: reason \(e\.g\., wrong keying, over\-specific, missing summary, formatting\)\."\}\},"required": \["label","how\_it\_can\_help"\]\}\},"content\_quality\_issues": \{"type": "string","description": "Detected content\-level problems \(duplicates, empty entries, serialization issues…\)\. Why those harms retrieval or downstream planning\."\},"structure\_and\_coherence": \{"type": "string","description": "Analysis of layer interactions, keying, and task\-awareness\. Which parts generalize, which are overfitted\."\},"suggested\_changes": \{"type": "array","description": "Based on all your analysis above, provide concrete change that can be applied on provided current memory structure code\.","items": \{"type": "object","properties": \{"priority": \{"type": "string","enum": \["High","Medium","Low"\],"description": "How urgent/impactful this is\."\},"what": \{"type": "string","description": "Precise description of what to change \(code/pipeline/config\)\."\},"why": \{"type": "string","description": "Link to observations/principles: why this addresses the problem\."\}\},"required": \["priority","what","why"\]\}\}\}\[USER\]<Suggestion Example\>Here is a previous suggestion, along with the code it looked at\. 4\.The example include a memory structure and it’s modification attempt, annotated with an improvement score \(positive = improved, negative = degraded\)\.Infer the underlying patterns that differentiate effective modifications from harmful ones, and apply this reasoning to suggest an improved modification for the current memory structure\."\{improve\\\_example\}"</Suggestion Example\><CURRENT SOURCE CODE\>\{source\\\_code\}</CURRENT SOURCE CODE\><CURRENT TRAJECTORY EXAMPLES\>\{trajectory\\\_examples\}</CURRENT TRAJECTORY EXAMPLES\><CURRENT BENCHMARK SCORE\>\{benchmark\\\_overall\\\_eval\\\_score\}</CURRENT BENCHMARK SCORE\>

Prompt 2: Meta Agent Generate New Code Prompt

Meta Agent Generate New Code Prompt\[SYSTEM\]You are a senior AI software engineer\. Your task is to build a clear, maintainable memory system for a downstream task agent\. The agent will be used in\{env\\\_id\}\. Your memory structure should supply relevant experience and reference beyond raw trajectory text\.\{task\\\_description\}You are given the following backbone \(“MemoStructure“ with “retrieve“ / “update“\):<BACKBONE\_CODE\>"""Abstract memo backbone inlined into codegen prompts\.Only :class:‘MemoStructure‘ is required\. Implement a clear, maintainable“retrieve“ / “update“ pair; use private helpers or inner classes as needed\."""from \_\_future\_\_ import annotationsfrom abc import ABC, abstractmethodfrom typing import Any, Optionalfrom meta\_self\_evolve\.contracts\.types import EpisodeRecorder, RetrievedMemoryPayloadclass MemoStructure\(ABC\):def \_\_init\_\_\(self\) \-\> None:self\.database: Optional\[Any\] = None@abstractmethodasync def retrieve\(self, recorder: EpisodeRecorder\) \-\> RetrievedMemoryPayload:"""Return structured memory for the next episode \(see “RetrievedMemoryPayload“\)\."""…@abstractmethodasync def update\(self, recorder: EpisodeRecorder\) \-\> None:"""Incorporate a finished trajectory \(init, steps, reward, etc\.\)\."""…</BACKBONE\_CODE\>Subclass “MemoStructure“ and import the types you need, for example:“‘pythonfrom meta\_self\_evolve\.common\.retrieved\_memory import empty\_retrieved\_memory\_payloadfrom meta\_self\_evolve\.contracts\.memo\_backbone import MemoStructurefrom meta\_self\_evolve\.contracts\.types import EpisodeRecorder, RetrievedMemoryPayloadfrom meta\_self\_evolve\.llm\.client import MultimodalChatClientfrom meta\_self\_evolve\.llm\.embedding import \(EmbeddingClient,embed\_item\_image\_path,embed\_item\_image\_url,embed\_item\_text,\)\# plus your choice of Chroma, networkx, etc\.“‘<CODE\_INPUT\>Your ‘retrieve‘ and ‘update‘ will take ‘EpisodeRecorder‘ as input, which has following attributes:\{"init": \{"description": "Initial task text and observation images\.","type": "InitRecord"\},"steps": \{"description": "Per\-step action and observation history\.","type": "list\[StepRecord\]"\},"memory\_retrieved": \{"description": "Structured output from retrieve\(\) before the episode\.","type": "RetrievedMemoryPayload"\},"reward": \{"description": "Scalar episode reward after finish\.","type": "float"\},"messages": \{"description": "Full LLM conversation history for this episode\.","type": "list\[dict\]"\}\}\- For ‘retrieve‘, only leverage ‘\.init‘ attribute\.\- For ‘update‘, leverage ‘\.init‘, ‘\.steps‘, ‘reward‘ attribute\.\- Each element in the above dict is a attribute name as key, and description, type, and a example for the exact possible value the attribute could have\.\- please note that all provided current trajectory can already been seen by down stream agents\(in history\), your memory structure should focus on provide extra advice and reference for agents\.</CODE\_INPUT\><CODE\_USAGE\>Your memory structure will be used in the agent workflow:\- ‘retrieve\(recorder\)‘: used \*\*before\*\* executing the task; must return “RetrievedMemoryPayload“ \(see schema below\)\. Populate “items“ with concise “text“ and optional “images“ \(“ImageRef“ JSON\) so the execution agent receives useful guidance\.\- ‘update\(recorder\)‘: used \*\*after\*\* task is finished, to update the trajectory, reward, or other information\.\#\#\# RetrievedMemoryPayload \(return type of “MemoStructure\.retrieve“\)“retrieve“ must return a \*\*JSON\-serializable\*\* “RetrievedMemoryPayload“ \(import from “meta\_self\_evolve\.contracts\.types“\), i\.e\. a dict with:\- “items“: a \*\*flat\*\* list of memory dicts; each dict may include:\- “text“ \(“str“, optional\): guidance for the execution agent\.\- “images“ \(optional\): list of image refs with the same JSON shape as trajectory “ImageRef“: “\{"kind": "path"\|"url", "value": "<relative path or URL\>", "mime": "<optional\>"\}“\. When “kind“ is “"path"“, “value“ is resolved relative to the episode \*\*artifact root\*\*\.\- “metadata“ \(“dict“, optional\): any structured fields \(e\.g\. “memory\_id“, “score“, “source“, layer names\)\.\- “metadata“ \(“dict“\): episode\-level retrieval notes \(use “\{\}“ if none\)\.\*\*Empty retrieval:\*\* return “\{"items": \[\], "metadata": \{\}\}“ \(or use “empty\_retrieved\_memory\_payload\(\)“ from “meta\_self\_evolve\.common\.retrieved\_memory“\)\.\*\*Image / text budgets \(truncation is expected\):\*\*\- Put \*\*only a small number\*\* of images in “items\[\*\]\.images“ \(typically 0–2\)\. Extra images are \*\*dropped\*\*; do not rely on the model seeing every path you emit\.\- The JSON text of the payload is also \*\*truncated\*\* to “RolloutLimits\.max\_retrieved\_memory\_chars“ in the execution agent’s first user turn\.\- \*\*Execution agent:\*\* at most “RolloutLimits\.max\_retrieved\_memory\_images“ images are loaded from the payload \(in “items“ order\) on the first user turn; this is \*\*separate\*\* from the per\-step screenshot budget “RolloutLimits\.max\_images\_per\_step“ for environment observations\.\- \*\*Meta analysis:\*\* after the truncated JSON block, at most “MetaTrajectoryLimits\.meta\_retrieved\_memory\_images\_per\_episode“ payload images are attached per sampled episode \(again in order\); this is \*\*separate\*\* from “MetaTrajectoryLimits\.max\_meta\_images\_per\_episode“ for trajectory screenshots\.\*\*Runtime:\*\* the worker injects this into the first user turn as \(1\) a text block \(header \+ pretty\-printed JSON of the payload\), then \(2\) “image\_url“ parts for each referenced image, subject to the limits above\.</CODE\_USAGE\>Here is the basic tools provided:<GRAPH\_DATABASE\_INTERACTION\>NETWORKX GRAPH CHEATSHEETContext:import networkx as nxG = nx\.Graph\(\)1\. NODE OPERATIONS\- G\.add\_node\(node, \*\*attrs\): Add a single node with optional attributes\.\- G\.add\_nodes\_from\(\[n1, n2\], \*\*common\_attrs\): Add multiple nodes at once \(shared attributes apply to all\)\.\- G\.remove\_node\(node\): Remove a node and all edges connected to it\.\- G\.remove\_nodes\_from\(\[n1, n2\]\): Remove multiple nodes\.\- node in G: Check if a node exists\.\- G\.nodes: Get all nodes \(NodeView\)\.\- G\.nodes\[node\]: Access node attributes as a dict\.\- nx\.set\_node\_attributes\(G, \{node: \{"attr": value\}\}\): Set attributes for nodes\.2\. EDGE OPERATIONS\- G\.add\_edge\(u, v, \*\*attrs\): Add an edge between two nodes\.\- G\.add\_edges\_from\(\[\(u, v\), \(x, y\)\], \*\*attrs\): Add multiple edges at once \(shared attributes apply to all\)\.\- G\.remove\_edge\(u, v\): Remove a single edge\.\- G\.remove\_edges\_from\(\[\(u, v\), \(x, y\)\]\): Remove multiple edges\.\- G\.has\_edge\(u, v\): Check if an edge exists\.\- G\.edges: Get all edges \(EdgeView\)\.\- G\.edges\[\(u, v\)\]: Access edge attributes as a dict\.\- nx\.set\_edge\_attributes\(G, \{\(u, v\): \{"weight": 1\.0\}\}\): Set attributes for edges\.3\. TRAVERSAL / NEIGHBORHOOD\- G\.neighbors\(node\): Get neighbors of a node\.\- G\.adj\[node\]: Get dict of neighbors with edge data\.\- nx\.shortest\_path\(G, source, target\): Find one shortest path between nodes\.\- nx\.shortest\_path\_length\(G, source, target\): Get shortest path length\.\- nx\.all\_simple\_paths\(G, source, target, cutoff\): Generate all simple paths up to a cutoff length\.\- nx\.connected\_components\(G\): Get connected components as node sets\.\- G\.subgraph\(\[n1, n2, n3\]\): Extract a subgraph induced by given nodes\.4\. ANALYSIS / CENTRALITY\- G\.degree\(node\): Get degree \(number of edges\) for a single node\.\- G\.degree\(\): Get degree for all nodes \(DegreeView\)\.\- nx\.degree\_centrality\(G\): Compute degree centrality \(dict of node \-\> score\)\.\- nx\.betweenness\_centrality\(G\): Compute betweenness centrality\.\- nx\.pagerank\(G\): Compute PageRank scores for nodes\.\- nx\.clustering\(G\): Compute local clustering coefficient\.\- nx\.is\_connected\(G\): Check if graph is connected\.\- nx\.number\_connected\_components\(G\): Count connected components\.5\. UTILITIES\- G\.copy\(\): Make a copy of the graph\.\- G\.clear\(\): Remove all nodes and edges\.\- nx\.to\_dict\_of\_dicts\(G\): Convert graph to adjacency dict\.\- nx\.to\_numpy\_array\(G\): Get adjacency matrix as a NumPy array\.</GRAPH\_DATABASE\_INTERACTION\><CHROMA\_DATABASE\_INTERACTION\>\#\# Initialize Chroma DBImport: ‘from langchain\_chroma import Chroma‘Use ‘embedder = EmbeddingClient\(\)‘ and ‘db = Chroma\(embedding\_function=embedder\)‘ to create the database\. DO NOT use persist\_dir\.The default LangChain helpers \(‘add\_texts‘, ‘similarity\_search\(query: str\)‘\) are \*\*text\-only\*\* on the query side: ‘similarity\_search‘ takes a string query, which ‘EmbeddingClient‘ embeds as a \*\*single fused text vector\*\* \(DashScope multimodal embedding in text mode\)\.For \*\*screenshots, multiple images, or text\+image fusion\*\*, do \*\*not\*\* assume ‘similarity\_search‘ sees pixels\. Build fused vectors explicitly with ‘await embedder\.get\_fused\_embedding\(\[…\]\)‘ \(see Embedding Client in TOOLS\)\. One ‘get\_fused\_embedding‘ call returns \*\*one\*\* fused vector; need multiple vectors → call \*\*multiple times\*\*\.Recommended multimodal storage / retrieval pattern:\- Keep the \*\*human\-readable memory content\*\* \(summary, advice, metadata\) in Chroma / graph / your normal memory structure\.\- Store each multimodal fused vector in a \*\*sidecar index\*\* keyed by a stable ‘memory\_id‘ \(for example a dict or list storing ‘\{memory\_id, embedding, metadata\}‘\)\.\- At retrieve time, build \*\*one fused query vector\*\*, compare it against stored multimodal vectors \(for example with ‘EmbeddingClient\.compute\_one\_to\_group\_similarity\(…\)‘\), rank by similarity, then map the top hits back to ‘memory\_id‘ and finally return the linked readable memory content\.\- For recorder screenshots, local image paths like ‘recorder\.init\.images\[i\]\.value‘ are relative to the current episode artifact root\. In normal worker execution, ‘EmbeddingClient‘ can resolve those relative paths automatically from runtime context\. If you work with images outside that runtime, pass ‘base\_dir=‘ explicitly\.\#\#\# Add Memory: Adds new text entries to the database and returns their unique IDs\.db\.add\_texts\(texts: List\[str\],metadatas: Optional\[Union\[str, int, float, bool, None\]\] = None,ids: Optional\[List\[str\]\] = None\) \-\> List\[str\]\- metadatas must be \*\*flat list\*\*: each value must be a single primitive type \(str, int, float, bool, or None\)\.\- You cannot pass lists, nested dicts, or other complex objects\.\- If you need to store structured data, serialize it to a JSON string:\#\#\# Retrieve Memorydb\.similarity\_search\(query: str,k: int = 4\) \-\> List\[Document\]‘query‘ is plain text only \(text embedding path\)\. For multimodal queries, use ‘get\_fused\_embedding\(…\)‘, run your own similarity step, then map results back to stored memory items\.return List\[Document\]: \[Document\(page\_content="the agent found a key",metadata=\{"type": "item"\}\)\]\#\#\# Get by IDdb\.get\(ids: Optional\[List\[str\]\] = None\) \-\> Dict\[str, List\]\#\#\# Delete Memorydb\.delete\(ids: Optional\[List\[str\]\] = None\) \-\> None</CHROMA\_DATABASE\_INTERACTION\><OTHER\_TOOLS\>TOOLS AVAILABLE:1\. Multimodal Chat ClientClass: MultimodalChatClient\- Purpose: Asynchronous wrapper around OpenAI\-compatible Chat Completions for text or multimodal user content\. Use this when you need summarisation, synthesis, planning, or structured JSON output inside the memory code\.\- Initialization \(memory structure code — \*\*must\*\* match the benchmark execution model\):from meta\_self\_evolve\.llm\.client import MultimodalChatClientclient = MultimodalChatClient\(\)\- Do \*\*not\*\* pass ‘model=…‘ inside generated memory code\.\- Key Methods:\- await client\.complete\(messages: List\[Dict\[str, Any\]\]\) \-\> str:Send a standard chat message list and return plain text\.\- await client\.complete\_with\_system\(\*, system\_prompt: str, user\_input: str \| list, history: Optional\[List\[Dict\]\] = None, …\) \-\> str:Convenience wrapper for one system prompt \+ one user input \(\+ optional history\)\. \*\*All parameters after ‘client‘ are keyword\-only\*\* \(the real signature uses ‘\*‘ before ‘system\_prompt=‘\)\.\- await client\.complete\_json\_object\(messages: List\[Dict\[str, Any\]\], \*, temperature: float \| None = None\) \-\> Dict\[str, Any\]:Request a JSON object response\. Pass ‘messages‘ positionally; ‘temperature=‘ is keyword\-only if needed\.\- await client\.complete\_json\_with\_system\(\*, system\_prompt: str, user\_input: str \| list, history: Optional\[List\[Dict\]\] = None, temperature: float \| None = None\) \-\> Dict\[str, Any\]:Convenience wrapper for structured JSON output with a system prompt\. \*\*All parameters after ‘client‘ are keyword\-only\*\* \(the real signature uses ‘\*‘ before ‘system\_prompt=‘\)\.\- Usage Example:reply = await client\.complete\_with\_system\(system\_prompt="You summarize browser trajectories\.",user\_input="Summarize the key failure pattern in these steps\.",\)structured = await client\.complete\_json\_with\_system\(system\_prompt="Return ONLY a JSON object matching the schema described below\.",user\_input="…",\)\- IMPORTANT:\- \*\*Never\*\* call ‘complete\_with\_system\(…\)‘ or ‘complete\_json\_with\_system\(…\)‘ with positional arguments \(e\.g\. ‘complete\_json\_with\_system\(system\_prompt, user\_input\)‘ will raise ‘TypeError‘\)\. Always use ‘system\_prompt=…‘, ‘user\_input=…‘, and optionally ‘history=…‘, ‘temperature=…‘ as keywords\.\- If you need structured output, use ‘complete\_json\_object\(…\)‘ or ‘complete\_json\_with\_system\(…\)‘\.\- JSON output schemas should still be described clearly inside your system prompt\. Format example:\{"location": \{"type": "string","description": "The location to get the weather for"\},"unit": \{"type": \["string", "null"\],"description": "The unit to return the temperature in","enum": \["F", "C"\]\}\}\- Maintain history explicitly by passing prior messages in ‘history=‘ or by constructing the full ‘messages‘ list yourself\.2\. Embedding ClientClass: EmbeddingClient\- Purpose: DashScope \*\*qwen3\-vl\-embedding\*\* \(fixed model and fixed vector dimension inside the class\)\. Supports \*\*fused\*\* vectors for text\-only, image\(s\), or text\+image mixed inputs\. Optional cosine similarity helpers\.\- Initialization \(do \*\*not\*\* pass an embedding model name\):from meta\_self\_evolve\.llm\.embedding import EmbeddingClient, embed\_item\_text, embed\_item\_image\_path, embed\_item\_image\_urlembedder = EmbeddingClient\(retries=3, retry\_delay=1\.0\)\- Key Methods:\- await embedder\.get\_embedding\(text: str\) \-\> List\[float\]:Single \*\*text\*\* string → one fused vector \(text\-only path\)\.\- await embedder\.get\_batch\_embeddings\(texts: List\[str\]\) \-\> List\[List\[float\]\]:One fused vector per string \(independent calls\)\.\- await embedder\.get\_fused\_embedding\(items: List, base\_dir: Optional\[Path \| str\] = None\) \-\> List\[float\]:\*\*Multimodal fusion\*\*: ‘items‘ is an ordered list of parts \(from the helpers above\):\- ‘embed\_item\_text\("…"\)‘ → ‘\{type:’text’, text:…\}‘\- ‘embed\_item\_image\_path\("relative/or/abs/path\.png"\)‘ → local image; if relative, ‘EmbeddingClient‘ resolves it against ‘base\_dir=‘ when provided, otherwise against the worker’s current artifact\-root runtime context\.\- ‘embed\_item\_image\_url\("https://…"\)‘ → image URL\.One call uses ‘enable\_fusion=True‘ and returns \*\*one\*\* vector\. If you need \*\*multiple\*\* vectors \(e\.g\. separate vectors per document\), call ‘get\_fused\_embedding‘ \*\*multiple times\*\*—do not pack unrelated fusion targets into one call expecting multiple vectors\.\- await EmbeddingClient\.compute\_similarity\(emb1: List\[float\], emb2: List\[float\], metric: str = "cosine"\) \-\> float:Computes similarity between two embeddings asynchronously\.\- await EmbeddingClient\.compute\_one\_to\_group\_similarity\(emb: List\[float\], group\_emb: List\[List\[float\]\], metric: str = "cosine"\) \-\> List\[float\]:Computes similarity between one embedding and a group of embeddings asynchronously\.\- ‘EmbeddingClient\(\)‘ also implements ‘embed\_query\(\)‘ / ‘embed\_documents\(\)‘ / ‘\_\_call\_\_‘ so it can be passed into Chroma’s ‘embedding\_function=…‘ for \*\*text\*\* document/query paths\.\- Notes:\* Requires ‘DASHSCOPE\_API\_KEY‘ in the environment for DashScope\.\* For recorder screenshots in normal evaluation/runtime, relative image paths can usually be passed directly because the worker sets an artifact\-root runtime context\. If you use the embedder outside that runtime, pass ‘base\_dir=‘ explicitly\.\* Similarity helpers support cosine similarity and run in parallel for efficiency\.\* Do not import from ‘utils\.hire\_agent‘ or ‘evals\.utils\.hire\_agent‘; use ‘meta\_self\_evolve\.llm\.\*‘ only\.</OTHER\_TOOLS\>\#\#\# Tooling constraints \(memory structure code\)\- Use ‘MultimodalChatClient\(\)‘ with \*\*no\*\* ‘model=‘ argument so chat calls use the same model as the benchmark agent \(‘–execution\-model‘\)\.\- Use ‘EmbeddingClient\(\)‘ with \*\*no\*\* embedding model argument; it uses the built\-in DashScope multimodal embedding with a fixed dimension\.\- For text\+image or multi\-image \*\*fusion\*\* vectors, use ‘await embedder\.get\_fused\_embedding\(\[…\], base\_dir=…\)‘ with ‘embed\_item\_text‘ / ‘embed\_item\_image\_path‘ / ‘embed\_item\_image\_url‘\. In normal worker execution, recorder screenshot paths can usually be passed directly without ‘base\_dir=‘ because the worker sets artifact\-root runtime context\. \*\*One\*\* call returns \*\*one\*\* fused vector; for multiple independent vectors, call \*\*multiple times\*\*\.\- For multimodal memory, keep a sidecar vector index keyed by ‘memory\_id‘, then retrieve by fused query vector → similarity ranking → map top hits back to readable memory content\.\- Do not hard\-code OpenAI embedding model names or arbitrary chat model IDs in memory code\.\#\#\# Your Task:Modify the code above so that it fully satisfies the following design goals:1\. \*\*Modular, clear design:\*\*\- Prefer a small, readable structure \(private helpers, inner classes, or a single ‘MemoStructure‘ with well\-named methods\)\. Avoid unnecessary scaffolding\.\- If you use several stores \(e\.g\. Chroma, graph\), give each a clear role and data shape\.2\. \*\*Retrieve / update orchestration:\*\*\- Implement ‘MemoStructure‘ with ‘retrieve‘ / ‘update‘\.\- ‘retrieve\(\)‘ can chain internal steps when that helps\.\- ‘update\(\)‘ should persist new evidence from the finished trajectory in a consistent order\.3\. \*\*Out\-of\-the\-Box Reasoning:\*\*\- Do not just mechanically call each layer one by one — think about the \*\*semantic flow of information\*\*\.\- Consider cases like:\- what type of memory layer can be used according to the analysis result or task description, with the aim to better assist the agent to finish it’s task?\- what order and input output should be best suitable for the analysis result or task description?\- Think about high\-level stategy: what can be a good memory structure, and has good ability to transfer to other area?\- Make sure each layer plays a meaningful role in the system\.\- Directly perform simple plans or content writing based on if/else patterns should be avoided, since this will hurt the transfer ability\.\- Keep the retrieved memory clean and useful, aviod cutting off meaningful texts, repeat same patterns in the retrieved memory\.4\. \*\*Integration with Utilities:\*\*\- Feel free to use any provided utility functions \(e\.g\., similarity calculation, interaction with databases, hire new agent\) if relevant\. The tools available will be listed in ‘TOOLS‘ section\.\- You can also create your own tools if neccessary, think out of the box\.5\. \*\*Code Quality:\*\*\- Output clean, runnable Python code following PEP8\.\- Ensure ‘retrieve\(\)‘ and ‘update\(\)‘ accept ‘EpisodeRecorder‘ and orchestrate the pipeline end\-to\-end\.\- Initialize any stores or clients in ‘MemoStructure\.\_\_init\_\_‘ as needed\.\- Define a module\-level factory function exactly as ‘def build\_memo\(\) \-\> MemoStructure‘, and return an instance of your top\-level ‘MemoStructure‘ subclass\.\- Do not overuse defensive programming; raise appropriate exceptions when unexpected conditions occur to facilitate debugging\.6\. \*\*Coherent Policy Logic:\*\*\- Avoid placeholders like pass or \# TODO\.\- Avoid hard\-coded if/else branches or enumerated case handling; instead, express the logic through modular policy functions, scoring mechanisms, or composable decision rules\.\- Instead of enumerating case\-specific rules, express generalizable principles that could apply across different families or new unseen tasks\.\- The logic should be adaptable and compositional, not dependent on predefined constants or string names\.\- Use abstractions instead of specific family identifiers\.The goal is to ensure the memory policy behaves consistently across tasks and supports generalization, not to hard\-code specific task behaviors\.\#\#\# Important:\- Think creatively about data flow — outputs of one layer can feed into the next\.\- Each layer’s functionality and stored data should be clearly designed\.\- Use project\-local imports under ‘meta\_self\_evolve\.\*‘; do not import from ‘utils\.hire\_agent‘ or ‘evals\.utils\.hire\_agent‘\.\- Provide \*\*only the final rewritten code\*\*, no explanations\.\[USER\]Here is the current code that you must edit:<CURRENT\_CODE\>\{source\\\_code\}</CURRENT\_CODE\>Here is the score of current code:<REWARD\>\{benchmark\\\_overall\\\_eval\\\_score\}</REWARD\>Here is the analysis result \(suggestions\):<ANALYSIS\_RESULT\>\{"trajectory\_score\_assessment": "\{trajectory\\\_score\\\_assessment\}","suggested\_changes": "\{suggested\\\_changes\}"\}</ANALYSIS\_RESULT\>

Prompt 3: Meta Agent Generate New Code Prompt Without Existing Source

Meta Agent Generate New Code Prompt Without Existing Source\[SYSTEM\]You are a senior AI software engineer\. Your task is to build a clear, maintainable memory system for a downstream task agent\. The agent will be used in\{env\\\_id\}\. Your memory structure should supply relevant experience and reference beyond raw trajectory text\.\{task\\\_description\}You are given the following backbone \(“MemoStructure“ with “retrieve“ / “update“\):<BACKBONE\_CODE\>"""Abstract memo backbone inlined into codegen prompts\.Only :class:‘MemoStructure‘ is required\. Implement a clear, maintainable“retrieve“ / “update“ pair; use private helpers or inner classes as needed\."""from \_\_future\_\_ import annotationsfrom abc import ABC, abstractmethodfrom typing import Any, Optionalfrom meta\_self\_evolve\.contracts\.types import EpisodeRecorder, RetrievedMemoryPayloadclass MemoStructure\(ABC\):def \_\_init\_\_\(self\) \-\> None:self\.database: Optional\[Any\] = None@abstractmethodasync def retrieve\(self, recorder: EpisodeRecorder\) \-\> RetrievedMemoryPayload:"""Return structured memory for the next episode \(see “RetrievedMemoryPayload“\)\."""…@abstractmethodasync def update\(self, recorder: EpisodeRecorder\) \-\> None:"""Incorporate a finished trajectory \(init, steps, reward, etc\.\)\."""…</BACKBONE\_CODE\>Subclass “MemoStructure“ and import the types you need, for example:“‘pythonfrom meta\_self\_evolve\.common\.retrieved\_memory import empty\_retrieved\_memory\_payloadfrom meta\_self\_evolve\.contracts\.memo\_backbone import MemoStructurefrom meta\_self\_evolve\.contracts\.types import EpisodeRecorder, RetrievedMemoryPayloadfrom meta\_self\_evolve\.llm\.client import MultimodalChatClientfrom meta\_self\_evolve\.llm\.embedding import \(EmbeddingClient,embed\_item\_image\_path,embed\_item\_image\_url,embed\_item\_text,\)\# plus your choice of Chroma, networkx, etc\.“‘<CODE\_INPUT\>Your ‘retrieve‘ and ‘update‘ will take ‘EpisodeRecorder‘ as input, which has following attributes:\{"init": \{"description": "Initial task text and observation images\.","type": "InitRecord"\},"steps": \{"description": "Per\-step action and observation history\.","type": "list\[StepRecord\]"\},"memory\_retrieved": \{"description": "Structured output from retrieve\(\) before the episode\.","type": "RetrievedMemoryPayload"\},"reward": \{"description": "Scalar episode reward after finish\.","type": "float"\},"messages": \{"description": "Full LLM conversation history for this episode\.","type": "list\[dict\]"\}\}\- For ‘retrieve‘, only leverage ‘\.init‘ attribute\.\- For ‘update‘, leverage ‘\.init‘, ‘\.steps‘, ‘reward‘ attribute\.\- Each element in the above dict is a attribute name as key, and description, type, and a example for the exact possible value the attribute could have\.\- please note that all provided current trajectory can already been seen by down stream agents\(in history\), your memory structure should focus on provide extra advice and reference for agents\.</CODE\_INPUT\><CODE\_USAGE\>Your memory structure will be used in the agent workflow:\- ‘retrieve\(recorder\)‘: used \*\*before\*\* executing the task; must return “RetrievedMemoryPayload“ \(see schema below\)\. Populate “items“ with concise “text“ and optional “images“ \(“ImageRef“ JSON\) so the execution agent receives useful guidance\.\- ‘update\(recorder\)‘: used \*\*after\*\* task is finished, to update the trajectory, reward, or other information\.\#\#\# RetrievedMemoryPayload \(return type of “MemoStructure\.retrieve“\)“retrieve“ must return a \*\*JSON\-serializable\*\* “RetrievedMemoryPayload“ \(import from “meta\_self\_evolve\.contracts\.types“\), i\.e\. a dict with:\- “items“: a \*\*flat\*\* list of memory dicts; each dict may include:\- “text“ \(“str“, optional\): guidance for the execution agent\.\- “images“ \(optional\): list of image refs with the same JSON shape as trajectory “ImageRef“: “\{"kind": "path"\|"url", "value": "<relative path or URL\>", "mime": "<optional\>"\}“\. When “kind“ is “"path"“, “value“ is resolved relative to the episode \*\*artifact root\*\*\.\- “metadata“ \(“dict“, optional\): any structured fields \(e\.g\. “memory\_id“, “score“, “source“, layer names\)\.\- “metadata“ \(“dict“\): episode\-level retrieval notes \(use “\{\}“ if none\)\.\*\*Empty retrieval:\*\* return “\{"items": \[\], "metadata": \{\}\}“ \(or use “empty\_retrieved\_memory\_payload\(\)“ from “meta\_self\_evolve\.common\.retrieved\_memory“\)\.\*\*Image / text budgets \(truncation is expected\):\*\*\- Put \*\*only a small number\*\* of images in “items\[\*\]\.images“ \(typically 0–2\)\. Extra images are \*\*dropped\*\*; do not rely on the model seeing every path you emit\.\- The JSON text of the payload is also \*\*truncated\*\* to “RolloutLimits\.max\_retrieved\_memory\_chars“ in the execution agent’s first user turn\.\- \*\*Execution agent:\*\* at most “RolloutLimits\.max\_retrieved\_memory\_images“ images are loaded from the payload \(in “items“ order\) on the first user turn; this is \*\*separate\*\* from the per\-step screenshot budget “RolloutLimits\.max\_images\_per\_step“ for environment observations\.\- \*\*Meta analysis:\*\* after the truncated JSON block, at most “MetaTrajectoryLimits\.meta\_retrieved\_memory\_images\_per\_episode“ payload images are attached per sampled episode \(again in order\); this is \*\*separate\*\* from “MetaTrajectoryLimits\.max\_meta\_images\_per\_episode“ for trajectory screenshots\.\*\*Runtime:\*\* the worker injects this into the first user turn as \(1\) a text block \(header \+ pretty\-printed JSON of the payload\), then \(2\) “image\_url“ parts for each referenced image, subject to the limits above\.</CODE\_USAGE\>Here is the basic tools provided:<GRAPH\_DATABASE\_INTERACTION\>NETWORKX GRAPH CHEATSHEETContext:import networkx as nxG = nx\.Graph\(\)1\. NODE OPERATIONS\- G\.add\_node\(node, \*\*attrs\): Add a single node with optional attributes\.\- G\.add\_nodes\_from\(\[n1, n2\], \*\*common\_attrs\): Add multiple nodes at once \(shared attributes apply to all\)\.\- G\.remove\_node\(node\): Remove a node and all edges connected to it\.\- G\.remove\_nodes\_from\(\[n1, n2\]\): Remove multiple nodes\.\- node in G: Check if a node exists\.\- G\.nodes: Get all nodes \(NodeView\)\.\- G\.nodes\[node\]: Access node attributes as a dict\.\- nx\.set\_node\_attributes\(G, \{node: \{"attr": value\}\}\): Set attributes for nodes\.2\. EDGE OPERATIONS\- G\.add\_edge\(u, v, \*\*attrs\): Add an edge between two nodes\.\- G\.add\_edges\_from\(\[\(u, v\), \(x, y\)\], \*\*attrs\): Add multiple edges at once \(shared attributes apply to all\)\.\- G\.remove\_edge\(u, v\): Remove a single edge\.\- G\.remove\_edges\_from\(\[\(u, v\), \(x, y\)\]\): Remove multiple edges\.\- G\.has\_edge\(u, v\): Check if an edge exists\.\- G\.edges: Get all edges \(EdgeView\)\.\- G\.edges\[\(u, v\)\]: Access edge attributes as a dict\.\- nx\.set\_edge\_attributes\(G, \{\(u, v\): \{"weight": 1\.0\}\}\): Set attributes for edges\.3\. TRAVERSAL / NEIGHBORHOOD\- G\.neighbors\(node\): Get neighbors of a node\.\- G\.adj\[node\]: Get dict of neighbors with edge data\.\- nx\.shortest\_path\(G, source, target\): Find one shortest path between nodes\.\- nx\.shortest\_path\_length\(G, source, target\): Get shortest path length\.\- nx\.all\_simple\_paths\(G, source, target, cutoff\): Generate all simple paths up to a cutoff length\.\- nx\.connected\_components\(G\): Get connected components as node sets\.\- G\.subgraph\(\[n1, n2, n3\]\): Extract a subgraph induced by given nodes\.4\. ANALYSIS / CENTRALITY\- G\.degree\(node\): Get degree \(number of edges\) for a single node\.\- G\.degree\(\): Get degree for all nodes \(DegreeView\)\.\- nx\.degree\_centrality\(G\): Compute degree centrality \(dict of node \-\> score\)\.\- nx\.betweenness\_centrality\(G\): Compute betweenness centrality\.\- nx\.pagerank\(G\): Compute PageRank scores for nodes\.\- nx\.clustering\(G\): Compute local clustering coefficient\.\- nx\.is\_connected\(G\): Check if graph is connected\.\- nx\.number\_connected\_components\(G\): Count connected components\.5\. UTILITIES\- G\.copy\(\): Make a copy of the graph\.\- G\.clear\(\): Remove all nodes and edges\.\- nx\.to\_dict\_of\_dicts\(G\): Convert graph to adjacency dict\.\- nx\.to\_numpy\_array\(G\): Get adjacency matrix as a NumPy array\.</GRAPH\_DATABASE\_INTERACTION\><CHROMA\_DATABASE\_INTERACTION\>\#\# Initialize Chroma DBImport: ‘from langchain\_chroma import Chroma‘Use ‘embedder = EmbeddingClient\(\)‘ and ‘db = Chroma\(embedding\_function=embedder\)‘ to create the database\. DO NOT use persist\_dir\.The default LangChain helpers \(‘add\_texts‘, ‘similarity\_search\(query: str\)‘\) are \*\*text\-only\*\* on the query side: ‘similarity\_search‘ takes a string query, which ‘EmbeddingClient‘ embeds as a \*\*single fused text vector\*\* \(DashScope multimodal embedding in text mode\)\.For \*\*screenshots, multiple images, or text\+image fusion\*\*, do \*\*not\*\* assume ‘similarity\_search‘ sees pixels\. Build fused vectors explicitly with ‘await embedder\.get\_fused\_embedding\(\[…\]\)‘ \(see Embedding Client in TOOLS\)\. One ‘get\_fused\_embedding‘ call returns \*\*one\*\* fused vector; need multiple vectors → call \*\*multiple times\*\*\.Recommended multimodal storage / retrieval pattern:\- Keep the \*\*human\-readable memory content\*\* \(summary, advice, metadata\) in Chroma / graph / your normal memory structure\.\- Store each multimodal fused vector in a \*\*sidecar index\*\* keyed by a stable ‘memory\_id‘ \(for example a dict or list storing ‘\{memory\_id, embedding, metadata\}‘\)\.\- At retrieve time, build \*\*one fused query vector\*\*, compare it against stored multimodal vectors \(for example with ‘EmbeddingClient\.compute\_one\_to\_group\_similarity\(…\)‘\), rank by similarity, then map the top hits back to ‘memory\_id‘ and finally return the linked readable memory content\.\- For recorder screenshots, local image paths like ‘recorder\.init\.images\[i\]\.value‘ are relative to the current episode artifact root\. In normal worker execution, ‘EmbeddingClient‘ can resolve those relative paths automatically from runtime context\. If you work with images outside that runtime, pass ‘base\_dir=‘ explicitly\.\#\#\# Add Memory: Adds new text entries to the database and returns their unique IDs\.db\.add\_texts\(texts: List\[str\],metadatas: Optional\[Union\[str, int, float, bool, None\]\] = None,ids: Optional\[List\[str\]\] = None\) \-\> List\[str\]\- metadatas must be \*\*flat list\*\*: each value must be a single primitive type \(str, int, float, bool, or None\)\.\- You cannot pass lists, nested dicts, or other complex objects\.\- If you need to store structured data, serialize it to a JSON string:\#\#\# Retrieve Memorydb\.similarity\_search\(query: str,k: int = 4\) \-\> List\[Document\]‘query‘ is plain text only \(text embedding path\)\. For multimodal queries, use ‘get\_fused\_embedding\(…\)‘, run your own similarity step, then map results back to stored memory items\.return List\[Document\]: \[Document\(page\_content="the agent found a key",metadata=\{"type": "item"\}\)\]\#\#\# Get by IDdb\.get\(ids: Optional\[List\[str\]\] = None\) \-\> Dict\[str, List\]\#\#\# Delete Memorydb\.delete\(ids: Optional\[List\[str\]\] = None\) \-\> None</CHROMA\_DATABASE\_INTERACTION\><OTHER\_TOOLS\>TOOLS AVAILABLE:1\. Multimodal Chat ClientClass: MultimodalChatClient\- Purpose: Asynchronous wrapper around OpenAI\-compatible Chat Completions for text or multimodal user content\. Use this when you need summarisation, synthesis, planning, or structured JSON output inside the memory code\.\- Initialization \(memory structure code — \*\*must\*\* match the benchmark execution model\):from meta\_self\_evolve\.llm\.client import MultimodalChatClientclient = MultimodalChatClient\(\)\- Do \*\*not\*\* pass ‘model=…‘ inside generated memory code\.\- Key Methods:\- await client\.complete\(messages: List\[Dict\[str, Any\]\]\) \-\> str:Send a standard chat message list and return plain text\.\- await client\.complete\_with\_system\(\*, system\_prompt: str, user\_input: str \| list, history: Optional\[List\[Dict\]\] = None, …\) \-\> str:Convenience wrapper for one system prompt \+ one user input \(\+ optional history\)\. \*\*All parameters after ‘client‘ are keyword\-only\*\* \(the real signature uses ‘\*‘ before ‘system\_prompt=‘\)\.\- await client\.complete\_json\_object\(messages: List\[Dict\[str, Any\]\], \*, temperature: float \| None = None\) \-\> Dict\[str, Any\]:Request a JSON object response\. Pass ‘messages‘ positionally; ‘temperature=‘ is keyword\-only if needed\.\- await client\.complete\_json\_with\_system\(\*, system\_prompt: str, user\_input: str \| list, history: Optional\[List\[Dict\]\] = None, temperature: float \| None = None\) \-\> Dict\[str, Any\]:Convenience wrapper for structured JSON output with a system prompt\. \*\*All parameters after ‘client‘ are keyword\-only\*\* \(the real signature uses ‘\*‘ before ‘system\_prompt=‘\)\.\- Usage Example:reply = await client\.complete\_with\_system\(system\_prompt="You summarize browser trajectories\.",user\_input="Summarize the key failure pattern in these steps\.",\)structured = await client\.complete\_json\_with\_system\(system\_prompt="Return ONLY a JSON object matching the schema described below\.",user\_input="…",\)\- IMPORTANT:\- \*\*Never\*\* call ‘complete\_with\_system\(…\)‘ or ‘complete\_json\_with\_system\(…\)‘ with positional arguments \(e\.g\. ‘complete\_json\_with\_system\(system\_prompt, user\_input\)‘ will raise ‘TypeError‘\)\. Always use ‘system\_prompt=…‘, ‘user\_input=…‘, and optionally ‘history=…‘, ‘temperature=…‘ as keywords\.\- If you need structured output, use ‘complete\_json\_object\(…\)‘ or ‘complete\_json\_with\_system\(…\)‘\.\- JSON output schemas should still be described clearly inside your system prompt\. Format example:\{"location": \{"type": "string","description": "The location to get the weather for"\},"unit": \{"type": \["string", "null"\],"description": "The unit to return the temperature in","enum": \["F", "C"\]\}\}\- Maintain history explicitly by passing prior messages in ‘history=‘ or by constructing the full ‘messages‘ list yourself\.2\. Embedding ClientClass: EmbeddingClient\- Purpose: DashScope \*\*qwen3\-vl\-embedding\*\* \(fixed model and fixed vector dimension inside the class\)\. Supports \*\*fused\*\* vectors for text\-only, image\(s\), or text\+image mixed inputs\. Optional cosine similarity helpers\.\- Initialization \(do \*\*not\*\* pass an embedding model name\):from meta\_self\_evolve\.llm\.embedding import EmbeddingClient, embed\_item\_text, embed\_item\_image\_path, embed\_item\_image\_urlembedder = EmbeddingClient\(retries=3, retry\_delay=1\.0\)\- Key Methods:\- await embedder\.get\_embedding\(text: str\) \-\> List\[float\]:Single \*\*text\*\* string → one fused vector \(text\-only path\)\.\- await embedder\.get\_batch\_embeddings\(texts: List\[str\]\) \-\> List\[List\[float\]\]:One fused vector per string \(independent calls\)\.\- await embedder\.get\_fused\_embedding\(items: List, base\_dir: Optional\[Path \| str\] = None\) \-\> List\[float\]:\*\*Multimodal fusion\*\*: ‘items‘ is an ordered list of parts \(from the helpers above\):\- ‘embed\_item\_text\("…"\)‘ → ‘\{type:’text’, text:…\}‘\- ‘embed\_item\_image\_path\("relative/or/abs/path\.png"\)‘ → local image; if relative, ‘EmbeddingClient‘ resolves it against ‘base\_dir=‘ when provided, otherwise against the worker’s current artifact\-root runtime context\.\- ‘embed\_item\_image\_url\("https://…"\)‘ → image URL\.One call uses ‘enable\_fusion=True‘ and returns \*\*one\*\* vector\. If you need \*\*multiple\*\* vectors \(e\.g\. separate vectors per document\), call ‘get\_fused\_embedding‘ \*\*multiple times\*\*—do not pack unrelated fusion targets into one call expecting multiple vectors\.\- await EmbeddingClient\.compute\_similarity\(emb1: List\[float\], emb2: List\[float\], metric: str = "cosine"\) \-\> float:Computes similarity between two embeddings asynchronously\.\- await EmbeddingClient\.compute\_one\_to\_group\_similarity\(emb: List\[float\], group\_emb: List\[List\[float\]\], metric: str = "cosine"\) \-\> List\[float\]:Computes similarity between one embedding and a group of embeddings asynchronously\.\- ‘EmbeddingClient\(\)‘ also implements ‘embed\_query\(\)‘ / ‘embed\_documents\(\)‘ / ‘\_\_call\_\_‘ so it can be passed into Chroma’s ‘embedding\_function=…‘ for \*\*text\*\* document/query paths\.\- Notes:\* Requires ‘DASHSCOPE\_API\_KEY‘ in the environment for DashScope\.\* For recorder screenshots in normal evaluation/runtime, relative image paths can usually be passed directly because the worker sets an artifact\-root runtime context\. If you use the embedder outside that runtime, pass ‘base\_dir=‘ explicitly\.\* Similarity helpers support cosine similarity and run in parallel for efficiency\.\* Do not import from ‘utils\.hire\_agent‘ or ‘evals\.utils\.hire\_agent‘; use ‘meta\_self\_evolve\.llm\.\*‘ only\.</OTHER\_TOOLS\>\#\#\# Tooling constraints \(memory structure code\)\- Use ‘MultimodalChatClient\(\)‘ with \*\*no\*\* ‘model=‘ argument so chat calls use the same model as the benchmark agent \(‘–execution\-model‘\)\.\- Use ‘EmbeddingClient\(\)‘ with \*\*no\*\* embedding model argument; it uses the built\-in DashScope multimodal embedding with a fixed dimension\.\- For text\+image or multi\-image \*\*fusion\*\* vectors, use ‘await embedder\.get\_fused\_embedding\(\[…\], base\_dir=…\)‘ with ‘embed\_item\_text‘ / ‘embed\_item\_image\_path‘ / ‘embed\_item\_image\_url‘\. In normal worker execution, recorder screenshot paths can usually be passed directly without ‘base\_dir=‘ because the worker sets artifact\-root runtime context\. \*\*One\*\* call returns \*\*one\*\* fused vector; for multiple independent vectors, call \*\*multiple times\*\*\.\- For multimodal memory, keep a sidecar vector index keyed by ‘memory\_id‘, then retrieve by fused query vector → similarity ranking → map top hits back to readable memory content\.\- Do not hard\-code OpenAI embedding model names or arbitrary chat model IDs in memory code\.\#\#\# Your Task:Modify the code above so that it fully satisfies the following design goals:1\. \*\*Modular, clear design:\*\*\- Prefer a small, readable structure \(private helpers, inner classes, or a single ‘MemoStructure‘ with well\-named methods\)\. Avoid unnecessary scaffolding\.\- If you use several stores \(e\.g\. Chroma, graph\), give each a clear role and data shape\.2\. \*\*Retrieve / update orchestration:\*\*\- Implement ‘MemoStructure‘ with ‘retrieve‘ / ‘update‘\.\- ‘retrieve\(\)‘ can chain internal steps when that helps\.\- ‘update\(\)‘ should persist new evidence from the finished trajectory in a consistent order\.3\. \*\*Out\-of\-the\-Box Reasoning:\*\*\- Do not just mechanically call each layer one by one — think about the \*\*semantic flow of information\*\*\.\- Consider cases like:\- what type of memory layer can be used according to the analysis result or task description, with the aim to better assist the agent to finish it’s task?\- what order and input output should be best suitable for the analysis result or task description?\- Think about high\-level stategy: what can be a good memory structure, and has good ability to transfer to other area?\- Make sure each layer plays a meaningful role in the system\.\- Directly perform simple plans or content writing based on if/else patterns should be avoided, since this will hurt the transfer ability\.\- Keep the retrieved memory clean and useful, aviod cutting off meaningful texts, repeat same patterns in the retrieved memory\.4\. \*\*Integration with Utilities:\*\*\- Feel free to use any provided utility functions \(e\.g\., similarity calculation, interaction with databases, hire new agent\) if relevant\. The tools available will be listed in ‘TOOLS‘ section\.\- You can also create your own tools if neccessary, think out of the box\.5\. \*\*Code Quality:\*\*\- Output clean, runnable Python code following PEP8\.\- Ensure ‘retrieve\(\)‘ and ‘update\(\)‘ accept ‘EpisodeRecorder‘ and orchestrate the pipeline end\-to\-end\.\- Initialize any stores or clients in ‘MemoStructure\.\_\_init\_\_‘ as needed\.\- Define a module\-level factory function exactly as ‘def build\_memo\(\) \-\> MemoStructure‘, and return an instance of your top\-level ‘MemoStructure‘ subclass\.\- Do not overuse defensive programming; raise appropriate exceptions when unexpected conditions occur to facilitate debugging\.6\. \*\*Coherent Policy Logic:\*\*\- Avoid placeholders like pass or \# TODO\.\- Avoid hard\-coded if/else branches or enumerated case handling; instead, express the logic through modular policy functions, scoring mechanisms, or composable decision rules\.\- Instead of enumerating case\-specific rules, express generalizable principles that could apply across different families or new unseen tasks\.\- The logic should be adaptable and compositional, not dependent on predefined constants or string names\.\- Use abstractions instead of specific family identifiers\.The goal is to ensure the memory policy behaves consistently across tasks and supports generalization, not to hard\-code specific task behaviors\.\#\#\# Important:\- Think creatively about data flow — outputs of one layer can feed into the next\.\- Each layer’s functionality and stored data should be clearly designed\.\- Use project\-local imports under ‘meta\_self\_evolve\.\*‘; do not import from ‘utils\.hire\_agent‘ or ‘evals\.utils\.hire\_agent‘\.\- Provide \*\*only the final rewritten code\*\*, no explanations\.\[USER\]Please generated new code based on your understanding about the task and requirements\.

Prompt 4: Meta Agent Reflection Repair Prompt

Meta Agent Reflection Repair Prompt\[SYSTEM\]You are a senior AI software engineer and code repair expert\.Your role is to carefully analyze the provided code and the error information, identify potential errors or design flaws, and directly rewrite or edit the code to fix those issues — while keeping the main design goals and intentions exactly the same\.You are given the following context and base classes:<BACKBONE\_CODE\>"""Abstract memo backbone inlined into codegen prompts\.Only :class:‘MemoStructure‘ is required\. Implement a clear, maintainable“retrieve“ / “update“ pair; use private helpers or inner classes as needed\."""from \_\_future\_\_ import annotationsfrom abc import ABC, abstractmethodfrom typing import Any, Optionalfrom meta\_self\_evolve\.contracts\.types import EpisodeRecorder, RetrievedMemoryPayloadclass MemoStructure\(ABC\):def \_\_init\_\_\(self\) \-\> None:self\.database: Optional\[Any\] = None@abstractmethodasync def retrieve\(self, recorder: EpisodeRecorder\) \-\> RetrievedMemoryPayload:"""Return structured memory for the next episode \(see “RetrievedMemoryPayload“\)\."""…@abstractmethodasync def update\(self, recorder: EpisodeRecorder\) \-\> None:"""Incorporate a finished trajectory \(init, steps, reward, etc\.\)\."""…</BACKBONE\_CODE\><CODE\_INPUT\>Your ‘retrieve‘, ‘update‘ will take ‘EpisodeRecorder‘ as input, which has following attributes:\{"init": \{"description": "Initial task text and observation images\.","type": "InitRecord"\},"steps": \{"description": "Per\-step action and observation history\.","type": "list\[StepRecord\]"\},"memory\_retrieved": \{"description": "Structured output from retrieve\(\) before the episode\.","type": "RetrievedMemoryPayload"\},"reward": \{"description": "Scalar episode reward after finish\.","type": "float"\},"messages": \{"description": "Full LLM conversation history for this episode\.","type": "list\[dict\]"\}\}\- For ‘retrieve‘, only leverage ‘\.init‘ attribute\.\- For ‘update‘, you can leverage all listed attribute\.</CODE\_INPUT\><CODE\_USAGE\>Your memory structure will be used in the agent workflow:\- ‘retrieve\(recorder\)‘: used \*\*before\*\* executing the task; must return “RetrievedMemoryPayload“ \(see schema below\)\.\- ‘update\(recorder\)‘: used \*\*after\*\* task is finished, to update the trajectory, reward, or other information\.\#\#\# RetrievedMemoryPayload \(return type of “MemoStructure\.retrieve“\)“retrieve“ must return a \*\*JSON\-serializable\*\* “RetrievedMemoryPayload“ \(import from “meta\_self\_evolve\.contracts\.types“\), i\.e\. a dict with:\- “items“: a \*\*flat\*\* list of memory dicts; each dict may include:\- “text“ \(“str“, optional\): guidance for the execution agent\.\- “images“ \(optional\): list of image refs with the same JSON shape as trajectory “ImageRef“: “\{"kind": "path"\|"url", "value": "<relative path or URL\>", "mime": "<optional\>"\}“\. When “kind“ is “"path"“, “value“ is resolved relative to the episode \*\*artifact root\*\*\.\- “metadata“ \(“dict“, optional\): any structured fields \(e\.g\. “memory\_id“, “score“, “source“, layer names\)\.\- “metadata“ \(“dict“\): episode\-level retrieval notes \(use “\{\}“ if none\)\.\*\*Empty retrieval:\*\* return “\{"items": \[\], "metadata": \{\}\}“ \(or use “empty\_retrieved\_memory\_payload\(\)“ from “meta\_self\_evolve\.common\.retrieved\_memory“\)\.\*\*Image / text budgets \(truncation is expected\):\*\*\- Put \*\*only a small number\*\* of images in “items\[\*\]\.images“ \(typically 0–2\)\. Extra images are \*\*dropped\*\*; do not rely on the model seeing every path you emit\.\- The JSON text of the payload is also \*\*truncated\*\* to “RolloutLimits\.max\_retrieved\_memory\_chars“ in the execution agent’s first user turn\.\- \*\*Execution agent:\*\* at most “RolloutLimits\.max\_retrieved\_memory\_images“ images are loaded from the payload \(in “items“ order\) on the first user turn; this is \*\*separate\*\* from the per\-step screenshot budget “RolloutLimits\.max\_images\_per\_step“ for environment observations\.\- \*\*Meta analysis:\*\* after the truncated JSON block, at most “MetaTrajectoryLimits\.meta\_retrieved\_memory\_images\_per\_episode“ payload images are attached per sampled episode \(again in order\); this is \*\*separate\*\* from “MetaTrajectoryLimits\.max\_meta\_images\_per\_episode“ for trajectory screenshots\.\*\*Runtime:\*\* the worker injects this into the first user turn as \(1\) a text block \(header \+ pretty\-printed JSON of the payload\), then \(2\) “image\_url“ parts for each referenced image, subject to the limits above\.</CODE\_USAGE\>\#\#\# Your Task:Carefully inspect the code, and error information, detect the root cause of the errors, structural issues, or missing implementations, and \*\*only fix the root cause code\*\*\. Here are some cheetsheet that could be useful:<GRAPH\_DATABASE\_INTERACTION\>NETWORKX GRAPH CHEATSHEETContext:import networkx as nxG = nx\.Graph\(\)1\. NODE OPERATIONS\- G\.add\_node\(node, \*\*attrs\): Add a single node with optional attributes\.\- G\.add\_nodes\_from\(\[n1, n2\], \*\*common\_attrs\): Add multiple nodes at once \(shared attributes apply to all\)\.\- G\.remove\_node\(node\): Remove a node and all edges connected to it\.\- G\.remove\_nodes\_from\(\[n1, n2\]\): Remove multiple nodes\.\- node in G: Check if a node exists\.\- G\.nodes: Get all nodes \(NodeView\)\.\- G\.nodes\[node\]: Access node attributes as a dict\.\- nx\.set\_node\_attributes\(G, \{node: \{"attr": value\}\}\): Set attributes for nodes\.2\. EDGE OPERATIONS\- G\.add\_edge\(u, v, \*\*attrs\): Add an edge between two nodes\.\- G\.add\_edges\_from\(\[\(u, v\), \(x, y\)\], \*\*attrs\): Add multiple edges at once \(shared attributes apply to all\)\.\- G\.remove\_edge\(u, v\): Remove a single edge\.\- G\.remove\_edges\_from\(\[\(u, v\), \(x, y\)\]\): Remove multiple edges\.\- G\.has\_edge\(u, v\): Check if an edge exists\.\- G\.edges: Get all edges \(EdgeView\)\.\- G\.edges\[\(u, v\)\]: Access edge attributes as a dict\.\- nx\.set\_edge\_attributes\(G, \{\(u, v\): \{"weight": 1\.0\}\}\): Set attributes for edges\.3\. TRAVERSAL / NEIGHBORHOOD\- G\.neighbors\(node\): Get neighbors of a node\.\- G\.adj\[node\]: Get dict of neighbors with edge data\.\- nx\.shortest\_path\(G, source, target\): Find one shortest path between nodes\.\- nx\.shortest\_path\_length\(G, source, target\): Get shortest path length\.\- nx\.all\_simple\_paths\(G, source, target, cutoff\): Generate all simple paths up to a cutoff length\.\- nx\.connected\_components\(G\): Get connected components as node sets\.\- G\.subgraph\(\[n1, n2, n3\]\): Extract a subgraph induced by given nodes\.4\. ANALYSIS / CENTRALITY\- G\.degree\(node\): Get degree \(number of edges\) for a single node\.\- G\.degree\(\): Get degree for all nodes \(DegreeView\)\.\- nx\.degree\_centrality\(G\): Compute degree centrality \(dict of node \-\> score\)\.\- nx\.betweenness\_centrality\(G\): Compute betweenness centrality\.\- nx\.pagerank\(G\): Compute PageRank scores for nodes\.\- nx\.clustering\(G\): Compute local clustering coefficient\.\- nx\.is\_connected\(G\): Check if graph is connected\.\- nx\.number\_connected\_components\(G\): Count connected components\.5\. UTILITIES\- G\.copy\(\): Make a copy of the graph\.\- G\.clear\(\): Remove all nodes and edges\.\- nx\.to\_dict\_of\_dicts\(G\): Convert graph to adjacency dict\.\- nx\.to\_numpy\_array\(G\): Get adjacency matrix as a NumPy array\.</GRAPH\_DATABASE\_INTERACTION\><CHROMA\_DATABASE\_INTERACTION\>\#\# Initialize Chroma DBImport: ‘from langchain\_chroma import Chroma‘Use ‘embedder = EmbeddingClient\(\)‘ and ‘db = Chroma\(embedding\_function=embedder\)‘ to create the database\. DO NOT use persist\_dir\.The default LangChain helpers \(‘add\_texts‘, ‘similarity\_search\(query: str\)‘\) are \*\*text\-only\*\* on the query side: ‘similarity\_search‘ takes a string query, which ‘EmbeddingClient‘ embeds as a \*\*single fused text vector\*\* \(DashScope multimodal embedding in text mode\)\.For \*\*screenshots, multiple images, or text\+image fusion\*\*, do \*\*not\*\* assume ‘similarity\_search‘ sees pixels\. Build fused vectors explicitly with ‘await embedder\.get\_fused\_embedding\(\[…\]\)‘ \(see Embedding Client in TOOLS\)\. One ‘get\_fused\_embedding‘ call returns \*\*one\*\* fused vector; need multiple vectors → call \*\*multiple times\*\*\.Recommended multimodal storage / retrieval pattern:\- Keep the \*\*human\-readable memory content\*\* \(summary, advice, metadata\) in Chroma / graph / your normal memory structure\.\- Store each multimodal fused vector in a \*\*sidecar index\*\* keyed by a stable ‘memory\_id‘ \(for example a dict or list storing ‘\{memory\_id, embedding, metadata\}‘\)\.\- At retrieve time, build \*\*one fused query vector\*\*, compare it against stored multimodal vectors \(for example with ‘EmbeddingClient\.compute\_one\_to\_group\_similarity\(…\)‘\), rank by similarity, then map the top hits back to ‘memory\_id‘ and finally return the linked readable memory content\.\- For recorder screenshots, local image paths like ‘recorder\.init\.images\[i\]\.value‘ are relative to the current episode artifact root\. In normal worker execution, ‘EmbeddingClient‘ can resolve those relative paths automatically from runtime context\. If you work with images outside that runtime, pass ‘base\_dir=‘ explicitly\.\#\#\# Add Memory: Adds new text entries to the database and returns their unique IDs\.db\.add\_texts\(texts: List\[str\],metadatas: Optional\[Union\[str, int, float, bool, None\]\] = None,ids: Optional\[List\[str\]\] = None\) \-\> List\[str\]\- metadatas must be \*\*flat list\*\*: each value must be a single primitive type \(str, int, float, bool, or None\)\.\- You cannot pass lists, nested dicts, or other complex objects\.\- If you need to store structured data, serialize it to a JSON string:\#\#\# Retrieve Memorydb\.similarity\_search\(query: str,k: int = 4\) \-\> List\[Document\]‘query‘ is plain text only \(text embedding path\)\. For multimodal queries, use ‘get\_fused\_embedding\(…\)‘, run your own similarity step, then map results back to stored memory items\.return List\[Document\]: \[Document\(page\_content="the agent found a key",metadata=\{"type": "item"\}\)\]\#\#\# Get by IDdb\.get\(ids: Optional\[List\[str\]\] = None\) \-\> Dict\[str, List\]\#\#\# Delete Memorydb\.delete\(ids: Optional\[List\[str\]\] = None\) \-\> None</CHROMA\_DATABASE\_INTERACTION\><OTHER\_TOOLS\>TOOLS AVAILABLE:1\. Multimodal Chat ClientClass: MultimodalChatClient\- Purpose: Asynchronous wrapper around OpenAI\-compatible Chat Completions for text or multimodal user content\. Use this when you need summarisation, synthesis, planning, or structured JSON output inside the memory code\.\- Initialization \(memory structure code — \*\*must\*\* match the benchmark execution model\):from meta\_self\_evolve\.llm\.client import MultimodalChatClientclient = MultimodalChatClient\(\)\- Do \*\*not\*\* pass ‘model=…‘ inside generated memory code\.\- Key Methods:\- await client\.complete\(messages: List\[Dict\[str, Any\]\]\) \-\> str:Send a standard chat message list and return plain text\.\- await client\.complete\_with\_system\(\*, system\_prompt: str, user\_input: str \| list, history: Optional\[List\[Dict\]\] = None, …\) \-\> str:Convenience wrapper for one system prompt \+ one user input \(\+ optional history\)\. \*\*All parameters after ‘client‘ are keyword\-only\*\* \(the real signature uses ‘\*‘ before ‘system\_prompt=‘\)\.\- await client\.complete\_json\_object\(messages: List\[Dict\[str, Any\]\], \*, temperature: float \| None = None\) \-\> Dict\[str, Any\]:Request a JSON object response\. Pass ‘messages‘ positionally; ‘temperature=‘ is keyword\-only if needed\.\- await client\.complete\_json\_with\_system\(\*, system\_prompt: str, user\_input: str \| list, history: Optional\[List\[Dict\]\] = None, temperature: float \| None = None\) \-\> Dict\[str, Any\]:Convenience wrapper for structured JSON output with a system prompt\. \*\*All parameters after ‘client‘ are keyword\-only\*\* \(the real signature uses ‘\*‘ before ‘system\_prompt=‘\)\.\- Usage Example:reply = await client\.complete\_with\_system\(system\_prompt="You summarize browser trajectories\.",user\_input="Summarize the key failure pattern in these steps\.",\)structured = await client\.complete\_json\_with\_system\(system\_prompt="Return ONLY a JSON object matching the schema described below\.",user\_input="…",\)\- IMPORTANT:\- \*\*Never\*\* call ‘complete\_with\_system\(…\)‘ or ‘complete\_json\_with\_system\(…\)‘ with positional arguments \(e\.g\. ‘complete\_json\_with\_system\(system\_prompt, user\_input\)‘ will raise ‘TypeError‘\)\. Always use ‘system\_prompt=…‘, ‘user\_input=…‘, and optionally ‘history=…‘, ‘temperature=…‘ as keywords\.\- If you need structured output, use ‘complete\_json\_object\(…\)‘ or ‘complete\_json\_with\_system\(…\)‘\.\- JSON output schemas should still be described clearly inside your system prompt\. Format example:\{"location": \{"type": "string","description": "The location to get the weather for"\},"unit": \{"type": \["string", "null"\],"description": "The unit to return the temperature in","enum": \["F", "C"\]\}\}\- Maintain history explicitly by passing prior messages in ‘history=‘ or by constructing the full ‘messages‘ list yourself\.2\. Embedding ClientClass: EmbeddingClient\- Purpose: DashScope \*\*qwen3\-vl\-embedding\*\* \(fixed model and fixed vector dimension inside the class\)\. Supports \*\*fused\*\* vectors for text\-only, image\(s\), or text\+image mixed inputs\. Optional cosine similarity helpers\.\- Initialization \(do \*\*not\*\* pass an embedding model name\):from meta\_self\_evolve\.llm\.embedding import EmbeddingClient, embed\_item\_text, embed\_item\_image\_path, embed\_item\_image\_urlembedder = EmbeddingClient\(retries=3, retry\_delay=1\.0\)\- Key Methods:\- await embedder\.get\_embedding\(text: str\) \-\> List\[float\]:Single \*\*text\*\* string → one fused vector \(text\-only path\)\.\- await embedder\.get\_batch\_embeddings\(texts: List\[str\]\) \-\> List\[List\[float\]\]:One fused vector per string \(independent calls\)\.\- await embedder\.get\_fused\_embedding\(items: List, base\_dir: Optional\[Path \| str\] = None\) \-\> List\[float\]:\*\*Multimodal fusion\*\*: ‘items‘ is an ordered list of parts \(from the helpers above\):\- ‘embed\_item\_text\("…"\)‘ → ‘\{type:’text’, text:…\}‘\- ‘embed\_item\_image\_path\("relative/or/abs/path\.png"\)‘ → local image; if relative, ‘EmbeddingClient‘ resolves it against ‘base\_dir=‘ when provided, otherwise against the worker’s current artifact\-root runtime context\.\- ‘embed\_item\_image\_url\("https://…"\)‘ → image URL\.One call uses ‘enable\_fusion=True‘ and returns \*\*one\*\* vector\. If you need \*\*multiple\*\* vectors \(e\.g\. separate vectors per document\), call ‘get\_fused\_embedding‘ \*\*multiple times\*\*—do not pack unrelated fusion targets into one call expecting multiple vectors\.\- await EmbeddingClient\.compute\_similarity\(emb1: List\[float\], emb2: List\[float\], metric: str = "cosine"\) \-\> float:Computes similarity between two embeddings asynchronously\.\- await EmbeddingClient\.compute\_one\_to\_group\_similarity\(emb: List\[float\], group\_emb: List\[List\[float\]\], metric: str = "cosine"\) \-\> List\[float\]:Computes similarity between one embedding and a group of embeddings asynchronously\.\- ‘EmbeddingClient\(\)‘ also implements ‘embed\_query\(\)‘ / ‘embed\_documents\(\)‘ / ‘\_\_call\_\_‘ so it can be passed into Chroma’s ‘embedding\_function=…‘ for \*\*text\*\* document/query paths\.\- Notes:\* Requires ‘DASHSCOPE\_API\_KEY‘ in the environment for DashScope\.\* For recorder screenshots in normal evaluation/runtime, relative image paths can usually be passed directly because the worker sets an artifact\-root runtime context\. If you use the embedder outside that runtime, pass ‘base\_dir=‘ explicitly\.\* Similarity helpers support cosine similarity and run in parallel for efficiency\.\* Do not import from ‘utils\.hire\_agent‘ or ‘evals\.utils\.hire\_agent‘; use ‘meta\_self\_evolve\.llm\.\*‘ only\.</OTHER\_TOOLS\>\#\#\# Output:Return \*\*only the final corrected Python code\*\*, no explanations or commentary\.\[USER\]Here’s the code with potential error:<CODE\_FOR\_MODIFY\>“‘python\{code\\\_str\}“‘</CODE\_FOR\_MODIFY\>And here’s corresponding error message:\{error\\\_msg\}Find the root cause first and then modify only the corresponding code to avoid the error\.

Prompt 5: WebVoyager Execution Prompt

WebVoyager Execution Prompt\[SYSTEM\]Imagine you are a robot browsing the web, just like humans\. Now you need to complete a task\. In each iteration, you will receive an Observation that includes a screenshot of a webpage and some texts\. This screenshot will feature Numerical Labels placed in the TOP LEFT corner of each Web Element\.Carefully analyze the visual information to identify the Numerical Label corresponding to the Web Element that requires interaction, then follow the guidelines and choose one of the following actions:1\. Click a Web Element\.2\. Delete existing content in a textbox and then type content\.3\. Scroll up or down\. Multiple scrolls are allowed to browse the webpage\. Pay attention\!\! The default scroll is the whole window\. If the scroll widget is located in a certain area of the webpage, then you have to specify a Web Element in that area\. I would hover the mouse there and then scroll\.4\. Wait\. Typically used to wait for unfinished webpage processes, with a duration of 5 seconds\.5\. Go back, returning to the previous webpage\.6\. Google, directly jump to the Google search page\. When you can’t find information in some websites, try starting over with Google\.7\. Answer\. This action should only be chosen when all questions in the task have been solved\.Correspondingly, Action should STRICTLY follow the format:\- Click \[Numerical\_Label\]\- Type \[Numerical\_Label\]; \[Content\]\- Scroll \[Numerical\_Label or WINDOW\]; \[up or down\]\- Wait\- GoBack\- Google\- ANSWER; \[content\]Key Guidelines You MUST follow:\* Action guidelines \*1\) To input text, NO need to click textbox first, directly type content\. After typing, the system automatically hits ‘ENTER‘ key\. Sometimes you should click the search button to apply search filters\. Try to use simple language when searching\.2\) You must Distinguish between textbox and search button, don’t type content into the button\! If no textbox is found, you may need to click the search button first before the textbox is displayed\.3\) Execute only one action per iteration\.4\) STRICTLY Avoid repeating the same action if the webpage remains unchanged\. You may have selected the wrong web element or numerical label\. Continuous use of the Wait is also NOT allowed\.5\) When a complex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions \(steps\)\. Flexibly combine your own abilities with the information in the web page\. Double check the formatting requirements in the task when ANSWER\.\* Web Browsing Guidelines \*1\) Don’t interact with useless web elements like Login, Sign\-in, donation that appear in Webpages\. Pay attention to Key Web Elements like search textbox and menu\.2\) Vsit video websites like YouTube is allowed BUT you can’t play videos\. Clicking to download PDF is allowed and will be analyzed by the Assistant API\.3\) Focus on the numerical labels in the TOP LEFT corner of each rectangle \(element\)\. Ensure you don’t mix them up with other numbers \(e\.g\. Calendar\) on the page\.4\) Focus on the date in task, you must look for results that match the date\. It may be necessary to find the correct year, month and day at calendar\.5\) Pay attention to the filter and sort functions on the page, which, combined with scroll, can help you solve conditions like ’highest’, ’cheapest’, ’lowest’, ’earliest’, etc\. Try your best to find the answer that best fits the task\.Your reply should strictly follow the format:Thought: \{Your brief thoughts \(briefly summarize the info that will help ANSWER\)\}Action: \{One Action format you choose\}Then the User will provide:Observation: \{A labeled screenshot Given by User\}\[USER\]Observation:\{labeled\\\_screenshot\\\_and\\\_text\}

Prompt 6: WebVoyager Auto Evaluation Prompt

WebVoyager Auto Evaluation Prompt\[SYSTEM\]As an evaluator, you will be presented with three primary components to assist you in your role:1\. Web Task Instruction: This is a clear and specific directive provided in natural language, detailing the online activity to be carried out\. These requirements may include conducting searches, verifying information, comparing prices, checking availability, or any other action relevant to the specified web service \(such as Amazon, Apple, ArXiv, BBC News, Booking etc\)\.2\. Result Screenshots: This is a visual representation of the screen showing the result or intermediate state of performing a web task\. It serves as visual proof of the actions taken in response to the instruction\.3\. Result Response: This is a textual response obtained after the execution of the web task\. It serves as textual result in response to the instruction\.– You DO NOT NEED to interact with web pages or perform actions such as booking flights or conducting searches on websites\.– You SHOULD NOT make assumptions based on information not presented in the screenshot when comparing it to the instructions\.– Your primary responsibility is to conduct a thorough assessment of the web task instruction against the outcome depicted in the screenshot and in the response, evaluating whether the actions taken align with the given instructions\.– NOTE that the instruction may involve more than one task, for example, locating the garage and summarizing the review\. Failing to complete either task, such as not providing a summary, should be considered unsuccessful\.– NOTE that the screenshot is authentic, but the response provided by LLM is generated at the end of web browsing, and there may be discrepancies between the text and the screenshots\.– Note the difference: 1\) Result response may contradict the screenshot, then the content of the screenshot prevails, 2\) The content in the Result response is not mentioned on the screenshot, choose to believe the content\.You should elaborate on how you arrived at your final evaluation and then provide a definitive verdict on whether the task has been successfully accomplished, either as ’SUCCESS’ or ’NOT SUCCESS’\.\[USER\]TASK: <task\>Result Response: <answer\><num\> screenshots at the end:\{trailing\\\_screenshot\\\_image\\\_parts\}Your verdict:

Prompt 7: AgentVista Execution Prompt

AgentVista Execution Prompt\[SYSTEM\]You are a visual reasoning agent for the AgentVista benchmark\.Each turn you must output:Thought: <your reasoning\>Action: <exactly one action\>Allowed actions \(JSON payloads must be valid JSON\):\- Action: WEB\_SEARCH\[\{"query": "…", "max\_results": 10\}\]\- Action: IMAGE\_SEARCH\[\{"query": "…", "max\_results": 10\}\]\- Action: VISIT\[\{"url": "https://…", "goal": "what to extract"\}\]\- Action: CODE\[\{"code": "…"\}\]\- Action: ANSWER\[your final answer\]When you are done, use Action: ANSWER\[…\]\. If the task expects a tagged final answer, include <answer\>…</answer\> inside the ANSWER payload\.Only one Action line per reply\.\[USER\]Solve the task using tools if needed\. Images are attached in order \(original\_image, original\_image\_1, …\)\.Question:\{question\\\_display\}

Prompt 8: AgentVista Evaluation Prompt

AgentVista Evaluation Prompt\[SYSTEM\]You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question\-answer pairs\. Compare the predicted answer with the correct answer and determine if they match meaningfully\. Consider synonyms or paraphrases as valid matches\.\[USER\]1\. \*\*Question\*\*:\{question\}2\. \*\*Ground Truth Answer\*\*:\{ground\\\_truth\}3\. \*\*Model Predicted Answer\*\*:\{prediction\}Evaluate the model’s prediction against the ground truth\. Output an integer score: 1 for correct, 0 for incorrect\.Respond using exactly: Score: 1 or Score: 0Explanation: <your explanation\>

Prompt 9: AgentVista Visit Tool Summary Prompt

AgentVista Visit Tool Summary Prompt\[SYSTEM\]You are a helpful assistant that summarizes webpage content based on user goals\.\[USER\]Please process the following webpage content and user goal to extract relevant information:\#\# \*\*Webpage Content\*\*\{content\}\#\# \*\*User Goal\*\*\{goal\}\#\# \*\*Task Guidelines\*\*1\. \*\*Content Scanning\*\*: Locate the \*\*specific sections/data\*\* directly related to the user’s goal within the webpage content2\. \*\*Key Extraction\*\*: Identify and extract the \*\*most relevant information\*\* from the content\. Never miss any important information\. Output the \*\*full original context\*\* as far as possible \(can be more than three paragraphs\)3\. \*\*Summary Output\*\*: Organize into a concise paragraph with logical flow, prioritizing clarity and judging the contribution of the information to the goal\#\# \*\*Output Format\*\*Please respond in JSON format with the following fields:\{"evidence": "Key quotes or facts from the page that are directly relevant to the goal","summary": "A concise summary of how the webpage content answers or relates to the user’s goal"\}

Prompt 10: Retrieved Memory Injection Prompt Fragment

Retrieved Memory Injection Prompt Fragment\[USER\]Retrieved memory \(from retrieve; use as guidance, not as task text\):\{retrieved\\\_memory\\\_payload\\\_json\}