基于动作条件的双模拟用于GUI智能体记忆管理

# ACTION-CONDITIONED BISIMULATION FOR GUI AGENT MEMORY
Source: [https://arxiv.org/html/2609.38778](https://arxiv.org/html/2609.38778)
Liuyang SongQuanquan LiDaqian YangYan WenZhengtao Yao††thanks:\*Equal contribution\. †Corresponding author:zyao9248@alumni\.usc\.edu\.

###### Abstract

An agent that remembers what it did on a web page must decide when two pages count as the same\. Memories built on observation similarity merge pages that look alike but behave differently, and GUIs are full of such pages: two tabs of one widget or two rows of one menu answer the same click differently\. We define the merge rule as an action\-conditioned bisimulation over the empirical predictive state graph a frozen agent fills as it acts\. Two states merge only when their shared actions lead to agreeing outcomes and successor blocks under an affordance label\. Observation similarity never enters the rule, and nothing is trained\. It replaces the merge rule of an existing outcome\-value memory, so a closed\-loop comparison isolates it\. On MiniWoB\+\+ it raises success rate over a memoryless agent, while a control taking identical exploratory detours, the prior successor\-representation merge, and the same criterion without action conditioning change nothing\.

###### Index Terms:

GUI agents, state abstraction, bisimulation, agent memory

††address:1Peking University2East China Normal University3University of Southern California## 1Introduction

A GUI agent that is not retrained between episodes can still improve, but only if it can recognise that it has been here before\. Every memory for such an agent therefore rests on one decision\. Given two observations, are they the same state? Get it wrong in one direction and each episode is a fresh graph, so nothing transfers; get it wrong in the other and experience from one page is recalled on a different page that merely resembles it\.

The prevailing answer is similarity\. Synapse\[[1](https://arxiv.org/html/2609.38778#bib.bib1)\]retrieves abstracted trajectories by embedding distance, on the benchmark used here; ExpeL\[[2](https://arxiv.org/html/2609.38778#bib.bib2)\]keeps successful trajectories beside insights distilled from them; Agent Workflow Memory\[[3](https://arxiv.org/html/2609.38778#bib.bib3)\]induces reusable sub\-routines; AutoGuide\[[4](https://arxiv.org/html/2609.38778#bib.bib4)\]recalls the guideline whose context matches the present state; A\-MEM\[[5](https://arxiv.org/html/2609.38778#bib.bib5)\]links notes into a network indexed by embedding; and successor\-representation memories\[[6](https://arxiv.org/html/2609.38778#bib.bib6)\]merge states whose discounted occupancy profiles agree\. They differ in what they store and agree on how they bring it back\. An item returns when the present observation resembles the one it was stored under\. That answers*which memory is relevant here*, not*is this the same state*\.

On a GUI the two questions come apart adversarially\. Two tabs of the same widget, two rows of the same menu, and a control before and after it has been enabled are near\-identical as observations and differ entirely in their response to the same click\. Merging them pools evidence across two different action semantics, and the agent then recalls a confident, wrong action\.

We replace the criterion with an action\-conditioned predictive state graph \(APSG, Fig\.[1](https://arxiv.org/html/2609.38778#S1.F1)\)\. A state is characterised by what it predicts under each action, the immediate outcome and the distribution over successor states, and two states merge only when those predictions agree for every action both have actually tried\. This is the bisimulation criterion\[[7](https://arxiv.org/html/2609.38778#bib.bib7),[8](https://arxiv.org/html/2609.38778#bib.bib8)\], which underpins representation learning for deep RL\[[9](https://arxiv.org/html/2609.38778#bib.bib9)\]and is still being revised\[[10](https://arxiv.org/html/2609.38778#bib.bib10),[11](https://arxiv.org/html/2609.38778#bib.bib11)\]\. That line learns an embedding offline, from reward, in continuous control\. We use the criterion as what it originally was, a partition, compute it by counting transitions a deployed agent has already made, and train nothing\.

Native GUI models such as UI\-TARS\[[12](https://arxiv.org/html/2609.38778#bib.bib12)\]improve the perception and grounding of the policy itself, and AgentOccam\[[13](https://arxiv.org/html/2609.38778#bib.bib13)\]shows on WebArena\[[14](https://arxiv.org/html/2609.38778#bib.bib14)\]that aligning the observation and action space with the policy outperforms added orchestration\. We take the same view from the memory side\.

![Refer to caption](https://arxiv.org/html/2609.38778v1/framework.png)Figure 1:The loop the criterion sits in\. A frozen policy acts on the GUI and every transition is recorded, so each state accumulates an empirical profile of what its tried actions led to\. States are compared only through those profiles, never through what the pages look like, and iterating the comparison to stability gives the partitionBB\. Everything to the right ofBBis inherited unchanged from the outcome\-value memory the criterion is dropped into\.
## 2Method

### 2\.1The empirical transition graph

A frozen agent interacting with a GUI produces transitions\(s,a,s′\)\(s,a,s^\{\\prime\}\), in whichssindexes an observation signatureσ⁡\(o\)\\sigma\(o\)andaaan action templateτ⁡\(a∣o\)\\tau\(a\\mid o\)\. A template replaces the transient parts of an action with descriptors that survive the episode, an element id by the element’s accessibility role and label and a goal\-supplied label by the slot it filled, so that clickingDonettain one episode andCheryin the next yields one template\. It resolves backwards against the current page, failing closed unless each descriptor matches one element\. The trajectorieso0,a0,o1,…o\_\{0\},a\_\{0\},o\_\{1\},\\dotsof every episode of a task then accumulate into the transition multiset

𝒟=\{\(σ⁡\(oi\),τ⁡\(ai∣oi\),σ⁡\(oi\+1\)\)\}i,\\mathcal\{D\}=\\bigl\\\{\\bigl\(\\sigma\(o\_\{i\}\),\\ \\tau\(a\_\{i\}\\mid o\_\{i\}\),\\ \\sigma\(o\_\{i\+1\}\)\\bigr\)\\bigr\\\}\_\{i\},\(1\)whose distinct first coordinates are the statesSS\. Withn⁡\(s,a,s′\)n\(s,a,s^\{\\prime\}\)counting the copies of\(s,a,s′\)\(s,a,s^\{\\prime\}\),n⁡\(s,a\)=∑s′n⁡\(s,a,s′\)n\(s,a\)=\\sum\_\{s^\{\\prime\}\}n\(s,a,s^\{\\prime\}\),A⁡\(s\)=\{a:n⁡\(s,a\)\>0\}A\(s\)=\\\{a:n\(s,a\)\>0\\\}the actions ever tried atss, andS±S^\{\\pm\}the states at which an episode terminated in success or failure, the entire description of a state used here is its empirical prediction under each action tried there,

P⁡\(s′∣s,a\)=n⁡\(s,a,s′\)n⁡\(s,a\),ρ⋆​\(s,a\)=∑s′∈S⋆​\(s\)P⁡\(s′∣s,a\),P\(s^\{\\prime\}\\mid s,a\)=\\frac\{n\(s,a,s^\{\\prime\}\)\}\{n\(s,a\)\},\\qquad\\rho^\{\\star\}\(s,a\)=\\\!\\\!\\sum\_\{s^\{\\prime\}\\in S^\{\\star\}\(s\)\}\\\!\\\!P\(s^\{\\prime\}\\mid s,a\),\(2\)withS±​\(s\)=S±S^\{\\pm\}\(s\)=S^\{\\pm\}and the progress successorsS△​\(s\)S^\{\\triangle\}\(s\)defined next\.

Progress needs care, since tabs and menus repaint a page in both directions and readingρ△\\rho^\{\\triangle\}as “the page changed” would credit oscillation\. Writingu↝vu\\rightsquigarrow vfor reachability over the edges withn⁡\(u,a,v\)\>0n\(u,a,v\)\>0, we grade a transition byc⁡\(s,s′\)c\(s,s^\{\\prime\}\), which is−1\-1on a self\-loop,00whens′↝ss^\{\\prime\}\\rightsquigarrow sand11otherwise\. Progress is then only what has no recorded way back,S△​\(s\)=\{s′:c⁡\(s,s′\)=1\}S^\{\\triangle\}\(s\)=\\\{s^\{\\prime\}:c\(s,s^\{\\prime\}\)=1\\\}\.

### 2\.2Action\-conditioned bisimulation distance

LetBBbe a partition of the states,B⁡\(s′\)B\(s^\{\\prime\}\)the block ofs′s^\{\\prime\}, andPBP\_\{B\}the pushforward ofPPonto blocks\. For an actionaatried at bothssandtt,

da​\(s,t\)=13​∑⋆∈\{\+,−,△\}\|ρ⋆​\(s,a\)−ρ⋆​\(t,a\)\|⏟immediate outcome\+γ​TV\(PB\(⋅∣s,a\),PB\(⋅∣t,a\)\)⏟successor blocks,\\begin\{split\}d\_\{a\}\(s,t\)=\\ &\\underbrace\{\\tfrac\{1\}\{3\}\\\!\\\!\\sum\_\{\\star\\in\\\{\+,\-,\\triangle\\\}\}\\\!\\\!\\bigl\|\\rho^\{\\star\}\(s,a\)\-\\rho^\{\\star\}\(t,a\)\\bigr\|\}\_\{\\text\{immediate outcome\}\}\\\\\[2\.0pt\] &\+\\ \\gamma\\,\\underbrace\{\\mathrm\{TV\}\\bigl\(P\_\{B\}\(\\cdot\\mid s,a\),P\_\{B\}\(\\cdot\\mid t,a\)\\bigr\)\}\_\{\\text\{successor blocks\}\},\\end\{split\}\(3\)withTV\\mathrm\{TV\}the total variation distance\. The state distance is the worst case over the actions that both states have tried, and is infinite unless they carry the same labelℓ\\ell,

d⁡\(s,t\)=\{∞if​ℓ​\(s\)≠ℓ⁡\(t\)or​A​\(s\)∩A⁡\(t\)=∅,maxa∈A⁡\(s\)∩A⁡\(t\)⁡da​\(s,t\)otherwise\.d\(s,t\)=\\begin\{cases\}\\infty&\\text\{if \}\\ell\(s\)\\neq\\ell\(t\)\\\\ &\\text\{or \}A\(s\)\\cap A\(t\)=\\emptyset,\\\\\[2\.0pt\] \\displaystyle\\max\_\{a\\in A\(s\)\\cap A\(t\)\}d\_\{a\}\(s,t\)&\\text\{otherwise\.\}\\end\{cases\}\(4\)The restriction to shared actions is the substantive modelling choice\. An action observed atssalone carries no evidence abouttt, and treating its absence as agreement would merge on missing data, which is how a similarity rule fails\. States with no evidence in common are never merged, however alike they look, and the trio of Fig\.[2](https://arxiv.org/html/2609.38778#S2.F2)shows the criterion separating look\-alikes that do share actions\. It also makes the action vocabulary load\-bearing\. A state carries only the few actions tried there, often one, so under actions named by the element id clicked every cross\-episode pair is at infinite distance and the abstraction degrades to the identity, which the templates of Sec\.[2\.1](https://arxiv.org/html/2609.38778#S2.SS1)prevent\.

The label is what bisimulation on a labelled system requires before it compares transitions at all\[[7](https://arxiv.org/html/2609.38778#bib.bib7)\], and the action\-conditioned term refines it rather than replacing it\. We takeℓ⁡\(s\)\\ell\(s\)to be the invariant affordance signature of the page,

ℓ⁡\(s\)=\(g^,\{\{role⁡\(e\):e∈E⁡\(s\)\}\},π⁡\(s\)\),\\ell\(s\)=\\bigl\(\\hat\{g\},\\ \\\{\\\!\\\{\\mathrm\{role\}\(e\):e\\in E\(s\)\\\}\\\!\\\},\\ \\pi\(s\)\\bigr\),\(5\)in whichg^\\hat\{g\}is the goal with each named target replaced by its slot position,E⁡\(s\)E\(s\)the elements of the page, andπ⁡\(s\)\\pi\(s\)the goal slots fillable on it, all properties of the task and not of the episode\. It tests what a page offers rather than what it displays, and is inherited from the memory we build on \(Sec\.[2\.4](https://arxiv.org/html/2609.38778#S2.SS4)\)\. Dropping it is not harmless\. A closed menu and the menu it opens into afford the same toggle and are each other’s successor under it, so both terms of \([3](https://arxiv.org/html/2609.38778#S2.E3)\) agree once the pair is merged, and the refinement settles on a coarse fixed point in which the pooled value of the toggle outranks the action that finishes the task\.

### 2\.3The partition is a fixed point

Equation \([3](https://arxiv.org/html/2609.38778#S2.E3)\) compares distributions over blocks ofBB, soBBappears on both sides of its own definition and we compute it as a fixed point\. WritingdBd^\{B\}for \([4](https://arxiv.org/html/2609.38778#S2.E4)\) evaluated with the pushforward ontoBBandUF⁡\(B,𝒫\)\\mathrm\{UF\}\(B,\\mathcal\{P\}\)for the union–find closure ofBBunder the pairs𝒫\\mathcal\{P\}in increasing order ofdBd^\{B\}, we start from the immediate\-outcome partitionB0B\_\{0\}, which is \([4](https://arxiv.org/html/2609.38778#S2.E4)\) atγ=0\\gamma\{=\}0, and iterate

Bk=UF⁡\(Bk−1,\{\(s,t\):dBk−1​\(s,t\)≤τb\}\)B\_\{k\}=\\mathrm\{UF\}\\bigl\(B\_\{k\-1\},\\ \\\{\(s,t\)\\ :\\ d^\{B\_\{k\-1\}\}\(s,t\)\\leq\\tau\_\{b\}\\\}\\bigr\)\(6\)untilBk=Bk−1B\_\{k\}=B\_\{k\-1\}or a budgetKKis spent; since \([6](https://arxiv.org/html/2609.38778#S2.E6)\) only coarsens, the iteration is monotone and settles after a few rounds in practice\. This is the usual bisimulation\-metric construction\[[8](https://arxiv.org/html/2609.38778#bib.bib8),[15](https://arxiv.org/html/2609.38778#bib.bib15)\], run on empirical rather than known transitions\. The graph of Sec\.[2\.1](https://arxiv.org/html/2609.38778#S2.SS1)carrying this partition is the APSG\.

A state withA⁡\(s\)=∅A\(s\)=\\emptysetsupports no comparison under \([4](https://arxiv.org/html/2609.38778#S2.E4)\), which is no corner case, since it covers every terminal and the state currently being decided\. Left a singleton it returns nothing from \([7](https://arxiv.org/html/2609.38778#S2.E7)\), so we place it instead by its incoming signature, which is action\-conditioned in the same sense as \([4](https://arxiv.org/html/2609.38778#S2.E4)\)\. Given a recorded\(u,a,s\)\(u,a,s\), it joins the block that transitions leavingB⁡\(u\)B\(u\)underaamost often landed in; an episode’s first page joins the block its counterparts in past episodes occupy, and only a terminal falls back to its role as success, failure, or neither\. Each placement obeys the label requirement\.

![Refer to caption](https://arxiv.org/html/2609.38778v1/figs/cell-tab1.png)

\(a\) Tab \#1 click\(18\)

![Refer to caption](https://arxiv.org/html/2609.38778v1/figs/cell-tab2.png)

\(b\) Tab \#2 click\(20\)

![Refer to caption](https://arxiv.org/html/2609.38778v1/figs/cell-tab3.png)

\(c\) Tab \#3 click\(22\)

Figure 2:One evaluation cell ofclick\-tab\-2\(goal “Switch between the tabs to find and click on the linkgravida\.”\)\. The panels are the three states reachable before the target is visible; only \(c\) holds the link\. They differ by a highlight and a paragraph of filler, so an observation\-similarity criterion merges them, while their responses to the same three clicks differ and \([4](https://arxiv.org/html/2609.38778#S2.E4)\) separates them\. Below, the element clicked at each step\. ReAct alternates Tab \#2 and Tab \#1 for all eight steps\. APSG emits the same five actions, then overrides at step66\(bold\), once the block holds enough recorded failures ofclick\(18\)for \([8](https://arxiv.org/html/2609.38778#S2.E8)\) to preferclick\(22\)\.
### 2\.4What is deliberately inherited

The abstraction is the whole method, so everything downstream is held fixed\. We implement the rule as a subclass of an existing outcome\-value memory that overrides the partition and nothing else\. That memory takesV=\(1−γ\)​\(I−γ​P𝒟\)−1​rV=\(1\-\\gamma\)\(I\-\\gamma P\_\{\\mathcal\{D\}\}\)^\{\-1\}rwithr=𝟏S\+−𝟏S−r=\\mathbf\{1\}\_\{S^\{\+\}\}\-\\mathbf\{1\}\_\{S^\{\-\}\}, the normalised discounted terminal occupancy of the recorded process, and scores a transition by the empirical advantageδ⁡\(u,v\)=g⁡\(v\)−V⁡\(u\)\\delta\(u,v\)=g\(v\)\-V\(u\), whose continuationg⁡\(v\)g\(v\)is±1\\pm 1at a terminal inS±S^\{\\pm\}andγ​V​\(v\)\\gamma V\(v\)otherwise\. The abstraction enters only now\. The evidence for a templateaaat the current statessis pooled over the whole blockB⁡\(s\)B\(s\),

qB\(a\)=∑\(u,a,v\):B⁡\(u\)=B⁡\(s\)n\(u,a,v\)\[δ\(u,v\)\+ηc¯\(u,v\)\]κ\+∑\(u,a,v\):B⁡\(u\)=B⁡\(s\)n\(u,a,v\),q\_\{B\}\(a\)=\\frac\{\\displaystyle\\sum\_\{\(u,a,v\):\\,B\(u\)=B\(s\)\}\\\!\\\!\\\!n\(u,a,v\)\\,\\bigl\[\\delta\(u,v\)\+\\eta\\,\\bar\{c\}\(u,v\)\\bigr\]\}\{\\displaystyle\\kappa\+\\\!\\\!\\\!\\sum\_\{\(u,a,v\):\\,B\(u\)=B\(s\)\}\\\!\\\!\\\!n\(u,a,v\)\},\(7\)withη​c¯\\eta\\,\\bar\{c\}a small dense term \(c¯=c\\bar\{c\}=c, but−1\-1at a recorded failure\) andκ\\kappaa prior against one transition dominating\. Given the policy’s own shortlista0,…,am−1a\_\{0\},\\dots,a\_\{m\-1\}in its own order, the memory returns

a⋆=arg⁡max0≤j<m​qB​\(aj\)−λ​j,a^\{\\star\}=\\arg\\max\_\{0\\leq j<m\}\\ q\_\{B\}\(a\_\{j\}\)\-\\lambda j,\(8\)withqB​\(aj\)=0q\_\{B\}\(a\_\{j\}\)=0where no evidence resolves ontoaja\_\{j\}, so that the rank penaltyλ\\lambdamakes an override require the recalled evidence to outweigh the policy’s own ordering anda0a\_\{0\}survives whenever the memory is silent\.

All of this is inherited verbatim, including the label of \([5](https://arxiv.org/html/2609.38778#S2.E5)\), which that memory already gates its own merges on, so that the change of criterion is not confounded with a relaxation of the label\. The three merge rules therefore differ in the single symbolBBof \([7](https://arxiv.org/html/2609.38778#S2.E7)\), and nothing else is added, no router, no certificate, no stagnation heuristic, and no branch on the task identity\.

### 2\.5MakingBBobservable

Two properties of a deployed greedy agent would make any comparison of merge rules vacuous\. First, \([8](https://arxiv.org/html/2609.38778#S2.E8)\) needs candidates, which sampling cannot supply once thinking is disabled, so we keep the policy’s greedy action as the heada0a\_\{0\}and append distinct alternatives from a second greedy call; a memory that never overrides then reproduces the baseline exactly\. Second, greedy execution leaves nothing to pool, since the executed action is a function of the page, so\|A⁡\(s\)\|=1\|A\(s\)\|=1however oftenssis visited andqBq\_\{B\}can only re\-propose the one action it ever saw, whateverBBis\. The reranking systems therefore take anϵ\\epsilon\-greedy detour offa0a\_\{0\}over the first third of each run, seeded by the environment seed and step index and never by the system, so a memoryless control makes the identical detours and only what is kept of them differs\.

## 3Experiments

Table 1:Closed\-loop MiniWoB\+\+ over 10 tasks with a frozen Qwen3\-8B, single seed\. SR is the benchmark’s own success rate on the evaluation half fixed before the run \(400 paired cells per system\), decoded greedily by every system\. Environment seeds are identical across systems, soΔ\\Deltaagainst ReAct is paired cell by cell, with a 95% percentile bootstrap interval and an exact McNemarpp\.SystemSRΔ\\Deltavs\. ReAct95% CIpp*Memoryless*ReAct0\.723–––Shortlist control0\.723\+0\.000\+0\.000\[\+0\.000,\+0\.000\]\[\+0\.000,\+0\.000\]1\.000*Memory, prior merge rules*SR merge0\.728\+0\.005\+0\.005\[−0\.007,\+0\.018\]\[\-0\.007,\+0\.018\]0\.727Observation merge0\.708−0\.015\-0\.015\[−0\.033,\+0\.003\]\[\-0\.033,\+0\.003\]0\.180*Memory, ours*APSG \(ours\)0\.782\+0\.060\+0\.060\[\+0\.033,\+0\.090\]\[\+0\.033,\+0\.090\]<<0\.001Figure 3:Both panels recomputed from the episode log\.*\(a\)*Cumulative success rate, with a 95% bootstrap band on our curve only\. The shortlist control takes the identical detours but keeps nothing, so it recovers to ReAct’s rate and stops there; under the action\-conditioned partition the same detours are paid off by episode3535\.*\(b\)*On the evaluation episodes:*evidence coverage*, the share of decisions whose block held recorded evidence for a candidate;*override rate*, the share at which \([8](https://arxiv.org/html/2609.38778#S2.E8)\) displaced the policy’s first choice;*override precision*, the share of displaced outcomes that moved from failure to success rather than the reverse\.We evaluate closed\-loop on MiniWoB\+\+\[[16](https://arxiv.org/html/2609.38778#bib.bib16)\]through BrowserGym\[[17](https://arxiv.org/html/2609.38778#bib.bib17)\], on ten navigation\-style tasks fixed beforehand in which the same page layouts recur across episodes while each episode draws a different environment instance, so cross\-episode state identity decides the outcome\. The policy is Qwen3\-8B\[[18](https://arxiv.org/html/2609.38778#bib.bib18)\]served locally, frozen, thinking disabled, and capped at eight steps\. Each task runs sixty episodes; episodes11–2020take the detours of Sec\.[2\.5](https://arxiv.org/html/2609.38778#S2.SS5)and episodes2121–6060are greedy for every system\. One setting serves every task and system \(ϵ=0\.35\\epsilon=0\.35,τb=0\.30\\tau\_\{b\}=0\.30,γ=0\.8\\gamma=0\.8,K=6K=6, and the inheritedλ=0\.015\\lambda=0\.015,κ=2\\kappa=2,η=0\.10\\eta=0\.10\)\. A hand\-written shortcut sits in front of the policy in all five systems, acting without a model call whenever exactly one visible element carries the label the goal names; it fired171171times for ReAct\[[19](https://arxiv.org/html/2609.38778#bib.bib19)\]and170170times for each of the others, so it cannot separate them\.

The ablation ladder\.Table[1](https://arxiv.org/html/2609.38778#S3.T1)is read from the bottom up\. Each rung holds the policy, the prompt, the model\-call budget and the exploration draws fixed and changes exactly one thing, and three of the four rungs fail to move\. The shortlist control pays the memories’ second model call and takes every one of their detours, yet ends the evaluation episodes without a single discordant pair against ReAct, so neither the extra call nor the exploration finds better actions by itself\. Keeping that same evidence under the SR merge is indistinguishable from keeping none of it, and the observation merge is if anything worse than not merging at all\. Only the last rung moves, by\+0\.060\+0\.060against ReAct \(p<10−3p<10^\{\-3\}\) and by\+0\.075\+0\.075against the observation merge it replaces\. What separates the rules is which states they merge, not how many\. The observation merge is the more cautious, collapsing the graph roughly twofold against our roughly fivefold, and it is the one that loses, since pooling evidence helps only when the states pooled answer the same action the same way\. Fig\.[3](https://arxiv.org/html/2609.38778#S3.F3)\(b\) locates the difference at the moment of decision\. Neither rule is silent, so the criterion does not make the memory quieter; it makes it right more often when it does speak\. Ours moves most of the outcomes it touches from failure to success and the observation merge moves most of them the other way, which is the aliasing of Fig\.[2](https://arxiv.org/html/2609.38778#S2.F2)showing up at the decision itself\. A confident wrong recall costs more than no recall, and the shared\-action requirement of \([4](https://arxiv.org/html/2609.38778#S2.E4)\) is the clause that refuses the pair that produces one\.

Where the gain falls and what it costs\.Before the run we split the suite by a stated property, not by outcome, and recorded that the gain should concentrate on one side and be absent on the other\. A container task is one whose goal names one of several interchangeable containers \(tabs, menus, disclosure triangles\), so that pages differing in identity coincide in behaviour\. Both halves of that prediction hold\. On the container tasks the difference against ReAct is\+0\.075\+0\.075, and on the others there is no discordant pair for any memory system on any cell: where there is nothing to alias, \([8](https://arxiv.org/html/2609.38778#S2.E8)\) returns the policy’s own action because no block has accumulated evidence against it, so those tasks are a negative control the mechanism passes, not a subset it is weak on\. Inside the container set the movement concentrates on the tab and collapsible tasks, where near\-identical pages differ only in what they open, while the tasks already at ceiling show the memory doing no harm\. One container task,click\-menu\-2, moves against us; its cause remains unresolved\. The gain is bought with exploration, and Fig\.[3](https://arxiv.org/html/2609.38778#S3.F3)\(a\) shows the cost repaid\. The memory starts behind, having spent a third of the run deliberately off\-policy, crosses ReAct within the run, and ends ahead of it over every episode, not only the evaluation ones, while the control pays the identical cost and recovers to ReAct’s rate and no further\. On the272272cells every system solves, where only directness can differ, it reaches the goal in fewer steps than any other system \(1\.701\.70against ReAct’s1\.811\.81\), because a filled block records which of the look\-alike pages is worth opening\.

What a merge rule trades, and why this one bounds it\.Everything the memory can do passes through \([7](https://arxiv.org/html/2609.38778#S2.E7)\), a mean of empirical advantages taken over a block, so a merge rule has one way to help and one way to hurt\. It helps by count\. A single page is visited a handful of times,κ\\kappadominates its denominator, and an unmerged memory is therefore silent rather than wrong; a block puts enough recorded transitions behind a template for the numerator to outweigh the prior, which is the coverage every rule in Fig\.[3](https://arxiv.org/html/2609.38778#S3.F3)\(b\) buys\. It hurts by bias\. The pooled mean stands in for the advantage ofaaatssonly insofar as the states merged withssansweraaasssdoes, and any disagreement left among them entersqB​\(a\)q\_\{B\}\(a\)carrying the full weight of their counts, with nothing downstream able to detect it\. The two move together under any rule, and the criterion is the only place the second is bounded\. Equation \([3](https://arxiv.org/html/2609.38778#S2.E3)\) is built from the two ways the summand of \([7](https://arxiv.org/html/2609.38778#S2.E7)\) can move betweenssandttunder one action\. Its advantageδ⁡\(u,v\)=g⁡\(v\)−V⁡\(u\)\\delta\(u,v\)=g\(v\)\-V\(u\)separates into the outcome recorded at the transition, which the first term compares directly, and a continuationγ​V​\(v\)\\gamma V\(v\), which depends onvvthrough its block alone exactly whenBBis a bisimulation of the recorded process, and which the second term therefore compares as a distribution over blocks\. Thresholdingda≤τbd\_\{a\}\\leq\\tau\_\{b\}caps the disagreement a merge admits, the maximum in \([4](https://arxiv.org/html/2609.38778#S2.E4)\) makes the cap a worst case over the shared actions and not an average, and refusingA⁡\(s\)∩A⁡\(t\)=∅A\(s\)\\cap A\(t\)=\\emptysetkeeps it from being a maximum over an empty set\. The cap is stated on the partition it is used to build, which is why \([6](https://arxiv.org/html/2609.38778#S2.E6)\) iterates: each round re\-evaluates every admitted merge against the coarser blocks its own predecessors created, so the property the bound assumes is the property the fixed point establishes\. Merging on appearance buys the same counts with no such object to iterate towards, and the bias it admits is invisible to the rule that admitted it\.

## References

- \[1\]L\. Zheng, R\. Wang, X\. Wang, and B\. An,*Synapse: Trajectory\-as\-Exemplar Prompting with Memory for Computer Control*, ICLR, 2024\.
- \[2\]A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\.\-J\. Liu, and G\. Huang,*ExpeL: LLM Agents Are Experiential Learners*, AAAI, 2024\.
- \[3\]Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig,*Agent Workflow Memory*, ICML, 2025\.
- \[4\]Y\. Fu, D\.\-K\. Kim, J\. Kim, S\. Sohn, L\. Logeswaran, K\. Bae, and H\. Lee,*AutoGuide: Automated Generation and Selection of Context\-Aware Guidelines for Large Language Model Agents*, NeurIPS, 2024\.
- \[5\]W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang,*A\-MEM: Agentic Memory for LLM Agents*, arXiv:2502\.12110, 2025\.
- \[6\]P\. Dayan,*Improving generalisation for temporal difference learning: The successor representation*, Neural Computation, vol\. 5, no\. 4, pp\. 613–624, 1993\.
- \[7\]R\. Givan, T\. Dean, and M\. Greig,*Equivalence notions and model minimization in Markov decision processes*, Artificial Intelligence, vol\. 147, nos\. 1–2, pp\. 163–223, 2003\.
- \[8\]N\. Ferns, P\. Panangaden, and D\. Precup,*Metrics for Finite Markov Decision Processes*, UAI, pp\. 162–169, 2004\.
- \[9\]A\. Zhang, R\. McAllister, R\. Calandra, Y\. Gal, and S\. Levine,*Learning Invariant Representations for Reinforcement Learning without Reconstruction*, ICLR, 2021\.
- \[10\]L\. Zhang, Z\. Wang, X\. Li, and Y\.\-H\. Li,*Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning*, arXiv:2507\.18519, 2025\.
- \[11\]Z\. Luo, T\. Ni, P\.\-L\. Bacon, D\. Precup, and X\. Si,*Understanding Behavioral Metric Learning: A Large\-Scale Study on Distracting Reinforcement Learning Environments*, arXiv:2506\.00563, 2025\.
- \[12\]Y\. Qin et al\.,*UI\-TARS: Pioneering Automated GUI Interaction with Native Agents*, arXiv:2501\.12326, 2025\.
- \[13\]K\. Yang, Y\. Liu, S\. Chaudhary, R\. Fakoor, P\. Chaudhari, G\. Karypis, and H\. Rangwala,*AgentOccam: A Simple Yet Strong Baseline for LLM\-Based Web Agents*, ICLR, 2025\.
- \[14\]S\. Zhou et al\.,*WebArena: A Realistic Web Environment for Building Autonomous Agents*, ICLR, 2024\.
- \[15\]P\. S\. Castro,*Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes*, AAAI, pp\. 10069–10076, 2020\.
- \[16\]E\. Z\. Liu, K\. Guu, P\. Pasupat, T\. Shi, and P\. Liang,*Reinforcement Learning on Web Interfaces Using Workflow\-Guided Exploration*, ICLR, 2018\.
- \[17\]T\. Le Sellier de Chezelles et al\.,*The BrowserGym Ecosystem for Web Agent Research*, TMLR, 2025\.
- \[18\]A\. Yang et al\.,*Qwen3 Technical Report*, arXiv:2505\.09388, 2025\.
- \[19\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao,*ReAct: Synergizing reasoning and acting in language models*, ICLR, 2023\.