Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents

# Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents
Source: [https://arxiv.org/html/2606.15017](https://arxiv.org/html/2606.15017)
1\]ServiceNow AI Research 2\]ÉTS Montreal\\affiliationbreak3\]University of British Columbia 4\]McGill University\\contribution\[\*\]Equal advising

Masih AminbeidokhtiJose DolzIsmail Ben Ayed Issam H\. LaradjiSpandana GellaNicolas Gontier\[\[\[\[

###### Abstract

Online web agents often augment a base actor with memory, workflow, or skill modules\. These modules can improve performance, but they also consume test\-time tokens, a cost rarely reported alongside the actor’s inference cost\. We study online augmentation, where this overhead is paid on every task, and re\-evaluate its benefits under a fixed total inference budget\. We compare AWM, ASI, and ReasoningBank with a token\-matched vanilla baseline that uses the same budget for additional actor steps\. Across three WebArena domains and three models, Gemini 3 Flash, GPT\-5\.4\-mini, and Qwen 3\.6\-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens\. We observe a similar trend on WorkArena\-L1 with Qwen 3\.6\-27B, indicating that the effect extends to enterprise knowledge\-work tasks\. Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget\-matched actor\. We further show that run\-to\-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents\.

## 1Introduction

Controlled total inference budget per taskOnline augmentationAWM / ASI / ReasoningBankBudget\-matched actorvanilla actor \+ more explorationactormodules10 actor steps \+ module callsactor15 actor stepsCost paid online:fewer tokens left for explorationBudget redirected:more interactionwith the environment

![Refer to caption](https://arxiv.org/html/2606.15017v1/x1.png)

Figure 1:Budget allocation under controlled test\-time inference cost\.\(top\) Online augmentation spends budget on workflow, skill, or memory modules, while a budget\-matched vanilla actor \(Vanilla\-IB\) spends it on additional observe\-act steps\. \(bottom\) On WebArena with Gemini 3 Flash\(gemini\-team\-2025\-gemini3\),Vanilla\-IBachieves the highest success rate with fewer tokens\.Web agents are evaluated on how often they solve tasks under a test\-time budget\. A growing line of work augments a base actor with modules that induce reusable workflows, maintain memory, or distill skills from prior trajectories\(wang\-etal\-2025\-asi;wang\-etal\-2024\-agent;ouyang\-etal\-2025\-reasoningbank;li\-etal\-2025\-polyskill\)\. Following Agent Skill Induction\(wang\-etal\-2025\-asi\)and SkillWeaver\(zheng\-etal\-2025\-skillweaver\), we use the term*skill*for executable code, typically a Python function, that wraps a reusable sequence of low\-level actions\.

These auxiliary modules are not free\. They consume tokens, add prompts and control logic, and in the online setting, couple performance to the order in which tasks are processed\. Their token cost is rarely reported next to the actor’s, which makes it hard to tell whether observed gains are justified within a fixed budget\. This creates a simple allocation problem: tokens spent inducing, retrieving, verifying, or injecting reusable knowledge are tokens unavailable for direct task execution, such as observing the current page, reasoning over its state, or taking additional actions\.

Figure[1](https://arxiv.org/html/2606.15017#S1.F1)illustrates this tradeoff\. Under the same total inference budget, an online augmented agent splits its budget between direct environment interaction and auxiliary memory, workflow, or skill machinery\. A budget\-matched vanilla actor instead redirects that budget toward additional observe\-act steps\. The bottom panel previews the main empirical pattern we observe on WebArena\(zhou\-etal\-2024\-webarena\): once total token usage is counted, a budget\-matched vanilla agent achieves the highest average success rate while using fewer tokens than the online augmentation methods\.

Prior work on augmentation methods largely assumed that base models lacked the capability to complete tasks end\-to\-end\(wang\-etal\-2024\-agent;wang\-etal\-2025\-asi\); more recent frontier models, however, may no longer face that constraint and may require less external scaffolding\. We therefore ask a direct question: under a fixed inference budget, is it better to allocate tokens to online memory and skill modules, or can a vanilla actor achieve comparable performance by spending the same budget on additional interaction steps? We focus on the*online*setting, where agents update their behavior using trajectories collected during evaluation, rather than relying on offline skill libraries constructed before deployment\. In this setting, auxiliary costs are incurred repeatedly, the knowledge accumulated by memory or skill modules depends on the order in which tasks are encountered rather than the full task distribution, and task\-level parallelism is constrained by the sequential dependence between tasks\.

To study this tradeoff, we compare three online augmentation methods, Agent Workflow Memory \(AWM;wang\-etal\-2024\-agent\), Agent Skill Induction \(ASI;wang\-etal\-2025\-asi\), and ReasoningBank \(ouyang\-etal\-2025\-reasoningbank\), against a budget\-matched vanilla actor\. Unlike the augmented methods, which use the default 10\-step actor horizon, this baseline allocates the same inference budget to a longer interaction horizon; we refer to it asVanilla\-Increased\-Budget \(Vanilla\-IB\)\. We do not presentVanilla\-IBas a new agent scaffold, but rather as a simple budget\-aware control designed to test whether the overhead of online augmentation is justified\.

We evaluate this comparison on WebArena\(zhou\-etal\-2024\-webarena\)and WorkArena\-L1\(drouin\-etal\-2024\-workarena\)\. WebArena provides the broadest sweep in our study, spanning three models, four agent configurations, and three domains: Shopping, Reddit, and Admin\. WorkArena\-L1 tests the same budget\-matching question in enterprise knowledge\-work tasks under Qwen 3\.6\-27B\.

Our main findings are as follows: \(1\) On WebArena,Vanilla\-IBachieves the highest aggregate success rate across all three models we test, while mostly using fewer total tokens than AWM, ASI, and ReasoningBank\. \(2\) On WorkArena\-L1,Vanilla\-IBremains competitive with the best\-performing augmented methods, indicating that the budget\-matching question is not specific to WebArena or consumer\-style websites\. \(3\) Beyond these aggregate results, we show that single\-run success rate hides high task\-level variance, and that reporting a method’s performance from a single run is a weak basis for comparison\.

These findings suggest three evaluation principles for online web agents\. First, comparisons should report*total token usage*across all modules, not only the actor\. Second, online augmentation methods should be compared against*baselines within the same budget*\. Third, evaluations should report*multi\-run variance*, since agent evaluation introduces stochasticity that aggregate single\-run metrics do not capture\.

## 2Related Work

#### Web\-agent benchmarks\.

Mind2Web\(deng\-etal\-2024\-mind2web\)and WebArena\(zhou\-etal\-2024\-webarena\)introduced large\-scale benchmarks for realistic multi\-step browser interaction\. Subsequent work has broadened this benchmark landscape to cover visual grounding in VisualWebArena\(koh\-etal\-2024\-visualwebarena\), enterprise workflows in WorkArena\(drouin\-etal\-2024\-workarena\)and WorkArena\+\+\(boisvert\-etal\-2024\-workarenaplusplus\), live websites in Online\-Mind2Web\(xue\-etal\-2025\-illusion\), and continuously generated open\-ended tasks in WebArena\-Infinity\(webarena\-infinity\)\. In this work, we focus on WebArena and WorkArena because they provide interactive environments in which agents must act, observe the consequences of their actions, and complete diverse tasks end\-to\-end\. This makes them well suited for evaluating test\-time compute mechanisms, where the cost of additional reasoning steps, tool calls, or memory access must be measured as part of the agent’s deployment\-time behavior\.

#### Online workflow, memory, and skill induction\.

A growing line of work augments base actors with modules that extract reusable knowledge from evaluation\-time trajectories\. AWM\(wang\-etal\-2024\-agent\)induces natural\-language workflows from successful trajectories and retrieves them for later tasks\. ASI\(wang\-etal\-2025\-asi\)synthesizes reusable Python skill functions called as higher\-level operations for subsequent tasks\. ReasoningBank\(ouyang\-etal\-2025\-reasoningbank\)distills reasoning strategies from both successes and failures into a growing memory bank queried at test time\. PolySkill\(li\-etal\-2025\-polyskill\)separates a skill’s abstract goal from its concrete realization for more general reuse, and PlugMem\(yang\-etal\-2026\-plugmem\)proposes a task\-agnostic memory plugin attachable to any LLM agent\. ACE\(zhang\-etal\-2025\-ace\)treats the agent’s context itself as an evolving playbook that accumulates and refines strategies through a generate\-reflect\-curate loop\. ERL\(allard\-etal\-2026\-erl\)reflects on task trajectories to extract transferable heuristics, which are retrieved at test time to guide execution on subsequent tasks\. All these methods share a common premise: part of the test\-time budget is allocated to producing and retrieving reusable knowledge, a cost that is rarely reported alongside the actor’s\. Our work asks whether that allocation is always the right choice\.

#### Offline reusable knowledge\.

A complementary line of work builds reusable knowledge before deployment begins, through an exploration or synthesis phase that is separate from the evaluation run\. SkillWeaver\(zheng\-etal\-2025\-skillweaver\)discovers and synthesizes Python API skills in an offline phase before deployment\. WALT\(prabhu\-etal\-2025\-walt\)learns stable tool abstractions reverse\-engineered from website structure prior to evaluation, and WebXSkill\(wang\-etal\-2026\-webxskill\)mines parameterized action programs from synthetic agent trajectories\. JEF\-Hinter\(nekoei\-etal\-2025\-jefhinter\)distills offline trajectories into compact, context\-aware hints, including from both successful and failed traces, and retrieves relevant hints for the agent’s current state at inference time\. Offline approaches amortize discovery or distillation cost over many subsequent uses, which is useful when an agent is deployed repeatedly on the same stable environment\. This tradeoff breaks down when the environment is visited infrequently, when its structure changes between discovery and deployment, or when the agent encounters a new domain\. These limitations motivate the online setting that is the focus of our work\.

#### LLM capability and the value of scaffolding\.

Frontier language models have rapidly improved in capabilities central to web agents: multi\-step reasoning, instruction following, and adaptive decision\-making under uncertainty\. Recent models such as GPT\-5\.4\(openai\-2026\-gpt54\), Claude Sonnet 4\.6\(anthropic\-2026\-sonnet46\), and Gemini 3\(gemini\-team\-2025\-gemini3\)can solve complex long\-horizon tasks with relatively lightweight scaffolding, often matching or outperforming more heavily engineered agent architectures\(bechard\-etal\-2026\-terminal\)\. This rapid progress has consequences for augmentation methods: gains that were genuine and substantial over weaker models may shrink or disappear as the base actor becomes more capable on its own\.

#### Alternative paradigms and evaluation robustness\.

Most web agents use a browser, accessibility tree, and fixed action vocabulary\. A line of work explores structurally different paradigms: API\-based agents substantially outperform browser agents on WebArena, with hybrid agents reaching even higher performance\(song\-etal\-2025\-beyond\), and terminal\-based agents operating through code and programmatic interfaces can match or outperform GUI\-based agents on enterprise automation tasks\(bechard\-etal\-2026\-terminal\)\. These approaches avoid some browser\-skill fragilities, but depend on stable, documented APIs that may not exist\. Within the browser\-based setting, evaluation reliability is itself a concern: timeouts, page\-load failures, and nondeterministic UI behavior can change outcomes independently of agent decisions\. Introducing realistic network failures into WebArena, WebVoyager\(he\-etal\-2024\-webvoyager\), and REAL\(garg2025realbenchmarkingautonomousagents\)has been shown to cause significant drops in success rates\(kara\-etal\-2025\-warex\)\. These issues reinforce the importance of multi\-run reporting as a first\-class evaluation criterion\.

## 3Experimental Settings

### 3\.1Tasks and Domains

We evaluate on WebArena\(zhou\-etal\-2024\-webarena\), a benchmark of realistic multi\-step web tasks across isolated website environments\. We report results on three domains\.Shopping, consisting of 187 tasks, is an e\-commerce site with product search, comparison, and checkout tasks\.Reddit, with 106 tasks, is a social discussion platform with navigation, posting, and search tasks\.Admin, which contains 182 tasks, is the administrator panel of the WebArena shopping site, covering product management, order processing, and account administration\. These domains differ in how much their tasks share navigational patterns, which is relevant to whether workflow or skill reuse can pay off\.

The setting is sequential and online: agents attempt tasks in order, and use modules with skill, memory, or workflow components to extract knowledge from earlier trajectories and apply it to later ones\.

We also evaluate on the Level\-1 collection of WorkArena\(drouin\-etal\-2024\-workarena\)tasks focused on common enterprise knowledge\-work interactions in the ServiceNow platform\. This benchmark consists of 33 tasks spanning ServiceNow operations such as dashboards, forms, knowledge\-base search, list filtering, list sorting, menu navigation, and service\-catalog requests\. For WorkArena\-L1, we conduct the same online comparison across all task types, using three seeds per task type for each method\. This benchmark replaces consumer\-style and community websites with enterprise software workflows, while preserving the browser\-based interaction setting used by our agents\.

### 3\.2Studied Systems

We compare four agent scaffolds in our experiments\. Three methods augment the actor with auxiliary modules while keeping the actor’s interaction horizon to at most 10 steps\. The fourth,Vanilla\-IB, is a vanilla actor without auxiliary modules but with an extended interaction budget\.

AWM\(wang\-etal\-2024\-agent\): online workflow memory, in which an LLM\-based auxiliary module induces and retrieves workflows from prior trajectories alongside the actor\.

ASI\(wang\-etal\-2025\-asi\): online programmatic skill induction, in which an LLM\-based auxiliary module synthesizes, verifies, and reuses Python skill functions\.

ReasoningBank\(ouyang\-etal\-2025\-reasoningbank\): online reasoning memory, in which an LLM\-based auxiliary module accumulates and retrieves reasoning strategies from prior successes and failures\.

Vanilla\-IB: a vanilla actor with its interaction horizon extended to 15 steps and with rule\-based pruning of the accessibility tree\. The pruning removes text duplicated between a parent node and its direct children, a common pattern in accessibility trees that inflates context length without adding information\. This pruning involves no LLM calls \(see Appendix[B](https://arxiv.org/html/2606.15017#A2)\)\. We do not present these modifications as a contribution; they are included only to give the vanilla actor more interaction steps under a comparable overall budget\.

The 15\-step horizon forVanilla\-IBtypically does not exceed the total token budget of the augmented methods, because their auxiliary modules consume a substantial fraction of the overall tokens\. Exact budget matching is difficult in exploratory web\-agent benchmarks such as WebArena and WorkArena\. Token usage varies across tasks because agents choose actions dynamically and task complexity differs, so there is no fixed per\-task cost to target\. Moreover, for augmented methods, a single fixed budget cannot be cleanly divided between the actor and its auxiliary modules\. The augmented methods in our comparison also use different aggregate budgets, so there is no single token value against which to matchVanilla\-IB\. The most practical control lever is therefore the actor’s maximum step count\. Setting this horizon to 15 is an approximation rather than an exact budget match\. In many tasks,Vanilla\-IBcould take additional steps while still remaining within the token budget of one or more augmented baselines\. In a small number of tasks, the reverse is true:Vanilla\-IBspends slightly more tokens than one or more augmented methods\. We acknowledge this as a limitation of step\-count\-based budget control, although such cases are uncommon in our experiments\.

Table 1:Per\-domain success rate \(%\) and total token usage per task \(K\) on WebArena\.Domain values are means over three independent runs \(mean±\\pmstd\); per\-run details are in Appendix[E](https://arxiv.org/html/2606.15017#A5)\. Avg\. is the mean over all tasks of Shopping, Reddit, and Admin domains\.
### 3\.3Models and Budget Accounting

We evaluate under three models,Gemini 3 Flash\(gemini\-team\-2025\-gemini3\),GPT\-5\.4\-mini\(openai\-2026\-gpt54mini\), andQwen 3\.6\-27B\(qwen\-team\-2025\-qwen36\), each across three independent runs on each domain\. On WorkArena\-L1, we report the results underQwen 3\.6\-27B\. For every configuration, we report tasksuccess rate \(SR\)andtotal token usage, which aggregates all LLM calls made by the system: actor and any auxiliary module calls \(i\.e\., workflow induction, skill synthesis, memory construction, retrieval, and verification\)\. Token usage is averaged per task in thousands of tokens \(K\)\. For both WebArena and WorkArena\-L1, metrics are reported as means and standard deviations over three runs\. Prior work typically reports metrics on one run, and often does not report token usage; we treat this accounting as a first\-class evaluation criterion\.

## 4Main Results

#### Performance and token usage\.

Table[1](https://arxiv.org/html/2606.15017#S3.T1)summarizes WebArena success rate and total token usage across domains, methods, and models\. Across WebArena domains,Vanilla\-IBachieves the best average success rate for all three base models and, for Gemini 3 Flash and Qwen 3\.6\-27B, is also the most token\-efficient configuration\. The augmented methods therefore do not simply trade extra tokens for higher accuracy; their workflow, skill, or memory modules often increase cost without producing proportional gains in task success\. This complicates the interpretation of scaffolded\-agent gains: auxiliary reasoning can make a system appear stronger by giving it more computation, but much of that advantage disappears when compared with a vanilla actor that spends a similar budget on additional environment interaction\. The effect is strongest for GPT\-5\.4\-mini, suggesting that scaffold quality is itself model\-dependent: weaker models can produce noisy workflows, skills, or reasoning memories that are difficult for the actor to ignore\. For stronger models, the augmented methods are less prone to bad insights but still do not overtakeVanilla\-IB, consistent with modern actors already internalizing many of the planning and navigation abilities that earlier scaffolds attempted to externalize\.

#### Reddit step\-sweep\.

The closest case for augmentation is Reddit with Gemini 3 Flash, where AWM andVanilla\-IBhave identical success rates\. We probe this case by extending theVanilla\-IBactor horizon further \(Figure[2](https://arxiv.org/html/2606.15017#S4.F2)\)\. As the actor horizon increases,Vanilla\-IBsteadily improves, matching AWM from 15 steps onward and surpassing it at larger horizons while still using fewer tokens up until 25 steps\. This comparison also clarifies the role of step count: the augmented methods already exceed the 15\-stepVanilla\-IBtoken budget at their standard 10\-step horizon, so giving them the same longer horizon would increase rather than resolve the cost imbalance\. The apparent benefit of workflow memory on Reddit is therefore at least partly explained by budget asymmetry rather than by a robust advantage of the auxiliary module itself\.

Table 2:Success rate \(%\) and total token usage per task \(K\) on WorkArena\-L1for Qwen 3\.6\-27B\. Values are means over three independent seeds per task type\.
#### WorkArena\-L1\.

The enterprise setting is more balanced but supports the same basic conclusion\. Table[2](https://arxiv.org/html/2606.15017#S4.T2)shows that, under Qwen 3\.6\-27B,Vanilla\-IBis effectively tied with ReasoningBank for the best success rate, outperforms AWM and ASI, and uses fewer tokens than ASI and ReasoningBank\. Thus,Vanilla\-IBremains on the success–cost frontier rather than being dominated by an augmented method\.

#### Where Does the Token Budget Go?

To identify the source of the token overhead, we decompose total usage into actor tokens from environment interaction and module tokens from workflow induction, skill synthesis, retrieval, and verification, in Table[3](https://arxiv.org/html/2606.15017#S4.T3)\. The overhead has two sources\. First, augmented methods pay explicit module costs\. Second, and less obvious, augmentation methods also inflate the actor’s prompt by injecting retrieved workflows, memories, or available skill functions at each step\. This implicit cost cannot be removed by making module calls cheaper: as long as retrieved knowledge is inserted into the actor context, the per\-step actor cost rises with the size of that content\. Augmentation therefore imposes a double cost, combining auxiliary inference with larger actor prompts\.

Table 3:Token breakdown by component under Gemini 3 Flash on WebArena, per task in thousands \(K\)\. Actor: tokens from LLM calls made during interaction steps\. Modules: tokens from auxiliary components \(workflow induction, skill synthesis, retrieval, verification\)\. Both are further split into prompt and completion tokens\. Values are means over three runs\.Vanilla\-IBhas no module calls by construction\.![Refer to caption](https://arxiv.org/html/2606.15017v1/x2.png)Figure 2:Vanilla\-IBhorizon sweep on Reddit under Gemini 3 Flash\.Success rate \(left\) and total tokens per task \(right\) are reported as a function of interaction steps\. The available budget is the token usage of the best\-performing augmentation baseline, in this case AWM, shaded in orange \(right\)\. At the 10\-step mark,Vanilla\-IBSR is below AWM, which in isolation might seem to justify augmentation\. However, the right panel shows AWM already spends roughly twice as many tokens at that point\. Extending theVanilla\-IBhorizon up to 25 steps closes this gap:Vanilla\-IBsurpasses AWM in SR while still using fewer tokens than AWM\.
#### Headroom forVanilla\-IB\.

We set theVanilla\-IBhorizon to 15 steps as a clean, domain\-agnostic choice, but in most cases it is a conservative setting\. As the Reddit sweep shows \(Figure[2](https://arxiv.org/html/2606.15017#S4.F2)\), token usage at 15 steps is still well below AWM’s budget, and success rate continues to rise before saturating around 25 steps\. The unused headroom is even larger on Shopping, where the token gap to augmentation methods is wider\. Together, these results suggest that comparisons should be made along the axis of total token cost rather than nominal step count\. AWM evaluated at 10 steps already spends more tokens thanVanilla\-IBat 15 steps, so extending its actor horizon would only widen this cost gap rather than close it\. For any step budget at which augmented methods are evaluated, a budget\-matched vanilla baseline can be obtained by increasing theVanilla\-IBhorizon accordingly; the same token\-efficiency imbalance therefore persists across operating points\.

## 5Audit Beyond Aggregate Success Rate

### 5\.1Run\-to\-Run Variance

#### Multi\-run reporting\.

A single\-run aggregate success rate hides properties that matter for online web agents\. Even at a fixed temperature, LLM outputs vary across runs due to hardware\-level non\-determinism in distributed inference\. A given task may succeed in one run and fail in another for reasons unrelated to agent capability: a different token sample produces a different action sequence, which may or may not navigate the same interface in the same way\. Another source of variance can be the web environment itself: BrowserGym\(chezelles\-etal\-2025\-browsergym\)and Playwright\(playwright\)can encounter timeouts, slow page loads, or intermittent server errors, and prior work shows that injected web failures can sharply reduce success rates on web tasks\(kara\-etal\-2025\-waber;kara\-etal\-2025\-warex\)\. As a result, two runs of the same agent on the same task set can produce nearly identical aggregate success rates while disagreeing on a meaningful fraction of individual tasks\. This aggregate stability is reassuring for reproducibility but hides large task\-level noise that single\-run evaluations do not expose\.

Multi\-run reporting helps absorb both sources of noise and provides a more reliable picture of what the agent can actually do\. A single reported success rate is an incomplete statistic: it conflates genuine agent capability with the particular set of environment states encountered in that run, and this conflation is especially dangerous when comparing methods that differ in the number or structure of their LLM calls, since some methods have more opportunities to encounter a bad environment state\. We therefore perform our experiments over three runs, and the detailed per\-run results are provided in Appendix[E](https://arxiv.org/html/2606.15017#A5)\.

#### Any\-of\-3 and All\-of\-3\.

To expose the task\-level instability, we compute two complementary bounds:*Any\-of\-3*, the fraction of tasks that succeed in at least one of the three runs \(an OR across runs\); and*All\-of\-3*, the fraction that succeed in all three runs \(an AND across runs\)\. Table[4](https://arxiv.org/html/2606.15017#S5.T4)reports both as illustrative bounds on the latent capability that single\-run averages hide, and shows that the gaps are relatively large for all the methods\. Single\-run success rate is therefore an incomplete estimate of task\-level capability, especially for methods whose learned state changes across the evaluation sequence\.

Table 4:Any\-of\-3 and All\-of\-3 success rates \(%\) on the Shopping domain of WebArenaacross three models\. Any\-of\-3 is the fraction of tasks succeeding in at least one run; All\-of\-3 is the fraction succeeding in all three runs\.Δ↑\\Delta\_\{\\uparrow\}andΔ↓\\Delta\_\{\\downarrow\}are the gaps from the three\-run mean \(Table[1](https://arxiv.org/html/2606.15017#S3.T1)\)\.

### 5\.2Programmatic Skill Fragility

A structural limit of programmatic skill approaches such as ASI is that a synthesized skill is tied to the accessibility\-tree state in which it was created\. These skills often wrap a sequence of low\-level actions and refer to UI elements by BrowserGym accessibility\-tree IDs \(BIDs\), which are assigned dynamically\. BIDs are not stable across interactions\. After the first action in a skill, the page may reload or update, and later BIDs may then point to different elements or to nothing at all\. The function then either fails silently, throws an error that disrupts the agent’s trajectory, or interacts with the wrong element\.

A vanilla agent that queries the LLM at each step adapts to state changes naturally, since it observes the current accessibility tree before choosing each action\. A function\-based approach lacks this adaptability unless it inserts an extra LLM call inside the function at each sub\-step, in which case its efficiency justification largely disappears\. Appendix[D](https://arxiv.org/html/2606.15017#A4)provides explanations of this and related failure modes across all three augmented methods, including concrete task\-level examples where augmented agents fail andVanilla\-IBsucceeds\.

### 5\.3Effect of Accessibility\-Tree Pruning

Vanilla\-IBcombines two changes relative to a plain 10\-step agent: an extended 15\-step actor horizon and rule\-based accessibility\-tree pruning \(Appendix[B](https://arxiv.org/html/2606.15017#A2)\)\. Table[5](https://arxiv.org/html/2606.15017#S5.T5)isolates their individual contributions\.

Table 5:Effect of extending the vanilla actor horizonfrom 10 to 15 steps and rule\-based accessibility\-tree pruning, across all three domains of WebArena under Gemini 3 Flash\. Values are means over three runs \(mean±\\pmstd\)\. The last row corresponds toVanilla\-IB\.The extended horizon is the main success\-rate improvement, while pruning is primarily a token\-reduction mechanism\. Pruning consistently reduces token usage at both horizons and more than offsets the cost of five additional interaction steps, soVanilla\-IBuses fewer tokens than the plain 10\-step baseline while matching or improving success rate\. Its effect on success rate is smaller and domain\-dependent, which is expected: the purpose of pruning is to remove redundant accessibility\-tree text, not to add new reasoning capability\. When pruning helps, we suspect it is because removing duplicate entries makes the context more information\-dense and shortens token distances between related elements, such as labels and fields\. These effects may make actionable content easier to track, but the mixed domain results suggest that this benefit is not uniform\.

## 6Practical Advantages of Stateless Evaluation

Beyond success rate and token cost, two practical properties differ betweenVanilla\-IBand the augmented methods, both stemming from the absence of cross\-task state\.

#### Parallelism\.

Vanilla\-IBtreats tasks as independent episodes and maintains no shared cross\-task state\. As a result, withNNworkers, tasks can be executed concurrently, reducing end\-to\-end wall\-clock time by up to a factor ofNN\. In contrast, AWM, ASI, and ReasoningBank are inherently sequential: each task must finish before the next begins because the memory or skill state derived from its trajectory is reused in subsequent tasks\. This makes parallel deployability an additional practical advantage ofVanilla\-IB, beyond its token efficiency\.

#### Fault tolerance\.

Web benchmarks are run against live or simulated website environments that are not always perfectly reliable: a server may become temporarily unresponsive, a page may time out, or a browser session may crash mid\-task for reasons unrelated to the agent\(kara\-etal\-2025\-waber\)\. ForVanilla\-IB, any such failure is local: the affected task can simply be re\-run in isolation and the result substituted, since no other task depends on its outcome\. For AWM, ASI, and ReasoningBank, this repair is not possible in general\. If taskkkfails due to an environment glitch, the knowledge state used by all subsequent tasksk\+1,k\+2,…k\{\+\}1,k\{\+\}2,\\ldotswas already formed without a potential successful trajectory from taskkk\. Re\-running taskkkin isolation does not retroactively correct the downstream state; a proper repair would require re\-running the entire experiment, which can be expensive\. A single transient environment failure can therefore silently degrade the reported performance of an online augmentation method in a way that is hard to detect\. In our experiments, when we identified a failure caused by the evaluation infrastructure rather than the agent’s decisions, such as a network issue or browser crash, we re\-ran the affected task immediately so that the augmentation methods could still potentially induce knowledge from it, and so that the reported numbers reflect each method’s performance under its intended operating conditions\.

## 7Conclusion

We re\-evaluated online web\-agent augmentation under a fixed total inference budget, asking whether test\-time tokens are better spent on memory, workflow, or skill modules, or directly on additional actor steps\. Across three WebArena domains and three models \(Gemini 3 Flash, GPT\-5\.4\-mini, and Qwen 3\.6\-27B\), a token\-matched vanilla actor matches or surpasses AWM, ASI, and ReasoningBank in aggregate success rate while often using fewer total tokens\. We observed the same trend on WorkArena\-L1 with Qwen 3\.6\-27B, indicating that the effect extends beyond consumer web tasks to enterprise knowledge\-work settings\. These results do not imply that skills or workflow memory are useless: they can provide gains in domains where task structure supports reuse\. Rather, our findings show that such gains are conditional and often disappear once augmentation overhead is compared against a budget\-matched actor\. We also show that run\-to\-run variance materially affects outcomes, motivating the inclusion of multi\-run experiments as a core evaluation criterion rather than an optional diagnostic\. Overall, our results argue for stronger evaluation hygiene in online web\-agent research: total test\-time cost should be reported transparently, augmented agents should be compared against budget\-aware vanilla baselines, and robustness analyses should accompany aggregate success rates\.

## 8Limitations

This paper reports a focused empirical comparison on WebArena under three models \(Gemini 3 Flash, GPT\-5\.4\-mini, and Qwen 3\.6\-27B\), plus WorkArena\-L1 under Qwen 3\.6\-27B, and several limitations follow from this scope\.

Our WebArena conclusions are restricted to three of WebArena’s five domains \(Shopping, Reddit, and Admin\), and methods might behave differently on the other domains\. Our WorkArena\-L1 results are restricted to Qwen 3\.6\-27B and three seeds per task type\. More broadly, the results should not be generalized to all web environments\. Settings with more homogeneous task distributions, longer deployment horizons, or environments built around stable high\-level APIs may yield a more favorable tradeoff for augmentation methods\. Besides, our budget\-matching argument applies to online augmentation only; offline methods such as SkillWeaver\(zheng\-etal\-2025\-skillweaver\)and WALT\(prabhu\-etal\-2025\-walt\)amortize skill\-discovery cost over many uses, and a fair comparison there requires different accounting\.

## References

## Appendix AImplementation Details

Vanilla\-IB, AWM, and ASI are all run using the unified codebase and prompts released bywang\-etal\-2025\-asito ensure that differences in results cannot be attributed to inconsistencies in agent scaffolding, prompt formatting, or environment interaction\. The codebase uses BrowserGym\(chezelles\-etal\-2025\-browsergym\)for interacting with the environment\. AWM and ASI come from the same research group\(wang\-etal\-2024\-agent;wang\-etal\-2025\-asi\), and AWM is included as a baseline in the more recent ASI codebase\. ReasoningBank is run using the codebase released byouyang\-etal\-2025\-reasoningbankdue to its unique prompt structure and agent logic\.

We set the actor LLM temperature ofVanilla\-IBto0, the same as the default value in AWM and ASI\. ReasoningBank sets the temperature to0\.70\.7and we kept the default intact\. The auxiliary modules of AWM, ASI, and ReasoningBank use their respective defaults, which we do not modify\. ReasoningBank’s codebase also ships a vendored WebArena with custom evaluation pipeline that replaces the standard BrowserGym harness\. All methods in this paper are evaluated using the unmodified BrowserGym evaluation harness to ensure identical assessment criteria across configurations\.

We host Qwen 3\.6\-27B on four NVIDIA H100 GPUs using vLLM\(kwon\-etal\-2023\-vllm\)\. We used AI assistants during implementation and manuscript preparation for code\-related tasks, including generating plotting scripts and LaTeX snippets\. We also used them to assist with writing, and for polishing and shortening paragraphs\.

## Appendix BAccessibility Tree Pruning

The accessibility\-tree simplification used inVanilla\-IBis a deterministic rule\-based function with no LLM call, model fine\-tuning, or learned component, and therefore no additional token cost\. The function operates on the BrowserGym\-style accessibility tree, in which each node has a role \(e\.g\.,button,StaticText,heading\) and an accessible name, and is rendered as one indented line per node\.

#### Pruning rule\.

AStaticTextchild node is removed when its text content is already contained verbatim in the accessible name of its immediate parent\. This handles a common pattern in web accessibility trees, where a parent node \(such as a labelled button or a list item\) carries the same text as itsStaticTextchild\. Removing the duplicate cuts context length without dropping information\.StaticTextnodes whose stripped content is empty are also removed\. Non\-StaticTextnodes, node IDs \(BIDs\), roles, and tree structure are not modified\. The rule has one exception: aStaticTextchild is kept whenever its parent’s role is one of \{article,paragraph,heading,strong,emphasis,mark,sectionheader\}, even when the child’s text is already in the parent’s name\. This avoids hollowing out actual prose blocks where theStaticTextchild is the canonical carrier of the rendered text\.

#### Icon characters\.

Before comparison, Unicode private\-use characters in the rangeU\+E600\-\-U\+E6FF\(typically icon\-font glyphs that render as visually empty placeholders\) are stripped from both the parent name and theStaticTextchild\. Literal\\uXXXXescapes that appear in the serialized tree are also stripped\. This prevents icon characters from blocking an otherwise valid redundancy match\.

## Appendix CPrompt Templates

Vanilla\-IB, AWM, and ASI use the prompt templates released bywang\-etal\-2025\-asiwithout modification, except as noted below\. Since the templates are publicly available in the ASI code repository, we do not report them here\.

#### Model\-specific modification\.

The prompts are model\-agnostic with one exception\. When running experiments with GPT\-5\.4\-mini, we observed two behaviors that interfered with evaluation\. First, the model frequently produced action outputs with no accompanying reasoning or explanation\. Such explanations are important for augmentation methods: the induction modules parse them to extract reusable knowledge from trajectories\. Second, the model sometimes generated turns asking the user for clarification or additional input, which is not possible during automated benchmark evaluation\. To address both issues, the following text was appended to the action prompt for GPT\-5\.4\-mini:

> You cannot ask for follow\-up questions or hand back to the user\. Do your best with the information already available\. Output a summary of your reasoning and the important information you have found, followed by the best actions at this step \(in one triple backticks block\)\.

It is worth noting thatVanilla\-IBis less dependent on reasoning traces in the output: the vanilla agent acts only on the current observation, so in many cases explanatory text is overhead that could in principle be removed to reduce completion tokens\. This represents an additional efficiency advantage ofVanilla\-IBthat our numbers do not capture\. However, to ensure a fair comparison across all methods, we applied the same prompt to all methods under each model, including the above addition for GPT\-5\.4\-mini\.

## Appendix DFailure Mode Analysis of Augmented Agents

Section[5\.2](https://arxiv.org/html/2606.15017#S5.SS2)discusses the structural fragility of programmatic skills in dynamic accessibility\-tree interfaces at a conceptual level\. This appendix extends that discussion with quantitative characterizations of failure modes across all three augmented methods, and presents concrete examples of tasks where augmented agents fail whileVanilla\-IBsucceeds\. Unless stated otherwise, quantities are averaged over three independent runs per domain–model configuration\.

### D\.1ASI: Verification Confounds Function Success with Agent Recovery

ASI induces Python functions from successful trajectories and adds them to a shared action library after a verification episode\. The intended guarantee is that only working functions enter the library\. In the verification procedure, the induced function is called as the first decision\. If the function changes the environment but fails on one of its underlying actions, the agent may still recover via primitive actions and solve the task\. In that case, the episode can be judged a success and the function is written to disk, without actual verification\. A common instance of this failure is when the function fails because an element ID has changed after the page reloaded for the verification episode\. Functions that require arguments whose values depend on a prior action’s page state are especially prone to this: the caller cannot supply a valid BID for an element that only becomes visible after an earlier click\. The induced functions are also typically short, averaging approximately 2\.4 primitive actions per function under Gemini 3 Flash and 2\.5 under GPT\-5\.4\-mini, which limits the abstraction benefit\. Some functions also hardcode task\-specific values that do not generalize across tasks; for example, the functionfill\_refund\_request\_formin the shopping library hardcodes the complaint message “It broke after just three days of use” regardless of the actual reason for the refund\.

Table[6](https://arxiv.org/html/2606.15017#A4.T6)quantifies the verification\-confound effect\. We define a*first\-step verification failure*as a verification run in which the induced function fails before the second agent action\. A*recovered*verification is one that was still judged correct despite that first\-step failure\. First\-step failures are not rare\. Under GPT, the induced function fails immediately in 72\.2% of Shopping verifications; over half of those are recovered and still produce a stored library entry\. Even under Gemini Shopping, where the first\-step failure rate is lower \(33\.3%\), two thirds of failures recover\. In both cases the result is a function library that contains entries whose correctness at verification time was not due to the function itself\.

Table 6:ASI verification failures at the first high\-level function callon WebArena\. Values are averages over three replicas\.*Attempts*: total number of function induction attempts by ASI across all tasks\.*First\-step fail*: number of attempts where the induced function failed before the second agent action\.*Recov\. rate*: fraction of first\-step failures where the verification episode was nonetheless judged correct, causing a potentially broken function to be stored in the shared library\.
### D\.2AWM: Generic Templates and Induction from Failed Trajectories

AWM induces abstract natural\-language workflows from trajectories the judge deems successful, then retrieves and injects the closest workflow into the actor’s prompt at each inference step\. Many induced workflows are too generic to add navigational value\. A stored shopping workflow for order\-history aggregation reduces to: “click the account link, click the full\-order\-history link, send a summary”\. This describes the high\-level intent without preserving operational details that determine correctness, such as whether pagination is needed, what the stopping condition is, or how totals are computed\. Similarly, a stored price\-range workflow directs the agent to sort, toggle direction, navigate to the last page, and report the range, but omits checks for sort direction availability, filtered result sets, or whether the last\-page element is reachable in the current state\. Because the workflow is injected into the actor’s prompt, the agent must decide at each step whether to follow it or ignore it, a burden that smaller models are especially ill\-equipped to handle, as it competes with the current accessibility tree and task instruction\.

A more fundamental problem is that AWM’s induction is gated only by the judge’s verdict\. Table[7](https://arxiv.org/html/2606.15017#A4.T7)shows the fraction of workflow\-induction events triggered by failed trajectories\. On Shopping, nearly half of all induction events, 49\.5% under Gemini and 52\.3% under GPT, originate from failed agent trajectories\. The contamination is substantially lower for Reddit \(11–19%\), which may help explain AWM’s stronger performance on that domain\. Because AWM’s duplicate\-suppression mechanism checks only whether a new workflow semantically overlaps an existing one, it does not filter out strategies that are simply wrong: once an incorrect workflow enters the library, it can be retrieved and applied to all future tasks in the same category\.

Table 7:AWM workflow\-induction statisticson WebArena, averaged per run\.*Induced*: total induction events triggered by the judge for a run\.*From failed \(%\)*: fraction originating from tasks the ground\-truth evaluator classified as failed\.*Final WFs*: distinct workflows in the final library after duplicate suppression\.
### D\.3ReasoningBank: Contamination and Memory Over\-Application

ReasoningBank stores a few short lessons from every task trajectory regardless of outcome, labeling each entry as a success or failure based on the judge’s verdict\. TheKKmost semantically similar entries are retrieved at inference time and injected into the actor’s context\. Table[8](https://arxiv.org/html/2606.15017#A4.T8)reports the size and contamination of the resulting memory corpus\. On Shopping tasks, more than half of all success\-labeled entries, 52\.9% for Gemini and 59\.5% for GPT, come from trajectories that failed according to the ground\-truth evaluator\. These entries are framed with positive guidance language regardless of their true provenance, so the retrieval mechanism cannot distinguish between lessons that worked and lessons that did not\.

DomainTotalmem\.Succ\.\-labeledFP insucc\. \(%\)FN infail \(%\)Gemini 3 FlashShopping5189652\.9%40\.7%Reddit2943230\.2%25\.3%Admin5078642\.6%40\.4%GPT\-5\.4\-miniShopping5446759\.5%18\.4%Reddit3072060\.0%17\.4%Admin5326248\.1%25\.2%

Table 8:ReasoningBank memory\-corpus statisticson WebArena, averaged over three runs\. Tasks processed per run: 187 \(Shopping\), 106 \(Reddit\), 182 \(Admin\)\.*Succ\.\-labeled*: tasks labeled successful by the judge\.*FP in succ\.*: fraction of success\-labeled tasks from trajectories that failed according to the ground\-truth evaluator\.*FN in fail*: fraction of fail\-labeled tasks from trajectories that succeeded according to the ground\-truth evaluator\.A separate concern is injection pressure\. Even though only a few memories are retrieved per task, they are referenced heavily in the agent traces: across all domain–model configurations, we observe an average of 8–12 explicit memory\-item mentions per task, with over 95% of Gemini tasks and over 75% of GPT tasks referencing at least one retrieved memory\. The retrieved memories therefore function as active policy hints rather than passive background\. When a memory is overly general or mildly mismatched to the current task, the agent may prioritize it over the current accessibility tree, producing extra navigation steps, unnecessary checks, or incorrect early termination\.Vanilla\-IB, which carries no retrieved memory layer, avoids this class of distraction entirely by grounding each action in the current observation\.

### D\.4Discussion: Structural Limits of Judge\-Based Quality Control

The contamination patterns described above for AWM and ReasoningBank, and the verification confound in ASI, share a common structural root: each method delegates quality control to an LLM judge whose input is fundamentally incomplete for the tasks it is asked to evaluate\.

The judge receives the agent’s textual output \(its reasoning trace and final response\) and, in some configurations, a snapshot of the browser state at the end of the episode\. It does not observe the sequence of intermediate page states the agent traversed, the specific elements it interacted with, or whether information the agent reports was actually encountered during navigation or confabulated in the reasoning trace\. For tasks where correctness depends on having visited the right pages and extracted the right values, a judge working from this terminal view cannot reliably reconstruct whether the work was actually done\.

This is not necessarily a matter of judge capability\. A more powerful model receiving the same partial input faces the same gap: the evidence needed to verify navigation\-dependent correctness is simply absent from the judge’s context\. Nor is it straightforwardly fixable by expanding what is passed to the judge\. Supplying the full trace at every step would require passing on the order of hundreds of thousands of tokens per judge call for a ten\-step task on a content\-heavy page, making each evaluation call more expensive than the agent trajectory it is judging, and possibly exceeding practical context limits entirely\.

Table 9:Example failures of augmented agentswhereVanilla\-IBsucceeds\. Tasks are selected from WebArena \(single runs shown; all from Gemini 3 Flash\)\.
### D\.5Illustrative Task\-Level Failures

Table[9](https://arxiv.org/html/2606.15017#A4.T9)presents representative task\-level failures drawn from our Gemini 3 Flash runs, showing in each case what the augmented agent does and howVanilla\-IBsucceeds on the same task\.

## Appendix EPer\-Run Variance Results

Tables[10](https://arxiv.org/html/2606.15017#A5.T10),[11](https://arxiv.org/html/2606.15017#A5.T11), and[12](https://arxiv.org/html/2606.15017#A5.T12)report success rate and token usage for each method across three independent runs under Gemini 3 Flash, GPT\-5\.4\-mini, and Qwen 3\.6\-27B, respectively\. Standard deviations are sample std over the three runs\.

Table 10:Per\-run success rate \(%\) and token usage \(K per task\)across three independent runs under Gemini 3 Flash on WebArena\. Standard deviations are sample std over the three runs\.Table 11:Per\-run success rate \(%\) and token usage per task \(K\)under GPT\-5\.4\-mini on WebArena\. Standard deviations are sample std over three runs\.Table 12:Per\-run success rate \(%\) and token usage per task \(K\)under Qwen 3\.6\-27B on WebArena\. Standard deviations are sample std over three runs\.