ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents

# ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents
Source: [https://arxiv.org/html/2605.16116](https://arxiv.org/html/2605.16116)
Chinmay Savadikar1,2,†,, Mingyu Zhao2,, Yuanzheng Zhu2,†, Han Li2, Shuang Xie2, Alberto Castelo2,Tianfu Wu1,Lingyun Wang2, 1North Carolina State University2ShopifyThis work was done as part of internship at ShopifyEqual contribution, order decided randomlyCorresponding author:lingyun\.wang@shopify\.com

###### Abstract

Developing and evaluating e\-commerce web agents requires environments that preserve meaningful task structure while enabling controllable, reproducible, and scalable scientific comparison\. Existing methodologies force a tradeoff: live storefronts provide realism but are non\-stationary, difficult to inspect, and irreproducible, while hand\-built sandbox benchmarks provide control but cover only a narrow range of layouts, catalogs, policies, and interaction patterns\. We argue that the core bottleneck is methodological: the field lacks a scalable way to construct evaluation settings that are simultaneously realistic, diverse, controllable, inspectable, and reproducible\. We introduce ShopGym, an integrated framework for realistic simulation and scalable benchmarking of e\-commerce web agents\. ShopGym is a framework for constructing e\-commerce simulation environments and grounded benchmark tasks\. Its simulation layer, ShopArena, converts live seed storefronts into self\-contained sandbox shops through anonymized shop specifications and a staged, validated generation process\. On top of these simulated storefronts, ShopGuru synthesizes benchmark tasks across seven skill categories, grounding each task in the shop’s catalog, navigation structure, policies, and interaction affordances\. Together, ShopArena and ShopGuru produce self\-contained, resettable, inspectable, and stable evaluation artifacts that preserve structural properties and agent\-evaluation signals relevant to shopping tasks\. We validate the framework through graph\-based structural analysis and agent\-based behavioral evaluation with 224 generated tasks across six sandbox shops: three constructed with synthetic data and three with real data\. Our results show that the synthetic shops preserve key structural properties of live storefronts, with agent performance on synthetic shops positively correlated with performance on live storefronts\.

## 1Introduction

As web agents become more capable, the field increasingly needs stable environments that support long\-horizon interaction in realistic, user\-facing settings\. E\-commerce is a particularly demanding domain for this purpose: shopping tasks require agents to navigate multi\-page interfaces, interpret product information, satisfy user constraints, compare alternatives, and execute actions such as selecting variants or adding items to cart\. These tasks combine perception, reasoning, and sequential decision\-making in a grounded setting, making storefronts a compelling testbed for web\-agent evaluation\. The importance of this domain is reflected in growing interest in shopping assistants\[[23](https://arxiv.org/html/2605.16116#bib.bib23),[25](https://arxiv.org/html/2605.16116#bib.bib25),[17](https://arxiv.org/html/2605.16116#bib.bib17)\], offline A/B testing\[[2](https://arxiv.org/html/2605.16116#bib.bib2),[15](https://arxiv.org/html/2605.16116#bib.bib15)\], and user behavior simulation for e\-commerce applications\[[10](https://arxiv.org/html/2605.16116#bib.bib10),[23](https://arxiv.org/html/2605.16116#bib.bib23),[26](https://arxiv.org/html/2605.16116#bib.bib26),[20](https://arxiv.org/html/2605.16116#bib.bib20)\]\.

Existing environments for e\-commerce agents face a fundamental tradeoff between realism and experimental control\. Live\-websites\[[21](https://arxiv.org/html/2605.16116#bib.bib21),[7](https://arxiv.org/html/2605.16116#bib.bib7),[11](https://arxiv.org/html/2605.16116#bib.bib11)\]offer strong realism: agents are run and evaluated against realistic layouts, and naturally occurring variation and interaction patterns\. But live storefronts are inherently non\-stationary\. Catalogs, layouts, and webpage design on live storefronts can change, collections can be reorganized, and storefronts participating in live A/B testing can serve separate variants for different requests\. This non\-stationary behavior makes reproducible training and evaluation of web agents challenging\. The evaluation performance can reflect not only agent capability but also incidental variation from website drift, location\-specific behavior, and operational noise, making results difficult to reproduce and agents difficult to compare fairly\[[21](https://arxiv.org/html/2605.16116#bib.bib21)\], and training runs difficult to reproduce\.

Sandbox environments\[[28](https://arxiv.org/html/2605.16116#bib.bib28),[8](https://arxiv.org/html/2605.16116#bib.bib8),[22](https://arxiv.org/html/2605.16116#bib.bib22),[16](https://arxiv.org/html/2605.16116#bib.bib16),[18](https://arxiv.org/html/2605.16116#bib.bib18),[12](https://arxiv.org/html/2605.16116#bib.bib12)\], in contrast, are controllable and repeatable by design, but at the cost of realism and diversity of live websites\. Manually constructing realistic storefront environments is expensive and covers only a narrow range of layouts, catalog structures, and interaction patterns\.

Existing work proposing environments for e\-commerce agents has largely explored the two methodologies, with works like DeepShop\[[11](https://arxiv.org/html/2605.16116#bib.bib11)\]using live websites, and WebShop\[[22](https://arxiv.org/html/2605.16116#bib.bib22)\], ShoppingBench\[[16](https://arxiv.org/html/2605.16116#bib.bib16)\], ShopSimulator\[[18](https://arxiv.org/html/2605.16116#bib.bib18)\], and WebMall\[[12](https://arxiv.org/html/2605.16116#bib.bib12)\]manually building sandbox environments\.We argue that the deeper bottleneck is methodological: the field lacks a scalable way to*constructreliable simulated environments that are realistic, controllable, inspectable, and reproducible*\.

To address this gap, we proposeShopGym, a framework for constructing realistic e\-commerce simulation environments and grounded web\-agent benchmarks\. ShopGym comprises two complementary components in a synergistic workflow: \(a\) the simulation environment layer,ShopArena, transforms one or more live seed storefronts into a self\-contained sandbox shop; and \(b\)ShopGurusynthesizes benchmark tasks grounded in the sandbox shops\.

![Refer to caption](https://arxiv.org/html/2605.16116v1/x1.png)Figure 1:ShopGymcomprises two components\.ShopArenaprovides a simulation environment populated with synthetic sandbox shops, along with a scalable pipeline that generates new sandbox shops from one or more live seed storefronts through specification synthesis followed by data and code generation\.ShopGuruthen consumes the resulting catalog, collections, pages, and shop statistics to generate both short\-horizon tasks covering primitive skills and long\-horizon shopping journeys that combine these skills\.ShopArena: validated simulation environment\.To generate the simulation environment grounded in real storefronts, ShopArena separates storefront understanding \(exploration\) from sandbox synthesis \(generation\) through an intermediate specification document that serves as the sole interface between exploration and generation\. Given one or more seed storefronts, ShopArena autonomously explores the sites and writes an anonymized specification capturing their navigation structure, user\-facing affordances, high\-level page organization, and catalog statistics\. When multiple seeds are provided, ShopArena composes their structural and behavioral signals into a single specification, so one sandbox shop can span diversity that no individual source storefront would cover\. A staged generation pipeline then synthesizes a runnable sandbox shop from the specification, iteratively refining the implementation through build checks, type checking, and multimodal visual validation\. ShopArena does not aim to clone a particular storefront; it producesbehaviorally aligned approximations– synthetic shops grounded in real storefront structure and interaction patterns, but with synthetic brand identity, products, descriptions, and imagery\. The goal is not visual or brand\-level fidelity but structural, behavioral, and statistical alignment sufficient for controlled evaluation\.*Because the specification is human\-readable, benchmark creators can also edit generated environments in controlled ways without re\-exploring the source storefronts*\.

ShopGuru: grounded task generation\.ShopGuru consumes the sandbox shop’s catalog, navigation graph, and store policies and synthesizes tasks that are valid against the environment they will be executed in\. This design separates live\-web exploration from benchmark execution\. The generated shops are stable and resettable, while the intermediate specification provides an inspectable control surface that benchmark creators can edit without re\-exploring the source storefronts\.

We evaluate ShopGym by generating synthetic sandbox shops and anonymized twin shops, then validating them using structural metrics and agent\-based behavioral comparisons\. Across paired tasks defined over source and generated shops, agent success rates on sandbox environments positively correlate with success rates on live storefronts, indicating that generated storefronts retain meaningful evaluative signal from live\-web settings while substantially improving control and reproducibility\. While it does not completely replace live\-web evaluation, ShopGym provides a stronger methodology for standardized benchmarking and analysis\. To summarize, our contributions are:

- •We identify the realism\-control tradeoff in e\-commerce environments and agent evaluation as a methodological bottleneck, and argue that the missing capability is scalable construction of realistic, controllable, and reproducible environments and evaluation benchmarks\.
- •We bridge this gap withShopGym, a framework for constructing realistic, controlable, and reproducible e\-commerce agent evaluation environments\.
- •We presentShopArena, a pipeline that converts live seed storefronts into anonymized, self\-contained sandbox shops through specification synthesis and staged code generation\.
- •We presentShopGuru, a grounded task\-generation pipeline that created short\-horizon and long\-horizon shopping tasks tied to each generated shop’s catalog, navigation structure, filters, and policies\.
- •We validate that the simulated environments preserve meaningful signal from live storefront evaluation, supporting their use as controlled, behaviorally aligned benchmarking environments\.

## 2ShopGym

ShopGym combines two complementary frameworks:ShopArena, a simulation environment composed of interactable sandbox shops, andShopGuru, an evaluation framework with a synthetic task generation pipeline grounded in the sandbox shops\. ShopArena consists of a simulation environment containing multiple sandbox shops\. It also provides a scalable pipeline to extend the environment by generating new synthetic sandbox shops: given one or more seed storefronts\{S\}N\\\{S\\\}\_\{N\}\(real e\-commerce websites\), the ShopArena generation pipeline can generate a stable, self\-contained, and anonymized sandbox shopEEusing a multi\-agent shop exploration and code synthesis pipeline\. The ShopGuru evaluation framework generates a set of instruction following tasksTTdefined over the sandbox shopEE\. Following standard practice\[[28](https://arxiv.org/html/2605.16116#bib.bib28),[1](https://arxiv.org/html/2605.16116#bib.bib1)\], we define a taskT=\(E,s0,p,V\)T=\(E,\{s\_\{0\}\},p,V\)as a sandbox shopEEwith initial states0s\_\{0\}, a natural\-language intentpp, and a verification functionVVthat maps the agent’s trajectory to a binary success score\. Figure[1](https://arxiv.org/html/2605.16116#S1.F1)illustrates the full pipeline\.

### 2\.1ShopArena

To generate each sandbox shop in the ShopArena environment, we propose a scalable pipeline that converts one or more live seed storefronts\{S\}N\\\{S\\\}\_\{N\}into a realistic, self\-contained, inspectable and reproducible sandbox shop\.*Importantly, the pipeline can be extended to any live storefront, making ShopArena an evolving environment that can be updated with new sandbox stores*\.

The generation pipeline\. Doing this end\-to\-end in one pass is an extremely long\-horizon task: it combines storefront exploration and understanding, synthetic catalog construction, and source\-code generation in a single trajectory that is hard to control and debug, and prone to incomplete outputs as agentic LLMs accumulate context\[[9](https://arxiv.org/html/2605.16116#bib.bib9),[14](https://arxiv.org/html/2605.16116#bib.bib14)\]\. We instead split synthesis into two phases connected by a single intermediate artifact: an anonymized design specificationMM\. The*exploration*phase \(§[2\.1\.1](https://arxiv.org/html/2605.16116#S2.SS1.SSS1)\) browses\{S\}N\\\{S\\\}\_\{N\}and writesMM; the*generation*phase \(§[2\.1\.2](https://arxiv.org/html/2605.16116#S2.SS1.SSS2)\) readsMMand writes the source code for a runnable sandbox shop\. This split confines all live\-web dependence to exploration, lets either phase be re\-run independently, and turns the human\-readable specification into a control surface for editing the generated environment without re\-exploring the source\.

Multi\-agent framework\.Both phases are organized as small sets of agents communicating through the file system\. Each agent is an instance of a coding agent \(Claude Code, with Claude Opus 4\.6 as the underlying model\) differentiated by \(a\) the tools it is given access to and \(b\) the objective described by its system prompt\. Agents share a workspace directory but do not share a conversational context\.

#### 2\.1\.1Exploration: From Live Storefront to Specifications

The exploration phase browses one or more seed storefronts\{S\}N\\\{S\\\}\_\{N\}, extracts the properties of a live storefront that matter for downstream environment generation, and generates a specification documentMM\. The specification consists of three facets: \(i\) a design manual, which is a natural language description of the store and its visual style; \(ii\) a structured list of its attributes that are typically present on storefronts such as filter attributes, navigation menu design, pagination information, etc\. \(iii\) statistics such as number of products, price range and distribution, etc\. Figure[2](https://arxiv.org/html/2605.16116#S2.F2)shows the full exploration pipeline\.

Static prefetch\. The phase begins with a deterministic prefetch of canonical surfaces \(homepage HTML, sitemap, search and cart endpoints, public catalog JSON, policy pages\)\.

Planner agent \(Agentplan\\textsc\{Agent\}\_\{\\text\{plan\}\}\)\.A single planner consumes the prefetched files and, with a small live\-browsing budget, writes a plan that decomposes exploration ofSiS\_\{i\}into focused subtasks\. The planner is shown an example plan but is required to adapt it toSiS\_\{i\}, so coverage is grounded in what the storefront actually exposes rather than in a fixed template\. An example plan is shown in Fig[G\.1](https://arxiv.org/html/2605.16116#A7.SS1)\.

Specification agent \(Agentspec\\textsc\{Agent\}\_\{\\text\{spec\}\}\)\.For each planned subtask, the harness spawns a fresh specification agent equipped with browser\-automation tools \(Playwright\)\. The agent performs targeted browsing, captures evidence \(screenshots, snapshots, optional XHR traces\), and writes an anonymized fragment ofMMfor its subtask\.Agentspec\\textsc\{Agent\}\_\{\\text\{spec\}\}is instructed to ignore brand names, product names, named individuals, and absolute URLs at write time, so everything downstream of exploration is anonymous by construction — no separate redaction pass is required\.

Consolidation and multi\-seed composition\.A non\-agentic LLM merges the per\-subtask fragments into one specification per feature, yieldingMM\. For a sandbox shop calibrated on multiple seeds, exploration runs independently on each storefront, producing oneM​\(Si\)M\(S\_\{i\}\)per seed; a second LLM merge composes them into a single composite specification\. Multi\-seed composition is how ShopArena scales*within*a shop: a single sandbox can span structural and behavioral diversity that no individual source storefront would cover\.

At the end of the exploration phase, the pipeline produces the anonymized shop specificationsMMfor the sandbox shop, with the design document divided into multiple predefined*features*\(website skeleton, homepage, collections, etc\.\) A full list of features and an example of the full specifications is provided in Appendix[F](https://arxiv.org/html/2605.16116#A6)

![Refer to caption](https://arxiv.org/html/2605.16116v1/x2.png)Figure 2:The exploration phase fetches key pages, and the public product catalogue\. A planner agent then writes a plan using a list of predefined features to implement\. Finally, Specification agents write anonymized specifications for each sub\-task, which are merged to form the shop specifications\.
#### 2\.1\.2Generation: From Specifications to Sandbox Shop

The generation phase synthesizes a runnable storefront fromMM\. Because it sees onlyMMand never the prefetched evidence, generation is anonymous by construction\. The generation phase has two parts: a catalog synthesis pipeline and a source\-code synthesis loop\.

Synthetic catalog generation\.The generation step first synthesizes the collection details \(description, associated products\), product names, product descriptions, and product images using the design manual and statistics fromMM\. The data synthesis is structured as simple LLM calls that generate the data for one collection at a time\. The details of the models used for the data generation are provided in Appendix[B\.1](https://arxiv.org/html/2605.16116#A2.SS1)\.

Stepwise source\-code synthesis\.ShopArena generates the storefront source code in a fixed sequence of*steps*, each scoped to one feature: site shell and homepage, collection pages, product detail page, cart, search and filtering, policy pages, and a final integration pass\. Each step is given only the slice ofMMrelevant to its feature, the catalog, and the current state of the codebase, and is executed by a dedicated execution–verification loop\. Steps run in a fixed order over a shared workspace, so later steps build on a stable surface produced by earlier ones, and a failure in one step can be repaired without regenerating the rest of the shop\.

![Refer to caption](https://arxiv.org/html/2605.16116v1/x3.png)Figure 3:The generation phase\. A staged sequence of feature\-scoped steps is each driven by an execution–verification loop in which a fresh Execution Agent edits the codebase and a separate Verification Agent writes natural\-language feedback\.Execution–verification loop\.Within each step, code generation is structured in the spirit of the Ralph technique111[https://ghuntley\.com/ralph/](https://ghuntley.com/ralph/): a fresh agent process is spawned at each iteration, reads the current state of the workspace from disk, edits the codebase, and exits; the harness then runs verification, writes feedback to disk, and spawns the next iteration\. Two specialised agents alternate within the loop:

- •Execution Agent \(Agentexec\\textsc\{Agent\}\_\{\\text\{exec\}\}\)\.At each iteration, a coding agent with the codebase and terminal access receives the feature to implement, the slice ofMM, the catalog, and \(if present\) the verifier’s most recent feedback\. It edits the source code and runs lightweight checks \(type checking, build\)\.
- •Verifiers\.The code generated byAgentexec\\textsc\{Agent\}\_\{\\text\{exec\}\}is checked by both rule\-based verifiers and a multimodal verification agent\. The rule\-based verifiers ensure the site builds successfully, required pages return valid HTTP success codes, and outputs are properly structured\. The verification agent then hosts the shop locally, uses browser automation to inspect the rendered site for visual and functional issues, and produces natural\-language feedback for the nextAgentexec\\textsc\{Agent\}\_\{\\text\{exec\}\}iteration\.

Long\-context management\.Each step could in principle be driven by one long\-running agent that holds the codebase, the verifier’s reports, and the slice ofMMin a single context window\. We avoid this because long\-horizon code generation degrades as context grows\[[9](https://arxiv.org/html/2605.16116#bib.bib9),[14](https://arxiv.org/html/2605.16116#bib.bib14)\], and we observed the same failure mode in our own development: partially broken outputs the agent itself reports as complete\. Spawning a fresh process at each iteration externalises three concerns that would otherwise pile up inside one prompt: the*state context*\(the current source code, read directly from disk\), the*implementation context*\(the agent’s prior tool\-use traces, discarded\), and the*evaluation context*\(the verifier’s most recent feedback, at a known path\)\. Each freshAgentexec\\textsc\{Agent\}\_\{\\text\{exec\}\}iteration sees only what the workspace and the latest verifier note convey; the prompt does not grow unboundedly, and the agent re\-establishes context from the artifacts that matter\. The loop runs for a fixed iteration budget \(30 by default\) or terminates early when the agent reports no further action is required\.

Final consolidation\.After all feature\-specific steps complete, ShopArena runs a final consolidation pass: a global agent navigates a tour of canonical routes, and fixes the remaining defects raised by the verifiers in prior steps\.

### 2\.2ShopGuru: Grounded Task Generation

ShopGuru uses the collections, product details, and statistics from ShopArena’s generation phase \(§[2\.1\.2](https://arxiv.org/html/2605.16116#S2.SS1.SSS2)\) and generates synthetic tasks grounded in the sandbox shop\. The pipeline uses a small set of*deterministic generators*, which construct short horizon tasks meant for evaluating primitive skills, and a single*LLM\-authored*generator for long\-horizon shopping journeys\. The pipeline is structured as an execution\-verification loop: the generated tasks \(using rule\-based or LLM\-based generators\) are verified using multiple rule\-based verifiers\. The verifiers validate that the collections, products, filter types and options, and pages referenced by the task are present in the shop’s data\. A detailed description of the verifiers can be found in Table[2](https://arxiv.org/html/2605.16116#A11.T2)\. Figure[4](https://arxiv.org/html/2605.16116#S2.F4)illustrates the full pipeline\.

#### 2\.2\.1Skill Catalog

We organize ShopGuru’s seven skill categories into three short\-horizon groups and one long\-horizon group, summarized in Table[1](https://arxiv.org/html/2605.16116#A8.T1)in Appendix[H](https://arxiv.org/html/2605.16116#A8)\. The three short\-horizon groups are themselves the underlying primitives that the long\-horizon journeys re\-combine\.

#### 2\.2\.2Short\-Horizon Task Generation

Short\-Horizon tasks are meant to evaluate the basic instruction following capabilities of agents\. The short\-horizon tasks are generated using rule\-based generators to evaluate three primitive skills:

Product Discovery \(search\-exact,search\-substitute\)\.The discovery generators sample products from the catalog and emit two task variants per product: one asking the agent to find the product by its exact title, and one asking for a semantically similar alternative\. To keep the dataset diverse, the sampler accepts at most one product perproduct\_type, and excludes inactive products, gift cards, and products with no available variant \(so an agent is never asked to add an out\-of\-stock item\)\.

Filter\-Selection \(browse,filter\)\.Browse tasks pick a populated, non\-generic collection \(at least three products; generic catchall handles likeall,sale,featured, orbest\-sellersexcluded\) and ask the agent to navigate to the collection page and add any product to cart\. The filter generator additionally pairs the collection with a realistic facet\(dim,value\)\(\\textit\{dim\},\\textit\{value\}\)\. The non\-trivial design decision here is feasibility: shop\-wide option statistics are a poor source for per\-collection filter facets because a dimension can be common in the catalog overall yet absent from a specific collection\. We therefore build a*per\-collection*option index from the collection’s own products and only emit a task when at least one product in the collection actually has the chosen\(dim,value\)\(\\textit\{dim\},\\textit\{value\}\)pair\. As a fallback, when no priority variant\-option dimension is feasible in a collection, we use the universal product fieldsVendor\(storefront\-rendered as “Brand”\) andProductType\(rendered as “Type”\), which every storefront filter UI exposes regardless of variant\-option coverage\.

Information\-Seeking \(shipping,returns\)\.The policy generator searches the shop’s extractedpages\.jsonfor pages whose title or handle matches a small keyword list \(shipping/deliveryfor shipping;return/refund/exchangefor returns\)\. Both the/pages/⟨\\langlehandle⟩\\rangleroute and the platform\-default/policies/⟨\\langlename⟩\\rangle\-policyroute are emitted as soft URL hints \(url\_containsplus aurl\_contains\_altlist\), since some storefronts surface the policy under a custom page handle and others under the platform default\.

![Refer to caption](https://arxiv.org/html/2605.16116v1/x4.png)Figure 4:The ShopGuru task generation pipeline\. Deterministic generators emit short\-horizon tasks \(search\-exact,search\-substitute,browse,filter,shipping,returns\); an LLM\-authored generator emits long\-horizon shopping journeys reconciled against the shop data via a validator\-driven polish loop\. All emitted tasks are checked by rule\-based verifiers \(Table[2](https://arxiv.org/html/2605.16116#A11.T2)\)\.
#### 2\.2\.3LLM\-Authored Long\-Horizon Journeys

Long\-horizon journeys are the workhorse of the benchmark’shard\_long\_horizonbundle\. They mimic the kind of multi\-step behavior we observe in real human shopper traces — filter→\\rightarrowsort→\\rightarrowinspect→\\rightarrowdetour to a policy page→\\rightarrowadd to cart→\\rightarrowedit quantity — which the primitive\-task generators by themselves do not cover\. Authoring such journeys by hand is prohibitively expensive at scale, so ShopGuru delegates the long\-horizon authoring to an LLM \(e\.g\. GPT\-5 or Gemini 3 Pro\) with two structural guards: \(i\) a few\-shot prompt grounded in the shop’s actual catalog, navigation, and policy pages, and \(ii\) a polish loop that uses the same validator from §[2\.2\.4](https://arxiv.org/html/2605.16116#S2.SS2.SSS4)as a feedback signal\.

Listing[3](https://arxiv.org/html/2605.16116#LST3)shows a representative end\-to\-end \(E2E\) task that chains collection navigation, a Color filter, a price sort option, and a final cart action\. The few\-shot examples in the user prompt template \(eight in total; the full prompt and three representative examples are in Appendix[L](https://arxiv.org/html/2605.16116#A12)\) are constructed to reflect commonly observed click\-stream patterns in human shopper sessions on production storefronts, drawing on prior characterizations of online shopping behavior\[[19](https://arxiv.org/html/2605.16116#bib.bib19),[10](https://arxiv.org/html/2605.16116#bib.bib10)\]: multi\-pet household shopping, returns\-cautious browse\-without\-purchase, free\-shipping\-threshold calculation, cross\-brand comparison, filter\-recover, first\-time\-visitor info\-page detour, homepage\-callout brand drilldown, and so on\.

The polish loop \(Appendix[J](https://arxiv.org/html/2605.16116#A10)\) follows from a known weakness of single\-pass LLM authoring: the model occasionally hallucinates a product into the wrong collection, asks for a variant option that a product does not expose \(e\.g\. “select a Color” on a gift card\), or names a filter dimension that has no realizable values in the target collection\. We close this gap by reusing the validator as a critic\. After each generation pass, the validator returns a list of per\-task issues; ShopGuru filters to the actionable subset \(every error\-severity finding plus the hallucination\-class warnings\) and constructs a polish prompt that lists*only the flagged tasks*alongside their specific issues and the shop context, and asks the LLM to regenerate*only those tasks*while preserving their IDs\. The polish loop runs for at most two rounds\. In our experiments, this is sufficient to drive the residual error rate to zero on most shops without human review\. When tasks still fail validation after two rounds, the build halts with a non\-zero exit code rather than silently shipping the flawed tasks; a human curator then either re\-runs generation with a different seed or hand\-authors replacements underdata\_sources/, which override the auto\-generated outputs in the next build\.

#### 2\.2\.4Post\-Generation Validation

A small dependency\-free validator runs over every emitted benchmark file as the final gate\. Its seven core rules \(Table[2](https://arxiv.org/html/2605.16116#A11.T2)\)—four error\-severity and three warning\-severity—target the failure modes we observed during multi\-shop benchmark regeneration; collectively they catch every class of grounding mismatch we encountered\. An actionable subset of the validator’s findings \(every error plus the two hallucination\-class warnings\) is also reused as the polish signal for the LLM\-authored journey generator \(§[2\.2\.3](https://arxiv.org/html/2605.16116#S2.SS2.SSS3)\), so the same audit logic both gates the output and steers the LLM toward feasible tasks; the remainingunknown\-pagewarning is advisory only\.

## 3Evaluating Sandbox Shop Quality

For the synthetic shops in the ShopArena environment to be a reasonable substitute for live storefronts, we want the sandbox shops to bestructurally and behaviorallysimilar to real websites\. Structural validation aims to evaluate the static structure of the synthetic shop against the real shops, whereas behavioral validation evaluates shopping workflows\.

To demonstrate and validate the ShopArena as a structurally and behaviorally aligned environment, we generate three synthetic sandbox shops spanning clothing, cookware and electronics hardware domains\. We refer to these shops assandbox shops\. The screenshots of the sandbox shops can be found in Section[E](https://arxiv.org/html/2605.16116#A5), along with the links to the live hosted sandbox shops\. We further generate three synthetic shops using real product data instead of synthetic data, which we refer to astwin shops\. When generating the twin shops, we use an additional visual verification agent that matches the visual theme and structure of the real storefronts\.

### 3\.1Structural Validity

To validate the structure of thesandbox shops,we compare them with real shops along two axes:Observation and Interaction Complexity, andState Transition Complexity\. We use 7 real shops and 3 synthetic shops for the comparisons\. We note that this is an unpaired validation\.

Observation/Interaction Complexity\. To compare the observation space, we use the depth of the simplified accessibility tree as a proxy for evaluating the nesting complexity of the websites222As defined by BrowserGym\. The nesting complexity evaluates how much structured content the agents must parse\. We compare the interaction complexity by counting the number of unique elements on the website that support thefillaction,clickaction, or elements that support select style controls \(choice\)\. Figure[5\(c\)](https://arxiv.org/html/2605.16116#S3.F5.sf3)shows that the synthetic sandbox shops are comparable in terms of observation and interaction complexity\.

State Transition Complexity\. To compare the state transition complexity, we treat the website as a directed state\-transition graph, where each node corresponds to a distinct UI state, as illustrated in Figure[5\(a\)](https://arxiv.org/html/2605.16116#S3.F5.sf1)\. Each pink node \(for example/\) denotes individual pages, and violet nodes \(/:navigation\) denote alternate UI configurations of those pages when interactive elements are opened\. Edges denote the actions that trigger transitions between the states, for instance, the red arrow shows clicking the navigation bar on/transitions the state to/:navigation\.

The aggregated statistics in Figure[5\(b\)](https://arxiv.org/html/2605.16116#S3.F5.sf2)show that the synthetic sandbox shops have a comparable number of nodes to the real shops, indicating similar coverage of distinct UI states\. However, they exhibit fewer edges and a lower average out\-degree, meaning that each state tends to have fewer outgoing transitions\. This reduction is largely because real storefronts contain additional navigation paths through external links, marketing pages, and auxiliary site components that are intentionally excluded from the sandbox environments\. Overall, the synthetic shops closely match real storefronts in observation complexity, action affordances, and transition\-graph scale, supporting their use as realistic sandbox environments\.

![Refer to caption](https://arxiv.org/html/2605.16116v1/x5.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.16116v1/x6.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.16116v1/x7.png)\(c\)

Figure 5:\(a\) An example directed graph visualization of the website; \(b\) the State Transition statistics; \(c\) the Observation/Interaction statistics
### 3\.2Behavioral Validation

![Refer to caption](https://arxiv.org/html/2605.16116v1/x8.png)\(a\)BrowserGym Harness\.
![Refer to caption](https://arxiv.org/html/2605.16116v1/x9.png)\(b\)Internal Harness\.

Figure 6:Success rates on real storefronts vs\. ShopArena twins\.We use thetwin shopsto perform behavioral validation of the ShopArena pipeline\. Thetwin shopsare visually and structurally similar to real shops, with the shop\-specific information such as names, locations, contact details, etc\. replaced with mock information\. We then generate the ShopGuru evaluation suite for the real and twin shops, and evaluate multiple frontier models on both variants\. We evaluate GPT\-5\-mini, GPT5\[[13](https://arxiv.org/html/2605.16116#bib.bib13)\], and Gemini 3 Flash\.

We evaluate the models using two harnesses: a BrowserGym\[[4](https://arxiv.org/html/2605.16116#bib.bib4)\]based implementation that uses accessibility trees \(AXTrees\), and a proprietary implementation that uses screenshots along with AXTrees\. We use GPT\-5 as a judge in both implementations, which analyzes the agent trajectories and provides a binary success rate\. The implementation details are provided in Appendix[B](https://arxiv.org/html/2605.16116#A2)\.

Figure[6](https://arxiv.org/html/2605.16116#S3.F6)shows that all models achieve similar performance on the real storefront and its twin variant\. Furthermore, the performance on the complex tasks improves with stronger models\. This shows that from the perspective of agent behavior, the sandbox shops are behaviorally consistent with real shops\.

## 4Validating ShopGuru task complexity

![Refer to caption](https://arxiv.org/html/2605.16116v1/x10.png)\(a\)BrowserGym Harness\.
![Refer to caption](https://arxiv.org/html/2605.16116v1/x11.png)\(b\)Internal Harness\.

Figure 7:Success rates on the synthetic sandbox shops\.To validate that the ShopGuru task generation pipeline can generate complex tasks, we evaluate frontier models with the ShopGuru evaluation suite on thesandbox shops\. We use the same web agent, harness, and judge setup as §[3\.2](https://arxiv.org/html/2605.16116#S3.SS2)\.

As seen in Figure[7\(a\)](https://arxiv.org/html/2605.16116#S4.F7.sf1)GPT\-5\-mini with the BrowserGym harness achieves a success rate of 47\.9% on the Long\-Horizon complex tasks, Gemini 3 Flash achieves 59%, and GPT\-5 achieves 62\.5%\. The results with the custom harness show similar performance across all models\. A low performance of frontier models shows that the ShopGuru task generation pipeline can generate meaningfully hard tasks which are useful for training and evaluation of next generation e\-commerce agents\.

## 5Conclusion

We presented ShopGym, an integrated framework for realistic simulation and scalable benchmarking of e\-commerce web agents\. ShopGym addresses a central methodological gap in current environments: live storefronts provide realism but are non\-stationary and irreproducible, while sandbox environments are controllable but limited in diversity and realism\. By combining ShopArena for sandbox shop generation and ShopGuru for grounded task synthesis, ShopGym produces evaluation artifacts that are self\-contained, inspectable, resettable, and stable across runs\.

Our results show that the generated shops are structurally comparable to real storefronts and preserve meaningful behavioral signal for agent evaluation\. In particular, agent performance on synthetic twins positively tracks performance on live storefronts, while fully synthetic composite shops yield benchmarks of comparable difficulty\. These findings suggest that ShopGym provides a practical middle ground between live\-web realism and sandbox control, and offers a scalable foundation for more reproducible and interpretable evaluation of e\-commerce web agents\.

## References

- Aggarwal et al\. \[2026\]Pranjal Aggarwal, Graham Neubig, and Sean Welleck\.Gym\-anything: Turn any software into an agent environment\.*arXiv preprint arXiv:2604\.06126*, 2026\.
- Castelo et al\. \[2026\]Alberto Castelo, Zahra Zanjani Foumani, Ailin Fan, Keat Yang Koay, Vibhor Malik, Yuanzheng Zhu, Han Li, Meysam Feghhi, Ronie Uliana, Shuang Xie, Zhaoyu Zhang, Angelo Ocana Martins, Mingyu Zhao, Francis Pelland, Jonathan Faerman, Nikolas LeBlanc, Aaron Glazer, Andrew McNamara, Lingyun Wang, and Zhong Wu\.Simgym: Traffic\-grounded browser agents for offline A/B testing in e\-commerce\.*CoRR*, abs/2602\.01443, 2026\.doi:10\.48550/ARXIV\.2602\.01443\.URL[https://doi\.org/10\.48550/arXiv\.2602\.01443](https://doi.org/10.48550/arXiv.2602.01443)\.
- Chae et al\. \[2026\]Hyungjoo Chae, Jungsoo Park, and Alan Ritter\.Safe and scalable web agent learning via recreated websites\.*CoRR*, abs/2603\.10505, 2026\.doi:10\.48550/ARXIV\.2603\.10505\.URL[https://doi\.org/10\.48550/arXiv\.2603\.10505](https://doi.org/10.48550/arXiv.2603.10505)\.
- de Chezelles et al\. \[2025\]Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F\. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados\.The browsergym ecosystem for web agent research\.*Transactions on Machine Learning Research*, 2025\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=5298fKGmv3](https://openreview.net/forum?id=5298fKGmv3)\.Expert Certification\.
- Deng et al\. \[2023\]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su\.Mind2web: Towards a generalist agent for the web\.In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,*Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023*, 2023\.URL[http://papers\.nips\.cc/paper\_files/paper/2023/hash/5950bf290a1570ea401bf98882128160\-Abstract\-Datasets\_and\_Benchmarks\.html](http://papers.nips.cc/paper_files/paper/2023/hash/5950bf290a1570ea401bf98882128160-Abstract-Datasets_and_Benchmarks.html)\.
- Gandhi and Neubig \[2026\]Apurva Gandhi and Graham Neubig\.Go\-browse: Training web agents with structured exploration\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=IpzRWE52yw](https://openreview.net/forum?id=IpzRWE52yw)\.
- He et al\. \[2024\]Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu\.Webvoyager: Building an end\-to\-end web agent with large multimodal models\.In Lun\-Wei Ku, Andre Martins, and Vivek Srikumar, editors,*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024*, pages 6864–6890\. Association for Computational Linguistics, 2024\.doi:10\.18653/V1/2024\.ACL\-LONG\.371\.URL[https://doi\.org/10\.18653/v1/2024\.acl\-long\.371](https://doi.org/10.18653/v1/2024.acl-long.371)\.
- Koh et al\. \[2024\]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po\-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried\.Visualwebarena: Evaluating multimodal agents on realistic visual web tasks\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024*, 2024\.URL[https://arxiv\.org/abs/2401\.13649](https://arxiv.org/abs/2401.13649)\.
- Liu et al\. \[2023\]Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics*, 12:157–173, 2023\.URL[https://api\.semanticscholar\.org/CorpusID:259360665](https://api.semanticscholar.org/CorpusID:259360665)\.
- Lu et al\. \[2025\]Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Jiri Gesi, Yaochen Xie, Qi He, Dakuo Wang, et al\.Can llm agents simulate multi\-turn human behavior? evidence from real online customer behavior data\.*arXiv preprint arXiv:2503\.20749*, 2025\.
- Lyu et al\. \[2025\]Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, and Xiuyi Chen\.Deepshop: A benchmark for deep research shopping agents\.*ArXiv*, abs/2506\.02839, 2025\.URL[https://api\.semanticscholar\.org/CorpusID:279118560](https://api.semanticscholar.org/CorpusID:279118560)\.
- Peeters et al\. \[2025\]Ralph Peeters, Aaron Steiner, Luca Schwarz, Julian Yuya Caspary, and Christian Bizer\.Webmall–a multi\-shop benchmark for evaluating web agents \[technical report\]\.*arXiv preprint arXiv:2508\.13024*, 2025\.
- Singh et al\. \[2025\]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El\-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al\.Openai gpt\-5 system card\.*arXiv preprint arXiv:2601\.03267*, 2025\.
- Sinha et al\. \[2025\]Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping\.The illusion of diminishing returns: Measuring long horizon execution in llms\.*ArXiv*, abs/2509\.09677, 2025\.URL[https://api\.semanticscholar\.org/CorpusID:281252776](https://api.semanticscholar.org/CorpusID:281252776)\.
- Wang et al\. \[2025a\]Dakuo Wang, Ting\-Yao Hsu, Yuxuan Lu, Hansu Gu, Limeng Cui, Yaochen Xie, William Headean, Bingsheng Yao, Akash Veeragouni, Jiapeng Liu, et al\.Agenta/b: Automated and scalable web a/btesting with interactive llm agents\.*arXiv preprint arXiv:2504\.09723*, 2025a\.
- Wang et al\. \[2025b\]Jiang Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jiandong Zhang, and Xiaoyi Zeng\.Shoppingbench: A real\-world intent\-grounded shopping benchmark for llm\-based agents\.In*AAAI Conference on Artificial Intelligence*, 2025b\.URL[https://api\.semanticscholar\.org/CorpusID:280536823](https://api.semanticscholar.org/CorpusID:280536823)\.
- Wang et al\. \[2026a\]Jiangyuan Wang, Kejun Xiao, Huaipeng Zhao, Tao Luo, and Xiaoyi Zeng\.Productresearch: Training e\-commerce deep research agents via multi\-agent synthetic trajectory distillation\.*CoRR*, abs/2602\.23716, 2026a\.doi:10\.48550/ARXIV\.2602\.23716\.URL[https://doi\.org/10\.48550/arXiv\.2602\.23716](https://doi.org/10.48550/arXiv.2602.23716)\.
- Wang et al\. \[2026b\]Pei Wang, Yanan Wu, Xiaoshuai Song, Weixun Wang, Gengru Chen, Zhongwen Li, Ke Yan, Ken Deng, Qi Liu, Shu\-Man Zhao, Shaopan Xiong, Xuepeng Liu, Xuefeng Chen, Wanxi Deng, Wenbo Su, and Bo Zheng\.Shopsimulator: Evaluating and exploring rl\-driven llm agent for shopping assistants\.*ArXiv*, abs/2601\.18225, 2026b\.URL[https://api\.semanticscholar\.org/CorpusID:285050373](https://api.semanticscholar.org/CorpusID:285050373)\.
- Wang et al\. \[2025c\]Ziyi Wang, Yuxuan Lu, Wenbo Li, Amir A\. Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia\-Jun Li, Lydia B\. Chilton, and Dakuo Wang\.Opera: A dataset of observation, persona, rationale, and action for evaluating llms on human online shopping behavior simulation\.*ArXiv*, abs/2506\.05606, 2025c\.URL[https://api\.semanticscholar\.org/CorpusID:279244562](https://api.semanticscholar.org/CorpusID:279244562)\.
- Wang et al\. \[2025d\]Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Jing Huang, and Dakuo Wang\.Customer\-r1: Personalized simulation of human behaviors via rl\-based llm agent in online shopping\.*arXiv preprint arXiv:2510\.07230*, 2025d\.
- Xue et al\. \[2025\]Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su\.An illusion of progress? assessing the current state of web agents\.*CoRR*, abs/2504\.01382, 2025\.doi:10\.48550/ARXIV\.2504\.01382\.URL[https://doi\.org/10\.48550/arXiv\.2504\.01382](https://doi.org/10.48550/arXiv.2504.01382)\.
- Yao et al\. \[2022\]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan\.Webshop: Towards scalable real\-world web interaction with grounded language agents\.In Sanmi Koyejo, S\. Mohamed, A\. Agarwal, Danielle Belgrave, K\. Cho, and A\. Oh, editors,*Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022*, 2022\.URL[http://papers\.nips\.cc/paper\_files/paper/2022/hash/82ad13ec01f9fe44c01cb91814fd7b8c\-Abstract\-Conference\.html](http://papers.nips.cc/paper_files/paper/2022/hash/82ad13ec01f9fe44c01cb91814fd7b8c-Abstract-Conference.html)\.
- Yu et al\. \[2026\]Zijian Yu, Kejun Xiao, Huaipeng Zhao, Tao Luo, and Xiaoyi Zeng\.Shopping companion: A memory\-augmented LLM agent for real\-world e\-commerce tasks\.*CoRR*, abs/2603\.14864, 2026\.doi:10\.48550/ARXIV\.2603\.14864\.URL[https://doi\.org/10\.48550/arXiv\.2603\.14864](https://doi.org/10.48550/arXiv.2603.14864)\.
- Yuan et al\. \[2026\]Peng Yuan, Yuyang Yin, Yuxuan Cai, and Zheng Wei\.Webforge: Breaking the realism\-reproducibility\-scalability trilemma in browser agent benchmark\.*arXiv preprint arXiv:2604\.10988*, 2026\.
- Zhang et al\. \[2024\]Shuo Zhang, Boci Peng, Xinping Zhao, Boren Hu, Yun Zhu, Yanjia Zeng, and Xuming Hu\.Llasa: Large language and e\-commerce shopping assistant\.*CoRR*, abs/2408\.02006, 2024\.doi:10\.48550/ARXIV\.2408\.02006\.URL[https://doi\.org/10\.48550/arXiv\.2408\.02006](https://doi.org/10.48550/arXiv.2408.02006)\.
- Zhang et al\. \[2025\]Yimeng Zhang, Jiri Gesi, Ran Xue, Tian Wang, Ziyi Wang, Yuxuan Lu, Sinong Zhan, Huimin Zeng, Qingjun Cui, Yufan Guo, et al\.See, think, act: Online shopper behavior simulation with vlm agents\.*arXiv preprint arXiv:2510\.19245*, 2025\.
- Zhou \[2026\]Shuyan Zhou\.Webarena\-infinity: Generating browser environments with verifiable tasks at scale\.*shuyanzhou\.com*, March 2026\.URL[https://webarena\.dev/webarena\-infinity/](https://webarena.dev/webarena-infinity/)\.
- Zhou et al\. \[2024\]Shuyan Zhou, Frank F\. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig\.Webarena: A realistic web environment for building autonomous agents\.In*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\. OpenReview\.net, 2024\.URL[https://openreview\.net/forum?id=oKn9c6ytLx](https://openreview.net/forum?id=oKn9c6ytLx)\.

## Appendix ARelated Work

##### Static web environments and benchmarks\.

Prior work on web\-agent evaluation has pursued two goals: broad coverage of real websites and reproducible executable environments\. Mind2Web\[[5](https://arxiv.org/html/2605.16116#bib.bib5)\]provides large\-scale offline traces from 137 real websites across 31 domains, emphasizing breadth and cross\-site generalization\. OPeRA\[[19](https://arxiv.org/html/2605.16116#bib.bib19)\]similarly collects observations, personas, rationales, and actions from real human shopping sessions on Amazon\. In contrast, WebArena\[[28](https://arxiv.org/html/2605.16116#bib.bib28)\]provides self\-hosted, fully functional websites with executable tasks, and VisualWebArena\[[8](https://arxiv.org/html/2605.16116#bib.bib8)\]extends this setting to visually grounded tasks\. Such environments support autonomous exploration and reproducible evaluation, unlike purely offline datasets\. Several benchmarks focus specifically on e\-commerce\. WebShop\[[22](https://arxiv.org/html/2605.16116#bib.bib22)\]offers a synthetic storefront with 1\.25M products\. ShoppingBench\[[16](https://arxiv.org/html/2605.16116#bib.bib16)\]provides a large shopping sandbox with API\-based interaction, aimed primarily at intent grounding\. ShopSimulator\[[18](https://arxiv.org/html/2605.16116#bib.bib18)\]introduces a Chinese web environment for multi\-turn shopping dialog and fine\-grained product differentiation\. WebMall\[[12](https://arxiv.org/html/2605.16116#bib.bib12)\]expands to four simulated shops with authentic product offers and comparison\-shopping tasks\. Although these benchmarks enable controlled evaluation, they remain tied to a fixed set of layouts, taxonomies, and policies, and therefore capture only a limited slice of real\-world e\-commerce diversity\.

Live\-website evaluation\.To address this limitation, Online\-Mind2Web\[[21](https://arxiv.org/html/2605.16116#bib.bib21)\], WebVoyager\[[7](https://arxiv.org/html/2605.16116#bib.bib7)\], and DeepShop\[[11](https://arxiv.org/html/2605.16116#bib.bib11)\]evaluate agents directly on live websites, including real shopping domains\. These settings improve diversity and realism, but depend on external sites whose layouts, inventories, flows, and access policies may change without notice\. This non\-stationarity makes comparisons over time difficult and limits reproducibility, especially for RL training\.

Automated environment and task generation\.A recent line of work uses LLMs to automate environment construction\. Gym\-Anything\[[1](https://arxiv.org/html/2605.16116#bib.bib1)\]treats setup as a multi\-agent pipeline that installs and configures existing software, producing CUA\-World with 10K\+ tasks\. WebArena\-Infinity\[[27](https://arxiv.org/html/2605.16116#bib.bib27)\]generates synthetic web apps from user manuals\. WebForge\[[24](https://arxiv.org/html/2605.16116#bib.bib24)\]uses a Plan/Generate/Refine/Validate pipeline to create self\-contained web environments at scale, and VeriEnv\[[3](https://arxiv.org/html/2605.16116#bib.bib3)\]constructs executable synthetic versions of real websites with deterministic rewards exposed through a Python SDK\. These approaches scale well, but often sacrifice fidelity: their environments are derived from manuals or ungrounded LLM design choices and may miss the layouts, integrations, and edge cases of production storefronts\. In contrast, ShopArena grounds generation in observed real shops: it first explores a live storefront to produce an anonymised Shop Manual capturing taxonomy, navigation, capabilities, and statistics, then synthesises a calibrated sandbox shop from that manual\. This preserves much more of the diversity of real storefronts while retaining determinism and reset\-ability\. Go\-Browse\[[6](https://arxiv.org/html/2605.16116#bib.bib6)\]proposes a multi\-agent pipeline for generating tasks and trajectories by navigating websites and maintaining a graph of visited URLs for efficient exploration\.

## Appendix BImplementation Details

### B\.1Models used for agent implementations

- •Planning Agent: Claude Code with Claude Opus 4\.6
- •Specification Agent: Claude Code with Claude Opus 4\.6
- •Collections generator: Claude Opus 4\.6
- •Product name and description generator: Claude Sonnet 4\.6
- •Product Image Generator: Gemini 3 Pro
- •Execution Agent: Claude Code with Claude Opus 4\.6
- •Visual Verification Agent: Claude Code with Claude Opus 4\.6, enabled with Playwright for browser automation

### B\.2Harness A: BrowserGym Agent

Observation\. The BrowserGym harness uses the accessibility tree as the observation\. We do not use screenshots for simplicity\.

Action Space\. The agent is allowed to use 20 actions:

- •noop: Do nothing, optionally waiting for the given number of milliseconds\.
- •send\_msg\_to\_user: Send a message to the user \(terminal action that ends the episode\)\.
- •scroll: Scroll the page horizontally and vertically by the given pixel deltas\.
- •fill: Type a value into an<input\>,<textarea\>, or\[contenteditable\]element\.
- •select\_option: Select one or multiple options in a<select\>element by value or label\.
- •click: Click an element with a given mouse button and optional modifier keys\.
- •dblclick: Double\-click an element with a given mouse button and optional modifier keys\.
- •hover: Move the mouse over an element to hover it\.
- •press: Focus an element and press a key combination \(e\.g\.Control\+Enter,ArrowDown\)\.
- •focus: Move keyboard focus to the matching element\.
- •clear: Clear the contents of an input field\.
- •drag\_and\_drop: Drag one element and drop it onto another\.
- •upload\_file: Click an element to open a file chooser and upload one or more files\.
- •go\_back: Navigate back to the previous page in browser history\.
- •go\_forward: Navigate forward to the next page in browser history\.
- •goto: Navigate the current tab to the given URL\.
- •tab\_close: Close the currently active tab\.
- •tab\_focus: Activate the tab at the given index, bringing it to the front\.
- •new\_tab: Open a new tab and make it the active one\.
- •report\_infeasible: Declare the task impossible with a short reason \(terminal action that ends the episode\)\.

Browser\. We use a headless chromium browser for all experiments with BrowserGym, with Playwright for browser interaction\.

Trajectory details\. The BrowserGym agent uses a think \- action \- memory loop, where only the observation from the current step is used as context\. The agent is instructed to reason first using<think\>\.\.\.</think\>tags, output a single action in<action\>\.\.\.</action\>tags, and write whatever it thinks is necessary to solve the task in<memory\>\.\.\.</memory\>tags\. The think, action and memory tags from all the steps are used as context in the future steps, along with any execution errors\. The agent is allowed a maximum of 30 steps, and the episode terminates when the agent sends a message to the user using thesend\_msg\_to\_useraction, reports the task as infeasible using thereport\_infeasibleaction, or the step threshold is reached\. Episodes that reach the step budget are graded as failed\.

### B\.3Harness B: Internal Harness

The internal harness implements a single\-call\-per\-step planner–executor agent that runs over a textual page projection and emits one Playwright\-grounded action per step\. Each task is evaluated across several independent browser sessions to absorb agent and storefront stochasticity\.

##### Observation\.

At each step, the current page is rendered into a tree\-shaped textual projection that fuses the rendered DOM and the accessibility tree into a single representation\. Every interactive element \(clickable control, text input, combobox, menu item, etc\.\) is tagged with a stable identifier of the form\[elem\-N\]that the agent uses to reference the element in its action\. For all experiments in this paper the agent additionally receives a viewport screenshot of the current page at each step, so the observation is multimodal \(page projection plus rendered pixels\)\. The agent prompt also includes the current URL and a running*memory*: a per\-step textual log of prior decisions, action outcomes \(success / error strings\), and the agent’s own analysis\. The full LLM transcript is discarded at the end of each step — only the memory summary, the fresh page projection, and \(optionally\) the latest screenshot survive into the next prompt — so context length grows linearly in*step count*, not in cumulative LLM output\.

##### Action space\.

Each step’s LLM call is constrained by a fixed structured\-output schema with the following fields:

- •analysis— free\-form chain\-of\-thought over the current state, prior memory, and the task goal\.
- •shouldEnd— boolean flag terminating the rollout\.
- •instruction— a short natural\-language description of the intended next action \(e\.g\. “click on the filter menu titled*Price*”, “type*lunch box for kids*in the header search field”\)\.
- •elementId— theelem\-Ntarget of the action, referencing one of the IDs in the current page projection\.
- •method— a Playwright interaction method \(click,fill,selectOption,press,check, etc\.\)\.
- •arguments— an optional list of method arguments \(e\.g\. the value to type into a text field\)\.

The harness translates the \(elementId,method,arguments\) tuple into a Playwright call against the live browser session\. Failures \(selector misses, navigation errors, timeouts\) are surfaced back to the agent as an error string in the next step’s prompt and recorded verbatim in memory\.

##### Termination and budgets\.

A rollout terminates on \(i\)shouldEnd=trueemitted by the agent, \(ii\) hitting the per\-task step budget of4040steps, or \(iii\) hitting the per\-task wall\-clock budget of850850s\. Rollouts that exit via the step or time budget are scored as failures regardless of the cart state at termination \(§[B\.4](https://arxiv.org/html/2605.16116#A2.SS4)\), since the agent itself did not declare goal completion\.

##### Browser\.

The agent runs against a remote, Playwright\-compatible browser session provisioned per task\. Each session is initialised at the task’s starting URL \(\*\_realfor the live source storefront or\*\_twinfor the ShopArena sandbox\), accepts a*cart\-only*terminal\-state convention \(every shopping task in ShopGuru ends with the agent ending the session once the cart contains the requested items, never by clicking a Checkout button\), and is torn down at the end of the rollout\.

##### LLM client and models\.

All LLM calls use an OpenAI\-compatible chat\-completion API with structured\-output \(JSON\) decoding\. We evaluate three models spanning compact and flagship capability tiers: GPT\-5 mini, Gemini 3 Flash, and GPT\-5\. Where the underlying model exposes it, reasoning effort is set tomedium\. Each LLM call is retried up to three times on transient API errors before the rollout is marked failed\.

##### Repeats and aggregation\.

We run three independent rollouts per \(task, model, environment\) cell\. Per\-task verdicts are averaged within each cell, and the error bars in Figs\.[6](https://arxiv.org/html/2605.16116#S3.F6)b and[7](https://arxiv.org/html/2605.16116#S4.F7)b span the per\-cell standard error of the mean across the three repeats\. Per\-bundle pass rates aggregate across all tasks in the bundle \(easy\_short\_horizonorhard\_long\_horizon\) and all repeats\.

### B\.4LLM\-as\-Judge

Both harnesses score every rollout with the same LLM\-as\-judge model but slightly different protocols\. After each rollout, the judge LLM \(GPT\-5\) receives:

- •the original task intent and the task’ssuccess\_criteriaJSON \(URL hint plus any cart/response constraints\);
- •the agent’s per\-step memory log \(textual summary of decisions and action outcomes\);
- •the agent’s URL trajectory \(every URL visited, in order\);
- •the rollout’sterminationType\(goal\_reached,steps\_limit\_reached, ortime\_limit\_reached\)\.

The judge prompt instructs the model to weigh task satisfaction first, followed by output quality, tool\-use effectiveness, and recovery from transient errors; it returns a structured JSON object with a binarysuccessfield and a free\-formreasoningexplanation\. The verdict is then post\-processed by a hard rule: rollouts that did not terminate via the agent’s ownshouldEndare forced tosuccess=falseregardless of the judge’s opinion\. This guards against a class of spurious successes where the agent times out on a page whose final state happens to look correct \(e\.g\. the requested product is on the page but was never added to cart\) but never emits a deliberate end\-of\-session signal\.

With the BrowserGym harness,url\_containsfield is treated as a hard gate, where the trajectory is graded as failed if the expected URL is not present in the trajectory\. The judge LLM receives the entire cart history, the agent’s reasoning, action and memory traces, as well as the screenshots of the webpages in the trajectory and outputs a binary pass/fail decision along with it’s reasoning\.

### B\.5Infrastructure Requirements

We perform all evaluations using the official APIs from the model providers\. To parallelize the evaluation suite, the agent harnesses are orchestrated on a standard cloud virtual machine\.

## Appendix CLimitations

ShopGym improves control, reproducibility, and scalability, but it is not a full substitute for live\-web evaluation\. By construction, ShopArena removes many sources of real\-world non\-stationarity, including inventory churn, personalization, A/B tests, third\-party integrations, latency variation, and other operational noise\. This is desirable for controlled benchmarking, but it also means that strong performance in ShopGym should be interpreted as evidence of capability under stabilized conditions, not as a guarantee of robustness on live storefronts\. More broadly, the generated shops are behaviorally aligned approximations rather than exact reconstructions, so some fine\-grained brand\-specific, visual, or platform\-specific cues are intentionally not preserved\.

Our validation metrics are proxies for realism: the structural analysis uses accessibility\-tree depth, interaction counts, and transition\-graph statistics, which capture meaningful aspects of interface and navigation complexity, but do not fully measure semantic ambiguity, visual fidelity, content quality, or all sources of user\-facing behavioral variation\. Likewise, the behavioral validation shows that model performance on twin shops positively tracks performance on live storefronts, but correlation in aggregate success rate does not imply that the same latent capabilities or failure modes are being exercised in exactly the same way\.

Finally, the current task suite covers only part of the e\-commerce agent problem\. ShopGuru focuses on product discovery, browsing, filtering, information seeking, and multi\-step in\-store shopping journeys\. It does not yet cover several important settings, including login\-protected flows, checkout and payment, recommendation modules and sponsored content, customer\-service interactions, cross\-store comparison, post\-purchase workflows, multi\-session memory, multilingual settings, or safety\- and policy\-critical edge cases\. Extending the framework to these settings is an important direction for future work\.

## Appendix DBroader Impact and Safeguards

ShopGym is intended to improve the reproducibility, controllability, and inspectability of e\-commerce web\-agent evaluation\. By converting live storefronts into sandboxed evaluation environments, the framework reduces reliance on non\-stationary live websites and enables more reliable comparison across agents\.

Potential positive impact\. ShopGym is a framework for constructing e\-commerce simulation environments and grounded benchmark tasks\. The simulation layer, ShopArena, provides a framework for creating diverse, stable and realistic simulation environments from live storefronts\. Beyond evaluations, the environments can be used for any agent simulations, such as training of e\-commerce agents using Reinforcement Learning\.

Potential negative impact\. We nevertheless identify several potential negative broader impacts\. First, progress on web\-agent evaluation may indirectly contribute to stronger agents that could be misused for unauthorized browsing automation, large\-scale scraping, spammy shopping behavior, inventory manipulation, or deceptive commercial activity\. Second, because the pipeline uses live storefronts during the exploration phase, careless use could raise privacy, copyright, fair\-use, or terms\-of\-service concerns if raw source\-store content, identifiable commercial information, or original assets were copied or redistributed\. Third, generated benchmarks may not be representative of all e\-commerce websites, merchant categories, languages, geographic regions, accessibility patterns, or user populations\. Claims about benchmark diversity should therefore be interpreted relative to the seed storefronts and task families used to instantiate a particular ShopGym benchmark\.

Mitigations\. We mitigate these risks through several safeguards\. ShopGym is designed as a sandboxed research environment rather than a tool for acting on live commercial websites\. Generated shops are self\-contained, locally hostable, resettable, and inspectable\. Login is disabled and checkout is mocked, so the benchmark does not support real purchases, account access, or interaction with real customers\. During exploration, the pipeline extracts public, user\-facing structural information such as navigation affordances, catalog statistics, page organization, and policy\-page structure\. Downstream generation uses anonymized specifications and does not preserve brand names, product names, named individuals, contact details, or absolute URLs\. Released benchmark artifacts use synthetic brand identities, products, descriptions, and imagery rather than direct copies of source\-store assets\.

We will document the intended use, limitations, task schema, environment setup, dependency versions, and licenses of released artifacts\. We do not release raw source\-store data or original commercial assets\. Users who instantiate ShopGym on additional storefronts should respect applicable website terms, robots policies where relevant, copyright restrictions, and regional legal requirements\. These safeguards are intended to support responsible research use while reducing the risk that the framework is used for unauthorized live\-web automation or redistribution of commercial content\.

## Appendix EShopArena Generation Artifact Example

This appendix demonstrates the sandbox shops generated by the ShopArena shop generation pipeline\. These sandbox shops are built from exploration artifacts generated from ShopArena exploration tasks\.

![Refer to caption](https://arxiv.org/html/2605.16116v1/x12.png)\(a\)Mock Clothing
![Refer to caption](https://arxiv.org/html/2605.16116v1/x13.png)\(b\)Mock Hardware
![Refer to caption](https://arxiv.org/html/2605.16116v1/x14.png)\(c\)Mock Cookware

Figure 8:Screenshots of Sandbox Shop HomepagesFigure[9](https://arxiv.org/html/2605.16116#A5.F9)shows more detailed interaction surfaces within the generated sandbox shops\. Built from specifications, the examples include a collection page with faceted filters, a homepage with a promotional popup, and a product detail page with search suggestions and purchasing controls\. These screenshots highlight that the generated shops contain not only realistic visual layouts, but also functional e\-commerce elements that support browsing, filtering, promotion handling, search, and product\-level decision making\.

![Refer to caption](https://arxiv.org/html/2605.16116v1/x15.png)\(a\)Collection page with filters
![Refer to caption](https://arxiv.org/html/2605.16116v1/x16.png)\(b\)Homepage with promotion popup
![Refer to caption](https://arxiv.org/html/2605.16116v1/x17.png)\(c\)Product detail page

Figure 9:Screenshots of Sandbox Shop DetailsLive versions of the sandbox shops can be accessed at:

- •
- •
- •

## Appendix FShopArena Exploration Artifact Example

This appendix reproduces the three top\-level outputs thatshop\_arena\.explorewrites for a single live storefront exploration run\. The example was produced from a premium hybrid cookware storefront; all store, brand, designer, product, and URL strings were anonymized at write time per the explorer’s constitution, so only structural and behavioral facts remain\.

### F\.1Shop Manual \(manual\.md\)

The manual instructs the generation pipeline to implement the cookware shop by dividing it into 7 features:Site Shell,Homepage,Collections & Navigation,Cart,Search,Floating UX,Policy & info pages\.

\#ShopManual\-\-PremiumHybridCookwareandKitchenAccessoriesShop

\#\#Overview

ThisisapremiumhybridcookwareandkitchenaccessoriesstorefrontpricedinUSD\.Thebrandvoiceisbold,chef\-endorsed,anddirect\-to\-consumer,leaningonculinary\-professionalcredibilityanda"buyonce"durabilitynarrative\.Aggregatereviewcountsinthe50,000\+rangeandastatedlifetimewarrantysignalasizeable,establishedcatalogratherthananicheboutique\.

\#\#Siteshell

Apersistentdismissibleannouncementbarsitsaboveatwo\-rowstickyheader\.Row1isautilitybar:aleftclusterofthreeeditorialtextlinksplusa"Partners"dropdownbuttonlisting2\-3namedcollaboratorpages,acenteredlogo,andarightclusterofsearch,account,andcarticonbuttons\.Row2istheprimarycategorynavigation\-\-eighttop\-levelentriesrenderedasinlinebuttons\.Mostentriesopenflyoutpanelsratherthanasinglemegamenu,sonavigationdepthtopsoutattwolevels\(top\-level\-\>sub\-link\)\.Eachflyoutisafull\-widthtwo\-columnpanel:averticalsub\-linklistendingin"ShopAll\[Category\]"ontheleft,plus2\-3featuredproductcardsontheright\(image,name,pricewithoptionalcompare\-at,savingsbadge,andstarrating\)\.Onetop\-levelentrybehavesasadirectlinkinsteadofopeningaflyout\.

Amobileoff\-canvasdrawermirrorsthesametaxonomyasaccordions,withextradirectlinksforsale,giftcards,"ShopAll",andthePartnersgroup\.

Thefooteroccupiesa\`contentinfo\`landmarkoneverypageandissplitintoatopbandandabottomband\.Thetopbandhasthreenamedlinkgroups\(Shop,Help,Media\)plusafourthcolumnoffoursocialiconlinksopeninginnewtabs\.Thebottombandisaflexrowpairingacopyrightlinewithtwolegallinks\(termsofservice,accessibilitycommitment\)againstabrandwordmarkimage\.Thefootercarriesnoinlinenewsletterform,nopaymenticons,andnolocaleorcurrencyswitcher\.

\#\#Homepage

Thehomepageisalong,modularcompositionofsixteensectionsinthisorder:

\-Full\-widthherowithautoplaymutedloopingbackgroundvideoandacenteredheadline\+singleCTAoverlay\.

\-"ShopbyCategory"chipslider\-\-ahorizontallyscrollingrowofsevenimage\-backedsub\-collectionchipswithprev/nextarrowsanda"ShopAll"link\.

\-Featuredsetspromoblock\-\-text\-left/product\-cards\-right,witha3\-cardrowshowingimage,savingsbadge,name,currentprice,andoriginalprice\.

\-Specialtyproductbanner\-\-areversedtwo\-columnblockpairingtwoproductcardsontheleftwithdescriptivecopyontherightintroducingalimited\-timesub\-line\.

\-Chefendorsementblock\-\-autoplayvideothumbnailwithaplayoverlayopeningfullscreenplayback,pairedwithanattributedblockquoteandastar\+aggregate\-review\-countsocial\-proofline\.

\-2x2featurecardgrid\-\-fourcardscoveringhigh\-heatcapability,PFAS\-freenonstick,dishwashersafety,anddurability/warranty,eachwithasmallattributebadge,anall\-capsheadline,andaone\-totwo\-sentencedescriptor\.

\-Reviewsbanner\-\-aggregaterating\+rotatingcustomerquoteononeside,athree\-icontruststrip\(lifetimewarranty,freeshipping,30\-dayreturns\)ontheother\.

\-"BestSellers"horizontalproductcarouselwithprev/nextarrowsanda"ShopAll"link\.

\-Knifecategoryherowithtwoproductcardsontheleftandacategoryvalue\-prop\+customerquotechipontheright\.

\-UGCvideocarousel\-\-sixvideothumbnailtileswithplayoverlays;tilesopenafullscreenpopupplayer\.

\-3\-upspecialtycategorybanners\-\-threeequal\-widthcardseachcombiningaproductimage,headline,descriptor,CTAlink,andacustomerquotechipwithstarrating\.

\-Culinarycouncilsectionintroducingthebrand'schefadvisorygroupwithfiveprofessionalnames\(rolesonly\)andaCTAtoadedicatedpage\.

\-"Recipes"carouselofrecipecardswithimage,title,yield,andtime,linkingintotherecipesblog\.

\-"Articles"carouselofeditorialblogcardsusingthesamecardchromeminustherecipemetadata\.

\-Foundermessage\-\-ashorttwo\-paragraphfirst\-personnotesignedwithaplain\-textname\.

\-Inlinenewslettersectionatthepagebottomwithanemailinput,hiddenhoneypot,and"Subscribe"button\.

Thereisnohomepagemodalpopup\.AseparateKlaviyo\-drivenpopupisconfiguredbutdidnotfireduringexploration\.

\#\#Collections&navigation

Collectionpagesrenderasathree\-columngridondesktop\.Eachcardshowsaproductimage,anoptionalpromotionaloverlaybadge\("SALE","New","Bestseller"\),producttitle,starratingwithreviewcount,andprice\(strikethroughoriginalpriceplusapercentage\-offbadgewhendiscounted\)\.Thecollectiontitledisplaysaliveproductcountinparentheses\.

Abovethegridsitsahorizontalfilterbarwithtwocollapsiblecontrols\.The"Filter"panelexposestwocheckboxfacets\-\-"Available"and"OnSale"\-\-thatupdateURLparametersimmediately,renderserver\-side,andcombinecleanly\.The"Sortby"dropdownofferseightoptions:featured,bestselling,alphabeticallyA\-Z,alphabeticallyZ\-A,priceascending,pricedescending,datenew\-to\-old,anddateold\-to\-new\.Sortingupdatesthe\`sort\_by\`URLparameterwithoutafullreload\.

Paginationusesa"Loadmore"button\.Theinitialrendershows24cards;clickingthebuttonappendstherestofthecollectioninasinglebatchandthebuttonthendisappears\.Thereisnoinfinitescroll,nonumberedpagination,andno\`page=\`URLparameter\.

\#\#Productpage

Theproductpageleadswithahorizontalimagecarouselthathasprev/nextarrowcontrolsandanoptionalleft\-sidethumbnailstripfordirectslideselection\.Belowthetitlesitsthepriceblock:prominentcurrentprice,optionalstrikethroughoriginalprice,optionalpercentagesavingsbadge,andabuy\-now\-pay\-laterfinancinglinethatopensamodal\.Atrustbadgerowbeneaththepricesummarizesfreeshipping,a30\-dayreturnguarantee,and\(whereapplicable\)alifetimewarranty,plusanexpandableclaimbuttonlistingchemical\-safetyattributesfornonstickitems\.

Variantselectorsrenderasradiobuttonsinasinglehorizontalrow\.Apparelexposesasizeaxis\(XS\-XXL\)witha"SizeGuide"linkabovethegroup\.Single\-variantproductsskiptheselectorentirely\.Aquantitystepper\(decrement,numericdisplay,increment\)sitsimmediatelyadjacenttothe"AddtoCart"CTA\.

Descriptionsvarybycategory\.Simplerproductsrenderaninlineproseparagraphplusabulletedfeaturelist\.Complexproductsuseanaccordion\-tablayout\(e\.g\."Description"\+"HybridTechnology"tabs\)followedbyahorizontalseven\-iconattributestrip\(warranty,utensilsafety,nonstick,oven\-safe,dishwasher\-safe,induction\-ready,stay\-coolhandle\)\.SomePDPsincludea"GiftWithPurchase"calloutabovethequantitysteppershowingabundledfreeitem\.

ReviewsuseanOkendo\-stylewidget\.Productswithoutreviewsshowa"Bethefirsttoleaveareview"promptanda"Writeareview"CTA\.Productswithreviewsshowanaggregatescore,aclickablestar\-distributionbreakdown,anauto\-generatedAIsummaryblock,searchandsort/filterdropdowns\(latest,oldest,highest,lowest,mosthelpful;rating1\-5orall\),andindividualreviewcardswithreviewername,starrating,relativedate,headline,body,andhelpful\-votebuttons\.

Recommendationrailsdifferbytemplate\.SomePDPsrendera"Customersalsolove"railofproductcardswithquick\-addbuttonsandinlinecolor\-swatchselectors\.Othersrendertwostackedrails\-\-a"FrequentlyBoughtTogether"triowithinline"AddtoCart"buttons,anda"RecentlyViewed"carouselwithprev/nextarrows\.Nocustomizationorpersonalizationfields\(engraving,monogram,giftnote\)areexposedonthePDP\.

\#\#Cart

Thecartopensasaright\-sideslide\-outdrawer\(ARIA\`dialog\`labeled"YourCart"\)triggeredbytheheadercarticon\.Theiconshowsanumericitem\-countbadgeoncethecarthasanyitems\.

Emptystateshowsthe"YourCart"headingwitha\`0\`badge,ashortempty\-cartsentence,aCTAlinkintoabest\-sellerscollection,a\`$0\.00\`subtotalline,andthesametrust\-badgetrio\(lifetimewarranty,freeshipping,30\-dayreturns\)repeatednearthefooter\.

Filledstatereplacestheemptycopywithafree\-shippingqualifierbanneroncethethresholdismet\.Eachlineitemshowsathumbnail,name\(bothlinkedtothePDP\),thepriceblock\(currentpriceplusstrikethroughcompare\-atwhendiscounted\),a"RemoveItem"affordance,andathree\-buttonquantitystepper\.QuantitychangesapplyviaAjaxandimmediatelyrecomputethreesummaryrows:productsatlistprice,totalsavings,andsubtotal\.Afull\-width"Checkout"buttonsitsbelow,withafootnoteasteriskforfree\-shippingeligibilityandthetrustbadgesrepeated\.

Thedrawercarriesa"FrequentlyBoughtTogether"upsellrail\(threeproductcardswithinline"AddtoCart"\)inbothstates;carteditemsareswappedoutoftherailforfreshsuggestions\.A"RecentlyViewed"railwithprev/nextarrowsappearsbeneath\.Thereisnoinlinepromo\-codefield\-\-discountcodesarecollectedonthenativecheckout\-\-andthereisnoinlineshippingestimatecalculator;free\-shippingeligibilityisconveyedonlyviathequalifierbannerandfootnote\.

\#\#Search

Searchistriggeredbyamagnifying\-glassiconbuttonintheheaderutilityrow\.Clickingitexpandsaninlinedropdowndirectlybelowtheicon\-\-thepageisnotdimmedandnavigationremainsvisible\.

Predictivesearchrendersastatic"Suggested"panelinsidethedropdown:typicallyfourlinkscombiningonecollectionshortcutandthreepre\-setsearch\-querylinks\.Thelistisfixedanddoesnotchangewithkeystrokes\.Thedropdownitselfdoesnotrenderproductthumbnails,prices,orproductcards\-\-onlysuggestedqueriesandcollectionshortcuts\.

A\`/search/suggest\.json\`predictiveendpointisactiveinthebackgroundandreturnsproduct\-shapedJSON\(title,handle,price,compare\-atrange,availability,URL,vendor,featuredimage,tags,type\)butitsresultsarenotsurfacedintheheaderdropdown\.Submittingtheformnavigatestoafull\-page\`/search?q=<query\>\`resultslisting\.Asecondsearchinputappearsinthemobileoff\-canvasdrawerwiththesameformactionandnopredictivepanel\.

\#\#FloatingUX

Athird\-partyCMPrendersafull\-widthcookieconsentbanneratthebottomoftheviewportonfirstvisit\.Thebanneroffersaprivacy\-policylink,"AllowAll","RejectAll",and"ConfirmMyChoices"buttons\.Granularpreferencecontrolscoverfourconsentcategories\(analytics,marketing,preferences,saleofdata\)\.ChoiceisstoredinlocalStorageandintegrateswithGoogleConsentModeV2fordownstreamtaggating\.

AKlaviyo\-powerednewsletterpopupisconfigured\(loadedviaasyncscript\)\.Itistime\-orsession\-gatedanddidnotappearduringcold\-sessionexploration\.Theinlinenewslettersectioninthepagefooterarearemainsthealways\-oncapturesurface\.

Nochatwidget,noagegate,andnohomepagemodalpopupwereobserved\.

\#\#Policy&infopages

Seveninformationalpagesareexposed\.Fourarenativepoliciesunder\`/policies/<slug\>\`\-\-shipping,refund,privacy,andtermsofservice\-\-allsharingaminimalsingle\-columntemplate:H1title,thenrichtextparagraphs,H2sub\-sections,andbulletlists,withnoimagesorsidebars\.

Threecustompagesliveunder\`/pages/<slug\>\`\.Thecontactpageusesaherosplitlayoutwithaheadline,aone\-totwo\-sentencebody,andaCTAbuttondeep\-linkingintoanexternalcustomer\-supportportal\(noinlineform\),followedbyathree\-cardinformationalgridcoveringwarranty,productregistration,and24/7support\.TheFAQpageusesacustomanchored\-sectiontemplate:anH1,ahorizontalin\-pagenavigationbarwithfouranchors\(ProductsandUsage,Orders,Contact,Warranty\),andfourH2sectionsofaccordion\-stylequestion/answeritems\.Theaboutpageisacustomeditoriallayoutwithacelebrity\-chefheroquote,abrand\-storysectionpairingcopywithimagery,afive\-rownumberedfeaturelist,athree\-link"Shop"CTArow,andaninlineemailsubscribeformwithahiddenhoneypot\.

### F\.2Capabilities \(capabilities\.json\)

\{

"version":"0\.1",

"shop":\{

"descriptor":"Premiumhybridcookwareandkitchenaccessoriesshop",

"category":"Cookware/Kitchen",

"currency":"USD",

"tone":\[

"premium",

"bold",

"chef\-endorsed",

"direct\-to\-consumer"

\]

\},

"site\_shell":\{

"has\_announcement\_bar":true,

"header\_style":"two\-rowsticky",

"has\_mega\_menu":false,

"nav\_depth":2,

"footer\_groups":3

\},

"homepage":\{

"section\_types":\[

"hero\_video",

"category\_chip\_slider",

"featured\_product\_grid",

"specialty\_product\_banner",

"video\_endorsement",

"feature\_cards\_grid",

"reviews\_banner",

"product\_carousel",

"product\_hero\_quote",

"ugc\_video\_carousel",

"category\_banners\_3up",

"culinary\_council",

"recipe\_carousel",

"article\_carousel",

"founder\_message",

"newsletter\_inline"

\],

"section\_count":16,

"has\_popup\_modal":false

\},

"collection":\{

"layout":"grid",

"columns\_desktop":3,

"filters":\[

"availability",

"on\_sale"

\],

"sort":\[

"featured",

"best\_selling",

"alphabetically\_az",

"alphabetically\_za",

"price\_asc",

"price\_desc",

"date\_new",

"date\_old"

\],

"pagination":"load\_more"

\},

"product":\{

"gallery\_style":"horizontal\_carousel\_with\_prev\_next",

"variant\_selectors":\[

"radio\_buttons"

\],

"has\_quantity\_selector":true,

"description\_layout":"accordion\_tabs",

"has\_reviews":true,

"has\_recommendations":true,

"has\_personalization":false

\},

"cart":\{

"type":"drawer",

"has\_promo\_input":false,

"has\_upsells":true,

"has\_shipping\_estimate":false

\},

"search":\{

"trigger":"icon\_button",

"has\_predictive":true,

"predictive\_types":\[

"suggested\_queries",

"collection\_shortcuts"

\],

"results\_layout":"fullpage"

\},

"intl":\{

"has\_locale\_switcher":null,

"has\_currency\_switcher":null

\},

"floating":\{

"has\_chat\_widget":false,

"has\_age\_gate":false,

"has\_cookie\_banner":true,

"has\_newsletter\_popup":true

\},

"info\_pages\_present":\[

"shipping\-policy",

"refund\-policy",

"privacy\-policy",

"terms\-of\-service",

"contact",

"faq",

"about\-us"

\]

\}

### F\.3Statistics \(stats\.json\)

\{

"products\_total":164,

"collections\_total":50,

"products\_per\_collection":\{

"avg":15\.42,

"median":8\.0,

"max":179

\},

"price":\{

"min":0\.98,

"max":5499\.0,

"median":59\.99,

"currency":"USD"

\},

"products\_with\_variants\_pct":0\.07317073170731707,

"variant\_axes\_observed":\[

"title",

"size"

\],

"navigation\_depth\_max":2,

"homepage\_section\_count":16,

"info\_pages\_count":7,

"feature\_count":48

\}

## Appendix GShopArena Exploration Plan Example

This appendix reproduces theplan\.mddocument emitted by the planner iteration ofshop\_arena\.explorefor the same exploration run shown in Appendix[F](https://arxiv.org/html/2605.16116#A6)\. The plan enumerates the per\-task work items the executor loop then consumes one task at a time; checkbox state \(\[x\]/\[\!\]\) is updated by the executor at the end of each iteration\.

### G\.1Plan \(plan\.md\)

\#Plan\-\-PremiumHybridCookwareShop

Adirect\-to\-consumershopspecializinginpremiumhybridcookware,knives,kitchentools,seasonings,andbrandedapparel,endorsedbyacelebritychef\.SellsviaUSDpricingwithUS\-focusedshipping\.Theshopusesatwo\-rowstickyheader,flyoutdropdownnavigation,andacartdrawerarchitecture\.

\#\#Tasks

\-\[x\]homepage\_sections\-\-navigateto\`/\`;captureherosection,eachbelow\-herosection\(categorygrid,endorsementblock,featuressection,best\-sellerscarousel,social\-proof/testimonials,specialty\-productbanners,recipecarousel\),noteanynewsletterpopupbehavior;full\-pagescreenshot\+per\-sectionevidence\[priority:9\]

\-\[x\]cart\_drawer\-\-add1producttocart;capturedrawerinemptystate,filledstate,qty\-changestate;checkforpromo\-codeinput,upsellrail,andshipping\-estimatewidget\[priority:8\]

\-\[x\]header\_navigation\-\-snapshotmainnavinclosedstate,thentriggereachtop\-levelflyoutbutton\(8categories\);captureoneopenflyout;recordnavdepthandsub\-linkcountpercategory\[priority:7\]

\-\[x\]search\_predictive\-\-opensearchmodal;type2shortprefixes\("s"and"pan"\);capturesuggestionpanelshowingproducts;captureonenetworkrequestvianetwork\.jsonl\[priority:7\]

\-\[x\]collection\_filters\-\-visitthelargestcookwarecollection;exercise"Available"and"OnSale"filtercheckboxes;testallsortoptions;scrolltobottomtoconfirmpaginationbehavior;screenshotgridandrecordcolumncount\[priority:6\]

\-\[x\]product\_variants\-\-visit2PDPs:oneapparelproduct\(hassizeradiobuttons\)andonecookwareset\(single\-variant\);capturegallerycarouselstates\(prev/next\),variantselectorstates,qtystepper,descriptionlayout,reviewswidget\(Okendo\),and"Customersalsolove"recommendationsrail\[priority:5\]

\-\[x\]footer\-\-screenshotfullfooter;recordlinkgroups,socialicons,andbottom\-barcopyright/legallinks\[priority:4\]

\-\[x\]info\_pages\-\-visit\`/policies/shipping\-policy\`,\`/policies/refund\-policy\`,\`/policies/privacy\-policy\`,\`/policies/terms\-of\-service\`,\`/pages/contact\`,\`/pages/faq\`,\`/pages/about\-us\`;onescreenshoteach;notepagelayouttype\(policytemplatevscustompage\)\[priority:3\]

\-\[x\]floating\_widgets\-\-waitforandcaptureKlaviyonewsletterpopup;inspectCrispchatwidgetlauncher;checkOneTrustcookiebannerpresence;onescreenshotperwidgetifvisible\[priority:1\]

\#\#OmittedAreas

\-mega\_menu\-\-navusesflyoutdropdownpanelspercategory\(single\-columnsub\-linklists\),notafull\-gridmegamenulayout;observedinhomepagesnapshot

\-intl\_switchers\-\-only\`en\_US\`localepresent;nolocaleorcurrencyswitchervisibleinheader,footer,orhomepagesnapshot;\`index\.html\`localemetadatais\`en\_US\`only

\-age\_gate\-\-notobservedon\`/\`homepagesnapshotorin\`index\.html\`HTMLstructure

## Appendix HShopGuru Skill Catalog

This appendix accompanies §[2\.2\.1](https://arxiv.org/html/2605.16116#S2.SS2.SSS1)and summarizes the seven skill categories generated by ShopGuru\.

Table 1:ShopGuru’s seven skill categories grouped by horizon\. Short\-horizon tasks are produced by deterministic, pure\-function generators over ShopArena’s extracted JSON; long\-horizon tasks are LLM\-authored from few\-shot examples reflecting human\-shopper behaviors and reconciled against the same extracted data via a validator\-driven polish loop \(§[2\.2\.3](https://arxiv.org/html/2605.16116#S2.SS2.SSS3)\)\.
## Appendix IExamples of ShopGuru tasks

\{

"id":"mock\_clothing\-filter\-1",

"type":"shopping",

"intent":"Navigatetothe\\"FrostSeasonCollection\\"collectiononthisstore\.FindandusetheColorfilter\(e\.g\.Black\)toselectanoption\.Ifproductsareshownafterfiltering,selectanyvariantofaproductandaddittocart\.Itisokifnoproductsmatchthefilter;usingfiltercorrectlymeansthenavigationtaskiscompletedsuccessfully\.",

"success\_criteria":\{

"url\_contains":"/collections/frost\-season\-collection",

"type":"navigation"

\},

\}

Listing 1:Example collection\-filter task emitted by the deterministic generator\.url\_containsis a soft URL hint to the LLM judge, not a hard pass/fail gate\.\{

"id":"mock\_hardware\-returns\-1",

"type":"navigation",

"intent":"Findthereturnsandrefundpolicypageonthisstore\.Lookforreturnpolicyinformationinthefooter,menu,oranynavigationlinks\.Readtherefundpolicydetails,thenleavethesite\.",

"success\_criteria":\{

"url\_contains":"/policies/refund\-policy",

"type":"page\_navigation"

\},

"url\_contains\_alt":\[

"/pages/return\-policy"

\],

\}

Listing 2:Example page\-navigation task to find specific policy on the shop, also emitted by the deterministic generator\.\{

"id":"mock\_cookware\-e2e\-v1\-3",

"type":"shopping",

"intent":"Shoppingforanewchef'sknifeonabudget\.NavigatetotheKnivescollection\.ApplyafilterforColorifavailable,orsortthecollectionbyPrice:LowtoHigh\.BrowsethefilteredorsortedresultsandpickanyChef'sKnife\.Openitspage,choosewhicheverColorvariantyoulikebest\(e\.g\.,BlackHandleorNaturalWoodHandle\),andaddittocart\.Oncetheknifeisinyourcart,endthesession\.DonotclickanyCheckoutbutton\.",

"success\_criteria":\{

"url\_contains":"/collections/knives",

"type":"cart\_after\_filter\_or\_sort"

\}

\}

Listing 3:Example LLM\-authored long\-horizon journey that chains a collection navigation, a Color filter, a price sort option, and a final cart action\.\{

"id":"mock\_clothing\-e2e\-v1\-1",

"type":"shopping",

"intent":"First\-timevisitorexploringthebrand'sethos\.OpentheAboutUspagefromthefooterormainmenutoreaduponMockClothing'slifestylepositioning\.Next,navigatetotheFreshDropscollectiontoseewhat'snewlyavailable\.Pickanyaccessory\(likeascrunchieorcap\)thatcatchesyoureye,openitsproductpage,selectanyavailableColorandSizevariants,andaddittoyourcart\.Oncetheitemisinyourcart,endthesession\.DonotclickanyCheckoutbutton\.",

"success\_criteria":\{

"url\_contains":"/pages/about",

"type":"cart\_after\_about\_page\_detour"

\},

\}

Listing 4:Another example LLM\-authored long\-horizon task consisting of a shopping journey after a page navigation detour\.
## Appendix JShopGuru Polish Loop

This appendix accompanies §[2\.2\.3](https://arxiv.org/html/2605.16116#S2.SS2.SSS3)and illustrates the LLM\-authored E2E polish loop\.

Shop context\(catalog, collections, pages\)\+\+system prompt\+\+8 few\-shot examplesLLM \(roundkk\)e\.g\. GPT\-5, Gemini 3 ProValidator\(7 rules\)issues?finaltasksPolish prompt:∙\\bulletflagged tasks \(preserve IDs\)∙\\bulletspecific validator issues∙\\bulletshop contextnoneflaggedregenerateflagged onlyat most 2 rounds, then emit

Figure 10:LLM\-authored E2E polish loop\. The same validator that audits deterministic generators \(§[2\.2\.4](https://arxiv.org/html/2605.16116#S2.SS2.SSS4)\) drives an iterative refinement of LLM\-authored tasks\. The round\-kkoutput is audited; if any tasks are flagged, the polish prompt bundles*only the flagged tasks*with their specific validator issues and the shop context, the LLM regenerates only those tasks while preserving their IDs, and the merged result is re\-audited\. The loop terminates after at most two rounds; tasks that still fail validation after two rounds are surfaced as a release blocker rather than silently shipped\.
## Appendix KVerifiers for ShopGuru pipeline

This appendix accompanies §[2\.2\.4](https://arxiv.org/html/2605.16116#S2.SS2.SSS4)and lists the rule set used by ShopGuru’s post\-generation validator\. The validator runs as the final gate on every emitted benchmark file:error\-severity findings halt the build with a non\-zero exit code, whilewarning\-severity findings are logged and either fed back to the LLM\-authored journey generator as polish\-loop signals \(rules marked†in Table[2](https://arxiv.org/html/2605.16116#A11.T2)\) or surfaced as advisory hints for the human curator\. See §[2\.2\.4](https://arxiv.org/html/2605.16116#S2.SS2.SSS4)for the narrative description and §[2\.2\.3](https://arxiv.org/html/2605.16116#S2.SS2.SSS3)for the polish\-loop integration\.

Table 2:ShopGuru’s seven core validator rules\. The validator returns a list ofIssuerecords rather than raising, so callers can decide whether to error or just warn\. Errors halt the build with a non\-zero exit code\. Rules marked with†are additionally fed back to the LLM\-authored journey generator as polish\-loop signals \(§[2\.2\.3](https://arxiv.org/html/2605.16116#S2.SS2.SSS3)\); the unmarked warning is advisory\.RuleSeverityCatchesunknown\-collection†errorurl\_containsreferences a collection handle absent fromcollections\.json\.unknown\-product†errorSame, for/products/⟨\\langlehandle⟩\\rangle\.infeasible\-filter†errorFilter\(dim,value\)\(\\textit\{dim\},\\textit\{value\}\)is not realized by any product in the targeted collection\.intent\-answer\-leak†errorAsuccess\_criteria\.response\_containsvalue appears verbatim in the intent — an agent could pass by echoing the prompt without visiting the page\.option\-mismatch†warningThe intent asks the agent to “select a⟨\\langleOption⟩\\ranglevariant” for a product that has no such option \(e\.g\. “select Color” on a gift card whose only option is denomination\)\.product\-not\-in\-collection†warningThe intent pairs a product with a collection that does not contain it\.unknown\-pagewarningSame asunknown\-collectionbut for/pages/⟨\\langlehandle⟩\\rangle\(softer because some storefronts surface policies via/policies/\*fallthrough routes\)\.
## Appendix LShopGuru E2E Prompt Templates

This appendix accompanies §[2\.2\.3](https://arxiv.org/html/2605.16116#S2.SS2.SSS3)and reproduces \(i\) the system prompt used by the LLM\-authored long\-horizon journey generator, \(ii\) the user prompt template skeleton \(placeholders only\), and \(iii\) three of the eight few\-shot examples that ground the LLM in human\-shopper behavior\. The selected examples are the ones that best span the behavior categories enumerated in Table[1](https://arxiv.org/html/2605.16116#A8.T1):*Multi\-pet household shopping*\(multi\-product cart with a quantity edit\),*Free\-shipping threshold calculation*\(cart\-state arithmetic with a localized banner\), and*Filter that returns zero results, then recover*\(filter\-recover\)\. The remaining five few\-shot examples are available in the released benchmark code\. All prompt text is reproduced verbatim modulo ASCII\-folding of a small number of non\-ASCII glyphs \(→\\rightarrow, currency symbols, accented Lithuanian characters\)\.

### L\.1System Prompt

Youareanexpertatdesigningevaluationtasksfore\-commercewebagents\.

Yourjobistoauthorrealisticend\-to\-endshoppingjourneysthatmirrorhow

realhumanbuyersactuallybrowseandtransactononlinestores\.

Hardrules:

\-Everytaskmustbe100%

products,collections,pagetitles,orvariantoptionsthatarenotin

thedatadumpbelow\.

\-Aproductmentionedinataskmustexistinthecatalog\(Title\+handle\)\.

\-Acollectionmentionedinataskmustexist\(Title\+handle\)\.

\-Ifyouasktheagentto"selectaColorvariant"or"selectaSize",

thenamedproductMUSThavethatvariantoptionANDvalues\.Otherwise

usegenericphrasinglike"selectanyvariant"orsimply"addittocart"\.

\-Aproduct/collectionpairinginthesametask\(e\.g\."NavigatetoX

collectionandaddproductY"\)isonlyvalidifYisactuallyamember

ofXaccordingtotheProduct\-to\-CollectionMappingssection\.

\-Everytaskmustendinawell\-definedstatetheagentcanreach:a

specificsetofitemsinthecart,aspecificURLvisitedfora

navigationtask\.Never"andthenbrowsearound"\.

\-\*\*NeverinstructtheagenttoclickCheckout,proceedtocheckout,head

toacheckoutpage,abandoncheckout,orotherwisetraversethecart\-\>

checkoutboundary\.\*\*Checkoutisintentionallydisabledonthesandbox

shopandtheLLMjudgegradestasksonthe\*\*finalcartstateonly\*\*\.

Endshoppingtaskswithwordinglike"Oncetheproductisinyourcart,

endthesession\.DonotclickanyCheckoutbutton\."Fortasksinvolving

cartedits\(add\-\>remove,quantitychange\),describetheexpectedfinal

cartexplicitly\.

\-Intentsshouldbesecond\-person,imperative\.Shoppingtasksshouldbe

oneortwoshortparagraphs;theymayincludenaturaldetours\(policy

lookup,Aboutpage,brandcomparison\)beforethecart\-stateterminal\.

\-Avoidtasksthatrequireuserauthentication,paymententry,orany

externally\-gatedflow\.

\-Donotproducetasksthatrequirehumanjudgmentcallstheagentcannot

verify\(e\.g\.,"pickthemoststylishproduct"\)\.

Qualitybar\(thisiswhatseparatesagreattaskfromabasicone\):

\-Openwithaone\-sentencepersonaormotivationthatframesthejourney

\("First\-timevisitor\.","You'reareturns\-cautiousshopper\.",

"Multi\-pethouseholdshopping\.","Saleshunting\."\)\.

\-ReferencespecificstorefrontUIelements\(homepagebanner,topmenu,

footerQuickLinks,brandmenu\)bynamewhentheyexist\.

\-Mixatleast2\-3distinctskillspertask\(search\+compare\+cartedit;

policylookup\+navdrilldown\+add\)\.Single\-step"searchandadd"

tasksaretooeasyandshouldbetheminority\.

\-Useexactbrand/product/collectiontitlesaswritteninthedata\-\-

neverparaphrase\("FreshDrops"not"newarrivals","WomensRunClub"

not"runninggear"\)\.

Outputformat:aJSONobjectwithasinglekey"tasks"containinganarray

oftaskobjects,eachwithfields:

id\(slug\-friendlystring\),

type\("shopping"or"navigation"\),

intent\(string\),

success\_criteria\(objectwithurl\_containsandadescriptivetypefield\)\.

Donotincludethe\`url\`field\-\-thatwillbefilledindownstream\.

### L\.2User Prompt Template Skeleton

The user prompt is built per shop by substituting the placeholders below with concrete data extracted by ShopArena\. Sections marked\{placeholder\}are filled in at generation time fromproducts\.json,collections\.json,stats\.json, andpages\.json\.

Author\{count\}end\-to\-endevaluationtasksforthefollowingstorefront\.

Followthesystemprompt'sschemaandqualitybar\.Coverasmanyofthe

behaviorcategoriesbelowasthisstoresupports,andskipanythatare

notapplicabletothisstore\.

\#\#\#Shopprofile

Name:\{store\_name\}

Description:\{store\_description\}

Country:\{country\},Currency:\{currency\},Language:\{language\}

Domain:\{real\_url\}

\#\#\#Topcollections\(title\-\-handle\)

\{collections\_list\}

\#\#\#Topproducttypes

\{product\_types\_list\}

\#\#\#Exampleproducts\(title\-\-handle,withREALvariantoptions&values\)

\{products\_list\}

\#\#\#Product\-to\-Collectionmemberships\(usethesetogroundmulti\-steptasks\)

\{mapping\_str\}

\#\#\#Collectionfacets/optionpatterns\(forfiltertasks\)

\{option\_patterns\_list\}

\#\#\#Policy/infopages\(title\-\-/pages/handle\)

\{pages\_list\}

\#\#\#Giftcardproducts

\{gift\_card\_list\_or\_none\}

\#\#\#Behaviorcategoriestocover\(aimforatleast10of14\)

1\.Search&atomicadd\-to\-cart

2\.Navdrilldown\(menu\-\>sub\-menu\-\>collection\-\>product\)

3\.Filter\+sort\(e\.g\.Format=Hardcoverthensortbyprice\)

4\.Filterthatreturnszeroresults,thenrecover

5\.Substitute\-matchdiscovery\(intendedproductmissing\-\>closealternative\)

6\.Review/detailreadonaproductpage

7\.Sizechart/fitguidelookup

8\.Shippingpolicylookup\(withcartactionafter\)

9\.Returns/refundslookup\(withcartactionafter\)

10\.Giftcardpurchase\(onlyiftheshopsellsgiftcards\)

11\.Multi\-productcartwithedit\(addA,addB,removeA,setqty2onB\)

12\.Cross\-collectionorcross\-brandcomparison\(compareAandB,pickone\)

13\.Contact/storelocator/aboutpage

14\.Free\-shippingthresholdorsale\-discountcalculation\(ifbannerexists\)

\#\#\#High\-qualityreferencetasks\(fromothershopsinthisbenchmarksuite\)

\[SeeAppendixsectionsbelowforthreerepresentativefew\-shotexamples\.

Fiveadditionalexamplesareincludedinthereleasedbenchmarkcode\.\]

\#\#\#Anti\-patternstoAVOID

\-"SearchforX\.PickaColorandSizevariant\.Addtocart\."\-\-toothin,

nopersona,nodetour,singleskill\.

\-Mentioning"Color"or"Size"foraproductwhoseoptionslistabovedoes

NOTincludethatname\.

\-GenericUIreferenceslike"navigatetothehomepage"withoutnaminga

specificsectionorCTA\.

\-Twoproductsinoneintentthatarenotco\-locatedinanycollection

accordingtothemembershiplistabove\.

Whenconstructingtasks,USEEXACTproducttitles,collectiontitles,and

optionnamesfromthelistsabove\.Usethestorefront'sownwordingfor

collectionnames\("FreshDrops"not"newarrivals"\)\.Formulti\-steptasks,

ensureeach\(collection,product\)pairyoumentionisarealmembership\.

RespondwithaJSONobjectwitha"tasks"keycontainingthearrayof

\{count\}tasks\.Each\`id\`shouldusetheformat

"\{shop\_slug\}\-e2e\-v1\-\{index\}"replacing\{index\}withnumbersstartingat1\.

### L\.3Few\-Shot Example A: Multi\-Pet Household Shopping

A multi\-product cart with a per\-item quantity edit and a cross\-collection nav\. Exercises behavior categories 2 \(Nav drilldown\), 11 \(Multi\-product cart with edit\), and the cart\-state\-only termination rule\.

intent:"Multi\-pethouseholdshopping\.NavigatetotheDogKibble

collectionviatheDogsmenuandaddanykibbleproducttocart\.

NavigatetoCats\-\>LitterfromthetopmenuandopenWorldsBestCat

Litter\.Selectanysizevariant\(forexampleOriginal28lb\)andaddit

tocart\.Openthefullcartpageandincreasethecatlitterquantity

from1to2\.Oncethecartcontainsonedogkibbleandtwounitsof

catlitter,endthesession\.DonotclickanyCheckoutbutton\."

success\_criteria\.type:"cart\_multi\_pet\_with\_quantity\_edit"

### L\.4Few\-Shot Example B: Free\-Shipping Threshold Calculation

Cart\-state arithmetic over a localized homepage banner\. Exercises behavior category 14 \(Free\-shipping threshold\) and demonstrates that the prompt is robust to non\-English storefronts \(here, Lithuanian\)\.

intent:"Hitthefree\-shippingthreshold\.Notethe'Nemokamas

pristatymasnuo49EUR'banneronthehomepage\.NavigatetoSunims\-\>

SkanestaiandaddatreatproductunderEUR25tothecart\.Openthe

cartpage,observehowmuchmoreyouneedtoreachEUR49forfree

shipping\.Gobacktothestore,navigatetothesameSkanestai

category,andaddasecondproductthatpushesthetotaloverEUR49\.

Returntocart,confirmthefree\-shippingbannerhasupdated\.Once

bothproductsareinyourcartwithatotaloverEUR49,endthe

session\.DonotclickanyCheckoutbutton\."

success\_criteria\.type:"cart\_after\_free\_shipping\_threshold"

### L\.5Few\-Shot Example C: Filter\-Recover

A filter that yields zero or sparse results, followed by a recovery strategy\. Exercises behavior category 4 \(Filter that returns zero results, then recover\) and demonstrates handling of edge cases that deterministic generators cannot synthesize on their own\.

intent:"NavigatetotheChristmas&HanukkahcollectionviatheShop

byHolidays&Eventsmenu\.ApplyaSizefilterfor'3\-6Months';

resultswilllikelybesparseorempty\.Observetheemptyorfiltered

state,thencleartheSizefilterandapplyadifferentonesuchas

Color=Blue\.Ifproductsappear,openthefirstone,selectany

variant,andaddittocart;ifstillempty,removeallfiltersand

pickanyproductfromtheunfilteredlist\.Onceaproductisinyour

cart,endthesession\.DonotclickanyCheckoutbutton\."

success\_criteria\.type:"cart\_after\_filter\_recover"