SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

# SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents
Source: [https://arxiv.org/html/2605.19219](https://arxiv.org/html/2605.19219)
Han Li, Vibhor Malik, Zahra Zanjani Foumani, Alberto Castelo, Shuang Xie, Ailin Fan, Keat Yang Koay, Yuanzheng Zhu, Meysam Feghhi, Ronie Uliana, Zhaoyu Zhang, Angelo Ocana Martins, Mingyu Zhao, Francis Pelland, Jonathan Faerman, Nikolas LeBlanc, Aaron Glazer, Andrew McNamara, Zhong Wu, Lingyun Wang Shopify Bellevue, Washington, USA

###### Abstract

A/B testing remains the gold standard for evaluating modifications to e\-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience\. We presentSimGym, a framework for simulating A/B tests on e\-commerce storefronts using vision\-language model \(VLM\) agents operating in a live browser\. The framework comprises three key components: \(a\) a traffic\-grounded persona generation pipeline that derives per\-shop buyer archetypes and intents from production clickstream data; \(b\) a live\-browser agent architecture that combines multimodal perception over visual and browser\-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and \(c\) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior\. We validateSimGymon A/B tests of visually driven UI theme changes from a major e\-commerce platform across diverse storefronts and product categories\. Empirical results show thatSimGymagents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add\-to\-cart shifts observed across interface variants in real\-buyer traffic\. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants\.

## 1Introduction

A/B testing remains the gold standard for evaluating modifications in e\-commerce storefronts, including UI and theme changes, while enabling data\-driven decisions that directly affect conversion rates and revenue\. However, traditional A/B testing often incurs substantial costs\. Diverting traffic to experimental variants exposes real users to potentially suboptimal experiences, achieving statistical significance often requires weeks of data collection, and unsuccessful treatments can degrade customer experience before they are detected and rolled back\. These limitations make merchants more risk\-averse in practice: instead of testing bold redesigns with potentially large effects on conversion, merchants often favor incremental, low\-risk changes, which constrain innovation and hinder business growth\.

Solving the aforementioned challenges motivates a natural question: cansynthetic buyersbe used to pre\-test interface modifications before those variants are exposed to real customers? Recent advances in large language model \(LLM\)\-based and vision\-language model \(VLM\)\-based agents make this possibility increasingly realistic\. Recent work has shown that such agents can navigate complex online interfaces and execute multi\-step tasks across diverse browser environmentsZhouet al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib7)\); Denget al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib17)\); Chezelleset al\.\([2024](https://arxiv.org/html/2605.19219#bib.bib8)\); Panet al\.\([2024](https://arxiv.org/html/2605.19219#bib.bib20)\)\. In parallel, persona and profile\-conditioned agents have begun to exhibit more realistic patterns of user behaviorParket al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib9)\); Wanget al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib10)\); Zhanget al\.\([2024](https://arxiv.org/html/2605.19219#bib.bib11)\)\. VLMs further strengthen this trend by enabling agents to reason over rendered webpages rather than relying solely on textual page structure, thereby narrowing the gap between agent perception and the visual experience of real shoppers\. In e\-commerce and design\-evaluation settings, systems such as PAARSMansouret al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib12)\), Shop\-R1Zhanget al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib13)\), and Customer\-R1Wanget al\.\([2025c](https://arxiv.org/html/2605.19219#bib.bib33)\)suggest that agent behavior can be aligned with historical customer data, while AgentA/BWanget al\.\([2025a](https://arxiv.org/html/2605.19219#bib.bib31)\), UXAgentLuet al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib32)\), and SimABRiederet al\.\([2026](https://arxiv.org/html/2605.19219#bib.bib34)\)take further steps toward synthetic experimentation and usability testing using persona\-conditioned LLM agents\.

Despite these advances, existing work remains limited in two important respects\. First, prior research has yet to establish an end\-to\-end framework for synthetic A/B testing in real e\-commerce settings\. Such a framework must comprise three key components: \(a\) a persona generation pipeline that grounds synthetic buyers in observed traffic and persona distributions, ensuring that the simulated population reflects the heterogeneity of real users; \(b\) a stable browser\-based interaction environment in which agent behavior can be evaluated under the same live interface conditions encountered by actual customers; and \(c\) a reliable evaluation module for validating simulated outcomes under interface interventions\. Although prior studies have addressed these components only in isolationMansouret al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib12)\); Zhouet al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib7)\); Chezelleset al\.\([2024](https://arxiv.org/html/2605.19219#bib.bib8)\); Wanget al\.\([2025a](https://arxiv.org/html/2605.19219#bib.bib31)\); Luet al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib32)\); Riederet al\.\([2026](https://arxiv.org/html/2605.19219#bib.bib34)\); Wanget al\.\([2025c](https://arxiv.org/html/2605.19219#bib.bib33)\), they have not integrated them into a unified framework\. Consequently, the field still lacks a rigorous basis for conducting synthetic A/B testing on real e\-commerce use cases\.

Second, the practical utility of the end\-to\-end simulation workflow depends on how well synthetic agents can predict real user responses to interface changes\. However, prior work has not demonstrated such predictive validity in heterogeneous real e\-commerce settings\. Specifically, existing approaches primarily focus on standardized offline benchmarks such as WebShopYaoet al\.\([2022](https://arxiv.org/html/2605.19219#bib.bib15)\)and ShoppingBenchWanget al\.\([2025b](https://arxiv.org/html/2605.19219#bib.bib16)\), behavioral\-similarity analyses on clickstream tracesSunet al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib14)\); Wanget al\.\([2025c](https://arxiv.org/html/2605.19219#bib.bib33)\), static\-screenshot A/B simulation without live\-browser interaction or magnitude validationRiederet al\.\([2026](https://arxiv.org/html/2605.19219#bib.bib34)\), or live\-browser usability simulation without validation against measured human intervention effectsLuet al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib32)\)\. Consequently, none of these approaches thoroughly evaluate whether simulated outcomes align with the observed shifts under real per\-shop A/B interventions, leaving a critical gap: an agent could behave reasonably at the level of navigation yet still fail to predict actual conversion shifts, thereby offering limited value for pre\-testing decisions in real e\-commerce applications\.

![Refer to caption](https://arxiv.org/html/2605.19219v1/x1.png)

Figure 1:SimGymframework overview\.In this paper, we presentSimGym, a framework for A/B test simulation in which VLM\-powered browser agents, grounded in a merchant’s observed customer distribution, autonomously browse two variants of the same online storefront\. Unlike prior offline simulation approaches that operate on static datasetsWanget al\.\([2025b](https://arxiv.org/html/2605.19219#bib.bib16)\); Zhanget al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib13)\),SimGymagents interact with live storefronts in a full browser environment, requiring real\-time multimodal perception, memory management, and action execution\. By aggregating agent behaviors and decision traces,SimGymproduces comparative metrics that quantify how interface changes, such as visually driven theme changes, affect add\-to\-cart \(A2C\) performance\.[Figure˜1](https://arxiv.org/html/2605.19219#S1.F1)illustrates the end\-to\-end framework\. We summarize our contributions as follows:

- •We present an end\-to\-end framework,SimGym, for synthetic A/B test simulation and evaluation with VLM\-powered browser agents\. The framework is fully modular, with components that can be independently customized, replaced, or extended\.
- •We introduce a novel technical realization ofSimGym, combining a traffic\-grounded synthetic\-buyer construction pipeline with a multimodal live\-browser agent architecture\. The pipeline transforms production clickstreams into shop\-specific buyer archetypes and shopping intents, while the agent architecture integrates visual observations, textual webpage representations, episodic memory, and execution guardrails to support coherent, persona\-grounded shopping behavior across diverse real\-world storefronts\.
- •We validate the framework on real\-world, visually driven theme\-change use cases from a production e\-commerce platform by comparing simulated predictions against observed human behavioral outcomes\. Empirical evaluation demonstrates strong directional alignment and correlation between simulated and observed outcome shifts across diverse storefronts and product categories\.

## 2SimGym Framework

As shown in[Figure˜1](https://arxiv.org/html/2605.19219#S1.F1), theSimGymframework consists of three key components: \(a\) a traffic\-grounded persona generation pipeline \([Section˜2\.1](https://arxiv.org/html/2605.19219#S2.SS1)\); \(b\) a multimodal live\-browser agent architecture \([Section˜2\.2](https://arxiv.org/html/2605.19219#S2.SS2)\); and \(c\) an evaluation protocol \([Section˜2\.3](https://arxiv.org/html/2605.19219#S2.SS3)\)\.SimGymis fully modular, and each component can be independently customized or extended\.

### 2\.1Persona Generation

The persona generation pipeline derives a population of synthetic shoppers from merchant\-specific clickstream traces through a six\-stage process\. Each synthetic shopper is represented by abuyer persona, which we define as the pairing of abuyer archetype\(a multi\-dimensional behavioral and values profile\) with abuyer intent\(a sampled product target paired with a purchase\-decision guide\)\. Rather than relying on hand\-crafted personas or generic shopper archetypes, the pipeline infers shopper heterogeneity from observed session and buyer\-level behavior\. At a high level, this process comprises: \(a\) clustering sessions to identify recurring purchase patterns; \(b\) extracting product preferences and generating buyer intents; \(c\) aggregating buyer\-level behavioral signals to construct buyer archetypes; and \(d\) composing intent and archetype into the buyer persona that drives the live\-browser agent\.

This design enablesSimGymto preserve each storefront’s empirical customer distribution, align agent profiles with observed engagement and conversion patterns, and scale persona construction across diverse merchants without manual tuning\. We describe the implementation details in[Section˜3](https://arxiv.org/html/2605.19219#S3)\.

### 2\.2Agent Architecture

Unlike existing offline simulation approachesZhanget al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib13)\),SimGymadopts a VLM\-powered browser agent architecture that executes shopping sessions directly on live storefronts through a full browser environment\. To operate reliably across heterogeneous and dynamically changing e\-commerce interfaces, this architecture combines multimodal web perception, action planning, memory management, and guarded browser execution, as shown in[Figure˜7](https://arxiv.org/html/2605.19219#A3.F7)\. We describe each component below\.

Multimodal Web Perception\.SimGymsimulation agents observe each page through two complementary modalities provided to the VLM at every decision step: \(a\) textual web structure information, represented by a DOM\-derived accessibility tree that abstracts the page into a hierarchy of elements, such as headings, buttons, and links, with unique reference IDs mapped to DOM locations for unambiguous action targetingDenget al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib17)\); Zhouet al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib7)\); and \(b\) visual page perception, represented by a page screenshot that captures the rendered storefront experience as it would be presented to a real shopper, including layout, imagery, typography, color, and visual emphasis\.

The accessibility tree provides a compact, action\-oriented representation that preserves the semantic structure needed for navigation and element selection, while the screenshot complements this representation with the rendered visual context conveyed by storefront design, promotional content, and product imagery\. Combining these modalities allowsSimGymsimulation agents to reason about interface changes whose effects are primarily visual, such as hero redesigns, banner prominence, and color\-led merchandising, while maintaining stable, DOM\-grounded action execution across heterogeneous storefront designs\.

Action Planning\.SimGymsimulation agents follow an observe\-plan\-act loop commonly used in autonomous agent architecturesParket al\.\([2023](https://arxiv.org/html/2605.19219#bib.bib9)\)\. In this design, the simulation agent serves as a high\-level planner: at each decision step, it determines whether the shopping session should terminate and, if not, generates the next browser\-level action to be executed by a dedicated browser controller\. To support goal\-directed and persona\-conditioned decisions,SimGymassembles a state\-conditioned planning context for the VLM that contains: \(a\) the assigned shopping goal; \(b\) the generated buyer persona \(see[Section˜3](https://arxiv.org/html/2605.19219#S3)\), including exploration depth, price sensitivity, product preferences, and value orientation; \(c\) session memory summarizing prior actions, observations, and intermediate outcomes; \(d\) the current browser state, represented by the URL, the DOM\-derived accessibility tree, and the page screenshot; and \(e\) guardrails specifying permissible checkout and cart operations\.

Given this context, each decision step proceeds through three stages: \(a\)*observe*, where the agent observes the current page state and retrieves the accumulated session memory; \(b\)*plan*, where the VLM produces a schema\-constrained response containing its reasoning, a termination decision, and a proposed next action if the session should continue; and \(c\)*act*, where the system either terminates the session or passes the predicted action to the dedicated browser controller for execution on the live page\. The action specification includes the action type, target element, and required arguments, making the model output directly parseable by downstream execution logic\.

After execution, the controller records the resulting page transition, action outcome, and relevant observations, then appends this information to session memory for subsequent planning steps\. This loop repeats until the simulation agent completes its shopping goal or triggers a guardrail\. The process keeps each decision grounded in the current page state while preserving session\-level continuity across the full browsing trajectory\. We provide an example agent trajectory in Appendix[E](https://arxiv.org/html/2605.19219#A5)\.

Memory Management\.The simulation agents maintain an episodic session memory that accumulates the complete browsing trajectory, including the initial navigation context, prior observations, model reasoning and decisions, executed actions, action outcomes, and any error states\. This memory is incorporated into the planning context at each decision step, allowing the agent to maintain temporal coherence across multi\-step shopping sessions\. By conditioning on previously viewed products, attempted interactions, and failed actions, the agent can reduce redundant behavior and make more consistent decisions across the browsing trajectory, mirroring how real shoppers maintain and use browsing context during product exploration\.

Guardrails\.To improve reliability in open\-ended web environments,SimGymuses guardrails that constrain execution and support recovery from common failure modes\. These mechanisms include: \(a\) loop protection, which detects repeated identical actions and prevents unbounded executionPanet al\.\([2024](https://arxiv.org/html/2605.19219#bib.bib20)\); \(b\) step and time budgets, which bound the duration of each simulation; \(c\) model retry logic, which retries failed VLM calls with error context; and \(d\) error propagation, which passes browser execution failures back to the agent to support informed recovery rather than blind retry\. These guardrails ensure that simulations recover or terminate gracefully when agents encounter unexpected page states, execution failures, or reasoning errors\.

### 2\.3Evaluation Protocol

The evaluation protocol connects simulated shopping behavior to behavioral shifts observed in human traffic on production storefronts, and is designed to evaluate whether simulated agents predict the effect of real interface changes, rather than merely producing plausible browsing traces\. The protocol consists of two stages: \(a\) ground\-truth construction that identifies paired control and treatment storefront variants with sufficient traffic and substantive UI differences, while excluding pairs whose A2C signal would be dominated by non\-theme factors such as overlapping promotional campaigns, merchandising or assortment shifts, pricing changes, and new\-shop ramp\-up periods; and \(b\) predictive\-validity measurement that compares simulated and observed human shifts in add\-to\-cart behavior using alignment rate and Pearson correlation\.

The protocol operates only on per\-agent session logs and observed storefront outcomes, and is therefore agnostic to the upstream persona generation pipeline and browser\-agent architecture\. Alternative persona generators or agent implementations can be integrated and evaluated using the same evaluation protocol under the modularized design\. We describe the ground\-truth dataset and metric definitions in detail in[Section˜4](https://arxiv.org/html/2605.19219#S4)\.

## 3Persona Generation

Building synthetic shoppers that can predict real customer behavior requires agent profiles grounded in each merchant’s traffic distribution\. Storefronts often serve distinct buyer populations with heterogeneous intents, preferences, and browsing behaviors\. For example, a boutique fitness brand can exhibit a significantly different traffic composition than a sporting\-goods retailer\. Unlike traditional persona generation which relies on UX intuition with fictional personas, we directly extract shopper intents and behavioral preferences from each shop’s clickstream data\. To prevent evaluation leakage, persona construction uses historical clickstreams disjoint from the A/B sessions used for ground\-truth A2C estimation\. Capturing merchant\-specific shopper heterogeneity at scale entails two design objectives: \(a\) generated behavioral preferences should reflect the empirical distribution of buyer types observed on each storefront; and \(b\) the pipeline should scale across a large and heterogeneous set of merchants without manual persona engineering or store\-specific tuning\. Guided by these design objectives, we construct agent personas through a six\-stage pipeline\.

Stage 1: Session\-Level Clustering\.We represent each storefront session as a feature vector spanning engagement \(duration, event count\), exploration depth \(product views, distinct items\), search behavior, funnel progression \(A2C, checkout, purchase\), and economic value \(cart value, order value\)\. Features are standardized via z\-scoring across sessions, and we apply k\-means clustering with k\-means\+\+Arthuret al\.\([2007](https://arxiv.org/html/2605.19219#bib.bib6)\)initialization\. We usek=5k=5selected by an elbow\-style analysis of k\-means inertia and cluster\-size balance\. Each session is assigned to its nearest centroid, with the standardized distance retained as a measure of assignment uncertainty\.

Stage 2: Product Preference Extraction\.For each \(shop, cluster\) pair, we construct an input record containing shop metadata \(name and industry\), the cluster identifier, and an aggregate summary of products browsed or purchased by sessions in the cluster\. We use GPT\-5Singhet al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib1)\)to extract product preferences from this record and output a structured JSON object with three fields: \(a\)product categories, listing up to ten broad categories such as “sneakers” or “athletic wear”; \(b\)individual products, listing up to ten frequently browsed or purchased products in the cluster; and \(c\) a brief reasoning\. We restrict categories to generic descriptors rather than shop\-specific product names to improve reuse in downstream persona generation\.

Stage 3: Buyer Intent Generation\.We generate a structured shopping intent for each agent with two components: \(a\)*product target*, sampled from the Stage 2 product categories associated with the agent’s cluster and paired with a buyer archetype from the same cluster, so that agents share a behavioral profile but vary in shopping targets; and \(b\)*purchase\-decision guide*, a fixed instruction that asks the agent to research available options, defer purchase by default, and add a product to cart only when the product clearly fits the assigned archetype and the storefront experience provides sufficient purchase confidence\. To keep intents reusable across storefront variants, we exclude bundles, sizes, discounts, and UI\-specific details from the product target\.

Stage 4: Buyer Behavior Aggregation\.For each \(shop, cluster\) pair, letnndenote the number of simulated agents allocated to that cluster\. We select thennsessions nearest to the cluster centroid, using centroid proximity to obtain representative behavioral examples for agent construction\. We then aggregate the selected sessions at the buyer level to compute cross\-session summaries including session counts, funnel outcomes \(A2C, checkout, purchase\), average cart and order values, and product interaction histories\.

Stage 5: Buyer Archetype Construction\.We construct buyer archetypes as multi\-dimensional profiles rather than single mutually exclusive labels, allowing multiple traits to coexist \(e\.g\., price\-sensitive and ethics\-oriented\)\. Each buyer is represented along five continuous dimensions grouped into behavioral and value dimensions \([Table˜1](https://arxiv.org/html/2605.19219#S3.T1)\)\. Behavioral dimensions describe how the buyer shops and are computed from aggregated clickstream statistics\. Value dimensions describe what product attributes appear to drive choices\. For each value axis, we identify relevant products using predefined keyword sets \(e\.g\., “handcrafted”, “organic”, “commercial grade”\) and compare their prevalence among browsed and purchased items, separating general interest from revealed purchase preference\. Finally, we use GPT\-5 to output a structured JSON with a score for each archetype dimension and a concise rationale\. The full schema and an example output are shown in[Figure˜5](https://arxiv.org/html/2605.19219#A2.F5)\.

Table 1:Buyer Archetype DimensionsStage 6: Prompt Composition\.We form each agent persona by pairing a Stage 3 shopping intent with a Stage 5 buyer archetype from the same cluster\. The final prompt combines the product target, purchase\-decision guide, behavioral and value profiles, and cluster\-level product preferences from Stage 2\. This yields personas that reflect observed product interests and multi\-dimensional buyer traits\.[Figure˜6](https://arxiv.org/html/2605.19219#A3.F6)shows an example persona output\.

## 4Evaluation Protocol

In this section, we describe the evaluation protocol along two dimensions: \(a\) constructing a ground\-truth dataset to capture how human shoppers respond to storefront changes; and \(b\) defining evaluation metrics to measure predictive validity across diverse buyer types\.

### 4\.1Ground Truth Construction

To assess the predictive validity ofSimGym, we construct a ground\-truth dataset from real A/B experiments of theme changes on the e\-commerce platform\. Each experiment compares a control storefront against a treatment storefront that differs in visual presentation or layout, allowing us to measure human A2C shifts directly from real buyer traffic exposed to the two variants\. We select shops using two criteria: \(a\) sufficient traffic in both control and treatment groups to support reliable human A2C estimation; and \(b\) substantial visual or layout differences between the paired storefront variants\.

Since the paired storefront variants vary in magnitude, from minor styling tweaks to complete layout redesigns, we employ Gemini 3 ProGoogle DeepMind \([2025b](https://arxiv.org/html/2605.19219#bib.bib3)\)as a strong VLM evaluator to jointly inspect page screenshots and the parsed DOM, characterize the change in each control\-treatment pair, and stratify the dataset by change magnitude\. We additionally apply quality\-control filters to exclude pairs whose A2C signal would be dominated by non\-theme factors \(overlapping promotional campaigns, merchandising or assortment shifts, pricing changes, and new\-shop ramp\-up periods\), and require that the remaining theme changes are non\-trivial\. This yields a final dataset of5050shops, each containing a paired control\-treatment storefront comparison, a summary of the theme change, and observed human A2C shifts between variants\.

### 4\.2Evaluation Methodology

We evaluateSimGymby measuring predictive validity against real human outcomes\. Rather than comparing exact browsing trajectories, we assess whether simulated agents predict the behavioral shifts induced by storefront changes\. We use A2C rate as the primary outcome, as it directly reflects purchase intent and corresponds to a key optimization target for merchants\. Our evaluation focuses on two dimensions of predictive validity: \(a\) for directional validity, we use alignment rate as the percentage of shops for which the sign of the agent\-predicted A2C shift between storefront variants matches the sign of the observed human A2C shift; and \(b\) for magnitude validity, we use the Pearson correlation coefficient between agent\-predicted and human\-observed A2C shifts across shops\. We report 95% confidence intervals for both metrics over then=50n\{=\}50shops\. Computation details are in[Appendix˜G](https://arxiv.org/html/2605.19219#A7)\.

## 5Experiments

In this section, we evaluateSimGymsimulations on 50 real\-world shops spanning 16 countries and 11 product categories \([Figure˜2](https://arxiv.org/html/2605.19219#S5.F2)\)\. These 50 shops have ground\-truth A2C shifts constructed from observed human behavior across control and treatment storefront variants with visually driven theme changes, using the procedure described in[Section˜4\.1](https://arxiv.org/html/2605.19219#S4.SS1)\. We report all metrics on theskimmerscohort, the largest non\-bouncer traffic segment, which accounts for64\.5%64\.5\\%of all engaged sessions\. This cohort exhibits substantive purchase intent with a roughly9\.5%9\.5\\%session\-level A2C rate, and its A2C shifts are highly correlated with whole\-shop A2C shifts\.111A2C shift denotes the change in human A2C rate between the control and treatment storefront variants\. Across the 50 shops, A2C shifts measured on the skimmers cohort are highly correlated with those measured on whole\-shop traffic after excluding immediate bouncers \(r≈0\.89r\\approx 0\.89; Appendix[F](https://arxiv.org/html/2605.19219#A6)\)\.

![Refer to caption](https://arxiv.org/html/2605.19219v1/x2.png)

Figure 2:Dataset distribution of the 50\-storefront golden set spanning 16 countries and 11 industries\. It includes smaller markets and specialized categories to support evaluation across diverse storefronts\.Table 2:Overall simulation evaluation under different LLMs for agent decision making\.Each agent uses the Stagehand browser\-automation libraryBrowserbase \([2026](https://arxiv.org/html/2605.19219#bib.bib2)\)as the browser controller for operating live storefronts\. In our experiments, we use proprietary models for agent decision making through public APIs \(Gemini 3 FlashGoogle DeepMind \([2025a](https://arxiv.org/html/2605.19219#bib.bib4)\)\) and serve open\-source models on an internal cluster with4040NVIDIA B200 GPUs \(GPT\-OSSAgarwalet al\.\([2025](https://arxiv.org/html/2605.19219#bib.bib5)\)\)\. In each simulation, we run600600agents per shop\. To reduce stochastic variation, we repeat each shop\-level simulation twice and report metrics averaged over the two trials\.

### 5\.1Overall Evaluation

Table[2](https://arxiv.org/html/2605.19219#S5.T2)reports the overall simulation performance under different models used for agent decision making\. We make the following key observations\. First, Gemini 3 Flash with visual integration achieves the strongest simulation quality, reaching77%77\\%directional alignment and0\.550\.55correlation with observed human A2C shifts\. This indicates that a strong backbone model paired with full multimodal agent perception provides the best predictive agreement with real behavioral responses\. Second, the comparison between Gemini 3 Flash with and without visual input demonstrates the value of multimodal perception, as the vision\-enabled agent improves both alignment \(77%77\\%vs\.70%70\\%\) and correlation \(0\.550\.55vs\.0\.490\.49\) over its text\-only counterpart\. This suggests that visually salient storefront features \(e\.g\., image quality, layout density, and trust cues\) provide behavioral signals that are only coarsely captured by DOM structure alone and help agents better predict buyer responses to interface changes\. Third, GPT\-OSS underperforms the Gemini 3 Flash configurations, suggesting that a performance gap remains between open\-source and leading proprietary models for agent decision making\. However, the GPT\-OSS text\-only agent still achieves59%59\\%alignment and0\.410\.41correlation, producing meaningful simulation signals despite lacking visual input\. This suggests thatSimGymmaintains reasonable predictive validity with less capable open\-source models, highlighting the robustness of the framework beyond proprietary multimodal systems\. Finally,SimGymsupports efficient per\-shop simulation across model configurations\. With a 600\-agent budget, a single\-shop simulation completes in 5\.3 minutes for Gemini 3 Flash with vision, 4\.5 minutes for text\-only Gemini 3 Flash, and under 13 minutes for GPT\-OSS\. These runtimes suggest thatSimGymsupports rapid storefront evaluation across different agent decision\-making models\.

### 5\.2Ablation Studies

This section presents two key ablations based on the Gemini 3 Flash vision\-enabled agent to quantify the impact ofSimGym’s persona generation and memory management components on overall simulation quality\.

#### 5\.2\.1Effect of Persona Generation Strategy

To quantify the effect of persona and intent inputs from the persona generation pipeline \(see[Section˜3](https://arxiv.org/html/2605.19219#S3)\), we compare three variants on the same 50\-shop golden set:Full Persona\(a buyer archetype with a shopping intent\),Shopping Intent Only\(a target product with a purchase\-decision guide\), andProduct Only\(only the target product to shop for without buyer archetype or purchase guide\)\.

[Table˜3](https://arxiv.org/html/2605.19219#S5.T3)demonstrates that the full persona representation is central to predictive validity\. First,Full Personaachieves the strongest performance, with77%77\\%directional alignment and0\.550\.55correlation with observed human A2C shifts\. Second, removing the buyer archetype substantially degrades both metrics:Shopping Intent Onlydrops to44%44\\%alignment with near\-zero correlation, indicating that product targets and purchase intentions alone do not capture sufficient behavioral heterogeneity\. Finally,Product Onlyperforms worst, with chance\-level alignment \(51%51\\%\) and the lowest correlation\. These results indicate that accurate simulation depends on population\-specific behavioral variation encoded by the buyer archetype\.[Figure˜3](https://arxiv.org/html/2605.19219#S5.F3)further supports this pattern at the shop level\.Full Personashows the strongest human\-agent agreement, whileShopping Intent OnlyandProduct Onlyscatter diffusely with weak correspondence to human A2C shifts\.

Table 3:Effect of persona and intent inputs on predictive validity\.![Refer to caption](https://arxiv.org/html/2605.19219v1/x3.png)\(a\)Full Persona
![Refer to caption](https://arxiv.org/html/2605.19219v1/x4.png)\(b\)Shopping Intent Only
![Refer to caption](https://arxiv.org/html/2605.19219v1/x5.png)\(c\)Product Only

Figure 3:Human\-agent agreement in A2C shifts\. Each panel plots human\-observed versus agent\-predictedΔ\\DeltaA2C rates under a persona input variant\. The dashed line denotesy=xy\{=\}x\.
#### 5\.2\.2Effect of Memory Management Strategy

We study the effect of episodic session memory on simulation quality by comparing the full agent with a variant that removes memory access\. In the no\-memory setting, the agent receives only the current page state and persona at each step, without access to prior actions or observations\.

[Table˜4](https://arxiv.org/html/2605.19219#S5.T4)shows that removing episodic session memory substantially degrades predictive validity\. Without memory, directional alignment drops to42%42\\%, and correlation collapses to0, indicating that the agent no longer recovers either the direction or magnitude of human A2C shifts\. This suggests that session memory is essential for coherent multi\-step shopping behavior and for producing simulation outcomes that remain predictive of real buyer responses\. A failure\-mode analysis further attributes this degradation to navigation incoherence\. Without memory,70\.7%70\.7\\%of diverged agents become stuck in navigation loops\. This suggests that memoryless agents often fail to track visited pages or attempted actions, preventing them from reaching the product\-evaluation stage where storefront changes can meaningfully affect A2C decisions\.

Table 4:Effect of session memory on predictive validity\.

### 5\.3Sensitivity Analysis

![Refer to caption](https://arxiv.org/html/2605.19219v1/x6.png)\(a\)Directional alignment
![Refer to caption](https://arxiv.org/html/2605.19219v1/x7.png)\(b\)Pearson correlation

Figure 4:Agent sample\-size sensitivity on the5050\-shop golden set\. Shaded bands denote the 10th–90th percentile range over10001000bootstrap resamples\.Since shop\-level simulated A2C shifts are estimated from a finite set of buyer\-agent sessions, the agent budget \(i\.e\., the number of agents per shop simulation\) determines both estimate variance and simulation cost\. We therefore select the agent budget by identifying where additional agents yield diminishing gains in aggregate predictive validity\. Using a completed Gemini 3 Flash run with vision and full persona on the5050\-shop golden set, we evaluate finite\-sample stability via bootstrap resampling\. For each agent budget from5050to700700, we resample buyer\-agent sessions with replacement within each shop, recompute shop\-level agentΔ\\DeltaA2C, and then recompute cross\-shop Pearson correlation and directional alignment against humanΔ\\DeltaA2C\. We perform10001000bootstrap resamples for each agent budget\.

[Figure˜4](https://arxiv.org/html/2605.19219#S5.F4)shows the finite\-sample behavior of the evaluation metrics with two key findings: \(a\) For directional validity, alignment rises from67%67\\%at5050agents to73%73\\%by300300agents, with little additional improvement through700700agents\. \(b\) For magnitude validity, correlation increases from0\.480\.48at a budget of5050to0\.540\.54by300300, after which gains plateau; the 10th–90th percentile band also narrows from0\.170\.17at5050agents to0\.050\.05at600600agents\. Based on these findings, we use an agent budget of600600in all experiments, which places the simulation beyond the observed stability plateau and provides reliable estimates with additional buffer for finite\-sample variability and session\-level failures\.

## 6Conclusion

We presentedSimGym, a framework for e\-commerce A/B test simulation that uses traffic\-grounded VLM\-powered browser agents to interact with live storefront variants and predict the effects of interface changes on user behavior\. The framework includes a traffic\-grounded persona generation pipeline, a multimodal browser agent architecture with memory and guardrails, and an evaluation protocol that validates simulated outcome shifts against observed human outcomes\. Empirical results on 50 real storefront changes show thatSimGymachieves strong agreement with observed human add\-to\-cart shifts while significantly reducing evaluation cycles, enabling rapid experimentation without exposing real buyers to candidate variants\.

Limitations and Future Work\.Our study is conducted on a single major e\-commerce platform\. While the evaluation covers a 50\-shop golden set of curated A/B\-tested storefront changes spanning 16 countries and 11 product industries, generalization beyond this setting remains to be tested\. We also report results without post\-training alignment on human traces, whose impact on predictive validity remains to be validated\. Important future directions include extending the framework beyond a single platform and beyond add\-to\-cart shifts to broader behavioral targets, and improving agent fidelity through post\-training alignment on human traces\. More broadly,SimGymcan serve as an inner loop for automated UI exploration, enabling candidate interface changes to be proposed, screened, and prioritized in simulation before merchant\-facing experiments\.

## References

- S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§5](https://arxiv.org/html/2605.19219#S5.p2.2)\.
- D\. Arthur, S\. Vassilvitskii,et al\.\(2007\)K\-means\+\+: the advantages of careful seeding\.InSoda,Vol\.7,pp\. 1027–1035\.Cited by:[§3](https://arxiv.org/html/2605.19219#S3.p2.1)\.
- Browserbase \(2026\)Stagehand: the ai browser automation framework\.Note:[https://github\.com/browserbase/stagehand](https://github.com/browserbase/stagehand)Cited by:[§5](https://arxiv.org/html/2605.19219#S5.p2.2)\.
- D\. Chezelles, T\. Le Sellier, S\. O\. Shayegan, L\. K\. Jang, X\. H\. Lù, O\. Yoran, D\. Kong, F\. F\. Xu, S\. Reddy, Q\. Cappart,et al\.\(2024\)The browsergym ecosystem for web agent research\.arXiv preprint arXiv:2412\.05467\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1)\.
- X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su \(2023\)Mind2web: towards a generalist agent for the web\.Advances in Neural Information Processing Systems36,pp\. 28091–28114\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.19219#S2.SS2.p2.1)\.
- B\. Efron \(1979\)Bootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.Cited by:[Appendix G](https://arxiv.org/html/2605.19219#A7.p2.6)\.
- Google DeepMind \(2025a\)Gemini 3 flash \- model card\.Note:[https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-Flash\-Model\-Card\.pdf](https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdf)Model card\. Published December 2025; updated 17 December 2025Cited by:[§5](https://arxiv.org/html/2605.19219#S5.p2.2)\.
- Google DeepMind \(2025b\)Gemini 3 pro \- model card\.Note:[https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-Pro\-Model\-Card\.pdf](https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf)Model card update: December 2025Cited by:[§4\.1](https://arxiv.org/html/2605.19219#S4.SS1.p2.1)\.
- Y\. Lu, B\. Yao, H\. Gu, J\. Huang, Z\. Wang, Y\. Li, J\. Gesi, Q\. He, T\. J\. Li, and D\. Wang \(2025\)UXAgent: an LLM\-agent\-based usability testing framework for web design\.InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems \(CHI EA ’25\),External Links:[Document](https://dx.doi.org/10.1145/3706599.3719729)Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1),[§1](https://arxiv.org/html/2605.19219#S1.p4.1)\.
- S\. Mansour, L\. Perelli, L\. Mainetti, G\. Davidson, and S\. D’Amato \(2025\)Paars: persona aligned agentic retail shoppers\.InProceedings of the 1st Workshop for Research on Agent Language Models \(REALM 2025\),pp\. 143–159\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1)\.
- Y\. Pan, D\. Kong, S\. Zhou, C\. Cui, Y\. Leng, B\. Jiang, H\. Liu, Y\. Shang, S\. Zhou, T\. Wu,et al\.\(2024\)Webcanvas: benchmarking web agents in online environments\.arXiv preprint arXiv:2406\.12373\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.19219#S2.SS2.p8.1)\.
- J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th annual acm symposium on user interface software and technology,pp\. 1–22\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.19219#S2.SS2.p4.1)\.
- T\. Rieder, M\. Schneider, M\. Truss, V\. Tsaplin, A\. Rublea, S\. Dere, F\. Chicharro Sanz, T\. Reiss, and M\. D\. Dogan \(2026\)SimAB: simulating A/B tests with persona\-conditioned AI agents for rapid design evaluation\.arXiv preprint arXiv:2603\.01024\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1),[§1](https://arxiv.org/html/2605.19219#S1.p4.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§3](https://arxiv.org/html/2605.19219#S3.p3.1)\.
- L\. Sun, S\. Fu, B\. Yao, Y\. Lu, W\. Li, H\. Gu, J\. Gesi, J\. Huang, C\. Luo, and D\. Wang \(2025\)LLM agent meets agentic ai: can llm agents simulate customers to evaluate agentic\-ai\-based shopping assistants?\.arXiv preprint arXiv:2509\.21501\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p4.1)\.
- D\. Wang, T\. Hsu, Y\. Lu, H\. Gu, L\. Cui, Y\. Xie, W\. Headean, B\. Yao, A\. Veeragouni, J\. Liu,et al\.\(2025a\)AgentA/b: automated and scalable web a/btesting with interactive llm agents\.arXiv preprint arXiv:2504\.09723\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1)\.
- J\. Wang, K\. Xiao, Q\. Sun, H\. Zhao, T\. Luo, J\. D\. Zhang, and X\. Zeng \(2025b\)ShoppingBench: a real\-world intent\-grounded shopping benchmark for llm\-based agents\.arXiv preprint arXiv:2508\.04266\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p4.1),[§1](https://arxiv.org/html/2605.19219#S1.p5.1)\.
- L\. Wang, J\. Zhang, X\. Chen, Y\. Lin, R\. Song, W\. X\. Zhao, and J\. Wen \(2023\)Recagent: a novel simulation paradigm for recommender systems\.arXiv preprint arXiv:2306\.02552\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1)\.
- Z\. Wang, Y\. Lu, Y\. Zhang, J\. Huang, and D\. Wang \(2025c\)Customer\-R1: personalized simulation of human behaviors via RL\-based LLM agent in online shopping\.arXiv preprint arXiv:2510\.07230\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1),[§1](https://arxiv.org/html/2605.19219#S1.p4.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022\)Webshop: towards scalable real\-world web interaction with grounded language agents\.Advances in Neural Information Processing Systems35,pp\. 20744–20757\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p4.1)\.
- A\. Zhang, Y\. Chen, L\. Sheng, X\. Wang, and T\. Chua \(2024\)On generative agents in recommendation\.InProceedings of the 47th international ACM SIGIR conference on research and development in Information Retrieval,pp\. 1807–1817\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1)\.
- Y\. Zhang, T\. Wang, J\. Gesi, Z\. Wang, Y\. Lu, J\. Lin, S\. Zhan, V\. Gao, R\. Jiao, J\. Liu,et al\.\(2025\)Shop\-r1: rewarding llms to simulate human behavior in online shopping via reinforcement learning\.arXiv preprint arXiv:2507\.17842\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p5.1),[§2\.2](https://arxiv.org/html/2605.19219#S2.SS2.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2023\)Webarena: a realistic web environment for building autonomous agents\.arXiv preprint arXiv:2307\.13854\.Cited by:[§1](https://arxiv.org/html/2605.19219#S1.p2.1),[§1](https://arxiv.org/html/2605.19219#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.19219#S2.SS2.p2.1)\.

## Appendix ABroader Impact

SimGymstudies offline simulation for A/B testing e\-commerce storefronts\. Its primary positive impact is to reduce the need to expose real customers to untested interface variants\. By allowing merchants to pre\-screen potentially risky redesigns in simulation,SimGymcan help reduce user exposure to suboptimal or frustrating shopping experiences, accelerate iteration from multi\-week online experiments to substantially faster offline evaluation, and prioritize which variants should be evaluated with real traffic\.

We also recognize several potential risks\. First, live\-browser agents should be clearly separated from real user traffic so that simulated sessions do not contaminate merchant analytics, marketing attribution, or downstream recommender training\. Second, automated UI optimization could be misused to reinforce manipulative or deceptive design patterns that improve short\-term conversion at the expense of user well\-being\. Any deployment should therefore couple predictive\-validity objectives with explicit ethical, usability, and user\-experience constraints\. Third, persona construction from production clickstream data is privacy\-sensitive\. Such data should be processed only under appropriate data\-use agreements, with de\-identification, aggregation where applicable, minimum\-cohort thresholds, and strict access controls to prevent leakage of individual browsing behavior\.

## Appendix BBuyer Archetype Construction

![Refer to caption](https://arxiv.org/html/2605.19219v1/x8.png)

Figure 5:Buyer Archetype Construction Framework\.[Figure˜5](https://arxiv.org/html/2605.19219#A2.F5)details the buyer archetype construction process\. Given aggregated buyer data from Stage 4, we compute scores along five continuous dimensions organized into two categories:

Behavioral dimensionscapturehowbuyers shop:

- •Price sensitivity: Measures the gap between maximum browsed price and average purchased price\. Buyers are labeled as budget \(\>\>50% gap\), mid\-range \(30–50% gap\), or premium \(<<30% gap\), with category\-aware normalization to account for product\-specific price distributions\.
- •Exploration depth: A 0–1 score derived from session duration, search count, and product views, mapped to shallow \(0–0\.35\), moderate \(0\.35–0\.65\), or deep \(0\.65–1\.0\) exploration regimes\.

Values dimensionscapturewhybuyers choose products:

- •Premium focus: Attention to luxury, craftsmanship, and prestige\.
- •Performance focus: Emphasis on durability, reliability, and specifications\.
- •Ethics focus: Interest in sustainability and ethical sourcing\.

For each values dimension, we identify products containing relevant keywords \(e\.g\., “handcrafted,” “commercial grade,” “organic”\) and compare the proportion in browsed vs\. purchased items to infer revealed preferences\. An LLM processes these signals with category\-aware taxonomy and deterministic decision rules, outputting a structured JSON with scores, confidence estimates, and reasoning traces\.

## Appendix CPersona Extracted Output

![Refer to caption](https://arxiv.org/html/2605.19219v1/x9.png)Figure 6:Persona Extracted Output\.[Figure˜6](https://arxiv.org/html/2605.19219#A3.F6)presents a representative output from our persona generation pipeline described in[Section˜2\.1](https://arxiv.org/html/2605.19219#S2.SS1)\. The visualization illustrates how the six\-stage pipeline transforms raw clickstream data into a structured agent prompt\.

The Intent component, generated in Stage 3, pairs a sampled product target \(here, “chairs”\) with the fixed shopping guide described in[Section˜3](https://arxiv.org/html/2605.19219#S3), which anchors purchase decisions to the archetype\. The Shopping Profile encodes two behavioral dimensions derived from Stage 4’s buyer behavior aggregation and Stage 5’s archetype construction:\(1\)\(1\)Price Tier is set to "Budget" \(price\-conscious and value\-focused\), reflecting category\-aware price sensitivity computed from the gap between browsed and purchased price points;\(2\)\(2\)Exploration Depth is classified as "Shallow" \(direct and focused\), derived from the buyer’s session duration, search frequency, and product view counts\.

The Values section captures the buyer’s "Performance & Reliability" orientation, one of three values dimensions in our framework \(see[Table˜1](https://arxiv.org/html/2605.19219#S3.T1)\)\. This classification emerges from Stage 5’s analysis of keyword\-matched product interaction patterns \(e\.g\., "durable," "professional\-grade," "certified"\)\.

Finally, the Shopping Experience Preferences operationalize the persona into actionable behavioral guidance\. These preferences are derived from the intersection of behavioral and values dimensions: Budget Tier Preferences specify responsiveness to discount signals, social proof, and urgency cues, while Performance Value Preferences emphasize attention to detailed specifications, customer reviews, and transparency about materials\. Together, these components enable the agent to exhibit coherent, persona\-consistent behavior throughout the shopping session\.

![Refer to caption](https://arxiv.org/html/2605.19219v1/x10.png)Figure 7:Agent Architecture\.
## Appendix DAgent Architecture

[Figure˜7](https://arxiv.org/html/2605.19219#A3.F7)visualizes the observe–plan–act loop introduced in[Section˜2\.2](https://arxiv.org/html/2605.19219#S2.SS2)\. At each step:

- •Observe\.The agent reads the current browser state \(URL, accessibility tree, and screenshot\) and retrieves the running session memory\.
- •Plan\.The VLM produces a schema\-constrained response containing its reasoning, a termination decision, and–if the session should continue–the next action to take\.
- •Act\.The browser controller executes the action on the live storefront and writes the resulting page transition and outcome back to session memory for the next step\.

An episodic session memory carries the full trajectory forward across steps for temporal coherence, and a guardrail layer \(loop protection, step and time budgets, model retry, and error propagation\) wraps the loop to enforce graceful termination on failure\. A full worked trajectory under this loop is given in[Appendix˜E](https://arxiv.org/html/2605.19219#A5)\.

## Appendix EAgent Trace Example

![Refer to caption](https://arxiv.org/html/2605.19219v1/x11.png)

Figure 8:Agent Reasoning During Initial Navigation\.![Refer to caption](https://arxiv.org/html/2605.19219v1/x12.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.19219v1/x13.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.19219v1/x14.png)\(c\)

Figure 9:Screenshots of Agent Browsing\. \(a\) Minis collection \($7–9\)\. \(b\) Dragons collection \($4–$30\)\. \(c\) Axolotls collection \($6–$18\)\.![Refer to caption](https://arxiv.org/html/2605.19219v1/x15.png)

Figure 10:Agent Reasoning During Collection Exploration\.In this section, we present a complete trace of aSimGymagent completing a shopping task on an anonymous store, specializing in3​D3Dprinted fidget toys and collectible figures\. The agent’s goal is to find and purchase a premium\-quality collectible mini, with a buyer persona emphasizing luxury materials, refined craftsmanship, and moderate exploration depth\.

[Figure˜8](https://arxiv.org/html/2605.19219#A5.F8)shows the agent’s initial reasoning as it begins navigating the store\. The agent explores multiple collections \([Figure˜9](https://arxiv.org/html/2605.19219#A5.F9)\), systematically rejecting low\-priced items that don’t match its premium preferences\.[Figure˜10](https://arxiv.org/html/2605.19219#A5.F10)captures the agent’s reasoning as it moves between collections:“The current collection does not contain any high\-priced, premium\-quality collectible minis that match the buyer profile’s emphasis on luxury materials and refined craftsmanship\.”Rather than settling for budget options priced at $4–7, the agent efficiently shifts between collections, a behavior consistent with its moderate exploration depth\.

![Refer to caption](https://arxiv.org/html/2605.19219v1/x16.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.19219v1/x17.png)\(b\)

Figure 11:Product Selection and Add\-to\-cart\. \(a\) Premium Crystal Wing Dragon product page \($33\.18\)\. \(b\) Add\-to\-cart confirmation\.![Refer to caption](https://arxiv.org/html/2605.19219v1/x18.png)

Figure 12:Agent Reasoning During Purchase Decision and Checkout\.After exploring multiple collections, the agent identifies a suitable product: the Premium Crystal Wing Dragon \([Figure˜11\(a\)](https://arxiv.org/html/2605.19219#A5.F11.sf1)\), priced at $33\.18\. The agent’s reasoning confirms the match:“I have found a product that matches my buyer profile: it is presented as a premium edition, uses high\-quality 3D printing, offers refined craftsmanship, and is priced higher than the low\-cost mystery packs\.”The agent selects product options and adds the item to cart \([Figure˜11\(b\)](https://arxiv.org/html/2605.19219#A5.F11.sf2)\)\.[Figure˜12](https://arxiv.org/html/2605.19219#A5.F12)shows the final reasoning as the agent proceeds to checkout, concluding:“Given the moderate exploration depth, further browsing is unnecessary; proceeding to checkout will fulfill the buying intent\.”

This trace illustrates persona\-consistent behavior throughout an extended shopping journey\. The agent’s premium preference led it to reject multiple budget\-priced collections before identifying a suitable $33\.18 product, while its moderate exploration depth prevented excessive browsing once a match was found\.

Table 5:Session\-cluster distribution on the5050\-shop golden set\. Skimmers \(highlighted\) is the cohort used throughout[Section˜5](https://arxiv.org/html/2605.19219#S5)\.Table 6:Per\-shop humanΔ\\DeltaA2C in the skimmers cohort vs\. whole\-shop signals \(Pearson correlation across5050shops\)\.
## Appendix FCohort Coverage and Representativeness

[Table˜5](https://arxiv.org/html/2605.19219#A5.T5)reports the cluster distribution of all sessions on the5050\-shop golden set, computed from the same session clustering used in[Section˜2\.1](https://arxiv.org/html/2605.19219#S2.SS1)\. The skimmers cohort is the largest cluster of*engaged*\(non\-bouncing\) sessions; the immediate\-bounce cluster is excluded from any reported metric since it carries no UI\-engagement signal, and the remaining three clusters are individually insignificant to support stable per\-shopΔ\\DeltaA2C estimates\.

To check that reporting on a single cohort does not distort the shop\-level picture, we compare the per\-shop humanΔ\\DeltaA2C in the skimmers cohort against \(i\) the whole\-shop signal across all sessions and \(ii\) the whole\-shop signal with the immediate\-bounce cluster excluded\. Pearson correlation across the5050shops isr≈0\.75r\{\\approx\}0\.75for \(i\) andr≈0\.89r\{\\approx\}0\.89for \(ii\); see[Table˜6](https://arxiv.org/html/2605.19219#A5.T6)\. The cohort signal is therefore a faithful proxy for the engaged shop\-level signal\.

## Appendix GConfidence Interval Computation

We report95%95\\%confidence intervals for the two headline predictive\-validity metrics, alignment rate and Pearson correlation, over then=50n\{=\}50shops in the golden set\. Each shop is the unit of analysis and contributes one paired\(Δ​A2Chuman,Δ​A2Csim\)\(\\Delta\\text\{A2C\}\_\{\\text\{human\}\},\\Delta\\text\{A2C\}\_\{\\text\{sim\}\}\)observation, averaged across the two trials per shop \([Section˜5](https://arxiv.org/html/2605.19219#S5)\)\.

Alignment rate\.We use the percentile bootstrapEfron \([1979](https://arxiv.org/html/2605.19219#bib.bib35)\)over shops\. For each ofB=10,000B\{=\}10\{,\}000resamples, we drawn=50n\{=\}50shops with replacement from the golden set and recompute alignment rate as the fraction of resampled shops on which the trial\-averaged sign of the agent A2C shift agrees with the sign of the human A2C shift\. The reported95%95\\%CI is the empirical2\.52\.5th–97\.597\.5th percentile interval over theBBbootstrap replicates\.

Pearson correlation\.We use the Fisherzz\-transform on then=50n\{=\}50paired shop observations:z=12​ln⁡\(1\+r1−r\)z=\\tfrac\{1\}\{2\}\\ln\\\!\\left\(\\tfrac\{1\+r\}\{1\-r\}\\right\)with standard error1/n−31/\\sqrt\{n\-3\}, after which the symmetriczz\-interval is back\-transformed to therrscale\. This is the standard parametric CI for the Pearson coefficient\.

Both procedures share the same unit of analysis \(the shop\) and the same sample size \(n=50n\{=\}50\), so the intervals are directly comparable across the two metrics within each table\.