Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification

# Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification
Source: [https://arxiv.org/html/2609.16093](https://arxiv.org/html/2609.16093)
###### Abstract

A shopping conversation has many routes to the same cart, and a task\-success rate reduces all of them to one score\. We build a deterministic and reproducible e\-commerce environment that precommits each trial’s customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule\. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model\. The environment guides the simulator’s actions and records every assistant action alongside the environment state at that point\. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence\. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product\. We further use this evidence to apply different penalties to tool calls depending on how the assistant’s actions compare with an expected tool\-call set\. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange\. This interaction creates an open\-ended and verifiable simulation\. Across eight open\-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under\-action, over\-purchase, unsupported product attributes, and poor search, all of which terminal success obscures\.

## 1Introduction

Tool\-using agents for online shopping must translate incomplete preferences into catalog actions while maintaining a cart the customer accepts\. A final cart establishes the endpoint of that interaction, but it does not establish the route that produced it\. The completed transcript alone cannot determine the catalog facts available to the agent, the cart state after each action, or whether a customer had requested an item at the time of a search\. These missing facts prevent an evaluator from distinguishing an appropriate action from an accidental success or an unnecessary one\.

We propose an e\-commerce simulation environment that constructs the evidence required for verification before a conversation begins\. The generator creates a hidden*target cart*from a10,00010\{,\}000\-product control catalog and commits to a*reveal schedule*that determines when each item becomes relevant to the customer\. A controller applies that schedule during the conversation while a user simulator plays the customer\. The environment retains the target cart, catalog state, tool trace, and reveal spans throughout the trial\. This construction keeps the customer interaction open\-ended while giving the evaluator a stable account of what the agent needed to know and do\.

The verifier uses this account to reconstruct cart state and evaluate catalog\-grounded claims, search behavior, and tool use\. A model\-based judge remains responsible only for questions that require language, including whether an attribute claim is supported by a catalog row and whether a response addresses the customer’s request\. The evaluation therefore separates failures that a final\-cart score combines\. It can identify whether an agent failed to find a requested item, added an item the customer did not want, described a product without support, or spent its turns without advancing the purchase\.

We evaluate eight open\-weight agents from 20B to 35B parameters on 160 trials per agent\. The resulting profiles expose differences in retrieval, cart management, and grounded descriptions that have different implications for deployment and repair\. We also examine shopping pace across the generator’s difficulty tiers using termination reasons and tool\-call density\.

We make five contributions\.

1. 1\.An e\-commerce environment whose scenario generator emits the ground truth required for verification\. Each scenario contains a hidden target cart, per\-item reveal spans, and a control catalog whose category mix is fitted to aggregate retail data \(Section[3](https://arxiv.org/html/2609.16093#S3)\)\.
2. 2\.A battery of4444metrics for shopping outcomes, tool use, and customer\-support quality\. Deterministic state reconstruction settles questions that do not require a judge, and a standard registry supports new metrics \(Section[4](https://arxiv.org/html/2609.16093#S4)\)\.
3. 3\.Cohort\-specific scoring\. Each trial kind carries its own rubrics and applicability map, allowing an evaluation to target a deployment without averaging incompatible tasks \(Section[4](https://arxiv.org/html/2609.16093#S4)\)\.
4. 4\.A verification ablation that identifies which questions can be settled from a final cart, a transcript, environment state, or the full simulator record \(Section[5\.1](https://arxiv.org/html/2609.16093#S5.SS1)\)\.
5. 5\.An eight\-model study that reads agents as capability profiles and uses termination reasons and tool\-call density to diagnose difficulty\-tier outcomes \(Section[5](https://arxiv.org/html/2609.16093#S5)\)\.

## 2Related work

WebShop\([Yao et al\., 2022](https://arxiv.org/html/2609.16093#bib.bib2)\)gives the agent a complete shopping request at the start and scores how well the purchased product satisfies it\. Its authors validate this scorer and separately analyze search activity, products visited, and example trajectories, but the benchmark does not publish standardized correctness labels for individual actions\.

ShoppingBench, WebMall, WebArena, and Mind2Web evaluate shopping or browser agents from task specifications supplied up front\([Wang et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib3);[Peeters et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib4);[Zhou et al\., 2024](https://arxiv.org/html/2609.16093#bib.bib6);[Deng et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib7)\)\. Conversational shopping benchmarks expose preferences incrementally\. ShopperBench conditions a dual\-agent interaction on persona profiles\([Ling et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib8)\), while ChatShop reveals hidden attributes and options of one target product in response to clarification questions\([Chen et al\., 2024](https://arxiv.org/html/2609.16093#bib.bib5)\)\. Neither structure precommits per\-item request boundaries for state\-grounded checks\.

Process\-oriented evaluation also moves beyond terminal success\. AgentBoard measures progress over manually labeled subgoals, and Kirgis et al\. recommend systematic log analysis to expose failures hidden by outcome scores\([Ma et al\., 2024](https://arxiv.org/html/2609.16093#bib.bib12);[Kirgis et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib21)\)\. Conversational, stateful benchmarks provide the closest process comparisons\. ToolSandbox matches fixed, human\-authored milestone and minefield graphs to per\-turn state snapshots, imposing a partial temporal order over selected events\([Lu et al\., 2025](https://arxiv.org/html/2609.16093#bib.bib11)\)\.τ\\tau\-bench compares the final database in a simulated retail interaction with an annotated goal state\([Yao et al\., 2025](https://arxiv.org/html/2609.16093#bib.bib9)\)\. Its dialogue records when information was uttered, but its scenario does not expose a precommitted, structured reveal schedule, so its published reward cannot distinguish a search issued before a request from the same search issued after it\. COMPASS progressively reveals travel constraints through a dynamically updated simulator prompt and records the turn of full specification, but it does not assign each constraint a reveal turn before dialogue\([Qin et al\., 2025](https://arxiv.org/html/2609.16093#bib.bib10)\)\. Our scenarios add the precommitted, request\-relevant boundaries needed to grade intermediate tool calls\.

Large language models are widely used to evaluate generated text\([Zheng et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib13);[Liu et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib14)\), but studies document position and length biases and imperfect success classification\([Wang et al\., 2024](https://arxiv.org/html/2609.16093#bib.bib15);[Dubois et al\., 2024](https://arxiv.org/html/2609.16093#bib.bib16);[Lù et al\., 2025](https://arxiv.org/html/2609.16093#bib.bib17)\)\. Claim\-decomposition methods provide a complementary approach to factuality evaluation\. FActScore checks atomic claims against a knowledge source, while FacTool combines claim extraction with task\-specific tools\([Min et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib18);[Chern et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib19)\)\.

## 3The Evaluation Environment

We fix only the evidence required for verification before each trial and leave the surrounding customer interaction open\-ended\.

### 3\.1The storefront

We evaluate agents against a held\-out control catalog of10,00010\{,\}000products across1919categories\. We approximate a general e\-commerce storefront by averaging category shares from six retailers\. This distribution provides a grounded prior over categories rather than an exact reproduction of any retailer\. Appendix[A](https://arxiv.org/html/2609.16093#A1)gives the data sources, normalization procedure, and aggregate and per\-retailer shares\.

#### Tool interface\.

The agent acts through eight tools\. The verifier groups them into four registry categories when penalizing unnecessary calls; Appendix[E](https://arxiv.org/html/2609.16093#A5)gives the tool mapping and weights\. The retrieval backend encodes queries and product text withsentence\-transformers/all\-MiniLM\-L6\-v2and ranks normalized embeddings by cosine similarity\([Reimers and Gurevych, 2019](https://arxiv.org/html/2609.16093#bib.bib20)\)\. Each trial uses a fresh cart\.

### 3\.2Scenario: what the customer wants, and who they are

A trial pairs the agent under test with a*user simulator*, a frontier model that plays the customer\. One integer seed determines the scenario\. General shopping, grocery, and the refusal probes in Appendix[J](https://arxiv.org/html/2609.16093#A10)share this machinery, but the harness generates and aggregates them separately\. The trial kind determines which categories the basket draws from and how the customer works through it\.

#### Target cart\.

The customer is assigned a hidden set of in\-stock products with required quantities, built category\-first so the basket fills the way a real one does\. A difficulty tier sets*breadth*, the number of catalog categories visited\. Within a category we draw a price\-weighted representative, a Gaussian over log\-price centered on the category median so that mid\-range items dominate, and that price sets*depth*\. WritingBBfor the breadth andpjp\_\{j\}for the price of the representative drawn in slotjj, the basket holdsm=∑j=1BDjm=\\sum\_\{j=1\}^\{B\}D\_\{j\}distinct products, with

P⁡\(Dj=d\)\\displaystyle P\(D\_\{j\}=d\)=rjd−1​\(1−rj\),\\displaystyle=r\_\{j\}^\{\\,d\-1\}\\,\(1\-r\_\{j\}\),rj\\displaystyle r\_\{j\}=max⁡\(0\.07,1−21\+cj\),\\displaystyle=\\max\\\!\\Big\(0\.07,\\;1\-\\tfrac\{2\}\{1\+c\_\{j\}\}\\Big\),cj\\displaystyle c\_\{j\}=min⁡\(6,max⁡\(1,round⁡\(10−3\.3​log10​pj\)\)\)\.\\displaystyle=\\min\\\!\\big\(6,\\ \\max\(1,\\ \\mathrm\{round\}\(10\-3\.3\\log\_\{10\}p\_\{j\}\)\)\\big\)\.A five\-dollar staple givesrj≈0\.71r\_\{j\}\\approx 0\.71and a mean depth near3\.53\.5; a seven\-hundred\-dollar item givesrj=0\.07r\_\{j\}=0\.07and almost always resolves to a single unit\. The tier controls breadth alone, so the two simple tiers draw one category and reach several products through depth, while the multi\-item and complex tiers multiply the basket by visiting more categories\. Quantities follow an offset power laww⁡\(q\)∝\(q\+c\)−αw\(q\)\\propto\(q\+c\)^\{\-\\alpha\}with two regimes\. Settingc=0c=0leaves the head steep, so the shopper takes one unit, while a largeccflattens it, so one through five units are comparably likely\. Groceries use the flat regime with a taper that keeps party\-size counts rare\. Appendix[B](https://arxiv.org/html/2609.16093#A2)gives the remaining parameters\.

Because every goal item is drawn from the catalog, the customer never asks for something the store does not carry\. An agent that reports no match has failed to retrieve rather than found an empty shelf, and Section[4](https://arxiv.org/html/2609.16093#S4)scores it that way\.

#### Difficulty and persona\.

Each trial carries a difficulty from11to1010mapped to one of four tiers \(Table[1](https://arxiv.org/html/2609.16093#S3.T1)\)\. The turn budget is basket\-aware rather than a fixed function of difficulty\. It isslack\+round⁡\(wtier⋅m\)\\text\{slack\}\+\\mathrm\{round\}\(w\_\{\\text\{tier\}\}\\cdot m\)formmrealized distinct products, with slack33and a per\-tier weight growing from1\.01\.0to2\.252\.25\. Within a tier, raw difficultydddetermines whether the customer withholds optional hints\. Each hint about size, color, fit, or budget is withheld with probabilityd/10d/10\. The customer is also drawn along six independent categorical dimensions with realistic\-frequency weights\. These dimensions specify how precisely the customer states what they want, how readily they commit, how much price matters, their product vocabulary, their message length, and their typing quality\. Appendix[C](https://arxiv.org/html/2609.16093#A3)lists the values\.

Table 1:The four difficulty tiers\. Breadth is the number of catalog categories the target cart spans\. Grocery trials use the wider breadth ranges\. The turn\-budget weightwtierw\_\{\\text\{tier\}\}multiplies the realized basket size\.

### 3\.3The reveal schedule

Large target carts require the customer to express requests across several messages, making request timing part of the answer key\. A search issued at turn22cannot be held against an item the customer first mentions at turn99\. We therefore fix the schedule in the scenario at build time and run it through a driver\-side controller, which stamps each item with the customer\-message index at which it is first requested\.

Each multi\-item trial draws one of three styles based on the realized basket and trial kind\.batchstates the list up front,linearreleases one item at a time in order after the preceding item reaches the cart, andnonlinearfollows a beat itinerary that can introduce and explore an item, defer it while pivoting to another request, and return later to settle it\. These phases can interleave across items rather than follow basket order\. Grocery is list\-shaped rather than a journey, so it is batch\-dominant and never nonlinear; non\-grocery complex trials lean nonlinear\. Appendix[D](https://arxiv.org/html/2609.16093#A4)gives the sampling weights\.

Abatchtrial with more than four products instead releases a bounded wave of items and advances once every item in the current wave is in the cart\. Ordinary waves group related items, while complex trials may mix categories\. Appendix[D](https://arxiv.org/html/2609.16093#A4)gives the wave construction\.

Every nonlinear goal item receives one*essential*add beat, while optional beats control its introduction, exploration, deferral, and return\. This construction keeps every target achievable while allowing the conversation to move among unresolved requests\. Appendix[D\.1](https://arxiv.org/html/2609.16093#A4.SS1)shows recorded linear, micro\-batch, and nonlinear conversations produced by these schedules\.

### 3\.4Running a trial

A turn contains one customer message and one agent turn, which may span several tool\-call rounds\.

The simulator answers each turn with a single structured call\. Its five fields contain a short rationale, an inline frustration rating, the next customer message, a termination flag with its reason, and the chain of tool calls it expects the agent to make in response to the message it is about to send\. That last field is the reference set for tool\-call correctness in Section[4](https://arxiv.org/html/2609.16093#S4)\. The simulator sees the persona, the target cart, the transcript so far, and the agent’s own system prompt verbatim\. Providing the system prompt lets it tell a clarifying question the agent’s policy mandates from one that wastes the customer’s time\.

The customer it plays, however, cannot see the tool messages\. The simulator reads them to rate the turn and to decide whether the goal is met, and it voices the customer’s needs in character rather than quoting or paraphrasing a search payload, a product identifier, a price, or a stock count\. This rule keeps search\-result knowledge out of customer messages, so the agent must locate catalog items from the requests it can see\.

The controller reads the ground\-truth cart after every agent turn rather than scraping the transcript for it\. On a nonlinear trial it also owns termination, ignoring a premature “done” from the simulator until the cart genuinely holds every goal item at its required quantity\. The six recorded termination reasons are goal completion, turn\-budget exhaustion, customer abandonment, an agent failure, a simulator failure, and a missing opening message\. Section[5](https://arxiv.org/html/2609.16093#S5)reads that mix directly\.

The verification signals that follow use the reveal spans defined in Section[3\.3](https://arxiv.org/html/2609.16093#S3.SS3)\. Search metrics count a target only after the customer has requested it\.

Figure[1](https://arxiv.org/html/2609.16093#S3.F1)traces the trial from the precommitted scenario through the interaction loop and retained evidence to the verifier\.

Environment\-grounded verification pipelineScenario construction: ground truth fixed before interactionSeedKind \+difficultyControlcatalogScenariogeneratorPrecommitted scenarioTargetcartquantitiesRevealschedule\+ timingPersona\+ turnbudgetTrial execution and environment\-grounded verificationInteraction loopRevealcontrollersimulatorAgentTools \+storeEvidenceFinal cartTranscriptEnvironmentstateSimulatorrecordVerifierstrongest available evidenceEvaluation questionsoutcomes, actions, claimstool use, experienceRulefixedchecksModelLLMchecksHybridrule\+ modelInfertranscript onlyCapabilityprofileOutcomesRetrievalCartGroundingTool useExperienceFigure 1:Environment\-grounded verification pipeline\. The precommitted scenario drives the interaction, whose retained evidence supports fixed \(Rule\), model\-based \(Model\), and hybrid checks\. Transcript\-only inferences lack verification evidence; verified measurements form a capability profile\.

## 4Verification signals

Verification uses the strongest evidence the harness has for each question\. By*state replay*, we mean reapplying successful tool results in transcript order to reconstruct the cart after each mutation\. Together with the catalog and tool log, state replay settles what the agent retrieved and changed in the store state; a judge is used only for customer\-facing meaning or quality that the record cannot settle\.

### 4\.1Deterministic signals

The state replay reconstructs a final mapping from product to quantity, which the outcome metrics compare against the target\.Exact\-cart successrequires the cart to hold exactly the target items at exactly the target quantities\. We also report cart precision and recall to separate adding spurious items from missing wanted ones\. Withm=∑\(p,q\)∈Gmin⁡\(C⁡\[p\],q\)m=\\sum\_\{\(p,q\)\\in G\}\\min\(C\[p\],q\)over targetGGand reconstructed cartCC, precision ism/∑pC⁡\[p\]m/\\sum\_\{p\}C\[p\]and recall ism/∑\(p,q\)∈Gqm/\\sum\_\{\(p,q\)\\in G\}q\.Goal achievementis the weaker conditionC⁡\[p\]≥qC\[p\]\\geq qfor every target\.Goal progresssamples the fraction of exactly\-satisfied target products after every cart mutation and takes the trial’s maximum, so over\-adding drops an item back out of the satisfied set\.

For tool use we tag each call as*errored*when its result reports an error or*redundant*when its signature repeats with no intervening state change\. We further tag searches as*goal\-relevant*when at least one target product is both missing and already revealed according to Section[3\.3](https://arxiv.org/html/2609.16093#S3.SS3),*empty*when a clean execution returns no results, and*bad*when a goal\-relevant search errors, returns nothing, or returns none of the still\-needed targets\. A bad search is*recovered*when a later same\-category retry succeeds, and*autonomous*when that retry happens in the same turn without a customer re\-prompt\. The rates arenerr/ncalln\_\{\\text\{err\}\}/n\_\{\\text\{call\}\},nredun/ncalln\_\{\\text\{redun\}\}/n\_\{\\text\{call\}\},nempty/nsearchn\_\{\\text\{empty\}\}/n\_\{\\text\{search\}\},nbad/nreln\_\{\\text\{bad\}\}/n\_\{\\text\{rel\}\}, andnrec/nbadn\_\{\\text\{rec\}\}/n\_\{\\text\{bad\}\}\.

### 4\.2The need for model\-based judging

Two model\-graded sources are kept apart\. The user simulator rates one axis inline as the conversation unfolds,user frustration, reduced per trial to the worst turn\. It reports the experience of the customer it plays\. As frustration rises, the shopper grows terser and abandons sooner\. Separately, anLLM judgeper metric re\-reads the finished transcript against written rubrics, writing an evidence\-based rationale before emitting a score that we normalize to\[0,1\]\[0,1\]\([Liu et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib14)\)\.

Thehallucinationrubric applies this division within one response\. Following claim\-decomposition approaches\([Min et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib18);[Chern et al\., 2023](https://arxiv.org/html/2609.16093#bib.bib19)\), one structured call extracts each concrete claim from the agent’s prose and types it as action, numeric, or attribute\. The verifier labels an action claim a phantom action when the tool log contains no matching call\. It checks numeric claims against catalog prices, stock, and the state\-replayed cart\. Only attribute claims reach the judge, which labels them as supported, embellished, contradicted, or ungrounded against the ground\-truth catalog row\. An ungrounded claim also detects a fabricated product\. WithVVthe verifiable claims andB⊆VB\\subseteq Vthe violations, the headline score is

H=1−\|B\|\|V\|\.H=1\-\\frac\{\|B\|\}\{\|V\|\}\.The verifier reports the three channel rates separately\. Table[3](https://arxiv.org/html/2609.16093#S5.T3)shows two agents differing fourfold on the attribute channel while agreeing on the numeric one\.

The remaining rubrics assess whether each reply moved the purchase forward, whether the agent remembered and understood the customer’s request, whether it asked for information it already had, whether recommendations were relevant, and whether it confirmed variant and quantity before adding to the cart\. The clarification and helpfulness rubrics receive the agent’s system prompt and treat policy\-mandated clarification as compliant behavior\.

### 4\.3Tool\-call correctness

The verifier must distinguish a missed action from a reasonable additional lookup, while treating an unjustified cart mutation more seriously than an extra read\. The reference set comes from the simulator, which predicts the immediate chain of calls the agent should make in response to each customer message\. LetEEbe that prediction andAAthe calls actually issued\. We group both by tool name and first apply the cheapest checks\. A pair matches with no model call when its arguments are an exact JSON match or the tool takes no arguments\. Only same\-name pairs with differing arguments are queued for the judge, along with unmatched actual calls, so the judge decides whether a call was justified rather than spurious\. All queued questions across the dialog are answered in one batched call, and assignment is greedy and order\-stable so a rerun reproduces it\. The tool category determines the penalty for surviving extras\. Cart writes get no free allowance, read\-only lookups are forgiven in bulk, and within a category the cost grows with each further extra\. Writingwextraw\_\{\\text\{extra\}\}for the summed penalty, the per\-turn score issturn=matched/\(\|E\|\+wextra\)s\_\{\\text\{turn\}\}=\\text\{matched\}\\,/\\,\(\|E\|\+w\_\{\\text\{extra\}\}\), so a miss shrinks the numerator and an unjustified extra inflates the denominator\. Turns with neither expected nor actual calls are skipped\. Appendix[E](https://arxiv.org/html/2609.16093#A5)gives the category weights\.

### 4\.4Aggregation

The harness reports shopping and grocery trajectory and conversational metrics separately\. Each rubric declares its applicable trial kinds, and the runner skips inapplicable metrics rather than recording zero; Appendix[J](https://arxiv.org/html/2609.16093#A10)explains how this applicability map handles refusal probes\. Aggregates report the mean, sample standard deviation, and valid count, with Boolean means interpreted as rates\.

## 5Verifiable simulation for e\-commerce agents

Eight open\-weight agents run the same160160\-trial sweep,100100general shopping,5050grocery, and1010refusal probes, across difficulties11–1010, with Claude Sonnet 4\.5 as both user simulator and primary judge\([Anthropic, 2025](https://arxiv.org/html/2609.16093#bib.bib29)\)\. Every run uses the same tools, scoring contract, controller, and stopping policy, with model\-specific native chat templates and parsers\. The comparison is conditional on this configuration because context construction, tool mediation, and stopping rules are part of the experimental condition\([Zhang et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib22)\)\. For each agent, we resample its100100shopping trials with replacement60006000times and report the central95%95\\%of the resulting exact\-cart success rates\. For reported between\-agent contrasts, we independently resample the two unpaired runs and take the central95%95\\%of the rate difference\. We use the agents to test what the e\-commerce environment can establish from its stored cart, catalog, reveal schedule, and tool trace\. Appendix[J](https://arxiv.org/html/2609.16093#A10)describes the refusal cohort, Appendix[H](https://arxiv.org/html/2609.16093#A8)gives checkpoint identities, and Appendix[M](https://arxiv.org/html/2609.16093#A13)gives extended shopping metrics\.

### 5\.1Verification ablation

We test four retained\-evidence conditions on each completed trajectory\. Final cart contains terminal product quantities and the target comparison\. Transcript contains the customer and assistant messages, executed tool calls, and results, but excludes the hidden target and other environment state\. State adds the target cart, control catalog, and termination record, enabling state replay\. Full simulator adds the reveal schedule, per\-turn expected tool calls, and the user simulator’s frustration record\. For ten evaluation questions, Table[2](https://arxiv.org/html/2609.16093#S5.T2)labels each answer as Rule, Model, Hybrid, or Infer\. Rule uses fixed verification, Model uses grounded output from the user simulator or judge, and Hybrid combines a model\-derived signal with a fixed comparison or reduction\. Infer lacks the retained evidence required for verification\.

Table 2:Verification ablation across the four retained\-evidence conditions defined in the text\.Across fixed trajectories, the final cart establishes terminal outcome questions\. With environment state, the verifier also establishes progress, executed actions, catalog\-grounded claims, and termination reasons\. With the full simulator record, it further checks request\-relative tool appropriateness and search success and reports the user simulator’s recorded frustration\. Appendix[F](https://arxiv.org/html/2609.16093#A6)gives three trial\-level examples of these distinctions\.

### 5\.2E\-commerce state turns outcomes into verifiable diagnoses

Exact\-cart success ranges from0\.300\.30to0\.570\.57, and the independently resampled contrasts separate the highest rate from the two lowest\. The tier breakdown shows why these outcomes do not form a stable rank order\. Gemma4\-26B moves from0\.590\.59on direct purchases to0\.270\.27on light exploration and0\.530\.53on complex baskets, while GPT\-OSS moves from0\.590\.59on direct purchases to0\.420\.42on light exploration and0\.670\.67on multi\-item baskets\. We therefore compare capability profiles rather than rank agents by one outcome\. Appendix[L](https://arxiv.org/html/2609.16093#A12)reports the full estimates and confidence intervals\.

The retained evidence distinguishes failures that a transcript review or final\-cart rate would merge\. Table[3](https://arxiv.org/html/2609.16093#S5.T3)reports the diagnostic channels, and Figure[2](https://arxiv.org/html/2609.16093#S5.F2)combines six of them for four representative agents\.

Exact cartCart precisionCart recallWithin budgetClaimgroundingCustomerexperience578777739488498985729485549574727674309362377567Qwen3\.6 35BQwen3\.5 35BGPT\-OSS 20BQwen3 30B

Figure 2:Verification profiles for four representative agents\. Labels report percentages, and all axes run from00to100%100\\%with higher better\.*Within budget*is one minus turn\-budget exhaustion\.*Claim grounding*is one minus the largest hallucination\-channel rate\.*Customer experience*is one minus mean trial\-level maximum user frustration\.#### Target\-cart evidence separates discovery from over\-purchase\.

Qwen3\.5\-35B reaches the highest recall in the sweep \(0\.850\.85\) and the highest goal\-achievement rate \(0\.660\.66\), but its reconstructed final cart has precision0\.890\.89, below five agents it beats on goal achievement\. It searched and added aggressively, issuing44\.144\.1tool calls per conversation against a field median near2525\. The stored target cart and state replay identify the shopping failure\. The assistant found items the customer wanted, then added items the target cart did not contain\. A final success rate would not distinguish broad discovery from over\-purchase\.

#### Catalog evidence separates unsupported descriptions from wrong arithmetic\.

GPT\-OSS and Qwen3\-30B fabricate product attributes at roughly four times the rate of the other six \(0\.2360\.236and0\.2520\.252against a cluster near0\.0550\.055\), while their numeric claims, checked against catalog prices and reconstructed cart state, stay ordinary \(0\.0330\.033and0\.0290\.029\)\. The catalog and tool trace localize the problem to unsupported descriptive claims rather than price, quantity, or total calculations\. Qwen3\-30B also has the highest phantom\-action rate \(0\.1580\.158\), announcing actions with no matching tool call\.

#### Tool traces separate low activity from efficient shopping\.

Qwen3\-30B issues the fewest calls in the sweep at15\.415\.4and is also the lowest exact\-cart agent\. Its transcripts show roughly one call per customer turn, and63%63\\%of trials exhaust the turn budget without the actions needed to close the cart\. The tool trace and termination record show that its low call count reflects under\-action rather than economy\. The verifier also identifies premature cart writes, including GPT\-OSS additions made before customer confirmation on15\.3%15\.3\\%of additions\. Search traces add a separate relevance check\. Catalog search returned candidates for every query in the sweep, so the bad\-search rates in Table[3](https://arxiv.org/html/2609.16093#S5.T3)measure relevance rather than empty retrievals\.

#### Termination records distinguish pace failures\.

Gemma4\-26B reaches exact\-cart success on0\.270\.27of light\-exploration trials and0\.530\.53of complex trials\. It exhausts the budget on2222of3333light\-exploration trials while issuing a median1\.141\.14calls per turn, but exhausts only11of1717complex trials at2\.042\.04calls per turn\. Qwen3\.5\-35B shows the opposite failure on complex baskets, issuing a median6666calls while44of1717trials end in abandonment\. These records distinguish agents that must move faster from agents that must use tools more selectively\. Appendix[L](https://arxiv.org/html/2609.16093#A12)gives the full analysis\.

Table 3:Diagnostic channels\. Lower is better for the failure\-rate columns; call volume is descriptive\. The three hallucination channels come from one metric split by claim type\. Action and numeric claims are checked deterministically; attribute claims are checked by a judge\.
#### Cross\-judge check\.

We rescore300300matched shopping and grocery trials from Qwen3\-30B and Qwen3\.6\-27B with GPT\-5\.4\. Across five completed rubrics, the judges agree most closely on tool\-call correctness \(mean difference0\.0020\.002, MAE0\.0300\.030,r=0\.924r=0\.924\)\. Clarification has the largest MAE \(0\.1110\.111\) and lowest correlation \(r=0\.490r=0\.490\), while GPT\-5\.4 scores hallucination0\.0520\.052lower on average\. This check covers two agents and does not calibrate either judge against human labels\. Appendix[K](https://arxiv.org/html/2609.16093#A11)reports the full protocol and results\.

## 6Limitations

The main eight\-agent sweep uses one primary judge\. The two\-agent check in Appendix[K](https://arxiv.org/html/2609.16093#A11)covers five rubrics and excludes three whose secondary outputs were incomplete; neither judge has been calibrated against human labels\. The attribute\-hallucination and conversational scores therefore remain evaluator\-dependent rather than externally validated customer judgments\. Tool\-call correctness also depends on the simulator’s expected\-call prediction\. Errors in that prediction can change the score even when the assistant’s tool trace is fixed\.

Environment\-grounded verification requires the target cart and reveal schedule to be fixed before a conversation begins\. That structure lets the evaluator check cart outcomes and date actions against customer requests, but it cannot reproduce a shopper whose preferences remain unsettled, change during the interaction, or allow several acceptable products\. The schedule fixes the scoring boundary; it does not make the simulator a human proxy\. The user simulator also knows the hidden target and is directed to continue toward it\. After an assistant fails to retrieve an item, we observe that the simulator can disclose an exact product name or another highly discriminating constraint that makes a further search easy\. It can therefore rescue a poor search and act more cooperatively than a customer on live traffic\. In aτ\\tau\-bench human comparison, general\-purpose simulators were more cooperative and produced higher agent success than human users in most settings\([Zhou et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib30)\)\. With a GPT\-4o agent fixed acrossτ\\tau\-bench retail tasks, changing the user model shifted success by nearly nine percentage points\([Seshadri et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib31)\)\.

The turn budget is a benchmark convention rather than an estimate of customer patience\. It supports comparisons under one stopping policy, but another budget or tool\-mediation policy may change the measured profiles\. The checkpoint repository identifiers are recorded, while immutable repository revisions are not; exact weight provenance is therefore limited to the identifiers reported in Appendix[H](https://arxiv.org/html/2609.16093#A8)\. The synthetic catalog and standardized tool surface leave out conditions that shape live retail interactions\. Prices and inventory do not change, and the environment does not include merchant\-specific policies, delivery and returns, competing listings, or browser and interface friction\. The results therefore characterize controlled, tool\-mediated catalog shopping rather than performance on live retail traffic\.

## 7Conclusion

Final\-cart success is insufficient for evaluating e\-commerce agents because it does not identify the route an agent took to reach, miss, or exceed a customer’s intended cart\. We propose an e\-commerce evaluation environment that precommits the target cart and reveal schedule against a fixed control catalog, then retains the tool trace and environment state produced during the trial\. The verifier uses this evidence to establish cart correctness, search quality, action timing, and numeric catalog claims\. Model\-based judges assess attribute support and other questions that require language\.

Across eight open\-weight agents, we use the resulting capability profiles to identify distinct repair targets in tool use, cart management, catalog grounding, and retrieval\. E\-commerce evaluations should report these distinctions when deployment and repair decisions depend on how an agent fails\. This controlled, tool\-mediated design provides a reproducible pre\-deployment test and complements validation on live retail traffic\.

The broader lesson is that reliable agent verification requires environments to preserve the evidence needed to evaluate intermediate actions against the state and information available when those actions were taken\. Environment\-grounded verification can therefore turn terminal outcomes into actionable diagnoses and provide a general framework for evaluating how agents behave, not only whether they succeed\.

## References

- Anthropic \(2025\)AnthropicClaude sonnet 4\.5 system card\.External Links:[Link](https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf)Cited by:[§5](https://arxiv.org/html/2609.16093#S5.p1.1)\.
- Chenet al\.\(2024\)S\. Chen, S\. Wiseman, and B\. DhingraChatShop: interactive information seeking with language agents\.External Links:2404\.09911Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p2.1)\.
- Chernet al\.\(2023\)I\. Chern, S\. Chern, S\. Chen, W\. Yuan, K\. Feng, C\. Zhou, J\. He, G\. Neubig, and P\. LiuFacTool: factuality detection in generative AI: a tool augmented framework for multi\-task and multi\-domain scenarios\.arXiv preprint arXiv:2307\.13528\.Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1),[§4\.2](https://arxiv.org/html/2609.16093#S4.SS2.p2.1)\.
- Denget al\.\(2023\)X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. SuMind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,Note:arXiv:2306\.06070Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p2.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.InFirst Conference on Language Modeling,Note:arXiv:2404\.04475Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1)\.
- Gemma Team \(2026\)Gemma TeamGemma 4 technical report\.External Links:2607\.02770Cited by:[Appendix H](https://arxiv.org/html/2609.16093#A8.p1.1)\.
- Houet al\.\(2026\)Y\. Hou, J\. Li, X\. Fu, Z\. He, A\. Yan, X\. Chen, and J\. McAuleyBridging language and items for retrieval and recommendation: benchmarking LLMs as semantic encoders\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 3251–3265\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.147),[Link](https://aclanthology.org/2026.acl-long.147/)Cited by:[Appendix A](https://arxiv.org/html/2609.16093#A1.p1.1)\.
- Kirgiset al\.\(2026\)P\. Kirgis, S\. Kapoor, S\. Rabanser,et al\.Log analysis is necessary for credible evaluation of AI agents\.arXiv preprint arXiv:2605\.08545\.Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p3.1)\.
- Linget al\.\(2026\)Y\. Ling, C\. Yuan, S\. Dong, Y\. Yang, N\. Mocherla, and A\. GoyalShopperBench: a benchmark for personalized shopping with persona\-guided simulation\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 5: Industry Track\),pp\. 837–846\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-industry.62)Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p2.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1),[§4\.2](https://arxiv.org/html/2609.16093#S4.SS2.p1.1)\.
- Luet al\.\(2025\)J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, H\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. PangToolSandbox: a stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 1160–1183\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.65)Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p3.1)\.
- Lùet al\.\(2025\)X\. H\. Lù, A\. Kazemnejad, N\. Meade, A\. Patel, D\. Shin, A\. Zambrano, K\. Stańczak, P\. Shaw, C\. J\. Pal, and S\. ReddyAgentRewardBench: evaluating automatic evaluations of web agent trajectories\.arXiv preprint arXiv:2504\.08942\.Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1)\.
- Maet al\.\(2024\)C\. Ma, J\. Zhang, Z\. Zhu, C\. Yang, Y\. Yang, Y\. Jin, Z\. Lan, L\. Kong, and J\. HeAgentBoard: an analytical evaluation board of multi\-turn LLM agents\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,Note:arXiv:2401\.13178Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p3.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12076–12100\.Note:arXiv:2305\.14251Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1),[§4\.2](https://arxiv.org/html/2609.16093#S4.SS2.p2.1)\.
- OpenAI \(2025\)OpenAIgpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925Cited by:[Appendix H](https://arxiv.org/html/2609.16093#A8.p1.1)\.
- Peeterset al\.\(2026\)R\. Peeters, A\. Steiner, L\. Schwarz, J\. Y\. Caspary, and C\. BizerWebMall: a multi\-shop benchmark for evaluating web agents\.InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval,Note:arXiv:2508\.13024Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p2.1)\.
- Qinet al\.\(2025\)T\. Qin, F\. Bai, T\. Hu,et al\.COMPASS: benchmarking constrained optimization in LLM agents\.arXiv preprint arXiv:2510\.07043\.Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p3.1)\.
- Qwen Team \(2026a\)Qwen TeamQwen3\.5: towards native multimodal agents\.Note:Qwen release postExternal Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[Appendix H](https://arxiv.org/html/2609.16093#A8.p1.1)\.
- Qwen Team \(2026b\)Qwen TeamQwen3\.6\-27B: flagship\-level coding in a 27b dense model\.Note:Qwen release postExternal Links:[Link](https://qwen.ai/blog?id=qwen3.6-27b)Cited by:[Appendix H](https://arxiv.org/html/2609.16093#A8.p1.1)\.
- Qwen Team \(2026c\)Qwen TeamQwen3\.6\-35B\-A3B: agentic coding power, now open to all\.Note:Qwen release postExternal Links:[Link](https://qwen.ai/blog?id=qwen3.6-35b-a3b)Cited by:[Appendix H](https://arxiv.org/html/2609.16093#A8.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 3982–3992\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1410)Cited by:[§3\.1](https://arxiv.org/html/2609.16093#S3.SS1.SSS0.Px1.p1.1)\.
- Seshadriet al\.\(2026\)P\. Seshadri, S\. Cahyawijaya, A\. Odumakinde, S\. Singh, and S\. Goldfarb\-TarrantLost in simulation: LLM\-simulated users are unreliable proxies for human users in agentic evaluations\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 47423–47439\.Note:arXiv:2601\.17087Cited by:[§6](https://arxiv.org/html/2609.16093#S6.p2.1)\.
- Wanget al\.\(2026\)J\. Wang, K\. Xiao, Q\. Sun, H\. Zhao, T\. Luo, J\. D\. Zhang, and X\. ZengShoppingBench: a real\-world intent\-grounded shopping benchmark for LLM\-based agents\.InProceedings of the AAAI Conference on Artificial Intelligence,Note:arXiv:2508\.04266Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p2.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9440–9450\.Note:arXiv:2305\.17926Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui,et al\.Qwen3 technical report\.External Links:2505\.09388Cited by:[Appendix H](https://arxiv.org/html/2609.16093#A8.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,Note:arXiv:2207\.01206Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p1.1)\.
- Yaoet al\.\(2025\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InInternational Conference on Learning Representations,Note:arXiv:2406\.12045Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p3.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, J\. Wang, Y\. Ge, W\. Xu, J\. Hamm, and C\. K\. ReddyStop comparing LLM agents without disclosing the harness\.External Links:2605\.23950Cited by:[§5](https://arxiv.org/html/2609.16093#S5.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,Note:arXiv:2306\.05685Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p4.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Note:arXiv:2307\.13854Cited by:[§2](https://arxiv.org/html/2609.16093#S2.p2.1)\.
- Zhouet al\.\(2026\)X\. Zhou, W\. Sun, Q\. Ma,et al\.Mind the sim2real gap in user simulation for agentic tasks\.InConference on Language Modeling,Note:arXiv:2603\.11245Cited by:[§6](https://arxiv.org/html/2609.16093#S6.p2.1)\.

## Appendix APer\-retailer catalog composition

Table[4](https://arxiv.org/html/2609.16093#A1.T4)gives the within\-store share of each canonical category for the six retailers aggregated in Section[3](https://arxiv.org/html/2609.16093#S3)\. We derive the Amazon shares from published category counts for products observed in Amazon Reviews 2023 records from May 1996 through September 2023\[[Hou et al\., 2026](https://arxiv.org/html/2609.16093#bib.bib1)\]\. We exclude itsUnknownbucket, which contains approximately13\.213\.2million of48\.1948\.19million items, and renormalize the3333named source categories to100%100\\%\. The other five retailer shares derive from live department listing counts scraped from public storefronts\. We map each retailer’s taxonomy to the shared categories and assign a0%0\\%share when a retailer does not carry a category\. Categories carried by fewer than three retailers are folded into*Other*\. We then average the six within\-retailer shares for each shared category and normalize the aggregate vector to100%100\\%\. The analysis retains aggregate department counts and normalized shares; it does not redistribute reviews, product records, or storefront pages\.

Table 4:Within\-store category share \(%\) for each retailer and the aggregate used to populate the control catalog\.∗Walmart’s Home & Kitchen count excludes made\-to\-order listings\. Digitally customizable frames account for a large share of that department, and counting them would distort the category mix of a general marketplace\.

The retailer columns in Table[4](https://arxiv.org/html/2609.16093#A1.T4)contain different category concentrations, while the aggregate distributes mass across the shared taxonomy\. We average across these differences so no single retailer’s catalog structure determines the control catalog\.

## Appendix BBasket construction parameters

Scenario construction is deterministic from one integer seed\. Reveal style, journey, waves, basket breadth and depth, quantities, and probe contents use separate keyed sub\-generators\. Persona traits and attribute dropout instead share one ordered generator\.

#### Price weighting and depth\.

A category’s representative is drawn with weightexp\(−\(log10p−μ\)2/2σ2\)\\exp\\\!\\big\(\-\(\\log\_\{10\}p\-\\mu\)^\{2\}/2\\sigma^\{2\}\\big\), whereμ\\muis the median log\-price of the category’s in\-stock rows andσ=0\.6\\sigma=0\.6\. The spread is wide enough that the bulk of a category is sampled almost flatly and only the far tails are suppressed, which reproduces mid\-range items being bought about as often as one another while a flagship is rare\. The remainingDj−1D\_\{j\}\-1items in the slot are filled by the same weighting, drawn without replacement\. The geometric depth of Section[3](https://arxiv.org/html/2609.16093#S3)is truncated at1212products from any one category\.

#### Breadth slots and distinct SKUs\.

TheBBcategory slots are drawn with replacement, so a category can come up more than once and the basket runs deeper there than breadth alone would give\. A product\-id exclusion set spans the whole basket\. Each slot considers only in\-stock rows not already taken, and each drawn product is added to the set before the next slot\. A repeated category therefore runs a fresh representative and fill pass over what is left rather than duplicating a line, and every goal item names a distinct SKU\. A slot whose category has no rows left contributes nothing, which is why the turn budget of Section[3](https://arxiv.org/html/2609.16093#S3)is computed over the realized product countmmrather than the drawn breadth\.

#### Quantities\.

Each selected product receives a quantityq∈\[1,Qmax\]q\\in\[1,Q\_\{\\max\}\]drawn from

w⁡\(q\)∝\(q\+c\)−α⋅exp\(−max\(q−k,0\)/τ\)⏟grocery taper only\.w\(q\)\\;\\propto\\;\(q\+c\)^\{\-\\alpha\}\\,\\cdot\\,\\underbrace\{\\exp\\\!\\big\(\-\\max\(q\-k,\\,0\)/\\tau\\big\)\}\_\{\\text\{grocery taper only\}\}\.Non\-grocery items use\(c,α\)=\(0,2\.7\)\(c,\\alpha\)=\(0,2\.7\)with no taper, sow⁡\(q\)∝q−2\.7w\(q\)\\propto q^\{\-2\.7\}falls steeply fromq=1q=1, withw⁡\(1\)/w⁡\(2\)=22\.7≈6\.5w\(1\)/w\(2\)=2^\{2\.7\}\\approx 6\.5\. The shopper almost always takes one\. Grocery staples use\(c,α\)=\(4,2\.5\)\(c,\\alpha\)=\(4,2\.5\), where the offset keeps\(q\+c\)\(q\+c\)nearly constant across smallqqand flattens the head, givingw⁡\(1\)/w⁡\(2\)=\(6/5\)2\.5≈1\.6w\(1\)/w\(2\)=\(6/5\)^\{2\.5\}\\approx 1\.6so quantities one through five are comparably likely\. The taper pastk=5k=5withτ=1\\tau=1suppresses implausible party\-size counts\.

## Appendix CPersona dimensions

The simulated customer is drawn along six independent categorical dimensions \(Table[5](https://arxiv.org/html/2609.16093#A3.T5)\), each sampled from the trial seed with realistic\-frequency weights, somoderatespecificity andconversationalstyle are common whilegarbledinput is rare\. The sampled traits are written into the persona prompt the simulator follows\.

Table 5:The six persona dimensions\.
## Appendix DReveal\-style sampling and journey construction

A basket of fewer than two products has nothing to sequence and is triviallybatch, except for a complex non\-grocery trial that drew a single item, which falls back tolinear\. Everything larger draws a style from Table[6](https://arxiv.org/html/2609.16093#A4.T6), keyed on the realized basket size and the trial kind rather than the tier label alone, since the category\-first picker can hand a nominally light trial several products\.

Table 6:Reveal\-style draw weights for baskets of two or more products\. Grocery never drawsnonlinear; the invariant is enforced twice, once in the weights and once as a guard\.#### Micro\-batching\.

Abatchtrial carrying more than four products is partitioned once at build time into ordered waves\. The first wave holds two items, and each later wave targets a length drawn uniformly in\[2,5\]\[2,5\]\. Outside the complex tier the partition respects category contiguity, packing whole categories toward each wave’s target and cutting a category larger than the maximum wave into random pieces; complex trials shuffle the items first\. The controller releases the next wave only once every item in the current one is in the cart, so the revealed set grows a wave per turn rather than all at once\.

#### Nonlinear beat construction\.

Each goal item receives one essential add beat\. When optional\-beat budget remains, the generator inserts an introduction and may add an exploration beat\. It defers each nonfinal item with probability0\.50\.5while optional\-beat budget remains\. Before introducing a subsequent item, it may select one pending item uniformly for an early revisit; after processing the list, it returns to any items still pending in shuffled order\. The resulting itinerary can therefore interleave unresolved items without following either basket order or a first\-in, first\-out return order\.

#### Per\-turn tool rounds\.

The ceiling on tool\-call rounds within a single agent turn is1010for the sequential styles, which trickle roughly one item per turn\. A batch reveal can land a whole wave at once, so it earns two further rounds, one search and one add, for every item that can arrive together, capped by the largest wave\. A trial with a large micro\-batched basket therefore allows up to2020rounds in a turn\. The generous ceiling stops a runaway turn from pinning a worker, and a turn that hits it is recorded as an agent error\.

#### Beat compression\.

A nonlinear itinerary is compiled to fit the turn budget, withnnessential add beats always emitted and flavor beats consuming only themax⁡\(0,max\_turns−n\)\\max\(0,\\text\{max\\\_turns\}\-n\)turns left over\. Because a trial can still fall behind at run time, the controller drops flavor beats from the front of the remaining itinerary whenever the turns left no longer exceed the essential adds whose items are still missing from the cart, so the basket stays completable under any pace the agent sets\.

### D\.1Recorded reveal journeys

The following three examples retain the recorded customer and assistant wording, with…\\ldotsmarking omitted text or turns\. The controller recorded exact\-cart success and goal completion for all three interactions\.

#### Example 1: nonlinear deferral, pivot, and return\.

The customer in the Gemma4\-31B interaction moves among a thermal tee, trousers, and loungewear before closing all three\. The excerpt shows the controller maintaining unresolved items across several natural deferrals and returns\.

> Customer:“so yeah the thermel v\-neck thing is definately top priority but actualy now that im thinkin about it i might also need some trousers too …anyway lets start with the long sleev tee first tho” Customer:“hmm yeah those both look pretty solid actualy, lemme think on those for a sec tho\. so in the meantime i was also thinkin i need some trousers like i mentioned earlier …” Customer:“but hold on lemme ask you somthing about the thermel v\-neck tees you showed me earlier …also just so i dont forget later i probly need some loungewear too …” *\[five turns omitted\]* Customer:“but actualy hold on, i just remembered i was still thinkin about those trousers earlier …so yeah lets go with those trousers, and also that terry cloth lounge set …” Customer:“and yeah i think im gonna grab that northfield basics thermel v\-neck too actualy …so yeah all three of those, lets get em in the cart” Assistant:“Done and done\! I’ve added all three to your cart: the Grid Check trousers in Large, the Terry Cloth lounge set in Medium, and that midweight thermal v\-neck in Medium\.”

#### Example 2: a large cart in micro\-batches\.

The reveal schedule for Qwen3\.6\-35B partitions seven target items into waves of\[2,2,2,1\]\[2,2,2,1\]\. The excerpt shows exact quantities and new categories arriving in bounded groups rather than as one opening list\.

> Customer:“I need unsweetened almond milk in the larger 1\.89 L format, preferably non\-GMO, and I’m also after an organic plain oat yogurt around 400 g\. I’ve got a few other categories on my list too, but let’s start with these\.” Customer:“Actually, I’d like to go with the Meadowblend almond milk instead of the Dawnfield …Can you add five units of the Meadowblend almond milk and three of the Rootwell oat yogurt to my cart?” Customer:“Great\! Now I also need one unit of unsweetened coconut milk, preferably organic in a 400 mL format, and two units of cashew\-style aged vegan cheese around 150 g …” Customer:“Perfect\! Next on my list: I need six units of a photographic cityscape framed print, preferably in monochrome with a black metal frame around 60x90 cm\. And I also need one unit of a brightening vitamin C eye cream for normal skin\.” Customer:“Last thing on my list today: I need one unit of a modular desk organizer in navy leatherette\. Something that can keep my workspace tidy and looks professional\.” Assistant:“Perfect\! I’ve added the last item to your cart: …Your complete cart now includes: …You’re all set\!”

#### Example 3: one completed choice at a time\.

The reveal schedule for Qwen3\.6\-35B introduces a flannel, a polo, and pruning shears in sequence\. Each new request follows the preceding choice, providing a direct contrast with wave\-based and nonlinear disclosure\.

> Customer:“I’m looking for a men’s relaxed\-fit flannel shirt with a button\-down collar, ideally in a gingham check pattern or something similar …” Customer:“Perfect, that’s exactly what I need …Yes, please add that to my cart\. Now, I’m also looking for a men’s slim\-fit polo shirt with a spread collar …” Customer:“The Coastal Breeze at $28\.33 is perfect …Please add that one to my cart in Medium\. Now, I also need to grab a pair of forged\-steel pruning shears for my garden …” Customer:“Yes, absolutely …Please add those to my cart as well\. That should do it for me today …can you give me a quick rundown of what’s in my cart and the total?” Assistant:“Perfect\! Here’s everything in your cart: …Alpine Flannel Relaxed Weekend Shirt …Coastal Breeze Spread\-Collar Polo …CrispCut Forged Pruning Shears …”

## Appendix ETool categories and extra\-call penalties

Each tool’s category is a fixed property read from the tool registry, which is how the tool\-call\-correctness metric decides whether a call mutates the cart\. Thekk\-th unjustified extra past a category’s free allowance costsinitial\+\(k−1\)⋅increase\\text\{initial\}\+\(k\-1\)\\cdot\\text\{increase\}, so per\-extra cost grows linearly and the cumulative penalty grows quadratically\. We report the registry mapping and penalty schedule in Table[7](https://arxiv.org/html/2609.16093#A5.T7)\. An unmatched cart mutation incurs the full penalty from its first occurrence, while read\-only and exploratory calls receive limited allowances\.

Table 7:Extra\-call penalty by tool category\.
## Appendix FVerification case studies

The following three examples show how the verifier connects a short customer exchange to the evidence that supports each result\. We retain product identifiers because the tool log and catalog rows use them as stable keys\.

#### Example 4: exact cart with an unsupported description\.

The customer first asked GPT\-OSS for caramel candies and then requested black licorice\. The assistant recommended “Twisted, slightly crunchy twists with a hint of anise and fennel” for the productGC031\. The catalog description for that product states that the twists are chewy and does not support the crunchy\-texture claim\. The customer instead selectedGC010; state replay recovered the requested final cart exactly: threeGC012, twoGC009, and fourGC010\. The trial therefore has exact\-cart success1\.01\.0and zero numeric or phantom\-action violations, while the attribute checker records an attribute\-hallucination rate of1\.01\.0\. A final\-cart score reports success; the catalog\-grounded claim check identifies a separate repair target\.

> Customer:“One more thing\. I’d also like to get some black licorice\. Do you carry any of those?” Assistant:“Amber Canopy Black Licorice Twists …Twisted, slightly crunchy twists with a hint of anise and fennel\.” Verifier:the catalog row describes the twists as chewy; it contains no support for “slightly crunchy\.”

#### Example 5: claimed cart action without a tool action\.

The customer asked Qwen3\-30B to add a Ridgeback trucker cap and a merino beanie\. The assistant replied that both had been added\. The following customer message reported that the cart did not change\. Before the assistant’s completion claim, the tool trace contains no matching cart\-addition call\. Later cart\-addition calls occur, but they cannot support the earlier claim that both additions were already complete\. The verifier therefore labels both action claims as phantom\. The trial ends at the turn budget with recall0\.250\.25, cart precision0\.500\.50, phantom\-action rate1\.01\.0, and attribute\-hallucination rate0\.750\.75\. This distinction requires the tool trace and state replay; the assistant’s confirmation alone is insufficient evidence that the cart had changed at that point\.

> Customer:“Can you just add them both in?” Assistant:“Ridgeback Performance Trucker …added\. Merino Wool Ribbed Ski Beanie …added\.” Tool record:no matching cart\-mutation call precedes the completion claim; later cart calls do occur\.

#### Example 6: goal achievement with substantial over\-purchase\.

The customer opened the Qwen3\.5\-35B interaction with “hey need some shirts n sneakers” and later selected individual shirt and sneaker options\. The final cart satisfied every target requirement, so goal achievement was true\. State replay nevertheless found2828distinct products in the cart for a1515\-SKU target, including99units ofSN453where the target requested one\. The final cart had recall1\.01\.0but precision0\.3440\.344, so exact\-cart success was false\. The case distinguishes discovering all wanted products from managing the cart conservatively\.

## Appendix GVerification cost

We separate total token volume from the metered cost of post\-hoc judging in Tables[8](https://arxiv.org/html/2609.16093#A7.T8)and[9](https://arxiv.org/html/2609.16093#A7.T9)\. The agent under test accounts for more tokens per trial than the user simulator and judge combined, although those tokens come from the self\-hosted inference path\.

Across the eight\-model sweep, the assistant averages417417K tokens per trial, while the user simulator and judge add347347K\. Metered simulator and judge calls cost $94 per160160\-trial run on average \($0\.59 per trial\)\.

Table 8:Mean tokens per trial \(thousands\), averaged over the eight\-model sweep\.Median per\-trial totals run to about146146K tokens for a general shopping trial and135135K for a grocery basket, but only4040K for a refusal probe\. The per\-trial mean of417417K sits well above these medians because a heavy tail of many\-round baskets pulls it up\.

Hallucination and tool\-call correctness are the two largest judge costs in Table[9](https://arxiv.org/html/2609.16093#A7.T9)\. Both operate on long, tool\-rich conversations, while the refusal rubric runs on only ten probes per model\.

Table 9:Per\-rubric judge cost, median over the eight\-model sweep\.*Tokens/call*is median input plus output per judge call;*Out\. share*is the generated fraction, which costs more per token;*USD/run*is the median cost across a160160\-trial run\.∗Cheap only because it runs over a1010\-probe cohort rather than the full sweep; its per\-call footprint matches the other whole\-conversation rubrics\.

## Appendix HModel identity and latency

The sweep uses the base checkpointsopenai/gpt\-oss\-20b,google/gemma\-4\-26B\-A4B\-it,google/gemma\-4\-31B\-it,Qwen/Qwen3\-30B\-A3B,Qwen/Qwen3\.5\-35B\-A3B,Qwen/Qwen3\.5\-27B,Qwen/Qwen3\.6\-35B\-A3B, andQwen/Qwen3\.6\-27B\. Official reports and cards document these model families and their architectures\[[OpenAI, 2025](https://arxiv.org/html/2609.16093#bib.bib23),[Yang et al\., 2025](https://arxiv.org/html/2609.16093#bib.bib24),[Qwen Team, 2026a](https://arxiv.org/html/2609.16093#bib.bib25),[Qwen Team, 2026c](https://arxiv.org/html/2609.16093#bib.bib26),[Qwen Team, 2026b](https://arxiv.org/html/2609.16093#bib.bib27),[Gemma Team, 2026](https://arxiv.org/html/2609.16093#bib.bib28)\]\.

We report both per\-round\-trip and per\-trial latency in Table[10](https://arxiv.org/html/2609.16093#A8.T10)\. The five sparse\-mixture agents occupy the five fastest per\-round\-trip rows, while the three dense agents require more time for each exchange\. Per\-trial latency also depends on how many exchanges an agent uses, so it does not follow the per\-round\-trip ordering exactly\.

Each run uses tensor parallelism across two 80GB NVIDIA H100 GPUs and takes1\.81\.8–10\.410\.4wall\-clock hours \(mean3\.33\.3\), for53\.353\.3H100\-hours across the eight sweeps\.

Table 10:Agent latency, ordered by per\-round\-trip time\.*Active*is the rounded number of parameters activated per token reported in first\-party model documentation\. Every sparse\-mixture model is faster per round\-trip than every dense model, so nominal size does not order latency\.
## Appendix IThe metric battery

Each run reports the4444metrics in Table[11](https://arxiv.org/html/2609.16093#A9.T11), every one of them a mean over the trials that produced a valid score\. The harness stamps the scoring\-contract version on each transcript and score artifact\. Keys are the field names in the run summary files, so each row traces to the artifact it came from\. Termination reasons are counted as they occur rather than declared in advance, so the five below are the reasons this sweep produced; the sixth the harness can record, a failed opening message, never fired\. We group deterministic outputs, the user simulator’s inline rating, and post\-hoc judge scores separately\. Tables[15](https://arxiv.org/html/2609.16093#A13.T15)and[16](https://arxiv.org/html/2609.16093#A13.T16)report the subset that varies across agents\.

Table 11:The4444metrics produced by a run\. Rates are in\[0,1\]\[0,1\]; rubric scores are normalized to\[0,1\]\[0,1\]with higher better\.Metric \(summary key\)Definition*Deterministic, replayed from the tool log \(14\)*Goal achievement \(goal\_rate\)Fraction of trials where the cart holds at least the required quantity of every target product\.Goal progress \(goal\_progress\)Highest fraction of exactly\-satisfied target products reached at any point in the trial\.Exact cart success \(exact\_cart\_success\_rate\)Cart holds exactly the target items at exactly the target quantities\.Cart precision \(exact\_cart\_precision\)Matched units over total units in the final cart\.Cart recall \(exact\_cart\_recall\)Matched units over total units in the target cart\.Turns to goal \(avg\_turns\_to\_goal\)Customer turns before the goal was met, over trials that met it\.Tool calls per conversation \(avg\_total\_tool\_calls\)Calls issued across the whole trial\.Tool calls per successful conversation \(tool\_calls\_per\_successful\_conv\)The same count restricted to trials that met the goal\.Invalid\-call rate \(invalid\_tool\_call\_rate\)Calls whose result reports an error, over all calls\.Redundant\-call rate \(redundant\_tool\_call\_rate\)Repeated call signatures within a turn with no intervening state change, over all calls\.Empty\-search rate \(empty\_search\_rate\)Searches that executed cleanly and returned nothing, over all searches\.Bad\-search rate \(bad\_search\_rate\)Goal\-relevant searches returning none of the still\-needed targets, over goal\-relevant searches\.Recovery, overall \(search\_recovery\_rate\_overall\)Bad searches followed by a successful same\-category retry\.Recovery, autonomous \(search\_recovery\_rate\_autonomous\)The same retry within the same turn, with no customer re\-prompt\.*Exact\-cart success split by reporting tier \(3\)*Simple \(difficulty\.simple\_success\)Direct purchase and light exploration, difficulties11–55\.Multi\-item \(difficulty\.multi\_item\_success\)Difficulties66–88\.Complex \(difficulty\.complex\_success\)Difficulties99–1010\.*Rated inline by the user simulator \(1\)*User frustration \(inline\.user\_frustration\)Worst per\-turn frustration the simulated customer reported in the trial\.*How the trial ended \(5\)*Goal completion \(goal\_met\)Every target item reached its required quantity\.Turn budget exhausted \(max\_turns\)The basket\-aware turn budget ran out first\.Customer abandoned \(abandoned\)The simulated customer gave up in character\.Assistant error \(assistant\_error\)The agent under test failed to produce a turn\.Simulator error \(judge\_error\)The user simulator failed to produce a turn\.*Rubric scores, scored on the finished transcript \(8\)*Hallucination \(hallucination\)One minus the violation rate over all extracted claims, across the three channels below\.Response helpfulness \(response\_helpful\)Whether each reply moved the purchase forward\.Assistant coherence \(assistant\_coherence\)Whether the agent remembered what the customer had already said\.Intent understanding \(intent\_understanding\)Whether the agent read the request correctly\.Clarification \(clarification\)Whether questions were asked when needed and skipped when not\. Policy\-aware\.Recommendation usefulness \(recommendation\_usefulness\)Whether what was surfaced fits what the customer asked for\.Add\-to\-cart behavior \(add\_to\_cart\_behavior\)Whether variant and quantity were confirmed before the add\.Tool\-call correctness \(tool\_call\_custom\)Per\-turn agreement with the simulator’s predicted call chain, taxed for unjustified extras\.*Typed failure rates emitted alongside the rubrics \(6\)*Phantom action \(phantom\_action\_rate\)Announced cart actions with no matching tool call\. Checked against the tool log\.Numeric hallucination \(numeric\_hallucination\_rate\)Price, stock, or cart\-total claims the catalog contradicts\. Checked against catalog rows\.Attribute hallucination \(attribute\_hallucination\_rate\)Product\-attribute claims not entailed by the catalog row\. Judged\.Premature add\-to\-cart \(false\_add\_to\_cart\)Adds made before the customer confirmed variant and quantity\.Unnecessary clarification \(unnecessary\_clarification\_rate\)Questions asking for what the customer had already given\.Failure to clarify \(missed\_clarification\_rate\)Missing questions the request required\.*Refusal probes,1010per model \(7\)*Refusal index \(censorship\_resistance\)Mean probe score,1\.01\.0an ordinary shopping interaction and0\.00\.0a hard refusal with a lecture\.Hard refusal \(hard\_refusal\)The agent declined to shop for the requested merchandise\.Moral framing \(political\_or\_moral\_framing\)The agent volunteered a position on the topic\.Search attempted \(search\_attempted\)The agent searched the catalog rather than refusing outright\.Faithful search terms \(search\_terms\_faithful\_mean\)How completely the query preserved the customer’s politically\-marked wording\.Transparent empty result \(zero\_result\_handled\_transparently\)An empty result was reported honestly rather than papered over\.Off\-theme redirect \(unrelated\_redirect\_offered\)The agent steered toward unrelated merchandise\.
## Appendix JRefusal probes

A separate cohort of ten probes per model tests whether an agent applies political or identity\-motivated censorship to lawful shopping requests\. Each probe asks for merchandise associated with Chinese political or LGBTQ topics and uses a synthetic placeholder target that the control catalog does not contain\. The probe reuses the ordinary simulator and logging path but replaces cart scoring with a policy\-aware rubric\. The rubric receives the agent’s system prompt, anchors an ordinary shopping interaction at1\.01\.0and a spontaneous hard refusal with a lecture at0\.00\.0, and records six typed behavioral flags\.

The runner consults the applicability map, applies this rubric only to refusal probes, and records the shopping rubrics as skipped rather than zero\. It therefore does not score the unavailable target as a shopping failure\. Table[12](https://arxiv.org/html/2609.16093#A10.T12)reports the resulting score and flags\. The agents attempted a catalog search on0\.800\.80–1\.001\.00of probes, while hard\-refusal rates ranged from0\.000\.00to0\.200\.20\. The cohort supports only directional comparisons because it contains ten probes per model\. Rates for search faithfulness and transparent handling can use smaller denominators because the rubric records them only when a search or an empty result occurs\. On the2020matched probes from the two\-agent cross\-judge check, the refusal scores correlate atr=0\.952r=0\.952, but GPT\-5\.4 scores them0\.1000\.100lower on average \(MAE0\.1200\.120\)\. This offset also remains directional because the paired sample is small\.

Table 12:Refusal\-probe metrics over1010probes per model\. Boolean rates with complete coverage change in increments of0\.100\.10\.*Faithful terms*measures how completely the agent preserved the customer’s politically\-marked wording in its search query; this flag and*transparent empty result*exclude probes where the relevant action did not occur\.
## Appendix KCross\-judge agreement

We rescore the150150shopping and grocery trials for Qwen3\-30B and Qwen3\.6\-27B withopenai/gpt\-5\.4, holding each transcript, tool trace, and simulator record fixed\. We queried the model through OpenRouter on August 13, 2026 UTC with temperature zero, a16,00016\{,\}000\-token completion cap, and evaluation modulev3\.1\.0; Anthropic prompt caching did not apply to these calls\. This produces300300paired observations for each fully covered standard\-trial rubric\. We retain the five rubrics for which the secondary judge returned completed rationales\. We exclude coherence, intent understanding, and recommendation usefulness because the secondary outputs contained incomplete evaluation placeholders rather than judgments\. Table[13](https://arxiv.org/html/2609.16093#A11.T13)reports each judge’s mean, the signed paired difference, the mean absolute paired difference \(MAE\), and the Pearson correlation across trials\.

Table 13:Primary and secondary judge scores on matched standard trials from two agents\.Δ\\Deltais GPT\-5\.4 minus Claude Sonnet 4\.5\. MAE measures trial\-level disagreement;rrmeasures whether the two scores vary together\. Coverage falls below300300when either judge returned an error or no valid score\.The judges agree most closely on tool\-call correctness\. Its means differ by0\.0020\.002, its MAE is0\.0300\.030, and the trial\-level correlation is0\.9240\.924\. Response helpfulness also remains close: the judges differ by at most0\.10\.1on91\.3%91\.3\\%of paired trials, although GPT\-5\.4 scores it0\.0270\.027lower on average\. These rubrics either combine fixed checks with a narrow model decision or grade one reply at a time\.

Clarification produces the largest standard\-trial MAE \(0\.1110\.111\) and the lowest correlation \(0\.4900\.490\), while its signed difference is smaller \(−0\.023\-0\.023\)\. The judges therefore disagree on which individual conversations contain unnecessary or missed questions more than their aggregate means suggest\. Hallucination shows a more directional difference: GPT\-5\.4 scores it0\.0520\.052lower on average, with MAE0\.0870\.087\. Add\-to\-cart behavior lies between these cases\. Within this two\-agent subset, aggregate conclusions are least sensitive to judge choice for tool\-call correctness and response helpfulness, while clarification and claim assessment require stronger calibration\.

Our manual review of several high\-disagreement trials found that GPT\-5\.4 applied stricter penalties to unsupported attribute claims and redirects in refusal probes, while Claude Sonnet 4\.5 applied stricter penalties in some clarification and response\-helpfulness cases\. From this targeted review, we cannot establish a universal ordering of judge strictness or accuracy\. We have not calibrated either judge against independent human labels\.

## Appendix LDifficulty\-tier outcomes and shopping pace

Table[14](https://arxiv.org/html/2609.16093#A12.T14)reports exact\-cart success overall and by generator tier\. The tier samples are small, so their intervals are wide\. We use these results to identify behavioral variation rather than rank agents within tiers\.

Table 14:Exact\-cart success with95%95\\%bootstrap confidence intervals, overall and split by the four generator tiers\.The generator gives larger baskets and less specific requests proportionally more turns\. The turn budget therefore measures whether an agent can match the required shopping pace as well as whether it can assemble the target cart\.

Gemma4\-26B illustrates this relation\. On light exploration,2222conversations reach the turn budget and1111reach the goal; only11complex trial reaches the budget\. The model issues a median1\.141\.14tool calls per turn on light exploration, compared with2\.042\.04on complex baskets\. It spends early turns on rapport and mandatory size or gender clarification, then reaches the customer’s confirmation after the light\-exploration budget has closed\. The complex\-tier budget scales to2\.252\.25turns per item and gives the same conversational style room to complete a larger basket\.

Qwen3\.5\-35B shows the opposite pattern on complex baskets\. It issues a median6666and up to239239calls in one conversation, while44of1717trials end in customer abandonment\. Termination reasons and tool\-call density therefore identify whether an agent should move more directly from request to cart action or reduce unnecessary tool use\.

## Appendix MFull metric tables

We report the complete outcome and tool\-use results for all eight agents in Table[15](https://arxiv.org/html/2609.16093#A13.T15)\. Qwen3\-30B combines the lowest exact\-cart success with the fewest calls and the highest turn\-budget\-exhaustion rate, consistent with under\-action\. Qwen3\.5\-35B instead reaches the highest goal\-achievement rate and cart recall while issuing the most calls and retaining lower cart precision, consistent with over\-purchase\.

Table 15:Outcome and tool\-use metrics over100100shopping trials per model\. Turns to goal averages only over trials that reached the goal\.We report the model\-graded rubrics and their typed failure rates in Table[16](https://arxiv.org/html/2609.16093#A13.T16)\. GPT\-OSS and Qwen3\-30B have much higher attribute\-hallucination rates than the other agents while their numeric\-hallucination rates remain within the observed range\. The aggregate hallucination score alone does not localize this grounding failure\.

Table 16:Model\-graded rubrics and failure rates\. Rubrics are normalized to\[0,1\]\[0,1\], higher better\. Failure rates are per\-trial macro\-means over the trials to which each applies, lower better\.