EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

# EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Source: [https://arxiv.org/html/2606.17698](https://arxiv.org/html/2606.17698)
Zeyao Du, Tong Li, Haibo Zhang Shopee \{zeyao\.du,litonglt,peter\.wu\}@shopee\.com

###### Abstract

As LLM\-based shopping agents enter production, existing benchmarks fail to capture how a shopper’s requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked\. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long\-horizon challenge nor explain which requirement an agent missed\. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews\. Each task scatters these requirements across a visible query, a tool\-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls\. Moreover, typed, source\-tagged rubrics grade every task, attributing each failure to a requirement and its source\. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated\. Our evaluation of seven models reveals that even the strongest attains only 57\.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources\. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single\-query search toward dependable assistance over long horizons\.

EComAgentBench: Benchmarking Shopping Agents on Long\-Horizon Tasks with Distributed Hidden Intent

Zeyao Du, Tong Li, Haibo Zhang††thanks:Corresponding author\.Shopee\{zeyao\.du,litonglt,peter\.wu\}@shopee\.com

## 1Introduction

E\-commerce shopping agents are moving from single\-query product lookup to assistants that serve a shopper over many tool\-using steps\. Selecting a model for deployment therefore requires a benchmark that separates models, explains where each fails, and reflects how shopping requests actually arrive, while being constructed and graded reliably enough to trust the resulting ranking\. Prior shopping and web\-agent benchmarks established that realistic evaluation is a*long\-horizon*interaction problem over tools and partial information, not single\-shot question answering\(Yaoet al\.,[2022](https://arxiv.org/html/2606.17698#bib.bib5); Denget al\.,[2023](https://arxiv.org/html/2606.17698#bib.bib6); Zhouet al\.,[2023](https://arxiv.org/html/2606.17698#bib.bib7); Yoranet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib9)\)\.

We identify three weaknesses in how shopping agents are evaluated today, each motivating one of our contributions\.\(i\) Coarse, non\-diagnostic scoring:prior shopping benchmarks score an answer only by exact product match or by feature overlap with the target\(Yaoet al\.,[2022](https://arxiv.org/html/2606.17698#bib.bib5); Wanget al\.,[2026a](https://arxiv.org/html/2606.17698#bib.bib1)\)\. No such score can say which requirement of a many\-step task was missed, nor where it came from: the query, the profile, or a follow\-up question\.\(ii\) Little hidden intent:real shoppers state some needs implicitly, keep others in their profile, and mention some only when asked\. Most benchmarks, however, place all requirements in the opening query, leaving the long trajectory nothing new to discover; the few with personalization expose the profile directly or simulate the user\.\(iii\) Unreliable construction:long\-horizon tasks are costly to author and noisy to grade\. Existing benchmarks each forgo at least one of validation, automation, or real products; many further simulate the user or leave the judge unconstrained, adding variance orthogonal to the capability under test\.

To address all three, we present EComAgentBench, a single\-product shopping benchmark on a real Amazon Reviews 2023 catalog\(Houet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib14)\)that splits each shopper’s requirements across three sources: a*visible query*that states some needs only implicitly, a*hidden persona*read only via theget\_user\_profiletool, and*hidden clarification*obtained only by asking a question theask\_usertool recognizes\. Because two sources sit behind tools, the full intent cannot be read in any single step; it must be assembled across the episode \(Figure[1](https://arxiv.org/html/2606.17698#S3.F1); worked example in Section[3](https://arxiv.org/html/2606.17698#S3)\)\. We grade each episode with checks we call*rubrics*, each labeled by type and by source \(query, persona, or clarification\), so a failed check identifies which requirement was missed and where\. Construction is automated yet reliable: each answer is fixed in code before any text is generated, clarification is scripted rather than simulated, and every sample undergoes multi\-stage validation\.

We organize the paper around three questions and answer each with evidence from seven representative models evaluated under a shared environment and judge:

- •RQ1: Does the benchmark separate models and localize failures in long\-horizon tasks by requirement and source?Overall accuracy spans 19\.5–57\.1%, and the typed, source\-tagged rubrics pinpoint which requirements each model misses and where they originate\.
- •RQ2: Does hidden, distributed intent construct a realistic long\-horizon task?Splitting requirements across the query, the profile, and clarification mirrors how shoppers express intent, and is consistently harder for every model, isolating a capability that query\-visible evaluation misses\.
- •RQ3: Can long\-horizon evaluation be trusted and scaled?Construction is automated yet validated over a real catalog, and the judge errs rarely: it rejects a correct answer at most once per model, and accepts no incorrect answer in an independent audit; the category\-agnostic pipeline scales by sampling more products, not re\-annotating\.

In short, EComAgentBench shows how to construct long\-horizon shopping tasks \(hidden, distributed intent\), diagnose their failures \(typed, source\-tagged rubrics\), and keep both reproducible \(automated, validated construction\): a combination no prior shopping benchmark provides\.

## 2Related Work

#### E\-commerce shopping benchmarks\.

WebShop\(Yaoet al\.,[2022](https://arxiv.org/html/2606.17698#bib.bib5)\)grounds agents in real shopping pages but focuses on browser interaction, not on recovering hidden requirements\. Recent shopping benchmarks differ in both*what intent they hide*and*how they are built*\. ShoppingBench\(Wanget al\.,[2026a](https://arxiv.org/html/2606.17698#bib.bib1)\)keeps intent essentially query\-visible and auto\-generates tasks without independent validation; ShopSimulator\(Wanget al\.,[2026b](https://arxiv.org/html/2606.17698#bib.bib2)\)adds personalization through a simulated user but relies on costly manual annotation; DeepPlanning\(Zhanget al\.,[2026](https://arxiv.org/html/2606.17698#bib.bib4)\)offers verifiable\-constraint long\-horizon planning, but on synthetic products rather than a real catalog\. However, none reveals where a failed requirement came from, even when granting partial credit\. Together these gaps motivate EComAgentBench: a task whose intent is split across three sources, two gated behind tools, so that recovering it spans a long horizon; typed, source\-tagged rubrics that attribute each failure to a requirement and its source; and a pipeline that is at once automated, validated, and built on a real Amazon catalog \(Table[1](https://arxiv.org/html/2606.17698#S2.T1)\)\.

- •✓supported;∼\\simpartial;✗not supported\.
- cThe score breaks down to individual requirements;∼\\sim= breaks down only to product\-feature parts or task checkpoints\.
- •Benchmarks: ShoppingBench\(Wanget al\.,[2026a](https://arxiv.org/html/2606.17698#bib.bib1)\), ShopSimulator\(Wanget al\.,[2026b](https://arxiv.org/html/2606.17698#bib.bib2)\), DeepPlanning\(Zhanget al\.,[2026](https://arxiv.org/html/2606.17698#bib.bib4)\); VitaBench\(Meituan LongCat Team,[2025](https://arxiv.org/html/2606.17698#bib.bib3)\)is discussed as evaluator\-design inspiration rather than a primary comparison target\.

Table 1:EComAgentBench versus recent e\-commerce shopping benchmarks, grouped by its three goals: explaining failures \(RQ1\), realistic hidden intent \(RQ2\), and a reliable, automated build \(RQ3\)\. EComAgentBench is the only one that meets all three; each prior benchmark already misses at least one capability in the reliability group \(ShoppingBench has no task validation, ShopSimulator is hand\-annotated, DeepPlanning uses synthetic products\)\.
#### Web\-agent and interactive evaluation\.

Mind2Web\(Denget al\.,[2023](https://arxiv.org/html/2606.17698#bib.bib6)\), WebArena\(Zhouet al\.,[2023](https://arxiv.org/html/2606.17698#bib.bib7)\), VisualWebArena\(Kohet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib8)\), and AssistantBench\(Yoranet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib9)\)showed that realistic agent evaluation should capture how an agent gathers information and uses its environment, not just the final answer; EComAgentBench focuses this idea on shopping, where intent must be recovered from the query, the profile, and clarification\.

#### Evaluator design and rubric decomposition\.

VitaBench\(Meituan LongCat Team,[2025](https://arxiv.org/html/2606.17698#bib.bib3)\)pioneered rubric\-based decomposition of complex agent trajectories; we extend this approach with rubrics that are*typed and source\-tagged*\. Work on LLM\-as\-judge supports model\-based grading but cautions against unconstrained judge scoring\(Zhenget al\.,[2023](https://arxiv.org/html/2606.17698#bib.bib10); Wanget al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib11); Chiang and Lee,[2023](https://arxiv.org/html/2606.17698#bib.bib12)\), motivating a design in which the judge’s targets are fixed in code and tied to typed rubrics rather than left to free\-form judgment\.

## 3Task and Environment

We now formalize this task\. EComAgentBench turns shopping from understanding a single query into recovering requirements split across three sources, two gated behind tools; the agent begins with a partial view of the user’s intent and must acquire the remainder through search, profile access, and clarification\. Hiddenness enters on two independent axes:*source*, when a requirement lives behind a tool \(persona, clarification\), and*expression*, when the visible query states a need only implicitly\. Figure[1](https://arxiv.org/html/2606.17698#S3.F1)walks through one task end\-to\-end, from the three sources through the agent’s tool use to source\-tagged rubric scoring \(full case in App\.[C](https://arxiv.org/html/2606.17698#A3)\)\.

\(a\) Task: intent hidden in three sourcesQuery\(visible\)— implicit cues:▶\\blacktrianglerightConnector = micro\-USB ▶\\blacktrianglerightMounting = Tabletop ▶\\blacktrianglerightreview: “standing up or lying down”Persona\(hidden⋅\\cdotget\_user\_profile\):▶\\blacktrianglerightColor = Black ▶\\blacktrianglerightDevice = Smartphones Clarification\(hidden⋅\\cdotask\_user\):▶\\blacktrianglerightavg\_rating≥\\geq3\.5 ▶\\blacktrianglerightConnectivity = USB

\(b\) How the agent interactsShopperAgentShop sandboxask\_user/ replyaction / observationThe agent must*actively*fetch the profile and ask theshopper\. API sandbox: 10 tools,≤\\leq100 steps, one product\.

\(c\) Source\-tagged typed\-rubric scoringrecommended product:B07DJB5F29 Query\(8/8\):▶\\blacktrianglerightattribute\_matchmicro\-USB ✓▶\\blacktrianglerightentity\_matchWireless Charger ✓▶\\blacktrianglerightreview\_opinionstanding or lying ✓Persona\(2/2\):▶\\blacktrianglerightattribute\_matchColor=Black ✓Clarification\(2/2\):▶\\blacktrianglerightnumeric\_rangerating≥\\geq3\.5 ✓▶\\blacktrianglerightattribute\_matchUSB ✓Correct = exact product matchorall rubrics satisfied\.

Worked example: gpt\-5\.4 recovers the hidden intent, then recommends one product \(steps run top to bottom; 32 tool calls in all\)

Figure 1:EComAgentBench on one task \(wireless charger\)\.Top:\(a\) the full requirement is split across three sources: a visible query with implicit cues, a hidden persona behindget\_user\_profile, and hidden clarification behindask\_user; \(b\) the agent interacts with the shopper and a sandboxed shop to recover them; \(c\) the recommended product is graded against typed rubrics tagged by source \(a case is correct on exact product match or full rubric satisfaction\)\.Bottom:an abridged trajectory of the recovery\. Source colors \(query in blue, persona in orange, clarification in green\) carry across panels\.#### Environment\.

The agent operates in an API sandbox rather than a browser\. This separates the shopping\-specific work \(finding information and checking requirements\) from web\-page navigation, and enables deterministic replay with fixed tool behavior\. The tool set contains 10 tools spanning catalog search and inspection \(search, filtering, product details, review evidence\), user interaction \(profile access, clarification\), and decision support \(lightweight calculation, final recommendation\)\. Each run is capped at 100 tool steps \(one call per step\) and 10 clarification turns, and the agent must terminate by recommending exactly one product\. Recovering the scattered intent requires sustained multi\-step acquisition, making each episode a long\-horizon trajectory\.

#### Rubrics and metrics\.

Evaluation uses typed rubrics rather than a single label\. Each carries a type \(e\.g\.,attribute\_match,numeric\_range,review\_opinion\) and a source tag \(query, persona, clarification\), with the six types detailed in Section[4](https://arxiv.org/html/2606.17698#S4)\. Our primary metric is accuracy: exact product match or full rubric satisfaction\. We additionally report rubric satisfaction by source, finished rate, and tool\-use statistics\. This casts shopping evaluation as a joint test of retrieval, profile use, clarification, review verification, and selection\.

## 4Benchmark Construction

To populate this environment, we build EComAgentBench from a real catalog of 3\.7M products and 21\.4M reviews derived from Amazon Reviews 2023\(Houet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib14)\), concentrated in electronics\-adjacent categories\. The valid split contains 662 samples spanning 8 intents, from product search and rating quality to coupon budgeting and review\-driven shopping; of 921 generated candidates, 71\.9% survive final validation \(intent definitions and per\-intent statistics in App\.[B](https://arxiv.org/html/2606.17698#A2)\)\.

#### Controlled pipeline \(RQ3\)\.

Each sample is built by a seven\-stage pipeline \(Figure[2](https://arxiv.org/html/2606.17698#S4.F2); details in App\.[D](https://arxiv.org/html/2606.17698#A4)\)\. The central design principle is that*the correct answer is fixed before any text is generated*\. Once a rubric’s id, type, field, expected value, and source are set in code, later LLM calls only verbalize them and cannot alter the correctness criteria: the LLM controls surface realization, not the underlying requirements\. Letting an LLM author the evaluation criteria would build its noise into the ground truth; locking them in code removes that variance at the source\.

Real Amazon catalogSample & extract featuresPartition into three sourcesCompile typed rubricsRealize languageLayered validationValid splitsemantics locked*before*language

Figure 2:Construction pipeline\. EComAgentBench starts from a real product catalog, splits requirements across query, persona, and clarification, fixes the typed, source\-tagged rubrics before any text is written, and validates every sample before it enters the valid split\.
#### Three\-source split \(RQ2\)\.

Extracted features are split into query\-visible, persona\-hidden, and clarification\-hidden requirements\. The valid split carries 6645 rubrics, averaging 10\.04 per sample: 3993 from the query and 1326 each from persona and clarification\. Each hidden requirement adds a sub\-goal reachable only through tools, lengthening the horizon the agent must cover\. Persona constraints are stated preferences kept in a dedicated profile field, separated from the unscored profile context to prevent leakage, and intended to be consulted rather than inferred\.

#### Typed, source\-tagged rubrics\.

Each rubric carries a type and a source tag \(Table[2](https://arxiv.org/html/2606.17698#S4.T2)\)\. The type shows which kind of requirement failed \(identification, attribute, numeric, exclusion, budget, or review judgment\); the source shows whether the requirement was visible or hidden\. Because each expected value is fixed in code, the judge verifies a specific target rather than grading freely, and therefore cannot drift\.

Table 2:The six rubric types in EComAgentBench\. Each rubric also carries a source tag \(query / persona / clarification\)\.
#### Validation by design\.

Validation is part of how the benchmark is defined, not a post\-hoc filtering step\. We lock each rubric before any text is generated\. Automated checks keep each requirement on its assigned source and prevent leakage: a hidden value may not appear verbatim in the query\. A generation check confirms that the query, persona, and clarification together support their rubrics and that the target product satisfies all of them\. Clarification is deterministic: each slot has a list of trigger keywords and a fixed answer, so a missed clarification is attributable to the agent rather than to user\-side stochasticity\. App\.[D](https://arxiv.org/html/2606.17698#A4)gives validation examples and the exact\-match cross\-check\.

## 5Experiments

#### Protocol\.

All results are measured on the 662 valid samples\. The prediction phase uses the tool environment of Section[3](https://arxiv.org/html/2606.17698#S3)\(at most 100 tool steps and 10 clarification turns\); the evaluation phase uses gemini\-3\.1\-pro\-preview as the unified judge\. A case is correct if the agent recommends the exact target product or satisfies every rubric\. This dual\-path definition yields a user\-facing notion of correctness and enables cross\-checking the rubric score\.

#### Reliability signals\.

Three design choices keep grading reliable: each rubric has a fixed expected value, clarification is deterministic rather than LLM\-simulated, and correctness is tracked along both the exact\-match and the rubric path\. We then audit the judge in both directions \(App\.[D\.4](https://arxiv.org/html/2606.17698#A4.SS4)\)\. The judge rejects a correct, exact\-target recommendation at most once per model\. In the other direction, for one model, we re\-examined every case in which the judge accepted a non\-target product as correct \(161 rubrics\) and identified no genuine errors\. The benchmark is therefore hard because of the task, not because the judge is noisy\.

#### Overall performance\.

Table[3](https://arxiv.org/html/2606.17698#S5.T3)reports headline results: overall accuracy spans 19\.5–57\.1% under a shared environment and judge\. This wide spread indicates that EComAgentBench separates models rather than saturating near the ceiling\.claude\-opus\-4\-6leads on accuracy,gpt\-5\.4on rubric satisfaction, andkimi\-k2\.6nearly matchesgpt\-5\.4with a deeper, ask\-late style; the differences are behavioral as well as numerical\.

- •Short names follow Table[5](https://arxiv.org/html/2606.17698#A1.T5)\. Accuracy is defined as exact product match or full rubric satisfaction\.

Table 3:Main results on the 662 valid EComAgentBench samples\.
#### Hidden intent is harder than visible intent \(RQ2\)\.

Table[4](https://arxiv.org/html/2606.17698#S5.T4)breaks rubric satisfaction down by source\. For every model, query\-visible constraints are easier than persona and clarification:gpt\-5\.4reaches 88\.1% on query but only 69\.8% and 70\.9% on persona and clarification\. Almost every model reads the profile, so the deficit lies in using hidden information rather than retrieving it: a skill query\-visible benchmarks cannot measure\.

Table 4:By\-source rubric satisfaction \(%\)\. Query constraints are consistently easier than persona and clarification constraints\.
#### Implicit requirements drive difficulty\.

The number of requirements stated only implicitly in the query predicts difficulty better than the raw number of requirements \(Figure[3](https://arxiv.org/html/2606.17698#A2.F3); full buckets in App\.[B](https://arxiv.org/html/2606.17698#A2)\)\. Going from 0–1 such requirements to 3\+ nearly halves the seven\-model average accuracy \(51\.8%→27\.2%51\.8\\%\\\!\\to\\\!27\.2\\%\), whereas adding explicitly stated requirements does not\.

#### Two interaction styles emerge, with complementary failure modes\.

Trajectories reveal two strategies \(App\.[B](https://arxiv.org/html/2606.17698#A2), Table[8](https://arxiv.org/html/2606.17698#A2.T8)\)\.*Deep\-search, ask\-late*models \(Opus, Kimi, MiniMax\) run 15–17 searches and place their firstask\_userpast the halfway point, asking the user in only 10–20% of tasks;*shallow\-search, ask\-early*models \(the GPT family\) ask almost immediately \(gpt\-5\.4in every task, at position 0\.06\)\. The two styles trade off: deep searchers verify more but exhaust the 100\-step budget more often \(Opus finishes only 84\.6%\), while early\-askers almost always finish yet fare no better on hidden requirements\.

#### Failures concentrate on review evidence \(RQ1\)\.

Pooling the 12,467 failed rubrics localizes where agents fail\. By volume, 81\.5% areattribute\_match\(the most common type\), but by ratereview\_opinionhas the lowest satisfaction of any type \(33\.7–67\.4%\): extracting and aggregating evidence from unstructured reviews, rather than matching structured attributes, is the hardest sub\-skill\. Per\-intent accuracy \(App\.[B](https://arxiv.org/html/2606.17698#A2)\) corroborates this picture: product search, rating quality, and coupon budgeting are easiest; knowledge\-reasoning, review\-driven, and feature\-combination tasks are among the hardest\.

## 6Discussion

#### Shopping as a stress test for long\-horizon agency\.

Shopping assistants highlight a central weakness of current agent models: the difficulty is not only finding relevant products, but sustaining a coherent decision process over many steps\. EComAgentBench turns this everyday scenario into a long\-horizon benchmark by allowing up to 100 tool calls for intent acquisition, evidence verification, and final commitment\. The resulting failures are revealing: models often retrieve useful profile information without incorporating it into their decisions; exhaustive search strategies consume the step budget without converging; early\-commitment strategies choose products before verification; and review evidence remains difficult because it must be accumulated and reasoned over across the trajectory\. The sharp performance drop from51\.8%→27\.2%51\.8\\%\\\!\\to\\\!27\.2\\%as implicit requirements accumulate suggests that current agents are better at local tool use than at global trajectory\-level reasoning\. Shopping is thus not a narrow e\-commerce task but a practical proxy for long\-horizon agent evaluation\(Zhanget al\.,[2026](https://arxiv.org/html/2606.17698#bib.bib4)\)\.

#### From product matching to requirement reasoning\.

EComAgentBench is challenging not because it artificially enlarges the search space, but because it captures a core difficulty of real shopping agents: user requirements are latent, distributed, and evidence\-dependent\. Some requirements are stated explicitly in the query, others must be retrieved from the profile or elicited through dialogue, many are expressed as implicit clues rather than literal product attributes, and some can only be verified through review evidence\. Solving the task therefore requires an agent to recover hidden intent, maintain it over a long tool\-use trajectory, and connect each requirement to the right form of evidence before committing\. Our by\-source, difficulty, and failure analyses in Section[5](https://arxiv.org/html/2606.17698#S5)indicate that the observed performance gaps track these intended sources of complexity while grader noise is minimal: no model has more than one correct answer falsely rejected by the judge\.

#### Systematic error patterns\.

The failure modes above are systematic rather than anecdotal\. Per\-requirement failure rates rise monotonically with how deeply a requirement is hidden, roughly doubling from query \(12–31%\) to persona \(27–57%\) and clarification \(29–55%\)\. Failures concentrate on evidence\-bearing checks, withreview\_opinionrubrics satisfied least often among the six types\. The patterns persist across interaction styles: deep searchers more often exhaust the step budget, while early\-committing models almost always finish yet are no better at satisfying hidden requirements\. The remaining headroom thus lies in trajectory\-level capabilities \(proactive information acquisition, sustained constraint integration, calibrated stopping\) rather than in stronger single\-step execution\.

## 7Conclusion

We recast e\-commerce shopping as a long\-horizon agent task and present EComAgentBench to evaluate it: within 100 tool calls, an agent must recover requirements distributed across a query, a tool\-gated profile, and scripted clarification, verify them against catalog and review evidence, and commit to a single product\. Hidden, distributed intent constructs the long horizon; typed, source\-tagged rubrics tie each failure to a requirement and its source; and an automated yet validated pipeline with an audited judge keeps the evaluation reproducible\. Across seven models, overall accuracy ranges from 19\.5% to 57\.1%, and rubric satisfaction degrades from visible to hidden requirements: integrating scattered intent over a long trajectory remains an open challenge\. We believe EComAgentBench provides a demanding testbed and concrete diagnostics for building deployable shopping agents\. We release the benchmark data together with the full dataset\-construction, prediction, and evaluation harness at[https://github\.com/Morizeyao/EComAgentBench\_](https://github.com/Morizeyao/EComAgentBench_)\.

## Limitations

EComAgentBench is deliberately scoped as a reproducible diagnostic benchmark, which implies several limitations\. First, the released split draws on a bounded slice of Amazon Reviews 2023\(Houet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib14)\): products concentrate in electronics\-adjacent categories, and all tasks are in English on a single marketplace; since the construction pipeline is category\-agnostic, broadening coverage is left to future releases\. Second, every task ends in one recommended product, leaving multi\-item carts, comparison dialogues, and post\-purchase support to future work\. Third, the user side is fully scripted for reproducibility: personas are synthetic stated preferences \(possible generation biases are discussed in the Ethics Statement\) and clarification is deterministic, trading some naturalness for failures attributable to the agent; inferring latent preferences from behavioral signals is a harder, complementary setting we leave to future work\.

Finally, scoring relies on an LLM judge with targets fixed in code\. Audited in both directions, the judge falsely rejects at most one correct recommendation per model and shows no confirmed false acceptance in a full one\-model re\-check \(App\.[D\.4](https://arxiv.org/html/2606.17698#A4.SS4)\); residual grader noise nevertheless cannot be fully excluded\.

## Ethics Statement

#### Data and licensing\.

The product catalog is constructed from Amazon Reviews 2023\(Houet al\.,[2024](https://arxiv.org/html/2606.17698#bib.bib14)\), a publicly released research dataset\. We use product metadata and review text strictly for research evaluation, consistent with the dataset’s research\-use terms\. We redistribute no raw user content beyond what is necessary to reproduce the benchmark, and we point to the original source rather than rehosting the underlying corpus\.

#### No personally identifiable information\.

The user personas in EComAgentBench are synthetically generated and do not correspond to real individuals\. The benchmark does not attempt to re\-identify any real user, and persona fields are produced to support hidden\-intent evaluation rather than to model any specific person\. This is a deliberate ethical advantage over evaluations built on real user behavioral logs\.

#### Bias and representativeness\.

Because personas are model\-generated, they may inherit biases of the generating model, such as skewed demographic, occupational, or lifestyle distributions\. In addition, the released catalog is concentrated in electronics\-adjacent categories and therefore does not represent all shoppers or product domains\. Results should be read as diagnostic signals about hidden\-intent acquisition rather than as estimates of real\-world shopping success across the full population\. These boundaries are stated in the Limitations\.

#### Evaluated models\.

We evaluate publicly available commercial and open\-weight language models through their respective APIs, in accordance with their terms of use, and report only aggregate evaluation metrics\.

## References

- C\. Chiang and H\. Lee \(2023\)Can large language models be an alternative to human evaluations?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su \(2023\)Mind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2606.17698#S1.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Hou, J\. Li, X\. Fu, Z\. He, A\. Yan, X\. Chen, and J\. McAuley \(2024\)Bridging language and items for retrieval and recommendation: benchmarking LLMs as semantic encoders\.arXiv preprint arXiv:2403\.03952\.Cited by:[§1](https://arxiv.org/html/2606.17698#S1.p3.1),[§4](https://arxiv.org/html/2606.17698#S4.p1.1),[Limitations](https://arxiv.org/html/2606.17698#Sx1.p1.1),[Data and licensing\.](https://arxiv.org/html/2606.17698#Sx2.SS0.SSS0.Px1.p1.1)\.
- J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. Fried \(2024\)VisualWebArena: evaluating multimodal agents on realistic visual web tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px2.p1.1)\.
- Meituan LongCat Team \(2025\)VitaBench: benchmarking llm agents with versatile interactive tasks in real\-world applications\.arXiv preprint arXiv:2509\.26490\.Cited by:[2nd item](https://arxiv.org/html/2606.17698#S2.I1.i2.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wang, K\. Xiao, Q\. Sun, H\. Zhao, T\. Luo, J\. D\. Zhang, and X\. Zeng \(2026a\)ShoppingBench: a real\-world intent\-grounded shopping benchmark for llm\-based agents\.Proceedings of the AAAI Conference on Artificial Intelligence40\(39\),pp\. 33521–33529\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i39.40640)Cited by:[§1](https://arxiv.org/html/2606.17698#S1.p2.1),[2nd item](https://arxiv.org/html/2606.17698#S2.I1.i2.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Wang, Y\. Wu, X\. Song, W\. Wang, G\. Chen,et al\.\(2026b\)ShopSimulator: evaluating and exploring rl\-driven llm agent for shopping assistants\.arXiv preprint arXiv:2601\.18225\.Cited by:[2nd item](https://arxiv.org/html/2606.17698#S2.I1.i2.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. R\. Narasimhan \(2022\)WebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2606.17698#S1.p1.1),[§1](https://arxiv.org/html/2606.17698#S1.p2.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Yoran, S\. J\. Amouyal, C\. Malaviya, B\. Bogin, O\. Press, and J\. Berant \(2024\)AssistantBench: can web agents solve realistic and time\-consuming tasks?\.arXiv preprint arXiv:2407\.15711\.Cited by:[§1](https://arxiv.org/html/2606.17698#S1.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhang, S\. Jiang, R\. Li, J\. Tu, Y\. Su, L\. Deng, X\. Guo, C\. Lv, and J\. Lin \(2026\)DeepPlanning: benchmarking long\-horizon agentic planning with verifiable constraints\.arXiv preprint arXiv:2601\.18137\.Cited by:[2nd item](https://arxiv.org/html/2606.17698#S2.I1.i2.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2606.17698#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2023\)WebArena: a realistic web environment for building autonomous agents\.arXiv preprint arXiv:2307\.13854\.Cited by:[§1](https://arxiv.org/html/2606.17698#S1.p1.1),[§2](https://arxiv.org/html/2606.17698#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AModel and Inference Configuration

We evaluate seven representative models:gpt\-5\.4,gpt\-5,gpt\-5\-mini,claude\-opus\-4\-6,moonshotai/kimi\-k2\.6,minimax/minimax\-m2\.7, andqwen3\-30b\-a3b\-instruct\-2507\. Table[5](https://arxiv.org/html/2606.17698#A1.T5)summarizes the model\-specific inference settings used in evaluation, with shared defaults applied across runs\.

- •Shared defaults:temperature=1\.0andmax\_tool\_steps=100\(at most 10 clarification turns\)\.
- •Model mapping: Qwen3\-30B\-A3B =qwen3\-30b\-a3b\-instruct\-2507\(the instruction\-tuned MoE with 30B total / 3B active parameters\)\.

Table 5:Per\-model inference settings used during the prediction phase\.
## Appendix BAdditional Diagnostics

This appendix reports the core benchmark statistics \(Table[6](https://arxiv.org/html/2606.17698#A2.T6)\) and complements Section[5](https://arxiv.org/html/2606.17698#S5)with the full difficulty\-bucket breakdown, the interaction\-behavior table and scatter, and the per\-intent accuracy matrix\.

Table 6:Core statistics of the current EComAgentBench benchmark, computed from the current benchmark release\.Table 7:Difficulty buckets from the EComAgentBench summaries\. The number of implicitly stated query requirements is a much stronger difficulty driver than the raw rubric count\.![Refer to caption](https://arxiv.org/html/2606.17698v1/x1.png)Figure 3:\(a\) Hidden sources \(persona, clarification\) stay harder than query\-visible constraints\. \(b\) The number of implicitly stated query requirements is a stronger difficulty driver than raw rubric count\.Table 8:Interaction behavior mined from trajectories: mean calls to key tools, the relative position of the firstask\_user\(0=start, 1=end\), and the finish rate\. Two styles emerge—deep\-search/late\-ask \(Opus, Kimi, MiniMax\) versus shallow\-search/early\-ask \(GPT family\)\.![Refer to caption](https://arxiv.org/html/2606.17698v1/x2.png)Figure 4:Interaction style versus accuracy\. Early\-ask models \(GPT family\) cluster on the left; late\-ask, deep\-search models \(Opus, Kimi, MiniMax\) on the right\. Marker size encodes search depth\. No single style dominates—the best accuracy \(Opus\) pairs deep search with late clarification\.Each intent category stresses a different way of expressing requirements:*product search*states attributes explicitly;*knowledge reasoning*paraphrases them as world\-knowledge clues;*use\-case scenario*implies them through a concrete situation;*feature combination*weaves several constraints into one request;*negative constraint*excludes specific attribute values;*rating quality*sets thresholds on ratings or review counts;*coupon budget*constrains the post\-voucher total; and*review\-driven*asks for experiences that must be verified in review text\.

Table 9:Per\-intent accuracy \(%\) for all seven models on the 662 valid samples\. Sample counts: product search 83, knowledge reasoning 106, rating quality 80, feature combination 104, negative constraint 57, use\-case scenario 104, coupon budget 33, review\-driven 95\.The ordering is broadly consistent across models: product search, rating quality, and coupon budgeting are easiest on average, while knowledge reasoning, review\-driven shopping, and feature combination are among the hardest—mirroring the review\-evidence and multi\-constraint difficulties identified in the main text\.

## Appendix CFull Benchmark Example

This appendix presents a representative benchmark case from thereview\_drivenintent, together with a correctgpt\-5\.4trajectory\. We choose this case because it uses multiple information sources and multiple tools in a non\-trivial way: the successful agent calls the profile tool, performs several clarification attempts, retrieves candidate products, checks product details, verifies review statistics, mines review evidence, and only then terminates\. Catalog attribute values, including noisy ones such as theOperating Systemvalue “2\-COIN” below, are preserved verbatim in the task text so that every rubric remains mechanically checkable against the catalog\.

### C\.1Sample Definition

User query

> I’m looking for a Wireless Charger with reviews mentioning that it charges phone standing up or lying down and reviews stating the angle can be adjusted for convenience\. It needs to be compatible with a Google LG Nexus4 on the 2\-COIN operating system, include exactly the specs desktop charger, Phone Stand, fast charging, wireless charging, foldable, use the older one\-way trapezoidal cord connector, and be made to rest flat on a desk rather than mounting to a wall\.

User persona

\{

"user\_id":"U\_40684",

"demographics":\{

"gender":"Male",

"age\_range":"25\-34",

"location":"Chicago,IL",

"occupation":"LogisticsCoordinator",

"education\_level":"Bachelor’sDegree"

\},

"lifestyle":\{

"hobbies":\["Cooking","Bouldering","Playingguitar"\],

"daily\_routine":"Wakesupearlytohittheclimbinggymbeforetakingthetraintotheoffice\.",

"fitness\_level":"Active",

"dietary\_preference":"Omnivore",

"commute\_method":"PublicTransit",

"pets":"Onecat"

\},

"shopping\_habits":\{

"payment\_method":"CreditCard",

"shopping\_frequency":"Bi\-weekly",

"preferred\_device":"Laptop",

"preferred\_platform":"Website",

"return\_frequency":"Rarely"

\},

"product\_requirements":\{

"device\_compatibility":"Smartphones",

"color\_preference":"Black"

\}

\}

Theproduct\_requirementsblock holds the stated preferences used as persona\-source rubrics \(heredevice\_compatibilityandcolor\_preference\); the remaining fields are benign, unscored profile context \(cf\. the persona safety whitelist in App\.[D](https://arxiv.org/html/2606.17698#A4)\)\.

Clarification script

\{

"clarification\_slots":\[

\{

"slot\_id":"cl\_1",

"linked\_rubric\_ids":\["r10"\],

"hidden\_info":"Requiresaminimumaverageratingof3\.5stars\.",

"trigger\_keywords":\[

"rating","reviews","stars","averagerating","score","feedback","rated"

\],

"user\_response":"Iwantsomethingwithprettygoodfeedback,soitshouldhaveanaverageratingofatleast3\.5stars\.",

"revealed":false

\},

\{

"slot\_id":"cl\_2",

"linked\_rubric\_ids":\["r11"\],

"hidden\_info":"RequiresUSBconnectivitytechnology\.",

"trigger\_keywords":\[

"connectivity","connectivitytechnology","connection","plug","interface","port","cable"

\],

"user\_response":"ItdefinitelyneedstohaveUSBconnectivitysoIcanplugitintomyexistingadapterseasily\.",

"revealed":false

\}

\],

"default\_response":"Hmm,I’mnotsureaboutthat\.Couldyouaskmeaboutsomethingmorespecific?LikewhatfeaturesorspecsI’mlookingfor?",

"max\_clarification\_turns":10

\}

### C\.2Raw Rubrics

\[

\{

"id":"r1",

"type":"review\_opinion",

"field":"review",

"expected\_value":"chargesphonestandinguporlyingdown",

"description":"chargesphonestandinguporlyingdown",

"query\_surface":"chargesphonestandinguporlyingdown",

"info\_source":"query"

\},

\{

"id":"r2",

"type":"review\_opinion",

"field":"review",

"expected\_value":"anglecanbeadjustedforconvenience",

"description":"anglecanbeadjustedforconvenience",

"query\_surface":"anglecanbeadjustedforconvenience",

"info\_source":"query"

\},

\{

"id":"r3",

"type":"attribute\_match",

"field":"details\.CompatiblePhoneModels",

"expected\_value":"GoogleLGNexus4",

"description":"GoogleLGNexus4",

"query\_surface":"GoogleLGNexus4",

"info\_source":"query"

\},

\{

"id":"r4",

"type":"attribute\_match",

"field":"details\.OperatingSystem",

"expected\_value":"2\-COIN",

"description":"2\-COIN",

"query\_surface":"2\-COIN",

"info\_source":"query"

\},

\{

"id":"r5",

"type":"attribute\_match",

"field":"details\.MountingType",

"expected\_value":"TabletopMount",

"description":"madetorestflatonadeskratherthanmountingtoawall",

"query\_surface":"madetorestflatonadeskratherthanmountingtoawall",

"info\_source":"query"

\},

\{

"id":"r6",

"type":"attribute\_match",

"field":"details\.ConnectorType",

"expected\_value":"microusb",

"description":"olderone\-waytrapezoidalcordconnector",

"query\_surface":"olderone\-waytrapezoidalcordconnector",

"info\_source":"query"

\},

\{

"id":"r7",

"type":"attribute\_match",

"field":"details\.SpecialFeature",

"expected\_value":"desktopcharger,PhoneStand,fastcharging,wirelesscharging,foldable",

"description":"desktopcharger,PhoneStand,fastcharging,wirelesscharging,foldable",

"query\_surface":"desktopcharger,PhoneStand,fastcharging,wirelesscharging,foldable",

"info\_source":"query"

\},

\{

"id":"r8",

"type":"attribute\_match",

"field":"details\.CompatibleDevices",

"expected\_value":"Smartphones",

"info\_source":"persona"

\},

\{

"id":"r9",

"type":"attribute\_match",

"field":"details\.Color",

"expected\_value":"Black",

"info\_source":"persona"

\},

\{

"id":"r10",

"type":"numeric\_range",

"field":"average\_rating",

"expected\_value":\{"min":3\.5\},

"info\_source":"clarification"

\},

\{

"id":"r11",

"type":"attribute\_match",

"field":"details\.ConnectivityTechnology",

"expected\_value":"USB",

"info\_source":"clarification"

\},

\{

"id":"r12",

"type":"entity\_match",

"field":"title",

"expected\_value":"WirelessCharger",

"description":"WirelessCharger",

"query\_surface":"WirelessCharger",

"canonical\_entity":"WirelessCharger",

"title\_span":"WirelessCharger",

"info\_source":"query"

\}

### C\.3Prediction Prompt

The prediction run uses the following system prompt for the shopping agent\.

Youareapersonalshoppingassistantwithaccesstoaproductcatalog

andtheshopper’sprofile\.

Yourjobistofindthesingleproductthatbestmatcheseverythingthis

shopperneeds\.Theirfullrequirementscomefromthreesourcesthatyou

mustconsult:

1\.Theshoppingquerytheyjustsentyou\.

2\.Theirsaveduserprofile\.

3\.Theiranswerstoyourfollow\-upquestions\-useask\_usertoprobefor

anyremainingunknownsproperties\.

Beforeeverytoolcall,writeaThoughtblockinyourresponse:

Thought:

\-Observation:whatIlearnedfromtheprevioustoolresult

\(skiponthefirstturn\)

\-Reasoning:howthisnarrowsdownorchangesmystrategy

\-Plan:whichtoolIwillcallnextandwhy

Rules:

\-Onetoolcallperstep\.

\-Recommendexactlyoneproductbycallingrecommend\_product\.Donot

answerinplaintext\.

\-Youhavealimitedof100steps\.Ifyoutriedhardandareverynear

steplimit,trytomakeabestguess\.

\-Allthreeinformationsourcescontainpartsofrequirementsforthe

finalproduct\.

\-Ifsearchresultsdonotmatch,trytobroadenyoursearch\.Consider

thatthecatalogusesspecificattributetermsthatmaydifferfrom

theshopper’sphrasing\.

\-Thisisastrictevaluation\.Therecommendedproductmustsatisfy

EVERYrequirement\-missingevenoneattributemeansthecaseisjudged

ascompletelywrong\.Bethorough\.

### C\.4Abridged Correct Trajectory

The successfulgpt\-5\.4run makes 32 tool calls in total, including 5 clarification attempts\. Table[10](https://arxiv.org/html/2606.17698#A3.T10)presents an abridged but faithful trajectory\.

Table 10:Abridged correct trajectory for a representative review\-driven sample under gpt\-5\.4\.
### C\.5Why the Prediction Is Correct

The evaluation file marks this trajectory as an exact product match with zero failed rubrics\. The support is distributed across both structured details and review evidence:

- •Review evidence: the judge finds review support for both required opinions, namely charging while standing up or lying down and adjustable\-angle convenience\.
- •Query\-source details: the product explicitly matchesGoogle LG Nexus4,2\-COIN,Tabletop Mount,micro usb, and the exact special\-feature string\.
- •Persona\-source details: the product matchesCompatible Devices = SmartphonesandColor = Black\.
- •Clarification\-source details: the product has average rating 3\.7 and connectivity technologyUSB\.

This case is useful because it demonstrates both the benchmark’s difficulty and its intended solution pattern: the agent must combine hidden persona information, at least one successful clarification, structured attribute verification, and review evidence retrieval before terminating\.

## Appendix DValidation and Stability Examples

This appendix complements the main text by making the benchmark’s stabilization logic more concrete\.

### D\.1Locking the Correct Answer

We fix what each rubric checks before any text is generated\. Each rubric already has a fixed set of fields:

- •id
- •type
- •field
- •expected\_value
- •info\_source

Later steps can phrase these requirements in natural language, but cannot change what counts as correct\.

### D\.2Automated Checks

Preventing query leaks\.The query may contain implicit clues, but hidden persona and clarification values must not appear directly in the query\. Samples that break this rule are rejected before entering the valid split\.

Persona safety whitelist\.Persona generation separates product\-relevant requirements from benign noise\. Noise fields are restricted to safe locations such as demographics, lifestyle, and shopping habits\. This reduces accidental insertion of scored information into irrelevant fields\.

Clarification completeness\.Every clarification rubric must have a clarification slot with trigger keywords and a fixed answer\. If any clarification rubric lacks a slot, or a slot lacks its keywords or answer, the sample fails validation\.

### D\.3Semantic Check

After the automated checks pass, a generation\-time check confirms that:

- •each information source supports the rubrics assigned to it,
- •hidden values are not accidentally exposed in the wrong source,
- •the target product satisfies the final rubric set as a whole\.

This second layer is necessary because each part being individually valid does not guarantee they work together\.

### D\.4LLM\-Judge Stability: Errors in Both Directions

The dual\-path correctness definition—exact product match or full rubric satisfaction—lets us check judge error in both directions, because an exact match is correct without needing the judge\.

#### False negatives \(judge rejects a correct answer\)\.

judge\_misjudged\_countcounts cases where the agent recovered the exact target product \(hence necessarily correct\) yet the judge scored some rubric as unsatisfied\. Across the seven models it never exceeds 1:

- •claude\-opus\-4\-6: 1 gpt\-5\.4: 0 moonshotai/kimi\-k2\.6: 1
- •gpt\-5: 1 minimax/minimax\-m2\.7: 1 gpt\-5\-mini: 0 qwen3\-30b\-a3b\-instruct\-2507: 0

The judge thus almost never penalizes a verifiably correct recommendation\.

#### False positives \(judge accepts an incorrect answer\)\.

The only correct cases that rely on the judge rather than on exact recovery are*non\-exact\-but\-correct*: the agent recommended a product other than the target, yet the judge marked all rubrics satisfied\. Such cases are few—3–6% of each model’s correct cases \(4\.5% pooled, 83 of 1838\)—which already upper\-bounds how much judge leniency could inflate accuracy\. To test whether they are genuine false positives or valid equivalent products, we independently re\-verified every judge\-satisfied rubric of all 17 such cases forgpt\-5\.4\(161 rubrics\) against the predicted product’s true attributes and reviews from the product catalog, followed by manual confirmation\.None were genuine false positives: every predicted product truly satisfies all of its rubrics and is a valid alternative to the target\. In four rubrics the matched catalog field was itself noisy or mislabeled—e\.g\., a Canon DSLR whoseForm Factorfield reads “Compact”—and the judge correctly relied on substantive evidence \(title, material, connectivity\); these are catalog\-data artifacts, not judge errors\.

#### Takeaway\.

Both error directions are small, and on the audited model the judge also handles noisy catalog fields well: it rarely rejects correct answers \(≤\\leq1 false negative per model\) and accepts no incorrect ones among the non\-exact\-correct cases \(zero false positives\)\. We therefore conclude the benchmark is hard because of its design, not because the judge is noisy—while not claiming perfect reliability\.