MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents

# MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents
Source: [https://arxiv.org/html/2607.29002](https://arxiv.org/html/2607.29002)
Zeying Hao1\\equalcontrib, Hao Guo1\\equalcontrib, Mengtao Xu1\\corresponding, Yimin Hu1, Yuheng Song1, Zesheng Zhou1, Jinsong Lan1\\corresponding, Xiaoyong Zhu1

###### Abstract

Online shoppers increasingly turn to AI shopping assistants, using images and multi\-turn dialogue to express and refine product needs that are difficult to articulate in text alone\. However, existing benchmarks largely rely on text\-only or synthetic requests, underrepresenting complex real\-world shopping requirements jointly expressed through images and language\. We introduce MMShopBench, the first real\-log benchmark for multimodal, multi\-turn shopping agents\. Built from carefully cleaned and manually annotated shopping logs, MMShopBench provides ground\-truth annotations of each request’s purchase intent and mandatory product requirements\. Agents must infer these requirements jointly from user images and multi\-turn dialogue, retrieve candidate products through image and text search, and verify that each candidate satisfies all requirements using its product images and structured attributes\. We evaluate representative open\-source and proprietary models using an evidence\-grounded multimodal protocol and construct a companion training set for fine\-tuning an open\-source model\. To ensure reproducible experimentation, we build an offline shopping sandbox, where fine\-tuning substantially narrows the performance gap between our open\-source model and leading proprietary models, demonstrating the effectiveness of our training data\.

Code and Data—https://github\.com/H\-cool/MMshopbench

## Introduction

Recent advances in large language models \(LLMs\) have accelerated the development of agents that can reason over multiple steps, invoke external tools, and act in interactive environments\(Yao et al\.[2023](https://arxiv.org/html/2607.29002#bib.bib21)\)\. Building on multimodal language models, these agents are increasingly able to process images alongside text, ground intermediate decisions in visual evidence, and select modality\-specific tools during task execution\(Zhang et al\.[2026b](https://arxiv.org/html/2607.29002#bib.bib24)\)\. As a domain of substantial practical value and broad application potential, e\-commerce has naturally become a key setting for evaluating agent capabilities\.

Within this setting, conversational shopping agents offer a new interaction paradigm in which users can express complex, intent\-driven needs through natural dialogue rather than relying on keyword\-based search\. Product search often begins before a shopper has a complete textual query\. A user may first upload an image to indicate a style, object, or printed list; refer back to it with an ambiguous phrase such as “this kind”; and only later add a budget, material, size, compatibility, or bundle constraint\. The operative request is therefore not the latest utterance, but a purchase specification accumulated across visual evidence and dialogue\. A competent shopping agent must derive the operative set of user constraints from the available visual and textual evidence, select the appropriate retrieval modality—text or image—and reject any candidate whose product evidence fails to establish compliance with every hard constraint\.

Existing evaluation settings cover these capabilities only in isolation\. Shopping\-agent benchmarks predominantly begin with self\-contained text instructions or constructed interactions and emphasize navigation, planning, preference elicitation, or final product choice\(Yao et al\.[2022](https://arxiv.org/html/2607.29002#bib.bib20); Wang et al\.[2026b](https://arxiv.org/html/2607.29002#bib.bib18),[c](https://arxiv.org/html/2607.29002#bib.bib19); Du, Li, and Zhang[2026](https://arxiv.org/html/2607.29002#bib.bib2)\)\. Multimodal search benchmarks instead focus on cross\-modal evidence acquisition for authored search questions, while multimodal dialogue datasets preserve images and interaction history without requiring end\-to\-end catalog retrieval and requirement\-level product verification\(Kottur et al\.[2021](https://arxiv.org/html/2607.29002#bib.bib7); Zhao et al\.[2022](https://arxiv.org/html/2607.29002#bib.bib25); Jiang et al\.[2025](https://arxiv.org/html/2607.29002#bib.bib6); Li et al\.[2025](https://arxiv.org/html/2607.29002#bib.bib8); Tao et al\.[2026](https://arxiv.org/html/2607.29002#bib.bib16); Zhang et al\.[2026a](https://arxiv.org/html/2607.29002#bib.bib23)\)\. These paradigms therefore do not capture the complete path from requirements expressed across an organic multimodal conversation, through modality\-aware retrieval, to evidence\-grounded product selection\. Addressing both limitations requires realistic request provenance together with a frozen, inspectable environment\.

![Refer to caption](https://arxiv.org/html/2607.29002v1/x1.png)Figure 1:Overview of MMShopBench and its offline shopping\-agent workflow\. Real multimodal, multi\-turn logs are curated into 289 cases with hidden purchase\-intent and mandatory\-requirement annotations\. Within the frozen 100K\-product sandbox, agents infer requirements, interleave text and region\-aware image retrieval, and apply evidence\-grounded verification and selection; the same environment supports teacher\-trajectory SFT and grounded evaluation\.MMShopBench addresses this gap through a unified benchmark, environment, and agent framework, as summarized in Figure[1](https://arxiv.org/html/2607.29002#Sx1.F1)\. Through rigorous filtering and cleaning of real\-world logs from an online AI shopping assistant, we construct 289 evaluation cases that retain the complete sequence of images and dialogue through which users progressively supplement, clarify, and refine their shopping needs over the course of an interaction\. Each case is manually annotated to identify the purchase intent, the mandatory attributes that a target product must satisfy, and one or more verified target products\. We pair these real\-log tasks with a frozen sandbox of 100,000 products and an agent framework that chooses between BM25 text search and requirement\-conditioned regional visual retrieval\. To preserve visual provenance across multi\-turn interactions, the framework assigns persistent identities to user images and conditions region selection on the accumulated purchase specification\. An Evidence\-Grounded Verification and Selection stage then checks and reranks retrieved candidates against fixed product images and structured attributes before the final response\. Finally, we construct a quality\-controlled SFT corpus and evaluate representative open\- and closed\-source models, enabling controlled study of how targeted supervision improves multimodal shopping agents\.

The key contributions of this work can be summarized as follows:

- •A real\-log multimodal shopping benchmark\.We curate 289 real multimodal, multi\-turn shopping\-assistant conversations, each annotated with purchase intent, mandatory requirements, and verified target products\.
- •A reproducible sandbox and evidence\-grounded agent framework\.We build a frozen 100,000\-product sandbox and an agent framework integrating text search, requirement\-conditioned regional visual retrieval, and Evidence\-Grounded Verification and Selection\.
- •Broad evaluation and effective supervision\.We benchmark representative open\- and closed\-source models and show that fine\-tuning on our quality\-controlled corpus yields substantial gains across model scales and narrows the gap to leading proprietary systems\.

## Related Work

### Product Search and Shopping Agents

Product\-search datasets offer strong catalog grounding but generally reduce a need to a single textual query\. The Shopping Queries Dataset, for example, contains real Amazon queries and manually judged query–product pairs for ranking and relevance classification\(Reddy et al\.[2022](https://arxiv.org/html/2607.29002#bib.bib14)\)\. WebShop turns compositional text instructions into navigation and purchase actions over a simulated site populated with real products\(Yao et al\.[2022](https://arxiv.org/html/2607.29002#bib.bib20)\)\. ShoppingBench adds complex intent grounded in product records, while generating evaluation instructions from sampled products\(Wang et al\.[2026b](https://arxiv.org/html/2607.29002#bib.bib18)\)\. Recent shopping environments extend evaluation to simulated multi\-turn preference discovery or long\-horizon hidden intent\(Wang et al\.[2026c](https://arxiv.org/html/2607.29002#bib.bib19); Du, Li, and Zhang[2026](https://arxiv.org/html/2607.29002#bib.bib2)\)\. Despite their realism at the product or interaction level, these benchmarks formulate shopping needs primarily as text\-only or manually constructed requests, and therefore underrepresent requirements that are conveyed visually and progressively clarified across multiple dialogue turns\. In contrast, MMShopBench derives its tasks from real multimodal interactions with AI shopping assistants: agents must jointly infer the user’s purchase intent and mandatory requirements from images and dialogue, retrieve candidate products through text and image search, and verify each constraint against frozen product images and structured attributes\.

### Multimodal Search Agent Benchmarks

Multimodal search\-agent benchmarks evaluate whether models can formulate queries, use retrieval and browsing tools, and synthesize evidence across visual and textual sources\. MMSearch decomposes multimodal search into requerying, reranking, summarization, and end\-to\-end search\(Jiang et al\.[2025](https://arxiv.org/html/2607.29002#bib.bib6)\)\. MM\-BrowseComp uses hand\-crafted questions whose prompts or supporting webpages contain essential image or video evidence, requiring agents to browse beyond text\-only cues\(Li et al\.[2025](https://arxiv.org/html/2607.29002#bib.bib8)\)\. MMSearch\-Plus requires fine\-grained visual cues to be extracted and propagated through iterative image–text retrieval under retrieval noise\(Tao et al\.[2026](https://arxiv.org/html/2607.29002#bib.bib16)\)\. BrowseComp\-V3V^\{3\}emphasizes deep cross\-modal, multi\-hop browsing with publicly searchable evidence and subgoal\-level process evaluation\(Zhang et al\.[2026a](https://arxiv.org/html/2607.29002#bib.bib23)\)\.

Other benchmarks focus on the structure and reliability of the search trajectory\. MC\-Search supplies long, step\-wise annotated reasoning chains and process\-level measures for retrieval and planning\(Ning et al\.[2026](https://arxiv.org/html/2607.29002#bib.bib11)\)\. MERRIN evaluates modality selection, multimodal evidence retrieval, and multi\-hop reasoning over noisy or conflicting web sources\(Wang et al\.[2026a](https://arxiv.org/html/2607.29002#bib.bib17)\), while InterLV\-Search requires visual and textual evidence to repeatedly condition subsequent search actions and provides a standardized agent interface for tool use and trajectory logging\(Hou et al\.[2026](https://arxiv.org/html/2607.29002#bib.bib5)\)\. Collectively, these benchmarks move beyond final\-answer accuracy by exposing intermediate capabilities such as query planning, modality choice, evidence acquisition, and cross\-modal integration\.

## MMShopBench

### Task Definition

An evaluation case isx=\(Ht,It,Rt\)x=\(H\_\{t\},I\_\{t\},R\_\{t\}\)at target turntt\. The historyHtH\_\{t\}contains all user images, user utterances, and assistant responses throughtt\.ItI\_\{t\}denotes the purchase intent, andRt=\{r1,…,rm\}R\_\{t\}=\\\{r\_\{1\},\\ldots,r\_\{m\}\\\}contains the mandatory attributes that any acceptable target product must satisfy\. The agent observesHtH\_\{t\}but not the annotations\. Using a bounded sequence of retrieval calls, it returns an ordered listPk=\(p1,…,pk\)P\_\{k\}=\(p\_\{1\},\\ldots,p\_\{k\}\)\.

A product is valid only when available evidence supports both the purchase intentItI\_\{t\}and every mandatory attributeri∈Rtr\_\{i\}\\in R\_\{t\}\. This conjunctive criterion is strict: if any mandatory attribute is violated or cannot be verified, the product is judged unsuccessful, even when it otherwise matches the requested product type\. This formulation distinguishes two sources of failure\. Intent\- or attribute\-inference errors yield an incorrect search or acceptance criterion, whereas grounding errors retrieve or accept a product without sufficient supporting evidence\.

### Data Collection, Curation, and Annotation

#### Log sampling\.

MMShopBench is derived from interaction logs of a deployed e\-commerce conversational shopping assistant\. We consider online multi\-turn sessions collected between June and July 2026 and randomly sample 10,000 conversations that contain both at least one user\-provided image and nonempty user text\. Multimodal production traffic is not restricted to purchase\-oriented requests: users may ask for product information, request generic visual recognition, or provide input that remains too ambiguous to support a concrete shopping action\. The initial pool therefore serves as a broad sample of real multimodal assistant usage rather than as the evaluation set itself\.

All textual data and annotations in MMShopBench are originally in Chinese, and the benchmark and reported evaluations use these original Chinese records\. To facilitate presentation, dialogue excerpts, annotations, and examples in this paper are translated into English\.

#### Intent stratification and target\-turn localization\.

We organize the sampled traffic into five Level\-1 intent categories:*Find Exact Same Product*,*Find Similar/Alternative/Recommendation*,*Product Knowledge and Comparative Decision\-Making*,*General Visual Recognition*, and*Ambiguous Intent*\. We use Claude Opus 4\.8\(Anthropic[2026](https://arxiv.org/html/2607.29002#bib.bib1)\)as a scalable pre\-annotation model to assign a Level\-1 category to each of the 10,000 conversations and to localize the target turntt, defined as the turn by which the accumulated dialogue history contains the user’s complete set of requirements\. This step identifies both whether a conversation expresses an actionable shopping need and how much of its preceding multimodal context is required to recover that need\.

#### Expert annotation and quality control\.

The first two Level\-1 categories directly require retrieval of a concrete product and are therefore retained as benchmark candidates\. We draw a balanced sample of 400 conversations across these two categories and ask professional annotators with e\-commerce expertise to inspect the multimodal history through the target turn\. For each case, annotators specify the purchase intent, enumerate the mandatory product attributes that cannot be traded off, and verify one or more target products when available\. During quality control, we exclude conversations containing personally identifiable information or other sensitive content to protect user privacy\. We also remove cases for which no product satisfying the annotated requirements can be found on the shopping platform\. After filtering, 289 evaluation cases remain\.

#### Hierarchical intent taxonomy\.

We further assign Level\-2 labels within the two retained shopping intents\.*Find Exact Same Product*maps to an exact\-match subtype, whereas*Find Similar/Alternative/Recommendation*is decomposed into need\-solving recommendation, similar\-product search, purchase decision/selection, and matching recommendation\. Figure[2](https://arxiv.org/html/2607.29002#Sx3.F2)reports the resulting hierarchy and distribution\. At the leaf level, the two most prevalent intents are*Find Exact Same Product*\(35\.8%\) and*Need\-Solving Recommendation*\(26\.0%\)\.

![Refer to caption](https://arxiv.org/html/2607.29002v1/x2.png)Figure 2:Hierarchical intent distribution of MMShopBench\. The inner ring shows the two retained Level\-1 shopping intents, and the outer ring shows their Level\-2 decomposition\. Percentages are computed over the final 289\-case evaluation set\.Each released record stores pseudonymous conversation and trace identifiers, the target turn, per\-turn user images and text, preceding assistant messages, the purchase intent, slash\-delimited mandatory\-attribute annotations, and verified item identifiers\. We preserve the original turn order because later requests often resolve referents introduced by an earlier image or assistant response\. The evaluation therefore exposes the complete history through the target turn rather than converting the interaction into a synthetic one\-shot query\.

Table[1](https://arxiv.org/html/2607.29002#Sx3.T1)summarizes the interaction structure\. The median target is turn 4, and the latest target occurs at turn 12\. Every case contains at least one user\-provided image by the target turn, with an average of 1\.38 images per case\. This universal visual coverage makes image history a structural component of the benchmark rather than an optional modality\.

Table 1:MMShopBench evaluation\-set statistics\.Table 2:Positioning of MMShopBench against representative shopping\-agent benchmarks\. Input modalities refer to shopper requests or dialogue history, not product media or tool observations\.✓\\checkmarkdenotes that the benchmark includes the capability, and×\\timesdenotes its absence\.

### Positioning

Table[2](https://arxiv.org/html/2607.29002#Sx3.T2)positions MMShopBench against representative shopping\-agent benchmarks\. Here,*real logs*means naturally occurring shopper–assistant conversations rather than generated requests over real products or purchase histories\.*Image input*and*text input*distinguish the modalities supplied in shopper requests or dialogue history; product images in a catalog do not by themselves constitute image input\.*Multi\-turn*requires shopper–assistant utterance exchange rather than repeated agent–environment actions, and*Visual object grounding*requires linking request\-side visual objects to fixed product evidence\. Prior benchmarks cover text\-based long\-horizon interaction or personalization, but none combines these capabilities with real multimodal dialogue\.

## Offline Shopping Sandbox

### Frozen Offline Product Catalog

We construct the sandbox catalog in a target\-grounded manner\. Starting from the identifiers of the human\-verified target products in MMShopBench, we retrieve the corresponding products from the full online catalog and use their category assignments as sampling anchors\. All available target products are retained\. We then perform stratified sampling over non\-target products, first drawing products from the same leaf categories and broader Level\-2 categories as the targets\. To preserve catalog diversity rather than restricting the sandbox to benchmark\-specific categories, we allocate the remaining quota uniformly across other product categories\. After deduplication, this procedure yields a frozen catalog of 100,000 distinct products\. Real\-world shopping requests often contain fine\-grained constraints that cannot be resolved from surface\-level metadata such as product titles, prices, and shop information alone\. We therefore augment each catalog record with available SKU\-level labels, structured product attributes, and OCR text extracted from product\-detail images, providing richer evidence for verifying whether a retrieved product satisfies every mandatory user requirement\.

### Multimodal Retrieval Tools

The sandbox exposes complementary text\- and image\-based product retrieval tools\. Text search applies BM25\(Robertson and Zaragoza[2009](https://arxiv.org/html/2607.29002#bib.bib15)\)over product titles, categories, attributes, and shop fields, returning up to ten products for each search query\. It is primarily used when the dialogue specifies category, function, brand, or other attribute constraints in language\. For visual retrieval, we use Marqo\-Ecommerce\-Embeddings\-L\(Zhu, Jung, and Clark[2025](https://arxiv.org/html/2607.29002#bib.bib27)\)to encode catalog product images and the user\-provided query image into a shared embedding space, rank products by cosine similarity, and return the identifiers of the top five candidates\.

We introduce two interface designs to make visual retrieval effective in multi\-turn, multi\-image interactions\.

Persistent image identities\.When images are passed only as content blocks in the model input, the model can inspect them during inference, but subsequent tool calls lack a stable handle for specifying which earlier image should be reused\. This ambiguity becomes particularly problematic when a conversation spans multiple turns or contains several images\. We instead promote every incoming image to a persistent, addressable session object and assign it a session\-global identifier,img\_idx\. The identifier is recorded in the dialogue history together with the image’s turn and within\-turn position, and image\-tool calls takeimg\_idxas an explicit argument\. At execution time, the sandbox resolves this identifier back to the original image, allowing the agent to recall, disambiguate, and reuse any previously observed image rather than limiting tool access to the latest input\. The identifier also preserves provenance by making each visual retrieval action traceable to a specific user image\.

Requirement\-conditioned region retrieval\.Whole\-image embeddings can be dominated by backgrounds, secondary objects, or other visually salient content unrelated to the user’s current shopping goal\. We therefore formulate the crop as an agent\-selected visual query rather than a fixed preprocessing operation\. After resolving the requestedimg\_idx, the agent jointly conditions on the selected image, purchase intent, and accumulated dialogue constraints to predict a normalized target region\(x1,y1,x2,y2\)\(x\_\{1\},y\_\{1\},x\_\{2\},y\_\{2\}\)\. The sandbox then crops the selected region, encodes the resulting crop with Marqo\-Ecommerce\-Embeddings\-L\(Zhu, Jung, and Clark[2025](https://arxiv.org/html/2607.29002#bib.bib27)\), the same model used to construct the catalog image index, and ranks catalog images by cosine similarity\. Because the region is generated at tool\-call time, the agent can shift to a different object or refine the spatial extent when retrieved candidates conflict with the request\. This establishes a feedback loop between cross\-modal requirement inference, visual grounding, and product retrieval while suppressing similarity signals from irrelevant image content\.

### Evidence\-Grounded Verification and Selection

Retrieval success does not always translate into recommendation success: a product that satisfies every requirement may surface in a tool result yet be dropped during the agent’s implicit final selection\. Evidence\-Grounded Verification and Selection \(EGVS\) targets this gap between what retrieval makes available and what the agent finally recommends\. After search terminates, it reconstructs the full candidate pool𝒞=\(⋃s=1S𝒞s\)∪F\\mathcal\{C\}=\(\\bigcup\_\{s=1\}^\{S\}\\mathcal\{C\}\_\{s\}\)\\cup F, combining products returned by every text or image search call with the agent’s original final picksFF\. For each candidatec∈𝒞c\\in\\mathcal\{C\}, EGVS assembles product\-side evidenceE​\(c\)E\(c\)comprising its title, primary image, structured attributes, SKU\-level labels, and product\-detail OCR text\. The policy model then operates in a self\-verification mode and receives the same inferred specificationS^t=\(I^t,R^t\)\\hat\{S\}\_\{t\}=\(\\hat\{I\}\_\{t\},\\hat\{R\}\_\{t\}\), derived solely fromHtH\_\{t\}, together withE​\(c\)E\(c\)\. It does not re\-infer or revise the specification for individual candidates\. A product is confirmed only when its evidence is compatible with the inferred purchase intentI^t\\hat\{I\}\_\{t\}and supports every requirement inR^t\\hat\{R\}\_\{t\}\. The human annotations\(It,Rt\)\(I\_\{t\},R\_\{t\}\), verified product identifiers, and external\-judge outputs are never exposed to the agent or self\-verifier\. Let𝒮=\{c∈𝒞:v​\(c\)=1\}\\mathcal\{S\}=\\\{c\\in\\mathcal\{C\}:v\(c\)\{=\}1\\\}denote the candidates the self\-verifiervvconfirms\. EGVS then refines rather than rebuilds the selection, composing the final top\-KKrecommendation \(KKis the recommendation size, withK≥3K\\geq 3to cover the reported cutoffs\) as

A=topK​\(\(𝒮∩F\)​‖\(𝒮∖F\)‖​\(F∖𝒮\)\),A=\\mathrm\{top\}\_\{K\}\\big\(\(\\mathcal\{S\}\\cap F\)\\;\\\|\\;\(\\mathcal\{S\}\\setminus F\)\\;\\\|\\;\(F\\setminus\\mathcal\{S\}\)\\big\),where∥\\\|concatenates the ordered segments andtopK\\mathrm\{top\}\_\{K\}keeps the firstKKitems\. Confirmed original picks𝒮∩F\\mathcal\{S\}\\cap Fcome first, followed by confirmed products the agent had missed𝒮∖F\\mathcal\{S\}\\setminus F, and any remaining original picksF∖𝒮F\\setminus\\mathcal\{S\}fill leftover slots\. Verification thus prioritizes confirmed products within the top\-KKbudget while keeping the agent’s other picks wherever slots remain, promoting and recovering products rather than replacing the agent’s list\. The self\-verifier uses the policy model with a separate prompt and context from the external evaluator and introduces no additional model dependency\.

### Agent Workflow

Given the multimodal dialogue history, the agent first consolidates the purchase intent and mandatory requirements into an inferred specification\(I^t,R^t\)\(\\hat\{I\}\_\{t\},\\hat\{R\}\_\{t\}\), then decides whether to invoke text search, requirement\-conditioned region retrieval, or both\. It alternates between reasoning and retrieval\(Yao et al\.[2023](https://arxiv.org/html/2607.29002#bib.bib21)\)for at most eight tool steps, revising textual queries or image regions when the returned candidates are inconsistent with the request\. After search terminates, EGVS applies the verification\-and\-selection procedure above to the accumulated retrieval trace\. The agent returns the resulting product identifiers, each paired with a concise textual description; because every identifier must originate from a tool result, each recommendation remains traceable to its supporting retrieval action\.

### Supervised Fine\-Tuning

Following the filtering and quality\-control procedure used to construct MMShopBench, we screen 900 multimodal, multi\-turn conversations from online shopping\-assistant logs to form a companion SFT corpus\. Within the sandbox, Gemini\-3\.1\-Pro\-Preview serves as the teacher policy and executes the agent workflow above, producing search trajectories that interleave dialogue\-conditioned decisions, text or region\-aware image retrieval calls, retrieved product evidence, and final verification and selection\. Given an interaction contextxxand a teacher\-generated agent\-token sequencey=\(y1,…,yL\)y=\(y\_\{1\},\\ldots,y\_\{L\}\), we optimize the student model with the autoregressive cross\-entropy objective

ℒSFT​\(θ\)=−∑i=1Llog⁡pθ​\(yi∣x,y<i\)\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\-\\sum\_\{i=1\}^\{L\}\\log p\_\{\\theta\}\(y\_\{i\}\\mid x,y\_\{<i\}\)\.This supervision targets the agent’s operational policy rather than a direct mapping from a dialogue to a product identifier\. It teaches the model to coordinate tools over multiple rounds, determine when visual or textual evidence should be acquired, refine subsequent search actions from tool feedback, and assess whether the accumulated evidence supports the user’s mandatory requirements\.

### Evaluation Metrics

We use GPT\-5\.5\(OpenAI[2026](https://arxiv.org/html/2607.29002#bib.bib12)\)as a multimodal LLM judge, kept separate from the policy\-model self\-verifier in EGVS\. For candidatepj\(n\)p\_\{j\}^\{\(n\)\}, letE​\(pj\(n\)\)E\(p\_\{j\}^\{\(n\)\}\)denote its frozen product evidence, comprising product images, title, structured attributes, SKU\-level labels, and OCR text extracted from product\-detail images\. The judge jointly considers the complete multimodal historyHt\(n\)H\_\{t\}^\{\(n\)\}and the human\-annotated purchase intentIt\(n\)I\_\{t\}^\{\(n\)\}and mandatory requirementsRt\(n\)R\_\{t\}^\{\(n\)\}when assessing the product evidence\. Letyn,jy\_\{n,j\}denote the resulting binary decision:yn,j=1y\_\{n,j\}=1only if the evidence is consistent with the constraints expressed in the contextual images and text while satisfying the annotated purchase intent and every mandatory requirement; any contradiction or insufficient support with respect to either source yieldsyn,j=0y\_\{n,j\}=0\.

Our primary evaluation reports four metrics\. Fork∈\{1,3\}k\\in\\\{1,3\\\}, the evidence\-grounded judge score is

Judge@k=1N∑n=1N𝟏\[∃j≤k:yn,j=1\]\.\\mathrm\{Judge@\}k=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbf\{1\}\\\!\\left\[\\exists j\\leq k:\\;y\_\{n,j\}=1\\right\]\.Judge@1 evaluates the first returned product, whereas Judge@3 counts a case as successful if any of the first three products satisfies this joint decision rule\. As a deterministic complement, letV\(n\)V^\{\(n\)\}denote the set of manually verified product identifiers for casenn, andPk\(n\)P\_\{k\}^\{\(n\)\}the firstkkproducts the system returns\. Fork∈\{1,3\}k\\in\\\{1,3\\\}, exact\-identifier retrieval is

ID​@​k=1N​∑n=1N𝟏​\[Pk\(n\)∩V\(n\)≠∅\]\.\\mathrm\{ID@\}k=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbf\{1\}\\\!\\left\[P\_\{k\}^\{\(n\)\}\\cap V^\{\(n\)\}\\neq\\emptyset\\right\]\.ID@1 tests whether the first product matches an annotated identifier, while ID@3 allows a match anywhere among the first three products\. Because the verified identifiers are retained in the offline catalog, both metrics directly measure recovery of an annotated target\. They are nevertheless conservative: the catalog may contain additional products that satisfyIt\(n\)I\_\{t\}^\{\(n\)\}and every requirement inRt\(n\)R\_\{t\}^\{\(n\)\}but are not enumerated inV\(n\)V^\{\(n\)\}\. Judge@kkcan therefore exceed ID@kkby recognizing such valid alternatives\. For EGVS analysis, we additionally report Judge@Pool and ID@Pool, which apply the same two success criteria to the reconstructed candidate pool𝒞\(n\)\\mathcal\{C\}^\{\(n\)\}rather than to the final top\-three output\. These pool\-level scores are retrieval oracles, not leaderboard metrics\. Since LLM judges may introduce systematic bias\(Zheng et al\.[2023](https://arxiv.org/html/2607.29002#bib.bib26)\), we freeze the judge prompt and evidence schema across all evaluated agents\.

## Experiments

### Experimental Setup

Evaluation\.We evaluate Gemini\-3\.1\-Pro\-Preview\(Google DeepMind[2026](https://arxiv.org/html/2607.29002#bib.bib4)\), Claude Opus 4\.8\(Anthropic[2026](https://arxiv.org/html/2607.29002#bib.bib1)\), Kimi\-K2\.6\(Moonshot AI[2026](https://arxiv.org/html/2607.29002#bib.bib10)\), MiniMax\-M2\.7\(MiniMax[2026](https://arxiv.org/html/2607.29002#bib.bib9)\), and Qwen3\.5\-9B, Qwen3\.5\-27B, and Qwen3\.5\-122B\-A10B\(Qwen Team[2026](https://arxiv.org/html/2607.29002#bib.bib13)\)\. Every model receives the same multimodal dialogue history and tool interface\. In the reported EGVS snapshot, Gemini\-3\.1\-Pro\-Preview, Claude Opus 4\.8, MiniMax\-M2\.7, and Kimi\-K2\.6 use Thinking mode, whereas all Qwen3\.5 configurations use Non\-Thinking mode\. GPT\-5\.5 serves as the frozen multimodal judge, and we report Judge@1, Judge@3, ID@1, and ID@3 as defined above\.

SFT implementation\.We use the same training configuration for Qwen3\.5\-9B, Qwen3\.5\-27B, and Qwen3\.5\-122B\-A10B\. For each model, we freeze the visual module and fully optimize all remaining parameters for four epochs with a learning rate of5×10−65\\times 10^\{\-6\}on 32 NVIDIA A100 GPUs\.

### Main Results

Table 3:Main results on MMShopBench, reported as percentages after EGVS\. Best and second\-best values in each column are shown in bold and underlined, respectively\.Table[3](https://arxiv.org/html/2607.29002#Sx5.T3)reports end\-to\-end performance under the four primary metrics\. Gemini\-3\.1\-Pro\-Preview attains the strongest results, with 64\.7% Judge@1 and 61\.4% ID@1, yet its top\-ranked product satisfies the complete request in fewer than two thirds of cases, underscoring the difficulty of MMShopBench\. Supervised fine\-tuning yields a pronounced improvement for Qwen3\.5\-122B\-A10B: Judge@1 rises from 5\.5% to 52\.9% and Judge@3 from 5\.9% to 67\.5%, gains of 47\.4 and 61\.6 percentage points, respectively, while ID@1 and ID@3 increase from 4\.8% and 5\.9% to 49\.8% and 55\.4%\. Although the fine\-tuned model does not surpass the leading proprietary systems, it approaches their performance while operating in Non\-Thinking mode, trailing the best proprietary result by 5\.9 points on Judge@3\.

### Ablation Studies

Table 4:Ablation of Evidence\-Grounded Verification and Selection \(EGVS\) on MMShopBench\. Base and \+EGVS are evaluated with Judge@3 and ID@3, while Judge@Pool and ID@Pool are oracle success rates over the union of the top\-20 retrieved candidates and final selections\. Best and second\-best results in each numeric column are shown in bold and underlined\.EGVS\.Table[4](https://arxiv.org/html/2607.29002#Sx5.T4)reports fixed\-denominator performance before and after evidence\-grounded selection\. Gemini\-3\.1\-Pro\-Preview obtains the highest Judge@3, reaching 73\.4 after verification, and the supervised Qwen3\.5 variants substantially outperform their instruction\-tuned counterparts even before EGVS\. Applied after retrieval, EGVS raises Judge@3 for every configuration, with the largest gain on Qwen3\.5\-9B SFT \(\+15\.9 points\)\. Qwen3\.5\-122B\-A10B SFT achieves the strongest open\-model result at 67\.5, trailing Gemini by 5\.9 points\.

Because ID@kkmatches returned identifiers against the human\-verified setV\(n\)V^\{\(n\)\}without model\-side evidence reasoning, it shares neither the evidence schema nor the acceptance logic of the self\-verifier and is immune to verifier–judge alignment\. Its consistent rise under EGVS—for nine of ten configurations, led by\+10\.4\+10\.4points on Qwen3\.5\-9B SFT—confirms that verification recovers annotated targets rather than re\-scoring candidates toward the judge’s preferences\. Where a Judge@3 gain exceeds the corresponding ID@3 gain \(e\.g\., Claude Opus 4\.8,\+6\.6\+6\.6vs\.\+2\.4\+2\.4\), the difference reflects EGVS promoting valid products absent from the incomplete setV\(n\)V^\{\(n\)\}rather than verifier–judge coupling\.

The pool\-level Oracle columns bound remaining headroom: Qwen3\.5\-9B Instruct reaches 20\.4 Judge@3 against a 20\.8 Judge@Pool ceiling, recovering nearly all satisfying candidates in its pool, whereas the SFT agents remain 2\.1–8\.0 points below their ceilings, leaving room for better verification and ranking\.

Necessity of multimodal evaluation\.To test whether MMShopBench captures capabilities omitted by text\-only benchmarks, we rescreen online shopping\-assistant logs to form a separate 300\-case diagnostic set\. Claude Opus 4\.8 stratifies the set into three equally sized regimes based on modality dependence:*text\-sufficient*cases can be resolved from text alone,*image\-required*cases depend on visual evidence, and*mixed*cases require joint reasoning over both modalities\. We compare full and text\-only inputs using the same agent and prompt\.

Figure[3](https://arxiv.org/html/2607.29002#Sx5.F3)yields two complementary findings\. Under full multimodal input, mixed cases obtain the lowest Judge@3, indicating that reconciling constraints distributed across images and dialogue is the most challenging regime\. Removing images produces the largest degradation on image\-required cases, followed by mixed cases, while text\-sufficient cases change only modestly\. This shows that user images encode indispensable product constraints unrecoverable from text alone, so text\-only evaluation omits a core capability of real shopping agents, motivating MMShopBench’s preservation of user images and multi\-turn context\.

![Refer to caption](https://arxiv.org/html/2607.29002v1/x3.png)Figure 3:Effect of removing dialogue images across modality strata\. Full uses the original multimodal input, whereas Text\-only retains only the query text\.

## Conclusion and Future Work

We introduced MMShopBench, a real\-log benchmark for multimodal, multi\-turn shopping agents, with a reproducible 100,000\-product sandbox\. It evaluates whether agents can infer intent and requirements from images and dialogue, retrieve candidates, and verify them against frozen product evidence\. Experiments show supervised trajectory tuning substantially improves open\-model agents, while EGVS recovers valid products overlooked during selection, highlighting the importance of cross\-modal requirement inference and evidence\-grounded selection for realistic shopping assistance\. Future work will explore broader training strategies and scale MMShopBench with more real\-log interactions\.

## References

- Anthropic \(2026\)Anthropic\. 2026\.Claude Opus 4\.8 System Card\.https://www\.anthropic\.com/claude\-opus\-4\-8\-system\-card\.Accessed: 2026\-07\-28\.
- Du, Li, and Zhang \(2026\)Du, Z\.; Li, T\.; and Zhang, H\. 2026\.EComAgentBench: Benchmarking Shopping Agents on Long\-Horizon Tasks with Distributed Hidden Intent\.*arXiv preprint arXiv:2606\.17698*\.
- Fan et al\. \(2026\)Fan, S\.; Min, S\.; Hu, Y\.; Xia, B\.; Unnikrishnan, J\.; Musselmann, R\.; Gao, Y\.; Yin, Q\.; Nigam, P\.; and Yin, B\. 2026\.Shopping Reasoning Bench: An Expert\-Authored Benchmark for Multi\-Turn Conversational Shopping Assistants\.*arXiv preprint arXiv:2606\.12608*\.
- Google DeepMind \(2026\)Google DeepMind\. 2026\.Gemini 3\.1 Pro Model Card\.https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro\.Accessed: 2026\-07\-28\.
- Hou et al\. \(2026\)Hou, B\.; Gu, J\.; Guo, J\.; Dang, R\.; Leng, S\.; Li, X\.; Song, X\.; and Yang, J\. 2026\.InterLV\-Search: Benchmarking Interleaved Multimodal Agentic Search\.*arXiv preprint arXiv:2605\.07510*\.
- Jiang et al\. \(2025\)Jiang, D\.; Zhang, R\.; Guo, Z\.; Wu, Y\.; Lei, J\.; Qiu, P\.; Lu, P\.; Chen, Z\.; Song, G\.; Gao, P\.; Liu, Y\.; Li, C\.; and Li, H\. 2025\.MMSearch: Unveiling the Potential of Large Models as Multi\-Modal Search Engines\.In*International Conference on Learning Representations*\.
- Kottur et al\. \(2021\)Kottur, S\.; Moon, S\.; Geramifard, A\.; and Damavandi, B\. 2021\.SIMMC 2\.0: A Task\-oriented Dialog Dataset for Immersive Multimodal Conversations\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, 4903–4912\. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics\.
- Li et al\. \(2025\)Li, S\.; Bu, X\.; Wang, W\.; Liu, J\.; Dong, J\.; He, H\.; Lu, H\.; Zhang, H\.; Jing, C\.; Li, Z\.; Li, C\.; Tian, J\.; Zhang, C\.; Peng, T\.; He, Y\.; Gu, J\.; Zhang, Y\.; Yang, J\.; Zhang, G\.; Huang, W\.; Zhou, W\.; Zhang, Z\.; Ding, R\.; and Wen, S\. 2025\.MM\-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents\.*arXiv preprint arXiv:2508\.13186*\.
- MiniMax \(2026\)MiniMax\. 2026\.The MiniMax\-M2 Series: Mini Activations Unleashing Max Real\-World Intelligence\.*arXiv preprint arXiv:2605\.26494*\.
- Moonshot AI \(2026\)Moonshot AI\. 2026\.Kimi K2\.6: Advancing Open\-Source Coding\.https://www\.kimi\.com/blog/kimi\-k2\-6\.Accessed: 2026\-07\-28\.
- Ning et al\. \(2026\)Ning, X\.; Fu, D\.; Wei, T\.; Ai, M\.; Zou, J\.; Li, T\.\-W\.; Tong, H\.; Zhu, Y\.; Hamann, H\.; and He, J\. 2026\.MC\-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains\.In*The Fourteenth International Conference on Learning Representations*\.
- OpenAI \(2026\)OpenAI\. 2026\.GPT\-5\.5 System Card\.https://openai\.com/index/gpt\-5\-5\-system\-card/\.Accessed: 2026\-07\-28\.
- Qwen Team \(2026\)Qwen Team\. 2026\.Qwen3\.5: Towards Native Multimodal Agents\.https://qwen\.ai/blog?id=qwen3\.5\.Accessed: 2026\-07\-28\.
- Reddy et al\. \(2022\)Reddy, C\. K\.; Màrquez, L\.; Valero, F\.; Rao, N\.; Zaragoza, H\.; Bandyopadhyay, S\.; Biswas, A\.; Xing, A\.; and Subbian, K\. 2022\.Shopping Queries Dataset: A Large\-Scale ESCI Benchmark for Improving Product Search\.*arXiv preprint arXiv:2206\.06588*\.
- Robertson and Zaragoza \(2009\)Robertson, S\.; and Zaragoza, H\. 2009\.The Probabilistic Relevance Framework: BM25 and Beyond\.*Foundations and Trends in Information Retrieval*, 3\(4\): 333–389\.
- Tao et al\. \(2026\)Tao, X\.; Teng, Y\.; Su, X\.; Fu, X\.; Wu, J\.; Tao, C\.; Liu, Z\.; Bai, H\.; Liu, R\.; and Kong, L\. 2026\.MMSearch\-Plus: Benchmarking Provenance\-Aware Search for Multimodal Browsing Agents\.In*International Conference on Learning Representations*\.
- Wang et al\. \(2026a\)Wang, H\.; Wan, D\.; Lee, H\.; Pham, T\.; Cankosyan, M\.; Chen, W\.; Stengel\-Eskin, E\.; Vu, T\.; and Bansal, M\. 2026a\.MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments\.*arXiv preprint arXiv:2604\.13418*\.
- Wang et al\. \(2026b\)Wang, J\.; Xiao, K\.; Sun, Q\.; Zhao, H\.; Luo, T\.; Zhang, J\. D\.; and Zeng, X\. 2026b\.Shoppingbench: A real\-world intent\-grounded shopping benchmark for llm\-based agents\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, 33521–33529\.
- Wang et al\. \(2026c\)Wang, P\.; Wu, Y\.; Song, X\.; Wang, W\.; Chen, G\.; Li, Z\.; Yan, K\.; Liu, Q\.; Deng, K\.; Zhao, S\.; Xiong, S\.; Liu, X\.; Chen, X\.; Deng, W\.; Su, W\.; and Zheng, B\. 2026c\.ShopSimulator: Evaluating and Exploring RL\-Driven LLM Agent for Shopping Assistants\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 35600–35619\.
- Yao et al\. \(2022\)Yao, S\.; Chen, H\.; Yang, J\.; and Narasimhan, K\. 2022\.WebShop: Towards Scalable Real\-World Web Interaction with Grounded Language Agents\.In*Advances in Neural Information Processing Systems*, volume 35, 20744–20757\.
- Yao et al\. \(2023\)Yao, S\.; Zhao, J\.; Yu, D\.; Du, N\.; Shafran, I\.; Narasimhan, K\.; and Cao, Y\. 2023\.ReAct: Synergizing Reasoning and Acting in Language Models\.In*International Conference on Learning Representations*\.
- Yu et al\. \(2026\)Yu, Z\.; Xiao, K\.; Zhao, H\.; Luo, T\.; and Zeng, X\. 2026\.Shopping Companion: Benchmarking and Training LLM Agents for Long\-Horizon Preference\-Grounded E\-Commerce Tasks\.*arXiv preprint arXiv:2603\.14864*\.
- Zhang et al\. \(2026a\)Zhang, H\.; Zhou, J\.; Li, B\.; Zhou, B\.; Shan, Y\.; Lu, H\.; Cao, Z\.; Chen, J\.; Han, Y\.; Sheng, Z\.; Tao, Z\.; Liang, H\.; Wu, J\.; Shi, Y\.; He, Y\.; Lin, J\.; Zhang, Q\.; Yan, G\.; Zhao, R\.; Li, Z\.; Yu, X\.; Mei, L\.; Chen, C\.; Zhang, W\.; and Cui, B\. 2026a\.BrowseComp\-V3V^\{3\}: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents\.*arXiv preprint arXiv:2602\.12876*\.
- Zhang et al\. \(2026b\)Zhang, Z\.; Miao, C\.; Su, J\.; Zhou, Z\.; Zhang, C\.; Wang, X\.; Liu, R\.; Zheng, K\.; Cai, J\.; Zhang, B\.; Li, Z\.; Xiang, S\.; and Yan, Y\. 2026b\.Visual\-Seeker: Towards Visual\-Native Multimodal Agentic Search via Active Visual Reasoning\.*arXiv preprint arXiv:2606\.15231*\.
- Zhao et al\. \(2022\)Zhao, N\.; Li, H\.; Wu, Y\.; and He, X\. 2022\.JDDC 2\.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, 12037–12051\. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics\.
- Zheng et al\. \(2023\)Zheng, L\.; Chiang, W\.\-L\.; Sheng, Y\.; Zhuang, S\.; Wu, Z\.; Zhuang, Y\.; Lin, Z\.; Li, Z\.; Li, D\.; Xing, E\. P\.; Zhang, H\.; Gonzalez, J\. E\.; and Stoica, I\. 2023\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.In*Advances in Neural Information Processing Systems*, volume 36, 46595–46623\.
- Zhu, Jung, and Clark \(2025\)Zhu, T\.; Jung, M\. C\.; and Clark, J\. 2025\.Generalized Contrastive Learning for Multi\-Modal Retrieval and Ranking\.In*Companion Proceedings of the ACM on Web Conference 2025*, 661–670\.

## Appendix S1Qualitative Examples

### S1\.1EGVS Case: Recovering a Valid Candidate from the Retrieval Pool

Figure[S1](https://arxiv.org/html/2607.29002#A1.F1)isolates the benefit of EGVS from retrieval quality: the satisfying product is already present in the candidate pool, so the gain comes from explicit verification and reselection rather than additional search\. By checking the referenced object, wood appearance, and shape evidence against the complete multimodal request, EGVS suppresses plausible distractors and recovers an evidence\-supported recommendation\.

![[Uncaptioned image]](https://arxiv.org/html/2607.29002v1/x4.png)

Figure S1:Qualitative EGVS case\. The dialogue refers to the wooden tray in the background rather than the foreground kettle\. Retrieval contains a satisfying tray, but the baseline final selection discards it; EGVS verifies each candidate against the cross\-turn textual and visual constraints and promotes the supported item\.

### S1\.2MMShopBench Data Visualization

![Refer to caption](https://arxiv.org/html/2607.29002v1/x5.png)Figure S2:MMShopBench example with three dialogue rounds\. The user requests fewer compartments and then lighter colors; the bottom panel records the associated annotated intent and mandatory attributes\.![Refer to caption](https://arxiv.org/html/2607.29002v1/x6.png)Figure S3:MMShopBench image\-grounded retrieval example\. The dialogue starts from a close\-up of an electric\-tricycle frame and asks for a mountable storage box; the bottom panel shows the associated annotated intent and mandatory attributes\.![[Uncaptioned image]](https://arxiv.org/html/2607.29002v1/x7.png)![[Uncaptioned image]](https://arxiv.org/html/2607.29002v1/x8.png)![[Uncaptioned image]](https://arxiv.org/html/2607.29002v1/x9.png)![[Uncaptioned image]](https://arxiv.org/html/2607.29002v1/x10.png)