APeB: Benchmarking Personalization Ability of Large Language Model Agents
Summary
Introduces APeB, a benchmark for evaluating personalization in LLM agents, focusing on inferring user intent and preferences from raw queries and interaction histories. Finds that current models struggle with early-stage queries and that history-aware refinement can help.
arXiv:2607.03162v1 Announce Type: new
Abstract: LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.
View Cached Full Text
Cached at: 07/07/26, 04:34 AM
# Benchmarking Personalization Ability of Large Language Model Agents Source: [https://arxiv.org/html/2607.03162](https://arxiv.org/html/2607.03162) [Garry Yang1,\*](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Zizhe Chen1,\*](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Xinru Chen2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Yongqiang Chen1](https://arxiv.org/html/2607.03162v1/mailto:[email protected]) [Jianxiang Wang2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Deyu Zou1](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Linyi Ding2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Jialiang Wu2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]) [Yunzhong He2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Yu Gong2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[James Cheng1,†](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Huaixiao Tou2,†](https://arxiv.org/html/2607.03162v1/mailto:[email protected]) 1The Chinese University of Hong Kong2ByteDance ###### Abstract LLM\-powered agents struggle with personalization when users issue raw, underspecified queries\. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives\. Existing benchmarks rarely test this capability, as they often rely on user\-refined queries or simplified histories\. We introduce personalized product search \(PPS\), a testbed for agentic personalization under raw queries and diverse histories\. We constructAgentPersonalizedBenchmark \(APeB\) from action logs, pairing underspecified intents with rich histories and user\-viewed candidate items\. Evaluating state\-of\-the\-art LLMs with multi\-step agent workflows, we find that models handle explicit queries well but struggle with early\-stage queries requiring intent and preference discovery\. Rubric analysis attributes this gap mainly to ineffective history use\. A simple history\-aware query\-refinement pipeline,VQRA, yields consistent gains, highlighting the need for dedicated history\-utilization modules in personalized agents\. APeB: Benchmarking Personalization Ability of Large Language Model Agents [Garry Yang1,\*](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Zizhe Chen1,\*](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Xinru Chen2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Yongqiang Chen1](https://arxiv.org/html/2607.03162v1/mailto:[email protected])[Jianxiang Wang2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Deyu Zou1](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Linyi Ding2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Jialiang Wu2](https://arxiv.org/html/2607.03162v1/mailto:[email protected])[Yunzhong He2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Yu Gong2](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[James Cheng1,†](https://arxiv.org/html/2607.03162v1/mailto:[email protected]),[Huaixiao Tou2,†](https://arxiv.org/html/2607.03162v1/mailto:[email protected])1The Chinese University of Hong Kong2ByteDance 11footnotetext:Equal contribution\.22footnotetext:Corresponding authors\.## 1Introduction Figure 1:Personalization challenge for agents: infer intent and preferences from diverse history to make a preference\-aligned decision\.Table 1:Comparison of existing benchmarks by whether and how they capture core personalization challenges\.Large Language Model \(LLM\)\-powered agents extend foundation models with additional modules for memory, planning, and execution, enabling multi\-step, goal\-directed reasoning over long contextsPeng et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib37)\); Wang et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib58)\); Zhang et al\. \([2025c](https://arxiv.org/html/2607.03162#bib.bib76)\); Hu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib20)\)\. As shown in Figure[1](https://arxiv.org/html/2607.03162#S1.F1), a central challenge for LLM\-powered agents is*personalization*: user objectives are not explicitly specified; preferences are implicit and partially observed through histories; and correctness is user\-centric rather than globally optimalRussell \([2021](https://arxiv.org/html/2607.03162#bib.bib46)\)\. In practice, users implicitly reason over their own past experience, translate vague intent into specific criteria, and discriminate between plausible optionsDowney et al\. \([2008](https://arxiv.org/html/2607.03162#bib.bib11)\); Punj and Moore \([2009](https://arxiv.org/html/2607.03162#bib.bib39)\); Liu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib26)\)\. Accordingly, a personalization agent should inherently align its planning and action to one particular user’s intents rather than generate generic responses following a fully specified goalZhang et al\. \([2025c](https://arxiv.org/html/2607.03162#bib.bib76)\)\. Therefore, we ask > *How can we benchmark the agentic personalization capability of LLMs?* To our knowledge, existing benchmarks individually address one or two of these axes but never combine noisy heterogeneous histories, vague\-to\-refined search trajectories, and closely competing candidates\.*Recommendation benchmarks*cast personalization as next action prediction from historical signals, emphasizing preference deduction over objective understandingZhu et al\. \([2018](https://arxiv.org/html/2607.03162#bib.bib78)\); Liu et al\. \([2016](https://arxiv.org/html/2607.03162#bib.bib25)\)\.*Rationale prediction benchmarks*evaluate localized reasoning, focusing on generating intermediate justifications in isolation rather than reaching a user\-aligned outcomeWang et al\. \([2025c](https://arxiv.org/html/2607.03162#bib.bib62)\); Yang et al\. \([2025b](https://arxiv.org/html/2607.03162#bib.bib68)\)\.*Multi\-turn Memory benchmarks*model personalization over largely homogeneous conversations, evaluating relevant information extraction instead of intent inference under ambiguityHu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib20)\); Huang et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib21)\)\. In all, these benchmarks sidestep the need to jointly reason over noisy histories, unrefined user queries, and closely competing candidates\. As a result, they also fail to evaluate whether agentic pipelines \(e\.g\., ReAct\) contribute to intent inference and preference\-aligned actionsDu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib12)\)\. Motivated by these limitations, we constructAPeB\(AgentPersonalizedBenchmark\) to evaluate agents’ personalization ability through the*Personalized Product Search*\(PPS\) task\. PPS frames personalization as a user\-specific choice among competing products conditioned on a shopping intent and long\-term behavioral preferenceRobertson and Zaragoza \([2009](https://arxiv.org/html/2607.03162#bib.bib45)\); Duan and Zhai \([2015](https://arxiv.org/html/2607.03162#bib.bib13)\)\. On online platforms, user behavior is noisy across domains, shopping intent is often vague and early\-stage, and purchases occur only after comparing similar productsLiu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib27)\); Reddy et al\. \([2022](https://arxiv.org/html/2607.03162#bib.bib42)\); Downey et al\. \([2008](https://arxiv.org/html/2607.03162#bib.bib11)\); Rendle et al\. \([2012](https://arxiv.org/html/2607.03162#bib.bib43)\)\. However, prior*PPS benchmarks*rely on sanitized histories, clarified queries, and easily separable candidatesLiu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib26)\); Hou et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib19)\), thereby failing to evaluate personalization under ambiguous, reasoning\-intensive conditions\. The preference\-aligned purchase also contrasts with*generic shopping benchmarks*that focus on isolated sub\-tasks \(e\.g\., product matching\) without grounding decisions in user\-specific intents and historiesJin et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib24)\); Yao et al\. \([2022](https://arxiv.org/html/2607.03162#bib.bib70)\)\. Comparison with related benchmarks is shown in Table[1](https://arxiv.org/html/2607.03162#S1.T1)\. Full related work is in Appendix[A](https://arxiv.org/html/2607.03162#A1)\. In turn,APeBcontains intricate cases drawn from a leading content platform with shopping functionality\. It includes 5,648 cases from 10\+ categories collected from a 30\-day retained\-session window with up to 60 days of pre\-session history \(Section[3](https://arxiv.org/html/2607.03162#S3)\)\. Users primarily consume videos and livestreams, yet can seamlessly transition to product search, browsing, and purchase within the same platform\. This end\-to\-end flow yields rich, heterogeneous interaction histories that capture real\-world personalization challenges \(see Appendix[B\.1](https://arxiv.org/html/2607.03162#A2.SS1)\)\. We disentangle user behavior into a long\-term history period and a personalization shopping session, delimited by the timestamp of a user\-issued search \(Section[2\.1\.1](https://arxiv.org/html/2607.03162#S2.SS1.SSS1)\)\. Sessions are retained only if the search leads to a query\-aligned purchase following reasoning\-intensive browsing \(Section[2\.1\.2](https://arxiv.org/html/2607.03162#S2.SS1.SSS2)\)\. The history period comprises diverse interactions with products and contextual content \(e\.g\., videos and livestreams\), providing long\-term personalization signals \(Section[2\.2\.1](https://arxiv.org/html/2607.03162#S2.SS2.SSS1)\)\. The shopping session contains user searches spanning vanilla intent to refined keywords \(Section[2\.2\.2](https://arxiv.org/html/2607.03162#S2.SS2.SSS2)\), together with a set of heavily viewed competitive candidates \(Section[2\.2\.3](https://arxiv.org/html/2607.03162#S2.SS2.SSS3)\) and a final observed purchase, enabling intent evaluation and verifiable user\-centric assessment\. We evaluate whether personalization agents can first infer a user’s latent preferences from vague intents and history hints \(*Planning*\) and select among close alternatives \(*Action*\) on the user’s behalf \(the pipeline defined in Figure[1](https://arxiv.org/html/2607.03162#S1.F1)\)\. We test latest LLMs with agentic workflows onAPeB\. Evaluation spansintentandrefinedqueries with random and hard candidates, and includes comparison against a PPS model UniSARShi et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib51)\)\. Results show that \(1\) in settings dominated by semantic alignment rather than personalization \(refinedqueries or random candidates\), LLMs substantially outperform UniSAR, and ReAct workflows further improve performance; \(2\) for tasks requiring user modeling and intent refinement \(intentwith hard candidates\), LLMs fall to the level of zero\-shot UniSAR, with limited gains from ReAct workflows; \(3\) rubric analysis shows thatintentqueries expose weak intent and history attribution, with ReAct further harming performance via poor multi\-step planning; \(4\) we find that a simple history\-groundedVagueQueryRefinementAgent \(VQRA\) pipeline improves intent inference and Hit@1 over single\-prompt underintent\. Overall,APeBserves as a testbed for evaluating agentic systems under realistic personalization constraints\. ## 2APeBConstruction Figure 2:Construction Pipeline ofAPeB: from behavioral logs to benchmark components: curated heterogeneous interactions, non\-trivial shopping session discovery \(search refinement, extensive browsing, and intent–order alignment\), andintent/refinedquery–candidate pair identification\.APeBtargets challenging personalization grounded in holistic user behavior\.We introduceAPeB, a benchmark designed for evaluatingagentic personalization capabilityon the PPS task\. To retain cases in which the personalized decisions are time\-consuming and interaction\-heavy \(Section[2\.1\.2](https://arxiv.org/html/2607.03162#S2.SS1.SSS2)\), we collectnon\-trivial user shopping sessionswith their heterogeneous histories mined from holistic user behavior logs \(Appendix[B](https://arxiv.org/html/2607.03162#A2)\)\. As shown in Figure[2](https://arxiv.org/html/2607.03162#S2.F2), a*session*is defined as a time\-bounded search\-to\-order window that begins with a search and ends with a purchase \(Section[2\.1\.1](https://arxiv.org/html/2607.03162#S2.SS1.SSS1)\)\. Non\-trivial sessions involve query reformulation and extensive product comparison before an intent\-aligned order \(Section[2\.1\.2](https://arxiv.org/html/2607.03162#S2.SS1.SSS2)\), reflecting how user history shapes query formulation, exploration, and decision\-makingShang et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib48)\); Zuo et al\. \([2022](https://arxiv.org/html/2607.03162#bib.bib81)\)\. These sessions inherently require an agent to jointly reason over long\-term user preferences, vanilla query intent, and fine\-grained candidate differences, rather than relying on surface\-level semantic matching\. For filtered sessions, we construct three benchmark components: \(i\)Heterogeneous histories–preference signals \(Figure[2](https://arxiv.org/html/2607.03162#S2.F2)Left, Section[2\.2\.1](https://arxiv.org/html/2607.03162#S2.SS2.SSS1)\); \(ii\)User\-issuedintent\(vague, early\-stage\) andrefined\(well\-specified, late\-stage\) queries–different levels of query specificity \(Figure[2](https://arxiv.org/html/2607.03162#S2.F2)Right, Section[2\.2\.2](https://arxiv.org/html/2607.03162#S2.SS2.SSS2)\); and \(iii\)Hard candidate set–products the user considered, together with theobserved purchased item\(Figure[2](https://arxiv.org/html/2607.03162#S2.F2)Right, Section[2\.2\.3](https://arxiv.org/html/2607.03162#S2.SS2.SSS3)\)\. ### 2\.1Non\-trivial Shopping Session Discovery To identify*non\-trivial shopping sessions*from behavior logs, we first retain users with sufficient interaction density over a 60\-day window \(see Appendix[B\.2](https://arxiv.org/html/2607.03162#A2.SS2)\)\. Following the long–short period division paradigmLiu et al\. \([2016](https://arxiv.org/html/2607.03162#bib.bib25)\), we then extractsearch\-to\-order sessionsby anchoring boundaries at search timestamps\. From these, we further selectnon\-trivial casescharacterized by query evolution, extensive content browsing, and cross\-product comparison prior to a personalized purchase decision\. #### 2\.1\.1Search\-to\-Order Session Disentangle We disentangle each user’s interaction sequence by the search action timestamp \(‘\{’ in Figure[2](https://arxiv.org/html/2607.03162#S2.F2)\)\. For a useruu, letℐu=\{\(xuk,tuk\)\}k=1Nu\\mathcal\{I\}\_\{u\}=\\\{\(x\_\{u\}^\{k\},t\_\{u\}^\{k\}\)\\\}\_\{k=1\}^\{N\_\{u\}\}denote the chronologically ordered interaction log, wherexukx\_\{u\}^\{k\}is an action andtukt\_\{u\}^\{k\}its timestamp\. We consider an initial search actionxum=query\(q\)x\_\{u\}^\{m\}=\\mathrm\{query\}\(q\)at timetumt\_\{u\}^\{m\}and a final purchase actionxun=order\(p\)x\_\{u\}^\{n\}=\\mathrm\{order\}\(p\)at timetunt\_\{u\}^\{n\}\. A*search\-to\-order session*is identified if the purchase occurs within a fixed time windowΔ=30\\Delta=30minutes after the search:tun−tum≤Δ\.t\_\{u\}^\{n\}\-t\_\{u\}^\{m\}\\leq\\Delta\.Users without any such session are removed; if multiple sessions exist, we retain one per user\. For each retained session, the search timetumt\_\{u\}^\{m\}and the purchase timetunt\_\{u\}^\{n\}define the disentanglement boundary: ℋu\\displaystyle\\mathcal\{H\}\_\{u\}=\{\(xuk,tuk\)∈ℐu:tuk<tum\},\\displaystyle=\\\{\(x\_\{u\}^\{k\},t\_\{u\}^\{k\}\)\\in\\mathcal\{I\}\_\{u\}:t\_\{u\}^\{k\}<t\_\{u\}^\{m\}\\\},\(1\)𝒮u\\displaystyle\\mathcal\{S\}\_\{u\}=\{\(xuk,tuk\)∈ℐu:tum≤tuk≤tun\}\.\\displaystyle=\\\{\(x\_\{u\}^\{k\},t\_\{u\}^\{k\}\)\\in\\mathcal\{I\}\_\{u\}:t\_\{u\}^\{m\}\\leq t\_\{u\}^\{k\}\\leq t\_\{u\}^\{n\}\\\}\.\(2\)Here,ℋu\\mathcal\{H\}\_\{u\}captures long\-term preference signals, while𝒮u\\mathcal\{S\}\_\{u\}contains goal\-driven queries and candidates\. This disentanglement separates preference inference from task execution , enabling reliable evaluation of personalization reasoning\. #### 2\.1\.2Non\-trivial Session Filtering For each useruuwith a session of ordered events𝒮u\\mathcal\{S\}\_\{u\}\(Section[2\.1\.1](https://arxiv.org/html/2607.03162#S2.SS1.SSS1)\), each event is one of*search*,*product browsing*,*media \(video/livestream\) browsing*, or*purchase*\. It can be represented as 𝒮u=\(𝒬usearch,𝒫uview,𝒱uview,𝒫uorder\),\\mathcal\{S\}\_\{u\}=\\big\(\\mathcal\{Q\}\_\{u\}^\{\\mathrm\{search\}\},\\ \\mathcal\{P\}\_\{u\}^\{\\mathrm\{view\}\},\\ \\mathcal\{V\}\_\{u\}^\{\\mathrm\{view\}\},\\ \\mathcal\{P\}\_\{u\}^\{\\mathrm\{order\}\}\\big\),where𝒬usearch\\mathcal\{Q\}\_\{u\}^\{\\mathrm\{search\}\}is the ordered list of search queries,𝒫uview\\mathcal\{P\}\_\{u\}^\{\\mathrm\{view\}\}the product\-interactions,𝒱uview\\mathcal\{V\}\_\{u\}^\{\\mathrm\{view\}\}the media\-interactions, and𝒫uorder\\mathcal\{P\}\_\{u\}^\{\\mathrm\{order\}\}the purchased products\. To obtain non\-trivial shopping sessions, we formulate these three conditions: \(1\) Iterative query refinement\.The user progressively clarifies intent through multiple searches: \|𝒬usearch\|≥2\.\|\\mathcal\{Q\}\_\{u\}^\{\\mathrm\{search\}\}\|\\geq 2\.\(3\) \(2\) Extensive\-browsing behavior\.The user exhibits substantial exploration prior to purchase: \|𝒫uview\|\+\|𝒱uview\|≥τv,\|\\mathcal\{P\}\_\{u\}^\{\\mathrm\{view\}\}\|\+\|\\mathcal\{V\}\_\{u\}^\{\\mathrm\{view\}\}\|\\geq\\tau\_\{v\},\(4\)whereτv\\tau\_\{v\}is a minimum interaction threshold\. \(3\) Intent–order alignment\.The final purchase must be semantically consistent with both the user’s expressed intent and explored candidates\. Specifically, there exists a purchased productpuo∈𝒫uorderp\_\{u\}^\{o\}\\in\\mathcal\{P\}\_\{u\}^\{\\mathrm\{order\}\}such that ∃q∈𝒬us\.t\.𝕀\[Align\(q,puo\)\]=1,\\displaystyle\\exists\\,q\\in\\mathcal\{Q\}\_\{u\}\\ \\text\{s\.t\.\}\\ \\mathbb\{I\}\[\\mathrm\{Align\}\(q,p\_\{u\}^\{o\}\)\]=1,\(5\)\|\{v∈𝒫uview:𝕀\[Align\(v,puo\)\]=1\}\|≥τa,\\displaystyle\\left\|\\left\\\{\\,v\\in\\mathcal\{P\}\_\{u\}^\{\\mathrm\{view\}\}:\\mathbb\{I\}\[\\mathrm\{Align\}\(v,p\_\{u\}^\{o\}\)\]=1\\right\\\}\\right\|\\geq\\tau\_\{a\},whereτa\\tau\_\{a\}is a minimum alignment count\. If multiple purchases satisfy, we retain only one\. Criteria are primarily enforced via rule\-based filtering \(Appendix[B\.3\.1](https://arxiv.org/html/2607.03162#A2.SS3.SSS1)\)\. Semantic alignment in \(3\) is assessed using GPT\-4\.1 with human verification \(Appendix[B\.3\.2](https://arxiv.org/html/2607.03162#A2.SS3.SSS2)\)\. We remove unsatisfied sessions and retain one session per user\. These criteria target a common high\-effort shopping regime, 67% of users exhibit multi\-step shopping behavior in a long session, defined as issuing at least two queries and comparing at least ten products before purchase \(Appendix[B\.5\.2](https://arxiv.org/html/2607.03162#A2.SS5.SSS2)\)\. ### 2\.2History & Query\-Candidate Identification For users with retained sessions, we construct their heterogeneous histories and aligned query–candidate pairs\. #### 2\.2\.1Heterogeneous History Curation Heterogeneous long\-term interactions provide rich signals for inferring user persona\. For a useruuwith historical interaction logℋu\\mathcal\{H\}\_\{u\}\(Eq\.[1](https://arxiv.org/html/2607.03162#S2.E1)\), we first filter out low–dwell\-time records \(Appendix[B\.4\.1](https://arxiv.org/html/2607.03162#A2.SS4.SSS1)\)\. We then consider two complementary types for the remaining interactions\. \(1\) Product interactions\.Each product interaction is defined as xu,productk=\(metapk,typek,Δtk\),x\_\{u,\\mathrm\{product\}\}^\{k\}=\(\\mathrm\{meta\}\_\{p\}^\{k\},\\ \\mathrm\{type\}^\{k\},\\ \\Delta t^\{k\}\),\(6\)wheremetapk=\(id,cat,title,detail,features\)\\mathrm\{meta\}\_\{p\}^\{k\}=\(\\mathrm\{id\},\\ \\mathrm\{cat\},\\ \\mathrm\{title\},\\ \\mathrm\{detail\},\\ \\mathrm\{features\}\)encodes product metadata,\{view, like, cart, order\}\\\{\\text\{view, like, cart, order\}\\\}denotes the interaction type,Δtk\\Delta t^\{k\}is the dwell time\. \(2\) Media interactions\.A media \(video or livestream\) interaction is represented as xu,mediak=\(metamk,Δtk\),x\_\{u,\\mathrm\{media\}\}^\{k\}=\(\\mathrm\{meta\}\_\{m\}^\{k\},\\ \\Delta t^\{k\}\),\(7\)wheremetamk=\(id,tags,title,detail,𝒫referk\),\\mathrm\{meta\}\_\{m\}^\{k\}=\(\\mathrm\{id\},\\ \\mathrm\{tags\},\\ \\mathrm\{title\},\\ \\mathrm\{detail\},\\ \\mathcal\{P\}\_\{\\mathrm\{refer\}\}^\{k\}\),and𝒫referk\\mathcal\{P\}\_\{\\mathrm\{refer\}\}^\{k\}denotes the set of products referenced or mentioned in the media, whileΔtk\\Delta t^\{k\}captures user engagement duration \(see Appendix[B\.4\.2](https://arxiv.org/html/2607.03162#A2.SS4.SSS2)\)\. #### 2\.2\.2Intent&RefinedQuery We distinguish two query types from the query refinement trajectory based on their position and role\. For each useruu, let𝒬usearch=\(qu1,…,qu\|𝒬usearch\|\)\\mathcal\{Q\}\_\{u\}^\{\\mathrm\{search\}\}=\(q\_\{u\}^\{1\},\\ldots,q\_\{u\}^\{\|\\mathcal\{Q\}\_\{u\}^\{\\mathrm\{search\}\}\|\}\)denote the temporally ordered search queries in the retained search\-to\-order session, and letpuop\_\{u\}^\{o\}be the purchased product\. We define: \(i\)Intentqueryquintentq\_\{u\}^\{\\mathrm\{\\text\{intent\}\}\}: the earliest query in𝒬u\\mathcal\{Q\}\_\{u\}that is semantically aligned withpuop\_\{u\}^\{o\}, denoting an intentional but ambiguous goal \(e\.g\.*campus ootd*\)\. \(ii\)Refinedqueryqurefinedq\_\{u\}^\{\\mathrm\{\\text\{refined\}\}\}: the last query issued before purchase that is explicit in product type/attributes, representing a well\-specified intent \(e\.g\.*oversized long\-sleeve*\)\. Query–product alignment is judged using GPT\-4\.1 with human verification\. We discard cases containing only a single aligned query \(Appendix[B\.3\.3](https://arxiv.org/html/2607.03162#A2.SS3.SSS3)\)\. Unlikerefined, which primarily evaluates semantic retrievalLiu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib26)\),intentreflects early\-stage search behavior requiring intent inferenceDowney et al\. \([2008](https://arxiv.org/html/2607.03162#bib.bib11)\)\. The distinction is defined by the session trajectory, not a fixed semantic boundary—identical query forms can reflect different levels of ambiguity at different stages of refinement\. Accordingly,intentqueries stress an agent’s capacity to resolve underspecified goals into actionable intent under personalization\. #### 2\.2\.3Hard Candidate Identification Explored but unbought items form*hard negatives*that are semantically close to the final choice\. For the retained session of useruu, the hard candidate set is defined as 𝒞uhard=𝒫uview∪\{puo\},\\mathcal\{C\}\_\{u\}^\{\\mathrm\{hard\}\}=\\mathcal\{P\}\_\{u\}^\{\\mathrm\{view\}\}\\cup\\\{p\_\{u\}^\{o\}\\\},\(8\)where each candidatec∈𝒞uhardc\\in\\mathcal\{C\}\_\{u\}^\{\\mathrm\{hard\}\}is represented by its product metadatametac\\mathrm\{meta\}\_\{c\}\. Distinguishingpuop\_\{u\}^\{o\}from𝒞uhard\\mathcal\{C\}\_\{u\}^\{\\mathrm\{hard\}\}requires agents to replicate the user’s personalized, preference\-aligned comparison process over fine\-grained attributes\. Appendix[B\.5\.1](https://arxiv.org/html/2607.03162#A2.SS5.SSS1)shows that the purchased item is more distinguishable when user\-history signals are considered\. ### 2\.3Benchmark Evaluation APeBevaluates models under two query settings: Input\.For query typeτ∈\{intent,refined\}\\tau\\in\\\{\\text\{intent\},\\text\{refined\}\\\}: Inputuτ=\(quτ,ℋu,𝒞uhard\)\.\\text\{Input\}\_\{u\}^\{\\tau\}=\(q\_\{u\}^\{\\tau\},\\mathcal\{H\}\_\{u\},\\mathcal\{C\}\_\{u\}^\{\\mathrm\{hard\}\}\)\.\(9\) Output and Metrics\.GivenInputuτ\\text\{Input\}\_\{u\}^\{\\tau\}, the model ranks products from𝒞uhard\\mathcal\{C\}\_\{u\}^\{\\mathrm\{hard\}\}and outputs akk\-size list: ℛ^u=Rankθ\(Inputuτ\),\|ℛ^u\|=k\.\\hat\{\\mathcal\{R\}\}\_\{u\}=\\operatorname\{Rank\}\_\{\\theta\}\(\\text\{Input\}\_\{u\}^\{\\tau\}\),\\quad\|\\hat\{\\mathcal\{R\}\}\_\{u\}\|=k\.\(10\)We setk=5and report Hit@1/5, measuring whether the purchased itempuop\_\{u\}^\{o\}appears in the ranked list\. ## 3Benchmark Post\-Processing AnonymizationBefore evaluation, user behavior data are anonymized to prevent personal information leakage\. Raw user identifiers are removed, and each interaction sequence is mapped to a randomly assigned surrogate ID with no cross\-split linkability\. Temporal signals are discretized into coarse\-grained time bins to avoid fine\-grained behavioral tracing\. We retain public item and content metadata to support heterogeneous personalization tasks\. Meanwhile, we apply automated personally identifiable information redaction to all free\-form text—particularly OCR and ASR outputs—to remove explicit and implicit identifiers\.APeBis planned for controlled release after compliance and privacy review with data minimization\. Appendix[B\.4\.3](https://arxiv.org/html/2607.03162#A2.SS4.SSS3)documents the details\. StatisticsAPeBcontains 5,648 retained cases in English, spanning 10\+ categories \(e\.g\.,*Personal Care*,*Clothing*,*Food & Beverages*\)\. On average, each user has 187 historical interactions, including 82 product and 105 media interactions\. These histories contain an average of 18 historical orders\. The average size of the candidate set is 14\.1\. Details are given in Appendix[B\.5](https://arxiv.org/html/2607.03162#A2.SS5)\. ## 4Experiments and Analysis Table 2:Hit@1/5 performance for LLM workflows \(Single\-Prompt \(60 category\-prior selected records\), ReAct, Deerflow and VQRA\) and PPS model UniSAR underintentandrefinedqueries\. VQRA denotes the setting where theintentquery has been rewritten with a shared backbone \(GPT\-4\.1\) using selected history context before evaluation\.We evaluate personalized shopping goal completion and intermediate reasoning onAPeBunderrefinedandintentacross 8 LLMs and their agentic variants, addressing the following questions: RQ1:How do LLM workflows perform compared to recommender PPS methods? We compare hit rates onintentandrefinedqueries with different history/candidate sets\. RQ2:Which types of flaws most adversely affect personalization tasks? We conduct a rubric analysis to attribute performance drop to intermediate reasoning errors\. RQ3:How can the failures in RQ2 be addressed? We use aVagueQueryRefinementAgent \(VQRA\) to demonstrate that explicit history utilization modules reduce this gap\. ### 4\.1Experimental Setup #### 4\.1\.1Evaluated Models We evaluate: \(1\) PPS baseline: UniSARShi et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib51)\)\. \(2\) LLMs: Qwen3\-235BYang et al\. \([2025a](https://arxiv.org/html/2607.03162#bib.bib67)\), DeepSeek\-R1Guo et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib16)\); GPT\-4/5 seriesOpenAI et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib34)\); Singh et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib52)\), Gemini\-2\.5\-ProGoogle DeepMind \([2025](https://arxiv.org/html/2607.03162#bib.bib15)\)\. \(3\) LLM Agents: ReActYao et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib71)\), DeerflowBytedance \([2025](https://arxiv.org/html/2607.03162#bib.bib5)\)\. Appendix[C\.2](https://arxiv.org/html/2607.03162#A3.SS2)discusses additional baselines\. #### 4\.1\.2Benchmark Utilization Due to the large semantic volume, we utilizeAPeB: \(1\) UniSAR \- Supervised TrainingFollowingSun et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib54)\), we construct SAR training data from historical query–order pairs and isolated purchases\. The training history is defined asℋuSAR=\{iuk\}k=1\|ℋuSAR\|\\mathcal\{H\}\_\{u\}^\{\\text\{SAR\}\}=\\\{i\_\{u\}^\{k\}\\\}\_\{k=1\}^\{\|\\mathcal\{H\}\_\{u\}^\{\\text\{SAR\}\}\|\}, where each interactioniuk∈\{\(quk,puk\),puk\}i\_\{u\}^\{k\}\\in\\\{\(q\_\{u\}^\{k\},p\_\{u\}^\{k\}\),\\,p\_\{u\}^\{k\}\\\}\. \(2\) LLM \- Single PromptTo fit histories, queries, and candidates within the LLM context window, we retainkkrecords fromℋu\\mathcal\{H\}\_\{u\}\. We evaluate category\-prior and time\-prior history selection \(same\-category label associated with the intent query vs\. most\-recent records\), keeping only essential fields per record \(title, price\)\. Details are in Appendix[C\.3](https://arxiv.org/html/2607.03162#A3.SS3)\. \(3\) Agent \- External Retrieval DatabaseAgents use a retrieval tool backed by a database over history and candidate metadata\. At each step, agents issue queries to retrieve extra information beyond the prompt \(Appendix[C\.4](https://arxiv.org/html/2607.03162#A3.SS4),[C\.5](https://arxiv.org/html/2607.03162#A3.SS5)\)\. #### 4\.1\.3Evaluation Method Algorithm 1Structured Product Recommendation StepInput:intent/refinedqueryqq, historiesℋ\\mathcal\{H\}, candidates𝒞hard\\mathcal\{C\}^\{\\mathrm\{hard\}\} Target inference:Infer user targetτ\\taufrom\(q,ℋ\)\(q,\\mathcal\{H\}\)\. Relevant history selection:ℋ⋆←\\mathcal\{H\}^\{\\star\}\\leftarrowsalient histories toτ\\tau History summarization:ℱ=\{\(fi,ri,Ii\)\}i=15\\mathcal\{F\}=\\\{\(f\_\{i\},r\_\{i\},I\_\{i\}\)\\\}\_\{i=1\}^\{5\},fif\_\{i\}– preference fact,ℋi⋆\\mathcal\{H\}\_\{i\}^\{\\star\}– supporting records,rir\_\{i\}– relevance rank\. Recommend items:ℛ⋆=\{\(ci,ri,ρi,ℱi\)\}i=15\\mathcal\{R\}^\{\\star\}=\\\{\(c\_\{i\},r\_\{i\},\\rho\_\{i\},\\mathcal\{F\}\_\{i\}\)\\\}\_\{i=1\}^\{5\},cic\_\{i\}– candidate,ρi\\rho\_\{i\}– rationale,ℱi⊆ℱ\\mathcal\{F\}\_\{i\}\\subseteq\\mathcal\{F\}– supporting features\. Structured Output:To evaluate both task success and intermediate steps, Algorithm[1](https://arxiv.org/html/2607.03162#alg1)prompts models to produce structured outputs, including user target, history summarization, and recommend items\. For ReAct, reasoning stepsReiRe\_\{i\}and retrieval queriesRqiRq\_\{i\}at each stepiiare logged\. Outcome & Reasoning:Personalization performance is measured using Hit@1/5 on the recommended itemsℛ⋆\\mathcal\{R\}^\{\\star\}\. A model succeeds when it ranks the purchased item above user\-considered alternatives\. We further apply rubric evaluation to intermediate outputs using*LLM\-as\-a\-Judge*, scoring each component independently\. The intermediate rubrics are anchored to the same user session evidence: they diagnose whether failures arise from intent inference, history\-grounded preference extraction, recommendation alignment, or multi\-step reasoning\. Detailed evaluation is in Section[4\.3](https://arxiv.org/html/2607.03162#S4.SS3)\. ### 4\.2Overall Results and Analysis \(RQ1\) #### 4\.2\.1LLMs Underperform onIntentQueries Table[2](https://arxiv.org/html/2607.03162#S4.T2)shows that while single\-prompt LLMs achieve strong Hit@5 across workflows under both query types, they slightly underperform UniSAR on Hit@1 forintentqueries\. This highlights theirweakness in precise personalized ranking under vanilla goals and noisy histories, where user modeling outweighs semantic matching\. Across LLM variants, proprietary closed\-source models \(e\.g\., GPT\-5\.2\) achieve higher performance than open\-source or smaller\-scale models under bothintentandrefinedqueries\. However, these gains are marginal—especially on Hit@1—suggestinglimitations of current LLM training for real\-world personalization\. We further evaluate agents with multi\-step reasoning and tool use\. In Table[2](https://arxiv.org/html/2607.03162#S4.T2), therefined\-query block shows that ReAct agents consistently outperform single\-prompt LLMs, while theintent\-query block shows that ReAct yields only marginal gains and Deerflow shows little benefit or even degradation\. These results indicate thatincreased semantic capacity alone does not guarantee improved personalization: when user goals are weakly specified, additional planning and retrieval steps can instead amplify reasoning errors\. Detailed analysis is in Section[4\.3](https://arxiv.org/html/2607.03162#S4.SS3)\. #### 4\.2\.2Hard Candidates and Noisy History LLMs struggle to benefit from history\.Table[3](https://arxiv.org/html/2607.03162#S4.T3)shows that with no\-history, models demonstrate basic semantic matching ability, whileintentremains more challenging\. When user history is introduced, trained UniSAR achieves substantial gains\. Short and long category\-prior histories \(@6/@60 CATE\) yield very similar Hit@1 underintentqueries; while the refined\-query block shows larger improvements once intent ambiguity is reduced\. Comparisons betweenintentandrefinedqueries imply that history possesses usable preference signal but current LLM workflows do not naturally benefit possibly because LLMs do not extract recoverable preferences underintentqueries\. Further analysis in Section[4\.4](https://arxiv.org/html/2607.03162#S4.SS4)provides stepwise evidence\. Appendices[B\.5\.1](https://arxiv.org/html/2607.03162#A2.SS5.SSS1)and[C\.2\.1](https://arxiv.org/html/2607.03162#A3.SS2.SSS1)demonstrate that with explicit preference extraction and memory modules, LLMs can better benefit from history\. Table 3:History ablation: Hit@1 \(%\) onintent/refinedqueries\. @0 = no history; @k Cate = k category\-prior records; arrows = change vs\. @0\.Table 4:Candidate ablation onintentqueries: Hit@1 \(%\) with random vs\. hard candidates\.Personalization choice lies beyond semantic matching\.Table[4](https://arxiv.org/html/2607.03162#S4.T4)shows that, even under vague queries, LLMs outperform the UniSAR baseline when evaluated with random candidate sets, which are sampled from all possible candidates in the dataset\. This gain mainly stems from LLMs’ ability to exploit coarse semantic cues in underspecified intents to filter out obviously mismatched negatives and locate the query\-aligned purchased item\. State\-of\-the\-art models \(e\.g\., GPT\-5\.2\) further widen this margin over older or open\-source models \(Qwen3, GPT\-4\.1\)\. Under hard candidates, both gains over UniSAR and inter\-LLM performance gaps largely disappear, indicating that current LLMs struggle to discriminate fine\-grained attributes among user\-viewed candidates\. This also shows that: random negatives often require coarse semantic matching rather than true personalization\. Hard candidates drawn from the same shopping trajectory better test whether a model can distinguish the purchased item based on user\-specific preferences\. ### 4\.3Rubrics on Intermediate Reasoning \(RQ2\) \(a\)Averaged scores over LLM models onIntent/VQRA/Refinedquery \(b\)Averaged scores over ReAct models onIntent/VQRA/Refinedquery \(c\)Per\-model analysis of intermediate reasoning components onIntentquery Figure 3:Rubric scores for evaluating models across intermediate stepwise outputs \(QI: Query Inference;PQ: Preference Quality;RQ: Recommendation Quality;PR: Minimum Observed\-Purchase Preference Rank↓\\downarrow;PC: Total Observed\-Purchase Preference Count;AC: Agent Coherence;QC: Query–Candidate;QH: Query–History;QQ: Query Quantity\)\. Diagnostic scores are averaged over Gemini\-2\.5\-Pro, GPT\-5\.2, and GPT\-4\.1 judges\.We conduct rubric\-based evaluation of intermediate outputs in Section[4\.1\.3](https://arxiv.org/html/2607.03162#S4.SS1.SSS3)to diagnose why LLMs and ReAct agents underperform onintentqueries and to compare behaviors across LLMs\. We first infer a diagnostic goal proxyggfrom in\-session interactions \(Eq\.[2](https://arxiv.org/html/2607.03162#S2.E2)\) \(Appendix[C\.7\.1](https://arxiv.org/html/2607.03162#A3.SS7.SSS1)\), and evaluate each output againstggusing*LLM\-as\-a\-Judge*\. The diagnostic scores are averaged across Gemini\-2\.5\-Pro, GPT\-5\.2, and GPT\-4\.1 judges with human verification \(Appendix[C\.7\.3](https://arxiv.org/html/2607.03162#A3.SS7.SSS3)\)\. These diagnostics correspond to general personalization stages: QI measures intent inference under ambiguity, PQ measures history\-conditioned preference extraction, RQ measures item/action selection quality, and AR measures whether tool\-using reasoning retrieves useful evidence \(see Table[21](https://arxiv.org/html/2607.03162#A3.T21)\)\. Multi\-Level RubricWe decompose the evaluation into stepwise complementary aspects \(Appendix[C\.7\.2](https://arxiv.org/html/2607.03162#A3.SS7.SSS2)\): \(1\)*Query Inference \(QI\)*—user target understanding from queries and histories; \(2\)*Historical Preference Reasoning*—via i\.*Preference Quality \(PQ\)*, assessing whether summarized preferencesℱ\\mathcal\{F\}are goal\-relevant and non\-hallucinatory, and ii\. Salience Statistics: minimum rank \(*PR*\) and total count \(*PC*\) of preferences supporting the purchased item; \(3\)*Recommendation Quality \(RQ\)*—whether the recommend resultsℛ⋆\\mathcal\{R\}^\{\\star\}align with the user goal without hallucination; \(4\)*Agent Reasoning \(AR\)*—coherence of reasoning stepsReiRe\_\{i\}\(AC\) and retrieval utility, including candidate/history queries \(QC, QH\) and query quantity \(QQ\), toward achievingggwithout hallucination\. Rubric–Hit@K alignment and judge\-sensitivity checks in Appendix[C\.7\.3](https://arxiv.org/html/2607.03162#A3.SS7.SSS3)and[C\.7\.4](https://arxiv.org/html/2607.03162#A3.SS7.SSS4)support using these scores as diagnostic evidence\. Intermediate ResultsFigure[3](https://arxiv.org/html/2607.03162#S4.F3)shows rubric scores forintentandrefinedqueries under single\-prompt and ReAct workflows\. We observe: *\(1\) Failures originate from intent understanding\.*As shown in Figure[3\(a\)](https://arxiv.org/html/2607.03162#S4.F3.sf1),*query inference*performance drops substantially onintentcompared torefinedqueries, undermining downstream preference grounding and leading to degraded*recommendation quality*\. Notably,*preference reasoning*remains limited even under explicitrefinedqueries, indicating identifying and summarizing relevant signals from heterogeneous histories remains an open challenge, even with clarified goals\. Under vague intents, inferring meaningful historical preferences is particularly difficult\. *\(2\) Agentic workflows amplify strengths and weaknesses\.*Comparing Figures[3\(a\)](https://arxiv.org/html/2607.03162#S4.F3.sf1)and[3\(b\)](https://arxiv.org/html/2607.03162#S4.F3.sf2), multi\-step retrieval improves recommendation quality under explicitrefinedqueries but offers little benefit forintentqueries\. Under vague intents, low reasoning coherence yields ineffective retrieval queries, causing agentic reasoning to compound intent errors, consistent with theintent\-query block of Table[2](https://arxiv.org/html/2607.03162#S4.T2)\. Moreover, retrieving relevant signals from user histories is substantially harder than retrieving candidate information under both query typesZou et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib80),[2026](https://arxiv.org/html/2607.03162#bib.bib79)\)\. Appendix[C\.4\.1](https://arxiv.org/html/2607.03162#A3.SS4.SSS1)reports additional ablations on round\-depth, prompts, and retrieval\-backend\. *\(3\) Models exhibit different intermediate behaviors\.*Figure[3\(c\)](https://arxiv.org/html/2607.03162#S4.F3.sf3)shows that models with comparable Hit@1 diverge in their intermediate reasoning steps\. Advanced reasoning models \(e\.g\., Gemini\-2\.5\-Pro, DeepSeek\-R1\) achieve better intermediate performance in history preference reasoning and recommendation quality\. While they prioritize salient preferences earlier \(lower FR\), they extract fewer preferences overall \(lower FC\)\. They also issue fewer retrieval queries under ReAct \(lower QQ\)\. These might become detrimental especially underintentqueries where insufficient preference coverage undermines recommendation\. ### 4\.4VQRA: RefineIntentQuery First \(RQ3\) Section[4\.3](https://arxiv.org/html/2607.03162#S4.SS3)and Figure[3](https://arxiv.org/html/2607.03162#S4.F3)attribute the performance drop onintentqueries to weak query inference due to underexplored histories\. To isolate this gap, we evaluate a history\-conditioned query\-refinement agent,VQRA, in which a shared backbone \(GPT\-4\.1\) first rewrites theintentquery using selected same\-category histories and the LLM backbone is then evaluated with the rewritten query under the single\-prompt pipeline \(Appendix[C\.8](https://arxiv.org/html/2607.03162#A3.SS8)\)\. Table[2](https://arxiv.org/html/2607.03162#S4.T2)shows thatVQRAimproves Hit@1, with the largest gain on GPT\-5\.2Think\. Appendix[C\.8](https://arxiv.org/html/2607.03162#A3.SS8)shows similar results with the same\-backbone\-rewrite variant\. Rubric scores in Figure[3\(a\)](https://arxiv.org/html/2607.03162#S4.F3.sf1),[3\(b\)](https://arxiv.org/html/2607.03162#S4.F3.sf2)further suggest that query refinement improves*query inference \(QI\)*toward the true shopping intent, leading to gains in*recommendation quality \(RQ\)*\. These indicate that inferring intent from vague queries and noisy histories remains underutilized: with explicit history grounding, LLMs already produce useful refinements, suggesting headroom for future personalization agent design\. ## 5Conclusion We introduceAPeB, a benchmark for agentic personalization that evaluates personalized product search under raw, underspecified queries and noisy interaction histories\. ThroughAPeB, we show that current LLMs handle explicit queries well but struggle in early\-stage personalization, where they must infer intent, recover preferences, and choose among competing alternatives\. Rubric analysis attributes this gap mainly to ineffective history use: models fail to extract relevant preferences and apply them to planning and candidate selection\. Furthermore,VQRA, a simple history\-aware query\-refinement pipeline, improves performance across backbones, highlighting the value of explicit history\-utilization modules\.APeBserves as a testbed for robust user modeling in real\-world personalization\. ## Limitations APeBis drawn from a single content\-commerce platform; behavior patterns, query styles, and candidate structures may not transfer to other e\-commerce ecosystems or non\-shopping personalization domains\. All evaluation is static and offline: fixed candidate sets and recorded purchases do not reflect real\-time inventory, pricing, promotions, or multi\-turn interaction\. Observed purchases serve as ground\-truth preference proxies but are confounded by ranking exposure, stock availability, and platform mediation\. We explore category\- and time\-based history selection without exhaustively studying advanced memory architectures or adaptive context\-length strategies\. Rubric analysis relies on LLM\-as\-a\-Judge, which may introduce systematic biases despite strong human agreement\. The benchmark covers limited locale and language; intent expression and attribute salience may vary across cultures\. ## Ethical Considerations This work studies personalization benchmarks derived from postprocessed behavioral shopping records\. Its positive impact is to make evaluation of preference\-conditioned product search more realistic and auditable, especially when users issue ambiguous queries\. The main risks are privacy leakage from long histories or product/media text, re\-identification through rare behavior patterns, category or platform bias, and misuse of preference inference for manipulative over\-personalization\. We mitigate these risks by releasing only anonymized benchmark instances rather than raw logs, removing direct identifiers and platform\-private fields, using surrogate identifiers without cross\-split linkability, binning temporal signals, minimizing retained interactions, and redacting sensitive free\-form text such as OCR/ASR\-derived content\. ## References - Ai et al\. \(2017\)Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W\. Bruce Croft\. 2017\.[Learning a hierarchical embedding model for personalized product search](https://doi.org/10.1145/3077136.3080813)\.In*Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR ’17, page 645–654, New York, NY, USA\. Association for Computing Machinery\. - Arora et al\. \(2025\)Rahul K\. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal\. 2025\.[Healthbench: Evaluating large language models towards improved human health](https://doi.org/10.48550/ARXIV.2505.08775)\.*CoRR*, abs/2505\.08775\. - Ashkan and Metzler \(2019\)Azin Ashkan and Donald Metzler\. 2019\.[Revisiting online personal search metrics with the user in mind](https://doi.org/10.1145/3331184.3331266)\.In*Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR’19, page 625–634, New York, NY, USA\. Association for Computing Machinery\. - Ben\-Shimon et al\. \(2015\)David Ben\-Shimon, Alexander Tsikinovsky, Michael Friedmann, Bracha Shapira, Lior Rokach, and Johannes Hoerle\. 2015\.Recsys challenge 2015 and the YOOCHOOSE dataset\.In*Proceedings of the 9th ACM Conference on Recommender Systems, RecSys 2015, Vienna, Austria, September 16\-20, 2015*, pages 357–358\. ACM\. - Bytedance \(2025\)Henry Li Bytedance, Daniel Walnut\. 2025\.Deerflow: Deep exploration and efficient research flow\.[https://github\.com/bytedance/deer\-flow](https://github.com/bytedance/deer-flow)\. - Chen et al\. \(2024\)Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun\. 2024\.[GUI\-WORLD: A dataset for gui\-oriented multimodal llm\-based agents](https://doi.org/10.48550/ARXIV.2406.10819)\.*CoRR*, abs/2406\.10819\. - Chen et al\. \(2026\)Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, and James Cheng\. 2026\.[Hista and numca: Estimate state value effectively for llm reinforcement learning](https://arxiv.org/abs/2605.29782)\.*Preprint*, arXiv:2605\.29782\. - Chhikara et al\. \(2025\)Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav\. 2025\.[Mem0: Building production\-ready AI agents with scalable long\-term memory](https://doi.org/10.48550/ARXIV.2504.19413)\.*CoRR*, abs/2504\.19413\. - Deng et al\. \(2025\)Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou\. 2025\.[Onerec: Unifying retrieve and rank with generative recommender and iterative preference alignment](https://arxiv.org/abs/2502.18965)\.*Preprint*, arXiv:2502\.18965\. - Deng et al\. \(2023\)Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su\. 2023\.Mind2web: Towards a generalist agent for the web\.In*Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023*\. - Downey et al\. \(2008\)Doug Downey, Susan T\. Dumais, Daniel J\. Liebling, and Eric Horvitz\. 2008\.[Understanding the relationship between searchers’ queries and information goals](https://doi.org/10.1145/1458082.1458143)\.In*Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM 2008, Napa Valley, California, USA, October 26\-30, 2008*, pages 449–458\. ACM\. - Du et al\. \(2025\)Yiming Du, Wenyu Huang, Danna Zheng, Zhaowei Wang, Sebastien Montella, Mirella Lapata, Kam\-Fai Wong, and Jeff Z\. Pan\. 2025\.[Rethinking memory in llm based agents: Representations, operations, and emerging topics](https://arxiv.org/abs/2505.00675)\.*Preprint*, arXiv:2505\.00675\. - Duan and Zhai \(2015\)Huizhong Duan and ChengXiang Zhai\. 2015\.[Mining coordinated intent representation for entity search and recommendation](https://doi.org/10.1145/2806416.2806557)\.In*Proceedings of the 24th ACM International on Conference on Information and Knowledge Management*, CIKM ’15, page 333–342, New York, NY, USA\. Association for Computing Machinery\. - Farshidi et al\. \(2024\)Siamak Farshidi, Kiyan Rezaee, Sara Mazaheri, Amir Rahimi, Ali Dadashzadeh, Morteza Ziabakhsh, Sadegh Eskandari, and Slinger Jansen\. 2024\.[Understanding user intent modeling for conversational recommender systems: a systematic literature review](https://doi.org/10.1007/s11257-024-09398-x)\.*User Modeling and User\-Adapted Interaction*, 34:1643–1706\. - Google DeepMind \(2025\)Google DeepMind\. 2025\.[Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://arxiv.org/abs/2507.06261)\.*Preprint*, arXiv:2507\.06261\. - Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z\. F\. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others\. 2025\.[Deepseek\-r1 incentivizes reasoning in llms through reinforcement learning](https://doi.org/10.1038/s41586-025-09422-z)\.*Nature*, 645\(8081\):633–638\. - Guo et al\. \(2019\)Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W\. Bruce Croft, and Xueqi Cheng\. 2019\.[A deep look into neural ranking models for information retrieval](https://arxiv.org/abs/1903.06902)\.*Preprint*, arXiv:1903\.06902\. - Harper and Konstan \(2015\)F\. Maxwell Harper and Joseph A\. Konstan\. 2015\.[The movielens datasets: History and context](https://doi.org/10.1145/2827872)\.*ACM Trans\. Interact\. Intell\. Syst\.*, 5\(4\)\. - Hou et al\. \(2024\)Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian J\. McAuley\. 2024\.[Bridging language and items for retrieval and recommendation](https://doi.org/10.48550/ARXIV.2403.03952)\.*CoRR*, abs/2403\.03952\. - Hu et al\. \(2025\)Yuanzhe Hu, Yu Wang, and Julian McAuley\. 2025\.[Evaluating memory in llm agents via incremental multi\-turn interactions](https://arxiv.org/abs/2507.05257)\.*Preprint*, arXiv:2507\.05257\. - Huang et al\. \(2025\)Zhaopei Huang, Qifeng Dai, Guozheng Wu, Xiaopeng Wu, Kehan Chen, Chuan Yu, Xubin Li, Tiezheng Ge, Wenxuan Wang, and Qin Jin\. 2025\.[Mem\-pal: Towards memory\-based personalized dialogue assistants for long\-term user\-agent interaction](https://arxiv.org/abs/2511.13410)\.*Preprint*, arXiv:2511\.13410\. - Jansen and Molina \(2006\)Jim Jansen and Paulo Molina\. 2006\.[The effectiveness of web search engines for retrieving relevant ecommerce links](https://doi.org/10.1016/j.ipm.2005.09.003)\.*Information Processing & Management*, 42:1075–1098\. - Jin et al\. \(2023\)Wei Jin, Haitao Mao, Zheng Li, Haoming Jiang, Chen Luo, Hongzhi Wen, Haoyu Han, Hanqing Lu, Zhengyang Wang, Ruirui Li, Zhen Li, Monica Xiao Cheng, Rahul Goutam, Haiyang Zhang, Karthik Subbian, Suhang Wang, Yizhou Sun, Jiliang Tang, Bing Yin, and Xianfeng Tang\. 2023\.Amazon\-m2: A multilingual multi\-locale shopping session dataset for recommendation and text generation\.In*Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023*\. - Jin et al\. \(2024\)Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, and 3 others\. 2024\.Shopping MMLU: A massive multi\-task online shopping benchmark for large language models\.In*Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024*\. - Liu et al\. \(2016\)Guimei Liu, Tam T\. Nguyen, Gang Zhao, Wei Zha, Jianbo Yang, Jianneng Cao, Min Wu, Peilin Zhao, and Wei Chen\. 2016\.[Repeat buyer prediction for e\-commerce](https://doi.org/10.1145/2939672.2939674)\.In*Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13\-17, 2016*, pages 155–164\. ACM\. - Liu et al\. \(2023\)Jiongnan Liu, Zhicheng Dou, Guoyu Tang, and Sulong Xu\. 2023\.[Jdsearch: A personalized product search dataset with real queries and full interactions](https://doi.org/10.1145/3539618.3591900)\.In*Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR ’23, page 2945–2952, New York, NY, USA\. Association for Computing Machinery\. - Liu et al\. \(2025\)Zhanyu Liu, Shiyao Wang, Xingmei Wang, Rongzhou Zhang, Jiaxin Deng, Honghui Bao, Jinghao Zhang, Wuchao Li, Pengfei Zheng, Xiangyu Wu, Yifei Hu, Qigen Hu, Xinchen Luo, Lejian Ren, Zixing Zhang, Qianqian Wang, Kuo Cai, Yunfan Wu, Hongtao Cheng, and 7 others\. 2025\.[Onerec\-think: In\-text reasoning for generative recommendation](https://arxiv.org/abs/2510.11639)\.*Preprint*, arXiv:2510\.11639\. - Lyu et al\. \(2025\)Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, and Xiuying Chen\. 2025\.[Deepshop: A benchmark for deep research shopping agents](https://arxiv.org/abs/2506.02839)\.*Preprint*, arXiv:2506\.02839\. - Magister et al\. \(2024\)Lucie Charlotte Magister, Katherine Metcalf, Yizhe Zhang, and Maartje ter Hoeve\. 2024\.[On the way to llm personalization: Learning to remember user conversations](https://arxiv.org/abs/2411.13405)\.*Preprint*, arXiv:2411\.13405\. - Maharana et al\. \(2024\)Adyasha Maharana, Dong\-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang\. 2024\.[Evaluating very long\-term conversational memory of llm agents](https://arxiv.org/abs/2402.17753)\.*Preprint*, arXiv:2402\.17753\. - Maynez et al\. \(2020\)Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald\. 2020\.[On faithfulness and factuality in abstractive summarization](https://doi.org/10.18653/v1/2020.acl-main.173)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 1906–1919, Online\. Association for Computational Linguistics\. - Min et al\. \(2023\)Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen\-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi\. 2023\.[FActScore: Fine\-grained atomic evaluation of factual precision in long form text generation](https://doi.org/10.18653/v1/2023.emnlp-main.741)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12076–12100, Singapore\. Association for Computational Linguistics\. - Ning et al\. \(2025\)Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao yong Wei, Shanru Lin, Hui Liu, Philip S\. Yu, and Qing Li\. 2025\.[A survey of webagents: Towards next\-generation ai agents for web automation with large foundation models](https://arxiv.org/abs/2503.23350)\.*Preprint*, arXiv:2503\.23350\. - OpenAI et al\. \(2024\)OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others\. 2024\.[Gpt\-4 technical report](https://arxiv.org/abs/2303.08774)\.*Preprint*, arXiv:2303\.08774\. - Packer et al\. \(2024\)Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G\. Patil, Ion Stoica, and Joseph E\. Gonzalez\. 2024\.[Memgpt: Towards llms as operating systems](https://arxiv.org/abs/2310.08560)\.*Preprint*, arXiv:2310\.08560\. - Peng et al\. \(2024\)Bo Peng, Xinyi Ling, Ziru Chen, Huan Sun, and Xia Ning\. 2024\.ecellm: Generalizing large language models for e\-commerce from large\-scale, high\-quality instruction data\.In*Forty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024*\. OpenReview\.net\. - Peng et al\. \(2025\)Qiyao Peng, Hongtao Liu, Hua Huang, Qing Yang, and Minglai Shao\. 2025\.[A survey on llm\-powered agents for recommender systems](https://arxiv.org/abs/2502.10050)\.*Preprint*, arXiv:2502\.10050\. - Peng et al\. \(2026\)Zhiyuan Peng, Xuyang Wu, Huaixiao Tou, Yi Fang, and Yu Gong\. 2026\.[Memrerank: Preference memory for personalized product reranking](https://arxiv.org/abs/2603.29247)\.*Preprint*, arXiv:2603\.29247\. - Punj and Moore \(2009\)Girish Punj and Robert Moore\. 2009\.[Information search and consideration set formation in a web\-based store environment](https://doi.org/10.1016/j.jbusres.2007.04.013)\.*Journal of Business Research*, 62\(6\):644–650\.Retailing Evolution Research\. - Rajput et al\. \(2023\)Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H\. Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q\. Tran, Jonah Samost, Maciej Kula, Ed H\. Chi, and Maheswaran Sathiamoorthy\. 2023\.[Recommender systems with generative retrieval](https://arxiv.org/abs/2305.05065)\.*Preprint*, arXiv:2305\.05065\. - Raza et al\. \(2026\)Shaina Raza, Mizanur Rahman, Safiullah Kamawal, Armin Toroghi, Ananya Raval, Farshad Navah, and Amirmohammad Kazemeini\. 2026\.[A comprehensive review of recommender systems: Transitioning from theory to practice](https://doi.org/10.1016/j.cosrev.2025.100849)\.*Computer Science Review*, 59:100849\. - Reddy et al\. \(2022\)Chandan K\. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian\. 2022\.[Shopping queries dataset: A large\-scale esci benchmark for improving product search](https://arxiv.org/abs/2206.06588)\.*Preprint*, arXiv:2206\.06588\. - Rendle et al\. \(2012\)Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt\-Thieme\. 2012\.[Bpr: Bayesian personalized ranking from implicit feedback](https://arxiv.org/abs/1205.2618)\.*Preprint*, arXiv:1205\.2618\. - Richardson et al\. \(2023\)Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy\. 2023\.[Integrating summarization and retrieval for enhanced personalization via large language models](https://arxiv.org/abs/2310.20081)\.*Preprint*, arXiv:2310\.20081\. - Robertson and Zaragoza \(2009\)Stephen Robertson and Hugo Zaragoza\. 2009\.[The probabilistic relevance framework: Bm25 and beyond](https://doi.org/10.1561/1500000019)\.*Found\. Trends Inf\. Retr\.*, 3\(4\):333–389\. - Russell \(2021\)Stuart Russell\. 2021\.[Human\-compatible artificial intelligence](https://doi.org/10.1093/oso/9780198862536.003.0001)\.In*Human\-Like Machine Intelligence*\. Oxford University Press\. - Salemi et al\. \(2024\)Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani\. 2024\.[Lamp: When large language models meet personalization](https://doi.org/10.18653/V1/2024.ACL-LONG.399)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024*, pages 7370–7392\. Association for Computational Linguistics\. - Shang et al\. \(2025\)Yu Shang, Peijie Liu, Yuwei Yan, Zijing Wu, Leheng Sheng, Yuanqing Yu, Chumeng Jiang, An Zhang, Fengli Xu, Yu Wang, Min Zhang, and Yong Li\. 2025\.[Agentrecbench: Benchmarking llm agent\-based personalized recommender systems](https://arxiv.org/abs/2505.19623)\.*Preprint*, arXiv:2505\.19623\. - Shao et al\. \(2023\)Weiqi Shao, Xu Chen, Jiashu Zhao, Long Xia, Jingsen Zhang, and Dawei Yin\. 2023\.[Sequential recommendation with user evolving preference decomposition](https://doi.org/10.1145/3624918.3625312)\.In*Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region*, SIGIR\-AP ’23, page 253–263, New York, NY, USA\. Association for Computing Machinery\. - Shen et al\. \(2023\)Chenhui Shen, Liying Cheng, Xuan\-Phi Nguyen, Yang You, and Lidong Bing\. 2023\.[Large language models are not yet human\-level evaluators for abstractive summarization](https://arxiv.org/abs/2305.13091)\.*Preprint*, arXiv:2305\.13091\. - Shi et al\. \(2024\)Teng Shi, Zihua Si, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Dewei Leng, Yanan Niu, and Yang Song\. 2024\.[Unisar: Modeling user transition behaviors between search and recommendation](https://arxiv.org/abs/2404.09520)\.*Preprint*, arXiv:2404\.09520\. - Singh et al\. \(2025\)Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El\-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker\-Whitcomb, Alex Beutel, Alex Karpenko, and 465 others\. 2025\.[Openai gpt\-5 system card](https://arxiv.org/abs/2601.03267)\.*Preprint*, arXiv:2601\.03267\. - Sun et al\. \(2025\)Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi R\. Fung, Hou Pong Chan, Kevin Small, ChengXiang Zhai, and Heng Ji\. 2025\.[Persona\-db: Efficient large language model personalization for response prediction with collaborative data refinement](https://arxiv.org/abs/2402.11060)\.*Preprint*, arXiv:2402\.11060\. - Sun et al\. \(2023\)Zhongxiang Sun, Zihua Si, Xiaoxue Zang, Dewei Leng, Yanan Niu, Yang Song, Xiao Zhang, and Jun Xu\. 2023\.[Kuaisar: A unified search and recommendation dataset](https://doi.org/10.1145/3583780.3615123)\.In*Proceedings of the 32nd ACM International Conference on Information and Knowledge Management*, CIKM ’23, page 5407–5411\. ACM\. - Tou et al\. \(2025\)Huaixiao Tou, Ying Zeng, Cong Ma, Muzhi Li, Minghao Li, Weijie Yuan, He Zhang, and Kai Jia\. 2025\.[Shoppingcomp: Are llms really ready for your shopping cart?](https://arxiv.org/abs/2511.22978)*Preprint*, arXiv:2511\.22978\. - Wang et al\. \(2023a\)Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar\. 2023a\.[Voyager: An open\-ended embodied agent with large language models](https://arxiv.org/abs/2305.16291)\.*Preprint*, arXiv:2305\.16291\. - Wang et al\. \(2025a\)Jiangyuan Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jiandong Zhang, and Xiaoyi Zeng\. 2025a\.[Shoppingbench: A real\-world intent\-grounded shopping benchmark for llm\-based agents](https://doi.org/10.48550/ARXIV.2508.04266)\.*CoRR*, abs/2508\.04266\. - Wang et al\. \(2024\)Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen\. 2024\.[A survey on large language model based autonomous agents](https://doi.org/10.1007/s11704-024-40231-1)\.*Frontiers of Computer Science*, 18\(6\)\. - Wang et al\. \(2025b\)Piaohong Wang, Motong Tian, Jiaxian Li, Yuan Liang, Yuqing Wang, Qianben Chen, Tiannan Wang, Zhicong Lu, Jiawei Ma, Yuchen Eleanor Jiang, and Wangchunshu Zhou\. 2025b\.[O\-mem: Omni memory system for personalized, long horizon, self\-evolving agents](https://arxiv.org/abs/2511.13593)\.*Preprint*, arXiv:2511\.13593\. - Wang et al\. \(2023b\)Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei\. 2023b\.[Augmenting language models with long\-term memory](https://arxiv.org/abs/2306.07174)\.*Preprint*, arXiv:2306\.07174\. - Wang et al\. \(2022\)Xiaolei Wang, Kun Zhou, Ji\-Rong Wen, and Wayne Xin Zhao\. 2022\.[Towards unified conversational recommender systems via knowledge\-enhanced prompt learning](https://doi.org/10.1145/3534678.3539382)\.In*Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, KDD ’22, page 1929–1937, New York, NY, USA\. Association for Computing Machinery\. - Wang et al\. \(2025c\)Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia\-Jun Li, Lydia B\. Chilton, and Dakuo Wang\. 2025c\.[Opera: A dataset of observation, persona, rationale, and action for evaluating llms on human online shopping behavior simulation](https://doi.org/10.48550/ARXIV.2506.05606)\.*CoRR*, abs/2506\.05606\. - Xie et al\. \(2025\)Shuyi Xie, Ziqin Liew, Hailing Zhang, Haibo Zhang, Ling Hu, Zhiqiang Zhou, Shuman Liu, and Anxiang Zeng\. 2025\.[Towards reliable evaluation of large language models for multilingual and multimodal e\-commerce applications](https://arxiv.org/abs/2510.20632)\.*Preprint*, arXiv:2510\.20632\. - Xu et al\. \(2023\)Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi\. 2023\.[A critical evaluation of evaluations for long\-form question answering](https://doi.org/10.18653/v1/2023.acl-long.181)\.pages 3225–3245\. - Xu et al\. \(2025\)Wei Xu, Jiasen Zheng, Junjiang Lin, Mingxuan Han, and Junliang Du\. 2025\.[Unified representation learning for multi\-intent diversity and behavioral uncertainty in recommender systems](https://doi.org/10.1109/ISCIPT67144.2025.11265095)\.pages 114–117\. - Yan et al\. \(2025\)Yuwei Yan, Yu Shang, Qingbin Zeng, Yu Li, Keyu Zhao, Zhiheng Zheng, Xuefei Ning, Tianji Wu, Shengen Yan, Yu Wang, Fengli Xu, and Yong Li\. 2025\.[Agentsociety challenge: Designing llm agents for user modeling and recommendation on web platforms](https://arxiv.org/abs/2502.18754)\.*Preprint*, arXiv:2502\.18754\. - Yang et al\. \(2025a\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025a\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\. - Yang et al\. \(2025b\)Yuqi Yang, Weiqi Wang, Baixuan Xu, Wei Fan, Qing Zong, Chunkit Chan, Zheye Deng, Xin Liu, Yifan Gao, Changlong Yu, Chen Luo, Yang Li, Zheng Li, Qingyu Yin, Bing Yin, and Yangqiu Song\. 2025b\.[Sessionintentbench: A multi\-task inter\-session intention\-shift modeling benchmark for e\-commerce customer behavior understanding](https://arxiv.org/abs/2507.20185)\.*Preprint*, arXiv:2507\.20185\. - Yang et al\. \(2023\)Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang\. 2023\.[Mm\-react: Prompting chatgpt for multimodal reasoning and action](https://arxiv.org/abs/2303.11381)\.*Preprint*, arXiv:2303\.11381\. - Yao et al\. \(2022\)Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan\. 2022\.Webshop: Towards scalable real\-world web interaction with grounded language agents\.In*Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022*\. - Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\. 2023\.[React: Synergizing reasoning and acting in language models](https://arxiv.org/abs/2210.03629)\.*Preprint*, arXiv:2210\.03629\. - Yu et al\. \(2025\)Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya\-Qin Zhang, Wei\-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou\. 2025\.[Memagent: Reshaping long\-context LLM with multi\-conv rl\-based memory agent](https://doi.org/10.48550/ARXIV.2507.02259)\.*CoRR*, abs/2507\.02259\. - Yue et al\. \(2023\)Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun\. 2023\.[Automatic evaluation of attribution by large language models](https://arxiv.org/abs/2305.06311)\.*Preprint*, arXiv:2305\.06311\. - Zhang et al\. \(2025a\)Beibei Zhang, Yanan Lu, Ruobing Xie, Zongyi Li, Siyuan Xing, Tongwei Ren, and Fen Lin\. 2025a\.[Harnessing multimodal large language models for personalized product search with query\-aware refinement](https://arxiv.org/abs/2509.18682)\.*Preprint*, arXiv:2509\.18682\. - Zhang et al\. \(2025b\)Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou\. 2025b\.[Qwen3 embedding: Advancing text embedding and reranking through foundation models](https://arxiv.org/abs/2506.05176)\.*Preprint*, arXiv:2506\.05176\. - Zhang et al\. \(2025c\)Yu Zhang, Shutong Qiao, Jiaqi Zhang, Tzu\-Heng Lin, Chen Gao, and Yong Li\. 2025c\.[A survey of large language model empowered agents for recommendation and search: Towards next\-generation information retrieval](https://arxiv.org/abs/2503.05659)\.*Preprint*, arXiv:2503\.05659\. - Zhou et al\. \(2024\)Shuyan Zhou, Frank F\. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig\. 2024\.Webarena: A realistic web environment for building autonomous agents\.In*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\. OpenReview\.net\. - Zhu et al\. \(2018\)Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai\. 2018\.[Learning tree\-based deep model for recommender systems](https://doi.org/10.1145/3219819.3219826)\.In*Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19\-23, 2018*, pages 1079–1088\. ACM\. - Zou et al\. \(2026\)Deyu Zou, Yongqiang Chen, Fan Feng, Mufei Li, Pan Li, Yu Gong, and James Cheng\. 2026\.On information self\-locking in reinforcement learning for active reasoning of llm agents\.*arXiv preprint arXiv:2603\.12109*\. - Zou et al\. \(2025\)Deyu Zou, Yongqiang Chen, Jianxiang Wang, Garry Yang, Mufei Li, Qing Da, James Cheng, Pan Li, and Yu Gong\. 2025\.Reducing belief deviation in reinforcement learning for active reasoning of llm agents\.In*The Fourteenth International Conference on Learning Representations*\. - Zuo et al\. \(2022\)Simiao Zuo, Qingyu Yin, Haoming Jiang, Shaohui Xi, Bing Yin, Chao Zhang, and Tuo Zhao\. 2022\.[Context\-aware query rewriting for improving users’ search experience on e\-commerce websites](https://arxiv.org/abs/2209.07584)\.*Preprint*, arXiv:2209\.07584\. ## Appendix ARelated Works LLM\-powered Personalization Agentstypically include*profile modeling*,*memory management*, and*task planning and execution*Wang et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib58)\); Zhang et al\. \([2025c](https://arxiv.org/html/2607.03162#bib.bib76)\)\. Profile modeling derives explicit or latent personas from user surveys or interactionsWang et al\. \([2025c](https://arxiv.org/html/2607.03162#bib.bib62)\); Salemi et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib47)\); Sun et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib53)\); Magister et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib29)\); Richardson et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib44)\); Huang et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib21)\)\. Memory management organizes long\-term histories of interactions and conversations into a shopping memory bank that supports retrieving salient actions and filtering noisePacker et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib35)\); Wang et al\. \([2023b](https://arxiv.org/html/2607.03162#bib.bib60)\); Maharana et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib30)\); Peng et al\. \([2026](https://arxiv.org/html/2607.03162#bib.bib38)\)\. Task planning executes subtasks such as preference reasoning, attribute grounding, auxiliary information search \(e\.g\., web search\), and candidate generation, aligning inferred personas with query intent and fine\-grained criteriaYang et al\. \([2025b](https://arxiv.org/html/2607.03162#bib.bib68),[2023](https://arxiv.org/html/2607.03162#bib.bib69)\); Wang et al\. \([2023a](https://arxiv.org/html/2607.03162#bib.bib56)\)\. However, they face challenges in tracking preference drift, retrieving task\-relevant memories from heterogeneous histories, and understanding vague intentsWang et al\. \([2025b](https://arxiv.org/html/2607.03162#bib.bib59)\); Zhang et al\. \([2025a](https://arxiv.org/html/2607.03162#bib.bib74)\); Yao et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib71)\)\. Recent studies further show that improving agent behavior under complex, long\-horizon tasks remains challengingZou et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib80),[2026](https://arxiv.org/html/2607.03162#bib.bib79)\); Chen et al\. \([2026](https://arxiv.org/html/2607.03162#bib.bib7)\), which aligns with the difficulty of history\-grounded personalization task\. Generic E\-commerce Benchmarksevaluate shopping tasks such as product search, web navigation, and multi\-level reasoning, using metrics including success rate, prediction/generation accuracy, and rubric\-based scoresGuo et al\. \([2019](https://arxiv.org/html/2607.03162#bib.bib17)\); Wang et al\. \([2022](https://arxiv.org/html/2607.03162#bib.bib61)\); Ning et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib33)\)\. Search datasets such asHou et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib19)\); Reddy et al\. \([2022](https://arxiv.org/html/2607.03162#bib.bib42)\); Liu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib26)\)focus on simple user queries, while others model complex queries with multiple constraintsLyu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib28)\); Tou et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib55)\)\. However, their queries are either overly explicit or synthetically constructed, failing to capture genuinely vague user intents\. Another line of work benchmarks web agents in interactive environments, requiring navigation, product comparison, and constraint satisfactionZhou et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib77)\); Yao et al\. \([2022](https://arxiv.org/html/2607.03162#bib.bib70)\)\. These settings often depend on synthetic or annotator trajectories and expose page\-level information such as HTMLDeng et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib10)\); Chen et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib6)\), with shopping goals that are manually designed rather than organically derived\. Recent benchmarks emphasize multi\-task knowledge and reasoning abilitiesPeng et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib36)\); Wang et al\. \([2025a](https://arxiv.org/html/2607.03162#bib.bib57)\); Salemi et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib47)\); Lyu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib28)\); Xie et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib63)\)\. For example,Jin et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib24)\)decompose shopping into subtasks such as concept understanding and attribute reasoning, typically framed as QA\-style items with rubric evaluationArora et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib2)\)\. Nonetheless, strong performance on isolated subtasks does not necessarily translate into successful user goal completionXu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib64),[2025](https://arxiv.org/html/2607.03162#bib.bib65)\); Farshidi et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib14)\)\. Behavior Datasets for Personalization\.Recommendation benchmarks primarily target next\-item/rating prediction on past interactionsJin et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib23)\); Liu et al\. \([2016](https://arxiv.org/html/2607.03162#bib.bib25)\); Zhu et al\. \([2018](https://arxiv.org/html/2607.03162#bib.bib78)\); Ben\-Shimon et al\. \([2015](https://arxiv.org/html/2607.03162#bib.bib4)\)\. They typically contain sanitized interactions and lack genuine objectives or viable candidates\. Personalized Product Search \(PPS\) requires modeling both user preference and query intent to locate itemsAi et al\. \([2017](https://arxiv.org/html/2607.03162#bib.bib1)\); Jansen and Molina \([2006](https://arxiv.org/html/2607.03162#bib.bib22)\); Duan and Zhai \([2015](https://arxiv.org/html/2607.03162#bib.bib13)\); Robertson and Zaragoza \([2009](https://arxiv.org/html/2607.03162#bib.bib45)\); Rajput et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib40)\); Deng et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib9)\)\. Existing datasetsLiu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib26)\); Hou et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib19)\)construct queries and purchases from structured user logs, treating unobserved ones as negatives\. However, explicit queries reduce personalization into surface\-level query–item matching, while unobserved candidates bypass intent inference, collapsing the task into preference\-only recommendationGuo et al\. \([2019](https://arxiv.org/html/2607.03162#bib.bib17)\); Ashkan and Metzler \([2019](https://arxiv.org/html/2607.03162#bib.bib3)\)\. Memory benchmarks further attempt to model personalization through multi\-turn interactions, often using synthetically generated dialogues from reviews and session logsHuang et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib21)\); Hu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib20)\); Shang et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib48)\); Yan et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib66)\)\. Nevertheless, the homogeneity of dialogues and the limited diversity of synthetic generation pipelines fail to capture the noisy and heterogeneous nature of real\-world user personasRaza et al\. \([2026](https://arxiv.org/html/2607.03162#bib.bib41)\); Shao et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib49)\)\.APeBfills these gaps with unstructured histories, user\-issuedintentqueries, and behavior\-grounded hard candidates\. ## Appendix BBenchmark Collection ### B\.1Holistic User Action Platform The base platform underlyingAPeBhosts billions of short\-form videos and livestreams and serves as a major venue for commerce\. Users primarily consume video and livestream content via continuous scrolling, but can seamlessly transition to search for videos, livestreams, or products, as well as navigate directly to product detail pages embedded within promotional content\. The queries collected inAPeBare restricted to those exhibiting at least weak e\-commerce intent\. We further illustrate this user interaction timeline in Figure[4](https://arxiv.org/html/2607.03162#A2.F4)\. All user data inAPeBare collected in compliance with the platform’s privacy policies and contractual data\-use requirements\. Figure 4:User interaction timeline on the base platform\. ### B\.2Active User Filtering We retain only active users with at least five product or media interactions per day\. Specifically, we identify users who exhibit at least one search\-to\-order session \(defined in Section[2\.1](https://arxiv.org/html/2607.03162#S2.SS1)\) on each day in a 30\-days window\. For each retained user, we collect up to 60 days of historical interactions prior to the purchase date, including e\-commerce–related search queries, livestreams, videos, and product interactions\. ### B\.3Session Filtering We detail the implementation of session filtering rules introduced in Section[2\.1\.2](https://arxiv.org/html/2607.03162#S2.SS1.SSS2)\. #### B\.3\.1Rule\-based Session Filtering We first apply deterministic, rule\-based filtering using SQL over raw interaction logs\. Specifically, we retain only sessions that satisfy: \(i\) extensive browsing behavior, enforced by a minimum threshold on the total number of viewed products and content interactions \(Eq\.[4](https://arxiv.org/html/2607.03162#S2.E4)\); and \(ii\) iterative query refinement, requiring at least two distinct search queries within a session \(Eq\.[3](https://arxiv.org/html/2607.03162#S2.E3)\); \(iii\) coarse intent–purchase consistency, requiring that a sufficient number of viewed products or media items share the same category label as the final purchased product \(Eq\.[5](https://arxiv.org/html/2607.03162#S2.E5)\)\. These criteria remove shallow or accidental purchases and ensure that retained sessions reflect non\-trivial exploration and intent evolution\. #### B\.3\.2Intent–purchase aligning verification For sessions passing the rule\-based filters, we further verify intent–purchase alignment \(Eq\.[5](https://arxiv.org/html/2607.03162#S2.E5)\)\. Specifically, we check whether the final purchased item is semantically consistent with at least one issued query\. Semantic alignment between queries, viewed items, and purchased products is assessed using GPT\-4\.1 and validated by human review\. Annotators reviewed 100 randomly sampled de\-identified sessions containing the query sequence, relevant viewed product metadata, and the purchased item, and judged whether at least one query/view trajectory semantically supported the final purchase\. GPT\-4\.1 decisions achieved 97\.1% human agreement, and GPT\-5\.2 achieved comparable agreement at 97\.8% under the same protocol\. Sessions failing this semantic consistency check are discarded; if multiple purchased items satisfy the criterion, we retain only one to ensure a unique target per session\. #### B\.3\.3Distinctintentandrefined Given a sessionuuwith query sequence𝒬u=\{qu1,…,quTu\}\\mathcal\{Q\}\_\{u\}=\\\{q\_\{u\}^\{1\},\\dots,q\_\{u\}^\{T\_\{u\}\}\\\}and purchased productpuop\_\{u\}^\{o\}, we locate two aligned queries using GPT\-4\.1 by prompting it to*return indices*in𝒬u\\mathcal\{Q\}\_\{u\}: tintent=min\{t:\\displaystyle t\_\{\\text\{intent\}\}=\\min\\\{t:qutis semantically aligned\\displaystyle q\_\{u\}^\{t\}\\text\{ is semantically aligned\}withpuo\},\\displaystyle\\text\{with \}p\_\{u\}^\{o\}\\\},the earliest aligned query, i\.e\.,intent, and trefined=max\{t:\\displaystyle t\_\{\\text\{refined\}\}=\\max\\\{t:qutis aligned and explicit\\displaystyle q\_\{u\}^\{t\}\\text\{ is aligned and explicit\}in product type/attributes\},\\displaystyle\\text\{in product type/attributes\}\\\},the last explicit aligned query before purchase, i\.e\.,refined\. We then setquintent=qutintentq\_\{u\}^\{\\text\{intent\}\}=q\_\{u\}^\{t\_\{\\text\{intent\}\}\}andqurefined=qutrefinedq\_\{u\}^\{\\text\{refined\}\}=q\_\{u\}^\{t\_\{\\text\{refined\}\}\}\. To ensure*distinct*intents, we apply a minimal rule\-based check after GPT\-4\.1 localization: we discard sessionuuifquintentq\_\{u\}^\{\\text\{intent\}\}andqurefinedq\_\{u\}^\{\\text\{refined\}\}are identical after normalization \(lowercasing and whitespace/punctuation stripping\), i\.e\.,norm\(quintent\)=norm\(qurefined\)\\mathrm\{norm\}\(q\_\{u\}^\{\\text\{intent\}\}\)=\\mathrm\{norm\}\(q\_\{u\}^\{\\text\{refined\}\}\)\. This removes cases where the session contains only a single aligned query \(or duplicated re\-issues\) and thus lacks an observable intent refinement trajectory\. We manually verify 100 random samples by showing annotators the de\-identified query sequence, relevant product metadata, and purchased item, and asking whether the selectedintentis the earliest aligned ambiguous query andrefinedis the later explicit aligned query\. Both GPT\-4\.1 and GPT\-5\.2 achieve93%93\\%success in correctly locating\(tintent,trefined\)\(t\_\{\\text\{intent\}\},t\_\{\\text\{refined\}\}\); this query\-localization check is used to audit the post\-hoc aligned\-trajectory construction\. ### B\.4Heterogeneous History Curation We curate heterogeneous user histories by unifying product\-centric and media\-centric interaction records into a structured sequence, enabling fine\-grained modeling of user intent and preference evolution\. #### B\.4\.1Dwell\-time filtering We discard video and product view interactions with dwell time shorter than 10 seconds, retaining only interactions that reflect meaningful user engagement\. #### B\.4\.2Heterogeneous history content ##### Product interactions\. Each product interactionxu,productkx\_\{u,\\mathrm\{product\}\}^\{k\}\(Eq\.[6](https://arxiv.org/html/2607.03162#S2.E6)\) corresponds to an explicit user action on an item\. The product metadatametapk\\mathrm\{meta\}\_\{p\}^\{k\}consist of an*encrypted*product identifierid\\mathrm\{id\}, a categorical taxonomy labelcategory\\mathrm\{category\}, and textual fieldstitle\\mathrm\{title\}anddetail\\mathrm\{detail\}, wheredetail\\mathrm\{detail\}is parsed from the product’s structured HTML page\. Thefeature\\mathrm\{feature\}field contains a set of salient product attributes extracted by platform\-level product understanding pipelines\. The interaction typetypek∈\{view,like,cart,order\}\\mathrm\{type\}^\{k\}\\in\\\{\\text\{view\},\\text\{like\},\\text\{cart\},\\text\{order\}\\\}reflects increasing engagement intensity, andΔtk\\Delta t^\{k\}denotes the dwell time associated with the interaction event\. ##### Media interactions\. Each media interactionxu,mediakx\_\{u,\\mathrm\{media\}\}^\{k\}\(Eq\.[7](https://arxiv.org/html/2607.03162#S2.E7)\) represents user engagement with a short video or livestream\. The media metadatametamk\\mathrm\{meta\}\_\{m\}^\{k\}include an*encrypted*media identifierid\\mathrm\{id\}, contenttags\\mathrm\{tags\}, and atitle\\mathrm\{title\}\. Thedetail\\mathrm\{detail\}field is constructed from platform\-generated multimodal summaries, including captions and anonymized OCR/ASR transcripts, to capture the semantic content of the media\. The referenced product set𝒫referk\\mathcal\{P\}\_\{\\mathrm\{refer\}\}^\{k\}consists of products explicitly linked or automatically annotated as mentioned within the media content\. The engagement durationΔtk\\Delta t^\{k\}measures the user’s viewing or participation time\. ##### Privacy and compliance\. All interaction records and metadata are collected in accordance with the platform’s privacy and data usage policies\. Personal identifiers are removed or anonymized, and only platform\-approved structured annotations and processed textual signals are retained, ensuring that the benchmark supports personalization research without exposing sensitive user information\. #### B\.4\.3Release Scope and Governance We releaseAPeBas an anonymized benchmark package rather than raw platform logs\. The package is designed for research evaluation of preference\-conditioned product search and excludes fields that are unnecessary for reproducing the benchmark task or that create avoidable re\-identification risk\. Table[5](https://arxiv.org/html/2607.03162#A2.T5)summarizes the release boundary\. Table 5:Release scope for postprocessed benchmark records\. The release keeps only fields needed to reproduce the task and evaluation\.Before release, each benchmark instance passes the following checks: \(i\) no raw user/account identifiers; \(ii\) no cross\-split persistent user identifier; \(iii\) no fine\-grained timestamp; \(iv\) no raw event\-log payload; \(v\) no device, IP, GPS, or account\-profile attribute; \(vi\) exact duplicate product exposures collapsed where needed for candidate construction; \(vii\) free\-form product/media text scanned for phone numbers, emails, addresses, usernames, URLs, and other unique identifiers; and \(viii\) spot checks on sampled records containing OCR/ASR\-derived text\. Data minimization is applied at both the user and field levels\. We retain only the history window and shopping\-session evidence needed to define the benchmark task, cap the number of retained interactions, and omit raw events outside the benchmark construction window\. In the retained benchmark, users average 187 historical interactions; under the source\-log accounting used for governance, the retained records correspond to about 200 interactions per user per month, approximately 10% of shopping\-related activity\. The benchmark is intended for research evaluation of personalized product search, preference reasoning, history\-aware ranking, and agentic shopping workflows\. It is not intended for identifying individuals, reconstructing behavioral timelines, building user profiles outside the benchmark task, joining records with external datasets, production ad targeting, or optimizing manipulative personalization strategies\. Dataset users should agree not to attempt re\-identification, linkage attacks, demographic or sensitive\-attribute inference about individuals, or redistribution outside the approved access terms\. Construction and release are subject to internal privacy/compliance review and platform data\-use policies\. The release package will document the approved access path, permitted research uses, license or terms of use, and a contact/takedown mechanism for privacy concerns\. These safeguards reduce but do not eliminate privacy risk: long interaction histories and rare product/media combinations may still carry residual linkage risk, and OCR/ASR\-derived text can contain sensitive content even after automated redaction\. ### B\.5Benchmark Statistics The following statistics summarize retained postprocessed benchmark records after the release\-scope filtering in Appendix[B\.4\.3](https://arxiv.org/html/2607.03162#A2.SS4.SSS3)\.APeBcovers 5,648 cases with the category distribution shown in Table[6](https://arxiv.org/html/2607.03162#A2.T6)\. As Section[3](https://arxiv.org/html/2607.03162#S3)shows, retained users average 187 historical interactions, including 82 product interactions and 105 media interactions, with 18 historical orders on average; candidate sets contain 14\.1 items on average\. Table 6:Category distribution of purchased products in the benchmark; lower\-frequency categories are grouped under Others\.#### B\.5\.1Hard\-Candidate Controls and Difficulty Diagnostics We audit hard candidates to ensure that the setting is not driven by candidate\-set artifacts\. For each retained session, the candidate set contains the purchased item and user\-viewed alternatives after validity filtering, semantic filtering, duplicate handling, and candidate\-count outlier removal\. Sessions with more than 25 candidates are removed; the remaining candidate sets have mean size 14\.1 and standard deviation 2\.8\. Semantic filtering keeps at least 9 similar items per retained case \(see Table[7](https://arxiv.org/html/2607.03162#A2.T7)\)\. Table 7:Hard\-candidate construction controls\. Values summarize the retained benchmark after filtering unless specified otherwise\.We additionally treat price as part of the user\-facing evidence available during the shopping decision and can reflect user preference\. However, promotion, stock, delivery, or exposure\-rank effects may also act as a residual commercial confounder\. For a diagnostic purchase\-validity check, we sample 500 retained sessions and compare the purchased item against hard negatives in pairwise judgments\. GPT\-5\.2 receives de\-identified product metadata, histories, and queries, then chooses which item better matches the user preference\. For each session, the purchase win rate is the fraction of pairwise comparisons in which the purchased item is preferred over a hard negative\. Table[8](https://arxiv.org/html/2607.03162#A2.T8)shows that user\-history preference evidence substantially improves purchase distinguishability, beyond what is explained by purely price\-based comparisons and supports interpreting the purchase label as a preference signal rather than an artifact of commercial confounders\. Table 8:Pairwise distinguishability on 500 sampled sessions\. The third row removes price from candidate metadata, isolating non\-price preference signal\.We further test whether useful preference evidence can be recovered from noisy histories\. Shown in Table[9](https://arxiv.org/html/2607.03162#A2.T9), a hard\-candidate decision diagnostic, performance improves from 67% with query\-only context to 81% after one round of preference extraction and 83% after multi\-round extraction, indicating that history contains recoverable signals that help distinguish near\-miss candidates\. Table 9:Effect of preference extraction on purchase identification among hard candidates\.Together, these controls reduce possible artifacts for hard\-candidate difficulty: retained cases are size\-controlled, invalid candidates are filtered, duplicate exposures are collapsed, and purchased items remain distinguishable from hard negatives when preference evidence is supplied\. We acknowledge that they do not eliminate all commercial confounders, which is a common limitation observed in online collected shopping benchmarksHarper and Konstan \([2015](https://arxiv.org/html/2607.03162#bib.bib18)\); Liu et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib26)\)\. #### B\.5\.2Representativeness & Purchase Validity To assess whetherAPeBreflects a common shopping pattern, we measure the prevalence of multi\-step shopping behavior in the source logs used for benchmark construction\. We define multi\-step shopping behavior as issuing at least two queries and comparing at least ten products before purchase\. Under this definition, 67% of source\-log users exhibit multi\-step behavior in a 30\-minute window, indicating that the retained setting captures a frequent high\-effort shopping regime\. The small gap in Table[10](https://arxiv.org/html/2607.03162#A2.T10)between low\- and high\-activity users suggests that the benchmark is not driven solely by users with unusually dense histories\. Table 10:Performance stability across user activity levels\. We split users by retained historical interaction count and report average Hit@1 over evaluated models onintenthard\-candidate cases\. #### B\.5\.3Scope and Transferability APeB is instantiated in personalized product search, but the benchmark is designed around a domain\-agnostic personalization structure: infer intent from an underspecified request, recover task\-relevant preferences from noisy history, and select among plausible candidates or actions\. The mapping in Table[11](https://arxiv.org/html/2607.03162#A2.T11)demonstrates why PPS is a concrete, behavior\-grounded testbed for challenges that recur across personalization settings\. Table 11:Structural mapping from APeB/PPS to broader personalization settings\. APeB provides an operational testbed for shared reasoning challenges rather than empirical coverage of every domain\. ## Appendix CExperiment Setting Due to architectural differences across*LLM\-only*,*ReAct agent*, and*DeerFlow agent*settings, we adopt model\-specific configurations\. Unless otherwise stated, all models use their default temperature\. The LLM/ReAct Hit@1/5 values in Table[2](https://arxiv.org/html/2607.03162#S4.T2)are averaged over three runs\. The detailed settings are as follows: 1. 1\.Gemini\-2\.5\-Pro\(GEMINIin Table[2](https://arxiv.org/html/2607.03162#S4.T2)\): When used as a standalone LLM or within DeerFlow, we adopt the default reasoning budget\. When deployed as a ReAct agent, we reduce the reasoning budget to 1024 tokens to ensure practical efficiency\. 2. 2\.GPT\-4\.1\-Mini\(4\.1MINI\): All experiments use the default configuration\. 3. 3\.GPT\-4o: All experiments use the default configuration\. 4. 4\.Qwen3\-235B\(Qwen3\-235B\): Although the model supports internal reasoning, we disable its thinking mode in all experiments due to interface limitations\. Other settings remain default\. 5. 5\.DeepSeek\-R1\(R1\): Thinking mode is enabled with the default reasoning budget\. 6. 6\.GPT\-4\.1: All experiments use the default configuration\. 7. 7\.GPT\-5\.2: Unless specified otherwise,GPT5\.2refers to the model configured with a minimal reasoning budget, with all other settings left unchanged\. 8. 8\.GPT\-5\.2Think: Unless specified otherwise,GPT5\.2Thinkrefers to the GPT\-5\.2 condition with the default reasoning budget, whileGPT5\.2uses the minimal reasoning\-budget condition; all remaining settings follow the defaults\. 9. 9\.Additional model\-family runs: Qwen3\.5, Gemini3Pro, GPT5\.5, and GPT5\.5Think follow the same prompt, history\-selection, retrieval\-tool, and parsing protocols as the corresponding main\-table LLM or reasoning\-model conditions\. ### C\.1Additional Model\-Family Results Table[12](https://arxiv.org/html/2607.03162#A3.T12)provides the appendix position for additional model\-family runs that extend the main result structure in Table[2](https://arxiv.org/html/2607.03162#S4.T2)\. Table 12:Additional model\-family results under the same hard\-candidate protocol as Table[2](https://arxiv.org/html/2607.03162#S4.T2)\. Values are Hit@1/5 percentages\. Deerflow andVQRAare not yet evaluated for these model families\. ### C\.2Additional Baseline Settings We choose UniSARShi et al\. \([2024](https://arxiv.org/html/2607.03162#bib.bib51)\)as the primary non\-LLM PPS/SAR baseline because it is publicly available and directly models user transition behavior between search and recommendation\. We report additional reproducible embedding, reranking, and memory\-agent checks, including Qwen3 embedding/reranking variantsZhang et al\. \([2025b](https://arxiv.org/html/2607.03162#bib.bib75)\), MemAgentYu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib72)\), and Mem0Chhikara et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib8)\)memory workflows\. Table 13:Additional baseline checks onintenthard\-candidate cases\. We report Hit@1 \(%\) under the same candidate protocol\.For embedding baselines, query and candidate text are encoded and ranked by similarity; the history\-conditioned variant appends selected user\-history evidence before scoring\. For trained embedding and reranker baselines, training uses the retained search\-purchase supervision available to the baseline protocol, and evaluation uses theAPeBhard\-candidate set\. The baselines slightly differ in signal access\. UniSAR and the trained embedding/reranker variants learn from historical search\-purchase supervision, whereas LLM prompting and agent workflows operate primarily through in\-context history evidence\. UniSAR also uses cross\-session negatives, which is standard in SAR/PPS trainingLiu et al\. \([2025](https://arxiv.org/html/2607.03162#bib.bib27)\)\. #### C\.2\.1Memory\-Agent Baselines We report Mem0 and MemAgent memory\-workflow results separately from embedding and supervised baselines because these methods wrap an evaluated LLM backbone with an explicit memory component\. Table[14](https://arxiv.org/html/2607.03162#A3.T14)is evaluated on a comparable LLM mix as Table[2](https://arxiv.org/html/2607.03162#S4.T2)\. Table 14:Memory\-agent experiment results under the hard\-candidate protocol\. Values are Hit@1/5 percentages when available; dashes denote settings not evaluated or not available in the current result snapshot\. ### C\.3Single\-Prompt LLM This subsection describes the shared direct\-prompt protocol; agent\-specific extensions are described in Appendix[C\.4](https://arxiv.org/html/2607.03162#A3.SS4)and Appendix[C\.5](https://arxiv.org/html/2607.03162#A3.SS5)\. #### C\.3\.1LLM Prompt For all LLMs evaluated under the same setting, we use an identical system prompt to ensure fair comparison\. The prompt is instantiated according to the scenario configuration\. For example, when the input consists of aintentquery with hard negative candidates, the system prompt takes the following form \(details omitted for brevity\): > TASK: Structured Product Recommendation \(Exactly 5 Items\) Your goal is to recommend exactly five products that best match the user’s explicit needs, using only data from: \(1\)<<history\>\>\-\-\- chronological user interactions with indices \(action type, product in this history, price\) \(2\)<<query\>\>\-\-\- the user’s current query \(may be ambiguous or intentional\) \(3\)<<candidates\>\>\-\-\- product candidates \(index, title, price\) STEP 0 \-\-\- Determine the User’s Explicit Target \(Ambiguous Query Handling\) \(\.\.\.\) STEP 1 \-\-\- Identify Important History Entries Based on the Query \(\.\.\.\) STEP 2 \-\-\- Extract Explicit Features \(Summarized\) & Rank Relevance \(\.\.\.\) STEP 3 \-\-\- Evaluate All Candidate Products \(\.\.\.\) STEP 4 \-\-\- Produce Exactly 5 Final Recommendations \(\.\.\.\) OUTPUT SPECIFICATION \(STRICT JSON ONLY\) Output raw JSON conforming to the Recommendation interfacewithoutwrapping it in triple back\-ticks\. Type schema: interface Product \{ rank: int; // 1\-\-5, unique ascending index: int; // candidate index product\_title: string; // rewritten title \(no new facts\) product\_description: string; // rewritten description product\_recommendation\_reason: List\[string\]; // grounded in query \+ Step 0 \+ summarized\_features \} interface SupportingHistoryFeature \{ summarized\_feature: string; support\_history\_index: List\[int\]; relevance\_rank: int; // from Step 2 support\_product: List\[int\]; // candidate indices supported \} interface Recommendation \{ target\_product: string; // derived in Step 0 concerned\_features: string; // derived in Step 2 products: List\[Product\]; // exactly 5 items histories: List\[SupportingHistoryFeature\]; //\>\>5 entries, sorted by relevance\_rank \} End schema\. In other scenarios, the system prompt may undergo minor specification changes, while its overall structure remains unchanged\. The<history\>,<query\>, and<candidates\>are supplied in the user prompt using a fixed template\. For each interaction, to avoid introducing irrelevant context, only essential fields—interaction type, title, and price—are included; all identifiers and sensitive attributes are encrypted\. > <<history\>\> 1\. Action type: order product; Product name: \[DOSEN SKINCARE APPROVED\] Emina Water Gel Moisturizer 30g \| Mencerahkan & Mengurangi Flek Hitam \| 48H Menjaga Kelembapan, Memperkuat Skin Barrier \| Mengurangi Jerawat dan Meredakan Kemerahan, Kulit Glowing \| Mengandung Niacinamide dan Vitamin C; Price: 3\.8 usd \(\.\.\.\) <</history\>\> <<query\>\> dd cream wardah <</query\>\> <<candidates\>\> 1\. Product name: \( DRY SKIN \) Hyalupure Emulsion for DRY SKIN \- Perawatan kulit KERING menjadi lembab dan kencang; Price: 15\.078408 usd \(\.\.\.\) <</candidates\>\> Finally, we parse model outputs using a Pydantic\-based parser and compute standard retrieval metrics, including HR@1 and HR@5\. #### C\.3\.2History Record Selection To control prompt length while preserving essential interaction semantics \(e\.g\., action type, product title, and price\), we limit the history to at most 60 records using an importance\-aware selection mechanism\. Results in Table[2](https://arxiv.org/html/2607.03162#S4.T2)primarily adopt the category\-prior strategy using the category label associated with the provided intent query\. ##### Importance scoring\. Each interaction is assigned an importance score determined by interaction type and dwell time, with stronger behavioral signals ranked asorder\>cart\>like\>view\\texttt\{order\}\>\\texttt\{cart\}\>\\texttt\{like\}\>\\texttt\{view\}\. For interactions of the same type and category, longer dwell time yields higher importance\. ##### Time\-prior selection\. From the full interaction history, we first retain records whose importance exceeds a predefined ratio threshold\. The final 60 records are then selected by recency within this importance\-filtered subset, ensuring that retained histories emphasize both behavioral salience and temporal relevance\. ##### Category\-prior selection\. We first assign the provided intent query to a coarse shopping category label, such as skin care, toys, or clothing, following the category taxonomy summarized in Appendix[B\.5](https://arxiv.org/html/2607.03162#A2.SS5)\. We then include interactions whose category matches this intent\-query category, up to 60 records, ranked by importance\. If fewer than 60 same\-category records are available, the remaining slots are filled using the time\-prior selection strategy above\. This approach maximizes query\-relevant category evidence while retaining salient and recent interactions\. This schema does not use the purchased item or reveal target\-product information\. We include this variant to test whether query\-relevant category histories help models extract user preferences from noisy behavior records\. #### C\.3\.3History Context Stress Tests In Table[15](https://arxiv.org/html/2607.03162#A3.T15), we run additional stress tests on 500 randomly selectedintenthard\-candidate sessions to isolate whether history gains come from useful user evidence\. Table 15:History context stress tests on 500 randomly selectedintenthard\-candidate sessions\. We report Hit@1 \(%\); only the supplied history context is varied\.Shuffling histories reduces Hit@1 for both R1 and Qwen3, indicating that performance depends on coherent user\-specific evidence\. Modality\-specific results show that product and media/video histories both provide preference signals, while increasing the context from 60 to 240 records does not improve performance\. These results demonstrate that useful history selection and grounding matter more than simply exposing longer histories\. #### C\.3\.4Contamination and Counterfactual History Check We also probedata contaminationthrough a memorization hypothesis: if decisions mainly rely on memorized query\-item associations, then replacing the user’s real history with unrelated counterfactual histories should have limited effect\. We therefore hold the query and candidate set fixed and compare query\-only, counterfactual\-history, and real\-history contexts on 500 sampled sessions\. For each sampled session, the query and candidate set remain unchanged\. In the query\-only condition, the model receives no historical records\. In the counterfactual condition, the true user’s history is replaced with a length\-matched history sampled from a different user, excluding records from the target user and session\. In the real\-history condition, the model receives the original user history under the same formatting and parsing protocol\. We report pairwise preference accuracy in Table[16](https://arxiv.org/html/2607.03162#A3.T16): the percentage of purchase\-vs\-hard\-negative comparisons in which the model selects the observed purchased item\. Table 16:Counterfactual\-history contamination check on 500 sampled sessions\. Query and candidate set are fixed across conditions\. We report pairwise preference accuracy \(%\) for selecting the observed purchased item over hard negatives\.Counterfactual histories stay close to query\-only performance for all three models, while real histories improve over counterfactual histories substantially\. This pattern suggests that useful performance depends on the matching user’s context rather than arbitrary additional history text\. The check provides evidence that memorized query\-item associations are unlikely to be the dominant source of the observed history\-conditioned gains\. To address possible leakage concerns during data collection and evaluation, Appendix[C\.7\.3](https://arxiv.org/html/2607.03162#A3.SS7.SSS3)separately audits GPT\-4\.1’s construction and diagnostic roles; GPT\-4\.1 does not determine Hit@K labels, and its evaluated task performance is not superior to the other evaluated models\. ### C\.4ReAct Agent Our ReAct agent is implemented using the open\-source LangChain framework\. Compared to the single\-prompt LLM setting, the system prompt follows the same high\-level structure but additionally specifies the available tools and their usage constraints\. > AVAILABLE TOOL 1 \-\-\- structured\_search\_tool Purpose: Use this tool only to retrieve literal factual clarifications about products or history entries\. Permitted data sources: \- Candidate product detailed descriptions \- History product detailed descriptions \- Candidate product detailed features \- History product detailed features \- History video descriptions \(OCR/ASR\) Tool Query Rules: 1\. You must generate your own search queries from: \- literal terms in the user query \- literal terms relevant to target\_product or concerned\_features \- literal candidate titles or features \- literal history text \- interpretation\-critical terms ONLY when clarifying explicit text 4\. You may only rely on: \- explicit user inputs \- explicit tool results AVAILABLE TOOL 2 \-\-\- web\_search\_tool \(\.\.\.\) Beyond the explicit system\-prompt specification, model providers \(e\.g\., GPT and Gemini\) impose additional constraints and instructions on tool invocation through their respective interfaces\. We strictly follow the official usage guidelines of each provider\. The user prompt is identical to that used in the single\-prompt LLM setting\. ##### Available Tools\. 1. 1\.Structured Search Tool\.In addition to item titles,APeBprovides rich auxiliary information, including product descriptions, video OCR/ASR transcripts, and structured attributes\. As these contents exceed prompt\-length limits, they are organized in a fixed schema and stored in a retrieval database, which the agent queries via this tool \(see Appendix[C\.6](https://arxiv.org/html/2607.03162#A3.SS6)\)\. 2. 2\.Web Search Tool\.We wrap the web search API provided by YOU111https://you\.com/homeas a callable tool\. Each query returns the top\-3 relevant text chunks\. 3. 3\.Crawl Tool\.We use the Jina AI crawling API222https://jina\.ai/to fetch web pages and convert them into agent\-friendly Markdown representations\. 4. 4\.Code Execution Tool\.We employ the Python REPL tool from LangChain333https://www\.langchain\.com/, enabling the agent to execute code and observe intermediate results\. #### C\.4\.1ReAct Failure\-Source Analysis We analyze logged ReAct traces under the same hard\-candidate evaluation setting used in the main tables\. In Table[17](https://arxiv.org/html/2607.03162#A3.T17), for each trace, we report the number of reasoning rounds reached, and the AC/QC/QH/QQ diagnostic scores defined in Appendix[C\.7\.2](https://arxiv.org/html/2607.03162#A3.SS7.SSS2)\. Retrieval quality is measured by rubric\-scored relevance to the issued query and utility to the user intent of the retrieved evidence\. MetricIntentRefinedCandidate retrieval relevance \(QC\-1\)8892History retrieval relevance \(QH\-1\)8186Candidate retrieval utility \(QC\-2\)6880History retrieval utility \(QH\-2\)4955Query total quantity \(QQ\)8\.18\.5Table 17:ReAct retrieval diagnostics: relevance and utility of retrieved candidate/history evidence, plus total query count\.The reasonable query quantity indicates that the ReAct degradation is not primarily due to agents failing to call the available tools\. The lower QH scores indicate that the harder bottleneck is formulating history\-relevant queries under vague goals: agents often produce locally plausible search terms, but these terms retrieve weaker personalization evidence than candidate metadata\. Table 18:Round\-depth and coherence for ReAct traces\. Ratio is the percentage of traces reaching each round; AC is the mean agent\-coherence score for that round\.Table[18](https://arxiv.org/html/2607.03162#A3.T18)shows that most traces stop after one round, and the small subset that continues does not show improved coherence\. This pattern argues against over\-exploration as the main cause of degradation and indicates limited spontaneous self\-correction\. Table 19:Self\-correction prompt ablation for ReAct onintenthard\-candidate cases\. Values are Hit@1 \(%\)\.We additionally report ReAct results with the explicit self\-correction instruction prompt in Table[19](https://arxiv.org/html/2607.03162#A3.T19)\. This produces only small, model\-dependent changes across models\. Table 20:Retrieval\-backend sensitivity for ReAct onintenthard\-candidate cases\. Values are Hit@1 \(%\)\.In Table[20](https://arxiv.org/html/2607.03162#A3.T20), we report Hit@1 results when varying the retrieval backend\. The minimal differences suggest that backend choice alone does not explain the ReAct gap\. These checks indicate that a redesigned agent architecture or task\-specific planner may be needed to improve personalization performance\. ### C\.5DeerFlow Agent System Figure 5:Overview of our modified DeerFlow agent system\.As the original DeerFlow agent system includes components unsuitable forAPeBevaluation \(e\.g\., coordinators and human feedback loops\), we simplify its architecture to the workflow illustrated in Figure[5](https://arxiv.org/html/2607.03162#A3.F5)\. For all DeerFlow\-based experiments, the evaluated model simultaneously assumes the roles of*Planner*,*Researcher*, and*Reporter*\. #### C\.5\.1Planner The Planner receives the user input—consisting of history, query, and candidates—directly as input\. It is instructed to group relevant historical interactions and candidate items, and to formulate a set of research tasks aimed at information gathering and user preference investigation\. The system prompt follows the form: > \(\.\.\.\) Follow these steps to generate a research plan: 1\.Understand query and products\- Understand the action content\. Reason about the hidden pattern\. Catch the relation between the action content, search query, and candidates\. Based on your thinking result, infer a grouping strategy that is helpful for researcher to dig out the important and detailed features\. 2\.Group products\- Based on the grouping strategy derived from step 1, break the history interactions and candidates into different groups\. In each groups, please clearly list the index of the interactions and products in this group, but you don’t need to include the interaction content or product title in the task description because it will be automatically passed to researcher based on your index output\. 3\.Check group size\- Consider whether the size of each group is reasonable to form a research task with medium difficulty and clear scope \(a desired size will be between 6 and 10 interactions and candidates related with each other\)\. If not, adjust the grouping strategy\. 4\.Formulate research goal\- Formulate a clear and concise research goal for each group\. The goal should be suitable enough to utilize the ability of researcher \(they are strong models with retrieval tools\)\. \(\.\.\.\) Output raw JSON conforming to the Plan interfacewithoutwrapping it in triple back\-ticks\. Type schema: interface Step \{ need\_search: bool; // Must be explicitly set for each step title: str; description: str; // Specify exactly the product link and title that belong to this group, and discuss what features to collect from these products\. interactions\_index\_list: List\[int\]; // The index of the products in this group candidates\_index\_list: List\[int\]; // The index of the candidates in this group \} interface Plan \{ has\_enough\_context: bool; thought: str; title: str; steps: List\[Step\]; // Research & Processing steps to get more context \} End schema\. \(\.\.\.\) #### C\.5\.2Researcher The Researcher agent shares the same implementation and toolset as the ReAct agent described in Section[C\.4](https://arxiv.org/html/2607.03162#A3.SS4)\. The only differences lie in the input and system prompt: the Researcher receives the research tasks generated by the Planner, and operates under the following system prompt: > \# Task Background You are aresearch agentin an e\-commerce recommendation system\. This recommendation system is responsible for predicting what user will order for current search query based on the history interactions list and the candidate list of products\. It made up of three agents: 1\.Planning agent\- Understand the user’s search query, group the history interactions and candidates by feature or category, and break the overall task into sub\-tasks on different group of products for the research agent\. 2\.Research agent \(you\)\- Uses various retrieval tools to explore the product information in depth for the allocated group and analyze the feature of this group that user might be interested in or not interested in\. 3\.Reporter agent\- Compile the results from the research agent and combine it with user’s search query to generate a final report on how much is a candidate suitable for user and what is the most suitable candidate for user\. You are dedicated to conducting in\-depth investigations about part of user history interactions and product candidates\. You will be provided with a research topic, and a research description, together with a group of history interactions and candidates that this research topic focus on, which is carefully selected and grouped\. \(\.\.\.\) \# Output Format \- Provide a structured response in markdown format\. \- Include the following sections: \-Problem Statement: Restate the problem for clarity\. \-Findings: Organize your findings by topic rather than by tool used\. For each major finding: \- Summarize the key information \- Track the sources of information but DO NOT include inline citations in the text \-Conclusion: Provide a synthesized response to the problem based on the gathered information\. \(\.\.\.\) #### C\.5\.3Reporter The Reporter agent receives the original user input, the research plan, and all research reports produced by the Researchers\. It is instructed to synthesize the collected evidence into a final report and produce a ranked list of candidates\. The system prompt takes the following form: > \(\.\.\.\) You should analyze the research result, user persona, and user history following these steps: 1\. List: In the research result of different groups, there are lots of features that are unique or similar to each other\. Please distinguish them and list them out clearly\. For those similar features, you can combine them if they share exactly the same meaning, but \*\*NEVER\*\* combine them into one general feature because it will make the report too general and not able to meet user’s needs\. 2\. Ranking: Based on user persona and research results, rank the primary features to consider first when the user issues such query content\. Notice that common features are not necessarily to be ranked higher, because details always matter to the user\. 3\. Choosing: Based on the user persona and ranking result, discuss why each candidate is or is not suitable for the user, and present the most suitable candidate for the user\. Finally, you should output a rank for all candidates based on how much it is suitable for the user\. \(\.\.\.\) Output raw JSON conforming to the Recommendation interfacewithoutwrapping it in triple back\-ticks\. Type schema: interface Product \{ product\_title: str; // The title of the product product\_description: str; // A detailed description of the product to make it more understandable product\_recommendation\_reason: List\[str\]; // The reasons that this product is recommended to the user rank: int; // The rank of the product in the recommendation, 1 being the most important index: int; // The original index of the product in the candidates list \} interface Recommendation \{ title: str; // Main topic of the recommendation summary: str; // A summary that previews the most important points of the recommendation products: List\[Product\]; // The products that are recommended to the user, ranked by their importance \} End schema\. ### C\.6Retrieval Database As discussed earlier, each interaction or candidate inAPeBcontains rich auxiliary information\. We organize this information into a retrieval database, which agents access via a retrieval tool\. Unlike conventional Retrieval\-Augmented Generation \(RAG\) setups that partition content into fixed\-length chunks and retrieve them via embedding similarity, such chunking would fragment individual interactions and degrade semantic coherence in our setting\. We therefore adopt a document\-level retrieval strategy, where the atomic retrieval unit corresponds to a single interaction\. Each document aggregates all information associated with one interaction and is organized using the following format: > \# Detail information of one candidate product Product name: WARDAH C\-Defense Vitamin C All Series Lengkap \- Panthenol Anti\-Pollution Radiant Creamy Wash, Energizing Whip Foam, Mousse Moisturizer, DD Cream Light/Natural, Face Mist, 10% Ferulic Acid Radiant Charge Serum \- Skincare Price: 2\.237636 usd Product description: <p\>Wardah C\-Defense Vitamin C Series</p\> <p\>1\. Wardah C\-Defense Vitamin C \+ Panthenol Anti\-Pollution Radiant Creamy Wash \(50 mL/100 mL\)</p\> <p\>Membantu menjaga pertahanan acid mantle sebagai garda terdepan skin barrier\-mu\. Mengandung Vitamin C dan Panthenol yang membantu kulit tampak lebih cerah dan terlindungi dari polusi\. \(\.\.\.\)</p\> Categories: Beauty & Personal Care, Skincare, Serums & Essences, Serums & Essences Brand name: wardah Shop name: wardahbeautyid Attributes: \{’Volume’: \[’55 ml’\], ’Negara Produsen’: \[’ID’\], ’Edisi’: \[’Edisi Reguler’\], ’Kuantitas per Kemasan’: \[’1’\], ’Mengandung Alkohol atau Aerosol’: \[’Tidak Mengandung Keduanya’\], ’Contains Dangerous Goods?’: \[’Tidak’\], ’Varian’: \[’Creamy Wash 100’, ’Creamy Wash 50’, ’DDCream Light’, ’DDCream Natural’, ’Face Mist’, ’Gel Cleanser’, ’Gel Moisturizer’, ’Mousse Moist’, ’Serum’, ’Whip Foam 100’, ’Whip Foam 50’\], ’Edition’: \[’Regular Edition’\], ’Region of Origin’: \[’ID’\], ’Contains Alcohol or Aerosol’: \[’Contains Neither’\], ’Quantity Per Pack’: \[’1’\], ’Is Suit’: \[’is\_suit’\]\} The document format is adapted to its source type \(product interaction, media interaction, or candidate\)\. For example, interaction documents additionally include the corresponding action type\. We also consider the suitability of standard embedding\-based retrieval for this setting\. Empirically, we observe that agents primarily issue title\-based queries when invoking the retrieval tool, using item titles to locate detailed information\. Consequently, we adopt BM25—a robust lexical retrieval method—as the backend of the retrieval tool, rather than embedding\-based similarity search\. ### C\.7Rubric Details #### C\.7\.1Diagnostic goal proxy summarization We infer a behavior\-grounded diagnostic goal proxyggat the session level from all interactions in Eq\.[2](https://arxiv.org/html/2607.03162#S2.E2), including issued queries, long\-viewed products \(≥\\geq15s\), and the purchased item, using GPT\-4\.1\. The model is prompted to reconstruct the latent user intent that best explains the complete search–purchase trajectory\. Specifically, GPT\-4\.1 outputs a concise goal proxyggthat minimally and coherently aligns the observed queries with the final purchase\. This proxy is used only as a reference for rubric diagnosis and does not determine Hit@K labels, which are computed from observed purchases in the ranked candidate lists\. #### C\.7\.2Rubric perspectives We detail the multi\-level rubric design in Section[4\.3](https://arxiv.org/html/2607.03162#S4.SS3)\. Diagnostic scores reported in the main analysis are averaged over Gemini\-2\.5\-Pro, GPT\-5\.2, and GPT\-4\.1 judges; judge\-sensitivity details are provided in Appendix[C\.7\.3](https://arxiv.org/html/2607.03162#A3.SS7.SSS3)\. Table 21:Behavior\-grounded targets for rubric diagnostics\. Hit@K remains the task label; rubrics diagnose intermediate failures against observed session evidence\.- •*Query Inference \(QI\)*evaluates intent understanding by measuring how well the model\-inferred targetτ\\taualigns with the user’s explicit query and stated intent\. Specifically, we prompt the judge to score the relevance between\(user\_query,user\_goal\)\(\\texttt\{user\\\_query\},\\texttt\{user\\\_goal\}\)and the inferred target \(target\_product\), producing an integer score in\[0,100\]\[0,100\]under a graded relevance rubric ranging from contradiction to direct satisfaction\. We use thequery\_intent\_to\_target\_relation\_scoreas the QI metric\. - •*Historical Preference Grounding* - –*Preference Quality \(PQ\)\.*We assess whether each summarized preference featureℱ\\mathcal\{F\}is grounded in user history, goal\-relevant to the inferred intent, and non\-hallucinatory\. A preference is considered valid only if it coherently contributes to explaining the purchased item choice without contradiction or fabrication\. - –*Preference Salience Statistics\.*We quantify the strength of historical grounding using two statistics over validated preferences:*Preference Minimum Rank \(PR\)*, defined as the rankrir\_\{i\}of the first preference supporting the purchased item \(lower is better\), and*preference count \(PC\)*, defined as the total number of distinct preferences that plausibly support the purchased item\. Both are computed under strict gating rules that heavily penalize irrelevant, contradictory, or ungrounded preferences\. - •*Recommendation Quality \(RQ\)*evaluates the recommended setR⋆R^\{\\star\}with fine\-grained, score\-based assessment along two dimensions\.*Faithfulness*checks each recommended itemcic\_\{i\}against reference product metadata, assigning separate scores fortitle\-\-truthanddescription\-\-truthconsistency to penalize hallucinated, exaggerated, or contradictory claims\.*Relevance*measures alignment with the user goalggvia two complementary scores: \(i\)product\-\-intent alignment, evaluating whether the recommended title and description address the user query and intent, and \(ii\)reason\-\-intent alignment, assessing whether the provided rationaleρi\\rho\_\{i\}coherently justifies the recommendation with respect togg\. An overall RQ score is computed by aggregating these aspect scores, with gating rules enforcing strong penalties under critical factual or consistency violations\. - •*Agent Reasoning* - –*Agent Coherence \(AC\)*evaluates the*coherence*of each reasoning stepReiRe\_\{i\}by assessing whether the agent’s explicit reasoning logically justifies its stop\-or\-continue decision under a ReAct\-style retrieval framework\. The evaluator scores the reasoning on intent understanding, task relevance, alignment with the purchased candidate, decision correctness, and logical completeness, penalizing unjustified leaps, hallucinated assumptions, or premature termination\. The resulting AC score reflects the decision validity of the agent’s stepwise reasoning\. - –*Candidate Query \(QC\)*evaluates the*quality*of each retrieved candidate by scoring the alignment between the issued search keywords, the agent’s reasoning, and the retrieved item\. We measure query relevance by scoring the retrieved evidence against the issued query\. Query utility measures whether the keywords accurately reflect the user query and diagnostic goal proxy, are logically derived from the ReAct reasoning, and effectively retrieve candidates that are consistent with the observed purchased product\. The resulting QC score captures how informative and decision\-relevant the retrieval step is, penalizing generic, accidental, or misaligned candidate retrievals\. - –*History Query \(QH\)*evaluates the*quality*of retrieval history queries by scoring how effectively the issued search keywords retrieve decision\-relevant user history\. We measure query relevance by scoring the retrieved evidence against the issued query\. Query utility measures whether the issued keywords retrieve history entries that provide meaningful signals for identifying the observed purchased candidate\. The resulting QH score reflects the usefulness of retrieved history for personalization, penalizing generic, accidental, or weakly informative history retrievals\. - –*Query Quantity \(QQ\)*measures the total number of retrieval queries issued by the agent, defined as the sum of candidate queries and history queries, capturing the overall extent of retrieval tool usage during reasoning\. For the ReAct failure\-source analysis in Appendix[C\.4\.1](https://arxiv.org/html/2607.03162#A3.SS4.SSS1), AC is averaged at the reasoning\-step or round level, QC/QH summarize the quality of retrieved candidate/history evidence, and QQ is summarized through total query count across candidate/history across multiple steps\. #### C\.7\.3LLM Role Audit and Judge Robustness We audit LLM usage by separating benchmark construction filters, evaluated model predictions, diagnostic goal proxies, rubric judging, and VQRA rewriting\. Hit@K labels remain observed purchases within behavior\-grounded hard\-candidate sets; LLMs only support filtering, localization, diagnostic scoring, or rewrite modules as summarized in Table[23](https://arxiv.org/html/2607.03162#A3.T23)\. Judge robustness is checked with Gemini\-2\.5\-Pro, GPT\-5\.2, and GPT\-4\.1 judges in Table[23](https://arxiv.org/html/2607.03162#A3.T23)\. Table 22:Audit of LLM roles in benchmark construction, diagnostics, and experimental components\.Table 23:Judge\-sensitivity check for diagnostic all rubric scores on 500 randomly sampled sessions withIntentqueries\. Scores are stable across Gemini\-2\.5\-Pro, GPT\-5\.2, and GPT\-4\.1 judges\.The three judges produce nearly identical relative rankings on this sample, and human pairwise preference checks agree with the rubric preference in 96 of 100 cases\. This supports using rubric scores as diagnostic evidence while preserving Hit@K as the task\-native evaluation metric\. #### C\.7\.4Rubric Validity via Correlation with Task Metrics The complementary judge\-sensitivity evidence is summarized in Appendix[C\.7\.3](https://arxiv.org/html/2607.03162#A3.SS7.SSS3); this subsection focuses on rubric–Hit@K alignment\. To evaluate output quality at the component level, we adopt an*LLM\-as\-a\-Judge*paradigm to score model behaviors along multiple rubric dimensions \(Appendix[C\.7](https://arxiv.org/html/2607.03162#A3.SS7)\)\. Prior work on LLM\-based evaluation typically establishes validity through agreement with human annotations or inter\-judge consistencyShen et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib50)\); Min et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib32)\)\. While such criteria assess annotator reliability, they do not directly verify whether rubric scores faithfully reflect task success\. Following recent work on metric validityMaynez et al\. \([2020](https://arxiv.org/html/2607.03162#bib.bib31)\); Yue et al\. \([2023](https://arxiv.org/html/2607.03162#bib.bib73)\), we instead assess our rubrics by examining their alignment with*task\-native success signals*\. Specifically, we use Hit@K as the reference metric family and analyze the plotted Hit@5 alignment between rubric\-based scores and task success across models and agentic workflows \(Figure[6](https://arxiv.org/html/2607.03162#A3.F6)\)\. Our key assumption is that, if subtask\-level rubric scores produced by a Judger LLM exhibit strong correlation with Hit@K within each model or experimental setting, these scores can be treated as supporting diagnostic evidence for diagnosing component\-level reasoning failures\. Beyond correlation strength, we further examine whether rubric scores preserve*relative performance differences*across models and workflows\. Figure[7](https://arxiv.org/html/2607.03162#A3.F7)compares model\-level mean rubric scores against mean Hit@5 from multiple rubric perspectives, showing that models with higher task success consistently achieve stronger rubric scores along corresponding dimensions\. This consistency supports the use of rubric\-based evaluation not only as a diagnostic signal within a model, but also as a meaningful basis for cross\-model and cross\-workflow comparison\. Together, these results support using rubric degradation as diagnostic evidence for task\-level failures, while preserving Hit@K as the primary task\-native outcome\. ### C\.8VQRADetail Section[4\.3](https://arxiv.org/html/2607.03162#S4.SS3)and Figure[3](https://arxiv.org/html/2607.03162#S4.F3)attribute the performance drop onintentqueries primarily to weak query inference\. To empirically verify this hypothesis, we construct a query\-refinement pipeline\. Given anintentquery, which is intentionally under\-specified and weakly aligned with the final purchased product, we refine it into an explicitrefinedquery context using selected user interaction history\. Selected same\-category context follows the intent\-query category\-prior convention in Appendix[C\.3\.2](https://arxiv.org/html/2607.03162#A3.SS3.SSS2)\. Concretely, in the main rewrite setting, we selectk=20k=20product and media interaction records from the selected same\-category history context as evidence\. A shared backbone \(GPT\-4\.1\) rewrites the originalintentquery into a precise and discriminative keyword query, and the rewritten query is then used as the user query for personalized product retrieval\. More than 92% of evaluated sessions have enough same\-category histories for this context\. We additionally evaluate a self\-rewrite variant, where the evaluated backbone rewrites its own query; VQRA\(GPT\-4\.1\) uses GPT\-4\.1 as a shared rewrite module\. Table 24:Self\-rewrite ablation for VQRA onintenthard\-candidate cases\. GPT\-5\.2T denotes GPT\-5\.2Think; both VQRA variants use the same intent\-category history context\.Self\-rewrite improves Hit@1 for all three reported backbones and is comparable to the GPT\-4\.1 rewrite variant, indicating that VQRA’s gains mostly come from how the model utilizes history, not solely due to GPT\-4\.1 acting as an external assistant\. By substituting the originalintentwith its refined counterpart while keeping the retrieval pipeline fixed, this design directly tests whether the model has potential to resolve query ambiguity through pipeline redesign and whether doing so can improve performance\. \(a\)QI vs\. Hit@5 \(b\)PQ vs\. Hit@5 \(c\)RQ vs\. Hit@5 \(d\)PC vs\. Hit@5 \(e\)PR vs\. Hit@5 \(↓\\downarrow\) \(f\)AC vs\. Hit@5 Figure 6:Session\-level alignment between diagnostic rubric scores and task\-native Hit@5\. Panels cover LLM diagnostics \(QI, PQ, RQ, PC, PR\) and ReAct agent coherence \(AC\); PR is a rank metric, so lower values indicate stronger preference salience\.\(a\)Mean QI vs\. mean Hit@5 \(b\)Mean PQ vs\. mean Hit@5 \(c\)Mean PR vs\. mean Hit@5 \(↓\\downarrow\) \(d\)Mean RQ vs\. mean Hit@5 Figure 7:Model\-level rubric–Hit@5 alignment under theintenthard\-candidate setting\. Each point is one model backbone; panels aggregate the diagnostics used in Figure[3](https://arxiv.org/html/2607.03162#S4.F3), showing whether cross\-model task performance tracks rubric aspects\.
Similar Articles
Benchmarking the Personalization Capabilities of Large Language Models
This paper introduces SDR-Bench, a benchmark for evaluating the personalization capabilities of large language models in a two-party Bayesian Persuasion framework, finding a consistent plateau across frontier LLMs and validating the framework with a field deployment.
LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
This paper introduces LUNAR, a benchmark for evaluating how large language models personalize responses from longitudinal app interaction histories across daily-life domains such as clothing, food, housing, and mobility. Experiments on 19 mainstream LLMs reveal that effective personalization depends on evidence selection and cross-domain integration, and that stronger personalization can come at the cost of privacy protection.
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
MCP-Persona is a benchmark evaluating LLM agents on personalized tools interacting with individual accounts and local databases. Experiments reveal significant challenges for state-of-the-art agents in personalized tool use.
Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents
This paper introduces PerMemBench, the first benchmark for evaluating personalized memory systems in LLM-based agents, and proposes a session-level storage gating framework that adapts memory policies to individual user contexts.
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
AdaPlanBench is a dynamic benchmark for evaluating LLM agents' ability to adaptively plan under progressively revealed world and user constraints through multi-turn interactions, showing current models struggle especially with user constraints.