A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing

# A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing
Source: [https://arxiv.org/html/2608.04625](https://arxiv.org/html/2608.04625)
\(2018\)

###### Abstract\.

Industrial recommendation strategy iteration heavily relies on large\-scale A/B experimentation\. Traditional tuning requires experts to repeatedly design strategies, configure experiments, analyze results, and adjust parameters, making the process labor\-intensive and time\-consuming\. Meanwhile, valuable knowledge from historical experiments is often fragmented, making systematic reuse difficult through manual expert effort alone\. Existing RAG agents partially alleviate this burden by retrieving prior strategies, but typically organize experience in a flat manner, overlooking the hierarchical relationships among business scenarios, recommendation stages, optimization objectives, and experimental contexts\. This often results in mismatched retrieval and limited cross\-scenario transfer, while preventing agents from continuously refining strategies and parameters through sequential A/B feedback\. To address these limitations, we proposeA/B Agent, a closed\-loop A/B agent for industrial recommendation strategy optimization\. The framework comprises three tightly coupled core components:Historical Strategy Knowledge Organization,Autonomous Target\-Aware Strategy Generation, andExperiment\-Guided Strategy Self\-Evolution\. It organizes historical strategies into a hierarchical experience tree, retrieves transferable evidence through multi\-path Tree\-RAG to generate executable strategies, and continuously analyzes online A/B feedback to guide autonomous tuning and update the experience tree for self\-evolution\. Extensive offline and online evaluations demonstrate its effectiveness, including a4\.829%improvement in GMV in a real\-world short\-video e\-commerce recommendation system while maintaining positive gains across all guardrail metrics\.

Transfer Learning, Large Language Models, Agent, A/B Test\.

††copyright:acmlicensed††journalyear:2018††doi:XXXXXXX\.XXXXXXX††conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY††ccs:Information systems Recommender systems## 1\.Introduction

In modern industrial recommender systems\(Panet al\.,[2026](https://arxiv.org/html/2608.04625#bib.bib2); Huet al\.,[2026](https://arxiv.org/html/2608.04625#bib.bib3); Jianget al\.,[2026a](https://arxiv.org/html/2608.04625#bib.bib1)\), continuous strategy development\(Penget al\.,[2025](https://arxiv.org/html/2608.04625#bib.bib14)\), large\-scale A/B testing\(Quinet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib15)\), and feedback\-driven parameter optimization\(Hanet al\.,[2019](https://arxiv.org/html/2608.04625#bib.bib13)\)are essential for improving recommendation quality and performance\. A strategy iteration\(Jadon and Patil,[2024](https://arxiv.org/html/2608.04625#bib.bib19); Denget al\.,[2025](https://arxiv.org/html/2608.04625#bib.bib17)\)typically spans objectives, ranking pipelines, scoring functions, parameter configurations, and engineering constraints, followed by small\-scale validation, traffic ramp\-up, and full deployment\. Because multiple business domains share limited exposure, optimizing one objective may suppress others and harm user experience\. Therefore, strategy iteration\(Guptaet al\.,[2019](https://arxiv.org/html/2608.04625#bib.bib25)\)must jointly optimize core metrics and satisfy multiple guardrail constraints\(Piet al\.,[2020](https://arxiv.org/html/2608.04625#bib.bib16)\)to ensure balanced traffic allocation and long\-term system health\.

However, as shown in Figure[1](https://arxiv.org/html/2608.04625#S1.F1)\(a\), existing industrial strategy iteration still relies heavily on human expertise\(Dmitrievet al\.,[2017](https://arxiv.org/html/2608.04625#bib.bib23)\)\. Faced with numerous interdependent—and sometimes conflicting—core and guardrail metrics\(Jeunenet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib21)\), engineers must identify performance gains\(Deng and Shi,[2016](https://arxiv.org/html/2608.04625#bib.bib24)\), assess statistical significance, attribute metric changes\(Xuet al\.,[2018](https://arxiv.org/html/2608.04625#bib.bib26)\), and balance potential risks\. Analyzing a typical experiment and determining the next parameter configuration often requires approximately hours of work by an experienced engineer\(Fabijanet al\.,[2018](https://arxiv.org/html/2608.04625#bib.bib22)\)\. Meanwhile, historical strategy knowledge\(Namakiet al\.,[2020](https://arxiv.org/html/2608.04625#bib.bib29)\)is fragmented across heterogeneous sources\(Schlegel and Sattler,[2023](https://arxiv.org/html/2608.04625#bib.bib27)\), including documents, source code, experiment configurations, and metric logs\. These sources lack a unified abstraction\(Réet al\.,[2019](https://arxiv.org/html/2608.04625#bib.bib28)\)of strategy mechanisms, applicability conditions, and transferability boundaries, making historical knowledge difficult to systematically retrieve and reuse\. Even when a similar historical strategy is identified, differences in user intent, ranking pipelines, feature availability, and business constraints\(Shenget al\.,[2021](https://arxiv.org/html/2608.04625#bib.bib31)\)require its parameters, thresholds, and traffic allocation to be adapted to the target scenario\(Zhanget al\.,[2022](https://arxiv.org/html/2608.04625#bib.bib32); Wanget al\.,[2023](https://arxiv.org/html/2608.04625#bib.bib33)\)and continuously refined through multiple rounds of online experimentation\(Wanget al\.,[2023](https://arxiv.org/html/2608.04625#bib.bib33)\)\.

![Refer to caption](https://arxiv.org/html/2608.04625v1/x1.png)Figure 1\.Motivation of A/B Agent\. Traditional A/B tuning requires repeated manual experimentation, while Other RAG Agent often misses scenario\- and stage\-specific context\. In contrast, A/B Agent leverages hierarchical historical knowledge, autonomous strategy generation, and self\-evolution to optimize strategies effectively\.Recent advances in large language models\(Jianget al\.,[2025a](https://arxiv.org/html/2608.04625#bib.bib4); Ninget al\.,[2025](https://arxiv.org/html/2608.04625#bib.bib50)\)and retrieval\-augmented generation\(Jianget al\.,[2025b](https://arxiv.org/html/2608.04625#bib.bib5); Yuanet al\.,[2025](https://arxiv.org/html/2608.04625#bib.bib7)\)provide new opportunities for understanding and reusing industrial strategy knowledge\. Nevertheless, conventional RAG agents remain insufficient for complex strategy iteration\(Sarthiet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib39); Edgeet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib12)\)\. Flat RAG systems primarily retrieve independent text chunks according to semantic similarity and cannot adequately model the hierarchical relationships among application scenarios, business pipelines, ranking stages, optimization objectives, parameter configurations, and experimental outcomes\(Yoranet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib42); Asaiet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib43)\)\. This limitation restricts reliable knowledge transfer across scenarios and pipelines\(Huanget al\.,[2025a](https://arxiv.org/html/2608.04625#bib.bib40); Taoet al\.,[2025](https://arxiv.org/html/2608.04625#bib.bib41)\)\. Moreover, existing RAG agents\(Shinnet al\.,[2023](https://arxiv.org/html/2608.04625#bib.bib44)\)typically provide only one\-shot strategy recommendations and lack a long\-horizon optimization framework that connects strategy generation, online experimentation, metric analysis, and parameter updates\. Consequently, as shown in Figure[1](https://arxiv.org/html/2608.04625#S1.F1)\(b\), they cannot autonomously refine strategies from continuous feedback or progressively consolidate successful practices and failed experiments into an strategy knowledge base\.

To address the aforementioned challenges,A/B Agentis proposed as an agentic framework for industrial recommendation strategy knowledge construction, strategy initialization, online optimization, and experience self\-evolution\. The framework transforms fragmented historical experiment records into structured and transferable strategy experiences\. Given a new business request, it retrieves relevant historical knowledge under the target context and deployment constraints to initialize an executable strategy\. After deployment, the system continuously analyzes core and guardrail metrics, compares successive experiment variants, and generates the strategy and parameter configuration for the next testing round\. Validated outcomes are subsequently written back to the strategy experience tree, forming a closed loop of experience accumulation, strategy initialization, online tuning, and knowledge evolution\.

The framework consists of three core components,Historical Strategy Knowledge Organization,Autonomous Target\-Aware Strategy Generation, andExperiment\-Guided Strategy Self\-Evolution\. To overcome fragmented experience, as shown in Figure[1](https://arxiv.org/html/2608.04625#S1.F1)\(c\), historical A/B records are decomposed into reusable strategy chunks containing motivations, mechanisms, parameters, applicable contexts, outcomes, and risks, and are organized into a hierarchical experience tree by domain, scenario, recommendation stage, and objective\. To improve the weak knowledge transfer of flat RAG, the initialization module performs multi\-path sparse and dense retrieval, hierarchy\-aware candidate enhancement, and reranking, thereby selecting context\-matched evidence and generating an executable strategy with feasible parameter settings and controlled guardrail risks\. After deployment, the self\-evolution module reduces manual and time\-consuming tuning by organizing successive strategies, parameter updates, and online metrics into an experiment tree\. Parent–child and sibling comparisons identify effective modifications and promising alternatives, enabling autonomous local parameter search or mechanism replacement when performance saturates, or guardrails are violated\. Validated strategies are then written back to the experience tree, forming a closed loop of knowledge organization, target\-aware initialization, autonomous A/B optimization, and experience self\-evolution\.

Extensive offline and online evaluations demonstrate the effectiveness of A/B Agent\. Across three industrial e\-commerce scenarios, A/B Agent achieves the highest average overall score of7\.2447\.244, outperforming Claude\-Sonnet\-4\.6 by1\.3%1\.3\\%\. Compared with the strongest RAG baseline in each scenario, it improves the overall score by25\.0%25\.0\\%,31\.7%31\.7\\%, and23\.5%23\.5\\%, respectively\. It also consistently surpasses GPT\-5\.5 in correctness and Claude\-Sonnet\-4\.6 in novelty\. In real\-world deployment, A/B Agent further achieves a4\.829%improvement in Cart GMV while maintaining positive gains across all guardrail metrics\.

The main contributions of this work are summarized as follows:

- •A/B Agent is introduced as a closed\-loop agent for historical strategy mining, cross\-scenario transfer, A/B analysis, and parameter optimization in industrial recommender systems\.
- •A hierarchical strategy experience tree is constructed to decompose historical strategies into transferable skills and support structured autonomous target\-aware retrieval and strategy generation\.
- •An experiment\-tree\-guided mechanism is developed for A/B tuning and experience self\-evolution, enabling joint metric analysis, parameter optimization, and continuous knowledge updates\.
- •An industrial A/B dataset is built with strategy configurations, experimental outcomes, and multidimensional metrics for evaluating strategy generation and tuning\.
- •Extensive offline and online experiments validate the effectiveness of A/B Agent, including a4\.829%improvement in GMV in real\-world deployment\.

## 2\.Related Work

LLM Agents for Recommendation\.Recent advances in large language models have promoted the development of agentic recommender systems\. Existing methods\(Huanget al\.,[2025b](https://arxiv.org/html/2608.04625#bib.bib45); Wanget al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib46)\)mainly use LLM agents for preference understanding, conversational recommendation, personalized reasoning, recommendation explanation, and user simulation\. By integrating user profiles, memory, planning, and tool use, these agents can better capture user intents and generate personalized recommendations\.Haoet al\.\([2025](https://arxiv.org/html/2608.04625#bib.bib47)\)proposed RecAgent, an LLM\-based generative agent framework that simulates users’ browsing, interaction, and social behaviors in recommendation environments\. However, most existing studies focus on user\-item\-level decisions, such as preference prediction, item generation, and feedback simulation\. Industrial recommendation systems additionally require strategy\-level support for designing ranking strategies, transferring historical experience, analyzing A/B results, and tuning online parameters\. A/B Agent shifts the focus from direct recommendation to industrial strategy iteration by mining historical strategies, retrieving transferable experience, generating strategy suggestions, and refining them through online A/B feedback\.

Agentic Systems for Strategy Optimization\.General LLM\-based agentic systems\(Jianget al\.,[2026b](https://arxiv.org/html/2608.04625#bib.bib6); Wuet al\.,[2026](https://arxiv.org/html/2608.04625#bib.bib8)\)integrate memory, planning, tool use, reflection, and environmental feedback to solve complex tasks\. These frameworks\(Yaoet al\.,[2022](https://arxiv.org/html/2608.04625#bib.bib48)\)enable agents to decompose tasks, interact with external tools, accumulate reusable experience, and iteratively improve their decisions\.Yanget al\.\([2024](https://arxiv.org/html/2608.04625#bib.bib49)\)proposed SWE\-agent, an LLM\-based software engineering agent equipped with a specialized agent–computer interface for repository inspection, code modification, and debugging\. Nevertheless, most existing systems target general tasks such as web navigation, software engineering, embodied control, and interactive problem solving\. They rarely consider recommendation strategy optimization, which requires jointly reasoning over domain knowledge, historical experiments, ranking mechanisms, business objectives, and online metrics\. A/B Agent addresses this gap through a recommendation\-specific closed loop that organizes historical strategies into reusable experiences, retrieves relevant knowledge for new requests, and uses A/B evidence to iteratively refine strategy parameters\.

![Refer to caption](https://arxiv.org/html/2608.04625v1/x2.png)Figure 2\.Overview of A/B Agent\. The framework consists of three components: Historical Strategy Knowledge Organization, Autonomous Target\-Aware Strategy Generation, and Experiment\-Guided Strategy Self\-Evolution\.
## 3\.Methodology

As illustrated in Figure[2](https://arxiv.org/html/2608.04625#S2.F2), A/B Agent consists of three components:Historical Strategy Knowledge Organization,Autonomous Target\-Aware Strategy Generation, andExperiment\-Guided Strategy Self\-Evolution\. Historical experiment records are first transformed into reusable strategy experiences and organized into a hierarchical experience tree\. Given a new optimization request, Tree\-RAG retrieves relevant experiences according to the business context and generates an executable initial strategy\. After deployment, successive A/B experiments are organized into an experiment tree, where strategy and parameter comparisons guide iterative optimization\.

### 3\.1\.Historical Strategy Knowledge Organization

#### Historical Experiment Record Structuring\.

Historical experiment reports are typically organized by projects or experiment batches rather than reusable strategy units\. A single report may contain multiple strategy changes, parameter versions, and intermediate failures, making direct document\-level indexing prone to irrelevant context and weak alignment between modifications and outcomes\. We therefore convert each record into a unified schema covering the business background, scenario, recommendation stage, optimization objective, strategy description, parameter update, core metrics, guardrail metrics, and deployment conditions\. This standardized representation serves as the input for strategy extraction and experience tree construction\.

#### Agentic Multi\-Perspective Strategy Extraction\.

A strategy agent decomposes each experiment record into atomic and self\-contained strategy chunks\. Each chunk corresponds to an independent modification and preserves its motivation, mechanism, parameter configuration, applicability conditions, observed outcomes, and potential risks\. To support diverse query formulations, complementary views are generated from problem, scenario, and mechanism perspectives\. Schema validation, metric consistency checking, and atomicity checking are further applied to prevent unsupported content or unrelated modifications from being merged\.

#### Hierarchical Experience Tree Construction\.

The extracted chunks are organized into a hierarchical strategy experience tree:

\(1\)𝒯E=\(𝒱E,ℰE\),\\mathcal\{T\}\_\{E\}=\\left\(\\mathcal\{V\}\_\{E\},\\mathcal\{E\}\_\{E\}\\right\),where𝒱E\\mathcal\{V\}\_\{E\}contains category and strategy nodes, andℰE\\mathcal\{E\}\_\{E\}represents their hierarchical relations\. Each strategy chunkccis assigned to a semantic path:

\(2\)π​\(c\)=\[dc,sc,lc,oc\],\\pi\(c\)=\[d\_\{c\},\\,s\_\{c\},\\,l\_\{c\},\\,o\_\{c\}\],wheredcd\_\{c\},scs\_\{c\},lcl\_\{c\}, andoco\_\{c\}denote the domain, scenario, recommendation stage, and optimization objective, respectively\.

Each leaf node stores the strategy description, parameter configuration, metric changes, deployment conditions, risks, and references to the original experiment\. Transferable strategies may be attached to multiple compatible paths, enabling both precise retrieval from matched branches and knowledge transfer across related branches\.

![Refer to caption](https://arxiv.org/html/2608.04625v1/x3.png)Figure 3\.End\-to\-end workflow of A/B Agent, including autonomous strategy generation via target\-aware multi\-path retrieval and strategy self\-evolution through online A/B experimentation, feedback\-driven tuning, and continuous experience updates\.

### 3\.2\.Autonomous Target\-Aware Strategy Generation

#### Multi\-Path Strategy Retrieval\.

Given an optimization requestqq, A/B Agent identifies its business scenario, recommendation stage, optimization objective, guardrail requirements, and deployment constraints, and maps them to a semantic path in the strategy experience tree\. Candidate strategy chunks are retrieved through complementary sparse and dense paths\. LetSm​\(c,q\)S\_\{m\}\(c,q\)denote the relevance score between candidate chunkccand requestqqfrom retrieval pathmm\. The initial retrieval score is

\(3\)Sret​\(c,q\)=∑m=1Mαm​Norm⁡\(Sm​\(c,q\)\),S\_\{\\mathrm\{ret\}\}\(c,q\)=\\sum\_\{m=1\}^\{M\}\\alpha\_\{m\}\\operatorname\{Norm\}\\left\(S\_\{m\}\(c,q\)\\right\),whereMMis the number of retrieval paths,αm\\alpha\_\{m\}is the corresponding weight, andNorm⁡\(⋅\)\\operatorname\{Norm\}\(\\cdot\)aligns score ranges across methods\. Sparse retrieval captures exact business terms, metric names, and parameter identifiers, while dense retrieval identifies strategies with similar mechanisms but different expressions\. Their combination balances exact matching and semantic recall\.

#### Context\-Aware Relevance Boosting and Reranking\.

Textual similarity alone may retrieve strategies that are inconsistent with the target business context\. A/B Agent therefore exploits the hierarchy of the experience tree to boost candidates that better match the request in terms of scenario, recommendation stage, and optimization objective\. Letvclv\_\{c\}^\{l\}andvqlv\_\{q\}^\{l\}denote the nodes associated with candidateccand requestqqat hierarchy levelll\. Their tree\-path relevance is defined as

\(4\)Btree​\(c,q\)=∑lβl​exp⁡\(−γl​d𝒯​\(vcl,vql\)\),B\_\{\\mathrm\{tree\}\}\(c,q\)=\\sum\_\{l\}\\beta\_\{l\}\\exp\\left\(\-\\gamma\_\{l\}d\_\{\\mathcal\{T\}\}\\left\(v\_\{c\}^\{l\},v\_\{q\}^\{l\}\\right\)\\right\),whered𝒯​\(⋅,⋅\)d\_\{\\mathcal\{T\}\}\(\\cdot,\\cdot\)denotes the shortest\-path distance between two nodes in the experience tree,βl\\beta\_\{l\}controls the importance of hierarchy levelll, andγl\\gamma\_\{l\}determines the distance\-decay rate\. Structurally matched candidates receive larger boosts, while candidates from nearby branches retain smaller weights to support knowledge transfer across related scenarios or objectives\. The tree\-path relevance is combined with the initial retrieval score to perform coarse ranking, from which the top\-KKcandidates are selected\. Each candidate chunk is then paired with the original request and passed to Qwen\-Reranker\. The reranker performs fine\-grained scoring based on semantic relevance, strategy mechanism, applicability conditions, and deployment constraints, and produces a refined ranking\. The highest\-ranked chunks are retained as supporting evidence for strategy generation\.

#### Evidence\-Grounded Strategy Generation and Judging\.

The strategy generator takes the optimization request, structured business context, and reranked historical experiences as input\. It extracts transferable mechanisms and adapts them to the target task by considering available model inputs, serving costs, valid parameter ranges, and deployment conditions\. The generated candidates are then evaluated by agent\-based judges in terms of contextual relevance, evidence consistency, engineering feasibility, expected business gain, and guardrail risk\. Candidates that lack sufficient evidence, contradict historical outcomes, or violate deployment constraints are filtered out\. The highest\-ranked candidate is converted into an executable initial strategy for A/B testing, including the strategy modification, parameter configuration, supporting evidence, expected gains, and potential risks\.

### 3\.3\.Experiment\-Guided Strategy Self\-Evolution

#### A/B Experiment Understanding and Outcome Extraction\.

After deployment, A/B Agent aligns experiment descriptions, configuration files, and platform metadata into a structured representation containing the business scenario, recommendation stage, optimization objective, strategy configuration, and guardrail requirements\. It further extracts changes in core and guardrail metrics together with experiment duration, traffic allocation, and statistical confidence\. To reduce the influence of noisy or inconclusive results, each metric change is adjusted as

\(5\)Δ​y~j=ρj​Δ​yj,\\widetilde\{\\Delta y\}\_\{j\}=\\rho\_\{j\}\\Delta y\_\{j\},whereΔ​yj\\Delta y\_\{j\}is the observed change of metricjj,ρj∈\[0,1\]\\rho\_\{j\}\\in\[0,1\]is its confidence weight, andΔ​y~j\\widetilde\{\\Delta y\}\_\{j\}is the confidence\-adjusted outcome\.

#### A/B Experiment Tree Construction\.

Successive experiment versions are organized into an A/B experiment tree:

\(6\)𝒯∗A=\(𝒱∗A,ℰ∗A\),\\mathcal\{T\}\*\{A\}=\(\\mathcal\{V\}\*\{A\},\\mathcal\{E\}\*\{A\}\),where𝒱∗A\\mathcal\{V\}\*\{A\}andℰA\\mathcal\{E\}\_\{A\}denote experiment nodes and transition edges\. Each node

\(7\)vt=\(at,𝜽t,𝐲t,xt\)v\_\{t\}=\(a\_\{t\},\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{y\}\_\{t\},x\_\{t\}\)stores the tested strategy, parameter configuration, observed metric changes, and experiment context\. Each directed edge records the strategy and parameter modifications between two versions\. The root represents the production baseline or initial strategy, child nodes represent subsequent modifications, and sibling nodes correspond to alternative configurations from the same parent\.

#### Experiment Comparison and Utility Evaluation\.

For the current experiment, A/B Agent retrieves nodes with similar scenarios, objectives, and strategy mechanisms, and performs both parent–child and sibling comparisons\. Parent–child comparison evaluates the effect of a specific modification, while sibling comparison identifies the most promising configuration\. To jointly consider core gains and guardrail risks, the utility of an experiment node is defined as

\(8\)U​\(v\)=∑j∈ℳcorewj​Δ​y~j−λ​∑k∈ℳguardwk​max⁡\(0,−Δ​y~k\),U\(v\)=\\sum\_\{j\\in\\mathcal\{M\}\_\{\\mathrm\{core\}\}\}w\_\{j\}\\widetilde\{\\Delta y\}\_\{j\}\-\\lambda\\sum\_\{k\\in\\mathcal\{M\}\_\{\\mathrm\{guard\}\}\}w\_\{k\}\\max\\left\(0,\-\\widetilde\{\\Delta y\}\_\{k\}\\right\),whereℳcore\\mathcal\{M\}\_\{\\mathrm\{core\}\}andℳguard\\mathcal\{M\}\_\{\\mathrm\{guard\}\}denote the sets of core and guardrail metrics, respectively;wjw\_\{j\}andwkw\_\{k\}are their importance weights; andλ\\lambdacontrols the penalty for guardrail degradation\. A higherU​\(v\)U\(v\)indicates a better balance between target improvement and guardrail safety\.

#### Iterative Strategy Tuning\.

Based on the current outcomes, related experiment nodes, and historical strategy experiences, A/B Agent recommends the strategy and parameter configuration for the next round\. When a branch consistently improves the target objective without violating guardrails, the agent performs local search around its promising parameter range\. When the branch saturates unacceptable degradation, the agent retrieves alternative mechanisms from the strategy experience tree and creates a new branch\. This process repeats until the gains converge, the experiment budget is exhausted, or guardrail constraints prevent further exploration\.

## 4\.Deployment

Figure[3](https://arxiv.org/html/2608.04625#S3.F3)illustrates the online workflow of A/B Agent, which consists of two main stages:Autonomous Strategy GenerationandStrategy Self\-Evolution\. In the current deployment, GLM\-5\.1 serves as the foundation model for strategy understanding, generation, and iterative optimization\. Given a strategy request, the system first identifies the business scenario, recommendation stage, optimization objective, guardrail constraints, and deployment requirements, and converts them into a structured query\. It then retrieves relevant knowledge from the pre\-organized historical strategy experience tree through Tree\-RAG and generates an executable candidate strategy, including parameter configurations, supporting evidence, expected gains, and potential risks\. Before deployment, rule\-based validators check formatting, numerical ranges, parameter types, and configuration consistency, while agent and human reviewers assess safety, rationality, and engineering feasibility\. Once approved, the strategy is launched on the online A/B testing platform\. During experimentation, A/B Agent periodically collects core and guardrail metrics, updates the experiment tree with new strategy variants and outcomes, and compares the current node with its parent and sibling nodes to identify effective changes in strategies and parameters\. It then performs local optimization around promising configurations or retrieves alternative mechanisms when the current branch saturates or violates guardrail constraints\. The updated strategy is redeployed for subsequent A/B rounds, while validated outcomes are distilled back into the strategy experience tree to support continuous knowledge accumulation and strategy self\-evolution\.

## 5\.Experiments

### 5\.1\.Experimental Setup

Table 1\.Dataset Statistics\.Note:Scenarios 1–3 represent three e\-commerce scenarios deployed at Kuaishou\.

Table 2\.Main experimental results across three recommendation scenarios in Kuaishou E\-commerce\. The best result in each column is highlighted inbold, while the second\-best result isunderlined\.MethodModelScenario 1Scenario 2Scenario 3OverallCorrect\.NoveltyOverallCorrect\.NoveltyOverallCorrect\.NoveltyOverallLLMGLM\-5\.16\.2705\.1305\.9096\.2175\.1305\.8796\.2045\.1075\.8455\.879DeepSeek\-V4\-Pro6\.1575\.6266\.0306\.1095\.6636\.0265\.9135\.7575\.9195\.992Kimi\-K2\.66\.7135\.7306\.4776\.5115\.5336\.2766\.5835\.6416\.3776\.384GPT\-5\.57\.3225\.8966\.8807\.2075\.8046\.7897\.1845\.8066\.7696\.816Claude\-Sonnet\-4\.67\.2526\.8877\.2007\.1636\.8047\.1477\.1266\.8547\.1017\.151RAG AgentVanilla RAG5\.9745\.8095\.7665\.7285\.5655\.5335\.9615\.7285\.7515\.692GAR5\.9225\.7745\.7125\.6305\.5435\.4835\.9905\.7095\.7715\.664HyDE6\.0355\.8265\.8055\.6965\.5875\.5246\.0395\.7775\.8235\.728CRAG5\.8875\.8265\.6875\.5985\.5765\.4506\.0195\.7185\.7995\.654Graph RAG5\.9135\.6435\.7005\.5655\.5655\.4225\.9225\.5735\.7075\.620A/B AgentOurs7\.3487\.1837\.2557\.3707\.2077\.2887\.2337\.1947\.1937\.244

Table 3\.Online A/B testing results throughout the iterative strategy\-tuning process\.Note:Dark and light red backgrounds indicate the best and second\-best results for the core and important metrics, respectively\. Red and green backgrounds indicate positive and negative guardrail changes, respectively\. The final row reports the absolute difference between Strategy 5 and the expert\-designed strategy\.

Dataset\.An industrial strategy generation benchmark is constructed from 310 historical recommendation strategies across three representative scenarios in Kuaishou E\-commerce\. As summarized in Table[1](https://arxiv.org/html/2608.04625#S5.T1), the benchmark covers diverse deployment stages, including retrieval, ranking, blending, and other recommendation components\. Its broad coverage across scenarios and pipeline stages enables a comprehensive evaluation of strategy understanding and generation in heterogeneous industrial settings\.

Baselines\.A/B Agent is compared with two groups of baselines\. The first group includes general\-purpose LLMs: GLM\-5\.1, DeepSeek\-V4\-Pro, Kimi\-K2\.6, GPT\-5\.5, and Claude\-Sonnet\-4\.6, which directly generate strategies from the given scenario and task description\. The second group consists of representative retrieval\-augmented generation methods, including Vanilla RAG, GAR\(Maoet al\.,[2021](https://arxiv.org/html/2608.04625#bib.bib9)\), HyDE\(Gaoet al\.,[2023](https://arxiv.org/html/2608.04625#bib.bib10)\), CRAG\(Yanet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib11)\), and Graph RAG\(Edgeet al\.,[2024](https://arxiv.org/html/2608.04625#bib.bib12)\), which retrieve relevant historical experience before generating the final strategy\. For a controlled comparison, all RAG baselines and A/B Agent use GLM\-5\.1 as the shared foundation model, ensuring that performance differences primarily stem from their knowledge organization, retrieval, and strategy\-generation mechanisms\.

Evaluation Protocol\.For each case, a recommendation strategy is generated according to the target scenario, optimization objective, and available context\. GPT\-5\.5 serves as the automatic evaluator and assesses each strategy along two dimensions:*Correctness*, covering technical feasibility, scenario relevance, and logical consistency; and*Novelty*, measuring the ability to provide meaningful insights beyond direct reuse of historical experience\. An*Overall*score is also reported to reflect holistic strategy quality\.

### 5\.2\.Main Results

Table[2](https://arxiv.org/html/2608.04625#S5.T2)presents the main results across three e\-commerce scenarios\. A/B Agent achieves the best performance in all evaluation dimensions, demonstrating consistent effectiveness across heterogeneous recommendation tasks\. Among the standalone LLMs, GPT\-5\.5 obtains the highest correctness, while Claude\-Sonnet\-4\.6 performs better in novelty and achieves the strongest average overall score of 7\.151, indicating a trade\-off between technical reliability and exploratory strategy generation\. The evaluated RAG agents do not consistently outperform standalone LLMs\. HyDE achieves the highest average overall score among the RAG baselines at 5\.728, followed by Vanilla RAG at 5\.692\. Compared with the strongest RAG method in each scenario, A/B Agent improves the overall score by 25\.0%, 31\.7%, and 23\.5%, respectively\. These improvements indicate that semantic similarity alone is insufficient, since the effectiveness of historical strategies depends strongly on the target scenario, pipeline stage, optimization objective, and deployment constraints\. By explicitly modeling these factors and evaluating strategy transferability, A/B Agent reduces noisy retrieval and enables more effective reuse of industrial experience\. Overall, A/B Agent achieves the highest average overall score of 7\.244, outperforming Claude\-Sonnet\-4\.6 by 1\.3%\. It also surpasses GPT\-5\.5 in correctness by 0\.4%, 2\.3%, and 0\.7%, while improving novelty over Claude\-Sonnet\-4\.6 by 4\.3%, 5\.9%, and 5\.0%, respectively\. These results demonstrate that A/B Agent generates strategies that are both practically reliable and meaningfully differentiated through structured industrial knowledge and transferability\-aware retrieval\.

### 5\.3\.Strategy Evolution

Table[3](https://arxiv.org/html/2608.04625#S5.T3)reports the online A/B testing results throughout the iterative strategy\-tuning process\. The overall trajectory follows an exploration\-and\-refinement pattern, in which the agent first explores the attainable performance frontier and then incorporates online feedback to improve guardrail safety\. Strategy 1 achieves moderate gains across the core and important metrics, but degrades Watch Time, Live GMV, and Platform GMV, indicating an unsatisfactory balance between local business gains and platform\-level stability\. Strategy 2 further increases GMV to\+2\.984%\+2\.984\\%and achieves the highest click improvement of\+1\.167%\+1\.167\\%\. However, all guardrail metrics remain negative, suggesting that aggressive optimization of short\-term engagement and conversion may adversely affect the broader recommendation ecosystem\. Strategy 3 continues the exploration and obtains strong improvements in GMV, GPM, OPM, and orders, reaching\+3\.299%\+3\.299\\%,\+3\.169%\+3\.169\\%,\+2\.320%\+2\.320\\%, and\+2\.526%\+2\.526\\%, respectively\. Most guardrail metrics also recover, although Live GMV remains slightly negative at−0\.067%\-0\.067\\%\. After incorporating these observations, Strategy 4 shifts toward constrained refinement, retaining strong business gains while achieving positive changes across all guardrail metrics\. Strategy 5 further strengthens the primary objectives, producing the largest GMV and GPM gains of\+4\.829%\+4\.829\\%and\+4\.677%\+4\.677\\%, respectively, while maintaining positive guardrail performance\. Compared with the expert\-designed strategy, Strategy 5 improves GMV and GPM by2\.150%2\.150\\%and2\.941%2\.941\\%points and improves Watch Time, Live GMV, and Platform Orders by0\.171%0\.171\\%,1\.448%1\.448\\%, and0\.262%0\.262\\%points, respectively\. Although its OPM, clicks, orders, and Platform GMV are lower, the final strategy achieves substantially stronger primary business gains while avoiding the expert strategy’s negative effects on Watch Time and Live GMV\.

### 5\.4\.Balancing Core Gains and Guardrail Safety

![Refer to caption](https://arxiv.org/html/2608.04625v1/x4.png)Figure 4\.Pareto\-oriented tuning trajectory of A/B Agent across core gains and guardrail safety\.Figure[4](https://arxiv.org/html/2608.04625#S5.F4)illustrates how A/B Agent progressively balances core\-metric gains and guardrail safety through iterative online A/B feedback\. Strategy 1 achieves a moderate GMV improvement of\+1\.123%\+1\.123\\%, but decreases the four guardrail metrics by0\.405%0\.405\\%on average, indicating that the initial configuration still introduces considerable ecosystem\-level risks\. After adjusting the objective weights and decision thresholds, Strategy 2 increases the GMV gain to\+2\.984%\+2\.984\\%and improves the average guardrail change to−0\.332%\-0\.332\\%\. Although it already outperforms the expert\-designed strategy in GMV, the guardrail performance remains negative\. Strategy 3 further strengthens the optimization of the core objectives, increasing GMV to\+3\.299%\+3\.299\\%, while the average guardrail change becomes positive for the first time at\+0\.223%\+0\.223\\%\. Strategy 4 achieves a comparable GMV gain of\+3\.253%\+3\.253\\%and maintains a positive guardrail average of\+0\.202%\+0\.202\\%\. The slight fluctuation between Strategies 3 and 4 reflects an intermediate stability\-validation stage rather than a substantial performance regression\. The final Strategy 5 reaches the strongest overall operating point, improving GMV and GPM by\+4\.829%\+4\.829\\%and\+4\.677%\+4\.677\\%, respectively\. Meanwhile, all four guardrail metrics become positive, with an average improvement of approximately\+0\.342%\+0\.342\\%\. Compared with the expert\-designed strategy, Strategy 5 further improves GMV and GPM by2\.150%2\.150\\%and2\.941%2\.941\\%points, respectively, while also delivering clear improvements in Watch Time, Live GMV, and Platform Orders\. Overall, the trajectory demonstrates that A/B Agent can continuously translate online feedback into effective parameter updates, gradually moving from a region with limited gains and guardrail risks toward a Pareto\-optimal solution that jointly improves business effectiveness and platform safety\.

### 5\.5\.Ablation Study

Table 4\.Ablation study on key designs of A/B Agent\.SettingCorrectnessNoveltyOverallDeclineA/B Agent7\.3487\.1837\.255–A\. Knowledge Basew/o Knowledge Base7\.1047\.0877\.0170\.238↓\\downarrowB\. Retrieval Architecturew/o Strategy\-Tree RAG \(Flat RAG\)7\.2967\.1227\.2400\.015↓\\downarroww/o Multi\-Path Retrieval7\.2007\.1577\.1440\.111↓\\downarrowC\. Retrieval Enhancementw/o Objective\-Aware Boosting7\.2527\.1137\.1850\.070↓\\downarroww/o Scenario\-Aware Boosting7\.2177\.0707\.1330\.122↓\\downarroww/o Re\-ranking Module7\.1747\.0707\.0880\.167↓\\downarrow

Table[4](https://arxiv.org/html/2608.04625#S5.T4)reports the ablation results in scenario 1\. Removing any component reduces performance, confirming that the proposed modules provide complementary benefits\. The knowledge base is the most important component: removing it decreases the overall score by 3\.3% and correctness by 3\.3%, highlighting the value of historical industrial experience for generating feasible strategies\. The re\-ranking module is also critical, with its removal causing a 2\.3% decline in overall performance\. The retrieval enhancements further improve strategy transfer\. Removing scenario\-aware boosting and multi\-path retrieval reduces the overall score by 1\.7% and 1\.5%, respectively, while removing objective\-aware boosting results in a 1\.0% decline\. These results indicate that effective strategy generation requires both broad evidence retrieval and explicit alignment with the target scenario and optimization objective\. Replacing Strategy\-Tree RAG with Flat RAG produces a smaller but consistent decrease of 0\.2%\. Although the margin is modest, it confirms that hierarchical knowledge organization enables more precise retrieval than treating historical strategies as an unstructured document collection\. Overall, the complete A/B Agent achieves the best performance across all metrics, validating the combined effectiveness of its knowledge base, retrieval architecture, boosting mechanisms, and re\-ranking module\.

### 5\.6\.Online A/B Test

Table 5\.Online A/B testing results of A/B Agent\.A/B Agent has been deployed in the e\-commerce short\-video scenario on the Kuaishou platform\. We conducted an online A/B test in a real\-world production environment to evaluate its practical effectiveness\. As shown in Table[5](https://arxiv.org/html/2608.04625#S5.T5), the strategy generated by A/B Agent improves GMV by 4\.829%, together with gains of 4\.677% in GPM, 0\.841% in OPM, and 0\.578% in CVR\. It also increases clicks and orders by 0\.370% and 1\.053%, respectively, demonstrating that A/B Agent can translate historical strategy knowledge into measurable online business improvements\.

## 6\.Conclusion

A/B Agent is presented as a self\-evolving agentic framework for industrial recommendation strategy iteration\. It consists of three tightly coupled components:Historical Strategy Knowledge Organization,Autonomous Target\-Aware Strategy Generation, andExperiment\-Guided Strategy Self\-Evolution\. A/B Agent organizes fragmented historical records into a hierarchical experience tree, retrieves transferable knowledge through multi\-path Tree\-RAG, and continuously refines strategies and parameters using online A/B feedback\. Extensive offline experiments demonstrate consistent improvements over general\-purpose LLMs and conventional RAG\-based agents across multiple recommendation scenarios and pipeline stages\. Real\-world deployment at Kuaishou further yields a 4\.829% increase in GMV, accompanied by consistent gains in GPM, OPM, CVR, clicks, and orders\. These results highlight the effectiveness and efficiency of A/B Agent for industrial strategy optimization\. Future work will focus on stronger requirement alignment, more accurate data and metric attribution, and automated engineering\-feasibility verification to support safer and more reliable strategy evolution\.

## References

- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-rag: learning to retrieve, generate, and critique through self\-reflection\.InInternational conference on learning representations,Vol\.2024,pp\. 9112–9141\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- A\. Deng and X\. Shi \(2016\)Data\-driven metric development for online controlled experiments: seven lessons learned\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 77–86\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- J\. Deng, S\. Wang, K\. Cai, L\. Ren, Q\. Hu, W\. Ding, Q\. Luo, and G\. Zhou \(2025\)Onerec: unifying retrieve and rank with generative recommender and iterative preference alignment\.arXiv preprint arXiv:2502\.18965\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- P\. Dmitriev, S\. Gupta, D\. W\. Kim, and G\. Vaz \(2017\)A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments\.InProceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining,pp\. 1427–1436\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, D\. Metropolitansky, R\. O\. Ness, and J\. Larson \(2024\)From local to global: a graph rag approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1),[§5\.1](https://arxiv.org/html/2608.04625#S5.SS1.p2.1)\.
- A\. Fabijan, P\. Dmitriev, H\. H\. Olsson, and J\. Bosch \(2018\)Effective online controlled experiment analysis at large scale\.In2018 44th Euromicro Conference on Software Engineering and Advanced Applications \(SEAA\),pp\. 64–67\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- L\. Gao, X\. Ma, J\. Lin, and J\. Callan \(2023\)Precise zero\-shot dense retrieval without relevance labels\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1762–1777\.Cited by:[§5\.1](https://arxiv.org/html/2608.04625#S5.SS1.p2.1)\.
- S\. Gupta, R\. Kohavi, D\. Tang, Y\. Xu, R\. Andersen, E\. Bakshy, N\. Cardin, S\. Chandran, N\. Chen, D\. Coey,et al\.\(2019\)Top challenges from the first practical online controlled experiments summit\.ACM SIGKDD Explorations Newsletter21\(1\),pp\. 20–35\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- J\. Han, Y\. Yu, F\. Liu, R\. Tang, and Y\. Zhang \(2019\)Optimizing ranking algorithm in recommender system via deep reinforcement learning\.In2019 international conference on artificial intelligence and advanced manufacturing \(AIAM\),pp\. 22–26\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- C\. Hao, S\. Wang, and K\. Zhou \(2025\)Uncertainty\-aware gui agent: adaptive perception through component recommendation and human\-in\-the\-loop refinement\.arXiv preprint arXiv:2508\.04025\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p1.1)\.
- Z\. Hu, Y\. Chen, Y\. Pan, X\. Yuan, Y\. Yin, D\. Wang, B\. Xia, Z\. Luo, H\. Wang, S\. Ni,et al\.\(2026\)Stop treating collisions equally: qualification\-aware semantic id learning for recommendation at industrial scale\.arXiv preprint arXiv:2603\.00632\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- H\. Huang, Y\. Huang, J\. Yang, Z\. Pan, Y\. Chen, K\. Ma, H\. Chen, and J\. Cheng \(2025a\)Retrieval\-augmented generation with hierarchical knowledge\.arXiv preprint arXiv:2503\.10150\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- X\. Huang, J\. Lian, Y\. Lei, J\. Yao, D\. Lian, and X\. Xie \(2025b\)Recommender ai agent: integrating large language models for interactive recommendations\.ACM Transactions on Information Systems43\(4\),pp\. 1–33\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p1.1)\.
- A\. Jadon and A\. Patil \(2024\)A comprehensive survey of evaluation techniques for recommendation systems\.InInternational Conference on Computation of Artificial Intelligence & Machine Learning,pp\. 281–304\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- O\. Jeunen, S\. Baweja, N\. Pokharna, and A\. Ustimenko \(2024\)Powerful a/b\-testing metrics and where to find them\.InProceedings of the 18th ACM Conference on Recommender Systems,pp\. 816–818\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- Z\. Jiang, Y\. Chen, S\. Wang, H\. Qu, Z\. Jindong, W\. Fan, L\. Qing, D\. Liang, and J\. Wang \(2026a\)Atomic intent reasoning: bringing llm semantics to industrial cross\-domain recommendations\.arXiv preprint arXiv:2606\.10357\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- Z\. Jiang, P\. Wu, Z\. Liang, P\. Q\. Chen, X\. Yuan, Y\. Jia, J\. Tu, C\. Li, P\. H\. Ng, and Q\. Li \(2025a\)Hibench: benchmarking llms capability on hierarchical structure reasoning\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5505–5515\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- Z\. Jiang, P\. Wu, X\. Yuan, W\. Fan, and Q\. Li \(2025b\)QA\-dragon: query\-aware dynamic rag system for knowledge\-intensive visual question answering\.arXiv preprint arXiv:2508\.05197\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- Z\. Jiang, X\. Yuan, H\. Qu, S\. Lin, K\. Liu, W\. Fan, and L\. Qing \(2026b\)SUPERGLASSES: benchmarking vision language models as intelligent agents for ai smart glasses\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 2165–2175\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p2.1)\.
- Y\. Mao, P\. He, X\. Liu, Y\. Shen, J\. Gao, J\. Han, and W\. Chen \(2021\)Generation\-augmented retrieval for open\-domain question answering\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 4089–4100\.Cited by:[§5\.1](https://arxiv.org/html/2608.04625#S5.SS1.p2.1)\.
- M\. H\. Namaki, A\. Floratou, F\. Psallidas, S\. Krishnan, A\. Agrawal, Y\. Wu, Y\. Zhu, and M\. Weimer \(2020\)Vamsa: automated provenance tracking in data science scripts\.InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 1542–1551\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- L\. Ning, Z\. Liang, Z\. Jiang, H\. Qu, Y\. Ding, W\. Fan, X\. Wei, S\. Lin, H\. Liu, P\. S\. Yu,et al\.\(2025\)A survey of webagents: towards next\-generation ai agents for web automation with large foundation models\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 6140–6150\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- Y\. Pan, Y\. Chen, Z\. Hu, X\. Yuan, D\. Wang, Y\. Yin, S\. Ni, H\. Wang, J\. Wang, F\. Ren,et al\.\(2026\)Beyond static collision handling: adaptive semantic id learning for multimodal recommendation at industrial scale\.arXiv preprint arXiv:2604\.23522\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- H\. Peng, X\. Jin, Q\. Huang, and S\. Liu \(2025\)E\-commerce intelligent recommendation optimization and personalized marketing strategy based on big model\.InProceedings of the 2025 International Conference on Information Economy, Data Modeling and Cloud Computing,pp\. 313–317\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- Q\. Pi, G\. Zhou, Y\. Zhang, Z\. Wang, L\. Ren, Y\. Fan, X\. Zhu, and K\. Gai \(2020\)Search\-based user interest modeling with lifelong sequential behavior data for click\-through rate prediction\.InProceedings of the 29th ACM International Conference on Information & Knowledge Management,pp\. 2685–2692\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- F\. Quin, D\. Weyns, M\. Galster, and C\. C\. Silva \(2024\)A/b testing: a systematic literature review\.Journal of Systems and Software211,pp\. 112011\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p1.1)\.
- C\. Ré, F\. Niu, P\. Gudipati, and C\. Srisuwananukorn \(2019\)Overton: a data system for monitoring and improving machine\-learned products\.arXiv preprint arXiv:1909\.05372\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- P\. Sarthi, S\. Abdullah, A\. Tuli, S\. Khanna, A\. Goldie, and C\. Manning \(2024\)Raptor: recursive abstractive processing for tree\-organized retrieval\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 32628–32649\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- M\. Schlegel and K\. Sattler \(2023\)Management of machine learning lifecycle artifacts: a survey\.ACM SIGMOD Record51\(4\),pp\. 18–35\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- X\. Sheng, L\. Zhao, G\. Zhou, X\. Ding, B\. Dai, Q\. Luo, S\. Yang, J\. Lv, C\. Zhang, H\. Deng,et al\.\(2021\)One model to serve all: star topology adaptive recommender for multi\-domain ctr prediction\.InProceedings of the 30th ACM International Conference on Information & Knowledge Management,pp\. 4104–4113\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- W\. Tao, X\. Xing, Y\. Chen, L\. Huang, and X\. Xu \(2025\)Treerag: unleashing the power of hierarchical storage for enhanced knowledge retrieval in long documents\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 356–371\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- Y\. Wang, Z\. Jiang, Z\. Chen, F\. Yang, Y\. Zhou, E\. Cho, X\. Fan, Y\. Lu, X\. Huang, and Y\. Yang \(2024\)Recmind: large language model powered agent for recommendation\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 4351–4364\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p1.1)\.
- Y\. Wang, X\. Zhao, B\. Chen, Q\. Liu, H\. Guo, H\. Liu, Y\. Wang, R\. Zhang, and R\. Tang \(2023\)PLATE: a prompt\-enhanced paradigm for multi\-scenario recommendations\.InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1498–1507\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- P\. Wu, P\. Q\. Chen, X\. Li, W\. Fan, and Q\. Li \(2026\)Datamart\-agent: llm\-driven game\-theoretic agent for data marketplace modeling\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 32509–32531\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p2.1)\.
- Y\. Xu, W\. Duan, and S\. Huang \(2018\)SQR: balancing speed, quality and risk in online experiments\.InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 895–904\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.
- S\. Yan, J\. Gu, Y\. Zhu, and Z\. Ling \(2024\)Corrective retrieval augmented generation\.Cited by:[§5\.1](https://arxiv.org/html/2608.04625#S5.SS1.p2.1)\.
- J\. Yang, C\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)Swe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p2.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§2](https://arxiv.org/html/2608.04625#S2.p2.1)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2024\)Making retrieval\-augmented language models robust to irrelevant context\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 29862–29883\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- X\. Yuan, L\. Ning, W\. Fan, and Q\. Li \(2025\)MKG\-rag: multimodal knowledge graph\-enhanced rag for visual question answering\.arXiv preprint arXiv:2508\.05318\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p3.1)\.
- Y\. Zhang, X\. Wang, J\. Hu, K\. Gao, C\. Lei, and F\. Fang \(2022\)Scenario\-adaptive and self\-supervised model for multi\-scenario personalized recommendation\.InProceedings of the 31st ACM International Conference on Information & Knowledge Management,pp\. 3674–3683\.Cited by:[§1](https://arxiv.org/html/2608.04625#S1.p2.1)\.

## Appendix AFurther Analysis

### A\.1\.Stage Analysis

Table 6\.Performance comparison across different recommendation stages\.Table[6](https://arxiv.org/html/2608.04625#A1.T6)compares different methods across four recommendation stages\. A/B Agent achieves the best performance in retrieval, ranking, blending, and other stages, with scores of 7\.269, 7\.096, 7\.280, and 7\.302, respectively\. Compared with the strongest baseline, Claude\-Sonnet\-4\.6, it improves the scores by 0\.118, 0\.043, 0\.080, and 0\.108 points, demonstrating robust effectiveness across heterogeneous components of the recommendation pipeline\. Among standalone LLMs, Claude\-Sonnet\-4\.6 performs best, followed by GPT\-5\.5, while the RAG\-based methods generally lag behind\. This indicates that directly retrieving historical records is insufficient when the knowledge is not explicitly organized by stage, scenario, and optimization objective\. Although ranking remains the most challenging stage due to its multi\-objective and tightly coupled design, A/B Agent still outperforms all baselines\. Overall, the results validate the effectiveness of its stage\-aware knowledge organization and retrieval mechanism\.

### A\.2\.Error Analysis

Figure[5](https://arxiv.org/html/2608.04625#A1.F5)presents the distribution of failure cases produced by A/B Agent\. The dominant error category is*solution mismatch*, accounting for 43\.1% of all failures\. These errors arise when a generated strategy is generally reasonable but does not fully align with the target scenario, optimization objective, pipeline stage, or deployment constraints\. This result indicates that fine\-grained strategy alignment remains the primary challenge for A/B Agent\.

Data\-, label\-, and attribution\-related errors constitute the second\-largest category at 29\.2%\. Such errors typically involve inaccurate assumptions about feature definitions, training labels, metric attribution, or the availability of historical data\. Engineering and serving issues account for 19\.4%, mainly reflecting incompatibilities with existing pipelines, unavailable features, or excessive deployment overhead\. Evaluation and specification errors represent only 8\.3% of the failures, suggesting that the task specification and evaluation process are comparatively reliable\.

Overall, the remaining errors are primarily associated with scenario\-specific understanding and practical constraint alignment rather than general strategy generation\. Further improvements should therefore focus on objective\-aware knowledge retrieval, explicit verification of data dependencies, and constraint\-aware evaluation before online experimentation\.

![Refer to caption](https://arxiv.org/html/2608.04625v1/x5.png)Figure 5\.Distribution of error categories for A/B Agent\.