HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

# HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
Source: [https://arxiv.org/html/2607.24779](https://arxiv.org/html/2607.24779)
\(2026\)

###### Abstract\.

Online advertising bidding systems typically deploy multiple offline\-trained expert models \(e\.g\., PID controllers, model predictive control, offline RL policies\) but face two critical limitations: lack of online adaptability to non\-stationary auction markets, and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints\. We propose HOBA \(Hierarchical On\-policy Bidding Agents\), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales\. At the high level, a large language model infers hyperparameters from contextual signals through a Think\-Act\-Observe\-Reflect loop with historical experience retrieval\. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias\. At the low level, a dynamic expert pool \(PID, MPC, IQL, Decision Transformer\) executes bids under high\-level constraints\. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability\. Experiments on the AuctionNet benchmark and a large\-scale A/B test demonstrate consistent improvements over state\-of\-the\-art baselines\. In a large\-scale online deployment, HOBA delivered substantial business value, achieving a \+3\.6% increase in target cost, proving the effectiveness of our hierarchical multi\-agent bidding paradigm\.

Online Advertising, Auto\-bidding, Reinforcement Learning, Agent, LLM

††journalyear:2026††copyright:cc††conference:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2; August 09–13, 2026; Jeju Island, Republic of Korea††booktitle:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2 \(KDD ’26\), August 09–13, 2026, Jeju Island, Republic of Korea††doi:10\.1145/3770855\.3818435††isbn:979\-8\-4007\-2259\-2/2026/08††ccs:Information systems Online advertising## 1\.Introduction

Online advertising auctions constitute a multi\-billion dollar market where advertisers compete in real\-time to maximize objectives \(conversions, revenue\) subject to budget and cost\-per\-acquisition \(CPA\) constraints\. A core challenge isautomatic biddingin non\-stationary markets where competitor strategies, user behavior, and platform dynamics continuously evolve\(Huet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib147)\), necessitating systems that adapt online while maintaining safety\.

Current industry practice deploys offline\-trained expert models—PID controllers\(Zhanget al\.,[2014](https://arxiv.org/html/2607.24779#bib.bib60)\), model predictive control \(MPC\)\(Hanet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib61)\), and offline RL policies\(Kumaret al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib62); Kostrikovet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib63); Chenet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib92)\)—as bidding controllers\. However, these systems face two critical limitations: \(1\)lack of online adaptability, as expert parameters remain fixed post\-deployment and cannot respond to market shifts without costly retraining, and \(2\)manual hyperparameter tuningof bid bounds, pacing rates, and exploration coefficients per campaign\. Enabling safe online learning remains challenging: on\-policy methods \(PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.24779#bib.bib81)\), A3C\(Mnihet al\.,[2016](https://arxiv.org/html/2607.24779#bib.bib82)\)\) risk catastrophic exploration that can exhaust budgets within minutes, while off\-policy methods suffer from action collapse and fail to generalize across diverse conditions\. Moreover, existing approaches conflate strategic planning with tactical execution, treating bidding as a monolithic optimization problem that lacks interpretability\.

We proposeHOBA \(Hierarchical On\-policy Bidding Agents\), a framework that decomposes bidding into three tiers: \(1\) ahigh\-level LLM agent\(1\-hour cycle\) infers hyperparameters𝜽t\\boldsymbol\{\\theta\}\_\{t\}through Think\-Act\-Observe\-Reflect with experience retrieval, \(2\) amid\-level SARSA agent\(2\-minute cycle\) selects experts using causal adjustment to debias historical data, and \(3\) alow\-level expert pool\(per\-auction\) executes bids under high\-level constraints\. We treat each layer as an autonomous agent with its own observation space, decision cycle, and optimization objective, forming a cooperative multi\-agent system under a shared revenue constraint\. This design confines online learning todiscrete expert selection\(5–10 validated models\) rather than continuous bid optimization, achieving safe adaptation without unconstrained exploration\.

Our contributions are:\(1\)We propose the first hierarchical multi\-agent bidding framework decoupling strategic reasoning, expert selection, and bid execution across time scales\.\(2\)We introduce LLM\-guided hyperparameter optimization with natural language reasoning and causal\-adjusted on\-policy expert selection for safe online adaptation\.\(3\)We demonstrate \+3\.6% cost target improvement in large\-scale A/B tests, validating practical viability\.

## 2\.Related Work

### 2\.1\.Automatic Bidding in Online Advertising

Automatic bidding systems aim to optimize advertiser objectives under budget and cost constraints\. Early approaches employed rule\-based controllers:PID controllers\(Zhanget al\.,[2014](https://arxiv.org/html/2607.24779#bib.bib60)\)adjust bid multipliers via proportional\-integral\-derivative feedback to regulate CPA deviation, whilemodel predictive control \(MPC\)\(Hanet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib61)\)solves constrained optimization over future horizons to balance conversions and budget consumption\. These methods offer interpretability and fast deployment but require extensive manual tuning and struggle to adapt to non\-stationary markets\.

Recent work has increasingly adoptedreinforcement learningfor bidding\.Caiet al\.\([2017](https://arxiv.org/html/2607.24779#bib.bib10)\)formulate bidding as an MDP and apply Q\-learning to optimize real\-time display advertising\.Hanet al\.\([2020](https://arxiv.org/html/2607.24779#bib.bib61)\)introduce a generalized framework optimizing diverse KPIs via constrained policy gradient methods\. However, these on\-policy approaches face safety risks in production deployment—unconstrained exploration can exhaust budgets or violate cost targets within minutes, causing advertiser churn\.

To enable safe learning, practitioners have turned tooffline RL\(Levineet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib96)\), training policies on logged data without online exploration\. Recent work has applied Conservative Q\-Learning \(CQL\)\(Kumaret al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib62)\)to mitigate overestimation in bidding, employed Implicit Q\-Learning \(IQL\)\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib63)\)for robust value estimation under reward noise, and leveraged Decision Transformer \(DT\)\(Chenet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib92)\)to model sequential dependencies via supervised sequence modeling\. While these methods avoid risky exploration, they suffer from action collapse—policies fail to generalize across diverse market conditions due to distributional shift between offline data and online deployment\(Fuet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib102)\)\.

Our work differs fundamentally bydecoupling expert training from online adaptation: we maintain a pool of diverse offline\-trained experts and confine online learning to discrete expert selection rather than continuous bid optimization, achieving safety without sacrificing adaptability\.

### 2\.2\.Hierarchical Reinforcement Learning

Hierarchical RL decomposes complex tasks into multiple levels of abstraction, enabling efficient learning and transfer\(Barto and Mahadevan,[2003](https://arxiv.org/html/2607.24779#bib.bib64); Pateriaet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib65)\)\.Options framework\(Suttonet al\.,[1999](https://arxiv.org/html/2607.24779#bib.bib66)\)extends MDPs with temporally extended actions, whilefeudal RL\(Dayan and Hinton,[1992](https://arxiv.org/html/2607.24779#bib.bib67); Vezhnevetset al\.,[2017](https://arxiv.org/html/2607.24779#bib.bib68)\)separates managers \(setting subgoals\) from workers \(achieving subgoals\)\.Nachumet al\.\([2018](https://arxiv.org/html/2607.24779#bib.bib100)\)propose hierarchical off\-policy learning with goal\-conditioned value functions\. In robotics,Osaet al\.\([2020](https://arxiv.org/html/2607.24779#bib.bib101)\)survey hierarchical methods for manipulation tasks requiring coordination across motion primitives\.

In online advertising, prior work has applied hierarchical methods mainly to budget allocation across campaigns or coordination among multiple advertisers\(Yuanet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib69)\), treating individual bidders as monolithic policies\. In contrast, HOBA decomposes a*single*advertiser’s bidding system into three hierarchical levels: strategic reasoning \(LLM for hyperparameter inference\), tactical selection \(SARSA for expert model choice\), and operational execution \(expert pool for bid generation\)\. This time\-scale separation confines online learning to discrete model selection rather than continuous bid optimization, significantly reducing exploration risk\.

### 2\.3\.LLM\-Based Decision Making and Agent Systems

Large language models have emerged as powerful tools for sequential decision\-making through natural language reasoning\(Huanget al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib70); Yaoet al\.,[2023](https://arxiv.org/html/2607.24779#bib.bib71)\)\. Methods like ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2607.24779#bib.bib71)\)combine reasoning with action execution, while Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.24779#bib.bib72)\)enables self\-improvement through experience replay\. Recent work has applied LLMs to hyperparameter tuning\(Yanget al\.,[2024](https://arxiv.org/html/2607.24779#bib.bib73)\)and AutoML configuration\(Chenet al\.,[2023](https://arxiv.org/html/2607.24779#bib.bib74)\), demonstrating superior sample efficiency in offline settings\. However, these approaches focus on simulated environments or offline optimization\. HOBA is the first to deploy LLM\-guided reasoning foronline hyperparameter inferencein a production advertising system, where a Think\-Act\-Observe\-Reflect loop with historical experience retrieval enables continual adaptation to non\-stationary markets without per\-auction LLM inference costs\.

### 2\.4\.Causal Inference in Reinforcement Learning

Off\-policy learning from logged data faces confounding bias when certain actions are preferentially chosen in favorable contexts\(Liet al\.,[2015](https://arxiv.org/html/2607.24779#bib.bib75); Swaminathan and Joachims,[2015](https://arxiv.org/html/2607.24779#bib.bib76)\)\. Inverse propensity scoring \(IPS\)\(Horvitz and Thompson,[1952](https://arxiv.org/html/2607.24779#bib.bib77)\)reweights samples by logging policy probabilities but suffers from high variance, while doubly robust estimation\(Bang and Robins,[2005](https://arxiv.org/html/2607.24779#bib.bib78); Dudíket al\.,[2014](https://arxiv.org/html/2607.24779#bib.bib79)\)combines model\-based prediction with importance weighting for improved robustness\. In advertising, prior work has applied these methods to click\-through rate prediction\(Liet al\.,[2010](https://arxiv.org/html/2607.24779#bib.bib97)\)and ranking evaluation\(Gilotteet al\.,[2018](https://arxiv.org/html/2607.24779#bib.bib98)\)\. Our mid\-level SARSA agent incorporates doubly robust causal adjustment \(Eq\.[12](https://arxiv.org/html/2607.24779#S4.E12)\) to debias expert selection, ensuring Q\-learning does not inherit spurious correlations from historical data where expert models were deployed under non\-random policies\.

## 3\.Preliminary

We formulate automatic bidding as a sequential decision problem where an advertiser aims to maximize conversion value subject to budget and cost constraints over a campaign horizon\.

### 3\.1\.Auto\-Bidding as Constrained Optimization

For advertiseriiparticipating in auctionsj∈\{1,…,J\}j\\in\\\{1,\\ldots,J\\\}, the bidding problem is formulated as\(Hanet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib61)\):

\(1\)maxbi​∑j𝒜i​j​\(bj\)⋅vi​js\.t\.\\displaystyle\\max\_\{b\_\{i\}\}\\sum\_\{j\}\\mathcal\{A\}\_\{ij\}\(b\_\{j\}\)\\cdot v\_\{ij\}\\quad\\text\{s\.t\.\}∑j𝒜i​j​\(bj\)⋅𝒫i​j≤Bi,\\displaystyle\\sum\_\{j\}\\mathcal\{A\}\_\{ij\}\(b\_\{j\}\)\\cdot\\mathcal\{P\}\_\{ij\}\\leq B\_\{i\},∑j𝒜i​j​\(bj\)⋅𝒫i​j∑j𝒜i​j​\(bj\)⋅vi​j≤Ci,\\displaystyle\\frac\{\\sum\_\{j\}\\mathcal\{A\}\_\{ij\}\(b\_\{j\}\)\\cdot\\mathcal\{P\}\_\{ij\}\}\{\\sum\_\{j\}\\mathcal\{A\}\_\{ij\}\(b\_\{j\}\)\\cdot v\_\{ij\}\}\\leq C\_\{i\},wherebjb\_\{j\}denotes bids from all advertisers,vi​j∈ℝ\+v\_\{ij\}\\in\\mathbb\{R\}\_\{\+\}is conversion value,𝒜i​j​\(bj\)∈\{0,1\}\\mathcal\{A\}\_\{ij\}\(b\_\{j\}\)\\in\\\{0,1\\\}indicates whether advertiseriiwins auctionjj, and𝒫i​j\\mathcal\{P\}\_\{ij\}is payment upon winning\. The constraints enforce a budget limitBiB\_\{i\}and maximum cost\-per\-value ratioCiC\_\{i\}\(equivalently, minimum return\-on\-ad\-spend\)\.

### 3\.2\.Challenges in Direct Policy Learning

Standard RL approaches face fundamental difficulties in online advertising:

Safety\-Adaptability Tradeoff\.On\-policy methods \(e\.g\., PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.24779#bib.bib81)\), A3C\(Mnihet al\.,[2016](https://arxiv.org/html/2607.24779#bib.bib82)\)\) require online exploration but risk catastrophic failures—a single poorly\-informed bid can exhaust the daily budget within minutes\. Off\-policy methods \(e\.g\., CQL\(Kumaret al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib62)\), IQL\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib63)\)\) train safely on logged data but suffer from distributional shift: policies fail when market conditions \(competitor behavior, traffic patterns\) deviate from historical data\(Levineet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib96)\)\.

Hyperparameter Sensitivity\.Performance depends critically on hyperparameters such as bid bounds, budget pacing rates, and exploration bonuses\. Manual tuning is labor\-intensive and campaign\-specific, while automated methods like Bayesian optimization\(Snoeket al\.,[2012](https://arxiv.org/html/2607.24779#bib.bib95)\)require hundreds of evaluations—infeasible when each evaluation involves deploying a policy for an entire campaign day\.

Interpretability Requirements\.Production systems demand transparency: advertisers need to understand bidding decisions, and platform operators must audit constraint compliance\. End\-to\-end neural policies provide limited interpretability, complicating debugging and trust\.

### 3\.3\.Hierarchical Decomposition Rationale

HOBA addresses these challenges through hierarchical decomposition across three time scales:

- •Strategic Layer \(hourly\): An LLM infers hyperparameters from campaign context via natural language reasoning, separatingwhat constraints to imposefromhow to bid\.
- •Tactical Layer \(minute\-scale\): A SARSA agent selects among pre\-validated expert models\. By learning over discrete expert choices \(\|𝒜\|≈5\|\\mathcal\{A\}\|\\approx 5\-1010\) rather than continuous bid values, exploration risk is dramatically reduced\.
- •Operational Layer \(per\-auction\): Offline\-trained experts execute bids under high\-level constraints, enabling fast inference without online policy updates\.

This design achieves: \(1\)Safetyvia bounded exploration, \(2\)Adaptabilityvia online expert selection responding to market shifts, and \(3\)Interpretabilityvia LLM reasoning traces and causal attribution\. Formal details follow in Section[4](https://arxiv.org/html/2607.24779#S4)\.

## 4\.Methodology

HOBA consists of three hierarchical tiers operating at different time scales \(Figure[1](https://arxiv.org/html/2607.24779#S4.F1)\): \(1\) a high\-level LLM agent making strategic decisions hourly, \(2\) a mid\-level SARSA agent selecting expert models every 2 minutes, and \(3\) a low\-level expert pool executing bids per auction\.This decomposition confines online learning to discrete expert selection rather than continuous bid optimization, reducing exploration risk while maintaining adaptability\.Global Objective Alignment\.All three layers optimize the single global objective in Eq\. \(1\)\. The LLM parameterizes the feasible constraint set; SARSA selects the best expert within that region; and experts enforce constraints by construction via double\-clipping \(Eq\. \(15\)\), guaranteeing every bid satisfies Eq\. \(1\)\. The penaltiesμ\\muandη\\etain Eq\. \(11\) are instrumental: switching instability causes budget waste \(violatingBiB\_\{i\}\), and pacing deviation causes CPA violations \(violatingCiC\_\{i\}\)\.

![Refer to caption](https://arxiv.org/html/2607.24779v1/x1.png)Figure 1\.Overview of HOBA’s three\-tier architecture\. High\-level LLM agent \(hour cycle\) generates hyperparameters𝜽t\\boldsymbol\{\\theta\}\_\{t\}via Think\-Act\-Observe\-Reflect; Mid\-level SARSA agent \(minute cycle\) selects expert modelsaτa\_\{\\tau\}with causal adjustment; Low\-level expert pool \(per\-auction\) executes bidsbτb\_\{\\tau\}with fixed policies\. Online learning is confined to mid\-level selection, ensuring safe adaptation\.### 4\.1\.High\-Level: LLM\-Guided Hyperparameter Optimization

The high\-level agent indirectly shapes the MDP of the mid\-level agent by dynamically constraining the feasible action space and reward scale\.The LLM agent infers optimal hyperparameters from campaign context through a Think\-Act\-Observe\-Reflect loop\(Yaoet al\.,[2023](https://arxiv.org/html/2607.24779#bib.bib71); Shinnet al\.,[2023](https://arxiv.org/html/2607.24779#bib.bib72)\), operating on a 1\-hour cycle to reduce exploration risk while enabling strategic adaptation\.

#### 4\.1\.1\.State Representation and Decision Loop

At timett, we construct high\-level state capturing budget status \(𝐛t\\mathbf\{b\}\_\{t\}\), performance metrics \(𝐫t\\mathbf\{r\}\_\{t\}\), market conditions \(𝐦t\\mathbf\{m\}\_\{t\}\), placement performance \(𝐩t\\mathbf\{p\}\_\{t\}\), and historical statistics \(𝐡t\\mathbf\{h\}\_\{t\}\):

\(2\)𝐬tH=\[𝐛t;𝐫t;𝐦t;𝐩t;𝐡t\]\.\\mathbf\{s\}\_\{t\}^\{H\}=\[\\mathbf\{b\}\_\{t\};\\mathbf\{r\}\_\{t\};\\mathbf\{m\}\_\{t\};\\mathbf\{p\}\_\{t\};\\mathbf\{h\}\_\{t\}\]\.This numeric vector is transformed into natural language via template𝒫state​\(𝐬tH\)\\mathcal\{P\}\_\{\\text\{state\}\}\(\\mathbf\{s\}\_\{t\}^\{H\}\)for LLM processing\.

Think Phase\.The LLM retrieves similar historical experiences from memory bank𝒟mem\\mathcal\{D\}\_\{\\text\{mem\}\}and performs structured reasoning:

\(3\)𝐚tthink=LLM​\(𝒫think​\(𝐬tH,ℳt\)\),\\mathbf\{a\}^\{\\text\{think\}\}\_\{t\}=\\text\{LLM\}\(\\mathcal\{P\}\_\{\\text\{think\}\}\(\\mathbf\{s\}\_\{t\}^\{H\},\\mathcal\{M\}\_\{t\}\)\),whereℳt\\mathcal\{M\}\_\{t\}contains top\-kkretrieved experiences based on semantic similarity, producing problem identification and root cause analysis\.

Act Phase\.Based on analysis, the LLM generates hyperparameters:

\(4\)𝜽t=LLM​\(𝒫act​\(𝐬tH,𝐚tthink\)\),\\boldsymbol\{\\theta\}\_\{t\}=\\text\{LLM\}\(\\mathcal\{P\}\_\{\\text\{act\}\}\(\\mathbf\{s\}\_\{t\}^\{H\},\\mathbf\{a\}^\{\\text\{think\}\}\_\{t\}\)\),where𝜽t=\{θtbounds,θtmode,θtcontrol,θtexplore\}\\boldsymbol\{\\theta\}\_\{t\}=\\\{\\theta^\{\\text\{bounds\}\}\_\{t\},\\theta^\{\\text\{mode\}\}\_\{t\},\\theta^\{\\text\{control\}\}\_\{t\},\\theta^\{\\text\{explore\}\}\_\{t\}\\\}includes bid bounds\[qlower,qupper\]\[q\_\{\\text\{lower\}\},q\_\{\\text\{upper\}\}\], response mode \(conservative/moderate/aggressive\), control targets \(CPA, budget pacing rateρ\\rho, bid change limits𝚫\\boldsymbol\{\\Delta\}\), and exploration bonus\. Safety is enforced via:

\(5\)𝜽t←Clip​\(𝜽t,𝜽min,𝜽max\)\.\\boldsymbol\{\\theta\}\_\{t\}\\leftarrow\\text\{Clip\}\(\\boldsymbol\{\\theta\}\_\{t\},\\boldsymbol\{\\theta\}\_\{\\min\},\\boldsymbol\{\\theta\}\_\{\\max\}\)\.
Observe\-Reflect\.After one\-hour execution, we aggregate observations and compute deviation from predictions:

\(6\)𝐨t=Aggregate​\(\{𝐱τ\}τ∈𝒯t\),𝜹t=\{δCPA,δconv,δbudget\}\.\\mathbf\{o\}\_\{t\}=\\text\{Aggregate\}\(\\\{\\mathbf\{x\}\_\{\\tau\}\\\}\_\{\\tau\\in\\mathcal\{T\}\_\{t\}\}\),\\quad\\boldsymbol\{\\delta\}\_\{t\}=\\\{\\delta\_\{\\text\{CPA\}\},\\delta\_\{\\text\{conv\}\},\\delta\_\{\\text\{budget\}\}\\\}\.The LLM evaluates decision quality:

\(7\)𝐫treflect=LLM​\(𝒫reflect​\(𝐚tthink,𝜽t,𝐨t,𝜹t\)\),\\mathbf\{r\}^\{\\text\{reflect\}\}\_\{t\}=\\text\{LLM\}\(\\mathcal\{P\}\_\{\\text\{reflect\}\}\(\\mathbf\{a\}^\{\\text\{think\}\}\_\{t\},\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{o\}\_\{t\},\\boldsymbol\{\\delta\}\_\{t\}\)\),producing prediction accuracysaccs\_\{\\text\{acc\}\}, overall quality score:

\(8\)squality=w1​sobjective\+w2​sacc\+w3​srisk\+w4​sefficiency,s\_\{\\text\{quality\}\}=w\_\{1\}s\_\{\\text\{objective\}\}\+w\_\{2\}s\_\{\\text\{acc\}\}\+w\_\{3\}s\_\{\\text\{risk\}\}\+w\_\{4\}s\_\{\\text\{efficiency\}\},and structured lessons\. This experience is stored in𝒟mem\\mathcal\{D\}\_\{\\text\{mem\}\}as:

\(9\)ei=\(𝐬iH,𝜽i,𝐨i,𝐫ireflect,𝐳i\),e\_\{i\}=\(\\mathbf\{s\}\_\{i\}^\{H\},\\boldsymbol\{\\theta\}\_\{i\},\\mathbf\{o\}\_\{i\},\\mathbf\{r\}^\{\\text\{reflect\}\}\_\{i\},\\mathbf\{z\}\_\{i\}\),with semantic embedding𝐳i\\mathbf\{z\}\_\{i\}via pre\-trained encoders\(Reimers and Gurevych,[2019](https://arxiv.org/html/2607.24779#bib.bib83)\)\. Retrieval uses cosine similarity with FAISS indexing\(Johnsonet al\.,[2019](https://arxiv.org/html/2607.24779#bib.bib84)\):

\(10\)ℳt=TopK​\(\{𝐳t⊤​𝐳i‖𝐳t‖​‖𝐳i‖\}i=1\|𝒟mem\|,k\)\.\\mathcal\{M\}\_\{t\}=\\text\{TopK\}\\left\(\\left\\\{\\frac\{\\mathbf\{z\}\_\{t\}^\{\\top\}\\mathbf\{z\}\_\{i\}\}\{\\\|\\mathbf\{z\}\_\{t\}\\\|\\\|\\mathbf\{z\}\_\{i\}\\\|\}\\right\\\}\_\{i=1\}^\{\|\\mathcal\{D\}\_\{\\text\{mem\}\}\|\},k\\right\)\.Algorithm[1](https://arxiv.org/html/2607.24779#alg1)summarizes the procedure\.

Algorithm 1High\-Level LLM Agent in HOBA1:Input:Campaign state

𝐬tH\\mathbf\{s\}\_\{t\}^\{H\}, memory bank

𝒟mem\\mathcal\{D\}\_\{\\text\{mem\}\}
2:Output:Hyperparameters

𝜽t\\boldsymbol\{\\theta\}\_\{t\}, reflection

𝐫treflect\\mathbf\{r\}\_\{t\}^\{\\text\{reflect\}\}
3:

4:// Think Phase

5:

𝐳t←Embed​\(StateToText​\(𝐬tH\)\)\\mathbf\{z\}\_\{t\}\\leftarrow\\text\{Embed\}\(\\text\{StateToText\}\(\\mathbf\{s\}\_\{t\}^\{H\}\)\)
6:

ℳt←TopK​\(\{sim​\(𝐳t,𝐳i\)\}i∈𝒟mem,k=3\)\\mathcal\{M\}\_\{t\}\\leftarrow\\text\{TopK\}\(\\\{\\text\{sim\}\(\\mathbf\{z\}\_\{t\},\\mathbf\{z\}\_\{i\}\)\\\}\_\{i\\in\\mathcal\{D\}\_\{\\text\{mem\}\}\},k=3\)
7:

𝐚tthink←LLM​\(𝒫think​\(𝐬tH,ℳt\)\)\\mathbf\{a\}\_\{t\}^\{\\text\{think\}\}\\leftarrow\\text\{LLM\}\(\\mathcal\{P\}\_\{\\text\{think\}\}\(\\mathbf\{s\}\_\{t\}^\{H\},\\mathcal\{M\}\_\{t\}\)\)
8:

9:// Act Phase

10:

𝜽t,𝐨texpected←LLM​\(𝒫act​\(𝐬tH,𝐚tthink\)\)\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{o\}\_\{t\}^\{\\text\{expected\}\}\\leftarrow\\text\{LLM\}\(\\mathcal\{P\}\_\{\\text\{act\}\}\(\\mathbf\{s\}\_\{t\}^\{H\},\\mathbf\{a\}\_\{t\}^\{\\text\{think\}\}\)\)
11:

𝜽t←Clip​\(𝜽t,𝜽min,𝜽max\)\\boldsymbol\{\\theta\}\_\{t\}\\leftarrow\\text\{Clip\}\(\\boldsymbol\{\\theta\}\_\{t\},\\boldsymbol\{\\theta\}\_\{\\min\},\\boldsymbol\{\\theta\}\_\{\\max\}\)
12:

13:// Observe Phase

14:

𝐨t←Aggregate​\(ExecuteOneHour​\(𝜽t\)\)\\mathbf\{o\}\_\{t\}\\leftarrow\\text\{Aggregate\}\(\\text\{ExecuteOneHour\}\(\\boldsymbol\{\\theta\}\_\{t\}\)\)
15:

𝜹t←ComputeDeviation​\(𝐨t,𝐨texpected\)\\boldsymbol\{\\delta\}\_\{t\}\\leftarrow\\text\{ComputeDeviation\}\(\\mathbf\{o\}\_\{t\},\\mathbf\{o\}\_\{t\}^\{\\text\{expected\}\}\)
16:

17:// Reflect Phase

18:

𝐫treflect←LLM​\(𝒫reflect​\(𝐚tthink,𝜽t,𝐨t,𝜹t\)\)\\mathbf\{r\}\_\{t\}^\{\\text\{reflect\}\}\\leftarrow\\text\{LLM\}\(\\mathcal\{P\}\_\{\\text\{reflect\}\}\(\\mathbf\{a\}\_\{t\}^\{\\text\{think\}\},\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{o\}\_\{t\},\\boldsymbol\{\\delta\}\_\{t\}\)\)
19:

𝐳t′←Embed​\(ExperienceToText​\(𝐬tH,𝜽t,𝐨t,𝐫treflect\)\)\\mathbf\{z\}\_\{t\}^\{\\prime\}\\leftarrow\\text\{Embed\}\(\\text\{ExperienceToText\}\(\\mathbf\{s\}\_\{t\}^\{H\},\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{o\}\_\{t\},\\mathbf\{r\}\_\{t\}^\{\\text\{reflect\}\}\)\)
20:

𝒟mem←𝒟mem∪\{\(𝐬tH,𝜽t,𝐨t,𝐫treflect,𝐳t′\)\}\\mathcal\{D\}\_\{\\text\{mem\}\}\\leftarrow\\mathcal\{D\}\_\{\\text\{mem\}\}\\cup\\\{\(\\mathbf\{s\}\_\{t\}^\{H\},\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{o\}\_\{t\},\\mathbf\{r\}\_\{t\}^\{\\text\{reflect\}\},\\mathbf\{z\}\_\{t\}^\{\\prime\}\)\\\}
21:

22:return

𝜽t\\boldsymbol\{\\theta\}\_\{t\},

𝐫treflect\\mathbf\{r\}\_\{t\}^\{\\text\{reflect\}\}

#### 4\.1\.2\.Prompt Engineering and Output Validation

Prompt effectiveness is critical for LLM\-guided optimization\. Our prompts incorporate: \(1\) role definition establishing domain expertise \(“You are an expert advertising optimization agent”\), \(2\) structured input with formatted performance metrics and placement breakdowns, \(3\) explicit reasoning framework \(problem→\\tocause→\\tostrategy\), and \(4\) JSON schema enforcement ensuring valid outputs with strict type constraints\.

To ensure reliability, we validate LLM outputs through multiple checks\. First, we verify JSON parsing success and schema compliance \(all required fields present with correct types\)\. Second, we apply domain constraints: bid bounds must satisfy0\.1≤qlower<qupper≤3\.00\.1\\leq q\_\{\\text\{lower\}\}<q\_\{\\text\{upper\}\}\\leq 3\.0, pacing rateρ∈\[0\.5,1\.5\]\\rho\\in\[0\.5,1\.5\], and exploration bonusθexplore∈\[0,0\.5\]\\theta^\{\\text\{explore\}\}\\in\[0,0\.5\]\. Third, we implement fallback mechanisms: if parsing fails or constraints are violated, we retry with error feedback up to 3 times before reverting to the previous valid configuration\. This multi\-layer validation ensures system robustness even when LLM outputs are occasionally malformed\. Critically, all hyperparameter outputs are hard\-clipped via Eq\. \(5\), ensuring stochasticity in the LLM reasoning trace does not propagate to bid execution\.

Model Optimization\.We optionally fine\-tune the LLM offline using Kahneman\-Tversky Optimization \(KTO\)\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2607.24779#bib.bib85)\), labeling configurations as desirable \(squality≥7s\_\{\\text\{quality\}\}\\geq 7\) or undesirable \(squality<4s\_\{\\text\{quality\}\}<4\) based on deployment outcomes\. KTO optimizes by pushing desirable configurations above an average reward baseline while pulling undesirable ones below, naturally encoding loss aversion that aligns with constraint\-sensitive bidding\. Fine\-tuning occurs weekly/monthly with gradual A/B rollout to production\.

### 4\.2\.Mid\-Level: SARSA\-Based Expert Selection with Causal Adjustment

The mid\-level agent selects expertaτ∈𝒜=\{PID,MPC,DT,IQL,CQL\}a\_\{\\tau\}\\in\\mathcal\{A\}=\\\{\\text\{PID\},\\text\{MPC\},\\text\{DT\},\\text\{IQL\},\\text\{CQL\}\\\}every 2 minutes based on context𝐱τ=\[𝐱τtime;𝐱τmarket;𝐱τperf;𝜽t;𝐱τexpert\]\\mathbf\{x\}\_\{\\tau\}=\[\\mathbf\{x\}^\{\\text\{time\}\}\_\{\\tau\};\\mathbf\{x\}^\{\\text\{market\}\}\_\{\\tau\};\\mathbf\{x\}^\{\\text\{perf\}\}\_\{\\tau\};\\boldsymbol\{\\theta\}\_\{t\};\\mathbf\{x\}^\{\\text\{expert\}\}\_\{\\tau\}\], encoding temporal features, market conditions, recent performance, high\-level hyperparameters, and per\-expert statistics\. Reward balances value, cost, and stability:

\(11\)rτ=valueτ−λ⋅costτ−μ⋅𝕀​\[aτ≠aτ−1\]−η⋅\|spend\_rateτ−target\_rateτ\|,r\_\{\\tau\}=\\text\{value\}\_\{\\tau\}\-\\lambda\\cdot\\text\{cost\}\_\{\\tau\}\-\\mu\\cdot\\mathbb\{I\}\[a\_\{\\tau\}\\neq a\_\{\\tau\-1\}\]\-\\eta\\cdot\|\\text\{spend\\\_rate\}\_\{\\tau\}\-\\text\{target\\\_rate\}\_\{\\tau\}\|,with switching penaltyμ=0\.1\\mu=0\.1and pacing penaltyη=0\.05\\eta=0\.05\.

#### 4\.2\.1\.Causal Adjustment and SARSA Selection

Historical data𝒟hist=\{\(𝐱i,ai,ri\)\}\\mathcal\{D\}\_\{\\text\{hist\}\}=\\\{\(\\mathbf\{x\}\_\{i\},a\_\{i\},r\_\{i\}\)\\\}suffers from confounding: experts selected in favorable contexts appear superior due to preferential selection\. We estimate causal effectμ​\(a\)=𝔼​\[r\|d​o​\(a\)\]\\mu\(a\)=\\mathbb\{E\}\[r\|do\(a\)\]via doubly robust estimator\(Bang and Robins,[2005](https://arxiv.org/html/2607.24779#bib.bib78); Dudíket al\.,[2014](https://arxiv.org/html/2607.24779#bib.bib79)\):

\(12\)μ^​\(a\)=1N​∑i=1N\[m^​\(𝐱i,a\)\+𝕀​\(ai=a\)π^​\(a\|𝐱i\)​\(ri−m^​\(𝐱i,a\)\)\],\\hat\{\\mu\}\(a\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[\\hat\{m\}\(\\mathbf\{x\}\_\{i\},a\)\+\\frac\{\\mathbb\{I\}\(a\_\{i\}=a\)\}\{\\hat\{\\pi\}\(a\|\\mathbf\{x\}\_\{i\}\)\}\(r\_\{i\}\-\\hat\{m\}\(\\mathbf\{x\}\_\{i\},a\)\)\\right\],where outcome modelm^​\(𝐱,a\)\\hat\{m\}\(\\mathbf\{x\},a\)\(XGBoost with 500 trees\) predicts𝔼​\[r\|𝐱,a\]\\mathbb\{E\}\[r\|\\mathbf\{x\},a\]and propensity modelπ^​\(a\|𝐱\)\\hat\{\\pi\}\(a\|\\mathbf\{x\}\)\(logistic regression with L2 regularization\) estimates logging policy probabilities\. Both models are trained daily on the last 7 days of data \( 500K samples\) with clipped propensity scoresπ^​\(a\|𝐱\)≥0\.05\\hat\{\\pi\}\(a\|\\mathbf\{x\}\)\\geq 0\.05to prevent extreme importance weights\.

We maintain linear Q\-functionQ​\(𝐱,a;ϕa\)=𝐰a⊤​ϕ​\(𝐱\)Q\(\\mathbf\{x\},a;\\boldsymbol\{\\phi\}\_\{a\}\)=\\mathbf\{w\}\_\{a\}^\{\\top\}\\phi\(\\mathbf\{x\}\)with polynomial features \(degree 2\)\. Expert selection combines Q\-values, causal effects, and UCB exploration:

\(13\)aτ=arg⁡maxa∈𝒜⁡\[Q​\(𝐱τ,a\)\+μ^​\(a\)\+θtexplore⋅β​log⁡\(τ\)Nτ​\(a\)\+1\],a\_\{\\tau\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\}\\left\[Q\(\\mathbf\{x\}\_\{\\tau\},a\)\+\\hat\{\\mu\}\(a\)\+\\theta^\{\\text\{explore\}\}\_\{t\}\\cdot\\beta\\sqrt\{\\frac\{\\log\(\\tau\)\}\{N\_\{\\tau\}\(a\)\+1\}\}\\right\],whereNτ​\(a\)N\_\{\\tau\}\(a\)counts selections,β=2\.0\\beta=2\.0controls exploration, andθtexplore\\theta^\{\\text\{explore\}\}\_\{t\}from high\-level modulates intensity\. After observing transition\(𝐱τ,aτ,rτ,𝐱τ\+1\)\(\\mathbf\{x\}\_\{\\tau\},a\_\{\\tau\},r\_\{\\tau\},\\mathbf\{x\}\_\{\\tau\+1\}\)and selecting next actionaτ\+1a\_\{\\tau\+1\}, we update via SARSA\(Sutton and Barto,[2018](https://arxiv.org/html/2607.24779#bib.bib87)\):

\(14\)ϕaτ←ϕaτ−α∇ϕaτ\(Q\(𝐱τ,aτ\)−\[rτ\+γQ\(𝐱τ\+1,aτ\+1\)\]\)2,\\boldsymbol\{\\phi\}\_\{a\_\{\\tau\}\}\\leftarrow\\boldsymbol\{\\phi\}\_\{a\_\{\\tau\}\}\-\\alpha\\nabla\_\{\\boldsymbol\{\\phi\}\_\{a\_\{\\tau\}\}\}\\left\(Q\(\\mathbf\{x\}\_\{\\tau\},a\_\{\\tau\}\)\-\[r\_\{\\tau\}\+\\gamma Q\(\\mathbf\{x\}\_\{\\tau\+1\},a\_\{\\tau\+1\}\)\]\\right\)^\{2\},withα=0\.01\\alpha=0\.01andγ=0\.99\\gamma=0\.99\. Critically, this updates only mid\-level Q\-network weights; low\-level expert models remain fixed\.

To handle market non\-stationarity, we detect distribution shifts via Kolmogorov\-Smirnov test\(Massey Jr,[1951](https://arxiv.org/html/2607.24779#bib.bib88)\)on reward CDFs\. WhenDK​SD\_\{KS\}exceeds threshold \(p<0\.05p<0\.05\), we temporarily double exploration and trigger expert fine\-tuning\.

### 4\.3\.Low\-Level: Dynamic Expert Pool

The pool contains\|𝒜\|∈\[5,10\]\|\\mathcal\{A\}\|\\in\[5,10\]experts executing bids per auction\.Expert parameters remain fixed during deployment—online learning is confined to mid\-level selection, ensuring safe adaptation\.

#### 4\.3\.1\.Expert Pool Composition

We initialize with five diverse offline\-trained experts:PID\(feedback control with gainsKp=0\.5K\_\{p\}=0\.5,Ki=0\.1K\_\{i\}=0\.1,Kd=0\.05K\_\{d\}=0\.05\),MPC\(constrained optimization over 1\-hour horizon via OSQP\(Stellatoet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib94)\)\), and three offline RL models—CQL\(Kumaret al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib62)\)\(conservative Q\-learning with penaltyα=5\.0\\alpha=5\.0\),IQL\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib63)\)\(expectile regression withτ=0\.7\\tau=0\.7\), andDT\(Chenet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib92)\)\(decision transformer with context length 20\)\. These experts exhibit complementary strengths \(Table[1](https://arxiv.org/html/2607.24779#S4.T1)\): PID for fast error correction, MPC for explicit budget planning, and offline RL for learning from logged data\.

Table 1\.Expert characteristics enabling adaptation across market conditions\.To adapt to market shifts, we dynamically expand the pool through fine\-tuned variants and hybrid ensembles\. Every 24 hours, we identify the best performer and create a fine\-tuned variant initialized from its parameters and trained for 100 gradient steps on recent data\. We also create weighted combinationsbτhybrid=w⋅bτ\(a1\)\+\(1−w\)⋅bτ\(a2\)b^\{\\text\{hybrid\}\}\_\{\\tau\}=w\\cdot b^\{\(a\_\{1\}\)\}\_\{\\tau\}\+\(1\-w\)\\cdot b^\{\(a\_\{2\}\)\}\_\{\\tau\}of complementary experts, where ensemble weightwwis updated online via gradient ascent on expected reward\. Underperforming dynamic experts are pruned when average reward falls below90%90\\%of the best static expert\.

#### 4\.3\.2\.Bid Execution and Constraints

Given activated expertaτa\_\{\\tau\}and auction features𝐟τ\\mathbf\{f\}\_\{\\tau\}, the final bid enforces hierarchical constraints:

\(15\)b~τ\\displaystyle\\tilde\{b\}\_\{\\tau\}=Expertaτ​\(𝐟τ,𝐬τ;ϕaτ\),\\displaystyle=\\text\{Expert\}\_\{a\_\{\\tau\}\}\(\\mathbf\{f\}\_\{\\tau\},\\mathbf\{s\}\_\{\\tau\};\\boldsymbol\{\\phi\}\_\{a\_\{\\tau\}\}\),bτ\\displaystyle b\_\{\\tau\}=Clip​\(Clip​\(b~τ,qlower⋅CPAtarget,qupper⋅CPAtarget\),bτ−1±𝚫t\),\\displaystyle=\\text\{Clip\}\\big\(\\text\{Clip\}\(\\tilde\{b\}\_\{\\tau\},q\_\{\\text\{lower\}\}\\cdot\\text\{CPA\}\_\{\\text\{target\}\},q\_\{\\text\{upper\}\}\\cdot\\text\{CPA\}\_\{\\text\{target\}\}\),b\_\{\\tau\-1\}\\pm\\boldsymbol\{\\Delta\}\_\{t\}\\big\),where first clipping enforces strategic bounds from high\-level LLM and second limits volatility \(typically±20%\\pm 20\\%\)\. This guarantees all bids respect hierarchical constraints regardless of expert behavior\. Static experts execute efficiently with total decision time ¡10ms, well within real\-time bidding requirements \(100ms timeout\)\. The three tiers operate asynchronously: high\-level updates𝜽t\\boldsymbol\{\\theta\}\_\{t\}hourly, mid\-level selectsaτa\_\{\\tau\}every 2 minutes, and low\-level executes bids per auction, ensuring safety, efficiency, and interpretability\.

## 5\.Experiments

We conduct extensive experiments in both offline environments and online A/B testing to validate the effectiveness of our proposed hierarchical reinforcement learning framework\. Four research questions guide our investigation:

- •RQ1: How does HOBA perform compared to state\-of\-the\-art auto\-bidding baselines?
- •RQ2: How do the hierarchical components contribute to HOBA’s overall performance?
- •RQ3: What is the impact of key hyperparameters and design choices?
- •RQ4: How does HOBA perform in real\-world deployment and production scenarios?

### 5\.1\.Experimental Setup

#### 5\.1\.1\.Dataset\.

We evaluate HOBA usingAuctionNet\(Zhaoet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib90)\), a large\-scale bidding benchmark released by Alibaba\. The benchmark consists of two distinct variants to assess model robustness: \(i\)AuctionNet\-Standard, featuring complete bidding trajectories with rich feedback, and \(ii\)AuctionNet\-Sparse, a challenging version characterized by a lower conversion density\. Each dataset comprises 500,000 trajectories across 10,000 episodes, with each episode spanning 48 discrete time steps\.

#### 5\.1\.2\.Evaluation Metrics

We adopt the AuctionNet evaluation protocol with the score metricscore=∑i\(oi​vi\)⋅min⁡\{\(C/CPA\)β,1\}\\text\{score\}=\\sum\_\{i\}\(o\_\{i\}v\_\{i\}\)\\cdot\\min\\\{\(C/\\text\{CPA\}\)^\{\\beta\},1\\\}whereβ=2\\beta=2, which jointly optimizes conversion value and cost\-per\-acquisition adherence\. Our evaluation uses an agent replacement protocol: we sequentially substitute all 48 time\-step agents with the candidate policy\. For each replacement configuration, we execute 30 independent trials and report the mean of the top\-5 outcomes to ensure statistical robustness\.

#### 5\.1\.3\.Baselines

We evaluate HOBA against competitive methods across three categories:

Rule\-Based Methods:

- •PID\(Zhanget al\.,[2014](https://arxiv.org/html/2607.24779#bib.bib60)\): Proportional\-integral\-derivative feedback controller with hand\-tuned control gains
- •MPC\(Hanet al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib61)\): Model predictive control optimizing over a 10\-step lookahead horizon

Offline RL Methods:

- •BCQ\(Fujimotoet al\.,[2019](https://arxiv.org/html/2607.24779#bib.bib91)\): Batch\-constrained Q\-learning using VAE for constrained action generation
- •CQL\(Kumaret al\.,[2020](https://arxiv.org/html/2607.24779#bib.bib62)\): Conservative Q\-learning applying penalties to out\-of\-distribution actions
- •IQL\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib63)\): Implicit Q\-learning through expectile regression \(τ=0\.8\\tau=0\.8\)
- •DT\(Chenet al\.,[2021](https://arxiv.org/html/2607.24779#bib.bib92)\): Decision Transformer formulating RL as conditional sequence modeling

Advanced Baselines:

- •HRL\-Bid\(Yuanet al\.,[2022](https://arxiv.org/html/2607.24779#bib.bib93)\): Hierarchical RL employing fixed manager\-worker architecture
- •BO\+SARSA: Combines Bayesian optimization\(Snoeket al\.,[2012](https://arxiv.org/html/2607.24779#bib.bib95)\)for hyperparameter search with SARSA\-based expert selection \(requires 50 offline policy evaluations per campaign, computationally expensive at scale\)

#### 5\.1\.4\.Implementation

We implement all methods in PyTorch and run experiments on GPUs\. HOBA’s configuration is as follows:

High\-Level LLM Agent:We employ GPT\-4o with sampling temperature 0\.7, maximum token limit 2048, and hourly update frequency\. Historical experience retrieval uses FAISS\(Johnsonet al\.,[2019](https://arxiv.org/html/2607.24779#bib.bib84)\)indexing with top\-k=3k=3neighbors selected via cosine similarity \(threshold 0\.7\)\. Structured JSON schemas ensure type\-safe hyperparameter generation with range validation\.

Mid\-Level SARSA Agent:State features are 64\-dimensional, encoding budget utilization, ROI trends, market competition, PID control errors, and temporal features\. We use learning rateα=0\.01\\alpha=0\.01, discount factorγ=0\.95\\gamma=0\.95, and UCB exploration coefficientβ=2\.0\\beta=2\.0\. The causal adjustment module trains an XGBoost outcome model \(500 trees, maximum depth 6\) and a multinomial logistic regression propensity model, both retrained daily on accumulated deployment data\.

Low\-Level Expert Pool:The initial pool contains 5 static experts \(PID, MPC, IQL, DT, CQL\), each pre\-trained on 6 months of historical auction data \( 50M transitions\)\. Dynamic experts are created weekly by fine\-tuning the best\-performing static expert on 1000 recent transitions for 100 gradient steps\. We prune underperforming dynamic experts when their Q\-value falls belowαprune=0\.8×maxa′⁡Q​\(s,a′\)\\alpha\_\{\\text\{prune\}\}=0\.8\\times\\max\_\{a^\{\\prime\}\}Q\(s,a^\{\\prime\}\)for 24 consecutive hours\.Complete hyperparameter settings are detailed in Table[2](https://arxiv.org/html/2607.24779#S5.T2)\.

Table 2\.Hyperparameter configuration for HOBA\.

### 5\.2\.Overall Performance Comparison \(RQ1\)

Table[3](https://arxiv.org/html/2607.24779#S5.T3)presents comprehensive performance comparison across both standard and sparse settings over five budget scales\. HOBA consistently outperforms all baselines across all configurations\.

Table 3\.Performance comparison on AuctionNet Standard and Sparse\. HOBA consistently outperforms all baselines across budget scales\. Mean score over 5 runs\.∗indicates statistical significance \(p<0\.05p<0\.05\) vs\. best baseline\.Key observations:

1. \(1\)Consistent superiority across settings: HOBA achieves state\-of\-the\-art performance in all 10 configurations, with improvements ranging from \+3\.0% \(standard, 50% budget\) to \+12\.1% \(sparse, 100% budget\)\.
2. \(2\)Larger gains in sparse scenarios: The performance gap widens significantly in sparse settings \(\+9\.7%\-12\.1% vs\. \+3\.0%\-4\.7%\), validating that HOBA’s hierarchical design provides greater value when long\-term credit assignment is challenging\. The LLM’s strategic reasoning and SARSA’s causal adjustment become more critical when conversion signals are scarce, with the largest improvement \(\+12\.1%\) observed at 100% budget where the balance between exploration and exploitation is most crucial\.
3. \(3\)Offline RL surpasses rules: Methods learning from logged data \(CQL, IQL, DT\) substantially outperform rule\-based approaches \(PID, MPC\) by 18\-25%, confirming the importance of data\-driven optimization\.
4. \(4\)Hyperparameter tuning matters: BO\+SARSA demonstrates that proper configuration improves performance over fixed settings \(\+5\.7% vs\. IQL on average\)\. However, BO\+SARSA requires 50 offline evaluations per campaign, making it computationally prohibitive for large\-scale deployment with thousands of campaigns\.
5. \(5\)Hierarchical decomposition excels: HOBA’s three\-tier architecture \(strategic LLM \+ tactical SARSA \+ operational experts\) achieves substantial gains through effective separation of concerns, demonstrating that decoupling reasoning, selection, and execution addresses the safety\-adaptability tradeoff\. The improvement is more pronounced in sparse settings where strategic planning becomes essential\.

Computational Efficiency: Offline training \(expert models, causal models\) takes 6 hours on GPUs \(one\-time cost amortized across campaigns\)\. Online inference achieves P99 latency of 38ms \(LLM hourly overhead: 15ms amortized, SARSA selection: 2ms, expert execution: 21ms\), comfortably meeting the ¡100ms real\-time constraint for production deployment\.

### 5\.3\.Ablation Study: Component Contributions \(RQ2\)

To isolate the contribution of each hierarchical component, we conduct systematic ablation experiments on the standard setting with 100% budget \(Table[4](https://arxiv.org/html/2607.24779#S5.T4)\)\.

Table 4\.Ablation study on AuctionNet \(100% budget\)\. Each row removes or replaces one component\.Critical findings:

1. \(1\)Expert selection is most critical: Removing SARSA\-based expert selection causes the largest performance drop \(\-10\.7%\), demonstrating that no single expert dominates across all market conditions\. Dynamic selection adapts to market shifts, while a fixed expert \(even the best offline: IQL\) fails under distribution changes\.
2. \(2\)LLM strategic reasoning provides substantial value: Removing the high\-level LLM degrades performance by 7\.1%\. This confirms that adaptive hyperparameter inference based on campaign context is critical for maintaining cost efficiency\.
3. \(3\)On\-policy SARSA outperforms off\-policy DQN: Replacing SARSA with DQN causes 6\.4% performance loss and 70\.6% more expert switching instability \(5\.8/hr vs\. 3\.4/hr\)\. This validates that on\-policy learning is more stable in non\-stationary auction environments, as DQN’s max operator leads to Q\-value overestimation and frequent action changes under market shifts\.
4. \(4\)Causal adjustment eliminates bias: Removing doubly robust estimation results in 4\.9% value loss, demonstrating that naive Q\-learning on biased historical data leads to suboptimal expert selection due to confounding between historical bidding strategies and observed outcomes\.
5. \(5\)Memory retrieval enhances decisions: Removing LLM memory retrieval degrades performance by 3\.6%, showing that learning from similar historical experiences improves hyperparameter quality\. The ”Think\-Observe\-Reflect” loop enables continual improvement\.
6. \(6\)Dynamic experts provide marginal gains: Removing dynamic expert adaptation causes only 2\.5% loss, suggesting that the 5 static experts already provide sufficient coverage\. However, dynamic experts offer insurance against unexpected market regimes\.

All components contribute synergistically to achieve state\-of\-the\-art performance, with expert selection and LLM\-guided hyperparameter optimization being the most critical\.

### 5\.4\.Parameter Sensitivity \(RQ3\)

We examine how key hyperparameters affect HOBA’s performance\. Figure[2](https://arxiv.org/html/2607.24779#S5.F2)visualizes sensitivity across four critical parameters\.

![Refer to caption](https://arxiv.org/html/2607.24779v1/x2.png)Figure 2\.Hyperparameter sensitivity analysis showing performance across different configurations\.LLM Update Frequency\.Hourly updates \(default\) achieve optimal performance\. Increasing frequency to 30 minutes yields minor improvement \(\+1\.2%\) while doubling API costs and introducing decision volatility\. Reducing frequency to 4 hours degrades performance by 3\.8% due to stale hyperparameters unable to track market dynamics\.

Exploration Coefficientβ\\beta\.Optimal balance occurs atβ=2\.0\\beta=2\.0\. Lower values \(β=0\.5\\beta=0\.5\) restrict exploration, causing the agent to overlook superior experts \(\-4\.5%\)\. Higher values \(β=5\.0\\beta=5\.0\) trigger excessive switching behavior and instability \(\-3\.2%\), as uncertainty dominates over value exploitation\.

Learning Rateα\\alpha\.SARSA exhibits stable performance acrossα∈\[0\.005,0\.02\]\\alpha\\in\[0\.005,0\.02\]\. Values below this range \(α=0\.001\\alpha=0\.001\) impede adaptation speed \(\-3\.1%\), while values above \(α=0\.05\\alpha=0\.05\) induce gradient noise and oscillation in non\-stationary markets \(\-4\.8%\)\.

Memory Retrievalkk\.Retrieved experience count plateaus atk=3k=3\. Single retrieval \(k=1k=1\) provides inadequate historical context \(\-2\.3%\)\. Large counts \(k=10k=10\) inject dissimilar experiences as noise and inflate prompt tokens \(\-1\.8%\)\.

HOBA maintains robust performance across reasonable hyperparameter ranges, minimizing the need for exhaustive tuning\.

### 5\.5\.Online A/B Testing \(RQ4\)

To provide the ultimate validation, we deployed HOBA in a large\-scale online A/B test on a major advertising platform\.

#### 5\.5\.1\.Experimental Setup\.

Baseline and Deployment:

Our baseline is the incumbent production system, a highly\-optimized framework combining Decision Transformer control with rule\-based hyperparameter tuning\. This baseline has been refined over 3\+ years of production use, serving billions of daily auctions across 100K\+ campaigns\. It represents a strong industrial benchmark that prioritizes stability and predictability\.

HOBA is deployed via a distributed serving system with GPU acceleration\. The architecture consists of: \(1\) High\-level LLM agent running on CPU servers, invoking GPT\-4o API hourly; \(2\) Mid\-level SARSA agent on GPUs, selecting experts every 2 minutes; \(3\) Low\-level expert pool executing bids at millisecond latency via TensorRT optimization\. The system ensures fault tolerance and load balancing across multiple instances\.

Multi\-Stage Experiment Design: To ensure safe and robust validation, we conducted a phased rollout:

- •Phase 1 \(Observation\): Dec 1\-7, 2025\. HOBA deployed on 5% of advertisements \(1,247 campaigns, $2\.1M total budget\) to validate stability and initial performance in a controlled setting\.
- •Phase 2 \(Scale\-up\): Dec 8\-14, 2025\. Following successful Phase 1 results, we scaled to 20% of advertisements \(4,982 campaigns, $8\.7M budget\) for broader validation across diverse campaign types\.
- •Phase 3 \(Full Rollout\): Dec 15\-28, 2025\. After confirming consistent gains, HOBA was promoted to 100% advertisements\.

Consistent performance improvements were observed across all phases, confirming HOBA’s robustness to varying advertising conditions, budget allocations, and campaign characteristics\.

Latency and Computational Cost: The deployed HOBA system achieves P99 latency of 38ms per auction request, compared to baseline’s 12ms, with the overhead dominated by the bid execution pipeline \(21ms, including SARSA selection, expert inference, and system overhead\) rather than the LLM, which contributes only∼\\sim15ms amortized per auction since it runs hourly rather than per\-bid\. While HOBA introduces additional overhead \(LLM: 15ms amortized per hour, SARSA: 2ms per 2min, expert inference: 21ms\), this latency comfortably meets the platform’s ¡100ms constraint\. LLM API costs average $0\.08 per campaign per day, negligible compared to $5K\-50K daily budgets\. Compared to manual hyperparameter tuning \($20 per adjustment, 2 adjustments/week\), HOBA reduces operational costs by 96%\.

#### 5\.5\.2\.Results and Business Impact

We define each online metric precisely\.Target Cost Achievementis the spend\-weighted fraction of campaigns where actual CPA stays within the advertiser target, directly reflecting constraint compliance from Eq\. \(1\)\.Conversion Valueis the cumulative∑jAi​j⋅vi​j\\sum\_\{j\}A\_\{ij\}\\cdot v\_\{ij\}across all won auctions\.Budget Completionis total spend divided by budget capBiB\_\{i\}\.ROIis Conversion Value divided by total spend\. Statistical significance is assessed via two\-samplett\-test with Bonferroni correction \(∗:p<0\.001p<0\.001,∗∗:p<0\.01p<0\.01\)\. Table[5](https://arxiv.org/html/2607.24779#S5.T5)summarizes A/B test results from Phase 2 \(Dec 8\-14, 2025\) with 4,982 campaigns providing sufficient statistical power\.

Table 5\.Online A/B test results \(Phase 2, 4,982 campaigns\)\.∗:p<0\.001p<0\.001,∗∗:p<0\.01p<0\.01\(two\-samplett\-test, Bonferroni correction\)\.Key business impacts: \(1\)Revenue growth: 8\.1% conversion value gain \($1\.50M additional value over 7 days\) demonstrates efficient budget utilization\. \(2\)Cost control: 3\.6% improvement in target cost achievement \(90\.0%→93\.2%\) improves advertiser trust, reflected in 7\.6% satisfaction increase\. \(3\)Budget efficiency: Near\-complete budget utilization \(97\.8% vs\. 94\.2%\) eliminates underspend waste\. \(4\)ROI improvement: 3\.3% gain \(2\.22 vs\. 2\.15\) validates more efficient ad spend allocation\.

#### 5\.5\.3\.Robustness Analysis

Distribution Shifts\.Consistent gains across phased rollout \(Phase 1: \+3\.0% ROI on 5% traffic, Phase 2: \+3\.3% on 20%, Phase 3: \+3\.5% on 100%\) confirm robustness despite varying traffic volumes, user demographics, and competitive landscapes\.

Cold\-Start Generalization\.We evaluate HOBA on 623 cold\-start campaigns \(Dec 15\-21, 2025\) lacking historical data—a stringent OOD test requiring generalization from learned bidding principles alone\. HOBA achieved \+4\.9% ROI improvement vs\. baseline, exceeding \+3\.3% warm\-start average by 48%, validating strong zero\-shot transfer capability\.

Failure Cases\.HOBA underperforms on extremely sparse campaigns \(¡3 conversions/day, 2\.8% of total\) where causal estimation suffers from insufficient samples\. The system gracefully falls back to conservative Decision Transformer control, preventing catastrophic failures\. Future work will investigate few\-shot meta\-learning for rare campaign types\.

Following validation, HOBA scaled to 100% traffic \(Dec 15, 2025\) with 1\-month monitoring confirming sustained improvements without degradation\.

## 6\.Conclusion

We present HOBA, a hierarchical reinforcement learning framework that addresses the safety\-adaptability tradeoff in online advertising bidding\. By decomposing decision\-making across three temporal scales—strategic hyperparameter inference via LLM\-guided reasoning with experience memory, tactical expert coordination through SARSA with causal debiasing, and operational bid generation via a dynamic expert ensemble—our approach restricts online learning to discrete model selection rather than continuous action exploration, substantially reducing deployment risk while preserving market responsiveness\. The hierarchical architecture enables interpretable strategic oversight through natural language reasoning traces and quantifiable expert contribution attribution via causal analysis\. Comprehensive evaluation on AuctionNet benchmarks and production A/B testing validates consistent superiority over competitive baselines\. Large\-scale deployment across 100K\+ campaigns demonstrates significant business impact, achieving \+3\.3% ROI improvement and \+8\.1% conversion value increase\. Future work will investigate meta\-learning techniques for cold\-start campaign generalization and extend the framework to multi\-objective optimization incorporating advertiser\-specific preference models\.

## References

- H\. Bang and J\. M\. Robins \(2005\)Doubly robust estimation in missing data and causal inference models\.Biometrics61\(4\),pp\. 962–973\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1),[§4\.2\.1](https://arxiv.org/html/2607.24779#S4.SS2.SSS1.p1.2)\.
- A\. G\. Barto and S\. Mahadevan \(2003\)Recent advances in hierarchical reinforcement learning\.Discrete event dynamic systems13\(1\-2\),pp\. 41–77\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- H\. Cai, R\. Kan, X\. Zhu,et al\.\(2017\)Real\-time bidding by reinforcement learning in display advertising\.InProceedings of the tenth ACM international conference on web search and data mining,pp\. 661–670\.Cited by:[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p2.1)\.
- L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. Mordatch \(2021\)Decision transformer: reinforcement learning via sequence modeling\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 15084–15097\.Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p3.1),[§4\.3\.1](https://arxiv.org/html/2607.24779#S4.SS3.SSS1.p1.5),[4th item](https://arxiv.org/html/2607.24779#S5.I3.i4.p1.1)\.
- Y\. Chen, Z\. Huang, H\. Wu, Y\. Lv, Y\. Zhang,et al\.\(2023\)Towards autonomous system: flexible modular production system enhanced with large language model agents\.arXiv preprint arXiv:2304\.14721\.Cited by:[§2\.3](https://arxiv.org/html/2607.24779#S2.SS3.p1.1)\.
- P\. Dayan and G\. E\. Hinton \(1992\)Feudal reinforcement learning\.Advances in neural information processing systems5\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- M\. Dudík, D\. Erhan, J\. Langford, and L\. Li \(2014\)Doubly robust policy evaluation and optimization\.InStatistical Science,Vol\.29,pp\. 485–511\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1),[§4\.2\.1](https://arxiv.org/html/2607.24779#S4.SS2.SSS1.p1.2)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)KTO: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[§4\.1\.2](https://arxiv.org/html/2607.24779#S4.SS1.SSS2.p3.2)\.
- J\. Fu, A\. Kumar, O\. Nachum, G\. Tucker, and S\. Levine \(2021\)D4RL: datasets for deep data\-driven reinforcement learning\.InNeurIPS Datasets Track,Cited by:[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p3.1)\.
- S\. Fujimoto, D\. Meger, and D\. Precup \(2019\)Off\-policy deep reinforcement learning without exploration\.InInternational Conference on Machine Learning,pp\. 2052–2062\.Cited by:[1st item](https://arxiv.org/html/2607.24779#S5.I3.i1.p1.1)\.
- A\. Gilotte, C\. Calauzènes, T\. Nedelec, A\. Abraham, and S\. Dollé \(2018\)Offline a/b testing for recommender systems\.InProceedings of the Eleventh ACM International Conference on Web Search and Data Mining,pp\. 198–206\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1)\.
- X\. Han, W\. Wang, and W\. Zhang \(2020\)A reinforcement learning framework for optimizing real\-time bidding\.IEEE TKDE32,pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2607.24779#S3.SS1.p1.2),[2nd item](https://arxiv.org/html/2607.24779#S5.I2.i2.p1.1)\.
- D\. G\. Horvitz and D\. J\. Thompson \(1952\)A generalization of sampling without replacement from a finite universe\.Journal of the American statistical Association47\(260\),pp\. 663–685\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1)\.
- Y\. Hu, C\. Han, T\. Guo, and H\. Xiao \(2022\)Applying opponent modeling for automatic bidding in online repeated auctions\.arXiv preprint arXiv:2212\.02723\.Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p1.1)\.
- W\. Huang, P\. Abbeel, D\. Pathak, and I\. Mordatch \(2022\)Language models as zero\-shot planners: extracting actionable knowledge for embodied agents\.International Conference on Machine Learning,pp\. 9118–9147\.Cited by:[§2\.3](https://arxiv.org/html/2607.24779#S2.SS3.p1.1)\.
- J\. Johnson, M\. Douze, and H\. Jégou \(2019\)Billion\-scale similarity search with GPUs\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.Cited by:[§4\.1\.1](https://arxiv.org/html/2607.24779#S4.SS1.SSS1.p4.3),[§5\.1\.4](https://arxiv.org/html/2607.24779#S5.SS1.SSS4.p2.1)\.
- I\. Kostrikov, A\. Nair, and S\. Levine \(2022\)Offline reinforcement learning with implicit q\-learning\.InICLR,Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.24779#S3.SS2.p2.1),[§4\.3\.1](https://arxiv.org/html/2607.24779#S4.SS3.SSS1.p1.5),[3rd item](https://arxiv.org/html/2607.24779#S5.I3.i3.p1.1)\.
- A\. Kumar, A\. Zhou, G\. Tucker, and S\. Levine \(2020\)Conservative q\-learning for offline reinforcement learning\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.24779#S3.SS2.p2.1),[§4\.3\.1](https://arxiv.org/html/2607.24779#S4.SS3.SSS1.p1.5),[2nd item](https://arxiv.org/html/2607.24779#S5.I3.i2.p1.1)\.
- S\. Levine, A\. Kumar, G\. Tucker, and J\. Fu \(2020\)Offline reinforcement learning: tutorial, review, and perspectives on open problems\.arXiv preprint arXiv:2005\.01643\.Cited by:[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.24779#S3.SS2.p2.1)\.
- L\. Li, W\. Chu, J\. Langford, and R\. E\. Schapire \(2010\)A contextual\-bandit approach to personalized news article recommendation\.InProceedings of the 19th international conference on World wide web,pp\. 661–670\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1)\.
- L\. Li, J\. Kim, and I\. Zitouni \(2015\)Toward predicting the outcome of an a/b experiment for search relevance\.Proceedings of the Eighth ACM International Conference on Web Search and Data Mining,pp\. 37–46\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1)\.
- F\. J\. Massey Jr \(1951\)The kolmogorov\-smirnov test for goodness of fit\.Journal of the American statistical Association46\(253\),pp\. 68–78\.Cited by:[§4\.2\.1](https://arxiv.org/html/2607.24779#S4.SS2.SSS1.p3.2)\.
- V\. Mnih, A\. P\. Badia, M\. Mirza, A\. Graves, T\. Lillicrap, T\. Harley, D\. Silver, and K\. Kavukcuoglu \(2016\)Asynchronous methods for deep reinforcement learning\.InInternational Conference on Machine Learning,pp\. 1928–1937\.Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§3\.2](https://arxiv.org/html/2607.24779#S3.SS2.p2.1)\.
- O\. Nachum, S\. S\. Gu, H\. Lee, and S\. Levine \(2018\)Data\-efficient hierarchical reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- T\. Osa, J\. Pajarinen, G\. Neumann, J\. A\. Bagnell, P\. Abbeel, and J\. Peters \(2020\)An algorithmic perspective on imitation learning\.Foundations and Trends in Robotics7\(1\-2\),pp\. 1–179\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- S\. Pateria, B\. Subagdja, A\. Tan, and C\. Quek \(2021\)Hierarchical reinforcement learning: a comprehensive survey\.ACM Computing Surveys \(CSUR\)54\(5\),pp\. 1–35\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 3982–3992\.Cited by:[§4\.1\.1](https://arxiv.org/html/2607.24779#S4.SS1.SSS1.p4.3)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§3\.2](https://arxiv.org/html/2607.24779#S3.SS2.p2.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 8634–8652\.Cited by:[§2\.3](https://arxiv.org/html/2607.24779#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.24779#S4.SS1.p1.1)\.
- J\. Snoek, H\. Larochelle, and R\. P\. Adams \(2012\)Practical bayesian optimization of machine learning algorithms\.InAdvances in Neural Information Processing Systems,Vol\.25\.Cited by:[§3\.2](https://arxiv.org/html/2607.24779#S3.SS2.p3.1),[2nd item](https://arxiv.org/html/2607.24779#S5.I4.i2.p1.1)\.
- B\. Stellato, G\. Banjac, P\. Goulart, A\. Bemporad, and S\. Boyd \(2020\)OSQP: an operator splitting solver for quadratic programs\.Mathematical Programming Computation12\(4\),pp\. 637–669\.Cited by:[§4\.3\.1](https://arxiv.org/html/2607.24779#S4.SS3.SSS1.p1.5)\.
- R\. S\. Sutton and A\. G\. Barto \(2018\)Reinforcement learning: an introduction\.MIT press\.Cited by:[§4\.2\.1](https://arxiv.org/html/2607.24779#S4.SS2.SSS1.p2.6)\.
- R\. S\. Sutton, D\. Precup, and S\. Singh \(1999\)Between mdps and semi\-mdps: a framework for temporal abstraction in reinforcement learning\.Artificial intelligence112\(1\-2\),pp\. 181–211\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- A\. Swaminathan and T\. Joachims \(2015\)Batch learning from logged bandit feedback through counterfactual risk minimization\.Journal of Machine Learning Research16\(1\),pp\. 1731–1755\.Cited by:[§2\.4](https://arxiv.org/html/2607.24779#S2.SS4.p1.1)\.
- A\. S\. Vezhnevets, S\. Osindero, T\. Schaul, N\. Heess, M\. Jaderberg, D\. Silver, and K\. Kavukcuoglu \(2017\)Feudal networks for hierarchical reinforcement learning\.InInternational Conference on Machine Learning,pp\. 3540–3549\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p1.1)\.
- C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen \(2024\)Large language models as optimizers\.arXiv preprint arXiv:2309\.03409\.Cited by:[§2\.3](https://arxiv.org/html/2607.24779#S2.SS3.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)React: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2607.24779#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.24779#S4.SS1.p1.1)\.
- W\. Yuan, H\. Wang, X\. Yu, N\. Liu, and Z\. Li \(2021\)Hierarchical reinforcement learning for integrated recommendation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 4521–4528\.Cited by:[§2\.2](https://arxiv.org/html/2607.24779#S2.SS2.p2.1)\.
- W\. Yuan, H\. Wang, X\. Yu, N\. Liu, and Z\. Li \(2022\)Multi\-agent reinforcement learning for online advertising\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,pp\. 4447–4457\.Cited by:[1st item](https://arxiv.org/html/2607.24779#S5.I4.i1.p1.1)\.
- W\. Zhang, S\. Yuan, and J\. Wang \(2014\)Optimal real\-time bidding for display advertising\.InKDD,pp\. 1077–1086\.Cited by:[§1](https://arxiv.org/html/2607.24779#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.24779#S2.SS1.p1.1),[1st item](https://arxiv.org/html/2607.24779#S5.I2.i1.p1.1)\.
- J\. Zhao, W\. Zhang, J\. Wang,et al\.\(2021\)AuctionNet: benchmark dataset for learning to bid in online advertising auctions\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,pp\. 2479–2489\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.24779#S5.SS1.SSS1.p1.1)\.

## Appendix ALLM Prompt Template Structure

Due to proprietary constraints, we provide the structural design of all four TAOR phases rather than verbatim templates\.

System Role\.The LLM is defined as a high\-level reasoning module operating on a 1\-hour cycle\. It observes campaign state, retrieves top\-KKsimilar historical experiences, and emits structured hyperparameters that constrain downstream layers\. It does*not*directly produce bid prices; it only sets bounds enforced by construction via Eq\. \(15\)\.

Think Phase\.Input includes campaign state \(budget utilization, CPA ratio, market competition, recent performance\) and top\-KKretrieved memory entries\. The LLM performs problem identification and root cause analysis following Eq\. \(3\)\.

Act Phase\.Based on Think output, the LLM generates hyperparameters via a strict JSON output schema:

```
{
  "mode": "conservative"|"moderate"|"aggressive",
  "bid_bounds": [q_lo, q_hi],   // 0.1 <= q_lo < q_hi <= 3.0
  "pacing_rate": float [0.5, 1.5],
  "theta_explore": float [0.0, 0.5]
}
```

Post\-hoc Clipping \(Eq\. \(5\)\)\.All outputs are hard\-clipped to valid ranges regardless of LLM sampling variance, ensuring stochasticity in the reasoning trace does not propagate to bid execution:

```
q_lo  = clip(q_lo, 0.1, 3.0)
q_hi  = clip(q_hi, q_lo, 3.0)
pacing = clip(pacing, 0.5, 1.5)
theta_explore = clip(theta_explore, 0.0, 0.5)
```

Observe Phase\.Aggregates one\-hour execution metrics: CPA deviationδCPA\\delta\_\{\\text\{CPA\}\}, conversion deviationδconv\\delta\_\{\\text\{conv\}\}, and budget deviationδbudget\\delta\_\{\\text\{budget\}\}\(Eq\. \(6\)\)\.

Reflect Phase\.Evaluates decision quality viasqualitys\_\{\\text\{quality\}\}\(Eq\. \(8\)\) and stores the structured experience tuple\(stH,𝜽t,ot,rtreflect,zt\)\(s^\{H\}\_\{t\},\\boldsymbol\{\\theta\}\_\{t\},o\_\{t\},r^\{\\text\{reflect\}\}\_\{t\},z\_\{t\}\)to memory bank𝒟mem\\mathcal\{D\}\_\{\\text\{mem\}\}\(Eq\. \(9\)\)\.