利用强化学习优化 Pinterest 自动竞价系统中的控制器参数

# Reinforcement Learning for Controller Parameters in the Auto-Bidding System at Pinterest
Source: [https://arxiv.org/html/2610.33248](https://arxiv.org/html/2610.33248)
DOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN:978\-1\-4503\-XXXX\-X/2018/06CCS:Information systems Online advertisingCCS:Computing methodologies Reinforcement learningCCS:Information systems Auction algorithmsYichu ZhouNote:Authors contributed equally to this research\.Affiliation:Pinterest Inc\.,Seattle,USAemail:[yichuzhou@pinterest\.com](mailto:[email protected])Qinyi ChenAffiliation:Pinterest Inc\.,San Francisco,USAemail:[qinyichen@pinterest\.com](mailto:[email protected]),Fei FengAffiliation:Pinterest Inc\.,New York,USAemail:[ffeng@pinterest\.com](mailto:[email protected]),Jacob LohAffiliation:Pinterest Inc\.,San Francisco,USAemail:[jloh@pinterest\.com](mailto:[email protected]),Mehdi Ben AyedAffiliation:Pinterest Inc\.,New York,USAemail:[mbenayed@pinterest\.com](mailto:[email protected]),Lin YangAffiliation:Pinterest Inc\.,University of California, Los Angeles,Los Angeles,USAemail:[linyang@pinterest\.com](mailto:[email protected]),Andrei CureleaAffiliation:Pinterest Inc\.,San Francisco,USAemail:[acurelea@pinterest\.com](mailto:[email protected])andJiajing XuAffiliation:Pinterest Inc\.,San Francisco,USAemail:[jiajing@pinterest\.com](mailto:[email protected])

2018

###### Abstract\.

Auto\-bidding systems in online advertising must balance competing objectives—maximizing budget delivery while minimizing cost per action—across heterogeneous campaigns under non\-stationary market conditions\. This trade\-off evolves over campaign lifespans: early on, cost optimization dominates when budgets roll over; near deadlines, budget delivery becomes urgent\. While PID controllers are widely used in production for their stability and interpretability, fixed tunings cannot adapt to diverse campaigns and shifting traffic patterns\. We presentRLBP\(Reinforcement Learning for Bidding Parameters\), a hybrid architecture deployed at Pinterest that uses RL as a meta\-controller to select campaign\-specific PID parameters daily while retaining PID for high\-frequency bid adjustments\.RLBPtrains a multi\-headQQ\-network using a relative\-lift reward formulation normalized against the production baseline, and performs inference\-time fusion that adapts the cost–delivery trade\-off to campaign states without retraining\. Through online A/B testing at Pinterest, RLBP achieves a statistically significant reduction in effective cost\-per\-click \(eCPC\) with no revenue degradation, and two months of post\-rollout measurements against a holdout also show consistent cost reductions\. These results demonstrate that meta\-controller RL can successfully augment production control systems at scale\.

###### Keywords:

auto\-bidding, reinforcement learning, PID control, budget pacing, computational advertising

## 1\.Introduction

Pinterest is a visual discovery platform that serves approximately619619million monthly active users and generated $4\.2 billion in advertising revenue in 2025\([Pinterest, Inc\., 2026](https://arxiv.org/html/2610.33248#bib.bib26)\)\. Like every large advertising platform, its ads delivery system must solve an*auto\-bidding*problem on behalf of each advertiser: given a budget, a lifetime schedule, and cost\-efficiency targets such as cost\-per\-click \(CPC\) or return\-on\-spend \(ROS\), decide how much to bid for each user\-level impression so that the campaign delivers its budget smoothly while satisfying its KPI constraints\([Aggarwal et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib1);[Balseiro et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib2)\)\. Because these decisions are taken billions of times per day, even fractional improvements to the bidding policy translate into substantial advertiser value and platform revenue\.

A long line of work formulates auto\-bidding as a budget\- and KPI\-constrained optimization problem, whose optimal solution, under standard assumptions and primal–dual arguments, reduces to bidding a scalar multiple of the predicted impression value\([Wu et al\., 2018](https://arxiv.org/html/2610.33248#bib.bib11);[He et al\., 2021](https://arxiv.org/html/2610.33248#bib.bib13);[Yang et al\., 2019](https://arxiv.org/html/2610.33248#bib.bib8);[Balseiro et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib2);[Aggarwal et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib1);[Chen et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib28)\)\. The operational question is then how to update that scalar*online*, under non\-stationary traffic and in the face of daily variation in user behavior, advertiser entry, and competitive pressure\. In production, this online update is often implemented using a proportional–integral–derivative \(PID\) feedback controller that tracks the running error between realized performance and a pacing target and adjusts a bid multiplier to close the gap\([Zhang et al\., 2016](https://arxiv.org/html/2610.33248#bib.bib6);[Agarwal et al\., 2014](https://arxiv.org/html/2610.33248#bib.bib4);[Yang et al\., 2019](https://arxiv.org/html/2610.33248#bib.bib8);[Stram et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib9);[Karlsson, 2020](https://arxiv.org/html/2610.33248#bib.bib7)\)\. PID controllers are simple, interpretable, and widely used, which is why they remain the backbone of industrial pacing systems at LinkedIn, Yahoo, Alibaba, and elsewhere\. Their weakness, however, is rigidity: a single fixed setup cannot be simultaneously appropriate for hundreds of thousands of heterogeneous campaigns and shifting market conditions\.

Reinforcement learning \(RL\) is a natural way to replace such rules with a policy that adapts to campaign states\. A large body of work formulates real\-time bidding as a Markov decision process and trains a deep RL agent to emit bids directly\([Cai et al\., 2017](https://arxiv.org/html/2610.33248#bib.bib10);[Wu et al\., 2018](https://arxiv.org/html/2610.33248#bib.bib11);[Jin et al\., 2018](https://arxiv.org/html/2610.33248#bib.bib12);[He et al\., 2021](https://arxiv.org/html/2610.33248#bib.bib13);[Wen et al\., 2022](https://arxiv.org/html/2610.33248#bib.bib14);[Guo et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib16)\)\. In production, however, a black\-box policy that outputs bids using advertisers’ budgets faces two stubborn obstacles\. First,*safety and explainability*: neural policies can fail catastrophically \(severe under\- or over\-delivery, cost performance degradation\), and such failures are hard to diagnose and remediate immediately and can lead to unrecoverable revenue impact and terrible advertiser experience\([Korenkevych et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib17)\)\. Second,*off\-policy learning*: because of safety concerns, online exploration with random bids is often unacceptable, so policies are constrained to be trained from logged data, exposing the agent to the full difficulty of offline RL\([Kumar et al\., 2020](https://arxiv.org/html/2610.33248#bib.bib18)\)\.

We presentRLBP\(*Reinforcement Learning for Bidding Parameters*\), the auto\-bidding pacing system deployed and evaluated with live traffic at Pinterest\. The central design choice is to*not*use RL to produce bids directly\. Instead, we retain the existing PID controller and use RL as a*meta\-controller*that configures the controller\. Specifically, at the beginning of each day and for each campaign, the RL agent observes the campaign state, produces a set of controller parameter multipliers and materializes these decisions as a lightweight lookup table for serving\. This table is fixed for the next 24 hours\. During online serving, the PID controller consumes the same table and continues to perform fine\-grained bid adjustment at the serving cadence\. We deliberately restrict the RL agent to operate on a small, bounded set of controller multipliers\. This confines its decisions to modest adjustments around a validated production configuration, rather than allowing it to generate bids directly or select unconstrained controller settings\. Beyond safety, this design is also lightweight in infrastructure cost, since only a flat lookup table—not a model artifact—is loaded at serving time\. Related designs include[Korenkevych et al\. \(2024\)](https://arxiv.org/html/2610.33248#bib.bib17), who use offline RL to optimize the parameters of an existing base policy at Meta, and[Yang et al\. \(2025\)](https://arxiv.org/html/2610.33248#bib.bib27), who apply deep RL to tune the ranking utility function at Pinterest\.RLBPshares the philosophy of augmenting an existing production bidder with a learned layer\.

Our main contributions are listed below:

- •A hybrid RL–PID architecture for production auto\-bidding\.We learn campaign\-specific PID parameter multipliers while retaining high\-frequency feedback control\. No model inference at serving time is required\.
- •Multi\-objective balancing without retraining\.We train a multi\-headQQ\-network for separate goals, budget utilization and cost efficiency, and then fuse the values at inference time for policy induction\. This design adapts the delivery–cost trade\-off dynamically based on campaign states\.
- •Relative\-lift rewards for clearer learning guidance\.Instead of using absolute values of the objective metrics \(e\.g\. revenue or cost in dollars\), we define rewards as the metric lift against the production baseline\. This transformation not only normalizes different metrics to a comparable scale but also reshapes the reward distribution and provides a clearer learning signal for the agent to improve policy performance\.
- •End\-to\-end production deployment and empirical validation at Pinterest\.We design an exploration–training–serving pipeline that integrates seamlessly into Pinterest’s production bidding system with negligible infrastructure overhead\. The online A/B test shows a statistically significant0\.25%0\.25\\%eCPC reduction with no revenue degradation\. The post\-rollout measurements against holdout traffic demonstrate consistent cost reduction and click\-through rate improvement\.

The remainder of the paper is organized as follows\. Section[2](https://arxiv.org/html/2610.33248#S2)reviews related work\. Section[3](https://arxiv.org/html/2610.33248#S3)describes the formulation of the meta\-controller MDP\. Section[4](https://arxiv.org/html/2610.33248#S4)details the central techniques including the multi\-head DQN, reward shaping, and adaptive fusion\. Section[5](https://arxiv.org/html/2610.33248#S5)covers production integration, and Section[6](https://arxiv.org/html/2610.33248#S6)presents offline and online experiments, followed by concluding remarks\.

## 2\.Related Work

### 2\.1\.Auto\-Bidding and Budget Pacing

Auto\-bidding is the problem of bidding in real\-time auctions on behalf of advertisers to maximize campaign value under budget and KPI constraints\([Aggarwal et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib1);[Balseiro et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib2);[Lucier et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib3)\)\. Classical analyses reduce the optimal bid to a scalar multiple of the predicted impression value, so the operational task reduces to online estimation of that scalar\([Wu et al\., 2018](https://arxiv.org/html/2610.33248#bib.bib11);[He et al\., 2021](https://arxiv.org/html/2610.33248#bib.bib13);[Yang et al\., 2019](https://arxiv.org/html/2610.33248#bib.bib8)\)\. Production systems realize this in two ways\. The*feedback\-control*lineage drives the running spend error to a pacing target via online feedback—from probabilistic throttling on a participation probability at LinkedIn\([Agarwal et al\., 2014](https://arxiv.org/html/2610.33248#bib.bib4)\), to PI\-style multiplicative bid control in Yahoo’s*Mystique*\([Stram et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib9)\), Alibaba’s multivariable controller\([Yang et al\., 2019](https://arxiv.org/html/2610.33248#bib.bib8)\), and the RTB work of[Zhang et al\. \(2016\)](https://arxiv.org/html/2610.33248#bib.bib6);[Karlsson \(2020\)](https://arxiv.org/html/2610.33248#bib.bib7)surveys this line\. The*optimization*lineage instead solves a parametric bid\-shading program online via dual\-based algorithms, such as those of Balseiro et al\.\([Balseiro et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib2)\)and Chen et al\.\([Chen et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib28)\)\.[Xu et al\. \(2015\)](https://arxiv.org/html/2610.33248#bib.bib5)take a complementary approach that allocates delivery rate across traffic segments\. Our work complements these pacing mechanisms by learning campaign\-specific controller settings and adapting the delivery–cost trade\-off over campaign lifecycles\.

### 2\.2\.Reinforcement Learning for Bidding

Starting with[Cai et al\. \(2017\)](https://arxiv.org/html/2610.33248#bib.bib10)’s MDP formulation of real\-time bidding, a substantial line of work has applied deep RL to produce bids directly\.[Wu et al\. \(2018\)](https://arxiv.org/html/2610.33248#bib.bib11)introduce reward shaping tailored for budget\-constrained bidding;[Jin et al\. \(2018\)](https://arxiv.org/html/2610.33248#bib.bib12)and[Wen et al\. \(2022\)](https://arxiv.org/html/2610.33248#bib.bib14)address the multi\-agent setting;[He et al\. \(2021\)](https://arxiv.org/html/2610.33248#bib.bib13)unify constrained bidding under a DDPG\-based controller;[Guo et al\. \(2024\)](https://arxiv.org/html/2610.33248#bib.bib16)reformulate bidding as generative trajectory modeling via diffusion\. On the offline\-RL side,[Kumar et al\. \(2020\)](https://arxiv.org/html/2610.33248#bib.bib18)provide the canonical CQL estimator,[Mou et al\. \(2022\)](https://arxiv.org/html/2610.33248#bib.bib15)propose an online RL framework with a safety\-constrained exploration policy that bridges the offline–online gap, and[Korenkevych et al\. \(2024\)](https://arxiv.org/html/2610.33248#bib.bib17)deploy at Meta a hybrid agent whose neural network is trained offline to adjust an existing base policy and then discarded at serving time\. In the Pinterest setting specifically,[Yang et al\. \(2025\)](https://arxiv.org/html/2610.33248#bib.bib27)apply deep RL to tune the*ranking utility*function, a different subsystem from pacing\. Across most of this line, the learned policy ultimately determines the bid itself, which creates deployment friction in safety\-sensitive production environments\.

### 2\.3\.Hybrid RL and Classical Control

Outside advertising, combining RL with classical controllers has a long history in robotics and process control: a line of work uses RL to*tune*the gains of a PID controller rather than replace it, trading off adaptivity against stability\([Lakhani et al\., 2022](https://arxiv.org/html/2610.33248#bib.bib21);[Bujgoi and Sendrescu, 2025](https://arxiv.org/html/2610.33248#bib.bib22);[van Niekerk et al\., 2025](https://arxiv.org/html/2610.33248#bib.bib23)\); the broader safe\-RL and constrained\-MDP literature studies explicit safety constraints\([Altman, 1999](https://arxiv.org/html/2610.33248#bib.bib19);[García and Fernández, 2015](https://arxiv.org/html/2610.33248#bib.bib20)\)\. The closest work to ours is[Korenkevych et al\. \(2024\)](https://arxiv.org/html/2610.33248#bib.bib17), who augment Meta’s production bidder with offline RL\. Our focus is daily, campaign\-specific selection over a discrete set of PID gain multipliers around the production configuration\. We further implement a multi\-headQQ\-network for separate business goals and use adaptive fusion to combine the values at inference time\.

Overall, our work sits at the intersection of the three lines above: we retain the PID feedback\-control structure, borrow offline training and IPS\-based screening\([Horvitz and Thompson, 1952](https://arxiv.org/html/2610.33248#bib.bib24);[Wang et al\., 2017](https://arxiv.org/html/2610.33248#bib.bib25)\)from the RL lineage, and validate the design through A/B tests on live Pinterest traffic\.

## 3\.Problem Formulation

This section formalizes the auto\-bidding problem as a reinforcement learning task\. We first describe how PID controllers are used in production bidding systems \([§3\.1](https://arxiv.org/html/2610.33248#S3.SS1)\), then present our meta\-controller MDP formulation where the RL agent learns to select PID controller parameters \([§3\.2](https://arxiv.org/html/2610.33248#S3.SS2)\)\.

### 3\.1\.PID Controller for Auto Bidding

Using PID controllers for online bidding is prevalent in industrial applications\([Agarwal et al\., 2014](https://arxiv.org/html/2610.33248#bib.bib4);[Zhang et al\., 2016](https://arxiv.org/html/2610.33248#bib.bib6);[Yang et al\., 2019](https://arxiv.org/html/2610.33248#bib.bib8);[Stram et al\., 2024](https://arxiv.org/html/2610.33248#bib.bib9)\)\. Based on campaign objectives, the controller can be designed to fulfill the budget smoothly or achieve desired cost performance\. The core idea is to track the error between the current performance metric \(e\.g\., spend rate, cost efficiency\) and a target value, then adjust the bid accordingly to minimize this error over time\.

Once the target is given, at time stepℓ\\ell, the controller adjusts the next bid valuebℓ\+1b\_\{\\ell\+1\}with

\(1\)bℓ\+1=f⁡\(bℓ,kp⋅e​r​r​Pℓ,ki⋅e​r​r​Iℓ,kd⋅e​r​r​Dℓ\),b\_\{\\ell\+1\}=f\(b\_\{\\ell\},~k\_\{p\}\\cdot errP\_\{\\ell\},~k\_\{i\}\\cdot errI\_\{\\ell\},~k\_\{d\}\\cdot errD\_\{\\ell\}\),where the functionffencapsulates the update formula, which typically follows a conventional summation or multiplicative approach\. The termse​r​r​Pℓ,e​r​r​Iℓ,e​r​r​DℓerrP\_\{\\ell\},errI\_\{\\ell\},errD\_\{\\ell\}represent the proportional, integral, and derivative error terms, respectively\. The parameterskp,ki,kd∈ℝ\+k\_\{p\},k\_\{i\},k\_\{d\}\\in\\mathbb\{R\}\_\{\+\}are known as the controller*coefficients*, which determine the relative influence of each error term on the bid adjustment\.

Although the PID controller has been adopted for many decades, it remains a practical first\-choice option for bidding systems due to its stability, interpretability, and effectiveness\. However, a critical challenge is dynamically selecting appropriate parameters: a fixed set of parameters cannot simultaneously accommodate the diverse characteristics of different campaigns \(e\.g\., budget size, lifetime, traffic patterns\) or adapt to dynamic market conditions\.

### 3\.2\.MDP for Controller\-based Bidding

We adopt the standard Markov decision process \(MDP\) formulation throughout the paper\. Specifically, an MDP is defined by a state space𝒮\{\\mathcal\{S\}\}, a finite action space𝒜\{\\mathcal\{A\}\}, a transition kernelP⁡\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\), a reward functionr:𝒮×𝒜→ℝr:\{\\mathcal\{S\}\}\\times\{\\mathcal\{A\}\}\\rightarrow\\mathbb\{R\}, and a discount factorγ∈\[0,1\)\\gamma\\in\[0,1\)\. A policyπ⁡\(a∣s\)\\pi\(a\\mid s\)specifies the probability of taking actiona∈𝒜a\\in\{\\mathcal\{A\}\}at states∈𝒮s\\in\{\\mathcal\{S\}\}, and induces the action\-value function

\(2\)Qπ\(s,a\):=𝔼π\[∑t=0∞γtr\(st,at\)\|s0=s,a0=a\]\.Q^\{\\pi\}\(s,a\):=\\mathbb\{E\}^\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{0\}=s,\\,a\_\{0\}=a\\right\]\.
The objective is to learn a policy that maximizes the expected cumulative discounted reward, or equivalently, the correspondingQQ\-values over the state\-action space\.

We formulate the online bidding process as an MDP, where the key design is to use RL as a*meta\-controller*which only selects PID parameters\(kp,ki,kd\)\(k\_\{p\},k\_\{i\},k\_\{d\}\)rather than directly producing bids\. In our implementation, the agent takes one step per day\. We usettto index daily RL decisions andℓ\\ellto index the controller updates in Eq\. \([1](https://arxiv.org/html/2610.33248#S3.E1)\)\. At the start of each day, the agent observes each campaign’s state and selects the corresponding parameters\(kp,ki,kd\)\(k\_\{p\},k\_\{i\},k\_\{d\}\)\. The PID controller then uses these parameters to update the campaign’s bids according to Eq\. \([1](https://arxiv.org/html/2610.33248#S3.E1)\) throughout that day\. Practitioners can use a different inference cadence as needed\.

State Space\.The statest∈𝒮s\_\{t\}\\in\{\\mathcal\{S\}\}encodes the campaign’s status at the beginning of daytt, including the campaign ID embedding, lifetime remaining days, daily budget, PID controller targets \(e\.g\., a campaign’s target spend curve\), and previous\-day aggregated metrics \(CTR, CPC, revenue\)\. These features collectively provide the agent with information about the campaign’s lifecycle stage, performance history, and operational constraints, enabling it to make informed decisions about appropriate controller tuning\.

Action Space\.Each actionat=\(mp,mi,md\)∈𝒜a\_\{t\}=\(m\_\{p\},m\_\{i\},m\_\{d\}\)\\in\{\\mathcal\{A\}\}specifies a multiplicative adjustment to the default controller parameters\(kp0,ki0,kd0\)\(k\_\{p\}^\{0\},k\_\{i\}^\{0\},k\_\{d\}^\{0\}\)\. Concretely, the deployed parameters are given by

\(3\)\(kp,ki,kd\)=\(mp⋅kp0,mi⋅ki0,md⋅kd0\)\.\(k\_\{p\},k\_\{i\},k\_\{d\}\)=\(m\_\{p\}\\cdot k\_\{p\}^\{0\},\\;m\_\{i\}\\cdot k\_\{i\}^\{0\},\\;m\_\{d\}\\cdot k\_\{d\}^\{0\}\)\.
In our implementation, we optimize only the proportional \(mpm\_\{p\}\) and integral multipliers \(mim\_\{i\}\), while keeping the derivative multiplier \(mdm\_\{d\}\) fixed at 1\. This design choice is motivated by the fact that the derivative error signale​r​r​DℓerrD\_\{\\ell\}is fairly noisy, making derivative control less reliable for policy optimization\. For each of the two tuned dimensions, the multiplier is chosen from the discrete set\{1−α,1,1\+α\}\{\\\{\}1\-\\alpha,\\,1,\\,1\+\\alpha\{\\\}\}, whereα∈\(0,1\)\\alpha\\in\(0,1\)controls the maximum deviation from the production baseline\. For example,α=0\.2\\alpha=0\.2corresponds to allowing±20%\\pm 20\\%adjustments around the default setting\. The resulting action space therefore contains32=93^\{2\}=9actions\. Both the number of actions and the allowable deviation are intentionally*kept small*to limit online exploration to bounded changes around the validated configuration\.

Transition Dynamics\.After the agent selects actionata\_\{t\}at the start of daytt, the PID controller uses the adjusted parameters from Eq\. \([3](https://arxiv.org/html/2610.33248#S3.E3)\) to serve bids throughout the day\. The next statest\+1s\_\{t\+1\}is determined by the aggregated outcomes of these bids: total spend, clicks, revenue, and remaining budget\. The transitionP⁡\(st\+1\|st,at\)P\(s\_\{t\+1\}\|s\_\{t\},a\_\{t\}\)is stochastic due to the inherent randomness in user behavior, auction competition, and ad delivery dynamics\. Importantly, the environment is*non\-stationary*: market conditions, user engagement patterns, and competitor strategies evolve daily, making adaptive policies crucial for sustained performance\.

Reward Function\.The rewardr⁡\(st,at\)r\(s\_\{t\},a\_\{t\}\)quantifies the campaign’s performance over dayttunder the selected PID parameters\. We aim to balance two primary objectives:

1. \(1\)Maximizing budget utilization \(BU\): Spending the campaign’s allocated budget to maximize revenue delivery;
2. \(2\)Minimizing cost per click \(CPC\): Achieving cost\-efficient delivery to meet advertiser ROI expectations\.

Note that these two objectives can easily conflict with each other\. Higher bids can increase delivery at the expense of higher CPC, while lower bids may improve cost efficiency but leave part of the budget unspent\. The intricacy here is to balance the two objectives based on campaign states, which we will detail later\.

Discount Factor\.In our deployment, we instantiate this RL framework withγ=0\\gamma=0, yielding a one\-step contextual\-bandit objective based on daily outcomes\. This choice avoids bootstrapping errors in offline training, and the cross\-day trade\-off between cost efficiency and delivery is instead handled by the lifecycle\-dependent fusion weight \(see[§4\.2](https://arxiv.org/html/2610.33248#S4.SS2)\)\. It is a deployment choice rather than a restriction of the meta\-controller architecture: practitioners can useγ∈\(0,1\)\\gamma\\in\(0,1\)and train theQQ\-heads to estimate discounted returns for longer\-term optimization, while retaining the state, action, and controller interface\. We discuss this extension in[§7](https://arxiv.org/html/2610.33248#S7)\.

## 4\.Proposed Methods

In this section, we presentRLBP, an RL framework for ads bidding that jointly optimizes budget utilization and cost efficiency\. We first describe the multi\-headQQ\-networks \([§4\.1](https://arxiv.org/html/2610.33248#S4.SS1)\), then explain how its predictions are fused \([§4\.2](https://arxiv.org/html/2610.33248#S4.SS2)\) and converted into a bidding\-control policy \([§4\.3](https://arxiv.org/html/2610.33248#S4.SS3)\)\. Finally, we specify the relative\-lift rewards used to train the agent \([§4\.4](https://arxiv.org/html/2610.33248#S4.SS4)\)\.

### 4\.1\.Multi\-headQQ\-Networks

Since we have multiple objectives for the agent to optimize, we define separate reward functions, e\.g\., reward for budget utilization and reward for cost reduction\. A conventional approach is to combine these rewards into a scalar and train aQQ\-network to predict the resulting value\. However, in production ads bidding systems, the relative importance of different optimization objectives often changes over the lifetime of a campaign\. Early in a campaign, since unspent budget can roll over to future days, we can prioritize cost performance\. Near the final day of a campaign, spending the remaining budget becomes a more immediate concern\. As a result, the desired trade\-off between cost efficiency and revenue generation is inherently dynamic rather than fixed\. Such a requirement of flexibility in objective weighting can be troublesome under a conventional RL paradigm where the reward is a fixed function\. To obtain different policies, practitioners often need to redefine the reward and retrain the model, which is computationally expensive and operationally inconvenient in a production environment\.

To address this issue, we propose a multi\-headQQ\-network\. The model feeds a common state representation into multiple separate branches, each predictingQQ\-values for one objective\. As illustrated in[Figure1](https://arxiv.org/html/2610.33248#S4.F1), we have a head for budget utilization that outputsQb​u​\(s,a\)Q^\{bu\}\(s,a\)and a head for CPC that outputsQc​p​c​\(s,a\)Q^\{cpc\}\(s,a\)\. The outputs are fused only at inference to derive a policy\. Depending on how the predictedQQ\-values are fused, we can generate different policies from a single trained model\.

Repeat×3\\times 3State RepresentationLinear layerLayer NormLeaky ReLUDropoutLinear LayerLinear layerLayer NormLeaky ReLUDropoutLinear LayerQbuQ^\{\\mathrm\{bu\}\}HeadQcpcQ^\{\\mathrm\{cpc\}\}HeadInference onlyAdaptive Fusion

Figure 1\.Multi\-headQQ\-network architecture\.#### 4\.1\.1\.Loss Functions

The total loss comprises temporal difference losses from all heads\. In our case, given a training dataset\{\(si,ai,rib​u,ric​p​c,si′\)\}i=1N\\\{\(s\_\{i\},a\_\{i\},r^\{bu\}\_\{i\},r^\{cpc\}\_\{i\},s^\{\\prime\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, the per\-objective losses and the total loss respectively are

\(4\)Lb​u​\(θ\)=1N​∑i=1N\(Qθb​u​\(si,ai\)−rib​u−γ⋅maxa∈𝒜⁡Qθb​u​\(si′,a\)\)2\\displaystyle L^\{bu\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(Q\_\{\\theta\}^\{bu\}\(s\_\{i\},a\_\{i\}\)\-r\_\{i\}^\{bu\}\-\\gamma\\cdot\\max\_\{a\\in\{\\mathcal\{A\}\}\}Q\_\{\\theta\}^\{bu\}\(s^\{\\prime\}\_\{i\},a\)\\right\)^\{2\}\(5\)Lc​p​c​\(θ\)=1N​∑i=1N\(Qθc​p​c​\(si,ai\)−ric​p​c−γ⋅mina∈𝒜⁡Qθc​p​c​\(si′,a\)\)2\\displaystyle L^\{cpc\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(Q\_\{\\theta\}^\{cpc\}\(s\_\{i\},a\_\{i\}\)\-r\_\{i\}^\{cpc\}\-\\gamma\\cdot\\min\_\{a\\in\{\\mathcal\{A\}\}\}Q\_\{\\theta\}^\{cpc\}\(s^\{\\prime\}\_\{i\},a\)\\right\)^\{2\}\(6\)Lt​o​t​a​l​\(θ\)=Lb​u​\(θ\)\+Lc​p​c​\(θ\)\\displaystyle L^\{total\}\(\\theta\)=~L^\{bu\}\(\\theta\)\+L^\{cpc\}\(\\theta\)whereθ\\thetadenotes the model parameters;rib​ur\_\{i\}^\{bu\}andric​p​cr\_\{i\}^\{cpc\}are the budget utilization and CPC rewards;NNis the total number of training examples\. The CPC head usesmin\\minin its Bellman backup because lower CPC lift indicates better cost performance\. In our deployment, withγ=0\\gamma=0, eachQQ\-head is trained to fit the corresponding daily rewards\.

#### 4\.1\.2\.State Representation

The state representation is constructed from the state definition in[§3](https://arxiv.org/html/2610.33248#S3)\. Before being fed into theQQ\-network, thecampaign’s target spend curveis normalized into an hourly distribution, and budget features are log\-transformed\. Categorical features are embedded, while all vector and scalar numerical features are concatenated and batch\-normalized\. The final state representation is the concatenation of the categorical embeddings and normalized numerical feature block\.

### 4\.2\.Inference\-time AdaptiveQQ\-Value Fusion

We define the fusedQQ\-value as a fusion of the two heads:

\(7\)Q^​\(s,a\)=Qb​u​\(s,a\)−β⁡\(d\)⋅Qc​p​c​\(s,a\)\\hat\{Q\}\(s,a\)=Q^\{bu\}\(s,a\)\\mathbin\{\-\}\\beta\(d\)\\cdot Q^\{cpc\}\(s,a\)wheredddenotes the number of remaining days in the campaign’s lifetime, andβ⁡\(d\)≥0\\beta\(d\)\\geq 0is a cost\-penalty weight that balances budget utilization against cost reduction\. The negative sign reflects that we seek to maximize budget utilization while minimizing CPC\. We designβ⁡\(d\)\\beta\(d\)to decrease asddapproaches zero, progressively reducing the cost penalty and shifting the policy’s emphasis toward budget utilization as the campaign approaches its deadline\. We compare alternative adaptive weighting functions in[§6\.1\.2](https://arxiv.org/html/2610.33248#S6.SS1.SSS2)\.

This inference\-time adaptive fusion design provides two main advantages\. First, it separates outcome prediction from business preferences by learning each objective separately, avoiding the difficulty of encoding a changing trade\-off in a single reward function\. Second, it offers practical flexibility in deployment, since the trade\-off can be adjusted directly through the fusion weightβ⁡\(d\)\\beta\{\(d\)\}as campaign conditions evolve\. This makesRLBPbetter suited for real\-world ad systems, where optimization priorities are often non\-stationary and must be updated in a lightweight and interpretable manner\.

### 4\.3\.Policy Induction

Given the fused scores in[§4\.2](https://arxiv.org/html/2610.33248#S4.SS2), we construct a policy over the discrete joint action space\.

Rather than using a greedy policy that always selects the highest\-scoring action, we adopt a top\-KKstochastic policy\. Specifically, the policy retains theKKactions with the highest fused scores, forming the state\-dependent candidate setA′​\(s\)A^\{\\prime\}\(s\)\. A temperature\-scaled softmax is applied over this reduced action set:

\(8\)π⁡\(a\|s\)=exp⁡\(Q^​\(s,a\)/τ\)∑a′∈A′​\(s\)exp⁡\(Q^​\(s,a′\)/τ\)for​a∈A′​\(s\)\.\\pi\(a\|s\)=\\frac\{\\exp\\left\(\\hat\{Q\}\(s,a\)/\\tau\\right\)\}\{\\sum\\limits\_\{a^\{\\prime\}\\in A^\{\\prime\}\(s\)\}\\exp\\left\(\\hat\{Q\}\(s,a^\{\\prime\}\)/\\tau\\right\)\}\\qquad\\text\{for~\}a\\in A^\{\\prime\}\(s\)\.whereQ^​\(s,a\)\\hat\{Q\}\(s,a\)is the fused score after numerical preprocessing andτ\>0\\tau\>0is the temperature; the probability is zero outsideA′​\(s\)A^\{\\prime\}\(s\)\. The final bidding\-control decision is then sampled from this distribution\.

### 4\.4\.Reward Shaping

We now specify the daily rewards used to train the multi\-headQQ\-networks in[§4\.1](https://arxiv.org/html/2610.33248#S4.SS1)\.

Learning directly from raw business metrics, especially CPC, presents significant challenges\. Since the CPC difference between candidate actions is often small relative to its absolute value \(for example, $1\.01 versus $1\.00 per click\), training theQQ\-network to accurately rank these actions can be difficult\. Furthermore, CPC ranges vary widely across campaigns, and campaigns with larger absolute prediction errors can disproportionately dominate the loss function during training\.

An additional challenge stems from the varying scales across distinct reward signals\. For instance, CPC values may span from 0 to 100, whereas budget utilization remains strictly bounded within the interval\[0,1\]\\left\[0,1\\right\]\. Such scale discrepancies generate imbalanced gradient signals backpropagated to the shared network layers during training, while simultaneously introducing complexity when tuning fusion weights at inference\.

![Two histograms show the relative gap between the best and second-best rewards. Absolute CPC gaps concentrate near zero; CPC lift gaps span more of the interval.](https://arxiv.org/html/2610.33248v1/content/resources/rel_gap_hist.png)Figure 2\.Reward relative gap distribution comparison between absolute metric values vs\. metric lifts\. The left panel shows the relative difference between the best and second\-best rewards constructed with raw CPC, while the right panel shows the corresponding relative difference by using CPC lift\.Two histograms show the relative gap between the best and second\-best rewards\. Absolute CPC gaps concentrate near zero; CPC lift gaps span more of the interval\.For policy selection, the key question is how candidate actions compare under the two objectives\. We therefore shape the reward to the metric lift over the production baseline rather than the raw metric value\. Formally, letv⁡\(s,a\)v\(s,a\)denote the observed metric \(e\.g\., CPC\) under statessand actionaa, and letvp​r​o​d​\(s\)v^\{prod\}\(s\)denote the corresponding metric estimated from matched observations under the production baseline\. CPC is total spend divided by total clicks, whereas BU is total spend divided by allocated budget\. Instead of usingv⁡\(s,a\)v\(s,a\)directly as the target reward, we define the reward as:

\(9\)r⁡\(s,a\)=v⁡\(s,a\)−vp​r​o​d​\(s\)vp​r​o​d​\(s\)r\(s,a\)=\\frac\{v\(s,a\)\-v^\{prod\}\(s\)\}\{v^\{prod\}\(s\)\}Applied to BU and CPC, this definition yields the two training rewardsrb​ur^\{bu\}andrc​p​cr^\{cpc\}used in[§4\.1](https://arxiv.org/html/2610.33248#S4.SS1), so that eachQQ\-function learns the relative metric change of an action over the baseline system\. Their outputs are then combined through the fusion rule to construct the action\-selection policy\.\.

This reward shaping provides two advantages\. First, it centers targets around the existing controller and expresses deviations on a relative scale\. Under squared\-error training, normalization by the referencevp​r​o​d​\(s\)v^\{prod\}\(s\)weights squared prediction errors by1/\(vp​r​o​d​\(s\)\)21/\(v^\{prod\}\(s\)\)^\{2\}, emphasizing relative discrepancies across campaigns\.[Figure2](https://arxiv.org/html/2610.33248#S4.F2)illustrates the relative action\-gap distributions under the two reward formulations\. In our offline study, we also evaluate their effect on policy outcomes; see[Table1](https://arxiv.org/html/2610.33248#S6.T1)\. Second, the lift values of different metrics, such as CPC and BU, tend to lie on more comparable scales than their raw absolute values\. This makes it easier to balance multiple objectives in the inference\-timeQQ\-value fusion discussed in[§4\.2](https://arxiv.org/html/2610.33248#S4.SS2), since the combination weights no longer need to compensate for large differences in metric magnitude\.

Overall, by shaping the reward as metric lift,RLBPlearns baseline\-relative action values rather than absolute metric values\. This design aligns target scales across campaigns and simplifies multi\-objective trade\-off control at inference time\.

## 5\.Production Integration

This section describes the integration ofRLBPinto Pinterest’s production auction and bidding system\. As illustrated in[Figure3](https://arxiv.org/html/2610.33248#S5.F3), the pipeline comprises three stages: online exploration\-and\-exploitation, offline training, and daily serving\.

Online Exploitation\-ExplorationOffline TrainingDaily ServingActionAssignmentTransition Log\(s,a,r,s′\)\(s,a,r,s^\{\\prime\}\)Batch DQNTrainingDailyInference\(kp,ki,kd\)\(k\_\{p\},k\_\{i\},k\_\{d\}\)TableProductionBiddera∼Unif⁡\(𝒜\)a\\sim\\mathrm\{Unif\}\(\{\\mathcal\{A\}\}\)a∼πa\\sim\\piQ^\\hat\{Q\}\-fusionFigure 3\.End\-to\-end training\-serving pipeline ofRLBP\.##### Online exploration\-and\-exploitation\.

Training the DQN requires logged tuples\(st,at,rtb​u,rtc​p​c,st\+1\)\(s\_\{t\},a\_\{t\},r\_\{t\}^\{bu\},r\_\{t\}^\{cpc\},s\_\{t\+1\}\)collected under a known logging policy\. While serving the learned policy, we reserve a dedicated portion of campaign traffic and budget for uniform random exploration\. Specifically, each campaign’s budget is split into two segments: the primary portion is allocated to the RL policy for exploitation, while the remaining portion is used for the uniform random exploration, with assignments refreshed daily\. Each segment uses distinct user buckets and separate budgets and bids\. This separation allows exploratory actions to be evaluated within their allocated traffic and budget\.

##### Offline training\.

A batch training job periodically aggregates the transition samples over a historical time window and retrains theQQ\-network\. The model learns separateQQ\-functions for the target objectives using the reward formulation described in Section[4](https://arxiv.org/html/2610.33248#S4)\. Once training is complete, the resulting checkpoint is used to generate serving\-time decisions for active campaigns\.

##### Daily serving\.

A scheduled job runsRLBPonce per day over all eligible campaigns: it computesQQ\-values for everya∈𝒜a\\in\{\\mathcal\{A\}\}, fuses the two heads by the adaptive rule in[§4\.2](https://arxiv.org/html/2610.33248#S4.SS2), selects an action under the top\-KKstochastic policy of[§4\.3](https://arxiv.org/html/2610.33248#S4.SS3), and materializes the chosen multipliers as a campaign\-level lookup table \([Figure4](https://arxiv.org/html/2610.33248#S5.F4)\)\. The production bidder then adjusts bids on the order of seconds, fetching the corresponding controller parameters from that table and applying them through the existing controller; it never loads the DQN or invokes the agent at request time, which minimizes inference latency\.

Daily Inference \(00:00\)Online ServingCampaignStateRL Agent\(DQN\)ControllerParameters\(kp,ki,kd\)\(k\_\{p\},k\_\{i\},k\_\{d\}\)Parameter Lookup TablePIDControllerCampaignSignalsBidAdjustmentobserveselect actionone\-time write per day\(kp,ki,kd\)\(k\_\{p\},k\_\{i\},k\_\{d\}\)Figure 4\.Two\-layer RL\+PID architecture\. The RL agent observes campaign state once daily and selects controller gains from a small, bounded set\. The PID controller uses those fixed gains for all bid adjustments over the next 24 hours, retaining the existing feedback\-control structure\.

## 6\.Experiments

In this section, we evaluateRLBPboth offline and online\. In the offline study, we evaluate the design choices and screen candidate configurations against the production baseline\. In the online experiment, we validated the efficacy ofRLBPthrough A/B testing on Pinterest’s production advertising platform\.

### 6\.1\.Offline Study

#### 6\.1\.1\.IPS\-based offline lift

We use 21 days of transition data for training and 7 days of held\-out data for evaluation\. Letπ^​\(a∣s\)\\hat\{\\pi\}\(a\\mid s\)denote the known logging propensity\. Suppose the held\-out dataset containsNNexamples,\{\(si,ai,ri,si′\)\}i=1N\\\{\(s\_\{i\},a\_\{i\},r\_\{i\},s^\{\\prime\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, wheresis\_\{i\}is the state,aia\_\{i\}is the action,rir\_\{i\}is the observed reward, andsi′s^\{\\prime\}\_\{i\}is the next state\. For a policyπ\\pi, we can estimate its value with IPS:

\(10\)IPS⁡\(π\)=1N​∑i=1Nπ⁡\(ai∣si\)π^​\(ai∣si\)⋅ri\\mathrm\{IPS\}\(\\pi\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{\\pi\(a\_\{i\}\\mid s\_\{i\}\)\}\{\\hat\{\\pi\}\(a\_\{i\}\\mid s\_\{i\}\)\}\\cdot r\_\{i\}The offline results below use uniformly randomized exploration logs, withπ^​\(a∣s\)=1/\|𝒜\|\\hat\{\\pi\}\(a\\mid s\)=1/\|\\mathcal\{A\}\|\. After rollout \([§6\.3](https://arxiv.org/html/2610.33248#S6.SS3)\), the logging policy mixes the deployed policy with uniform exploration\.

To measure the improvements, we compare the IPS of an offline\-trained policyπ\\piagainst the production policyπprod\\pi\_\{\\text\{prod\}\}, i\.e\.,

\(11\)lift⁡\(π\)=IPS⁡\(π\)−IPS⁡\(πprod\)IPS⁡\(πprod\)\.\\mathrm\{lift\}\(\\pi\)=\\frac\{\\mathrm\{IPS\}\(\\pi\)\-\\mathrm\{IPS\}\(\\pi\_\{\\mathrm\{prod\}\}\)\}\{\\mathrm\{IPS\}\(\\pi\_\{\\mathrm\{prod\}\}\)\}\.In the following section, we present the IPS lift achieved by each technique\. Because our framework incorporates dual reward functions—one designed to boost budget utilization and the other to minimize costs—we report the corresponding IPS lifts separately for each objective\.

#### 6\.1\.2\.Ablation study

We isolate the contribution of two core techniques ofRLBP: the relative\-lift reward formulation \([§4\.4](https://arxiv.org/html/2610.33248#S4.SS4)\) and the choice of adaptive functionβ⁡\(d\)\\beta\(d\)in the inference\-timeQQ\-fusion \([§4\.2](https://arxiv.org/html/2610.33248#S4.SS2)\)\. In each ablation, all other components are held fixed atRLBP’s deployed configuration and results are averaged across five training runs with different random seeds\. Bold rows mark the production choice\.

Table 1\.Reward formulation ablation\. Each row reports IPS lift \(%\) over the production baseline\. Revenue lift is evaluated on the final day of the campaign lifetime \(d=0d=0\), where the policy prioritizes budget utilization; CPC lift is evaluated ond\>0d\>0, where cost efficiency is the primary objective\.Boldmarks the production choice\.##### Reward Shaping\.

We compare training with absolute metric values against relative lift as the reward signal\.[Table1](https://arxiv.org/html/2610.33248#S6.T1)quantifies the final policy quality: relative lift achieves2\.77%2\.77\\%revenue lift on the final campaign day and3\.02%3\.02\\%CPC reduction on earlier days, compared to0\.08%0\.08\\%and2\.17%2\.17\\%for absolute rewards\. The difference in revenue\-lift is especially pronounced\.

Table 2\.Adaptive function ablation\. Each row reports IPS lift \(%\) on a cohort of campaigns tracked across their full last\-seven\-day lifecycle\.Boldmarks the production choice\.
##### Adaptive function\.

We compare three forms of the fusion weightβ⁡\(d\)\\beta\(d\)\([Figure5](https://arxiv.org/html/2610.33248#S6.F5)a\): a*step*function switching abruptly atd=1d=1, a*linear*rampβ⁡\(d\)∝d\\beta\(d\)\\propto d, and a*sigmoid*between them\. All increase cost emphasis with remaining days, differing in their transition profiles\.[Figure5](https://arxiv.org/html/2610.33248#S6.F5)b shows the consequence for policy behavior: greater cost emphasis yields more CPC\-reducing actions, gradually under the linear schedule and abruptly under the step\.

![Refer to caption](https://arxiv.org/html/2610.33248v1/content/figures/adaptive_function_shapes.png)Figure 5\.\(a\) Three adaptive functionsβ⁡\(d\)\\beta\(d\)controlling cost emphasis as a function of lifetime remaining days\. \(b\) Resulting learned policy behavior: fraction of campaigns assigned a cost\-reduction action under each adaptive function\.[Table2](https://arxiv.org/html/2610.33248#S6.T2)reports the overall IPS lift on a cohort of campaigns in the last week of their lifetime, and[Figure6](https://arxiv.org/html/2610.33248#S6.F6)breaks down CPC and revenue lift by lifetime remaining days across the same cohort\. Day\-level lifts vary across the lifecycle\. The linear schedule is the only evaluated variant with both nonnegative revenue lift \(\+0\.05%\+0\.05\\%\) and reduced CPC \(−0\.34%\-0\.34\\%\) in the point estimates \([Table2](https://arxiv.org/html/2610.33248#S6.T2)\), so we adopt it in production\. Sigmoid has higher revenue \(\+1\.41%\+1\.41\\%\) alongside higher CPC \(\+0\.49%\+0\.49\\%\)\.

![Refer to caption](https://arxiv.org/html/2610.33248v1/content/resources/ips_by_remaining_days_cohort.png)Figure 6\.Offline IPS CPC and revenue lift \(%\) by lifetime remaining days for different adaptive functions, evaluated on campaigns in the last seven days of their lifetime\.

### 6\.2\.Online A/B Experiments

We evaluateRLBPin a seven\-day live A/B test using the following deployed configuration: relative\-lift reward, linear adaptive fusion, and top\-KKsoftmax inference\. All experiment arms share the same serving infrastructure; the production baseline uses the default controller with no RL\-driven adjustment\. Against this baseline, we report revenue, clicks, and effective cost\-per\-click \(eCPC\)\. We report results at two levels of aggregation:all campaigns, which includes every treated campaign\-day and reflects overall online impact, andlifetime\-ended campaigns, restricted to campaigns whose lifetime completed during the experiment\. The latter captures campaign\-end behavior using observations within the test window\.

#### 6\.2\.1\.Overall results

[Table3](https://arxiv.org/html/2610.33248#S6.T3)summarizes online performance at two levels of aggregation\. Across all treated campaign\-days,RLBPachieves a statistically significant0\.25%0\.25\\%reduction in eCPC \(p<0\.05p<0\.05\) while keeping revenue approximately unchanged, indicating that the learned policy routes spend more cost\-efficiently\. For lifetime\-ended campaigns, revenue and budget utilization are particularly relevant because the opportunity to spend the remaining budget ends with the campaign\. In this cohort, we observe a\+0\.43%\+0\.43\\%revenue lift and\+0\.25%\+0\.25\\%click lift, alongside\+0\.18%\+0\.18\\%eCPC lift\. This pattern is consistent with the intended shift toward delivery near campaign end\.

Table 3\.Online A/B results\. Bold numbers are statistically significant \(pp\-value<0\.05<0\.05\)\.
#### 6\.2\.2\.Segmented analysis

We further break results down by budget level and historical CPC tier \([Table4](https://arxiv.org/html/2610.33248#S6.T4)\)\. eCPC point estimates are negative across all reported segments, with the largest gains in the high\-budget \(−0\.90%\-0\.90\\%eCPC,\+0\.81%\+0\.81\\%clicks\) and high\-CPC \(−0\.64%\-0\.64\\%eCPC\) segments where there is the most room for optimization\.

Table 4\.eCPC and click lifts in online A/B experiments, segmented by budget level and historical CPC tier\. Bold numbers are statistically significant \(pp\-value<0\.05<0\.05\)\.

### 6\.3\.Post\-Rollout Performance

Following the online A/B experiments in[§6\.2](https://arxiv.org/html/2610.33248#S6.SS2), we measuredRLBP’s performance over two full months after rollout at Pinterest against a small slice of*holdout control*traffic using default parameters\(kp0,ki0,kd0\)\(k\_\{p\}^\{0\},k\_\{i\}^\{0\},k\_\{d\}^\{0\}\)\. This extends the evaluation beyond the seven\-day A/B test to a larger campaign population\.

##### Measurements\.

We divide lifetime\-budget click\-optimized campaigns into two groups for evaluation: campaign\-days with seven or more remaining days of lifetime, and those with fewer\. This division aligns with our linear adaptive fusion deployment, in which the weighting progressively transitions from cost efficiency to budget utilization starting when seven days remain \([§4\.2](https://arxiv.org/html/2610.33248#S4.SS2)\)\. Since the holdout control has much less traffic and budget, we compare BU \(instead of absolute revenue\), eCPC, and click\-through rate \(CTR\)\. Within each group we sum spend, budget, clicks and impressions over the campaigns active on each day, form that day’s metrics, and average across days with equal weight\. Due to the imbalanced traffic, no statistical test is conducted\.

##### Results\.

The results for the two lifecycle groups are consistent with the linear adaptive fusion function \(see[Table5](https://arxiv.org/html/2610.33248#S6.T5)\)\. Where cost receives the greatest emphasis,RLBPachieves nearly1%1\\%lower eCPC, with slightly lower BU than the control; where the weighting turns toward BU, delivery is above the control while eCPC remains lower\. These patterns are consistent with both earlier offline and online evaluations: offline, the linear schedule was selected because it reduced CPC while yielding nonnegative revenue lift; online, the A/B test showed an eCPC reduction overall, with a positive revenue\-lift point estimate among lifetime\-ended campaigns\. These figures are a directional reference on long\-run behavior, with[§6\.2](https://arxiv.org/html/2610.33248#S6.SS2)remaining the controlled estimate\.

Table 5\.Post\-rollout performance ofRLBPrelative to the control holdout by two lifecycle stages\. BU denotes budget utilization \(delivered spend over budget\)\.

## 7\.Discussion

In this section, we discuss several directions for extending the RLBP framework in industrial systems\.

##### Multi\-step learning\.

As a first step toward industrial deployment, we instantiate RLBP withγ=0\\gamma=0\. Nevertheless, this framework is not confined to a single\-step setup\. The meta\-controller architecture, the reward design, and the multi\-headQQ\-network \(trained with temporal\-difference errors\) with adaptive inference\-time fusion all extend naturally to multi\-step settings, enabling long\-horizon optimization\. One caveat in the multi\-step setting is that each head bootstraps through its own greedy policy, so the fusedQQ\-value is generally not the optimalQQ\-value of the combined reward\. Fusion then serves as a practical heuristic, and developing fusion schemes with formal guarantees is a promising direction for future work\.

##### Expanded action space\.

In this work, we limit RL actions to a small number of multipliers around the production default parameters mainly due to safety considerations\. This narrows down the scope of exploration and simplifies deployment, at the cost of adjustment precision and range\. A potential next step is to experiment with a finer action grid or continuous gain multipliers for more precise tuning, but this would require broader exploration coverage, additional safeguards, and more robust offline evaluation\.

##### More frequent inference\.

Another direction of improvement is to increase the inference frequency\. Currently, inference occurs once per day as an initial implementation\. We are actively exploring shorter cadences, such as hourly or minute\-level updates\. Elevating the inference rate enables the state representation to track intra\-day fluctuations, traffic spikes, and shifting market dynamics with greater precision\. This allows the RL agent to adjust decisions more dynamically, thereby reducing suboptimal performance driven by update delays\. However, transitioning toward near\-real\-time serving renders offline lookup tables unusable, requiring online model deployment and leading to higher infrastructure costs\.

## 8\.Conclusion

We presentedRLBP, an RL meta\-controller that augments \(rather than replaces\) a production PID auto\-bidder, providing a practical path for deploying RL in safety\-sensitive advertising systems\.RLBPintroduces a multi\-head DQN with inference\-time adaptive fusion and a relative\-lift reward formulation\. Deployed at Pinterest,RLBPachieves statistically significant eCPC reductions with approximately unchanged revenue in online A/B experiments, demonstrating that RL can deliver measurable value on live advertising traffic at scale\. Future directions built upon the current framework include extending to multi\-step optimization for longer\-lived campaigns, relaxing to continuous action spaces, and generalizing to additional campaign types and controller parameters\.

###### Acknowledgements\.

We would like to express our sincere gratitude to all those who contributed to the development of our RLBP model and pipeline\. We especially thank Akanksha Baid, Adam Fer, and Amol Goyal for their general support in delivering the project; Aniket Ketkar, Sameer Bhide, Jorge Alberto Bayona Romo, and Jen\-An Lien for providing guidance on infrastructure implementation; Ludek Cigler and Christina Huang for sharing their experience with real\-time bidding\. Their expertise, dedication, and collaboration were essential to the success of this project\.

## References

- Agarwalet al\.\(2014\)D\. Agarwal, S\. Ghosh, K\. Wei, and S\. YouBudget pacing for targeted online advertisements at LinkedIn\.InProceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining \(KDD\),New York, NY, USA,pp\. 1613–1619\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.33248#S3.SS1.p1.1)\.
- Aggarwalet al\.\(2024\)G\. Aggarwal, A\. Badanidiyuru, S\. R\. Balseiro, K\. Bhawalkar, Y\. Deng, Z\. Feng, G\. Goel, C\. Liaw, H\. Lu, M\. Mahdian, J\. Mao, A\. Mehta, V\. Mirrokni, R\. Paes Leme, A\. Perlroth, G\. Piliouras, J\. Schneider, A\. Schvartzman, B\. Sivan, K\. Spendlove, Y\. Teng, D\. Wang, H\. Zhang, M\. Zhao, W\. Zhu, and S\. ZuoAuto\-bidding and auctions in online advertising: a survey\.Note:arXiv preprint arXiv:2408\.07685Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p1.1),[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1)\.
- Altman \(1999\)E\. AltmanConstrained Markov decision processes\.CRC Press\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p1.1)\.
- Balseiroet al\.\(2024\)S\. R\. Balseiro, K\. Bhawalkar, Z\. Feng, H\. Lu, V\. Mirrokni, B\. Sivan, and D\. WangA field guide for pacing budget and ROS constraints\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235,Vienna, Austria,pp\. 2607–2638\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p1.1),[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1)\.
- Bujgoi and Sendrescu \(2025\)G\. Bujgoi and D\. SendrescuTuning of PID controllers using reinforcement learning for nonlinear system control\.Processes13\(3\),pp\. 735\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p1.1)\.
- Caiet al\.\(2017\)H\. Cai, K\. Ren, W\. Zhang, K\. Malialis, J\. Wang, Y\. Yu, and D\. GuoReal\-time bidding by reinforcement learning in display advertising\.InProceedings of the Tenth ACM International Conference on Web Search and Data Mining \(WSDM\),New York, NY, USA,pp\. 661–670\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Chenet al\.\(2024\)Q\. Chen, P\. H\. Nguyen, and D\. GligorijevicOptimization\-based budget pacing in eBay sponsored search\.InCompanion Proceedings of the ACM Web Conference 2024 \(WWW Companion\),New York, NY, USA,pp\. 328–337\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1)\.
- García and Fernández \(2015\)J\. García and F\. FernándezA comprehensive survey on safe reinforcement learning\.Journal of Machine Learning Research16\(1\),pp\. 1437–1480\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p1.1)\.
- Guoet al\.\(2024\)J\. Guo, Y\. Huo, Z\. Zhang, T\. Wang, C\. Yu, J\. Xu, Y\. Zhang, and B\. ZhengAIGB: generative auto\-bidding via conditional diffusion modeling\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\),New York, NY, USA,pp\. 5038–5049\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Heet al\.\(2021\)Y\. He, X\. Chen, D\. Wu, J\. Pan, Q\. Tan, C\. Yu, J\. Xu, and X\. ZhuA unified solution to constrained bidding in online display advertising\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\),New York, NY, USA,pp\. 2993–3001\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Horvitz and Thompson \(1952\)D\. G\. Horvitz and D\. J\. ThompsonA generalization of sampling without replacement from a finite universe\.Journal of the American Statistical Association47\(260\),pp\. 663–685\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p2.1)\.
- Jinet al\.\(2018\)J\. Jin, C\. Song, H\. Li, K\. Gai, J\. Wang, and W\. ZhangReal\-time bidding with multi\-agent reinforcement learning in display advertising\.InProceedings of the 27th ACM International Conference on Information and Knowledge Management \(CIKM\),New York, NY, USA,pp\. 2193–2202\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Karlsson \(2020\)N\. KarlssonFeedback control in programmatic advertising: the frontier of optimization in real\-time bidding\.IEEE Control Systems Magazine40\(5\),pp\. 40–77\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1)\.
- Korenkevychet al\.\(2024\)D\. Korenkevych, F\. Cheng, A\. Balakir, A\. Nikulkov, L\. Gao, Z\. Cen, Z\. Xu, and Z\. ZhuOffline reinforcement learning for optimizing production bidding policies\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\),New York, NY, USA\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§1](https://arxiv.org/html/2610.33248#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p1.1)\.
- Kumaret al\.\(2020\)A\. Kumar, A\. Zhou, G\. Tucker, and S\. LevineConservative Q\-learning for offline reinforcement learning\.InAdvances in Neural Information Processing Systems 33 \(NeurIPS\),Red Hook, NY, USA\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Lakhaniet al\.\(2022\)A\. I\. Lakhani, M\. A\. Chowdhury, and Q\. LuStability\-preserving automatic tuning of PID control with reinforcement learning\.Complex Engineering Systems2\(1\)\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p1.1)\.
- Lucieret al\.\(2024\)B\. Lucier, S\. Pattathil, A\. Slivkins, and M\. ZhangAutobidders with budget and ROI constraints: efficiency, regret, and pacing dynamics\.InProceedings of the 37th Conference on Learning Theory \(COLT\),Cited by:[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1)\.
- Mouet al\.\(2022\)Z\. Mou, Y\. Huo, R\. Bai, M\. Xie, C\. Yu, J\. Xu, and B\. ZhengSustainable online reinforcement learning for auto\-bidding\.InAdvances in Neural Information Processing Systems 35 \(NeurIPS\),Red Hook, NY, USA\.Cited by:[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Pinterest, Inc\. \(2026\)Pinterest, Inc\.Pinterest announces fourth quarter and full year 2025 results, delivers 14% revenue growth and record users\.Note:[https://s204\.q4cdn\.com/369458543/files/doc\_earnings/2025/q4/earnings\-result/Q425\-PressRelease\.pdf](https://s204.q4cdn.com/369458543/files/doc_earnings/2025/q4/earnings-result/Q425-PressRelease.pdf)Press release, February 12, 2026; 619M global monthly active users; $4\.2B full\-year 2025 revenueCited by:[§1](https://arxiv.org/html/2610.33248#S1.p1.1)\.
- Stramet al\.\(2024\)R\. Stram, R\. Abboud, A\. Shtoff, O\. Somekh, A\. Raviv, and Y\. KorenMystique: a budget pacing system for performance optimization in online advertising\.InCompanion Proceedings of the ACM Web Conference 2024 \(WWW Companion\),New York, NY, USA,pp\. 433–442\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.33248#S3.SS1.p1.1)\.
- van Niekerket al\.\(2025\)J\. A\. van Niekerk, J\. D\. Le Roux, and I\. K\. CraigReinforcement learning based automatic tuning of PID controllers in multivariable grinding mill circuits\.Control Engineering Practice165,pp\. 106522\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p1.1)\.
- Wanget al\.\(2017\)Y\. Wang, A\. Agarwal, and M\. DudíkOptimal and adaptive off\-policy evaluation in contextual bandits\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research,Sydney, Australia,pp\. 3589–3597\.Cited by:[§2\.3](https://arxiv.org/html/2610.33248#S2.SS3.p2.1)\.
- Wenet al\.\(2022\)C\. Wen, M\. Xu, Z\. Zhang, Z\. Zheng, Y\. Wang, X\. Liu, Y\. Rong, D\. Xie, X\. Tan, C\. Yu, J\. Xu, F\. Wu, G\. Chen, X\. Zhu, and B\. ZhengA cooperative\-competitive multi\-agent framework for auto\-bidding in online advertising\.InProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining \(WSDM\),New York, NY, USA,pp\. 1129–1139\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Wuet al\.\(2018\)D\. Wu, X\. Chen, X\. Yang, H\. Wang, Q\. Tan, X\. Zhang, J\. Xu, and K\. GaiBudget constrained bidding by model\-free reinforcement learning in display advertising\.InProceedings of the 27th ACM International Conference on Information and Knowledge Management \(CIKM\),New York, NY, USA,pp\. 1443–1451\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§1](https://arxiv.org/html/2610.33248#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Xuet al\.\(2015\)J\. Xu, K\. Lee, W\. Li, H\. Qi, and Q\. LuSmart pacing for effective online ad campaign optimization\.InProceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining \(KDD\),New York, NY, USA,pp\. 2217–2226\.Cited by:[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1)\.
- Yanget al\.\(2025\)X\. Yang, M\. Ben Ayed, L\. Zhao, F\. Zhou, Y\. Shen, A\. Engle, J\. Zhuang, L\. Leng, J\. Xu, C\. Rosenberg, and P\. DeshikacharDeep reinforcement learning for ranking utility tuning in the ad recommender system at Pinterest\.InProceedings of the Nineteenth ACM Conference on Recommender Systems \(RecSys\),New York, NY, USA\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.33248#S2.SS2.p1.1)\.
- Yanget al\.\(2019\)X\. Yang, Y\. Li, H\. Wang, D\. Wu, Q\. Tan, J\. Xu, and K\. GaiBid optimization by multivariable control in display advertising\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining \(KDD\),New York, NY, USA,pp\. 1966–1974\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.33248#S3.SS1.p1.1)\.
- Zhanget al\.\(2016\)W\. Zhang, Y\. Rong, J\. Wang, T\. Zhu, and X\. WangFeedback control of real\-time display advertising\.InProceedings of the Ninth ACM International Conference on Web Search and Data Mining \(WSDM\),New York, NY, USA,pp\. 407–416\.Cited by:[§1](https://arxiv.org/html/2610.33248#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.33248#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.33248#S3.SS1.p1.1)\.