Generative Optimization for Incentivized Advertising with Global Level Constraints

# Generative Optimization for Incentivized Advertising with Global Level Constraints
Source: [https://arxiv.org/html/2608.04421](https://arxiv.org/html/2608.04421)
\(2026\)

###### Abstract\.

Incentivized advertising allocates monetary or virtual rewards to drive user engagement, where a key challenge is optimizing continuous incentive magnitudes under strict global constraints\. This problem is complicated by high\-frequency interactions, delayed feedback, and non\-Markovian user dynamics such as fatigue, which limit the effectiveness of existing uplift modeling and constrained reinforcement learning approaches\. To address these challenges, we propose GOAL, a constraint\-aware generative framework that formulates incentive allocation as a conditional sequence generation problem\. GOAL directly generates incentive magnitudes conditioned on user histories and system\-level global pressure, and integrates a hierarchical causal state encoder to capture both local behavioral dynamics and long\-range dependencies\. To enable flexible constraint control, we introduceSafeConstrainedPolicyOptimization \(SCPO\), which learns a single generative policy that generalizes across a spectrum of ROI constraints without retraining\. Experiments on large\-scale real\-world data and a synthetic fatigue\-aware environment show that GOAL improves long\-term revenue and user retention while substantially reducing ROI violation rates compared to strong baselines\.

Autoregressive Generation, Incentivized Advertisement, Policy Optimization

††journalyear:2026††copyright:cc††conference:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2; August 09–13, 2026; Jeju Island, Republic of Korea††booktitle:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2 \(KDD ’26\), August 09–13, 2026, Jeju Island, Republic of Korea††doi:10\.1145/3770855\.3818423††isbn:979\-8\-4007\-2259\-2/2026/08††ccs:Information systems Recommender systems## 1\.Introduction

Incentivized advertising has become a core mechanism for driving user engagement in modern recommender and advertising systems, where platforms dynamically allocate monetary incentives \(e\.g\., virtual coins or coupons\) to stimulate user interactions\(Zhanget al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib85); Chenet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib86)\)\. Unlike conventional ad recommendation settings that focus solely on content selection\(Denget al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib95); Rajputet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib75)\), incentivized advertising requires jointly deciding how much incentive to allocate for each interaction, under strict global constraints\(Wuet al\.,[2018](https://arxiv.org/html/2608.04421#bib.bib78); Goldenberget al\.,[2020](https://arxiv.org/html/2608.04421#bib.bib77)\)as Fig\.[1](https://arxiv.org/html/2608.04421#S1.F1)shows\. Excessive incentives may yield short\-term engagement gains but severely harm long\-term profitability, while overly conservative strategies fail to sustain user participation\. Designing an incentive allocation policy that balances long\-term value and financial safety remains a fundamental challenge in large\-scale industrial systems\.

From a decision\-making perspective, incentive allocation can be viewed as a sequential optimization problem with delayed and cumulative effects\. Each incentive decision not only affects immediate user response but also influences future behavior through latent dynamics such as user fatigue and incentive saturation\. However, existing approaches struggle to address this challenge effectively\. Uplift modeling methods, widely adopted in production systems, focus on short\-term treatment effects\(Peiet al\.,[2019](https://arxiv.org/html/2608.04421#bib.bib82); Zouet al\.,[2019](https://arxiv.org/html/2608.04421#bib.bib83); Zhenget al\.,[2018](https://arxiv.org/html/2608.04421#bib.bib84)\)and rely on two\-stage pipelines that are highly sensitive to prediction errors, leading to sub\-optimal decisions under constraints such as Return on Investment \(ROI\)\(Hansotia and Rukstales,[2002](https://arxiv.org/html/2608.04421#bib.bib80); Zhaoet al\.,[2019](https://arxiv.org/html/2608.04421#bib.bib81); Goldenberget al\.,[2020](https://arxiv.org/html/2608.04421#bib.bib77)\)\. Offline reinforcement learning \(RL\) methods explicitly optimize long\-term objectives, but they typically rely on Markovian assumptions that are violated in high\-frequency user interactions\(Hausknecht and Stone,[2015](https://arxiv.org/html/2608.04421#bib.bib99); Zhaoet al\.,[2018](https://arxiv.org/html/2608.04421#bib.bib100); Liet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib101)\)and tend to learn overly conservative policies due to distributional shift and limited exploration\(Kiyoharaet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib87); Korenkevychet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib88); Liuet al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib89)\)\.

![Refer to caption](https://arxiv.org/html/2608.04421v1/x1.png)Figure 1\.Overview of the incentivized advertising framework\. The platform leverages online models to directly generate incentive amounts based on user context and historical behaviors, distributing them to users to encourage participation in ad interactions\. Corresponding user interaction histories are collected to enable the platform’s continuous optimization under cost and performance constraints\.Recently, generative sequence modeling paradigms inspired by large language models have shown promise in modeling complex user behaviors by directly generating actions conditioned on historical trajectories\(Genget al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib93); Denget al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib95); Guoet al\.,[2025a](https://arxiv.org/html/2608.04421#bib.bib107); Tayet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib108)\)\. Despite their expressive power, standard Transformer\-based architectures exhibit a critical limitation in incentivized advertising\(Liet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib90); Xiaoet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib91)\): decision\-critical signals, such as abrupt user fatigue or intent shifts, are often highly localized in time, while global self\-attention tends to dilute these signals\(Zhouet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib102); Liuet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib103)\)when processing long interaction histories\. Moreover, existing generative alignment methods primarily optimize unconstrained scalar rewards, making them ill\-suited for strict, system\-level constraints that couple decisions across time\(Xiaoet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib91)\)\.

To address the aforementioned challenges, we introduceGOAL, a generative optimization framework with global\-level constraints, specifically tailored for incentivized advertising\. Our improvements primarily focus on enhancing generative decision models along two critical dimensions: \(i\)capturing non\-Markovian and localized user dynamics in long interaction histories, and \(ii\)enforcing strict global\-level ROI constraints within a unified generative decision framework\.Concretely, GOAL integrates a hierarchical causal state encoder that explicitly models dense local behavioral dynamics before performing global temporal reasoning, together with a constraint\-conditioned generative policy that adapts incentive strategies according to system\-level pressure\. On top of this architecture, we introduce Safe Constrained Policy Optimization \(SCPO\), which trains a single generative policy over a distribution of Lagrange multipliers\. By separating and normalizing preference signals under each constraint level during alignment, SCPO enables the learned policy to adapt its incentive strategy to different global constraint targets at inference time without retraining\. To summarize, our main contributions are as follows:

- •Generative Framework for Incentivized Advertising\.We propose GOAL and explore the application of a generative paradigm to incentivized advertising, enabling the model to capture finer\-grained behavioral patterns as well as both short and long\-term dependencies\.
- •SCPOAlignment Strategy\.We proposeSCPO, a constraint\-aware alignment algorithm for safe exploration\. Unlike GRPO, which may inducerevenue\-cost imbalanceby solely maximizing system value,SCPOintegrates generative policy optimization with Lagrangian multiplier to strictly enforce global constraints while optimizing long\-term ecosystem value\.
- •Validation on Industrial Scale\.We validate the superior performance of our approach on a diverse set of real\-world industrial and synthetic datasets\. The results demonstrate significant improvements in core metrics\.

## 2\.Preliminary

In this section, we formally formulate the incentive decision problem as a generative sequence modeling task and introduce the constrained optimization framework based on Lagrangian duality\.

### 2\.1\.Problem Formulation

Given a dataset𝒟=\{\(ℋi,λi,ai\)\}i=1N\\mathcal\{D\}=\\\{\(\\mathcal\{H\}\_\{i\},\\lambda\_\{i\},a\_\{i\}\)\\\}\_\{i=1\}^\{N\}, whereNNis the dataset size\. For theii\-th example \(corresponding to a specific decision momentttfor a user\),ℋi\\mathcal\{H\}\_\{i\}represents the historical interaction trajectory derived via a sliding window of lengthLL:ℋ​i=\[𝐱t−L,…,𝐱t−1\]\\mathcal\{H\}i=\[\\mathbf\{x\}\_\{t\-L\},\\dots,\\mathbf\{x\}\_\{t\-1\}\],where each composite event𝐱τ=\(𝐬τ,aτ\)∈𝒮×𝒜\\mathbf\{x\}\_\{\\tau\}=\(\\mathbf\{s\}\_\{\\tau\},a\_\{\\tau\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}consists of the high\-dimensional state features𝐬τ\\mathbf\{s\}\_\{\\tau\}and the received incentiveaτa\_\{\\tau\}at stepτ\\tau\. Additionally,λi∈ℝ\+\\lambda\_\{i\}\\in\\mathbb\{R\}^\{\+\}represents the system\-level constraint multiplier reflecting the real\-time system pressure, andai∈𝒜a\_\{i\}\\in\\mathcal\{A\}is the ground\-truth incentive allocated at the current steptt\.

To harness the sequential reasoning capabilities of generative architectures for continuous control, we reformulate the incentive decision as an autoregressive generation task\. Specifically, we introduce a discrete vocabulary𝒱\\mathcal\{V\}, where each token represents a quantized numerical unit\. We decompose the continuous target actionaia\_\{i\}into a sequence of tokens𝒈i=\(gi1,gi2,…,giTi\)\\bm\{g\}\_\{i\}=\(g\_\{i\}^\{1\},g\_\{i\}^\{2\},\\dots,g\_\{i\}^\{T\_\{i\}\}\), wheregik∈𝒱g\_\{i\}^\{k\}\\in\\mathcal\{V\}denotes the token at thekk\-th generation step, andTiT\_\{i\}denotes the sequence length\. Conversely, we design a deterministic mapping functionϕ​\(⋅\)\\phi\(\\cdot\)that reconstructs the original scalar valueaia\_\{i\}fromgig\_\{i\}, i\.e\.,ai=ϕ​\(gi\)=∑k=1Tiϕ​\(gik\)∈ℝ\+a\_\{i\}=\\phi\(g\_\{i\}\)=\\sum\_\{k=1\}^\{T\_\{i\}\}\\phi\(g\_\{i\}^\{k\}\)\\in\\mathbb\{R\}^\{\+\}, whereϕ​\(gik\)\\phi\(g\_\{i\}^\{k\}\)denotes the numerical magnitude of the token\. The details of vocabulary construction and mapping are presented in Section[3\.1](https://arxiv.org/html/2608.04421#S3.SS1)\. Our goal is to train a generative policyπθ\\pi\_\{\\theta\}which, given the user history and constraint characteristics\(ℋi,λi\)\(\\mathcal\{H\}\_\{i\},\\lambda\_\{i\}\), generates the corresponding predicted token sequence𝒈^i=\(g^i1,g^i2,…,g^iTi\)\\bm\{\\hat\{g\}\}\_\{i\}=\(\\hat\{g\}\_\{i\}^\{1\},\\hat\{g\}\_\{i\}^\{2\},\\dots,\\hat\{g\}\_\{i\}^\{T\_\{i\}\}\)\. In turn, the predicted incentive valuea^i\\hat\{a\}\_\{i\}, reconstructed viaa^i=ϕ​\(g^i\)=∑k=1Tiϕ​\(g^ik\)\\hat\{a\}\_\{i\}=\\phi\(\\hat\{g\}\_\{i\}\)=\\sum\_\{k=1\}^\{T\_\{i\}\}\\phi\(\\hat\{g\}\_\{i\}^\{k\}\), approximates the actual incentiveaia\_\{i\}while adhering to the cost\-efficiency trade\-off governed byλi\\lambda\_\{i\}\.

![Refer to caption](https://arxiv.org/html/2608.04421v1/x2.png)Figure 2\.The overall architecture of GOAL\. \(1\) Hierarchical Causal Encoder: synergizes DCC and self\-attention to capture both high\-frequency local dynamics and global temporal dependencies\. \(2\)λ\\lambda\-Conditioned Generative Decoder: replaces standard FFNs with MoE to enable the experts to perceive different levels ofλ\\lambda\-induced constraint pressure\. \(3\) Optimization: optimized via a weighted NTP objectiveℒW−N​T​P\\mathcal\{L\}\_\{W\-NTP\}\.
### 2\.2\.Primal\-Dual Constrained Policy Optimization

While the generative policy maximizes user value given the system\-level stateλ\\lambda, the definition ofλ\\lambdais governed by global business requirements, particularly strict ROI constraints in incentive scenarios\. Primal Problem\.Consider a trajectory ofNNinteractions, where actionaia\_\{i\}incurs costcic\_\{i\}and yields revenuerir\_\{i\}\. The global ROI is defined as∑i=1Nri∑i=1Nci\\frac\{\\sum\_\{i=1\}^\{N\}r\_\{i\}\}\{\\sum\_\{i=1\}^\{N\}c\_\{i\}\}, which is required to exceed a minimum thresholdτ\\tau\. The resulting reward decision problem is formulated as

\(1\)max\{ai\}​∑i=1Nris\.t\.∑i=1Nri∑i=1Nci≥τ,0≤ci≤Cmax\.\\max\_\{\\\{a\_\{i\}\\\}\}\\sum\_\{i=1\}^\{N\}r\_\{i\}\\quad\\text\{s\.t\.\}\\quad\\frac\{\\sum\_\{i=1\}^\{N\}r\_\{i\}\}\{\\sum\_\{i=1\}^\{N\}c\_\{i\}\}\\geq\\tau,\\quad 0\\leq c\_\{i\}\\leq C\_\{\\max\}\.The fractional ROI constraint couples decisions across time, making direct optimization challenging\. Lagrangian Duality and Decoupling\.We leverage the Lagrangian duality framework to decouple the global constraint\. Specifically, we linearize the ROI constraint and introduce a Lagrange multiplierλ′≥0\\lambda^\{\\prime\}\\geq 0to incorporate it into the optimization objective as:

\(2\)ℒ​\(π,λ′\)=∑i=1Nri\+λ′​\(∑i=1Nri−τ​∑i=1Nci\)=\(1\+λ′\)​∑i=1Nri−λ′​τ​∑i=1Nci\.\\begin\{split\}\\mathcal\{L\}\(\\pi,\\lambda^\{\\prime\}\)&=\\sum\_\{i=1\}^\{N\}r\_\{i\}\+\\lambda^\{\\prime\}\\left\(\\sum\_\{i=1\}^\{N\}r\_\{i\}\-\\tau\\sum\_\{i=1\}^\{N\}c\_\{i\}\\right\)\\\\ &=\(1\+\\lambda^\{\\prime\}\)\\sum\_\{i=1\}^\{N\}r\_\{i\}\-\\lambda^\{\\prime\}\\tau\\sum\_\{i=1\}^\{N\}c\_\{i\}\.\\end\{split\}Optimizing this objective is mathematically equivalent to maximizing the normalized return:

\(3\)maxπ∈Π⁡ℒ​\(π,λ\)=maxπ∈Π​∑i=1N\(ri−λ​ci\),where​λ=λ′​τ1\+λ′\.\\max\_\{\\pi\\in\\Pi\}\\mathcal\{L\}\(\\pi,\\lambda\)=\\max\_\{\\pi\\in\\Pi\}\\sum\_\{i=1\}^\{N\}\\left\(r\_\{i\}\-\\lambda c\_\{i\}\\right\),\\quad\\text\{where \}\\lambda=\\frac\{\\lambda^\{\\prime\}\\tau\}\{1\+\\lambda^\{\\prime\}\}\.Under the mild assumption that a feasible policy exists, strong duality holds\. Thus, the primal problem is equivalent to the Lagrangian dual:minλ∈ℝ\+⁡maxπ∈Π⁡ℒ​\(π,λ\)\\min\_\{\\lambda\\in\\mathbb\{R\}^\{\+\}\}\\max\_\{\\pi\\in\\Pi\}\\mathcal\{L\}\(\\pi,\\lambda\)\.

Therefore, given the optimal dual variableλ∗\\lambda^\{\*\}, the optimization objective reduces to:

\(4\)maxπ​∑i=1N\(ri−λ∗​ci\)\.\\max\_\{\\pi\}\\sum\_\{i=1\}^\{N\}\(r\_\{i\}\-\\lambda^\{\*\}c\_\{i\}\)\.

## 3\.Methodology

As illustrated in Figure[2](https://arxiv.org/html/2608.04421#S2.F2), we propose GOAL, departing from conventional two\-stage incentive optimization pipelines by unifying generation, constraint modeling, and policy optimization into a single generative framework\. The model operates in two tightly coupled stages\. First, a causal state encoder captures the temporally consistent evolution of user interests, while a Lagrangian multiplier embedding is injected as a global control signal to explicitly parameterize constraint tightness\. Second, an autoregressive decoder equipped with constraint\-aware MoE routing generates incentive sequences, dynamically switching between conservative and aggressive strategies based on the constraint context\. On top of this architecture, we introduce aλ\\lambda\-generalized safe constrained policy optimization scheme \(SCPO\), enabling a single trained model to adapt to diverse ROI constraints\. Together, these components embed constraint awareness into both the representation space and the optimization dynamics, yielding a flexible and scalable solution to constrained incentive generation\. In the following subsections, we elaborate on the implementation details of these components\.

### 3\.1\.Incentive Vocabulary Construction

0:Incentive dataset

𝒜=\{aj\}j=1N\\mathcal\{A\}=\\\{a\_\{j\}\\\}\_\{j=1\}^\{N\}, start percentile

qstartq\_\{\\text\{start\}\}, end percentile

qendq\_\{\\text\{end\}\}, decay rate

δ\\delta, restoration error threshold

ϵ1\\epsilon\_\{1\}, and minimal precision threshold

ϵ2\\epsilon\_\{2\}\.

0:Constructed vocabulary

𝒱\\mathcal\{V\}
1:Initialize an empty vocabulary

𝒱←\{\}\\mathcal\{V\}\\leftarrow\\\{\\\}
2:Sort

𝒜\\mathcal\{A\}in descending order to obtain

𝒜^=\{a^j\}j=1N\\hat\{\\mathcal\{A\}\}=\\\{\\hat\{a\}\_\{j\}\\\}\_\{j=1\}^\{N\}
3:Initialize iteration counter

i←1i\\leftarrow 1, error metric

e​r​r←∞err\\leftarrow\\infty, and current percentile

q←qstartq\\leftarrow q\_\{\\text\{start\}\}
4:while

e​r​r\>ϵ1err\>\\epsilon\_\{1\}do

5:Compute the

qq\-percentile value

oio\_\{i\}of the residual set

𝒜^\\hat\{\\mathcal\{A\}\}
6:if

oi≤ϵ2o\_\{i\}\\leq\\epsilon\_\{2\}then

7:break\{Terminate if the token precision falls below the minimum threshold\}

8:endif

9:Generate a new token

wiw\_\{i\}such that

ϕ​\(wi\)=oi\\phi\(w\_\{i\}\)=o\_\{i\}
10:

𝒱←𝒱∪\{wi\}\\mathcal\{V\}\\leftarrow\\mathcal\{V\}\\cup\\\{w\_\{i\}\\\}\{Insert the unique token into the vocabulary\}

11:Update the residual values in

𝒜^\\hat\{\\mathcal\{A\}\}:

12:for

j=1j=1to

NNdo

13:if

a^j≥oi\\hat\{a\}\_\{j\}\\geq o\_\{i\}then

14:

a^j←a^j−oi\\hat\{a\}\_\{j\}\\leftarrow\\hat\{a\}\_\{j\}\-o\_\{i\}
15:endif

16:endfor

17:Update the error metric:

e​r​r←max1≤j≤N⁡\{a^jaj\}err\\leftarrow\\max\_\{1\\leq j\\leq N\}\\left\\\{\\frac\{\\hat\{a\}\_\{j\}\}\{a\_\{j\}\}\\right\\\}
18:Update the percentile with decay rate

δ\\delta:

q←max⁡\(q⋅δ,qend\)q\\leftarrow\\max\(q\\cdot\\delta,q\_\{\\text\{end\}\}\)
19:

i←i\+1i\\leftarrow i\+1
20:endwhile

21:return

𝒱\\mathcal\{V\}

Algorithm 1Incentive Vocabulary ConstructionIncentive allocation differs from language generation in that it involves continuous numerical values with strict magnitude constraints, where existing tokenization\(Wallaceet al\.,[2019](https://arxiv.org/html/2608.04421#bib.bib121)\)or bucketing methods\(Wanget al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib73); Jieet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib74)\)fail to preserve numerical precision\. This limitation stems from two inherent characteristics of the action space: a wide value range \(e\.g\., 1 to 3,000 coins\) and a heavy\-tailed distribution\. Standard one\-step regression faces a severe optimization bottleneck under such a vast range, as uniform losses struggle to balance low\-value regions with high\-magnitude scales\. Moreover, due to the long\-tailed nature of marketing traffic, regression models are easily overwhelmed by high\-frequency low\-value samples, leading to severe underfitting on rare but decision\-critical large incentives\. To address these challenges, we adopt a data\-driven vocabulary construction approach based on dynamic quantile adjustment\(Maet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib76)\), as detailed in Algorithm[1](https://arxiv.org/html/2608.04421#alg1)\.

The constructed vocabulary𝒱=\{w1,w2,…,w\|𝒱\|\}\\mathcal\{V\}=\\\{w\_\{1\},w\_\{2\},\\dots,w\_\{\|\\mathcal\{V\}\|\}\\\}guarantees that each token is unique, enabling the comprehensive representation of the continuous incentive space using a finite set of discrete units\. To transform a specific incentiveaia\_\{i\}into a token sequence𝒈i=\(gi1,gi2,…,giTi\)\\bm\{g\}\_\{i\}=\(g\_\{i\}^\{1\},g\_\{i\}^\{2\},\\dots,g\_\{i\}^\{T\_\{i\}\}\), we must ensure that the original value is reconstructible with bounded error while minimizing the sequence lengthMM\. To achieve these dual objectives, we employ a deterministic greedy decomposition algorithm that iteratively decomposes the total rewardata\_\{t\}by always selecting the largest available token from𝒱\\mathcal\{V\}less than or equal to the current residual value\. Crucially, this strategy naturally enforces a monotonicity constraintϕ​\(gi1\)≥ϕ​\(gi2\)≥⋯≥ϕ​\(giTi\)\\phi\(g\_\{i\}^\{1\}\)\\geq\\phi\(g\_\{i\}^\{2\}\)\\geq\\dots\\geq\\phi\(g\_\{i\}^\{T\_\{i\}\}\)\. This captures a hierarchical coarse\-to\-fine structure, ensuring that the leading tokens encode the dominant magnitude of the incentive, while subsequent tokens efficiently handle residual precision\.

### 3\.2\.Autoregressive Incentive Generation

The generation module serves as the actor in our framework, which maps the composite state inputs into a precise incentive action\.

#### 3\.2\.1\.Hierarchical Causal Encoder

Distinct from conventional encoders that directly feed raw embeddings into Transformer layers, we introduce a specialized causal state encoder module to explicitly capture the dynamic causality between historical behaviors and current states\. The encoding process operates sequentially, moving from initial feature projection to causal modeling, and finally to global context integration\.

Given the user historical trajectoryℋu=\[𝒙t−L,…,𝒙t−1\]\\mathcal\{H\}\_\{u\}=\[\\bm\{x\}\_\{t\-L\},\\dots,\\bm\{x\}\_\{t\-1\}\]defined in the problem formulation, which corresponds to the sequence\(𝒙1,…,𝒙L\)\\left\(\\bm\{x\}\_\{1\},\\ldots,\\bm\{x\}\_\{L\}\\right\)illustrated in Fig\.[2](https://arxiv.org/html/2608.04421#S2.F2), we first project these features into dense continuous vectors\. This process yields an initial embedding sequence𝑬=\(𝒆1,…,𝒆L\)\\bm\{E\}=\(\\bm\{e\}\_\{1\},\\dots,\\bm\{e\}\_\{L\}\), which serves as the input to the causal module\. To simulate the user’s dynamic interests while strictly adhering to temporal causality, we employ a Dilated Causal Convolution \(DCC\) block\(Bai,[2018](https://arxiv.org/html/2608.04421#bib.bib116)\)\. This module aggregates previous behaviors to form a causal state representation\. Let𝑯\(0\)=𝑬\\bm\{H\}^\{\(0\)\}=\\bm\{E\}be the input to the DCC\. We generate the hidden representations𝑯\(l\)=\(𝒉1\(l\),⋯,𝒉L\(l\)\)\\bm\{H\}^\{\(l\)\}=\\left\(\\bm\{h\}\_\{1\}^\{\(l\)\},\\cdots,\\bm\{h\}\_\{L\}^\{\(l\)\}\\right\)for thell\-th convolutional layer via DCC:

\(5\)𝑯\(l\)=DilatedCausalConv​\(𝑯\(l−1\)\),l=1,…,L1\.\\bm\{H\}^\{\(l\)\}=\\text\{DilatedCausalConv\}\(\\bm\{H\}^\{\(l\-1\)\}\),\\quad l=1,\\dots,L\_\{1\}\.Specifically, the hidden state at timett, denoted as𝒉t\(l\)\\bm\{h\}^\{\(l\)\}\_\{t\}, is computed by convolving filter weightsffwith past states:

\(6\)𝒉t\(l\)=∑i=0kc−1f​\(i\)⋅𝒉t−dc⋅i\(l−1\),\\bm\{h\}^\{\(l\)\}\_\{t\}=\\sum\_\{i=0\}^\{k\_\{c\}\-1\}f\(i\)\\cdot\\bm\{h\}^\{\(l\-1\)\}\_\{t\-d\_\{c\}\\cdot i\},wheredcd\_\{c\}andkck\_\{c\}represent the dilation factor and filter size, respectively\. This formulation ensures that𝒉t\(l\)\\bm\{h\}^\{\(l\)\}\_\{t\}only depends on historical information, providing strict causality\. We further apply residual connections and normalization to enhance gradient flow\. Finally, a linear transformation with LeakyReLU activation is applied to the DCC’s output𝑯\(L1\)\\bm\{H\}^\{\(L\_\{1\}\)\}to obtain the sequence of causal states𝑺~=\(𝒔~1,…,𝒔~L\)\\tilde\{\\bm\{S\}\}=\(\\tilde\{\\bm\{s\}\}\_\{1\},\\dots,\\tilde\{\\bm\{s\}\}\_\{L\}\):

\(7\)𝑺~=LeakyReLU​\(𝑾s​𝑯\(L1\)\+𝑩s\),\\tilde\{\\bm\{S\}\}=\\text\{LeakyReLU\}\(\\bm\{W\}\_\{s\}\\bm\{H\}^\{\(L\_\{1\}\)\}\+\\bm\{B\}\_\{s\}\),where𝑾s\\bm\{W\}\_\{s\}and𝑩s\\bm\{B\}\_\{s\}are learnable parameters\.

To render the generative process ROI\-constraint\-aware, we introduce a constraint embedding mechanism\. Specifically, the scalar Lagrange multiplierλi\\lambda\_\{i\}is projected into a dense vector𝒆λi\\bm\{e\}\_\{\\lambda\_\{i\}\}via a Multi\-Layer Perceptron \(MLP\), serving as a global condition token:𝒆λ=MLP​\(λi\)\\bm\{e\}\_\{\\lambda\}=\\text\{MLP\}\(\\lambda\_\{i\}\)\. We prepend this constraint embedding to the causal state sequence, forming an augmented sequence\[𝒆λi;𝑺~\]\[\\bm\{e\}\_\{\\lambda\_\{i\}\};\\tilde\{\\bm\{S\}\}\]\.

While DCC captures local causal dependencies within a finite receptive field, we introduce a self\-attention layer to model global dependencies across the entire interaction history, conditioned onλi\\lambda\_\{i\}\. This enables the capture of long\-range behavioral correlations beyond the reach of convolution\. The resulting sequence is processed by self\-attention followed by an FFN:

\(8\)𝑯e​n​c=FFN​\(SelfAttn​\(\[𝒆λi;𝑺~\]\)\)\.\\bm\{H\}\_\{enc\}=\\mathrm\{FFN\}\\big\(\\mathrm\{SelfAttn\}\(\[\\bm\{e\}\_\{\\lambda\_\{i\}\};\\tilde\{\\bm\{S\}\}\]\)\\big\)\.The output𝑯e​n​c\\bm\{H\}\_\{enc\}provides a semantic representation for the subsequent decoding stage\.

#### 3\.2\.2\.λ\\lambda\-Conditioned Generative Decoder

This module formulates incentive generation as an autoregressive sequence generation task\. To overcome the precision loss inherent in mapping continuous values to discrete tokens\(Iidaet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib71); Wanget al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib73); Jinet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib72)\), we allow each prediction step the flexibility to choose from the vocabulary\.

Let𝒁\(l\)\\bm\{Z\}^\{\(l\)\}denote the intermediate representation𝑯e​n​c\\bm\{H\}\_\{enc\}after attention mechanisms\. To enforce the global ROI constraint while maintaining computational efficiency, we substitute the standard Feed\-Forward Network \(FFN\) in the decoder with a constraint\-aware Mixture\-of\-Experts \(MoE\) layer\(Duet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib97); Zoph,[2022](https://arxiv.org/html/2608.04421#bib.bib133)\)\. For the input vector𝒛∈𝒁\(l\)\\bm\{z\}\\in\\bm\{Z\}^\{\(l\)\}, the output𝒐d​e​c\\bm\{o\}\_\{dec\}is computed by aggregating the outputs of a sparse subset of experts:

\(9\)𝒐dec\\displaystyle\\bm\{o\}\_\{\\text\{dec\}\}=∑j=1Nexpαj​\(𝒛,𝒄λ\)⋅Expertj​\(𝒛\),\\displaystyle=\\sum\_\{j=1\}^\{N\_\{\\text\{exp\}\}\}\\alpha\_\{j\}\(\\bm\{z\},\\bm\{c\}\_\{\\lambda\}\)\\cdot\\text\{Expert\}\_\{j\}\(\\bm\{z\}\),𝒓\\displaystyle\\bm\{r\}=Softmax​\(𝑾g⋅\[𝒛;𝒄λ\]\),\\displaystyle=\\text\{Softmax\}\\\!\\left\(\\bm\{W\}\_\{g\}\\cdot\[\\bm\{z\};\\bm\{c\}\_\{\\lambda\}\]\\right\),αj​\(𝒛,𝒄λ\)\\displaystyle\\alpha\_\{j\}\(\\bm\{z\},\\bm\{c\}\_\{\\lambda\}\)=\{rj∑e∈𝒯re,j∈𝒯,𝒯=Top\-​k​\(𝒓\)0,otherwise\.\\displaystyle=
whereNe​x​pN\_\{exp\}denotes the total number of experts, andαj\\alpha\_\{j\}represents the routing weight for thejj\-th expert\. In this way, the FFN is able to learn different ROI constraints and assign distinct weights to different experts \(see Appendix[D](https://arxiv.org/html/2608.04421#A4)\)\.

Finally, the MoE output𝒐d​e​c\\bm\{o\}\_\{dec\}is projected into the vocabulary space to predict the next token\. The generation probability for a tokengi^∈𝒱\\hat\{g\_\{i\}\}\\in\\mathcal\{V\}is given by:

\(10\)Pθ​\(g^it∣𝑯e​n​c,g^i<t\)=Softmax​\(𝑾o​u​t⋅𝒐d​e​c\)\.P\_\{\\theta\}\(\\hat\{g\}\_\{i\}^\{t\}\\mid\\bm\{H\}\_\{enc\},\\hat\{g\}\_\{i\}^\{<t\}\)=\\text\{Softmax\}\(\\bm\{W\}\_\{out\}\\cdot\\bm\{o\}\_\{dec\}\)\.
Weighted Next\-Token Prediction Mechanism\.Following standard practice, we augment the vocabulary with special tokens<SOS\>,<EOS\>, and<PAD\>to denote sequence boundaries and enable batch padding\. These tokens carry zero incentive value\.

As illustrated in Fig\.[2](https://arxiv.org/html/2608.04421#S2.F2), the decoder generates the incentive token sequence𝒈^i=\(g^i1,…,g^iTi\)\\hat\{\\bm\{g\}\}\_\{i\}=\(\\hat\{g\}\_\{i\}^\{1\},\\dots,\\hat\{g\}\_\{i\}^\{T\_\{i\}\}\)conditioned on the encoder output𝑯e​n​c\\bm\{H\}\_\{enc\}and the preceding subsequenceg^i<t\\hat\{g\}\_\{i\}^\{<t\}\. Specifically, at generation steptt, the output tokeng^it\\hat\{g\}\_\{i\}^\{t\}is computed as:

\(11\)g^it=arg​maxg∈𝒱⁡Pθ​\(w∣𝑯e​n​c,g^i<t\)\.\\hat\{g\}\_\{i\}^\{t\}=\\operatorname\*\{arg\\,max\}\_\{g\\in\\mathcal\{V\}\}P\_\{\\theta\}\(w\\mid\\bm\{H\}\_\{enc\},\\hat\{g\}\_\{i\}^\{<t\}\)\.
To equip the model with fundamental market perception and account for the numerical significance of coin values, we employ a Position\-Weighted Next Token Prediction objective\. In our scenario, tokens in the early stages of the decomposition sequence represent higher orders of magnitude\. Therefore, a misprediction at these positions leads to a more substantial bias in the final reward\. The objective is defined as:

\(12\)ℒW−N​T​P=−1N​∑i=1N∑t=1Tiωt⋅log⁡Pθ​\(g^ti∣𝑯e​n​c,g^<ti\)\\mathcal\{L\}\_\{W\-NTP\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\omega\_\{t\}\\cdot\\log P\_\{\\theta\}\\left\(\\hat\{g\}\_\{t\}^\{i\}\\mid\\bm\{H\}\_\{enc\},\\hat\{g\}\_\{<t\}^\{i\}\\right\)where he weightωt\\omega\_\{t\}is formally defined as follows:

\(13\)ωt=log10⁡\(v​\(gti\)\+1\)\+δ,\\omega\_\{t\}=\\log\_\{10\}\\left\(v\\left\(g\_\{t\}^\{i\}\\right\)\+1\\right\)\+\\delta,wherev​\(⋅\)v\(\\cdot\)maps a token to its physical coin value, andδ\\deltais a smoothing constant to maintain a baseline gradient for small\-value tokens\. Distinct from the traditional NTP loss, which treats all token\-level prediction errors indiscriminately, our weighted formulation shifts the model’s focus toward numerical precision\. By penalizing discrepancies based on their physical magnitude, this objective significantly refines the pre\-trained model’s accuracy in high\-value estimation\. Meanwhile, to expedite convergence, we also adopt the Teacher Forcing \(TF\)\(Venkatramanet al\.,[2015](https://arxiv.org/html/2608.04421#bib.bib123)\)mechanism, where the ground truth tokengitg\_\{i\}^\{t\}is fed into the decoder at stept\+1t\+1to explicitly guide the generation process\.

### 3\.3\.Reward Model Training

Due to the extreme sparsity of final conversions, using consumption cost alone as the reward signal leads to unstable learning\. To provide denser supervision, we incorporate a set of auxiliary behavioral signals𝑺o​b​j\\bm\{S\}\_\{obj\}\(e\.g\., revenue contribution, scroll depth, retention, and task\-level engagement\) to enrich the reward modeling\.

To aggregate these heterogeneous objectives, we adopt a neural\-based fusion model to learn a personalized Preference Score \(P\-Score\)\(Caoet al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib117)\)\. The model follows a multi\-tower architecture, where each tower encodes objective\-specific features, and the resulting representations are fused by a shared MLP to produce a unified utility score\. The model is trained with a weighted multi\-objective objective to balance trade\-offs among different signals:

\(14\)ℒP\-Score=∑k∈𝒮o​b​jwk⋅ℒk​\(y^,yk∗\),\\mathcal\{L\}\_\{\\text\{P\-Score\}\}=\\sum\_\{k\\in\\mathcal\{S\}\_\{obj\}\}w\_\{k\}\\cdot\\mathcal\{L\}\_\{k\}\(\\hat\{y\},y\_\{k\}^\{\*\}\),wherewkw\_\{k\}represents the adaptive weight balancing the trade\-off between objectives\. For binary classification tasks \(e\.g\., retention or click\-through prediction\),ℒk\\mathcal\{L\}\_\{k\}is formulated as the standard Binary Cross\-Entropy \(BCE\) loss:

\(15\)ℒk=−1N​∑i=1N\[yi,k∗​log⁡\(p^i,k\)\+\(1−yi,k∗\)​log⁡\(1−p^i,k\)\],\\mathcal\{L\}\_\{k\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[y\_\{i,k\}^\{\*\}\\log\(\\hat\{p\}\_\{i,k\}\)\+\(1\-y\_\{i,k\}^\{\*\}\)\\log\(1\-\\hat\{p\}\_\{i,k\}\)\\right\],whereyi,k∗∈\{0,1\}y\_\{i,k\}^\{\*\}\\in\\\{0,1\\\}denotes the ground\-truth label for sampleiion objectivekk, andp^i,k\\hat\{p\}\_\{i,k\}is the predicted probability\.

![Refer to caption](https://arxiv.org/html/2608.04421v1/x3.png)Figure 3\.The workflow of Safe Constrained Policy Optimization \(SCPO\)\.
### 3\.4\.Safe Constrained Policy Optimization \(SCPO\)

As formulated in Section[2\.2](https://arxiv.org/html/2608.04421#S2.SS2), the optimal policy for the condition\-constrained alignment problem can be theoretically derived via the dual problemminλ≥0⁡maxπ⁡ℒ​\(π,λ\)\\min\_\{\\lambda\\geq 0\}\\allowbreak\\max\_\{\\pi\}\\mathcal\{L\}\(\\pi,\\lambda\)\. However, traditional primal\-dual methods require iterative updates of the multiplierλ\\lambda, necessitating policy re\-training whenever the ROI threshold shifts\. This is computationally prohibitive for large\-scale generative models\.

To achieve constraint adaptation, we propose aλ\\lambda\-Generalization framework\(Zhanget al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib85)\)integrated with Group Relative Policy Optimization \(GRPO\)\. We fundamentally reformulate the single\-point optimization problem into a universal policy learning problem over the distribution of possible Lagrange multipliers\.

Universal Lagrangian Objective\.Instead of treatingλ\\lambdaas a static hyperparameter, we elevate it to a stochastic state variable\. Letλ∼P​\(Λ\)\\lambda\\sim P\(\\Lambda\)be a multiplier sampled from a uniform distribution over a predefined range\[λmin,λmax\]\[\\lambda\_\{\\min\},\\lambda\_\{\\max\}\]\. Complementary to this, the range\[λmin,λmax\]\[\\lambda\_\{\\min\},\\lambda\_\{\\max\}\]is chosen to cover both unconstrained and strictly constrained regimes\. We detail the process of determining the candidate set forλ\\lambdain Appendix[A](https://arxiv.org/html/2608.04421#A1)\. We extend the state space tos~=\[s;λ\]\\tilde\{s\}=\[s;\\lambda\]and parameterize the policy asπθ​\(a\|s,λ\)\\pi\_\{\\theta\}\(a\|s,\\lambda\)\.

The optimization objective is transformed to maximize the expected Lagrangian\-conditioned reward over the entireλ\\lambdaspectrum:

\(16\)𝒥G​e​n​\(θ\)=𝔼λ∼P​\(Λ\)​\[𝔼τ∼πθ\(⋅\|λ\)​\[∑t=0Trλ​\(st,at\)\]\],\\mathcal\{J\}\_\{Gen\}\(\\theta\)=\\mathbb\{E\}\_\{\\lambda\\sim P\(\\Lambda\)\}\\left\[\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\(\\cdot\|\\lambda\)\}\\left\[\\sum\_\{t=0\}^\{T\}r\_\{\\lambda\}\(s\_\{t\},a\_\{t\}\)\\right\]\\right\],where, consistent with the dual formulation in Eq\.[3](https://arxiv.org/html/2608.04421#S2.E3), the composite step\-wise rewardrλr\_\{\\lambda\}dynamically subsumes the cost penalty based on the currentλ\\lambda:

\(17\)rλ​\(s,oi\)=R​\(s,oi\)−λ​C​\(s,oi\)r\_\{\\lambda\}\\left\(s,o\_\{i\}\\right\)=R\\left\(s,o\_\{i\}\\right\)\-\\lambda C\\left\(s,o\_\{i\}\\right\), here,RRis the P\-Score derived in Section[3\.3](https://arxiv.org/html/2608.04421#S3.SS3), andCCrepresents the monetary cost of the action\. This formulation incentivizes the model to learn a continuous family of strategies: aggressive revenue maximization whenλ→0\\lambda\\to 0, and conservative cost control whenλ\\lambdais large\.

After training, the learned policy can be instantiated with different constraint levels without retraining\. At inference time, a specificλ∗\\lambda^\{\*\}is selected to satisfy the ROI constraint with the detailed selection procedure provided in Appendix[B](https://arxiv.org/html/2608.04421#A2)\.

SCPO with ROI\-Sensitive Advantage\.To achieve a policy adaptable to arbitrary constraint levels via preference alignment, we modify the rollout mechanism of the standard GRPO\. Specifically, as illustrated in Fig\.[3](https://arxiv.org/html/2608.04421#S3.F3), given states∈\{s1,s2,…,sm\}s\\in\\\{s\_\{1\},s\_\{2\},\\ldots,s\_\{m\}\\\}and a set of parameters\{λ1,…,λn\}\\\{\\lambda\_\{1\},\\ldots,\\lambda\_\{n\}\\\}sampled from\[0,λmax\]\[0,\\lambda\_\{\\max\}\]\(A detailed formulation of the candidate selection strategy forλ\\lambdais provided in Appendix[A](https://arxiv.org/html/2608.04421#A1)\.\), their Cartesian product is constructed as\{λ1,…,λn\}×\{s1,s2,…,sm\}\\left\\\{\\lambda\_\{1\},\\ldots,\\lambda\_\{n\}\\right\\\}\\times\\left\\\{s\_\{1\},s\_\{2\},\\ldots,s\_\{m\}\\right\\\}\. For each pair\(λi,sj\)\(\\lambda\_\{i\},s\_\{j\}\), we first generate a set of candidate responses\{oiλj\}i=1m\\\{o\_\{i\}^\{\\lambda\_\{j\}\}\\\}\_\{i=1\}^\{m\}\. Correspondingly, utilizing Eq\.[17](https://arxiv.org/html/2608.04421#S3.E17), the reward evaluation expands from the original set\{r1,…,rm\}\\\{r\_\{1\},\\dots,r\_\{m\}\\\}to a constraint\-conditioned matrix\{r1λj\},…,\{rmλj\}\\left\\\{r\_\{1\}^\{\\lambda\_\{j\}\}\\right\\\},\\dots,\\left\\\{r\_\{m\}^\{\\lambda\_\{j\}\}\\right\\\}, i\.e\.,

\(18\)ℛλj=\{rλj​\(s,o1\),…,rλj​\(s,om\)\}\\mathcal\{R\}\_\{\\lambda\_\{j\}\}=\\\{r\_\{\\lambda\_\{j\}\}\(s,o\_\{1\}\),\\dots,r\_\{\\lambda\_\{j\}\}\(s,o\_\{m\}\)\\\}in each Groupλj\{\\lambda\_\{j\}\}\.

This augmented dataset and reward computation integrate the state, policy, and constraint variableλ\\lambda, facilitating the post\-training optimization of the generative policyπ​\(s,λ\)\\pi\(s,\\lambda\)to derive the optimal strategy under any ROI constraint\. Finally, during the inference phase, by simply inputting the target optimalλ∗\\lambda^\{\*\}, the model generates the optimal incentive allocation strategy\.

Based on the computed constraint\-conditioned reward matrix\{riλj\}\\\{r^\{\\lambda\_\{j\}\}\_\{i\}\\\}, the subsequent step involves estimating the advantage to guide the policy gradient update\. However, a critical challenge in training with variable constraint inputs is the significant scale discrepancy in rewards across different pressure levels \(λ\\lambda\)\. A high penalty coefficient naturally suppresses the total reward, which could mislead the optimizer into interpreting high\-λ\\lambdatrajectories as poor performance solely due to the stricter constraint rather than the policy’s quality\. Standard global normalization fails to account for this intrinsic shift\.

To address this, we define the advantageA​\(s,oi\)A\(s,o\_\{i\}\)using a Constraint\-Conditional Normalization strategy \(Appendix[C](https://arxiv.org/html/2608.04421#A3)\)\. This approach strictly isolates the evaluation context: for a specific constraint levelλj\\lambda\_\{j\}, we compute the advantage statistics solely over themmtrajectories generated under that same condition\. The group statistics are computed as:

\(19\)μλj=1m​∑i=1mrλj​\(s,oi\),σλj=1m​∑i=1m\(rλj​\(s,oi\)−μλj\)2\.\\mu\_\{\\lambda\_\{j\}\}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}r\_\{\\lambda\_\{j\}\}\(s,o\_\{i\}\),\\quad\\sigma\_\{\\lambda\_\{j\}\}=\\sqrt\{\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\left\(r\_\{\\lambda\_\{j\}\}\(s,o\_\{i\}\)\-\\mu\_\{\\lambda\_\{j\}\}\\right\)^\{2\}\}\.Consequently, the final advantage is formulated as the standardized score within its constraint group:

\(20\)A​\(s,oi\)=rλj​\(s,oi\)−μλjσλj\+ϵ,A\(s,o\_\{i\}\)=\\frac\{r\_\{\\lambda\_\{j\}\}\(s,o\_\{i\}\)\-\\mu\_\{\\lambda\_\{j\}\}\}\{\\sigma\_\{\\lambda\_\{j\}\}\+\\epsilon\},which effectively decouples the policy’s performance from the intrinsic difficulty imposed by the constraint levelλ\\lambda\.

The final policy optimization objective is computed by maximizing the following surrogate loss, which incorporates a PPO\-style clipping mechanism and a KL\-divergence regularization term:

\(21\)ℒSCPO​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\text\{SCPO\}\}\(\\theta\)=𝔼s\[1m​n∑i=1m∑j=1n\(min\(ρi,jAi,j,clip\(ρi,j,1−ϵ,1\+ϵ\)Ai,j\)\\displaystyle=\\mathbb\{E\}\_\{s\}\\Bigg\[\\frac\{1\}\{mn\}\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}\\Big\(\\min\\big\(\\rho\_\{i,j\}A\_\{i,j\},\\text\{clip\}\(\\rho\_\{i,j\},1\-\\epsilon,1\+\\epsilon\)A\_\{i,j\}\\big\)−β𝔻KL\(πθ∥πref\)\)\]\.\\displaystyle\-\\beta\\,\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\Big\)\\Bigg\]\.
whereρi,j=πθ​\(oi\|s,λj\)πold​\(oi\|s,λj\)\\rho\_\{i,j\}=\\frac\{\\pi\_\{\\theta\}\(o\_\{i\}\|s,\\lambda\_\{j\}\)\}\{\\pi\_\{\\text\{old\}\}\(o\_\{i\}\|s,\\lambda\_\{j\}\)\}denotes the probability ratio between the current and old policies\.ϵ\\epsilonis the clipping hyperparameter that restricts the policy update step size, andπref\\pi\_\{\\text\{ref\}\}represents the reference policy used to prevent excessive deviation during training\.

## 4\.Experiments

In this section, we conduct comprehensive experiments to demonstrate the effectiveness of GOAL\. Specifically, we explore the following research questions:

- •RQ1:Does GOAL achieve superior long\-term revenue and ROI stability compared to state\-of\-the\-art industrial baselines?
- •RQ2:Can the proposed framework dynamically adapt to varying ROI constraints without re\-training?
- •RQ3:How do the key components \(e\.g\., Causal Encoder, SCPO\) contribute to the overall performance?
- •RQ4:How does GOAL perform in online deployment compared to existing methods?

### 4\.1\.Experimental Settings

Datasets\.We evaluate our method on two real\-world and synthetic datasets, with the experimental setup as described below\. Specifically, we use the initial six days of the data as the training set and designate the final day as the test set\.

- •IA:This dataset is collected from the incentive advertising system of a leading short video platform, covering the period from November 5, 2025, to November 12, 2025\. It comprises approximately 130,000 daily active users and 1\.84 million daily incentive exposure events\.
- •Synthetic IA:Constructed based on the IA dataset to maintain consistent feature alignment, this simulator introduces explicit user fatigue dynamics to benchmark long\-term ROI controllability\. It contains 10,000 diverse trajectories generated via a mixed behavior policy \(details in Appendix[E](https://arxiv.org/html/2608.04421#A5)\)\.

Evaluation Metrics\.We adopt a comprehensive set of metrics to assess the proposed framework from the perspectives of business growth, cost efficiency, and long\-term sustainability:

- •Average Revenue \(REV\):The average monetary revenue accrued by the platform from user transactions following incentive exposure\.
- •Return on Investment \(ROI\):Defined as the ratio of Total Revenue to Total Incentive Cost \(ROI=RevenueCost\\text\{ROI\}=\\frac\{\\text\{Revenue\}\}\{\\text\{Cost\}\}\)\. This measures the economic efficiency of the allocation strategy\.
- •ROI Vialate Rate \(RVR\):We propose this new metric to measure constraint satisfaction\. It is defined as the percentage of sliding windows in which the realized ROI falls below the target threshold\. Lower is better\.

### 4\.2\.Overall Performance \(RQ1\)

To demonstrate the comprehensive superiority of GOAL, we compare it withλ\\lambda=0\.5 against three representative state\-of\-the\-art approaches, covering representative paradigms of offline decision making, including generative sequence modeling methods \(DT\(Chenet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib126)\)and CDT\(Liuet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib127)\)\), unconstrained offline reinforcement learning \(IQL\(Kostrikovet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib14)\)\) and constrained reinforcement learning approaches \(CAL\(Wuet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib128)\)and TREBI\(Linet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib129)\)\)\. More details about datasets are shown in Appendix[F](https://arxiv.org/html/2608.04421#A6)\.

The overall performance comparisons of different approaches on the industrial dataset are reported in Table[1](https://arxiv.org/html/2608.04421#S4.T1)\. From the experimental results, we can draw the following key observations:\(1\) Superior Performance:GOAL achieves a substantial improvement in total revenue while simultaneously attaining the highest ROI and a lower RVR, indicating that GOAL is able to effectively exploit incentive opportunities without sacrificing efficiency or constraint stability\.\(2\) Beyond MDP\-Based Optimization: The generative\-based methods consistently outperform MDP\-based reinforcement learning approaches across multiple evaluation metrics, clearly demonstrating the advantages of generative approaches in balancing constraint satisfaction and revenue optimization\.\(3\) Explicit constraint\-conditioned generation is crucial for incentivized decision\-making:GOAL consistently outperforms CDT and constrained reinforcement learning methods in terms of Revenue, ROI, and RVR, indicating that explicitly conditioning the generative policy on global constraints is a more effective paradigm than post\-hoc constrained optimization\.

Table 1\.Overall performance of GOAL on the IA dataset\. The best results are shown inbold, and the second\-best results areunderlined\. Note that the REV is measured in USD\.MethodIA DatasetSynthetic IAREV↑\\uparrowROI↑\\uparrowRVR↓\\downarrowREV↑\\uparrowROI↑\\uparrowRVR↓\\downarrowDT0\.274\.7418\.23%0\.193\.6413\.51%CDT0\.224\.6015\.00%0\.364\.1210\.24%IQL0\.253\.6824\.69%0\.153\.5918\.05%CAL0\.193\.7021\.82%0\.123\.6114\.49%TREBI0\.193\.9220\.21%0\.093\.7214\.21%GOAL \(Ours\)0\.324\.9312\.24%0\.294\.239\.20%*Improv\.**\+18\.52%**\+4\.01%**\+18\.40%**–**\+2\.40%**\+9\.86%*

### 4\.3\.Dynamic Constraint Adaptation \(RQ2\)

To answer RQ2 regarding dynamic adaptability, we conduct a sensitivity analysis on the Lagrange multiplierλ\\lambda, which controls the trade\-off between incentive cost and revenue\. By sweepingλ\\lambdaon the test set, we examine how the learned policy adjusts its behavior under different constraint tightness levels, as shown in Fig\.[4](https://arxiv.org/html/2608.04421#S4.F4)\.

As illustrated in Fig\.[4\(a\)](https://arxiv.org/html/2608.04421#S4.F4.sf1), increasingλ\\lambdaleads to a clear and monotonic reduction in the average incentive cost, indicating that higher constraint pressure encourages the policy to adopt more cost\-efficient actions\. Meanwhile, Fig\.[4\(b\)](https://arxiv.org/html/2608.04421#S4.F4.sf2)shows that the average revenue also decreases asλ\\lambdaincreases, reflecting the expected trade\-off between revenue maximization and cost control\.

![Refer to caption](https://arxiv.org/html/2608.04421v1/x4.png)\(a\)Average incentive cost vs\.λ\\lambda\.
![Refer to caption](https://arxiv.org/html/2608.04421v1/x5.png)\(b\)Average revenue vs\.λ\\lambda\.

Figure 4\.Effect of the Lagrangian multiplierλ\\lambdaon incentive cost and revenue\.In the low\-λ\\lambdaregime, the model follows an aggressive strategy by allocating larger incentives to stimulate user engagement, resulting in higher cost and revenue\. Asλ\\lambdaincreases, the policy gradually shifts toward more conservative behavior, selectively reducing incentives for low\-efficiency interactions\. These results demonstrate thatλ\\lambdaserves as an effective inference\-time control knob, enabling smooth and predictable adjustment of the policy between aggressive and conservative regimes without retraining\.

### 4\.4\.Ablation Study \(RQ3\)

To assess the effectiveness of each component in GOAL, we conducted a series of ablation studies\. Specifically, we build several variants of GOAL:

- •w/o SCPO:A variant of GOAL without the Safe Constrained Policy Optimization stage, which directly uses the supervised fine\-tuned \(SFT\) model to generate incentive tokens\.
- •w/o DCC:A variant of GOAL without the DCC module, which utilizes standard self\-attention mechanisms\.
- •SCPO w/oλ\\lambda\-generalizaition:A variant of GOAL that applies SCPO with a fixed Lagrange multiplierλ\\lambda, without conditioning the policy onλ\\lambda\.
- •w/o Tokenizer:A variant that replaces the autoregressive tokenizer with a direct regression head, outputting a single scalar instead of a token sequence under the same configurations\.

Table 2\.Ablation study on IA dataset\. The target ROI threshold is set toτ=3\.0\\tau=3\.0\.MethodREV\(↑\\uparrow\)ROI\(↑\\uparrow\)RVR\(↓\\downarrow\)w/o SCPO0\.27±\\pm0\.014\.57±\\pm0\.0415\.64%±\\pm0\.02w/o DCC0\.25±\\pm0\.014\.91±\\pm0\.0512\.29%±\\pm0\.02SCPO w/oλ\\lambda\-gen0\.25±\\pm0\.034\.89±\\pm0\.0214\.12%±\\pm0\.04w/o Tokenizer0\.29±\\pm0\.024\.89±\\pm0\.0313\.85%±\\pm0\.03GOAL \(Ours\)0\.32±\\pm0\.014\.93±\\pm0\.0212\.24%±\\pm0\.01

Overall, the ablation results in Table[2](https://arxiv.org/html/2608.04421#S4.T2)demonstrate that our model’s effectiveness arises from the complementary roles of its components\. Specifically, the ”\-w/o SCPO” variant leads to a pronounced degradation in ROI accompanied by an increase in CVR, indicating that the SCPO module is critical for enforcing strict ROI constraints\. The ”\-w/o DCC” variant shows substantial performance degradation across all metrics; relying on standard self\-attention, the model is more prone to overfitting spurious correlations in offline data\. The ”\-SCPO w/oλ\\lambda\-gen” variant achieves high ROI with low violation rates but yields the lowest revenue, indicating that a staticλ\\lambdalacks the flexibility to adapt to dynamic system states\. Finally, the “w/o Tokenizer” variant shows a clear drop in both REV and ROI\. Without structured token generation, the model has difficulty handling the large value range and heavy\-tailed distribution of incentive allocations, leading to less accurate modeling of high\-stakes cases\.

### 4\.5\.Online Experimental Testing \(RQ4\)

![Refer to caption](https://arxiv.org/html/2608.04421v1/x6.png)Figure 5\.Online Deployment of GOALTo validate the effectiveness of our model on live traffic, we adapt our framework to the online serving environment, as shown in Fig\.[5](https://arxiv.org/html/2608.04421#S4.F5)\. We conducted an extended online A/B test on an incentivized advertising system spanning 4 weeks, where each experimental group encompassed 65% of randomly selected users\. The control group utilized the production baseline, while the experimental group deployed GOAL\. Compared to the baseline, GOAL achieved a2\.184%increase in ROI and a2\.559%lift in revenue\. Crucially, the ROI improvement remained highly stable over this longer observation window, while secondary metrics such as user dwell time and DAU exhibited no significant adverse changes\. These core business gains reached statistical significance with app\-value of0\.030\.03, providing robust evidence of our approach’s long\-term effectiveness and financial safety in live commercial systems\.

## 5\.Related Works

### 5\.1\.Incentive Constraint Rewards Decision

A parallel domain is budget\-constrained coupon allocation, which optimizes GMV subject to a fixed budget, distinct from our objective of maximizing revenue under strict ROI constraints\. Mainstream methodologies in this domain fall into two primary categories: Resource Allocation \(RA\) and Direct Learning Methods\(DLM\)\. Approaches based on RA have evolved from disjointed two\-stage paradigms\(Aiet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib109); Albert and Goldenberg,[2022](https://arxiv.org/html/2608.04421#bib.bib110); Wanget al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib111)\)to end\-to\-end Decision\-Focused Learning\(Zhouet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib112); Zhanget al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib113)\)\. Works such as\(Aiet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib109); Albert and Goldenberg,[2022](https://arxiv.org/html/2608.04421#bib.bib110); Wanget al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib111)\)typically adopt a predict\-then\-optimize paradigm, utilizing causal inference models to estimate average treatment effects and subsequently formulating resource allocation as a knapsack problem\. In contrast,\(Zhouet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib112); Zhanget al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib113)\)addresses the non\-differentiability of decision losses to achieve the end\-to\-end integration of prediction and optimization modules\. Nevertheless, these approaches are limited by the difficulty of counterfactual inference, compounded by an inherent myopia that neglects future ecosystem value\. The second line of research, DLM, bypasses the separation of ML and OR by formulating the problem as a constrained optimization problem, where a substantial body of work\(Xiaoet al\.,[2019](https://arxiv.org/html/2608.04421#bib.bib114); Zhanget al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib85); Guoet al\.,[2025b](https://arxiv.org/html/2608.04421#bib.bib115)\)frames the coupon allocation task in sequential incentive marketing as a Constrained Markov Decision Process \(CMDP\) and adopts RL to solve it\. These methods typically employ Lagrangian multipliers to incorporate resource constraints directly into the reward function\. While capable of modeling long\-term rewards, their application to incentivized advertising is hindered by the strict Markovian assumption and the inherent conservatism of offline RL algorithms\(Kiyoharaet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib87); Korenkevychet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib88); Liuet al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib89)\)\.

Crucially, different from our business scenario, existing methods model incentives as unordered categorical labels\. However, in our incentive system \(e\.g\., coin distribution\), no such categorical labels are available, making these methods not directly applicable to our precise and fine\-grained continuous decision\-making task\.

### 5\.2\.Generative Sequence Modeling and Preference Alignment

The field of user behavior modeling has witnessed a fundamental paradigm shift, transitioning from discriminative matching to generative sequence modeling\. Propelled by the triumph of Large Language Models \(LLMs\), this generative paradigm has gained significant traction\. Notably, recent breakthroughs in generative recommendation\(Denget al\.,[2025](https://arxiv.org/html/2608.04421#bib.bib95); Genget al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib93)\)and search\(Guoet al\.,[2025a](https://arxiv.org/html/2608.04421#bib.bib107); Tayet al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib108)\)have underscored the immense potential of formulating user modeling as a sequence generation task\. To steer these generative paradigms towards specific goals, researchers have adopted Preference Alignment techniques\. These methods aim to calibrate the model’s generative distribution to match specific reward signals\. In pursuit of this alignment, several paradigms have emerged\. RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2608.04421#bib.bib105)\)typically employs PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2608.04421#bib.bib31)\)to maximize a learned reward function, yet its reliance on the Actor\-Critic architecture often leads to training instability and high computational costs\. To mitigate this, DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib106)\)derives an analytical solution to implicitly optimize the reward, treating alignment as a stable classification task\. Nevertheless, this approach limits the model’s ability to explore beyond historical data boundaries\. More recently, GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib98)\)estimates advantages via group sampling, eliminating the need for a value network to enhance efficiency\. However, these methods predominantly focus on maximizing unconstrained scalar rewards, fundamentally overlooking the non\-negligible cost constraints inherent to incentivized advertising\.

## 6\.Conclusion

In this paper, we introduced GOAL, the first generative framework designed for incentivized advertising under strict ROI constraints\. By unifying causal user state modeling, constraint\-aware generation, and SCPO, GOAL addresses the limitations of conventional myopic uplift models and rigid offline RL approaches\. Extensive experiments on large\-scale industrial datasets demonstrate that GOAL not only outperforms state\-of\-the\-art baselines in maximizing Revenue and user retention but also exhibits superior stability in constraint adherence\. Notably, our analysis highlights the efficacy of the proposed Constraint\-Aware MoE andλ\\lambda\-generalization mechanism, which empowers the model to adapt to dynamic operational requirements without retraining\. This work marks a paradigm shift from traditional prediction\-allocation pipelines to a unified generative control process, providing a scalable and flexible solution for complex decision\-making in high\-stakes commercial environments\. In the future, we plan to extend this framework to multi\-objective optimization scenarios, such as simultaneously balancing platform revenue, advertiser ROI, and user experience\.

## References

- M\. Ai, B\. Li, H\. Gong, Q\. Yu, S\. Xue, Y\. Zhang, Y\. Zhang, and P\. Jiang \(2022\)Lbcf: a large\-scale budget\-constrained causal forest algorithm\.InProceedings of the ACM Web Conference 2022,pp\. 2310–2319\.Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- J\. Albert and D\. Goldenberg \(2022\)E\-commerce promotions personalization via online multiple\-choice knapsack with uplift modeling\.InProceedings of the 31st ACM International Conference on Information & Knowledge Management,pp\. 2863–2872\.Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- S\. Bai \(2018\)An empirical evaluation of generic convolutional and recurrent networks for sequence modeling\.arXiv preprint arXiv:1803\.01271\.Cited by:[§3\.2\.1](https://arxiv.org/html/2608.04421#S3.SS2.SSS1.p2.6)\.
- J\. Cao, P\. Xu, Y\. Cheng, K\. Guo, J\. Tang, S\. Wang, D\. Leng, S\. Yang, Z\. Liu, Y\. Niu,et al\.\(2025\)Pantheon: personalized multi\-objective ensemble sort via iterative pareto policy optimization\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 5575–5582\.Cited by:[§3\.3](https://arxiv.org/html/2608.04421#S3.SS3.p2.7)\.
- F\. Chen, X\. Liu, B\. Tang, F\. Xiong, S\. Hwang, and G\. Zhuang \(2022\)BCRLSP: an offline reinforcement learning framework for sequential targeted promotion\.arXiv preprint arXiv:2207\.07790\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p1.1)\.
- L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. Mordatch \(2021\)Decision transformer: reinforcement learning via sequence modeling\.Advances in neural information processing systems34,pp\. 15084–15097\.Cited by:[1st item](https://arxiv.org/html/2608.04421#A6.I1.i1.p1.1),[§4\.2](https://arxiv.org/html/2608.04421#S4.SS2.p1.1)\.
- J\. Deng, S\. Wang, K\. Cai, L\. Ren, Q\. Hu, W\. Ding, Q\. Luo, and G\. Zhou \(2025\)Onerec: unifying retrieve and rank with generative recommender and iterative preference alignment\.arXiv preprint arXiv:2502\.18965\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p1.1),[§1](https://arxiv.org/html/2608.04421#S1.p3.1),[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong, D\. Lepikhin, Y\. Xu, M\. Krikun, Y\. Zhou, A\. W\. Yu, O\. Firat,et al\.\(2022\)Glam: efficient scaling of language models with mixture\-of\-experts\.InInternational conference on machine learning,pp\. 5547–5569\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.04421#S3.SS2.SSS2.p2.4)\.
- S\. Geng, S\. Liu, Z\. Fu, Y\. Ge, and Y\. Zhang \(2022\)Recommendation as language processing \(rlp\): a unified pretrain, personalized prompt & predict paradigm \(p5\)\.InProceedings of the 16th ACM conference on recommender systems,pp\. 299–315\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1),[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- D\. Goldenberg, J\. Albert, L\. Bernardi, and P\. Estevez \(2020\)Free lunch\! retrospective uplift modeling for dynamic promotions recommendation within roi constraints\.InProceedings of the 14th ACM Conference on Recommender Systems,pp\. 486–491\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p1.1),[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- X\. Guo, B\. Chen, S\. Wang, Y\. Yang, C\. Lei, Y\. Ding, and H\. Li \(2025a\)OneSug: the unified end\-to\-end generative framework for e\-commerce query suggestion\.arXiv preprint arXiv:2506\.06913\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1),[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- Z\. Guo, W\. Zhou, S\. Wang, and W\. Li \(2025b\)Constraint\-conditioned actor\-critic for offline safe reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- B\. J\. Hansotia and B\. Rukstales \(2002\)Direct marketing for multichannel retailers: issues, challenges and solutions\.Journal of Database Marketing & Customer Strategy Management9\(3\),pp\. 259–266\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- M\. J\. Hausknecht and P\. Stone \(2015\)Deep recurrent q\-learning for partially observable mdps\.\.InAAAI fall symposia,Vol\.45,pp\. 141\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- H\. Iida, D\. Thai, V\. Manjunatha, and M\. Iyyer \(2021\)Tabbie: pretrained representations of tabular data\.arXiv preprint arXiv:2105\.02584\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.04421#S3.SS2.SSS2.p1.1)\.
- Z\. Jie, J\. Li, and W\. Lu \(2022\)Learning to reason deductively: math word problem solving as complex relation extraction\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5944–5955\.Cited by:[§3\.1](https://arxiv.org/html/2608.04421#S3.SS1.p1.1)\.
- Z\. Jin, X\. Jiang, X\. Wang, Q\. Liu, Y\. Wang, X\. Ren, and H\. Qu \(2021\)Numgpt: improving numeracy ability of generative pre\-trained models\.arXiv preprint arXiv:2109\.03137\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.04421#S3.SS2.SSS2.p1.1)\.
- H\. Kiyohara, K\. Kawakami, and Y\. Saito \(2021\)Accelerating offline reinforcement learning application in real\-time bidding and recommendation: potential use of simulation\.arXiv preprint arXiv:2109\.08331\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- D\. Korenkevych, F\. Cheng, A\. Balakir, A\. Nikulkov, L\. Gao, Z\. Cen, Z\. Xu, and Z\. Zhu \(2024\)Offline reinforcement learning for optimizing production bidding policies\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 5251–5259\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- I\. Kostrikov, A\. Nair, and S\. Levine \(2021\)Offline reinforcement learning with implicit q\-learning\.arXiv preprint arXiv:2110\.06169\.Cited by:[3rd item](https://arxiv.org/html/2608.04421#A6.I1.i3.p1.1),[§4\.2](https://arxiv.org/html/2608.04421#S4.SS2.p1.1)\.
- N\. Li, X\. Ban, C\. Ling, C\. Gao, L\. Hu, P\. Jiang, K\. Gai, Y\. Li, and Q\. Liao \(2024\)Modeling user fatigue for sequential recommendation\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 996–1005\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- W\. Li, H\. Luo, Z\. Lin, C\. Zhang, Z\. Lu, and D\. Ye \(2023\)A survey on transformers in reinforcement learning\.arXiv preprint arXiv:2301\.03044\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1)\.
- Q\. Lin, B\. Tang, Z\. Wu, C\. Yu, S\. Mao, Q\. Xie, X\. Wang, and D\. Wang \(2023\)Safe offline reinforcement learning with real\-time budget constraints\.InInternational Conference on Machine Learning,pp\. 21127–21152\.Cited by:[5th item](https://arxiv.org/html/2608.04421#A6.I1.i5.p1.1),[§4\.2](https://arxiv.org/html/2608.04421#S4.SS2.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1)\.
- T\. Liu, Q\. Xu, W\. Shi, Z\. Hua, and S\. Yang \(2025\)Session\-level dynamic ad load optimization using offline robust reinforcement learning\.arXiv preprint arXiv:2501\.05591\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- Z\. Liu, Z\. Guo, Y\. Yao, Z\. Cen, W\. Yu, T\. Zhang, and D\. Zhao \(2023\)Constrained decision transformer for offline safe reinforcement learning\.InInternational conference on machine learning,pp\. 21611–21630\.Cited by:[2nd item](https://arxiv.org/html/2608.04421#A6.I1.i2.p1.1),[§4\.2](https://arxiv.org/html/2608.04421#S4.SS2.p1.1)\.
- H\. Ma, K\. Tian, T\. Zhang, X\. Zhang, H\. Zhou, C\. Chen, H\. Li, J\. Guan, and S\. Zhou \(2024\)Generative regression based watch time prediction for short\-video recommendation\.arXiv preprint arXiv:2412\.20211\.Cited by:[§3\.1](https://arxiv.org/html/2608.04421#S3.SS1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- C\. Pei, X\. Yang, Q\. Cui, X\. Lin, F\. Sun, P\. Jiang, W\. Ou, and Y\. Zhang \(2019\)Value\-aware recommendation based on reinforcement profit maximization\.InThe World Wide Web Conference,pp\. 3123–3129\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- S\. Rajput, N\. Mehta, A\. Singh, R\. Hulikal Keshavan, T\. Vu, L\. Heldt, L\. Hong, Y\. Tay, V\. Tran, J\. Samost,et al\.\(2023\)Recommender systems with generative retrieval\.Advances in Neural Information Processing Systems36,pp\. 10299–10315\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- Y\. Tay, V\. Tran, M\. Dehghani, J\. Ni, D\. Bahri, H\. Mehta, Z\. Qin, K\. Hui, Z\. Zhao, J\. Gupta,et al\.\(2022\)Transformer memory as a differentiable search index\.Advances in Neural Information Processing Systems35,pp\. 21831–21843\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1),[§5\.2](https://arxiv.org/html/2608.04421#S5.SS2.p1.1)\.
- A\. Venkatraman, M\. Hebert, and J\. Bagnell \(2015\)Improving multi\-step prediction of learned time series models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.29\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.04421#S3.SS2.SSS2.p8.5)\.
- E\. Wallace, Y\. Wang, S\. Li, S\. Singh, and M\. Gardner \(2019\)Do nlp models know numbers? probing numeracy in embeddings\.arXiv preprint arXiv:1909\.07940\.Cited by:[§3\.1](https://arxiv.org/html/2608.04421#S3.SS1.p1.1)\.
- C\. Wang, X\. Shi, S\. Xu, Z\. Wang, Z\. Fan, Y\. Feng, A\. You, and Y\. Chen \(2023\)A multi\-stage framework for online bonus allocation based on constrained user intent detection\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 5028–5038\.Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- Z\. Wang, H\. Dong, R\. Jia, J\. Li, Z\. Fu, S\. Han, and D\. Zhang \(2021\)Tuta: tree\-based transformers for generally structured table pre\-training\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,pp\. 1780–1790\.Cited by:[§3\.1](https://arxiv.org/html/2608.04421#S3.SS1.p1.1),[§3\.2\.2](https://arxiv.org/html/2608.04421#S3.SS2.SSS2.p1.1)\.
- D\. Wu, X\. Chen, X\. Yang, H\. Wang, Q\. Tan, X\. Zhang, J\. Xu, and K\. Gai \(2018\)Budget constrained bidding by model\-free reinforcement learning in display advertising\.InProceedings of the 27th ACM International Conference on Information and Knowledge Management,pp\. 1443–1451\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p1.1)\.
- Z\. Wu, B\. Tang, Q\. Lin, C\. Yu, S\. Mao, Q\. Xie, X\. Wang, and D\. Wang \(2024\)Off\-policy primal\-dual safe reinforcement learning\.arXiv preprint arXiv:2401\.14758\.Cited by:[4th item](https://arxiv.org/html/2608.04421#A6.I1.i4.p1.1),[§4\.2](https://arxiv.org/html/2608.04421#S4.SS2.p1.1)\.
- S\. Xiao, L\. Guo, Z\. Jiang, L\. Lv, Y\. Chen, J\. Zhu, and S\. Yang \(2019\)Model\-based constrained mdp for budget allocation in sequential incentive marketing\.InProceedings of the 28th ACM International Conference on Information and Knowledge Management,pp\. 971–980\.Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- T\. Xiao, M\. Singh, E\. Mintun, T\. Darrell, P\. Dollár, and R\. Girshick \(2021\)Early convolutions help transformers see better\.Advances in neural information processing systems34,pp\. 30392–30400\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1)\.
- S\. Zhang, H\. Zhou, J\. Zheng, G\. Jiang, B\. Cheng, W\. Lin, and G\. Chen \(2025\)Bi\-level decision\-focused causal learning for large\-scale marketing optimization: bridging observational and experimental data\.arXiv preprint arXiv:2510\.19517\.Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- Y\. Zhang, B\. Tang, Q\. Yang, D\. An, H\. Tang, C\. Xi, X\. Li, and F\. Xiong \(2021\)BCORLE \(λ\\lambda\): an offline reinforcement learning and evaluation framework for coupons allocation in e\-commerce market\.Advances in Neural Information Processing Systems34,pp\. 20410–20422\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p1.1),[§3\.4](https://arxiv.org/html/2608.04421#S3.SS4.p2.1),[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- K\. Zhao, J\. Hua, L\. Yan, Q\. Zhang, H\. Xu, and C\. Yang \(2019\)A unified framework for marketing budget allocation\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 1820–1830\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- X\. Zhao, L\. Xia, L\. Zhang, Z\. Ding, D\. Yin, and J\. Tang \(2018\)Deep reinforcement learning for page\-wise recommendations\.InProceedings of the 12th ACM conference on recommender systems,pp\. 95–103\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- G\. Zheng, F\. Zhang, Z\. Zheng, Y\. Xiang, N\. J\. Yuan, X\. Xie, and Z\. Li \(2018\)DRN: a deep reinforcement learning framework for news recommendation\.InProceedings of the 2018 world wide web conference,pp\. 167–176\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.
- H\. Zhou, R\. Huang, S\. Li, G\. Jiang, J\. Zheng, B\. Cheng, and W\. Lin \(2024\)Decision focused causal learning for direct counterfactual marketing optimization\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 6368–6379\.Cited by:[§5\.1](https://arxiv.org/html/2608.04421#S5.SS1.p1.1)\.
- H\. Zhou, S\. Zhang, J\. Peng, S\. Zhang, J\. Li, H\. Xiong, and W\. Zhang \(2021\)Informer: beyond efficient transformer for long sequence time\-series forecasting\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 11106–11115\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p3.1)\.
- B\. Zoph \(2022\)Designing effective sparse expert models\.In2022 IEEE International Parallel and Distributed Processing Symposium Workshops \(IPDPSW\),pp\. 1044–1044\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.04421#S3.SS2.SSS2.p2.4)\.
- L\. Zou, L\. Xia, Z\. Ding, J\. Song, W\. Liu, and D\. Yin \(2019\)Reinforcement learning to optimize long\-term user engagement in recommender systems\.InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 2810–2818\.Cited by:[§1](https://arxiv.org/html/2608.04421#S1.p2.1)\.

## APPENDIX

## Appendix ASelection of Candidateλ\\lambdaSets

To enable the policy to perceive and adapt to varying business requirements without retraining, we formalize a Goal\-conditioned Policyπ​\(a\|s,λ\)\\pi\(a\|s,\\lambda\)\. In this framework, the state space is augmented with the Lagrangian multiplierλ\\lambdaas a contextual indicator\. The objective is to learn a universal strategy manifold that approximates the Pareto frontier between revenue and ROI\.

A fundamental challenge is to define an effective sampling interval\[λmin,λmax\]\[\\lambda\_\{\\min\},\\lambda\_\{\\max\}\]that covers all potentially optimal strategies\. Ifλ\\lambdais too small, the policy fails to satisfy ROI constraints; ifλ\\lambdais excessively large, the policy may collapse prematurely\. We derive the boundaries based on the marginal utility of actions:

- •Lower Bound \(λmin=0\\lambda\_\{\\min\}=0\):Whenλ=0\\lambda=0, the objectiveri−λ​cir\_\{i\}\-\\lambda c\_\{i\}reduces to pure revenue maximization\. This represents the Unconstrained Boundary, where the policy explores the maximum possible incentive intensity to establish the upper limit of revenue\.
- •Upper Bound \(λmax=τ\\lambda\_\{\\max\}=\\tau\):According to the derived dual relationshipλ=λ′​τ1\+λ′\\lambda=\\frac\{\\lambda^\{\\prime\}\\tau\}\{1\+\\lambda^\{\\prime\}\}in Eq\.[3](https://arxiv.org/html/2608.04421#S2.E3), the effective penalty coefficientλ\\lambdaasymptotically approaches the ROI thresholdτ\\tauas the dual penaltyλ′→∞\\lambda^\{\\prime\}\\to\\infty\.

To ensure the theoretical rigor of this boundary, we introduce the following property:

Proposition 1\. For a discrete action set𝒜\\mathcal\{A\}with ascending costsc1<c2<⋯<ckc\_\{1\}<c\_\{2\}<\\dots<c\_\{k\}, there exists a finite thresholdλ¯=maxi\>1⁡ri−r1ci−c1\\bar\{\\lambda\}=\\max\_\{i\>1\}\\frac\{r\_\{i\}\-r\_\{1\}\}\{c\_\{i\}\-c\_\{1\}\}such that for allλ≥λ¯\\lambda\\geq\\bar\{\\lambda\}, the optimal actiona=arg⁡maxai∈𝒜⁡\(ri−λ​ci\)a=\\arg\\max\_\{a\_\{i\}\\in\\mathcal\{A\}\}\(r\_\{i\}\-\\lambda c\_\{i\}\)consistently converges to the minimum\-cost actiona1a\_\{1\}\.

Proof\.Consider the objective functionℱ​\(λ,ai\)=ri−λ​ci\\mathcal\{F\}\(\\lambda,a\_\{i\}\)=r\_\{i\}\-\\lambda c\_\{i\}\. We aim to find the condition wherea1a\_\{1\}is the unique optimizer, i\.e\.,a1=arg⁡maxai⁡ℱ​\(λ,ai\)a\_\{1\}=\\arg\\max\_\{a\_\{i\}\}\\mathcal\{F\}\(\\lambda,a\_\{i\}\), which must satisfy the system of linear inequalitiesr1−λ​c1\>ri−λ​cir\_\{1\}\-\\lambda c\_\{1\}\>r\_\{i\}\-\\lambda c\_\{i\}for alli∈\{2,…,k\}i\\in\\\{2,\\dots,k\\\}\. Isolating the terms involvingλ\\lambdayieldsλ​\(ci−c1\)\>ri−r1\\lambda\(c\_\{i\}\-c\_\{1\}\)\>r\_\{i\}\-r\_\{1\}\. Given the premise of ascending costsci\>cjc\_\{i\}\>c\_\{j\}fori\>ji\>j, the coefficient\(ci−c1\)\(c\_\{i\}\-c\_\{1\}\)is strictly positive for alli\>1i\>1\. Dividing both sides by this term defines a set of lower bounds for the multiplierλ\\lambdaasλ\>ri−r1ci−c1\\lambda\>\\frac\{r\_\{i\}\-r\_\{1\}\}\{c\_\{i\}\-c\_\{1\}\}for alli∈\{2,…,k\}i\\in\\\{2,\\dots,k\\\}\. We define the critical thresholdλ¯\\bar\{\\lambda\}as the supremum of these individual action\-wise bounds:λ¯=maxi∈\{2,…,k\}⁡\(ri−r1ci−c1\)\\bar\{\\lambda\}=\\max\_\{i\\in\\\{2,\\dots,k\\\}\}\\left\(\\frac\{r\_\{i\}\-r\_\{1\}\}\{c\_\{i\}\-c\_\{1\}\}\\right\)\. It follows that for anyλ≥λ¯\\lambda\\geq\\bar\{\\lambda\}, the inequalityℱ​\(λ,a1\)≥ℱ​\(λ,ai\)\\mathcal\{F\}\(\\lambda,a\_\{1\}\)\\geq\\mathcal\{F\}\(\\lambda,a\_\{i\}\)is guaranteed for the entire action set𝒜\\mathcal\{A\}\. In this regime, the marginal penalty imposed byλ\\lambdaper unit of cost exceeds the marginal reward gain of any alternative actionaia\_\{i\}relative toa1a\_\{1\}\. Thus, the optimal policyπ∗\\pi^\{\*\}necessarily converges to the point mass ona1a\_\{1\}\. This completes the proof\.□\\square

In practical scenarios, business requirements often specify a minimum ROI thresholdτ\\tauthat varies across different marketing campaigns\. To ensure our policy can generalize to the most stringent constraints, we identify the maximum value among these potential ROI targets, denoted asτmax\\tau\_\{\\max\}\. Sinceτmax\\tau\_\{\\max\}typically exceeds the marginal utility thresholdλ¯\\bar\{\\lambda\}, we utilize it as the practical sampling ceilingλmax\\lambda\_\{\\max\}\. This ensures that the training interval\[0,λmax\]\[0,\\lambda\_\{\\max\}\]encompasses all operationally relevant strategy transitions, from unconstrained growth to the most conservative efficiency requirements\.

To implement this in our training pipeline, we adopt a discretization and uniform sampling approach\. Specifically, the continuous interval\[0,λmax\]\[0,\\lambda\_\{\\max\}\]is partitioned into a candidate setΛ=\{0,Δ​λ,2​Δ​λ,…,λmax\}\\Lambda=\\\{0,\\Delta\\lambda,2\\Delta\\lambda,\\dots,\\lambda\_\{\\max\}\\\}with a fixed step sizeΔ​λ\\Delta\\lambda\. For instance, withλmax=3\.0\\lambda\_\{\\max\}=3\.0andΔ​λ=0\.5\\Delta\\lambda=0\.5, the model explores 7 distinct cost\-penalty scenarios\. During each training iteration, a targetλ\\lambdais randomly drawn fromΛ\\Lambdaand concatenated with the system statess, enabling the policyπ​\(a\|s,λ\)\\pi\(a\|s,\\lambda\)to internalize the mapping between varying ROI constraints and their corresponding optimal decision logics\. This stochastic exposure ensures the model approximates a continuous Pareto frontier rather than over\-fitting to a single trade\-off point\.

## Appendix BSelection of Optimalλ∗\\lambda^\{\*\}at Inference

To ensure the learned policy strictly adheres to the target ROI thresholdτ\\tauduring deployment, we employ a binary search procedure to identify the optimal dual variableλ∗\\lambda^\{\*\}, leveraging the monotonicity of the realized ROI with respect toλ\\lambda, as detailed in Algorithm[2](https://arxiv.org/html/2608.04421#alg2)\.

0:Trained Policy

πθ\\pi\_\{\\theta\}, Calibration Dataset

𝒟\\mathcal\{D\}, Target ROI threshold

τ\\tau
0:Constraint spectrum

Λc​a​n​d=\{λ1,…,λn\}\\Lambda\_\{cand\}=\\\{\\lambda\_\{1\},\\dots,\\lambda\_\{n\}\\\}\(sorted in ascending order\)

10:Tolerance

ϵ\\epsilon
21:Initialize:

l​o​w←1,h​i​g​h←n,λ∗←λnlow\\leftarrow 1,\\ high\\leftarrow n,\\ \\lambda^\{\*\}\\leftarrow\\lambda\_\{n\}\{Default to safety\}

2:while

l​o​w≤h​i​g​hlow\\leq highdo

3:

m​i​d←⌊\(l​o​w\+h​i​g​h\)/2⌋mid\\leftarrow\\lfloor\(low\+high\)/2\\rfloor
34:

λc​u​r​r←Λc​a​n​d​\[m​i​d\]\\lambda\_\{curr\}\\leftarrow\\Lambda\_\{cand\}\[mid\]
5:\{Empirical Policy Evaluation\}

6:Rollout: Generate actions

a∼πθ\(⋅\|s,λc​u​r​r\)a\\sim\\pi\_\{\\theta\}\(\\cdot\|s,\\lambda\_\{curr\}\)for all

s∈𝒟s\\in\\mathcal\{D\}
47:Calculate Realized ROI:

ℛ^←∑ri∑ci\\hat\{\\mathcal\{R\}\}\\leftarrow\\frac\{\\sum r\_\{i\}\}\{\\sum c\_\{i\}\}
8:\{Binary Search Range Update\}

9:if

ℛ^≥τ\\hat\{\\mathcal\{R\}\}\\geq\\tauthen

10:

λ∗←λc​u​r​r\\lambda^\{\*\}\\leftarrow\\lambda\_\{curr\}\{ROI threshold satisfied\}

11:

h​i​g​h←m​i​d−1high\\leftarrow mid\-1
12:else

13:

l​o​w←m​i​d\+1low\\leftarrow mid\+1\{ROI threshold violated\}

14:endif

515:endwhile

16:return

λ∗\\lambda^\{\*\}

Algorithm 2λ\\lambdaCalibration
## Appendix CSafe Constrained Policy Optimization

The algorithmic details of SCPO are presented in Algorith[3](https://arxiv.org/html/2608.04421#alg3)\.

0:User state dataset

𝒟\\mathcal\{D\}, Constraint spectrum

Λ=\{λ1,…,λn\}\\Lambda=\\\{\\lambda\_\{1\},\\dots,\\lambda\_\{n\}\\\}
0:Policy

πθ\\pi\_\{\\theta\}, Reference policy

πref\\pi\_\{\\text\{ref\}\}, Group size

mm
0:Hyperparameters: clipping

ϵ\\epsilon, KL coefficient

β\\beta, Learning rate

η\\eta
1:whilenot convergeddo

2:Sample a batch of states

Bs∼𝒟B\_\{s\}\\sim\\mathcal\{D\}
3:foreach state

s∈Bss\\in B\_\{s\}do

4:\{Candidate Generation\}

5:Sample

mmcandidate responses

\{oi\}i=1m∼πθ\(⋅\|s\)\\\{o\_\{i\}\\\}\_\{i=1\}^\{m\}\\sim\\pi\_\{\\theta\}\(\\cdot\|s\)
6:\{Cartesian Augmentation & Reward Evaluation\}

7:foreach constraint level

λj∈Λ\\lambda\_\{j\}\\in\\Lambdado

8:Compute rewards for all candidates under the condition

λj\\lambda\_\{j\}:

19:

ri,j←R​\(s,oi\)−λj​C​\(s,oi\),∀i∈\{1,…,m\}r\_\{i,j\}\\leftarrow R\(s,o\_\{i\}\)\-\\lambda\_\{j\}C\(s,o\_\{i\}\),\\quad\\forall i\\in\\\{1,\\dots,m\\\}
10:\{Constraint\-Conditional Normalization\}

11:Compute group statistics specifically for

λj\\lambda\_\{j\}:

212:

μλj←1m​∑i=1mri,j,σλj←1m​∑i=1m\(ri,j−μλj\)2\\mu\_\{\\lambda\_\{j\}\}\\leftarrow\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}r\_\{i,j\},\\quad\\sigma\_\{\\lambda\_\{j\}\}\\leftarrow\\sqrt\{\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\(r\_\{i,j\}\-\\mu\_\{\\lambda\_\{j\}\}\)^\{2\}\}
13:Compute normalized advantages \(Eq\.[20](https://arxiv.org/html/2608.04421#S3.E20)\):

14:

Ai,j←ri,j−μλjσλj\+ϵ,∀i∈\{1,…,m\}A\_\{i,j\}\\leftarrow\\frac\{r\_\{i,j\}\-\\mu\_\{\\lambda\_\{j\}\}\}\{\\sigma\_\{\\lambda\_\{j\}\}\+\\epsilon\},\\quad\\forall i\\in\\\{1,\\dots,m\\\}
15:endfor

316:endfor

17:\{Policy Optimization\}

18:Construct the surrogate objective

ℒSCPO​\(θ\)\\mathcal\{L\}\_\{\\text\{SCPO\}\}\(\\theta\)over the augmented batch:

19:

ℒ←1\|Bs\|​m​n∑s∑i=1m∑j=1n\[min\(ρi,jAi,j,clip\(ρi,j,1−ϵ,1\+ϵ\)Ai,j\)−βDKL\]\\begin\{aligned\} \\mathcal\{L\}\\leftarrow&\\frac\{1\}\{\|B\_\{s\}\|\\,m\\,n\}\\sum\_\{s\}\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}\\Big\[\\min\\\!\\left\(\\rho\_\{i,j\}A\_\{i,j\},\\operatorname\{clip\}\\\!\\left\(\\rho\_\{i,j\},1\-\\epsilon,\\right\.\\right\.\\\\ &\\left\.\\left\.1\+\\epsilon\\right\)A\_\{i,j\}\\right\)\-\\beta D\_\{\\mathrm\{KL\}\}\\Big\]\\end\{aligned\}
20:Update policy parameters:

θ←θ\+η​∇θℒ\\theta\\leftarrow\\theta\+\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}
21:endwhile

Algorithm 3Safe Constrained Policy Optimization \(SCPO\)
## Appendix DConstraint\-aware Experts

To understand how the constraint\-aware MoE internalizes the global constraint signal, we analyze the routing behavior of the gating network under different values of the Lagrange multiplierλ\\lambda\. Recall thatλ\\lambdaexplicitly parameterizes the strength of the ROI constraint and is provided to the gating network as a global conditioning signal\.

Fig\.[6](https://arxiv.org/html/2608.04421#A4.F6)visualizes the average expert selection probabilities asλ\\lambdavaries from loose to strict constraint regimes\. Whenλ\\lambdais small, corresponding to a relaxed ROI constraint, the gating network predominantly activates Expert 0\. This expert is consistently assigned the highest routing probability and learns to generate aggressive incentive allocations that prioritize revenue maximization with limited regard for cost efficiency\. Asλ\\lambdaincreases, the routing probability gradually shifts from Expert 0 toward Expert 1 and Expert 2\. This smooth transition indicates that the gating network does not treatλ\\lambdaas a discrete switch, but instead continuously modulates the composition of experts to balance revenue and cost under moderate constraint pressure\. In the high\-λ\\lambdaregime, the routing mass is almost entirely concentrated on Expert 3, which specializes in conservative, cost\-sensitive incentive generation and strongly suppresses low\-efficiency actions\.

Importantly, this expert specialization is not manually imposed\. All experts share the same architecture and are trained jointly, and the observed division of labor emerges naturally from conditioning the MoE onλ\\lambdaduring training\. These results demonstrate that the constraint\-aware MoE effectively translates global constraint pressure into structured, expert\-level behavioral specialization, enabling the model to adapt its incentive strategy across different ROI regimes without retraining\.

![Refer to caption](https://arxiv.org/html/2608.04421v1/x7.png)Figure 6\.Expert specialization induced by the constraint signalλ\\lambdain the constraint\-aware MoE on IA dataset\.
## Appendix ESynthetic ROI\-Constrained Environment

To isolate policy controllability under long\-term ROI constraints away from live\-data confounders, we introduce a minimal, reproducible synthetic testbed capturing the tension between short\-term incentive gains and long\-term fatigue\-induced ROI degradation\. Each session evolves overTTtimesteps\. At steptt, the agent observes statest=\(ft,rt−1\)s\_\{t\}=\(f\_\{t\},r\_\{t\-1\}\), whereftf\_\{t\}represents user fatigue andrt−1∈\{0,1\}r\_\{t\-1\}\\in\\\{0,1\\\}is the prior engagement outcome\. The agent selects a discrete incentiveat∈\{0,1,…,K\}a\_\{t\}\\in\\\{0,1,\\dots,K\\\}with a linear costCt=atC\_\{t\}=a\_\{t\}\.

Our fatigue model is intentionally simplified as a controlled testbed for mechanistic verification\. Fatigue evolves deterministically asft\+1=ρ​ft\+η​atf\_\{t\+1\}=\\rho f\_\{t\}\+\\eta a\_\{t\}, whereρ∈\(0,1\)\\rho\\in\(0,1\)governs natural decay andη\>0\\eta\>0tracks accumulation\. We adopt this exponential decay to align with empirical observations in advertising and recommendation: fatigue from repeated exposures dissipates rapidly at first, then tapers off gradually\.

User engagementrt∼Bernoulli​\(pt\)r\_\{t\}\\sim\\mathrm\{Bernoulli\}\(p\_\{t\}\)yields the instantaneous rewardRt=rtR\_\{t\}=r\_\{t\}\. To incorporate saturation effects, fatigue scales user engagement non\-linearly via a sigmoid function, yielding the success probabilitypt=σ​\(α​at−β​ft\)p\_\{t\}=\\sigma\(\\alpha a\_\{t\}\-\\beta f\_\{t\}\), whereα\>0\\alpha\>0dictates the immediate utility of incentives andβ\>0\\beta\>0governs fatigue sensitivity\. Trajectory\-level ROI is defined asROI=∑rt/\(∑at\+ϵ\)\\mathrm\{ROI\}=\\sum r\_\{t\}/\(\\sum a\_\{t\}\+\\epsilon\)\.

To build the offline dataset, we sample10,00010,000trajectories via a mixed behavior policy \(greedy, random, and fixed medium incentives\) to ensure broad action coverage and fair evaluation\. Default parameters are set toK=10K=10,T=100T=100,α=0\.8\\alpha=0\.8,β=1\.2\\beta=1\.2,ρ=0\.9\\rho=0\.9, andη=0\.5\\eta=0\.5\.

## Appendix FDataset Details

- •DT\(Chenet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib126)\): A generative approach that casts offline RL as a conditional sequence modeling task, generating actions conditioned on the target return\-to\-go\.
- •CDT\(Liuet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib127)\): A safe RL variant of DT that conditions generation on cost thresholds\.
- •IQL\(Kostrikovet al\.,[2021](https://arxiv.org/html/2608.04421#bib.bib14)\): A state\-of\-the\-art offline RL method that mitigates out\-of\-distribution overestimation via expectile\-based value learning\.
- •CAL\(Wuet al\.,[2024](https://arxiv.org/html/2608.04421#bib.bib128)\): It solves the constrained problem by iteratively updating the policy and the dual variable\.
- •TREBI\(Linet al\.,[2023](https://arxiv.org/html/2608.04421#bib.bib129)\): It tackles constrained policy optimization by optimizing over trajectory distributions while enforcing strict constraint satisfaction\.