A Pinch of SFT, A Dash of RL: When Reinforcement Learning Helps Long-Horizon Advertising Agents

# A Pinch of SFT, A Dash of RL:When Reinforcement Learning Helps Long-Horizon Advertising Agents
Source: [https://arxiv.org/html/2609.22194](https://arxiv.org/html/2609.22194)
Sahika GencAffiliation:Amazon Web Services Agentic AIEmail:[sahika@amazon\.com](mailto:[email protected])Bunyamin SismanAffiliation:Amazon AdvertisingEmail:[bunyamis@amazon\.com](mailto:[email protected])Shahriar ShariatAffiliation:Amazon AdvertisingEmail:[sshariat@amazon\.com](mailto:[email protected])Shree Vandana KachrooAffiliation:Amazon AdvertisingEmail:[kacshree@amazon\.com](mailto:[email protected])Avishek SahaAffiliation:Amazon AdvertisingEmail:[avisaha@amazon\.com](mailto:[email protected])Qianli WuAffiliation:Amazon AdvertisingEmail:[qianliwu@amazon\.com](mailto:[email protected])Ari SingerAffiliation:Amazon AdvertisingEmail:[arising@amazon\.com](mailto:[email protected])Benoit DumoulinAffiliation:Amazon AdvertisingEmail:[bdumouli@amazon\.com](mailto:[email protected])

###### Abstract

Enterprise analytics agents solve long\-horizon tool\-use problems over distributed business data, requiring retrieval, reasoning, API calls, code execution, and adaptation to intermediate observations\. Supervised fine\-tuning \(SFT\) calibrates tool syntax and teacher\-supported behavior, whereas reinforcement learning \(RL\) can explore reward\-supported behaviors beyond demonstrations; applied uniformly, however, RL can perturb already\-calibrated skills\. We study how to balance SFT and RL under production\-mirroring beta APIs\. We observe that, in our controlled experiment, checkpoint trajectories retrospectively separated into three regimes:*Imitation*, where SFT captured reliable teacher behavior;*Lift*, where both stages helped; and*Discovery*, where useful reward\-observable behavior lay outside reliable teacher support\. We leverage this prospectively, using teacher support and reward\-observable headroom to route features to SFT only, SFT→\\rightarrowRL, increased RL allocation, or further environment development\. Across 18 subsequent feature\-specific experiments, the diagnostic predicted 15/18 observed trajectories\. On GPT\-OSS 120B, targeted SFT→\\rightarrowRL produced positive point estimates on 7/8 advertiser skills relative to a frontier Control; five positive gains had paired 95% confidence intervals excluding zero, while one skill had a confidence\-supported regression\. The largest gain was non\-disclosure \(\+11\.27\+11\.27points; 95% CI\[\+9\.72,\+12\.82\]\[\+9\.72,\+12\.82\]\)\. A separate SME audit surfaced that targeted RL reduces standard leakage from 11\.8% to 2\.9% and adversarial leakage from 22\.9% to 6\.8% relative to SFT while preserving actionability \(86\.2% to 85\.7%\)\. In a matched uniform\-versus\-targeted comparison with shared rewards and optimization, targeted RL improved the seven\-skill mean delta from\+1\.62\+1\.62to\+3\.57\+3\.57while using 43% less incremental RL compute\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/rl_pipeline_and_diagnostic_flowchart_2.png)Figure 1:Targeted post\-training pipeline\.Episodes interleave reasoning, beta\-API calls, and tool observations\. Stage I uses teacher\-distilled SFT with tool\-response masking\. The original eight\-skill study motivates a three\-regime hypothesis; a subsequent prospective diagnostic uses teacher support and frozen\-SFT reward headroom to route new features before RL\. Stage II applies GRPO only where reward\-observable headroom justifies on\-policy optimization, using tool\-validity and skill\-aware jury rewards with a KL anchor toπSFT\\pi\_\{\\mathrm\{SFT\}\}\.## 1Introduction

An advertiser running a holiday campaign across hundreds of audience segments does not ask a single, well\-scoped question\. They ask*“why is my return on ad spend down 12% this week, which audiences are driving the drop, and where should I move my budget?”*Answering this requires identifying affected campaigns, retrieving metrics across a date window, decomposing the drop by segment, and recommending spend shifts that respect advertiser objectives\. Such requests are naturally long\-horizon tool\-use problems: success is defined by outcomes over a sequence of API calls, code execution steps, and intermediate observations, not by a single next\-token target\. This makes reinforcement learning attractive, but not uniformly beneficial\. Pure RL from a cold\-started base model collapses on long\-horizon advertiser skills: when nearly every rollout fails, the group\-relative advantages that drive GRPO updates\([Shao et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib30)\)contain little usable signal\.

The central question is how to allocate SFT and RL under limited rollout budget without broad capability regression\. Teacher\-distilled SFT calibrates tool syntax and common reasoning traces, but can plateau when demonstrations do not reliably cover desired behavior\. Post\-SFT RL therefore poses a sharper allocation problem:*when does on\-policy optimization discover reward\-supported behavior beyond the teacher, and when does it merely perturb behavior SFT has already calibrated?*

In the original controlled experiment, we observe three retrospective training trajectories\. In*Imitation*, demonstrations are reliable and SFT closes most of the gap\. In*Lift*, SFT establishes a competent policy and RL adds further improvement\. In*Discovery*, the desired behavior is reward\-observable but weakly represented in teacher demonstrations\. We subsequently operationalize this hypothesis into a pre\-RL diagnostic using teacher support and reward\-observable headroom\. The diagnostic determines whether a feature receives SFT only, SFT followed by RL, increased RL allocation, or additional environment \(RL Gym\) development\. This separates the retrospective evidence that motivated the taxonomy from prospective evidence about whether it can guide new experiments\.

Our contributions are:\(i\) A regime hypothesis and prospective diagnostic:we operationalize the three regimes using teacher support and reward\-observable headroom measured before feature\-specific RL; across 18 subsequent experiments, the diagnostic predicts 15/18 observed cases\.\(ii\) A targeted post\-training recipe with a matched comparison:teacher\-distilled, tool\-response\-masked SFT is followed by GRPO only where exploration is reward\-distinguishable\. Under matched rewards and optimization settings, targeted allocation improves the seven\-skill mean delta from\+1\.62\+1\.62to\+3\.57\+3\.57and uses 43% less incremental RL compute than uniform RL\.\(iii\) SME\-grounded evaluation with paired uncertainty and human safety validation:every headline score is Avg@32 on fixed held\-out examples; we report paired 95% intervals and separately audit non\-disclosure with human SMEs, showing substantially lower leakage without a material loss in actionability\.\(iv\) Production\-scale evidence and accounting:we report failed and successful large\-scale runs, explicit offline release\-readiness criteria, complete per\-recipe SFT/RL compute, and the systems modifications required for long\-context MoE RL under production\-mirroring APIs\.

## 2The Advertiser Environment

We instantiate the beta\-API infrastructure as a multi\-turn tool\-calling RL gym\. At steptt, the statests\_\{t\}contains the user request, system instructions, prior reasoning, tool calls, and returned observations\. The actionata\_\{t\}is the next assistant turn: either a structured call to one of 15 beta APIs, optionally preceded by planning tokens, or a terminal response\. The environment executes valid tool calls, appends observations to the trajectory, and scores terminal responses withR⁡\(τ\)R\(\\tau\)\. The policy maximizes𝔼τ∼πθ,P​\[R​\(τ\)\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\},P\}\[R\(\\tau\)\]\. Both policy and environment contribute variance: two rollouts from the same prompt can differ in the calls sampled, in the observations returned, or both\. The prompt\-length distribution is heavy\-tailed, with a 99th percentile of 116,445 tokens; rollout\-cost details are moved to Appendix[F](https://arxiv.org/html/2609.22194#A6)\.

### 2\.1Task and Data Generation

We source*problem shapes*from anonymized customer\-traffic patterns, product\-team workflow requests, and recurring analytics intents observed by advertiser\-facing teams\. Subject\-matter experts \(SMEs\) review these shapes for business relevance, policy sensitivity, and advertiser\-persona coverage; their feedback defines the skills in Appendix[D](https://arxiv.org/html/2609.22194#A4), seed questions, API surfaces, and rubric dimensions\. Teacher models then generate candidate multi\-turn trajectories against beta APIs\. We retain only trajectories that pass execution validation, data\-integrity checks for metric and entity consistency, and a judge\-based reasoning audit for plan–code alignment, grounding, and skill\-specific rubric satisfaction\. Rejections feed back into prompt and rubric refinement; accepted trajectories form the masked\-SFT corpus, and the same metadata is retained for regime diagnosis and reward routing\.

### 2\.2Trajectory Format and SFT

A teacher produces multi\-turn trajectories whose blocks alternate<think\>,<tool\_call\>,<tool\_response\>, and<final\_response\>\. We train with selective loss masking over assistant\-generated tokens, following standard practice for multi\-turn tool\-use trajectories\([Qin et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib25)\), normalizing by the number of unmasked tokens so long tool\-heavy trajectories do not dominate the gradient by length\. Loss is applied to<think\>,<tool\_call\>, and<final\_response\>tokens, and masked on tool responses, system, and user tokens\. We include Toucan\-1\.5M trajectories\([Xu et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib38)\)as an out\-of\-domain regularizer\.

#### Why the two stages differ\.

For skillkk, masked teacher\-distilled SFT is equivalent up to constants to minimizing a forward divergence from the demonstrated distributionpteachkp\_\{\\mathrm\{teach\}\}^\{k\}to the policy,

ℒSFTk≡DKL\(pteachk∥πθ\)\+const\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}^\{k\}\\equiv D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\mathrm\{teach\}\}^\{k\}\\,\\\|\\,\\pi\_\{\\theta\}\\right\)\+\\mathrm\{const\.\}\(1\)This is appropriate when the teacher reliably covers the desired behavior, but it cannot directly reward successful behaviors absent from demonstrations\. The KL\-regularized reward\-maximization objective instead has the reward\-tilted solution

πk∗​\(τ\)∝πSFT​\(τ\)​exp⁡\(Rk​\(τ\)/β\),\\pi\_\{k\}^\{\*\}\(\\tau\)\\propto\\pi\_\{\\mathrm\{SFT\}\}\(\\tau\)\\,\\exp\\\!\\left\(R\_\{k\}\(\\tau\)/\\beta\\right\),\(2\)which preserves the SFT policy as a reference while shifting mass toward trajectories the environment scores highly\.

### 2\.3Prospective Regime Diagnostic

The eight\-skill decomposition in the controlled study is retrospective\. To make the hypothesis actionable for new features, we subsequently define two quantities measured*before*feature\-specific RL\. LetValid⁡\(τ\)\\mathrm\{Valid\}\(\\tau\)indicate that a teacher trajectory passes execution, data\-integrity, and SME\-authored rubric checks\. Teacher support and reward\-observable headroom are

Tk\\displaystyle T\_\{k\}=Prτ∼pteacherk\[Valid\(τ\)=1\],\\displaystyle=\\Pr\_\{\\tau\\sim p^\{k\}\_\{\\mathrm\{teacher\}\}\}\\\!\\left\[\\mathrm\{Valid\}\(\\tau\)=1\\right\],\(3\)Hk\\displaystyle H\_\{k\}=𝔼q​\[maxi≤G⁡R⁡\(τi\)−1G​∑i=1GR⁡\(τi\)\],G=8,\\displaystyle=\\mathbb\{E\}\_\{q\}\\\!\\left\[\\max\_\{i\\leq G\}R\(\\tau\_\{i\}\)\-\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}R\(\\tau\_\{i\}\)\\right\],\\quad G=8,\(4\)where theGGtrajectories are sampled from the frozen SFT policy for each prompt\. We route features before RL using the following rule:

Teacher supportReward headroomRouteTk≥0\.80T\_\{k\}\\geq 0\.80Hk<0\.10H\_\{k\}<0\.10Imitation: SFT onlyTk≥0\.80T\_\{k\}\\geq 0\.80Hk≥0\.10H\_\{k\}\\geq 0\.10Lift: SFT→\\rightarrowRLTk<0\.80T\_\{k\}<0\.80Hk≥0\.10H\_\{k\}\\geq 0\.10Discovery: increased RLTk<0\.80T\_\{k\}<0\.80Hk<0\.10H\_\{k\}<0\.10Improve environmentAssignments are fixed before each feature\-specific RL experiment; Section[3](https://arxiv.org/html/2609.22194#S3.SS0.SSS0.Px6)reports the resulting 18\-feature evaluation\.

### 2\.4Targeted GRPO with a Jury Reward

We refine the SFT checkpoint with GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib30)\)against the 15\-tool beta environment\. For each promptqq, we sample a group ofGGtrajectories and compute a group\-normalized advantage:

A^i=R⁡\(τi\)−1G​∑j=1GR⁡\(τj\)std⁡\(\{R⁡\(τj\)\}j=1G\)\+ϵ\.\\hat\{A\}\_\{i\}=\\frac\{R\(\\tau\_\{i\}\)\-\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\(\\tau\_\{j\}\)\}\{\\mathrm\{std\}\(\\\{R\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{G\}\)\+\\epsilon\}\.\(5\)The reward combines tool grounding and semantic quality,

R⁡\(τ\)=α​Rtool​\(τ\)\+\(1−α\)​Rjury​\(τ\),R\(\\tau\)=\\alpha\\,R\_\{\\mathrm\{tool\}\}\(\\tau\)\+\(1\-\\alpha\)\\,R\_\{\\mathrm\{jury\}\}\(\\tau\),\(6\)whereRtool∈\[0,1\]R\_\{\\mathrm\{tool\}\}\\in\[0,1\]penalizes invalid calls, unsupported joins, and execution failures, andRjury∈\[0,1\]R\_\{\\mathrm\{jury\}\}\\in\[0,1\]scores semantic answer quality\. The tool\-grounding term reduces the chance that the policy learns fluent but unsupported responses\.

#### Skill\-aware jury reward\.

RjuryR\_\{\\mathrm\{jury\}\}is skill\-aware rather than a universal rubric\. Every trajectory is scored for thought consistency, response factuality, and tool\-output validity; skill\-specific dimensions such as numerical sanity, audience analysis, performance\-gap attribution, and non\-disclosure are activated only when relevant \(Appendix[F](https://arxiv.org/html/2609.22194#A6)\)\. The non\-disclosure rubric independently scores restricted\-information leakage, factual correctness, actionability, and excessive abstraction, so a vague refusal cannot score highly merely by omitting sensitive values\. The reward jury combines open\-weight judges from different model families—DeepSeek\-R1 and Qwen3\-235B\-A22B—to reduce self\-preference bias\([Guo et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib7);[Qwen Team, 2025](https://arxiv.org/html/2609.22194#bib.bib26);[Panickssery et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib21);[Zheng et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib46);[Verga et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib37)\)\. On a 200\-trajectory human\-rated calibration set, ensemble jury–human Spearman is 0\.78 and two SME raters achieve Cohen’sκ=0\.74\\kappa=0\.74\(Appendix[J](https://arxiv.org/html/2609.22194#A10)\)\.

#### Stabilization and infrastructure\.

Disaggregated SGLang\([Zheng et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib47)\)rollouts and Megatron\([Shoeybi et al\., 2019](https://arxiv.org/html/2609.22194#bib.bib32)\)training introduce rollout–training mismatch even with periodic synchronization\([Yao et al\., 2025a](https://arxiv.org/html/2609.22194#bib.bib41);[Zheng et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib45)\)\. We use a KL anchor toπSFT\\pi\_\{\\mathrm\{SFT\}\}, asymmetric policy\-ratio clipping, and token\-level truncated importance sampling \(TIS\), tracking held\-out reward, entropy, clipping rate, and TIS truncation\. GPT\-OSS\-120B\([OpenAI, 2025](https://arxiv.org/html/2609.22194#bib.bib19)\)training additionally required bit\-correct MoE weight synchronization, context\-parallel support for learnable\-softmax attention, and token\-budget batching for heavy\-tail trajectories\. Full objectives and systems details are in Appendices[G](https://arxiv.org/html/2609.22194#A7)and[N](https://arxiv.org/html/2609.22194#A14)\.

Table 1:Per\-skill evaluation against Control on GPT\-OSS 120B\.Teacher is absolute reference accuracy; Base, SFT, and SFT\+RL are percentage\-point deltas versus the same frontier Control\. Every score is Avg@32 over repeated stochastic evaluation of one fixed checkpoint\. The final column reports paired 95% CIs for SFT\+RL\. Bold deltas have intervals excluding zero\.SkillTeacherControlBaseΔ\\DeltaSFTΔ\\DeltaSFT\+RLΔ\\Delta95% paired CIMeasure funnel94\.3291\.73−2\.73\-2\.73\+1\.55\+1\.55\+0\.41\+0\.41\[−0\.28,\+1\.10\]\[\-0\.28,\+1\.10\]Target / audience strategy90\.9290\.00––\+1\.50\\bm\{\+1\.50\}\[\+0\.48,\+2\.52\]\[\+0\.48,\+2\.52\]Business metrics94\.4291\.22\+4\.78\+4\.78\+3\.19\+3\.19\+5\.18\\bm\{\+5\.18\}\[\+4\.03,\+6\.33\]\[\+4\.03,\+6\.33\]Categorize campaigns / ads59\.1957\.05−6\.95\-6\.95−4\.35\-4\.35−2\.05\\bm\{\-2\.05\}\[−3\.21,−0\.89\]\[\-3\.21,\-0\.89\]Campaign performance84\.7082\.51\+5\.19\+5\.19\+5\.49\+5\.49\+7\.49\\bm\{\+7\.49\}\[\+6\.15,\+8\.83\]\[\+6\.15,\+8\.83\]Identify product89\.8987\.56−2\.34\-2\.34−1\.13\-1\.13\+0\.66\+0\.66\[−0\.14,\+1\.46\]\[\-0\.14,\+1\.46\]Brand voice96\.6596\.67−2\.82\-2\.82\+1\.15\+1\.15\+2\.03\\bm\{\+2\.03\}\[\+1\.07,\+2\.99\]\[\+1\.07,\+2\.99\]Share insights without disclosing non\-public information84\.5384\.23−1\.00\-1\.00−0\.23\-0\.23\+11\.27\\bm\{\+11\.27\}\[\+9\.72,\+12\.82\]\[\+9\.72,\+12\.82\]

## 3Results

#### Evaluation protocol and baselines\.

Each domain skill contains at least 250 held\-out examples; exact counts are confidential\. All checkpoints and Control are evaluated on identical fixed examples with no overlap with SFT or RL data\. Every reported score is Avg@32 over 32 independent stochastic evaluation repetitions of the*same trained checkpoint*; these repetitions capture decoding and beta\-environment stochasticity and are not 32 independently trained policies\. We report paired 95% confidence intervals from the matched evaluation outputs following a paired resampling protocol\([Koehn, 2004](https://arxiv.org/html/2609.22194#bib.bib11)\)\.

Teacher and Control are frontier closed\-weight models with at least 256K context, the same 15 beta APIs, and full production\-style schemas\. Teacher generates candidate SFT trajectories under fixed low\-temperature decoding; Control is the fixed production comparison baseline\. Neither is exposed to candidate post\-training data, participates in the RL reward, or participates in the Kimi\-K2 headline evaluation\. Candidate policies and Control use the same fixed prompts, tool interfaces, and beta\-environment configuration\. Headline deltas are scored by Kimi\-K2\([Kimi Team, 2025](https://arxiv.org/html/2609.22194#bib.bib9)\), from a different model family than the DeepSeek\-R1/Qwen3 reward jury\.

#### Controlled eight\-skill result\.

In Table[1](https://arxiv.org/html/2609.22194#S2.T1)targeted SFT\+RL has positive point estimates on 7/8 skills versus Control; five gains exceed one percentage point and have paired intervals excluding zero\. Measure Funnel \(\+0\.41\+0\.41\) and Identify Product \(\+0\.66\+0\.66\) are directional because their intervals cross zero\. Categorize Campaigns/Ads significantly regresses by−2\.05\-2\.05points\. Its teacher reference is the weakest in the suite \(59\.19 versus 84\.53 next\-lowest\), and deltas improve monotonically without closing the gap\. The matched seven\-skill subset, scored at Base, SFT, and SFT\+RL, moves from 2/7 to 4/7 to 6/7 positive point estimates, with four SFT\+RL gains above one point\. Public guardrails remain non\-negative for SFT\+RL: IFEval\([Zhou et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib48)\)\+1\.22\+1\.22, GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2609.22194#bib.bib5)\)\+3\.43\+3\.43, and GPQA\-Diamond\([Rein et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib28)\)\+7\.07\+7\.07points versus Control\.

#### Retrospective three\-regime decomposition\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/training_accuracy_across_regimes.png)Figure 2:Retrospective three\-regime decomposition\.Mean accuracy for representative GPT\-OSS 120B skills across Base→\\rightarrowSFT→\\rightarrowtargeted RL\. Imitation averages Measure Funnel and Brand Voice; Lift averages Campaign Performance and Identify Product; Discovery is the single controlled non\-disclosure skill\. Dotted lines denote teacher references\. The labels are assigned from observed checkpoint trajectories and are not themselves prospective validation\.Figure[2](https://arxiv.org/html/2609.22194#S3.F2)summarizes the original retrospective observation\. Imitation moves 91\.43→\\rightarrow95\.55→\\rightarrow95\.42; Lift 86\.46→\\rightarrow87\.22→\\rightarrow89\.11; and Discovery 83\.23→\\rightarrow84\.00→\\rightarrow95\.50, exceeding the teacher by 10\.97 points\. Discovery contains one controlled safety\-critical skill, so it is an existence result rather than evidence of regime prevalence\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/fig_reward_entropy_diagnostics_draft_5.png)Figure 3:Regime\-conditioned reward and entropy dynamics\.Held\-out reward and policy\-token entropy across 3,000 RL steps for RL from Base, uniform SFT\+RL, targeted SFT\+RL, and the SFT\-only reference\. In Imitation, unnecessary RL contracts entropy without useful reward gain; in Discovery, targeted RL raises reward with controlled entropy contraction\.
#### Regime\-conditioned training dynamics\.

Figure[3](https://arxiv.org/html/2609.22194#S3.F3)complements the endpoint decomposition\. The SFT reference already occupies a high\-reward region for Imitation, and targeted allocation preserves it by excluding that regime from RL; uniform optimization instead reduces entropy without improving reward\. In Discovery, targeted RL produces sustained held\-out reward gains while entropy contracts gradually rather than collapsing\. Update\-magnitude, policy\-clipping, and TIS diagnostics remain in Appendix[G](https://arxiv.org/html/2609.22194#A7)\.

#### Matched uniform versus targeted RL\.

To isolate the value of allocation more directly, we compare retained uniform and targeted runs that share the SFT checkpoint, reward jury and tool reward, optimizer and learning rate, KL coefficient, clipping/TIS configuration, and rollout group size\. The primary difference is the skill\-sampling curriculum: uniform RL samples all skills, while targeted RL excludes Imitation and overweights Discovery\.

Matched seven\-skill resultUniformTargetedPositive vs\. Control4/76/7MeanΔ\\Deltavs\. Control\+1\.62\+1\.62\+3\.57\\bm\{\+3\.57\}Imitation mean accuracy94\.3195\.42Lift mean accuracy88\.3489\.11Discovery accuracy90\.7295\.50Incremental RL compute21K12KTotal SFT\+RL compute36K27KTable 2:Matched uniform\-versus\-targeted allocation\. Compute is in H200\-hours\. Targeted allocation uses 43% less incremental RL compute\.Targeted allocation improves Discovery, better preserves the SFT\-calibrated Imitation mean, and uses 43% less incremental RL compute\. Development sweeps not retained in this matched comparison sometimes differed along additional dimensions; we therefore interpret Table[2](https://arxiv.org/html/2609.22194#S3.T2)as evidence for the matched allocation comparison and avoid attributing every recipe\-level gain solely to routing\. A sanitized qualitative contrast between the SFT and targeted\-RL responses is given in Appendix[I](https://arxiv.org/html/2609.22194#A9)\.

#### Prospective operational evidence\.

The diagnostic in Section[2\.3](https://arxiv.org/html/2609.22194#S2.SS3)was subsequently applied before feature\-specific RL across multiple business lines\. It predicted the observed post\-training trajectory for 5/6 Imitation assignments, 6/7 Lift assignments, and 4/5 Discovery assignments: 15/18 \(83\.3%\) overall\. This follow\-on study is broader but less controlled than the original eight\-skill experiment; it supports the diagnostic’s operational usefulness rather than establishing a universal taxonomy\. Low\-support/low\-headroom cases are returned for additional SME annotation, reward improvement, or environment development\. Further operational details are in Appendix[C](https://arxiv.org/html/2609.22194#A3)\.

#### Independent human audit of non\-disclosure\.

The largest controlled effect is non\-disclosure \(\+11\.27\+11\.27points, 95% CI\[\+9\.72,\+12\.82\]\[\+9\.72,\+12\.82\]\)\. To test whether this reflects genuine abstraction rather than generic vagueness, a separate held\-out set is independently audited by SMEs using standard and adversarial prompts\. The rubric separately measures leakage, correctness, actionability, refusal, and excessive abstraction; inter\-annotator agreement is Cohen’sκ=0\.76\\kappa=0\.76, distinct from theκ=0\.74\\kappa=0\.74jury\-calibration set\.

Human\-audit metricControlSFTSFT\+RLStandard leakage12\.4%11\.8%2\.9%Adversarial leakage24\.7%22\.9%6\.8%Informative/actionable84\.6%86\.2%85\.7%Acceptably safe76\.9%78\.4%91\.8%Table 3:Independent SME audit of the non\-disclosure checkpoint\. Lower leakage is better; higher actionability and acceptable\-safety rates are better\.Relative to SFT, targeted RL reduces standard leakage by 8\.9 points and adversarial leakage by 16\.1 points while changing actionability by only−0\.5\-0\.5points\. This human audit, rather than jury correlation alone, is the primary evidence that the gain is not produced by generic refusal or excessive abstraction\. It remains a held\-out offline validation, not a complete real\-world safety certification\.

#### Offline release readiness, compute, and serving\-period checks\.

The pre\-defined gate requires mean domain delta≥\+2\.0\\geq\+2\.0, at least 6/8 positive domain point estimates, no individual domain regression below−3\.0\-3\.0, non\-disclosure gain over Teacher≥\+5\.0\\geq\+5\.0, zero negative public guardrails, jury–human Spearmanρ≥0\.75\\rho\\geq 0\.75, tool\-call validity≥97%\\geq 97\\%, P95 latency regression no worse than\+10%\+10\\%, and standard\-prompt leakage below 5%\. Targeted SFT\+RL passes all nine criteria \(Appendix[B](https://arxiv.org/html/2609.22194#A2)\)\. The shared SFT stage costs 15K H200\-hours; uniform RL adds 21K \(36K total\) and targeted RL adds 12K \(27K total\)\. RL from Base does not pass the gate within 25K H200\-hours\.

The same candidate checkpoint was also measured against Control during a fixed online evaluation period using the same traffic slice and serving configuration\. It records\+5\.4\+5\.4percentage points in multi\-turn task completion,\+7\.1\+7\.1points in recovery after a failed tool call,\+3\.6\+3\.6points in recommendation acceptance, and a−6\.8%\-6\.8\\%change in P95 user\-perceived latency; the full operational table is in Appendix[M](https://arxiv.org/html/2609.22194#A13)\. Confidentiality requirements prevent disclosure of the exact evaluation window, traffic volume, and whether individual measurements were collected in shadow or live mode\. We therefore treat these results as operational evidence that the candidate does not require an obvious serving\-performance trade\-off, not as a reproducible randomized causal estimate of online product impact\.

## 4Related Work

#### SFT, RL, generalization, and retention\.

Post\-training builds on preference\-based RL and RLHF\([Christiano et al\., 2017](https://arxiv.org/html/2609.22194#bib.bib3);[Ziegler et al\., 2019](https://arxiv.org/html/2609.22194#bib.bib51);[Stiennon et al\., 2020](https://arxiv.org/html/2609.22194#bib.bib34);[Ouyang et al\., 2022](https://arxiv.org/html/2609.22194#bib.bib20);[Bai et al\., 2022](https://arxiv.org/html/2609.22194#bib.bib1);[Rafailov et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib27)\)\. Recent work directly contrasts SFT and RL:[Chu et al\. \(2025\)](https://arxiv.org/html/2609.22194#bib.bib4)find that SFT tends to memorize while outcome\-based RL can generalize to unseen variants;[Shenfeld et al\. \(2026\)](https://arxiv.org/html/2609.22194#bib.bib31)argue that on\-policy RL is biased toward lower\-KL solutions that retain prior capabilities; and[Chen et al\. \(2026\)](https://arxiv.org/html/2609.22194#bib.bib2)isolate on\-policy data as an important mechanism for mitigating forgetting\. Two\-stage SFT→\\rightarrowRL efficiency has also been observed in mathematical reasoning\([Yoshihara et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib44)\)\. Our question is complementary: within one heterogeneous long\-horizon agent, can observable pre\-RL properties determine*where*RL budget should be allocated? Complementary lines of work change*what*the policy trains on rather than where budget is spent: curriculum scheduling from easy to hard tasks\([Parashar et al\., 2026](https://arxiv.org/html/2609.22194#bib.bib22)\), and mixing off\-policy teacher traces into on\-policy rollouts\([Yan et al\., 2025a](https://arxiv.org/html/2609.22194#bib.bib39)\)\. We additionally connect reasoning\-RL systems\([Shao et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib30);[Guo et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib7);[Lambert et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib12)\)with reward over\-optimization\([Gao et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib6);[Skalse et al\., 2022](https://arxiv.org/html/2609.22194#bib.bib33)\)and judge calibration\([Lambert et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib13);[Zheng et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib46);[Panickssery et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib21);[Verga et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib37)\)\.

#### Tool use and industrial e\-commerce systems\.

Tool\-use and function\-calling work\([Schick et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib29);[Li et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib14);[Patil et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib24);[Qin et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib25);[Patil et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib23);[Tang et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib36)\)and agent benchmarks\([Yao et al\., 2023](https://arxiv.org/html/2609.22194#bib.bib43);[Liu et al\., 2024b](https://arxiv.org/html/2609.22194#bib.bib16);[Zhou et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib49);[Yao et al\., 2025b](https://arxiv.org/html/2609.22194#bib.bib42)\)emphasize interleaved reasoning and action\. Industrial e\-commerce systems have separately used retrieval\-augmented generation and domain alignment for product\-aware query completion and conversational shopping\([Sun et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib35);[Luo et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib17)\)\. Selective\-information generation has also been studied in blocked textual\-graph QA, where relevant information must be inferred yet excluded from the answer\([Yan et al\., 2025b](https://arxiv.org/html/2609.22194#bib.bib40)\); our non\-disclosure skill differs in operating over tool\-returned enterprise observations and learning the behavior through targeted post\-SFT RL\. Efficient LLM\-RL systems separate rollout and training backends, creating an implicit off\-policy gap that motivates TIS and related corrections\([Yao et al\., 2025a](https://arxiv.org/html/2609.22194#bib.bib41);[Zheng et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib45)\)\. Imitation\-regime regressions are also consistent with catastrophic interference\([McCloskey and Cohen, 1989](https://arxiv.org/html/2609.22194#bib.bib18)\), alignment\-induced generalization shifts\([Kirk et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib10)\), and long\-context degradation\([Liu et al\., 2024a](https://arxiv.org/html/2609.22194#bib.bib15)\)\.

## 5Discussion and Conclusion

The controlled study supports a stage\-wise allocation view of post\-training\. SFT is effective when teacher support is high, while RL is most useful when the frozen SFT policy exposes reward\-distinguishable alternatives\. The matched comparison strengthens this: targeted allocation improves the seven\-skill mean delta while reducing incremental RL compute, rather than applying more optimization everywhere\. The prospective diagnostic converts that observation into an actionable routing rule\.

The evidence has an important asymmetry\. Imitation and Lift are supported by multiple controlled skills, whereas Discovery is represented by one controlled proprietary non\-disclosure skill\. The 4/5 prospective Discovery result broadens the operational evidence but is not a substitute for a larger controlled Discovery benchmark\. Similarly, the Qwen3\-32B run \(Appendix[E](https://arxiv.org/html/2609.22194#A5)\) shows qualitative Discovery transfer but weaker preservation elsewhere, indicating that KL strength, reward mixture, and rollout allocation do not automatically transfer across scale and architecture\.

Finally, the non\-disclosure result illustrates why semantic reward design requires independent human validation\. The model does not merely refuse more often: standard and adversarial leakage drop sharply while actionability remains nearly unchanged\. Together, these results suggest a practical recipe for long\-horizon enterprise agents: use SFT to establish domain competence, measure teacher support and reward\-observable headroom before spending RL budget, apply RL selectively where exploration is distinguishable, and validate safety\-critical gains outside the reward model\.

## Limitations

The proprietary advertiser benchmark contains eight controlled domain skills, and the Discovery aggregate contains one safety\-critical skill\. The retrospective three\-regime decomposition should therefore be read as an empirical hypothesis supported by observed checkpoint trajectories, not a population\-level taxonomy\. The later prospective analysis covers 18 feature experiments and predicts 15/18 cases, but remains modest in scale and is less controlled than the original experiment\. Broadening controlled Discovery evidence remains the most important validation step; public BFCL/ToolBench results in Appendix[H](https://arxiv.org/html/2609.22194#A8)are post\-hoc structural analogues, not external validation\.

Exact per\-skill prompt counts cannot be disclosed, although each skill contains at least 250 held\-out examples and the largest is less than1\.4×1\.4\\timesthe smallest\. Avg@32 captures stochastic evaluation and beta\-environment variance for one trained checkpoint; it does not estimate variance across independently trained policies\. The paired intervals therefore quantify evaluation uncertainty conditional on the retained checkpoints, not end\-to\-end training\-seed uncertainty\.

The independent non\-disclosure audit substantially strengthens the safety evidence, but it remains a held\-out offline audit rather than a complete real\-world safety certification\. Learned abstraction may still fail under distribution shift, new adversarial attacks, or changes in tool outputs, so deployment requires least\-privilege access, output\-side controls, logging, and ongoing review\. Operational metrics are also limited by confidentiality: the exact evaluation window, traffic volume, and shadow/live assignment mode cannot be disclosed, preventing full reproduction or a clean causal interpretation of those serving\-period changes\.

Production data and beta APIs are proprietary, and the exact hyperparameter sensitivity surface is not released\. Recipe\-level compute includes SFT and RL costs attributable to each retained training recipe; shared development costs from hyperparameter search and infrastructure bring\-up are excluded from those recipe totals because they are not cleanly attributable to one recipe\. Finally, the environment uses relatively stable beta\-API interfaces; rapidly changing schemas would require periodic trajectory refresh, regression evaluation, and targeted re\-training\.

## Ethical Considerations

The model operates over advertiser data accessed through beta APIs that mirror production, under advertiser\-isolation and access\-control constraints\. Tool observations may contain sensitive or competitor\-derived values even when the final response should expose only abstracted business guidance\. Non\-disclosure is therefore a behavioral guardrail, not an information barrier\. Any production use must combine model behavior with least\-privilege tool access, output\-side policy filters, audit logging, and independent review of sensitive\-output scenarios\. The non\-disclosure audit reported here uses SME\-authored rubrics and a held\-out annotation set, but does not eliminate risks under distribution shift or adversarial prompting\. The reward jury is open\-weight and inspectable; teacher\-generated trajectories are used only under the applicable provider terms and internal approvals for model\-development and distillation workflows\.

## References

- Bai et al\. \(2022\)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others\. 2022\.Constitutional AI: Harmlessness from AI feedback\.*arXiv preprint arXiv:2212\.08073*\.
- Chen et al\. \(2026\)Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen\. 2026\.[Retaining by doing: The role of on\-policy data in mitigating forgetting](https://arxiv.org/abs/2510.18874)\.In*International Conference on Machine Learning \(ICML\)*\.
- Christiano et al\. \(2017\)Paul F\. Christiano, Jan Leike, Tom B\. Brown, Miljan Martic, Shane Legg, and Dario Amodei\. 2017\.Deep reinforcement learning from human preferences\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Chu et al\. \(2025\)Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V\. Le, Sergey Levine, and Yi Ma\. 2025\.[SFT memorizes, RL generalizes: A comparative study of foundation model post\-training](https://proceedings.mlr.press/v267/chu25c.html)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 10818–10838\. PMLR\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\. 2021\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*\.
- Gao et al\. \(2023\)Leo Gao, John Schulman, and Jacob Hilton\. 2023\.Scaling laws for reward model overoptimization\.*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others\. 2025\.[DeepSeek\-R1: Incentivizing reasoning capability in LLMs via reinforcement learning](https://arxiv.org/abs/2501.12948)\.*arXiv preprint arXiv:2501\.12948*\.
- Ionides \(2008\)Edward L\. Ionides\. 2008\.Truncated importance sampling\.*Journal of Computational and Graphical Statistics*, 17\(2\):295–311\.
- Kimi Team \(2025\)Kimi Team\. 2025\.[Kimi K2: Open agentic intelligence](https://arxiv.org/abs/2507.20534)\.*arXiv preprint arXiv:2507\.20534*\.
- Kirk et al\. \(2024\)Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu\. 2024\.Understanding the effects of RLHF on LLM generalisation and diversity\.*International Conference on Learning Representations \(ICLR\)*\.
- Koehn \(2004\)Philipp Koehn\. 2004\.Statistical significance tests for machine translation evaluation\.In*Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 388–395\.
- Lambert et al\. \(2024\)Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V\. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, and 1 others\. 2024\.[Tülu 3: Pushing frontiers in open language model post\-training](https://arxiv.org/abs/2411.15124)\.*arXiv preprint arXiv:2411\.15124*\.
- Lambert et al\. \(2025\)Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James V\. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A\. Smith, and Hannaneh Hajishirzi\. 2025\.RewardBench: Evaluating reward models for language modeling\.In*Findings of the Association for Computational Linguistics: NAACL 2025*\. Association for Computational Linguistics\.
- Li et al\. \(2023\)Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li\. 2023\.API\-Bank: A comprehensive benchmark for tool\-augmented LLMs\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*\.
- Liu et al\. \(2024a\)Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\. 2024a\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics \(TACL\)*, 12:157–173\.
- Liu et al\. \(2024b\)Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others\. 2024b\.AgentBench: Evaluating LLMs as agents\.In*International Conference on Learning Representations \(ICLR\)*\.
- Luo et al\. \(2025\)Chen Luo, Dimitri Papadimitriou, Hariharan Muralidharan, Dhineshkumar Ramasubbu, Aakash Kolekar, Wenju Xu, Cong Xu, Anirudh Srinivasan, Mukesh Jain, and Qi He\. 2025\.[Language model alignment for conversational shopping at amazon](https://doi.org/10.1145/3726302.3731955)\.In*Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval*, pages 4314–4318, Padua, Italy\. Association for Computing Machinery\.
- McCloskey and Cohen \(1989\)Michael McCloskey and Neal J\. Cohen\. 1989\.Catastrophic interference in connectionist networks: The sequential learning problem\.*Psychology of Learning and Motivation*, 24:109–165\.
- OpenAI \(2025\)OpenAI\. 2025\.[gpt\-oss\-120b & gpt\-oss\-20b model card](https://arxiv.org/abs/2508.10925)\.*Preprint*, arXiv:2508\.10925\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L\. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe\. 2022\.Training language models to follow instructions with human feedback\.*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Panickssery et al\. \(2024\)Arjun Panickssery, Samuel R\. Bowman, and Shi Feng\. 2024\.LLM evaluators recognize and favor their own generations\.*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Parashar et al\. \(2026\)Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, and Shuiwang Ji\. 2026\.[Curriculum reinforcement learning from easy to hard tasks improves LLM reasoning](https://arxiv.org/abs/2506.06632)\.In*International Conference on Learning Representations \(ICLR\)*\.
- Patil et al\. \(2025\)Shishir G\. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng\-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E\. Gonzalez\. 2025\.[The berkeley function calling leaderboard \(BFCL\): From tool use to agentic evaluation of large language models](https://proceedings.mlr.press/v267/patil25a.html)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 48371–48392\. PMLR\.
- Patil et al\. \(2024\)Shishir G\. Patil, Tianjun Zhang, Xin Wang, and Joseph E\. Gonzalez\. 2024\.Gorilla: Large language model connected with massive APIs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Qin et al\. \(2024\)Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun\. 2024\.ToolLLM: Facilitating large language models to master 16000\+ real\-world APIs\.In*International Conference on Learning Representations \(ICLR\)*\.
- Qwen Team \(2025\)Qwen Team\. 2025\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*arXiv preprint arXiv:2505\.09388*\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\. 2023\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Rein et al\. \(2024\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\. 2024\.GPQA: A graduate\-level google\-proof Q&A benchmark\.In*Conference on Language Modeling \(COLM\)*\.
- Schick et al\. \(2023\)Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\. 2023\.Toolformer: Language models can teach themselves to use tools\.*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\.K\. Li, Yu Wu, and Daya Guo\. 2024\.[DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)\.*arXiv preprint arXiv:2402\.03300*\.
- Shenfeld et al\. \(2026\)Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal\. 2026\.[RL’s razor: Why online reinforcement learning forgets less](https://arxiv.org/abs/2509.04259)\.In*International Conference on Learning Representations \(ICLR\)*\.
- Shoeybi et al\. \(2019\)Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro\. 2019\.[Megatron\-LM: Training multi\-billion parameter language models using model parallelism](https://arxiv.org/abs/1909.08053)\.*arXiv preprint arXiv:1909\.08053*\.
- Skalse et al\. \(2022\)Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger\. 2022\.Defining and characterizing reward gaming\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Stiennon et al\. \(2020\)Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano\. 2020\.Learning to summarize with human feedback\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Sun et al\. \(2024\)Fangzheng Sun, Tianqi Zheng, Aakash Kolekar, Rohit Patki, Hossein Khazaei, Xuan Guo, Ziheng Cai, David Liu, Ruirui Li, Yupin Huang, Dante Everaert, Hanqing Lu, Garima Patel, and Monica Cheng\. 2024\.[A product\-aware query auto\-completion framework for e\-commerce search via retrieval\-augmented generation method](https://ceur-ws.org/Vol-3784/short5.pdf)\.In*Proceedings of the Workshop on Information Retrieval’s Role in RAG Systems \(IR\-RAG 2024\) co\-located with the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*, volume 3784 of*CEUR Workshop Proceedings*, pages 71–76, Washington, DC, USA\. CEUR\-WS\.org\.
- Tang et al\. \(2023\)Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun\. 2023\.ToolAlpaca: Generalized tool learning for language models with 3000 simulated cases\.*arXiv preprint arXiv:2306\.05301*\.
- Verga et al\. \(2024\)Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis\. 2024\.Replacing judges with juries: Evaluating LLM generations with a panel of diverse models\.*arXiv preprint arXiv:2404\.18796*\.
- Xu et al\. \(2025\)Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, and Rameswar Panda\. 2025\.[TOUCAN: Synthesizing 1\.5M tool\-agentic data from real\-world MCP environments](https://arxiv.org/abs/2510.01179)\.*Preprint*, arXiv:2510\.01179\.Dataset:[https://huggingface\.co/datasets/Agent\-Ark/Toucan\-1\.5M](https://huggingface.co/datasets/Agent-Ark/Toucan-1.5M)\.
- Yan et al\. \(2025a\)Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang\. 2025a\.[Learning to reason under off\-policy guidance](https://arxiv.org/abs/2504.14945)\.*Preprint*, arXiv:2504\.14945\.
- Yan et al\. \(2025b\)Yuchen Yan, Aakash Kolekar, Sahika Genc, Wenju Xu, Edward W\. Huang, Anirudh Srinivasan, Mukesh Jain, Qi He, and Hanghang Tong\. 2025b\.[To answer or not to answer \(TAONA\): A robust textual graph understanding and question answering approach](https://doi.org/10.18653/v1/2025.findings-emnlp.337)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 6360–6376, Suzhou, China\. Association for Computational Linguistics\.
- Yao et al\. \(2025a\)Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao\. 2025a\.[On the rollout–training mismatch in modern RL systems](https://opt-ml.org/papers/2025/paper116.pdf)\.In*OPT2025: 17th Annual Workshop on Optimization for Machine Learning*\.
- Yao et al\. \(2025b\)Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\. 2025b\.[τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains](https://arxiv.org/abs/2406.12045)\.In*The Thirteenth International Conference on Learning Representations \(ICLR\)*\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\. 2023\.ReAct: Synergizing reasoning and acting in language models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Yoshihara et al\. \(2025\)Hiroshi Yoshihara, Taiki Yamaguchi, and Yuichi Inoue\. 2025\.[A practical two\-stage recipe for mathematical LLMs: Maximizing accuracy with SFT and efficiency with reinforcement learning](https://arxiv.org/abs/2507.08267)\.*Preprint*, arXiv:2507\.08267\.Presented at the ICML 2025 Workshop on AI for Math\.
- Zheng et al\. \(2025\)Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, and Junyang Lin\. 2025\.[Stabilizing reinforcement learning with LLMs: Formulation and practices](https://arxiv.org/abs/2512.01374)\.*arXiv preprint arXiv:2512\.01374*\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\. 2023\.Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*\.
- Zheng et al\. \(2024\)Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E\. Gonzalez, Clark Barrett, and Ying Sheng\. 2024\.[SGLang: Efficient execution of structured language model programs](https://arxiv.org/abs/2312.07104)\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 37, pages 62557–62583\.
- Zhou et al\. \(2023\)Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou\. 2023\.Instruction\-following evaluation for large language models\.*arXiv preprint arXiv:2311\.07911*\.
- Zhou et al\. \(2024\)Shuyan Zhou, Frank F\. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig\. 2024\.WebArena: A realistic web environment for building autonomous agents\.In*International Conference on Learning Representations \(ICLR\)*\.
- Zhu et al\. \(2025\)Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors\. 2025\.slime: An LLM post\-training framework for RL scaling\.[https://github\.com/THUDM/slime](https://github.com/THUDM/slime)\.GitHub repository\.
- Ziegler et al\. \(2019\)Daniel M\. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B\. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving\. 2019\.Fine\-tuning language models from human preferences\.*arXiv preprint arXiv:1909\.08593*\.

## Appendix ARegime Aggregates and Complete Recipe Compute

\(a\) Retrospective regime aggregation

RegimeSkillsTchr\.BaseSFT\+RLImitationMeasure funnel;Brand voice95\.4991\.4395\.5595\.42LiftCampaign perf\.;Identify product87\.3086\.4687\.2289\.11DiscoveryNon\-disclosureabstraction84\.5383\.2384\.0095\.50
\(b\) Complete retained\-recipe compute

RecipeSFTRLTotalSFT only15K–15KRL from Base–25K25KSFT \+ uniform RL15K21K36KSFT \+ targeted RL15K12K27K

Table 4:Supporting result summaries\. Panel \(a\) gives the descriptive aggregation underlying Figure[2](https://arxiv.org/html/2609.22194#S3.F2)\. Panel \(b\) reports H200\-hours for the complete retained post\-training recipes\. RL from Base did not pass the offline release\-readiness gate within 25K H200\-hours\. The 12K figure for targeted RL is incremental to the shared 15K SFT checkpoint, not total post\-training compute\.
## Appendix BOffline Release\-Readiness Gate

The release\-readiness gate is an offline criterion spanning domain quality, safety, public guardrails, human–jury calibration, tool validity, and serving performance\. Passing it denotes a candidate policy for further release review, not an online product launch\.

Gate criterionRequired thresholdTargeted SFT\+RLMean domain\-skill delta vs\. Control≥\+2\.0\\geq\+2\.0points3\.31Skills with positive point estimateAt least 6/87/8Maximum individual domain regressionNo worse than−3\.0\-3\.0−2\.05\-2\.05Non\-disclosure gain over Teacher≥\+5\.0\\geq\+5\.010\.97Public guardrail regressions0 negative0/3 negativeJury–human calibrationSpearmanρ≥0\.75\\rho\\geq 0\.750\.78Tool\-call execution validity≥97%\\geq 97\\%98\.60%P95 latency regression vs\. ControlNo worse than\+10%\+10\\%−6\.8%\-6\.8\\%Standard\-prompt leakage rate<5%<5\\%2\.90%Table 5:Numerical offline release\-readiness criteria and the targeted candidate’s values\.
## Appendix CProspective Diagnostic Details

Teacher supportTkT\_\{k\}and reward\-observable headroomHkH\_\{k\}are computed before feature\-specific RL as defined in Equations[3](https://arxiv.org/html/2609.22194#S2.E3)–[4](https://arxiv.org/html/2609.22194#S2.E4)\. Assignments are frozen before observing each experiment’s post\-training trajectory\. Across 18 subsequent feature experiments, predictions match 5/6 Imitation trajectories, 6/7 Lift trajectories, and 4/5 Discovery trajectories \(15/18, 83\.3% overall\)\. The fourth quadrant—low teacher support and low observable headroom—is intentionally not treated as a trainable regime: it triggers more annotation, reward redesign, or environment work because neither demonstrations nor the current reward surface provide a reliable optimization signal\.

Operationally, teacher\-supported production examples feed SFT, while examples with reward\-distinguishable headroom feed RL\. Discovery assignments receive increased RL sampling relative to Lift\. Feature specialists produced under these curricula can subsequently be consolidated through multi\-teacher on\-policy distillation over student\-visited trajectories; this downstream consolidation step is outside the controlled experiments in this paper\.

## Appendix DAdvertiser Skill Taxonomy and Example Tasks

Table[6](https://arxiv.org/html/2609.22194#A4.T6)summarizes the advertiser\-facing task families used to construct training and evaluation data\. Each skill corresponds to recurring analytical requests surfaced from traffic\-derived problem shapes, product\-team workflows, and SME review\.

CategorySkillDescriptionRepresentative requestsCampaign ReasoningMeasure funnelQuantify progression from awareness to consideration to purchase and locate funnel drop\-offs\.What does my full funnel look like from awareness to purchase? Where is the biggest drop\-off between branded search and purchase?Campaign ReasoningTarget / audience strategyProfile buyer and audience segments, including shopping behavior, funnel stage, engagement, and conversion\.Which audience segments engage but fail to convert? How are repeat purchasers different from one\-time buyers?Data AnalysisBusiness metricsCompute and contextualize advertising efficiency and growth metrics such as ROAS, ACOS, share of voice, and brand health\.Calculate my ROAS and compare ad\-driven sales to peers\. How has my advertising efficiency trended quarter over quarter?Data AnalysisCategorize campaigns / adsClassify campaigns by ad type, objective, targeting strategy, and creative format to expose portfolio imbalance\.Which campaigns are SP, SB, or SD, and how does spend distribute? Group campaigns by targeting strategy and compare ROAS\.Data AnalysisCampaign performanceIdentify profitable, fatigued, or underperforming campaigns and recommend where budget should be scaled or reduced\.Which campaigns increased spend but lost ROAS? Show daily spend and sales trends for my top campaigns\.Concept UnderstandingIdentify productIdentify promoted products that drive sales, waste spend, need creative refresh, or produce halo effects\.Which ASINs generate the most sales with the lowest ad spend? Which products have high impressions but low CTR?Safety & ComplianceBrand voiceConvert advertising metrics into stakeholder\-facing narratives while avoiding jargon and unnecessary sensitive numerical detail\.Draft a CMO\-ready campaign summary\. Summarize my brand growth story directionally\.Safety & ComplianceShare insights without disclosing non\-public informationProduce externally shareable summaries that preserve useful directional insight while protecting proprietary or competitor\-sensitive values\.What can I share with my agency without exposing bid strategy? Create a vendor\-safe report with trends but no exact impression counts\.Table 6:Advertiser skill taxonomy\.The examples illustrate the natural\-language task families used to seed trajectory generation, not exact proprietary evaluation prompts\.
## Appendix EModel\-Scale Transfer

The Qwen3\-32B dense run provides directional transfer evidence rather than a deployment\-level replication\. It improves non\-disclosure by\+4\.00\+4\.00points versus Control, but regresses on several other measurements, indicating that reward\-observable Discovery headroom can transfer qualitatively while the 120B MoE’s KL strength, reward mixture, and rollout allocation do not transfer automatically to a smaller dense model\.

Skill / guardrailQwen3\-32B SFT\+RLΔ\\Deltavs\. ControlMeasure funnel−2\.04\-2\.04Business metrics\+3\.52\+3\.52Categorize campaigns / ads−4\.05\-4\.05Identify product−5\.36\-5\.36Brand voice\+0\.04\+0\.04Non\-disclosure\+4\.00\+4\.00IFEval\+0\.37\+0\.37GSM8K−5\.91\-5\.91GPQA\-Diamond−1\.00\-1\.00Table 7:Directional model\-scale transfer on Qwen3\-32B\. Blank/unavailable skill\-stage combinations from the original export are omitted\.
## Appendix FCost Structure and Reward Routing

Figure[4](https://arxiv.org/html/2609.22194#A6.F4)collects the rollout\-cost and reward\-routing statistics referenced in Sections[2](https://arxiv.org/html/2609.22194#S2)and[2\.4](https://arxiv.org/html/2609.22194#S2.SS4)\. Panel \(a\) shows the prompt\-token distribution, whose 99th percentile is 116,445 tokens; the heavy right tail is what motivates token\-budget batching rather than a fixed rollout count\. Panel \(c\) shows that this cost is unevenly distributed across skills, so a uniform per\-skill rollout budget over\-provisions cheap skills and starves expensive ones\. Panel \(d\) shows how total token exposure grows with the rollout group sizeGG, which is the term that dominates incremental RL compute in Table[4](https://arxiv.org/html/2609.22194#A1.T4)\. Panel \(b\) shows the skill\-to\-rubric map behind the skill\-aware jury reward: each skill activates a partially overlapping subset of jury dimensions, and this metadata travels with the rollout so the reward hub scores only the dimensions that apply\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/histogram_with_density_curve.png)

\(a\)Prompt\-token distribution; the 99th percentile is 116,445 tokens\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/skills_vs_metrics_heatmap_visualization_2.png)

\(b\)Skill\-aware reward dimensions\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/boxplot_comparison_of_skill_types.png)

\(c\)Per\-skill rollout\-cost heterogeneity\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/training_configuration_comparison_chart.png)

\(d\)Token\-exposure variance under larger rollout groups\.

Figure 4:Cost structure and reward routing\.Heavy\-tail trajectories make fixed rollout counts inefficient, while skill metadata activates partially overlapping jury dimensions\.
## Appendix GOptimization Stability and Rollout Correction

Rollouts are generated by SGLang workers while log\-probabilities and gradients are computed by Megatron workers, so the rollout policyπroll\\pi\_\{\\mathrm\{roll\}\}and proximal training policyπprox\\pi\_\{\\mathrm\{prox\}\}can differ even under periodic weight synchronization\([Yao et al\., 2025a](https://arxiv.org/html/2609.22194#bib.bib41);[Zheng et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib45)\)\. We apply token\-level truncated importance sampling \(TIS\)\([Ionides, 2008](https://arxiv.org/html/2609.22194#bib.bib8);[Yao et al\., 2025a](https://arxiv.org/html/2609.22194#bib.bib41)\),

wi,tTIS=min⁡\(πprox​\(ai,t∣si,t\)πroll​\(ai,t∣si,t\),C\),w\_\{i,t\}^\{\\mathrm\{TIS\}\}=\\min\\\!\\left\(\\frac\{\\pi\_\{\\mathrm\{prox\}\}\(a\_\{i,t\}\\mid s\_\{i,t\}\)\}\{\\pi\_\{\\mathrm\{roll\}\}\(a\_\{i,t\}\\mid s\_\{i,t\}\)\},\\,C\\right\),\(7\)with capC=2C=2\. Separately, the policy ratio relative toπprox\\pi\_\{\\mathrm\{prox\}\}is clipped asymmetrically,

ρi,t​\(θ\)=πθ​\(ai,t∣si,t\)πprox​\(ai,t∣si,t\),ρ¯i,t​\(θ\)=clip\[0\.80,1\.28\]⁡\(ρi,t​\(θ\)\)\.\\rho\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{i,t\}\\mid s\_\{i,t\}\)\}\{\\pi\_\{\\mathrm\{prox\}\}\(a\_\{i,t\}\\mid s\_\{i,t\}\)\},\\quad\\bar\{\\rho\}\_\{i,t\}\(\\theta\)=\\operatorname\{clip\}\_\{\[0\.80,\\,1\.28\]\}\\\!\\left\(\\rho\_\{i,t\}\(\\theta\)\\right\)\.\(8\)The TIS\-corrected clipped surrogate and final objective are

ℓi,t​\(θ\)=wi,tTIS​min⁡\(ρi,t​\(θ\)​A^i,ρ¯i,t​\(θ\)​A^i\),\\ell\_\{i,t\}\(\\theta\)=w\_\{i,t\}^\{\\mathrm\{TIS\}\}\\,\\min\\\!\\left\(\\rho\_\{i,t\}\(\\theta\)\\hat\{A\}\_\{i\},\\ \\bar\{\\rho\}\_\{i,t\}\(\\theta\)\\hat\{A\}\_\{i\}\\right\),\(9\)𝒥GRPO\(θ\)=𝔼i,t\[ℓi,t\(θ\)\]−βKL\(πθ∥πSFT\)\.\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{i,t\}\\\!\\left\[\\ell\_\{i,t\}\(\\theta\)\\right\]\-\\beta\\,\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{SFT\}\}\\right\)\.\(10\)TIS limits variance from rollout–training mismatch; policy\-ratio clipping limits local update magnitude; and the KL anchor retains the SFT checkpoint as a behavioral reference\. We track token entropy, clipped\-token ratepclipp\_\{\\mathrm\{clip\}\}, and TIS\-truncation ratepTISp\_\{\\mathrm\{TIS\}\}to separate entropy collapse, excessive local movement, and rollout mismatch\.

![Refer to caption](https://arxiv.org/html/2609.22194v1/fig_update_stability_tis_draft_3.png)Figure 5:Optimization stability and rollout correction\.Targeted SFT\+RL exhibits smaller update magnitudes, lower clipping activity, and weaker TIS correction pressure than RL from Base or uniform SFT\+RL\. The gradient clipping threshold is 10; the policy\-ratio interval is\[0\.80,1\.28\]\[0\.80,1\.28\]; TIS usesC=2C=2\.
## Appendix HPublic\-Benchmark Consistency Check

We evaluate Base, SFT, and SFT\+RL on BFCL\([Patil et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib23)\)and ToolBench\([Qin et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib25)\)subsets not used to select the proprietary RL mix\. These labels are assigned after observing checkpoint behavior and are therefore structural analogues rather than prospective external validation\.

Observed patternBenchmark subsetBaseSFT\+RLΔRL\\Delta\_\{\\mathrm\{RL\}\}Imitation\-likeBFCL \(Single\-Tool\)82\.1%93\.4%92\.1%−1\.3\-1\.3Lift\-likeToolBench \(Multi\-Step\)74\.2%81\.5%84\.3%\+2\.8\+2\.8Discovery\-likeBFCL \(Missing Params\)61\.4%68\.2%81\.5%\+13\.3\+13\.3Table 8:External consistency check on public tool\-use subsets\.BFCL Single\-Tool has a narrow action space, and SFT learns syntax and selection convention before RL slightly regresses the calibrated mode\. ToolBench Multi\-Step admits multiple valid tool orderings; both SFT and RL add value\. BFCL Missing Parameters requires recognizing absent required parameters rather than blindly emitting a call; RL provides the largest additional gain\. This last pattern is a structural analogue of Discovery behavior, not a proxy for the proprietary non\-disclosure task\.

## Appendix IQualitative Case Study: Discovery\-Regime Abstraction

To understand why RL can break through the SFT plateau on non\-disclosure, we analyze sanitized trajectories where a tool returns competitor\-sensitive bidding behavior alongside advertiser metrics\.

#### SFT failure mode: imitation and leakage\.

The SFT model can remain factually responsive while exposing a restricted raw value:

> “Your impression share dropped because Competitor X increased their top\-of\-search bid to $4\.50, outbidding your maximum of $3\.00\.”

#### RL success mode: constrained abstraction\.

The targeted RL model instead produces an actionable abstraction:

> “Your impression share dropped significantly due to increased competitive density at the top\-of\-search\. To regain visibility, consider directionally increasing your top\-of\-search modifier while monitoring efficiency\.”

This contrast illustrates the reward target but is not itself safety evidence; the independent SME audit in Appendix[J](https://arxiv.org/html/2609.22194#A10)provides the quantitative held\-out validation\.

## Appendix JJury Calibration and Independent Human Audit

#### Jury calibration\.

We calibrate the open\-weight jury \(DeepSeek\-R1, Qwen3\-235B\-A22B\) against a 200\-trajectory human\-rated seed set stratified across completed domain skills\. Two SME raters independently score each trajectory’s<final\_response\>on a 5\-point quality scale; inter\-rater agreement is Cohen’sκ=0\.74\\kappa=0\.74\. Per\-juror Spearman with averaged human scores is 0\.76 \(DeepSeek\-R1\) and 0\.79 \(Qwen3\-235B\-A22B\); inter\-juror Spearman is 0\.81; ensemble jury versus averaged human is 0\.78\. The pre\-specified deployment threshold isρ≥0\.75\\rho\\geq 0\.75\.

#### Independent non\-disclosure audit\.

The safety audit is a separate held\-out annotation set from the 200\-trajectory calibration sample\. SME\-authored standard and adversarial prompts are used to evaluate direct/indirect leakage, factual correctness, actionability, excessive vagueness, refusal, and an overall acceptable\-safety judgment\. Two independent SME annotators achieve Cohen’sκ=0\.76\\kappa=0\.76; disagreements are adjudicated by a second SME cohort\.

MetricControlSFTSFT\+RLStandard\-prompt leakage rate12\.40%11\.80%2\.90%Adversarial\-prompt leakage rate24\.70%22\.90%6\.80%Informative/actionable response rate84\.60%86\.20%85\.70%Refusal rate5\.80%5\.50%6\.40%Acceptable safety rate76\.90%78\.40%91\.80%Table 9:Full independent SME audit of non\-disclosure\. The result separates leakage from usefulness and refusal behavior\.

## Appendix KPer\-Skill SFT Trajectory Volumes and RL Mix

SFT trajectories are unevenly distributed across skill families, weighted by empirical difficulty and downstream importance\. Exact per\-skill training counts are proprietary; relative weights are summarized below\. The RL mix re\-weights toward exploration\-heavy skills identified by the routing diagnostic and excludes Imitation\-regime skills\.

RegimeSFT mix weightRL mix weightImitation1\.0×\\times0×\\times\(excluded\)Lift1\.0×\\times1\.0×\\timesDiscovery1\.5×\\times2\.5×\\timesOOD \(Toucan, regularizer\)0\.3×\\times–Table 10:Trajectory mix weights normalized within each stage relative to Lift at 1\.0×\\times\.
## Appendix LHyperparameters and Empirical Sweeps

GRPO uses group sizeG=8G=8, KL coefficientβ=0\.04\\beta=0\.04\(constant; no annealing\), reward weightingα=0\.4\\alpha=0\.4forRtoolR\_\{\\mathrm\{tool\}\}and1−α=0\.61\{\-\}\\alpha=0\.6forRjuryR\_\{\\mathrm\{jury\}\}, rollout context limit 64K tokens, and learning rate5×10−75\\times 10^\{\-7\}with AdamW and linear warmup over 100 steps\. We use asymmetric policy\-ratio clipping withϵlow=0\.20\\epsilon\_\{\\mathrm\{low\}\}=0\.20andϵhigh=0\.28\\epsilon\_\{\\mathrm\{high\}\}=0\.28, giving\[0\.80,1\.28\]\[0\.80,1\.28\]\. TIS usesC=2C=2\. SFT uses learning rate1×10−51\\times 10^\{\-5\}, batch size 64 trajectories, and three epochs\.

The specific values forα\\alpha,β\\beta, sequence limits, and data\-mix ratios were selected following 83 large\-scale development experiments, including unsuccessful RL\-from\-Base attempts\. The exact sensitivity surfaces are proprietary\. Recipe compute in Table[4](https://arxiv.org/html/2609.22194#A1.T4)includes only the attributable SFT/RL compute of the retained recipes; shared hyperparameter\-search and infrastructure bring\-up compute are excluded from recipe totals because they are shared development costs rather than cleanly attributable to one recipe\.

## Appendix MOperational Metrics

The targeted candidate and Control were measured over the same traffic slice and serving configuration during a fixed online evaluation period\. Exact traffic volume, evaluation dates, and shadow/live assignment mode cannot be disclosed under confidentiality requirements\. Accordingly, the table reports normalized changes only and is interpreted as operational evidence rather than a reproducible randomized causal estimate of online impact\.

Operational metricChange vs\. ControlSupported QPS\+10%\+10\\%Generation throughput\+2\.9%\+2\.9\\%tokens/sP50 user\-perceived latency−4\.2%\-4\.2\\%P95 user\-perceived latency−6\.8%\-6\.8\\%Recommendation acceptance rate\+3\.6\+3\.6percentage pointsMulti\-turn task\-completion rate\+5\.4\+5\.4percentage pointsRecovery after a failed tool call\+7\.1\+7\.1percentage pointsAverage turns to resolution−5%\-5\\%Session abandonment rate−2\.1\-2\.1percentage pointsTable 11:Serving\-period changes for the targeted candidate relative to Control\. Negative latency, turns\-to\-resolution, and abandonment changes are improvements\.
## Appendix NInfrastructure for GPT\-OSS Reinforcement Learning

The advertiser environment is built on a Megatron\-backed\([Shoeybi et al\., 2019](https://arxiv.org/html/2609.22194#bib.bib32)\)SLIME\([Zhu et al\., 2025](https://arxiv.org/html/2609.22194#bib.bib50)\)stack with a disaggregated SGLang\([Zheng et al\., 2024](https://arxiv.org/html/2609.22194#bib.bib47)\)rollout engine\. We extend this stack to support multi\-turn beta\-API rollouts, skill\-aware reward shaping, heavy\-tailed token lengths, and GPT\-OSS\-120B MoE synchronization\. We describe framework\-level changes; proprietary reward functions, datasets, and product\-specific tool implementations are not disclosed\.

#### Disaggregated RL setup\.

Training and inference run on disjoint GPU pools: while Megatron performs the optimizer step for batchkk, SGLang generates rollouts for batchk\+1k\{\+\}1and reward workers score batchk−1k\{\-\}1\. This asynchronous design hides rollout latency behind training compute\. The training policy and rollout engine use different parallel decompositions and weight layouts, so each policy update requires explicit Megatron→\\rightarrowSGLang synchronization\.

#### SLIME: custom rollout generator and reward hub\.

SLIME dispatches inference to SGLang through a load\-balancing router, computes rewards in a separateRewardHubactor, and returns batches to Megatron\. We replace the default single\-turn completion with a multi\-turngenerate\_rolloutloop implementing model action→\\rightarrowtool dispatch→\\rightarrowenvironment observation→\\rightarrownext state\. The generator dispatches structured JSON tool calls, parses outputs, appends observations, and masks tool\-output tokens from the loss\. The reward hub is extended for the multi\-juror ensemble and skill\-aware rubric routing\.

#### MoE weight synchronization for disaggregated rollout\.

GPT\-OSS uses fused, interleaved gate/up projections for experts, whereas Megatron emits unfused per\-expert tensors\. During synchronization, these tensors are repacked into the fusedw13/w2layout served by SGLang\. We add an expert\-parallel\-aware load path that materializes only local experts and restores the SwiGLU gate/up ordering\. Under tensor parallelism, the all\-gather that reconstructs the GLU layout must be applied consistently to weights and biases; extending the same re\-chunking transformation to biases makes Megatron→\\rightarrowSGLang transfer bit\-correct\.

#### Context parallelism for learnable\-softmax attention\.

GPT\-OSS combines learnable\-softmax attention with interleaved full\-attention and sliding\-window layers\. We add context\-parallel support through a custom attention module\. Full\-attention layers use zigzag ring attention over shardedq/k/vq/k/vtensors; the learnable\-softmax scale is threaded through forward and backward\. Sliding\-window layers instead all\-gatherQ/K/VQ/K/V, run local flash attention with the window mask, and re\-slice outputs, because the global zigzag permutation breaks locality\.

#### Heavy\-tail handling and reward routing\.

We bucket rollouts by total context length and dynamically resize each batch to a fixed token budget, keeping memory approximately stable without truncating long trajectories or heavily padding short ones\. Skill metadata travels with each rollout through generation and reward scoring so the reward hub can activate the appropriate rubric dimensions\. Together, asynchronous rollout, MoE synchronization, context\-parallel attention support, and token\-budget batching make multi\-node long\-context GRPO feasible for GPT\-OSS\-120B\.