SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation

# Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Source: [https://arxiv.org/html/2608.03092](https://arxiv.org/html/2608.03092)
\\correspondingauthor

zhoumengyu\.zmy@alibaba\-inc\.com

Jiahua Bao‡Qwen Large Model Application Team, AlibabaTu Yongsiqi‡Qwen Large Model Application Team, AlibabaYihao Liu‡Qwen Large Model Application Team, AlibabaHaotian Zhou‡Qwen Large Model Application Team, AlibabaHaoxuan Ma‡Qwen Large Model Application Team, AlibabaMengyu Zhou†Qwen Large Model Application Team, AlibabaWenkui Fan‡Qwen Large Model Application Team, AlibabaJunwei HeUniversity of Chinese Academy of SciencesXiaoxi JiangQwen Large Model Application Team, AlibabaGuanjun JiangQwen Large Model Application Team, Alibaba

###### Abstract

We aim to improve model performance in multi\-reward reinforcement learning training process\. Existing Group reward\-Decoupled Normalization Policy Optimization \(GDPO\) has mitigated the issue of reward signals masking one another during direct scalarization by normalizing each reward dimension separately before aggregation\. However, our experiments show that GDPO still struggles to balance reward signals with different granularities\. Specifically, in some particular training tasks, the model may receive a dense reward that assigns fine\-grained scores ranging from 0\.1 to 1\.0, together with a sparse reward that provides only binary feedback of either 0 or 1\. In such cases, we find that the sparse reward may provide an insufficient optimization signal, preventing its corresponding capability from being effectively reinforced\. Therefore,*how can we strengthen the optimization signal from the sparse reward without sacrificing the capability already learned from the fine\-grained reward?*To overcome this limitation, we proposeSpecialize\-and\-Merge Online Policy Distillation \(SMOPD\), a two\-stage training method for multi\-reward optimization\.*Stage1\-Specialize*: SMOPD first employs reward\-priority configurations to train multiple reward\-specialized teachers, allowing each reward to be learned under conditions where its signal can effectively drive optimization\.*Stage2\-Merge*: SMOPD then utilizes online policy distillation to combine the reward\-specialized capabilities of these teachers into a single student policy, while maintaining balanced task\-level optimization\. To validate our method, we conduct experiments on two multi\-reward settings: complementary rewards \(tool\-calling accuracy and format\) and conflicting rewards \(helpful and harmless rewards\)\. Based on above settings, SMOPD outperforms GDPO across 1\.5B, 3B and 7B backbones\.

## 1Introduction

Reinforcement learning from human feedback \(RLHF\) has become the dominant paradigm for aligning large language models with human preferencesouyang2022instructgpt;christiano2017rlhf\. In multi\-reward alignment, prior work commonly adopts GRPOshao2024deepseekmathto optimize a single policy with several reward signals\. However, because GRPO first sums different rewards and then computes group\-relative advantages, reward dimensions with different scales or reward combinations can mask one another during scalarization, causing the final advantage to lose reward\-specific information\. GDPOliu2026gdpomitigates this aggregation\-level issue by normalizing each reward dimension separately before aggregation, thereby preserving the contribution of each reward in the training signal\. Building on this reward\-level decomposition, GD2POliu2026gd2pofurther addresses conflicts among reward dimensions by filtering rollouts with severe reward\-wise disagreement and reweighting queries according to reward consensus\. Despite these improvements, GDPO and GD2PO still focus on how observed reward signals are normalized and combined within a single policy\.

This leaves open a different problem:Reward dimensions can differ not only in scale, but also in how often they provide informative learning signals\.Group\-based advantage estimation relies on reward differences among rollouts sampled for the same prompt\. Dense reward distributions can rank sampled responses in most rollout groups, continuously providing reliable optimization signals\. Sparse reward distributions, by contrast, may assign the same value to most responses in a group, leaving that reward dimension with little useful within\-group variation\. In such groups, per\-reward normalization cannot create an informative signal where no response\-level distinction exists\. As a result, under balanced priorities, training is still dominated by dense reward dimensions, while the occasional signal from sparse reward distributions can be overwhelmed before it accumulates\. Figure[1](https://arxiv.org/html/2608.03092#S1.F1)\(a\) illustrates that the balanced GDPO advantage remains closely aligned with the dense reward\.

![Refer to caption](https://arxiv.org/html/2608.03092v1/x1.png)

\(a\) Reward\-density effect\.

![Refer to caption](https://arxiv.org/html/2608.03092v1/x2.png)

\(b\) Performance comparison\.

Figure 1:\(a\)In a batch of88prompt groups with44rollouts each, the dense reward vector is\[0,1,2,3\]\[0,1,2,3\]in all88groups, whereas the sparse reward vector is\[0,0,0,0\]\[0,0,0,0\]in77groups and\[1,0,0,0\]\[1,0,0,0\]in the remaining group\. The bars show batch\-averaged advantage profiles under balanced weights\(0\.5,0\.5\)\(0\.5,0\.5\)and sparse\-priority weights\(0\.1,0\.9\)\(0\.1,0\.9\)\.\(b\)Main results comparing SMOPD with multi\-reward baselines on RLLA Test \(1\.5B\), API\-Bank LLM\-Judge \(1\.5B\), and Safe Domain \(7B\)\.Therefore, single\-policy multi\-reward RL faces an inherent tension: Raising the priority of a sparse reward to make it learnable inevitably skews the final balance, yet keeping a balanced priority leaves it overshadowed\. Our experiments confirm this issue in tool callingschick2023toolformer;qin2024toolllm, where a dense accuracy reward is paired with a sparse binary format reward\. Under balanced priorities, GDPO behaves similarly to GRPO, with format compliance remaining below9%9\\%\. When the format reward is made dominant, the model reliably learns the required output structure, but the resulting policy is optimized under a deliberately skewed objective\. This suggests that GDPO alleviates aggregation\-level reward masking, yet still struggles with the signal\-density imbalance between sparse and dense reward distributions\.

To resolve this tension, we proposeSpecialize\-and\-Merge Online Policy Distillation \(SMOPD\), a two\-stage method for multi\-reward optimization\. SMOPD is designed to preserve the strengths learned under dense reward distributions while improving the model’s sensitivity to sparse reward distributions\.*Stage1\-Specialize*\. SMOPD uses reward\-priority configurations to train multiple reward\-specialized teachers from the same base policy\. Instead of balancing all rewards within a single policy, we assign each teacher a reward\-priority profile that amplifies its target reward in the optimization signal\. In particular, assigning higher priority to a sparse reward counteracts its weaker and less frequent learning signal, enabling the policy to more effectively capture the optimization direction induced by that reward\. Figure[1](https://arxiv.org/html/2608.03092#S1.F1)\(a\) illustrates: a sparse\-reward priority profile recovers the sparse reward direction\. This allows sparse or hard\-to\-learn reward distributions to be acquired in a favorable regime, while capabilities learned from dense reward distributions are preserved by complementary teachers\.*Stage2\-Merge*\. SMOPD merges these reward\-specialized teachers into a single student policy through online policy distillation\. The student learns from the teachers’ reward\-specialized behaviors on its own rollouts, while a balanced GDPO anchor maintains task\-level optimization over the original multi\-reward objective\. In this way, SMOPD first acquires different reward strengths through specialization and then balances them through policy\-level merging, producing one unified policy instead of a set of separate specialists\. Figure[2](https://arxiv.org/html/2608.03092#S1.F2)summarizes the full workflow\.

![Refer to caption](https://arxiv.org/html/2608.03092v1/x3.png)Figure 2:Overview of SMOPD\.*Stage 1*: complementary GDPO priority profiles produce an accuracy teacher and a format teacher from the same base policy\.*Stage 2 \(a\)*: on the student’s rollout prefix, frozen teachers emit top\-κ\\kappanext\-token distributions\. The accuracy teacher peaks on tool\-selection tokens, the format teacher peaks on XML boundary tokens, and the union\-top\-κ\\kappamixture retains both\.*Stage 2 \(b\)*: the student is updated through two\-level signals\. Token\-level OPD fits its distribution to the teacher mixture via forward KL, while the sequence\-level task anchor optimizes the balanced multi\-reward objective\.Experimentally, we validate SMOPD across model scales across 1\.5B, 3B and 7B with different model families\(Qwen2\.5qwen2024qwen25and Llama\-3\.2grattafiori2024llama3\) under two reward structures, including complementary rewards \(tool\-calling accuracy and format\) and conflicting rewards\(helpful and harmless rewards\)bai2022hh;bai2022constitutional\. Our teacher analysis further surfaces structure that a scalarized objective cannot represent\.

Our contributions are:

- •SMOPD resolves the reward\-balancing tension in multi\-reward RL\.Sparse reward distributions are often learnable only when made dominant, but such skewed priorities sacrifice other objectives in a single policy\. SMOPD addresses this tension by specializing teachers under complementary reward\-priority profiles, then merging them through token\-level on\-policy distillation with a parameter\-free uniform teacher mixture and a sequence\-level anchor\.
- •SMOPD improves both complementary and conflicting reward settings\.Across three backbones \(1\.5B, 3B, and 7B\), SMOPD consistently exceeds GDPO across different model families\. Specifically, in the complementary setting, the improvement is particularly pronounced—peaking at the 1\.5B model with a\+48%composite gain and a dramatic format compliance jump from 8\.8% to97\.5%\. In the conflicting setting, SMOPD achieves superior Safe Domain benchmark performance over GDPO across every setting\.
- •The merged student surpasses its own teachers\.On safe alignment it exceeds both teachers and the scalarized baseline across three backbones; even at 7B, merging weak teachers yields a clear gain from complementary knowledge unused by scalarized training\.

## 2Related Work

### 2\.1Reinforcement Learning for LLMs

RLHF aligns LLMs with human preferenceschristiano2017rlhf;ziegler2019finetuning;stiennon2020summarize;ouyang2022instructgpt, classically with PPOschulman2017ppoor its preference\-based shortcut DPOrafailov2023dpo, though optimizing against learned reward models is prone to over\-optimizationgao2023overoptimization\. GRPOshao2024deepseekmathremoves the critic by normalizing rewards within a rollout group, and later variants sharpen this estimator, e\.g\. DAPO’s decoupled clipping and dynamic samplingyu2025dapo, GSPO’s sequence\-level importance ratioszheng2025gspo, and simpler REINFORCE\-style baselinesahmadian2024back\. For multiple rewards, the standard treatment scalarizes them into one objective and refines the aggregation: GDPOliu2026gdponormalizes each reward separately and supports priority weights, DVAOdvao2026adapts weights to per\-reward gradient magnitudes, and SAWhe2026sawreweights objectives by learning speed; other lines constrain or trade off objectives instead of summing themxu2024cgpo;dai2024safe;zhou2023modpo;wang2024arithmetic, or merge separately\-trained policies in weight spacewortsman2022soups;rame2024rewarded;rame2024warp\. All of these ultimately ask a single set of parameters to absorb every reward at once, freezing one trade\-off at training time; SMOPD instead trains one teacher per reward and defers balancing to a distillation\-based merge\.

### 2\.2On\-Policy Distillation

Knowledge distillation originally trains a student on a teacher’s soft targetshinton2015distilling, extended to autoregressive LMs by sequence\-level KD on teacher\-generated textkim2016seqkd\. Because teacher\-generated data mismatches what the student sees at inference, GKDagarwal2024gkddistills on the student’s*own*rollouts—on\-policy distillation—with the divergence choice studied by MiniLLMgu2024minillm, DistiLLMko2024distillm, andff\-divergence KDwen2023fdistill, and rollout selection refined by PG\-OPDzhao2026pgopd; we follow this line and use forward KL on student rollouts\. Recent work scales OPD to*multiple teachers*, fusing separately\-trained domain\-specialized policies \(math, coding, instruction following\) into one model more effectively than reward mixing, cascade RL, or parameter mergingma2026mopd; MiMomimo2026flashscales this recipe to frontier post\-training, and G\-OPDyang2026gopdmerges domain\-specialized policies back into a shared base\. Because each prompt belongs to a single domain, all of these route it to the one teacher that owns it\. Our problem is orthogonal:*within a single domain*, several rewards act on the*same*prompt at once, so no routing can separate them; SMOPD instead combines reward\-specialized teachers at every token, turning on\-policy distillation into a mechanism for multi\-reward balancing\.

## 3Method: SMOPD Framework

The core philosophy of SMOPD is to let each reward be learned where it is easiest, in a*teacher*trained to prioritize it, and then to transfer the teachers’ competence into a single student at the granularity where rewards actually act:*individual tokens*\. This section presents the two stages in turn\.

### 3\.1Stage 1: Reward\-Specialized Teacher Training

Group\-based methods sampleGGrollouts\{o1,…,oG\}\\\{o\_\{1\},\\ldots,o\_\{G\}\\\}per prompt and score each rolloutoio\_\{i\}withKKreward dimensionsri=\(ri\(1\),…,ri\(K\)\)r\_\{i\}=\(r\_\{i\}^\{\(1\)\},\\ldots,r\_\{i\}^\{\(K\)\}\); we writeμ\(k\),σ\(k\)\\mu^\{\(k\)\},\\sigma^\{\(k\)\}for the group mean and standard deviation of dimensionkk\. GRPOshao2024deepseekmathnormalizes the*summed*reward within the group,A^iGRPO=\(∑kri\(k\)−μS\)/\(σS\+ϵ\)\\hat\{A\}\_\{i\}^\{\\text\{GRPO\}\}=\(\\sum\_\{k\}r\_\{i\}^\{\(k\)\}\-\\mu\_\{S\}\)/\(\\sigma\_\{S\}\+\\epsilon\), whereμS,σS\\mu\_\{S\},\\sigma\_\{S\}are the group statistics of the sum andϵ\\epsilonis a small stabilizing constant\. GDPOliu2026gdpoinstead \(i\) normalizes each dimension within the group, \(ii\) aggregates the resulting advantages under priority weights𝐰=\(w1,…,wK\)\\mathbf\{w\}=\(w\_\{1\},\\ldots,w\_\{K\}\)\(equal by default\), and \(iii\) applies a final*batch*\-wise whitening over all responses in the update:

A^i\(k\)\\displaystyle\\hat\{A\}\_\{i\}^\{\(k\)\}=ri\(k\)−μ\(k\)σ\(k\),\\displaystyle=\\frac\{r\_\{i\}^\{\(k\)\}\-\\mu^\{\(k\)\}\}\{\\sigma^\{\(k\)\}\},\(1\)Aisum\\displaystyle A\_\{i\}^\{\\text\{sum\}\}=∑k=1Kwk⋅A^i\(k\),\\displaystyle=\\sum\_\{k=1\}^\{K\}w\_\{k\}\\cdot\\hat\{A\}\_\{i\}^\{\(k\)\},\(2\)A^iGDPO\\displaystyle\\hat\{A\}\_\{i\}^\{\\text\{GDPO\}\}=Aisum−μbatch​\(Asum\)σbatch​\(Asum\)\+ϵ\.\\displaystyle=\\frac\{A\_\{i\}^\{\\text\{sum\}\}\-\\mu\_\{\\text\{batch\}\}\(A^\{\\text\{sum\}\}\)\}\{\\sigma\_\{\\text\{batch\}\}\(A^\{\\text\{sum\}\}\)\+\\epsilon\}\.\(3\)Both estimators train the policyπθ\\pi\_\{\\theta\}with the standard clipped surrogate objective on these advantagesschulman2017ppo;shao2024deepseekmath\.

GDPO’s priority weights thus let a single policy be steered toward a selected trade\-off\. SMOPD repurposes this control as*reward\-priority specialization*: a set of complementary priority profiles constructs multiple teachers, one for each reward dimension, before they are merged in Stage 2\. For a reward dimensionkkwe wish to specialize on, we setwk≫wjw\_\{k\}\\gg w\_\{j\}forj≠kj\\neq k:

A^teacher,k=wkhigh⋅A^\(k\)\+∑j≠kwjlow⋅A^\(j\)\\hat\{A\}\_\{\\mathrm\{teacher\},k\}=w\_\{k\}^\{\\text\{high\}\}\\cdot\\hat\{A\}^\{\(k\)\}\+\\sum\_\{j\\neq k\}w\_\{j\}^\{\\text\{low\}\}\\cdot\\hat\{A\}^\{\(j\)\}\(4\)
For example, withK=2K=2\(accuracy and format\), we train aformat teacherwith𝐰=\(0\.1,0\.9\)\\mathbf\{w\}=\(0\.1,0\.9\), amplifying the format reward signal by9×9\\timesrelative to accuracy, and anaccuracy teacherwith𝐰=\(0\.9,0\.1\)\\mathbf\{w\}=\(0\.9,0\.1\), and vice versa\.

Because per\-reward normalization gives eachA^\(k\)\\hat\{A\}^\{\(k\)\}zero mean and unit variance, the9×9\\timesweight ratio directly translates to a9×9\\timesexpected gradient contribution from the favored reward \(the normalized advantages are placed on a common scale, so the weights alone set their relative influence\)\. This makes the model “zoom in” on its target dimension, reliably maximizing it; the non\-target dimensions may degrade \(as for the accuracy teacher\) or, when the dimensions are not in conflict, be retained \(as for the format teacher, Section[4\.2](https://arxiv.org/html/2608.03092#S4.SS2.SSS0.Px1)\)\. Either way the teachers become*complementary*, each contributing a distinct strength to the subsequent merge\.

### 3\.2Stage 2: Multi\-Teacher Online Policy Distillation

GivenMMteacher teachers\{π1,…,πM\}\\\{\\pi\_\{1\},\\ldots,\\pi\_\{M\}\\\}, SMOPD distills their token\-level knowledge into a studentπθ\\pi\_\{\\theta\}\. Unlike offline distillation, the student generates its own responses \(on\-policy\), and teachers provide top\-κ\\kappalog\-probabilities on the student’s sequences—following the on\-policy distillation paradigmagarwal2024gkd, which avoids the exposure bias of distilling on a fixed teacher\-generated corpus\.

#### Teacher mixture distribution\.

At each token positionttin a student\-generated response, the teacher target is theα\\alpha\-weighted mixture of theMMteachers’ next\-token distributions:

pTmix​\(v∣st\)∝∑m=1Mαm​pπm​\(v∣st\),p\_\{T\}^\{\\text\{mix\}\}\(v\\mid s\_\{t\}\)\\;\\propto\\;\\sum\_\{m=1\}^\{M\}\\alpha\_\{m\}\\,p\_\{\\pi\_\{m\}\}\(v\\mid s\_\{t\}\),\(5\)wherevvranges over vocabulary tokens,sts\_\{t\}is the student’s generated prefix up to positiontt, andαm\\alpha\_\{m\}are the mixture weights\. SMOPD usesuniformweightsαm=1/M\\alpha\_\{m\}=1/M\(e\.g\.\[0\.5,0\.5\]\[0\.5,0\.5\]for two teachers\), the simplest and, as we show, strongest choice in our setting; input\-dependent gating alternatives are explored in Supplementary material[E](https://arxiv.org/html/2608.03092#A5)\.

Operating over the full vocabulary for every teacher at every position is prohibitive, so we realize Eq\. equation[5](https://arxiv.org/html/2608.03092#S3.E5)with a*component\-wise top\-κ\\kappa*construction \(κ=16\\kappa=16by default, distinct from the reward\-dimension countKK; sensitivity is studied in Section[4\.3](https://arxiv.org/html/2608.03092#S4.SS3.SSS0.Px2)\)\. Each teachermmemits only its own top\-κ\\kappatoken set𝒯mκ\\mathcal\{T\}\_\{m\}^\{\\kappa\}with log\-probabilitieslog⁡pπm\\log p\_\{\\pi\_\{m\}\}\. We shift each bylog⁡αm\\log\\alpha\_\{m\}, pool theM×κM\\\!\\times\\\!\\kappacandidates, and keep theκ\\kappalargest as the mixture target:

p~Tmix\(⋅∣st\)=top​\-​κm=1,…,Mv∈𝒯mκ\[logαm\+logpπm\(v∣st\)\]\.\\tilde\{p\}\_\{T\}^\{\\text\{mix\}\}\(\\cdot\\mid s\_\{t\}\)=\\operatorname\*\{top\\text\{\-\}\\kappa\}\_\{\\begin\{subarray\}\{c\}m=1,\\ldots,M\\\\ v\\in\\mathcal\{T\}\_\{m\}^\{\\kappa\}\\end\{subarray\}\}\\big\[\\,\\log\\alpha\_\{m\}\+\\log p\_\{\\pi\_\{m\}\}\(v\\mid s\_\{t\}\)\\,\\big\]\.\(6\)This is a fast approximation to the exact union\-and\-renormalize mixture \(log​∑mαm​pm\\log\\sum\_\{m\}\\alpha\_\{m\}p\_\{m\}\): a token in several teachers’ top\-κ\\kappasets may appear more than once among the candidates, and tokens outside every teacher’s top\-κ\\kappaset are dropped\. With high\-quality teachers whose top\-κ\\kappamass dominates, the two coincide closely\. The result is a single per\-position top\-κ\\kappatarget that jointly carries both teachers\.

#### Distillation loss\.

The student minimizes the forward KL divergence from the \(top\-κ\\kappa\) teacher mixture, evaluated over the mixture’s support:

ℒOPD=∑tKL\(p~Tmix\(⋅∣st\)∥pθ\(⋅∣st\)\)\\mathcal\{L\}\_\{\\text\{OPD\}\}=\\sum\_\{t\}\\operatorname\{KL\}\\left\(\\tilde\{p\}\_\{T\}^\{\\text\{mix\}\}\(\\cdot\\mid s\_\{t\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\right\)\(7\)We use forward \(rather than reverse\) KL deliberately: forward KL is mode\-coveringgu2024minillm;ko2024distillm, encouraging the student to place mass on*all*high\-probability tokens of the teacher mixture—important here because different teachers may be confident about different tokens \(e\.g\., format wrappers vs\. function arguments\) at the same position, and the union\-top\-κ\\kappatarget in Eq\. equation[6](https://arxiv.org/html/2608.03092#S3.E6)preserves both\. Section[4\.3](https://arxiv.org/html/2608.03092#S4.SS3.SSS0.Px2)ablates both choices, while Supplementary material[C](https://arxiv.org/html/2608.03092#A3)details the sampled\-token reverse\-KL estimator\.

#### GDPO anchor\.

To prevent the student from only imitating teacher distributions without task\-level grounding, we add a GDPO loss with equal weights𝐰equal=\(1/K,…,1/K\)\\mathbf\{w\}\_\{\\text\{equal\}\}=\(1/K,\\ldots,1/K\):

ℒanchor=ℒGDPO​\(πθ;𝐰equal\),\\mathcal\{L\}\_\{\\text\{anchor\}\}=\\mathcal\{L\}\_\{\\text\{GDPO\}\}\(\\pi\_\{\\theta\};\\mathbf\{w\}\_\{\\text\{equal\}\}\),\(8\)whereℒGDPO​\(πθ;𝐰\)\\mathcal\{L\}\_\{\\text\{GDPO\}\}\(\\pi\_\{\\theta\};\\mathbf\{w\}\)denotes the clipped policy\-gradient loss driven by the GDPO advantage of Eq\. equation[3](https://arxiv.org/html/2608.03092#S3.E3)under weights𝐰\\mathbf\{w\}\.

#### Total objective\.

The student optimizes:

ℒtotal=ℒanchor\+λ⋅ℒOPD\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{anchor\}\}\+\\lambda\\cdot\\mathcal\{L\}\_\{\\text\{OPD\}\}\(9\)whereλ\\lambdacontrols the distillation strength \(defaultλ=1\.0\\lambda=1\.0\)\. The anchor provides sequence\-level direction \(“this response should be upweighted/downweighted”\), while OPD provides token\-level guidance \(“at this position, the distribution should look like this mixture of teachers”\)\.

Table 1:Main results on complementary rewards setting \(RLLA\), across Qwen2\.5\-\{1\.5B, 3B\}\-Instruct\.In\-domaincolumns use the held\-out RLLA\-4K test split;Generalizationcolumns are held\-out tool\-use benchmarks\. For API\-Bank,qwen3\.7\-plusserves as the semantic judge for original exact\-match failures while exact\-match successes are retained; All metrics are defined in Section[4\.1](https://arxiv.org/html/2608.03092#S4.SS1);Boldmarks the best andunderlinethe second best among the baselines and SMOPD;Table 2:Main results on conflicting rewards\(Safe\-alignment benchmarks\) across Qwen2\.5\-3B, 7B\-Instruct, and Llama\-3\.2\-3B\-Instruct, reported per benchmark with separate Useful \(U\.\) and Harmless \(H\.\) scores and their average \(Avg\);Overallis the mean Avg across the three benchmarks\.

## 4Experiments

### 4\.1Experimental Setup

#### Training\.

We train SMOPD on five backbones\. Within each setting, the baselines, reward\-specialized teachers, and SMOPD students share the same training recipe, differing only in the GDPO priority profile and, for SMOPD, the added distillation loss; all runs utilize the verl frameworksheng2024hybridflowon 8 H100 GPUs\.Complementary rewards\(RLLA\)\.We train Qwen2\.5\-\{1\.5B, 3B\}\-Instructqwen2024qwen25withG=8G\{=\}8rollouts per prompt on the RLLA\-4K*training split*from ToolRLqian2025toolrl\. The*accuracy reward*\(racc∈\[−3,3\]r\_\{\\text\{acc\}\}\\in\[\-3,3\]\) scores function\-name and parameter matching, while the binary*format reward*\(rfmt∈\{0,1\}r\_\{\\text\{fmt\}\}\\in\\\{0,1\\\}\) checks the required XML structure \(<think\>,<tool\_call\>, and<response\>tags\)\. The format and accuracy teachers are trained with complementary priority profiles,𝐰=\(0\.1,0\.9\)\\mathbf\{w\}=\(0\.1,0\.9\)and\(0\.9,0\.1\)\(0\.9,0\.1\), respectively; we additionally conduct an ablation study on other priority configurations in Supplementary material[D](https://arxiv.org/html/2608.03092#A4)\.Conflicting rewards\(helpful \+ harmless\)\.We train Qwen2\.5\-\{3B, 7B\}\-Instruct and Llama\-3\.2\-3B\-Instruct withG=4G\{=\}4on prompt\-only Alpacataori2023alpaca\. Useful and harmless rewards are produced by dual reward models trained on PKU\-SafeRLHF preference datadai2024safe, and the corresponding teachers employee the complementary profiles,\[0\.7,0\.3\]\[0\.7,0\.3\]and\[0\.3,0\.7\]\[0\.3,0\.7\]\. Full training hyperparameters are provided in Supplementary material[A](https://arxiv.org/html/2608.03092#A1)\(Tables[6](https://arxiv.org/html/2608.03092#A1.T6)and[6](https://arxiv.org/html/2608.03092#A1.T6)\)\.

#### Evaluation\.

All models are evaluated with vLLMkwon2023vllmusing temperature 0\.0 and top\-pp1\.0;Complementary rewards\(RLLA\)\.On the held\-out RLLA\-4K test split \(80 prompts\),RLLA Meanis the primary composite metricracc\+rfmtr\_\{\\text\{acc\}\}\+r\_\{\\text\{fmt\}\}; we additionally report itsAcc Rewardcomponent and the binaryFormat Passrate\. Generalization is evaluated on two held\-out tool\-use benchmarks:BFCL ASTis BFCL\-v4patil2023gorillafunction\-calling accuracy averaged over the non\-live and live AST categories, andAPI\-Bank \(LLM\-Judge\)li2023apibankmeasures functional accuracy on Level\-1/2/3\. For API\-Bank, exact\-match successes remain correct and an LLM judgezheng2023judgingreviews only exact\-match failures for functionally equivalent tool calls; we reportL1–L3and micro\-accuracy over all 597 items \(Avg\)\. The BFCL category breakdown is in Supplementary material[G](https://arxiv.org/html/2608.03092#A7), and the complete API\-Bank protocol, original strict scores, and failure analysis are in Supplementary material[F](https://arxiv.org/html/2608.03092#A6)\.Conflicting rewards\(helpful \+ harmless\)\.Every model is scored by the same dual reward models on held\-out HH\-RLHFbai2022hh, PKU\-SafeRLHF, and Alpaca prompts \(mean@1\)\. We report per\-benchmarkUseful,Harmless, and their average;Overallis the mean of these averages across the three benchmarks\.

### 4\.2Main Results

#### Reward\-priority specialization makes sparse reward distributions learnable\.

As shown in the 1\.5B setting of Table[1](https://arxiv.org/html/2608.03092#S3.T1)\(rows 2–4\), GDPO struggles to optimize the format reward under balanced priorities, achieving only 8\.8% format compliance on the RLLA test set\. This is because the binary format reward follows a sparse reward distribution: in most rollout groups, responses receive identical format scores, leaving little within\-group variation for group\-based RL to exploit\. Although the non\-zero format score indicates that the model can occasionally generate correctly formatted responses, this sparse signal is overwhelmed by the denser accuracy reward distribution during balanced multi\-reward optimization\. By increasing the priority of the format reward, the format\-specialized teacher \(𝐰=\[0\.1,0\.9\]\\mathbf\{w\}=\[0\.1,0\.9\]\) amplifies this otherwise underutilized signal and raises format compliance to 97\.5%\. Importantly, this gain does not come at the cost of accuracy performance\. It shows that reward\-priority specialization can make the sparse reward distribution learnable while preserving the dense\-reward capability\.

#### SMOPD under complementary rewards\.

We first study the complementary reward setting, where accuracy and format rewards supervise different aspects of tool\-calling behavior\. Table[1](https://arxiv.org/html/2608.03092#S3.T1)reports results on both backbones\. On Qwen2\.5\-1\.5B, SMOPD achieves an RLLA score of 2\.740, yielding a\+48%improvement over the GDPO baseline \(1\.849\), while preserving the structured\-output capability learned by the format\-specialized teacher \(97\.5% format compliance\)\. Meanwhile, the student retains accuracy\-oriented tool\-use ability: BFCL AST reaches 70\.7%, and API\-Bank LLM\-judge Avg reaches 87\.1%—the best among all 1\.5B methods in Table[1](https://arxiv.org/html/2608.03092#S3.T1)and\+3\.5%points above the strongest scalarized baseline\. We report the API\-Bank LLM\-judge metric because the original strict exact\-match protocol can penalize functionally equivalent tool calls due to superficial differences in argument formatting or values; therefore, we apply semantic judging only to exact\-match failures while preserving exact successes\. The complete judging procedure and the corresponding strict exact\-match results are provided in Supplementary material[F](https://arxiv.org/html/2608.03092#A6)\. These results show that SMOPD can merge complementary teacher capabilities instead of collapsing them into a single averaged behavior\.

#### SMOPD under conflicting rewards\.

We further evaluate SMOPD under conflicting rewards, where helpfulness and harmlessness require balancing competing alignment objectives \(Section[4\.2](https://arxiv.org/html/2608.03092#S4.SS2.SSS0.Px1)\)\. Across three backbones \(Table[2](https://arxiv.org/html/2608.03092#S3.T2)\), SMOPD consistently improves over scalarized baselines by merging reward\-specialized capabilities; adaptive teacher mixing variants are compared in Supplementary material[E](https://arxiv.org/html/2608.03092#A5)\. On Qwen2\.5\-3B, SMOPD achieves the best Overall score \(5\.669\), outperforming the strongest scalarized baseline GD2PO \(5\.650\) and both single\-reward teachers\. The gain becomes more evident on Qwen2\.5\-7B: although each individual teacher provides only marginal improvement over GDPO \(5\.500 and 5\.522 versus 5\.498\), SMOPD further improves the merged policy to 5\.646, exceeding all baselines and teachers\. This suggests that even under conflicting objectives, reward\-specialized teachers retain complementary capabilities that can be recovered through policy\-level merging\. On Llama\-3\.2\-3B, where teacher specialization is the most challenging, both individual teachers degrade substantially, yet SMOPD recovers the performance to 5\.590, approaching the strongest scalarized baseline GD2PO \(5\.605\) while surpassing GDPO \(5\.583\)\. The remaining gap to GD2PO is mainly attributed to the task anchor rather than the teacher\-merging mechanism: SMOPD uses GDPO as its sequence\-level anchor, whereas GD2PO improves the scalarized optimization objective itself through a refined advantage estimator\. Therefore, improving the anchor provides an orthogonal direction to further enhance SMOPD, while the consistent gain over GDPO demonstrates the effectiveness of reward\-specialized capability merging\.

### 4\.3Ablation and Analysis

#### Ablating the Task Anchor\.

SMOPD combines two training signals operating at different levels of granularity\. Thesequence\-levelGDPO anchor determines whether a sampled response should be upweighted or downweighted according to reward\-normalized advantages, providing task\-level optimization direction but only a coarse scalar signal for the entire response\. In contrast,token\-levelOPD distillation shapes the student’s distribution toward the teacher mixture at every generation step, providing dense supervision through hundreds of token\-level gradient signals within a single response\.

This combination is particularly valuable for format compliance, where the reward is binary \(0/1 per response\) but the required behavior—correct XML structure spanning many tokens—needs position\-level guidance\. The GDPO anchor alone cannot teach format because it reduces to a single advantage shared by all tokens in a response\. OPD provides the missing token\-level granularity: the format teacher’s high confidence on XML wrapper tokens \(<tool\_call\>,</tool\_call\>\) and the accuracy teacher’s specialization on function\-argument tokens jointly shape the student’s behavior at each position\.

To test whether both signals are necessary, we ablate the anchor on Qwen2\.5\-7B safe alignment at the controlled settingκ=32\\kappa=32, optimizing only the OPD objective,ℒtotal=ℒOPD\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{OPD\}\}\(Table[3](https://arxiv.org/html/2608.03092#S4.T3)\)\. OPD alone already improves over the individual teachers, reaching5\.5445\.544Overall compared with5\.5005\.500for the useful teacher and5\.5225\.522for the harmless teacher\. However, it remains close to teacher performance, suggesting that pure distillation is limited by the quality of the teacher mixture it imitates\. Reintroducing the anchor lifts SMOPD to5\.6395\.639Overall, exceeding both teachers\. The anchor therefore supplies an additional task\-level optimization signal beyond teacher imitation, allowing the student to improve after absorbing complementary teacher behaviors\. Supplementary material[B](https://arxiv.org/html/2608.03092#A2)formalizes this distinction: the forward\-KL objective in Eq\. equation[7](https://arxiv.org/html/2608.03092#S3.E7)reaches its minimum at the teacher mixture, where the distillation gradient vanishes, whereas the policy\-gradient anchor can continue optimizing task reward\.

Table 3:Ablating the task anchor on Qwen2\.5\-7B\-Instruct safe alignment at the controlled settingκ=32\\kappa=32\(dual\-RM mean@1;Overall==mean of Useful and Harmless\)\.
#### Ablating OPD Design Choices\.

The top\-κ\\kappaapproximation controls the number of teacher candidates retained at each position, trading a broader approximation to the teacher distributions for distillation cost\. We varyκ∈\{16,32,64\}\\kappa\\in\\\{16,32,64\\\}for SMOPD on Qwen2\.5\-7B\-Instruct safe alignment while keeping the training and evaluation protocol unchanged, and separately compare the sampled\-token reverse\-KL update detailed in Supplementary material[C](https://arxiv.org/html/2608.03092#A3)\(Table[4](https://arxiv.org/html/2608.03092#S4.T4)\)\. Forward\-KL performance remains highly stable across support sizes, with Overall varying by only0\.0070\.007betweenκ=16\\kappa=16andκ=64\\kappa=64\. To understand why a small support is sufficient, we measure the teacher probability mass retained by the top\-κ\\kappaapproximation\. The top\-16 support already preserves approximately0\.9940\.994of the teacher mass throughout training \(Figure[3](https://arxiv.org/html/2608.03092#S4.F3)\), explaining why increasingκ\\kappaprovides little additional benefit\.

The sampled\-token reverse\-KL variant reaches 5\.563 Overall,0\.0820\.082below forward KL withκ=16\\kappa=16\. This gap is consistent with the different behaviors of the two divergence directions in our multi\-teacher setting\. Forward KL is mode\-covering, encouraging the student to preserve the diverse high\-probability regions in the teacher mixture, which is important when different teachers contribute distinct behaviors\. In contrast, reverse KL is mode\-seeking and may concentrate on a subset of dominant teacher modes, potentially losing specialized capabilities from other teachers\. Thus, forward KL is preferable in this setting, although this single\-backbone result does not imply that reverse KL is universally inferior\.

![Refer to caption](https://arxiv.org/html/2608.03092v1/x4.png)Figure 3:Top\-16 teacher probability mass during Qwen2\.5\-7B safe\-alignment training; light and dark curves show the raw and smoothed values\.Table 4:Top\-κ\\kappasupport and KL\-direction ablation on Qwen2\.5\-7B safe alignment\. Useful and Harmless are averaged across HH\-RLHF, PKU\-SafeRLHF, and Alpaca; Overall is their mean\.

## 5Conclusion

We identified a reward\-density imbalance in multi\-reward reinforcement learning: even after per\-reward normalization, sparse rewards may provide too little within\-group variation to effectively influence a shared policy\. To overcome this limitation, we introduced SMOPD, which decouples capability acquisition from reward balancing by training reward\-specialized teachers and merging their capabilities into a single student through online policy distillation with a balanced task anchor\. Across complementary rewards and conflicting rewards settings, SMOPD consistently improves over GDPO on 1\.5B, 3B, and 7B backbones\. These results highlight the effectiveness of our method in multi\-reward optimization: reward\-specific specialization preserves complementary capabilities, while subsequent merging produces a balanced final policy without over\-prioritizing any single objective\.

## References

## Appendix ATraining and Evaluation Details

All models are trained with the verl framework using Ray\-based distributed training\. Table[6](https://arxiv.org/html/2608.03092#A1.T6)lists the configuration of the accuracy\+format \(RLLA\) experiments and Table[6](https://arxiv.org/html/2608.03092#A1.T6)that of the safe\-alignment experiments\. Within each setting, GDPO, the teachers, and the SMOPD students share the same recipe, differing only in the GDPO reward weights \(and, for SMOPD, the added distillation loss\); GD2PO changes only the advantage estimator, as detailed below\. The RLLA runs use 150 steps for Qwen2\.5\-1\.5B and 100 for Qwen2\.5\-3B; safe\-alignment runs use 100 steps for the 3B backbones and 66 steps for Qwen2\.5\-7B\. The confidence\-failure gates \(Sec\.[E](https://arxiv.org/html/2608.03092#A5)\) use weight floorϕ=0\.2\\phi=0\.2, exponentsγ=η=1\\gamma=\\eta=1, confidence floorc0=0\.05c\_\{0\}=0\.05, blend strengthβ=0\.5\\beta=0\.5, andϵ=10−6\\epsilon=10^\{\-6\}in all settings\.

#### GD2PO baseline\.

The GD2PO rows in Tables[1](https://arxiv.org/html/2608.03092#S3.T1)and[2](https://arxiv.org/html/2608.03092#S3.T2)use the hard conflict\-filtering variant of GD2PO\(liu2026gd2po\)\. Starting from GDPO’s separately group\-normalized advantage for each reward, GD2PO\-Hard removes a rollout whenever its nonzero reward\-wise scalar advantages contain opposing signs\. It then scales each prompt group’s surviving advantages by that group’s retained\-rollout fraction and whitens over retained response tokens only\. Reward weights are equal, and every other model, data, rollout, and optimization setting is identical to the corresponding GDPO run\.

Table 5:Training and evaluation configuration for the accuracy \+ format experiments \(both backbones\)\.
Table 6:Training and evaluation configuration for the safe\-alignment experiments \(all three backbones\)\.

## Appendix BWhy the Anchor Breaks the Teacher Ceiling

This supplementary material formalizes the claim of Section[4\.3](https://arxiv.org/html/2608.03092#S4.SS3.SSS0.Px1)that the two training signals of Eq\. equation[9](https://arxiv.org/html/2608.03092#S3.E9)play distinct, complementary roles\. The forward\-KL OPD loss can at best reproduce the teacher mixture, whereas the anchor’s policy gradient is the only term that remains active at that point\. Moreover, on a sparse reward, the anchor becomes informative only after OPD has lifted the student’s success rate\. Throughout, we treat the frozen teacher mixture of Eq\. equation[5](https://arxiv.org/html/2608.03092#S3.E5)as a fixed conditional distributionq\(⋅∣s\)q\(\\cdot\\mid s\), writedθd\_\{\\theta\}for the state \(prefix\) distribution induced by the student’s own rollouts, and ignore PPO\-style clipping, which is inactive at the on\-policy point where the importance ratio equals one\.

#### Pure OPD is capped at the teacher mixture\.

The on\-policy distillation objective of Eq\. equation[7](https://arxiv.org/html/2608.03092#S3.E7)is

ℒOPD\(θ\)=𝔼s∼dθ\[KL\(q\(⋅∣s\)∥pθ\(⋅∣s\)\)\]\.\\mathcal\{L\}\_\{\\text\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{s\\sim d\_\{\\theta\}\}\\\!\\left\[\\operatorname\{KL\}\\\!\\big\(q\(\\cdot\\mid s\)\\,\\big\\\|\\,p\_\{\\theta\}\(\\cdot\\mid s\)\\big\)\\right\]\.\(10\)SinceKL⁡\(q∥pθ\)≥0\\operatorname\{KL\}\(q\\,\\\|\\,p\_\{\\theta\}\)\\geq 0with equality iffpθ\(⋅∣s\)=q\(⋅∣s\)p\_\{\\theta\}\(\\cdot\\mid s\)=q\(\\cdot\\mid s\)on the support ofqq, any student matching the mixture on its own visited states attains the global minimumℒOPD=0\\mathcal\{L\}\_\{\\text\{OPD\}\}=0\. Moreover, this minimum is a stationary point: the pointwise gradient is

∇θKL⁡\(q∥pθ\)=−∑vq​\(v∣s\)​∇θlog⁡pθ​\(v∣s\),\\nabla\_\{\\theta\}\\operatorname\{KL\}\\big\(q\\,\\\|\\,p\_\{\\theta\}\\big\)=\-\\sum\_\{v\}q\(v\\mid s\)\\,\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(v\\mid s\),\(11\)which atpθ=qp\_\{\\theta\}=qequals−∑v∇θpθ​\(v∣s\)=−∇θ1=0\-\\sum\_\{v\}\\nabla\_\{\\theta\}\\,p\_\{\\theta\}\(v\\mid s\)=\-\\nabla\_\{\\theta\}1=0; the contribution of∇θdθ\\nabla\_\{\\theta\}d\_\{\\theta\}vanishes as well because the integrand is pointwise zero at the minimum\. Gradient descent onℒOPD\\mathcal\{L\}\_\{\\text\{OPD\}\}alone therefore terminates at the mixture policy: the student inherits the teachers’ behavior but receives*no*signal that would push its expected task reward above that of the mixture\. This is the imitation ceiling observed in Table[3](https://arxiv.org/html/2608.03092#S4.T3), where the OPD\-only student reaches5\.5445\.544Overall, barely above the teachers it imitates \(5\.5005\.500/5\.5225\.522\)\.

#### The anchor’s gradient survives at the ceiling\.

The anchor is a policy\-gradient term driven by the GDPO advantageA^GDPO\\hat\{A\}^\{\\text\{GDPO\}\}of Eq\. equation[3](https://arxiv.org/html/2608.03092#S3.E3):

−∇θℒanchor=𝔼o∼πθ​\[A^GDPO​\(o\)​∇θlog⁡πθ​\(o\)\],\-\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{anchor\}\}=\\mathbb\{E\}\_\{o\\sim\\pi\_\{\\theta\}\}\\\!\\left\[\\hat\{A\}^\{\\text\{GDPO\}\}\(o\)\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(o\)\\right\],\(12\)an ascent direction on the group\-normalized task reward\. Its stationary points are local optima of the reward, not matches to any teacher; in particular, the mixtureqqis not a stationary point ofℒanchor\\mathcal\{L\}\_\{\\text\{anchor\}\}unlessqqalready locally maximizes the task reward\. Hence at the OPD fixed pointpθ=qp\_\{\\theta\}=q,

∇θℒtotal=∇θℒanchor\+λ⋅0=∇θℒanchor≠0\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{total\}\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{anchor\}\}\+\\lambda\\cdot 0=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{anchor\}\}\\neq 0\(13\)in general, so optimization of Eq\. equation[9](https://arxiv.org/html/2608.03092#S3.E9)continues*through*the imitation ceiling in the direction that increases task reward—exactly the\+0\.095\+0\.095Overall lift of SMOPD over its anchor\-free counterpart in Table[3](https://arxiv.org/html/2608.03092#S4.T3)\.

#### Under a sparse reward, OPD is what activates the anchor\.

Why not rely on the anchor alone, then? For a binary reward with per\-rollout success probabilitypθsuccp^\{\\text\{succ\}\}\_\{\\theta\}under the current policy, a group ofGGi\.i\.d\. rollouts receives identical rewards on that dimension with probability\(pθsucc\)G\+\(1−pθsucc\)G\(p^\{\\text\{succ\}\}\_\{\\theta\}\)^\{G\}\+\(1\-p^\{\\text\{succ\}\}\_\{\\theta\}\)^\{G\}, and by Eq\. equation[1](https://arxiv.org/html/2608.03092#S3.E1)such a degenerate group contributes exactly zero advantage on that dimension\. The anchor’s expected signal on the sparse dimension is therefore proportional to the probability of a*mixed*group,

1−\(pθsucc\)G−\(1−pθsucc\)G≈G​pθsucc\(pθsucc→0\),1\-\(p^\{\\text\{succ\}\}\_\{\\theta\}\)^\{G\}\-\(1\-p^\{\\text\{succ\}\}\_\{\\theta\}\)^\{G\}\\;\\approx\\;G\\,p^\{\\text\{succ\}\}\_\{\\theta\}\\quad\(p^\{\\text\{succ\}\}\_\{\\theta\}\\to 0\),\(14\)which vanishes linearly with the success rate: nearpθsucc≈0p^\{\\text\{succ\}\}\_\{\\theta\}\\approx 0the anchor is inert on exactly the dimension that needs it most\. The OPD gradient of Eq\. equation[11](https://arxiv.org/html/2608.03092#A2.E11), by contrast, is*dense*: it is nonzero at every position where the student deviates from the mixture, independent of within\-group reward variance, and the format teacher concentrates its mass precisely on the wrapper tokens the student is missing\. Distillation therefore raisespθsuccp^\{\\text\{succ\}\}\_\{\\theta\}rapidly; once the success rate is bounded away from0and11, mixed groups occur with constant probability, the sparse dimension’s advantage becomes non\-degenerate, and the anchor’s ascent direction of Eq\. equation[12](https://arxiv.org/html/2608.03092#A2.E12)takes over\.

In summary, the two losses are complementary by construction: OPD supplies the dense, reward\-independent gradient that carries the student to the teachers’ level and activates the sparse dimension’s group signal, while the anchor is the only term whose gradient survives at the imitation ceiling—and is thus what pushes the student*beyond*its teachers\.

## Appendix CSampled\-Token Reverse\-KL Estimator

Section[4\.3](https://arxiv.org/html/2608.03092#S4.SS3.SSS0.Px1)compares forward KL against reverse KL while keeping uniform teacher mixing, the GDPO anchor, the training recipe, and the evaluation checkpoint fixed\. Because the reverse direction places the student in the first argument, we implement it in its natural on\-policy, sampled\-token form\(gu2024minillm;lu2025onpolicy\)\. For each rollout tokenyt∼πoldy\_\{t\}\\sim\\pi\_\{\\mathrm\{old\}\}, every teacher returns the scalar log\-probability of that token and the mixture is formed exactly:

log⁡qmix​\(yt\)=logsumexpm⁡\(log⁡αm\+log⁡pπm​\(yt\)\),\\log q\_\{\\mathrm\{mix\}\}\(y\_\{t\}\)=\\operatorname\{logsumexp\}\_\{m\}\\\!\\left\(\\log\\alpha\_\{m\}\+\\log p\_\{\\pi\_\{m\}\}\(y\_\{t\}\)\\right\),\(15\)where conditioning on the prefixsts\_\{t\}is implicit\. LetDrev​\(θ\):=DKL​\(πθ∥qmix\)D\_\{\\mathrm\{rev\}\}\(\\theta\):=D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|q\_\{\\mathrm\{mix\}\}\)\. The desired reverse divergence and its score\-function gradient\(williams1992reinforce\)are

Drev​\(θ\)\\displaystyle D\_\{\\mathrm\{rev\}\}\(\\theta\)=𝔼y∼πθ​\[log⁡πθ​\(y\)−log⁡qmix​\(y\)\],\\displaystyle=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\\!\\left\[\\log\\pi\_\{\\theta\}\(y\)\-\\log q\_\{\\mathrm\{mix\}\}\(y\)\\right\],∇θDrev​\(θ\)\\displaystyle\\nabla\_\{\\theta\}D\_\{\\mathrm\{rev\}\}\(\\theta\)=𝔼y∼πθ​\[δθ​\(y\)​∇θlog⁡πθ​\(y\)\],\\displaystyle=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\\!\\left\[\\delta\_\{\\theta\}\(y\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\)\\right\],δθ​\(y\)\\displaystyle\\delta\_\{\\theta\}\(y\)=log⁡πθ​\(y\)−log⁡qmix​\(y\)\.\\displaystyle=\\log\\pi\_\{\\theta\}\(y\)\-\\log q\_\{\\mathrm\{mix\}\}\(y\)\.\(16\)Accordingly, we detach the sampled log\-ratio and use it as a token\-level advantage in a clipped policy\-ratio update:

k1​\(yt\)\\displaystyle k\_\{1\}\(y\_\{t\}\)=log⁡πold​\(yt\)−log⁡qmix​\(yt\),\\displaystyle=\\log\\pi\_\{\\mathrm\{old\}\}\(y\_\{t\}\)\-\\log q\_\{\\mathrm\{mix\}\}\(y\_\{t\}\),At\\displaystyle A\_\{t\}=−stopgrad⁡\[k1​\(yt\)\],\\displaystyle=\-\\operatorname\{stopgrad\}\\\!\\left\[k\_\{1\}\(y\_\{t\}\)\\right\],ρt\\displaystyle\\rho\_\{t\}=πθ​\(yt\)πold​\(yt\),\\displaystyle=\\frac\{\\pi\_\{\\theta\}\(y\_\{t\}\)\}\{\\pi\_\{\\mathrm\{old\}\}\(y\_\{t\}\)\},ℒrev​\-​PG\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{rev\\text\{\-\}PG\}\}=−𝔼t​\[clip⁡\(ρt,1−ϵ,1\+ϵ\)​At\]\.\\displaystyle=\-\\mathbb\{E\}\_\{t\}\\\!\\left\[\\operatorname\{clip\}\(\\rho\_\{t\},1\-\\epsilon,1\+\\epsilon\)A\_\{t\}\\right\]\.\(17\)At the on\-policy pointπθ=πold\\pi\_\{\\theta\}=\\pi\_\{\\mathrm\{old\}\}, the expected gradient of Eq\. equation[17](https://arxiv.org/html/2608.03092#A3.E17)matches Eq\. equation[16](https://arxiv.org/html/2608.03092#A3.E16)\.

This detached policy\-gradient construction is essential\. Although a sampledk3k\_\{3\}scalar\(schulman2020kl\)has the reverse\-KL value in expectation, directly differentiating it while treating the rollout sampling distribution as fixed does not recover the reverse\-KL gradient; it therefore cannot serve as a valid direction ablation\. We exclude that invalid control and report only the sampled\-token estimator in Table[4](https://arxiv.org/html/2608.03092#S4.T4)\. The sampled\-token reverse\-KL update remains below forward KL in this setting, but the comparison does not establish that reverse KL is universally inferior\.

## Appendix DAblation: Sensitivity to the Priority Profile

Table 7:*Moderate*\-skew ablation on Qwen2\.5\-3B: teachers use softened GDPO priority profiles rather than the aggressive skew of the main results in Table[1](https://arxiv.org/html/2608.03092#S3.T1)\. API\-Bank was not run for these moderate checkpoints and is therefore omitted\. The GDPO baseline is reproduced for reference;boldmarks the moderate\-profile SMOPD result\.Our main experiments specialize teachers with aggressive priority profiles \(\[0\.9,0\.1\]/\[0\.1,0\.9\]\[0\.9,0\.1\]/\[0\.1,0\.9\]\)\. Here we ablate the*strength*of that skew, re\-running the entire acc\+format pipeline on Qwen2\.5\-3B with softened*moderate*weights \(\[0\.7,0\.3\]\[0\.7,0\.3\]accuracy\-heavy,\[0\.3,0\.7\]\[0\.3,0\.7\]format\-heavy\); Table[7](https://arxiv.org/html/2608.03092#A4.T7)reports the results\. The moderate weights yield two competitive, complementary teachers \(94\.4%94\.4\\%/97\.5%97\.5\\%format\), and merging works at least as well as under aggressive weights: moderate SMOPD attains2\.7572\.757—the best result among all 3B acc\+format runs—beating both of its own teachers \(2\.7072\.707/2\.7092\.709\), the GDPO baseline \(2\.7282\.728\), and the aggressive\-profile student \(2\.7382\.738\)\. The advantage of SMOPD is thus robust to how sharply the rewards are skewed: as long as the teachers remain complementary, merging surpasses the scalarized baseline, and the softer skew performs, if anything, marginally better\.

## Appendix EAdaptive Teacher\-Mixing Gates

SMOPD’s default mixes teachers with*uniform*weightsαm=1/M\\alpha\_\{m\}=1/M\(Section[3\.2](https://arxiv.org/html/2608.03092#S3.SS2)\)\. Here we ask whether a more adaptive, input\-dependent gate could do better by routing more weight to the teacher that is most relevant at each sequence or token, and we define two*confidence\-failure*gates that steer the uniform prior1/M1/Musing three signals derived from the teachers and rewards: each teacher’s*confidence*\(how peaked its top\-κ\\kappadistribution is\), the student’s*reward failure*on that teacher’s target dimension, and inter\-teacher*disagreement*\.

#### Signals\.

At tokentt, teachermm’s confidence combines the normalized negative entropy and the top\-two margin of its \(top\-κ\\kappa, renormalized\) distributionpπm\(⋅∣st\)p\_\{\\pi\_\{m\}\}\(\\cdot\\mid s\_\{t\}\):

cm,t=12​\(1−H\[pπm\(⋅∣st\)\]log⁡κ\)\+12​\(pπm\(1\)​\(st\)−pπm\(2\)​\(st\)\),c\_\{m,t\}=\\tfrac\{1\}\{2\}\\Big\(1\-\\tfrac\{H\\\!\\left\[p\_\{\\pi\_\{m\}\}\(\\cdot\\mid s\_\{t\}\)\\right\]\}\{\\log\\kappa\}\\Big\)\+\\tfrac\{1\}\{2\}\\big\(p\_\{\\pi\_\{m\}\}^\{\(1\)\}\(s\_\{t\}\)\-p\_\{\\pi\_\{m\}\}^\{\(2\)\}\(s\_\{t\}\)\\big\),\(18\)whereH​\[⋅\]H\[\\cdot\]is Shannon entropy andp\(1\)≥p\(2\)p^\{\(1\)\}\\\!\\geq p^\{\(2\)\}are the two largest probabilities\. Reward failure uses the response’s rewardrmr\_\{m\}on teachermm’s dimension, min–max normalized to\[0,1\]\[0,1\]:

fm=1−clamp⁡\(rm−rminrmax−rmin,0,1\),f\_\{m\}=1\-\\operatorname\{clamp\}\\\!\\Big\(\\tfrac\{r\_\{m\}\-r\_\{\\min\}\}\{r\_\{\\max\}\-r\_\{\\min\}\},\\,0,\\,1\\Big\),\(19\)where\[rmin,rmax\]\[r\_\{\\min\},r\_\{\\max\}\]is the reward range of that dimension; a teacher is thus upweighted exactly where the student is failing its reward\. Disagreementdt∈\[0,1\]d\_\{t\}\\in\[0,1\]is the fraction of teacher pairs whose top\-1 tokens differ atsts\_\{t\}\.

#### Sequence\-level gate\.

Averagingcm,tc\_\{m,t\}over the response mask givesc¯m\\bar\{c\}\_\{m\}, and the raw weight combines the three signals over the uniform prior:

α~m=1M​\(c¯m\+c0\)η​\(fm\+ϵ\)γ,αm=α~m∑jα~j\.\\tilde\{\\alpha\}\_\{m\}=\\tfrac\{1\}\{M\}\\,\(\\bar\{c\}\_\{m\}\+c\_\{0\}\)^\{\\eta\}\\,\(f\_\{m\}\+\\epsilon\)^\{\\gamma\},\\quad\\alpha\_\{m\}=\\frac\{\\tilde\{\\alpha\}\_\{m\}\}\{\\sum\_\{j\}\\tilde\{\\alpha\}\_\{j\}\}\.\(20\)Averaging disagreement tod¯\\bar\{d\}, we blend back toward the prior and apply a floorϕ\\phiso no teacher is silenced:

αm\\displaystyle\\alpha\_\{m\}←Φϕ​\[\(1−β​d¯\)​αm\+β​d¯⋅1M\],\\displaystyle\\leftarrow\\Phi\_\{\\phi\}\\\!\\left\[\(1\-\\beta\\bar\{d\}\)\\,\\alpha\_\{m\}\+\\beta\\bar\{d\}\\cdot\\tfrac\{1\}\{M\}\\right\],\(21\)Φϕ​\[x\]m\\displaystyle\\Phi\_\{\\phi\}\[x\]\_\{m\}=ϕ\+\(1−ϕ​M\)​xm∑jxj\.\\displaystyle=\\phi\+\(1\-\\phi M\)\\,\\tfrac\{x\_\{m\}\}\{\\sum\_\{j\}x\_\{j\}\}\.

#### Token\-level gate\.

The same construction is applied*per position*:cm,tc\_\{m,t\}anddtd\_\{t\}are used directly \(no averaging\), yielding position\-specific weightsαm,t\\alpha\_\{m,t\}that let different teachers dominate at different tokens\. Reward failurefmf\_\{m\}remains sequence\-level \(one reward per response\)\. Gate hyperparameters are fixed across all settings and listed in Sec\.[A](https://arxiv.org/html/2608.03092#A1)\.

#### Results\.

Tables[8](https://arxiv.org/html/2608.03092#A5.T8)and[9](https://arxiv.org/html/2608.03092#A5.T9)compare the two gates against SMOPD’s uniform mixing on all five settings\. The 1\.5B RLLA block uses the aggressive main profile, while the 3B RLLA block uses the moderate\-profile ablation of Table[7](https://arxiv.org/html/2608.03092#A4.T7)\. Two findings stand out\. First,the three variants stay within a narrow band: on RLLA Mean the spread never exceeds0\.060\.06, and on safety Overall it never exceeds0\.020\.02\. Second,uniform mixing is the strongest RLLA rule in both displayed profile settings: it reaches 2\.740 at 1\.5B and 2\.757 in the moderate 3B ablation, compared with 2\.684/2\.690 and 2\.733/2\.730 for the sequence/token gates\. Token\-level routing wins only on Qwen2\.5\-7B safe alignment \(5\.651 vs\. 5\.646\), while uniform ties or leads in the remaining safety settings\. The adaptive mechanisms therefore add no consistent benefit over the parameter\-free mixture, motivating uniform mixing as SMOPD’s default\.

Table 8:Teacher\-mixing comparison on the accuracy \+ format setting\. Rows are SMOPD with uniform mixing \(our main method\) and the two confidence\-failure gates\. The 1\.5B rows use aggressive teachers; the 3B rows use the moderate teachers from the ablation in Table[7](https://arxiv.org/html/2608.03092#A4.T7), not the aggressive\-profile 3B checkpoints in the main table\. Per column and per backbone,boldmarks the best andunderlinethe second best among the three mixing rules \(ties share the rank\)\.Table 9:Teacher\-mixing comparison on safe alignment\. Rows are SMOPD with uniform mixing \(our main method\) and the two confidence\-failure gates\. Per column and per backbone,boldmarks the best andunderlinethe second best among the three mixing rules \(ties share the rank\)\.

## Appendix FAPI\-Bank LLM\-Judge Protocol and Original Exact\-Match Analysis

The main text reports API\-Bank under a semantic LLM\-judge metric rather than the benchmark’s original strict matcher\. Figure[4](https://arxiv.org/html/2608.03092#A6.F4)summarizes the complete evaluation path, and Figure[5](https://arxiv.org/html/2608.03092#A6.F5)shows the verbatim judge prompt\. For every saved model response, we first apply the original exact matcher\. Exact successes are retained as correct; only exact failures are sent to the judge \(qwen3\.7\-plus, one query per failed item\), together with the recent dialogue turns, the gold tool call, and the predicted call\. The judge accepts a failure only when it uses the correct tool and its arguments are functionally equivalent to the requested call\. Benign casing or formatting differences and harmless optional parameters may therefore be rescued, whereas wrong tools, missing required fields, unsupported values, or outcome\-changing arguments remain incorrect\. No model is re\-run\.

The final metric is\(Nexact\+Nrescued\)/Ntotal\(N\_\{\\mathrm\{exact\}\}\+N\_\{\\mathrm\{rescued\}\}\)/N\_\{\\mathrm\{total\}\}, computed separately for L1/L2/L3 and as micro\-accuracy over all 597 items\. Under this metric \(Table[1](https://arxiv.org/html/2608.03092#S3.T1)\), SMOPD obtains the highest Overall score at both scales:87\.10%87\.10\\%at 1\.5B and77\.39%77\.39\\%at 3B, respectively3\.523\.52and1\.011\.01percentage points above the strongest scalarized baseline\.

![Refer to caption](https://arxiv.org/html/2608.03092v1/x5.png)Figure 4:API\-Bank LLM\-judge evaluation pipeline\. The original exact matcher is applied first, and its successes remain correct\. Only failed exact matches undergo semantic review using the saved dialogue, reference call, and prediction\. A semantically equivalent call is rescued; a functionally different call remains incorrect\. The final per\-level and Overall scores combine original exact successes with judge\-rescued failures\.You are a strict evaluator of tool\-calling correctness for the API\-Bank benchmark\.You are given a multi\-turn dialogue, the GROUND\-TRUTH tool call that correctly solves the user’s final need, and a MODEL’s predicted tool call\. Decide whether the model’s prediction is FUNCTIONALLY CORRECT: it must call the right tool AND supply parameters that faithfully fulfill the user’s request as expressed in the dialogue, i\.e\. be semantically equivalent to the ground truth\.Judge CORRECT \(Yes\) when the prediction differs from the ground truth ONLY in ways that do not change the outcome, for example:•Case, whitespace, or punctuation differences in parameter values\.•Synonyms / equivalent phrasings that refer to the same entity or value\.•Reasonable interpretations of time boundaries that still capture the user’s stated intent \(e\.g\. “up to March 12th” as 2023\-03\-12 23:59:59 vs 2023\-03\-12 00:00:00\)\.•Equivalent formatting of the same value \(e\.g\. 5 vs “5”, equivalent date formats\)\.•Extra optional parameters that are consistent with the dialogue and do not alter the core action\.Judge INCORRECT \(No\) when the prediction:•Calls a different or wrong tool, or one that does not fulfill the request\.•Omits a required parameter, or fills a required parameter with a wrong or contradictory value\.•Uses a value that changes the meaning or outcome \(different user, amount, target, time span, etc\.\)\.•Hallucinates values not supported by the dialogue\.Only the FINAL required tool call matters\. Base your decision on the user’s actual need in the dialogue, not on superficial string matching\. Be strict: do not pass genuinely wrong calls, but do not fail calls that are merely phrased or formatted differently\.\# Dialogue \(most recent turns\) \{dialogue\}\# Ground\-truth tool call \{gold\}\# Model predicted tool call\(s\) \{pred\}Respond in EXACTLY this format and nothing else: Reasoning: <one concise sentence\> Judgment: <Yes or No\>

Figure 5:Verbatim prompt of the API\-Bank LLM judge \(qwen3\.7\-plus\)\. The placeholders\{dialogue\},\{gold\}, and\{pred\}are filled with the recent dialogue turns, the reference tool call, and the model’s predicted call\(s\) of each exact\-match failure; the judge returns a one\-sentence rationale and a binary verdict\.Table 10:Representative API\-Bank items where the 3B model calls the correct tool but is scored wrong by exact name\-and\-parameter matching\. Each 3B output is semantically correct or more complete than the reference; the failure is purely a literal\-string mismatch\.#### Original exact\-match metric\.

The original API\-Bank scorer requires exact equality of both the function name and the*entire*parameter dictionary; any difference in casing, specificity, formatting, or additional fields counts as a failure\. Table[11](https://arxiv.org/html/2608.03092#A6.T11)preserves these strict scores for comparison with the original ToolRL/GD2PO evaluation protocol\. Unlike the LLM\-judge results in the main text, they primarily measure literal agreement with the reference call\.

MethodL1L2L3OverallQwen2\.5\-1\.5BGRPO Baseline62\.6653\.7348\.8558\.63GDPO Baseline67\.4256\.7247\.3361\.81GD2PO Baseline64\.6653\.7345\.0459\.13Accuracy Teacher \[0\.9,0\.1\]64\.9158\.2149\.6260\.80Format Teacher \[0\.1,0\.9\]72\.1862\.6938\.1763\.65SMOPD74\.4464\.1839\.6965\.66
MethodL1L2L3OverallQwen2\.5\-3BGRPO Baseline66\.6747\.7627\.4855\.95GDPO Baseline65\.4156\.7238\.1758\.46GD2PO Baseline66\.9250\.7541\.9859\.63Accuracy Teacher \[0\.9,0\.1\]67\.6752\.2430\.5357\.79Format Teacher \[0\.1,0\.9\]68\.6752\.2442\.7561\.14SMOPD67\.9249\.2532\.8258\.12

Table 11:Original strict API\-Bank exact\-match accuracy \(%\) on the same saved generations as Table[1](https://arxiv.org/html/2608.03092#S3.T1)\.Overallis micro\-accuracy over all 597 items \(399 L1, 67 L2, and 131 L3\)\. These are the original name\-and\-full\-parameter\-dictionary matching scores, not the semantic LLM\-judge metric used in the main text\. Per column and backbone,boldmarks the best andunderlinethe second best among the three scalarized baselines and SMOPD; single\-reward teachers are shown for context and excluded from ranking\.
#### Quantitative diagnosis of strict matching\.

On the 597 items shared by the 1\.5B and 3B SMOPD runs, the exact matcher counts 392 correct for 1\.5B and 347 for 3B\. Of the 66 items that 1\.5B passes but 3B fails,62are cases where 3B invokes the*correct*tool but its argument string does not match the reference literally, and only 4 are genuine refusals or clarification requests\. These disagreements concentrate on the simplest Level\-1 items \(38 of 66\), precisely where the target is a short literal string and extra model capability cannot help\. The lower exact\-match score of Qwen2\.5\-3B is therefore a literal\-matching artifact rather than a regression in tool\-calling ability: the more capable backbone tends to produce richer argument strings—more specific values, fuller context, or extra optional fields \(Table[10](https://arxiv.org/html/2608.03092#A6.T10)\)—which deviate from the short reference strings more often, so the strict matcher penalizes exactly the additional capability that the semantic judge rescues\.

#### Representative cases\.

Table[10](https://arxiv.org/html/2608.03092#A6.T10)lists 3B outputs that are semantically correct—often*more*complete or faithful to the user’s request—yet scored wrong by exact matching\.

We use the semantic judge in the main table because it measures functional equivalence rather than literal reproduction, while retaining Table[11](https://arxiv.org/html/2608.03092#A6.T11)for direct comparability with prior exact\-match reporting\. Together with BFCL, the judge results show that strict matching can understate functional tool\-use ability without changing the central within\-backbone comparison: at both scales, SMOPD attains the highest judged API\-Bank accuracy of all compared methods\.

Table 12:Full per\-category BFCL\-v4 accuracy \(%\) for the aggressive\-profile models of Table[1](https://arxiv.org/html/2608.03092#S3.T1)\.AST \(ours\)is the mean of the*Non\-Live*and*Live AST*block accuracies \(our main\-text BFCL metric\); P\.M\. abbreviates Parallel\-Multiple\. The full\-suiteOverallaverage is dominated by categories these single\-turn tool callers never see \(Multi\-Turn; Web\-Search and Memory return no score and are omitted\), motivating the AST metric used in the main text\.

## Appendix GFull BFCL\-v4 Category Breakdown

Table[12](https://arxiv.org/html/2608.03092#A6.T12)reports every BFCL\-v4 category for the aggressive\-profile models of Table[1](https://arxiv.org/html/2608.03092#S3.T1)\. It shows why the full\-suite*Overall*score \(last column, 15–23%\) is misleadingly low: it averages in categories our single\-turn tool models were never trained for—*Multi\-Turn*is0%0\\%for every model, and the Web\-Search/Memory categories return no score—while the models are in fact strong on the abstract\-syntax\-tree \(AST\) function\-calling categories they target \(76–84% non\-live, 60–72% live\)\. Our main\-textBFCL ASTmetric is the mean of the Non\-Live and Live AST columns\. At 3B every SMOPD variant matches or exceeds the GDPO baseline’s AST accuracy, whereas at 1\.5B the scalarized baselines remain strongest \(GD2PO73\.3%73\.3\\%vs\. SMOPD70\.7%70\.7\\%\)\.