CONCUR：用于多任务、多奖励强化学习的一致性辅助奖励

# CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Source: [https://arxiv.org/html/2610.36437](https://arxiv.org/html/2610.36437)
Jixuan LengSi SiFei WangCho\-Jui Hsieh††thanks:Work done while interning at Google\.Jiong ZhangHsiang\-Fu YuInderjit S\. DhillonAffiliation:Google

###### Abstract

When a Large Language Model \(LLM\) is trained with Reinforcement Learning \(RL\) on multiple tasks, a task’s primary reward can*gate*auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary\. However, group\-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary’s order, and at the primary’s ceiling only auxiliaries order responses\. Keeping gated auxiliaries may hurt some tasks, while dropping them may hurt others\. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary’s ceiling receive equal scores\. CONCUR\-T applies both conditions throughout training, and CONCUR\-A lifts the second for an auxiliary that agrees with the primary’s order more often than it reverses it among passing responses\. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped\.

## 1Introduction

Reinforcement Learning \(RL\) is now a standard way to improve the capabilities of Large Language Models \(LLMs\)\([Guo et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib21);[Lambert et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib5)\)\. One setup trains a single model on several tasks in one run\([Pardinas et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib46);[Ramesh et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib9)\), where each task can score a*response*with several*reward components*\. We study this multi\-task, multi\-reward setup, in which one component of each task is the*primary*that measures the task’s own success, such as the correctness of a math answer\. Some of the others are auxiliary components that the primary*gates*\. We follow the reward design of[Bai et al\. \(2026\)](https://arxiv.org/html/2610.36437#bib.bib7), in which a response*passes*if its primary is at least one half, and a*gated auxiliary*keeps its own score on a response that passes and cannot exceed the primary on one that fails\. Group\-relative advantage estimators treat a gated auxiliary like any other component\. For example, GDPO\([Liu et al\., 2026c](https://arxiv.org/html/2610.36437#bib.bib6)\)standardizes each component within a*group*of responses to the same prompt and sums the standardized scores at the*declared weights*specified by the reward design\.

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/concur.png)Figure 1:Left: GDPO standardizes reward components in a group and sums them at the declared weights, and the insets show where an auxiliary departs from its primary\. Right: CONCUR projects each gated auxiliary’s scores onto the closest ones that concur with the primary, and in CONCUR\-A earned trust decides whether a ceiling is tied\.What should an estimator do with a gated auxiliary? We train three LLMs with GDPO on a mixture of math, code, instruction following \(IF\), and writing, in which every task’s primary gates a length reward and writing’s primary also gates a reward\-model score \(RM\)\. We call a task together with one of its gated auxiliaries a*gated pair*, and the mixture has five\. Neither keeping nor dropping the gated auxiliaries is satisfactory\. As a*reference*, we run GDPO with every gated auxiliary at weight00\. GDPO at the declared weights scores well below the reference on math and below it on code, but above it on writing\. The left of Figure[1](https://arxiv.org/html/2610.36437#S1.F1)illustrates the two places where a gated auxiliary departs from its primary during training: against the primary’s order, where a response with a lower primary scores higher, and*at the ceiling*, where the primary equals its maximum and cannot separate responses\. We measure each as a share of the sum of an auxiliary’s squared standardized scores\. As Figure[2](https://arxiv.org/html/2610.36437#S3.F2)shows, the share against the order grows to more than half on writing, and the share at the ceiling is largest on math, whose binary primary scores correct responses equally\. At the ceiling the auxiliary acts alone\. In a group whose responses all reach it, the primary’s standardized scores are zero, while standardization scales the auxiliary’s raw differences, however small, to unit variance\.

The gate gives the primary priority over the auxiliaries that it gates, but it constrains only the responses that fail\. We propose CONCUR, an advantage estimator that extends this priority to every pair of responses in a group, as the right of Figure[1](https://arxiv.org/html/2610.36437#S1.F1)illustrates\. CONCUR does this with a projection: within each group, it replaces each gated auxiliary’s standardized scores with the closest scores that*concur*with the primary\. Scores concur if they satisfy two constraints\. The*order constraint*requires that no response with a higher primary scores lower, and the*ceiling constraint*, where it applies, requires that responses at the ceiling score alike\. When the ceiling constraint applies to a gated pair, we say that the pair’s ceiling is*tied*\. CONCUR\-T ties every gated pair’s ceiling throughout training, has no setting and no state, and is our default\. CONCUR\-A is the adaptive form, and we call its rule*earned trust*\. It lifts a gated pair’s tie while the auxiliary agrees with the primary’s order more often than it reverses it among the passing responses that the primary separates\. CONCUR uses no gradient and keeps every reward component at its declared weight\.

Our contributions are summarized as follows:

- •We analyze how a gated auxiliary enters group\-relative advantages, show that in training it departs from its primary both against the primary’s order and at its ceiling, and find that neither keeping nor dropping gated auxiliaries is satisfactory\.
- •We propose CONCUR, an advantage estimator that projects each gated auxiliary’s standardized scores onto the closest scores that concur with the primary, in a default and an adaptive variant\.
- •We prove that the projection never reverses the primary’s order within a group and that, with a binary primary, it keeps only the auxiliary’s difference between correct and wrong responses\.
- •We show that, on three models from two families and across math, code, instruction\-following, and writing benchmarks, both variants achieve higher scores on average than seven existing estimators and than when the gated auxiliaries are dropped\.

## 2Related Work

Multi\-reward RL\.Group\-relative estimators differ in where they normalize: GRPO standardizes the weighted sum within a group, GDPO and MO\-GRPO standardize each component before the sum, and RDPO maps each component to quantile scores\([Shao et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib1);[Liu et al\., 2026c](https://arxiv.org/html/2610.36437#bib.bib6);[Ichihara et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib60);[Bai et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib7)\)\. Others account for the correlation between components\([Gupta et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib22);[Hu et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib52)\), change the weights of losses, tasks, or reward components during training\([Kendall et al\., 2018](https://arxiv.org/html/2610.36437#bib.bib49);[Chen et al\., 2018](https://arxiv.org/html/2610.36437#bib.bib48);[Lu et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib20);[Ramesh et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib9);[He et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib8)\), or filter out responses whose per\-component advantages disagree in sign\([Liu et al\., 2026b](https://arxiv.org/html/2610.36437#bib.bib10)\)\. Two recent works drop the division by the standard deviation of the rewards in a group\([Kassenaar et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib56);[Wang, 2026](https://arxiv.org/html/2610.36437#bib.bib57)\), as Dr\. GRPO already does\([Liu et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib4)\)\. All these methods treat every reward component alike, whereas CONCUR changes only the gated auxiliaries’ scores in each response group\.

Priority among rewards\.A common reward design, called conditioning or gating, lets one reward count in full only when another one reaches a threshold\([Kimi Team, 2025](https://arxiv.org/html/2610.36437#bib.bib64);[Arora and Zanette, 2025](https://arxiv.org/html/2610.36437#bib.bib65);[Aggarwal and Welleck, 2025](https://arxiv.org/html/2610.36437#bib.bib66);[Song and Zheng, 2025](https://arxiv.org/html/2610.36437#bib.bib67);[Liu et al\., 2026c](https://arxiv.org/html/2610.36437#bib.bib6);[Bai et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib7)\)\. CONCUR takes such a gate as given and makes its priority hold for any two responses of a group\. Other designs let a second reward order the correct responses only\([Tan et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib44);[Peng et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib45);[Li et al\., 2026a](https://arxiv.org/html/2610.36437#bib.bib43);[Fang et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib53)\)\. On a binary primary, they make the opposite choice from CONCUR\. They keep the auxiliary’s differences among correct responses, which CONCUR’s ceiling constraint removes, and drop its difference between correct and wrong responses, which CONCUR keeps\. Appendix[A](https://arxiv.org/html/2610.36437#A1)discusses other ways to enforce a priority, such as constrained RL, lexicographic RL, and methods that act in gradient space\.

Order\-preserving projections\.Isotonic regression finds the values that respect a given order and are closest to the input values, and the pool\-adjacent\-violators algorithm computes it along a chain\([Ayer et al\., 1955](https://arxiv.org/html/2610.36437#bib.bib59)\)\. The most related work is POISE, which applies this projection to the score labels in the fixed training corpus of a robot reward model, so that they follow the order of pairwise preference labels\([Wang et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib58)\)\. CONCUR projects at every training step onto the order of the primary, which is another reward component of the same responses, and adds the ceiling constraint\.

## 3Preliminaries

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/gate.png)Figure 2:GDPO at the declared weights onQwen3\-4B\-Base, one line per gated pair\. The panels show the auxiliary’s cosine with its primary, its shares of energy against the order and at the ceiling from Equation[3](https://arxiv.org/html/2610.36437#S3.E3), as 10\-step running means, and the cumulative mean of the agreement of Equation[4](https://arxiv.org/html/2610.36437#S3.E4), which is not defined for Math’s binary primary\.### 3\.1Multi\-Task RL with Gated Rewards

The policyπθ\\pi\_\{\\theta\}samples a*group*ofGGresponses to each promptxxof taskt∈𝒯t\\in\\mathcal\{T\}\. Each*reward component*k∈𝒦tk\\in\\mathcal\{K\}\_\{t\}gives responseiia scoreri​k∈\[0,1\]r\_\{ik\}\\in\[0,1\]and has a*declared weight*wkw\_\{k\}\. One component of each task, indexed byjj, is its*primary*, which measures the task’s own success\. We writepi=ri​jp\_\{i\}=r\_\{ij\}for the primary score of responseii, and the primary*separates*two responsesiiandi′i^\{\\prime\}ifpi≠pi′p\_\{i\}\\neq p\_\{i^\{\\prime\}\}\. A response*passes*ifpi≥12p\_\{i\}\\geq\\frac\{1\}\{2\}and*fails*otherwise, and it is*at the ceiling*ifpi=1p\_\{i\}=1, which is the primary’s maximum\. The primary*gates*a set𝒜t⊂𝒦t∖\{j\}\\mathcal\{A\}\_\{t\}\\subset\\mathcal\{K\}\_\{t\}\\setminus\\\{j\\\}of*gated auxiliaries*:ri​k=min⁡\{si​k,pi\}r\_\{ik\}=\\min\\\{s\_\{ik\},p\_\{i\}\\\}ifpi<12p\_\{i\}<\\frac\{1\}\{2\}andri​k=si​kr\_\{ik\}=s\_\{ik\}otherwise, wheresi​k∈\[0,1\]s\_\{ik\}\\in\[0,1\]is the auxiliary’s own score\. The components, their gates, and the declared weights form the*reward design*\.

We mix equal shares of math, code, IF, and writing prompts, with every declared weight11\. The primary is binary correctness for math, the unit\-test pass rate for code, and a weighted share of the criteria that a response meets for IF and writing\([Li et al\., 2026b](https://arxiv.org/html/2610.36437#bib.bib23)\)\. A program checks IF’s criteria\. Writing’s criteria form a rubric, and a judge model decides which of them a response meets\. Every task’s primary gates a*length reward*, which is full for a response no longer than a per\-prompt reference length and falls quadratically to zero at twice that length, and writing’s primary also gates a*reward\-model score*\(RM\)\([Liu et al\., 2026a](https://arxiv.org/html/2610.36437#bib.bib18)\), rescaled to\[0,1\]\[0,1\]\. Each gated auxiliary and its task together form a*gated pair*, of which the mixture has five: Math, Code, IF, and Writing are each a task with its own length reward, and Writing \(RM\) is writing with its RM\. IF also has one component that is not gated, which is the judge’s score on one further criterion\. Appendix[B\.1](https://arxiv.org/html/2610.36437#A2.SS1)shows every component’s role, and Appendix[D\.1](https://arxiv.org/html/2610.36437#A4.SS1)gives the data and rewards\.

Group\-relative advantages\.GDPO extends GRPO by standardizing each reward component within a group before taking the weighted sum:

zi​k\\displaystyle z\_\{ik\}=ri​k−r¯kdk,ui=∑k∈𝒦twk​zi​k,\\displaystyle=\\frac\{r\_\{ik\}\-\\bar\{r\}\_\{k\}\}\{d\_\{k\}\},\\qquad u\_\{i\}=\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\}w\_\{k\}z\_\{ik\},\(1\)Ai\\displaystyle A\_\{i\}=𝒲​\(u\)i=\(ui−u¯\)/σ^u,\\displaystyle=\\mathcal\{W\}\(u\)\_\{i\}=\(u\_\{i\}\-\\bar\{u\}\)/\\hat\{\\sigma\}\_\{u\},\(2\)wherer¯k\\bar\{r\}\_\{k\}anddkd\_\{k\}are the group’s mean and sample standard deviation,zi​k=0z\_\{ik\}=0ifdk=0d\_\{k\}=0, and the*batch whitening*𝒲\\mathcal\{W\}, which preserves the order ofuu, computesu¯\\bar\{u\}andσ^u\\hat\{\\sigma\}\_\{u\}over all tasks\. We callzi​kz\_\{ik\}a*standardized score*anduiu\_\{i\}a*summed score*, and we writezk=\(zi​k\)i∈ℝGz\_\{k\}=\(z\_\{ik\}\)\_\{i\}\\in\\mathbb\{R\}^\{G\}for the group as a whole\.

### 3\.2A Motivating Experiment

We trainQwen3\-4B\-Base\([Yang et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib19)\)with GDPO at the declared weights and, on every training batch, compare each gated auxiliary’s standardized scoreszzwith its primary’s\. Figure[2](https://arxiv.org/html/2610.36437#S3.F2)plots the comparison, and Table[5](https://arxiv.org/html/2610.36437#A4.T5)in the appendix gives its values\. We also train GDPO with every gated auxiliary at weight00, which is a different reward design\. We call this run the*reference*and label it GDPO \(no aux\.\)\. An auxiliary’s*energy*in a group is∥z∥2\\lVert z\\rVert^\{2\}\. Letz^≤\\hat\{z\}^\{\\leq\}be the scores closest tozzin which no response with a higher primary scores lower, and letz^=\\hat\{z\}^\{=\}be the closest scores that are also equal on the responses at the ceiling\. The energy of every nonzerozzthen splits into exactly three nonnegative shares that sum to one \(more in Section[5](https://arxiv.org/html/2610.36437#S5)\):

against the order\\displaystyle\\text\{against the order\}=1−∥z^≤∥2/∥z∥2,\\displaystyle=1\-\\lVert\\hat\{z\}^\{\\leq\}\\rVert^\{2\}/\\lVert z\\rVert^\{2\},\(3\)at the ceiling\\displaystyle\\text\{at the ceiling\}=\(∥z^≤∥2−∥z^=∥2\)/∥z∥2,\\displaystyle=\\bigl\(\\lVert\\hat\{z\}^\{\\leq\}\\rVert^\{2\}\-\\lVert\\hat\{z\}^\{=\}\\rVert^\{2\}\\bigr\)/\\lVert z\\rVert^\{2\},kept\\displaystyle\\text\{kept\}=∥z^=∥2/∥z∥2\.\\displaystyle=\\lVert\\hat\{z\}^\{=\}\\rVert^\{2\}/\\lVert z\\rVert^\{2\}\.We also measure whether an auxiliary orders passing responses as its primary does\. Take a group and a gated pair of taskttwith auxiliarykk, and letP=\{i:pi≥12\}P=\\\{i:p\_\{i\}\\geq\\frac\{1\}\{2\}\\\}be the group’s passing responses\. If the primary separates at least two of them, the group gives a*reading*of the pair’s*agreement*,

ax,k=∑\(i,i′\)∈P2sign⁡\(pi−pi′\)​sign⁡\(ri​k−ri′​k\)\|\{\(i,i′\)∈P2:pi≠pi′\}\|\.a\_\{x,k\}=\\frac\{\\sum\_\{\(i,i^\{\\prime\}\)\\in P^\{2\}\}\\operatorname\{sign\}\(p\_\{i\}\-p\_\{i^\{\\prime\}\}\)\\operatorname\{sign\}\(r\_\{ik\}\-r\_\{i^\{\\prime\}k\}\)\}\{\\lvert\\\{\(i,i^\{\\prime\}\)\\in P^\{2\}:p\_\{i\}\\neq p\_\{i^\{\\prime\}\}\\\}\\rvert\}\.\(4\)Among the pairs inPPthat the primary separates, it is the share that the auxiliary orders as the primary does minus the share that it orders the other way\.

\(O1\) At the start of training, a gated auxiliary mostly repeats its primary on math, code, and IF\.In the first panel, the cosine between an auxiliary’s standardized scores and its primary’s is at least 0\.75 on Math, Code, and IF over the first 20 training steps\. Even a high cosine leaves room for a large effect on training\. On math, where this cosine is highest, the reference scores well above GDPO\.

\(O2\) An auxiliary departs from its primary in two places: against its order and at its ceiling\.On Math, the share against the order is zero by construction, because the gate sets a wrong response’s auxiliary to00\. It stays small on Code and IF, and on the two writing pairs it grows during training to more than half\. The share at the ceiling is largest on Math\. Over the last 20 training steps, a share 0\.16 of the math groups in the update have every response at the ceiling\. There the primary is constant, and the other components alone order the group\.

\(O3\) Dropping the gated auxiliaries raises the score on math and lowers it on writing\.On our evaluation benchmarks, the reference scores 7\.9 points above GDPO on math, above it on code as well, and 5\.3 points below it on writing \(more in Section[6\.2](https://arxiv.org/html/2610.36437#S6.SS2)\)\. Whether an auxiliary agrees with its primary varies by pair\. In Figure[2](https://arxiv.org/html/2610.36437#S3.F2), the agreement ends positive for writing’s RM and negative for writing’s length reward, each by a few hundredths\.

These observations motivate the two constraints of CONCUR: its order constraint removes the share against the order, and its ceiling constraint, wherever it applies, removes the share at the ceiling\.

## 4Concordant AuxiliaryRewards

Within each group, CONCUR replaces each gated auxiliary’s standardized scores with the closest scores that*concur*with the primary\. Scores concur if they satisfy the order constraint and, at the steps where the pair’s ceiling is*tied*, the ceiling constraint as well\. The*ceiling policy*decides when a pair’s ceiling is tied, and the two variants of CONCUR differ only in this policy\. CONCUR\-T uses the constant policy that keeps all ceilings tied, and we recommend it as the default variant\. The adaptive form CONCUR\-A lifts a pair’s tie while the pair’s auxiliary has*earned trust*\. The other components keep their standardized scores, and the sum of all scores at the declared weights is then whitened over the whole batch\. CONCUR needs nothing beyond the reward design\. It uses the gate as it is declared, without having to know what a given gated auxiliary rewards\. Algorithm[1](https://arxiv.org/html/2610.36437#alg1)in the appendix states the full computation for one batch\.

### 4\.1The Projection

Take one group, with primariesp∈\[0,1\]Gp\\in\[0,1\]^\{G\}\. The vectors of scoresv∈ℝGv\\in\\mathbb\{R\}^\{G\}that satisfy the order constraint form the*order cone*C≤​\(p\)=\{v∈ℝG:pi\>pi′⇒vi≥vi′\}C\_\{\\leq\}\(p\)=\\\{v\\in\\mathbb\{R\}^\{G\}:\\ p\_\{i\}\>p\_\{i^\{\\prime\}\}\\;\\Rightarrow\\;v\_\{i\}\\geq v\_\{i^\{\\prime\}\}\\\}\. The order constraint has no margin: any difference in the primary, however small, outranks the auxiliary\. At the ceiling, equal primaries are not an artifact of sampling, since a prompt that the policy solves puts its whole group at the ceiling, where the components other than the primary determine the entire order and each has unit variance\. The ceiling therefore gets its own constraint, andC=​\(p\)=\{v∈C≤​\(p\):pi=pi′=1⇒vi=vi′\}C\_\{=\}\(p\)=\\\{v\\in C\_\{\\leq\}\(p\):\\ p\_\{i\}=p\_\{i^\{\\prime\}\}=1\\;\\Rightarrow\\;v\_\{i\}=v\_\{i^\{\\prime\}\}\\\}is the cone of those vectors that satisfy both the order constraint and this one\.

CONCUR moves an auxiliary’s standardized scores as little as the constraints allow: for each gated auxiliaryk∈𝒜tk\\in\\mathcal\{A\}\_\{t\}and for each group, it takes the Euclidean projection of the group’s scoreszkz\_\{k\},

z^k=ΠC​\(zk\)=arg​minv∈C⁡∥v−zk∥2,\\hat\{z\}\_\{k\}=\\Pi\_\{C\}\(z\_\{k\}\)=\\argmin\_\{v\\in C\}\\lVert v\-z\_\{k\}\\rVert^\{2\},\(5\)whereC=C=​\(p\)C=C\_\{=\}\(p\)if the pair’s ceiling is tied at that training step andC=C≤​\(p\)C=C\_\{\\leq\}\(p\)if it is not, and the summed scoreuiu\_\{i\}then usesz^k\\hat\{z\}\_\{k\}in place ofzkz\_\{k\}:

ui\\displaystyle u\_\{i\}=∑k∈𝒦t∖𝒜twk​zi​k\+∑k∈𝒜twk​z^i​k,\\displaystyle=\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\\setminus\\mathcal\{A\}\_\{t\}\}w\_\{k\}z\_\{ik\}\+\\sum\_\{k\\in\\mathcal\{A\}\_\{t\}\}w\_\{k\}\\hat\{z\}\_\{ik\},\(6\)Ai\\displaystyle A\_\{i\}=𝒲​\(u\)i\.\\displaystyle=\\mathcal\{W\}\(u\)\_\{i\}\.The projection exists, is unique, and leaveszkz\_\{k\}unchanged exactly whenzkz\_\{k\}concurs \(more in Section[5](https://arxiv.org/html/2610.36437#S5)\)\. Its exact computation has two steps, which Appendix[B\.2](https://arxiv.org/html/2610.36437#A2.SS2)details\. First, if the ceiling is tied, the scores of the responses at the ceiling are replaced by their mean\. Then the pool\-adjacent\-violators algorithm of isotonic regression pools neighboring responses whose scores are out of order along the primary, giving each pool its mean\. The projected scores are not standardized again\. That would restore unit variance to what the projection leaves\.

### 4\.2Earned Trust

Tying a ceiling has a cost\. It removes all the credit that the auxiliary assigns among the responses at the ceiling, which can be most of a task’s passing responses\. CONCUR\-A restores this credit for a gated pair whose auxiliary orders passing responses as the primary does, which it measures in every group where the primary separates two of them\.

For a gated pair\(t,k\)\(t,k\), the*pooled agreement*a¯t,k\\bar\{a\}\_\{t,k\}is the mean of all readingsax,ka\_\{x,k\}of its agreement since the start of training, including those of the current batch, andset,k\\mathrm\{se\}\_\{t,k\}is its standard error\. A pair’s ceiling is tied by default, and it is not tied at a step ifa¯t,k\>2​set,k\\bar\{a\}\_\{t,k\}\>2\\,\\mathrm\{se\}\_\{t,k\}\. While its ceiling is not tied, we say that the pair’s auxiliary*has earned trust*\. A binary primary scores every passing response alike, so it never gives a reading\. Its ceiling stays tied throughout training, and the two variants compute the same summed scores on such a task\. With a graded primary, CONCUR\-A suits a reward design in which an auxiliary that agrees with the primary should keep ordering the responses at the ceiling, and CONCUR\-T suits one in which it should not\. Appendix[B\.3](https://arxiv.org/html/2610.36437#A2.SS3)details the rule and its assumption\.

## 5Analysis

Appendix[C](https://arxiv.org/html/2610.36437#A3)provides the derivations and the complete proofs of the two statements in this section\.

The ceiling\.Take a group whose responses are all at the ceiling\. The primary is constant there, sozi​j=0z\_\{ij\}=0for every responseii, while every componentkkwithdk\>0d\_\{k\}\>0has standardized scores of unit sample variance, whatever the size of its raw differences\. For example, supposeG=8G=8and a binary primary gates one auxiliary, both at weight11\. If all eight responses are correct and exactly one of them has a lower auxiliary score than the others, whether by0\.010\.01or by0\.50\.5, that response hasui=−7/8≈−2\.47u\_\{i\}=\-7/\\sqrt\{8\}\\approx\-2\.47\. If four responses are correct with a full auxiliary score and four are wrong, each wrong response hasui=−7/2≈−1\.87u\_\{i\}=\-\\sqrt\{7/2\}\\approx\-1\.87\. A difference of0\.010\.01in the auxiliary thus costs a correct response more summed score in the first group than a wrong answer costs in the second\. Under CONCUR with the ceiling tied, the gated auxiliary contributes nothing to that first group\.

###### Proposition 5\.1\(The projection\)\.

For primariespp, take any vectorz∈ℝGz\\in\\mathbb\{R\}^\{G\}andC∈\{C≤​\(p\),C=​\(p\)\}C\\in\\\{C\_\{\\leq\}\(p\),C\_\{=\}\(p\)\\\}\.

1. \(i\)*Existence\.*CCis a closed convex cone that contains the constant vectors, so the projectionz^=ΠC​\(z\)\\hat\{z\}=\\Pi\_\{C\}\(z\)ofzzexists and it is unique\.
2. \(ii\)*Computation\.*Averaging the scores at a tied ceiling and then pooling the adjacent violators along the primary computesz^\\hat\{z\}exactly\.
3. \(iii\)*Mean\.*∑iz^i=∑izi\\sum\_\{i\}\\hat\{z\}\_\{i\}=\\sum\_\{i\}z\_\{i\}\.
4. \(iv\)*Energy\.*∥z∥2=∥z^∥2\+∥z−z^∥2\\lVert z\\rVert^\{2\}=\\lVert\\hat\{z\}\\rVert^\{2\}\+\\lVert z\-\\hat\{z\}\\rVert^\{2\}, andz^=z\\hat\{z\}=zexactly whenz∈Cz\\in C\. Withz^≤=ΠC≤​\(p\)​\(z\)\\hat\{z\}^\{\\leq\}=\\Pi\_\{C\_\{\\leq\}\(p\)\}\(z\)andz^==ΠC=​\(p\)​\(z\)\\hat\{z\}^\{=\}=\\Pi\_\{C\_\{=\}\(p\)\}\(z\), we have∥z^=∥≤∥z^≤∥≤∥z∥\\lVert\\hat\{z\}^\{=\}\\rVert\\leq\\lVert\\hat\{z\}^\{\\leq\}\\rVert\\leq\\lVert z\\rVert, so forz≠0z\\neq 0the shares of Equation[3](https://arxiv.org/html/2610.36437#S3.E3)are nonnegative and sum to one\.
5. \(v\)*Order\.*\(pi−pi′\)​\(z^i−z^i′\)≥0\(p\_\{i\}\-p\_\{i^\{\\prime\}\}\)\(\\hat\{z\}\_\{i\}\-\\hat\{z\}\_\{i^\{\\prime\}\}\)\\geq 0for every pair of responses, so the group covariance ofz^\\hat\{z\}with any nondecreasing function of the primary is nonnegative, and in particular⟨z^,zj⟩≥0\\langle\\hat\{z\},z\_\{j\}\\rangle\\geq 0for the primary’s own scoreszjz\_\{j\}\.

By \(iv\), a projected auxiliary has no more energy than under GDPO\. By \(v\), on a task with positive declared weights whose primary gates every other component, CONCUR’s summed score always orders any two responses of a group with different primaries as the primary does, whereas GDPO’s need not\. A component that is not gated, such as IF’s judged criterion, can still reverse this order\.

###### Corollary 5\.2\(A binary primary\)\.

Let the primary of a given task be binary and gate its auxiliarykk\.

1. \(i\)A failing response hasri​k=0r\_\{ik\}=0, sozk∈C≤​\(p\)z\_\{k\}\\in C\_\{\\leq\}\(p\): the order constraint changes nothing, and with the ceiling not tied CONCUR’s sumuuis GDPO’s on any group of the task\.
2. \(ii\)With the ceiling tied, in a group with both passes and failures,z^k=c​zj\\hat\{z\}\_\{k\}=c\\,z\_\{j\}withc=⟨zk,zj⟩/∥zj∥2∈\[0,1\]c=\\langle z\_\{k\},z\_\{j\}\\rangle/\\lVert z\_\{j\}\\rVert^\{2\}\\in\[0,1\], which is the cosine betweenzkz\_\{k\}andzjz\_\{j\}wheneverzkz\_\{k\}is nonzero\.
3. \(iii\)With the ceiling tied, in a group of passes only,z^k=0\\hat\{z\}\_\{k\}=0, whatever the scores inzkz\_\{k\}are\.

On a task such as math, whose only components are a binary primary and one gated auxiliary at weight11each, CONCUR’s summed score with the ceiling tied isu=\(1\+c\)​zju=\(1\+c\)\\,z\_\{j\}in a group with passes and failures andu=0u=0in a group of passes only\. These are the summed scores of GDPO without that auxiliary, multiplied by a per\-group factor1\+c∈\[1,2\]1\+c\\in\[1,2\]\. The gated auxiliary therefore keeps its difference between passes and failures, which adds to the primary’s, and what the projection removes is only its order among the passes within a group\.

## 6Experiments

In this section, we run experiments withQwen3\-4B\-Base,Qwen3\-8B\-Base\([Yang et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib19)\), andOLMo\-3\-7B\-Instruct\-SFT\([Team Olmo et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib17)\)to address four questions: \(1\) Does CONCUR improve on the estimators at the declared weights? \(2\) Does it improve on dropping the gated auxiliaries? \(3\) What does each part of CONCUR add? \(4\) Is CONCUR in effect a change of weight?

MathCodeIFWritingAIME24AIME25MATHOBenchMinervaAMC23MeanHEval\+MBPP\+LCB\-v6MeanIFEvalIFBenchGBenchMeanWBenchCW\-v3LBWriteMeanAvgQwen3\-4B\-BaseBase7\.817\.0867\.2635\.6231\.6143\.9132\.2165\.0655\.5813\.2044\.6147\.3617\.0024\.8629\.7432\.4224\.7644\.4433\.8734\.53GRPO11\.9812\.6080\.2544\.6345\.2854\.6941\.5782\.7471\.1624\.0059\.3086\.4537\.6777\.9367\.3574\.2547\.3269\.2163\.5954\.68GDPO9\.069\.9079\.2944\.6645\.4859\.6141\.3381\.8370\.8724\.4059\.0385\.7336\.3380\.6167\.5671\.9445\.9769\.0162\.3154\.31GD2PO11\.886\.7777\.6743\.5547\.1957\.7340\.8083\.4171\.4623\.4959\.4587\.6537\.6779\.0868\.1370\.1644\.6868\.9461\.2654\.09GEOMA11\.5610\.6378\.8543\.8446\.9455\.3941\.2080\.2470\.2125\.6658\.7087\.7736\.0078\.5067\.4269\.5345\.8665\.8360\.4153\.79SAW12\.5011\.1577\.8443\.8648\.1456\.6441\.6980\.7370\.1122\.8057\.8886\.9340\.3381\.0069\.4269\.8845\.0666\.2160\.3854\.21MT\-GRPO10\.736\.1576\.9942\.5145\.9859\.7740\.3579\.0970\.0522\.6357\.2686\.6934\.3379\.6566\.8970\.1444\.5166\.8160\.4953\.07PAPO9\.277\.5076\.5341\.4846\.5353\.5239\.1483\.0570\.7123\.5459\.1085\.8533\.3380\.7166\.6366\.5046\.5363\.9759\.0052\.60CONCUR\-T19\.4816\.3585\.5152\.8549\.1269\.2248\.7684\.0268\.6024\.7459\.1287\.7737\.6779\.1768\.2080\.1947\.8477\.7468\.5958\.69CONCUR\-A20\.5218\.7585\.2853\.5849\.2269\.3849\.4582\.4471\.5125\.1459\.7088\.1340\.3377\.3568\.6079\.4344\.7176\.3166\.8258\.80Qwen3\-8B\-BaseBase11\.358\.5472\.4639\.4440\.9150\.2337\.1680\.0667\.2820\.9156\.0853\.0017\.3345\.5938\.6436\.7526\.7251\.4238\.3041\.47GRPO15\.8315\.5280\.6447\.9449\.8664\.3845\.7083\.7274\.6626\.5161\.6386\.8142\.0081\.8670\.2276\.4852\.4869\.7866\.2557\.90GDPO21\.3514\.3882\.9949\.3249\.6162\.1946\.6484\.0974\.5226\.2361\.6188\.1339\.3384\.5570\.6776\.0952\.9069\.5766\.1958\.35GD2PO16\.2512\.1981\.2148\.5649\.9259\.6144\.6283\.9675\.0826\.2961\.7889\.3343\.0083\.6972\.0073\.8948\.1669\.8863\.9857\.40GEOMA14\.0612\.5079\.6345\.9949\.7763\.5244\.2482\.7475\.6125\.7161\.3687\.2934\.0084\.3668\.5575\.9450\.4068\.9765\.1056\.70SAW15\.3115\.5281\.9148\.1351\.7762\.2745\.8285\.8574\.3426\.6962\.2987\.6540\.0084\.2670\.6475\.4049\.6771\.0365\.3757\.99MT\-GRPO14\.4815\.7380\.5845\.7551\.0565\.6345\.5385\.0675\.2627\.6662\.6686\.9337\.3383\.1169\.1271\.7050\.3367\.4563\.1657\.20PAPO12\.1910\.1078\.4844\.8149\.3362\.0342\.8282\.5073\.4126\.6360\.8586\.3337\.3384\.3669\.3473\.2051\.1565\.5063\.2855\.82CONCUR\-T31\.2523\.9689\.6159\.5555\.4078\.5256\.3887\.3874\.8727\.2063\.1588\.8539\.3383\.4070\.5387\.1952\.4281\.2273\.6164\.01CONCUR\-A28\.7523\.1390\.1360\.3754\.1978\.9155\.9182\.7473\.2026\.3460\.7688\.1338\.6783\.4070\.0688\.0649\.9281\.6073\.1963\.17OLMo\-3\-7B\-Instruct\-SFTBase8\.547\.4070\.1236\.2735\.2944\.5333\.6969\.2161\.2416\.8649\.1084\.4126\.6761\.4257\.5044\.7437\.9157\.4746\.7144\.14GRPO29\.4824\.6984\.9555\.6739\.9470\.8650\.9377\.3265\.7922\.3455\.1587\.4132\.0056\.4358\.6168\.4749\.3764\.2160\.6855\.26GDPO27\.8122\.6085\.2455\.8940\.3669\.5350\.2476\.3464\.7623\.1454\.7586\.4529\.6750\.8655\.6665\.3147\.1463\.2758\.5753\.89GD2PO28\.3324\.2785\.7256\.0740\.1272\.1951\.1276\.7764\.7923\.2054\.9287\.6532\.3361\.1360\.3765\.7448\.3168\.2160\.7555\.66GEOMA23\.7521\.9883\.1852\.7439\.9763\.9147\.5974\.6364\.7422\.3453\.9088\.0130\.3346\.6454\.9966\.9949\.5965\.3260\.6352\.94SAW26\.1525\.2184\.4953\.8240\.3466\.8849\.4876\.3465\.4221\.8954\.5588\.6132\.6754\.7058\.6667\.3647\.9164\.9360\.0754\.45MT\-GRPO36\.5628\.1386\.9158\.0441\.9677\.4254\.8475\.7364\.9722\.6354\.4487\.0532\.0054\.5157\.8567\.6950\.7066\.3761\.5956\.71PAPO22\.1920\.3180\.8849\.7438\.5659\.3845\.1774\.7665\.0821\.8953\.9187\.1729\.6758\.6458\.4965\.3546\.9864\.7359\.0252\.35CONCUR\-T34\.4829\.9089\.4161\.5744\.0579\.8456\.5483\.2971\.0629\.0361\.1387\.6533\.6758\.5459\.9566\.1449\.0469\.2361\.4759\.13CONCUR\-A36\.3532\.1990\.5362\.4144\.8183\.5258\.3082\.7467\.5430\.3460\.2187\.7732\.0056\.9158\.8965\.4549\.2168\.4961\.0559\.35

Table 1:Evaluation scores of the seven estimators and the two variants of CONCUR\. Each model’s best score is in bold, its second best is underlined, and the base policy is in gray italics\. Eight benchmark names are shortened: MATH is MATH\-500, OBench is OlympiadBench, HEval\+ is HumanEval\+, LCB\-v6 is LiveCodeBench v6, GBench is GuideBench, WBench is WritingBench, CW\-v3 is Creative Writing v3, and LBWrite denotes LongBench\-Write\.Baselines\.We compare CONCUR with seven other advantage estimators at the declared weights:

- •GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib1)\)\. Standardizes the weighted sum of the components within a group\.
- •GDPO\([Liu et al\., 2026c](https://arxiv.org/html/2610.36437#bib.bib6)\)\. Standardizes each component within a group before summing them\.
- •GD2PO\([Liu et al\., 2026b](https://arxiv.org/html/2610.36437#bib.bib10)\)\. Hard variant: drops responses whose component advantages conflict\.
- •GEOMA\([Gupta et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib22)\)\. Aggregates preconditioned components by Nash social welfare\.
- •SAW\([He et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib8)\)\. Rescales each component’s weight by its batch coefficient of variation\.
- •MT\-GRPO\([Ramesh et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib9)\)\. Reweights the tasks toward those that are currently weaker\.
- •PAPO\([Tan et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib44)\)\. Normalizes each process reward among the passing responses only\.

Training\.We train on the four\-task mixture withG=8G=8responses to each prompt\. Following prior work\([Khatri et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib27);[Rastogi et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib28)\), we exclude from the update the groups whose responses all receive identical component scores\. Every run takes 300 steps of 512 prompts, at a constant learning rate of10−610^\{\-6\}\(more in Appendix[D\.2](https://arxiv.org/html/2610.36437#A4.SS2)\)\.

Evaluation\.We evaluate the base policy and each run’s final checkpoint on 15 benchmarks in four*task groups*: 6 math\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.36437#bib.bib30);[Lightman et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib31);[Lewkowycz et al\., 2022](https://arxiv.org/html/2610.36437#bib.bib32);[He et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib33)\), 3 code\([Chen et al\., 2021](https://arxiv.org/html/2610.36437#bib.bib34);[Austin et al\., 2021](https://arxiv.org/html/2610.36437#bib.bib35);[Liu et al\., 2023](https://arxiv.org/html/2610.36437#bib.bib36);[Jain et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib37)\), 3 IF\([Zhou et al\., 2023](https://arxiv.org/html/2610.36437#bib.bib14);[Pyatkin et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib38);[Diao et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib62)\), and 3 writing\([Wu et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib15);[Paech, 2025](https://arxiv.org/html/2610.36437#bib.bib40);[Bai et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib63)\)\. A task group’s score is the mean over its benchmarks, and*Avg*is the mean over all benchmarks\. We score the writing benchmarks with the judge model used in training, which we call the*training judge*\. The same responses are also scored by a second judge, which comes from another model family and is not used in training\.

### 6\.1Q1: CONCUR Has the Highest Avg

As shown in Table[1](https://arxiv.org/html/2610.36437#S6.T1), on all three models CONCUR in either variant has the highest Avg of the runs at the declared weights, more than four points above GDPO, and both variants are also the top two under the second judge\. Their largest gain is on math, at least 6\.3 points above GDPO and above every estimator\. Five of the other estimators change how components are normalized, weighted, or filtered but treat a gated auxiliary like any other component, and none of them moves Avg by more than three points from GDPO on any model\. PAPO keeps the auxiliary’s order among correct responses, which CONCUR removes on math, and has the lowest math score of the seven estimators on every model\. Under the second judge, both variants score above GDPO on the writing task group on every model and above all seven estimators on the two Qwen models, but below five of them onOLMo\-3\-7B\-Instruct\-SFT, as Table[8](https://arxiv.org/html/2610.36437#A5.T8)in the appendix shows\.

### 6\.2Q2: CONCUR Beats Dropping Auxiliaries

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/MTMR-ablations-qwen3-4b-base.png)

\(a\)Qwen3\-4B\-Base

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/MTMR-ablations-qwen3-8b-base.png)

\(b\)Qwen3\-8B\-Base

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/MTMR-ablations-olmo-3-7b-it-sft.png)

\(c\)OLMo\-3\-7B\-Instruct\-SFT

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/aux.png)

\(d\) Auxiliary scores onQwen3\-4B\-Base

Figure 3:Panels \(a\) to \(c\): each task group’s score and Avg minus those of GDPO\. Panel \(d\): each gated auxiliary’s mean score over the passing responses, as the mean over the last 20 training steps minus the base policy’s level\.Figure[3](https://arxiv.org/html/2610.36437#S6.F3)\(a\) to \(c\) compare both variants with GDPO and with the reference GDPO \(no aux\.\)\. On every model, dropping the gated auxiliaries raises the math score by more than seven points and lowers the writing score by more than four, so neither choice is better on both\. The reference scores below GDPO on all three writing benchmarks and under both judges\. Both variants keep most of the gain on math and score above GDPO on writing, and on Avg they are above both runs\. OnOLMo\-3\-7B\-Instruct\-SFT, the reference still scores above both variants on math and on code\.

What happens to the auxiliaries’ scores?To track what the auxiliaries reward, we look at training onQwen3\-4B\-Base\. Figure[3](https://arxiv.org/html/2610.36437#S6.F3)\(d\) shows how each gated auxiliary’s mean score over passing responses changes over training\. Under both variants, this score rises on code, on IF, and for writing’s RM about as much as under GDPO, and it falls on math and for writing’s length reward\. These two pairs correspond to the two places that the motivating experiment measured: on math, every passing response is at the ceiling, and on writing, more than half of the length reward’s energy is against the primary’s order\. Writing’s RM also has more than half of its energy against the order, but unlike the length reward, it agrees with the primary more often than it reverses it among passing responses\. In the reference, this score ends lower than under either variant on all five pairs, so both variants give up less of what the auxiliaries reward than dropping them does\. The same holds on the other two models, and onOLMo\-3\-7B\-Instruct\-SFT, where code’s ceiling is tied at every step, this score also falls on code under both variants\. Appendix[E\.4](https://arxiv.org/html/2610.36437#A5.SS4)shows their panels and every benchmark’s scores\.

What happens to the primary?The bottom row of Figure[4](https://arxiv.org/html/2610.36437#S6.F4)tracks the primary’s mean score over all responses\. On math and on writing, it rises more under both variants than under GDPO and nearly as much as under the reference, on all three models \(more in Appendix[E\.4](https://arxiv.org/html/2610.36437#A5.SS4)\)\. The reference trains these two tasks on the primary alone, so under GDPO the gated auxiliaries hold the primary back, and under CONCUR they do much less\. On writing, the reference reaches about the same primary as both variants but scores below GDPO on the benchmarks, which also measure what the auxiliaries reward, and CONCUR keeps more of it\.

### 6\.3Q3: What Each Part of CONCUR Adds

We also run the ablation CONCUR\-O, which applies the order constraint alone\. Figure[3](https://arxiv.org/html/2610.36437#S6.F3)\(a\) to \(c\) show its scores along with those of the other runs\.

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/alignment.png)Figure 4:Cosine between each task’s advantages and the standardized scores of its primary \(top\) and the primary’s mean score over all responses \(bottom\) during training onQwen3\-4B\-Base, shown as 10\-step running means\.The order constraint\.Compared with GDPO, CONCUR\-O raises the writing score on every model, by at least 4\.5 points, and writing is the task where the share against the order is large\. Under the second judge, CONCUR\-O also scores above GDPO on writing on every model\. Few of writing’s passing responses are at the ceiling, so in almost every writing group CONCUR\-T and CONCUR\-A compute what CONCUR\-O computes\. In training, the three runs reach about the same primary on writing, above that of GDPO\. OnQwen3\-4B\-Base, Figure[4](https://arxiv.org/html/2610.36437#S6.F4)shows that the cosine between writing’s advantages and its primary’s standardized scores falls under GDPO and stays high under both variants\. Over the last 20 steps on all three models, this cosine is at most 0\.76 under GDPO and at least 0\.92 under both variants\. On code and IF, where the share against the order stays small, CONCUR\-O scores about as high as GDPO or higher on the benchmarks\. By Corollary[5\.2](https://arxiv.org/html/2610.36437#S5.Thmtheorem2), CONCUR\-O computes the summed scores of GDPO on every math group, so the order constraint does not act on math\.

The ceiling constraint\.CONCUR\-T adds the ceiling constraint on all pairs at all steps, and its math score is at least 6\.6 points above that of CONCUR\-O\. The gain on math comes from the ceiling constraint\. CONCUR\-T, CONCUR\-A, and the reference are the three runs that give math’s auxiliary no credit among correct responses, and all three score far above GDPO and CONCUR\-O on math, with a higher math accuracy in training as well\. Yet math takes a smaller part of the update, since the ceiling constraint sets to zero the summed scores of a math group whose responses are all correct, and such a group stays in the batch whitening\. Over the last 20 steps, math’s share of the batch’s summed squared advantages is at least 0\.24 under GDPO and at most 0\.18 under CONCUR\-T \(more in Appendix[E\.3](https://arxiv.org/html/2610.36437#A5.SS3)\)\. The ceiling constraint changes the other three task groups less, and its largest effect there is on the code score ofOLMo\-3\-7B\-Instruct\-SFT, which it raises by 5\.5 points relative to CONCUR\-O\. Under GDPO, the share at the ceiling on code is 0\.21 on that model and only 0\.04 on each of the two Qwen models\.

The ceiling policy\.The two variants end within one point of each other on Avg\. They differ in what they compute, as Table[6](https://arxiv.org/html/2610.36437#A4.T6)in the appendix details\. CONCUR\-A keeps the tie for math, whose binary primary gives no reading, and for writing’s length reward, and lifts it at almost every step for IF and for writing’s RM\. On code it lifts the tie at most steps on the two Qwen models and keeps it at every step onOLMo\-3\-7B\-Instruct\-SFT, where code’s length reward reverses the primary’s order more often than it agrees with it\. There CONCUR\-A also scores 4\.6 points above CONCUR\-O on code\. Lifting a tie changes scores only where passing responses reach the ceiling\. IF and code on the two Qwen models have more than half of their passing responses there, while writing’s RM has almost none\. OnQwen3\-8B\-Base, code’s pooled agreement is barely positive, CONCUR\-A lifts the tie at almost every step, and its code score is 2\.4 points below that of CONCUR\-T\. As readings accumulate, the standard error shrinks, so the rule tests only the sign of the pooled agreement\. This case is one reason why CONCUR\-T is our default variant\.

### 6\.4Q4: Is CONCUR a Change of Weight?

A weight scales an auxiliary on every response of every group by one factor, whereas the projection changes an auxiliary’s scores within a group, differently in each group, and not at all where they already concur\. On a binary primary that gates one auxiliary, as on math, CONCUR comes close to a change of weight\. By Corollary[5\.2](https://arxiv.org/html/2610.36437#S5.Thmtheorem2), its summed scores in a group are GDPO’s without the auxiliary, scaled by the group’s own factor, and the math scores of the two variants are within 2\.9 points of the reference’s\. At a smaller positive weight, the auxiliary alone still orders a group whose responses are all at the ceiling\. GDPO \(w=0\.5w=0\.5\), which sets every gated auxiliary to weight0\.50\.5, gains far less on math than the reference does and scores below both variants on Avg on all three models\. The length reward’s score on math moves under it about as under GDPO but falls under both variants\. On a graded primary, the projection is not a single factor on the auxiliary\. On writing, GDPO at each of the three weights11,0\.50\.5, and00scores below CONCUR\-O and below the two variants on all three models\.

Dropping the division\.GRPO \(no std\), which drops GRPO’s division by the group’s standard deviation, scores more than six points below the reference on math, and more than three points below both variants on Avg \(more in Appendix[E\.4](https://arxiv.org/html/2610.36437#A5.SS4)\)\. Without this division, math’s auxiliary still gets credit among correct responses, as under GDPO\.

## 7Conclusion

CONCUR extends the gate’s priority to every pair of responses in a group: it projects each gated auxiliary’s standardized scores onto the closest scores that concur with the primary\. On Avg over 15 benchmarks, the default CONCUR\-T and the adaptive CONCUR\-A both score more than four points above GDPO on all three models, and they are also above every other estimator in our comparison and above a reference that drops the gated auxiliaries\. The order constraint alone raises the writing score, the ceiling constraint raises the math score, and the two variants end within one point of each other on Avg\. In training, both variants raise the primary on math and on writing nearly as much as dropping the gated auxiliaries does, and they give up less of what the auxiliaries reward, as every gated auxiliary’s mean score over the passing responses ends higher than under the reference on every model\.

## Limitations

Our evidence comes from one four\-task mixture\. Its gated auxiliaries are four length rewards and one reward\-model score\. We test one intermediate weight\. We measure what the auxiliaries reward only on training rollouts, since the benchmarks do not score it\. The order constraint is a design choice, and it is wrong when the designer would let an auxiliary outrank a small difference in the primary\. It constrains how credit is assigned within a group, not the policy’s expected primary after an update\. Where the primary is a judge’s score, as on writing, CONCUR makes the gated auxiliaries follow that judge, which may share preferences with writing’s reward model\([Singhal et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib13);[Dubois et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib39)\)\. The gains over GDPO on the writing task group are smaller under a second judge, as Table[8](https://arxiv.org/html/2610.36437#A5.T8)reports\. Earned trust is also a design choice\. It assumes that an auxiliary’s agreement with the primary where the primary separates passing responses carries over to the ceiling, where it separates none, and our mixture does not test this\.

## References

- Achiamet al\.\(2017\)J\. Achiam, D\. Held, A\. Tamar, and P\. AbbeelConstrained policy optimization\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.70,pp\. 22–31\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Aggarwal and Welleck \(2025\)P\. Aggarwal and S\. WelleckL1: controlling how long a reasoning model thinks with reinforcement learning\.InConference on Language Modeling \(COLM\),Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Arora and Zanette \(2025\)D\. Arora and A\. ZanetteTraining language models to reason efficiently\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Ayeret al\.\(1955\)M\. Ayer, H\. D\. Brunk, G\. M\. Ewing, W\. T\. Reid, and E\. SilvermanAn empirical distribution function for sampling with incomplete information\.The Annals of Mathematical Statistics26\(4\),pp\. 641–647\.Cited by:[§C\.2](https://arxiv.org/html/2610.36437#A3.SS2.p4.1),[§2](https://arxiv.org/html/2610.36437#S2.p3.1)\.
- Baiet al\.\(2026\)Y\. Bai, K\. Liu, Z\. Zhuang, J\. Zhou, R\. Weng, X\. Chen, J\. Wang, and X\. CaiMulti\-objective and mixed\-reward reinforcement learning via reward\-decorrelated policy optimization\.arXiv preprint arXiv:2605\.13641\.Cited by:[§1](https://arxiv.org/html/2610.36437#S1.p1.1),[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Baiet al\.\(2025\)Y\. Bai, J\. Zhang, X\. Lv, L\. Zheng, S\. Zhu, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongWriter: unleashing 10,000\+ word generation from long context LLMs\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 36528–36546\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Chenet al\.\(2018\)Z\. Chen, V\. Badrinarayanan, C\. Lee, and A\. RabinovichGradNorm: gradient normalization for adaptive loss balancing in deep multitask networks\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 794–803\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Cuiet al\.\(2025\)G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, Y\. Zhang, J\. Chen, W\. Li, B\. He, Y\. Fan, T\. Yu,et al\.Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p1.1)\.
- Daiet al\.\(2024\)J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. YangSafe RLHF: safe reinforcement learning from human feedback\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 50750–50777\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Diaoet al\.\(2025\)L\. Diao, X\. Xu, W\. Sun, C\. Yang, and Z\. ZhangGuideBench: benchmarking domain\-oriented guideline following for LLM agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11361–11399\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Duet al\.\(2018\)Y\. Du, W\. M\. Czarnecki, S\. M\. Jayakumar, M\. Farajtabar, R\. Pascanu, and B\. LakshminarayananAdapting auxiliary losses using gradient similarity\.arXiv preprint arXiv:1812\.02224\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.InConference on Language Modeling \(COLM\),Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1),[Limitations](https://arxiv.org/html/2610.36437#Sx1.p1.1)\.
- Fanget al\.\(2026\)S\. Fang, Y\. Zhao, W\. Tian, J\. Lu, Y\. Zheng, Y\. Wang, and X\. QiuORPG: reconciling multiple reward objectives through objective\-wise policy gradients\.arXiv preprint arXiv:2609\.34985\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Gemma Teamet al\.\(2026\)Gemma Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon,et al\.Gemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p3.1)\.
- Girgiset al\.\(2026\)R\. Girgis, R\. de Schaetzen, L\. Rowe, A\. Robitaille, C\. Pal, and L\. PaullConstrained group relative policy optimization\.arXiv preprint arXiv:2602\.05863\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§1](https://arxiv.org/html/2610.36437#S1.p1.1)\.
- Guptaet al\.\(2024\)T\. Gupta, R\. Madhavan, X\. Zhang, N\. Natarajan, C\. Bansal, and S\. RajmohanMulti\-preference optimization: generalizing DPO via set\-level contrasts\.arXiv preprint arXiv:2412\.04628\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Guptaet al\.\(2026\)T\. Gupta, P\. Srivastava, R\. Madhavan, K\. Shanmugam, and A\. RaghuveerGEOMA: geometric and econometric objectives for multi\-reward alignment\.InThe 1st Workshop on Scaling Post\-training for LLMs,Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[4th item](https://arxiv.org/html/2610.36437#S6.I1.i4.p1.1)\.
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. SunOlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Heet al\.\(2026\)Y\. He, B\. Bi, S\. Liu, H\. Liao, Y\. Ge, B\. Wan, S\. Tong, J\. Chen, J\. Guo, and X\. ChengSAW: stage\-aware dynamic weighting for multi\-objective reinforcement learning in large language models\.arXiv preprint arXiv:2606\.07705\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[5th item](https://arxiv.org/html/2610.36437#S6.I1.i5.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Huet al\.\(2026\)W\. Hu, H\. Jing, H\. Shi, Y\. Liu, H\. Li, and Y\. SongCorrGRPO: correlation\-normalized GRPO for multi\-reward learning\.arXiv preprint arXiv:2609\.36820\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Hugging Face \(2025\)Hugging FaceOpen R1: a fully open reproduction of DeepSeek\-R1\.External Links:[Link](https://github.com/huggingface/open-r1)Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p1.1)\.
- Ichiharaet al\.\(2026\)Y\. Ichihara, Y\. Jinnai, T\. Morimura, M\. Sakamoto, R\. Mitsuhashi, and E\. UchibeMO\-GRPO: mitigating reward hacking of group relative policy optimization on multi\-objective problems\.Transactions of the Association for Computational Linguistics14,pp\. 1867–1882\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Kanchetiet al\.\(2026\)S\. S\. Kancheti, A\. Kanade, R\. Sinha, V\. N\. Balasubramanian, and T\. GanuFaithful GRPO: improving visual spatial reasoning in multimodal language models via constrained policy optimization\.InConference on Language Modeling \(COLM\),Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Kassenaaret al\.\(2026\)G\. Kassenaar, Z\. Yang, and V\. François\-LavetLearning when to think: adaptive reasoning for test\-time compute allocation\.arXiv preprint arXiv:2608\.20256\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Kendallet al\.\(2018\)A\. Kendall, Y\. Gal, and R\. CipollaMulti\-task learning using uncertainty to weigh losses for scene geometry and semantics\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 7482–7491\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Khatriet al\.\(2026\)D\. Khatri, L\. Madaan, R\. Tiwari, R\. Bansal, S\. S\. Duvvuri, M\. Zaheer, I\. S\. Dhillon, D\. Brandfonbrener, and R\. AgarwalThe art of scaling reinforcement learning compute for LLMs\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p3.1)\.
- Kimi Team \(2025\)Kimi TeamKimi k1\.5: scaling reinforcement learning with LLMs\.arXiv preprint arXiv:2501\.12599\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Lambertet al\.\(2025\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. Le Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. HajishirziTulu 3: pushing frontiers in open language model post\-training\.InConference on Language Modeling \(COLM\),Cited by:[§1](https://arxiv.org/html/2610.36437#S1.p1.1)\.
- Lenget al\.\(2025\)J\. Leng, C\. Huang, B\. Zhu, and J\. HuangTaming overconfidence in LLMs: reward calibration in RLHF\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Lenget al\.\(2026\)J\. Leng, S\. Si, H\. Yu, V\. Raman, and I\. S\. DhillonGroupDPO: memory\-efficient group\-wise direct preference optimization\.arXiv preprint arXiv:2604\.15602\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. MisraSolving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Liet al\.\(2026a\)G\. Li, Y\. Chen, M\. Lin, and T\. YangDRPO: efficient reasoning via decoupled reward policy optimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Liet al\.\(2026b\)S\. Li, J\. Zhao, H\. Ren, Z\. Wei, Y\. Zhou, J\. Yang, S\. Liu, K\. Zhang, and C\. WeiRubricHub: a comprehensive and highly discriminative rubric dataset via automated coarse\-to\-fine generation\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 31320–31344\.Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.36437#S3.SS1.p2.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Liuet al\.\(2026a\)C\. Y\. Liu, L\. Zeng, Y\. Xiao, J\. He, J\. Liu, C\. Wang, R\. Yan, W\. Shen, F\. Zhang, J\. Xu, and Y\. LiuSkywork\-Reward\-V2: scaling preference data curation via human\-AI synergy\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p3.1),[§3\.1](https://arxiv.org/html/2610.36437#S3.SS1.p2.1)\.
- Liuet al\.\(2026b\)H\. Liu, Y\. Liu, J\. Ni, S\. Huang, X\. Liu, P\. Cheng, J\. Song, R\. Ding, J\. Li, Z\. Yu, M\. Zhou, H\. Xu, X\. Jiang, and G\. JiangGD2\{\}^\{2\}PO: mitigating multi\-reward conflicts via group\-dynamic reward\-decoupled policy optimization\.arXiv preprint arXiv:2606\.16771\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[3rd item](https://arxiv.org/html/2610.36437#S6.I1.i3.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation\.Advances in Neural Information Processing Systems36,pp\. 21558–21572\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Liuet al\.\(2026c\)S\. Liu, X\. Dong, X\. Lu, S\. Diao, P\. Belcak, M\. Liu, M\. Chen, H\. Yin, Y\. F\. Wang, K\. Cheng, Y\. Choi, J\. Kautz, and P\. MolchanovGDPO: group reward\-decoupled normalization policy optimization for multi\-reward RL optimization\.InProceedings of the 43rd International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.36437#S1.p1.1),[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[§2](https://arxiv.org/html/2610.36437#S2.p2.1),[2nd item](https://arxiv.org/html/2610.36437#S6.I1.i2.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding R1\-Zero\-like training: a critical perspective\.InConference on Language Modeling \(COLM\),Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Luet al\.\(2026\)Y\. Lu, Z\. Wang, S\. Li, X\. Liu, C\. Yu, Q\. Yin, Z\. Shi, Z\. Zhang, and M\. JiangLearning to optimize multi\-objective alignment through dynamic reward weighting\.Transactions of the Association for Computational Linguistics14,pp\. 1051–1073\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Luoet al\.\(2025\)M\. Luo, S\. Tan, J\. Wong, X\. Shi, W\. Y\. Tang, M\. Roongta, C\. Cai, J\. Luo, L\. E\. Li, R\. A\. Popa, and I\. StoicaDeepScaleR: surpassing O1\-Preview with a 1\.5B model by scaling RL\.Note:Notion blogExternal Links:[Link](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2)Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p1.1)\.
- Munnet al\.\(2026\)H\. Munn, B\. Tidd, P\. Böhm, M\. Gallagher, and D\. HowardScalable multi\-objective robot reinforcement learning through gradient conflict resolution\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 9154–9161\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Paech \(2025\)S\. J\. PaechEQ\-Bench creative writing benchmark v3\.External Links:[Link](https://github.com/EQ-bench/creative-writing-bench)Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Pardinaset al\.\(2026\)R\. Pardinas, E\. Kamalloo, D\. Vazquez, and A\. DrouinApriel\-Reasoner: RL post\-training for general\-purpose and efficient reasoning\.InConference on Language Modeling \(COLM\),Cited by:[§1](https://arxiv.org/html/2610.36437#S1.p1.1)\.
- Penget al\.\(2026\)K\. Peng, Y\. Ouyang, X\. Liu, Z\. Tian, R\. Han, Y\. Yuan, and L\. DingThink dense, not long: dynamic decoupled conditional advantage for efficient reasoning\.arXiv preprint arXiv:2602\.02099\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Pyatkinet al\.\(2025\)V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. HajishirziGeneralizing verifiable instruction following\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Vol\.38\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Qiuet al\.\(2026\)R\. Qiu, R\. Wang, G\. Yang, X\. Li, and Z\. ShaoLPPG\-RL: lexicographically projected policy gradient reinforcement learning with subproblem exploration\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 25009–25017\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-35B\-A3B: agentic coding power, now open to all\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-35b-a3b)Cited by:[§D\.3](https://arxiv.org/html/2610.36437#A4.SS3.p1.1)\.
- Rameshet al\.\(2026\)S\. S\. Ramesh, X\. Ji, M\. Zimmer, S\. Yoon, Z\. Wang, H\. Bou Ammar, A\. Lucchi, and I\. BogunovicMulti\-task GRPO: reliable LLM reasoning across tasks\.InProceedings of the 43rd International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.36437#S1.p1.1),[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[6th item](https://arxiv.org/html/2610.36437#S6.I1.i6.p1.1)\.
- Rastogiet al\.\(2025\)A\. Rastogi, A\. Q\. Jiang, A\. Lo, G\. Berrada, G\. Lample, J\. Rute, J\. Barmentlo, K\. Yadav, K\. Khandelwal, K\. R\. Chandu,et al\.Magistral\.arXiv preprint arXiv:2506\.10910\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p3.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§D\.2](https://arxiv.org/html/2610.36437#A4.SS2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1),[1st item](https://arxiv.org/html/2610.36437#S6.I1.i1.p1.1)\.
- Singhalet al\.\(2024\)P\. Singhal, T\. Goyal, J\. Xu, and G\. DurrettA long way to go: investigating length correlations in RLHF\.InConference on Language Modeling \(COLM\),Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1),[Limitations](https://arxiv.org/html/2610.36437#Sx1.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, L\. Hammond, C\. Griffin, and A\. AbateLexicographic multi\-objective reinforcement learning\.InProceedings of the Thirty\-First International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 3430–3436\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Songet al\.\(2024\)F\. Song, B\. Yu, M\. Li, H\. Yu, F\. Huang, Y\. Li, and H\. WangPreference ranking optimization for human alignment\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 18990–18998\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Song and Zheng \(2025\)M\. Song and M\. ZhengWalk before you run\! Concise LLM reasoning via reinforcement learning\.arXiv preprint arXiv:2505\.21178\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1)\.
- Tanet al\.\(2026\)Z\. Tan, Z\. Yu, B\. Lin, Z\. Geng, H\. Geng, Y\. Zhang, M\. Zhang, Y\. Chen, S\. Hu, Z\. Yin, C\. Zhang, and L\. BaiPAPO: stabilizing rubric integration training via decoupled advantage normalization\.InProceedings of the 2026 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p2.1),[7th item](https://arxiv.org/html/2610.36437#S6.I1.i7.p1.1)\.
- Team Olmoet al\.\(2025\)Team Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl,et al\.Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p1.1),[§6](https://arxiv.org/html/2610.36437#S6.p1.1)\.
- Tercan and Prabhu \(2024\)A\. Tercan and V\. S\. PrabhuThresholded lexicographic ordered multiobjective reinforcement learning\.InProceedings of the 27th European Conference on Artificial Intelligence \(ECAI\),Frontiers in Artificial Intelligence and Applications, Vol\.392,pp\. 3006–3014\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Tessleret al\.\(2019\)C\. Tessler, D\. J\. Mankowitz, and S\. MannorReward constrained policy optimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Wanget al\.\(2026\)Y\. Wang, Y\. Zhan, Z\. Feng, Z\. Cui, Z\. Zhou, R\. Liang, J\. Zhu, Z\. Yang, Y\. Zhao, Z\. Mao, B\. Jia, H\. Ni, C\. Xie, B\. Liu, Y\. Zhang, Y\. Dai, X\. Ju, W\. Ye, and S\. ZhangTrustRoboReward: preference\-ordered isotonic score editing for multi\-paradigm robot reward models\.arXiv preprint arXiv:2608\.08491\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p3.1)\.
- Wang \(2026\)Y\. WangThe dark room in the reward channel: dense prediction rewards collapse GRPO\-trained LLM agents – and the channel, not the content, decides what works\.arXiv preprint arXiv:2607\.21273\.Cited by:[§2](https://arxiv.org/html/2610.36437#S2.p1.1)\.
- Wuet al\.\(2025\)Y\. Wu, J\. Mei, M\. Yan, C\. Li, S\. Lai, Y\. Ren, Z\. Wang, J\. Zhang, M\. Wu, Q\. Jin, and F\. HuangWritingBench: a comprehensive benchmark for generative writing\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Vol\.38\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.2](https://arxiv.org/html/2610.36437#S3.SS2.p1.2),[§6](https://arxiv.org/html/2610.36437#S6.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu,et al\.DAPO: an open\-source LLM reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§D\.1](https://arxiv.org/html/2610.36437#A4.SS1.p1.1)\.
- Yuet al\.\(2020\)T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. FinnGradient surgery for multi\-task learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 5824–5836\.Cited by:[Appendix A](https://arxiv.org/html/2610.36437#A1.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§6](https://arxiv.org/html/2610.36437#S6.p4.1)\.

## Appendix AAdditional Related Work

In this section, we discuss other ways to enforce a priority among rewards and further work on reward models and group\-wise objectives\. Constrained RL maximizes one reward subject to a constraint on another\([Achiam et al\., 2017](https://arxiv.org/html/2610.36437#bib.bib51);[Tessler et al\., 2019](https://arxiv.org/html/2610.36437#bib.bib26);[Dai et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib11);[Girgis et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib41);[Kancheti et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib42)\), lexicographic RL maximizes its rewards in a fixed order of priority\([Skalse et al\., 2022](https://arxiv.org/html/2610.36437#bib.bib12);[Tercan and Prabhu, 2024](https://arxiv.org/html/2610.36437#bib.bib47)\), and methods in gradient space weight or project one gradient so that it does not oppose another\([Du et al\., 2018](https://arxiv.org/html/2610.36437#bib.bib61);[Yu et al\., 2020](https://arxiv.org/html/2610.36437#bib.bib50);[Qiu et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib54);[Munn et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib55)\)\. CONCUR applies the priority to every pair of responses in a group, directly on their scores, with no gradient and no multiplier\. Reward models and automatic judges can favor longer or more confident responses regardless of quality\([Singhal et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib13);[Dubois et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib39);[Leng et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib68)\), and in our mixture the RM is an auxiliary that writing’s primary gates\. Outside RL, preference optimization goes beyond pairs and contrasts several responses to the same prompt jointly, as a ranked list or as a set\([Song et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib70);[Gupta et al\., 2024](https://arxiv.org/html/2610.36437#bib.bib71);[Leng et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib69)\)\.

## Appendix BMethod Details

In this section, we list the role of every reward component in the mixture, describe how the projection is computed, and specify the rule of earned trust\.

### B\.1Gates and Roles

In Table[2](https://arxiv.org/html/2610.36437#A2.T2), math’s primary is binary, and the primaries of code, IF, and writing are graded\. In our implementation, a response is at the ceiling if its primary is within10−610^\{\-6\}of the maximum value11\.

Table 2:Reward components of the mixture: a task’s primary gates each auxiliary in its row, all at weight11\.
### B\.2The Projection

For one group and one gated auxiliarykk, the projection of the standardized scoreszkz\_\{k\}in Equation[5](https://arxiv.org/html/2610.36437#S4.E5)is computed in two steps, as Algorithm[1](https://arxiv.org/html/2610.36437#alg1)states\.*Averaging step\.*If the pair’s ceiling is tied, the scores of the responses at the ceiling are replaced by their mean\.*Pooling step\.*The responses are sorted in ascending order by primary, and those with equal primaries by their score after the averaging step\. The sorted scores are then scanned in order\. Each score starts a block of its own, and while the last block has a lower mean than the one before it, the two merge\. At the end, every response receives the mean of the block that holds it\. Responses that share a primary below the ceiling are not constrained among themselves\. Their equal primaries can be an artifact of sampling, since another response to the same prompt can have a higher primary, and the order constraint then keeps all of their auxiliary scores at or below that response’s\.

Algorithm 1CONCUR for one batch\. Line 2 is earned trust, which only CONCUR\-A carries out\.1:Input \(CONCUR\-A\):for every gated pair

\(t,k\)\(t,k\), the number

nn, the mean

a¯t,k\\bar\{a\}\_\{t,k\}, and the standard error

set,k\\mathrm\{se\}\_\{t,k\}of the readings

ax,ka\_\{x,k\}of Equation[4](https://arxiv.org/html/2610.36437#S3.E4)from all groups so far, including this batch’s

2:For each taskttand gated auxiliaryk∈𝒜tk\\in\\mathcal\{A\}\_\{t\}

3:

tied⁡\(t,k\)←\\mathrm\{tied\}\(t,k\)\\leftarrowtrue

4:CONCUR\-A:

tied⁡\(t,k\)←\\mathrm\{tied\}\(t,k\)\\leftarrowfalse if

n≥61n\\geq 61and

a¯t,k\>2​set,k\\bar\{a\}\_\{t,k\}\>2\\,\\mathrm\{se\}\_\{t,k\}
5:For each group \(promptxxof tasktt\)

6:

zi​k←ri​kz\_\{ik\}\\leftarrow r\_\{ik\}standardized within the group, for every

k∈𝒦tk\\in\\mathcal\{K\}\_\{t\}
7:for

k∈𝒜tk\\in\\mathcal\{A\}\_\{t\}do

8:if

tied⁡\(t,k\)\\mathrm\{tied\}\(t,k\)thenreplace the

zi​kz\_\{ik\}of the responses at the ceiling with their mean

9:

z^k←\\hat\{z\}\_\{k\}\\leftarrowsort in ascending order by \(primary, score\), then pool adjacent violators

10:endfor

11:

ui←∑k∈𝒦t∖𝒜twk​zi​k\+∑k∈𝒜twk​z^i​ku\_\{i\}\\leftarrow\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\\setminus\\mathcal\{A\}\_\{t\}\}w\_\{k\}z\_\{ik\}\+\\sum\_\{k\\in\\mathcal\{A\}\_\{t\}\}w\_\{k\}\\hat\{z\}\_\{ik\}
12:For the batch

13:

A←𝒲⁡\(u\)A\\leftarrow\\mathcal\{W\}\(u\)

### B\.3Earned Trust

Leta1,…,ana\_\{1\},\\dots,a\_\{n\}be all readings of the gated pair\(t,k\)\(t,k\)since the start of training, including those of the current step’s batch\. Their mean and its standard error are given by

a¯t,k\\displaystyle\\bar\{a\}\_\{t,k\}=1n​∑m=1nam,\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{m=1\}^\{n\}a\_\{m\},\(7\)set,k\\displaystyle\\mathrm\{se\}\_\{t,k\}=1n⁡\(n−1\)​∑m=1n\(am−a¯t,k\)2\.\\displaystyle=\\sqrt\{\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{m=1\}^\{n\}\\bigl\(a\_\{m\}\-\\bar\{a\}\_\{t,k\}\\bigr\)^\{2\}\}\.At a step, the pair’s ceiling is not tied ifn≥61n\\geq 61anda¯t,k\>2​set,k\\bar\{a\}\_\{t,k\}\>2\\,\\mathrm\{se\}\_\{t,k\}, and it is tied otherwise\. The rule has three settings: the margin of two standard errors, the minimum number of readings, and pooling over all of training\. The minimum of6161readings comes from Student’sttdistribution: with6161readings and thus6060degrees of freedom, its two\-sided95%95\\%critical value is2\.002\.00to two decimals\. A pair with a graded primary has6161readings within a few training steps, and the standard error shrinks as readings accumulate, so the rule tests only the sign of the pooled agreement\. The readings are not independent draws from one distribution, since the policy changes between steps and every reading has the same weight, so we use the margin as a fixed threshold on the pooled agreement and attach no significance level to it\. A ceiling that is not tied at one step can be tied again at a later one\. No reading compares two responses at the ceiling: they have the same primary, so every pair of responses that Equation[4](https://arxiv.org/html/2610.36437#S3.E4)counts has a response below the ceiling\. The rule assumes that if an auxiliary agrees with the primary’s order on such pairs more often than it reverses that order, it also tracks what the primary measures among the responses at the ceiling\.

## Appendix CAnalysis and Proofs

In this section, we show how standardization acts at the ceiling, prove Proposition[5\.1](https://arxiv.org/html/2610.36437#S5.Thmtheorem1)and Corollary[5\.2](https://arxiv.org/html/2610.36437#S5.Thmtheorem2), and add a few remarks on these two statements\.

A group hasG≥2G\\geq 2responses with primariesp∈\[0,1\]Gp\\in\[0,1\]^\{G\}, andH=\{i:pi=1\}H=\\\{i:p\_\{i\}=1\\\}is the set of responses at the ceiling\. Vectors lie inℝG\\mathbb\{R\}^\{G\}, with⟨a,b⟩=∑iai​bi\\langle a,b\\rangle=\\sum\_\{i\}a\_\{i\}b\_\{i\}and∥a∥2=⟨a,a⟩\\lVert a\\rVert^\{2\}=\\langle a,a\\rangle, and𝟏\\mathbf\{1\}is the vector of ones\. The standardized scoreszkz\_\{k\}of Equation[1](https://arxiv.org/html/2610.36437#S3.E1)satisfy⟨zk,𝟏⟩=0\\langle z\_\{k\},\\mathbf\{1\}\\rangle=0, and sincedkd\_\{k\}is the sample standard deviation withdk2=1G−1​∑i\(ri​k−r¯k\)2d\_\{k\}^\{2\}=\\frac\{1\}\{G\-1\}\\sum\_\{i\}\(r\_\{ik\}\-\\bar\{r\}\_\{k\}\)^\{2\}, they also satisfy∥zk∥2=G−1\\lVert z\_\{k\}\\rVert^\{2\}=G\-1ifdk\>0d\_\{k\}\>0\. The two facts below are standard\. LetC⊂ℝGC\\subset\\mathbb\{R\}^\{G\}be a closed convex cone andz∈ℝGz\\in\\mathbb\{R\}^\{G\}\.*Fact 1\.*The functionv↦∥v−z∥2v\\mapsto\\lVert v\-z\\rVert^\{2\}has exactly one minimizer onCC, which we writeΠC​\(z\)\\Pi\_\{C\}\(z\)\.*Fact 2\.*A pointz^∈C\\hat\{z\}\\in CequalsΠC​\(z\)\\Pi\_\{C\}\(z\)if and only if it satisfies

⟨z−z^,z^⟩\\displaystyle\\langle z\-\\hat\{z\},\\,\\hat\{z\}\\rangle=0and\\displaystyle=0\\quad\\text\{and\}\(8\)⟨z−z^,v⟩\\displaystyle\\langle z\-\\hat\{z\},\\,v\\rangle≤0for every​v∈C\.\\displaystyle\\leq 0\\quad\\text\{for every \}v\\in C\.\(9\)
### C\.1Standardization at the Ceiling

In a group whose responses are all at the ceiling, every responseiihasri​j=1r\_\{ij\}=1, sodj=0d\_\{j\}=0, which giveszi​j=0z\_\{ij\}=0\. A componentkkwithdk\>0d\_\{k\}\>0has∥zk∥2=G−1\\lVert z\_\{k\}\\rVert^\{2\}=G\-1whatever the size of its raw differences, so its standardized scores have the sample variance1G−1​∑izi​k2=1\\frac\{1\}\{G\-1\}\\sum\_\{i\}z\_\{ik\}^\{2\}=1\. By Equation[1](https://arxiv.org/html/2610.36437#S3.E1),ui=∑k≠jwk​zi​ku\_\{i\}=\\sum\_\{k\\neq j\}w\_\{k\}z\_\{ik\}, so the remaining components alone determine the whole order of such a group\.

The example\.LetG=8G=8, and let a binary primary gate one auxiliary, both at weight11\. In the first group, all responses are correct, seven with the auxiliary scoressand one withs−δs\-\\delta, whereδ\>0\\delta\>0\. The deviations from the mean areδ/8\\delta/8for the seven and−7δ/8\-7\\delta/8for the one, and the sample variance isdk2=17​\(7⋅δ264\+49​δ264\)=δ28d\_\{k\}^\{2\}=\\frac\{1\}\{7\}\\bigl\(7\\cdot\\frac\{\\delta^\{2\}\}\{64\}\+\\frac\{49\\,\\delta^\{2\}\}\{64\}\\bigr\)=\\frac\{\\delta^\{2\}\}\{8\}\. The one response haszi​k=−7​δ/8δ/8=−78≈−2\.47z\_\{ik\}=\-\\frac\{7\\delta/8\}\{\\delta/\\sqrt\{8\}\}=\-\\frac\{7\}\{\\sqrt\{8\}\}\\approx\-2\.47, which does not depend onδ\\delta, and sincezi​j=0z\_\{ij\}=0, this is itsuiu\_\{i\}\. In the second group, four responses are correct with auxiliary score11and four are wrong\. A wrong response has primary00, so the gate sets its auxiliary score tomin⁡\{si​k,0\}=0\\min\\\{s\_\{ik\},0\\\}=0, and both components score the group\(1,1,1,1,0,0,0,0\)\(1,1,1,1,0,0,0,0\), with mean12\\frac\{1\}\{2\}and sample variance17⋅8⋅14=27\\frac\{1\}\{7\}\\cdot 8\\cdot\\frac\{1\}\{4\}=\\frac\{2\}\{7\}\. Their standardized scores are±12/2/7=±7/8\\pm\\frac\{1\}\{2\}\\big/\\sqrt\{2/7\}=\\pm\\sqrt\{7/8\}, so a wrong response hasui=−2​7/8=−7/2≈−1\.87u\_\{i\}=\-2\\sqrt\{7/8\}=\-\\sqrt\{7/2\}\\approx\-1\.87\.

### C\.2Proof of Proposition[5\.1](https://arxiv.org/html/2610.36437#S5.Thmtheorem1)

Item \(i\) \(existence\)\.The setC≤​\(p\)C\_\{\\leq\}\(p\)is the intersection of the closed half\-spaces\{v:vi≥vi′\}\\\{v:v\_\{i\}\\geq v\_\{i^\{\\prime\}\}\\\}over the pairs withpi\>pi′p\_\{i\}\>p\_\{i^\{\\prime\}\}, or all ofℝG\\mathbb\{R\}^\{G\}if there is no such pair\. The boundary of each half\-space contains the origin, so each is a closed convex cone, and so is their intersection\. The vectors that are constant onHHform a linear subspaceSS, andC=​\(p\)=C≤​\(p\)∩SC\_\{=\}\(p\)=C\_\{\\leq\}\(p\)\\cap Sis again a closed convex cone\. A constant vector satisfies every inequality and every equality above, so both cones contain the constant vectors, among them00and±𝟏\\pm\\mathbf\{1\}\. By Fact 1,z^=ΠC​\(z\)\\hat\{z\}=\\Pi\_\{C\}\(z\)exists and is unique for everyz∈ℝGz\\in\\mathbb\{R\}^\{G\}\.

Item \(ii\) \(computation\)\.LetPSP\_\{S\}be the map that replaces the entries of a vector onHHwith their mean and leaves its other entries unchanged\. Lety=PS​zy=P\_\{S\}zif the ceiling is tied andy=zy=zif it is not, so thatyyis the result of the averaging step of Appendix[B\.2](https://arxiv.org/html/2610.36437#A2.SS2), and letw=ΠC≤​\(p\)​\(y\)w=\\Pi\_\{C\_\{\\leq\}\(p\)\}\(y\)\.*Step \(a\):wm≤wm′w\_\{m\}\\leq w\_\{m^\{\\prime\}\}for any two responsesmmandm′m^\{\\prime\}with the same primary andym≤ym′y\_\{m\}\\leq y\_\{m^\{\\prime\}\}\.*Letw′w^\{\\prime\}bewwwith the entriesmmandm′m^\{\\prime\}exchanged\. The two responses have the same primary, sow′∈C≤​\(p\)w^\{\\prime\}\\in C\_\{\\leq\}\(p\)\. Suppose thatwm\>wm′w\_\{m\}\>w\_\{m^\{\\prime\}\}, so thatw′≠ww^\{\\prime\}\\neq w\. Expanding the squares gives∥w′−y∥2−∥w−y∥2=2​\(wm−wm′\)​\(ym−ym′\)≤0\\lVert w^\{\\prime\}\-y\\rVert^\{2\}\-\\lVert w\-y\\rVert^\{2\}=2\\,\(w\_\{m\}\-w\_\{m^\{\\prime\}\}\)\(y\_\{m\}\-y\_\{m^\{\\prime\}\}\)\\leq 0, becauseym≤ym′y\_\{m\}\\leq y\_\{m^\{\\prime\}\}\. Thenw′∈C≤​\(p\)w^\{\\prime\}\\in C\_\{\\leq\}\(p\)is at least as close toyyaswwis, which contradicts the uniqueness ofww\.

*Step \(b\):w=z^w=\\hat\{z\}\.*If the ceiling is not tied, this holds by definition\. If it is tied, the vectorz−yz\-yis zero outsideHHand sums to zero overHH, where everyv∈Sv\\in Sis constant, so⟨z−y,v⟩=0\\langle z\-y,v\\rangle=0forv∈Sv\\in S\. Forv∈Sv\\in Sthe vectorv−yv\-ylies inSS, and therefore∥v−z∥2=∥v−y∥2\+∥y−z∥2\\lVert v\-z\\rVert^\{2\}=\\lVert v\-y\\rVert^\{2\}\+\\lVert y\-z\\rVert^\{2\}, whose last term does not depend onvv\. SinceC=​\(p\)⊂SC\_\{=\}\(p\)\\subset S, the point ofC=​\(p\)C\_\{=\}\(p\)closest tozzis the point ofC=​\(p\)C\_\{=\}\(p\)closest toyy, soz^=ΠC=​\(p\)​\(y\)\\hat\{z\}=\\Pi\_\{C\_\{=\}\(p\)\}\(y\)\. The vectoryyis constant onHH, whose responses share one primary, so step \(a\) giveswm≤wm′w\_\{m\}\\leq w\_\{m^\{\\prime\}\}andwm′≤wmw\_\{m^\{\\prime\}\}\\leq w\_\{m\}for allm,m′∈Hm,m^\{\\prime\}\\in H, andw∈C≤​\(p\)∩S=C=​\(p\)w\\in C\_\{\\leq\}\(p\)\\cap S=C\_\{=\}\(p\)\. The pointwwis closest toyyinC≤​\(p\)C\_\{\\leq\}\(p\)and lies in the subsetC=​\(p\)C\_\{=\}\(p\), and thereforew=ΠC=​\(p\)​\(y\)=z^w=\\Pi\_\{C\_\{=\}\(p\)\}\(y\)=\\hat\{z\}\.

*Step \(c\): the pooling step returnsww\.*Number the responses in the order of the pooling step’s sort, so thatp1≤p2≤⋯≤pGp\_\{1\}\\leq p\_\{2\}\\leq\\dots\\leq p\_\{G\}, andym≤ym\+1y\_\{m\}\\leq y\_\{m\+1\}wheneverpm=pm\+1p\_\{m\}=p\_\{m\+1\}\. LetK=\{v:v1≤v2≤⋯≤vG\}K=\\\{v:v\_\{1\}\\leq v\_\{2\}\\leq\\dots\\leq v\_\{G\}\\\}, which is a closed convex cone\. First,K⊂C≤​\(p\)K\\subset C\_\{\\leq\}\(p\): ifpi\>pi′p\_\{i\}\>p\_\{i^\{\\prime\}\}, theni\>i′i\>i^\{\\prime\}in this numbering, andvi≥vi′v\_\{i\}\\geq v\_\{i^\{\\prime\}\}forv∈Kv\\in K\. Second,wwlies inKK: ifpm<pm\+1p\_\{m\}<p\_\{m\+1\}, thenwm≤wm\+1w\_\{m\}\\leq w\_\{m\+1\}becausew∈C≤​\(p\)w\\in C\_\{\\leq\}\(p\), and ifpm=pm\+1p\_\{m\}=p\_\{m\+1\}, thenwm≤wm\+1w\_\{m\}\\leq w\_\{m\+1\}by step \(a\)\. Third,wwis closest toyyinC≤​\(p\)C\_\{\\leq\}\(p\)and lies in the subsetKK, sow=ΠK​\(y\)w=\\Pi\_\{K\}\(y\), which is the isotonic regression of the sequencey1,…,yGy\_\{1\},\\dots,y\_\{G\}\. After the sort, the pooling step applies the pool\-adjacent\-violators algorithm toy1,…,yGy\_\{1\},\\dots,y\_\{G\}, which computes the isotonic regression of a sequence\([Ayer et al\., 1955](https://arxiv.org/html/2610.36437#bib.bib59)\), so that the two steps together returnw=z^w=\\hat\{z\}\.

Item \(iii\) \(mean\)\.The coneCCcontains±𝟏\\pm\\mathbf\{1\}, so Equation[9](https://arxiv.org/html/2610.36437#A3.E9)withv=𝟏v=\\mathbf\{1\}and withv=−𝟏v=\-\\mathbf\{1\}gives⟨z−z^,𝟏⟩=0\\langle z\-\\hat\{z\},\\mathbf\{1\}\\rangle=0, and it follows that∑iz^i=∑izi\\sum\_\{i\}\\hat\{z\}\_\{i\}=\\sum\_\{i\}z\_\{i\}\.

Item \(iv\) \(energy\)\.By Equation[8](https://arxiv.org/html/2610.36437#A3.E8),∥z∥2=∥z^\+\(z−z^\)∥2=∥z^∥2\+∥z−z^∥2\\lVert z\\rVert^\{2\}=\\lVert\\hat\{z\}\+\(z\-\\hat\{z\}\)\\rVert^\{2\}=\\lVert\\hat\{z\}\\rVert^\{2\}\+\\lVert z\-\\hat\{z\}\\rVert^\{2\}, so∥z^∥≤∥z∥\\lVert\\hat\{z\}\\rVert\\leq\\lVert z\\rVert\. Ifz∈Cz\\in C, thenzzis itself the closest point ofCC, soz^=z\\hat\{z\}=z\. Conversely, ifz^=z\\hat\{z\}=z, thenz∈Cz\\in Cbecausez^∈C\\hat\{z\}\\in C\. Letz^≤=ΠC≤​\(p\)​\(z\)\\hat\{z\}^\{\\leq\}=\\Pi\_\{C\_\{\\leq\}\(p\)\}\(z\)andz^==ΠC=​\(p\)​\(z\)\\hat\{z\}^\{=\}=\\Pi\_\{C\_\{=\}\(p\)\}\(z\)\. SinceC=​\(p\)⊂C≤​\(p\)C\_\{=\}\(p\)\\subset C\_\{\\leq\}\(p\), the closest point of the smaller cone is at least as far fromzz, so∥z−z^=∥≥∥z−z^≤∥\\lVert z\-\\hat\{z\}^\{=\}\\rVert\\geq\\lVert z\-\\hat\{z\}^\{\\leq\}\\rVert\. The energy identity for each cone then gives∥z^=∥2=∥z∥2−∥z−z^=∥2≤∥z∥2−∥z−z^≤∥2=∥z^≤∥2\\lVert\\hat\{z\}^\{=\}\\rVert^\{2\}=\\lVert z\\rVert^\{2\}\-\\lVert z\-\\hat\{z\}^\{=\}\\rVert^\{2\}\\leq\\lVert z\\rVert^\{2\}\-\\lVert z\-\\hat\{z\}^\{\\leq\}\\rVert^\{2\}=\\lVert\\hat\{z\}^\{\\leq\}\\rVert^\{2\}\. Forz≠0z\\neq 0, the three shares of Equation[3](https://arxiv.org/html/2610.36437#S3.E3)are therefore nonnegative, and they sum to one, since∥z^≤∥2\\lVert\\hat\{z\}^\{\\leq\}\\rVert^\{2\}and∥z^=∥2\\lVert\\hat\{z\}^\{=\}\\rVert^\{2\}cancel\.

Item \(v\) \(order\)\.Sincez^∈C⊂C≤​\(p\)\\hat\{z\}\\in C\\subset C\_\{\\leq\}\(p\),pi\>pi′p\_\{i\}\>p\_\{i^\{\\prime\}\}impliesz^i≥z^i′\\hat\{z\}\_\{i\}\\geq\\hat\{z\}\_\{i^\{\\prime\}\}, so\(pi−pi′\)​\(z^i−z^i′\)≥0\(p\_\{i\}\-p\_\{i^\{\\prime\}\}\)\(\\hat\{z\}\_\{i\}\-\\hat\{z\}\_\{i^\{\\prime\}\}\)\\geq 0for every pair\. Letffbe a nondecreasing function andfi=f⁡\(pi\)f\_\{i\}=f\(p\_\{i\}\)\. Then\(fi−fi′\)​\(z^i−z^i′\)≥0\(f\_\{i\}\-f\_\{i^\{\\prime\}\}\)\(\\hat\{z\}\_\{i\}\-\\hat\{z\}\_\{i^\{\\prime\}\}\)\\geq 0for every pair as well: ifpi\>pi′p\_\{i\}\>p\_\{i^\{\\prime\}\}both factors are nonnegative, and ifpi=pi′p\_\{i\}=p\_\{i^\{\\prime\}\}the first factor is zero\. For any two vectorsa,ba,bwith meansa¯,b¯\\bar\{a\},\\bar\{b\},

∑i,i′\(ai−ai′\)​\(bi−bi′\)=2​G​∑i\(ai−a¯\)​\(bi−b¯\)\.\\sum\_\{i,i^\{\\prime\}\}\(a\_\{i\}\-a\_\{i^\{\\prime\}\}\)\(b\_\{i\}\-b\_\{i^\{\\prime\}\}\)=2G\\sum\_\{i\}\(a\_\{i\}\-\\bar\{a\}\)\(b\_\{i\}\-\\bar\{b\}\)\.Withai=fia\_\{i\}=f\_\{i\}andb=z^b=\\hat\{z\}, the left side is a sum of nonnegative terms, so the group covariance ofz^\\hat\{z\}withf⁡\(p\)f\(p\)is nonnegative\. Forf⁡\(p\)=pf\(p\)=panddj\>0d\_\{j\}\>0, the right side is2​G​dj​⟨z^,zj⟩2G\\,d\_\{j\}\\,\\langle\\hat\{z\},z\_\{j\}\\rangle, becausepi−p¯=dj​zi​jp\_\{i\}\-\\bar\{p\}=d\_\{j\}\\,z\_\{ij\}and∑izi​j=0\\sum\_\{i\}z\_\{ij\}=0, so⟨z^,zj⟩≥0\\langle\\hat\{z\},z\_\{j\}\\rangle\\geq 0\. Ifdj=0d\_\{j\}=0, thenzj=0z\_\{j\}=0and⟨z^,zj⟩=0\\langle\\hat\{z\},z\_\{j\}\\rangle=0\. ∎

### C\.3Proof of Corollary[5\.2](https://arxiv.org/html/2610.36437#S5.Thmtheorem2)

A response withpi=1p\_\{i\}=1passes and is at the ceiling, and a response withpi=0p\_\{i\}=0fails, soHHis the set of passes, and we writeFFfor the set of failures andn=\|H\|n=\\lvert H\\rvert\. Under the gate, a failure hasri​k=min⁡\{si​k,0\}=0r\_\{ik\}=\\min\\\{s\_\{ik\},0\\\}=0, while a pass hasri​k=si​k≥0r\_\{ik\}=s\_\{ik\}\\geq 0\.

*Part \(i\)\.*The only pairs withpi\>pi′p\_\{i\}\>p\_\{i^\{\\prime\}\}havei∈Hi\\in Handi′∈Fi^\{\\prime\}\\in F\. Ifdk=0d\_\{k\}=0, thenzk=0∈C≤​\(p\)z\_\{k\}=0\\in C\_\{\\leq\}\(p\), and ifdk\>0d\_\{k\}\>0, thenzi​k−zi′​k=\(ri​k−ri′​k\)/dk=si​k/dk≥0z\_\{ik\}\-z\_\{i^\{\\prime\}k\}=\(r\_\{ik\}\-r\_\{i^\{\\prime\}k\}\)/d\_\{k\}=s\_\{ik\}/d\_\{k\}\\geq 0for such a pair\. In both caseszk∈C≤​\(p\)z\_\{k\}\\in C\_\{\\leq\}\(p\), andΠC≤​\(p\)​\(zk\)=zk\\Pi\_\{C\_\{\\leq\}\(p\)\}\(z\_\{k\}\)=z\_\{k\}by item \(iv\) of Proposition[5\.1](https://arxiv.org/html/2610.36437#S5.Thmtheorem1)\. The share against the order in Equation[3](https://arxiv.org/html/2610.36437#S3.E3)is therefore zero on a binary primary\. If none of the task’s gated pairs has a tied ceiling, Equation[6](https://arxiv.org/html/2610.36437#S4.E6)gives its groups the summed scores of Equation[1](https://arxiv.org/html/2610.36437#S3.E1)\.

*Part \(ii\)\.*Both passes and failures are present, so1≤n≤G−11\\leq n\\leq G\-1anddj\>0d\_\{j\}\>0\. Ifdk=0d\_\{k\}=0, thenzk=0z\_\{k\}=0andz^k=0\\hat\{z\}\_\{k\}=0, and the claim holds withc=0c=0\. Letdk\>0d\_\{k\}\>0, and lets¯H=1n​∑i∈Hri​k≥0\\bar\{s\}\_\{H\}=\\frac\{1\}\{n\}\\sum\_\{i\\in H\}r\_\{ik\}\\geq 0be the mean auxiliary score of the passes\. WithSSandPSP\_\{S\}as in the proof of Proposition[5\.1](https://arxiv.org/html/2610.36437#S5.Thmtheorem1), the vectory=PS​zky=P\_\{S\}z\_\{k\}has the entries\(s¯H−r¯k\)/dk\(\\bar\{s\}\_\{H\}\-\\bar\{r\}\_\{k\}\)/d\_\{k\}onHHand−r¯k/dk\-\\bar\{r\}\_\{k\}/d\_\{k\}onFF\. It is constant onHH, and its entries onHHexceed those onFFbys¯H/dk≥0\\bar\{s\}\_\{H\}/d\_\{k\}\\geq 0, soy∈C=​\(p\)y\\in C\_\{=\}\(p\)\. By step \(b\) of that proof and item \(iv\),z^k=ΠC=​\(p\)​\(y\)=y\\hat\{z\}\_\{k\}=\\Pi\_\{C\_\{=\}\(p\)\}\(y\)=y\. The vectorz^k\\hat\{z\}\_\{k\}is constant onHHand constant onFF, and it sums to zero by item \(iii\)\. The same holds forzjz\_\{j\}, whose entries are\(1−nG\)/dj\(1\-\\frac\{n\}\{G\}\)/d\_\{j\}onHHand−nG/dj\-\\frac\{n\}\{G\}/d\_\{j\}onFF, andzj≠0z\_\{j\}\\neq 0\. The vectors that are constant onHH, constant onFFand sum to zero form a line, soz^k=c​zj\\hat\{z\}\_\{k\}=c\\,z\_\{j\}withc=⟨z^k,zj⟩/∥zj∥2c=\\langle\\hat\{z\}\_\{k\},z\_\{j\}\\rangle/\\lVert z\_\{j\}\\rVert^\{2\}\. Sincezj∈Sz\_\{j\}\\in Sandzk−PS​zkz\_\{k\}\-P\_\{S\}z\_\{k\}is orthogonal toSS,⟨z^k,zj⟩=⟨zk,zj⟩\\langle\\hat\{z\}\_\{k\},z\_\{j\}\\rangle=\\langle z\_\{k\},z\_\{j\}\\rangle, which givesc=⟨zk,zj⟩/∥zj∥2c=\\langle z\_\{k\},z\_\{j\}\\rangle/\\lVert z\_\{j\}\\rVert^\{2\}\. The entries onHHexceed those onFFbys¯H/dk\\bar\{s\}\_\{H\}/d\_\{k\}inz^k\\hat\{z\}\_\{k\}and by1/dj1/d\_\{j\}inzjz\_\{j\}, soc=s¯H​dj/dk≥0c=\\bar\{s\}\_\{H\}\\,d\_\{j\}/d\_\{k\}\\geq 0\. Since∥zk∥2=∥zj∥2=G−1\\lVert z\_\{k\}\\rVert^\{2\}=\\lVert z\_\{j\}\\rVert^\{2\}=G\-1,c=⟨zk,zj⟩/\(∥zk∥​∥zj∥\)c=\\langle z\_\{k\},z\_\{j\}\\rangle/\(\\lVert z\_\{k\}\\rVert\\,\\lVert z\_\{j\}\\rVert\)is their cosine, soc≤1c\\leq 1\.

*Part \(iii\)\.*In a group of passes only, every response is inHH, soC=​\(p\)C\_\{=\}\(p\)is the set of constant vectors\. The constant vector closest tozkz\_\{k\}is its mean times𝟏\\mathbf\{1\}, which is zero\. ∎

### C\.4Remarks on the Statements

The factor1\+c1\+cthat follows Corollary[5\.2](https://arxiv.org/html/2610.36437#S5.Thmtheorem2)is22exactly when the group’s correct responses share one positive auxiliary score\. Take a binary primary, a group with both passes and failures, and an auxiliary withdk\>0d\_\{k\}\>0\. By the proof of part \(ii\) of the corollary, the tied ceiling keepsz^k=PS​zk\\hat\{z\}\_\{k\}=P\_\{S\}z\_\{k\}, so the partzk−z^kz\_\{k\}\-\\hat\{z\}\_\{k\}that it removes is zero on the failures and equals\(ri​k−s¯H\)/dk\(r\_\{ik\}\-\\bar\{s\}\_\{H\}\)/d\_\{k\}on the passes\. PAPO’s standardization in Equation[11](https://arxiv.org/html/2610.36437#A4.E11)divides the same differencesri​k−s¯Hr\_\{ik\}\-\\bar\{s\}\_\{H\}by the standard deviation over the passes and sets the failures to zero\. Up to a positive factor, it therefore keeps exactly the part ofzkz\_\{k\}that the tied ceiling removes, and it drops the partz^k=c​zj\\hat\{z\}\_\{k\}=c\\,z\_\{j\}that the tied ceiling keeps instead\.

## Appendix DExperimental Details

In this section, we describe the data and rewards, the training runs and baselines, and the evaluation\.

Task groupBenchmarkProblemsResponsesResponse LenTemperatureTop\-ppOtherMetricJudgeMathAIME24303232,7680\.60\.95–accuracy–AIME25303232,7680\.60\.95–accuracy–MATH\-5005003232,7680\.60\.95–accuracy–OlympiadBench6743232,7680\.60\.95–accuracy–Minerva2723232,7680\.60\.95–accuracy–AMC23403232,7680\.60\.95–accuracy–CodeHumanEval\+1641032,7680\.60\.95–pass@1–MBPP\+3781032,7680\.60\.95–pass@1–LiveCodeBench v61751032,7680\.60\.95–pass@1–IFIFEval541132,7680\.60\.95–instruction\-level strict accuracy–IFBench300132,7680\.60\.95–prompt\-level loose accuracy–GuideBench1,042132,76801\.0–accuracy–WritingWritingBench1,000132,7680\.70\.8top\-kk20overall scoreGemma\-4\-26B\-A4B\-ITCreative Writing v396132,7680\.71\.0min\-pp0\.1rubric scoreGemma\-4\-26B\-A4B\-ITLongBench\-Write120132,7680\.51\.0–overall scoreGemma\-4\-26B\-A4B\-IT

Table 3:Settings of the evaluation for each benchmark\. Every metric is reported on a scale from 0 to 100\.### D\.1Data and Rewards

Data\.Math prompts come from DeepScaleR\([Luo et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib24)\)and DAPO\-Math\-17k\([Yu et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib3)\), code prompts from Eurus\-2\-RL\([Cui et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib16)\), Dolci\-RL\-Zero\-Code\([Team Olmo et al\., 2025](https://arxiv.org/html/2610.36437#bib.bib17)\), and Open\-R1\([Hugging Face, 2025](https://arxiv.org/html/2610.36437#bib.bib25)\), and IF and writing prompts from RubricHub\([Li et al\., 2026b](https://arxiv.org/html/2610.36437#bib.bib23)\)\. We deduplicate prompts across tasks, remove exact and embedded copies of evaluation questions, and keep prompts of at most 2,048 tokens\. For each base policy, we sample eight*base responses*to every candidate prompt at temperature11and top\-pp11, cut them off at the model’s response limit in training, and score them with the primaries and, on writing, with the reward model \(RM\)\. On math, code, and writing, we drop the prompts whose eight base responses all end before the limit with a primary of11, and those on which no base response passes make up a fixed 10% of each task’s training prompts\. On IF, we set no quota and keep the prompts whose base responses all have a primary of11, since the judged criterion can still separate their responses\. Each of the four tasks has 9,838 training prompts onQwen3\-4B\-Base, 11,705 onQwen3\-8B\-Base, and 10,850 onOLMo\-3\-7B\-Instruct\-SFT\.

The length reward\.For responseiiof\|yi\|\|y\_\{i\}\|generated tokens to a promptxxwith*reference length*ℓx\\ell\_\{x\}, the length reward’s own scoresi​ks\_\{ik\}is given by

si​k=max⁡\{0,1−\(\(\|yi\|−ℓx\)\+ℓx\)2\},s\_\{ik\}=\\max\\Bigl\\\{0,\\;1\-\\Bigl\(\\frac\{\(\|y\_\{i\}\|\-\\ell\_\{x\}\)\_\{\+\}\}\{\\ell\_\{x\}\}\\Bigr\)^\{2\}\\Bigr\\\},\(10\)where\(a\)\+=max⁡\{a,0\}\(a\)\_\{\+\}=\\max\\\{a,0\\\}\. The reference lengthℓx\\ell\_\{x\}is the mean number of generated tokens over the prompt’s*reference set*, which consists of its base responses that end before the limit and have a primary of11\. On code, IF, and writing, a prompt with fewer than two such responses instead uses its base responses that end before the limit and pass\. A prompt whose reference set is empty takes a task\-level reference length\. Over the task’s prompts with a reference set, we regress the logarithm of the reference length on the share of base responses outside the set\. The task\-level length is the fit at a share of11if the95%95\\%bootstrap interval of the slope over 2,000 resamples lies above zero, and it is the mean reference length of these prompts otherwise\.

Judge and reward model\.The training judge isGemma\-4\-26B\-A4B\-IT\([Gemma Team et al\., 2026](https://arxiv.org/html/2610.36437#bib.bib29)\), and the RM isSkywork\-Reward\-V2\-Qwen3\-8B\([Liu et al\., 2026a](https://arxiv.org/html/2610.36437#bib.bib18)\)\. The RM’s raw scoreρ\\rhois rescaled to\(ρ−ρlo\)/\(ρhi−ρlo\)\(\\rho\-\\rho\_\{\\mathrm\{lo\}\}\)/\(\\rho\_\{\\mathrm\{hi\}\}\-\\rho\_\{\\mathrm\{lo\}\}\)and clipped to\[0,1\]\[0,1\], whereρlo\\rho\_\{\\mathrm\{lo\}\}andρhi\\rho\_\{\\mathrm\{hi\}\}are the 1st and 99th percentiles of its raw scores on all base responses of the training writing prompts, and they stay fixed for all runs of a base policy throughout their training\.

Table 4:Hyperparameters of the runs\.
### D\.2Training, Runs, and Baselines

Training\.Table[4](https://arxiv.org/html/2610.36437#A4.T4)lists the training hyperparameters\. The estimator’s advantages enter a clipped surrogate objective\([Schulman et al\., 2017](https://arxiv.org/html/2610.36437#bib.bib2)\), without a KL or entropy term\. Every run except MT\-GRPO excludes the groups with identical component scores before applying its estimator\. Each run uses 4 nodes of 8 NVIDIA RTX PRO 6000 GPUs, which also serve the training judge and the RM\.

Table 5:The gated auxiliaries under GDPO at the declared weights onQwen3\-4B\-Base, early \(steps 1–20\) and late \(the last 20\)\. Passing at ceiling is the share of passing responses at the ceiling, and Agreement is pooled over the run\.Table 6:What earned trust reads and decides for each gated pair: the share of passing responses at the ceiling over the last 20 training steps, the pooled agreement of CONCUR\-A at its last training step, each run’s pooled agreement in standard errors, which CONCUR\-T and CONCUR\-O only log, and the share of all training steps at which CONCUR\-A leaves the pair’s ceiling not tied\.Runs and baselines\.On each model, we run the seven estimators at the declared weights, the two variants of CONCUR, CONCUR\-O, the reference, GDPO \(w=0\.5w=0\.5\), and GRPO \(no std\)\. All runs on a model, including the reference, share the training prompts, the training recipe, and the rewards with their gates\. Each baseline uses the settings of its paper or public code where reported and the shared recipe otherwise\. MT\-GRPO keeps its own filter and its ratio\-preserving rejection sampling with the settings of its released script, so it generates more responses per step than every other run\. All runs whiten their summed scores over the batch except the five whose estimators lack this step: GRPO, GRPO \(no std\), GEOMA, MT\-GRPO, and PAPO\.

PAPO\.We apply PAPO’s advantage rule with a task’s primary as its outcome reward, the passing responsesP=\{i:pi≥12\}P=\\\{i:p\_\{i\}\\geq\\frac\{1\}\{2\}\\\}as its correct set, and each gated auxiliaryk∈𝒜tk\\in\\mathcal\{A\}\_\{t\}as a process reward:

z~i​k\\displaystyle\\tilde\{z\}\_\{ik\}=\{\(ri​k−r¯kP\)/dkPi∈P,0i∉P,\\displaystyle=\\begin\{cases\}\(r\_\{ik\}\-\\bar\{r\}^\{P\}\_\{k\}\)/d^\{P\}\_\{k\}&i\\in P,\\\\ 0&i\\notin P,\\end\{cases\}\(11\)Ai\\displaystyle A\_\{i\}=∑k∈𝒦t∖𝒜twk​zi​k\+∑k∈𝒜twk​z~i​k,\\displaystyle=\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\\setminus\\mathcal\{A\}\_\{t\}\}w\_\{k\}z\_\{ik\}\+\\sum\_\{k\\in\\mathcal\{A\}\_\{t\}\}w\_\{k\}\\tilde\{z\}\_\{ik\},wherer¯kP\\bar\{r\}^\{P\}\_\{k\}anddkPd^\{P\}\_\{k\}are the mean and the sample standard deviation of the auxiliary overPP, andz~i​k\\tilde\{z\}\_\{ik\}is zero on every response of a group in which fewer than two responses pass ordkPd^\{P\}\_\{k\}is numerically zero\.

### D\.3Evaluation

Table[3](https://arxiv.org/html/2610.36437#A4.T3)gives each benchmark’s settings, metric, and judge, which are the same for every run\. Creative Writing v3 is scored by its rubric alone, and LongBench\-Write’s score is the mean of a quality score from the judge and a length score\. The training judge scores each writing benchmark by its own criteria, and the second judge isQwen3\.6\-35B\-A3B\([Qwen Team, 2026](https://arxiv.org/html/2610.36437#bib.bib72)\)with thinking off\. Table[8](https://arxiv.org/html/2610.36437#A5.T8)gives the scores of every run under this judge\.

## Appendix EAdditional Results

In this section, we report the training statistics behind the motivating experiment, the decisions of earned trust, the shares of energy and of the update during training, and the results of the two variants, the ablation, and the reference on all three models\.

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/aux-qwen3-8b-base.png)

![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/aux-olmo-3-7b-it-sft.png)

Figure 5:Each gated auxiliary’s mean score over the passing responses onQwen3\-8B\-Base\(left\) andOLMo\-3\-7B\-Instruct\-SFT\(right\): the mean over the last 20 training steps minus the base policy’s level of the same model\.![Refer to caption](https://arxiv.org/html/2610.36437v1/figures/primary.png)Figure 6:The primary’s mean score over all responses during training onQwen3\-8B\-Base\(top\) andOLMo\-3\-7B\-Instruct\-SFT\(bottom\) under GDPO, the reference, and CONCUR, shown as 10\-step running means\.MathCodeIFWritingAIME24AIME25MATHOBenchMinervaAMC23MeanHEval\+MBPP\+LCB\-v6MeanIFEvalIFBenchGBenchMeanWBenchCW\-v3LBWriteMeanAvgQwen3\-4B\-BaseGDPO9\.069\.9079\.2944\.6645\.4859\.6141\.3381\.8370\.8724\.4059\.0385\.7336\.3380\.6167\.5671\.9445\.9769\.0162\.3154\.31GDPO \(w=0\.5w=0\.5\)14\.5810\.3181\.7948\.4646\.1258\.3643\.2782\.0171\.1924\.1759\.1286\.8139\.3379\.5668\.5770\.7438\.1167\.4158\.7554\.60GDPO \(no aux\.\)23\.4418\.7584\.7653\.1649\.5265\.7049\.2285\.4972\.3324\.7460\.8584\.7740\.3375\.9167\.0167\.3539\.3364\.3657\.0156\.66GRPO \(no std\)12\.408\.3379\.4944\.8145\.8054\.5340\.8980\.4971\.1924\.4658\.7187\.2933\.6778\.7966\.5865\.6342\.5366\.0058\.0653\.03CONCUR\-O11\.8814\.5880\.3245\.7346\.2554\.1442\.1581\.1069\.6624\.1758\.3186\.6939\.0078\.9868\.2277\.3145\.4277\.7866\.8455\.53CONCUR\-T19\.4816\.3585\.5152\.8549\.1269\.2248\.7684\.0268\.6024\.7459\.1287\.7737\.6779\.1768\.2080\.1947\.8477\.7468\.5958\.69CONCUR\-A20\.5218\.7585\.2853\.5849\.2269\.3849\.4582\.4471\.5125\.1459\.7088\.1340\.3377\.3568\.6079\.4344\.7176\.3166\.8258\.80Qwen3\-8B\-BaseGDPO21\.3514\.3882\.9949\.3249\.6162\.1946\.6484\.0974\.5226\.2361\.6188\.1339\.3384\.5570\.6776\.0952\.9069\.5766\.1958\.35GDPO \(w=0\.5w=0\.5\)19\.2715\.4284\.0651\.1251\.2263\.4447\.4283\.7872\.8326\.8061\.1487\.7737\.6783\.9769\.8075\.0451\.5671\.2065\.9358\.34GDPO \(no aux\.\)27\.9223\.9690\.2459\.9253\.9877\.1155\.5285\.9874\.8727\.3762\.7486\.5733\.6781\.3867\.2175\.0043\.8965\.9561\.6160\.52GRPO \(no std\)20\.6314\.7984\.1351\.5050\.8063\.5947\.5786\.3476\.0324\.9762\.4589\.4536\.3384\.8470\.2170\.3346\.6671\.0662\.6858\.10CONCUR\-O19\.6916\.5683\.4350\.0449\.7864\.6947\.3785\.3075\.1127\.0962\.5089\.2138\.6782\.9270\.2686\.7851\.6482\.9773\.8060\.26CONCUR\-T31\.2523\.9689\.6159\.5555\.4078\.5256\.3887\.3874\.8727\.2063\.1588\.8539\.3383\.4070\.5387\.1952\.4281\.2273\.6164\.01CONCUR\-A28\.7523\.1390\.1360\.3754\.1978\.9155\.9182\.7473\.2026\.3460\.7688\.1338\.6783\.4070\.0688\.0649\.9281\.6073\.1963\.17OLMo\-3\-7B\-Instruct\-SFTGDPO27\.8122\.6085\.2455\.8940\.3669\.5350\.2476\.3464\.7623\.1454\.7586\.4529\.6750\.8655\.6665\.3147\.1463\.2758\.5753\.89GDPO \(w=0\.5w=0\.5\)29\.5827\.2987\.0457\.5141\.5373\.9852\.8277\.7464\.9224\.8055\.8287\.5332\.6751\.9257\.3766\.5048\.1063\.7359\.4455\.66GDPO \(no aux\.\)38\.3333\.6590\.8863\.7446\.1583\.9159\.4485\.1870\.5031\.5462\.4184\.7733\.3355\.7657\.9555\.4638\.2155\.9849\.8957\.83GRPO \(no std\)31\.7727\.6087\.5158\.4441\.8173\.0553\.3677\.3865\.3423\.6055\.4487\.7731\.3355\.4758\.1963\.1846\.9867\.5359\.2355\.92CONCUR\-O26\.7722\.0884\.3653\.2839\.9267\.5048\.9976\.6564\.5025\.6655\.6087\.2931\.6760\.1759\.7170\.2149\.7872\.0464\.0155\.46CONCUR\-T34\.4829\.9089\.4161\.5744\.0579\.8456\.5483\.2971\.0629\.0361\.1387\.6533\.6758\.5459\.9566\.1449\.0469\.2361\.4759\.13CONCUR\-A36\.3532\.1990\.5362\.4144\.8183\.5258\.3082\.7467\.5430\.3460\.2187\.7732\.0056\.9158\.8965\.4549\.2168\.4961\.0559\.35

Table 7:Evaluation scores of GDPO, GDPO \(w=0\.5w=0\.5\), the reference GDPO \(no aux\.\), GRPO \(no std\), CONCUR\-O, and the two variants of CONCUR on every benchmark, with the same benchmark names and formatting as Table[1](https://arxiv.org/html/2610.36437#S6.T1)\.Table 8:Writing benchmarks and Avg under the second judge, with the names and formatting of Table[1](https://arxiv.org/html/2610.36437#S6.T1)\.### E\.1Gated Auxiliaries under GDPO

Table[5](https://arxiv.org/html/2610.36437#A4.T5)lists the training statistics behind the three observations of the motivating experiment: the cosine of \(O1\), the shares of \(O2\), and the agreement of \(O3\)\. The share at the ceiling grows on IF to nearly its value on Math and stays small on Code\. A training statistic of one step pools a task’s groups in that step’s batch: in a cosine or a share of Equation[3](https://arxiv.org/html/2610.36437#S3.E3), each inner product and each squared norm is summed over the groups before the ratio is taken\. The share of passing responses at the ceiling is taken over the task’s passing responses in the batch, and early and late values are means over 20 steps\. Among a task’s groups that enter the update, the share with all responses at the ceiling is 0\.13 early and 0\.16 late on math, 0\.05 and 0\.02 on code, and 0\.03 and 0\.20 on IF, and it stays 0\.00 on writing\.

An auxiliary independent of the primary\.Let a group’s primaries be distinct, and let the auxiliary’s scores be independent normal draws that do not depend on the primary\. For a standard normal vector inℝG\\mathbb\{R\}^\{G\}, the expected squared norm of its projection ontoC≤​\(p\)C\_\{\\leq\}\(p\)isHG=∑m=1G1/mH\_\{G\}=\\sum\_\{m=1\}^\{G\}1/m, of which the constant vectors account for11, and the vector’s direction is independent of its norm\. The projection therefore keeps an expected share\(HG−1\)/\(G−1\)\(H\_\{G\}\-1\)/\(G\-1\)of the auxiliary’s energy, and the remainder is the expected share against the order, which rounds to0\.750\.75at the group size ofG=8G=8\.

### E\.2Trust during Training

A decision of earned trust changes a group’s scores only if at least two of its responses are at the ceiling\. Over the last 20 training steps of the CONCUR\-A run onQwen3\-4B\-Base, all responses are at the ceiling in a share 0\.21 of the IF groups and 0\.03 of the code groups that enter the update\. In these groups, the auxiliary keeps GDPO’s standardized scores at the steps where the pair’s tie is lifted\.

### E\.3Shares during Training

Over the last 20 training steps of the CONCUR\-A run onQwen3\-4B\-Base, the share against the order is zero on math by construction, 0\.02 on code, 0\.04 on IF, 0\.67 for writing’s length reward, and 0\.58 for its RM\. Over the same steps, the share at the ceiling is 0\.76 on math, 0\.05 on code, 0\.18 on IF, and no more than 0\.01 for writing’s two gated auxiliaries\.

Shares of the update\.Over the last 20 training steps, math’s share of the batch’s summed squared advantages is 0\.27, 0\.28, and 0\.24 under GDPO and 0\.18, 0\.12, and 0\.10 under CONCUR\-T onQwen3\-4B\-Base,Qwen3\-8B\-Base, andOLMo\-3\-7B\-Instruct\-SFT, respectively\.

### E\.4Variants, Ablation, and Reference

Table[8](https://arxiv.org/html/2610.36437#A5.T8)gives every benchmark for GDPO, GDPO \(w=0\.5w=0\.5\), the reference, GRPO \(no std\), CONCUR\-O, and the two variants of CONCUR\. Figure[5](https://arxiv.org/html/2610.36437#A5.F5)shows the auxiliaries’ scores onQwen3\-8B\-BaseandOLMo\-3\-7B\-Instruct\-SFT, as Figure[3](https://arxiv.org/html/2610.36437#S6.F3)\(d\) does onQwen3\-4B\-Base\. In both figures, the*base policy’s level*of a gated pair is the auxiliary’s mean score over the task’s passing responses at the first step, averaged over the model’s runs, and a run’s change is its mean score over the last 20 training steps minus this level\. Under GDPO \(w=0\.5w=0\.5\), which both figures leave out, the length reward’s change on math is \+0\.10, \+0\.10, and−\-0\.20 on the three models in that order, compared with \+0\.12, \+0\.13, and−\-0\.13 under GDPO\. As the bottom row of Figure[4](https://arxiv.org/html/2610.36437#S6.F4)does onQwen3\-4B\-Base, Figure[6](https://arxiv.org/html/2610.36437#A5.F6)shows the primary’s mean score over all responses onQwen3\-8B\-BaseandOLMo\-3\-7B\-Instruct\-SFTfor the same runs\. OnOLMo\-3\-7B\-Instruct\-SFT, where code’s share at the ceiling is the largest, code’s pass rate in training also ends higher under both variants than under GDPO and CONCUR\-O\.