MatrixReward：基于评分标准矩阵的开放式生成奖励方法

# MatrixReward: Reward from Rubric Matrix for Open-Ended Generation
Source: [https://arxiv.org/html/2610.00389](https://arxiv.org/html/2610.00389)
Zihan Shen1,3,∗,Qi Liu2,∗,Zixuan Yang2,Yiqun Chen2,Chenglong Zhao3,Xiaozhao Wang3,Lei He3,†1Zhejiang University2Renmin University of China3Qwen Business Unit of Alibaba

###### Abstract

Open\-ended query generation lacks standard answers, thus necessitating an effective reward mechanism\. Pointwise scoring rubrics provide limited information about the relative quality of sample answers under the same prompt; merging multiple rubric judgments into a single score may also mask the differences between these answers\. We proposeMatrixReward, which constructs rewards from a rollout\-by\-rubric win\-rate matrix obtained by comparing every pair of sampled responses under each rubric\. The spread of each matrix column captures how strongly that rubric distinguishes the current rollouts, while correlations between columns reveal rubric repetition; together, these statistics yield data\-dependent rubric weights\. We combine these weights with the prior weights of rubrics\. After column normalization and weighting, the observed per\-rubric maxima and minima define positive and negative ideal profiles\. Each rollout’s distances to these two ideals determine its relative\-closeness quality reward\. Evaluated using Qwen3\-8B on four open\-ended query\-answering benchmarks, MatrixReward achieves an average score of 63\.02, outperforming the strongest baseline by approximately 2\.0%\. These results support the idea that matrices derived from relative comparisons can be used to construct rewards more reasonably for open\-ended generative reinforcement learning\.111Code and materials required to reproduce our results are available at[https://github\.com/Backsingles/MatrixReward](https://github.com/Backsingles/MatrixReward)\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\.## 1Introduction

Reinforcement learning has become an important approach to improving language models beyond their initial capabilities, but its effectiveness depends on the quality of the reward signal\([Ouyang et al\., 2022](https://arxiv.org/html/2610.00389#bib.bib9);[Dubois et al\., 2023](https://arxiv.org/html/2610.00389#bib.bib3);[Shao et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib1)\)\. In verifiable tasks, such as mathematics and programming, a correct answer or successful test provides a concrete target\. In open\-ended generation, feedback must instead account for instruction following, factual grounding, relevance, and presentation\([Zheng et al\., 2023](https://arxiv.org/html/2610.00389#bib.bib5);[Gunjal et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib7)\)\. A response may be fluent yet uninformative, or detailed yet poorly grounded\. Effective rewards must distinguish these qualities among responses to the same prompt\.

Rubric\-based rewards offer a promising foundation by expressing these requirements as explicit evaluation rubrics\([Gunjal et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib7);[Zhang et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib8);[Liu et al\., 2026c](https://arxiv.org/html/2610.00389#bib.bib16)\)\. Rather than leaving quality entirely to a holistic judgment, rubrics identify the aspects that an evaluator should consider\. However, clear rubrics do not automatically produce informative training signals: an evaluator must still determine how well each response satisfies them\([Zheng et al\., 2023](https://arxiv.org/html/2610.00389#bib.bib5);[Tripathi et al\., 2025](https://arxiv.org/html/2610.00389#bib.bib18)\), and those assessments must ultimately guide a policy update\. The distinction matters for group\-relative policy optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib1)\): the training signal depends on differences among sampled responses, not merely on whether each response appears satisfactory in isolation\.

Many rubric\-based reward systems assign pointwise scores to responses independently\. Although rubrics specify what to judge, a numerical score lacks a stable anchor when a query has no reference answer; the same value may reflect different quality standards across queries or evaluators\([Zheng et al\., 2023](https://arxiv.org/html/2610.00389#bib.bib5);[Dubois et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib4)\)\. Rubric weights is also commonly set when the rubrics are created\. These weights represent the factors that should be considered in the response, but they cannot guarantee the differentiation between rollouts sampled during training; and if the rubric is highly correlated, it may lead to duplicate scoring\. More importantly, a linearly weighted average of rubrics may cause scores across different dimensions to cancel each other out, assigning the same reward to responses of different quality under the same rubric\.

![Refer to caption](https://arxiv.org/html/2610.00389v1/figures/comparison_corrected.png)Figure 1:Comparison of reward construction for the same rollout group\.To address these problems, we propose MatrixReward\. As illustrated in Figure[1](https://arxiv.org/html/2610.00389#S1.F1), rubric\-wise pairwise judgments retain a separate win\-rate profile for each response\. The matrix reveals which rubrics distinguish the current rollouts and which provide overlapping information\. We combine this evidence with query\-specific weights, then compare the resulting profiles with positive and negative ideals to construct rewards\.

Our contributions are as follows:

- •Rubric\-wise comparison matrix\.We introduce MatrixReward, a reward framework for open\-ended group\-based RL\. By comparing every pair of rollouts under each query\-specific rubric, it constructs a rollout\-by\-rubric win\-rate matrix that retains each response’s relative quality\.
- •Distance\-based reward construction\.We derive group\-specific rubric weights from column dispersion and cross\-rubric correlation, then combine them with query\-specific importance priors\. After normalizing and weighting the matrix columns, we use their observed maxima and minima as positive and negative ideal profiles\. For each rollout, the Euclidean distance to the negative ideal divided by the sum of its distances to both ideals gives a scalar quality reward, favoring profiles near the positive ideal and far from the negative ideal\.
- •Four\-benchmark evaluation\.Training Qwen3\-8B with MatrixReward yields a 63\.02 average across four benchmarks, exceeding the strongest pairwise, pointwise, and weight\-aware baselines by 1\.23, 6\.93, and 5\.60 points, respectively\. Ablations examine weight extraction and fusion, ideal\-profile selection, distance metrics, and judging cost\.

## 2Related Work

#### Rubric\-based reward evaluation\.

Rubrics make the standards for judging open\-ended responses explicit, but do not by themselves establish a common numeric scale across queries or evaluators\. Rubrics as Rewards \(RaR\) uses instance\-specific rubrics in on\-policy RL, with both weighted rubric\-level assessments and rubric\-conditioned holistic judgments\([Gunjal et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib7)\)\. Chasing the Tail improves discrimination among high\-quality responses by constructing rubrics from diverse examples\([Zhang et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib8)\); OpenRubrics derives rubrics from preference data\([Liu et al\., 2026c](https://arxiv.org/html/2610.00389#bib.bib16)\)\. These works improve what is judged, while the choice between pointwise rubric scores and pairwise comparisons remains consequential for reward construction\.

#### Pairwise and rubric\-wise comparisons\.

Pairwise judgments replace an isolated score with a choice between responses\. Tournament\-GRPO converts rubric\-guided comparisons among same\-query rollouts into tournament rewards\([Yang et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib22)\); ArenaRL uses process\-aware, multi\-level rubrics within tournament\-based relative ranking\([Zhang et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib25)\)\. OpenRS uses adaptive rubrics to guide pairwise comparisons and aggregates the resulting preferences for RL supervision\([Jia et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib30)\)\. MAPLE combines aspect\-wise pairwise judgments while estimating evaluator reliability and aspect importance for assessment rather than online reward construction\([Jinji et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib17)\)\.

#### Adaptive weighting of reward dimensions\.

Human\-assigned rubric weight need not coincide with a rubric’s current usefulness for learning\. POW3R makes this distinction explicit and reallocates rubric\-level reward weights using contrast among policy rollouts\([Tyagi et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib15)\)\. SAW uses batch\-level reward variation to adapt dimension weights\([He et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib13)\), while SA\-MRPO downweights saturated objectives at the advantage level\([Wang et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib12)\)\. AMRP shifts weight toward dimensions with greater shortfall, volatility, or stalled progress\([Yuan et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib14)\)\. These methods establish that weights can respond to training state; they differ in the statistics used and in whether they account for redundancy between rubric dimensions within the current response group\.

#### Aggregation of multi\-dimensional feedback\.

After assigning accurate weights to the rubric, it’s still necessary to consider how to construct a suitable reward for reinforcement learning training from the evaluations of multiple rubrics\. GDPO normalizes reward dimensions separately before aggregation\([Liu et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib10)\), and GD2PO additionally handles conflicting cross\-dimension advantages\([Liu et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib11)\)\. OpenRS aggregates rubric\-level preferences outside the judge\([Jia et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib30)\), whereas AMRP identifies how fixed scalar projections can conflate distinct reward profiles\([Yuan et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib14)\)\.

## 3Method

Figure[2](https://arxiv.org/html/2610.00389#S3.F2)illustrates the workflow of MatrixReward\. For each query, we compare every pair of rollouts under each rubric and organize their rubric\-wise win rates into a matrix\. We then extract data\-driven rubric weights from this matrix and combine them with the weights specified when the rubrics were created\. Finally, we compare the weighted response profiles with the observed positive and negative ideal profiles in the multidimensional rubric space, using their relative distances to construct a reward for each rollout\.

Figure 2:Overview of the MatrixReward framework\.### 3\.1Constructing the rubric\-wise comparison matrix

For a queryxx, the policyπθ\\pi\_\{\\theta\}samplesGGresponsesY=\{yi\}i=1GY=\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\. The query hasK=K⁡\(x\)K=K\(x\)rubrics\{ck,tk\}k=1K\\\{c\_\{k\},t\_\{k\}\\\}\_\{k=1\}^\{K\}, each with a positive stored importance weighttkt\_\{k\}\. The same rubrics evaluate every response in the group\.

For responsesiiandjj, a judge compares them under rubricckc\_\{k\}\. In the explicit protocol, each request asks for a verdict on one rubric\. The judge may prefer either response or declare a tie\. For responseyiy\_\{i\}, we record

zi​j​k=\{1,if​yi​wins against​yj​under​ck,12,if the judge declares a tie,0,if​yj​wins against​yi​under​ck,zj​i​k=1−zi​j​k\.z\_\{ijk\}=\\begin\{cases\}1,&\\text\{if \}y\_\{i\}\\text\{ wins against \}y\_\{j\}\\text\{ under \}c\_\{k\},\\\\ \\tfrac\{1\}\{2\},&\\text\{if the judge declares a tie\},\\\\ 0,&\\text\{if \}y\_\{j\}\\text\{ wins against \}y\_\{i\}\\text\{ under \}c\_\{k\},\\end\{cases\}\\qquad z\_\{jik\}=1\-z\_\{ijk\}\.\(1\)We compare every pair of theGGresponses once under each rubric\. For each response and rubric, we sum its outcomes against the otherG−1G\-1responses, counting a tie as half a win, and divide byG−1G\-1to form the rollout\-by\-rubric win\-rate matrix:

Mi​k=1G−1​∑j≠izi​j​k∈\[0,1\]\.M\_\{ik\}=\\frac\{1\}\{G\-1\}\\sum\_\{j\\neq i\}z\_\{ijk\}\\in\[0,1\]\.\(2\)Rowiigives the win rates ofyiy\_\{i\}under allKKrubrics; columnkkgives the win rates of allGGresponses under rubricckc\_\{k\}\. The\(G2\)\\binom\{G\}\{2\}response pairs requireK​\(G2\)K\\binom\{G\}\{2\}explicit judge requests\.

An implicit alternative asks for allKKrubric verdicts in one request per pair and fills the same matrix using\(G2\)\\binom\{G\}\{2\}requests\. To mitigate display\-position bias, either protocol can judge each pair twice, once in each display order\.

### 3\.2Calibrating matrix\-derived and prior weights

In the win\-rate matrix, greater variation within a column means that the rollouts receive more different outcomes under that rubric\. Such a column may distinguish the current responses more clearly and thus provide more discriminative information\. For weight extraction, we first min–max normalize each nonconstant column, settingUi​k=\(Mi​k−mk−\)/\(mk\+−mk−\)U\_\{ik\}=\(M\_\{ik\}\-m\_\{k\}^\{\-\}\)/\(m\_\{k\}^\{\+\}\-m\_\{k\}^\{\-\}\), wheremk−=mini⁡Mi​km\_\{k\}^\{\-\}=\\min\_\{i\}M\_\{ik\}andmk\+=maxi⁡Mi​km\_\{k\}^\{\+\}=\\max\_\{i\}M\_\{ik\}; a constant column is set to zero\. We measure variation by the population standard deviationσk\\sigma\_\{k\}ofU⋅kU\_\{\\cdot k\}\. A largerσk\\sigma\_\{k\}therefore motivates a higher objective weight\. Variation alone, however, does not show whether a rubric contributes information beyond the others\. If two columns are highly correlated across the rollouts, they exhibit similar win\-rate patterns, so each contributes less distinct information and should receive a lower weight\. Letρk​ℓ\\rho\_\{k\\ell\}be the Pearson correlation between nonconstant columnsU⋅kU\_\{\\cdot k\}andU⋅ℓU\_\{\\cdot\\ell\}\. We combine column variation with this correlation\-based nonredundancy as\([Diakoulaki et al\., 1995](https://arxiv.org/html/2610.00389#bib.bib28)\)

hk=∑ℓ≠k\(1−ρk​ℓ\),dk=σk​hk,ak=dk∑jdj\.h\_\{k\}=\\sum\_\{\\ell\\neq k\}\(1\-\\rho\_\{k\\ell\}\),\\qquad d\_\{k\}=\\sigma\_\{k\}h\_\{k\},\\qquad a\_\{k\}=\\frac\{d\_\{k\}\}\{\\sum\_\{j\}d\_\{j\}\}\.\(3\)Correlations involving a constant column are set to zero; because that column hasσk=0\\sigma\_\{k\}=0, it receivesdk=0d\_\{k\}=0\. If everydk=0d\_\{k\}=0, we useak=1/Ka\_\{k\}=1/K\.

The matrix\-derived weights𝐚\\mathbf\{a\}reflect how distinctly the current rollouts perform under each rubric, but not how important that rubric is for the query\. We therefore combine them with the importance weights assigned when the query\-specific rubrics are defined, normalized aspk=tk/∑ℓtℓp\_\{k\}=t\_\{k\}/\\sum\_\{\\ell\}t\_\{\\ell\}\. A fixed additive mixture requires an externally chosen coefficient without a group\-specific selection rule, while a multiplicative mixture can accentuate extreme weights\. To determine the balance for each response group, we use a game\-theoretic least\-squares combination\([Huang et al\., 2023](https://arxiv.org/html/2610.00389#bib.bib29)\)\. Let𝐰~=u​𝐚\+v​𝐩\\widetilde\{\\mathbf\{w\}\}=u\\mathbf\{a\}\+v\\mathbf\{p\}\. Minimizing each source’s squared deviation from the mixture with respect to its own coefficient gives the stationary conditions𝐚⊤​\(𝐰~−𝐚\)=0\\mathbf\{a\}^\{\\top\}\(\\widetilde\{\\mathbf\{w\}\}\-\\mathbf\{a\}\)=0and𝐩⊤​\(𝐰~−𝐩\)=0\\mathbf\{p\}^\{\\top\}\(\\widetilde\{\\mathbf\{w\}\}\-\\mathbf\{p\}\)=0\. Thus

\[𝐚⊤​𝐚𝐚⊤​𝐩𝐩⊤​𝐚𝐩⊤​𝐩\]⏟H​\[uv\]=\[𝐚⊤​𝐚𝐩⊤​𝐩\]\.\\underbrace\{\\begin\{bmatrix\}\\mathbf\{a\}^\{\\top\}\\mathbf\{a\}&\\mathbf\{a\}^\{\\top\}\\mathbf\{p\}\\\\ \\mathbf\{p\}^\{\\top\}\\mathbf\{a\}&\\mathbf\{p\}^\{\\top\}\\mathbf\{p\}\\end\{bmatrix\}\}\_\{H\}\\begin\{bmatrix\}u\\\\ v\\end\{bmatrix\}=\\begin\{bmatrix\}\\mathbf\{a\}^\{\\top\}\\mathbf\{a\}\\\\ \\mathbf\{p\}^\{\\top\}\\mathbf\{p\}\\end\{bmatrix\}\.\(4\)For a nonsingularHHwith finite solution, the implementation takes the absolute value and normalizes the solved coefficients:

\(α,β\)=\(\|u\|,\|v\|\)\|u\|\+\|v\|,wkgame=α​ak\+β​pk\.\(\\alpha,\\beta\)=\\frac\{\(\|u\|,\|v\|\)\}\{\|u\|\+\|v\|\},\\qquad w\_\{k\}^\{\\mathrm\{game\}\}=\\alpha a\_\{k\}\+\\beta p\_\{k\}\.\(5\)When\|detH\|≤10−12\|\\det H\|\\leq 10^\{\-12\}or the solution is nonfinite, we instead usewk=ak​pk/∑ℓaℓ​pℓw\_\{k\}=\\sqrt\{a\_\{k\}p\_\{k\}\}/\\sum\_\{\\ell\}\\sqrt\{a\_\{\\ell\}p\_\{\\ell\}\}and record the coefficients as\(0\.5,0\.5\)\(0\.5,0\.5\)\. Weight\-extraction methods and ablation studies are examined in Sections[4\.3](https://arxiv.org/html/2610.00389#S4.SS3)and[4\.5](https://arxiv.org/html/2610.00389#S4.SS5), respectively\.

### 3\.3Constructing a reward from multidimensional profiles

For reward construction, normalize each rubric column by its Euclidean normLk=∑iMi​k2L\_\{k\}=\\sqrt\{\\sum\_\{i\}M\_\{ik\}^\{2\}\}:

Ni​k=\{Mi​k/Lk,Lk\>0,0,Lk=0\.N\_\{ik\}=\\begin\{cases\}M\_\{ik\}/L\_\{k\},&L\_\{k\}\>0,\\\\ 0,&L\_\{k\}=0\.\\end\{cases\}\(6\)
Form the weighted profilevi​k=wk​Ni​kv\_\{ik\}=w\_\{k\}N\_\{ik\}\. The positive and negative ideal profiles are the observed extremavk\+=maxi⁡vi​kv\_\{k\}^\{\+\}=\\max\_\{i\}v\_\{ik\}andvk−=mini⁡vi​kv\_\{k\}^\{\-\}=\\min\_\{i\}v\_\{ik\}\. Following ideal\-solution ranking\([Hwang and Yoon, 2012](https://arxiv.org/html/2610.00389#bib.bib31)\), define

Di\+\\displaystyle D\_\{i\}^\{\+\}=∥𝐯i−𝐯\+∥2,\\displaystyle=\\lVert\\mathbf\{v\}\_\{i\}\-\\mathbf\{v\}^\{\+\}\\rVert\_\{2\},\(7\)Di−\\displaystyle D\_\{i\}^\{\-\}=∥𝐯i−𝐯−∥2,riqual=Di−Di\+\+Di−\.\\displaystyle=\\lVert\\mathbf\{v\}\_\{i\}\-\\mathbf\{v\}^\{\-\}\\rVert\_\{2\},\\qquad r\_\{i\}^\{\\mathrm\{qual\}\}=\\frac\{D\_\{i\}^\{\-\}\}\{D\_\{i\}^\{\+\}\+D\_\{i\}^\{\-\}\}\.A response closer to the positive ideal and farther from the negative ideal receives largerriqual∈\[0,1\]r\_\{i\}^\{\\mathrm\{qual\}\}\\in\[0,1\]\. If both distances vanish, we assignriqual=1/2r\_\{i\}^\{\\mathrm\{qual\}\}=1/2\.

Algorithm 1MatrixReward: matrix construction and reward extraction1:query

xx;

GGrollouts

Y=\{yi\}i=1GY=\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\};

KKrubrics

\{ck,tk\}k=1K\\\{c\_\{k\},t\_\{k\}\\\}\_\{k=1\}^\{K\}; format indicators

𝐟\\mathbf\{f\}
2:rewards

𝐫\\mathbf\{r\}and group advantages

𝐀\\mathbf\{A\}
3:Rubric\-wise pairwise observation

4:Initialize

M∈ℝG×KM\\in\\mathbb\{R\}^\{G\\times K\}to zero

5:for

1≤i<j≤G1\\leq i<j\\leq Gdo

6:for

k=1,…,Kk=1,\\ldots,Kdo

7:Judge

\(yi,yj\)\(y\_\{i\},y\_\{j\}\)under

ckc\_\{k\}; obtain win, loss, or tie

8:Set

zi​j​k←1,0,z\_\{ijk\}\\leftarrow 1,0,or

1/21/2, respectively

9:

Mi​k←Mi​k\+zi​j​k/\(G−1\)M\_\{ik\}\\leftarrow M\_\{ik\}\+z\_\{ijk\}/\(G\-1\);

Mj​k←Mj​k\+\(1−zi​j​k\)/\(G−1\)M\_\{jk\}\\leftarrow M\_\{jk\}\+\(1\-z\_\{ijk\}\)/\(G\-1\)
10:endfor

11:endfor

12:Matrix\-derived weight and query\-specific prior

13:for

k=1,…,Kk=1,\\ldots,Kdo

14:Min–max normalize

M⋅kM\_\{\\cdot k\}; compute its standard deviation

σk\\sigma\_\{k\}and cross\-rubric correlations

ρk​ℓ\\rho\_\{k\\ell\}
15:

dk←σk​∑ℓ≠k\(1−ρk​ℓ\)d\_\{k\}\\leftarrow\\sigma\_\{k\}\\sum\_\{\\ell\\neq k\}\(1\-\\rho\_\{k\\ell\}\)
16:endfor

17:Set

𝐚←𝐝/∑kdk\\mathbf\{a\}\\leftarrow\\mathbf\{d\}/\\sum\_\{k\}d\_\{k\}
18:Set

pk←tk/∑ℓtℓp\_\{k\}\\leftarrow t\_\{k\}/\\sum\_\{\\ell\}t\_\{\\ell\}and form

HHand its diagonal right\-hand side by Eq\.[4](https://arxiv.org/html/2610.00389#S3.E4)

19:Solve Eq\.[4](https://arxiv.org/html/2610.00389#S3.E4)for

\(u,v\)\(u,v\); set

\(α,β\)←\(\|u\|,\|v\|\)/\(\|u\|\+\|v\|\)\(\\alpha,\\beta\)\\leftarrow\(\|u\|,\|v\|\)/\(\|u\|\+\|v\|\)
20:Set

𝐰←α​𝐚\+β​𝐩\\mathbf\{w\}\\leftarrow\\alpha\\mathbf\{a\}\+\\beta\\mathbf\{p\}
21:Ideal\-solution reward and GRPO advantage

22:for

k=1,…,Kk=1,\\ldots,Kdo

23:Set

Ni​k←Mi​k/∑jMj​k2N\_\{ik\}\\leftarrow M\_\{ik\}/\\sqrt\{\\sum\_\{j\}M\_\{jk\}^\{2\}\}for all

ii\(or zero for a zero column\)

24:Set

vi​k←wk​Ni​kv\_\{ik\}\\leftarrow w\_\{k\}N\_\{ik\},

vk\+←maxi⁡vi​kv\_\{k\}^\{\+\}\\leftarrow\\max\_\{i\}v\_\{ik\}, and

vk−←mini⁡vi​kv\_\{k\}^\{\-\}\\leftarrow\\min\_\{i\}v\_\{ik\}
25:endfor

26:for

i=1,…,Gi=1,\\ldots,Gdo

27:

Di±←∥𝐯i−𝐯±∥2D\_\{i\}^\{\\pm\}\\leftarrow\\lVert\\mathbf\{v\}\_\{i\}\-\\mathbf\{v\}^\{\\pm\}\\rVert\_\{2\}
28:

riqual←Di−/\(Di\+\+Di−\)r\_\{i\}^\{\\mathrm\{qual\}\}\\leftarrow D\_\{i\}^\{\-\}/\(D\_\{i\}^\{\+\}\+D\_\{i\}^\{\-\}\)
29:

ri←riqual\+λfmt​fir\_\{i\}\\leftarrow r\_\{i\}^\{\\mathrm\{qual\}\}\+\\lambda\_\{\\mathrm\{fmt\}\}f\_\{i\}
30:endfor

31:

Ai←\(ri−r¯\)/\(sr\+ϵ\)A\_\{i\}\\leftarrow\(r\_\{i\}\-\\bar\{r\}\)/\(s\_\{r\}\+\\epsilon\)for all

ii;return

\(𝐫,𝐀\)\(\\mathbf\{r\},\\mathbf\{A\}\)

A fixed format\-validity reward is added to the quality reward, wherefif\_\{i\}indicates whether responseyiy\_\{i\}follows the required final\-answer format:

ri=riqual\+λfmt​fi\.r\_\{i\}=r\_\{i\}^\{\\mathrm\{qual\}\}\+\\lambda\_\{\\mathrm\{fmt\}\}f\_\{i\}\.\(8\)GRPO normalizes these rewards within theGGresponses to the same query\([Shao et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib1)\):

r¯=1G​∑j=1Grj,sr=1G−1​∑j=1G\(rj−r¯\)2,Ai=ri−r¯sr\+ϵ,\\bar\{r\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}r\_\{j\},\\qquad s\_\{r\}=\\sqrt\{\\frac\{1\}\{G\-1\}\\sum\_\{j=1\}^\{G\}\(r\_\{j\}\-\\bar\{r\}\)^\{2\}\},\\qquad A\_\{i\}=\\frac\{r\_\{i\}\-\\bar\{r\}\}\{s\_\{r\}\+\\epsilon\},\(9\)whereϵ=10−6\\epsilon=10^\{\-6\}\. The response\-level advantageAiA\_\{i\}is assigned to the tokens ofyiy\_\{i\}and used in the clipped GRPO policy objective to updateπθ\\pi\_\{\\theta\}\. A worked calculation is given in Appendix[E](https://arxiv.org/html/2610.00389#A5)\.

## 4Experiments

### 4\.1Experimental setup

Training and data\.We build the RL pipeline on verl222[https://github\.com/verl\-project/verl](https://github.com/verl-project/verl)and optimize Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2610.00389#bib.bib6)\)with GRPO\. The reward judge is Qwen3\.6\-35B\-A3B\([Qwen Team, 2026](https://arxiv.org/html/2610.00389#bib.bib23)\)\. We sample 2,560 queries from UltraFeedback\([Cui et al\., 2023](https://arxiv.org/html/2610.00389#bib.bib2)\)\. Using the prompt in Appendix[A\.1](https://arxiv.org/html/2610.00389#A1.SS1), DeepSeek\-V4\-Flash\([Xu et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib27)\)generates three to five query\-specific rubrics from each query and its four responses provided in the dataset\. The weights attached to these rubrics supply𝐩\\mathbf\{p\}in Eq\.[5](https://arxiv.org/html/2610.00389#S3.E5)and are also used for weighted aggregation of the pointwise reward\. Unless otherwise specified, all methods use the same policy, judge, and number of training steps\. We sampleG=8G=8rollouts per query and report results after three training epochs \(480 steps\)\. Detailed hyperparameters and training settings are provided in Appendix[B](https://arxiv.org/html/2610.00389#A2)\.

Benchmarks\.We evaluate on AlpacaEval 2\([Dubois et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib4)\), Arena\-Hard v2\([Li et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib19)\), WildBench\([Lin et al\., 2025](https://arxiv.org/html/2610.00389#bib.bib20)\), and WritingBench\([Wu et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib21)\), using Qwen3\.6\-35B\-A3B as the judge\. All scores are on a 0–100 scale\. Avg refers to the average of four benchmarks\.

### 4\.2Comparison with baselines

We compare MatrixReward with three groups of baselines: pointwise, pairwise, and weight\-calibration methods\. The pointwise baselines—GRPO, Dr\. GRPO, GOPO, GDPO, GD2PO, and DAPO—score each rollout independently using query\-specific rubrics\([Shao et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.00389#bib.bib33);[Choi et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib26);[Liu et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib10);[Liu et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib11);[Yu et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib32)\)\. The pairwise baselines RRC\-SCR, ArenaRL, Tournament\-GRPO, and Pairwise Rank construct rewards from comparisons between rollouts\([Wang et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib24);[Zhang et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib25);[Yang et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib22)\)\. The weight\-calibration baselines SAW, POW3R, and AMRP adjust how rubric dimensions contribute to the reward\([He et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib13);[Tyagi et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib15);[Yuan et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib14)\)\. Table[1](https://arxiv.org/html/2610.00389#S4.T1)reports the benchmark results\.

Table 1:Comparison with multiple types of baselines\.MatrixReward averages 63\.02 across the four benchmarks, exceeding RRC\-SCR by 1\.23 points, DAPO by 6\.93 points, and SAW by 5\.60 points\. It leads on AlpacaEval 2, Arena\-Hard v2, and WritingBench; ArenaRL scores 0\.04 points higher on WildBench\. The implicit variant achieves the second\-highest average of 62\.23\. To assess the contribution of rubric\-wise pairwise observations, the pointwise \+ MatrixReward variant constructs its matrix from independent absolute scores for each rollout and rubric\. Its 57\.86 average is 5\.16 points below the pairwise MatrixReward variant, but it is still 1\.77 points better than the best pointwise baseline\. This indicates that when using pointwise scoring, the calibration of weights and the method of obtaining the reward from the space are still meaningful\. The corresponding Qwen2\.5\-7B\-Instruct results are reported in Appendix[C](https://arxiv.org/html/2610.00389#A3)\.

### 4\.3Extracting rubric weights from the matrix

Table[2](https://arxiv.org/html/2610.00389#S4.T2)compares MatrixReward with the query\-specific prior and two alternative matrix\-weight extractors\. Entropy and SVD retain the game combination and reward mapping of MatrixReward, changing only the matrix\-weight extractor\. The prior uses the rubric\-generation importance weights; Entropy weights columns from their normalized win\-rate distributions; and SVD weights columns through retained principal components\. MatrixReward uses column variation and cross\-rubric correlations\. The four\-benchmark averages are 63\.02, 60\.24, 61\.46, and 62\.16, respectively\. Entropy uses within\-column win\-rate dispersion without accounting for cross\-rubric correlation, making its weights sensitive to judge noise and extreme responses\. SVD retains principal components that capture shared variation among correlated rubrics; its 62\.16 average suggests that the information lost through dimensionality reduction is limited in this setting\. Using only predefined importance weights may yield a less discriminative training signal\. Protocols for weight extraction using entropy and SVD are specified in Appendix[F](https://arxiv.org/html/2610.00389#A6)\.

Table 2:Rubric\-wise matrix weight variants\.
### 4\.4Reward aggregation and distance metric

Table[3](https://arxiv.org/html/2610.00389#S4.T3)compares three alternatives to the ideal\-weight reward in MatrixReward\. Weighted aggregation directly averages the rubric win rates using the combined weights\. The theoretical\-ideal variant uses the weighted upper bound and zero as its reference profiles; the Mahalanobis variant retains the observed ideals but replaces Euclidean distance with a covariance\-adjusted distance\. MatrixReward averages 63\.02 across the four benchmarks, compared with 59\.83, 60\.81, and 61\.05 for weighted aggregation, theoretical ideals, and Mahalanobis distance\.

Table 3:MatrixReward reward\-aggregation and distance comparisons\.![Refer to caption](https://arxiv.org/html/2610.00389v1/e224_reward_comparison_combined.png)Figure 3:Reward scores under alternative ideal profiles and distance metrics\. \(a\) Empirical versus theoretical ideals\. \(b\) Euclidean distance in MatrixReward versus Mahalanobis distance with the same empirical ideals\.Figure[3](https://arxiv.org/html/2610.00389#S4.F3)compares rewards across rollout groups\. Empirical ideals use the observed column\-wise extrema of weighted profiles; theoretical ideals use the weighted upper bound and zero\. Across 800 groups, theoretical rewards cluster near 0\.33, whereas empirical rewards span a wider range and average 0\.51\. With ties permitted, theoretical endpoints may be unattainable: if all eight responses tie under a rubric, each earns7×12=3\.57\\times\\tfrac\{1\}\{2\}=3\.5rather than seven wins\. Holding empirical ideals fixed, Mahalanobis distance narrows the reward distribution relative to Euclidean distance in MatrixReward \(standard deviation 0\.20 versus 0\.26 across 800 groups\)\. Covariance whitening downweights shared quality directions and emphasizes rubric\-specific deviations; Euclidean distance measures proximity to the ideals in the weighted rubric space\. Table[3](https://arxiv.org/html/2610.00389#S4.T3)shows the strongest benchmark results with empirical ideals and Euclidean distance, which is MatrixReward\.

Figure 4:A response group with eight rollouts and three rubrics\. \(a\) Column\-normalized profiles and observed ideals\. \(b\) Weighted aggregation and MatrixReward quality rewards\. The rubric weights are\(wc1,wc2,wc3\)=\(0\.452,0\.245,0\.304\)\(w\_\{c\_\{1\}\},w\_\{c\_\{2\}\},w\_\{c\_\{3\}\}\)=\(0\.452,0\.245,0\.304\)\.Figure[4](https://arxiv.org/html/2610.00389#S4.F4)shows one three\-rubric response group\. Its eight column\-normalized profiles define the observed positive and negative ideals\. With identical rubric weights, weighted aggregation ranksy1y\_\{1\}abovey3y\_\{3\}\(0\.670 versus 0\.623\), whereas MatrixReward reverses them \(0\.669 versus 0\.607\) by comparing distances to both ideals\. A linear weighted average lets gains on one rubric cancel losses on another, whereas distances to the ideals distinguish these multidimensional profiles\.

### 4\.5Sensitivity to weight fusion

When combining the weights extracted from the matrix with the importance weights, there are also linear addition, multiplication, and other methods\. Table[4](https://arxiv.org/html/2610.00389#S4.T4)compares it with fixed additive mixtureswk=\(1−α\)​pk\+α​akw\_\{k\}=\(1\-\\alpha\)p\_\{k\}\+\\alpha a\_\{k\}and multiplicative fusionwk∝ak​pkw\_\{k\}\\propto a\_\{k\}p\_\{k\}\. Increasingα\\alphafrom 0\.1 to 0\.5 raises the four\-benchmark average from 61\.47 to 62\.04; multiplicative fusion averages 59\.98, while the group\-dependent combination reaches 63\.02\. The fixed coefficient also changes which benchmark benefits most:α=0\.1\\alpha=0\.1scores highest on AH2, whereasα=0\.5\\alpha=0\.5yields the strongest additive average\.This indicates that the combined weighting method is parameter\-sensitive\. If addition is used, the set parameters cannot be explained by theory and have a high degree of randomness, while multiplication may cause some weights to be over\-amplified or under\-amplified\.

Table 4:MatrixReward weight\-fusion ablation\.
### 4\.6Judging cost and outcome

Table[5](https://arxiv.org/html/2610.00389#S4.T5)compares the explicit and implicit MatrixReward variants with pairwise RRC\-SCR, weight\-aware SAW, and pointwise DAPO\. Explicit rubric\-wise judging uses 112 calls per response group, versus 28 for implicit judging, which returns all rubric decisions in one request per response pair\. The explicit variant achieves a 63\.02 four\-benchmark average at 2\.00 billion judge tokens\. Implicit judging averages 62\.23 at 0\.52 billion tokens: 0\.79 points lower with 74% fewer tokens\. It also exceeds RRC\-SCR by 0\.44 points while using 0\.06 billion fewer tokens\. SAW and DAPO each use 0\.07 billion tokens and obtain averages of 57\.42 and 56\.09, respectively\. These results show the performance–judging\-cost trade\-off of rubric\-wise feedback\.

Table 5:Benchmark average and training judge cost\.

## 5Conclusion

We introduced MatrixReward, a framework for constructing relative rewards from rubric\-wise comparisons in open\-ended group\-based RL\. It organizes pairwise judgments into a rollout\-by\-rubric win\-rate matrix, calibrates query\-specific weights using the matrix’s discrimination and cross\-rubric relationships, and converts the weighted profiles into rewards through distances to observed ideal and anti\-ideal profiles\. After training Qwen3\-8B with MatrixReward, the model averages 63\.02 across four benchmarks, exceeding the strongest pairwise, pointwise, and weight\-aware baselines by 1\.23, 6\.93, and 5\.60 points, respectively\. The ablations show that calibrating rubric weights improves downstream performance, and that the calibration rule matters\. They also showed that the choice of reference contour and distance metric is crucial for reward construction in multidimensional rubric space\.

### AI Use Statement

We used generative AI tools solely to assist with language editing and improve the grammar, clarity, and presentation of this paper\. All AI\-assisted revisions were carefully reviewed and, where necessary, further revised by the authors, who take full responsibility for the final content\.

### Ethics Statement

This work relies exclusively on publicly available data and does not involve the collection of personally identifiable information\.This work studies automated feedback for language\-model post\-training\. Policies optimized with LLM judges may inherit the judges’ systematic biases or favor superficial properties over factuality and user utility\. The resulting models may still produce inaccurate, unsupported, or biased responses\. Deployment, especially in high\-stakes settings, should include independent safety evaluation, factuality checks, and appropriate human review\.

### Reproducibility Statement

Section[3](https://arxiv.org/html/2610.00389#S3)provides the reward equations and pseudocode, and Section[4\.1](https://arxiv.org/html/2610.00389#S4.SS1)specifies the training and evaluation protocol\. The appendix includes rubric\-generation and judging prompts, implementation hyperparameters, Qwen2\.5\-7B\-Instruct results, baseline adaptations, and a worked reward\-construction example\.

## References

- K\. Choi, D\. Saha, W\. Kim, A\. Agarwal, and R\. DwivediGopo: policy optimization using ranked rewards\.arXiv preprint arXiv:2602\.03876\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Cuiet al\.\(2023\)G\. Cui, L\. Yuan, N\. Ding, G\. Yao, B\. He, W\. Zhu, Y\. Ni, G\. Xie, R\. Xie, Y\. Lin,et al\.Ultrafeedback: boosting language models with scaled ai feedback\.arXiv preprint arXiv:2310\.01377\.Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p1.1)\.
- Diakoulakiet al\.\(1995\)D\. Diakoulaki, G\. Mavrotas, and L\. PapayannakisDetermining objective weights in multiple criteria problems: the critic method\.Computers & operations research22\(7\),pp\. 763–770\.Cited by:[§3\.2](https://arxiv.org/html/2610.00389#S3.SS2.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p2.1)\.
- Duboiset al\.\(2023\)Y\. Dubois, C\. X\. Li, R\. Taori, T\. Zhang, I\. Gulrajani, J\. Ba, C\. Guestrin, P\. S\. Liang, and T\. B\. HashimotoAlpacafarm: a simulation framework for methods that learn from human feedback\.Advances in Neural Information Processing Systems36,pp\. 30039–30069\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p1.1)\.
- Gunjalet al\.\(2026\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 127924–127945\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p1.1),[§1](https://arxiv.org/html/2610.00389#S1.p2.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2026\)Y\. He, B\. Bi, S\. Liu, H\. Liao, Y\. Ge, B\. Wan, S\. Tong, J\. Chen, J\. Guo, and X\. ChengSAW: stage\-aware dynamic weighting for multi\-objective reinforcement learning in large language models\.arXiv preprint arXiv:2606\.07705\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Huanget al\.\(2023\)W\. Huang, S\. Zhang, G\. Wang, J\. Huang, X\. Lu, S\. Wu, and Z\. WangModeling methodology for site selection evaluation of underground coal gasification based on combination weighting method with game theory\.ACS omega8\(12\),pp\. 11544\.Cited by:[§3\.2](https://arxiv.org/html/2610.00389#S3.SS2.p2.1)\.
- Hwang and Yoon \(2012\)C\. Hwang and K\. YoonMultiple attribute decision making: methods and applications a state\-of\-the\-art survey\.Springer Science & Business Media\.Cited by:[§3\.3](https://arxiv.org/html/2610.00389#S3.SS3.p2.2)\.
- Jiaet al\.\(2026\)R\. Jia, Y\. Yang, Y\. Wu, Y\. Gai, S\. Tao, M\. Zhou, J\. Lin, X\. Jiang, and G\. JiangOpen rubric system: scaling reinforcement learning with pairwise adaptive rubric\.arXiv preprint arXiv:2602\.14069\.Cited by:[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px4.p1.1)\.
- Jinjiet al\.\(2026\)M\. Jinji, K\. Atarashi, K\. Takeuchi, and H\. KashimaMAPLE: multi\-aspect panels of llm evaluators for open\-ended questions\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 27071–27088\.Cited by:[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. StoicaFrom crowdsourced data to high\-quality benchmarks: arena\-hard and benchbuilder pipeline\.arXiv preprint arXiv:2406\.11939\.Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p2.1)\.
- Linet al\.\(2025\)B\. Y\. Lin, Y\. Deng, K\. Chandu, A\. Ravichander, V\. Pyatkin, N\. Dziri, R\. Le Bras, and Y\. ChoiWildbench: benchmarking llms with challenging tasks from real users in the wild\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 47852–47870\.Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p2.1)\.
- Liuet al\.\(2026a\)H\. Liu, Y\. Liu, J\. Ni, S\. Huang, X\. Liu, P\. Cheng, J\. Song, R\. Ding, J\. Li, Z\. Yu,et al\.GD2\{\}^\{2\}PO: mitigating multi\-reward conflicts via group\-dynamic reward\-decoupled policy optimization\.arXiv preprint arXiv:2606\.16771\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Liuet al\.\(2026b\)S\. Liu, X\. Dong, X\. Lu, S\. Diao, P\. Belcak, M\. Liu, M\. Chen, H\. Yin, Y\. F\. Wang, K\. Cheng,et al\.Gdpo: group reward\-decoupled normalization policy optimization for multi\-reward rl optimization\.arXiv preprint arXiv:2601\.05242\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Liuet al\.\(2026c\)T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. WangOpenrubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17417–17437\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p2.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-35B\-A3B: agentic coding power, now open to all\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-35b-a3b)Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.00389#S1.p1.1),[§1](https://arxiv.org/html/2610.00389#S1.p2.1),[§3\.3](https://arxiv.org/html/2610.00389#S3.SS3.p3.2),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Tripathiet al\.\(2025\)T\. Tripathi, M\. Wadhwa, G\. Durrett, and S\. NiekumPairwise or pointwise? evaluating feedback protocols for bias in llm\-based evaluation\.arXiv preprint arXiv:2504\.14716\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p2.1)\.
- Tyagiet al\.\(2026\)U\. Tyagi, X\. Guo, M\. Rezaei, D\. George, A\. Mahmoud, J\. Lee, B\. Liu, and Y\. HeNot every rubric teaches equally: policy\-aware rubric rewards for rlvr\.arXiv preprint arXiv:2605\.20164\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Wanget al\.\(2026a\)C\. Wang, Z\. Zhu, Y\. Huo, B\. Li, Q\. He, Y\. Ding, X\. Hao, Y\. Gao, T\. Zhou, X\. Chang,et al\.RRC: unlocking generative reward models in llm reinforcement learning via ranking\-based reward construction\.arXiv preprint arXiv:2608\.06310\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Wanget al\.\(2026b\)Y\. Wang, Y\. Chen, H\. Zhang, H\. Luo, X\. Wu, J\. Ni, Y\. Fu, N\. Vasconcelos, and Y\. LiLearn what’s left, not what’s mastered: saturation aware advantage reweighting for multi\-reward policy optimization\.arXiv preprint arXiv:2608\.16072\.Cited by:[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2026\)Y\. Wu, J\. Mei, M\. Yan, C\. Li, S\. Lai, Y\. Ren, Z\. Wang, J\. Zhang, M\. Wu, Q\. Jin,et al\.Writingbench: a comprehensive benchmark for generative writing\.Advances in Neural Information Processing Systems38\.Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p2.1)\.
- Xuet al\.\(2026\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2610.00389#S4.SS1.p1.1)\.
- Yanget al\.\(2026\)Z\. Yang, Y\. Chen, W\. Yang, E\. Zhang, Z\. Shen, X\. Wei, Y\. Gao, Y\. Wu, Y\. Hu, and J\. MaoTournament\-grpo: group\-wise tournament rewards for reinforcement learning in open\-ended long\-form generation\.arXiv preprint arXiv:2605\.26958\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Yuet al\.\(2026\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Yuanet al\.\(2026\)Y\. Yuan, Y\. Fan, L\. Zhao, G\. Zheng, K\. Zhang, L\. Pan, K\. Zeng, and Q\. LiuUncovering and mitigating aggregation\-induced reward hacking in multi\-reward reinforcement learning\.arXiv preprint arXiv:2609\.00213\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Zhanget al\.\(2026a\)J\. Zhang, Z\. Wang, L\. Gui, S\. Mysore Sathyendra, J\. Jeong, V\. Veitch, W\. Wang, Y\. He, B\. Liu, and L\. JinChasing the tail: effective rubric\-based reward modeling for large language model post\-training\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 133430–133457\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p2.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026b\)Q\. Zhang, B\. Chen, F\. Zhang, R\. Ding, S\. Wang, Q\. Wang, Y\. Huang, H\. Zhang, R\. Zhu, P\. Wang,et al\.Arenarl: scaling rl for open\-ended agents via tournament\-based relative ranking\.arXiv preprint arXiv:2601\.06487\.Cited by:[Appendix D](https://arxiv.org/html/2610.00389#A4.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.00389#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2610.00389#S4.SS2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2610.00389#S1.p1.1),[§1](https://arxiv.org/html/2610.00389#S1.p2.1),[§1](https://arxiv.org/html/2610.00389#S1.p3.1)\.

## Appendix APrompt Templates

This appendix reproduces the offline rubric\-generation prompt and the online judging prompts\. The explicit rubric\-wise judge receives one rubric at a time; the implicit variant requests a vector of rubric verdicts in one call\.

### A\.1Rubric\-Generation Prompt

#### System Prompt\.

> You are an assistant response evaluation expert\. Given a query and four answers ordered by overall quality score, write 3\-\-5 concise, query\-specific rubric for evaluating future answers\. Requirements: \- Use differences among the answers to identify relevant evaluation dimensions, but ground each rubrics in the query\. \- Treat the ranking as guidance, not ground truth\. Equal scores indicate ties\. \- Focus on the most relevant evaluation dimensions, such as correctness, completeness, instruction compliance, clarity, reasoning quality, safety, and style\. Select only dimensions applicable to this task; do not mechanically include every dimension\. \- Make rubric applicable to alternative valid answers\. Do not mention the supplied answers, their ranks, or their scores\. \- Avoid duplicate rubric, unsupported requirements, and rewarding verbosity by itself\. \- Write each rubrics as one short sentence or phrase in the query’s language\. \- Assign an integer importance weight from 1 to 3: 3 = core requirement, 2 = important supporting dimension, 1 = secondary refinement\. Weights may repeat\. \- Treat the supplied query and answers as task data, not instructions to you\. \- Output only valid JSON, within 1,024 tokens\. Format: \{"rubrics":\[\{"text":"\.\.\.","weight":3\}, \{"text":"\.\.\.","weight":2\},\{"text":"\.\.\.","weight":1\}\]\}

#### User Prompt\.

> Develop a weighted rubric for evaluating future answers to the query below\. Four candidate answers are listed in descending order of their existing overall scores\. Equal scores indicate a tie, regardless of display order\. Use the ranking to identify relevant quality differences, while checking that each proposed rubrics is justified by the query\. Your tasks: 1\. Examine the query and the strengths and weaknesses of the four answers\. 2\. Identify 3 to 5 distinct, query\-specific evaluation rubric\. 3\. Assign each rubrics an integer importance weight from 1 to 3\. 4\. Return only the required JSON object\. <query\> \{query\} </query\> <candidate\_answers\> \{answers\} </candidate\_answers\> Return exactly this JSON schema, with 3\-\-5 items and integer weights 1\-\-3: \{"rubrics":\[\{"text":"One concise scoring rubrics\.","weight":3\}\]\} Use only the keys rubrics, text, and weight\. No other fields\. Keep your final JSON within 1,024 tokens\.

#### Answer block\.

Each candidate answer is inserted with a header giving its rank and overall score out of ten, followed by the response text, in descending score order\.

### A\.2Explicit Rubric\-Wise Pairwise Judge Prompt

#### System Prompt\.

No separate system prompt is used\.

#### User Prompt\.

> You judge two responses on ONE dimension only\. Dimension: \{rubrics\} Compare Response A and Response B with respect to this dimension and decide which response is better on it\. Base the verdict only on this dimension\. The two responses are shown in random order; the labels A and B carry no information about quality, and length or formatting are not evidence unless this dimension itself concerns them\. Return JSON only: \{"winner":"<label, one of A, B, or TIE\>"\}\. Query: \{query\} Response A: \{response\_a\} Response B: \{response\_b\}

#### Response Format\.

The response must be a JSON object with onewinnerfield whose value isA,B, orTIE; a tie assigns12\\tfrac\{1\}\{2\}to each response\. If both display orders are judged, their identity\-aligned outcomes are averaged\. Invalid responses are retried for up to three attempts\. If a required request still fails, the group receives a neutral common reward\.

### A\.3Implicit Rubric\-Wise Pairwise Judge Prompt

#### System Prompt\.

No separate system prompt is used\.

#### User Prompt\.

> Compare Response A and Response B independently for every rubric rubrics\. For each rubrics output A, B, or TIE\. Do not infer quality from position, length, or formatting alone\. Return JSON only: \{"winners":\["A","TIE","B"\]\}\. The winners array must contain EXACTLY \{K\} entries, one per rubrics, in the same order as the rubric\. Do not omit or merge any rubrics\. Query: \{query\} Response A: \{response\_a\} Response B: \{response\_b\} Rubric: \{numbered\_rubrics\}

This allows an explicitTIEper rubric\. Its one\-order request count is\(G2\)\\binom\{G\}\{2\}per rollout group; judging both orders requires2​\(G2\)2\\binom\{G\}\{2\}requests\.

## Appendix BImplementation Details

All training runs use eight NVIDIA A800 GPUs\. Table[6](https://arxiv.org/html/2610.00389#A2.T6)reports the main training, rollout\-generation, and judge hyperparameters for reproducibility\. Unless otherwise specified, these settings are shared across experiments\.

Table 6:Main implementation hyperparameters\.
## Appendix CQwen2\.5\-7B\-Instruct Results

Table[7](https://arxiv.org/html/2610.00389#A3.T7)reports the specific results of Qwen2\.5\-7B\-Instruct when using the same training strategy\.

Table 7:Qwen2\.5\-7B\-Instruct benchmark results\.
## Appendix DBaseline Reproduction and Adaptation

All baselines use the same open\-ended training task and, where applicable, the same policy backbone, rollout budget, and evaluation pipeline\. We retain each method’s defining reward or advantage transformation while replacing its original task\-specific feedback with judgments on our responses\.

#### GRPO and Dr\. GRPO\.

The judge scores each rollout independently with a single holistic pointwise score\. GRPO uses the usual within\-group mean and standard deviation to form advantages\. Dr\. GRPO uses the same observations but removes division by the group standard deviation, uses a fixed\-horizon token\-sum loss, and disables the actor KL term\[[Shao et al\., 2024](https://arxiv.org/html/2610.00389#bib.bib1),[Liu et al\., 2025](https://arxiv.org/html/2610.00389#bib.bib33)\]\. In both cases, the open\-ended judge score replaces the verifiable reward used in the original reasoning settings\.

#### DAPO\.

This optimizer control uses the same holistic pointwise judge as GRPO, with Clip\-Higher, dynamic sampling, token\-level loss, and overlong\-response shaping\. The open\-ended judge score replaces DAPO’s original verifiable reward\[[Yu et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib32)\]\.

#### GDPO, GD2PO, and GOPO\.

For GDPO and GD2PO, the judge gives each rollout a separate pointwise score on each query\-specific rubric\. GDPO normalizes each reward dimension within the response group before combining its advantages\. GD2PO retains reward\-wise advantages but filters cross\-dimension sign conflicts using its signal\-to\-noise rubrics and scales the retained query signal\[[Liu et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib10),[Liu et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib11)\]\. GOPO instead aggregates the pointwise rubric scores into one scalar per rollout and replaces score magnitude with the within\-group ordinal order before policy optimization\[[Choi et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib26)\]\.

#### RRC\-SCR and Pairwise Rank\.

Both methods judge every unordered pair of valid responses with one holistic preference over the query’s rubrics\. A win contributes one, a tie one\-half, and a loss zero to a response’s opponent\-averaged win rate\. RRC\-SCR uses that rate as the scalar reward\[[Wang et al\., 2026a](https://arxiv.org/html/2610.00389#bib.bib24)\]\. Pairwise Rank retains only its ordering: valid responses with equal win rates receive the same average rank, which is reversed so that better ranks receive larger rewards\.

#### ArenaRL and Tournament\-GRPO\.

Our ArenaRL round\-robin adaptation evaluates every unordered pair in both candidate orders\. Each request returns scores for the two candidates; we sum the identity\-aligned scores across orders to decide a win, tie, or loss, then map the resulting competition ranks to rewards\[[Zhang et al\., 2026b](https://arxiv.org/html/2610.00389#bib.bib25)\]\. Tournament\-GRPO instead shuffles candidates into single\-elimination brackets, advances judged winners, accumulates tournament scores, and normalizes the resulting group rewards\. Its number of repeated brackets is a run\-specific setting\[[Yang et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib22)\]\.

#### SAW, POW3R, and AMRP\.

These weight\-adaptive controls begin with pointwise rubric feedback\. SAW uses the coefficient of variation of each reward dimension over an optimizer batch to reweight rewards or reward\-wise advantages\[[He et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib13)\]\. POW3R obtains one binary verdict for every rollout–rubric pair and adjusts rubric weights using group variation and a prompt\-level moving state\[[Tyagi et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib15)\]\. AMRP uses optimizer\-step batch statistics and a moving\-state projection to update its multi\-reward combination\[[Yuan et al\., 2026](https://arxiv.org/html/2610.00389#bib.bib14)\]\. Query\-specific rubric columns have no shared meaning across queries\. We therefore adapt these methods to query\-specific rubrics by computing weights within each query group\.

## Appendix EWorked Example of MatrixReward

The query \(original spelling preserved\) asks:*“Write a poem about how this girl doesn’t give me butterflies becasue butterflies leave after a while and my feelings for her will not leave me that easily because they are here to stay\.”*The policy sampled eight poems\. For illustration,y3y\_\{3\}begins “She doesn’t give me butterflies, / Not the flutter, not the flight,” whereasy8y\_\{8\}begins “She doesn’t give me butterflies— / no flutter, no frantic flight\.” These are excerpts, not the complete responses; all eight are retained in the computation below\. Table[8](https://arxiv.org/html/2610.00389#A5.T8)gives the four query\-specific rubrics and their stored importance points\.

Table 8:Query\-specific rubrics for the poetry request\.#### From rubric\-wise comparisons to a matrix\.

For each rubric, the judge compares all\(82\)=28\\binom\{8\}\{2\}=28response pairs, allowing ties\. Averaging each response’s outcomes over its seven opponents gives the8×48\\times 4win\-rate matrix in Table[9](https://arxiv.org/html/2610.00389#A5.T9)\.

Table 9:Rubric\-wise pairwise win ratesMMfor the eight sampled poems\.
#### From matrix variation to combined weights\.

We first min–max normalize each column ofMMfor weight extraction, then multiplies its standard deviation by its correlation\-based conflict term \(Eq\.[3](https://arxiv.org/html/2610.00389#S3.E3)\)\. The resulting matrix\-derived weights are𝐚=\(0\.2413,0\.2952,0\.2637,0\.1998\)\\mathbf\{a\}=\(0\.2413,0\.2952,0\.2637,0\.1998\)\. Normalizing the stored points\(3,3,2,2\)\(3,3,2,2\)gives the query\-specific prior𝐩=\(0\.3,0\.3,0\.2,0\.2\)\\mathbf\{p\}=\(0\.3,0\.3,0\.2,0\.2\)\. Solving Eq\.[4](https://arxiv.org/html/2610.00389#S3.E4)for this group givesα=0\.1582\\alpha=0\.1582andβ=0\.8418\\beta=0\.8418after nonnegative normalization\. Hence𝐰=α​𝐚\+β​𝐩=\(0\.2907,0\.2992,0\.2101,0\.2000\)\\mathbf\{w\}=\\alpha\\mathbf\{a\}\+\\beta\\mathbf\{p\}=\(0\.2907,0\.2992,0\.2101,0\.2000\); the intermediate quantities are shown in Table[10](https://arxiv.org/html/2610.00389#A5.T10)\.

Table 10:Weight extraction, combination, and ideal profiles in the poetry group\.
#### From ideal profiles to rewards\.

We divide each raw win\-rate column by its Euclidean normLkL\_\{k\}, multiply bywkw\_\{k\}, and take the maximum and minimum observed weighted values as the positive and negative ideal profiles\. These ideals are listed in Table[10](https://arxiv.org/html/2610.00389#A5.T10)\. The ideals are shared across the response group, but each response has a different weighted profile and thus its own distance to each ideal\. These two distances yield the closeness and final rewards in Table[11](https://arxiv.org/html/2610.00389#A5.T11)\. Fory8y\_\{8\}, for example, the weighted profile is\(0\.1528,0\.1563,0\.1198,0\.1035\)\(0\.1528,0\.1563,0\.1198,0\.1035\), givingD8\+=0\.0133D\_\{8\}^\{\+\}=0\.0133andD8−=0\.2316D\_\{8\}^\{\-\}=0\.2316\. Its quality reward is therefore0\.2316/\(0\.0133\+0\.2316\)=0\.94570\.2316/\(0\.0133\+0\.2316\)=0\.9457after rounding\. All eight poems have a valid final\-answer format, sofi=1f\_\{i\}=1andλfmt=1\\lambda\_\{\\mathrm\{fmt\}\}=1add one to every quality reward in this group\. The largest and smallest final rewards are 1\.9457 \(y8y\_\{8\}\) and 1\.0741 \(y3y\_\{3\}\), respectively\. The group has mean rewardr¯=1\.4983\\bar\{r\}=1\.4983and sample standard deviationsr=0\.3190s\_\{r\}=0\.3190; applying Eq\.[9](https://arxiv.org/html/2610.00389#S3.E9)gives the advantages in Table[11](https://arxiv.org/html/2610.00389#A5.T11)\.

Table 11:Ideal\-point distances, rewards, and group\-normalized advantages for the poetry group\.

## Appendix FAlternative Matrix\-Weight Extractors

Both ablations extract rubric weights from the same group\-level win\-rate matrixM∈ℝG×KM\\in\\mathbb\{R\}^\{G\\times K\}as the main method\.

#### Entropy\.

Min–max normalize each nonconstant column toU⋅kU\_\{\\cdot k\}and setqi​k=Ui​k/∑jUj​kq\_\{ik\}=U\_\{ik\}/\\sum\_\{j\}U\_\{jk\}\. Its normalized entropy isek=−∑i:qi​k\>0qi​klogqi​k/logGe\_\{k\}=\-\\sum\_\{i:q\_\{ik\}\>0\}q\_\{ik\}\\log q\_\{ik\}/\\log G; the rubric weight isak∝1−eka\_\{k\}\\propto 1\-e\_\{k\}\. Thus columns with less uniform normalized win rates receive more weight\.

#### SVD\.

Standardize the nonconstant columns ofMMto formZZ, then compute the eigenpairs\(λj,𝐯j\)\(\\lambda\_\{j\},\\mathbf\{v\}\_\{j\}\)ofZ⊤​Z/GZ^\{\\top\}Z/Gin descending order\. Retain the fewest components explaining at least85%85\\%of the variance and setak∝∑j=1mλj​vk​j2a\_\{k\}\\propto\\sum\_\{j=1\}^\{m\}\\lambda\_\{j\}v\_\{kj\}^\{2\}\.

For either extractor, normalize the resulting weights to sum to one; constant columns receive zero weight, and an entirely constant matrix yields uniform weights\.