CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation

# CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation
Source: [https://arxiv.org/html/2607.20862](https://arxiv.org/html/2607.20862)
Hehao Zhang Danli Wang Xinyuan Wang Xuange Gao Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China zhanghehao2023@ia\.ac\.cndanli\.wang@ia\.ac\.cn wangxinyuan2024@ia\.ac\.cngaoxuange2022@ia\.ac\.cn

###### Abstract

At present, reliable evaluation of non\-verifiable tasks remains challenging\. Existing approaches often fail to adequately capture the diverse evaluative criteria underlying human preferences in such tasks\. To this end, we propose Constrained Shared\-Private Fusion \(CSPF\), a fusion method that treats heterogeneous frozen reward models as complementary evaluators and learns to integrate their hidden\-state representations under pairwise human\-preference supervision\.CSPFdecomposes each expert signal into shared and expert\-private representations, encouraging cross\-expert alignment while preserving complementary viewpoints\. Across experiments on LM\-Arena target\-domain adaptation and PPE out\-of\-distribution preference evaluation,CSPFachieves the best performance on the primary metrics among the evaluated single\-expert reward\-model, scalar\-score multi\-expert, and rubric\-judge baselines\. Overall,CSPFsuggests that fusing hidden\-state representations provides a more expressive basis for preference assessment, offering a practical route toward integrated evaluative signals for non\-verifiable preference tasks\.

## 1Introduction

After pretraining, LLMs are often further adapted through post\-training to improve target capabilities\. Reinforcement learning is an important post\-training approach: it optimizes a policy model by using evaluators, such as reward models, verifiers, or checkers, to assess generated responses and guide optimization\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.20862#bib.bib34)\)\. This dependence on assessment quality makes the source of evaluation crucial\. In verifiable domains such as mathematical reasoning and code generation, objective ground\-truth answers or executable tests make assessment relatively straightforward, enabling reinforcement learning with rule\-based or test\-based rewards to improve mathematical reasoning and coding abilities\(Shaoet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib38); Gehringet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib12)\)\. By contrast, many alignment\-relevant tasks are non\-verifiable: open\-ended dialogue, creative writing, subjective question answering, and safety\-sensitive instruction following often lack a single ground\-truth standard, and their quality depends on multi\-criteria human judgments\(Jiaet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib18); Gunjalet al\.,[2026](https://arxiv.org/html/2607.20862#bib.bib13)\)\. Reliable evaluation for such tasks is therefore difficult but essential, because post\-training can only optimize LLM behavior toward objectives that the evaluator can accurately assess\.

Existing work has developed three main evaluator families for non\-verifiable preference tasks: preference\-based reward models, rubric\-based evaluators, and multi\-evaluator methods\. Preference\-based reward models learn holistic reward functions from human comparisons, rankings, or ratings, providing scalable supervision for LLM post\-training\(Maliket al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib32)\)\. Rubric\-based evaluators, often implemented as LLM judges, make criteria explicit and score or compare responses under natural\-language rubrics\(Gunjalet al\.,[2026](https://arxiv.org/html/2607.20862#bib.bib13)\)\. Multi\-evaluator methods select, route among, or aggregate heterogeneous reward models to exploit complementary strengths across evaluators\(Wu and Lu,[2026](https://arxiv.org/html/2607.20862#bib.bib46); Wanget al\.,[2024b](https://arxiv.org/html/2607.20862#bib.bib42)\)\. These approaches move beyond direct correctness checking, but they differ in how evaluative information is represented: holistic preference labels, explicit natural\-language criteria, or final scalar scores from multiple experts\.

Despite this progress, existing mechanisms remain limited when preferences are composite\. In preference\-based reward modeling, pairwise comparisons and scalar rewards provide scalable supervision, but they collapse multidimensional quality judgments into holistic labels\. Such holistic signals offer limited fine\-grained credit assignment, leaving unclear which criteria drive the preference, which spans are problematic, or where revision is needed\(Wuet al\.,[2023](https://arxiv.org/html/2607.20862#bib.bib45)\)\. Rubric\-based evaluators make criteria explicit, but natural\-language rubrics can be incomplete, overlapping, or misaligned with the intended preference direction\. Their scores still depend on how the judge model interprets each criterion, which may introduce misalignment with human preferences\(Shenet al\.,[2026](https://arxiv.org/html/2607.20862#bib.bib39)\)\. Methods that use multiple evaluators address heterogeneity, but selection and routing reduce feedback to a chosen expert and may miss complementary criteria that should be considered jointly\. Scalar aggregation uses multiple experts, yet operates on final scores whose scales and semantics are shaped by different data sources, objectives, and calibration regimes\(Nguyenet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib33); Wu and Lu,[2026](https://arxiv.org/html/2607.20862#bib.bib46); Wanget al\.,[2024b](https://arxiv.org/html/2607.20862#bib.bib42)\)\. These limitations motivate evaluation mechanisms that integrate multidimensional evidence in a semantically aligned representation space and model interactions across criteria, rather than relying solely on holistic labels, judge\-dependent rubric scores, or uncalibrated scalar rewards\.

To this end, we introduce Constrained Shared\-Private Fusion \(CSPF\), a hidden\-state fusion method for evaluating LLM responses in non\-verifiable preference tasks\.CSPFtreats frozen reward models as complementary evaluative perspectives and learns, under pairwise human\-preference supervision, to fuse their hidden\-state representations rather than relying only on final scalar scores\. Its constrained shared\-private fusion structure separates cross\-expert shared representations from expert\-private representations, encouraging common preference\-relevant signals to align while preserving complementary expert viewpoints\. Because all reward\-expert backbones remain frozen, adaptation is localized within the fusion module, making the approach modular and extensible to newly released or domain\-specialized reward models\.

Our work makes three main contributions:

1. \(1\)CSPFas a method for non\-verifiable preference evaluation\.We proposeCSPF, a hidden\-representation\-level method for fusing multiple reward models, which implicitly models interactions among latent evaluative factors to improve non\-verifiable preference evaluation\.
2. \(2\)Hidden\-state fusion mechanism beyond scalar scores\.We compare hidden\-state and scalar\-score fusion methods, showing that representation\-level fusion achieves stronger performance and supports more sample\-dependent use of complementary reward experts\.
3. \(3\)Experimental validation of fusion design factors\.We analyze signal representation, expert\-pool composition, and fusion structure, showing that effective multi\-expert evaluation depends on coordinated design choices rather than simply adding more experts or scores\.

## 2Related Work

### 2\.1Preference\-Based Reward Models

Preference\-based reward models convert human comparisons, rankings, or ratings into learned scalar evaluators for LLM post\-training\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.20862#bib.bib34)\)\. Recent open models illustrate this ecosystem: Skywork\-Reward\-V2 emphasizes scalable preference\-data curation and general\-purpose reward modeling\(Liuet al\.,[2025a](https://arxiv.org/html/2607.20862#bib.bib28)\), OffsetBias improves robustness against evaluation bias\(Parket al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib35)\), and ArmoRM combines multiple interpretable reward objectives before producing an overall preference score\(Wanget al\.,[2024a](https://arxiv.org/html/2607.20862#bib.bib41)\)\. Benchmarks such as RewardBench, RewardBench 2, and PPE assess reward models across instruction following, reasoning, safety, and human\-preference settings\(Lambertet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib22); Maliket al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib32); Fricket al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib11)\)\. Despite this progress, holistic preference RMs collapse multiple quality dimensions into a single score, obscuring the criteria and trade\-offs underlying each judgment\. Consequently, improvements in aggregate preference can mask regressions in specific dimensions\(Wuet al\.,[2023](https://arxiv.org/html/2607.20862#bib.bib45)\)\.

### 2\.2Rubric\-Based LLM Evaluators

Rubric\-based LLM evaluators make criteria explicit by prompting or training judge models to score, compare, or critique responses under natural\-language rubrics\. Representative open evaluators include Prometheus\-2, which supports direct assessment and pairwise ranking with user\-defined criteria\(Kimet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib21)\), and R3, which develops rubric\-agnostic reward models with reasoned score assignments\(Anugrahaet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib1)\)\. Rubric\-based methods provide an interpretable interface, but their reliability depends on rubric coverage and the judge model’s interpretation of each criterion; rubrics can also be incomplete, redundant, overlapping, or misaligned with the intended preference direction\(Shenet al\.,[2026](https://arxiv.org/html/2607.20862#bib.bib39)\)\. This motivates complementary approaches that integrate multiple evaluative perspectives while grounding evaluation in human\-preference supervision rather than judge\-specific criterion interpretations\.

### 2\.3Multi\-Expert Reward\-Model Evaluation

A growing line of work studies how to use multiple reward models rather than relying on a single evaluator\. Reward\-model selection and routing methods choose among candidate reward models for each instance\(Nguyenet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib33); Wu and Lu,[2026](https://arxiv.org/html/2607.20862#bib.bib46)\), while scalar aggregation methods combine expert scores, such as reward\-model ensembles\(Eisensteinet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib7)\)and log\-sigmoid\-centered reward aggregation\(Wanget al\.,[2024b](https://arxiv.org/html/2607.20862#bib.bib42)\)\. These methods show that heterogeneous reward models can provide complementary signals, but selection, routing, or scalar\-score aggregation leaves limited room to model cross\-expert interactions before scalar compression\.

### 2\.4Hidden Representations Beyond Final Outputs

Prior work has shown that neural models encode useful information in internal representations that may not be fully exposed by final outputs\. Analyses of Transformer representations show that different layers capture different linguistic and task\-relevant properties\(Rogerset al\.,[2020](https://arxiv.org/html/2607.20862#bib.bib37)\), and reward\-model work has used hidden\-state regularization to improve generalization under distribution shift\(Yanget al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib47)\)\. Related work on multi\-model collaboration also suggests that internal expert representations can support prediction beyond final generated outputs or scalar decisions\(Fein\-Ashleyet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib9)\)\. However, how to align and fuse hidden states from multiple frozen reward experts for non\-verifiable preference evaluation remains underexplored\.

## 3Method

To address the limitations of existing work, we propose Constrained Shared\-Private Fusion \(CSPF\), a hidden\-state fusion method that learns a target\-domain reward function from frozen reward experts under pairwise human\-preference supervision\. As shown in Figure[1](https://arxiv.org/html/2607.20862#S3.F1), the method is specified by three design factors: signal representation, expert pool, and fusion structure\. We detail these factors and the training objective below\.

![Refer to caption](https://arxiv.org/html/2607.20862v1/x1.png)Figure 1:Overview of hidden\-state evidence fusion from multiple reward experts for non\-verifiable preference modeling\. The figure abstracts three design factors in our study: signal interface, expert pool, and fusion structure\. The structured fusion block is instantiated byCSPF\. The radar chart and expert\-pool map are illustrative\.### 3\.1Signal Representation

Letxxdenote a prompt andyya candidate response, and letEkE\_\{k\}, fork∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}, denote thekk\-th ofKKfrozen reward experts\. Each expert exposes a scalar reward scoresk​\(x,y\)∈ℝs\_\{k\}\(x,y\)\\in\\mathbb\{R\}\. Because raw score scales can differ across experts, we normalizesk​\(x,y\)s\_\{k\}\(x,y\)using target\-domain training statistics:

s~k​\(x,y\)=sk​\(x,y\)−μkmax⁡\{σk,ϵ\}\\tilde\{s\}\_\{k\}\(x,y\)=\\frac\{s\_\{k\}\(x,y\)\-\\mu\_\{k\}\}\{\\max\\\{\\sigma\_\{k\},\\epsilon\\\}\}\(1\)whereμk\\mu\_\{k\}andσk\\sigma\_\{k\}are estimated on the training split and fixed thereafter, andϵ\>0\\epsilon\>0lower\-bounds the denominator for numerical stability\. The normalized scores~k​\(x,y\)\\tilde\{s\}\_\{k\}\(x,y\)is retained as an auxiliary calibration signal\.

Each expert also exposes a hidden representation before its final scalar output\. Because these hidden states lie in model\-specific representation spaces and may differ in dimensionality, we map a selected hidden\-state readouthk​\(x,y\)∈ℝdk,hh\_\{k\}\(x,y\)\\in\\mathbb\{R\}^\{d\_\{k,h\}\}into a common representation space:

zk​\(x,y\)=Pk​\(LNk​\(hk​\(x,y\)\)\)∈ℝdz\_\{k\}\(x,y\)=P\_\{k\}\\\!\\left\(\\mathrm\{LN\}\_\{k\}\(h\_\{k\}\(x,y\)\)\\right\)\\in\\mathbb\{R\}^\{d\}\(2\)whereLNk\\mathrm\{LN\}\_\{k\}andPkP\_\{k\}are expert\-specific normalization and projection layers\. The projected representationzk​\(x,y\)z\_\{k\}\(x,y\)is the main signal consumed by adapters and fusion modules\. Appendix[A\.2](https://arxiv.org/html/2607.20862#A1.SS2)details the hidden\-state extraction protocol; Section[5\.3](https://arxiv.org/html/2607.20862#S5.SS3)evaluates alternative layer and span choices\.

### 3\.2Expert Pool

We denote the candidate pool ofKKpretrained reward experts byℰ\\mathcal\{E\}and the active expert index set for each configuration byCC:

ℰ=\{Ek\}k=1K,C⊆\{1,…,K\}\\mathcal\{E\}=\\\{E\_\{k\}\\\}\_\{k=1\}^\{K\},\\qquad C\\subseteq\\\{1,\\ldots,K\\\}\(3\)Only expertsEkE\_\{k\}withk∈Ck\\in Ccontribute the normalized scores~k​\(x,y\)\\tilde\{s\}\_\{k\}\(x,y\)and projected hidden representationzk​\(x,y\)z\_\{k\}\(x,y\)defined in Section[3\.1](https://arxiv.org/html/2607.20862#S3.SS1)to fusion\. All expert backbones remain frozen; only modules built on these signals are trained\.

The setCCis fixed across examples rather than selected per instance; sample\-dependent interactions among expert signals are modeled by the fusion module\. The experts are treated as complementary, potentially overlapping evaluative perspectives, without assuming a one\-to\-one mapping to human\-defined criteria\. Concrete expert models and roles are described in Section[4\.2](https://arxiv.org/html/2607.20862#S4.SS2)and Appendix[A\.1](https://arxiv.org/html/2607.20862#A1.SS1)\.

### 3\.3Structured Fusion:CSPF

![Refer to caption](https://arxiv.org/html/2607.20862v1/x2.png)Figure 2:Architecture ofCSPF\. The left branch shows the per\-expert signal path; the right branch illustrates shared\-private fusion over hidden representations from the active expert pool\. Normalized scalar scores are retained as auxiliary calibration signals\.CSPFinstantiates the fusion structure by decomposing each projected hidden representation into a cross\-expert shared component and an expert\-specific private component, as illustrated in Figure[2](https://arxiv.org/html/2607.20862#S3.F2)\.

For each expertEkE\_\{k\}, we apply a shared encoderEsE\_\{s\}and an expert\-specific private encoderEp,kE\_\{p,k\}:

ck​\(x,y\)=Es​\(zk​\(x,y\)\)pk​\(x,y\)=Ep,k​\(zk​\(x,y\)\)\\begin\{array\}\[\]\{rcl\}c\_\{k\}\(x,y\)&=&E\_\{s\}\\\!\\left\(z\_\{k\}\(x,y\)\\right\)\\\\\[2\.0pt\] p\_\{k\}\(x,y\)&=&E\_\{p,k\}\\\!\\left\(z\_\{k\}\(x,y\)\\right\)\\end\{array\}\(4\)whereckc\_\{k\}captures shared preference\-relevant information across experts andpkp\_\{k\}preserves expert\-specific information\.

For an active expert poolC=\{k1,…,km\}C=\\\{k\_\{1\},\\ldots,k\_\{m\}\\\}, we form a sequence of shared and private expert tokens,

XC​\(x,y\)=\[ck1,pk1,…,ckm,pkm\]X\_\{C\}\(x,y\)=\[c\_\{k\_\{1\}\},p\_\{k\_\{1\}\},\\ldots,c\_\{k\_\{m\}\},p\_\{k\_\{m\}\}\]\(5\)and map it to a fused representation with a fusion moduleℱθ\\mathcal\{F\}\_\{\\theta\}:

vC​\(x,y\)=vec⁡\(ℱθ​\(XC​\(x,y\)\)\)v\_\{C\}\(x,y\)=\\operatorname\{vec\}\\\!\\left\(\\mathcal\{F\}\_\{\\theta\}\(X\_\{C\}\(x,y\)\)\\right\)\(6\)The final reward combines this fused hidden representation with normalized scalar scores used only as calibration signals:

rθ​\(x,y\)\\displaystyle r\_\{\\theta\}\(x,y\)=gθ​\(\[vC​\(x,y\);s~C​\(x,y\)\]\)\\displaystyle=g\_\{\\theta\}\\\!\\left\(\[v\_\{C\}\(x,y\);\\tilde\{s\}\_\{C\}\(x,y\)\]\\right\)\(7\)s~C​\(x,y\)\\displaystyle\\tilde\{s\}\_\{C\}\(x,y\)=\[s~k1​\(x,y\);…;s~km​\(x,y\)\]\\displaystyle=\[\\tilde\{s\}\_\{k\_\{1\}\}\(x,y\);\\ldots;\\tilde\{s\}\_\{k\_\{m\}\}\(x,y\)\]This reward function is trained with the pairwise preference objective and auxiliary constraints described next\.

### 3\.4Training Objective

We train all trainable evaluators onNNpairwise preference examples\(xi,yi\+,yi−\)\(x\_\{i\},y\_\{i\}^\{\+\},y\_\{i\}^\{\-\}\), whereyi\+y\_\{i\}^\{\+\}is preferred overyi−y\_\{i\}^\{\-\}\. With marginΔi=rθ​\(xi,yi\+\)−rθ​\(xi,yi−\)\\Delta\_\{i\}=r\_\{\\theta\}\(x\_\{i\},y\_\{i\}^\{\+\}\)\-r\_\{\\theta\}\(x\_\{i\},y\_\{i\}^\{\-\}\), the primary task loss is the Bradley–Terry objective

ℒBT=1N​∑i=1Nsoftplus⁡\(−Δi\)\\mathcal\{L\}\_\{\\mathrm\{BT\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\operatorname\{softplus\}\(\-\\Delta\_\{i\}\)\(8\)
ForCSPF, we further constrain the shared and private representations\. For each response sideρ∈\{\+,−\}\\rho\\in\\\{\+,\-\\\}, let𝐂¯kρ∈ℝB×d\\overline\{\\mathbf\{C\}\}\_\{k\}^\{\\rho\}\\in\\mathbb\{R\}^\{B\\times d\}collect the feature\-wise standardized shared representations of expertEkE\_\{k\}over a minibatch ofBBpreference pairs\. Let𝒫C=\{\(a,b\):a,b∈C,a<b\}\\mathcal\{P\}\_\{C\}=\\\{\(a,b\):a,b\\in C,\\ a<b\\\}\. We use a mean\-reduced Barlow Twins\-style objective\(Zbontaret al\.,[2021](https://arxiv.org/html/2607.20862#bib.bib49)\):

Ra​bρ\\displaystyle R\_\{ab\}^\{\\rho\}=B−1​\(𝐂¯aρ\)⊤​𝐂¯bρ,\\displaystyle=B^\{\-1\}\\left\(\\overline\{\\mathbf\{C\}\}\_\{a\}^\{\\rho\}\\right\)^\{\\top\}\\overline\{\\mathbf\{C\}\}\_\{b\}^\{\\rho\},\(9\)ℒBarlow\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{Barlow\}\}=𝔼ρ,\(a,b\)\[‖diag⁡\(Ra​bρ\)−𝟏‖22d\\displaystyle=\\mathbb\{E\}\_\{\\rho,\(a,b\)\}\\left\[\\frac\{\\left\\\|\\operatorname\{diag\}\(R\_\{ab\}^\{\\rho\}\)\-\\mathbf\{1\}\\right\\\|\_\{2\}^\{2\}\}\{d\}\\right\.\+β​‖off⁡\(Ra​bρ\)‖F2d2\]\\displaystyle\\hskip 52\.00005pt\\left\.\+\\frac\{\\beta\\left\\\|\\operatorname\{off\}\(R\_\{ab\}^\{\\rho\}\)\\right\\\|\_\{F\}^\{2\}\}\{d^\{2\}\}\\right\]whereoff⁡\(⋅\)\\operatorname\{off\}\(\\cdot\)sets the diagonal to zero,β\\betaweights the off\-diagonal penalty, and𝔼ρ,\(a,b\)\\mathbb\{E\}\_\{\\rho,\(a,b\)\}averages uniformly over the two response sides and the expert pairs in𝒫C\\mathcal\{P\}\_\{C\}\. The two sides are standardized separately before averaging\.

For the private representations, let𝒬=\{\(qu,eu\)\}u=1M\\mathcal\{Q\}=\\\{\(q\_\{u\},e\_\{u\}\)\\\}\_\{u=1\}^\{M\}collect theM=2​B​\|C\|M=2B\|C\|representations from both response sides, witheue\_\{u\}denoting source\-expert identity\. WithP​\(u\)=\{p≠u:ep=eu\}P\(u\)=\\\{p\\neq u:e\_\{p\}=e\_\{u\}\\\}andsu​a=exp⁡\(sim⁡\(qu,qa\)/τ\)s\_\{ua\}=\\exp\(\\operatorname\{sim\}\(q\_\{u\},q\_\{a\}\)/\\tau\), the supervised contrastive objective\(Khoslaet al\.,[2020](https://arxiv.org/html/2607.20862#bib.bib20)\)is

ℒSupCon=−𝔼u​𝔼p∈P​\(u\)​log⁡su​p∑a≠usu​a\.\\mathcal\{L\}\_\{\\mathrm\{SupCon\}\}=\-\\mathbb\{E\}\_\{u\}\\mathbb\{E\}\_\{p\\in P\(u\)\}\\log\\frac\{s\_\{up\}\}\{\\sum\_\{a\\neq u\}s\_\{ua\}\}\.\(10\)where both expectations are uniform,sim⁡\(⋅,⋅\)\\operatorname\{sim\}\(\\cdot,\\cdot\)is cosine similarity, andτ\\tauis the temperature\.

With weightsλB\\lambda\_\{\\mathrm\{B\}\}andλS\\lambda\_\{\\mathrm\{S\}\}for the shared\- and private\-space constraints, the complete objective is

ℒCSPF=ℒBT\+λB​ℒBarlow\+λS​ℒSupCon\\mathcal\{L\}\_\{\\mathrm\{CSPF\}\}=\\mathcal\{L\}\_\{\\mathrm\{BT\}\}\+\\lambda\_\{\\mathrm\{B\}\}\\mathcal\{L\}\_\{\\mathrm\{Barlow\}\}\+\\lambda\_\{\\mathrm\{S\}\}\\mathcal\{L\}\_\{\\mathrm\{SupCon\}\}\(11\)Thus,ℒBT\\mathcal\{L\}\_\{\\mathrm\{BT\}\}provides task supervision to both branches, while the two auxiliary losses impose shared\- and private\-space structure, respectively\.

## 4Experimental Setup

### 4\.1Datasets

#### Target\-domain adaptation and validation\.

We adapt and validate on a cleaned LM\-Arena preference dataset derived from crowdsourced Chatbot Arena comparisons of LLM responses\(Chianget al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib4)\), matching our non\-verifiable pairwise evaluation setting\.

#### Out\-of\-distribution evaluation\.

We use the human\-preference split of Preference Proxy Evaluations \(PPE\)\(Fricket al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib11)\), which targets non\-verifiable preferences rather than objective correctness\. PPE is excluded from training and score\-normalization statistics and used only for OOD evaluation\.

### 4\.2Frozen Reward\-Expert Pool

The expert pool contains seven frozen scalar reward models that expose both reward scores and hidden\-state signals: Skywork\-Reward, AceMath, OffsetBias, ArmoRM, AceCodeRM, SARM, and URM\(Liuet al\.,[2025a](https://arxiv.org/html/2607.20862#bib.bib28),[b](https://arxiv.org/html/2607.20862#bib.bib26); Parket al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib35); Wanget al\.,[2024a](https://arxiv.org/html/2607.20862#bib.bib41); Zenget al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib50); Zhanget al\.,[2026](https://arxiv.org/html/2607.20862#bib.bib51); Louet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib29)\)\. Skywork\-Reward serves as the general anchor expert in single\-expert baselines and controlled comparisons\. Each experiment uses an active subset of the candidate experts, denoted byCCin Section[3\.2](https://arxiv.org/html/2607.20862#S3.SS2); the active subset is reported in the corresponding table or figure\. We focus on scalar reward models and exclude generative judges that require rationale or critique generation before scoring\. Detailed model versions and intended expert roles are summarized in Table[A\.1](https://arxiv.org/html/2607.20862#A1.T1)\.

### 4\.3Baselines

#### Single\-expert reward models\.

We evaluate individual reward\-expert scalar scores directly\. BecauseCSPFis adapted on LM\-Arena, we also adapt Skywork\-Reward with LoRA on the same data, giving a matched parameter\-update baseline with a comparable trainable\-parameter budget: 15\.34M for LoRA versus 17\.45M forCSPF, excluding frozen backbones\. A frozen\-backbone Skywork adapter is also included; it trains a prediction head over the same hidden\-state and normalized\-score signal interface asCSPF, but using a single expert\.

#### Rubric evaluators\.

The rubric\-family baselines include two open\-source evaluators, Prometheus\-2\-7B and R3\-Qwen3\-8B\-4k\(Kimet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib21); Anugrahaet al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib1)\)\. Prometheus\-2\-7B represents a pairwise rubric\-conditioned LLM judge, while R3\-Qwen3\-8B\-4k represents a rubric\-aware pointwise evaluator with scalar rubric scores\. Their pairwise and pointwise evaluation protocols are detailed in Appendix[A\.4](https://arxiv.org/html/2607.20862#A1.SS4)\.

#### Multi\-expert reward\-model fusion\.

Using the same active expert pool asCSPF, RM Ensemble averages normalized expert scores\(Eisensteinet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib7)\), whereas LSC adapts log\-sigmoid\-centered aggregation to the frozen\-expert pairwise setting\(Wanget al\.,[2024b](https://arxiv.org/html/2607.20862#bib.bib42)\); both fuse final scalar scores rather than hidden representations\. Definitions are provided in Appendix[A\.4](https://arxiv.org/html/2607.20862#A1.SS4)\.

### 4\.4Metrics

We use pairwise accuracy as the primary evaluation metric throughout\. On LM\-Arena, we report validation accuracy as the target\-domain metric\. On PPE, we report PPE off6, the equal\-weighted average of pairwise accuracy over the six human\-preference slices used in our evaluation, as the primary OOD metric\. This follows PPE’s metric analysis, which identifies pairwise accuracy as a strong predictor of downstream post\-RLHF human preference scores among human\-preference metrics\(Fricket al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib11)\)\. Overall non\-tie accuracy and slice\-level PPE results are used as diagnostics when relevant\.

### 4\.5Training Details

All trainable baselines andCSPFare optimized with the pairwise preference objective in Section[3\.4](https://arxiv.org/html/2607.20862#S3.SS4)\. For frozen\-expert methods, we precompute scalar scores and selected hidden\-state readouts on LM\-Arena and PPE; the Skywork LoRA baseline is trained separately because it updates the reward\-model backbone\. Appendix[A](https://arxiv.org/html/2607.20862#A1)details hidden\-state extraction, training configurations, and baseline settings\.

## 5Results and Analysis

### 5\.1Main Results

MethodInterfaceExpertsLM\-ArenaVal\. Acc\.PPE OODoff6HardEasyIFCodeMathSim\.O/NTSingle\-expert reward modelsOffsetBias\-8Bscalar scoreO48\.6255\.7955\.5955\.8956\.6257\.0054\.4155\.2459\.17ArmoRM\-8Bscalar scoreA54\.5955\.0855\.2655\.5555\.4455\.8353\.7054\.7059\.11Skywork\-V2\-8Bscalar scoreS60\.4360\.2460\.3761\.2960\.3860\.6059\.0359\.7867\.59Skywork\-V2 LoRAparameter updateS66\.2659\.7660\.6059\.1160\.1859\.9259\.52\\bm\{59\.52\}59\.2767\.02Skywork\-V2 adapter \(ours\)hidden stateS66\.84¯\\underline\{66\.84\}61\.17¯\\underline\{61\.17\}61\.60¯\\underline\{61\.60\}61\.84¯\\underline\{61\.84\}61\.09¯\\underline\{61\.09\}62\.19¯\\underline\{62\.19\}59\.35¯\\underline\{59\.35\}60\.94¯\\underline\{60\.94\}68\.78¯\\underline\{68\.78\}Rubric evaluatorsPrometheus\-2\-7Bnatural languageP58\.3053\.7753\.9753\.4652\.5255\.2553\.0254\.3757\.17R3\-8Bnatural languageR56\.2253\.4953\.9252\.4854\.1553\.5754\.1952\.6154\.69Multi\-expert reward\-model fusionRM Ensemblescalar scoresS\+O\+A56\.4958\.9359\.0959\.5459\.3360\.4756\.4058\.7465\.28LSCscalar scoresS\+O\+A56\.5859\.5460\.0059\.9459\.7661\.5056\.7259\.3566\.13CSPF\(ours\)hidden statesS\+O\+A68\.04\\bm\{68\.04\}61\.67\\bm\{61\.67\}62\.04\\bm\{62\.04\}62\.33\\bm\{62\.33\}62\.28\\bm\{62\.28\}63\.11\\bm\{63\.11\}59\.0761\.20\\bm\{61\.20\}69\.17\\bm\{69\.17\}

Table 1:Main comparison ofCSPFwith three families of evaluator baselines\. Values are pairwise accuracies \(%\);PPEoff6 averages the six PPE human\-preference slices, and O/NT denotes overall non\-tie accuracy\. S, O, A, P, and R denote Skywork, OffsetBias, ArmoRM, Prometheus, and R3, respectively\. Bold and underlined values mark the best and second\-best results\.![Refer to caption](https://arxiv.org/html/2607.20862v1/x3.png)Figure 3:Sample\-level expert\-contribution shares on the same 200 random PPE responses\. RM Ensemble is uniform by construction \(coincident traces are slightly offset for visibility\); LSC andCSPFvary across responses, withCSPFshowing stronger sample\-dependent shifts\.Table[1](https://arxiv.org/html/2607.20862#S5.T1)reports the main comparison on LM\-Arena target\-domain validation and PPE out\-of\-distribution evaluation\. Overall,CSPFachieves the highest LM\-Arena validation accuracy \(68\.04\) and PPE off6 accuracy \(61\.67\) among all evaluated methods, as well as the highest O/NT accuracy\.

CSPFoutperforms all raw single\-expert reward models\. Skywork LoRA, trained on the same LM\-Arena data, improves target\-domain accuracy but slightly lowers PPE off6 relative to raw Skywork, revealing a target\-domain/OOD trade\-off rather than improved OOD transfer\. Relative to Skywork LoRA,CSPFimproves LM\-Arena and PPE off6 by 1\.78 and 1\.91 percentage points, respectively\. The frozen\-backbone Skywork hidden\+score adapter ranks second on both primary metrics and O/NT, showing that augmenting a frozen expert with hidden\-state\-based adaptation is effective even without multi\-expert fusion\. The additional gains ofCSPFare consistent with a benefit from integrating complementary expert signals\.

For the two evaluated rubric models,CSPFalso achieves higher target\-domain and OOD accuracy\. Their gap from the leading preference\-trained reward models may partly reflect a mismatch between a judge model’s interpretation of explicit criteria and the holistic judgments reflected in human preferences\.

RM Ensemble and LSC use the same active expert pool asCSPFbut fuse only final scalar scores; details are provided in Appendix[A\.4](https://arxiv.org/html/2607.20862#A1.SS4)\.CSPFoutperforms LSC, the stronger multi\-expert baseline, by 11\.46 and 2\.13 percentage points on LM\-Arena and PPE off6, respectively\. Both multi\-expert methods also remain below stronger single\-expert baselines, suggesting that scalar\-score aggregation mainly smooths or averages expert scores rather than producing complementary gains beyond the strongest individual expert\.

To further understand the fusion behavior behind these multi\-expert methods, we convert each method into a sample\-level expert\-contribution share vector\. RM Ensemble has fixed uniform shares, LSC uses transformed scalar\-score contributions, andCSPFuses exact group\-Shapley attribution as a post\-hoc diagnostic\(Jullumet al\.,[2021](https://arxiv.org/html/2607.20862#bib.bib19)\)\. These shares are used only for diagnosis, not as learned router weights; formal definitions are given in Appendix[B](https://arxiv.org/html/2607.20862#A2)\.

Figure[3](https://arxiv.org/html/2607.20862#S5.F3)visualizes sample\-level expert\-contribution shares\. RM Ensemble is uniform by construction\. LSC shows moderate sample\-level variation, but remains relatively balanced across experts, behaving close to an averaging scheme\. By contrast,CSPFis overall more Skywork\-dominant, consistent with the strongest general reward model carrying greater evaluative reliance\. At the same time,CSPFvaries across samples: OffsetBias or ArmoRM receives larger shares on some examples, indicating that complementary experts can become more influential for particular cases\. A complementaryL1L\_\{1\}\-deviation analysis, reported in Appendix[B\.3](https://arxiv.org/html/2607.20862#A2.SS3), shows the same trend\.

The above analysis suggests thatCSPFdoes not merely average frozen reward experts, but learns a more flexible sample\-dependent fusion of complementary expert signals\.

### 5\.2Component Ablation ofCSPF

ConfigurationS/Penc\.SharedBarlowPrivateSupConScorecalib\.LM\-ArenaVal\. Acc\.PPE off6O/NTCSPFw/o S/P encoders–––✓\\checkmark67\.95±\\pm0\.3261\.11±\\pm0\.6668\.73±\\pm0\.69CSPFw/o auxiliary constraints✓\\checkmark––✓\\checkmark68\.03±\\pm0\.3061\.53±\\pm0\.2469\.03±\\pm0\.28CSPFw/o private SupCon✓\\checkmark✓\\checkmark–✓\\checkmark68\.01±\\pm0\.2661\.62±\\pm0\.2469\.12±\\pm0\.20CSPFw/o score calibration✓\\checkmark✓\\checkmark✓\\checkmark–67\.90±\\pm0\.1761\.37±\\pm0\.2868\.92±\\pm0\.59CSPF✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark68\.04±\\pm0\.1361\.67±\\pm0\.1969\.17±\\pm0\.13Table 2:Component ablation ofCSPF\. All settings follow the mainCSPFconfiguration except for the listed components\. Values are mean±\\pmstandard deviation over three seeds\.We ablate the main components ofCSPF\. The*Hidden fusion w/o S/P enc\.*control uses the same frozen expert signals and scalar\-score calibration path asCSPF, but feeds hidden representations directly into a fusion head\. We then cumulatively add shared/private encoders, the shared\-space Barlow constraint, and the private SupCon constraint\.

Table[2](https://arxiv.org/html/2607.20862#S5.T2)shows that shared/private encoders provide the primary OOD gain: replacing direct hidden fusion with shared/private factorization improves PPE off6 and O/NT, suggesting that organizing heterogeneous expert representations into shared and expert\-specific factors is more transferable than direct concatenation\. The Barlow and SupCon constraints add smaller but consistent gains, while score calibration provides only modest improvement, indicating that normalized scalar scores mainly act as auxiliary calibration signals\.

The ablation also shows improved seed\-level stability\. Shared/private encoders reduce the standard deviation on PPE off6 and O/NT relative to direct hidden fusion, and both auxiliary constraints further reduce dispersion\. Overall, the shared/private fusion structure is the main factor behindCSPF’s gains, while auxiliary constraints and scalar\-score calibration provide refinement\.

### 5\.3Signal Representation Analysis

Hidden\-state signalLayerLM\-Arena Val\. Acc\.PPE off6O/NTLast\-nonpad0\.25​L0\.25L66\.5255\.2661\.050\.375​L0\.375L67\.0956\.8462\.890\.5​L0\.5L67\.9358\.5665\.410\.625​L0\.625L68\.2859\.8366\.720\.75​L0\.75L67\.8461\.0468\.580\.875​L0\.875L67\.7760\.5867\.86LL67\.3959\.9967\.14Last\-nonpad \+ response mean0\.25​L0\.25L67\.0755\.7361\.620\.375​L0\.375L67\.0157\.1563\.500\.5​L0\.5L67\.9358\.5265\.480\.625​L0\.625L68\.1561\.3468\.910\.75​L0\.75L67\.8361\.4169\.100\.875​L0\.875L67\.8161\.4768\.94LL67\.7260\.3867\.50Last\-nonpad \+ all\-token mean0\.25​L0\.25L67\.2455\.7161\.780\.375​L0\.375L67\.0957\.1163\.380\.5​L0\.5L67\.4758\.3865\.330\.625​L0\.625L67\.8961\.2368\.850\.75​L0\.75L67\.9761\.5269\.210\.875​L0\.875L67\.5961\.7069\.05LL67\.2160\.9568\.02Table 3:Signal representation analysis forCSPF\. All rows use S\+O\+A and vary only the hidden\-state layer and span\. Intermediate layers are generally stronger, and the best layer–span choice differs across evaluation targets\.We analyze how hidden\-state signal representation affectsCSPF\. All settings follow the mainCSPFconfiguration except for the hidden\-state readout\. Table[3](https://arxiv.org/html/2607.20862#S5.T3)varies only the hidden\-state readout depth and span;LLdenotes the final block, and fractional values denote depth\-normalized intermediate blocks\.

Table[3](https://arxiv.org/html/2607.20862#S5.T3)shows two patterns\. First, final\-layer signals are not strongest: LM\-Arena peaks at the intermediate0\.625​L0\.625Llast\-nonpad signal, while very early signals transfer poorly to PPE\. This suggests that useful expert information is concentrated in intermediate layers, after preference\-relevant abstraction forms but before final scalar\-reward compression\. Second, the preferred representation differs by evaluation target: LM\-Arena favors a compact mid\-layer signal, whereas PPE benefits more from pooled span\-level evidence at later intermediate layers\. Thus, effective fusion depends not only on which experts are used, but also on which internal representations are exposed\.

### 5\.4Expert\-Pool Composition Analysis

![Refer to caption](https://arxiv.org/html/2607.20862v1/x4.png)Figure 4:Expert\-pool ladder ablation forCSPF\. \+AM, \+AC, \+SA, and \+U cumulatively add AceMath, AceCodeRM, SARM, and URM after S\+O\+A\. Points and error bars show three\-seed means and standard deviations\.We examine expert\-pool composition by varying only the active pool under a fixedCSPFstructure\. Starting from the Skywork anchor, we add experts along the cumulative ladder in Figure[4](https://arxiv.org/html/2607.20862#S5.F4)to test whether pool expansion yields monotonic gains\.

Performance improves from S to S\+O and peaks at S\+O\+A, the pool used by the mainCSPFconfiguration\. All subsequent larger pools score below S\+O\+A on all three metrics, with the largest drops on PPE off6 and O/NT\. Thus, adding experts can improve performance, but expert count alone is not a reliable scaling rule\. WithinCSPF, expert\-pool expansion is therefore a system\-level design decision that should account for both the fusion structure and intended application domain\.

## 6Discussion

#### Implicit Multi\-Perspective Evaluation\.

CSPFcan be viewed as an implicit multi\-perspective evaluator for non\-verifiable preference tasks\. Such tasks often depend on multiple coupled criteria that are difficult to specify, weight, or aggregate explicitly\. Rather than asking an LLM judge to interpret natural\-language rubrics,CSPFlearns under pairwise human\-preference supervision to fuse hidden representations from multiple frozen reward experts\. This does not imply thatCSPFrecovers explicit human criteria; instead, it provides a practical way to integrate latent evaluative factors that may underlie composite human preferences\.

Taken together, the results positionCSPFbetween three evaluator families\. Like holistic single\-expert reward models,CSPFis trained toward overall human preferences through pairwise supervision; unlike them, it integrates multiple frozen evaluators\. Like rubric\-based judges, it reflects the need for multiple evaluative perspectives, but obtains these perspectives from reward\-expert representations rather than explicit natural\-language criteria\. Compared with scalar\-score multi\-expert fusion, it models interactions among latent evaluative factors at the representation level\. This combination helps explain whyCSPFachieves stronger non\-verifiable preference evaluation on both LM\-Arena and PPE\.

#### Multi\-Expert Fusion as a Coupled Design\.

Our analyses show that, in theCSPFframework, multi\-expert hidden\-state fusion is a coupled design problem\. The signal representation determines which expert information is exposed, the expert pool determines which evaluative perspectives participate, and the fusion structure determines how heterogeneous representations are organized\. Thus, hidden representations provide a richer interface than final scalar rewards, but effective fusion still requires coordinated choices about where representations are extracted, which experts are included, and how their signals are fused\.

#### Limitations\.

Several limitations remain\. This work studiesCSPFas a static pairwise evaluator, using LM\-Arena for adaptation and PPE for OOD evaluation; whether its gains transfer to downstream settings, including best\-of\-NNselection, DPO, and other RLHF\-style policy\-optimization pipelines, remains to be tested\.CSPFalso focuses on scalar reward models that expose hidden states, leaving generative judges and critique\-then\-score evaluators outside the main fusion setting\. Future work should investigate these directions to better understand and extend multi\-expert hidden\-state fusion\.

## 7Conclusion

This work examined how to build evaluators for non\-verifiable preference tasks, where open\-ended LLM responses must be judged through composite human preferences rather than deterministic correctness signals\. Motivated by this direction, we proposedCSPF, a non\-verifiable preference evaluation method that fuses multiple frozen reward experts\. The method integrates complementary expert perspectives at the hidden\-representation level and implicitly models interactions among expert signals through constrained shared\-private fusion\. Empirically,CSPFyields the strongest primary results among the evaluated evaluator families\. The analyses further show that the gains depend not only on using multiple experts, but also on coordinated choices of signal representation, expert\-pool composition, and fusion structure\. Overall, our results suggest that hidden\-state fusion is a practical representation\-level interface for constructing integrated evaluative signals for non\-verifiable preference evaluation\.

## References

- D\. Anugraha, Z\. Tang, L\. J\. V\. Miranda, H\. Zhao, M\. R\. Farhansyah, G\. Kuwanto, D\. Wijaya, and G\. I\. Winata \(2025\)R3: Robust Rubric\-Agnostic Reward Models\.Note:https://arxiv\.org/abs/2505\.13388v3Cited by:[§2\.2](https://arxiv.org/html/2607.20862#S2.SS2.p1.1),[§4\.3](https://arxiv.org/html/2607.20862#S4.SS3.SSS0.Px2.p1.1)\.
- W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, H\. Zhang, B\. Zhu, M\. Jordan, J\. E\. Gonzalez, and I\. Stoica \(2024\)Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference\.Note:https://arxiv\.org/abs/2403\.04132v1Cited by:[§4\.1](https://arxiv.org/html/2607.20862#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Eisenstein, C\. Nagpal, A\. Agarwal, A\. Beirami, A\. N\. D’Amour, K\. D\. Dvijotham, A\. Fisch, K\. A\. Heller, S\. R\. Pfohl, D\. Ramachandran, P\. Shaw, and J\. Berant \(2024\)Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking\.InFirst Conference on Language Modeling,Cited by:[§2\.3](https://arxiv.org/html/2607.20862#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2607.20862#S4.SS3.SSS0.Px3.p1.1)\.
- J\. Fein\-Ashley, D\. Parikh, R\. Kannan, and V\. Prasanna \(2025\)Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say\.Note:https://arxiv\.org/abs/2509\.21164v1Cited by:[§2\.4](https://arxiv.org/html/2607.20862#S2.SS4.p1.1)\.
- E\. Frick, T\. Li, C\. Chen, W\. Chiang, A\. Angelopoulos, J\. Jiao, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica \(2025\)How to Evaluate Reward Models for RLHF\.International Conference on Learning Representations2025,pp\. 18128–18163\.Cited by:[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20862#S4.SS1.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2607.20862#S4.SS4.p1.1)\.
- J\. Gehring, K\. Zheng, J\. Copet, V\. Mella, Q\. Carbonneaux, T\. Cohen, and G\. Synnaeve \(2024\)RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning\.Note:https://arxiv\.org/abs/2410\.02089v2Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p1.1)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx \(2026\)Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p1.1),[§1](https://arxiv.org/html/2607.20862#S1.p2.1)\.
- R\. Jia, Y\. Yang, Y\. Gai, K\. Luo, S\. Huang, J\. Lin, X\. Jiang, and G\. Jiang \(2025\)Writing\-Zero: Bridge the Gap Between Non\-verifiable Tasks and Verifiable Rewards\.arXiv\.External Links:2506\.00103,[Document](https://dx.doi.org/10.48550/arXiv.2506.00103)Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p1.1)\.
- M\. Jullum, A\. Redelmeier, and K\. Aas \(2021\)groupShapley: Efficient prediction explanation with Shapley values for feature groups\.arXiv\.External Links:2106\.12228,[Document](https://dx.doi.org/10.48550/arXiv.2106.12228)Cited by:[§B\.2](https://arxiv.org/html/2607.20862#A2.SS2.p1.2),[§5\.1](https://arxiv.org/html/2607.20862#S5.SS1.p5.1)\.
- P\. Khosla, P\. Teterwak, C\. Wang, A\. Sarna, Y\. Tian, P\. Isola, A\. Maschinot, C\. Liu, and D\. Krishnan \(2020\)Supervised Contrastive Learning\.Advances in Neural Information Processing Systems33,pp\. 18661–18673\.Cited by:[§3\.4](https://arxiv.org/html/2607.20862#S3.SS4.p3.5)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024\)Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4334–4353\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.248)Cited by:[§2\.2](https://arxiv.org/html/2607.20862#S2.SS2.p1.1),[§4\.3](https://arxiv.org/html/2607.20862#S4.SS3.SSS0.Px2.p1.1)\.
- N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. Hajishirzi \(2025\)RewardBench: Evaluating Reward Models for Language Modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1755–1797\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96),ISBN 979\-8\-89176\-195\-7Cited by:[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1)\.
- C\. Y\. Liu, L\. Zeng, Y\. Xiao, J\. He, J\. Liu, C\. Wang, R\. Yan, W\. Shen, F\. Zhang, J\. Xu, Y\. Liu, and Y\. Zhou \(2025a\)Skywork\-Reward\-V2: Scaling Preference Data Curation via Human\-AI Synergy\.Note:https://arxiv\.org/abs/2507\.01352v3Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.2.3.1.1),[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.
- Z\. Liu, Y\. Chen, M\. Shoeybi, B\. Catanzaro, and W\. Ping \(2025b\)AceMath: Advancing Frontier Math Reasoning with Post\-Training and Reward Modeling\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3993–4015\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.206),ISBN 979\-8\-89176\-256\-5Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.3.3.1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.
- X\. Lou, D\. Yan, W\. Shen, Y\. Yan, J\. Xie, and J\. Zhang \(2024\)Uncertainty\-aware Reward Model: Teaching Reward Models to Know What is Unknown\.Note:https://arxiv\.org/abs/2410\.00847v2Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.8.3.1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.
- S\. Malik, V\. Pyatkin, S\. Land, J\. Morrison, N\. A\. Smith, H\. Hajishirzi, and N\. Lambert \(2025\)RewardBench 2: Advancing Reward Model Evaluation\.Note:https://arxiv\.org/abs/2506\.01937v2Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1)\.
- D\. Nguyen, A\. Prasad, E\. Stengel\-Eskin, and M\. Bansal \(2025\)LASeR: Learning to Adaptively Select Reward Models with Multi\-Arm Bandits\.Advances in Neural Information Processing Systems38,pp\. 116838–116876\.Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.20862#S2.SS3.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1)\.
- J\. Park, S\. Jwa, R\. Meiying, D\. Kim, and S\. Choi \(2024\)OffsetBias: Leveraging Debiased Data for Tuning Evaluators\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1043–1067\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.57)Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.4.3.1.1),[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.
- A\. Rogers, O\. Kovaleva, and A\. Rumshisky \(2020\)A Primer in BERTology: What We Know About How BERT Works\.Transactions of the Association for Computational Linguistics8,pp\. 842–866\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00349)Cited by:[§2\.4](https://arxiv.org/html/2607.20862#S2.SS4.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.Note:https://arxiv\.org/abs/2402\.03300v3Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p1.1)\.
- W\. F\. Shen, X\. Qiu, C\. Whitehouse, L\. Alazraki, S\. Goel, F\. Barbieri, T\. Willi, A\. Mathur, and I\. Leontiadis \(2026\)Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open\-ended Tasks\.arXiv\.External Links:2602\.05125,[Document](https://dx.doi.org/10.48550/arXiv.2602.05125)Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.20862#S2.SS2.p1.1)\.
- H\. Wang, W\. Xiong, T\. Xie, H\. Zhao, and T\. Zhang \(2024a\)Interpretable Preferences via Multi\-Objective Reward Modeling and Mixture\-of\-Experts\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10582–10592\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.620)Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.5.3.1.1),[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.
- Z\. Wang, C\. Nagpal, J\. Berant, J\. Eisenstein, A\. N\. D’Amour, S\. Koyejo, and V\. Veitch \(2024b\)Transforming and Combining Rewards for Aligning Large Language Models\.InProceedings of the 41st International Conference on Machine Learning,pp\. 51161–51176\.External Links:ISSN 2640\-3498Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p2.1),[§1](https://arxiv.org/html/2607.20862#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.20862#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2607.20862#S4.SS3.SSS0.Px3.p1.1)\.
- X\. Wu and Y\. Lu \(2026\)Reward Model Routing in Alignment\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p2.1),[§1](https://arxiv.org/html/2607.20862#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.20862#S2.SS3.p1.1)\.
- Z\. Wu, Y\. Hu, W\. Shi, N\. Dziri, A\. Suhr, P\. Ammanabrolu, N\. Smith, M\. Ostendorf, and H\. Hajishirzi \(2023\)Fine\-Grained Human Feedback Gives Better Rewards for Language Model Training\.Advances in Neural Information Processing Systems36,pp\. 59008–59033\.Cited by:[§1](https://arxiv.org/html/2607.20862#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.20862#S2.SS1.p1.1)\.
- R\. Yang, R\. Ding, Y\. Lin, H\. Zhang, and T\. Zhang \(2024\)Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs\.Advances in Neural Information Processing Systems37,pp\. 62279–62309\.External Links:[Document](https://dx.doi.org/10.52202/079017-1990)Cited by:[§2\.4](https://arxiv.org/html/2607.20862#S2.SS4.p1.1)\.
- J\. Zbontar, L\. Jing, I\. Misra, Y\. LeCun, and S\. Deny \(2021\)Barlow Twins: Self\-Supervised Learning via Redundancy Reduction\.arXiv\.External Links:2103\.03230,[Document](https://dx.doi.org/10.48550/arXiv.2103.03230)Cited by:[§3\.4](https://arxiv.org/html/2607.20862#S3.SS4.p2.5)\.
- H\. Zeng, D\. Jiang, H\. Wang, P\. Nie, X\. Chen, and W\. Chen \(2025\)ACECODER: Acing Coder RL via Automated Test\-Case Synthesis\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12023–12040\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.587),ISBN 979\-8\-89176\-251\-0Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.6.3.1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.
- S\. Zhang, W\. Shi, S\. Li, J\. Liao, T\. Liang, H\. Cai, and X\. Wang \(2026\)Interpretable Reward Model via Sparse Autoencoder\.Proceedings of the AAAI Conference on Artificial Intelligence40\(41\),pp\. 34808–34816\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v40i41.40783)Cited by:[Table A\.1](https://arxiv.org/html/2607.20862#A1.T1.1.7.3.1.1),[§4\.2](https://arxiv.org/html/2607.20862#S4.SS2.p1.1)\.

## Appendix

## Appendix AExperimental details

### A\.1Frozen reward expert pool

Table[A\.1](https://arxiv.org/html/2607.20862#A1.T1)summarizes the frozen reward experts considered in our study\.

ExpertAbbrev\.Model versionRoleSkyworkSSkywork\-Reward\-V2\-Qwen3\-8B\(Liuet al\.,[2025a](https://arxiv.org/html/2607.20862#bib.bib28)\)general\-purpose anchorAceMathAMAceMath\-7B\-RM\(Liuet al\.,[2025b](https://arxiv.org/html/2607.20862#bib.bib26)\)math\-orientedOffsetBiasOLlama\-3\-OffsetBias\-RM\-8B\(Parket al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib35)\)bias\-aware preferenceArmoRMAArmoRM\-Llama3\-8B\-v0\.1\(Wanget al\.,[2024a](https://arxiv.org/html/2607.20862#bib.bib41)\)multi\-objective preferenceAceCodeRMACAceCodeRM\-7B\(Zenget al\.,[2025](https://arxiv.org/html/2607.20862#bib.bib50)\)code\-domainSARMSALlama\-SARM\-4B\(Zhanget al\.,[2026](https://arxiv.org/html/2607.20862#bib.bib51)\)factuality / focusURMUURM\-LLaMa\-3\.1\-8B\(Louet al\.,[2024](https://arxiv.org/html/2607.20862#bib.bib29)\)attribute / calibrationTable A\.1:Frozen scalar reward experts considered in our study\. Depending on the experiment, we use their reward scores, hidden states, or both\.
### A\.2Hidden\-state signal extraction

For token\-level hidden\-state signals,*last\-nonpad*denotes the final non\-padding token representation,*response mean*denotes mean pooling over assistant\-response tokens, and*all\-token mean*denotes mean pooling over non\-padding prompt\-response tokens\. When multiple hidden\-state signals are used together, they are concatenated before layer normalization and expert\-specific projection\. Layer positions are specified as proportions of the expert backbone depth; for example,0\.75​L0\.75Ldenotes the proportional layer at three quarters of the transformer block stack\.

### A\.3Training hyperparameters

Table[A\.2](https://arxiv.org/html/2607.20862#A1.T2)summarizes the default training configuration used for the mainCSPFrow and matched ablations\. Unless otherwise specified, all trainable adapters and fusion heads are trained on LM\-Arena with frozen reward experts and evaluated with the same LM\-Arena validation and PPE protocols as in Section[4](https://arxiv.org/html/2607.20862#S4)\. Score\-normalization statistics are estimated only on the LM\-Arena training split and then fixed for validation and OOD evaluation\.

ItemValueActive expert poolS\+O\+AHidden\-state signallast\-nonpad \+ response meanLayer specification0\.75​L0\.75Lproportional layerFusion structureshared/private encoders \+ structured hidden fusionScore calibrationfinal concat of normalized scalar scoresMain objectiveBradley–Terry pairwise lossEpochs3Batch size128Learning rate1×10−41\\times 10^\{\-4\}Weight decay1×10−21\\times 10^\{\-2\}Dropout0\.1Gradient clipping1\.0Seeds40, 43, 45Shared\-spaceconstraintBarlow loss \(λB=1×10−3\\lambda\_\{\\mathrm\{B\}\}=1\\times 10^\{\-3\},β=0\.005\\beta=0\.005\)Private\-spaceconstraintSupCon loss \(λS=1×10−4\\lambda\_\{\\mathrm\{S\}\}=1\\times 10^\{\-4\},τ=0\.10\\tau=0\.10\)Score normalizationLM\-Arena train split onlyReward expertsfrozenTable A\.2:Default training configuration for the mainCSPFrow and matched ablations\.
### A\.4Baseline definitions

This appendix defines the baselines used in the main comparison\. Frozen\-expert baselines use one or both of the scalar\-score and hidden\-state interfaces defined in Section[3\.1](https://arxiv.org/html/2607.20862#S3.SS1), and all trainable baselines are optimized with the pairwise objective in Section[3\.4](https://arxiv.org/html/2607.20862#S3.SS4)\.

#### Raw scalar reward\.

The raw\-score baseline directly uses the scalar output of a single reward expert:

rraw,k​\(x,y\)=sk​\(x,y\)r\_\{\\mathrm\{raw\},k\}\(x,y\)=s\_\{k\}\(x,y\)\(A\.1\)

#### Single\-expert adapter\.

For frozen single\-expert adaptation, the reward expert remains fixed and we train a prediction head on top of its hidden\-state and normalized scalar\-score signals:

radapter,k​\(x,y\)=fθ​\(\[zk​\(x,y\);s~k​\(x,y\)\]\)r\_\{\\mathrm\{adapter\},k\}\(x,y\)=f\_\{\\theta\}\\\!\\left\(\[z\_\{k\}\(x,y\);\\tilde\{s\}\_\{k\}\(x,y\)\]\\right\)\(A\.2\)The adapter uses the optimizer settings in Table[A\.2](https://arxiv.org/html/2607.20862#A1.T2)but is trained for one epoch; the reported result is averaged over seeds 40–45\. For the LoRA baseline, we instead adapt theq/k/v/oq/k/v/oattention projections \(r=16r=16,α=32\\alpha=32, dropout=0\.05=0\.05\) together with the native reward head of Skywork\. It is trained with the same pairwise objective for one epoch using a learning rate of2×10−52\\times 10^\{\-5\}; the reported result uses seed 43\.

#### RM Ensemble \(Mean\)\.

RM Ensemble \(Mean\) adapts standard reward\-model ensemble aggregation to our frozen heterogeneous\-expert setting by averaging normalized scalar scores over the active expert pool:

rmean​\(x,y\)=1\|C\|​∑k∈Cs~k​\(x,y\)r\_\{\\mathrm\{mean\}\}\(x,y\)=\\frac\{1\}\{\|C\|\}\\sum\_\{k\\in C\}\\tilde\{s\}\_\{k\}\(x,y\)\(A\.3\)

#### LSC aggregation\.

LSC adapts log\-sigmoid\-centered reward aggregation to the same frozen expert pool\. For each expert, letqkq\_\{k\}be the median normalized score over LM\-Arena training responses:

qk=median\(x,y\)∈𝒟train⁡s~k​\(x,y\)q\_\{k\}=\\operatorname\{median\}\_\{\(x,y\)\\in\\mathcal\{D\}\_\{\\mathrm\{train\}\}\}\\tilde\{s\}\_\{k\}\(x,y\)\(A\.4\)The LSC reward is

rLSC​\(x,y\)=∑k∈Clog⁡σ​\(s~k​\(x,y\)−qk\)r\_\{\\mathrm\{LSC\}\}\(x,y\)=\\sum\_\{k\\in C\}\\log\\sigma\\\!\\left\(\\tilde\{s\}\_\{k\}\(x,y\)\-q\_\{k\}\\right\)\(A\.5\)

#### Rubric\-evaluator protocols\.

Both rubric baselines use fixed, dataset\-independent general\-quality rubrics\. For each baseline, the same rubric is applied to LM\-Arena and PPE without dataset\- or slice\-specific modification\. Figure[A\.1](https://arxiv.org/html/2607.20862#A1.F1)presents the exact rubric text and required decision/score formats used in our experiments\.

Prometheus\-2: Pairwise EvaluationReference\-answer fieldNo reference answer is provided because this is a non\-verifiable preference task\. Judge the responses only according to the instruction and the score rubric\.Evaluation rubricChoose the response that a human evaluator would prefer for the given instruction\. Consider: \(1\) instruction following and responsiveness to the user’s request; \(2\) correctness, factuality, and sound reasoning; \(3\) completeness, usefulness, and actionable detail; \(4\) safety, appropriateness, and avoidance of harmful or misleading advice; and \(5\) clarity, organization, and style\. For math, code, or technical tasks, give special weight to correctness and executable reasoning\. When criteria conflict, prefer the response that is more helpful and reliable overall\.Output constraintReturn the final decision as\[RESULT\] Aor\[RESULT\] B\.

R3: Pointwise EvaluationEvaluation rubricUse the following 1–5 scale to judge how good the response is for the instruction\.5 = Excellent:fully follows the instruction, is correct and reliable, complete, useful, safe, and clearly written\.4 = Good:mostly follows the instruction and is useful, with only minor omissions or weaknesses\.3 = Fair:partially useful but has noticeable omissions, weak reasoning, unclear presentation, or limited helpfulness\.2 = Poor:substantially incomplete, unhelpful, unsafe, or contains significant mistakes\.1 = Very poor:fails the task, is mostly incorrect, harmful, or irrelevant\.Consider instruction following, correctness/factuality, reasoning quality, completeness, usefulness, safety, and clarity\. For math, code, or technical tasks, give special weight to correctness and executable reasoning\. For open\-ended non\-verifiable tasks, prefer the response that would be more helpful and reliable to a human user\.Output constraintBegin the output withScore: <one number from 1 to 5\>, followed by a brief assessment\.

Figure A\.1:Exact rubric text and required decision formats used for the two rubric evaluators\. Prometheus\-2 compares a response pair directly, whereas R3 scores the two responses independently\.For Prometheus\-2, response position is balanced by deterministic hashing, and unparsed decisions receive half credit\. A reverse\-order audit on 10% of the examples is used only for diagnosis\. For R3, the higher\-scoring response is predicted as preferred; equal scores and unparsed outputs receive half credit\. Prometheus\-2 uses greedy decoding, whereas R3 uses non\-thinking decoding with temperature 0\.6, top\-pp0\.95, and top\-kk20; both use at most 512 generated tokens, with input budgets of 7,168 and 8,192 tokens, respectively\.

## Appendix BExpert\-Use Share Diagnostics

This appendix defines post\-hoc expert\-contribution shares for RM Ensemble, LSC, andCSPF; these diagnostics are neither learned router weights nor part of training or inference\.

### B\.1Scalar\-Score Contribution Shares

For a sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\), RM Ensemble assigns each active expert a fixed uniform share:

ui,k\(Mean\)=1\|C\|,k∈Cu\_\{i,k\}^\{\(\\mathrm\{Mean\}\)\}=\\frac\{1\}\{\|C\|\},\\qquad k\\in C\(B\.1\)
For LSC, let

ℓi,k=log⁡σ​\(s~k​\(xi,yi\)−qk\)\\ell\_\{i,k\}=\\log\\sigma\\\!\\left\(\\tilde\{s\}\_\{k\}\(x\_\{i\},y\_\{i\}\)\-q\_\{k\}\\right\)\(B\.2\)whereqkq\_\{k\}is the training\-set reference defined in Appendix[A\.4](https://arxiv.org/html/2607.20862#A1.SS4)\. We use the normalized absolute transformed\-score contribution

ui,k\(LSC\)=\|ℓi,k\|∑j∈C\|ℓi,j\|u\_\{i,k\}^\{\(\\mathrm\{LSC\}\)\}=\\frac\{\|\\ell\_\{i,k\}\|\}\{\\sum\_\{j\\in C\}\|\\ell\_\{i,j\}\|\}\(B\.3\)

### B\.2CSPFGroup\-Shapley Shares

ForCSPF, the shared representation, private representation, and normalized scalar score from each active expert form one group\. For each sample, we compute exact Group\-Shapley values over these groups\(Jullumet al\.,[2021](https://arxiv.org/html/2607.20862#bib.bib19)\), usingFi​\(T\)=rθ\(T\)​\(xi,yi\)F\_\{i\}\(T\)=r\_\{\\theta\}^\{\(T\)\}\(x\_\{i\},y\_\{i\}\)as the coalition value and replacing groups outsideTTwith their LM\-Arena training\-split means\. For S\+O\+A, all eight coalitions are evaluated\.

We convert the signed values to normalized absolute contribution shares

pi,k=\|ϕi,k\|∑j∈C\|ϕi,j\|p\_\{i,k\}=\\frac\{\|\\phi\_\{i,k\}\|\}\{\\sum\_\{j\\in C\}\|\\phi\_\{i,j\}\|\}\(B\.4\)After excluding zero\-denominator samples, we set𝐮i\(CSPF\)=\(pi,k\)k∈C\\mathbf\{u\}\_\{i\}^\{\(\\mathrm\{CSPF\}\)\}=\(p\_\{i,k\}\)\_\{k\\in C\}\.

### B\.3L1L\_\{1\}Deviation from Global Expert\-Use Mean

To quantify sample\-level variation in expert use, we compute theL1L\_\{1\}distance between each sample’s expert\-use share vector and the method\-specific global mean\. For a methodmmwithNNvalid samples, let𝐮i\(m\)∈ℝ\|C\|\\mathbf\{u\}\_\{i\}^\{\(m\)\}\\in\\mathbb\{R\}^\{\|C\|\}denote its normalized expert\-use share vector and let

𝐮¯\(m\)=1N​∑i=1N𝐮i\(m\)\\bar\{\\mathbf\{u\}\}^\{\(m\)\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{u\}\_\{i\}^\{\(m\)\}\(B\.5\)
The sample\-level deviation is

di\(m\)=‖𝐮i\(m\)−𝐮¯\(m\)‖1d\_\{i\}^\{\(m\)\}=\\left\\\|\\mathbf\{u\}\_\{i\}^\{\(m\)\}\-\\bar\{\\mathbf\{u\}\}^\{\(m\)\}\\right\\\|\_\{1\}\(B\.6\)
A larger value indicates stronger deviation from the method’s average expert\-use pattern\. RM Ensemble has zero deviation by construction and is therefore omitted from Figure[B\.1](https://arxiv.org/html/2607.20862#A2.F1)\.

![Refer to caption](https://arxiv.org/html/2607.20862v1/x5.png)Figure B\.1:Distribution of sample\-level expert\-use variation on 2000 randomly sampled PPE responses\. TheL1L\_\{1\}distance is computed between each sample’s expert\-use share vector and the method\-specific global mean\.CSPFhas a more dispersed and longer\-tailed distribution than LSC\.