Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

# Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
Source: [https://arxiv.org/html/2608.06400](https://arxiv.org/html/2608.06400)
Yifan Wang1,Jinyi Mu2,Mayank Jobanputra1,Yu Wang3, Soyoung Oh1,Isabel Valera1,4,Vera Demberg1,5 1Saarland University2University of California San Diego3Bielefeld University 4Max Planck Institute for Software Systems5Max Planck Institute for Informatics yifwang@lst\.uni\-saarland\.de

###### Abstract

Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging\. Recent sparse Mixture\-of\-Experts \(MoE\) reward models seek to improve interpretability by routing prompts to specialized experts and characterizing experts through examples with high routing weights\. However, routing weights only reveal which prompts an expertreceives, not how itjudgesresponses, providing only a partial account of expert behavior\. We therefore proposeContribution\-Contrast \(CoCo\) response\-level interpretation, which faithfully characterizes experts’ roles using chosen–rejected response pairs with the largest contribution contrasts, jointly capturing routing and preference behavior\. Across automatic and human evaluations, CoCo yields more coherent, faithful, and specialized interpretations than router\-based, score\-based, and sparse autoencoder\-based alternatives while maintaining competitive reward modeling accuracy\. To the best of our knowledge, this is the first systematic study of interpretation methods for MoE reward models\.

Beyond Routing Weights: Faithful Response\-Level Interpretation of Mixture\-of\-Experts Reward Models via Contribution Contrast

Yifan Wang1, Jinyi Mu2, Mayank Jobanputra1, Yu Wang3,Soyoung Oh1,Isabel Valera1,4,Vera Demberg1,51Saarland University2University of California San Diego3Bielefeld University4Max Planck Institute for Software Systems5Max Planck Institute for Informaticsyifwang@lst\.uni\-saarland\.de

## 1Introduction

Reward models play a central role in reinforcement learning from human feedback \(RLHF\) and in aligning large language models \(LLMs\) with human preferences\(Baiet al\.,[2022](https://arxiv.org/html/2608.06400#bib.bib1); Kaufmannet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib2)\)\. However, understanding which properties drive their preference predictions remains challenging\(Luoet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib27)\)\. Recent work therefore seeks to discover interpretable preference patterns directly from standard preference data, without relying on costly fine\-grained annotations\(Zhanget al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib76); Movvaet al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib57)\)\.

Sparse mixture\-of\-experts \(MoE\) reward models offer a promising approach by using a prompt\-conditioned router to assign inputs to specialized response\-scoring experts\(Wanget al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib78)\)\. With sparsity and diversity regularization, these models learn coherent routing patterns and experts specialized in distinct domains\. However, the choice of signal for ranking and selecting examples to interpret MoE experts remains underexplored\.Wanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)interpret experts using examples with the highest routing weights, but these weights reveal only which prompts an expertreceives, not how itjudgesresponses\. Such interpretations thus provide only an incomplete account of experts’ roles in MoE, and primarily capture the prompt\-level domains in which each expert specializes, rather than the response\-level preference dimensions that directly explain the model’s decisions\.111See Appendix[A](https://arxiv.org/html/2608.06400#A1)for a detailed discussion of related work\.

To address these limitations, we proposeCoCo, aContributionContrast\-based response\-level interpretation method for MoE reward models\. CoCo uses contribution contrast as the interpretation signal and characterizes experts using response pairs with large product of the routing weight and absolute expert score difference, thereby capturing the interaction between where an expert is used and how it distinguishes responses\. We further adapt existing interpretability regularizers to encourage sparse and diverse contribution contrast patterns\. Across automatic and human evaluations on two datasets, CoCo produces more coherent, faithful, and specialized interpretations than router\-based, score\-based, and sparse autoencoder \(SAE\) alternatives while maintaining competitive reward modeling accuracy\. Controlled comparisons within the same MoE and across signal\-specific training objectives further show that CoCo provides a more informative account of expert behavior than routing weights or expert score differences alone\.

MethodInterpretationSAE\-basedCaptures response\-level features, but their weights are typically fixed, making the method less suitable for modeling flexible input\-dependent preferences\.Router\-basedCaptures prompt\-level domains or contexts in which each expert is used, but not how it distinguishes between responses\.Score\-basedCaptures response\-level preference dimensions, but may emphasize examples where the expert has little influence because of low routing weight\.CoCoCaptures response\-level preference dimensions that are faithful to model decisions by accounting for the interaction between routing weights and expert scores\.Table 1:Conceptual comparison of different interpretability signals used for interpreting reward models\.
## 2Interpreting MoE Reward Models via CoCo

We first introduce the MoE reward model architecture, the contribution contrast\-based interpretability regularizers, and the procedure for generating CoCo interpretations\. We also define two alternative MoE interpretation methods using router\-based and score\-based signals as comparison baselines\.

#### MoE Reward Model

To capture heterogeneous and interpretable preference patterns, prior work implements reward models using the MoE architecture\(Wanget al\.,[2024b](https://arxiv.org/html/2608.06400#bib.bib12); Shenet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib11)\)\. Given a preference pair\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\), an MoE reward model consists of a prompt\-conditioned router𝝅ϕ​\(x\)∈ΔK−1\\boldsymbol\{\\pi\}\_\{\\phi\}\(x\)\\in\\Delta^\{K\-1\}andKKreward experts\{rθk\}k=1K\\\{r\_\{\\theta\_\{k\}\}\\\}\_\{k=1\}^\{K\}\. The router produces a softmax distribution over experts, whereπϕ,k​\(x\)\\pi\_\{\\phi,k\}\(x\)denotes the weight assigned to expertkk\. The final reward is represented as the weighted average of expert scores:

r​\(x,y\)=∑k=1Kπϕ,k​\(x\)​rθk​\(x,y\)\.r\(x,y\)=\\sum\_\{k=1\}^\{K\}\\pi\_\{\\phi,k\}\(x\)\\,r\_\{\\theta\_\{k\}\}\(x,y\)\.\(1\)Conventionally, the MoE reward model is trained on a preference dataset𝒟=\{\(x,yw,yl\)\}\\mathcal\{D\}=\\\{\(x,y\_\{w\},y\_\{l\}\)\\\}using the Bradley–Terry objective\.

#### CoCo\-Based Interpretability Regularization

Wanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)introduce three interpretability regularizers for learning disentangled and specialized experts: local sparsityℒls\\mathcal\{L\}\_\{\\mathrm\{ls\}\}, global balanceℒgb\\mathcal\{L\}\_\{\\mathrm\{gb\}\}, and expert diversityℒdiv\\mathcal\{L\}\_\{\\mathrm\{div\}\}\. These regularizers respectively promote sparse routing weights, balanced expert utilization, and diverse expert behavior\. Since they use routing weights as the interpretability signal, these regularizers primarily operate onπϕ​\(x\)\\pi\_\{\\phi\}\(x\)\. CoCo instead interprets experts through their contribution contrasts to the final rewards and thus applies the regularizers to pairwise contribution contrasts\. For expertkk, we define its signed contribution contrast and magnitude as:

Δ​ck​\(x,yw,yl\)=\\displaystyle\\Delta c\_\{k\}\(x,y\_\{w\},y\_\{l\}\)=πϕ,k​\(x\)​\[rθk​\(x,yw\)−rθk​\(x,yl\)\],\\displaystyle\\pi\_\{\\phi,k\}\(x\)\\left\[r\_\{\\theta\_\{k\}\}\(x,y\_\{w\}\)\-r\_\{\\theta\_\{k\}\}\(x,y\_\{l\}\)\\right\],\(2\)ck​\(x,yw,yl\)\\displaystyle c\_\{k\}\(x,y\_\{w\},y\_\{l\}\)=\|Δ​ck​\(x,yw,yl\)\|\.\\displaystyle=\\left\|\\Delta c\_\{k\}\(x,y\_\{w\},y\_\{l\}\)\\right\|\.The magnitudeckc\_\{k\}measures how strongly the expert distinguishes the responses, while the sign ofΔ​ck\\Delta c\_\{k\}indicates which response it supports\.

Let𝐜​\(x,yw,yl\)=\[c1,…,cK\]\\mathbf\{c\}\(x,y\_\{w\},y\_\{l\}\)=\[c\_\{1\},\\ldots,c\_\{K\}\]denote the contribution contrast profile of a preference pair\. We apply the sparsity, balance, and diversity regularizers to these profiles rather than to routing weights, encouraging interpretable and distinct contribution patterns \(formal definitions are provided in Appendix[B](https://arxiv.org/html/2608.06400#A2)\)\. This lightweight adaptation leaves the MoE architecture unchanged, while aligning the interpretability constraints with the contrastive contribution signal used by CoCo\. An ablation study in Appendix[C](https://arxiv.org/html/2608.06400#A3)analyzes the effect of the CoCo\-based interpretability regularization\.

#### CoCo Interpretation Extraction

FollowingMovvaet al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib57)\), who show that preferences are better characterized through contrastive examples, we interpret each MoE expert using chosen–rejected response pairs\. For expertkk, we select the 10 validation examples with the largest contribution contrast magnitudesck​\(x,yw,yl\)c\_\{k\}\(x,y\_\{w\},y\_\{l\}\)and prompt an LLM to summarize their shared contrastive pattern\. We provide the input prompt and both responses, and indicate which response receives the larger contribution from the target expert\. The resulting interpretation may capture a shared context, a consistent preference dimension, or both\.

#### Alternative Interpretation Signals

While CoCo captures the interaction between routing and expert behavior, we define two alternative interpretation signals as comparison baselines: router\-based and score\-based interpretation\. When interpreting expertkk, router\-based interpretation ranks examples by the routing weightπϕ,k​\(x\)\\pi\_\{\\phi,k\}\(x\),222The router\-based baseline is identical to the interpretation method used inWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)\.whereas score\-based interpretation ranks them by the absolute expert score difference\|rθk​\(x,yw\)−rθk​\(x,yl\)\|\|r\_\{\\theta\_\{k\}\}\(x,y\_\{w\}\)\-r\_\{\\theta\_\{k\}\}\(x,y\_\{l\}\)\|\.

DatasetMethodRM Acc\.Interpretation QualityDecision FaithfulnessExpert SpecializationFidelityRedundancy \(↓\\downarrow\)EM Agree\.Removal FlipExpert Acc\.Expert Adv\.700KSARM75\.720\.1410\.64164\.440\.6652\.310\.95WIMHF60\.430\.2020\.51659\.657\.1954\.25\-7\.63SMoE83\.780\.3670\.57987\.100\.5977\.406\.25CoCo83\.560\.4110\.58687\.551\.5382\.243\.04RedditSARM62\.730\.1730\.67268\.261\.4056\.560\.69WIMHF65\.470\.2320\.51662\.628\.9455\.970\.52SMoE72\.410\.3410\.77057\.611\.2851\.15\-0\.39CoCo70\.740\.3740\.59493\.447\.9478\.936\.21

Table 2:Task performance and interpretability results of different interpretable reward models on 700K and Reddit\. Best scores are marked inbold\. CoCo achieves the strongest overall interpretability across both datasets\.

## 3Experiments

We show CoCo’s improved interpretability over baselines through automatic and human evaluations across two datasets\. Appendix[E](https://arxiv.org/html/2608.06400#A5)provides representative examples from different methods and analyzes how their content and style are affected by their respective interpretation signals\.

### 3\.1Experimental Setups

#### Model

FollowingWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\), we use[GRM\-Llama3\.2\-3B](https://huggingface.co/Ray2333/GRM-Llama3.2-3B-rewardmodel-ft)as a frozen backbone and trainK=20K=20linear reward heads as experts\. The router is implemented as a one\-hidden\-layer MLP with 128 hidden units\. We fix the coefficients ofℒls\\mathcal\{L\}\_\{\\mathrm\{ls\}\},ℒgb\\mathcal\{L\}\_\{\\mathrm\{gb\}\}, andℒdiv\\mathcal\{L\}\_\{\\mathrm\{div\}\}to0\.50\.5,1\.01\.0, and1\.01\.0, respectively\. We evaluate CoCo on two binary preference datasets:[700K](https://huggingface.co/datasets/hendrydong/preference_700K)\(Donget al\.,[2024](https://arxiv.org/html/2608.06400#bib.bib63)\)and[Reddit](https://huggingface.co/datasets/stanfordnlp/SHP)\(Ethayarajhet al\.,[2022](https://arxiv.org/html/2608.06400#bib.bib64)\)\. The complete sets of CoCo interpretations on both datasets are provided in Appendix[D](https://arxiv.org/html/2608.06400#A4)\.

#### Baselines

We compare CoCo with existing interpretable reward models and alternative interpretation signals for the MoE architecture\. For existing interpretable baselines, we include the sparse MoE reward model interpreted through routing weights \(SMoE;Wanget al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib78)\)\. We also include two SAE\-based methods: SARM\(Zhanget al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib76)\)and WIMHF\(Movvaet al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib57)\)\. SARM first trains a sequence\-level SAE encoder on unsupervised data and then fine\-tunes a reward model on the target preference dataset\. WIMHF instead learns contrastive features directly from the embedding contrasts of chosen–rejected response pairs\. Both methods interpret their features by prompting an LLM to summarize highly activating examples\.

To isolate the effect of the interpretation signal, we extract router\-based, score\-based, and CoCo interpretations from the same trained MoE model\. We also compare these interpretations on MoE variants trained with interpretability objectives targeted at routing weights, score contrasts, and contribution contrasts, respectively\. This allows us to distinguish the effect of post\-hoc interpretation from that of the corresponding interpretability objective\. Table[1](https://arxiv.org/html/2608.06400#S1.T1)summarizes the conceptual differences between CoCo and baseline interpretation methods\. Implementation details are provided in Appendix[F](https://arxiv.org/html/2608.06400#A6)\.

### 3\.2Evaluation Metrics

We evaluate each method along three dimensions, with scores averaged across experts:

#### Interpretation Quality

We directly assess the quality of the LLM\-generated interpretations\.Fidelitymeasures how well an interpretation generalizes to held\-out examples, computed as the Spearman correlation between interpretation signal values and an LLM’s judgments of whether each example follows the interpretation\. Higher correlation indicates stronger alignment between the expert signal and its interpretation\.Redundancyis the average pairwise cosine similarity between embeddings of expert interpretations\. Lower redundancy indicates more distinct preference patterns\.

#### Decision Faithfulness

Since interpretations are derived from top\-ranked examples, we evaluate whether these examples faithfully reflect the expert’s role in the model’s decisions\.Expert–Model Agreement\(EM Agree\.\) measures the proportion of the top\-ranked examples on which the target expert and the full MoE prefer the same response\.Expert Removal Flip Rate\(Removal Flip\) measures the proportion for which removing the target expert actually changes the MoE prediction, providing interventional evidence of the expert’s influence\. High values on both metrics indicate that the selected examples capture behavior that is aligned with and consequential to the MoE’s decisions\.

#### Expert Specialization

We likewise evaluate specialization on each expert’s top\-ranked examples\.Expert Accuracy\(Expert Acc\.\) measures the target expert’s preference accuracy on these examples\.Relative Expert Advantage\(Expert Adv\.\) is the target expert’s accuracy minus the mean accuracy of all other experts on the same set\. Together, these metrics capture both absolute competence and relative specialization of experts on their top\-ranked examples\.

To reduce sensitivity to the choice of selection threshold, we compute the decision faithfulness and expert specialization metrics atp∈\[0\.01,0\.025,0\.05,0\.075,0\.1\]p\\in\[0\.01,0\.025,0\.05,0\.075,0\.1\], whereppdenotes the proportion of top\-ranked examples, and report the normalized AUC\. We also report the reward model accuracy \(RM Acc\.\) of each reward model\.

### 3\.3Comparison with Existing Baselines

Table[2](https://arxiv.org/html/2608.06400#S2.T2)shows that CoCo achieves the strongest overall interpretability among existing interpretable reward models across both datasets, obtaining the highest fidelity, expert–model agreement, and expert accuracy while retaining strong reward modeling accuracy\. On Reddit, it also achieves the largest relative expert advantage\. While WIMHF yields lower redundancy and higher removal flip rates, it substantially underperforms on the remaining metrics\. SMoE attains a strong expert advantage on 700K, but trails CoCo in fidelity, decision faithfulness, and expert accuracy, and performs notably worse on Reddit\. Overall, CoCo yields the most coherent, faithful, and specialized expert interpretations while maintaining strong task performance\. A qualitative comparison in Appendix[E](https://arxiv.org/html/2608.06400#A5)further shows that different methods capture distinct aspects of the experts’ roles, reflecting the signals used for their interpretation\.

![Refer to caption](https://arxiv.org/html/2608.06400v1/x1.png)Figure 1:Human annotators rate CoCo highest in both pattern coherence and description quality\.#### Human Evaluation

FollowingWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\), we sample five experts/features per method on Reddit and ask five annotators to evaluate their top ten examples for pattern coherence and description quality on 1–5 scales\. Figure[1](https://arxiv.org/html/2608.06400#S3.F1)shows that CoCo produces the most coherent example sets for human annotators\. Moreover, because its interpretations capture both shared contexts and response\-level preference dimensions, they also receive the highest description quality ratings\.

### 3\.4Analysis of MoE Interpretation Signals

We compare CoCo with router\- and score\-based interpretation signals using their top\-ranked Reddit examples\. For a controlled comparison, we first apply all three signals to the same CoCo\-regularized MoE\. We then train separate MoE variants using signal\-specific regularization and evaluate each interpretation signal on its corresponding model\.

MethodDecision FaithfulnessExpert SpecializationEM Agree\.Removal FlipExpert Acc\.Expert Adv\.MoE with CoCo Interpretability RegularizationRouter\-based77\.868\.2964\.241\.21Score\-based85\.597\.5164\.926\.58CoCo93\.447\.9478\.936\.21MoE with Signal\-Specific Interpretability RegularizationRouter\-based57\.611\.2851\.15\-0\.39Score\-based62\.202\.5758\.135\.51CoCo93\.447\.9478\.936\.21

Table 3:Reddit results of interpretations derived from different signals\. Best scores are marked inbold\.Table[3](https://arxiv.org/html/2608.06400#S3.T3)shows that CoCo obtains the strongest overall decision faithfulness and expert specialization\. On the same MoE, it achieves the highest expert–model agreement and expert accuracy, while remaining competitive in removal flip and relative expert advantage\. Router\- and score\-based interpretations also remain substantially weaker when evaluated on MoE variants trained with their corresponding signal\-specific regularization\. Overall, contribution contrast emerges as the most effective signal for selecting informative example sets for MoE interpretation\.

## 4Conclusion

We introducedCoCo, which interprets MoE reward model experts through contribution contrast, jointly capturing routing and expert preference behavior\. Across automatic and human evaluations, CoCo produces more coherent, faithful, and specialized interpretations than router\-based, score\-based, and SAE\-based alternatives while maintaining strong reward modeling accuracy\. These findings support contribution contrast as a more informative basis for interpreting MoE reward models than routing weights or expert scores alone\.

## Limitations

Our experimental evaluation has several limitations that should be acknowledged\. First, the quality of CoCo interpretations depends on the structure learned by the underlying MoE reward model\. CoCo can expose how routing and expert scores jointly contribute to model decisions, but it cannot guarantee that the learned experts correspond to clean or semantically meaningful preference dimensions\. As a data\-centric approach, it may also fail to recover patterns that are absent, weakly represented, or insufficiently separable in the training data\. Its effectiveness therefore remains sensitive to the dataset, model design, and training configuration\. Second, interpretability in our experiments is evaluated through indirect proxies, including fidelity, redundancy, decision faithfulness, expert specialization, and human judgments\. Because the ground truth latent structure of preference data is unknown, these metrics cannot establish that an interpretation uniquely or completely characterizes an expert\. Our results should therefore be understood as evidence that CoCo produces more behaviorally coherent and decision\-aligned interpretations under the evaluated criteria, rather than as proof of ground truth disentanglement\.

## Ethical Considerations

CoCo is intended as an auditing tool for understanding reward model behavior, not as evidence that the learned expert patterns represent desirable or universally shared human values\. Since the method is data\-centric, its interpretations may reflect biases, harmful content, or demographic imbalances present in the underlying preference datasets\. In particular, Reddit\-derived examples may contain sensitive, offensive, or personally revealing content\. We therefore avoid interpreting learned patterns as normative judgments or inferring user identities and attributes from them\.

## Acknowledgements

This work was funded by the DFG project GRK 2853 "Neuroexplicit Models of Language, Vision, and Action" \(project number 471607914\)\.

## References

- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan, N\. Joseph, S\. Kadavath, J\. Kernion, T\. Conerly, S\. E\. Showk, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, T\. Hume, S\. Johnston, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, D\. Amodei, T\. B\. Brown, J\. Clark, S\. McCandlish, C\. Olah, B\. Mann, and J\. Kaplan \(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.CoRRabs/2204\.05862\.External Links:[Link](https://doi.org/10.48550/arXiv.2204.05862),[Document](https://dx.doi.org/10.48550/ARXIV.2204.05862),2204\.05862Cited by:[§1](https://arxiv.org/html/2608.06400#S1.p1.1)\.
- Whose boat does it float? improving personalization in preference tuning via inferred user personas\.Vienna, Austria,pp\. 3371–3393\.External Links:[Link](https://aclanthology.org/2025.acl-long.168/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.168),ISBN 979\-8\-89176\-251\-0Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- S\. Chakraborty, J\. Qiu, H\. Yuan, A\. Koppel, D\. Manocha, F\. Huang, A\. Bedi, and M\. Wang \(2024\)MaxMin\-RLHF: alignment with diverse human preferences\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 6116–6135\.External Links:[Link](https://proceedings.mlr.press/v235/chakraborty24b.html)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- D\. Chen, Y\. Chen, A\. Rege, Z\. Wang, and R\. K\. Vinayak \(2025\)PAL: sample\-efficient personalized reward modeling for pluralistic alignment\.External Links:[Link](https://openreview.net/forum?id=1kFDrYCuSu)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- K\. Chidambaram, K\. V\. Seetharaman, and V\. Syrgkanis \(2026\)Direct preference optimization with unobserved preference heterogeneity: the necessity of ternary preferences\.External Links:[Link](https://openreview.net/forum?id=7RFcY3ljMO)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- Y\. Choi, S\. Cho, M\. Lee, M\. Park, Y\. Ko, J\. Ok, and D\. Kim \(2025\)CoPL: collaborative preference learning for personalizing LLMs\.Suzhou, China,pp\. 12875–12893\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.650/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.650),ISBN 979\-8\-89176\-332\-6Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- H\. Dong, W\. Xiong, B\. Pang, H\. Wang, H\. Zhao, Y\. Zhou, N\. Jiang, D\. Sahoo, C\. Xiong, and T\. Zhang \(2024\)RLHF workflow: from reward modeling to online RLHF\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=a13aYUU9eU)Cited by:[§3\.1](https://arxiv.org/html/2608.06400#S3.SS1.SSS0.Px1.p1.7)\.
- K\. Ethayarajh, Y\. Choi, and S\. Swayamdipta \(2022\)Understanding dataset difficulty with𝒱\\mathcal\{V\}\-usable information\.pp\. 5988–6008\.Cited by:[§3\.1](https://arxiv.org/html/2608.06400#S3.SS1.SSS0.Px1.p1.7)\.
- T\. Kaufmann, P\. Weng, V\. Bengs, and E\. Hüllermeier \(2025\)A survey of reinforcement learning from human feedback\.Transactions on Machine Learning Research\.Note:Survey CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=f7OkIurx4b)Cited by:[§1](https://arxiv.org/html/2608.06400#S1.p1.1)\.
- Y\. Lee, J\. Williams, H\. Marklund, A\. Sharma, E\. Mitchell, A\. Singh, and C\. Finn \(2024\)Test\-time alignment via hypothesis reweighting\.CoRRabs/2412\.08812\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.08812),[Document](https://dx.doi.org/10.48550/ARXIV.2412.08812),2412\.08812Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- D\. Liu, J\. Li, Z\. Fu, Y\. Tu, J\. Li, Z\. Mao, and Y\. Zhang \(2026\)SparseRM: a lightweight preference modeling with sparse autoencoder\.Proceedings of the AAAI Conference on Artificial Intelligence40\(38\),pp\. 32051–32059\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40476),[Document](https://dx.doi.org/10.1609/aaai.v40i38.40476)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Liu, Y\. Zhu, S\. Wang, X\. Wei, E\. Min, Y\. Lu, S\. Wang, D\. Yin, and Z\. Dou \(2025\)LLMs \+ persona\-plug = personalized LLMs\.Vienna, Austria,pp\. 9373–9385\.External Links:[Link](https://aclanthology.org/2025.acl-long.461/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.461),ISBN 979\-8\-89176\-251\-0Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- F\. Luo, R\. Yang, H\. Sun, C\. Deng, J\. Yao, J\. Shen, H\. Zhang, and H\. Chen \(2025\)Rethinking diverse human preference learning through principal component analysis\.Vienna, Austria,pp\. 19857–19870\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1019/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1019),ISBN 979\-8\-89176\-256\-5Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.06400#S1.p1.1)\.
- R\. Movva, S\. Milli, S\. Min, and E\. Pierson \(2026\)What’s in my human feedback? learning interpretable descriptions of preference data\.External Links:[Link](https://openreview.net/forum?id=sC6A1bFDUt)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px2.p1.1),[Appendix F](https://arxiv.org/html/2608.06400#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.06400#S1.p1.1),[§2](https://arxiv.org/html/2608.06400#S2.SS0.SSS0.Px3.p1.2),[§3\.1](https://arxiv.org/html/2608.06400#S3.SS1.SSS0.Px2.p1.1)\.
- S\. Poddar, Y\. Wan, H\. Ivison, A\. Gupta, and N\. Jaques \(2024\)Personalizing reinforcement learning from human feedback with variational preference learning\.pp\. 52516–52544\.External Links:[Document](https://dx.doi.org/10.52202/079017-1664),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/5e1c255653eb98cef13f45b2d337c882-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- S\. Quan \(2024\)DMoERM: recipes of mixture\-of\-experts for effective reward modeling\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7006–7028\.External Links:[Link](https://aclanthology.org/2024.findings-acl.418/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.418)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- J\. Shen, J\. Yao, R\. Yang, Y\. Sun, F\. Luo, R\. Pan, T\. Zhang, and H\. Zhao \(2025\)MiCRo: mixture modeling and context\-aware routing for personalized preference learning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 17447–17463\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.882/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.882),ISBN 979\-8\-89176\-332\-6Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1),[Appendix B](https://arxiv.org/html/2608.06400#A2.p1.1),[§2](https://arxiv.org/html/2608.06400#S2.SS0.SSS0.Px1.p1.6)\.
- H\. Wang, Y\. Lin, W\. Xiong, R\. Yang, S\. Diao, S\. Qiu, H\. Zhao, and T\. Zhang \(2024a\)Arithmetic control of LLMs for diverse user preferences: directional preference alignment with multi\-objective rewards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 8642–8655\.External Links:[Link](https://aclanthology.org/2024.acl-long.468/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.468)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- H\. Wang, W\. Xiong, T\. Xie, H\. Zhao, and T\. Zhang \(2024b\)Interpretable preferences via multi\-objective reward modeling and mixture\-of\-experts\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10582–10592\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.620/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.620)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.06400#S2.SS0.SSS0.Px1.p1.6)\.
- Y\. Wang, J\. Mu, M\. Jobanputra, Y\. Wang, J\. Lee, S\. Oh, I\. Valera, and V\. Demberg \(2026\)Sparse mixture\-of\-experts reward models learn interpretable and specialized experts for personalized preference modeling\.arXiv preprint arXiv:2606\.04284\.Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px2.p2.1),[Appendix B](https://arxiv.org/html/2608.06400#A2.p1.1),[Appendix B](https://arxiv.org/html/2608.06400#A2.p2.3),[Appendix F](https://arxiv.org/html/2608.06400#A6.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2608.06400#A6.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.06400#S1.p2.1),[§2](https://arxiv.org/html/2608.06400#S2.SS0.SSS0.Px2.p1.5),[§3\.1](https://arxiv.org/html/2608.06400#S3.SS1.SSS0.Px1.p1.7),[§3\.1](https://arxiv.org/html/2608.06400#S3.SS1.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2608.06400#S3.SS3.SSS0.Px1.p1.1),[footnote 2](https://arxiv.org/html/2608.06400#footnote2)\.
- X\. Wu \(2025\)A comprehensive survey on learning from rewards for large language models: reward models and learning strategies\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 17847–17875\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.970/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.970),ISBN 979\-8\-89176\-335\-7Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- X\. Wu and Y\. Lu \(2026\)Reward model routing in alignment\.External Links:[Link](https://openreview.net/forum?id=i3OKIHSsHC)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- R\. Yu, S\. Wan, Y\. Wang, C\. Gao, L\. Gan, Z\. Zhang, and D\. Zhan \(2025\)Reward models in deep reinforcement learning: a survey\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence, IJCAI\-25,J\. Kwok \(Ed\.\),pp\. 10807–10816\.Note:Survey TrackExternal Links:[Document](https://dx.doi.org/10.24963/ijcai.2025/1199),[Link](https://doi.org/10.24963/ijcai.2025/1199)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- M\. J\. Zhang, Z\. Wang, J\. D\. Hwang, Y\. Dong, O\. Delalleau, Y\. Choi, E\. Choi, X\. Ren, and V\. Pyatkin \(2025\)Diverging preferences: when do annotators disagree and do models know?\.External Links:[Link](https://openreview.net/forum?id=qWgAAVhoXb)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- S\. Zhang, W\. Shi, S\. Li, J\. Liao, T\. Liang, H\. Cai, and X\. Wang \(2026\)Interpretable reward model via sparse autoencoder\.Proceedings of the AAAI Conference on Artificial Intelligence40\(41\),pp\. 34808–34816\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40783),[Document](https://dx.doi.org/10.1609/aaai.v40i41.40783)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.06400#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.06400#S3.SS1.SSS0.Px2.p1.1)\.
- S\. Zhao, J\. Dang, and A\. Grover \(2024\)Group preference optimization: few\-shot alignment of large language models\.External Links:[Link](https://openreview.net/forum?id=DpFeMH4l8Q)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhou, J\. Liu, J\. Shao, X\. Yue, C\. Yang, W\. Ouyang, and Y\. Qiao \(2024\)Beyond one\-preference\-fits\-all alignment: multi\-objective direct preference optimization\.Bangkok, Thailand,pp\. 10586–10613\.External Links:[Link](https://aclanthology.org/2024.findings-acl.630/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.630)Cited by:[Appendix A](https://arxiv.org/html/2608.06400#A1.SS0.SSS0.Px1.p1.1)\.

## Appendix ARelated Work

#### Data\-Centric Preference Decomposition

Most reward models learn a single global preference function and therefore struggle to capture heterogeneous human values\(Yuet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib7); Wu,[2025](https://arxiv.org/html/2608.06400#bib.bib6); Chakrabortyet al\.,[2024](https://arxiv.org/html/2608.06400#bib.bib9); Zhanget al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib58)\)\. Prior work addresses this by conditioning reward models on annotated attributes, user identities, or predefined groups\(Wanget al\.,[2024a](https://arxiv.org/html/2608.06400#bib.bib16); Quan,[2024](https://arxiv.org/html/2608.06400#bib.bib13); Wanget al\.,[2024b](https://arxiv.org/html/2608.06400#bib.bib12); Zhouet al\.,[2024](https://arxiv.org/html/2608.06400#bib.bib19); Wu and Lu,[2026](https://arxiv.org/html/2608.06400#bib.bib66); Zhaoet al\.,[2024](https://arxiv.org/html/2608.06400#bib.bib31); Choiet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib29); Liuet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib30); Balepuret al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib33); Chenet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib40)\), but such supervision is costly and restricts personalization to known preference dimensions\. More recent methods instead infer latent preference structure directly from standard binary chosen–rejected data using latent variables, prototypes, representation decomposition, or MoE models\(Poddaret al\.,[2024](https://arxiv.org/html/2608.06400#bib.bib41); Chidambaramet al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib42); Leeet al\.,[2024](https://arxiv.org/html/2608.06400#bib.bib28); Luoet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib27); Shenet al\.,[2025](https://arxiv.org/html/2608.06400#bib.bib11)\)\. However, their learned components are not always semantically interpretable, limiting transparent personalization\.

#### Interpretable Reward Models

To address the limitations above, recent work has explored aligning reward models with human\-interpretable preference patterns\. One line of work uses sparse autoencoders \(SAEs\) to decompose reward model representations into sparse latent features, which can then be interpreted through highly activating examples or used for preference modeling\(Movvaet al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib57); Liuet al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib75); Zhanget al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib76)\)\. While these methods provide fine\-grained representation\-level explanations, they are typically post\-hoc: sparse features are extracted from a separately trained SAE encoder and are not necessarily optimized to align with the target preference distribution\. This two\-stage pipeline also introduces additional training and analysis costs\. Moreover, SAE\-based reward models often use a fixed set of feature weights across contexts, which limits their flexibility in modeling context\-dependent preference patterns\.

A more recent direction builds interpretability directly into the reward model through sparse Mixture\-of\-Experts \(SMoE\) architectures\(Wanget al\.,[2026](https://arxiv.org/html/2608.06400#bib.bib78)\)\. SMoE reward models learn specialized experts from standard binary preference data with sparsity and diversity constraints and have been shown to improve both interpretability and personalization performance\. However,Wanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)characterize SMoE experts through examples with high routing weights, so the resulting interpretations capture prompt\-level topic or task clusters rather than response\-level preference patterns\. Moreover, they provide only a partial account of expert behavior, describing which examples an expertreceivesbut not how itjudgesthem\. In contrast, CoCo interpretations capture the interaction between routing patterns and expert behavior, providing a more faithful and informative account of response\-level preference dimensions\.

## Appendix BCoCo\-Based MoE Reward Model Training

FollowingShenet al\.\([2025](https://arxiv.org/html/2608.06400#bib.bib11)\)andWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\), we train the model by marginalizing the expert\-level Bradley–Terry probabilities:

ℒRM=−1\|𝒟\|\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{RM\}\}=\-\\frac\{1\}\{\|\\mathcal\{D\}\|\}∑\(x,yw,yl\)∈𝒟log∑k=1K\(πϕ,k\(x\)⋅\\displaystyle\\sum\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\log\\sum\_\{k=1\}^\{K\}\\big\(\\pi\_\{\\phi,k\}\(x\)\\cdot\(3\)σ\(rθk\(x,yw\)−rθk\(x,yl\)\)\)\.\\displaystyle\\sigma\\\!\(r\_\{\\theta\_\{k\}\}\(x,y\_\{w\}\)\-r\_\{\\theta\_\{k\}\}\(x,y\_\{l\}\)\)\\big\)\.
We adapt the interpretability regularization terms fromWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)to encourage sparse, disentangled, and diversecontribution patternsin each expert\. Specifically, we have defined the𝐜​\(x,yw,yl\)=\[c1,…,cK\]\\mathbf\{c\}\(x,y\_\{w\},y\_\{l\}\)=\[c\_\{1\},\\ldots,c\_\{K\}\]to denote the absolute contribution contrasts of all experts\. When applying the local sparsity and global balance regularizations, we convert𝐜​\(x,yw,yl\)\\mathbf\{c\}\(x,y\_\{w\},y\_\{l\}\)to a distribution𝐜~​\(x,yw,yl\)\\tilde\{\\mathbf\{c\}\}\(x,y\_\{w\},y\_\{l\}\):

c~k​\(x,yw,yl\)=ck​\(x,yw,yl\)∑j=1Kcj​\(x,yw,yl\)\+ϵ\\tilde\{c\}\_\{k\}\(x,y\_\{w\},y\_\{l\}\)=\\frac\{c\_\{k\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\sum\_\{j=1\}^\{K\}c\_\{j\}\(x,y\_\{w\},y\_\{l\}\)\+\\epsilon\}\(4\)and𝐜~​\(x,yw,yl\)=\[c~1,…,c~K\]\\tilde\{\\mathbf\{c\}\}\(x,y\_\{w\},y\_\{l\}\)=\[\\tilde\{c\}\_\{1\},\\ldots,\\tilde\{c\}\_\{K\}\]\. Then the CoCo\-adapted interpretability regularization terms are defined based on𝐜\\mathbf\{c\}and𝐜~\\tilde\{\\mathbf\{c\}\}\.

\(1\) Local Sparsity: We encourage sparse contribution contrast𝐜​\(x,yw,yl\)\\mathbf\{c\}\(x,y\_\{w\},y\_\{l\}\)by minimizing the entropy of its normalized distribution𝐜~​\(x,yw,yl\)\\tilde\{\\mathbf\{c\}\}\(x,y\_\{w\},y\_\{l\}\):

ℒls=𝔼\(x,yw,yl\)∼𝒟\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{ls\}\}=\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\[H​\(𝐜~​\(x,yw,yl\)\)log⁡K\],\\displaystyle\\left\[\\frac\{H\\\!\\big\(\\tilde\{\\mathbf\{c\}\}\(x,y\_\{w\},y\_\{l\}\)\\big\)\}\{\\log K\}\\right\],\(5\)H​\(𝐜~\)=−\\displaystyle H\(\\tilde\{\\mathbf\{c\}\}\)=\-∑k=1Kc~k​log⁡c~k\.\\displaystyle\\sum\_\{k=1\}^\{K\}\\tilde\{c\}\_\{k\}\\log\\tilde\{c\}\_\{k\}\.
\(2\) Global Balance: We encourage balanced utilization of experts to prevent model collapse\. Specifically, we maximize the entropy of the average contribution contrast𝐜~​\(x,yw,yl\)\\tilde\{\\mathbf\{c\}\}\(x,y\_\{w\},y\_\{l\}\)within a batchℬ\\mathcal\{B\}:

ℒgb=𝔼ℬ∼𝒟​\[−H​\(𝔼\(x,yw,wl\)∼ℬ​\[𝐜~​\(x,yw,yl\)\]\)log⁡K\]\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{gb\}\}=\\mathbb\{E\}\_\{\\mathcal\{B\}\\sim\\mathcal\{D\}\}\\left\[\-\\frac\{H\(\\mathbb\{E\}\_\{\(x,y\_\{w\},w\_\{l\}\)\\sim\\mathcal\{B\}\}\[\\tilde\{\\mathbf\{c\}\}\(x,y\_\{w\},y\_\{l\}\)\]\)\}\{\\log K\}\\right\]\.\(6\)\(3\) Expert Diversity: We also encourage experts to contribute diversely to the final reward by minimizing the average pairwise Pearson correlations between theirckc\_\{k\}:

ℒdiv=2K​\(K−1\)​∑i<jcorr​\(ci,cj\)2,\\mathcal\{L\}\_\{\\mathrm\{div\}\}=\\frac\{2\}\{K\(K\-1\)\}\\sum\_\{i<j\}\\mathrm\{corr\}\(c\_\{i\},c\_\{j\}\)^\{2\},\(7\)where the Pearson correlations are computed over a batchℬ\\mathcal\{B\}of preference pairs\.

The final training objective is defined as:

ℒ=ℒRM\+λls​ℒls\+λgb​ℒgb\+λdiv​ℒdiv,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{RM\}\}\+\\lambda\_\{\\mathrm\{ls\}\}\\mathcal\{L\}\_\{\\mathrm\{ls\}\}\+\\lambda\_\{\\mathrm\{gb\}\}\\mathcal\{L\}\_\{\\mathrm\{gb\}\}\+\\lambda\_\{\\mathrm\{div\}\}\\mathcal\{L\}\_\{\\mathrm\{div\}\},\(8\)whereλls\\lambda\_\{\\mathrm\{ls\}\},λgb\\lambda\_\{\\mathrm\{gb\}\}, andλdiv\\lambda\_\{\\mathrm\{div\}\}control the strengths of local sparsity, global balance, and expert diversity regularization, respectively\.

## Appendix CAblating CoCo\-Based Interpretation Regularization

We apply the CoCo interpretation procedure to MoE models where the interpretability objectives are applied to the routing weights and score contrasts\. Results in Table[4](https://arxiv.org/html/2608.06400#A3.T4)show that CoCo\-based regularization yields more disentangled and diverse contribution patterns, resulting in more faithful and specialized CoCo interpretations\.

MethodDecision FaithfulnessExpert SpecializationEM Agree\.Removal FlipExpert Acc\.Expert Adv\.Router\-based64\.490\.1659\.914\.43Score\-based61\.710\.1258\.184\.80CoCo93\.447\.9478\.936\.21

Table 4:Interpretability of CoCo interpretations on MoE models trained with different interpretability objectives on Reddit\.
## Appendix DComplete CoCo Interpretations

Tables[5](https://arxiv.org/html/2608.06400#A4.T5)and[6](https://arxiv.org/html/2608.06400#A4.T6)present the complete CoCo interpretations for all 20 experts on 700K and Reddit, respectively\. CoCo can capture both shared input contexts and response\-level preference dimensions, therefore yields diverse and informative expert interpretations\.

ExpertInterpretation0Math problems involving Python answered with brief steps and a minimal calculation\.1Instruction\-heavy prompts answered with structured, step\-by\-step lists\.2Step\-by\-step reasoning rather than brief answers\.3Task\-specific, step\-by\-step reasoning rather than generic exposition\.4Thorough, domain\-specific answers with concrete details rather than brief or dismissive replies\.5Detailed, structured explanations rather than refusals or deflections\.6Competitive\-programming tasks answered with a numbered plan followed by Python code\.7Lengthy, conversational elaborations rather than concise, direct answers\.8Tailored, concrete solutions with examples rather than vague or generic replies\.9Detailed instructions followed with specific, task\-complete outputs rather than vague summaries\.10Generic, textbook\-style exposition rather than direct answers to the prompt\.11Structured, detailed, and example\-backed explanations\.12Exact adherence to output\-format constraints\.13Math word problems answered with step\-by\-step calculations and a boxed final answer\.14Informational prompts answered with concrete, example\-backed explanations rather than vague meta\-responses or deflection\.15Exact adherence to output\-format constraints\.16Broad, tangential exposition rather than direct answers or instruction following\.17Correction of false premises rather than accepting them\.18Concrete, step\-by\-step guidance rather than vague generalities\.19Exact adherence to output\-format constraints\.Table 5:CoCo interpretations of the 20 experts on 700K\.ExpertInterpretation0Science\-fiction and comic\-book lore questions answered with specific in\-universe explanations\.1Concise, specific recommendations rather than lengthy lists or explanations\.2Cooking questions answered with references to specific authoritative sources and links\.3Practical, anecdote\-backed explanations rather than abstract speculation\.4Energy and climate AMAs answered with skeptical questions about feasibility and trade\-offs\.5Detailed, experience\-based advice with concrete steps rather than brief replies\.6Fiction and fandom questions answered with witty, in\-universe explanations\.7Personal, experience\-based advice with specific reasoning rather than one\-line suggestions\.8Blunt, sardonic humor rather than polite, measured replies\.9Specific, concrete answers rather than vague generalities\.10Concise, authoritative answers with actionable advice rather than personal anecdotes\.11Witty, humorous replies rather than detailed step\-by\-step advice\.12Practical advice questions answered with specific, experience\-based details rather than generic replies\.13Concrete, actionable advice with specific examples or mechanisms rather than generic opinions\.14Legal advice questions answered with statute citations and actionable steps\.15Legal or HR dispute questions answered with assertive, actionable escalation advice\.16Advice questions answered with blunt, decisive guidance rather than hedged or meandering replies\.17Procedural and technical explanations that correct misconceptions rather than provide vague reassurance\.18Concise, opinionated tips rather than recipe links or lengthy lists\.19First\-person, authoritative explanations with specific details rather than terse, impersonal replies\.Table 6:CoCo interpretations of the 20 experts on Reddit\.
## Appendix EQualitative Analysis

Table[7](https://arxiv.org/html/2608.06400#A5.T7)presents representative interpretations produced by different methods\. Their content and style vary substantially, reflecting the signals used for interpretation\. CoCo is the only contrastive method that jointly considers prompt context and expert preference behavior, and thus captures both domains of specialization and response\-level preference dimensions\. In contrast, router\-based interpretation and SMoE consider only highly ranked prompts, producing descriptions of the contexts assigned to each expert\. Both SAE\-based methods recover response\-level features, but SARM operates on individual responses, whereas WIMHF summarizes chosen–rejected pairs\. Consequently, WIMHF yields contrastive descriptions while SARM generally does not\. Score\-based interpretation also captures response\-level expert behavior, but ignores routing weights\. Its top\-ranked examples are therefore often repetitive, leading to highly redundant interpretations\.

MethodRepresentative InterpretationsCoCoScience\-fiction and comic\-book lore questions answered with specific in\-universe explanations\.Legal advice questions answered with specific statute citations and actionable steps\.Cooking questions answered with references to authoritative sources and links\.First\-person, authoritative explanations with specific details rather than terse, impersonal replies\.Blunt, sardonic humor rather than polite, measured replies\.Router\-basedAsks for in\-universe lore or continuity analysis of a popular fictional franchise\.Seeks baking advice or ideas, including recipes, techniques, ingredient pairings, or uses for leftovers\.Solicits community recommendations, tips, or personal stories\.Long, first\-person advice\-seeking posts describing complex real\-life situations with extensive background details\.Seeks practical next\-step advice for a real\-world problem, often legal or medical\.Score\-basedDetailed, example\-driven explanations rather than brief assertions\.Concrete, experience\-backed specifics rather than generic commentary\.First\-person anecdotes and practical, experience\-based tips rather than terse or theoretical replies\.Advice questions answered with concrete, detailed, and actionable guidance\.Terse, witty one\-liners rather than detailed explanations\.SMoEAsks a specific question seeking factual information, advice, or an opinion\.Asks for legal advice or information about rights and laws\.Asks a question or seeks advice in a Reddit\-post format\.Asks for troubleshooting or technical explanations of cooking or baking issues\.Asks for advice or information about a specific profession or career\.SARMProvides a direct, definitive answer to a factual or technical question\.Provides a brief, direct, or conversational response\.Provides an incorrect or factually hallucinated answer\.Provides a step\-by\-step solution to a mathematics problem\.Provides a brief, direct, or conversational response\.WIMHFOffers advice and recommendations rather than asking questions, sharing anecdotes, or linking sources\.Gives a terse, one\-line reply without explanation or reasoning\.Provides serious, information\-oriented responses without jokes, insults, or generic encouragement\.Includes specific recommendations or named examples\.Provides detailed, multi\-point advice with specific examples or resources\.Table 7:Representative interpretations produced by CoCo, alternative MoE interpretation signals, and existing interpretable reward model baselines on Reddit\.
## Appendix FExperimental Details

#### Datasets and Training

We adopt the same data pre\-processing pipeline and training configuration fromWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)for a controlled comparison\.

#### LLM Usage for Interpretation Extraction

FollowingMovvaet al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib57)\), we usegpt\-5\-lowto summarize top\-ranked examples into expert interpretations andgpt\-5\-mini\-lowas the judge for computing fidelity\. To measure redundancy, we encode all natural language interpretations using[Qwen3\-Embedding\-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B)and compute their pairwise cosine similarities\.

#### Baselines

We train SARM and SMoE on our data splits following their official implementations\. WIMHF is trained on its own pre\-defined splits of the Reddit and 700K datasets and learns 32 features, compared with our 20 experts\. We therefore evaluate WIMHF on its corresponding test splits and report interpretability results averaged over all 32 features\.

#### LLM Instructions

Figures[2](https://arxiv.org/html/2608.06400#A6.F2)and[3](https://arxiv.org/html/2608.06400#A6.F3)present the prompts used to generate LLM\-based interpretations from top\-ranked examples and to judge whether held\-out examples match the resulting interpretations\.

Prompt: Describing experts from top\-ranked examplesTask OverviewYou are interpreting one component of a mixture\-of\-experts reward model\.The model contains a router and multiple reward experts\. The router depends only on the user prompt, whereas each expert score depends on the full prompt\-\-response sequence\. An expert’s contribution is its router weight multiplied by its score\. High contribution may reflect a prompt\-level routing pattern, a response\-level scoring pattern, or both\.The examples below were selected because this component made a large contribution to their final rewards\. Identify the single most coherent pattern associated with high contribution\.InstructionsInspect the prompts and responses and identify the most specific pattern consistently supported across the examples\.The pattern may describe:•a prompt\-level pattern, such as topic, domain, task type, or user intent;•a response\-level property, such as style, reasoning, specificity, refusal behavior, or instruction following;•a coherent combination of prompt type and response property\.Prefer one of the following forms:•‘‘<prompt or task type\>’’•‘‘<response property\> rather than <contrast\>’’•‘‘<prompt or task type\> answered with <response property\>’’Examples:•‘‘science\-fiction questions’’•‘‘gives concise recommendations rather than extended explanations’’•‘‘science\-fiction questions answered with detailed in\-universe reasoning’’•‘‘cooking questions answered with mechanistic explanations of ingredients’’•‘‘responds directly to sensitive requests rather than refusing’’•‘‘follows exact output\-format constraints’’Use only properties recurring across multiple distinct examples\. If only the prompt topic is coherent, describe the routing pattern\. If response behavior is coherent across varied prompts, describe the response\-level property\. Do not combine unrelated clusters\.Additional Rules•Be objective, concrete, and concise\.•Identify one pattern, not a list of possible properties\.•Do not treat duplicated prompts as independent evidence\.•Do not assume high contribution implies higher quality or correctness\.•Avoid vague labels unless made behaviorally specific\.•Avoid ‘‘either,’’ long lists, and unrelated clauses\.•Do not mention the expert, router, score, contribution, activation, or examples in the final answer\.Examples\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
\{examples\}
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-OutputOutput exactly one concise description, beginning with ‘‘\-’’ and enclosed in quotation marks\.Your response is:\-"Figure 2:Instruction used to summarize the common feature of top\-ranked examples\.Prompt: Judging whether an example matches expert descriptionYou are evaluating a response\-level interpretation of a reward\-model expert\.Interpretation\{description\}Both responses answer the same user prompt\. Decide which response expresses the interpretation more strongly, taking the prompt into account\.Prompt\{prompt\}Response A\{response\_a\}Response B\{response\_b\}InstructionsOutput exactly one of the following labels:•A: Response A expresses the interpretation more strongly\.•B: Response B expresses the interpretation more strongly\.•NOT\_RELEVANT: The interpretation does not distinguish the responses, both express it equally, neither expresses it, or the evidence is unclear\.Consider general response quality only when it is part of the interpretation\. Do not output any explanation or additional text\.Answer:Figure 3:Instruction used to judge which response in a pair more strongly matches a given natural language expert interpretation\.
#### Human Study

We adopt the same human evaluation pipeline as inWanget al\.\([2026](https://arxiv.org/html/2608.06400#bib.bib78)\)\. Annotators were compensated at an hourly rate comparable to the local average hourly salary for researchers and informed about how their data would be used\.