LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

# Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Source: [https://arxiv.org/html/2607.29559](https://arxiv.org/html/2607.29559)
Manith Adikari1,4, Bei Peng2, Samuele Vinanzi3, Angelo Cangelosi1,4

###### Abstract

Reinforcement Learning \(RL\) systems are typically trained using a single, well\-specified scalar reward function\. However, real\-world decision\-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground\-truth reward functions are difficult to specify or inaccessible\. While Multi\-Objective RL \(MORL\) addresses such trade\-offs by modeling rewards as vectors, existing approaches typically assume access to a well\-specified reward function for each objective, inheriting the same challenges faced by single\-objective RL\. Meanwhile, Preference\-based RL \(PbRL\) has shown great potential in solving complex tasks without access to a pre\-defined reward function through reward learning from human feedback, yet has largely been studied in single\-objective settings\. In this work, we bridge this gap with LEMUR: Learning to Align with Multi\-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi\-objective policies\. Our approach jointly learns policies and multiple objective\-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning\. We evaluate LEMUR on a variety of benchmark multi\-objective tasks, and empirical results demonstrate its superior performance over baseline methods\. Our method presents a promising direction for solving multi\-objective decision\-making tasks without pre\-defined reward functions\.

## 1Introduction

Reinforcement Learning \(RL\) has achieved remarkable success in training autonomous agents, from game\-playing\(Mnihet al\.[2013](https://arxiv.org/html/2607.29559#bib.bib383)\)to robotics\(Tanet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib523)\)\. By formalizing learning as the maximization of cumulative rewards through trial and error learning\(Sutton and Barto[1998](https://arxiv.org/html/2607.29559#bib.bib101)\), RL provides a natural framework for training autonomous ‘goal\-seeking’ agents\(McCarthy[1997](https://arxiv.org/html/2607.29559#bib.bib385)\)\. However, standard RL relies on two critical assumptions: that the goal can be represented by a singlescalar reward, and that this reward function iswell\-specified\. In practice, these assumptions rarely hold in complex, real\-world domains\.

Real\-world tasks often involvemultiple, competingobjectives\(Dulac\-Arnoldet al\.[2019](https://arxiv.org/html/2607.29559#bib.bib554)\), such as balancing speed versus safety in autonomous driving\(Wanget al\.[2026](https://arxiv.org/html/2607.29559#bib.bib575)\), or throughput versus energy efficiency in robotics\(Huanget al\.[2022](https://arxiv.org/html/2607.29559#bib.bib573); Kouritemet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib572)\)\.

The field of Multi\-Objective Reinforcement Learning \(MORL\) addresses this by modelingrewards as vectorsto find a set of Pareto\-optimal policies\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525)\): policies where no objective’s expected returns can be increased without decreasing the returns of other objectives\. However, existing MORL approaches typically assume that the ground\-truth reward function for each objective is accessible and manually specified\(Hayeset al\.[2022b](https://arxiv.org/html/2607.29559#bib.bib139)\)\. Thus, they inherit the same challenges of reward specification from the single\-objective RL domain, now across multiple objectives\. Manually designing a reward function that can achieve an adequate balance between competing objectives is challenging and can lead to oversimplification, which results in suboptimal policies\(Knoxet al\.[2012](https://arxiv.org/html/2607.29559#bib.bib545)\)and possible reward exploitation\(Amodeiet al\.[2016](https://arxiv.org/html/2607.29559#bib.bib526)\)\. To avoid complicated reward engineering, Preference\-based RL \(PbRL\) learns reward models directly from human feedback\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520)\), which leads to better alignment between the system’s behavior and human preferences\. Despite progress in reward learning for single\-objective RL, this problem remains largely unexplored in the MORL setting\. While some works consider reward learning in MORL, they are typically limited to Large Language Model \(LLM\) post\-training settings or narrow natural language\-based tasks\(Bakkeret al\.[2022](https://arxiv.org/html/2607.29559#bib.bib498); Rameet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib30); Yanget al\.[2024](https://arxiv.org/html/2607.29559#bib.bib532)\), and do not study the joint learning of policies and rewards in environments with multiple, competing objectives\.

Reward specification is already a major bottleneck in scaling single\-objective RL settings, and becomes even more critical in MORL\. Two key challenges arise\. First, the agent must optimize multiple objectives whose underlying reward functions are complex, implicit, or inaccessible\. Second, even when human feedback is available, collapsing multi\-criteria reward signals into a single scalar obscures the very trade\-off structure that MORL is designed to optimize\(Vamplewet al\.[2011](https://arxiv.org/html/2607.29559#bib.bib576); Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525); Sorensenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib32)\)\. Consider training a robotic system:non\-expertoperators can reliably judge task\-level success, such as whether objects were grasped or placed correctly, while expert operators are required to assess fine\-grained criteria such as grasp stability, long\-term wear, or safety\-related objectives\. Collapsing suchheterogeneousfeedback into a single reward risks conflatingdistinctobjectives and diluting expert signals, motivating the need forseparate, objective\-specific reward models learned from the appropriate source of feedback\.

This creates a crucial gap:How can agents learn optimal trade\-offs between conflicting objectives when the reward functions are unknown and must be inferred from feedback?

To address this gap, we introduceLEMUR:Learning to Align withMulti\-ObjectiveReinforcement Learning from Preference Feedback, a framework for learning to balance multiple objectives without pre\-defined reward functions\. Our approach jointly learns policies and multiple objective\-specific reward models from preference feedback, enabling agents to effectively balance competing objectives during learning\. By removing the assumption of known reward functions and explicitly modeling multiple objectives, our method tackles a key challenge to scaling MORL in real\-world, human\-aligned domains\. To summarize, our main contributions are:

- •We propose a novel framework, LEMUR, which learns*multiple*objective\-specific reward models from preference feedback and then optimizes policies against these reward models using multi\-objective reinforcement learning\. This enables agents to effectively solve multi\-objective decision\-making tasks without access to pre\-defined reward functions, while naturally accommodating heterogeneous sources of feedback, where different annotators may hold expertise over different objectives\.
- •Our extensive experiments demonstrate that LEMUR outperforms baselines across a range of benchmark multi\-objective environments, namely high\-dimensional continuous control tasks, and further show LEMUR’s robustness to label noise, constrained feedback budgets, and scaling to more objectives\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/Figures/lemur.png)Figure 1:Illustration of our framework LEMUR: \(1\) Unsupervised Pre\-training for the MORL agent to explore and collect diverse experiences via maximizing state entropy H\(s\)\. \(2\) Reward learning from Preference feedback, where each reward model is learned separately from the preferences queried from each teacher\. The reward models are used to dynamically relabel the state\-action pairs as a reward vector for each objective \(i\.e\., each teacher’s preferences\)\. \(3\) Multi\-Objective RL agent denoted byπϕ\\pi\_\{\\phi\}uses each of the trained reward models to do multi\-objective policy optimization to maximize the expected vector rewards\.
## 2Preliminaries

Multi\-Objective Reinforcement Learning\.We formulate the problem as a Multi\-Objective Markov Decision Process \(MOMDP\)\(White[1982](https://arxiv.org/html/2607.29559#bib.bib534)\), defined by the tuple⟨𝒮,𝒜,𝒯,γ,𝐫⟩\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\gamma,\\mathbf\{r\}\\rangle\. Here,𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote state and action spaces respectively,𝒯\\mathcal\{T\}the transition dynamics, andγ∈\[0,1\)\\gamma\\in\[0,1\)the discount factor\. Unlike standard RL, the reward is a vector𝐫​\(s,a\)∈ℝm\\mathbf\{r\}\(s,a\)\\in\\mathbb\{R\}^\{m\}comprisingmmdistinct objectives\. The agent’s goal is to maximize the expected discounted vector returnsJ​\(π\)=𝔼π​\[∑t=0∞γt​𝐫​\(𝐬t,𝐚t\)\]J\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathbf\{r\}\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)\]\. A policyπ\\pimaps states to action distributions\. Since no single policy typically maximizes all objectives in MORL, the agent learns a set of policiesΠ\\Pirepresenting optimal trade\-offs, defined by Pareto dominance\(Hayeset al\.[2022a](https://arxiv.org/html/2607.29559#bib.bib524)\)\. A policyπ\\pidominatesπ′\\pi^\{\\prime\}\(denotedJ​\(π\)≻J​\(π′\)J\(\\pi\)\\succ J\(\\pi^\{\\prime\}\)\) if it is superior in at least one objective and no worse in others\. The solution set is the Pareto Frontierℱ=\{π∈Π∣∄​π′∈Π:J​\(π′\)≻J​\(π\)\}\\mathcal\{F\}=\\\{\\pi\\in\\Pi\\mid\\nexists\\pi^\{\\prime\}\\in\\Pi:J\(\\pi^\{\\prime\}\)\\succ J\(\\pi\)\\\}\.

Soft Actor\-Critic \(SAC\)\.SAC\(Haarnojaet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib535)\)is an off\-policy actor\-critic algorithm grounded in the maximum entropy framework\. It augments standard single\-objective RL with an entropy term to encourage exploration\. The agent aims to maximizeJ\(π\)=𝔼π\[∑tγt\(rt\+αℋ\(π\(⋅\|𝐬t\)\)\)\]J\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{t\}\\gamma^\{t\}\(r\_\{t\}\+\\alpha\\mathcal\{H\}\(\\pi\(\\cdot\|\\mathbf\{s\}\_\{t\}\)\)\)\]\.

Reward Learning from Preference Feedback\.We follow the standard Preference\-based RL \(PbRL\) framework, which uses ‘latent rewards’ as a proxy for values\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520)\)\. Each human’s reward function is learned independently using their respective preference feedback\. Similar to prior work, we use preferences over pairs of trajectory segments,\(σ0,σ1\)\(\\sigma^\{0\},\\sigma^\{1\}\)\. An expert teacher \(e\.g\., human\) provides a labely∈\{0,1,0\.5\}y\\in\\\{0,1,0\.5\\\}to indicate their preference\. To learn a reward functionr^\\hat\{r\}parameterized byψ\\psi, we employ the Bradley\-Terry model\(Bradley and Terry[1952](https://arxiv.org/html/2607.29559#bib.bib536)\), modeling the preference probability asP​\[σ1≻σ0;ψ\]=exp​∑r^​\(σ1\)exp​∑r^​\(σ1\)\+exp​∑r^​\(σ0\)P\[\\sigma^\{1\}\\succ\\sigma^\{0\};\\psi\]=\\frac\{\\exp\\sum\\hat\{r\}\(\\sigma^\{1\}\)\}\{\\exp\\sum\\hat\{r\}\(\\sigma^\{1\}\)\+\\exp\\sum\\hat\{r\}\(\\sigma^\{0\}\)\}\. Given a dataset𝒟\\mathcal\{D\}, the reward function is trained by minimizing the cross\-entropy loss:

ℒC​E​\(ψ\)=−𝔼𝒟​\[\(1−y\)​log⁡P​\[σ0≻σ1\]\+y​log⁡P​\[σ1≻σ0\]\]\.\\mathcal\{L\}^\{CE\}\(\\psi\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\Big\[\(1\-y\)\\log P\[\\sigma^\{0\}\\succ\\sigma^\{1\}\]\+y\\log P\[\\sigma^\{1\}\\succ\\sigma^\{0\}\]\\Big\]\.\(1\)The learned reward function can then be used to update the policy with any RL algorithm to maximize expected returns\.

## 3Problem Setup

In this section, we present our formulation of Multi\-Objective Reinforcement Learning \(MORL\)without pre\-definedreward functions formultiple, conflictingobjectives\.

#### Latent Reward Vector\.

In standard MORL, the reward function is a known vector𝐫​\(s,a\)∈ℝm\\mathbf\{r\}\(s,a\)\\in\\mathbb\{R\}^\{m\}\. However, in our work, the agentdoes nothave access to the ground\-truth reward function\. Instead, we assume the existence ofmmdistinct multiple, conflicting human users \(objectives\), where each dimensionrir\_\{i\}corresponds to the latent reward function of theii\-th specific user\. Since these rewards are inaccessible, we must approximate them\. We define a parameterized reward vector𝐫^ψ​\(s,a\)=\[r^ψ1​\(s,a\),…,r^ψm​\(s,a\)\]T\\mathbf\{\\hat\{r\}\}\_\{\\psi\}\(s,a\)=\[\\hat\{r\}\_\{\\psi\_\{1\}\}\(s,a\),\\dots,\\hat\{r\}\_\{\\psi\_\{m\}\}\(s,a\)\]^\{T\}, where each component is a reward model learned from human preference feedback\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520)\)\.

#### Multi\-Objective RL Optimization\.

The agent’s goal is to maximize the expected discounted vector returns\. We adopt the most prevalent MORL formalism called the utility\-based approach\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525)\), where we define a scalarization functionf𝐰​\(𝐫\)=𝐰⊤​𝐫f\_\{\\mathbf\{w\}\}\(\\mathbf\{r\}\)=\\mathbf\{w\}^\{\\top\}\\mathbf\{r\}, where𝐰∈ℝm\\mathbf\{w\}\\in\\mathbb\{R\}^\{m\}is a preference weight vector on the simplex \(i\.e\.,∑wi=1\\sum w\_\{i\}=1\)\. Thus, we define an optimal MORL agent to be the policies belonging to the Convex Coverage Set \(CCS\), the subset ofℱ\\mathcal\{F\}optimal for linearly scalarized preferences\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525)\)\.

To summarize, the MORL agent learns these policies by maximizing the expected returns of thelearnedlatent reward vector linearly scalarized by the weightw\{w\}, which determines the trade\-offs between objectives:

J​\(π\)=𝔼π​\[∑t=0∞γt​𝐰T​𝐫^ψ​\(st,at\)\]\.J\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathbf\{\{w\}\}^\{T\}\\mathbf\{\\hat\{r\}\_\{\\psi\}\}\(s\_\{t\},a\_\{t\}\)\\right\]\.\(2\)

## 4LEMUR

LEMUR\(Learning to Align withMulti\-ObjectiveReinforcement Learning from Preference Feedback\), illustrated in Figure[1](https://arxiv.org/html/2607.29559#S1.F1)proceeds in three stages: \(1\) unsupervised pre\-training, where the agent explores via intrinsic rewards to collect diverse experiences \(Section[4\.1](https://arxiv.org/html/2607.29559#S4.SS1)\); \(2\) reward learning, where multiple human teachers are queried for preference feedback to train objective\-specific reward models \(Section[4\.2](https://arxiv.org/html/2607.29559#S4.SS2)\); and \(3\) multi\-objective RL training against the learned reward models \(Section[4\.3](https://arxiv.org/html/2607.29559#S4.SS3)\)\. Stages 2 and 3 repeat, continually improving both the reward models and the multi\-objective policies\. Full pseudocode is provided in Appendix[B](https://arxiv.org/html/2607.29559#A2)\.

### 4\.1Unsupervised Pre\-training

Standard PbRL suffers from uninformative queries caused by the limited coverage of random initialization\. To generate informative queries, LEMUR employs an unsupervised pre\-training phase driven by intrinsic motivation\(Leeet al\.[2021b](https://arxiv.org/html/2607.29559#bib.bib500)\)\. We encourage exploration by maximizing state entropy, approximated via a particle\-basedkk\-nearest neighbors \(kk\-NN\) estimator\(Liu and Abbeel[2021](https://arxiv.org/html/2607.29559#bib.bib546)\)\. The intrinsic rewardri​n​t​\(st\)=log⁡\(‖st−stk‖\)r^\{int\}\(s\_\{t\}\)=\\log\(\\\|s\_\{t\}\-s\_\{t\}^\{k\}\\\|\)is the normalized distance to thekk\-th nearest neighbor inℬ\\mathcal\{B\}, and the agent maximizes𝒥i​n​t​\(ϕ\)=𝔼πϕ​\[∑t=0Tγt​ri​n​t​\(st\)\]\\mathcal\{J\}\_\{int\}\(\\phi\)=\\mathbb\{E\}\_\{\\pi\_\{\\phi\}\}\[\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\textbf\{r\}^\{int\}\(s\_\{t\}\)\]\. This populates the buffer with diverse behaviors, accelerating the subsequent multi\-objective reward learning\.

### 4\.2Reward learning of Multiple Objectives from Preferences

A core challenge in our setup is that the agent does not have access to the ground\-truth rewards, and themmconflicting objectives are characterized instead by the conflicting preferences ofmmhumans\. Following the PbRL formulation in Section[2](https://arxiv.org/html/2607.29559#S2), LEMUR learns a separate reward model𝐫^ψj​\(s,a\)\\hat\{\\mathbf\{r\}\}\_\{\\psi\_\{j\}\}\(s,a\)per teacher, trained by minimizing the cross\-entropy loss between the model’s predictions and that teacher’s labels \(Equation[1](https://arxiv.org/html/2607.29559#S2.E1)\)\.

Weight\-Conditioned Reward Models\.Rather than learning each teacher’s reward in isolation, we condition every objective\-specific model on the shared objective space\. Each teacherjjis assigned a reward model𝐫^ψj​\(s,a\)\\hat\{\\mathbf\{r\}\}\_\{\\psi\_\{j\}\}\(s,a\), a lightweight MLP predicting the full objective vector, whose scalar utility is obtained by projecting onto that teacher’s preference anchor,r^j​\(s,a\)=𝐚j⊤​𝐫^ψj​\(s,a\)\\hat\{r\}\_\{j\}\(s,a\)=\\mathbf\{a\}\_\{j\}^\{\\top\}\\hat\{\\mathbf\{r\}\}\_\{\\psi\_\{j\}\}\(s,a\)\. This couples themmlearned models to a common vector\-reward structure, so that a policy conditioned on𝐰\\mathbf\{w\}reads a consistent per\-teacher utility at inference, and the reward models remain directly comparable as the number of conflicting teachers grows\. We deliberately adopt this simple architecture, as in prior approaches\(Muet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib585)\); we find it sufficient to recover strong compromise policies while keeping reward learning fast enough to remain in the loop with online policy optimization\.

Query Sampling Strategy\.We sample trajectory pairs\(σ0,σ1\)\(\\sigma^\{0\},\\sigma^\{1\}\)uniformly at random from the bufferℬ\\mathcal\{B\}, so that queries span the diverse state\-action distributions explored by all policies\. While more sophisticated disagreement\-based strategies exist, uniform sampling offers simplicity and avoids bias toward particular regions of the objective space during early training\.

### 4\.3Multi\-Objective RL Training

Given the parameterized reward vector𝐫^ψ​\(s,a\)\\mathbf\{\\hat\{r\}\}\_\{\\psi\}\(s,a\), LEMUR trains the MORL agent to maximize expected latentvectorrewards \(Equation[2](https://arxiv.org/html/2607.29559#S3.E2)\)\. For policy optimization we leverage MORL/D, a state\-of\-the\-art Multi\-Objective Soft Actor\-Critic \(MO\-SAC\) algorithm\(Feltenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib22)\)which learns a set of independent SAC policies and applies an evolutionary strategy for policy search\. This off\-policy choice is deliberate: reusing past experience from the replay buffer is essential for sample efficiency under a limited human feedback budget\.

Weight Vector Initialization and Adaptation\.The scalarization weight vectors\{𝐰\}\\\{\\mathbf\{w\}\\\}determine the trade\-offs between objectives\. Given𝐰∈ℝm\\mathbf\{w\}\\in\\mathbb\{R\}^\{m\}, we optimize for policies using SAC\(Haarnojaet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib535)\)on the scalarized reward of Equation[2](https://arxiv.org/html/2607.29559#S3.E2)\. We employ a Pareto Simulated Annealing \(PSA\) approach similar to\(Feltenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib22)\), adapting the weight vector in response to the current policies and their distance to non\-dominated solutions, allowing the agent to focus training on feasible regions of the objective space while maintaining policy diversity\.

Cooperation via Shared Buffer\.To facilitate information exchange across policies learning different trade\-offs, all policies store and sample from a common replay bufferℬ\\mathcal\{B\}\. This enables policies to learn from diverse experiences collected under different preference weightings, improving sample efficiency, a critical consideration given the limited human feedback budget\.

Relabeling of Vector Rewards\. Combining off\-policy RL with a reward function learned from preferences introduces non\-stationarity: as the reward model is updated with new feedback, rewards associated with past transitions in the buffer become stale, destabilizing learning\. To address this, LEMUR employs a vector reward relabeling strategy inspired by prior work\(Leeet al\.[2021b](https://arxiv.org/html/2607.29559#bib.bib500)\)\. Rather than storing rewards, we store only the transitions, and compute vector rewards on the fly when a batch is sampled, using the most up\-to\-date reward models\. This ensures the agent always trains on updated rewards, synchronizing policy and reward learning while preserving the sample efficiency of our off\-policy approach\.

## 5Experiments

Our experiments address three questions: \(1\) Can LEMUR learn multi\-objective policies that balance multiplelearnedreward models frommultipleteachers? \(2\) How does LEMUR compare to existing baselines on multi\-objective benchmarks? \(3\) Does explicitly learning multiple reward models for conflicting feedback outperform aggregating feedback into a single reward model? For all experiments, we report the mean across five random seeds with standard error\. Additional implementation details are reported in Appendices[D](https://arxiv.org/html/2607.29559#A4)&[E](https://arxiv.org/html/2607.29559#A5)\.

Benchmark Environments & Setup\.We evaluate LEMUR on high\-dimensional environments from theMORL\-Generalizationbenchmark\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\)\. Following standard practice in PbRL\(Leeet al\.[2021a](https://arxiv.org/html/2607.29559#bib.bib586); Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520)\), we use scripted teachers that generate feedback according to the components of the ground\-truth vector reward, enabling quantitative evaluation; the ground\-truth rewards remain inaccessible to the agent, which must jointly learn the conflicting preferences and optimize to find a balance\. Our main experiments use two conflicting teachers, the fundamental version of the problem; Section[5\.1](https://arxiv.org/html/2607.29559#S5.SS1)demonstrates scaling to more objectives\. We evaluate on:MO\-Lunarlander, where Teacher A rewards precise, stable landings and Teacher B prioritizes fuel conservation;MO\-HopperandMO\-Cheetah, continuous control locomotion tasks where Teacher A prefers fast locomotion and Teacher B prefers slow, energy\-efficient gaits; andMO\-MetaWorld\(Drawer\-Close\), a robotic\-manipulation task from the Meta\-World suite\(Yuet al\.[2020](https://arxiv.org/html/2607.29559#bib.bib587)\)\. Meta\-World tasks are natively single\-objective; we convert Drawer\-Close into a two\-objective task by pairing the native task\-progress reward \(Teacher A\) with a control\-effort penalty \(Teacher B\), mirroring the reward decomposition standard in the MORL benchmark suite\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\)\. Full environment details are given in Appendix[G](https://arxiv.org/html/2607.29559#A7)\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/x1.png)Figure 2:Learning curves on all benchmark environments:MO\-LunarLander,MO\-Hopper,MO\-Cheetah, andMO\-MetaWorld\. Curves depict the true objective returns \(inaccessible to the agent\), averaged across five seeds, with shaded regions representing standard error\.LEMUR\(blue\) most closely tracks theOracle\(red\) on both objectives simultaneously\. Baselines that aggregate conflicting feedback \(Naive,Utilitarian\) fail to make progress, while the external baselines \(MORAL,PbMORL,FPbRL\) learn but consistently trail LEMUR\.Baselines\.We compare against five baselines spanning distinct strategies for preference aggregation and learning from multiple objectives: \(1\) aUtilitarianagent, a single SAC agent optimizing the arithmetic mean of the independently learned rewards; \(2\)Naivedata pooling, which trains one monolithic reward model on all conflicting feedback, akin to standard Reinforcement Learning from Human Feedback \(RLHF\); \(3\)MORAL\(Peschlet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib295)\), which recovers per\-teacher rewards via Adversarial Inverse Reinforcement Learning \(AIRL\) and learns a scalarization over them; \(4\)PbMORL\(Muet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib585)\), a recent preference\-based multi\-objective method learning a weight\-conditioned vector reward from pairwise feedback; and \(5\)FPbRL\(Siddiqueet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib584)\), which aggregates learned per\-teacher rewards through a Generalized Gini Welfare scalarization to optimize for fairness\. We additionally report an \(6\)Oracletrained on ground\-truth rewards as an upper bound\. Unless otherwise noted, every baseline shares LEMUR’s interactive learning loop, reward\-model architecture, pre\-training stage, teacher weight vectors, query budget, and environment\-step budget; the primary distinction lies in*how conflicting reward signals are aggregated and optimized*\. Where a baseline’s original policy optimizer would disadvantage it in our environments, we adapt in the baseline’s favour; all deviations are disclosed in Appendix[E](https://arxiv.org/html/2607.29559#A5)\.

### 5\.1Results & Analysis

Figure[2](https://arxiv.org/html/2607.29559#S5.F2)presents the learning curves for all methods across the benchmark environments\. Across every environment, LEMUR is the method that most closely tracks the Oracle on both objectives simultaneously, effectively recovering policies that balance multiple, conflicting objectives\. The Utilitarian and Naive agents remain flat with suboptimal returns throughout, supporting our hypothesis that aggregating conflicting reward signals into a single scalar degrades performance in such multi\-objective settings\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/x2.png)Figure 3:Task Success Rate \(%\) learning curves on MetaWorld \(Drawer Close Task\)\.While MORAL improves early in LunarLander and Hopper, it fails to sustain this progress: in MO\-Hopper its returns peak mid\-training and then steadily declines\. This is consistent with the known failure of out\-of\-distribution optimization due to static reward models; MORAL infers its per\-teacher rewards offline from expert demonstrations via AIRL and holds them fixed\. LEMUR instead does online policy optimization and addresses non\-stationarity throughvector reward relabeling\.

PbMORL and FPbRL, which both learn vector rewards from preference feedback, perform better than the aggregation baselines, yet still fall short of LEMUR\. We attribute this to the assumptions both methods inherit\. PbMORL trains a single weight\-conditioned reward model over pooled feedback, implicitly assuming all preferences originate from one single teacher; under conflicting teachers the pooled model must average over conflicting labels, degrading the reward signal, most visibly in MO\-Hopper and MO\-Cheetah where it consistently trails LEMUR on both objectives\. FPbRL preserves the vector structure but employs a fixed Generalized Gini welfare scalarization a priori, converging to a single welfare\-optimal policy rather than a set of trade\-offs: it achieves reasonable returns on MetaWorld, but fails to achieve task success \(shown in Figure[3](https://arxiv.org/html/2607.29559#S5.F3)\) nor make progress on the other environments\. LEMUR avoids both failure modes by maintaining objective\-specific reward models and adapting the trade\-off online\. On MO\-MetaWorld, Figure[3](https://arxiv.org/html/2607.29559#S5.F3)reports the task success rate: LEMUR reaches closest to the Oracle, while PbMORL plateaus\.

#### Multi\-Objective Metrics\.

Table 1:Hypervolume and Sparsity for LEMUR and PbMORL\. Full results are in Appendix[F\.1](https://arxiv.org/html/2607.29559#A6.SS1)\.We evaluate LEMUR using standard multi\-objective metrics\(Hayeset al\.[2022a](https://arxiv.org/html/2607.29559#bib.bib524)\)\.Hypervolume\(HV\) measures the volume of objective space, rewarding policies that are both high\-performing and broadly spread\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\);Sparsity\(SPS\) measures the average distance between policies along the front, with lower values indicating more uniform coverage\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\)\. As summarised in Table[1](https://arxiv.org/html/2607.29559#S5.T1), LEMUR attains the highest or comparable Hypervolume across all four environments while achieving markedly lower sparsity than PbMORL, indicating that it learns policies that is both higher\-performing and more uniformly distributed over the trade\-off space\. MORAL is excluded from these set\-based metrics, as its single\-objective policy optimization against a scalarized reward recovers only one solution rather than a front\. Full results are reported in Appendix[F\.1](https://arxiv.org/html/2607.29559#A6.SS1)\.

#### Reward Model Alignment\.

We additionally evaluate the learned reward models directly against the ground\-truth teacher rewards, following established PbRL evaluation practice\(Leeet al\.[2021a](https://arxiv.org/html/2607.29559#bib.bib586)\)\. We reportSpearmanrank correlation, which measures how accurately the learned reward models rank individual states compared to the teachers’ ground\-truth reward; theTrajectory Alignment Coefficient\(TAC\)\(Muslimaniet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib593)\), which compares rankings over whole trajectories rather than individual transitions\. Table[2](https://arxiv.org/html/2607.29559#S5.T2)shows that LEMUR’s reward models recover their teachers’ preference orderings with consistently strong correlation, and outperform both PbMORL and FPbRL \(Appendix[F\.2](https://arxiv.org/html/2607.29559#A6.SS2)\)\.

Table 2:LEMUR reward model alignment, reporting Spearman rank correlation and Trajectory Alignment Coefficient \(TAC\)\. Per\-metric comparisons against PbMORL and FPbRL are in Appendix[F\.2](https://arxiv.org/html/2607.29559#A6.SS2)\.
#### Scaling to More Objectives\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/x3.png)Figure 4:LEMUR scalability to higher\-dimensional objective spaces\. Episode returns for\(a\)3\-objective and\(b\)4\-objective tasks, comparing policies trained with LEMUR \(blue\) versus ground\-truth oracle rewards \(orange\)\.![Refer to caption](https://arxiv.org/html/2607.29559v1/x4.png)

\(a\) Buffer Relabeling

![Refer to caption](https://arxiv.org/html/2607.29559v1/x5.png)

\(b\) Noisy Labels

![Refer to caption](https://arxiv.org/html/2607.29559v1/x6.png)

\(c\) Query Budget

Figure 5:Ablation studies onMO\-Cheetahevaluating the impact of \(a\) the shared buffer, vector reward relabeling, and unsupervised pre\-training, \(b\) noisy teacher labels, and \(c\) varying the total query budget per teacher on agent returns for both objectives; by default LEMUR uses 3900 queries \(green\)\. The results are averaged over multiple runs across five seeds\.We next vary the teacher configuration onMO\-Cheetah\. Figure[4](https://arxiv.org/html/2607.29559#S5.F4)extends LEMUR to three and four conflicting teachers: the learned\-reward policies closely track the ground\-truth oracle across all objectives, demonstrating the per\-teacher decomposition scales without modification, each additional objective adding one reward model\.

### 5\.2Ablation Studies

To validate the components of LEMUR, we conduct ablation studies on the high\-dimensionalMO\-Cheetahdomain; Figure[5](https://arxiv.org/html/2607.29559#S5.F5)visualizes the learning curves\.

Impact of Shared Buffer, Relabeling, and Pre\-training\.Figure[5](https://arxiv.org/html/2607.29559#S5.F5)\(a\) isolates the contributions of the shared replay buffer, vector reward relabeling, and unsupervised pre\-training\. Disabling the shared buffer causes the most severe degradation\. With the shared buffer intact, removing relabeling alone produces a modest but consistent drop relative to full LEMUR, confirming that recomputing rewards under the current models stabilizes learning against reward non\-stationarity\. Removing pre\-training accelerates the earliest phase of training, but converges to lower final returns, indicating that the diverse initial buffer ultimately yields better reward models and policies\.

Robustness to Label Noise\.Figure[5](https://arxiv.org/html/2607.29559#S5.F5)\(b\) corrupts a fraction of teacher labels \(flipping preferences with probability up to 15%\), following PbRL benchmarking protocol\(Leeet al\.[2021a](https://arxiv.org/html/2607.29559#bib.bib586)\)\. LEMUR degrades gracefully: performance is essentially unaffected up to 10% noise, and at 15% the agent still learns effective compromise policies on both objectives, albeit with slower convergence and higher variance, indicating tolerance to levels of annotator error\.

Impact of Feedback Budget\.Figure[5](https://arxiv.org/html/2607.29559#S5.F5)\(c\) varies the total query budget from 260 to 5,200 per teacher\. Performance improves with budget, and larger budgets learn faster; notably, even 260 total learns adequate policies on both objectives\. This feedback efficiency is particularly important in preference\-based RL, where human queries are limited\.

Reward Model Ablation\.To verify that LEMUR’s gains stem from its weight\-conditioned reward model rather than the surrounding pipeline, we re\-run LEMUR replacing this component with an ensemble of three unconditioned MLPs\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520); Leeet al\.[2021b](https://arxiv.org/html/2607.29559#bib.bib500)\), holding all else fixed\. The weight\-conditioned variant converges to6,812±396\{,\}812\\pm 39and4,404±224\{,\}404\\pm 22on the two objectives, against4,556±3694\{,\}556\\pm 369and2,902±2452\{,\}902\\pm 245for the ensemble\. \(For full results, refer to Appendix[C\.7](https://arxiv.org/html/2607.29559#A3.SS7)\)\.

Additional Experiments\.In Appendices[C\.1](https://arxiv.org/html/2607.29559#A3.SS1)\-[C\.7](https://arxiv.org/html/2607.29559#A3.SS7), we demonstrate that LEMUR accommodates changing teachers/objectives mid\-training, varying levels of conflict between teachers, and also non\-stationary preferences without reinitialization\. Query ablations reveal that segment length is important to performance, and we verify that LEMUR maintains performance even when teachers are in agreement with overlapping preferences\.

## 6Related Work

#### Reward Learning from Preference Feedback\.

Designing reward functions is a primary bottleneck in scaling RL, as manual crafting is impractical and can induce unsafe behavior\(Amodeiet al\.[2016](https://arxiv.org/html/2607.29559#bib.bib526)\); prior work instead learns rewards from demonstrations\(Ng and Russell[2000](https://arxiv.org/html/2607.29559#bib.bib531); Abbeel and Ng[2004](https://arxiv.org/html/2607.29559#bib.bib527)\), language\(Linet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib528)\), or human feedback\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520)\)\. Preference\-based RL \(PbRL\) learns rewards from pairwise comparisons\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520); Leeet al\.[2021b](https://arxiv.org/html/2607.29559#bib.bib500)\); popularized as RLHF\(Stiennonet al\.[2020](https://arxiv.org/html/2607.29559#bib.bib529); Ouyanget al\.[2022](https://arxiv.org/html/2607.29559#bib.bib539)\), it typically trains a single reward model, aggregating diverse feedback into one scalar\(Ouyanget al\.[2022](https://arxiv.org/html/2607.29559#bib.bib539)\)and failing to capture the multi\-objective nature of human values\(Sorensenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib32)\)\. Offline variants train rewards on fixed datasets before policy optimization\(Shinet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib515)\), but static models suffer distribution shift as the policy diverges from the offline coverage, causing reward exploitation\(Gaoet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib564); Yeet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib579)\)\. Online, iterative RLHF mitigates this by collecting feedback alongside policy optimization\(Donget al\.[2024](https://arxiv.org/html/2607.29559#bib.bib578); Gaoet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib564); Yeet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib579)\), which is more critical still in MORL, where the agent must span a space of diverse policies\(Hayeset al\.[2022a](https://arxiv.org/html/2607.29559#bib.bib524)\); offline MORL instead presupposes specified rewards or adequate dataset coverage\(Yuanet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib270); Zhuet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib582)\)\. LEMUR circumvents this by jointly and interactively optimizing both the reward models and the multi\-objective policy online\.

#### Multi\-Objective Reinforcement Learning \(MORL\)\.

MORL learns a set of policies approximating the Pareto frontier\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525); Hayeset al\.[2022b](https://arxiv.org/html/2607.29559#bib.bib139)\), via single\-policy scalarization, weight\-conditioned, or multi\-policy methods; multi\-task and Meta\-RL are closely related\(Chenet al\.[2019](https://arxiv.org/html/2607.29559#bib.bib542); Abdolmalekiet al\.[2020](https://arxiv.org/html/2607.29559#bib.bib543); Sener and Koltun[2018](https://arxiv.org/html/2607.29559#bib.bib574)\)\. Most of this literature assumes a vector of ground\-truth reward functions\(Hayeset al\.[2022b](https://arxiv.org/html/2607.29559#bib.bib139)\), which is impractical in complex, real\-world tasks\. Reward\-free MORL\(Chenet al\.[2026](https://arxiv.org/html/2607.29559#bib.bib547)\)relaxes this only partially, using reward\-free exploration as an auxiliary objective while still assuming an extrinsically specified ground\-truth reward\. LEMUR extends the MORL paradigm to the setting where the objectives are never observed and must be inferred directly from preferences\.

#### Learning & Alignment with Diverse Objectives\.

Standard RLHF fails to capture thepluralisticnature of human values\(Sorensenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib32)\), and existing remedies rely on manual aggregation functions\(Rodriguez\-Sotoet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib544)\), expensive consensus datasets\(Tessleret al\.[2024](https://arxiv.org/html/2607.29559#bib.bib37)\), static offline learning\(Bakkeret al\.[2022](https://arxiv.org/html/2607.29559#bib.bib498)\), or model heterogeneous feedback as hidden context without optimizing the trade\-off between preferences\(Siththaranjanet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib530)\)\. Our closest baselines learn rewards for multiple objectives but inherit strong coherence assumptions: MORAL\(Peschlet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib295)\)requires expert demonstrations and freezes AIRL\-learned rewards\(Fuet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib541)\), PbMORL\(Muet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib585)\)assumes a single teacher, and FPbRL\(Siddiqueet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib584)\)fixes a welfare scalarization a priori\. LEMUR instead learns objective\-specific reward models from separate feedback streams and jointly optimizes a multi\-objective policy online viavector reward relabeling, without offline pre\-training or expert demonstrations\.

## 7Conclusion & Limitations

We propose LEMUR, a framework for Multi\-Objective RL in domains where reward functions are unknown and must be inferred from the conflicting preferences of multiple teachers\. LEMUR jointly learns the objectives and the policies that balance them, without expert demonstrations, pre\-defined rewards, or a priori aggregation rules that existing methods require\. Across multi\-objective RL control and robotic manipulation benchmark environments, LEMUR outperforms aggregation baselines and recent preference\-based multi\-objective methods, and remains robust to label noise, reduced feedback budgets, and scaling to additional objectives\. Several directions for future work are as follows\. Our evaluation uses scripted teachers, standard practice in PbRL for controlled and reproducible comparison\(Leeet al\.[2021a](https://arxiv.org/html/2607.29559#bib.bib586)\); our noise ablations suggest the framework tolerates the label error real annotators exhibit, and a human study is the natural next validation\. We adopt linear scalarization, and extending to non\-linear scalarization would allow the framework to learn policies in non\-convex regions of the front\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525); Hayeset al\.[2022b](https://arxiv.org/html/2607.29559#bib.bib139)\)\. Finally, active querying\(Akrouret al\.[2012](https://arxiv.org/html/2607.29559#bib.bib594)\)offers a route to further reducing the number of queries, which is a promising path for deploying preference\-based MORL in real\-time\.

## References

- P\. Abbeel and A\. Y\. Ng \(2004\)Apprenticeship learning via inverse reinforcement learning\.InProceedings of the twenty\-first international conference on Machine learning,pp\. 1\.Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Abdolmaleki, S\. Huang, L\. Hasenclever, M\. Neunert, F\. Song, M\. Zambelli, M\. Martins, N\. Heess, R\. Hadsell, and M\. Riedmiller \(2020\)A distributional view on multi\-objective policy optimization\.InInternational conference on machine learning,pp\. 11–22\.Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px2.p1.1)\.
- R\. Akrour, M\. Schoenauer, and M\. Sebag \(2012\)April: active preference learning\-based reinforcement learning\.InJoint European conference on machine learning and knowledge discovery in databases,pp\. 116–131\.Cited by:[§7](https://arxiv.org/html/2607.29559#S7.p1.1)\.
- D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. Mané \(2016\)Concrete problems in ai safety\.arXiv preprint arXiv:1606\.06565\.Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p3.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- R\. Bahlous\-Boldi, I\. Puri, I\. Shenfeld, A\. Kumar, M\. Damani, S\. Risi, O\. Khattab, Z\. Hong, and P\. Agrawal \(2026\)Vector policy optimization: training for diversity improves test\-time search\.arXiv preprint arXiv:2605\.22817\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px3.p1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1)\.
- M\. Bakker, M\. Chadwick, H\. Sheahan, M\. Tessler, L\. Campbell\-Gillingham, J\. Balaguer, N\. McAleese, A\. Glaese, J\. Aslanides, M\. Botvinick,et al\.\(2022\)Fine\-tuning language models to find agreement among humans with diverse preferences\.Advances in neural information processing systems35,pp\. 38176–38189\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2607.29559#S1.p3.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- M\. Bowling, J\. D\. Martin, D\. Abel, and W\. Dabney \(2023\)Settling the reward hypothesis\.InInternational Conference on Machine Learning,pp\. 3003–3020\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§2](https://arxiv.org/html/2607.29559#S2.p3.6)\.
- X\. Chen, A\. Ghadirzadeh, M\. Björkman, and P\. Jensfelt \(2019\)Meta\-learning for multi\-objective reinforcement learning\.In2019 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 977–983\.Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px2.p1.1)\.
- Y\. Chen, W\. Hung, B\. Wu, Z\. Hong, and P\. Hsieh \(2026\)A reward\-free viewpoint on multi\-objective reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=IwiwmY3Mzz)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px2.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep Reinforcement Learning from Human Preferences\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2607.29559#S1.p3.1),[§2](https://arxiv.org/html/2607.29559#S2.p3.6),[§3](https://arxiv.org/html/2607.29559#S3.SS0.SSS0.Px1.p1.5),[§5\.2](https://arxiv.org/html/2607.29559#S5.SS2.p5.4),[§5](https://arxiv.org/html/2607.29559#S5.p2.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Ding, J\. Zhang, J\. Clune, L\. Spector, and J\. Lehman \(2024\)Quality diversity through human feedback: towards open\-ended diversity\-driven optimization\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=9zlZuAAb08)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px3.p1.1)\.
- H\. Dong, W\. Xiong, B\. Pang, H\. Wang, H\. Zhao, Y\. Zhou, N\. Jiang, D\. Sahoo, C\. Xiong, and T\. Zhang \(2024\)RLHF workflow: from reward modeling to online RLHF\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=a13aYUU9eU)Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Dulac\-Arnold, D\. Mankowitz, and T\. Hester \(2019\)Challenges of real\-world reinforcement learning\.External Links:[Link](https://openreview.net/forum?id=S1xtR52NjN)Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p2.1)\.
- F\. Felten, E\. Talbi, and G\. Danoy \(2024\)Multi\-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework\.Journal of Artificial Intelligence Research79,pp\. 679–723\(en\)\.External Links:ISSN 1076\-9757,[Link](https://jair.org/index.php/jair/article/view/15702),[Document](https://dx.doi.org/10.1613/jair.1.15702)Cited by:[Appendix D](https://arxiv.org/html/2607.29559#A4.p1.1),[Appendix E](https://arxiv.org/html/2607.29559#A5.SS0.SSS0.Px3.p2.1),[§4\.3](https://arxiv.org/html/2607.29559#S4.SS3.p1.1),[§4\.3](https://arxiv.org/html/2607.29559#S4.SS3.p2.2)\.
- A\. Fickinger, S\. Zhuang, D\. Hadfield\-Menell, and S\. Russell \(2020\)Multi\-principal assistance games\.arXiv preprint arXiv:2007\.09540\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1)\.
- J\. Fu, K\. Luo, and S\. Levine \(2018\)Learning robust rewards with adverserial inverse reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rkHywl-A-)Cited by:[Appendix E](https://arxiv.org/html/2607.29559#A5.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx \(2025\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.InNeurIPS 2025 Workshop on Efficient Reasoning,External Links:[Link](https://openreview.net/forum?id=21UFlJrmS2)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p2.1)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[§2](https://arxiv.org/html/2607.29559#S2.p2.1),[§4\.3](https://arxiv.org/html/2607.29559#S4.SS3.p2.2)\.
- C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz, E\. Howley, A\. A\. Irissappane, P\. Mannion, A\. Nowé, G\. Ramos, M\. Restelli, P\. Vamplew, and D\. M\. Roijers \(2022a\)A practical guide to multi\-objective reinforcement learning and planning\.Auton\. Agent\. Multi\. Agent\. Syst\.36\(1\) \(en\)\.Cited by:[§F\.1](https://arxiv.org/html/2607.29559#A6.SS1.p1.1),[§2](https://arxiv.org/html/2607.29559#S2.p1.14),[§5\.1](https://arxiv.org/html/2607.29559#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz, E\. Howley, A\. A\. Irissappane, P\. Mannion, A\. Nowé, G\. Ramos, M\. Restelli, P\. Vamplew, and D\. M\. Roijers \(2022b\)A Practical Guide to Multi\-Objective Reinforcement Learning and Planning\.Autonomous Agents and Multi\-Agent Systems36\(1\),pp\. 26\(en\)\.Note:arXiv:2103\.09568 \[cs\]External Links:ISSN 1387\-2532, 1573\-7454,[Link](http://arxiv.org/abs/2103.09568),[Document](https://dx.doi.org/10.1007/s10458-022-09552-y)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.29559#S1.p3.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2607.29559#S7.p1.1)\.
- S\. Huang, A\. Abdolmaleki, G\. Vezzani, P\. Brakel, D\. J\. Mankowitz, M\. Neunert, S\. Bohez, Y\. Tassa, N\. Heess, M\. Riedmiller,et al\.\(2022\)A constrained multi\-objective reinforcement learning framework\.InConference on Robot Learning,pp\. 883–893\.Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p2.1)\.
- B\. Ibarz, J\. Leike, T\. Pohlen, G\. Irving, S\. Legg, and D\. Amodei \(2018\)Reward learning from human preferences and demonstrations in atari\.Advances in neural information processing systems31\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1)\.
- R\. Kirk, I\. Mediratta, C\. Nalmpantis, J\. Luketina, E\. Hambro, E\. Grefenstette, and R\. Raileanu \(2024\)Understanding the effects of RLHF on LLM generalisation and diversity\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PXD3FAVHJT)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px3.p1.1)\.
- W\. B\. Knox, B\. D\. Glass, B\. C\. Love, W\. T\. Maddox, and P\. Stone \(2012\)How humans teach agents: a new experimental perspective\.International Journal of Social Robotics4\(4\),pp\. 409–421\.Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p3.1)\.
- S\. A\. Kouritem, M\. I\. Abouheaf, N\. Nahas, and M\. Hassan \(2022\)A multi\-objective optimization design of industrial robot arms\.Alexandria Engineering Journal61\(12\),pp\. 12847–12867\.Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p2.1)\.
- K\. Lee, L\. Smith, A\. Dragan, and P\. Abbeel \(2021a\)B\-pref: benchmarking preference\-based reinforcement learning\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 1\),External Links:[Link](https://openreview.net/forum?id=ps95-mkHF_)Cited by:[§C\.3](https://arxiv.org/html/2607.29559#A3.SS3.p1.1),[§F\.2](https://arxiv.org/html/2607.29559#A6.SS2.p1.8),[§5\.1](https://arxiv.org/html/2607.29559#S5.SS1.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2607.29559#S5.SS2.p3.1),[§5](https://arxiv.org/html/2607.29559#S5.p2.1),[§7](https://arxiv.org/html/2607.29559#S7.p1.1)\.
- K\. Lee, L\. M\. Smith, and P\. Abbeel \(2021b\)PEBBLE: feedback\-efficient interactive reinforcement learning via relabeling experience and unsupervised pre\-training\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 6152–6163\.External Links:[Link](https://proceedings.mlr.press/v139/lee21i.html)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[§C\.3](https://arxiv.org/html/2607.29559#A3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.29559#S4.SS1.p1.6),[§4\.3](https://arxiv.org/html/2607.29559#S4.SS3.p4.1),[§5\.2](https://arxiv.org/html/2607.29559#S5.SS2.p5.4),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Lin, D\. Fried, D\. Klein, and A\. Dragan \(2022\)Inferring rewards from language in context\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8546–8560\.Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- H\. Liu and P\. Abbeel \(2021\)Behavior from the void: unsupervised active pre\-training\.Advances in Neural Information Processing Systems34,pp\. 18459–18473\.Cited by:[§4\.1](https://arxiv.org/html/2607.29559#S4.SS1.p1.6)\.
- Y\. J\. Ma, J\. Hejna, C\. Fu, D\. Shah, J\. Liang, Z\. Xu, S\. Kirmani, P\. Xu, D\. Driess, T\. Xiao, O\. Bastani, D\. Jayaraman, W\. Yu, T\. Zhang, D\. Sadigh, and F\. Xia \(2025\)Vision language models are in\-context value learners\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 33984–34009\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/54854cf15a24fff9f5134a8641136fe4-Paper-Conference.pdf)Cited by:[§F\.2](https://arxiv.org/html/2607.29559#A6.SS2.SSS0.Px1.p1.1)\.
- J\. McCarthy \(1997\)What is artificial intelligence?\.Note:Stanford UniversityAlso available in a 2007 version at http://www\-formal\.stanford\.edu/jmc/whatisai\.pdfCited by:[§1](https://arxiv.org/html/2607.29559#S1.p1.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. A\. Riedmiller \(2013\)Playing atari with deep reinforcement learning\.CoRRabs/1312\.5602\.External Links:[Link](http://arxiv.org/abs/1312.5602),1312\.5602Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p1.1)\.
- N\. Mu, Y\. Luan, and Q\. Jia \(2025\)Preference\-based multi\-objective reinforcement learning\.IEEE Transactions on Automation Science and Engineering\.Cited by:[Table 5](https://arxiv.org/html/2607.29559#A4.T5.8.8.10.2.1.1),[Appendix E](https://arxiv.org/html/2607.29559#A5.SS0.SSS0.Px3),[§4\.2](https://arxiv.org/html/2607.29559#S4.SS2.p2.5),[§5](https://arxiv.org/html/2607.29559#S5.p3.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- R\. Munos, M\. Valko, D\. Calandriello, M\. Gheshlaghi Azar, M\. Rowland, Z\. D\. Guo, Y\. Tang, M\. Geist, T\. Mesnard, C\. Fiegel, A\. Michi, M\. Selvi, S\. Girgin, N\. Momchev, O\. Bachem, D\. J\. Mankowitz, D\. Precup, and B\. Piot \(2024\)Nash learning from human feedback\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 36743–36768\.External Links:[Link](https://proceedings.mlr.press/v235/munos24a.html)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p2.1)\.
- C\. Muslimani, K\. Johnstonbaugh, S\. Chandramouli, S\. Booth, W\. B\. Knox, and M\. E\. Taylor \(2025\)Towards improving reward design in RL: a reward alignment metric for RL practitioners\.InReinforcement Learning Conference,External Links:[Link](https://openreview.net/forum?id=XZBYLXNGjT)Cited by:[§F\.2](https://arxiv.org/html/2607.29559#A6.SS2.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2607.29559#S5.SS1.SSS0.Px2.p1.1)\.
- A\. Y\. Ng and S\. J\. Russell \(2000\)Algorithms for inverse reinforcement learning\.InProceedings of the Seventeenth International Conference on Machine Learning,pp\. 663–670\.Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- B\. Pásztor, T\. K\. Buening, and A\. Krause \(2025\)Stackelberg learning from human feedback: preference optimization as a sequential game\.InNeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists,External Links:[Link](https://openreview.net/forum?id=NPBd0gBogw)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p2.1)\.
- M\. Peschl, A\. Zgonnikov, F\. A\. Oliehoek, and L\. C\. Siebert \(2022\)MORAL: aligning ai with human norms through multi\-objective reinforced active learning\.InProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems,pp\. 1038–1046\.Cited by:[Table 6](https://arxiv.org/html/2607.29559#A4.T6),[Table 7](https://arxiv.org/html/2607.29559#A4.T7),[Appendix E](https://arxiv.org/html/2607.29559#A5.SS0.SSS0.Px5),[§5](https://arxiv.org/html/2607.29559#S5.p3.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- T\. Pierrot, G\. Richard, K\. Beguir, and A\. Cully \(2022\)Multi\-objective quality diversity optimization\.InProceedings of the genetic and evolutionary computation conference,pp\. 139–147\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px3.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1)\.
- A\. Rame, G\. Couairon, C\. Dancette, J\. Gaya, M\. Shukor, L\. Soulier, and M\. Cord \(2023\)Rewarded soups: towards pareto\-optimal alignment by interpolating weights fine\-tuned on diverse rewards\.Advances in Neural Information Processing Systems36,pp\. 71095–71134\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2607.29559#S1.p3.1)\.
- M\. Rodriguez\-Soto, J\. A\. Rodriguez\-Aguilar, M\. Lopez\-Sanchez, and A\. Nowé \(2023\)Multi\-objective reinforcement learning for guaranteeing alignment with multiple values\.In 2023 Adaptive and Learning Agents Workshop at AAMAS\.\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- D\. M\. Roijers, P\. Vamplew, S\. Whiteson, and R\. Dazeley \(2013\)A survey of multi\-objective sequential decision\-making\.Journal of Artificial Intelligence Research48\(1\),pp\. 67–113\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1),[§F\.1](https://arxiv.org/html/2607.29559#A6.SS1.SSS0.Px1.p1.4),[§F\.1](https://arxiv.org/html/2607.29559#A6.SS1.p1.1),[§1](https://arxiv.org/html/2607.29559#S1.p3.1),[§1](https://arxiv.org/html/2607.29559#S1.p4.1),[§3](https://arxiv.org/html/2607.29559#S3.SS0.SSS0.Px2.p1.4),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2607.29559#S7.p1.1)\.
- S\. Ross, G\. Gordon, and D\. Bagnell \(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the fourteenth international conference on artificial intelligence and statistics,pp\. 627–635\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1)\.
- O\. Sener and V\. Koltun \(2018\)Multi\-task learning as multi\-objective optimization\.Advances in neural information processing systems31\.Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px2.p1.1)\.
- W\. F\. Shen, X\. Qiu, C\. Whitehouse, L\. Alazraki, S\. Goel, F\. Barbieri, T\. Willi, A\. Mathur, and I\. Leontiadis \(2026\)Rethinking rubric generation for improving llm judge and reward modeling for open\-ended tasks\.arXiv preprint arXiv:2602\.05125\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p2.1)\.
- D\. Shin, A\. D\. Dragan, and D\. S\. Brown \(2023\)Benchmarks and Algorithms for Offline Preference\-Based Reward Learning\.arXiv\(en\)\.Note:arXiv:2301\.01392 \[cs\]Comment: Transactions on Machine Learning Research\. arXiv admin note: text overlap with arXiv:2107\.09251External Links:[Link](http://arxiv.org/abs/2301.01392)Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- U\. Siddique, A\. Sinha, and Y\. Cao \(2023\)Fairness in preference\-based reinforcement learning\.InICML 2023 Workshop The Many Facets of Preference\-Based Learning,External Links:[Link](https://openreview.net/forum?id=ColATVnkEl)Cited by:[Table 5](https://arxiv.org/html/2607.29559#A4.T5.8.8.13.5.1.1),[Appendix E](https://arxiv.org/html/2607.29559#A5.SS0.SSS0.Px4),[§5](https://arxiv.org/html/2607.29559#S5.p3.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- D\. Silver, S\. Singh, D\. Precup, and R\. S\. Sutton \(2021\)Reward is enough\.Artificial intelligence299,pp\. 103535\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1)\.
- A\. Siththaranjan, C\. Laidlaw, and D\. Hadfield\-Menell \(2024\)Distributional preference learning: understanding and accounting for hidden context in RLHF\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=0tWTxYYPnW)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p2.1),[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- J\. M\. V\. Skalse and A\. Abate \(2023\)The reward hypothesis is false\.External Links:[Link](https://openreview.net/forum?id=M4UxoupR3az)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1)\.
- T\. Sorensen, J\. Moore, J\. Fisher, M\. L\. Gordon, N\. Mireshghallah, C\. M\. Rytting, A\. Ye, L\. Jiang, X\. Lu, N\. Dziri, T\. Althoff, and Y\. Choi \(2024\)Position: a roadmap to pluralistic alignment\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 46280–46302\.External Links:[Link](https://proceedings.mlr.press/v235/sorensen24a.html)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2607.29559#S1.p4.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.Advances in neural information processing systems33,pp\. 3008–3021\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- R\. S\. Sutton and A\. G\. Barto \(1998\)Reinforcement learning: an introduction\.Adaptive computation and machine learning,MIT Press,Cambridge, Mass\(en\)\.External Links:ISBN 978\-0\-262\-19398\-6Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p1.1)\.
- J\. Tan, T\. Zhang, E\. Coumans, A\. Iscen, Y\. Bai, D\. Hafner, S\. Bohez, and V\. Vanhoucke \(2018\)Sim\-to\-real: learning agile locomotion for quadruped robots\.Preprint\.Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p1.1)\.
- J\. Teoh, P\. Varakantham, and P\. Vamplew \(2025\)On generalization across environments in multi\-objective reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tuEP424UQ5)Cited by:[Appendix D](https://arxiv.org/html/2607.29559#A4.p1.1),[§F\.1](https://arxiv.org/html/2607.29559#A6.SS1.SSS0.Px1.p1.4),[Appendix G](https://arxiv.org/html/2607.29559#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.29559#S5.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.29559#S5.p2.1)\.
- M\. H\. Tessler, M\. A\. Bakker, D\. Jarrett, H\. Sheahan, M\. J\. Chadwick, R\. Koster, G\. Evans, L\. Campbell\-Gillingham, T\. Collins, D\. C\. Parkes, M\. Botvinick, and C\. Summerfield \(2024\)AI can help humans find common ground in democratic deliberation\.Science386\(6719\),pp\. eadq2852\(en\)\.External Links:ISSN 0036\-8075, 1095\-9203,[Link](https://www.science.org/doi/10.1126/science.adq2852),[Document](https://dx.doi.org/10.1126/science.adq2852)Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px3.p1.1)\.
- M\. Umer, M\. A\. Mohsin, A\. Bilal, A\. Chaudhry, A\. Haupt, S\. Koyejo, E\. Fox, and J\. M\. Cioffi \(2026\)General preference reinforcement learning\.arXiv preprint arXiv:2605\.18721\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p2.1)\.
- P\. Vamplew, R\. Dazeley, A\. Berry, R\. Issabekov, and E\. Dekker \(2011\)Empirical evaluation methods for multiobjective reinforcement learning algorithms\.Machine learning84\(1\),pp\. 51–80\.Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p4.1)\.
- P\. Vamplew, B\. J\. Smith, J\. Källström, G\. Ramos, R\. Rădulescu, D\. M\. Roijers, C\. F\. Hayes, F\. Heintz, P\. Mannion, P\. J\. Libin,et al\.\(2022\)Scalar reward is not enough: a response to silver, singh, precup and sutton \(2021\)\.Autonomous Agents and Multi\-Agent Systems36\(2\),pp\. 41\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1)\.
- H\. Van Seijen, M\. Fatemi, J\. Romoff, R\. Laroche, T\. Barnes, and J\. Tsang \(2017\)Hybrid reward architecture for reinforcement learning\.Advances in neural information processing systems30\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px1.p1.1)\.
- Z\. Wang, S\. Rahmani, D\. Cornelisse, B\. Sarkar, A\. D\. Goldie, J\. N\. Foerster, and S\. Whiteson \(2026\)Learning to drive in new cities without human demonstrations\.InWorkshop on Simulation for Autonomous Driving,External Links:[Link](https://openreview.net/forum?id=szVQsLccbW)Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p2.1)\.
- D\. White \(1982\)Multi\-objective infinite\-horizon discounted markov decision processes\.Journal of mathematical analysis and applications89\(2\),pp\. 639–647\.Cited by:[§2](https://arxiv.org/html/2607.29559#S2.p1.14)\.
- Z\. Wu, Y\. Hu, W\. Shi, N\. Dziri, A\. Suhr, P\. Ammanabrolu, N\. A\. Smith, M\. Ostendorf, and H\. Hajishirzi \(2023\)Fine\-grained human feedback gives better rewards for language model training\.Advances in Neural Information Processing Systems36,pp\. 59008–59033\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Xu, Y\. Tian, P\. Ma, D\. Rus, S\. Sueda, and W\. Matusik \(2020\)Prediction\-guided multi\-objective reinforcement learning for continuous robot control\.InInternational conference on machine learning,pp\. 10607–10616\.Cited by:[§F\.1](https://arxiv.org/html/2607.29559#A6.SS1.SSS0.Px2.p1.5)\.
- R\. Yang, X\. Pan, F\. Luo, S\. Qiu, H\. Zhong, D\. Yu, and J\. Chen \(2024\)Rewards\-in\-context: multi\-objective alignment of foundation models with dynamic preference adjustment\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=QLcBzRI3V3)Cited by:[§1](https://arxiv.org/html/2607.29559#S1.p3.1)\.
- R\. Yang, X\. Sun, and K\. Narasimhan \(2019\)A generalized algorithm for multi\-objective reinforcement learning and policy adaptation\.Advances in neural information processing systems32\.Cited by:[Appendix E](https://arxiv.org/html/2607.29559#A5.SS0.SSS0.Px3.p2.1)\.
- C\. Ye, W\. Xiong, Y\. Zhang, H\. Dong, N\. Jiang, and T\. Zhang \(2024\)Online iterative reinforcement learning from human feedback with general preference model\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=TwdX1W3M6S)Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- T\. Yu, D\. Quillen, Z\. He, R\. Julian, K\. Hausman, C\. Finn, and S\. Levine \(2020\)Meta\-world: a benchmark and evaluation for multi\-task and meta reinforcement learning\.InConference on robot learning,pp\. 1094–1100\.Cited by:[§5](https://arxiv.org/html/2607.29559#S5.p2.1)\.
- Y\. Yuan, Z\. Zheng, Z\. Dong, and J\. Hao \(2024\)MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi\-Objective Reinforcement Learning\.arXiv\(en\)\.Note:arXiv:2408\.15501 \[cs\]Comment: 23 pages, 7 figuresExternal Links:[Link](http://arxiv.org/abs/2408.15501)Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.
- T\. Zhi\-Xuan, M\. Carroll, M\. Franklin, and H\. Ashton \(2025\)Beyond preferences in ai alignment: t\. zhi\-xuan et al\.\.Philosophical Studies182\(7\),pp\. 1813–1863\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px4.p1.1)\.
- Z\. Zhou, J\. Liu, J\. Shao, X\. Yue, C\. Yang, W\. Ouyang, and Y\. Qiao \(2024\)Beyond one\-preference\-fits\-all alignment: multi\-objective direct preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 10586–10613\.Cited by:[Appendix A](https://arxiv.org/html/2607.29559#A1.SS0.SSS0.Px2.p1.1)\.
- B\. Zhu, M\. Dang, and A\. Grover \(2023\)Scaling pareto\-efficient decision making via offline multi\-objective RL\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Ki4ocDm364)Cited by:[§6](https://arxiv.org/html/2607.29559#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix

## Appendix AExtended Related Works

#### Vector Rewards and the Limits of Scalar Reward\.

A key premise in modern RL is thereward hypothesis: that goals can be adequately captured by maximizing a singlescalarreward\(Silveret al\.[2021](https://arxiv.org/html/2607.29559#bib.bib550)\)\. A growing body of work contests the sufficiency of this framing, arguing that many objectives of interest cannot always be expressed by scalar reward\(Vamplewet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib551); Skalse and Abate[2023](https://arxiv.org/html/2607.29559#bib.bib553); Bowlinget al\.[2023](https://arxiv.org/html/2607.29559#bib.bib552); Van Seijenet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib583)\)and are more naturally represented asvectors\(Vamplewet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib551); Skalse and Abate[2023](https://arxiv.org/html/2607.29559#bib.bib553)\)\. Vector rewards and Pareto\-optimal policy sets are common in multi\-objective decision\-making\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525); Hayeset al\.[2022b](https://arxiv.org/html/2607.29559#bib.bib139)\), where general policy optimization recovers the Pareto front by conditioning a single policy\-network on a sampled weight vector\. This literature, however, largely assumes the ground\-truth vector reward function is given\. The reward\-free viewpoint of\(Chenet al\.[2026](https://arxiv.org/html/2607.29559#bib.bib547)\)relaxes part of this assumption by using preference\-guided exploration as an auxiliary task, yet still requires the ground\-truth multi\-objective reward during training\. In contrast, LEMUR avoids these assumptions\. Instead, it posits that true objectives are never directly observed and must be learned, in this case from preference feedback\. Thus, the agent jointly learns reward models and an optimized policy that balances its various components\.

#### Multi\-Dimensional and Fine\-Grained Preference Learning\.

Standard RLHF distills human comparisons into a single scalar reward\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520); Ouyanget al\.[2022](https://arxiv.org/html/2607.29559#bib.bib539)\), which can conflate distinct criteria and collapse the multi\-objective structure of human values\(Sorensenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib32)\)\. A line of work therefore decomposes feedback into finer\-grained components: Fine\-Grained RLHF\(Wuet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib548)\)attaches rewards to localized segments and categories, while Rewarded Soups\(Rameet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib30)\)and MODPO\(Zhouet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib549)\)obtain a separate signal per objective and expose a Pareto family through weight interpolation or scalarized preference optimization\. These methods combine per\-objective signalspost hocrather than jointly learning the rewards and a trade\-off policy, and are developed almost exclusively for LLM post\-training\. A related direction forgoes the reward model entirely, optimizing the policy directly from preference data \(e\.g\., DPO\(Rafailovet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib560)\)\); such methods are effective but train on static or periodically refreshed preferences, forgoing the online, interactive feedback and policy optimization, ideal for reliable reward learning under policy improvement\(Christianoet al\.[2017](https://arxiv.org/html/2607.29559#bib.bib520); Leeet al\.[2021b](https://arxiv.org/html/2607.29559#bib.bib500); Gaoet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib564); Baiet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib563); Rosset al\.[2011](https://arxiv.org/html/2607.29559#bib.bib562); Ibarzet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib561); Stiennonet al\.[2020](https://arxiv.org/html/2607.29559#bib.bib529)\)\.

A further perspective treats heterogeneous feedback as hidden context, learning a distribution over reward functions\(Siththaranjanet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib530)\); this captureswhichpreferences are present but does not, on its own, optimize an explicit compromise between them\. LEMUR differs from all three: it learns an explicit, objective\-specific reward model per feedback streamonline, and jointly optimizes a set of policies over the resulting vector reward in continuous control\. A complementary line moves beyond the Bradley\-Terry preference model assumption itself\. General preference models represent preferences with a richer \(e\.g\., skew\-symmetric,kk\-dimensional\) structure that admitsintransitivecycles, optimizing policies directly against this structure rather than a scalar reward\(Umeret al\.[2026](https://arxiv.org/html/2607.29559#bib.bib566)\)\. Related game\-theoretic formulations cast alignment as a Nash or Stackelberg equilibrium over preferences\(Munoset al\.[2024](https://arxiv.org/html/2607.29559#bib.bib567); Pásztoret al\.[2025](https://arxiv.org/html/2607.29559#bib.bib568)\)\. Intransitivity is a different failure mode of scalar rewards from the one we study; it concerns theshapeof preferences rather than the presence of multiple objectives, and we regard bridging it with multi\-objective preference optimization as promising future work\. In the LLM setting, rubric\-based rewards have similarly been used to supply multi\-dimensional supervision for reinforcement fine\-tuning, though their reliability is sensitive to rubric coverage and correlated criteria\(Gunjalet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib570); Shenet al\.[2026](https://arxiv.org/html/2607.29559#bib.bib571)\)\. These directions are largely orthogonal to LEMUR, which targets online vector\-reward learning and trade\-off policy optimization\.

#### Connections to Diversity and Foundation\-Model Post\-Training\.

A contemporary line of work observes that scalar RL post\-training inducesentropy collapse, eroding the solution diversity required by inference\-time search\(Kirket al\.[2024](https://arxiv.org/html/2607.29559#bib.bib556)\)\. Most directly, Vector Policy Optimization\(Bahlous\-Boldiet al\.[2026](https://arxiv.org/html/2607.29559#bib.bib555)\)samples scalarizations over the reward simplex and trains a policy to output a set of solutions spanning the Pareto front, improving downstream best\-of\-NNsearch\. Such methods share LEMUR’s premise that collapsing a vector reward into a scalar discards useful structure\. They differ from our setting in three respects: \(i\) they target LLMtest\-time search, seeking a diverse candidate pool for a downstream selector, whereas LEMUR seeks a policy thatcompromisesbetween objectivesduring deployment; \(ii\) they assume the reward components areknown and observable\(e\.g\., per\-test\-case correctness\), whereas LEMUR mustlearnthem from feedback; and \(iii\) they operate on natural language generation rather than continuous control\. We therefore treat this literature as motivating context rather than comparable works, and leave the study of train\-time multi\-objective learning from feedback to improve test\-time diversity\(Dinget al\.[2024](https://arxiv.org/html/2607.29559#bib.bib557); Pierrotet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib558)\)as future work\.

#### Pluralistic Alignment and Multiple Principals\.

LEMUR’s multi\-teacher formulation connects topluralistic alignment, which holds that a single reward model cannot represent the plurality of human values\(Sorensenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib32)\)and that an agent should instead balance multiple objectives to reach a compromise across them\. A closely related framing casts this as servingmultiple principals: a single agent acting on behalf of several stakeholders must respect the preferences of each, turning alignment into the problem of representing and trading off competing principals rather than satisfying one\(Fickingeret al\.[2020](https://arxiv.org/html/2607.29559#bib.bib559)\)\. How to reconcile these principals is itself contested\. Naively aggregating preferences via RLHF has been shown to behave as a Borda count over latent objectives, with limited normative justification\(Siththaranjanet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib530)\), and others question whether aggregation is the right frame at all, proposing non\-aggregative alternatives for reconciling plural values\(Zhi\-Xuanet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib569)\)\. Practical efforts that do aggregate rely on curated consensus datasets or hand\-specified aggregation functions\(Bakkeret al\.[2022](https://arxiv.org/html/2607.29559#bib.bib498); Rodriguez\-Sotoet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib544); Tessleret al\.[2024](https://arxiv.org/html/2607.29559#bib.bib37)\), both costly to obtain a priori and brittle when preferences shift during deployment\. Rather than collapsing principals to a consensus in advance or assuming their reward functions are known, LEMUR preserves each principal’s objective as a separate learned reward model and recovers an explicit compromise through multi\-objective optimization\.

## Appendix BLEMUR Pseudocode

Algorithm 1LEMUR1:Feedback frequency

KK, queries per session

MM, number of objectives

mm
2:Initialize policy

ϕ\\phiand reward models

\{ψj\}j=1m\\\{\\psi\_\{j\}\\\}\_\{j=1\}^\{m\}
3:Initialize shared buffer

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptysetand preference datasets

𝒟j←∅\\mathcal\{D\}\_\{j\}\\leftarrow\\emptyset
4:

5:

ℬ←Explore​\(πϕ,ri​n​t\)\\mathcal\{B\}\\leftarrow\\textsc\{Explore\}\(\\pi\_\{\\phi\},r^\{int\}\)⊳\\trianglerightPre\-training

6:

7:foreach iterationdo

8:ifiteration

modK==0\\bmod~K==0then⊳\\trianglerightReward learning

9:foreach teacher

j∈\{1,…,m\}j\\in\\\{1,\\dots,m\\\}do

10:Sample queries

\(σ0,σ1\)∼ℬ\(\\sigma^\{0\},\\sigma^\{1\}\)\\sim\\mathcal\{B\}
11:

𝒟j←𝒟j∪\{\(σ0,σ1,yj\)\}n=1M\\mathcal\{D\}\_\{j\}\\leftarrow\\mathcal\{D\}\_\{j\}\\cup\\\{\(\\sigma^\{0\},\\sigma^\{1\},y\_\{j\}\)\\\}\_\{n=1\}^\{M\}
12:Update

r^ψj\\hat\{r\}\_\{\\psi\_\{j\}\}on

𝒟j\\mathcal\{D\}\_\{j\}via cross\-entropy loss

13:endfor

14:endif

15:

16:Collect transitions

\(st,at,st\+1\)\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)with

πϕ\\pi\_\{\\phi\}and store in

ℬ\\mathcal\{B\}
17:foreach gradient stepdo⊳\\trianglerightMulti\-objective policy optimization

18:Sample batch

\(s,a,s′\)∼ℬ\(s,a,s^\{\\prime\}\)\\sim\\mathcal\{B\}
19:Relabel vector rewards:

rw=𝐰⊤​𝐫^ψ​\(s,a\)r\_\{w\}=\\mathbf\{w\}^\{\\top\}\\mathbf\{\\hat\{r\}\}\_\{\\psi\}\(s,a\)
20:Update actor and critic using MO\-SAC

21:endfor

22:endfor

## Appendix CAdditional Experiments

### C\.1Adding a Teacher Mid\-Training

![Refer to caption](https://arxiv.org/html/2607.29559v1/x7.png)Figure 6:Adding a third teacher mid\-training \(MO\-Cheetah\)\.A third teacher is introduced at4×1054\\times 10^\{5\}steps \(dotted line\) into an already\-training two\-teacher run\. The new objective \(bottom\) is learned from scratch while the two existing objectives \(top, middle\) are preserved, so no retraining from scratch is required\. Mean±\\pmstd over five seeds\.A practical deployment of preference\-based MORL is unlikely to have a fixed, known set of stakeholders at the outset: new preference sources appear over time\. A framework that requires retraining from scratch whenever a teacher joins is therefore of limited practical use\. Because LEMUR maintains one weight\-conditioned reward model per teacher and couples them only through the shared MORL/D population, adding a teacher requires instantiating a single new reward model and extending the vector reward, leaving the existing reward models and the trained policy population intact\.

We test this directly: a run begins with the standard two\-teacherMO\-Cheetahsetup and a third teacher is introduced at4×1054\\times 10^\{5\}environment steps, with training continuing uninterrupted from the existing population\. Figure[6](https://arxiv.org/html/2607.29559#A3.F6)shows the outcome\. The newly added Objective Three is optimized from a cold start and rises steeply once its teacher joins, while Objectives One and Two, already near convergence at the changepoint, are retained rather than degraded, settling into a marginally adjusted equilibrium that accommodates the new objective\. Crucially, the framework absorbs the new preference source*online*, without reinitialising either the reward models or the policy population, demonstrating that the cost of adding a teacher is incremental rather than a full retraining cycle\.

### C\.2Non\-Stationary Preferences

Human preferences are not static, and a teacher may revise its trade\-off during the course of training\. Since LEMUR re\-queries every teacher and relabels the shared replay buffer at each feedback interval, a revised preference propagates into the learned reward and hence into the policy objective without any special\-case handling\. To test this, Teacher A’s weight vector is altered at2×1052\\times 10^\{5\}steps mid\-training\.

Figure[7](https://arxiv.org/html/2607.29559#A3.F7)shows that all three objectives continue to improve monotonically across the changepoint: the policy adapts to the revised utility rather than collapsing or plateauing, and the widening variance band immediately after the change reflects the transient period during which the reward models are being re\-fit to the new preference before the population re\-converges\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/x8.png)Figure 7:Non\-stationary preferences \(MO\-Cheetah\)\.Teacher A’s preference vector is changed at2×1052\\times 10^\{5\}steps \(dotted line\)\. Returns continue to improve across the changepoint, indicating that the online re\-query and buffer\-relabeling loop tracks the revised utility\. Mean±\\pmstd over five seeds\.
### C\.3Ablation: Query Length

![Refer to caption](https://arxiv.org/html/2607.29559v1/x9.png)Figure 8:Effect of query segment length \(MO\-Cheetah\)\.Ground\-truth return per objective for preference queries of length5050\(default\),3535and11transition, at a fixed budget of300300queries per teacher\. Shading is±1\\pm 1std; the length\-3535and length\-11arms are single seeds and carry a nominal band\. Shorter segments learn faster early but plateau by∼2×105\{\\sim\}2\\times 10^\{5\}steps, while length5050overtakes them and continues improving\.LEMUR elicits preferences overtrajectory segments, determining the behavioral context teachers see per comparison\. At a fixed budget of 300 queries per teacher, we tested segment lengths of 50, 35, and 1 transition \(Figure[8](https://arxiv.org/html/2607.29559#A3.F8)\)\. LEMUR maintains performance even with shorter segments of feedback\. The improvement in performance due to longer segments is potentially due to more context for reward learning\(Leeet al\.[2021a](https://arxiv.org/html/2607.29559#bib.bib586),[b](https://arxiv.org/html/2607.29559#bib.bib500)\)\.

### C\.4Ablation: Query Sampling Strategy

![Refer to caption](https://arxiv.org/html/2607.29559v1/x10.png)Figure 9:Uniform vs\. entropy\-based query sampling \(MO\-Cheetah\)\.Ground\-truth return per objective for the default uniform sampler and an entropy\-based sampler that scores a10×10\\timescandidate pool by reward\-model uncertainty\. Shading is±1\\pm 1std; the entropy arm is a single seed and carries a nominal band\. Entropy\-based selection is consistently ahead, but by a small margin relative to the effect of query length \(Figure[8](https://arxiv.org/html/2607.29559#A3.F8)\)\.We also tested ifwhichsegments are queried matters at a fixed budget and length\. We compared LEMUR’s default uniform sampler against an entropy\-based sampler that selects the most uncertain segment pairs from a 10×\\timescandidate pool \(Figure[9](https://arxiv.org/html/2607.29559#A3.F9)\)\.

Entropy\-based selection yields a modest gain in performance\. Uniform sampling recovers most of the achievable return without requiring uncertainty estimates, candidate pools, or extra forward passes\. Combined with our query\-length findings, this shows that under a fixed budget,how much behavior each query coversmatters significantly more thanwhichspecific segments are chosen\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/x11.png)Figure 10:Overlapping preferences \(MO\-Cheetah\)\.With near\-aligned teacher anchors, LEMUR converges close to the ground\-truth oracle on both objectives, showing that the method degrades gracefully when teachers largely agree\. Mean±\\pmstd over five seeds\.
### C\.5Overlapping Preferences

Our main results consider teachers whose anchors are genuinely conflicting\. A natural question is whether the machinery required to resolve conflict imposes a cost when the teachers largely*agree*\. We therefore repeat the experiment with overlapping \(near\-aligned\) teacher weight vectors, comparing LEMUR against the ground\-truth oracle upper bound under the same overlap condition\.

Figure[10](https://arxiv.org/html/2607.29559#A3.F10)shows that LEMUR tracks the oracle closely on both objectives, converging to a comparable final return with only a modest lag in sample efficiency\. This indicates that the weight\-conditioned formulation degrades gracefully toward the single\-preference case, when there is little conflict to resolve\.

### C\.6Ablation: Varying Levels of Teacher Conflict

![Refer to caption](https://arxiv.org/html/2607.29559v1/x12.png)Figure 11:Varying levels of teacher conflict \(MO\-Cheetah\)\.Each column is one conflict level, set by the teacher preference anchors: medium \(w=\[0\.6,0\.4\]/\[0\.4,0\.6\]w=\[0\.6,0\.4\]/\[0\.4,0\.6\]\), hard \(\[0\.75,0\.25\]/\[0\.25,0\.75\]\[0\.75,0\.25\]/\[0\.25,0\.75\]\) and harder \(\[0\.9,0\.1\]/\[0\.1,0\.9\]\[0\.9,0\.1\]/\[0\.1,0\.9\]\)\. Rows give each objective’s ground\-truth return\. Solid lines are LEMUR, dashed the ground\-truth\-reward Oracle under the identical configuration; shading is±1\\pm 1std across seeds, with a commonyy\-scale per row\.This ablation isolates thedegreeof teacher disagreement\. HoldingMO\-Cheetah, the MORL/D backbone, explorer, and query budget fixed, we sweep the anchors from mildly to severely opposed and train LEMUR alongside an Oracle given the same anchors at each level\. Comparing against a per\-level Oracle separates degradation caused by the trade\-off becoming harder from degradation caused by reward learning failing under disagreement\. LEMUR remains competitive across the sweep \(Figure[11](https://arxiv.org/html/2607.29559#A3.F11)\)\.

### C\.7Ablation: Weight\-Conditioned Reward Model vs\. Reward Ensemble

![Refer to caption](https://arxiv.org/html/2607.29559v1/x13.png)Figure 12:Reward\-model ablation \(MO\-Cheetah\)\.The weight\-conditioned reward model substantially outperforms the reward\-ensemble variant on both objectives under an identical optimizer, explorer, and query budget\. Mean±\\pmstd over five seeds\.LEMUR’s central architectural choice is to condition each teacher’s reward model on a preference weight vector, rather than learning an ensemble of unconditioned reward models as in the earlier formulation\. To isolate the contribution of this choice we hold everything else fixed, the same MORL/D backbone, explorer, teacher anchors, and query budget, and vary only the reward model\.

Figure[12](https://arxiv.org/html/2607.29559#A3.F12)shows a substantial and consistent gap on both objectives: the weight\-conditioned model converges to6,812±396\{,\}812\\pm 39and4,404±224\{,\}404\\pm 22, against4,556±3694\{,\}556\\pm 369and2,902±2452\{,\}902\\pm 245for the ensemble\. The ensemble also exhibits an order\-of\-magnitude larger standard error, indicating greater seed\-to\-seed instability\.

## Appendix DLEMUR Implementation Details

Training Details\.We utilize the MORL/D algorithm\(Feltenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib22)\), a Multi\-Objective Soft Actor\-Critic \(MO\-SAC\) implementation from theMORL\-Generalizationbenchmark\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\)\. A summary of the hyperparameters is provided in Tables[3](https://arxiv.org/html/2607.29559#A4.T3)and[4](https://arxiv.org/html/2607.29559#A4.T4)\.

In the initialExplorationphase \(Stage 0\), an intrinsic\-motivation explorer bootstraps a replay buffer of environment transitions, run for 50,000 timesteps on LunarLander, 80,000 on Hopper, and 100,000 on Cheetah and MetaWorld\. This buffer is shared identically with every baseline, so no method receives more exploration data than another\.

In theReward Learningphase \(Stage 1\), each of themmteachers trains its ownweight\-conditionedreward model: a 2\-layer MLP with 256 hidden units that takes the state\-action pair*concatenated with a preference weight vector*𝐰\\mathbf\{w\}, trained by Bradley–Terry cross\-entropy over pairwise trajectory\-segment comparisons of lengthH=50H=50\. Rather than querying each teacher only at its own fixed anchor𝐰j\\mathbf\{w\}\_\{j\}, weights are sampled from a Dirichlet distribution centred on that anchor with concentrationκ=30\\kappa=30, so a single model generalises across a neighbourhood of the anchor instead of memorising one point on the simplex\. At each iteration of reward\-learning and policy\-optimization, the per\-teacher query budgets areM=200M=200\(LunarLander\),500500\(Hopper\), and300300\(Cheetah, MetaWorld\), each trained for 100 epochs with Adam optimizer\.

ForMulti\-Objective Policy Optimization\(Stages 2\-3\), a population of 6 MO\-SAC policies is trained on themm\-dimensional learned vector reward, coupled by a shared replay buffer, weighted\-sum scalarization, PSA weight adaptation, and weight transfer between neighbouring policies \(neighbourhood size 2\)\. Learning rates and exchange frequencies are adapted per environment \(Table[4](https://arxiv.org/html/2607.29559#A4.T4)\) while the population size is held constant at 6 across all environments\. Crucially, reward learning does not terminate after Stage 1: everyfeedback\_intervalsteps the pipeline re\-queries all teachers, performs 30 online reward\-update epochs, and*relabels the shared replay buffer*with the updated reward models, so the policy and the reward model co\-adapt throughout training\.

Table 3:Hyperparameters for Exploration \(Stage 0\) and Reward Learning \(Stage 1\)\.Table 4:Hyperparameters for Multi\-Objective Policy Optimization \(Stages 2–3\)\.HyperparameterValueHyperparameterValuePbMORL\(Muet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib585)\)Reward ModelWeight\-conditioned MLPHidden Dim256Query Budget \(MM\)300 \(total, split across teachers\)Query Length \(HH\)50Reward LR / Epochs2\.5​e\-​42\.5\\text\{e\-\}4/ 100Reward Batch Size64Optimizer \(policy\)Envelope \(LunarLander\), MORL/D \(Others\)Population Size6FPbRL\(Siddiqueet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib584)\)Reward ModelGGF welfare MLP \(KK\-dim\)Hidden Dim256Query Budget \(MM\) / Length300 / 50Reward LR / Epochs2\.5​e\-​42\.5\\text\{e\-\}4/ 100Policy AlgorithmPPO \(continuous\), SAC\-discrete \(LunarLander\)Policy LR3​e\-​43\\text\{e\-\}4PPO Steps / Iter2,048PPO Minibatches32PPO Update Epochs10PPO Clip Coef0\.2GAEλ\\lambda0\.95GGF WeightsDecreasing on sorted utilities

Table 5:Hyperparameters for the PbMORL and FPbRL baselines\.Table 6:MORAL\(Peschlet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib295)\): hyperparameters for expert demonstration collection and adversarial AIRL reward training\.Table 7:MORAL\(Peschlet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib295)\): hyperparameters for online policy optimization \(Stage 2\)\. Note we optimize with SAC rather than the original work’s PPO; see Appendix[E](https://arxiv.org/html/2607.29559#A5)\.
## Appendix EBaselines Implementation Details

To ensure a fair comparison, all baselines share LEMUR’s Stage\-0 explorer, scripted teacher weight vectors, query budgetMM, query length, interaction frequencyKK, reward\-model capacity, and total environment\-step budget, and are evaluated and logged under identical protocols \(Tables[3](https://arxiv.org/html/2607.29559#A4.T3)and[4](https://arxiv.org/html/2607.29559#A4.T4)\)\. The primary distinction between methods therefore lies in*how conflicting reward signals are aggregated and optimized*, not in the data or compute they receive\.

Each baseline retains its own defining reward\-learning mechanism: MORAL’s adversarial AIRL rewards, PbMORL’s weight\-conditioned Bradley–Terry model, and FPbRL’s GGF welfare model\. Where a method’s original policy optimizer would place it at an unfair disadvantage in our environments, we adapt*in the baseline’s favour*, upgrading the optimizer rather than reproducing the paper verbatim, so that reported gaps reflect differences in reward learning and preference aggregation, the object of study, rather than differences in policy optimization\. Most notably, MORAL is optimized with SAC rather than the PPO used in the original work, substantially improving its sample efficiency on our continuous\-control tasks\. All such deviations are disclosed below\.

#### Utilitarian Agent\.

This baseline imposes ana prioriscalarization on the objectives\. Like LEMUR, it learnsmmdistinct reward models\{r^ψj\}j=1m\\\{\\hat\{r\}\_\{\\psi\_\{j\}\}\\\}\_\{j=1\}^\{m\}, one per teacher, but rather than learning a set of trade\-off policies it trains asingleSAC agent to optimize their arithmetic mean:

rutil​\(s,a\)=1m​∑j=1mr^ψj​\(s,a\)\.r\_\{\\text\{util\}\}\(s,a\)=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\hat\{r\}\_\{\\psi\_\{j\}\}\(s,a\)\.\(3\)Learning rates, batch sizes, and buffer sizes are identical to LEMUR, with the population size set to11\(standard single\-objective SAC\)\.

#### Naive Data Pooling\.

This baseline aggregates conflicting preferences at the*data*level, mimicking standard RLHF applied to heterogeneous feedback\. Rather than maintaining separate datasets𝒟j\\mathcal\{D\}\_\{j\}per objective, all feedback tuples are stored in a single monolithic dataset𝒟pool=⋃j=1m𝒟j\\mathcal\{D\}\_\{\\text\{pool\}\}=\\bigcup\_\{j=1\}^\{m\}\\mathcal\{D\}\_\{j\}, over which one reward modelr^pool\\hat\{r\}\_\{\\text\{pool\}\}is trained to minimize the cross\-entropy loss\. The policy is then trained with standard SAC to maximize

rnaive​\(s,a\)=r^pool​\(s,a\)\.r\_\{\\text\{naive\}\}\(s,a\)=\\hat\{r\}\_\{\\text\{pool\}\}\(s,a\)\.\(4\)As with the Utilitarian baseline, we use the environment\-specific hyperparameters in Table[4](https://arxiv.org/html/2607.29559#A4.T4)with the population size fixed to11\.

#### PbMORL\(Muet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib585)\)\.

The original Pb\-MORL assumes a single, internally\-consistent teacher whose preferences are valid under any sampled scalarization weight, and therefore has no mechanism for multiple, independently\-opinionated teachers\. We retain its weight\-conditionedmm\-dimensional reward model, trained with the Bradley–Terry cross\-entropy objective over pairwise trajectory comparisons, and extend it to our setting in the two most natural ways:*PbMORL\-naive*pools all teachers’ preferences into one shared weight\-conditioned model, while*PbMORL\-utilitarian*trains a separate model per teacher and combines them at inference by a fixed uniform average\. Each teacher answers every query under its own fixed weight vector, exactly as in LEMUR\.

The original paper pairs its reward model with Envelope Q\-learning\(Yanget al\.[2019](https://arxiv.org/html/2607.29559#bib.bib591)\), a discrete, value\-based method\. We retain this paper\-faithful pairing on LunarLander, but Envelope’s Q\-network requires an argmax over actions and is structurally inapplicable to continuous control; on Hopper, HalfCheetah, and MetaWorld we therefore substitute MORL/D\(Feltenet al\.[2024](https://arxiv.org/html/2607.29559#bib.bib22)\), the same population\-based optimizer LEMUR uses\. This is the only structural substitution, and it equalizes the optimizer between PbMORL and LEMUR on those environments, isolating the reward\-learning strategy as the sole difference\.

#### FPbRL\(Siddiqueet al\.[2023](https://arxiv.org/html/2607.29559#bib.bib584)\)\.

We implement FPbRL’sKK\-dimensional welfare reward model, trained from Generalized Gini Welfare \(GGF\)\-based preferences over the sameK=2K=2scripted teachers across which LEMUR and the other baselines compromise\. At each policy\-update iteration, the learnedKK\-dimensional reward is scalarized by the GGF weight assignment, which places the largest weight on the currently worst\-off objective, refreshed from a running per\-objective return estimate, the standard practical form of Siddique et al\.’s fair policy gradient\.

Unlike MORAL, we retain PPO for continuous\-action environments, matching the original paper: FPbRL’s fair policy gradient is formulated for an on\-policy optimizer, and substituting an off\-policy method would alter the method’s semantics rather than simply strengthen it\. On LunarLander we use SAC\-discrete, as no discrete PPO implementation is used elsewhere in our pipeline\. In addition to the per\-teacher returns reported for all methods, we log FPbRL’s own fairness metrics \(welfare, coefficient of variation, and minimum objective\), so that it is also evaluated on the criterion it is explicitly designed to optimize\.

#### MORAL\(Peschlet al\.[2022](https://arxiv.org/html/2607.29559#bib.bib295)\)\.

We reimplement MORAL’s pipeline in three stages: \(i\) two expert policies are trained on the ground\-truth per\-teacher scalarized reward and used to collect demonstrations; \(ii\) for each teacher, an AIRL\(Fuet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib541)\)discriminator is trained*adversarially*against a live generator, rather than against a fixed pool of shuffled expert transitions, which would render training non\-adversarial, yielding a per\-teacher learned rewardg​\(s\)g\(s\); and \(iii\) an active\-learning wrapper scalarizes the two frozen AIRL rewards with a Bradley\-Terry weight posterior, updated online via volume\-removal preference queries, which a single\-objective policy optimizes\. Since the original MORAL was evaluated only on grid\-world tasks, we adopt the AIRL hyperparameters and network architectures of Fu et al\.\(Fuet al\.[2018](https://arxiv.org/html/2607.29559#bib.bib541)\)for our high\-dimensional MuJoCo experiments; these are detailed in Tables[6](https://arxiv.org/html/2607.29559#A4.T6)and[7](https://arxiv.org/html/2607.29559#A4.T7)\.

We deliberately preserve MORAL’s defining contribution, adversarial IRL reward learning combined with an actively\-queried scalarization posterior, and do not replace it with our own preference\-based reward model, as doing so would no longer constitute a MORAL baseline\. We do, however, strengthen its policy optimization: the original work uses PPO, whereas we optimize with SAC \(SAC\-continuous, or SAC\-discrete on LunarLander\) across all environments\. This off\-policy upgrade materially improves MORAL’s sample efficiency under an identical environment\-step budget and matches the backbone family used by LEMUR, ensuring MORAL is not penalized for an on\-policy optimizer choice unrelated to its reward\-learning contribution\.

### E\.1MORAL Baseline: Learned Weights and Optimiser Choice

#### Learned scalarisation weights\.

MORAL maintains a posterior over a scalarisation weight vector, refined through active queries, and it is this posterior, rather than a per\-teacher reward decomposition, that carries its notion of whose preferences the policy is serving\. Figure[13](https://arxiv.org/html/2607.29559#A5.F13)tracks both components over training\.

#### SAC vs\. PPO\.

As described in Appendix[E](https://arxiv.org/html/2607.29559#A5), we optimize the MORAL baseline with SAC rather than the PPO used in the original work, on the grounds that an on\-policy optimizer would disadvantage the baseline for reasons unrelated to its reward\-learning contribution\. This ablation verifies that the substitution is genuinely favourable to MORAL and therefore that our reported comparison is conservative\.

Figure[14](https://arxiv.org/html/2607.29559#A5.F14)confirms this: holding MORAL’s adversarial AIRL reward learning and active\-query scalarisation posterior fixed and varying only the policy optimizer, the SAC variant dominates PPO on both objectives throughout training, and the gap widens as training proceeds\. The PPO variant additionally displays a pronounced sawtooth characteristic of on\-policy updates \(shown here under heavy smoothing\)\. Reporting MORAL with SAC therefore strengthens the baseline relative to a faithful reproduction of the original paper, and the weight collapse in Figure[13](https://arxiv.org/html/2607.29559#A5.F13)is a property of MORAL’s scalarization posterior rather than a symptom of the optimizer\.

![Refer to caption](https://arxiv.org/html/2607.29559v1/x14.png)Figure 13:MORAL’s learned scalarisation weights in \(MO\-Cheetah\)\.The two components of MORAL’s active\-query scalarisation posterior over training\.![Refer to caption](https://arxiv.org/html/2607.29559v1/x15.png)Figure 14:MORAL optimizer ablation \(MO\-Cheetah\)\.With MORAL’s reward learning held fixed, SAC outperforms the original work’s PPO on both objectives, confirming that our SAC upgrade strengthens the baseline\. Curves are heavily smoothed to expose the trend beneath PPO’s on\-policy oscillation\. Mean±\\pmstd over five seeds\.

## Appendix FAdditional Results

### F\.1Multi\-Objective RL Metrics

Evaluating a multi\-objective agent requires assessing the*set*of policies it recovers rather than any single return, so we adopt two metrics standard in the MORL literature\(Hayeset al\.[2022a](https://arxiv.org/html/2607.29559#bib.bib524); Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525)\)\. Let𝒫=\{J​\(π1\),…,J​\(πP\)\}\\mathcal\{P\}=\\\{J\(\\pi\_\{1\}\),\\dots,J\(\\pi\_\{P\}\)\\\}denote the set of objective\-value vectors attained by the learned policy population\.

#### Hypervolume \(HV\)\.

Given a reference point𝐳\\mathbf\{z\}dominated by all solutions, Hypervolume is the volume of the region dominated by𝒫\\mathcal\{P\}and bounded by𝐳\\mathbf\{z\},

HV​\(𝒫,𝐳\)=Λ​\(⋃𝐩∈𝒫\[𝐳,𝐩\]\),\\mathrm\{HV\}\(\\mathcal\{P\},\\mathbf\{z\}\)=\\Lambda\\\!\\left\(\\bigcup\_\{\\mathbf\{p\}\\in\\mathcal\{P\}\}\[\\mathbf\{z\},\\mathbf\{p\}\]\\right\),\(5\)whereΛ\\Lambdais the Lebesgue measure\. It simultaneously rewards solutions that are high\-performing \(pushing the front outward\) and diverse \(covering more of the objective space\), and is the most widely adopted MORL quality indicator because it is the only common metric strictly monotonic with Pareto dominance: any set that dominates another is guaranteed a higher score\(Roijerset al\.[2013](https://arxiv.org/html/2607.29559#bib.bib525)\)\. We use the reference point supplied by theMORL\-Generalizationbenchmark\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\)so that values are comparable across methods within an environment; absolute magnitudes are not comparable*across*environments, since they depend on both the reference point and the reward scale\.

#### Sparsity \(SPS\)\.

Sparsity measures how evenly solutions are distributed along the recovered front\. Sorting the\|𝒫\|\|\\mathcal\{P\}\|solutions by each objectiveiiand writingP~i​\(k\)\\tilde\{P\}\_\{i\}\(k\)for thekk\-th value,

SPS​\(𝒫\)=1\|𝒫\|−1​∑i=1m∑k=1\|𝒫\|−1\(P~i​\(k\)−P~i​\(k\+1\)\)2,\\mathrm\{SPS\}\(\\mathcal\{P\}\)=\\frac\{1\}\{\|\\mathcal\{P\}\|\-1\}\\sum\_\{i=1\}^\{m\}\\sum\_\{k=1\}^\{\|\\mathcal\{P\}\|\-1\}\\left\(\\tilde\{P\}\_\{i\}\(k\)\-\\tilde\{P\}\_\{i\}\(k\+1\)\\right\)^\{2\},\(6\)with lower values indicating more uniform coverage\(Xuet al\.[2020](https://arxiv.org/html/2607.29559#bib.bib5)\)\. Sparsity must be read alongside Hypervolume rather than independently: a degenerate front that collapses to a single solution reports a low, apparently favourable value despite failing to cover the objective space, which is why we report both\. This is the case for FPbRL, whose fixed welfare scalarization converges to a single policy, so no front is recovered and sparsity is undefined \(−\-\) on three of four environments\.

Table 8:Hypervolume and Sparsity across benchmark environments\. FPbRL’s fixed welfare scalarization fails to recover a set of trade\-off policies, converging instead to a single solution: sparsity is therefore undefined \(−\-\) where no front exists, and its near\-zero value on MetaWorld\-DrawerClose reflects this collapse rather than uniform coverage\. PbMORL is deterministic under our protocol on Hopper and HalfCheetah, hence zero variance\.

### F\.2Reward Model Evaluation Metrics

Policy return alone cannot distinguish a reward model that has genuinely recovered a teacher’s utility from one merely correlated with it on the visited state distribution\. We therefore evaluate the learned reward models directly against the ground\-truth teacher utilities, following PbRL benchmarking practice\(Leeet al\.[2021a](https://arxiv.org/html/2607.29559#bib.bib586)\)\. Letr^ψj\\hat\{r\}\_\{\\psi\_\{j\}\}denote teacherjj’s learned reward model andrj​\(s,a\)=𝐰j⊤​𝐫​\(s,a\)r\_\{j\}\(s,a\)=\\mathbf\{w\}\_\{j\}^\{\\top\}\\mathbf\{r\}\(s,a\)its ground\-truth utility, where𝐫\\mathbf\{r\}is the environment’s native vector reward and𝐰j\\mathbf\{w\}\_\{j\}teacherjj’s scripted weight vector\. All metrics lie in\[−1,1\]\[\-1,1\], higher is better, and are averaged across themmteachers\.

#### Per\-state correlation\.

Over states sampled from evaluation rollouts,*Spearman*rank correlation measures how faithfully the learned reward orders individual transitions\. It is our primary metric because preference\-based rewards are identifiable only up to a positive monotone transform, making an order\-preserving measure the appropriate notion of correctness\. We also report*Pearson*correlation on raw values, which is stricter in penalising any nonlinear distortion; following prior reward\-evaluation work we refer to this as the*Value\-Order Correlation*\(VOC\)\(Maet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib595)\)\.

#### Trajectory and policy ranking\.

Per\-state correlation can be high even when a reward model induces the wrong ordering over whole*behaviours*, which is what the policy ultimately optimizes\. We therefore roll out each policy in the MORL/D population and compare the induced rankings using Kendall’sτ\\tau\-b over trajectory returns, and the*Trajectory Alignment Coefficient*\(TAC\) of\(Muslimaniet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib593)\), computed identically but on*discounted*returns so that alignment is weighted by the same temporal discounting the agent optimizes\.*Trajectory VOC*additionally averages the within\-trajectory correlation between learned and ground\-truth per\-step reward sequences, capturing whether the model tracks the shape of the signal within an episode rather than only across episodes\.

#### Results\.

Tables[9](https://arxiv.org/html/2607.29559#A6.T9)\-[11](https://arxiv.org/html/2607.29559#A6.T11)report all metrics\. LEMUR attains the strongest per\-state correlations on every environment, with the largest margins on Hopper and MetaWorld\-DrawerClose where the teachers’ anchors are most opposed, consistent with per\-teacher decomposition mattering most under genuine conflict\. Trajectory\-level metrics are more mixed: FPbRL attains a higher Kendallτ\\tauand TAC on Hopper despite substantially weaker per\-state correlation, reflecting that its welfare scalarization orders whole behaviours consistently even where the underlying reward is poorly calibrated\. Reporting both families is what makes this distinction visible, and we recommend the same practice for future work in this setting\.

Table 9:Reward\-Model Evaluation Metrics : HalfCheetah\.Table 10:Reward\-Model Evaluation Metrics: MetaWorld\-DrawerClose\.Table 11:Reward\-Model Evaluation Metrics: Hopper\.

## Appendix GBenchmark Environment Details

Table[12](https://arxiv.org/html/2607.29559#A7.T12)summarises the native multi\-objective structure of each benchmark environment and the scripted teacher weight vectors used throughout our experiments\. In every case, teacherjj’s ground\-truth utility is the linear scalarisationrj​\(s,a\)=𝐰j⊤​𝐫​\(s,a\)r\_\{j\}\(s,a\)=\\mathbf\{w\}\_\{j\}^\{\\top\}\\mathbf\{r\}\(s,a\)of the environment’s native vector reward𝐫\\mathbf\{r\}, and it is this quantity thatvalidation/gt\_teacher\_\{a,b\}\_returntracks during training\. The weight vectors are chosen to be genuinely conflicting: each teacher places its largest weight on a different objective, so no single policy can simultaneously maximise both utilities\.

#### MO\-LunarLander\.

A four\-objective variant of the classic LunarLander domain with a discrete action space, taken from the MORL\-Generalization benchmark\(Teohet al\.[2025](https://arxiv.org/html/2607.29559#bib.bib323)\)\. The native reward vector comprises the shaping term \(progress toward the landing pad\), the main\-engine fuel cost, the side\-engine fuel cost, and the terminal landing/crash outcome\. Teacher A weights the two engine\-cost terms asymmetrically against Teacher B, producing a fuel\-allocation conflict on top of a shared landing objective\.

#### MO\-Hopper\.

A three\-objective continuous\-control locomotion task in which the native reward vector is\[vx,h,−c​‖a‖2\]\[\\,v\_\{x\},\\ h,\\ \-c\\,\\\|a\\\|^\{2\}\\,\]: forward velocity, hop height, and a negated energy/control cost\. Teacher A \(\[0\.8,0\.1,0\.1\]\[0\.8,0\.1,0\.1\]\) strongly prefers fast locomotion, while Teacher B \(\[0\.3,0\.5,0\.2\]\[0\.3,0\.5,0\.2\]\) prefers a higher, more energy\-efficient and stable gait\.

#### MO\-Cheetah\.

A two\-objective continuous\-control task whose native reward vector is\[reward\_forward,reward\_ctrl\]\[\\,\\texttt\{reward\\\_forward\},\\ \\texttt\{reward\\\_ctrl\}\\,\], i\.e\. forward velocity against control cost\. The teacher anchors\[0\.6,0\.4\]\[0\.6,0\.4\]and\[0\.4,0\.6\]\[0\.4,0\.6\]place opposing emphasis on velocity versus energy efficiency\.

#### MO\-MetaWorld

\(Drawer\-Close\)\. Meta\-World tasks are natively*single*\-objective, providing only a dense task\-progress reward\. We convertDrawer\-Closeinto a two\-objective task by pairing this native reward with a control\-effort penalty, yielding the vector reward\[rtask,−λ​‖a‖22\]\[\\,r\_\{\\text\{task\}\},\\ \-\\lambda\\\|a\\\|\_\{2\}^\{2\}\\,\]with control\-cost weightλ=0\.1\\lambda=0\.1and a fixed horizon of 500 steps\. This mirrors the forward\-reward/control\-cost decomposition standard to the MuJoCo suite, but applied to a more complex robot\-manipulation domain\. Teacher A is rewarded by task completion, Teacher B by smooth, energy\-efficient actuation\.

Table 12:Benchmark environments: native objective decomposition and scripted teacher weight vectors\.mmdenotes the dimensionality of the native vector reward\.

## Appendix HCompute Resources

In all experiments, we use 12 CPUs and a single GPU, of type either NVIDIA A100 or L40\. Training in all environments takes approximately three to five hours on average\.