多目标强化学习中后见重标注所致Preference Coverage Collapse研究

# On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning
Source: [https://arxiv.org/html/2609.26918](https://arxiv.org/html/2609.26918)
###### Abstract

> Hindsight relabeling which retroactively replacing a transition’s goal with the outcome the agent actually achieved is an effective tool for improving sample\-efficiency in Reinforcement Learning \(RL\)\. A natural extension to preference\-conditioned multi\-objective RL \(MORL\) relabels transitions with the preference direction the agent achieved rather than the one asked for\. We show that this extension is frequently harmful: across four preference\-conditioned off\-policy algorithms spanning two critic backbones and two preference\-sampling schemes on the continuous\-control MO\-Gymnasium suite, it degrades 19 of 36 algorithm\-environment settings by as much as four standard deviations, improves only one, and leaves the rest unaffected\. The harm is not a symptom of noisy relabels; denoising the target recovers almost nothing, and neither prioritized sampling nor any buffer\-structural choice reproduces it\. Instead, repeated relabeling collapses the critic’s coverage onto whatever narrow region of the preference space the agent happened to visit\. We name this failure mode*Preference Coverage Collapse*, and quantify it with abandoned preference mass \(APM\), a value\-aware statistic that tracks the harm \(ρ=−0\.73\\rho=\-0\.73\) where a purely structural coverage count does not\. We then introduceher\_mix, a single\-parameter convex combination pulling the achieved direction back towards the requested preference\. At one fixed value across every algorithm and environment, it returns 16 of the 19 harmed settings to baseline, preserves and even improves the one setting in which relabeling helps, and cuts abandoned preference mass from69%69\\%to6%6\\%\. Protecting coverage over the preference simplex, not filtering noisy relabels, is what makes hindsight relabeling safe for MORL\.

1Mila \(Quebec AI Institute\)2Université Laval3CIFAR AI Chair

## 1Introduction

Although Reinforcement Learning \(RL\) has been shown to be effective across a range of reward\-driven problems\([Mnih et al\. 2015](https://arxiv.org/html/2609.26918#bib.bib19);[Duan et al\. 2016](https://arxiv.org/html/2609.26918#bib.bib20);[Haarnoja et al\. 2018](https://arxiv.org/html/2609.26918#bib.bib8)\), many RL algorithms require large amounts of training data and remain especially sample\-inefficient in the presence of sparse or delayed rewards\. In Multi\-Objective Reinforcement Learning \(MORL\), this inefficiency is compounded\. Rather than learning a single behavior, a preference\-conditioned agent must learn to cover an entire space of trade\-offs from the same finite experience pool, effectively multiplying the amount it needs to learn without a corresponding increase in data\([Yang et al\. 2019](https://arxiv.org/html/2609.26918#bib.bib16);[Abels et al\. 2019](https://arxiv.org/html/2609.26918#bib.bib17)\)\. Hindsight Experience Replay \(HER\)\([Andrychowicz et al\. 2017](https://arxiv.org/html/2609.26918#bib.bib1)\), introduced for sparse\-reward goal\-conditioned RL, is among the most effective remedies\. Rather than discarding a trajectory that fails to reach its assigned goal, HER relabels it with the outcome the agent actually achieved and retrains on it, turning failures into usable learning signal\. The substitution costs nothing because a goal and an achieved outcome are the same kind of object : whatever the agent reached is itself a perfectly valid goal in hindsight\.

The same principle admits an analogue in preference\-conditioned MORL, where the policy is conditioned on a preference vector rather than a goal\. Instead of substituting an achieved state for the goal, the agent substitutes the trade\-off it actually achieved for the preference it was given\. This substitution has already been adopted in prior work\([Basaklar et al\. 2022](https://arxiv.org/html/2609.26918#bib.bib3);[Shianifar et al\. 2026](https://arxiv.org/html/2609.26918#bib.bib4)\)\(discussed in Section[3\.2](https://arxiv.org/html/2609.26918#S3.SS2)\), justified by its relationship to HER rather than by independent analysis\.

The case for relabeling in preference\-conditioned MORL is, at first glance, strong: relabeling promises to multiply off\-policy data for free, requiring no additional environment interaction and no architectural changes\. Yet, the precondition behind HER’s substitution does not obviously carry over\. Rewards in MORL are typically dense and continuous\-valued, and not every objective contains a natural notion of*achievement*in the way a goal is achieved \(a cost objective, for instance, is never reached so much as minimized\)\. Consider a hopper agent asked to trade off forward speed against energy \(control costs\) that falls over after some small number of timesteps\. The trade\-off it “achieved” is whatever brief forward shuffle it managed before collapsing, which says little about the speed versus energy tradeoff it was asked for\. Relabeling nonetheless trains the critic to treat that shuffle as the preference the agent wanted, and repeating this across many short episodes teaches the critic to value only the narrow corner it happened to reach \(Figure[1](https://arxiv.org/html/2609.26918#S1.F1)\)\. The technique adopted to fill an agent’s coverage of the preference space instead collapses it\. A failure mode we name*Preference Coverage Collapse*\(PCC\)\.

Whether the substitution remains well\-defined once these structural differences are accounted for has not been directly tested\. As we show, the answer determines not only whether relabeling helps or hinders learning in MORL, but how it can be made safe when it doesn’t\. We therefore ask:

“does HER\-style preference relabeling improve off\-policy MORL, where does it fail, why, and can it be repaired?”

The answer is sharper than current practice assumes: relabeling is harmful in the majority of settings where it has any effect at all\. Our primary contributions are as follows:

1. 1\.An empirical study of preference relabeling in off\-policy MORL\.We provide the first controlled evaluation of whether HER\-style preference relabeling helps or harms off\-policy MORL performance across four preference\-conditioned algorithms spanning two critic backbones and two preference\-sampling schemes\. It degrades 19 of 36 algorithm\-environment settings by as much as four standard deviations and improves exactly one\.
2. 2\.A hidden failure mode in relabeling design, and its resolution\.We identify a clipping artifact in the standard relabeling scheme that causes it to silently ignore cost objectives, and show that a sign\-robust fix resolves most of the apparent harm previously attributed to relabeling on cost\-objective environments\.
3. 3\.A mechanism for the harm, and a value\-sensitive diagnostic for it\.We show that the remaining harm is not explained by noisy relabeling targets, prioritization, or buffer structure, but rather by a collapse in the critic’s coverage over the preference simplex\. We introduce Abandoned Preference Mass \(APM\), which tracks the harm far more tightly \(ρ=−0\.73\\rho=\-0\.73\) than a purely structural count of the trade\-offs a policy offers \(\+0\.40\+0\.40\), because it is value\-aware\.
4. 4\.A fix that generalizes without tuning\.We introduceher\_mix, a single\-parameter interpolation between the achieved and requested preference\. Rolled out untuned at one fixed coefficient across all 36 settings\. It recovers 16 of the 19 harmed settings, preserves and amplifies the one setting where relabeling helps and leaves the neutral settings unaffected\.
5. 5\.A negative result on replay\-buffer structure\.We show that storage, partitioning, eviction, and capacity choices produce no harm at 16\-seed power, isolating relabeling as the buffer design axis that degrades performance\.

![Refer to caption](https://arxiv.org/html/2609.26918v1/hopper_intro.png)Figure 1:A single training episode collapses the preferences a policy trains on\.Ahopperagent falls within a few dozen steps; relabeling substitutes that narrow achieved trade\-off for every preference requested during the episode\. Repeated, the critic sees a cluster of achieved directions instead of the full requested range\.
## 2Background

### 2\.1Reinforcement Learning

We use the standard RL formulation\([Sutton et al\. 1998](https://arxiv.org/html/2609.26918#bib.bib5)\): a fully observable environment\(𝒮,𝒜,r,p,γ\)\(\\mathcal\{S\},\\mathcal\{A\},r,p,\\gamma\)withr:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}andγ∈\[0,1\)\\gamma\\in\[0,1\), a policyπ⁡\(a\|s\)\\pi\(a\|s\), discounted returnGtG\_\{t\}, and action\-value functionQπ\(s,a\)=𝔼π\[Gt∣st=s,at=a\]Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\[G\_\{t\}\\mid s\_\{t\}=s,a\_\{t\}=a\]\.

### 2\.2Multi\-Objective Reinforcement Learning

In MORL, the scalar reward is replaced by a vector\-valued reward function𝐫:𝒮×𝒜→ℝd\\mathbf\{r\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}^\{d\}, whereddis the number of objectives\. MORL algorithms can be broadly divided into multi\-policy approaches, which learn a set of policies to jointly approximate the Pareto front, and single\-policy \(preference\-conditioned\) approaches, which learn a single policy that adapts its behavior at inference time based on a preference vector\([Roijers et al\. 2013](https://arxiv.org/html/2609.26918#bib.bib18)\)\. We focus on the latter\.

The agent is conditioned on a preference vector𝐰∈𝒲\\mathbf\{w\}\\in\\mathcal\{W\}, typically constrained to the\(d−1\)\(d\-1\)\-simplex specifying a relative weighting over objectives\. Behavior is evaluated by a scalarized return, commonly the linear scalarization𝐰⊤​Gt\\mathbf\{w\}^\{\\top\}G\_\{t\}of the vector\-valued discounted returnGt∈ℝdG\_\{t\}\\in\\mathbb\{R\}^\{d\}, yielding the preference\-conditioned action\-value functionQπ\(s,a,𝐰\)=𝔼π\[𝐰⊤Gt∣st=s,at=a\]\.Q^\{\\pi\}\(s,a,\\mathbf\{w\}\)=\\mathbb\{E\}\_\{\\pi\}\\big\[\\mathbf\{w\}^\{\\top\}G\_\{t\}\\mid s\_\{t\}=s,a\_\{t\}=a\\big\]\.

### 2\.3Evaluation Metrics

Because MORL algorithms are generally evaluated on how well they recover the Pareto front rather than a single scalar return\([Hayes et al\. 2022](https://arxiv.org/html/2609.26918#bib.bib7)\), we adopt hypervolume and expected utility as our primary evaluation metrics\.

Hypervolume \(HV\)\.measures the volume of objective space dominated by a set of achieved returns relative to a reference point𝐫ref\\mathbf\{r\}\_\{\\text\{ref\}\}, rewarding both the extent and spread of the recovered front; it is, however, sensitive to choice of𝐫ref\\mathbf\{r\}\_\{\\text\{ref\}\}\.

Expected Utility Metric \(EUM\)\.Is the expected scalarized return under a preference distributionp⁡\(𝐰\)p\(\\mathbf\{w\}\)\([Zintgraf et al\. 2015](https://arxiv.org/html/2609.26918#bib.bib6);[Hayes et al\. 2022](https://arxiv.org/html/2609.26918#bib.bib7)\)\.EUM=𝔼𝐰∼p⁡\(𝐰\)​\[𝐰⊤​Gπ𝐰\],\\text\{EUM\}=\\mathbb\{E\}\_\{\\mathbf\{w\}\\sim p\(\\mathbf\{w\}\)\}\\left\[\\,\\mathbf\{w\}^\{\\top\}G^\{\\pi\_\{\\mathbf\{w\}\}\}\\,\\right\],whereGπ𝐰G^\{\\pi\_\{\\mathbf\{w\}\}\}is the return achieved by the policy conditioned on𝐰\\mathbf\{w\}\. Unlike HV, EUM requires no reference point𝐫ref\\mathbf\{r\}\_\{\\text\{ref\}\}and scales directly with the objectives the agent is optimizing for under each sampled preference\.

Effect size\.We report standardized effect sizes as Cohen’sdd\([Cohen 2013](https://arxiv.org/html/2609.26918#bib.bib24)\), the difference in means between an intervention and baseline divided by their pooled standard deviation,d=x¯treat−x¯basespooled\.d=\\frac\{\\bar\{x\}\_\{\\text\{treat\}\}\-\\bar\{x\}\_\{\\text\{base\}\}\}\{s\_\{\\text\{pooled\}\}\}\.

### 2\.4Replay Buffers and Prioritized Experience Replay

Off\-policy actor\-critic algorithms store past transitions in a*replay buffer*𝒟\\mathcal\{D\}, a set of tuples\(s,a,𝐫,s′,𝐰\)\(s,a,\\mathbf\{r\},s^\{\\prime\},\\mathbf\{w\}\)collected from interaction with the environment, and train by repeatedly sampling minibatches from𝒟\\mathcal\{D\}rather than using only recent experience\. The standard buffer samples transitions uniformly at random, implicitly treating each stored transition as equally useful for learning\([Lillicrap et al\. 2019](https://arxiv.org/html/2609.26918#bib.bib11)\)\.

Prioritized Experience Replay \(PER\)\([Schaul et al\. 2015](https://arxiv.org/html/2609.26918#bib.bib12)\)relaxes this by sampling transitions with probability proportional to\|δi\|α\|\\delta\_\{i\}\|^\{\\alpha\}, whereδi\\delta\_\{i\}is the TD error andα\\alphacontrols the strength of prioritization\. It is important to note that PER changes only how often a transition is sampled, leaving the transition’s content completely unmodified\.

### 2\.5Hindsight Experience Replay

Whereas PER changes how*often*a transition is sampled, HER changes what a transition*says*\. HER was introduced for goal\-conditioned RL under sparse, binary rewards,r\(s,a,g\)=𝟙\[∥𝕤′−𝕘∥<ϵ\]r\(s,a,g\)=\\mathbbold\{1\}\[\\\|s^\{\\prime\}\-g\\\|<\\epsilon\]: an agent rarely reaches its assigned goal early in training, so most episodes yield an uninformative zero reward regardless of how transitions are sampled or prioritized\. HER addresses this by relabeling a failed transition’s goal post hoc\. Given the transition\(s,a,s′,g\)\(s,a,s^\{\\prime\},g\), it additionally stores\(s,a,s′,g′\)\(s,a,s^\{\\prime\},g^\{\\prime\}\)for some stateg′g^\{\\prime\}achieved later in the same trajectory, turning a failure in a successful and informative transition\.

This substitution is valid because a goal and an achieved outcome are both elements of𝒮\\mathcal\{S\}, and success is defined purely by proximity to whichever one is supplied\. Thus, swappingggforg′g^\{\\prime\}leaves the reward function’s definition intact\. This property, rather than the relabeling procedure itself, is what our results explore once the same substitution is applied to preference vectors𝐰\\mathbf\{w\}in place of goalsgg\.

### 2\.6Preference Relabeling in MORL

Extending HER’s substitution to the preference\-conditioned setting means replacing the relabeled goalg′g^\{\\prime\}with a relabeled preference𝐰′\\mathbf\{w\}^\{\\prime\}\. For a stored transition\(s,a,𝐫,s′,𝐰\)\(s,a,\\mathbf\{r\},s^\{\\prime\},\\mathbf\{w\}\), the natural parallel to the “the goal actually reached” is the objective vector the agent actually achieved along the trajectory \(the per\-step reward vector𝐫\\mathbf\{r\}, or an aggregate such as return\-to\-go\)\. We refer to this achieved objective vector as𝐨^\\hat\{\\mathbf\{o\}\}\. The default operator,her\_achieved, maps𝐨^\\hat\{\\mathbf\{o\}\}onto the\(d−1\)\(d\-1\)\-simplex𝒲\\mathcal\{W\}via𝐰′=normalize​\(clip​\(𝐨^,0,∞\)\)\\mathbf\{w\}^\{\\prime\}=\\text\{normalize\}\(\\text\{clip\}\(\\hat\{\\mathbf\{o\}\},0,\\infty\)\),normalize​\(𝐱\)=𝐱/∑ixi\\text\{normalize\}\(\\mathbf\{x\}\)=\\mathbf\{x\}/\\sum\_\{i\}x\_\{i\}, clipping negative entries to zero so𝐰′\\mathbf\{w\}^\{\\prime\}is valid; the relabeled transition\(s,a,𝐫,s′,𝐰′\)\(s,a,\\mathbf\{r\},s^\{\\prime\},\\mathbf\{w\}^\{\\prime\}\)is stored alongside the original\.

This differs from HER’s original setting in two ways\. First, HER solves a sparse\-reward problem, whereas MORL’s scalarized reward𝐰⊤​𝐫\\mathbf\{w\}^\{\\top\}\\mathbf\{r\}is generically nonzero for any𝐰\\mathbf\{w\}: the motivation here is data reuse, not signal recovery\. Second, and more consequentially, HER’s validity argument \(Section[2\.5](https://arxiv.org/html/2609.26918#S2.SS5)\) does not transfer\. It relied on success being defined by proximity to whichever goal is supplied, which makes any achieved state an automatically valid stand\-in\. A preference is never “reached” in that sense:𝐨^\\hat\{\\mathbf\{o\}\}is a noisy trade\-off realized on one trajectory, not an outcome the agent aimed for, and nothing guarantees that𝐰′\\mathbf\{w\}^\{\\prime\}corresponds to a trajectory the agent would produce if it optimized for𝐰′\\mathbf\{w\}^\{\\prime\}from the start\.

The clip is also silently ill\-posed for objectives that are negative by construction: ifo^i<0\\hat\{o\}\_\{i\}<0on effectively every transition thenclip​\(o^i,0,∞\)=0\\text\{clip\}\(\\hat\{o\}\_\{i\},0,\\infty\)=0always, and𝐰′\\mathbf\{w\}^\{\\prime\}can never place weight on objectiveiiwhatever the trajectory’s true trade\-off\. Section[6](https://arxiv.org/html/2609.26918#S6)quantifies this and evaluates a sign\-robust alternative,her\_scaled, based on per\-objective min\-max normalization\.

## 3Related Work

### 3\.1HER in single\-objective RL

As discussed in Section[2](https://arxiv.org/html/2609.26918#S2), HER relabels failed goal\-conditioned trajectories with achieved outcomes; the original paper found that relabeling toward states from the trajectory’s near future consistently outperformed alternative relabeling targets, which the authors attributed to the relevance of near\-future states as possible goals\([Andrychowicz et al\. 2017](https://arxiv.org/html/2609.26918#bib.bib1)\)\. Subsequent work has continued to treat the relabeling target as a nontrivial design choice\. HER is known to perform poorly when the assigned goal is far from the agent’s initial state\([Ren et al\. 2019](https://arxiv.org/html/2609.26918#bib.bib14)\), motivating curriculum\-based extensions that prioritize intermediate goals\([Fang et al\. 2019](https://arxiv.org/html/2609.26918#bib.bib13)\)\. Even where the achieved\-outcome substitution is valid by construction,*which*outcome to relabel toward, and under what conditions, impacts its benefit\. As we discuss, this scrutiny has not carried over to MORL’s adoption of the same substitution\.

### 3\.2HER\-style relabeling applied to MORL

The single\-objective substitution principle has been carried over to preference\-conditioned MORL\. PD\-MORL\([Basaklar et al\. 2022](https://arxiv.org/html/2609.26918#bib.bib3)\)adopts a HER\-inspired buffer for both its MO\-DDQN\-HER and MO\-TD3\-HER variants, explicitly following\([Andrychowicz et al\. 2017](https://arxiv.org/html/2609.26918#bib.bib1)\); however, its relabeling draws preferences independent of the trajectory, rather than substituting the achieved trade\-off itself, and as such does not test the substitution HER’s original motivation actually implies\. More recently, Hindsight Preference Replay \(HPR\)\([Shianifar et al\. 2026](https://arxiv.org/html/2609.26918#bib.bib4)\)implements the achieved trade\-off substitution directly for Concave\-Augmented Pareto Q\-Learning \(CAPQL\)\([Lu et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib2)\), discussed further in Section[5\.1](https://arxiv.org/html/2609.26918#S5.SS1), motivating the approach as a direct generalization of the HER principle into preference space\. In both cases, adoption is motivated by HER’s single\-objective success rather than by analysis of whether HER’s underlying precondition, that an achieved outcome and a requested goal are directly substitutable, holds in the preference\-conditioned multi\-objective setting\.

### 3\.3Off\-policy MORL algorithms

Off\-policy MORL algorithms fall broadly along two main design axes: the underlying critic/update rule inherited from single\-objective RL, and the scheme used to sample or condition on preferences during training\([Lu et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib2)\)\. Early off\-policy approaches extended DQN\-style value learning to multiple objectives through scalarized targets\([Mossalam et al\. 2016](https://arxiv.org/html/2609.26918#bib.bib22)\), building on scalarization techniques developed for tabular multi\-objective Q\-learning\([Van Moffaert and Nowé 2014](https://arxiv.org/html/2609.26918#bib.bib23)\)\. More recent work has moved toward continuous control backbones \(SAC, TD3\) paired with preference\-conditioned critics capable of representing an entire Pareto front within a single policy\([Yang et al\. 2019](https://arxiv.org/html/2609.26918#bib.bib16)\)\. Preference\-sampling schemes vary similarly, with some methods drawing preferences uniformly from thedd\-simplex during training\([Basaklar et al\. 2022](https://arxiv.org/html/2609.26918#bib.bib3)\)while others bias sampling toward regions expected to be more informative, such as cone\-based sampling around directions that the policy is already competent in\([Lu et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib2)\), or Generalized Policy Improvement \(GPI\) over a maintained set of preference\-conditioned policies\([Alegre et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib9)\)\.

### 3\.4Diversity metrics for coverage

Measuring how evenly a set of samples is distributed, rather than only their quantity, is a long\-studied problem, where Hill numbers\([Hill 1973](https://arxiv.org/html/2609.26918#bib.bib21)\)and related entropy\-based diversity indices\([Jost 2006](https://arxiv.org/html/2609.26918#bib.bib25)\)quantify the “effective number” of categories represented in a population, collapsing to the raw category count under a uniform distribution and shrinking as mass concentrates\. We adapt the same construction to the preference simplex, treating each relabeled preference as a sample and asking how many effectively distinct regions of the simplex it covers\. Diversity\-style objectives also appear within RL, in quality\-diversity, novelty\-search and skill\-discovery methods\([Lehman and Stanley 2011](https://arxiv.org/html/2609.26918#bib.bib26);[Pugh et al\. 2016](https://arxiv.org/html/2609.26918#bib.bib27);[Eysenbach et al\. 2018](https://arxiv.org/html/2609.26918#bib.bib28)\)\.

Coverage collapse in preference\-conditioned MORL specifically has also been addressed directly at the architecture\-level, independent of the replay buffer\.[Kubo et al\.](https://arxiv.org/html/2609.26918#bib.bib29)show that under Smooth Tchebycheff \(STCH\) scalarization, the map from preference to optimal objective vector is Lipschitz continuous, and they derive an update rule \(CMDPI\) that exploits this for better preference\-space coverage\. Conversely, D3PO\([Ambadkar et al\. 2026](https://arxiv.org/html/2609.26918#bib.bib30)\)introduces a diversity\-driven regularizer that directly penalizes representational collapse across the preference simplex\. These approaches prevent collapse structurally, at the level of the scalarization or loss function\. Whether a collapse\-resistant architecture such as D3PO remains robust onceher\_achieved\-style relabeling is layered on top is, to our knowledge, untested\.

Our use of a diversity metric differs in purpose; rather than optimizing for diversity, or building an architecture robust to its loss, we use it purely diagnostically, to characterize a change in what a fixed training procedure covers as a side effect of relabeling\.

## 4Preference Coverage Collapse

### 4\.1Coverage

*Preference Coverage Collapse*\(PCC\) is the narrowing of the trade\-offs a policy actually serves under achieved\-preference relabeling: a covering policy answers different regions of𝒲\\mathcal\{W\}with different points of its achieved return setℱ\\mathcal\{F\}, and a collapsed policy serves the whole simplex from one corner\. We quantify this in two ways\.

Effective coverage \(C\)\.For each𝐰∈W\\mathbf\{w\}\\in W, we letpk∈ℱp\_\{k\}\\in\\mathcal\{F\}be the maximizing return andρk\\rho\_\{k\}the share ofWWfor whichpkp\_\{k\}is optimal\.CChere is the exponential Shannon entropy of this distribution \(a Hill number giving the effective number of distinct trade\-offs offered\):

C\(ℱ\)=exp\(−∑k=1\|ℱ\|ρklnρk\)\.C\(\\mathcal\{F\}\)=\\exp\\left\(\-\\sum\_\{k=1\}^\{\|\\mathcal\{F\}\|\}\\rho\_\{k\}\\ln\\rho\_\{k\}\\right\)\.\(1\)
C∈\[1,\|ℱ\|\]C\\in\[1,\|\\mathcal\{F\}\|\]:C≈1C\\approx 1under total collapse \(one return optimal almost everywhere\),CCapproaches\|ℱ\|\|\\mathcal\{F\}\|when every front point serves a distinct region ofWW\.CCis scale\-invariant and comparable across environments\.

Abandoned Preference Mass \(APM\)\.CCis purely structural and cannot distinguish healthy specialization from uniform failure\. APM complementsCC, measuring the fraction ofWWwhere utility falls meaningfully below a baseline front:

APM⁡\(ℱ,ℱ0\)=\|𝒜τ\|\|W\|,\\mathrm\{APM\}\(\\mathcal\{F\};\\mathcal\{F\}\_\{0\}\)\\;=\\;\\frac\{\|\\mathcal\{A\}\_\{\\tau\}\|\}\{\|W\|\},\(2\)where𝒜τ=\{𝐰∈W:Uℱ​\(𝐰\)<\(1−τ\)​Uℱ0​\(𝐰\)\}\\mathcal\{A\}\_\{\\tau\}=\\\{\\mathbf\{w\}\\in W:U\_\{\\mathcal\{F\}\}\(\\mathbf\{w\}\)<\(1\-\\tau\)\\,U\_\{\\mathcal\{F\}\_\{0\}\}\(\\mathbf\{w\}\)\\\}is the set of abandoned preferences andτ\\tausets what counts as meaningful; we useτ=0\.1\\tau=0\.1throughout, so a preference is abandoned when the policy realises less than90%90\\%of the baseline’s utility there\. APM lies in\[0,1\]\[0,1\]and equals00for the baseline itself; unlikeCCit is value\-aware, reading near00wherever the policy matches or beats the baseline\.

Both are functionals of the same realised fronts as EUM, so APM recasts the harm as lost coverage rather than independently explaining it; the causal account lives at the training level and is established empirically in Section[6\.4](https://arxiv.org/html/2609.26918#S6.SS4)\.

### 4\.2Mixed Relabeling

If the failure mode is a collapse of the conditioning distribution, the repair is to keep that distribution spread out\.her\_mixblends, per transition, the achieved direction with the originally collected preference:

𝐰used=\(1−λ\)​𝐰collected\+λ​𝐰achieved,\\mathbf\{w\}\_\{\\text\{used\}\}=\(1\-\\lambda\)\\,\\mathbf\{w\}\_\{\\text\{collected\}\}\+\\lambda\\,\\mathbf\{w\}\_\{\\text\{achieved\}\},\(3\)
a convex combination of two points of𝒲\\mathcal\{W\}and therefore itself a valid preference\. It adds one hyperparameter and one line of code; we fixλ=0\.25\\lambda=0\.25everywhere, with no per\-setting tuning\. As a falsification control, we also evaluate a TD\-gated variant that rejects relabels above a running TD\-error percentile but substitutes fully on the transition it accepts\. If the harm is coverage loss rather than noisy outliers, gating should underperform mixing\.

## 5Experiments

### 5\.1Algorithms and environments

We evaluate four preference\-conditioned off\-policy algorithms, chosen to span the two design axes of Section[3\.3](https://arxiv.org/html/2609.26918#S3.SS3)rather than to represent them exhaustively\.CAPQL\([Lu et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib2)\)pairs a SAC\([Haarnoja et al\. 2018](https://arxiv.org/html/2609.26918#bib.bib8)\)backbone with an angle\-cone preference sampler;GPI\-PD\([Alegre et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib9)\)pairs TD3\([Fujimoto et al\. 2018](https://arxiv.org/html/2609.26918#bib.bib10)\)with GPI weight\-support sampling\. They differ on both axes at once, so an effect that reproduces across both is unlikely to be an artifact of either choice\. The two ablations swap one component each:MO\-TD3is TD3 with the cone sampler,CAPQL\-uniformis SAC with uniform simplex sampling: completing a2×22\\times 2grid over\{update rule\}×\{sampler\}\\\{\\text\{update rule\}\\\}\\times\\\{\\text\{sampler\}\\\}that separates an effect tracking the backbone from one tracking the sampler\. Appendix A of the supplementary material gives the full descriptions\. All four run with library\-default hyperparameters; the replay buffer is the only component that varies\.

The benchmark is the continuous\-action MO\-Gymnasium suite\([Felten et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib15)\): eight MO\-MuJoCo v5 tasks \(hopper\-2obj,hopper,walker2d,halfcheetah,swimmer,ant\-2obj,ant,humanoid\) andmountaincar\-continuous, with two or three objectives each111mo\-reacher\-v5is excluded due to its discrete action space\.\. Crossing algorithms with environments yields 36 settings, the unit of analysis below\.

### 5\.2Protocol

Every run trains for 150k environment steps with evaluation every 15k\. Final EUM is the value at the last evaluation point\. Comparisons that support a claim use 16 random seeds per setting \(32 on the CAPQL hopper tasks\); mechanism probes use 3 seeds at 60k steps\. Seeds are shared across conditions, so all statistics are paired\. Our buffer implementation is verified as a drop\-in replacement: with all interventions disabled it reproduces CAPQL’s internal buffer exactly, in storage order and sampling behavior\. To check that the budget is not itself driving the verdicts, we re\-ran the two environments whose baselines are flat at 150k for 400k steps: both then learn \(\+34\.9\+34\.9\[\+19\.9,\+59\.0\+19\.9,\+59\.0\] and\+5\.6\+5\.6\[\+3\.4,\+9\.7\+3\.4,\+9\.7\] EUM\) and relabeling remains harmless on them \(d=−0\.06d=\-0\.06,\+0\.04\+0\.04\), so the classification is a property of those tasks rather than of the horizon\.

### 5\.3Statistical methodology

The primary metric is final EUM; HV is secondary \(as𝐫ref\\mathbf\{r\}\_\{\\text\{ref\}\}dependence introduces a free parameter\), and an HV result is only reported when its sign is stable across a2×22\\times 2grid of reference points and normalization schemes\. Effect sizes are Cohen’sddwith 95% BCa bootstrap confidence intervals \(10K resamples, paired by seed\), corrected by Holm\-Bonferroni\([Holm 1979](https://arxiv.org/html/2609.26918#bib.bib31)\)within each family of simultaneous tests \(one family per intervention across the nine environments\)\. Claims of no effect require equivalence under TOST with a standardized marginδd=0\.5\\delta\_\{d\}=0\.5, and a claim of harm additionally requires that the baseline itself learns on that task \(final versus early EUM, CI excluding zero\)\.

Alongside EUM and HV, which are scalarized and so cannot on their own distinguish a collapsed policy from a diverse one achieving similar value, we report the coverage statisticsCCand APM \(Section[4\.1](https://arxiv.org/html/2609.26918#S4.SS1)\), computed from each run’s saved final Pareto front on the same preference gridWWused for EUM\.

## 6Results

We test the three claims of Section[4](https://arxiv.org/html/2609.26918#S4)in order\. We first verify that the relabeling operator itself is well posed \(Section[6\.1](https://arxiv.org/html/2609.26918#S6.SS1)\), then establish that relabeling degrades learning and that its incidence is algorithm\- and environment\-dependent, rule out the obvious confounds, identify the harm as preference coverage collapse, and finally show that mixed relabeling repairs it at a single fixed coefficient\.

### 6\.1The Standard Operator Silently Ignores Cost Objectives

The apparent effect of relabeling depends first on how the achieved outcome is normalized into a preference\. The standard operator normalizesclip​\(achieved,0,∞\)\\text\{clip\}\(\\text\{achieved\},0,\\infty\), so an objective that is negative by convention contributes zero: on the four suite tasks carrying a control or energy cost, 86–100% of relabels degenerate to a corner or uniform vector, confounding the measured harm with the agent being trained to ignore an objective\. A sign\-robust operator \(her\_scaled, per\-objective min\-max normalization\) removes the artifact where the clip binds\. The effect onant\-2objgoes fromd=−1\.41d=\-1\.41to\+0\.04\+0\.04\. This leaves the harm unchanged where it does not, as onhopper\-2obj\(Appendix B of the supplementary material\)\. All results below use the sign\-robust operator, so any remaining effect reflects the relabeling principle rather than its implementation\.

### 6\.2Well\-Posed Relabeling Degrades Learning

Even with a well\-defined operator, relabeling harms performance in over half of the settings where it has any effect\. Across 36 settings, it harms 19 \(d≤−0\.5d\\leq\-0\.5with CI excluding zero\), leaves 16 statistically indistinguishable from baseline, and improves exactly one, with effects ranging fromd=−4\.03d=\-4\.03\(GPI\-PD/halfcheetah\) and\+1\.03\+1\.03\(GPI\-PD/ant\); Figure[2](https://arxiv.org/html/2609.26918#S6.F2)\(orange\) reports every setting\.

Figure 2:Effect of preference relabeling \(orange\) and of mixed relabeling atλ=0\.25\\lambda=0\.25\(blue\) versus baseline, Cohen’sddon final EUM, across all 36 algorithm\-environment settings, sorted by the relabeling effect\. The grey band marks\|d\|<0\.5\|d\|<0\.5; arrows connect the two conditions per setting\.Harm has a two\-level structure\. A core is algorithm\-independent: the early\-terminating bipeds: three\-objectivehopperdegrades under all four algorithms \(ddfrom−1\.30\-1\.30to−2\.56\-2\.56\),hopper\-2objandwalker2dunder three\. Outside that core, incidence tracks the algorithm: GPI\-PD is severely harmed on halfcheetah \(−4\.03\-4\.03\) where CAPQL is near\-null, MO\-TD3 is harmed on ant where GPI\-PD improves, and humanoid is unaffected throughout\. Whether relabeling hurts is a joint property of the environment’s failure structure and of how the algorithm consumes its conditioning preference\. The improvements are not weak\-baseline artifacts: every verdict passes the learning gate, and on GPI\-PD ant relabeling lifts final EUM from108108to124124\.

### 6\.3The Harm is intrinsic to Relabeling

Three families of alternative explanation fail to account for the effect \(statistics in Appendices C and D of the supplementary material\)\. Prioritized sampling is not responsible: PER alone is indistinguishable from baseline, and the PER×\\timesHER interaction is identifiable on only two of eight environments, pointing in opposite directions\. Buffer structure is not either: re\-tested at 16 seeds \(256 runs, MDE\|d\|≈1\.02\|d\|\\approx 1\.02\), no storage, partitioning, eviction, or capacity variant harms performance\. Nor is it about which transitions get relabeled: swimmer and halfcheetah never terminate early yet are among the most harmed \(d=−2\.87d=\-2\.87,−4\.03\-4\.03\), and de\-noising the target with return\-to\-go recovers only 7% of the loss\.

### 6\.4The Harm Is Preference Coverage Collapse

Measured with the statistics of Section[4\.1](https://arxiv.org/html/2609.26918#S4.SS1), coverage collapses under relabeling\. Effective coverageCC\(Equation[1](https://arxiv.org/html/2609.26918#S4.E1)\) falls on 22 of 32 settings with usable fronts \(mean→1\.92\.6\\\!\\rightarrow\\\!1\.9effective trade\-offs\), and on the harmed settings relabeling abandons a mean APM of69%69\\%of the preference simplex\. The harm tracks APM \(Figure[3](https://arxiv.org/html/2609.26918#S6.F3)a; Spearmanρ=−0\.73\\rho=\-0\.73,p<10−5p<10^\{\-5\}\) far more tightly thanCC\(\+0\.40\+0\.40\): APM flags the most\-harmed CAPQL/hopper\-2obj setting \(APM=100%=100\\%\), whichCCmisses since that front keeps its diversity while losing value everywhere, and it reads near\-zero on the one setting where relabeling helps \(APM=10%=10\\%on GPI\-PD/ant\), where collapsing onto a good corner is benign specialization rather than failure\. Mixed relabeling reclaims the coverage: on the harmed settings it cuts APM from69%69\\%to6%6\\%\(Figure[3](https://arxiv.org/html/2609.26918#S6.F3)b\) and restores87%87\\%of the lostCC, matching its recovery of EUM \(Section[6\.5](https://arxiv.org/html/2609.26918#S6.SS5)\)\.

Figure 3:Coverage collapse is general, and mixing reclaims it\.\(a\)harm versus abandoned preference mass across all 32 settings with usable fronts; marker shape denotes the algorithm\.\(b\)on the harmed settings,her\_mixatλ=0\.25\\lambda=0\.25returns the abandoned mass from a mean of69%69\\%to6%6\\%\.Decomposing performance by preferences shows why\. Recomputing per\-preference utility from saved Pareto fronts, three of four decisive harmed settings preserve utility near the direction the agent actually achieved and degrade steeply elsewhere\.; relabeling concentrated training on the corner reached, abandoning the rest of the simplex\. This is predominantly a uniform downward sift rather than seed\-level collapse \(which dominated on only four of eight harmed settings\)\.

The dose\-response is a cliff, not a slope \(Figure[4](https://arxiv.org/html/2609.26918#S6.F4)a\)\. Relabeling transitions independently with probabilityffon hopper\-2obj leaves performance intact throughf=0\.5f=0\.5\(d=\+0\.15d=\+0\.15atf=0\.25f\{=\}0\.25,\+0\.02\+0\.02atf=0\.5f\{=\}0\.5\), degrades atf=0\.75f=0\.75\(−0\.49\-0\.49\), collapsing atf=1f=1\(−3\.48\-3\.48\)\. A minority of surviving collected preferences suffices to hold the critic’s coverage together; the failure mode requires near\-total substitution\.

Figure 4:The failure requires near\-total substitution\.Both panels are CAPQL×\\timeshopper\-2obj\.\(a\)relabeling each transition with probabilityff: performance holds untilf→1f\\rightarrow 1\.\(b\)varying the mixing weightλ\\lambda: recovery stays near\-complete well past the\(1−λ\)\(1\-\\lambda\)dilution line\.ffandλ\\lambdaare different interventions\. Bars are 95% bootstrap CIs\.Noisy value targets are not the mechanism; instrumenting both critic families, the relabeled TD error is smaller than the collected\-conditioning TD error in 34 of 36 settings, and its ratio correlates positively with harm \(ρ=\+0\.36\\rho=\+0\.36, CI\[\+0\.01,\+0\.68\]\[\+0\.01,\+0\.68\]\), the opposite of what a noisy\-target account predicts; our prespecified criterion for that account fails on all 7 environments where algorithms diverge\. The harmed critics are the ones that fit the achieved direction best\. This rules out repairs that filter high\-error relabels, and motivates repairing the conditioning distribution instead\.

### 6\.5Mixed Relabeling Restores Coverage

We evaluateher\_mix\(Equation[3](https://arxiv.org/html/2609.26918#S4.E3),λ=0\.25\\lambda=0\.25fixed in Section[4\.2](https://arxiv.org/html/2609.26918#S4.SS2)\) alongside a TD\-gated control the coverage account predicts should underperform, on settings spanning all three regimes against criteria fixed in advance \(Table[1](https://arxiv.org/html/2609.26918#S6.T1)\)\. Mixed relabeling recovers 87 to 117% of the harm on all five harmed settings, preserves and amplifies the single benefit \(ant:d=\+1\.03d=\+1\.03under full relabeling,\+2\.42\+2\.42under mixing,\+4\.46\+4\.46atλ=0\.5\\lambda=0\.5\), and leaves the neutral control unchanged \(\+0\.08\+0\.08, CI\[−0\.51,\+0\.60\]\[\-0\.51,\+0\.60\]\)\. The TD gate recovers only 62% on halfcheetah and swimmer and fails its criterion on both, despite accepting 75 to 88% of relabels, as predicted\.

Mixing is not merely attenuated relabeling\. Under pure attenuation, recovery would scale as\(1−λ\)\(1\-\\lambda\)\(Figure[4](https://arxiv.org/html/2609.26918#S6.F4)b\); measured recovery is instead flat at 94 to 103% for allλ≤0\.5\\lambda\\leq 0\.5on hopper\-2obj and collapses only asλ→1\\lambda\\rightarrow 1, mirroring the dose\-response cliff\. On ant, mixing outperforms both of its endpoints, which no interpolation of their outcomes can produce; the blend reaches trade\-offs neither pure strategy trains\. The recovery is also stable in time, exceeding 86% over the second half of training on every harmed setting\.

Table 1:Mixed relabeling on the seven pre\-registered settings, spanning the three regimes\.ddis Cohen’sddon final EUM against baseline; recovery is\(mixed−relabel\)/\(baseline−relabel\)\(\\text\{mixed\}\-\\text\{relabel\}\)/\(\\text\{baseline\}\-\\text\{relabel\}\)with a paired bootstrap CI, so100%100\\%is a full return to baseline and0%0\\%is no better than relabeling\. All arms use the same fixedλ=0\.25\\lambda=0\.25\.
### 6\.6Generalisation across the suite

Finally, we deploy the same untunedλ=0\.25\\lambda=0\.25on every remaining setting of the study \(465 additional runs\)\. Figure[2](https://arxiv.org/html/2609.26918#S6.F2)shows the outcome on all 36 settings: 32 satisfy their criterion\. Harm is recovered on 16 of 19 harmed settings, across all four algorithms; the three exceptions are settings where relabeling’s own effect is small \(\|d\|≈0\.6\|d\|\\approx 0\.6\), so the recovery ratio is noisy even though mixing’s absolute effect already lies within the equivalence band\. The helped setting keeps its benefit, and 15 of 16 neutral settings are unaffected, with a single regression on GPI\-PD mountaincar \(d=−0\.66d=\-0\.66, CI\[−1\.07,−0\.36\]\[\-1\.07,\-0\.36\]\)\. A single fixed coefficient thus converts preference relabeling from an intervention that can cost four standard deviations into one that is neutral to beneficial across two critic families and nine environments\.

## 7Discussion, Limitations, and Future Work

Our results reframe HER\-style preference relabeling from a free sample\-efficiency tool into an intervention whose safety depends on an under\-examined precondition: that an achieved trade\-off is a valid substitute for a requested one\. That precondition frequently fails, and where it does the substitution is not merely noisy but actively harmful\. Any relabeling scheme that substitutes an*achieved*outcome for a*requested*one should therefore be treated as suspect by default, not assumed safe by analogy to HER’s single\-objective success\.

The failure is repairable, and diagnosing it takes more than coverage\.her\_mixremoves the harm without per\-environment tuning, and on the one setting where relabeling helps it exceeds both the pure\-relabel and the pure\-baseline condition: the gain is access to trade\-offs neither endpoint reaches, not attenuation\. Yet effective coverageCCcorrelates only weakly with harm \(ρ=\+0\.40\\rho=\+0\.40\) where APM tracks it tightly \(ρ=−0\.73\\rho=\-0\.73\), and APM correctly reads near\-zero on that same helped setting, where collapsing onto a corner is legitimate specialization\. Measuring how much of the preference space an agent covers is not the same as measuring how well it performs across that space, and a diagnostic built only on the first can be misled in either direction\.

Three limitations bound these claims\. All results are on MO\-MuJoCo with two or three objectives and four algorithm/sampler combinations over two critic backbones, so we do not claim generality across off\-policy MORL: discrete\-action benchmarks, more than three objectives, and multi\-policy or model\-based algorithms remain untested\.her\_mixis a strong default rather than a guarantee: untuned atλ=0\.25\\lambda=0\.25it recovers or preserves 32 of 36 verdicts, with one regression \(GPI\-PD/mountaincar\) and partial recovery on the largest original harm \(GPI\-PD/halfcheetah\), both consistent with an account predicting that mixing helps wherever collapse drives the harm, not that one coefficient closes every gap\. And our sign\-robust operator resolves the specific clip failure we identified without ruling out other ways an operator could misrepresent an achieved trade\-off\.

Each points to a next step: establishing whether the collapse mechanism is general to preference\-conditioned critics or specific to the single\-policy off\-policy setting studied here; an adaptiveλ\\lambda, annealed with training progress or driven by APM as an online proxy, which is already computable from checkpointed fronts; and a lightweight precondition check \(flagging majority\-negative objectives, say\) that catches this failure automatically rather than by dedicated analysis\.

## References

- Abelset al\.\(2019\)A\. Abels, D\. Roijers, T\. Lenaerts, A\. Nowé, and D\. SteckelmacherDynamic weights in multi\-objective deep reinforcement learning\.InInternational conference on machine learning,pp\. 11–20\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p1.1)\.
- Alegreet al\.\(2023\)L\. N\. Alegre, A\. L\. Bazzan, D\. M\. Roijers, A\. Nowé, and B\. C\. da SilvaSample\-efficient multi\-objective learning via generalized policy improvement prioritization\.arXiv preprint arXiv:2301\.07784\.Cited by:[Appendix A](https://arxiv.org/html/2609.26918#A1.p3.1),[§3\.3](https://arxiv.org/html/2609.26918#S3.SS3.p1.1),[§5\.1](https://arxiv.org/html/2609.26918#S5.SS1.p1.1)\.
- Ambadkaret al\.\(2026\)T\. Ambadkar, S\. Panda, S\. Kale, J\. Dodge, and A\. VermaPreference conditioned multi\-objective reinforcement learning: decomposed, diversity\-driven policy optimization\.arXiv preprint arXiv:2602\.07764\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p2.1)\.
- Andrychowiczet al\.\(2017\)M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, O\. Pieter Abbeel, and W\. ZarembaHindsight experience replay\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p1.1),[§3\.1](https://arxiv.org/html/2609.26918#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2609.26918#S3.SS2.p1.1)\.
- Basaklaret al\.\(2022\)T\. Basaklar, S\. Gumussoy, and U\. Y\. OgrasPd\-morl: preference\-driven multi\-objective reinforcement learning algorithm\.arXiv preprint arXiv:2208\.07914\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p2.1),[§3\.2](https://arxiv.org/html/2609.26918#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2609.26918#S3.SS3.p1.1)\.
- Cohen \(2013\)J\. CohenStatistical power analysis for the behavioral sciences\.routledge\.Cited by:[§2\.3](https://arxiv.org/html/2609.26918#S2.SS3.p4.1)\.
- Duanet al\.\(2016\)Y\. Duan, X\. Chen, R\. Houthooft, J\. Schulman, and P\. AbbeelBenchmarking deep reinforcement learning for continuous control\.InInternational conference on machine learning,pp\. 1329–1338\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p1.1)\.
- Eysenbachet al\.\(2018\)B\. Eysenbach, A\. Gupta, J\. Ibarz, and S\. LevineDiversity is all you need: learning skills without a reward function\.arXiv preprint arXiv:1802\.06070\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p1.1)\.
- Fanget al\.\(2019\)M\. Fang, T\. Zhou, Y\. Du, L\. Han, and Z\. ZhangCurriculum\-guided hindsight experience replay\.Advances in neural information processing systems32\.Cited by:[§3\.1](https://arxiv.org/html/2609.26918#S3.SS1.p1.1)\.
- Feltenet al\.\(2023\)F\. Felten, L\. N\. Alegre, A\. Nowé, A\. L\. C\. Bazzan, E\. G\. Talbi, G\. Danoy, and B\. C\. d\. SilvaA toolkit for reliable benchmarking and research in multi\-objective reinforcement learning\.InProceedings of the 37th Conference on Neural Information Processing Systems \(NeurIPS 2023\),Cited by:[§5\.1](https://arxiv.org/html/2609.26918#S5.SS1.p2.1)\.
- Fujimotoet al\.\(2018\)S\. Fujimoto, H\. Hoof, and D\. MegerAddressing function approximation error in actor\-critic methods\.InInternational conference on machine learning,pp\. 1587–1596\.Cited by:[Appendix A](https://arxiv.org/html/2609.26918#A1.p3.1),[§5\.1](https://arxiv.org/html/2609.26918#S5.SS1.p1.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[Appendix A](https://arxiv.org/html/2609.26918#A1.p2.1),[§1](https://arxiv.org/html/2609.26918#S1.p1.1),[§5\.1](https://arxiv.org/html/2609.26918#S5.SS1.p1.1)\.
- Hayeset al\.\(2022\)C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz,et al\.A practical guide to multi\-objective reinforcement learning and planning: cf hayes et al\.\.Autonomous Agents and Multi\-Agent Systems36\(1\),pp\. 26\.Cited by:[§2\.3](https://arxiv.org/html/2609.26918#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2609.26918#S2.SS3.p3.1)\.
- Hill \(1973\)M\. O\. HillDiversity and evenness: a unifying notation and its consequences\.Ecology54\(2\),pp\. 427–432\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p1.1)\.
- Holm \(1979\)S\. HolmA simple sequentially rejective multiple test procedure\.Scandinavian journal of statistics,pp\. 65–70\.Cited by:[§5\.3](https://arxiv.org/html/2609.26918#S5.SS3.p1.1)\.
- Jost \(2006\)L\. JostEntropy and diversity\.Oikos113\(2\),pp\. 363–375\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p1.1)\.
- Kuboet al\.\(2026\)A\. Kubo, K\. Nakanishi, and S\. IshiiA single deep preference\-conditioned policy for learning pareto coverage sets\.arXiv preprint arXiv:2605\.08946\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p2.1)\.
- Lehman and Stanley \(2011\)J\. Lehman and K\. O\. StanleyAbandoning objectives: evolution through the search for novelty alone\.Evolutionary computation19\(2\),pp\. 189–223\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p1.1)\.
- Lillicrapet al\.\(2019\)T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. WierstraContinuous control with deep reinforcement learning\.External Links:1509\.02971,[Link](https://arxiv.org/abs/1509.02971)Cited by:[§2\.4](https://arxiv.org/html/2609.26918#S2.SS4.p1.1)\.
- Luet al\.\(2023\)H\. Lu, D\. Herman, and Y\. YuMulti\-objective reinforcement learning: convexity, stationarity and pareto optimality\.InThe Eleventh International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.26918#A1.p2.1),[§3\.2](https://arxiv.org/html/2609.26918#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2609.26918#S3.SS3.p1.1),[§5\.1](https://arxiv.org/html/2609.26918#S5.SS1.p1.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.Human\-level control through deep reinforcement learning\.nature518\(7540\),pp\. 529–533\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p1.1)\.
- Mossalamet al\.\(2016\)H\. Mossalam, Y\. M\. Assael, D\. M\. Roijers, and S\. WhitesonMulti\-objective deep reinforcement learning\.arXiv preprint arXiv:1610\.02707\.Cited by:[§3\.3](https://arxiv.org/html/2609.26918#S3.SS3.p1.1)\.
- Pughet al\.\(2016\)J\. K\. Pugh, L\. B\. Soros, and K\. O\. StanleyQuality diversity: a new frontier for evolutionary computation\.Frontiers in Robotics and AI3,pp\. 40\.Cited by:[§3\.4](https://arxiv.org/html/2609.26918#S3.SS4.p1.1)\.
- Renet al\.\(2019\)Z\. Ren, K\. Dong, Y\. Zhou, Q\. Liu, and J\. PengExploration via hindsight goal generation\.Advances in Neural Information Processing Systems32\.Cited by:[§3\.1](https://arxiv.org/html/2609.26918#S3.SS1.p1.1)\.
- Roijerset al\.\(2013\)D\. M\. Roijers, P\. Vamplew, S\. Whiteson, and R\. DazeleyA survey of multi\-objective sequential decision\-making\.Journal of Artificial Intelligence Research48,pp\. 67–113\.Cited by:[§2\.2](https://arxiv.org/html/2609.26918#S2.SS2.p1.1)\.
- Schaulet al\.\(2015\)T\. Schaul, J\. Quan, I\. Antonoglou, and D\. SilverPrioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[§2\.4](https://arxiv.org/html/2609.26918#S2.SS4.p2.1)\.
- Shianifaret al\.\(2026\)J\. Shianifar, M\. Schukat, and K\. MasonHindsight preference replay improves preference\-conditioned multi\-objective reinforcement learning\.arXiv preprint arXiv:2601\.11604\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p2.1),[§3\.2](https://arxiv.org/html/2609.26918#S3.SS2.p1.1)\.
- Suttonet al\.\(1998\)R\. S\. Sutton A\. G\. Bartoet al\.Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§2\.1](https://arxiv.org/html/2609.26918#S2.SS1.p1.1)\.
- Van Moffaert and Nowé \(2014\)K\. Van Moffaert and A\. NowéMulti\-objective reinforcement learning using sets of pareto dominating policies\.The Journal of Machine Learning Research15\(1\),pp\. 3483–3512\.Cited by:[§3\.3](https://arxiv.org/html/2609.26918#S3.SS3.p1.1)\.
- Yanget al\.\(2019\)R\. Yang, X\. Sun, and K\. NarasimhanA generalized algorithm for multi\-objective reinforcement learning and policy adaptation\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2609.26918#S1.p1.1),[§3\.3](https://arxiv.org/html/2609.26918#S3.SS3.p1.1)\.
- Zintgrafet al\.\(2015\)L\. M\. Zintgraf, T\. V\. Kanters, D\. M\. Roijers, F\. Oliehoek, and P\. BeauQuality assessment of morl algorithms: a utility\-based approach\.InBenelearn 2015: proceedings of the 24th annual machine learning conference of Belgium and the Netherlands,Cited by:[§2\.3](https://arxiv.org/html/2609.26918#S2.SS3.p3.1)\.

## Appendix AAlgorithm Details

All four algorithms condition the critic on the preference vector𝐰\\mathbf\{w\}and differ in the critic update rule and in how training preferences are drawn\. All run with library\-default hyperparameters; the replay buffer is the only component we vary\. We also release the full implementation \(the swappable buffer, both algorithm adapters, the run manifests, and the analysis scripts that regenerate every number and figure reported here\)\.

CAPQL\([Lu et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib2)\)builds on Soft Actor\-Critic\([Haarnoja et al\. 2018](https://arxiv.org/html/2609.26918#bib.bib8)\), conditioning both actor and critic on𝐰\\mathbf\{w\}: the actor outputsπ⁡\(a∣s,𝐰\)\\pi\(a\\mid s,\\mathbf\{w\}\)and the critic estimatesQ⁡\(s,a,𝐰\)Q\(s,a,\\mathbf\{w\}\)\. Because linear scalarization concentrates optimal policies at vertices of the achievable value\-function set, CAPQL augments the reward with an entropy term, which makes that set strictly convex and the preference\-to\-value mapping continuous\. Training preferences are drawn from an angle cone around the diagonal\.

GPI\-PD\([Alegre et al\. 2023](https://arxiv.org/html/2609.26918#bib.bib9)\)builds on TD3\([Fujimoto et al\. 2018](https://arxiv.org/html/2609.26918#bib.bib10)\), likewise conditioning the critic on𝐰\\mathbf\{w\}\. Rather than shaping a single conditioned critic’s output, it acts by evaluating a support setℳ\\mathcal\{M\}of previously learned weight vectors under the query preference and taking the best resulting action,

πGPI​\(s,𝐰\)∈arg⁡maxa​max𝐰′∈ℳ⁡Q⁡\(s,a,𝐰′\)⋅𝐰,\\pi^\{\\text\{GPI\}\}\(s;\\mathbf\{w\}\)\\in\\arg\\max\_\{a\}\\max\_\{\\mathbf\{w\}^\{\\prime\}\\in\\mathcal\{M\}\}Q\(s,a,\\mathbf\{w\}^\{\\prime\}\)\\cdot\\mathbf\{w\},which is guaranteed to perform at least as well as any single policy in the set\. The same support set supplies the conditioning preferences its critic is trained on\.

MO\-TD3andCAPQL\-uniformare diagnostic ablations that each swap one component\. MO\-TD3 pairs the TD3 backbone with CAPQL’s cone sampler; CAPQL\-uniform pairs the SAC backbone with uniform sampling over the simplex\. Together with CAPQL and GPI\-PD they form a2×22\\times 2grid over\{update rule\}×\{sampler\}\\\{\\text\{update rule\}\\\}\\times\\\{\\text\{sampler\}\\\}, which is what lets us attribute an effect to the backbone, to the sampler, or to their combination\.

## Appendix BThe Clip Artifact on Cost Objectives

Four MO\-MuJoCo tasks carry an objective that is a pure control or energy cost and is therefore negative by construction\. The standardher\_achievedoperator normalizesclip​\(𝐨^,0,∞\)\\text\{clip\}\(\\hat\{\\mathbf\{o\}\},0,\\infty\), so on those tasks the clip zeroes that entry on effectively every transition: the relabeled preference can never point toward the cost objective and degenerates to a corner or uniform vector\.

Table[2](https://arxiv.org/html/2609.26918#A2.T2)reports the fraction of genuine \(non\-degenerate\) relabels alongside the effect of relabeling under both operators\. Replacing the clip with per\-objective min\-max normalization \(her\_scaled\) restores informative relabels: the effect onant\-2objvanishes entirely and becomes statistically equivalent to baseline, and it shrinks by more than half on swimmer and halfcheetah\. Walker2d retains substantial harm, andhopper\-2obj, where the clip never binds, is unchanged\. The artifact therefore explains the cost\-objective tasks but not the phenomenon itself, which is why all results in the main text use the sign\-robust operator\.

Table 2:Effect of relabeling \(Cohen’sddon final EUM, CAPQL\) under the standard clip\-based operator and the sign\-robust operator\. Genuine % is the fraction of relabels that are not corner or uniform degenerate under the clip\.†Inflated by low pooled variance\.

## Appendix CAlternative Explanations for the Harm

The main text states three verdicts on alternative explanations for the harm; this appendix gives the underlying statistics\. Unless noted, the environment ishopper\-2objwith termination on, 16 seeds, and effect sizes are Cohen’sddon final EUM with paired BCa intervals\.

### C\.1Prioritized sampling

PER on its own is close to inert, ranging fromd=\+0\.05d=\+0\.05to−0\.43\-0\.43across the suite\. For the interaction we report the difference\-in\-differences contrast on raw EUM,I=\(PER\+HER\)−PER−HER\+baselineI=\(\\text\{PER\}\{\+\}\\text\{HER\}\)\-\\text\{PER\}\-\\text\{HER\}\+\\text\{baseline\}, so that no verdict rests on a ratio of two noisy effect sizes;I<0I<0means PER amplifies the harm\. Table[3](https://arxiv.org/html/2609.26918#A3.T3)gives the per\-environment result\.

Table 3:PER×\\timesHER interaction, 16 seeds per cell, on theher\_achievedcells \(the only ones with a complete2×22\\times 2\)\. The interaction is identifiable on 2 of 8 environments, and those two point in opposite directions\.On six of eight environments the interval crosses zero, so the interaction is not identifiable at 16 seeds and no directional claim should be read from the point estimate\. Where it is identifiable the two environments disagree in sign\. PER is therefore neither an independent source of harm nor a consistent amplifier of it\.

### C\.2Early termination

Whether hopper’s early termination amplifies the harm is atermination×HER\\text\{termination\}\\times\\text\{HER\}interaction, which we report as the difference\-in\-differencesJJon raw EUM under the primary sign\-robust operator\. With termination on, relabeling costsd=−2\.86d=\-2\.86\[−5\.46\-5\.46,−1\.12\-1\.12\]; with it disabled,d=−0\.51d=\-0\.51\[−0\.78\-0\.78,\+0\.11\+0\.11\], givingJ=−105\.3J=\-105\.3EUM \[−139\.9\-139\.9,−66\.0\-66\.0\],p<0\.001p<0\.001\. Termination therefore amplifies the harm substantially, but does not create it: swimmer and halfcheetah never terminate early and are among the most harmed settings under the TD3\-family algorithms \(d=−2\.87d=\-2\.87and−4\.03\-4\.03\)\.

The same interaction is not identifiable under theher\_achievedclip \(J=−14\.4J=\-14\.4\[−59\.1\-59\.1,\+34\.9\+34\.9\],p=0\.54p=0\.54\), because the clip behaves differently again when termination is disabled\. This is a further illustration of why the operator has to be fixed before any mechanism question can be asked\.

### C\.3Relabel\-direction noise and episode length

Two interventions test whether the harm is an artefact of*which*transitions are relabeled rather than of relabeling itself\. Replacing the one\-step achieved direction with the far less noisy return\-to\-go direction \(her\_future\) recovers 7% of the loss\. Suppressing relabels from episodes shorter thanLminL\_\{\\min\}steps recovers a third atLmin=50L\_\{\\min\}=50, rising to 61% atLmin=200L\_\{\\min\}=200while relabeling is still active on 76% of transitions \(Table[4](https://arxiv.org/html/2609.26918#A3.T4)\)\.

Table 4:Targeted fixes onhopper\-2obj\(her\_achieved, 16 seeds\)\. Recovery is\(arm−HER\)/\(baseline−HER\)\(\\text\{arm\}\-\\text\{HER\}\)/\(\\text\{baseline\}\-\\text\{HER\}\)\.Short failed episodes are thus disproportionately responsible, which is consistent with the coverage account: they are exactly the transitions whose achieved direction is degenerate\. But no filter closes the gap, and de\-noising the target barely moves it, so neither the noise of the relabel target nor the population of episodes it is drawn from is the mechanism\.

## Appendix DStructural Buffer Design at Conclusion\-Grade Power

The structural knobs were originally screened at 4 seeds, where the minimum detectable effect is\|d\|≈2\.38\|d\|\\approx 2\.38and no equivalence verdict is possible\. We re\-ran them at 16 seeds \(256 runs\), which brings the minimum detectable effect to\|d\|≈1\.02\|d\|\\approx 1\.02, and tested equivalence with TOST at the preregistered marginδd=0\.5\\delta\_\{d\}=0\.5\. Two environments were used:hopper\-2obj, which carries the harm, andswimmer, where the 4\-seed screen produced the noisiest estimates; Table[5](https://arxiv.org/html/2609.26918#A4.T5)reports every cell\. Capacity at1×1\\timesand4×4\\timesthe training budget is omitted: the buffer never fills at 150k steps, so those settings are mechanically identical to baseline rather than empirically equivalent to it\.

Table 5:Structural knobs versus baseline, 16 seeds per cell, CAPQL\.ppis Holm\-corrected across the 16 tests\. No cell shows harm; the single Holm\-significant cell is a benefit that does not replicate across environments\.Six of sixteen cells are equivalent to baseline atδd=0\.5\\delta\_\{d\}=0\.5and nine are inconclusive at that margin, which is a statement about the margin rather than about the effects: every inconclusive interval is well inside±1\\pm 1, an order of magnitude short of relabeling’sd=−4\.03d=\-4\.03\. The single Holm\-significant cell is per\-objective partitioning helping onhopper\-2obj\(d=\+0\.67d=\+0\.67,p=0\.022p=0\.022\), which reverses sign onswimmer\(−0\.34\-0\.34\) and so does not support a general recommendation\. We report it rather than omit it, but it does not bear on the paper’s claim: no structural choice we tested harms performance, and none approaches the magnitude of the relabeling effect\.

## Appendix EPer\-Setting Validation of the Fixed Coefficient

The main paper reports that one untunedλ=0\.25\\lambda=0\.25satisfies 32 of the 36 algorithm\-environment settings, and shows the outcome as a forest plot\. Table[6](https://arxiv.org/html/2609.26918#A5.T6)gives the underlying per\-setting numbers\. Settings are grouped by the role HER’s own effect assigns them — harmed, helped, or neutral — and sorted within each group by that effect\. Recovery is\(mixed−relabel\)/\(baseline−relabel\)\(\\text\{mixed\}\-\\text\{relabel\}\)/\(\\text\{baseline\}\-\\text\{relabel\}\), so100%100\\%is a full return to baseline; a checkmark marks the pre\-registered criterion being met \(harmed: recovery≥70%\\geq 70\\%or TOST\-equivalent to baseline; helped:d≥\+0\.5d\\geq\+0\.5retained; neutral: not significantly harmed\)\.

Table 6:Per\-setting validation ofher\_mixat a single fixedλ=0\.25\\lambda=0\.25\. Blocks are harmed, helped, then neutral settings\.Sixteen of the nineteen harmed settings are recovered\. The three exceptions are settings where relabeling’s own effect is small \(\|d\|≈0\.6\|d\|\\approx 0\.6\), so the recovery ratio is noisy even though mixing’s absolute effect already lies inside the equivalence band\. The single helped setting keeps its benefit, and fifteen of sixteen neutral settings are unaffected, the exception being a regression on GPI\-PD/mountaincar\(d=−0\.66d=\-0\.66, CI\[−1\.07,−0\.36\]\[\-1\.07,\-0\.36\]\)\.