Training and Evaluating Ethical Reinforcement Learning Agents on Per-Episode Distributions

# Training and Evaluating Ethical Reinforcement Learning Agents on Per-Episode Distributions
Source: [https://arxiv.org/html/2608.14642](https://arxiv.org/html/2608.14642)
###### Abstract

Reinforcement Learning \(RL\) agents trained on a single reward signal exploit the gap between the designed reward and the intended behavior\. This is particularly a problem when we are trying to imbue ethical behavior into RL agents\. An agent can look ethical*on average*while concentrating its violations in a few bad episodes, and a creature in the environment harmed in one episode is not restored by good conduct in another\. We compare four ways of training ethical behavior in Craftax, an open\-ended survival benchmark\. The four are: scalar penalties with termination, a linear multi\-objective weight sweep, an adaptive Lagrangian constraint, and a non\-compensatory utility optimized per episode under the Expected Scalarized Returns \(ESR\) criterion\. All are evaluated under a single detector\-based protocol that counts every violation in every episode without censoring\. On the frontier of mean return against mean violation rate, the four methods are indistinguishable; per episode they separate sharply\. At matched mean return, the ESR agent holds its stated budget of one violation in effectively every episode \(worst\-decile1\.04±0\.071\.04\\pm 0\.07violations\), the Lagrangian leaks past the same budget \(1\.14±0\.031\.14\\pm 0\.03\), and the weight sweep’s worst episodes double it \(2\.20±0\.202\.20\\pm 0\.20\)\. An observation\-augmentation control attributes the separation to the training objective rather than to what the agent observes, and the per\-episode guarantee costs nothing on the mean frontier\. When ethical violations do not average away across episodes, we argue both training and evaluation must target the per\-episode distribution rather than the mean\.

## 1Introduction

Reinforcement Learning \(RL\) agents that chase a single reward signal often find ways to score well that their designers never wanted\. This is specification gaming, or reward hacking\(Krakovnaet al\.[2020](https://arxiv.org/html/2608.14642#bib.bib1); Skalseet al\.[2022](https://arxiv.org/html/2608.14642#bib.bib2)\), and it arises whenever the reward we can write down is only a rough stand\-in for what we actually care about\(Amodeiet al\.[2016](https://arxiv.org/html/2608.14642#bib.bib3)\)\. The damage is mild in small closed tasks but severe in open\-ended worlds, the setting that most resembles real deployment\. Craftax\(Matthewset al\.[2024](https://arxiv.org/html/2608.14642#bib.bib4)\)is one such world, a fast JAX survival game in which an agent gathers resources, climbs a technology tree, and survives across procedurally generated levels\. Its speed is a gift for research, but it also lets a misaligned agent practice harmful shortcuts at enormous scale\. Rewarded only for progress, the agent will strip a forest bare or kill a harmless creature for a few more points\.

Ethical training fails in a particular way\. An agent whose violations are rare on average may still concentrate them in a handful of bad episodes, and harm done in one episode is not undone by good conduct in another\. Mean violation rate, the standard target for both training and evaluation, cannot tell a uniformly almost\-clean agent apart from one that is perfect most of the time and catastrophic occasionally\. For the goal of learning ethically aligned behavior, this hidden tail is unacceptable\.

We address this challenge with two design choices, the first being the*shape*of the trade\-off\. Rather than a hand\-tuned linear weighting, we formalize the ethical requirement as a non\-compensatory utility, a thresholded\-lexicographic rule under which task return counts only while an episode stays within a stated violation budget\. As a secondary variant we*distill*the same stated ordering into a differentiable function via Bradley–Terry preference learning\(Wirthet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib23); Christianoet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib24)\)\. The labels come from our own rule, so this distills a stated ethic rather than learning ethics from behavior\. The second choice is*when*the trade\-off applies\. A utility can be applied after averaging over episodes, letting a bad episode be offset by good ones, or inside each episode\(Roijerset al\.[2013](https://arxiv.org/html/2608.14642#bib.bib7); Hayeset al\.[2022](https://arxiv.org/html/2608.14642#bib.bib8)\)\. For ethical behavior we take the second view, so violations are counted per episode and never averaged away\.

Measurement needs the same care as objective design: if we collapse violations to a single mean, we reintroduce the same cross\-episode compensation that makes rare catastrophic episodes invisible\. We score every trained agent, agnostic of its training method, in one shared detector\-only environment with uncensored per\-step counts and report the per\-episode violation*distribution*, its standard deviation, the probability of any violation, and the worst\-decile mean \(CVaR\)\. Across three Craftax dilemmas, we compare four training methods under this protocol: scalar penalties with termination, a linear weight sweep, an adaptive Lagrangian constraint, and the per\-episode non\-compensatory objective\. The objective*removes cross\-episode compensation*, so it cannot be satisfied by being ethical on average, but it does not guarantee a clean episode, since policies and environments stay stochastic\.

Our contributions are as follows:

- •A confound\-controlled comparison of four training methods for ethical RL behavior, with controls separating the objective \(ESR vs\. SER\), the observation \(SER\+Racc\+R\_\{\\mathrm\{acc\}\}\), the utility source \(stated vs\. distilled\), and the discounting approximation \(γ=1\\gamma\{=\}1\)\.
- •An evaluation methodology, a shared detector\-only environment, uncensored counts, and per\-episode distributional metrics, with the argument that mean rates cannot distinguish “ethical” from “ethical on average”\.
- •A reward\-isolated ethical\-dilemma benchmark in Craftax, three dilemmas×\\timesthree enforcement mechanisms, released with code and training pipeline\.
- •We find that the four methods are indistinguishable based on the mean frontier approach yet cleanly separated per episode\. The non\-compensatory objective holds its budget in effectively every episode at no cost in mean return, and the observation control attributes the per\-episode budget ceiling to the objective rather than to what the agent sees\.

## 2Related Work

#### Specification gaming\.

Reward hacking, an agent exploiting the gap between a proxy reward and the intended one, is a well known safety concern\(Krakovnaet al\.[2020](https://arxiv.org/html/2608.14642#bib.bib1); Amodeiet al\.[2016](https://arxiv.org/html/2608.14642#bib.bib3)\), made precise by asking when improving a proxy can lower the true reward\(Skalseet al\.[2022](https://arxiv.org/html/2608.14642#bib.bib2)\)\. We study it in an open\-ended world where such shortcuts are plentiful and cheap to repeat\.

#### Multi\-Objective Reinforcement Learning \(MORL\)\.

MORL keeps objectives as a vector and applies a utility to act\(Roijerset al\.[2013](https://arxiv.org/html/2608.14642#bib.bib7); Hayeset al\.[2022](https://arxiv.org/html/2608.14642#bib.bib8)\), and most systems fix a linear utility with hand\-chosen weights\. A weight sweep recovers only the convex hull of the achievable front\(Vamplewet al\.[2011](https://arxiv.org/html/2608.14642#bib.bib9)\), whereas non\-linear scalarizations reach more\(Van Moffaert and Nowé[2014](https://arxiv.org/html/2608.14642#bib.bib10)\), and our stated utility is a smoothed thresholded\-lexicographic ordering\(Gáboret al\.[1998](https://arxiv.org/html/2608.14642#bib.bib11)\)\. More important here is*when*the utility applies\. Under Scalarized Expected Returns \(SER\) it follows the expectation, under Expected Scalarized Returns \(ESR\) it precedes it\(Roijerset al\.[2018](https://arxiv.org/html/2608.14642#bib.bib12); Rădulescuet al\.[2020](https://arxiv.org/html/2608.14642#bib.bib13)\)\. We take the ESR view, since an ethical violation happens within a single episode\. Algorithmically we are closest to the ESR actor\-critic ofReymondet al\.\([2023](https://arxiv.org/html/2608.14642#bib.bib14)\), which conditions the policy on the accrued reward to optimize a non\-linear utility\. Our ESR\-PPO is that idea in PPO form, the vehicle rather than the contribution, and what we add is the per\-episode ethical budget it encodes and the controlled, distribution\-level comparison against the linear alternatives\.

#### Safe and constrained reinforcement learning\.

A separate tradition enforces safety as a constraint\. Constrained MDPs cap expected cost\(Altman[2021](https://arxiv.org/html/2608.14642#bib.bib15)\), many methods respect such limits\(García and Fernández[2015](https://arxiv.org/html/2608.14642#bib.bib16); Achiamet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib17)\), and shielding blocks unsafe actions outright\(Alshiekhet al\.[2018](https://arxiv.org/html/2608.14642#bib.bib18)\)but leaves the safety in the machinery rather than the agent\(Ghasemi and Crowley[2026](https://arxiv.org/html/2608.14642#bib.bib22)\)\. We do not use this approach, but we do implement a dual\-ascent Lagrangian\(Altman[2021](https://arxiv.org/html/2608.14642#bib.bib15); Achiamet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib17); Rayet al\.[2019](https://arxiv.org/html/2608.14642#bib.bib19)\)as a first\-class baseline, since an expected\-cost constraint is exactly what a per\-episode claim must beat\. Saute RL\(Sootlaet al\.[2022](https://arxiv.org/html/2608.14642#bib.bib20)\)augments the state with a cost budget to satisfy a hard cap almost surely, mechanically like our accrued\-return augmentation, but we optimize a graded utility over the return vector, not a single cap\. Risk\-sensitive RL controls tails via CVaR or chance constraints\(Chowet al\.[2018](https://arxiv.org/html/2608.14642#bib.bib21)\)\. We use CVaR to*evaluate*all methods, not to train\.

#### Learning rewards from preferences\.

Preference\-based reward learning fits a model to trajectory comparisons\(Wirthet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib23)\), often with a Bradley–Terry likelihood\(Bradley and Terry[1952](https://arxiv.org/html/2608.14642#bib.bib25); Christianoet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib24)\)\. What is learned depends on where the comparisons come from\. Most use human raters, whereas we generate them mechanically from a hand\-coded ethics\-gated rule plus a monotonicity prior\. Our fitted utility thus*distills*a stated ordering into a differentiable function, an ablation on whether fitting helps rather than ethics learned from behavior\. The pool it is fit to simply reuses agents trained elsewhere in this study\.

#### Machine ethics as multiple objectives\.

Alignment is widely framed as pluralistic\(Sorensenet al\.[2024](https://arxiv.org/html/2608.14642#bib.bib26); Grahamet al\.[2013](https://arxiv.org/html/2608.14642#bib.bib27)\), andVamplewet al\.\([2018](https://arxiv.org/html/2608.14642#bib.bib28)\)argue that ethical, legal, and safety constraints are competing objectives a linear scalar utility cannot serve\. Closest to us,Rodriguez\-Sotoet al\.\([2021](https://arxiv.org/html/2608.14642#bib.bib29)\)embed ethics in a multi\-objective process but weight it linearly as designers, and MORAL\(Peschlet al\.[2022](https://arxiv.org/html/2608.14642#bib.bib30)\)tunes a distribution over linear weights through human queries\. We differ by using a non\-linear utility optimized under ESR, so each episode is judged on its own, and the non\-linearity is what lets a breach resist averaging\. We run in Craftax\(Matthewset al\.[2024](https://arxiv.org/html/2608.14642#bib.bib4)\), an open\-ended JAX benchmark in the lineage of Crafter\(Hafner[2022](https://arxiv.org/html/2608.14642#bib.bib5)\)and NetHack\(Küttleret al\.[2020](https://arxiv.org/html/2608.14642#bib.bib6)\), and its speed doubles as a tool to stress test various things\.

## 3Training Methods

### 3\.1Problem Setting and Optimization Criteria

We model each dilemma as a multi\-objective MDP⟨𝒮,𝒜,P,𝐫,γ⟩\\langle\\mathcal\{S\},\\mathcal\{A\},P,\\mathbf\{r\},\\gamma\\ranglewith a two\-component reward𝐫t=\[rext,t,reth,t\]\\mathbf\{r\}\_\{t\}=\[\\,r\_\{\\mathrm\{ext\},t\},\\;r\_\{\\mathrm\{eth\},t\}\\,\]\. Hererext,tr\_\{\\mathrm\{ext\},t\}is the native game reward, andreth,t∈\{0,−ρ\}r\_\{\\mathrm\{eth\},t\}\\in\\\{0,\-\\rho\\\}is an ethical penalty that is nonzero iff an own\-dilemma violation occurs at steptt, withρ=10\\rho=10\. Every utility function then refers to𝐆τ\\mathbf\{G\}\_\{\\tau\}, the episode’s \(undiscounted\) vector return:

𝐆τ=\[Rext,Reth\]=∑t=0T−1𝐫t\.\\mathbf\{G\}\_\{\\tau\}\\;=\\;\\big\[\\,R\_\{\\mathrm\{ext\}\},\\;R\_\{\\mathrm\{eth\}\}\\,\\big\]\\;=\\;\\sum\_\{t=0\}^\{T\-1\}\\mathbf\{r\}\_\{t\}\.\(1\)A utilityu:ℝ2→ℝu:\\mathbb\{R\}^\{2\}\\to\\mathbb\{R\}turns the vector into one optimizable number, and the two optimization criteria differ in where the expectation sits\(Roijerset al\.[2013](https://arxiv.org/html/2608.14642#bib.bib7); Hayeset al\.[2022](https://arxiv.org/html/2608.14642#bib.bib8)\):

VSERπ=u​\(𝔼τ∼π​\[𝐆τ\]\),VESRπ=𝔼τ∼π​\[u​\(𝐆τ\)\]\.V^\{\\pi\}\_\{\\mathrm\{SER\}\}\\;=\\;u\\big\(\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\[\\mathbf\{G\}\_\{\\tau\}\]\\big\),\\qquad V^\{\\pi\}\_\{\\mathrm\{ESR\}\}\\;=\\;\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\big\[u\(\\mathbf\{G\}\_\{\\tau\}\)\\big\]\.\(2\)For a linearu​\(𝐠\)=𝐰⊤​𝐠u\(\\mathbf\{g\}\)=\\mathbf\{w\}^\{\\top\}\\mathbf\{g\}the two coincide by linearity of expectation\. They can differ only for non\-linearuu\. SER permits cross\-episode compensation \(a bad episode offset by good ones\), while ESR judges every episode on its own totals\. Two structural facts shape the experiments\. First, for any fixed𝐰⪰0\\mathbf\{w\}\\succeq 0the maximizers of𝐰⊤​𝔼​\[𝐆τ\]\\mathbf\{w\}^\{\\top\}\\mathbb\{E\}\[\\mathbf\{G\}\_\{\\tau\}\]lie on the convex hull of the achievable expected\-return set, so a weight sweep can only recover hull points\(Roijerset al\.[2013](https://arxiv.org/html/2608.14642#bib.bib7); Vamplewet al\.[2011](https://arxiv.org/html/2608.14642#bib.bib9)\), though non\-linear scalarization can express more\(Van Moffaert and Nowé[2014](https://arxiv.org/html/2608.14642#bib.bib10)\)\. Second, and decisive for how results must be read,*any*statistic that is linear in the state\-action occupancy measure, including the mean return\-versus\-violation frontier itself, cannot separate a per\-episode objective from an in\-expectation one\. The separation, if it exists, must appear in higher moments of the per\-episode distribution\. Our evaluation is designed around this point\.

### 3\.2Dilemmas and the Ethical Penalty Layer

We work in Craftax\(Matthewset al\.[2024](https://arxiv.org/html/2608.14642#bib.bib4)\), where the agent maximizes the native game reward \(achievement scores plus health shaping\)\. On top of it we define three dilemmas, each a situation where the unethical action genuinely earns more score\.

InNo Killing, passive creatures \(cows, bats, snails\) are easy food and a violation is killing one, and farming \(placing and eating plants\) is a viable ethical alternative, so a high\-return ethical policy exists\.

InSustainability, wood is valuable, and each unit harvested past a per\-episode budget of 8 \(below the unconstrained agent’s typical harvest, so the constraint binds\) is a violation\.

InProportional Force, hunting mobs to clear the area is easy, but a violation is killing a hostile creature \(melee or ranged\) that was more than 5 tiles away before the step, while closer kills count as defensive\.

The ethical layer is an event\-triggered penalty wrapper\. A JAX\-native detector classifies each step, and the violating event of the run’s own dilemma receives a fixed−10\-10penalty \(no positive shaping; each configuration touches only its own dilemma\), so the native reward stays dominant and ethics enters as a margin, not a competing reward stream to game\. Violations are detected from creature\-death state differences \(Section[4](https://arxiv.org/html/2608.14642#S4)\), not Craftax’s achievement flags, which censor repeats\. For scalar training the reward isrt=rext,t\+reth,tr\_\{t\}=r\_\{\\mathrm\{ext\},t\}\+r\_\{\\mathrm\{eth\},t\}under one of three enforcement mechanisms, mechanical rather than philosophical\.

Absoluteterminates at the first violation,calculatedterminates once the cumulative penalty passes−30\-30\(three are tolerated, the fourth ends it\), andheuristicpenalizes but continues\. The wrapper carries a reward\-machine\-style automaton whose state never affects rewards here\. We describe the layer as an event\-triggered penalty and nothing more\.

### 3\.3Linear Baselines: SER Weight Sweep and Lagrangian Constraint

SER \(linear MORL\)\.The reward is the vector\[rext,reth\]\[r\_\{\\mathrm\{ext\}\},r\_\{\\mathrm\{eth\}\}\], scalarized aswext​rext\+weth​rethw\_\{\\mathrm\{ext\}\}r\_\{\\mathrm\{ext\}\}\+w\_\{\\mathrm\{eth\}\}r\_\{\\mathrm\{eth\}\}before the advantage computation, so a single scalar critic suffices\(Roijerset al\.[2013](https://arxiv.org/html/2608.14642#bib.bib7)\)\. Sweepingwethw\_\{\\mathrm\{eth\}\}on a log scale\{0\.1,0\.3,1,3,10\}\\\{0\.1,0\.3,1,3,10\\\}traces a return\-versus\-violation trade\-off curve\. For a linear utility, SER and ESR coincide, and the sweep can only recover points on the convex hull of the achievable front\(Vamplewet al\.[2011](https://arxiv.org/html/2608.14642#bib.bib9)\)\.

Lagrangian \(Constrained MDP\)\.This baseline comes from the constrained tradition and uses the same vector reward\. With per\-step costct=max⁡\(0,−reth,t\)/ρc\_\{t\}=\\max\(0,\-r\_\{\\mathrm\{eth\},t\}\)/\\rho\(the violations at steptt\) and episodic costCτ=∑tctC\_\{\\tau\}=\\sum\_\{t\}c\_\{t\}, the CMDP is

maxπ⁡𝔼τ∼π​\[Rext\]s\.t\.𝔼τ∼π​\[Cτ\]≤d,\\max\_\{\\pi\}\\;\\;\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\big\[R\_\{\\mathrm\{ext\}\}\\big\]\\quad\\text\{s\.t\.\}\\quad\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\big\[C\_\{\\tau\}\\big\]\\;\\leq\\;d,\(3\)optimized through the relaxed rewardr~t=rext,t−λ​ct\\tilde\{r\}\_\{t\}=r\_\{\\mathrm\{ext\},t\}\-\\lambda\\,c\_\{t\}with one dual\-ascent step per policy update,

λ←clip​\(λ\+η​\(C^−d\),0,λmax\),\\lambda\\;\\leftarrow\\;\\mathrm\{clip\}\\\!\\big\(\\lambda\+\\eta\\,\(\\hat\{C\}\-d\),\\;0,\\;\\lambda\_\{\\max\}\\big\),\(4\)whereC^\\hat\{C\}is the empirical mean episodic cost in the update window \(held when the window completes no episodes\)\. The budgetddis swept over\{0\.5,1,2,4\}\\\{0\.5,1,2,4\\\}violations per episode\. As the principled\-constraint baseline, this remains a linear combination in expectation and is subject to the same convex\-hull and on\-average limitations\. Both baselines train on the penalty\-only \(heuristic\) configurations where their trade\-off is set by the weight or budget, but never by termination\.

### 3\.4A Non\-Compensatory Utility over Episode Returns

The utility is defined on the episode’s accumulated pair\[Rext,Reth\]\[R\_\{\\mathrm\{ext\}\},R\_\{\\mathrm\{eth\}\}\], not per step, in two variants\.

Stated \(primary\)\.An explicit smoothed thresholded\-lexicographic \(TLO\) form\(Gáboret al\.[1998](https://arxiv.org/html/2608.14642#bib.bib11)\)\. With the episode’s violation countv=max⁡\(0,−Reth\)/ρv=\\max\(0,\-R\_\{\\mathrm\{eth\}\}\)/\\rhoand the logisticσ​\(x\)=\(1\+e−x\)−1\\sigma\(x\)=\(1\+e^\{\-x\}\)^\{\-1\},

uTLO​\(Rext,Reth\)=σ​\(s​\(τtol−v\)\)​Rext−ρu​v,u\_\{\\mathrm\{TLO\}\}\\big\(R\_\{\\mathrm\{ext\}\},R\_\{\\mathrm\{eth\}\}\\big\)\\;=\\;\\sigma\\\!\\big\(s\\,\(\\tau\_\{\\mathrm\{tol\}\}\-v\)\\big\)\\;R\_\{\\mathrm\{ext\}\}\\;\-\\;\\rho\_\{u\}\\,v,\(5\)with sharpnesss=12s=12\. The gate multiplies return credit by≈1\\approx 1while the episode is within tolerance and by≈0\\approx 0once it is not, and every violation additionally costsρu\\rho\_\{u\}, making the utility non\-compensatory in both directions\. All moral parameters are stated, visible, and hand\-chosen\. The form also givesuTLO​\(𝟎\)=0u\_\{\\mathrm\{TLO\}\}\(\\mathbf\{0\}\)=0, which the telescoping identity below requires\. The zero\-tolerance instance \(τtol=0\.5\\tau\_\{\\mathrm\{tol\}\}=0\.5,ρu=10\\rho\_\{u\}=10\) encodes strict compliance\. The*budget family*generalizes it\. Settingτtol=k\+0\.5\\tau\_\{\\mathrm\{tol\}\}=k\+0\.5tolerateskkviolations per episode before the gate closes, with a small within\-budget charge \(ρu=1\\rho\_\{u\}=1, deliberately below the marginal return value of a violation on No Killing, about2\.52\.5, so the budget region is actually exercised\), swept overk∈\{1,2,4\}k\\in\\\{1,2,4\\\}on the binding dilemmas\. This is ESR’s analogue of the SER weight sweep and the Lagrangian budget grid, a family of exactly stated tolerance\-kkpreferences\. It carries the property the experiments test, namely that a per\-episode budget is a preference that no linear, occupancy\-additive objective can express\.

Distilled \(secondary\)\.We roll out a pool of trained agents \(unconstrained, the ERM agents, the SER sweep\), record each episode’s\[Rext,Reth\]\[R\_\{\\mathrm\{ext\}\},R\_\{\\mathrm\{eth\}\}\], and label preference pairs by the ethics\-gated rule above plus a monotonicity prior\. A small MLPuθu\_\{\\theta\}minimizes the Bradley–Terry loss\(Bradley and Terry[1952](https://arxiv.org/html/2608.14642#bib.bib25); Christianoet al\.[2017](https://arxiv.org/html/2608.14642#bib.bib24)\)over winners𝐆w\\mathbf\{G\}\_\{w\}and losers𝐆l\\mathbf\{G\}\_\{l\},

ℒ​\(θ\)=−𝔼\(w,l\)​\[log⁡σ​\(uθ​\(𝐆w\)−uθ​\(𝐆l\)\)\],\\mathcal\{L\}\(\\theta\)\\;=\\;\-\\,\\mathbb\{E\}\_\{\(w,l\)\}\\Big\[\\log\\sigma\\big\(u\_\{\\theta\}\(\\mathbf\{G\}\_\{w\}\)\-u\_\{\\theta\}\(\\mathbf\{G\}\_\{l\}\)\\big\)\\Big\],\(6\)and its output is rescaled at load time to game\-return units \(a positive constant, preserving the ordering and the ESR argmax\)\. The labels are ours, so the fitted utility*distills*a stated ordering\. The variant tests whether fitting helps or hurts relative to stating it\.

### 3\.5ESR\-PPO: Optimizing the Utility per Episode

The ESR trainer maximizesVESRπ=𝔼​\[u​\(𝐆τ\)\]V^\{\\pi\}\_\{\\mathrm\{ESR\}\}=\\mathbb\{E\}\[\\,u\(\\mathbf\{G\}\_\{\\tau\}\)\\,\]of Eq\. \([2](https://arxiv.org/html/2608.14642#S3.E2)\), the utility of each episode’s own totals, rather thanVSERπ=u​\(𝔼​\[𝐆τ\]\)V^\{\\pi\}\_\{\\mathrm\{SER\}\}=u\(\\mathbb\{E\}\[\\mathbf\{G\}\_\{\\tau\}\]\)\(Roijerset al\.[2018](https://arxiv.org/html/2608.14642#bib.bib12); Rădulescuet al\.[2020](https://arxiv.org/html/2608.14642#bib.bib13); Hayeset al\.[2022](https://arxiv.org/html/2608.14642#bib.bib8)\)\. Under a non\-compensatoryuu, a violating episode cannot be paid for by other episodes\. Two standard devices make this trainable with PPO\. First, a telescoping pseudo\-reward\. With the within\-episode accumulator𝐑tacc=∑k≤t𝐫k\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}=\\sum\_\{k\\leq t\}\\mathbf\{r\}\_\{k\}\(reset at episode start,𝐑−1acc=𝟎\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{\-1\}=\\mathbf\{0\}\), each step’s scalar reward is the utility increment

rtu\\displaystyle r^\{u\}\_\{t\}=u​\(𝐑tacc\)−u​\(𝐑t−1acc\),\\displaystyle\\;=\\;u\\big\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\\big\)\-u\\big\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\-1\}\\big\),\(7\)∑t=0T−1rtu\\displaystyle\\sum\_\{t=0\}^\{T\-1\}r^\{u\}\_\{t\}=u​\(𝐆τ\)−u​\(𝟎\)=u​\(𝐆τ\)\.\\displaystyle\\;=\\;u\\big\(\\mathbf\{G\}\_\{\\tau\}\\big\)\-u\(\\mathbf\{0\}\)\\;=\\;u\\big\(\\mathbf\{G\}\_\{\\tau\}\\big\)\.so standard PPO/GAE onrur^\{u\}optimizes the ESR objective\. The identity is exact atγ=1\\gamma=1but forγ<1\\gamma<1it introduces a bias we quantify with aγ=1\\gamma=1arm\. Second, since the ESR\-optimal action depends on what has accumulated \(the distance to the gate makes the criterion non\-stationary in the plain state\), the observation is extended to𝐨~t=\[𝐨t;ν​\(𝐑tacc\)\]\\tilde\{\\mathbf\{o\}\}\_\{t\}=\[\\,\\mathbf\{o\}\_\{t\}\\,;\\;\\nu\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\)\\,\]withν\\nuas a fixed normalizer\. Both devices follow the accrued\-return ESR line ofReymondet al\.\([2023](https://arxiv.org/html/2608.14642#bib.bib14)\)\. They are the vehicle, not the contribution\. The augmentation is a potential confound, so we also runSER\+Racc\+R\_\{\\mathrm\{acc\}\}, which is an identical setup with a*linear*utility, whose increments collapse to the per\-step SER reward𝐰⊤​𝐫t\\mathbf\{w\}^\{\\top\}\\mathbf\{r\}\_\{t\}, i\.e\. the SER objective under the ESR observation\. If ESR wins only through the extra input, this control reproduces the win\. If it does not, the objective is responsible\. Everything else \(network, hyperparameters, environment stack\) is identical across methods, isolating the objective\.

## 4Evaluation Methodology

Every trained agent, however trained, is rolled out in the same detector\-only environment \(the detector runs but adds no reward and never terminates\)\. The sustainability budget is read from each run’s training config so all face the same threshold\.

Uncensored counting\.Violations are counted from creature\-death state differences, with per\-step counts \(a step where two creatures are killed, counts as two\)\. The alternative, Craftax’s achievement flags, latches once per creature type per episode which would understate unsafe agents and make per\-episode dispersion a deterministic function of the mean \(std=m​\(1−m\)\\mathrm\{std\}=\\sqrt\{m\(1\-m\)\}for a near\-binary count\), voiding the distributional comparison\. The detector is unit\-tested both ways, so that a never\-attacking policy registers zero kills and an always\-attacking one registers many\.

Distributions and not only means\.For per\-episode violation counts\{Vi\}i=1N\\\{V\_\{i\}\\\}\_\{i=1\}^\{N\}we report the mean and standard deviation, the violating fractionP\(≥1\)=1N​∑i𝟏​\[Vi≥1\]P\(\\geq 1\)=\\tfrac\{1\}\{N\}\\sum\_\{i\}\\mathbf\{1\}\[V\_\{i\}\\geq 1\], and the tail meanCVaR0\.1\\mathrm\{CVaR\}\_\{0\.1\}\(mean of the worst⌈0\.1​N⌉\\lceil 0\.1N\\rceilepisodes\)\. The main metric is the*matched\-return tail comparison*\. Each ESR point is paired with the return\-nearest SER weight, Lagrangian budget, and SER\+Racc\+R\_\{\\mathrm\{acc\}\}control point on the same dilemma, and we compare distributions at equal mean returns\. This is the only place where a tail difference is attributable to the objective rather than to a position on the curve\. We also report the frontier, the cost of ethicsgap​\(m\)=R¯uncon−R¯m\\mathrm\{gap\}\(m\)=\\bar\{R\}\_\{\\mathrm\{uncon\}\}\-\\bar\{R\}\_\{m\}with a variance\-propagated 95% CI overn=3n=3seeds, and hypervolume, but theγ=1\\gamma=1component is reported separately\.

## 5Experimental Setup

All experiments use Craftax\-Symbolic\-v1\(Matthewset al\.[2024](https://arxiv.org/html/2608.14642#bib.bib4)\)\. Episodes end by in\-game death, by timeout, or, during ERM training only, by the enforcement mechanism\. Every method trains the same agent, PPO with a GRU recurrent actor\-critic \(hidden size 512\), 1024 parallel environments,10910^\{9\}environment steps per run, Adam with linear learning\-rate decay,γ=0\.99\\gamma=0\.99, and GAEλ=0\.8\\lambda=0\.8\(full hyperparameter table in the supplementary material\)\. The run matrix uses 3 seeds throughout\. It covers ERM at 3 dilemmas×\\times3 enforcements plus unconstrained \(30 runs\), SER at 3 dilemmas×\\times5 weights \(45\), Lagrangian at 3 dilemmas×\\times4 budgets \(36\), and ESR at 3 dilemmas×\\times\{stated, distilled\} \(18\)\. To these we add the budget arms on the two binding dilemmas,k∈\{1,2,4\}k\\in\\\{1,2,4\\\}\(18\), the SER\+Racc\+R\_\{\\mathrm\{acc\}\}control at two matched weights \(18\), and theγ=1\\gamma=1twins \(18\), for 183 runs in total\. All agents are scored by one evaluation job of 64 environments×\\times4096 steps per agent \(≈2,400\\approx 2\{,\}400completed episodes per operating point across seeds\)\. Each training run takes≈3\\approx 3hours on a single H100 GPU \(≈92\\approx 92k steps/s\) on national HPC clusters\. The ESR stage was retrained from scratch on a second cluster after a mid\-study migration and reproduced its results\.

## 6Results

### 6\.1Mean Return and Violation Rates

As intended, all four methods reach a common return\-versus\-violation frontier, so the aggregate view cannot separate them\. Every method trains stably, with final returns between 30 and 37 \(Figure[1](https://arxiv.org/html/2608.14642#S6.F1)\)\. The unconstrained agent scores37\.05±0\.7837\.05\\pm 0\.78and is genuinely tempted, killing a passive creature in 95% of No\-Killing episodes and over\-harvesting about 5 units per Sustainability episode\. Driving violations to near zero costs every method roughly the same amount\. Mean return falls to 31\.5–31\.9 for ERM, 30\.9–31\.0 for the strictest SER weights, and 30\.9–31\.1 for both ESR variants, in each case about 84% of the unconstrained return\. This 5–6 point gap is the price of compliance in this environment rather than a property of any single method\. The pattern holds along the whole frontier \(Figure[2](https://arxiv.org/html/2608.14642#S6.F2)\)\. The ESR budget points lie on the SER and Lagrangian envelope rather than above it, and adding ESR to the coverage set raises hypervolume only slightly \(0\.0264 against 0\.0249 on No Killing\)\. This is what our problem setting anticipates, and it is the outcome we want\. The mean return\-versus\-violation frontier is linear in the state\-action occupancy measure, so every point on it is attainable by some linear weighting, and a per\-episode objective cannot distinguish itself there\. A method that appeared to beat the linear baselines here would point to under\-tuned baselines rather than to a better objective\. Any difference between the methods must instead appear in how violations are distributed across episodes, which is what the matched\-return comparison below is designed to expose\.

![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/training_return_main.png)Figure 1:Training return on No Killing \(mean with min–max band over 3 seeds\)\. Every method family converges with the same shape, and the compliant methods plateau together about 6 points below the unconstrained agent, so the shared frontier of Figure[2](https://arxiv.org/html/2608.14642#S6.F2)reflects the cost of compliance rather than an under\-tuned baseline\.![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/frontier_main.png)Figure 2:No Killing frontier in per\-episode units \(mean±\\pm95% CI, 3 seeds, right axis % of unconstrained return\)\. The SER weight sweep and Lagrangian budgets trace one shared envelope, and the best point of each ESR version \(stated budgetk=1k\{=\}1and distilled\) lands on it\. The shaded band marks the matched\-return comparison of Figures[3](https://arxiv.org/html/2608.14642#S6.F3)–[4](https://arxiv.org/html/2608.14642#S6.F4)\. The full ESR family is Figure[5](https://arxiv.org/html/2608.14642#S6.F5), and the second binding dilemma \(Sustainability\) is in the supplementary material\.
### 6\.2Per\-Episode Violation Distributions

The methods separate clearly once we examine how violations are distributed across episodes rather than only their average\. We compare three agents at closely matched mean return on No Killing and read off their per\-episode violation counts \(Figure[3](https://arxiv.org/html/2608.14642#S6.F3), Table[1](https://arxiv.org/html/2608.14642#S6.T1)\)\. The ESR budget agent and the Lagrangian sit at 34\.2 and 34\.3, and the return\-nearest SER weight sits a little higher at 35\.3, so the comparison is if anything generous to SER\. The ESR budget agent \(k=1k\{=\}1\) stays within its stated budget of one violation per episode\. Across roughly 2,400 episodes it exceeds the budget in 0\.4% of them and never records three violations, and its worst\-decile mean isCVaR10%=1\.04±0\.07\\mathrm\{CVaR\}\_\{10\\%\}=1\.04\\pm 0\.07\. The Lagrangian \(b=0\.5b\{=\}0\.5\) meets the same budget on average but not within each episode\. In 1\.3% of its episodes it exceeds one violation, and its worst decile averages1\.14±0\.031\.14\\pm 0\.03\(Welcht≈5\.5t\\approx 5\.5against ESR\)\. The return\-nearest SER weight \(w=0\.1w\{=\}0\.1\) has a heavier tail again, with 13\.8% of episodes over budget and a worst decile of2\.20±0\.202\.20\\pm 0\.20\(t≈23t\\approx 23\), twice the ESR agent’s\. The SER\+Racc\+R\_\{\\mathrm\{acc\}\}control isolates the cause\. It gives the linear objective the same accrued\-return observation as ESR, yet it stays on the SER frontier rather than reaching ESR’s operating point\. At its own return of 32\.5 its tail matches the nearby SER weight \(CVaR 1\.12 against 1\.18 for plain SER at 32\.9\), not ESR’s\. The tighter tail therefore comes from the objective, not from the extra input\.

Widening the tolerance does not change the picture \(Figure[4](https://arxiv.org/html/2608.14642#S6.F4)\)\. Raisingkkfrom 1 to 2 to 4 leaves the mean return essentially flat \(34\.15, 34\.00, 33\.24\) while the worst\-decile tail grows toward, but never past, each new ceiling \(CVaR 1\.04, 1\.50, 2\.10\)\. In this dilemma, the temptation is concentrated in the first violation\. The same ordering holds on Sustainability more sharply, there the agents match at about 34\.5 mean return\. The ESR budget agent’s worst\-decile tail isCVaR10%=1\.14\\mathrm\{CVaR\}\_\{10\\%\}=1\.14against the return\-nearest Lagrangian’s 3\.73, and it exceeds one violation in 10% of episodes against the Lagrangian’s 55%\. Sustainability is a weaker test in other respects, because our single within\-budget charge ofρu=1\\rho\_\{u\}=1sat close to the marginal return value of a violation there, about 1\.1 against about 2\.5 on No Killing, leaving little incentive to spend the budget, which was therefore only lightly exercised\.

A single strong SER weight dominates the region on the mean, and one seed collapses, but the per\-episode gap runs in the same direction as on No Killing\. Stated precisely, the objective removes cross\-episode compensation and, on No Killing, holds violations within budget in 99\.6% of episodes\. It does not make any individual episode certain to be clean, since the policy and environment remain stochastic\.

![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/claimb_distribution.png)Figure 3:Per\-episode violation\-count distributions for the three return\-matched agents on No Killing \(about 2,400 episodes each\)\. The ESR budget agent’s mass stops at its one\-violation budget, with 0\.4% of episodes at two kills and none beyond\. The Lagrangian reaches three, and the matched SER weight develops a longer tail \(12\.1% of episodes at two kills and 1\.7% at three or more\)\.![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/budget_sweep.png)Figure 4:Budget sweep on No Killing\. Widening the tolerancek=1→2→4k=1\\rightarrow 2\\rightarrow 4buys no return \(top\) and only widens the worst\-decile tail toward, but not past, the budget ceiling \(bottom\)\. The temptation is concentrated in the first violation\.![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/esr_variations.png)Figure 5:The ESR family on the No\-Killing frontier, with the SER/Lagrangian envelope in gray for reference\. The stated budget sweep \(k=1,2,4k=1,2,4\) climbs the envelope, and the strict stated point \(k=0k\{=\}0\) and the distilled utilityuθu\_\{\\theta\}sit together at the compliance corner\. The variants choose*where*on the shared frontier to operate\.Table 1:Per\-episode violation statistics for the three agents compared at closely matched mean return on No Killing\. The return\-nearest SER weight sits a little higher at 35\.3, so the comparison favors SER, yet the non\-compensatory ESR objective still holds the tightest tail\. The SER\+Racc\+R\_\{\\mathrm\{acc\}\}control is discussed in the text, and complete tables for every method and dilemma are released with the code\.
### 6\.3The Cost of Compliance

A tighter tail might be expected to cost return, but it does not\. At the strict\-compliance corner every method loses between 5\.2 and 6\.2 return relative to the unconstrained agent \(about 16%\), and at a one\-violation budget the ESR and Lagrangian agents lose statistically the same amount \(2\.90±1\.062\.90\\pm 1\.06and2\.72±1\.702\.72\\pm 1\.70, Table[2](https://arxiv.org/html/2608.14642#S6.T2)\)\. The per\-episode guarantee therefore adds no return penalty over the linear baselines\. Proportional Force is a limitation of the benchmark\. Our detector shows that the constraint does not bind, since the unconstrained agent strikes preemptively only1\.4×10−51\.4\\times 10^\{\-5\}of the time per step\. There the distilled utility, fitted to a pool whose worst episode contains a single violation, loses about 1\.5 return for no gain in safety, the failure mode that our stated\-versus\-distilled comparison is designed to reveal \(supplementary material\)\.

Table 2:Cost of compliance on No Killing\. We report mean return, own\-dilemma violation rate,P\(≥1\)P\(\\geq 1\), and gap to the unconstrained agent \(means over three seeds\)\. Every compliant method pays a similar 5–6 point gap at the strict corner, and the ESR budget arm and the Lagrangian pay the same to operate at a one\-violation budget\. Full tables for all methods and dilemmas are released with the code\.
### 6\.4Ablations and Robustness Checks

ESR variants\.Figure[5](https://arxiv.org/html/2608.14642#S6.F5)places the full ESR family on the No\-Killing frontier\. The stated budget sweep runs along the frontier from the compliance corner upward, and the strict stated point \(k=0k\{=\}0\) and the distilled utility coincide at the corner\. The variants differ in where on the frontier they operate, not in whether they reach it\. Adding ESR to the coverage set raises hypervolume only slightly \(\+6%\+6\\%on No Killing\), consistent with the shared frontier\.

Stated versus distilled utility\.The two match at the corner \(30\.9 against 31\.1 return\), but the distilled utility is noisier wherever its training pool held few violations\. On Proportional Force it scores35\.2±4\.535\.2\\pm 4\.5against the stated utility’s36\.7±1\.136\.7\\pm 1\.1\(utility surfaces in the supplementary material\)\. Fitting the utility adds nothing over stating it and can cost return\.

Discounting\.Comparingγ=1\\gamma=1withγ=0\.99\\gamma=0\.99shows that the telescoping approximation introduces a real but dilemma\-dependent bias, costing about 3 return on No Killing at held compliance while improving Sustainability\. We report it as a sensitivity check, and the No\-Killing result is stable across both settings\.

Reproducibility\.Every method family converges without collapse\. The ESR runs reproduced their corner returns to within±0\.3\\pm 0\.3when retrained from scratch on different hardware\.

## 7Discussion

Every method that optimizes expected returns shares one blind spot\. A bad episode can be offset by good ones\. Whether that trade is available is decided by the shape of the utility, not by the training algorithm\. This gives our results a two\-sided reading\. Judged on mean return and mean violation rate, as most multi\-objective and constrained RL is judged, the four methods are interchangeable and the Lagrangian is the simplest good choice\. Judged per episode, the three expectation\-level methods spread their violations unevenly across episodes, which for an ethical constraint is the behavior that matters\. The per\-episode evaluation is therefore not a secondary diagnostic\. It is what makes the effect visible\. In practical terms, a per\-episode budget utility is the right tool when compliance must hold within each episode, an expected\-cost constraint is enough when only long\-run rates matter, and a utility should be distilled from data only when that data contains violations to learn from\.

#### Relation to risk\-sensitive RL\.

Because we report CVaR, our approach may be read as risk\-sensitive RL, but the two act on different quantities\. Risk\-sensitive methods optimize a tail statistic, a CVaR or a chance constraint, of the*return*distribution, reshaping the spread of rewards, and they take the risk level as a training input\. We instead make the per\-episode ethics budget the argument of the utility and use CVaR only to evaluate, so the tail we control is the tail of the*violation*count rather than of return\. The two are therefore complementary rather than competing, and a CVaR\-constrained agent would be a natural additional baseline\.

#### Scope\.

We do not claim that RL should abandon mean\-based evaluation in general\. The argument applies when violations do not compensate across episodes\. Ethical harms are of this kind, since a harm in one episode is not undone by good conduct in another, and their natural target is the per\-episode distribution that a mean averages away\. When a constraint really is about long\-run behavior, such as a power or bandwidth budget or an average\-cost target, violations are fungible across episodes, the mean is the correct target, and a simpler in\-expectation method is appropriate\. Our claim is that in the non\-compensatory case both the training objective and the evaluation statistic should move from the expectation to the distribution\. We demonstrate this in one environment and expect it to hold wherever that structure is present, rather than for RL evaluation as a whole\.

#### Limitations\.

Our proportionality constraint is defined by distance alone, so an agent that kills a distant ranged attacker is counted as striking preemptively, and the dilemma captures distance rather than threat\. The training penalty is applied per step and slightly under\-penalizes the rare step that removes two creatures at once, although measurement uses exact counts\. Separately, a kill is missed if a new creature spawns into the same slot within the step\. The telescoping identity is exact only atγ=1\\gamma=1\. Atγ=0\.99\\gamma=0\.99it introduces a bounded bias of up to 3–4 return that we measure rather than remove, shared across all ESR runs\. Finally, we use three seeds, so intervals are wide\. The main tail comparisons are significant \(Welcht≈5\.5t\\approx 5\.5and2323\), but one Sustainability seed collapsed, and all results come from a single environment with distillation labels generated by our own rule\.

## 8Conclusion and Future Work

We compared four ways of training ethical behavior across three Craftax dilemmas under a single per\-episode evaluation protocol\. The four methods are indistinguishable on the mean return\-versus\-violation frontier, but per episode only the non\-compensatory objective keeps its stated violation budget in effectively every episode, at no cost in mean return, and a control attributes this to the objective rather than to the agent’s observation\. The broader point concerns measurement as much as training\. When violations cannot be undone across episodes, an agent that is ethical on average is not ethical, and an evaluation that reports only mean rates cannot see the difference\. Future work includes preferences from human raters, social\-dilemma environments in which the penalty layer maps onto native metrics, and a Saute\-style hard\-budget baseline\(Sootlaet al\.[2022](https://arxiv.org/html/2608.14642#bib.bib20)\)\.

## References

- J\. Achiam, D\. Held, A\. Tamar, and P\. Abbeel \(2017\)Constrained policy optimization\.InInternational Conference on Machine Learning \(ICML\),pp\. 22–31\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Alshiekh, R\. Bloem, R\. Ehlers, B\. Könighofer, S\. Niekum, and U\. Topcu \(2018\)Safe reinforcement learning via shielding\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- E\. Altman \(2021\)Constrained Markov decision processes\.Routledge\.Note:First published 1999 by Chapman & Hall/CRCCited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. Mané \(2016\)Concrete problems in AI safety\.External Links:1606\.06565Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p1.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px1.p1.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2608.14642#S3.SS4.p3.4)\.
- Y\. Chow, M\. Ghavamzadeh, L\. Janson, and M\. Pavone \(2018\)Risk\-constrained reinforcement learning with percentile risk criteria\.Journal of Machine Learning Research18\(167\),pp\. 1–51\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.30\.Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p3.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2608.14642#S3.SS4.p3.4)\.
- Z\. Gábor, Z\. Kalmár, and C\. Szepesvári \(1998\)Multi\-criteria reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 197–205\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.4](https://arxiv.org/html/2608.14642#S3.SS4.p2.2)\.
- J\. García and F\. Fernández \(2015\)A comprehensive survey on safe reinforcement learning\.Journal of Machine Learning Research16,pp\. 1437–1480\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Ghasemi and M\. Crowley \(2026\)Objective decoupling in social reinforcement learning: recovering ground truth from sycophantic majorities\.External Links:2602\.08092Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Graham, J\. Haidt, S\. Koleva, M\. Motyl, R\. Iyer, S\. P\. Wojcik, and P\. H\. Ditto \(2013\)Moral foundations theory: the pragmatic validity of moral pluralism\.InAdvances in Experimental Social Psychology,Vol\.47,pp\. 55–130\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- D\. Hafner \(2022\)Benchmarking the spectrum of agent capabilities\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz, E\. Howley, A\. A\. Irissappane, P\. Mannion, A\. Nowé, G\. Ramos, M\. Restelli, P\. Vamplew, and D\. M\. Roijers \(2022\)A practical guide to multi\-objective reinforcement learning and planning\.Autonomous Agents and Multi\-Agent Systems36\(1\),pp\. 26\.Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p3.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.14642#S3.SS1.p1.8),[§3\.5](https://arxiv.org/html/2608.14642#S3.SS5.p1.5)\.
- V\. Krakovna, J\. Uesato, V\. Mikulik, M\. Rahtz, T\. Everitt, R\. Kumar, Z\. Kenton, J\. Leike, and S\. Legg \(2020\)Specification gaming: the flip side of AI ingenuity\.Note:DeepMind BlogCited by:[§1](https://arxiv.org/html/2608.14642#S1.p1.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Küttler, N\. Nardelli, A\. H\. Miller, R\. Raileanu, M\. Selvatici, E\. Grefenstette, and T\. Rocktäschel \(2020\)The NetHack learning environment\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 7671–7684\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- M\. Matthews, M\. Beukman, B\. Ellis, M\. Samvelyan, M\. T\. Jackson, S\. Coward, and J\. N\. Foerster \(2024\)Craftax: a lightning\-fast benchmark for open\-ended reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 35104–35137\.Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p1.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1),[§3\.2](https://arxiv.org/html/2608.14642#S3.SS2.p1.1),[§5](https://arxiv.org/html/2608.14642#S5.p1.14)\.
- M\. Peschl, A\. Zgonnikov, F\. A\. Oliehoek, and L\. C\. Siebert \(2022\)MORAL: aligning AI with human norms through multi\-objective reinforced active learning\.InInternational Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- R\. Rădulescu, P\. Mannion, D\. M\. Roijers, and A\. Nowé \(2020\)Multi\-objective multi\-agent decision making: a utility\-based analysis and survey\.Autonomous Agents and Multi\-Agent Systems34\(1\),pp\. 10\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.5](https://arxiv.org/html/2608.14642#S3.SS5.p1.5)\.
- A\. Ray, J\. Achiam, and D\. Amodei \(2019\)Benchmarking safe exploration in deep reinforcement learning\.Note:Technical report, OpenAICited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Reymond, C\. F\. Hayes, D\. Steckelmacher, D\. M\. Roijers, and A\. Nowé \(2023\)Actor\-critic multi\-objective reinforcement learning for non\-linear utility functions\.Autonomous Agents and Multi\-Agent Systems37\(2\),pp\. 23\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.5](https://arxiv.org/html/2608.14642#S3.SS5.p1.13)\.
- M\. Rodriguez\-Soto, M\. Lopez\-Sanchez, and J\. A\. Rodriguez\-Aguilar \(2021\)Multi\-objective reinforcement learning for designing ethical environments\.InProceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 545–551\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- D\. M\. Roijers, D\. Steckelmacher, and A\. Nowé \(2018\)Multi\-objective reinforcement learning for the expected utility of the return\.InAdaptive and Learning Agents Workshop at AAMAS/FAIM,Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.5](https://arxiv.org/html/2608.14642#S3.SS5.p1.5)\.
- D\. M\. Roijers, P\. Vamplew, S\. Whiteson, and R\. Dazeley \(2013\)A survey of multi\-objective sequential decision\-making\.Journal of Artificial Intelligence Research48,pp\. 67–113\.Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p3.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.14642#S3.SS1.p1.12),[§3\.1](https://arxiv.org/html/2608.14642#S3.SS1.p1.8),[§3\.3](https://arxiv.org/html/2608.14642#S3.SS3.p1.4)\.
- J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)Defining and characterizing reward gaming\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 9460–9471\.Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p1.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Sootla, A\. I\. Cowen\-Rivers, T\. Jafferjee, Z\. Wang, D\. H\. Mguni, J\. Wang, and H\. Ammar \(2022\)Sauté RL: almost surely safe reinforcement learning using state augmentation\.InInternational Conference on Machine Learning \(ICML\),pp\. 20423–20443\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2608.14642#S8.p1.1)\.
- T\. Sorensen, L\. Jiang, J\. D\. Hwang, S\. Levine, V\. Pyatkin, P\. West, N\. Dziri, X\. Lu, K\. Rao, C\. Bhagavatula, M\. Sap, J\. Tasioulas, and Y\. Choi \(2024\)Value kaleidoscope: engaging AI with pluralistic human values, rights, and duties\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19937–19947\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- P\. Vamplew, R\. Dazeley, A\. Berry, R\. Issabekov, and E\. Dekker \(2011\)Empirical evaluation methods for multiobjective reinforcement learning algorithms\.Machine Learning84\(1–2\),pp\. 51–80\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.14642#S3.SS1.p1.12),[§3\.3](https://arxiv.org/html/2608.14642#S3.SS3.p1.4)\.
- P\. Vamplew, R\. Dazeley, C\. Foale, S\. Firmin, and J\. Mummery \(2018\)Human\-aligned artificial intelligence is a multiobjective problem\.Ethics and Information Technology20\(1\),pp\. 27–40\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px5.p1.1)\.
- K\. Van Moffaert and A\. Nowé \(2014\)Multi\-objective reinforcement learning using sets of Pareto dominating policies\.Journal of Machine Learning Research15,pp\. 3483–3512\.Cited by:[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.14642#S3.SS1.p1.12)\.
- C\. Wirth, R\. Akrour, G\. Neumann, and J\. Fürnkranz \(2017\)A survey of preference\-based reinforcement learning methods\.Journal of Machine Learning Research18\(136\),pp\. 1–46\.Cited by:[§1](https://arxiv.org/html/2608.14642#S1.p3.1),[§2](https://arxiv.org/html/2608.14642#S2.SS0.SSS0.Px4.p1.1)\.

## Supplementary Material

## Appendix AHyperparameters

Table[3](https://arxiv.org/html/2608.14642#A1.T3)lists the PPO hyperparameters used for every run in the study\. They are held fixed across all methods and dilemmas so that the comparison isolates the training objective\.

Table 3:Training hyperparameters, identical for every method\.
## Appendix BAdditional Figures

#### Frontiers on the binding dilemmas\.

The main paper reports the No Killing frontier in per\-episode units\. Figure[8](https://arxiv.org/html/2608.14642#A2.F8)adds Sustainability, the second dilemma that binds; it shows the same shared\-envelope pattern, although the ESR budget arms there were only lightly exercised \(see the Cost of Compliance section of the main paper\)\. Proportional Force is omitted because it does not bind under state\-diff detection: the unconstrained agent’s preemptive\-strike rate is1\.4×10−51\.4\\times 10^\{\-5\}per step, so a return\-versus\-violation frontier there carries no signal, and all methods collapse to the same near\-zero\-violation corner\. Its role is instead the distillation case study of Figure[6](https://arxiv.org/html/2608.14642#A2.F6)\. Figure[7](https://arxiv.org/html/2608.14642#A2.F7)shows the Lagrangian dual ascent; training convergence is reported in the main paper\.

#### Why the distilled utility degrades on a violation\-free pool\.

The distilled utility is fit to the return pairs the exemplar agents actually realise, so it is constrained only where those agents produced data\. Figure[6](https://arxiv.org/html/2608.14642#A2.F6)slices the fitted utility at the violation counts a policy visits\. The No Killing and Sustainability pools contain episodes with up to 11 and 26 violations, and both fits behave as intended: a sharp cliff separates clean episodes from violating ones, and utility falls monotonically as violations accumulate\. The Proportional Force pool is different\. Its worst single episode contains one violation, so the fittedRethR\_\{\\mathrm\{eth\}\}axis spans only\[−10,0\]\[\-10,0\]and nothing constrains the fit beyond a single violation\. The cliff at the first violation still forms, because the pool does contain both clean and violating episodes, but the ordering among violating episodes inverts: atRext=40R\_\{\\mathrm\{ext\}\}=40the fitted utility assigns−21\.9\-21\.9to three violations against−26\.2\-26\.2to one, ranking the worse episode higher\. An agent trained on this utility receives incoherent guidance as soon as it violates at all, which is why it pays about 1\.5 return for no gain in safety\. This is the intended function of the stated\-versus\-distilled ablation: it flags exactly the regime, a pool that does not exercise the ethical axis, where distillation should not be trusted\.

![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/utility_slices.png)Figure 6:Distilled utility sliced at fixed violation countsvv, in game\-return units\. On No Killing and Sustainability the fit is well behaved: a sharp cliff separatesv=0v=0fromv≥1v\\geq 1, and utility decreases monotonically with further violations\. On Proportional Force the cliff survives but the ordering among violating episodes inverts at high return, where the fit scores three violations above one\.![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/dual_ascent.png)Figure 7:Lagrangian dual ascent on No Killing \(seed mean\)\. The multiplierλ\\lambdarises only for budgets tighter than the unconstrained agent’s violation rate, and stays near zero for the loose budgets that never bind, as designed\.![Refer to caption](https://arxiv.org/html/2608.14642v1/figures/frontier_appendix.png)Figure 8:Return vs\. per\-episode violations on the two binding dilemmas \(mean±\\pm95% CI, 3 seeds\)\. No Killing \(left\) reproduces the main\-paper frontier in full; Sustainability \(right\) shows the same shared\-envelope pattern, although its ESR budget arms were under\-exercised\. Proportional Force is omitted as non\-binding\.