AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL

# AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL
Source: [https://arxiv.org/html/2606.31178](https://arxiv.org/html/2606.31178)
Woosung Kim1\*Youngjun Suh1\*Jinho Lee1\*Jongmin Lee2†Byung\-Jun Lee1,3†

1Korea University2Yonsei University3Gauss Labs Inc\. \{wsk208,jinho0997,youngjunsuh, byungjunlee\}@korea\.ac\.kr jongminlee@yonsei\.ac\.kr \*Equal contribution†Corresponding authors

###### Abstract

Optimizing nonlinear preferences in multi\-objective reinforcement learning \(MORL\) is essential for capturing complex trade\-offs like risk aversion or fairness\. However, such non\-linearity has historically bifurcated nonlinear MORL objectives into two distinct paradigms: Scalarized Expected Return \(SER\) and Expected Scalarized Return \(ESR\)\. While SER requires global\-level optimization and ESR requires non\-Markovian policies, leading to fragmented optimization strategies, we bridge this divide through the Aggregation–Expectation–Transformation \(AET\) framework\. By unifying both criteria through a tripartite decomposition of scalarization, AET provides a principled foundation for general nonlinear MORL\. Building on this framework, we propose AETDICE, a tractable offline RL algorithm for AET objectives\. By utilizing DICE\-style density\-ratio estimation in an augmented state space, AETDICE enables sample\-based optimization from static datasets\. Our framework resolves long\-standing barriers and captures respective trade\-offs induced by AET framework, which existing methods fail to address\.

## 1Introduction

Multi\-objective reinforcement learning \(MORL\) optimizes sequential decisions under multiple, often competing objectivesVan Moffaert and Nowé \([2014](https://arxiv.org/html/2606.31178#bib.bib13)\); Cheung \([2019](https://arxiv.org/html/2606.31178#bib.bib12)\); Yanget al\.\([2019](https://arxiv.org/html/2606.31178#bib.bib15)\)\. The trade\-off among objectives is governed by a scalarization function that maps vector\-valued returns to a scalar objective\. Linear scalarization reduces to standard RL via a weighted sum, while nonlinear scalarization captures richer preferences such as diminishing returns, fairness and efficiencyRoijerset al\.\([2013](https://arxiv.org/html/2606.31178#bib.bib1)\); Agarwalet al\.\([2022](https://arxiv.org/html/2606.31178#bib.bib6)\)\.

In nonlinear MORL, two canonical criteria arise from the ordering of scalarization and expectation: the Scalarized Expected Return \(SER\), which applies nonlinearity to the expected return, and the Expected Scalarized Return \(ESR\), which applies nonlinearity to each trajectory’s return\. SER and ESR represent different decision\-making contexts with fundamentally different optimal policy structures, and existing methods have accordingly been developed separately for eachRoijerset al\.\([2018](https://arxiv.org/html/2606.31178#bib.bib5)\); Hayeset al\.\([2021](https://arxiv.org/html/2606.31178#bib.bib16),[2025](https://arxiv.org/html/2606.31178#bib.bib2)\)\.

We focus on the offline setting, where policies must be learned from a fixed dataset\. Offline methods exist for linear scalarizationZhuet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib3)\); Linet al\.\([2024](https://arxiv.org/html/2606.31178#bib.bib4)\)and SER\(Kimet al\.,[2025](https://arxiv.org/html/2606.31178#bib.bib9)\), but ESR remains unaddressed: trajectory\-level nonlinearity makes the optimal action depend on previously accumulated rewards, and learning such reward\-conditioned policies from fixed data has not been explored\. Moreover, applying nonlinear utility to each trajectory’s return vector yet aggregating the results linearly can neglect lower\-performing utility dimensions—the same imbalance that nonlinear aggregation corrects in SER\. While expectation\-level nonlinearity can counteract this imbalance between nonlinear trajectory\-level utilities, no existing method addresses this\.

We address both gaps through the Aggregation–Expectation–Transformation \(AET\) framework, which decomposes nonlinear scalarization into a trajectory\-level transformation \(FF\), expectation, and an expectation\-level aggregation \(GG\)\. AET subsumes SER and ESR as special cases while enabling a richer class of objectives that combines both levels of nonlinearity\. To optimize AET objectives offline, we introduce AETDICE—offline*AET*optimization via*DI*stribution*C*orrection*E*stimation\. We propose a transformed reward that absorbs nonlinearFFinto per\-step rewards, enabling the first offline ESR optimization, and develop a finite\-horizon DICE formulation over the augmented state space to handle the global aggregation of concaveGG\. Empirically, AETDICE optimizes ESR, SER, and novel AET objectives within a single framework, revealing distinct policy behaviors that arise from the interaction of both levels of nonlinearity\.

## 2Preliminaries

### 2\.1Finite\-horizon MOMDPs and MORL objectives

We consider a finite\-horizon multi\-objective Markov Decision Process \(MOMDP\)ℳ=\(𝒮,𝒜,P,H,p0,𝐫\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,\\allowbreak H,\\allowbreak p\_\{0\},\\allowbreak\\mathbf\{r\}\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces,P​\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)is a time\-homogeneous transition kernel,HHis the horizon,p0​\(s\)p\_\{0\}\(s\)is the initial\-state distribution, and𝐫​\(s,a\)∈ℝm\\mathbf\{r\}\(s,a\)\\in\\mathbb\{R\}^\{m\}specifies anmm\-dimensional reward vector formmobjectives\. A time\-dependent policyπ​\(a\|s,t\)\\pi\(a\|s,t\)induces a trajectory distribution overτ=\(s0,a0,…,sH−1,aH−1\)\\tau=\(s\_\{0\},a\_\{0\},\\ldots,s\_\{H\-1\},a\_\{H\-1\}\), and the cumulative multi\-objective return is given by𝐑​\(τ\)=∑t=0H−1𝐫​\(st,at\)\\mathbf\{R\}\(\\tau\)=\\sum\_\{t=0\}^\{H\-1\}\\mathbf\{r\}\(s\_\{t\},a\_\{t\}\)\.

Multi\-objective reinforcement learning \(MORL\) employs a scalarization functionu:ℝm→ℝu:\\mathbb\{R\}^\{m\}\\to\\mathbb\{R\}to convert vector\-valued returns into a scalar objective\. The classical choice is linear scalarization,u​\(𝐱\)=𝐰⊤​𝐱u\(\\mathbf\{x\}\)=\\mathbf\{w\}^\{\\top\}\\mathbf\{x\}, which yields

Jlin​\(π;𝐰\)=𝔼τ∼π​\[𝐰⊤​𝐑​\(τ\)\],J\_\{\\mathrm\{lin\}\}\(\\pi;\\mathbf\{w\}\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\[\\mathbf\{w\}^\{\\top\}\\mathbf\{R\}\(\\tau\)\],\(1\)where𝐰∈ℝm\\mathbf\{w\}\\in\\mathbb\{R\}^\{m\}specifies linear trade\-off between objectives\. Since linearuucommutes with the expectation, the problem reduces to single\-objective RL with scalar rewardr𝐰=𝐰⊤​𝐫r\_\{\\mathbf\{w\}\}=\\mathbf\{w\}^\{\\top\}\\mathbf\{r\}\. To capture more general preference structures, nonlinear MORL allowsuuto be nonlinear\. In this setting, the expectation and scalarization no longer commute, giving rise to two canonical optimality criteria: the*scalarized expected return*\(SER\) and the*expected scalarized return*\(ESR\),

JSER​\(π;u\)=u​\(𝔼τ∼π​\[𝐑​\(τ\)\]\),JESR​\(π;u\)=𝔼τ∼π​\[u​\(𝐑​\(τ\)\)\]\.J\_\{\\mathrm\{SER\}\}\(\\pi;u\)=u\\left\(\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\[\\mathbf\{R\}\(\\tau\)\]\\right\),\\qquad J\_\{\\mathrm\{ESR\}\}\(\\pi;u\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\[u\(\\mathbf\{R\}\(\\tau\)\)\]\.\(2\)SER and ESR represent different decision\-making contexts and place different roles on the scalarizationuu\. Under SER,uSERu\_\{\\mathrm\{SER\}\}aggregates the expected return vector into a scalar objective and is typically concave, promoting balanced performance across objectives at the expectation levelKimet al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib9)\); Rădulescuet al\.\([2020](https://arxiv.org/html/2606.31178#bib.bib22)\)\. Under ESR,uESRu\_\{\\mathrm\{ESR\}\}encodes a trajectory\-level preference and is not restricted to concave functionsCheung \([2019](https://arxiv.org/html/2606.31178#bib.bib12)\); Fanet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib18)\)\. This distinction leads to fundamentally different optimal policy structures and optimization approaches, establishing the two central paradigms of nonlinear MORL\.

### 2\.2Optimization challenges in nonlinear MORL

Figure[1](https://arxiv.org/html/2606.31178#S2.F1)illustrates how optimal policies differ across MORL with linear, SER, and ESR objectives in a simple two\-step MOMDP withuNSW​\(𝐱\)=∑i=1mlog⁡xiu\_\{\\mathrm\{NSW\}\}\(\\mathbf\{x\}\)=\\sum\_\{i=1\}^\{m\}\\log x\_\{i\}\(Nash Social WelfareKaneko and Nakamura \([1979](https://arxiv.org/html/2606.31178#bib.bib23)\)\)\. Nonlinear MORL produces behaviors absent in linear MORL: ESR selects an action no linear MORL policy chooses, and SER yields a stochastic optimal policy\. Beyond these behavioral differences, each criterion poses a distinct optimization challenge\.

s0s\_\{0\}s1s\_\{1\}sTs\_\{T\}a0:\(0,0\)\{a\_\{0\}\}:\(0,0\)a1:\(9,1\)\{a\_\{1\}\}:\(9,1\)a2:\(4,4\)\{a\_\{2\}\}:\(4,4\)a3:\(1,9\)\{a\_\{3\}\}:\(1,9\)

Obj\.Optimalπ∗\\pi^\{\*\}uNSW​\(𝔼​\[𝐑\]\)u\_\{\\text\{NSW\}\}\(\\mathbb\{E\}\[\\mathbf\{R\}\]\)𝔼​\[uNSW​\(𝐑\)\]\\mathbb\{E\}\[u\_\{\\text\{NSW\}\}\(\\mathbf\{R\}\)\]SER0\.5​a1\+0\.5​a30\.5\\,a\_\{1\}\+0\.5\\,a\_\{3\}𝐥𝐨𝐠⁡\(𝟐𝟓\)\\boldsymbol\{\\log\(25\)\}log⁡\(9\)\\log\(9\)ESRa2a\_\{2\}log⁡\(16\)\\log\(16\)𝐥𝐨𝐠⁡\(𝟏𝟔\)\\boldsymbol\{\\log\(16\)\}Lineara1a\_\{1\}ora3a\_\{3\}log⁡\(9\)\\log\(9\)log⁡\(9\)\\log\(9\)

Figure 1:Optimal policies of MORL with linear, SER, and ESR objectives in a two\-step MOMDP withuNSWu\_\{\\text\{NSW\}\}\. The optimal linear MORL policy selects the action corresponding to the higher\-weighted objective\. Full derivation in Appendix[B](https://arxiv.org/html/2606.31178#A2)\.#### SER challenge: global\-level optimization\.

In standard RL, the contribution of each state\-action pair can be evaluated locally through rewards or advantages, enabling Bellman\-style dynamic programming\. In SER, however, nonlinear scalarization is applied outside the expectation, coupling all state\-action pairs globally: the marginal contribution of any action depends on the policy’s overall expected return vector, not on local information alone\. This precludes local improvement and necessitates optimization at the level of the policy\-induced distribution\. In Figure[1](https://arxiv.org/html/2606.31178#S2.F1), action probabilities must be optimized jointly, and the SER\-optimal policy is stochastic—strictly outperforming all deterministic policies, unlike in single\-objective RL or linear MORL\. Prior workKimet al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib9)\)addresses this by reformulating the problem as convex optimization over occupancy measures, bypassing Bellman updates entirely\.

#### ESR challenge: Non\-Markovian policy\.

ESR applies nonlinearity at the trajectory level, making the optimal action depend not only on the current state but also on the accumulated reward𝐑tacc:=∑k=0t−1𝐫​\(sk,ak\)\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}:=\\sum\_\{k=0\}^\{t\-1\}\\mathbf\{r\}\(s\_\{k\},a\_\{k\}\)\. For anyt\>1t\>1, the ESR objective can be written as

JESR​\(π;uESR\)=𝔼π​\[uESR​\(𝐑tacc\+∑k=tH−1𝐫​\(sk,ak\)\)\]\.\\displaystyle J\_\{\\mathrm\{ESR\}\}\(\\pi;u\_\{\\mathrm\{ESR\}\}\)=\\mathbb\{E\}\_\{\\pi\}\\left\[u\_\{\\mathrm\{ESR\}\}\\left\(\\mathbf\{R\}\_\{t\}^\{\\text\{acc\}\}\+\\sum\_\{k=t\}^\{H\-1\}\\mathbf\{r\}\(s\_\{k\},a\_\{k\}\)\\right\)\\right\]\.Due to the nonlinearity, the future reward and𝐑tacc\\mathbf\{R\}\_\{t\}^\{\\text\{acc\}\}cannot be optimized separately\. The optimal action at statests\_\{t\}therefore depends on𝐑tacc\\mathbf\{R\}\_\{t\}^\{\\text\{acc\}\}: trajectories reaching the same state may require different actions depending on their reward history\. ESR\-optimal policies cannot, in general, be Markovian over the original state spaceCheung \([2019](https://arxiv.org/html/2606.31178#bib.bib12)\); Fanet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib18)\)\. Replacing𝐫​\(s0,a0\)\\mathbf\{r\}\(s\_\{0\},a\_\{0\}\)with=\(2,0\)=\(2,0\)in Figure[1](https://arxiv.org/html/2606.31178#S2.F1)shifts the ESR\-optimal action ats1s\_\{1\}froma3a\_\{3\}toa2a\_\{2\}, confirming that the optimal action depends on the reward history \(derivation in Appendix[B](https://arxiv.org/html/2606.31178#A2)\)\. Prior works on ESR optimizationYuet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib30)\);[Siddiqueet al\.](https://arxiv.org/html/2606.31178#bib.bib31); Penget al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib7)\)address this by augmenting the state with𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}, restoring Markovianity in the augmented space\.

## 3AET: A Unified Framework for Nonlinear Scalarization in MORL

To address offline optimization of nonlinear trajectory\-wise utilities and their nonlinear aggregation at the expectation level, we present AET, a unified framework that accommodates both levels of nonlinearity within a single formulation\. AET decomposes nonlinear scalarization into three components: Aggregation \(A\), Expectation \(E\), and Transformation \(T\)\. The AET objective is

JA​E​T​\(π;F,G\)=G​\(𝔼τ∼π​\[F​\(𝐑​\(τ\)\)\]\),\\displaystyle J\_\{AET\}\(\\pi;F,G\)=G\\\!\\left\(\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[F\(\\mathbf\{R\}\(\\tau\)\)\\right\]\\right\),\(3\)whereF:ℝm→ℝnF:\\mathbb\{R\}^\{m\}\\to\\mathbb\{R\}^\{n\}is a trajectory\-level transformation composed ofnntrajectory\-wise utilitiesF​\(𝐑\)=\(f1​\(𝐑\),…,fn​\(𝐑\)\)F\(\\mathbf\{R\}\)=\(f\_\{1\}\(\\mathbf\{R\}\),\\allowbreak\\dots,\\allowbreak f\_\{n\}\(\\mathbf\{R\}\)\), andG:ℝn→ℝG:\\mathbb\{R\}^\{n\}\\to\\mathbb\{R\}is an expectation\-level aggregation\. We assume eachfif\_\{i\}is smooth andGGis concave to ensure a well\-behaved optimization landscape\.

Eachfif\_\{i\}encodes a distinct preference over the return vector, mapping themm\-dimensional return vector tonnutility dimensions whose expectations are then aggregated byGG\. For instance,fi​\(𝐑\)=log⁡\(Ri\)f\_\{i\}\(\\mathbf\{R\}\)=\\log\(R\_\{i\}\)captures diminishing marginal utility, amplifying gains in low\-performing objectives;fi​\(𝐑\)=exp⁡\(Ri\)f\_\{i\}\(\\mathbf\{R\}\)=\\exp\(R\_\{i\}\)captures increasing marginal utility, amplifying gains in already high\-performing objectives; and non\-convex Cobb\-Douglasfi​\(R1,R2\)=R1ρ/\(R2\)1−ρf\_\{i\}\(R\_\{1\},R\_\{2\}\)=R\_\{1\}^\{\\rho\}/\(R\_\{2\}\)^\{1\-\\rho\}captures the efficiency trade\-off ratio between two objectives\. LinearGGaggregates the expected utilities linearly, while concaveGGamplifies gains in lower\-performing exptected utilities, encouraging balanced performance across all utility dimensions\.

#### Landscape of Nonlinear MORL

The AET formulation organizes existing nonlinear MORL objectives by where nonlinearity is placed\. Based on the nonlinearity ofFFandGG, we identify four regimes: \(i\) Linear MORL \(linearFF, linearGG\), recovered byF​\(𝐑\)=𝐑F\(\\mathbf\{R\}\)=\\mathbf\{R\}andG​\(𝐯\)=𝐰⊤​𝐯G\(\\mathbf\{v\}\)=\\mathbf\{w\}^\{\\top\}\\mathbf\{v\}; \(ii\) SER\-MORL \(linearFF, concaveGG\), recovered by retaining the identity transformation but adopting a concave aggregationGG; \(iii\) ESR\-MORL \(nonlinearFF, linearGG\), recovered by settingF=\(f1,…,fn\)F=\(f\_\{1\},\\dots,f\_\{n\}\)withG​\(𝐯\)=𝟏⊤​𝐯G\(\\mathbf\{v\}\)=\\mathbf\{1\}^\{\\top\}\\mathbf\{v\}; and \(iv\) AET\-MORL \(nonlinearFF, concaveGG\), recovered by aggregatingF=\(f1,…,fn\)F=\(f\_\{1\},\\dots,f\_\{n\}\)with concaveGG\. We note that ESR\-MORL can also be described asF=f1\+⋯\+fnF=f\_\{1\}\+\\cdots\+f\_\{n\}withG​\(v\)=vG\(v\)=v, reducing to a single scalar utility; we retain the vector representation to clearly distinguish it from AET\-MORL, where concaveGGpreserves the distinction among multiple utility dimensions\.

This taxonomy makes explicit which regimes lack offline methods: while Linear MORL and SER\-MORL are addressed by prior workZhuet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib3)\); Linet al\.\([2024](https://arxiv.org/html/2606.31178#bib.bib4)\); Kimet al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib9)\), no existing approach handlesESR\-MORLorAET\-MORL\. ESR\-MORL has been studied in online and value iteration settingYuet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib30)\);[Siddiqueet al\.](https://arxiv.org/html/2606.31178#bib.bib31); Peng and Fain \([2023](https://arxiv.org/html/2606.31178#bib.bib11)\), but \(i\) no offline method addresses the additional challenge of distribution shift, and \(ii\) current approaches cannot be extended to optimizing AET objectives with two nonlinearities, as they do not account for nonlinear aggregationGG\. The AET objective and its offline optimization are novel to this work\. We fill these gaps by addressing each source of nonlinearity in turn:[Section˜4](https://arxiv.org/html/2606.31178#S4)tackles nonlinearFF,[Section˜5](https://arxiv.org/html/2606.31178#S5)tackles nonlinearGG, and their combination yields AETDICE\.

## 4Augmented MOMDP for Nonlinear Transformation F

We present an augmented MOMDP framework for optimizing nonlinear transformationFF\. While we adopt the augmented state space from prior workPenget al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib7)\)to restore Markovianity, we define a transformed reward that captures the incremental contribution of each action toFF\. This serves two purposes: it enables offline optimization of ESR\-MORL by reducing it to standard single\-objective RL in the augmented MOMDP, and it extends naturally to AET\-MORL by reformulating it as SER\-MORL in the augmented MOMDP\.

### 4\.1Transformed Reward in the Augmented Space

We define the augmented finite\-horizon MOMDPℳ~=\(𝒮~,𝒜,P~,H,p~0,𝐫~\)\\tilde\{\\mathcal\{M\}\}=\(\\tilde\{\\mathcal\{S\}\},\\mathcal\{A\},\\tilde\{P\},H,\\tilde\{p\}\_\{0\},\\tilde\{\\mathbf\{r\}\}\), where each augmented state is given bys~t=\(st,𝐑tacc,t\)\\tilde\{s\}\_\{t\}=\(s\_\{t\},\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\},t\)\. The initial augmented state iss~0=\(s0,𝟎,0\)\\tilde\{s\}\_\{0\}=\(s\_\{0\},\\mathbf\{0\},0\)with initial distributionp~0​\(s~0\)=p0​\(s0\)\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\_\{0\}\)=p\_\{0\}\(s\_\{0\}\)\. The transition kernelP~\\tilde\{P\}is defined as

P~\(s~t\+1\|s~t,at\)=P\(st\+1\|st,at\)⋅𝕀\[𝐑t\+1acc=𝐑tacc\+𝐫\(st,at\)\],\\displaystyle\\begin\{split\}\\tilde\{P\}\(\\tilde\{s\}\_\{t\+1\}\|\\tilde\{s\}\_\{t\},a\_\{t\}\)=P\(s\_\{t\+1\}&\|s\_\{t\},a\_\{t\}\)\\cdot\\mathbb\{I\}\[\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\+1\}=\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\+\\mathbf\{r\}\(s\_\{t\},a\_\{t\}\)\],\\\\ \\end\{split\}\(4\)where the timestep advances deterministically fromtttot\+1t\+1\. Since reward accumulation is absorbed into the transition dynamics and our objective focuses on the trajectory\-wise utilityF​\(𝐑​\(τ\)\)F\(\\mathbf\{R\}\(\\tau\)\), we define a transformed reward to capture the incremental contribution of each action toFF:

𝐫~​\(s~t,at\)=F​\(𝐑tacc\+𝐫​\(st,at\)\)−F​\(𝐑tacc\),for​t≥1\\displaystyle\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)=F\\big\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\+\\mathbf\{r\}\(s\_\{t\},a\_\{t\}\)\\big\)\-F\\big\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\\big\),\\quad\\text\{for\}\\ t\\geq 1\(5\)with𝐫~​\(s~0,a0\)=F​\(𝐫​\(s0,a0\)\)\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\}\_\{0\},a\_\{0\}\)=F\(\\mathbf\{r\}\(s\_\{0\},a\_\{0\}\)\)\. This induces an exact telescoping sum:∑t=0H−1𝐫~​\(s~t,at\)=F​\(𝐑​\(τ\)\)\\sum\_\{t=0\}^\{H\-1\}\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)=F\(\\mathbf\{R\}\(\\tau\)\), reducing the trajectory\-level objective to a standard cumulative reward in the augmented MOMDP\. WhenFFis linear, the transformed reward reduces to𝐫~​\(s~t,at\)=F​\(𝐫​\(st,at\)\)\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)=F\(\\mathbf\{r\}\(s\_\{t\},a\_\{t\}\)\), which no longer depends on𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}; the optimal policy is then independent of the accumulated reward, making the augmentation redundant\.

Under this construction, a policy must be conditioned on the augmented states~t\\tilde\{s\}\_\{t\}to account for the dependence on accumulated rewards\. We denote such policies asπ~​\(a\|s~t\)\\tilde\{\\pi\}\(a\|\\tilde\{s\}\_\{t\}\), which are Markovian inℳ~\\tilde\{\\mathcal\{M\}\}but non\-Markovian in the original MOMDPℳ\\mathcal\{M\}\. This reduces ESR\-MORL to solving single\-objective RL inℳ~\\tilde\{\\mathcal\{M\}\}, as we show below\.

### 4\.2Offline ESR\-MORL

As described in[Section˜3](https://arxiv.org/html/2606.31178#S3), whenGGis linear, it commutes with the expectation, allowing us to collapse the AET objective into a single\-utility case \(nn=1\) by definingF~=G∘F:ℝm→ℝ\\tilde\{F\}=G\\circ F\\colon\\mathbb\{R\}^\{m\}\\to\\mathbb\{R\}\. ESR\-MORL then becomes standard single\-objective RL inℳ~\\tilde\{\\mathcal\{M\}\}, as the trajectory\-level nonlinearity is fully absorbed into the transformed reward\. This enables existing offline RL algorithms, such as IQLKostrikovet al\.\([2021](https://arxiv.org/html/2606.31178#bib.bib32)\), to be applied with only minor modifications to account for the augmented states~t\\tilde\{s\}\_\{t\}\.

###### Proposition 4\.1\(Reduction to single\-objective RL under linearGG\)\.

LetGGbe linear\. Then for any policyπ~\\tilde\{\\pi\}in the augmented MOMDPℳ~\\tilde\{\\mathcal\{M\}\},

JAET​\(π~;F,G\)=G​\(𝔼τ∼π​\[F​\(𝐑​\(τ\)\)\]\)=𝔼τ∼π~​\[F~​\(𝐑​\(τ\)\)\]=𝔼τ∼π~​\[∑t=0H−1r~​\(s~t,at\)\],\\displaystyle J\_\{\\mathrm\{AET\}\}\(\\tilde\{\\pi\};F,G\)=G\\\!\\left\(\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[F\(\\mathbf\{R\}\(\\tau\)\)\\right\]\\right\)=\\mathbb\{E\}\_\{\\tau\\sim\\tilde\{\\pi\}\}\\\!\\left\[\\tilde\{F\}\(\\mathbf\{R\(\\tau\)\}\)\\right\]=\\mathbb\{E\}\_\{\\tau\\sim\\tilde\{\\pi\}\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\tilde\{r\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\\right\],\(6\)wherer~\\tilde\{r\}is defined in \([5](https://arxiv.org/html/2606.31178#S4.E5)\) withF~=G∘F\\tilde\{F\}=G\\circ F\.

#### Comparison with existing ESR methods\.

Prior ESR optimization methodsYuet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib30)\);[Siddiqueet al\.](https://arxiv.org/html/2606.31178#bib.bib31)modify the value function update, rather than transforming the reward, to account for trajectory\-wise nonlinearity, achieving effective online optimization\. However, this approach does not align with two goals in our framework: offline optimization and AET optimization\. Designing a value function update that simultaneously handles both nonlinearity and distribution shift is nontrivial\. Our transformed reward decouples these concerns: the nonlinearity ofFFis absorbed into the reward, while the value function update relies entirely on existing offline RL methods\. Furthermore, value\-function\-based approaches are inherently local and incompatible with the global optimization required by concaveGGin AET\-MORL\. This motivates the visitation\-distribution\-based formulation in the augmented MOMDP, which we develop in the next section\.

In addition, prior work on ESRPenget al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib7)\)conditions the policy on accumulated reward vector𝐑tacc\\mathbf\{R\}^\{\\text\{acc\}\}\_\{t\}and assigns the transformed returnF​\(𝐑​\(τ\)\)F\(\\mathbf\{R\}\(\\tau\)\)only as a terminal reward\. In contrast, our formulation provides a denser reward signal𝐫~\\tilde\{\\mathbf\{r\}\}that captures the incremental contribution of each action toF​\(𝐑​\(τ\)\)F\(\\mathbf\{R\}\(\\tau\)\), making it amenable for sample\-based optimization\.

## 5Convex Optimization for Nonlinear Aggregation G

With nonlinearFFabsorbed into the transformed reward, optimizing AET\-MORL reduces to SER\-MORL inℳ~\\tilde\{\\mathcal\{M\}\}: maximizing a concaveGGover multiple expected utilities\. To handle the global\-level optimization required by concaveGG, we extend the Distribution Correction Estimation \(DICE\) frameworkKimet al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib9)\); Leeet al\.\([2021](https://arxiv.org/html/2606.31178#bib.bib10)\)to the augmented MOMDPℳ~\\tilde\{\\mathcal\{M\}\}, yielding AETDICE—offline*AET*optimization via*Di*stribution*C*orrection*E*stimation\.

### 5\.1Visitation Distribution in the Augmented MOMDP

In the augmented MOMDP, the visitation distribution induced by a policyπ~\\tilde\{\\pi\}is defined asdtπ~​\(s~,a\):=Pr⁡\(s~t=s~,at=a∣π~\)d^\{\\tilde\{\\pi\}\}\_\{t\}\(\\tilde\{s\},a\)\\;:=\\;\\Pr\(\\tilde\{s\}\_\{t\}=\\tilde\{s\},a\_\{t\}=a\\mid\\tilde\{\\pi\}\), representing the probability thatπ~\\tilde\{\\pi\}visits the augmented state–action pair\(s~,a\)\(\\tilde\{s\},a\)at timesteptt\. While the timestep is already encoded within the augmented states~\\tilde\{s\}, we retain the indexttfor notational clarity in the derivations below\. Crucially, the expected utility vector becomes linear in the visitation distributiondtπ~d^\{\\tilde\{\\pi\}\}\_\{t\}:

𝔼τ∼π~​\[F​\(𝐑​\(τ\)\)\]=𝔼τ∼π~​\[∑t=0H−1𝐫~​\(s~t,at\)\]=∑t=0H−1∑s~,adtπ~​\(s~,a\)​𝐫~​\(s~,a\)\.\\mathbb\{E\}\_\{\\tau\\sim\\tilde\{\\pi\}\}\\\!\\left\[F\(\\mathbf\{R\(\\tau\)\}\)\\right\]=\\mathbb\{E\}\_\{\\tau\\sim\\tilde\{\\pi\}\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\\right\]=\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{\\tilde\{s\},a\}d^\{\\tilde\{\\pi\}\}\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\.\(7\)Therefore, AET\-MORL reduces to a convex optimization problem over visitation distribution inℳ~\\tilde\{\\mathcal\{M\}\}, subject to finite\-horizon Bellman flow constraints that ensure the validity of𝐝=\{dt\}t=0H−1\\mathbf\{d\}=\\\{d\_\{t\}\\\}\_\{t=0\}^\{H\-1\}:

max𝐝≥0\\displaystyle\\max\_\{\\mathbf\{d\}\\geq 0\}\\quadG​\(∑t=0H−1∑s~,adt​\(s~,a\)​𝐫~​\(s~,a\)\)\\displaystyle G\\\!\\left\(\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\\right\)\(8a\)s\.t\.∑ad0​\(s~,a\)=p~0​\(s~\),∀s~\\displaystyle\\sum\_\{a\}d\_\{0\}\(\\tilde\{s\},a\)=\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\),\\quad\\forall\\tilde\{s\}\(8b\)∑a′dt\+1​\(s~′,a′\)=∑s~,adt​\(s~,a\)​P~​\(s~′∣s~,a\),∀s~′,t∈\{0,…,H−2\},\\displaystyle\\sum\_\{a^\{\\prime\}\}d\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\},a^\{\\prime\}\)=\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\),\\quad\\forall\\tilde\{s\}^\{\\prime\},\\,t\\in\\\{0,\\dots,H\-2\\\},\(8c\)wheres~′=\(s′,𝐑tacc\+𝐫​\(s,a\),t\+1\)\\tilde\{s\}^\{\\prime\}=\(s^\{\\prime\},\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\+\\mathbf\{r\}\(s,a\),t\+1\)denotes the next augmented state\. This formulation enables the global optimization required by SER\-like objectives with nonlinearGG, and the optimal policy inducing optimal𝐝∗\\mathbf\{d\}^\{\*\}can be recovered viaπ~t∗​\(a\|s~\)=dt∗​\(s~,a\)/∑adt∗​\(s~,a\)\\tilde\{\\pi\}\_\{t\}^\{\*\}\(a\|\\tilde\{s\}\)=d^\{\*\}\_\{t\}\(\\tilde\{s\},a\)/\\sum\_\{a\}d^\{\*\}\_\{t\}\(\\tilde\{s\},a\)\. However, solving this convex optimization directly requires access to the transition dynamics ofℳ~\\tilde\{\\mathcal\{M\}\}\. Moreover, even when the original state and action spaces are finite, the augmented state space𝒮~\\tilde\{\\mathcal\{S\}\}is generally continuous, as the accumulated reward𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}takes continuous values under general reward functions\. This makes the convex formulation impractical even in tabular domains, motivating a sample\-based approach\.

### 5\.2AETDICE: Regularized AET for sample\-based optimization

To derive a sample\-based objective from \([8](https://arxiv.org/html/2606.31178#S5.E8)\) in the offline setting, AETDICE extends the DICE\-RL framework to the augmented MOMDP with two key modifications: \(i\)ϕ\\phi\-divergence regularization applied to the per\-timestep data distributiondtDd\_\{t\}^\{D\}, correcting distribution shift at each step of the finite horizon, and \(ii\) a slack variable𝐤\\mathbf\{k\}that decouples the concave aggregationGGfrom the expectation over𝐝\\mathbf\{d\}\. The regularized primal problem is:

max𝐝∈𝒟,𝐤∈ℝm⁡G​\(𝐤\)−β​∑t=0H−1Dϕ​\(dt∥dtD\),s\.t\.∑t=0H−1∑s~,adt​\(s~,a\)​𝐫~​\(s~,a\)=𝐤,\\displaystyle\\max\_\{\\mathbf\{d\}\\in\\mathcal\{D\},\\mathbf\{k\}\\in\\mathbb\{R\}^\{m\}\}G\(\\mathbf\{k\}\)\-\\beta\\sum\_\{t=0\}^\{H\-1\}D\_\{\\phi\}\(d\_\{t\}\\\|d^\{D\}\_\{t\}\),\\quad\\text\{s\.t\.\}\\quad\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)=\\mathbf\{k\},\(9\)where𝒟\\mathcal\{D\}denotes the set of non\-negative visitation distribution satisfying constraints \([8b](https://arxiv.org/html/2606.31178#S5.E8.2)\)–\([8c](https://arxiv.org/html/2606.31178#S5.E8.3)\),β\>0\\beta\>0is a regularization coefficient, andDϕ​\(dt∥dtD\):=∑s~,adtD​\(s~,a\)​ϕ​\(dt​\(s~,a\)/dtD​\(s~,a\)\)D\_\{\\phi\}\(d\_\{t\}\\,\\\|\\,d^\{D\}\_\{t\}\):=\\sum\_\{\\tilde\{s\},a\}d\_\{t\}^\{D\}\(\\tilde\{s\},a\)\\phi\\left\(d\_\{t\}\(\\tilde\{s\},a\)/d\_\{t\}^\{D\}\(\\tilde\{s\},a\)\\right\)is theϕ\\phi\-divergence betweendtd\_\{t\}and the data distributiondtDd\_\{t\}^\{D\}at time steptt\. The full derivation, required assumptions, and complete algorithm are provided in Appendix[C](https://arxiv.org/html/2606.31178#A3)\.

#### Dual formulation\.

We form the Lagrangian of \([9](https://arxiv.org/html/2606.31178#S5.E9)\) by introducing𝝁∈ℝn\\boldsymbol\{\\mu\}\\in\\mathbb\{R\}^\{n\}for the slack constraint, and time\-indexed multipliersνt​\(s~\)\\nu\_\{t\}\(\\tilde\{s\}\)for the Bellman flow constraints \([8b](https://arxiv.org/html/2606.31178#S5.E8.2)\)–\([8c](https://arxiv.org/html/2606.31178#S5.E8.3)\), replacing the time\-independent multiplier used in standard DICE approaches\(Kimet al\.,[2025](https://arxiv.org/html/2606.31178#bib.bib9); Leeet al\.,[2021](https://arxiv.org/html/2606.31178#bib.bib10)\)to reflect the finite\-horizon structure of the augmented MOMDP\. The primal problem becomesmax𝐝≥0,𝐤⁡min𝝂,𝝁⁡ℒ\\max\_\{\\mathbf\{d\}\\geq 0,\\mathbf\{k\}\}\\min\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\}\\mathcal\{L\}, where:

ℒ\(𝐝,𝐤,𝝂,𝝁\):=G\(𝐤\)−𝝁⊤𝐤\+∑s~ν0\(s~\)p~0\(s~\)−β∑t=0H−1Dϕ\(dt\|\|dtD\)\+∑s~,a,tdt\(s~,a\)e𝝂,𝝁,t\(s~,a\)\\displaystyle\\mathcal\{L\}\(\\mathbf\{d\},\\mathbf\{k\},\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\):=G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\top\}\\mathbf\{k\}\+\\sum\_\{\\tilde\{s\}\}\\nu\_\{0\}\(\\tilde\{s\}\)\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\)\-\\beta\\sum\_\{t=0\}^\{H\-1\}D\_\{\\phi\}\(d\_\{t\}\|\|d^\{D\}\_\{t\}\)\+\\sum\_\{\\tilde\{s\},a,t\}d\_\{t\}\(\\tilde\{s\},a\)e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)withe𝝂,𝝁,t​\(s~,a\)=𝝁⊤​𝐫~​\(s~,a\)\+𝔼s~′​\[νt\+1​\(s~′\)\]−νt​\(s~\)e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)=\\boldsymbol\{\\mu\}^\{\\top\}\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\+\\mathbb\{E\}\_\{\\tilde\{s\}^\{\\prime\}\}\[\\nu\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\}\)\]\-\\nu\_\{t\}\(\\tilde\{s\}\)andνH​\(s~\)=0\\nu\_\{H\}\(\\tilde\{s\}\)=0for consistency\.

#### Derivation of training objectives\.

Under strong duality, we swap the optimization order tomin𝝂,𝝁⁡max𝐝≥0,𝐤⁡ℒ\\min\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\}\\max\_\{\\mathbf\{d\}\\geq 0,\\mathbf\{k\}\}\\mathcal\{L\}\. The inner maximization over𝐝\\mathbf\{d\}yields a closed\-form optimal visitation distribution ratiowt∗​\(s~,a\)=dt∗​\(s~,a\)/dtD​\(s~,a\)=max⁡\(0,\(ϕ′\)−1​\(e𝝂,𝝁,t​\(s~,a\)/β\)\)w^\{\*\}\_\{t\}\(\\tilde\{s\},a\)=d^\{\*\}\_\{t\}\(\\tilde\{s\},a\)/d^\{D\}\_\{t\}\(\\tilde\{s\},a\)=\\max\\left\(0,\(\\phi^\{\\prime\}\)^\{\-1\}\\left\(e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)/\\beta\\right\)\\right\)\. We leverage Fenchel conjugatesϕ\+∗​\(y\):=supx≥0\{x​y−ϕ​\(x\)\}\\phi\_\{\+\}^\{\*\}\(y\):=\\sup\_\{x\\geq 0\}\\\{xy\-\\phi\(x\)\\\}andG∗​\(𝐲\):=sup𝐱\{𝐱⊤​𝐲\+G​\(𝐱\)\}G^\{\*\}\(\\mathbf\{y\}\):=\\sup\_\{\\mathbf\{x\}\}\\\{\\mathbf\{x\}^\{\\top\}\\mathbf\{y\}\+G\(\\mathbf\{x\}\)\\\}to simplify the min–max optimization and yield an unconstrained objective that depends only on expectations overp~0\\tilde\{p\}\_\{0\}and the offline datadtDd^\{D\}\_\{t\}:

min𝝂,𝝁⁡ℒA​E​T​\(𝝂,𝝁\):=𝔼p~0​\[ν0​\(s~\)\]\+∑t=0H−1𝔼dtD​\[β​ϕ\+∗​\(e𝝂,𝝁,t​\(s~,a\)/β\)\]\+G∗​\(−𝝁\)\.\\displaystyle\\min\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\}\\mathcal\{L\}\_\{AET\}\(\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\):=\\mathbb\{E\}\_\{\\tilde\{p\}\_\{0\}\}\[\\nu\_\{0\}\(\\tilde\{s\}\)\]\+\\sum\_\{t=0\}^\{H\-1\}\\mathbb\{E\}\_\{d\_\{t\}^\{D\}\}\\left\[\\beta\\phi\_\{\+\}^\{\*\}\\left\(e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)/\\beta\\right\)\\right\]\+G^\{\*\}\(\-\\boldsymbol\{\\mu\}\)\.\(10\)After minimizingℒAET\\mathcal\{L\}\_\{\\mathrm\{AET\}\}to obtainν∗\\nu^\{\*\}and𝝁∗\\boldsymbol\{\\mu\}^\{\*\}, the policy that induces the optimal visitation distribution can be extracted using the optimal visitation ratiowt∗w^\{\*\}\_\{t\}via weighted behavior cloning followingLeeet al\.\([2021](https://arxiv.org/html/2606.31178#bib.bib10)\):

maxπ⁡𝔼\(s~,a\)∼dtD​\[wt∗​\(s~,a\)​log⁡π​\(a\|s~t\)\]\.\\displaystyle\\max\_\{\\pi\}\\;\\mathbb\{E\}\_\{\(\\tilde\{s\},a\)\\sim d\_\{t\}^\{D\}\}\\\!\\left\[w^\{\*\}\_\{t\}\(\\tilde\{s\},a\)\\log\\pi\(a\|\\tilde\{s\}\_\{t\}\)\\right\]\.

#### Comparison with FairDICE\.

AETDICE differs from the DICE frameworkKimet al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib9)\); Leeet al\.\([2021](https://arxiv.org/html/2606.31178#bib.bib10)\)in three key aspects: \(i\) prior DICE approaches impose Bellman flow constraints in a discounted infinite\-horizon MDP, whereas AETDICE imposes finite\-horizon Bellman flow constraints over the augmented state space \([8b](https://arxiv.org/html/2606.31178#S5.E8.2)\)–\([8c](https://arxiv.org/html/2606.31178#S5.E8.3)\); \(ii\) the Lagrangian multipliers and policy are conditioned on the augmented states~\\tilde\{s\}, and offline datadtDd\_\{t\}^\{D\}must be preprocessed to include the accumulated reward vector𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}; and \(iii\) AETDICE extends FairDICE, which handles only SER objectives, to support a broader class of nonlinear objectives including ESR and AET, though it operates in a higher\-dimensional state space\.

## 6Experiments

We conduct two sets of experiments to demonstrate that AETDICE enables offline optimization of AET objectives with nonlinearFFandGG\. First, we focus on offline Fair MORL with concaveFFandGG, optimizing and comparing different fairness objectives within a single framework\. Second, we go beyond concavity to optimize convex and non\-convex trajectory\-wise utilities, along with their concave aggregation, on complex continuous\-control domains\.

#### Algorithms\.

We evaluate offline methods corresponding to each nonlinear MORL regime identified in[Section˜3](https://arxiv.org/html/2606.31178#S3): \(1\)OptiDICE\(Leeet al\.,[2021](https://arxiv.org/html/2606.31178#bib.bib10)\): a DICE\-based offline RL method with linearFFand linearGG; \(2\)FairDICE\(Kimet al\.,[2025](https://arxiv.org/html/2606.31178#bib.bib9)\): a DICE\-based offline MORL method with linearFFand concaveGG; \(3\)ESR\-IQL\(ours\): an IQL\-based method on the augmented MOMDP with nonlinearFFand linearGG; and \(4\)AETDICE\-Obj\(ours\): a DICE\-based method on the augmented MOMDP, applicable to any combination ofFFandGG\.

### 6\.1Offline Fair MORL: ESR, SER, and BSR

![Refer to caption](https://arxiv.org/html/2606.31178v1/x1.png)
![Refer to caption](https://arxiv.org/html/2606.31178v1/x2.png)

Figure 2:\(a\)Fair\-Taxi: a taxi agent serving two passenger groups\. Each objective corresponds to serving a passenger group\.\(b\)Distribution of behavior patterns across varying nonlinear MORL objectives\. Full details in Appendix[E](https://arxiv.org/html/2606.31178#A5)\.Fair MORLKimet al\.\([2025](https://arxiv.org/html/2606.31178#bib.bib9)\); Parket al\.\([2024](https://arxiv.org/html/2606.31178#bib.bib36)\)applies concave scalarization to promote equitable performance across objectives, where improvements to lower\-performing objectives yield larger gains in overall utility\. As discussed in[Section˜3](https://arxiv.org/html/2606.31178#S3), the AET framework subsumes existing Fair MORL criteria as special cases\. To further bridge trajectory\-level and expectation\-level fairness, we adopt theα\\alpha\-fairness utility\(Mo and Walrand,[2002](https://arxiv.org/html/2606.31178#bib.bib17)\)to constructBalanced Scalarized Return \(BSR\), a novel objective family parameterized byα\\alphathat interpolates between trajectory\-level and expectation\-level fairness:

JBSR​\(π;α\)=Gα​\(𝔼τ∼π​\[Fα​\(𝐑​\(τ\)\)\]\),Fα​\(𝐱\)=u1−α​\(𝐱\),Gα​\(𝐱\)=∑i=1nuα​\(xi\)\.J\_\{\\mathrm\{BSR\}\}\(\\pi;\\alpha\)=G\_\{\\alpha\}\\\!\\left\(\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[F\_\{\\alpha\}\(\\mathbf\{R\}\(\\tau\)\)\\right\]\\right\),\\qquad F\_\{\\alpha\}\(\\mathbf\{x\}\)=u\_\{1\-\\alpha\}\(\\mathbf\{x\}\),\\quad G\_\{\\alpha\}\(\\mathbf\{x\}\)=\\sum\_\{i=1\}^\{n\}u\_\{\\alpha\}\(x\_\{i\}\)\.\(11\)whereuα​\(x\)=x1−α−11−αu\_\{\\alpha\}\(x\)=\\frac\{x^\{1\-\\alpha\}\-1\}\{1\-\\alpha\}forα≠1\\alpha\\neq 1andu1​\(x\)=log⁡xu\_\{1\}\(x\)=\\log x\. By varyingα∈\[0,1\]\\alpha\\in\[0,1\], BSR controls the relative nonlinearity applied at the trajectory level \(FαF\_\{\\alpha\}\) and the aggregation level \(GαG\_\{\\alpha\}\), recovering ESR atα=0\\alpha=0and SER atα=1\\alpha=1\. We analyze ESR\-, BSR\-, and SER\-optimal policies obtained by AETDICE and illustrate their distinct behaviors in Fair\-Taxi environment \([Figure˜2](https://arxiv.org/html/2606.31178#S6.F2)\)\. Details and full results including other environments including MO\-PointMaze\-3obj are presented in Appendix[E](https://arxiv.org/html/2606.31178#A5)\.

#### Behavioral differences\.

Figure[2](https://arxiv.org/html/2606.31178#S6.F2)\(a\) illustrates three episode\-level behavior patterns of optimal nonlinear MORL policies in Fair\-Taxi: \(A\) serving both groups, \(B\) serving only group 1, and \(C\) serving only group 2\. Figure[2](https://arxiv.org/html/2606.31178#S6.F2)\(b\) shows their distribution over 100 episodes\. The ESR\-optimal policy consistently produces pattern \(A\), balancing return across objectives within each episode\. The SER\-optimal policy yields a stochastic mixture of \(B\) and \(C\), where each trajectory specializes in a single objective and balance is achieved only in expectation\. The BSR\-optimal policy smoothly interpolates between these extremes: asα\\alphaincreases toward SER, specialized trajectories gradually emerge, while lowerα\\alpharecovers the balanced ESR behavior\.

![Refer to caption](https://arxiv.org/html/2606.31178v1/x3.png)

Figure 3:Non\-Markovian behavior of ESR and BSR\-optimal policies in Fair\-Taxi environment\.
#### Non\-Markovian policy\.

The non\-Markovian structure formalized in[Section˜4](https://arxiv.org/html/2606.31178#S4)manifests concretely in the ESR\- and BSR\-optimal policies\. Under these objectives, the concavity ofFFtransforms the original reward through𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}: as one objective accumulates higher returns, its transformed marginal reward diminishes, steering the policy toward the under\-served objective\. Since the transformed reward depends on𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}, the optimal action from the same state varies with trajectory history, making the policy inherently non\-Markovian\. In contrast, the SER\-optimal policy is unaffected, as its reward does not depend on𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\. Figure[3](https://arxiv.org/html/2606.31178#S6.F3)captures a state att=25t=25where group 1 has already been served three times\. The ESR\-optimal policy shifts to group 2 due to diminished marginal reward for group 1, while the SER\-optimal policy continues serving group 1 independently of history\. The BSR\-optimal policy \(α=0\.45\\alpha=0\.45\) stochastically chooses between the two strategies\.

#### Quantitative evaluation\.

The quantile plot in Figure[4](https://arxiv.org/html/2606.31178#S6.F4)illustrates how different Fair MORL objectives shape the per\-trajectory return distribution across individual objective dimensions\. The SER\-optimal policy achieves higher expected returns but with large variance across trajectories, while the ESR\-optimal policy yields lower expected returns with significantly reduced variance\. The BSR\-optimal policy lies between these extremes, offering a tunable trade\-off\. Prior work has noted that SER is suited for repeated policy execution and ESR for single\-shot deploymentRoijerset al\.\([2013](https://arxiv.org/html/2606.31178#bib.bib1)\)\. BSR accommodates practitioners who seek the benefits of repeated execution while constraining the variability across objectives, interpolating between the two regimes viaα\\alpha\.

![Refer to caption](https://arxiv.org/html/2606.31178v1/x4.png)

Figure 4:Quantile distributions of per\-trajectory returns in MO\-PointMaze\-3obj environment

### 6\.2Beyond Concavity: General AET Objectives

Beyond Fair MORL, the AET framework accommodates diverse trajectory\-level utilities and their aggregation by varyingFFandGG\. We demonstrate this on the D4MORL benchmarkZhuet al\.\([2023](https://arxiv.org/html/2606.31178#bib.bib3)\), a more complex continuous\-control domain, with four configurations: \(i\) convexF​\(𝐱\)=exp⁡\(𝐱/50\)F\(\\mathbf\{x\}\)=\\exp\(\\mathbf\{x\}/50\)with linearG​\(𝐱\)G\(\\mathbf\{x\}\), which prioritizes high\-performing objectives; \(ii\) the same convexFFwith concaveG​\(𝐱\)=∑ilog⁡\(xi\)G\(\\mathbf\{x\}\)=\\sum\_\{i\}\\log\(x\_\{i\}\), which combines trajectory\-level specialization with expectation\-level balance; and \(iii\) non\-convex Cobb\-DouglasF​\(𝐱\)=x1ρ/x21−ρF\(\\mathbf\{x\}\)=x\_\{1\}^\{\\rho\}/\{x\_\{2\}\}^\{1\-\\rho\}with linearGG, a single trajectory\-wise utility case that optimizes trajectory\-wise efficiency; and \(iv\) non\-convex Cobb\-Douglasf1f\_\{1\}for efficiency and concave safety\-thresholdf2​\(x\)=log⁡\(x−Rt​h\)f\_\{2\}\(x\)=\\log\(x\-R\_\{th\}\)designed to penalize collapse, aggregated by concaveGG\.

#### Convex utility F\.

We apply convexFFwith increasing marginal utility to MO\-Ant, where the two objectives correspond to movement in orthogonal directions\. ConvexFFamplifies gains from concentrating on a single objective, encouraging the policy to commit to one direction per episode rather than distributing effort across both\. Under Linear MORL, the agent moves diagonally according to the preference weights \(Behavior C in Figure[5](https://arxiv.org/html/2606.31178#S6.F5)\)\. With convexFFand linearGG\(i\), however, the amplification effect causes the policy to specialize exclusively in one direction \(Behavior A\), even under equal weighting\. Pairing the same convexFFwith concaveGG\(ii\) addresses this at the expectation level: the concave aggregation amplifies under\-served utility dimensions, producing a stochastic mixture of Behavior A and Behavior B, where each trajectory specializes in a different direction yet balance is maintained across episodes—a behavior that is difficult to achieve with linearGG, as it requires precise weight tuning to counteract the amplification of convexFF\.

![Refer to caption](https://arxiv.org/html/2606.31178v1/x5.png)
![Refer to caption](https://arxiv.org/html/2606.31178v1/x6.png)
![Refer to caption](https://arxiv.org/html/2606.31178v1/x7.png)

Figure 5:\(a\)MO\-Ant\-v2: AETDICE with convexFFand concaveGGproduces a stochastic mixture of specialized behaviors \(Behavior A: 67%, Behavior B: 33%\), while Linear MORL with equal preference weights produces balanced diagonal movement \(Behavior C\)\.\(b\)Over\-optimization of Cobb\-Douglas utility leads to agent collapse\.\(c\)Adding safety utilityf2f\_\{2\}maintains Cobb\-Douglas efficiency while preventing collapse\.
#### Cobb\-Douglas utility\.

We focus on MO\-Walker2d and MO\-Halfcheetah as two objectives, forward locomotion and energy consumption, present a natural efficiency trade\-off\. The Cobb\-Douglas utilityF​\(𝐱\)=x1ρ/\(x2\)1−ρF\(\\mathbf\{x\}\)=x\_\{1\}^\{\\rho\}/\(x\_\{2\}\)^\{1\-\\rho\}is convex in one objective and concave in the other, making it non\-convex overall\. Despite this, the utility has a natural interpretation in these environments: by treating energy consumption as the denominator, the Cobb\-Douglas utility effectively optimizes the movement\-to\-energy ratio, capturing trajectory\-wise efficiency\. By varyingρ\\rho, one controls the relative emphasis on forward locomotion versus energy consumption\. However, at excessively low values \(e\.g\.,ρ=0\.1\\rho=0\.1\), the MO\-Walker2d agent frequently collapses, as the utility overly prioritizes energy saving at the expense of maintaining stable locomotion\.

#### Cobb\-Douglas utility with safety utility\.

To counteract the collapse caused by over\-optimizing efficiency, we introduce a safety utilityf2​\(x\)=log⁡\(x−Rth\)f\_\{2\}\(x\)=\\log\(x\-R\_\{\\mathrm\{th\}\}\)that penalizes trajectories approaching a safety threshold, analogous to a log\-barrier\. This is aggregated with the Cobb\-Douglas utility via concaveGG, forming a two\-utility AET objective:

JA​E​T​\(π~\)=log⁡\(𝔼τ∼π~​\[R​\(τ\)ρC​\(τ\)1−ρ\]\)\+log⁡\(𝔼τ∼π~​\[Rs​\(τ\)−Rt​h\]\)\\displaystyle J\_\{AET\}\(\\tilde\{\\pi\}\)=\\log\\left\(\\mathbb\{E\}\_\{\\tau\\sim\\tilde\{\\pi\}\}\\left\[\\frac\{R\(\\tau\)^\{\\rho\}\}\{C\(\\tau\)^\{1\-\\rho\}\}\\right\]\\right\)\+\\log\\left\(\\mathbb\{E\}\_\{\\tau\\sim\\tilde\{\\pi\}\}\[R\_\{s\}\(\\tau\)\-R\_\{th\}\]\\right\)\(12\)whereRt​h=0\.8​HR\_\{th\}=0\.8His set to 80% of the maximum achievable safety return\. Figure[5](https://arxiv.org/html/2606.31178#S6.F5)\(b,c\) compares AETDICE policies optimizing Cobb\-Douglas utility alone versus with the additional safety utility, showing that the safety utility successfully prevents the Walker2d agent from collapsing\. These results demonstrate that the AET framework extends beyond concave transformations to handle a broader class of practical objectives\. Full results are presented in Appendix[F](https://arxiv.org/html/2606.31178#A6)\.

Table 1:Results on MO\-Ant with convex trajectory\-level utilityFF\. Linear is abbreviated as Lin\.

## 7Conclusion

We presented AET, a unified framework that organizes nonlinear MORL objectives by the position of nonlinearity—trajectory\-level transformationFFand expectation\-level aggregationGG—and identifies two previously unaddressed regimes: offline ESR\-MORL and AET\-MORL\. To fill these gaps, we proposed AETDICE, which introduces a transformed reward that absorbs nonlinearFFinto per\-step rewards in an augmented MOMDP, and a finite\-horizon DICE formulation that handles the global optimization required by concaveGG\. Empirically, AETDICE optimizes ESR, SER, BSR, and novel AET objectives within a single framework, revealing distinct policy behaviors—including non\-Markovian strategies and stochastic mixtures—that emerge from the interaction of both levels of nonlinearity\. Limitations and broader impact are discussed in Appendix[J](https://arxiv.org/html/2606.31178#A10)\.

## 8Acknowledgements

This work was partly supported by Institute of Information&\\&Communications Technology Planning&\\&Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(No\. RS\-2022\-II220311, Development of Goal\-Oriented Reinforcement Learning Techniques for Contact\-Rich Robotic Manipulation of Everyday Objects, No\. RS\-2024\- 00457882, AI Research Hub Project, No\. RS\-2019\- II190079, Artificial Intelligence Graduate School Program \(Korea University\), the IITP \(Institute of Information&\\&Communications Technology Planning&\\&Evaluation\)\-ITRC \(Information Technology Research Center\) grant funded by the Korea government \(Ministry of Science and ICT\) \(IITP\-2026\-RS\-2024\-00436857\), BK21 Four project of the National Research Foundation of Korea, the National Research Foundation of Korea \(NRF\) grant funded by the Korea government \(MSIT\) \(RS2025\-00560367\), the IITP under the Artificial Intelligence Star Fellowship support program to nurture the best talents \(IITP\-2026\-RS\-2025\-02304828\) grant funded by the Korea government \(MSIT\)\. This work was also supported by the Institute of Information&\\&Communications Technology Planning&\\&Evaluation \(IITP\) grant \(RS\-2020\-II201361, Artificial Intelligence Graduate School Program \(Yonsei University\), and the AI Computing Infrastructure Enhancement \(GPU Rental Support\) User Support Program funded by the Ministry of Science and ICT \(MSIT\) \(No\. RQT\-25\-090109\), Republic of Korea\.

## References

- \[1\]\(2022\)Multi\-objective reinforcement learning with non\-linear scalarization\.InProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems,pp\. 9–17\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.31178#S1.p1.1)\.
- \[2\]L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. Mordatch\(2021\)Decision transformer: reinforcement learning via sequence modeling\.Advances in neural information processing systems34,pp\. 15084–15097\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p1.1)\.
- \[3\]W\. C\. Cheung\(2019\)Regret minimization for reinforcement learning with vectorial feedback and complex objectives\.Advances in Neural Information Processing Systems32\.Cited by:[§1](https://arxiv.org/html/2606.31178#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.31178#S2.SS1.p2.9),[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.SSS0.Px2.p1.11)\.
- \[4\]Z\. Fan, N\. Peng, M\. Tian, and B\. Fain\(2023\)Welfare and fairness in multi\-objective reinforcement learning\.InProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems,pp\. 1991–1999\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px2.p1.1),[§E\.1](https://arxiv.org/html/2606.31178#A5.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.31178#S2.SS1.p2.9),[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.SSS0.Px2.p1.11)\.
- \[5\]J\. Fu, A\. Kumar, O\. Nachum, G\. Tucker, and S\. Levine\(2020\)D4RL: datasets for deep data\-driven reinforcement learning\.External Links:2004\.07219Cited by:[§E\.2](https://arxiv.org/html/2606.31178#A5.SS2.p1.1)\.
- \[6\]C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz,et al\.\(2021\)A practical guide to multi\-objective reinforcement learning and planning\.arXiv preprint arXiv:2103\.09568\.Cited by:[§1](https://arxiv.org/html/2606.31178#S1.p2.1)\.
- \[7\]C\. F\. Hayes, T\. Verstraeten, D\. M\. Roijers, E\. Howley, and P\. Mannion\(2025\)Expected scalarised returns dominance: a new solution concept for multi\-objective decision making\.Neural Computing and Applications37\(19\),pp\. 13079–13099\.Cited by:[§1](https://arxiv.org/html/2606.31178#S1.p2.1)\.
- \[8\]M\. Kaneko and K\. Nakamura\(1979\)The nash social welfare function\.Econometrica: Journal of the Econometric Society,pp\. 423–435\.Cited by:[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.p1.1)\.
- \[9\]W\. Kim, J\. Lee, J\. Lee, and B\. Lee\(2025\)FairDICE: fairness\-driven offline multi\-objective reinforcement learning\.arXiv preprint arXiv:2506\.08062\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p2.1),[§C\.1](https://arxiv.org/html/2606.31178#A3.SS1.SSS0.Px2.p1.8),[§C\.1](https://arxiv.org/html/2606.31178#A3.SS1.p1.3),[§1](https://arxiv.org/html/2606.31178#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.31178#S2.SS1.p2.9),[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.31178#S3.SS0.SSS0.Px1.p2.3),[§5\.2](https://arxiv.org/html/2606.31178#S5.SS2.SSS0.Px1.p1.3),[§5\.2](https://arxiv.org/html/2606.31178#S5.SS2.SSS0.Px3.p1.3),[§5](https://arxiv.org/html/2606.31178#S5.p1.5),[§6](https://arxiv.org/html/2606.31178#S6.SS0.SSS0.Px1.p1.8),[§6\.1](https://arxiv.org/html/2606.31178#S6.SS1.p1.2)\.
- \[10\]I\. Kostrikov, A\. Nair, and S\. Levine\(2021\)Offline reinforcement learning with implicit q\-learning\.arXiv preprint arXiv:2110\.06169\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2606.31178#S4.SS2.p1.5)\.
- \[11\]A\. Kumar, A\. Zhou, G\. Tucker, and S\. Levine\(2020\)Conservative q\-learning for offline reinforcement learning\.Advances in neural information processing systems33,pp\. 1179–1191\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p1.1)\.
- \[12\]J\. Lee, W\. Jeon, B\. Lee, J\. Pineau, and K\. Kim\(2021\)Optidice: offline policy optimization via stationary distribution correction estimation\.InInternational Conference on Machine Learning,pp\. 6120–6130\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p1.1),[§C\.1](https://arxiv.org/html/2606.31178#A3.SS1.SSS0.Px1.p1.10),[§C\.4](https://arxiv.org/html/2606.31178#A3.SS4.p1.6),[§5\.2](https://arxiv.org/html/2606.31178#S5.SS2.SSS0.Px1.p1.3),[§5\.2](https://arxiv.org/html/2606.31178#S5.SS2.SSS0.Px2.p1.11),[§5\.2](https://arxiv.org/html/2606.31178#S5.SS2.SSS0.Px3.p1.3),[§5](https://arxiv.org/html/2606.31178#S5.p1.5),[§6](https://arxiv.org/html/2606.31178#S6.SS0.SSS0.Px1.p1.8)\.
- \[13\]Q\. Lin, C\. Yu, Z\. Liu, and Z\. Wu\(2024\)Policy\-regularized offline multi\-objective reinforcement learning\.InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems,pp\. 1201–1209\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p2.1),[§1](https://arxiv.org/html/2606.31178#S1.p3.1),[§3](https://arxiv.org/html/2606.31178#S3.SS0.SSS0.Px1.p2.3)\.
- \[14\]D\. J\. Lizotte, M\. H\. Bowling, and S\. A\. Murphy\(2010\)Efficient reinforcement learning with multiple reward functions for randomized controlled trial analysis\.\.InICML,Vol\.10,pp\. 695–702\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px1.p1.1)\.
- \[15\]J\. Mo and J\. Walrand\(2002\)Fair end\-to\-end window\-based congestion control\.IEEE/ACM Transactions on networking8\(5\),pp\. 556–567\.Cited by:[§6\.1](https://arxiv.org/html/2606.31178#S6.SS1.p1.2)\.
- \[16\]G\. Park, W\. Byeon, S\. Kim, E\. Havakuk, A\. Leshem, and Y\. Sung\(2024\)The max\-min formulation of multi\-objective reinforcement learning: from theory to a model\-free algorithm\.InInternational Conference on Machine Learning,pp\. 39616–39642\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px2.p1.1),[§6\.1](https://arxiv.org/html/2606.31178#S6.SS1.p1.2)\.
- \[17\]N\. Peng and B\. Fain\(2023\)Nonlinear multi\-objective reinforcement learning with provable guarantees\.arXiv preprint arXiv:2311\.02544\.Cited by:[§3](https://arxiv.org/html/2606.31178#S3.SS0.SSS0.Px1.p2.3)\.
- \[18\]N\. Peng, M\. Tian, and B\. Fain\(2025\)Multi\-objective reinforcement learning with nonlinear preferences: provable approximation for maximizing expected scalarized return\.InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems,pp\. 1632–1640\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px2.p1.1),[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.SSS0.Px2.p1.11),[§4\.2](https://arxiv.org/html/2606.31178#S4.SS2.SSS0.Px1.p2.4),[§4](https://arxiv.org/html/2606.31178#S4.p1.2)\.
- \[19\]R\. Rădulescu, P\. Mannion, Y\. Zhang, D\. M\. Roijers, and A\. Nowé\(2020\)A utility\-based analysis of equilibria in multi\-objective normal\-form games\.The Knowledge Engineering Review35,pp\. e32\.Cited by:[§2\.1](https://arxiv.org/html/2606.31178#S2.SS1.p2.9)\.
- \[20\]D\. M\. Roijers, P\. Vamplew, S\. Whiteson, and R\. Dazeley\(2013\)A survey of multi\-objective sequential decision\-making\.Journal of Artificial Intelligence Research48,pp\. 67–113\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.31178#S1.p1.1),[§6\.1](https://arxiv.org/html/2606.31178#S6.SS1.SSS0.Px3.p1.1)\.
- \[21\]D\. Roijers, D\. Steckelmacher, and A\. Nowé\(2018\)Multi\-objective reinforcement learning for the expected utility of the return\.InAdaptive Learning Agents Workshop 2018,Cited by:[§1](https://arxiv.org/html/2606.31178#S1.p2.1)\.
- \[22\]U\. Siddique, P\. Li, and Y\. CaoLearning fair pareto\-optimal policies in multi\-objective reinforcement learning\.InThe Seventeenth Workshop on Adaptive and Learning Agents,Cited by:[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.SSS0.Px2.p1.11),[§3](https://arxiv.org/html/2606.31178#S3.SS0.SSS0.Px1.p2.3),[§4\.2](https://arxiv.org/html/2606.31178#S4.SS2.SSS0.Px1.p1.2)\.
- \[23\]U\. Siddique, A\. Sinha, and Y\. Cao\(2023\)Fairness in preference\-based reinforcement learning\.arXiv preprint arXiv:2306\.09995\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px2.p1.1)\.
- \[24\]J\. Skalse and A\. Abate\(2023\)On the limitations of markovian rewards to express multi\-objective, risk\-sensitive, and modal tasks\.InUncertainty in Artificial Intelligence,pp\. 1974–1984\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px2.p1.1)\.
- \[25\]K\. Van Moffaert, M\. M\. Drugan, and A\. Nowé\(2013\)Hypervolume\-based multi\-objective reinforcement learning\.InInternational Conference on Evolutionary Multi\-Criterion Optimization,pp\. 352–366\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px1.p1.1)\.
- \[26\]K\. Van Moffaert, M\. M\. Drugan, and A\. Nowé\(2013\)Scalarized multi\-objective reinforcement learning: novel design techniques\.In2013 IEEE symposium on adaptive dynamic programming and reinforcement learning \(ADPRL\),pp\. 191–199\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px1.p1.1)\.
- \[27\]K\. Van Moffaert and A\. Nowé\(2014\)Multi\-objective reinforcement learning using sets of pareto dominating policies\.The Journal of Machine Learning Research15\(1\),pp\. 3483–3512\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.31178#S1.p1.1)\.
- \[28\]R\. Wu, Y\. Zhang, Z\. Yang, and Z\. Wang\(2021\)Offline constrained multi\-objective reinforcement learning via pessimistic dual value iteration\.Advances in Neural Information Processing Systems34,pp\. 25439–25451\.Cited by:[Appendix A](https://arxiv.org/html/2606.31178#A1.SS0.SSS0.Px3.p2.1)\.
- \[29\]R\. Yang, X\. Sun, and K\. Narasimhan\(2019\)A generalized algorithm for multi\-objective reinforcement learning and policy adaptation\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2606.31178#S1.p1.1)\.
- \[30\]G\. Yu, U\. Siddique, and P\. Weng\(2023\)Fair deep reinforcement learning with preferential treatment\.\.InECAI,pp\. 2922–2929\.Cited by:[§2\.2](https://arxiv.org/html/2606.31178#S2.SS2.SSS0.Px2.p1.11),[§3](https://arxiv.org/html/2606.31178#S3.SS0.SSS0.Px1.p2.3),[§4\.2](https://arxiv.org/html/2606.31178#S4.SS2.SSS0.Px1.p1.2)\.
- \[31\]B\. Zhu, M\. Dang, and A\. Grover\(2023\)Scaling pareto\-efficient decision making via offline multi\-objective rl\.arXiv preprint arXiv:2305\.00567\.Cited by:[§E\.3](https://arxiv.org/html/2606.31178#A5.SS3.SSS0.Px2.p1.1),[§E\.3](https://arxiv.org/html/2606.31178#A5.SS3.p1.1),[Table 11](https://arxiv.org/html/2606.31178#A8.T11),[Table 11](https://arxiv.org/html/2606.31178#A8.T11.18.2),[§1](https://arxiv.org/html/2606.31178#S1.p3.1),[§3](https://arxiv.org/html/2606.31178#S3.SS0.SSS0.Px1.p2.3),[§6\.2](https://arxiv.org/html/2606.31178#S6.SS2.p1.11)\.

## Appendix ARelated Works

#### Multi\-Objective Reinforcement Learning

Multi\-objective reinforcement learning \(MORL\) studies sequential decision\-making problems with vector\-valued rewards, where no single policy is universally optimal under the Pareto dominance relation\[[20](https://arxiv.org/html/2606.31178#bib.bib1)\]\. As a result, MORL methods often aim to compute a set of trade\-off solutions \(e\.g\., Pareto\-optimal or coverage sets\) so that a decision maker can later select a policy once preferences are specified\[[26](https://arxiv.org/html/2606.31178#bib.bib33),[27](https://arxiv.org/html/2606.31178#bib.bib13)\]\. A widely used instantiation is linear scalarization, which reduces the problem to a standard single\-objective MDP by optimizing a weighted sum of objectives and thereby enables the use of conventional RL algorithms\[[14](https://arxiv.org/html/2606.31178#bib.bib29),[25](https://arxiv.org/html/2606.31178#bib.bib14)\]\. This perspective has led to two main solution paradigms: set\-based approaches, which learn a collection of Pareto\-optimal policies, and preference\-conditioned methods, which adapt a single policy to different trade\-offs\[[20](https://arxiv.org/html/2606.31178#bib.bib1),[26](https://arxiv.org/html/2606.31178#bib.bib33),[27](https://arxiv.org/html/2606.31178#bib.bib13)\]\.

#### Nonlinear Preference in MORL

However, linear scalarization cannot capture richer preference structures such as fairness, balance, or risk sensitivity\[[24](https://arxiv.org/html/2606.31178#bib.bib34)\]\. To address this limitation, recent work has explored nonlinear scalarization based on concave utility functions applied to return vectors\. Under appropriate smoothness and concavity conditions, such nonlinear objectives remain tractable to optimize\[[1](https://arxiv.org/html/2606.31178#bib.bib6),[18](https://arxiv.org/html/2606.31178#bib.bib7)\]\. This line of work includes welfare\-oriented criteria such as Nash social welfare and Gini\-based measures, which encourage equitable performance across objectives\[[4](https://arxiv.org/html/2606.31178#bib.bib18),[23](https://arxiv.org/html/2606.31178#bib.bib35)\]\. Related approaches also study max–min formulations that explicitly promote balanced outcomes, often combined with entropy regularization to stabilize learning in model\-free online settings\[[16](https://arxiv.org/html/2606.31178#bib.bib36)\]\. Together, these works reflect a broader shift from linear trade\-offs toward more expressive preference modeling in MORL\.

#### Offline RL and Offline MORL

Offline reinforcement learning \(RL\) focuses on learning policies from fixed datasets without additional environment interaction, where distributional shift is a central challenge\. Existing approaches address this issue through conservative value estimation\[[11](https://arxiv.org/html/2606.31178#bib.bib37)\], implicit value regularization\[[10](https://arxiv.org/html/2606.31178#bib.bib32)\], divergence\-regularized optimization\[[12](https://arxiv.org/html/2606.31178#bib.bib10)\], and return\-conditioned sequence modeling\[[2](https://arxiv.org/html/2606.31178#bib.bib38)\]\.

In the multi\-objective setting, most offline approaches rely on linear scalarization and require explicit preference conditioning during training or evaluation\[[28](https://arxiv.org/html/2606.31178#bib.bib8),[13](https://arxiv.org/html/2606.31178#bib.bib4)\]\. FairDICE\[[9](https://arxiv.org/html/2606.31178#bib.bib9)\]adapts stationary distribution correction to a nonlinear welfare objective, showing that certain nonlinear criteria can be handled in offline MORL\. However, it is tailored to a specific welfare formulation and does not directly generalize to broader classes of nonlinear objectives with different placements of nonlinearity in the return structure\.

## Appendix BOptimal Policies in the Two\-Step MOMDP Example

In this section, we derive the optimal policies for linear, SER, and ESR objectives in the two\-step MOMDP shown in Figure[6](https://arxiv.org/html/2606.31178#A2.F6)\(reproduced from Figure[1](https://arxiv.org/html/2606.31178#S2.F1)for convenience\)\. Through these derivations, we illustrate the distinct optimization challenges that arise from nonlinear MORL objectives\.

s0s\_\{0\}s1s\_\{1\}sTs\_\{T\}a0:\(0,0\)\{a\_\{0\}\}:\(0,0\)a1:\(9,1\)\{a\_\{1\}\}:\(9,1\)a2:\(4,4\)\{a\_\{2\}\}:\(4,4\)a3:\(1,9\)\{a\_\{3\}\}:\(1,9\)

Obj\.Optimalπ∗\\pi^\{\*\}uNSW​\(𝔼​\[𝐑\]\)u\_\{\\text\{NSW\}\}\(\\mathbb\{E\}\[\\mathbf\{R\}\]\)𝔼​\[uNSW​\(𝐑\)\]\\mathbb\{E\}\[u\_\{\\text\{NSW\}\}\(\\mathbf\{R\}\)\]SER0\.5​a1\+0\.5​a30\.5\\,a\_\{1\}\+0\.5\\,a\_\{3\}𝐥𝐨𝐠⁡\(𝟐𝟓\)\\boldsymbol\{\\log\(25\)\}log⁡\(9\)\\log\(9\)ESRa2a\_\{2\}log⁡\(16\)\\log\(16\)𝐥𝐨𝐠⁡\(𝟏𝟔\)\\boldsymbol\{\\log\(16\)\}Lineara1a\_\{1\}ora3a\_\{3\}log⁡\(9\)\\log\(9\)log⁡\(9\)\\log\(9\)

Figure 6:Optimal policies of MORL with linear, SER, and ESR objectives in a two\-step MOMDP withuNSW​\(𝐱\)=∑ilog⁡\(Ri\)u\_\{\\text\{NSW\}\}\(\\mathbf\{x\}\)=\\sum\_\{i\}\\log\(R\_\{i\}\)\.#### Linear scalarization\.

For linear preference weights\[w1,w2\]∈ℝ\+2\[w\_\{1\},w\_\{2\}\]\\in\\mathbb\{R\}^\{2\}\_\{\+\}, the linear MORL objective is defined asJlin​\(π;𝐰\)=𝔼π​\[w1​R1\+w2​R2\]J\_\{\\mathrm\{lin\}\}\(\\pi;\\mathbf\{w\}\)=\\mathbb\{E\}\_\{\\pi\}\[w\_\{1\}R\_\{1\}\+w\_\{2\}R\_\{2\}\]\. The optimal policy selectsa1a\_\{1\}whenw1\>w2w\_\{1\}\>w\_\{2\}anda3a\_\{3\}whenw1<w2w\_\{1\}<w\_\{2\}, with both SERuNSW​\(𝔼​\[𝐑\]\)u\_\{\\text\{NSW\}\}\(\\mathbb\{E\}\[\\mathbf\{R\}\]\)and ESR𝔼​\[uNSW​\(𝐑\)\]\\mathbb\{E\}\[u\_\{\\text\{NSW\}\}\(\\mathbf\{R\}\)\]remaining constant atlog⁡9\\log 9in either case\. The exception isw1=w2w\_\{1\}=w\_\{2\}, wherea1a\_\{1\}anda3a\_\{3\}are equally optimal, making any stochastic mixture also optimal\. However, Scalarized Expected Return \(SER\) is no longer constant in this case, as it varies with the mixing probability\.

#### Scalarized Expected Return \(SER\)\.

SER maximizesJSER​\(π\)=uNSW​\(𝔼π​\[𝐑\]\)J\_\{\\mathrm\{SER\}\}\(\\pi\)=u\_\{\\mathrm\{NSW\}\}\(\\mathbb\{E\}\_\{\\pi\}\[\\mathbf\{R\}\]\)\. Unlike the linear case, optimizing SER is not straightforward: lettingp1p\_\{1\},p2p\_\{2\},p3p\_\{3\}denote the probabilities of selectinga1a\_\{1\},a2a\_\{2\},a3a\_\{3\}, one must solve a convex optimization over the full policy distribution\. This reflects a key challenge of SER optimization—the nonlinearity applies to the expected return, requiring global optimization over the entire policy rather than local, per\-action evaluation\. The convex optimization can be written as:

maxp1,p2,p3\\displaystyle\\max\_\{p\_\{1\},p\_\{2\},p\_\{3\}\}\\quadlog⁡\(9​p1\+4​p2\+p3\)\+log⁡\(p1\+4​p2\+9​p3\)\\displaystyle\\log\(9p\_\{1\}\+4p\_\{2\}\+p\_\{3\}\)\+\\log\(p\_\{1\}\+4p\_\{2\}\+9p\_\{3\}\)s\.t\.p1\+p2\+p3=1,p1,p2,p3≥0\.\\displaystyle p\_\{1\}\+p\_\{2\}\+p\_\{3\}=1,\\quad p\_\{1\},p\_\{2\},p\_\{3\}\\geq 0\.\(13\)where the optimal solution isp1∗=0\.5p\_\{1\}^\{\*\}=0\.5,p2∗=0p\_\{2\}^\{\*\}=0,p3∗=0\.5p\_\{3\}^\{\*\}=0\.5, yielding a stochastic policy that mixesa1a\_\{1\}anda3a\_\{3\}equally\.

#### Expected Scalarized Return \(ESR\)\.

ESR maximizesJESR​\(π\)=𝔼π​\[uNSW​\(𝐑\)\]J\_\{\\mathrm\{ESR\}\}\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\[u\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\)\]\. Since the nonlinearity is applies before the expectation, each action can be evaluated independently:uNSW​\(𝐑​\(a1\)\)=log⁡9u\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\(a\_\{1\}\)\)=\\log 9,uNSW​\(𝐑​\(a2\)\)=log⁡16u\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\(a\_\{2\}\)\)=\\log 16,uNSW​\(𝐑​\(a3\)\)=log⁡9u\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\(a\_\{3\}\)\)=\\log 9\. The optimal policy is therefore deterministic:π∗=a2\\pi^\{\*\}=a\_\{2\}\.

### B\.1Demonstrating ESR non\-Markovianity

By changing the initial state\-action pair\(s0,a0\)\(s\_\{0\},a\_\{0\}\)such that the reward ats0s\_\{0\}becomes\(2,0\)\(2,0\)instead of\(0,0\)\(0,0\), the previously accumulated reward ats1s\_\{1\}shifts from\(0,0\)\(0,0\)to\(2,0\)\(2,0\)\. This changes each trajectory’s return vector to𝐑​\(a1\)=\(11,1\)\\mathbf\{R\}\(a\_\{1\}\)=\(11,1\),𝐑​\(a2\)=\(6,4\)\\mathbf\{R\}\(a\_\{2\}\)=\(6,4\),𝐑​\(a3\)=\(3,9\)\\mathbf\{R\}\(a\_\{3\}\)=\(3,9\), yieldinguNSW​\(𝐑​\(a1\)\)=log⁡11u\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\(a\_\{1\}\)\)=\\log 11,uNSW​\(𝐑​\(a2\)\)=log⁡24u\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\(a\_\{2\}\)\)=\\log 24,\(𝐑\(a3\)\)NSW=log27\{\}\_\{\\mathrm\{NSW\}\}\(\\mathbf\{R\}\(a\_\{3\}\)\)=\\log 27\. The optimal action ats1s\_\{1\}shifts froma2a\_\{2\}toa3a\_\{3\}, demonstrating that the ESR\-optimal policy depends on the accumulated reward and is therefore non\-Markovian in the original state space\.

## Appendix CDerivation of the AETDICE Loss Functions

In this section, we provide the full derivation of the AETDICE loss functions\. AETDICE assumes the augmented MOMDPℳ~\\tilde\{\\mathcal\{M\}\}defined in[Section˜4](https://arxiv.org/html/2606.31178#S4)with the transformed reward:

𝐫~​\(s~t,at\)=F​\(𝐑tacc\+𝐫​\(st,at\)\)−F​\(𝐑tacc\)∈ℝn\.\\displaystyle\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)=F\\\!\\left\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\+\\mathbf\{r\}\(s\_\{t\},a\_\{t\}\)\\right\)\-F\\\!\\left\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\\right\)\\;\\in\\;\\mathbb\{R\}^\{n\}\.\(14\)The AET\-MORL objectivemaxπ~⁡JAET​\(π~;F,G\)\\max\_\{\\tilde\{\\pi\}\}J\_\{\\mathrm\{AET\}\}\(\\tilde\{\\pi\};F,G\)is reformulated to a convex optimization in terms of𝐝=\{dt​\(s~,a\)\}t=0H−1\\mathbf\{d\}=\\\{d\_\{t\}\(\\tilde\{s\},a\)\\\}\_\{t=0\}^\{H\-1\}, the visitation distributions inℳ~\\tilde\{\\mathcal\{M\}\}as

max𝐝≥0\\displaystyle\\max\_\{\\mathbf\{d\}\\geq 0\}\\quadG​\(∑t=0H−1∑s~,adt​\(s~,a\)​𝐫~​\(s~,a\)\)\\displaystyle G\\\!\\left\(\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\\right\)\(15a\)s\.t\.∑ad0​\(s~,a\)=p~0​\(s~\),∀s~\\displaystyle\\sum\_\{a\}d\_\{0\}\(\\tilde\{s\},a\)=\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\),\\quad\\forall\\tilde\{s\}\(15b\)∑a′dt\+1​\(s~′,a′\)=∑s~,adt​\(s~,a\)​P~​\(s~′∣s~,a\),∀s~′,t∈\{0,…,H−2\},\\displaystyle\\sum\_\{a^\{\\prime\}\}d\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\},a^\{\\prime\}\)=\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\),\\quad\\forall\\tilde\{s\}^\{\\prime\},\\,t\\in\\\{0,\\dots,H\-2\\\},\(15c\)wheres~′=\(s′,𝐑tacc\+𝐫​\(s,a\),t\+1\)\\tilde\{s\}^\{\\prime\}=\(s^\{\\prime\},\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\+\\mathbf\{r\}\(s,a\),t\+1\)\. No flow constraint is imposed att=H−1t=H\-1, as all transitions fromdH−1d\_\{H\-1\}lead to the terminal state\.

### C\.1Regularized Primal Problem

To enable sample\-based optimization of the problem in offline setting, we follow the derivation from\[[9](https://arxiv.org/html/2606.31178#bib.bib9)\]by \(i\) addingϕ\\phi\-divergence regularization against per\-timestep data distributiondtDd^\{D\}\_\{t\}and \(ii\) introducing a slack variable𝐤∈ℝn\\mathbf\{k\}\\in\\mathbb\{R\}^\{n\}to relocate the expected utility vector outside the concave aggregation:

max𝐝≥0,𝐤∈ℝn\\displaystyle\\max\_\{\\mathbf\{d\}\\geq 0,\\,\\mathbf\{k\}\\in\\mathbb\{R\}^\{n\}\}\\quadG​\(𝐤\)−β​∑t=0H−1Dϕ​\(dt∥dtD\)\\displaystyle G\(\\mathbf\{k\}\)\-\\beta\\sum\_\{t=0\}^\{H\-1\}D\_\{\\phi\}\(d\_\{t\}\\,\\\|\\,d\_\{t\}^\{D\}\)\(16a\)s\.t\.∑t=0H−1∑s~,adt​\(s~,a\)​𝐫~​\(s~,a\)=𝐤,\\displaystyle\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)=\\mathbf\{k\},\(16b\)∑ad0​\(s~,a\)=p~0​\(s~\),∀s~∈𝒮~,\\displaystyle\\sum\_\{a\}d\_\{0\}\(\\tilde\{s\},a\)=\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\),\\qquad\\forall\\tilde\{s\}\\in\\tilde\{\\mathcal\{S\}\},\(16c\)∑a′dt\+1​\(s~′,a′\)=∑s~,adt​\(s~,a\)​P~​\(s~′∣s~,a\),∀s~′,t∈\{0,…,H−2\},\\displaystyle\\sum\_\{a^\{\\prime\}\}d\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\},a^\{\\prime\}\)=\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\),\\quad\\forall\\tilde\{s\}^\{\\prime\},\\,t\\in\\\{0,\\dots,H\-2\\\},\(16d\)where no flow constraint is imposed att=Ht=H, as all transitions fromdH−1d\_\{H\-1\}lead to the terminal state\.

#### Assumptions\.

We require: \(i\)GGis concave and continuous; \(ii\)ϕ\\phiis strictly convex, withϕ​\(1\)=0\\phi\(1\)=0; and \(iii\) the data distribution satisfiesdtD​\(s~,a\)\>0d^\{D\}\_\{t\}\(\\tilde\{s\},a\)\>0for all reachable\(s~,a\)\(\\tilde\{s\},a\)andtt, so that theϕ\\phi\-divergence is well\-defined and Slater’s condition is satisfied\. Assumption \(iii\) is stronger than the corresponding coverage assumption in DICE\-based offline RL\[[12](https://arxiv.org/html/2606.31178#bib.bib10)\], which only requires coverage over the original state\-action space\. In the augmented MOMDP, coverage must additionally hold over the accumulated reward𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}, yet the set of𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}values reachable from a given state is generally a strict subset ofℝm\\mathbb\{R\}^\{m\}\. This can be partially mitigated by initializing trajectories in the data with nonzero accumulated rewards, but the coverage requirement remains fundamentally more demanding than in the original MOMDP\.

#### Role of the slack variable k\.

The slack variable𝐤\\mathbf\{k\}is a technique first introduced in\[[9](https://arxiv.org/html/2606.31178#bib.bib9)\], which decouples the expectation over𝐝\\mathbf\{d\}from the concave aggregationGG\. By enforcing \([16b](https://arxiv.org/html/2606.31178#A3.E16.2)\), the expected transformed reward is computed separately and passed toGGthrough𝐤\\mathbf\{k\}\. This decoupling is essential for deriving an unbiased sample\-based approximation of the Lagrangian dual: without𝐤\\mathbf\{k\}, estimatingG​\(∑t∑s~,adt​\(s~,a\)​𝐫~​\(s~,a\)\)G\\\!\\left\(\\sum\_\{t\}\\sum\_\{\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\\right\)from samples would require applyingGGto an empirical average, which is biased by Jensen’s inequality\.

### C\.2Lagrangian Dual

We derive the Lagrangian dual of \([16](https://arxiv.org/html/2606.31178#A3.E16)\) by introducing multipliersνt​\(s~\)\\nu\_\{t\}\(\\tilde\{s\}\)for each Bellman\-flow constraints \([16c](https://arxiv.org/html/2606.31178#A3.E16.3)\)–\([16d](https://arxiv.org/html/2606.31178#A3.E16.4)\), and𝝁∈ℝn\\boldsymbol\{\\mu\}\\in\\mathbb\{R\}^\{n\}for the slack constraint \([16b](https://arxiv.org/html/2606.31178#A3.E16.2)\):

max𝐝≥0,𝐤⁡min𝝂,𝝁⁡ℒ​\(𝐝,𝐤,𝝂,𝝁\)\\displaystyle\\max\_\{\\mathbf\{d\}\\geq 0,\\mathbf\{k\}\}\\,\\min\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\}\\ \\mathcal\{L\}\(\\mathbf\{d\},\\mathbf\{k\},\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\)=G​\(𝐤\)−𝝁⊤​\(∑t,s~,adt​\(s~,a\)​𝐫~​\(s~,a\)−𝐤\)−β​∑t=0H−1Dϕ​\(dt∥dtD\)\\displaystyle=G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\\!\\top\}\\\!\\left\(\\sum\_\{t,\\tilde\{s\},a\}d\_\{t\}\(\\tilde\{s\},a\)\\,\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\-\\mathbf\{k\}\\right\)\-\\beta\\sum\_\{t=0\}^\{H\-1\}D\_\{\\phi\}\(d\_\{t\}\\,\\\|\\,d\_\{t\}^\{D\}\)\+∑s~ν0​\(s~\)​\(p~0​\(s~\)−∑ad0​\(s~,a\)\)\\displaystyle\\quad\+\\sum\_\{\\tilde\{s\}\}\\nu\_\{0\}\(\\tilde\{s\}\)\\\!\\left\(\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\)\-\\sum\_\{a\}d\_\{0\}\(\\tilde\{s\},a\)\\right\)\+∑t=0H−2∑s~νt​\(s~′\)​\(∑s~,aP~​\(s~′∣s~,a\)​dt​\(s~,a\)−∑adt\+1​\(s~′,a\)\)\.\\displaystyle\\quad\+\\sum\_\{t=0\}^\{H\-2\}\\sum\_\{\\tilde\{s\}\}\\nu\_\{t\}\(\\tilde\{s\}^\{\\prime\}\)\\\!\\left\(\\sum\_\{\\tilde\{s\},a\}\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\)\\,d\_\{t\}\(\\tilde\{s\},a\)\-\\sum\_\{a\}d\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\},a\)\\right\)\.\(17\)
To enable sample\-based evaluation of the dual function, we reorganize the last term of the dual function into expectations with respect to the visitation distribution𝐝\\mathbf\{d\}:

∑s~ν0​\(s~\)​p~0​\(s~\)−∑s~,aν0​\(s~\)​d0​\(s~,a\)\\displaystyle\\sum\_\{\\tilde\{s\}\}\\nu\_\{0\}\(\\tilde\{s\}\)\\,\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\)\\;\-\\;\\sum\_\{\\tilde\{s\},a\}\\nu\_\{0\}\(\\tilde\{s\}\)\\,d\_\{0\}\(\\tilde\{s\},a\)\+∑s~,ad0​\(s~,a\)​∑s~′P~​\(s~′∣s~,a\)​ν1​\(s~′\)−∑s~,aν1​\(s~\)​d1​\(s~,a\)\\displaystyle\\quad\+\\;\\sum\_\{\\tilde\{s\},a\}d\_\{0\}\(\\tilde\{s\},a\)\\sum\_\{\\tilde\{s\}^\{\\prime\}\}\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\)\\,\\nu\_\{1\}\(\\tilde\{s\}^\{\\prime\}\)\\;\-\\;\\sum\_\{\\tilde\{s\},a\}\\nu\_\{1\}\(\\tilde\{s\}\)\\,d\_\{1\}\(\\tilde\{s\},a\)\+∑s~,ad1​\(s~,a\)​∑s~′P~​\(s~′∣s~,a\)​ν2​\(s~′\)−⋯\\displaystyle\\quad\+\\;\\sum\_\{\\tilde\{s\},a\}d\_\{1\}\(\\tilde\{s\},a\)\\sum\_\{\\tilde\{s\}^\{\\prime\}\}\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\)\\,\\nu\_\{2\}\(\\tilde\{s\}^\{\\prime\}\)\\;\-\\;\\cdots−∑s~,aνH−1​\(s~\)​dH−1​\(s~,a\)\+∑s~,adH−1​\(s~,a\)​∑s~′P~​\(s~′∣s~,a\)​νH​\(s~′\)⏟:=0\.\\displaystyle\\quad\-\\;\\sum\_\{\\tilde\{s\},a\}\\nu\_\{H\-1\}\(\\tilde\{s\}\)\\,d\_\{H\-1\}\(\\tilde\{s\},a\)\+\\;\\sum\_\{\\tilde\{s\},a\}d\_\{H\-1\}\(\\tilde\{s\},a\)\\sum\_\{\\tilde\{s\}^\{\\prime\}\}\\tilde\{P\}\(\\tilde\{s\}^\{\\prime\}\\mid\\tilde\{s\},a\)\\,\\underbrace\{\\nu\_\{H\}\(\\tilde\{s\}^\{\\prime\}\)\}\_\{:=\\,0\}\.\(18\)where we adoptνH​\(s~\):=0\\nu\_\{H\}\(\\tilde\{s\}\):=0for alls~∈𝒮~\\tilde\{s\}\\in\\tilde\{\\mathcal\{S\}\}for notational uniformity\. This enables the derivation of the reorganized Lagrangian dual:

ℒ​\(d,𝐤,𝝂,𝝁\)=\\displaystyle\\mathcal\{L\}\(d,\\mathbf\{k\},\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\)=G\(𝐤\)−𝝁⊤𝐤−β∑t=0H−1Dϕ\(dt\|\|dtD\)\+∑s~ν0\(s~\)p~0\(s~\)\+∑s~,a,tdt\(s~,a\)e𝝂,𝝁,t\(s~,a\)\\displaystyle G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\top\}\\mathbf\{k\}\-\\beta\\sum\_\{t=0\}^\{H\-1\}D\_\{\\phi\}\(d\_\{t\}\|\|d^\{D\}\_\{t\}\)\+\\sum\_\{\\tilde\{s\}\}\\nu\_\{0\}\(\\tilde\{s\}\)\\tilde\{p\}\_\{0\}\(\\tilde\{s\}\)\+\\sum\_\{\\tilde\{s\},a,t\}d\_\{t\}\(\\tilde\{s\},a\)e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)wheree𝝂,𝝁,t​\(s~,a\)=𝝁⊤​𝐫~​\(s~,a\)\+𝔼s~′​\[νt\+1​\(s~′\)\]−νt​\(s~\)e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)=\\boldsymbol\{\\mu\}^\{\\top\}\\tilde\{\\mathbf\{r\}\}\(\\tilde\{s\},a\)\+\\mathbb\{E\}\_\{\\tilde\{s\}^\{\\prime\}\}\[\\nu\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\}\)\]\-\\nu\_\{t\}\(\\tilde\{s\}\)\.

### C\.3Switching the order of optimization via strong duality

We leverage strong duality of \([16](https://arxiv.org/html/2606.31178#A3.E16)\) enabled by the data coverage assumption and satisfaction of Slater’s condition to swap the optimization order tomin𝝂,𝝁⁡max𝐝≥0,𝐤⁡ℒ\\min\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\}\\;\\max\_\{\\mathbf\{d\}\\geq 0,\\,\\mathbf\{k\}\}\\;\\mathcal\{L\}\. This yields a closed\-form solution for𝐝\\mathbf\{d\}and enables simplification of the dual function via conjugate functions\.

We define the per\-timestep visitation ratiowt​\(s~,a\):=dt​\(s~,a\)dtD​\(s~,a\)w\_\{t\}\(\\tilde\{s\},a\):=\\frac\{d\_\{t\}\(\\tilde\{s\},a\)\}\{d\_\{t\}^\{D\}\(\\tilde\{s\},a\)\}prior to obtaining the closed\-form solution to further simplify the Lagrangian:

ℒ=G​\(𝐤\)−𝝁⊤​𝐤\+𝔼p~0​\[ν0​\(s~\)\]\+∑t=0H−1𝔼\(s~,a\)∼dtD​\[wt​\(s~,a\)​e𝝂,𝝁,t​\(s~,a\)−β​ϕ​\(wt​\(s~,a\)\)\]\.\\displaystyle\\mathcal\{L\}=G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\\!\\top\}\\mathbf\{k\}\+\\mathbb\{E\}\_\{\\tilde\{p\}\_\{0\}\}\[\\nu\_\{0\}\(\\tilde\{s\}\)\]\+\\sum\_\{t=0\}^\{H\-1\}\\mathbb\{E\}\_\{\(\\tilde\{s\},a\)\\sim d\_\{t\}^\{D\}\}\\\!\\Big\[w\_\{t\}\(\\tilde\{s\},a\)\\,e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)\-\\beta\\,\\phi\(w\_\{t\}\(\\tilde\{s\},a\)\)\\Big\]\.\(19\)
Given\(𝝂,𝝁,𝐤\)\(\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},\\mathbf\{k\}\), the inner maximization in \([19](https://arxiv.org/html/2606.31178#A3.E19)\) decouples pointwise across\(t,s~,a\)\(t,\\tilde\{s\},a\):

maxw≥0⁡\{w⋅e𝝂,𝝁,t​\(s~,a\)−β​ϕ​\(w\)\}\.\\displaystyle\\max\_\{w\\geq 0\}\\;\\Big\\\{\\,w\\cdot e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)\-\\beta\\,\\phi\(w\)\\,\\Big\\\}\.\(20\)The first\-order conditione𝝂,𝝁,t​\(s~,a\)−β​ϕ′​\(w\)=0e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)\-\\beta\\,\\phi^\{\\prime\}\(w\)=0yields the closed\-form maximizer

wt∗​\(s~,a\)=max⁡\(0,\(ϕ′\)−1​\(e𝝂,𝝁,t​\(s~,a\)β\)\),\\displaystyle w\_\{t\}^\{\*\}\(\\tilde\{s\},a\)=\\max\\\!\\left\(0,\\;\(\\phi^\{\\prime\}\)^\{\-1\}\\\!\\left\(\\frac\{e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)\}\{\\beta\}\\right\)\\right\),\(21\)where themax⁡\(0,⋅\)\\max\(0,\\cdot\)enforces the non\-negativity constraint\.

#### Fenchel\-conjugate substitution\.

Define the non\-negative\-restricted Fenchel conjugate ofϕ\\phi,

ϕ\+∗​\(y\):=supw≥0\{w​y−ϕ​\(w\)\}\.\\displaystyle\\phi^\{\*\}\_\{\+\}\(y\):=\\sup\_\{w\\geq 0\}\\,\\big\\\{\\,wy\-\\phi\(w\)\\,\\big\\\}\.\(22\)By a direct change of variables,

supw≥0\{w​z−β​ϕ​\(w\)\}=β​supw≥0\{w​zβ−ϕ​\(w\)\}=β​ϕ\+∗​\(zβ\)\.\\displaystyle\\sup\_\{w\\geq 0\}\\,\\big\\\{\\,wz\-\\beta\\phi\(w\)\\,\\big\\\}=\\beta\\,\\sup\_\{w\\geq 0\}\\,\\Big\\\{\\,w\\,\\tfrac\{z\}\{\\beta\}\-\\phi\(w\)\\,\\Big\\\}=\\beta\\,\\phi^\{\*\}\_\{\+\}\\\!\\left\(\\tfrac\{z\}\{\\beta\}\\right\)\.\(23\)Substituting \([23](https://arxiv.org/html/2606.31178#A3.E23)\) withz=e𝝂,𝝁,t​\(s~,a\)z=e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)into \([19](https://arxiv.org/html/2606.31178#A3.E19)\) eliminates the inner maximization overwtw\_\{t\}:

max𝐝≥0⁡ℒ=G​\(𝐤\)−𝝁⊤​𝐤\+𝔼p~0​\[ν0​\(s~\)\]\+∑t=0H−1𝔼dtD​\[β​ϕ\+∗​\(e𝝂,𝝁,t​\(s~,a\)β\)\]\.\\displaystyle\\max\_\{\\mathbf\{d\}\\geq 0\}\\;\\mathcal\{L\}=G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\\!\\top\}\\mathbf\{k\}\+\\mathbb\{E\}\_\{\\tilde\{p\}\_\{0\}\}\[\\nu\_\{0\}\(\\tilde\{s\}\)\]\+\\sum\_\{t=0\}^\{H\-1\}\\mathbb\{E\}\_\{d\_\{t\}^\{D\}\}\\\!\\left\[\\beta\\,\\phi^\{\*\}\_\{\+\}\\\!\\left\(\\frac\{e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)\}\{\\beta\}\\right\)\\right\]\.\(24\)The remaining𝐤\\mathbf\{k\}\-dependent part of the Lagrangian isG​\(𝐤\)−𝝁⊤​𝐤G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\\!\\top\}\\mathbf\{k\}, maximized unconstrained over𝐤∈ℝn\\mathbf\{k\}\\in\\mathbb\{R\}^\{n\}\. Following the convention used elsewhere in the paper, define the \(concave\) conjugate ofGGasG∗​\(𝐲\):=sup𝐱\{𝐱⊤​𝐲\+G​\(𝐱\)\}G^\{\*\}\(\\mathbf\{y\}\):=\\sup\_\{\\mathbf\{x\}\}\\,\\big\\\{\\,\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{y\}\+G\(\\mathbf\{x\}\)\\,\\big\\\}\. Then

max𝐤∈ℝn⁡\{G​\(𝐤\)−𝝁⊤​𝐤\}=max𝐤∈ℝn⁡\{𝐤⊤​\(−𝝁\)\+G​\(𝐤\)\}=G∗​\(−𝝁\)\.\\displaystyle\\max\_\{\\mathbf\{k\}\\in\\mathbb\{R\}^\{n\}\}\\,\\big\\\{\\,G\(\\mathbf\{k\}\)\-\\boldsymbol\{\\mu\}^\{\\\!\\top\}\\mathbf\{k\}\\,\\big\\\}=\\max\_\{\\mathbf\{k\}\\in\\mathbb\{R\}^\{n\}\}\\,\\big\\\{\\,\\mathbf\{k\}^\{\\\!\\top\}\(\-\\boldsymbol\{\\mu\}\)\+G\(\\mathbf\{k\}\)\\,\\big\\\}=G^\{\*\}\(\-\\boldsymbol\{\\mu\}\)\.\(25\)
Combining the two conjugate formulations reduces the dual problem to a sample\-based loss function given by,

min𝝂,𝝁⁡ℒAET​\(𝝂,𝝁\):=𝔼p~0​\[ν0​\(s~\)\]\+∑t=0H−1𝔼\(s~,a\)∼dtD​\[β​ϕ\+∗​\(e𝝂,𝝁,t​\(s~,a\)β\)\]\+G∗​\(−𝝁\)\.\\displaystyle\\min\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\}\\;\\mathcal\{L\}\_\{\\mathrm\{AET\}\}\(\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\}\):=\\mathbb\{E\}\_\{\\tilde\{p\}\_\{0\}\}\[\\nu\_\{0\}\(\\tilde\{s\}\)\]\+\\sum\_\{t=0\}^\{H\-1\}\\mathbb\{E\}\_\{\(\\tilde\{s\},a\)\\sim d\_\{t\}^\{D\}\}\\\!\\left\[\\beta\\,\\phi^\{\*\}\_\{\+\}\\\!\\left\(\\frac\{e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)\}\{\\beta\}\\right\)\\right\]\+G^\{\*\}\(\-\\boldsymbol\{\\mu\}\)\.\(26\)Crucially,ℒAET\\mathcal\{L\}\_\{\\mathrm\{AET\}\}is expressed entirely as expectations overp~0\\tilde\{p\}\_\{0\}and the offline dataset\{dtD\}t=0H−1\\\{d\_\{t\}^\{D\}\\\}\_\{t=0\}^\{H\-1\}\. It does not require knowledge of the augmented transition kernelP~\\tilde\{P\}\(theP~\\tilde\{P\}\-expectation insidee𝝂,𝝁,te\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}is replaced by single\-sample plug\-inνt\+1​\(s~t\+1\)\\nu\_\{t\+1\}\(\\tilde\{s\}\_\{t\+1\}\)at offline transitions\(s~t,at,s~t\+1\)\(\\tilde\{s\}\_\{t\},a\_\{t\},\\tilde\{s\}\_\{t\+1\}\)\), nor does it require explicit access to the primal visitation distributions\{dt\}\\\{d\_\{t\}\\\}\. It can therefore be optimized using standard stochastic gradient descent on neural\-network parametrizationsνθ,t\\nu\_\{\\theta,t\}and𝝁\\boldsymbol\{\\mu\}\.

### C\.4Policy Extraction

After minimizing \([26](https://arxiv.org/html/2606.31178#A3.E26)\) to obtain\(𝝂∗,𝝁∗\)\(\\boldsymbol\{\\nu\}^\{\*\},\\boldsymbol\{\\mu\}^\{\*\}\), the optimal visitation ratiowt∗w\_\{t\}^\{\*\}is recovered from \([21](https://arxiv.org/html/2606.31178#A3.E21)\) usinge𝝂∗,𝝁∗,te\_\{\\boldsymbol\{\\nu\}^\{\*\},\\boldsymbol\{\\mu\}^\{\*\},t\}\. The optimal augmented\-state policyπ~t∗​\(a∣s~\)\\tilde\{\\pi\}^\{\*\}\_\{t\}\(a\\mid\\tilde\{s\}\)inducing the optimal visitationdt∗=wt∗​dtDd\_\{t\}^\{\*\}=w\_\{t\}^\{\*\}\\,d\_\{t\}^\{D\}is then extracted\. In the tabular case, this reduces to direct normalization:π~t∗​\(a∣s~\)=wt∗​\(s~,a\)​dtD​\(s~,a\)∑a′wt∗​\(s~,a′\)​dtD​\(s~,a′\)\\tilde\{\\pi\}^\{\*\}\_\{t\}\(a\\mid\\tilde\{s\}\)=\\frac\{w\_\{t\}^\{\*\}\(\\tilde\{s\},a\)\\,d\_\{t\}^\{D\}\(\\tilde\{s\},a\)\}\{\\sum\_\{a^\{\\prime\}\}w\_\{t\}^\{\*\}\(\\tilde\{s\},a^\{\\prime\}\)\\,d\_\{t\}^\{D\}\(\\tilde\{s\},a^\{\\prime\}\)\}\. In continuous case, we use weighted behavior cloning following\[[12](https://arxiv.org/html/2606.31178#bib.bib10)\]:

maxπ~⁡𝔼\(s~,a\)∼dtD​\[wt∗​\(s~,a\)​log⁡π~​\(a∣s~\)\]\.\\displaystyle\\max\_\{\\tilde\{\\pi\}\}\\;\\mathbb\{E\}\_\{\(\\tilde\{s\},a\)\\sim d\_\{t\}^\{D\}\}\\\!\\left\[w\_\{t\}^\{\*\}\(\\tilde\{s\},a\)\\,\\log\\tilde\{\\pi\}\(a\\mid\\tilde\{s\}\)\\right\]\.\(27\)

### C\.5Choice ofϕ\\phi\-Divergence and AggregationGG

We use theχ2\\chi^\{2\}\-divergence forϕ\\phiand a piecewise\-log function for the concave aggregationGG\.

#### χ2\\chi^\{2\}divergence\.

ϕ​\(x\)=12​\(x−1\)2\\phi\(x\)=\\tfrac\{1\}\{2\}\(x\-1\)^\{2\}forx≥0x\\geq 0\. Its convex conjugate is

ϕ\+∗​\(y\)=12​\[\[y\+1\]\+\]2−12,wt∗​\(s~,a\)=\[e𝝂,𝝁,t​\(s~,a\)/β\+1\]\+,\\displaystyle\\phi^\{\*\}\_\{\+\}\(y\)\\;=\\;\\tfrac\{1\}\{2\}\\bigl\[\\,\[y\+1\]\_\{\+\}\\,\\bigr\]^\{2\}\\;\-\\;\\tfrac\{1\}\{2\},\\qquad w\_\{t\}^\{\*\}\(\\tilde\{s\},a\)\\;=\\;\\bigl\[\\,e\_\{\\boldsymbol\{\\nu\},\\boldsymbol\{\\mu\},t\}\(\\tilde\{s\},a\)/\\beta\+1\\,\\bigr\]\_\{\+\},\(28\)where\[⋅\]\+:=max⁡\(0,⋅\)\[\\,\\cdot\\,\]\_\{\+\}:=\\max\(0,\\cdot\)\.

#### Piecewise\-log aggregation\.

In practice, we avoid usingG​\(𝐱\)=∑ilog⁡\(xi\)G\(\\mathbf\{x\}\)=\\sum\_\{i\}\\log\(x\_\{i\}\)directly, as logarithm is undefined for negative values that may arise during optimization\. Instead, we use a piecewise extensionG​\(𝐱\)=∑i=1dg​\(xi\)G\(\\mathbf\{x\}\)=\\sum\_\{i=1\}^\{d\}g\(x\_\{i\}\)with

g​\(x\)=\{log⁡\(x\),x≥1,−12​\(x−2\)2\+12,x<1\.\\displaystyle g\(x\)\\;=\\;\\begin\{cases\}\\log\(x\),&x\\geq 1,\\\\ \-\\tfrac\{1\}\{2\}\(x\-2\)^\{2\}\+\\tfrac\{1\}\{2\},&x<1\.\\end\{cases\}\(29\)which is concave, strictly increasing, and well\-defined on all ofℝ\\mathbb\{R\}\. WithG∗​\(𝒑\):=sup𝐱\{𝒑⊤​𝐱\+G​\(𝐱\)\}G^\{\*\}\(\\boldsymbol\{p\}\):=\\sup\_\{\\mathbf\{x\}\}\\\{\\boldsymbol\{p\}^\{\\\!\\top\}\\mathbf\{x\}\+G\(\\mathbf\{x\}\)\\\}, its conjugate isG∗​\(−𝝁\)=∑i=1dg∗​\(−μi\)G^\{\*\}\(\-\\boldsymbol\{\\mu\}\)=\\sum\_\{i=1\}^\{d\}g^\{\*\}\(\-\\mu\_\{i\}\)with

g∗​\(−μ\)=\{−1−log⁡\(μ\),0<μ<1,12​μ2−2​μ\+12,μ≥1\.\\displaystyle g^\{\*\}\(\-\\mu\)\\;=\\;\\begin\{cases\}\-1\-\\log\(\\mu\),&0<\\mu<1,\\\\ \\tfrac\{1\}\{2\}\\mu^\{2\}\-2\\mu\+\\tfrac\{1\}\{2\},&\\mu\\geq 1\.\\end\{cases\}\(30\)

### C\.6Full Algorithm of AETDICE

We present the full algorithm of AETDICE in Algorithm[1](https://arxiv.org/html/2606.31178#alg1)\. A key implementation consideration is that the Lagrangian multipliersνt​\(s~\)\\nu\_\{t\}\(\\tilde\{s\}\)are defined per timestep, and the neural network must be able to represent this time dependence\. In standard offline RL datasets, trajectories are stored with timestep information, but each sample’s contribution to the loss does not depend on when it occurs within the episode\. This allows data augmentation along the time axis, which we describe in the following section\.

Algorithm 1AETDICE: Offline AET Optimization via Distribution Correction Estimation1:Offline dataset

𝒟=\{\(st,at,𝐫t,st\+1,t\)\}\\mathcal\{D\}=\\\{\(s\_\{t\},a\_\{t\},\\mathbf\{r\}\_\{t\},s\_\{t\+1\},t\)\\\}from the original MOMDP

ℳ\\mathcal\{M\}; transformation

F:ℝm→ℝnF:\\mathbb\{R\}^\{m\}\\\!\\to\\\!\\mathbb\{R\}^\{n\}; aggregation

G:ℝn→ℝG:\\mathbb\{R\}^\{n\}\\\!\\to\\\!\\mathbb\{R\}; dual network

νψ\\nu\_\{\\psi\}, policy network

π~θ\\tilde\{\\pi\}\_\{\\theta\}; divergence temperature

β\>0\\beta\>0; learning rate

η\\eta
2:Augmented\-state policy

π~θ\\tilde\{\\pi\}\_\{\\theta\}approximating the AET\-optimal policy

3:// Stage 1: Dataset preprocessing \(one\-time\)

4:Construct augmented dataset

𝒟~\\widetilde\{\\mathcal\{D\}\}: for each trajectory in

𝒟\\mathcal\{D\}, compute accumulated rewards

𝐑tacc=∑k=0t−1𝐫​\(sk,ak\)\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}=\\sum\_\{k=0\}^\{t\-1\}\\mathbf\{r\}\(s\_\{k\},a\_\{k\}\)and form augmented transitions

\(s~t,at,𝐫~t,s~t\+1\)\(\\tilde\{s\}\_\{t\},a\_\{t\},\\tilde\{\\mathbf\{r\}\}\_\{t\},\\tilde\{s\}\_\{t\+1\}\)with

s~t=\(st,𝐑tacc,t\)\\tilde\{s\}\_\{t\}=\(s\_\{t\},\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\},t\)
5:Compute the transformed reward for each transition:

𝐫~t=F​\(𝐑tacc\+𝐫​\(st,at\)\)−F​\(𝐑tacc\)∈ℝn\\tilde\{\\mathbf\{r\}\}\_\{t\}\\;=\\;F\\\!\\left\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\+\\mathbf\{r\}\(s\_\{t\},a\_\{t\}\)\\right\)\-F\\\!\\left\(\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}\\right\)\\;\\in\\;\\mathbb\{R\}^\{n\}
6:// Stage 2: Initialization

7:Initialize

ψ\\psi,

θ\\theta
8:if

GGis nonlinearthen

9:Initialize

𝝁∈ℝn\\boldsymbol\{\\mu\}\\in\\mathbb\{R\}^\{n\}
10:else

11:Fix

𝝁=𝟏\\boldsymbol\{\\mu\}=\\mathbf\{1\}or preference weights of a choice \(no slack constraint needed when

GGis linear\)

12:endif

13:// Stage 3: Joint dual / policy optimization

14:whilenot convergeddo

15:Sample minibatch

ℬ=\{\(s~t,at,𝐫~t,s~t\+1\)\}\\mathcal\{B\}=\\\{\(\\tilde\{s\}\_\{t\},a\_\{t\},\\tilde\{\\mathbf\{r\}\}\_\{t\},\\tilde\{s\}\_\{t\+1\}\)\\\}from

𝒟~\\widetilde\{\\mathcal\{D\}\}and initial states

s~0∼p~0\\tilde\{s\}\_\{0\}\\sim\\tilde\{p\}\_\{0\}
16:Compute empirical slack\-augmented advantage at each transition:

e^𝝁,ψ,t​\(s~t,at\)=𝝁⊤​𝐫~t\+νψ​\(s~t\+1\)−νψ​\(s~t\)\\hat\{e\}\_\{\\boldsymbol\{\\mu\},\\psi,t\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\\;=\\;\\boldsymbol\{\\mu\}^\{\\\!\\top\}\\tilde\{\\mathbf\{r\}\}\_\{t\}\\;\+\\;\\nu\_\{\\psi\}\(\\tilde\{s\}\_\{t\+1\}\)\\;\-\\;\\nu\_\{\\psi\}\(\\tilde\{s\}\_\{t\}\)
17:\(single\-sample plug\-in for

𝔼s~′∼P~​\[νt\+1​\(s~′\)\]\\mathbb\{E\}\_\{\\tilde\{s\}^\{\\prime\}\\sim\\tilde\{P\}\}\[\\nu\_\{t\+1\}\(\\tilde\{s\}^\{\\prime\}\)\]; boundary

νψ​\(s~H\):=0\\nu\_\{\\psi\}\(\\tilde\{s\}\_\{H\}\):=0at terminal transitions\)

18:Update dual parameters

ψ\\psi\(and

𝝁\\boldsymbol\{\\mu\}if

GGis nonlinear\) by minimizing the empirical AET dual loss:

ℒ^AET​\(ψ,𝝁\)=𝔼s~0∼p~0​\[νψ​\(s~0\)\]\+𝔼\(s~t,at,s~t\+1\)∼𝒟~​\[β​ϕ\+∗​\(e^𝝁,ψ,t​\(s~t,at\)β\)\]\+G∗​\(−𝝁\)\\widehat\{\\mathcal\{L\}\}\_\{\\mathrm\{AET\}\}\(\\psi,\\boldsymbol\{\\mu\}\)\\;=\\;\\mathbb\{E\}\_\{\\tilde\{s\}\_\{0\}\\sim\\tilde\{p\}\_\{0\}\}\\\!\\big\[\\nu\_\{\\psi\}\(\\tilde\{s\}\_\{0\}\)\\big\]\\;\+\\;\\mathbb\{E\}\_\{\(\\tilde\{s\}\_\{t\},a\_\{t\},\\tilde\{s\}\_\{t\+1\}\)\\sim\\widetilde\{\\mathcal\{D\}\}\}\\\!\\left\[\\beta\\,\\phi^\{\*\}\_\{\+\}\\\!\\\!\\left\(\\frac\{\\hat\{e\}\_\{\\boldsymbol\{\\mu\},\\psi,t\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\}\{\\beta\}\\right\)\\right\]\\;\+\\;G^\{\*\}\(\-\\boldsymbol\{\\mu\}\)
19:via

ψ←ψ−η​∇ψℒ^AET\\psi\\leftarrow\\psi\-\\eta\\nabla\_\{\\psi\}\\widehat\{\\mathcal\{L\}\}\_\{\\mathrm\{AET\}\}and

𝝁←𝝁−η​∇𝝁ℒ^AET\\boldsymbol\{\\mu\}\\leftarrow\\boldsymbol\{\\mu\}\-\\eta\\nabla\_\{\\boldsymbol\{\\mu\}\}\\widehat\{\\mathcal\{L\}\}\_\{\\mathrm\{AET\}\}
20:Compute optimal visitation ratio at sampled transitions:

w∗​\(s~t,at\)=max⁡\(0,\(ϕ′\)−1​\(e^𝝁,ψ,t​\(s~t,at\)β\)\)w^\{\*\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\\;=\\;\\max\\\!\\left\(0,\\;\(\\phi^\{\\prime\}\)^\{\-1\}\\\!\\\!\\left\(\\frac\{\\hat\{e\}\_\{\\boldsymbol\{\\mu\},\\psi,t\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\}\{\\beta\}\\right\)\\right\)
21:Update policy parameters

θ\\thetavia weighted behavior cloning:

ℒθ=−𝔼\(s~t,at\)∼𝒟~​\[w∗​\(s~t,at\)​log⁡π~θ​\(at∣s~t\)\]\\mathcal\{L\}\_\{\\theta\}\\;=\\;\-\\,\\mathbb\{E\}\_\{\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\\sim\\widetilde\{\\mathcal\{D\}\}\}\\\!\\left\[w^\{\*\}\(\\tilde\{s\}\_\{t\},a\_\{t\}\)\\,\\log\\tilde\{\\pi\}\_\{\\theta\}\(a\_\{t\}\\mid\\tilde\{s\}\_\{t\}\)\\right\]
22:via

θ←θ−η​∇θℒθ\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\theta\}
23:endwhile

24:return

π~θ\\tilde\{\\pi\}\_\{\\theta\}

## Appendix DTime\-Index Augmentation in the Augmented MOMDP

We describe the data augmentation procedure used to support learning in the augmented MOMDP\. The goal is to enrich the dataset with additional time\-indexed contexts while preserving the original state\-action distribution\.

### D\.1Dataset and Augmentation Strategy

We begin with a dataset𝒟0\\mathcal\{D\}\_\{0\}collected by a behavior policyπb\\pi\_\{b\}, consisting of tuples\(st,at,𝐑tacc,t\)\(s\_\{t\},a\_\{t\},\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\},t\), where𝐑tacc=∑k=0t−1𝐫​\(sk,ak\)\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}=\\sum\_\{k=0\}^\{t\-1\}\\mathbf\{r\}\(s\_\{k\},a\_\{k\}\)is recorded during data collection\. No assumptions are made about the optimality ofπb\\pi\_\{b\}\. Augmentation is performed solely over the discrete time indext∈\{0,1,…,H\}t\\in\\\{0,1,\\dots,H\\\}; all other components of the augmented state—includingssand𝐑acc\\mathbf\{R\}^\{\\mathrm\{acc\}\}—are taken directly from𝒟0\\mathcal\{D\}\_\{0\}and are never resampled or synthetically modified\.

### D\.2Multi\-Head Lagrangian Multiplier

Augmentation is implemented through the parameterization of the Lagrangian multiplierν​\(s,𝐑acc,t\)\\nu\(s,\\mathbf\{R\}^\{\\mathrm\{acc\}\},t\)\. The neural network consists of a shared trunk processing\(s,𝐑acc\)\(s,\\mathbf\{R\}^\{\\mathrm\{acc\}\}\), followed by a multi\-head output indexed by time, producing\[ν​\(s,𝐑acc,0\),…,ν​\(s,𝐑acc,H\)\]\[\\nu\(s,\\mathbf\{R\}^\{\\mathrm\{acc\}\},0\),\\dots,\\nu\(s,\\mathbf\{R\}^\{\\mathrm\{acc\}\},H\)\]\. For each data point\(s,𝐑acc\)\(s,\\mathbf\{R\}^\{\\mathrm\{acc\}\}\)from𝒟0\\mathcal\{D\}\_\{0\}, the full set of time\-indexed outputs is available during training\. This exposes the dual objective to a richer collection of time\-conditioned contexts without altering the underlying state\-action distribution, and ensures that the multipliers are well\-trained across the entire episode horizon\.

## Appendix EEnvironment Details

![Refer to caption](https://arxiv.org/html/2606.31178v1/x8.png)\(a\)Fair\-Taxi
![Refer to caption](https://arxiv.org/html/2606.31178v1/x9.png)\(b\)MO\-PointMaze\-2obj
![Refer to caption](https://arxiv.org/html/2606.31178v1/x10.png)\(c\)MO\-PointMaze\-3obj

Figure 7:Visualization of the multi\-objective environments\. Each environment balances multiple objectives over a finite horizon \(H=50H=50for Fair\-Taxi andH=300H=300MO\-PointMaze\)\.### E\.1Finite Fair\-Taxi

The Finite Fair\-Taxi environment is adapted from the multi\-objective taxi domain of Fan et al\.\[[4](https://arxiv.org/html/2606.31178#bib.bib18)\]and formulated as a finite\-horizon grid\-world task\. A taxi operates on anN×NN\\times Ngrid and serves passengers associated with different origin–destination pairs, where each passenger group defines one reward dimension\. Because only one passenger can be transported at a time and episodes have a fixed horizon, the agent must allocate time across groups, introducing a fairness trade\-off between objectives\. All experiments use a two\-objective setting\.

#### State and Action Spaces\.

The state is discrete and encodes the taxi location\(x,y\)\(x,y\), a binary indicator of whether a passenger is onboard, and the index of the passenger group currently being served \(or a special value if none\)\. The action space consists of six discrete actions: move north, south, east, west, pick up, and drop off\. Movement actions are clipped at grid boundaries\. Pickup is valid only when the taxi is at an origin location and not already carrying a passenger, and drop\-off is valid only when the taxi is at the corresponding destination of the onboard passenger\.

#### Reward Function\.

Rewards are vector\-valued with dimensionm=2m=2\. At each time step, the reward vector𝐫t∈ℝ2\\mathbf\{r\}\_\{t\}\\in\\mathbb\{R\}^\{2\}is defined as follows: a successful delivery for groupiiyieldsrt\(i\)=30r\_\{t\}^\{\(i\)\}=30andrt\(j\)=0r\_\{t\}^\{\(j\)\}=0forj≠ij\\neq i; invalid pickup or drop\-off actions incurrt\(1\)=rt\(2\)=−10r\_\{t\}^\{\(1\)\}=r\_\{t\}^\{\(2\)\}=\-10; all other transitions produce zero reward\. To introduce asymmetry between objectives, the rewards and penalties for group 1 are scaled by a factor of two, resulting inrt\(1\)∈\{60,−20,0\}r\_\{t\}^\{\(1\)\}\\in\\\{60,\-20,0\\\}while group 2 retains\{30,−10,0\}\\\{30,\-10,0\\\}\. Rewards are therefore sparse and event\-driven\.

#### Episode Termination and Layout\.

Episodes terminate after a fixed horizon of 50 time steps\. The two origin–destination pairs are placed in distant regions of the grid, with each origin uniquely paired to a destination\. Serving one group typically requires traveling in a direction that does not overlap with the other, so dedicating time to one group reduces the opportunity to serve the other\. This creates a temporal trade\-off between objectives within each episode\.

#### Offline Dataset Construction

For the Fair\-Taxi domain, offline datasets are generated using a mixture of single\-objective policies following the strategy\. Each policy is a tabular Q\-function optimized for a different linear scalarization of the reward vector\. During data collection, policies are alternated within each episode according to fixed time intervals, producing trajectories that contain segments optimized for different objectives\. This switching scheme encourages diverse return profiles across episodes and prevents the dataset from being dominated by trajectories specialized to a single objective\. Episodes start from a fixed initial taxi location to ensure comparability across policies\. Each episode runs for at most 50 steps\. The dataset stores state, action, next state, reward vector, timestep index, terminal flag, and accumulated return vector at each step\.

### E\.2MO\-PointMaze

The MO\-PointMaze environment is a continuous\-state multi\-objective navigation task adapted from the D4RL maze domains\[[5](https://arxiv.org/html/2606.31178#bib.bib25)\]\. A point\-mass agent moves in a 2D maze containing multiple spatial goal regions, where each goal corresponds to one reward dimension\. Because the goals are spatially separated by walls and corridors, the agent must allocate time across regions, inducing a trade\-off between objectives within each episode\.

#### State and Action Spaces\.

The observation space is inherited from the underlying D4RL PointMaze environment and includes the agent’s 2D position and velocity\. Experiments use a discretized action space consisting of movements along the four cardinal directions with several fixed magnitudes\. This discretization stabilizes training and ensures consistent coverage during offline data collection\. The episode horizon is set to 300 steps\.

#### Maze Layouts\.

Two layouts are used\. In the two\-objective setting[7\(b\)](https://arxiv.org/html/2606.31178#A5.F7.sf2), the agent starts bottom left region of a compact maze with two goal regions located in different corridors\. In the three\-objective setting[7\(c\)](https://arxiv.org/html/2606.31178#A5.F7.sf3), the maze contains three goals positioned in distinct regions\. Reaching one goal generally requires moving away from the others, making simultaneous proximity infeasible\.

#### Reward Function\.

Rewards are vector\-valued with dimension equal to the number of goalsmm\. At each time step, the reward for objectiveiidepends on the Euclidean distance between the agent’s position𝐬t\\mathbf\{s\}\_\{t\}and theii\-th goal location𝐠i\\mathbf\{g\}\_\{i\}:

rt\(i\)=exp⁡\(−∥𝐬t−𝐠i∥d0\),r\_\{t\}^\{\(i\)\}=\\exp\\\!\\left\(\-\\frac\{\\lVert\\mathbf\{s\}\_\{t\}\-\\mathbf\{g\}\_\{i\}\\rVert\}\{d\_\{0\}\}\\right\),whered0d\_\{0\}is a scaling constant proportional to the maze cell size\. Rewards decay rapidly with distance, so meaningful reward in one dimension requires the agent to remain near the corresponding goal region for a sustained period\.

In the two\-objective setting, the first reward dimension is scaled by a factor of two to introduce asymmetry between objectives\. In the three\-objective setting, all reward dimensions are equally scaled\.

#### Multi\-Objective Trade\-off\.

Due to the rapid spatial decay of rewards and the maze topology, the agent cannot remain close to multiple goals at the same time\. Policies must therefore schedule visits to different regions within an episode, creating a temporal trade\-off between objectives\. This makes the environment well suited for analyzing differences between trajectory\-level and expectation\-level scalarization\.

![Refer to caption](https://arxiv.org/html/2606.31178v1/x11.png)Figure 8:Visualization of the offline dataset return distributions for the MO\-PointMaze environment\.\(Left\)The cumulative returns in the 2\-objective setting, illustrating the trade\-off between the two goals\.\(Right\)The spatial distribution of returns in the 3\-objective setting\.
#### Offline Dataset Collection\.

Offline datasets are generated using heuristic navigation policies that move toward specific goal regions following shortest\-path routes\. By varying goal targets and switching patterns over time, the dataset covers diverse behaviors, including specialization on a single objective and sequential visitation of multiple goals\. This diversity ensures that the dataset contains trajectories with a wide range of return trade\-offs, which is essential for evaluating multi\-objective offline RL methods\. The resulting return distributions are visualized in Figure[8](https://arxiv.org/html/2606.31178#A5.F8)\.

### E\.3MO\-MuJoCo \(D4MORL\)

We use the multi\-objective MuJoCo benchmark from D4MORL\[[31](https://arxiv.org/html/2606.31178#bib.bib3)\], which extends the standard MuJoCo locomotion suite by replacing scalar rewards with vector\-valued multi\-objective rewards while leaving the underlying physics, observation space, and action space unchanged\. Five environments are used: MO\-Ant, MO\-Walker2d, MO\-HalfCheetah\. For all environments we use the*expert\-uniform*and*amateur\-uniform*dataset splits provided by\[[31](https://arxiv.org/html/2606.31178#bib.bib3)\], in which the behaviour\-policy preferences are sampled uniformly from the 2\-simplex during data collection; each split contains 50,000 transitions per env\.

#### State and Action Spaces\.

States and continuous actions are inherited from the underlying MuJoCo environments\. State and action dimensions are Ant 27/8, HalfCheetah 17/6, Walker2d 17/6, Hopper 11/3, Swimmer 8/2 — verify\. We follow the standard MuJoCo convention of clipping actions to\[−1,1\]\[\-1,1\]before computing the reward\.

#### Reward Function\.

At each time step the reward is a vector𝐫t∈ℝ2\\mathbf\{r\}\_\{t\}\\in\\mathbb\{R\}^\{2\}whose two components capture distinct physical aspects of the task: forward and lateral velocity in MO\-Ant, forward speed and energy saving in MO\-HalfCheetah and MO\-Walker2d, forward speed and energy efficiency in MO\-Swimmer, and running and jumping in MO\-Hopper\. Both components include a shared survival bonus and a shared action\-norm penalty, so the two objectives are positively oriented \(higher is better\) and partially correlated\. We refer to\[[31](https://arxiv.org/html/2606.31178#bib.bib3)\]for exact formulae\. To study the effect of asymmetry between objectives, the per\-objective rewards are min–max normalized at dataset load time:

r~t\(i\)=ci⋅rt\(i\)−rmin\(i\)rmax\(i\)−rmin\(i\),\\tilde\{r\}\_\{t\}^\{\(i\)\}\\;=\\;c\_\{i\}\\cdot\\frac\{r\_\{t\}^\{\(i\)\}\-r\_\{\\min\}^\{\(i\)\}\}\{r\_\{\\max\}^\{\(i\)\}\-r\_\{\\min\}^\{\(i\)\}\},wherermin\(i\),rmax\(i\)r\_\{\\min\}^\{\(i\)\},r\_\{\\max\}^\{\(i\)\}are computed over the offline dataset andci∈ℝ\+c\_\{i\}\\in\\mathbb\{R\}\_\{\+\}is a per\-objective scaling factor\. We use the symmetric default𝐜=\(1,1\)\\mathbf\{c\}=\(1,1\)for all environments and additionally𝐜=\(2,1\)\\mathbf\{c\}=\(2,1\)in MO\-Ant for an objective\-1\-favouring variant\.

#### Episode Termination and Horizon\.

All environments use a fixed horizon ofH=500H=500steps\. Standard MuJoCo early\-termination conditions apply \(height/angle bounds in MO\-Walker2d and MO\-Hopper, instability bounds in MO\-Ant\); transitions following early termination contribute zero reward and are masked out by the buffer\.

#### Cobb\-Douglas Reward Reshaping\.

Two of our experiments \(G\-linear withFF\-Cobb\-Douglas, and G\-log with combinedFF\-Cobb\-Douglas andFF\-threshold\) require the reward vector to be split into a clean \(return, cost\) pair\. The relevant D4MORL environments admit such a split because their second objective takes the saving formconst−‖𝐚‖2\+alive\\text\{const\}\-\\\|\\mathbf\{a\}\\\|^\{2\}\+\\text\{alive\}\. For MO\-HalfCheetah, we recover the action\-norm cost asrt\(1\)←5−rt\(1\)r\_\{t\}^\{\(1\)\}\\leftarrow 5\-r\_\{t\}^\{\(1\)\}, yielding a\(forward speed,‖𝐚‖2\)\(\\text\{forward speed\},\\;\\\|\\mathbf\{a\}\\\|^\{2\}\)layout\. For MO\-Walker2d we use a 3\-dimensional layout\(speed,cost,safety\)\(\\text\{speed\},\\,\\text\{cost\},\\,\\text\{safety\}\), where speed and cost match the MO\-HalfCheetah reshape and the safety channel is a quadratic balance rewardrtsafety=\(1−2​hnorm2\)\+\(1−2​anorm2\)r\_\{t\}^\{\\text\{safety\}\}=\(1\-2h\_\{\\text\{norm\}\}^\{2\}\)\+\(1\-2a\_\{\\text\{norm\}\}^\{2\}\)computed from the height and torso\-angle observations normalized to the env’s safety bounds \(sortsafety∈\[−2,\+2\]r\_\{t\}^\{\\text\{safety\}\}\\in\[\-2,\+2\]in\-bounds, peaking at the upright centre\)\. The Cobb\-Douglas utility consumes \(speed, cost\), and the safety channel feeds a separate piecewise\-log threshold; the two combine through the outer aggregationGG\.

## Appendix FAdditional Experiment Results

### F\.1Fair\-Taxi

We report quantitative performance on the Finite Fair\-Taxi environment under ESR, BSR \(α=0\.5\\alpha=0\.5\), and SER objectives\. All values are log\-scaled utilities \(mean±\\pmstandard deviation over three seeds\), so numerical differences reflect multiplicative gaps in the underlying objective values\. Each policy is trained using the same AET framework with a different nonlinear objective and evaluated under all three criteria; entry\(i,j\)\(i,j\)in Table[2](https://arxiv.org/html/2606.31178#A6.T2)reports the performance of a policy trained for objectiveiiwhen evaluated using objectivejj\.

Across all objectives, policies achieve their highest performance when evaluated under the same criterion used during training, indicating that ESR, BSR, and SER correspond to distinct optimization targets in practice\. Policies trained under BSR exhibit intermediate behavior: while optimized for BSR, they retain comparatively higher performance under ESR and SER evaluations than policies trained under the extreme objectives\. Overall, these results demonstrate that nonlinear objectives are not interchangeable, and that objective mismatch can lead to substantial degradation even when numerical differences appear small due to log scaling\.

Table 2:Performance of offline Fair MORL in Fair\-Taxi environment\. Rows correspond to the nonlinear objective used for training, and columns correspond to the evaluation objective\. All values are log\-scaled utilities \(mean±\\pmstandard deviation over 3 random seeds\)\.
### F\.2MO\-PointMaze

We next examine how these behavioral differences translate into return trade\-offs in a continuous\-state setting\. Figure[9](https://arxiv.org/html/2606.31178#A6.F9)shows the achieved return vectors\. In the two\-objective case, ESR produces balanced trajectories along the diagonal trade\-off, whereas SER concentrates near extreme regions that prioritize a single objective per episode\. BSR \(α=0\.5\\alpha=0\.5\) spans intermediate trade\-offs\. The same pattern is more pronounced in the three\-objective setting, where SER lies near the corners, ESR occupies the balanced interior, and BSR fills the space between\. ESR\-IQL aligns with ESR\-style trade\-offs, while AETDICE covers the full spectrum\.

Table[3](https://arxiv.org/html/2606.31178#A6.T3)reports performance under multiple evaluation criteria \(ESR, BSR, SER, and LSR\), reflecting different preference structures\. The AETDICE variants achieve strong performance on their respective target objectives, demonstrating that the proposed framework can effectively optimize a range of nonlinear criteria within a unified approach\. In the three\-objective setting, ESR\-IQL attains the highest ESR score, consistent with its specialization\. Linear scalarization achieves high linear scalarized return \(LSR\), as expected, but yields lower performance under nonlinear utilities, often producing policies that prioritize a single objective rather than balancing across objectives\.

Table 3:Comparison of ESR, BSR, SER, and LSR \(linear scalarization return with coefficients of 1\) scores in MO\-PointMaze\. For each method, performance is reported with respect to the objective it directly optimizes \(mean±\\pmstandard deviation over 3 random seeds\)\.![Refer to caption](https://arxiv.org/html/2606.31178v1/x12.png)

Figure 9:Performance results in the MO\-PointMaze\-2obj \(left\) and MO\-PointMaze\-3obj \(right\) environment\.
### F\.3D4MORL

#### Environments and reward modifications\.

We run three experiment families on D4MORL\. The first is experiment on the performance of various AETDICE objectives on MO\-Ant and MO\-Walker2d with the original D4MORL rewards and datasets\. The second is the Cobb–Douglas \(CD\) experiment on MO\-Walker2d and MO\-HalfCheetah, where we reshape the reward into a \(return, cost\) layout because the CD utility requires a monotone\-increasing reward in dimension 0 and a monotone\-increasing cost in dimension 1\. The third is the safety utility aggregated CD experiment on MO\-Walker2d, which adds a safety reward to prevent a “drop\-and\-die” exploit\.

CD reshape \(MO\-Walker2d, MO\-HalfCheetah\)\.Both environments expose a forward\-speed rewardrv=vx\+raliver^\{v\}=v\_\{x\}\+r^\{\\text\{alive\}\}and an energy\-saving rewardre=4−‖a‖2\+raliver^\{e\}=4\-\\\|a\\\|^\{2\}\+r^\{\\text\{alive\}\}\. We keep the forward speed unchanged and recover the action\-energy cost fromrer^\{e\}:

R0=rv,R1=5−re=‖a‖2\.R\_\{0\}\\;=\\;r^\{v\},\\qquad R\_\{1\}\\;=\\;5\-r^\{e\}\\;=\\;\\\|a\\\|^\{2\}\.
Safety utility aggregated CD \(MO\-Walker2d\)\.Walker2d ends an episode when the torso heighthhleaves\[0\.8,2\.0\]\[0\.8,2\.0\]or the torso angleθ\\thetaleaves\[−1,1\]\[\-1,1\]\. A CD agent at lowρ\\rhocan game this by ending the episode early to shrink the cost denominator\. To prevent this we add a third reward that scores the posture or location of the upper body of walker2d agent inside the safe zone\. Normalizinghhandθ\\thetaso the safe zone maps to\[−1,1\]\[\-1,1\]viah¯=\(h−1\.4\)/0\.6\\bar\{h\}=\(h\-1\.4\)/0\.6andθ¯=θ\\bar\{\\theta\}=\\theta, we define

R2=\(−2​h¯2\+1\)\+\(−2​θ¯2\+1\)\.R\_\{2\}\\;=\\;\\bigl\(\-2\\,\\bar\{h\}^\{2\}\+1\\bigr\)\\;\+\\;\\bigl\(\-2\\,\\bar\{\\theta\}^\{2\}\+1\\bigr\)\.Per step,R2∈\[−2,2\]R\_\{2\}\\in\[\-2,2\]inside the safe zone, peaking at\+2\+2at the upright center and dropping to−2\-2at the env bounds\.R2R\_\{2\}feeds the threshold branch of theFF\-utility whileR0R\_\{0\}andR1R\_\{1\}feed the Cobb–Douglas branch, so the safety floor is decoupled from the speed/cost trade\-off\.

Table 4:Results of MO\-Ant Expert with Concave G and Convex F\. \(mean ± standard deviation over 5 random seeds\)Table 5:Results of MO\-Ant Amateur with Concave G and Convex F\. \(mean ± standard deviation over 5 random seeds\)Table 6:Results of MO\-Walker2d Expert with Concave G and Convex F\. \(mean ± standard deviation over 5 random seeds\)Table 7:Results of MO\-Walker2d Amateur with Concave G and Convex F\. \(mean ± standard deviation over 5 random seeds\)Table 8:Results of Walker2d Cobb\-Douglas objectives\.Table 9:Results of HalfCheetah Cobb\-Douglas objectives\.

## Appendix GAblation Study: Impact of Regularization Strength

![Refer to caption](https://arxiv.org/html/2606.31178v1/x13.png)Figure 10:Performance of AETDICE with varyingβ\\betavalues on MO\-PointMaze\-2obj dataset\.We study how the regularization coefficientβ\\beta, which controls the strength of theϕ\\phi\-divergence penalty in AETDICE, affects performance under different AET objectives\. Figure[10](https://arxiv.org/html/2606.31178#A7.F10)reports evaluation scores under ESR, BSR, and SER criteria asβ\\betavaries on MO\-PointMaze\-2obj\. The effective range ofβ\\betain our experiments is considerably smaller than values commonly reported for DICE\-style methods in single\-objective settings\. This is expected: offline MORL datasets are collected from policies trained under diverse preference weights, so trajectories optimal for one trade\-off can be highly suboptimal for another AET objective\. Smaller regularization is needed to permit the larger distribution shift required for finding the target objective’s optimum within such heterogeneous data\. This makes achieving strong offline performance more demanding, as the optimizer must deviate further from the data distribution\. Nevertheless, with sufficiently smallβ\\beta, AETDICE achieves strong performance across all objectives, and performance varies smoothly asβ\\betachanges, indicating stable optimization\. These results demonstrate that AETDICE enables effective optimization of AET objectives—including those previously intractable—given appropriate regularization\.

## Appendix HHyperparameter

Table[10](https://arxiv.org/html/2606.31178#A8.T10)summarizes the default hyperparameters used for AETDICE\. The policyπθ\\pi\_\{\\theta\}, criticνψ\\nu\_\{\\psi\}, and preference parametersμ\\muare all optimized using Adam with a learning rate of3×10−43\\times 10^\{\-4\}and a batch size of 256\. Both the policy and critic are implemented as three\-layer multilayer perceptrons with hidden dimension 768, except in MO\-PointMaze\-3obj where we increase the width to 1024 to accommodate the higher\-dimensional return structure\. Training is performed for10610^\{6\}gradient steps\.

We use the Pearsonχ2\\chi^\{2\}divergence for occupancy regularization\. The divergence coefficientβ\\betais tuned over a logarithmic rangeβ∈\{10−6,5×10−7,10−7,5×10−8,10−8,5×10−9,10−9,5×10−10,10−10\}\\beta\\in\\\{10^\{\-6\},5\\times 10^\{\-7\},10^\{\-7\},5\\times 10^\{\-8\},10^\{\-8\},5\\times 10^\{\-9\},10^\{\-9\},5\\times 10^\{\-10\},10^\{\-10\}\\\}\. This range is smaller than what is typically used in single\-objective DICE settings, as nonlinear multi\-objective scalarization and trajectory\-level return conditioning change the effective scale of the dual residuals\. Using a log\-scale sweep ensures a balanced trade\-off between stability and objective optimization\.

### H\.1IQL Baseline Implementation

For the ESR\-based IQL baseline, we adopt the same network architecture as AETDICE to ensure a fair comparison\. In particular, we use identical state encoders, symlog\-transformed sinusoidal embeddings for the accumulated return inputs, and three\-layer multilayer perceptrons with the same hidden dimensions\.

We tune the expectile parameterτ\\tauover\{0\.8,0\.85,0\.9\}\\\{0\.8,0\.85,0\.9\\\}and the temperature parameterβIQL\\beta\_\{\\text\{IQL\}\}over\{10,12,14,16,18\}\\\{10,12,14,16,18\\\}\. IQL models are trained for3×1053\\times 10^\{5\}gradient steps using the Adam optimizer with the same learning rate and batch size as AETDICE\.

Table 10:Hyperparameters for AETDICE on MO\-PointMaze\.Table 11:Hyperparameters for AETDICE on D4MORL\[[31](https://arxiv.org/html/2606.31178#bib.bib3)\]\(MO\-Ant\-v2, MO\-Walker2d\-v2, MO\-HalfCheetah\-v2\)\.

## Appendix IComputational Resources

All experiments were run on a single workstation with an Intel® Xeon® Gold 6330 CPU \(256 GB RAM\) and an NVIDIA RTX 3090 GPU\. Training one AETDICE or IQL policy per task took approximately 15\-20 minutes on average, with GPU memory consumption staying below 20 GB throughout training\.

## Appendix JLimitation and Impact statement

#### Limitation

While AETDICE enables offline optimization of a broader class of nonlinear MORL objectives, it inherits the fundamental challenges of offline MORL\. First, as discussed in the ablation study \(Appendix[G](https://arxiv.org/html/2606.31178#A7)\), offline MORL datasets are collected from policies trained under diverse preference weights, making trajectories optimal for one objective highly suboptimal for another\. This requires smaller regularization to permit larger distribution shift, making offline optimization more demanding than in the single\-objective setting\. Second, the augmented state space introduces additional scalability challenges\. Incorporating𝐑tacc\\mathbf\{R\}^\{\\mathrm\{acc\}\}\_\{t\}into the state increases the dimensionality bymm\(the number of objectives\), and the data must provide sufficient coverage over this augmented space\. As the number of objectives grows, the coverage requirement becomes increasingly difficult to satisfy, potentially limiting the applicability of AETDICE to problems with a large number of objectives\.

#### Impact statement

AETDICE advances offline MORL by enabling optimization of nonlinear objectives that capture fairness and balanced performance across multiple criteria\. This is relevant to real\-world equitable decision\-making, where policies must balance competing objectives both within and across episodes\. As a methodological contribution, we do not anticipate direct negative societal impacts\. However, as with any offline RL method, the quality of learned policies depends on the data, and biases present in the dataset may propagate to the optimized policy\.