Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning
Summary
This paper introduces a novel preference-conditioned Bellman operator based on Chebyshev scalarization to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes, proving its convergence and effectiveness in capturing the entire Pareto frontier.
View Cached Full Text
Cached at: 06/26/26, 05:18 AM
# Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning Source: [https://arxiv.org/abs/2606.26397](https://arxiv.org/abs/2606.26397) [View PDF](https://arxiv.org/pdf/2606.26397) > Abstract:Real\-world decision\-making often requires balancing multiple conflicting objectives, a challenge that standard Reinforcement Learning \(RL\) frequently addresses by aggregating rewards into a single scalar signal\. While effective for simple tasks, this approach often fails to capture the full spectrum of optimal trade\-offs, known as the Pareto frontier\. In this paper, we introduce a novel preference\-conditioned Bellman operator, motivated from the Chebyshev scalarization, designed to compute deterministic Pareto\-optimal policies for Multi\-Objective Markov Decision Processes \(MOMDPs\)\. We prove that this operator satisfies an enveloping property, where the estimated value functions upper\-bound the true Pareto frontier, and demonstrate that it monotonically converges to a coverage set of this frontier\. Furthermore, we also show how to extract deterministic policies from these converged Q\-estimates\. This ensures the agent can recover a policy for any given preference, capturing the entire Pareto\-optimal frontier while guaranteeing each synthesized policy remains approximately Pareto\-optimal\. Experimental results validate that our algorithm successfully recovers complex trade\-offs, providing a solution for deterministic Pareto\-optimal policy synthesis\. ## Submission history From: Aniruddha Joshi \[[view email](https://arxiv.org/show-email/6ea3df43/2606.26397)\] **\[v1\]**Wed, 24 Jun 2026 21:28:49 UTC \(109 KB\)
Similar Articles
Bellman Policy Optimization
Bellman Policy Optimization (BPO) is a critic-free reinforcement learning method that reformulates Policy Mirror Descent using the Bellman equation for autoregressive generation with terminal rewards, improving mathematical reasoning in large language models.
Fair Policy Optimization in Major-Minor Weakly Coupled Markov Decision Processes
A HEC Montréal/MILA paper proposes fair policy optimization for major-minor weakly coupled MDPs, replacing the utilitarian objective with monotone concave fairness functions and introducing a count-proportion-based deep RL approach with a priority-based sampler, validated on machine replacement and NYC taxi pricing/relocation tasks.
Preference-Conditioned Multi-Objective Reinforcement Learning for Runtime-Tunable Transit Signal Priority
This paper presents a preference-conditioned multi-objective reinforcement learning controller for transit signal priority that allows runtime tuning of the trade-off between bus priority and overall traffic delay without retraining. Experiments show it outperforms fixed-time and rule-based baselines while maintaining feasibility constraints.
From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
This paper proposes a unified framework for personalized agentic reinforcement learning that decouples generic task rewards from personalized preference rewards, introducing PARPO and PSGM for preference-aligned policy optimization and skill retrieval.
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
This paper introduces LEMUR, a framework that combines multi-objective reinforcement learning with preference-based learning from multiple human feedback to learn Pareto-optimal policies without predefined reward functions.