Multi-Objective Constraint Inference using Inverse reinforcement learning

# Multi-Objective Constraint Inference using Inverse reinforcement learning
Source: [https://arxiv.org/html/2605.06951](https://arxiv.org/html/2605.06951)
Syed Ihtesham Hussain Shah Faculty of Sciences Vrije Universiteit Amsterdam s\.i\.h\.shah@vu\.nl &Floris den Hengst Faculty of Sciences Vrije Universiteit Amsterdam f\.den\.hengst@vu\.nl Aneta Lisowska Faculty of Sciences Vrije Universiteit Amsterdam a\.j\.lisowska@vu\.nl &Annette ten Teije Faculty of Sciences Vrije Universiteit Amsterdam annette\.ten\.teije@vu\.nl

###### Abstract

Constraint inference is widely considered essential to align reinforcement learning agents with safety boundaries and operational guidelines by observing expert demonstrations\. However, existing approaches typically assume homogeneous demonstrations \(i\.e\., generated by a single expert or multiple experts with identical objectives\)\. They also have limited ability to capture individual preferences and often suffer from computational inefficiencies\. In this paper, we introduce Multi\-Objective Constraint Inference \(MOCI\), a novel framework designed to jointly extract shared constraints and individual preferences from heterogeneous expert trajectories, where multiple experts pursue different objectives\. MOCI effectively models and learns from diverse, and potentially conflicting, behaviors\. Empirical evaluations demonstrate that MOCI significantly outperforms existing baselines, achieving improved predictive performance, and maintaining competitive computational efficiency on a standard grid\-world benchmark\. These results establish MOCI as an accurate, flexible, and computationally practical approach for real\-world constraint inference and preference learning tasks\.

## 1Introduction

Inverse Reinforcement Learning \(IRL\) is a framework for learning the underlying objectives of an expert by observing their behavior\[[13](https://arxiv.org/html/2605.06951#bib.bib3)\]\. Instead of manually specifying a reward function, IRL infers the reward that best explains the expert’s demonstrated actions\. Traditionally, the intent of an agent is modeled by a learned reward function, however, accurately matching the expert’s behavior can make the reward overly complex and brittle to small changes\. Instead of capturing complex and diverse motivations of the expert in a single reward function, expert behavior can be more easily explained by jointly learning a simple reward function and inferring a set of clear hard constraints\[[12](https://arxiv.org/html/2605.06951#bib.bib1)\]\. Although jointly learning constraints and reward functions yield simpler reward models that better explain expert trajectories, existing work\[[8](https://arxiv.org/html/2605.06951#bib.bib13),[7](https://arxiv.org/html/2605.06951#bib.bib21)\]suffers from two restrictive assumptions: \(1\) the dataset is homogeneous, i\.e\. generated by a single type of expert and \(2\) the expert’s reward function is completely knowna priori\. These assumptions are problematic in real\-world settings where observed data typically emerge from a heterogeneous population of experts, and the reward function is not known\. For instance, in urban driving, different drivers exhibit varying driving styles, ranging from aggressive to cautious, yet all are subject to the same shared physical and legal constraints, such as speed limits and lane boundaries\. At the same time, drivers have different driving and navigation preferences such as reaching the destination with minimum distance, or taking a longer route with more pleasant sceneries\. In this paper, we address the challenge of inferring shared constraints and learning individual preferences from heterogeneous expert demonstrations\. We propose Multi\-Objective Constraint Inference \(MOCI\), a novel framework to jointly recover shared constraints and preferences from unlabeled demonstrations generated by multiple experts with varying preferences and in an environment with hard constraints shared across all experts\. MOCI iteratively clusters expert trajectories while learning personalized reward weights\. Based on these clusters \(groups\), MOCI then identifies shared constraintsCCby evaluating each state using the joint log\-likelihood method\[[18](https://arxiv.org/html/2605.06951#bib.bib19)\]\. A hard constraint is identified for a state only if its removal from the feasible environment increases the likelihood of the observed demonstrations across all clusters simultaneously\. Hence, MOCI jointly learns shared constraints and personalized reward weights by applying maximum entropy inverse reinforcement learning to each group, allowing the separation of individual expert preferences from shared environmental constraints\. We evaluate our approach on a multi\-objective GridWorld environment, a well\-established benchmark for sequential decision\-making problems\. This controlled setting allows us to systematically validate the proposed method\. The remainder of this paper is organized as follows: Section[2](https://arxiv.org/html/2605.06951#S2)formalizes the preliminaries and foundational concepts upon which the framework is built, including Constrained Markov Decision Processes \(CMDPs\) and Maximum Entropy Inverse Reinforcement Learning \(IRL\)\. Section[3](https://arxiv.org/html/2605.06951#S3)introduces the proposed Multi\-Objective Constraint Inference \(MOCI\) algorithm and provides a detailed analysis of its computational complexity\. Section[4](https://arxiv.org/html/2605.06951#S4)details the experimental setup using a simulated heterogeneous Gridworld environment and discusses the empirical results of the framework\. Section[5](https://arxiv.org/html/2605.06951#S5)provides a performance comparison between MOCI and existing baseline techniques\. Finally, Section[6](https://arxiv.org/html/2605.06951#S6)summarizes the study’s conclusions and discusses current limitations\.

## 2Preliminaries

In this section, we formalize the foundational concepts upon which the Multi\-Objective Constraint Inference \(MOCI\) framework is built\. We define the environments, the structure of the agent’s objectives, and the methodologies used to infer those objectives from the data\.

### 2\.1Constrained Markov Decision Processes

A standard Markov Decision Process \(MDP\) is defined as a tupleℳ=⟨𝒮,𝒜,𝒯,γ,R⟩\\mathcal\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\gamma,R\\rangle, where𝒮\\mathcal\{S\}is the state space,𝒜\\mathcal\{A\}is the action space,𝒯​\(s′∣s,a\)∈𝒮×𝒜→Δ​\(𝒮\)\\mathcal\{T\}\(s^\{\\prime\}\\mid s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\Delta\(\\mathcal\{S\}\)is the transition probability distribution,γ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor,R​\(s,a\)∈𝒮×𝒜→ℝR\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}is the reward function\[[15](https://arxiv.org/html/2605.06951#bib.bib5)\]\. The behavior of an agent is defined by a policyπ​\(a∣s\)∈𝒮→Δ​\(𝒜\)\\pi\(a\\mid s\)\\in\\mathcal\{S\}\\to\\Delta\(\\mathcal\{A\}\), which assigns states to a probability distribution over actions\. AConstrained Markov Decision Process\(CMDP\)\[[16](https://arxiv.org/html/2605.06951#bib.bib4)\]extends this framework by restricting the set of allowable policies\. Although CMDP can be formulated using cost functions and budget thresholds, we focus on the context of hard environmental constraints \(such as walls or strict physical limitations\), defined as a set of forbidden or unsafe states denotedC⊂𝒮C\\subset\\mathcal\{S\}following recent related work byKimet al\.\[[6](https://arxiv.org/html/2605.06951#bib.bib17)\], Qadriet al\.\[[10](https://arxiv.org/html/2605.06951#bib.bib16)\]\. The CMDP is thus augmented toℳC=⟨𝒮,𝒜,𝒯,γ,R,C⟩\\mathcal\{M\}\_\{C\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\gamma,R,C\\rangle\. A trajectoryξ=\{\(s0,a0\),\(s1,a1\),…,\(sT,aT\)\}\\xi=\\\{\(s\_\{0\},a\_\{0\}\),\(s\_\{1\},a\_\{1\}\),\\dots,\(s\_\{T\},a\_\{T\}\)\\\}of lengthH∈ℕH\\in\\mathbb\{N\}is considered valid if and only if it does not violate any constraints inCC\[[11](https://arxiv.org/html/2605.06951#bib.bib6)\]\. This is formalized using a binary indicator function:

𝕀C​\(ξ\)=\{1if​st∉C​for all​t∈\{0,…,H\}0otherwise\\mathbb\{I\}^\{C\}\(\\xi\)=\\begin\{cases\}1&\\text\{if \}s\_\{t\}\\notin C\\text\{ for all \}t\\in\\\{0,\\dots,H\\\}\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(1\)

### 2\.2Multi\-Objective MDP

AMulti\-Objective Markov Decision Process\(MOMDP\) extends traditional MDPs with a vectorial reward function𝑹∈𝒮×𝒜→ℝd\\boldsymbol\{R\}\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\}to model environments withddpotentially competing objectives \(e\.g\., minimizing travel time while maximizing safety\)\[[3](https://arxiv.org/html/2605.06951#bib.bib14),[5](https://arxiv.org/html/2605.06951#bib.bib15)\]\. This vectorial reward can be collapsed into a scalar reward if the particular way in which these multiple objectives are to be combined for some particular individual or use case, are known\. In case this so\-called*scalarization*function is linear in the objectives, we refer to its weightsw:=\[w1,…,wp\]w:=\\left\[w\_\{1\},\\dots,w\_\{p\}\\right\]as the preferences of that individual such that their reward functionR=w⊤​𝑹R=w^\{\\top\}\\boldsymbol\{R\}\.

In practice, this is often expressed via a feature formulation whereϕ​\(s,a\)∈ℝd\\phi\(s,a\)\\in\\mathbb\{R\}^\{d\}represents a vector of state\-action features, and an agent’s specific preference is defined by a weight vectorww\. The scalarized reward\[[14](https://arxiv.org/html/2605.06951#bib.bib2)\]for an agent with preferencewwcan be defined as:

Rw​\(s,a\)=w⊤​ϕ​\(s,a\)R\_\{w\}\(s,a\)=w^\{\\top\}\\phi\(s,a\)\(2\)In a heterogeneous multi\-agent setting\[[19](https://arxiv.org/html/2605.06951#bib.bib8)\], different expertsk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}share the same state\-action featuresϕ\\phibut possess distinct, private preference weightswkw\_\{k\}, leading to diverse optimal policies within the same underlying environment\.

### 2\.3Inverse Reinforcement Learning

Inverse Reinforcement Learning\(IRL\) addresses the problem of extracting an agent’s underlying reward function given its demonstrated behavior\[[2](https://arxiv.org/html/2605.06951#bib.bib9)\]\. Formally, we assume access to the MDPs transition function but not its reward function, denoted asℳ∖R\\mathcal\{M\}\\setminus R, and a dataset of expert trajectories𝒟=\{ξ1,…,ξN\}\\mathcal\{D\}=\\\{\\xi\_\{1\},\\dots,\\xi\_\{N\}\\\}generated by an expert policyπE\\pi\_\{E\}\.

The goal of IRL is to find a reward functionR∗R^\{\*\}such that the expert’s policyπE\\pi\_\{E\}is optimal\[[1](https://arxiv.org/html/2605.06951#bib.bib10)\]\. If the reward is parameterized linearly asR​\(s,a\)=w⊤​ϕ​\(s,a\)R\(s,a\)=w^\{\\top\}\\phi\(s,a\), the IRL problem can be framed as finding a weight vectorw∗w^\{\*\}such that the expected feature counts of the expert match the expected feature counts of a policy optimizingw∗w^\{\*\}:

𝔼πE​\[∑t=0Tγt​ϕ​\(st,at\)\]=𝔼πw∗​\[∑t=0Tγt​ϕ​\(st,at\)\]\\mathbb\{E\}\_\{\\pi\_\{E\}\}\\left\[\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\phi\(s\_\{t\},a\_\{t\}\)\\right\]=\\mathbb\{E\}\_\{\\pi\_\{w^\{\*\}\}\}\\left\[\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\phi\(s\_\{t\},a\_\{t\}\)\\right\]\(3\)However, this problem is inherently ill\-posed, as multiple reward functions \(including a trivial reward of all zeros\) can explain the same behavior\.

### 2\.4Maximum Entropy IRL

To resolve the ambiguity of the IRL problem,\[[20](https://arxiv.org/html/2605.06951#bib.bib18)\]introducedMaximum Entropy Inverse Reinforcement Learning\(MaxEnt IRL\)\. MaxEnt IRL applies the principle of maximum entropy to select the probability distribution over trajectories that matches the expert’s empirical feature expectations while making no other assumptions \(i\.e\., being as random as possible otherwise\)\.

Under the MaxEnt framework, the probability of an agent choosing a specific trajectoryξ\\xiis exponentially proportional to the total accumulated reward of that trajectory:

P​\(ξ∣w\)=1Z​\(w\)​exp⁡\(∑\(s,a\)∈ξw⊤​ϕ​\(s,a\)\)=1Z​\(w\)​eRw​\(ξ\)P\(\\xi\\mid w\)=\\frac\{1\}\{Z\(w\)\}\\exp\\left\(\\sum\_\{\(s,a\)\\in\\xi\}w^\{\\top\}\\phi\(s,a\)\\right\)=\\frac\{1\}\{Z\(w\)\}e^\{R\_\{w\}\(\\xi\)\}\(4\)whereZ​\(w\)Z\(w\)is the partition function, representing the integral \(or sum\) over all possible trajectories originating from the start state:

Z​\(w\)=∑ξ′∈ΞeRw​\(ξ′\)Z\(w\)=\\sum\_\{\\xi^\{\\prime\}\\in\\Xi\}e^\{R\_\{w\}\(\\xi^\{\\prime\}\)\}\(5\)
The reward weightswware then found by maximizing the log\-likelihood of the demonstrated trajectories𝒟\\mathcal\{D\}:

ℒ​\(w\)=∑i=1\|𝒟\|log⁡P​\(ξi∣w\)=∑i=1\|𝒟\|\(w⊤​ϕ​\(ξi\)−log⁡Z​\(w\)\)\\mathcal\{L\}\(w\)=\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\log P\(\\xi\_\{i\}\\mid w\)=\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\left\(w^\{\\top\}\\phi\(\\xi\_\{i\}\)\-\\log Z\(w\)\\right\)\(6\)The gradient of this log\-likelihood neatly reduces to the difference between the empirical feature counts of the demonstrations and the expected feature counts under the current weight vectorww, allowing for efficient optimization via gradient ascent\.

## 3Multi\-Objective Constraint Inference \(MOCI\)

In this section, we present our approach for jointly learning shared constraints and individual preferences\. We also provide a theoretical analysis of the computational complexity of the proposed algorithm, highlighting its scalability with respect to the number of states, actions, and demonstrations\.

### 3\.1Approach

Let𝒟=\{ξ1,…,ξN\}\\mathcal\{D\}=\\\{\\xi\_\{1\},\\dots,\\xi\_\{N\}\\\}be a dataset of demonstrated trajectories of maximum lengthHH\. We assume aConstraint Multi\-Objective Markov Decision Process\(CMOMDP\)⟨𝒮,𝒜,𝒯,γ,𝑹,C⟩\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\gamma,\\boldsymbol\{R\},C\\rangleand the existence ofKKlatent expert types \(clusters\), where each typek∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}is characterized by a specific preference weight vectorwkw\_\{k\}such thatRk=wk⊤​𝑹R\_\{k\}=w\_\{k\}^\{\\top\}\\boldsymbol\{R\}, and a prior probabilityπk=P​\(k\)\\pi\_\{k\}=P\(k\)\. Crucially, the demonstrations are not labelled by their associated expert typek∈Kk\\in Kas inferring expert preferences is a key goal of this work\.

Although experts differ in their preferences, all agents share the same state\-action feature functionϕ\\phi, and operate subject to a shared set of hard physical constraints, denoted byCC\. Following the Maximum Entropy Inverse Reinforcement Learning \(MaxEnt IRL\) framework byZiebartet al\.\[[20](https://arxiv.org/html/2605.06951#bib.bib18)\], the probability of observing a specific trajectoryξ\\xi, given that it was generated by an expert of typekksubject to constraintsCC, is defined as:

P​\(ξ∣C,wk\)=1Z​\(C,wk\)​eRwk​\(ξ\)​𝕀C​\(ξ\)P\(\\xi\\mid C,w\_\{k\}\)=\\frac\{1\}\{Z\(C,w\_\{k\}\)\}e^\{R\_\{w\_\{k\}\}\(\\xi\)\}\\mathbb\{I\}^\{C\}\(\\xi\)\(7\)where,Rwk​\(ξ\)=∑\(s,a\)∈ξwk⊤​ϕ​\(s,a\)R\_\{w\_\{k\}\}\(\\xi\)=\\sum\_\{\(s,a\)\\in\\xi\}w\_\{k\}^\{\\top\}\\phi\(s,a\)is the cumulative reward of the trajectory under preferencewkw\_\{k\}\.Z​\(C,wk\)Z\(C,w\_\{k\}\)is the partition function over all feasible paths in the constrained Markov Decision Process \(MDP\)\. The detailed analysis and proof are given in the appendix\-[B](https://arxiv.org/html/2605.06951#A2)\.𝕀C​\(ξ\)\\mathbb\{I\}^\{C\}\(\\xi\)is an indicator function that equals11if the trajectoryξ\\xidoes not violate any constraint inCCand0otherwise as presented in equation \([1](https://arxiv.org/html/2605.06951#S2.E1)\)\. Marginalizing over the latent assignment of demonstrations to expert types, the likelihood of a single demonstration is:

P​\(ξ∣C,\{wk\},\{πk\}\)=∑k=1Kπk​eRwk​\(ξ\)Z​\(C,wk\)​𝕀C​\(ξ\)P\(\\xi\\mid C,\\\{w\_\{k\}\\\},\\\{\\pi\_\{k\}\\\}\)=\\sum\_\{k=1\}^\{K\}\\pi\_\{k\}\\frac\{e^\{R\_\{w\_\{k\}\}\(\\xi\)\}\}\{Z\(C,w\_\{k\}\)\}\\mathbb\{I\}^\{C\}\(\\xi\)\(8\)
Our objective is to jointly infer the shared constraintsCC, the latent reward weights\{wk\}\\\{w\_\{k\}\\\}, and the priors\{πk\}\\\{\\pi\_\{k\}\\\}by maximizing the joint log\-likelihood of the dataset𝒟\\mathcal\{D\}:

ℒ​\(C,\{wk\},\{πk\}\)=∑i=1\|𝒟\|log⁡\(∑k=1Kπk​eRwk​\(ξi\)Z​\(C,wk\)​𝕀C​\(ξi\)\)\\mathcal\{L\}\(C,\\\{w\_\{k\}\\\},\\\{\\pi\_\{k\}\\\}\)=\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\log\\left\(\\sum\_\{k=1\}^\{K\}\\pi\_\{k\}\\frac\{e^\{R\_\{w\_\{k\}\}\(\\xi\_\{i\}\)\}\}\{Z\(C,w\_\{k\}\)\}\\mathbb\{I\}^\{C\}\(\\xi\_\{i\}\)\\right\)\(9\)
Because the assignments of the demonstrations to agent types or clusters are unobserved, directly optimizing this joint log\-likelihood is intractable\. We therefore optimize equation \([9](https://arxiv.org/html/2605.06951#S3.E9)\) using an Expectation\-Maximization \(EM\) approach\[[4](https://arxiv.org/html/2605.06951#bib.bib11)\], which alternates between estimating the posterior probability of cluster assignments and updating the model parameters alongside the constraint set\.

In the Expectation step \(E\-step\), we fix the current constraintsCC, weights\{wk\}\\\{w\_\{k\}\\\}, and priors\{πk\}\\\{\\pi\_\{k\}\\\}, and compute the responsibilityγi,k\\gamma\_\{i,k\}, which represents the posterior probability that trajectoryξi\\xi\_\{i\}was generated by expertkk:

γi,k=πk​P​\(ξi∣C,wk\)∑j=1Kπj​P​\(ξi∣C,wj\)\\gamma\_\{i,k\}=\\frac\{\\pi\_\{k\}P\(\\xi\_\{i\}\\mid C,w\_\{k\}\)\}\{\\sum\_\{j=1\}^\{K\}\\pi\_\{j\}P\(\\xi\_\{i\}\\mid C,w\_\{j\}\)\}\(10\)In the Maximization step \(M\-step\), we update the parameters to maximize the expected log\-likelihood over the entire dataset\. The cluster priors are updated as the empirical mean of the responsibilities, such that

πk=1\|𝒟\|​∑i=1\|𝒟\|γi,k\\pi\_\{k\}=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\gamma\_\{i,k\}\(11\)To update the reward weightswkw\_\{k\}for each cluster, we perform gradient ascent where the gradient for trajectoryiiand clusterkkis weighted by the responsibilityγi,k\\gamma\_\{i,k\}, ensuring that the weights adapt to the trajectories probabilistically assigned to that cluster:

∇wkℒ=∑i=1\|𝒟\|γi,k​\(ϕ​\(ξi\)−𝔼P​\(ξ∣C,wk\)​\[ϕ​\(ξ\)\]\)\\nabla\_\{w\_\{k\}\}\\mathcal\{L\}=\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\gamma\_\{i,k\}\\left\(\\phi\(\\xi\_\{i\}\)\-\\mathbb\{E\}\_\{P\(\\xi\\mid C,w\_\{k\}\)\}\[\\phi\(\\xi\)\]\\right\)\(12\)where𝔼P​\(ξ∣C,wk\)​\[ϕ​\(ξ\)\]\\mathbb\{E\}\_\{P\(\\xi\\mid C,w\_\{k\}\)\}\[\\phi\(\\xi\)\]is the expected feature count under the current constraints and cluster weights\.wk←wk\+α​∇wkℒw\_\{k\}\\leftarrow w\_\{k\}\+\\alpha\\nabla\_\{w\_\{k\}\}\\mathcal\{L\}

Finally, to update the shared constraintsCC, we execute a greedy search over the set of candidate constraints \(states not visited by any expert in𝒟\\mathcal\{D\}\)\. For a candidate constraintcc, we define its score as the joint log\-likelihood evaluated with the augmented constraint set;

Score​\(c\)=ℒ​\(C∪\{c\},\{wk\},\{πk\}\)\\text\{Score\}\(c\)=\\mathcal\{L\}\(C\\cup\\\{c\\\},\\\{w\_\{k\}\\\},\\\{\\pi\_\{k\}\\\}\)\(13\)The algorithm iteratively adds the candidatec^=arg⁡maxc⁡Score​\(c\)\\hat\{c\}=\\arg\\max\_\{c\}\\text\{Score\}\(c\)toCC\. This greedy addition terminates when the reduction in Kullback\-Leibler \(KL\) divergence falls below a specified thresholddD​K​Ld\_\{DKL\}, which is mathematically equivalent to stopping when the increase in log\-likelihood satisfies;

ℒ​\(C∪\{c^\},⋅\)−ℒ​\(C,⋅\)≤dD​K​L\\mathcal\{L\}\(C\\cup\\\{\\hat\{c\}\\\},\\cdot\)\-\\mathcal\{L\}\(C,\\cdot\)\\leq d\_\{DKL\}\(14\)This thresholding mechanism serves as a regularizer, preventing the algorithm from overfitting to noise by rejecting constraints that yield only marginal improvements to the model’s likelihood\.

The algorithm for MOCI is presented in appendix \-[D](https://arxiv.org/html/2605.06951#A4)\. The goal of the algorithm is to employs an Expectation\-Maximization \(EM\) approach to infer shared constraints and multi\-objective preferences\.

### 3\.2Computational Complexity

The MOCI algorithm calculates the likelihood of each trajectory in𝒟\\mathcal\{D\}demonstrations belonging to each groupk∈Kk\\in K\. It computes partition functionZZ, which iterates through the maximum horizon \(trajectory length\)HH, checking all actions\|𝒜\|\|\\mathcal\{A\}\|for all valid states\|𝒮\|\|\\mathcal\{S\}\|\. Which yields complexity:

𝒪E​\(K⋅H⋅\(\|𝒮\|⋅\|𝒜\|\+\|𝒟\|\)\)\\mathcal\{O\}\_\{E\}\(K\\cdot H\\cdot\(\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\+\|\\mathcal\{D\}\|\)\)\(15\)For each clusterKK, the algorithm performsII​R​LI\_\{IRL\}gradient descent steps\. In each step, it recomputesZZto find the expected feature counts and compares them to the empirical features of the demonstrations\. It can be shown as:

𝒪M​\(K⋅II​R​L⋅H⋅\(\|𝒮\|⋅\|𝒜\|\+\|𝒟\|\)\)\\mathcal\{O\}\_\{M\}\(K\\cdot I\_\{IRL\}\\cdot H\\cdot\(\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\+\|\\mathcal\{D\}\|\)\)\(16\)The algorithm also tests unvisited candidate states to see if adding them to the constraint set𝒞\\mathcal\{C\}reduces the log\-likelihood by less than threshold \(dD​K​Ld\_\{DKL\}\)\. Let𝒞a​d​d​e​d\\mathcal\{C\}\_\{added\}be the number of constraints successfully inferred in a single step\. For each added constraint, the algorithm tests a subset of candidates \(Mt​e​s​tM\_\{test\}\)\.

𝒪𝒞​\(𝒞a​d​d​e​d⋅Mt​e​s​t⋅K⋅\|𝒮\|⋅\|𝒜\|⋅H\)\\mathcal\{O\}\_\{\\mathcal\{C\}\}\(\\mathcal\{C\}\_\{added\}\\cdot M\_\{test\}\\cdot K\\cdot\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\\cdot H\)\(17\)To find the total computational cost, we sum the complexities of all the steps \(Expectation step𝒪E\\mathcal\{O\}\_\{E\}, Weights𝒪M\\mathcal\{O\}\_\{M\}, and Constraints𝒪𝒞\\mathcal\{O\}\_\{\\mathcal\{C\}\}\), and multiply by the total number of iterations \(II​t​e​rI\_\{Iter\}\)\.

Cost=𝒪​\(II​t​e​r⋅K⋅\|𝒮\|⋅\|𝒜\|⋅H⋅\(1\+II​R​L\+𝒞a​d​d​e​d⋅Mt​e​s​t\)\)\\text\{Cost\}=\\mathcal\{O\}\\Big\(I\_\{Iter\}\\cdot K\\cdot\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\\cdot H\\cdot\(1\+I\_\{IRL\}\+\\mathcal\{C\}\_\{added\}\\cdot M\_\{test\}\)\\Big\)\(18\)To simplify complexity equation, we isolate the terms that grow the fastest as the problem scales toward infinity\. In Inverse Reinforcement Learning, the hyper\-parameters \(II​t​e​r,K,II​R​L,Mt​e​s​t,𝒞a​d​d​e​dI\_\{Iter\},K,I\_\{IRL\},M\_\{test\},\\mathcal\{C\}\_\{added\}\) can be treated as constants\. Furthermore, the size of the state\-action space multiplied by the horizon \(\|𝒮\|⋅\|𝒜\|⋅H\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\\cdot H\) will always asymptotically dominate the number of sampled demonstrations \(\|𝒟\|\|\\mathcal\{D\}\|\)\. By dropping the constants and non\-dominant terms, the total theoretical computational cost reduces to:

Cost=𝒪​\(\|𝒮\|⋅\|𝒜\|⋅H\)\\text\{Cost\}=\\mathcal\{O\}\(\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\\cdot H\)\(19\)The cost of MOCI is highly dependent on the total number of states\|𝒮\|\|\\mathcal\{S\}\|, the total number of actions per state\|𝒜\|\|\\mathcal\{A\}\|, and the horizon \(maximum length of a trajectory\)HH\. For a specific environment, e\.g\., Gridworld, the total number of states\|𝒮\|=N2\|\\mathcal\{S\}\|=N^\{2\}, hence the MOCI algorithm scales quadratically with the grid dimensions\.

![Refer to caption](https://arxiv.org/html/2605.06951v1/Diagrams/True_GridWorld_Env_and_Demos.png)Figure 1:Demonstration of the MOCI algorithm in a 6x6 Gridworld\. Ground\-truth environment with water hard constraints \(blue\) and expert trajectories for a Grass\-Lover \(lime\) and Rock\-Lover \(orange\)\.
![Refer to caption](https://arxiv.org/html/2605.06951v1/Diagrams/MOCI-inferred-constraints.png)Figure 2:MOCI\-inferred constraints \(red hatched\), showing successful recovery of true constraints alongside false positives in unvisited states\.

## 4Experiments: Heterogeneous Gridworld

To validate the Multi\-Objective Constraint Inference \(MOCI\) framework, we designed a simulated multi\-feature Gridworld environment\. This environment serves as a controlled testbed for evaluating the MOCI algorithm\.

### 4\.1Environment Setup

The environment is modeled as a deterministic Markov Decision Process \(MDP\) on aN×NN\\times Ngrid as shown in figure\-[2](https://arxiv.org/html/2605.06951#S3.F2)\. The state space𝒮\\mathcal\{S\}consists of grid cells, and the action space𝒜\\mathcal\{A\}allows for standard movement:\{Up, Down, Left, Right, Stay\}\\\{\\textit\{Up, Down, Left, Right, Stay\}\\\}\. The environment features a "Start" state in the top\-left corner and a "Goal" state in the bottom\-right corner\. Each states∈𝒮s\\in\\mathcal\{S\}is assigned a categorical feature vectorϕ​\(s\)∈\{0,1\}4\\phi\(s\)\\in\\\{0,1\\\}^\{4\}that represents four different types of terrain: grasslands \(green\), Rocks\(brown\), Water \(blue\) and Normal \(light beige\) tiles\. Exactly one feature is active for any given state\. The true underlying environment contains a set of impassable obstacles \(blue water tiles in this case\) that represent our set of ground\-truth hard constraintsC∗C^\{\*\}\. Transitions into statess∈C∗s\\in C^\{\*\}are strictly prohibited in the environment dynamics\. To simulate a heterogeneous population, we defined two distinct expert types with divergent reward preferences, parameterized by their ground\-truth weight vectorswk∗w\_\{k\}^\{\*\}:

1. 1\.Expert 1 \(Grass\-Lover\):Highly prefers traveling on the Grass tiles
2. 2\.Expert 2 \(Rock\-Lover\):Highly prefers traveling on the Rock tiles

The agent’s objectives are to reach the goal state by taking the shortest path while navigating their preferred terrain\. Both experts share the same terminal goal state and are fundamentally restricted by the Water tiles \(C∗C^\{\*\}\)\. To generate the demonstration dataset𝒟\\mathcal\{D\}, we computed the optimal Maximum Entropy policyπk∗\\pi\_\{k\}^\{\*\}for each expert using Soft Value Iteration based on their respective reward weightswk∗w\_\{k\}^\{\*\}\. We then sampled\|𝒟1\|=10\|\\mathcal\{D\}\_\{1\}\|=10trajectories from Expert 1 and\|𝒟2\|=10\|\\mathcal\{D\}\_\{2\}\|=10trajectories from Expert 2\. The final dataset provided to the inference algorithms was an unlabeled, randomly shuffled mixture of these 20 trajectories,𝒟=𝒟1∪𝒟2\\mathcal\{D\}=\\mathcal\{D\}\_\{1\}\\cup\\mathcal\{D\}\_\{2\}\. In Figure\-[2](https://arxiv.org/html/2605.06951#S3.F2), the trajectory of Expert 1 \(Grass\-Lover\) is shown in lime, while the trajectory of Expert 2 \(Rock\-Lover\) is shown in orange\.

### 4\.2Results

All results reported in this paper were obtained by running experiments on aMacBook M3usingPython 3\.11\.5\. The figure\-[2](https://arxiv.org/html/2605.06951#S3.F2)visualizes the constraints inferred by the MOCI algorithm after observing the expert trajectories shown in figure\-[2](https://arxiv.org/html/2605.06951#S3.F2)\. The algorithm successfully places "Inferred Constraints" \(marked by red hatched boxes\) over the blue water tiles\. By observing that neither the grass\-lover nor the rock\-lover ever stepped on these tiles, MOCI correctly deduced they are forbidden\. The red hatched boxes also appear on several non\-water tiles \(e\.g\., a normal tile on the top right\)\. This occurs because these specific location was never visited by either expert\. The algorithm conservatively \(and incorrectly\) inferred that these unvisited states must also be hard constraints\. It shows that while MOCI can successfully learn heterogeneous preferences and recover ground\-truth constraints \(the water\), limited dataset coverage can lead the algorithm to hallucinate constraints in regions of the state space that were simply ignored by the expert’s specific goals\. The figure\-[3](https://arxiv.org/html/2605.06951#S4.F3)illustrates the MOCI algorithm’s capability to untangle and learn the distinct underlying reward functions of different types of experts from a shared environment\. The visualization is split into two side\-by\-side bar charts, each representing a specific expert profile evaluated across four terrain features: Sand, Grass, Rocks, and Water\. This left\-hand chart compares the original Ground Truth weights \(light gray bars\) against the MOCI Learned weights \(green bars\)\. The algorithm accurately captures the expert’s defining trait: a strong positive preference for Grass\. It also correctly identifies the negative polarity for Rocks and Water, meaning the algorithm learned that the expert avoids these terrains\. The right\-hand chart contrasts the Ground Truth weights \(light gray bars\) with the MOCI Learned weights \(brown bars\) for the second expert\. Here, the algorithm successfully identifies the opposite behavior, showing a strong positive preference for Rocks\. It correctly assigns negative weights to Grass and Water, capturing the expert’s specific aversions\. Tuning ThresholddD​K​Ld\_\{DKL\}:The adjustment of the threshold value is arguably the most critical step in the implementation of MOCI\. The threshold \(dD​K​Ld\_\{DKL\}dictates how much the likelihood of a trajectory must drop before the algorithm considers a state to be a hard constraint\. If it is too high, the algorithm misses the real walls \(false negatives\)\. If it is too low, the algorithm looks at a perfectly navigable tile that the experts simply disliked and declares it a physical wall \(a false positive\)\. We have presented some of the factors on which the performance of MOCI is really dependent in Appendix\-[C](https://arxiv.org/html/2605.06951#A3)\. To ensure that the threshold remains invariant with the size of the dataset\|𝒟\|\|\\mathcal\{D\}\|, we normalize the joint log\-likelihood to represent the average information gained per trajectory:

La​v​g=1\|𝒟\|​∑i=1\|𝒟\|log⁡P​\(ξi\|wk,C\)L\_\{avg\}=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\log P\(\\xi\_\{i\}\|w\_\{k\},C\)\(20\)By evaluating the normalized change,Δ​La​v​g\\Delta L\_\{avg\}, the divergence thresholddD​K​Ld\_\{DKL\}can be robustly set to a constant value across varying expert population sizes\. Established literature suggests an effective range of0\.01≤dD​K​L≤20\.01\\leq d\_\{DKL\}\\leq 2for this parameter\.

![Refer to caption](https://arxiv.org/html/2605.06951v1/Diagrams/preference_recovery_barchart.png)Figure 3:Joint recovery of heterogeneous preferences using the MOCI algorithm\. The chart compares the ground\-truth reward weights with the learned weights across four terrain features for two distinct expert profiles \(a Grass\-Lover and a Rock\-Lover\)\.![Refer to caption](https://arxiv.org/html/2605.06951v1/Diagrams/Threshhold_Plot.png)Figure 4:Effect of the number of expert demonstrations \(\|𝒟\|\|\\mathcal\{D\}\|\) on the False Positive Rate \(FPR\) during constraint inference evaluated across four thresholds \(dD​K​Ld\_\{DKL\}\) in a 5x5 Gridworld\.
![Refer to caption](https://arxiv.org/html/2605.06951v1/Diagrams/FPR_vs_GridSize.png)Figure 5:Robustness of the algorithm’s False Positive Rate \(FPR\) relative to environmental scaling \(grid sizes from 5x5 to 10x10\) given a fixed dataset of\|D\|=20\|D\|=20expert demonstrations\.

Figure\-[5](https://arxiv.org/html/2605.06951#S4.F5)illustrates the False Positive Rate \(FPR\) with respect to the four different thresholds \(dD​K​L∈\{0\.01,0\.05,0\.5,2\.0\}d\_\{DKL\}\\in\\\{0\.01,0\.05,0\.5,2\.0\\\}\)\. The performance of the algorithm is evaluated on different number of Expert Demonstrations \(\|𝒟\|\|\\mathcal\{D\}\|\)\. Despite the different threshold values, the FPR does not escalate as the number of expert demonstrations increases\. Instead, the metric remains highly stable and shows a trend toward convergence\. When the algorithm receives very few demonstrations \(e\.g\.\|𝒟\|=1\|\\mathcal\{D\}\|=1\), there is a higher degree of uncertainty\. This lack of data causes the algorithm to over\-infer constraints, resulting in a higher FPR\. As the number of expert demonstrations increases, the FPR drops and converges\. The highest threshold \(dD​K​L=2\.0d\_\{DKL\}=2\.0, represented by the blue line\) consistently yields the lowest False Positive Rate for this particular gird size\. This proves that the algorithm successfully normalizes the log\-likelihood against the dataset size, preventing metric inflation as more data are introduced\. Robustness:Figure[5](https://arxiv.org/html/2605.06951#S4.F5)presents an evaluation of the algorithm’s robustness in terms of its False Positive Rate \(FPR\) as the environment scales \(grid sizes\), while keeping the amount of expert data fixed at\|𝒟\|=20\|\\mathcal\{D\}\|=20demonstrations\. As the grid scales from5×55\\times 5to10×1010\\times 10, the total number of states increases quadratically \(from 25 to 100\)\. Consequently, a fixed dataset of 20 trajectories covers a progressively smaller percentage of the environment\. Because the expert’s coverage becomes sparser in larger grids, the algorithm’s uncertainty increases slightly, leading to a minor upward trend in the FPR\. Additionally, despite the state space quadrupling in size, the FPR remains remarkably low\. Scalability effect on runtime:Figure\-[6](https://arxiv.org/html/2605.06951#S4.F6)illustrates the execution time \(in seconds\) of the MOCI algorithm as the complexity of the environment increases from a5×55\\times 5grid to a10×1010\\times 10grid\. The blue line \(short trajectory\) represents an optimal direct path to the goal, where the horizon scales roughly twice the dimension of the grid \(H≈2​NH\\approx 2N\)\. The red line \(long trajectory\) represents a sub\-optimal or highly exploratory path, where the horizon could scale at five times the grid dimension \(H≈5​NH\\approx 5N\)\. For optimal paths \(2​N~2N\), the computational cost increases at a manageable rate\. In a5×55\\times 5grid, the execution takes roughly 0\.3 seconds, and it smoothly increases to just under 1\.6 seconds for a10×1010\\times 10grid\. For exploratory paths \(5​N~5N\), the algorithm requires significantly more computation\. The execution time curves increase much more sharply, starting around 0\.4 seconds for a5×55\\times 5grid but rising to nearly 4\.7 seconds for a10×1010\\times 10grid\. This graph perfectly validates the theoretical computational cost previously defined as𝒪​\(N2⋅\|𝒜\|⋅H\)\\mathcal\{O\}\(N^\{2\}\\cdot\|\\mathcal\{A\}\|\\cdot H\)\. In summary, these findings validate MOCI as a uniquely robust, scalable, and computationally viable framework for extracting shared constraints from complex, heterogeneous expert behaviors\.

![Refer to caption](https://arxiv.org/html/2605.06951v1/Diagrams/Runtime_vs_GridSize_Dynamic_Horizons.png)
Figure 6:Scalability analysis of run\-time versus grid size with dynamically scaling trajectory lengths\. The execution time exhibits a steepening curve, effectively validating the theoretical complexity𝒪​\(\|𝒮\|⋅\|𝒜\|⋅H\)\\mathcal\{O\}\(\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\\cdot H\)\.Table 1:Performance Comparison with existing techniques

## 5Comparison with existing techniques

To rigorously evaluate the efficacy of the proposed Multi\-Objective Constraint Inference \(MOCI\) framework, it is imperative to benchmark its performance against established state\-of\-the\-art techniques\. Table\-[1](https://arxiv.org/html/2605.06951#S4.T1)presents the performance comparison with existing techniques\. It compares the MOCI techniques with the two existing baselines, Maximum Likelihood Constraint Inference \(MLCI\)\[[8](https://arxiv.org/html/2605.06951#bib.bib13)\]and Inverse Constrained Reinforcement Learning \(ICRL\)\[[7](https://arxiv.org/html/2605.06951#bib.bib21)\]across four evaluation metrics\.

Constraint mean squared error \(CMSE\)is computed as the mean squared error between the true constraint function and the recovered constraint function\. Run\-timerepresents the execution or computational time required by the technique, including training and inferring the constraints\. Trajectory Typesindicates whether the model handles "Homogeneous" or "Heterogeneous" data\. Preference Learningis a boolean metric indicating whether the technique utilizes preference learning \("Yes" or "No"\)\. We measured these performace matrices over a5×55\\times 5gridworld and results are summarized in Table\-[1](https://arxiv.org/html/2605.06951#S4.T1)\. Our proposed MOCI framework demonstrates superior predictive accuracy, achieving a Mean Squared Error \(MSE\) of 0\.027, which represents a substantial improvement over the baseline techniques MLCI \(0\.25\) and ICRL \(0\.36\)\. Although MLCI remains the most computationally efficient model with a run\-time of 0\.2927, MOCI maintains a highly competitive run\-time of 0\.6917\. This is because in MLCI, there are no explicit weight updates during the constraint inference loop\. The nominal MDP has the expert’s reward function completely knowna priori\. The only thing that changes is the set of forbidden states\. On the other hand, MOCI maintains a list of weights and it performs gradient descent to actively update the reward parameters for each cluster based on the responsibilities calculated in the each step\. ICRL has a comparatively longer run\-time \(8\.90\), mainly due to its neural network–based soft parameterization of the indicator set over constrained trajectories\. Furthermore, unlike MLCI and ICRL, which are restricted to homogeneous trajectory types and lack preference learning capabilities, MOCI is uniquely able to process heterogeneous trajectories while explicitly incorporating preference learning\. These advantages highlight MOCI’s enhanced flexibility and robustness without sacrificing computational efficiency\.

## 6Conclusion

In this study, we presented MOCI, a novel Multi\-Objective Constraint Inference framework that successfully addresses the limitations of current constraint learning techniques\. By leveraging preference learning, MOCI is uniquely process heterogeneous trajectory data, allowing it to accurately capture diverse expert behaviors and nuanced multi\-objective trade\-offs\. Our comparative analysis reveals that MOCI achieves superior predictive accuracy, reducing the Mean Squared Error to 0\.027\. This is a substantial improvement over established baselines\. Crucially, MOCI achieves this high accuracy while bypassing the severe computational overhead associated with iterative reinforcement learning loops, resulting in highly competitive execution times\. These findings highlight MOCI’s potential as a scalable and robust tool for safe reinforcement learning in complex environments where expert data is varied and multifaceted\.

Limitations:

1. 1\.MOCI currently requires the hyperparameterKK\(the number of distinct expert types \) to be defined a priori\. In real\-world, unannotated datasets, the exact number of heterogeneous behaviors is rarely known in advance\. IfKKis set incorrectly, MOCI may either force distinct behaviors into a single distorted reward function or split a single behavior into redundant clusters\.
2. 2\.The MOCI focuses strictly on hard constraints \(states that are absolutely forbidden\)\. It does not natively support the inference of soft constraints \(e\.g\., a speed limit that an expert might occasionally violate but generally avoids\) or probabilistic chance constraints\.

Future work:

1. 1\.To eliminate the need for a predefined number of expert types, the EM clustering step could be replaced with Bayesian non\-parametric methods, such as a Dirichlet Process Mixture Model \(DPMM\)\[[9](https://arxiv.org/html/2605.06951#bib.bib20)\]\. This would allow the algorithm to automatically deduce the optimal number of distinct expert profiles \(KK\) directly from the complexity and diversity of the demonstration data\.
2. 2\.This work can be extended to real\-world healthcare applications\. In cancer treatment, patients are typically subject to shared clinical constraints, such as treatment guidelines, toxicity limits, and eligibility criteria for specific therapies \(e\.g\., chemotherapy, radiotherapy\)\. However, individual preferences can vary substantially\. Some patients may prioritize prolonging survival, even at the cost of aggressive treatments and severe side effects, while others may prefer to preserve quality of life and opt for less intensive treatments\. Furthermore, treatment effects are highly heterogeneous\[[17](https://arxiv.org/html/2605.06951#bib.bib12)\], as patients may respond differently to the same therapy due to clinical, biological, and personal factors\. By leveraging MOCI to learn shared constraints, we aim to optimize treatment strategies that align with individual patient preferences while adhering common constraints and safety boundaries\.

AcknowledgmentsThis work has been partly supported by the PersOn project \(P21\-03\) and the Hybrid Intelligence center \(grant no\. 024\.004\.022\), both funded by Nederlandse Organisatie voor Wetenschappelijk Onderzoek \(NWO\)\.

## References

- \[1\]\(2022\)A survey of inverse reinforcement learning\.Artificial Intelligence Review55\(6\),pp\. 4307–4346\.Cited by:[§2\.3](https://arxiv.org/html/2605.06951#S2.SS3.p2.5)\.
- \[2\]S\. Arora and P\. Doshi\(2021\)A survey of inverse reinforcement learning: challenges, methods and progress\.Artificial Intelligence297,pp\. 103500\.Cited by:[§2\.3](https://arxiv.org/html/2605.06951#S2.SS3.p1.3)\.
- \[3\]L\. Barrett and S\. Narayanan\(2008\)Learning all optimal policies with multiple criteria\.InProceedings of the 25th international conference on Machine learning,pp\. 41–47\.Cited by:[§2\.2](https://arxiv.org/html/2605.06951#S2.SS2.p1.4)\.
- \[4\]M\. Hamidi, M\. Sheikhalishahi, and F\. Martinelli\(2018\)Privacy preserving expectation maximization \(em\) clustering construction\.InInternational Symposium on Distributed Computing and Artificial Intelligence,pp\. 255–263\.Cited by:[§3\.1](https://arxiv.org/html/2605.06951#S3.SS1.p4.1)\.
- \[5\]C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz,et al\.\(2022\)A practical guide to multi\-objective reinforcement learning and planning: cf hayes et al\.\.Autonomous Agents and Multi\-Agent Systems36\(1\),pp\. 26\.Cited by:[§2\.2](https://arxiv.org/html/2605.06951#S2.SS2.p1.4)\.
- \[6\]K\. Kim, G\. Swamy, Z\. Liu, D\. Zhao, S\. Choudhury, and S\. Z\. Wu\(2023\)Learning shared safety constraints from multi\-task demonstrations\.Advances in Neural Information Processing Systems36,pp\. 5808–5826\.Cited by:[§2\.1](https://arxiv.org/html/2605.06951#S2.SS1.p1.12)\.
- \[7\]S\. Malik, U\. Anwar, A\. Aghasi, and A\. Ahmed\(2021\)Inverse constrained reinforcement learning\.InInternational conference on machine learning,pp\. 7390–7399\.Cited by:[§1](https://arxiv.org/html/2605.06951#S1.p1.1),[Table 1](https://arxiv.org/html/2605.06951#S4.T1.4.1.3.2.1),[§5](https://arxiv.org/html/2605.06951#S5.p1.1)\.
- \[8\]D\. L\. McPherson, K\. C\. Stocking, and S\. S\. Sastry\(2021\)Maximum likelihood constraint inference from stochastic demonstrations\.In2021 IEEE conference on control technology and applications \(CCTA\),pp\. 1208–1213\.Cited by:[§1](https://arxiv.org/html/2605.06951#S1.p1.1),[Table 1](https://arxiv.org/html/2605.06951#S4.T1.4.1.2.1.1),[§5](https://arxiv.org/html/2605.06951#S5.p1.1)\.
- \[9\]S\. Niekum and A\. Barto\(2011\)Clustering via dirichlet process mixture models for portable skill discovery\.Advances in neural information processing systems24\.Cited by:[item 1](https://arxiv.org/html/2605.06951#S6.I2.i1.p1.1)\.
- \[10\]M\. Qadri, G\. Swamy, J\. Francis, M\. Kaess, and A\. Bajcsy\(2025\)Your learned constraint is secretly a backward reachable tube\.Reinforcement Learning Journal6,pp\. 478–492\.Cited by:[§2\.1](https://arxiv.org/html/2605.06951#S2.SS1.p1.12)\.
- \[11\]A\. Schlaginhaufen and M\. Kamgarpour\(2023\)Identifiability and generalizability in constrained inverse reinforcement learning\.InInternational conference on machine learning,pp\. 30224–30251\.Cited by:[§2\.1](https://arxiv.org/html/2605.06951#S2.SS1.p1.12)\.
- \[12\]D\. R\. Scobee and S\. S\. Sastry\(2019\)Maximum likelihood constraint inference for inverse reinforcement learning\.arXiv preprint arXiv:1909\.05477\.Cited by:[§1](https://arxiv.org/html/2605.06951#S1.p1.1)\.
- \[13\]S\. I\. H\. Shah, A\. Coronato, M\. Naeem, and G\. De Pietro\(2022\)Learning and assessing optimal dynamic treatment regimes through cooperative imitation learning\.IEEE Access10,pp\. 78148–78158\.Cited by:[§1](https://arxiv.org/html/2605.06951#S1.p1.1)\.
- \[14\]S\. I\. H\. Shah, G\. De Pietro, G\. Paragliola, and A\. Coronato\(2023\)Projection based inverse reinforcement learning for the analysis of dynamic treatment regimes\.Applied Intelligence53\(11\),pp\. 14072–14084\.Cited by:[§2\.2](https://arxiv.org/html/2605.06951#S2.SS2.p2.3)\.
- \[15\]R\. Singh, A\. Gupta, and N\. B\. Shroff\(2022\)Learning in constrained markov decision processes\.IEEE Transactions on Control of Network Systems10\(1\),pp\. 441–453\.Cited by:[§2\.1](https://arxiv.org/html/2605.06951#S2.SS1.p1.12)\.
- \[16\]A\. Wachi and Y\. Sui\(2020\)Safe reinforcement learning in constrained markov decision processes\.InInternational Conference on Machine Learning,pp\. 9797–9806\.Cited by:[§2\.1](https://arxiv.org/html/2605.06951#S2.SS1.p1.12)\.
- \[17\]H\. Ye\(2025\)Deep reinforcement learning\-driven efficacy\-toxicity balance optimization strategy for personalized drug combination in cancer patients\.Journal of Science, Innovation & Social Impact1\(1\),pp\. 307–317\.Cited by:[item 2](https://arxiv.org/html/2605.06951#S6.I2.i2.p1.1)\.
- \[18\]S\. Zeng, C\. Li, A\. Garcia, and M\. Hong\(2022\)Maximum\-likelihood inverse reinforcement learning with finite\-time guarantees\.Advances in Neural Information Processing Systems35,pp\. 10122–10135\.Cited by:[§1](https://arxiv.org/html/2605.06951#S1.p1.1)\.
- \[19\]Y\. Zhong, J\. G\. Kuba, X\. Feng, S\. Hu, J\. Ji, and Y\. Yang\(2024\)Heterogeneous\-agent reinforcement learning\.Journal of Machine Learning Research25\(32\),pp\. 1–67\.Cited by:[§2\.2](https://arxiv.org/html/2605.06951#S2.SS2.p2.6)\.
- \[20\]B\. D\. Ziebart, A\. L\. Maas, J\. A\. Bagnell, A\. K\. Dey,et al\.\(2008\)Maximum entropy inverse reinforcement learning\.\.InAaai,Vol\.8,pp\. 1433–1438\.Cited by:[§2\.4](https://arxiv.org/html/2605.06951#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2605.06951#S3.SS1.p2.5)\.

## Appendices

## Appendix ANotations and description

Table\-[2](https://arxiv.org/html/2605.06951#A1.T2)summarizes the notation used in this paper along with their descriptions\.

Table 2:Notations and Descriptions for Multi\-Objective Constraint Inference \(MOCI\)
## Appendix BConstraint Muti\-Objective Maximum Entropy IRL

Letξ\\xidenote a trajectory andf​\(ξ\)∈ℝmf\(\\xi\)\\in\\mathbb\{R\}^\{m\}its associated feature vector encodingmmobjectives\. For each expert clusterkk, preferences over objectives are represented by a weight vectorwk∈ℝmw\_\{k\}\\in\\mathbb\{R\}^\{m\}\. The trajectory\-level reward is defined as

Rwk​\(ξ\)=wk⊤​f​\(ξ\)\.R\_\{w\_\{k\}\}\(\\xi\)=w\_\{k\}^\{\\top\}f\(\\xi\)\.\(21\)
We assume the presence of a set of hard constraintsCCsuch that any trajectory violating these constraints is considered infeasible\. This is encoded using an indicator function

𝕀C​\(ξ\)=\{1,if​ξ​satisfies all constraints in​C,0,otherwise\.\\mathbb\{I\}^\{C\}\(\\xi\)=\\begin\{cases\}1,&\\text\{if \}\\xi\\text\{ satisfies all constraints in \}C,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(22\)
The goal of Maximum Entropy Inverse Reinforcement Learning is to find a probability distributionP​\(ξ∣C,wk\)P\(\\xi\\mid C,w\_\{k\}\)over feasible trajectories that maximizes entropy while matching the empirical feature expectations of expert demonstrations\. Formally, we solve

maxP​\(ξ\)\\displaystyle\\max\_\{P\(\\xi\)\}\\quad−∑ξP​\(ξ\)​log⁡P​\(ξ\)\\displaystyle\-\\sum\_\{\\xi\}P\(\\xi\)\\log P\(\\xi\)\(23\)s\.t\.∑ξP​\(ξ\)=1,\\displaystyle\\sum\_\{\\xi\}P\(\\xi\)=1,\(24\)∑ξP​\(ξ\)​f​\(ξ\)=f^,\\displaystyle\\sum\_\{\\xi\}P\(\\xi\)f\(\\xi\)=\\hat\{f\},\(25\)P​\(ξ\)=0if​𝕀C​\(ξ\)=0\.\\displaystyle P\(\\xi\)=0\\quad\\text\{if \}\\mathbb\{I\}^\{C\}\(\\xi\)=0\.\(26\)
Introducing Lagrange multipliersλ\\lambdaandwkw\_\{k\}for the normalization and feature\-matching constraints, the Lagrangian becomes

ℒ=−∑ξP​\(ξ\)​log⁡P​\(ξ\)\+λ​\(∑ξP​\(ξ\)−1\)\+wk⊤​\(∑ξP​\(ξ\)​f​\(ξ\)−f^\)\.\\mathcal\{L\}=\-\\sum\_\{\\xi\}P\(\\xi\)\\log P\(\\xi\)\+\\lambda\\left\(\\sum\_\{\\xi\}P\(\\xi\)\-1\\right\)\+w\_\{k\}^\{\\top\}\\left\(\\sum\_\{\\xi\}P\(\\xi\)f\(\\xi\)\-\\hat\{f\}\\right\)\.\(27\)
Taking the derivative ofℒ\\mathcal\{L\}with respect toP​\(ξ\)P\(\\xi\)and setting it to zero yields

−log⁡P​\(ξ\)−1\+λ\+wk⊤​f​\(ξ\)=0\.\-\\log P\(\\xi\)\-1\+\\lambda\+w\_\{k\}^\{\\top\}f\(\\xi\)=0\.\(28\)
Solving forP​\(ξ\)P\(\\xi\)gives

P​\(ξ\)=exp⁡\(wk⊤​f​\(ξ\)\)​exp⁡\(λ−1\)\.P\(\\xi\)=\\exp\\\!\\big\(w\_\{k\}^\{\\top\}f\(\\xi\)\\big\)\\exp\(\\lambda\-1\)\.\(29\)
The normalization constant, or partition function, is defined over feasible trajectories as

Z​\(C,wk\)=∑ξ:𝕀C​\(ξ\)=1exp⁡\(Rwk​\(ξ\)\)\.Z\(C,w\_\{k\}\)=\\sum\_\{\\xi:\\mathbb\{I\}^\{C\}\(\\xi\)=1\}\\exp\\\!\\big\(R\_\{w\_\{k\}\}\(\\xi\)\\big\)\.\(30\)
Substitutingexp⁡\(λ−1\)=1/Z​\(C,wk\)\\exp\(\\lambda\-1\)=1/Z\(C,w\_\{k\}\)yields the final form:

P​\(ξ∣C,wk\)=1Z​\(C,wk\)​exp⁡\(Rwk​\(ξ\)\),P\(\\xi\\mid C,w\_\{k\}\)=\\frac\{1\}\{Z\(C,w\_\{k\}\)\}\\exp\\\!\\big\(R\_\{w\_\{k\}\}\(\\xi\)\\big\),\(31\)for allξ\\xisatisfying the constraints, andP​\(ξ∣C,wk\)=0P\(\\xi\\mid C,w\_\{k\}\)=0otherwise\. The idea is taken from Maximum Entropy Inverse Reinforcement Learning by Zeibert et al\.

## Appendix CSensitivity analysis of MOCI

The magnitude of the change in log\-likelihood,Δ​L\\Delta L, upon adding a candidate state to the constraint setCCis intrinsically dependent on three environmental factors:

- •Dataset Size \(\|D\|\|D\|\):The total log\-likelihoodLLis an unnormalized sum over all trajectories in the expert dataset\. Consequently, the magnitude ofΔ​L\\Delta Lscales linearly with the number of expert demonstrations\.
- •State Space Complexity:In environments with limited pathways \(e\.g\., a heavily constrained Gridworld\), removing a single valid state drastically reduces the partition functionZZ\. Conversely, in highly redundant state spaces, the reduction inZZis minimal unless the state represents a critical navigational chokepoint\.
- •State Centrality:Unvisited states located on optimal paths yield a massiveΔ​L\\Delta Lwhen constrained, whereas unvisited states in peripheral or suboptimal regions yield negligible changes to the partition function\.

### C\.1Ablation: latent mixture versus a single preference model

Heterogeneous demonstrations are modeled as a mixture overKKlatent linear preferences with shared hard constraints\. A natural baseline is to*remove*mixture capacity by settingK=1K\{=\}1, i\.e\., to explain the same pooled trajectories with a single MaxEnt component while retaining the remainder of the MOCI procedure \(EM updates, constraint search, and the same hyperparameters otherwise\)\. We instantiate MOCI withK∈\{1,2\}K\{\\in\}\\\{1,2\\\}on the*same*unlabeled dataset𝒟\\mathcal\{D\}for each random demo seed: trajectories are drawn i\.i\.d\. from two expert policies with distinct terrain preferences, then pooled and shuffled\. This isolates whether mixture structure materially affects fit and constraint recovery when the generative process truly uses two preference types\.

Table[3](https://arxiv.org/html/2605.06951#A3.T3)summarizes means±\\pmstandard deviations over5050independent demo seeds on the heterogeneous GridWorld protocol described above \(dDKL=0\.05d\_\{\\mathrm\{DKL\}\}\{=\}0\.05,1010EM iterations per run\)\. We report average marginal log\-likelihood per trajectory under the fitted model, constraint mean squared error \(CMSE\), precision and recall for forbidden states, their harmonic mean \(F1\), and false positive rate \(FPR\)\.

Table 3:Pooled heterogeneous demonstrations: single\-preference MOCI \(K=1K\{=\}1\) versus mixture MOCI \(K=2K\{=\}2\)\. Means±\\pmstd\. over5050seeds\.The ablation supports the hypothesis that mixture structure improves*explanation*of pooled heterogeneous data:K=2K\{=\}2achieves a systematic, statistically significant lift in marginal likelihood relative toK=1K\{=\}1\. Effects on overlap\-based constraint scores are directionally favorable forK=2K\{=\}2\. Accordingly, we treatK=1K\{=\}1primarily as a likelihood\-controlled sanity check on mixture capacity rather than as a substitute for the full model in heterogeneous regimes\.

## Appendix DMOCI Algorithm

The MOCI algorithm\-[1](https://arxiv.org/html/2605.06951#algorithm1)employs an Expectation\-Maximization \(EM\) style approach to infer shared constraints and multi\-objective rewards jointly\. During the E\-step, it calculates the responsibility of each reward cluster for the observed expert trajectories\. The M\-step then updates the cluster priors, optimizes the reward weights for each objective via Maximum Entropy Inverse Reinforcement Learning, and greedily adds candidate constraints from unvisited states as long as they significantly improve the data likelihood without falling below the divergence thresholddD​K​Ld\_\{DKL\}\.

Input:MDPℳ\\mathcal\{M\}, Demonstrations𝒟=\{ξ1,…,ξN\}\\mathcal\{D\}=\\\{\\xi\_\{1\},\\dots,\\xi\_\{N\}\\\}, Number of clustersKK, thresholddD​K​Ld\_\{DKL\}, Priors\{πk\}k=1K\\\{\\pi\_\{k\}\\\}\_\{k=1\}^\{K\}, Learning rateα\\alpha

Output:Inferred constraints

CC, Reward weights

\{wk\}k=1K\\\{w\_\{k\}\\\}\_\{k=1\}^\{K\}
1

21\.5mm

31\.5mmInitialization:

4

C←∅C\\leftarrow\\emptyset
𝒞c​a​n​d←\{s∈𝒮∣s∉ξ​for all​ξ∈𝒟\}\\mathcal\{C\}\_\{cand\}\\leftarrow\\\{s\\in\\mathcal\{S\}\\mid s\\notin\\xi\\text\{ for all \}\\xi\\in\\mathcal\{D\}\\\}
//Unvisited states

5Initialize

wkw\_\{k\}randomly,

πk←1/K\\pi\_\{k\}\\leftarrow 1/Kfor

k∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}
6

72mm

8repeat

9

1\.5mm/\*E\-Step: Evaluate Responsibilities\*/

10for*k=1k=1toKK*do

11Compute partition function

Z​\(C,wk\)Z\(C,w\_\{k\}\)via backward pass

12

13end for

14for*i=1i=1toNN*do

15for*k=1k=1toKK*do

16

γi,k←πk​P​\(ξi∣C,wk\)∑j=1Kπj​P​\(ξi∣C,wj\)\\gamma\_\{i,k\}\\leftarrow\\frac\{\\pi\_\{k\}P\(\\xi\_\{i\}\\mid C,w\_\{k\}\)\}\{\\sum\_\{j=1\}^\{K\}\\pi\_\{j\}P\(\\xi\_\{i\}\\mid C,w\_\{j\}\)\}
17

18end for

19

20end for

21

222mm

/\*M\-Step A: Update Priors\*/

23for*k=1k=1toKK*do

24

πk←1N​∑i=1Nγi,k\\pi\_\{k\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\gamma\_\{i,k\}
25

26end for

27

282mm

/\*M\-Step B: Update Reward Weights \(MaxEnt IRL\)\*/

29for*k=1k=1toKK*do

30

∇wkℒ←∑i=1Nγi,k​\(ϕ​\(ξi\)−𝔼P​\(ξ∣C,wk\)​\[ϕ​\(ξ\)\]\)\\nabla\_\{w\_\{k\}\}\\mathcal\{L\}\\leftarrow\\sum\_\{i=1\}^\{N\}\\gamma\_\{i,k\}\\left\(\\phi\(\\xi\_\{i\}\)\-\\mathbb\{E\}\_\{P\(\\xi\\mid C,w\_\{k\}\)\}\[\\phi\(\\xi\)\]\\right\)
31

wk←wk\+α​∇wkℒw\_\{k\}\\leftarrow w\_\{k\}\+\\alpha\\nabla\_\{w\_\{k\}\}\\mathcal\{L\}
32end for

33

342mm

/\*M\-Step C: Update Shared Constraints\*/

35while*𝒞c​a​n​d≠∅\\mathcal\{C\}\_\{cand\}\\neq\\emptyset*do

36

c∗←arg⁡maxc∈𝒞c​a​n​d⁡ℒ​\(C∪\{c\},\{wk\},\{πk\}\)c^\{\*\}\\leftarrow\\arg\\max\_\{c\\in\\mathcal\{C\}\_\{cand\}\}\\mathcal\{L\}\(C\\cup\\\{c\\\},\\\{w\_\{k\}\\\},\\\{\\pi\_\{k\}\\\}\)
37

Δ​ℒ←ℒ​\(C∪\{c∗\},⋅\)−ℒ​\(C,⋅\)\\Delta\\mathcal\{L\}\\leftarrow\\mathcal\{L\}\(C\\cup\\\{c^\{\*\}\\\},\\cdot\)\-\\mathcal\{L\}\(C,\\cdot\)
38if*Δ​ℒ≤dD​K​L\\Delta\\mathcal\{L\}\\leq d\_\{DKL\}*then

break

//Stop adding constraints to prevent overfitting

39

40else

41

C←C∪\{c^\}C\\leftarrow C\\cup\\\{\\hat\{c\}\\\}
42

𝒞c​a​n​d←𝒞c​a​n​d∖\{c^\}\\mathcal\{C\}\_\{cand\}\\leftarrow\\mathcal\{C\}\_\{cand\}\\setminus\\\{\\hat\{c\}\\\}
43

44end if

45

46end while

47

48until*convergence ofℒ​\(C,\{wk\},\{πk\}\)\\mathcal\{L\}\(C,\\\{w\_\{k\}\\\},\\\{\\pi\_\{k\}\\\}\)*

Algorithm 1Multi\-Objective Constraint Inference \(MOCI\)