Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization

# Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization
Source: [https://arxiv.org/html/2607.03140](https://arxiv.org/html/2607.03140)
11institutetext:Josef Ressel Centre for Intelligent and Secure Industrial Automation,
Salzburg University of Applied Sciences, Salzburg, Austria22institutetext:Department of Artificial Intelligence and Human Interfaces,
Paris Lodron University of Salzburg, Salzburg, Austria
22email:georg\.schaefer@fh\-salzburg\.ac\.at###### Abstract

Industrial automation increasingly demands control strategies that balance operational performance with strict energy efficiency requirements\. A common approach to solving this multi\-objective problem, particularly within the framework of reinforcement learning \(RL\), is to formulate a single, scalar reward function that linearly combines the competing objectives\. However, the manual weighting of these different objectives is heavily reliant on domain intuition, incredibly time\-consuming, prone to human bias, and frequently fails to uncover optimal trade\-off solutions\. This work addresses the critical challenge of automating the weight selection process to systematically and efficiently discover the Pareto front of optimal trade\-off policies\. We formulate the weight selection process as a multi\-objective Bayesian optimization \(MOBO\)problem and evaluate its sample efficiency against a standard uniform grid search baseline\. Using a physical Quanser Aero 2 testbed configured for 1\-Degree\-of\-Freedom \(DoF\)pitch control, our results demonstrate that the MOBOapproach, utilizing the Expected Hypervolume Improvement \(qEHVI\) acquisition function, consistently outperforms uniform grid sampling\. MOBOachieves superior hypervolume and maximum spread, successfully identifying high\-quality, diverse trade\-off policies with a reduced evaluation budget, thereby enabling highly efficient energy\-aware control in complex mechatronic systems\.

## 1Introduction

The rapid modernization of industrial automation has brought forward a critical design paradigm: systems must no longer be optimized solely for operational performance, but must concurrently adhere to strict energy efficiency and sustainability constraints\[[2](https://arxiv.org/html/2607.03140#bib.bib9)\]\. Traditional control methodologies, while robust, often struggle to adapt to the highly nonlinear dynamics, complex environmental interactions, and shifting objectives present in modern mechatronic systems\. Consequently, RLhas emerged as a powerful, data\-driven framework capable of learning optimal control policies through continuous interaction with an environment\[[6](https://arxiv.org/html/2607.03140#bib.bib10)\]\. Indeed, recent empirical evaluations have demonstrated that policy gradient methods can offer highly competitive adaptability and rapid response characteristics when benchmarked against traditional Model Predictive Control \(MPC\) on non\-linear mechatronic systems\[[15](https://arxiv.org/html/2607.03140#bib.bib39)\]\.

However, integrating energy efficiency into the RLframework fundamentally transforms the control task into a multi\-objective optimization problem\. The agent must balance at least two inherently conflicting goals: maximizing primary operational performance, such as precise setpoint regulation, dynamic trajectory tracking, or process throughput, and minimizing the physical energy expended during actuation\. Traditional approaches frequently treat energy consumption as a secondary heuristic via fixed, manually tuned reward penalties\. While our approach utilizes a similar linear scalarization to combine competing objectives, we address the critical limitation of manual tuning by formulating a framework that automates the discovery of the optimal trade\-off surface\[[13](https://arxiv.org/html/2607.03140#bib.bib4)\]\.

In industrial practice, this selection is predominantly achieved through manual tuning\. A control engineer intuitively selects a weight, trains a complete RLagent, evaluates the physical performance, and iteratively adjusts the weight based on the observed outcome\. This methodology is incredibly time\-consuming, highly susceptible to human bias, and frequently fails to uncover the true Pareto front, i\.e\., the set of optimal solutions where no single objective can be improved without degrading another\[[4](https://arxiv.org/html/2607.03140#bib.bib13)\]\. Because training an RLagent on physical hardware or high\-fidelity simulators is computationally expensive and subjects machinery to physical wear, there is a pressing need for sample\-efficient automation of this process\.

To address these challenges, this paper proposes an automated framework for modeling the optimal trade\-off surface in energy\-aware continuous control\. Our main contributions are as follows:

- •We formulate the multi\-objective reward weighting process as a black\-box optimization problem and apply MOBOto automate the discovery of the Pareto front\.
- •We conduct a rigorous empirical comparison between the proposed MOBOapproach and a standard uniform grid search baseline, utilizing a high\-fidelity environment of a Quanser Aero 2 mechatronic testbed\.
- •We demonstrate that MOBOachieves superior Pareto front approximations across multiple quantitative metrics \(hypervolume, spacing, maximum spread\) while drastically reducing the required computational budget and environment interactions\.

The remainder of this paper is structured as follows\.[Section˜2](https://arxiv.org/html/2607.03140#S2)provides the necessary theoretical background and discusses related work in the fields of multi\-objective RL and Bayesian optimization\.[Section˜3](https://arxiv.org/html/2607.03140#S3)details the system architecture of the mechatronic testbed and formally defines the reinforcement learning problem\.[Section˜4](https://arxiv.org/html/2607.03140#S4)outlines our implementation and evaluation methodology\. Finally,[Section˜5](https://arxiv.org/html/2607.03140#S5)discusses the empirical results, followed by concluding remarks and directions for future research in[Section˜6](https://arxiv.org/html/2607.03140#S6)\.

## 2Background and Related Work

### 2\.1Multi\-Objective Reinforcement Learning

Standard RLoperates on the premise of maximizing a single scalar reward signal, traditionally modeled as a Markov decision process \(MDP\)\[[20](https://arxiv.org/html/2607.03140#bib.bib11)\]\. However, industrial applications inherently involve multiple conflicting feedback signals\. As Roijers et al\.\[[13](https://arxiv.org/html/2607.03140#bib.bib4)\]establish, converting a multi\-objective problem into a single\-objective one via fixed scalarization is often undesirable when the end\-user’s specific performance\-versus\-energy preferences are unknown a priori\. In such scenarios, the objective shifts from finding a single optimal policy to identifying the set of Pareto\-optimal policies\. Recent literature emphasizes that explicitly multi\-objective methods are required to solve complex planning problems efficiently\[[5](https://arxiv.org/html/2607.03140#bib.bib5)\]\. Let the performance of a policyπ\\pibe evaluated by a vector of objective functions𝐉​\(π\)=\[J1​\(π\),J2​\(π\),…,Jk​\(π\)\]T\\mathbf\{J\}\(\\pi\)=\[J\_\{1\}\(\\pi\),J\_\{2\}\(\\pi\),\\dots,J\_\{k\}\(\\pi\)\]^\{T\}\. A policyπ∗\\pi^\{\*\}is considered Pareto optimal if there does not exist another policyπ\\pisuch thatJi​\(π\)≤Ji​\(π∗\)J\_\{i\}\(\\pi\)\\leq J\_\{i\}\(\\pi^\{\*\}\)for all objectivesii, andJj​\(π\)<Jj​\(π∗\)J\_\{j\}\(\\pi\)<J\_\{j\}\(\\pi^\{\*\}\)for at least one objectivejj\[[8](https://arxiv.org/html/2607.03140#bib.bib14)\]\. The set of all such non\-dominated policy evaluations forms the Pareto front\.

### 2\.2Energy\-Aware Control in Industrial Automation

In modern industrial robotics, balancing system performance, such as precise trajectory tracking or manipulation, with energy efficiency is critical\. Traditional approaches frequently treat energy consumption as a secondary heuristic or a simple reward penalty\. However, sustainable operation demands that actuation energy is explicitly regulated\. Recent frameworks address this by treating the task as a Constrained MDPor a multi\-objective problem, seeking to minimize long\-term energy budgets while preserving strict operational reliability\[[7](https://arxiv.org/html/2607.03140#bib.bib8)\]\. Our approach aligns with this paradigm by directly modeling the trade\-off between tracking error and power consumption, focusing specifically on automating the discovery of the optimal trade\-off surface\.

### 2\.3Bayesian Optimization for Pareto Front Modeling

To map the Pareto front without exhaustively training RLpolicies for every possible weight combination, sample\-efficient black\-box optimization is necessary\. Bayesian optimization \(BO\)\[[19](https://arxiv.org/html/2607.03140#bib.bib15)\]has become a standard approach for optimizing expensive functions, utilizing a Gaussian process \(GP\)\[[12](https://arxiv.org/html/2607.03140#bib.bib16)\]surrogate model to approximate the objective landscape and quantify uncertainty\. In the multi\-objective context \(MOBO\), Gaussian Processes are combined with specialized acquisition functions, such as Expected Hypervolume Improvement \(qEHVI\)\[[3](https://arxiv.org/html/2607.03140#bib.bib17)\], to intelligently select query points that maximally expand the non\-dominated solution set\. Recent advancements focus on efficiently evaluating Pareto\-frontier entropy\[[21](https://arxiv.org/html/2607.03140#bib.bib6)\]and selecting diverse batches of parameters to ensure broad coverage\[[1](https://arxiv.org/html/2607.03140#bib.bib7)\]\. Our work directly applies these MOBOprinciples to automate the hyperparameter weighting of an RLreward function\.

## 3System Architecture and Problem Formulation

### 3\.1The Quanser Aero 2 Testbed

To rigorously validate our approach, we require a mechatronic system featuring non\-linear dynamics and measurable physical energy states\. We utilize the Quanser Aero 2 system \([Fig\.˜1](https://arxiv.org/html/2607.03140#S3.F1)\)\. For the scope of this evaluation, the testbed is configured in a 1\-DoFsetup: the yaw axis is mechanically locked, while the pitch axis remains unlocked and free to rotate\.

![Refer to caption](https://arxiv.org/html/2607.03140v1/figures/aero2.png)Figure 1:The Quanser Aero 2 mechatronic testbed\. For this study, the system is configured in a 1\-DoF mode where the yaw axis is locked and the pitch axis is controlled via the dual DC motors\.The system is actuated by two DC motors equipped with propellers\. Letu0u\_\{0\}andu1u\_\{1\}represent the voltage signals applied to the respective motors\. In the 1\-DoFpitch configuration, the control inputuuis symmetric and defined asu=u0=−u1u=u\_\{0\}=\-u\_\{1\}\. The primary outputy=Θy=\\Thetarepresents the pitch angle of the main beam\. The system dynamics are inherently non\-linear due to the aerodynamic drag, friction, and thrust characteristics of the propellers, providing an established, highly representative benchmark for evaluating advanced learning\-based control algorithms\[[15](https://arxiv.org/html/2607.03140#bib.bib39)\]\.

### 3\.2State and Action Space Formulation

The task assigned to the Proximal Policy Optimization \(PPO\)agent\[[18](https://arxiv.org/html/2607.03140#bib.bib12)\]is to orient the beam to a desired, time\-varying target angle while minimizing the electrical power consumed by the DC motors\. The state space𝒮\\mathcal\{S\}is fully observable and represented as a continuous vector containing the current pitch angle, the angular velocity, and the current target reference angle:

st=\[Θt,Θ˙t,Θtr​e​f\]Ts\_\{t\}=\\left\[\\Theta\_\{t\},\\dot\{\\Theta\}\_\{t\},\\Theta^\{ref\}\_\{t\}\\right\]^\{T\}\(1\)The action space𝒜\\mathcal\{A\}comprises the continuous voltage signal applied to the motors, bounded by the physical limits of the hardware:

at=\[ut\]∈\[−24​V,24​V\]a\_\{t\}=\[u\_\{t\}\]\\in\[\-24\\text\{V\},24\\text\{V\}\]\(2\)

### 3\.3Reward Function Design

The behavior of the agent is entirely dictated by the parameterα\\alphain the scalarized reward function\. At each time steptt, the rewardRtR\_\{t\}is computed as:

Rt=−\(1−α\)⋅\|Δt\|−α⋅PtR\_\{t\}=\-\(1\-\\alpha\)\\cdot\|\\Delta\_\{t\}\|\-\\alpha\\cdot P\_\{t\}\(3\)To ensure stable neural network updates and meaningful scalarization, the termsΔt\\Delta\_\{t\}\(absolute tracking error\) andPtP\_\{t\}\(power consumption\) are normalized to a\[0,1\]\[0,1\]scale\. As demonstrated in our prior investigations regarding the formulation of RL environments, strict normalization of observation spaces and actions is a crucial prerequisite for stable policy convergence and the successful transfer of simulated policies to the physical system\[[14](https://arxiv.org/html/2607.03140#bib.bib48)\]\. The power consumptionPtP\_\{t\}is calculated dynamically based on the applied voltage and the internal resistance characteristics of the Aero 2 motors\. By manipulatingα∈\[0,1\]\\alpha\\in\[0,1\]in our previous work\[[17](https://arxiv.org/html/2607.03140#bib.bib53)\], we observed that the problem spans the entire spectrum from maximum tracking performance to maximum energy conservation, validating its use as the core tunable hyperparameter\.

## 4Methodology

### 4\.1Software and Implementation Details

The RLenvironment was built as a high\-fidelity digital twin of the physical Quanser Aero 2 system to ensure that all trained policies are highly deployable\. This environment utilizes our previously established Python\-Simulink co\-simulation framework\[[16](https://arxiv.org/html/2607.03140#bib.bib38)\], which enables the seamless integration of numerically robust Simulink models with flexible Python\-based machine learning environments\. The PPO\[[18](https://arxiv.org/html/2607.03140#bib.bib12)\]agent was implemented utilizing the Stable Baselines3 framework\[[11](https://arxiv.org/html/2607.03140#bib.bib1)\], backed by PyTorch\[[10](https://arxiv.org/html/2607.03140#bib.bib3)\]for automatic differentiation and neural network optimization\. Both the actor and critic networks feature continuous multi\-layer perceptron \(MLP\) architectures\.

### 4\.2MOBO\-RL Integration Workflow

To automate the discovery of the Pareto front, we integrate the RLtraining process within an outer\-loop optimization framework managed by the BoTorch/Ax library\[[9](https://arxiv.org/html/2607.03140#bib.bib2)\]\. The optimization is initialized by defining the continuous parameter boundsα∈\[0\.0,1\.0\]\\alpha\\in\[0\.0,1\.0\]and the two minimization objectives: tracking error \(f1f\_\{1\}\) and power consumption \(f2f\_\{2\}\)\.

Stable Baselines3Ax / BoTorch APIMOBO Controller\(GP \+ qEHVI\)Suggests optimal weightα\\alphaRL Training PhasePPO Rollout & UpdateConverge to policyπi\\pi\_\{i\}Policy EvaluationRun validation profile:Measuref1f\_\{1\},f2f\_\{2\}Surrogate UpdateComplete trial via APIUpdate GP posteriorSampleαi\\alpha\_\{i\}TrainedPolicyπi\\pi\_\{i\}Yield\(f1,i,f2,i\)\(f\_\{1,i\},f\_\{2,i\}\)LogDataFigure 2:The proposed MOBO\-RL integration workflow\. The Bayesian optimizer iteratively samples the reward weighting parameterα\\alpha\. For each sampled weight, an RLagent is trained to convergence and its physical performance is evaluated\. The resulting empirical observations update the surrogate model to intelligently guide future sampling\.The structure of this framework is illustrated in[Fig\.˜2](https://arxiv.org/html/2607.03140#S4.F2)\. The surrogate model approximates the objective functions using a GP\. For each sequential trial, the optimizer queries this surrogate model to suggest the next most promisingα\\alphaweight utilizing the Expected Hypervolume Improvement \(qEHVI\)acquisition function\. A fresh PPOagent is instantiated and trained to convergence using this specific scalarized reward formulation\. Following training, the resultant policyπ\\piis deployed in a deterministic evaluation environment to record the true tracking error and electrical power draw\. Finally, these empirical results are passed back to the optimizer via the API, dynamically updating the posterior distribution of the surrogate model and informing the subsequent iteration\.

### 4\.3Exploration Strategies

To approximate the Pareto front, we enforce a strict, heavily constrained maximum budget of 11 full RLtraining runs \(evaluations\)\. This budget simulates the real\-world constraints of hardware training times\. We compare two distinct strategies for exploring the parameter space ofα\\alpha:

- •Uniform Grid Search \(Baseline\):We manually select 11 equally spaced values forα∈\[0,1\]\\alpha\\in\[0,1\]\. While providing simple, uniform coverage of the parameter space, this method is fundamentally inefficient as it cannot adapt to the underlying non\-linear mapping betweenα\\alphaand the bi\-objective performance\.
- •Multi\-Objective BO \(MOBO\):To prevent premature convergence and provide a baseline statistical variance, the MOBOoptimization is initialized with 5 quasi\-random Sobol points\. This is followed by 6 sequentially guided BOselections\. The total remains exactly 11 trials to ensure a mathematically fair comparison against the baseline\.

### 4\.4Quantitative Evaluation Metrics

We quantitatively assess the quality of the generated Pareto fronts using three standard multi\-objective metrics\. Let𝒫\\mathcal\{P\}be the approximated Pareto front\.

1. 1\.Hypervolume \(HV\):Evaluates the total dominated n\-dimensional volume in the objective space between the Pareto solutions and a predefined, worst\-case reference point𝐫\\mathbf\{r\}\. HV​\(𝒫\)=Λ​\(⋃𝐱∈𝒫\[f1​\(𝐱\),r1\]×\[f2​\(𝐱\),r2\]\)\\text\{HV\}\(\\mathcal\{P\}\)=\\Lambda\\left\(\\bigcup\_\{\\mathbf\{x\}\\in\\mathcal\{P\}\}\[f\_\{1\}\(\\mathbf\{x\}\),r\_\{1\}\]\\times\[f\_\{2\}\(\\mathbf\{x\}\),r\_\{2\}\]\\right\)\(4\)whereΛ\\Lambdadenotes the Lebesgue measure\. A larger volume indicates a front that is closer to the true Pareto optimal set and possesses a better spread\.
2. 2\.Spacing \(SS\):Measures the uniformity of the solution distribution along the front\. It is defined as the variance in the Euclidean distance between adjacent solutions: S=1n−1​∑i=1n\(d¯−di\)2S=\\sqrt\{\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\}\(\\bar\{d\}\-d\_\{i\}\)^\{2\}\}\(5\)wheredid\_\{i\}is the distance to the nearest neighbor for theii\-th solution, andd¯\\bar\{d\}is the mean of alldid\_\{i\}\. Lower values indicate a more evenly distributed set of choices\.
3. 3\.Maximum Spread \(M​SMS\):Assesses the maximal range coverage by calculating the Euclidean distance between the extreme boundary points of the discovered front in the objective space\. LetKKbe the number of objectives\. The maximum spread is defined as the Euclidean norm of the objective ranges: M​S=∑k=1K\(max𝐱∈𝒫⁡fk​\(𝐱\)−min𝐱∈𝒫⁡fk​\(𝐱\)\)2MS=\\sqrt\{\\sum\_\{k=1\}^\{K\}\\left\(\\max\_\{\\mathbf\{x\}\\in\\mathcal\{P\}\}f\_\{k\}\(\\mathbf\{x\}\)\-\\min\_\{\\mathbf\{x\}\\in\\mathcal\{P\}\}f\_\{k\}\(\\mathbf\{x\}\)\\right\)^\{2\}\}\(6\)A higherM​SMSvalue indicates that the algorithm successfully explored the absolute outer limits of the performance trade\-off, providing a broader spectrum of operational modes\.

## 5Results and Discussion

### 5\.1Quantitative Results

[Table˜1](https://arxiv.org/html/2607.03140#S5.T1)summarizes the final performance of both exploration strategies after the strict budget of 11 evaluations was exhausted\. The automated MOBOapproach consistently outperforms the uniform grid sampling across all three defined multi\-objective metrics\.

Table 1:Comparison of Pareto front quality metrics after 11 trials\.The higher Hypervolume demonstrates that MOBOdiscovered policies that objectively dominate those found by the uniform grid, as visualized in[Fig\.˜3](https://arxiv.org/html/2607.03140#S5.F3)\. Crucially, the non\-linear relationship between the weightα\\alphaand the actual physical system performance causes a uniform grid search in theparameterspace to map to a highly non\-uniform cluster of points in theobjectivespace\. As observed in our experiments, the Grid Search wasted a significant portion of its 11 evaluations on adjacentα\\alphavalues that resulted in similar physical system behavior, largely failing to map the critical“knee”of the Pareto front where the most viable trade\-offs reside\.

![Refer to caption](https://arxiv.org/html/2607.03140v1/x1.png)Figure 3:Comparison of all sampled configurations between the uniform grid search and MOBO selection, highlighting the discovered Pareto front\.
### 5\.2Sample Efficiency and Progression

The most significant and practical advantage of the MOBOapproach is its remarkable sample efficiency, which directly translates to saved computational time and reduced hardware degradation during physical training\. Analyzing the progression of the hypervolume \([Fig\.˜4](https://arxiv.org/html/2607.03140#S5.F4)\), MOBOrapidly identifies high\-quality solutions\. Because the qEHVI acquisition function explicitly targets areas of the objective space that maximize expected improvement, it intelligently avoids redundant regions of the parameter space\.

![Refer to caption](https://arxiv.org/html/2607.03140v1/x2.png)Figure 4:Progression of hypervolume across the evaluation trials compared to the grid search baseline\.Remarkably, the MOBOapproach achieves the final, maximum hypervolume of the 11\-trial Grid Search baseline in just 4 evaluation trials\. In a real\-world industrial context, where training a single policy to convergence might take hours or days of continuous operation, matching baseline performance in 4 trials represents a reduction in computational and temporal cost of approximately 64%\.

![Refer to caption](https://arxiv.org/html/2607.03140v1/x3.png)Figure 5:Progression of maximum spread across the evaluation trials compared to the grid search baseline\.Similarly, the maximum spread progression \([Fig\.˜5](https://arxiv.org/html/2607.03140#S5.F5)\) highlights the capacity of BOfor active boundary exploration\. The algorithm matched the grid search’s maximum spread by trial 8\. This proves that MOBOdoes not merely exploit a single high\-performing region, but actively pushes the boundaries of the known objective space\. It ensures that fundamentally diverse trade\-off policies, spanning from ultra\-precise, high\-energy trajectory tracking to loose, highly conservative energy\-saving modes, are systematically discovered and made available to the control engineer without requiring tedious manual intervention\.

## 6Conclusion and Future Work

This work rigorously validates Multi\-Objective Bayesian Optimization as a highly effective, sample\-efficient, and automated method for multi\-objective weight selection in industrial RLcontinuous control tasks\. By successfully balancing electrical energy consumption and tracking performance on the Quanser Aero 2 testbed, we demonstrated that MOBOfundamentally eliminates the biases and inefficiencies of manual heuristic weighting\. The approach consistently creates a superior approximation of the Pareto front while significantly reducing the required evaluation budget, marking a critical step toward accessible, sustainable intelligent automation\.

Future work will focus on extending this methodology to the full, highly coupled 2\-DoF\(pitch and yaw\) configuration of the Quanser Aero 2 to handle more complex trajectory tracking tasks\. Furthermore, because exploring the boundaries of the weight parameter space, specifically where tracking performance is heavily prioritized over energy conservation \(α→0\\alpha\\to 0\), can lead to aggressive, potentially hardware\-damaging“bang\-bang”policies, we intend to integrate strict Safe RL constraints into the optimization framework\. This ensures that boundary exploration remains physically safe during early training phases\. Finally, we will investigate the transferability of these findings, utilizing the identified Pareto optimal weights from the Aero 2 to jump\-start and accelerate RLtraining processes on similar, yet mathematically distinct, industrial mechatronic systems\.

#### Acknowledgments

Financial support for this study was provided by the Christian Doppler Research Association \(CDG\) through the Josef Ressel Centre for Intelligent and Secure Industrial Automation, the corresponding WISS Co\-project of Land Salzburg, and by the European Interreg project BA0100172 AI4GREEN\.

## References

- \[1\]A\. Ahmadianshalchi, S\. Belakaria, and J\. R\. Doppa\(2024\)Pareto front\-diverse batch multi\-objective bayesian optimization\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 10845–10853\.Cited by:[§2\.3](https://arxiv.org/html/2607.03140#S2.SS3.p1.1)\.
- \[2\]K\. Biel and C\. H\. Glock\(2020\)Energy\-efficient automation: a review of control strategies\.International Journal of Production Research58\(15\),pp\. 4771–4793\.Cited by:[§1](https://arxiv.org/html/2607.03140#S1.p1.1)\.
- \[3\]S\. Daulton, M\. Balandat, and E\. Bakshy\(2020\)Differentiable expected hypervolume improvement for parallel multi\-objective bayesian optimization\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9851–9864\.Cited by:[§2\.3](https://arxiv.org/html/2607.03140#S2.SS3.p1.1)\.
- \[4\]K\. Deb\(2014\)Multi\-objective optimization using evolutionary algorithms\.John Wiley & Sons\.Cited by:[§1](https://arxiv.org/html/2607.03140#S1.p3.1)\.
- \[5\]C\. F\. Hayes, R\. Rădulescu, E\. Bargiacchi, J\. Källström, M\. Macfarlane, M\. Reymond, T\. Verstraeten, L\. M\. Zintgraf, R\. Dazeley, F\. Heintz,et al\.\(2022\)A practical guide to multi\-objective reinforcement learning and planning\.Autonomous Agents and Multi\-Agent Systems36\(1\),pp\. 26\.Cited by:[§2\.1](https://arxiv.org/html/2607.03140#S2.SS1.p1.8)\.
- \[6\]J\. Kober, J\. A\. Bagnell, and J\. Peters\(2013\)Reinforcement learning in robotics: a survey\.The International Journal of Robotics Research32\(11\),pp\. 1238–1274\.Cited by:[§1](https://arxiv.org/html/2607.03140#S1.p1.1)\.
- \[7\]C\. Liu, Y\. Zhang, and P\. Xu\(2026\)Energy\-aware reinforcement learning for robotic manipulation of articulated components in infrastructure operation and maintenance\.arXiv preprint arXiv:2602\.12288\.Cited by:[§2\.2](https://arxiv.org/html/2607.03140#S2.SS2.p1.1)\.
- \[8\]K\. Miettinen\(1998\)Nonlinear multiobjective optimization\.Springer Science & Business Media\.Cited by:[§2\.1](https://arxiv.org/html/2607.03140#S2.SS1.p1.8)\.
- \[9\]M\. Olson, E\. Santorella, L\. C\. Tiao, S\. Cakmak, M\. Garrard, S\. Daulton, Z\. J\. Lin, S\. Ament, B\. Beckerman, E\. Onofrey, P\. Igusti, C\. Lara, B\. Letham, C\. Cardoso, S\. S\. Shen, A\. C\. Lin, M\. Grange, E\. Kashtelyan, D\. Eriksson, M\. Balandat, and Eytan\. Bakshy\(2025\)Ax: A Platform for Adaptive Experimentation\.InAutoML 2025 ABCD Track,Cited by:[§4\.2](https://arxiv.org/html/2607.03140#S4.SS2.p1.3)\.
- \[10\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga,et al\.\(2019\)Pytorch: an imperative style, high\-performance deep learning library\.Advances in neural information processing systems32\.Cited by:[§4\.1](https://arxiv.org/html/2607.03140#S4.SS1.p1.1)\.
- \[11\]A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. Dormann\(2021\)Stable\-baselines3: reliable reinforcement learning implementations\.Journal of Machine Learning Research22\(268\),pp\. 1–8\.External Links:[Link](https://jmlr.org/papers/v22/20-1364.html)Cited by:[§4\.1](https://arxiv.org/html/2607.03140#S4.SS1.p1.1)\.
- \[12\]C\. E\. Rasmussen and C\. K\. Williams\(2006\)Gaussian processes for machine learning\.MIT press\.Cited by:[§2\.3](https://arxiv.org/html/2607.03140#S2.SS3.p1.1)\.
- \[13\]D\. M\. Roijers, P\. Vamplew, S\. Whiteson, and R\. Dazeley\(2013\)A survey of multi\-objective sequential decision\-making\.Journal of Artificial Intelligence Research48,pp\. 67–113\.Cited by:[§1](https://arxiv.org/html/2607.03140#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.03140#S2.SS1.p1.8)\.
- \[14\]G\. Schäfer, T\. Krau, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2025\-05\)The Crucial Role of Problem Formulation in Real\-World Reinforcement Learning\.In2025 IEEE 8th International Conference on Industrial Cyber\-Physical Systems \(ICPS\),Emden, Germany\.External Links:[Document](https://dx.doi.org/10.1109/ICPS65515.2025.11087883)Cited by:[§3\.3](https://arxiv.org/html/2607.03140#S3.SS3.p1.8)\.
- \[15\]G\. Schäfer, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2024\-08\)Comparison of Model Predictive Control and Proximal Policy Optimization for a 1\-DOF Helicopter System\.In2024 IEEE 22ndInternational Conference on Industrial Informatics\(INDIN’24\),Beijing, China\.Note:Best presentation awardExternal Links:[Document](https://dx.doi.org/10.1109/INDIN58382.2024.10774357)Cited by:[§1](https://arxiv.org/html/2607.03140#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.03140#S3.SS1.p2.5)\.
- \[16\]G\. Schäfer, M\. Schirl, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2024\-09\)Python\-Based Reinforcement Learning on Simulink Models\.In11th International Conference on Soft Methods in Probability and Statistics \(SMPS 2024\),Salzburg, Austria\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-65993-5%5F55)Cited by:[§4\.1](https://arxiv.org/html/2607.03140#S4.SS1.p1.1)\.
- \[17\]G\. Schäfer, R\. Seliger, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2025\-08\)Multi\-Objective Reinforcement Learning for Energy\-Efficient Industrial Control\.InDatabase and Expert Systems Applications \- DEXA 2025 Workshops,Communications in Computer and Information Science,Bangkok, Thailand,pp\. 67–72\.External Links:[Document](https://dx.doi.org/10.1007/978-3-032-02003-1),ISBN 978\-3\-032\-02002\-4Cited by:[§3\.3](https://arxiv.org/html/2607.03140#S3.SS3.p1.8)\.
- \[18\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§3\.2](https://arxiv.org/html/2607.03140#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.03140#S4.SS1.p1.1)\.
- \[19\]B\. Shahriari, K\. Swersky, Z\. Wang, R\. P\. Adams, and N\. De Freitas\(2015\)Taking the human out of the loop: a review of bayesian optimization\.Proceedings of the IEEE104\(1\),pp\. 148–175\.Cited by:[§2\.3](https://arxiv.org/html/2607.03140#S2.SS3.p1.1)\.
- \[20\]R\. S\. Sutton and A\. G\. Barto\(2018\)Reinforcement learning: an introduction\.MIT press\.Cited by:[§2\.1](https://arxiv.org/html/2607.03140#S2.SS1.p1.8)\.
- \[21\]S\. Suzuki, S\. Takeno, T\. Tamura, K\. Shitara, and M\. Karasuyama\(2020\)Multi\-objective bayesian optimization using pareto\-frontier entropy\.InProceedings of the 37th International Conference on Machine Learning,pp\. 9279–9288\.Cited by:[§2\.3](https://arxiv.org/html/2607.03140#S2.SS3.p1.1)\.