Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System

# Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
Source: [https://arxiv.org/html/2607.03125](https://arxiv.org/html/2607.03125)
RLreinforcement learningPPOproximal policy optimizationDRLdeep reinforcement learningMDPMarkov decision processMPCmodel predictive control1\-DoFone\-degree\-of\-freedomLQRlinear\-quadratic regulatorLQIlinear\-quadratic\-integral regulatorLTIlinear Time\-InvariantSB3Stable Baselines3TRPOtrust region policy optimizationICPSindustrial cyber\-physical systemCPScyber\-physical systemPIDproportional\-integral\-derivativePINNphysics\-informed neural networkCMDPconstrained Markov decision processCBFcontrol barrier functionRK44th order Runge\-KuttaODEordinary differential equationReLUrectified linear unit
11institutetext:Josef Ressel Centre for Intelligent and Secure Industrial Automation,
Salzburg University of Applied Sciences, Salzburg, Austria22institutetext:Paris Lodron University of Salzburg, Salzburg, Austria
22email:georg\.schaefer@fh\-salzburg\.ac\.at###### Abstract

Deep reinforcement learning \(DRL\) offers powerful control for industrial cyber\-physical systems \(ICPSs\), but its“black\-box”exploration risks violating strict hardware safety limits\. Typically, these constraints are managed through complex reward shaping\. In this work\-in\-progress paper, we embed a differentiable physics model directly into the proximal policy optimization \(PPO\) actor loss function\. By simulating short\-horizon future trajectories during training, the policy is penalized for anticipated safety violations independent of the task\-reward signal\. Evaluated on a simulated 1\-degree\-of\-freedom helicopter testbed with strict pitch constraints, our physics\-informed soft regularizations substantially reduce constraint violations while maintaining reliable target tracking\.

## 1Introduction

\\Acp

icps integrate physical processes with computational control, underpinning modern manufacturing and smart production\.\\Acrl has demonstrated significant success in optimizing these complex environments, offering data\-driven solutions where mathematical modeling is challenging\[[4](https://arxiv.org/html/2607.03125#bib.bib5)\]\. However, a primary limitation of standard[reinforcement learning](https://arxiv.org/html/2607.03125#id1.1.id1)\([RL](https://arxiv.org/html/2607.03125#id1.1.id1)\) is its“black\-box”exploration strategy, which can cause irreversible hardware damage in real\-world mechatronic systems\. The target system is the Quanser Aero 2111[https://www\.quanser\.com/products/aero\-2/](https://www.quanser.com/products/aero-2/), a mechatronic laboratory testbed suitable for controlling the angleθ\\thetaof a bar by actuating two rotors, see\[[7](https://arxiv.org/html/2607.03125#bib.bib29)\]for details\. Traditionally, safety constraints in[RL](https://arxiv.org/html/2607.03125#id1.1.id1)are encoded purely within the reward definition, leading to complex and brittle reward shaping\[[3](https://arxiv.org/html/2607.03125#bib.bib7)\]\. Other safe[RL](https://arxiv.org/html/2607.03125#id1.1.id1)paradigms rely on[constrained Markov decision processes](https://arxiv.org/html/2607.03125#id16.16.id16)\[[1](https://arxiv.org/html/2607.03125#bib.bib1)\],[control barrier functions](https://arxiv.org/html/2607.03125#id17.17.id17)\[[2](https://arxiv.org/html/2607.03125#bib.bib2)\], or offline[model predictive control](https://arxiv.org/html/2607.03125#id5.5.id5)\([MPC](https://arxiv.org/html/2607.03125#id5.5.id5)\) projections\[[9](https://arxiv.org/html/2607.03125#bib.bib48)\]\. To achieve anticipatory and safe control without massive computational overhead, we propose embedding a differentiable physics model directly into the loss function of the[proximal policy optimization](https://arxiv.org/html/2607.03125#id2.2.id2)\([PPO](https://arxiv.org/html/2607.03125#id2.2.id2)\) algorithm\. This approach conditions the actor network regarding safety constraints by penalizing predicted future violations via differentiable simulation\[[6](https://arxiv.org/html/2607.03125#bib.bib3)\]\. The scientific value of this work\-in\-progress lies in demonstrating constraint satisfaction decoupled from the primary task\-reward signal, providing a scalable mechanism for safe[RL](https://arxiv.org/html/2607.03125#id1.1.id1)deployment\.

## 2Methodology

Our base algorithm is[PPO](https://arxiv.org/html/2607.03125#id2.2.id2)\[[11](https://arxiv.org/html/2607.03125#bib.bib4)\], chosen for its stable performance and continuous action space capabilities\. To embed physical foresight, the system dynamics are modeled via known differential equations representing the Quanser Aero 2\. We integrate the[physics\-informed neural network](https://arxiv.org/html/2607.03125#id15.15.id15)\([PINN](https://arxiv.org/html/2607.03125#id15.15.id15)\) via an auto\-regressive unroll mechanism\. An overview of this dual\-path architecture \(evaluating actions concurrently in the environment for task reward and in the differentiable physics model for safety constraints\) is illustrated in Fig\.[1](https://arxiv.org/html/2607.03125#S2.F1)\.

sts\_\{t\}Actorπθ\\pi\_\{\\theta\}Physics \(RK4\)EnvironmentℒSafety\\mathcal\{L\}\_\{\\text\{Safety\}\}ℒPPO\\mathcal\{L\}\_\{\\text\{PPO\}\}Σ\\Sigma𝐱t\\mathbf\{x\}\_\{t\}ata\_\{t\}ata\_\{t\}θ\(1​…​H\)\\theta^\{\(1\.\.\.H\)\}rt,st\+1r\_\{t\},s\_\{t\+1\}⋅λ\\cdot\\lambdaCombined gradients flow backwardFigure 1:Architecture overview: Actions are evaluated concurrently in the environment and the differentiable physics model\. Both the task objective \(ℒPPO\\mathcal\{L\}\_\{\\text\{PPO\}\}\) and the anticipatory safety penalty \(ℒSafety\\mathcal\{L\}\_\{\\text\{Safety\}\}, scaled byλ\\lambda\) combine into a total loss, whose gradients flow backward to update the policy\.- •Lookahead Horizon:We project 0\.3 s into the future \(3 steps at 10 Hz\)\. This balances the need to capture system inertia against the computational cost of long\-term simulation\.
- •System Dynamics vs\.[RL](https://arxiv.org/html/2607.03125#id1.1.id1)Observation:For every timesteptt, we distinguish the agent’s full observation statests\_\{t\}from the physical state𝐱𝐭\\mathbf\{x\_\{t\}\}used for safety prediction\. While the[PPO](https://arxiv.org/html/2607.03125#id2.2.id2)agent observes an augmented state\[[8](https://arxiv.org/html/2607.03125#bib.bib64)\], the differentiable physics model isolates strictly the[one\-degree\-of\-freedom](https://arxiv.org/html/2607.03125#id6.6.id6)\([1\-DoF](https://arxiv.org/html/2607.03125#id6.6.id6)\) Quanser Aero 2 system state:𝐱=\[θ,θ˙,ω0,ω1\]T\\mathbf\{x\}=\[\\theta,\\dot\{\\theta\},\\omega\_\{0\},\\omega\_\{1\}\]^\{T\}\(pitch angle, pitch velocity, and internal motor angular velocities\)\. We use these core variables to initialize the non\-linear continuous\-time dynamics𝐱˙=f​\(𝐱,𝐮\)\\mathbf\{\\dot\{x\}\}=f\(\\mathbf\{x\},\\mathbf\{u\}\)natively in PyTorch\. During training,𝐱t\\mathbf\{x\}\_\{t\}is extracted directly from the simulator as privileged information, and the current policy is sampled for new actions using the predicted state as an input\.
- •Numerical Integration:We utilize a[4th order Runge\-Kutta](https://arxiv.org/html/2607.03125#id18.18.id18)\([RK4](https://arxiv.org/html/2607.03125#id18.18.id18)\) integration method, operating at 5 micro\-steps per environment step, ensuring accurate trajectory simulation\.

Safety Constraint Definition:To rigorously test the mechanism, we impose an artificial pitch limit of±30∘\\pm 30^\{\\circ\}\. We simultaneously train the agent to track target signals of±40∘\\pm 40^\{\\circ\}\. While contradictory to standard operation, this extreme configuration effectively showcases the algorithm’s ability to prioritize safety bounds over task\-reward maximization\.

Loss Modification:The standard[PPO](https://arxiv.org/html/2607.03125#id2.2.id2)actor loss is augmented with a safety penalty derived from the differentiable physics model asℒ=ℒPPO\+λ⋅ℒSafety\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{PPO\}\}\+\\lambda\\cdot\\mathcal\{L\}\_\{\\text\{Safety\}\}, whereλ\\lambdaacts as a weighting factor\. Because the numerical[RK4](https://arxiv.org/html/2607.03125#id18.18.id18)integration is built entirely from trackable tensor operations, the computational graph records every simulation step\. Consequently, via the chain rule, gradients from anticipated safety violations flow backward through the unrolled differential equations into the actor network\. To computeℒSafety\\mathcal\{L\}\_\{\\text\{Safety\}\}, we unroll the system dynamics over the prediction horizon ofH=3H=3steps\. Letθ\(h\)\\theta^\{\(h\)\}denote the predicted pitch angle at horizon stephh, simulated via the differentiable[RK4](https://arxiv.org/html/2607.03125#id18.18.id18)integrator\. The safety penalty acts as a soft regularizer, relying on a[rectified linear unit](https://arxiv.org/html/2607.03125#id20.20.id20)\([ReLU](https://arxiv.org/html/2607.03125#id20.20.id20)\) to penalize the policy strictly when the absolute predicted pitch exceeds the predefined safety limitθmax=30∘\\theta\_\{\\text\{max\}\}=30^\{\\circ\}\. The normalized safety loss is defined as:

ℒSafety=1H​∑h=1Hmax⁡\(0,\|θ\(h\)\|θmax−1\)\\mathcal\{L\}\_\{\\text\{Safety\}\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\max\(0,\\frac\{\|\\theta^\{\(h\)\}\|\}\{\\theta\_\{\\text\{max\}\}\}\-1\)\(1\)
Because this formulation is purely composed of differentiable tensor operations, the resulting gradients correctly map anticipated future constraint violations back to the actor network’s current action distribution\.

## 3Experimental Setup and Results

We evaluate our approach on a simulated Quanser Aero 2 system in a[1\-DoF](https://arxiv.org/html/2607.03125#id6.6.id6)configuration\. The underlying system dynamics and base[RL](https://arxiv.org/html/2607.03125#id1.1.id1)problem formulation have been established in previous studies\[[7](https://arxiv.org/html/2607.03125#bib.bib29)\], and the environment is accessed via a standard Gym API framework using Python\-Simulink interfaces\[[10](https://arxiv.org/html/2607.03125#bib.bib28)\]\. We tested three primary configurations, running 10 random seeds for each to ensure statistical significance:*Naive Baseline*\(λ=0\\lambda=0\),*PINN Balanced*\(λ=150\\lambda=150\), and*PINN Over\-penalized*\(λ=1000\\lambda=1000\)\. The penalty weightλ\\lambdawas chosen empirically; a rigorous statistical evaluation of all 10 seeds and aλ\\lambdaPareto sweep are reserved for future work\.

Training Stability:All configurations successfully converged\. We observed that the value loss naturally increases for higherλ\\lambdaconfigurations due to the conflicting objectives \(tracking a40∘40^\{\\circ\}target vs\. stopping at30∘30^\{\\circ\}\), but the policy loss remained comparable across the board\.

Performance Trade\-off and Safety:The evaluation results \(see Fig\.[2](https://arxiv.org/html/2607.03125#S3.F2)\) highlight a clear trade\-off between target tracking and constraint satisfaction\. The*Naive Baseline*achieves high task reward by accurately tracking the40∘40^\{\\circ\}target, but heavily violates the30∘30^\{\\circ\}safety threshold\. The*PINN Over\-penalized*model exhibits very few safety violations but results in sluggish overall tracking, failing to adapt dynamically even when within safe regions\. The*PINN Balanced*configuration demonstrates a suitable choice ofλ\\lambda\. It aggressively tracks the reference signal and largely learns to decelerate to respect the30∘30^\{\\circ\}threshold\. Although brief transient violations can still occur during abrupt, extreme setpoint changes, it drastically reduces both predicted and actual pitch violations compared to the baseline\.

![Refer to caption](https://arxiv.org/html/2607.03125v1/x1.png)Figure 2:Simulation evaluation profile demonstrating the trade\-off between target tracking and constraint satisfaction\. The*Naive Baseline*ignores the30∘30^\{\\circ\}safety threshold, the*PINN Over\-penalized*struggles to track the signal, and the*PINN Balanced*successfully adheres to the constraints while maintaining strong tracking performance\.
## 4Conclusion and Future Work

In this work\-in\-progress paper, we demonstrated that embedding differentiable physics models into the[PPO](https://arxiv.org/html/2607.03125#id2.2.id2)actor loss function provides a scalable and effective mechanism for soft anticipatory safety regularization in[industrial cyber\-physical systems](https://arxiv.org/html/2607.03125#id12.12.id12)\. This approach enables anticipatory constraint satisfaction and mitigates the need for complex, task\-specific reward shaping\. Ongoing and future research will focus on the sim\-to\-real transfer of this[PINN](https://arxiv.org/html/2607.03125#id15.15.id15)\-trained agent directly onto the physical Quanser Aero 2 hardware\. Beyond zero\-shot transfer, we plan to conduct further fine\-tuning directly on the real hardware to close the sim\-to\-real gap\. Because relying on a static[ordinary differential equation](https://arxiv.org/html/2607.03125#id19.19.id19)\([ODE](https://arxiv.org/html/2607.03125#id19.19.id19)\) model during this phase could introduce systematic errors, this fine\-tuning should be coupled with active parameter identification\. Building upon our recent work on parameter identification methodologies for the testbed\[[5](https://arxiv.org/html/2607.03125#bib.bib65)\], we plan to dynamically update the parameters of the differentiable physics model to match observed real\-world trajectories\. Finally, we aim to complement our anticipatory training loss with a reactive safety mechanism \(such as a safety shield or[CBF](https://arxiv.org/html/2607.03125#id17.17.id17)\) that acts as a real\-time execution filter\. While the predictive loss smoothly guides the policy toward safe behaviors during training, a reactive architectural constraint would intervene directly during deployment, instantly overriding proposed actions if a forward\-simulation predicts an imminent pitch violation\.

### Acknowledgments\.

Financial support for this study was provided by the Christian Doppler Research Association \(CDG\) through the Josef Ressel Centre for Intelligent and Secure Industrial Automation, the corresponding WISS Co\-project of Land Salzburg, and by the European Interreg project BA0100172 AI4GREEN\. During the preparation of this work, the authors used AI tools for language editing and formatting assistance\.

## References

- \[1\]E\. Altman\(2021\)Constrained markov decision processes\.Routledge\.Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1)\.
- \[2\]A\. D\. Ames, S\. Coogan, M\. Egerstedt, G\. Notomista, K\. Sreenath, and P\. Tabuada\(2019\)Control barrier functions: theory and applications\.In2019 18th European control conference \(ECC\),pp\. 3420–3431\.Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1)\.
- \[3\]J\. Garcıa and F\. Fernández\(2015\)A comprehensive survey on safe reinforcement learning\.Journal of Machine Learning Research16\(1\),pp\. 1437–1480\.Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1)\.
- \[4\]J\. Kober, J\. A\. Bagnell, and J\. Peters\(2013\)Reinforcement learning in robotics: a survey\.The International Journal of Robotics Research32\(11\),pp\. 1238–1274\.Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1)\.
- \[5\]J\. Langschwert, G\. Schäfer, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2026\)Reinforcement Learning for Optimal Experiment Design in Parameter Identification of Mechatronic Systems\.InDatabase and Expert Systems Applications \- DEXA 2026 Workshops,Communications in Computer and Information Science,Graz, Austria\.Note:submittedCited by:[§4](https://arxiv.org/html/2607.03125#S4.p1.1)\.
- \[6\]M\. Raissi, P\. Perdikaris, and G\. E\. Karniadakis\(2019\)Physics\-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations\.Journal of Computational physics378,pp\. 686–707\.Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1)\.
- \[7\]G\. Schäfer, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2024\-08\)Comparison of Model Predictive Control and Proximal Policy Optimization for a 1\-DOF Helicopter System\.In2024 IEEE 22nd International Conference on Industrial Informatics \(INDIN’24\),Beijing, China\.External Links:[Document](https://dx.doi.org/10.1109/INDIN58382.2024.10774357)Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1),[§3](https://arxiv.org/html/2607.03125#S3.p1.5)\.
- \[8\]G\. Schäfer, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2026\)Anticipatory Reinforcement Learning for Trajectory Tracking\.InDatabase and Expert Systems Applications \- DEXA 2026 Workshops,Communications in Computer and Information Science,Graz, Austria\.Note:submittedCited by:[2nd item](https://arxiv.org/html/2607.03125#S2.I1.i2.p1.6)\.
- \[9\]G\. Schäfer, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2026\-02\)Safe reinforcement learning using ideas from model predictive control\.InComputer Aided Systems Theory – EUROCAST 2026 – Extended Abstracts,Vol\.20\.Cited by:[§1](https://arxiv.org/html/2607.03125#S1.p1.1)\.
- \[10\]G\. Schäfer, M\. Schirl, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2024\-09\)Python\-Based Reinforcement Learning on Simulink Models\.In11th International Conference on Soft Methods in Probability and Statistics \(SMPS 2024\),Salzburg, Austria\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-65993-5%5F55)Cited by:[§3](https://arxiv.org/html/2607.03125#S3.p1.5)\.
- \[11\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2607.03125#S2.p1.1)\.