Anticipatory Reinforcement Learning for Trajectory Tracking

# Anticipatory Reinforcement Learning for Trajectory Tracking
Source: [https://arxiv.org/html/2607.03132](https://arxiv.org/html/2607.03132)
RLreinforcement learningPPOproximal policy optimizationDRLdeep reinforcement learningMDPMarkov decision processMPCmodel predictive control1\-DoFone\-degree\-of\-freedomLQRlinear\-quadratic regulatorLQIöinear\-quadratic\-integral regulatorLTIlinear Time\-InvariantSB3Stable Baselines3TRPOtrust region policy optimizationICPSindustrial cyber\-physical systemCPScyber\-physical systemPIDproportional\-integral\-derivative
11institutetext:Josef Ressel Centre for Intelligent and Secure Industrial Automation,
Salzburg University of Applied Sciences, Salzburg, Austria22institutetext:Paris Lodron University of Salzburg, Salzburg, Austria
22email:georg\.schaefer@fh\-salzburg\.ac\.at###### Abstract

Deep reinforcement learning \(DRL\) in industrial control often suffers from lag and overshoot due to purely reactive control based on the current tracking error\. To achieve anticipatory control without high computational overhead, we introduce a predictive formulation that augments the DRL state space with target velocities and future reference horizons\. Evaluating eight configurations using proximal policy optimization \(PPO\) on a 1\-degree\-of\-freedom \(1\-DoF\) helicopter testbed, simulation results showed a 9\-fold error reduction, lowering the mean absolute deviation from2\.73∘2\.73^\{\\circ\}to0\.31∘0\.31^\{\\circ\}\. However, zero\-shot transfer to physical hardware revealed a sim\-to\-real gap\. Interestingly, a simpler configuration using a single, further look\-ahead horizon matched the real\-world top performance of the most complex model \(1\.11∘1\.11^\{\\circ\}\)\. Overall, evaluating various combinations of prediction horizons and target velocities demonstrated that highly granular predictive data is not necessarily required for physical transfer\.

## 1Introduction

\\Acp

icps integrate physical processes with computational control and underpin modern manufacturing, energy management, and smart production lines\.\\Acrl has demonstrated success in optimizing these complex, dynamic environments, offering data\-driven control solutions where mathematical modeling is challenging\[[2](https://arxiv.org/html/2607.03132#bib.bib8)\]\. However, many[deep reinforcement learning](https://arxiv.org/html/2607.03132#id3.3.id3)\([DRL](https://arxiv.org/html/2607.03132#id3.3.id3)\) policies typically compute their actions based on the current state and the instantaneous reference, without anticipating future changes in the reference trajectory\. For example, standard benchmark environments and baseline controllers often rely solely on the instantaneous deviation from a target \(e\.g\.,\[[1](https://arxiv.org/html/2607.03132#bib.bib6),[4](https://arxiv.org/html/2607.03132#bib.bib3)\]\)\. Consequently, these agents frequently exhibit lag and overshoot, as they cannot anticipate upcoming trajectory changes\. In contrast, classic control strategies such as[model predictive control](https://arxiv.org/html/2607.03132#id5.5.id5)\([MPC](https://arxiv.org/html/2607.03132#id5.5.id5)\) systematically incorporate future reference trajectories\. However, computing optimal actuating sequences requires solving an optimization problem at every time step\. This high computational demand limits[MPC](https://arxiv.org/html/2607.03132#id5.5.id5)in real\-time industrial applications\.\\Acdrl offers computationally lightweight online inference once trained\. To bridge the gap between the foresight of[MPC](https://arxiv.org/html/2607.03132#id5.5.id5)and the computational efficiency of[DRL](https://arxiv.org/html/2607.03132#id3.3.id3), this work\-in\-progress paper introduces a predictive problem formulation for industrial mechatronic systems\. Our primary contributions are twofold: \(i\) We embed“foresight”into the[DRL](https://arxiv.org/html/2607.03132#id3.3.id3)state space by augmenting the agent’s observation with future target variables and the current target velocity\. \(ii\) We systematically evaluate the sim\-to\-real transferability of these predictive agents on a physical[one\-degree\-of\-freedom](https://arxiv.org/html/2607.03132#id6.6.id6)\([1\-DoF](https://arxiv.org/html/2607.03132#id6.6.id6)\) helicopter testbed to assess their performance with regards to reference tracking\.

## 2Problem Formulation and Experimental Setup

The experimental setup utilizes the Quanser Aero 2 testbed in a[1\-DoF](https://arxiv.org/html/2607.03132#id6.6.id6)configuration\. We access the simulation and the physical system as described in\[[5](https://arxiv.org/html/2607.03132#bib.bib4)\]\. Building upon our previous problem formulation\[[4](https://arxiv.org/html/2607.03132#bib.bib3)\], we redefine the tracking problem to better align with the dynamic constraints of physical hardware\. Instead of step functions as reference signals, which are physically impossible to track for mechanical systems, we shift the target generation to continuous S\-curve trajectories and constant sections\. These S\-curves provide physically feasible, differentiable trajectories\. This prevents the agent from overfitting its policy to the unrealistic, instantaneous error jumps caused by rigid step functions, forcing it instead to learn generalized, dynamic tracking behavior\. In a purely reactive setup, the environment state at timettis defined asst=\(θt,ωt,rt\)s\_\{t\}=\(\\theta\_\{t\},\\omega\_\{t\},r\_\{t\}\), whereθt\\theta\_\{t\}is the pitch angle,ωt\\omega\_\{t\}is the angular velocity \(computed asΔ​θ/Δ​t\\Delta\\theta/\\Delta t\), andrtr\_\{t\}is the current reference\. To embed predictive“foresight”, we augment this state space by including the target velocityr˙t\\dot\{r\}\_\{t\}\(computed analytically\) and a horizon of future target points sampled directly from the trajectory\. The augmented state space is formulated asst=\(θt,ωt,rt,r˙t,rt\+1⋅Δ​t,…,rt\+N⋅Δ​t\)s\_\{t\}=\(\\theta\_\{t\},\\omega\_\{t\},r\_\{t\},\\dot\{r\}\_\{t\},r\_\{t\+1\\cdot\\Delta t\},\.\.\.,r\_\{t\+N\\cdot\\Delta t\}\)\. Here,NNrepresents the number of future targets provided to the agent, andΔ​t\\Delta tdefines the fixed time interval between these future trajectory points\. To solve this control task, we selected the[proximal policy optimization](https://arxiv.org/html/2607.03132#id2.2.id2)\([PPO](https://arxiv.org/html/2607.03132#id2.2.id2)\) algorithm due to its theoretical policy improvement guarantees and proven stability in continuous action spaces\. The algorithm was implemented using the Stable Baselines3 library\[[3](https://arxiv.org/html/2607.03132#bib.bib7)\]\. To isolate the impact of predictive features, we systematically evaluated eight state\-space configurations formed by augmenting the purely reactive baseline with various combinations of the target velocityr˙\\dot\{r\}and look\-ahead horizons \(NNfuture targets at varying intervalsΔ​t\\Delta t\)\. \(2\)Target Velocity:augmented solely withr˙t\\dot\{r\}\_\{t\};Future Targets:augmented withNNfuture targets at varying intervals ofΔ​t\\Delta t; and \(4\)Combined:integrating bothr˙t\\dot\{r\}\_\{t\}and future targets\.

To ensure statistical significance, each configuration was trained in simulation over 10 independent runs using varying random seeds\. The primary evaluation metric was the mean absolute deviation to the target trajectory\. Finally, to evaluate the sim\-to\-real gap, the single best\-performing agent from each configuration was deployed directly onto the physical hardware and evaluated on a single representative S\-curve trajectory\.

## 3Results and Discussion

The experimental results of all eight evaluated configurations are summarized in[Table˜1](https://arxiv.org/html/2607.03132#S3.T1)\. In the simulation environment, the introduction of predictive state variables yielded a roughly 9\-fold improvement in tracking accuracy\. The purely reactive baseline agent struggled to track the continuous S\-curves, resulting in a mean absolute deviation of2\.73∘±0\.18∘2\.73^\{\\circ\}\\pm 0\.18^\{\\circ\}\. In contrast, the fully combined predictive agent \(utilizing both the target velocityr˙t\\dot\{r\}\_\{t\}andN=4N=4future targets atΔ​t=0\.5\\Delta t=0\.5s intervals\) achieved near\-perfect tracking with a deviation of just0\.31∘±0\.10∘0\.31^\{\\circ\}\\pm 0\.10^\{\\circ\}\. A critical finding from the simulation data is the impact of the look\-ahead horizon length\. When providing the agent with only a single future target \(N=1N=1\), a time interval ofΔ​t=1\.0\\Delta t=1\.0s significantly outperformed a shortΔ​t=0\.1\\Delta t=0\.1s interval \(0\.85∘±0\.06∘0\.85^\{\\circ\}\\pm 0\.06^\{\\circ\}versus1\.70∘±0\.13∘1\.70^\{\\circ\}\\pm 0\.13^\{\\circ\}, respectively\)\. This behavior can be attributed to the physical inertia of the[1\-DoF](https://arxiv.org/html/2607.03132#id6.6.id6)system\. A0\.10\.1s look\-ahead does not provide sufficient time for the agent to accelerate or decelerate the beam\. Conversely, a1\.01\.0s horizon allows the agent to smoothly“pre\-steer”and apply control efforts proactively, minimizing lag\. To visualize this improvement[Fig\.˜1](https://arxiv.org/html/2607.03132#S3.F1)illustrates the tracking behavior of the baseline agent compared to the predictive agent against the reference signal\.

![Refer to caption](https://arxiv.org/html/2607.03132v1/x1.png)Figure 1:Simulation evaluation profile of the reactive baseline versus the best predictive agent, showing the S\-curve pitch tracking \(top\), target deviation error \(middle\), and applied control voltage \(bottom\)\.While the simulation results demonstrate the significant improvement of the predictive formulation, the real\-world validation on the physical hardware reveals a classic sim\-to\-real gap\. The most complex agent \(r˙t\\dot\{r\}\_\{t\},N=4N=4withΔ​t=0\.5\\Delta t=0\.5s\) performed best in simulation\. However, during physical hardware validation, this most augmented configuration did not outperform simpler models; instead, it yielded the same top\-tier performance as an agent with no target velocity and only a single future target withΔ​t=1\.0​s\\Delta t=1\.0\\,s\. This phenomenon provides valuable insights into mitigating the sim\-to\-real gap\. By providing the agent with highly granular, dense trajectory data, the neural network is able to exploit the idealized, noise\-free dynamics of the simulation model\. However, physical systems suffer from unmodeled friction, communication delays, and sensor noise\. As observed in recent literature regarding the sim\-to\-real gap\[[6](https://arxiv.org/html/2607.03132#bib.bib2)\], policies that perfectly optimize for simulated dynamics often become fragile in reality\. Because the simpler predictive formulation matches the hardware performance of the most complex agent, it becomes clear that highly dense trajectory data is not strictly required for optimal physical transfer\. Instead, a simpler, broader look\-ahead horizon acts as an effective regularizer\. It provides enough foresight to overcome the system’s inertia while remaining abstract enough to ignore minor dynamic discrepancies between the simulation and the real\-world testbed, offering a more computationally efficient state space without sacrificing real\-world accuracy\.

Table 1:Comparison of tracking performance across different predictive state representations\.r˙\\dot\{r\}: Target velocity included\.NN: Number of future target points\.Δ​t\\Delta t: Time interval between future targets in seconds\. Values represent the absolute pitch deviation in degrees \(∘\)\. Worst, Best, Mean, and Std refer to the maximum, minimum, average, and standard deviation of the tracking error across the 10 independent simulation runs\.
## 4Conclusion and Future Work

We demonstrated that the performance limitations of standard[DRL](https://arxiv.org/html/2607.03132#id3.3.id3)in physical tracking stem largely from their purely reactive problem formulation\. Embedding a predictive look\-ahead horizon into the state space bridges the gap between reactive AI and optimal control\. This augmented formulation allows the agent to anticipate trajectory changes and compensate for physical inertia, resulting in highly accurate continuous tracking\. For industry, this approach offers a highly desirable trade\-off: lightweight online inference of neural networks while maintaining the high precision required to stabilize complex plants\. Furthermore, sim\-to\-real validation highlighted that a coarser prediction horizon acts as a robust regularizer, preventing overfitting to simulated dynamics\. Future work will expand this approach\. First, to mitigate the reliance ona prioritarget trajectories, we will explore learning trajectory prediction models for task\-specific movement patterns via supervised learning\. Second, we plan fine\-tuning directly on hardware and conducting statistically rigorous hardware evaluations across a broader\(N,Δ​t\)\(N,\\Delta t\)grid\. Third, we aim to scale this predictive formulation to multi\-DoF robotic systems for complex manufacturing tasks\. Finally, we will benchmark the real\-time performance and computational overhead of our agents against advanced[MPC](https://arxiv.org/html/2607.03132#id5.5.id5)\.

### Acknowledgments\.

Financial support for this study was provided by the Christian Doppler Research Association \(CDG\) through the Josef Ressel Centre for Intelligent and Secure Industrial Automation, the corresponding WISS Co\-project of Land Salzburg, and by the European Interreg project BA0100172 AI4GREEN\. During the preparation of this work, the authors used AI tools for language editing and formatting assistance\.

## References

- \[1\]G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. Zaremba\(2016\)Openai gym\.arXiv preprint arXiv:1606\.01540\.Cited by:[§1](https://arxiv.org/html/2607.03132#S1.p1.2)\.
- \[2\]R\. Nian, J\. Liu, and B\. Huang\(2020\)A review on reinforcement learning: introduction and applications in industrial process control\.Computers & Chemical Engineering139,pp\. 106886\.Cited by:[§1](https://arxiv.org/html/2607.03132#S1.p1.2)\.
- \[3\]A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. Dormann\(2021\)Stable\-baselines3: reliable reinforcement learning implementations\.Journal of machine learning research22\(268\),pp\. 1–8\.Cited by:[§2](https://arxiv.org/html/2607.03132#S2.p1.17)\.
- \[4\]G\. Schäfer, T\. Krau, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2025\)The crucial role of problem formulation in real\-world reinforcement learning\.In2025 IEEE 8th International Conference on Industrial Cyber\-Physical Systems \(ICPS\),pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2607.03132#S1.p1.2),[§2](https://arxiv.org/html/2607.03132#S2.p1.17)\.
- \[5\]G\. Schäfer, M\. Schirl, J\. Rehrl, S\. Huber, and S\. Hirlaender\(2024\)Python\-based reinforcement learning on simulink models\.InInternational Conference on Soft Methods in Probability and Statistics,pp\. 449–456\.Cited by:[§2](https://arxiv.org/html/2607.03132#S2.p1.17)\.
- \[6\]A\. Wagenmaker, K\. Huang, L\. Ke, K\. Jamieson, and A\. Gupta\(2024\)Overcoming the sim\-to\-real gap: leveraging simulation to learn to explore for real\-world rl\.Advances in Neural Information Processing Systems37,pp\. 78715–78765\.Cited by:[§3](https://arxiv.org/html/2607.03132#S3.p2.4)\.