Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control

# Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control
Source: [https://arxiv.org/html/2609.17697](https://arxiv.org/html/2609.17697)
Azita DabiriBart De Schutter††thanks:This research has received funding from the European Research Council \(ERC\) under the European Union’s Horizon 2020 research and innovation programme \(Grant agreement No\. 101018826 \- ERC Advanced Grant CLariNet\)\.††thanks:The authors are with the Delft Center for Systems and Control, Delft University of Technology, Delft, The Netherlands \{g\.oenuer, a\.dabiri, b\.deschutter\}@tudelft\.nl\.

###### Abstract

Integrated deep reinforcement learning \(DRL\) and model predictive control \(MPC\) methods are increasingly used to control autonomous systems by combining their complementary capabilities\. DRL learns control policies through interaction with the environment\. MPC uses a system model to optimize control inputs while accounting for constraints\. In DRL\-MPC frameworks with shared control authority, both the DRL agent and the MPC controller each determine part of the control inputs\. However, common learning formulations treat MPC as part of the environment and therefore do not explicitly account for MPC’s contribution to control or its interaction with the DRL agent\. This paper proposes a novel composite\-gradient learning \(CGL\) method that integrates the MPC controller into the learning process by representing the DRL and MPC control inputs as a joint action and accounting for their interaction when updating the DRL agent during training\. CGL is evaluated on two multi\-class freeway traffic networks with different strengths of interaction between the DRL and MPC control inputs and it is compared with alternative methods that treat MPC as part of the environment or that only partially incorporate MPC into learning\. The results show that CGL offers limited benefit under weak interaction, but learns higher\-performing control policies than the alternative methods in a subset of training runs under strong interaction, although the average control\-performance gains remain modest\.

## IIntroduction

Model predictive control \(MPC\) and deep reinforcement learning \(DRL\) have become prominent approaches for controlling complex systems in applications ranging from industrial process control to robotics\[[1](https://arxiv.org/html/2609.17697#bib.bib9),[2](https://arxiv.org/html/2609.17697#bib.bib4)\]\. Although both methods address sequential decision making, they employ different approaches\. MPC\[[1](https://arxiv.org/html/2609.17697#bib.bib9)\]uses an explicit prediction model to forecast the system evolution over a finite horizon, optimizes a sequence of control inputs while explicitly accounting for system constraints, applies only the first control input of the optimized sequence, and repeats this procedure in a receding\-horizon fashion\. On the other hand, DRL\[[2](https://arxiv.org/html/2609.17697#bib.bib4)\]learns a control policy directly through interaction with the environment without requiring an explicit system model, allowing control policies to be obtained when model knowledge is limited or the system dynamics are uncertain\. DRL also enables fast control input computation after training, since the trained policy network directly maps the current observation to an action through a single forward pass without requiring online optimization\.

The complementary characteristics of DRL and MPC motivate hybrid frameworks that combine MPC’s built\-in optimization and constraint\-handling capabilities with DRL’s fast online computation and model independence\[[3](https://arxiv.org/html/2609.17697#bib.bib11)\]\. Hierarchical DRL\-MPC frameworks exploit this combination by assigning different tasks and time scales to either method\. Such frameworks improve control performance in automation applications such as traffic management\[[4](https://arxiv.org/html/2609.17697#bib.bib7)\]and microgrid energy management\[[5](https://arxiv.org/html/2609.17697#bib.bib12)\]\. However, despite overall progress, integrating DRL and MPC remains a developing area with limited practical applications\.

The current paper considers a hierarchical DRL\-MPC framework\[[6](https://arxiv.org/html/2609.17697#bib.bib8)\], in which the control authority and control inputs are divided between DRL and MPC to allow MPC to run at a low frequency for control measures whose slower update rate accommodates MPC’s high computation time, while DRL controls high\-frequency control measures by leveraging its short deployment time\. For example, high\-frequency and low\-frequency control inputs can correspond, respectively, to ramp metering rates and vehicle splitting rates in freeway networks\[[7](https://arxiv.org/html/2609.17697#bib.bib10)\], actuator commands and reference trajectories in robotic motion control\[[8](https://arxiv.org/html/2609.17697#bib.bib1)\], and semiconductor switching commands and motor\-current references in power\-electronic converter control\[[9](https://arxiv.org/html/2609.17697#bib.bib2)\]\. The hierarchical DRL\-MPC framework therefore reduces online computation relative to purely MPC\-based control while preserving MPC’s optimization and constraint\-handling capabilities at the upper level\.

In this hierarchical DRL\-MPC framework\[[6](https://arxiv.org/html/2609.17697#bib.bib8)\], the MPC controller is considered part of the environment while training the DRL agent, as treating the MPC controller as part of the environment is a common approach in such frameworks\[[3](https://arxiv.org/html/2609.17697#bib.bib11)\]\. However, when the control authority and control inputs are divided between MPC and DRL, since the MPC controller requires future DRL actions to predict the system evolution over its optimization horizon, the DRL agent influences the computation of the MPC control input\. When MPC is treated as part of the environment, this dependence is not explicitly incorporated into the learned assessment of control performance or into the update of the DRL agent, thereby leaving the learning formulation incomplete\. This may limit performance when the control inputs of the MPC controller and the DRL agent are strongly coupled\.

For example, such coupling may arise in a freeway traffic control problem in which an MPC controller at the upper level uses route guidance to regulate the traffic flow directed toward a metered on\-ramp, while a DRL agent at the lower level uses ramp metering to control vehicle entry onto the freeway mainline from that on\-ramp\. The high\-level MPC control input determines the traffic demand arriving at the metered on\-ramp and therefore the operating conditions faced by the lower\-level DRL agent\. Conversely, the predicted DRL actions over the MPC horizon influence the route guidance control input of the MPC controller by shaping the predicted traffic evolution\. Together, these interactions create bidirectional coupling between the MPC controller and the DRL agent\.

To incorporate this coupling into the learning process, we formulate the hierarchical control problem as a Markov decision process \(MDP\) in which the DRL action and the MPC control input form a joint action space\. Based on this formulation, we propose a novel composite\-gradient learning method that captures both the direct effect of the DRL action and its indirect effect through the MPC control input, thereby addressing the incomplete learning formulation that treats MPC as part of the environment\[[3](https://arxiv.org/html/2609.17697#bib.bib11)\]\.

The main contributions of this paper are as follows:

- •We formulate a hierarchical DRL\-MPC control framework with shared control authority as an MDP in which the lower\-level DRL action and the upper\-level MPC control input form a joint action to capture the interaction between the two control levels\.
- •We propose a novel composite\-gradient learning method in which the learned assessment of control performance accounts for the joint action, and the update of the DRL agent captures both the direct effect of the DRL action and its indirect effect through the MPC control input\.
- •We develop a hierarchy\-aware data storage and sampling scheme that reduces the computational cost of the learning process by reusing MPC computations across the data collected between two consecutive MPC control input updates\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2609.17697#S2)describes the hierarchical DRL\-MPC framework with shared control authority\. Section[III](https://arxiv.org/html/2609.17697#S3)presents the proposed composite\-gradient learning method\. Section[IV](https://arxiv.org/html/2609.17697#S4)introduces the multi\-class freeway traffic case studies and evaluates the proposed method under different DRL\-MPC control input coupling conditions\. Finally, Section[V](https://arxiv.org/html/2609.17697#S5)concludes the paper and outlines directions for future work\.

## IIHierarchical DRL\-MPC Framework with Shared Control Authority

Fig\. 1:\(a\) Block diagram illustrating the proposed DRL\-MPC framework that divides control inputs between the DRL agent and the MPC controller\. \(b\) Time scales for the low\-frequency control inputs of the high\-level MPC controller and the high\-frequency control inputs of the low\-level DRL agent\.The hierarchical DRL\-MPC framework divides control authority and control inputs between DRL and MPC\[[6](https://arxiv.org/html/2609.17697#bib.bib8)\], where the MPC controller uses a copy of the trained DRL agent, whose policy is held fixed during deployment, to obtain high\-frequency control inputs required by MPC for predicting the future states of the controlled system \(see Figure[1](https://arxiv.org/html/2609.17697#S2.F1)\(a\)\)\. This allows the MPC controller to predict the future states of the controlled system over its prediction window, even though the control input of the lower level is not determined by the MPC controller but by the DRL agent\. Through this coupling, the DRL agent influences the MPC solution by affecting the state predictions used in the MPC optimization\.

Let𝒙⁡\(k\)∈𝒳\\boldsymbol\{x\}\(k\)\\in\\mathcal\{X\}denote the state of the controlled system at sampling stepkk, where𝒳\\mathcal\{X\}is the state space of the system andTTis the length of the sampling time interval\. The high\- and low\-level control inputs jointly determine the evolution of the controlled system according to the discrete\-time dynamical model

𝒙⁡\(k\+1\)=f⁡\(𝒙⁡\(k\),𝒖¯h​\(k\),𝒖¯l​\(k\),𝒅⁡\(k\)\),\\boldsymbol\{x\}\(k\+1\)=f\\big\(\\boldsymbol\{x\}\(k\),\\boldsymbol\{\\bar\{u\}\}\_\{\\mathrm\{h\}\}\(k\),\\boldsymbol\{\\bar\{u\}\}\_\{\\mathrm\{l\}\}\(k\),\\boldsymbol\{d\}\(k\)\\big\),\(1\)whereffdenotes the system dynamics,𝒖¯h​\(k\)\\boldsymbol\{\\bar\{u\}\}\_\{\\mathrm\{h\}\}\(k\)and𝒖¯l​\(k\)\\boldsymbol\{\\bar\{u\}\}\_\{\\mathrm\{l\}\}\(k\)denote the high\- and low\-level control inputs applied to the controlled system at sampling stepkk, respectively, and𝒅⁡\(k\)\\boldsymbol\{d\}\(k\)denotes the exogenous disturbances\.

Letklk\_\{\\mathrm\{l\}\}represent the control step counter for the low\-level DRL agent, whose control update interval has lengthTlT\_\{\\mathrm\{l\}\}\(see Figure[1](https://arxiv.org/html/2609.17697#S2.F1)\(b\)\)\. The relationship between the low\-level control interval and the sampling time interval isTl=ml​TT\_\{\\mathrm\{l\}\}=m\_\{\\mathrm\{l\}\}T,k=ml​klk=m\_\{\\mathrm\{l\}\}k\_\{\\mathrm\{l\}\},ml∈ℕ\+m\_\{\\mathrm\{l\}\}\\in\\mathbb\{N\}^\{\+\},ml≥1m\_\{\\mathrm\{l\}\}\\geq 1, wheremlm\_\{\\mathrm\{l\}\}is the number of sampling intervals within one low\-level control interval,k=ml​klk=m\_\{\\mathrm\{l\}\}k\_\{\\mathrm\{l\}\}identifies the sampling step corresponding to low\-level update stepklk\_\{\\mathrm\{l\}\}, andℕ\+\\mathbb\{N\}^\{\+\}denotes the set of positive integers\. At every low\-level control step, the DRL agent receives the observation𝒙rl​\(kl\)∈𝒳rl\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)\\in\\mathcal\{X\}\_\{\\mathrm\{rl\}\}\. Let𝜼rl​\(kl\)\\boldsymbol\{\\eta\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)denote auxiliary information available to the agent that is not necessarily contained in the current controlled\-system state, such as exogenous measurements or previous states and control inputs\. The DRL observation may combine selected or transformed components of the system state with this auxiliary information and is defined by the observation mapHrlH\_\{\\mathrm\{rl\}\}as

𝒙rl​\(kl\)=Hrl​\(𝒙⁡\(ml​kl\),𝜼rl​\(kl\)\)\.\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)=H\_\{\\mathrm\{rl\}\}\\big\(\\boldsymbol\{x\}\(m\_\{\\mathrm\{l\}\}k\_\{\\mathrm\{l\}\}\),\\boldsymbol\{\\eta\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)\\big\)\.\(2\)
The DRL agent computes the high\-frequency control inputs everyTlT\_\{\\mathrm\{l\}\}time units as𝒖l​\(kl\)=πθ​\(𝒙rl​\(kl\)\)\\boldsymbol\{u\}\_\{\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\)=\\pi\_\{\\theta\}\\big\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)\\big\),𝒖l​\(kl\)∈𝒰l\\boldsymbol\{u\}\_\{\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\)\\in\\mathcal\{U\}\_\{\\mathrm\{l\}\}, whereπθ\\pi\_\{\\theta\}is the deterministic DRL policy with parametersθ\\theta, and𝒰l\\mathcal\{U\}\_\{\\mathrm\{l\}\}is the admissible low\-level control\-input set\. Then, the low\-level control inputs that are applied to the controlled system during the control interval\[kl​Tl,\(kl\+1\)​Tl\)\[k\_\{\\mathrm\{l\}\}T\_\{\\mathrm\{l\}\},\(k\_\{\\mathrm\{l\}\}\+1\)T\_\{\\mathrm\{l\}\}\)are calculated as𝒖¯l​\(ml​kl\+s\)=𝒖l​\(kl\)\\boldsymbol\{\\bar\{u\}\}\_\{\\mathrm\{l\}\}\(m\_\{\\mathrm\{l\}\}k\_\{\\mathrm\{l\}\}\+s\)=\\boldsymbol\{u\}\_\{\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\),s∈𝕀mls\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{l\}\}\}, where𝕀m=\{0,1,…,m−1\}\\mathbb\{I\}\_\{m\}=\\\{0,1,\\dots,m\-1\\\}form∈ℕ\+m\\in\\mathbb\{N\}^\{\+\}\. Hence, the control input𝒖l​\(kl\)\\boldsymbol\{u\}\_\{\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\)is applied over an interval ofTlT\_\{\\mathrm\{l\}\}time units using a zero\-order hold \(ZOH\) strategy\.

Let𝒖h​\(kh\)\\boldsymbol\{u\}\_\{\\mathrm\{h\}\}\(k\_\{\\mathrm\{h\}\}\)denote the high\-level control input applied at high\-level control stepkhk\_\{\\mathrm\{h\}\}\(see Figure[1](https://arxiv.org/html/2609.17697#S2.F1)\(b\)\), and let𝒖h,j\|kh\\boldsymbol\{u\}\_\{\\mathrm\{h\},j\\mid k\_\{\\mathrm\{h\}\}\}denote the high\-level control input at high\-level control stepkh\+jk\_\{\\mathrm\{h\}\}\+jcalculated at high\-level control stepkhk\_\{\\mathrm\{h\}\}, wherej∈𝕀Npj\\in\\mathbb\{I\}\_\{N\_\{\\mathrm\{p\}\}\}\. The MPC controller computes the control sequence

𝒖~h​\(kh\)=\[𝒖h,0\|kh⊤,𝒖h,1\|kh⊤,…,𝒖h,Np−1\|kh⊤\]⊤\\boldsymbol\{\\tilde\{u\}\}\_\{\\mathrm\{h\}\}\(k\_\{\\mathrm\{h\}\}\)=\\left\[\\boldsymbol\{u\}\_\{\\mathrm\{h\},0\\mid k\_\{\\mathrm\{h\}\}\}^\{\\top\},\\boldsymbol\{u\}\_\{\\mathrm\{h\},1\\mid k\_\{\\mathrm\{h\}\}\}^\{\\top\},\\dots,\\boldsymbol\{u\}\_\{\\mathrm\{h\},N\_\{\\mathrm\{p\}\}\-1\\mid k\_\{\\mathrm\{h\}\}\}^\{\\top\}\\right\]^\{\\top\}over a prediction horizon ofNpN\_\{\\mathrm\{p\}\}high\-level time steps, each with a duration ofThT\_\{\\mathrm\{h\}\}time units, whereTh=mh​TT\_\{\\mathrm\{h\}\}=m\_\{\\mathrm\{h\}\}T,k=mh​khk=m\_\{\\mathrm\{h\}\}k\_\{\\mathrm\{h\}\},mh=q​mlm\_\{\\mathrm\{h\}\}=qm\_\{\\mathrm\{l\}\},q∈ℕ\+q\\in\\mathbb\{N\}^\{\+\}\. Here,qqis the number of low\-level control intervals within one high\-level control interval, aligning the high\-level and low\-level control input updates \(see Figure[1](https://arxiv.org/html/2609.17697#S2.F1)\(b\)\)\. At coincident high\-level and low\-level update time instants, their step counters therefore satisfykl=q​khk\_\{\\mathrm\{l\}\}=qk\_\{\\mathrm\{h\}\}\.

At high\-level control time stepkhk\_\{\\mathrm\{h\}\}, let𝒙^ℓ\|kh=𝒙^​\(mh​kh\+ℓ\)\\boldsymbol\{\\hat\{x\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\}=\\boldsymbol\{\\hat\{x\}\}\(m\_\{\\mathrm\{h\}\}k\_\{\\mathrm\{h\}\}\+\\ell\)denote the predicted future state of the controlled systemℓ\\ellsampling steps ahead forℓ∈𝕀mh​Np\+1\\ell\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}N\_\{\\mathrm\{p\}\}\+1\}\. The predicted future states are stacked as

𝒙~​\(kh\)=\[𝒙^1\|kh⊤,𝒙^2\|kh⊤,…,𝒙^mh​Np\|kh⊤\]⊤\.\\boldsymbol\{\\tilde\{x\}\}\(k\_\{\\mathrm\{h\}\}\)=\\left\[\\boldsymbol\{\\hat\{x\}\}\_\{1\\mid k\_\{\\mathrm\{h\}\}\}^\{\\top\},\\boldsymbol\{\\hat\{x\}\}\_\{2\\mid k\_\{\\mathrm\{h\}\}\}^\{\\top\},\\dots,\\boldsymbol\{\\hat\{x\}\}\_\{m\_\{\\mathrm\{h\}\}N\_\{\\mathrm\{p\}\}\\mid k\_\{\\mathrm\{h\}\}\}^\{\\top\}\\right\]^\{\\top\}\.The states are predicted at the sampling steps to evaluate the objective function accurately, even though the high\-level control inputs are updated only at the high\-level control steps\. The relative\-index notation is extended analogously to the predicted high\-level inputs, low\-level inputs, and disturbances, denoted by𝒖^h,ℓ\|kh\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{h\},\\ell\\mid k\_\{\\mathrm\{h\}\}\},𝒖^l,ℓ\|kh\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{l\},\\ell\\mid k\_\{\\mathrm\{h\}\}\}, and𝒅^ℓ\|kh\\boldsymbol\{\\hat\{d\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\}, respectively, forℓ∈𝕀mh​Np\\ell\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}N\_\{\\mathrm\{p\}\}\}\. The disturbance forecasts𝒅^ℓ\|kh\\boldsymbol\{\\hat\{d\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\}are assumed to be available to the MPC controller\. Let𝒖l,i\|kh\\boldsymbol\{u\}\_\{\\mathrm\{l\},i\\mid k\_\{\\mathrm\{h\}\}\}and𝒙^rl,i\|kh\\boldsymbol\{\\hat\{x\}\}\_\{\\mathrm\{rl\},i\\mid k\_\{\\mathrm\{h\}\}\}denote the predicted DRL action and the corresponding predicted DRL observationiilow\-level control steps ahead, respectively, wherei∈𝕀q​Npi\\in\\mathbb\{I\}\_\{qN\_\{\\mathrm\{p\}\}\}\. Let𝜼^rl,i\|kh\\boldsymbol\{\\hat\{\\eta\}\}\_\{\\mathrm\{rl\},i\\mid k\_\{\\mathrm\{h\}\}\}denote the auxiliary information associated with that predicted DRL observation, which is obtained as𝒙^rl,i\|kh=Hrl​\(𝒙^ml​i\|kh,𝜼^rl,i\|kh\)\\boldsymbol\{\\hat\{x\}\}\_\{\\mathrm\{rl\},i\\mid k\_\{\\mathrm\{h\}\}\}=H\_\{\\mathrm\{rl\}\}\\big\(\\boldsymbol\{\\hat\{x\}\}\_\{m\_\{\\mathrm\{l\}\}i\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{\\eta\}\}\_\{\\mathrm\{rl\},i\\mid k\_\{\\mathrm\{h\}\}\}\\big\), analogous to \([2](https://arxiv.org/html/2609.17697#S2.E2)\)\.

The MPC controller computes the control inputs over the prediction horizon by solving the following optimization problem:

min𝒖~h​\(kh\),𝒙~​\(kh\)\\displaystyle\\min\_\{\\boldsymbol\{\\tilde\{u\}\}\_\{\\mathrm\{h\}\}\(k\_\{\\mathrm\{h\}\}\),\\boldsymbol\{\\tilde\{x\}\}\(k\_\{\\mathrm\{h\}\}\)\}\\;\\;∑ℓ∈𝕀mh​NpLmpc​\(𝒙^ℓ\|kh,𝒖^h,ℓ\|kh,𝒖^l,ℓ\|kh,𝒅^ℓ\|kh\)\\displaystyle\\sum\_\{\\ell\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}N\_\{\\mathrm\{p\}\}\}\}L\_\{\\mathrm\{mpc\}\}\\big\(\\boldsymbol\{\\hat\{x\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{h\},\\ell\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{l\},\\ell\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{d\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\}\\big\)s\.t\.𝒙^ℓ\+1\|kh=f^​\(𝒙^ℓ\|kh,𝒖^h,ℓ\|kh,𝒖^l,ℓ\|kh,𝒅^ℓ\|kh\),\\displaystyle\\boldsymbol\{\\hat\{x\}\}\_\{\\ell\+1\\mid k\_\{\\mathrm\{h\}\}\}=\\hat\{f\}\\big\(\\boldsymbol\{\\hat\{x\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{h\},\\ell\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{l\},\\ell\\mid k\_\{\\mathrm\{h\}\}\},\\boldsymbol\{\\hat\{d\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\}\\big\),for​ℓ∈𝕀mh​Np,\\displaystyle\\indent\\text\{for \}\\ell\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}N\_\{\\mathrm\{p\}\}\},𝒙^0\|kh=𝒙⁡\(mh​kh\),\\displaystyle\\boldsymbol\{\\hat\{x\}\}\_\{0\\mid k\_\{\\mathrm\{h\}\}\}=\\boldsymbol\{x\}\(m\_\{\\mathrm\{h\}\}k\_\{\\mathrm\{h\}\}\),\(3a\)𝒖^h,mh​j\+s\|kh=𝒖h,j\|kh,\\displaystyle\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{h\},m\_\{\\mathrm\{h\}\}j\+s\\mid k\_\{\\mathrm\{h\}\}\}=\\boldsymbol\{u\}\_\{\\mathrm\{h\},j\\mid k\_\{\\mathrm\{h\}\}\},for​s∈𝕀mh,j∈𝕀Np,\\displaystyle\\indent\\text\{for \}s\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}\},\\;j\\in\\mathbb\{I\}\_\{N\_\{\\mathrm\{p\}\}\},\(3b\)𝒖^l,ml​i\+s\|kh=πθ​\(𝒙^rl,i\|kh\),\\displaystyle\\boldsymbol\{\\hat\{u\}\}\_\{\\mathrm\{l\},m\_\{\\mathrm\{l\}\}i\+s\\mid k\_\{\\mathrm\{h\}\}\}=\\pi\_\{\\theta\}\\big\(\\boldsymbol\{\\hat\{x\}\}\_\{\\mathrm\{rl\},i\\mid k\_\{\\mathrm\{h\}\}\}\\big\),for​s∈𝕀ml,i∈𝕀q​Np,\\displaystyle\\indent\\text\{for \}s\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{l\}\}\},\\;i\\in\\mathbb\{I\}\_\{qN\_\{\\mathrm\{p\}\}\},\(3c\)𝒙^ℓ\|kh∈𝒞x,ℓ∈𝕀mh​Np\+1\\\{0\},\\displaystyle\\boldsymbol\{\\hat\{x\}\}\_\{\\ell\\mid k\_\{\\mathrm\{h\}\}\}\\in\\mathcal\{C\}\_\{\\mathrm\{x\}\},\\quad\\ell\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}N\_\{\\mathrm\{p\}\}\+1\}\\backslash\\\{0\\\},\(3d\)𝒖h,j\|kh∈𝒞uh,j∈𝕀Np\.\\displaystyle\\boldsymbol\{u\}\_\{\\mathrm\{h\},j\\mid k\_\{\\mathrm\{h\}\}\}\\in\\mathcal\{C\}\_\{\\mathrm\{u\_\{h\}\}\},\\quad j\\in\\mathbb\{I\}\_\{N\_\{\\mathrm\{p\}\}\}\.\(3e\)In \([3](https://arxiv.org/html/2609.17697#S2.E3)\),LmpcL\_\{\\mathrm\{mpc\}\}denotes the stage\-cost function, andf^\\hat\{f\}denotes the prediction model for the controlled system\. Constraint \([3a](https://arxiv.org/html/2609.17697#S2.E3.1)\) sets the initial condition of the predicted state trajectory to the measured state at the current high\-level control step\. Constraint \([3b](https://arxiv.org/html/2609.17697#S2.E3.2)\) maps the MPC control inputs from high\-level control steps to sampling steps using a ZOH strategy\. Constraint \([3c](https://arxiv.org/html/2609.17697#S2.E3.3)\) specifies that the low\-level control inputs used in the prediction are generated by a copy of the DRL policy based on the predicted DRL observation \(see Figure[1](https://arxiv.org/html/2609.17697#S2.F1)\(a\)\) and held over the corresponding low\-level control interval using a ZOH strategy\. During deployment, the policy parametersθ\\thetaremain fixed, and the same policy is used both to generate the low\-level control inputs applied to the controlled system and to generate their predictions within the MPC problem\. Finally, \([3d](https://arxiv.org/html/2609.17697#S2.E3.4)\) and \([3e](https://arxiv.org/html/2609.17697#S2.E3.5)\) collect the state and high\-level input constraints, respectively\. Here,𝒞x⊆𝒳\\mathcal\{C\}\_\{\\mathrm\{x\}\}\\subseteq\\mathcal\{X\}denotes the admissible state constraint set, and𝒞uh⊆𝒰h\\mathcal\{C\}\_\{\\mathrm\{u\_\{h\}\}\}\\subseteq\\mathcal\{U\}\_\{\\mathrm\{h\}\}denotes the high\-level control\-input constraint set imposed by the MPC controller, where𝒰h\\mathcal\{U\}\_\{\\mathrm\{h\}\}denotes the admissible high\-level control\-input set\.

The resulting high\-level control input corresponds to the first element of the optimized control sequence,𝒖h​\(kh\)=𝒖h,0\|kh⋆\\boldsymbol\{u\}\_\{\\mathrm\{h\}\}\(k\_\{\\mathrm\{h\}\}\)=\\boldsymbol\{u\}^\{\\star\}\_\{\\mathrm\{h\},0\\mid k\_\{\\mathrm\{h\}\}\}, and can be expressed as

𝒖h​\(kh\)=πmpc​\(𝒙⁡\(mh​kh\),πθ,𝜼mpc​\(kh\)\),\\boldsymbol\{u\}\_\{\\mathrm\{h\}\}\(k\_\{\\mathrm\{h\}\}\)=\\pi\_\{\\mathrm\{mpc\}\}\\left\(\\boldsymbol\{x\}\(m\_\{\\mathrm\{h\}\}k\_\{\\mathrm\{h\}\}\),\\pi\_\{\\theta\},\\boldsymbol\{\\eta\}\_\{\\mathrm\{mpc\}\}\(k\_\{\\mathrm\{h\}\}\)\\right\),\(4\)whereπmpc\\pi\_\{\\mathrm\{mpc\}\}denotes the MPC control law obtained by solving \([3](https://arxiv.org/html/2609.17697#S2.E3)\), and𝜼mpc​\(kh\)\\boldsymbol\{\\eta\}\_\{\\mathrm\{mpc\}\}\(k\_\{\\mathrm\{h\}\}\)collects the time\-varying auxiliary information required to solve the MPC problem that is not represented by the measured state or the DRL policy\. Disturbance forecasts𝒅^\\boldsymbol\{\\hat\{d\}\}and predicted DRL\-observation auxiliary information𝜼^rl\\boldsymbol\{\\hat\{\\eta\}\}\_\{\\mathrm\{rl\}\}are examples of this type of information\. The high\-level control input is applied to the controlled system using a ZOH strategy as𝒖¯h​\(mh​kh\+s\)=𝒖h​\(kh\)\\boldsymbol\{\\bar\{u\}\}\_\{\\mathrm\{h\}\}\(m\_\{\\mathrm\{h\}\}k\_\{\\mathrm\{h\}\}\+s\)=\\boldsymbol\{u\}\_\{\\mathrm\{h\}\}\(k\_\{\\mathrm\{h\}\}\),s∈𝕀mhs\\in\\mathbb\{I\}\_\{m\_\{\\mathrm\{h\}\}\}\.

## IIIComposite\-Gradient Learning

This section formulates the hierarchical control problem as an MDP and presents CGL together with a hierarchy\-aware data storage and sampling scheme\.

### III\-AMDP Formulation

Assuming Markov\-sufficient observations, the learning problem is represented by the MDP tuple

\(𝒳rl,𝒰,𝒫,r,γ\),𝒰=𝒰l×𝒰h\.\\left\(\\mathcal\{X\}\_\{\\mathrm\{rl\}\},\\mathcal\{U\},\\mathcal\{P\},r,\\gamma\\right\),\\qquad\\mathcal\{U\}=\\mathcal\{U\}\_\{\\mathrm\{l\}\}\\times\\mathcal\{U\}\_\{\\mathrm\{h\}\}\.Here,𝒳rl\\mathcal\{X\}\_\{\\mathrm\{rl\}\}is the MDP state space formed by the DRL observations,𝒰\\mathcal\{U\}is the joint action space formed by the admissible low\- and high\-level control\-input sets,𝒫\\mathcal\{P\}is the state\-transition probability kernel,r:𝒳rl×𝒰→ℝr:\\mathcal\{X\}\_\{\\mathrm\{rl\}\}\\times\\mathcal\{U\}\\rightarrow\\mathbb\{R\}is the one\-step reward, andγ\\gammais the discount factor\. One MDP step corresponds to one low\-level control interval\.

The MDP state is the DRL observation𝒙rl​\(kl\)∈𝒳rl\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)\\in\\mathcal\{X\}\_\{\\mathrm\{rl\}\}\. The joint action𝒖⁡\(kl\)=\[𝒖l⊤​\(kl\),𝒖h\|l⊤​\(kl\)\]⊤∈𝒰\\boldsymbol\{u\}\(k\_\{\\mathrm\{l\}\}\)=\[\\boldsymbol\{u\}^\{\\top\}\_\{\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\),\\boldsymbol\{u\}^\{\\top\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\)\]^\{\\top\}\\in\\mathcal\{U\}comprises the low\- and high\-level control inputs, both represented on the low\-level time scale\. Unlike the DRL input, which is recomputed at every low\-level control step, the MPC input is updated once everyqqlow\-level control steps and held constant between updates\. Accordingly, the high\-level input represented on the low\-level time scale is denoted by𝒖h\|l​\(kl\)\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\)and defined as𝒖h\|l​\(kl\)=𝒖h​\(⌊kl/q⌋\),\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\)=\\boldsymbol\{u\}\_\{\\mathrm\{h\}\}\\left\(\\left\\lfloor k\_\{\\mathrm\{l\}\}/q\\right\\rfloor\\right\),where⌊⋅⌋\\lfloor\\cdot\\rfloordenotes the floor operation, which returns the greatest integer less than or equal to its argument\.

The transition probability kernel𝒫\(⋅∣𝒙rl,𝒖\)\\mathcal\{P\}\(\\cdot\\mid\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\},\\boldsymbol\{u\}\)gives the conditional probability distribution of the next MDP state, represented by the next DRL observation\. For a current DRL observation𝒙rl\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}, a joint control input𝒖\\boldsymbol\{u\}, and any measurable set𝒳rl\+⊆𝒳rl\\mathcal\{X\}^\{\+\}\_\{\\mathrm\{rl\}\}\\subseteq\\mathcal\{X\}\_\{\\mathrm\{rl\}\}of possible next observations𝒙rl\+​\(kl\)=𝒙rl​\(kl\+1\)\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)=\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\+1\), the transition probability kernel is given by

𝒫⁡\(𝒳rl\+∣𝒙rl,𝒖\)\\displaystyle\\mathcal\{P\}\\big\(\\mathcal\{X\}^\{\+\}\_\{\\mathrm\{rl\}\}\\mid\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\},\\boldsymbol\{u\}\\big\)=Pr\(𝑿rl\(kl\+1\)∈𝒳rl\+\|𝑿rl\(kl\)=𝒙rl,𝑼\(kl\)=𝒖\),\\displaystyle=\\Pr\\\!\\left\(\\boldsymbol\{X\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\+1\)\\in\\mathcal\{X\}^\{\+\}\_\{\\mathrm\{rl\}\}\\ \\middle\|\\ \\boldsymbol\{X\}\_\{\\mathrm\{rl\}\}\(k\_\{\\mathrm\{l\}\}\)=\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\},\\boldsymbol\{U\}\(k\_\{\\mathrm\{l\}\}\)=\\boldsymbol\{u\}\\right\),wherePr\(⋅∣⋅\)\\Pr\(\\cdot\\mid\\cdot\)denotes the conditional probability operator and𝑿rl\\boldsymbol\{X\}\_\{\\mathrm\{rl\}\}and𝑼\\boldsymbol\{U\}denote random variables with realized values𝒙rl\\boldsymbol\{x\}\_\{\\mathrm\{rl\}\}and𝒖\\boldsymbol\{u\}, respectively\. One transition covers the evolution of the controlled system and the auxiliary information from sampling stepml​klm\_\{\\mathrm\{l\}\}k\_\{\\mathrm\{l\}\}to sampling stepml​\(kl\+1\)m\_\{\\mathrm\{l\}\}\(k\_\{\\mathrm\{l\}\}\+1\)\. Accordingly,𝒫\\mathcal\{P\}captures the controlled\-system dynamics in \([1](https://arxiv.org/html/2609.17697#S2.E1)\) together with the evolution of the auxiliary information over this interval\.

The rewardrrevaluates the control performance associated with the MDP state and joint action over themlm\_\{\\mathrm\{l\}\}sampling intervals that constitute the low\-level control interval\.

### III\-BLearning Algorithm

We adopt Deep Deterministic Policy Gradient \(DDPG\)\[[10](https://arxiv.org/html/2609.17697#bib.bib5)\]as the base actor\-critic algorithm for developing composite\-gradient learning \(CGL\), motivated by the successful application of DDPG across a range of control benchmarks\[[2](https://arxiv.org/html/2609.17697#bib.bib4)\]and, in particular, within the considered DRL\-MPC framework\[[6](https://arxiv.org/html/2609.17697#bib.bib8)\]\. DDPG is suitable for the considered framework because its off\-policy learning feature allows previously collected experience to be reused, reducing the repeated MPC computations required to collect new training experience\.

Building on DDPG, CGL learns a deterministic DRL policy through an actor networkπθ\\pi\_\{\\theta\}, parameterized byθ\\theta, which maps the observed state to a DRL action\. A critic networkQϕQ\_\{\\phi\}, parameterized byϕ\\phi, estimates the expected discounted return associated with a state\-action pair under the current policy\. To improve learning stability, CGL also maintains a target actor networkπθ′\\pi\_\{\\theta^\{\\prime\}\}and a target critic networkQϕ′Q\_\{\\phi^\{\\prime\}\}, whose parametersθ′\\theta^\{\\prime\}andϕ′\\phi^\{\\prime\}are updated gradually relative to those of the main networks\. Past experiences are stored in a replay buffer𝒟\\mathcal\{D\}, with the oldest experiences removed once the buffer reaches its capacity\. At each network update, a mini\-batchℳ\\mathcal\{M\}comprisingMMstored experiences is sampled from𝒟\\mathcal\{D\}and used to calculate the actor and critic updates\.

For each low\-level transition, CGL requires the MPC\-initialization states associated with the current and next MDP states\. The current MPC\-initialization state is used to calculate the high\-level control input and its contribution to the composite policy gradient \(see \([9](https://arxiv.org/html/2609.17697#S3.E9)\) and \([13](https://arxiv.org/html/2609.17697#S3.E13)\)\), whereas the next MPC\-initialization state is used to construct the critic target \(see \([6](https://arxiv.org/html/2609.17697#S3.E6)\) and \([7](https://arxiv.org/html/2609.17697#S3.E7)\)\)\.

At low\-level control stepklk\_\{\\mathrm\{l\}\}, the MPC\-initialization state is defined as

𝒙mpc​\(kl\)=𝒙⁡\(mh​⌊klq⌋\),\\boldsymbol\{x\}\_\{\\mathrm\{mpc\}\}\(k\_\{\\mathrm\{l\}\}\)=\\boldsymbol\{x\}\\left\(m\_\{\\mathrm\{h\}\}\\left\\lfloor\\frac\{k\_\{\\mathrm\{l\}\}\}\{q\}\\right\\rfloor\\right\),\(5\)which is the measured controlled\-system state used as an input toπmpc\\pi\_\{\\mathrm\{mpc\}\}in \([4](https://arxiv.org/html/2609.17697#S2.E4)\) at the most recent MPC update\. Consequently,𝒙mpc​\(kl\)\\boldsymbol\{x\}\_\{\\mathrm\{mpc\}\}\(k\_\{\\mathrm\{l\}\}\)remains unchanged over theqqlow\-level control steps between two consecutive MPC updates\.

Letkl,ik\_\{\\mathrm\{l\},i\}denote the low\-level control step at which experienceiiis collected\. The MPC\-initialization states associated with the current and next MDP states of experienceiiare denoted by

𝒙mpc,i=𝒙mpc​\(kl,i\),𝒙mpc,i\+=𝒙mpc​\(kl,i\+1\),\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},i\}=\\boldsymbol\{x\}\_\{\\mathrm\{mpc\}\}\(k\_\{\\mathrm\{l\},i\}\),\\qquad\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{mpc\},i\}=\\boldsymbol\{x\}\_\{\\mathrm\{mpc\}\}\(k\_\{\\mathrm\{l\},i\}\+1\),respectively, and are identical unless the transition ends at a high\-level control input update instant\. CGL stores the augmented experience tuples\(𝒙mpc,i,𝒙rl,i,𝒖i,ri,𝒙mpc,i\+,𝒙rl,i\+,δi\)\(\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},i\},\\allowbreak\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\},\\allowbreak\\boldsymbol\{u\}\_\{i\},\\allowbreak r\_\{i\},\\allowbreak\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{mpc\},i\},\\allowbreak\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{rl\},i\},\\allowbreak\\delta\_\{i\}\)in the replay buffer𝒟\\mathcal\{D\}, whereδi∈\{0,1\}\\delta\_\{i\}\\in\\\{0,1\\\}indicates whether the transition is terminal\.

For an experienceiisampled from the replay buffer, the critic target is calculated as

yi=ri\+γ⁡\(1−δi\)​Qϕ′​\(𝒙rl,i\+,𝒖θ′,i\+\),y\_\{i\}=r\_\{i\}\+\\gamma\(1\-\\delta\_\{i\}\)Q\_\{\\phi^\{\\prime\}\}\\left\(\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{rl\},i\},\\boldsymbol\{u\}^\{\+\}\_\{\\theta^\{\\prime\},i\}\\right\),\(6\)where

𝒖θ′,i\+=\[𝒖l,θ′,i\+𝒖h\|l,θ′,i\+\]\\boldsymbol\{u\}^\{\+\}\_\{\\theta^\{\\prime\},i\}=\\begin\{bmatrix\}\\boldsymbol\{u\}^\{\+\}\_\{\\mathrm\{l\},\\theta^\{\\prime\},i\}\\\\ \\boldsymbol\{u\}^\{\+\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta^\{\\prime\},i\}\\end\{bmatrix\}is the joint control input of the target policy associated with the next MDP state\. The low\-level control input of the target policy is calculated using the target actor network as𝒖l,θ′,i\+=πθ′​\(𝒙rl,i\+\)\.\\boldsymbol\{u\}^\{\+\}\_\{\\mathrm\{l\},\\theta^\{\\prime\},i\}=\\pi\_\{\\theta^\{\\prime\}\}\(\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{rl\},i\}\)\.The high\-level control input of the target policy, represented on the low\-level time scale, is calculated as

𝒖h\|l,θ′,i\+=πmpc​\(𝒙mpc,i\+,πθ′,𝜼mpc,i\+\),\\boldsymbol\{u\}^\{\+\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta^\{\\prime\},i\}=\\pi\_\{\\mathrm\{mpc\}\}\\left\(\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{mpc\},i\},\\pi\_\{\\theta^\{\\prime\}\},\\boldsymbol\{\\eta\}^\{\+\}\_\{\\mathrm\{mpc\},i\}\\right\),\(7\)where𝜼mpc,i\+\\boldsymbol\{\\eta\}^\{\+\}\_\{\\mathrm\{mpc\},i\}denotes the auxiliary information associated with the MPC update relevant to the next low\-level control step\.

The loss for the critic network is calculated as

L⁡\(ϕ\)=1M​∑i=1M\(yi−Qϕ​\(𝒙rl,i,𝒖i\)\)2,L\(\\phi\)=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\left\(y\_\{i\}\-Q\_\{\\phi\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\},\\boldsymbol\{u\}\_\{i\}\\right\)\\right\)^\{2\},\(8\)whereiiindexes theMMexperiences in the mini\-batchℳ\\mathcal\{M\}\. The critic parametersϕ\\phiare updated by minimizing \([8](https://arxiv.org/html/2609.17697#S3.E8)\) using a gradient\-based optimizer, such as adaptive moment estimation \(Adam\)\[[11](https://arxiv.org/html/2609.17697#bib.bib3)\]\.

The actor parametersθ\\thetaare subsequently updated by maximizing the return estimated by the critic\. For experienceii, the joint control input induced by the current actor policy is defined as𝒖θ,i=\[𝒖l,θ,i⊤,𝒖h\|l,θ,i⊤\]⊤\\boldsymbol\{u\}\_\{\\theta,i\}=\[\\boldsymbol\{u\}^\{\\top\}\_\{\\mathrm\{l\},\\theta,i\},\\boldsymbol\{u\}^\{\\top\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta,i\}\]^\{\\top\}\. The low\-level component is calculated as𝒖l,θ,i=πθ​\(𝒙rl,i\)\\boldsymbol\{u\}\_\{\\mathrm\{l\},\\theta,i\}=\\pi\_\{\\theta\}\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\}\)\. The corresponding high\-level component is calculated as

𝒖h\|l,θ,i=πmpc​\(𝒙mpc,i,πθ,𝜼mpc,i\)\.\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta,i\}=\\pi\_\{\\mathrm\{mpc\}\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},i\},\\pi\_\{\\theta\},\\boldsymbol\{\\eta\}\_\{\\mathrm\{mpc\},i\}\\right\)\.\(9\)Unlike the joint input𝒖i\\boldsymbol\{u\}\_\{i\}stored in the replay buffer,𝒖θ,i\\boldsymbol\{u\}\_\{\\theta,i\}is calculated using the current actor parameters\. The actor objective is defined as

J⁡\(θ\)=1M​∑i=1MQϕ​\(𝒙rl,i,𝒖θ,i\)\.J\(\\theta\)=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}Q\_\{\\phi\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\},\\boldsymbol\{u\}\_\{\\theta,i\}\\right\)\.\(10\)Based on \([10](https://arxiv.org/html/2609.17697#S3.E10)\), the composite policy gradient is decomposed as

∇θJ​\(θ\)=𝒈rl\+𝒈mpc,\\nabla\_\{\\theta\}J\(\\theta\)=\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}\+\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\},where𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}and𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}denote the DRL and MPC contributions to the composite policy gradient, respectively \(see Figure[2](https://arxiv.org/html/2609.17697#S3.F2)\)\. The DRL contribution is calculated as

𝒈rl=1M​∑i=1M\(∇θ𝒖l,θ,i\)⊤​∇𝒖lQϕ​\(𝒙rl,i,𝒖θ,i\)\.\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\left\(\\nabla\_\{\\theta\}\\boldsymbol\{u\}\_\{\\mathrm\{l\},\\theta,i\}\\right\)^\{\\\!\\top\}\\nabla\_\{\\boldsymbol\{u\}\_\{\\mathrm\{l\}\}\}Q\_\{\\phi\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\},\\boldsymbol\{u\}\_\{\\theta,i\}\\right\)\.\(11\)The sensitivity∇θ𝒖l,θ,i\\nabla\_\{\\theta\}\\boldsymbol\{u\}\_\{\\mathrm\{l\},\\theta,i\}in \([11](https://arxiv.org/html/2609.17697#S3.E11)\) is obtained by differentiating the actor network with respect to the actor parametersθ\\thetaas∇θ𝒖l,θ,i=∇θπθ​\(𝒙rl,i\)\.\\nabla\_\{\\theta\}\\boldsymbol\{u\}\_\{\\mathrm\{l\},\\theta,i\}=\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\}\\right\)\.

The MPC contribution is calculated as

𝒈mpc=1M​∑i=1M\(∇θ𝒖h\|l,θ,i\)⊤​∇𝒖h\|lQϕ​\(𝒙rl,i,𝒖θ,i\)\.\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\left\(\\nabla\_\{\\theta\}\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta,i\}\\right\)^\{\\\!\\top\}\\nabla\_\{\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\}\}\}Q\_\{\\phi\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\},\\boldsymbol\{u\}\_\{\\theta,i\}\\right\)\.\(12\)The sensitivity∇θ𝒖h\|l,θ,i\\nabla\_\{\\theta\}\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta,i\}in \([12](https://arxiv.org/html/2609.17697#S3.E12)\) is obtained by differentiating the MPC control law111The MPC problem is assumed to admit a locally unique solution and to have a control law that is differentiable with respect toθ\\thetaat the sampled experiences\.as

∇θ𝒖h\|l,θ,i=∇θπmpc​\(𝒙mpc,i,πθ,𝜼mpc,i\)\.\\nabla\_\{\\theta\}\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta,i\}=\\nabla\_\{\\theta\}\\pi\_\{\\mathrm\{mpc\}\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},i\},\\pi\_\{\\theta\},\\boldsymbol\{\\eta\}\_\{\\mathrm\{mpc\},i\}\\right\)\.\(13\)Here,∇θπmpc\\nabla\_\{\\theta\}\\pi\_\{\\mathrm\{mpc\}\}denotes the derivative of the MPC control input with respect to the actor parameters through the policyπθ\\pi\_\{\\theta\}embedded in the MPC problem\. It therefore accounts for the effect of the actor parameters on the predicted low\-level control inputs in \([3c](https://arxiv.org/html/2609.17697#S2.E3.3)\) and, consequently, on the optimized high\-level control input\. Thus, the DRL contribution in \([11](https://arxiv.org/html/2609.17697#S3.E11)\) captures the direct effect of the DRL action on the estimated return, whereas the MPC contribution in \([12](https://arxiv.org/html/2609.17697#S3.E12)\) captures the indirect effect of the DRL policy through the MPC control input, as illustrated in Figure[2](https://arxiv.org/html/2609.17697#S3.F2)\. The actor parameters are then updated using gradient ascent as

θ←θ\+αθ​∇θJ​\(θ\),\\theta\\leftarrow\\theta\+\\alpha\_\{\\theta\}\\nabla\_\{\\theta\}J\(\\theta\),\(14\)whereαθ\>0\\alpha\_\{\\theta\}\>0denotes the actor learning rate\.

Fig\. 2:Computational graph of the direct DRL contribution𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}and indirect MPC contribution𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}within one group starting atii\.Following the critic and actor updates, the target\-network parameters are updated using Polyak averaging as in\[[10](https://arxiv.org/html/2609.17697#bib.bib5)\]to improve learning stability as

θ′←τ​θ\+\(1−τ\)​θ′,ϕ′←τ​ϕ\+\(1−τ\)​ϕ′,\\theta^\{\\prime\}\\leftarrow\\tau\\theta\+\(1\-\\tau\)\\theta^\{\\prime\},\\quad\\phi^\{\\prime\}\\leftarrow\\tau\\phi\+\(1\-\\tau\)\\phi^\{\\prime\},
whereτ∈\(0,1\)\\tau\\in\(0,1\)is the smoothing factor\.

During experience collection, noise is added to the low\-level DRL actions to encourage exploration\. Before being applied to the controlled system, each resulting exploratory low\-level action is projected into the admissible control\-input set𝒰l\\mathcal\{U\}\_\{\\mathrm\{l\}\}\.

Fig\. 3:Hierarchy\-aware reuse of MPC evaluations under the target policy across consecutive experience groups\. The red\-hatched elements show the excluded final transition and its avoided additional MPC evaluation\.
### III\-CHierarchy\-Aware Data Storage and Sampling Scheme

The critic target in \([6](https://arxiv.org/html/2609.17697#S3.E6)\) requires evaluating the MPC control law under the target actor policy according to \([7](https://arxiv.org/html/2609.17697#S3.E7)\)\. For the actor update in \([14](https://arxiv.org/html/2609.17697#S3.E14)\), the MPC contribution in \([12](https://arxiv.org/html/2609.17697#S3.E12)\) requires evaluating the control law under the current actor policy and calculating its sensitivity according to \([13](https://arxiv.org/html/2609.17697#S3.E13)\)\. For fixed actor parameters, MPC evaluations with the same MPC\-initialization state and auxiliary information yield identical control inputs and sensitivities\. Accordingly, the proposed data storage and sampling scheme groups transitions by their associated MPC updates to reuse these computations during training\.

Letjjindex the groups, each containing theqqlow\-level transitions between two consecutive MPC updates\. Letbjb\_\{j\}denote the index of the first experience in groupjj, so its experiences have indicesbj\+sb\_\{j\}\+sfors∈𝕀qs\\in\\mathbb\{I\}\_\{q\}\(see Figure[3](https://arxiv.org/html/2609.17697#S3.F3)\)\. The transition\-specific entries of the augmented experience tuple are denoted by𝝃i=\(𝒙rl,i,𝒖i,ri,𝒙rl,i\+,δi\)\\boldsymbol\{\\xi\}\_\{i\}=\(\\boldsymbol\{x\}\_\{\\mathrm\{rl\},i\},\\boldsymbol\{u\}\_\{i\},r\_\{i\},\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{rl\},i\},\\delta\_\{i\}\)\. The replay buffer𝒟\\mathcal\{D\}stores these transitions in groups of the formEj=\(𝒙mpc,bj,\{𝝃bj\+s\}s∈𝕀q\)E\_\{j\}=\(\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},b\_\{j\}\},\\\{\\boldsymbol\{\\xi\}\_\{b\_\{j\}\+s\}\\\}\_\{s\\in\\mathbb\{I\}\_\{q\}\}\)\.

Since the MPC\-initialization state𝒙mpc,bj\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},b\_\{j\}\}is shared by all transitions in groupjj, it is stored only once\. Experiences and groups are indexed in collection order, withbj\+1=bj\+qb\_\{j\+1\}=b\_\{j\}\+qfor consecutive groups within an episode\. According to \([5](https://arxiv.org/html/2609.17697#S3.E5)\), the next MPC\-initialization state associated with transition𝝃bj\+s\\boldsymbol\{\\xi\}\_\{b\_\{j\}\+s\}is given by

𝒙mpc,bj\+s\+=\{𝒙mpc,bj,s<q−1,𝒙mpc,bj\+1,s=q−1\.\\boldsymbol\{x\}^\{\+\}\_\{\\mathrm\{mpc\},b\_\{j\}\+s\}=\\begin\{cases\}\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},b\_\{j\}\},&s<q\-1,\\\\ \\boldsymbol\{x\}\_\{\\mathrm\{mpc\},b\_\{j\+1\}\},&s=q\-1\.\\end\{cases\}
For groupjj, the MPC control law in \([4](https://arxiv.org/html/2609.17697#S2.E4)\) is evaluated under either actor policy as

𝒖h\|l,ϑ,bj=πmpc​\(𝒙mpc,bj,πϑ,𝜼mpc,bj\),\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\vartheta,b\_\{j\}\}=\\pi\_\{\\mathrm\{mpc\}\}\\left\(\\boldsymbol\{x\}\_\{\\mathrm\{mpc\},b\_\{j\}\},\\pi\_\{\\vartheta\},\\boldsymbol\{\\eta\}\_\{\\mathrm\{mpc\},b\_\{j\}\}\\right\),whereϑ∈\{θ,θ′\}\\vartheta\\in\\\{\\theta,\\theta^\{\\prime\}\\\}and𝜼mpc,bj\\boldsymbol\{\\eta\}\_\{\\mathrm\{mpc\},b\_\{j\}\}denotes the auxiliary information associated with the MPC update of groupjj\. The high\-level control input of the current policy in \([9](https://arxiv.org/html/2609.17697#S3.E9)\) and its sensitivity in \([13](https://arxiv.org/html/2609.17697#S3.E13)\) are calculated once per group because their values are identical for all experiences in groupjj\.

Using the MPC control inputs evaluated for each group, the high\-level control input of the target policy in \([7](https://arxiv.org/html/2609.17697#S3.E7)\) is obtained as

𝒖h\|l,θ′,bj\+s\+=\{𝒖h\|l,θ′,bj,s<q−1,𝒖h\|l,θ′,bj\+1,s=q−1\.\\boldsymbol\{u\}^\{\+\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta^\{\\prime\},b\_\{j\}\+s\}=\\begin\{cases\}\\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta^\{\\prime\},b\_\{j\}\},&s<q\-1,\\\\ \\boldsymbol\{u\}\_\{\\mathrm\{h\}\\mid\\mathrm\{l\},\\theta^\{\\prime\},b\_\{j\+1\}\},&s=q\-1\.\\end\{cases\}Consequently, the critic targets in \([6](https://arxiv.org/html/2609.17697#S3.E6)\) for the last transition of groupj−1j\-1and the firstq−1q\-1transitions of groupjjuse the same high\-level control input of the target policy \(see Figure[3](https://arxiv.org/html/2609.17697#S3.F3)\)\.

At each network update,BBsequences ofNeN\_\{\\mathrm\{e\}\}consecutive groups from the same episode are sampled uniformly from𝒟\\mathcal\{D\}, with starting group indicesj0\(r\)j\_\{0\}^\{\(r\)\},r∈\{1,…,B\}r\\in\\\{1,\\ldots,B\\\}\. The final transition of each selected sequence is excluded to avoid evaluating \([7](https://arxiv.org/html/2609.17697#S3.E7)\) for the subsequent, unselected group, as illustrated in Figure[3](https://arxiv.org/html/2609.17697#S3.F3)\. For tasks with a terminal condition, the final episode transition can instead be retained to learn that no future reward follows, usingyi=riy\_\{i\}=r\_\{i\}\. The mini\-batch is defined as

ℳ=⨄r=1B\{𝝃i∣bj0\(r\)≤i<bj0\(r\)\+q​Ne−1\},\\mathcal\{M\}=\\biguplus\_\{r=1\}^\{B\}\\left\\\{\\boldsymbol\{\\xi\}\_\{i\}\\mid b\_\{j\_\{0\}^\{\(r\)\}\}\\leq i<b\_\{j\_\{0\}^\{\(r\)\}\}\+qN\_\{\\mathrm\{e\}\}\-1\\right\\\},where⨄\\biguplusdenotes multiset union, preserving repeated transitions if sampled sequences overlap\. TheM=B⁡\(q​Ne−1\)M=B\(qN\_\{\\mathrm\{e\}\}\-1\)augmented experience tuples are reconstructed from the selected transitions and their group\-level information, then relabeled withi∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}for the critic loss in \([8](https://arxiv.org/html/2609.17697#S3.E8)\) and actor objective in \([10](https://arxiv.org/html/2609.17697#S3.E10)\)\. Each sequence requiresNeN\_\{\\mathrm\{e\}\}target\-policy MPC evaluations for itsq​Ne−1qN\_\{\\mathrm\{e\}\}\-1critic targets in \([6](https://arxiv.org/html/2609.17697#S3.E6)\)\. Likewise,NeN\_\{\\mathrm\{e\}\}current\-policy MPC evaluations and sensitivity calculations in \([13](https://arxiv.org/html/2609.17697#S3.E13)\) are reused across these transitions for the MPC gradient contribution in \([12](https://arxiv.org/html/2609.17697#S3.E12)\)\.

The number of MPC evaluations under the target policy per critic target using the proposed hierarchy\-aware data storage and sampling scheme is thereforeNe/\(q​Ne−1\)N\_\{\\mathrm\{e\}\}/\(qN\_\{\\mathrm\{e\}\}\-1\)\. If groups were sampled independently rather than consecutively, each group would require MPC evaluations under the target policy for both itself and its successor\. Without reuse between sampled groups, retaining allqqtransitions per group would therefore require2​Ne2N\_\{\\mathrm\{e\}\}evaluations forq​NeqN\_\{\\mathrm\{e\}\}critic targets, giving2​Ne/\(q​Ne\)=2/q2N\_\{\\mathrm\{e\}\}/\(qN\_\{\\mathrm\{e\}\}\)=2/qevaluations per target\. For the proposed scheme, the ratioNe/\(q​Ne−1\)N\_\{\\mathrm\{e\}\}/\(qN\_\{\\mathrm\{e\}\}\-1\)decreases monotonically toward1/q1/qasNeN\_\{\\mathrm\{e\}\}increases, approaching half the number of MPC evaluations per critic target required by independent sampling\. By reusing identical MPC computations under fixed network parameters, the proposed scheme reduces the MPC evaluation cost per retained transition without introducing additional approximations into its critic target or composite\-gradient terms\.

## IVCase Study

This section evaluates CGL on two multi\-class freeway traffic networks and compares it with alternative methods\.

### IV\-AFreeway Traffic Control Settings

We consider the two freeway networks in Figure[4](https://arxiv.org/html/2609.17697#S4.F4), each with two vehicle classes\. We use a multi\-class METANET model\[[7](https://arxiv.org/html/2609.17697#bib.bib10)\]to simulate these networks with a sampling interval ofT=10T=10s, owing to its balance between computational efficiency and modeling accuracy\. Each 1\-km link contributes seven state elements comprising the class\-dependent mean speeds, densities, and outflows and the total density, while each origin contributes four state elements comprising the class\-dependent queue lengths and outflows\. Network 1 has nine links and three origins, yielding a 75\-dimensional state vector𝒙⁡\(k\)\\boldsymbol\{x\}\(k\)\. Network 2 has eight links and four origins, yielding a 72\-dimensional state vector\. At each step, the demand vector𝒅⁡\(k\)\\boldsymbol\{d\}\(k\)contains66class\-specific demands of the three exogenous demand streams\.

![Refer to caption](https://arxiv.org/html/2609.17697v1/figures/network_structure_1.png)Fig\. 4:Freeway layouts for Network 1 \(top\) and Network 2 \(bottom\)\. The green\-and\-red arrow symbol denotes the route split controlled byuhu\_\{\\mathrm\{h\}\}, and the traffic\-light symbols denote the ramp\-metering locations controlled byul,1u\_\{\\mathrm\{l\},1\}andul,2u\_\{\\mathrm\{l\},2\}\.Each network uses one vehicle\-splitting input,uhu\_\{\\mathrm\{h\}\}, to distribute traffic between two routes and two ramp\-metering inputs,ul,1u\_\{\\mathrm\{l\},1\}andul,2u\_\{\\mathrm\{l\},2\}, to regulate traffic entering the freeway from the two controlled on\-ramps\. The network structures yield different coupling strengths\. In Network 1,uhu\_\{\\mathrm\{h\}\}splits mainline traffic between two routes that later merge with separately metered on\-ramps\. In Network 2,uhu\_\{\\mathrm\{h\}\}directly splits a shared demand stream between the two metered on\-ramps, creating stronger bidirectional coupling with the ramp\-metering actions\.

The low\-level DRL action𝒖l=\[ul,1,ul,2\]⊤∈\[0,1\]2\\boldsymbol\{u\}\_\{\\mathrm\{l\}\}=\[u\_\{\\mathrm\{l\},1\},u\_\{\\mathrm\{l\},2\}\]^\{\\top\}\\in\[0,1\]^\{2\}determines two ramp\-metering rates everyTl=120T\_\{\\mathrm\{l\}\}=120s\. The high\-level MPC inputuh∈\[0,1\]u\_\{\\mathrm\{h\}\}\\in\[0,1\]is a traffic splitting rate updated everyTh=600T\_\{\\mathrm\{h\}\}=600s\. Hence,q=5q=5, and each MPC input is held fixed while five successive DRL actions are applied\. The MPC prediction horizon is 20 min, and each controlled episode lasts 30 min\. Initial traffic conditions for both networks are established through a warm\-up period before controlled operation\.

### IV\-BLearning Methods and Experimental Setup

CGL is compared with three alternative learning methods: CGL\-D \(direct\), CGL\-S \(scaled\), and MPC\-E\. CGL\-D uses the joint\-action critic but updates the actor using only the direct gradient𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}, omitting the indirect contribution𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}through MPC\. CGL\-S uses𝒈rl\+α​𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}\+\\alpha\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}withα=nh​‖𝒈rl‖nl​‖𝒈mpc‖\\alpha=\\frac\{n\_\{\\mathrm\{h\}\}\\\|\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}\\\|\}\{n\_\{\\mathrm\{l\}\}\\\|\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}\\\|\}to balance the DRL and MPC contribution norms in proportion to their control input counts, where∥⋅∥\\\|\\cdot\\\|denotes the Euclidean norm andnln\_\{\\mathrm\{l\}\}andnhn\_\{\\mathrm\{h\}\}are the dimensions of the low\- and high\-level control input vectors, respectively\. MPC\-E treats MPC as part of the environment, omits the high\-level MPC input from the critic’s action input, and updates the actor using only𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}\.

The actor and critic use the reduced observation𝒙~rl\\tilde\{\\boldsymbol\{x\}\}\_\{\\mathrm\{rl\}\}, which augments the traffic state with the current demand vector and the previously applied splitting rate\. The reduced observation is treated as an approximate MDP state and has dimensions 82 and 79 for Networks 1 and 2, respectively\.

The low\-level reward over each 120\-s interval is the negative sum of the total time spent \(TTS\), a queue\-constraint penalty, and an input\-change penalty\. Here, the TTS is the aggregate time that vehicles spend traveling on the freeway links or waiting in the origin queues, so reducing it promotes efficient traffic movement\. The queue\-constraint penalty is one\-sided and squared: no penalty is applied below a queue limit, and the penalty grows quadratically above it\. Queue limits are 200 vehicles at mainline origins in both networks and 50 and 120 at each controlled on\-ramp in Networks 1 and 2, respectively\. Network 2 also includes a density\-constraint penalty for the total density above3333veh/km/lane downstream of each controlled merge, with a weight of 2000\. The input\-change penalty applies to changes in all three control inputs with a weight of 0\.4, encouraging smoother control operation\. The reward is scaled by1/301/30to control its magnitude, motivated by evidence that reward scaling can affect learning speed and asymptotic performance in off\-policy actor\-critic learning\[[12](https://arxiv.org/html/2609.17697#bib.bib6)\]\. MPC minimizes the negative of this reward over a prediction horizon ofNp=2N\_\{\\mathrm\{p\}\}=2high\-level control steps, equivalent to 120 sampling steps, thereby aligning its objective with the DRL objective\.

Similarly to\[[4](https://arxiv.org/html/2609.17697#bib.bib7)\], the actor has two 256\-unit hidden layers with rectified linear unit activations and layer normalization\. The joint\-action critic processes the observation and joint action through separate 256\- and 128\-unit branches, followed by 256\- and 128\-unit hidden layers\. All methods use a replay\-buffer capacity of 10 000, a configured mini\-batch size of 120,γ=0\.99\\gamma=0\.99, actor and critic learning rates of10−310^\{\-3\}, andτ=0\.01\\tau=0\.01\. The hierarchy\-aware sampling scheme usesNe=2N\_\{\\mathrm\{e\}\}=2, yielding 108 retained transitions acrossB=12B=12sampled sequences per update\. Decaying Ornstein\-Uhlenbeck noise is added to the DRL actions during training, as suggested in\[[10](https://arxiv.org/html/2609.17697#bib.bib5)\]\.

The nonlinear optimization problem in \([3](https://arxiv.org/html/2609.17697#S2.E3)\) is solved using Optimistix\[[13](https://arxiv.org/html/2609.17697#bib.bib13)\]with at most 500 iterations\. The sensitivity in \([13](https://arxiv.org/html/2609.17697#S3.E13)\) is computed by reverse\-mode automatic differentiation of the finite\-iteration solver map using a recursive checkpoint adjoint scheme, which, unlike implicit differentiation, does not require the solver to converge to a local minimum\[[13](https://arxiv.org/html/2609.17697#bib.bib13)\]\.

Each method is trained using 10 independent neural\-network\-initialization and exploration seeds, with 960 complete episodes per run distributed over 64 parallel environments\. The implementations and full evaluation outputs are available athttps://github\.com/GirayOnur/composite\-gradient\-learning\.

### IV\-CResults and Discussion

Figure[5](https://arxiv.org/html/2609.17697#S4.F5)compares the evaluation returns obtained by the learning methods in both networks\. The return is the sum of the undiscounted rewards over one episode\.

Fig\. 5:Evaluation returns \(higher is better\) for Network 1 \(left\) and Network 2 \(right\)\. Each point shows the mean return of one trained policy over 100 evaluations with different demand\-noise realizations\. Horizontal bars show the mean over the 10 policies for each method\.For Network 1, the return distributions overlap strongly, and the variation across trained policies is larger than the differences between the method means\. Taking CGL as the reference, CGL\-D has a lower mean return, whereas MPC\-E has a higher mean return\. The large overlap shows no clear advantage for any method in Network 1\. For Network 2, the mean returns of CGL\-D and MPC\-E are both lower than that of CGL by approximately 2\.4% of its magnitude\. Their trained policies reach nearly identical returns, while three of the 10 CGL policies achieve distinctly higher returns\. These results suggest that CGL may learn higher\-performing control policies than the alternative methods in a subset of training runs when the DRL and MPC control inputs interact strongly, although the average control\-performance gains remain modest\. The similar returns of CGL\-D and MPC\-E also show that including the high\-level input in the critic is not sufficient to achieve this improvement; the actor update must also include the indirect contribution through MPC\. Figure[5](https://arxiv.org/html/2609.17697#S4.F5)also shows that CGL\-S does not improve on CGL in Network 1 despite amplifying𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}, whose magnitude is much smaller than that of𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}in this network \(see also Figure[6](https://arxiv.org/html/2609.17697#S4.F6)\)\. It is therefore not considered in Network 2, where the stronger DRL\-MPC interaction results in a larger indirect MPC contribution\.

Figure[6](https://arxiv.org/html/2609.17697#S4.F6)shows the norm share of the indirect MPC gradient, expressed as a percentage, and the cosine similarity for CGL during training\. The norm share of the indirect MPC gradient is‖𝒈mpc‖/\(‖𝒈rl‖\+‖𝒈mpc‖\)\\\|\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}\\\|/\(\\\|\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}\\\|\+\\\|\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}\\\|\)expressed as a percentage, measuring the relative magnitude of the indirect MPC contribution\. The cosine similarity measures the directional alignment between𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}and𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}, where positive values indicate similar directions and negative values indicate opposing directions\.

Fig\. 6:Distributions of the norm share of the indirect MPC gradient \(left\) and branch cosine similarity between𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}and𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}\(right\) during CGL training\. Each network has 1 720 logged updates \(172 per policy\); the Network 1 norm\-share distribution excludes one zero\-gradient update\. Violin shapes, boxes, horizontal lines, and diamonds show the distributions, interquartile ranges, medians, and means, respectively\.In Network 1, the mean share of𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}is only3\.0%3\.0\\%, and the median cosine similarity between𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}and𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}is 0\.03\. In Network 2, the mean share of𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}increases to27\.3%27\.3\\%, possibly due to the stronger interaction between the DRL and MPC control inputs, giving the indirect MPC contribution a larger role in learning\. Moreover, the median cosine similarity decreases to−0\.40\-0\.40, showing that𝒈mpc\\boldsymbol\{g\}\_\{\\mathrm\{mpc\}\}often opposes𝒈rl\\boldsymbol\{g\}\_\{\\mathrm\{rl\}\}, while including this opposing contribution can redirect the actor update toward the higher\-return control policies shown in Figure[5](https://arxiv.org/html/2609.17697#S4.F5)\.

## VConclusions

This paper has proposed composite\-gradient learning \(CGL\) for hierarchical control frameworks in which deep reinforcement learning \(DRL\) and model predictive control \(MPC\) share control authority\. CGL integrates the MPC controller into the learning process by representing the DRL and MPC control inputs as a joint action and accounting for their interaction when updating the DRL agent during training\. This paper has also proposed a hierarchy\-aware data storage and sampling scheme that enables the reuse of MPC computations during training to reduce the computational burden of CGL\.

In the freeway traffic control case studies, CGL offered limited benefit when the DRL and MPC control inputs interacted weakly\. Under strong interaction, however, CGL learned policies that markedly outperformed the alternatives in some training runs, highlighting its potential to improve control performance despite modest average gains\.

Future work will focus on detecting strong DRL\-MPC control input coupling and selectively computing and using the composite gradient in such cases to improve the computational efficiency of CGL\.

## References

- \[1\]J\. B\. Rawlings, D\. Q\. Mayne, and M\. M\. Diehl\(2017\)Model predictive control: theory, computation, and design\.2nd edition,Nob Hill Publishing,Madison, WI, USA\.Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p1.1)\.
- \[2\]X\. Wang, S\. Wang, X\. Liang, D\. Zhao, J\. Huang, X\. Xu, B\. Dai, and Q\. Miao\(2024\)Deep reinforcement learning: a survey\.IEEE Transactions on Neural Networks and Learning Systems35\(4\),pp\. 5064–5078\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2022.3207346)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p1.1),[§III\-B](https://arxiv.org/html/2609.17697#S3.SS2.p1.1)\.
- \[3\]R\. Reiter, J\. Hoffmann, D\. Reinhardt, F\. Messerer, K\. Baumgärtner, S\. Sawant, J\. Bödecker, M\. Diehl, and S\. Gros\(2026\)Synthesis of model predictive control and reinforcement learning: survey and classification\.Annual Reviews in Control61\.Note:Art\. no\. 101045External Links:[Document](https://dx.doi.org/10.1016/j.arcontrol.2026.101045)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p2.1),[§I](https://arxiv.org/html/2609.17697#S1.p4.1),[§I](https://arxiv.org/html/2609.17697#S1.p6.1)\.
- \[4\]D\. Sun, A\. Jamshidnejad, and B\. De Schutter\(2024\)A novel framework combining MPC and deep reinforcement learning with application to freeway traffic control\.IEEE Transactions on Intelligent Transportation Systems25\(7\),pp\. 6756–6769\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2023.3342651)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p2.1),[§IV\-B](https://arxiv.org/html/2609.17697#S4.SS2.p4.1)\.
- \[5\]M\. A\. Aljabri, M\. O\. Bahabri, N\. A\. Alakhrash, F\. A\. Hariri, and M\. N\. Ajour\(2026\)AMI\-informed hierarchical deep reinforcement learning–model predictive control for coordinated EV, PV, and battery energy management in campus microgrids\.Energies19\(13\)\.Note:Art\. no\. 3210External Links:[Document](https://dx.doi.org/10.3390/en19133210)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p2.1)\.
- \[6\]G\. Onur, A\. Dabiri, and B\. De Schutter\(2026\)Sharing the control authority between deep reinforcement learning and model predictive control: application to multi\-class transportation networks\.arXiv preprint arXiv:2608\.20858\.Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p3.1),[§I](https://arxiv.org/html/2609.17697#S1.p4.1),[§II](https://arxiv.org/html/2609.17697#S2.p1.1),[§III\-B](https://arxiv.org/html/2609.17697#S3.SS2.p1.1)\.
- \[7\]C\. Pasquale, S\. Sacone, S\. Siri, and B\. De Schutter\(2017\)A multi\-class model\-based control scheme for reducing congestion and emissions in freeway networks by combining ramp metering and route guidance\.Transportation Research Part C: Emerging Technologies80,pp\. 384–408\.External Links:[Document](https://dx.doi.org/10.1016/j.trc.2017.04.007)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p3.1),[§IV\-A](https://arxiv.org/html/2609.17697#S4.SS1.p1.1)\.
- \[8\]U\. Rosolia, A\. Singletary, and A\. D\. Ames\(2022\)Unified multirate control: from low\-level actuation to high\-level planning\.IEEE Transactions on Automatic Control67\(12\),pp\. 6627–6640\.External Links:[Document](https://dx.doi.org/10.1109/TAC.2022.3184664)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p3.1)\.
- \[9\]C\. Xue, L\. Ding, Z\. Quan, and Y\. Li\(2024\)Multirate modeling and predictive control for WBG\-device\-based high\-switching\-frequency power converters\.IEEE Transactions on Industrial Electronics71\(1\),pp\. 93–103\.External Links:[Document](https://dx.doi.org/10.1109/TIE.2023.3250765)Cited by:[§I](https://arxiv.org/html/2609.17697#S1.p3.1)\.
- \[10\]T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. Wierstra\(2016\)Continuous control with deep reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§III\-B](https://arxiv.org/html/2609.17697#S3.SS2.p1.1),[§III\-B](https://arxiv.org/html/2609.17697#S3.SS2.p10.1),[§IV\-B](https://arxiv.org/html/2609.17697#S4.SS2.p4.1)\.
- \[11\]D\. P\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§III\-B](https://arxiv.org/html/2609.17697#S3.SS2.p7.2)\.
- \[12\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1861–1870\.Cited by:[§IV\-B](https://arxiv.org/html/2609.17697#S4.SS2.p3.1)\.
- \[13\]J\. Rader, T\. Lyons, and P\. Kidger\(2024\)Optimistix: modular optimisation in JAX and Equinox\.arXiv preprint arXiv:2402\.09983\.Cited by:[§IV\-B](https://arxiv.org/html/2609.17697#S4.SS2.p5.1)\.