Sample-efficient Transfer Reinforcement Learning via Adaptive Reward Shaping and Policy-Ratio Reweighting Strategy

# Sample-efficient Transfer Reinforcement Learning via Adaptive Reward Shaping and Policy-Ratio Reweighting Strategy
Source: [https://arxiv.org/html/2606.26527](https://arxiv.org/html/2606.26527)
Wenjie Huang, Yang Li,Member, IEEE, Jingjia Teng, Mingwei Jin, Kai Song, Yougang Bian,Member, IEEE, Yongfu Li,Senior Member, IEEE, Qisong Yang, Helai HuangThis work was supported by the National NSF of China \(52302493\) and the National K&D Program of China \(2023YFB2504700\)\. \(Corresponding Author: Yang Li\)Wenjie Huang, Yang Li, Jingjia Teng, Mingwei Jin, Kai Song, Yougang Bian are with the State Key Laboratory of Advanced Design and Manufacturing Technology for Vehicle, College of Mechanical and Vehicle Engineering, Hunan University, Changsha 410082, China \(e\-mail: huangwenjie@hnu\.edu\.cn; lyxc56@gmail\.com; tengjingjia@foxmail\.com; tipsy13103562826@163\.com; song\_kaivip@163\.com; byg10@foxmail\.com\)\.Yongfu Li is with the Key Laboratory of Intelligent Air\-Ground Cooperative Control for Universities in Chongqing, School of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, China \(e\-mail: liyongfu@ieee\.org\)\. Qisong Yang is with the Xi’an Institute of High\-Tech, Xi’an 710025, China\. \(e\-mail: qisong\.yang\.93@outlook\.com\)\.Helai Huang is with the School of Traffic and Transportation Engineering, Central South University, Changsha, Hunan 410083, China\. \(email: huanghelai@csu\.edu\.cn\)

###### Abstract

Transfer learning improves policy learning efficiency by reusing knowledge from source tasks, offering a practical paradigm for safe and efficient autonomous highway lane\-changing decision\-making\. However, existing methods often suffer from transfer mismatch caused by distribution shifts between source and target domains, resulting in training instability and performance degradation\. Moreover, target\-domain adaptation still relies on exploratory interactions, making it difficult to ensure training safety in safety\-critical lane\-changing scenarios\. To address these issues, this study proposes a safe transfer reinforcement learning framework for autonomous highway lane changing\. First, an adaptive teacher intervention mechanism based on instantaneous safety cost is developed to reduce unsafe exploration and gradually decay intervention intensity, while the return bounds under the mixed\-behavior policy are theoretically analyzed\. The intervention process also generates dual\-source samples for joint training\. Second, a teacher\-guided safe transfer learning mechanism incorporates the teacher policy’s action\-evaluation information into student learning as a reward\-shaping term to improve training safety and efficiency, while gradually weakening teacher guidance as policy safety improves\. Third, a teacher\-guided optimization weighting mechanism reweights sample contributions during policy optimization via a likelihood\-ratio\-based factor, thereby improving transfer stability\. Experimental results under different traffic densities, together with an additional evaluation on the real\-world NGSIM dataset, show that the proposed method achieves improvements of more than 52\.2% in safety and 5\.0% in efficiency over baseline methods\. These results demonstrate the effectiveness and robustness of the proposed method in enabling safety\-aware transfer for autonomous highway lane changing across diverse traffic scenarios\.

## IIntroduction

Safe and efficient lane\-changing decision\-making is critical for autonomous driving systems and intelligent transportation systems envisioned for the 15\-Minute City, and reinforcement learning \(RL\) has shown great potential in autonomous lane\-changing tasks\[[19](https://arxiv.org/html/2606.26527#bib.bib16)\]\. However, learning a lane\-changing policy from scratch typically requires extensive environment interactions and exploration, resulting in high training costs and considerable safety risks\[[20](https://arxiv.org/html/2606.26527#bib.bib3)\]\. To address these issues, transfer reinforcement learning leverages prior knowledge from source tasks to reduce the training burden and improve convergence efficiency\[[27](https://arxiv.org/html/2606.26527#bib.bib1)\],\[[32](https://arxiv.org/html/2606.26527#bib.bib12)\]\. Nevertheless, existing transfer reinforcement learning methods still suffer from several limitations\. First, distribution discrepancies between source and target domains often lead to knowledge mismatch, which may further result in poor data utilization, low sample efficiency, training instability, negative transfer, and degraded convergence performance\[[44](https://arxiv.org/html/2606.26527#bib.bib4)\]\. Second, although transfer mechanisms can improve learning efficiency, target\-domain training still generally requires exploration, making it difficult to ensure training safety and limiting the deployment of such methods in safety\-critical scenarios\[[13](https://arxiv.org/html/2606.26527#bib.bib5)\]\. Therefore, existing transfer reinforcement learning methods for autonomous driving still find it challenging to simultaneously achieve training safety and policy learning efficiency\.

![Refer to caption](https://arxiv.org/html/2606.26527v1/x1.png)Figure 1:Illustration of the key components of our method, which transfers the knowledge learned from the source task \(⋄\\diamond\) \(left\) to the target task \(∘\\circ\) \(right\) effectively, which accelerates the learning process, improves data efficiency, and safety performance\.To address the aforementioned challenges, existing transfer reinforcement learning methods can be broadly classified into three categories: safety\-oriented transfer methods\[[45](https://arxiv.org/html/2606.26527#bib.bib6),[38](https://arxiv.org/html/2606.26527#bib.bib17),[43](https://arxiv.org/html/2606.26527#bib.bib9)\], efficiency\-oriented transfer methods\[[24](https://arxiv.org/html/2606.26527#bib.bib10),[26](https://arxiv.org/html/2606.26527#bib.bib11)\], and domain adaptation\-based transfer methods\[[28](https://arxiv.org/html/2606.26527#bib.bib7),[42](https://arxiv.org/html/2606.26527#bib.bib8)\]\. Safety\-oriented transfer methods focus on improving training safety during transfer, typically by reducing hazardous exploration in target\-domain training through mechanisms such as teacher guidance\[[45](https://arxiv.org/html/2606.26527#bib.bib6)\], robust control\[[38](https://arxiv.org/html/2606.26527#bib.bib17)\], and safety constraints\[[43](https://arxiv.org/html/2606.26527#bib.bib9)\]\. However, unsafe behaviors may still arise during early adaptation to a new environment\. Efficiency\-oriented transfer methods focus on improving transfer efficiency by reusing prior knowledge from source tasks to reduce the training burden and accelerate policy convergence\[[24](https://arxiv.org/html/2606.26527#bib.bib10),[26](https://arxiv.org/html/2606.26527#bib.bib11)\]\. Nevertheless, these methods usually lack explicit treatment of target\-domain training safety, as well as the distribution shift and training instability caused by source\-target discrepancies\. Domain adaptation\-based transfer methods focus on modeling source\-target discrepancies, typically through domain transfer\[[28](https://arxiv.org/html/2606.26527#bib.bib7)\], correspondence learning, or representation transfer to improve cross\-domain transfer performance\[[42](https://arxiv.org/html/2606.26527#bib.bib8)\]\. However, a unified and effective treatment of the distribution shift, training instability, and negative transfer caused by source\-target discrepancies is still lacking\. Therefore, it remains imperative to improve the convergence efficiency and stability of policy learning while ensuring safety, so as to enable fast, stable, and safe reinforcement learning\-based decision\-making\.

To address the above issues, this paper first employs the policy learned from the source task to provide safety guidance for the target task, as illustrated in Fig\.[1](https://arxiv.org/html/2606.26527#S1.F1)\. On this basis, a teacher\-guided safe transfer reinforcement learning framework for autonomous highway lane changing is further developed, as shown in Fig\.[2](https://arxiv.org/html/2606.26527#S1.F2)\. The proposed framework integrates adaptive teacher intervention, teacher\-guided policy learning, and teacher\-guided optimization within a unified transfer learning paradigm\. Through this design, the method aims to reduce exploration risk, improve the effectiveness of teacher guidance, and enhance transfer stability during target\-domain learning\.

![Refer to caption](https://arxiv.org/html/2606.26527v1/x2.png)Figure 2:Overall framework of the proposed safer transfer learning method\. A teacher Soft Actor\-Critic \(SAC\) agent is first trained offline in a simple highway environment\. During student training in a complex highway environment, the teacher provides safer action intervention, safety\-guided policy learning signals, and teacher\-intervened samples for dual\-source data learning\.The main contributions are summarized as follows:

- •We propose an adaptive teacher intervention mechanism based on instantaneous safety cost, with an adaptive threshold to achieve intervention decay, while the upper and lower bounds of the return under the mixed\-behavior policy are theoretically analyzed\. In addition, the intervention process naturally generates two types of samples, which can be jointly exploited for dual\-source data training according to a mixing ratio determined by the intervention rate\.
- •We propose a teacher\-guided safe transfer learning mechanism, in which the action\-evaluation information of the teacher policy is incorporated into the student policy learning process as a reward\-shaping term\. The guiding effect under safety constraints is modulated by the safety\-related weight, thereby enhancing the student policy’s safety awareness and learning efficiency\.
- •We further develop a teacher\-guided optimization weighting mechanism\. By constructing a likelihood\-ratio\-based weighting factor, the update contribution of samples in policy optimization is reweighted, thereby improving the stability of the transfer process and enhancing the efficiency of teacher knowledge utilization\.
- •Comparative experiments conducted under different traffic densities, together with an additional experiment based on the real\-world NGSIM dataset, demonstrate that the proposed method is both effective and robust, and achieves a favorable balance between traffic efficiency and safety\.

The remainder of this paper is organized as follows: Section[II](https://arxiv.org/html/2606.26527#S2)reviews related work\. Section[III](https://arxiv.org/html/2606.26527#S3)describes the problem setting and formulation\. Sections[IV](https://arxiv.org/html/2606.26527#S4),[V](https://arxiv.org/html/2606.26527#S5), and[VI](https://arxiv.org/html/2606.26527#S6)present the proposed methodology and the theoretical analysis\. Sections[VII](https://arxiv.org/html/2606.26527#S7)and[VIII](https://arxiv.org/html/2606.26527#S8)provide the experimental setup and results\. Section[IX](https://arxiv.org/html/2606.26527#S9)concludes the whole paper\.

## IIRelated Work

### II\-ATransfer Reinforcement Learning

Existing studies on transfer reinforcement learning have improved policy adaptation mainly by reusing demonstrations, pretrained policies, or online teacher assistance\. Demonstration\-based methods accelerate early exploration by incorporating expert trajectories into policy optimization\[[36](https://arxiv.org/html/2606.26527#bib.bib45),[18](https://arxiv.org/html/2606.26527#bib.bib46)\], while policy transfer and distillation methods reuse pretrained behavioral or representational knowledge across related tasks\[[29](https://arxiv.org/html/2606.26527#bib.bib50),[41](https://arxiv.org/html/2606.26527#bib.bib51)\]\. In parallel, teacher\-guided approaches further improve learning efficiency through action advising or selective intervention during online training\[[37](https://arxiv.org/html/2606.26527#bib.bib56),[2](https://arxiv.org/html/2606.26527#bib.bib58)\]\. Such methods are particularly appealing in reinforcement learning because they introduce external guidance without requiring the student to fully rely on offline supervision or direct policy cloning\. This online assistance provides a more flexible transfer mechanism than static demonstrations or fixed pretrained policies\. However, most existing teacher\-assisted methods still provide guidance primarily at the action level\. Although such mechanisms can locally reduce unsafe exploration\[[14](https://arxiv.org/html/2606.26527#bib.bib78)\], they often fail to transfer teacher\-side evaluation signals into the student’s value learning\. As a result, the student may follow corrected actions without fully internalizing their long\-term safety implications\. Moreover, most guidance mechanisms are fixed, heuristic, or only weakly coupled with the student’s learning status\[[11](https://arxiv.org/html/2606.26527#bib.bib76)\], making them less suitable for safety\-critical tasks that require adaptive guidance\.

Different from current methods, we propose a teacher\-guided safe transfer mechanism that incorporates teacher\-side action evaluation into student policy learning via reward shaping and adaptively modulates the guidance effect with a safety\-related weight under safety constraints, thereby improving safety awareness and learning efficiency\.

### II\-BSafe Transfer Reinforcement Learning

Safety assurance in teacher\-student reinforcement learning is commonly achieved in two different ways\. Some studies enforce safety at the execution level by correcting or replacing unsafe student actions before execution; for instance, action shielding methods\[[1](https://arxiv.org/html/2606.26527#bib.bib32)\]and their extensions to partially observable settings\[[5](https://arxiv.org/html/2606.26527#bib.bib33)\]explicitly prevent actions that may violate safety requirements, while model\-based recovery schemes further improve the flexibility of such protection through recovery control or adaptive replanning\[[23](https://arxiv.org/html/2606.26527#bib.bib34)\]\. Instead of directly correcting unsafe student actions at execution time, another line of research improves training safety by incorporating the teacher into the learning process, either through direct guidance on student decisions or through broader protection of the training process\. Some methods provide selective teacher advice only when necessary\[[16](https://arxiv.org/html/2606.26527#bib.bib38)\], while others further combine such advice with imitation learning to help the student acquire safer behaviors from teacher\-supported decisions\[[15](https://arxiv.org/html/2606.26527#bib.bib39)\]\. However, existing studies usually use fixed or heuristic intervention schemes, without adapting intervention intensity to the student’s instantaneous safety condition or competence\. Moreover, they rarely exploit the student\-generated and teacher\-guided samples induced by the intervention as complementary data sources for joint training\.

Unlike existing approaches, we develop an adaptive teacher intervention framework driven by instantaneous safety cost, in which teacher involvement is progressively reduced through an adaptive threshold, and the resulting mixed\-behavior policy is further analyzed theoretically\. In addition, the resulting student\-generated and teacher\-guided samples are jointly utilized for dual\-source training under an intervention\-dependent mixing strategy\.

### II\-CPolicy Optimization with Heterogeneous Training Data

Reinforcement learning is typically trained from the agent’s own online interactions, which often suffer from inefficient exploration and unstable policy improvement\. To address this issue, early studies augment training with external experience, such as demonstrations or successful trajectories\. Some studies exploit demonstrations from related or multiple tasks to improve learning efficiency\[[10](https://arxiv.org/html/2606.26527#bib.bib61),[7](https://arxiv.org/html/2606.26527#bib.bib62)\], whereas others incorporate expert state sequences or successful trajectories to guide learning\[[9](https://arxiv.org/html/2606.26527#bib.bib63),[25](https://arxiv.org/html/2606.26527#bib.bib64)\]\. Building on this idea, subsequent studies began to exploit heterogeneous experience during training, such as combining offline data with online interaction\[[33](https://arxiv.org/html/2606.26527#bib.bib65)\], balancing prior data with newly collected samples in offline\-to\-online reinforcement learning\[[21](https://arxiv.org/html/2606.26527#bib.bib66)\], and jointly utilizing demonstrations, successful episodes, and teacher\-related data\[[31](https://arxiv.org/html/2606.26527#bib.bib67)\]\. This setting becomes particularly relevant in teacher\-student reinforcement learning, where training experience can also be generated online through teacher intervention\. However, these studies usually do not explicitly distinguish student\-generated samples from teacher\-intervened samples\. Consequently, the unique roles of mixed\-behavior samples in policy optimization remain insufficiently explored, and their weighting in policy optimization is rarely calibrated according to their relevance to the current policy\.

In contrast to conventional methods, we develop a teacher\-guided optimization weighting mechanism, where a likelihood\-ratio\-based factor calibrates the weighting of mixed\-behavior samples in policy optimization, thereby improving transfer stability and enhancing the efficiency of teacher knowledge utilization\.

## IIIProblem Description

This section provides the necessary background on constrained Markov decision processes \(CMDPs\) and formulates the transfer learning problem considered in this work\.

### III\-AConstrained Markov Decision Processes

We consider tasks formulated by constrained Markov decision processes \(CMDPs\)\[[4](https://arxiv.org/html/2606.26527#bib.bib75)\]\. A CMDP can be described by a tupleℳ=⟨𝒮,𝒜,𝒫,r,c,d,γ⟩\\mathcal\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},r,c,d,\\gamma\\rangle, where𝒮\\mathcal\{S\}denotes the state space and𝒜\\mathcal\{A\}denotes the action space\.𝒫:𝒮×𝒜×𝒮→\[0,1\]\\mathcal\{P\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\rightarrow\[0,1\]specifies the transition probability from statessto the next states′s^\{\\prime\}after taking actionaa, withs∈𝒮s\\in\\mathcal\{S\}anda∈𝒜a\\in\\mathcal\{A\}\. The functionr:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}denotes the reward function, andc:𝒮×𝒜→ℝc:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}denotes the cost function\.d∈ℝ\+d\\in\\mathbb\{R\}\_\{\+\}andγ∈\[0,1\)\\gamma\\in\[0,1\)represent the cost limit and the discount factor, respectively\. In a safe RL problem, an agent interacts with the environment over time and generates a trajectory⟨\(s0,a0,r0,c0,s0′\),\(s1,a1,r1,c1,s1′\),…⟩\\langle\(s\_\{0\},a\_\{0\},r\_\{0\},c\_\{0\},s\_\{0\}^\{\\prime\}\),\(s\_\{1\},a\_\{1\},r\_\{1\},c\_\{1\},s\_\{1\}^\{\\prime\}\),\\ldots\\rangle\. Starting from the initial states0s\_\{0\}, at each time steptt, the agent is in a statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}and takes an actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}\. It then receives the corresponding rewardrt=r​\(st,at\)r\_\{t\}=r\(s\_\{t\},a\_\{t\}\)and costct=c​\(st,at\)c\_\{t\}=c\(s\_\{t\},a\_\{t\}\), and transitions to the next statest′∼𝒫\(⋅∣st,at\)s\_\{t\}^\{\\prime\}\\sim\\mathcal\{P\}\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\. This process restarts withst\+1=st′s\_\{t\+1\}=s\_\{t\}^\{\\prime\}and repeats until a termination condition is met, after which a new trajectory is initiated\. The objective is to learn an optimal policy \(π∗\\pi^\{\*\}\) that maximizes the expected discounted return while ensuring that the expected discounted cost return remains belowdd:

π∗=\\displaystyle\\pi^\{\*\}=arg⁡maxπ∈Π⁡𝔼\(st,at\)∼ρπ​\[∑t=0∞γt​rt\],\\displaystyle\\arg\\max\_\{\\pi\\in\\Pi\}\\;\\underset\{\(s\_\{t\},a\_\{t\}\)\\sim\\rho\_\{\\pi\}\}\{\\mathbb\{E\}\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\\right\],\(1\)s\.t\.𝔼\(st,at\)∼ρπ​\[∑t=0∞γt​ct\]≤d,\\displaystyle\\text\{s\.t\.\}\\quad\\;\\quad\\ \\ \\underset\{\(s\_\{t\},a\_\{t\}\)\\sim\\rho\_\{\\pi\}\}\{\\mathbb\{E\}\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}c\_\{t\}\\right\]\\leq d,whereρπ\\rho\_\{\\pi\}is the trajectory distribution induced by the policyπ\\pi,Π\\Piis the set of all feasible policies\.

### III\-BProblem Formulation

We formalize our problem setting using the transfer learning framework for RL\. Given a source task and a target task, transfer learning aims to accelerate the learning process on the target task by leveraging knowledge acquired from the source task\[[40](https://arxiv.org/html/2606.26527#bib.bib27)\]\. The source taskℳ⋄\\mathcal\{M\}^\{\\diamond\}is expected to provide useful knowledge𝒦=\{π⋄\}\\mathcal\{K\}=\\\{\\pi^\{\\diamond\}\\\}to the agent learning the target taskℳ∘\\mathcal\{M\}^\{\\circ\}, thereby enabling the agent to learn the target policy more efficiently onℳ∘\\mathcal\{M\}^\{\\circ\}\.

#### III\-B1State Space

A state describes the environment configuration with which the agent interacts\. The state of each vehicleiiis defined assi=\(pi,Δ​xi,Δ​yi,Δ​vix,Δ​viy\)s\_\{i\}=\(p\_\{i\},\\Delta x\_\{i\},\\Delta y\_\{i\},\\Delta v\_\{i\}^\{x\},\\Delta v\_\{i\}^\{y\}\), wherepip\_\{i\}is a presence indicator specifying whether vehicleiiis observable;\(Δ​xi,Δ​yi\)\(\\Delta x\_\{i\},\\Delta y\_\{i\}\)denote the longitudinal and lateral positions of the surrounding vehicleiirelative to the ego vehicle, and\(Δ​vix,Δ​viy\)\(\\Delta v\_\{i\}^\{x\},\\Delta v\_\{i\}^\{y\}\)represent the relative longitudinal and lateral velocities between the surrounding vehicleiiand the ego vehicle, respectively\. The overall state representation is given byS=\[s1,s2,…,si,…,sM\]S=\[s\_\{1\},s\_\{2\},\\ldots,s\_\{i\},\\ldots,s\_\{M\}\], whereMMis the maximum number of observed vehicles\.

#### III\-B2Action Space

The agent selects actions from the action space𝒜\\mathcal\{A\}, which consists of high\-level control commands\. The action space𝒜\\mathcal\{A\}is defined as:

𝒜=\[A←,A→,A↑,A↓,A⊘\],\\mathcal\{A\}=\\left\[A^\{\\leftarrow\},A^\{\\rightarrow\},A^\{\\uparrow\},A^\{\\downarrow\},A^\{\\oslash\}\\right\],\(2\)whereA←A^\{\\leftarrow\},A→A^\{\\rightarrow\},A↑A^\{\\uparrow\},A↓A^\{\\downarrow\}, andA⊘A^\{\\oslash\}correspond to turning left, turning right, accelerating, decelerating, and cruising, respectively\.

#### III\-B3Reward Function

The source and target tasks adopt different reward functions\. Given the stringent safety requirements of safety\-critical applications for transfer, we construct a source task that is free of extrinsic task rewards to train the teacher agent, with the primary objective of acquiring safety\-relevant knowledge\. During training, no external task return is used; instead, to improve exploration efficiency, an intrinsic auxiliary reward is introduced to encourage active exploration and improve state\-space coverage\.

r⋄=‖𝐩t−𝐩t−1‖2,r^\{\\diamond\}=\\left\\\|\\mathbf\{p\}\_\{t\}\-\\mathbf\{p\}\_\{t\-1\}\\right\\\|\_\{2\},\(3\)where𝐩t=\(xt,yt\)\\mathbf\{p\}\_\{t\}=\(x\_\{t\},y\_\{t\}\)denotes the position of the ego vehicle at time steptt\.

In contrast, the target task follows a standard task\-driven extrinsic reward mechanism, aiming to simultaneously promote safety, traffic efficiency, and task completion\. In the highway lane\-change scenario considered in this paper, the ego vehicle must complete a lane change toward the target lane and maintain stable driving within multi\-lane traffic, while avoiding collisions, maintaining appropriate driving speed, and reaching the target lane, thereby enabling safe and efficient longitudinal progression and lateral decision\-making\. Accordingly, the reward received by the student agent at each time step is defined as a weighted sum of several components, including: a safety term that penalizes collision events, an efficiency term that encourages maintaining an appropriate driving speed, and a goal term that guides the policy toward the target lane\. Specifically, the instantaneous reward received is defined as:

r∘=ws​rs\+we​re\+wg​rg,r^\{\\circ\}=w\_\{s\}r\_\{s\}\+w\_\{e\}r\_\{e\}\+w\_\{g\}r\_\{g\},\(4\)wherersr\_\{s\},rer\_\{e\}, andrgr\_\{g\}denote the safe reward, efficiency reward, and goal reward, respectively\.wsw\_\{s\},wew\_\{e\}, andwgw\_\{g\}are the corresponding weighting coefficients\.

A safe reward is imposed such that:

rs=\{−1,if a collision occurs,0,otherwise,r\_\{s\}=\\begin\{cases\}\-1,&\\text\{if a collision occurs\},\\\\ 0,&\\text\{otherwise\},\\end\{cases\}\(5\)which discourages unsafe behaviors that may cause collisions\.

To promote driving efficiency, the speed reward is defined as:

re=min⁡\(vt−vminvmax−vmin,1\),r\_\{e\}=\\min\\bigl\(\\frac\{v\_\{t\}\-v\_\{\\min\}\}\{v\_\{\\max\}\-v\_\{\\min\}\},1\\bigr\),\(6\)wherevtv\_\{t\}denotes the speed of the ego vehicle at time steptt\.

To facilitate task completion in the highway lane\-change scenario, a goal term is further introduced as:

rg=1−\|ℓt−ℓg\|ℒ,r\_\{g\}=1\-\\frac\{\\left\|\\ell\_\{t\}\-\\ell\_\{g\}\\right\|\}\{\\mathcal\{L\}\},\(7\)whereℓt\\ell\_\{t\}andℓg\\ell\_\{g\}denote the indices of the ego vehicle’s current lane and the goal lane, respectively, andℒ\\mathcal\{L\}is the total number of lanes on the current road\. This normalized shaping reward increases as the ego vehicle approaches the target lane, thereby encouraging timely lane\-change completion\.

#### III\-B4Cost Function

In addition to the reward design, we design a safety\-related cost function to explicitly penalize situations with insufficient longitudinal safety margins\. Specifically, we consider the time headway \(THW\) between the ego vehicle and the closest leading vehicle in the current lane\. Letdhead​\(t\)≥0d\_\{\\text\{head\}\}\(t\)\\geq 0denote the longitudinal distance between the ego vehicle and the leading vehicle at time steptt\. The THW is defined as:

THWt=dhead​\(t\)max⁡\(vt,ε\),\\mathrm\{THW\}\_\{t\}=\\frac\{d\_\{\\text\{head\}\}\(t\)\}\{\\max\(v\_\{t\},\\varepsilon\)\},\(8\)whereε\>0\\varepsilon\>0is a small constant to avoid numerical issues\.

Subsequently, a THW\-based cost term is incorporated to penalize cases where the time headway falls below a predefined safety thresholdTsafeT\_\{\\mathrm\{safe\}\}\. The instantaneous cost is defined as:

cthw​\(t\)=max⁡\(0,1−THWtTsafe\)\.c\_\{\\mathrm\{thw\}\}\(t\)=\\max\(0,1\-\\dfrac\{\\mathrm\{THW\}\_\{t\}\}\{T\_\{\\mathrm\{safe\}\}\}\)\.\(9\)This cost encourages the agent to maintain an adequate following distance, thereby reducing the risk of rear\-end collisions and the possibility of longitudinal collision\.

## IVTransfer Learning Strategy

This section introduces the core components of the proposed method, including the adaptive action intervention mechanism and the teacher\-guided safe policy learning scheme\. These components establish the key principles for safe interaction and knowledge transfer, which will later be integrated into a unified training framework\.

### IV\-ASafe Exploration via Adaptive Action Intervention

After completing the training of the teacher model, we train a new policy, referred to as the student policy, dedicated to the target task\. To enhance safety during training and improve the student’s performance, we incorporate the teacher policy into the training loop, allowing the teacher policyπt\\pi^\{t\}and the student policyπs\\pi^\{s\}to work together and form the mixed behavior policyπmix\\pi^\{\\mathrm\{mix\}\}\. The term “teacher” only indicates that this policy is introduced to facilitate student training, and the optimality of the teacher model is not considered in this paper\.

#### IV\-A1Action Intervention Function

During student training, the intervention function determines whether the action at the current time step is generated by the teacher policy networkπt\\pi^\{t\}or the student policy networkπs\\pi^\{s\}\. The student policy is then updated using the data collected under such interventions\. One intuitive design principle for the intervention function is to trigger intervention when the student’s behavior deviates from that of the teacher\. In this work, we design a cost\-based intervention function based on the instantaneous cost, as defined below:

at=\{att,if​ct−1\>τ,ats,otherwise,a\_\{t\}=\\begin\{cases\}a\_\{t\}^\{t\},&\\text\{if \}c\_\{t\-1\}\>\\tau,\\\\ a\_\{t\}^\{s\},&\\text\{otherwise,\}\\end\{cases\}\(10\)whereτ\\tauis the intervention threshold, which controls when the teacher policy intervenes\.atta^\{t\}\_\{t\}andatsa^\{s\}\_\{t\}denote teacher action and student action at the time steptt, respectively\. The student policy keeps sampling withπmix=πs\\pi^\{\\mathrm\{mix\}\}=\\pi^\{s\}at the start of a trajectory\. Once the first eventct−1\>τc\_\{t\-1\}\>\\tauis encountered, we setπmix=πt\\pi^\{\\mathrm\{mix\}\}=\\pi^\{t\}until the end of the episode\. Consequently, the teacher policy serves to improve the safety of data collection\. Based on this intervention function, the mixed behavior policy can be expressed in the following form:

πmix\(⋅\|s\)=𝒯\(s\)πt\(⋅\|s\)\+\(1−𝒯\(s\)\)πs\(⋅\|s\),\\pi^\{\\text\{mix\}\}\(\\cdot\|s\)=\\mathcal\{T\}\(s\)\\pi^\{t\}\(\\cdot\|s\)\+\(1\-\\mathcal\{T\}\(s\)\)\\pi^\{s\}\(\\cdot\|s\),\(11\)where𝒯​\(s\)=1\\mathcal\{T\}\(s\)=1indicates that the teacher intervenes in action selection, whereas𝒯​\(s\)=0\\mathcal\{T\}\(s\)=0indicates otherwise\. The mixed behavior policy defined in \([11](https://arxiv.org/html/2606.26527#S4.E11)\) provides an intuitive mechanism for safe exploration by allowing the teacher to intervene whenever necessary\. A theoretical analysis of its effectiveness is presented in Section[VI](https://arxiv.org/html/2606.26527#S6)\.

#### IV\-A2Intervention Decay

Teacher intervention can prevent unsafe behaviors and accelerate the student’s learning process, substantially improving the student’s performance in the early stages of training\. However, excessive reliance on the teacher’s guidance may hinder the student’s final performance\. To address this issue, we introduce an intervention decay mechanism that gradually reduces the teacher’s intervention, thereby reducing the student’s dependence on the teacher\. During training, this mechanism adaptively updates the intervention thresholdτ\\taubased on recent safety performance\. Specifically, the violation rate is defined as the proportion of time steps within an epoch for which the instantaneous cost is greater than zero\. Letr∗r^\{\*\}denote the desired target violation rate, andr^​\(e\)\\hat\{r\}\(e\)denote the average violation rate at epocheecomputed using a sliding window over the most recentWWepochs\. The threshold update rule is given by:

τ​\(e\+1\)=clip​\(τ​\(e\)\+κ​\(r∗−r^​\(e\)\),0,τmax\),\\tau\(e\+1\)=\\mathrm\{clip\}\\Big\(\\tau\(e\)\+\\kappa\\big\(r^\{\*\}\-\\hat\{r\}\(e\)\\big\),\\,0,\\,\\tau\_\{\\max\}\\Big\),\(12\)whereκ\>0\\kappa\>0is the update step size,τmax\\tau\_\{\\max\}is the maximum allowable threshold, andclip​\(⋅\)\\mathrm\{clip\}\(\\cdot\)ensuresτ\\tauremains within a feasible range\. This update rule admits an intuitive interpretation\. Ifr^​\(e\)\>r∗\\hat\{r\}\(e\)\>r^\{\*\}, i\.e\., the current policy is riskier than desired, thenr∗−r^​\(e\)<0r^\{\*\}\-\\hat\{r\}\(e\)<0andτ​\(e\+1\)<τ​\(e\)\\tau\(e\+1\)<\\tau\(e\), which lowers the intervention threshold and leads to more frequent teacher interventions\. Conversely, ifr^​\(e\)<r∗\\hat\{r\}\(e\)<r^\{\*\}, thenτ​\(e\+1\)\>τ​\(e\)\\tau\(e\+1\)\>\\tau\(e\), allowing the student to act more often and thereby progressively decaying teacher intervention\.

### IV\-BSafety\-Guided Policy Learning

To accelerate learning under safety constraints without making the student overly dependent on the teacher, we incorporate teacher\-guidance information into an entropy\-regularized actor–critic framework from two aspects, namely value\-target construction and optimization weighting, as illustrated in Fig\.[3](https://arxiv.org/html/2606.26527#S4.F3)\. Specifically, the teacher’s action\-evaluation signal is introduced as a reward\-shaping term in the value\-target update, yielding a target with an adaptive safety weight\. In addition, teacher\-intervened transitions are assigned a likelihood\-ratio\-based optimization weight according to the relative likelihood of the executed action under the current student and teacher policies\. In this way, teacher guidance improves both learning safety and transfer stability during iterative optimization\.

![Refer to caption](https://arxiv.org/html/2606.26527v1/pictures/policy-learning.png)Figure 3:Safety\-guided policy learning framework\. The proposed framework incorporates teacher guidance into the student policy optimization process to improve learning safety and transfer stability under safety constraints\.#### IV\-B1Teacher\-induced Reward Shaping

For any transition\(s,a,r,c,s′\)\(s,a,r,c,s^\{\\prime\}\), in addition to the extrinsic environment rewardrr, we compute an auxiliary signal from the teacher policy to quantify how well the actionaaaligns with the teacher’s behavior:

bonus=log⁡πt​\(a∣s\)\.\\text\{bonus\}=\\log\\pi^\{t\}\(a\\mid s\)\.\(13\)This term encourages the student to produce teacher\-consistent actions more frequently during early exploration, thereby improving sampling safety and enhancing learning efficiency\. Moreover, to adaptively adjust the influence of the teacher signal according to the current safety status, we scale this auxiliary signal by a safety weightβ\\beta, which inherently serves as a Lagrange multiplier associated with safety cost constraints\. This weight varies dynamically with the degree to which the current policy satisfies the safety constraint, strengthening teacher guidance when the policy is unsafe and gradually attenuating it as safety improves, thereby achieving an adaptive trade‑off between safety guidance and task performance\.

#### IV\-B2Value Target Construction

Within the maximum\-entropy actor–critic framework, the update of the action\-value function is typically based on a target value obtained by marginalizing over the next\-state action distribution, thereby incorporating entropy regularization into value estimation\. Specifically, given the next states′s^\{\\prime\}, the policyπθ\(⋅∣s′\)\\pi\_\{\\theta\}\(\\cdot\\mid s^\{\\prime\}\)outputs the action probabilitiespθ\(⋅∣s′\)p\_\{\\theta\}\(\\cdot\\mid s^\{\\prime\}\)and the corresponding log\-likelihoodslogpθ\(⋅∣s′\)\\log p\_\{\\theta\}\(\\cdot\\mid s^\{\\prime\}\)\. The expected next\-state value is then computed as:

Vnext​\(s′\)=∑a′pθ​\(a′∣s′\)​\(mini=1,2⁡Qi​\(s′,a′\)−α​log⁡pθ​\(a′∣s′\)\),V\_\{\\text\{next\}\}\(s^\{\\prime\}\)=\\sum\_\{a^\{\\prime\}\}p\_\{\\theta\}\(a^\{\\prime\}\\mid s^\{\\prime\}\)\\Big\(\\min\_\{i=1,2\}Q\_\{i\}\(s^\{\\prime\},a^\{\\prime\}\)\-\\alpha\\log p\_\{\\theta\}\(a^\{\\prime\}\\mid s^\{\\prime\}\)\\Big\),\(14\)whereα\\alphais the entropy regularization coefficient, andQitargQ\_\{i\}^\{\\text\{targ\}\}denotes the target action\-value function parameterized by a slowly updated target network to stabilize learning\.

To incorporate teacher\-guidance information into policy learning, we treat the teacher’s evaluation signalbonusfor the current action as an auxiliary bonus reward and add it to the extrinsic environment rewardrr, yielding an augmented target:

Qtarget​\(s,a\)=r\+β⋅bonus\+γ​Vnext​\(s′\),Q\_\{\\text\{target\}\}\(s,a\)=r\+\\beta\\cdot\\text\{bonus\}\+\\gamma V\_\{\\text\{next\}\}\(s^\{\\prime\}\),\(15\)whereγ\\gammais the discount factor\. Compared with the standard maximum\-entropy update, this work introduces an additional safety gain term induced by the teacher signal\. By unifying teacher priors, safety constraints, and maximum\-entropy value estimation within a single update framework, the learning objective of the student critic is no longer merely to achieve optimal expected return in the environmental sense, but is instead transformed into a risk\-sensitive value optimality criterion that incorporates safety\-preference\-aware corrections\.

#### IV\-B3Optimization Weight Construction

Besides incorporating the teacher’s evaluation signal into the value target, we further adjust the contribution of teacher\-intervened transitions during optimization by introducing a likelihood\-ratio\-based weighting factor\. Specifically, for a teacher\-intervened actionaa, we define:

ρ=exp⁡\(log⁡πs​\(a\|s\)−log⁡πt​\(a\|s\)\),\\rho=\\exp\(\\log\\pi^\{s\}\(a\|s\)\-\\log\\pi^\{t\}\(a\|s\)\),\(16\)whereπs\\pi^\{s\}andπt\\pi^\{t\}denote the student and teacher policies, respectively\. For student\-generated transitions, we simply setρ=1\\rho=1\. This coefficient is used to reweight the optimization losses of both the actor and the critic: when the student policy can adequately explain the teacher’s actions, the corresponding samples are assigned larger update weights, thereby facilitating effective transfer of safety knowledge to the student policy; conversely, when a significant mismatch exists between the student and teacher policies, the weights are clipped and suppressed to prevent unstable updates caused by safety priors in distributionally inconsistent regions\. Accordingly, the critic network losses can be written in the weighted form:

ℒQi=12​𝔼​\[ρ​\(Qi​\(s,a\)−Qtarget​\(s,a\)\)2\],i∈\{1,2\},\\mathcal\{L\}\_\{Q\_\{i\}\}=\\frac\{1\}\{2\}\\,\\mathbb\{E\}\\left\[\\rho\\big\(Q\_\{i\}\(s,a\)\-Q\_\{\\text\{target\}\}\(s,a\)\\big\)^\{2\}\\right\],\\quad i\\in\\\{1,2\\\},\(17\)whereℒQi\\mathcal\{L\}\_\{Q\_\{i\}\}is the critic loss\. The same weighting strategy is also applied in the policy optimization step\.

## VTeacher\-Guided Training Framework

Based on the transfer learning strategy described above, this section further presents the overall teacher\-guided training framework of the proposed method\. We first introduce the dual\-source data training scheme, which jointly utilizes student\-generated and teacher\-guided samples for policy optimization\. Then, the complete algorithm architecture is summarized to show how safe exploration, teacher guidance, and constrained SAC updates are integrated into a unified training process\.

### V\-AModel Training with Dual\-source Data

The intervention\-driven interaction naturally induces two types of transition samples with distinct origins\. Let each transition be denoted by

x=\(s,a,r,c,s′\)\.x=\(s,a,r,c,s^\{\\prime\}\)\.\(18\)According to whether teacher intervention is triggered, the collected transitions can be divided into a student\-generated replay buffer and a teacher\-generated replay buffer, which are defined as follows:

𝒟s=\{x∣𝒯​\(s\)=0\},𝒟t=\{x∣𝒯​\(s\)=1\},\\mathcal\{D\}^\{s\}=\\left\\\{x\\mid\\mathcal\{T\}\(s\)=0\\right\\\},\\qquad\\mathcal\{D\}^\{t\}=\\left\\\{x\\mid\\mathcal\{T\}\(s\)=1\\right\\\},\(19\)where𝒯​\(s\)∈\{0,1\}\\mathcal\{T\}\(s\)\\in\\\{0,1\\\}is the intervention indicator, with𝒯​\(s\)=1\\mathcal\{T\}\(s\)=1indicating teacher intervention and𝒯​\(s\)=0\\mathcal\{T\}\(s\)=0indicating student interaction\.

Accordingly, the replay data used for training form a dual\-source replay structure composed of𝒟s\\mathcal\{D\}^\{s\}and𝒟t\\mathcal\{D\}^\{t\}, rather than a single\-source experience distribution\. Samples in𝒟t\\mathcal\{D\}^\{t\}are mainly collected in safety\-critical states and provide safety\-aware guidance, whereas samples in𝒟s\\mathcal\{D\}^\{s\}are generated during autonomous interaction and preserve task\-driven exploration in nominal operating states\. Therefore, the two replay sources play complementary roles in policy learning\. To jointly utilize them, the mini\-batch for each update is constructed as

ℬ=ℬt∪ℬs,\|ℬt\|=η​\|ℬ\|,\|ℬs\|=\(1−η\)​\|ℬ\|,\\mathcal\{B\}=\\mathcal\{B\}^\{t\}\\cup\\mathcal\{B\}^\{s\},\\quad\|\\mathcal\{B\}^\{t\}\|=\\eta\|\\mathcal\{B\}\|,\\quad\|\\mathcal\{B\}^\{s\}\|=\(1\-\\eta\)\|\\mathcal\{B\}\|,\(20\)whereℬt⊂𝒟t\\mathcal\{B\}^\{t\}\\subset\\mathcal\{D\}^\{t\}andℬs⊂𝒟s\\mathcal\{B\}^\{s\}\\subset\\mathcal\{D\}^\{s\}denote the teacher\-generated and student\-generated subsets contained in the current mini\-batch, respectively, andη\\etais a mixing hyperparameter determined by the teacher intervention rate during training\.

Under this dual\-source replay setting, the SAC actor is optimized on the mixed mini\-batch as

Jπ​\(θ\)=𝔼s∼ℬ,a∼πθ\(⋅\|s\)​\[α​log⁡πθ​\(a\|s\)−mini=1,2⁡Qi​\(s,a\)\],J\_\{\\pi\}\(\\theta\)=\\mathbb\{E\}\_\{s\\sim\\mathcal\{B\},\\,a\\sim\\pi\_\{\\theta\}\(\\cdot\|s\)\}\\left\[\\alpha\\log\\pi\_\{\\theta\}\(a\|s\)\-\\min\_\{i=1,2\}Q\_\{i\}\(s,a\)\\right\],\(21\)whereπθ\\pi\_\{\\theta\}is the student policy,α\\alphais the temperature coefficient, andQ1Q\_\{1\}andQ2Q\_\{2\}are the twin critics\. Sinceℬ\\mathcal\{B\}contains both teacher\-generated and student\-generated samples, the actor update is performed on replay states drawn from both intervention\-triggered and autonomous interaction regions\.

### V\-BAlgorithm Architecture

Algorithm 1Teacher\-Guided Safe Transfer0:Student environment

ℰs\\mathcal\{E\}^\{s\}, teacher policy

πt\\pi^\{t\}, student policy

πs\\pi^\{s\}, reward critics

Qϕ1Q\_\{\\phi\_\{1\}\},

Qϕ2Q\_\{\\phi\_\{2\}\}, cost critic

QψQ\_\{\\psi\}, replay buffers

𝒟s\\mathcal\{D\}^\{s\},

𝒟t\\mathcal\{D\}^\{t\}\.

1:Initialize

θ\\theta,

ϕ1\\phi\_\{1\},

ϕ2\\phi\_\{2\},

ψ\\psiand target networks, entropy coefficient

α\\alpha, cost penalty coefficient

β\\beta, intervention threshold

τ\\tau, mixing ratio

η\\etaand intervention flag

FintF\_\{\\text\{int\}\}\.

2:foreach interaction stepdo

3:Obtain current student state

ss\.

4:if

FintF\_\{\\text\{int\}\}= Truethen

5:Select action

a∼πt\(⋅\|s\)a\\sim\\pi^\{t\}\(\\cdot\|s\)\.

6:Compute

ρ←exp⁡\(log⁡πs​\(a\|s\)−log⁡πt​\(a\|s\)\)\\rho\\leftarrow\\exp\(\\log\\pi^\{s\}\(a\|s\)\-\\log\\pi^\{t\}\(a\|s\)\)\.

7:Compute teacher guidance signal

bonus←log⁡πt​\(a\|s\)\\text\{bonus\}\\leftarrow\\log\\pi^\{t\}\(a\|s\)\.

8:else

9:Select action

a∼πs\(⋅\|s\)a\\sim\\pi^\{s\}\(\\cdot\|s\)\.

10:Set

ρ←1\\rho\\leftarrow 1, and compute

bonus←log⁡πt​\(a\|s\)\\text\{bonus\}\\leftarrow\\log\\pi^\{t\}\(a\|s\)\.

11:endif

12:Execute

aain

ℰs\\mathcal\{E\}^\{s\}, observe

\(s,a,r,c,s′\)\(s,a,r,c,s^\{\\prime\}\)\.

13:if

c\>τc\>\\tauthen

14:Activate teacher intervention for the remaining episode:

Fint=TrueF\_\{\\text\{int\}\}=\\text\{True\}\.

15:endif

16:if

𝒯​\(s\)=0\\mathcal\{T\}\(s\)=0then

17:Store

\(s,a,r,bonus,ρ,s′,c\)\(s,a,r,\\text\{bonus\},\\rho,s^\{\\prime\},c\)into

𝒟s\\mathcal\{D\}^\{s\}\.

18:else

19:Store

\(s,a,r,bonus,ρ,s′,c\)\(s,a,r,\\text\{bonus\},\\rho,s^\{\\prime\},c\)into

𝒟t\\mathcal\{D\}^\{t\}\.

20:endif

21:Set

s←s′s\\leftarrow s^\{\\prime\}\.

22:ifend of episodethen

23:

Fint=FalseF\_\{\\text\{int\}\}=\\text\{False\}\.

24:endif

25:ifupdate condition is satisfiedthen

26:Composite sampling from

𝒟s\\mathcal\{D\}^\{s\}and

𝒟t\\mathcal\{D\}^\{t\}with

η\\eta\.

27:Update target values

QtargetQ\_\{\\text\{target\}\}and

Q​ctargetQc\_\{\\text\{target\}\}\.

28:Update

Qϕ1Q\_\{\\phi\_\{1\}\}and

Qϕ2Q\_\{\\phi\_\{2\}\}using

QtargetQ\_\{\\text\{target\}\}weighted by

ρ\\rho\.

29:Update

QψQ\_\{\\psi\}using

Q​ctargetQc\_\{\\text\{target\}\}weighted by

ρ\\rho\.

30:Update

πs\\pi^\{s\}with the constrained SAC objective\.

31:Update

α\\alpha,

β\\beta, and softly update target networks\.

32:endif

33:ifend of epochthen

34:Estimate violation rate

r^\\hat\{r\}and optionally update

τ←clip​\(τ\+κ​\(r∗−r^\),0,τmax\)\.\\tau\\leftarrow\\mathrm\{clip\}\\big\(\\tau\+\\kappa\(r^\{\\ast\}\-\\hat\{r\}\),\\,0,\\,\\tau\_\{\\max\}\\big\)\.
35:Update

η\\etabased on teacher intervention rate\.

36:endif

37:endfor

The overall training procedure of the proposed method is summarized in Algorithm[1](https://arxiv.org/html/2606.26527#alg1)\. At a high level, the proposed framework integrates teacher\-guided safe interaction and constrained policy optimization into a unified SAC\-based training procedure\. During interaction, the student acts autonomously as long as the current behavior remains safe; once the instantaneous safety cost exceeds the intervention threshold, the teacher takes over control for the remainder of the current trajectory to prevent unsafe exploration and provide corrective guidance \(Lines 3–15\)\. For each executed action, teacher guidance is incorporated in two complementary ways\. First, the teacher’s action evaluation is transformed into a reward\-shaping bonus and injected into the value target, encouraging the student to favor actions that are more consistent with the teacher’s safe behavior\. The influence of this bonus is modulated by the adaptive cost penalty coefficientβ\\beta, which scales the strength of teacher guidance according to the current safety status of the student policy\. Second, for teacher\-intervened transitions, a likelihood\-ratio\-based coefficientρ\\rhois computed from the relative likelihood between the student and teacher action probabilities and is used to reweight the optimization losses \(Lines 6–10 and 27–30\)\. This design strengthens the influence of teacher\-guided samples that are more compatible with the current student policy while suppressing highly mismatched ones\. To exploit heterogeneous experience more effectively, two replay buffers are maintained to separately store student\-generated and teacher\-intervened transitions \(Lines 16–20\)\. When the update condition is met, mini\-batches are sampled from both buffers with a mixing ratioη\\etaand merged for joint optimization\. The reward critics and cost critics are then updated with their corresponding targets, followed by the updates of the policy, entropy coefficient, penalty coefficient, and target networks \(Lines 25–31\)\. In addition, the intervention thresholdτ\\taucan be adaptively adjusted according to the epoch\-level safety violation rate \(Lines 33–34\), allowing stronger teacher protection during unsafe stages and gradually reducing teacher reliance as training becomes safer\. Through this architecture, the proposed method achieves a balance among safe exploration, effective teacher\-to\-student knowledge transfer, and stable policy improvement\.

## VITheoretical Analysis

This section provides a theoretical analysis of the proposed teacher\-policy intervention mechanism\. We establish its effectiveness by showing that teacher interventions can raise the student’s attainable performance upper bound while preserving a guaranteed lower bound\.

###### Lemma 1\.

Consider a MDP with discount factorγ∈\(0,1\)\\gamma\\in\(0,1\)and bounded rewardr​\(s,a\)∈\[0,Rm​a​x\]r\(s,a\)\\in\[0,R\_\{max\}\]\. For any stochastic policyπ\\pi, define the discounted state distribution ofπ\\pias:

dπ​\(s\)=\(1−γ\)​∑t=0∞γt​Pr⁡\(st=s\|π,d0\),d\_\{\\pi\}\(s\)=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\Pr\\\!\\left\(s\_\{t\}=s\\,\\middle\|\\,\\pi,d\_\{0\}\\right\),\(22\)whered0d\_\{0\}is the initial\-state distribution andPr⁡\(st=s∣π,d0\)\\Pr\(s\_\{t\}=s\\mid\\pi,d\_\{0\}\)denotes the state visitation probability at time stepttunder policyπ\\pi\.

Then the following bound holds\[[39](https://arxiv.org/html/2606.26527#bib.bib23)\]:

\|J\(π\)−J\(π′\)\|≤Rmax\(1−γ\)2𝔼s∼dπ∥π\(⋅∣s\)−π′\(⋅∣s\)∥1\.\\displaystyle\\left\|J\(\\pi\)\-J\(\\pi^\{\\prime\}\)\\right\|\\leq\\frac\{R\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\,\\mathbb\{E\}\_\{s\\sim d\_\{\\pi\}\}\\left\\\|\\pi\(\\cdot\\mid s\)\-\\pi^\{\\prime\}\(\\cdot\\mid s\)\\right\\\|\_\{1\}\.\(23\)

###### Proof\.

According to the Performance Difference Lemma\[[17](https://arxiv.org/html/2606.26527#bib.bib24)\], for any two policiesπ\\piandπ′\\pi^\{\\prime\}, the difference in their discounted returns can be expressed as follows:

J​\(π\)−J​\(π′\)=11−γ​𝔼s∼dπ​\[𝔼a∼π\(⋅\|s\)​Aπ′​\(s,a\)\],J\(\\pi\)\-J\(\\pi^\{\\prime\}\)=\\frac\{1\}\{1\-\\gamma\}\\,\\mathbb\{E\}\_\{s\\sim d\_\{\\pi\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)\\right\],\(24\)whereAπ′​\(s,a\)≜Qπ′​\(s,a\)−Vπ′​\(s\)A\_\{\\pi^\{\\prime\}\}\(s,a\)\\triangleq Q\_\{\\pi^\{\\prime\}\}\(s,a\)\-V\_\{\\pi^\{\\prime\}\}\(s\)denotes the advantage function with respect toπ′\\pi^\{\\prime\}\.

For a fixed statess,

𝔼a∼π\(⋅\|s\)​Aπ′​\(s,a\)=∑aπ​\(a\|s\)​Aπ′​\(s,a\)\.\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)=\\sum\_\{a\}\\pi\(a\|s\)A\_\{\\pi^\{\\prime\}\}\(s,a\)\.\(25\)
From the definition of the advantage function, we have:

𝔼a∼π′\(⋅\|s\)​Aπ′​\(s,a\)=0\.\\mathbb\{E\}\_\{a\\sim\\pi^\{\\prime\}\(\\cdot\|s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)=0\.\(26\)
Therefore,

𝔼a∼π\(⋅∣s\)​Aπ′​\(s,a\)−𝔼a∼π′\(⋅∣s\)​Aπ′​\(s,a\)\\displaystyle\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\\mid s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)\-\\mathbb\{E\}\_\{a\\sim\\pi^\{\\prime\}\(\\cdot\\mid s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)\(27\)=\\displaystyle=∑a\(π​\(a∣s\)−π′​\(a∣s\)\)​Aπ′​\(s,a\),\\displaystyle\\sum\_\{a\}\\bigl\(\\pi\(a\\mid s\)\-\\pi^\{\\prime\}\(a\\mid s\)\\bigr\)\\,A\_\{\\pi^\{\\prime\}\}\(s,a\),and using \([26](https://arxiv.org/html/2606.26527#S6.E26)\) we obtain:

𝔼a∼π\(⋅\|s\)​Aπ′​\(s,a\)=∑a\(π​\(a\|s\)−π′​\(a\|s\)\)​Aπ′​\(s,a\)\.\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)=\\sum\_\{a\}\\bigl\(\\pi\(a\|s\)\-\\pi^\{\\prime\}\(a\|s\)\\bigr\)A\_\{\\pi^\{\\prime\}\}\(s,a\)\.\(28\)
Applying Hölder’s inequality\[[8](https://arxiv.org/html/2606.26527#bib.bib25)\]to \([28](https://arxiv.org/html/2606.26527#S6.E28)\) yields

\|𝔼a∼π\(⋅\|s\)Aπ′\(s,a\)\|≤∥π\(⋅\|s\)−π′\(⋅\|s\)∥1⋅∥Aπ′\(s,⋅\)∥∞\.\\left\|\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\_\{\\pi^\{\\prime\}\}\(s,a\)\\right\|\\leq\\\|\\pi\(\\cdot\|s\)\-\\pi^\{\\prime\}\(\\cdot\|s\)\\\|\_\{1\}\\cdot\\\|A\_\{\\pi^\{\\prime\}\}\(s,\\cdot\)\\\|\_\{\\infty\}\.\(29\)
Because0≤r​\(s,a\)≤Rmax0\\leq r\(s,a\)\\leq R\_\{\\max\}, for any policyπ′\\pi^\{\\prime\}we have:

Qπ′​\(s,a\)≤∑t=0∞γt​Rmax=Rmax1−γ,Vπ′​\(s\)≤Rmax1−γ\.Q\_\{\\pi^\{\\prime\}\}\(s,a\)\\leq\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{\\max\}=\\frac\{R\_\{\\max\}\}\{1\-\\gamma\},\\quad V\_\{\\pi^\{\\prime\}\}\(s\)\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\.\(30\)
Hence,

\|Aπ′​\(s,a\)\|=\|Qπ′​\(s,a\)−Vπ′​\(s\)\|≤Rmax1−γ,\|A\_\{\\pi^\{\\prime\}\}\(s,a\)\|=\|Q\_\{\\pi^\{\\prime\}\}\(s,a\)\-V\_\{\\pi^\{\\prime\}\}\(s\)\|\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\},\(31\)which implies‖Aπ′​\(s,⋅\)‖∞≤Rmax1−γ\\\|A\_\{\\pi^\{\\prime\}\}\(s,\\cdot\)\\\|\_\{\\infty\}\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\.

Substituting \([29](https://arxiv.org/html/2606.26527#S6.E29)\) and \([31](https://arxiv.org/html/2606.26527#S6.E31)\) into \([24](https://arxiv.org/html/2606.26527#S6.E24)\) gives

\|J​\(π\)−J​\(π′\)\|\\displaystyle\|J\(\\pi\)\-J\(\\pi^\{\\prime\}\)\|≤11−γ𝔼s∼dπ\[∥π\(⋅\|s\)−π′\(⋅\|s\)∥1⋅Rmax1−γ\]\\displaystyle\\leq\\frac\{1\}\{1\-\\gamma\}\\,\\mathbb\{E\}\_\{s\\sim d\_\{\\pi\}\}\\left\[\\\|\\pi\(\\cdot\|s\)\-\\pi^\{\\prime\}\(\\cdot\|s\)\\\|\_\{1\}\\cdot\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\\right\]\(32\)=Rmax\(1−γ\)2𝔼s∼dπ∥π\(⋅\|s\)−π′\(⋅\|s\)∥1,\\displaystyle=\\frac\{R\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\,\\mathbb\{E\}\_\{s\\sim d\_\{\\pi\}\}\\\|\\pi\(\\cdot\|s\)\-\\pi^\{\\prime\}\(\\cdot\|s\)\\\|\_\{1\},which is exactly Lemma[1](https://arxiv.org/html/2606.26527#Thmlemma1)\. ∎

###### Assumption 1\(Policy Smoothness in Finite Discrete Action Space\)

Consider a finite discrete action space𝒜\\mathcal\{A\}with\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty\. The student policyπs\\pi^\{s\}is assumed to be uniformly lower bounded, i\.e\., there exists a constantδ\>0\\delta\>0such that for all statess∈𝒮s\\in\\mathcal\{S\}and all actionsa∈𝒜a\\in\\mathcal\{A\},

πs​\(a∣s\)≥δ\.\\pi^\{s\}\(a\\mid s\)\\geq\\delta\.\(33\)

###### Theorem 1\.

With the action intervention function incorporated, the lower and upper bounds of the return of the behavior policyJ​\(πmix\)J\(\\pi^\{\\mathrm\{mix\}\}\)are respectively expressed as:

J​\(πt\)−2​\(1−ω\)​Rmax​−log⁡δ\(1−γ\)2\\displaystyle J\(\\pi^\{t\}\)\-\\frac\{\\sqrt\{2\}\(1\-\\omega\)R\_\{\\max\}\\sqrt\{\-\\log\\delta\}\}\{\(1\-\\gamma\)^\{2\}\}\(34\)≤\\displaystyle\\leqJ​\(πmix\)\\displaystyle J\(\\pi^\{\\mathrm\{mix\}\}\)≤\\displaystyle\\leqJ​\(πt\)\+2​\(1−ω\)​Rmax​−log⁡δ\(1−γ\)2,\\displaystyle J\(\\pi^\{t\}\)\+\\frac\{\\sqrt\{2\}\(1\-\\omega\)R\_\{\\max\}\\sqrt\{\-\\log\\delta\}\}\{\(1\-\\gamma\)^\{2\}\},whereω=𝔼s∼dπmix∥𝒯\(s\)\[πt\(⋅\|s\)−πs\(⋅\|s\)\]∥1𝔼s∼dπmix∥πt\(⋅\|s\)−πs\(⋅\|s\)∥1\\omega=\\frac\{\\mathbb\{E\}\_\{s\\sim d\_\{\\pi^\{\\mathrm\{mix\}\}\}\}\\left\\\|\\mathcal\{T\}\(s\)\\left\[\\pi^\{t\}\(\\cdot\|s\)\-\\pi^\{s\}\(\\cdot\|s\)\\right\]\\right\\\|\_\{1\}\}\{\\mathbb\{E\}\_\{s\\sim d\_\{\\pi^\{\\mathrm\{mix\}\}\}\}\\left\\\|\\pi^\{t\}\(\\cdot\|s\)\-\\pi^\{s\}\(\\cdot\|s\)\\right\\\|\_\{1\}\}is the weighted expected intervention rate determined by the intervention function\.

###### Proof\.

Using the mixed behavior policy as defined in \([11](https://arxiv.org/html/2606.26527#S4.E11)\),

𝔼s∼dmix∥πmix\(⋅\|s\)−πt\(⋅\|s\)∥1\\displaystyle\\mathbb\{E\}\_\{s\\sim d\_\{\\text\{mix\}\}\}\\left\\\|\\pi^\{\\text\{mix\}\}\(\\cdot\|s\)\-\\pi^\{t\}\(\\cdot\|s\)\\right\\\|\_\{1\}\(35\)=𝔼s∼dmix∥𝒯\(s\)πt\(⋅\|s\)\+\(1−𝒯\(s\)\)πs\(⋅\|s\)−πt\(⋅\|s\)∥1\\displaystyle=\\mathbb\{E\}\_\{s\\sim d\_\{\\text\{mix\}\}\}\\left\\\|\\mathcal\{T\}\(s\)\\pi^\{t\}\(\\cdot\|s\)\+\\bigl\(1\-\\mathcal\{T\}\(s\)\\bigr\)\\pi^\{s\}\(\\cdot\|s\)\-\\pi^\{t\}\(\\cdot\|s\)\\right\\\|\_\{1\}=\(1−ω\)𝔼s∼dmix∥πs\(⋅\|s\)−πt\(⋅\|s\)∥1\.\\displaystyle=\(1\-\\omega\)\\mathbb\{E\}\_\{s\\sim d\_\{\\text\{mix\}\}\}\\left\\\|\\pi^\{s\}\(\\cdot\|s\)\-\\pi^\{t\}\(\\cdot\|s\)\\right\\\|\_\{1\}\.
Applying Pinsker’s inequality\[[6](https://arxiv.org/html/2606.26527#bib.bib26)\], we have:

∥πs\(⋅\|s\)−πt\(⋅\|s\)∥1≤2DKL\(πt\(⋅\|s\)∥πs\(⋅\|s\)\)\.\\left\\\|\\pi^\{s\}\(\\cdot\|s\)\-\\pi^\{t\}\(\\cdot\|s\)\\right\\\|\_\{1\}\\leq\\sqrt\{2D\_\{\\text\{KL\}\}\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\parallel\\pi^\{s\}\(\\cdot\|s\)\\bigr\)\}\.\(36\)
For any fixed statess, the Kullback\-Leibler \(KL\) divergence between the teacher policy and the student policy is defined as:

DKL\(πt\(⋅\|s\)∥πs\(⋅\|s\)\)=∑aπt\(a\|s\)logπt​\(a\|s\)πs​\(a\|s\)\.D\_\{\\text\{KL\}\}\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\parallel\\pi^\{s\}\(\\cdot\|s\)\\bigr\)=\\sum\_\{a\}\\pi^\{t\}\(a\|s\)\\log\\frac\{\\pi^\{t\}\(a\|s\)\}\{\\pi^\{s\}\(a\|s\)\}\.\(37\)
Sinceπs​\(a\|s\)≥δ\\pi^\{s\}\(a\|s\)\\geq\\delta, it follows that1πs​\(a\|s\)≤1δ\\frac\{1\}\{\\pi^\{s\}\(a\|s\)\}\\leq\\frac\{1\}\{\\delta\}\. Therefore:

DKL\(πt\(⋅\|s\)∥πs\(⋅\|s\)\)\\displaystyle D\_\{\\text\{KL\}\}\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\parallel\\pi^\{s\}\(\\cdot\|s\)\\bigr\)\(38\)≤∑aπt​\(a∣s\)​log⁡πt​\(a∣s\)δ\\displaystyle\\leq\\sum\_\{a\}\\pi^\{t\}\(a\\mid s\)\\log\\frac\{\\pi^\{t\}\(a\\mid s\)\}\{\\delta\}=∑aπt​\(a∣s\)​log⁡πt​\(a∣s\)−log⁡δ​∑aπt​\(a∣s\)\.\\displaystyle=\\sum\_\{a\}\\pi^\{t\}\(a\\mid s\)\\log\\pi^\{t\}\(a\\mid s\)\-\\log\\delta\\sum\_\{a\}\\pi^\{t\}\(a\\mid s\)\.
Using∑aπt​\(a\|s\)=1\\sum\_\{a\}\\pi^\{t\}\(a\|s\)=1and the definition of entropyH\(πt\(⋅\|s\)\)=−∑aπt\(a\|s\)logπt\(a\|s\)≥0H\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\bigr\)=\-\\sum\_\{a\}\\pi^\{t\}\(a\|s\)\\log\\pi^\{t\}\(a\|s\)\\geq 0, we obtain:

DKL\(πt\(⋅\|s\)∥πs\(⋅\|s\)\)≤−H\(πt\(⋅\|s\)\)−logδ≤−logδ\.D\_\{\\text\{KL\}\}\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\parallel\\pi^\{s\}\(\\cdot\|s\)\\bigr\)\\leq\-H\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\bigr\)\-\\log\\delta\\leq\-\\log\\delta\.\(39\)
By combining \([35](https://arxiv.org/html/2606.26527#S6.E35)\), \([36](https://arxiv.org/html/2606.26527#S6.E36)\), \([39](https://arxiv.org/html/2606.26527#S6.E39)\) with Lemma[1](https://arxiv.org/html/2606.26527#Thmlemma1), the following result can be obtained:

\|J​\(πmix\)−J​\(πt\)\|\\displaystyle\\left\|J\(\\pi^\{\\text\{mix\}\}\)\-J\(\\pi^\{t\}\)\\right\|\(40\)≤Rmax\(1−γ\)2𝔼s∼dmix∥πmix\(⋅\|s\)−πt\(⋅\|s\)∥1\\displaystyle\\leq\\frac\{R\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\mathbb\{E\}\_\{s\\sim d\_\{\\text\{mix\}\}\}\\left\\\|\\pi^\{\\text\{mix\}\}\(\\cdot\|s\)\-\\pi^\{t\}\(\\cdot\|s\)\\right\\\|\_\{1\}=\(1−ω\)​Rmax\(1−γ\)2𝔼s∼dmix∥πs\(⋅\|s\)−πt\(⋅\|s\)∥1\\displaystyle=\\frac\{\(1\-\\omega\)R\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\mathbb\{E\}\_\{s\\sim d\_\{\\text\{mix\}\}\}\\left\\\|\\pi^\{s\}\(\\cdot\|s\)\-\\pi^\{t\}\(\\cdot\|s\)\\right\\\|\_\{1\}≤2​\(1−ω\)​Rmax\(1−γ\)2​𝔼s∼dmix​DKL\(πt\(⋅\|s\)∥πs\(⋅\|s\)\)\\displaystyle\\leq\\frac\{\\sqrt\{2\}\(1\-\\omega\)R\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\mathbb\{E\}\_\{s\\sim d\_\{\\text\{mix\}\}\}\\sqrt\{D\_\{\\text\{KL\}\}\\bigl\(\\pi^\{t\}\(\\cdot\|s\)\\parallel\\pi^\{s\}\(\\cdot\|s\)\\bigr\)\}≤2​\(1−ω\)​Rmax​−log⁡δ\(1−γ\)2\.\\displaystyle\\leq\\frac\{\\sqrt\{2\}\(1\-\\omega\)R\_\{\\max\}\\sqrt\{\-\\log\\delta\}\}\{\(1\-\\gamma\)^\{2\}\}\.
Therefore, we obtain:

J​\(πt\)−2​\(1−ω\)​Rmax​−log⁡δ\(1−γ\)2\\displaystyle J\(\\pi^\{t\}\)\-\\frac\{\\sqrt\{2\}\(1\-\\omega\)R\_\{\\max\}\\sqrt\{\-\\log\\delta\}\}\{\(1\-\\gamma\)^\{2\}\}\(41\)≤\\displaystyle\\leqJ​\(πmix\)\\displaystyle J\(\\pi^\{\\mathrm\{mix\}\}\)≤\\displaystyle\\leqJ​\(πt\)\+2​\(1−ω\)​Rmax​−log⁡δ\(1−γ\)2\.\\displaystyle J\(\\pi^\{t\}\)\+\\frac\{\\sqrt\{2\}\(1\-\\omega\)R\_\{\\max\}\\sqrt\{\-\\log\\delta\}\}\{\(1\-\\gamma\)^\{2\}\}\.∎

## VIIExperimental Setup

### VII\-ASimulation Environment

The proposed method is evaluated in a highway lane\-changing scenario\. In this setting, the ego vehicle makes lane\-changing decisions under the dynamic interactions and influences of surrounding vehicles, with the objective of safely completing a lane change toward the rightmost target lane and maintaining stable driving\. At the beginning of each episode, the ego vehicle and surrounding vehicles are randomly initialized on the road\. The simulation parameters are set as follows: the lane width is 5 m; the vehicle length and width are 5 m and 2 m, respectively; the initial vehicle speed is randomly sampled from 23 to 25 m/s; the maximum speed is set to 30 m/s; and the desired speed is specified within 20–30 m/s\. If the ego vehicle makes an improper lane\-change decision during driving that results in an excessively close distance to other vehicles, a collision, or departure from the road boundary, the episode is deemed a failure, and the environment is immediately reset to start the next episode\.

### VII\-BImplementation Details

The simulation environment is developed based on a modified version of the open\-sourcegym\-highway\-envsimulator\[[22](https://arxiv.org/html/2606.26527#bib.bib41)\]\. Vehicle motion is propagated using a kinematic bicycle model\[[30](https://arxiv.org/html/2606.26527#bib.bib42)\]\. For HDVs, longitudinal dynamics and lateral lane\-changing behaviors are governed by the Intelligent Driver Model \(IDM\)\[[35](https://arxiv.org/html/2606.26527#bib.bib43)\]and the MOBIL model\[[3](https://arxiv.org/html/2606.26527#bib.bib44)\], respectively\. The simulation frequency is set to 10 Hz, while the decision\-making frequency is set to 2 Hz\. To ensure reproducibility and statistical robustness, all algorithms are trained by interacting with the environment for5×1055\\times 10^\{5\}steps using three different random seeds\. The discount factor and learning rate are set to0\.990\.99and1×10−41\\times 10^\{\-4\}, respectively\. Parameter gradients are updated every 100 steps, and each update samples a mini\-batch of 256 transitions from the replay buffer\. Further details are shown in Table[I](https://arxiv.org/html/2606.26527#S7.T1)\. All experiments are conducted on a personal PC equipped with an Intel Core i5\-10210U CPU and 16 GB RAM111[https://github\.com/HuangWJ\-12/TG\-STRL](https://github.com/HuangWJ-12/TG-STRL)\.

TABLE I:Hyperparameters Setting\.ParameterDescriptionValueTsafeT\_\{\\text\{safe\}\}Safe threshold for time headway1\.2 sηπ\\eta\_\{\\pi\}Actor network learning rate1×10−41\\times 10^\{\\mathrm\{\-4\}\}ηQ\\eta\_\{Q\}Reward critic network learning rate1×10−41\\times 10^\{\\mathrm\{\-4\}\}ηC\\eta\_\{C\}Cost critic network learning rate1×10−41\\times 10^\{\\mathrm\{\-4\}\}ddCost limit0\.05γ\\gammaDiscount factor0\.99ℬ\\mathcal\{B\}Mini\-batch size256nen\_\{e\}Total steps per epoch4000τinit\\tau\_\{\\text\{init\}\}Initial intervention threshold0\.0κ\\kappaIntervention threshold update step size0\.5We compare it against several state\-of\-the\-art methods: SAC\[[12](https://arxiv.org/html/2606.26527#bib.bib72)\], PPO\-Lag\[[34](https://arxiv.org/html/2606.26527#bib.bib73)\], and TS2C\[[39](https://arxiv.org/html/2606.26527#bib.bib23)\]\. Considering safety and traffic efficiency, the following evaluation metrics are adopted: \(i\) Average Reward\(↑\)\(\\uparrow\): the average reward obtained by the ego vehicle per episode; \(ii\) Average Cost\(↓\)\(\\downarrow\): the average cost of the ego vehicle for each episode; \(iii\) Crash Ratio\(↓\)\(\\downarrow\): the ratio of the ego vehicle that collides with surrounding vehicles, represented as a decimal value between 0 and 1; \(iv\) Average Velocity\(↑\)\(\\uparrow\): the average speed maintained by the vehicle during driving\.

## VIIIExperiment and Result Analysis

### VIII\-AConvergence Performance

Fig\.[4](https://arxiv.org/html/2606.26527#S8.F4)presents the average reward and average cost curves, and the crash\-ratio boxplots for all compared methods during training\. As shown by the average reward curves Fig\.[4](https://arxiv.org/html/2606.26527#S8.F4)\(a\), the proposed method exhibits rapid performance improvement in the early stage of training and quickly reaches a high reward level with relatively few environment interactions\. In contrast, although SAC gradually attains an average reward comparable to, or even slightly higher than, that of the proposed method in the later stage of training, it shows substantial fluctuations during the early training phase\. PPO\-Lag also achieves a rapid increase in return at the beginning of training, but its final reward remains clearly lower than that of the proposed method\. Notably, the reward curve of TS2C becomes nearly stable at an early stage and shows almost no further improvement throughout training\. This is due to TS2C is able to form a relatively fixed behavioral pattern quickly, yet its subsequent policy optimization capability is weak, making it more prone to getting trapped in a local optimum\. As a result, its reward level is markedly lower than that of the proposed method and the other baselines\.

From the perspective of safety, the average cost curves as shown in Fig\.[4](https://arxiv.org/html/2606.26527#S8.F4)\(b\) indicate that the proposed method rapidly reduces the cost to a low level in the early training stage and maintains only small fluctuations throughout the entire training process, demonstrating strong safety and training stability\. By contrast, the average cost of SAC remains consistently high and fluctuates considerably\. Although PPO\-Lag yields a lower average cost than SAC overall, it is still significantly higher than that of the proposed method, suggesting that traditional Lagrangian\-constrained methods remain limited in their ability to control risk in dynamic traffic environments\. TS2C maintains a relatively low average cost, but its unsatisfactory reward level indicates that this low cost mainly results from an overly conservative policy\. Its safety improvement is therefore achieved at the expense of task efficiency, rather than through genuinely efficient and safe decision\-making\.

The crash\-ratio boxplots in Fig\.[4](https://arxiv.org/html/2606.26527#S8.F4)\(c\) further support these observations\. The proposed method exhibits a concentrated crash\-ratio distribution, with both the median and interquartile range remaining at relatively low levels, indicating not only superior average safety but also better robustness and consistency across different evaluation episodes\. In contrast, SAC and PPO\-Lag show more dispersed crash\-ratio distributions and contain more high\-crash outliers, implying that their safety performance is less stable across different scenarios\. Although TS2C has a relatively low median crash ratio, it contains more outliers than the proposed method\. Overall, our method not only ensures safety but also improves training efficiency and stability in autonomous highway lane\-changing tasks\.

![Refer to caption](https://arxiv.org/html/2606.26527v1/pictures/Comparative_experiment.png)Figure 4:Comparative study\. \(a\) Average reward, \(b\) Average cost, \(c\) Crash ratio\. Training curves of average reward and average cost versus training steps for different methods in the target\-task environment, together with a box plot of the crash ratio\. The compared methods include the proposed method, SAC\[[12](https://arxiv.org/html/2606.26527#bib.bib72)\], PPO\-Lag\[[34](https://arxiv.org/html/2606.26527#bib.bib73)\], and TS2C\[[39](https://arxiv.org/html/2606.26527#bib.bib23)\]\. The results show that the proposed method achieves a superior balance between safety and efficiency, with faster and more stable convergence, lower costs, and a reduced crash ratio while maintaining competitive task returns\.TABLE II:Comparative Study Results\.MethodAverage RewardAverage CostCrash RatioAverage Velocity \(m/s\)LowMediumHighLowMediumHighLowMediumHighLowMediumHighOurs48\.0147\.3446\.030\.090\.110\.160\.010\.000\.0121\.9721\.8021\.30SAC\[[12](https://arxiv.org/html/2606.26527#bib.bib72)\]45\.3443\.5743\.161\.921\.691\.840\.060\.080\.0921\.3720\.7620\.46PPO\-Lag\[[34](https://arxiv.org/html/2606.26527#bib.bib73)\]42\.9942\.5442\.160\.640\.680\.850\.050\.050\.0620\.3820\.3120\.18TS2C\[[39](https://arxiv.org/html/2606.26527#bib.bib23)\]27\.0427\.0027\.000\.160\.230\.250\.000\.000\.016\.796\.656\.42

#### VIII\-A1Comparative Visualization

![Refer to caption](https://arxiv.org/html/2606.26527v1/pictures/Comparative-Visualization.png)Figure 5:Visualization of lane\-changing behaviors under different methods\. The proposed method completes the lane change smoothly and maintains stable driving, whereas SAC exhibits aggressive consecutive lane changes that lead to collision, and PPO\-Lag and TS2C show increasingly conservative behaviors that reduce driving efficiency\.

### VIII\-BPerformance Evaluation

#### VIII\-B1Traffic Safety

For traffic safety, the proposed method demonstrates strong safety performance across all three traffic density scenarios\. In the medium\-density setting, it achieves an average cost of 0\.11, representing reductions of approximately 93\.5% and 83\.8% compared to SAC \(1\.69\) and PPO\-Lag \(0\.68\), respectively\. Moreover, its collision rate is 0\.00, outperforming SAC \(0\.08\) and PPO\-Lag \(0\.05\)\. In both low\- and high\-density scenarios, the proposed method maintains similarly stable safety performance\. In contrast, SAC consistently exhibits higher cost and collision risk across all densities\. Although PPO\-Lag improves safety relative to SAC, it remains inferior overall\. While TS2C achieves a collision rate comparable to the proposed method, its safety performance largely stems from an overly conservative control strategy\. Overall, the proposed method effectively satisfies safety constraints under varying traffic densities, demonstrating strong safety robustness\.

#### VIII\-B2Traffic Efficiency

In terms of traffic efficiency, the proposed method maintains high task performance while ensuring safety\. As shown in Table[II](https://arxiv.org/html/2606.26527#S8.T2), in the medium\-density scenario, it attains an average speed of 21\.80 m/s, exceeding SAC \(20\.76 m/s\) and PPO\-Lag \(20\.31 m/s\) by 5\.0% and 7\.3%, respectively, indicating its ability to sustain high traffic efficiency under safety constraints\. In contrast, TS2C achieves an average speed of only 6\.65 m/s, reflecting excessive conservatism and failing to meet the fundamental requirements of lane\-changing tasks in terms of efficiency and task completion quality\. In both low\- and high\-density scenarios, the proposed method exhibits stable performance\. Although SAC also achieves relatively high speeds, its significantly higher safety cost and collision rate indicate that its efficiency is obtained at the expense of safety\. PPO\-Lag performs poorly compared to the proposed method, while TS2C consistently exhibits conservative behavior\. In summary, the proposed method achieves a favorable balance between safety and efficiency across different traffic densities\.

To provide a more intuitive assessment of the capability of the proposed method, we visualize the execution process of the highway lane\-changing task\. Fig\.[5](https://arxiv.org/html/2606.26527#S8.F5)visualizes the comparative results of the proposed method and the baseline methods\. As shown in Fig\.[5](https://arxiv.org/html/2606.26527#S8.F5)\(a\), the proposed method is able to successfully complete the lane\-changing task at an early stage of training and maintain stable driving thereafter\. By contrast, SAC, shown in Fig\.[5](https://arxiv.org/html/2606.26527#S8.F5)\(b\), adopts a more aggressive decision\-making strategy\. In an effort to maintain a high speed, it performs consecutive cross\-lane maneuvers without sufficiently accounting for the dynamic gaps and constraints of vehicles in adjacent lanes, which ultimately leads to a collision\. Although PPO\-Lag \(Fig\.[5](https://arxiv.org/html/2606.26527#S8.F5)\(c\)\) also completes the lane\-changing task successfully, its policy is relatively conservative, causing the vehicle to gradually decelerate during subsequent driving and maintain an increasingly large distance from the preceding vehicle\. This phenomenon is even more pronounced in TS2C \(Fig\.[5](https://arxiv.org/html/2606.26527#S8.F5)\(d\)\), where the policy exhibits stronger conservatism and significantly reduces the driving efficiency of the vehicle\.

### VIII\-CReal\-World Data Validation

We further validate the proposed method using the US Highway 101 \(US\-101\) dataset from the Next Generation Simulation \(NGSIM\) open data, with the corresponding scenario illustrated in Fig\.[6](https://arxiv.org/html/2606.26527#S8.F6), and compare its performance with the baseline method PPO\-Lag\. From the US\-101 dataset, we extract information about surrounding vehicles, which then travel in the simulation environment with their initial positions and velocities provided by the dataset\.

![Refer to caption](https://arxiv.org/html/2606.26527v1/pictures/NGSIM.png)Figure 6:Illustration of the NGSIM US\-101 Scenario\. The green vehicle denotes the ego vehicle, while the yellow vehicles denote surrounding vehicles\.TABLE III:Performance Comparison on the US\-101 DatasetMethodAverageRewardAverageCostCrashRatioAverage Velocity\(m/s\)Ours37\.460\.740\.0029\.90PPO\-Lag20\.970\.730\.3510\.55As shown in Table[III](https://arxiv.org/html/2606.26527#S8.T3), from the perspective of traffic safety, the proposed method yields an average cost of 0\.74 on the US\-101 dataset, which is comparable to the 0\.73 achieved by PPO\-Lag, while attaining a crash ratio of 0, substantially outperforming PPO\-Lag’s 0\.35\. In addition, the proposed method achieves an average reward of 37\.46, significantly higher than PPO\-Lag’s 20\.97, and increases the average velocity from 10\.55 m/s to 29\.9 m/s, indicating that it can maintain higher traffic efficiency under realistic traffic interactions\. These results demonstrate that, in real\-world traffic scenarios based on the NGSIM dataset, the proposed method achieves superior traffic efficiency while maintaining a high level of safety, thereby further validating its effectiveness and robustness in realistic driving environments\.

### VIII\-DAblation Study

![Refer to caption](https://arxiv.org/html/2606.26527v1/pictures/Ablation_study.png)Figure 7:Ablation study\. \(a\) Average reward, \(b\) Average cost, \(c\) Crash ratio\. Training curves of average reward and average cost versus training steps under different ablation configurations in the target\-task environment, together with a box plot of the crash ratio\. The compared settings include the complete method, w/o SG, w/o SG and ID, and w/o SG, ID, and DS\. The results show that the complete method achieves higher returns, lower costs, and a reduced crash ratio compared with all ablated variants, confirming the benefit of integrating safety\-guided, intervention decay, and dual\-source data training\.TABLE IV:Ablation Study Results\.SGIDDSAverage RewardAverage CostCrash RatioAverage Velocity \(m/s\)LowMediumhighLowMediumHighLowMediumHighLowMediumHigh\-\-\-34\.7537\.3036\.230\.760\.750\.760\.370\.390\.3420\.5220\.0921\.41\-\-✓42\.2741\.7638\.870\.440\.440\.570\.130\.210\.2419\.9719\.3719\.63\-✓✓47\.3346\.9945\.820\.230\.280\.380\.020\.010\.0319\.4019\.2719\.77✓✓✓48\.0147\.3446\.200\.090\.110\.140\.010\.000\.0121\.9721\.8021\.44To systematically evaluate the contribution of each module, we conduct an ablation study with four configurations: \(i\) the complete method, which includes safety\-guided \(SG\), intervention decay \(ID\), and dual\-source data training \(DS\); \(ii\) w/o SG, which removes the safety\-guided policy learning module, including both the teacher\-induced reward shaping and the optimization weight construction; \(iii\) w/o SG and ID, which further disables the intervention decay mechanism by fixing the intervention threshold; and \(iv\) w/o SG, ID and DS, which serves as the baseline without any proposed modules\. Fig\.[7](https://arxiv.org/html/2606.26527#S8.F7)presents the training curves of average reward and average cost, along with the box plots of crash ratio\. After training, all algorithms are evaluated using three random seeds with 100 test episodes per seed, and the averaged results are summarized in Table[IV](https://arxiv.org/html/2606.26527#S8.T4)\.

#### VIII\-D1Impact of the Safety\-Guided Mechanism

To evaluate the contribution of the safety\-guided mechanism, we compare the complete method with the variant in which SG is removed\. As shown in Fig\.[7](https://arxiv.org/html/2606.26527#S8.F7), both methods can achieve the high reward levels, indicating that even without SG, the student policy can still benefit from the high\-quality samples generated by teacher intervention\. However, the complete method consistently maintains a lower average cost throughout the entire training process, demonstrating that SG plays a significant role in improving training safety\. The evaluation results in Table[IV](https://arxiv.org/html/2606.26527#S8.T4)further confirm this observation\. In the medium\-density scenario, for instance, the full method outperforms the variant without SG by increasing the average reward from 46\.99 to 47\.34, while simultaneously reducing the average cost from 0\.28 to 0\.11 \(a 60\.7% reduction\) and lowering the collision rate from 0\.01 to 0\.00\. Meanwhile, the average speed increases from 19\.27 m/s to 21\.80 m/s\. These results indicate that, by incorporating the teacher’s behavioral evaluation into the student’s learning process, the SG mechanism enhances the quality of knowledge transfer, leading to more stable transfer performance and a more effective balance between safety and reward\.

#### VIII\-D2Impact of Intervention Decay

To analyze the role of intervention decay, we further compare the variant without SG with the variant in which both SG and ID are removed\. As shown in Fig\.[7](https://arxiv.org/html/2606.26527#S8.F7), once the intervention decay mechanism is disabled, the model exhibits low reward and high cost during the early stage of training\. Although its performance subsequently approaches the convergence region rapidly, both reward and safety deteriorate noticeably in the later stage of training, accompanied by larger fluctuations and an overall degradation in performance\. The evaluation results in Table[IV](https://arxiv.org/html/2606.26527#S8.T4)show the same trend\. After removing the ID module, the average cost increases by up to 91\.3% across low\-, medium\-, and high\-density scenarios, while the collision rates rise significantly to 0\.13, 0\.21, and 0\.24, respectively\. These findings suggest that the intervention decay mechanism plays a crucial role in balancing early\-stage safety protection and late\-stage policy autonomy\. Without adaptive decay, teacher intervention cannot be relaxed in a timely manner as the student policy becomes safer, hindering a smooth transition from teacher\-assisted behavior to autonomous decision\-making\. As a consequence, both reward performance and safety outcomes are substantially compromised\.

#### VIII\-D3Impact of Dual\-Source Data Training

Finally, to assess the effectiveness of dual\-source data training, we compare the variant without SG and ID with the variant in which SG, ID, and DS are all removed\. As can be observed from the training curves in Fig\.[7](https://arxiv.org/html/2606.26527#S8.F7), introducing DS generally leads to better learning performance, with lower cost and a more stable optimization process\. The crash\-ratio boxplot also indicates that the collision frequency is significantly reduced when DS is enabled\. Table[IV](https://arxiv.org/html/2606.26527#S8.T4)further confirms this gain\. The baseline method \(w/o SG, ID, and DS\) exhibits a collision rate as high as 0\.39\. Although it achieves relatively high speed, this is largely a superficial effect resulting from aggressive decision\-making rather than effective optimization under safety constraints\. These results demonstrate that DS enhances both the diversity and safety of replay data by jointly leveraging student\-generated samples and teacher\-intervened samples, thereby mitigating unsafe behavior patterns commonly observed in single\-source learning\. Consequently, even in the absence of SG and ID, DS remains an effective mechanism for improving training stability and reducing collision risk\.

In summary, the ablation results verify that the three modules contribute in different yet complementary ways\. SG improves the efficiency of teacher knowledge transfer and achieves a better balance between safety and performance; ID enables a smooth transition from teacher intervention to autonomous control and is crucial for stable learning; and DS enhances safety robustness by enriching the training data distribution\. Their integration yields the best overall performance, thereby validating the effectiveness of the proposed teacher\-guided safe transfer reinforcement learning framework\.

### VIII\-ESensitivity Analysis

![Refer to caption](https://arxiv.org/html/2606.26527v1/pictures/Sensitivity_analysis.png)Figure 8:Sensitivity analysis of the intervention thresholdτinit\\tau\_\{\\text\{init\}\}\. \(a\) Average reward, \(b\) Average cost\. Training curves of average reward \(top\) and average cost \(bottom\) under different threshold settings \(τinit∈\{0\.0,0\.25,0\.4\}\\tau\_\{\\text\{init\}\}\\in\\\{0\.0,0\.25,0\.4\\\}\), illustrating their impact on learning dynamics and convergence behavior\. Among the three settings,τinit=0\.0\\tau\_\{\\text\{init\}\}=0\.0yields the best overall performance\.To evaluate the impact of the initial intervention threshold on algorithm performance, a sensitivity analysis is conducted by varying its initial value, withτinit∈\{0\.0,0\.25,0\.4\}\\tau\_\{\\text\{init\}\}\\in\\\{0\.0,0\.25,0\.4\\\}, and training each configuration independently\. The corresponding training curves of average reward and average cost are illustrated in Fig\.[8](https://arxiv.org/html/2606.26527#S8.F8)\. From the overall results, all configurations converge to similar performance levels, indicating that the proposed method exhibits strong robustness to the choice of the threshold\. However, in the early stage of training, the learning dynamics are significantly influenced by different threshold settings\. When the threshold is small \(e\.g\.,τinit=0\.0\\tau\_\{\\text\{init\}\}=0\.0\), more frequent teacher intervention enables the policy to achieve higher initial rewards while maintaining lower costs, reflecting stronger safety guidance and higher\-quality early decisions\. In contrast, larger thresholds \(e\.g\.,τinit=0\.25\\tau\_\{\\text\{init\}\}=0\.25andτinit=0\.4\\tau\_\{\\text\{init\}\}=0\.4\) allow for greater autonomous exploration\. Although this expands the policy search space, it results in slower reward improvement and larger cost fluctuations in the early stage, indicating increased uncertainty and risk exposure during exploration\. Despite these differences, all configurations eventually stabilize and achieve comparable performance, further demonstrating the stability and adaptability of the proposed method under varying intervention intensities\.

## IXConclusion

In this paper, we propose a sample\-efficient transfer reinforcement learning framework via adaptive reward shaping and policy\-ratio reweighting strategy for autonomous highway lane changing\. The proposed method combines adaptive teacher intervention, teacher\-guided value\-target construction, and optimization weighting of teacher\-intervened samples within a unified constrained reinforcement learning framework, thereby improving exploration safety, transfer stability, and the utilization of teacher knowledge during training\. These components are tightly coupled within a unified constrained reinforcement learning framework, enabling safer exploration in the early stage and more stable knowledge transfer throughout training\. In addition, the effect of the mixed\-behavior policy was further analyzed through theoretical return bounds\. Experimental results show that the proposed method achieves a favorable balance between traffic efficiency and safety and outperforms existing baseline methods in autonomous highway lane\-changing tasks\. In future work, we will investigate fusing the teacher and student actions in a more flexible manner and explore the meta\-transfer reinforcement learning paradigm for safe and few\-shot learning\.

## References

- \[1\]\(2018\)Safe reinforcement learning via shielding\.InProc\. AAAI Conf\. Artif\. Intell\.,Vol\.32\.Cited by:[§II\-B](https://arxiv.org/html/2606.26527#S2.SS2.p1.1)\.
- \[2\]D\. Anand, V\. Gupta, P\. Paruchuri, and B\. Ravindran\(2021\)An enhanced advising model in teacher\-student framework using state categorization\.InProc\. AAAI Conf\. Artif\. Intell\.,Vol\.35,pp\. 6653–6660\.Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[3\]T\.J\. Ayres, L\. Li, D\. Schleuning, and D\. Young\(2001\)Preferred time\-headway of highway drivers\.InProc\. 4th IEEE Intell\. Transp\. Syst\. Conf\. \(ITSC\),Vol\.,pp\. 826–829\.Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p1.3)\.
- \[4\]V\. S\. Borkar\(2005\)An actor\-critic algorithm for constrained markov decision processes\.Syst\. Control Lett\.54\(3\),pp\. 207–213\.Cited by:[§III\-A](https://arxiv.org/html/2606.26527#S3.SS1.p1.24)\.
- \[5\]S\. Carr, N\. Jansen, S\. Junges, and U\. Topcu\(2023\)Safe reinforcement learning via shielding under partial observability\.InProc\. AAAI Conf\. Artif\. Intell\.,Vol\.37,pp\. 14748–14756\.Cited by:[§II\-B](https://arxiv.org/html/2606.26527#S2.SS2.p1.1)\.
- \[6\]T\. M\. Cover and J\. A\. Thomas\(2006\)Elements of information theory\.2 edition,Wiley\.Cited by:[§VI](https://arxiv.org/html/2606.26527#S6.11.p2.1)\.
- \[7\]A\. Filos, C\. Lyle, Y\. Gal, S\. Levine, N\. Jaques, and G\. Farquhar\(2021\)Psiphi\-learning: reinforcement learning with demonstrations using successor features and inverse temporal difference learning\.InProc\. 35th Int\. Conf\. Mach\. Learn\.,pp\. 3305–3317\.Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[8\]G\. B\. Folland\(1999\)Real analysis: modern techniques and their applications\.John Wiley & Sons\.Cited by:[§VI](https://arxiv.org/html/2606.26527#S6.6.p6.1)\.
- \[9\]X\. Guo, S\. Chang, M\. Yu, G\. Tesauro, and M\. Campbell\(2019\)Hybrid reinforcement learning with expert state sequences\.InProc\. AAAI Conf\. Artif\. Intell\.,Vol\.33,pp\. 3739–3746\.Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[10\]Y\. Guo, J\. Gao, Z\. Wu, C\. Shi, and J\. Chen\(2023\)Reinforcement learning with demonstrations from mismatched task under sparse reward\.InProc\. Conf\. Robot Learn\.,pp\. 1146–1156\.Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[11\]Y\. Guo, X\. Zhang, S\. Stepputtis, J\. Campbell, and K\. P\. Sycara\(2025\)Adaptive action advising with different rewards\.InProc\. Mach\. Learn\. Res\.,pp\. 252–267\.Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[12\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProc\. 35th Int\. Conf\. Mach\. Learn\.,pp\. 1861–1870\.Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p2.4),[Figure 4](https://arxiv.org/html/2606.26527#S8.F4),[TABLE II](https://arxiv.org/html/2606.26527#S8.T2.3.1.4.1)\.
- \[13\]K\. Hsu, A\. Z\. Ren, D\. P\. Nguyen, A\. Majumdar, and J\. F\. Fisac\(2023\)Sim\-to\-lab\-to\-real: safe reinforcement learning with shielding and generalization guarantees\.Artif\. Intell\.314,pp\. 103811\.Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p1.1)\.
- \[14\]Z\. Huang, Z\. Sheng, C\. Ma, and S\. Chen\(2024\)Human as ai mentor: enhanced human\-in\-the\-loop reinforcement learning for safe and efficient autonomous driving\.Commun\. Transp\. Res\.,pp\. 100127\.External Links:[Document](https://dx.doi.org/10.1016/j.commtr.2024.100127)Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[15\]E\. Ilhan, J\. Gow, and D\. Perez Liebana\(2021\)Action advising with advice imitation in deep reinforcement learning\.InProc\. 20th Int\. Conf\. Auton\. Agents Multiagent Syst\.,AAMAS ’21,pp\. 629–637\.Cited by:[§II\-B](https://arxiv.org/html/2606.26527#S2.SS2.p1.1)\.
- \[16\]E\. Ilhan, J\. Gow, and D\. Perez\-Liebana\(2019\)Teaching on a budget in multi\-agent deep reinforcement learning\.Proc\. 2019 IEEE Conf\. Games \(CoG\),pp\. 1–8\.Cited by:[§II\-B](https://arxiv.org/html/2606.26527#S2.SS2.p1.1)\.
- \[17\]S\. Kakade and J\. Langford\(2002\)Approximately optimal approximate reinforcement learning\.InProc\. 19th Int\. Conf\. Mach\. Learn\.,pp\. 267–274\.Cited by:[§VI](https://arxiv.org/html/2606.26527#S6.1.p1.2)\.
- \[18\]B\. Kang, Z\. Jie, and J\. Feng\(2018\)Policy optimization with demonstrations\.InProc\. 35th Int\. Conf\. Mach\. Learn\.,Vol\.80,pp\. 2474–2483\.Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[19\]B\. R\. Kiran, I\. Sobh, V\. Talpaert, P\. Mannion, A\. A\. A\. Sallab, S\. Yogamani, and P\. Pérez\(2022\)Deep reinforcement learning for autonomous driving: a survey\.IEEE Trans\. Intell\. Transp\. Syst\.23\(6\),pp\. 4909–4926\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2021.3054625)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p1.1)\.
- \[20\]H\. Krasowski, X\. Wang, and M\. Althoff\(2020\)Safe reinforcement learning for autonomous lane changing using set\-based prediction\.InIEEE Int\. Conf\. Intell\. Transp\. Syst\. \(ITSC\),pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1109/ITSC45102.2020.9294259)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p1.1)\.
- \[21\]S\. Lee, Y\. Seo, K\. Lee, P\. Abbeel, and J\. Shin\(2022\)Offline\-to\-online reinforcement learning via balanced replay and pessimistic q\-ensemble\.InProc\. Conf\. Robot Learn\.,pp\. 1702–1712\.Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[22\]E\. Leurent\(2018\)An environment for autonomous driving decision\-making\.GitHub repository\.Note:[https://github\.com/eleurent/highway\-env](https://github.com/eleurent/highway-env)Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p1.3)\.
- \[23\]S\. Li and O\. Bastani\(2020\)Robust model predictive shielding for safe reinforcement learning with stochastic dynamics\.InProc\. IEEE Int\. Conf\. Robot\. Autom\. \(ICRA\),Vol\.,pp\. 7166–7172\.Cited by:[§II\-B](https://arxiv.org/html/2606.26527#S2.SS2.p1.1)\.
- \[24\]X\. Liang, Y\. Liu, T\. Chen, M\. Liu, and Q\. Yang\(2019\)Federated transfer reinforcement learning for autonomous driving\.arXiv preprint arXiv:1910\.06001\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1910.06001)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.
- \[25\]G\. Libardi, G\. De Fabritiis, and S\. Dittert\(2021\)Guided exploration with proximal policy optimization using a single demonstration\.InProc\. 38th Int\. Conf\. Mach\. Learn\.,pp\. 6611–6620\.Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[26\]H\. Lu, C\. Lu, H\. Wang, J\. Gong, M\. Zhu, and H\. Yang\(2024\)Scenario\-level knowledge transfer for motion planning of autonomous driving via successor representation\.Transp\. Res\. Pt\. C\-Emerg\. Technol\.168,pp\. 104899\.External Links:[Document](https://dx.doi.org/10.1016/j.trc.2024.104899)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.
- \[27\]Z\. Ma, X\. Liu, and Y\. Huang\(2024\)Unsupervised reinforcement learning for multi\-task autonomous driving: expanding skills and cultivating curiosity\.IEEE Trans\. Intell\. Transp\. Syst\.25\(10\),pp\. 14209–14219\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2024.3400224)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p1.1)\.
- \[28\]R\. Moni and B\. Gyires\-Tóth\(2025\)Self\-supervised domain transfer for reinforcement learning\-based autonomous driving agent\.Expert Syst\. Appl\.284,pp\. 127809\.External Links:[Document](https://dx.doi.org/10.1016/j.eswa.2025.127809)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.
- \[29\]E\. Parisotto, J\. L\. Ba, and R\. Salakhutdinov\(2015\)Actor\-mimic: deep multitask and transfer reinforcement learning\.arXiv preprint arXiv:1511\.06342\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1511.06342)Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[30\]P\. Polack, F\. Altché, B\. d’Andréa\-Novel, and A\. de La Fortelle\(2017\)The kinematic bicycle model: a consistent model for planning feasible trajectories for autonomous vehicles?\.InProc\. IEEE Intell\. Veh\. Symp\. \(IV 2017\),Vol\.,pp\. 812–818\.External Links:[Document](https://dx.doi.org/10.1109/IVS.2017.7995816)Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p1.3)\.
- \[31\]D\. Seita, A\. Gopal, Z\. Mandi, and J\. Canny\(2021\)DCUR: data curriculum for teaching via samples with reinforcement learning\.InarXiv preprint arXiv:2109\.07380,Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[32\]H\. Shu, T\. Liu, X\. Mu, and D\. Cao\(2022\)Driving tasks transfer using deep reinforcement learning for decision\-making of autonomous vehicles in unsignalized intersection\.IEEE Trans\. Veh\. Technol\.71\(1\),pp\. 41–52\.External Links:[Document](https://dx.doi.org/10.1109/TVT.2021.3121985)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p1.1)\.
- \[33\]Y\. Song, Y\. Zhou, A\. Sekhari, J\. A\. Bagnell, A\. Krishnamurthy, and W\. Sun\(2023\)Hybrid rl: using both offline and online data can make rl efficient\.InProc\. 11th Int\. Conf\. Learn\. Represent\. \(ICLR\),Cited by:[§II\-C](https://arxiv.org/html/2606.26527#S2.SS3.p1.1)\.
- \[34\]A\. Stooke, J\. Achiam, and P\. Abbeel\(2020\)Responsive safety in reinforcement learning by pid lagrangian methods\.InProc\. 37th Int\. Conf\. Mach\. Learn\.,pp\. 9133–9143\.Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p2.4),[Figure 4](https://arxiv.org/html/2606.26527#S8.F4),[TABLE II](https://arxiv.org/html/2606.26527#S8.T2.3.1.5.1)\.
- \[35\]M\. Treiber, A\. Hennecke, and D\. Helbing\(2000\)Congested traffic states in empirical observations and microscopic simulations\.Phys\. Rev\. E62\(2\),pp\. 1805\.Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p1.3)\.
- \[36\]M\. Vecerik, T\. Hester, J\. Scholz, F\. Wang, O\. Pietquin, B\. Piot, N\. Heess, T\. Rothörl, T\. Lampe, and M\. Riedmiller\(2017\)Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards\.arXiv preprint arXiv:1707\.08817\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1707.08817)Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[37\]Z\. Wang and M\. E\. Taylor\(2017\)Improving reinforcement learning with confidence\-based demonstrations\.InProc\. 26th Int\. Joint Conf\. Artif\. Intell\.,IJCAI’17,pp\. 3027–3033\.External Links:ISBN 9780999241103Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[38\]Z\. Xu, C\. Tang, and M\. Tomizuka\(2018\)Zero\-shot deep reinforcement learning driving policy transfer for autonomous vehicles based on robust control\.InProc\. 21th IEEE Intell\. Transp\. Syst\. Conf\. \(ITSC\),pp\. 2865–2871\.External Links:[Document](https://dx.doi.org/10.1109/ITSC.2018.8569612)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.
- \[39\]Z\. Xue, Z\. Peng, Q\. Li, Z\. Liu, and B\. Zhou\(2023\)Guarded policy optimization with imperfect online demonstrations\.Proc\. 11th Int\. Conf\. Learn\. Represent\. \(ICLR\)\.Cited by:[§VII\-B](https://arxiv.org/html/2606.26527#S7.SS2.p2.4),[Figure 4](https://arxiv.org/html/2606.26527#S8.F4),[TABLE II](https://arxiv.org/html/2606.26527#S8.T2.3.1.6.1),[Lemma 1](https://arxiv.org/html/2606.26527#Thmlemma1.p2.1.1)\.
- \[40\]Q\. Yang, T\. D\. Simão, N\. Jansen, S\. H\. Tindemans, and M\. T\. Spaan\(2023\)Reinforcement learning by guided safe exploration\.InProc\. 26th Eur\. Conf\. Artif\. Intell\.,pp\. 2858–2865\.Cited by:[§III\-B](https://arxiv.org/html/2606.26527#S3.SS2.p1.4)\.
- \[41\]H\. Yin and S\. Pan\(2017\)Knowledge transfer for deep reinforcement learning with hierarchical experience replay\.InProc\. AAAI Conf\. Artif\. Intell\.,Vol\.31\.Cited by:[§II\-A](https://arxiv.org/html/2606.26527#S2.SS1.p1.1)\.
- \[42\]H\. You, R\. Dong, Y\. Chi, and Y\. Zhu\(2022\)Cross\-domain adaptive transfer reinforcement learning based on state\-action correspondence\.InProc\. Conf\. Uncertainty Artif\. Intell\.,Vol\.180,pp\. 1640–1652\.Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.
- \[43\]Q\. Zhang, C\. Wu, H\. Tian, Y\. Gao, W\. Yao, and L\. Wu\(2024\)Safety reinforcement learning control via transfer learning\.Automatica166,pp\. 111714\.External Links:[Document](https://dx.doi.org/10.1016/j.automatica.2024.111714)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.
- \[44\]Y\. Zhang, J\. Liu, C\. Li, Y\. Niu, Y\. Yang, Y\. Liu, and W\. Ouyang\(2024\)A perspective of q\-value estimation on offline\-to\-online reinforcement learning\.InProc\. AAAI Conf\. Artif\. Intell\.,Vol\.38,pp\. 16908–16916\.Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p1.1)\.
- \[45\]R\. Zhou, J\. Huang, M\. Li, H\. Li, H\. Cao, and X\. Song\(2025\)Knowledge transfer from simple to complex: a safe and efficient reinforcement learning framework for autonomous driving decision\-making\.Adv\. Eng\. Inform\.\.External Links:[Document](https://dx.doi.org/10.1016/j.aei.2025.103188)Cited by:[§I](https://arxiv.org/html/2606.26527#S1.p2.1)\.