Mitigating Bus Bunching with Reinforcement Learning Enhanced by Semantic Stop Embedding

# Mitigating Bus Bunching with Reinforcement Learning Enhanced by Semantic Stop Embedding
Source: [https://arxiv.org/html/2608.10207](https://arxiv.org/html/2608.10207)
\\TRBauthor

Xin Dong\*Department of Civil and Environmental Engineering, The Pennsylvania State Universityxjd5036@psu\.edu\[University Park, PA, 16803\]\[0009\-0000\-9420\-6496\]\\TRBauthorVikash V\. GayahDepartment of Civil and Environmental Engineering, The Pennsylvania State Universitygayah@engr\.psu\.edu\[University Park, PA, 16803\]\[0000\-0002\-0648\-3360\]

\\TRBtitlefootnote

\*Corresponding author

\\AuthorHeaders

Dong, Gayah

## 1Abstract

Bus bunching degrades service regularity and increases passenger waiting in high\-frequency transit\. Existing reinforcement\-learning\-based holding controllers primarily rely on instantaneous operational variables or route\-specific stop identifiers, which provide limited information about the functional and operational context of individual stops and constrain policy reuse across routes\. This study introduces an LLM\-assisted semantic stop representation for event\-driven bus holding control\. An LLM is used offline to transform heterogeneous stop information, including physical attributes, surrounding activity context, and historical operational characteristics, into fixed semantic embeddings that are incorporated into a deep Q\-learning controller without requiring real\-time LLM inference\. Experiments are conducted in stochastic simulations calibrated with observed data from two bus routes\. Compared with the best calibrated Daganzo baseline, the semantic controller reduces headway variability, bunching events, and passenger waiting time by 32\.0%, 69\.2%, and 24\.0%, respectively\. A route\-specific stop identifier does not improve the spacing\-only controller, whereas semantic stop information improves headway regularity, waiting time, and holding effort, providing a more favorable overall trade\-off across control objectives\. Cross\-route experiments further show that zero\-shot transfer provides limited immediate generalization, while warm\-start fine\-tuning accelerates early\-stage learning and improves transferred policies; cold\-start training nevertheless achieves the best final performance\. These findings suggest that semantic state representations can complement conventional operational states and support adaptation\-based policy reuse across related transit routes\. Key Words:Reinforcement learning; semantic stop embeddings; large language models; transfer learning; bus holding control\.

## 2Introduction

Bus bunching is a persistent problem in high\-frequency transit\. Small disturbances in travel time, dwell time, passenger arrivals, traffic conditions, signals, or driver behavior can disrupt vehicle spacing and trigger a self\-reinforcing processDaganzoPilachowski2011Cooperation\. A delayed bus encounters more passengers and experiences longer dwell times, while the following bus serves fewer passengers and gradually closes the gapDaganzoPilachowski2011Cooperation\. These dynamics can amplify modest headway deviations into persistent bunching, resulting in longer and less predictable passenger waits, uneven vehicle loads, and increased in\-vehicle delayCatsEtAl2011Holding\. At the system level, bunching also reduces service reliability, effective fleet capacity, and operational productivity\. Bus\-control literature includes schedule\- and headway\-based holding, stop\-skipping, boarding limits, speed control, signal priority, short turning, and combinations of these measures\. Holding remains especially attractive because it can be implemented with existing vehicle\-location and driver\-communication systems\.

Classical bus holding control has largely been developed through analytical and rule\-based formulations that translate observed headway deviations into real\-time holding decisions\.Daganzo2009Headwayderived an adaptive headway\-based rule designed to contain local disturbances without requiring substantial schedule slack, whileDaganzoPilachowski2011Cooperationshowed that two\-way spacing information and bus\-to\-bus cooperation can stabilize service\. Subsequent studies demonstrated that the effectiveness of such strategies depends on the holding criterion and control\-point configurationCatsEtAl2011Holding\. An empirical comparison byBerrebiEtAl2018Comparingfound that schedule\-based methods required relatively little holding but provided limited headway stabilization, whereas prediction\-based methods achieved a better regularity–holding tradeoff but were sensitive to prediction accuracy\. Recent reviews further identify passenger\-oriented objectives, coordinated control, simplified demand and network assumptions, and limited real\-world evaluation as continuing research gapsGkiotsalitisCats2021AtStop;RezazadaEtAl2024Review\. Although feedback and rule\-based strategies provide interpretable and computationally tractable control, their decisions are constrained by predefined functional forms and calibrated parameters, limiting their ability to represent complex operational tradeoffs and adapt to changing system conditions\. A parallel stream of research has formulated bus holding as a real\-time optimization problem in which passenger costs, vehicle operations, and service constraints are represented explicitly\.EberleinEtAl2001Holdingdeveloped a rolling\-horizon formulation using real\-time vehicle information, whileKoehlerEtAl2011IQPintroduced an iterative quadratic procedure for control at multiple holding points\.DelgadoEtAl2012Holdingfurther incorporated vehicle\-capacity constraints, passenger boarding processes, and joint holding and boarding control\. Later work compared deterministic and stochastic predictive\-control formulations under different operating conditionsMunozEtAl2013Dynamic, extended holding optimization to corridors with interacting bus servicesHernandezEtAl2015MultiLine, and incorporated dynamically varying passenger demand and link travel timesSanchezMartinezEtAl2016Dynamic\. These models rely on explicit representations of bus and passenger dynamics and require repeated online optimization as new information becomes available\. Although existing solution procedures can support real\-time implementation, their computational requirements may increase substantially with the prediction horizon and system complexity\.

Reinforcement\-learning\-based control shifts the computationally intensive process of policy optimization to pre\-deployment trainingliu2021privacy;hou2026mobility\. Once trained, the RL\-based controller can generate state\-dependent holding actions during operation through a forward pass of the policy network, without repeatedly solving an online optimization problem\. Early studies introduced deep value\-based learning for discrete holding decisionsAlesianiGkiotsalitis2018RLand combined approximate dynamic programming with Q\-learning to evaluate downstream effects over multiple stagesHeEtAl2022Qlearning\. Later work formulated fleet\-wide holding as an event\-driven decision processZhangEtAl2026SingleAgentand developed coordinated multi\-agent controllers for asynchronous decisions and inter\-vehicle interactionsChenEtAl2016MARL\. Subsequent extensions considered joint holding and stop\-skippingRodriguezEtAl2023Cooperative, robust and multi\-objective controlWangSun2023Robust;WangSun2023MultiObjective, hierarchical action selectionYuEtAl2024Hierarchical, and graph\-based coordination in congested multi\-line corridorsLiEtAl2026GraphAware\. Despite these advances, existing state representations largely rely on operational variables, such as headways, vehicle loads and passenger occupancywood2023development, passenger flows, previous actions, and categorical vehicle or stop identifiers\. These features describe the current fleet state, but categorical identifiers provide little information about the functional and land\-use context of individual stops or the relationships between stops across routes\. Recent evidence shows that state and action design can matter as much as the selected RL algorithm\. In particular, a systematic benchmark found that forward and backward spacing alone can outperform a richer state containing a stop identifier, while discrete holding actions can perform comparably to continuous actionsXuEtAl2025Systematic\. This result does not imply that stops are operationally interchangeable; rather, it suggests that an arbitrary identifier is a weak representation of the contextual factors that distinguish them\. This representation gap suggests a potential role for LLMs, whose application in bus holding control remains limited\. Existing bus control work has primarily used LLMs to generate and refine reward functionsYuEtAl2025LLMReward, rather than to construct semantic state representations\. Whether LLM\-derived stop semantics can improve within\-route control and support cross\-route policy adaptation therefore remains insufficiently explored\.

To summarize, three key gaps remain in existing RL\-based bus holding and LLM\-enhanced control studies: \(1\) conventional state representations provide limited information about stop\-level functional context; \(2\) route\-specific representations constrain policy reuse across routes; and \(3\) existing LLM applications mainly focus on reward design rather than semantic state representation\. To address these gaps, this study introduces an LLM\-assisted semantic state representation for RL\-based bus holding control\. Rather than treating stops as arbitrary route\-specific identifiers, the proposed approach represents their functional and operational characteristics in a shared feature space that can be used across routes\. The LLM is employed offline to construct the stop representations, while online holding decisions remain governed by the learned RL policy, avoiding real\-time LLM inference\. The study investigates whether semantic stop information can improve holding control within a route and facilitate policy reuse across routes\. In particular, it examines both the immediate generalization of a pretrained policy and its ability to adapt to a new operating environment through further training\. These questions are evaluated in stochastic, data\-calibrated bus operation environments through comparisons with classical control strategies and alternative RL state representations\.

The main contributions of this study are as follows:

- •We introduce an LLM\-assisted semantic representation of bus stops for reinforcement learning\-based holding control\. The representation combines functional and access\-related labels derived from geospatial context with discretized historical operational characteristics, and encodes them in a common continuous feature space across routes\. The stop representations are constructed offline and reused throughout RL training and deployment, avoiding repeated and computationally costly semantic feature construction\.
- •We use controlled within\-route comparisons to isolate the contribution of semantic stop information from that of route\-specific stop identity and directly observed passenger\-flow variables\. The analysis demonstrates the added value of semantic representations for selected control settings and clarifies their role beyond conventional spacing\-based state designs\.
- •We evaluate cross\-route policy reuse through zero\-shot transfer, target\-route fine\-tuning, and cold\-start training\. The findings reveal that target\-route adaptation substantially improves transferred policies and early\-stage learning, highlighting the distinction between immediate generalization and adaptation\-based reuse\.

The remainder of this paper is organized as follows\. Section[3](https://arxiv.org/html/2608.10207#S3)presents the proposed methodology, including the event\-driven MDP formulation, semantic stop embedding construction, and semantic\-enhanced RL holding policy\. Section[4](https://arxiv.org/html/2608.10207#S4)describes the experimental design\. Section[5](https://arxiv.org/html/2608.10207#S5)reports and discusses the same\-route and cross\-route experimental results\. Finally, Section[6](https://arxiv.org/html/2608.10207#S6)concludes the paper and discusses its limitations and directions for future research\.

## 3Methodology

This section presents a semantic\-enhanced reinforcement learning framework for bus holding control\. At each bus\-departure event, a shared DQN selects a discrete holding duration based on a local observation that combines the bus’s spatial relationship with its neighboring buses and a fixed semantic embedding of the current stop\. The framework consists of three components: an event\-driven Markov decision process formulation, an offline semantic stop embedding procedure, and a DQN\-based policy\-learning method\.

### 3\.1Event\-Driven MDP Formulation

Consider a bus route represented as a directed loop𝒢=\(𝒮,ℰ\)\\mathcal\{G\}=\(\\mathcal\{S\},\\mathcal\{E\}\), where𝒮=\{1,…,N\}\\mathcal\{S\}=\\\{1,\\ldots,N\\\}is the set of stops andℰ=\{\(i,i\+1\):i=1,…,N\}\\mathcal\{E\}=\\\{\(i,i\+1\):i=1,\\ldots,N\\\}is the set of consecutive links, with stopN\+1N\+1identified as stop11\. Letℓi\\ell\_\{i\}denote the length of linkiiandL=∑i=1NℓiL=\\sum\_\{i=1\}^\{N\}\\ell\_\{i\}denote the total loop length\. A fleetℬ=\{1,…,B\}\\mathcal\{B\}=\\\{1,\\ldots,B\\\}operates continuously along the route\. The complete system state at timett, denoted byX​\(t\)∈𝒳X\(t\)\\in\\mathcal\{X\}, contains the information required to characterize the bus operation process, including bus positions, onboard passenger loads by destination, stop\-level passenger queues, and the service, holding, or travel status of each bus\. System transitions are governed by stochastic passenger arrivals, passenger service, link travel times, and holding actions\. Holding decisions are made asynchronously at bus\-departure events\. Letτnb\\tau\_\{n\}^\{b\}denote thennth decision epoch of busbb, defined as the time at which the bus completes passenger service at stopinbi\_\{n\}^\{b\}and is ready to enter the downstream link\. Atτnb\\tau\_\{n\}^\{b\}, the controller receives a local observation𝐨nb=Ω​\(X​\(τnb\)\)\\mathbf\{o\}\_\{n\}^\{b\}=\\Omega\(X\(\\tau\_\{n\}^\{b\}\)\), selects an actionanba\_\{n\}^\{b\}, and receives the corresponding reward when the same bus reaches its next decision epochτn\+1b\\tau\_\{n\+1\}^\{b\}\. The problem is formulated as an event\-driven, partially observed Markov decision process:

ℳ=\(𝒳,𝒪,𝒜,P,R,γ,Ω\)\.\\mathcal\{M\}=\(\\mathcal\{X\},\\mathcal\{O\},\\mathcal\{A\},P,R,\\gamma,\\Omega\)\.\(1\)
Here,𝒳\\mathcal\{X\}and𝒪\\mathcal\{O\}denote the full\-state and observation spaces, respectively;𝒜\\mathcal\{A\}is the holding\-action space;PPdenotes the transition dynamics between consecutive decision events;RRis the reward function;γ\\gammais the discount factor; andΩ\\Omegamaps the full system state to the local controller observation\. Although the elapsed time between consecutive decision epochs is variable, each bus\-departure event is treated as one decision step by the RL controller\.

#### 3\.1\.1State

The dynamic component of the observation describes the spatial relationship between the controlled bus and its nearest neighboring buses\. Letxb​\(τnb\)∈\[0,L\)x\_\{b\}\(\\tau\_\{n\}^\{b\}\)\\in\[0,L\)denote the longitudinal position of busbbalong the loop at decision epochτnb\\tau\_\{n\}^\{b\}\. The normalized forward and backward spatial spacings are defined as:

g¯n\+,b\\displaystyle\\bar\{g\}\_\{n\}^\{\+,b\}=1L​minb′∈ℬ∖\{b\}⁡\[\(xb′​\(τnb\)−xb​\(τnb\)\)modL\],\\displaystyle=\\frac\{1\}\{L\}\\min\_\{b^\{\\prime\}\\in\\mathcal\{B\}\\setminus\\\{b\\\}\}\\left\[\\bigl\(x\_\{b^\{\\prime\}\}\(\\tau\_\{n\}^\{b\}\)\-x\_\{b\}\(\\tau\_\{n\}^\{b\}\)\\bigr\)\\bmod L\\right\],\(2\)g¯n−,b\\displaystyle\\bar\{g\}\_\{n\}^\{\-,b\}=1L​minb′∈ℬ∖\{b\}⁡\[\(xb​\(τnb\)−xb′​\(τnb\)\)modL\]\.\\displaystyle=\\frac\{1\}\{L\}\\min\_\{b^\{\\prime\}\\in\\mathcal\{B\}\\setminus\\\{b\\\}\}\\left\[\\bigl\(x\_\{b\}\(\\tau\_\{n\}^\{b\}\)\-x\_\{b^\{\\prime\}\}\(\\tau\_\{n\}^\{b\}\)\\bigr\)\\bmod L\\right\]\.\(3\)
These quantities measure the route distances from the controlled bus to the nearest bus ahead and behind, respectively\. Equal forward and backward spacings indicate that the controlled bus is locally centered between its neighboring buses\.

Let𝐳inb∈ℝd\\mathbf\{z\}\_\{i\_\{n\}^\{b\}\}\\in\\mathbb\{R\}^\{d\}denote the fixed semantic embedding of the stop at which the decision is made\. The proposed semantic\-enhanced observation is

𝐨n,semb=\[g¯n\+,b,g¯n−,b,𝐳inb\]∈ℝ2\+d\.\\mathbf\{o\}\_\{n,\\mathrm\{sem\}\}^\{b\}=\\left\[\\bar\{g\}\_\{n\}^\{\+,b\},\\bar\{g\}\_\{n\}^\{\-,b\},\\mathbf\{z\}\_\{i\_\{n\}^\{b\}\}\\right\]\\in\\mathbb\{R\}^\{2\+d\}\.\(4\)
The semantic vector provides contextual information that cannot be inferred directly from instantaneous bus spacing, including the physical, functional, and operational characteristics of the current stop\. Its construction is described in Section[3\.2](https://arxiv.org/html/2608.10207#S3.SS2)\.

Two alternative observation designs are used as benchmarks: the spacing\-only observation excludes stop\-level contextual information; and the stop\-ID observation supplements the spacing variables with the categorical indexinbi\_\{n\}^\{b\}of the current stop:

𝐨n,semb=\[g¯n\+,b,g¯n−,b,𝐢nb\]∈ℝ3\.\\mathbf\{o\}\_\{n,\\mathrm\{sem\}\}^\{b\}=\\left\[\\bar\{g\}\_\{n\}^\{\+,b\},\\bar\{g\}\_\{n\}^\{\-,b\},\\mathbf\{i\}\_\{n\}^\{b\}\\right\]\\in\\mathbb\{R\}^\{3\}\.\(5\)

#### 3\.1\.2Action

At decision points, the controller selects an action index from𝒜=\{0,…,A−1\}\\mathcal\{A\}=\\\{0,\\ldots,A\-1\\\}\. LetHmaxH\_\{\\max\}denote the maximum allowable holding time\. The holding duration associated with actiona∈𝒜a\\in\\mathcal\{A\}is

H​\(a\)=aA−1​Hmax\.H\(a\)=\\frac\{a\}\{A\-1\}H\_\{\\max\}\.\(6\)

#### 3\.1\.3Reward

The reward for actionanba\_\{n\}^\{b\}is assigned when busbbreaches its next decision epochτn\+1b\\tau\_\{n\+1\}^\{b\}\. It combines a spacing\-equalization term, a penalty for newly detected bunching conditions, and a regularization term favoring shorter holding durations\.

LetC​\(t\)C\(t\)denote the cumulative number of bunching conditions detected by timett\. The three reward components are defined as

Rn\+1eq,b\\displaystyle R\_\{n\+1\}^\{\\mathrm\{eq\},b\}=exp⁡\(−\|g¯n\+1\+,b−g¯n\+1−,b\|\),\\displaystyle=\\exp\\\!\\left\(\-\\left\|\\bar\{g\}\_\{n\+1\}^\{\+,b\}\-\\bar\{g\}\_\{n\+1\}^\{\-,b\}\\right\|\\right\),\(7\)Δ​Cn\+1b\\displaystyle\\Delta C\_\{n\+1\}^\{b\}=max⁡\{0,C​\(τn\+1b\)−C​\(τnb\)\},\\displaystyle=\\max\\\!\\left\\\{0,C\(\\tau\_\{n\+1\}^\{b\}\)\-C\(\\tau\_\{n\}^\{b\}\)\\right\\\},\(8\)Rnhold,b\\displaystyle R\_\{n\}^\{\\mathrm\{hold\},b\}=exp⁡\(−H​\(anb\)Hmax\)\.\\displaystyle=\\exp\\\!\\left\(\-\\frac\{H\(a\_\{n\}^\{b\}\)\}\{H\_\{\\max\}\}\\right\)\.\(9\)
The final reward is

rn\+1b=Rn\+1eq,b−κ​Δ​Cn\+1b\+ω​Rnhold,b,r\_\{n\+1\}^\{b\}=R\_\{n\+1\}^\{\\mathrm\{eq\},b\}\-\\kappa\\Delta C\_\{n\+1\}^\{b\}\+\\omega R\_\{n\}^\{\\mathrm\{hold\},b\},\(10\)whereκ\>0\\kappa\>0andω\>0\\omega\>0determine the relative weights assigned to bunching prevention and holding reduction\. Their values are reported in Section[4\.4](https://arxiv.org/html/2608.10207#S4.SS4)\.

### 3\.2Semantic Stop Embedding

To augment the dynamic operating state with stop\-level contextual information, we construct a fixed semantic embedding𝐳i\\mathbf\{z\}\_\{i\}for each stopii\. The embedding is generated offline, cached before RL training, and remains unchanged during policy learning and evaluation\. Its construction consists of two stages: constrained semantic annotation of heterogeneous stop information and projection of the resulting representation into a shared low\-dimensional feature space\.

#### 3\.2\.1Fact Assembly and Semantic Annotation

Let𝝃i\\boldsymbol\{\\xi\}\_\{i\}denote the structured fact record associated with stopii\. The record contains three categories of information: physical stop attributes, surrounding activity context, and operational descriptors\. Physical attributes characterize the stop facility and its local spatial configuration\. Surrounding activity context describes nearby land\-use functions and their approximate accessibility, while operational descriptors summarize recurrent passenger\-demand and service characteristics\.

A deterministic prompt\-construction functionΠ​\(⋅\)\\Pi\(\\cdot\)converts the structured record into a constrained prompt\. A large language modelGψG\_\{\\psi\}maps the supplied facts to structured semantic metadata, after which a deterministic normalization function𝒩​\(⋅\)\\mathcal\{N\}\(\\cdot\)enforces the predefined schema and controlled vocabularies:

𝐲i=𝒩​\(Gψ​\(Π​\(𝝃i\)\)\)=\(si,𝒯irole,ri,𝒯idemand,𝒯iquality\)\.\\mathbf\{y\}\_\{i\}=\\mathcal\{N\}\\\!\\left\(G\_\{\\psi\}\\\!\\left\(\\Pi\(\\boldsymbol\{\\xi\}\_\{i\}\)\\right\)\\right\)=\\left\(s\_\{i\},\\mathcal\{T\}\_\{i\}^\{\\mathrm\{role\}\},r\_\{i\},\\mathcal\{T\}\_\{i\}^\{\\mathrm\{demand\}\},\\mathcal\{T\}\_\{i\}^\{\\mathrm\{quality\}\}\\right\)\.\(11\)
Here,sis\_\{i\}is a short natural\-language summary,𝒯irole\\mathcal\{T\}\_\{i\}^\{\\mathrm\{role\}\}is a set of functional role tags,rir\_\{i\}is a route\-context category,𝒯idemand\\mathcal\{T\}\_\{i\}^\{\\mathrm\{demand\}\}contains demand\-related tags, and𝒯iquality\\mathcal\{T\}\_\{i\}^\{\\mathrm\{quality\}\}contains data\-quality indicators\.

The free\-text summarysis\_\{i\}is retained for inspection but is not included in the embedding input\. Instead, a leakage\-controlled subset of the original attributes, denoted by𝝃¯i\\bar\{\\boldsymbol\{\\xi\}\}\_\{i\}, is combined with the normalized semantic labels through a deterministic key–value serialization:

ti=S​\(𝝃¯i,𝒯irole,ri,𝒯idemand,𝒯iquality\)\.t\_\{i\}=S\\\!\\left\(\\bar\{\\boldsymbol\{\\xi\}\}\_\{i\},\\mathcal\{T\}\_\{i\}^\{\\mathrm\{role\}\},r\_\{i\},\\mathcal\{T\}\_\{i\}^\{\\mathrm\{demand\}\},\\mathcal\{T\}\_\{i\}^\{\\mathrm\{quality\}\}\\right\)\.\(12\)
To reduce route\-specific leakage, the serialized embedding input excludes explicit route names and raw stop\-sequence indices\. Continuous or highly variable attributes are represented using predefined categorical bins rather than raw values\. Exact geographic coordinates are used only to retrieve nearby contextual information and are not directly embedded\. The retained fields are selected to describe stop characteristics that can be compared across routes, including facility type, nearby activity categories, accessibility, demand patterns, service variability, and observation confidence\.

#### 3\.2\.2Embedding Construction and Alignment

The canonical text representationtit\_\{i\}is mapped to a high\-dimensional vector using a pretrained text\-embedding modelEϕE\_\{\\phi\}:

𝐞i=Eϕ​\(ti\)∈ℝD\.\\mathbf\{e\}\_\{i\}=E\_\{\\phi\}\(t\_\{i\}\)\\in\\mathbb\{R\}^\{D\}\.\(13\)
Because the raw text embedding is substantially larger than the dynamic operational state, it is standardized and projected into a compact feature space before being supplied to the RL controller\. Let𝒰\\mathcal\{U\}denote the set of stops used to fit the representation transformation, and let𝝁e\\boldsymbol\{\\mu\}\_\{e\}and𝝈e\\boldsymbol\{\\sigma\}\_\{e\}denote the featurewise mean and standard deviation of their raw embeddings\. The standardized embedding is

𝐞~i=\(𝐞i−𝝁e\)⊘max⁡\(𝝈e,ε\),\\widetilde\{\\mathbf\{e\}\}\_\{i\}=\(\\mathbf\{e\}\_\{i\}\-\\boldsymbol\{\\mu\}\_\{e\}\)\\oslash\\max\(\\boldsymbol\{\\sigma\}\_\{e\},\\varepsilon\),\(14\)
where⊘\\oslashdenotes elementwise division andε\>0\\varepsilon\>0ensures numerical stability\.

Let𝐕d∈ℝD×d\\mathbf\{V\}\_\{d\}\\in\\mathbb\{R\}^\{D\\times d\}contain the leadingddprincipal\-component loading vectors fitted from the standardized embeddings\. The reduced representation is

𝐩i=𝐕d𝖳​𝐞~i∈ℝd\.\\mathbf\{p\}\_\{i\}=\\mathbf\{V\}\_\{d\}^\{\\mathsf\{T\}\}\\widetilde\{\\mathbf\{e\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\}\.\(15\)
The PCA coordinates are standardized again and clipped to limit extreme feature values:

𝐳i=clip\[−c,c\]⁡\(\(𝐩i−𝝁p\)⊘max⁡\(𝝈p,ε\)\)∈ℝd,\\mathbf\{z\}\_\{i\}=\\operatorname\{clip\}\_\{\[\-c,c\]\}\\\!\\left\(\(\\mathbf\{p\}\_\{i\}\-\\boldsymbol\{\\mu\}\_\{p\}\)\\oslash\\max\(\\boldsymbol\{\\sigma\}\_\{p\},\\varepsilon\)\\right\)\\in\\mathbb\{R\}^\{d\},\(16\)
where𝝁p\\boldsymbol\{\\mu\}\_\{p\}and𝝈p\\boldsymbol\{\\sigma\}\_\{p\}are the featurewise mean and standard deviation of the PCA coordinates\. The proposed controller usesd=8d=8andc=3c=3\.

The semantic representations are constructed in a shared feature space so that stops from different routes can be compared using the same coordinate system\. This design allows a policy trained on one route to receive semantically consistent stop descriptors when applied to another route\. Once constructed, the stop vectors are fixed throughout training and evaluation and are used as part of the local observation at each holding decision\.

### 3\.3Semantic\-Enhanced RL Holding Policy

The semantic\-enhanced observation defined in Equation \([4](https://arxiv.org/html/2608.10207#S3.E4)\) is mapped to holding actions using a deep Q\-learning policy shared by all buses\. Parameter sharing allows decision experiences collected from different buses and stops to update a common action\-value functionQθ​\(𝐨,a\)Q\_\{\\theta\}\(\\mathbf\{o\},a\)\. The Q\-networkFθF\_\{\\theta\}maps an observation to the action values associated with theAAcandidate holding durations:

𝐪θ​\(𝐨\)=Fθ​\(𝐨\)∈ℝA,Qθ​\(𝐨,a\)=\[𝐪θ​\(𝐨\)\]a\.\\mathbf\{q\}\_\{\\theta\}\(\\mathbf\{o\}\)=F\_\{\\theta\}\(\\mathbf\{o\}\)\\in\\mathbb\{R\}^\{A\},\\qquad Q\_\{\\theta\}\(\\mathbf\{o\},a\)=\\left\[\\mathbf\{q\}\_\{\\theta\}\(\\mathbf\{o\}\)\\right\]\_\{a\}\.\(17\)
For the proposed policy, the semantic stop vector𝐳i\\mathbf\{z\}\_\{i\}is concatenated directly with the dynamic spacing variables before being passed to the network\. For the stop\-ID benchmark, the categorical stop index is first mapped to a learned embedding\. All observation designs otherwise use the same policy\-learning procedure\. Each transition connects two consecutive decision epochs of the same bus and is stored in the replay buffer as

em=\(𝐨m,am,rm,𝐨m′\)\.e\_\{m\}=\\left\(\\mathbf\{o\}\_\{m\},a\_\{m\},r\_\{m\},\\mathbf\{o\}^\{\\prime\}\_\{m\}\\right\)\.\(18\)
For transitionmm, the temporal\-difference target is computed as

ym=rm\+γ​sg⁡\[maxa′∈𝒜⁡Qθ​\(𝐨m′,a′\)\],y\_\{m\}=r\_\{m\}\+\\gamma\\,\\operatorname\{sg\}\\\!\\left\[\\max\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}Q\_\{\\theta\}\(\\mathbf\{o\}^\{\\prime\}\_\{m\},a^\{\\prime\}\)\\right\],\(19\)
wheresg⁡\[⋅\]\\operatorname\{sg\}\[\\cdot\]denotes the stop\-gradient operation\. The same online Q\-network is used to evaluate both the current and next observations, while the bootstrap target is treated as constant during backpropagation\.

The network parameters are updated by minimizing the mean squared temporal\-difference error over a minibatchℬm\\mathcal\{B\}\_\{m\}:

ℒ​\(θ\)=1\|ℬm\|​∑m∈ℬm\(Qθ​\(𝐨m,am\)−ym\)2\.\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{B\}\_\{m\}\|\}\\sum\_\{m\\in\\mathcal\{B\}\_\{m\}\}\\left\(Q\_\{\\theta\}\(\\mathbf\{o\}\_\{m\},a\_\{m\}\)\-y\_\{m\}\\right\)^\{2\}\.\(20\)
During training, actions are selected using anϵ\\epsilon\-greedy policy\. During evaluation, the controller selects the action with the highest estimated value:

anb=arg⁡maxa∈𝒜⁡Qθ​\(𝐨nb,a\)\.a\_\{n\}^\{b\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\}Q\_\{\\theta\}\(\\mathbf\{o\}\_\{n\}^\{b\},a\)\.\(21\)

## 4Experimental Design

This section describes the simulation environment, data preparation, control models, and evaluation settings used to investigate two research questions: \(1\) whether incorporating semantic stop information improves the same\-route performance of an RL\-based holding policy relative to benchmark control models, and \(2\) whether a policy learned on a source route can be effectively transferred to a target route\. And it is examined through a cross\-route experiment involving three policy training and deployment settings: zero\-shot transfer without target\-route training, warm\-start fine\-tuning initialized from the source\-route checkpoint, and cold\-start training from random initialization\. The experiments use two Pennsylvania State University campus shuttle routes operating in State College, Pennsylvania, USA: the White Loop \(WL\) and Blue Loop \(BL\)\. In the cross\-route experiment, WL serves as the source route and BL as the target route\. Figure[1](https://arxiv.org/html/2608.10207#S4.F1)shows the stop locations and directions of travel for both routes\.

![Refer to caption](https://arxiv.org/html/2608.10207v1/figures/route_maps_wl_bl.png)Figure 1:Route maps of the White Loop \(WL\) and Blue Loop \(BL\)\. Stop labels indicate stop names, and arrows indicate the direction of travel\.### 4\.1Simulation Environment

Each simulation episode represents two hours of bus operation at a one\-second time resolution, with buses initialized evenly along the route at the start of each episode\. Passenger boarding and alighting occur at constant per\-second service rates, and each stop enforces a fixed number of loading berths under a first\-in\-first\-out, no\-overtaking queue discipline\. Stochastic passenger demand and link travel times are generated from route\-specific parameters estimated from the historical data described in Section[4\.2](https://arxiv.org/html/2608.10207#S4.SS2)\. When holding control is enabled, the controller is activated once a bus completes passenger service at a stop and selects a holding duration from a discrete action set before departing for the downstream link\. Table[1](https://arxiv.org/html/2608.10207#S4.T1)lists the exact simulation and control parameter values used throughout the experiments\.

Table 1:Simulator and control settings\.
### 4\.2Data

Two categories of data are used in the experiment: \(1\) transit operational data used to construct and calibrate the simulation environment, and \(2\) stop\-level contextual data used to construct the semantic representations introduced in Section[3\.2](https://arxiv.org/html/2608.10207#S3.SS2)\.

Transit operational data\.The simulator inputs are derived from historical CATA automatic vehicle location \(AVL\) records collected from August 15 to December 15, 2017, retaining daytime observations \(8:00–20:00\) after removing records with missing or invalid operational information\. These records are used to estimate stop\-specific Poisson passenger\-arrival rates, gravity\-model alighting probabilities incorporating walking\-distance impedance, and link\-specific lognormal travel\-time distributions fitted from consecutive vehicle observations\. Table[2](https://arxiv.org/html/2608.10207#S4.T2)summarizes the resulting route\-level inputs: compared with WL, BL has more stops, a longer route, and greater link travel\-time variability, while WL has a slightly higher total arrival rate — differences that provide distinct operating conditions for same\-route control and cross\-route transfer evaluation\.

Table 2:Route\-specific simulator inputs estimated from the AVL records\.Semantic stop data\.The semantic stop data characterize the physical, functional, and operational context of each stop on the WL \(13 stops\) and BL \(15 stops\) routes: stop name, curbside type, route\-context attributes, distances to adjacent stops, nearby points of interest \(POIs\), and AVL\-derived demand descriptors \(mean arrival rate, boarding variability, mean headway, and sample size\)\. Nearby POIs are retrieved from OpenStreetMap via the Overpass API, mapped to a predefined campus land\-use vocabulary \(academic buildings, residence halls, libraries, student centers, athletic facilities, event venues, administrative facilities, parking areas, landmarks\), with walking times estimated at a pedestrian speed of 1\.35 m/s\. For instance, the WL stop*Pattee TC WB*is a transit\-center stop serving three routes \(WL, WE, RL\) at sequence position 6 of 13\. It is located approximately one minute on foot from Pattee Library \(library\), three minutes from the West Halls residence complex \(residence hall\), and six to eight minutes from Old Main \(administrative building\), the Willard Building \(academic building\), and the HUB\-Robeson Center \(student center\)\. The stop also exhibits relatively high and variable passenger demand, with a mean arrival rate of 0\.69 pax/min, a boarding coefficient of variation of 0\.93, and 6,871 valid observations\. These collected facts serve as inputs to the semantic representation procedure described in Section[3\.2](https://arxiv.org/html/2608.10207#S3.SS2)\.

### 4\.3Benchmark Control Models

The proposed*Spacing \+ semantic*RL policy combines normalized spacing features with the semantic stop representation described in Section[3\.2](https://arxiv.org/html/2608.10207#S3.SS2)\. It is compared with four benchmark control models\. The first is the*no\-control*baseline, which applies no holding intervention and allows buses to depart immediately after completing passenger service\. The second is a calibrated rule\-based implementation of the headway\-based holding model proposed by DaganzoDaganzo2009Headway, with the control gainα\\alphaand slack fractionffselected through grid search and the maximum holding time fixed at 60 s\. The other two benchmarks are RL\-based models that share the same network architecture, action space, and training procedure as the proposed policy but use different state representations\. The*Spacing*model uses only normalized forward and backward spatial spacings, whereas the*Spacing \+ stop ID*model augments these spacing features with a learned categorical stop embedding\.

### 4\.4Model Training and Evaluation

The no\-control and rule\-based benchmarks are evaluated directly, whereas each RL controller is trained for 200 episodes, validated every 10 episodes, and selected by the checkpoint with the lowest mean validation headway variation \(ties broken by total bunching count\)\. Final evaluation uses independent seeds and deterministic greedy actions\. All methods are compared on four metrics: headway variation \(service regularity\), total bunching count, average holding time \(control intervention\), and average passenger waiting time\. Lower values are preferred throughout, though holding time is treated as an operational cost to be weighed against gains in the other three metrics\. Space\-time diagrams are used only for visual and diagnostic analysis\.

Performance metrics\.Four performance metrics are reported for each evaluation episode\.Bunching countis defined as the total number of stop\-level bunching detections over the episode\. A bunching detection occurs when at least two buses are simultaneously present at the same stop\.Headway variationis the standard deviation of the actual departure headways observed across bus departure events\.Average waiting timeis the cumulative passenger waiting time in all stop queues divided by the total number of passenger arrivals\.Average holding timeis the mean selected holding duration across all holding decisions, including zero\-second holding actions\.

## 5Results

This section evaluates within\-route control performance, cross\-route policy reuse, and the behavioral patterns induced by the learned policies\.

### 5\.1Within\-Route Performance

Table[3](https://arxiv.org/html/2608.10207#S5.T3)compares the control methods using the four previously introduced performance metrics\. First, all three RL controllers achieve lower mean headway variation, fewer bunching events, and shorter passenger waiting times than both benchmark methods—the Daganzo rule and the no\-control baseline\. These findings suggest that the learned policies adapt more effectively to the stochastic and state\-dependent operating conditions represented in the simulation\. Second, the comparison among the RL state representations highlights the value of semantic stop information\. Adding a categorical stop identifier does not improve the spacing\-only policy in terms of headway variation, bunching count, or passenger waiting time\. This finding is consistent with prior evidence that arbitrary stop identifiers provide limited additional information beyond compact spacing\-based statesXuEtAl2025Systematic\. In contrast, augmenting the state with semantic stop information improves performance across multiple dimensions\. Third, the results reflect the multi\-objective nature of the control problem, which requires balancing service regularity, passenger delay, and control effort\. Among the RL controllers, our semantic policy achieves the lowest mean headway variation, passenger waiting time, and holding time, whereas the spacing\-only policy produces marginally fewer bunching events at the cost of substantially longer holding\. Overall, the semantic policy provides the most favorable balance across the evaluated metrics\.

Table 3:Within\-Route Performance
### 5\.2Cross\-Route Performance

Overall, these results support the feasibility of semantic\-enhanced RL policy transfer across routes\. Figure[2](https://arxiv.org/html/2608.10207#S5.F2)first examines the training behavior of the semantic\-enhanced RL policy by comparing cold\-start training with warm\-start fine\-tuning\. The training curves show that the warm\-start policy begins from a better initial performance level, with higher total reward and headway reward, as well as lower headway variation and passenger waiting time during the early training episodes\. It also reaches a stable performance range more quickly\. These trajectories suggest that the policy learned on the source route contains transferable information that can be reused on the target route, thereby reducing the amount of target\-route training needed to obtain a reasonable control policy\. Table[4](https://arxiv.org/html/2608.10207#S5.T4)reports the final target\-route control performance evaluated over several random seeds\. Cold\-start training achieves the lowest headway variation, bunching count, and passenger waiting time among the RL variants\. This result is expected because the policy is trained entirely on the target route and can therefore specialize to route\-specific dynamics\. However, this performance is achieved with greater holding effort than the transferred policies\. In contrast, zero\-shot transfer improves bunching count and passenger waiting time relative to the no\-control baseline, but provides little improvement in headway variation, indicating that direct policy reuse is feasible but insufficient without target\-route adaptation\. After target\-route adaptation \(warm\-start with fine\-tuning\), the transferred policy further improves over zero\-shot transfer and reaches a more balanced performance level\. Notably, it requires the lowest holding time among the active controllers, suggesting that warm\-start fine\-tuning can adapt the source\-route policy to the target route while maintaining a lower control effort\. Together with the faster early\-stage learning shown in Figure[2](https://arxiv.org/html/2608.10207#S5.F2), these results indicate that source\-route knowledge can reduce target\-route training effort and support effective cross\-route policy adaptation\.

![Refer to caption](https://arxiv.org/html/2608.10207v1/figures/bl_warmstart_training_curves.png)Figure 2:Cold\-start and warm\-start training trajectories on the target route\. Panels show total reward, headway reward, headway variation, passenger waiting time, holding time, and passenger riding time\. Thin lines denote seed\-level episode means, and shaded bands indicate±1\\pm 1standard deviation across five seeds\.Table 4:Transfer Performance for the Spacing\+Semantic RL Policy
### 5\.3Bunching Patterns and Holding Actions

Figure[3](https://arxiv.org/html/2608.10207#S5.F3)complements the aggregate metrics by showing representative simulated space–time diagrams for both routes\. For each route, all control methods are evaluated under the same random seed and route configuration; therefore, the diagrams are intended to illustrate temporal control behavior rather than provide a statistical ranking\. Red crosses indicate the stop–time locations where bunching events are detected, showing not only how many bunching events occur but also where they emerge and propagate along the route\. In the no\-control cases, bunching events are widely distributed across both routes and often appear repeatedly over consecutive stops, suggesting that once headway disturbances form, they can persist and propagate downstream without intervention\. These locations can be interpreted as less resilient portions of the route in this realization, where the service has limited ability to recover naturally from headway deviations\. With active control, the number of bunching events decreases and the remaining events become more localized, indicating that holding actions can interrupt the propagation of bunching and improve route\-level resilience\. In the WL realization, the bunching count decreases from 57 under no control to 30 under the Daganzo rule, 22 under spacing\-only RL, and 1 under spacing\+semantic RL\. The corresponding counts on BL are 57, 6, 3, and 2\. The increasingly clear separation between vehicle trajectories further illustrates how active holding can maintain more regular service over time\.

![Refer to caption](https://arxiv.org/html/2608.10207v1/figures/spacetime_grid_comparison.png)Figure 3:Representative simulated space\-time trajectories for WL and BL under no control, best\-tuned Daganzo rule, spacing\-only RL, and semantic\-enhanced RL\. Colored lines denote vehicle trajectories, gray and pink horizontal segments denote normal dwell and holding, respectively, and red crosses denote detected bunching events\. All panels use the same evaluation seed for illustration\.Figure[4](https://arxiv.org/html/2608.10207#S5.F4)compares the stop\-level hold probabilities of the spacing\-only RL policy and the semantic\-enhanced RL policy on the WL and BL routes\. The nonuniform profiles show that both policies learn stop\-specific control patterns rather than applying holding uniformly along the route\. Adding semantic stop information further changes where holding is applied: the unweighted stop\-level average hold probability decreases from 0\.612 to 0\.508 on WL and from 0\.582 to 0\.561 on BL, while the locations with increased or reduced holding also shift\. For example, the semantic\-enhanced RL policy substantially reduces holding at Westgate on WL and University Club on BL, while retaining or increasing holding at Lot 83 W on WL and Lot 83 E on BL\. These changes suggest that semantic stop information affects the spatial allocation of control effort and may help the policy identify more suitable locations for regulating downstream service\. Several of these stops correspond to recognizable campus activity or access points, making their role as potential control locations plausible\.

![Refer to caption](https://arxiv.org/html/2608.10207v1/figures/holdprob_grid_comparison.png)Figure 4:Stop\-level probability of selecting a positive holding action for \(a\) WL spacing\-only RL, \(b\) WL spacing\+semantic RL, \(c\) BL spacing\-only RL, and \(d\) BL spacing\+semantic RL\. Bars show the mean of the per\-seed positive\-hold probability across ten evaluation seeds, and error bars indicate±1\\pm 1standard deviation\. A positive holding action has a selected holding time greater than zero\.

## 6Conclusion

This paper develops and evaluates a language\-model\-assisted semantic stop representation for reinforcement\-learning\-based bus holding control\. The representation complements dynamic operational states by capturing the functional context and recurring operational characteristics that differentiate individual stops\. It is constructed from stop attributes, surrounding activity context, historical demand and service summaries, and schema\-constrained LLM outputs\. The resulting representation is integrated into an event\-driven RL controller\. The simulation results show that the semantic\-enhanced controller provides the most favorable overall balance among the evaluated RL state designs in the same\-route experiment\. It achieves the lowest headway variation, passenger waiting time, and holding time among the RL controllers, while attaining the second\-lowest bunching count\. Stop\-level action patterns further suggest that semantic information changes the spatial allocation of holding effort rather than simply increasing holding intensity\. In the cross\-route experiment, zero\-shot deployment of the source\-route policy provides partial benefits but does not fully adapt to the target route\. Fine\-tuning improves the transferred policy and provides better initialization and early\-stage learning, whereas cold\-start training on the target route achieves the strongest final regularity and passenger\-delay performance after convergence\.

Overall, the findings indicate that semantic stop representations can complement conventional bus holding states and support source\-to\-target policy reuse and adaptation in the studied setting\. From an implementation perspective, the representations are constructed and cached offline, so language\-model inference, text embedding, and representation alignment do not add computation or latency to the real\-time control loop\. Representing stops in a shared feature space also provides a practical basis for reusing learned policies across related routes\. Such reuse may reduce target\-route training and retraining needs\. It may also enable agencies to test, update, and deploy control strategies more rapidly than developing a separate policy from scratch for each route\. However, the study is limited to simulation\-based experiments on two university shuttle routes and one primary transfer direction\. Future work should evaluate the framework on larger and more heterogeneous transit networks, quantify the data and computational savings associated with policy reuse, examine additional transfer directions, test robustness under demand shifts and disruptions, and further isolate the effects of semantic labels from those of the underlying operational information\.

## GENERATIVE AI DISCLOSURE

ChatGPT \(OpenAI\) was used to assist with language editing and rephrasing portions of the manuscript\. The authors reviewed and verified all generated text and take full responsibility for the content of the manuscript\.

## AUTHOR CONTRIBUTIONS

All authors contributed to the study conception and design, data collection, methodology development, analysis and interpretation of the results, and manuscript preparation and revision\. All authors reviewed the results and approved the final version of the manuscript\.

## DECLARATION OF CONFLICTING INTERESTS

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article\.

## FUNDING

The authors disclosed no financial support for the research, authorship, and/or publication of this article\.

## References