TraveL: Transformer-based Multi-view Path Distributional Representation Learning

# TraveL: Transformer-based Multi-view Path Distributional Representation Learning
Source: [https://arxiv.org/html/2609.03427](https://arxiv.org/html/2609.03427)
Conference:31st ACM International Conference on Information and Knowledge Management; October 17–22, 2022; Atlanta, Georgia, the USA31st ACM International Conference on Information and Knowledge Management \(CIKM ’22\), October 17–22, 2022, Atlanta, Georgia, the USA,Tao\-Yang FuAffiliation:The Pennsylvania State Universityemail:[txf225@psu\.edu](mailto:[email protected])andWang\-chien LeeAffiliation:The Pennsylvania State Universityemail:[wlee@cse\.psu\.edu](mailto:[email protected])

© none

###### Abstract\.

Path representation learning \(PRL\) for road networks has received increasing research attention, due to various path\-related applications\. Existing works on PRL typically exploit the co\-occurrence relationship among road segments and paths to learn a vector as the path representation, without exploring the varied traveler behaviors and the regional correlation on the path\. In this work, we propose to learndistributional representations, which provide valuable information for use in path\-related applications, by capturing thevaried traveler behaviorsas well as thevarious dependencies within regions of road segments\. We propose a novel Transformer\-based Multi\-view Distributional Representation Learning \(TraveL\) framework to encode a path along with a travel starting time to a distributional representation, which can be used to decode possible samples of on\-path traveler behavior\. Moreover, by analyzing theregional correlationwhich reveals various road segment relationships, we propose aregional attentionto encode these correlations in a path\. Also, we explore the idea of Kolmogorov–Smirnov \(K\-S\) test to compare the sampled traveler behavior against the collected ground truth to facilitate training\. Experimental results show that the proposed TraveL model outperforms the state\-of\-the\-art methods on both synthetic and real\-world datasets, by 14\.7% in Mean K\-S distance for travel time distribution estimation, 16\.7% in Mean Absolute Error \(MAE\) for path similarity prediction, and 3\.97% in MAE for destination prediction\.

###### Keywords:

Path representation learning; Distributional representation; Regional attention

## 1\.Introduction

With the rapid growth of GPS\-enabled devices and location\-aware applications, a large volume of trajectory data are collected, which provides opportunities to study and improve various applications in intelligent transportation systems \(ITS\)\. Among them, many ITS applications involvepaths, i\.e\., a sequence of consecutive road segments, such as path travel time estimation, path recommendation, and destination prediction \(given a partial path\)\.111Note that a path isnota trajectory which is a sequence of GPS points\.In support of these applications, in this work, we study the problem ofPath Representation Learning \(PRL\), aiming to encode a given path in the road network into a general\-purpose representation, e\.g\., a low\-dimension latent vector\. Similar to existing representation learning models in different domains, e\.g\., Doc2Vec for document representation learning\([Le and Mikolov, 2014](https://arxiv.org/html/2609.03427#bib.bib11)\)and HIN2Vec for network representation learning\([Fu et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib6)\), PRL may reduce both storage cost and human labors required for extensive feature engineering in path\-related applications, showing its great values\.

![Refer to caption](https://arxiv.org/html/2609.03427v1/Figures/path-highway-view.PNG)Figure 1\.Regional Correlation of a Path in Highway ViewFor PRL in a road network, two kinds of information are important and essential: i\)the varied behaviors from travelerson the same path\. In addition to the static features of a path \(e\.g\., the road network structure\), it’s imperative to capture the varied traveler behaviors, especially in terms of travel speed along the path, which intuitively benefits the travel time distribution estimation\. It may also benefit path similarity estimation, especially when the similarity measurement of interest is related to the traveler behaviors on the path \(e\.g\., finding another path with a similar travel time fluctuation as a given path\); and ii\)regional correlation in the paths, brought by the co\-occurrence and same\-type relationships among road segments, indicating various dependencies among the travel behaviors on different road segments\. For example, as shown in Figure[1](https://arxiv.org/html/2609.03427#S1.F1), drivers drive along a path consisting of 7 road segments \(denoted asr1r\_\{1\}tor7r\_\{7\}respectively\)\. We observe thatr1r\_\{1\},r2r\_\{2\}andr3r\_\{3\}are in the urban area, whiler4r\_\{4\},r5r\_\{5\}andr6r\_\{6\}are on the highway\. We argue that the driving behavior, e\.g\., speed, from an urban road to a highway may change significantly, while the driving behavior on continuous road segments of the same type \(i\.e\., a path region of highway road segments\) is relatively stable\. It would be helpful, yet challenging, to capture the regional correlation in paths for PRL\.

Existing works on PRL starts by learning task\-specific path representations\. Liu et al\.\([Liu et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib14)\), Li et al\.\([Li et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib13)\)and Yang et al\.\([Yang and Yang, 2019](https://arxiv.org/html/2609.03427#bib.bib25)\)exploit various recurrent neural network based models to aggregate road segment embeddings in the path as the path representation for their targeted tasks\. Aiming for general\-purpose PRL, Yang et al\. propose Path InfoMax \(PIM\)\([Yang et al\., 2021](https://arxiv.org/html/2609.03427#bib.bib24)\), with a curriculum negative sampling strategy to generate negative paths to learn path representations in a general graph\. Trembr, recently proposed by Fu et al\.\([Fu and Lee, 2020](https://arxiv.org/html/2609.03427#bib.bib5)\), exploits pre\-trained road segment embeddings to capture road segment co\-occurrence and same\-type relationship, and encodes a path into its representation with an LSTM\-based encoder\-decoder structure\. PIM and Trembr do not capture the varied traveler behaviors, and thus fail to learn path representations well\.

In this work, we propose to capture both the varied traveler behaviors and the regional correlation for PRL\. We propose to explore an encoder\-decoder framework which first encodes the targeted path into a probabilistic distribution as the representation, capable of generating expected travel behaviors on the path\. However, we facethree issues: i\) conventional latent vectors are insufficient to cover the varied traveler behaviors\. To address the issue, we explore the idea ofdistributional representation, which learns a distribution as the representation of a path, with higher capacity than a latent vector\. ii\) the decoded travel speeds/behaviors of different travelers along a path do not follow a known parameterized distribution, making it inappropriate to learn the parameters based on some known distribution of travel behaviors \(in speed\)\. Instead, we propose to generate a set ofon\-path sequences \(OP\-Seqs\)from the distributional representation, each of which refers to a possible travel trace on the path, i\.e\., a trace consists of the path and the travel time spent on each road segment in the path\. As such, we can optimize the path distributional representation by measuring and minimizing the dissimilarity between the generated and ground\-truth OP\-Seqs\. iii\) to bridge from the distributional representation to the OP\-Seqs is challenging\. To resolve the issue, we explore a sampling\-based approach in the decoding phase, which regards a point sampled from the distributional representation as the representation of a possible OP\-Seq\. As such, the distributional representation covers all possible OP\-Seqs and thus acts as a proper path representation\.

In addition, we capture various correlations in path regions for PRL where a path region is some sequence of consecutive road segments in the path\. We propose two schemes to form path regions to explore regional correlation: i\) road\-type based\. The traveler behaviors in a region of a same road type \(shown in Figure[1](https://arxiv.org/html/2609.03427#S1.F1)with red bounding boxes\) may be correlated; and ii\) hop based, traveler behaviors within a number of hops along the path, may be correlated\. We use the term,view, to denote the path structure exhibiting a specific regional correlation, e\.g\., highway view as shown in Figure[1](https://arxiv.org/html/2609.03427#S1.F1)\. To capture the regional correlation, we propose the idea ofregional attention, by letting each road segment correlate other road segments in the same region to enhance the road segment representations\.

![Refer to caption](https://arxiv.org/html/2609.03427v1/TRAVEL-Framework.png)Figure 2\.The TraveL FrameworkTo realize our ideas, we propose theTransformer\-based Multi\-view Distributional RepresentationLearning \(TraveL\) framework, to capture both varied traveler behaviors and the regional correlations to learn path distributional representations\. As shown in Figure[2](https://arxiv.org/html/2609.03427#S1.F2), the TraveL framework consists of aPath Encoderto encode an input path, i\.e\.,pp, and a departure time \(we considerrush hourandnormal hourfor simplicity\) into a distributional representation, and anOn\-path Sequence Generatorwhich decodes a set of OP\-Seqs from the path distributional representation to approach the ground truth, i\.e\., the historical OP\-Seqs onpp\. More specifically, Path Encoder first exploits the Road2Vec framework\([Fu and Lee, 2020](https://arxiv.org/html/2609.03427#bib.bib5)\)to learn a latent vector as the initial embedding for each road segment\. The sequence of road segment embeddings inpp, is then fed to a Multi\-view Path Transformer, which applies regional attention to capture the regional correlation to output a distribution representation forpp, denoted asNp​\(μp,Σp\)N\_\{p\}\(\\mu\_\{p\},\\Sigma\_\{p\}\)\(here we learn a Gaussian Distribution due to its generality\)\. FromNp​\(μp,Σp\)N\_\{p\}\(\\mu\_\{p\},\\Sigma\_\{p\}\), we samplennpoints, each of which is regarded as a representation of a possible OP\-Seq\. Each point is then fed to a Long\-short Term Memory \(LSTM\) model to both generate the OP\-Seq and recoverppwith a series of predictions of the next road segments\. To facilitate training, we measure the error for both OP\-Seqs generation \(i\.e\., travel time recovery\) and the next road segment prediction \(i\.e\., path recovery\)\. It is worth noting that we explore a novel idea of using K\-S test\([Massey Jr, 1951](https://arxiv.org/html/2609.03427#bib.bib15)\)to evaluate the error of OP\-Seqs generation\. Also note that, while the traditional variational autoencoder \(VAE\) also follows an encoder\-decoder paradigm with a prior distribution included, TraveL and VAE are totally different\. VAE learns to generate all the input paths from a prior distribution, where each path is generated from a latent vector sampled from the same prior\. On the other hand, TraveL encodes each path to its own distributional representation in order to capture all possible traveler behaviors on the path\. Finally, to deal with the data sparsity issue of real\-world datasets, we generate a synthetic dataset, Syn\-Porto, for a complete evaluation of TraveL against several state\-of\-the\-art models\. We also conduct extensive experiments on two real\-world datasets, i\.e\., Porto and Tokyo\. The results show that TraveL outperforms all baseline models for three applications on all the datasets\.

Major contributions made in this work are as follows\.

- •We propose a novel idea ofdistributional representationto capture the varied traveler behaviors for PRL\.
- •We analyze theregional correlationin a path, and propose aregional attentionmechanism to capture it for PRL\.
- •We proposeTraveL, a new end\-to\-end framework, to capture varied traveler behaviors and regional correlation for PRL\. We propose a novel sampling\-based approach for varied traveler behavior generation, and exploit the idea of K\-S test for the evaluation\.
- •We conduct extensive experiments on both synthetic and real\-world datasets to evaluate TraveL against the state\-of\-the\-arts PRL models\. Experimental results show thatTraveLoutperforms the state of the arts under various metrics for all three downstream applications, by reducing the mean K\-S distance by 14\.7% for travel time distribution estimation\.

## 2\.Related Work

We briefly review the related work on path representation learning and distributional representation learning\.

### 2\.1\.Path Representation Learning

The goal of representation learning is to encode raw data into general\-purpose low\-dimensional latent vectors, i\.e\., embeddings, that are effectively fed as inputs to downstream machine learning and data mining methods for various applications\. In recent years, neural network based representation learning methods have attracted a lot of research interests in various domains, e\.g\., text processing\([Mikolov et al\., 2013](https://arxiv.org/html/2609.03427#bib.bib16);[Le and Mikolov, 2014](https://arxiv.org/html/2609.03427#bib.bib11);[Devlin et al\., 2018](https://arxiv.org/html/2609.03427#bib.bib3);[Zhang et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib26)\), graph analytics\([Grover and Leskovec, 2016](https://arxiv.org/html/2609.03427#bib.bib8);[Fu et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib6);[Sun et al\., 2019](https://arxiv.org/html/2609.03427#bib.bib20);[Goyal et al\., 2020](https://arxiv.org/html/2609.03427#bib.bib7);[Xie et al\., 2016](https://arxiv.org/html/2609.03427#bib.bib23)\), computer vision\([Radford et al\., 2015](https://arxiv.org/html/2609.03427#bib.bib17);[Xia et al\., 2014](https://arxiv.org/html/2609.03427#bib.bib22);[Dosovitskiy et al\., 2020](https://arxiv.org/html/2609.03427#bib.bib4);[Chen et al\., 2020](https://arxiv.org/html/2609.03427#bib.bib2);[Lee et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib12)\), etc\. Only recently, research on representation learning has been extended to path data, which are categorized into two folds\. First, some existing works learn path representations for a specific targeted task, e\.g\., proximity search\([Liu et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib14)\), cascade prediction in a social network\([Li et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib13)\)and path ranking\([Yang and Yang, 2019](https://arxiv.org/html/2609.03427#bib.bib25)\)\. The goal of these methods is not to learn general\-purpose embeddings of paths, but is to exploit various recurrent neural network based models to aggregate the embeddings of road segments in a given path to generate a path representation for their specific\. Second, aligned with our goal, some existing works learn general\-purpose path embeddings\. Among them, Yang et al\. propose PathInforMax \(PIM\), with a curriculum negative sampling strategy to generate negative paths and two discriminators to distinguish the difference between the representation of the input path and those of its negative paths to learn path representations\([Yang et al\., 2021](https://arxiv.org/html/2609.03427#bib.bib24)\)\. Besides, BERT, a famous language model proposed by Delvin et al\.\([Devlin et al\., 2018](https://arxiv.org/html/2609.03427#bib.bib3)\)to capture long\-term dependencies among nodes, is also examined by Yang et al\. for PRL in their study\. However, BERT is not designed for road networks, failing to capture the regional correlation brought by road types\. Fu et al\. propose Trembr, by exploiting pre\-trained road segment embeddings to capture static information in the road network\. Trembr encodes an input path \(i\.e\., a sequence of road segments\) into its representation by using an LSTM\-based encoder\-decoder neural network structure\([Fu and Lee, 2020](https://arxiv.org/html/2609.03427#bib.bib5)\)\. However, Trembr does not capture the varied traveler behaviors in the road network, and thus may not learn a proper path representations\.

### 2\.2\.Distributional Representation Learning

Variational Autoencoder \(VAE\) is a famous framework that learns the mapping from a prior distribution to the distribution of the inputs\([Kingma and Welling, 2013](https://arxiv.org/html/2609.03427#bib.bib10);[Rezende et al\., 2014](https://arxiv.org/html/2609.03427#bib.bib19)\)\. While VAE maps the input to a latent distribution, VAE is to learn a generator to generate outputs following the same distribution as the inputs from the prior, instead of learning a distributional representation for each input\. Recently, Ren et al\. propose BETAE, which embeds entities and queries as Beta distributions, to capture the uncertainty of the queries, for multi\-hop reasoning over knowledge graphs\([Ren and Leskovec, 2020](https://arxiv.org/html/2609.03427#bib.bib18)\)\. However, to the best of our knowledge, our work is the first attempt to learn path distributional representations in the road network\.

## 3\.RESEARCH Problem and Challenges

In this section, we introduce some key terms, define the targeted research problem and discuss the challenges\.

###### Definition 0\.

Road Network\. A road network can be represented as a directed graphG=\(V,E,Ψ\)G=\(V,E,\\Psi\), whereVVis a set of nodes representing intersections, associated with its coordinates\(v\.lon,v\.lat\)\(v\.lon,v\.lat\)\(i\.e\., longitude and latitude\);E⊆V×VE\\subseteq V\\times Vis a set of directed edges representing road segments; andΨ:E→ℱ\\Psi:E\\to\\mathcal\{F\}is a function mapping an edge to its features, e\.g\., a road segment is athree\-lanehighway\.

As the traveler behaviors may be highly dependent on the departure timedd\(in terms of rush hours or normal hours\), we distinguish the two paths with the same road segments but different departure times\. In other words, we define the termpathwithdd\.

###### Definition 0\.

Path\. A pathp=\{r1,r2,r3,…,r\|p\|;d\}p=\\\{r\_\{1\},r\_\{2\},r\_\{3\},\.\.\.,r\_\{\|p\|\};d\\\}is a sequence of connected road segments with a departure timedd, whereri∈Er\_\{i\}\\in Eis theii\-th road segment inpp,\|p\|\|p\|is the number of road segments inpp, andddis eitherrush hourornormal hour\.

###### Definition 0\.

On\-path Sequence\. An on\-path sequenceT=\{\(r1,tr1\),\(r2,tr2\),…,\(r\|p\|,tr\|p\|\)\}T=\\\{\(r\_\{1\},t\_\{r\_\{1\}\}\),\(r\_\{2\},t\_\{r\_\{2\}\}\),\.\.\.,\(r\_\{\|p\|\},t\_\{r\_\{\|p\|\}\}\)\\\}consists of a pathppand the travel time spent on each road segment, whererir\_\{i\}is theii\-th road segment inppandtrit\_\{r\_\{i\}\}is the travel time onrir\_\{i\}\.

As the traveler behaviors are well captured in trajectory data, we formally define a trajectory as follows\.

###### Definition 0\.

Trajectory\. A trajectory is a sequence of spatio\-temporal sample points, each of which contains a location \(i\.e\., longitude and latitude\) and a timestamp, generated from the movement of a traveler on a path\.

While trajectory \(which is widely studied in many existing works\) and path are related, please note the following difference to avoid confusion – a path consists of a sequence of road segments, while a trajectory consists of a sequence of spatio\-temporal sample points\.

To observe and analyze the real\-world traveler behavior on a path, i\.e\., travel time spent on a path, we project real\-world trajectory data onto a road network using existing map mapping methods to obtain on\-path sequences\. To capture the varied traveler behaviors and to increase the capacity of path representation, we explore to learn adistributional representationfor a path\. We choose Gaussian distribution as the form of path representations due to its generality\. We define the research problem as below\.

###### Definition 0\.

Path Representation Learning in Road Networks\. Given a dataset of selective pathsD=\{pi\}i=1\|D\|D=\\big\\\{p\_\{i\}\\big\\\}\_\{i=1\}^\{\|D\|\}in a road networkGG, wherepip\_\{i\}is theii\-th path inDD\. The task of Path Representation Learning is to learn to map a pathppto a Gaussian distributionNp​\(μp,Σp\)N\_\{p\}\(\\mu\_\{p\},\\Sigma\_\{p\}\), whereμp∈ℝd\\mu\_\{p\}\\in\\mathbb\{R\}^\{d\}andΣp∈ℝd×d\\Sigma\_\{p\}\\in\\mathbb\{R\}^\{d\\times d\}, in add\-dimension space in support of a variety of path mining tasks\.

In our work, we explore encoder\-decoder paradigm, which first encodes various signals on the targeted path, with the regional correlation captured, into the distributional representation, and then decodes the representation to generate varied traveler behaviors\. To realize the idea, we face the following challenges: \(1\)Recovery of the varied traveler behaviors\. The travel time spent on the road segments by travelers along a path may not follow a known distribution \(e\.g\., Gamma or Chi\-squared distribution\)\. Thus we are not able to statistically fit the historical travel times to a known parameterized distribution as the path representation\. How to reasonably generate varied traveler behaviors, i\.e\., varied travel times, from the distributional representation? How to evaluate the generated travel times with the ground truth? How to deal with the data sparsity issue in real\-world datasets, i\.e\., the lack of ground truth travel times on each path? \(2\)Capturing the regional correlation in the path\.Existing models fail to capture regional correlation brought by various road segment relationships\. What road segment relationships bring the regional correlation, that should be captured? How to capture regional correlation with Path Encoder? In the following, we introduce our design of TraveL to address these questions\.

## 4\.Design of TraveL

In this section, we first introduce the proposed TraveL framework and then detail our design of its components\.

### 4\.1\.The TraveL Framework

To capture both the varied traveler behaviors and the regional correlation on the path for PRL, we follow the encoder\-decoder paradigm in the design of TraveL\. As shown in Figure[2](https://arxiv.org/html/2609.03427#S1.F2), TraveL includes three components: 1\) aPath Encoderto encode the sequence of road segments in the target pathpp, together with the departure time, to its distributional representation in the latent space; 2\) anOn\-path Sequence \(OP\-Seq\) Generatorwhich generates a set of possible OP\-Seqs onpp, which are expected to follow the same distribution in travel behaviors \(i\.e\., travel times\) as the ground\-truth, i\.e\., the historical OP\-Seqs onpp, and recoversppby sequentially predicting the next road segment; and 3\) anOn\-path Sequence Evaluatorto evaluate both the recovery of the path and the generation of OP\-Seqs to facilitate training\. Here, we propose a sampling\-based approach to generate the OP\-Seqs: we regard a sample point from the path distributional representation as the representation of a possible OP\-Seq\. As such, the distributional representation covers all possible OP\-Seqs on the path and thus acts as a proper path representation with the varied traveler behaviors well captured\.

Specifically, given a pathpp=\{r1,r2,…,r\|p\|\}\\\{r\_\{1\},r\_\{2\},\.\.\.,r\_\{\|p\|\}\\\},ppis fed to the Path Encoder to generate itsdistributional representationfollowed Gaussian distribution, denoted asN⁡\(μp,Σp\)N\(\\mu\_\{p\},\\Sigma\_\{p\}\)\. A sampling process is then applied to samplennpoints, i\.e\.,s1,s2,…,sns\_\{1\},s\_\{2\},\.\.\.,s\_\{n\}, fromNp​\(μp,Σp\)N\_\{p\}\(\\mu\_\{p\},\\Sigma\_\{p\}\)\. After that, the sample pointsjs\_\{j\}, is fed to the OP\-Seq Generator to generate an OP\-SeqTj′=\{\(r1,t′r1j\),…,\(r\|p\|,t′r\|p\|j\)\}T^\{\\prime\}\_\{j\}=\\\{\(r\_\{1\},\{t^\{\\prime\}\}^\{j\}\_\{r\_\{1\}\}\),\.\.\.,\(r\_\{\|p\|\},\{t^\{\\prime\}\}^\{j\}\_\{r\_\{\|p\|\}\}\)\\\}, wheret′rkj\{t^\{\\prime\}\}^\{j\}\_\{r\_\{k\}\}is the generated travel time onrkr\_\{k\}\. Meanwhile, the OP\-Seq Generator predicts the next road segment, i\.e\.,rk\+1′r^\{\\prime\}\_\{k\+1\}, given the partial path tillrkr\_\{k\}\. With these outputs fed, to evaluate the generated travel times, OP\-Seq Evaluator collects historical OP\-Seqs onpp, and measure the dissimilarity between the set of generated travel times and the set of historical travel times with K\-S distance\. Besides, OP\-Seq Evaluator treats the next road segment prediction as a classification task, with a maximum likelihood based loss function applied\. With both the paths and historical OP\-Seqs fed, we train TraveL end to end\. Owing to the proven power of these loss functions, we leave the idea of learning a neural net based OP\-Seq Evaluator with adversarial learning in future\. Note that, to simplify the sampling process, we assume that the learned Gaussian distribution hasΣp=σp2​I\\Sigma\_\{p\}=\{\\sigma\_\{p\}\}^\{2\}I\. Owing to the power of the OP\-Seq Generator to learn complicated transformation fromNp​\(μp,σp2​I\)N\_\{p\}\(\\mu\_\{p\},\{\\sigma\_\{p\}\}^\{2\}I\)to the historical OP\-Seqs, this assumption \(and the adoption of Gaussian distribution as the path distributional representation\) may not compromise the model’s ability to recover Op\-Seqs\.

### 4\.2\.Path Encoder

In this section, we detail our design of Path Encoder, which encodes a path and its departure time to a distributional representation\. As shown in Figure[3](https://arxiv.org/html/2609.03427#S4.F3), Path Encoder first encodes each road segment to an embedding with a process of pre\-training and feature transformation, and then feed the new road segment embeddings to a Multi\-view Path transformer to capture the regional correlation to enhance road segment embeddings, and finally aggregate the road segment embeddings to generate the path distributional representation\. Now, we detail all the steps\.

Figure 3\.The Structure of Path Encoder#### 4\.2\.1\.Road Segment Embedding Initialization\.

Owing to the proven power of Road2Vec for road segment embedding pre\-training\([Fu and Lee, 2020](https://arxiv.org/html/2609.03427#bib.bib5)\), in this work, we first exploit Road2Vec to generate the initial road segment embeddings, denoted as\{ei,i=1,2,…,\|p\|\}\\\{e\_\{i\},i=1,2,\.\.\.,\|p\|\\\}forpp\. In addition, we exploit a two\-dimension one\-hot departure time embedding,ede\_\{d\}, which is\(0,1\)T\(0,1\)^\{T\}whenddis in rush hour \(8\-10AM and 5\-7PM\), and\(1,0\)T\(1,0\)^\{T\}whenddis in normal hours or\(0,0\)T\(0,0\)^\{T\}whenddis not known\. Then we exploit a one\-layer feed\-forward net \(FFN\) to transform the concatenation ofeie\_\{i\},ede\_\{d\}and a positional embeddingep​o​se\_\{pos\}\(i\.e\., a sinusoidal function used in Transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib21)\)\) to a same latent space, i\.e\.,xi0=F​F​N​\(\[ei,et,ep​o​s\]\)=W2​R​e​L​U​\(W1​\[ei,et,ep​o​s\]\+b1\)\+b2x^\{0\}\_\{i\}=FFN\(\[e\_\{i\},e\_\{t\},e\_\{pos\}\]\)=W\_\{2\}ReLU\(W\_\{1\}\[e\_\{i\},e\_\{t\},e\_\{pos\}\]\+b\_\{1\}\)\+b\_\{2\}, whereW1W\_\{1\},W2W\_\{2\}are two feature transformation matrices;b1b\_\{1\},b2b\_\{2\}are two bias vectors; and ReLU is the non\-linear activation function\.

![Refer to caption](https://arxiv.org/html/2609.03427v1/Figures/path-lane-view.PNG)Figure 4\.Regional Correlation of a Path in Lane View
#### 4\.2\.2\.Multi\-view Path Transformer

Here we first analyze the regional correlation in the path\. We use the termpath regionto denote some sequence of consecutive road segments in the path, where the traveler behaviors within a path region are correlated\. We observe two types of road segment relationship bringing the regional correlation: i\) same road\-type relationship\. The traveler behaviors in a region of the same road type, may be correlated\. Figure[4](https://arxiv.org/html/2609.03427#S4.F4)shows an example where travelers start the trip from the left\-bottom corner and the destination is at the right\-upper corner\. We observe thatr2r\_\{2\}andr3r\_\{3\}are three\-lane roads, while the rest are two\-lane roads\. Intuitively, the traveler behavior may change significantly when the traveler drives from a two\-lane road to a three\-lane road, while keep stable on a consecutive sequence of two\-lane road segments; and ii\) co\-occurrence relationship\. Traveler behaviors within a number of hops along the path, may be correlated\.

Thus, we propose to define differentviewsof the paths, where each view denotes a path structure exhibiting some specific regional correlation, e\.g\., the lane view which has three path regions \(as shown in Figure[4](https://arxiv.org/html/2609.03427#S4.F4)with green bounding boxes\)\. More generally, Algorithm[1](https://arxiv.org/html/2609.03427#alg1)shows the process to generate path regions under a type\-based view for the type𝒯\\mathcal\{T\}\. In this work, we explore thehighway viewand thelane viewas the representative type\-based views\. In addition, we propose ahop view, where everyHH\(a hyper\-parameter to decide\) continuous road segments in the path form a region, i\.e\.,\{r1,…,rH\}\\\{r\_\{1\},\.\.\.,r\_\{H\}\\\}as the first region and\{rH\+1,…,r2​H\}\\\{r\_\{H\+1\},\.\.\.,r\_\{2H\}\\\}as the second region, etc\. However, co\-occurrence relationships among road segments around the region borders, e\.g\.,rHr\_\{H\}andrH\+1r\_\{H\+1\}, are not shown as they are in different regions\. To resolve the issue, we explore the idea ofregion shiftingto shift all regions right together, by one road segment, to form a new view\. As such, with at most\(H−1\)\(H\-1\)times shift, each pair of two correlated road segments are guaranteed to be in the same region at least in one view\. All the\(H−1\)\(H\-1\)views work together to exhibit the co\-occurrence relationships\.

Algorithm 1Region generation under a type\-based view0:input path

p=r1,r2,…,r\|p\|p=\{r\_\{1\},r\_\{2\},\.\.\.,r\_\{\|p\|\}\}; road segment type values

𝒯1,𝒯2,…,𝒯\|p\|\\mathcal\{T\}\_\{1\},\\mathcal\{T\}\_\{2\},\.\.\.,\\mathcal\{T\}\_\{\|p\|\}of a given type

𝒯\\mathcal\{T\};

1:Initialize the set of regions

ℛ=∅\\mathcal\{R\}=\\varnothing, road segment index \(to assign\)

i=1i=1, region \(to generate\) index

k=1k=1;

2:repeat

3:Initialize

ℛk=∅\\mathcal\{R\}\_\{k\}=\\varnothing
4:while

i≤\|p\|i\\leq\|p\|and \(

ℛk\\mathcal\{R\}\_\{k\}is

∅\\varnothingor \(

ri−1∈ℛkr\_\{i\-1\}\\in\\mathcal\{R\}\_\{k\}and

𝒯i−1=𝒯i\\mathcal\{T\}\_\{i\-1\}=\\mathcal\{T\}\_\{i\}\)\)do

5:

ℛk\\mathcal\{R\}\_\{k\}=

ℛk∪\{ri\}\\mathcal\{R\}\_\{k\}\\cup\\\{r\_\{i\}\\\},

i=i\+1i=i\+1
6:endwhile

7:

ℛ=ℛ∪\{ℛk\}\\mathcal\{R\}=\\mathcal\{R\}\\cup\\\{\\mathcal\{R\}\_\{k\}\\\},

k=k\+1k=k\+1
8:until

i\>\|p\|i\>\|p\|

To capture the regional correlation under each view, we propose the idea ofregional attention, which correlates each road segment to other road segments in the same region\. We implement regional attention in each of the stacked Multi\-view Regional Attention \(MVRA\) Layers in the Multi\-view Path Transformer, and Figure[5](https://arxiv.org/html/2609.03427#S4.F5)shows the structure of thell\-th MVRA layer\. Generally, road segment embeddings are first fed to a Multi\-view Attention Layer to capture the regional correlation to generate a new embedding for each road segment\. Then the new embeddings generated from different views are averaged and fed to a Multi\-head Path Self\-attention Layer to capture the long\-term dependency among the road segments\. Finally, the road segment embeddings are fed to an FFN for a non\-linear transformation, and then fed with a residual connection and a layer normalization to be the road segment representations input to the\(l\+1\)\(l\+1\)MVRA layer\.

Figure 5\.Thell\-th Multi\-view Regional Attention LayerMore specifically, we denote the embedding ofrir\_\{i\}input to thell\-th MVGA layer asxil−1x^\{l\-1\}\_\{i\}\. Under a viewvv, the path is split toKvK\_\{v\}regions, which are denoted asR1\(v\)R^\{\(v\)\}\_\{1\}, …,RKv\(v\)R^\{\(v\)\}\_\{K\_\{v\}\}respectively\. Then, for thekk\-th region, i\.e\.,Rk\(v\)R^\{\(v\)\}\_\{k\}, and each road segmentri∈Rk\(v\)r\_\{i\}\\in R^\{\(v\)\}\_\{k\}, the regional attention is to generate a new embedding, denoted asxil,vx^\{l,v\}\_\{i\}, forrir\_\{i\}by capturing its correlation with other road segments in the region with the following steps: \(1\) we apply feature transformation on eachxil−1x^\{l\-1\}\_\{i\}∈\\inRk\(v\)R^\{\(v\)\}\_\{k\}with a query matrixWQl,vW^\{l,v\}\_\{Q\}, an answer matrixWAl,vW^\{l,v\}\_\{A\}and a value matrixWHl,vW^\{l,v\}\_\{H\}to transformxil−1x^\{l\-1\}\_\{i\}to its query vectorqil,vq^\{l,v\}\_\{i\}, answer vectorail,va^\{l,v\}\_\{i\}and value vectorhil,vh^\{l,v\}\_\{i\}byqil,v=WQl,v⋅xil−1q^\{l,v\}\_\{i\}=W^\{l,v\}\_\{Q\}\\cdot x^\{l\-1\}\_\{i\},ail,v=WAl,v⋅xil−1a^\{l,v\}\_\{i\}=W^\{l,v\}\_\{A\}\\cdot x^\{l\-1\}\_\{i\}andhil,v=WHl,v⋅xil−1h^\{l,v\}\_\{i\}=W^\{l,v\}\_\{H\}\\cdot x^\{l\-1\}\_\{i\}, respectively\. \(2\) for eachrj∈Rk\(v\)r\_\{j\}\\in R^\{\(v\)\}\_\{k\}, we propose to derive the impact ofrjr\_\{j\}onrir\_\{i\}by calculating an attention score,αi​jl,v=fαl,v​\(qil,v,ajl,v\)=\(qil,v\)T​Wαl,v​ajl,v\\alpha^\{l,v\}\_\{ij\}=f^\{l,v\}\_\{\\alpha\}\(q^\{l,v\}\_\{i\},a^\{l,v\}\_\{j\}\)=\(\{q^\{l,v\}\_\{i\}\}\)^\{T\}W^\{l,v\}\_\{\\alpha\}a^\{l,v\}\_\{j\}, wherefαl,v​\(⋅,⋅\)f^\{l,v\}\_\{\\alpha\}\(\\cdot,\\cdot\)is the attention function withWαl,vW^\{l,v\}\_\{\\alpha\}as a feature transformation matrix to learn\. \(3\) we normalize the attention scores with a softmax function, and exploit the normalized attention scores to aggregate the value vectors to generatexil,vx^\{l,v\}\_\{i\}as follows,

\(1\)βi​jl,v=e​x​p​\(αi​jl,v\)∑rj∈Rk\(v\)e​x​p​\(αi​jl,v\),xil,v=∑rj∈Rk\(v\)βi​jl,v​hjl,v\\small\\beta^\{l,v\}\_\{ij\}=\\frac\{exp\(\\alpha^\{l,v\}\_\{ij\}\)\}\{\\sum\_\{r\_\{j\}\\in R^\{\(v\)\}\_\{k\}\}exp\(\\alpha^\{l,v\}\_\{ij\}\)\},\\hskip 9\.24994ptx^\{l,v\}\_\{i\}=\\sum\_\{r\_\{j\}\\in R^\{\(v\)\}\_\{k\}\}\\beta^\{l,v\}\_\{ij\}h^\{l,v\}\_\{j\}The generated embeddings ofrir\_\{i\}under different views, i\.e\.,xil,1x^\{l,1\}\_\{i\}, …,xil,Vx^\{l,V\}\_\{i\}, are then averaged to bex′il\{x^\{\\prime\}\}^\{l\}\_\{i\}to feed to the following Multi\-head Path Self\-Attention Layer, which follows the attention design in Transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib21)\), to capture the long\-term dependency among road segments to outputzilz^\{l\}\_\{i\}as the embedding ofrir\_\{i\}\. Finally,zilz^\{l\}\_\{i\}is fed to a two\-layer FFN for further feature transformation, and then fed to a layer\-normalization layer with a residual link built to generatexilx^\{l\}\_\{i\}, i\.e\.,xil=L​a​y​e​r​N​o​r​m​\(F​F​N​\(zil\)\+xil−1\)x^\{l\}\_\{i\}=LayerNorm\(\{FFN\(z^\{l\}\_\{i\}\)\}\+x^\{l\-1\}\_\{i\}\)\. By stackingLLMVRA layers, road segment representations have more opportunity to correlate with other road segments to capture more complicated relationships\. At the end,\{xiL,i=1,2,…,\|p\|\}\\\{x^\{L\}\_\{i\},i=1,2,\.\.\.,\|p\|\\\}is fed to a Self\-gating Aggregation Layer to generate the path distributional representation\.

For aggregation, the idea is to first transform the embedding ofrir\_\{i\}into two embeddings in different latent spaces, i\.e\.,xiμx^\{\\mu\}\_\{i\}andxiσx^\{\\sigma\}\_\{i\}, with the former for predictionμp\\mu\_\{p\}and the latter for predictionσp\\sigma\_\{p\}; and then exploit a weighted sum of\{xiμ\}\\\{x^\{\\mu\}\_\{i\}\\\}asμp\\mu\_\{p\}and a weighted sum of\{xiσ\}\\\{x^\{\\sigma\}\_\{i\}\\\}asσp\\sigma\_\{p\}\. Formally, we generatexiμx^\{\\mu\}\_\{i\}andxiσx^\{\\sigma\}\_\{i\}by,xiμ=t​a​n​h​\(Wμ​xiL\+bμ\)x^\{\\mu\}\_\{i\}=tanh\(W\_\{\\mu\}x^\{L\}\_\{i\}\+b\_\{\\mu\}\)andxiσ=t​a​n​h​\(Wσ​xiL\+bσ\)x^\{\\sigma\}\_\{i\}=tanh\(W\_\{\\sigma\}x^\{L\}\_\{i\}\+b\_\{\\sigma\}\)\. Then we adopt a self\-gating mechanism to derive the weight of each road segment to the path, and normalize the weights as follows,

\(2\)γiμ=e​x​p​\(fμ​\(xiμ\)\)∑ie​x​p​\(fμ​\(xiμ\)\),γiσ=e​x​p​\(fσ​\(xiσ\)\)∑ie​x​p​\(fσ​\(xiσ\)\)\\gamma^\{\\mu\}\_\{i\}=\\frac\{exp\(f\_\{\\mu\}\(x^\{\\mu\}\_\{i\}\)\)\}\{\\sum\_\{i\}exp\(f\_\{\\mu\}\(x^\{\\mu\}\_\{i\}\)\)\},\\gamma^\{\\sigma\}\_\{i\}=\\frac\{exp\(f\_\{\\sigma\}\(x^\{\\sigma\}\_\{i\}\)\)\}\{\\sum\_\{i\}exp\(f\_\{\\sigma\}\(x^\{\\sigma\}\_\{i\}\)\)\}wherefμ​\(⋅,⋅\)f\_\{\\mu\}\(\\cdot,\\cdot\)andfσ​\(⋅,⋅\)f\_\{\\sigma\}\(\\cdot,\\cdot\)are two two\-layer FFNs with output dimension as 1 to derive the weights and a Softmax function is used for normalization\. At the end, we aggregate the road segment embeddings with the normalized weights to be the path distributional representation byμp=∑i=1\|p\|γiμ​xiμ\\mu\_\{p\}=\\sum^\{\|p\|\}\_\{i=1\}\\gamma^\{\\mu\}\_\{i\}x^\{\\mu\}\_\{i\}andσp=∑i=1\|p\|γiσ​xiσ\\sigma\_\{p\}=\\sum^\{\|p\|\}\_\{i=1\}\\gamma^\{\\sigma\}\_\{i\}x^\{\\sigma\}\_\{i\}\. Next, the path distributional representation,N⁡\(μp,σp2​I\)N\(\\mu\_\{p\},\{\\sigma\_\{p\}\}^\{2\}I\), is fed to an On\-Path Sequence Generator to generate possible OP\-Seqs onpp\.

### 4\.3\.On\-path Sequence Generator

As shown in Figure[6](https://arxiv.org/html/2609.03427#S4.F6), the OP\-Seq Generator generates a set of possible OP\-Seqs fromN⁡\(μp,σp2​I\)N\(\\mu\_\{p\},\{\\sigma\_\{p\}\}^\{2\}I\)while recoveringpp\. To generate OP\-Seqs,nnpoints, i\.e\.,\{sj,j=1,2,…,n\}\\\{s\_\{j\},j=1,2,\.\.\.,n\\\}, are sampled222To facilitate training, we follow VAE to exploit a reparameterization trick\.fromN⁡\(μp,σp2​I\)N\(\\mu\_\{p\},\{\\sigma\_\{p\}\}^\{2\}I\)\. Thensjs\_\{j\}is fed to a Long Short Term Memory \(LSTM\), a commonly used sequential data generator, to jointly generate the travel time spent on each road segment \(marked as blue in Figure[6](https://arxiv.org/html/2609.03427#S4.F6)\) and predict the next road segments one by one\. At time point 0,r0r\_\{0\}, which is always a START road segment telling the LSTM to start the generation, is fed to the LSTM to predict the first road segment, i\.e\.,r1′r^\{\\prime\}\_\{1\}\. Then at time stepii, LSTM takesrir\_\{i\}as the input and outputoio\_\{i\}as a summarization of the partial path\{r1,r2,…,ri\}\\\{r\_\{1\},r\_\{2\},\.\.\.,r\_\{i\}\\\}\.oio\_\{i\}is then fed to two FFNs, respectively, to generate: 1\)t′rij\{t^\{\\prime\}\}^\{j\}\_\{r\_\{i\}\}, i\.e\., the travel time spent onrir\_\{i\}; and ii\) the probability of each road segment to be the next road segment with a softmax function applied\. As a special case, we do not care aboutt′r0j\{t^\{\\prime\}\}^\{j\}\_\{r\_\{0\}\}since it is not a meaningful travel time on any road segment\. Meanwhile, we do maximize the predicted probability of the END road segment to ber\|p\|\+1′r^\{\\prime\}\_\{\|p\|\+1\}\.

![Refer to caption](https://arxiv.org/html/2609.03427v1/on-path-sequence-generator.png)Figure 6\.Structure of On\-path Sequence Generator
### 4\.4\.Loss Design in On\-path Sequence Evaluator

In this section, we detail the design of our loss function in the OP\-Seq Evaluator\. Note that the output of OP\-Seq Generator includes two parts: i\) the generated OP\-Seqs, i\.e\.,\{Tj′\|j=1,2,…,n\}\\\{T^\{\\prime\}\_\{j\}\|j=1,2,\.\.\.,n\\\}wherennis the number of sample points; and ii\) the predicted probability distribution of the next road segments\. We design loss functions for them respectively\.

Loss for OP\-Seqs Generation\.Given an input pathpp, we letSp′=\{t′p1,…,t′pn\}\{S\}^\{\\prime\}\_\{p\}=\\\{\{t^\{\\prime\}\}^\{1\}\_\{p\},\.\.\.,\{t^\{\\prime\}\}^\{n\}\_\{p\}\\\}denote the travel times spent on thenngenerated OP\-Seqs, i\.e\.,t′pj=∑i=1\|p\|t′rij\{t^\{\\prime\}\}^\{j\}\_\{p\}=\\sum^\{\|p\|\}\_\{i=1\}\{t^\{\\prime\}\}^\{j\}\_\{r\_\{i\}\}\. Suppose thatpphasmmhistorical OP\-Seqs\. Let the set of travel times spent on these historical OP\-Seqs be denoted asSp=\{tp1,tp2,…,tpm\}S\_\{p\}=\\\{t^\{1\}\_\{p\},t^\{2\}\_\{p\},\.\.\.,t^\{m\}\_\{p\}\\\}\. To measure the distance betweenSp′S^\{\\prime\}\_\{p\}andSpS\_\{p\}, we exploit the two\-sample Kolmogorov–Smirnov \(K\-S\) test, which tests the equality of two continuous one\-dimensional probability distributions by measuring the distance, calledK\-S distance, between two sets of values sampled from the two distributions respectively\. Thus in TraveL, we propose to exploit the K\-S distance as the loss function for OP\-Seq generation, i\.e\.,

\(3\)ℒP​a​t​h​T​i​m​e​\(θ,p\)=K​S​\(Sp′​\(θ\),Sp\)\\mathcal\{L\}\_\{PathTime\}\(\\theta,p\)=KS\(S^\{\\prime\}\_\{p\}\(\\theta\),S\_\{p\}\)whereθ\\thetapoints to the TraveL model parameters andSp′​\(θ\)S^\{\\prime\}\_\{p\}\(\\theta\)is theSp′S^\{\\prime\}\_\{p\}generated withθ\\theta, and KS\(⋅,⋅\\cdot,\\cdot\) is the K\-S distance function \(see Section 5\.8\)\. Data Sparsity Issue\.We observe the data sparsity issue in collected real\-world data: the number of historical OP\-Seqs on a pathpp, i\.e\.,\|Sp\|\|S\_\{p\}\|, may be so small, that there may be a large bias between the observedSpS\_\{p\}and the real travel time distribution onpp, rendering the K\-S distance defined above not working\. To address the issue, we propose to apply a regularization on the travel time generation, by asking the generated travel time distribution on each road segmentri∈pr\_\{i\}\\in pto also approach the historical travel time distribution onrir\_\{i\}\. Denote the set of generated travel time onrir\_\{i\}asSri′\(θ\)=\{t′rij\|j=1,2,…,n\}S^\{\\prime\}\_\{r\_\{i\}\}\(\\theta\)=\\\{\{t^\{\\prime\}\}^\{j\}\_\{r\_\{i\}\}\|j=1,2,\.\.\.,n\\\}with TraveL model parametersθ\\theta, and the set of historical travel time \(not necessary to be onpp\) onrir\_\{i\}asSriS\_\{r\_\{i\}\}\. We define the loss for road segment travel time generation as the K\-S distance betweenSri′S^\{\\prime\}\_\{r\_\{i\}\}andSriS\_\{r\_\{i\}\}, i\.e\.,ℒ⁡\(θ,ri\)=K​S​\(Sri′​\(θ\),Sri\)\\mathcal\{L\}\(\\theta,r\_\{i\}\)=KS\(S^\{\\prime\}\_\{r\_\{i\}\}\(\\theta\),S\_\{r\_\{i\}\}\)\. We sum the loss for all road segments in pathppas follows\.

\(4\)ℒR​s​T​i​m​e​\(θ,p\)=∑ri∈pℒ⁡\(θ,ri\)=∑ri∈pK​S​\(Sri′​\(θ\),Sri\)\\small\\mathcal\{L\}\_\{RsTime\}\(\\theta,p\)=\\sum\_\{r\_\{i\}\\in p\}\\mathcal\{L\}\(\\theta,r\_\{i\}\)=\\sum\_\{r\_\{i\}\\in p\}KS\(S^\{\\prime\}\_\{r\_\{i\}\}\(\\theta\),S\_\{r\_\{i\}\}\)which acts as a regularization term to supplementℒP​a​t​h​T​i​m​e​\(θ,p\)\\mathcal\{L\}\_\{PathTime\}\(\\theta,p\)\.

Loss for the Next Road Segment Prediction\.In addition to evaluation on the generated OP\-Seqs, we also evaluate the prediction of the next road segment\. We treat the next road segment prediction task as a classification problem, and train the model by maximizing the probability of the ground truth next road segment to be the next road segment at each time step of prediction\. Thus, we explore a maximum likelihood based loss function for the next road segment prediction as follows\.

\(5\)ℒR​s​P​r​e​d\(θ,p\)=−∑j=1n∑i=0\|p\|logP\(ri\+1\|sj,r0,r1,…,ri,θ\)\\mathcal\{L\}\_\{RsPred\}\(\\theta,p\)=\-\\sum^\{n\}\_\{j=1\}\\sum^\{\|p\|\}\_\{i=0\}logP\(r\_\{i\+1\}\|s\_\{j\},r\_\{0\},r\_\{1\},\.\.\.,r\_\{i\},\\theta\)
Finally, we regularize the learned distributional representationNN\(μp,σp2​I\\mu\_\{p\},\\sigma^\{2\}\_\{p\}I\) withOPENℒP​a​t​h​R​e​p​\(θ,p\)=K​L​\(N⁡\(μp,σp2​I\),N⁡\(0,I\)\)\)\\mathcal\{L\}\_\{PathRep\}\(\\theta,p\)=KL\(N\(\\mu\_\{p\},\\sigma^\{2\}\_\{p\}I\),N\(0,I\)\)\), and the model parametersθ\\thetawith a two\-norm regularization\. We sum these losses for all paths in the datasetDDto get the final loss function as follows\.

\(6\)ℒ⁡\(θ\)=1\|D\|​∑p∈D\(λ1​ℒP​a​t​h​T​i​m​e​\(θ,p\)\+λ2​ℒR​s​T​i​m​e​\(θ,p\)CLOSEOPEN\+λ3​ℒR​s​P​r​e​d​\(θ,p\)\+λ4​ℒP​a​t​h​R​e​p​\(θ,p\)\)\+λ5​\|\|θ\|\|22\\small\\begin\{split\}\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{\|D\|\}\\sum\_\{p\\in D\}\(\\lambda\_\{1\}\\mathcal\{L\}\_\{PathTime\}\(\\theta,p\)\+\\lambda\_\{2\}\\mathcal\{L\}\_\{RsTime\}\(\\theta,p\)\\\\ \+\\lambda\_\{3\}\\mathcal\{L\}\_\{RsPred\}\(\\theta,p\)\+\\lambda\_\{4\}\\mathcal\{L\}\_\{PathRep\}\(\\theta,p\)\)\+\\lambda\_\{5\}\{\\lvert\\lvert\\theta\\rvert\\rvert\}^\{2\}\_\{2\}\\end\{split\}where\|D\|\|D\|is the dataset size and∑i=13λi=1\\sum^\{3\}\_\{i=1\}\\lambda\_\{i\}=1to weight the losses\.

## 5\.Performance Evaluation

In the following, we introduce the datasets, baseline models for comparison, and a typical representation learning process\. Then we evaluate TraveL distributional representations against baselines on three tasks, i\.e\., travel time distribution estimation, path similarity prediction and destination prediction, for all the datasets\.

### 5\.1\.Datasets

We collect two commonly used real\-world trajectory datasets, Porto and Tokyo, and generate a synthetic dataset, Syn\-Porto, on the map of Porto for the evaluation\. We briefly introduce the datasets below\.

Portocollects 1\.7 minion taxi trajectories of 442 taxis in Porto, Portugal from January 2013 to June 2014333http://www\. geolink\.pt/ecmlpkdd2015\-challenge/\.\.

Tokyocollects 78 million GPS sample points from 617K users walking or taking vehicles in Tokyo\([Kashiyama et al\., 2017](https://arxiv.org/html/2609.03427#bib.bib9)\)\. We extract the GPS point sequences for users taking bike or car, and then segment the sequences into trajectories with a 45\-second gap\.

Syn\-Portois the synthetic dataset including 200 million OP\-Seqs of moving taxis in Porto, generated by a simulation system\. Because that we have no information about the real travel time distribution for synthesis, we create three versions of Syn\-Porto, denoted asDn​o​rD\_\{nor\},Dl​o​g​nD\_\{logn\}andDm​i​xD\_\{mix\}respectively, with road segment travel times sampled from a Normal distribution, a Log\-normal distribution, or a mixture of them\. The three versions are used to prove the consistent superiority of TraveL under various assumptions\.

For Porto and Tokyo, we remove trajectories with less than 10 GPS sample points \(too short\) and finally obtain 1\.2 million and 0\.29 million trajectories, respectively\. To obtain OP\-Seqs, we adopt Barefoot444Barefoot can be found at https://github\.com/bmwcarit/barefoot/\., a Hidden Markov Model based map matching tool, to project a trajectory onto a road network to yield an OP\-Seq\. We further exploit the three\-sigma principle to remove OP\-Seqs with outlier road segment travel time\.

### 5\.2\.Baseline Models

In this section, we introduce the state\-of\-the\-art path representation learning models compared in the evaluation\.

Node2Veclearns node presentations in a graph\([Grover and Leskovec, 2016](https://arxiv.org/html/2609.03427#bib.bib8)\)\. We average the representations of nodes along a path as the path representation\.

RoadSegment \(RS\)feeds the sequence of road segments in the path to a seq2seq model, where the encoder and decoder are both a single layer LSTM model, to generate the path representation\.

InfoGraphis an unsupervised model to learn a representation for the whole graph\([Sun et al\., 2019](https://arxiv.org/html/2609.03427#bib.bib20)\)\. In this work, we regard a path as a special graph and learn the path representation with InfoGraph\.

BERTis an unsupervised language representation learning model\([Devlin et al\., 2018](https://arxiv.org/html/2609.03427#bib.bib3)\)\. We treat a road segment as a word and a path as a sentence\. To enable training, we split a path P into two sub\-paths P1 and P2 \(with equal number of road segments\), and consider \(P1, P2\) as a valid pair of two continuous sentences and \(P2, P1\) as invalid because the latter does not form a real path in the road network\.

Path InfoMax \(PIM\)is an unsupervised path representation learning framework which first generates negative samples with curriculum negative sampling, and then employs mutual information maximization to learn path representations\([Yang et al\., 2021](https://arxiv.org/html/2609.03427#bib.bib24)\)\.

Trembris a state\-of\-the\-art trajectory representation learning model which first maps a trajectory to a path on road network to learn the trajectory representation\([Fu and Lee, 2020](https://arxiv.org/html/2609.03427#bib.bib5)\)\. To learn a trajectory representation, Trembr feeds the sequence of road segments with pre\-trained embeddings into a seq2seq model with a special designed loss function to distinguish relationships among road segments\. In this work, we adapt Trembr to learn the path representation\.

TraveLfeeds the sequence of road segments into the proposed TraveL framework to generate path distributional representations\.

### 5\.3\.Representation Learning Process

Given a datasetDDconsisting of a set of paths and their on\-path sequences, we randomly split the set of all paths into 90% and 10% as training and validation set, respectively\. We train the TraveL model with the paths in the training set and choose the optimal model parameters which achieve the lowest loss on the validation set\. The initial learning rate is set to 0\.0001 and we use Adam optimizer for training\. We further tune the model hyper\-parameters to minimize the loss\. The optimal hyper\-parameter setting is detailed in Section 5\.9\. We also tune the best parameter settings for all baseline models\. The dimensionality of path representations learned by baseline models is finally set as 256\.

### 5\.4\.Travel Time Distribution Estimation \(TTDE\)

To evaluate all the PRL models for the TTDE task, we first learn the path representations for each model following the training process in Section 5\.3\. For each dataset, once we learn the path representations, we randomly split all the paths, associated with their learned representations, into 80%, 10% and 10%, as the training, validation and testing set, respectively, for the TTDE task\. For each path in the training set, we randomly sample a set of points from its distributional representation, and feed these points to a two\-layer FFN to generate a set of travel times, respectively\. We train the FFN by minimizing the K\-S distance between the generated travel times and the ground truth, with the validation sets to select the optimal FFN\. Finally, we evaluate the TraveL representations on the testing set with K\-S distance between the generated \(with the trained FFN\) and ground\-truth travel time set as the metric\. However, one issue is that, for baseline models, each path representation is a vector, which cannot be used for sampling\. To fill the gap, we train another two FFNs to first map the vector to a pair of 128\-dim mean vector and standard deviation vector, and then follow the sampling process introduced above to train all FFNs with the K\-S loss end to end\.

For Porto and Tokyo, due to the data sparsity issue, we evaluate the representations of sub\-paths \(of a fixed lengthLL\), which have sufficient OP\-Seqs in the real\-world datasets\. We collect the sub\-paths and their OP\-Seqs, and train all models to obtain sub\-path representations following the representation learning process introduced in Section 5\.3\. Then we evaluate sub\-path representations, in a same way as full path representations\. We varyLLas 5, 10, 15 and ‘Full’ \(where we learn representations for full paths\) to evaluate the impact ofLLto the model performance\.

Table 1\.Travel Time Distribution Estimation on Syn\-PortoDm​i​xD\_\{mix\}Dn​o​rD\_\{nor\}Dl​o​g​nD\_\{logn\}ModelShortMidLongShortMidLongShortMidLongNode2Vec0\.310\.340\.380\.280\.330\.370\.270\.300\.36RS0\.320\.360\.390\.300\.340\.380\.290\.320\.38BERT0\.380\.400\.420\.360\.390\.410\.360\.380\.40InfoGraph0\.350\.370\.400\.330\.360\.420\.320\.340\.39PIM0\.290\.320\.35∗0\.26∗0\.300\.350\.260\.290\.35Trembr0\.28∗0\.31∗0\.35∗0\.26∗0\.28∗0\.33∗0\.25∗0\.28∗0\.34∗TraveL0\.240\.290\.330\.230\.270\.310\.230\.280\.29

Evaluation of ModelsTable[1](https://arxiv.org/html/2609.03427#S5.T1)shows the Mean K\-S distance \(MKS\) achieved by different models on the three synthetic datasets\. To analyze the impact of the path length to the model performance, we categorize all paths to Short \(<3km\), Mid \(3km\-6km\), or Long \(\>6km\), and show the MKS achieved for paths of each category\. As shown, TraveL outperforms all the baselines consistently, for various lengths on all three synthetic datasets, by reducing MKS from 3\.57% \(0\.27 v\.s\. 0\.28 by Trembr onDn​o​rD\_\{nor\}\) to 36\.1% \(0\.23 v\.s\. 0\.36 by BERT onDl​o​g​nD\_\{logn\}\)\. We observe that model performance onDm​i​xD\_\{mix\}is worse thanDn​o​rD\_\{nor\}andDl​o​g​nD\_\{logn\}, which fits our intuition that a more complicated travel time distribution is harder to predict\. Among the baselines, we observe that PIM and Trembr outperform others because PIM distinguishes the co\-occurrence relationships among nodes and paths, and Trembr captures same\-type relationship among the road segments as well into the pre\-trained road segment embeddings\.

In addition, we observe all models perform worse \(i\.e\., achieve higher MKS\) for longer paths\. An interesting observation is that, for most models, the increase of MKS from Mid to Long is larger than that from Short to Mid \(e\.g\., MKS by Trembr increases 0\.05 from Mid to Long v\.s\. 0\.02 from Short to Mid\), which indicates that these models get worse more rapidly for longer paths\. The challenge for long path travel time distribution prediction comes from both the dependency capturing for road segments in a longer distance and the more complicated travel time distribution \(since it is a sum of more road segments’ travel time distribution\)\. Instead, MKS by TraveL increases 0\.04 onDm​i​xD\_\{mix\}from Mid to Long and 0\.05 from Short to Mid, which indicates that TraveL works well for learning representations for long paths owing to its ability to capture long\-term dependencies and the varied traveler behaviors\.

![Refer to caption](https://arxiv.org/html/2609.03427v1/Figures/real-travel-time-estimation.PNG)Figure 7\.Travel Time Estimation on Porto and TokyoOn Porto and Tokyo, we observe that TraveL also outperforms the baseline models for sub\-paths of all lengths \(shown in Figure[7](https://arxiv.org/html/2609.03427#S5.F7)\)\. As shown, all models perform worse with a larger path length \(LL\), which suggests the challenge brought by long paths\. We observe that, BERT, an baseline that also exploits the Transformer structure to capture the long\-term dependencies, does not work as well as TraveL\. The reason may be that TraveL also captures the regional correlation with the regional attention mechanism and the varied traveler behaviors, which are both essential for the TTDE task\.

### 5\.5\.Path Similarity Prediction \(PSP\)

Here we learn path representations with the representation learning process introduced in Section 5\.3 for the PSP task\. The learned path representations are believed to capture rich information if they can be used to predict various kinds of path similarity\. In this experiment, we target on the following two similarities: i\) the weighted Jaccard similarity \(WJ\), a commonly used path similarity function, which is the ratio of the shared road segments’ length to the total length of the two paths; and ii\) the speed\-relevant weighted Jaccard similarity \(SWJ\) we propose, which is the ratio of the expected travel time spent on the shared road segments to the expected total travel time spent on the two paths\. We define SWJ between two paths,p\(1\)p^\{\(1\)\}andp\(2\)p^\{\(2\)\}, as follows\.

\(7\)S​W​J​\(p\(1\),p\(2\)\)=∑r∈p\(1\)1\[r∈p\(2\)\]ET\(r\)∑r∈p\(1\)E​T​\(r\)\+∑r∈p\(2\)E​T​\(r\)SWJ\(p^\{\(1\)\},p^\{\(2\)\}\)=\\frac\{\\sum\_\{r\\in p^\{\(1\)\}\}\{\\textbf\{1\}\_\{\[r\\in p^\{\(2\)\}\]\}ET\(r\)\}\}\{\\sum\_\{r\\in p^\{\(1\)\}\}ET\(r\)\+\\sum\_\{r\\in p^\{\(2\)\}\}ET\(r\)\}where1\[r∈p\(2\)\]\\textbf\{1\}\_\{\[r\\in p^\{\(2\)\}\]\}is an indicator function which equals to 1 whenr∈p\(2\)r\\in p^\{\(2\)\}and equals to 0 otherwise,E​T​\(r\)ET\(r\)is the expected travel time onrr, i\.e\., the length ofrrdivided by the speed limit onrr\. Intuitively, predicting SWJ is more challenging \(than WJ\), since it requires to capture speed\-relevant information in addition to lengths\.

Next, we generate a datasetDp​pD\_\{pp\}consisting of path pairs, from each path dataset\. For Syn\-Porto, given a pathpp, we follow an existing literature, PathRank, to collect candidate paths with the same origin and destination aspp, which are then paired withpp\. For Porto and Tokyo, a path may not have any candidate paths due to the limited number of paths\. Thus we keep randomly selecting two paths and put them intoDp​pD\_\{pp\}if their WJ or SWJ is larger than 0\.5, and we finally collect 1 million path pairs\. Note that inDp​pD\_\{pp\}, we remove all the departure time information from each path before PRL, so that the evaluation of the similarity prediction focuses on how well the road segments static information and correlations are captured\. After the PRL process withDp​pD\_\{pp\}, we concatenate the representations of two paired paths and train a linear regression \(LR\) model to predict the WJ or SWJ similarity between the two paths with the concatenated representation\. Specifically, for TraveL, we concatenateμp\\mu\_\{p\}andσp\\sigma\_\{p\}to form a vector representation ofpp, which is then fed to the LR model to facilitate training\. Finally, we exploit Mean Absolute Error \(MAE\) and Mean Absolute Percentage Error \(MAPE\), between the predicted and the ground truth similarity scores of the path pairs for testing, as the evaluation metrics\. In addition, for Syn\-Porto, we evaluate the path representations by ranking all candidate paths, for a given path, in terms of the similarity, and use Kendall rank correlation coefficient \(denoted byτ\\tau\) and Spearman’s rank correlation coefficient \(denoted byρ\\rho\) to measure the consistency between the ranking derived by the predicted similarity scores and the ranking derived by the ground truth similarity scores\. Higherτ\\tauandρ\\rhoindicate higher accuracy\.

Table 2\.Path Similarity Prediction onDm​i​xD\_\{mix\}Weighted JaccardSpeed\-relevant Weighted JaccardModelMAEMAPEτ\\tauρ\\rhoMAEMAPEτ\\tauρ\\rhoNode2Vec0\.1830\.10\.620\.660\.2540\.40\.520\.58RS0\.1624\.80\.660\.730\.2336\.60\.580\.64BERT0\.2438\.70\.500\.530\.2744\.90\.440\.48InfoGraph0\.1729\.50\.650\.690\.2134\.80\.500\.55PIM0\.12∗17\.9∗0\.73∗0\.77∗0\.1624\.10\.640\.69Trembr0\.1319\.80\.720\.750\.15∗21\.3∗0\.66∗0\.70∗TraveL0\.1014\.40\.760\.790\.1115\.10\.700\.74

Evaluation of Models\.As shown in Table[2](https://arxiv.org/html/2609.03427#S5.T2), all PRL models perform worse for SWJ prediction onDm​i​xD\_\{mix\}, which validates our intuition that SWJ is more challenging to predict owing to its requirement to capture speed\-relevant information\. By comparing Trembr with PIM, we observe that Trembr loses for WJ prediction, while wins for SWJ prediction\. The reason may be that Trembr captures the same\-type relationship \(which, e\.g\., the road segment is on highway, may indicate the speeds\) into road segment embeddings to help SWJ prediction\. Finally, owing to the ability to capture varied traveler behaviors, i\.e\., the varied travel times \(which is relevant to speeds\), TraveL performs the best among all PRL models by reducing MAE by 16\.7% \(0\.10 v\.s\. 0\.12 by PIM\) for WJ, and MAE by 26\.7% \(0\.11 v\.s\. 0\.15 by Trembr\) for SWJ\. The results for MAPE,τ\\tauandρ\\rhoalso suggest similar insights as MAE\. Besides, we have similar observations from the evaluation results on both Porto and Tokyo \(as shown in Table[3](https://arxiv.org/html/2609.03427#S5.T3)\), which validates our ideas again\.

Table 3\.Path Similarity Prediction on Real\-world Datasets\-Porto\-WJPorto\-SWJTokyo\-WJTokyo\-SWJModelMAEMAPEMAEMAPEMAEMAPEMAEMAPENode2Vec0\.1930\.10\.2133\.00\.1726\.60\.2031\.2RS0\.2132\.00\.2235\.40\.1524\.80\.1727\.1BERT0\.2336\.20\.2741\.90\.2235\.30\.2640\.3InfoGraph0\.2235\.70\.2537\.70\.2032\.10\.3539\.0PIM0\.1728\.10\.1931\.80\.1422\.30\.2628\.5Trembr0\.15∗24\.6∗0\.18∗28\.6∗0\.13∗20\.4∗0\.16∗24\.9∗TraveL0\.1319\.90\.1421\.90\.1217\.50\.1320\.8

### 5\.6\.Destination Prediction \(DP\)

Here we evaluate all PRL models for the DP task by exploiting a partial path’s representation to predict the path destination, i\.e\., the coordinate of the end of the last road segment in the synthetic dataset or the last coordinate generated by map matching for real\-world datasets\. For each dataset, given a pathpp, we generate a partial path starting frompp’s origin with length as⌊δ×\|p\|⌋\\lfloor\\delta\\times\|p\|\\rfloor, whereδ\\deltais the ratio to control the partial path length\. We train all PRL models to obtain the representation for each partial path\. Similar to the evaluation process of PSP, we split all partial paths as 90% and 10% as the training and testing set respectively, and train two LR models to map the partial path representation to the latitude and longitude of the destination respectively, with the training set\. On the testing set, we calculate the MAE of the geographic distance between the predicted coordinate and the destination in kilometers as the metric for DP\. Finally, we varyδ\\deltaas 0\.5, 0\.625, 0\.75 and 0\.875 to evaluate its impacts on the model performance\.

Figure[8](https://arxiv.org/html/2609.03427#S5.F8)shows the results of DP on all three datasets\. We have two main observations: i\) asδ\\deltaincreases, all models achieve a significantly lower MAE, which suggests the decrease of the difficulty of the task; and ii\) Although DP is intuitively the least relevant, among the three targeted tasks, to the travel time information, TraveL \(i\.e\., the green line\) still outperforms other baseline models consistently on all the three datasets, by reducing MAE \(whenδ=0\.75\\delta=0\.75\) from 3\.97% \(i\.e\., 1\.69 v\.s\. 1\.76 by Trembr on Tokyo\) to 10\.6% \(0\.67 v\.s\. 0\.75 by Trembr on Porto\), which indicates the importance of capturing the regional correlations and the varied traveler behaviors for PRL\.

![Refer to caption](https://arxiv.org/html/2609.03427v1/Figures/destination-prediction.PNG)Figure 8\.Evaluation of Destination Prediction
### 5\.7\.Ablation Study

In this section, we perform an ablation study to assess the impacts of various components in TraveL on its performance\. We consider the following variants of TraveL by removing some components from TraveL framework\. i\)Complete: the complete version of TraveL; ii\),NoViews: removing all the multi\-view attention layers from Path Encoder; iii\)NoHW, removing Highway\-view attention; iv\)NoLane: removing Lane\-view attention; v\)NoHop: removing Hop\-view attention; vi\)NoDistr: replacing the distribution representation by a vector representation; and vii\)OnlyPath: with OP\-Seq Generator predicting the next road segments without generating travel times\.

Table[4](https://arxiv.org/html/2609.03427#S5.T4)shows the results of the ablation study on the three datasets for all three tasks\. We observe thatCompleteoutperforms all variants on the PSP task\. BetweenOnlyPathandNoViews, we observe the former has more performance deterioration againstCompletefor PSP than the latter, indicating that capturing varied traveler behaviors is more important than capturing the regional correlation for PSP\. AmongNoViews,NoHW,NoLaneandNoHop, we observe thatNoViewsalways performs the worst, which indicates that all these three views benefit PRL in a complementary way\. Besides, we observe thatNoHopperforms worse thanNoHWandNoLanefor DP, whileNoHWandNoLaneperform worse thanNoHopfor PSP, which suggests that different views may have distinguished benefits for different tasks\. Finally, we observe thatNoDistrperforms the worst \(for TTDE and DP\) and the second worst \(for PSP\) among all variants, validating our idea that a distributional representation has larger capacity than a vector, which is essential to help capture the varied traveler behaviors for PRL\.

Table 4\.Ablation Study of TraveLTTDE \(MKS\)PSP \(MAE\)DP \(MAE\)ModelDm​i​xD\_\{mix\}PortoTokyoDm​i​xD\_\{mix\}PortoTokyoDm​i​xD\_\{mix\}PortoTokyoComplete0\.290\.320\.280\.110\.140\.131\.060\.841\.88NoViews0\.310\.330\.300\.140\.170\.141\.150\.881\.94NoHW0\.300\.320\.290\.140\.150\.131\.100\.861\.92NoLane0\.310\.330\.300\.140\.150\.141\.080\.841\.89NoHop0\.300\.330\.300\.120\.140\.131\.120\.851\.90NoDistr0\.320\.340\.330\.140\.170\.161\.160\.871\.93OnlyPath0\.310\.340\.310\.130\.180\.171\.090\.891\.91

### 5\.8\.Details of Kolmogorov–Smirnov Test

Kolmogorov–Smirnov Test is a classic test of the equality of two continuous one\-dimensional probability distributions based on two sets of values sampled from the two distributions respectively\. For example, given the generated travel time set \(Sp′S^\{\\prime\}\_\{p\}\) and the historical travel time set \(SpS\_\{p\}\), K\-S test transforms each set to an empirical distribution function, i\.e\.,Fn​\(x\)=1n​∑i=1n1\[−∞,x\]​\(Xi\)F\_\{n\}\(x\)=\\frac\{1\}\{n\}\\sum^\{n\}\_\{i=1\}\{\\textbf\{1\}\_\{\[\-\\infty,x\]\}\(X\_\{i\}\)\}, whereXiX\_\{i\}is theii\-th smallest value inSp′S^\{\\prime\}\_\{p\},1\[−∞,x\]​\(Xi\)\\textbf\{1\}\_\{\[\-\\infty,x\]\}\(X\_\{i\}\)is the indicator function, equal to 1 ifXi≤xX\_\{i\}\\leq xand equal to 0 otherwise\. And similarly, we obtainFm​\(x\)F\_\{m\}\(x\)fromSpS\_\{p\}\. Then K\-S test defines the K\-S distance between the two sample sets asK​S​\(Sp′,Sp\)=supx\|Fn​\(x\)−Fm​\(x\)\|KS\(S^\{\\prime\}\_\{p\},S\_\{p\}\)=\{\\sup\}\_\{x\}\|F\_\{n\}\(x\)\-F\_\{m\}\(x\)\|, which is then exploited to judge whether the two sample sets are from a same distribution\.

### 5\.9\.Hyper\-parameter Settings

In this section, we detail the optimal hyper\-parameter setting of TraveL: i\) the number of multi\-view regional attention layers \(denoted asLL\) in Multi\-view Path Transformer as 6; ii\) the number of hops in Hop\-view as 3; iii\) the dimensionality ofμp\\mu\_\{p\}andσp\\sigma\_\{p\}as 128; and vi\) the number of sample points from the distributional representation as 100\. For Syn\-porto, we haveλ1\\lambda\_\{1\}as 0\.3,λ2\\lambda\_\{2\}as 0\.0 \(since synthetic data does not face the data sparsity issue\),λ3\\lambda\_\{3\}as 0\.7,λ4\\lambda\_\{4\}as 0\.1 andλ5\\lambda\_\{5\}as 0\.05 to weight the loss\. For Porto and Tokyo, we haveλ1\\lambda\_\{1\}as 0\.1,λ2\\lambda\_\{2\}as 0\.3,λ3\\lambda\_\{3\}as 0\.6,λ4\\lambda\_\{4\}as 0\.1 andλ5\\lambda\_\{5\}as 0\.05\. We conduct parameter sensitivity tests to evaluate the impacts of these parameters\. We observe that increasingLLfrom 6 to 8 leads to a worse performance of TraveL on all three tasks on Porto and Tokyo, but better performances onDm​i​xD\_\{mix\}\. The reason may be that the larger amount of synthetic data \(than those of the real\-world datasets\) supports training a deeper neural network\. Other details are not shown due to the space limit\.

## 6\.Conclusion

We propose the TraveL framework for path representation learning in the road network, where the idea of distributional representation is explored, together with a sampling based On\-path Sequence Generator, to capture varied traveler behaviors on the path, and a multi\-view regional attention is developed to capture various correlations within regions of road segments\. We explore the idea of K\-S test to facilitate model training and evaluation\. Empirically, we demonstrate the superiority of TraveL to the state of the arts\. As for our next step, we plan to explore distributional representations in other forms to capture other types of traveler behaviors for PRL\.

## References

- Chen et al\.\(2020\)Yen\-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu\. 2020\.Uniter: Universal image\-text representation learning\. In*European conference on computer vision*\. Springer, 104–120\.
- Devlin et al\.\(2018\)Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\. 2018\.Bert: Pre\-training of deep bidirectional transformers for language understanding\.*arXiv preprint arXiv:1810\.04805*\(2018\)\.
- Dosovitskiy et al\.\(2020\)Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al\.2020\.An image is worth 16x16 words: Transformers for image recognition at scale\.*arXiv preprint arXiv:2010\.11929*\(2020\)\.
- Fu and Lee \(2020\)Tao\-Yang Fu and Wang\-Chien Lee\. 2020\.TremBR: Exploring road networks for trajectory representation learning\.*ACM Transactions on Intelligent Systems and Technology \(TIST\)*11, 1 \(2020\), 1–25\.
- Fu et al\.\(2017\)Tao\-yang Fu, Wang\-Chien Lee, and Zhen Lei\. 2017\.Hin2vec: Explore meta\-paths in heterogeneous information networks for representation learning\. In*Proceedings of the 2017 ACM on Conference on Information and Knowledge Management*\. 1797–1806\.
- Goyal et al\.\(2020\)Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo\. 2020\.dyngraph2vec: Capturing network dynamics using dynamic graph representation learning\.*Knowledge\-Based Systems*187 \(2020\), 104816\.
- Grover and Leskovec \(2016\)Aditya Grover and Jure Leskovec\. 2016\.node2vec: Scalable feature learning for networks\. In*Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining*\. 855–864\.
- Kashiyama et al\.\(2017\)Takehiro Kashiyama, Yanbo Pang, and Yoshihide Sekimoto\. 2017\.Open PFLOW: Creation and evaluation of an open dataset for typical people mass movement in urban areas\.*Transportation research part C: emerging technologies*85 \(2017\), 249–267\.
- Kingma and Welling \(2013\)Diederik P Kingma and Max Welling\. 2013\.Auto\-encoding variational bayes\.*arXiv preprint arXiv:1312\.6114*\(2013\)\.
- Le and Mikolov \(2014\)Quoc Le and Tomas Mikolov\. 2014\.Distributed representations of sentences and documents\. In*International conference on machine learning*\. PMLR, 1188–1196\.
- Lee et al\.\(2017\)Hsin\-Ying Lee, Jia\-Bin Huang, Maneesh Singh, and Ming\-Hsuan Yang\. 2017\.Unsupervised representation learning by sorting sequences\. In*Proceedings of the IEEE international conference on computer vision*\. 667–676\.
- Li et al\.\(2017\)Cheng Li, Jiaqi Ma, Xiaoxiao Guo, and Qiaozhu Mei\. 2017\.Deepcas: An end\-to\-end predictor of information cascades\. In*Proceedings of the 26th international conference on World Wide Web*\. 577–586\.
- Liu et al\.\(2017\)Zemin Liu, Vincent W Zheng, Zhou Zhao, Fanwei Zhu, Kevin Chen\-Chuan Chang, Minghui Wu, and Jing Ying\. 2017\.Semantic proximity search on heterogeneous graph by proximity embedding\. In*Proceedings of the AAAI Conference on Artificial Intelligence*, Vol\. 31\.
- Massey Jr \(1951\)Frank J Massey Jr\. 1951\.The Kolmogorov\-Smirnov test for goodness of fit\.*Journal of the American statistical Association*46, 253 \(1951\), 68–78\.
- Mikolov et al\.\(2013\)Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean\. 2013\.Efficient estimation of word representations in vector space\.*arXiv preprint arXiv:1301\.3781*\(2013\)\.
- Radford et al\.\(2015\)Alec Radford, Luke Metz, and Soumith Chintala\. 2015\.Unsupervised representation learning with deep convolutional generative adversarial networks\.*arXiv preprint arXiv:1511\.06434*\(2015\)\.
- Ren and Leskovec \(2020\)Hongyu Ren and Jure Leskovec\. 2020\.Beta embeddings for multi\-hop logical reasoning in knowledge graphs\.*Advances in Neural Information Processing Systems*33 \(2020\), 19716–19726\.
- Rezende et al\.\(2014\)Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra\. 2014\.Stochastic backpropagation and approximate inference in deep generative models\. In*International conference on machine learning*\. PMLR, 1278–1286\.
- Sun et al\.\(2019\)Fan\-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang\. 2019\.Infograph: Unsupervised and semi\-supervised graph\-level representation learning via mutual information maximization\.*arXiv preprint arXiv:1908\.01000*\(2019\)\.
- Vaswani et al\.\(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\. 2017\.Attention is all you need\.*Advances in neural information processing systems*30 \(2017\)\.
- Xia et al\.\(2014\)Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan\. 2014\.Supervised hashing for image retrieval via image representation learning\. In*Twenty\-eighth AAAI conference on artificial intelligence*\.
- Xie et al\.\(2016\)Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun\. 2016\.Representation learning of knowledge graphs with entity descriptions\. In*Proceedings of the AAAI Conference on Artificial Intelligence*, Vol\. 30\.
- Yang et al\.\(2021\)Sean Bin Yang, Chenjuan Guo, Jilin Hu, Jian Tang, and Bin Yang\. 2021\.Unsupervised Path Representation Learning with Curriculum Negative Sampling\.*arXiv preprint arXiv:2106\.09373*\(2021\)\.
- Yang and Yang \(2019\)Sean Bin Yang and Bin Yang\. 2019\.PathRank: A Multi\-Task Learning Framework to Rank Paths in Spatial Networks\.*arXiv preprint arXiv:1907\.04028*\(2019\)\.
- Zhang et al\.\(2017\)Ye Zhang, Matthew Lease, and Byron Wallace\. 2017\.Active discriminative text representation learning\. In*Proceedings of the AAAI Conference on Artificial Intelligence*, Vol\. 31\.