Location-Aware Language Models via Secondary Embeddings

# Location-Aware Language Models via Secondary Embeddings
Source: [https://arxiv.org/html/2609.00454](https://arxiv.org/html/2609.00454)
###### Abstract

Pretrained transformer\-based language models achieve strong performance across a wide range of NLP tasks but remain limited in encoding geo\-locational semantics, leading to suboptimal representations of place names and spatial entities\. In this work, we propose a lightweight, model\-agnostic approach for injecting geo\-spatial awareness into pretrained embeddings without modifying the tokenizer or requiring costly retraining\. Our method augments input representations with structured geographic signals by combining location names with their corresponding latitude and longitude, and employs a location\-focused masking to better align textual representations with real\-world spatial relationships\. This design allows the model to incorporate geo\-spatial context while preserving existing semantic and syntactic knowledge\. Experimental results demonstrate substantial improvements in geo\-spatial alignment while maintaining comparable performance on standard NLP benchmarks such as GLUE\. The method is computationally efficient, requiring only minutes of additional training, and generalizes across multiple model architectures and scales\.

###### Keywords:

Language models Masked language modeling \(MLM\) Geo\-locations

## 1Introduction

Recent advances in pretrained language models such as BERT\[[3](https://arxiv.org/html/2609.00454#bib.bib1)\]and RoBERTa\[[11](https://arxiv.org/html/2609.00454#bib.bib3)\]have demonstrated a remarkable ability to capture semantic and syntactic relationships between words\. Despite this success, these models often struggle with specialized domains that require additional contextual grounding\. In particular, geo\-locational awareness is essential for understanding spatial relationships and providing contextually relevant outputs such as route descriptions, targeted advertisements, and localized search results\. However, existing models largely fail to encode geographic semantics effectively for entities such as cities, landmarks, or regions\.

![Refer to caption](https://arxiv.org/html/2609.00454v1/embedding_model_comparison_tsne_v2.png)Figure 1:Comparison between the actual geographic location of cities \(left\) and their corresponding t\-SNE visualization in the BERT embedding space \(right\)\. The visualization shows that cities which are geographically distant often appear close together in the embedding space, highlighting the lack of geo\-spatial awareness in pretrained language models\.As illustrated in Figure[1](https://arxiv.org/html/2609.00454#S1.F1), embeddings of city names are not well structured in the latent space, largely due to the absence of explicit geographic grounding in language models\. For instance, the citiesSaarbrückenandSalzburgappear close to each other in BERT’s embedding space, even though they are geographically distant\. This discrepancy highlights the model’s inability to reflect real\-world spatial relationships\.

A contributing factor to this limitation lies in the tokenization process\. While subword tokenization has proven effective for general text representation, it fragments named entities, eroding their unique semantic identity\. For example, the city nameIngolstadtis split into subwords"ing","\#\#ols", and"\#\#tadt", resulting in partial embeddings that fail to capture the entity as a single coherent concept\. The issue worsens when different locations share identical names \(e\.g\., Paris, France vs\. Paris, Texas\), as the model has no geo\-spatial information to distinguish between them\. Consequently, embeddings of distinct entities often collapse into the same region of the vector space, misrepresenting their relational proximity\.

Existing approaches to incorporating geographic information often require architectural modifications or costly retraining procedures, limiting their scalability and practical applicability\[[9](https://arxiv.org/html/2609.00454#bib.bib13),[10](https://arxiv.org/html/2609.00454#bib.bib14),[4](https://arxiv.org/html/2609.00454#bib.bib15)\]\. A seemingly straightforward solution would be to expand the tokenizer’s vocabulary to include all place names\. However, this approach is impractical as it significantly increases model size and training costs\. For instance, adding merely 1,000 new tokens to the BERT\-base vocabulary increases the embedding parameters by approximately 3\.2%\. Moreover, retraining such a model to integrate the new tokens into its representational space is computationally expensive and may degrade performance on downstream NLP tasks\.

To address these limitations, we propose a lightweight and model\-agnostic method for integrating geographic information into pretrained embeddings without modifying the tokenizer or requiring large\-scale retraining\. Our approach injects structured geo\-spatial signals into the embedding space, aligning representations with real\-world spatial relationships while preserving existing linguistic knowledge\. As a result, the model gains location awareness with minimal computational overhead and no loss in downstream performance\.

The main contributions of this work are as follows:

- •We propose a lightweight method for integrating geo\-locational information into pretrained language model embeddings without modifying the tokenizer or vocabulary\.
- •Our approach enriches the embedding space with structured geographic signals while preserving existing syntactic and semantic knowledge\.
- •We demonstrate substantial improvements in geo\-spatial awareness without degrading performance on downstream benchmarks such as GLUE\.
- •We also validate the generalizability of our method across different model architectures, scales, and multilingual settings\.

## 2Related Work

Prior work has shown that incorporating geographic information into language models can improve the representation of spatial entities and performance on geo\-text understanding tasks\[[9](https://arxiv.org/html/2609.00454#bib.bib13),[10](https://arxiv.org/html/2609.00454#bib.bib14),[4](https://arxiv.org/html/2609.00454#bib.bib15),[14](https://arxiv.org/html/2609.00454#bib.bib16)\]\. SpaBERT\[[9](https://arxiv.org/html/2609.00454#bib.bib13)\]employs a masked entity prediction objective alongside masked language modeling to capture geographic relationships among entities\. GeoLM\[[10](https://arxiv.org/html/2609.00454#bib.bib14)\]combines masked language modeling with contrastive learning to jointly model linguistic and geographic information\. Similarly, MGeo\[[4](https://arxiv.org/html/2609.00454#bib.bib15)\]introduces a multimodal architecture with a dedicated geographic encoder for query and point\-of\-interest \(POI\) matching\. While these approaches demonstrate the benefits of geo\-spatial modeling, they depend on specialized architectures, additional learning objectives, or extensive training to acquire location awareness\.

In contrast, our method injects geographic signals into pretrained language models through lightweight input augmentation, requiring no tokenizer modifications, no additional learning objectives beyond masked language modeling, and only a short period of training, while preserving general semantic knowledge\.

## 3Method

![Refer to caption](https://arxiv.org/html/2609.00454v1/loc_emb_v2.jpg)Figure 2:Overview of the proposed framework for incorporating geo\-spatial information into pretrained language model embeddings\. Given an input sentence containing location entities, we construct a secondary embedding that encodes the corresponding geographic information \(city name, latitude, and longitude\)\. This information is integrated into the input sequence using special tokens \(<l​o​c\><loc\>,</loc\></loc\>, and<l​\_​s​e​p\><l\\\_sep\>\) to inject structured location signals into the model\. During training, we employ two masking strategies: \(1\) standard masked language modeling \(MLM\) with random token masking \(left\), and \(2\) a location\-focused masking scheme that selectively masks city names \(right\)\. The model is trained to recover masked entities using both contextual and geo\-spatial cues, enabling improved location\-aware representations\.### 3\.1Dataset Generation

We construct a synthetic dataset using information derived from the General Transit Feed Specification \(GTFS\) for long\-distance train schedules in Germany111https://gtfs\.de/en/feeds/\. After pre\-processing, we extract 820 unique location names corresponding to train stations, which serve as geographic entities\.

To generate natural language training data, we employ two large language models: Mistral\[[7](https://arxiv.org/html/2609.00454#bib.bib5),[6](https://arxiv.org/html/2609.00454#bib.bib6)\]and LLaMA 3\[[5](https://arxiv.org/html/2609.00454#bib.bib7)\], to produce journey descriptions containing station names, arrival and departure times, and other contextual details\. This results in a corpus of 5,965 text samples with a total of 62,758 occurrences of location entities\.

In addition to textual data, we associate each location with its corresponding latitude and longitude coordinates, forming a structured geo\-spatial reference that is later incorporated into the training process\.

### 3\.2Continual Pretraining with Secondary Embeddings

To enable geo\-spatial awareness, we use three special tokens:<l​o​c\><loc\>\(start of location\),</loc\></loc\>\(end of location\), and<l​\_​s​e​p\><l\\\_sep\>\(location separator\)\. These tokens explicitly define the boundaries of location\-specific information within the input sequence\.

We construct a mapping between eachstop\_nameand its geographic coordinates \(latitude and longitude\), which we refer to assecondary embeddings\. Unlike standard token embeddings learned purely from text, these embeddings encode structured spatial information and act as an auxiliary signal during training\.

During tokenization, each occurrence of a location entity is augmented into the following structured format:

<l​o​c\><loc\>stop\_name<l​\_​s​e​p\><l\\\_sep\>latitude<l​\_​s​e​p\><l\\\_sep\>longitude</loc\></loc\>

This representation injects both textual and coordinate\-based information directly into the model’s input stream, enabling the model to associate entities with their corresponding spatial attributes\.

We perform continual pretraining using the masked language modeling \(MLM\) objective with two different masking strategies \(Figure[2](https://arxiv.org/html/2609.00454#S3.F2)\):

- •Random Masking \(ℛ\\mathcal\{R\}\):Following the standard BERT training procedure\[[3](https://arxiv.org/html/2609.00454#bib.bib1)\], a subset of tokens is randomly masked\.
- •Location Masking \(𝕃\\mathbb\{L\}\):We specifically mask thestop\_nametokens within the<l​o​c\><loc\>blocks while retaining the associated latitude and longitude information\. The model is trained to predict the masked location entity using both contextual and geo\-spatial cues\. This targeted objective encourages alignment between textual representations and their underlying geographic structure\.

We train separate models under each masking strategy and compare their effectiveness in inducing geo\-spatial awareness\.

## 4Experiment Details

### 4\.1Evaluation

The primary objective of our approach is to enrich pretrained language model embeddings with geo\-locational information while preserving their existing semantic knowledge\. To verify that location\-aware pretraining does not degrade linguistic capability, we evaluate the models on the GLUE benchmark\[[15](https://arxiv.org/html/2609.00454#bib.bib11)\]\. We report F1\-scores for MRPC, Pearson correlation for STS\-B, and accuracy for the remaining tasks\. Performance comparable to the baseline indicates that the injected geo\-spatial information does not compromise general language understanding\.

To evaluate location awareness, we analyze the alignment between distances in the embedding space and real\-world geographic distances\. We compute pairwise geographic distances between locations using the Haversine formula based on latitude and longitude coordinates, and treat this as the ground\-truth distance matrix\.

Since model embeddings reside in a high\-dimensional space, we measure distances between location embeddings using Euclidean distance and cosine similarity\. Our empirical evaluation on ground\-truth data demonstrates a strong correlation between embedding\-based distances and Haversine distance, indicating that Euclidean distance and cosine similarities provide reliable proxies for geographic distance in the embedding space\.

We then compute Pearson and Spearman correlation coefficients between the geographic distance matrix and the corresponding embedding distance matrices\. Higher correlation indicates better preservation of real\-world spatial relationships in the learned embedding space\.

### 4\.2Implementation Details

We train the models on the synthetic dataset described in Section[3\.1](https://arxiv.org/html/2609.00454#S3.SS1)for 25 epochs using a batch size of 60 and a learning rate of1×10−41\\times 10^\{\-4\}with the AdamW optimizer\. We use pretrained models using the Hugging Face Transformers library\[[16](https://arxiv.org/html/2609.00454#bib.bib12)\]\.

For GLUE tasks, fine\-tuning is performed for up to 50 epochs with a batch size of 256 and a learning rate of1×10−51\\times 10^\{\-5\}using bfloat16 precision\. We apply early stopping with a patience of 5 to prevent overfitting\.

Unless otherwise specified, all experiments are conducted using the base variant of each pretrained model\. Location\-aware training completes within approximately 15 minutes on a single NVIDIA H100 GPU, demonstrating the computational efficiency of the proposed approach\.

Table 1:GLUE benchmark results for baseline and location\-aware models across multiple architectures\. Here,ℛ\\mathcal\{R\}and𝕃\\mathbb\{L\}denote models trained with random masking and location\-specific masking, respectively\. The results show that location\-aware training preserves general language performance while achieving comparable or slightly improved scores\.

## 5Results and Discussion

Table[1](https://arxiv.org/html/2609.00454#S4.T1)presents the GLUE benchmark results for our location\-aware models across multiple architectures\. Overall, the results show that the approach preserves general language understanding, with performance comparable to baseline pretrained models across all tasks\.

A closer inspection of Table[1](https://arxiv.org/html/2609.00454#S4.T1)shows that location\-aware variants often match or slightly improve upon baseline performance\. For instance, BERTbaseimproves from 81\.16 to 81\.96 with location masking, while ALBERT shows a noticeable gain from 81\.44 to 82\.07\. These improvements indicate that injecting geo\-spatial information does not interfere with linguistic knowledge and, in some cases, provides additional useful signals\.

Across architectures, location\-specific masking \(𝕃\\mathbb\{L\}\) tends to yield more consistent gains compared to random masking \(ℛ\\mathcal\{R\}\), particularly for BERT\[[3](https://arxiv.org/html/2609.00454#bib.bib1)\]and ALBERT\[[8](https://arxiv.org/html/2609.00454#bib.bib9)\]\. This suggests that explicitly masking location entities creates a stronger and more targeted learning signal for aligning textual and geo\-spatial representations\. In contrast, random masking primarily preserves baseline performance without consistently improving it\.

#### Embedding Analysis\.

To evaluate how location information is encoded, we compare two embedding extraction strategies: the\[CLS\]token and mean pooling over all tokens\. While both capture location\-aware signals, mean pooling consistently exhibits stronger alignment with geographic structure\. For example, in RoBERTa, the Pearson correlation improves from 0\.2997 \(baseline\) to 0\.3343 using\[CLS\], and further to 0\.4503 using mean embeddings\. This indicates that geo\-spatial information is distributed across tokens rather than concentrated in a single representation\.

#### Impact of Tokenization\.

We observe that tokenizer choice significantly influences the effectiveness of location\-aware adaptation\. Models based on WordPiece tokenization \(e\.g\., BERT and DistilBERT\) show more stable and consistent improvements compared to byte\-level BPE \(RoBERTa\) and SentencePiece\-based models \(XLM\-R, mBERT\)\. WordPiece decomposes rare words into meaningful subword units, which helps preserve partial semantic structure of location names\. In contrast, byte\-level BPE operates at the character level, and SentencePiece treats text as a continuous Unicode stream, which may dilute the representation of structured entities such as location names and numeric coordinates\.

Interestingly, uncased models tend to benefit more consistently from location\-aware training, whereas cased models show less pronounced improvements\. This suggests that case sensitivity may introduce additional variability in entity representations, making it harder to learn consistent geo\-spatial relationships\.

### 5\.1Effect of Model Size

Table 2:Correlation results comparing embedding distances with geographic distances\.rrandρ\\rhodenote Pearson and Spearman correlation coefficients, respectively\. Higher values indicate improved geo\-spatial alignment\.To study the impact of model scale, we evaluate BERT variants of different sizes, including BERTbase, BERTlarge, and DistilBERT\. The configurations of these models are shown in Table[3](https://arxiv.org/html/2609.00454#S5.T3)\.

Table 3:Configurations of the models used in our experiment\.Table[2](https://arxiv.org/html/2609.00454#S5.T2)reports the correlation between embedding distances and ground\-truth geographic distances using both Euclidean and cosine similarity\. The results show that larger models benefit more from location\-aware training\. BERTlargeachieves the highest correlation \(Pearsonr=0\.550r=0\.550\), indicating a stronger alignment between embedding space and real\-world geography\. BERTbaseand DistilBERT also show substantial improvements over their respective baselines, demonstrating that the proposed method generalizes across model scales\.

We further evaluate a smaller model, BERT\-tiny\[[13](https://arxiv.org/html/2609.00454#bib.bib8)\], which shows only marginal improvement \(from 0\.0123 to 0\.0139\)\. This suggests that very small models lack sufficient capacity to effectively incorporate geo\-spatial signals\.

Additional experiments with RoBERTa\[[11](https://arxiv.org/html/2609.00454#bib.bib3)\]show a significant improvement in correlation \(from 0\.2665 to 0\.4296\), further confirming the effectiveness of our approach in larger architectures\.

### 5\.2Effect of Multilingual Models

We evaluate the impact of location\-aware training on multilingual models, including mBERT\[[2](https://arxiv.org/html/2609.00454#bib.bib2)\]and XLM\-R\[[1](https://arxiv.org/html/2609.00454#bib.bib10)\]\. In contrast to monolingual models, we observe limited or no consistent improvement in geo\-spatial alignment\.

This can be attributed to the significantly larger and more diverse vocabularies used in multilingual models \(e\.g\.,∼\\sim105k tokens for mBERT and∼\\sim250k for XLM\-R\), as well as the increased variability in entity representation across languages\. These factors make it more challenging for the model to learn consistent mappings between textual and geographic representations\.

### 5\.3Effect of Other Architectures

We further evaluate the generalizability of our approach on alternative architectures such as ALBERT\[[8](https://arxiv.org/html/2609.00454#bib.bib9)\]and MobileBERT\[[12](https://arxiv.org/html/2609.00454#bib.bib4)\]\. For ALBERT, we observe a substantial improvement in geo\-spatial alignment, with correlation increasing from 0\.0039 to 0\.1746\. Similarly, MobileBERT shows a improvement from 0\.0123 to 0\.0345\. These results demonstrate that our method is not limited to standard transformer architectures and can generalize to parameter\-efficient and compressed models\.

## 6Conclusion

In this work, we introduced a lightweight and effective approach for incorporating geo\-spatial information into pretrained language models\. By augmenting the input with structured location signals and applying targeted masking strategies, our method enables conventional language models to become location\-aware without requiring architectural modifications, vocabulary expansion, or costly retraining\. Experimental results demonstrate that our approach improves the alignment between the embedding space and real\-world geographic relationships, while maintaining comparable performance on standard NLP benchmarks such as GLUE\. These findings confirm that location awareness can be introduced without degrading the model’s existing semantic and syntactic capabilities\. Furthermore, the proposed method is computationally efficient, requiring only around 15 minutes of training to adapt pretrained models\. This makes it a practical and scalable solution for enhancing language models with geo\-spatial grounding in real\-world applications\.

## 7Limitations

Despite the promising results, several limitations remain\. First, while our approach improves geo\-spatial alignment, a deeper understanding of how location information is encoded within the embedding space—particularly in interaction with semantic and syntactic features—requires further investigation\. Second, our method operates as a post\-hoc adaptation to pretrained models\. Since most representational capacity is learned during the initial pretraining phase, incorporating geo\-spatial information directly during large\-scale pretraining may yield stronger and more robust representations\. Investigating this direction constitutes an important area for future research\. Finally, our current work focuses on encoder\-based models\. Extending this approach to generative language models remains an open challenge and an important direction for future exploration\.

## Acknowledgments

This work has been supported by the Verkehrsverbund Großraum Ingolstadt \(VGI\) as part of the project newMIND\.

## References

- \[1\]A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov\(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§5\.2](https://arxiv.org/html/2609.00454#S5.SS2.p1.1)\.
- \[2\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1,Minneapolis, Minnesota\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§5\.2](https://arxiv.org/html/2609.00454#S5.SS2.p1.1)\.
- \[3\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§1](https://arxiv.org/html/2609.00454#S1.p1.1),[1st item](https://arxiv.org/html/2609.00454#S3.I1.i1.p1.1),[§5](https://arxiv.org/html/2609.00454#S5.p3.1)\.
- \[4\]R\. Ding, B\. Chen, P\. Xie, F\. Huang, X\. Li, Q\. Zhang, and Y\. Xu\(2023\)Mgeo: multi\-modal geographic language model pre\-training\.InProceedings of the 46th international ACM SIGIR conference on research and development in information retrieval,pp\. 185–194\.Cited by:[§1](https://arxiv.org/html/2609.00454#S1.p4.1),[§2](https://arxiv.org/html/2609.00454#S2.p1.1)\.
- \[5\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3\.1](https://arxiv.org/html/2609.00454#S3.SS1.p2.1)\.
- \[6\]A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.\(2024\)Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§3\.1](https://arxiv.org/html/2609.00454#S3.SS1.p2.1)\.
- \[7\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de Las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7b\.ArXivabs/2310\.06825\.External Links:[Link](https://api.semanticscholar.org/CorpusID:263830494)Cited by:[§3\.1](https://arxiv.org/html/2609.00454#S3.SS1.p2.1)\.
- \[8\]Z\. Lan, M\. Chen, S\. Goodman, K\. Gimpel, P\. Sharma, and R\. Soricut\(2020\)ALBERT: a lite bert for self\-supervised learning of language representations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1eA7AEtvS)Cited by:[§5\.3](https://arxiv.org/html/2609.00454#S5.SS3.p1.1),[§5](https://arxiv.org/html/2609.00454#S5.p3.1)\.
- \[9\]Z\. Li, J\. Kim, Y\. Chiang, and M\. Chen\(2022\)SpaBERT: a pretrained language model from geographic data for geo\-entity representation\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 2757–2769\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.200/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.200)Cited by:[§1](https://arxiv.org/html/2609.00454#S1.p4.1),[§2](https://arxiv.org/html/2609.00454#S2.p1.1)\.
- \[10\]Z\. Li, W\. Zhou, Y\. Chiang, and M\. Chen\(2023\)GeoLM: empowering language models for geospatially grounded language understanding\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5227–5240\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.317/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.317)Cited by:[§1](https://arxiv.org/html/2609.00454#S1.p4.1),[§2](https://arxiv.org/html/2609.00454#S2.p1.1)\.
- \[11\]Y\. Liu\(2019\)Roberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692364\.Cited by:[§1](https://arxiv.org/html/2609.00454#S1.p1.1),[§5\.1](https://arxiv.org/html/2609.00454#S5.SS1.p4.1)\.
- \[12\]Z\. Sun, H\. Yu, X\. Song, R\. Liu, Y\. Yang, and D\. Zhou\(2020\)MobileBERT: a compact task\-agnostic BERT for resource\-limited devices\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.195)Cited by:[§5\.3](https://arxiv.org/html/2609.00454#S5.SS3.p1.1)\.
- \[13\]I\. Turc, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Well\-read students learn better: on the importance of pre\-training compact models\.arXiv preprint arXiv:1908\.08962\.Cited by:[§5\.1](https://arxiv.org/html/2609.00454#S5.SS1.p3.1)\.
- \[14\]H\. Wan, Y\. Lin, S\. Guo, and Y\. Lin\(2022\)Pre\-training time\-aware location embeddings from spatial\-temporal trajectories\.IEEE Transactions on Knowledge and Data Engineering34\(11\),pp\. 5510–5523\.External Links:[Document](https://dx.doi.org/10.1109/TKDE.2021.3057875)Cited by:[§2](https://arxiv.org/html/2609.00454#S2.p1.1)\.
- \[15\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,Brussels, Belgium\.External Links:[Document](https://dx.doi.org/10.18653/v1/W18-5446)Cited by:[§4\.1](https://arxiv.org/html/2609.00454#S4.SS1.p1.1)\.
- \[16\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the Conference on Empirical Methods in NLP: System Demonstrations,Cited by:[§4\.2](https://arxiv.org/html/2609.00454#S4.SS2.p1.1)\.