Tokenizing Crosslingual Homographs

# Tokenizing Crosslingual Homographs
Source: [https://arxiv.org/html/2607.17689](https://arxiv.org/html/2607.17689)
Rotem Brillant Yuval Pinter Stein Faculty of Computer and Information Science Ben\-Gurion University of the Negev Beer Sheva, Israel \{brillant@post,uvp@cs\}\.bgu\.ac\.il

###### Abstract

Multilingual language models rely on shared subword vocabularies to represent multiple languages within a limited number of token units\. While such sharing is often useful, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ\. We investigate this limitation through cross\-lingual homographs and false friends, and examine whether introducing language information earlier in the tokenization process can improve their treatment\. We propose a simple tokenizer\-level intervention based on language cues: language\-specific characters replacing initial characters of shared\-vocabulary words, reducing common identity during vocabulary construction\. In intrinsic analysis, we find through tokenizer\-level statistics that BPE and UnigramLM often treat cross\-lingual homographs in a largely language\-agnostic way, whereas the context\-sensitive SaGe tokenizer diverges more strongly; our intervention removes this gap\. In downstream English\-to\-X machine translation, our cues yield modest improvements in several settings, especially under BPE, although the effect is not consistent across all languages and evaluation sets\. Overall, the findings suggest that adding lightweight language information at the tokenizer level is a promising direction for further exploration\.

## 1Introduction

Multilingual large language models have significantly advanced natural language processing by enabling a single model to represent multiple languages within a shared parameter space\(Aharoni et al\.,[2019](https://arxiv.org/html/2607.17689#bib.bib2); Conneau et al\.,[2020](https://arxiv.org/html/2607.17689#bib.bib4)\)\. A central component of these models is the tokenization process, which constructs a subword vocabulary used to encode textual input across languages\. Tokenization plays a critical role in balancing computational efficiency with linguistic expressiveness, particularly in multilingual settings\(Limisiewicz et al\.,[2023](https://arxiv.org/html/2607.17689#bib.bib15)\)\.

![Refer to caption](https://arxiv.org/html/2607.17689v1/figures/language_cue_intro.png)Figure 1:Illustration of language cues applied to a shared homograph before tokenizer training\.Because vocabulary size is limited, selecting an appropriate tokenizer configuration becomes a key architectural decision\(Liang et al\.,[2023](https://arxiv.org/html/2607.17689#bib.bib14)\)\. Multilingual contexts introduce unique design considerations, such as how vocabulary capacity should be distributed across languages, whether to train a shared tokenizer or aggregate language\-specific ones, and how increasing the number of languages affects cross\-lingual transfer performance\.

In a shared vocabulary, an important challenge emerges when training tokenizers on multiple related languages: the presence of shared tokens\. Related languages often contain identical surface forms that are assigned a single subword unit in the learned vocabulary\. An example is the wordbarn, shared between English and Swedish\. These shared tokens receive a single embedding representation that is jointly trained across languages during pre\-training\(Vernikos and Popescu\-Belis,[2021](https://arxiv.org/html/2607.17689#bib.bib24)\)\. However, this shared representation may not equally capture the meanings and usage patterns of all contributing languages\. Such imbalance can arise due to differences in corpus size, token frequency, or contextual usage patterns across languages\.

We investigate the shared\-token phenomenon through a specific class of words: cross\-lingual homographs, words that share identical surface forms across different languages\. While many such words preserve similar meanings across languages \(or denote identical entities\), they may appear in distinct contextual distributions\. Moreover, some homographs correspond to*false friends*, carrying different meanings despite their identical surface forms\. For example, the formbankin English can refer to a financial institution or the side of a river, as in “They sat by the river bank to rest\.” In German, however,bankcommonly means a bench, as in “Ich bin auf der Bank mit dem Kopf gestoßen\.” \(I hit my head on the bench\)\. False friends introduce additional complexity to multilingual tokenization: how can a single shared token represent language\-dependent meanings and usage patterns?

We begin by showing that mainstream tokenizer algorithms, such as Byte\-Pair Encoding\(BPE; Sennrich et al\.,[2016](https://arxiv.org/html/2607.17689#bib.bib21)\)and Unigram Language Modeling\(ULM; Kudo,[2018](https://arxiv.org/html/2607.17689#bib.bib9)\), often treat homographs in a language\-agnostic manner during vocabulary construction\. Because the tokenizer is trained on a concatenated multilingual corpus without explicit language conditioning, identical surface forms are typically assigned a single shared subword unit\. As a result, the downstream embedding representation aggregates signals from multiple languages, potentially reinforcing the imbalance we described\. In contrast, the SaGe tokenizer\(Yehezkel and Pinter,[2023](https://arxiv.org/html/2607.17689#bib.bib26)\), which incorporates contextual information during vocabulary construction, creates vocabularies that segment shared surface forms differently across languages more often\.

To improve homograph handling even more, we introduce*language cues*into the tokenizer training process\. These cues are language\-specific Unicode characters injected into the training corpus\. For each shared homograph, the first character is mapped to a language\-unique cue, allowing the tokenizer to observe minimal but systematic language distinctions during vocabulary learning\. The multilingual tokenizer is thus encouraged to distinguish homographs across languages, potentially allocating distinct subword units per language and enabling more balanced embeddings\.

To examine whether this approach is useful beyond vocabulary construction itself, we evaluate the proposed language cues in a multilingual machine translation setting\. We compare baseline and cue\-based English\-to\-X translation models across several target languages, using both BPE and ULM tokenization and evaluating on general as well as homograph\-focused test sets\. While the results do not suggest a uniform improvement in every setting, they indicate that language cues can lead to small and encouraging gains, particularly in some evaluation conditions involving homographs and on more challenging benchmarks\. These findings provide preliminary evidence that introducing lightweight language signals during tokenization may help mitigate problematic shared\-token behavior in multilingual downstream tasks\.

## 2Homographs in Tokenization

Over the years, a range of approaches has been proposed to improve multilingual language models\. Studies have looked into specialized vocabulary allocation schemes, composition of multilingual tokenizers from monolingual ones, increasing multilingual vocabularies, alternative encoding methods, vocabulary extension, and data\-balancing or adaptation strategies for low\-resource languages\. Together, these methods address important challenges such as vocabulary bottlenecks, unfair allocation across languages, and weak performance on underrepresented languages\. At the same time, they mostly treat multilingual tokenization as a problem of global vocabulary design, corpus balance, or model adaptation\.

One common assumption in multilingual language modeling is that some degree of vocabulary overlap across languages is useful\.Pires et al\. \([2019](https://arxiv.org/html/2607.17689#bib.bib19)\)show that multilingual BERT\(Devlin et al\.,[2019](https://arxiv.org/html/2607.17689#bib.bib5)\)can transfer knowledge between languages even without explicit cross\-lingual training, which suggests that shared structure can be useful in multilingual models\.Limisiewicz et al\. \([2023](https://arxiv.org/html/2607.17689#bib.bib15)\)show that vocabulary overlap can be beneficial in some settings, particularly for sentence\-level cross\-lingual tasks, even if it is not equally helpful for all tasks\.Kallini et al\. \([2025](https://arxiv.org/html/2607.17689#bib.bib7)\)refine this picture by asking what kind of overlap is actually helpful\. They show that overlap is usually better than having completely separate vocabularies, but mostly for words with similar meanings\. Overlap between words with different meanings is much less useful , highlighting the concern for the cases of homographs and false friends\.

Table 1:Distribution of tokenization patterns for false friends under BPE and BPE\_SaGe\.Recently,Kitamura et al\. \([2025](https://arxiv.org/html/2607.17689#bib.bib8)\)andTanwar et al\. \([2026](https://arxiv.org/html/2607.17689#bib.bib22)\)showed that models often struggle to disambiguate cross\-lingual homographs and may rely too heavily on orthographic similarity rather than semantic interpretation, adding to the motivation of the present work\. We do not treat token sharing only as a question of vocabulary size, clustering, or vocabulary allocation\. Instead, we focus on cases where identical surface forms may be tokenized in the same way, even when their meanings or usage differ across languages\. Our goal is not to eliminate overlap altogether, but to provide the tokenizer with a signal so that potentially misleading shared forms can be distinguished earlier in the pipeline\. In that sense, the proposed language cues operate at a different level from approaches such asLample and Conneau \([2019](https://arxiv.org/html/2607.17689#bib.bib11)\)’s, where language embeddings are introduced on the model side during pretraining\. Here, the language signal is introduced directly into the input data, at the point where shared subword units are first created\.

### 2\.1Analysis of Homograph Tokenization

To further motivate the proposed approach, we first examine how cross\-lingual homographs are tokenized by existing subword tokenizers in monolingual and bilingual settings\. If multilingual tokenizers tend to segment homographs in the same way as their monolingual counterparts, even when meanings differ across languages, this would suggest that the same written form is being segmented similarly regardless of language\. Such a pattern would support the need for an explicit language signal during tokenizer training\.

We use a false friends dataset\(aari1995,[2026](https://arxiv.org/html/2607.17689#bib.bib1); Wiktionary contributors,[2026](https://arxiv.org/html/2607.17689#bib.bib25)\)to focus on the clearest case in which identical surface forms should not necessarily be tokenized in the same way across languages\.[Table 1](https://arxiv.org/html/2607.17689#S2.T1)compares monolingual and multilingual tokenizers on this subset under two settings: BPE,111SentencePiece version\(Kudo and Richardson,[2018](https://arxiv.org/html/2607.17689#bib.bib10)\)\.and SaGe with a BPE\-initialized vocabulary\. All tokenizers are trained on monolingual and concatenated multilingual training corpora\(Goldhahn et al\.,[2012](https://arxiv.org/html/2607.17689#bib.bib6); Leipzig Corpora Collection,[2021](https://arxiv.org/html/2607.17689#bib.bib13)\)\. The token sequence triplets resulting from each setting \(English, L2∈\{\\in\\\{German, French\}\\\}, bilingual\) are grouped into several cases:same\_splitsdenotes cases where the English monolingual, L2 monolingual, and multilingual tokenizers all produce the same segmentation;en\_t=mult\_tandL2\_t=mult\_tdenote cases where the multilingual tokenizer matches only one of the monolingual tokenizers;en\_t=L2\_tdenotes cases where the two monolingual tokenizers agree with each other but differ from the multilingual tokenizer; anddifferent\_splitsdenotes cases where all three tokenizations differ\.

The comparison shows a clear contrast between tokenizer families\. Standard BPE often produces the same segmentation in monolingual and multilingual settings, while SaGe diverges more strongly\. To determine whether this pattern extends beyond the false\-friend subset, we next examine the broader set of homographs\. We build a homograph dataset using a dictionary repository\(Altun,[2026](https://arxiv.org/html/2607.17689#bib.bib3)\), adding four more languages to our set, and use Earth mover’s distance \(EMD\) to compare the tokenization distributions produced by the BPE and SaGe tokenizer families\. Unlike exact category counts, EMD provides a single measure of how different the overall tokenization patterns are\. Our EMD function is detailed in[Appendix B](https://arxiv.org/html/2607.17689#A2)\.

BPEULMLanguageENL2ENAllAllEMDENL2ENAllAllEMDPair=M=M=L2SameDiff=M=M=L2SameDiffEN\-DE1924131417021058940\.662001134619120088610\.73w/SaGe1141750470510353612185602784213930EN\-FR1263105519018538010\.521217128012818107270\.67w/SaGe79787930078624006776095085082860EN\-ES573761698614200\.54583777639493120\.65w/SaGe44045117330413164903832162391356EN\-IT1328565516308090\.478391140469364220\.56w/SaGe53728337020519885655143053011698EN\-RO13277127515335380\.70947109811415434830\.68w/SaGe60741046138923186075144254192220EN\-SV606402529321730\.67652437548212010\.65w/SaGe4222682212869683802662222551042Table 2:Distribution of tokenization patterns for homographs under BPE \(left\) and ULM \(right\)\.[Table 2](https://arxiv.org/html/2607.17689#S2.T2)shows that the broader homograph set follows the same general pattern observed for false friends\. Across all language pairs, BPE produces many more cases in thesame\_splitscategory, indicating that monolingual and multilingual tokenizers often preserve the same segmentation\. In contrast, the SaGe variant yields substantially more cases in thedifferent\_splitscategory, suggesting greater divergence between monolingual and multilingual tokenization\. UnigramLM \([Table 2](https://arxiv.org/html/2607.17689#S2.T2), right\) follows a similar pattern to BPE, with monolingual and multilingual tokenizers often producing the same segmentation for homographs, and SaGe reversing the trend\. Taken together, these results suggest that standard BPE and ULM tend to treat cross\-lingual homographs in a more language\-agnostic way\.

## 3Language Cues for Homographs

![Refer to caption](https://arxiv.org/html/2607.17689v1/figures/cue_injection_flow.png)Figure 2:Overview of multilingual language model pipeline: standard flow vs\. language\-cue injection\.We introduce lightweight language\-specific cues during tokenizer training, at the stage where subword units are learned from text\. First, we construct language\-pair\-specific homograph sets for English and each target language\. Using these sets, we modify the training corpus by inserting language\-specific Unicode cues into selected words, replacing the first character of each marked homograph with a language\-unique cue\. We then train tokenizers on the modified corpus and use them in a downstream machine translation setting, where we compare cue\-based systems with standard baselines across several English\-to\-L2 language pairs\.

### 3\.1Homograph Set Construction

To construct the homograph sets used in this work, we extracted candidate words from the all\-words\-in\-all\-languages repository\(Altun,[2026](https://arxiv.org/html/2607.17689#bib.bib3)\)\. For each English–L2 language pair, we found the intersection between the English word list and the corresponding target\-language word list, yielding a set of shared surface forms\. We then applied two filtering steps: first, we removed words shorter than three characters, in order to avoid irregular forms such as Roman numerals \(e\.g\., ii\)\. Then, we filtered out words that appeared less than 5 times in either language\-specific training corpus\. This excluded words that are either very rare overall or strongly imbalanced across the two languages, and are therefore less likely to provide reliable evidence during tokenizer training\. The resulting homograph sets are diverse and include names, entities, false friends, and medical terms, among other forms\.

### 3\.2Language Cues

Our method introduces language\-specific cues directly into the tokenizer training data\.[Appendix A](https://arxiv.org/html/2607.17689#A1)contains the full mapping from lowercase English letters to the language\-specific Unicode cue characters we used and explains our selection\. For each language, we define a unique set of 26 Unicode characters, where each cue is mapped to one of the 26 lowercase English letters\.222In all experiments, we lowercased the data in pre\-processing\.This creates a simple one\-to\-one correspondence between the first letter of a homograph and a language\-specific replacement character\. When a word from the homograph set is encountered in the corpus, its first character is replaced with the cue corresponding to that language and letter\.

The rationale behind our cue protocol is to provide a lightweight language signal without forcing the two forms into fully separate representations\. Recoverability of the original text is maintained, as the cue replaces only the first character in a systematic one\-to\-one manner\. We verified that the selected characters are extremely rare in the corpora we used for all tasks, and never exist in beginnings of words where all other characters are standard\. The cue serves as an additional indicator of the linguistic context in which the homograph appears, while the remaining characters can still be shared across languages when the tokenizer finds this useful\. For example, if the English and German forms ofbankreceive different initial cues, the tokenizer may still learn shared subword units such asankornk\. This allows the method to distinguish language\-specific uses of a homograph while preserving the possibility of partial cross\-lingual sharing\.

## 4Evaluation

Before evaluating the proposed method in downstream settings, we examine its static vocabulary features\. We compare multilingual tokenizers trained with language cues against their standard baseline counterparts, with the goal of identifying how cue injection changes basic tokenization behavior\. To characterize these differences, we use several tokenizer\-level measures that capture whether cue injection changes the compactness of the learned vocabulary and the granularity of word segmentation\.[Table 3](https://arxiv.org/html/2607.17689#S4.T3)presents statistics for token length at the vocabulary level and for Rényi efficiency\(Zouhar et al\.,[2023](https://arxiv.org/html/2607.17689#bib.bib28)\)over a held\-out development set for English–German and English–French\. Cue\-based multilingual tokenizers differ only modestly from their baseline counterparts, but the differences are fairly consistent\.333Similar patterns are observed across the remaining language pairs; the corresponding tables and fertility plots are provided in[Appendix C](https://arxiv.org/html/2607.17689#A3)\.In both language pairs, over both tokenizer variants, cued tokens are slightly shorter and efficiency is slightly higher\. The former is expected: as alphabet size increases, more tokens are necessary to encode sequences\. The latter is encouraging: higher Rényi efficiency signifies a more balanced distribution of token types across a corpus\.

Table 3:Vocabulary length and Rényi efficiency for English–German and English–French tokenizers\.![Refer to caption](https://arxiv.org/html/2607.17689v1/figures/03_fertility_train_en_de_BPE.png)

![Refer to caption](https://arxiv.org/html/2607.17689v1/figures/03_fertility_train_en_de_UNI.png)

Figure 3:Token fertility comparison between baseline and cue\-based multilingual tokenizers for EN–DE, using BPE \(top\) and UnigramLM \(bottom\)\.![Refer to caption](https://arxiv.org/html/2607.17689v1/figures/03_fertility_train_en_fr_BPE.png)

![Refer to caption](https://arxiv.org/html/2607.17689v1/figures/03_fertility_train_en_fr_UNI.png)

Figure 4:Token fertility comparison between baseline and cue\-based multilingual tokenizers for EN–FR, using BPE \(top\) and UnigramLM \(bottom\)\.Figures[3](https://arxiv.org/html/2607.17689#S4.F3)–[4](https://arxiv.org/html/2607.17689#S4.F4)compare the*fertility*, presented as tokens per word distributions, between baselines and cue\-based multilingual tokenizers\. Cued tokenizers produce fewer single\-token words and more words split into three to five subwords, suggesting that cue injection encourages somewhat finer segmentation of the training data\.

Although these intrinsic changes are modest, they indicate that cue injection shifts tokenizer behavior in a consistent direction before any downstream training is performed\.

### 4\.1Language Modeling

Table 4:BPE token\-level and word\-level perplexity results for baseline and cued EN–L2 models\. Bolded values indicate the lower perplexity for each language pair and metric\.Table 5:UnigramLM token\-level and word\-level perplexity results for baseline and cued EN–L2 models\. Bolded values indicate the lower perplexity for each language pair and metric\.For each evaluated setup, we train a small GPT language model\(Radford et al\.,[2018](https://arxiv.org/html/2607.17689#bib.bib20)\)to evaluate the effect of language cues through perplexity\. Model architecture details are in[Appendix D](https://arxiv.org/html/2607.17689#A4)\. The models are trained on multilingual text from the OPUS\-100\(Zhang et al\.,[2020](https://arxiv.org/html/2607.17689#bib.bib27); Tiedemann,[2012](https://arxiv.org/html/2607.17689#bib.bib23)\)parallel training data, split into 90% training and 10% evaluation\. For each tokenizer type, we train one model on the baseline\-tokenized data and one model on the cue\-tokenized data, and compare their perplexity on the evaluation split\.

[Table 4](https://arxiv.org/html/2607.17689#S4.T4)and[Table 5](https://arxiv.org/html/2607.17689#S4.T5)report token\-level and word\-level perplexity for the baseline and cue\-based models\. Token\-level perplexity is lower for the cue\-based models across all language pairs under both BPE and UnigramLM\. However, word\-level perplexity is generally higher for the cue\-based models, with the only exception being EN–IT under UnigramLM, where it improves slightly\. Since token\-level perplexity depends on segmentation, word\-level perplexity provides the more stable comparison in the present setting\. Under this normalization, the cue\-based models do not show a consistent intrinsic language\-modeling advantage\. The token\-level perplexity results should therefore be interpreted cautiously\.

### 4\.2Machine Translation

To further test downstream evaluation, we apply the cue\-based tokenization setup to machine translation from English to L2\. For each language pair, we compare two main systems: a baseline model trained on the original parallel data, and a cue\-based model trained on a modified version of the same data in which homographs from the constructed set are marked with language\-specific cues\. Aside from the cue injection step, the baseline and cue\-based systems follow the same training setup\. Models use a shared vocabulary for both languages, and a shared input–output embedding matrix structure, providing a direct assessment of the cue protocol in its intended setting\. We used vocabulary size 8,000\. Further model architecture and implementation details are in[Appendix E](https://arxiv.org/html/2607.17689#A5)\.

#### Data

We evaluate the translation models on both standard parallel training data and additional evaluation sets in order to assess the proposed method under different conditions\. As an in\-domain benchmark, we use data from OPUS\-100\(Tiedemann,[2012](https://arxiv.org/html/2607.17689#bib.bib23); Zhang et al\.,[2020](https://arxiv.org/html/2607.17689#bib.bib27)\), a large multilingual parallel corpus covering many language pairs\. In addition, we source FLORES\+\(NLLB Team et al\.,[2024](https://arxiv.org/html/2607.17689#bib.bib16); Open Language Data Initiative,[2024](https://arxiv.org/html/2607.17689#bib.bib17)\), a higher\-quality multilingual dataset designed to support more reliable cross\-lingual evaluation\. We use it to examine whether the effects of the proposed method remain visible under stricter evaluation conditions\. Finally, from each dataset we derive ahomograph\-focusedsubset: this includes, for each language pair, all and only the sentences that contain at least one homograph from the constructed homograph set on the source side\.

#### Results

Table 6:BLEU scores for BPE across the general and homograph\-focused test sets \(HG\)\.Table 7:BLEU scores for UnigramLM across the general and homograph\-focused test sets \(HG\)\.Tables[6](https://arxiv.org/html/2607.17689#S4.T6)and[7](https://arxiv.org/html/2607.17689#S4.T7)present the BLEU scores for baseline and cued systems across the general and homograph\-focused evaluation sets\. Overall, the effect of cue injection is mixed, but several encouraging patterns emerge\. Under BPE, the cued system improves over the baseline in many settings\. These improvements are usually small, but they appear across both the full test sets and the homograph\-focused subsets\. At the same time, there are still cases in which the baseline remains stronger, such as EN–DE on OPUS\-100 and EN–SV on FLORES\+\. A second pattern is that, for BPE, the gains are often more visible on FLORES\+ than on OPUS\-100\. This is especially noticeable for EN–IT, EN–RO, and EN–DE\. The homograph\-focused subsets generally follow the same direction as the full test sets, although they do not consistently show larger gains\. This suggests that the cues may influence translation behavior beyond the selected homograph cases, but not in a uniformly targeted way\.

The UnigramLM results are less stable\. Some language pairs show gains, such as EN–DE and EN–FR on OPUS\-100 and EN–ES and EN–RO on FLORES\+, but in other settings the baseline remains stronger, particularly for EN–SV\. Compared with BPE, UnigramLM shows a less consistent response to cue injection across languages and evaluation sets, possibly because its probabilistic vocabulary selection makes it less sensitive to inserted cues\.

Taken together, these results suggest that our cueing method is promising, but its effect is clearly tokenizer\- and language\-dependent\. BPE appears to benefit more consistently from the cues, while UnigramLM produces a more mixed picture\. Although the gains are modest, their repeated appearance across several settings supports the idea that lightweight language signals at the tokenizer level can have a meaningful downstream effect\.

#### Statistical Robustness

For statistical robustness, we apply paired bootstrap resampling with 1,000 iterations for each evaluation set\. In each case, we compute the bootstrap distribution of the BLEU difference, defined as cue BLEU minus baseline BLEU, and report its mean, standard deviation, and 95% confidence interval\. This allows the observed BLEU differences to be examined beyond a single point estimate and helps assess whether the direction of the effect remains stable under repeated resampling\. The full results are in[Appendix F](https://arxiv.org/html/2607.17689#A6)\. On BPE, two confidence intervals out of 24 do not cross zero, favoring the cued model: English–Italian on both FLORES\+ and its homograph subset \(mean diff 0\.71 and 0\.75, resp\.\)\. In UnigramLM, three confidence intervals favor the base model: English–Romanian on OPUS\-100 \(−\-0\.82\), and English–Swedish on FLORES\+ and its homograph subset \(−\-2\.28 and−\-2\.17, resp\.\)\. The aggregate means for the bootstrap setup are\+\+0\.1724 for BPE, and−\-0\.2495 for UnigramLM, greatly dominated by the two Swedish FLORES\+ datasets \(without which the difference drops to 0\.0699\)\.

The BPE results seem to suggest that the cue\-based system often moves in a favorable direction, echoing the relatively stronger results \(compared to UnigramLM\) on Rényi efficiency and perplexity\. UnigramLM’s results do not replicate the overall success cues had on intrinsic and language modeling benchmarks\. While the differences are still small in magnitude, the number of language pairs and setups considered allow us to speculate as to the reason these two algorithms behave differently: BPE is bottom\-up, merging tokens based on their components’ co\-occurrence statistics, which focuses the cues’ role as alphabet items that follow similar patterns as the characters they replaced; tokens with cues will form at a slower rate than tokens with the original characters but at a similar order, which will eventually lend consistency to the resulting vocabulary\. UnigramLM, on the other hand, is a top\-down algorithm prioritizing candidate token likelihood, which causes it to be less sensitive to low\-frequency tokens like the ones cues appear in, making its behavior on them more erratic and denying them consistency\. The cues’ differentiating role may be enough to improve intrinsic statistics and LM prediction, but the semantics\-heavy task of translation requires more\.

#### Instance Analysis

It is difficult to find examples in well\-established translation datasets where false friends occur in contexts that can throw English\-to\-L2 translations off\. However, we found several instances where homograph words were translated into native words in the cued models, where the baseline kept the source form, as borrowed\-from\-English variants\. One example is the word*tour*in the English sentencethe rock band was due to tour the united states and canada until september 16\.The baseline Spanish model translated this tola banda de rock se debió altoura los estados unidos y a canadá hasta el 16 de septiembre\., while the cued model producedla banda de rock se debía a lagirapor los estados unidos y canadá hasta el 16 de septiembre\.In another example, the Englishitaly’s national football, along with german national football team is the second most successful team in the world and were the fifa world cup champions in 2006\.was translated by the baseline Italian model toil calcio nazionale dell’italia, insieme alla nazionale di calcio tedesca, è il secondoteamdi successo nel mondo ed è stato il campionato della fifa nel 2006\., with the cued model preferringsquadra\.

## 5Conclusion

Multilingual tokenization relies on shared vocabularies in order to represent many languages within a limited number of subword units\. While this sharing is often beneficial, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ\. We explored this problem through cross\-lingual homographs and proposed a simple way to introduce language information earlier in the tokenization process\.

The results presented in this work are modest, but generally encouraging\. The proposed language cues do not lead to consistent gains across all settings, and their effect depends on the tokenizer, the language pair, and the evaluation set\. At the same time, the repeated appearance of small improvements suggests that this type of intervention may be capturing a limitation of standard multilingual tokenization\. This leaves several promising directions for future work, such as exploring whether language cues can be developed into a more general tokenization method rather than remaining a targeted intervention, or extending them to multi\-script settings\.

## Limitations

One limitation of this work is its experimental scope\. The downstream evaluation is restricted to English\-to\-L2 machine translation task for a specific set of languages, most of which are fairly related and use Latin\-based scripts\. This means that the findings should be interpreted with caution, since it is not yet clear how well the same behavior would carry over to more distant language pairs, other writing systems, or broader multilingual settings\.

The proposed cueing method is also intentionally narrow: it applies only to a selected set of cross\-lingual homographs, and therefore does not address other forms of potentially problematic token sharing\. Other forms of cross\-lingual ambiguity may not be captured by this design\. At the same time, the empirical gains are generally modest and not fully consistent across tokenizers, datasets, and language pairs\. Taken together, these limitations suggest that the current results are best viewed as encouraging initial evidence rather than as a complete solution\. Future work will be needed to test whether similar ideas remain useful in larger multilingual models, on other tasks, and in more general language\-aware tokenization schemes\.

## Acknowledgements

This research was supported by the Israel Science Foundation \(grant No\. 1166/23\)\.

## References

- aari1995 \(2026\)aari1995\. 2026\.False friends english–german\.[https://huggingface\.co/datasets/aari1995/false\_friends\_en\_de](https://huggingface.co/datasets/aari1995/false_friends_en_de)\.Hugging Face dataset, accessed March 11, 2026\.
- Aharoni et al\. \(2019\)Roee Aharoni, Melvin Johnson, and Orhan Firat\. 2019\.[Massively multilingual neural machine translation](https://doi.org/10.18653/v1/N19-1388)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 3874–3884, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Altun \(2026\)Eymen Efe Altun\. 2026\.all\-words\-in\-all\-languages\.[https://github\.com/eymenefealtun/all\-words\-in\-all\-languages](https://github.com/eymenefealtun/all-words-in-all-languages)\.GitHub repository, accessed March 11, 2026\.
- Conneau et al\. \(2020\)Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov\. 2020\.[Unsupervised cross\-lingual representation learning at scale](http://arxiv.org/abs/1911.02116)\.
- Devlin et al\. \(2019\)Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\. 2019\.[BERT: Pre\-training of deep bidirectional transformers for language understanding](https://doi.org/10.18653/v1/N19-1423)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 4171–4186, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Goldhahn et al\. \(2012\)Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff\. 2012\.[Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages](https://aclanthology.org/L12-1154/)\.In*Proceedings of the Eighth International Conference on Language Resources and Evaluation \(LREC’12\)*, pages 759–765, Istanbul, Turkey\. European Language Resources Association \(ELRA\)\.
- Kallini et al\. \(2025\)Julie Kallini, Dan Jurafsky, Christopher Potts, and Martijn Bartelds\. 2025\.[False friends are not foes: Investigating vocabulary overlap in multilingual language models](http://arxiv.org/abs/2509.18750)\.
- Kitamura et al\. \(2025\)Yuka Kitamura, Jiahao Huang, and Akiko Aizawa\. 2025\.[Doppelganger\-JC: Benchmarking the LLMs’ understanding of cross\-lingual homographs between Japanese and Chinese](https://doi.org/10.18653/v1/2025.ijcnlp-long.96)\.In*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*, pages 1779–1794, Mumbai, India\. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics\.
- Kudo \(2018\)Taku Kudo\. 2018\.[Subword regularization: Improving neural network translation models with multiple subword candidates](https://doi.org/10.18653/v1/P18-1007)\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 66–75, Melbourne, Australia\. Association for Computational Linguistics\.
- Kudo and Richardson \(2018\)Taku Kudo and John Richardson\. 2018\.[Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](http://arxiv.org/abs/1808.06226)\.
- Lample and Conneau \(2019\)Guillaume Lample and Alexis Conneau\. 2019\.[Cross\-lingual language model pretraining](http://arxiv.org/abs/1901.07291)\.
- Lee et al\. \(2025\)Junyoung Lee, Marco Cognetta, Sangwhan Moon, and Naoaki Okazaki\. 2025\.[Jamo\-level subword tokenization in low\-resource Korean machine translation](https://doi.org/10.18653/v1/2025.loresmt-1.8)\.In*Proceedings of the Eighth Workshop on Technologies for Machine Translation of Low\-Resource Languages \(LoResMT 2025\)*, pages 66–80, Albuquerque, New Mexico, U\.S\.A\. Association for Computational Linguistics\.
- Leipzig Corpora Collection \(2021\)Leipzig Corpora Collection\. 2021\.[Leipzig corpora used in this study](https://wortschatz.uni-leipzig.de/en/download/)\.This work used the following Leipzig corpora for tokenizer training: \[English Wikipedia 2016 1M, German Wikipedia 2021 1M, French Wikipedia 2021 1M, Spanish Wikipedia 2021 1M, Italian Wikipedia 2021 1M, Romanian Wikipedia 2021 1M, Swedish Wikipedia 2021 1M\]\. Available from the Leipzig Corpora Collection download portal\.
- Liang et al\. \(2023\)Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa\. 2023\.[XLM\-V: Overcoming the vocabulary bottleneck in multilingual masked language models](https://doi.org/10.18653/v1/2023.emnlp-main.813)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 13142–13152, Singapore\. Association for Computational Linguistics\.
- Limisiewicz et al\. \(2023\)Tomasz Limisiewicz, Jiří Balhar, and David Mareček\. 2023\.[Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages](https://doi.org/10.18653/v1/2023.findings-acl.350)\.In*Findings of the Association for Computational Linguistics: ACL 2023*, pages 5661–5681, Toronto, Canada\. Association for Computational Linguistics\.
- NLLB Team et al\. \(2024\)NLLB Team, Marta R\. Costa\-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang\. 2024\.[Scaling neural machine translation to 200 languages](https://doi.org/10.1038/s41586-024-07335-x)\.*Nature*, 630\(8018\):841–846\.
- Open Language Data Initiative \(2024\)Open Language Data Initiative\. 2024\.Flores\+: A multilingual dataset for evaluation\.[https://huggingface\.co/datasets/openlanguagedata/flores\_plus](https://huggingface.co/datasets/openlanguagedata/flores_plus)\.Hugging Face dataset card, accessed April 11, 2026\.
- Ott et al\. \(2019\)Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli\. 2019\.fairseq: A fast, extensible toolkit for sequence modeling\.In*Proceedings of NAACL\-HLT 2019: Demonstrations*\.
- Pires et al\. \(2019\)Telmo Pires, Eva Schlinger, and Dan Garrette\. 2019\.[How multilingual is multilingual BERT?](https://doi.org/10.18653/v1/P19-1493)In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4996–5001, Florence, Italy\. Association for Computational Linguistics\.
- Radford et al\. \(2018\)Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al\. 2018\.Improving language understanding by generative pre\-training\.
- Sennrich et al\. \(2016\)Rico Sennrich, Barry Haddow, and Alexandra Birch\. 2016\.[Neural machine translation of rare words with subword units](https://doi.org/10.18653/v1/P16-1162)\.In*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1715–1725, Berlin, Germany\. Association for Computational Linguistics\.
- Tanwar et al\. \(2026\)Eshaan Tanwar, Gayatri Oke, and Tanmoy Chakraborty\. 2026\.[Multilingual llms struggle to link orthography and semantics in bilingual word processing](http://arxiv.org/abs/2501.09127)\.
- Tiedemann \(2012\)Jörg Tiedemann\. 2012\.[Parallel data, tools and interfaces in OPUS](http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf)\.In*Proceedings of the Eighth International Conference on Language Resources and Evaluation \(LREC’12\)*, pages 2214–2218, Istanbul, Turkey\. European Language Resources Association \(ELRA\)\.
- Vernikos and Popescu\-Belis \(2021\)Giorgos Vernikos and Andrei Popescu\-Belis\. 2021\.[Subword mapping and anchoring across languages](https://doi.org/10.18653/v1/2021.findings-emnlp.224)\.In*Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 2633–2647, Punta Cana, Dominican Republic\. Association for Computational Linguistics\.
- Wiktionary contributors \(2026\)Wiktionary contributors\. 2026\.Appendix: Glossary of false friends\.[https://en\.wiktionary\.org/wiki/Appendix:Glossary\_of\_false\_friends](https://en.wiktionary.org/wiki/Appendix:Glossary_of_false_friends)\.Wiktionary, accessed March 11, 2026\.
- Yehezkel and Pinter \(2023\)Shaked Yehezkel and Yuval Pinter\. 2023\.[Incorporating context into subword vocabularies](https://doi.org/10.18653/v1/2023.eacl-main.45)\.In*Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pages 623–635, Dubrovnik, Croatia\. Association for Computational Linguistics\.
- Zhang et al\. \(2020\)Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich\. 2020\.[Improving massively multilingual neural machine translation and zero\-shot translation](https://doi.org/10.18653/v1/2020.acl-main.148)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 1628–1639, Online\. Association for Computational Linguistics\.
- Zouhar et al\. \(2023\)Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell\. 2023\.[Tokenization and the noiseless channel](https://doi.org/10.18653/v1/2023.acl-long.284)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5184–5207, Toronto, Canada\. Association for Computational Linguistics\.

## Appendix ACue Replacement Mappings

![[Uncaptioned image]](https://arxiv.org/html/2607.17689v1/figures/cue_table.png)Table 8:Mapping from English ASCII characters to language\-specific Unicode cue replacements\. EN denotes the English\-side cue, and X denotes the cue used for the non\-English language in the pair\.[Table 8](https://arxiv.org/html/2607.17689#A1.T8)presents the full mapping between English ASCII characters and those for other languages\.

The language cues were selected from the Extended Latin Unicode block\. This choice was made for practical reasons related to SentencePiece training\. First, these characters remain compatible with SentencePiece normalization\. Second, they make it possible to keep the split\_by\_unicode\_script argument set to true, which is preferred in order to avoid mixing characters from unrelated scripts within the same token\. At the same time, because the cues belong to a Latin\-based block, they can still participate in merge operations during tokenization training\.

## Appendix BEarth Mover’s Distance

Earth mover’s distance \(EMD\) is used for comparing two distributions where the “distances” between the points on the supports are meaningful\. For geographic histograms, this is the most natural: euclidean distance dictates the amount of effort needed to “move earth” from one location to another; EMD then minimizes the total effort needed given the complete distributions\. For less\-trivial cases like ours, custom distance functions can be defined\. The function we implement ford​\(x,y\)d\(x,y\)is given in[Table 9](https://arxiv.org/html/2607.17689#A2.T9), intended to distance bins that have different counts of identical tokenizations and bring closer the bins where the multilingual vocabulary behaves regularly\.

Table 9:Earth Mover’s Distance between tokenizer vocabulary consistency bins\.
## Appendix CIntrinsic Results on All Language Pairs

Figures[5](https://arxiv.org/html/2607.17689#A3.F5),[6](https://arxiv.org/html/2607.17689#A3.F6),[7](https://arxiv.org/html/2607.17689#A3.F7), and[8](https://arxiv.org/html/2607.17689#A3.F8)present the fertility comparisons for English and Spanish, Italian, Swedish, and Romanian, respectively\.

![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_es_BPE.png)

![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_es_UNI.png)

Figure 5:Token fertility comparison between baseline and cue\-based multilingual tokenizers for EN–ES, using BPE \(top\) and UnigramLM \(bottom\)\.![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_it_BPE.png)

![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_it_UNI.png)

Figure 6:Token fertility comparison between baseline and cue\-based multilingual tokenizers for EN–IT, using BPE \(top\) and UnigramLM \(bottom\)\.![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_se_BPE.png)

![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_se_UNI.png)

Figure 7:Token fertility comparison between baseline and cue\-based multilingual tokenizers for EN–SE, using BPE \(top\) and UnigramLM \(bottom\)\.![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_ro_BPE.png)

![Refer to caption](https://arxiv.org/html/2607.17689v1/appendix_images/03_fertility_train_en_ro_UNI.png)

Figure 8:Token fertility comparison between baseline and cue\-based multilingual tokenizers for EN–RO, using BPE \(top\) and UnigramLM \(bottom\)\.
## Appendix DGPT Experimental Setup

The parameters used for the language model experiments are presented in[Table 10](https://arxiv.org/html/2607.17689#A4.T10)\.

Table 10:GPT training and model configuration parameters\.
## Appendix ENMT Experimental Setup

All machine translation experiments were conducted using the fairseq\-based\(Ott et al\.,[2019](https://arxiv.org/html/2607.17689#bib.bib18)\)implementation described byLee et al\. \([2025](https://arxiv.org/html/2607.17689#bib.bib12)\)\. The only modification made to fairseq was in the post\-processing of the validation and test outputs, so that BLEU would be computed on the correctly reconstructed text\. Final BLEU scores were computed with SacreBLEU after detokenization, matching the post\-processing pipeline used for the generated outputs\. Apart from this step, the training pipeline followed the standard fairseq workflow, with parameters provided in[Table 11](https://arxiv.org/html/2607.17689#A5.T11)\. The tokenizer used in all experiments was SentencePiece with the standard settings\. The translation architecture used in all experiments was transformer\_iwslt\_de\_en\. As a result, each model used a single tokenizer with a shared vocabulary of size 8,000, trained either with BPE or with UnigramLM\. Training took between 2 and 4 hours for all models on a given setup \(baseline / cues×\\timesBPE / Unigram\) on an NVidia GeForce RTX 4090 GPU server\.

Table 11:Fairseq model hyperparameters used in the experiments \(Encoder and Decoder\)\.
## Appendix FBootstrap MT

Tables[12](https://arxiv.org/html/2607.17689#A6.T12)and[13](https://arxiv.org/html/2607.17689#A6.T13)present full results for the bootstrap tests on BPE and UnigramLM, respectively\.

Table 12:Bootstrap BLEU results for BPE across all language pairs and evaluation sets\. Bold values mark the better BLEU mean between the baseline and cued systems\.Table 13:Bootstrap BLEU results for UnigramLM across all language pairs and evaluation sets\. Bold values mark the better BLEU mean between the baseline and cued systems\.