What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?

# What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?
Source: [https://arxiv.org/html/2608.17325](https://arxiv.org/html/2608.17325)
###### Abstract

Tokenization is a fundamental component of language modeling pipelines\. Despite its importance, it is often fixed, even though it significantly impacts model performance across languages\. In this work, we analyze what tokens are learned when tokenization is jointly optimized with language modeling\. We comparetokenizer\-freeapproaches such asSSLMs andH\-Nets with fixed tokenizers across 18 typologically and script\-diverse languages\. Our results show that joint optimization fundamentally alters token structure\.SSLMs recover morphologically aligned and contextually efficient tokens, whereasH\-Nets prioritize byte\-level efficiency, producing longer tokens with very low overlap with standard subword vocabularies\. We further show that tokenization behavior varies across language typologies\. Agglutinative languages exhibit more dynamic segmentation patterns while learning\. Through downstream evaluation, with pretrained\-then\-finetuned BERT models, we find thatSSLM\-based pretokenization consistently reduces language modeling perplexity and achieves competitive downstream performance despite distinct vocabularies\. Overall,tokenizer\-freeapproaches optimize for contextual and computational efficiency rather than strict morphological structure, resulting in fundamentally different yet effective vocabularies for downstream NLP\.

## 1Introduction

Tokenization is a fundamental preprocessing step in language models \(LMs\)\. In transformer\-based LMs[41](https://arxiv.org/html/2608.17325#bib.bib6), it is often performed by learning a tokenizer through subword algorithms such as Byte\-pair Encoding \(BPE,[14](https://arxiv.org/html/2608.17325#bib.bib11);[36](https://arxiv.org/html/2608.17325#bib.bib1)\), Unigram Language Model \(ULM,[20](https://arxiv.org/html/2608.17325#bib.bib2)\), or WordPiece \(WPC,[35](https://arxiv.org/html/2608.17325#bib.bib12)\)\. These algorithms are built on simple statistical priors, such as the assumption that frequently co\-occurring characters should be part of the same token[36](https://arxiv.org/html/2608.17325#bib.bib1)\. Despite their widespread adoption, it is unclear whether such approaches provide best performance, especially when working with languages across diverse morphological complexities, typologies, and scripts\. This concern is further amplified from the standardtokenization\-LM\-detokenizationpipeline, where a tokenizer is learned independently of language model training and remains static\. We refer to these asfixed\-tokenizerapproaches\. Assessing the best tokenizer for such LMs is both difficult and expensive, since the tokenizer remains fixed prior to training and cannot be modified thereafter\.

Figure 1:Morphological alignment evaluation oftokenizer\-free\(H\-Nets,SSLMs\) and fixed\-tokenizer approaches\.SSLMs show consistently higher alignment, whereasH\-Nets trade morphological alignment for longer, computationally efficient tokens\.Various modifications and novel algorithms have been proposed to learn a tokenizer\. This includes advances in pre\-tokenization[6](https://arxiv.org/html/2608.17325#bib.bib37);[17](https://arxiv.org/html/2608.17325#bib.bib17), robustness[19](https://arxiv.org/html/2608.17325#bib.bib13), morphological alignment[16](https://arxiv.org/html/2608.17325#bib.bib32);[21](https://arxiv.org/html/2608.17325#bib.bib14);[45](https://arxiv.org/html/2608.17325#bib.bib15), vocabulary pruning[11](https://arxiv.org/html/2608.17325#bib.bib18)or modifications[10](https://arxiv.org/html/2608.17325#bib.bib33), and cross\-token111Tokens that can span multiple words, for example “by the way”\.approaches[34](https://arxiv.org/html/2608.17325#bib.bib16);[23](https://arxiv.org/html/2608.17325#bib.bib19)\. However, no single approach can be concluded to be the “best”, as conclusions depend on various factors such as: 1\) languages chosen and their morphological properties, 2\) experimental setup like model architecture, dataset size and other hyperparameters[46](https://arxiv.org/html/2608.17325#bib.bib26);[31](https://arxiv.org/html/2608.17325#bib.bib10)\.

Recently, new architectures have been proposed that overcome the aforementionedtokenization\-LM\-detokenizationpipeline\. These are often referred to astokenizer\-freeLMs\. Architectures such as Dynamic Token Pooling Transformers[27](https://arxiv.org/html/2608.17325#bib.bib20)andH\-Nets[18](https://arxiv.org/html/2608.17325#bib.bib3)incorporate data\-dependent boundary prediction \(i\.e\., tokenization\) and byte\-level language modeling into a single end\-to\-end pipeline\.H\-Nets, in particular, provide a stable architecture\. On the other hand, models such as subword segmental language model \(SSLM,[24](https://arxiv.org/html/2608.17325#bib.bib4);[25](https://arxiv.org/html/2608.17325#bib.bib5)\) use long short\-term memory \(LSTMs,[15](https://arxiv.org/html/2608.17325#bib.bib24)\) or transformers[41](https://arxiv.org/html/2608.17325#bib.bib6)to marginalize and optimize over all possible segmentations\. Despite being computationally heavy and unfeasible at large scale,SSLMs provide us with a joint architecture which we can utilize for analysis\. Thesetokenizer\-freeapproaches avoid any statistical or heuristic priors, thus providing us with a tokenization that is jointly learned with the LM\.

In this work, we investigate “what tokens are learned when tokenization is optimized jointly with language modeling\.” Our contributions:

1. 1\.We analyzehowandwhattokens are learned throughtokenizer\-freeapproaches, namelySSLMandH\-Net, by evaluating their intrinsic and linguistic properties across a diverse set of languages\.
2. 2\.We compare these approaches withfixed\-tokenizermethods, such asBPEandULM, to assesshow similartheir learned vocabularies are\.
3. 3\.We evaluate the impact of these tokenization approaches on downstream task performance to assesswhetherjointly learned tokenizers offer improvements\.

## 2Related Work

Prior work analyzing tokens learned bytokenizer\-freeLMs remains limited\. Recently,[25](https://arxiv.org/html/2608.17325#bib.bib5)studied the learning dynamics of tokens learned bySSLMs[24](https://arxiv.org/html/2608.17325#bib.bib4)\. They tracked subword learning dynamics from a linguistic perspective by evaluating morphological alignment, productivity, idiosyncrasy, and fertility, and identified distinct stages of subword learning\. However, because their study focused on learning dynamics, the analysis was limited to three languages\. Moreover, critical confounding factors such as dataset size and disparities across languages were not accounted for when drawing conclusions across languages\.

On the other hand, many studies have analyzed fixed tokenizers and their modifications\. These studies have largely focused on the impact of subword algorithms on downstream LM performance\. For example,[8](https://arxiv.org/html/2608.17325#bib.bib28)and[42](https://arxiv.org/html/2608.17325#bib.bib29)compared different algorithms, showing thatULMperforms better on downstream tasks and recovers subwords that are more morphologically aligned\. While, works such as[40](https://arxiv.org/html/2608.17325#bib.bib40)argued that inference strategy can matter as or more than the tokenizer construction algorithm\.

## 3Methodology

### 3\.1Languages and Datasets

Table 1:List of typologically and script diverse languages included in this study\. Detailed version of this table can be found in Table[11](https://arxiv.org/html/2608.17325#A2.T11)of Appendix\.Languages vary widely in morphology, typology, and script, which may affect the best tokenization approach\. We therefore study a broad, representative set of 18 languages across these diversity, listed in Table[1](https://arxiv.org/html/2608.17325#S3.T1)\(detailed version in Table[11](https://arxiv.org/html/2608.17325#A2.T11)of Appendix\)\.

An ideal comparison across languages requires parallel training and test data, which is difficult to obtain at this scale\. We therefore use non\-parallel monolingual corpora while carefully controlling the data source and size\. To balance data sizes, we extract 250,000 English sentences and measure their size in bytes\. Then, for other languages, we scale the data size according tobyte\-premiums222Byte premium \(BP\) is the ratio of bytes required to encode a comparable amount of information or parallel text in one language relative to another language \(typically designated as English\)\.\(BP\)[4](https://arxiv.org/html/2608.17325#bib.bib23)and extract the desired number of sentences333This is necessary for our comparative analysis involving UTF\-8 encoding, as it is known to produce disparities across languages with diverse scripts[4](https://arxiv.org/html/2608.17325#bib.bib23)\.\. We use WMT News Crawl corpora444[https://data\.statmt\.org/news\-crawl/](https://data.statmt.org/news-crawl/)[9](https://arxiv.org/html/2608.17325#bib.bib38), fixing the domain to News for most languages, falling back to the NLLB corpus[38](https://arxiv.org/html/2608.17325#bib.bib25)where unavailable\. Statistics of the resultant pretraining data along with available morphologically annotated data is listed in Table[12](https://arxiv.org/html/2608.17325#A2.T12)of the Appendix\.

### 3\.2Tokenization Approaches

We include two distincttokenizer\-freeapproaches from recent literatures:

- •Transformer\-based version of subword segmental language model;SSLM[25](https://arxiv.org/html/2608.17325#bib.bib5); that marginalize over all possible segmentations while jointly optimizing for language modeling\.
- •End\-to\-endH\-Nets[18](https://arxiv.org/html/2608.17325#bib.bib3)that jointly perform data\-dependent boundary prediction along with byte\-level language modeling\.

To compare these withfixed\-tokenizerapproaches, we include both standard and improved algorithms\. Standard algorithms includeBPE[36](https://arxiv.org/html/2608.17325#bib.bib1),ULM[20](https://arxiv.org/html/2608.17325#bib.bib2), andWPC[35](https://arxiv.org/html/2608.17325#bib.bib12)\. We also studySaGe[44](https://arxiv.org/html/2608.17325#bib.bib30), which prefers subword units occurring in fewer distinct contexts relative to their frequency, and unsupervised morphological tokenizers such asMorfessor[37](https://arxiv.org/html/2608.17325#bib.bib36)\. We further include methods that modify inference \(such asBPE\-dropout[32](https://arxiv.org/html/2608.17325#bib.bib31)andPathPiece[33](https://arxiv.org/html/2608.17325#bib.bib21)\), pre\-tokenization \(such asMorphBPE[6](https://arxiv.org/html/2608.17325#bib.bib37)\), encoding \(such asMYTE[22](https://arxiv.org/html/2608.17325#bib.bib34)\) and vocabulary construction \(such asPickyBPE[10](https://arxiv.org/html/2608.17325#bib.bib33)\)\. We further include cross\-token approaches such asSuperBPE[23](https://arxiv.org/html/2608.17325#bib.bib19)andBoundlessBPE[34](https://arxiv.org/html/2608.17325#bib.bib16), which have been shown to be more efficient\. Table[13](https://arxiv.org/html/2608.17325#A2.T13)in Appendix describes these approaches and §[B\.3](https://arxiv.org/html/2608.17325#A2.SS3)lists their hyperparameters\.

### 3\.3Evaluation

We evaluate both intrinsic properties and extrinsic performance of the mentioned tokenization approaches\. We consider intrinsic metrics such ascontextual exponenceandeffective vocabulary size, as well as linguistic metrics such as morphological alignment\. Table[14](https://arxiv.org/html/2608.17325#A2.T14)in the Appendix provides a comprehensive description of the metrics we evaluate\. This broad intrinsic evaluation provides us with a meaningful assessment of the tokens produced by each tokenization approach\.

To measure the impact of each tokenization approach on downstream performance, we pretrain BERT[12](https://arxiv.org/html/2608.17325#bib.bib27)models555Note that this requires treatingtokenizer\-freeapproach as pretokenization\. We discuss the reasons behind this in §[4\.4](https://arxiv.org/html/2608.17325#S4.SS4)\.at a scale of 12M parameters666We perform an ablation study at different model scales and discuss the results in §[4\.4](https://arxiv.org/html/2608.17325#S4.SS4)and in the Appendix\.and finetune them on tasks: Sentiment Analysis, POS Tagging, Named Entity Recognition \(NER\), and Dependency Parsing\. We limit the downstream evaluation to three typologically distinct languages: English, Hindi, and Telugu; to keep the computational cost manageable\.

### 3\.4Experimental Setup

We take a backward approach from the ideal experimental setup as noted by[31](https://arxiv.org/html/2608.17325#bib.bib10), allowing us to better isolate the effect of the tokenizer\. We trainSSLMs,H\-Nets, andfixed\-tokenizeralgorithms on the train subset of the dataset\. Training is monitored on the validation subset, and all tokenization approaches are evaluated on the test subset\. Total parameters count ofH\-Netmodels remains approximately 3M and that ofSSLMs at 2M\. This ensures that the ratio of token\-to\-parameter count is at least 2 across languages \(see Table[12](https://arxiv.org/html/2608.17325#A2.T12)\)\. WhileH\-Nets do not require any vocabulary for initialization,SSLMs require a fixed lexicon provided beforehand\. We consider10,00010,000most frequent words for this, following[25](https://arxiv.org/html/2608.17325#bib.bib5), and set maximum token length777Measured in terms of unicode character length, and not as length in raw UTF\-8 bytesas55\. For downstream evaluation, we scrape a larger dataset containing 10M sentences from same data source; NewsCrawl[9](https://arxiv.org/html/2608.17325#bib.bib38)\. This ensures a reasonable data size for 2M\-30M parameter BERT models which we analyze\. Detailed hyperparameters for each models are listed in the Appendix[B](https://arxiv.org/html/2608.17325#A2)\.

EnglishHebrewHindiTamilHungarianIndonesianreapingמשטרתייםकार्यकर्ताओंவகுத்துbrigádokkalceritakanreap ingמשטרתי יםकार्यकर्ता ओंவகு த்துbrigád okkalcerita kanSegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1rea ping0משט רתיים0का र्यकर ्ताओं0வக ுத்து0bri g ádo kkal0ceri takan0re ap ing0\.67מ שטר תי ים0\.5कार्य कर्ता ओं0\.67வகு த்து1br igá d okkal0\.5c erita kan0\.67reap ing1משטר תי ים0\.67कार्य कर्ता ओं0\.67வகு த்து1br igá dok kal0c erita kan0\.67reap ing1משטר ת י ים0\.5कार्य कर्ता ओं0\.67வ குத்த ு0b rigád okkal0\.67c erita kan0\.67reap ing1משטר ת י ים0\.5कार्य कर्ता ओं0\.67வ குத்த ு0b rigád okkal0\.67c erita k an0\.5Table 2:Evolution of segmentation of few candidate wordforms produced bytokenizer\-freeapproachSSLMwith morphological alignment F1\-scores\. English, Hebrew, and Hindi are fusional or analytical languages, while Tamil, Hungarian, and Indonesian are agglutinative languages\. More examples in Table[10](https://arxiv.org/html/2608.17325#A2.T10)in Appendix\.

## 4Experiments & Results

### 4\.1Q1: How are the tokens learned?

We analyze the learning dynamics ofSSLMs similar to[25](https://arxiv.org/html/2608.17325#bib.bib5), but with broader language coverage, script variation, and, more importantly, dataset sizes adjusted according tobyte\-premiums[4](https://arxiv.org/html/2608.17325#bib.bib23)\. We evaluate and track morphological alignment against MorphScore[3](https://arxiv.org/html/2608.17325#bib.bib7);[5](https://arxiv.org/html/2608.17325#bib.bib8)data, along with intrinsic properties such as effective vocabulary size and number of distinct neighbouring tokens, i\.e\., contextual exponence[44](https://arxiv.org/html/2608.17325#bib.bib30)888Refer to Table[14](https://arxiv.org/html/2608.17325#A2.T14)in Appendix for brief description of these metric\.\. Our results confirm the findings of[25](https://arxiv.org/html/2608.17325#bib.bib5)with inclusion of more diverse languages\.

#### Morphological Alignment

Figure[2](https://arxiv.org/html/2608.17325#S4.F2)plots the variation in morphological alignment of tokens produced bySSLM, as its training progress, for a subset of languages; a similar plot containing all 18 languages is plotted in figure[9](https://arxiv.org/html/2608.17325#A2.F9)in Appendix\.

Findings:Across all languages, we observe four distinct phases in the morphscore dynamics: 1\)Rapid initial evolutionof morphemes, 2\)Fluctuationfor a brief period which is more consistent and pronounced in case of Agglutinative languages, 3\)Convergenceto a fixed alignment, and 4\)Saturation, where alignment remains static implying convergence to a solution\. Overall, higher alignment in Templatic and Fusional languages as compared to Agglutinative languages\. This implies that a jointly learned tokenizer effectively identifies the root\-and\-pattern structures \(as in Hebrew\) and the inflectional suffixes \(as in Hindi\) very early in training\. Despite being highly synthetic, Tamil converges to comparatively much lower F1\-score \(≈0\.3\\approx 0\.3\)\. Interestingly, the tokens reach mid\-range alignment, but then regresses to a lower alignment\. This is observed in both Tamil and Indonesian\. Hungarian also shows similar trends like Tamil, but converges to higher alignment following another fluctuation\. English, on the other hand, reaches a stable, mid\-range alignment \(≈0\.5\\approx 0\.5\) without any major fluctuations\. This may reflect its relatively simple morphology\.

Figure 2:Evolution of morphological alignment \(F1\-score\) oftokenizer\-freeapproachSSLMshows that agglutinative languages show larger fluctuations compared to other languages\.We identify and list candidate data\-points from morphscore dataset \(see Table[2](https://arxiv.org/html/2608.17325#S3.T2)\), which depicts the overall trend\. For Analytic and Fusional languages, we observe how quickly the jointly learned tokenizer identifies the morphemes\. In English, it instantly progresses from nonsensical split \(“rea ping”, at step 2200\) to a perfect morphologically aligned segment \(“reap ing”, at step 2950\)\. Similarly in Hindi, it converges to a stable segmentation \(“कार्य\+कर्ता\+ओं”, at step 800\) capturing the core semantic units\. This implies that in languages where word structures are comparatively less complex, it converges on a stable “lexicon” of subwords\. However, in Agglutinative languages such as Tamil, it reaches a perfect alignment \(“வகு\+த்து”, at step 450\), but then converges on an inaccurate split \(“வ\+குத்த\+ ு”, at step 7150\), implying that the jointly learned tokenizer found it beneficial to trade\-off morphological alignment in such languages\.

#### Intrinsic Properties

We perform similar analysis for intrinsic properties such as contextual exponence and effective vocabulary size on test split\.

Findings:Figure[3\(a\)](https://arxiv.org/html/2608.17325#S4.F3.sf1)and[3\(b\)](https://arxiv.org/html/2608.17325#S4.F3.sf2)reveal script\-wise similarities in dynamics\. Latin\-script languages such as English, Hungarian, and Indonesian show largest effective vocabulary sizes, reaching between14,00014,000to15,50015,500unique tokens\. This suggests that jointly learned tokenizer combines frequent character sequences into larger dedicated units to optimize contextual efficiency\. We thereby see a steep drop in contextual exponence in these languages\.

\(a\)Contextual Exponence\(b\)Effective Vocabulary size
Figure 3:Evolution of number of distinct neighbouring tokens \(i\.e\., contextual exponence\) and effective vocabulary size oftokenizer\-freeapproachSSLMshows a script\-wise similarity\. Latin languages saturates at lower contextual exponence through higher effective vocabulary size\.Among the subset of languages here, non\-Latin languages such as Tamil and Hindi show smaller effective vocabulary size and higher contextual exponence\. This suggests that joint tokenizer prefers to learn a small, flexible set of fundamental morphemes rather than memorizing long, specific wordforms in such scripts\. Dravidian\-script languages such as Tamil combine many parts together, so one learned subword can still work with many different neighboring units because words often have many added affixes\. This is observed through higher contextual exponence and lower vocabulary size\. In contrast, Hebrew reaches its stable effective vocabulary size much earlier than other scripts\. This shows again the ability of jointly learned tokenizers to quickly identify the root\-and\-pattern \(introflexive\) structures inherent in the script\.

### 4\.2Q2: What tokens are learned?

In this section, we evaluate bothtokenizer\-freeapproaches999We consider the tokens corresponding to the best checkpoint, i\.e\., the checkpoint corresponding to the least validation loss\.;H\-Nets andSSLMs; and compare them withfixed\-tokenizerapproaches mentioned in §[3\.2](https://arxiv.org/html/2608.17325#S3.SS2)\. We focus on metrics listed in Table[14](https://arxiv.org/html/2608.17325#A2.T14)of Appendix\.Note:H\-Nets andSSLMs can, by design, produce fundamentally different types of tokens;H\-Nets can produce cross\-tokens \(or superwords\) frequently, whileSSLMs are bounded to produce only subwords\. This is reflected in the further results and discussions\.

#### Morphological Alignment

Findings:As observed in figure[1](https://arxiv.org/html/2608.17325#S1.F1), we identify a significant difference among tokens produced by the twotokenizer\-freeapproaches\.SSLMs trade\-off computational efficiency for high linguistic fidelity, whileH\-Nets prioritize byte\-level computational efficiency at the direct expense of morphological alignment\. This can be observed byH\-Net’s low morphscore\-F1 ranging below0\.10\.1across all languages\. On the other hand, joint optimization approach \(i\.e\.,SSLM\) showcase significantly higher alignment ranging from≈0\.26\\approx 0\.26in Telugu to≈0\.65\\approx 0\.65in Hindi\. Notably,SSLMs show consistently higher alignment as compared to popular subword algorithms such asBPEorWPC, even converging to the most aligned tokens in languages such as Malayalam and Turkish\. However in other languages it tends to align less compared to approaches such asULM,MorphULM,MYTE, orMorfessor\. In Hebrew, despite suchfixed\-tokenizerapproaches providing nearly perfect alignment with morphology,SSLMs show comparatively lower alignment \(≈0\.63\\approx 0\.63\)\. These findings suggest that the desirable morphological alignment of jointly learned tokenizer is indeed dependent on a language’s morphological properties\.

#### Intrinsic Properties

Findings:As observed in figure[4\(a\)](https://arxiv.org/html/2608.17325#S4.F4.sf1), bothtokenizer\-freeapproaches exhibit higher fertility thanfixed\-tokenizerbaselines, especially in agglutinative languages such as Malayalam, Tamil, and Telugu; this effect is strongest inH\-Nets \(≈5\.0\\approx 5\.0to≈5\.5\\approx 5\.5\), while analytic or fusional languages remain below≈2\.0\\approx 2\.0\.

\(a\)Fertility\(b\)Contextual exponence\(c\)Mean Token Length
Figure 4:Intrinsic evaluation \(fertility, contextual exponence, and mean token length\) oftokenizer\-freeapproaches \(SSLMs andH\-Nets\) andfixed\-tokenizerapproaches\.Tokenizer\-freeapproaches show comparatively higher fertility and lower contextual exponence\. Mean token length ofH\-Nets are significantly higher for non\-Latin languages\.Across languages,tokenizer\-freeapproaches consistently yield lower contextual exponence thanfixed\-tokenizermethods \(see figure[4\(b\)](https://arxiv.org/html/2608.17325#S4.F4.sf2)\), withSSLMandH\-Netaveraging less than7575neighbouring tokens\. This likely arises because minimizing language\-modeling loss favors units with more predictable grammatical roles and fewer valid neighbours, improving semantic quality and downstream performance[44](https://arxiv.org/html/2608.17325#bib.bib30)\.

![Refer to caption](https://arxiv.org/html/2608.17325v1/vocab_overlap_combined_agglutinative.png)\(a\)Agglutinative Languages
![Refer to caption](https://arxiv.org/html/2608.17325v1/vocab_overlap_combined_fusional_analytic.png)\(b\)Analytic and Fusional Languages

Figure 5:Jaccard token overlap betweentokenizer\-freeapproaches such asSSLMandH\-Net, and fixed\-tokenizer approaches on the test split\. The lower triangle of the heatmap denotes the Jaccard overlap, while the upper triangle denotes the standard deviation of the overlap\.Interestingly, mean token length reveals strong divergence betweentokenizer\-freeapproaches \(see figure[4\(c\)](https://arxiv.org/html/2608.17325#S4.F4.sf3)\), especially across scripts\. In non\-Latin languages,H\-Nets produce exceptionally long tokens \(≈17\\approx 17to18\.518\.5characters\), likely because either data\-dependent boundary prediction favors long phrases to minimize byte\-level loss or struggles to identify boundaries in complex scripts\. In contrast,SSLMs and most fixed tokenizers maintain shorter, stable lengths \(≈2\.5\\approx 2\.5to5\.05\.0characters\) which follows from its design of explicit maximum token\-length constraints\.

### 4\.3Q3: How similar are the learned tokens?

\(a\)English \(Analytic\)\(b\)Telugu \(Agglutinative\)\(c\)Hindi \(Fusional\)
Figure 6:Variation in the validation perplexity of BERT models during pretraining\. Models trained with tokenizers usingSSLMas pretokenization consistently achieve the lowest perplexity\.TokenizerSentiment Analysis \(Acc\)POS Tagging \(F1\)Named Entity Recognition \(F1\)Dependency Parsing \(LAS\)Languageeng\_latntel\_teluhin\_devaeng\_latntel\_teluhin\_devaeng\_latntel\_teluhin\_devaeng\_latntel\_teluhin\_devaStandard AlgorithmsBPE85\.5562\.8576\.6795\.0988\.4197\.1589\.9894\.7290\.6975\.6466\.3188\.20UnigramLM83\.6164\.8877\.6395\.1189\.7297\.2089\.6194\.8091\.0875\.9568\.8888\.73WordPiece84\.8665\.9176\.1095\.2289\.5197\.3488\.3795\.3390\.7378\.1166\.3188\.63Contextual AlgorithmsSaGe85\.3263\.2273\.6191\.6881\.9991\.6783\.3291\.8888\.3262\.1755\.1470\.07Inference ModificationsBPE\-dropout80\.2862\.6672\.8593\.0683\.5095\.3784\.7092\.7889\.4168\.3859\.0683\.47PathPiece84\.7564\.3277\.0692\.2787\.1095\.0883\.8992\.6287\.0462\.7658\.7677\.47Vocabulary ModificationsPickyBPE73\.1660\.4458\.7078\.7476\.5777\.5468\.9688\.6184\.2749\.9645\.7756\.66Cross\-token AlgorithmsBoundlessBPE83\.4965\.2575\.1491\.9883\.5095\.2584\.8493\.6287\.1561\.6557\.5578\.08SuperBPE83\.8364\.5174\.7693\.8285\.1796\.1788\.9393\.0588\.2168\.3262\.6980\.85Pre\-tokenization using MorfessorMorphBPE86\.2463\.9676\.1095\.1889\.1297\.2490\.6394\.6691\.0275\.5669\.1888\.11MorphULM85\.8966\.9177\.0695\.2889\.3897\.3390\.0394\.9090\.5275\.2869\.0388\.24MorphWPC87\.3867\.2875\.9195\.2690\.0397\.2388\.1094\.9590\.7577\.6569\.0388\.35Pre\-tokenization using SSLMSSLM\-BPE85\.66④64\.70⑦77\.06④95\.09⑥88\.42⑧97\.18⑥88\.93⑤94\.38⑨91\.03④74\.90⑧66\.77⑦87\.47⑧SSLM\-ULM84\.17⑨63\.40⑪76\.10⑧95\.06⑧90\.17①97\.16⑧88\.88⑥94\.58⑧91\.30②73\.35⑨68\.58⑥86\.95⑨SSLM\-WPC84\.75⑩65\.25⑤74\.38⑫94\.95⑨90\.10②97\.17⑦87\.05⑨94\.84⑥91\.41①76\.78③68\.73⑤87\.51⑦Table 3:Downstream performance of 12M\-parameter BERT models trained with different tokenizer variants\. Best results for each task\-language pair are inbold; superscripts denote tokenizer ranks with SSLM pretokenization\. Scores are averaged over three finetuning runs \(seeds 42, 43, 44\)\. We report accuracy for Sentiment Analysis, F1 for POS tagging and NER, and LAS for Dependency Parsing\. Variants improved bySSLMpretokenization areunderlined\.We measure pairwise Jaccard overlap on the test split to compare tokens learned bytokenizer\-freeandfixed\-tokenizerapproaches across language typologies\. Figure[5](https://arxiv.org/html/2608.17325#S4.F5)shows the corresponding heatmap\.

Findings:H\-Nets exhibit near\-zero overlap with fixed tokenizers across typologies, indicating that end\-to\-end data\-dependent boundary prediction learns token sets fundamentally different from frequency\-based subword methods by prioritizing byte\-level sequence efficiency over standard subword boundaries\.SSLMs, on the other hand, show comparatively higher \(though still low\) overlap with fixed tokenizers, particularly with frequency\-based methods such asBPEandWPC; the strongest similarity occurs betweenSSLMs andBPEvariants across all typologies\. Overlap is lowest for agglutinative languages \(≈0\.12\\approx 0\.12to0\.180\.18Jaccard\), suggesting that productive morphology generates many possible subwords, encouraging joint optimization to learn smaller atomic morphemes while fixed tokenizers merge them into larger but linguistically arbitrary units\.BPE,BPE\-dropout,PathPiece, andMorphBPEform a high\-overlap cluster \(greater than0\.800\.80Jaccard\), indicating that changes to pre\-tokenization or inference do not fundamentally alter learned vocabularies\. In contrast,SaGeshows lower overlap, suggesting that prioritizing contextual exponence substantially changes the vocabulary\.

### 4\.4Q4: Are the learned tokens better?

To determine if these fundamentally distinct tokens learned throughtokenizer\-freeapproaches improve language understanding, we analyze the downstream performance of BERT[12](https://arxiv.org/html/2608.17325#bib.bib27)models pretrained with each tokenizer and finetuned for the tasks listed in §[3\.3](https://arxiv.org/html/2608.17325#S3.SS3), with results reported in Table[3](https://arxiv.org/html/2608.17325#S4.T3)\. As noted,tokenizer\-freeapproaches do not have a fixed vocabulary, while BERT models require static, fixed, pre\-defined vocabulary\. Therefore, we usetokenizer\-freeapproaches101010We avoid considering pretokenization withH\-Nets, as it is observed that it mostly segments at whitespaces and it is not straightforward to limit and fix its vocabulary due to its cross\-token flexibility\.as pretokenization and learn a standard fixed tokenizer on the pre\-segmented corpus\. This approach fixes the model architecture and provides a fair downstream comparison\. We refer to these with prefixSSLM, i\.e\.,SSLM\-BPE,SSLM\-ULM, andSSLM\-WPC\.

Findings:As illustrated in Figure[6](https://arxiv.org/html/2608.17325#S4.F6)and summarized in Table[4](https://arxiv.org/html/2608.17325#S4.T4),tokenizer\-freeapproaches consistently achieve the lowest validation perplexity across all three evaluated languages and converge earlier compared to fixed tokenizers\. Only in Hindi, we observeSaGeoutperformingtokenizer\-freeapproaches with nearly perfect perplexity of11\. This improvement in perplexity is even more pronounced as compared to other pretokenization\-based modifications such as that ofMorfessor\. We also observe that Telugu has the highest perplexity without pretokenization, but the lowest perplexity when thetokenizer\-freeSSLMis used for pretokenization\.

Table 4:Aligning tokens learned by fixed\-tokenizer approaches with that learned bySSLMthrough pretokenization significantly reduces the perplexity, showcasing improvement in language modeling\. The scores are averages of perplexities acrossBPE,ULM, andWPC\.The downstream results \(see Table[3](https://arxiv.org/html/2608.17325#S4.T3)above\) showcase the practical implications of each approach\. Despite having fundamentally distinct vocabulary,tokenizer\-freeapproaches remain competitive\. In contrast, the performance of many modifiedfixed\-tokenizerapproaches such asSaGe,PickyBPE, andBPE\-dropoutremains inconsistent and often significantly lower across different tasks and languages\. Notably,SSLM\-ULMachieves the highest F1\-score for POS tagging in Telugu \(90\.17%90\.17\\%\), whileSSLM\-WPCachieves highest for NER in Hindi \(91\.41%91\.41\\%\)\. Pretokenization withMorfessorremains the best performing approach overall\. It consistently improves the performance over standard approaches\. Overall, the results suggest that jointly learned tokenizers learn meaningful tokens that transfer effectively to downstream NLP tasks\.

We additionally perform an ablation study across different model scales and observe consistent trends\.SSLM\-based pretokenization continues to achieve lower validation perplexity and competitive downstream performance across scales, although the relative gains reduce slightly for larger models\. Detailed experimental setup, hyperparameters, and complete results are provided in the §[A](https://arxiv.org/html/2608.17325#A1)of Appendix\.

## 5Conclusion

In this work, we showed that jointly learned tokenization fundamentally changes the structure and behavior of learned tokens across typologically diverse languages\. In particular,SSLMs learn more morphologically aligned and contextually efficient segments than standard frequency\-based tokenizers, whileH\-Nets prioritize byte\-level efficiency and produce substantially different token structures\. Although morphology\-aware pretokenization remains strongest overall,SSLM\-based pretokenization consistently remains competitive on downstream performance\. Overall, our findings show that jointly optimized tokenizers can learn linguistically meaningful token structures that remain effective for downstream NLP despite having substantially different vocabularies from standard tokenization approaches\.

## Limitations

While we do our best to handle confounding factors such as dataset sizes, model sizes, and other experimental setup, few limitations remain:

1. 1\.Model and dataset sizes: We focused on spanning an extensive set of tokenization approaches, languages, and their combinations\. Therefore, our analysis was limited to models and dataset at relatively smaller scale\. For instance, we limittokenizer\-freeLMs to 3M parameters and dataset of 250,000 sentences\. Downstream evaluation was performed on BERT models at 12M parameters and dataset of 10M sentences\. Although we perform an ablation study withBPEandSSLM\-BPEtokenization approaches, we leave scalability analysis of our findings as future work\. In this work, we trade\-off model and dataset sizes for larger coverage of languages and tokenization approaches\.
2. 2\.Generalizability of downstream results: For every combination of tokenization approach and language, we had to pretrain a model\. Hence, our downstream evaluation was limited to three typologically distinct languages: English, Hindi, and Telugu\. We do not evaluate downstream performance across all 18 languages, in order to keep computational cost manageable\. We focus on evaluating all tokenization approaches for these selected languages in a controlled experimental setting\.
3. 3\.Hyperparameters oftokenizer\-freeLMs: We do not sweep over certain hyperparameters oftokenizer\-freeLMs, for example, maximum token length inSSLM\. We fix them following heuristics from previous works, since sweeping over these hyperparameters require significant computation resources, which we limit\.

## Ethics Statement

This work focuses on the analysis of tokenization strategies across languages and does not involve human subjects or the use of sensitive personal data\. All experiments are conducted on publicly available text corpora\. However, differences in dataset quality, and linguistic coverage across languages may introduce biases that affect the observed behavior of tokenization methods\.

## References

- Akhtaret al\.\(2016\)M\. S\. Akhtar, A\. Kumar, A\. Ekbal, and P\. BhattacharyyaA hybrid deep learning architecture for sentiment analysis\.InProceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers,Y\. Matsumoto and R\. Prasad \(Eds\.\),Osaka, Japan,pp\. 482–493\.External Links:[Link](https://aclanthology.org/C16-1047/)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Alqahtaniet al\.\(2026\)S\. Alqahtani, M\. T\. Nayeem, M\. T\. R\. Laskar, T\. Mohiuddin, and M\. S\. BariStop taking tokenizers for granted: they are core design decisions in large language models\.External Links:2601\.13260,[Link](https://arxiv.org/abs/2601.13260)Cited by:[Table 14](https://arxiv.org/html/2608.17325#A2.T14)\.
- Arnett and Bergen \(2025\)C\. Arnett and B\. BergenWhy do language models perform worse for morphologically complex languages?\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 6607–6623\.External Links:[Link](https://aclanthology.org/2025.coling-main.441/)Cited by:[Table 12](https://arxiv.org/html/2608.17325#A2.T12),[§4\.1](https://arxiv.org/html/2608.17325#S4.SS1.p1.1)\.
- Arnettet al\.\(2024\)C\. Arnett, T\. A\. Chang, and B\. BergenA bit of a problem: measurement disparities in dataset sizes across languages\.InProceedings of the 3rd Annual Meeting of the Special Interest Group on Under\-resourced Languages @ LREC\-COLING 2024,M\. Melero, S\. Sakti, and C\. Soria \(Eds\.\),Torino, Italia,pp\. 1–9\.External Links:[Link](https://aclanthology.org/2024.sigul-1.1/)Cited by:[Table 12](https://arxiv.org/html/2608.17325#A2.T12),[§3\.1](https://arxiv.org/html/2608.17325#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.17325#S4.SS1.p1.1),[footnote 3](https://arxiv.org/html/2608.17325#footnote3)\.
- Arnettet al\.\(2025\)C\. Arnett, M\. Hudspeth, and B\. O’ConnorEvaluating morphological alignment of tokenizers in 70 languages\.External Links:2507\.06378,[Link](https://arxiv.org/abs/2507.06378)Cited by:[Table 12](https://arxiv.org/html/2608.17325#A2.T12),[§4\.1](https://arxiv.org/html/2608.17325#S4.SS1.p1.1)\.
- Banerjee and Bhattacharyya \(2018\)T\. Banerjee and P\. BhattacharyyaMeaningless yet meaningful: morphology grounded subword\-level NMT\.InProceedings of the Second Workshop on Subword/Character LEvel Models,M\. Faruqui, H\. Schütze, I\. Trancoso, Y\. Tsvetkov, and Y\. Yaghoobzadeh \(Eds\.\),New Orleans,pp\. 55–60\.External Links:[Link](https://aclanthology.org/W18-1207/),[Document](https://dx.doi.org/10.18653/v1/W18-1207)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.19.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Batsurenet al\.\(2021\)K\. Batsuren, G\. Bella, and F\. GiunchigliaMorphyNet: a large multilingual database of derivational and inflectional morphology\.InProceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology,G\. Nicolai, K\. Gorman, and R\. Cotterell \(Eds\.\),Online,pp\. 39–48\.External Links:[Link](https://aclanthology.org/2021.sigmorphon-1.5/),[Document](https://dx.doi.org/10.18653/v1/2021.sigmorphon-1.5)Cited by:[Table 12](https://arxiv.org/html/2608.17325#A2.T12)\.
- Bostrom and Durrett \(2020\)K\. Bostrom and G\. DurrettByte pair encoding is suboptimal for language model pretraining\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4617–4624\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.414/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.414)Cited by:[§2](https://arxiv.org/html/2608.17325#S2.p2.1)\.
- Chelbaet al\.\(2013\)C\. Chelba, T\. Mikolov, M\. Schuster, Q\. Ge, T\. Brants, P\. Koehn, and T\. RobinsonOne billion word benchmark for measuring progress in statistical language modeling\.Technical reportGoogle\.External Links:[Link](http://arxiv.org/abs/1312.3005)Cited by:[§3\.1](https://arxiv.org/html/2608.17325#S3.SS1.p2.1),[§3\.4](https://arxiv.org/html/2608.17325#S3.SS4.p1.1)\.
- Chizhovet al\.\(2024\)P\. Chizhov, C\. Arnett, E\. Korotkova, and I\. P\. YamshchikovBPE gets picky: efficient vocabulary refinement during tokenizer training\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 16587–16604\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.925/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.925)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.17.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Cognettaet al\.\(2024\)M\. Cognetta, T\. Hiraoka, R\. Sennrich, Y\. Pinter, and N\. OkazakiAn analysis of BPE vocabulary trimming in neural machine translation\.InProceedings of the Fifth Workshop on Insights from Negative Results in NLP,S\. Tafreshi, A\. Akula, J\. Sedoc, A\. Drozd, A\. Rogers, and A\. Rumshisky \(Eds\.\),Mexico City, Mexico,pp\. 48–50\.External Links:[Link](https://aclanthology.org/2024.insights-1.7/),[Document](https://dx.doi.org/10.18653/v1/2024.insights-1.7)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p1.1),[§3\.3](https://arxiv.org/html/2608.17325#S3.SS3.p2.1),[§4\.4](https://arxiv.org/html/2608.17325#S4.SS4.p1.1)\.
- Doddapaneniet al\.\(2023\)S\. Doddapaneni, R\. Aralikatte, G\. Ramesh, S\. Goyal, M\. M\. Khapra, A\. Kunchukuttan, and P\. KumarTowards leaving no Indic language behind: building monolingual corpora, benchmark and models for Indic languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 12402–12426\.External Links:[Link](https://aclanthology.org/2023.acl-long.693/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.693)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Gage \(1994\)P\. GageA new algorithm for data compression\.C Users J\.12\(2\),pp\. 23–38\.External Links:ISSN 0898\-9788Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.3.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p1.1)\.
- Hochreiter and Schmidhuber \(1997\)S\. Hochreiter and J\. SchmidhuberLong short\-term memory\.Neural Comput\.9\(8\),pp\. 1735–1780\.External Links:ISSN 0899\-7667,[Link](https://doi.org/10.1162/neco.1997.9.8.1735),[Document](https://dx.doi.org/10.1162/neco.1997.9.8.1735)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p3.1)\.
- Hofmannet al\.\(2022\)V\. Hofmann, H\. Schuetze, and J\. PierrehumbertAn embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 385–393\.External Links:[Link](https://aclanthology.org/2022.acl-short.43/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-short.43)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Huet al\.\(2025\)Y\. Hu, F\. Liang, D\. Zhao, J\. Geuter, V\. Reddy, C\. W\. Schmidt, and C\. TannerEntropy\-driven pre\-tokenization for byte\-pair encoding\.External Links:2506\.15889,[Link](https://arxiv.org/abs/2506.15889)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Hwanget al\.\(2025\)S\. Hwang, B\. Wang, and A\. GuDynamic chunking for end\-to\-end hierarchical sequence modeling\.External Links:2507\.07955,[Link](https://arxiv.org/abs/2507.07955)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p3.1),[2nd item](https://arxiv.org/html/2608.17325#S3.I1.i2.p1.1)\.
- Kudo and Richardson \(2018\)T\. Kudo and J\. RichardsonSentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,E\. Blanco and W\. Lu \(Eds\.\),Brussels, Belgium,pp\. 66–71\.External Links:[Link](https://aclanthology.org/D18-2012/),[Document](https://dx.doi.org/10.18653/v1/D18-2012)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Kudo \(2018\)T\. KudoSubword regularization: improving neural network translation models with multiple subword candidates\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 66–75\.External Links:[Link](https://aclanthology.org/P18-1007/),[Document](https://dx.doi.org/10.18653/v1/P18-1007)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.5.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Libovický and Helcl \(2024\)J\. Libovický and J\. HelclLexically grounded subword segmentation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7403–7420\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.421/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.421)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Limisiewiczet al\.\(2024\)T\. Limisiewicz, T\. Blevins, H\. Gonen, O\. Ahia, and L\. ZettlemoyerMYTE: morphology\-driven byte encoding for better and fairer multilingual language modeling\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15059–15076\.External Links:[Link](https://aclanthology.org/2024.acl-long.804/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.804)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.21.1.1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Liuet al\.\(2025\)A\. Liu, J\. Hayase, V\. Hofmann, S\. Oh, N\. A\. Smith, and Y\. ChoiSuperBPE: space travel for language models\.External Links:2503\.13423,[Link](https://arxiv.org/abs/2503.13423)Cited by:[§B\.3](https://arxiv.org/html/2608.17325#A2.SS3.SSS0.Px3.p1.1),[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.25.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Meyer and Buys \(2022\)F\. Meyer and J\. BuysSubword segmental language modelling for nguni languages\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 6636–6649\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.494/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.494)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p3.1),[§2](https://arxiv.org/html/2608.17325#S2.p1.1)\.
- Meyer and Buys \(2025\)F\. Meyer and J\. BuysThe learning dynamics of subword segmentation for morphologically diverse languages\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),Mumbai, India,pp\. 647–661\.External Links:[Link](https://aclanthology.org/2025.ijcnlp-long.36/),ISBN 979\-8\-89176\-298\-5Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p3.1),[§2](https://arxiv.org/html/2608.17325#S2.p1.1),[1st item](https://arxiv.org/html/2608.17325#S3.I1.i1.p1.1),[§3\.4](https://arxiv.org/html/2608.17325#S3.SS4.p1.1),[§4\.1](https://arxiv.org/html/2608.17325#S4.SS1.p1.1)\.
- Mukku and Mamidi \(2017\)S\. S\. Mukku and R\. MamidiACTSA: annotated corpus for Telugu sentiment analysis\.InProceedings of the First Workshop on Building Linguistically Generalizable NLP Systems,E\. Bender, H\. Daumé III, A\. Ettinger, and S\. Rao \(Eds\.\),Copenhagen, Denmark,pp\. 54–58\.External Links:[Link](https://aclanthology.org/W17-5408/),[Document](https://dx.doi.org/10.18653/v1/W17-5408)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Nawrotet al\.\(2023\)P\. Nawrot, J\. Chorowski, A\. Lancucki, and E\. M\. PontiEfficient transformers with dynamic token pooling\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 6403–6417\.External Links:[Link](https://aclanthology.org/2023.acl-long.353/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.353)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p3.1)\.
- Nivreet al\.\(2020\)J\. Nivre, M\. de Marneffe, F\. Ginter, J\. Hajič, C\. D\. Manning, S\. Pyysalo, S\. Schuster, F\. Tyers, and D\. ZemanUniversal Dependencies v2: an evergrowing multilingual treebank collection\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 4034–4043\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.497/),ISBN 979\-10\-95546\-34\-4Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Nivre and Fang \(2017\)J\. Nivre and C\. FangUniversal Dependency evaluation\.InProceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies \(UDW 2017\),M\. de Marneffe, J\. Nivre, and S\. Schuster \(Eds\.\),Gothenburg, Sweden,pp\. 86–95\.External Links:[Link](https://aclanthology.org/W17-0411/)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Panet al\.\(2017\)X\. Pan, B\. Zhang, J\. May, J\. Nothman, K\. Knight, and H\. JiCross\-lingual name tagging and linking for 282 languages\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 1946–1958\.External Links:[Link](https://aclanthology.org/P17-1178/),[Document](https://dx.doi.org/10.18653/v1/P17-1178)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Poelmanet al\.\(2025\)W\. Poelman, T\. Bauwens, and M\. de LhoneuxConfounding factors in relating model performance to morphology\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 7262–7287\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.369/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.369),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1),[§3\.4](https://arxiv.org/html/2608.17325#S3.SS4.p1.1)\.
- Provilkovet al\.\(2020\)I\. Provilkov, D\. Emelianenko, and E\. VoitaBPE\-dropout: simple and effective subword regularization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 1882–1892\.External Links:[Link](https://aclanthology.org/2020.acl-main.170/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.170)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.13.1.1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Schmidtet al\.\(2024\)C\. W\. Schmidt, V\. Reddy, H\. Zhang, A\. Alameddine, O\. Uzan, Y\. Pinter, and C\. TannerTokenization is more than compression\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 678–702\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.40/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.40)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.15.1.1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Schmidtet al\.\(2025\)C\. W\. Schmidt, V\. Reddy, C\. Tanner, and Y\. PinterBoundless byte pair encoding: breaking the pre\-tokenization barrier\.External Links:2504\.00178,[Link](https://arxiv.org/abs/2504.00178)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.23.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Schuster and Nakajima \(2012\)M\. Schuster and K\. NakajimaJapanese and korean voice search\.InInternational Conference on Acoustics, Speech and Signal Processing,pp\. 5149–5152\.Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.7.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Sennrichet al\.\(2016\)R\. Sennrich, B\. Haddow, and A\. BirchNeural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1715–1725\.External Links:[Link](https://aclanthology.org/P16-1162/),[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.3.1.1.1),[§1](https://arxiv.org/html/2608.17325#S1.p1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Smitet al\.\(2014\)P\. Smit, S\. Virpioja, S\. Grönroos, and M\. KurimoMorfessor 2\.0: toolkit for statistical morphological segmentation\.InProceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics,S\. Wintner, M\. Tadić, and B\. Babych \(Eds\.\),Gothenburg, Sweden,pp\. 21–24\.External Links:[Link](https://aclanthology.org/E14-2006/),[Document](https://dx.doi.org/10.3115/v1/E14-2006)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.11.1.1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1)\.
- Teamet al\.\(2022\)N\. Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood, B\. Akula, L\. Barrault, G\. M\. Gonzalez, P\. Hansanti, J\. Hoffman, S\. Jarrett, K\. R\. Sadagopan, D\. Rowe, S\. Spruit, C\. Tran, P\. Andrews, N\. F\. Ayan, S\. Bhosale, S\. Edunov, A\. Fan, C\. Gao, V\. Goswami, F\. Guzmán, P\. Koehn, A\. Mourachko, C\. Ropers, S\. Saleem, H\. Schwenk, and J\. WangNo language left behind: scaling human\-centered machine translation\.External Links:2207\.04672,[Link](https://arxiv.org/abs/2207.04672)Cited by:[Table 12](https://arxiv.org/html/2608.17325#A2.T12),[§3\.1](https://arxiv.org/html/2608.17325#S3.SS1.p2.1)\.
- Tjong Kim Sang and De Meulder \(2003\)E\. F\. Tjong Kim Sang and F\. De MeulderIntroduction to the CoNLL\-2003 shared task: language\-independent named entity recognition\.InProceedings of the Seventh Conference on Natural Language Learning at HLT\-NAACL 2003,pp\. 142–147\.External Links:[Link](https://aclanthology.org/W03-0419/)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Uzanet al\.\(2024\)O\. Uzan, C\. W\. Schmidt, C\. Tanner, and Y\. PinterGreed is all you need: an evaluation of tokenizer inference methods\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 813–822\.External Links:[Link](https://aclanthology.org/2024.acl-short.73/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-short.73)Cited by:[§2](https://arxiv.org/html/2608.17325#S2.p2.1)\.
- Vaswaniet al\.\(2023\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. PolosukhinAttention is all you need\.External Links:1706\.03762,[Link](https://arxiv.org/abs/1706.03762)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p1.1),[§1](https://arxiv.org/html/2608.17325#S1.p3.1)\.
- Vemulaet al\.\(2025\)S\. R\. Vemula, S\. Dandapat, D\. Sharma, and P\. KrishnamurthyRethinking tokenization for rich morphology: the dominance of unigram over BPE and morphological alignment\.InThe 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,S\. T\.y\.s\.s, S\. Shimizu, and Y\. Gong \(Eds\.\),Mumbai, India,pp\. 232–252\.External Links:[Link](https://aclanthology.org/2025.ijcnlp-srw.20/),ISBN 979\-8\-89176\-304\-3Cited by:[§2](https://arxiv.org/html/2608.17325#S2.p2.1)\.
- Wanget al\.\(2018\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,T\. Linzen, G\. Chrupała, and A\. Alishahi \(Eds\.\),Brussels, Belgium,pp\. 353–355\.External Links:[Link](https://aclanthology.org/W18-5446/),[Document](https://dx.doi.org/10.18653/v1/W18-5446)Cited by:[§B\.4](https://arxiv.org/html/2608.17325#A2.SS4.p2.1)\.
- Yehezkel and Pinter \(2023\)S\. Yehezkel and Y\. PinterIncorporating context into subword vocabularies\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 623–635\.External Links:[Link](https://aclanthology.org/2023.eacl-main.45/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.45)Cited by:[Table 13](https://arxiv.org/html/2608.17325#A2.T13.2.9.1.1.1),[Table 14](https://arxiv.org/html/2608.17325#A2.T14.2.7.2.1.1),[§3\.2](https://arxiv.org/html/2608.17325#S3.SS2.p2.1),[§4\.1](https://arxiv.org/html/2608.17325#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.17325#S4.SS2.SSSx2.p2.1)\.
- Zhuet al\.\(2025\)Q\. Zhu, X\. Hu, P\. Ji, W\. Wu, and K\. TuUnsupervised morphological tree tokenizer\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 22299–22312\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1146/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1146),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Zhuet al\.\(2019\)Y\. Zhu, B\. Heinzerling, I\. Vulić, M\. Strube, R\. Reichart, and A\. KorhonenOn the importance of subword information for morphological tasks in truly low\-resource languages\.InProceedings of the 23rd Conference on Computational Natural Language Learning \(CoNLL\),M\. Bansal and A\. Villavicencio \(Eds\.\),Hong Kong, China,pp\. 216–226\.External Links:[Link](https://aclanthology.org/K19-1021/),[Document](https://dx.doi.org/10.18653/v1/K19-1021)Cited by:[§1](https://arxiv.org/html/2608.17325#S1.p2.1)\.
- Zouharet al\.\(2023\)V\. Zouhar, C\. Meister, J\. Gastaldi, L\. Du, M\. Sachan, and R\. CotterellTokenization and the noiseless channel\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5184–5207\.External Links:[Link](https://aclanthology.org/2023.acl-long.284/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.284)Cited by:[Table 14](https://arxiv.org/html/2608.17325#A2.T14.2.14.1.1.1)\.

## Appendix AAblation Study: Model Scale

In our downstream evaluation, we fix the size of BERT models at approximately 12M parameters\. This was scaled accordingly with respect to the data size of 10M sentences\. Here, we perform an ablation study by changing the size of our models while keeping the data size same\. We perform additional evaluation of BERT models at approximately 2M and 30M parameters\. To limit the computational resources, we limit our analysis to two tokenization approaches:BPEandSSLM\-BPE\. This allows us to focus on the affect of pretokenization usingtokenizer\-freeapproachSSLM\.

Model ScaleSentiment Analysis \(Acc\)POS Tagging \(F1\)Named Entity Recognition \(F1\)Dependency Parsing \(LAS\)Languageeng\_latntel\_teluhin\_devaeng\_latntel\_teluhin\_devaeng\_latntel\_teluhin\_devaeng\_latntel\_teluhin\_devaStandard Algorithm: BPE2M77\.7559\.1565\.9789\.1662\.2292\.5770\.7986\.1575\.9054\.7856\.0474\.1212M85\.5562\.8576\.6795\.0988\.4197\.1589\.9894\.7290\.6975\.6466\.3188\.2030M88\.4265\.4378\.0195\.9792\.8297\.6491\.6595\.6691\.9180\.2972\.9690\.44Using SSLM as pretokenization: SSLM\-BPE2M76\.9559\.8958\.8988\.0561\.7691\.2856\.1883\.3776\.6150\.9953\.6272\.3812M85\.6664\.7077\.0695\.0988\.4297\.1888\.9394\.3891\.0374\.9066\.7787\.4730M86\.0164\.7078\.0196\.1392\.7897\.4591\.5995\.7092\.7280\.2070\.0989\.88Table 5:Downstream performance of BERT models at different scales trained with tokenizers BPE and SSLM\-BPE\. Best results for each task\-language pair are inbold\. Scores are averaged over three finetuning runs \(seeds 42, 43, 44\)\. We report accuracy for Sentiment Analysis, F1 for POS tagging and NER, and LAS for Dependency Parsing\. Variants improved bySSLMpretokenization areunderlined\.Figure 7:Validation perplexity of BERT models at different scales\. Aligning tokens withtokenizer\-freeapproaches through pretokenization consistently results in lower perplexity\.\(a\)Sentiment analysis \(Accuracy\)\(b\)POS tagging \(F1\-score\)\(c\)NER \(F1\-score\)\(d\)Dependency parsing \(LAS\)
Figure 8:Performance of BERT models across model scales on sentiment analysis, POS tagging, NER, and dependency parsing\.Findings: As observed in Figure[7](https://arxiv.org/html/2608.17325#A1.F7), models trained usingSSLMas pretokenization consistently show lower perplexity across all model sizes\. However, as model size increases, the improvement saturates\. This indicates thatSSLM\-BPEgenerally provides more efficient language modeling\. We also perform evaluation on downstream tasks at different scales\. The results are listed in Table[5](https://arxiv.org/html/2608.17325#A1.T5)and plotted in Figure[8](https://arxiv.org/html/2608.17325#A1.F8)\. Across all three languages,SSLM\-BPEoften matchesBPEon downstream tasks such as POS tagging and NER\. We observe that whenever model benefits with larger size, the gap betweenSSLM\-BPEandBPEreduces, especially for POS tagging and NER tasks\.Note: Ourtokenizer\-freeLMs were trained 250,000 sentence corpora, rather than on the entire 10M sentence corpora which we use for downstream evaluation\. This was due to high computation requirement of such models \(approximately10×10\\timesslower than transformers\)\. We leave further scaling analysis accounting for these factors as our future work\.

## Appendix BHyperparameters & Experimental Setup

### B\.1H\-Nets

Table[6](https://arxiv.org/html/2608.17325#A2.T6)details the primary hyperparameter configurations used for pretraining the Hierarchical Network \(H\-Net\) models across all evaluated languages\. The model employs a structural layout ofm1m\_\{1\}at the bottom level andT2T\_\{2\}at the top level, utilizing an embedding and hidden dimension of 256 for both levels\. The bottom level \(m1m\_\{1\}\) operates purely without an FFN block, while the top level \(T2T\_\{2\}\) uses an intermediate feed\-forward dimension of 640\. This setup operates directly on byte sequences \(vocabulary size of 256\) and results in a lightweight model with approximately 3M parameters\. We trained the model using an AdamW optimizer \(withβ1=0\.9,β2=0\.95\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.95\) and a batch size of 128 for a maximum of 70 epochs\. Optimization was guided by a cosine learning rate scheduler with a peak base learning rate of3×10−43\\times 10^\{\-4\}after an initial 10% linear warmup phase\. To prevent overfitting, early stopping was employed with a patience of 5 epochs alongside a weight decay of 0\.01\. TheH\-Netspecific structural compression constraints were maintained constantly throughout training, with the ratio loss scale set to 1\.0 and zero warmup compression epochs\.

Table 6:Hyperparameter configurations and approximate parameter count for the Hierarchical Network \(H\-Net\) pre\-training phase\.
### B\.2SSLMs

Table[7](https://arxiv.org/html/2608.17325#A2.T7)details the comprehensive set of hyperparameters utilized during the pre\-training of our Subword Segmental Language Models \(SSLMs\) across all evaluated languages\. We adopted a lightweight Transformer decoder architecture tailored to subword segmental modeling, bringing the total parameter count to 2,105,836\. Key architectural adjustments include scaling down the number of layers, embedding dimensions, and feed\-forward dimensions to ensure fast and efficient training with a limited parameter budget\. The maximum segment length was constrained to 5 spatial units to balance compositional flexibility and computational overhead\. Optimization was performed using the Adam optimizer coupled with an inverse square root learning rate scheduler and a linear warmup phase\.

Table 7:Hyperparameter configurations and parameter count for the lightweight Subword Segmental Language Model \(SSLM\) pre\-training phase\.
### B\.3Fixed\-tokenizer Variants

To ensure a fair and comprehensive comparison across different tokenization strategies, we standardized the training hyperparameter configurations where applicable, while setting strategy\-specific parameters to their best\-known or recommended values\.

##### Global Settings

All tokenizers were trained across three final target vocabulary sizes:T∈\{5000,10000,20000\}T\\in\\\{5000,10000,20000\\\}\. For tokenizers relying on the SentencePiece library \(e\.g\., standardBPE,Unigram, andPathPiece\), the character coverage parameter was explicitly set to1\.01\.0\(100%\) to prevent any heuristic character dropping and uniformly ensure complete representation of the datasets’ alphabets\. We consider fixed\-tokenizer variants with final vocabulary size of10,00010,000for a fair comparison withSSLMs with initial lexicon of size10,00010,000\.

##### Standard Subword Architectures

ForWordPiece, we utilized the HuggingFace tokenizers library, configuring it with NFKC Unicode normalization and lowercasing\. Pre\-tokenization was handled via a standard whitespace split, and the continuation prefix was assigned as “\#\#”\. The standard special tokens strictly enforced were\[UNK\], \[CLS\], \[SEP\], \[PAD\], and\[MASK\]\. StandardBPEandUnigramLMmodels relied on SentencePiece defaults beyond the mandatory complete character coverage\.

##### SuperBPE

Following the two\-stage subword acquisition strategy ofSuperBPE, the base vocabulary size \(the transition limitttbetween stage 1 and stage 2\) was configured att=4000t=4000when final vocabulary size desired wasT=10000T=10000, andt=10000t=10000whenT=25000T=25000\(i\.e\.,T=t×2\.5T=t\\times 2\.5, as found in[23](https://arxiv.org/html/2608.17325#bib.bib19)\)\.

##### BoundlessBPE

ForBoundlessBPE, theτ\\tauparameter, managing the trade\-off threshold, was set to0\.90\.9\. The vocabulary pruning recalculation interval was configured to occur every10001000iterations \(recalc = 1000\)\. We utilized theultimate2regular expression split pattern constraints, taking advantage of the active vocabulary blowup tolerance \(blowup = 1\)\.

##### PickyBPE

The intersection\-over\-sequence threshold regulating token acceptance inPickyBPEwas strictly confined to0\.90\.9\.

##### Morphologically\-Informed Tokenizers \(MorphBPE, MorphULM, MorphWP\)

To create tokenizers aligned with linguistic boundaries, an unsupervisedMorfessorBaseline model was first trained on the raw corpus to dictate morphological decisions\. Texts were pre\-tokenized along the predicted morphological seams, creating a pseudo\-corpus separated by whitespace\. The subsequentBPE,Unigram, andWordPiececounterparts \(MorphBPE,MorphULM, andMorphWP\) were exclusively trained on this constrained configuration, forcing the statistically induced vocabularies to respect Morfessor’s proposed boundaries\.

### B\.4Downstream Evaluation

As discussed, we perform downstream evaluation by pretraining BERT[12](https://arxiv.org/html/2608.17325#bib.bib27)models and later finetuning them on each task\. Table[8](https://arxiv.org/html/2608.17325#A2.T8)lists the hyperparameters we used while pretraining BERT models\. Similarly, Table[9](https://arxiv.org/html/2608.17325#A2.T9)lists the hyperparameters while finetuning for each task\.

We describe our downstream tasks as follows: POS tagging: Involves assigning a grammatical category such as noun, verb, adjective, to each word in a sentence\. It involves reasoning through both the definition of the word and its context\. We consider the class of word as the class corresponding to its first subword\. We use Universal Dependency \(UD\) treebanks[28](https://arxiv.org/html/2608.17325#bib.bib41)via HuggingFace universal dependency dataset:en\_ewtfor English,te\_mtgfor Telugu, andhi\_hdtbfor Hindi\. Labels are drawn from the 17 universal POS tags \(upos\)\. Performance is measured with weighted\-average F1\-score\. Sentiment Analysis: This tasks involves determining the sentiment of a sentence, e\.g\., positive, negative, or neutral\. We use ACTSA \(Annotated Corpus for Telugu Sentiment Analysis\) dataset for Telugu[26](https://arxiv.org/html/2608.17325#bib.bib42), IITP Product Review dataset[1](https://arxiv.org/html/2608.17325#bib.bib43), and SST\-2 dataset from GLUE benchmark[43](https://arxiv.org/html/2608.17325#bib.bib39)\. Named Entity Recognition\(NER\): This tasks involves identifying and classifying named entities into predefined categories such as persons, organizations, locations, etc\. We use CoNLL NER dataset[39](https://arxiv.org/html/2608.17325#bib.bib44)for English, and WikiAnn dataset[30](https://arxiv.org/html/2608.17325#bib.bib45);[13](https://arxiv.org/html/2608.17325#bib.bib46)\. WikiAnn consists of coarse grained labels as follows: Person \(POS\), Organization \(ORG\), and Location \(LOC\)\. Whereas, CoNLL NER contains an additional Miscellaneous \(MISC\) tag\. Dependency Parsing\(DP\): This task involves analyzing the grammatical structure of a sentence by identifying the relationship between head words and their dependents\. We again use Universal Dependency \(UD\) treebank dataset[28](https://arxiv.org/html/2608.17325#bib.bib41)\. We report Labeled Attachment Score \(LAS\)[29](https://arxiv.org/html/2608.17325#bib.bib47)\.

Table 8:Architecture and pre\-training configurations for the three BERT model scales \(2M / 12M / 30M\)\. Where values differ across scales, they are listed in order from smallest to largest model\.Table 9:Hyperparameters used for fine\-tuning BERT across all downstream tasks\.EnglishHebrewHindiTamilHungarianIndonesianBangladeshisריכוזייםडिस्ट्रीब्यूटरोंதொடுக்கzeneszerzőzseniketkehabisanBangladeshi sריכוזי יםडिस्ट्रीब्यूटर ोंதொடு க்கzeneszerzőzseni ketke habis anSegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1Ba ngl ades his0ריכו ז יים0डिस् ट्री ब्यू टरों0தொ டுக்க0zene szerz őz sen iket0keha bisa n0Ba ngla deshi s0\.5ריכ וזי ים0\.67डिस् ट्री ब्यू ट र ों0\.33தொட ு க்க0\.67zene szerz ő zs en i ket0\.29ke habi s an0\.8Ba ngla deshi s0\.5רי כוז י ים0\.5डिस् ट्री ब्यू ट र ों0\.33தொ டுக்க0zene szerz ő zs en ik et0keha bis an0\.5Ba ngla deshi s0\.5רי כוז י ים0\.5डिस् ट्री ब्यू टर ों0\.4தொ டுக்க0zene szerz ő zs eni ket0\.33keha bis an0\.5Ba ngla deshi s0\.5רי כוז י ים0\.5डिस् ट्री ब्यू टर ों0\.4தொ டுக்க0zene szerz ő zs eni ket0\.33keha bis an0\.5meadowlarksגבשושייםरजिस्ट्रारोंவசித்துmegtörténhetésénekdisucikanmeadowlark sגבשושי יםरजिस्ट्रार ोंவசி த்துmegtörténhetés énekdi suci kanSegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1mea do wl arks0ג בשו שיים0रजि स्ट्र ारों0வசி த்து1megt örtén he té sének0dis uci kan0\.5mea dow lark s0\.5ג בש ושי ים0\.5रजिस् ट्रा र ों0\.5வசி த்து1meg törté nhet és ének0\.4di s uci kan0\.80mea dow lark s0\.5ג בש ושי ים0\.67रजिस् ट्रा रों0வசி த்து1meg törté nhet ésé nek0dis uci kan0\.5mea dow lark s0\.5ג בש וש י ים0\.4रजिस् ट्रा र ों0\.5வசி த்த ு0\.67meg törté n het és ének0\.33dis uci kan0\.5mea dow lark s0\.5ג בש וש י ים0\.4रजिस् ट्रा र ों0\.5வசி த்த ு0\.67meg törté n het és ének0\.33dis uci kan0\.5eschewingמאושפזותपुरातत्वविदोंஎல்லைய்szavazatábólkegigihaneschew ingמ אושפז ותपुरातत्वविद ोंஎல்லை ய்szavazat ábólke gigih anSegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1es che wing0מא ושפ זות0पुरा तत् ववि दों0எல்லை ய்1szav azat ából0\.67keg igi han0esc hew ing0\.67מאו שפ ז ות0\.4पुरा तत् वविद ों0\.5எல்லை ய ்0\.67szava zat ából0\.67ke gi g ih an0\.67esc hew ing0\.67מ א ושפ ז ות0\.67पु रात त्व विद ों0\.4எ ல்லைய ்0sz avaz atá ból0ke g igi h an0\.67esc hew ing0\.67מא ושפ ז ות0\.4पु रात त्व विद ों0\.4எ ல்லைய ்0sz avaz at ából0\.5ke g igi h an0\.67esc hew ing0\.67מא ושפ ז ות0\.4पु रात त्व विद ों0\.4எல்லை ய ்0\.67sz avaz at ából0\.5keg igi h an0\.4Veteransאידיאליםप्रतिष्ठानोंபடையைச்tartozásánakperumahanVeteran sאידיאל יםप्रतिष्ठान ोंபடை யைச்tartozás ánakpe rumah anSegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1Ve ter ans0אידי אלים0प्रति ष्ठा नों0படை யைச்1tar tozás ának0\.67per uma han0V et eran s0\.5אי דיאל ים0\.67प्रति ष्ठ ान ों0\.5படை யைச்1tarto zás ának0\.67pe rumah an1Vet eran s0\.67אי דיאל ים0\.67प्रति ष्ठ ान ों0\.5படை யைச்1tarto zásá nak0per umah an0\.5Vet eran s0\.67אי דיאל ים0\.67प्र तिष्ठ ान ों0\.5ப டை யைச்0\.67tarto zás ának0\.67per umah an0\.5Vet eran s0\.67אי דיאל י ם0\.5प्र तिष्ठ ान ों0\.5ப டை யைச்0\.67tarto zás ának0\.67per umah an0\.5roastedעיתונאיתस्लाइड्सமுடிவெடுக்கப்atrocitásokbergabungroast edעיתונאי תस्लाइड ्सமுடிவெடு க்கப்atrocitás okber gabungSegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1SegmentationF1roa sted0עיתונ אית0स्ला इड ्स0\.67முடிவ ெடுக் கப்0atr oci tások0berg abung0roast ed1\.0עיתונ אי ת0\.67स्ल ाइड ्स0\.67முடிவ ெடு க்க ப்0\.5a tro c itás ok0\.4ber gab ung0\.67roast ed1\.0עיתונ אי ת0\.67स ्लाइ ड ्स0\.5முடிவ ெடு க்க ப்0\.5a tro c itás ok0\.4ber gab ung0\.67roast ed1\.0ע יתונא י ת0\.5स ्लाइ ड ्स0\.5முடிவ ெ டுக்க ப்0a troc itás ok0\.5ber gab ung0\.67roast ed1\.0ע יתונא י ת0\.5स्ल ाइड ्स0\.67முடிவ ெ டுக்க ப்0atroc itás ok0\.67b ergab ung0\.5Table 10:Evolution of segmentation of few more candidate wordforms produced bytokenizer\-freeapproachSSLMwith morphological alignment F1\-scores\. English, Hebrew, and Hindi are fusional or analytical languages, while Tamil, Hungarian, and Indonesian are agglutinative languages\.LanguageISO 639\-3ISO 15924TypologyFamilyGroupAgglutinativeFinnishfinlatnAgglutinative \(with some Fusion\)UralicFinno\-UgricHungarianhunlatnAgglutinative \(Suffixing\)UralicFinno\-UgricMalayalammalmlymAgglutinative \(Highly Synthetic\)DravidianDravidianTamiltamtamlAgglutinative \(Highly Synthetic\)DravidianDravidianTelugutelteluAgglutinative \(Highly Synthetic\)DravidianDravidianKyrgyzkircyrlAgglutinative \(Suffixing\)TurkicTurkicTurkishturlatnAgglutinative \(Highly Productive Suffixing\)TurkicTurkicMongolianmoncyrlAgglutinative \(Suffixing\)MongolicMongolicIndonesianindlatnAgglutinative \(Reduplication\-Heavy\)AustronesianMalayo\-PolynesianFusionalSanskritsandevaFusional \(Polysynthetic Tendencies\)Indo\-EuropeanIndo\-AryanHindihindevaFusional / Analytic \(Split\-Ergative\)Indo\-EuropeanIndo\-AryanSindhisndarabFusional \(Moderate\)Indo\-EuropeanIndo\-AryanCroatianhrvlatnFusional \(Highly Synthetic\)Indo\-EuropeanSlavicRussianruscyrlFusional \(Highly Synthetic\)Indo\-EuropeanSlavicPersianfasarabFusional \(Weakly Synthetic\)Indo\-EuropeanIndo\-IranianAnalytic & IntroflexiveEnglishenglatnAnalytic / Weakly FusionalIndo\-EuropeanGermanicSwedishswelatnAnalytic / Weakly FusionalIndo\-EuropeanGermanicHebrewhebhebrIntroflexive \(Non\-concatenative\)AfroasiaticSemiticTable 11:Languages categorized by morphological typology\. Grouping highlights the structural similarities in word formation across different language families\.LanguageISO 15924BP\#sentences\#wordsASL\#MS\#MNData SourceAgglutinativeFinnishlatn1\.0589051432,4224,532,82410\.48101723,745,139NewsCrawlHungarianlatn1\.0199851280,2764,883,41617\.4263501,044,996NewsCrawlMalayalammlym2\.8852389438,5014,053,8599\.24131\-NewsCrawlTamiltaml2\.7292892397,2064,216,31310\.611179\-NewsCrawlTelugutelu2\.6198705475,4634,582,0989\.648092\-NewsCrawlKyrgyzcyrl1\.963557410,0755,154,29012\.574221\-NewsCrawlTurkishlatn1\.0444815351,2924,733,45713\.4730076\-NewsCrawlMongoliancyrl1\.8046135623,1475,719,2499\.18\-16,221NLLBIndonesianlatn1\.1788023371,7106,252,11616\.822785\-NewsCrawlFusionalSanskritdeva2\.5428913857,2985,159,6966\.0216184\-NLLBHindideva2\.3701629427,0156,936,48416\.241301\-NewsCrawlSindhiarab1\.5880165614,0557,258,01411\.823874\-NLLBCroatianlatn0\.9897218309,0905,652,25218\.2977491,765,011NewsCrawlRussiancyrl1\.8228284341,5005,296,13215\.51215691,414,063NewsCrawlPersianarab1\.6790492444,4787,197,90316\.1911859\-NewsCrawlAnalytic & IntroflexiveEnglishlatn1250,0006,249,150253688874,725NewsCrawlSwedishlatn1\.0210256492,3456,002,54912\.196223140,843NLLBHebrewhebr1\.3555346508,2505,248,16410\.334641\-NLLBTable 12:Comparative Statistics after byte\-premium \(BP\)[4](https://arxiv.org/html/2608.17325#bib.bib23)adjustments and Dataset Sources for Languages Categorized by Morphological Typology\. Abbreviations are BP: Byte\-premiums, ASL: Average Sentence Length \(in number of words\), \#MS: Number of items in MorphScore[3](https://arxiv.org/html/2608.17325#bib.bib7);[5](https://arxiv.org/html/2608.17325#bib.bib8), \#MN: Number of items in MorphyNet[7](https://arxiv.org/html/2608.17325#bib.bib9), NLLB: No Language Left Behind[38](https://arxiv.org/html/2608.17325#bib.bib25)\.Table 13:List of Fixed\-tokenizer approaches included in this study, organized by the stage at which they modify the tokenization pipeline\.MetricDescriptionRequirementsWhat it AddressesCoverageSubword FertilityAverage number of tokens produced per word\.Tokenizer and representative corpus\.Sequence length and basic storage efficiency\.Compression RateRatio of raw text size to tokenized sequence length\.Tokenizer and representative corpus\.Storage and context window efficiency\.Token Frequency DistributionDistributional profile of token occurrences \(e\.g\., long\-tail vs\. concentrated usage\)\.Token counts from a representative corpus\.Statistical balance of vocabulary usage and fragmentation patterns\.Token Length DistributionDistributional profile of tokens grouped by lengths from a representative corpusVocabulary of tokenizerGranularity balance between short and long tokens, which affects efficiency and segmentation behavior\.Contextual ExponenceNumber of distinct neighbours each token encounters[44](https://arxiv.org/html/2608.17325#bib.bib30)\. We consider only immediate neighbourhoods in our evaluation\.Representative corpus and its corresponding tokensDegree to which the tokenizer optimizes tokens’ contextual soundnessEffective Vocabulary SizeTotal number of unique tokens available in the tokenizer vocabulary\. Fortokenizer\-freeapproaches, we consider number of unique tokens when a corpus is segmented using it\.Trained tokenizer specification or model\.Memory footprint, embedding\-table size, and granularity trade\-offs\.GeneralizabilityType\-Token Ratio \(TTR\)Ratio of unique token types to total token occurrences in a corpus sample\.Tokenized corpus with fixed sample size \(or moving\-window normalization\)\.Lexical diversity and degree of repetition; helps compare whether a tokenizer yields broader vs\. concentrated token usage\.Linguistic AlignmentMorphological AlignmentDivergence between token segments and known morphemes\.Morphological analyzer and gold\-standard data\.Linguistic fidelity and grammatical preservation\.RobustnessRényi Efficiency[47](https://arxiv.org/html/2608.17325#bib.bib22)Information\-theoretic metric penalizing skewed distributions\.Token frequency data from a corpus\.Statistical balance and information density\.Table 14:Summary of Tokenizer Evaluation Metrics grouped by the proposed multidimensional framework in[2](https://arxiv.org/html/2608.17325#bib.bib35)Figure 9:Evolution of Morphological Alignment \(F1\-score\) in Tranformer\-based Subword Segmental Language Models \(SSLM\)\.\(a\)Fertility by script\(b\)Contextual exponence by script\(c\)Effective vocabulary size by typology\(d\)Compression by typology\(e\)Rényi efficiency by typology\(f\)Type\-token ratio by typology
Figure 10:Intrinsic properties oftokenizer\-freeapproaches \(H\-Nets andSSLMs\) compared to fixed\-tokenizer approaches across scripts and language typologies\.