SuTRA : Structurally-Unified Tokenization with Root Awareness

# SuTRA : Structurally-Unified Tokenization with Root Awareness
Source: [https://arxiv.org/html/2608.18087](https://arxiv.org/html/2608.18087)
\\interspeechcameraready

Rathore Gole Telwadkar Bhatia Ruparel Surekha Bhargava Motilal Oswal Financial Services Ltd\.India Indian Institute of Technology \(IIT\) BombayIndia

###### Abstract

Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes\. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables \(aksharas\) rather than letters\. Frequency\-based methods over\-fragment words, arbitrarily splitting roots and affixes—a phenomenon we termMorphological Shattering\. We proposeSuTRA111Project Page:[https://mo\-vaibhavr\-43300\.github\.io/SuTRA/](https://mo-vaibhavr-43300.github.io/SuTRA/)\(Structurally\-Unified Tokenization with Root Awareness\), a morphology\-aware algorithm that preservesaksharaindivisibility and penalizes merges crossing morphological boundaries\. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati\.SuTRAreduces shattering, achieving peak gains of \+14\.7% in morphological alignment \(Boundary F1\) and \+34% in semantic recoverability \(Hindi\) overBpe\. These structural gains yield an average improvement of \+8\.08chrF2in machine translation\.

###### keywords:

Large Language Models,Tokenization,Morphological Integrity,Indic Languages

## 1Introduction

ModernNLPpipelines rely on tokenizers as the foundational bridge to resolve the input\-representation gap, converting raw text into the discrete tokens that populate the high\-dimensional vector spaces of Language Models\[pennington2014glove,mikolov2013distributed,sarzynska2021detecting\]\. Currentde factostandards—such as Byte\-Pair Encoding \(Bpe\)\[Gage1994\-ds,Sennrich2016\-uh\],WordPiece\[Schuster2012\-ev\], andUnigram\[Kudo2018\-ue\]—function primarily as statistical data compression techniques\. While they efficiently reduce sequence lengths and out\-of\-vocabulary \(OOV\) rates, they remain fundamentally agnostic to the morphological structure of language\[Hofmann2022\-td,Hofmann2020\-lg\]\. Despite explorations into tokenization\-free architectures\[deiseroth2024t,clark2022canine,xue2022byt5,tay2021charformer\], state\-of\-the\-art models\[brown2020language,guo2025deepseek,yang2025qwen3,team2025gemma\]continue to rely onBpe\-style segmentation and inherit its structural limitations\.

Because these tokenizers ignore morphological rules, they often segment multimorphemic words into sub\-tokens that misalign with lexical roots and affixes\. We refer to this tokenizer\-level misalignment asMorphological Shattering\(Figure[1](https://arxiv.org/html/2608.18087#S1.F1)\)\. The problem is amplified in morphologically rich languages, especially those written in Indic scripts such as Devanagari\. These scripts are abugidas: base consonants and dependent vowels \(matras\) together form orthographic syllables \(aksharas\) that function as atomic written units\[akshara\_ref\]\. Off\-the\-shelf tokenizers frequently split inside these units—for example, separating amatrafrom its base consonant\. Many Indo\-Aryan languages also exhibitSandhi, where segments at word and morpheme boundaries change or fuse in the surface form\[sandhan2022translist,gaikwad2021state\], further obscuring the underlying morpheme boundaries\. Together with the well\-documentedIndic Taxof higher token fertility \(more tokens per word\)\[kumar2026sanskrit,pattnayak2025tokenization,tamang2024evaluating,chaudhari2023significance\], these properties lead to more severe Morphological Shattering\. We useSemantic Blindnessfor the corresponding representation\-level effect: root semantics become harder to recover with simple linear probes, and embeddings become overly sensitive to small orthographic changes that leave the underlying morphemes intact\[asgari2025morphbpe,arnett2025evaluating,morphtok2024,isac2025slip\]\.

![Refer to caption](https://arxiv.org/html/2608.18087v1/Images/teaser_6.png)Figure 1:Morphological Shattering vs\. Root Preservation\.For the Hindi wordasādhāraṇ, standardBpefuses the negation prefix into the root \(\[asā\]\+\[dhāraṇ\]\), whileSuTRAcleanly separates prefix and root \(\[a\]\+\[sādhāraṇ\]\)\. We term such frequency\-driven prefix–root fusionMorphological Shattering;SuTRA's root\-preserving segmentations yield more stable subword units and reduce semantic blindness\. Figure generated using PaperBanana\[paperbanana\]\.To address Morphological Shattering and its semantic consequences in complex languages, we make three contributions:

- •SuTRA\(Structurally\-Unified Tokenization with Root Awareness\): We introduce a structure\-guided extension ofBpethat augments frequency\-based merging with lightweight linguistic priors\. It combines script\-aware grouping of akshara\-like units with morphology\-aligned merge scoring and a root\-preserving boundary constraint to penalize splits across valid morphological units\.
- •Indic Morphological Gold\-Standard Dataset: We construct a large\-scale, LLM\-assisted morphological segmentation dataset tailored for three Indic languages \(Hindi, Marathi, and Gujarati\), providing a robust common benchmark for evaluating morphological alignment\.
- •Intrinsic and Downstream Evaluation: We design a comprehensive evaluation suite spanning morphological alignment, semantic recoverability, robustness under orthographic perturbations, and machine translation\. Across these four axes,SuTRAreduces Morphological Shattering and yields more stable, structurally aligned representations: it better respects morphemic boundaries without arbitrary over\-segmentation and improves both intrinsic semantic recoverability and downstream quality compared to standard baselines\.

## 2Related Work

### 2\.1Frequency\-Driven Tokenization

Standard vocabulary construction relies on compression algorithms likeBpe\[Gage1994\-ds,Sennrich2016\-uh\],WordPiece\[Schuster2012\-ev\],Unigram\[Kudo2018\-ue\], andSuperBpe\[liu2025superbpespacetravellanguage\]\. By treating text as a language\-agnostic character stream, these methods maximize statistical coverage but frequently override linguistic boundaries, systematically provokingMorphological Shattering\[Banerjee2018\-hd,Hofmann2020\-lg\]\. Prior mitigations, such as stochastic regularization \(Bpe\-Dropout\[provilkov\-etal\-2020\-bpe\]\) or post\-hoc realignment \(Flota\[hofmann\-etal\-2022\-embarrassingly\]\), remain auxiliary interventions that fail to correct the structural dissociation ingrained during initial vocabulary creation\. Furthermore, standard evaluation metrics like Compression Ratio and Perplexity\[haga2025babylmchallengeexploringeffect,ali2024tokenizer\]mask these representational inefficiencies because they measure sequence predictability rather than sub\-lexical integrity\.

### 2\.2Character\-level and Token\-free Architectures

To bypass the limitations of subword vocabularies, several works have proposed tokenization\-free or character\-level architectures such asCanine\[clark2022canine\],ByT5\[xue2022byt5\], andCharformer\[tay2021charformer\]\. These models operate directly on raw bytes or characters, theoretically avoiding the "shattering" problem entirely by allowing the model to learn its own internal representations of morphology\[deiseroth2024t\]\. However, these architectures suffer from a significantCompute Penalty, as character\-level processing drastically increases sequence lengths, making them computationally expensive for long\-context tasks\. Consequently, state\-of\-the\-art models continue to utilize subword\-basedBpe\[guo2025deepseek,yang2025qwen3,team2025gemma\], inheriting its structural limitations while benefiting from its efficiency\.

### 2\.3Morphological Challenges in IndicLLMs

Morphologically Rich Languages\(Mrls\), particularly those in theIndicfamily, suffer from a disproportionateIndic Taxcharacterized by high fertility rates and poor semantic anchoring\[morphtok2024,Lian2025\-ch\]\. TraditionalBpefails to captureSandhi\(phonetic fusions\) and routinely violates the integrity of theAbugidascript by decouplingmatras\(dependent vowels\) from base consonants\[Banerjee2018\-hd\]\. Prior work has explored morphology\-aware tokenization and attention\-guided segmentation\[morphtok2024,asgari2025morphbpe,jabbar2024morphpiecelinguistictokenizer\], but either treats morphology as a pre\-tokenization heuristic or targets specific languages \(English, Hebrew, Turkish\) rather than Indic abugidas\[jabbar2024morphpiecelinguistictokenizer,seker\-etal\-2022\-alephbert,Toraman\_2023\]\. Although contemporary approaches likeAg\-Bpe\[charlet\_2025\_agbpe\_v3\]target semantic awareness through attention\-guided scoring, it fails to explicitly preserve the atomic boundaries of Indic scripts\. In contrast, our approach works with penalty driven morphological violations, which exhibit higher sensitivity to preserving token\-level semantic coherence\.

## 3Morphological Dataset Preparation

A critical bottleneck forMRLsubword tokenization is the lack of verified ground\-truth segmentation\. Large\-scale resources like Unimorph 4\.0\[batsuren2022unimorph40universalmorphology\]and MorphyNet\[batsuren\-etal\-2021\-morphynet\]are primarilyparadigm\-centric—mapping lemmas to inflected forms without defining explicit boundaries\. Similarly, specific datasets like GujMORPH\[baxi\-bhatt\-2022\-gujmorph\]rely on rule\-based stemming, failing to restore oblique roots or handle phonetic fusion \(Sandhi\)\. To address this, we constructed aGold Standard Morphological Split DatasetforHindi,Marathi, andGujarati\. As shown in Table[1](https://arxiv.org/html/2608.18087#S3.T1), our dataset enforcesRoot Restoration\(e\.g\., mapping obliquegharato rootghar\) via an LLM\-in\-the\-loop pipeline\.

Table 1:Morphological Datasets\.We provide the first high\-scale, LLM\-verified morphological coverage for Indic scripts\.### 3\.1Dataset Pipeline

We employ a three\-stage hybrid pipeline—corpus extraction, unsupervised decomposition, and LLM verification \(details inSup\. Mat\.\)\. First, we extract a diverse vocabulary fromIndicCorp\[ai4bharat\_corpus\]\. For decomposition, we adaptSampoNlp\[chelombitko2025samponlp\]to the Indo\-Aryan context, leveraging itsMinimum Description Lengthatomicity scoring while modifying degeneracy constraints to accommodate single\-character Indic affixes andmatras\. Next,Gemini 3\.0 Flash\[team2023gemini\]resolves ambiguities \(e\.g\.,Sandhi\) and corrects over\-segmentation\. This yields a Gold Standard lexicon of∼\\sim560,000verified words \(Table[2](https://arxiv.org/html/2608.18087#S3.T2)\), acting asSuTRA's morphological probe\. Crucially, we employ adual\-layer logic: the lexicon stores purecanonical splits, whileSuTRAmaps these tosurface boundary indicesduring training\. We prioritizesurface\-form integrityover canonical purity to guarantee zero\-overhead detokenization, enabling exact recovery via simple concatenation\.

Table 2:Gold Standard Statistics\.Hybrid of rule\-based extraction andLLM\-verified morphological segmentation\.

## 4Methodology

![Refer to caption](https://arxiv.org/html/2608.18087v1/Images/sutra_final.png)Figure 2:Overview ofSuTRA\.Phase 1 \(Pre\-tokenization\)applies orthographic rulesΦ\\Phito map each word into akshara\-like units and uses a gold morphological lexicon or a seq2seq model to markforbidden boundaries\(morpheme boundaries that merges should not cross\)\.Phase 2 \(Morphology\-Aware Merging\)runs a BPE\-style algorithm with scoresS​\(a,b\)=f​\(a,b\)​Ψ​\(a,b\)γtS\(a,b\)=f\(a,b\)\\,\\Psi\(a,b\)^\{\\gamma\_\{t\}\}, whereΨ\\Psidownweights merges that violate forbidden boundaries andγt\\gamma\_\{t\}controls rigidity over training, biasing vocabulary toward merges that respect Indic script structure and morpheme boundaries\. Figure generated using PaperBanana\[paperbanana\]\.We introduceSuTRA, a two\-phase framework that integrates morphological priors into statistical subword learning \(Figure[2](https://arxiv.org/html/2608.18087#S4.F2)\)\. The pipeline systematically transitions tokenization from purely frequency\-driven to morphologically grounded segmentation\.

Phase 1: Pre\-tokenization\.Building onMorphTok, this phase defines atomic units and marks morphologically sensitive boundaries\. Standard tokenizers often isolate dependent vowels \(matras\), breaking the akshara structure of Indic abugida scripts\. To avoid such script\-breaking splits, we apply orthographic rules \(Φ\\Phi\) that map each wordwwto a sequence of akshara\-like unitsU=\[u1,u2,…,un\]U=\[u\_\{1\},u\_\{2\},\\dots,u\_\{n\}\], binding modifiers to their base consonants\. In parallel, we perform a morphological lookup to flagforbiddenboundaries—boundaries between akshara units that coincide with morpheme boundaries in our gold lexicon\. A curated lexicon is used for known words, and a fine\-tuned character\-level seq2seq model infers boundaries for out\-of\-vocabulary items\.

Phase 2: Training \(Morphology Aware Merging\)\.WhileMorphTokrelies on rigid, pre\-computed boundary enforcement,SuTRAdifferentiates itself by integrating these morphological priors directly into a score\-basedBpemerging process\. Instead of pure frequency\-based merging,SuTRAdynamically penalizes candidate merges that cross the forbidden boundaries identified in Phase 1\. Letf​\(a,b\)f\(a,b\)be the corpus frequency of a candidate pair\(a,b\)\(a,b\), andχ​\(a,b\)\\chi\(a,b\)be itsconflict count\(the number of occurrences where merging\(a,b\)\(a,b\)crosses a forbidden boundary\)\. The morphological validity probabilityΨ​\(a,b\)∈\[0,1\]\\Psi\(a,b\)\\in\[0,1\]is:

Ψ​\(a,b\)=1−χ​\(a,b\)f​\(a,b\)\\Psi\(a,b\)=1\-\\frac\{\\chi\(a,b\)\}\{f\(a,b\)\}To govern the merging process, we introduce a dynamicRigidity Constraint\(γt\\gamma\_\{t\}\)\. The final merge scoreS​\(a,b\)S\(a,b\)is defined as:

S​\(a,b\)=f​\(a,b\)⋅Ψ​\(a,b\)γtS\(a,b\)=f\(a,b\)\\cdot\\Psi\(a,b\)^\{\\gamma\_\{t\}\}During training,γt\\gamma\_\{t\}is annealed fromγs​t​a​r​t\\gamma\_\{start\}toγe​n​d\\gamma\_\{end\}\. This exponential decay forms a curriculum: the algorithm strictly prioritizes merging safe, continuous lexical roots early in training \(high rigidity\) before relaxing the constraint to attach functional affixes later \(low rigidity\)\. Figure[3](https://arxiv.org/html/2608.18087#S4.F3)shows thatSuTRAconstraints translate into cleaner segmentations that preserve the akshara structure and morpheme boundaries as compared to other tokenizers\. The detailed algorithm along with complexity analysis is given inSup\. Mat\.

![Refer to caption](https://arxiv.org/html/2608.18087v1/Images/ComparisonTable.png)Figure 3:Qualitative Comparison of Morphological Segmentation Across Tokenizers\.SuTRAconsistently matches the Gold Standard by respecting phonetic and morphological boundaries\.
## 5Experiments and Results

To evaluate the efficacy ofSuTRA, we conduct experiments on three morphologically rich Indic languages: Hindi \(HI\), Marathi \(MR\), and Gujarati \(GU\)\. Our evaluation is organized around four questions that mirror our central claims: \(1\) doesSuTRAbetter align subwords with gold morpheme boundaries? \(2\) do these boundaries make whole\-word semantics more directly recoverable from subword embeddings? \(3\) do the resulting representations yield measurable gains on downstream machine translation? and \(4\) doesSuTRAmaintain morphological robustness under orthographic perturbations? We therefore assessSuTRAalong four complementary axes: morphological alignment, semantic recoverability, translation quality, and robustness to surface noise\.

### 5\.1Morphological Alignment Evaluation

We first evaluate whether tokenizer boundaries align with gold morpheme segmentations in Hindi \(HI\), Marathi \(MR\), and Gujarati \(GU\)\.Setup and Metrics\.We normalize tokenizer outputs by stripping subword artifacts \(e\.g\.,\#\#,\_\) to enable direct string matching with gold morphemes\. We report: \(1\)Boundary F1—the harmonic mean of precision and recall over predicted vs\. gold boundary indices; and \(2\)Fertility Ratio—the mean number of predicted tokens per gold segment \(\|Tpred\|/\|Tgold\|\|T\_\{\\text\{pred\}\}\|/\|T\_\{\\text\{gold\}\}\|\)\.

Table 3:Morphological Alignment Evaluation\.Evaluation of Boundary F1 and Fertility Ratio across Hindi, Marathi, and Gujarati\. Higher F1 indicates superior structural integrity, while controlled fertility prevents arbitrary character\-level fragmentation\. Best F1 scores are in bold\.Results and Discussion\.Table[3](https://arxiv.org/html/2608.18087#S5.T3)shows thatSuTRAachieves the highest Boundary F1 for Hindi \(0\.586\) and Marathi \(0\.617\), and remains competitive for Gujarati, while keeping fertility within a moderate range\. High\-fertility baselines such as SuperBPE andMorphTokproduce many more tokens per morpheme yet still underperform on F1, indicating over\-fragmentation rather than genuine morphological alignment\. By enforcingaksharaatomicity and penalizing merges that cross morpheme boundaries,SuTRA's extra segments tend to correspond to functional affixes rather than the Morphological Shattering observed in purely frequency\-driven tokenizers\. Extended metrics are reported in theSup\. Mat\.

### 5\.2Semantic Recoverability

Arbitrary subword fragmentation forces models to expend capacity reconstructing basic lexical semantics\. We therefore ask whetherSuTRA's boundaries make whole\-word meaning more directly recoverable from subword embeddings\.

Setup & Metrics\.We train a Word2Vec model\[mikolov2013efficientestimationwordrepresentations\]on a joint vocabulary containing both words and subwords\. After training, we freeze these embeddings and train lightweight diagnostic models to reconstruct a target whole\-word embedding from the mean\-pooled embeddings of its constituent subwords\. We measuresemantic recoverabilityusingR2R^\{2\}, the proportion of variance in the target embedding that is predictable from the subwords\. ALinearprobe \(no hidden layers\) tests simple additive composition, while aMulti\-Layer Perceptron \(MLP\)probe tests whether more complex morphological interactions remain recoverable \(SeeSup\. Mat\.for details\.\)

Table 4:Semantic Recoverability \(R2R^\{2\}\)\.Linear models evaluate immediate compositionality, while deeper MLP models test the preservation of recoverable structural signals\.Results & Discussion\.Table[4](https://arxiv.org/html/2608.18087#S5.T4)shows thatSuTRAgenerally preserves semantic information more effectively thanBpe\. In Hindi,SuTRAyields a \+34% relative gain in LinearR2R^\{2\}overBpe, and the MLP provides only marginal improvement forBpe\(0\.33→\\rightarrow0\.34\), suggesting that little additional structure is recoverable beyond simple addition\. In Marathi and Gujarati, Linear scores are similar, reflecting dense orthography and complex stem alternations, butSuTRAbenefits substantially from the MLP \(R2\>0\.50R^\{2\}\>0\.50in both cases\), whereasBpeimproves only modestly\. This confirmsSuTRApreserves recoverable structural signals that arbitrary fragmentation destroys\.

### 5\.3Machine Translation Performance

To assess downstream utility, we evaluateSuTRAon aHindi↔\\leftrightarrowMarathitranslation task using a 3\-layer Transformer\[vaswani2017attention\]trained on theBhasaAnuvaadcorpus\[jain2024bhasaanuvaad\]\.

Implementation\.We use a standard Transformer encoder–decoder \(L=3,H=4,dff=400L=3,H=4,d\_\{\\mathrm\{ff\}\}=400\) implemented infairseq\[ott2019fairseq\]\. All models are trained for 100k updates with a shared 32k vocabulary per tokenizer andIndicNLPnormalization, keeping architecture and optimization hyperparameters identical across baselines to isolate the impact of segmentation \(seeSup\. Mat\.for full details\)\.

Results\.Table[5\.3](https://arxiv.org/html/2608.18087#S5.SS3)shows thatSuTRAattains the best scores forMarathi→\\rightarrowHindiand remains competitive with the strongest baseline in the reverse direction, indicating that improved morphological alignment and semantic recoverability translate into meaningful gains for neural MT\. A secondary evaluation on causal language modeling, along with metric derivations and full hyperparameter settings, is provided in theSup\. Mat\.; these tasks follow standard evaluation protocols in prior work\[morphtok2024\]\.

Table 5:Machine Translation\.SuTRAachieves the highest scores inMarathi→\\rightarrowHindiand remains comparable to the strongest baseline in the reverse task, demonstrating that root contiguity improves cross\-lingual alignment\.### 5\.4Morphological Robustness Evaluation

We evaluate tokenizer resilience to small surface changes \(e\.g\., typos or suffix substitutions\) that should not affect the underlying root but often trigger unstable segmentations\. Using an adversarial suite of 10,000 words per language \(HI, MR, GU\) with synthetic orthographic and inflectional perturbations, we measure robustness viaJaccard Overlap\(Jac\.↑\\uparrow\) between original and perturbed segmentations andRoot\-Affected Distance\(R\.Aff\.↓\\downarrow\), which tracks how often perturbations alter the subwords covering the lexical root\.

Results & Discussion\.As shown in Table[6](https://arxiv.org/html/2608.18087#S5.T6), purely statistical baselines \(Bpe,Unigram\) exhibit high R\.Aff\., indicating that minor variations frequently bleed into the core lexeme\. In contrast,SuTRAattains near\-zero R\.Aff\. while maintaining higher Jaccard overlap, showing that morphological priors effectively insulate roots from peripheral noise\. This structural stability complements our semantic\-recoverability and MT results, confirming thatSuTRAmaintains consistent tokenization under perturbations\. Extended details are in theSup\. Mat\.\.

Table 6:Morphological Robustness Comparison\.Results show thatSuTRAeffectively mitigates root fragmentation\.## 6Conclusion and Future Work

Our results tell a consistent story\. When tokenization is constrained to respect Indic script structure and morpheme boundaries, Morphological Shattering is reduced, and whole\-word semantics become more linearly recoverable from subword embeddings\. This, in turn, increases robustness and yields tangible gains on downstream translation, without increasing token fertility or vocabulary size\.SuTRAthus validates our core hypothesis: guiding frequency\-based subword learning with lightweight morphological priors is a practical way to improve both interpretability and utility ofLLMtokenizers for morphologically rich languages\. Future work includes extendingSuTRAto other morphologically rich languages, exploring unsupervised anchoring for low\-resource scripts, and assessing its impact on additional text and speech tasks such as Text to Speech \(TTS\) and Automatic Speech Recognition \(ASR\)\.

## 7Acknowledgements

This research was conducted at Motilal Oswal Financial Services Limited\. The authors gratefully acknowledge the organization for providing the resources, infrastructure, and support that made this work possible\.

## 8Use of Generative AI Disclosure

In accordance with Interspeech 2026 guidelines, the authors disclose the following uses of Generative AI tools during the execution of this research and the preparation of this manuscript:

- •Dataset Construction:We utilized Gemini 3\.0 Flash\[team2023gemini\]as an LLM\-in\-the\-loop verifier within our dataset pipeline to resolve morphophonological ambiguities \(e\.g\.,Sandhi\) and correct algorithmic over\-segmentation\.
- •Visualizations:Figure[1](https://arxiv.org/html/2608.18087#S1.F1)and Figure[2](https://arxiv.org/html/2608.18087#S4.F2)was generated utilizing PaperBanana\[paperbanana\]\.
- •Writing Assistance:LLMs were utilized for minor language polishing, structural refinement, and LaTeX formatting during the drafting process\.

The human authors rigorously reviewed, modified, and validated all AI\-assisted outputs and take full responsibility for the final content, scientific accuracy, and integrity of this work\.

## References

SuTRA: Supplementary Material

Summary of Supplementary Contents

1. Section 1:Methodology for Morphological Dataset Preparation\(Page[9](https://arxiv.org/html/2608.18087#S9)\)

Details on the three\-stage pipeline \(SampoNLP, statistical filtering, and LLM\-based verification\) used to construct the Gold Standard dataset\.

Section 2:The Rigidity Constraint \(γ\\gamma\)\(Page[10](https://arxiv.org/html/2608.18087#S10)\) Formal derivation of the dynamic annealing schedule and the linguistic justification for the curriculum\-based merging approach\.

Section 3:Algorithm forSuTRA\(Page[1](https://arxiv.org/html/2608.18087#alg1)\) Complete pseudocode for the Semantic\-Unit Tokenization with Rigidity Annealing process, including pre\-tokenization and trilateral enforcement\.

Section 4:Computational Complexity Analysis\(Page[12](https://arxiv.org/html/2608.18087#S12)\) A theoretical comparison of training and inference time complexities betweenSuTRA, standardBpe, andMorphTok\.

Section 5:Extended Analysis of Morphological Alignment\(Page[9](https://arxiv.org/html/2608.18087#S13.T9)\) Full evaluation metrics \(Exact Match, Precision, Recall\) across all target languages, expanding on the main paper's alignment results\.

Section 6:Experimental Setup for Neural Machine Translation\(Page[14\.1](https://arxiv.org/html/2608.18087#S14.SS1)\) Detailed architectural configurations, optimization hyperparameters, and data preprocessing protocols for the translation tasks\.

Section 7:Experiment: Causal Language Modeling\(Page[11](https://arxiv.org/html/2608.18087#S15.T11)\) Evaluation of predictive efficiency, perplexity derivations, and training loss curves for generative modeling\.

Section 8:Extended Analysis of Morphological Robustness\(Page[12](https://arxiv.org/html/2608.18087#S16.T12)\) A deep dive into the "Morphological Shattering" phenomenon and secondary metrics quantifying tokenizer resilience under noise\.

Section 9:Limitations\(Page[17](https://arxiv.org/html/2608.18087#S17)\) Discussion on vocabulary scaling and the boundaries of semantic constraint saturation\.

## 9Methodology for Morphological Dataset Preparation

We developed a robust, three\-stage pipeline to construct the Gold Standard Morphological Split Dataset\. Our approach integrates unsupervised information\-theoretic segmentation with semantic verification usingGemini 3\.0 Flash\[team2023gemini\]\. This hybrid methodology allows us to leverage the high recall of statistical segmentation while ensuring precision through expert\-aligned semantic reasoning\.

### 9\.1Stage 1: Adapted Iterative Morphological Decomposition \(SampoNLP\)

To generate initial candidate splits, we adapted theSampoNLPpipeline\[chelombitko2025samponlp\]\. This framework operates on the principle ofSelf\-Referential Atomicity Scoring, where the "atomic" nature of a morpheme is determined iteratively by its ability to optimally compress the lexicon\.

While the core logic is language\-agnostic, the original implementation relied on Latin\-script constraints\. We adapted the pipeline for Indic languages \(Hindi,Marathi,Gujarati\) through the following modifications:

1. 1\.Script\-Specific Hard Filtering:We redefined the valid character setΣ\\Sigmato include the Unicode blocks for Devanagari \(U\+0900–U\+097F\) and Gujarati \(U\+0A80–U\+0AFF\)\.
2. 2\.Atomic Whitelist Expansion \(WW\):We significantly expanded the whitelistWWto includeMatras \(Dependent Vowels\)\(e\.g\.,A\(U\+093E\)\) and short grammatical suffixes\. This ensures that legitimate single\-character morphemes are not aggressively merged into adjacent roots\.

### 9\.2Stage 2: Limitations of Statistical Segmentation

While the adapted pipeline successfully identified high\-frequency recurring patterns, we observed a systematic error pattern defined asStatistical Over\-segmentation\. The algorithm would frequently split a root word if it contained a substring resembling a high\-frequency suffix \(e\.g\., splitting Hindishriman\(Mr\.\) intoshri\+man\), ignoring the semantic integrity of the root\. This necessitated a semantic verification layer\.

### 9\.3Stage 3: LLM\-Based Semantic Verification

To resolve over\-segmentation and restore oblique roots, we employedGemini 3\.0 Flash\[team2023gemini\]\. We utilized a unified system prompt template across all languages, dynamically injecting language\-specific morphological rules to handle the distinction between agglutinative Indic scripts and fusional English morphology\.

Unified System Prompt Template Used for VerificationRole:You are an expert linguist specializing in\[Target\_Language\]Morphology\. Task:Your task is to segment the given word into its constituent morphemes: the Root and any Suffixes\.\[Target\_Language\]is a morphologically rich language\. Pay close attention to:\[Language\_Specific\_Rules\] Return the answer in strict JSON format:``` { "is_correct": boolean, "correct_split": "Prefix+Root+Suffix", "morphology": { "root": "The base dictionary form (e.g., 'ghar', 'happy')", "prefixes": [], "suffixes": ["grammatical suffixes"], "stem": "Modified stem if applicable" }, "origin": "Native [Target_Language]", "error_type": "None" } ```

Language\-Specific Rule InjectionsMarathi / Hindi Rules:1\.Oblique forms \(Samanya Rup\):Restore the root to its original dictionary form \(e\.g\., 'ghara'→\\rightarrowRoot: 'ghar', Suffix: 'cha'\)\.2\.Plural markers\.3\.Case markers \(Vibhakti\)\.Gujarati Rules:1\.Root Identification:Identify the base dictionary form \(e\.g\., for 'gharnu', the root is 'ghar'\)\.2\.Oblique Forms:Restore the root if it changed form before a suffix was added \(e\.g\., 'chokraao'→\\rightarrowRoot: 'chokro', Suffix: 'ao'\)\.3\.Suffixes:Isolate grammatical markers for case \(nu/ni/no/na\), plurality \(o\), and postpositions\.English Rules:1\.Root Restoration:Restore the root's original spelling if it changed during affixation\.•Example: 'happiness'→\\rightarrowRoot: 'happy' \(NOT 'happi'\)\.•Example: 'running'→\\rightarrowRoot: 'run' \(NOT 'runn'\)\.2\.Affixation:Identify standard prefixes \(e\.g\., 'un\-', 're\-'\) and suffixes \(e\.g\., '\-ing', '\-ed'\)\.3\.Compound Words:If the word is a compound, split it into its constituent free bases\.

Justification for the Prompt Design:The prompt explicitly enforcesRoot Restoration\. Standard statistical segmenters operate strictly on surface forms and cannot recover the original dictionary root \(e\.g\., mapping \*happi\* back to \*happy\* or \*ghara\* back to \*ghar\*\)\. By enforcing a structured JSON output with a separate field for the "root", we ensure that the resulting dataset captures the deep morphological structure, not just surface segmentation\.

## 10The Rigidity Constraint \(γ\\gamma\)

In theSuTRAframework, the merging process is governed by a dynamic Rigidity Constraint \(γt\\gamma\_\{t\}\), which acts as an exponential penalty on the semantic validity probabilityΨ​\(a,b\)\\Psi\(a,b\)of a candidate merge pair\. The score is calculated asS​\(a,b\)=f​\(a,b\)⋅Ψ​\(a,b\)γtS\(a,b\)=f\(a,b\)\\cdot\\Psi\(a,b\)^\{\\gamma\_\{t\}\}\.

Annealing Formulation\.To transition the vocabulary construction from strict morphological adherence to optimal statistical compression, we annealγt\\gamma\_\{t\}linearly over the course of theBpetraining steps\. Letttrepresent the current merge iteration andTTrepresent the total number of target merges to reach the desired vocabulary size\. The constraint at stepttis defined as:γ\_t =γ\_start \- \(γ\_start \-γ\_end \) ⋅\( tT \)

Justification for the Curriculum Approach\.The intuition behind this dynamic decay is rooted in curriculum learning\. Early in theBpetraining process, the algorithm combines short character sequences to form the foundational roots and affixes of the language\. If these foundational units are permitted to cross morphological boundaries, the error propagates, causing theMorphological Shatteringobserved in standard tokenizers\. By enforcing a high rigidity constraint early on, we force the algorithm to build "safe," semantically pure subwords\. As the vocabulary grows \(t→Tt\\to T\), the algorithm needs to form longer, composite tokens \(e\.g\., highly frequent full words\) to ensure statistical efficiency and maintain a competitive compression ratio\. Relaxing the constraint allows these necessary larger merges to occur once the foundational roots are already securely established\.

Whyγ∈\[4,0\]\\gamma\\in\[4,0\]?We specifically tune the annealing schedule to boundγt\\gamma\_\{t\}between44and0\.

- •Starting atγs​t​a​r​t=4\\gamma\_\{start\}=4:Because the semantic validityΨ​\(a,b\)\\Psi\(a,b\)is a probability between0and11, raising it to the power of44acts as an aggressive, non\-linear amplifier for any boundary conflicts\. For example, if a candidate merge crosses a forbidden boundary in just10%10\\%of its occurrences \(Ψ=0\.9\\Psi=0\.9\), its score is severely penalized \(Ψ4≈0\.65\\Psi^\{4\}\\approx 0\.65\), effectively blocking the merge in favor of structurally safer alternatives\.
- •Ending atγe​n​d=0\\gamma\_\{end\}=0:Asγt\\gamma\_\{t\}approaches0, the penalty factorΨ​\(a,b\)0\\Psi\(a,b\)^\{0\}approaches11\. At the very last merge step \(t=Tt=T\), the scoring function cleanly reduces toS​\(a,b\)=f​\(a,b\)⋅1S\(a,b\)=f\(a,b\)\\cdot 1, which is the exact formulation of standardBpe\. This ensures that the final stages of vocabulary construction are purely frequency\-driven, allowingSuTRAto match the fertility and sequence compression efficiency of baseline tokenizers without sacrificing the structural integrity built during the early phases\.

Table 7:Effect of the Annealed Rigidity Constraint\.At the start of training \(t=0,γ=4t=0,\\gamma=4\), the exponential penalty prioritizes structurally safe merges \(Ψ≈1\.0\\Psi\\approx 1\.0\), forcing the tokenizer to build valid semantic roots despite lower raw frequencies\. By the end of training \(t=T,γ=0t=T,\\gamma=0\), the penalty disappears \(Ψ0=1\\Psi^\{0\}=1\), reducing the score to standardBpefrequency to optimize compression\.## 11Algorithm forSuTRA

Algorithm 1SuTRA: Semantic\-Unit Tokenization with Rigidity Annealing1:Corpus

𝒞\\mathcal\{C\}, Target Vocabulary Size

KK, Annealing Bounds

\[γs​t​a​r​t,γe​n​d\]\[\\gamma\_\{start\},\\gamma\_\{end\}\]
2:Final Vocabulary

𝒱\\mathcal\{V\}, Merge Rules

ℛ\\mathcal\{R\}
3:Phase 1: Pre\-tokenization & Boundary Identification

4:

𝒞′←ApplyOrthographicRules​\(𝒞,Φ\)\\mathcal\{C\}^\{\\prime\}\\leftarrow\\text\{ApplyOrthographicRules\}\(\\mathcal\{C\},\\Phi\)⊳\\trianglerightGroup chars into Aksharas

5:

𝒱0←UniqueUnits​\(𝒞′\)\\mathcal\{V\}\_\{0\}\\leftarrow\\text\{UniqueUnits\}\(\\mathcal\{C\}^\{\\prime\}\)
6:

ℬf​o​r​b​i​d​d​e​n←IdentifyMorphologicalBoundaries​\(𝒞\)\\mathcal\{B\}\_\{forbidden\}\\leftarrow\\text\{IdentifyMorphologicalBoundaries\}\(\\mathcal\{C\}\)⊳\\trianglerightLexicon/Seq2Seq

7:Phase 2: Morphology\-Aware Merging \(Curriculum Learning\)

8:

𝒱←𝒱0,ℛ←∅\\mathcal\{V\}\\leftarrow\\mathcal\{V\}\_\{0\},\\mathcal\{R\}\\leftarrow\\emptyset
9:while

\|𝒱\|<K\|\\mathcal\{V\}\|<Kdo

10:

p←\|𝒱\|−\|𝒱0\|K−\|𝒱0\|p\\leftarrow\\frac\{\|\\mathcal\{V\}\|\-\|\\mathcal\{V\}\_\{0\}\|\}\{K\-\|\\mathcal\{V\}\_\{0\}\|\}⊳\\trianglerightTraining Progress

11:

γt←γs​t​a​r​t⋅\(γe​n​dγs​t​a​r​t\)p\\gamma\_\{t\}\\leftarrow\\gamma\_\{start\}\\cdot\(\\frac\{\\gamma\_\{end\}\}\{\\gamma\_\{start\}\}\)^\{p\}⊳\\trianglerightExponential Annealing

12:Step A: Collect Statistics

13:for alladjacent pairs

\(a,b\)∈𝒞′\(a,b\)\\in\\mathcal\{C\}^\{\\prime\}do

14:

f​\(a,b\)←CountFreq​\(a,b\)f\(a,b\)\\leftarrow\\text\{CountFreq\}\(a,b\)
15:

χ​\(a,b\)←CountForbiddenBoundaryCrossings​\(a,b,ℬf​o​r​b​i​d​d​e​n\)\\chi\(a,b\)\\leftarrow\\text\{CountForbiddenBoundaryCrossings\}\(a,b,\\mathcal\{B\}\_\{forbidden\}\)
16:endfor

17:Step B: Score and Merge

18:

Ψ​\(a,b\)←1−χ​\(a,b\)f​\(a,b\)\\Psi\(a,b\)\\leftarrow 1\-\\frac\{\\chi\(a,b\)\}\{f\(a,b\)\}⊳\\trianglerightMorphological Validity

19:

\(a∗,b∗\)←arg⁡max⁡\(f​\(a,b\)⋅Ψ​\(a,b\)γt\)\(a^\{\*\},b^\{\*\}\)\\leftarrow\\arg\\max\\left\(f\(a,b\)\\cdot\\Psi\(a,b\)^\{\\gamma\_\{t\}\}\\right\)
20:

𝒱←𝒱∪\{a∗​b∗\}\\mathcal\{V\}\\leftarrow\\mathcal\{V\}\\cup\\\{a^\{\*\}b^\{\*\}\\\}
21:

ℛ←ℛ∪\{\(a∗,b∗\)→a∗​b∗\}\\mathcal\{R\}\\leftarrow\\mathcal\{R\}\\cup\\\{\(a^\{\*\},b^\{\*\}\)\\to a^\{\*\}b^\{\*\}\\\}
22:

𝒞′←ApplyMerge​\(𝒞′,a∗,b∗\)\\mathcal\{C\}^\{\\prime\}\\leftarrow\\text\{ApplyMerge\}\(\\mathcal\{C\}^\{\\prime\},a^\{\*\},b^\{\*\}\)
23:endwhile

24:Phase 3: Structural Bottleneck \(Post\-processing\)

25:for allword

w∈𝒞′w\\in\\mathcal\{C\}^\{\\prime\}do

26:if

SegmentCount​\(w\)\>3\\text\{SegmentCount\}\(w\)\>3then

27:

IdentifyRootIndices​\(w,ℬf​o​r​b​i​d​d​e​n\)\\text\{IdentifyRootIndices\}\(w,\\mathcal\{B\}\_\{forbidden\}\)
28:

w←ForceMergeRootTokens​\(w\)w\\leftarrow\\text\{ForceMergeRootTokens\}\(w\)⊳\\trianglerightEnforceW=\[P\]⊕R⊕\[S\]W=\[P\]\\oplus R\\oplus\[S\]

29:endif

30:endfor

## 12Computational Complexity Analysis forSuTRA

This section details the time complexity ofSuTRAduring vocabulary construction \(training\) and text encoding \(inference\), compared against standardBpeandMorphTok\. LetNNbe the total characters in the training corpus,VVbe the target vocabulary size \(number of merges\),MMbe the number of unique active candidate pairs,Vi​nV\_\{in\}be the number of unique words in the training corpus, and\|w\|\|w\|be the length of a given word\.

We utilize ByT5 for morphological boundary prediction in the baselineMorphTokand the pre\-segmentation phase ofSuTRA\. As a byte\-level Transformer, ByT5 exhibits quadratic time complexity with respect to sequence length, yielding an inference complexity of𝒪​\(\|w\|2\)\\mathcal\{O\}\(\|w\|^\{2\}\)per word\.

### 12\.1Training Complexity

StandardBpe:The initial character pair frequency calculation takes𝒪​\(N\)\\mathcal\{O\}\(N\)\. ExecutingVVmerges requires updating the frequencies of adjacent pairs, taking𝒪​\(log⁡M\)\\mathcal\{O\}\(\\log M\)per update\. The total training complexity is𝒪​\(N\+V​log⁡M\)\\mathcal\{O\}\(N\+V\\log M\)\.

MorphTok:MorphTokrequires pre\-segmenting the corpus\. Generating boundaries forVi​nV\_\{in\}unique words using ByT5 takes𝒪​\(Vi​n⋅\|w\|2\)\\mathcal\{O\}\(V\_\{in\}\\cdot\|w\|^\{2\}\)\. After segmentation, standardBpeis executed within the constrained boundaries\. The total training complexity is𝒪​\(Vi​n⋅\|w\|2\+N\+V​log⁡M\)\\mathcal\{O\}\(V\_\{in\}\\cdot\|w\|^\{2\}\+N\+V\\log M\)\.

SuTRA\(Ours\):SuTRAinvolves two primary training phases:

1. 1\.Boundary Extraction:Similar toMorphTok, extracting boundaries for unique corpus words via ByT5 takes𝒪​\(Vi​n⋅\|w\|2\)\\mathcal\{O\}\(V\_\{in\}\\cdot\|w\|^\{2\}\)\.
2. 2\.Score Calculation and Merging:Initializing the boundary conflict countsχ​\(a,b\)\\chi\(a,b\)and valid frequenciesf​\(a,b\)f\(a,b\)across the corpus takes𝒪​\(N\)\\mathcal\{O\}\(N\)\. During each of theVVmerges, the algorithm updates the semantic validity probabilityΨ​\(a,b\)\\Psi\(a,b\)and calculates the annealed scoreS​\(a,b\)=f​\(a,b\)⋅Ψ​\(a,b\)γtS\(a,b\)=f\(a,b\)\\cdot\\Psi\(a,b\)^\{\\gamma\_\{t\}\}\. The exponentiation and scalar multiplication add an𝒪​\(1\)\\mathcal\{O\}\(1\)arithmetic operation per pair update\. The merge phase therefore remains bounded by𝒪​\(V​log⁡M\)\\mathcal\{O\}\(V\\log M\)\.

The total training complexity forSuTRAis𝒪​\(Vi​n⋅\|w\|2\+N\+V​log⁡M\)\\mathcal\{O\}\(V\_\{in\}\\cdot\|w\|^\{2\}\+N\+V\\log M\), which is asymptotically identical toMorphTok\.

### 12\.2Inference Complexity

StandardBpe:Encoding a word applies learned merge rules iteratively\. The inference complexity is𝒪​\(\|w\|\)\\mathcal\{O\}\(\|w\|\)\.

MorphTok:MorphTokenforces boundaries at runtime\. Unseen words must pass through the ByT5 model before subword merging can occur\. The inference complexity is dominated by the Transformer forward pass, resulting in𝒪​\(\|w\|2\)\\mathcal\{O\}\(\|w\|^\{2\}\)per out\-of\-vocabulary word\.

SuTRA\(Ours\):SuTRAincorporates the morphological constraints directly into the final vocabulary merge rules during training\. At inference, boundary prediction and score calculations are omitted\. The model relies entirely on the learned merges, resulting in an inference complexity of𝒪​\(\|w\|\)\\mathcal\{O\}\(\|w\|\)\.

Table 8:Time Complexity Summary\.Comparison of training and inference time complexity\. Variables:NN\(corpus size\),VV\(vocab size\),MM\(unique candidate pairs\),Vi​nV\_\{in\}\(unique corpus words\), and\|w\|\|w\|\(word length\)\.## 13Extended Analysis of Morphological Alignment\.

The streamlined results in Table[9](https://arxiv.org/html/2608.18087#S13.T9)highlight the tension between subword compression \(Fertility\) and linguistic integrity \(Boundary F1\):

Table 9:Comparison of Tokenizers \(Normalized\) across Hindi, Marathi, and Gujarati\. EM: Exact Match Acc, Prec: Boundary Precision, Rec: Boundary Recall, F1: Boundary F1, Fert: Fertility Ratio\. Best F1 scores are in bold\.1. 1\.Efficiency\-Accuracy Frontier:SuTRA\(Ours\) achieves the highestBoundary F1in Hindi \(0\.5860\.586\) and Marathi \(0\.6170\.617\), establishing a new performance baseline\. Notably, it maintains high alignment scores despite a slightly higher fertility ratio than standard BPE\. This suggests that the additional segments produced bySuTRAare not "noise" but rather accurate captures of functional affixes that purely statistical methods often miss\.
2. 2\.The Morphological Shattering Threshold:We observe a clear inverse correlation between extreme fertility and boundary alignment\. Models that "shatter" words into excessive units, such asMorphTokandSuperBPE\(Fert\.\>2\.5\>2\.5\), consistently yield the lowest F1 scores \(<0\.25<0\.25\)\. This confirms that over\-segmentation in Indic scripts typically results in linguistically void fragments rather than meaningful sub\-morphemic units\.
3. 3\.Statistical Instability in Marathi:The catastrophic failure ofSentencePiecein Marathi \(Fertility:1\.1831\.183, F1:0\.0830\.083\) underscores the risk of purely data\-driven tokenization\. By reverting to character\-level splitting, the model loses all semantic anchoring\. In contrast, the phonetic guardrails inSuTRAensure stable segmentation \(0\.6170\.617F1\) even in the presence of complex Marathi conjuncts\.
4. 4\.Unigram Robustness in Gujarati:In Gujarati,Unigramachieves a peak F1 of0\.6690\.669with the lowest fertility \(1\.1371\.137\)\. This suggests that for certain scripts, the probabilistic pruning of a large initial vocabulary may capture stable roots more efficiently than the bottom\-up merging approach of BPE\-based architectures\.

These results validate that integrating orthographic constraints \(Φ\\Phi\) and morphological priors \(Ψ\\Psi\) creates a more robustsemantic anchorfor Indic LLMs, effectively mitigating theMorphological Shatteringprevalent in purely data\-driven baselines\.

## 14Experimental Setup for Neural Machine Translation

### 14\.1Neural Machine Translation

We evaluate the downstream utility ofSuTRAvia aHindi→\\rightarrowMarathitranslation task\. Our framework is designed to isolate the impact of tokenization by keeping architectural and optimization variables constant across all baseline comparisons\.

#### 14\.1\.1Data and Preprocessing

We utilize theBhasaAnuvaadparallel corpus\[jain2024bhasaanuvaad\], employing official train \(581379\), validation \(32298\), and test \(32300\) splits\. All tokenizer variants are trained and evaluated on identical sentence pairs to ensure a controlled environment\.

- •Normalization:Before tokenization, we apply Unicode canonicalization and language\-specific normalization viaIndicNLPto reduce orthographic variance\.
- •Vocabularies:We target a vocabulary size of 32k units per language\. Data is binarized usingfairseq\-preprocess\[ott2019fairseq\]with a shared subword dictionary betweenHindiandMarathi\.

![Refer to caption](https://arxiv.org/html/2608.18087v1/Images/Samples_Hi_Marathi.jpg)Figure 4:Sample examples from the dataset used for Hindi to Marathi Machine Translation Task
#### 14\.1\.2Model Architecture

We employ a standard Transformer encoder–decoder with the following configuration:

- •Depth and Width:3 encoder and 3 decoder layers; embedding dimensiondmodel=100d\_\{\\mathrm\{model\}\}=100; feed\-forward dimensiondff=400d\_\{\\mathrm\{ff\}\}=400\.
- •Attention:4 attention heads per layer with a dropout rate of 0\.2\.
- •Parameter Tying:Decoder input and output embeddings are tied to improve convergence and reduce parameter count\.

#### 14\.1\.3Training and Optimization

Models are trained for 100k updates using the Adam optimizer \(β1=0\.9,β2=0\.98\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.98\) with a label smoothing factor of 0\.1\. We utilize an inverse\-square\-root learning rate schedule with a peak of5×10−45\\times 10^\{\-4\}and 4,000 warmup updates\. Gradient clipping is enforced at 1\.0, and we use a token\-based batch size of 4,096 per update\. Best models are selected based on validation loss\.

#### 14\.1\.4Evaluation Protocol

Final performance is measured on detokenized outputs using beam search \(size 5\) and a length penalty of 1\.0\. We report:

- •Translation Quality:Bleu\[papineni\-etal\-2002\-bleu\]andchrF\[popovic\-2015\-chrf\]scores computed on detokenized text\.
- •Intrinsic Metrics:Validation Perplexity \(Ppl\), tokenfertilityratios, and Out\-of\-Vocabulary \(Oov\) percentages\.

## 15Experiment Causal Language Modeling

### 15\.1Metric Derivation: Perplexity

For the downstream Causal Language Modeling \(CLM\) task, we evaluate the predictive efficiency ofSuTRAusingPerplexity \(PPL\)\. Perplexity measures the model's uncertainty when predicting the next token in a sequence\. Formally, for a test sequence of tokensW=\(w1,w2,…,wN\)W=\(w\_\{1\},w\_\{2\},\\dots,w\_\{N\}\), the perplexity is derived from the totalNegative Log\-Likelihood \(NLL\)\.

LetP​\(wi∣w<i\)P\(w\_\{i\}\\mid w\_\{<i\}\)be the probability assigned by the model to theii\-th token given the preceding context\. The total NLL for the sequence is:

NLLtotal=−∑i=1Nlog⁡P​\(wi∣w<i\)\\text\{NLL\}\_\{\\text\{total\}\}=\-\\sum\_\{i=1\}^\{N\}\\log P\(w\_\{i\}\\mid w\_\{<i\}\)\(1\)
In our implementation, we calculate the average NLL per predicted token to account for varying sequence lengths:

Avg NLL=1Npred​NLLtotal\\text\{Avg NLL\}=\\frac\{1\}\{N\_\{\\text\{pred\}\}\}\\text\{NLL\}\_\{\\text\{total\}\}\(2\)whereNpredN\_\{\\text\{pred\}\}is the count of non\-padding tokens\. The Perplexity is then defined as the exponential of the average NLL:

P​P​L=exp⁡\(Avg NLL\)=e\(1Npred​NLLtotal\)PPL=\\exp\(\\text\{Avg NLL\}\)=e^\{\\left\(\\frac\{1\}\{N\_\{\\text\{pred\}\}\}\\text\{NLL\}\_\{\\text\{total\}\}\\right\)\}\(3\)

### 15\.2Implementation ofSuTRAPre\-tokenization

To ensure morphological grounding for Indic scripts, ourSCBPEPreTokenizerutilizes a regex\-based grouping strategy before BPE merging\. The pattern targets phonetic clusters \(aksharas\) across Devanagari and Gujarati Unicode blocks:

- •Devanagari:\\u0900\-\\u097F\(Base\) and\\u093A\-\\u094F\(Marks\)\.
- •Gujarati:\\u0A80\-\\u0AFF\(Base\) and\\u0ABC\-\\u0ACD\(Marks\)\.

This ensures that dependent vowels \(matras\), halants, and nasalization marks \(anusvara\) are never separated from their base consonant during the initial tokenization phase\. Additionally, a word\-boundary marker</w\>is appended to the final phonetic unit of every word to preserve lexical boundaries\.

### 15\.3Model Configuration and Training Hyperparameters

We trained a GPT\-2 style Transformer architecture from scratch using the Hugging FaceTrainerAPI\. The detailed hyperparameters are provided in Table[10](https://arxiv.org/html/2608.18087#S15.T10)\.

Table 10:Hyperparameters for the downstream Causal Language Modeling task\.
### 15\.4Evaluation Protocol

Out\-of\-distribution \(OOD\) perplexity was calculated on held\-out parquet datasets\. To ensure robustness, we used a row\-batch size of 256 and a sequence length limit of 256 tokens\. The model was evaluated ineval\(\)mode withtorch\.no\_grad\(\)to ensure deterministic NLL calculations\.

### 15\.5Results

![Refer to caption](https://arxiv.org/html/2608.18087v1/Images/loss_comparison_final_sutra.png)Figure 5:Loss curves for BPE andSuTRATable 11:Downstream performance on Causal Language Modeling \(CLM\)\. We report average Negative Log\-Likelihood \(Avg NLL\) and Perplexity \(PPL\) on out\-of\-distribution \(OOD\) corpora\. Lower values indicate better compression and predictive efficiency\.## 16Extended Analysis of Morphological Robustness Evaluation

### 16\.1Quantifying Morphological Resilience

Standard tokenizers often lack linguistic awareness, leading to arbitrary root fragmentation when faced with noisy text—a phenomenon we termMorphological Shattering\. We center our investigation around the following research question:

> To what extent can a semantics\-constrained tokenizer maintain the integrity of a language's root\-affix structure when faced with orthographic and morphological variations?

We evaluate tokenizer robustness using an adversarial test set of 10,000 unique words per language, generating variants via synthetic orthographic noise \(e\.g\., character swaps, deletions , substitute , insertions\) to simulate real\-world typos and morphological shifts\. To quantify the tokenizer's susceptibility to catastrophic re\-segmentation we compareSuTRAagainst six baselines using three metrics:Jaccard Overlap \(Jac\.↑\\uparrow\), which calculates the token\-level intersection over union to measure structural consistency;Root Affected Distance \(R\.Aff\.↓\\downarrow\), which assesses semantic vulnerability by measuring how deeply noise alters the core root's tokenization\. As shown in Table[12](https://arxiv.org/html/2608.18087#S16.T12), standard frequency\-based methods \(Bpe,Unigram\) exhibit high Root Affected Distance \(\>0\.22\>0\.22\), indicating that minor orthographic noise forces them to recalculate boundaries from scratch, thereby corrupting the semantic anchor\. In contrast,SuTRAachieves a Root Affected Distance near zero \(≈0\.04\\approx 0\.04\) \. This confirms that enforcing morphological priors successfully insulates the root, preserving linguistic utility even under adversarial conditions\.

Table 12:Morphological Shattering Comparison\.Results show thatSuTRAeffectively mitigates root fragmentation\.To provide a rigorous evaluation of how tokenizers handle the complex morphology of Indic languages, we analyze the relationship between eight distinct metrics\. These metrics collectively quantify theShattering effect—the phenomenon where a tokenizer loses semantic consistency when faced with orthographic variations\.

## 17Limitations

Vocabulary Scaling\. To ensure a controlled comparison across all tokenizers, we maintained a uniform vocabulary size of 32k\. While this provides a consistent baseline for evaluating architectural efficiency, the performance ofSuTRAunder larger vocabulary regimes \(e\.g\., 64k or 128k\) remains unexplored\. Future benchmarking is required to determine if increased capacity further mitigates Morphological Shattering or if the benefits of semantic constraints saturate at higher scales\.