超越原子标记：音节分解在语言模型预训练中的应用

# Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
Source: [https://arxiv.org/html/2609.21362](https://arxiv.org/html/2609.21362)
Journal:Artificial Intelligence JournalNghia Hieu Nguyena, Thai Bao Huynha, Binh\-An Dinh\-Lea, Phu Gia Hoangb, Dat Tien Nguyenc, Kiet Van Nguyena, Ngan Luu\-Thuy NguyenaAffiliation:University of Information Technology, Vietnam National University, Ho Chi Minh city, Viet NamAffiliation:Independent Researcher, GermanyAffiliation:Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Abu Dhabi, UEA

###### Abstract

Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies\. We introducePhonemic Tokenizer, a linguistically motivated tokenizer for Vietnamese and Chinese that converts each syllable into IPA and factorizes it into three phonological components: onset, rime, and tone\. The three components jointly occupy one contextual position, preserving syllable\-level sequence length while enabling representation sharing across phonologically related syllables\. Non\-phonological and unsupported units are handled through character\-level fallback\. This deterministic design requires no corpus\-dependent vocabulary learning and yields vocabularies of only 112 entries for Chinese and 256 for Vietnamese\. Intrinsic evaluation shows that the tokenizer achieves substantially higher Rényi efficiency in both languages, represents every entry in a standard Vietnamese syllable dictionary with a Fertility of exactly one, and generally produces shorter Vietnamese sequences than existing pretrained tokenizers\. We further instantiate the tokenizer inPhonemicBERT, which combines factorized component embeddings and reconstructs complete masked syllables using three prediction heads\. Under a controlled Chinese pretraining setup, PhonemicBERT\-Zh is competitive with or outperforms character, subword, and SubChar alternatives across diverse language\-understanding tasks\. PhonemicBERT\-Vi also achieves competitive or superior results to established Vietnamese and multilingual pretrained models\. These results establish phonemic factorization as a compact, efficient, and interpretable alternative to atomic and statistically segmented text representations\.

###### Keywords:

Tokenization , Natural Language Representation , Transformer , Vietnamese , Chinese

## 1Introduction

Tokenization determines the computational units through which pretrained language models \(PLMs\) access text\. Most PLMs rely on corpus\-induced algorithms such as Byte Pair Encoding \(BPE\)\[[16](https://arxiv.org/html/2609.21362#bib.bib3)\], WordPiece\[[15](https://arxiv.org/html/2609.21362#bib.bib4)\], or the Unigram Language Model\[[9](https://arxiv.org/html/2609.21362#bib.bib5)\]\. Although effective, these methods do not guarantee that the resulting tokens correspond to meaningful linguistic units\. Instead, each token is represented by an atomic identifier whose internal structure and relationships with other tokens remain implicit\.

This limitation is particularly relevant to Chinese and Vietnamese\. Despite their different writing systems, both are tonal languages in which the syllable is a prominent unit and can be analyzed into an onset, a rime, and a lexical tone\. Chinese characters do not transparently encode pronunciation\[[3](https://arxiv.org/html/2609.21362#bib.bib6)\], but Pinyin provides an intermediate phonological notation\. Vietnamese uses an alphabetic orthography with explicit orthographic\-syllable boundaries and relatively systematic grapheme–phoneme correspondences\. Character tokenization preserves Chinese syllable boundaries but treats characters as unrelated atomic entries, whereas statistical subword methods may merge syllables or fragment a Vietnamese syllable into pieces without independent linguistic interpretations\. Neither paradigm explicitly represents the syllable’s internal organization\.

Sub\-character tokenization partially addresses this issue by converting Chinese characters into Pinyin or Wubi strings before applying statistical segmentation\[[17](https://arxiv.org/html/2609.21362#bib.bib1)\]\. Its final units, however, remain corpus\-dependent and need not preserve syllable structure\. More generally, atomic token identifiers offer little explanation of which linguistic properties are shared across input units\. Explicit linguistic representations do not make neural models fully interpretable, but they provide an analyzable account of how surface text is transformed into model inputs\.

We introducePhonemic Tokenizer, a computational representation framework that preserves one contextual position per syllable while factorizing it into three phonological components:onset, rime, and tone\. Each orthographic unit is first mapped to IPA and then deterministically converted into a phonological tuple\. Syllables sharing a component reuse its representation, rather than being assigned entirely independent token identifiers\. The framework requires neither corpus\-dependent vocabulary learning nor statistical segmentation and retains syllable\-level sequence length\.

We instantiate the framework for Mandarin Chinese and Vietnamese\. Chinese characters are assigned phrase\-aware Pinyin pronunciations before IPA conversion, while Vietnamese orthographic syllables are converted directly using deterministic rules\. The two instantiations share the same factorization scheme but use language\-specific phonological inventories\. Missing components are represented by\[EMPTY\], and unsupported inputs are handled through character\-level fallback\. Including fallback and special symbols, the resulting vocabularies contain only 112 entries for Chinese and 256 for Vietnamese\.

We evaluate the framework intrinsically through vocabulary size, fertility, Rényi efficiency, and tokenized sequence length\. We then developPhonemicBERTas an experimental instantiation, combining projected component embeddings with tuple\-level masked prediction\. For Chinese, we follow the corpus, model configurations, and evaluation protocol of[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1), enabling controlled comparisons with character, subword, and SubChar tokenization\. We also evaluate robustness to synthetic homophone substitutions\. For Vietnamese, we compare PhonemicBERT\-Vi with established base\-sized monolingual and multilingual PLMs across diverse downstream tasks\. The intrinsic results demonstrate compact and efficient vocabulary utilization, while the downstream experiments show that phonemic factorization remains competitive with and outperforms established alternatives on multiple tasks across both writing systems\.

Our contributions are threefold:

- 1\.We propose an IPA\-based computational representation that factorizes each syllable into onset, rime, and tone while preserving one contextual position and avoiding corpus\-dependent segmentation\.
- 2\.We instantiate and intrinsically evaluate the framework for Mandarin Chinese and Vietnamese, obtaining compact vocabularies across logographic and alphabetic writing systems\.
- 3\.We introduce PhonemicBERT to evaluate the representation in pretraining, providing controlled Chinese comparisons, broad Vietnamese evaluation, and an analysis of robustness to homophone substitutions\.

## 2Related Work

### 2\.1Statistical and Language\-Specific Tokenization

Most pretrained language models use corpus\-induced subword tokenizers such as BPE\[[16](https://arxiv.org/html/2609.21362#bib.bib3)\], WordPiece\[[15](https://arxiv.org/html/2609.21362#bib.bib4)\], or the Unigram Language Model\[[9](https://arxiv.org/html/2609.21362#bib.bib5)\]\. These methods balance character\- and word\-level vocabularies by extracting frequent patterns, but their units are not guaranteed to align with linguistic boundaries\[[7](https://arxiv.org/html/2609.21362#bib.bib9)\]\. For Chinese and Vietnamese, they may merge multiple syllables or fragment a Vietnamese syllable into pieces without independent linguistic interpretations\. Character tokenization avoids fragmentation in Chinese but represents phonologically related characters as unrelated atomic entries\.

Vietnamese PLMs illustrate these alternatives\. PhoBERT\[[11](https://arxiv.org/html/2609.21362#bib.bib8)\]applies word segmentation followed by BPE, whereas WikiBERT\[[12](https://arxiv.org/html/2609.21362#bib.bib17)\], mBERT\[[4](https://arxiv.org/html/2609.21362#bib.bib2)\], and XLM\-R\[[2](https://arxiv.org/html/2609.21362#bib.bib10)\]use monolingual or multilingual statistical vocabularies\. DistilBERT\[[14](https://arxiv.org/html/2609.21362#bib.bib18)\]retains its teacher’s tokenizer while reducing model size\. None explicitly represents the onset, rime, and tone of each Vietnamese syllable\. In contrast, Phonemic Tokenizer preserves one contextual position per syllable while exposing reusable phonological components\.

### 2\.2Linguistically Informed and Phonological Representations

Prior work incorporates sub\-character information into Chinese language models\. ChineseBERT\[[18](https://arxiv.org/html/2609.21362#bib.bib7)\]augments character embeddings with glyph and Pinyin features\. SubChar\[[17](https://arxiv.org/html/2609.21362#bib.bib1)\]converts characters into Wubi or Pinyin strings before applying statistical subword segmentation\. These approaches demonstrate the value of sub\-character information, but either retain characters as the primary units or derive their final boundaries through corpus\-dependent segmentation\. Related work on spelling correction and robust language modeling similarly uses Pinyin as an auxiliary signal rather than as the primary structured input representation\.

IPA\-based tokenization has also been explored across languages\.[Miletić et al\. \[10\]](https://arxiv.org/html/2609.21362#bib.bib13)convert orthographic corpora into IPA and train BPE or Unigram tokenizers over the resulting phoneme sequences, reducing script\-dependent disparities and increasing cross\-lingual symbol overlap\. Our objective differs: we apply no statistical segmentation after IPA conversion\. Instead, each syllable is deterministically represented as one onset–rime–tone tuple\. Chinese and Vietnamese use separate phonological inventories; IPA provides a consistent notation for applying the same computational factorization across their different writing systems\.

### 2\.3Intrinsic Tokenizer Evaluation

Downstream results jointly reflect tokenization, model architecture, pretraining data, and optimization\. Intrinsic metrics therefore provide complementary information about tokenizer behavior\. Fertility measures the number of tokens required to represent an input unit and quantifies fragmentation\[[13](https://arxiv.org/html/2609.21362#bib.bib19)\], while average sequence length reflects the number of contextual positions processed by the model\. Rényi efficiency measures how effectively the available vocabulary is utilized\[[22](https://arxiv.org/html/2609.21362#bib.bib20)\]\. However, it is not a reliable standalone predictor of downstream performance\[[1](https://arxiv.org/html/2609.21362#bib.bib14)\]\.

We consequently evaluate Phonemic Tokenizer using vocabulary size, fertility, Rényi efficiency, and average sequence length, and report these properties separately from the downstream performance of PhonemicBERT\.

## 3Phonemic Tokenization

We proposePhonemic Tokenizer, a computational representation framework that maps each phonologically analyzable syllable to a structured onset–rime–tone tuple\. Given an input sequence

C=\[c1,…,cn\],C=\[c\_\{1\},\\ldots,c\_\{n\}\],\(1\)the tokenizer produces

P=\[p1,…,pn\],pi=\(Oi,Ri,Ti\),P=\[p\_\{1\},\\ldots,p\_\{n\}\],\\qquad p\_\{i\}=\(O\_\{i\},R\_\{i\},T\_\{i\}\),\(2\)whereOiO\_\{i\},RiR\_\{i\}, andTiT\_\{i\}denote the onset, rime, and tone of the syllable corresponding tocic\_\{i\}\. The three components are aligned at one contextual position rather than emitted as three sequential tokens\. The framework therefore exposes syllable\-internal structure without tripling sequence length\.

### 3\.1Orthography\-to\-IPA Conversion

The tokenizer first converts each native input unit into a normalized IPA syllable:

ϕℓ:ci⟼si,ℓ∈\{Zh,Vi\},\\phi\_\{\\ell\}:c\_\{i\}\\longmapsto s\_\{i\},\\qquad\\ell\\in\\\{\\mathrm\{Zh\},\\mathrm\{Vi\}\\\},\(3\)whereℓ\\elldenotes the language\. IPA provides a consistent intermediate notation, but each language retains its own phonological inventory and conversion rules\.

Vietnamese orthographic syllables are converted directly using deterministic grapheme\-to\-phoneme correspondences\. Because pronunciation varies across regions, we adopt a composite standard based on established descriptions of Vietnamese phonology\[[20](https://arxiv.org/html/2609.21362#bib.bib16),[19](https://arxiv.org/html/2609.21362#bib.bib15)\]\. It provides a normalized computational representation rather than a narrow transcription of a single dialect\.

Chinese characters do not transparently encode pronunciation\. We therefore first assign an aligned Pinyin sequence

ΨZh:\[c1,…,cn\]⟼\[q1,…,qn\],\\Psi\_\{\\mathrm\{Zh\}\}:\[c\_\{1\},\\ldots,c\_\{n\}\]\\longmapsto\[q\_\{1\},\\ldots,q\_\{n\}\],\(4\)using phrase\-level pronunciation lookup to resolve covered polyphonic characters\. Phrase entries are matched longest\-first; unmatched characters receive their default isolated pronunciation\. Each selected Pinyin syllable is then deterministically converted into IPA:

ρZh:qi⟼si\.\\rho\_\{\\mathrm\{Zh\}\}:q\_\{i\}\\longmapsto s\_\{i\}\.\(5\)Pinyin thus serves only as a pronunciation\-bearing bridge between Chinese orthography and IPA\. Polyphonic readings covered by the phrase lexicon are resolved before factorization, whereas distinct homophonous characters may still map to the same phonemic representation\. We analyze this trade\-off in Section[7\.2\.4](https://arxiv.org/html/2609.21362#S7.SS2.SSS4)\.

### 3\.2Syllable Factorization

Each IPA syllable is factorized using a language\-specific function

γℓ:si⟼\(Oi,Ri,Ti\)\.\\gamma\_\{\\ell\}:s\_\{i\}\\longmapsto\(O\_\{i\},R\_\{i\},T\_\{i\}\)\.\(6\)The onset contains the consonantal material preceding the rime\. Internally, the tokenizer identifies the rime as

Ri=Gi​‖Vi‖​Fi,R\_\{i\}=G\_\{i\}\\\|V\_\{i\}\\\|F\_\{i\},\(7\)whereGiG\_\{i\},ViV\_\{i\}, andFiF\_\{i\}denote an optional glide, the vowel nucleus, and an optional final\. These elements are used only during parsing; their combination is retained as one atomic rime component\.

The tone is represented independently\. Syllables differing only in tone therefore share their onset and rime entries, while syllables sharing an onset or rime reuse the corresponding component\. This explicit sharing distinguishes phonemic factorization from assigning an unrelated identifier to every character or complete syllable\.

If an onset or an overt tone is absent under the language\-specific convention, its slot is assigned\[EMPTY\]\. This symbol is distinct from\[UNK\], which denotes unsupported input, and\[PAD\], which is used for sequence padding\. A valid native syllable must contain a recognized rime\.

### 3\.3Coverage and Complexity

Units outside the native syllable inventory, including foreign strings, URLs, punctuation, and malformed text, are handled through character\-level fallback\. A fallback characterxxis represented as

p⁡\(x\)=\(x,x,x\),p\(x\)=\(x,x,x\),\(8\)preserving the three\-slot interface\. Characters absent from both the native and fallback inventories are mapped to\[UNK\]\. Consequently, native syllables occupy one contextual position, while an unsupported multi\-character unit may expand into several fallback positions\.

The conversion and factorization rules are fixed independently of the pretraining corpus\. With dictionary lookup and bounded phrase matching, the tokenizer runs in linear time with respect to the input length\. Including fallback and special symbols, the Chinese and Vietnamese instantiations contain 112 and 256 vocabulary entries, respectively\. Complete algorithms, pronunciation resources, and IPA mapping tables are provided in Appendices[A](https://arxiv.org/html/2609.21362#A1)and[B](https://arxiv.org/html/2609.21362#A2)\.

## 4PhonemicBERT

To evaluate the proposed representation in language model pretraining, we developPhonemicBERT, a BERT encoder adapted to process onset–rime–tone tuples\. The Transformer architecture remains unchanged; only the input representation and masked prediction objective are modified\. We independently pretrainPhonemicBERT\-ZhandPhonemicBERT\-Viusing their language\-specific tokenizers and vocabularies\.

### 4\.1Factorized Input Representation

Given a sequence

P=\[p1,…,pn\],pi=\(Oi,Ri,Ti\),P=\[p\_\{1\},\\ldots,p\_\{n\}\],\\qquad p\_\{i\}=\(O\_\{i\},R\_\{i\},T\_\{i\}\),\(9\)all three components ofpip\_\{i\}remain aligned at one contextual position\. Let𝒱ℓ\\mathcal\{V\}\_\{\\ell\}be the complete vocabulary for languageℓ\\ell, including phonological components, fallback characters, and special symbols\. PhonemicBERT uses a shared embedding table

Eℓ:𝒱ℓ→ℝd,E\_\{\\ell\}:\\mathcal\{V\}\_\{\\ell\}\\rightarrow\\mathbb\{R\}^\{d\},\(10\)whereddis the Transformer hidden size\. The three component embeddings are concatenated and projected from3​d3dtodd:

e~i=Wproj​\[Eℓ​\(Oi\)⊕Eℓ​\(Ri\)⊕Eℓ​\(Ti\)\]\+bproj,\\tilde\{e\}\_\{i\}=W\_\{\\mathrm\{proj\}\}\\left\[E\_\{\\ell\}\(O\_\{i\}\)\\oplus E\_\{\\ell\}\(R\_\{i\}\)\\oplus E\_\{\\ell\}\(T\_\{i\}\)\\right\]\+b\_\{\\mathrm\{proj\}\},\(11\)whereWproj∈ℝd×3​dW\_\{\\mathrm\{proj\}\}\\in\\mathbb\{R\}^\{d\\times 3d\}\. Standard positional and token\-type embeddings are then added before the sequence is passed to the BERT encoder\. Thus, a sequence ofnnsyllables containsnn, rather than3​n3n, Transformer positions\.

The shared table enables syllables with common components to reuse the corresponding embeddings, while the projection learns interactions among onset, rime, and tone\. Fallback and special units follow the same interface: a fallback characteraais represented as\(a,a,a\)\(a,a,a\), while absent and unknown components use\[EMPTY\]and\[UNK\], respectively\.

### 4\.2Holistic Masked Language Modeling

Masking only one component would reveal the remaining phonological information at the same position\. We therefore introduceHolistic Masked Language Modeling\(H\-MLM\), which masks and reconstructs the complete tuple\.

We sample a setℳ\\mathcal\{M\}containing 15% of the sequence positions\. For everyi∈ℳi\\in\\mathcal\{M\}, all components are replaced simultaneously:

\(Oi,Ri,Ti\)⟼\(\[MASK\],\[MASK\],\[MASK\]\)\.\(O\_\{i\},R\_\{i\},T\_\{i\}\)\\longmapsto\\left\(\\texttt\{\[MASK\]\},\\texttt\{\[MASK\]\},\\texttt\{\[MASK\]\}\\right\)\.\(12\)H\-MLM does not use the standard 80/10/10 replacement strategy: every selected tuple is fully masked, forcing reconstruction from surrounding context\.

Lethi∈ℝdh\_\{i\}\\in\\mathbb\{R\}^\{d\}be the final contextual representation at a masked position\. Three independent heads predict the original components:

y^iO\\displaystyle\\hat\{y\}\_\{i\}^\{O\}=Softmax⁡\(WO​hi\+bO\),\\displaystyle=\\operatorname\{Softmax\}\(W\_\{O\}h\_\{i\}\+b\_\{O\}\),\(13\)y^iR\\displaystyle\\hat\{y\}\_\{i\}^\{R\}=Softmax⁡\(WR​hi\+bR\),\\displaystyle=\\operatorname\{Softmax\}\(W\_\{R\}h\_\{i\}\+b\_\{R\}\),y^iT\\displaystyle\\hat\{y\}\_\{i\}^\{T\}=Softmax⁡\(WT​hi\+bT\)\.\\displaystyle=\\operatorname\{Softmax\}\(W\_\{T\}h\_\{i\}\+b\_\{T\}\)\.Each head predicts over the complete vocabulary𝒱ℓ\\mathcal\{V\}\_\{\\ell\}, allowing the same objective to handle native syllables and fallback characters\. The training loss is the unweighted sum of the three cross\-entropy losses:

ℒH​\-​MLM=ℒO\+ℒR\+ℒT\.\\mathcal\{L\}\_\{\\mathrm\{H\\text\{\-\}MLM\}\}=\\mathcal\{L\}\_\{O\}\+\\mathcal\{L\}\_\{R\}\+\\mathcal\{L\}\_\{T\}\.\(14\)H\-MLM therefore preserves one prediction position per syllable while requiring the shared contextual representation to reconstruct its complete phonological structure\.

## 5Experiments

We evaluate Phonemic Tokenizer intrinsically and use PhonemicBERT to assess whether its representations support language model pretraining\. Chinese provides a controlled comparison with alternative tokenization pipelines, whereas Vietnamese evaluates practical competitiveness against existing monolingual and multilingual PLMs\.

### 5\.1Pretraining Data and Configurations

PhonemicBERT\-Zh uses the 2\.3GB Baidu Baike corpus released by[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\. After separating punctuation, we retain lines in which at least 70% of the remaining characters are Chinese, yielding 2\.1GB of pretraining text\. PhonemicBERT\-Vi uses the 20GB corpus employed by PhoBERT\[[11](https://arxiv.org/html/2609.21362#bib.bib8)\]\. A Vietnamese unit is considered native when it can be factorized into a valid onset–rime–tone tuple\. Lines containing at least 70% native units after punctuation separation are retained, resulting in 18\.54GB\. Filtering is applied at the line level; unsupported material within retained lines is handled through character fallback\.

ConfigurationZhViSource size2\.3 GB20 GBRetained size2\.1 GB18\.54 GBTransformer layers6 / 1212Hidden size768768Attention heads1212FFN size3,0723,072Vocabulary size112256Maximum length512512Batch size256256Learning rate5×10−45\\times 10^\{\-4\}5×10−45\\times 10^\{\-4\}Training steps1M1MTable 1:Pretraining data and model configurations\. Both models use AdamW, 1% warmup, and 15% tuple masking\.For Chinese, we follow the model capacities, training schedule, and evaluation protocol of[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1), using six\- and twelve\-layer encoders\. PhonemicBERT\-Vi uses a twelve\-layer BERT\-base encoder\. Complete preprocessing and optimization details are provided in Table[1](https://arxiv.org/html/2609.21362#S5.T1)\.

### 5\.2Intrinsic Tokenizer Evaluation

For Chinese, we compare Phonemic Tokenizer with Character, Subword, SubChar\-Wubi, and SubChar\-Pinyin tokenizers from[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\. For Vietnamese, we compare it with the released tokenizers of PhoBERT\[[11](https://arxiv.org/html/2609.21362#bib.bib8)\], WikiBERT\[[12](https://arxiv.org/html/2609.21362#bib.bib17)\], mBERT\[[4](https://arxiv.org/html/2609.21362#bib.bib2)\], and XLM\-R\[[2](https://arxiv.org/html/2609.21362#bib.bib10)\]\. Model\-specific preprocessing, including PhoBERT word segmentation, is retained, while model control symbols are excluded from all measurements\.

We report vocabulary size, Rényi efficiency, and average tokenized sequence length for both languages\. For Vietnamese, we additionally report fertility:

Fertility⁡\(τ\)=∑x∈𝒟Npos​\(τ⁡\(x\)\)∑x∈𝒟Nunit​\(x\),\\operatorname\{Fertility\}\(\\tau\)=\\frac\{\\sum\_\{x\\in\\mathcal\{D\}\}N\_\{\\mathrm\{pos\}\}\(\\tau\(x\)\)\}\{\\sum\_\{x\\in\\mathcal\{D\}\}N\_\{\\mathrm\{unit\}\}\(x\)\},\(15\)whereNunit​\(x\)N\_\{\\mathrm\{unit\}\}\(x\)counts whitespace\-delimited input units andNposN\_\{\\mathrm\{pos\}\}counts output contextual positions\. One phonemic tuple counts as one position, while fallback may expand a non\-native unit into multiple character positions\.

Following[Zouhar et al\. \[22\]](https://arxiv.org/html/2609.21362#bib.bib20), Rényi efficiency is computed as

REα⁡\(τ\)=11−α​log​∑v∈𝒱τp​\(v\)αlog⁡\|𝒱τ\|,α=2\.5,\\operatorname\{RE\}\_\{\\alpha\}\(\\tau\)=\\frac\{\\frac\{1\}\{1\-\\alpha\}\\log\\sum\_\{v\\in\\mathcal\{V\}\_\{\\tau\}\}p\(v\)^\{\\alpha\}\}\{\\log\|\\mathcal\{V\}\_\{\\tau\}\|\},\\qquad\\alpha=2\.5,\(16\)wherep⁡\(v\)p\(v\)is the empirical unigram probability of vocabulary entryvv\. For Phonemic Tokenizer, component occurrences in all three aligned slots are included in this distribution, although their tuple contributes only one contextual position to sequence length\. We treat Rényi efficiency as a descriptive intrinsic property rather than a standalone predictor of downstream performance\[[1](https://arxiv.org/html/2609.21362#bib.bib14)\]\.

### 5\.3Pretrained Baselines

For Chinese, we compare PhonemicBERT\-Zh with the CharTokenizer, Subword, SubChar\-Wubi, and SubChar\-Pinyin models reported by[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\. These represent atomic character input, character\-level BPE, glyph\-based SubChar tokenization, and Pinyin\-based SubChar tokenization, respectively\. The comparison matches the source corpus, encoder capacities, training schedule, and downstream protocol as closely as possible\. Because PhonemicBERT also introduces factorized projection and H\-MLM, the results compare complete tokenization\-and\-pretraining pipelines rather than changing only a scalar tokenizer\.

For Vietnamese, we compare PhonemicBERT\-Vi with WikiBERT, PhoBERT, mBERT, DistilBERT\[[14](https://arxiv.org/html/2609.21362#bib.bib18)\], and XLM\-R\-base\. These models differ in pretraining data and objectives; the comparison therefore evaluates practical PLM performance rather than providing a controlled tokenizer ablation\. Baseline results are taken from VLUE\[[6](https://arxiv.org/html/2609.21362#bib.bib11)\], except PhoNER and VietMed results, which follow[Dinh et al\. \[5\]](https://arxiv.org/html/2609.21362#bib.bib12)\. We exclude CafeBERT and HuTieuBERT because they differ substantially in model scale, continued\-pretraining history, or encoder architecture\.

### 5\.4Downstream Benchmarks

The Chinese evaluation covers eleven benchmarks: TNEWS, IFLYTEK, and THUCNews for text classification; AFQMC, BQ, and CSL for sentence\-pair understanding; OCNLI and CLUEWSC for inference and coreference; C3 and CMRC for reading comprehension; and CLUENER for named\-entity recognition\. Following baseline availability, the twelve\-layer comparison covers the nine benchmarks reported by[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1), while CMRC and CLUENER are evaluated only with the six\-layer configuration\.

The Vietnamese evaluation covers twelve datasets: VSMEC, ViHSD, ViHOS, UIT\-ViSFD, ViOCD, ViCTSD, and UIT\-VSFC for social\-media classification and span detection; NIIVTB POS for part\-of\-speech tagging; PhoNER and VietMed for general\- and medical\-domain named\-entity recognition; ViNLI for natural language inference; and ViNewsQA for extractive question answering\.

### 5\.5Fine\-Tuning and Reporting

For Chinese, we follow[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1): the official development set is used as the fixed test set, while each run samples 10% of the official training set as a new development set\. We report accuracy for sequence\-level tasks, exact match for CMRC, and entity\-level F1 for CLUENER\.

For Vietnamese, we use the official splits and fine\-tuning setup of VLUE\[[6](https://arxiv.org/html/2609.21362#bib.bib11)\]\. Classification, NLI, and NIIVTB POS use macro\-F1; ViHOS uses token\-level macro\-F1; PhoNER and VietMed use span\-level F1; and ViNewsQA uses extractive QA F1\.

We fine\-tune PhonemicBERT with four random seeds for every task and report the mean and standard deviation\. For Chinese, each seed also produces a new random development split\. Published baseline are reported by their source studies and may not use the same number of runs\.

## 6Intrinsic Evaluation of the Phonemic Tokenizer

### 6\.1Vietnamese Tokenization Results

We compare the Vietnamese Phonemic Tokenizer with the released tokenizers of PhoBERT, WikiBERT, mBERT, and XLM\-RoBERTa\. The evaluation considers vocabulary compactness, syllable\-normalized Fertility, Rényi efficiency, and average tokenized sequence length\. We subsequently analyze the occurrence of units that cannot be phonologically factorized as standard Vietnamese syllables\.

#### 6\.1\.1Vocabulary compactness

TokenizerVocabularyPretraining corpusPhoBERT64,00020 GBWikiBERT30,000\-mBERT119,537\-XLM\-RoBERTa250,0022\.5 TBPhonemic25618\.54 GBTable 2:Vocabulary sizes and pretraining\-corpus scales of the Vietnamese tokenizers\. Corpus sizes describe the complete pretraining resources of the corresponding models and are not directly comparable across monolingual and multilingual settings\.Table[2](https://arxiv.org/html/2609.21362#S6.T2)compares the vocabulary sizes and pretraining\-corpus scales of the evaluated tokenizers\. The Phonemic Tokenizer uses only 256 vocabulary entries, including phonological components, fallback characters, and special symbols\. This vocabulary is approximately 117×\\timessmaller than that of WikiBERT, 250×\\timessmaller than that of PhoBERT, 467×\\timessmaller than that of mBERT, and 977×\\timessmaller than that of XLM\-RoBERTa\.

The compact vocabulary results from representing a syllable compositionally through shared onset, rime, and tone entries rather than assigning an atomic entry to every syllable or statistically induced subword\. Corpus scale is reported only to contextualize the pretrained tokenizers and should not be interpreted as a controlled comparison: multilingual corpora contain different proportions of Vietnamese data, and their total sizes are not directly comparable with the Vietnamese\-only corpus used by PhonemicBERT\-Vi\.

#### 6\.1\.2Syllable\-level Fertility

\#DatasetPhoBERTWikiBERTmBERTXLM\-RPhonemic1VSMEC1\.08821\.33561\.27711\.18481\.05622ViHOS2\.45611\.29731\.44051\.38801\.01163ViHSD2\.43741\.32541\.43941\.39421\.02804NIIVTB POS1\.02001\.34261\.16941\.16741\.04225PhoNER1\.03411\.36111\.15581\.18341\.09746UIT\-ViSFD1\.24281\.50391\.39911\.29481\.16437ViOCD2\.72611\.33201\.45021\.37471\.01398ViCTSD1\.21221\.47331\.34421\.23121\.13889UIT\-VSFC1\.02081\.35711\.08421\.13811\.006210VietMed1\.01421\.30821\.11141\.04131\.000711ViNewsQA1\.19301\.50581\.31091\.20041\.136312ViNLI1\.22861\.53141\.30601\.22921\.167813Standard Dictionary1\.06601\.40191\.48871\.37981\.0000

Table 3:Syllable\-normalized Fertility on Vietnamese downstream datasets and the Standard Dictionary\. Lower values indicate fewer contextual positions per input syllabic unit\. For the Phonemic Tokenizer, an onset–rime–tone tuple counts as one contextual position\. Values below one for statistical tokenizers can result from merging multiple syllabic units into a single token\.Table[3](https://arxiv.org/html/2609.21362#S6.T3)reports the average number of contextual positions produced per whitespace\-delimited input unit\. The Phonemic Tokenizer obtains the lowest Fertility on 10 of the 12 downstream datasets\. The only exceptions are NIIVTB POS and PhoNER, on which PhoBERT achieves slightly lower values: 1\.0200 versus 1\.0422 on NIIVTB POS and 1\.0341 versus 1\.0974 on PhoNER\. Across the remaining datasets, the Phonemic Tokenizer maintains Fertility between 1\.0007 and 1\.1678, indicating that most input units are represented using one contextual position\.

The advantage is particularly pronounced on ViHOS, ViHSD, and ViOCD\. PhoBERT produces 2\.4561, 2\.4374, and 2\.7261 positions per input unit on these datasets, respectively, whereas the Phonemic Tokenizer produces only 1\.0116, 1\.0280, and 1\.0139\. WikiBERT, mBERT, and XLM\-RoBERTa also exhibit greater fragmentation on most datasets, with Fertility commonly ranging from approximately 1\.2 to 1\.5\. These results suggest that the statistical vocabularies of the pretrained tokenizers are more sensitive to variation across domains and writing styles, whereas deterministic syllable factorization provides more stable positional behavior\.

The Fertility of the Phonemic Tokenizer nevertheless exceeds one on natural text because not every whitespace\-delimited unit is a well\-formed Vietnamese syllable\. Social\-media datasets such as VSMEC, ViHOS, ViHSD, UIT\-ViSFD, ViOCD, ViCTSD, and UIT\-VSFC contain teen\-code spellings, non\-standard abbreviations, expressive character sequences, and emoticons\. Such forms cannot always be factorized into valid onset–rime–tone tuples and are therefore represented through character\-level fallback\. The degree of expansion varies across these datasets because it depends on both the frequency and character length of the unsupported units\. For example, UIT\-ViSFD and ViCTSD yield Fertility values of 1\.1643 and 1\.1388, whereas ViHOS, ViHSD, and ViOCD remain much closer to one\.

Elevated Fertility is also observed on PhoNER, ViNewsQA, and ViNLI, with values of 1\.0974, 1\.1363, and 1\.1678, respectively\. These corpora contain online\-news or Wikipedia\-derived text and consequently include foreign names, transliterated entities, technical terminology, and other non\-Vietnamese expressions\. Character\-level fallback preserves these units without forcing them into the Vietnamese phonological inventory, but may expand one input unit into multiple contextual positions\.

#### 6\.1\.3Vocabulary utilization

\#DatasetPhoBERTWikiBERTmBERTXLM\-RPhonemic1VSMEC0\.47400\.48760\.43540\.38630\.56612ViHOS0\.27020\.49230\.17010\.35350\.57773ViHSD0\.27100\.49450\.17200\.35720\.58194NIIVTB POS0\.44050\.47060\.41500\.34120\.53005PhoNER0\.38310\.43150\.37340\.30460\.46926UIT\-ViSFD0\.48950\.44810\.40060\.39070\.49967ViOCD0\.25960\.45700\.14710\.33010\.52578ViCTSD0\.50200\.47430\.42820\.41400\.54109UIT\-VSFC0\.37010\.42250\.35720\.28850\.464310VietMed0\.43310\.46720\.42110\.38900\.544011ViNewsQA0\.45290\.46990\.42350\.39850\.540312ViNLI0\.45900\.47720\.41580\.39810\.531913Standard Dictionary0\.64200\.48740\.46140\.44610\.8232

Table 4:Rényi efficiency of Vietnamese tokenizers\. Higher values indicate more balanced utilization of the corresponding vocabulary\.Table[4](https://arxiv.org/html/2609.21362#S6.T4)reports Rényi efficiency\. The Phonemic Tokenizer achieves the highest value on every evaluated dataset and on the Standard Dictionary\. Its advantage is particularly pronounced on ViHOS, ViHSD, ViOCD, and the dictionary, where the distributions induced by several pretrained tokenizers are comparatively concentrated\.

These results indicate that the compact phonemic inventory is used more evenly than the substantially larger statistical vocabularies\. The dictionary result of 0\.8232 further shows that the component inventory provides broad and balanced coverage of well\-formed Vietnamese syllables\. Nevertheless, Rényi efficiency measures vocabulary\-distribution properties rather than downstream quality and should be interpreted jointly with Fertility, sequence length, and the PhonemicBERT results\.

#### 6\.1\.4Average tokenized sequence length

\#DatasetInputPhoBERTWikiBERTmBERTXLM\-RPhonemic1VSMEC14\.0115\.24 \(\+01\.23\)18\.71 \(\+04\.70\)17\.89 \(\+03\.88\)16\.59 \(\+02\.58\)14\.79 \(\+00\.78\)2ViHOS14\.3235\.16 \(\+20\.84\)18\.57 \(\+04\.25\)20\.62 \(\+06\.30\)19\.87 \(\+05\.55\)14\.48 \(\+00\.16\)3ViHSD13\.1131\.96 \(\+18\.85\)17\.38 \(\+04\.27\)18\.87 \(\+05\.76\)18\.28 \(\+05\.17\)13\.48 \(\+00\.37\)4NIIVTB POS28\.8629\.43 \(\+00\.57\)38\.74 \(\+09\.88\)33\.74 \(\+04\.88\)33\.68 \(\+04\.82\)30\.07 \(\+01\.21\)5PhoNER34\.6335\.81 \(\+01\.18\)47\.13 \(\+12\.50\)40\.02 \(\+05\.39\)40\.98 \(\+06\.35\)38\.00 \(\+03\.37\)6UIT\-ViSFD36\.1944\.97 \(\+08\.78\)54\.42 \(\+18\.23\)50\.63 \(\+14\.44\)46\.85 \(\+10\.66\)42\.13 \(\+05\.94\)7ViOCD31\.7786\.61 \(\+54\.84\)42\.32 \(\+10\.55\)46\.07 \(\+14\.30\)43\.67 \(\+11\.90\)32\.21 \(\+00\.44\)8ViCTSD29\.3835\.62 \(\+06\.24\)43\.29 \(\+13\.91\)39\.50 \(\+10\.12\)36\.18 \(\+06\.80\)33\.46 \(\+04\.08\)9UIT\-VSFC14\.2314\.52 \(\+00\.29\)19\.31 \(\+05\.08\)15\.43 \(\+01\.20\)16\.19 \(\+01\.96\)14\.32 \(\+00\.09\)10VietMed23\.3723\.71 \(\+00\.34\)30\.58 \(\+07\.21\)25\.98 \(\+02\.61\)24\.34 \(\+00\.97\)23\.39 \(\+00\.02\)11ViNewsQA39\.1246\.67 \(\+07\.55\)58\.90 \(\+19\.78\)51\.28 \(\+12\.16\)46\.96 \(\+07\.84\)44\.45 \(\+05\.33\)12ViNLI25\.0530\.77 \(\+05\.72\)38\.36 \(\+13\.31\)32\.71 \(\+07\.66\)30\.79 \(\+05\.74\)29\.25 \(\+04\.20\)

Table 5:Average input and tokenized sequence lengths on Vietnamese downstream datasets\. Parenthesized values indicate the absolute change relative to the average input length\. A positive value denotes sequence expansion\. Bold indicates the shortest tokenized sequence for each dataset\. Lengths count contextual positions and exclude model\-specific special tokens\.Table[5](https://arxiv.org/html/2609.21362#S6.T5)reports the average number of contextual positions per example\. The results closely follow the Fertility patterns\. The Phonemic Tokenizer produces the shortest average sequences on 10 of the 12 datasets, matching the datasets on which it obtains the lowest Fertility\. PhoBERT produces slightly shorter sequences only on NIIVTB POS and PhoNER\.

Larger increases occur on datasets containing more non\-standard or non\-Vietnamese material\. The Phonemic Tokenizer expands UIT\-ViSFD by 5\.94 positions, ViNewsQA by 5\.33, ViNLI by 4\.20, and PhoNER by 3\.37 on average\. These increases are consistent with the use of character\-level fallback for foreign terminology, named entities, URLs, and malformed or non\-standard forms\. Nevertheless, the resulting sequences remain shorter than those produced by the other evaluated tokenizers on UIT\-ViSFD, ViNewsQA, and ViNLI\.

#### 6\.1\.5Type\-level coverage of native Vietnamese units

DatasetUnique typesVietnameseCoverageVSMEC4,7503,57775%ViHOS7,3014,45561%ViHSD11,8525,62447%NIIVTB POS8,3164,69856%PhoNER2,7062,12278%UIT\-ViSFD6,4733,77558%ViOCD4,5983,06867%ViCTSD6,9354,28962%UIT\-VSFC2,4512,01982%VietMed2,7632,48090%ViNewsQA13,8204,74234%ViNLI11,2984,36639%

Table 6:Type\-level coverage of phonologically analyzable Vietnamese units\. Coverage is the proportion of unique observed types recognized as standard Vietnamese syllables\.To understand the effect of non\-Vietnamese material, we first examine the number of unique units in each dataset\. Table[6](https://arxiv.org/html/2609.21362#S6.T6)reports the total number of distinct observed types, the number that can be phonologically factorized as standard Vietnamese syllables, and their percentage\.

Type\-level coverage varies substantially across datasets\. VietMed exhibits the highest coverage at 90%, followed by UIT\-VSFC at 82% and PhoNER at 78%\. In contrast, only 34% and 39% of the distinct types in ViNewsQA and ViNLI, respectively, are recognized as standard Vietnamese\. These low type\-level percentages indicate a long tail of names, foreign expressions, numerals, symbols, malformed strings, and other dataset\-specific forms that cannot be analyzed through the native syllable inventory\.

#### 6\.1\.6Token\-level coverage of native Vietnamese units

DatasetOccurrencesVietnameseCoverageVSMEC102,46285,08083%ViHOS160,121124,23378%ViHSD450,182333,65874%NIIVTB POS617,413503,24282%PhoNER381,038264,01969%UIT\-ViSFD468,599350,34075%ViOCD176,651148,72084%ViCTSD334,603276,28083%UIT\-VSFC231,582200,51987%VietMed216,753214,81899%ViNewsQA2,754,2142,321,24084%ViNLI14,998,70711,500,83077%

Table 7:Token\-level coverage of phonologically analyzable Vietnamese units\. Unlike Table[6](https://arxiv.org/html/2609.21362#S6.T6), repeated occurrences are retained\.Type counts assign the same weight to frequent and rare forms\. We therefore also compute token\-level coverage, counting repeated occurrences\. Table[7](https://arxiv.org/html/2609.21362#S6.T7)shows that native Vietnamese syllables constitute the majority of observed occurrences in every dataset\. Coverage ranges from 69% on PhoNER to 99% on VietMed\.

The difference between type\- and token\-level coverage is especially large on ViNewsQA and ViNLI\. Although only 34% and 39% of their unique types are recognized as standard Vietnamese, these types account for 84% and 77% of all occurrences, respectively\. Thus, much of the unsupported vocabulary belongs to a diverse but relatively infrequent tail, whereas frequently occurring text remains predominantly analyzable\. Nevertheless, every unsupported occurrence may invoke character\-level fallback, explaining why the Fertility of the Phonemic Tokenizer can exceed one even when native Vietnamese accounts for most corpus tokens\.

#### 6\.1\.7Overall findings

The Vietnamese intrinsic evaluation reveals three complementary properties\. First, phonological factorization reduces the vocabulary from tens or hundreds of thousands of entries to 256 shared components\. Second, the resulting vocabulary is used more evenly, as reflected by consistently higher Rényi efficiency\. Third, the tokenizer preserves one contextual position for every analyzable Vietnamese syllable, while deviations from unit Fertility arise primarily from non\-native or malformed material handled through fallback\. The type\- and token\-level analyses further show that such material is diverse but often infrequent, allowing the tokenizer to maintain broad operational coverage without incorporating it into the native phonological inventory\.

### 6\.2Chinese Tokenization Results

We compare the Chinese Phonemic Tokenizer with Character, Subword, SubChar\-Wubi, and SubChar\-Pinyin tokenization\. Because the Chinese input and the corresponding phonemic representation are aligned at the character level, we do not separately report Fertility\. For the Phonemic and Character tokenizers, each input character deterministically contributes one contextual position, while the Fertility of Subword and SubChar tokenizers would primarily restate the sequence\-compression patterns already captured by average tokenized length\. We therefore focus on Rényi efficiency, sequence length, vocabulary size, and native\-character coverage\.

#### 6\.2\.1Rényi efficiency

\#DatasetCharacterSubwordSubChar\-WubiSubChar\-PinyinPhonemic1TNEWS0\.50020\.49200\.49270\.51580\.69932IFLYTEK0\.52420\.53890\.55070\.54970\.68613CLUEWSC0\.43760\.46880\.52060\.50630\.67454AFQMC0\.37410\.36670\.32560\.34820\.66075OCNLI0\.47720\.49150\.52570\.52020\.66516CSL0\.50640\.51000\.53590\.50960\.68747C30\.48670\.48250\.50260\.46390\.67138THUCNews0\.48730\.50780\.53300\.52640\.69619BQ0\.45290\.44880\.46680\.45150\.678910CMRC0\.49820\.52640\.54510\.55500\.709911CLUENER0\.49880\.51490\.52610\.53510\.6970

Table 8:Rényi efficiency of Chinese tokenizers\. Higher values indicate more balanced utilization of the corresponding vocabulary\. For the Phonemic Tokenizer, the distribution is computed over identifiers occurring in the three aligned component slots\.Table[8](https://arxiv.org/html/2609.21362#S6.T8)reports Rényi efficiency on eleven Chinese benchmarks\. The Phonemic Tokenizer achieves the highest efficiency on every dataset, with values ranging from 0\.6607 on AFQMC to 0\.7099 on CMRC\. By contrast, the strongest value among the four baselines ranges from 0\.3741 to 0\.5550\. The advantage is observed consistently across short\-text classification, long\-document classification, sentence\-pair understanding, reading comprehension, and sequence\-labeling datasets\.

The largest difference occurs on AFQMC, where the Phonemic Tokenizer obtains 0\.6607, compared with 0\.3741 for the strongest baseline\. Substantial differences are also observed on BQ \(0\.6789 versus 0\.4668\), TNEWS \(0\.6993 versus 0\.5158\), and C3 \(0\.6713 versus 0\.5026\)\. These results indicate that the compact onset–rime–tone inventory is used more evenly than the much larger statistical or character vocabularies\.

#### 6\.2\.2Average tokenized sequence length

\#DatasetInputCharacterSubwordSubChar\-WubiSubChar\-PinyinPhonemic1TNEWS22\.2422\.2420\.3215\.9616\.1822\.242IFLYTEK288\.17288\.17254\.35182\.46184\.49288\.173CLUEWSC66\.6466\.6461\.8152\.2751\.9366\.644AFQMC13\.3513\.3512\.5110\.9810\.5313\.355OCNLI17\.8217\.8215\.5812\.0312\.4117\.826CSL206\.08206\.08176\.40125\.39130\.61206\.087C36\.416\.416\.004\.454\.646\.418THUCNews851\.29851\.29741\.83541\.62552\.27851\.299BQ11\.8611\.8610\.858\.258\.5411\.8610CMRC262\.02262\.02230\.67170\.38171\.97262\.0211CLUENER37\.3937\.3932\.9324\.6525\.0237\.39

Table 9:Average input and tokenized sequence lengths on Chinese benchmarks\. Lengths count contextual positions and exclude model\-specific special tokens\. Bold indicates the shortest tokenized sequence\. The Phonemic and Character tokenizers preserve one position per input character\.Table[9](https://arxiv.org/html/2609.21362#S6.T9)compares the average number of contextual positions produced by each tokenizer\. The Phonemic Tokenizer and Character tokenizer preserve the original sequence length on every dataset because each Chinese character occupies exactly one contextual position\. For example, the average input length on TNEWS is 22\.24 characters, and both tokenizers produce 22\.24 positions\. The same one\-to\-one correspondence holds for long inputs such as THUCNews, where all three values are 851\.29\.

Subword and SubChar tokenizers produce shorter sequences by merging recurring character or sub\-character patterns into larger units\. SubChar\-Wubi produces the shortest sequences on nine datasets, while SubChar\-Pinyin is shortest on CLUEWSC and AFQMC\. Both substantially compress the character sequences through statistical merging\. On IFLYTEK, for example, SubChar\-Wubi reduces the average length from 288\.17 to 182\.46, whereas SubChar\-Pinyin produces 184\.49 positions\. On THUCNews, the corresponding reductions are from 851\.29 to 541\.62 and 552\.27 positions\.

These results expose a deliberate difference in design objectives\. SubChar tokenization uses part of its vocabulary capacity to store frequent multi\-character combinations and thereby improve sequence compression\. The Phonemic Tokenizer instead preserves character–syllable alignment while reducing vocabulary size through within\-syllable factorization\. It therefore does not shorten Chinese sequences, but it avoids splitting a character into multiple sequential units and maintains direct alignment for character\-level tasks such as NER and span extraction\.

#### 6\.2\.3Vocabulary compactness

TokenizerVocabulary sizeCharacter22,675Subword22,675SubChar\-Wubi22,675SubChar\-Pinyin22,675Phonemic112Table 10:Vocabulary sizes of the Chinese tokenizers under the controlled configuration\. The Phonemic count includes phonological, fallback, and special entries\.Table[10](https://arxiv.org/html/2609.21362#S6.T10)compares the vocabulary sizes of the evaluated tokenizers\. Character, Subword, SubChar\-Wubi, and SubChar\-Pinyin each use 22,675 entries under the controlled configuration of[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\. The Phonemic Tokenizer requires only 112 entries, including its phonological components, fallback characters, and special symbols\. Its vocabulary is therefore approximately 202 times smaller, corresponding to a reduction of approximately 99\.51% relative to the baseline vocabulary size\. This reduction is achieved without assigning an atomic vocabulary entry to every Chinese character\. Instead, all covered characters are represented compositionally through their onset, rime, and tone\. The vocabulary result and sequence\-length result capture different efficiency dimensions: the Phonemic Tokenizer substantially reduces the number of embedding entries, whereas SubChar primarily reduces the number of contextual positions\.

#### 6\.2\.4Overall findings

The Chinese intrinsic evaluation reveals a trade\-off between vocabulary compactness and sequence compression\. The Phonemic Tokenizer preserves the same sequence length as character tokenization while reducing the vocabulary from 22,675 to 112 entries and achieving substantially higher Rényi efficiency across all datasets\. SubChar tokenizers produce shorter sequences through cross\-character statistical merging, but retain a vocabulary more than 200 times larger\. The proposed tokenizer therefore improves representational compactness and vocabulary utilization without sacrificing the one\-to\-one alignment between Chinese characters and contextual positions\.

## 7Downstream Evaluation of PhonemicBERT

### 7\.1Vietnamese PLM Comparison

#### 7\.1\.1Token\- and span\-level linguistic analysis

ModelViHOSNIIVTB POSVietMedPhoNERWikiBERT77\.0575\.52––PhoBERT75\.6977\.6056\.04±4\.4456\.04\\pm 4\.4483\.83¯±0\.15\\underline\{83\.83\}\\pm 0\.15mBERT76\.2281\.3459\.81±0\.5759\.81\\pm 0\.5772\.82±0\.4372\.82\\pm 0\.43DistilBERT75\.7280\.05––XLM\-RoBERTa74\.6781\.7661\.76±0\.88\\mathbf\{61\.76\}\\pm 0\.8869\.75±0\.3869\.75\\pm 0\.38PhonemicBERT\-Vi83\.40±1\.28\\mathbf\{83\.40\}\\pm 1\.2882\.64±1\.02\\mathbf\{82\.64\}\\pm 1\.0260\.20¯±0\.84\\underline\{60\.20\}\\pm 0\.8484\.66±1\.10\\mathbf\{84\.66\}\\pm 1\.10

Table 11:Results on Vietnamese token\- and span\-level tasks\. ViHOS uses token\-level macro\-F1, NIIVTB POS uses macro\-F1, and VietMed and PhoNER use span\-level F1\. PhonemicBERT\-Vi results are means over four random seeds\. Bold and underlining indicate the highest and second\-highest reported means, respectively\. Results on ViHOS and NIIVTB POS are reported from\[[6](https://arxiv.org/html/2609.21362#bib.bib11)\], while results on the VietMed and PhoNER are from\[[5](https://arxiv.org/html/2609.21362#bib.bib12)\]\.Table[11](https://arxiv.org/html/2609.21362#S7.T11)reports results on tasks requiring predictions aligned with individual input positions or textual spans\. PhonemicBERT\-Vi achieves the highest reported mean on ViHOS, NIIVTB POS, and PhoNER, while ranking second on the medical\-domain VietMed dataset\.

The clearest improvement occurs on ViHOS, where PhonemicBERT\-Vi obtains83\.40±1\.2883\.40\\pm 1\.28, exceeding WikiBERT, the strongest reported baseline, by 6\.35 points\. On NIIVTB POS, it reaches82\.64±1\.0282\.64\\pm 1\.02, compared with 81\.76 for XLM\-RoBERTa\. PhonemicBERT\-Vi also obtains the highest reported PhoNER mean at84\.66±1\.1084\.66\\pm 1\.10, although its 0\.83\-point difference from PhoBERT is small relative to the variation across runs and should be interpreted as competitive rather than conclusive superiority\.

These results are consistent with the one\-position\-per\-syllable design of the Phonemic Tokenizer\. Token\- and span\-level annotations can be aligned directly with syllable representations without reconstructing labels from fragmented subwords\. Factorization also allows syllables sharing an onset, rime, or tone to reuse component embeddings\. Nevertheless, the experiments evaluate the complete PhonemicBERT pipeline and do not isolate the effects of syllable alignment, factorized projection, and H\-MLM\.

On VietMed, PhonemicBERT\-Vi obtains60\.20±0\.8460\.20\\pm 0\.84, ranking second behind XLM\-RoBERTa at61\.76±0\.8861\.76\\pm 0\.88but outperforming the reported PhoBERT and mBERT means\. The broader multilingual pretraining of XLM\-RoBERTa may be beneficial for medical terminology and foreign entity forms, many of which cannot be directly factorized as native Vietnamese syllables\.

#### 7\.1\.2Social\-media and feedback understanding

ModelVSMECViHSDViSFDViOCDViCTSDUIT\-VSFCCons\.ToxicTopicSent\.WikiBERT57\.64–71\.46–––––PhoBERT59\.91–––82\.11\\mathbf\{82\.11\}±1\.00\\pm 1\.0072\.5372\.53±1\.66\\pm 1\.6678\.78¯\\underline\{78\.78\}±0\.71\\pm 0\.7180\.95¯\\underline\{80\.95\}±1\.26\\pm 1\.26mBERT54\.5964\.2070\.2791\.6181\.62¯\\underline\{81\.62\}±0\.72\\pm 0\.7268\.5368\.53±3\.67\\pm 3\.6775\.8075\.80±0\.71\\pm 0\.7178\.3078\.30±1\.46\\pm 1\.46DistilBERT53\.8362\.5070\.9790\.50––––XLM\-RoBERTa61\.8963\.68––81\.62¯\\underline\{81\.62\}±0\.61\\pm 0\.6173\.29¯\\underline\{73\.29\}±2\.09\\pm 2\.0977\.6677\.66±1\.79\\pm 1\.7980\.5780\.57±1\.72\\pm 1\.72PhonemicBERT\-Vi60\.79¯\\underline\{60\.79\}±1\.54\\pm 1\.5464\.67\\mathbf\{64\.67\}±0\.31\\pm 0\.3175\.36\\mathbf\{75\.36\}±0\.33\\pm 0\.3392\.69\\mathbf\{92\.69\}±1\.11\\pm 1\.1181\.5981\.59±1\.04\\pm 1\.0479\.88\\mathbf\{79\.88\}±1\.10\\pm 1\.1078\.94\\mathbf\{78\.94\}±0\.25\\pm 0\.2582\.21\\mathbf\{82\.21\}±1\.22\\pm 1\.22

Table 12:Macro\-F1 results on Vietnamese social\-media and feedback benchmarks\. ViCTSD includes constructiveness and toxicity detection, while UIT\-VSFC includes topic and sentiment classification\. Results on VSMEC, ViHSD, ViSFD, and ViOCD are reported from\[[6](https://arxiv.org/html/2609.21362#bib.bib11)\], while results on the ViCTSD and UIT\-VSFC are from\[[5](https://arxiv.org/html/2609.21362#bib.bib12)\]\.Table[12](https://arxiv.org/html/2609.21362#S7.T12)presents results on social\-media classification, hate\-speech detection, complaint detection, toxicity, topic, and sentiment analysis\. PhonemicBERT\-Vi achieves the highest reported mean on six of the eight prediction targets and ranks second on VSMEC\.

PhonemicBERT\-Vi obtains the highest reported results on ViHSD \(64\.67±0\.3164\.67\\pm 0\.31\), UIT\-ViSFD \(75\.36±0\.3375\.36\\pm 0\.33\), and ViOCD \(92\.69±1\.1192\.69\\pm 1\.11\)\. Its largest improvement among these datasets occurs on UIT\-ViSFD, where it exceeds WikiBERT by 3\.90 points\. On ViOCD, it improves over mBERT by 1\.08 points, while its 0\.47\-point advantage over mBERT on ViHSD is comparatively small\.

The two multi\-target datasets reveal different patterns\. On ViCTSD, PhonemicBERT\-Vi obtains79\.88±1\.1079\.88\\pm 1\.10on toxicity detection, 6\.59 points above XLM\-RoBERTa, the strongest reported baseline\. On constructiveness, however, its score of81\.59±1\.0481\.59\\pm 1\.04is effectively comparable to PhoBERT, mBERT, and XLM\-RoBERTa, whose reported means range from 81\.62 to 82\.11\.

On UIT\-VSFC, PhonemicBERT\-Vi achieves the highest reported means for both topic classification \(78\.94±0\.2578\.94\\pm 0\.25\) and sentiment classification \(82\.21±1\.2282\.21\\pm 1\.22\)\. The differences from PhoBERT are only 0\.16 and 1\.26 points, respectively, so these results are best characterized as competitive rather than substantial improvements\. On VSMEC, PhonemicBERT\-Vi ranks second with60\.79±1\.5460\.79\\pm 1\.54, behind XLM\-RoBERTa at 61\.89\.

These results demonstrate that the complete phonemic pipeline remains effective on informal text despite the prevalence of teen code, emoticons, foreign expressions, and malformed forms\. Such inputs invoke the character\-level fallback mechanism rather than phonological factorization\. The results therefore show operational robustness in the presence of fallback, but do not establish that fallback itself is responsible for the performance gains\.

#### 7\.1\.3Inference and question answering

ModelViNewsQAViNLIWikiBERT82\.85–PhoBERT–78\.05mBERT83\.1973\.62DistilBERT–66\.77XLM\-RoBERTa–77\.01PhonemicBERT\-Vi82\.16±1\.3582\.16\\pm 1\.3573\.47±1\.2073\.47\\pm 1\.20Table 13:Results on Vietnamese inference and question\-answering benchmarks\. ViNewsQA is evaluated using extractive QA F1, while ViNLI uses macro\-F1\.Table[13](https://arxiv.org/html/2609.21362#S7.T13)reports results on ViNewsQA and ViNLI, which emphasize document\-level question answering and sentence\-pair semantic reasoning, respectively\. On ViNewsQA, PhonemicBERT\-Vi obtains82\.16±1\.3582\.16\\pm 1\.35, compared with 83\.19 for mBERT and 82\.85 for WikiBERT\. Its mean is 1\.03 points below the strongest reported result, indicating broadly competitive question\-answering performance\. A larger deficit appears on ViNLI\. PhonemicBERT\-Vi obtains73\.47±1\.2073\.47\\pm 1\.20, compared with 78\.05 for PhoBERT and 77\.01 for XLM\-RoBERTa\. Unlike POS tagging and NER, natural language inference depends less directly on local syllable alignment and more strongly on sentence\-level lexical and semantic distinctions\.

#### 7\.1\.4Model and vocabulary scale

Pretrained model\#Params\#Layers\#HeadsHiddenVocab\.LanguagePretraining sourcemBERT179M1212768119,547MultilingualWikipediaMultilingual DistilBERT134M612768119,547MultilingualWikipediaXLM\-RoBERTa\-base270M1212768250,002MultilingualCommonCrawlWikiBERT–121276820,101MonolingualWikipediaPhoBERT\-base135M121276864,001MonolingualWikipedia, newsPhonemicBERT\-Vi87\.94M1212768256MonolingualWikipedia, news

Table 14:Architectural and pretraining characteristics of the Vietnamese and multilingual pretrained models\. Parameter and vocabulary counts correspond to the evaluated checkpoints\. A dash indicates that the parameter count is not reported\.Table[14](https://arxiv.org/html/2609.21362#S7.T14)contextualizes the downstream results with respect to model capacity, vocabulary size, and pretraining source\. Phonemi cBERT\-Vi uses the same 12\-layer depth, 768\-dimensional hidden representations, and 12 attention heads as a standard base\-sized encoder\. However, its shared phonemic vocabulary contains only 256 entries, compared with 20,101–250,002 entries for the pretrained baselines\.

PhonemicBERT\-Vi consequently contains 87\.94M parameters, approximately 34\.9% fewer than PhoBERT\-base, 50\.9% fewer than mBERT, and 67\.4% fewer than XLM\-RoBERTa\-base\. Although the factorized input projection and three H\-MLM prediction heads introduce additional parameters, their cost remains substantially smaller than that of the large embedding and output matrices required by conventional token vocabularies\.

#### 7\.1\.5Overall findings

Across the three task groups, PhonemicBERT\-Vi achieves the highest reported mean on nine of fourteen prediction targets\. Its strongest results occur on token\- and span\-aligned tasks and on several social\-media benchmarks, while its performance is less competitive on natural language inference\. These results show that a model pretrained with the Phonemic Tokenizer can remain competitive with larger monolingual and multilingual PLMs, although its effectiveness varies across task types\.

### 7\.2Controlled Chinese Comparison

We compare PhonemicBERT\-Zh with character, subword, and SubChar models under the Chinese experimental protocol of[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\. Results are organized into text classification, sentence\-pair and reasoning, and reading comprehension and sequence\-labeling tasks\. We discuss the six\- and twelve\-layer configurations separately rather than relying on an aggregate score\.

#### 7\.2\.1Text classification

TokenizerTNEWSIFLYTEKTHUCNews6\-layer encoderCharTokenizer64\.19±0\.18\\mathbf\{64\.19\}\\pm 0\.1855\.83±0\.5055\.83\\pm 0\.5096\.95±0\.0496\.95\\pm 0\.04Subword64\.09±0\.2864\.09\\pm 0\.2854\.88±0\.3954\.88\\pm 0\.3997\.14±0\.03\\mathbf\{97\.14\}\\pm 0\.03SubChar\-Wubi63\.89±0\.2563\.89\\pm 0\.2558\.64¯±0\.27\\underline\{58\.64\}\\pm 0\.2797\.02±0\.0497\.02\\pm 0\.04SubChar\-Pinyin63\.68±0\.2563\.68\\pm 0\.2558\.81±0\.28\\mathbf\{58\.81\}\\pm 0\.2897\.04¯±0\.04\\underline\{97\.04\}\\pm 0\.04PhonemicBERT\-Zh64\.13¯±0\.21\\underline\{64\.13\}\\pm 0\.2158\.14±0\.3158\.14\\pm 0\.3197\.02±0\.0497\.02\\pm 0\.0412\-layer encoderCharTokenizer64\.39¯±0\.13\\underline\{64\.39\}\\pm 0\.1358\.52±0\.4658\.52\\pm 0\.4697\.02±0\.0397\.02\\pm 0\.03SubChar\-Pinyin64\.19±0\.1464\.19\\pm 0\.1459\.67¯±0\.23\\underline\{59\.67\}\\pm 0\.2397\.12¯±0\.03\\underline\{97\.12\}\\pm 0\.03PhonemicBERT\-Zh64\.40±0\.12\\mathbf\{64\.40\}\\pm 0\.1260\.47±0\.25\\mathbf\{60\.47\}\\pm 0\.2597\.16±0\.04\\mathbf\{97\.16\}\\pm 0\.04Table 15:Accuracy on Chinese text\-classification benchmarks, reported as mean±\\pmstandard deviation\. Bold and underlining indicate the highest and second\-highest means within each encoder configuration\.Table[15](https://arxiv.org/html/2609.21362#S7.T15)reports results on TNEWS, IFLYTEK, and THUCNews\. With six layers, PhonemicBERT\-Zh remains competitive on all three datasets\. It obtains64\.13±0\.2164\.13\\pm 0\.21on TNEWS, only 0\.06 points below CharTokenizer;58\.14±0\.3158\.14\\pm 0\.31on IFLYTEK, compared with58\.81±0\.2858\.81\\pm 0\.28for SubChar\-Pinyin; and97\.02±0\.0497\.02\\pm 0\.04on THUCNews, compared with97\.14±0\.0397\.14\\pm 0\.03for Subword\.

Scaling PhonemicBERT\-Zh to twelve layers improves its mean on all three datasets\. It obtains the highest reported means on TNEWS \(64\.40±0\.1264\.40\\pm 0\.12\), IFLYTEK \(60\.47±0\.2560\.47\\pm 0\.25\), and THUCNews \(97\.16±0\.0497\.16\\pm 0\.04\)\. The clearest difference occurs on IFLYTEK, where it exceeds SubChar\-Pinyin by 0\.80 points\. On TNEWS and THUCNews, the differences from the second\-highest means are only 0\.01 and 0\.04 points, respectively, and the runs exhibit overlapping variation\. These results should therefore be interpreted as competitive performance rather than substantial improvements\.

Overall, phonemic factorization preserves text\-classification performance despite reducing the vocabulary from 22,675 entries to 112\. The larger gain on IFLYTEK may indicate that the deeper encoder is better able to contextualize factorized syllable representations over its comparatively long input descriptions, although the current experiments do not isolate this effect\.

#### 7\.2\.2Sentence\-pair and reasoning tasks

TokenizerCLUEWSCAFQMCOCNLICSLBQ6\-layer encoderCharTokenizer63\.39±1\.9563\.39\\pm 1\.9568\.68±0\.4668\.68\\pm 0\.4668\.19±0\.3968\.19\\pm 0\.3982\.67±0\.4682\.67\\pm 0\.4681\.99¯±0\.47\\underline\{81\.99\}\\pm 0\.47Subword62\.67±2\.8762\.67\\pm 2\.8769\.25±0\.42\\mathbf\{69\.25\}\\pm 0\.4269\.03±0\.44\\mathbf\{69\.03\}\\pm 0\.4483\.20¯±0\.27\\underline\{83\.20\}\\pm 0\.2781\.94±0\.2881\.94\\pm 0\.28SubChar\-Wubi64\.61±2\.0964\.61\\pm 2\.0968\.75±0\.5968\.75\\pm 0\.5968\.93¯±0\.38\\underline\{68\.93\}\\pm 0\.3882\.81±0\.4682\.81\\pm 0\.4681\.70±0\.2981\.70\\pm 0\.29SubChar\-Pinyin65\.90±1\.45\\mathbf\{65\.90\}\\pm 1\.4568\.89±0\.4268\.89\\pm 0\.4267\.98±0\.4567\.98\\pm 0\.4582\.87±0\.4082\.87\\pm 0\.4081\.74±0\.2481\.74\\pm 0\.24PhonemicBERT\-Zh65\.17¯±1\.62\\underline\{65\.17\}\\pm 1\.6269\.25±0\.38\\mathbf\{69\.25\}\\pm 0\.3867\.73±0\.4067\.73\\pm 0\.4083\.22±0\.23\\mathbf\{83\.22\}\\pm 0\.2383\.08±0\.27\\mathbf\{83\.08\}\\pm 0\.2712\-layer encoderCharTokenizer68\.09±1\.5968\.09\\pm 1\.5969\.00±0\.3569\.00\\pm 0\.3570\.40¯±0\.34\\underline\{70\.40\}\\pm 0\.3482\.77¯±0\.33\\underline\{82\.77\}\\pm 0\.3383\.49¯±0\.38\\underline\{83\.49\}\\pm 0\.38SubChar\-Pinyin71\.71±2\.03\\mathbf\{71\.71\}\\pm 2\.0369\.30¯±0\.24\\underline\{69\.30\}\\pm 0\.2470\.43±0\.25\\mathbf\{70\.43\}\\pm 0\.2582\.23±0\.2782\.23\\pm 0\.2782\.28±0\.1682\.28\\pm 0\.16PhonemicBERT\-Zh68\.47¯±1\.75\\underline\{68\.47\}\\pm 1\.7569\.81±0\.28\\mathbf\{69\.81\}\\pm 0\.2869\.15±0\.2769\.15\\pm 0\.2783\.37±0\.35\\mathbf\{83\.37\}\\pm 0\.3584\.33±0\.24\\mathbf\{84\.33\}\\pm 0\.24

Table 16:Accuracy on Chinese sentence\-pair and reasoning benchmarks, reported as mean±\\pmstandard deviation\.Table[16](https://arxiv.org/html/2609.21362#S7.T16)presents results on CLUEWSC, AFQMC, OCNLI, CSL, and BQ\. In the six\-layer setting, PhonemicBERT\-Zh ties Subword for the highest AFQMC mean at 69\.25 and obtains the highest reported means on CSL \(83\.22±0\.2383\.22\\pm 0\.23\) and BQ \(83\.08±0\.2783\.08\\pm 0\.27\)\.

In the twelve\-layer setting, PhonemicBERT\-Zh achieves the highest reported means on AFQMC \(69\.81±0\.2869\.81\\pm 0\.28\), CSL \(83\.37±0\.3583\.37\\pm 0\.35\), and BQ \(84\.33±0\.2484\.33\\pm 0\.24\)\. Its BQ mean exceeds CharTokenizer by 0\.84 points, while its AFQMC and CSL means exceed the strongest baselines by 0\.51 and 0\.60 points, respectively\. Together with the six\-layer results, this pattern shows that the factorized representation remains effective for semantic equivalence and keyword–abstract consistency tasks\.

The results are weaker on CLUEWSC and OCNLI\. On twelve\-layer CLUEWSC, PhonemicBERT\-Zh obtains68\.47±1\.7568\.47\\pm 1\.75, compared with71\.71±2\.0371\.71\\pm 2\.03for SubChar\-Pinyin\. On OCNLI, it reaches69\.15±0\.2769\.15\\pm 0\.27, trailing SubChar\-Pinyin by 1\.28 points\. The standard deviations indicate considerable run\-to\-run variation on CLUEWSC, whereas the OCNLI difference is more consistent\.

Both tasks require fine\-grained contextual distinctions, and phonemic factorization does not uniformly preserve every orthographic distinction available to character\- or indexed Pinyin\-based representations\. In particular, different Chinese characters can map to the same phonemic tuple\. We investigate the relationship between such homophonic collisions and downstream behavior in Section[7\.2\.4](https://arxiv.org/html/2609.21362#S7.SS2.SSS4)\.

#### 7\.2\.3Reading comprehension and sequence labeling

TokenizerC3CMRC†CLUENER†6\-layer encoderCharTokenizer53\.17±0\.5653\.17\\pm 0\.5656\.5869\.61Subword53\.32±0\.4453\.32\\pm 0\.4455\.8567\.94SubChar\-Wubi54\.68±0\.77\\mathbf\{54\.68\}\\pm 0\.7754\.4570\.63SubChar\-Pinyin53\.03±0\.4753\.03\\pm 0\.4755\.1870\.77PhonemicBERT\-Zh53\.69¯±0\.56\\underline\{53\.69\}\\pm 0\.5655\.49±0\.1155\.49\\pm 0\.1170\.71¯±0\.03\\underline\{70\.71\}\\pm 0\.0312\-layer encoderCharTokenizer54\.22±0\.4054\.22\\pm 0\.40––SubChar\-Pinyin55\.92±0\.26\\mathbf\{55\.92\}\\pm 0\.26––PhonemicBERT\-Zh55\.33¯±0\.21\\underline\{55\.33\}\\pm 0\.21––Table 17:Results on Chinese reading\-comprehension and sequence\-labeling benchmarks\. C3 is evaluated using accuracy, CMRC using exact match \(EM\), and CLUENER using entity\-level F1\. All results are reported as mean±\\pmstandard deviation except the published CMRC and CLUENER baselines marked with†\\dagger, for which[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)report single\-run results\. Twelve\-layer CMRC and CLUENER baseline results are unavailable\.Table[17](https://arxiv.org/html/2609.21362#S7.T17)presents task\-standard scores on C3, CMRC, and CLUENER\. C3 is a multiple\-choice reading\-comprehension benchmark evaluated using accuracy, whereas CMRC is a span\-extraction benchmark evaluated using exact match\. CLUENER is evaluated using entity\-level F1\.

In the six\-layer setting, PhonemicBERT\-Zh obtains53\.69±0\.5653\.69\\pm 0\.56accuracy on C3, ranking second behind SubChar\-Wubi at54\.68±0\.7754\.68\\pm 0\.77\. Scaling to twelve layers improves its C3 accuracy to55\.33±0\.2155\.33\\pm 0\.21, leaving it 0\.59 points below SubChar\-Pinyin\.

On CMRC, PhonemicBERT\-Zh obtains55\.49±0\.1155\.49\\pm 0\.11EM, compared with the single\-run CharTokenizer result of 56\.58 and Subword result of 55\.85\. On CLUENER, it reaches70\.71±0\.0370\.71\\pm 0\.03entity\-level F1, closely matching the single\-run SubChar\-Pinyin score of 70\.77\.

#### 7\.2\.4Analysis on Homophone in Chinese

Figure 1:Illustration of synthetic homophone generation\[[17](https://arxiv.org/html/2609.21362#bib.bib1)\]\.A phonemic representation maps homophonous characters to the same pronunciation tuple\. This property may remove orthographic distinctions between characters with different meanings, but it should also make the tokenizer robust to homophone substitutions, a common category of Chinese input errors\. We therefore investigate how synthetic homophones affect PhonemicBERT\-Zh\.

We follow the evaluation procedure of SubChar\[[17](https://arxiv.org/html/2609.21362#bib.bib1)\]\. Given a clean input sequence, a specified proportion of Chinese characters is randomly selected and replaced with different characters having the same Pinyin pronunciation, including lexical tone\. The replacement character is therefore orthographically and potentially semantically different from the original character, while preserving its pronunciation\. For example, the characters "世" \(world\) and "事" \(affair\) in "世事" \(shìshì\) can be replaced by the homophones "适" \(suitable\) and "逝" \(pass away\), respectively, because all four characters are pronounced asshi④\(Figure[1](https://arxiv.org/html/2609.21362#S7.F1)\)\. We evaluate corruption ratios of7\.5%7\.5\\%,15%15\\%,22\.5%22\.5\\%,30%30\\%, and37\.5%37\.5\\%\.

This procedure directly tests whether the model’s input representation is stable under pronunciation\-preserving orthographic perturbations\. Character and conventional subword tokenizers assign different token identities to the original and substituted characters\. By contrast, Phonemic Tokenizer factorizes homophones into the same onset, rime, and tone components when their assigned pronunciations are identical\. We conduct the analysis on TNEWS, OCNLI, and C3 and report accuracy\. Results for the Character, Subword, SubChar\-Pinyin, and SubChar\-Pinyin\-NoIndex baselines are taken from[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\.

DatasetTokenizerClean7\.5%15\.0%22\.5%30\.0%37\.5%TNEWSCharacter64\.1063\.0958\.9650\.9138\.3325\.20Subword64\.0962\.8257\.7548\.6736\.3725\.27SubChar\-Pinyin63\.6861\.9556\.6745\.2230\.7127\.53SubChar\-Pinyin\-NoIndex63\.2863\.2863\.2863\.2863\.2863\.28PhonemicBERT\-Zh64\.0763\.7163\.1763\.1063\.0963\.39OCNLICharacter68\.1964\.8956\.5847\.6540\.4836\.36Subword69\.0364\.3356\.4948\.0742\.6838\.28SubChar\-Pinyin67\.7061\.9354\.3946\.0140\.2437\.33SubChar\-Pinyin\-NoIndex67\.9167\.9167\.9167\.9167\.9167\.91PhonemicBERT\-Zh67\.3266\.8866\.6167\.2266\.9267\.12C3Character53\.1351\.4649\.2247\.7146\.7843\.95Subword53\.5551\.6649\.4947\.8146\.2443\.58SubChar\-Pinyin52\.8750\.4547\.4644\.5042\.4240\.07SubChar\-Pinyin\-NoIndex53\.6553\.6553\.6553\.6553\.6553\.65PhonemicBERT\-Zh53\.7052\.1552\.0452\.1851\.8151\.81

Table 18:Accuracy under synthetic homophone substitutions\. Corruption percentages indicate the proportion of Chinese characters replaced with different characters having the same Pinyin pronunciation and tone\. Baseline results are taken from[Si et al\. \[17\]](https://arxiv.org/html/2609.21362#bib.bib1)\. Bold value indicate the best result within the pronunciation\-based representations for each setting\.Table[18](https://arxiv.org/html/2609.21362#S7.T18)shows a sharp contrast between orthography\-dependent and pronunciation\-based representations\. Character, Subword, and SubChar\-Pinyin models deteriorate progressively as the corruption ratio increases\. At37\.5%37\.5\\%corruption, the Character model drops by 38\.90 points on TNEWS, 31\.83 points on OCNLI, and 9\.18 points on C3\. The corresponding decreases for the Subword model are 38\.82, 30\.75, and 9\.97 points\. Applying BPE after Pinyin conversion does not eliminate this sensitivity: SubChar\-Pinyin loses 36\.15, 30\.37, and 12\.80 points on the three datasets, respectively\. Statistical segmentation can change when characters are substituted even if their pronunciations are preserved\.

PhonemicBERT\-Zh is substantially more stable\. From the clean condition to37\.5%37\.5\\%corruption, its accuracy changes from 64\.07 to 63\.39 on TNEWS, from 67\.32 to 67\.12 on OCNLI, and from 53\.70 to 51\.81 on C3, corresponding to decreases of only 0\.68, 0\.20, and 1\.89 points\. Thus, even at the highest corruption level, the model retains nearly all of its clean\-input performance on TNEWS and OCNLI and most of its performance on C3\. These results confirm that phonological factorization provides robustness to orthographic substitutions that preserve pronunciation\.

SubChar\-Pinyin\-NoIndex is exactly invariant across corruption levels because homophonous characters are deliberately mapped to the same unindexed romanized representation\. PhonemicBERT\-Zh exhibits a similar, although not perfect, invariance\. This difference is attributable to the phrase\-aware Chinese pronunciation pipeline\. Phonemic Tokenizer applies longest\-match\-first lookup to assign contextual pronunciations; replacing a character can break or create a phrase match, thereby changing the pronunciation assigned to that position or a neighboring position even when the substituted character is a homophone in isolation\. This effect explains the small fluctuations observed across corruption ratios\. It is most visible on C3, where longer and more context\-dependent inputs provide more opportunities for phrase matches to change\.

Overall, these results reveal both the strength and the limitation of the Phonemic Tokenizer\. By mapping pronunciation\-preserving substitutions to identical or highly similar phonemic tuples, it substantially improves robustness to homophone\-based noise\. However, the same factorization removes the orthographic identity that distinguishes homophonous characters with different meanings\. Such characters become indistinguishable at the input level, requiring PhonemicBERT\-Zh to recover their semantic distinctions entirely from the surrounding context\. Phonemic Tokenizer therefore trades character\-level lexical specificity for greater representational sharing and robustness to orthographic variation\.

## 8Conclusion

We introduced thePhonemic Tokenizer, a linguistically motivated alternative to character and statistical subword tokenization for Vietnamese and Chinese\. Instead of treating a syllable as an atomic symbol or fragmenting it into statistically derived pieces, the proposed tokenizer factorizes each syllable into three phonological components—onset, rime, and tone—while preserving one contextual position per syllable\. Its deterministic IPA\-based formulation requires no corpus\-dependent vocabulary learning and represents the phonological systems of Chinese and Vietnamese with compact vocabularies of 112 and 256 entries, respectively\.

Our intrinsic evaluation shows that this factorization achieves consistently high Rényi efficiency in both languages\. In Vietnamese, it represents every entry in the standard syllable dictionary with a Fertility of exactly one and generally produces shorter sequences than the evaluated pretrained tokenizers\. In Chinese, it preserves character\-level sequence length while using a substantially smaller vocabulary than character, subword, and SubChar alternatives\. These findings demonstrate that compactness need not be obtained by increasing sequence length or fragmenting syllables into multiple contextual positions\.

To evaluate whether the proposed representation remains effective for language modeling, we instantiated it inPhonemicBERT, which combines the three component embeddings through a learned projection and reconstructs complete masked syllables using three prediction heads\. PhonemicBERT\-Zh remains competitive with character, subword, and SubChar models under a controlled Chinese pretraining setup, while PhonemicBERT\-Vi outperforms established monolingual and multilingual pretrained models on several Vietnamese downstream tasks despite its much smaller vocabulary and parameter count\. The synthetic homophone experiments further demonstrate that collapsing pronunciation\-equivalent characters into shared phonemic representations yields stable downstream performance under homophone substitutions, although it also removes orthographic distinctions among semantically different homophones\.

## References

- \[1\]\(2024\)Two counterexamples to tokenization and the noiseless channel\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 16897–16906\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1469/)Cited by:[§2\.3](https://arxiv.org/html/2609.21362#S2.SS3.p1.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p3.2)\.
- \[2\]A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov\(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p1.1)\.
- \[3\]F\. Coulmas\(1991\)Writing systems of the world\.The Language Library,Wiley\.External Links:ISBN 9780631180289,LCCN 88025130,[Link](https://books.google.com.vn/books?id=VOywmavmZ3UC)Cited by:[§1](https://arxiv.org/html/2609.21362#S1.p2.1)\.
- \[4\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p1.1)\.
- \[5\]A\. T\. D\. Dinh, K\. H\. N\. Vo, T\. T\. Ta, V\. C\. Doan, and T\. Quan\(2026\)When morphology hides in plain sight: breaking the isolation in Vietnamese and beyond\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 10377–10392\.External Links:[Link](https://aclanthology.org/2026.acl-long.472/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.472),ISBN 979\-8\-89176\-390\-6Cited by:[§5\.3](https://arxiv.org/html/2609.21362#S5.SS3.p2.1),[Table 11](https://arxiv.org/html/2609.21362#S7.T11),[Table 11](https://arxiv.org/html/2609.21362#S7.T11.5),[Table 12](https://arxiv.org/html/2609.21362#S7.T12),[Table 12](https://arxiv.org/html/2609.21362#S7.T12.5)\.
- \[6\]P\. N\. Do, S\. Q\. Tran, P\. G\. Hoang, K\. V\. Nguyen, and N\. L\. Nguyen\(2024\)VLUE: a new benchmark and multi\-task knowledge transfer learning for vietnamese natural language understanding\.InFindings of the Association for Computational Linguistics: NAACL 2024,Mexico City, Mexico,pp\. 211–222\.External Links:[Link](https://aclanthology.org/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.15)Cited by:[§5\.3](https://arxiv.org/html/2609.21362#S5.SS3.p2.1),[§5\.5](https://arxiv.org/html/2609.21362#S5.SS5.p2.1),[Table 11](https://arxiv.org/html/2609.21362#S7.T11),[Table 11](https://arxiv.org/html/2609.21362#S7.T11.5),[Table 12](https://arxiv.org/html/2609.21362#S7.T12),[Table 12](https://arxiv.org/html/2609.21362#S7.T12.5)\.
- \[7\]X\. Gutierrez\-Vasques, C\. Bentz, and T\. Samardžić\(2023\)Languages through the looking glass of BPE compression\.Computational Linguistics49\(4\),pp\. 943–1001\.External Links:[Link](https://aclanthology.org/2023.cl-4.5/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00489)Cited by:[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p1.1)\.
- \[8\]C\. X\. Hạo\(1998\)Tiếng việt mấy vấn đề ngữ âm \- ngữ pháp \- ngữ nghĩa\.Nhà xuất bản Giáo dục Việt Nam\.Cited by:[§A\.1](https://arxiv.org/html/2609.21362#A1.SS1.p1.1)\.
- \[9\]T\. Kudo\(2018\)Subword regularization: improving neural network translation models with multiple subword candidates\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 66–75\.External Links:[Link](https://aclanthology.org/P18-1007/),[Document](https://dx.doi.org/10.18653/v1/P18-1007)Cited by:[§1](https://arxiv.org/html/2609.21362#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p1.1)\.
- \[10\]M\. Miletić, J\. Kallini, and E\. Shutova\(2026\)Phonemes to the rescue: multilingual tokenization based on international phonetic alphabet\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 40323–40349\.External Links:[Link](https://aclanthology.org/2026.acl-long.1872/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1872),ISBN 979\-8\-89176\-390\-6Cited by:[§2\.2](https://arxiv.org/html/2609.21362#S2.SS2.p2.1)\.
- \[11\]D\. Q\. Nguyen and A\. Tuan Nguyen\(2020\)PhoBERT: pre\-trained language models for Vietnamese\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1037–1042\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.92/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.92)Cited by:[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p2.1),[§5\.1](https://arxiv.org/html/2609.21362#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p1.1)\.
- \[12\]S\. Pyysalo, J\. Kanerva, A\. Virtanen, and F\. Ginter\(2021\)WikiBERT models: deep transfer learning for many languages\.InProceedings of the 23rd Nordic Conference on Computational Linguistics \(NoDaLiDa\),S\. Dobnik and L\. Øvrelid \(Eds\.\),Reykjavik, Iceland \(Online\),pp\. 1–10\.External Links:[Link](https://aclanthology.org/2021.nodalida-main.1/)Cited by:[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p1.1)\.
- \[13\]P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. Gurevych\(2021\)How good is your tokenizer? on the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3118–3135\.External Links:[Link](https://aclanthology.org/2021.acl-long.243/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243)Cited by:[§2\.3](https://arxiv.org/html/2609.21362#S2.SS3.p1.1)\.
- \[14\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p2.1),[§5\.3](https://arxiv.org/html/2609.21362#S5.SS3.p2.1)\.
- \[15\]M\. Schuster and K\. Nakajima\(2012\)Japanese and korean voice search\.In2012 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 5149–5152\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP.2012.6289079)Cited by:[§1](https://arxiv.org/html/2609.21362#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p1.1)\.
- \[16\]R\. Sennrich, B\. Haddow, and A\. Birch\(2016\)Neural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1715–1725\.External Links:[Link](https://aclanthology.org/P16-1162/),[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[§1](https://arxiv.org/html/2609.21362#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.21362#S2.SS1.p1.1)\.
- \[17\]C\. Si, Z\. Zhang, Y\. Chen, F\. Qi, X\. Wang, Z\. Liu, Y\. Wang, Q\. Liu, and M\. Sun\(2023\)Sub\-character tokenization for Chinese pretrained language models\.Transactions of the Association for Computational Linguistics11,pp\. 469–487\.External Links:[Link](https://aclanthology.org/2023.tacl-1.28/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00560)Cited by:[§1](https://arxiv.org/html/2609.21362#S1.p3.1),[§1](https://arxiv.org/html/2609.21362#S1.p6.1),[§2\.2](https://arxiv.org/html/2609.21362#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.21362#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2609.21362#S5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p1.1),[§5\.3](https://arxiv.org/html/2609.21362#S5.SS3.p1.1),[§5\.4](https://arxiv.org/html/2609.21362#S5.SS4.p1.1),[§5\.5](https://arxiv.org/html/2609.21362#S5.SS5.p1.1),[§6\.2\.3](https://arxiv.org/html/2609.21362#S6.SS2.SSS3.p1.1),[Figure 1](https://arxiv.org/html/2609.21362#S7.F1),[Figure 1](https://arxiv.org/html/2609.21362#S7.F1.4),[§7\.2\.4](https://arxiv.org/html/2609.21362#S7.SS2.SSS4.p2.1),[§7\.2\.4](https://arxiv.org/html/2609.21362#S7.SS2.SSS4.p3.1),[§7\.2](https://arxiv.org/html/2609.21362#S7.SS2.p1.1),[Table 17](https://arxiv.org/html/2609.21362#S7.T17),[Table 18](https://arxiv.org/html/2609.21362#S7.T18),[Table 18](https://arxiv.org/html/2609.21362#S7.T18.5)\.
- \[18\]Z\. Sun, X\. Li, X\. Sun, Y\. Meng, X\. Ao, Q\. He, F\. Wu, and J\. Li\(2021\)ChineseBERT: chinese pretraining enhanced by glyph and pinyin information\.arXiv preprint arXiv:2106\.16038\.Cited by:[§2\.2](https://arxiv.org/html/2609.21362#S2.SS2.p1.1)\.
- \[19\]Đ\. T\. Thuật\(2016\)Ngữ âm tiếng việt\.Nhà xuất bản Đại học Quốc gia Hà Nội\.Cited by:[§A\.1](https://arxiv.org/html/2609.21362#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.21362#S3.SS1.p2.1)\.
- \[20\]H\. C\. Xuan\(1985\)Phonologie et linearite\. reflexions critiques sur les postulats de la phonologie contemporaine\.Societe d’Etudes Linguistiques et Anthropologiques de France \(SELAF\)\.Cited by:[§A\.1](https://arxiv.org/html/2609.21362#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.21362#S3.SS1.p2.1)\.
- \[21\]H\. C\. Xuan\(1998\)The problem of phoneme in vietnamese\.Vietnamese studies\.Cited by:[§A\.1](https://arxiv.org/html/2609.21362#A1.SS1.p1.1)\.
- \[22\]V\. Zouhar, C\. Meister, J\. Gastaldi, L\. Du, M\. Sachan, and R\. Cotterell\(2023\)Tokenization and the noiseless channel\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5184–5207\.External Links:[Link](https://aclanthology.org/2023.acl-long.284/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.284)Cited by:[§2\.3](https://arxiv.org/html/2609.21362#S2.SS3.p1.1),[§5\.2](https://arxiv.org/html/2609.21362#S5.SS2.p3.1)\.

## Appendix AVietnamese Phonology and Orthography

This appendix summarizes the syllable structure of Vietnamese and the systematic relationship between its modern orthography and phonological representation\. These linguistic properties provide the basis for the deterministic factorization performed by Phonemic Tokenizer\.

### A\.1Syllable Structure

Vietnamese belongs to the Vietic branch of the Austroasiatic language family\. Modern Vietnamese is written inChữ Quốc Ngữ, a Latin based orthography developed to represent Vietnamese pronunciation\[[20](https://arxiv.org/html/2609.21362#bib.bib16),[21](https://arxiv.org/html/2609.21362#bib.bib21),[8](https://arxiv.org/html/2609.21362#bib.bib22),[19](https://arxiv.org/html/2609.21362#bib.bib15)\]\. In this writing system, whitespace generally separates orthographic syllables rather than necessarily delimiting complete lexical words\. A multisyllabic Vietnamese word may therefore consist of multiple whitespace\-separated orthographic units\.

Phonologically, a Vietnamese syllable follows the canonical structure

S=\(O,R,T\),S=\(O,R,T\),\(17\)whereOO,RR, andTTdenote the onset, rime, and lexical tone, respectively\. The onset is optional, whereas the rime constitutes the obligatory core of the syllable\. The rime can be further analyzed as

R=\(G,V,F\),R=\(G,V,F\),\(18\)whereGGis an optional medial glide,VVis the vocalic nucleus, andFFis an optional coda\.

For example, under our composite pronunciation standard, the orthographic syllablehoàngis transcribed as

hoàng⟼\\longmapsto/huıaŋ/\.

Its IPA representation can be analyzed as

O = /h/,

G = /uı/,

V = /a/,

F = /ŋ/,

T = //

The medial, nucleus, and coda are then combined into the atomic rime

R = /waŋ/

giving the final phonological factorization

hoàng⟼\\longmapsto\(/h/, /waŋ/, //\)\.

Vietnamese distinguishes six lexical tones:ngang//,huyền//,sắc//,hỏi//,ngã/P/, andnặng/P/\. They form an integral part of the syllable and can distinguish lexical meaning\.

### A\.2Correspondence between Orthography and Phonology

Chữ Quốc Ngữexhibits a high degree of correspondence between graphemes and phonological categories\. Given a well\-formed Vietnamese syllable, its pronunciation can generally be determined from its spelling using a finite set of context\-sensitive rules\. This correspondence remains stable across grammatical contexts because Vietnamese words do not undergo the extensive inflectional changes found in many morphologically rich languages\.

Grapheme\(s\)PhonemeExampleExample IPAb/b/ba/ba/t/t/ta/ta/th/th/tha/tha/c,k,q/k/ca,kim,qua/ka/, /kim/, /kwa/ph/f/pha/fa/đ/d/đa/da/g,gh/7/ga,ghe/7a/, /7E/gi/z/gia/za/d/j/da/ja/x/s/xa/sa/s/ s/sa/ sa/ch/c͡C/cha/c͡Ca/tr/t͡ s/tra/t͡ sa/ng,ngh/ŋ/nga,nghe/ŋa/, /ŋE/nh//nha/a/l/l/la/la/r/r/ra/ra/kh/x/kha/xa/v/v/va/va/m/m/ma/ma/n/n/na/na/h/h/ha/ha/

Table 19:Orthographic onset\-to\-phoneme correspondences under the composite Vietnamese pronunciation standard\. Tone contours in the examples describe phonetic realizationsThe correspondence is not strictly one\-to\-one at the level of individual letters \(Table[19](https://arxiv.org/html/2609.21362#A1.T19),[20](https://arxiv.org/html/2609.21362#A1.T20),[21](https://arxiv.org/html/2609.21362#A1.T21), and[22](https://arxiv.org/html/2609.21362#A1.T22)\)\. A phoneme may be represented by different graphemes depending on the neighboring vowel or its position within the syllable\. Nevertheless, these alternations are systematic rather than arbitrary\. Representative cases include:

- 1\.The same velar onset /k/ is written asc,k, or as part ofqu\. The spellingkis typically used before front vowels,quintroduces a following medial glide, andcoccurs in the remaining environments\.
- 2\.The spelling pairsg/ghandng/nghrepresent the same respective onset phonemes /7/ and /ŋ/\. The formsghandnghoccur before particular front\-vowel spellings, whereasgandngare used elsewhere\.
- 3\.A medial glide /uı/ may be written as eitherooru, with its orthographic realization determined by the surrounding onset and nucleus\.
- 4\.The diphthong /ie/ represented phonologically as a single nucleus may appear asiê,yê,ia, orya\. The selected spelling depends on whether the syllable contains a medial or a following coda\.
- 5\.The diphthong families /uo/ \(uô\-ua\) and /W9/ \(ươ\-ưa\) exhibit similar alternations\. Forms containingôorơoccur before a coda, whereasuaorưaoccurs in open syllables\.
- 6\.The vowel and semivowel spellingsi\-y/iı/ ando\-u/uı/ are conditioned by their syllabic positions and neighboring phonological components\.

For example, the initial velar stop /k/ has different surface spellings inca/ka/,kim/kim/, andqua/kuıa/, but these forms can be normalized to the same onset category\. Similarly, the nuclei inchia/c͡Cie/,kiên/kien/,yên/ien/, andkhuya/xuıie/ exhibit different spellings whose selection is predictable from the surrounding syllable structure\.

ComponentGrapheme\(s\)PhonemeExamplesMedialo,u/uı/hoa,thuê,khuyênMonophthonga/a/ba,anMonophthongă;abeforey, u/ă/ăn,tay,cauMonophthongâ/9̆/ân,câyMonophthongi,y/i/tin,yMonophthongê/e/êm,kếtMonophthonge/E/em,senMonophthongu/u/thu,ungMonophthongư/W/tư,ưngMonophthongo/O/ong,conMonophthongoo/O:/xoongMonophthongô/o/ông,tốtMonophthongơ/9/ơ,sơnDiphthongiê,yê,ia,ya/ie/biên,yên,bia,khuyaDiphthonguô,ua/uo/buôn,muaDiphthongươ,ưa/W9/hương,mưa

Table 20:Orthographic correspondences for the Vietnamese medial glide and vocalic nuclei\. Multiple spellings in the same row are normalized to the same phonemic category\.The relationship between orthography and pronunciation is therefore better characterized ashighly regular and context\-conditionedthan as perfectly one\-to\-one\. Several orthographic forms may map to the same phonological component, but their mappings can be resolved using the internal structure of the syllable\. Consequently, Phonemic Tokenizer can normalize surface spelling variants into canonical onset, rime, and tone components without learning a statistical segmentation model from a pretraining corpus\.

This regularity also explains why phonological factorization does not require Vietnamese graphemes to be emitted as separate sequential tokens\. Once an orthographic syllable has been analyzed, its internal phonological structure can be represented as a single aligned tuple:

ci⟼\(Oi,Ri,Ti\),c\_\{i\}\\longmapsto\(O\_\{i\},R\_\{i\},T\_\{i\}\),\(19\)thereby preserving one contextual position per orthographic syllable while making its internal structure explicit\.

bia⟼/bie/,\\displaystyle\\longmapsto/\{bie\{\}\\hbox\{\\kern 0\.27771pt\{\\raise 0\.0pt\\hbox\{\{\\char 128\\relax\}\}\}\\kern 0\.0pt\{\\char 191\\relax\}\}\}/,biên⟼/bien/,\\displaystyle\\longmapsto/\{bien\{\}\\hbox\{\\kern 0\.27771pt\{\\raise 0\.0pt\\hbox\{\{\\char 128\\relax\}\}\}\\kern 0\.0pt\{\\char 191\\relax\}\}\}/,\(20\)khuya⟼/xuıie/,\\displaystyle\\longmapsto/\{x\{\\vtop\{\\hbox\{u\}\\kern 0\.86108pt\\hbox\{\\hbox to5\.55557pt\{\\hss\\vbox to4\.30554pt\{\\hbox\{\\hbox\{\\char 16\\relax\}\}\\vss\}\\hss\}\}\}\}ie\{\}\\hbox\{\\kern 0\.27771pt\{\\raise 0\.0pt\\hbox\{\{\\char 128\\relax\}\}\}\\kern 0\.0pt\{\\char 191\\relax\}\}\}/,khuyên⟼/xuıien/\.\\displaystyle\\longmapsto/\{x\{\\vtop\{\\hbox\{u\}\\kern 0\.86108pt\\hbox\{\\hbox to5\.55557pt\{\\hss\\vbox to4\.30554pt\{\\hbox\{\\hbox\{\\char 16\\relax\}\}\\vss\}\\hss\}\}\}\}ien\{\}\\hbox\{\\kern 0\.27771pt\{\\raise 0\.0pt\\hbox\{\{\\char 128\\relax\}\}\}\\kern 0\.0pt\{\\char 191\\relax\}\}\}/\.\(21\)
Grapheme\(s\)PhonemeExamplesi,y/iı/mai,tayo,u/uı/sao,đaum/m/nam,êmn/n/an,ênng/ŋ/ang,ôngnh//anh,ênhp/p/áp,hợpt/t/át,kếtc/k/ác,ứcch/c/ách,ích

Table 21:Orthographic coda\-to\-phoneme correspondences used in the Vietnamese pronunciation normalization\.ToneMarkExampleNgangUnmarkedmaHuyềnGravemàSắcAcutemáHỏiHook abovemảNgãTildemãNặngDot belowmạTable 22:Mapping from Vietnamese orthographic tone marks to the tone representations used by Phonemic Tokenizer\.

## Appendix BMandarin Chinese Phonology and Pronunciation Representation

This appendix describes the syllable structure of Mandarin Chinese and the relationship among Chinese characters, Pinyin, and IPA\. Unlike VietnameseChữ Quốc Ngữ, Chinese orthography does not transparently encode pronunciation\. Phonemic Tokenizer therefore uses Pinyin as an intermediate pronunciation\-bearing representation before converting each syllable into its IPA\-based onset–rime–tone factorization\.

### B\.1Mandarin Syllable Structure

Modern Standard Mandarin is a tonal and predominantly syllable\-based language\. In most cases, one Chinese character corresponds to one spoken syllable, although the same character may have different pronunciations in different lexical contexts\. Each Mandarin syllable is analyzed using the canonical structure

S=\(O,R,T\),S=\(O,R,T\),\(22\)whereOO,RR, andTTdenote the onset, rime, and lexical tone, respectively\. The onset is optional, whereas the rime forms the obligatory core of the syllable\. Internally, a Mandarin rime may be analyzed as

R=\(G,V,F\),R=\(G,V,F\),\(23\)whereGGdenotes an optional medial glide,VVdenotes the vocalic nucleus, andFFdenotes an optional coda\. As in the Vietnamese instantiation, these internal elements are used only to determine the rime\. Phonemic Tokenizer stores their combination as one atomic rime component\.

For example, the character光\(light\) is represented in Pinyin asguāngand transcribed as

光⟼\\longmapstoguāng⟼\\longmapsto/kuıaŋ/\.

Its pronunciation can be analyzed as

O = /k/,

G = /uı/,

V = /a/,

F = /ŋ/,

T = //\.

The glide, nucleus, and coda are combined into the atomic rime

R = /uıaŋ/,

giving the final factorization

光⟼\\longmapsto\(/k/, /uıaŋ/, //\)\.

The same three\-slot structure also represents syllables without an overt onset\. For example, the character安\(peaceful\), written asānin Pinyin and transcribed as /an/, is factorized as

安⟼\\longmapsto\(\[EMPTY\], /an/, //\)\.

Here,\[EMPTY\]represents the absence of an overt consonantal onset and is distinct from\[UNK\]and\[PAD\]\.

Mandarin distinguishes four principal lexical tones, conventionally represented in Pinyin using tone marks or the numerical labels 1–4\. The corresponding normalized pitch contours are high\-level, rising, dipping, and falling\. A neutral\-tone category additionally occurs in unstressed syllables\. The tone representation used by the tokenizer is assigned after phrase\-level pronunciation retrieval, so the same character may receive different tone components in different lexical expressions\.

### B\.2Relationship among Chinese Characters, Pinyin, and IPA

Chinese characters primarily encode lexical and morphemic identity rather than directly specifying pronunciation\. Consequently, the mapping between characters and spoken syllables is neither one\-to\-one nor recoverable from character shape alone\. Two forms of ambiguity are particularly relevant\.

First, many orthographically distinct characters are homophones\. For example,意\(intention\),义\(meaning\),异\(strange\), and议\(debate\) can all be pronounced asyì\. Once converted into IPA, they share the same phonemic factorization:

\{意,义,异,议\}⟼\\longmapstoyì⟼\\longmapsto/i/\.

This many\-to\-one mapping enables representation sharing and robustness to homophone substitutions, but it also removes the orthographic distinction among characters with different meanings\.

Second, a single character may be polyphonic\. For example,行is pronounced asxíngin行为\(behavior\) but ashángin银行\(bank\):

行为⟼xíngwéi,\\displaystyle\\longmapsto\\textit\{xíngwéi\},\(24\)银行⟼yínháng\.\\displaystyle\\longmapsto\\textit\{yínháng\}\.\(25\)Pronunciation assignment must therefore precede IPA conversion and cannot always be performed independently for each character\.

Pinyin bridges this gap by explicitly encoding the pronunciation of a Chinese syllable through an initial, a final, and a tone mark\. Once the appropriate Pinyin reading has been selected, its conversion into IPA is highly systematic\. We consequently decompose Chinese conversion into two functions:

ψZh:C⟼Q,\\psi\_\{\\mathrm\{Zh\}\}:C\\longmapsto Q,\(26\)whereψZh\\psi\_\{\\mathrm\{Zh\}\}assigns a context\-sensitive Pinyin sequenceQ=\[q1,…,qn\]Q=\[q\_\{1\},\\ldots,q\_\{n\}\]to a character sequence, and

ρZh:qi⟼si,\\rho\_\{\\mathrm\{Zh\}\}:q\_\{i\}\\longmapsto s\_\{i\},\(27\)whereρZh\\rho\_\{\\mathrm\{Zh\}\}deterministically converts a selected Pinyin syllable into its normalized IPA representation\. The complete conversion is therefore

ϕZh=ρZh∘ψZh\.\\phi\_\{\\mathrm\{Zh\}\}=\\rho\_\{\\mathrm\{Zh\}\}\\circ\\psi\_\{\\mathrm\{Zh\}\}\.\(28\)

### B\.3Systematic Pinyin\-to\-IPA Conversion

Although the relationship between Chinese characters and pronunciation is opaque, the relationship between Pinyin and IPA is highly regular\. Pinyin letters should not, however, always be interpreted as independent Latin phonemes\. Their pronunciation can depend on their position and neighboring letters within the syllable\.

For example, the Pinyin formsgāandgēare transcribed under our normalization as

gā⟼\\longmapsto/ka/,

gē⟼\\longmapsto/k@/\.

Although their rimes differ, the Pinyin initialgcorresponds to the same normalized onset/k//\{k\}/in both syllables\.

Pinyin also employs the lettersyandwas orthographic devices at the beginning of syllables that lack a consonantal initial\. For example,

衣\(yī\)⟼\\longmapsto/i/,

乌\(wū\)⟼\\longmapsto/u/\.

The initial lettersyandwdo not introduce independent consonantal onsets in these examples\. The two syllables are instead factorized as

衣⟼\\longmapsto\(\[EMPTY\], /i/, //\),

乌⟼\\longmapsto\(\[EMPTY\], /u/, //\)\.

These examples show that Pinyin\-to\-IPA conversion ishighly regular but context\-conditioned\. Once phrase\-level lookup has selected the intended Pinyin reading, deterministic syllable rules can normalize its spelling into an onset, an atomic rime, and a tone\. Pinyin therefore serves as the pronunciation\-bearing interface that makes IPA\-based phonemic factorization applicable to a logographic writing system\.

### B\.4Pinyin\-to\-IPA Correspondences

Unlike Vietnamese Quốc Ngữ, Chinese characters do not provide a systematic graphemic representation of their pronunciation\. Phonemic Tokenizer therefore first assigns a Pinyin pronunciation to each character and then converts the selected Pinyin syllable into IPA\. Once the Pinyin form has been determined, this conversion is largely systematic, although several letters have context\-conditioned realizations\.

A Pinyin syllable is analyzed as

qi⟼\(Oi,Ri,Ti\),q\_\{i\}\\longmapsto\(O\_\{i\},R\_\{i\},T\_\{i\}\),\(29\)whereOiO\_\{i\},RiR\_\{i\}, andTiT\_\{i\}denote its onset, atomic rime, and lexical tone\. The following tables present the normalized mappings used by Phonemic Tokenizer\. They represent broad phonological categories rather than narrow phonetic realizations of individual speakers\.

#### B\.4\.1Initial Correspondences

Pinyin initialNormalized IPAExampleExample IPAb/p/bā/pa/p/ph/pā/pha/m/m/mā/ma/f/f/fā/fa/d/t/dā/ta/t/th/tā/tha/n/n/nā/na/l/l/lā/la/g/k/gā/ka/k/kh/kā/kha/h/x/hā/xa/j/t͡/jī/t͡i/q/t͡h/qī/t͡hi/x//xī/i/zh/t͡ s/zhā/t͡ sa/ch/t͡ sh/chā/t͡ sha/sh/ s/shā/ sa/r/ z/ráo/ zau/z/t͡s/zā/t͡sa/c/t͡sh/cā/t͡sha/s/s/sā/sa/

Table 23:Mapping from Pinyin initials to the normalized IPA onsets used by Phonemic Tokenizer\.Pinyin distinguishes 21 consonantal initials\. Table[23](https://arxiv.org/html/2609.21362#A2.T23)maps them to their normalized IPA representations\. Pinyin letters such asb,d, andgrepresent voiceless unaspirated stops rather than the voiced stops suggested by their conventional Latin values\. Their counterpartsp,t, andkare distinguished by aspiration\.

A syllable without a consonantal initial is assigned\[EMPTY\]in its onset slot\. The Pinyin lettersyandwat syllable\-initial position generally indicate the organization of an onsetless final rather than independent consonantal onsets\. For example,

衣​\(yī\)\\displaystyle\\textit\{衣\}~\(\\textit\{y\\=\{\\i\}\}\)⟼\(\[EMPTY\],/i/,//\),\\displaystyle\\longmapsto\\left\(\\texttt\{\[EMPTY\]\},/\{i\}/,/\{\{\}\\hbox\{\\kern 0\.27771pt\{\\raise 0\.0pt\\hbox\{\{\\char 128\\relax\}\}\}\\kern 0\.0pt\{\\char 191\\relax\}\}\}/\\right\),\(30\)乌​\(wū\)\\displaystyle\\textit\{乌\}~\(\\textit\{w\\=\{u\}\}\)⟼\(\[EMPTY\],/u/,//\)\.\\displaystyle\\longmapsto\\left\(\\texttt\{\[EMPTY\]\},/\{u\}/,/\{\{\}\\hbox\{\\kern 0\.27771pt\{\\raise 0\.0pt\\hbox\{\{\\char 128\\relax\}\}\}\\kern 0\.0pt\{\\char 191\\relax\}\}\}/\\right\)\.\(31\)

#### B\.4\.2Basic Final Correspondences

Pinyin finalNormalized IPAExamplea/a/māo/wo/bōe/7/gēê/E/êai/ai/láiei/ei/lèiao/au/hǎoou/ou/dōuan/an/ānen/9n/ēnang/aŋ/mángeng/9ŋ/dēngong/Uŋ/dōnger/a r/érTable 24:Mapping from basic Pinyin finals to normalized atomic rimes\.Table[24](https://arxiv.org/html/2609.21362#A2.T24)lists finals without a leading high\-vowel medial\. Each complete final is mapped to one atomic rime, even when it contains a nucleus and a nasal or rhotic coda\. The final written asois realized with a labial transition in syllables such asbō,pō,mō, andfó\. The precise phonetic realization varies, but it is normalized to a single rime category in the tokenizer\.

#### B\.4\.3Finals withi,u, andü

Pinyin finalNormalized IPAExamplei/i/mǐia/ia/jiāie/iE/xiěiao/iau/xiǎoiou/iou/liúian/iEn/tiānin/in/xīniang/iaŋ/liànging/iŋ/míngiong/iUŋ/xiōng

Table 25:Mapping of Pinyin finals containing theimedial\. The underlying finaliouis written asiuafter an initial\.GroupPinyin finalNormalized IPAExampleuu/u/lùua/ua/huāuo/uo/guóuai/uai/kuàiuei/uei/duìuan/uan/guānuen/u9n/lùnuang/uaŋ/guāngueng/u9ŋ/wēngüü/y/lǜüe/yE/lüèüan/yEn/juǎnün/yn/jūn

Table 26:Mapping of Pinyin finals containing theuandümedials\. The table shows underlying final forms before Pinyin spelling contractions are applied\.Tables[25](https://arxiv.org/html/2609.21362#A2.T25)and[26](https://arxiv.org/html/2609.21362#A2.T26)present finals containing the three Mandarin medials conventionally represented byi,u, andü\. The IPA representations include the medial as part of the atomic rime\.

#### B\.4\.4Context\-Conditioned Pinyin Spellings

Several Pinyin finals have different written forms depending on whether a consonantal initial is present\. Phonemic Tokenizer expands these surface forms before converting them into IPA, as summarized in Table[27](https://arxiv.org/html/2609.21362#A2.T27)\.

Surface formNormalized finalExampleNormalized IPAiuafter an initialiouliú/liou/uiafter an initialueiduì/tuei/unafter an initialuenlùn/lu9n/yi,ya,yei,ia,ieyī,yā,yè/i/, /ia/, /iE/yao,youiao,iouyào,yǒu/iau/, /iou/yan,yin,yang,yingian,in,iang,ingyán,yīn,yáng,yīngCorrespondingi\-group rimeswu,wa,wou,ua,uowū,wā,wǒCorrespondingu\-group rimeswai,wei,wan,wenuai,uei,uan,uenwài,wèi,wǎn,wènCorrespondingu\-group rimeswang,wenguang,uengwáng,wēngCorrespondingu\-group rimesyu,yue,yuan,yunü,üe,üan,ünyú,yuè,yuán,yúnCorrespondingü\-group rimesuafterj,q, orxüjū,qù,xū/y/ within the corresponding rime

Table 27:Context\-conditioned Pinyin spellings normalized before IPA conversion\. Initialyandwin these forms are treated as orthographic markers rather than consonantal onset tokens\.A further context\-dependent case concerns the Pinyin finali\. Following ordinary initials, it represents the high front vowel/i//\{i\}/\. Following the dental sibilantsz,c, ands, or the retroflex initialszh,ch,sh, andr, it represents a syllabic apical segment rather than/i//\{i\}/\. These contexts are assigned separate normalized rime entries:

mi\\displaystyle\\textit\{mi\}⟼/mi/,\\displaystyle\\longmapsto/\{mi\}/,\(32\)\{zi,ci,si\}\\displaystyle\\\{\\textit\{zi\},\\textit\{ci\},\\textit\{si\}\\\}⟼dental apical rime,\\displaystyle\\longmapsto\\text\{dental apical rime\},\{zhi,chi,shi,ri\}\\displaystyle\\\{\\textit\{zhi\},\\textit\{chi\},\\textit\{shi\},\\textit\{ri\}\\\}⟼retroflex apical rime\.\\displaystyle\\longmapsto\\text\{retroflex apical rime\}\.
This distinction prevents the surface letterifrom incorrectly collapsing phonologically different Mandarin rimes\.

#### B\.4\.5Tone Correspondences

Pinyin represents four principal lexical tones using diacritics or the numerical labels 1–4\. An unstressed neutral tone is left unmarked or marked with the numeral 5\. Table[28](https://arxiv.org/html/2609.21362#A2.T28)gives the normalized tone mappings\.

TonePinyin markExampleTokenizer valueFirstMacronmā//SecondAcutemá//ThirdCaronmǎ//FourthGravemà//NeutralUnmarkedma\[EMPTY\]Table 28:Mapping from Pinyin tone marks to the normalized tone representations used by Phonemic Tokenizer\. The neutral tone is assigned\[EMPTY\]because it has no overt tone mark\.Taken together, these mappings define the deterministic second stage of the Chinese tokenization pipeline:

ci→ψZhqi→ρZhsi⟼\(Oi,Ri,Ti\),c\_\{i\}\\xrightarrow\{\\psi\_\{\\mathrm\{Zh\}\}\}q\_\{i\}\\xrightarrow\{\\rho\_\{\\mathrm\{Zh\}\}\}s\_\{i\}\\longmapsto\(O\_\{i\},R\_\{i\},T\_\{i\}\),\(33\)whereψZh\\psi\_\{\\mathrm\{Zh\}\}assigns a Pinyin pronunciation andρZh\\rho\_\{\\mathrm\{Zh\}\}converts that pronunciation into normalized IPA\. Detailed Pinyin\-to\-IPA mappings are provided in Tables[23](https://arxiv.org/html/2609.21362#A2.T23)–[28](https://arxiv.org/html/2609.21362#A2.T28)\.