CNM-BERT: A Drop-In Structural Embedding for Chinese Characters via Ideographic Description Sequences

# CNM-BERT: A Drop-In Structural Embedding for Chinese Characters via Ideographic Description Sequences
Source: [https://arxiv.org/html/2608.05167](https://arxiv.org/html/2608.05167)
Liqian Yan1,211footnotemark:1 1Shanghai Starriver Bilingual School 2LinkScape \{thomas, eric\}@linkscape\.appCorresponding author\.

###### Abstract

Token\-based encoders like BERT treat Chinese characters as atomic identifiers, ignoring their recursive orthographic structure\. Consequently, models rely on contextual co\-occurrence, degrading performance on rare and out\-of\-vocabulary \(OOV\) characters\. We propose the Compositional Network Model \(CNM\), a lightweight augmentation that injects discrete compositional structure into Transformer encoders\. CNM parses Ideographic Description Sequences \(IDS\) into trees, encodes them via a recursive Tree\-MLP, and fuses the structural embeddings into BERT without modifying the backbone\. Evaluated on theWu et al\. \([2025](https://arxiv.org/html/2608.05167#bib.bib22)\)structural\-probing benchmark, CNM\-BERT outperforms the strongest baseline \(ChineseBERT\) on long\-tail and OOV characters by \+9\.8 Structure accuracy and \+7\.7 Radical F1\. Furthermore, CNM\-BERT achieves consistent gains across CLUE, MRC, and NER tasks at both base and large scales, demonstrating that explicit structural injection delivers both robust OOV understanding and tangible downstream value\.

CNM\-BERT: A Drop\-In Structural Embedding for Chinese Characters via Ideographic Description Sequences

Thomas Sing\-wing Wu1,2††thanks:Equal contribution\.and Liqian Yan1,211footnotemark:1††thanks:Corresponding author\.1Shanghai Starriver Bilingual School2LinkScape\{thomas, eric\}@linkscape\.app

## 1Introduction

Modern Transformer language models rely on tokenization, which enables stable training but imposes a rigid information bottleneck\. Once text is segmented, any internal linguistic structure within a token becomes opaque\. While largely benign for alphabetic languages, this abstraction is fundamentally lossy for logographic scripts like Chinese, where characters are not arbitrary atomic symbols but*recursive compositions*of semantic and phonetic components \(e\.g\., 辯 = ⿲\(⿱\(立,十\), ⿳\(亠,二,口\), ⿱\(立,十\)\)\)\. Mapping characters to atomic IDs discards this structure, forcing models to learn character semantics indirectly through contextual co\-occurrence\. The cost of this indirection falls heavily on the*long tail*: rare characters receive few gradient updates, and standard embedding tables cannot systematically share parameters across orthographically related characters\. Crucially, this limitation isarchitectural, not scale\-bound—increasing parameter count or pre\-training data cannot recover information the input interface has already discarded\. An out\-of\-vocabulary \(\[UNK\]\) or under\-trained character remains opaque regardless of model size\.

#### Contribution\.

We propose theCompositional Network Model \(CNM\), a lightweight, drop\-in augmentation that injects*discrete*sub\-character structure into Transformer encoders without modifying the backbone, vocabulary, or output space\. CNM canonicalizes Ideographic Description Sequences \(IDS\) into rooted parse trees, encodes each character with a recursiveTree\-MLP, and fuses the resulting structural embedding into the standard BERT embedding layer \(Figure[2](https://arxiv.org/html/2608.05167#S3.F2)\)\. Because structure is symbolic and computed only once per unique character per batch, CNM is highly efficient, adding minimal parameters and incurring just≈5%\\approx 5\\%training overhead\.

#### Positioning\.

We evaluate CNM under a dual empirical framework\. Our goal is to demonstrate that explicit structural modeling closes a specific representation gap that token\-only models cannot resolve via scaling, while strictly preserving general NLU performance:

- •Primary: structural probing\.On the*Chinese Character Dataset*\(CCD\)Wu et al\. \([2025](https://arxiv.org/html/2608.05167#bib.bib22)\)—an external benchmark testing sub\-character understanding \(e\.g\., layout, radical decomposition\)—CNM\-BERT improves Structure accuracy on the OOV slice by\+9\.8 pointsand Radical F1 by\+7\.7 pointsover the strongest visual baseline\. This confirms that symbolic decomposition recovers structural signals that scale\-only approaches miss\.
- •Secondary: general NLU\.On standard CLUE, MRC, and NER benchmarks, CNM\-BERT consistently matches or exceeds the best Chinese PLMs of equivalent scale\. This establishes explicit structural injection as a strict refinement of the token interface, conferring specialized OOV robustness without downstream regression\.

## 2Related Work

#### Chinese pre\-trained encoders\.

BERT\(Devlin et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib5)\)established character\-level MLM as the standard for Chinese, and subsequent work has primarily refined the*masking strategy*: BERT\-wwm\(Cui et al\.,[2020a](https://arxiv.org/html/2608.05167#bib.bib2)\)and ERNIE\(Zhang et al\.,[2019b](https://arxiv.org/html/2608.05167#bib.bib28)\)introduce Whole Word and entity\-level masking; MacBERT\(Cui et al\.,[2020a](https://arxiv.org/html/2608.05167#bib.bib2)\)replaces masked tokens with synonyms; ZEN\(Diao et al\.,[2020](https://arxiv.org/html/2608.05167#bib.bib6)\)and NEZHA\(Wei et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib20)\)inject N\-gram information or relative positional encodings\. CNM is orthogonal to all of these—we adopt WWM as best practice but modify the*embedding interface*rather than the masking objective\.

#### Visual and phonological augmentation\.

Glyce\(Meng et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib10)\), ChineseBERT\(Sun et al\.,[2021](https://arxiv.org/html/2608.05167#bib.bib18)\), and MECT\(Wu et al\.,[2021](https://arxiv.org/html/2608.05167#bib.bib21)\)extract sub\-character signals from rendered glyphs via CNNs\. These methods rely on*implicit feature extraction*from continuous pixel statistics, which introduces font sensitivity and substantial compute overhead\. Pinyin\- and Bopomofo\-based variants\(Zhang et al\.,[2021](https://arxiv.org/html/2608.05167#bib.bib26); Tan et al\.,[2022](https://arxiv.org/html/2608.05167#bib.bib19); Si et al\.,[2023](https://arxiv.org/html/2608.05167#bib.bib15)\)replace orthography with pronunciation but suffer from severe homophone ambiguity\(Du and Way,[2017](https://arxiv.org/html/2608.05167#bib.bib7)\), since pronunciation correlates weakly with character semantics compared to compositional structure\.

#### Compositional and structural modeling\.

Static embeddings such as cw2vec\(Cao et al\.,[2018](https://arxiv.org/html/2608.05167#bib.bib1)\)and radical\-level embeddings\(Yin et al\.,[2016](https://arxiv.org/html/2608.05167#bib.bib24); Sun et al\.,[2014](https://arxiv.org/html/2608.05167#bib.bib17)\)established that compositional decomposition yields richer character semantics\. Sub\-character tokenization\(Si et al\.,[2023](https://arxiv.org/html/2608.05167#bib.bib15); Zhang et al\.,[2019a](https://arxiv.org/html/2608.05167#bib.bib27)\)flattens components into the vocabulary, but this alters the output space and risks generating invalid characters\(Nikolov et al\.,[2018](https://arxiv.org/html/2608.05167#bib.bib11)\);Si et al\. \([2023](https://arxiv.org/html/2608.05167#bib.bib15)\)explicitly note that their method is unsuitable for open\-ended generation\. CNM differs by treating each character as a*recursive symbolic function*: rather than pixels or flattened sequences, we encode the IDS*tree*with a recursive Tree\-MLP, preserving the standard tokenizer interface while exposing discrete compositional structure\.

#### Sequence\-level structural integration\.

Lattice\-BERT\(Lai et al\.,[2021](https://arxiv.org/html/2608.05167#bib.bib8)\)integrates word\-level lattice information to disambiguate Chinese segmentation by exposing multiple candidate tokenizations to the encoder\. CNM is*complementary*rather than competitive: Lattice\-BERT refines the*sequence of tokens*, while CNM refines the*representation of each token*via sub\-character composition\. The two operate at orthogonal granularities and could in principle be combined\.

## 3Model

![Refer to caption](https://arxiv.org/html/2608.05167v1/x1.png)Figure 1:Canonical IDS parse tree for the character 辯 \(U\+8FAF\)\. Internal nodes \(blue badges\) are layout operators rendered as their schematic IDC glyphs \(⿲: left\-middle\-right; ⿱: top\-bottom; ⿳: top\-middle\-bottom\); leaves \(orange boxes\) are atomic Unicode components\. The Tree\-MLP encoder computes a structural embedding bottom\-up from this tree, with operator\-conditioned MLPs for binary and ternary nodes\.We propose theCompositional Network Model \(CNM\), a lightweight architectural augmentation for Chinese Transformer encoders\. CNM preserves the*entire*Transformer backbone—all self\-attention and feed\-forward layers remain*identical*to a baseline BERT encoder—and modifies only the*input embedding interface*\. Specifically, CNM introduces a*structure encoder*that computes a per\-character structural embedding from a deterministic*canonicalization*of Ideographic Description Sequences \(IDS\) \(Figure[1](https://arxiv.org/html/2608.05167#S3.F1)\)\. The structural embedding is then fused with the standard BERT embedding at the input layer and propagated through the unchanged encoder stack; the resulting dual\-stream architecture is summarized in Figure[2](https://arxiv.org/html/2608.05167#S3.F2)\.

CNM is designed to satisfy three practical constraints: \(i\)drop\-in compatibilitywith standard BERT fine\-tuning pipelines \(same sequence length and vocabulary interface\), \(ii\)explicit discrete structurederived from symbolic character composition rather than font\-dependent rasterized glyphs, and \(iii\)efficiency, by caching and vectorizing structure computation over unique characters per batch\.

### 3\.1Inputs and Tokenization

We adoptcharacter\-level tokenizationfor Chinese, consistent with common Chinese BERT practice: each CJK Unified Ideograph occupies one token position\.111Non\-Han characters \(such as Latin letters, digits, and punctuation\) are tokenized by the baseline WordPiece rules; CNM attaches a null structural embedding to these positions \(§[3\.4](https://arxiv.org/html/2608.05167#S3.SS4)\)\.Let a batch have sizeBBand padded sequence lengthTT\. CNM consumes the standard BERT inputsinput\_ids∈ℕB×T\\in\\mathbb\{N\}^\{B\\times T\}\(andattention\_mask,token\_type\_idsas usual\), together with a structural index tensorstruct\_idx∈ℕB×T\\texttt\{struct\\\_idx\}\\in\\mathbb\{N\}^\{B\\times T\}\.

#### Structural indexing\.

struct\_idxb,t\\texttt\{struct\\\_idx\}\_\{b,t\}is computed directly from the raw Unicode string*before*mapping to token IDs\. It indexes a precomputed table of canonical IDS parses for the original character at position\(b,t\)\(b,t\), even if the correspondinginput\_idsentry maps to\[UNK\]under the baseline vocabulary\.

### 3\.2Structural Representation from IDS

![Refer to caption](https://arxiv.org/html/2608.05167v1/x2.png)Figure 2:CNM as a drop\-in augmentation\. The character\-level tokenizer feeds two parallel streams: the baseline character embedding and a Tree\-MLP encoder operating on the IDS parse tree of each unique character in the batch \(Figure[1](https://arxiv.org/html/2608.05167#S3.F1)\)\. The fusion layer projects\[𝐞t​o​k;𝐬x\]\[\\mathbf\{e\}^\{tok\};\\mathbf\{s\}\_\{x\}\]back to the model hidden size, leaving the Transformer backbone, vocabulary, and output head identical to vanilla BERT\.CNM represents each Han character using anIDS parse treethat encodes its spatial composition \(illustrated in Figure[1](https://arxiv.org/html/2608.05167#S3.F1)for the character 辯\)\. IDS strings are formed from*Ideographic Description Characters*\(IDCs\), which specify binary or ternary layout operators, and component leaves \(Unicode codepoints\)\. Let𝒱c​h​a​r\\mathcal\{V\}\_\{char\}be the set of Unicode Han characters observed in pre\-training and downstream data\. For each characterx∈𝒱c​h​a​rx\\in\\mathcal\{V\}\_\{char\}, we construct a rooted ordered tree𝒯x\\mathcal\{T\}\_\{x\}whose internal nodes are IDCs and whose leaves are component codepoints\. We denote the set of unique leaf codepoints by𝒱c​m​p\\mathcal\{V\}\_\{cmp\}and the set of IDC operators used in our implementation by𝒱o​p\\mathcal\{V\}\_\{op\}, restricted to a fixed set of standard Unicode binary/ternary layout operators\.222We list𝒱o​p\\mathcal\{V\}\_\{op\}and report IDS coverage statistics in §[4](https://arxiv.org/html/2608.05167#S4)\.

#### Canonicalization\.

IDS decompositions are not guaranteed to be unique\. CNM applies a deterministiccanonicalizationthat maps each characterxxto exactly one tree𝒯x\\mathcal\{T\}\_\{x\}\. We \(i\) discard candidates with private\-use codepoints or non\-standard markers, \(ii\) resolve intermediate aliases until all leaves are Unicode codepoints in𝒱c​m​p\\mathcal\{V\}\_\{cmp\}, and \(iii\) restrict internal nodes to binary/ternary IDCs in𝒱o​p\\mathcal\{V\}\_\{op\}\. Among remaining candidates we select by lexicographic order over \(a\) tree depth, \(b\) the predicate that every operator lies in the high\-frequency subset \{⿰, ⿱\}, \(c\) total node count, and \(d\) a stable lexicographic operator hash for tie\-breaking\. The full procedure is formalized in Appendix[B](https://arxiv.org/html/2608.05167#A2)\. Characters without a valid IDS entry \(∼3%\\sim 3\\%of the vocabulary\) map to a learnable structural embedding𝐬u​n​k\\mathbf\{s\}\_\{unk\}\.

### 3\.3Recursive Tree\-MLP Structure Encoder

To map the discrete tree𝒯x\\mathcal\{T\}\_\{x\}into a dense structural embedding𝐬x∈ℝds\\mathbf\{s\}\_\{x\}\\in\\mathbb\{R\}^\{d\_\{s\}\}, we introduce aTree\-MLP Encodercomputed bottom\-up\. LetEc​m​p∈ℝ\|𝒱c​m​p\|×dsE\_\{cmp\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{cmp\}\|\\times d\_\{s\}\}andEo​p∈ℝ\|𝒱o​p\|×dsE\_\{op\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{op\}\|\\times d\_\{s\}\}be learnable embedding matrices for components and operators, respectively\. For any nodennin𝒯x\\mathcal\{T\}\_\{x\}, we compute a hidden state𝐡n∈ℝds\\mathbf\{h\}\_\{n\}\\in\\mathbb\{R\}^\{d\_\{s\}\}\.

#### Leaf nodes\.

Ifnnis a leaf corresponding to componentc∈𝒱c​m​pc\\in\\mathcal\{V\}\_\{cmp\},

𝐡n=LayerNorm​\(Ec​m​p​\(c\)\)\.\\mathbf\{h\}\_\{n\}=\\text\{LayerNorm\}\\\!\\left\(E\_\{cmp\}\(c\)\\right\)\.\(1\)

#### Internal nodes\.

Ifnnis an internal node corresponding to operatoro∈𝒱o​po\\in\\mathcal\{V\}\_\{op\}with ordered childrenc1,…,ckc\_\{1\},\\dots,c\_\{k\}wherek∈\{2,3\}k\\in\\\{2,3\\\}, we apply an operator\-conditioned MLP with a bounded residual path:

𝐡c​a​t=\[Eo​p​\(o\);𝐡c1;…;𝐡ck\]∈ℝ\(k\+1\)​ds,\\mathbf\{h\}\_\{cat\}=\[\\,E\_\{op\}\(o\);\\ \\mathbf\{h\}\_\{c\_\{1\}\};\\ \\dots;\\ \\mathbf\{h\}\_\{c\_\{k\}\}\\,\]\\in\\mathbb\{R\}^\{\(k\+1\)d\_\{s\}\},\(2\)𝐡n=LayerNorm​\(GELU​\(𝐖k​𝐡c​a​t\+𝐛k\)\+1k​∑i=1k𝐡ci\),\\mathbf\{h\}\_\{n\}=\\text\{LayerNorm\}\\\!\\left\(\\text\{GELU\}\(\\mathbf\{W\}\_\{k\}\\mathbf\{h\}\_\{cat\}\+\\mathbf\{b\}\_\{k\}\)\+\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{h\}\_\{c\_\{i\}\}\\right\),\(3\)where𝐖2∈ℝds×3​ds\\mathbf\{W\}\_\{2\}\\in\\mathbb\{R\}^\{d\_\{s\}\\times 3d\_\{s\}\},𝐖3∈ℝds×4​ds\\mathbf\{W\}\_\{3\}\\in\\mathbb\{R\}^\{d\_\{s\}\\times 4d\_\{s\}\}, and𝐛k∈ℝds\\mathbf\{b\}\_\{k\}\\in\\mathbb\{R\}^\{d\_\{s\}\}\. The structural embedding for characterxxis defined as the root hidden state:

𝐬x=𝐡r​o​o​t​\(𝒯x\)\.\\mathbf\{s\}\_\{x\}=\\mathbf\{h\}\_\{root\}\(\\mathcal\{T\}\_\{x\}\)\.\(4\)

#### Batch efficiency\.

Computing𝐬x\\mathbf\{s\}\_\{x\}independently at each token position would repeat work across identical characters\. During training and inference, CNM therefore computes structure embeddings only for the set of unique Han characters appearing in the current batch, denoted𝒱b​a​t​c​h\\mathcal\{V\}\_\{batch\}\. We precompile each𝒯x\\mathcal\{T\}\_\{x\}into a topologically sorted instruction buffer \(post\-order traversal\) so that the Tree\-MLP can be vectorized across nodes\. We then gather the corresponding𝐬x\\mathbf\{s\}\_\{x\}back to token positions viastruct\_idx\. This reduces per\-step structural computation fromO​\(B​T\)O\(BT\)tree evaluations toO​\(\|𝒱b​a​t​c​h\|\)O\(\|\\mathcal\{V\}\_\{batch\}\|\)character evaluations, with small constant factors\.

### 3\.4Fusion at the Embedding Interface

CNM injects structural information at theembedding interfacewhile leaving the Transformer encoder layers unchanged\. Let𝐞it​o​k∈ℝd\\mathbf\{e\}^\{tok\}\_\{i\}\\in\\mathbb\{R\}^\{d\}denote the baseline token embedding at positionii, and let𝐩i∈ℝd\\mathbf\{p\}\_\{i\}\\in\\mathbb\{R\}^\{d\}and𝐠i∈ℝd\\mathbf\{g\}\_\{i\}\\in\\mathbb\{R\}^\{d\}denote the baseline position and segment embeddings, respectively\. CNM computes a structure vector𝐬xi∈ℝds\\mathbf\{s\}\_\{x\_\{i\}\}\\in\\mathbb\{R\}^\{d\_\{s\}\}from the original characterxix\_\{i\}at positionii; for positions without a defined IDS \(e\.g\., special tokens\[CLS\],\[SEP\], punctuation, Latin characters\), we use a learnable null structural embedding𝐬∅\\mathbf\{s\}\_\{\\emptyset\}, and for characters missing from the IDS table we use𝐬u​n​k\\mathbf\{s\}\_\{unk\}\.

We fuse the token and structure streams by concatenation followed by a linear projection to the model hidden size:

𝐳i=LayerNorm​\(𝐖f​\[𝐞it​o​k;𝐬xi\]\+𝐛f\),\\mathbf\{z\}\_\{i\}=\\text\{LayerNorm\}\\\!\\left\(\\mathbf\{W\}\_\{f\}\[\\mathbf\{e\}^\{tok\}\_\{i\};\\ \\mathbf\{s\}\_\{x\_\{i\}\}\]\+\\mathbf\{b\}\_\{f\}\\right\),\(5\)where𝐖f∈ℝd×\(d\+ds\)\\mathbf\{W\}\_\{f\}\\in\\mathbb\{R\}^\{d\\times\(d\+d\_\{s\}\)\}and𝐛f∈ℝd\\mathbf\{b\}\_\{f\}\\in\\mathbb\{R\}^\{d\}\. The final input embedding to the Transformer is then formed as in BERT:

𝐞i=𝐳i\+𝐩i\+𝐠i,\\mathbf\{e\}\_\{i\}=\\mathbf\{z\}\_\{i\}\+\\mathbf\{p\}\_\{i\}\+\\mathbf\{g\}\_\{i\},\(6\)followed by the baseline embedding LayerNorm and dropout\. See implementation details in §[5](https://arxiv.org/html/2608.05167#S5)\. The fused embeddings\{𝐞i\}i=1T\\\{\\mathbf\{e\}\_\{i\}\\\}\_\{i=1\}^\{T\}are fed to the unchanged Transformer encoder layers\.

#### Parameters and overhead\.

CNM introduces parameters inEc​m​pE\_\{cmp\},Eo​pE\_\{op\},\(𝐖2,𝐖3\)\(\\mathbf\{W\}\_\{2\},\\mathbf\{W\}\_\{3\}\), and𝐖f\\mathbf\{W\}\_\{f\}\(plus𝐬∅\\mathbf\{s\}\_\{\\emptyset\}and𝐬u​n​k\\mathbf\{s\}\_\{unk\}\)\. We report parameter counts and training/inference throughput impact in §[5](https://arxiv.org/html/2608.05167#S5)\.

### 3\.5Pre\-training Objective

We train CNM\-BERT end\-to\-end with WWM\-MLM at 15% corruption with the standard 80/10/10 replacement strategy, applying the same corruption pattern to the aligned structural indices to prevent gold\-structure leakage\. We additionally use an auxiliary component\-prediction loss: for each masked positionmmwith gold leaf components\{c1,…,cLm\}\\\{c\_\{1\},\\ldots,c\_\{L\_\{m\}\}\\\}, we predict eachclc\_\{l\}from a*target*structural embedding computed only from the gold tree \(i\.e\., not visible to the Transformer input\)\. The total loss isℒ=ℒMLM\+λ​ℒaux\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{MLM\}\}\+\\lambda\\,\\mathcal\{L\}\_\{\\text\{aux\}\}withλ=0\.1\\lambda=0\.1\. The full formal derivation is given in Appendix[D](https://arxiv.org/html/2608.05167#A4)\.

## 4Experiment Setup

In this section, we introduce our baselines, pretraining corpus, evaluation benchmarks, and experimental settings\.

### 4\.1Baselines

We compare CNM\-BERT against a broad set of Chinese PLMs of comparable scale\.Masking variants:BERT\(Devlin et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib5)\), BERT\-wwm and BERT\-wwm\-ext\(Cui et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib3)\), RoBERTa\-wwm\-ext, MacBERT\(Cui et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib3)\), ERNIE\(Zhang et al\.,[2019b](https://arxiv.org/html/2608.05167#bib.bib28)\)\.Visual/phonological augmentation:ChineseBERT\(Sun et al\.,[2021](https://arxiv.org/html/2608.05167#bib.bib18)\)\.Sub\-character tokenization\(added in this revision\): SubChar\-Wubi and SubChar\-Pinyin\(Si et al\.,[2023](https://arxiv.org/html/2608.05167#bib.bib15)\), which replace the vocabulary with Wubi keystrokes or Pinyin syllables\. We were unable to obtain Lattice\-BERT\(Lai et al\.,[2021](https://arxiv.org/html/2608.05167#bib.bib8)\)because its released code repository is no longer accessible; we discuss its complementary positioning in §[2](https://arxiv.org/html/2608.05167#S2)\.

#### Controlled re\-training\.

To attribute gains cleanly, we additionally re\-trained BERT and MacBERT under an identical recipe to CNM\-BERT—same corpus, vocabulary, optimizer, schedule, batch size, update count, FP16 / gradient\-clipping settings, and fine\-tuning protocol—so that controlled comparisons differ only in the presence of the CNM structural pathway \(and WWM, which is absent in the BERT baseline\)\. All other models in our tables use their released checkpoints under our common fine\-tuning protocol\.

### 4\.2Pretraining Data

FollowingSun et al\. \([2021](https://arxiv.org/html/2608.05167#bib.bib18)\), we pretrain on Chinese Wikipedia plus filtered CommonCrawl \(≈\\approx4B tokens\), with HTML/dedup/non\-Chinese filtering, Jieba333[https://github\.com/fxsjy/jieba](https://github.com/fxsjy/jieba)segmentation for WWM, and IDS structural decompositions from the BabelStone database444[https://babelstone\.co\.uk/CJK/IDS\.TXT](https://babelstone.co.uk/CJK/IDS.TXT); covers 97,680 CJK Unified Ideographs\.\. Trees use 12 IDS operators with maximum depth 6 over a component vocabulary of≈\\approx5K atomic radicals; the∼\\sim3% of characters lacking a valid IDS entry fall back to a learnable embedding𝐬u​n​k\\mathbf\{s\}\_\{unk\}\.

### 4\.3Evaluation Data

We evaluate CNM\-BERT under two complementary lenses corresponding to its primary and secondary contributions\.

#### Primary: structural probing \(CCD\)\.

TheChinese Character Dataset\(CCD\)\(Wu et al\.,[2025](https://arxiv.org/html/2608.05167#bib.bib22)\)is an external diagnostic benchmark designed specifically to test sub\-character understanding\. Each example is a single character formatted as\[CLS\] x \[SEP\], with annotations for \(i\) top\-level layout structure \(8\-way macro\-F1\), \(ii\) radical decomposition \(set F1\), \(iii\) stroke count \(MAE\), and \(iv\) stroke\-type sequence \(F1\)\. We evaluate on three splits: anIIDcharacter split, aLong\-tailtier split \(train on high/mid\-frequency, test on lowest\-frequency tier\), and anOOV sliceof the long\-tail test set restricted to characters that map to\[UNK\]under the model’s tokenizer\. The OOV slice isolates the regime where token\-only models structurally collapse\.

#### Secondary: general NLU\.

We evaluate on twelve standard Chinese NLU tasks:CLUE\(TNEWS, IFLYTEK, AFQMC, CMNLI, CSL, CLUEWSC2020\)\(Xu et al\.,[2020](https://arxiv.org/html/2608.05167#bib.bib23)\),MRC\(CMRC 2018\(Cui et al\.,[2020b](https://arxiv.org/html/2608.05167#bib.bib4)\), DRCD\(Shao et al\.,[2019](https://arxiv.org/html/2608.05167#bib.bib14)\), C3\(Sun et al\.,[2020](https://arxiv.org/html/2608.05167#bib.bib16)\)\), andNER\(MSRA\(Levow,[2006](https://arxiv.org/html/2608.05167#bib.bib9)\), OntoNotes 4\.0\(Pradhan et al\.,[2011](https://arxiv.org/html/2608.05167#bib.bib13)\), Weibo\(Peng and Dredze,[2015](https://arxiv.org/html/2608.05167#bib.bib12)\)\)\. We use the official CLUE splits and canonical MRC/NER splits; full statistics are in Appendix[A](https://arxiv.org/html/2608.05167#A1)\.

### 4\.4Hyper\-parameters

We train two scales following standard BERT configurations:base\(12 layers, hidden 768\) initialized frombert\-base\-chinese, andlarge\(24 layers, hidden 1024\) initialized fromhfl/chinese\-roberta\-wwm\-ext\-large\. Structural components useds=256d\_\{s\}=256, hidden 512, max tree depth 6, and a∼\\sim5K\-component, 16\-operator vocabulary\. The fusion layer is initialized with Identity\+Zero weights to preserve pretrained representations\. We pretrain for 1M steps with effective batch size 256 \(base\) / 512 \(large\), LAMB optimizer\(You et al\.,[2020](https://arxiv.org/html/2608.05167#bib.bib25)\), peak LR1×10−41\{\\times\}10^\{\-4\}, 10K warmup steps, FP16 on 8×\\timesA100 80GB GPUs \(≈\\approx7 / 14 days\)\. Finetuning uses a learning\-rate grid\{1,2,3,5\}×10−5\\\{1,2,3,5\\\}\{\\times\}10^\{\-5\}with 5 seeds and early stopping on dev\. Full architectural and optimization detail is in Appendix[E](https://arxiv.org/html/2608.05167#A5)\.

## 5Experiment Results

We organize results around the dual evaluation contract introduced in §1:primarystructural\-probing evidence on CCD \(§[5\.1](https://arxiv.org/html/2608.05167#S5.SS1)\),secondarygeneral\-NLU evidence on CLUE/MRC/NER \(§[5\.2](https://arxiv.org/html/2608.05167#S5.SS2)–[3](https://arxiv.org/html/2608.05167#S5.T3)\), anablationthat isolates which components of CNM produce which gains \(§[5\.4](https://arxiv.org/html/2608.05167#S5.SS4)\), and a briefefficiencynote \(§[5\.5](https://arxiv.org/html/2608.05167#S5.SS5)\)\.

#### Reproduction methodology\.

Published Chinese\-PLM results vary substantially across hyperparameter grids, finetuning scripts, seeds, and library versions, which makes canonical comparisons fragile\. We therefore re\-run*all*baselines from official HuggingFace checkpoints under a single identical protocol \(§[4](https://arxiv.org/html/2608.05167#S4)\)—five seeds, same grid, same hardware—and our reproduced baseline numbers fall within±0\.1\\pm 0\.1of published values on most tasks\.

### 5\.1Primary Result: Structural Probing on CCD

CCD is the most direct test of the claim CNM\-BERT actually makes: that explicit symbolic decomposition recovers sub\-character information that token\-only models cannot represent\. Table[1](https://arxiv.org/html/2608.05167#S5.T1)reports Structure macro\-F1, Radical F1, stroke\-count MAE, and Stroke\-type F1 across three splits of increasing difficulty: IID, long\-tail, and the OOV slice within the long\-tail split\.

Table 1:CCD diagnostic results \(primary evaluation\)\.Structure and Radical metrics measure symbolic compositional understanding; stroke metrics measure visual rendering\. CNM\-BERT is the strongest model on*every*symbolic metric across*every*split, with the largest margins exactly where token\-only models fail \(OOV slice:\+9\.8\+9\.8Struct,\+7\.7\+7\.7Radical over ChineseBERT\)\. ChineseBERT, which has access to rendered glyph pixels, leads on the visual stroke metrics—an expected and complementary outcome\.![Refer to caption](https://arxiv.org/html/2608.05167v1/x3.png)Figure 3:CCD Structure F1 across character\-frequency tiers\. Token\-only baselines collapse on the OOV slice; CNM\-BERT degrades gracefully because the structural pathway provides a fallback when the token pathway fails\. The widening gap from Head \(\+1\.2\+1\.2\) to OOV \(\+9\.8\+9\.8over the strongest baseline\) is the qualitative signature of an architectural prior\.#### Findings\.

Three observations characterize the CCD results\. First, on theOOV slicethe gap between token\-only baselines \(Structure≤14\.0\\leq 14\.0, Radical≤7\.4\\leq 7\.4\) and structurally\-aware models is enormous: when the tokenizer fails, the encoder has no fallback, and the model collapses to chance\. CNM\-BERT lifts OOV Structure to76\.076\.0and Radical to56\.156\.1, exceeding the strongest visual baseline \(ChineseBERT\) by\+9\.8\+9\.8and\+7\.7\+7\.7respectively, and exceeding the strongest sub\-character\-tokenization baseline \(SubChar\-Wubi\) by\+10\.8\+10\.8and\+11\.1\+11\.1\. Second, the gain*grows*with distributional shift: from\+1\.3\+1\.3Structure on IID to\+4\.0\+4\.0on long\-tail to\+9\.8\+9\.8on OOV\. This is the qualitative signature of an architectural prior, not a corpus artifact\. Third, on visual stroke metrics ChineseBERT remains best, which is consistent with its design \(it sees pixels\)\. CNM\-BERT and ChineseBERT thus capture genuinely complementary signals—symbolic composition and visual rendering—and the appropriate evaluation question is which of these is more transferable to downstream tasks\. We address that question next\.

### 5\.2Secondary Result: CLUE Benchmark

Table 2:CLUE benchmark \(test accuracy %\)\. All baselines reproduced under identical finetuning conditions; CNM\-BERT achieves the best average at both scales but the absolute margin over the strongest external baseline is small \(\+0\.18\+0\.18base,\+0\.19\+0\.19large\)\. The relevant claim is parity, not state of the art—structural injection is a strict refinement of the standard token interface and does not regress general NLU\.CNM\-BERT achieves the best average on CLUE at both base \(71\.0371\.03\) and large \(73\.5673\.56\) scales, surpassing every reproduced baseline\. The margin over the strongest external baseline is small \(\+0\.18\+0\.18base,\+0\.19\+0\.19large\) but statistically reliable across five fine\-tuning seeds and consistent at both scales\. Against our matched\-recipe MacBERT and BERT baselines the gains are\+0\.50\+0\.50and\+2\.69\+2\.69avg respectively\. The two SubChar baselines reveal a typical sub\-character trade\-off: they gain on TNEWS/CSL \(\+5−6\+5\{\-\}6pts\) but lose on AFQMC/IFLYTEK \(−6\-6pts\) because vocabulary substitution destroys standard token semantics\. CNM\-BERT shows no such trade\-off, because it*augments*rather than*replaces*the token interface\.

### 5\.3Secondary Result: MRC and NER

CMRC 2018DRCDC3NER \(span F1\)ModelEMF1EMF1DevTestMSRAOnto\.WeiboBase ModelsBERT65\.584\.583\.189\.965\.764\.594\.9380\.8767\.33RoBERTa\-wwm\-ext67\.487\.286\.692\.567\.166\.595\.4280\.3768\.15MacBERT68\.587\.989\.494\.369\.368\.2———ChineseBERT69\.688\.387\.893\.470\.469\.195\.8481\.6569\.02CNM\-BERT \(ours\)69\.488\.888\.994\.671\.270\.195\.9081\.9270\.15Large ModelsRoBERTa\-wwm\-ext70\.088\.689\.694\.872\.171\.296\.1481\.3968\.35MacBERT70\.788\.990\.795\.673\.272\.0———ChineseBERT71\.689\.790\.595\.474\.072\.896\.5282\.1870\.80CNM\-BERT \(ours\)71\.389\.590\.695\.574\.473\.896\.4882\.4071\.35

Table 3:Reading comprehension \(EM/F1; C3accuracy\) and named entity recognition \(span F1 %\)\. NER results for noisy social\-media text \(Weibo\) show the largest CNM\-BERT gains \(\+1\.13\+1\.13over ChineseBERT base,\+0\.55\+0\.55large\), consistent with the CCD long\-tail finding: structural priors help most when surface statistics are unreliable\.Reading comprehension and NER show the same pattern as CLUE: CNM\-BERT is competitive everywhere and best on roughly half the metrics, with the cleanest gains appearing on noisy or rare\-character tasks \(Weibo NER\+1\.13\+1\.13base; C3Test\+1\.0\+1\.0\)\. On clean span\-extraction \(CMRC, DRCD\) CNM\-BERT and ChineseBERT trade leads within a fraction of a point\. We interpret this as further evidence that structural injection is a refinement, not a regression\.

### 5\.4Ablation Study

Table 4:Ablation: CLUE\-Avg \(dev\) and CCD long\-tail OOV Structure accuracy\. The hierarchical Tree\-MLP fusion \(rows 2–3\) is the dominant source of structural gains; the auxiliary loss alone \(row 4\) is insufficient; flattening the tree to a bag\-of\-components \(row 5\) loses17\.717\.7pts on OOV\.To isolate which component of CNM produces which gain—the Tree\-MLP fusion pathway, the auxiliary component\-prediction loss, or the recursive hierarchy itself—we ablate each independently on a single A100 80GB GPU\. We report CLUE Avg on the development set \(general NLU\) and CCD\-OOV Structure accuracy \(the regime where the architectural prior matters most\)\. Results are in Table[4](https://arxiv.org/html/2608.05167#S5.T4)\.

The ablation supports three conclusions\.\(i\) Structural injection is what matters: aux loss without Tree\-MLP fusion \(row 4\) lifts CCD\-OOV by only\+7\.6\+7\.6vs\.\+41\.8\+41\.8for the full model, ruling out multi\-task regularization as the source of gain\.\(ii\) Hierarchy carries real signal: flattening to a bag\-of\-components \(row 5\) loses17\.717\.7pts on CCD\-OOV \(76\.0→58\.376\.0\\to 58\.3\), so the IDS*tree*structure, not just the components, is informationally load\-bearing\.\(iii\) The auxiliary loss is a small but reliable addition: rows \(2\) vs\. \(3\) showλ=0\.1\\lambda\{=\}0\.1adds\+0\.22\+0\.22CLUE\-Avg and\+1\.5\+1\.5CCD\-OOV overλ=0\\lambda\{=\}0, consistent across five seeds, so we keep it as a regularizer rather than as a load\-bearing component\.

### 5\.5Efficiency

CNM\-BERT adds only≈2\.5\\approx 2\.5M parameters over BERT\-base for the structural pathway—the component table \(≈\\approx1\.28M\), operator table \(4K\), operator\-conditioned MLPs \(≈\\approx460K\), and fusion projection \(≈\\approx790K\); the full breakdown is in Appendix[C](https://arxiv.org/html/2608.05167#A3)\. This is≈18×\\approx 18\\timesless than the≈45\\approx 45M ChineseBERT adds for its glyph CNN and Pinyin embeddings\. Training is correspondingly cheap:≈5%\\approx 5\\%slowdown over vanilla BERT \(142→135142\\to 135samples/sec at batch size 32, sequence length 512, single A100\), substantially faster than ChineseBERT \(9898samples/sec,−30%\-30\\%\)\. The overhead is bounded by our caching strategy: the Tree\-MLP is evaluated only on the set of unique characters in each batch, not at every token position, reducing per\-step structural compute fromO​\(B​T\)O\(BT\)tree evaluations toO​\(\|𝒱b​a​t​c​h\|\)O\(\|\\mathcal\{V\}\_\{batch\}\|\)\.

## 6Discussion and Conclusion

We presented theCompositional Network Model \(CNM\), a lightweight, drop\-in augmentation that exposes discrete sub\-character structure to a Transformer encoder via deterministic IDS canonicalization and a recursive Tree\-MLP\. The contribution rests on two empirical claims\. First, in the regime that token\-only models cannot represent—rare and OOV characters—explicit symbolic decomposition produces large, qualitatively different gains:\+9\.8\+9\.8Structure and\+7\.7\+7\.7Radical points over the strongest visual baseline on the CCD OOV slice\. Second, on broad NLU CNM\-BERT achieves the highest average on CLUE, MRC and NER at both scales, with margins that are small in absolute terms \(≈0\.2\\approx 0\.2–0\.50\.5avg\) but statistically reliable across five seeds and consistent at both scales\. Ablations attribute the bulk of the gain to the hierarchical Tree\-MLP fusion\. Together these results show that an architectural prior targeting sub\-character structure can deliver real downstream value*and*close the OOV structural gap without trading off against general NLU—a dual property that, to our knowledge, no prior method achieves\. The broader implication is that the sub\-character information gap is*architectural*: it cannot be closed by scale alone, because no amount of contextual co\-occurrence recovers structure the input interface has discarded, and even much larger generative models that share the character\-as\-atom assumption inherit this limitation\.

## Limitations

We note four limitations\. First, CNM depends on the coverage and quality of an external IDS database; characters lacking a valid IDS entry fall back to a learnable embedding \(∼3%\\sim 3\\%of our vocabulary, mostly rare variants\), so the structural pathway provides no benefit for those characters\. Second, the scope of this work is restricted to Chinese\. The methodology is in principle transferable to other Han\-derived scripts \(Japanese Kanji, Korean Hanja, Vietnamese Chũ Nôm\) and to morphologically rich non\-logographic languages, but we leave empirical verification to future work\. Third, the paper’s strongest empirical claim depends heavily on CCD\. This is appropriate given the paper’s structural\-probing goal, but we want to be explicit about the relationship between CCD labels and the IDS source used by CNM: both ultimately derive from compositional decomposition resources for Han characters\. If the two share substantial overlap, the \+9\.8 OOV gain is best interpreted as evidence of*structured\-resource transfer*—CNM correctly propagates the structural information available in its training\-time database to the evaluation\-time probing task—rather than as evidence of an independent emergent capability\. We believe both readings are scientifically valuable, but the more conservative one should be preferred until cross\-resource probes \(e\.g\., CCD\-style labels derived from a disjoint compositional taxonomy\) are available\. Fourth, our CLUE/MRC/NER results are best characterized as small but consistent improvements over the strongest existing Chinese PLM baselines\. We report that CNM\-BERT achieves the highest average score on every general\-NLU table we report, by margins that are statistically reliable across five fine\-tuning seeds but practically small \(typically0\.20\.2–0\.50\.5avg points over MacBERT and ChineseBERT\)\. The contribution is not to dominate these benchmarks but to demonstrate that an architectural prior targeting sub\-character structure can deliver this small but real improvement on general NLU*while also*closing a large gap on out\-of\-vocabulary structural probes—a dual property that no prior method we are aware of achieves\.

## Acknowledgments

This work was conducted entirely independently by the listed authors\. No mentorship, assistance, or guidance of any kind contributed to any aspect of this paper\.

We thank Alibaba Cloud for generously providing the computational resources used in this work, which enabled our experiments and analyses\.

## References

- Cao et al\. \(2018\)Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li\. 2018\.[cw2vec: Learning chinese word embeddings with stroke n\-gram information](https://www.aaai.org/ojs/index.php/AAAI/article/view/12029)\.In*Proceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence*, pages 5053–5061\. Association for the Advancement of Artificial Intelligence\.
- Cui et al\. \(2020a\)Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu\. 2020a\.[Revisiting pre\-trained models for Chinese natural language processing](https://doi.org/10.18653/v1/2020.findings-emnlp.58)\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 657–668, Online\. Association for Computational Linguistics\.
- Cui et al\. \(2019\)Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu\. 2019\.[Pre\-training with whole word masking for chinese bert](https://doi.org/10.48550/arXiv.1906.08101)\.In*arXiv preprint arXiv:1906\.08101*, pages 1–9\. arXiv\.
- Cui et al\. \(2020b\)Yiming Cui, Ting Liu, Ziqing Yang, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu\. 2020b\.[A sentence cloze dataset for Chinese machine reading comprehension](https://doi.org/10.18653/v1/2020.coling-main.589)\.In*Proceedings of the 28th International Conference on Computational Linguistics*, pages 6717–6723, Barcelona, Spain \(Online\)\. International Committee on Computational Linguistics\.
- Devlin et al\. \(2019\)Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\. 2019\.[BERT: Pre\-training of deep bidirectional transformers for language understanding](https://doi.org/10.18653/v1/N19-1423)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 4171–4186, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Diao et al\. \(2020\)Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang\. 2020\.[ZEN: Pre\-training Chinese text encoder enhanced by n\-gram representations](https://doi.org/10.18653/v1/2020.findings-emnlp.425)\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 4729–4740, Online\. Association for Computational Linguistics\.
- Du and Way \(2017\)Jinhua Du and Andy Way\. 2017\.Pinyin as subword unit for chinese\-sourced neural machine translation\.In*Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science \(AICS\)*\.
- Lai et al\. \(2021\)Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, and Dongyan Zhao\. 2021\.Lattice\-BERT: Leveraging multi\-granularity representations in Chinese pre\-trained language models\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*, pages 1716–1731\.
- Levow \(2006\)Gina\-Anne Levow\. 2006\.[The third international Chinese language processing bakeoff: Word segmentation and named entity recognition](https://aclanthology.org/W06-0115/)\.In*Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing*, pages 108–117, Sydney, Australia\. Association for Computational Linguistics\.
- Meng et al\. \(2019\)Yuxian Meng, Wei Wu, Fei Wang, Xin Li, Ping Nie, Fan Yin, Ming Li, Qinghong Han, Xiaoyan Sun, and Jiwei Li\. 2019\.Glyce: Glyph\-vectors for chinese character representations\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Nikolov et al\. \(2018\)Nikola I\. Nikolov, Yuhuang Hu, Mi Xue Tan, and Richard H\.R\. Hahnloser\. 2018\.[Character\-level Chinese\-English translation through ASCII encoding](https://doi.org/10.18653/v1/W18-6302)\.In*Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 10–16, Brussels, Belgium\. Association for Computational Linguistics\.
- Peng and Dredze \(2015\)Nanyun Peng and Mark Dredze\. 2015\.[Named entity recognition for Chinese social media with jointly trained embeddings](https://doi.org/10.18653/v1/D15-1064)\.In*Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 548–554, Lisbon, Portugal\. Association for Computational Linguistics\.
- Pradhan et al\. \(2011\)Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue\. 2011\.[CoNLL\-2011 shared task: Modeling unrestricted coreference in OntoNotes](https://aclanthology.org/W11-1901/)\.In*Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task*, pages 1–27, Portland, Oregon, USA\. Association for Computational Linguistics\.
- Shao et al\. \(2019\)Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai\. 2019\.[Drcd: a chinese machine reading comprehension dataset](https://arxiv.org/abs/1806.00920)\.*Preprint*, arXiv:1806\.00920\.
- Si et al\. \(2023\)Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun\. 2023\.[Sub\-character tokenization for Chinese pretrained language models](https://doi.org/10.1162/tacl_a_00560)\.*Transactions of the Association for Computational Linguistics*, 11:469–487\.
- Sun et al\. \(2020\)Kai Sun, Dian Yu, Dong Yu, and Claire Cardie\. 2020\.[Investigating prior knowledge for challenging Chinese machine reading comprehension](https://doi.org/10.1162/tacl_a_00305)\.*Transactions of the Association for Computational Linguistics*, 8:141–155\.
- Sun et al\. \(2014\)Yaming Sun, Lei Lin, Nan Yang, Zhenzhou Ji, and Xiaolong Wang\. 2014\.Radical\-enhanced chinese character embedding\.In*International Conference on Neural Information Processing*, pages 279–286\. Springer\.
- Sun et al\. \(2021\)Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, and Jiwei Li\. 2021\.[ChineseBERT: Chinese pretraining enhanced by glyph and Pinyin information](https://doi.org/10.18653/v1/2021.acl-long.161)\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 2065–2075, Online\. Association for Computational Linguistics\.
- Tan et al\. \(2022\)Minghuan Tan, Yong Dai, Duyu Tang, Zhangyin Feng, Guoping Huang, Jing Jiang, Jiwei Li, and Shuming Shi\. 2022\.[Exploring and adapting Chinese GPT to Pinyin input method](https://doi.org/10.18653/v1/2022.acl-long.133)\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1899–1909, Dublin, Ireland\. Association for Computational Linguistics\.
- Wei et al\. \(2019\)Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu\. 2019\.[NEZHA: Neural contextualized representation for chinese language understanding](https://arxiv.org/abs/1909.00204)\.*arXiv preprint arXiv:1909\.00204*\.
- Wu et al\. \(2021\)Shuang Wu, Xiaoning Song, and Zhenhua Feng\. 2021\.[MECT: Multi\-metadata embedding based cross\-transformer for Chinese named entity recognition](https://doi.org/10.18653/v1/2021.acl-long.121)\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 1529–1539, Online\. Association for Computational Linguistics\.
- Wu et al\. \(2025\)Xiaofeng Wu, Karl Stratos, and Wei Xu\. 2025\.[The impact of visual information in Chinese characters: Evaluating large models’ ability to recognize and utilize radicals](https://doi.org/10.18653/v1/2025.naacl-long.16)\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 331–350, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Xu et al\. \(2020\)Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, and 13 others\. 2020\.[CLUE: A Chinese language understanding evaluation benchmark](https://doi.org/10.18653/v1/2020.coling-main.419)\.In*Proceedings of the 28th International Conference on Computational Linguistics*, pages 4762–4772, Barcelona, Spain \(Online\)\. International Committee on Computational Linguistics\.
- Yin et al\. \(2016\)Rongchao Yin, Quan Wang, Peng Li, Rui Li, and Bin Wang\. 2016\.[Multi\-granularity Chinese word embedding](https://doi.org/10.18653/v1/D16-1100)\.In*Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pages 981–986, Austin, Texas\. Association for Computational Linguistics\.
- You et al\. \(2020\)Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho\-Jui Hsieh\. 2020\.[Large batch optimization for deep learning: Training bert in 76 minutes](https://arxiv.org/abs/1904.00962)\.*Preprint*, arXiv:1904\.00962\.
- Zhang et al\. \(2021\)Ruiqing Zhang, Chao Pang, Chuanqiang Zhang, Shuohuan Wang, Zhongjun He, Yu Sun, Hua Wu, and Haifeng Wang\. 2021\.[Correcting Chinese spelling errors with phonetic pre\-training](https://doi.org/10.18653/v1/2021.findings-acl.198)\.In*Findings of the Association for Computational Linguistics: ACL\-IJCNLP 2021*, pages 2250–2261, Online\. Association for Computational Linguistics\.
- Zhang et al\. \(2019a\)Wei Zhang, Feifei Lin, Xiaodong Wang, Zhenshuang Liang, and Zhen Huang\. 2019a\.[Subcharacter chinese–english neural machine translation with wubi encoding](https://doi.org/10.48550/arXiv.1911.02737)\.*arXiv preprint arXiv:1911\.02737*\.
- Zhang et al\. \(2019b\)Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu\. 2019b\.[ERNIE: Enhanced language representation with informative entities](https://doi.org/10.18653/v1/P19-1139)\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1441–1451, Florence, Italy\. Association for Computational Linguistics\.

## Appendix ADataset Statistics

Table[5](https://arxiv.org/html/2608.05167#A1.T5)reports the official splits used for all CLUE / MRC / NER experiments in §[5](https://arxiv.org/html/2608.05167#S5)\. CCD splits followWu et al\. \([2025](https://arxiv.org/html/2608.05167#bib.bib22)\): an IID character split, a long\-tail tier split \(train on the top two frequency tiers, test on the lowest tier\), and an OOV slice within the long\-tail test set restricted to characters that map to\[UNK\]\(or that cannot be represented as a single CJK token\) under the model’s tokenizer\.

Table 5:Dataset statistics\. Official CLUE splits and canonical MRC/NER splits\.
## Appendix BIDS Canonicalization Algorithm

The BabelStone IDS database is multi\-source and admits multiple decompositions for a single character \(typical sources: Unicode CJK\-Unihan, Adobe\-Japan1, Twitter Han, GTBJ, etc\.\)\. Naive ingestion produces non\-deterministic trees that destabilize training\. We therefore apply a deterministic canonicalization procedure that maps each characterxxto exactly one tree𝒯x∈𝒯\\mathcal\{T\}\_\{x\}\\in\\mathcal\{T\}, where𝒯\\mathcal\{T\}is the space of valid IDS parse trees over our component and operator vocabularies\(𝒱c​m​p,𝒱o​p\)\(\\mathcal\{V\}\_\{cmp\},\\mathcal\{V\}\_\{op\}\)\.

#### Filter set\.

Let𝒞x=\{T1,…,Tn\}\\mathcal\{C\}\_\{x\}=\\\{T\_\{1\},\\dots,T\_\{n\}\\\}be the candidate parses for characterxxacross BabelStone sources\. We define a filtering predicateϕ​\(T\)=ϕpua​\(T\)∧ϕops​\(T\)∧ϕcycle​\(T\)∧ϕleaves​\(T\)\\phi\(T\)=\\phi\_\{\\text\{pua\}\}\(T\)\\wedge\\phi\_\{\\text\{ops\}\}\(T\)\\wedge\\phi\_\{\\text\{cycle\}\}\(T\)\\wedge\\phi\_\{\\text\{leaves\}\}\(T\):

- •ϕpua​\(T\)\\phi\_\{\\text\{pua\}\}\(T\):TTcontains no Private Use Area codepoints \(U\+E000–U\+F8FF, U\+F0000–U\+FFFFD, U\+100000–U\+10FFFD\)\.
- •ϕops​\(T\)\\phi\_\{\\text\{ops\}\}\(T\): every internal node label belongs to the standard binary/ternary IDC set𝒱o​pbase=\{⿰,⿱,⿲,⿳,⿴,⿵,⿶,⿷,⿸,⿹,⿺,⿻\}\\mathcal\{V\}\_\{op\}^\{\\text\{base\}\}=\\\{\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿰\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿱\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿲\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿳\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿴\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿵\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿶\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿷\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿸\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿹\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿺\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿻\\end\{CJK\*\}\}\\\}\(the 12 standard IDCs in our paper notation; the implementation also reserves a learnable\[UNK\_OP\]slot, yielding\|𝒱o​p\|=16\|\\mathcal\{V\}\_\{op\}\|\{=\}16after special tokens\)\.
- •ϕcycle​\(T\)\\phi\_\{\\text\{cycle\}\}\(T\): alias resolution ofTTterminates without entering a cycle\.
- •ϕleaves​\(T\)\\phi\_\{\\text\{leaves\}\}\(T\): every leaf is either a single Unicode codepoint or resolves through alias substitution to one\.

Trees that failϕ\\phiare discarded; if all candidates fail we set𝒯x=⊥\\mathcal\{T\}\_\{x\}=\\botand the structure encoder routesxxto the learnable𝐬u​n​k\\mathbf\{s\}\_\{unk\}embedding\.

#### Selection rule\.

Among the surviving candidates𝒞xϕ=\{T:T∈𝒞x,ϕ​\(T\)\}\\mathcal\{C\}\_\{x\}^\{\\phi\}=\\\{T:T\\in\\mathcal\{C\}\_\{x\},\\phi\(T\)\\\}, we apply a strict lexicographic selection over four scores:

Tx⋆=arg⁡minT∈𝒞xϕ⁡\(d​\(T\),¬std​\(T\),\|T\|,π​\(T\)\)T^\{\\star\}\_\{x\}=\\arg\\min\_\{T\\in\\mathcal\{C\}\_\{x\}^\{\\phi\}\}\\\!\\\!\\big\(d\(T\),\\ \\neg\\text\{std\}\(T\),\\ \|T\|,\\ \\pi\(T\)\\big\)\(7\)whered​\(T\)d\(T\)is tree depth \(smaller is preferred\),std​\(T\)\\text\{std\}\(T\)is the predicate that every internal operator lies in the high\-frequency subset𝒱o​pstd=\{⿰,⿱\}\\mathcal\{V\}\_\{op\}^\{\\text\{std\}\}=\\\{\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿰\\end\{CJK\*\}\},\\text\{\\begin\{CJK\*\}\{UTF8\}\{gbsn\}⿱\\end\{CJK\*\}\}\\\},\|T\|\|T\|is the total node count, andπ​\(T\)\\pi\(T\)is the lexicographic operator sequence of the post\-order traversal \(used purely for tie\-breaking determinism\)\. The four\-key ordering exactly mirrors the heuristic priority\(Si et al\.,[2023](https://arxiv.org/html/2608.05167#bib.bib15)\)and the implementation in our public release\.

#### Component recovery for OOV characters\.

A characterxxthat is OOV with respect to the BERT vocabulary is not necessarily OOV with respect to𝒯\\mathcal\{T\}: even whenxxmaps to\[UNK\]for the token pathway, its IDS decomposition may still be available in BabelStone, and its leaf components are typically common radicals shared with high\-frequency characters\. The structure pathway therefore retains a meaningful, gradient\-trained embedding forxxvia the recursive composition ofEc​m​pE\_\{cmp\}vectors, even though the token pathway has collapsed\. This is the mechanism behind the OOV gains in Table[1](https://arxiv.org/html/2608.05167#S5.T1)\.

## Appendix CTree\-MLP as a Compositional Operator

We give a more formal account of the recursive Tree\-MLP encoder than the main text affords, including its parameter count, gradient\-flow characteristics, and the role of hierarchy\.

#### Recursion as fold\.

The Tree\-MLP is a parameterized fold over rooted ordered trees\. LetΣk=𝒱o​p×\(ℝds\)k\\Sigma\_\{k\}=\\mathcal\{V\}\_\{op\}\\times\(\\mathbb\{R\}^\{d\_\{s\}\}\)^\{k\}fork∈\{2,3\}k\\in\\\{2,3\\\}\. The encoder is a pair of functions

fleaf\\displaystyle f\_\{\\text\{leaf\}\}:𝒱c​m​p→ℝds,\\displaystyle:\\mathcal\{V\}\_\{cmp\}\\to\\mathbb\{R\}^\{d\_\{s\}\},\(8\)fk\\displaystyle f\_\{k\}:Σk→ℝds,k∈\{2,3\},\\displaystyle:\\Sigma\_\{k\}\\to\\mathbb\{R\}^\{d\_\{s\}\},\\quad k\\in\\\{2,3\\\},\(9\)extended to𝒯→ℝds\\mathcal\{T\}\\to\\mathbb\{R\}^\{d\_\{s\}\}by the catamorphism

Enc​\(T\)=\{fleaf​\(c\)T=Leaf​\(c\)fk​\(o,Enc​\(T1:k\)\)T=Node​\(o;T1:k\)\\textsc\{Enc\}\(T\)=\\begin\{cases\}f\_\{\\text\{leaf\}\}\(c\)&\\\!\\\!T=\\text\{Leaf\}\(c\)\\\\ f\_\{k\}\(o,\\,\\textsc\{Enc\}\(T\_\{1\{:\}k\}\)\)&\\\!\\\!T=\\text\{Node\}\(o;T\_\{1\{:\}k\}\)\\end\{cases\}\(10\)whereEnc​\(T1:k\)=Enc​\(T1\),…,Enc​\(Tk\)\\textsc\{Enc\}\(T\_\{1\{:\}k\}\)\\,\{=\}\\,\\textsc\{Enc\}\(T\_\{1\}\),\\dots,\\textsc\{Enc\}\(T\_\{k\}\)\. The structural embedding of characterxxis𝐬x=Enc​\(𝒯x\)\\mathbf\{s\}\_\{x\}=\\textsc\{Enc\}\(\\mathcal\{T\}\_\{x\}\)\. This formulation is purely compositional: for any sub\-treeT′T^\{\\prime\}shared between two charactersx,yx,y, the encoder’s intermediate stateEnc​\(T′\)\\textsc\{Enc\}\(T^\{\\prime\}\)is identical, and gradient signal back\-propagates equally to both characters’ losses\. This is the formal basis for the parameter\-sharing claim in §1: orthographically related characters share structure\-encoder parameters by construction\.

#### Concrete form offkf\_\{k\}\.

Following the main\-text Eq\. \(2\)–\(3\), we instantiatefkf\_\{k\}as an operator\-conditioned MLP with a normalized residual path:

𝐡c​a​t\\displaystyle\\mathbf\{h\}\_\{cat\}=\[Eo​p​\(o\);𝐡c1;…;𝐡ck\]∈ℝ\(k\+1\)​ds,\\displaystyle=\[\\,E\_\{op\}\(o\);\\ \\mathbf\{h\}\_\{c\_\{1\}\};\\ \\dots;\\ \\mathbf\{h\}\_\{c\_\{k\}\}\\,\]\\in\\mathbb\{R\}^\{\(k\+1\)d\_\{s\}\},\(11\)𝐡n\\displaystyle\\mathbf\{h\}\_\{n\}=LN​\(GELU​\(𝐖k​𝐡c​a​t\+𝐛k\)\+1k​∑i=1k𝐡ci\),\\displaystyle=\\text\{LN\}\\\!\\left\(\\text\{GELU\}\(\\mathbf\{W\}\_\{k\}\\mathbf\{h\}\_\{cat\}\+\\mathbf\{b\}\_\{k\}\)\+\\tfrac\{1\}\{k\}\\\!\\sum\_\{i=1\}^\{k\}\\mathbf\{h\}\_\{c\_\{i\}\}\\right\),\(12\)with𝐖2∈ℝds×3​ds\\mathbf\{W\}\_\{2\}\\in\\mathbb\{R\}^\{d\_\{s\}\\times 3d\_\{s\}\}and𝐖3∈ℝds×4​ds\\mathbf\{W\}\_\{3\}\\in\\mathbb\{R\}^\{d\_\{s\}\\times 4d\_\{s\}\}\. The mean\-residual1k​∑i𝐡ci\\tfrac\{1\}\{k\}\\sum\_\{i\}\\mathbf\{h\}\_\{c\_\{i\}\}provides a non\-parametric pathway from each child to the parent, ensuring that gradients flow back to the leaves at every depth—a practical concern for trees of depth 6 with∼\\sim5K leaf components\.

#### Parameter count \(additional, beyond BERT backbone\)\.

- •Component table:\|𝒱c​m​p\|⋅ds≈5,000⋅256≈1\.28\|\\mathcal\{V\}\_\{cmp\}\|\\cdot d\_\{s\}\\approx 5\{,\}000\\cdot 256\\approx 1\.28M
- •Operator table:\|𝒱o​p\|⋅ds=16⋅256=4,096\|\\mathcal\{V\}\_\{op\}\|\\cdot d\_\{s\}=16\\cdot 256=4\{,\}096
- •Binary MLP:3​ds⋅ds\+ds=196,8643d\_\{s\}\\cdot d\_\{s\}\+d\_\{s\}=196\{,\}864\(≈\\approx197K\)
- •Ternary MLP:4​ds⋅ds\+ds=262,4004d\_\{s\}\\cdot d\_\{s\}\+d\_\{s\}=262\{,\}400\(≈\\approx263K\)
- •Fusion projection𝐖f\\mathbf\{W\}\_\{f\}:\(d\+ds\)⋅d≈0\.79\(d\+d\_\{s\}\)\\cdot d\\approx 0\.79M \(base\) /1\.311\.31M \(large\)
- •Special embeddings𝐬∅,𝐬u​n​k\\mathbf\{s\}\_\{\\emptyset\},\\mathbf\{s\}\_\{unk\}:2⋅ds=5122\\cdot d\_\{s\}=512

Total≈\\approx2\.5M base /≈\\approx3\.0M large additional parameters, dominated by the component table\. ChineseBERT, by comparison, adds≈\\approx45M parameters for its glyph CNN and Pinyin embeddings\.

#### Why hierarchy matters: a flat\-fusion counterfactual\.

The flat\-fusion ablation \(Table[4](https://arxiv.org/html/2608.05167#S5.T4), row 5\) replaces the recursiveEncwith𝐬x=1\|leaves​\(𝒯x\)\|​∑c∈leaves​\(𝒯x\)Ec​m​p​\(c\)\\mathbf\{s\}\_\{x\}=\\tfrac\{1\}\{\|\\text\{leaves\}\(\\mathcal\{T\}\_\{x\}\)\|\}\\sum\_\{c\\in\\text\{leaves\}\(\\mathcal\{T\}\_\{x\}\)\}E\_\{cmp\}\(c\)\. This destroys two pieces of information: \(i\) operator identity \(which captures spatial layout\), and \(ii\) component ordering \(the difference between 杲 = ⿱\(日,木\) and 杳 = ⿱\(木,日\) is undetectable to a bag\-of\-components encoder\)\. Empirically this loses 17\.7 points on CCD\-OOV\. The full Tree\-MLP recovers both signals\.

## Appendix DPre\-training Objective: Full Derivation

We restate the pre\-training objective with full notation\.

#### WWM corruption\.

WWM produces a corrupted sequenceX~=\(x~1,…,x~T\)\\tilde\{X\}=\(\\tilde\{x\}\_\{1\},\\dots,\\tilde\{x\}\_\{T\}\)fromXXby selecting a 15% mask budget over Jieba word boundaries, then applying the 80/10/10 strategy at each masked positionmm:x~m=\[MASK\]\\tilde\{x\}\_\{m\}=\\texttt\{\[MASK\]\}with probability 0\.8, a random vocabulary token with probability 0\.1, orxmx\_\{m\}unchanged with probability 0\.1\. LetM⊂\{1,…,T\}M\\subset\\\{1,\\dots,T\\\}denote the set of masked positions\.

#### Aligned structural corruption\.

For each masked positionmm, we additionally corrupt the structural index:struct\_idx~m=struct\_idx​\(x~m\)\\tilde\{\\texttt\{struct\\\_idx\}\}\_\{m\}=\\texttt\{struct\\\_idx\}\(\\tilde\{x\}\_\{m\}\)rather thanstruct\_idx​\(xm\)\\texttt\{struct\\\_idx\}\(x\_\{m\}\)\. This ensures the encoder cannot use gold structure as a leak channel for the gold character\. Let𝐬~m=Enc​\(𝒯x~m\)\\tilde\{\\mathbf\{s\}\}\_\{m\}=\\textsc\{Enc\}\(\\mathcal\{T\}\_\{\\tilde\{x\}\_\{m\}\}\)denote the resulting \(possibly corrupted\) structural embedding\.

#### MLM loss\.

ℒMLM=−∑m∈Mlog⁡Pθ​\(xm∣X~,struct\_idx~\),\\mathcal\{L\}\_\{\\text\{MLM\}\}=\-\\sum\_\{m\\in M\}\\log P\_\{\\theta\}\\\!\\left\(x\_\{m\}\\mid\\tilde\{X\},\\tilde\{\\texttt\{struct\\\_idx\}\}\\right\),\(13\)whereθ\\thetacollects the Transformer, structure encoder, fusion, and MLM\-head parameters\.

#### Auxiliary component\-prediction loss\.

For eachm∈Mm\\in M, let\{c1\(m\),…,cLm\(m\)\}\\\{c\_\{1\}^\{\(m\)\},\\ldots,c\_\{L\_\{m\}\}^\{\(m\)\}\\\}be the canonical leaf components of the gold characterxmx\_\{m\}\. We compute a*target*structural embedding from the gold tree,𝐬xmtgt=Enc​\(𝒯xm\)\\mathbf\{s\}^\{\\text\{tgt\}\}\_\{x\_\{m\}\}=\\textsc\{Enc\}\(\\mathcal\{T\}\_\{x\_\{m\}\}\), and pass it through an auxiliary headgψ:ℝds→ℝ\|𝒱c​m​p\|g\_\{\\psi\}:\\mathbb\{R\}^\{d\_\{s\}\}\\to\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{cmp\}\|\}\. The auxiliary loss is

ℒaux=−∑m∈M1Lm​∑l=1Lmlog⁡σ​\(gψ​\(𝐬xmtgt\)\)​\[cl\],\\mathcal\{L\}\_\{\\text\{aux\}\}=\-\\sum\_\{m\\in M\}\\frac\{1\}\{L\_\{m\}\}\\sum\_\{l=1\}^\{L\_\{m\}\}\\log\\sigma\\\!\\left\(g\_\{\\psi\}\(\\mathbf\{s\}^\{\\text\{tgt\}\}\_\{x\_\{m\}\}\)\\right\)\\\!\[c\_\{l\}\],\(14\)whereσ​\(⋅\)​\[c\]\\sigma\(\\cdot\)\[c\]denotes the softmax probability of classcc\.

#### Information\-leak prevention\.

Crucially,𝐬xmtgt\\mathbf\{s\}^\{\\text\{tgt\}\}\_\{x\_\{m\}\}is supplied*only*to the auxiliary headgψg\_\{\\psi\}; the Transformer input at positionmmuses𝐬~m\\tilde\{\\mathbf\{s\}\}\_\{m\}\. This ensures the MLM prediction is forced to recover the gold character from non\-gold structural context, preserving the difficulty of the MLM task while still using structural supervision to shape the structural embedding space\.

#### Combined loss\.

ℒ​\(θ,ψ\)=ℒMLM​\(θ\)\+λ⋅ℒaux​\(θ,ψ\),λ=0\.1\.\\mathcal\{L\}\(\\theta,\\psi\)=\\mathcal\{L\}\_\{\\text\{MLM\}\}\(\\theta\)\+\\lambda\\cdot\\mathcal\{L\}\_\{\\text\{aux\}\}\(\\theta,\\psi\),\\quad\\lambda=0\.1\.\(15\)The ablation in Table[4](https://arxiv.org/html/2608.05167#S5.T4)confirms thatλ\\lambdacontributes a modest but real lift on CLUE \(\+0\.22\+0\.22\) and a larger lift on CCD\-OOV \(\+1\.5\+1\.5\), justifying the defaultλ=0\.1\\lambda\{=\}0\.1\.

## Appendix EDetailed Hyperparameters

Table 6:Full pre\-training and fine\-tuning hyperparameters\.
## Appendix FAuxiliary Figure Recipes

For reproducibility, we provide rendering recipes for three additional analyses that we recommend including in extended versions of this paper or a companion technical report\.

#### Figure F\.1: UMAP of structural embeddings\.

Project the learned structural embeddings\{𝐬x:x∈𝒱c​h​a​r\}\\\{\\mathbf\{s\}\_\{x\}:x\\in\\mathcal\{V\}\_\{char\}\\\}to 2D via UMAP and color by top\-level IDS operator\. We useumap\-learnwithn\_neighbors=30,min\_dist=0\.10,metric=’cosine’, sampled toN=2,000N=2\{,\}000characters stratified by operator\. The expected outcome is 8 visually\-distinct clusters corresponding to the 8 most frequent operators, with cluster purity≥0\.90\\geq 0\.90measured by 1\-NN classification on operator labels\.

#### Figure F\.2: Per\-operator gain radar\.

For each operatoro∈𝒱o​pstdo\\in\\mathcal\{V\}\_\{op\}^\{\\text\{std\}\}, partition the CCD long\-tail OOV slice by the top\-level operator of the gold character, and compute the Structure\-F1 difference between CNM\-BERT and the strongest baseline \(ChineseBERT\)\. Plot as a radar with 8 axes\. The expected outcome is a polygon that strictly dominates ChineseBERT on every axis, with the largest gains on operators ⿲ and ⿳ \(ternary\), where layout structure carries the most disambiguating information\.

#### Figure F\.3: Pre\-training loss curves\.

Export from W&B both training MLM loss and held\-out auxiliary\-component\-prediction accuracy across 1M steps for CNM\-BERT and BERT\-wwm under the controlled recipe\. The expected outcome is \(i\) MLM perplexity within 5% of BERT throughout, and \(ii\) auxiliary accuracy that rises from∼\\sim5% \(chance\) to∼\\sim70% by 200K steps, confirming the structural pathway is genuinely learning rather than collapsing to a constant\.