Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

# Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Source: [https://arxiv.org/html/2607.05691](https://arxiv.org/html/2607.05691)
###### Abstract

Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte\-pair encoding \(BPE\) from natural language with little scrutiny\. In natural language, BPE’s principal alternative, Unigram\-LM, is known to build structurally different vocabularies\. Whether that contrast survives in chemistry was open: the complete glyph base already covers every conformant molecule, so the learned pieces add compression rather than coverage, and a tiny alphabet under hard valence constraints could drive two frequency\-based algorithms to converge\. We report a controlled comparison of BPE and Unigram\-LM over a fixed 165\-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies \(diverse, drug\-like, natural\-products\) and both pre\-tokenization boundary policies\. The two do not converge\. In all 22 matched conditions they build*near\-disjoint*subword vocabularies: cross\-algorithm Jaccard overlap on the learned pieces above the shared base never exceeds0\.1610\.161, and at most0\.050\.05once weighted toward the high\-frequency pieces a model updates most\. Unigram\-LM also segments held\-out molecules into2929–41%41\\%more tokens; the arms largely agree on*where*to cut but not*how deeply*, so BPE’s segmentation is a strict coarsening of Unigram\-LM’s on8080–99%99\\%of molecules\. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale, past where embeddings remain learnable; only token\-frequency imbalance attenuates in magnitude, shrinking with vocabulary size and most on the natural\-products corpus, without closing\. The subword algorithm is therefore a modeling decision, not a free default\. We release all trained tokenizers and per\-condition measurements\.

![Refer to caption](https://arxiv.org/html/2607.05691v1/x1.png)Figure 1:Nicotine and serotonin under the two algorithms’V=1024V\{=\}1024vocabularies \(molecules in rows, algorithms in columns\)\. Hue marks the algorithm \(blue Smirk\-GPE/BPE, orange Unigram\-LM\); darker shading marks a larger piece, on the same scale for both\. BPE builds a few large pieces spanning whole rings and chains; Unigram\-LM stays near\-atomic, a scatter of small beads, emitting more tokens per molecule \(\+5\+5and\+7\+7here, a wider gap than the∼\\simone\-third average across held\-out molecules\)\.## 1Introduction

Every chemical language model that operates on a SMILES string\[[53](https://arxiv.org/html/2607.05691#bib.bib53)\]begins with a tokenizer that maps it to integer IDs, fixing the vocabulary the model embeds, the granularity at which it operates, and hence the effective sequence length\. It is a foundational design choice\[[2](https://arxiv.org/html/2607.05691#bib.bib2),[16](https://arxiv.org/html/2607.05691#bib.bib16)\], almost always made by default: SMILES tokenizers inherit byte\-pair encoding \(BPE\)\[[43](https://arxiv.org/html/2607.05691#bib.bib43)\]from natural\-language practice, and its chemistry descendants \(SPE, APE, Smirk\-GPE\) are all BPE variants\. The principal alternative, Unigram\-LM\[[19](https://arxiv.org/html/2607.05691#bib.bib19)\], sees occasional chemistry use but has never been compared to BPE at matched conditions on a fixed chemistry\-grammatical base\.

That choice was long overshadowed by a more basic axis, vocabulary*coverage*: whether every string gets a token or part falls through to\[UNK\]\.Wadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]found that coverage, not the subword scheme, separates chemistry tokenizers downstream, but only across heterogeneous bases at large native vocabularies, never isolating the subword\-*algorithm*axis\. Smirk\[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]closes coverage with a complete165165\-token OpenSMILES base\[[15](https://arxiv.org/html/2607.05691#bib.bib15)\],111FollowingWadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]:158158OpenSMILES glyphs plus seven special tokens \(including\[UNK\]\)\. We write “glyph” for the158158chemistry\-grammatical pieces\.emitting no\[UNK\]on conformant input\. Our study begins there: pushing the vocabulary into the small regime, we isolate the subword algorithm, still inherited from natural language, as the design choice under study\.

The inheritance is not obviously safe\. Chemistry SMILES is statistically unlike language, with a tiny base alphabet and strings under hard structural constraints, so natural\-language priors may not transfer, and the ones that do cut both ways\. Priors point toward divergence:Bostrom and Durrett \[[5](https://arxiv.org/html/2607.05691#bib.bib5)\]on natural language and the one prior chemistry head\-to\-head both report it, though under confounds we revisit in §[2\.2](https://arxiv.org/html/2607.05691#S2.SS2)\. Yet a convergence null stays live: in natural language a key segmentation statistic \(whole\-pretoken absorption\) converges across the two algorithms at largeVV\[[36](https://arxiv.org/html/2607.05691#bib.bib36)\], and in chemistry both select pieces by frequency over a small,158158\-glyph alphabet where valence and ring\-closure constraints sharply concentrate which glyph sequences occur; that concentration could drive them to absorb the same high\-frequency substructures and converge\.

We therefore ask:*on chemistry SMILES, holding everything but the algorithm fixed, does the choice between BPE and Unigram\-LM still shape the structure of the resulting tokenizer, or does chemistry’s small, valence\-constrained alphabet drive the two to converge on the same vocabulary?*We find that the algorithm shapes the tokenizer\. At a matched condition \(same corpus, vocabulary size, and boundary policy\), BPE and Unigram\-LM build measurably different vocabularies \(Figure[1](https://arxiv.org/html/2607.05691#S0.F1)\) in three senses:*which*pieces a vocabulary contains \(membership\),*how finely*it segments held\-out molecules \(granularity\), and*how unevenly*token usage is distributed \(distribution\); two further results fix its scope\. All five are stable properties of the algorithm pair, holding across corpus typology, boundary policy, and vocabulary size, with only the distribution gap’s magnitude attenuating on one corpus \(Section[4](https://arxiv.org/html/2607.05691#S4)\)\.

This is a tokenizer\-level study: we train no language models and make no claim about which algorithm is better downstream\. It is the precondition for that comparison, establishing that the two do not yield interchangeable vocabularies, dislodging the field’s inherited default\. Our contributions are fourfold: \(i\) the first controlled, chemistry\-grammatical, small\-vocabulary comparison of BPE and Unigram\-LM across a corpus\-typology range and both boundary policies; \(ii\) chemistry\-side measurements, previously unreported for SMILES, of three of the four mechanism diagnostics carried from the natural\-language literature \(dead\-zone surplus, scaffold fraction, and whole\-pretoken absorption; the remaining one, segmentation entropy, is intrinsic to Unigram\-LM\), alongside a structural\-character and non\-canonicity battery that localizes the divergence; \(iii\) two results fixing its scope, that the near\-disjoint vocabularies nonetheless parse compatibly and that the divergence persists at8×8\\timesthe headline vocabulary, past the learnable regime; and \(iv\) all trained tokenizers and per\-condition measurements, released for re\-analysis\.

## 2Background and related work

### 2\.1Chemistry SMILES tokenizers

Chemistry\-side tokenizers are best organized by how each handles*coverage*\(§[1](https://arxiv.org/html/2607.05691#S1)\)\.*Atom\-wise*regex tokenizers\[[42](https://arxiv.org/html/2607.05691#bib.bib42)\]split on atom symbols and treat each bracketed atom as one token; DeepChem’s implementation\[[35](https://arxiv.org/html/2607.05691#bib.bib35)\], popularized by ChemBERTa\[[7](https://arxiv.org/html/2607.05691#bib.bib7)\], is widely used\. Because a bracketed atom jointly encodes element, isotope, chirality, charge, and hydrogen count, the OpenSMILES bracketed\-atom space exceeds2828trillion permutations\[[51](https://arxiv.org/html/2607.05691#bib.bib51)\], far beyond any fixed vocabulary; the unrepresented remainder becomes\[UNK\]\.*SMILES Pair Encoding*\(SPE\)\[[25](https://arxiv.org/html/2607.05691#bib.bib25)\]and*Atom Pair Encoding*\(APE\)\[[23](https://arxiv.org/html/2607.05691#bib.bib23)\]are BPE adapted to SMILES with atom\-level initialization, learning∼\\sim3,000–5,300 tokens; both inherit this bracketed\-atom\-as\-token convention and still emit\[UNK\]\[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]; for SPE this reaches∼\\sim19% of tokens on MoleculeNet\[[54](https://arxiv.org/html/2607.05691#bib.bib54)\]and∼\\sim50% on tmQM\[[4](https://arxiv.org/html/2607.05691#bib.bib4)\]\.*Atom\-in\-SMILES*\[[50](https://arxiv.org/html/2607.05691#bib.bib50)\]replaces atomic tokens with environment\-aware tokens; it is bijective but has an implicit, open\-ended vocabulary\.*Smirk*\[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]is a regex glyph decomposition of OpenSMILES over a165165\-token base; any OpenSMILES\-conformant string decomposes into these glyphs with no\[UNK\]\.*Smirk\-GPE*is BPE on top of that glyph base\. A separate line of work sets coverage aside: rather than constructing a vocabulary, it post\-processes SMILES token sequences for compression \(e\.g\. trie\-based refinement and transition\-graph filtering\[[34](https://arxiv.org/html/2607.05691#bib.bib34)\]\), building on the same frequency\-driven subword primitives whose structure we characterize here\.

We study SMILES throughout; alternative notations such as SELFIES\[[18](https://arxiv.org/html/2607.05691#bib.bib18)\], whose strings always decode to valid molecules, and DeepSMILES\[[31](https://arxiv.org/html/2607.05691#bib.bib31)\]vary the representation rather than the subword algorithm and are out of scope\. That notation choice is consequential \(SMILES\-based generators outperform SELFIES on distribution\-learning metrics, and the invalid strings SMILES admits act as implicit quality control\[[44](https://arxiv.org/html/2607.05691#bib.bib44)\]\), yet orthogonal to our algorithm contrast\.

### 2\.2BPE versus Unigram\-LM

BPE and Unigram\-LM construct their vocabularies in opposite directions \(Appendix Fig\.[6](https://arxiv.org/html/2607.05691#A1.F6)\)\. BPE\[[43](https://arxiv.org/html/2607.05691#bib.bib43)\]builds bottom\-up, greedily merging the most\-frequent adjacent symbol pair until the target size is reached; it is deterministic given the corpus and target size\. Unigram\-LM\[[19](https://arxiv.org/html/2607.05691#bib.bib19)\]builds top\-down, fitting a unigram probability distribution over a large seed vocabulary by expectation\-maximization and pruning the pieces whose removal least hurts corpus likelihood; it provides per\-segmentation probabilities and supports subword regularization at training time\. In chemistry it appears in models such as BARTSmiles\[[6](https://arxiv.org/html/2607.05691#bib.bib6)\]and ReactionT5 / CompoundT5\[[38](https://arxiv.org/html/2607.05691#bib.bib38)\]\. \(WordPiece\[[41](https://arxiv.org/html/2607.05691#bib.bib41)\]builds bottom\-up like BPE, differing only in its merge criterion, and is out of scope\.\)

Both algorithms run downstream of a pre\-tokenizer, the step that first splits the raw string into*pretokens*\(the chunks within which subword merges stay confined\)\. In natural language that boundary is the highest\-impact design choice across studies, above vocabulary size and corpus\[[52](https://arxiv.org/html/2607.05691#bib.bib52),[36](https://arxiv.org/html/2607.05691#bib.bib36)\]\. Recent methods relax it for compression\[[27](https://arxiv.org/html/2607.05691#bib.bib27),[40](https://arxiv.org/html/2607.05691#bib.bib40)\]\. We treat its chemistry analog, whether merges may cross the bracketed\-atom boundary, as a first\-class axis \(§[3\.3](https://arxiv.org/html/2607.05691#S3.SS3)\)\.

Prior work has measured*where*BPE and Unigram\-LM diverge, each study supplying a diagnostic we carry to chemistry\.Bostrom and Durrett \[[5](https://arxiv.org/html/2607.05691#bib.bib5)\]found Unigram\-LM matches or beats BPE on natural language for structural reasons, though the advantage varied by an order of magnitude across their two corpora; that structural account has three legs: better morphological alignment, less over\-merging of frequent affixes, and a smaller dead\-zone surplus \(BPE carries∼\\sim1,500 more intermediate\-merge tokens that fire too rarely to train an embedding\)\. Morphological alignment has no gold\-standard SMILES segmentation to score against, so Bostrom’s better\-alignment metric does not transfer; the over\-merging leg becomes our granularity contrast \(fertility\), and the dead\-zone surplus is a symmetric cross\-arm count we measure directly \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\)\.Lian et al\. \[[26](https://arxiv.org/html/2607.05691#bib.bib26)\]quantify the BPE\-side fraction as6\.07%6\.07\\%“scaffold tokens” for the LLaMA\-32K tokenizer\[[49](https://arxiv.org/html/2607.05691#bib.bib49)\]and show that removing them yields downstream gains at 6\.7B parameters\.Reddy et al\. \[[36](https://arxiv.org/html/2607.05691#bib.bib36)\]report a related diagnostic, the fraction of pretokens absorbed as single tokens, and find it “remarkably high across all algorithms,” increasing with vocabulary size\.

The single prior chemistry\-side head\-to\-head isTemizer et al\. \[[47](https://arxiv.org/html/2607.05691#bib.bib47)\], who learned BPE, WordPiece, and Unigram vocabularies on∼\\sim2\.3M ChEMBL\[[55](https://arxiv.org/html/2607.05691#bib.bib55)\]SMILES atV∈\{8​K,16​K,32​K\}V\\in\\\{8\\text\{K\},16\\text\{K\},32\\text\{K\}\\\}\. They report that BPE and WordPiece vocabularies overlap by∼\\sim80% while Unigram overlaps the others by at most∼\\sim47%, that the overlap shrinks asVVgrows, and that Unigram “chemical words” run longer and are less often RDKit\-valid\. Their study establishes that a contrast is*visible*\. But it runs on raw SMILES \(no fixed chemistry base, confounding the algorithm with the absence of an atomic level\) and one drug\-like corpus under a single pre\-tokenization policy, leaving open whether the contrast is algorithmic, corpus\-stable, or boundary\-robust \(the three questions we test\)\. A fourth confound, largeVV, places it outside the learnable small\-vocabulary regime \(§[2\.3](https://arxiv.org/html/2607.05691#S2.SS3)\)\.

### 2\.3Corpus typology and the small\-vocabulary regime

The chemistry analog of the English\-versus\-Japanese axis ofBostrom and Durrett \[[5](https://arxiv.org/html/2607.05691#bib.bib5)\]is not morphology but alphabet diversity×\\timestopology×\\timesfunctional\-group distribution\. We span it with three headline corpora:PubChem\[[17](https://arxiv.org/html/2607.05691#bib.bib17)\]\(diverse\),ZINC\-22\[[48](https://arxiv.org/html/2607.05691#bib.bib48)\]\(drug\-like\), andCOCONUT\[[45](https://arxiv.org/html/2607.05691#bib.bib45)\]\(natural\-products\), plus the 1% Sample of Enamine’s make\-on\-demandREAL\-Space\[[12](https://arxiv.org/html/2607.05691#bib.bib12),[51](https://arxiv.org/html/2607.05691#bib.bib51)\], which serves only as an anchor \(per\-corpus characteristics in Table[1](https://arxiv.org/html/2607.05691#S3.T1)\)\. Combinatorially enumerated from a fixed building\-block set, REAL\-Space has a narrow alphabet, and even a 1% sample is large enough to exhaust the available merges, making it a*super\-saturated*drug\-like probe\.

Vocabulary size is constrained by embedding learnability: above roughly1,0001\{,\}000tokens learned pieces can fire too rarely to train embeddings, a token\-usage dead\-zone \(the regime boundary, not the cross\-arm surplus of §[2\.2](https://arxiv.org/html/2607.05691#S2.SS2)\) thatWadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]report on the super\-saturated REAL\-Space \(their SI Fig\. S1\)\. The small\-vocabulary regime is therefore where the choice between the two algorithms is most consequential: with slots scarce, each must commit to*which*high\-frequency patterns to absorb \(§[1](https://arxiv.org/html/2607.05691#S1)\)\. We track each condition against the learnability bar ofGowda and May \[[11](https://arxiv.org/html/2607.05691#bib.bib11)\]\(§[3\.5](https://arxiv.org/html/2607.05691#S3.SS5)\)\.

## 3Methods

### 3\.1Experimental design: the grid

Each trained tokenizer \(one*arm*\) is an \(algorithm, corpus,VV, boundary\) tuple\. A*condition*, equivalently a grid cell, is a \(corpus,VV, boundary\) point; in a*matched*condition both arms are trained, so the only cross\-arm difference is the selection algorithm\. One control and two robustness axes keep that contrast clean: \(i\) fixing the base alphabet \(§[3\.3](https://arxiv.org/html/2607.05691#S3.SS3)\) removes coverage as a confound; \(ii\) spanning three corpus typologies \(§[3\.2](https://arxiv.org/html/2607.05691#S3.SS2)\) separates an algorithmic effect from a single\-corpus artifact; and \(iii\) varying the boundary policy \(§[3\.3](https://arxiv.org/html/2607.05691#S3.SS3)\) separates that effect from a bracket\-boundary artifact\. Stability of the membership and granularity contrasts across these last two axes is what we report as robustness;VVchecks scale\.

The headline grid is2​algorithm×3​corpus×3​V×2​boundary=362\\,\\text\{algorithm\}\\times 3\\,\\text\{corpus\}\\times 3\\,V\\times 2\\,\\text\{boundary\}=36tokenizers \(1818matched conditions\), withV∈\{256,512,1024\}V\\in\\\{256,512,1024\\\}\. To it we add a four\-tokenizer PubChemV=2048V\{=\}2048sensitivity block and a four\-tokenizer REAL\-SpaceV=1024V\{=\}1024anchor block \(two matched conditions each\), for4444trained tokenizers, the two arms of each of2222matched conditions\. At a single coordinate no pair can be formed: on ZINC\-22 atV=2048V\{=\}2048the narrow alphabet pushes both arms past the learnable regime, the Unigram\-LM arm so far that we leave it untrained \(§[4](https://arxiv.org/html/2607.05691#S4)\)\. We train only the BPE arm there, to demonstrate that ceiling, and report it single\-arm; its two boundary variants are additional to the4444\.

### 3\.2Corpora and preprocessing

All four corpora pass through the same policy: RDKit\[[22](https://arxiv.org/html/2607.05691#bib.bib22)\]isomeric canonicalization, exact\-string deduplication, and a base\-conformance filter \(procedure, drop rates, and pinned versions in Appendix[A\.1](https://arxiv.org/html/2607.05691#A1.SS1)\)\. We pin the RDKit build, since canonical SMILES is toolkit\-dependent\[[30](https://arxiv.org/html/2607.05691#bib.bib30)\], and deduplicate because both algorithms select by frequency\. The policy is otherwise narrower than a standard pipeline: we do*not*salt\-strip \(real input carries multi\-component salts and the\.disconnection symbol\), neutralize charges \(collapsing the charged bracketed atoms\[O\-\],\[NH3\+\]central to the boundary axis\), or cap heavy\-atom count \(clipping the large COCONUT molecules that define its typology\)\.

Table 1:The four corpora: three headline typologies plus the REAL\-Space anchor\. Counts are post\-preprocessing totals, after any subsampling and before the 5% held\-out split\.A final filter enforces the study’s premise: the Smirk base covers any OpenSMILES\-conformant string with no\[UNK\], but RDKit’s non\-Kekulé canonicalization can emit off\-base atoms \(aromatic Si, Te\)\. We drop any molecule the bare base cannot cover without\[UNK\], closing each corpus under the base; left in, these atoms would contaminate both vocabularies and perturb the overlap\. Only PubChem loses molecules \(under0\.004%0\.004\\%\); the rest are fully conformant\.

The corpora span∼\\sim740K \(COCONUT, total\) to∼\\sim136M molecules \(Table[1](https://arxiv.org/html/2607.05691#S3.T1)\); each carries a deterministic held\-out test split, used for every held\-out\-evaluated measurement, leaving COCONUT∼\\sim702K molecules to train on\.

### 3\.3Tokenizer training

Both algorithms train from Smirk’s165165\-token base behind a shared pre\-tokenization module, so each consumes an identical stream of glyph\-id words\. The BPE arm is Smirk’sGpeTrainer; the Unigram\-LM arm is a sibling trainer we add that delegates to HuggingFacetokenizers’UnigramTrainer\[[19](https://arxiv.org/html/2607.05691#bib.bib19),[28](https://arxiv.org/html/2607.05691#bib.bib28)\]\. Both run at their reference defaults \(with three by\-design exceptions, none of them tuning, Appendix[A\.2](https://arxiv.org/html/2607.05691#A1.SS2)\), so neither is hand\-tuned to favor the contrast\.

The base is installed as length\-1 pieces, so both arms target the sameVV\(Table[A1](https://arxiv.org/html/2607.05691#A1.T1)\)\. A*matched*condition matches this*target*VV; the arms need not realize it identically\. BPE keeps the full base and fills the remaining budget with merges; Unigram\-LM seeds a large pool and prunes back, halting at or below target\. The two realize near\-equal sizes on diverse corpora, but on a narrow alphabet at largerVVthe Unigram\-LM arm falls well below target\. Realized per\-arm sizes are reported per condition \(Table[A13](https://arxiv.org/html/2607.05691#A6.T13); effect on overlap in §[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\)\. Because several measurements carry no confidence interval, every tokenizer is also trained twice and asserted byte\-identical\.

Theboundary policyis the one knob we vary on that shared pre\-tokenization module: under*no\-merge\-brackets*\(NMB\) a bracketed atom is opaque and merges cannot cross it, while under*merge\-brackets*\(MB\) the boundary is permeable\. MB is Smirk\-GPE’s default;Wadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]also trained an NMB variant, reporting “similar results” at their single merge\-exhausted vocabulary, a finding we test systematically across the small\-VVregime\.

### 3\.4What we measure

We compute seven structural quantities per condition: some from the realized vocabularies, the dead\-zone audit from the training corpus, and the rest on the single per\-corpus held\-out test split\. The first three are the*direct cross\-arm contrasts*, one per sense of a “different vocabulary” \(the remaining four are mechanism diagnostics, gated by the learnability bar and defined in §[3\.5](https://arxiv.org/html/2607.05691#S3.SS5)\):

- •Membership: vocabulary overlap \(Jaccard\)\.The cross\-arm overlap on the multi\-glyph subword set is J=\|𝒱BPEmulti∩𝒱ULmulti\|\|𝒱BPEmulti∪𝒱ULmulti\|,J=\\frac\{\|\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{\\text\{BPE\}\}\\cap\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{\\text\{UL\}\}\|\}\{\|\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{\\text\{BPE\}\}\\cup\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{\\text\{UL\}\}\|\},\(1\)writing𝒱amulti\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{a\}for the multi\-glyph pieces \(length≥2\\geq 2glyphs\) armaalearns above the shared base, as distinct from the full vocabulary𝒱a\\mathcal\{V\}\_\{a\}and the target sizeVV\(a scalar\)\. We exclude the shared base, which both arms retain in full and which would otherwise dominate the overlap at smallVV\(≈0\.5\\approx 0\.5atV=256V\{=\}256\) while saying little about what each algorithm selects\.
- •Granularity: fertility\.The standard subwords\-per\-word tokenizer metric\[[37](https://arxiv.org/html/2607.05691#bib.bib37)\], here mean held\-out tokens per molecule, with glyphs per token as the compression ratio \(both tabulated per condition with CIs in Appendix[F\.2](https://arxiv.org/html/2607.05691#A6.SS2)\); we report the absolute gap\|Δ​f\|\|\\Delta f\|and, as the headline contrast, the relative gaprel​\|Δ​f\|\\mathrm\{rel\}\|\\Delta f\|, \|Δ​f\|=\|fBPE−fUL\|,rel​\|Δ​f\|=\|Δ​f\|f¯,f¯=12​\(fBPE\+fUL\)\.\|\\Delta f\|=\|f\_\{\\text\{BPE\}\}\-f\_\{\\text\{UL\}\}\|,\\qquad\\mathrm\{rel\}\|\\Delta f\|=\\frac\{\|\\Delta f\|\}\{\\bar\{f\}\},\\qquad\\bar\{f\}=\\tfrac\{1\}\{2\}\(f\_\{\\text\{BPE\}\}\+f\_\{\\text\{UL\}\}\)\.\(2\)
- •Distribution: token\-frequency imbalance\.The divergence from uniform\[[11](https://arxiv.org/html/2607.05691#bib.bib11)\]is D=12​∑i\|pi−1/V\|,D=\\tfrac\{1\}\{2\}\\sum\_\{i\}\\big\|p\_\{i\}\-1/V\\big\|,\(3\)withpip\_\{i\}the held\-out relative frequency of tokenii; the cross\-arm contrast is the gap\|Δ​D\|=\|DBPE−DUL\|\|\\Delta D\|=\|D\_\{\\text\{BPE\}\}\-D\_\{\\text\{UL\}\}\|, and we reportDDwith normalized Shannon entropyη=H/log⁡V\\eta=H/\\log Vand Rényi efficiency atα=2\.5\\alpha\{=\}2\.5\[[56](https://arxiv.org/html/2607.05691#bib.bib56)\]as within\-family diagnostics\[[39](https://arxiv.org/html/2607.05691#bib.bib39),[8](https://arxiv.org/html/2607.05691#bib.bib8)\]\(per\-condition values in Appendix[F\.4](https://arxiv.org/html/2607.05691#A6.SS4)\)\.

#### Membership robustness\.

Three companions test whether this membership overlap is an artifact of counting, crossing two axes into a2×22\\times 2:*weighting*\(unweighted vs\. frequency\-weighted\) by*masking*\(all multi\-glyph pieces vs\. structural pieces only\)\. The frequency\-weighted variant weights each subwordxxby its held\-out emission count \(normalized per arm, with a bootstrap CI\), reflecting the pieces a model actually sees:

Jw=∑xmin⁡\(wBPE​\(x\),wUL​\(x\)\)∑xmax⁡\(wBPE​\(x\),wUL​\(x\)\)\.J\_\{\\mathrm\{w\}\}=\\frac\{\\sum\_\{x\}\\min\\\!\\big\(w\_\{\\text\{BPE\}\}\(x\),w\_\{\\text\{UL\}\}\(x\)\\big\)\}\{\\sum\_\{x\}\\max\\\!\\big\(w\_\{\\text\{BPE\}\}\(x\),w\_\{\\text\{UL\}\}\(x\)\\big\)\}\.\(4\)The structural variantsJstructJ\_\{\\mathrm\{struct\}\}andJw,structJ\_\{\\mathrm\{w,struct\}\}recomputeJJandJwJ\_\{\\mathrm\{w\}\}over*structural*pieces only \(dropping bracket\-internal pieces, those that only ever occur inside a bracketed atom\), isolating cross\-pretoken merges; together they separate a lowJwJ\_\{\\mathrm\{w\}\}into genuine high\-frequency structural disagreement versus an artifact of Unigram\-LM’s bracket\-internal pieces \(Appendix[F](https://arxiv.org/html/2607.05691#A6)gives the bracket\-internal definition and the structural renormalization\)\.

#### Granularity, read positionally\.

Because both arms segment the*same*glyph stream, both draw their cuts from the same inter\-glyph positions, so beyond*how many*tokens each emits we can ask*where*its cuts fall relative to the other arm’s\. At each position the two arms either agree to cut, agree to merge,*nest*\(Unigram\-LM cuts where BPE merges\), or*conflict*\(BPE cuts where Unigram\-LM merges\)\. We summarize this as the boundary JaccardJ∂J\_\{\\partial\}\(agreement over cut positions, so agree\-cut over agree\-cut\+\+nest\+\+conflict\), the nest and conflict rates \(over all positions\), the fraction of molecules with zero conflict \(those whose BPE parse is a strict coarsening of Unigram\-LM’s\), and, for the conflicts that remain, their localization by substructure class \(heteroatom, unsaturated carbon, saturated carbon\)\. This adds no new quantity: the nest rate is the fertility gap counted cut by cut\. Per\-condition values are in Appendix[F\.3](https://arxiv.org/html/2607.05691#A6.SS3)\.

### 3\.5Learnability gate and mechanism diagnostics

The comparison is posed where token embeddings are learnable, so we certify that regime per condition\. For each arm we compute the clearancecnc\_\{n\}, the fraction of its*learned*vocabulary pieces \(those above the shared base\) firing at leastnntimes in the training corpus, and apply the learnability barFp,n:cn≥pF\_\{p,n\}\\\!:c\_\{n\}\\geq p, withF95%,100F\_\{95\\%,100\}\(c100≥0\.95c\_\{100\}\\geq 0\.95\) the headline\. We applyGowda and May \[[11](https://arxiv.org/html/2607.05691#bib.bib11)\]’s bar to the learned vocabulary because the fixed165165\-token base is identical across both arms and chosen by neither algorithm\. This is an undertrained\-token audit, the training\-corpus analog of the weight\-based audits on deployed LLMs\[[21](https://arxiv.org/html/2607.05691#bib.bib21)\]and one of the low\-cost tokenizer diagnosticsAlqahtani et al\. \[[2](https://arxiv.org/html/2607.05691#bib.bib2)\]advocate, applied here to chemistry: it flags pieces that fire too rarely in training to learn a reliable embedding \(Appendix[A\.4](https://arxiv.org/html/2607.05691#A1.SS4); the per\-armcnc\_\{n\}sweep is in Appendix[F\.5](https://arxiv.org/html/2607.05691#A6.SS5), from which the bar at anyppfollows\)\. A condition whose arm fails the bar is flagged and never pooled into a cross\-corpus reading\. Because the bar is an absolute count, a failure indexes corpus size, not typology\. The same size/typology confound colors COCONUT’s one corpus\-dependent contrast, the distribution gap, which the size\-matched probe \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\) resolves as typological\. The bar gates only the rare\-token\-tail diagnostics, not the overlap and fertility contrasts; its clearancecnc\_\{n\}also supplies the dead\-zone surplus that opens the diagnostics below\.

The remaining four are*mechanism diagnostics*:

- •Dead\-zone surplus\.The cross\-arm difference in clearance at the bar’s countnn,Δ​cn=cnBPE−cnUL\\Delta c\_\{n\}=c^\{\\text\{BPE\}\}\_\{n\}\-c^\{\\text\{UL\}\}\_\{n\}, the chemistry analog of the dead\-zone surplus ofBostrom and Durrett \[[5](https://arxiv.org/html/2607.05691#bib.bib5)\]\.
- •Whole\-pretoken absorption\.The fraction of pretokens absorbed as a single token\[[36](https://arxiv.org/html/2607.05691#bib.bib36)\]\.
- •Scaffold fraction\.The BPE\-side fraction of intermediate\-merge “scaffold” tokens\[[26](https://arxiv.org/html/2607.05691#bib.bib26)\], which we identify on the trained vocabulary as pieces whose end\-of\-training standalone frequency falls below the last committed merge’s candidate frequency \(a static analog of their per\-iteration removal rule, hence a conservative lower bound\); zero for Unigram\-LM by construction\.
- •Segmentation entropy\.Mean Shannon entropy of the fitted Unigram\-LM’s distribution over a held\-out molecule’s segmentations\[[19](https://arxiv.org/html/2607.05691#bib.bib19)\], normalized per glyph \(total entropy over total glyphs, the reportedΔ​Hg\\Delta H\_\{g\}gap\); zero for BPE by construction\.

#### Quantifying and reading the measurements\.

Bootstrap CIs \(95%,10001000resamples, molecule as the resample unit\) accompany the held\-out\-evaluated quantities; the exact\-set quantities \(JJ,JstructJ\_\{\\mathrm\{struct\}\}, scaffold fraction\) carry no CI\. We read each measurement as an effect size against the scale of measurement noise, not against a pass/fail threshold: for membership, the cross\-arm overlap is scaled against the internal self\-overlap ceiling \(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\) and the prior\-work overlap values we cite \(§[2\.2](https://arxiv.org/html/2607.05691#S2.SS2)\), never gated on a fixed value\. The one measured threshold we apply is on the distribution sense, whose gap is small enough to check against the corpus\-draw noise floor estimated from the subsample redraws \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\)\.

### 3\.6Sensitivity analysis and structural probes

Beyond the headline grid we run asensitivity analysis: each training hyperparameter is swept one\-factor\-at\-a\-time across a ladder bracketing its reference default \(Table[A1](https://arxiv.org/html/2607.05691#A1.T1)\), with four pairwise interactions and four structural probes \(Appendix[A\.3](https://arxiv.org/html/2607.05691#A1.SS3)\)\. Because BPE imposes no piece\-length cap, its only free training knob is the merge frequency, so the sweep is necessarily Unigram\-LM\-weighted\. To stay tractable the battery runs on a fixed representative subsample while the headline grid stays full\-corpus\.

## 4Results

We establish the divergence across the three direct contrasts \(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\), show it stable across the grid \(§[4\.2](https://arxiv.org/html/2607.05691#S4.SS2)\) and robust under hyperparameter and structural probes \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\), trace its mechanism \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\), and find the granularity gap generalizes off\-domain \(§[4\.5](https://arxiv.org/html/2607.05691#S4.SS5)\)\. All numbers below are generated from the deposited per\-condition measurements by the accompanying scripts\. Our analysis covers the2222matched conditions of §[3\.1](https://arxiv.org/html/2607.05691#S3.SS1)\(both arms trained\); the two single\-arm ZINC\-22V=2048V\{=\}2048coordinates are excluded, the BPE arm there clearing onlyc100≈0\.52c\_\{100\}\\approx 0\.52\(below theF95%,100F\_\{95\\%,100\}bar\) and the Unigram\-LM arm, unsafe by a wider margin, left untrained\.

Table[2](https://arxiv.org/html/2607.05691#S4.T2)collects all seven measurements across the2222conditions, grouped into the three direct contrasts and the four mechanism diagnostics; the subsections below read its columns in turn, the contrasts in §[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)and the diagnostics in §[4\.4](https://arxiv.org/html/2607.05691#S4.SS4), with the full per\-condition detail and bootstrap CIs in Appendix[F](https://arxiv.org/html/2607.05691#A6)\.

Table 2:Cross\-algorithm measurements per condition \(point estimates\)\.*Direct contrasts:*JJvocabulary overlap \(Jaccard\), rel\|Δ​f\|\|\\Delta f\|relative fertility gap,\|Δ​D\|\|\\Delta D\|token\-imbalance gap\.*Mechanism diagnostics:*Δ​c100\\Delta c\_\{100\}dead\-zone surplus \(cross\-arm clearance gap at theF95%,100F\_\{95\\%,100\}bar\),Δ\\Deltaabs whole\-pretoken\-absorption gap,Δ\\Deltascaf scaffold\-fraction gap,Δ​Hg\\Delta H\_\{g\}segmentation\-entropy gap per glyph\. Two\-sidedΔ​c100\\Delta c\_\{100\}andΔ\\Deltaabs are signed BPE−\-Unigram; one\-sidedΔ\\Deltascaf \(BPE\-only\) andΔ​Hg\\Delta H\_\{g\}\(Unigram\-only\) print the non\-zero arm’s magnitude\. Bootstrap CIs for the held\-out scalars are in the Appendix[F](https://arxiv.org/html/2607.05691#A6)detail tables; the exact\-setJJandΔ\\Deltascaf carry none\. Largest rel\|Δ​f\|\|\\Delta f\|inbold\.### 4\.1The two algorithms build measurably different vocabularies

We report in turn the three senses in which the vocabularies differ \(membership, granularity, distribution\)\. They carry different weight: membership is the more*surprising*contrast, the one a convergence would show up in first; granularity the more*robust and actionable*, carrying a bootstrap CI and untouched by the realized\-size asymmetry; and distribution the weakest\. Read positionally, granularity yields a fourth result that is not a fourth sense of difference but an answer to a question the first two raise: the near\-disjoint vocabularies nonetheless parse*compatibly*\(*Compatibility*, below\)\.

#### Membership\.

Given the same glyph base and corpus, BPE and Unigram\-LM build vocabularies that barely overlap, and they overlap least on the highest\-frequency pieces\. The frequency\-weighted overlapJwJ\_\{\\mathrm\{w\}\}\(Eq\.[4](https://arxiv.org/html/2607.05691#S3.E4)\) is at most0\.050\.05and falls to0\.0020\.002; the unweighted overlapJJnever exceeds0\.1610\.161and falls as low as0\.0100\.010\(Table[A12](https://arxiv.org/html/2607.05691#A6.T12), Fig\.[2](https://arxiv.org/html/2607.05691#S4.F2); per\-condition split in Fig\.[10](https://arxiv.org/html/2607.05691#A2.F10)\)\. Figure[3](https://arxiv.org/html/2607.05691#S4.F3)makes this concrete on three molecules segmented by the same PubChemV=1024V\{=\}1024\(MB\) tokenizers: BPE forms multi\-glyph chunks \(CC,CC=CC\) and keeps the stereocenter\[C@@H\]whole, whereas Unigram\-LM stays near\-atomic, even fragmenting\[C@@H\]into\[C@@,H,\]\. The two partition the identical glyph stream into almost entirely different units\.

![Refer to caption](https://arxiv.org/html/2607.05691v1/x2.png)Figure 2:Cross\-algorithm vocabulary overlap for2222matched conditions, frequency\-weighted \(JwJ\_\{\\mathrm\{w\}\}, left\) and unweighted \(JJ, right\) on a shared log axis; color and marker by corpus, filled NMB / open MB, with targetVVon thexx\-axis\. Every condition stays near\-disjoint, andJw<JJ\_\{\\mathrm\{w\}\}<Jthroughout\. Structural variants in Fig\.[15](https://arxiv.org/html/2607.05691#A6.F15); bootstrap CIs onJwJ\_\{\\mathrm\{w\}\}are smaller than the markers\.![Refer to caption](https://arxiv.org/html/2607.05691v1/x3.png)Figure 3:Segmentation and nesting on three held\-out molecules under the matched PubChemV=1024V\{=\}1024\(MB\) pair, the per\-molecule face of all three contrasts\. Each panel draws the shared glyph stream with Unigram\-LM tokens as boxes above \(orange\) and BPE below \(blue\); a box is*filled*when it is a learned multi\-glyph piece and*hollow*when it is a single base glyph, and dashed guides mark BPE cuts\.Membership:the two arms make different substructures atomic: Unigram\-LM’s pieces here areccccc,\[C@@,\[O\-,\[Na, BPE’s areCC,=O,\[C@@H\],CC=CC\.Granularity:per\-row token counts and compression \(glyphs/token\) give the fertility gap per molecule\.Compatibility:every BPE box spans a run of Unigram\-LM boxes \(*nest*\) with no*conflict*\(neither arm cutting where the other merges across it\), so BPE’s parse is a strict coarsening of Unigram\-LM’s, the9696–99%99\\%case across theV=1024V\{=\}1024cells \(Table[A17](https://arxiv.org/html/2607.05691#A6.T17)\)\.The right scale for this overlap is internal: the*same*algorithm retrained on the same corpus reproduces its vocabulary exactly \(self\-overlapJ=1\.0J\{=\}1\.0; the determinism check makes every cell byte\-identical on retrain, Appendix[A\.2](https://arxiv.org/html/2607.05691#A1.SS2)\), andJJmoves at most0\.0050\.005across independent subsample redraws \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\)\. Against that near\-unity ceiling the cross\-arm0\.0100\.010–0\.1610\.161is a qualitatively different regime, and the disagreement is sharpest exactly where the convergence null predicted agreement\.JwJ\_\{\\mathrm\{w\}\}falls*below*the unweightedJJin all2222conditions \(0\.0210\.021vs\.0\.0540\.054on PubChemV=256V\{=\}256NMB;0\.0020\.002vs\.0\.1050\.105on REAL\-Space\): the pieces the two arms*share*are disproportionately rare, so they disagree most on the high\-frequency core\. This refutes the convergence null of §[1](https://arxiv.org/html/2607.05691#S1): weighting by the mass a model sees does not soften the divergence but deepens it\.

The two arms disagree not just on*how many*pieces they share but on*what kind*of substructure each makes atomic \(Table[A6](https://arxiv.org/html/2607.05691#A3.T6)\)\. Aromatic\-ring pieces \(cccc,ccccc\) make up2727–37%37\\%of BPE’s exclusive pieces across the three corpora but only0–7%7\\%of Unigram\-LM’s, which instead skews to aliphatic heteroatom chains \(up to92%92\\%of its exclusive set on ZINC\-22\): BPE greedily merges ring fragments into single tokens while Unigram\-LM keeps them near\-atomic, the population\-level form of the segmentation example\. This BPE\-side ring\-merging is also visible in large\-VVchemistry vocabulary extension\[[16](https://arxiv.org/html/2607.05691#bib.bib16)\], where a BPE\-style tokenizer likewise captures aromatic rings as single tokens\. The same split is visible at the level of glyph adjacencies within pieces \(Fig\.[13](https://arxiv.org/html/2607.05691#A3.F13)\): Unigram\-LM’s pieces run longer on average and skew to sulfur and oxygen chains, while BPE reaches further into the aromatic–aliphatic junction\.

That disagreement on the high\-frequency core reflects genuine chemical substructure, not an artifact of how bracketed atoms are split: restricting the overlap to structural pieces barely moves it in either weighting \(Jw,structJ\_\{\\mathrm\{w,struct\}\}exceedsJwJ\_\{\\mathrm\{w\}\}by at most0\.0260\.026,JstructJ\_\{\\mathrm\{struct\}\}never exceeds0\.1780\.178, both still near\-disjoint; Table[A12](https://arxiv.org/html/2607.05691#A6.T12)\)\. The bracket\-internal component is largest on the scaffold\-rich COCONUT and negligible elsewhere, yet even there the structural disagreement dominates\.

On narrow alphabets the two arms diverge for a second, structural reason: they do not even build the same\-size vocabulary\. AtV=1024V\{=\}1024the Unigram\-LM arm saturates well below target \(ZINC\-22310310–357357vs\. BPE’s867867multi\-glyph pieces, REAL\-Space267267–293293vs\.867867\), its narrow alphabet offering too few high\-likelihood pieces for the pruning to reach the budget\. This mechanically caps the*unweighted*overlap, but the weighted headline is immune \(a piece an arm never realizes carries no emission mass\), soJwJ\_\{\\mathrm\{w\}\}stays at0\.0020\.002–0\.0090\.009on these cells \(ceilings in Appendix[D\.1](https://arxiv.org/html/2607.05691#A4.SS1)\)\.

#### Granularity\.

![Refer to caption](https://arxiv.org/html/2607.05691v1/x4.png)Figure 4:Held\-out fertility atV=1024V\{=\}1024as a dumbbell chart: each row joins BPE \(blue\) to Unigram\-LM \(orange\) on a shared tokens\-per\-molecule axis, so connector length is the absolute fertility gap, comparable across corpora\. Unigram\-LM lies to the right of BPE \(more tokens, less compression\) in every corpus and under both boundary policies \(NMB, permeable MB\), with the relative gap rel\|Δ​f\|\|\\Delta f\|\(Eq\.[2](https://arxiv.org/html/2607.05691#S3.E2)\) annotated per row \(3030–41%41\\%\)\. The absolute gap is largest on the natural\-products corpus \(COCONUT\), whose molecules are longest\. The relative gap’s rise withVVis in Appendix Fig\.[16](https://arxiv.org/html/2607.05691#A6.F16); per\-condition values across allVV, with bootstrap CIs, are in Table[A16](https://arxiv.org/html/2607.05691#A6.T16)\.The vocabularies do not just differ in membership; they segment molecules differently, and consistently so\. Unigram\-LM emits*more*tokens per held\-out molecule than BPE at matchedVVin every condition \(Figure[4](https://arxiv.org/html/2607.05691#S4.F4)\), with a relative fertility gap of29\.229\.2–41\.0%41\.0\\%\(the41\.0%41\.0\\%maximum on the REAL\-Space anchor; Table[2](https://arxiv.org/html/2607.05691#S4.T2), column rel\|Δ​f\|\|\\Delta f\|\)\. In absolute terms the gap is large: on PubChemV=1024V\{=\}1024\(NMB\) the average held\-out molecule is36\.136\.1tokens under BPE against50\.950\.9under Unigram\-LM \(per\-arm absolute fertilities, glyphs\-per\-token compression ratios, and bootstrap CIs for every condition in Appendix[F\.2](https://arxiv.org/html/2607.05691#A6.SS2), Table[A16](https://arxiv.org/html/2607.05691#A6.T16)\)\. Seen as compression the same gap is corpus\-stable where absolute length is not: BPE packs∼\\sim1\.41\.4–1\.81\.8glyphs per token across all three corpora while Unigram\-LM stays near\-atomic at∼\\sim1\.01\.0–1\.21\.2, barely above the one\-glyph\-per\-token floor\.

Unlike membership, granularity is unaffected by the realized\-size asymmetry: it is read from held\-out segmentation, so the unequal vocabulary sizes that cap the unweighted overlap on narrow alphabets leave it untouched even at those cells\. It is thus the contrast that most cleanly separates the two algorithms everywhere\. Beyond the training grid, this gap also*transfers*: it survives reuse off\-domain and on chemistry far outside the training distribution \(§[4\.5](https://arxiv.org/html/2607.05691#S4.SS5)\)\.

#### Compatibility: the near\-disjoint vocabularies nonetheless parse compatibly\.

The near\-disjoint membership and the fertility gap together raise a question they do not answer: do the two arms cut molecules in fundamentally incompatible places, or in the same places to different*depths*? Because both segment the same glyph stream, we can read the answer position by position \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4), Figure[3](https://arxiv.org/html/2607.05691#S4.F3)\)\. The gap is not merely a difference in count; it is*ordered*\. The arms agree on most cuts \(the boundary JaccardJ∂J\_\{\\partial\}is0\.650\.65–0\.740\.74across the grid\) and nearly all of their disagreement is*nesting*rather than crossing: Unigram\-LM cuts where BPE merges \(the nest rate,0\.240\.24–0\.340\.34, is the fertility gap read positionally\) while genuine crossing conflict stays near zero, below0\.7%0\.7\\%of positions in every condition and at most0\.1%0\.1\\%onceV≥1024V\{\\geq\}1024\. The molecule\-level consequence is sharp: on PubChemV=1024V\{=\}1024\(NMB\) the BPE segmentation is a strict coarsening of Unigram\-LM’s on97\.0%97\.0\\%of held\-out molecules, rising above99%99\\%on theV=1024V\{=\}1024ZINC\-22 and REAL\-Space cells, and never dropping below80%80\\%even atV=256V\{=\}256\(Table[A17](https://arxiv.org/html/2607.05691#A6.T17)\)\. This resolves the apparent paradox: the two algorithms build near\-disjoint*vocabularies*yet impose near\-compatible*parses*\. BPE lays down essentially the cut skeleton Unigram\-LM does and then merges further along it\. What little conflict remains is chemically structured, falling on heteroatom and unsaturated pieces rather than the saturated\-alkyl backbone, on the diverse and natural\-products corpora \(thecutc\\mathrm\{cut\}^\{c\}columns of Table[A17](https://arxiv.org/html/2607.05691#A6.T17)\)\.

#### Distribution\.

The third sense, token\-frequency distribution, separates the two algorithms in the same direction but by a smaller margin\. The token\-imbalance gap\|Δ​D\|\|\\Delta D\|exceeds the corpus\-draw noise floor \(∼\\sim0\.002, the one measured threshold we apply; §[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\) in every condition, the smallest gap \(0\.0380\.038, COCONUTV=1024V\{=\}1024\) still sitting an order of magnitude above it \(Table[2](https://arxiv.org/html/2607.05691#S4.T2), column\|Δ​D\|\|\\Delta D\|\)\. Distribution is corpus\-sensitive only in the*magnitude*of the gap, not whether it separates, a typological attenuation we examine in §[4\.2](https://arxiv.org/html/2607.05691#S4.SS2)\.

### 4\.2The divergence is stable across corpus, boundary, and scale

![Refer to caption](https://arxiv.org/html/2607.05691v1/x5.png)Figure 5:Scale does not dissolve the divergence, and could not be pushed much further if it did\. Both panels track PubChem \(NMB\) across the headline grid and theV=8192V\{=\}8192anchor on a log\-VVaxis\.Left:the frequency\-weighted overlapJwJ\_\{\\mathrm\{w\}\}and the relative granularity gaprel​\|Δ​f\|\\mathrm\{rel\}\|\\Delta f\|run flat through the anchor,8×8\\timesthe headline vocabulary, so neither contrast converges\.Right:theF95%,100F\_\{95\\%,100\}learnability clearance for each arm, with the0\.950\.95bar \(shaded region below is the undertrained regime\)\. The knee is sharp: the Unigram\-LM arm crosses the bar byV=2048V\{=\}2048and BPE byV=8192V\{=\}8192, so the anchor already sits past where the embeddings are learnable\. Per\-arm clearances across the grid are in Table[A20](https://arxiv.org/html/2607.05691#A6.T20), with theV=8192V\{=\}8192anchor in Appendix[A\.4](https://arxiv.org/html/2607.05691#A1.SS4)\.The divergence is stable along all three axes\. Across corpus typology \(Table[2](https://arxiv.org/html/2607.05691#S4.T2)\), membership and granularity separate the two algorithms in all six headline\(V,boundary\)\(V,\\text\{boundary\}\)conditions; across boundary policy no contrast’s outcome flips between the opaque\-bracket \(NMB\) and permeable\-bracket \(MB\) regimes \(Appendix Fig\.[9](https://arxiv.org/html/2607.05691#A1.F9)\); and across scale the fertility gap rises only slightly withVVwhile overlap stays near\-disjoint throughout, rising on PubChem and ZINC\-22 and falling on COCONUT \(Appendix Fig\.[16](https://arxiv.org/html/2607.05691#A6.F16)\)\. That overlap rise is one\-directional, BPE’s later merges absorbing substructures Unigram\-LM had already isolated at lowerVV\(marginal cross\-arm Jaccard≤0\.016\\leq 0\.016\); COCONUT instead starts high, its dominant natural\-product motifs shared early and then diluted as the near\-disjoint vocabulary grows\.

To test whether the divergence is a small\-VVartifact, we train one matched pair far above the grid, atV=8192V\{=\}8192on PubChem \(NMB\), where the wide alphabet lets*both*arms fill to target \(no Unigram saturation\), so the comparison is clean\. It does not close: the overlapsJ=0\.083J\{=\}0\.083andJw=0\.022J\_\{\\mathrm\{w\}\}\{=\}0\.022barely move from theirV=1024V\{=\}1024values of0\.0800\.080and0\.0220\.022, withJJshifting0\.0030\.003\(inside the≲0\.005\\lesssim 0\.005subsample\-redraw band, §[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\) andJwJ\_\{\\mathrm\{w\}\}by only0\.00040\.0004, and Unigram\-LM still segments34\.2%34\.2\\%more finely, on a flat3232–34%34\\%trend fromV=256V\{=\}256throughV=8192V\{=\}8192\. Whatever large\-VVconvergence the natural\-language literature reports \(§[5](https://arxiv.org/html/2607.05691#S5)\) therefore lies past the entire learnable range, not just the small\-VVregime we headline \(Figure[5](https://arxiv.org/html/2607.05691#S4.F5)\)\.

The one corpus\-sensitive feature is the*magnitude*of the distribution gap: it stays above the noise floor in every condition but on the natural\-products corpus runs about half that on the diverse and drug\-like corpora atV=1024V\{=\}1024\(\|Δ​D\|=0\.038\|\\Delta D\|=0\.038vs\.∼\\sim0\.07\), a size\-matched probe resolving that attenuation as typological, not a small\-sample artifact \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\)\. Beyond the gap, the absolute intrinsics show a coherent within\-family signature: both arms are highly imbalanced \(D≈0\.76D\\approx 0\.76–0\.970\.97\), but BPE is the more uniform of the two in all2222conditions \(lowerDD, higher normalized entropy and Rényi efficiency; Appendix[F\.4](https://arxiv.org/html/2607.05691#A6.SS4), Fig\.[17](https://arxiv.org/html/2607.05691#A6.F17)\)\.

### 4\.3Robustness of the contrasts

Robustness is assessed by a full sensitivity battery, not isolated spot\-checks \(§[3\.6](https://arxiv.org/html/2607.05691#S3.SS6), Appendix[A\.3](https://arxiv.org/html/2607.05691#A1.SS3), Figs\.[7](https://arxiv.org/html/2607.05691#A1.F7)–[9](https://arxiv.org/html/2607.05691#A1.F9)\): the separation survives every rung of every ladder and every interaction surface by a wide margin\.

#### Hyperparameter sweeps\.

Read as response curves, the two headline contrasts hold on every rung: overlap stays near\-disjoint \(J≤0\.123J\\leq 0\.123, within the≤0\.161\\leq 0\.161headline ceiling\) and the fertility gap stays large \(rel​\|Δ​f\|≥23%\\mathrm\{rel\}\|\\Delta f\|\\geq 23\\%\), both on the fixed PubChemV=512V\{=\}512NMB subsample, with the seed pool and BPE merge frequency inert\. The most overlap\-favorable hyperparameter corner we could construct is the shortest\-piece \(L=4L\{=\}4\) setting on the narrow\-alphabet corpora, where minimal Unigram\-LM pieces most resemble BPE’s merges\. Even thereJJpeaks at only0\.290\.29\(Appendix Fig\.[8](https://arxiv.org/html/2607.05691#A1.F8)\), above the headline≤0\.161\\leq 0\.161range but still leaving the two vocabularies more than70%70\\%disjoint\. One rung is a caveat rather than a contrast: the shrinking factor never moves the cross\-arm separation, but read against the default schedule it is the one knob that perturbs the Unigram vocabulary itself, which we revisit in §[5](https://arxiv.org/html/2607.05691#S5)\.

#### Structural probes\.

These hold the algorithm knobs fixed and complete the picture\. Across three independent subsample redraws the cross\-arm contrasts barely move \(overlap spread≲0\.005\\lesssim 0\.005, imbalance\-gap spread≲0\.0017\\lesssim 0\.0017\); the latter sets the corpus\-draw noise floor for the distribution sense, which every condition’s\|Δ​D\|≳0\.038\|\\Delta D\|\\gtrsim 0\.038clears by more than an order of magnitude\. A within\-PubChem size sweep locates the marginal PubChemV=512V\{=\}512NMB case as size\-driven rather than typological, and the REAL\-Space merge\-exhaustion anchor confirms the continuity premise \(Smirk’sGpeTrainerterminating naturally at\|𝒱\|=4,331\|\\mathcal\{V\}\|=4\{,\}331on Enamine’s published REAL 1% Sample, placing the small\-VVregime on the same merge trajectory as prior large\-VVwork\)\. Finally, asize\-matched typologyprobe resolves the size×\\timestypology confound on COCONUT \(§[3\.5](https://arxiv.org/html/2607.05691#S3.SS5)\) and the distribution\-magnitude attenuation of §[4\.2](https://arxiv.org/html/2607.05691#S4.SS2): subsampling PubChem and ZINC\-22 to COCONUT’s∼\\sim702K training size, the token\-imbalance gap stays near0\.060\.06on both non\-natural\-products corpora while COCONUT at the same size reads roughly two\-thirds that \(∼\\sim0\.0380\.038\), so the smaller natural\-products gap is a typology effect, not a small\-sample artifact\. At this reduced size the overlap rises on the narrow\-alphabet ZINC\-22 toJ≈0\.34J\\approx 0\.34, above the full\-corpus≤0\.161\\leq 0\.161headline but still leaving the two vocabularies about two\-thirds disjoint\.

### 4\.4How the two algorithms diverge

#### Mechanism diagnostics\.

Table[2](https://arxiv.org/html/2607.05691#S4.T2)collects all seven scalars per condition\. Of the four, three line up with the segmentation contrast and the fourth, the dead\-zone surplus, does not transfer\. The substantive one is whole\-pretoken absorption, higher for BPE in every condition \(the per\-pretoken counterpart of the fertility gap\): BPE absorbs almost every pretoken as a single token \(0\.890\.89–1\.001\.00, e\.g\.0\.970\.97on PubChemV=1024V\{=\}1024NMB\) against Unigram\-LM’s∼\\sim0\.720\.72\(Appendix[F\.5](https://arxiv.org/html/2607.05691#A6.SS5)\)\. The other two are one\-sided by construction and so weaker: segmentation entropy is carried entirely by Unigram\-LM, and the BPE scaffold fraction is non\-zero \(Unigram\-LM’s is exactly zero\) but faint at these sizes \(at most0\.6%0\.6\\%\)\. That is an order of magnitude below the∼\\sim6%Lian et al\. \[[26](https://arxiv.org/html/2607.05691#bib.bib26)\]report atV=32V\{=\}32K, a gap that reflects both the smallerVVand our conservative static criterion \(§[3\.5](https://arxiv.org/html/2607.05691#S3.SS5)\); the direction, a large\-VVphenomenon, is unchanged\.

#### The dead\-zone surplus\.

If anything, it cuts against the natural\-language account: whereBostrom and Durrett \[[5](https://arxiv.org/html/2607.05691#bib.bib5)\]found BPE carrying the larger dead zone, in chemistry both arms clear theF95%,100F\_\{95\\%,100\}bar wherever the tail can be certified, with negligible surplus \(\|Δ​c100\|≤0\.043\|\\Delta c\_\{100\}\|\\leq 0\.043, mixed sign; Table[A14](https://arxiv.org/html/2607.05691#A6.T14)\)\. The large surpluses occur only in conditions too small to certify, and there the sign reverses, Unigram\-LM stranding the rarely\-firing near\-atomic pieces its top\-down pruning keeps, so the asymmetry is corpus\-size\-driven, not algorithmic\. Bostrom’s BPE\-heavy surplus joins morphological alignment as a natural\-language leg that does not carry to chemistry SMILES\.

#### One mechanism, several views\.

The near\-disjoint vocabularies, the consistently signed fertility gap, and the absorption gap are not independent confirmations: absorption and fertility are the per\-pretoken and per\-molecule faces of the same coarser BPE segmentation\. The realized\-size asymmetry is one more view: on narrow alphabets at largeVV, Unigram\-LM’s pruning stops short while BPE fills to target \(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\), the same preference for finer pieces that drives the fertility gap and depresses the overlap\.

#### A structural reading\.

The same picture emerges from structure alone: BPE’s bottom\-up merging makes its vocabulary*compositionally closed*\(every piece concatenates two it already holds\), whereas Unigram\-LM’s likelihood pruning leaves roughly half its pieces unable to decompose into in\-vocab parts, the same long heteroatom chains it keeps exclusively\. Read on real molecules, BPE keeps the multiply\-bonded heteroatoms that define functional groups \(carbonyl==O, nitrile\#\\\#N\) inside a single token almost always \(0\.950\.95–1\.001\.00\), where Unigram\-LM almost never does \(≤0\.03\\leq 0\.03\)\. Five appendix analyses, piece composition, length, glyph co\-occurrence, compositional closure, and functional\-bond locality \(Appendix[C](https://arxiv.org/html/2607.05691#A3)\), all resolve this same greedy\-versus\-probabilistic signature\.

### 4\.5Generalization: the fertility gap is a property of the algorithm pair, not the corpus

Table 3:Cross\-corpus transfer: off\-domain fertility \(V=1024V\{=\}1024, NMB\), each cell BPE / Unigram\-LM\. Each value is the fertility of the*train*\-corpus tokenizer on the*eval*corpus’s held\-out split, divided by the eval corpus’s native tokenizer of the same arm; self\-transfer on the diagonal is omitted \(trivially1\.001\.00\)\. BPE transfers at near\-native fertility everywhere; Unigram\-LM is modestly domain\-sensitive, largest where the combinatorial REAL\-Space specialist meets natural products \(REAL\-Space→\\toCOCONUT, in bold\)\. Atom\-level OOV is below0\.01%0\.01\\%in every cell \(shared165165\-token base\)\.Sections[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)–[4\.2](https://arxiv.org/html/2607.05691#S4.SS2)established the granularity gap and its stability across the training grid\. We now show it is a property of the*algorithm pair*itself rather than of any particular training corpus: it persists when a tokenizer is read on a corpus it was never fit to \(cross\-domain transfer\) and even on chemistry far outside its training distribution \(out\-of\-distribution\), while coverage stays a non\-issue throughout\. This is a granularity\-specific check by construction: a vocabulary’s membership overlap is fixed at training and cannot be re\-probed on new corpora, whereas fertility is re\-measured wherever the tokenizer is read\. Reading the trained pairs on non\-canonical rewrites of their own held\-out split closes the section, yielding both a final robustness check on the gap and a new contrast in its own right,*write\-stability*\.

#### Cross\-domain transfer\.

The near\-disjoint vocabularies raise a practical question: is a tokenizer fit to one corpus usable on another, or must each domain retrain? We read everyV=1024V\{=\}1024\(NMB\) tokenizer on the held\-out split of each*other*corpus \(Table[3](https://arxiv.org/html/2607.05691#S4.T3), off\-diagonal; the diagonal native baseline is each corpus’s own\-tokenizer fertility, set to1\.001\.00\)\. Two findings\. First, coverage is a non\-issue: the shared165165\-token base keeps atom\-level out\-of\-vocabulary below0\.01%0\.01\\%in every cell\. Unlike natural\-language tokenizers, chemical ones do not shatter off\-domain input into unknowns, because the atomic alphabet is small and nearly universal\. Second, the off\-domain*fertility*penalty is small and arm\-dependent: BPE transfers within±1%\\pm 1\\%of native everywhere, so its sequence length is effectively corpus\-agnostic, whereas Unigram\-LM is modestly domain\-sensitive \(−6%\-6\\%to\+8%\+8\\%of native\), with the largest inflation exactly where the combinatorial REAL\-Space specialist meets the natural\-products corpus \(REAL\-Space→\\toCOCONUT,\+8%\+8\\%\)\. The algorithms diverge in*which*substructures they make atomic, not in how efficiently they tokenize off\-domain chemistry\.

Table 4:Out\-of\-distribution generalization: the PubChem diverse\-corpus generalist \(V=1024V\{=\}1024, NMB\) read on adversarial in\-spec chemistry it was not fit to\. Fertility is mean tokens per molecule; bootstrap CIs negligible at thesenn\. rel\|Δ​f\|\|\\Delta f\|is the cross\-arm relative fertility gap \(Eq\.[2](https://arxiv.org/html/2607.05691#S3.E2)\)\. The BPE\-coarser / Unigram\-finer divergence persists off\-distribution, and atom\-level OOV stays negligible \(shared165165\-token base\), so the contrast is genuine granularity, not coverage\.
#### Out\-of\-distribution chemistry\.

The transfer matrix stays within ordinary organics, so it never asks whether the contrast holds on genuinely exotic chemistry\. We test that directly by reading the PubChem diverse\-corpus generalist \(V=1024V\{=\}1024, NMB, both arms\) on two adversarial in\-spec corpora it was not fit to \(Table[4](https://arxiv.org/html/2607.05691#S4.T4)\): tmQM\[[4](https://arxiv.org/html/2607.05691#bib.bib4)\],∼\\sim97K transition\-metal complexes whose rare bracketed metals \(\[Sc\+3\],\[Y\+3\],\[MoH2\+2\]\) probe the coverage axis, and CycPeptMPDB\[[24](https://arxiv.org/html/2607.05691#bib.bib24)\],∼\\sim8K cyclic peptides \(∼\\sim110–150 tokens each\) whose length probes the fertility axis\. Coverage holds even here: atom\-level OOV is negligible for both arms \(zero on CycPeptMPDB;0\.002%0\.002\\%of tmQM tokens, from a residual off\-base aromatic silicon that survives the dative\-bond derivation documented in Appendix[A\.1](https://arxiv.org/html/2607.05691#A1.SS1)\), so even transition\-metal chemistry stays covered under the shared165165\-token base\. The granularity contrast*survives off\-distribution*: Unigram\-LM still segments more finely than BPE on chemistry neither arm was trained on, by29\.5%29\.5\\%on the cyclic peptides, just inside the29\.229\.2–41\.0%41\.0\\%in\-distribution band, and by23\.5%23\.5\\%on the metal complexes, more attenuated still, below that band\. That attenuation is the expected one: on the rare tokens neither arm learned a merge for, BPE falls back toward the shared glyph base and Unigram\-LM’s already\-near\-atomic segmentation, so the two partially converge, exactly the regime the metal corpus occupies\. The fertility gap therefore belongs to the algorithm pair, not the training corpora\.

#### Robustness to non\-canonical input\.

The first two checks vary the chemistry; this one varies the*notation*instead\. Reading the trained pairs on identity\-preserving SMILES rewrites of their held\-out split \(five rewrite families, Appendix[E](https://arxiv.org/html/2607.05691#A5)\) gives two results\. The gap*survives*: on the randomized orbit Unigram\-LM stays2323–35%35\\%finer than BPE in all2222conditions, the relative gap attenuating only slightly off\-canonical while both arms’ absolute fertilities rise, so the canonical fertilities lower\-bound a deployed model’s sequence length\. The orbit also separates the arms in a*new*sense,*write\-stability*: Unigram\-LM’s piece selection moves less under rewriting, its bag\-instability \(the fraction of the token multiset that changes\) running0\.110\.11–0\.220\.22against BPE’s0\.240\.24–0\.380\.38and holding the same ranking under an independent canonicalizer \(OpenBabel:0\.090\.09–0\.150\.15vs\.0\.190\.19–0\.240\.24\), in all2222conditions\.

## 5Discussion

The structural difference thatBostrom and Durrett \[[5](https://arxiv.org/html/2607.05691#bib.bib5)\]found between BPE and Unigram\-LM on natural language is present in chemistry SMILES, and not marginally\. At the vocabulary sizes where embeddings are learnable, the two algorithms build near\-disjoint vocabularies and segment molecules to systematically different lengths, robustly across the diverse→\\todrug\-like→\\tonatural\-products range and under both boundary policies, yet their cuts nest far more than they cross \(Appendix[F\.3](https://arxiv.org/html/2607.05691#A6.SS3)\)\. A small, valence\-constrained alphabet could plausibly have forced the two to converge\. That it does not, and that the divergence is algorithmic, corpus\-stable, and boundary\-robust rather than an artifact of the confounds prior work left open, is what the controlled design establishes\. The one prior chemistry head\-to\-head,Temizer et al\. \[[47](https://arxiv.org/html/2607.05691#bib.bib47)\]’s Unigram\-versus\-others comparison on raw ChEMBL, glimpsed the same contrast but at largeVVwithout a fixed base; our controlled design recovers it in a far more extreme form\.

Why the algorithm separates here when the field\-wide picture suggests it should not\.On the same three intrinsic metrics we use,Wadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]found the whole field of chemistry tokenizers scoring within a single90%90\\%quantile band, and argued that coverage, not subword scheme, separates them downstream\. Our result is the controlled complement of theirs, not a contradiction: their survey spans heterogeneous bases at each tokenizer’s native, mostly large vocabulary, where coverage dominates; once the base is fixed and the vocabulary pushed into the small regime, the algorithm axis their survey could not isolate emerges sharply on overlap and fertility\. Consistent with their reading, our*weakest*contrast is precisely token imbalance \(the metric on which they report all schemes clustering nearD≈0\.5D\\approx 0\.5\), and it is the one whose margin shrinks most across typology, a typological attenuation that never overturns the separation \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\)\. Its monotonic contraction withVVechoes the large\-VVwashoutReddy et al\. \[[36](https://arxiv.org/html/2607.05691#bib.bib36)\]report for natural language, but only that weakest contrast does so: theV=8192V\{=\}8192anchor leaves membership and granularity essentially unmoved from theirV=1024V\{=\}1024values \(Jw=0\.022J\_\{\\mathrm\{w\}\}\{=\}0\.022, gap34\.0%34\.0\\%\)\.

Why not a language\-model\-scale vocabulary?The divergence is not an artifact of staying small: chemistry’s*learnable*ceiling is low and theV=8192V\{=\}8192anchor already sits above it\. TheF95%,100F\_\{95\\%,100\}audit that gates the grid shows the Unigram\-LM arm crossing into the undertrained tail byV=2048V\{=\}2048even on PubChem, and both arms unsafe by the anchor, a ceiling set by the165165\-token base and consistent with deployed chemistry tokenizers staying small \(§[3\.5](https://arxiv.org/html/2607.05691#S3.SS5), Appendix[A\.4](https://arxiv.org/html/2607.05691#A1.SS4)\)\. Pushing a chemistry vocabulary toward language\-model scale would therefore not extend the present comparison but dissolve it: both arms would fill with pieces too rare to train, so any rise in overlap would reflect a shared exhaustion of the substructure space rather than agreement on what to make atomic\.

What this means for chemical\-LM builders\.The practical takeaway is that the subword algorithm is a genuine modeling decision, not a neutral default to inherit from natural\-language tooling\. Even with the atomic level pinned by a fixed165165\-token base, swapping BPE for Unigram\-LM changes which substructures become atomic embedding units \(near\-disjoint vocabularies, overlapping least on the high\-frequency pieces a model updates most\) and the effective sequence length by roughly a third \(the fertility gap\)\. BPE buys shorter sequences \(cheaper attention, longer context, and the NMT compression that tracks downstream quality at a fixed vocabulary\[[9](https://arxiv.org/html/2607.05691#bib.bib9)\]\) at the cost of coarser, less granular pieces; Unigram\-LM retains finer pieces and supports principled subword regularization\[[19](https://arxiv.org/html/2607.05691#bib.bib19)\], where BPE has only merge dropout\[[33](https://arxiv.org/html/2607.05691#bib.bib33)\]\. The training*corpus*, by contrast, matters little for efficiency: reused off\-domain a tokenizer segments at near\-native fertility \(§[4\.5](https://arxiv.org/html/2607.05691#S4.SS5)\), so the consequential choice is the algorithm, not the corpus\. Which arm is preferable downstream is the comparison this study sets up rather than settles, the more so since intrinsic separation need not translate into a downstream gap\[[1](https://arxiv.org/html/2607.05691#bib.bib1)\]; the choice is nonetheless worth justifying\.

Limits of the mechanism we can claim\.We claim the base\-independent part of the mechanism: near\-disjoint vocabularies, a consistently signed fertility gap, higher whole\-pretoken absorption for BPE, and the one\-sided segmentation\-entropy signal, which together show the two algorithms partition the same constrained glyph stream differently\. We do*not*claim the morphological\-alignment leg of the natural\-language account \(SMILES has no gold\-standard segmentation\), nor that the gap’s*direction*is portable: on protein sequences the same comparison finds Unigram\-LM the*more*compressive arm\[[46](https://arxiv.org/html/2607.05691#bib.bib46)\], so which algorithm segments more finely is domain\-specific\. One caveat survives on the Unigram reading itself, the prune\-schedule sensitivity flagged in §[4\.3](https://arxiv.org/html/2607.05691#S4.SS3), the one knob that visibly moves its piece set\.

## 6Limitations

As a tokenizer\-level study, this work does not measure the downstream language\-model quality a follow\-on takes up, nor does it exercise Unigram\-LM’s subword\-regularization capability \(§[5](https://arxiv.org/html/2607.05691#S5)\)\.

The headline contrast rests in part on two exact\-set Jaccard quantities \(JJandJstructJ\_\{\\mathrm\{struct\}\}\) that carry no per\-condition error bar\. Two variance sources bound them\. The determinism check measures the*retrain*jitter \(the symmetric\-difference piece count, same data retrained\) at zero for every cell, bounding it below∼\\sim0\.001 on theJJscale; the three subsample redraws \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\) bound the*corpus\-draw*variance directly, withJJmoving≲0\.005\\lesssim 0\.005across redraws; because these run at a smaller subsample than the full\-corpus headline cells, that bound is conservative for them\. Both are far smaller than the cross\-arm overlaps they qualify \(0\.0100\.010–0\.1610\.161\) and than the gap to self\-identity \(J=1\.0J\{=\}1\.0\), so the near\-disjointness is safe; a conclusion resting onJJresolved to finer than∼\\sim0\.005 would not be, but none here is\.

The corpus axis carries a size confound \(the three headline corpora differ in size as well as typology\); the REAL\-Space anchor, the within\-PubChem size sweep, and the size\-matched typology probe \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\) address it, the last by subsampling PubChem and ZINC\-22 to COCONUT’s training size and showing the one corpus\-dependent contrast is typological rather than size\-driven\. The learnability barF95%,100F\_\{95\\%,100\}is NMT\-derived\[[11](https://arxiv.org/html/2607.05691#bib.bib11)\]and its transfer to chemistry is itself an assumption\. We fix the base alphabet \(Smirk\), the canonicalization \(RDKit isomeric, non\-Kekulé\), and the algorithm set \(BPE, Unigram\-LM\); a different base, a Kekulé representation, and WordPiece are well\-posed separate experiments\. More broadly, the study presupposes a fixed\-vocabulary subword tokenizer; learned\-boundary and byte\-level architectures, such as dynamic chunking\[[14](https://arxiv.org/html/2607.05691#bib.bib14)\], hierarchical byte\-word models\[[29](https://arxiv.org/html/2607.05691#bib.bib29)\], and byte\-latent patching\[[32](https://arxiv.org/html/2607.05691#bib.bib32)\], dissolve the discrete vocabulary altogether and so lie outside the BPE\-versus\-Unigram contrast drawn here\.

## 7Conclusion

On chemistry SMILES, BPE and Unigram\-LM do not build the same vocabulary\. Across all2222matched conditions \(three corpus typologies, two boundary policies, and the small vocabulary sizes where embeddings are learnable\), the two algorithms produce near\-disjoint subword vocabularies \(the learned pieces above the shared165165\-token base; overlap≤0\.161\\leq 0\.161\) and Unigram\-LM segments molecules into2929–41%41\\%more tokens, with the contrast stable across corpus and boundary and only one secondary measure attenuating in magnitude on a single corpus, which we flag explicitly\. The convergence that a small, valence\-constrained alphabet could have forced does not happen, and it does not arrive with scale either: aV=8192V\{=\}8192anchor, eight times the headline vocabulary, leaves both contrasts essentially unmoved\. This removes the premise that the two algorithms are interchangeable at the tokenizer level, the default the field has inherited without testing\. The lever is the algorithm, not the training*corpus*: the vocabularies transfer across domains at near\-native cost \(within±1%\\pm 1\\%for BPE\)\. Whether either arm serves a downstream model better is the experiment this result now motivates\.

## Data and code availability

All trained tokenizers \(the4444two\-arm artifacts of the2222matched conditions, plus the single\-arm coordinates, the sensitivity sweep, and structural probes\), the per\-condition measurement data, and the table\- and figure\-generating scripts are released as a versioned archive \[DOI:10\.5281/zenodo\.21228245\]\. This article and its figures are licensed under CC\-BY 4\.0\.

## Acknowledgments

This work builds directly on the Smirk tokenizer and its fixed OpenSMILES glyph base released byWadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]\(the BattModels group\); both arms of the comparison are trained over that base using a pinned fork of their implementation\. I thank them for making the tokenizer and its configuration openly available\.

## Funding

This work received no external funding\.

## Competing interests

The author declares no competing interests\.

## References

- Ali et al\. \[2024\]Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Schulze Buschhoff, Charvi Jain, Alexander E\. Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suárez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores\-Herr\.Tokenizer choice for llm training: Negligible or crucial?In*Findings of the Association for Computational Linguistics: NAACL 2024*, pages 3907–3924, 2024\.doi:10\.18653/v1/2024\.findings\-naacl\.247\.URL[https://aclanthology\.org/2024\.findings\-naacl\.247](https://aclanthology.org/2024.findings-naacl.247)\.
- Alqahtani et al\. \[2026\]Sawsan Alqahtani, Mir Tafseer Nayeem, Md Tahmid Rahman Laskar, Tasnim Mohiuddin, and M Saiful Bari\.Stop taking tokenizers for granted: They are core design decisions in large language models\.In*Proceedings of the 2026 Conference of the European Chapter of the Association for Computational Linguistics*, pages 8410–8432\. Association for Computational Linguistics, 2026\.doi:10\.18653/v1/2026\.eacl\-long\.394\.URL[https://aclanthology\.org/2026\.eacl\-long\.394/](https://aclanthology.org/2026.eacl-long.394/)\.
- Arús\-Pous et al\. \[2019\]Josep Arús\-Pous, Simon V\. Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyrchan, Jean\-Louis Reymond, Hongming Chen, and Ola Engkvist\.Randomized SMILES strings improve the quality of molecular generative models\.*Journal of Cheminformatics*, 11\(1\):71, 2019\.doi:10\.1186/s13321\-019\-0393\-0\.
- Balcells and Skjelstad \[2020\]David Balcells and Bastian Bjerkem Skjelstad\.tmQM dataset—quantum geometries and properties of 86k transition metal complexes\.*Journal of Chemical Information and Modeling*, 60\(12\):6135–6146, 2020\.doi:10\.1021/acs\.jcim\.0c01041\.
- Bostrom and Durrett \[2020\]Kaj Bostrom and Greg Durrett\.Byte pair encoding is suboptimal for language model pretraining\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 4617–4624, 2020\.doi:10\.18653/v1/2020\.findings\-emnlp\.414\.URL[https://aclanthology\.org/2020\.findings\-emnlp\.414](https://aclanthology.org/2020.findings-emnlp.414)\.
- Chilingaryan et al\. \[2024\]Gayane Chilingaryan, Hovhannes Tamoyan, Ani Tevosyan, Nelly Babayan, Lusine Khondkaryan, Karen Hambardzumyan, Zaven Navoyan, Hrant Khachatrian, and Armen Aghajanyan\.BARTSmiles: Generative masked language models for molecular representations\.*Journal of Chemical Information and Modeling*, 64\(15\):5832–5843, 2024\.doi:10\.1021/acs\.jcim\.4c00512\.
- Chithrananda et al\. \[2020\]Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar\.Chemberta: Large\-scale self\-supervised pretraining for molecular property prediction, 2020\.
- Cognetta et al\. \[2024\]Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki\.Two counterexamples to tokenization and the noiseless channel\.In*Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation*, pages 13674–13683, 2024\.URL[https://aclanthology\.org/2024\.lrec\-main\.1469](https://aclanthology.org/2024.lrec-main.1469)\.
- Gallé \[2019\]Matthias Gallé\.Investigating the effectiveness of BPE: The power of shorter sequences\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 1375–1381\. Association for Computational Linguistics, 2019\.doi:10\.18653/v1/D19\-1141\.URL[https://aclanthology\.org/D19\-1141/](https://aclanthology.org/D19-1141/)\.
- Ganeeva et al\. \[2025\]Veronika Ganeeva, Kuzma Khrabrov, Artur Kadurin, and Elena Tutubalina\.Measuring chemical LLM robustness to molecular representations: a SMILES variation\-based framework\.*Journal of Cheminformatics*, 17\(1\):164, 2025\.doi:10\.1186/s13321\-025\-01079\-0\.
- Gowda and May \[2020\]Thamme Gowda and Jonathan May\.Finding the optimal vocabulary size for neural machine translation\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 3955–3964\. Association for Computational Linguistics, 2020\.doi:10\.18653/v1/2020\.findings\-emnlp\.352\.URL[https://aclanthology\.org/2020\.findings\-emnlp\.352/](https://aclanthology.org/2020.findings-emnlp.352/)\.
- Grygorenko et al\. \[2020\]Oleksandr O\. Grygorenko, Dmytro S\. Radchenko, Igor Dziuba, Alexander Chuprina, Kateryna E\. Gubina, and Yurii S\. Moroz\.Generating multibillion chemical space of readily accessible screening compounds\.*iScience*, 23\(11\):101681, 2020\.doi:10\.1016/j\.isci\.2020\.101681\.
- Heidenreich \[2026\]Hunter Heidenreich\.Smirk fork for the vocabulary–tokenizer comparison study: shared glyph\-id front\-end withGpeTrainer\(bpe\) and a unigram\-lm sibling trainer\.GitHub releasevtc\-2026\-05\-24, 2026\.URL[https://github\.com/hunter\-heidenreich/smirk/releases/tag/vtc\-2026\-05\-24](https://github.com/hunter-heidenreich/smirk/releases/tag/vtc-2026-05-24)\.Public, pinned fork of Smirk\. The BPE and Unigram\-LM arms share thecompute\_alphabet/tokenize\_wordsglyph\-id front\-end; the Unigram arm bridges to HuggingFace’sUnigramTrainervia a bijective Private\-Use\-Area glyph↔\\leftrightarrowchar map\.
- Hwang et al\. \[2025\]Sukjun Hwang et al\.Dynamic chunking for end\-to\-end hierarchical sequence modeling, 2025\.URL[https://arxiv\.org/abs/2507\.07955](https://arxiv.org/abs/2507.07955)\.
- James \[2016\]Craig A\. James\.OpenSMILES specification, version 1\.0, 2016\.URL[http://opensmiles\.org/opensmiles\.html](http://opensmiles.org/opensmiles.html)\.Blue Obelisk project; dated 2016\-05\-15\.
- Kalamkar et al\. \[2025\]Prathamesh Kalamkar, Ned Letcher, Meissane Chami, Sahger Lad, Shayan Mohanty, and Prasanna Pendse\.The tokenization bottleneck: How vocabulary extension improves chemistry representation learning in pretrained language models, 2025\.
- Kim et al\. \[2023\]Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A\. Shoemaker, Paul A\. Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E\. Bolton\.PubChem 2023 update\.*Nucleic Acids Research*, 51\(D1\):D1373–D1380, 2023\.doi:10\.1093/nar/gkac956\.
- Krenn et al\. \[2020\]Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru\-Guzik\.Self\-referencing embedded strings \(SELFIES\): A 100% robust molecular string representation\.*Machine Learning: Science and Technology*, 1\(4\):045024, 2020\.doi:10\.1088/2632\-2153/aba947\.
- Kudo \[2018\]Taku Kudo\.Subword regularization: Improving neural network translation models with multiple subword candidates\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 66–75, Melbourne, Australia, 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/P18\-1007\.URL[https://aclanthology\.org/P18\-1007](https://aclanthology.org/P18-1007)\.
- Kudo and Richardson \[2018\]Taku Kudo and John Richardson\.Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 66–71, 2018\.doi:10\.18653/v1/D18\-2012\.URL[https://aclanthology\.org/D18\-2012](https://aclanthology.org/D18-2012)\.
- Land and Bartolo \[2024\]Sander Land and Max Bartolo\.Fishing for Magikarp: Automatically detecting under\-trained tokens in large language models\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 11631–11646\. Association for Computational Linguistics, 2024\.doi:10\.18653/v1/2024\.emnlp\-main\.649\.URL[https://aclanthology\.org/2024\.emnlp\-main\.649/](https://aclanthology.org/2024.emnlp-main.649/)\.
- Landrum et al\. \[2026\]Greg Landrum, Paolo Tosco, Brian Kelley, David Cosgrove, Riccardo Vianello, et al\.RDKit: Open\-source cheminformatics, version 2026\.03\.1, 2026\.URL[https://doi\.org/10\.5281/zenodo\.19250388](https://doi.org/10.5281/zenodo.19250388)\.Zenodo, release2026\_03\_1\.
- Leon et al\. \[2024\]Miguelangel Leon, Yuriy Perezhohin, Fernando Peres, Aleš Popovič, and Mauro Castelli\.Comparing SMILES and SELFIES tokenization for enhanced chemical language modeling\.*Scientific Reports*, 14\(1\):25016, 2024\.doi:10\.1038/s41598\-024\-76440\-8\.
- Li et al\. \[2023\]Jianan Li, Keisuke Yanagisawa, Masahito Sugita, Takuya Fujie, Masahito Ohue, and Yutaka Akiyama\.CycPeptMPDB: A comprehensive database of membrane permeability of cyclic peptides\.*Journal of Chemical Information and Modeling*, 63\(7\):2240–2250, 2023\.doi:10\.1021/acs\.jcim\.2c01573\.
- Li and Fourches \[2021\]Xinhao Li and Denis Fourches\.SMILES pair encoding: A data\-driven substructure tokenization algorithm for deep learning\.*Journal of Chemical Information and Modeling*, 61\(4\):1560–1569, 2021\.doi:10\.1021/acs\.jcim\.0c01127\.
- Lian et al\. \[2024\]Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Hui Chen, Peng Liu, Jungong Han, and Guiguang Ding\.Scaffold\-bpe: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal, 2024\.URL[https://arxiv\.org/abs/2404\.17808](https://arxiv.org/abs/2404.17808)\.
- Liu et al\. \[2025\]Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A\. Smith, and Yejin Choi\.Superbpe: Space travel for language models\.In*Proceedings of the Second Conference on Language Modeling \(COLM\)*, 2025\.URL[https://arxiv\.org/abs/2503\.13423](https://arxiv.org/abs/2503.13423)\.
- Moi and Patry \[2023\]Anthony Moi and Nicolas Patry\.HuggingFace’s tokenizers: Fast state\-of\-the\-art tokenizers optimized for research and production\.[https://github\.com/huggingface/tokenizers](https://github.com/huggingface/tokenizers), 2023\.URL[https://github\.com/huggingface/tokenizers](https://github.com/huggingface/tokenizers)\.
- Neitemeier et al\. \[2025\]Pit Neitemeier et al\.Hierarchical autoregressive transformers: Combining byte\- and word\-level processing for robust, adaptable language models, 2025\.URL[https://arxiv\.org/abs/2501\.10322](https://arxiv.org/abs/2501.10322)\.
- O’Boyle \[2012\]Noel M\. O’Boyle\.Towards a universal SMILES representation \- a standard method to generate canonical SMILES based on the InChI\.*Journal of Cheminformatics*, 4\(1\):22, 2012\.doi:10\.1186/1758\-2946\-4\-22\.
- O’Boyle and Dalke \[2018\]Noel M\. O’Boyle and Andrew Dalke\.DeepSMILES: An adaptation of SMILES for use in machine\-learning of chemical structures\.*ChemRxiv*, 2018\.doi:10\.26434/chemrxiv\.7097960\.v1\.
- Pagnoni et al\. \[2024\]Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer\.Byte latent transformer: Patches scale better than tokens, 2024\.URL[https://arxiv\.org/abs/2412\.09871](https://arxiv.org/abs/2412.09871)\.
- Provilkov et al\. \[2020\]Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita\.BPE\-dropout: Simple and effective subword regularization\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 1882–1892, 2020\.doi:10\.18653/v1/2020\.acl\-main\.170\.
- Radhakrishnan et al\. \[2026\]Sridhar Radhakrishnan, Krish Mody, Arvind Venkatesh, and Ananth Venkatesh\.Optimizing SMILES token sequences via trie\-based refinement and transition graph filtering\.*Journal of Cheminformatics*, 18\(1\):13, 2026\.doi:10\.1186/s13321\-025\-01143\-9\.
- Ramsundar et al\. \[2019\]Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu\.*Deep Learning for the Life Sciences: Applying Deep Learning to Genomics, Microscopy, Drug Discovery, and More*\.O’Reilly Media, 2019\.ISBN 978\-1492039839\.
- Reddy et al\. \[2025\]Varshini Reddy, Craig W\. Schmidt, Yuval Pinter, and Chris Tanner\.How much is enough? the diminishing returns of tokenization training data, 2025\.URL[https://arxiv\.org/abs/2502\.20273](https://arxiv.org/abs/2502.20273)\.ICML 2025 Tokenization Workshop \(TokShop\)\.
- Rust et al\. \[2021\]Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych\.How good is your tokenizer? on the monolingual performance of multilingual language models\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 3118–3135\. Association for Computational Linguistics, 2021\.doi:10\.18653/v1/2021\.acl\-long\.243\.URL[https://aclanthology\.org/2021\.acl\-long\.243/](https://aclanthology.org/2021.acl-long.243/)\.
- Sagawa and Kojima \[2023\]Tatsuya Sagawa and Ryosuke Kojima\.ReactionT5: a large\-scale pre\-trained model towards application of limited reaction data\.*arXiv preprint*, 2023\.doi:10\.48550/arxiv\.2311\.06708\.
- Schmidt et al\. \[2024\]Craig W\. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris C\. Tanner\.Tokenization is more than compression\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 678–702\. Association for Computational Linguistics, 2024\.doi:10\.18653/v1/2024\.emnlp\-main\.40\.URL[https://aclanthology\.org/2024\.emnlp\-main\.40/](https://aclanthology.org/2024.emnlp-main.40/)\.
- Schmidt et al\. \[2025\]Craig W\. Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter\.Boundless byte pair encoding: Breaking the pre\-tokenization barrier\.In*Conference on Language Modeling \(COLM\)*, 2025\.URL[https://arxiv\.org/abs/2504\.00178](https://arxiv.org/abs/2504.00178)\.
- Schuster and Nakajima \[2012\]Mike Schuster and Kaisuke Nakajima\.Japanese and Korean voice search\.In*2012 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, pages 5149–5152\. IEEE, 2012\.doi:10\.1109/ICASSP\.2012\.6289079\.
- Schwaller et al\. \[2018\]Philippe Schwaller, Théophile Gaudin, Dávid Lányi, Costas Bekas, and Teodoro Laino\.“found in translation”: predicting outcomes of complex organic chemistry reactions using neural sequence\-to\-sequence models\.*Chemical Science*, 9\(28\):6091–6098, 2018\.doi:10\.1039/c8sc02339e\.
- Sennrich et al\. \[2016\]Rico Sennrich, Barry Haddow, and Alexandra Birch\.Neural machine translation of rare words with subword units\.In*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics*, pages 1715–1725, 2016\.doi:10\.18653/v1/P16\-1162\.URL[https://aclanthology\.org/P16\-1162](https://aclanthology.org/P16-1162)\.
- Skinnider \[2024\]Michael A\. Skinnider\.Invalid SMILES are beneficial rather than detrimental to chemical language models\.*Nature Machine Intelligence*, 6\(4\):437–448, 2024\.doi:10\.1038/s42256\-024\-00821\-x\.
- Sorokina et al\. \[2021\]Maria Sorokina, Peter Merseburger, Kohulan Rajan, Mehmet Aziz Yirik, and Christoph Steinbeck\.COCONUT online: Collection of open natural products database\.*Journal of Cheminformatics*, 13\(1\):2, 2021\.doi:10\.1186/s13321\-020\-00478\-9\.
- Suyunu et al\. \[2024\]Burak Suyunu, Enes Taylan, and Arzucan Özgür\.Linguistic laws meet protein sequences: A comparative analysis of subword tokenization methods, 2024\.
- Temizer et al\. \[2024\]Asu Büşra Temizer, Gökçe Uludoğan, Rıza Özçelik, Taha Koulani, Elif Özkırımlı, Kutlu Ö\. Ülgen, Nilgün Karalı, and Arzucan Özgür\.Exploring data\-driven chemical SMILES tokenization approaches to identify key protein\-ligand binding moieties\.*Molecular Informatics*, 43\(3\):e202300249, 2024\.doi:10\.1002/minf\.202300249\.
- Tingle et al\. \[2023\]Benjamin I\. Tingle, Khanh G\. Tang, Mar Castanon, John J\. Gutierrez, Munkhzul Khurelbaatar, Chinzorig Dandarchuluun, Yurii S\. Moroz, and John J\. Irwin\.ZINC\-22 – a free multi\-billion\-scale database of tangible compounds for ligand discovery\.*Journal of Chemical Information and Modeling*, 63\(4\):1166–1176, 2023\.doi:10\.1021/acs\.jcim\.2c01253\.
- Touvron et al\. \[2023\]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample\.LLaMA: Open and efficient foundation language models, 2023\.URL[https://arxiv\.org/abs/2302\.13971](https://arxiv.org/abs/2302.13971)\.
- Ucak et al\. \[2023\]Umit V\. Ucak, Islambek Ashyrmamatov, and Juyong Lee\.Improving the quality of chemical language model outcomes with atom\-in\-SMILES tokenization\.*Journal of Cheminformatics*, 15\(1\):55, 2023\.doi:10\.1186/s13321\-023\-00725\-9\.
- Wadell et al\. \[2026\]Alexius Wadell, Anoushka Bhutani, and Venkatasubramanian Viswanathan\.Tokenization for molecular foundation models\.*Journal of Chemical Information and Modeling*, 66\(3\):1384–1393, 2026\.doi:10\.1021/acs\.jcim\.5c01856\.
- Wegmann et al\. \[2025\]Anna Wegmann, Dong Nguyen, and David Jurgens\.Tokenization is sensitive to language variation\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 10958–10983\. Association for Computational Linguistics, 2025\.doi:10\.18653/v1/2025\.findings\-acl\.572\.URL[https://aclanthology\.org/2025\.findings\-acl\.572/](https://aclanthology.org/2025.findings-acl.572/)\.
- Weininger \[1988\]David Weininger\.SMILES, a chemical language and information system\. 1\. introduction to methodology and encoding rules\.*Journal of Chemical Information and Computer Sciences*, 28\(1\):31–36, 1988\.doi:10\.1021/ci00057a005\.
- Wu et al\. \[2018\]Zhenqin Wu, Bharath Ramsundar, Evan N\. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S\. Pappu, Karl Leswing, and Vijay S\. Pande\.MoleculeNet: a benchmark for molecular machine learning\.*Chemical Science*, 9\(2\):513–530, 2018\.doi:10\.1039/c7sc02664a\.
- Zdrazil et al\. \[2023\]Barbara Zdrazil, Eloy Félix, Fiona Hunter, Emma J\. Manners, James Blackshaw, Sybilla Corbett, Marleen De Veij, Harris Ioannidis, David Méndez, Juan F Mosquera, María Paula Magariños, Nicolas Bosc, Ricardo Arcila, Tevfik Kizilören, Anna Gaulton, A\. Patrícia Bento, Melissa F\. Adasme, Peter Monecke, Gregory A\. Landrum, and Andrew R\. Leach\.The ChEMBL database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods\.*Nucleic Acids Research*, 52\(D1\):D1180–D1192, 2023\.doi:10\.1093/nar/gkad1004\.
- Zouhar et al\. \[2023\]Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell\.Tokenization and the noiseless channel\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5184–5207\. Association for Computational Linguistics, 2023\.doi:10\.18653/v1/2023\.acl\-long\.284\.URL[https://aclanthology\.org/2023\.acl\-long\.284/](https://aclanthology.org/2023.acl-long.284/)\.

## Appendix AMethods and reproducibility detail

This appendix collects the study’s methodological detail: corpus construction and conformance filtering \(Appendix[A\.1](https://arxiv.org/html/2607.05691#A1.SS1)\), tokenizer training and byte\-level determinism \(Appendix[A\.2](https://arxiv.org/html/2607.05691#A1.SS2)\), the one\-factor\-at\-a\-time sensitivity battery and structural probes \(Appendix[A\.3](https://arxiv.org/html/2607.05691#A1.SS3)\), and the learnable\-regime confirmation \(Appendix[A\.4](https://arxiv.org/html/2607.05691#A1.SS4)\)\.

### A\.1Preprocessing and corpus construction

Each corpus passes through canonicalization, deduplication, and conformance filtering\. RDKit isomeric canonicalization maps every inputMolFromSmiles→\\toMolToSmiles\(non\-Kekulé, so aromatic systems are emitted in RDKit’s aromatic form\); inputs RDKit cannot parse or round\-trip are dropped, the per\-corpus drop rate is recorded, and the RDKit version is pinned \(rdkit==2026\.03\.1\), since canonical output is version\-dependent\. Deduplication is exact\-string only, so tautomers and protonation states survive as distinct entries\.

A base\-conformance filter then closes each corpus under the Smirk base\. The bare base tokenizer \(the165165\-token regex with no merges\) is the oracle: a molecule is dropped iff its decomposition emits\[UNK\]\(id0\), a frequency\-independent per\-molecule test that needs no trained tokenizer, so it runs ahead of subsampling, the split, and training\. We certified all four corpora \(∼\\sim344M molecules scanned\): COCONUT, ZINC\-22, and REAL\-Space are100%100\\%conformant \(0dropped across738,801738\{,\}801,83\.683\.6M, and136\.4136\.4M molecules\), while PubChem drops4,3514\{,\}351of123\.4123\.4M \(0\.0035%0\.0035\\%\)\. All4,3514\{,\}351PubChem offenders carry an aromatic\-silicon or aromatic\-tellurium atom that RDKit emits but OpenSMILES does not define, or one of a handful of Si\-family strings RDKit itself cannot re\-parse, which the base\-\[UNK\]test catches where an RDKit check would error\. Dropped molecules are deposited per corpus for inspection\.

Corpora exceeding the target size are reduced to the per\-corpus totals of Table[1](https://arxiv.org/html/2607.05691#S3.T1)by deterministic hash\-partition subsampling, so the conformance\-scan counts above are pre\-subsample\. Each corpus’s held\-out test split is5%5\\%, capped at10610^\{6\}molecules, and is disjoint from training\.

The two out\-of\-distribution evaluation corpora \(§[4\.5](https://arxiv.org/html/2607.05691#S4.SS5)\) are read with the trained PubChem tokenizers and need only light handling; CycPeptMPDB needs none\. tmQM ships a dative\-bond serialization \(\-\>/<\-\) that is not OpenSMILES and would route to\[UNK\]on the\>glyph; we derive a conformant encoding by converting the dative bonds to single bonds and recomputing valence, a representation choice that shifts implicit\-hydrogen bookkeeping on metal\-bound atoms, so the tmQM reading rests on our derivation rather than an independent reference encoding\. The derivation deliberately bypasses the canonicalize\-and\-dedup stage \(and hence the base\-conformance filter above\) to avoid re\-perceiving the metal coordination, so a residual0\.115%0\.115\\%of complexes retain an off\-base aromatic\-silicon atom; these account for the only\[UNK\]either arm emits on tmQM \(0\.002%0\.002\\%of tokens\)\.

### A\.2Tokenizer training and determinism

Both arms call the same alphabet\-construction and pre\-tokenization routines \(compute\_alphabet,tokenize\_words\), so the boundary policy is applied inside those shared routines rather than per arm\. The Unigram\-LM arm’sUnigramTrainer\[[19](https://arxiv.org/html/2607.05691#bib.bib19)\]performs frequent\-substring seeding scored by frequency×\\timeslength, an EM/pruning loop, and finalization, following the SentencePiece reference\[[20](https://arxiv.org/html/2607.05691#bib.bib20)\]\. Reference defaults are SentencePiece’s for Unigram\-LM \(maximum piece length1616, seed pool10610^\{6\}, two EM sub\-iterations, shrinking factor0\.750\.75\) and a minimum merge frequency of22for BPE \(Sennrich’s default and, since BPE imposes no piece\-length cap, its only free training knob\)\. Table[A1](https://arxiv.org/html/2607.05691#A1.T1)collects every knob, its default, and its sensitivity ladder\.

BPE\(greedy, bottom\-up\)

1:boundary\-filtered glyph\-id words; target

VV
2:

𝒱←\\mathcal\{V\}\\leftarrow165165\-token base

3:while

\|𝒱\|<\|\\mathcal\{V\}\|<target

VVdo

4:count adjacent token pairs

5:

\(p∗,c∗\)←\(p^\{\*\},c^\{\*\}\)\\leftarrowmost\-frequent pair

6:if

c∗<fc^\{\*\}<fthenstop⊳\\trianglerightmin\_frequency

7:merge

p∗p^\{\*\}; add to

𝒱\\mathcal\{V\}
8:return

𝒱\\mathcal\{V\}, ordered merge list

Unigram\-LM\(probabilistic, top\-down\)

1:boundary\-filtered glyph\-id words; target

VV
2:seed

←\\leftarrowsubstrings,

len≤L\\text\{len\}\\leq L⊳\\trianglerightmax\_piece\_length

3:capped at top

SS⊳\\trianglerightseed\_size

4:assign each piece a probability

5:while

\|𝒱\|\>\|\\mathcal\{V\}\|\>target

VVdo

6:for

mmsub\-iterationsdo⊳\\trianglerightn\_sub\_iterations

7:E\-step: expected piece usage

8:M\-step: re\-estimate; drop

E​\[use\]<0\.5\\mathrm\{E\}\[\\text\{use\}\]<0\.5
9:prune:

max⁡\(V,s​\|𝒱\|\)\\max\(V,s\|\\mathcal\{V\}\|\)⊳\\trianglerightshrinking\_factor

Figure 6:BPE grows a vocabulary bottom\-up; Unigram\-LM prunes a large seed top\-down\. The⊳\\trianglerightcomments mark where each training hyperparameter enters: one for BPE, four for Unigram\-LM \(defaults in Table[A1](https://arxiv.org/html/2607.05691#A1.T1)\)\.Table A1:Tokenizer training hyperparameters, by arm\. Each numeric knob is held at its reference\-implementation default for the headline grid and swept across a ladder bracketing that default for the sensitivity analysis \(§[3\.1](https://arxiv.org/html/2607.05691#S3.SS1)\);boldmarks the reference default within each ladder\. Design\-forced settings depart from the SentencePiece reference by construction, not tuning \(§[3\.3](https://arxiv.org/html/2607.05691#S3.SS3)\)\. Reference defaults are the published library defaults: SentencePiece’sTrainerSpec\(sentencepiece\_model\.proto\) and Sennrich’s subword\-nmt \(learn\_bpe\.py,\-\-min\-frequency\)\.ParameterRoleRef\. defaultHeadlineSensitivity ladder*Unigram\-LM*\(SentencePiece reference\[[20](https://arxiv.org/html/2607.05691#bib.bib20)\]\)max\_piece\_lengthmax glyphs per piece161616164,8,𝟏𝟔,32,64,1284,8,\\mathbf\{16\},32,64,128seed\_sizeseed\-pool cap10610^\{6\}10610^\{6\}\(2\.5,5,𝟏𝟎,20,40,80\)×105\(2\.5,5,\\mathbf\{10\},20,40,80\)\\\!\\times\\\!10^\{5\}n\_sub\_iterationsEM iters per prune22221,𝟐,3,41,\\mathbf\{2\},3,4shrinking\_factorkept fraction per prune0\.750\.750\.750\.750\.5,0\.6,0\.75,0\.9,0\.950\.5,0\.6,\\mathbf\{0\.75\},0\.9,0\.95*BPE*\(subword\-nmt reference\[[43](https://arxiv.org/html/2607.05691#bib.bib43)\]\)min\_frequencymin pair count to merge22220,1,𝟐,4,80,1,\\mathbf\{2\},4,8*Shared design axes*vocab\_sizeVVtarget size—256/512/1024256/512/1024\+2048\+2048; anchor10241024boundarybracket permeability—NMB / MB—*Design\-forced*\(deviate from SentencePiece by construction\)character\_coverageOOV coverage0\.99950\.99951\.01\.0—num\_threadsparallelism161611—pre\-tokenizationstring splittingwhitespace / scriptSmirk Layer\-A/B—Three settings depart from the SentencePiece reference by design, not tuning\. Character coverage is1\.01\.0rather than0\.99950\.9995, because Smirk’s base spans the OpenSMILES alphabet by construction and leaves no rare\-character tail\. Training is single\-threaded, which makes every artifact byte\-reproducible \(the multi\-threaded reference defaults are not\)\. SentencePiece’s text\-oriented splitting \(whitespace, Unicode script, dummy prefix\) is replaced wholesale by Smirk’s chemistry\-aware Layer\-A/B pre\-tokenizer, the point of fixing the base\. The two defaults that most directly govern piece granularity, the Unigram maximum piece length \(16→12816\\to 128\) and the BPE merge\-frequency floor \(2→02\\to 0\), are treated as ablation axes \(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\)\. The trainers live in a public, pinned fork of Smirk\[[13](https://arxiv.org/html/2607.05691#bib.bib13)\]\.

The two trainers budget the six non\-\[UNK\]special tokens differently: BPE counts them inside the targetVV, while the Unigram\-LM trainer adds them on top\. At a matchedVVthe Unigram\-LM arm therefore carries six more content pieces, an offset of≤2\.4%\\leq 2\.4\\%ofVV\(six pieces at the smallestV=256V\{=\}256\)\. This lets the Unigram\-LM arm compress slightly more than it otherwise would, so it can only shrink the fertility gap reported in the body, never inflate it\.

For determinism, every tokenizer is trained twice and its two artifacts are asserted byte\-identical; a failed assertion is flagged rather than silently passed\.

### A\.3Sensitivity analysis and structural probes

The sensitivity battery sweeps each training hyperparameter one\-factor\-at\-a\-time across its ladder in Table[A1](https://arxiv.org/html/2607.05691#A1.T1): geometric \(×2\\times 2\) ladders for the unbounded knobs \(the Unigram maximum piece length, the seed pool, and the BPE merge frequency\) and within\-range grids for the bounded knobs, where doubling is undefined \(the EM sub\-iterations and shrinking factor\)\. Four pairwise interactions probe covariation between knobs: maximum piece length×\\timescorpus typology, BPE merge frequency×\\timesUnigram maximum piece length, sub\-iterations×\\timesshrinking factor, and maximum piece length×V\\times V\. To stay tractable the battery runs on a fixed representative subsample rather than each full corpus, justified by the within\-PubChem size sweep\. Alongside the knob sweeps we retain four structural probes: subsample redraws, a within\-PubChem size sweep, a size\-matched typology probe, and a REAL\-Space merge\-exhaustion continuity run\.

![Refer to caption](https://arxiv.org/html/2607.05691v1/x6.png)Figure 7:Per\-knob sensitivity response curves on the fixed PubChemV=512V\{=\}512NMB subsample: vocabulary overlap \(JJ, top\) and the relative fertility gap \(rel​\|Δ​f\|\\mathrm\{rel\}\|\\Delta f\|, bottom\) as each training hyperparameter is swept across its ladder \(the Unigram maximum piece length, seed pool, sub\-iterations, and shrinking factor, plus the BPE merge frequency\)\. The gray vertical lines mark the reference default\. Both contrasts hold their separation on every rung; the seed pool and BPE merge frequency are inert \(flat\)\.![Refer to caption](https://arxiv.org/html/2607.05691v1/x7.png)Figure 8:The four pairwise hyperparameter interactions, each a heatmap of the cross\-arm vocabulary overlapJJ\(per cell annotated\): \(a\) sub\-iterations×\\timesshrinking factor, \(b\) maximum piece length×V\\times V, \(c\) maximum piece length×\\timescorpus typology, and \(d\) BPE merge frequency×\\timesUnigram maximum piece length \(both arms run permissive at once\)\. Every cell stays more than70%70\\%disjoint \(J≤0\.29J\\leq 0\.29\); overlap rises above the headline≤0\.161\\leq 0\.161range only at the shortest\-piece \(max\-piece\-length44\) setting, peaking on the narrow\-alphabet corpora\.![Refer to caption](https://arxiv.org/html/2607.05691v1/x8.png)Figure 9:Algorithm×\\timesboundary interaction: the signed NMB−\-MB difference per contrast, plotted againstVVwith corpus given by color and marker\. The panels are, top to bottom, membership overlapJJ, the signed fertility gapfBPE−fULf\_\{\\mathrm\{BPE\}\}\-f\_\{\\mathrm\{UL\}\}\(in tokens\), and the signed imbalance gapDBPE−DULD\_\{\\mathrm\{BPE\}\}\-D\_\{\\mathrm\{UL\}\}; each carries a zero baseline off which the sign reads\. No contrast’s outcome flips between the opaque\-bracket \(NMB\) and permeable\-bracket \(MB\) policies; the interaction terms are modest for overlap and imbalance and larger \(in token units\) but outcome\-preserving for fertility\. Only PubChem reachesV=2048V\{=\}2048; REAL\-Space is a singleV=1024V\{=\}1024point\.
### A\.4Learnability confirmation

The learnable\-regime check computes the clearancecnc\_\{n\}, the fraction of*learned*vocabulary pieces \(those above the shared165165\-token base, which is identical across both arms and chosen by neither algorithm\) firing at leastnntimes in the full training corpus, forn∈\{50,100,200\}n\\in\\\{50,100,200\\\}\. The barFp,nF\_\{p,n\}requirescn≥pc\_\{n\}\\geq p; the headlineF95%,100F\_\{95\\%,100\}\(c100≥0\.95c\_\{100\}\\geq 0\.95\) is reported per arm in Table[A14](https://arxiv.org/html/2607.05691#A6.T14), and the full per\-armcnc\_\{n\}sweep is in Appendix[F\.5](https://arxiv.org/html/2607.05691#A6.SS5), Table[A20](https://arxiv.org/html/2607.05691#A6.T20)\. Reading eachcnc\_\{n\}againstp∈\{0\.90,0\.95,0\.99\}p\\in\\\{0\.90,0\.95,0\.99\\\}recovers all nine\(p,n\)\(p,n\)bar outcomes from that sweep, so the bar’s robustness to both thresholds can be checked directly\.

#### The learnable ceiling and language\-model scale\.

Modern language models tokenize with vocabularies one to two orders of magnitude larger than ours \(3232K for Llama 2 and Mistral,128128K for Llama 3,256256K for Gemma\), which raises the question whether the divergence \(§[5](https://arxiv.org/html/2607.05691#S5)\) is an artifact of staying small\. It is not\. TheF95%,100F\_\{95\\%,100\}audit shows the Unigram\-LM arm crossing into the undertrained tail byV=2048V\{=\}2048even on PubChem, our largest headline corpus \(clearance0\.570\.57, below the0\.950\.95bar\), while BPE, whose coarser pieces fire more often, holds at0\.9970\.997; by theV=8192V\{=\}8192anchor both arms are unsafe \(BPE0\.680\.68, Unigram\-LM0\.130\.13\), a third of BPE’s pieces and almost nine\-tenths of Unigram\-LM’s firing too rarely to learn a reliable embedding\. The cause is the base: a158158\-glyph alphabet under hard valence and ring constraints admits few high\-frequency substructures, which is why Smirk’s merge trainer exhausts naturally atV≈4,331V\\approx 4\{,\}331\(§[4\.3](https://arxiv.org/html/2607.05691#S4.SS3)\) and why deployed chemistry tokenizers stay small \(SPE and APE learn∼\\sim3,000–5,300 pieces; §[2\.1](https://arxiv.org/html/2607.05691#S2.SS1)\)\. Language\-model vocabularies are large because they are sized to the lexical diversity of multilingual text and code, not a constrained formal language\.

## Appendix BVocabulary inventory

This appendix inventories the vocabulary in two parts: the fixed base both arms share \(Appendix[B\.1](https://arxiv.org/html/2607.05691#A2.SS1)\), and the multi\-glyph pieces each arm selects above it \(Appendix[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)\), read three ways: the complete PubChemV=256V\{=\}256snapshot, the shared core’s growth withVV, and the mirror\-image arm\-exclusive sets\.

### B\.1The base glyph alphabet

Table A2:The complete Smirk base: the 165\-token OpenSMILES glyph alphabet \(158 chemistry\-grammatical glyphs and 7 special tokens\), grouped by OpenSMILES role\. Any OpenSMILES\-conformant string decomposes into these glyphs with no\[UNK\]\. Inside a bracket atom\+and\-also denote charge, and\-doubles as the single bond\.Both arms train over the same fixed Smirk base, installed as length\-11pieces, so the only cross\-arm difference is the multi\-glyph pieces each*selects*above it \(§[3\.3](https://arxiv.org/html/2607.05691#S3.SS3)\)\. Table[A2](https://arxiv.org/html/2607.05691#A2.T2)lists all165165base tokens by OpenSMILES role\. Because any OpenSMILES\-conformant string decomposes into them with no\[UNK\], the bare base also serves as the conformance oracle of Appendix[A\.1](https://arxiv.org/html/2607.05691#A1.SS1)\.

### B\.2Learned multi\-glyph pieces

Above the shared base \(Appendix[B\.1](https://arxiv.org/html/2607.05691#A2.SS1)\), each arm selects only a small set of multi\-glyph pieces, and these learned sets are what the overlapJJ\(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\) compares\.

#### TheV=256V\{=\}256snapshot\.

Table A3:Learned multi\-glyph pieces for the PubChemV=256V\{=\}256matched pair, split three ways by cross\-algorithm membership \(shared by both arms, selected only by BPE, and selected only by Unigram\-LM\) under each boundary policy, with the per\-policy overlapJJin the block header\. Each piece is its concatenated glyph surface, ordered by glyph count then alphabetically; pieces also selected under the other boundary policy \(*boundary\-robust*: selected under both\) aregrayed, leaving the policy\-specific pieces plain\. Classes larger than 18 pieces are abridged to their first 14 and last 4 \(the parenthetical gives the full class size\); complete lists are in the data deposit\.To make the near\-disjointness concrete, we partition the*complete*learned set for the PubChemV=256V\{=\}256matched pair under both boundary policies three ways by cross\-algorithm membership \(Table[A3](https://arxiv.org/html/2607.05691#A2.T3)\)\. Of the186186distinct pieces the two arms select under NMB, only1010are shared \(J=0\.054J\{=\}0\.054;1313of183183under MB,J=0\.071J\{=\}0\.071\)\. The shared set is dominated by alkyl chains \(CCCC…CCCCCCCCCCCCCCCC\) with a few short ether and amine linkers, the motifs both algorithms agree to absorb; BPE’s exclusive pieces are short functional fragments \(=O,C=C,C\#N\) while Unigram\-LM’s run longer, toward heteroatom and homopolymeric chains \(COCCOCCOCCN,=C=C=C=C=C=C=C=C\)\. Permitting merges to cross the bracket boundary \(MB\) adds a few bracket\-internal pieces \(\[C@,\[Si\) and the aromatic ringcccccto the shared set; the picture is otherwise unchanged\. The boundary policy is the minor axis:124124of the186186NMB pieces recur under MB \(grayed in both tables; the policy\-specific ones left plain\), so the selection is largely boundary\-robust\. Of those124124, only two even change cross\-algorithm membership with the policy: the aromatic ringccccc, BPE\-only under NMB but shared under MB, and the heptyl chainCCCCCCC, shared under NMB but Unigram\-LM\-only under MB\. Neither is a bracket piece, so the policy moves their membership only indirectly, by perturbing the merge budget\.

#### Growth of the shared core acrossVV\.

Table A4:Growth of the BPE∩\\capUnigram\-LM shared core for the PubChem matched pair under each boundary policy, as the target vocabulary growsV∈\{256,512,1024,2048\}V\\in\\\{256,512,1024,2048\\\}\. The core is strictly nested \(no piece ever leaves\), so each row lists only the pieces*newly*shared at thatVV, and the cumulative core at a givenVVis that row plus all above it\. Cumulative core size grows10→50→128→31110\\to 50\\to 128\\to 311at overlapJ=0\.054,0\.076,0\.080,0\.090J=0\.054,\\,0\.076,\\,0\.080,\\,0\.090\(NMB\) and13→59→158→34513\\to 59\\to 158\\to 345at overlapJ=0\.071,0\.091,0\.100,0\.100J=0\.071,\\,0\.091,\\,0\.100,\\,0\.100\(MB\)\. Pieces also in the other boundary’s shared core \(*boundary\-robust*: shared under both\) aregrayed, leaving the boundary\-specific shared pieces plain; pieces are ordered by glyph count then alphabetically\. Layers larger than 18 pieces are abridged to their first 14 and last 4 \(the parenthetical gives the full layer size\); complete lists are in the data deposit\.The three\-way split above is a snapshot atV=256V\{=\}256; tracking the shared set up the vocabulary\-size ladder shows it to be a genuine, growing*core*\. AcrossV∈\{256,512,1024,2048\}V\\in\\\{256,512,1024,2048\\\}the PubChem BPE∩\\capUnigram\-LM set is strictly nested under both boundary policies \(no piece the two arms once agreed on is ever dropped\), growing from1010to311311pieces under NMB and1313to345345under MB, with the overlap rising fromJ=0\.054J\{=\}0\.054to0\.0900\.090\(NMB\) and from0\.0710\.071to0\.1000\.100\(MB, where it plateaus\) \(Table[A4](https://arxiv.org/html/2607.05691#A2.T4)\)\. The growth is one\-directional: the newly shared pieces are overwhelmingly substructures Unigram\-LM had already isolated at a smallerVV, which BPE’s later merges only then absorb \(175175of the183183pieces added atV=2048V\{=\}2048under NMB; the reverse essentially never occurs\), the mechanism the body invokes for the rising\-overlap trend \(§[4\.2](https://arxiv.org/html/2607.05691#S4.SS2)\)\. Even atV=2048V\{=\}2048the core is a small fraction of either arm’s∼\\sim1,890 multi\-glyph pieces, and the agreement is concentrated in alkyl chains and common heteroatom and ring motifs: the two algorithms converge only slowly, and on the least distinctive substructures\. The shared core is itself largely boundary\-robust:261261of its pieces belong to both the NMB and MB cores \(grayed in both tables\), leaving only5050NMB\-specific and8484MB\-specific shared pieces\.

#### The arm\-exclusive sets\.

Table A5:The arm\-exclusive multi\-glyph sets for the PubChem matched pair acrossVV: counts and glyph\-length statistics of the pieces each arm selects but the other does not\. The two arms select near\-equal*numbers*of exclusive pieces at everyVV, and these dwarf the shared core \(Table[A4](https://arxiv.org/html/2607.05691#A2.T4)\), so the near\-disjointness persists at every scale\. The Unigram\-LM \(UL\) exclusive pieces never exceed1616glyphs \(itsmax\_piece\_length, Table[A1](https://arxiv.org/html/2607.05691#A1.T1)\), whereas BPE imposes no length cap, so its exclusive pieces run far longer \(degenerate periodic chains such as a9696\-glyphOCCOCC…\\dotsatV=2048V\{=\}2048\)\.Exclusive piecesMean glyph len\.Max glyph len\.VVBPEULBPEULBPEUL*No\-merge\-brackets \(NMB\)*25689872\.96\.36165123053033\.87\.6191610247397375\.09\.0481620481,5811,5786\.310\.59616*Merge\-brackets \(MB\)*25686843\.05\.16165122962943\.76\.2181610247097074\.78\.1481620481,5461,5445\.910\.09616The mirror image of the shared core is what each arm keeps to itself\. Table[A5](https://arxiv.org/html/2607.05691#A2.T5)summarizes the BPE\-only and Unigram\-LM\-only multi\-glyph sets acrossVV, and three points stand out\. First, the two arms select a near\-equal*number*of exclusive pieces at everyVV\(1,5811\{,\}581vs1,5781\{,\}578atV=2048V\{=\}2048, NMB\), and these exclusive sets dwarf the shared core by roughly5×5\\timeseven at the largestVV, so the near\-disjointness is not a small\-VVartifact but persists at every scale\. Second, the fate of the two sets is asymmetric, the per\-piece face of the one\-directional catch\-up above: every BPE\-only piece atV=256V\{=\}256is still BPE\-only atV=2048V\{=\}2048\(Unigram\-LM never adopts it\), whereas mostV=256V\{=\}256Unigram\-LM\-only pieces \(7272of8787, NMB\) are absorbed into the shared core as BPE’s later merges catch up\. Third, the exclusive pieces differ sharply in length \(the table’s mean\- and max\-length columns\): Unigram\-LM’s run longer on average, BPE’s in an uncapped tail, a contrast taken up in full in Appendix[C\.2](https://arxiv.org/html/2607.05691#A3.SS2)\.

![Refer to caption](https://arxiv.org/html/2607.05691v1/x9.png)Figure 10:The learned multi\-glyph vocabularies are near\-disjoint by composition, in every matched condition\. Each bar splits the two arms’ combined learned\-piece set \(their union, normalized to100%100\\%\) outward from the center: BPE\-only pieces to the left \(blue\), Unigram\-LM\-only to the right \(orange\), with the shared core the thin neutral spine at the center, its width the unweighted overlapJJ\. The spine stays a sliver throughout, and the two wings are near\-symmetric except on the narrow\-alphabet ZINC\-22V=1024V\{=\}1024and REAL\-Space cells \(and, more mildly, COCONUTV=1024V\{=\}1024\), where Unigram\-LM’s pruning saturates and its wing shrinks\. The arms agree least where it matters most: the shared core is dominated by rare pieces, so its frequency\-weighted share is smaller still \(Jw≤0\.05J\_\{\\mathrm\{w\}\}\\leq 0\.05; Fig\.[2](https://arxiv.org/html/2607.05691#S4.F2)\)\. Rows group by corpus \(typology order\) thenVVand boundary; the right column gives the combined set’s absolute size, with per\-arm realized counts in Table[A13](https://arxiv.org/html/2607.05691#A6.T13)\.

## Appendix CStructural character of the learned pieces

The five analyses in this section, which the body reads as one signature of the greedy\-versus\-probabilistic mechanism \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\), characterize the internal structure and chemistry of each arm’s pieces: what kind of substructure each arm makes atomic \(Appendix[C\.1](https://arxiv.org/html/2607.05691#A3.SS1)\), how long its pieces run \(Appendix[C\.2](https://arxiv.org/html/2607.05691#A3.SS2)\), which glyphs co\-occur within them \(Appendix[C\.3](https://arxiv.org/html/2607.05691#A3.SS3)\), whether the pieces decompose into in\-vocabulary parts \(Appendix[C\.4](https://arxiv.org/html/2607.05691#A3.SS4)\), and whether they keep functional bonds inside one token \(Appendix[C\.5](https://arxiv.org/html/2607.05691#A3.SS5)\)\. Together they resolve a single contrast: BPE builds compositionally closed pieces that bind whole rings and functional groups, while Unigram\-LM keeps non\-closed heteroatom and unsaturated chains\. How the cross\-arm divergence varies across corpus typologies is taken up in Appendix[D](https://arxiv.org/html/2607.05691#A4); how it survives non\-canonical input \(Appendix[E](https://arxiv.org/html/2607.05691#A5)\) and out\-of\-distribution chemistry \(§[4\.5](https://arxiv.org/html/2607.05691#S4.SS5)\)\.

### C\.1Substructure classes of the learned pieces

Table A6:Substructure composition of the shared, BPE\-only, and Unigram\-LM\-only multi\-glyph sets, by corpus \(matched pairs atV=1024V\{=\}1024, NMB\)\. Each piece is classed from its glyphs by priority aromatic atom\>\>aliphatic heteroatom\>\>unsaturated carbon\>\>saturated carbon, and each row gives the percentage of that set’s pieces in each class\. Aromatic\-ring pieces are a BPE specialty the Unigram\-LM arm almost never forms; bracket\-internal pieces arise only under MB \(§[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)\) and do not occur here\.The piece lists \(Appendix[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)\) suggest the two arms favor different*kinds*of substructure; Table[A6](https://arxiv.org/html/2607.05691#A3.T6)classes every multi\-glyph piece and confirms it\. The aromatic\-ring split the body reads off it \(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\) is the clearest signal: BPE merges ring fragments \(cccc,ccccc\) into whole tokens where Unigram\-LM stays near\-atomic\. On Unigram\-LM’s side the table gives the mirror image: its exclusive pieces skew to aliphatic heteroatom chains, and on the natural\-products corpus to long unsaturated carbon runs \(49%49\\%of COCONUT’s Unigram\-LM\-only set, the cumulated=C=C=…\\dots\(cumulene\) motifs\)\. The shared core, by contrast, is heteroatom\-led like Unigram\-LM’s, but it also holds the saturated alkyl and \(on COCONUT\) aromatic motifs both algorithms absorb early: the common substructures, not either arm’s specialty\.

### C\.2Piece\-length distributions

![Refer to caption](https://arxiv.org/html/2607.05691v1/x10.png)Figure 11:Glyph\-length distribution of the BPE\-only and Unigram\-LM\-only multi\-glyph pieces \(PubChem, ZINC\-22, COCONUT;V=1024V\{=\}1024, NMB\)\. Unigram\-LM is capped at1616glyphs \(max\_piece\_length, dotted line\); BPE is uncapped, with a corpus\-dependent tail \(annotated maximum per panel\)\. Each panel’sxx\-axis spans its own length range \(theyy\-axis is shared\), so the annotated maxima, not the on\-page tail widths, carry the cross\-corpus comparison\. On ZINC\-22 the narrow alphabet keeps even BPE’s longest piece below the cap\.![Refer to caption](https://arxiv.org/html/2607.05691v1/x11.png)Figure 12:Unigram\-LM piece\-length distribution on PubChem \(V=1024V\{=\}1024, NMB\) as itsmax\_piece\_lengthcap is swept over\{4,8,16,32,64,128\}\\\{4,8,16,32,64,128\\\}\(dotted lines mark each cap\)\. The right edge tracks the cap exactly, so the1616\-glyph wall in Figure[11](https://arxiv.org/html/2607.05691#A3.F11)is the hyperparameter, not an intrinsic limit; the mode stays short throughout\.Figure[11](https://arxiv.org/html/2607.05691#A3.F11)plots the glyph\-length distribution of each arm’s exclusive pieces, and at first it cuts against the fertility gap\.*On average*Unigram\-LM’s pieces are the longer ones, in all2222matched conditions \(mean glyph length9\.09\.0vs\.5\.05\.0on PubChemV=1024V\{=\}1024,8\.98\.9vs\.5\.15\.1on ZINC\-22V=1024V\{=\}1024\)\. What BPE owns is the*tail*, and only where the corpus supports long repeats: uncapped, it accretes a few very long merges on diverse PubChem \(to4848glyphs,9696atV=2048V\{=\}2048\) and scaffold\-rich COCONUT \(to3232\), while Unigram\-LM stops at a hard wall at itsmax\_piece\_lengthdefault of1616\(Table[A1](https://arxiv.org/html/2607.05691#A1.T1)\)\. On narrow\-alphabet ZINC\-22 and REAL\-Space, BPE cannot form such repeats, and even its longest exclusive piece \(1010–1313glyphs\) is*shorter*than Unigram\-LM’s cap\. The mean and the tail thus disagree: Unigram\-LM wins the mean everywhere, BPE the tail only where long substructure exists to merge\. That asymmetry is moreover partly an artifact of Unigram\-LM’s cap: retraining it across themax\_piece\_lengthladder \(Figure[12](https://arxiv.org/html/2607.05691#A3.F12)\) tracks the right edge from44to128128glyphs and lifts the mean \(3\.23\.2to15\.015\.0\), yet even at the most permissive cap the bulk stays short, the cap shifting the tail and not the mode\. None of this coarsens Unigram\-LM’s segmentation: it samples likely segmentations rather than greedily applying its longest pieces, so its longer vocabulary pieces coexist with its higher fertility \(more, shorter tokens in use\)\.

### C\.3Glyph co\-occurrence within learned pieces

![Refer to caption](https://arxiv.org/html/2607.05691v1/x12.png)Figure 13:Adjacent base\-glyph pairs within learned multi\-glyph pieces \(PubChemV=2048V\{=\}2048\), over the core backbone glyphs \(organic, aromatic, bond, charge, chirality, and structural glyphs, grouped by OpenSMILES role with dividers; rare bracket\-atom element symbols omitted\)\. Rows are the boundary policy \(no\-merge\-brackets NMB, merge\-brackets MB\); the first two columns are Smirk\-GPE \(BPE\) and Unigram\-LM on a shared log count scale, the third their log\-ratio \(blue: BPE\-heavier, red: Unigram\-LM\-heavier; cells empty in both are grey\)\. Under NMB the bracket delimiters\[/\]never co\-occur \(merges cannot cross them\); MB switches the\[row on\. Both arms are dominated by the carbon backbone, weighted differently: on average Unigram\-LM builds longer, more homo\-atomic pieces\.The piece inventories can also be read at the level of constituent glyphs: for each learned multi\-glyph piece we count its adjacent base\-glyph pairs \(CCOcontributesC–CandC–O\)\. Figure[13](https://arxiv.org/html/2607.05691#A3.F13)maps these for the PubChemV=2048V\{=\}2048pair over the core backbone glyphs, grouped by OpenSMILES role \(the∼\\sim100100rare bracket\-atom symbols, which merge sparsely and only under MB, are omitted; Appendix[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)\)\. The glyph level adds two findings the piece\-level tables cannot show\. First, the boundary policy is a structural switch: under no\-merge\-brackets \(NMB\)*no*adjacency touches a bracket delimiter \(\[,\]; merges cannot cross the boundary\), whereas merge\-brackets \(MB\) turns on the\[row of bracket\-internal pairs \(\[–C,\[–N,\[–ring\-digit;3\.4%3\.4\\%of BPE adjacencies and1\.5%1\.5\\%of Unigram\-LM’s then involve\[\)\. Second, at near\-equal piece counts Unigram\-LM packs∼\\sim1\.65×1\.65\\timesas many adjacencies, the glyph\-level face of its longer pieces \(Appendix[C\.2](https://arxiv.org/html/2607.05691#A3.SS2)\)\. Both arms anchor on theC–Cbackbone but weight it differently, echoing the composition split \(right column\): Unigram\-LM leans to sulfur and oxygen chains \(C–S,O–O\), BPE into the aromatic–aliphatic junction \(C–c\)\.

### C\.4Compositional closure of the learned vocabularies

Table A7:Per\-condition within\-arm compositional closure\. Per matched pair \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\), read from the realized vocabulary alone\.cbinc\_\{\\mathrm\{bin\}\}is the binary\-split closure: the fraction of multi\-glyph pieces with some in\-vocab splitp=a⋅bp\{=\}a\{\\cdot\}b\(both parts in the base\-plus\-multi vocabulary\)\. It is BPE’s merge\-closure invariant read off the realized set, socbinBPE=1c\_\{\\mathrm\{bin\}\}^\{\\mathrm\{BPE\}\}\{=\}1exactly \(the correctness anchor\) and the orphan ratecorphBPE=0c\_\{\\mathrm\{orph\}\}^\{\\mathrm\{BPE\}\}\{=\}0, the latter omitted for BPE\.corphULc\_\{\\mathrm\{orph\}\}^\{\\mathrm\{UL\}\}is the Unigram\-LM \(UL\) orphan rate: the fraction of its length\-≥3\\geq 3pieces with*no*proper≥2\\geq 2\-glyph sub\-piece in vocabulary \(pieces that share no building block with the rest of the vocabulary\)\.cfullc\_\{\\mathrm\{full\}\}is the stronger full\-substring closure \(every≥2\\geq 2\-glyph substring in vocabulary\), non\-trivial for both arms\. Closure is an exact\-set quantity and carries no CI\. Unigram\-LM is far less self\-referential than BPE in every condition: roughly half its pieces \(as few as0\.120\.12on the narrowest alphabet\) do not decompose into in\-vocab parts\.The arm\-exclusive sets \(Appendix[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)\) and length distributions \(Appendix[C\.2](https://arxiv.org/html/2607.05691#A3.SS2)\) describe*which*pieces each arm keeps; a complementary question is how those pieces relate to one another*within*a single vocabulary\. BPE answers it by construction: built bottom\-up, every learned piece is the concatenation of two pieces already in the vocabulary, so the BPE vocabulary is*compositionally closed*\. Unigram\-LM, pruning a seed pool top\-down by likelihood, is under no such constraint, so whether its pieces decompose into in\-vocab parts is an empirical property\. We read it off the realized vocabularies alone \(no corpus\); it gives a construction\-independent view of the same greedy\-versus\-probabilistic mechanism the body traces \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\)\.

We measure closure identically for both arms from the realized set, so BPE’s value is a measured anchor rather than a restatement of its rule\. Let𝒱\\mathcal\{V\}be the full realized vocabulary \(the165165\-token base plus the learned multi\-glyph pieces\) andM=\{p∈𝒱:\|p\|≥2\}M=\\\{p\\in\\mathcal\{V\}:\|p\|\\geq 2\\\}the multi\-glyph set the overlap compares\. Because the base is complete, every single glyph is trivially in𝒱\\mathcal\{V\}, so the metrics range over splits and sub\-pieces of length≥2\\geq 2, where the question is non\-trivial\. We report three \(Table[A7](https://arxiv.org/html/2607.05691#A3.T7)\): the*binary\-split closure*cbinc\_\{\\mathrm\{bin\}\}, the fraction ofMMadmitting a splitp=a⋅bp\{=\}a\{\\cdot\}bwith both parts in𝒱\\mathcal\{V\}\(BPE’s merge\-closure invariant, socbinBPE=1c\_\{\\mathrm\{bin\}\}^\{\\mathrm\{BPE\}\}\{=\}1exactly\); the*orphan rate*corphc\_\{\\mathrm\{orph\}\}, the fraction of length\-≥3\\geq 3pieces with*no*proper≥2\\geq 2\-glyph sub\-piece in𝒱\\mathcal\{V\}\(pieces that share no building block with the rest of the vocabulary, zero for BPE by the same invariant\); and the stronger*full\-substring closure*cfullc\_\{\\mathrm\{full\}\}, the fraction whose every≥2\\geq 2\-glyph substring is in𝒱\\mathcal\{V\}, which is non\-trivial for both arms\.

Three readings\. First, Unigram\-LM is far less self\-referential than BPE in every one of the2222matched conditions:cbinULc\_\{\\mathrm\{bin\}\}^\{\\mathrm\{UL\}\}runs0\.360\.36–0\.640\.64on the diverse and natural\-products corpora and falls to0\.120\.12on the narrowest alphabet \(ZINC\-22V=256V\{=\}256\), so roughly half its pieces, and on ZINC\-22 nearly nine in ten, do not decompose into in\-vocab parts\. Second, the gap survives the stronger metric and an exact BPE anchor: even full\-substring closure separates the arms by a wide margin \(cfullBPEc\_\{\\mathrm\{full\}\}^\{\\mathrm\{BPE\}\}0\.550\.55–0\.770\.77againstcfullUL≤0\.05c\_\{\\mathrm\{full\}\}^\{\\mathrm\{UL\}\}\\leq 0\.05\), and that BPE’s owncfullc\_\{\\mathrm\{full\}\}sits well below11confirms the measurement is reading genuine structure, not echoing the merge rule \(merge\-closure guarantees one split, not all substrings\)\. Third, the orphan population is largely the arm\-exclusive tail the other appendices isolate: Unigram\-LM’s non\-closed pieces are overwhelmingly structural rather than bracket\-internal and are the long heteroatom chains \(COCOCOCOCOC,OCCOOCCOOCCO,OCCONOCCO\) it keeps without keeping their parts \(Appendix[C\.1](https://arxiv.org/html/2607.05691#A3.SS1), Appendix[C\.2](https://arxiv.org/html/2607.05691#A3.SS2)\), because likelihood scores each piece independently where merging must first own the sub\-pieces\. The orphan rate is non\-increasing inVV\(PubChem NMB0\.45→0\.180\.45\\to 0\.18overV=256→1024V\{=\}256\\to 1024\), the same convergence\-from\-below the overlap and imbalance trends show, but it never vanishes, and on the narrow alphabets its orphans are the saturated arm’s pruned\-back survivors \(Appendix[D\.1](https://arxiv.org/html/2607.05691#A4.SS1)\)\.

### C\.5Chemical functional\-bond locality

The closure and composition analyses \(Appendix[C\.4](https://arxiv.org/html/2607.05691#A3.SS4), Appendix[C\.1](https://arxiv.org/html/2607.05691#A3.SS1)\) describe each vocabulary as a set of pieces; a complementary question is whether the*realized segmentation*respects chemistry: whether the learned tokens align with chemically meaningful units, the cheminformatic analogue of asking whether a subword tokenizer’s boundaries respect morphemes\. We make this precise with the chemically salient bonds hardest to keep whole: the*multiply\-bonded heteroatoms*\. A non\-carbon atom joined by a double or triple bond is the core of a canonical functional group: the==O of a carbonyl, the\#\\\#N of a nitrile, the==N of an imine, the==O on sulfur, phosphorus, or nitrogen \(sulfonyl, phosphoryl, nitro\), the==S of a thiocarbonyl\. It is read straight off the molecular graph, so the measure needs no curated list of surface forms\. On each held\-out molecule, for every such bond, we ask whether the arm keeps it*token\-local*: whether the heteroatom shares a token with its==/\#\\\#bond glyph\. We measure locality per heteroatom\-and\-bond rather than per whole functional group because SMILES branch syntax scatters even a two\-atom group across a parenthesis \(a carbonyl is writtenC\(=O\), its carbon a shared backbone atom\), so the chemically meaningful unit is the==O, not the fullC\(=O\)\.

Table A8:Per\-condition within\-arm chemical functional\-bond locality\. Per matched pair \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\), read on the held\-out split\. A*functional bond*is a multiply\-bonded heteroatom, a non\-carbon atom joined by a double or triple bond \(the==O of a carbonyl, the\#\\\#N of a nitrile, the==N of an imine, the==O on sulfur, phosphorus, or nitrogen, the==S of a thiocarbonyl\), the cores of the canonical functional groups, read straight off the molecular graph\. Localityℓ\\ellis the fraction of those bonds the arm keeps inside a single token \(the heteroatom sharing a token with its bond glyph\);Δ​ℓ=ℓBPE−ℓUL\\Delta\\ell\{=\}\\ell^\{\\mathrm\{BPE\}\}\{\-\}\\ell^\{\\mathrm\{UL\}\}\.ℓC=O\\ell\_\{\\mathrm\{C\{=\}O\}\}isolates the carbonyl class\. BPE keeps nearly every functional bond local \(the carbonyl essentially always\); Unigram\-LM \(UL\) keeps almost none \(the carbonyl columnℓC=OUL\\ell^\{\\mathrm\{UL\}\}\_\{\\mathrm\{C\{=\}O\}\}is0\.0000\.000to three places in every condition\), spending its unsaturation budget on long homo\-atomic carbon runs, not on binding the heteroatom to its bond\. 95% molecule\-resampled bootstrap CIs are in the deposited per\-condition records\.Table[A8](https://arxiv.org/html/2607.05691#A3.T8)reports per\-arm localityℓ\\elland the gapΔ​ℓ=ℓBPE−ℓUL\\Delta\\ell=\\ell^\{\\mathrm\{BPE\}\}\-\\ell^\{\\mathrm\{UL\}\}over the2222matched conditions\. The separation is near\-total and uniform: BPE keeps0\.950\.95–1\.001\.00of functional bonds inside a single token, Unigram\-LM at most0\.030\.03, a gap of0\.930\.93–0\.990\.99on every corpus, vocabulary size, and boundary policy\. The carbonyl makes it sharpest: BPE binds the==O in essentially every occurrence \(ℓC=OBPE≥0\.995\\ell^\{\\mathrm\{BPE\}\}\_\{\\mathrm\{C=O\}\}\\geq 0\.995\), while Unigram\-LM does so in*none*, withℓC=OUL=0\.000\\ell^\{\\mathrm\{UL\}\}\_\{\\mathrm\{C=O\}\}=0\.000to three places in all2222conditions, over held\-out splits of up to10610^\{6\}molecules\. The one class Unigram\-LM keeps with any regularity is the nitrile \(ℓC​\#​NUL\\ell^\{\\mathrm\{UL\}\}\_\{\\mathrm\{C\\\#N\}\}up to0\.260\.26\), the terminalC\#Nthat is also one of its own short pieces\.

This is the in\-context face of the vocabulary findings\. The orphan rate \(Appendix[C\.4](https://arxiv.org/html/2607.05691#A3.SS4)\) and the arm\-exclusive chains \(Appendix[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)\) say Unigram\-LM spends its multi\-glyph budget on long homo\-atomic and unsaturated runs; locality confirms the consequence on real molecules\. Where BPE’s frequency\-greedy merging crystallizes the small functional motif \(=O,C\#N\) as a token, Unigram\-LM’s likelihood pruning leaves the heteroatom split from its bond, because the same==glyph it declines to bind to oxygen it spends on the=C=C=Ccumulene runs it keeps exclusively \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\)\. The reading is on the held\-out split with95%95\\%molecule\-resampled bootstrap CIs; at these denominators the intervals are tight to the third decimal\.

## Appendix DCorpus\-specific contrasts

Appendix[B\.2](https://arxiv.org/html/2607.05691#A2.SS2)inventories the learned pieces on PubChem’s diverse alphabet; this section follows the membership contrast across corpus typologies: the narrow drug\-like alphabet of ZINC\-22 \(Appendix[D\.1](https://arxiv.org/html/2607.05691#A4.SS1)\), the natural\-products corpus COCONUT \(Appendix[D\.2](https://arxiv.org/html/2607.05691#A4.SS2)\), and whether the cross\-algorithm agreement is itself corpus\-specific \(Appendix[D\.3](https://arxiv.org/html/2607.05691#A4.SS3)\)\.

### D\.1Narrow\-alphabet contrast: ZINC\-22

Table A9:The ZINC\-22 matched pair acrossVV: each arm’s multi\-glyph vocabulary size, the three\-way cross\-algorithm split \(shared / BPE\-only / Unigram\-LM\-only\), and the overlapJJ, under both boundary policies \(UL denotes Unigram\-LM\)\.Multi\-glyph vocab\.Cross\-arm splitVVBPEULSharedBPE\-onlyUL\-onlyJJ*No\-merge\-brackets \(NMB\)*2569897296950\.010512355310183372920\.0281024867310997682110\.092*Merge\-brackets \(MB\)*2569897494930\.021512355353253303280\.03710248673571267412310\.115ZINC\-22, the drug\-like corpus, has a far narrower alphabet than PubChem, where both arms reach the target vocabulary; Table[A9](https://arxiv.org/html/2607.05691#A4.T9)shows the narrow alphabet changes the picture in three ways\. First, the divergence is*sharper*at smallVV: the overlap isJ=0\.010J\{=\}0\.010atV=256V\{=\}256\(NMB\), roughly five times lower than PubChem’s0\.0540\.054, with only two shared multi\-glyph pieces\. Second, the Unigram\-LM arm*saturates*: its narrow alphabet offers too few high\-likelihood pieces for the pruning to reach target, so it settles at∼\\sim310310pieces and adds essentially nothing fromV=512V\{=\}512toV=1024V\{=\}1024, while BPE fills to target \(867867\)\. The two vocabularies are thus sharply unequal in size \(867867vs310310atV=1024V\{=\}1024\) where on PubChem they are near\-equal \(867867vs865865\); atV=2048V\{=\}2048the Unigram\-LM arm, unsafe by a wide margin, is left untrained, leaving no matched pair\. Third, the Unigram\-LM\-only set*shrinks*withVV\(292→211292\\to 211, NMB\), the opposite of PubChem: Unigram\-LM stops adding pieces while BPE’s later merges absorb its small fixed set into the shared core\. The overlap does rise withVV\(to0\.0920\.092/0\.1150\.115atV=1024V\{=\}1024\), but partly mechanically: the size gap caps the attainable overlap at\|𝒱ULmulti\|/\|𝒱BPEmulti\|≈0\.36\|\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{\\mathrm\{UL\}\}\|/\|\\mathcal\{V\}^\{\\mathrm\{multi\}\}\_\{\\mathrm\{BPE\}\}\|\\approx 0\.36, and the observed value is about a quarter of that ceiling, so genuine disagreement still carries the result \(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\)\. The combinatorial REAL\-Space corpus, narrow\-alphabet for the same reason, behaves identically: its size gap caps the attainable overlap at≈0\.31\\approx 0\.31, and the observed0\.1050\.105sits at about a third of that ceiling\. On a narrow alphabet, then, the two algorithms diverge for a second reason beyond selecting different pieces: Unigram\-LM simply runs out of pieces to select\.

### D\.2Natural\-products contrast: COCONUT

Table A10:The COCONUT matched pair acrossVV: each arm’s multi\-glyph vocabulary size, the three\-way cross\-algorithm split \(shared / BPE\-only / Unigram\-LM\-only\), and the overlapJJ, under both boundary policies \(UL denotes Unigram\-LM\)\.Multi\-glyph vocab\.Cross\-arm splitVVBPEULSharedBPE\-onlyUL\-onlyJJ*No\-merge\-brackets \(NMB\)*25698972474730\.140512355353652902880\.10110248676401397285010\.102*Merge\-brackets \(MB\)*25698972771700\.161512355353832722700\.13310248677691746935950\.119COCONUT, the natural\-products corpus, completes the typology and inverts the cross\-VVtrend\. Table[A10](https://arxiv.org/html/2607.05691#A4.T10)shows the overlap is*highest*at smallVV\(J=0\.161J\{=\}0\.161atV=256V\{=\}256, MB; about twice the highest small\-VVoverlap on any other corpus\) and*falls*asVVgrows \(to0\.1190\.119atV=1024V\{=\}1024\), the opposite of PubChem and ZINC\-22, where it rises\. The reason is visible in the split: both arms agree on far more pieces at smallVVthan elsewhere \(2424–2727shared atV=256V\{=\}256, against PubChem’s1010and ZINC\-22’s22\), because the dominant natural\-product motifs \(long alkyl chains and aromatic rings\) are frequent enough that both algorithms absorb them immediately\. The shared core still grows in absolute terms \(24→65→13924\\to 65\\to 139, NMB\), but the near\-disjoint remainder grows faster, so the shared*fraction*dilutes\. As the smallest corpus, COCONUT also begins to starve the Unigram\-LM arm atV=1024V\{=\}1024\(it realizes640640pieces against BPE’s867867under NMB\), a milder version of the ZINC\-22 saturation\. This early\-agreement\-then\-dilution is not a step toward convergence: the large\-VVanchor \(§[4\.2](https://arxiv.org/html/2607.05691#S4.SS2)\) confirms the remainder stays near\-disjoint well beyond the grid\.

### D\.3Universality of the agreement across corpora

A final question: when the two algorithms*do*agree, do they agree on the*same*pieces regardless of corpus? Largely not\. The three shared cores atV=1024V\{=\}1024\(NMB; sizes128128,9999, and139139for PubChem, ZINC\-22, and COCONUT\) overlap only weakly:2323pieces between PubChem and ZINC\-22,3131between PubChem and COCONUT, and1515between ZINC\-22 and COCONUT\. Just99pieces belong to all three cores\. That universal core is chemically unremarkable: long alkyl chains \(CCCCCCCCC,CCCCCCCCCC,CCCCCCCCCCC\) plus a handful of generic motifs \(N\#C,C\#CC,SCCS,C\#CCN,SCCCS,C=CC=CC\)\. Even cross\-algorithm agreement is therefore largely corpus\-specific: each shared core is dominated by pieces frequent in*that*corpus, not a universal chemical vocabulary the two algorithms converge on\.

## Appendix ERobustness to non\-canonical SMILES

![Refer to caption](https://arxiv.org/html/2607.05691v1/x13.png)Figure 14:Write\-stability under non\-canonical SMILES atV=1024V\{=\}1024, one dumbbell per corpus×\\timesboundary joining BPE \(blue\) to Unigram\-LM \(orange\) by bag\-instability \(fraction of the token multiset that changes on rewrite\), with the rewrite axes on a shared scale from mildest \(OpenBabel\) to catastrophic \(Explicit\-H\)\. The arm gap flips sign by axis: Unigram\-LM is the more write\-stable arm \(its dot left of BPE’s\) under randomization and the cross\-toolkit OpenBabel swap, but the*less*stable under Kekulé \(dot to the right\), which destroys the aromatic\-lowercase pieces it leans on\. Explicit hydrogens are catastrophic for both arms\. All2222conditions, with CIs, are in Table[A11](https://arxiv.org/html/2607.05691#A5.T11)\.Table A11:Per\-condition within\-arm robustness to non\-canonical SMILES\. Per matched pair \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\), on a seeded held\-out subsample; the paired columnsbBPEb^\{\\mathrm\{BPE\}\}andbULb^\{\\mathrm\{UL\}\}are the BPE and Unigram\-LM \(UL\) arms\.bbis the*bag\-instability*: the fraction of an arm’s token multiset that changes when the molecule is rewritten \(mean11minus the multiset Jaccard versus the canonical string\), under four identity\-preserving rewrite axes: Randomized \(RDKit restricted randomization, the augmentation\-realistic distribution ofArús\-Pous et al\. \[[3](https://arxiv.org/html/2607.05691#bib.bib3)\]\), Kekulé, Explicit\-H \(all\-explicit\-hydrogen, AMORE’s catastrophic axis\[[10](https://arxiv.org/html/2607.05691#bib.bib10)\]\), and OpenBabel \(the cross\-toolkit swap to OpenBabel’s canonical SMILES, gated to identity\-preservation by a round\-trip through RDKit\)\.gcg\_\{\\mathrm\{c\}\}andgrg\_\{\\mathrm\{r\}\}are the relative fertility gaprel​\|Δ​f\|\\mathrm\{rel\}\|\\Delta f\|\(Eq\.[2](https://arxiv.org/html/2607.05691#S3.E2)\) on the canonical strings and on the randomized orbit; their closeness shows the granularity gap survives the orbit\. A ring\-digit\-relabel floor \(omitted\) leaves the token count exactly invariant\. Figure[14](https://arxiv.org/html/2607.05691#A5.F14)plots the per\-axis arm\-stability pattern atV=1024V\{=\}1024; 95% molecule\-resampled bootstrap CIs are in the deposited per\-condition records\.Every measurement so far reads the tokenizers on the corpus’s*canonical*held\-out strings, yet a molecule has many equivalent SMILES, and a deployed model sees non\-canonical ones routinely \(a different toolkit’s canonical form, or the randomized strings used for augmentation\)\. We ask, model\-free on the trained families, what happens to each arm’s segmentation across that rewrite orbit, and whether the two algorithms differ\. For a seeded subsample of each held\-out split \(10,00010\{,\}000molecules\) we build five identity\-preserving rewrites per molecule\. Four are RDKit\-internal, ordered mild to catastrophic to match the difficulty spectrum a chemical*model*shows: a ring\-closure\-digit relabel; a Kekulé form;KKrandomized SMILES \(RDKit’s default*restricted*randomization, the augmentation\-realistic distribution; the unrestricted variant is a strict superset and would only widen the gaps\); and an all\-explicit\-hydrogen form \(Cto\[CH4\]\)\. The fifth swaps in OpenBabel’s canonical SMILES in place of RDKit’s, gated to identity\-preservation by a round\-trip back through RDKit; unlike the four rewrites of RDKit’s own output, it is a second canonicalizer’s legitimate form, the most realistic non\-canonical input a multi\-source pipeline meets, and a molecule on which the two toolkits agree contributes a genuine zero\. For each arm and axis we measure two senses of movement against the canonical string: the relative fertility dispersion \(does the token*count*move\) and the*bag\-instability*, the fraction of the token multiset that changes \(does the set of*pieces*move\), each with a95%95\\%molecule\-resampled bootstrap CI\. Table[A11](https://arxiv.org/html/2607.05691#A5.T11)shows the bag\-instability per axis and the cross\-arm relative fertility gap on canonical versus randomized strings \(gcg\_\{\\mathrm\{c\}\},grg\_\{\\mathrm\{r\}\}\); the per\-arm dispersions are deposited but not tabulated\.

Five readings\. First, the granularity gap is not a canonical\-notation artifact: read on the randomized orbit, Unigram\-LM stays2323–35%35\\%finer than BPE in every one of the2222matched conditions, attenuating only slightly from the canonical gap\. Both arms’ absolute fertility rises off\-canonical \(canonical strings are structurally simpler, with fewer branches and brackets\), so the reported canonical fertilities lower\-bound what a deployed model emits; the relative gap itself is marginally*smaller*on the randomized orbit \(gr<gcg\_\{\\mathrm\{r\}\}<g\_\{\\mathrm\{c\}\}in2121of2222cells\), so the canonical gap is, if anything, a slight over\-estimate of the deployed one\. Second, Unigram\-LM is the more write\-stable arm: its randomization bag\-instability runs0\.110\.11–0\.220\.22against BPE’s0\.240\.24–0\.380\.38, lower in all2222cells\. The mechanism is the same greedy\-versus\-probabilistic split the body traces: BPE applies a fixed merge*sequence*, so reordering the glyph stream changes which pairs merge first and the realized pieces shift; Unigram\-LM scores each piece independently and Viterbi\-decodes, a near\-context\-free segmentation that moves less\. Third, the arm ranking*flips*by axis \(Figure[14](https://arxiv.org/html/2607.05691#A5.F14)\): under Kekulé, Unigram\-LM is the*less*stable arm \(bag\-instability above BPE’s in2121of2222cells\), because Kekulé destroys every aromatic\-lowercase piece and Unigram\-LM leans hardest on aromatic runs \(Appendix[C\.1](https://arxiv.org/html/2607.05691#A3.SS1)\)\. So which algorithm is more robust depends on the rewrite, a richer answer than a single ranking\.

Fourth, the tokenizer’s difficulty ordering diverges from a model’s\. The ring\-digit relabel leaves the token count*exactly*invariant \(a clean floor\), and Kekulé barely moves it, yet Kekulé is the most piece\-destructive of the mild axes: it preserves how*many*tokens but rewrites*which*ones, a transform a model finds easy\. Explicit hydrogens are catastrophic for both arms \(bag\-instability0\.540\.54–0\.830\.83, with the token count inflating several\-fold\), the worst axis at the tokenizer level as it is for models, because every bare atom becomes a bracketed atom the learned pieces rarely fired on in an implicit\-hydrogen training corpus, the same embedding\-tail starvation the dead\-zone confirmation measures \(§[3\.5](https://arxiv.org/html/2607.05691#S3.SS5)\); the merge\-brackets policy softens it for BPE, which can then absorb bracket\-internal glyphs, tying this axis back to the boundary axis of the grid\.

Fifth, the write\-stability gap survives the most realistic non\-canonical input of all: a second toolkit’s canonical form\. Swapping each molecule’s RDKit canonical SMILES for OpenBabel’s, Unigram\-LM’s bag\-instability runs0\.090\.09–0\.150\.15against BPE’s0\.190\.19–0\.240\.24, lower in all2222cells, so an independent canonicalizer moves BPE’s pieces about twice as far as Unigram\-LM’s\. This is the strongest form of the stability finding: the gap is not an artifact of RDKit’s own randomizer or rewrite rules, but holds against a genuinely separate implementation that a multi\-source corpus encounters whenever its records were canonicalized by different software\. The swap is milder than full randomization \(a single canonical form rather than the average ofKKrandom rewrites\) yet sits well above the ring\-relabel floor, and like randomization shifts*which*pieces are chosen, not how many\.

## Appendix FPer\-condition measurement tables

The body reports the headline values; the exhaustive per\-condition measurements behind them are collected here, the summary tables first and then four families of measurement\-specific detail\. Tables tied to a specific analysis stay with it instead: compositional closure and functional\-bond locality in Appendix[C](https://arxiv.org/html/2607.05691#A3), non\-canonical rewrites in Appendix[E](https://arxiv.org/html/2607.05691#A5)\.

### F\.1Overlap, vocabulary, and summary scalars

Table[A12](https://arxiv.org/html/2607.05691#A6.T12)gives the four vocabulary\-overlap weightings, Table[A13](https://arxiv.org/html/2607.05691#A6.T13)the realized per\-arm multi\-glyph vocabulary, Table[A14](https://arxiv.org/html/2607.05691#A6.T14)the dead\-zone surplus with per\-armF95%,100F\_\{95\\%,100\}clearance, and Table[A15](https://arxiv.org/html/2607.05691#A6.T15)the robustness probes, the one table here that steps outside the per\-condition grid\. These expand the direct\-contrast columns of the body’s Table[2](https://arxiv.org/html/2607.05691#S4.T2)\. All are at the SentencePiece defaultL=16L\{=\}16\.

The two structural variants drop*bracket\-internal*pieces \(those whose every training occurrence falls inside a bracketed atom\) and recompute the overlap over the survivors:JstructJ\_\{\\mathrm\{struct\}\}the unweighted Jaccard \(Eq\.[1](https://arxiv.org/html/2607.05691#S3.E1)\) andJw,structJ\_\{\\mathrm\{w,struct\}\}the frequency\-weighted one \(Eq\.[4](https://arxiv.org/html/2607.05691#S3.E4)\), each arm’s weights renormalized over the structural held\-out mass and carrying its own bootstrap CI\.

Table A12:Cross\-algorithm vocabulary overlap \(Jaccard\)\.JJunweighted,JstructJ\_\{\\mathrm\{struct\}\}structural\-subword,JwJ\_\{\\mathrm\{w\}\}frequency\-weighted,Jw,structJ\_\{\\mathrm\{w,struct\}\}frequency\-weighted over structural subwords only; all four are vocabulary Jaccards on matched arm pairs\.JJandJstructJ\_\{\\mathrm\{struct\}\}are exact set quantities; the molecule\-resampled bootstrap CIs onJwJ\_\{\\mathrm\{w\}\}andJw,structJ\_\{\\mathrm\{w,struct\}\}span≤0\.002\\leq 0\.002in every condition and are deposited in full\. The smallest and largestJJare inbold\.![Refer to caption](https://arxiv.org/html/2607.05691v1/x14.png)Figure 15:Structural\-subword overlap for all2222matched conditions, the appendix companion to Fig\.[2](https://arxiv.org/html/2607.05691#S4.F2): frequency\-weighted \(Jw,structJ\_\{\\mathrm\{w,struct\}\}, left\) and unweighted \(JstructJ\_\{\\mathrm\{struct\}\}, right\), recomputed after dropping bracket\-internal pieces\. Same encoding as Fig\.[2](https://arxiv.org/html/2607.05691#S4.F2)\. Restricting to structural pieces barely moves the overlap \(every condition stays near\-disjoint\), so the near\-disjointness is genuine cross\-pretoken disagreement, not an artifact of how bracketed atoms are split\.Table A13:Realized per\-arm multi\-glyph vocabulary per condition\. Per arm, the sets the Jaccards compare, excluding the shared165165\-token base\. BPE keeps the full atomic base and fills the rest with merges to target; Unigram\-LM \(UL\) prunes to at or below target and may shed rarely\-used base glyphs, reallocating to multi\-glyph pieces, so on diverse corpora it packs comparably many \(UL/BPE near11\)\. On narrow alphabets at largerVVits pruning runs short of high\-likelihood pieces and bottoms out well below target \(UL/BPE≪1\\ll 1\)\. The Jaccard is a set ratio robust to this gap; its effect on the overlap ceiling is discussed in §[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\.![Refer to caption](https://arxiv.org/html/2607.05691v1/x15.png)Figure 16:Cross\-VVtrends of the three direct contrasts across the three multi\-VVcorpora \(the single\-VVREAL\-Space anchor is omitted\), the visual companion to Table[2](https://arxiv.org/html/2607.05691#S4.T2); color and marker give corpus, solid vs\. dashed the boundary policy \(NMB, MB\)\. Overlap rises withVVon PubChem and ZINC\-22 and falls on COCONUT, staying near\-disjoint throughout; the relative fertility gap rises slightly withVVand stays large; the imbalance gap\|Δ​D\|\|\\Delta D\|shrinks monotonically withVVbut stays above the noise floor at everyVV\.Table A14:Dead\-zone surplus: per\-armF95%,100F\_\{95\\%,100\}clearancec100c\_\{100\}and its cross\-arm differenceΔ​c100\\Delta c\_\{100\}\.c100BPEc\_\{100\}^\{\\mathrm\{BPE\}\},c100ULc\_\{100\}^\{\\mathrm\{UL\}\}\(columns BPE, UL; UL==Unigram\-LM\) are the fraction of each arm’s vocabulary clearing theF95%,100F\_\{95\\%,100\}rare\-token\-tail bar;Δ​c100\\Delta c\_\{100\}is their difference \(BPE−\-UL\)\.†\\daggercorpus too small to certify the tail at thisVV\.Table A15:Robustness extras: subsample, size sweep, seed\-cap, prune\-schedule, merge\-exhaustion\. Outside the headline grid\. Subsample redraw: Unigramc100c\_\{100\}across three independent draws of the same corpus\. Size sweep: the same cell trained at 5M / 15M / 50M molecules\. Seed cap and prune schedule: multi\-glyph vocabulary Jaccard of a one\-armed hyperparameter probe against its baseline \(1 = identical piece set\)\. Merge exhaustion: the realized vocabulary whereGpeTrainerterminated naturally below a50,00050\{,\}000cap\.
### F\.2Fertility and compression detail

Table A16:Per\-condition absolute fertility and compression ratio\. Per matched pair\.ffmean held\-out tokens per molecule \(95% molecule\-resampled bootstrap CI in brackets\);gt\\tfrac\{g\}\{t\}glyphs per token \(the compression ratio of §[3\.4](https://arxiv.org/html/2607.05691#S3.SS4), higher==coarser\);\|Δ​f\|\|\\Delta f\|the absolute and rel\|Δ​f\|\|\\Delta f\|the relative cross\-arm fertility gap \(Eq\.[2](https://arxiv.org/html/2607.05691#S3.E2)\)\. Unigram\-LM \(UL\) segments to more tokens at a lower compression ratio than BPE in every condition\.The body \(§[4\.1](https://arxiv.org/html/2607.05691#S4.SS1)\) and the headline table \(Table[2](https://arxiv.org/html/2607.05691#S4.T2)\) report the granularity contrast only as the*relative*gap rel\|Δ​f\|\|\\Delta f\|\. This appendix gives the absolute numbers behind it, the granularity counterpart to the membership inventory of Appendix[B](https://arxiv.org/html/2607.05691#A2)\. Table[A16](https://arxiv.org/html/2607.05691#A6.T16)lists, for every matched condition, each arm’s mean held\-out tokens per molecule \(with its bootstrap CI\), the glyphs\-per\-token compression ratio defined in §[3\.4](https://arxiv.org/html/2607.05691#S3.SS4), and the absolute and relative cross\-arm gaps; the per\-arm fertilities are shown as a dumbbell chart in the body \(Figure[4](https://arxiv.org/html/2607.05691#S4.F4)\)\.

Two things the relative gap alone hides\. First, the absolute gap is large: Unigram\-LM emits∼\\sim1212–1919more tokens per molecule than BPE, on molecules only∼\\sim3030–5555tokens long under BPE\. Second, the per\-arm CIs are tight: the interval spans at most∼\\sim0\.20\.2token on the diverse, drug\-like, and combinatorial corpora and∼\\sim11token even on the smaller natural\-products corpus, so the1212–1919\-token gap clears measurement noise by more than an order of magnitude in every condition\. This held\-out CI is the one the cross\-arm scalar table \(Table[2](https://arxiv.org/html/2607.05691#S4.T2)\) omits\.

### F\.3Boundary nestedness detail

Table A17:Per\-condition cross\-arm boundary nestedness\. Per matched pair \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\)\. Both arms cut the same glyph stream, so their boundaries are subsets of the same inter\-glyph positions\.J∂J\_\{\\partial\}the boundary Jaccard over*cut*positions \(agree\-cut over agree\-cut\+\+nest\+\+conflict\);*nest*the share of positions where Unigram\-LM cuts and BPE merges \(the fertility gap of Table[A16](https://arxiv.org/html/2607.05691#A6.T16)read positionally\) and*conflict*the share where BPE cuts and Unigram\-LM merges \(genuine crossing\), both over all positions;*nested*the fraction of molecules with zero conflict, i\.e\. whose BPE parse is a strict coarsening of Unigram\-LM’s\.cutc\\mathrm\{cut\}^\{c\}localizes conflict: the fraction of multi\-glyph Unigram\-LM pieces of classcc\(heteroatom, unsaturated carbon, saturated carbon\) that BPE cuts through, with no entry for a class the corpus never emits\. Conflict is near zero in every condition\.The granularity contrast has a positional reading \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\): because both arms cut the same glyph stream, their boundaries are comparable position by position\. Table[A17](https://arxiv.org/html/2607.05691#A6.T17)gives, for every matched condition, the boundary JaccardJ∂J\_\{\\partial\}, the nest and conflict rates, the fraction of molecules whose BPE parse strictly coarsens Unigram\-LM’s, and the per\-class conflict localization\. Two readings stand out\. First, the arms agree on most cuts and nest rather than cross: conflict is below0\.7%0\.7\\%of positions in every condition, and the nested\-molecule fraction exceeds0\.950\.95at everyV≥1024V\{\\geq\}1024cell while staying above0\.800\.80even atV=256V\{=\}256, so the fertility gap is depth, not disagreement\. Second, the residual conflict is chemically structured rather than uniform across substructure classes \(heteroatom\- and unsaturation\-led on the diverse and natural\-products corpora, backbone\-led on alkane\-rich ZINC\-22\), which is why we localize it by class rather than report a single conflict rate\.

### F\.4Token\-distribution detail

![Refer to caption](https://arxiv.org/html/2607.05691v1/x16.png)Figure 17:Token\-distribution intrinsics atV=1024V\{=\}1024, one dumbbell per corpus×\\timesboundary joining BPE \(blue\) to Unigram\-LM \(orange\) on each metric’s own axis: imbalanceDD\(divergence from uniform; left\), normalized Shannon entropyη\\eta\(center\), and Rényi efficiency atα=2\.5\\alpha\{=\}2\.5\(right\)\. BPE is the more uniform arm in every condition and on all three metrics \(lowerDD, higherη\\eta, higherRR\), a coherent within\-family signature, while both arms sit far from uniform \(D≳0\.85D\\gtrsim 0\.85\)\. Per\-condition values with CIs in Table[A18](https://arxiv.org/html/2607.05691#A6.T18)\.Table A18:Per\-condition token\-distribution intrinsics \(imbalance, entropy, Rényi efficiency\)\. Within\-family, per matched pair \(§[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\), each a held\-out per\-arm value with a 95% molecule\-resampled bootstrap CI\.DDtoken\-frequency imbalance \(divergence from uniform, Eq\.[3](https://arxiv.org/html/2607.05691#S3.E3);0uniform,11maximally concentrated\),η\\etanormalized Shannon entropy,RRRényi efficiency atα=2\.5\\alpha\{=\}2\.5\. BPE is more uniform than Unigram\-LM \(UL\) in every condition\. The cross\-arm gap\|Δ​D\|\|\\Delta D\|is in Table[2](https://arxiv.org/html/2607.05691#S4.T2)\.The distribution sense is reported in the body only through the cross\-arm gap\|Δ​D\|\|\\Delta D\|\(Table[2](https://arxiv.org/html/2607.05691#S4.T2)\); the three within\-family quantities defined in §[3\.4](https://arxiv.org/html/2607.05691#S3.SS4)\(token\-frequency imbalanceDD, normalized Shannon entropyη\\eta, and Rényi efficiency atα=2\.5\\alpha\{=\}2\.5\) are collected per arm here\. Table[A18](https://arxiv.org/html/2607.05691#A6.T18)gives all three for every matched condition, each with its bootstrap CI; Figure[17](https://arxiv.org/html/2607.05691#A6.F17)plots them atV=1024V\{=\}1024\.

Two readings stand out\. First, the three metrics form a*coherent*signature: in every one of the2222conditions BPE is the more uniform arm \(lowerDD, higherη\\eta, higherRR\), so the distribution difference, though the weakest of the three senses, is consistent in sign across all three measures and not an artifact of the particularDDstatistic we report\. Second, both arms sit*far*from uniform: absoluteDDruns0\.760\.76–0\.970\.97, roughly1\.51\.5–2×2\\timesthe∼\\sim0\.50\.5at whichWadell et al\. \[[51](https://arxiv.org/html/2607.05691#bib.bib51)\]report the chemistry\-tokenizer field clustering\. That offset is partly the small\-vocabulary regime \(a few pieces carry most of the mass\) and partly definitional, and we flag it rather than read across it: what we report is the cross\-arm gap, not the absolute level\. The per\-arm CIs are tight \(≤0\.001\\leq 0\.001on the diverse, drug\-like, and combinatorial corpora\), so the arm separation on each metric far exceeds its CI\.

### F\.5Mechanism\-diagnostic detail

Two of the four mechanism diagnostics are reported in the body only through a cross\-arm gap; their per\-arm detail is collected here: whole\-pretoken absorption below, and the dead\-zone clearance, whose headlinec100c\_\{100\}\(Table[A14](https://arxiv.org/html/2607.05691#A6.T14)\) we extend to the fullnn\-sweep\. The other two, scaffold fraction and segmentation entropy, are one\-sided by construction, so the gap reported in Table[2](https://arxiv.org/html/2607.05691#S4.T2)*is*the absolute\.

Table A19:Per\-condition whole\-pretoken absorption\. Per matched pair: the held\-out fraction of pretokens emitted as a single token\[[36](https://arxiv.org/html/2607.05691#bib.bib36)\], per arm with a 95% molecule\-resampled bootstrap CI, and the cross\-arm gapΔ\\Deltaabs \(BPE−\-UL\)\. BPE absorbs almost every pretoken whole; Unigram\-LM \(UL\) splits far more, the per\-pretoken face of the fertility gap \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\)\.#### Whole\-pretoken absorption\.

Table[A19](https://arxiv.org/html/2607.05691#A6.T19)gives each arm’s held\-out fraction of pretokens emitted as a single token, with CIs\. The contrast is stark and the body’sΔ\\Deltaabs hides it: BPE absorbs almost every pretoken whole \(0\.890\.89–1\.001\.00across conditions\) while Unigram\-LM keeps only∼\\sim0\.720\.72\(0\.710\.71–0\.740\.74\) intact, splitting the rest\. This is the per\-pretoken face of the fertility gap \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\); the permeable \(MB\) boundary lifts BPE’s absorption further, toward1\.01\.0, since merges may then cross the bracket\.

Table A20:Rare\-token clearancecnc\_\{n\}across the firing\-count sweep \(n∈\{50,100,200\}n\\in\\\{50,100,200\\\},p=0\.95p\{=\}0\.95\)\. FollowingGowda and May \[[11](https://arxiv.org/html/2607.05691#bib.bib11)\],cnc\_\{n\}is the fraction of an arm’s vocabulary firing at leastnntimes in the training corpus\. The learnability barFp,nF\_\{p,n\}iscn≥pc\_\{n\}\\geq p, so reading eachcnc\_\{n\}againstp∈\{0\.90,0\.95,0\.99\}p\\in\\\{0\.90,0\.95,0\.99\\\}gives every\(p,n\)\(p,n\)bar outcome;c100c\_\{100\}atp=0\.95p\{=\}0\.95is the headlineF95%,100F\_\{95\\%,100\}\(Table[A14](https://arxiv.org/html/2607.05691#A6.T14)\)\.†\\daggercorpus too small to certify the tail at thisVV\(as in Table[A14](https://arxiv.org/html/2607.05691#A6.T14)\)\.
#### Rare\-token clearance sweep\.

Table[A20](https://arxiv.org/html/2607.05691#A6.T20)gives the per\-arm clearancecnc\_\{n\}\(fraction of vocabulary firing≥n\\geq ntimes\) atn∈\{50,100,200\}n\\in\\\{50,100,200\\\}, the sweep behind the learnability bar of §[3\.5](https://arxiv.org/html/2607.05691#S3.SS5)\. The headlineF95%,100F\_\{95\\%,100\}is thec100≥0\.95c\_\{100\}\\geq 0\.95column; reading anycnc\_\{n\}againstp∈\{0\.90,0\.95,0\.99\}p\\in\\\{0\.90,0\.95,0\.99\\\}recovers the full\(p,n\)\(p,n\)grid, so the bar’s robustness to both thresholds is visible here\. BPE clears the bar atn≤100n\{\\leq\}100in every condition except its smallest corpus \(COCONUT\) atV=1024V\{=\}1024; the strictern=200n\{=\}200pulls a few narrow\-alphabet ZINC\-22 cells just under as well \(0\.9350\.935–0\.9480\.948\)\. Unigram\-LM’s near\-atomic pieces thin it much more on the smaller corpora, the regime where the dead\-zone sign reverses \(§[4\.4](https://arxiv.org/html/2607.05691#S4.SS4)\)\.