Tag
This paper systematically compares BPE and Unigram-LM tokenization methods for SMILES strings in chemistry, finding they produce near-disjoint vocabularies and differ significantly in segmentation granularity. The results show that the choice of subword algorithm is a critical modeling decision, not a free default.