The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Models
Summary
This article analyzes BPE tokenization in Polish, highlighting its limits in inflectional languages and proposing grammatical form anchoring to improve language modeling.
View Cached Full Text
Cached at: 09/17/26, 08:52 AM
# The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Models Source: [https://arxiv.org/abs/2609.17553](https://arxiv.org/abs/2609.17553) [View PDF](https://arxiv.org/pdf/2609.17553) > Abstract:This article analyzes BPE tokenization in Polish as a test case for the limits of statistical segmentation in an inflectional language\. It asks whether frequency\-based tokenization preserves linguistically relevant units, including orthographic form, phonemic and syllabic segmentation, derivational structure, inflectional endings, grammatical form, and the speaking subject\. The material includes diagnostic words, a children's text, selected forms from the Preamble to the Constitution of the Republic of Poland, word\-family tests, and examples with Polish diacritics and nasal vowels\. BPE tokenizers may produce segments that coincide with syllabic or morphologically interpretable divisions, but remain dependent on the frequency of written forms\. They do not systematically map orthographic representation onto phonemic structure or context\-dependent phonetic realization\. The results show that BPE stabilizes frequent surface fragments of grammatical exponents rather than grammatical categories themselves\. A form such as ustanawiamy is not merely a sequence ending in \-y, but a verbal form anchored in conjugation, person, number, tense, mood, and aspect\. The article develops the concept of grammatical form anchoring\. In Polish, forms such as poszlam, zrobilam, or bylam can establish the position of the speaking subject without an explicit pronoun\. In interaction with AI, this exposes a further problem: a language model does not possess a stable grammatical "I", but reconstructs it contextually and may mirror the user's forms or shift grammatical gender\. Roclawski's segmentation\-flexional forms are proposed as a diagnostic framework for evaluating tokenization boundaries\. More stable modeling of Polish may require sublexical stabilization, anchoring grammatical form in the inflectional system, representing sentence patterns and verbal valency, and maintaining the grammatical "I" in dialogue\. ## Submission history From: Elżbieta Dawidek \[[view email](https://arxiv.org/show-email/22f5e60c/2609.17553)\] **\[v1\]**Fri, 17 Jul 2026 17:51:52 UTC \(691 KB\)
Similar Articles
Writing-System-Level Tokenizer Adaptation for Byte-Level BPE
This paper introduces BPE-guided insertion for post-hoc tokenizer adaptation on byte-level BPE models, keeping vocabulary size fixed and preserving most token-ID assignments. The method reduces Ukrainian token counts by ~33-36% while minimizing impact on English and other European languages.
Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment
This paper identifies that BPE tokenization fragments safety-critical words into subword pieces, creating exploitable gaps in LLM alignment. Character-level perturbations can bypass safety by disrupting token boundaries, achieving 80-100% refusal flip on HarmBench prompts with 48% harmful outputs across five model families.
Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding
This paper proposes Pruned BPE, a post-training method that prunes low-exposure tokens from a BPE vocabulary and reallocates slots to better-exposed candidates, reducing encoded length without increasing model-visible vocabulary size. Experiments on English and Chinese corpora show approximately 0.27–0.36% encoded length reduction over standard BPE.
Incremental BPE Tokenization
This paper introduces an incremental algorithm for Byte Pair Encoding (BPE) tokenization that processes each byte in O(log^2 t) time, enabling efficient partial tokenization in streaming settings and achieving speedups over existing implementations.
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
This paper introduces BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced multilingual corpus for classical Indian languages. Subword fertility analysis shows significant improvements in tokenization efficiency, reducing sequence length by up to 90% compared to baseline tokenizers, thereby enhancing effective context length for downstream language models.