Examining the Limits of Word2Vec with Toki Pona
Summary
This paper investigates whether Word2Vec can generate meaningful semantic embeddings for Toki Pona, a constructed language with only ~130 words, using a corpus of 1.4 million sentences, and examines the effect of non-Toki Pona tokens on embedding quality.
View Cached Full Text
Cached at: 06/17/26, 05:39 AM
# Examining the Limits of Word2Vec with Toki Pona Source: [https://arxiv.org/abs/2606.17299](https://arxiv.org/abs/2606.17299) [View PDF](https://arxiv.org/pdf/2606.17299) > Abstract:Word2Vec's effectiveness at generating semantic embeddings has been widely validated, yet it has been tested almost exclusively on languages with large vocabulary inventories\. This study examines whether Word2Vec can successfully capture semantic relationships within an extremely reduced vocabulary using data from Toki Pona, a constructed language with approximately 130 words\. We sourced 1\.4 million sentences \(7\.95 million tokens\) from the Toki Pona community for training\. Approximately 23% of sentences in the corpus contain non\-Toki Pona tokens such as named entities, loanwords, and neologisms\. To investigate whether this linguistic noise enhances or hinders performance \-\- a topic rarely addressed in word embedding literature \-\- we trained two distinct models: one retaining these incidental tokens and another filtering them out completely\. Evaluation was conducted using quantitative methods measuring word proximity to semantic category centroids, automated silhouette scores via agglomerative clustering, and qualitative analysis utilizing representational similarity matrices compared against English\. The results indicate that while sparse, non\-core tokens do not affect the relative structure of the learned embeddings, they actually draw similar words closer together in the vector space\. Importantly, Word2Vec's effectiveness depends more on distributional patterns than lexicon size even at this extreme lower bound\. ## Submission history From: Daniel Huang \[[view email](https://arxiv.org/show-email/e39c3326/2606.17299)\] **\[v1\]**Mon, 15 Jun 2026 21:07:31 UTC \(985 KB\)
Similar Articles
Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India
Researchers develop KokborokMT, a neural MT system for the low-resource Kokborok language, achieving BLEU scores of 17.30 en→trp and 38.56 trp→en by fine-tuning NLLB-200 on a 36k-sentence parallel corpus.
QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages
This paper presents QuechuaTok, a benchmark for evaluating tokenization strategies for Southern Quechua, and introduces Morphological Boundary Accuracy (MorphAcc) as a necessary metric. It shows that BPE achieves low fertility but poor morphological accuracy, while a morphology-aware PRPE tokenizer achieves 83% MorphAcc, demonstrating that fertility rate alone is insufficient for agglutinative languages.
InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation
InsightTok introduces content-aware perceptual losses to improve discrete visual tokenization for better text and face reconstruction, enhancing autoregressive image generation quality.
The Rise of Verbal Tics in Large Language Models: A Systematic Analysis Across Frontier Models
A systematic study of repetitive, formulaic verbal tics in eight frontier LLMs, introducing the Verbal Tic Index (VTI) and revealing significant inter-model variation and negative impact on perceived naturalness.
Continuity and Ordinality Matter: Constraining Time Series Tokens for Effective Time Series Analysis with Large Language Models
This paper proposes COM, a method that enforces continuity and ordinality constraints on time series token embeddings to improve the performance of token-based time series large language models.