grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP
Summary
Introduces grapheme-kit, an open-source Python library that extends lexical distance and evaluation metrics to operate on grapheme clusters instead of Unicode code points, with improved processing for Tamil and Sinhala.
View Cached Full Text
Cached at: 07/27/26, 07:41 AM
# grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP Source: [https://arxiv.org/abs/2607.22456](https://arxiv.org/abs/2607.22456) [View PDF](https://arxiv.org/pdf/2607.22456) > Abstract:Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points\. We introduce grapheme\-kit, an open\-source Python library that extends these metrics to operate on grapheme clusters instead\. The library also provides improved grapheme processing for Tamil and Sinhala, including accurate grapheme cluster identification and grapheme composition/decomposition utilities\. Through an OCR case study, we demonstrate that grapheme\-level metrics provide a more faithful evaluation of complex scripts\. ## Submission history From: Menan Velayuthan \[[view email](https://arxiv.org/show-email/9e6ff006/2607.22456)\] **\[v1\]**Fri, 24 Jul 2026 16:14:37 UTC \(1,293 KB\)
Similar Articles
Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages
This paper trains five GPT-2-style models from scratch to compare dedicated monolingual models for Tamil, Telugu, Kannada, and Malayalam against a joint multilingual model, finding monolingual models outperform mGPT on sentiment classification and NER with more efficient tokenizers.
KG2Cypher: Data-Centric Pipeline for Building Enterprise Text-to-Cypher Systems
KG2Cypher presents a data-centric pipeline for building enterprise text-to-Cypher systems from existing knowledge graphs. It uses LLMs to generate natural language question-Cypher pairs, validated by an LLM judge and human review, and achieves significant performance improvements on Korean enterprise datasets with LoRA-based fine-tuning.
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
This paper introduces BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced multilingual corpus for classical Indian languages. Subword fertility analysis shows significant improvements in tokenization efficiency, reducing sequence length by up to 90% compared to baseline tokenizers, thereby enhancing effective context length for downstream language models.
Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamil
This paper evaluates the mathematical reasoning capabilities of large language models in Sinhala and Tamil, two low-resource South Asian languages, using a parallel dataset of independently authored problems. The study demonstrates that while basic arithmetic transfers well across languages, complex reasoning tasks show significant performance degradation in non-English languages, with implications for deploying AI tutoring tools in multilingual educational contexts.
Grammar Engineering Meets LLMs: Development of Cantonese and Irish ParGram Treebanks
This paper presents the development of Cantonese and Irish treebanks within the ParGram Project and investigates the potential and limitations of using multilingual LLMs (OpenAI's gpt-oss-120b) for grammar engineering tasks such as translation and syntactic structure generation.