Tag
The paper addresses tokenization errors in large language models when applied to Brahmic scripts by formalizing orthographic constraints in Agda and developing a provably correct fix for tokenization, with practical implementations in SentencePiece and a Rust library.