homographs

Tag

Cards List
#homographs

Tokenizing Crosslingual Homographs

arXiv cs.CL · 2026-07-21 Cached

This paper investigates how multilingual tokenizers handle cross-lingual homographs (identical surface forms with different meanings across languages) and proposes a lightweight language-cue intervention that introduces language-specific characters to reduce token sharing. Experiments show modest improvements in machine translation, particularly with BPE tokenization.

0 favorites 0 likes
← Back to home

Submit Feedback