tokenizer-evaluation

This paper introduces TokEval, a framework for evaluating language model tokenizers using intrinsic metrics that correlate with downstream task performance.