Tag
Hugging Face has released version 1 of their tokenizers library, featuring multiple language support, multi-thread scaling, and minimal package size.