Tag
This preprint argues that learned embedding tables dominate small language models' size and shows a frozen 16 MB vocabulary of 512-bit codes can match or beat trained tables on benchmarks, enabling a 46 MB BitNet ternary model with 5-6x faster CPU inference at a seventh of the memory.
An interactive map of GPT-2's token embedding space that lets users tap any token to explore its embeddings.