byte-level-models

Cartesia 分享关于分层网络（H-Nets）的研究博客，展示无需分词器的字节级语言模型在受控实验中实现 3-5 倍计算效率提升，2 阶段 H-Net 以 10 倍更少计算与数据超越匹配的开源模型，并在代码等领域表现出色。

该论文研究无分词器的字节级语言模型，发现配合 token-superposition 训练与哈希嵌入后，扁平 Transformer 随规模扩大能持续超越子词模型，并能自发学习出类似分词器的局部文本抽象，据此可将投机解码的接受率提升 3.4 倍。

A research paper from Meta demonstrates that byte-level language models, initially inferior to token-based models, can outperform them as computational resources increase, shown through distilled 1B models trained on up to 1 trillion bytes.

A paper from Meta shows that byte-level models start behind token models but surpass them with increasing compute, demonstrated with distilled 1B models trained on up to 1 trillion bytes.