tokenization

CZ commented that more countries will realize issuing local stablecoins boosts demand for their currency and government bonds, citing HSBC's Hong Kong dollar stablecoin RedCoin reaching 50,000 users and HK$87.75M in circulation within its first week.

Matt Dancho explains why understanding tokenization is a must-have skill for Data Scientists moving into AI Engineering, highlighting how tokens affect context windows, cost, embeddings, and multilingual performance, and links a tokenization handbook plus a free workshop.

美国证券交易委员会（SEC）发布了2027财年检查优先事项，将先进AI（包括“超级智能”）以及代币化证券列为重点监管对象，涵盖市场基础设施、网络安全和投资者保护等风险领域。

SpecSlot 是一种像素空间 rectified-flow Transformer，通过将空间锚点扩展为分配不相交 DCT 系数带的多个槽位，并仅让最低频槽参与全局注意力，在不增加计算成本的情况下提升图像细节生成质量。

Cartesia 分享关于分层网络（H-Nets）的研究博客，展示无需分词器的字节级语言模型在受控实验中实现 3-5 倍计算效率提升，2 阶段 H-Net 以 10 倍更少计算与数据超越匹配的开源模型，并在代码等领域表现出色。

Cartesia 的 From Tokens to Bytes 系列第一篇探讨 Hierarchical Networks (H-Nets) 的扩展规律，展示学习式 tokenization 带来 3-5 倍计算效率提升，2 阶段 H-Net 以 10 倍更少算力和数据超越匹配的开源模型，并呈现 in-context tokenization 等新行为以及在生物与视觉等跨模态任务上的潜力。

ARMOR-DP is a modular framework for multilingual BabyLM that dynamically adjusts language sampling based on per-language validation loss, showing that the weakest language shifts during training and that dynamic protection outperforms fixed Chinese-protection baselines.

该论文提出了一个统一的六维分析框架，系统综述了2017–2026年大模型训练全流程（数据与分词、预训练、SFT与偏好对齐/强化学习）的395篇核心文献，并揭示了跨阶段效应与多个开放问题。

AdaptTok-Indic 提出一种面向印地语系语言的脚本感知自适应分词方法，结合 Unicode 归一化预处理、显式印度语连字处理和基于脚本路由的字节级边界预测，在孟加拉语和印地语上取得比 SentencePiece BPE 更低的 token 生长率和更好的下游表现。

The paper shows that letters-only pre-tokenizer regexes in Llama-3 and Qwen3 impose an irreducible ~2.2x token premium on Nepali, and proposes an English-preserving regex repair that, with 32k new merges, reduces the premium to 1.01x, testing it via continued pretraining experiments.

该论文挑战了离散视觉令牌必然存在语义损失的假设，提出因子化向量量化（Factorized Vector Quantization），在多模态理解上达到与连续视觉表示相当的水平，并在离散表示模型中取得最先进性能。

The paper introduces Common-Support Distillation (CSD), a unified logits-based framework for cross-tokenizer knowledge transfer that constructs a shared support between teacher and student and matches induced distributions with a single objective, outperforming existing cross-tokenizer distillation methods across multiple teacher-student settings.

The paper identifies that existing time series tokenization strategies (patch and phase tokenization) introduce spectral distortions such as spectral leakage and aliasing, and proposes TimeFT, a parameter-free, nearly distortion-free frequency-based tokenizer that serves as a drop-in replacement and improves performance across forecasting, classification, and anomaly detection tasks.

A Nature paper introduces 'byteification', a method converting subword language models into byte-level ones with under 1% of a typical pretraining budget, producing models like Bolmo 7B that outperform prior byte-level baselines while retaining subword-level performance.

Someone open-sourced the KRONOS model, which treats each candlestick as a token to predict the next candle (pre-trained on 12 billion candlesticks). It can be integrated 100% free into Claude Code and TradingView to build a full AI crypto trading and prediction system.

论文提出 Mediator，一种离散、本体无关的 EHR 分词器，将任意临床概念映射到共享向量量化码本，使封闭词汇模型可迁移至 eICU，零样本任务 AUPRC 相对提升 29%。

Thunder-DeID+ 提出一种边界感知的韩语去标识化方法，通过改进的 BPE 分词器与三层 PII 分类体系，将法院判决书上的 Strict Typed F1 从 0.76-0.77 提升至 0.87-0.88，并在微调后达到 0.93，实现向更广泛敏感文档的迁移。

这篇论文通过在Tagalog上的受控实验（MorphAustro-Tok与BPE及打乱形态素的对照组比较），发现形态感知分词确实能提升机器翻译效果，但形态正确性本身并不贡献收益——增益仅来自分词粒度与一致性，而非语言学上正确的切分。

该论文指出，跨语料的层级消融实验中出现的“深度专门化”效应可能只是目标token分布差异（如空白符）造成的假象，并提出在报告专门化结论前应按token类型分解并进行混合匹配验证。

The paper introduces Tokka-Bench, an open-source framework for evaluating tokenizers across 100 natural languages and 20 programming languages using five metrics, finding that vocabulary allocation strategy matters more than raw vocabulary size among seven BPE tokenizers including GPT-4, Llama 3.1, Gemma 3, Qwen3, and Kimi K2.