sparse-transformers

提出了名为 Nestor 的嵌套稀疏机制，将条件计算从 MoE 扩展到注意力算子，使稀疏性层级组合，在同等计算与稀疏度下优于传统 MoE，并在长上下文与领域持续预训练中表现更佳。

Extropic introduces Z1T, a sparse transformer-like model optimized for their probabilistic Z1 hardware to enhance energy efficiency in transformer inference through algorithmic-hardware co-design.

This paper presents Token-Selective Attention (TSA), a differentiable token routing mechanism that learns to skip unnecessary computations per token in transformer layers, reducing token-layer operations by 14–23% with minimal quality loss on language modeling tasks.