Tag
CausalWAM proposes adding a sparse routing path to a pretrained World Action Model's action head so that generated actions depend only on action-relevant latent variables, with provable identification of causal structure up to permutation without temporal independence assumptions. Experiments on synthetic data and real robots show improved closed-loop success efficiency.
Qwen team 提出 ID Balancing,将极端稀疏 MoE 的负载不平衡问题建模为 PID 闭环控制问题,在 768 专家 Top-3 路由设置下将最差 MaxVio 降低超过 50%,且无需辅助损失,优势随稀疏度增加而扩大,对 Qwen 4.0 等超大稀疏 MoE 模型的扩展意义重大。
EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free LLMs, using dynamic byte patches as routing units to enable sparse conditional computation. Experiments show it achieves the lowest held-out bits-per-byte among baselines while maintaining downstream accuracy.
Introduces Sparse-Reslim, a plug-in routing module that processes only 25% of spatial tokens through expensive transformer blocks for efficient weather prediction, achieving up to 3.18x speedup and improved forecast accuracy.
A Zhihu contributor's half-year-old prediction that the next Transformer would absorb loops, recurrent state, sparse routing, and latent reasoning is gaining relevance as Loop Engineering advances. The article explores how future Transformer architectures may evolve into hybrid models blending linear-complexity layers for background context with attention for precise reasoning, plus finer-grained sparsity and native System 2 reasoning.