sparse-routing

Tag

Cards List
#sparse-routing

CausalWAM: Learning the Causes of Actions in World Action Models

Hugging Face Daily Papers ↗ · 2026-10-30 Cached

CausalWAM proposes adding a sparse routing path to a pretrained World Action Model's action head so that generated actions depend only on action-relevant latent variables, with provable identification of causal structure up to permutation without temporal independence assumptions. Experiments on synthetic data and real robots show improved closed-loop success efficiency.

0 favorites 0 likes
#sparse-routing

@NFT_Chen: Must-read! Qwen lead Liu Dayi Heng's team treats load imbalance in extremely sparse MoE as a closed-loop control proble…

X AI KOLs Timeline ↗ · 2026-10-09 Cached

Qwen team 提出 ID Balancing,将极端稀疏 MoE 的负载不平衡问题建模为 PID 闭环控制问题,在 768 专家 Top-3 路由设置下将最差 MaxVio 降低超过 50%,且无需辅助损失,优势随稀疏度增加而扩大,对 Qwen 4.0 等超大稀疏 MoE 模型的扩展意义重大。

0 favorites 0 likes
#sparse-routing

EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

arXiv cs.AI ↗ · 2026-08-10 Cached

EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free LLMs, using dynamic byte patches as routing units to enable sparse conditional computation. Experiments show it achieves the lowest held-out bits-per-byte among baselines while maintaining downstream accuracy.

0 favorites 0 likes
#sparse-routing

Less Tokens, Better Forecasts: Sparse Residual Routing for Efficient Weather Prediction

arXiv cs.LG ↗ · 2026-07-07 Cached

Introduces Sparse-Reslim, a plug-in routing module that processes only 25% of spatial tokens through expensive transformer blocks for efficient weather prediction, achieving up to 3.18x speedup and improved forecast accuracy.

0 favorites 0 likes
#sparse-routing

@ZhihuFrontier: Half a year ago, a Zhihu contributor predicted that the next Transformer would absorb loops, recurrent state, sparse ro…

X AI KOLs Timeline ↗ · 2026-06-26 Cached

A Zhihu contributor's half-year-old prediction that the next Transformer would absorb loops, recurrent state, sparse routing, and latent reasoning is gaining relevance as Loop Engineering advances. The article explores how future Transformer architectures may evolve into hybrid models blending linear-complexity layers for background context with attention for precise reasoning, plus finer-grained sparsity and native System 2 reasoning.

0 favorites 0 likes
← Back to home

Submit Feedback