sparse-moe

该论文提出 Elastic Expert Routing，通过从以 k 为中心的局部离散分布中随机采样激活专家数量，平滑传统 top-k 路由的硬边界，在保持计算成本不变的前提下提升稀疏 MoE 模型在微调与从头预训练场景下的下游表现。

MiMo-V2.6-Flash-RL is a multimodal AI model that scales reinforcement learning for self-improvement, featuring a sparse mixture-of-experts architecture with 309B total parameters and 1M token context length.

This paper investigates the predictive value of earlier expert selections in sparse mixture-of-experts models beyond the most recent layer, finding significant gains in routing prediction across layers.

Edge0-35B-A3B-preview is a sparse MoE model that enables efficient AI inference on mobile devices by using streaming expert offloading and quantization, achieving 15 tok/s with under 3 GiB of memory.

Apple's Lily engine optimizes on-device LLM inference for Apple silicon by leveraging unified memory and hardware, outperforming MLX-LM, and is tuned for the Qwen3.6-35B-A3B model's architecture.

Alibaba released Qwen3.8-Max, a 2.4 trillion-parameter sparse MoE model with 95B active parameters per token, 1M token context, and strong agentic and benchmark results, including autonomously coding for days, circuit design, and outperforming rivals on Terminal Bench and PaperBench.

This paper analyzes why compression statistics for LLM pruning can be reproducible yet select suboptimal endpoints, introducing information boundaries and observation fibers to model the gap. It proposes group-resolved and model-specific mask selection methods that improve worst-group perplexity across dense LLMs and OLMoE.

The author reflects on how AI model pricing per token has dropped dramatically, but real-world costs remain flat because cheaper models get re-run more often. They argue that cost per completed step is the metric that matters, not cost per million tokens.

This paper introduces the Expert Subspace Separation Index (ESSI) to disentangle route coherence, candidate quality, and candidate-by-context interaction in sparse Mixture-of-Experts language models, revealing a pattern of coherent overlap where routing selects token-relevant experts from a shared geometric neighborhood yet multi-expert computation remains useful.

SpecPrefetch proposes a parameter-efficient expert prefetching framework for sparse MoE models, using a lightweight adapter to predict next-layer experts for asynchronous transfer while preserving native routing semantics. It achieves up to 20% decoding throughput improvement on a Snapdragon 8 Elite device, demonstrating practical benefits for memory-constrained deployment.

SKT released A.X K2, a 688B-parameter sparse MoE language model with 33B active parameters, natively trained in FP8 and featuring Think/Non-Think reasoning modes, on Hugging Face.

The author shares a strategy for reducing costs in multi-model agent loops by using a cheap fast executor for repetitive nodes and a strong planner for high-level reasoning, with experience using Ling-3.0-flash on OpenRouter.

Motif Technologies releases an intermediate beta checkpoint of Motif-3, a large-scale Mixture-of-Experts language model with ~314B total parameters (~13B active), 256K context length, and custom architectures like Grouped Differential Latent Attention (GDLA), openly available for non-commercial research.

LingBot-Video is a 13B sparse-MoE video diffusion transformer (1.4B active) post-trained with RL as an action-conditioned world model, open-sourced with weights and code. It includes a physical-plausibility reward graded by a VLM and frames itself as a policy evaluator and action planner, though closed-loop robot results are absent.

LingBot-Video, a 30B-parameter video model with sparse MoE, designed for embodied intelligence, is open-sourced. It outperforms existing models on RBench, trained on 70K+ hours of embodied data.

OpenMythos is an open-source project that theoretically reproduces the Claude Mythos architecture, fully implementing the Recurrent-Depth Transformer (RDT), supporting MLA/GQA attention mechanisms and sparse MoE, providing preset configurations from 1B to 1T parameters, and is installable via pip.

Mistral AI releases Leanstral 1.5, an updated Lean 4 formal proof engineering model optimized for automated theorem proving and autoformalization, with 119B total parameters and 6.5B active parameters.

This paper proposes SARA, a framework that aligns routing distributions of multilingual inputs using Jensen-Shannon divergence to improve expert sharing for low-resource languages in sparse Mixture-of-Experts models. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct show improvements on multilingual benchmarks.

The article provides a detailed explanation of Mixture of Experts (MoE) in transformers, covering routing, load balancing, and recent innovations like fine-grained experts. It also highlights the significance of Noam Shazeer's research contributions and his move from Google to OpenAI.

StepFun's Step 3.7 Flash, a 198B sparse MoE model with 11B active parameters, matches 97% of Claude Opus 4.6's coding performance on SWE-Bench Verified at roughly one-ninth the cost, using an Advisor Mode strategy that reserves expensive frontier model calls for critical decision points.