Tag
LingBot-Video is a 13B sparse-MoE video diffusion transformer (1.4B active) post-trained with RL as an action-conditioned world model, open-sourced with weights and code. It includes a physical-plausibility reward graded by a VLM and frames itself as a policy evaluator and action planner, though closed-loop robot results are absent.
LingBot-Video, a 30B-parameter video model with sparse MoE, designed for embodied intelligence, is open-sourced. It outperforms existing models on RBench, trained on 70K+ hours of embodied data.
OpenMythos is an open-source project that theoretically reproduces the Claude Mythos architecture, fully implementing the Recurrent-Depth Transformer (RDT), supporting MLA/GQA attention mechanisms and sparse MoE, providing preset configurations from 1B to 1T parameters, and is installable via pip.
Mistral AI releases Leanstral 1.5, an updated Lean 4 formal proof engineering model optimized for automated theorem proving and autoformalization, with 119B total parameters and 6.5B active parameters.
This paper proposes SARA, a framework that aligns routing distributions of multilingual inputs using Jensen-Shannon divergence to improve expert sharing for low-resource languages in sparse Mixture-of-Experts models. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct show improvements on multilingual benchmarks.
The article provides a detailed explanation of Mixture of Experts (MoE) in transformers, covering routing, load balancing, and recent innovations like fine-grained experts. It also highlights the significance of Noam Shazeer's research contributions and his move from Google to OpenAI.
StepFun's Step 3.7 Flash, a 198B sparse MoE model with 11B active parameters, matches 97% of Claude Opus 4.6's coding performance on SWE-Bench Verified at roughly one-ninth the cost, using an Advisor Mode strategy that reserves expensive frontier model calls for critical decision points.
Step 3.7 Flash, an open-weight 198B sparse MoE model, claims 98% agent reliability on tau2-bench across all difficulty levels, with mid raw capability but strong multi-step consistency.
Step 3.7 Flash is a 198B-parameter sparse MoE vision-language model with 11B active parameters per token, supporting 256k context and three reasoning levels, designed for high-throughput agentic workflows.
DECO is a sparse MoE architecture that matches dense Transformer performance with only 20% activated experts and a 3x acceleration kernel, utilizing ReLU-based routing, learnable scaling, and the NormSiLU activation function.
Nucleus-Image is an open-source text-to-image diffusion transformer with 17B parameters across 64 routed experts, activating only ~2B parameters per forward pass. It matches or exceeds leading models like Qwen-Image and Imagen4 while maintaining high efficiency, released with full model weights, training code, and dataset.