Tag
Alibaba released Qwen3.8-Max, a 2.4 trillion-parameter sparse MoE model with 95B active parameters per token, 1M token context, and strong agentic and benchmark results, including autonomously coding for days, circuit design, and outperforming rivals on Terminal Bench and PaperBench.
Presents an emerging retail portfolio management application that uses personalized, tax-aware reinforcement learning with natural language goal input, featuring a three-phase pipeline and integration with live brokerage APIs.
A systematic study from Meta FAIR, Reality Labs, and Oxford on multimodal pretraining, revealing asymmetric knowledge flow between modalities, synergy vs. competition dynamics, the benefits of early unification, and efficient training recipes validated with 13.5B MoE models.
This paper introduces MESH, a memory-efficient Sinkhorn-based optimizer for Mixture-of-Experts (MoE) training that restores temporal momentum without storing full optimizer state, reducing memory by 62.5% while maintaining competitive evaluation loss compared to AdamW.
This paper introduces elbow-based routing, a training-free inference-time method for MoE models that dynamically adjusts the number of active experts per token by detecting the elbow point in router probability distributions, achieving a 5.3% average latency reduction while maintaining accuracy.
This paper introduces MSRT, a framework with a resource-aware Mixture of Speech Encoders (MoSE) to overcome the curse of multilinguality in many-to-many speech-to-text translation. The 4B-parameter model achieves state-of-the-art results across 45 languages, particularly improving low-resource speech translation with only 10 hours of paired data per language.
LG AI Research presents K-EXAONE 2.0, a 750B-parameter MoE foundation model upcycled from K-EXAONE, supporting 256K context and six languages, with self-speculative decoding for efficient inference.
This paper introduces SpecDrop, a parameter-free category-conditioned routing scheme for modular networks, showing that on vision tasks it achieves competitive accuracy while on fuzzy language partitions it reduces to no-routing baselines, suggesting granularity alignment matters more than router design.
The paper proposes C²MOE, a Consistency and Complementarity-guided Mixture of Experts framework for incomplete multimodal emotion recognition in conversations, using information-theoretic decomposition to improve robustness when modalities are missing.
DyPES-VLA is a cross-embodiment VLA model that learns shared dynamics priors via future prediction and uses an embodiment-specific Mixture-of-Experts action head to control robots in their native action spaces, achieving state-of-the-art results on LIBERO, RoboCasa, and RoboTwin benchmarks.
Cursor open-sources Mixture-of-Kittens (MoK), a deterministic MoE training megakernel for NVIDIA Blackwell GPUs that fuses computation and communication, delivering up to 2.37x speedup over baseline implementations.
This paper proposes MoEGen, a parameter-efficient fine-tuning framework that uses mixture-of-experts to generate instance-adaptive LoRA updates via expert codes and a lightweight hypernetwork, improving performance on commonsense reasoning benchmarks without storing separate adapters per expert.
AcceptMoE is a verifier-side expert selection method for speculative decoding of MoE LLMs, which reduces expert-weight traffic and improves throughput by up to 2.06x under expert offloading with minimal accuracy loss.
This paper systematically explores the physics of multimodal pretraining, uncovering how knowledge flows between modalities, conditions for synergy vs. competition, the benefits of early unification, and efficient pretraining recipes validated on 13.5B MoE models.
K-EXAONE 2.0 is an open-weight multilingual MoE foundation model from LG AI Research with 750B total parameters and 37B active, supporting 10 languages and 256K context, with notable gains in agentic coding, long-context understanding, and safety.
Cursor is open-sourcing Mixture-of-Kittens (MoK), a fused MoE training megakernel for NVIDIA NVL72 that runs up to 2.37x faster than the strongest public baselines.
InclusionAI released the Ling-3.0-flash model weights on Hugging Face with MIT license, including BF16 and official FP8 versions. The model uses a fine-grained MoE architecture with 512 experts and 8 active per token, totaling 127.5B params with 5.1B active.
A research paper presenting scaling laws and design principles for mixture-of-experts diffusion language models, trained as LLaDA MoE v2 (30B-A3B) on 23.5T tokens, approaching Qwen3 on several benchmarks with fewer pretraining tokens.
Mind Lab claims its Macaron-V1 model surpasses GLM-5.2 in benchmarks, using five LoRA expert modules attached to GLM-5.1 with dynamic expert switching and continual learning via distilled LoRA adapters.
DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.