mixture-of-experts

A newly released large MoE model, Motif 3 (314B with 13B active parameters, NVFP4 available), seems promising; the author asks for community experiences.

This paper introduces MOSAIC, a framework that jointly optimizes sparse Mixture-of-Experts model architecture and hardware systems for large-scale pretraining, showing that compute-optimal sparsity is not necessarily cluster-optimal when MFU, communication costs, and parallel layouts are considered.

This paper proposes UniF-MoE, a unified framework for token-adaptive Mixture-of-Experts computation that first shares reusable computation across experts and then routes the remaining residual demand, improving performance while reducing activated computation, latency, and memory on DomainBed and GLUE benchmarks.

NVIDIA released Nemotron 3.5 Lightning, a 30B-parameter MoE model with only 3B active parameters, optimized for agent execution tasks. It claims faster, cheaper tool calls and agent execution while staying fully open-source under OpenMDW-1.1.

NVIDIA announced Nemotron 3.5 Lightning, a 30B mixture-of-experts open model optimized for high-volume agentic AI workloads, alongside NeMo Switchyard, an open-source library for intelligent model routing across heterogeneous model ecosystems.

DoGMA is a central-dogma-guided foundation model for pan-cancer multi-omics analysis, using a Transformer-MoE architecture with directed attention to align DNA-RNA-protein flows and pretraining via masked hierarchical omics reconstruction. It shows strong performance across cancer representation learning, survival prediction, and metastasis prediction tasks.

A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.

This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.

This paper investigates how trace-driven evaluation can mislead assessments of MoE expert caching, identifying replay semantics, workload contamination, and operating regimes as confounding axes that can reverse policy rankings. After correcting these issues, it shows that a large offline-optimal gap overstates the gains actually recovered by lightweight causal caching mechanisms.

This paper proposes using router weight sensitivity under lightweight fine-tuning (e.g., LoRA) to identify and prune experts in Mixture-of-Experts models, enabling significant memory and latency reductions with minimal accuracy loss.

This paper introduces LorExperts and BTExperts, router-preserving compression methods for Mixture-of-Experts LLMs that cluster experts and represent non-dominant members as low-rank corrections, improving compression quality over prior methods like D2-MoE.

Presents TEXAS, a method for downstream adaptation of Mixture-of-Experts LLMs that discovers task-relevant experts via correctness-conditioned activations and applies token-level supervision allocation, improving performance across multiple benchmarks.

The paper presents Contribution-Contrast (CoCo), a novel response-level interpretation method for Mixture-of-Experts reward models, which captures routing and preference behavior more faithfully than routing-weight-based approaches.

EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free LLMs, using dynamic byte patches as routing units to enable sparse conditional computation. Experiments show it achieves the lowest held-out bits-per-byte among baselines while maintaining downstream accuracy.

Motif 3 is a 314B-parameter Mixture-of-Experts language model with 13.2B active parameters per token, featuring Grouped Differential Latent Attention and trained on 12.5T tokens, demonstrating competitive performance across reasoning, coding, and long-context tasks.

UniMoMo compresses MoE-based recommendation models by merging experts based on functional behavior and routing traffic, preserving quality while speeding up inference.

Alibaba released Qwen3.8-Max, a 2.4 trillion-parameter sparse MoE model with 95B active parameters per token, 1M token context, and strong agentic and benchmark results, including autonomously coding for days, circuit design, and outperforming rivals on Terminal Bench and PaperBench.

Presents an emerging retail portfolio management application that uses personalized, tax-aware reinforcement learning with natural language goal input, featuring a three-phase pipeline and integration with live brokerage APIs.

A systematic study from Meta FAIR, Reality Labs, and Oxford on multimodal pretraining, revealing asymmetric knowledge flow between modalities, synergy vs. competition dynamics, the benefits of early unification, and efficient training recipes validated with 13.5B MoE models.

This paper introduces MESH, a memory-efficient Sinkhorn-based optimizer for Mixture-of-Experts (MoE) training that restores temporal momentum without storing full optimizer state, reducing memory by 62.5% while maintaining competitive evaluation loss compared to AdamW.