mixture-of-experts

InclusionAI released the Ling-3.0-flash model weights on Hugging Face with MIT license, including BF16 and official FP8 versions. The model uses a fine-grained MoE architecture with 512 experts and 8 active per token, totaling 127.5B params with 5.1B active.

A research paper presenting scaling laws and design principles for mixture-of-experts diffusion language models, trained as LLaDA MoE v2 (30B-A3B) on 23.5T tokens, approaching Qwen3 on several benchmarks with fewer pretraining tokens.

Mind Lab claims its Macaron-V1 model surpasses GLM-5.2 in benchmarks, using five LoRA expert modules attached to GLM-5.1 with dynamic expert switching and continual learning via distilled LoRA adapters.

DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.

Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.

A 13-step visual walkthrough explaining how the Switch Transformer works, covering sparse mixture-of-experts routing and why models like GPT-4, Claude, DeepSeek-V3, and Kimi use this architecture to stay efficient.

Qwen released Qwen-CUA, a native computer-use agent with a 397B-A17B mixture-of-experts backbone, achieving state-of-the-art results on OSWorld-Verified and ranking #2 on WebArena. A technical report is available on Papers with Code.

AI9Stars released G9v3-39A5B, an open-weights 39B MoE language model with 5 active experts, targeting reasoning, coding, and assistant tasks under Apache 2.0.

This paper introduces Mixture-of-Translators (MoT), a framework for translating KV caches across heterogeneous LLMs, enabling cache reuse between different architectures. Experiments show preserved QA performance and long-context quality across Qwen2.5, GPT-2, and OPT models.

The paper introduces Hierarchical Copula-Gumbel-Top-K (H-CGA) routing, a method to control joint dependence among token routing choices in frozen Mixture-of-Experts models while keeping each token's routing law exactly fixed. It provides theoretical trade-offs between coherence and load dispersion and validates the mechanism with a small-scale pilot.

TheTom releases an MLX quantized version of DeepSeek V4 Flash (284B MoE, 21B active) at 3.05 bpw, fitting in 101 GiB to run on 128GB Apple Silicon, with GGUF sibling also available.

This paper presents ReBA (Relax Within, Balance Across), a geometry-guided load-balancing method for vision-language mixture-of-experts models. It addresses modality-specific routing imbalance by balancing image and text separately and grouping correlated visual tokens, improving robustness across resolution and composition shifts.

WASTE is a new open-source C inference engine that streams expert weights from disk to run the 2.78-trillion-parameter Kimi K3 model on a consumer laptop with just 29 GB of RAM, achieving 0.49–0.54 tokens/s.

This paper introduces TIER-MoE, a risk-guided subspace mixture-of-experts model for multimodal biomedical classification that estimates sample-specific modality reliability from out-of-fold predictions and routes modalities to experts, improving performance and calibration on four public datasets.

WASTE is an open-source inference engine that runs models larger than available RAM by storing expert weights on NVMe. It demonstrates running Kimi K3, a 2.78T parameter MoE model, on a MacBook Pro with 64GB unified memory.

Thinking Machines released Inkling-Small, an efficient open-weights Mixture-of-Experts model with 276B total and 12B active parameters, achieving comparable performance to its larger sibling Inkling at a quarter of the size. It features native reasoning over audio and images, variable thinking effort, and a 1M-token context window.

Thinking Machines released Inkling-Small, a 276B-parameter mixture-of-experts model with 12B active parameters, 1M context, and native image/audio understanding, now available on Modal with NVIDIA B300 support.

Kimi K3 by Moonshot is an open-weight model ranking fourth among 580 models, featuring innovations like Kimi Delta Attention to reduce KV cache memory, Quantile Balancing for expert load balancing, and AgentENV for efficient RL training sandboxing.

A new mixture-of-experts approach for reconstructing handwriting trajectories from IMU sensor data, with separate experts for touching and hovering phases, and a new public benchmark dataset.

This paper proposes modifications to encoder-decoder transformers for unconstrained molecular structure elucidation from IR spectra, using a Mixture-of-Experts decoder and contrastive alignment loss, achieving over 10 percentage points improvement in Top-K accuracy.