mixture-of-experts

This paper introduces KITE, a KV-invariant transformer expansion method that efficiently scales LLMs by reducing inference costs while maintaining performance. It presents the SST model that achieves lower training loss and reduced inference cost compared to baselines.

Qwen3.8-Omni-Flash is a natively multimodal agentic model designed for real-world productivity, improving multimodal understanding and reasoning with a MoE architecture and extended context window, along with releasing open-source frameworks for multimodal applications.

Hunyuan-A13B is an open-source large language model with a Mixture-of-Experts architecture, activating 13 billion parameters during inference, and featuring a dual-mode chain-of-thought framework for adaptive reasoning across complex tasks.

This paper introduces methods to bound four key memory peaks in large Mixture-of-Experts training, enabling training at 1M context length with fixed GPU memory and up to 10.4× throughput improvement over baselines.

dots3-note Preview is a multimodal Mixture-of-Experts AI model with 280B total parameters and 16B activated, supporting up to 512K token context, released as an open-weight model on Hugging Face.

The paper proposes CS-MoE, a novel Transformer architecture that shares neural experts across layers to improve parameter utilization, achieving lower perplexity with only 55% of parameters activated.

This paper formulates speculative decoding in Mixture-of-Experts models as a Stochastic Shortest Path problem and uses a diagnostic Oracle to demonstrate that optimal decisions follow a necessary condition balancing cost and progress.

Xiaomi has released two state-of-the-art AI models, MiMo-V2.6 Pro RL for maximum capability and MiMo-V2.6 Flash RL for maximum efficiency, both featuring sparse mixture-of-experts architecture, 1M context length, MIT licensing, and native omni-modal support for text, image, video, and audio.

OneBid is a unified auto-bidding foundation model that addresses challenges in diverse oCPX advertising scenarios by using Mixture-of-Experts architecture and CROP optimization, with deployment at Kuaishou showing significant improvements like +13.1% in ROAS.

IntBMoE is a novel Mixture-of-Experts method that decouples participation, execution, and materialization costs through block-level conditioning and sparse execution, demonstrating improvements in image classification, language modeling, and recommendation systems, with real-world deployment in AMap's generative recommendation system.

This paper proposes L0-MoE, a lightweight Mixture-of-Experts approach using L0-regularization to accelerate dense Large Language Models with up to 2.5x speedup while maintaining competitive performance.

This paper introduces Attention-Aware Routing (AAR), a method that enhances Mixture-of-Experts language models by incorporating attention weights into the router, improving mathematical reasoning performance and revealing coupled dynamics between routing and attention.

The paper introduces ScriptMoE, a script-aware mixture-of-experts architecture for all-in-one multilingual scene text recognition, along with the TextMuSS-10M synthetic dataset, achieving state-of-the-art accuracy on benchmarks.

This article explains in detail MoE (Mixture of Experts) inference engineering, corrects misconceptions about activated parameters and deployment costs, and delves into technical details such as router selection, runtime grouping, GPU execution, memory management, and expert parallelism.

The article explains the engineering aspects of Mixture-of-Experts (MoE) inference, detailing token routing, expert batching, GPU distribution, and performance trade-offs for efficient serving.

Flyweight is an open-source C++/CUDA inference engine for running large MoE models on consumer GPUs with system RAM, offering optimized performance and compatibility with models like Qwen and DeepSeek.

An unofficial fork of llama.cpp introduces a persistent expert pool for MoE models, optimized to reduce expert re-copies over PCIe on 16GB AMD gfx906 GPUs, thereby improving decode throughput for large context lengths.

OceanMoE introduces a Mixture-of-Experts framework with structured conditional sparse computation to balance shared ocean context and adaptive specialization for multivariate ocean forecasting, demonstrating improved accuracy in long-horizon predictions.

This paper proposes an Infinite-Parameter LLM architecture that uses a hypernetwork to generate weights from live data via Bayesian updates, enabling continuous adaptation and outperforming in-context learning.

Edge0 is a streaming MoE inference engine that uses trained routing prediction to serve 35B Mixture-of-Experts models from SSD, achieving near-fp16 performance on consumer hardware with low memory usage.