Expert expansion with llama.cpp
Summary
A developer built a custom branch of llama.cpp that implements expert expansion for Mixture-of-Experts models, tested it on Metal, and is seeking cross-platform feedback.
Similar Articles
CUDA: fuse shared experts into MMVQ by am17an · Pull Request #29184 · ggml-org/llama.cpp
A llama.cpp pull request (#29184) by am17an fuses shared experts into the MMVQ (matrix-matrix vector quantization) CUDA kernel, optimizing inference for mixture-of-experts models on GPUs.
Experts first llama.cpp
A developer created an experimental fork of llama.cpp that offloads only used experts instead of entire layers to VRAM, achieving speed improvements for MoE models on GPUs with limited VRAM like the RTX 2060 12GB. The author is asking for testers to validate performance on other Nvidia GPUs.
llama: add Maple 20B-A1B ternary MoE architecture (CPU) by AlexGabbia · Pull Request #27000 · ggml-org/llama.cpp
This pull request adds support for the Maple 20B-A1B ternary Mixture of Experts architecture for CPU inference in the llama.cpp framework.
llama.cpp expert-pool fork for Qwen 3.8 Flash next IQ4 + 16Gb VRAM tested on MI50 gfx906 with parameters
An unofficial fork of llama.cpp introduces a persistent expert pool for MoE models, optimized to reduce expert re-copies over PCIe on 16GB AMD gfx906 GPUs, thereby improving decode throughput for large context lengths.
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.