Expert expansion with llama.cpp
Summary
A developer built a custom branch of llama.cpp that implements expert expansion for Mixture-of-Experts models, tested it on Metal, and is seeking cross-platform feedback.
Similar Articles
Experts first llama.cpp
A developer created an experimental fork of llama.cpp that offloads only used experts instead of entire layers to VRAM, achieving speed improvements for MoE models on GPUs with limited VRAM like the RTX 2060 12GB. The author is asking for testers to validate performance on other Nvidia GPUs.
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.
llama.cpp slower on P-Cores than on E-Cores with MoE Model and GPU+CPU offloading?
Observation that llama.cpp runs slower on P-cores than E-cores when running Mixture of Experts models with GPU+CPU offloading.
Emergent Modularity in Mixture-of-Experts Models (8 minute read)
Ai2 releases EMO, a 14B-parameter mixture-of-experts language model trained to develop emergent modularity. It allows using a small subset of experts for specific tasks while maintaining near full-model performance.
I kept rewriting parameters every time I swapped models on vLLM and llama.cpp, so I built a tool to manage them (llmux, MIT)
A developer built llmux, an open-source tool to simplify managing model profiles across different engines like vLLM and llama.cpp, allowing one-click switching with Docker and NVIDIA GPU support.