llama.cpp slower on P-Cores than on E-Cores with MoE Model and GPU+CPU offloading?
Summary
Observation that llama.cpp runs slower on P-cores than E-cores when running Mixture of Experts models with GPU+CPU offloading.
Similar Articles
Experts first llama.cpp
A developer created an experimental fork of llama.cpp that offloads only used experts instead of entire layers to VRAM, achieving speed improvements for MoE models on GPUs with limited VRAM like the RTX 2060 12GB. The author is asking for testers to validate performance on other Nvidia GPUs.
A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM
A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.
Drastically improve prompt processing speed for --n-cpu-moe partially offloaded models
The article shares a performance optimization trick for llama.cpp, showing that increasing the micro-batch size (`-ub`) combined with partial CPU offloading (`--n-cpu-moe`) can drastically improve prompt processing speed for large models like gpt-oss-120b on consumer GPUs.
exllamav3 comfortably beats llama.cpp running CPU-offloaded Qwen-3.8-Flash-Next on my setup!
A user benchmarks exllamav3 against llama.cpp for CPU-offloaded inference, showing exllamav3 is faster for Qwen models but slower for others, depending on hardware and model architecture.
Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
This paper presents an empirical study on scheduling multiple LLMs on shared heterogeneous hardware, focusing on performance implications of CPU-GPU offloading and preemption. It finds that offloading causes non-linear decode degradation, especially for smaller models, and preemption overhead is dominated by model state reload, providing design guidance for future multi-model schedulers.