A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM

Reddit r/LocalLLaMA Tools
llama.cpp moe cuda gpu inference optimization caching

Summary

A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.

A new llama.cpp PR (#26563) adds a heatmap that tracks which MoE experts are used most often. Instead of keeping every expert on the GPU or offloading all of them, it caches the frequently selected experts in VRAM while the cold experts continue running on the CPU. The author’s results on Qwen3.6-35B-A3B with 8GB VRAM: Q2_M: 33.25 → 56.0 tok/s (1.68x) Q5_K_P: 17.34 → 35.93 tok/s (2.07x) Autofit enabled with --expert-hot-s -1 The negative results are probably more interesting: Qwen3.5-122B-A10B and Laguna-S-2.1 were actually slower with caching enabled. So this clearly isn’t a universal “make MoE faster” switch. My guess is that it only helps when expert reuse is high enough to outweigh the extra tracking and cache-management overhead. Current limitations: CUDA only Only active during single-token decoding Output can vary slightly depending on which experts are cached Still an open PR and not merged into llama.cpp This seems like a useful direction for running larger MoE models on consumer GPUs without destroying them with extremely low quants. Has anyone tested the branch on a 3060, 4060 or another 8–12GB card? I’d especially like to see hit rate and tok/s compared across coding, normal chat and long-context workloads. Source: llama.cpp PR #26563
Original Article

Similar Articles

Experts first llama.cpp

Reddit r/LocalLLaMA

A developer created an experimental fork of llama.cpp that offloads only used experts instead of entire layers to VRAM, achieving speed improvements for MoE models on GPUs with limited VRAM like the RTX 2060 12GB. The author is asking for testers to validate performance on other Nvidia GPUs.

24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)

Reddit r/LocalLLaMA

A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.

Llama.cpp PR 8% speed boost

Reddit r/LocalLLaMA

A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.