Tag
llama.cpp PR #29887 adds an LRU MoE expert cache that exploits locality of reference, boosting Qwen3.8-Flash-Next (94GB) runs to 67.8 tok/s on an RTX 5090 (2.2x), with SGLang merging a similar 'paged experts' feature days later.
A detailed blog post describing how to dramatically speed up GLM-5.2 inference on a dual Grace Hopper system from 2.5 tok/s to over 50 tok/s by stopping model cross-module traffic and grafting an FP8 MTP head onto the INT4 base.
TIDE is a lossless inference system for diffusion large language models that leverages temporal stability of expert activations to reduce I/O overhead and computation, achieving up to 1.4-1.5x throughput improvements on single GPU-CPU systems.