Llama cpp metal moe optimization
Summary
A developer shared a Metal optimization for llama.cpp that improves decode speed for IQ3_XXS models on Apple Silicon, with a GitHub pull request and a request for community testing.
Similar Articles
Anyone customizing and Optimizing llama.cpp per model?
The author proposes optimizing llama.cpp for specific models by stripping unnecessary code and using AI to automate the process, aiming for 2x+ performance improvements.
A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s
A llama.cpp pull request adds an x86 VNNI implementation for Q2_0 × Q8_0 dot products, delivering 3.0–3.6x CPU-only speedups on Bonsai models with bit-for-bit kernel correctness and 99.2% token agreement.
Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp
A pull request to llama.cpp refactors Intel XMX matrix engine support, delivering massive improvements to speculative decoding performance on Intel hardware.
Llama.cpp PR 8% speed boost
A llama.cpp PR moves sampling from CPU to GPU, yielding 8% faster tokens on an RTX 5090 and ~4% on a Tesla P40 for Qwen3.6-35B inference.
I ported EXL3 to run well on Apple Silicon - PonyExl3
Ported the EXL3 LLM codec to run on Apple Silicon via Metal, achieving high prefill and generation speeds on M5 Max (e.g., ~600 tok/s prefill, 17-80 tok/s gen on various models).