Apple M5 isn't making full use of its matmul cores yet
Summary
Apple M5 silicon supports INT8 activations for matrix multiplication, but inference backends like MLX and Llama.cpp currently use 16-bit; custom w8a8 kernels achieve up to 1.4x speedup on Gemma4 prefill tasks.
Similar Articles
We added W8A8 activation quantization to MLX — prefill went from 2.84s to 2.52s on M5 Pro
Mininglamp AI released Cider, a small SDK that adds W8A8 activation quantization to Apple's MLX framework, achieving up to 1.84x speedup on prefill for large language models on M5 Pro via custom Metal kernels. The tool works with any MLX model, with INT8 TensorOps support for M5 and above.
@ollama: Gemma 4 is now nearly 90% faster on Apple Silicon with Ollama using MLX! The speedup comes from improved multi-token pr…
Ollama announces that Gemma 4 is now nearly 90% faster on Apple Silicon using MLX, thanks to improved multi-token prediction enabled by default, with automatic tuning to avoid slowdown.
@ivanfioravanti: Apple M5 Max + MLX = raw power! Look at this demo I'm playing with "FasterLivePortrait-MLX" I started with MPS but resu…
The author demonstrates that migrating a LivePortrait implementation from MPS to Apple's MLX framework on an M5 Max chip results in significantly better performance and speed.
Gemma4 26b MoE running in MLX with turboquant (and custom kernel)
A developer successfully ran Gemma4 26b MoE on Apple MacBook Air M5 using MLX with turboquant and a custom kernel, achieving faster prompt processing and generation speeds than llama.cpp with lower memory usage. The implementation includes instructions for local deployment.
Optimizing On-Device Inference for Apple Silicon (20 minute read)
Apple's Lily engine optimizes on-device LLM inference for Apple silicon by leveraging unified memory and hardware, outperforming MLX-LM, and is tuned for the Qwen3.6-35B-A3B model's architecture.