Apple M5 isn't making full use of its matmul cores yet
Summary
Apple M5 silicon supports INT8 activations for matrix multiplication, but inference backends like MLX and Llama.cpp currently use 16-bit; custom w8a8 kernels achieve up to 1.4x speedup on Gemma4 prefill tasks.
Similar Articles
We added W8A8 activation quantization to MLX — prefill went from 2.84s to 2.52s on M5 Pro
Mininglamp AI released Cider, a small SDK that adds W8A8 activation quantization to Apple's MLX framework, achieving up to 1.84x speedup on prefill for large language models on M5 Pro via custom Metal kernels. The tool works with any MLX model, with INT8 TensorOps support for M5 and above.
@ollama: Gemma 4 is now nearly 90% faster on Apple Silicon with Ollama using MLX! The speedup comes from improved multi-token pr…
Ollama announces that Gemma 4 is now nearly 90% faster on Apple Silicon using MLX, thanks to improved multi-token prediction enabled by default, with automatic tuning to avoid slowdown.
@ivanfioravanti: Apple M5 Max + MLX = raw power! Look at this demo I'm playing with "FasterLivePortrait-MLX" I started with MPS but resu…
The author demonstrates that migrating a LivePortrait implementation from MPS to Apple's MLX framework on an M5 Max chip results in significantly better performance and speed.
Gemma4 26b MoE running in MLX with turboquant (and custom kernel)
A developer successfully ran Gemma4 26b MoE on Apple MacBook Air M5 using MLX with turboquant and a custom kernel, achieving faster prompt processing and generation speeds than llama.cpp with lower memory usage. The implementation includes instructions for local deployment.
@0x0SojalSec: Apple hid 15.8 TFLOPS of raw AI power in every M4 Mac & iPhone. They only let you use the Neural Engine for inference. …
A developer reverse-engineered Apple's private APIs to enable training neural networks directly on the Apple Neural Engine (ANE) in M4 Macs and iPhones, bypassing CoreML and GPU. The project demonstrates that ANE hardware is capable of training, though with limitations like low utilization and CPU fallbacks for some operations.