Splash on a 40-core M5 Max: +20% decode by tuning the kernels for your own chip

Reddit r/LocalLLaMA 09/25/26, 01:13 PM Tools

Summary

The article describes how to use the 'make tune-kernels' tool in Splash to optimize AI model decoding on 40-core M5 Max chips, achieving up to 20% speed improvement by tuning kernel layouts for specific hardware.

FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernels” that tests every available way of running each quantised matrix-multiply on your hardware. On my machine it found that the "split-K" layouts (each input row split four ways, with the partial sums combined at the end) are much faster for the 8-row step that checks draft tokens. Written up for Inco (https://github.com/incoai/splash/issues/154) In the meantime try it: build Splash from source (git clone https://github.com/incoai/splash, git checkout 1.0.2, make; needs Xcode 26+ with the Metal toolchain), then run build/engine-tests/tune-kernels build/splash.metallib <your model folder> --confirm on an idle Mac. The --confirm step tells you whether the winners actually speed up the whole forward pass on your chip. Use your model of choice to patch in. Swift Model conversions also available on hugging face here: https://huggingface.co/SiliconSpecies/Swift-1.5-Qwen3.8-27B-Splash

Original Article

Similar Articles

@jianchen1799: Local models can now handle agentic workloads. Local inference engines need to catch up. Today we’re releasing Splash: …

X AI KOLs Timeline

Inco AI releases Splash, an open-source inference engine optimized for Apple silicon, claiming up to 3× faster decode speeds for local model serving, enabling agentic workloads on devices like M5 Max MacBook Pro.

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro

Reddit r/LocalLLaMA

Inco Splash is an open-source inference engine optimized for Apple silicon, offering significant speed improvements for running AI models like Qwen3.8-27B on M-series MacBooks.

@svpino: DeepSeek-V4-Flash running at 5.71 token/s on a Mac M5 Pro. Every day, we get better models running on consumer hardware…

X AI KOLs Timeline

The tweet highlights DeepSeek-V4-Flash running at 5.71 tokens per second on a Mac M5 Pro, emphasizing advancements in local AI inference on consumer hardware, with a mention of Tencent's open-source Palm-Infra for Apple Silicon optimization.

Apple M5 isn't making full use of its matmul cores yet

Reddit r/LocalLLaMA

Apple M5 silicon supports INT8 activations for matrix multiplication, but inference backends like MLX and Llama.cpp currently use 16-bit; custom w8a8 kernels achieve up to 1.4x speedup on Gemma4 prefill tasks.

Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra

arXiv cs.LG

This paper presents a systematic optimization study of real-time diffusion model inference on the Apple M3 Ultra, achieving 22.7 FPS at 512x512 resolution using CoreML conversion and a distillation model, revealing that CUDA-optimized techniques do not directly transfer to Apple's unified memory architecture.