Show HN: Maple-Preview – ternary 20B MoE running at 120 tok/s on a iPhone
Summary
Maple-Preview is a ternary 20B MoE model that runs at 120 tokens per second on an iPhone, showcasing efficient on-device inference.
Similar Articles
deepgrove/maple-preview
DeepGrove releases Maple-Preview, an open-source 20B-A1B ternary-weight reasoning LLM with SOTA reasoning for its weight class, capable of 200+ tokens/sec on a Mac mini M4 and competitive with larger models.
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.
@rohanpaul_ai: So much possibilities for on-device small models. Here @adrgrondin is running Google’s Gemma 4 E2B on iPhone 17 Pro. ~4…
Google's Gemma 4 E2B is demonstrated running on an iPhone 17 Pro via MLX optimization, achieving ~40 tokens/second with 128K context and offline thinking mode for coding and math.
24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)
A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.
Inkling-Small 276B-A12B at ~2.9 tok/s on <10gb memory
Mference, a Swift + Metal inference engine, now supports Inkling-Small 276B-A12B, running it at ~2.9 tok/s on under 10GB memory, enabling large MoE models on consumer Apple hardware.