Show HN: Maple-Preview – ternary 20B MoE running at 120 tok/s on a iPhone
Summary
Maple-Preview is a ternary 20B MoE model that runs at 120 tokens per second on an iPhone, showcasing efficient on-device inference.
Similar Articles
I've added Maple-Preview to Mference, got 40 tps generation with 500MB of used RAM on Air M4
The author adds Maple-Preview support to Mference, a tool that streams MoE experts from disk to run large models on low-RAM devices, achieving 40 tps with 500MB RAM on an Air M4.
35B MoE running at 20.5 tok/s on an iPhone, without skipping experts
OpenBMB 的工程实践展示了如何通过 Lookahead Expert Scheduling(LES)在 iPhone 上以 20.51 tok/s 运行 35B-A3B MoE 模型,无需跳过专家或改变路由,相关实现已在 llama.cpp-omni 开源。
deepgrove/maple-preview
DeepGrove releases Maple-Preview, an open-source 20B-A1B ternary-weight reasoning LLM with SOTA reasoning for its weight class, capable of 200+ tokens/sec on a Mac mini M4 and competitive with larger models.
Edge0/Edge0-35B-A3B-preview
Edge0-35B-A3B-preview is a sparse MoE model that enables efficient AI inference on mobile devices by using streaming expert offloading and quantization, achieving 15 tok/s with under 3 GiB of memory.
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.