I thought I’d done something extraordinary by running massive models on standard smartphones but

Reddit r/AI_Agents Tools
open-source mobile-ai moe-models edge-inference llama-cpp

Summary

Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.

Hi everyone, I’m the creator of bigedgeonmoe, an open-source codebase that allows you to run massive MoE models (ranging from Qwen 35B to open-source 120B models) on mobile devices or consumer PCs. And at impressive speeds, too: Qwen 35B (Q4) runs at 6 tokens/second on a mid-range phone with 12GB of RAM. It’s true that there aren't any specific use cases yet (or at least not any obvious ones), but I see other projects doing the same thing on Macs (using high-end GPUs and RAM) go viral, whereas my project handles everything on the CPU. It’s also modular relative to llama.cpp, so any model or quantization works as long as it’s supported by llama.cpp; plus, if a new model comes out, registering the architecture takes just a single line of code. Sorry for the rant, but this is a project I’ve really poured myself into.
Original Article

Similar Articles

24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)

Reddit r/LocalLLaMA

A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.

MobileMoE: Scaling On-Device Mixture of Experts

Hugging Face Daily Papers

MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.