I thought I’d done something extraordinary by running massive models on standard smartphones but

Reddit r/AI_Agents Tools
open-source mobile-ai moe-models edge-inference llama-cpp

Summary

Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.

Hi everyone, I’m the creator of bigedgeonmoe, an open-source codebase that allows you to run massive MoE models (ranging from Qwen 35B to open-source 120B models) on mobile devices or consumer PCs. And at impressive speeds, too: Qwen 35B (Q4) runs at 6 tokens/second on a mid-range phone with 12GB of RAM. It’s true that there aren't any specific use cases yet (or at least not any obvious ones), but I see other projects doing the same thing on Macs (using high-end GPUs and RAM) go viral, whereas my project handles everything on the CPU. It’s also modular relative to llama.cpp, so any model or quantization works as long as it’s supported by llama.cpp; plus, if a new model comes out, registering the architecture takes just a single line of code. Sorry for the rant, but this is a project I’ve really poured myself into.
Original Article

Similar Articles

@aehyok: Share an open-source project FreeToken, a local inference engine specifically for running ultra-large Mixture-of-Experts (MoE) models on consumer-grade computers. Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → R…

X AI KOLs Timeline

FreeToken is an open-source local inference engine designed to run large Mixture-of-Experts (MoE) models on consumer-grade computers, offering significantly faster performance than alternatives like Ollama with easy installation and native GUI.

poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding

Reddit r/LocalLLaMA

作者将 llama.cpp 的 server 分支为 AgrillaMoE，针对 Qwen3.6-35B-A3B MoE 模型在 16GB 显存 GPU 上优化，配合 2-bit 量化可达 ~57-60 tok/s，并支持 OpenAI/Anthropic API 兼容接口；其 MoE 扩展补丁在运行时增加每 token 激活专家数（20 vs 8），无需重训即可在 GPQA-Diamond 上提升约 2.5 分。

24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)

Reddit r/LocalLLaMA

A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.