Tag
A high school student built a custom C++ engine (fork of llama.rn) that clusters two Android phones over USB via RPC to run a 27B ternary-quantized LLM locally, using runtime Hadamard transforms for addition-only math that keeps the phone cool at ~30°C.
A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.
Demonstrates that Qwen 3.8 Flash Next can run locally on a mid-range Android phone at 3.5 tokens per second with optimizations and low quantization.
A user shares a hands-on comparison of running Gemma 4 with LiteRT-LM on mobile devices versus their previous llama.cpp setup, noting significantly better memory usage (1.5-2 GB vs 4-5 GB) and faster inference (2-4 seconds vs 7-10 seconds) on smartphones like Samsung S25 Ultra and iPhone 13 Pro Max.
Cactus-Compute released Needle, a 26M parameter open-source model distilled from Gemini for efficient on-device function calling using a novel Simple Attention Network architecture without MLPs.