I turned an Android phone into a Vulkan-accelerated local LLM node (GGUF + LiteLLM + Tailscale)

Reddit r/LocalLLaMA 06/03/26, 11:15 PM Tools

android vulkan llm gguf litellm tailscale self-hosted inference

Summary

An Android phone is repurposed as a portable GGUF inference server with Vulkan acceleration, exposing an OpenAI-compatible endpoint via LiteLLM and Tailscale mesh for integration into a self-hosted AI cluster.

Hey everyone — I’ve been working on something that finally reached a stable enough point to share. I’ve been experimenting with using an Android device as a local inference node inside a self-hosted AI mesh. The goal wasn’t “run a chatbot on Android,” but to make the phone behave like a portable GGUF inference server that plugs into an existing cluster. \## What it currently does \- Loads GGUF models locally on-device \- Uses Vulkan for mobile GPU acceleration \- Exposes an OpenAI-compatible endpoint on the mesh \- Routes through LiteLLM like any other backend \- Joins the cluster through Tailscale \- Supports fallback routing to larger local nodes \- Can run standalone when the rest of the mesh is unavailable \## Architecture \`\`\`text \[Android Pocket Node / Z Fold 6\] GGUF + Vulkan (gpu\_layers=89) llama.cpp JNI/NDK bridge OpenAI-compatible local endpoint ↓ \[Tailscale Mesh\] ↓ \[Edge Gate on neo-x510uar\] request pre-flight battery / thermal / prompt-size routing ↓ \[LiteLLM Router on neo-x510uar\] OpenAI-compatible gateway model aliases fallback routing ↓ \[Fallback Nodes\] sheens-mac-studio — heavier reasoning / judge models moolah — RTX box for GPU-heavy workloads

Original Article

Similar Articles

I made this android app which runs ai models locally

Reddit r/artificial

A developer created an Android app that runs AI models locally, supporting GGUF and LiteRT formats with multiple ways to add models.

I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating)

Reddit r/LocalLLaMA

A high school student built a custom C++ engine (fork of llama.rn) that clusters two Android phones over USB via RPC to run a 27B ternary-quantized LLM locally, using runtime Hadamard transforms for addition-only math that keeps the phone cool at ~30°C.

LFM3-1.7B and LFM3-VL-2B running fully offline on iPhone and Android, plus what it took to export LFM3 to ExecuTorch (open source app)

Reddit r/LocalLLaMA

LFM3-1.7B and LFM3-VL-2B now run fully offline on iPhone and Android via the open-source Private Mind app, with detailed notes on what it took to export LFM3's novel architecture (gated convolutions, hybrid attention, sliding-window cache tweaks) to ExecuTorch, achieving ~40 tok/s on an iPhone 16 Pro GPU.

@victormustar: 300tok/s on mobile is insane... open source must win

X AI KOLs Following

Open source AI inference reaches 300 tok/s on mobile, with a WebGPU framework pushing Liquid AI's LFM2.5 230M to 1,400 tok/s in browser.

@TheTechDiggest: [OpenSource - Distributed AI & Mesh LLM Inference] Buying an expensive enterprise GPU isn't the only way to run massive…

X AI KOLs Timeline

A tweet thread introduces mesh-llm, an open-source tool that pools local network devices into a unified, OpenAI-compatible API for running large LLMs without expensive enterprise GPUs.