[Show / Question] Building an on-device, fully local Agent on a 4B model (Gemma 4 / Ministral) across Mobile & Desktop. Facing the reality of on-device limits—where should on-device agents go from here?

English isn't my first language,Sorry for any weird wording,using a translator here! Like many here, I’m obsessed with true privacy sovereignty and local-first AI. Over the past few months, I've been building Agro — an open-source, 100% on-device cross-platform LLM and autonomous agent client running on Android, iOS, macOS, Windows, and Linux. The project is built on Kotlin Multiplatform (Compose Multiplatform) on top of Google’s native LiteRT-LM C++ runtime (with Apple Metal, WebGPU Dawn, Vulkan, and OpenCL acceleration). It runs models like Gemma 3 / 4 (4B) and Ministral-3-3B quite smoothly on mid-to-high-end phones and modern laptops. GitHub: https://github.com/Onion99/Agro Releases (APK, DMG, EXE, AppImage): https://github.com/Onion99/Agro/releases 🧗 The Dilemma While basic tool-calling works well with a 3B~4B model, I find myself at an architectural crossroads. Running autonomous agents locally on edge devices faces brutal hardware constraints (thermal throttling, 4-8GB mobile RAM ceilings, slow token generation, and tiny effective context windows). I'd love to learn from experienced builders in this community: If you were aiming to make on-device agents genuinely useful (rather than just a toy), HWhat possible directions would you consider? Any feedback on the architecture, technical critique, or directional advice would be deeply appreciated! If you have an device, feel free to try the binaries from the release page and let me know how it performs on your hardware.