on-device-ai

Google AI Edge Gallery v1.0.13 & v1.0.14 updates add support for Gemma 4 with multi-token prediction, Pixel TPU optimization, experimental MCP, new skills, and chat history saving, enhancing on-device generative AI capabilities.

General Instinct launches a deployment layer that enables frontier AI models to run on constrained edge hardware like Jetsons and mobile NPUs, helping robotics and physical AI teams achieve low-latency offline inference.

A tweet argues the next AI boom will be compact intelligence on edge devices rather than larger data centers, with Liquid AI supporting the vision of running AI on phones, cars, and everyday devices.

VecML demonstrates its AI-PC software running RAG on 200K documents using the new Snapdragon X2 laptop, achieving low-token and low-memory retrieval. The software integrates multiple database functions into one platform, and controlled testing for macOS is now open.

A user shares a hands-on comparison of running Gemma 4 with LiteRT-LM on mobile devices versus their previous llama.cpp setup, noting significantly better memory usage (1.5-2 GB vs 4-5 GB) and faster inference (2-4 seconds vs 7-10 seconds) on smartphones like Samsung S25 Ultra and iPhone 13 Pro Max.

Jina releases MLX-native embedding models simultaneously with PyTorch versions, highlighting the growing importance of Apple's MLX framework for local AI deployment.

Cactus-Compute released Needle, a 26M parameter open-source model distilled from Gemini for efficient on-device function calling using a novel Simple Attention Network architecture without MLPs.

The article highlights Supertonic, an open-source text-to-speech model that runs entirely on-device, claiming superior speed and formatting accuracy compared to cloud-based services like ElevenLabs and OpenAI.

The author announces their live talk titled 'On-device Intelligence using MLX' at the aiDotEngineer event, expressing gratitude to the organizers and community contributors.

Papel is a new research-focused social platform that leverages AI-powered vector search and on-device RAG to help researchers discover, discuss, and quiz themselves on academic papers. It offers personalized feeds, local AI chat via Apple Intelligence or MLX, and gamified learning features.

The author introduces SAVI, an iOS app designed for ADHD users that converts voice brain dumps into structured tasks and reminders using on-device AI like Whisper and GPT-4o.

Google Chrome is automatically downloading a 4GB Gemini Nano model weights file to users' devices to power on-device AI features like scam detection and writing assistance, often without clear notification about storage requirements. Users can disable the On-Device AI toggle in Chrome settings to remove the file and prevent re-downloads.

Garry Tan highlights a model with a 1M token context window and coding agent capabilities running locally on a 128GB MacBook Pro, expressing excitement about the milestone.

atomic.chat has optimized Gemma 4 26B inference in LLaMA.cpp, achieving ~40% faster token generation on MacBook Pro M5 Max using Multi-Token Prediction (MTP) speculative decoding. This is a notable win for local AI users running desktop apps, coding agents, and private on-device assistants.

This technical report introduces X-OmniClaw, a unified mobile agent system designed for multimodal understanding and interaction on Android devices. It details the architecture for perception, memory management, and action execution using on-device AI capabilities.

Supertonic 3 is a lightweight, open-weight text-to-speech model designed for fast on-device inference, expanding support to 31 languages with improved stability and expression tags.

MIT researchers developed a new framework called FTTE that accelerates privacy-preserving federated learning by 81%, enabling efficient AI training on resource-constrained edge devices like smartwatches and sensors.

Tencent's AngelSlim team released Hy-MT1.5-1.8B-1.25bit, a highly compressed 1.25-bit machine translation model supporting 33 languages that fits in 440MB for on-device use. It utilizes the Sherry quantization algorithm to achieve world-class translation quality comparable to much larger models.

Google DeepMind releases Gemma 4, a family of open-weights multimodal models featuring Multi-Token Prediction (MTP) for up to 2x decoding speedups, supporting text, image, video, and audio with enhanced reasoning and coding capabilities.

A user reflects on why more apps don’t run local LLMs directly on phones, noting Gemma 2-4B models already work offline and could eliminate server costs while maintaining near-GPT-4o quality.