edge-ai

This paper proposes a knowledge-adaptive edge expert agent architecture for ecological monitoring, separating visual perception from reasoning to reduce reliance on cloud resources and enable sustainable on-device AI in remote deployments.

ExecuTorch, PyTorch's on-device AI deployment framework, won the Best Industry Paper Award at MLSysConf 2026. The paper introduces a unified solution for running models on diverse hardware, from microcontrollers to SoCs.

A user shares a hands-on comparison of running Gemma 4 with LiteRT-LM on mobile devices versus their previous llama.cpp setup, noting significantly better memory usage (1.5-2 GB vs 4-5 GB) and faster inference (2-4 seconds vs 7-10 seconds) on smartphones like Samsung S25 Ultra and iPhone 13 Pro Max.

Sipeed's new K3 RISC-V single-board computers feature 32GB LPDDR5 and a 60 TOPS NPU, enabling local inference of large language models at up to 15 tokens per second.

The article highlights the ability to run Qwen3-35B-A3B locally on a laptop for free using llama.cpp and Unsloth 4-bit quantization.

A developer successfully runs a quantized TinyStories transformer model locally on a stock Game Boy Color using custom ROM and fixed-point math.

This article introduces ExecuTorch, a unified PyTorch-native deployment framework designed to run AI models on diverse edge devices without requiring model conversion or reimplementation.

This study reveals a 'Smart Pruning Paradox' where activation-aware pruning methods like Wanda preserve perplexity but significantly amplify bias in Large Language Models deployed on edge devices.

OpenBMB released MiniCPM-V 4.6, a 1.3B parameter multimodal model. Using high-resolution visual processing and efficient compression, it achieves fast inference on consumer hardware and mobile phones, outperforming larger models. It is fully open-source and supports multiple inference and quantization frameworks.

Demonstrates running Gemma 4 offline in the browser using WebGPU and Transformers.js to control a Reachy Mini robot via WebSerial.

This paper introduces a neuro-symbolic pipeline using 2.5-D decomposition to improve LLM-based spatial construction accuracy by offloading vertical coordinate calculation to a deterministic executor, achieving high accuracy on benchmarks and edge hardware.

A new 20B+ parameter MoE model from OpenAI, quantized to 3-bit via TurboQuant and optimized with MLX, allows for high-performance local LLM inference on standard 16GB MacBooks.

Google released Gemma 4, an open-source AI model optimized for local execution on standard laptops, offering 3x faster performance and a 256k context window for free under an Apache 2.0 license.

A discussion post exploring where edge AI will have the greatest impact: autonomy and robotics, low-power vision systems, private local LLMs, or bandwidth-constrained industrial deployments.

The article discusses the growing viability of local AI models for everyday tasks, suggesting a shift toward hybrid architectures that optimize for cost and latency rather than relying solely on frontier cloud models.

MiniCPM-o 4.5 is a 9B parameter multimodal model featuring Omni-Flow, a framework enabling real-time full-duplex interaction where the model can simultaneously perceive and respond proactively. It achieves state-of-the-art open-source performance comparable to Gemini 2.5 Flash and runs on edge devices with less than 12GB RAM.

Tencent's AngelSlim team released Hy-MT1.5-1.8B-1.25bit, a highly compressed 1.25-bit machine translation model supporting 33 languages that fits in 440MB for on-device use. It utilizes the Sherry quantization algorithm to achieve world-class translation quality comparable to much larger models.

Anker unveiled its custom Thus AI chip using compute-in-memory architecture to enable local AI on tiny devices, starting with upcoming Soundcore flagship earbuds for superior call noise cancellation.

NVIDIA and Hugging Face publish a hands-on demo showing Gemma 4 running as a vision-language-action model entirely on the Jetson Orin Nano Super, using local STT/TTS and webcam input.

Soul Player C64 implements a real 2-layer decoder-only transformer with ~25,000 int8 parameters in hand-written 6502/6510 assembly, running entirely on an unmodified 1 MHz Commodore 64 loaded from a floppy disk. The project includes training scripts to build and quantize custom models, assemble C64 binaries, and run inference at roughly 60 seconds per token.