edge-ai

The ExecuTorch Hackathon is a two-day event in San Francisco where developers form teams to build and deploy PyTorch models on edge hardware using the ExecuTorch framework, with tracks for compute, mobile+XR, and IoT, sponsored by Meta, Qualcomm, and others.

PrismML showcases its tiny 1-bit Bonsai LLM at Qualcomm's Snapdragon Summit, which can run locally on smart glasses powered by the Snapdragon AR1 Gen 1 Platform, enabling real-time vision and language processing.

PAANI is an on-device perception-to-guidance architecture for river-robot simulation that combines YOLO11n and MobileNetV3-Small models with evidence fusion on Arduino UNO Q to provide explainable advisories for navigation, evaluated with promising results for edge AI applications.

A two-day hackathon at San Francisco State University, organized by Meta and Arm, focused on building AI applications on the edge using ExecuTorch on Arm-powered devices, with technical talks, workshops, and prizes.

By 2031, open and on-prem models are predicted to dominate routine inference in privacy-sensitive and cost-sensitive organizations, sidelining cloud-based AI companies like OpenAI and AnthropicAI.

Needle 3 is a compact AI foundation model optimized for edge devices like mobiles and wearables, offering tool calling, structured extraction, and text embedding in a single 8-29 MB file.

HP's ZGX Fury AI station is now orderable, featuring the GB300 superchip with 748GB unified memory for edge AI inference, and is paired with Red Hat AI Factory and NVIDIA collaboration for enterprise deployment.

This paper presents a federated learning approach for indoor fire detection that addresses bandwidth constraints, Byzantine attacks, and fixed-server issues through compressed updates and a rotating coordinator.

This paper proposes a knowledge distillation framework to compress vision-language models for on-device fire detection, showing that compact models can retain most of their teacher's capability while being deployable on embedded hardware.

Edge0-35B-A3B-preview is a sparse MoE model that enables efficient AI inference on mobile devices by using streaming expert offloading and quantization, achieving 15 tok/s with under 3 GiB of memory.

The article demonstrates voice conversations between Gemma 4 models on GPU and Jetson Orin hardware, using the open-source Cortexist Little Gemma engine for efficient inference with lip sync and gestures.

Ling 3.0 flash Fin AD quants can run locally on terminal devices via Atomic Chat to analyze Excel files and generate markdown summaries, enabling offline domain-specific AI workflows.

sanoTTS is a family of compact TTS models, with the smallest being 294k parameters, optimized for microcontrollers and outperforming larger models in benchmarks.

Liquid AI has released Liquid Nanos, a family of small foundation models (350M–2.6B parameters) that deliver frontier-grade performance on specialized tasks while running on everyday devices.

This research paper analyzes capability-stratified degradation in ternary quantized language models, showing that while factual knowledge deteriorates significantly, commonsense reasoning and downstream task adaptability are retained, making the models viable for efficient edge deployment.

This paper proposes a Generalized Optimization Engine (GOE) to accelerate AI inference on resource-constrained edge devices by integrating various model compression techniques, demonstrating that the choice of compression method affects task accuracy for language models deployed on GPU-less CPUs.

A tiny latent flow transformer with 2.4-4 million parameters, quantized to int8, implemented on an RP2350 microcontroller to generate 128x128 face images in approximately 20 seconds.

Reverse-engineered the Axera AX8850 NPU's int8 weight format to enable direct GGUF inference in llama.cpp, achieving up to 24.5 t/s decode and 716 t/s prefill on a Raspberry Pi 5, outperforming the vendor's runtime by 1.5×.

Thermo-FL is a federated LoRA fine-tuning framework for large language models on edge devices that uses device temperature to regulate training and transmission, paired with a robust aggregation method to defend against adversarial attacks, enhancing stability and performance.

Open-source inference engine FreeToken integrates CPU, GPU, and memory, enabling consumer hardware like 8GB VRAM laptops to run 35B MoE models, and home single-GPU gaming laptops to run 290B+ models, completely solving the VRAM limitation.