Tag
Liquid AI released LFM2.5-2.6B, a 2.69B parameter model with 128K context and tool calling, optimized for multi-step agent workflows and capable of running at 30 tok/s on a phone with a 1.67GB Q4_K_M GGUF, though coding and knowledge-heavy tasks remain weak compared to larger models.
Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.
Microsoft releases VibeVoice-ASR-BitNet, a compressed multilingual ASR model for real-time CPU inference. It achieves 1.6-2.3x faster inference than Whisper.cpp with real-time capability on as few as 3 CPU threads.
StickyMoE proposes a differentiable routing consistency loss that encourages adjacent tokens to activate the same experts in MoE models, reducing expert-swapping overhead and cache misses during inference on edge devices by up to 3.92× while improving perplexity.
BaseRT is a native Metal inference runtime for LLMs on Apple Silicon, achieving up to 1.56x higher decode throughput than llama.cpp and 1.35x higher than MLX across tested models.
This paper presents a cascaded multi-granularity pruning framework for deploying LLMs on Industrial IoT edge devices, achieving up to 13.8x compression with minimal accuracy loss on MHA+GELU architectures while exposing a collapse on GQA+SwiGLU designs.
Liquid AI released LFM2.5-230M, a compact 230M-parameter hybrid model optimized for on-device deployment with fast edge inference speeds (213 tok/s on Galaxy S25 Ultra) and built for agentic tasks via reinforcement learning.
Using TurboQuant, the user achieved 20 tokens per second on a Qwen 3.6 35B MoE model running on a GTX1060 3GB, showcasing impressive performance on outdated hardware.
Loft Orbital's YAM-9 satellite runs Google's Gemma 3 vision-language model onboard for real-time image analysis, reducing downlink bandwidth and latency by deciding what data to send to Earth.
This paper systematically studies variants of QKV projection sharing in transformers, finding that sharing key and value projections (Q-K=V) achieves 50% KV cache reduction with only 3.1% perplexity degradation, and combining with GQA/MQA can reach up to 96.9% cache reduction—enabling practical on-device inference with minimal quality loss.
A developer successfully ran the 284B-parameter DeepSeek-V4-Flash model on a Raspberry Pi 5 at over 1 tok/s, using an untouched GGUF file from antirez after extensive experimentation.
A tweet demonstrates that Multi-Token Prediction (MTP) achieves significant speedups for Qwen models on dual RTX 5090 hardware, suggesting that local inference can now rival cloud-model performance.
A developer built a fully offline suitcase robot named Sparky using a Jetson Orin NX and Gemma 4 E4B model, achieving ~200ms cached TTFT and 14-15 tok/s with 30+ sensors feeding into the prompt as natural language, all without network connectivity.
A developer ran 10 concurrent agents of the 35B-parameter Qwen3.6 model on a single 74W GB10 GPU at 436 tok/s total using vLLM, demonstrating high-efficiency edge deployment.
Bitnet.cpp presents a mixed-precision matrix multiplication library for efficient edge inference of ternary LLMs like BitNet b1.58, achieving up to 6.25x speedup over full-precision baselines. The system is open-sourced on GitHub.