Tag
A personal project describes a village simulation where a 144M-param julia-1 'gut' model scores villagers' actions on CPU, embeddingGemma retrieves relevant memories, and Gemma 4 e4b (6GB laptop GPU) is only invoked when plans fail or surprises occur. Results show the gut is rarely uncertain, and nightly rule-learning from mistakes cut injuries from 2.73 to 0.53 per valley.
LFM3-1.7B and LFM3-VL-2B now run fully offline on iPhone and Android via the open-source Private Mind app, with detailed notes on what it took to export LFM3's novel architecture (gated convolutions, hybrid attention, sliding-window cache tweaks) to ExecuTorch, achieving ~40 tok/s on an iPhone 16 Pro GPU.
OpenBMB 的工程实践展示了如何通过 Lookahead Expert Scheduling（LES）在 iPhone 上以 20.51 tok/s 运行 35B-A3B MoE 模型，无需跳过专家或改变路由，相关实现已在 llama.cpp-omni 开源。
作者利用 Apple 设备内置的 3B 参数本地模型（通过 Apple Neural Engine 运行）为其 OpenClaw 智能体团队构建了一个答案校验系统，用以检测并拦截工具输出中的幻觉或错误回答，且不占用主模型算力和带宽。
Gradium 发布了首个开源模型 Phonon，这是一组 31M 和 40M 参数、专为边缘/设备端运行设计的 TTS 模型，其推理栈完全开源并支持 CPU、GPU、NPU 等 10 多种后端（大多基于 Rust）。
Tinkerbox 是一个可将 LiquidAI d1-3B 等决策模型高效运行在 Apple Neural Engine 上的应用，支持通过 macOS Shortcuts 触发器实现无代码自动化，延迟低至约 40ms，并计划集成 vLLM/Qwen3.5 系列模型。
@TheAhmadOsman 展示了在 RTX Spark 设备上运行 Qwen 3.8 Next Flash 模型可以实现一些有趣的效果，体现了小型本地模型在边缘硬件上的推理潜力。
Attendee reports that demo queues for RTX Spark laptop demos by Mike Bradley and Ahmad Osman were longer than queues to try the laptops themselves, highlighting strong interest in on-device 70B model inference on the new RTX Spark devices.
OpenBMB发布了MiniCPM-V 4.7的35B-A3B版本，这是MiniCPM-V系列的首个MoE模型，借助Lookahead Expert Scheduling等技术成为首个能在设备端流畅运行且不损失性能的开源MoE多模态模型，同时在性能与效率上达到新的帕累托前沿。
LemonSeed Studio 0.5.8 brings on-device LLM inference to iPadOS via an eGPU (AMD R9700 in a Thunderbolt enclosure), running Qwen3.8-27B Q4 at ~159 tok/s with DFlash2 speculative decoding across iPadOS, macOS and Linux using the unified LemonSeed Engine.
MiniCPM-V 4.7 35B-A3B is the first MoE model in the MiniCPM-V series, released in GGUF-quantized form for llama.cpp, enabling lossless on-device multimodal inference at 13+ tokens/s with only 3GB of memory on iPhone 17 Pro Max while surpassing Qwen3.6 35B-A3B in performance-efficiency.
Nishant Joshi argues that personal devices on users' desks already hold massive unused compute (≈15 TFLOPS, 46 GB RAM), and that shipping data to the cloud for work whose context is already local is pure waste. He contends servers should only be used for coordination, authority, and sync, though he acknowledges limits around engineering cost, billing trust, and frontier models.
Cactus Compute released Whistle, a 16.9 MB speech-to-text model running on CPU within the same engine as Needle, supporting seven languages and claiming mostly better accuracy than Whisper base at 9x smaller size and 6x speed.
A position paper introducing 'Model as a Library' (MaaL), a software architecture that packages small, community-enrolled speech models as versioned on-device dependencies for offline, hallucination-free data collection in low-resource African languages. It proposes keyword spotting to turn closed-vocabulary digital forms into voice-based, on-device data collection for low-literacy populations, and transpiling existing form tools into MaaL schemas.
HybridInfer introduces a thermal-aware reinforcement-learning router for multi-tier LLM inference, achieving higher quality and cost efficiency than heuristics on real Android devices.
This paper introduces a streamlined pipeline for training and deploying machine learning models on the WeBe Band wearable device, focusing on system-level automation and hardware-aware optimization to enable rapid iteration and on-device evaluation for edge AI applications.
LeanStream is a streaming speculate-and-refine framework that enables efficient on-device LLM inference by progressively refining computation and I/O operations, reducing memory usage and improving throughput.
Apple's Lily engine optimizes on-device LLM inference for Apple silicon by leveraging unified memory and hardware, outperforming MLX-LM, and is tuned for the Qwen3.6-35B-A3B model's architecture.
Ling-3.0-flash MXFP4, a quantized model, has been released and runs locally on a single DGX Spark, achieving ~80 tok/s decoding and 2,500-3,500 tok/s long-input prefilling, enabling private on-device inference for coding, agents, and offline batch jobs.
MacPaw partners with Liquid AI to bring on-device AI inference and local memory to its products and app store, planning to offer the tech stack to developers and introduce credit-based AI pricing.