CPU-only inference on a Celeron N5095 SBC: 6 models from 0.6B to 8B, benchmarked
Summary
This post benchmarks six AI models from 0.6B to 8B parameters running CPU-only inference on a Celeron N5095 single-board computer, providing performance comparisons.
Similar Articles
Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.
The author optimized a matmul kernel for BitNet's ternary models on CPU, achieving 29x speedup in isolation, but found that the model is memory-bound, resulting in only 6-10% end-to-end gain. The inference engine is available as open-source.
AMD's tiny AI PC points to a more local future for model inference
AMD's Ryzen AI Max platform with 128GB unified memory enables local inference of large models up to 200 billion parameters, aiming to shift AI workloads from cloud to compact personal hardware.
@liquidai: Introducing LFM2.5-230M: our smallest model yet, built to run fast anywhere (CPUs, NPUs, and GPUs) to enable agentic ta…
Liquid AI releases LFM2.5-230M, a small 230M parameter model optimized for fast inference on CPUs, NPUs, and GPUs, targeting agentic tasks on devices like phones and robots.
@0xSero: Best models for your hardware this week. 8-12GB - https://huggingface.co/LiquidAI/LFM2.5-8B-A1B… incredible model, so f…
A curated weekly roundup of the best AI models for different hardware configurations, from 8GB to 768GB VRAM, highlighting performance and benchmarks.
Built a from-scratch BitNet inference engine in pure C — 1.8× faster than bitnet.cpp on Xeon (36 tok/s), zero dependencies [BitNet & Bonsai CPU testers wanted]
Project Zero is a from-scratch C99 LLM inference engine that runs BitNet and Qwen Bonsai-27B on CPU with zero dependencies, achieving 1.8× speedup over bitnet.cpp on Xeon. The project seeks community benchmarks for both models.