CPU-only inference on a Celeron N5095 SBC: 6 models from 0.6B to 8B, benchmarked
Summary
This post benchmarks six AI models from 0.6B to 8B parameters running CPU-only inference on a Celeron N5095 single-board computer, providing performance comparisons.
Similar Articles
Benchmarking Pocket-Scale Inference
This article benchmarks AI inference performance on the iPhone 17 Pro, evaluating metrics like generation time and model intelligence across various tasks to assess real-world mobile device usage.
Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.
The author optimized a matmul kernel for BitNet's ternary models on CPU, achieving 29x speedup in isolation, but found that the model is memory-bound, resulting in only 6-10% end-to-end gain. The inference engine is available as open-source.
AMD's tiny AI PC points to a more local future for model inference
AMD's Ryzen AI Max platform with 128GB unified memory enables local inference of large models up to 200 billion parameters, aiming to shift AI workloads from cloud to compact personal hardware.
1-bit 27B in the browser: 25–30 tok/s on a 6 GB RTX 3060 Laptop (WebGPU, no install)
Mentria.ai's browser-based inference engine achieves 25-30 tokens per second with a 1-bit 27B parameter model on a 6 GB RTX 3060 laptop GPU using WebGPU, enabling efficient AI inference without server installation.
Releasing smolbenchmark: Helps you choose the best model for your hardware!
smolbenchmark is a new resource that benchmarks small AI models on consumer hardware devices, providing metrics like speed, efficiency, and thermals to help users choose the best model for their specific hardware.