A barebones CPU-only inference engine for Qwen 3, written from scratch in pure C
Summary
A minimal CPU-only inference engine for Qwen 3 models implemented from scratch in pure C.
Similar Articles
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
First attempts at a CPU setup - MS-02 Intel 285hx, trying Qwen3, Qwen3.6 and Gemma4
Testing AI models Qwen3, Qwen3.6, and Gemma4 on a CPU setup using the Intel 285hx processor (MS-02).
I ran Qwen3.5-0.8B on a sub-$20 CPU chip in under 512MiB of memory
The author ran Qwen3.5-0.8B on a $10-20 Amlogic A113X CPU chip with a custom C runtime, achieving 1.82 tok/s decode and under 490 MiB peak RSS, demonstrating that small LLM inference can run on deployed edge hardware without a GPU.
Running Qwen3.5-122B on Mac Studio 96GB: Fixed 3 bugs that made long-context inference usable
Fixed three bugs in a qMLX fork for running Qwen3.5-122B on Mac Studio, reducing prefill time from minutes to sub-seconds for long-context inference; open-sourced the fork and benchmark script.
Qwen 3.8 27B available on Cerebras at 1500 tokens/s
Cerebras has announced the availability of the Qwen 3.8 27B model on its inference platform with a speed of 1500 tokens per second, detailing its model compression techniques such as quantization and pruning.