You don't need a GPU to run gemma-4-26B-A4B

Reddit r/LocalLLaMA 06/07/26, 07:24 AM Models

cpu-inference gemma llm open-source efficient koboldcpp no-gpu

Summary

The author demonstrates that the Gemma-4-26B-A4B model runs efficiently on a CPU-only system using Koboldcpp, achieving 7 tokens per second on an old desktop, suggesting that powerful GPUs may not be necessary for local LLM inference.

I've been running LLMs on my old potato i5-8500 with 32GB of RAM and \*no GPU\* for awhile now, running up to 12B dense models which run slow but perfectly useable. But this Gemma-4-26B-A4B simply flies on this CPU - only machine using Koboldcpp on Linux. That's right, an old used $150 desktop computer is running state of the art LLMs with something like 7 T/s. Yeah, go ahead and scoff. You can brag about your super-rig that costs more than a used car, but I'm bragging about a crappy old desktop I bought of ebay running the same thing that costs less than a night out. I keep thinking about buying a GPU but it's beginning to look like it might not be necessary. These smaller models are amazing without a GPU.

Original Article

Similar Articles

A 10 year old Xeon is all you need

Hacker News Top

A blog post detailing how to run the Gemma 4 AI model on a 10-year-old Xeon server with only CPU and DDR3 RAM, using customized llama.cpp optimizations.

@analogalok: Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what yo…

X AI KOLs Timeline

Alok demonstrates running Gemma 4 26B MoE on 8GB VRAM using Unsloth's QAT quant and the -cmoe flag in llama.cpp, achieving 20 tokens/sec with 250k context, marking a major milestone for budget local AI.

@analogalok: i just ran Google's brand new Unsloth Gemma4 12B dense GGUF on my RTX 4060 using llama.cpp + CUDA 13.2 21 tokens per se…

X AI KOLs Timeline

Google's new Gemma 4 12B is a single decoder-only transformer with encoder-free multimodal input, achieving strong benchmarks while being small enough to run locally on a budget GPU. It is released under Apache 2.0 license.

Gemma 4 26B Hits 600 Tok/s on One RTX 5090

Reddit r/LocalLLaMA

A benchmark shows that using vLLM with DFlash speculative decoding boosts Gemma 4 26B inference to ~578 tokens per second on a single RTX 5090, achieving a 2.56x speedup over baseline.

Ran gemma 4 12b on my 3090 yesterday and I think the local model game just changed

Reddit r/artificial

A user reports running Google's Gemma 4 12B model locally on a single RTX 3090 via GGUF quantization, finding strong performance including real 256k context, multimodal capabilities, and function calling that outperforms larger 70B models for coding tasks.