@leopardracer: SAME GPU SAME MODEL SAME CONTEXT AND 2X THE SPEED rtx 4060, gemma 4 12b, 48k context just switched the quantization fro…
Summary
Changing quantization from q4_k_m to q4_k_xl in llama.cpp doubles inference speed on the same GPU without hardware or driver changes, as demonstrated with Gemma 4 12B on an RTX 4060.
View Cached Full Text
Cached at: 06/09/26, 01:36 AM
SAME GPU SAME MODEL SAME CONTEXT AND 2X THE SPEED
rtx 4060, gemma 4 12b, 48k context
just switched the quantization from q4_k_m to q4_k_xl and went from 15 tokens per second to 32
no new hardware, no new drivers, no new model, just one parameter changed in llama.cpp
most people running local llms are leaving half their gpu’s speed on the table and don’t even know it
the full breakdown of tools and configs is in the article below ↓
Similar Articles
RTX5090, gemma-4-31B-it-Q6_K.gguf. Context: before - 35k, after - 80k!
Running the quantized Gemma-4-31B model on an RTX 5090 increases context length from 35k to 80k, showcasing significant performance improvement.
@leopardracer: GEMMA 4 26B ON AN RTX 4060 WITH A 248K TOKEN CONTEXT WINDOW 20 tokens per second and a context window so large you can …
Gemma 4 26B runs on an RTX 4060 with 248K token context at 20 tokens per second using llama.cpp and Q4_K_XL quantization, enabling local processing of entire codebases on consumer hardware.
@DogukanUrker: Gemma 4 12B on a single RTX 3060: the full 262,144 context at ~100 tok/s. (config below) dense model -> MTP speculative…
DogukanUrker demonstrates running Gemma 4 12B with full 262,144 context at ~100 tok/s on a single RTX 3060 using speculative decoding and KV cache splitting, achieving nearly full GPU utilization without CPU offload.
Ran gemma 4 12b on my 3090 yesterday and I think the local model game just changed
A user reports running Google's Gemma 4 12B model locally on a single RTX 3090 via GGUF quantization, finding strong performance including real 256k context, multimodal capabilities, and function calling that outperforms larger 70B models for coding tasks.
@analogalok: Gemma 4 12B QAT (dense) achieves 1000+ tokens/sec prefill on 8GB VRAM with 120k context Gemma 4 12B QAT (dense), TurboQ…
Gemma 4 12B QAT (dense) achieves over 1000 tokens per second prefill on an 8GB RTX 4060 with 120k context using TurboQuant, enabling full GPU layer offloading. This represents a 42% increase in prefill speed over previous methods.