@DogukanUrker: Gemma 4 12B on a single RTX 3060: the full 262,144 context at ~100 tok/s. (config below) dense model -> MTP speculative…
Summary
DogukanUrker demonstrates running Gemma 4 12B with full 262,144 context at ~100 tok/s on a single RTX 3060 using speculative decoding and KV cache splitting, achieving nearly full GPU utilization without CPU offload.
View Cached Full Text
Cached at: 07/21/26, 06:47 PM
Gemma 4 12B on a single RTX 3060: the full 262,144 context at ~100 tok/s. (config below)
dense model -> MTP speculative decoding hits the full ~2.4 x 42 -> ~100 tok/s just by turning the drafter on (the 26B MoE only managed ~1.3x).
100% on the GPU, zero CPU offload, pinned at ~97% of the 12GB.
the trick to fit full context + MTP: split the KV cache – q8_0 on the target, f16 on the draft. quantize the draft cache and acceptance dies. holds ~70% at 262K.
llama-swap config:
llama-server -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf –model-draft mtp-gemma-4-12B-it.gguf -ngld 99 –spec-type draft-mtp –spec-draft-n-max 4 -ngl 99 -c 262144 -fa on –jinja -np 1 -ctk q8_0 -ctv q8_0 -ctkd f16 -ctvd f16 -b 4096 -ub 1280 –temp 1.0 –top-p 0.95 –top-k 64 –repeat-penalty 1.0
Similar Articles
Gemma 4 26B Hits 600 Tok/s on One RTX 5090
A benchmark shows that using vLLM with DFlash speculative decoding boosts Gemma 4 26B inference to ~578 tokens per second on a single RTX 5090, achieving a 2.56x speedup over baseline.
@analogalok: Gemma 4 12B QAT (dense) achieves 1000+ tokens/sec prefill on 8GB VRAM with 120k context Gemma 4 12B QAT (dense), TurboQ…
Gemma 4 12B QAT (dense) achieves over 1000 tokens per second prefill on an 8GB RTX 4060 with 120k context using TurboQuant, enabling full GPU layer offloading. This represents a 42% increase in prefill speed over previous methods.
Ran gemma 4 12b on my 3090 yesterday and I think the local model game just changed
A user reports running Google's Gemma 4 12B model locally on a single RTX 3090 via GGUF quantization, finding strong performance including real 256k context, multimodal capabilities, and function calling that outperforms larger 70B models for coding tasks.
@leopardracer: GEMMA 4 26B ON AN RTX 4060 WITH A 248K TOKEN CONTEXT WINDOW 20 tokens per second and a context window so large you can …
Gemma 4 26B runs on an RTX 4060 with 248K token context at 20 tokens per second using llama.cpp and Q4_K_XL quantization, enabling local processing of entire codebases on consumer hardware.
24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)
A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.