@DogukanUrker: Ornith-1.5-9B at Q5 on a single RTX 3060: 200k context at ~52 tok/s, ~1700 tok/s prefill. 11.8 of the 12GB, zero cpu of…
Summary
The post details running the Ornith-1.5-9B AI model on an RTX 3060 with 200k context, achieving high inference speeds using advanced quantization techniques.
View Cached Full Text
Cached at: 08/21/26, 01:09 PM
Ornith-1.5-9B at Q5 on a single RTX 3060: 200k context at ~52 tok/s, ~1700 tok/s prefill. 11.8 of the 12GB, zero cpu offload. (config below)
32 layers, only 8 of them carry a kv cache, the other 24 hold a fixed recurrent state. that’s why 200k sits next to the weights at f16 – no quantized cache needed here.
quant is @atomic_chat_hq’s AD-Q5_K-Q4_K at 5.93gb: smaller than stock Q5_K_M and lower kl divergence at the same time, 0.0255 vs 0.0299.
@ornith_’s 35B is the better model – if you have 16gb ram and you’re fine with moe offload, run that one instead. this is for people who want everything resident on the card and their system ram untouched. (35B config in the replies)
llama-server -m Ornith-1.5-9B-AD-Q5_K-Q4_K.gguf -ngl 99 -c 200000 -fa on –jinja -np 1 –temp 0.6 –top-p 0.95 –top-k 20 –min-p 0.0 –presence-penalty 0.0 –repeat-penalty 1.0 –reasoning on –reasoning-format none
Similar Articles
Ornith-1.5-35B-A3B Q4 running 60tk/s on 4070Ti.
A user reports achieving 60 tokens per second with the Ornith-1.5-35B-A3B MOE model on an NVIDIA RTX 4070 Ti, demonstrating efficient local inference optimization.
Ornith-397B running at Q4 on a single RTX PRO 6000 Blackwell 96GB - 2,354 tok/s prefill, ~20–24 tok/s decode
Krasis, a MoE-focused runtime, enables running the 397B-parameter Ornith model on a single RTX PRO 6000 Blackwell 96GB GPU with ~20-24 tok/s decode by dynamically managing expert residency in VRAM.
@ItsmeAjayKV: Quick update: I tried Ornith-1.0-35B-Q5_K_M on my 3090, and i have mixed feelings. The good: it's really fast. I measur…
User tests Ornith-1.0-35B on an RTX 3090, finding fast inference speeds (1560 tok/s prompt, 78 tok/s generation) but consistently worse coding performance on Three.js tasks compared to Qwen 3.6, even after multiple attempts.
Local benchmarks with a RTX 3090 - Qwen3.6 27b vs Ornith
User runs local benchmarks comparing Qwen3.6 27b, Gemma4 26B, and Ornith1.0 35B on an RTX 3090 using inspect-ai. Results show Qwen leading in knowledge and coding, while Ornith is competitive in grounding and recall.
@DogukanUrker: Gemma 4 12B on a single RTX 3060: the full 262,144 context at ~100 tok/s. (config below) dense model -> MTP speculative…
DogukanUrker demonstrates running Gemma 4 12B with full 262,144 context at ~100 tok/s on a single RTX 3060 using speculative decoding and KV cache splitting, achieving nearly full GPU utilization without CPU offload.