An update on the Ornith-1.0-35B GGUF model introduces a native MTP speculative-decode graft for faster inference on a single GPU, achieving ~1.3-1.35x decode speedup while maintaining near-identical token distribution. Benchmark numbers for throughput, TTFT, and long-context performance across multiple quants are provided.
Follow-up to my previous Ornith-1.0-35B Q3_K_M post. I grafted a native MTP draft head onto the IQ4_XS body (head at Q6) for self-speculative decode, single GPU, llama.cpp: 1.3-1.35x single-stream decode (172.6 -> 233.8 tok/s). Next-token distribution is byte-identical to target-only (KLD 0.0, 32/32). BF16 KLD 0.073 — slightly better than Q4_K_M. Issue: not bit-exact to target-only over long deterministic gens (6/8 exact, 93.4% token match). Where it sits on the KLD ladder (top-64 next-token KL vs BF16, lower is better): Quant Mean KLD Top-1 Size Q8_0 0.011 96.9% 36.9 GB Q6_K 0.017 100.0% 28.5 GB Q5_K_M 0.035 93.8% 24.7 GB IQ4_XS-MTP graft (new) 0.073 90.6% ~19.6 GB Q4_K_M 0.086 90.6% 21.2 GB IQ4_XS 0.143 84.4% 18.9 GB Q3_K_M 0.362 84.4% 16.8 GB Fidelity ladder chart Performance numbers I added to the card: Throughput + p95 TTFT vs concurrency for all six quants (Q4_K_M ~243 tok/s @c1 -> ~656 tok/s @c16, p95 TTFT ~76 ms @c1). Long-context TTFT, single stream: prefill scales 94 ms @512 tokens -> ~6.3 s @32k (the IQ4_XS body and the graft prefill a bit faster than Q4_K_M at every length). Notes: Q4/Q5/Q6/Q8 are upstream artifacts I mirrored + revalidated; Q3_K_M, IQ4_XS, and the MTP graft are produced locally. REASONING=off is still the pinned serving default (the reasoning-mode bug from last post). Single workstation GPU (RTX PRO 6000 Blackwell 96 GB), tp=1 only. 🔗 https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 https://preview.redd.it/4kljd5aci2ah1.png?width=1800&format=png&auto=webp&s=f71b72f3fd40f3c64004c1910eb97304c98dcbc6 https://preview.redd.it/i7nro4aci2ah1.png?width=1800&format=png&auto=webp&s=65fef9870e76c5920799c884b181dc1d423bc995 https://preview.redd.it/5sdod4aci2ah1.png?width=1800&format=png&auto=webp&s=72f775e164cfa056172d705e7ff6f33e720d1380 https://preview.redd.it/cl2dw4aci2ah1.png?width=1800&format=png&auto=webp&s=690a525335066ff297666f3f6b0502a65db9c9bf https://preview.redd.it/270cq3aci2ah1.png?width=1680&format=png&auto=webp&s=ea5944912b2f876d1daf9f36ac42fbd5ca369e68 https://preview.redd.it/0tgp54aci2ah1.png?width=2200&format=png&auto=webp&s=e2487187d455833ba41516cf0f93560c3c68a20b https://preview.redd.it/2nuao3aci2ah1.png?width=1192&format=png&auto=webp&s=76f8b368e1c3e2b990c0545d0ba6e3c0e04f49bd https://preview.redd.it/o1u7n3aci2ah1.png?width=1192&format=png&auto=webp&s=14354bf5001b38159a56752c367a84da5bd47a63
The Ornith 1.5 35B A3B model's MTP tensors appear to be uninitialized, causing poor speculative decoding performance, and grafting the trained head from Qwen3.6-35B-A3B improves speed by 29%.
Announces Ornith-1.0-35B, a coding fine-tune of Qwen3.6-35B-A3B that slightly outperforms the base model on aider benchmarks. Also promotes the club-3090 repository for running LLMs on RTX 3090s.
Ornith-1.0-35B Q3_K_M is a 3-bit quantized version of a 35B parameter model, requiring about 17 GB VRAM, with KLD checking against BF16 to ensure fidelity.
Introduced MTP speculative decoding to the Ornith 35B coding model in FP8 precision, achieving approximately 18% faster inference with minimal extra VRAM.
A user asks why Multi-Token Prediction (MTP) speculative decoding in llama.cpp yields lower throughput (~15 tok/s) than standard decode (~30 tok/s) for Ornith 1.5 9B, despite a high 0.95 draft acceptance rate, noting halved memory bandwidth and GPU utilization.