If you are wondering why Ornith 1.5 35B A3B with MTP is so slow, this is why
Summary
The Ornith 1.5 35B A3B model's MTP tensors appear to be uninitialized, causing poor speculative decoding performance, and grafting the trained head from Qwen3.6-35B-A3B improves speed by 29%.
View Cached Full Text
Cached at: 08/20/26, 08:56 PM
ornith-ai/Ornith-1.5-35B-A3B · mtp.* tensors look like random init, not trained weights
Source: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B/discussions/10 First off, 1.5 is seriously impressive work. The 35B-A3B numbers on Terminal-Bench and SWE-bench for 3B active params are remarkable, and it’s been great to run locally. Thank you for releasing it under MIT.
While poking at the weights with some remote tensor probes I think I found a small packaging issue: the mtp.* tensors don’t look trained. Every projection (q/k/v/o, fc, the experts, shared_expert) has std of exactly 0.0200 with clean Gaussian stats, which is just initializer_range=0.02. The norm weights sit around 0.02 where you’d expect ~1. For comparison, Qwen3.6-35B-A3B’s MTP head (same shapes) has fc kurtosis in the hundreds and norms between 0.87 and 1.93.
It shows up in practice too. Used as a speculative draft in llama.cpp (b10507, Q4_K_M main + Q8_0 draft on a 3090) the shipped head accepts about 13% of drafted tokens, mean run 1.5, which is basically chance.
The nice part: since the 1.5 tower is only ~0.2-1% away from Qwen3.6 in cosine, grafting Qwen3.6’s trained head across just works. Acceptance goes to 55-75% (mean run 3.2-4.0) and generation goes from 131 to 169 tok/s on my box, same outputs on a small pass@1 set. So everything around the head slot is healthy, and a free +29% is sitting there for anyone who wants it.
Merged model demonstrating it, numbers in the card:https://huggingface.co/EryriLabs/Ornith-1.5-35B-A3B-BigBang-MTP
Happy to share the probe scripts if useful. Was the untrained head reserved for a later drop, or did a trained one just miss the upload? Either way, thanks again for the model.
Similar Articles
Ornith-1.0-35B GGUF update: native MTP speculative-decode graft + full serving/TTFT/long-context numbers (llama.cpp, tp=1)
An update on the Ornith-1.0-35B GGUF model introduces a native MTP speculative-decode graft for faster inference on a single GPU, achieving ~1.3-1.35x decode speedup while maintaining near-identical token distribution. Benchmark numbers for throughput, TTFT, and long-context performance across multiple quants are provided.
I added MTP to local SoTA Agentic Coding Model Ornith 35B FP8 E4M3
Introduced MTP speculative decoding to the Ornith 35B coding model in FP8 precision, achieving approximately 18% faster inference with minimal extra VRAM.
@malikwas1f: Ornith-1.0-35B: a Qwen3.6-35B-A3B coding fine-tune that edges the base on real coding (aider 15/30 vs 13) — full 262K a…
Announces Ornith-1.0-35B, a coding fine-tune of Qwen3.6-35B-A3B that slightly outperforms the base model on aider benchmarks. Also promotes the club-3090 repository for running LLMs on RTX 3090s.
Ornith-1.0-35B Q3_K_M: ~17 GB VRAM, KLD-checked against BF16
Ornith-1.0-35B Q3_K_M is a 3-bit quantized version of a 35B parameter model, requiring about 17 GB VRAM, with KLD checking against BF16 to ensure fidelity.
Ornith 35B works reasonably well with Qwen3.6 35B DFlash speculative model
Ornith 35B shows 30-40% token generation speedup when paired with Qwen3.6 35B DFlash speculative model in llama-server, achieving 80% acceptance rate on mixed code and text, though prompt processing suffers.