Two days tuning a single Tesla P40: 35B MoE, full 262k context, ~60 t/s short and ~30 t/s at 100k deep. No sub-Q4 quants.

Reddit r/LocalLLaMA Tools
llama.cpp inference-optimization tesla-p40 mixture-of-experts mtp-draft quantization hardware-hacking

Summary

A hobbyist details two days of tuning a single Tesla P40 24GB (PCIe x1) to run the 35B-A3B Ornith 1.5 MoE model at full 262k context, achieving ~60 t/s short-prompt and ~30 t/s at 100k depth without dropping below Q4 weights or q8 KV, via ECC disabling, int8 dp4a attention, and a trained MTP draft head check.

i Got a bit obsessed with my P40 this week trying to max speed on Ornith 1.5 35B-A3B (qwen3.5 MoE arch) without going below Q4 weights or q8 KV. Shrinking quants to win a benchmark has not really been useful to me so ai wanted a logical middle ground that was still “fast” Setup: one Tesla P40 24GB on PCIe x1 (yes, x1), Windows, my own llama.cpp fork. Q4_0 built from the official BF16, MTP draft (n=2), full 262k context, q8_0 KV. What actually moved it: - ECC off +1.5 GB VRAM and ~9% decode. - q8_0 KV, not q4_0. On Pascal q4 KV falls apart with depth (30.6 vs 42.2 t/s at 32k). - Check your MTP head. The first upload of this model, and quants built from it, have an untrained MTP head. `blk.40.nextn.shared_head_norm` mean ~1.02 = untrained, ~2.93 = trained. Draft acceptance went from ~45% to 60-85%. - int8 dp4a for the KQ half of the flash-attention tile kernel. Pascal has no tensor cores and crippled fp16, but dp4a is fast. Prefill +26% at 32k, +41% at 100k. - MTP prefill window. The MTP head is a full-attention layer, so on long prompts it costs ~13% prefill. Letting it read only the last 8k of the prompt gets that back (121k prompt: 327 -> 373 t/s). Output doesn't change, the main model still verifies every draft. Didn't work: f16 KV (faster in llama-bench, but slower and less context once MTP is on in a real server), dequantizing V inside the kernel (-5%), the official Q4_K_M (same quality score as my Q4_0, slower, less context). llama-bench, no MTP, 2 runs each: depth prefill t/s decode t/s 0 1104 60.8 8k 880 57.0 32k 552 46.0 64k 366 35.8 100k 265 29.1 190k - 18.8 Real server with MTP: 65-74 t/s on short prompts. A 101k-token prompt reads at ~415 t/s and generates at 35-39 t/s. I couldn't find anyone else posting P40 numbers at depth, so hopefully the curve helps someone. Happy to share the patches if you want to try them on your own Pascal card.
Original Article

Similar Articles

Qwen3.6:35b UD Q4_K_M 80 tok/s on Nvidia P40

Reddit r/LocalLLaMA

A user shares achieving 80 tok/s on a Qwen3.6 35B model with Q4_K_M quantization and 100k context on a single Nvidia P40 using TheTom's TurboQuant fork of llama.cpp, highlighting various optimizations.

24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)

Reddit r/LocalLLaMA

A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.

Qwen3.6-35B-A3B Q4 262k context on 8GB 3070 Ti = +30tps

Reddit r/LocalLLaMA

The author shares detailed tuning tips for running the Qwen3.6-35B-A3B MoE model on an 8GB RTX 3070 Ti with up to 262k context using llama.cpp, achieving 30+ tps, and notes a 25% speed boost when switching from Windows to Ubuntu Server.