Extened garlic to run Qwen3.5 35B A3B float8 at 55 tok/s on RTX 5060 Ti
Summary
A technique called 'Extened Garlic' enables running Qwen3.5 35B A3B model at 55 tok/s using float8 precision on an RTX 5060 Ti.
Similar Articles
Success running Qwen 3.8 27B EXL3 on RTX 3060 + 5060 Ti
A user successfully runs the Qwen 3.8 27B AI model on a mixed setup of RTX 3060 and 5060 Ti GPUs using tensor parallelism with exllamav3, achieving around 50 tokens per second with MTP enabled.
Qwen3.6:35b UD Q4_K_M 80 tok/s on Nvidia P40
A user shares achieving 80 tok/s on a Qwen3.6 35B model with Q4_K_M quantization and 100k context on a single Nvidia P40 using TheTom's TurboQuant fork of llama.cpp, highlighting various optimizations.
@ItsmeAjayKV: Achievement Unlocked: Running Qwen3.6-27b dense Thanks to the RTX 3090, now I can do this. Running @Alibaba_Qwen Qwen 3…
User benchmarks Qwen3.6-27B on an RTX 3090 using llama.cpp, achieving 35 tok/s generation and 1247 tok/s prompt processing.
EXPERIMENT: Qwen3.8-2.4T-A95B running locally on an RTX 5090 + RTX 5060 Ti at ~0.80 tok/s
An experiment running the Qwen3.8-2.4T-A95B MoE model locally on dual consumer GPUs (RTX 5090 + 5060 Ti) with llama.cpp, achieving ~0.8 tok/s with MTP speculative decoding enabled.
Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)
A developer showcases a custom `--moe-direct-io` feature for llama.cpp that achieves 10–15x speedup (20–24 tok/s) running the 125B MoE Qwen3.5-Flash Next (IQ2_XS, 68GB) on just an RTX 3060 12GB with 16GB RAM, while keeping output bit-exact to stock.