The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
If anyone is looking for a good high-speed setup with \~190k context, this config has been working insanely well for me. I’m using my laptop as a server over Tailscale. Installed Linux on it and running: \- Qwen3.6 35B A3B \- RTX 4060 8GB VRAM \- 32GB DDR5 5600MHz RAM \- Q5 quant models Current models tested: \- \`mudler/Qwen3.6-35B-A3B-APEX-GGUF\` \- \~40 tok/sec → 37 tok/sec \- \`hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-GGUF\` \- \~43 tok/sec → 37 tok/sec I can push it up to \~51 tok/sec by tweaking: \- \`--ctx-size 192640\` \- \`--n-gpu-layers 430\` \- \`--n-cpu-moe 35\` and adjusting those values slightly higher/lower depending on stability and memory usage. Here’s my current config: \#!/bin/bash \# --- LLAMA SERVER LAUNCHER SCRIPT --- \#SELECTED\_MODEL="/home/atulloq/.lmstudio/models/hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled.Q5\_K\_M.gguf" SELECTED\_MODEL="/home/atulloq/.lmstudio/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Balanced.gguf" echo "Starting Llama Server..." echo "Model: $SELECTED\_MODEL" /home/atulloq/llama-cpp-turboquant/build/bin/llama-server \\ \--model "$SELECTED\_MODEL" \\ \--host [0.0.0.0](http://0.0.0.0) \\ \--port 8085 \\ \--ctx-size 192640 \\ \--n-gpu-layers 430 \\ \--n-cpu-moe 35 \\ \--cache-type-k "turbo4" \\ \--cache-type-v "turbo4" \\ \--flash-attn on \\ \--batch-size 2048 \\ \--parallel 1 \\ \--no-mmap \\ \--mlock \\ \--ubatch-size 512 \\ \--threads 6 \\ \--cont-batching \\ \--timeout 300 \\ \--temp 0.2 \\ \--top-p 0.95 \\ \--min-p 0.05 \\ \--top-k 20 \\ \--metrics \\ \--chat-template-kwargs '{"preserve\_thinking": true}' I’m using this fork of llama.cpp with TurboQuant support: [https://github.com/TheTom/turboquant\_plus#build-llamacpp-with-turboquant](https://github.com/TheTom/turboquant_plus#build-llamacpp-with-turboquant) A few honest notes: \- Q4 is noticeably worse for long-context reasoning compared to Q5 on these models. \- \`--no-mmap\` + \`--mlock\` helped reduce weird slowdowns for me. \- TurboQuant KV cache makes a massive difference at high context sizes. \- Linux performs way better than Windows for this setup. \- Don’t expect these speeds if your RAM bandwidth is bad. DDR5 matters here. If anyone has optimizations for: \- better long-context stability, \- higher token throughput, \- or smarter \`n-cpu-moe\` tuning, I’d love to test them.
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.
A user compares running quantized Qwen3 Next 80B and Qwen3.5 122B on a 64GB RAM system, noting the trade-offs in speed, quality, and memory usage for local LLM inference.
The author shares detailed tuning tips for running the Qwen3.6-35B-A3B MoE model on an 8GB RTX 3070 Ti with up to 262k context using llama.cpp, achieving 30+ tps, and notes a 25% speed boost when switching from Windows to Ubuntu Server.
This article describes how to use the SYCL backend with llama.cpp to achieve over 60 tokens per second on the Qwen 3.6-35B-A3B model using an Intel Arc Pro B70 GPU, with the entire model and KV cache in VRAM.
The article compares llama.cpp backends for running Qwen 3.6 27B on an RTX 3090 24GB, finding ik_llama.cpp with IQ4_KS quantization yields the best performance (1261 tok/s prefill, 72.9 tok/s decode).