Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAM

Reddit r/LocalLLaMA Tools
llama.cpp local-inference qwen mixture-of-experts low-vram gguf-quantization multimodal

Summary

一位用户分享了在 RTX 2060 6GB 显存 + 32GB 内存的消费级硬件上运行 Qwen 3.6 35B A3B（Q4_K_M 量化）的 llama.cpp 完整启动配置，实现了 131K 上下文与视觉能力，约 600 tok/s 预填充与 23 tok/s 解码速度，通过将 MoE 专家层与视觉投影器卸载到 CPU 来控制显存占用。

Running HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM. Speeds start at ~600 tok/s prefill / 23 tok/s decode on an empty KV cache. As context grows they settle down - around 90k context it stabilizes at roughly 485 tok/s prefill and 15 tok/s decode, and holds there. The vision projector runs on CPU (--no-mmproj-offload), which keeps VRAM usage under ~5.2 GB and avoids OOM / GPU crashes. Image encoding is slower on CPU, but it buys ~1GB of VRAM. Most MoE expert layers also run on CPU (--n-cpu-moe 39), which is how a 35B model fits in 6GB VRAM in the first place. Launch command: bat @echo off cd /d "%~dp0" "%~dp0llama-server.exe" ^ -m "C:\Qwen3.6-35B-A3B-Uncensored-Q4_K_M\Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf" ^ --mmproj "C:\Qwen3.6-35B-A3B-Uncensored-Q4_K_M\mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf" ^ --no-mmproj-offload ^ -ngl 99 ^ --n-cpu-moe 39 ^ -c 131072 ^ -np 1 ^ -t 6 ^ -tb 10 ^ -b 2048 ^ -ub 2048 ^ -fa on ^ -ctk q8_0 ^ -ctv q8_0 ^ --load-mode mmap+mlock ^ --jinja ^ --reasoning-format deepseek ^ --reasoning-preserve ^ --spec-type none ^ --image-min-tokens 1024 ^ --temp 0.6 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0 ^ --alias Qwen3.6-35B-A3B-Uncensored-Q4_K_M ^ --host 127.0.0.1 ^ --port 8081 pause Hardware: RTX 2060 6GB + 32GB DDR4 RAM + i5-10400F CPU Context: 131072 tokens, Q8_0 KV cache Hope this helps someone. If anyone has tips to make the launch command even better, drop them in the comments - I'm out of ideas :D
Original Article

Similar Articles

Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context

Reddit r/LocalLLaMA

The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.

Qwen 30b MoE - 30tps - 6GB vram - Done!

Reddit r/LocalLLaMA

A user shares success running Qwen 30B MoE on an RTX 3050 6GB with 90k context at 30-35 tokens per second, celebrating low-end hardware performance.