A discussion thread sharing configurations and tips for running AI models like Qwen3.8-27B on 16 GB VRAM Windows systems, focusing on memory optimization techniques.
What models and configs are we using? Please share here On windows, I am using this copium pared down model https://huggingface.co/Bucoid/Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF with MTP disabled, q4 k/q4 v mmproj banished to CPU/RAM and a small ub to save whatever context I can (90k-100k) so everything stays in the vram If you are on linux or have an iGPU, you don't have to deal with windows eating 1.5 gb vram and so have more than 14.5 GB of VRAM to use and probably aren't in purgatory. @echo off .\ikllama\llama-server.exe ^ -m "D:\AI models\qwen3.8\Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF.gguf" ^ :: gpu offload all layers (99 is more than the max which means it will offload everything) -ngl 99 ^ :: this depends on your cpu -t 8 ^ :: literally can't fit in anything at higher q to save vram --cache-type-k q4_0 ^ --cache-type-v q4_0 ^ :: check your max context size with fit, at around >100k context rot sets in -c 100100 ^ :: this flag should always be on to optimise speed and memory use -fa on ^ :: You need a fixed jinja file to prevent it rambling forever, I believe this deefaults to xhigh --chat-template-file chat-template.jinja ^ --chat-template-kwargs "{\"preserve_thinking\": true, \"enable_thinking\": true}" ^ :: you ain't getting more than this -np 1 ^ :: use the mmproj and banish it to CPU/RAM land to save vram (probably ~800-900 mb vram saving) --mmproj mmproj-F16.gguf ^ --no-mmproj-offload ^ :: we need reasoning --reasoning on ^ :: the image mmproj needs this line --image-min-tokens 1024 ^ --metrics ^ --port 8080 ^ :: reduce vram spikes saving some vram --batch-size 1024 ^ --ubatch-size 256 ^ :: allows more caching in RAM. According to Claude it's mostly for your context slot checkpoints that there is literally no room for --cache-ram 24576 ^ --ctx-checkpoints 32 ^ :: Delta net architecture apparently has a bug where it just stalls forever saving and shifting contexts this is apparently supposed to help with this according to Cl*ude --no-context-shift ^ :: force mtp header into the CPU/RAM (cl*ude estimates ~200 mb savings) --override-tensor nextn=CPU ^ --jinja pause
The user discusses their experience with Qwen 3.6 27B for local coding tasks and asks for recommendations for larger models (100B+) suitable for systems with 224GB of VRAM.
Discussion about running LLMs on 12GB VRAM, noting current focus on dense models like Muse Glimmer 30B and Qwen 3.8 27B, and questioning whether upgrading to 24GB VRAM is needed.
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.
A user benchmarks Qwen 35B-A3B (a 35B MoE model) on a 12GB RTX 3060, finding that 12GB VRAM is a practical sweet spot for running the model with 32k context, achieving ~47 t/s generation.
The article shares an optimal llama.cpp configuration for running the Qwen 3.8 27B model on 16GB VRAM with 73k context, demonstrating its performance in agentic coding workflows through a real-world software engineering project.