1M context with 17 GB model in 24 GB VRAM: "for the first time I was able to load a context of almost 1M tokens and extract 7 needles from various parts of the text"
Summary
A user reports successfully running a 1M-token context on a single RTX 3090 using a Qwen-based 35B A3B model (17GB VRAM) with KVarN 4-bit KV-cache quantization in a BeeLlama.cpp fork, extracting 7 needles from different parts of the text.
Similar Articles
Getting close to 100K context on 32GB VRAM with Qwen3.6-27 at Q8
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.
500k context on 48gb VRAM!! - 21tok/s (coding)
A user reports successful deployment of a quantized Nemotron-3 Super model supporting 500k context and agentic coding on consumer-grade dual Titan RTX hardware.
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K
This article demonstrates the practical setup and performance benchmarks for running the Qwen3.8-27B model on a single NVIDIA RTX 5090 with a 262K token context window using vLLM, including detailed metrics and configuration steps.
Qwen 3.6-35B-A3B with 977 tk/s prompt processing and 262k context window on Intel Arc B70 Pro
This article describes how to use the SYCL backend with llama.cpp to achieve over 60 tokens per second on the Qwen 3.6-35B-A3B model using an Intel Arc Pro B70 GPU, with the entire model and KV cache in VRAM.