Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K

This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used. The short version: the full 262,144-token window fits together with vision, FP8 KV, prefix caching, tool calling, and a normal KDE desktop. Decode is 77.2 tok/s after a 1K prompt and 64.7 tok/s with 128K already resident. A successful 262,000-token prefill took 166 seconds. This is not a claim that 262K is fast; it is proof that it genuinely fits and completes. Model: joshebbs/qwen3.8-27b-uncensored-nvfp4-modelopt, pinned to revision e5ff4986938dcd0dd05ab4cce89da1b052be6ce3. It is an NVFP4 ModelOpt export of JonathanColetti/Qwen3.8-27B-Uncensored. The checkpoint is 19.18 GiB of safetensors and retains the vision tower and MTP head. The model is a 64-layer hybrid: 48 Gated DeltaNet layers and 16 full-attention layers. Results All runs hit the already-warm daily vLLM server through /v1/completions, concurrency 1, random token prompts, --ignore-eos, and temperature 0. PP means accepted input tokens divided by TTFT. TG means 1000 / mean_TPOT_ms, so it excludes prefill. The non-prefix runs had zero prefix-cache hits according to the server counters. Workload Runs PP tok/s TTFT Steady TG tok/s End-to-end output tok/s 8,192 in -> 1 out 5 7,005 1.169 s mean / 1.167 s median n/a n/a 32,768 in -> 1 out 3 6,148 5.330 s mean / 5.332 s median n/a n/a 131,072 in -> 256 out 1 2,781 47.128 s 64.7 5.01, because the 47 s prefill dominates 262,000 in -> 1 out 1 1,578 166.004 s n/a n/a 1,024 in -> 512 out 5 not used as a PP test 119.3 ms mean / 116.9 ms median 77.2 75.95 The short-context decode run had 12.959 ms mean TPOT and a measured peak of 78 output tok/s. At 128K resident context, TPOT rose to 15.463 ms, so generation fell by about 16.2% to 64.7 tok/s. The 128K and 262K rows are one run each. Treat those as measured operating points and fit checks, not a distribution. The 8K, 32K, and short-context decode rows are multi-run results. The drop in PP as context grows is substantial: this is a hybrid model, not a fully linear-attention model. Sixteen layers still use full attention. Prefix caching Fresh test with one shared 36,864-token prefix, a 16-token unique suffix, one output token, five sequential requests: Cold TTFT: 6.437 s Four cached TTFTs: 0.288, 0.282, 0.296, 0.288 s Cached median: 0.288 s Cold-to-cached speedup: 22.3x An older note in my launcher said 6.61 -> 0.20 s, or 33x. I could not reproduce the 0.20 s number in this fresh run, so 22.3x is the number I would use today. Prefix caching is still the difference between a usable long agent conversation and re-prefilling the entire transcript on every turn. Important caveat: vLLM puts the hybrid Mamba/DeltaNet cache in experimental align mode when prefix caching is enabled. If you see corrupted output, disabling prefix caching is the first control I would test. Hardware and software Part Exact measured setup GPU NVIDIA GeForce RTX 5090, 32,607 MiB reported VRAM, 600 W power limit CPU Intel Core i7-14700K, 20 cores / 28 threads RAM 32 GiB installed, 31 GiB visible OS Arch Linux, kernel 7.1.8-arch1-3 Desktop KDE/Wayland, with Firefox and terminals open during the VRAM snapshots NVIDIA driver 610.57.04 (nvidia-open / nvidia-utils 610.57.04) CUDA toolkit Arch cuda 13.3.1-1, nvcc 13.3.73 Python 3.13.13 vLLM 0.27.1, release wheel PyTorch 2.13.0+cu130 Transformers 5.15.0 FlashInfer 0.6.16.post3 Triton 3.7.1 compressed-tensors 0.17.0 The runtime auto-selected these paths from the startup log: modelopt_fp4 quantization FlashInfer CUTLASS NVFP4 GEMMs FlashInfer attention for the text model, flashinfer-native decode on SM120 Triton/FLA GDN prefill kernels Flash Attention for the vision encoder full and piecewise CUDA graphs; speculation is off Actual VRAM budget The important distinction is between model weight size, vLLM's process allocation, and the card-wide number from nvidia-smi. Item Measured Checkpoint safetensors on disk 19.18 GiB Model load reported by vLLM 18.51 GiB Manually pinned KV pool 9,150,000,000 bytes = 8.52 GiB GPU KV capacity reported by vLLM 268,170 tokens Maximum 262,144-token concurrency reported by vLLM 1.02x Live VLLM::EngineCore process 29,322 MiB Final whole-card snapshot 30,532 MiB used / 1,610 MiB free Across idle snapshots with the loaded server, free VRAM varied from 1,610 to 1,818 MiB as the desktop changed. This is real working margin, but it is not generous. I would not call this a headless-only fit: KDE, Firefox, and terminals were running, but a second large CUDA workload will obviously break it. --gpu-memory-utilization 0.92 is only a startup admission gate in this configuration. Because --kv-cache-memory-bytes 9150000000 pins the KV pool, vLLM explicitly says that the allocation does not respect gpu_memory_utilization. Lowering 0.92 does not shrink this KV pool or the context window; it merely lets the process start with a normal desktop consuming VRAM. --max-num-seqs 3 does not mean three simultaneous 262K requests. The KV pool only has 1.02x full-window capacity. The three slots help with shorter real requests that share the same pool. Exact install and model revision I already had a working Arch NVIDIA driver and /opt/cuda. This creates the Python environment used above and pins the CUDA 13.0 vLLM/PyTorch wheel family: uv venv --python 3.13 qwen38-env uv pip install --python qwen38-env/bin/python 'vllm==0.27.1' --torch-backend=cu130 ./qwen38-env/bin/hf download \ joshebbs/qwen3.8-27b-uncensored-nvfp4-modelopt \ --revision e5ff4986938dcd0dd05ab4cce89da1b052be6ce3 \ --local-dir Qwen3.8-27B-Uncensored-NVFP4-modelopt Weight hashes: 5db0ff93ebdf68034770a6acec123971e618928684bd2d5f3f51346990254911 model.safetensors 90fa0e3eed5a647c035c6df9ecabc416c0f8d573ff84ac12485b085f00a7cdf2 model-mtp-grafted.safetensors Do not delete model-mtp-grafted.safetensors from this revision just because speculation is off; the checkpoint index contains 15 tensors mapped to it. With speculation off, vLLM skips using the MTP head at runtime, but keeping the downloaded revision intact avoids an incomplete checkpoint. Chat template used in my daily setup The speed tests use the raw completions endpoint, so the chat template does not affect these numbers. It does affect my daily chat/tool behavior. My exact template is froggeric/Qwen-Fixed-Chat-Templates v22.2 at revision f64494d7b8a768222ab799d8c81f6e89dd272ac3, plus a small system-prompt terseness block. The upstream repo has since moved on, so pin the revision: mkdir -p chat-templates/froggeric-fixed chat-templates/sharp-v22.2 ./qwen38-env/bin/hf download froggeric/Qwen-Fixed-Chat-Templates \ chat_template.jinja \ --revision f64494d7b8a768222ab799d8c81f6e89dd272ac3 \ --local-dir chat-templates/froggeric-fixed cp chat-templates/froggeric-fixed/chat_template.jinja \ chat-templates/sharp-v22.2/chat_template.jinja I changed the first-line version string to qwen3.8-froggeric-v22.2-sharp, then inserted this immediately after {%- set _msgs = messages[head.count:] %}: {%- set _terse %} Answer directly, after thinking. Lead with the answer, then only what it needs to be correct and usable. Never: open with preamble or pleasantries; restate the question; add filler transitions; hedge with niceties; or repeat a point you've already made. Always: keep essential steps, caveats, uncertainties, and specifics — never drop correctness or a needed warning for brevity. Keep the final answer lean. Use the least structure that conveys it (plain prose when short; lists or code only when they earn their place). If genuinely uncertain, say so and explain why — never omit uncertainty for the sake of brevity. If a user request is genuinely ambiguous, ask a sharp question, don't guess. {%- endset %} {%- if not _sc %} {%- set _sc = _terse | trim %} {%- else %} {%- set _sc = (_sc | trim) ~ '\n\n' ~ (_terse | trim) %} {%- endif %} Checksums: 55d027bfded4407d214e5718e2f2804de73e8439148297397655ff507a30f2a5 froggeric v22.2 base 34dd122439410ee026562207c87124cb843ad92069568a9ab2f02e6d20626fea my sharp-v22.2 result If you do not care about my style block, use the pinned Froggeric v22.2 file directly. The model's stock template also works with TEMPLATE=stock, but it is not my exact daily setup. Exact effective server command This is the effective command from the live process. Paths can be changed, but the values are the ones used for the results above. export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True export MAX_JOBS=3 export TORCHINDUCTOR_COMPILE_THREADS=3 export CUDA_HOME=/opt/cuda export PATH="$CUDA_HOME/bin:$PATH" systemd-run --user --scope \ -p MemoryMax=24G \ -p MemorySwapMax=8G \ ./qwen38-env/bin/vllm serve \ ./Qwen3.8-27B-Uncensored-NVFP4-modelopt \ --host 0.0.0.0 \ --port 8888 \ --served-model-name qwen3.8-27b \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --kv-cache-memory-bytes 9150000000 \ --max-num-seqs 3 \ --max-num-batched-tokens 512 \ --gpu-memory-utilization 0.92 \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 \ --mm-processor-kwargs '{"max_pixels": 4014080}' \ --chat-template ./chat-templates/sharp-v22.2/chat_template.jinja I leave quantization and attention backend on auto. vLLM correctly resolves modelopt_fp4 and FlashInfer here. Adding explicit flags is fine for testing, but it would no longer be the exact command behind these numbers. Security note: --host 0.0.0.0 exposes an unauthenticated OpenAI-compatible endpoint to every reachable interface. That is intentional on my trusted LAN. Use 127.0.0.1, a firewall, or vLLM's API-key option if your network is not trusted. Why the less obvious values: 9150000000 KV bytes was chosen because 8,939,000,000 bytes was 288 tokens short: vLLM estimated a 261,856-token maximum and refused a 262,144 max length. The current value reports 268,170 tokens of capacity. --max-num-batched-tokens 512 protects the small activation margin. Raising it improves prefill but costs VRAM. --mm-processor-kwargs '{"max_pixels": 4014080}' caps a large image at about 3,908 tokens in my measurements. Make sure tokenizer.json has "truncation": null; an old export with a baked-in max_length: 2048 made larger images fail with HTTP 400. /opt/cuda/bin/nvcc must be visible. FlashInfer JIT failed here when CUDA was installed in /opt/cuda but CUDA_HOME was unset. Speculative decoding: why it is off The default is deliberately SPEC=off. The external DSpark/dflash drafter trained for the base checkpoint did not load against this abliterated ModelOpt export. The loader failed with a tensor dimension mismatch (128 versus 256). The built-in MTP head does load, but it made this checkpoint slower. My earlier controlled results were: Mode Decode Speculation off 78.55 tok/s Built-in MTP, 1 token 58.41 tok/s, 78.7% acceptance Built-in MTP, 3 tokens 45.02 tok/s, 57.9% acceptance The grafted MTP head is BF16 and does not get the target model's NVFP4 kernels. MTP also needs a larger KV pool: at least about 9.75e9 bytes for one speculative token or 1.005e10 for three. Three-token MTP with --max-num-batched-tokens 8192 OOMed at 262K on this card. I prefer the faster, simpler non-speculative setup. Accuracy/quality caveat This post measures fit and serving speed, not model quality. vLLM warns that this checkpoint does not provide calibrated FP8 KV q/prob scaling and uses scale 1.0, which may affect accuracy. I have not quantified that accuracy delta here. Prefix caching on the hybrid cache is also marked experimental by vLLM. Those two caveats matter more than another decimal place of tok/s. Reproduce the benchmark With the server already running: V=./qwen38-env/bin/vllm TOK=./Qwen3.8-27B-Uncensored-NVFP4-modelopt COMMON=(bench serve --backend openai --base-url http://127.0.0.1:8888 \ --endpoint /v1/completions --model qwen3.8-27b --tokenizer "$TOK" \ --dataset-name random --max-concurrency 1 --ignore-eos --temperature 0 \ --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99) "$V" "${COMMON[@]}" --seed 8182 --random-input-len 8192 \ --random-output-len 1 --num-prompts 5 "$V" "${COMMON[@]}" --seed 8184 --random-input-len 32768 \ --random-output-len 1 --num-prompts 3 "$V" "${COMMON[@]}" --seed 8183 --random-input-len 1024 \ --random-output-len 512 --num-prompts 5 "$V" "${COMMON[@]}" --seed 8186 --random-input-len 131072 \ --random-output-len 256 --num-prompts 1 "$V" "${COMMON[@]}" --seed 8185 --random-input-len 262000 \ --random-output-len 1 --num-prompts 1 For the shared-prefix test, replace the random dataset arguments with: ./qwen38-env/bin/vllm bench serve \ --backend openai --base-url http://127.0.0.1:8888 \ --endpoint /v1/completions --model qwen3.8-27b \ --tokenizer ./Qwen3.8-27B-Uncensored-NVFP4-modelopt \ --dataset-name prefix_repetition \ --prefix-repetition-prefix-len 36864 \ --prefix-repetition-suffix-len 16 \ --prefix-repetition-num-prefixes 1 \ --prefix-repetition-output-len 1 \ --num-prompts 5 --max-concurrency 1 \ --ignore-eos --seed 8187 --temperature 0 --save-detailed If another 5090 owner reproduces this, I would especially like to compare 128K PP/TG, free VRAM with a headless session versus a desktop, and whether FP8-KV quality changes are noticeable in long-context retrieval. AI-use disclosure: the machine, launcher decisions, failed experiments, and measurements are mine. I used Codex to inspect the live process, run the stated local benchmarks, verify the saved JSON/telemetry, and organize this write-up. I reviewed the numerical claims against the raw outputs; no benchmark numbers were generated or estimated by an LLM.