@ciruai: Finally 256k context for 16GB cards on smart models at high speeds! Using a 4080 Super 16GB I show you how to get full …
Summary
Demonstrates achieving 256k context on a 16GB RTX 4080 Super using Ternary Bonsai 27B Q2_0 model with llama.cpp, achieving up to 141 tok/s generation speed.
View Cached Full Text
Cached at: 07/16/26, 04:20 PM
Finally 256k context for 16GB cards on smart models at high speeds! Using a 4080 Super 16GB I show you how to get full 256k context OR up to 141 tok/s depending on your needs. Configs at : https://llm.ciru.ai/research/4080-27b.html… Model: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf…
RTX 4080 SUPER + Ternary Bonsai 27B Benchmarks
Source: https://llm.ciru.ai/research/4080-27b.html HeadlinePlatformTarget-onlyDSparkConfiguration## Benchmark headline
262,144Maximum validated target-only context with KV4. 817.36Prompt tokens/s at the full 262,016-token pass. 57.71Generation tokens/s at the full-window pass. 141.27Peak decoded DSpark generation tokens/s.
GPU, model, and runtime
GPUNVIDIA GeForce RTX 4080 SUPER16,376 MiB VRAM · compute capability 8.9Target modelTernary Bonsai 27B Q2_0Ternary-Bonsai-27B-Q2_0.gguf · 6.66 GiBDSpark drafterTernary Bonsai 27B DSpark Q4_1Ternary-Bonsai-27B-dspark-Q4_1.gguf · 1.81 GiBRuntimePrismML llama.cpp · Windows CUDA 12.4Build b1-62061f9 · commit 62061f9
Target-only throughput
The KV4 lane validates long-context operation. The F16 lane captures the high-throughput context ladder using the standard llama-bench profile.
KV4 full-context scaling
Prompt / contextPP tok/sTG tok/sPeak VRAMMinimum free16,3841,963.3166.189,158 MiB6,888 MiB32,7681,793.5464.939,499 MiB6,547 MiB65,5361,588.4666.6510,030 MiB6,016 MiB131,0721,229.4966.9311,478 MiB4,568 MiB262,144817.3657.7114,329 MiB1,717 MiB
KV4 VRAM scaling
F16 context ladder
Prompt tokensPP tok/sTG tok/sPeak VRAMMinimum free5122,157.87 ± 35.9468.22 ± 0.088,543 MiB7,503 MiB4,0962,111.83 ± 4.2868.04 ± 0.178,758 MiB7,288 MiB8,1921,925.5261.148,972 MiB7,074 MiB16,3841,851.6662.739,474 MiB6,572 MiB32,7681,673.0559.8710,633 MiB5,413 MiB65,5361,609.6568.3112,729 MiB3,317 MiB98,3041,407.0466.2014,783 MiB1,263 MiB110,5921,339.0368.1015,576 MiB470 MiB
**One card. 27 billion parameters.**Target-only KV4 and GPU-resident DSpark profiles.
DSpark speculative throughput
Decoded generation speed and draft acceptance for the successful GPU-resident DSpark profiles.
Workload throughput
GPU-resident KV4 context
ContextWorkloadDecoded TGDraftedAcceptedAcceptancePeak VRAMMinimum free1,024Code · temp 0.7117.907 t/s1449465.278%13,165 MiB2,881 MiB2,048Code · temp 0.7113.789 t/s1449465.278%14,089 MiB1,957 MiB2,048Code · greedy123.237 t/s1329773.485%14,346 MiB1,700 MiB2,048Arithmetic · temp 0.7141.272 t/s11210190.179%14,374 MiB1,672 MiB GPU DSpark KV4 contextPP tok/sTG tok/sAcceptanceFree after request16,3841,185.8498.1157.89% · 88/1521,806 MiB20,4801,056.7097.5257.89% · 88/152871 MiB22,5281,274.5095.8957.89% · 88/152472 MiB
Validated configuration
Target-only · F16 throughput### Standard llama-bench profile
-ngl 99 -fa on
-ctk f16 -ctv f16
-t 8 -b 512 -ub 512
Target-only · maximum context### 262K KV4 profile
-ngl 99 -fa on
-ctk q4_0 -ctv q4_0
-c 262144 -np 1
-b 512 -ub 512
Target-only · balanced headroom### 131K KV4 profile
-ngl 99 -fa on
-ctk q4_0 -ctv q4_0
-c 131072 -np 1
-b 512 -ub 512
DSpark · decoded speed### GPU-resident drafter
--spec-type draft-dspark
--spec-draft-n-max 4
-ngl 99 -ngld 99
-c 1024
Sampler: temperature 0.7 · top-p 0.95 · top-k 20 · seed 42.
DSpark · practical API context### 20,480-token GPU KV4 profile
--spec-type draft-dspark
--spec-draft-n-max 4
-c 20480 -np 1
-b 128 -ub 128
-ngl 99 -ngld 99
-ctk q4_0 -ctv q4_0
-ctkd q4_0 -ctvd q4_0
Model files### Target + paired drafter
Target:
Ternary-Bonsai-27B-Q2_0.gguf
Drafter:
Ternary-Bonsai-27B-dspark-Q4_1.gguf
Similar Articles
Getting close to 100K context on 32GB VRAM with Qwen3.6-27 at Q8
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.
500k context on 48gb VRAM!! - 21tok/s (coding)
A user reports successful deployment of a quantized Nemotron-3 Super model supporting 500k context and agentic coding on consumer-grade dual Titan RTX hardware.
7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context
A guide on optimizing VRAM usage on an AMD 7900XTX to run a 27B Qwen model with Q6K quantization and 131k context by compiling llama.cpp with OpenBLAS and CUDA_FA_ALL_QUANTS, and using kvcache quantization at q5_0/q4_0.
@sudoingX: anyone running a 16gb card, stop scrolling. @pupposandro and @davideciffa got qwen 35b-a3b down to 13.3gb, measured on …
A technique called luce spark allows Qwen 35B-a3B MoE model to run on a 16GB GPU (like RTX 3090) by learning which experts are frequently used and streaming the rest from RAM, achieving ~100 tok/s without VRAM bottleneck.
RTX 5080 16GB: Qwen3.6 35B MoE at 128k context — 56 tok/s, and why MTP doesn't help
Detailed benchmarks of Qwen3.6 35B MoE on RTX 5080 16GB show that MTP (Multi-Token Prediction) does not improve inference speed at 128k context due to VRAM constraints; the best configuration is Q4_K_XL without MTP, achieving ~56 tok/s generation at 128k context.