This article provides a detailed guide on running the Qwen3.8-Flash AI model on an RTX 3090 with 64GB RAM, discussing performance metrics, quantization settings, and deployment steps using tools like llamacpp.
I've got Qwen3.8-Flash-next running on RTX 3090, Ryzen 9 3950X, a PCIe 3.0 motherboard, and 64GB DDR RAM from 2020. IQ4_XS weights, full kvarn5 context, vision on GPU, experts in host RAM, n-grams on disk. MTP works but actually slows decode down even with 80% draft acceptance, as expected since every rejected token eats into the host RAM bandwidth. I get 160 tok/s prefill 16 tok/s decode, which makes it a decent option whenever I know I'll be AFK for at least a couple of hours, but not usable for interactive work. Variant setups kvarn5 is unrecognizable from q8/q8 on the KLD charts for the Qwen models. If you don't want to use Beellama, q5_0/q5_0 is also fine (just a very minor drop). Nonetheless, there's plenty of headroom so you can bump up the KV quant to q8/q8 if you prefer. You can go down to 16GB VRAM, with enough room for desktop, if you drop KV to kvarn4 and offload the vision tower to CPU - but then you'll need to make sure you don't breathe too hard because you're going to have very little spare host RAM for running anything else. I do not recommend using plain q4_0/q4_0 KV as the drop starts being measurable. You can fit in a 12GB card by further dropping ub from 2048 to 512, but your prefill will halve. How to deploy One-line deployment (CUDA Linux): https://github.com/crusaderky/pixi-llm-recipes . Choose llamacpp-source-cuda when starting the server. It should also work on Vulkan and ROCm, but it's untested. Just the llamacpp fork: https://github.com/crusaderky/llama.cpp/releases/tag/beellama-staging-v0.4.4-r9; if you wait it will eventually land in the main beellama branch. Or you can just use llamacpp master without kvarn. Just the llamacpp preset: https://github.com/crusaderky/pixi-llm-recipes/blob/26ed50ace2a40772aa2b45d1358aaf0993fd5596/models.ini#L3-L94 u/andbeeld can we have one more merge from llamacpp main before v0.4.4 final? Your latest merge is *just* before support for Qwen3.8-Flash was added. But I heard that you should never reduce KV cache quant below q8/q8? I don't care about people's vibes. I have not tested this model yet but I have tested
The article benchmarks llama.cpp and ik_llama.cpp for running the Qwen3.8-Flash-Next model on dual RTX 3060 hardware, showing that optimizing tensor splitting and ubatch settings can dramatically improve prefill speed from 36 t/s to 400 t/s, with comparisons of RAM usage.
User shares performance results and setup details for running the Qwen3.8-Flash-Next model with Llama.cpp on a system featuring an NVIDIA 5090 GPU and 64GB RAM, noting unexpected low RAM usage during inference.
This guide explains how to configure DFlash2 speculative decoding with llama.cpp to achieve up to 1.72x faster inference for the Qwen3.8-27B model on an RTX 4080 GPU with 16GB VRAM.
A user inquires about the minimum hardware specifications, such as RAM and VRAM, required to run the Qwen3.8-Flash-Next AI model, including expected performance metrics.
The author demonstrates how to run a 182B Qwen model on a 4080 GPU with 64GB DDR5 by offloading ngrams to SSD, achieving faster and more intelligent performance than a 27B model, making large model inference accessible on consumer hardware.