Qwen3.8 flash next ISTA-DASLab GGUF 50t/s TG and 1500t/s PP with 12GB VRAM and 64GB RAM Laptop on 'Strata' engine
Summary
The article highlights the Strata inference engine, which significantly outperforms llama.cpp for running Qwen3.8 models on a laptop with 12GB VRAM and 64GB RAM, achieving up to 50 tokens per second for text generation and 1500 tokens per second for prompt processing.
Similar Articles
Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s
Strata 是一个免费开源的推理引擎,可在 RTX 4090 等消费级显卡(12GB+ 显存)本地运行 Qwen3.8-Flash-Next(125B)模型,写入速度约 100 tokens/s,且支持 Windows/Linux 和 NVIDIA/AMD 显卡。
@coldniko: Run Qwen3.8-Flash-next on 8GB+ AMD GPU RX 7900 XTX runs the model at a sustained 52-60 output tokens per second + 1250 …
Strata is an open-source tool that runs Qwen3.8-Flash-Next, a 125B-parameter model, on consumer gaming PCs with quantization, delivering 52-60 tokens/s on an AMD RX 7900 XTX and 60-95 tokens/s on an RTX 5070, with KV cache optimizations like k8v4 for an extra 10% speedup.
Anyone sitting on a lot of slow system memory and a modest GPU.. try Strata + Qwen3.8 Next.
A user reports that Strata with Qwen3-Next 80B IQ3_XXS weights (~80GB) runs at 45-70 tokens/s on a DDR4 + 7900 XTX rig, significantly beating Llama.cpp tuned setups (~22.5 t/s), suggesting Strata is a promising inference tool for large MoE models on modest GPUs and slow system memory.
Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM
The article reports that the Qwen3.8-Flash-Next model achieves 120 tokens/second generation speed and 12k tokens/second prefill on a system with 4x AMD R9700 GPUs using optimized vLLM and a custom Docker image.
Qwen 3.6-35B-A3B with 977 tk/s prompt processing and 262k context window on Intel Arc B70 Pro
This article describes how to use the SYCL backend with llama.cpp to achieve over 60 tokens per second on the Qwen 3.6-35B-A3B model using an Intel Arc Pro B70 GPU, with the entire model and KV cache in VRAM.