@Gorden_Sun: Strata: Running a 100-billion-parameter model with just 12GB VRAM On a regular computer with 12GB VRAM, locally running…
Summary
Strata 项目实现在 12GB VRAM + 64GB RAM 的普通电脑上本地运行 125B 参数的 Qwen3.8-Flash-Next 模型，通过让常用专家模型常驻 GPU、全模型放入系统内存，并在 GPU、内存与 SSD 之间协同推理和投机采样来突破显存限制。
View Cached Full Text
Cached at: 10/04/26, 09:19 PM
Strata: Running a 100-billion-parameter model with just 12GB VRAM
On a regular computer with 12GB VRAM, locally running the 125B-parameter Qwen3.8-Flash-Next model. Strata keeps the most commonly used expert models resident on the GPU, places the full model in system memory, and coordinates inference and speculative sampling across the GPU, memory, and SSD.
This project is nothing short of god-tier: with 12GB VRAM + 64GB RAM, you can run the full version of the Qwen3.8-Flash-Next model; even with 32GB RAM, you can run the Coder version of the model. Qwen3.8-Flash-Next’s capabilities are no slouch—it can handle repetitive daily tasks with ease.
Github:
Similar Articles
@lxfater: With 64GB RAM, the 100-billion-parameter model is already deployed. We've got the hammer — now where's the nail?
A 12GB GPU with 64GB of system RAM is enough to run a 100-billion-parameter large model: the open-source project Strata can run quantized versions of Qwen3.8-Flash-Next — about 94 tokens/s with the Q2_0 quantization and about 53 tokens/s with IQ3_S on an RTX 5070. The author remarks that the hammer is ready; the question is where the nail (i.e., the application scenario) is.
Anyone sitting on a lot of slow system memory and a modest GPU.. try Strata + Qwen3.8 Next.
A user reports that Strata with Qwen3-Next 80B IQ3_XXS weights (~80GB) runs at 45-70 tokens/s on a DDR4 + 7900 XTX rig, significantly beating Llama.cpp tuned setups (~22.5 t/s), suggesting Strata is a promising inference tool for large MoE models on modest GPUs and slow system memory.
@MinLiBuilds: I just finished running a round of experiments with Strata. This is truly an awesome inference framework—VRAM is still …
A creator shares benchmark experiments with the Strata inference framework, systematically limiting VRAM budgets for the expert cache on an RTX 4090 to measure how speed degrades across different simulated GPU memory tiers, using Qwen3.8-Flash-Next Q2_0 at 8192 context and temperature 0.
@servasyy_ai: Strata 0.1.39 Multi-GPU Optimization is Here! No wonder—some folks' speeds have doubled!
Strata 0.1.39 has been released with multi-GPU optimization. According to user feedback, inference speeds in some scenarios have doubled — benchmarks on a 4090 48G running the hundred-billion-parameter Qwen3.8-Flash-Next showed a significant boost in tokens/s.
@Lonely__MH: Holy crap?! Is it really that badass? It's said that thanks to "MoE heterogeneous parallelism," you can run the 125B pa…
Strata is an open-source tool that uses MoE heterogeneous parallelism to run the 125B-parameter Qwen3.8-Flash-Next model locally on a gaming PC with just 12GB VRAM, reaching over 60-94 tokens/s on NVIDIA RTX 5070 and AMD RX 9070 XT GPUs. The project claims full local, private inference with support for NVIDIA and AMD cards on Windows and Linux.