Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090
Summary
Basalt 0.4.0 adds multi-model and router support (DeepSeek V4.1 Flash), video and file support, and its first dense model Qwen 3.8 27B NVFP4, reaching up to 776 tok/s structured output on an RTX 5090 with 92.8% top-1 fidelity vs BF16.
Similar Articles
Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)
Basalt 是一个专为 Blackwell 架构(5090 + 5060 Ti)深度优化的 Strata 分支推理引擎,据称吞吐量可达 Strata 的 2.6 倍,支持 Qwen3.8 Flash-Next、MTP、最多 8 路并发以及自定义视觉编码器,以 MIT 协议完全开源。
Basalt: NVFP4 + faster prefill for Qwen3.8 Flash-Next on Blackwell (single 5090)
Basalt inference tool adds native NVFP4 support and faster prefill for Qwen3.8 Flash-Next on a single RTX 5090, delivering up to 479 tok/s decode and 8,582 tok/s prefill with near-Q8 quality retention.
Basalt 0.5.0: Tencent Hy4 Flash with 1M context and vision, 84 to 172 tok/s on a 5090 + 5060 Ti (NVFP4)
Basalt 0.5.0 发布,新增对腾讯 Hy4 Flash 模型的支持,提供 1M 上下文与视觉能力,在 5090 + 5060 Ti 上实现 84-172 tok/s 吞吐,但量化对其精度影响较大。
[Update Benchmark] Basalt's new PRs (Strata fork) new record IQ3_S Qwen3.8 Flash Next on single RTX 5080 and 51 GB RAM with 5500 tok/s prompt processing and up to 275 tok/s peak generation (during opencode coding live session)
Basalt (a fork of Strata) lands new PRs setting a new benchmark record with IQ3_S Qwen3.8 Flash Next on a single RTX 5080 with 64GB DDR5, reaching 5500 tok/s prompt processing and up to 275 tok/s peak generation during a live opencode coding session.
Qwen3.8-FN at NVFP4 ( Basalt vs cachy-flash vs ninfer-ext)
对 Qwen3.8-Flash-Next NVFP4 模型在单张 RTX 5090 上的三个推理引擎(basalt、capy-flash、ninfer-ext)进行全 262k 上下文基准测试,basalt 在所有上下文长度和两种解码模式下均领先(prefill 8.1k-9.2k tok/s,greedy decode 176-265 tok/s),解码排名与 MTP 草稿接受率一致。