Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

Reddit r/LocalLLaMA Tools

Summary

Basalt 0.4.0 adds multi-model and router support (DeepSeek V4.1 Flash), video and file support, and its first dense model Qwen 3.8 27B NVFP4, reaching up to 776 tok/s structured output on an RTX 5090 with 92.8% top-1 fidelity vs BF16.

Hi everyone! I wanted to post a quick update on Basalt. Since last I posted, multi model and router support was implemented, starting with DeepSeek V4.1 Flash (IQ1_M 90 tok/s conv, 246 tok/s struct on a 5090 + 5060 Ti). Also new: video support, files in chats, and a bunch of fixes and improvements. Now 0.4.0 is out featuring Basalt's first dense model, Qwen 3.8 27B NVFP4. It's fast and accurate, top-1 92.8% vs the full BF16 model. This, in preparation for Qwen 4, so we had a headstart on the dense kernels. The numbers: Up to 776 tok/s on structured output, 262 conversation, 194 on prose. Single stream. Prefill, 12,000 tok/s at 4k (slower as context grows) 92.8% top-1 vs BF16 (0.0262 KLD) Chat, tools, thinking, images and video, up to 262K context Links: GitHub: https://github.com/jesdga95/basalt Site: https://basalt.jgulfo.com 27B quant: https://huggingface.co/jesdga/Qwen3.8-27B-Basalt Thanks to everyone who has filed issues, sent PRs or posted benchmark rows, special thanks to the sponsors who keep this going, happy to see Basalt gain so much traction in such a very short time!
Original Article

Similar Articles

Qwen3.8-FN at NVFP4 ( Basalt vs cachy-flash vs ninfer-ext)

Reddit r/LocalLLaMA

对 Qwen3.8-Flash-Next NVFP4 模型在单张 RTX 5090 上的三个推理引擎(basalt、capy-flash、ninfer-ext)进行全 262k 上下文基准测试,basalt 在所有上下文长度和两种解码模式下均领先(prefill 8.1k-9.2k tok/s,greedy decode 176-265 tok/s),解码排名与 MTP 草稿接受率一致。