For V100 Users: SGLang running Qwen+Dflash and Laguna
Summary
Modified SGLang to support Qwen and Laguna models on V100 GPUs using custom FlashAttention and Marlin kernels, achieving decent throughput on 4xV100 hardware.
Similar Articles
Qwen3.8-Flash-Next NVFP4 Day-3 support for 4xV100
RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100, enabling full context operation on 4xV100 GPUs with performance metrics showing high throughput and context handling up to 256k tokens.
I spent 3 weeks testing local Qwen3.8 on the new low-latency SGLang/vLLM recipes: DFlash2 2.8x. Builds: RadixArk + Inferact 27B NVFP4, 27B BF16, orcarouter 27B Uncensored, Flash-Next NVFP4
作者在单张 RTX PRO 6000 上对三款 Qwen3.8 27B 检查点进行 10 项统一测试,Flash-Next 赢下多数任务,而 DFlash2 投机解码将 27B 的推理速度从 75 提升到 210 tok/s(2.8 倍),并对比了 SGLang 与 vLLM 在不同量化导出下的性能差异。
Getting Qwen3.8-27B with decent speed on my 4080 with 16Gb card
This guide explains how to configure DFlash2 speculative decoding with llama.cpp to achieve up to 1.72x faster inference for the Qwen3.8-27B model on an RTX 4080 GPU with 16GB VRAM.
366 t/s Qwen3.6 27B NVFP4 on v100s
Introduces v100-skinny, a custom kernel library enabling fast NVFP4 inference on V100 (sm70) GPUs, achieving 366 t/s for Qwen3.6 27B in best-case extraction, with lower speeds for structured generation and code.
@lxfater: Two old V100s can also run the 321 billion parameter GLM-5.3-Flash locally!! This open-source project is called Project…
开源项目 Project Maya 基于 Strata 引擎并适配 GLM,让两张 V100 32GB 显卡加约 32GB 内存和 NVMe SSD 即可本地运行 3210 亿参数的 GLM-5.3-Flash,生成速度可达 40 tokens/s,还内置 Web 聊天、图像理解和 OpenAI/Anthropic API 兼容。