Tag
SGLang has updated its deployment recipes for the Qwen3.8-27B model on RTX 5090 and RTX Pro 6000 hardware, adding variants for different configurations with tuning options.
A 10-week, 30-minutes-a-day roadmap for engineers to learn LLM inference serving, covering vLLM, SGLang, load testing, quantization, and optimization techniques to build a reproducible benchmark.
Qwen3.8-27B, a 27B-parameter multimodal AI model from Alibaba, is now open source with day-0 support in SGLang, offering high inference speeds and superior performance in coding and office tasks.
A detailed benchmark of DeepSeek V4 Flash on a dual GH200 workstation, comparing SGLang and vLLM at 1M context with DSpark speculative decoding, finding SGLang faster at ~317 vs ~276 tok/s.
DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.
Google Cloud and RadixArk are partnering to bring the SGLang open-source inference framework to Google Cloud TPUs, initially via SGL-JAX and later with SGL-torchtpu for PyTorch-native support, enabling developers to run production workloads seamlessly across GPUs and TPUs.
An updated benchmark shows self-hosting Kimi K3 on 8×B300 nodes achieves 86.4% task resolution at roughly 20% higher hardware cost compared to GLM-5.2 on B200 nodes, though with lower throughput.
Modified SGLang to support Qwen and Laguna models on V100 GPUs using custom FlashAttention and Marlin kernels, achieving decent throughput on 4xV100 hardware.
PyTorch Foundation CTO Matt White will speak at AMD's AdvancingAI event about optimizing AI inference economics using open-source tools like vLLM and SGLang, advocating for right-sized models and intelligent routing to improve dollar per intelligence.
KDFlow is a novel knowledge distillation framework for large language models that uses a decoupled architecture with SGLang for teacher inference and FSDP2 for student training, achieving 1.44x to 6.36x speedup over existing frameworks.
A curated collection of LLM inference recipes and performance benchmarks for popular hardware configurations, including NVIDIA RTX Pro 6000 Blackwell, H100, AMD Strix Halo, and RTX 5090, with detailed batch size, quantization, and throughput data.
A new implementation of the fastest speculative decoding for Qwen models is available, with support in sglang via a Hugging Face fork and upcoming support in the uzu engine.
This article explores whether FlashAttention-3/4 optimizations benefit RTX GPUs, concluding that FA-2 is the ceiling due to hardware limitations on consumer cards.
This thread recommends three key papers for mastering LLM inference: PagedAttention, Sarathi-Serve, and SGLang, which introduce efficient memory management, chunked prefills, and structured generation techniques used in modern inference engines like vLLM and TensorRT-LLM.
SGLang now integrates DSpark, a confidence-driven speculative decoding method that verifies only confident draft tokens, improving inference efficiency under load.
A former Berkeley PhD who leads SGLang at xAI explains how they serve Grok on 100K GPUs using split prefill/decode, expert sharding, and communication/computation overlap to achieve DeepSeek-API-killing prices.
Testing GLM 5.2 with FP8 quantization and FP8 KV cache on H200 yields a score of 79.8% on Terminal-Bench 2.1, with one timeout not rerun.
The article explores how agent tools are being used to encode development workflows for SGLang, turning debugging, benchmarking, and profiling into executable skills and reproducible experiments, with efforts like KDA-Pilot already producing merged PRs.
An educational blog post explaining how SGLang works, including its runtime, frontend language, RadixAttention mechanism, and comparison to vLLM.
GLM 5.2 post-training code is open-sourced, using Megatron-LM for training and SGLang for rollout generation, forming a continuous RL loop with synchronized weights.