sglang

Baseten's 'Inference Engineering' is a systematic book that explains AI inference optimization techniques from CUDA to production deployment, helping engineers efficiently run open-source models in production environments.

LLM2Jev is an open-source tool that adapts local HuggingFace models to perform structured decisions with Choice, Score, and Noul frameworks, offering prefill-only inference and integration with Transformers and SGLang.

The article explains how to build a local decision engine using open-source LLMs and SGLang, enabling efficient scoring and probability distributions for fixed choices without full text generation, compared to systems like Jev.

The zero-to-sglang course has hit 1,000 GitHub stars, with new chapters released covering mini-sglang and the journey of a request including code walkthroughs.

A developer built a project in 8 hours using the Qwen3.8-Flash-Next model on a single NVIDIA DGX Spark, generating around 10k lines of code and consuming 800k tokens.

This article introduces an open-source GPU profiler that serves metrics in JSON format, allowing AI agents to automate performance tuning for CUDA-based AI engines like vLLM and SGLang without manual trace analysis.

The article discusses the tradeoff between generality and specialization in AI inference engines, with vLLM and SGLang as examples, and notes that coding agents are reducing engineering costs for creating specialized engines.

SGLang-Diffusion with VDN-H3 from MiniMax enables fast and scalable video generation, achieving 14.4s of 768p video in just 9.0s on 8× B200 GPUs for faster-than-real-time inference.

The article explains RadixAttention, a technique in SGLang that uses a radix tree to efficiently cache and reuse overlapping KV caches across branching requests in AI serving.

Benchmarks comparing SGLang, llama.cpp, and FreeToken on Qwen3.8-Flash-Next at full context show SGLang achieves the fastest time to first token at 35.4s, while llama.cpp baseline takes 258.4s, with speculative decoding providing performance improvements.

AdaptiveSpec is a training-free per-step speculative decoding method that adaptively adjusts token verification and draft tree shape to enhance LLM inference throughput, improving performance by up to 56% while maintaining high accuracy across benchmarks.

SGLang has added deployment recipes for DeepSeek-V4-Flash-Vision and DeepSeek-V4-Flash-0731 models on 2x DGX Spark hardware, with support for various configurations and optimizations.

RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100, enabling full context operation on 4xV100 GPUs with performance metrics showing high throughput and context handling up to 256k tokens.

A new checkpoint for Qwen3.8 Flash-Next offloads the ngram lookup table to SSD, reducing RAM usage by 48 GB while maintaining inference speed in SGLang.

An educational overview of key concepts in LLM inference engineering, covering techniques like KV cache, PagedAttention, Flash Attention, and continuous batching to optimize inference performance.

The article benchmarks the acceleration of MiniMax-H3 video generation on 8×H200 GPUs using SGLang Diffusion, achieving up to 6.24× speedup with quality measured by SSIM.

Josh Tobin reports a success in automated research where AI identified and helped fix edge cases that could impact inference performance in vLLM and SGLang, specifically in the FlashInfer library.

Liquid AI releases DSpark draft model checkpoints for the LFM2.5 family, enabling up to 3.2x faster inference on GPUs and devices with minimal quality trade-off, and with day-one support for open-source tools like llama.cpp and SGLang.

SGLang is announced as the native rollout engine for Miles v0.1, an open-source reinforcement learning framework for LLMs and multimodal models, aimed at improving throughput, cache efficiency, and stability in RL training at scale.

SGLang has updated its deployment recipes for the Qwen3.8-27B model on RTX 5090 and RTX Pro 6000 hardware, adding variants for different configurations with tuning options.