vllm

The vLLM Conference concluded successfully with high community attendance, and session recordings are available in the thread.

OpenJev is an open-source server offering a free API compatible with TypeSafe's Jev for probabilistic predictions, with performance benchmarks showing competitive latency.

Optimized performance for running the Qwen3.8 27b NVFP4 model on a single AMD Radeon R9700 GPU, achieving up to 153 tokens per second in decode and 3,619 tokens per second in prefill with improved concurrency.

This article introduces an open-source GPU profiler that serves metrics in JSON format, allowing AI agents to automate performance tuning for CUDA-based AI engines like vLLM and SGLang without manual trace analysis.

The article discusses the tradeoff between generality and specialization in AI inference engines, with vLLM and SGLang as examples, and notes that coding agents are reducing engineering costs for creating specialized engines.

A technique to offload the KV cache of Qwen3.8-Flash-Next to system RAM is demonstrated, allowing long-context inference with minimal decode slowdown by leveraging the model's efficient architecture.

This paper presents a measured study on learned request routing for disaggregated LLM serving, evaluating a calibrated router that uses admission-time features to assign requests, achieving higher mean goodput compared to traditional methods on a real vLLM/NIXL cluster.

Google Cloud and Inferact announce a partnership to optimize vLLM for TPU, making it a first-class citizen in the open model ecosystem for agentic production serving.

A software developer shares their experience switching from a MacBook to an RTX3090 Linux setup for running AI models, achieving significantly higher inference speeds with Qwen 3.8 27B and potentially replacing their Claude subscription.

Lorivo is a serverless platform that allows sharing GPU servers for hosting multiple LoRA adapters, simplifying deployment and reducing costs for fine-tuned AI models.

The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.

Nicolò Lucchesi will present at PyTorch Conference North America 2026 on the evolution of disaggregated serving in vLLM for hybrid models, collaborating with AWS and RedHat.

The article shares a curated reading list for getting started with AI infrastructure, focusing on large model inference topics such as vLLM, continuous batching, and performance benchmarking.

At PyTorch Conference North America, Ricardo Noriega de Soto and Alexander Brooks will demonstrate how extending vLLM's prefix caching mechanism to multistage pipelines boosts inference speeds while reducing GPU memory overhead, providing practical strategies for optimizing complex AI workloads.

Running the DeepSeek-V4-Flash-Vision-Exp 285B MoE model on 10-12x RTX 3090 GPUs achieves over 60-120 tok/s decode speeds with vision and tool support, fully documented for reproducibility.

Benchmark tests on Ling-3.0-flash show that higher acceptance length in multi-token prediction decreases prose throughput, with n=1 being the most efficient setting for the evaluated workloads.

Author released an open-source tool called cache-pressure that benchmarks how well local LLM inference engines actually retain KV cache contexts under pressure, allowing users to verify real cache capacity against advertised claims.

NVIDIA's open kernel driver for DGX Spark received two fixes that return freed GPU memory to the OS when a process exits and enable huge pages for GPU page faults on system memory, boosting first-touch bandwidth from 0.4 to 19.6 GiB/s.

TorchSpec is a PyTorch-native framework for training speculative decoding draft models, released in collaboration with vllm and demonstrated with Kimi K3 draft models on NVIDIA GB200 hardware.

User @ViC305 successfully runs DeepSeek-V4-Flash-Vision with EXL3 MixedK and DSpark speculative decoding on a single DGX Spark, achieving improved performance and fixing technical issues for multimodal AI deployment.