vllm

A DIY inference machine built with dual RTX 3090 GPUs, a ThinkPad motherboard, and a VW Golf radiator, capable of running AI models like Qwen3.8-27B via vLLM.

Nokia Applied Research has open-sourced WiSP, a vLLM plug-in for running MoE models on GPUs with insufficient VRAM by intelligently paging experts and reallocating memory with the KV cache, achieving up to 2.0× higher decode throughput.

The author introduces a learning track on large model inference optimization, covering topics such as KV Cache, Continuous Batching, PagedAttention, and a comparison between vLLM and SGLang, highlighting it as cutting-edge in AI deployment.

The article promotes a presentation on Elastic Expert Parallelism in vLLM at the PyTorch Conference North America 2026, discussing how to dynamically add or remove GPUs in Mixture-of-Experts deployments with minimal downtime.

The article introduces 1Cat-vLLM, a fork of vLLM optimized for NVIDIA V100 GPUs, and compares its performance with llama.cpp for serving large language models like Qwen3.6-35b.

At PyTorch Conference North America, Maajid Khan from Fujitsu Research India will present a talk on optimizing Mixture of Experts LLM inference on ARM CPUs using vLLM and OpenVINO.

A user shares their experience building a multi-GPU system with RTX 5070 Ti and 5060 Ti for running AI models like Qwen3.8-27B-FP8 using vLLM on Linux, detailing hardware setup, benchmarks, and challenges.

The blog post details the implementation of watermarking in vLLM for establishing text provenance in AI-generated content, leveraging randomness in sampling to balance non-distortion, robustness, and speed.

The preview of Qwen3.8-27B-S-experimental, a quantized AI model using Mirai S codec, is now publicly available on Hugging Face. It supports inference on Apple Silicon with uzu and NVIDIA with vLLM, offering performance details and usage instructions.

This paper synthesizes research on efficient distributed LLM serving, connecting vLLM and llm-d as complementary layers and proposing an Inference Execution Planner for future scheduler development.

vLLM is introducing hardware-agnostic layers to balance high performance on cutting-edge hardware with portability across different accelerators, addressing compatibility issues with torch.compile.

This article identifies and provides fixes for three bugs encountered when serving the MiMo-V2.6-Flash model with vLLM, including empty responses in streaming mode, reasoning loss in tool loops, and a hidden token output cap.

Introduces hardware-agnostic layers in vLLM to maintain high performance while ensuring portability across diverse hardware, as announced in a PyTorch Foundation blog post.

The article announces PyTorch Conference North America 2026, where Andrey Talman from Meta will discuss modernizing the PyTorch release process, including coordination with Triton and vllm, and the use of AI agentic workflows.

Shopify built a continual learning loop using PyTorch and vLLM to improve their GraphQL agent, reducing costs by 96% and outperforming frontier models through production-driven updates.

Audio8 ASR Infinite is a native streaming speech recognition model with selectable audio clock and configurable transcription delay, supporting unlimited-length audio transcription without drifting.

Baseten's 'Inference Engineering' is a systematic book that explains AI inference optimization techniques from CUDA to production deployment, helping engineers efficiently run open-source models in production environments.

This article provides a comparative analysis of self-hosted inference orchestrators for AI model deployment as of September 2026, covering features like multi-machine support, cache-aware routing, and platform compatibility.

A user shares their positive experience using Remotion with the GLM 5.3 Flash model to generate a motion graphics video about stock market analysis, noting its effectiveness and potential for future affordability.

Baidu's open-source Unlimited-OCR model processes multiple PDF pages simultaneously, outperforming baselines like DeepSeek-OCR and supporting local execution with community integrations.