[OSS] dlmserve - first serving engine for diffusion language models

Reddit r/LocalLLaMA Tools

Summary

dlmserve is the first open-source serving engine for diffusion language models, providing an OpenAI-compatible API, continuous batching, and 2.5x throughput over Hugging Face, all within 12GB VRAM.

Spent the last few months building this on a single **RTX 5070**. Quick context: **diffusion language models** (like [LLaDA](https://huggingface.co/gsai-ml/LLaDA-8B-Instruct) from gsai-ml) are a different beast from GPT-style autoregressive LLMs. Instead of generating one token at a time, they start with a fully masked sentence and iteratively *denoise* the whole thing in parallel. Cool tech, but mainstream serving engines are all built around the autoregressive contract, so none of them serve diffusion LLMs. **dlmserve** fills that gap: * OpenAI-compatible HTTP API (`/v1/chat/completions`) * Automatic continuous batching at the **denoising-step level** * Optional **LocalLeap** acceleration baked in * **Token-identical** to the reference HF implementation at `temperature=0` * **2.5x throughput** vs HF at `batch=4`, plus another **\~1.8x** from LocalLeap Runs in **12 GB VRAM** (RTX 3090/4090/5070 all fit). MIT licensed. **Repo:** [https://github.com/iOptimizeThings/dlmserve](https://github.com/iOptimizeThings/dlmserve) **Install:** `pipx install dlmserve` (or `pip install dlmserve` if you're in a venv) First public OSS project of this size for me. Genuinely curious what people think. Feedback and code review very welcome, also happy to answer questions about the diffusion serving architecture Edit: Roadmap: - v0.1 ✓ LLaDA-8B-Instruct + LLaDA-1.5 - v0.2 Dream-7B + DiffuLLaMA (issues already open) - v0.3 block diffusion + LLaDA-2.0 + Fast-dLLM KV cache
Original Article

Similar Articles

@_avichawla: https://x.com/_avichawla/status/2077653695123378321

X AI KOLs Timeline

This article argues that vLLM and similar serving frameworks are inefficient for running multiple small AI models on a single GPU due to design limitations. It introduces the SIE open-source inference engine as a solution for serving many models together to reduce costs.