Tag
清华大学等团队发布了 TokenRouter，一个支持小模型与大模型在单次回答中进行 token 级协作的推理服务引擎，通过减少模型切换的等待开销，在双卡 A100 上将 8 路并发的总吞吐从约 135 tokens/s 提升到 372 tokens/s（2.76 倍）。代码与论文均已开源公开。
An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.
dlmserve is the first open-source serving engine for diffusion language models, providing an OpenAI-compatible API, continuous batching, and 2.5x throughput over Hugging Face, all within 12GB VRAM.