@zhyncs42: Qwen inference team is super great — they achieved 540 TPS on TokenSpeed for agentic workloads Looking forward to them …
Summary
Qwen inference team announced TokenSpeed, a high-performance LLM inference engine for agentic workloads, achieving 540 TPS, with open-source preview available.
View Cached Full Text
Cached at: 05/24/26, 10:27 AM
Qwen inference team is super great — they achieved 540 TPS on TokenSpeed for agentic workloads 🚀 Looking forward to them sharing more optimization details soon. Stay tuned. https://t.co/R5W1RL7TTQ Enjoy! https://t.co/eZtNq1Hi3w
lightseekorg/tokenspeed
Source: https://github.com/lightseekorg/tokenspeed
TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. Our goal is to be the most performant inference engine for production agentic workloads.
Core components:
- Modeling layer: local-SPMD design with a static compiler that generates collective communication from module-boundary placement annotations, so users do not hand-write parallelism logic.
- Scheduler: C++ control plane and Python execution plane. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time.
- Kernels: pluggable, layered kernel system with a portable public API and a centralized registry including one of the fastest MLA (Multi-head Latent Attention) implementations on Blackwell for agentic workload.
- Entrypoint: SMG-integrated AsyncLLM for low-overhead CPU-side request handling.
Performance Comparison
Preview Status
This version is a preview release for reproducing the Kimi K2.5 on B200 and TokenSpeed MLA on B200 results from the TokenSpeed blog. Several major PRs are still in progress and have not been merged yet.
Ongoing work includes:
- Model coverage: Qwen 3.6, DeepSeek V4, and MiniMax M2.7.
- Runtime features: PD, EPLB, KV store, Mamba cache, VLM, and metrics.
- Platform optimization: Hopper optimization, MI350 optimization, and related runtime improvements.
These features are still being cleaned up and will be merged into main over
the next few weeks. TokenSpeed is currently under heavy development and is
intended to showcase the new runtime design and technical direction. Do not use
this preview release for production deployments.
Documentation
Start here:
Similar Articles
I pushed Qwen3.8-27B to 99 tps single request and 1150 tps with a batch request on a RTX 3090
The author optimized the Qwen3.8-27B model inference on an RTX 3090 GPU, achieving up to 99 tokens per second for single requests and 1150 tps with batch processing through various quantization and optimization techniques, and released the updated code on GitHub.
@MiaAI_lab: Nvidia did it again! @NVIDIAAI's Qwen 3.6 27B NVFP4 is faster than Unsloth's Qwen 3.6 27B NVFP4 by a whopping ~41% on D…
Nvidia's optimized Qwen 3.6 27B NVFP4 model achieves 41% faster single-session inference and 23-25% faster concurrent inference on DGX Spark compared to Unsloth's version.
Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
The article details an experiment achieving 50 tokens per second inference with Qwen3.8-27B at 256K context on a 24GB GPU using Multi-Token Prediction and custom optimizations.
Qwen3.6 27B on a 5090, 6.4k sample tok/s distribution after tuning MTP/cache settings
Running Qwen3.6 27B on an RTX 5090, achieving 6.4k tokens per second after tuning MTP and cache settings, demonstrating optimization techniques for inference.
Qwen 3.5 4B IQ2_XS: +16.67% Reasoning Performance From Tensor-Level Allocation
ByteOtter replicates tensor-level allocation on Qwen 3.5 4B, achieving a 16.67% relative improvement in reasoning performance with only a 0.412% increase in model size, marking the first cross-family application outside Gemma.