throughput

DSpark is a speculative decoding framework that combines semi-autoregressive draft generation with confidence-scheduled verification to accelerate LLM inference and improve throughput in high-concurrency settings.

Engineers successfully serve GLM 5.2 on AMD MI355X at 2626 tok/s per node and 213 tok/s single stream, achieving ~80% of B200 throughput at over 2x lower cost than Blackwell.

Etched announced it is coming out of stealth after successful A0 tapeout, with $1B+ customer contracts and $800M raised, claiming SOTA inference performance on its first racks shipping this summer.

Etched, an AI inference hardware startup, exited stealth after raising $800M and securing over $1B in customer contracts. Their first racks ship this summer, claiming state-of-the-art throughput, latency, and power efficiency.

DSpark from DeepSeek AI integrates speculative decoding ideas to achieve 1.5x to 5x higher throughput in production systems. This thread explains 10 key ideas from the basics.

Achieved DeepSeek-V4-Flash MTP speculative decoding on 2× RTX PRO 6000 with a 38% throughput increase by fixing a mis-routed quantization format issue.

SGLang provided Day-0 support for DeepSeek-V4, and collaboration between LMSys and NVIDIA engineering teams achieved up to 5x throughput increase in production, with improvements shown on the SemiAnalysis InferenceX dashboard.

Reports a peak throughput of 19 tokens per second for the Minimax M3 model running on 8-16 MI50 GPUs.

Ray Serve LLM achieves up to 4x higher throughput on prefill-heavy workloads and 24x on decode-heavy workloads in Ray 2.56, matching rust-based routing frameworks like vllm-router in production benchmarks, announced in partnership with Google Cloud GKE team.

The Kimi K2.7 Code High Speed model offers 5x throughput at 2x cost, leading to selective routing within an agent system.

This guide explains the discipline of AI inference engineering, covering the split between prefill and decoding phases, the shift from closed to open models, and optimization techniques for latency, throughput, and cost.

Z Lab, SGLang, and Modal release DFlash, a new speculative decoding model for Qwen 3.5 397B-A17B that uses block diffusion and KV injection to achieve over 4x throughput improvement over baseline and 1.5x over native MTP.

A collaboration between Modal, SGLang, and Z Lab integrates DFlash speculation into SGLang, achieving up to 4.3x throughput improvement for Alibaba's Qwen 397B-A17B model, advancing open intelligence.

New research on DFlash and Spec V2 speculative decoding methods achieves >4.3X baseline throughput for LLM inference, released as the default speculative decoding engine in SGLang.

A multi-agent collaboration using Gemma models achieved major throughput gains and exhibited emergent social behaviors like forming coalitions, issuing ethical statements, and coordinating resources, with over 60 agents and 250 submissions in 48 hours.

Xiaomi and TileRT achieved over 1,000 tokens per second inference on a 1-trillion parameter model using standard commodity GPUs, suggesting a major alternative to custom silicon.

Packed Twin Inference (PTI) is a technique that achieves ~2× LLM throughput by running multiple token sequences in a single batch decode, exploiting weight sharing in llama.cpp without needing a draft model or additional VRAM.

A tweet discusses the idea of building a transactional database with infinite throughput using S3-like object storage and content-addressing, where blocks are written in parallel and the root hash is updated periodically.

Amazon unveiled 'Resilient Network Graphs' (RNG), a data center network design that reduces hardware needs by 69% and increases throughput by 33%, now default for most AWS workloads after quiet deployment since last year.

Benchmarks for Qwen 3.6 27B and 35B models on dual RTX PRO 6000 GPUs using VLLM, showing generation throughput up to 3500 tokens per second.