throughput

New research on DFlash and Spec V2 speculative decoding methods achieves >4.3X baseline throughput for LLM inference, released as the default speculative decoding engine in SGLang.

A multi-agent collaboration using Gemma models achieved major throughput gains and exhibited emergent social behaviors like forming coalitions, issuing ethical statements, and coordinating resources, with over 60 agents and 250 submissions in 48 hours.

Xiaomi and TileRT achieved over 1,000 tokens per second inference on a 1-trillion parameter model using standard commodity GPUs, suggesting a major alternative to custom silicon.

Packed Twin Inference (PTI) is a technique that achieves ~2× LLM throughput by running multiple token sequences in a single batch decode, exploiting weight sharing in llama.cpp without needing a draft model or additional VRAM.

A tweet discusses the idea of building a transactional database with infinite throughput using S3-like object storage and content-addressing, where blocks are written in parallel and the root hash is updated periodically.

Amazon unveiled 'Resilient Network Graphs' (RNG), a data center network design that reduces hardware needs by 69% and increases throughput by 33%, now default for most AWS workloads after quiet deployment since last year.

Benchmarks for Qwen 3.6 27B and 35B models on dual RTX PRO 6000 GPUs using VLLM, showing generation throughput up to 3500 tokens per second.

Open-dLLM adapts Qwen3.6 to use diffusion-based generation, achieving over 3,000 tok/s on an RTX 5090 for short sequences, with code released on GitHub.

Kimi K2.6 achieves 56 tokens per second on a single MI300X GPU; user plans further throughput benchmarking.

K2.6 successfully downloaded and deployed the Qwen3.5-0.8B model locally on a Mac, using the niche Zig language to implement and optimize inference, demonstrating the new model’s generalization ability. After 4,000+ tool calls and 12+ hours of continuous operation, K2.6 iterated 14 times, boosting throughput from ~15 tokens/s to ~193 tokens/s, ultimately achieving 20% faster inference than LM Studio.

The article introduces core concepts from the book "Systems Performance" regarding latency, throughput, cache hierarchies, etc., and references latency numbers from experts like Jeff Dean, emphasizing the importance of hands-on practice for performance engineering.