throughput

Benchmark tests on Ling-3.0-flash show that higher acceptance length in multi-token prediction decreases prose throughput, with n=1 being the most efficient setting for the evaluated workloads.

The paper proposes Random Attention, a KV cache eviction method that uses random selection instead of scoring, matching selective methods while improving throughput in reasoning tasks.

NVIDIA reports up to 30× higher agentic throughput per megawatt with Vera Rubin compared to GB300, but emphasizes that production benchmarks should include additional metrics beyond tokens per watt.

NVIDIA's Vera Rubin NVL72 system demonstrates up to 30x higher throughput per megawatt for agentic AI workloads, setting a new efficiency standard for AI infrastructure.

Wi-Fi 8 shifts focus from speed to reliability, aiming to reduce interference and latency while maintaining similar data rates as Wi-Fi 7.

The LFM2.5-2.6B AI model is free to use on OpenRouter with good throughput, and is promoted for agentic harness applications.

The user tested scaling local AI agents with a Qwen 27B model, finding that adding more agents increases throughput only up to a point due to memory bandwidth limits, with long prompts benefiting more from parallelism.

DFlash 2 improves speculative decoding by predicting tokens in parallel, achieving over 20% more output per verification pass with minimal latency, and is integrated into major inference engines like SGLang and vLLM.

An observation of Ling-3.0-flash (124B) on one DGX Spark generating 15,128 tokens in a single response with stable decode throughput around 35.6 tok/s, highlighting long-context decoding performance.

A PlanetScale engineering post analyzes a MySQL outage caused by a long-running transaction and high concurrency, explaining how parallelism can degrade throughput and how Vitess's transaction pool handled (and amplified) the issue.

A practical tutorial on measuring transformer inference performance, covering metrics like latency, TTFT, throughput, memory usage, and benchmarking techniques for LLMs.

Intern-S2 Mobius is a Qwen3.5-35B derived model with an architectural difference claimed to improve throughput and reduce token consumption.

Wafer announces that it can serve Kimi K3 on AMD MI355X at 3.8x higher throughput and 71% lower cost than on B200 nodes, arguing that AMD's large VRAM and software support make it the best performance-per-dollar choice for frontier models.

Kimi K3 reached #10 on OpenRouter within two days of launch, processing 140 billion tokens daily, but high load caused throughput to drop and latency to spike.

NVIDIA introduces a series on AI Model Co-Design, explaining how model dimensions affect GPU performance and the trade-offs between throughput and interactivity for LLM deployment. The first post provides a practical primer on designing hardware-friendly LLMs to improve system throughput and user responsiveness.

This article presents a recipe for low-precision (NVFP4) RL training that balances throughput and stability, addressing issues from forward and backward pass quantization errors.

A real-time demo shows 16 concurrent users chatting with Qwen3.6-35B on a single DGX Spark, achieving peak 440 tok/s total and 105 tok/s per user using NVFP4 + MTP-3 on vLLM.

DeepSeek-V4-Flash-DSpark achieves 328 tok/s single inference and 1.7k tok/s batch throughput on 4x RTX PRO 6000 GPUs.

This thread recommends three key papers for mastering LLM inference: PagedAttention, Sarathi-Serve, and SGLang, which introduce efficient memory management, chunked prefills, and structured generation techniques used in modern inference engines like vLLM and TensorRT-LLM.

The paper introduces Nemotron-Labs-Diffusion, a tri-mode language model that unifies autoregressive, diffusion, and self-speculation decoding, achieving superior throughput and efficiency compared to existing models.