throughput

NVIDIA 推出 Nemotron 3.5 Super VL 开放模型（120B 总参数/12B 激活），支持文本、图像、文档和视频理解及工具使用，Nous Research 宣布 Hermes Agent 是使用该模型及 Bolt 模型的最佳平台，Super VL 在 Hermes Bench 上得分 63.44，本地吞吐量最高提升 5 倍。

清华团队发表 TokenRouter 论文，提出一种按 token 级别在大小模型间路由的 LLM 服务系统，通过独立服务器协作与 KV cache 传递，在 5 种路由方法上实现相比现有最强框架 2.01 到 64.15 倍的吞吐提升。

Tailscale details upcoming performance improvements to its networking product, including reduced memory overhead for small packets and planned throughput enhancements for late 2026.

The tweet identifies inference serving as a prime target for autoresearch, emphasizing end-to-end optimization with constraints on latency, quality, and throughput, covering various aspects in a unified search space and hinting at future developments.

Benchmark tests on Ling-3.0-flash show that higher acceptance length in multi-token prediction decreases prose throughput, with n=1 being the most efficient setting for the evaluated workloads.

The paper proposes Random Attention, a KV cache eviction method that uses random selection instead of scoring, matching selective methods while improving throughput in reasoning tasks.

NVIDIA reports up to 30× higher agentic throughput per megawatt with Vera Rubin compared to GB300, but emphasizes that production benchmarks should include additional metrics beyond tokens per watt.

NVIDIA's Vera Rubin NVL72 system demonstrates up to 30x higher throughput per megawatt for agentic AI workloads, setting a new efficiency standard for AI infrastructure.

Wi-Fi 8 shifts focus from speed to reliability, aiming to reduce interference and latency while maintaining similar data rates as Wi-Fi 7.

The LFM2.5-2.6B AI model is free to use on OpenRouter with good throughput, and is promoted for agentic harness applications.

The user tested scaling local AI agents with a Qwen 27B model, finding that adding more agents increases throughput only up to a point due to memory bandwidth limits, with long prompts benefiting more from parallelism.

DFlash 2 improves speculative decoding by predicting tokens in parallel, achieving over 20% more output per verification pass with minimal latency, and is integrated into major inference engines like SGLang and vLLM.

An observation of Ling-3.0-flash (124B) on one DGX Spark generating 15,128 tokens in a single response with stable decode throughput around 35.6 tok/s, highlighting long-context decoding performance.

A PlanetScale engineering post analyzes a MySQL outage caused by a long-running transaction and high concurrency, explaining how parallelism can degrade throughput and how Vitess's transaction pool handled (and amplified) the issue.

A practical tutorial on measuring transformer inference performance, covering metrics like latency, TTFT, throughput, memory usage, and benchmarking techniques for LLMs.

Intern-S2 Mobius is a Qwen3.5-35B derived model with an architectural difference claimed to improve throughput and reduce token consumption.

Wafer announces that it can serve Kimi K3 on AMD MI355X at 3.8x higher throughput and 71% lower cost than on B200 nodes, arguing that AMD's large VRAM and software support make it the best performance-per-dollar choice for frontier models.

Kimi K3 reached #10 on OpenRouter within two days of launch, processing 140 billion tokens daily, but high load caused throughput to drop and latency to spike.

NVIDIA introduces a series on AI Model Co-Design, explaining how model dimensions affect GPU performance and the trade-offs between throughput and interactivity for LLM deployment. The first post provides a practical primer on designing hardware-friendly LLMs to improve system throughput and user responsiveness.

This article presents a recipe for low-precision (NVFP4) RL training that balances throughput and stability, addressing issues from forward and backward pass quantization errors.