Tag
NVIDIA BioNeMo Inference Runtime accelerates biomolecular structure-prediction models using optimized kernels, CUDA Graphs, and Ray for high throughput while maintaining PyTorch workflow integration.
Benchmark tests on Ling-3.0-flash show that higher acceptance length in multi-token prediction decreases prose throughput, with n=1 being the most efficient setting for the evaluated workloads.
ShardFlow, a distributed LLM inference framework, achieves 28 TPS on Qwen2.5-7B across cloud regions by using speculative decoding and CUDA Graphs to mitigate WAN latency.
The posters for PyTorch Con North America are now available, showcasing research in areas like compiler optimization, CUDA Graphs, AI architectures, and reinforcement learning. The conference will be held in San Jose, CA on October 20-21, with early registration discounts.
SonicSampler presents a unified suite of tile-aware Triton kernels that vertically fuse the entire LLM sampling pipeline, supporting dynamic per-request behaviors and speculative verification, achieving up to 16x speedup over state-of-the-art baselines.
A detailed tutorial on implementing CUDA Graphs in an LLM inference server Tokn, covering FastAPI server setup, engine initialization, and CUDA Graph capture for optimized decode phases.
This paper investigates the performance gap in batch-1 LLM decode for physical AI systems, finding that faster memory bandwidth does not proportionally reduce latency due to launch overheads, and that quantization efficiency varies significantly across hardware.