Tag
Linux 7.3 merges kernel patches that improve VRAM management, enhancing performance stability in games when physical VRAM is exceeded through optimized memory eviction and caching.
Llama.cpp v0.1.0 is a C/C++ implementation for efficient LLM and VLM inference, supporting a wide range of hardware with minimal setup and high performance.
mlx-dspark v0.10.0 adds support for Qwen3.8-27B on Apple Silicon, providing up to 3x faster inference through speculative decoding with lossless verification.
NInfer adds day-0 support for the Qwen3.8-27B model, achieving around 200 tokens per second on a single RTX 5090 with speculative decoding, and includes engine improvements like concurrent requests and kernel optimizations.
AMD upstreamed optimizations to PyTorch/TorchTitan and TorchAO for FP8 training on AMD Instinct GPUs, achieving up to 13.4% throughput gains on Llama3-8B and recovering 89% of FP8 quantization overhead on DeepSeek-V3 via fused Triton kernels.
TokTier is a stateful tokenization service that cuts tokenization overhead in agentic LLM serving by reusing stable token boundaries and running exact CPU/GPU tokenization, reducing time-to-first-token by 16–34% under vLLM.
An open-source tool called XProf Cubism Reducer reduces TensorBoard trace sizes by over 90% for JAX/XLA, making performance profiling more efficient.
LMCache is an open-source KV cache management layer that separates cache I/O from compute, plugging into vLLM, SGLang, and TensorRT-LLM to achieve up to 14x faster time-to-first-token and 4x faster decoding by parallelizing cache lookups and sharing GPU memory.
Gergely Orosz praises Sirupsen for his deep understanding of networking and latency basics, referencing an interview where Sirupsen used napkin math to build a 10x cheaper or faster product.
This article emphasizes the importance of LLM caching in production systems to reduce latency, GPU utilization, and costs, and introduces LMCache, an open-source KV cache management layer for scalable LLM inference.
This blog post from PyTorch presents novel kernel fusion techniques for normalization ops like LayerNorm and RMSNorm, achieving significant speedups by reducing memory-IO overhead. Techniques include Lazy Pre-Norm and Multi-CTA Norm Fusion, hiding up to 90% of normalization latency when fused with GEMMs, and the FlashNormAttention algorithm achieving up to 35% kernel speedup.
A thread highlighting a technical blog series on how Hugging Face's transformers library loads models efficiently, covering meta device, safetensors, CUDA caching, and more.
The article provides the optimal deployment configuration for GLM-5.2 on 8xB200 nodes, showing that NVFP4 with two TP=4 replicas achieves roughly 2x throughput over FP8 TP=8, with detailed performance data and caveats.
A user achieved a 29x speedup for DeepSeek V4 Flash inference on 4x RTX 3090 GPUs by optimizing llama.cpp, reducing a 23k prompt from 25 minutes to 53 seconds.
This paper audits three performance-optimization benchmarks (GSO, SWE-Perf, SWE-efficiency) for coding agents, finding that runtime instability, scoring rules, and task coverage significantly affect reliability, and that many tasks are already solved by at least one public submission.
ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that predicts expert activations from prefill signatures and routes requests to minimize latency, implemented in vLLM and achieving 5.9-13.9% reduction in median TPOT on up to 40 GPUs.
Superpowers 6大幅提升了开发速度和成本效率，通过Fable的优化实现最高50%更快构建和60%更低token消耗，同时改进了对多个AI模型和编码代理的支持。
Manticore's KNN search gets up to 29% faster with 2-pass HNSW, batched distances, compile-time distance specialization, and AVX-512 support.
EGG is an expert-guided agent framework that decomposes GPU kernel generation into algorithmic structure design and hardware-specific tuning, using a stage-aware multi-agent collaboration mechanism. It achieves a 2.13x average speedup over PyTorch on KernelBench and real-world workloads.
The article draws parallels between managing engineering teams and managing AI agents, using Andy Grove's management principles to build reliable agent loops, illustrated through a performance debugging case study at Cursor.