cuda

This article introduces an open-source GPU profiler that serves metrics in JSON format, allowing AI agents to automate performance tuning for CUDA-based AI engines like vLLM and SGLang without manual trace analysis.

This pull request introduces Flash Attention tuning optimizations for CUDA/HIP in the llama.cpp project, targeting gfx1201 hardware to enhance inference performance.

The author is experimenting with an adaptive memory governor for PyTorch to prevent CUDA OOM errors on 8GB GPUs, sharing code and seeking community feedback.

NVIDIA Cosmos3, a 64B parameter image generation model, is released with INT4 quantization for local deployment on CUDA and MLX, with code and weights available and performance demonstrated on Apple Silicon.

Cohere introduces a megakernel serving engine for North Mini Code that achieves 1.25× to 1.41× faster inference than vLLM on H100 GPUs by optimizing memory bandwidth for autoregressive decoding.

Paddock is an open-source Rust/C++ inference engine with custom CUDA kernels, demonstrating competitive performance against vLLM and SGLang in benchmarks while supporting OpenAI/Anthropic style APIs and GGUF/safetensors formats.

PyTorch 2.14 release introduces fault tolerance as a first-class c10d concept with in-place process-group reconfiguration, alongside features like NVGEMM kernels, a new nccl2 backend, and native linear algebra for Apple Silicon.

A bug has been identified in DGX Spark, with the author urging NVIDIA to ensure it receives the latest CUDA updates without delay.

PyTorch 2.14 introduces significant updates including NVGEMM for CUTLASS kernels in Inductor, a new nccl2 backend for distributed computing, first-class fault tolerance, and native linear algebra on Apple Silicon.

This paper introduces mutation analysis as a metric to evaluate benchmark oracles for GPU kernels, revealing limitations in current testing methods and proposing optimizations for better fault detection in LLM-generated code.

The author shares their DIY CUDA box setup using CMP170 mining cards to run AI models like Qwen Flash Next, achieving strong inference performance.

A GitHub repository providing a structured 100-day learning plan for LLM inference engineering, covering topics from CUDA kernels to autoscaling, with runnable scripts.

A developer forked NInfer, a C++20/CUDA inference engine, to add tensor-parallelism and YaRN rope scaling, enabling Qwen3.8-27B to run with a 1M token context on dual 5090 GPUs and outperforming vLLM in specific decode scenarios.

Chinese robots dominate global humanoid shipments at 86%, and Nvidia is expanding its CUDA-style ecosystem into the concentrated robotics market, as reported by WSJ.

An adaptive KV cache streaming fork of llama.cpp enables running the Qwen 3.8 27B model with 262K context on a 16GB RTX 5070 Ti GPU, achieving ~25 tok/s by efficiently managing memory between RAM and VRAM.

This paper introduces DataKernelBench, a benchmark for evaluating LLMs on optimizing GPU kernels for database queries, achieving speedups over baseline methods like torch.compile.

This is a fork of NVIDIA's open GPU kernel modules that enables PCIe peer-to-peer communication between consumer GeForce GPUs (3090, 4090, 5090), significantly improving multi-GPU performance by allowing direct data transfer over PCIe.

The discussion compares CUDA's maturity to MLX in the context of Apple's M5 Ultra Mac Studio, stressing the need for competition to optimize hardware for local AI, which could enable Apple to surpass NVIDIA.

This paper presents a feature-major codebook layout for memory-efficient sparse-binary self-organizing maps, enabling the training of large-scale maps up to 1.05 million neurons on a single GPU with significant speed improvements for MEDLINE data.

At Hot Chips 2026, Nvidia announced plans to extend CUDA support to RISC-V CPUs, outlining specific hardware requirements like server-grade CPUs, ACPI, and PCIe coherency to enable efficient GPU compute.