cuda

An agentic CUDA kernel optimizer that automates GPU implementation generation through iterative code generation, correctness checks, benchmarking, and refinement, powered by LangGraph and OpenAI models.

Baseten's 'Inference Engineering' is a systematic book that explains AI inference optimization techniques from CUDA to production deployment, helping engineers efficiently run open-source models in production environments.

laya.cpp is an optimized C++ implementation for near-instant decision making, built on ggml with custom CUDA kernels, demonstrating significant speed improvements over Python for the Laya model.

Atlas is an open-source LLM inference engine written in pure Rust and CUDA, achieving high performance and compatibility with NVIDIA and AMD hardware.

A user configured a mismatched pair of Tesla V100 GPUs (16GB and 32GB) into a capable local LLM lab using llama.cpp with tensor split and other optimizations, achieving high prompt and decode speeds with the Qwen3.8 27B model.

The user has implemented nvfp4 KV cache support for the Qwen3.8 model on a heterogeneous GPU setup using custom CUDA kernels and quantization to optimize performance.

Flyweight is an open-source C++/CUDA inference engine for running large MoE models on consumer GPUs with system RAM, offering optimized performance and compatibility with models like Qwen and DeepSeek.

At PyTorch Conference North America 2026, Dhritiman Das will present a torch-native retrieval engine that uses PyTorch as the primary runtime for retrieval, filtering, and ranking, designed for scalable use cases like feed and search at LinkedIn.

Remote development has added AMD R9700 (RDNA 4) support to the Atlas LLM inference engine, enabling it to serve models like Qwen3.8-27B and Ornith-9B on the hardware.

This article introduces an open-source GPU profiler that serves metrics in JSON format, allowing AI agents to automate performance tuning for CUDA-based AI engines like vLLM and SGLang without manual trace analysis.

This pull request introduces Flash Attention tuning optimizations for CUDA/HIP in the llama.cpp project, targeting gfx1201 hardware to enhance inference performance.

The author is experimenting with an adaptive memory governor for PyTorch to prevent CUDA OOM errors on 8GB GPUs, sharing code and seeking community feedback.

NVIDIA Cosmos3, a 64B parameter image generation model, is released with INT4 quantization for local deployment on CUDA and MLX, with code and weights available and performance demonstrated on Apple Silicon.

Cohere introduces a megakernel serving engine for North Mini Code that achieves 1.25× to 1.41× faster inference than vLLM on H100 GPUs by optimizing memory bandwidth for autoregressive decoding.

Paddock is an open-source Rust/C++ inference engine with custom CUDA kernels, demonstrating competitive performance against vLLM and SGLang in benchmarks while supporting OpenAI/Anthropic style APIs and GGUF/safetensors formats.

PyTorch 2.14 release introduces fault tolerance as a first-class c10d concept with in-place process-group reconfiguration, alongside features like NVGEMM kernels, a new nccl2 backend, and native linear algebra for Apple Silicon.

A bug has been identified in DGX Spark, with the author urging NVIDIA to ensure it receives the latest CUDA updates without delay.

PyTorch 2.14 introduces significant updates including NVGEMM for CUTLASS kernels in Inductor, a new nccl2 backend for distributed computing, first-class fault tolerance, and native linear algebra on Apple Silicon.

This paper introduces mutation analysis as a metric to evaluate benchmark oracles for GPU kernels, revealing limitations in current testing methods and proposing optimizations for better fault detection in LLM-generated code.

The author shares their DIY CUDA box setup using CMP170 mining cards to run AI models like Qwen Flash Next, achieving strong inference performance.