cuda

Infermeld is an experimental open-source Linux companion kit (v0.1.0) for running one GGUF model across mixed AMD and NVIDIA GPUs using llama.cpp, packaging device selection, build instructions, preflight checks, and a thermal guard. Tested on an RX 6000 XT + RTX 3080 with Qwen3.6-35B-A3B, with the maintainer seeking other hardware combos for reproduction.

SparkTrace LLM 是一个新基准，用 Blackwell GPU 的真实计时结果评估 LLM 阅读 Nsight Compute 性能剖析输出后，能否正确选择最快的 CUDA kernel 重写版本。实验发现包括 GPT 5.4 nano 在内的一些模型会被性能剖析信息误导而选错 kernel。

A llama.cpp pull request (#29184) by am17an fuses shared experts into the MMVQ (matrix-matrix vector quantization) CUDA kernel, optimizing inference for mixture-of-experts models on GPUs.

A hand-written GEMM kernel for NVIDIA Blackwell (B200), built with CUTLASS's CuTeDSL from raw primitives (tcgen05, TMA, mbarriers, TMEM), goes from 164 to 1401 TFLOP/s — about 97% of cuBLAS — with one key change doing most of the work and prefill vs decode workloads yielding different optimizations.

A long-form tech deep dive analyzing whether Google's TPU chips can challenge Nvidia's dominance in GPUs — arguing that the outcome hinges not only on the hardware itself, but on the CUDA software ecosystem, cost economics, and the way large customers like Anthropic are spreading their bets across both types of chips.

ninfer-ext is an extended fork of NInfer that adds support for larger Qwen models, faster speculative decoding, agent serving features, and EXL3 quantization for Qwen3.8-27B on NVIDIA RTX 5090.

This article explains how CUDA Kernels work on GPUs, covering the basics of CUDA, GPU architecture, threads, blocks, and their role in AI and machine learning.

MEM v3 is a memory management tool for PyTorch that dynamically adjusts batch size and gradient accumulation to prevent CUDA OOM crashes during training and fine-tuning, featuring chaos resistance, crash-proof checkpoints, and live telemetry.

KernelZero introduces a co-evolution framework with Proposer and Coder models to enhance GPU kernel generation, achieving superior performance on CUDA and Triton benchmarks.

A user built an open-source inference engine called Strata to optimize running Qwen3.8-Flash-Next on modest hardware, achieving up to 65.1 tok/s from an initial 15 tok/s.

User shares their positive experience using Qwen3.8 model with ExLlamaV3 on 6x3090 GPUs, achieving 80-120 tokens per second, and controlling Hermes agent via Matrix for daily use.

An agentic CUDA kernel optimizer that automates GPU implementation generation through iterative code generation, correctness checks, benchmarking, and refinement, powered by LangGraph and OpenAI models.

Baseten's 'Inference Engineering' is a systematic book that explains AI inference optimization techniques from CUDA to production deployment, helping engineers efficiently run open-source models in production environments.

laya.cpp is an optimized C++ implementation for near-instant decision making, built on ggml with custom CUDA kernels, demonstrating significant speed improvements over Python for the Laya model.

Atlas is an open-source LLM inference engine written in pure Rust and CUDA, achieving high performance and compatibility with NVIDIA and AMD hardware.

A user configured a mismatched pair of Tesla V100 GPUs (16GB and 32GB) into a capable local LLM lab using llama.cpp with tensor split and other optimizations, achieving high prompt and decode speeds with the Qwen3.8 27B model.

The user has implemented nvfp4 KV cache support for the Qwen3.8 model on a heterogeneous GPU setup using custom CUDA kernels and quantization to optimize performance.

Flyweight is an open-source C++/CUDA inference engine for running large MoE models on consumer GPUs with system RAM, offering optimized performance and compatibility with models like Qwen and DeepSeek.

At PyTorch Conference North America 2026, Dhritiman Das will present a torch-native retrieval engine that uses PyTorch as the primary runtime for retrieval, filtering, and ranking, designed for scalable use cases like feed and search at LinkedIn.

Remote development has added AMD R9700 (RDNA 4) support to the Atlas LLM inference engine, enabling it to serve models like Qwen3.8-27B and Ornith-9B on the hardware.