cuTile Rust introduces a tile-based programming model that leverages Rust's ownership to guarantee memory safety and data-race freedom for GPU kernels, and the Grout inference engine built on it achieves competitive throughput with vLLM/SGLang for Qwen3 models.
I maintain cuTile Rust and just posted the paper "Fearless Concurrency on the GPU." As more GPU code gets AI-generated, the bottleneck moves from writing it to trusting it. cuTile Rust lets you write or generate GPU kernels whose memory safety and data-race freedom are verified by the compiler, through Rust's ownership and borrow checking. You get those guarantees by construction. It's a tile-based programming model that lowers to CUDA Tile IR, carrying Rust's ownership model across the launch boundary. You partition a mutable output into disjoint mutable sub-tensors, pass inputs as shared references, and write tile kernels with single-threaded semantics that the compiler maps to thread blocks. End to end, we built Grout, a Qwen3 inference engine, on cuTile Rust with Hugging Face. At batch-1 decode it reaches 171 tok/s for Qwen3-4B on an RTX 5090 and 82 tok/s for Qwen3-32B on a B200, competitive with vLLM and SGLang. Batch-1 decode is memory-bandwidth-bound, and Grout's throughput is consistent with our HBM roofline analysis. Many of Grout's kernels still use the unsafe path today, but they can be migrated to safe variants, providing a verifiable target for generated kernels. We've started a collection of such kernels in the cutile-kernels crate in the repo. If this is your thing, contributing safe variants helps grow a library of safe, high-performance kernels that future kernel synthesis can draw from. On the kernel side, the safety is effectively free. On a B200 the safe GEMM is within 0.3% of a hand-written low-level version (~92% of dense f16 peak), and element-wise hits ~7 TB/s, matching cuTile Python within measurement noise. Some additional caveats worth noting: Grout is batch-1 with a small set of supported models (a research case study, not a drop-in server), it's NVIDIA-only (lowers to Tile IR), and GEMM still slightly trails cuBLAS at some sizes. - Paper: https://arxiv.org/abs/2606.15991 - Code: https://github.com/nvlabs/cutile-rs - Grout: https://github.com/huggingface/grout Hope you enjoy the paper and learn something new! Happy to answer any questions :)
NVIDIA Labs releases cuTile Rust, a tile-based system for writing memory-safe, data-race-free GPU kernels in idiomatic Rust. It extends Rust's ownership model to GPU kernels, JIT-compiles Rust AST to GPU code, and achieves performance close to native CUDA.
Excited to share cuTile Rust, bringing Rust's fearless concurrency to GPU kernel programming. Their paper 'Fearless Concurrency on the GPU' is now on arXiv.
VectorWare announces that Rust's portable SIMD (core::simd) now works on the GPU, mapping SIMD vectors to warp lanes and enabling familiar Rust abstractions for GPU programming.
Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.
The Rust frontend for vLLM has been officially merged, offering a drop-in alternative to the Python API server with up to 5x throughput improvement on preprocess-heavy workloads.