We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)
Summary
Paddock is an open-source Rust/C++ inference engine with custom CUDA kernels, demonstrating competitive performance against vLLM and SGLang in benchmarks while supporting OpenAI/Anthropic style APIs and GGUF/safetensors formats.
Similar Articles
Building a Rust Inference Engine That Matches Llama.cpp
Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.
Fearless Concurrency on the GPU: Safe GPU inference in Rust, competitive with vLLM/SGLang [R]
cuTile Rust introduces a tile-based programming model that leverages Rust's ownership to guarantee memory safety and data-race freedom for GPU kernels, and the Grout inference engine built on it achieves competitive throughput with vLLM/SGLang for Qwen3 models.
@SuJinYan123: https://github.com/openinfer-project/openinfer/pull/522… openinfer p/d 分离支持，based on store（pegaflow p2p) based p/d pull…
OpenInfer 0.2.0 released, supporting speculative decoding, p/d separation, and GLM5.2 FP8 inference; this project is a pure Rust + CUDA LLM inference engine with no dependencies on frameworks like PyTorch.
@npashi: Finally able to talk about what I've been heads-down on for 6 months at @nvidia We just open-sourced cuda-oxide — an ex…
NVIDIA has open-sourced cuda-oxide, an experimental rustc backend that allows developers to write CUDA kernels directly in pure Rust without DSLs, FFI, or source-to-source translation.
An open handbook on LLM inference at scale (GPU internals, KV cache, batching, vLLM/SGLang/TensorRT-LLM) [P]
An open, in-progress handbook explaining LLM inference internals including GPU memory hierarchy, KV cache, batching, and popular inference engines like vLLM and TensorRT-LLM.