@QingQ77: Pure Rust LLM inference engine with custom CUDA kernels for each hardware × model × quantization combination, achieving higher inference speed than vLLM and TensorRT-LLM. https://github.com/Avarok-Cybersecurity/a…

X AI KOLs Timeline Tools
rust llm-inference open-source cuda-kernels performance inference-engine

Summary

Atlas is a pure Rust LLM inference engine that delivers faster inference than vLLM and TensorRT-LLM by customizing CUDA kernels for each hardware × model × quantization combination.

Pure Rust LLM inference engine, with custom CUDA kernels for each hardware × model × quantization combination, achieving higher inference speed than vLLM and TensorRT-LLM. https://github.com/Avarok-Cybersecurity/atlas… Atlas is a pure Rust LLM inference engine, with the entire architecture using traits to split model loading, layer computation, GPU backend, communication, and storage into pluggable modules. It currently targets NVIDIA GB10 first, with 12 hand-tuned kernel goals, running Qwen3.5/3.6/3-Next/3-VL, Gemma-4, Mistral-Small-4, MiniMax-M2, Nemotron-H models. Qwen3.5-35B-A3B using MTP speculative decoding achieves 131 tok/s, faster than NVIDIA's own vLLM on the same GB10. KV Cache has six quantization levels from BF16 to Turbo3 (3-bit Lloyd-Max). Turbo series uses Walsh-Hadamard rotation to reduce error at the same bit width.
Original Article Export to Word Export to PDF
View Cached Full Text

Cached at: 05/08/26, 05:37 PM

Atlas Inference Engine Pure Rust LLM Inference Universal Inference At Unimaginable Speeds

Similar Articles

@LinQingV: When exploring LLM inference chip architectures previously, I reviewed the architectures of the four major AI inference ASIC companies: Groq, SambaNova, Tenstorrent, and Cerebras. While the first three have different emphases, their underlying logic falls within the same framework: large on-chip SRAM + dataflow architecture + deterministic scheduling...

X AI KOLs Timeline

The article analyzes the AI inference ASIC architectures of Groq, SambaNova, Tenstorrent, and Cerebras, highlighting Cerebras's unique wafer-scale engine design. It discusses the benefits of deterministic latency and high bandwidth for LLM inference, while noting challenges like yield, cost, and KV cache bottlenecks.