Tag
mistral.rs now natively supports Agent Skills, enabling locally-run small LLMs to perform complex agentic workflows for scientific tasks, with full control over models, data, and execution.
A developer built a from-scratch Rust inference engine with custom GPU kernels that outperforms vLLM on Nemotron-30B decoding, achieving 75.7 vs 57 tok/s, submitted to the Build Small hackathon.
FlashQwen is a minimal from-scratch C++/CUDA inference engine for Qwen3-8B with no external dependencies, supporting multi-turn streaming chat and benchmark mode, with a binary size of only 1.2MB.
The tweet explains that running AI models is really about running optimized kernels, and that inference engines and their kernel implementations are critical for performance, not just the model or hardware.
Sage is a local AI inference engine built in Rust that runs entirely on the user's machine with kernel-level sandbox security, no telemetry, and no cloud dependency.
TRINE is a single-bitstream FPGA accelerator and compiler for end-to-end multimodal inference, unifying diverse layers and incorporating runtime-adaptive compute modes, token pruning, and dependency-aware offloading, achieving up to 22.57x latency reduction over an RTX 4090 at 20-21W.
vLLM v0.22.0 released with 459 commits, featuring DeepSeek V4 hardening, experimental Rust frontend, and batch-invariant Cutlass FP8, reducing end-to-end latency by 28.9%.
Tiny-vLLM is a high-performance LLM inference engine implemented in C++ and CUDA, offering features like continuous batching and PagedAttention, and serves as an educational resource.
Conifer is a new open-source local inference runtime built by a Princeton team, optimized for Apple Silicon with custom Rust kernels. It aims to outperform llama.cpp and MLX for small models and supports fully local agents with OS-level permission enforcement, now entering a limited 100-person beta.
Qwen inference team announced TokenSpeed, a high-performance LLM inference engine for agentic workloads, achieving 540 TPS, with open-source preview available.
A blog post comparing MLX inference engines, concluding oMLX as the top choice, with benchmarks on M5 Max 64GB using Qwen3.6-35B-A3B-4bit.
The article discusses how interaction models pose new challenges for AI model inference engines, with a focus on the vLLM project's solution as covered in a podcast episode featuring Woosuk Kwon.
Redis creator antirez released an open-source project called ds4, a DeepSeek V4 Flash local inference engine optimized for Mac Metal, featuring disk KV caching, ultra-long context, and excellent performance.
This pull request adds support for continuing generation on reasoning models within the llama.cpp server and WebUI.
This article reports on tests of the DS4 inference engine written in C by @antirez, noting its impressive speed when running a GPT-4o-equivalent model on a MacBook Pro with 128GB of RAM.
Atlas is a pure Rust LLM inference engine that delivers faster inference than vLLM and TensorRT-LLM by customizing CUDA kernels for each hardware × model × quantization combination.
A pull request has been merged into llama.cpp to add support for the Mimo v2.5 model, enhancing the framework's compatibility with this specific AI architecture.
ServiceNow engineers detail their migration from vLLM V0 to V1, focusing on resolving backend correctness issues like logprob semantics and runtime defaults to ensure stable reinforcement learning training dynamics.
Antirez released GGUF quantizations of DeepSeek V4 Flash specifically tailored for the DS4 inference engine, providing optimized configurations for different RAM sizes and enabling local execution of the large MoE model.
Developer wrote a Zig-based LLM inference engine from zero on macOS in 12 hours, boosting Qwen 3.5 0.8B throughput from 15 to 193 tokens per second.