Tag
Conifer is a new open-source local inference runtime built by a Princeton team, optimized for Apple Silicon with custom Rust kernels. It aims to outperform llama.cpp and MLX for small models and supports fully local agents with OS-level permission enforcement, now entering a limited 100-person beta.
Qwen inference team announced TokenSpeed, a high-performance LLM inference engine for agentic workloads, achieving 540 TPS, with open-source preview available.
A blog post comparing MLX inference engines, concluding oMLX as the top choice, with benchmarks on M5 Max 64GB using Qwen3.6-35B-A3B-4bit.
The article discusses how interaction models pose new challenges for AI model inference engines, with a focus on the vLLM project's solution as covered in a podcast episode featuring Woosuk Kwon.
Redis creator antirez released an open-source project called ds4, a DeepSeek V4 Flash local inference engine optimized for Mac Metal, featuring disk KV caching, ultra-long context, and excellent performance.
This pull request adds support for continuing generation on reasoning models within the llama.cpp server and WebUI.
This article reports on tests of the DS4 inference engine written in C by @antirez, noting its impressive speed when running a GPT-4o-equivalent model on a MacBook Pro with 128GB of RAM.
Atlas is a pure Rust LLM inference engine that delivers faster inference than vLLM and TensorRT-LLM by customizing CUDA kernels for each hardware × model × quantization combination.
A pull request has been merged into llama.cpp to add support for the Mimo v2.5 model, enhancing the framework's compatibility with this specific AI architecture.
ServiceNow engineers detail their migration from vLLM V0 to V1, focusing on resolving backend correctness issues like logprob semantics and runtime defaults to ensure stable reinforcement learning training dynamics.
Antirez released GGUF quantizations of DeepSeek V4 Flash specifically tailored for the DS4 inference engine, providing optimized configurations for different RAM sizes and enabling local execution of the large MoE model.
Developer wrote a Zig-based LLM inference engine from zero on macOS in 12 hours, boosting Qwen 3.5 0.8B throughput from 15 to 193 tokens per second.