Tag
Quail is an open source AI-SQL engine that integrates query planning with LLM inference to achieve over 1 billion input tokens per minute on an H100 GPU.
The article reports on running DeepSeek-V4-Flash-0731 on two Radeon AI PRO R9700 GPUs with the affinity inference engine, achieving 40-50 tok/s decode speed via a prebuilt quantization and stability fixes.
Gewell announces an inference engine designed for Gemma4 AI models to streamline deployment and execution.
The zero-to-sglang course has hit 1,000 GitHub stars, with new chapters released covering mini-sglang and the journey of a request including code walkthroughs.
Inco Splash is an open-source inference engine optimized for Apple silicon, offering significant speed improvements for running AI models like Qwen3.8-27B on M-series MacBooks.
Inco AI releases Splash, an open-source inference engine optimized for Apple silicon, claiming up to 3× faster decode speeds for local model serving, enabling agentic workloads on devices like M5 Max MacBook Pro.
The vLLM Conference concluded successfully with high community attendance, and session recordings are available in the thread.
Intel releases OpenVINO 2026.4 with expanded AI model support, performance enhancements like multi-token prediction, and new features for profiling and inference across CPUs, GPUs, and NPUs.
Edge0 is a streaming MoE inference engine that uses trained routing prediction to serve 35B Mixture-of-Experts models from SSD, achieving near-fp16 performance on consumer hardware with low memory usage.
A high-throughput inference engine for structured information extraction on Apple Silicon using MLX, offering parallel constrained decoding with 5.6x to 7.0x latency reductions and 100% schema validity.
SGLang is an open-source high-performance inference service framework from LMSYS, leveraging RadixAttention technology to achieve KV Cache reuse, supporting various hardware and models, and widely used in production environments.
Cherenkov is a new inference engine for Apple Silicon that enables efficient memory-constrained inference of large AI models like Qwen3.8-Flash-Next using predictive expert streaming and mixed-precision execution.
Cohere introduces a megakernel serving engine for North Mini Code that achieves 1.25× to 1.41× faster inference than vLLM on H100 GPUs by optimizing memory bandwidth for autoregressive decoding.
The author shares insights about the development time for inference engines they are working on.
The author describes consolidating four small AI models from separate services into a single server using Superlinked's inference engine to reduce operational overhead, while discussing trade-offs like GPU sharing and blast radius concerns.
Paddock is an open-source Rust/C++ inference engine with custom CUDA kernels, demonstrating competitive performance against vLLM and SGLang in benchmarks while supporting OpenAI/Anthropic style APIs and GGUF/safetensors formats.
Perplexity has open-sourced Lily, a local inference engine optimized for Qwen3.6-35B-A3B on Apple Silicon, achieving 1.35x faster inference than MLX-LM.
VoxGen is a new lightweight native inference engine for VoxCPM2 models, optimized for AMD cards using Rust and Vulkan compute to enhance performance and remove Python/PyTorch dependencies.
The user shares benchmark results showing vLLM's significantly faster prefill performance compared to llama.cpp and other engines, and questions the technical reasons behind this speed difference.
A developer forked NInfer, a C++20/CUDA inference engine, to add tensor-parallelism and YaRN rope scaling, enabling Qwen3.8-27B to run with a 1M token context on dual 5090 GPUs and outperforming vLLM in specific decode scenarios.