Tag
The author optimized a matmul kernel for BitNet's ternary models on CPU, achieving 29x speedup in isolation, but found that the model is memory-bound, resulting in only 6-10% end-to-end gain. The inference engine is available as open-source.
Project Zero is a from-scratch C99 LLM inference engine that runs BitNet and Qwen Bonsai-27B on CPU with zero dependencies, achieving 1.8× speedup over bitnet.cpp on Xeon. The project seeks community benchmarks for both models.
QuixiAI releases embeddinggemma.c, a fast cross-platform embedding inference engine written in C, supporting multiple backends (CPU, Metal, CUDA, ROCm, SYCL) and Matryoshka embeddings with a standard HTTP API.
A developer compares two inference stacks (production build vs SignalNine's q27) on the same Qwen model and finds they produce different honesty under pressure, with one fabricating progress and the other refusing appropriately, suggesting inference engines can affect model behavior beyond speed and quality metrics.
Moondream 3.1 is a vision language model with mixture-of-experts architecture (9B total parameters, 2B active), delivering state-of-the-art visual reasoning, detection, pointing, and captioning, deployable locally via the Photon inference engine or through the Moondream Cloud API.
The author shares their work over 6-8 months in ML systems and AI infrastructure, including a lightweight Python LLM inference engine (tachyon) that achieves 600+ tokens/s on consumer hardware with continuous batching and prefix caching, alongside blog posts on CUDA/CUTE DSL and collective communication, and contributions to SGLang and vLLM.
Lithos announces its inference engine serving Kimi K2.7 Code, achieving over 1,000 tokens/sec per user on a single 8×B200 node at native precision, 3.4–5.7× faster than major providers.
Colibri is an inference engine written in pure C, approximately 1300 lines of code, zero dependencies. It can run the 744-billion-parameter GLM-5.2 MoE model on a consumer machine with 25GB RAM, achieved by streaming loaded routing experts and efficient caching, no GPU or Python runtime needed.
TensorSharp is an open-source .NET library for running LLM inference locally, supporting GGUF models and offering a CLI, web chatbot, and OpenAI-compatible APIs with multiple backend options (CUDA, Metal, CPU).
Michael Goin announced removing PagedAttention from vLLM, a significant change to the open-source LLM inference engine.
Moondream's Photon inference engine eliminates GPU bubbles through pipelined decoding, achieving near-realtime VLM inference with up to 35% higher decode throughput.
OpenInfer, a pure Rust+CUDA LLM inference engine, quickly added support for DeepSeek's DSpark speculative decoding technique on RTX 5090, achieving nearly 500 tok/s per user and scaling to ~2.4K aggregate tok/s, outperforming DFlash on non-random workloads.
A minimal CPU-only inference engine for Qwen 3 models implemented from scratch in pure C.
Eyal Toledano built an LLM inference engine using pure WebGPU/WGSL, running on-device in browser and Node without API keys, and is seeking peer review.
Enze Xie announces Sol Video Inference Engine, an agent-native, training-free full-stack accelerator for video diffusion that auto-tunes cache, sparse attention, token pruning, quantization, and kernel fusion, achieving >2× end-to-end speedup on large models like 64B Cosmos3-Super and 22B LTX-2.3.
A detailed technical comparison of two dominant LLM serving frameworks, SGLang and vLLM, covering architectural differences in KV cache management (RadixAttention vs PagedAttention), throughput, latency, and deployment considerations for self-hosted environments.
DwarfStar is a self-contained native inference engine optimized for DeepSeek V4 Flash and PRO models, supporting Metal, CUDA, and ROCm backends, with a focus on high-end personal machines and Mac Studios.
Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs, achieving 5x cost cuts over DeepSeek's API; she also built FlexGen and helped build Chatbot Arena.
cuTile Rust introduces a tile-based programming model that leverages Rust's ownership to guarantee memory safety and data-race freedom for GPU kernels, and the Grout inference engine built on it achieves competitive throughput with vLLM/SGLang for Qwen3 models.
mistral.rs now natively supports Agent Skills, enabling locally-run small LLMs to perform complex agentic workflows for scientific tasks, with full control over models, data, and execution.