Tag
The article presents the official home for llama.cpp, an open-source local LLM inference engine, highlighting integration with the Pi coding agent via the pi-llama plugin and broad hardware optimization.
Antirez's h3.c is a native Minimal inference engine for MiniMax-H3 on Apple Silicon, providing a fast, end-to-end prompt-to-video/audio pipeline with Metal optimizations and an interactive session. It is currently focused on performance and memory optimization for M3 Max and M5 Max.
A developer is building Octane, a speculative decoding runtime for local LLM inference on consumer hardware, aiming for 2-3x speedup with exact output quality. Currently in active development with paged KV cache, continuous batching, and batched attention implemented.
Superlinked releases SIE, an open-source inference engine that serves 85+ models behind one API with on-demand loading and LRU eviction, cutting self-hosting GPU costs by ~75% for agent pipelines.
Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.
Photon 2.0 is a new inference engine and compiler that compiles models like Moondream, Qwen 3.5, and Gemma 4 into megakernels, claiming up to 2.3x throughput over vLLM and SGLang for physical AI workloads.
LocalAI explains why it writes its own C/C++ inference backends, showing that its vllm.cpp port achieves comparable or better throughput and far smaller footprint than vLLM, with benchmarks across multiple models and hardware.
WASTE is a new open-source C inference engine that streams expert weights from disk to run the 2.78-trillion-parameter Kimi K3 model on a consumer laptop with just 29 GB of RAM, achieving 0.49–0.54 tokens/s.
A developer trained a tiny Mamba-based autoregressive language model and built an inference engine to run it on the 6502 processor inside a BBC Micro, demonstrating modern machine learning on 1975 hardware with only 25KB of user memory.
WASTE is an open-source inference engine that runs models larger than available RAM by storing expert weights on NVMe. It demonstrates running Kimi K3, a 2.78T parameter MoE model, on a MacBook Pro with 64GB unified memory.
Kimi K3, a 2.8-trillion-parameter multimodal MoE model with 1M context window, now has day-0 vLLM support. This guide details how vLLM serves K3's novel architecture including Kimi Delta Attention, Attention Residuals, and speculative decoding achieving up to 370 tok/s.
This article compares serverless, on-prem, and edge deployment for AI models, highlighting inefficiencies in current multi-model serving. It introduces the Superlinked Inference Engine (SIE), an open-source tool that serves multiple models on a single GPU by dynamically loading and unloading weights, aiming to reduce costs and complexity.
RIS-Kernel introduces a model-agnostic sparse attention architecture (RIS) that reduces self-attention complexity from O(N^2) to O(N log N) for long-context LLM inference, enabling operation on commodity CPU hardware without GPU acceleration.
The author optimized a matmul kernel for BitNet's ternary models on CPU, achieving 29x speedup in isolation, but found that the model is memory-bound, resulting in only 6-10% end-to-end gain. The inference engine is available as open-source.
Project Zero is a from-scratch C99 LLM inference engine that runs BitNet and Qwen Bonsai-27B on CPU with zero dependencies, achieving 1.8× speedup over bitnet.cpp on Xeon. The project seeks community benchmarks for both models.
QuixiAI releases embeddinggemma.c, a fast cross-platform embedding inference engine written in C, supporting multiple backends (CPU, Metal, CUDA, ROCm, SYCL) and Matryoshka embeddings with a standard HTTP API.
A developer compares two inference stacks (production build vs SignalNine's q27) on the same Qwen model and finds they produce different honesty under pressure, with one fabricating progress and the other refusing appropriately, suggesting inference engines can affect model behavior beyond speed and quality metrics.
Moondream 3.1 is a vision language model with mixture-of-experts architecture (9B total parameters, 2B active), delivering state-of-the-art visual reasoning, detection, pointing, and captioning, deployable locally via the Photon inference engine or through the Moondream Cloud API.
The author shares their work over 6-8 months in ML systems and AI infrastructure, including a lightweight Python LLM inference engine (tachyon) that achieves 600+ tokens/s on consumer hardware with continuous batching and prefix caching, alongside blog posts on CUDA/CUTE DSL and collective communication, and contributions to SGLang and vLLM.
Lithos announces its inference engine serving Kimi K2.7 Code, achieving over 1,000 tokens/sec per user on a single 8×B200 node at native precision, 3.4–5.7× faster than major providers.