Tag
A developer forked NInfer, a C++20/CUDA inference engine, to add tensor-parallelism and YaRN rope scaling, enabling Qwen3.8-27B to run with a 1M token context on dual 5090 GPUs and outperforming vLLM in specific decode scenarios.
The article details progress in achieving over 50 tokens per second inference speed for DeepSeek-V4-Flash on M3 Ultra using the DwarfStar inference engine, with improvements in performance and quality metrics.
The article explores how malicious LLMs could exploit vulnerabilities in inference engines like vLLM to execute arbitrary code and control host machines, citing real-world CVEs.
FreeToken is an open-source local inference engine designed to run large Mixture-of-Experts (MoE) models on consumer-grade computers, offering significantly faster performance than alternatives like Ollama with easy installation and native GUI.
Sharp v22.1 template reduces Qwen model output tokens by 42% without speed loss, integrated into an NInfer fork for 5090 via C++ modifications.
Open-source inference engine FreeToken integrates CPU, GPU, and memory, enabling consumer hardware like 8GB VRAM laptops to run 35B MoE models, and home single-GPU gaming laptops to run 290B+ models, completely solving the VRAM limitation.
The article presents the official home for llama.cpp, an open-source local LLM inference engine, highlighting integration with the Pi coding agent via the pi-llama plugin and broad hardware optimization.
Antirez's h3.c is a native Minimal inference engine for MiniMax-H3 on Apple Silicon, providing a fast, end-to-end prompt-to-video/audio pipeline with Metal optimizations and an interactive session. It is currently focused on performance and memory optimization for M3 Max and M5 Max.
Redis creator antirez releases h3.c, a pure C + Metal inference engine for MiniMax H3 that runs natively on Apple Silicon, enabling text-to-video/audio without Python, PyTorch, or ComfyUI.
A developer is building Octane, a speculative decoding runtime for local LLM inference on consumer hardware, aiming for 2-3x speedup with exact output quality. Currently in active development with paged KV cache, continuous batching, and batched attention implemented.
Superlinked releases SIE, an open-source inference engine that serves 85+ models behind one API with on-demand loading and LRU eviction, cutting self-hosting GPU costs by ~75% for agent pipelines.
Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.
Photon 2.0 is a new inference engine and compiler that compiles models like Moondream, Qwen 3.5, and Gemma 4 into megakernels, claiming up to 2.3x throughput over vLLM and SGLang for physical AI workloads.
LocalAI explains why it writes its own C/C++ inference backends, showing that its vllm.cpp port achieves comparable or better throughput and far smaller footprint than vLLM, with benchmarks across multiple models and hardware.
WASTE is a new open-source C inference engine that streams expert weights from disk to run the 2.78-trillion-parameter Kimi K3 model on a consumer laptop with just 29 GB of RAM, achieving 0.49–0.54 tokens/s.
A developer trained a tiny Mamba-based autoregressive language model and built an inference engine to run it on the 6502 processor inside a BBC Micro, demonstrating modern machine learning on 1975 hardware with only 25KB of user memory.
WASTE is an open-source inference engine that runs models larger than available RAM by storing expert weights on NVMe. It demonstrates running Kimi K3, a 2.78T parameter MoE model, on a MacBook Pro with 64GB unified memory.
Kimi K3, a 2.8-trillion-parameter multimodal MoE model with 1M context window, now has day-0 vLLM support. This guide details how vLLM serves K3's novel architecture including Kimi Delta Attention, Attention Residuals, and speculative decoding achieving up to 370 tok/s.
This article compares serverless, on-prem, and edge deployment for AI models, highlighting inefficiencies in current multi-model serving. It introduces the Superlinked Inference Engine (SIE), an open-source tool that serves multiple models on a single GPU by dynamically loading and unloading weights, aiming to reduce costs and complexity.
RIS-Kernel introduces a model-agnostic sparse attention architecture (RIS) that reduces self-attention complexity from O(N^2) to O(N log N) for long-context LLM inference, enabling operation on commodity CPU hardware without GPU acceleration.