inference-engine

Flide is an efficient and exact inference engine for LLM decision models that decouples logical tree structures from a single flattened token stream via ForkAttention, answering queries up to 9.5x faster than vLLM and SGLang while staying closest to a float32 reference.

Kapitsa 0.2.26, an open-source Apple Silicon inference engine, publishes M5 Max benchmarks for Qwen, Nemotron, Flint and Clover models, featuring its KFlash best-first tree verification and SSD offloading that runs a 180B MoE at 61 tok/s on a 48GB MacBook Pro.

The author measures how lossy inference-engine flags (routing bias and tail skip) degrade model quality, showing that shipped defaults cost ~1.5 accuracy points with plenty of VRAM but ~3x more on small cards, while aggressive settings produce confident nonsense.

A detailed benchmark shows QEngine 1.3.1 (compiled for RTX 5090s) is the fastest Qwen3.8-Flash-Next engine for single-stream decode, ~1.4-1.9x faster than Strata and ~3x faster than tuned vLLM, with a KV-in-RAM mode that dramatically improves warm returns to idle conversations.

TryMirai open-sourced uzu, an Apple Silicon inference engine, featuring Weaver + DFlash speculative decoding that makes a 9B LLM (Qwen3.5) 4x faster locally and 2.6x faster than standard speculative decoding.

NInfer v0.3.0, an inference engine for running Qwen3.6/8-27B on a modded 22 GB RTX 2080 Ti, adds native Windows support and prebuilt releases, cutting binaries to 0.28 GB while delivering up to 47.5 tok/s decode with MTP speculative decoding.

作者测试了 Strata 0.2.7.1 的两项路由优化（STRATA_ROUTE_PRIOR 和 STRATA_ROUTE_TAIL_SKIP）在 Qwen 3.8 Flash Next IQ3_S 上对编码、数学和指令遵循任务的影响，发现 route prior 可大幅提升解码速度且质量损失在长推理下基本可忽略。

Magnitude is an open-source Rust LLM inference engine that auto-tunes GPU kernels on your local hardware, claiming 2.7x faster decode and 1.6x faster prefill than Ollama on a MacBook, with one-click integration into agents like Claude Code and Codex.

Delta Inference Engine 是一个 Windows 原生推理引擎，可在单张消费级 NVIDIA GPU 上以 2-bit 量化运行 5520 亿参数的 DeepSeek-V4.1-Flash-1017，RTX 3090 上生成速度达 41 tok/s，并提供 OpenAI 兼容 API。

OpenJet 的新版本在单张 RTX 3090 上通过自定义 CUDA megakernel 引擎并行服务最多 4 个编码代理，总吞吐达约 290 tok/s，是 llama-server --parallel 的 3-4 倍。核心思路是每步将所有活跃代理的工作堆叠成一个批处理，权重只读一次并用张量核同时处理所有行。

对 Qwen3.8-Flash-Next NVFP4 模型在单张 RTX 5090 上的三个推理引擎（basalt、capy-flash、ninfer-ext）进行全 262k 上下文基准测试，basalt 在所有上下文长度和两种解码模式下均领先（prefill 8.1k-9.2k tok/s，greedy decode 176-265 tok/s），解码排名与 MTP 草稿接受率一致。

Kapitsa, an open-source inference engine optimized for Apple Silicon, achieves 128 tok/s decode on a MacBook Pro M4 Pro 48GB running Recursive's new Flint 40B Q4 model, outperforming llama.cpp and Recursive's own engine in long-context decode and prefill benchmarks.

Nace.AI's Kapitsa 0.2 ships KFlash, a best-first tree-based speculative decoding method for Apple Silicon, benchmarked against Splash 1.4.0's DFlash2, mlx-serve, Ollama, llama.cpp and others on Qwen models across M4 Pro and M1 Pro MacBooks, showing top decode throughput (~65-201 tok/s).

Blockway 团队发布了 Volundr 32B 模型及自研 Agens Engine 推理引擎（MIT 开源），通过让 72 层中的 54 层使用固定大小状态替代增长的 KV cache，使 262K 长上下文能在 16 GB GPU 预算内运行，并演示了 110K token 冷读约 69 秒、后续追问 0.5 秒回答的实测。

Nace.ai 发布开源（Apache 2.0）Apple Silicon 优化 Metal 推理引擎 Kapitsa，专为 Qwen 和 Nemotron 模型定制内核，并在 M1 Pro、M4 Pro、M5 Max 上公布对比 MLX、llama.cpp、Ollama 等引擎的解码基准。

Blockway 发布 Agens Volundr 32B v1，修复 Preview 版的循环与推理截断问题，并开源 MIT 许可的 Agens Engine 推理引擎，其混合状态架构配合 2/3/4 混合量化可在 16 GB 显存预算内运行 262K 上下文（实测至 239K）。

Basalt inference tool adds native NVFP4 support and faster prefill for Qwen3.8 Flash-Next on a single RTX 5090, delivering up to 479 tok/s decode and 8,582 tok/s prefill with near-Q8 quality retention.

Xolotl 是一个基于 Strata 的推理引擎，可在单张 RTX 4090 消费级 PC 上以约 27 tok/s 运行 552B 的 DeepSeek-V4.1-Flash，并即将支持新的 1017 量化版本。

antirez's DwarfStar (ds4) project adds support for DeepSeek V4.1 Flash in PR #1225, expanding its specialized native inference engine for running open-weight models like DeepSeek, GLM, and Qwen on consumer hardware across Metal, CUDA, and ROCm.

capy-flash 是基于 ninfer-tp2 进化的本地推理引擎，可在两张 16GB 消费级 RTX 5070 Ti 上以 NVFP4 全精度量化运行 Qwen3.8-Flash-Next 125B-A6B（MoE 专家卸载至内存），生成速度约 100 tok/s，超过 Strata。作者还发现并修复了张量并行下双重 BF16 舍入导致的误差放大问题，改用 FP16 传输部分和即可获得与单 GPU 相当的数值精度。