@SuJinYan123: https://github.com/openinfer-project/openinfer/pull/522… openinfer p/d 分离支持，based on store（pegaflow p2p) based p/d pull…

X AI KOLs Timeline 07/03/26, 12:37 PM Tools

llm-inference open-source rust cuda framework p-d-separation

Summary

OpenInfer 0.2.0 released, supporting speculative decoding, p/d separation, and GLM5.2 FP8 inference; this project is a pure Rust + CUDA LLM inference engine with no dependencies on frameworks like PyTorch.

https://github.com/openinfer-project/openinfer/pull/522… OpenInfer p/d separation support, based on store (pegaflow p2p) based p/d pull mode. Afterwards, there may be a router option; tentatively using Dynamo, since it's also written in Rust, let's support it. Next step is to simply modify vLLM's pegaflow. This way, vLLM and OpenInfer can share a p/d protocol, meaning vLLM can be the P, OpenInfer the D, or vice versa. On the other hand, the store-based approach doesn't emphasize roles (like P, D — roles can be freely defined), but TTFT is not as good as the direct-based approach (but not terrible either). The best part of this architecture is truly pooled KV cache, fully managed by a single system. This brings up redundancy issues, which pegaflow might discuss in a future tech report. So OpenInfer 0.2.0: speculative decoding, p/d separation, GLM5.2 FP8 support!

Original Article

View Cached Full Text

Cached at: 07/03/26, 10:43 PM

openinfer

Pure Rust + CUDA LLM inference engine. No PyTorch. No model framework runtime.

Quickstart · Models · API · Performance · Architecture · Blog

Similar Articles

We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

Reddit r/LocalLLaMA

Paddock is an open-source Rust/C++ inference engine with custom CUDA kernels, demonstrating competitive performance against vLLM and SGLang in benchmarks while supporting OpenAI/Anthropic style APIs and GGUF/safetensors formats.

@SuJinYan123: Just 6 hours after DeepSeek open-sourced the Qwen DSpark weights, OpenInfer already has DSpark support running on RTX 5…

X AI KOLs Timeline

OpenInfer, a pure Rust+CUDA LLM inference engine, quickly added support for DeepSeek's DSpark speculative decoding technique on RTX 5090, achieving nearly 500 tok/s per user and scaling to ~2.4K aggregate tok/s, outperforming DFlash on non-random workloads.

Building a Rust Inference Engine That Matches Llama.cpp

Hacker News Top

Ferrox is a pure-Rust inference engine that loads GGUF models and runs local LLMs on CPU, Metal, or CUDA, with a CLI and an OpenAI-compatible server. It aims to match llama.cpp's performance while being written from scratch with no bindings.

@FakeMaidenMaker: Incredible! This open-source project can significantly speed up and save VRAM for self-hosted large model inference. It has garnered 9.2K stars on GitHub, joined the PyTorch Foundation, and NVIDIA's Dynamo has integrated it. GitHub: https://github.com/LMC…

X AI KOLs Timeline

LMCache is a KV cache management layer that accelerates large model inference and reduces VRAM consumption by caching and reusing KV cache. It has received 9.2K stars and joined the PyTorch Foundation, and is integrated by NVIDIA Dynamo.

Building Conifer, an open-source local inference runtime (free + open source)

Reddit r/artificial

Conifer is a new open-source local inference runtime built by a Princeton team, optimized for Apple Silicon with custom Rust kernels. It aims to outperform llama.cpp and MLX for small models and supports fully local agents with OS-level permission enforcement, now entering a limited 100-person beta.