Tag
The discussion points out that DeepSeek's open-source DeepInfer is an inference framework deeply customized for DeepSeek 4.1F. Its KV cache is extremely optimized at just 890 bytes per token, meaning task scheduling never needs to account for VRAM capacity. In contrast, the Kimi series models have KV cache overhead 15–30x higher, resulting in poor inference performance and user experience.
Colibri 是一个纯 C 编写的本地推理引擎，利用专家路由机制（热数据常驻内存、专家按需从磁盘读取），让仅 8GB 内存的普通电脑即可运行 7440 亿参数的 GLM-5.2 等 MoE 大模型，无需 GPU，并兼容 OpenAI 与 Anthropic 接口。
Cascadia 论文展示了在11台 Intel Core Ultra X7 358H AI PC 上分布式常驻推理 975B 总参数（41B 激活参数）的 Inkling MoE 模型，通过自定义 MoE 引擎与 OpenVINO 融合算子实现 60.29 聚合 decode tokens/s 的性能。
Nokia Applied Research开源了AnyJev库，可将任意LLM转化为Jev风格的决策模型，通过旋转预算和闭式头拟合实现无微调的概率校准输出，大幅降低order-flip率并提升自动决策比例。
The article promotes a talk at the PyTorch Conference North America focused on making enterprise agentic inference production-ready using PyTorch and vLLM, covering ecosystem updates and registration details.
Nokia Applied Research has open-sourced WiSP, a vLLM plug-in for running MoE models on GPUs with insufficient VRAM by intelligently paging experts and reallocating memory with the KV cache, achieving up to 2.0× higher decode throughput.
Wafer outperformed Cerebras on latency for YC's AI Office Hours, with GLM-5.2 achieving 379 ms versus 674 ms for Gemma 4, resulting in 44% lower latency and increased user engagement.
Google Cloud and Inferact announce a partnership to optimize vLLM for TPU, making it a first-class citizen in the open model ecosystem for agentic production serving.
Nicolò Lucchesi will present at PyTorch Conference North America 2026 on the evolution of disaggregated serving in vLLM for hybrid models, collaborating with AWS and RedHat.
Nunchux AI has launched Modelverse, a multimodal generative AI inference service providing fast, affordable access to over 30 image, video, and avatar models through a single API.
Z.ai announced that it served the GLM-5.3-Flash model entirely on Chinese AI chips with per-token costs comparable to Nvidia GPUs, using a custom inference engine optimized for memory-constrained hardware.
Nari Labs details their method for achieving sub-50 ms response times in text-to-speech models, emphasizing low-latency, real-time multimodal inference.
Fireworks AI's inclusion in trending and fastest-growing categories highlights the growing importance of custom models and owning the AI intelligence stack, with broader trends in AI SaaS vendor growth.
FreeToken is an edge-native serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines, enabling efficient execution of models up to 753B on a single GPU.
NVIDIA showcases the high-throughput performance of serving the Qwen3-8B 2.4T parameter model on GB300 NVL72 hardware, achieving over 4k tokens per second per GPU.
Superlinked releases SIE, an open-source inference engine that serves 85+ models behind one API with on-demand loading and LRU eviction, cutting self-hosting GPU costs by ~75% for agent pipelines.
A technical guide demonstrating how to serve five specialized small models (SLM, OCR, NER, reranker, object detector) on a single GPU using open-source tools, covering memory management, batching, and the Superlinked Inference Engine.
Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.
This article compares serverless, on-prem, and edge deployment for AI models, highlighting inefficiencies in current multi-model serving. It introduces the Superlinked Inference Engine (SIE), an open-source tool that serves multiple models on a single GPU by dynamically loading and unloading weights, aiming to reduce costs and complexity.
BentoML is an open-source Python framework that simplifies packaging, serving, and deploying AI models as REST APIs with containerization and multi-model orchestration.