Tag
Nunchux AI has launched Modelverse, a multimodal generative AI inference service providing fast, affordable access to over 30 image, video, and avatar models through a single API.
Z.ai announced that it served the GLM-5.3-Flash model entirely on Chinese AI chips with per-token costs comparable to Nvidia GPUs, using a custom inference engine optimized for memory-constrained hardware.
Nari Labs details their method for achieving sub-50 ms response times in text-to-speech models, emphasizing low-latency, real-time multimodal inference.
Fireworks AI's inclusion in trending and fastest-growing categories highlights the growing importance of custom models and owning the AI intelligence stack, with broader trends in AI SaaS vendor growth.
FreeToken is an edge-native serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines, enabling efficient execution of models up to 753B on a single GPU.
NVIDIA showcases the high-throughput performance of serving the Qwen3-8B 2.4T parameter model on GB300 NVL72 hardware, achieving over 4k tokens per second per GPU.
Superlinked releases SIE, an open-source inference engine that serves 85+ models behind one API with on-demand loading and LRU eviction, cutting self-hosting GPU costs by ~75% for agent pipelines.
A technical guide demonstrating how to serve five specialized small models (SLM, OCR, NER, reranker, object detector) on a single GPU using open-source tools, covering memory management, batching, and the Superlinked Inference Engine.
Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.
This article compares serverless, on-prem, and edge deployment for AI models, highlighting inefficiencies in current multi-model serving. It introduces the Superlinked Inference Engine (SIE), an open-source tool that serves multiple models on a single GPU by dynamically loading and unloading weights, aiming to reduce costs and complexity.
BentoML is an open-source Python framework that simplifies packaging, serving, and deploying AI models as REST APIs with containerization and multi-model orchestration.
MiaAI Lab released an upgraded recipe for serving DeepSeek V4 Flash on two DGX Spark nodes using vLLM with DSpark speculative decoding and NVFP4 KV-cache, achieving up to 153.7 tokens per second with six concurrent sessions.
This paper introduces Director, a distributed MoE serving system that minimizes end-to-end latency using prediction-driven, online proactive expert placement. It employs a lightweight predictor and a relaxation-based optimizer to achieve up to 55% latency reduction for models like Mistral, DeepSeek, and Qwen.
Hugging Face announces the transformers vLLM modeling backend achieving native or faster speeds compared to custom vLLM implementations, allowing model authors to automatically leverage ultra-fast inference without porting code.
The user asks why DeepSeek v4 Flash (284B parameters) is so cheap to run compared to smaller models like Qwen 27B, questioning if it's due to pricing dumping or architectural differences. The answer likely involves its MoE architecture and efficient inference techniques.
A tweet listing essential skills for AI infrastructure engineers, covering GPU fundamentals, inference optimization, distributed training, and production deployment.
A user asks about hardware requirements for serving GLM-5.2 in NVFP4 format, which vLLM now supports with reduced memory footprint and maintained accuracy.
A turnkey Docker setup to serve the GLM-5.2-NVFP4-REAP-469B model on 4× RTX PRO 6000 Blackwell GPUs using vLLM, with detailed instructions and configuration options.
This project decouples Alibaba DAMO Academy's ZipEnhancer noise reduction model from the ModelScope pipeline, rewrites the inference logic in pure PyTorch, and packages it as a FastAPI service. It supports FP16 half-precision and long audio segmentation, providing multiple noise reduction model switching and API interfaces.
Luce Spark is an open-source tool that enables running 35B MoE models on 16GB GPUs by intelligently caching hot experts on the GPU while keeping the rest in system RAM, using a calibrated placement and bounded async cache to maintain high throughput without the usual offload speed cliff.