Tag
This paper introduces Director, a distributed MoE serving system that minimizes end-to-end latency using prediction-driven, online proactive expert placement. It employs a lightweight predictor and a relaxation-based optimizer to achieve up to 55% latency reduction for models like Mistral, DeepSeek, and Qwen.
Hugging Face announces the transformers vLLM modeling backend achieving native or faster speeds compared to custom vLLM implementations, allowing model authors to automatically leverage ultra-fast inference without porting code.
The user asks why DeepSeek v4 Flash (284B parameters) is so cheap to run compared to smaller models like Qwen 27B, questioning if it's due to pricing dumping or architectural differences. The answer likely involves its MoE architecture and efficient inference techniques.
A tweet listing essential skills for AI infrastructure engineers, covering GPU fundamentals, inference optimization, distributed training, and production deployment.
A user asks about hardware requirements for serving GLM-5.2 in NVFP4 format, which vLLM now supports with reduced memory footprint and maintained accuracy.
A turnkey Docker setup to serve the GLM-5.2-NVFP4-REAP-469B model on 4× RTX PRO 6000 Blackwell GPUs using vLLM, with detailed instructions and configuration options.
This project decouples Alibaba DAMO Academy's ZipEnhancer noise reduction model from the ModelScope pipeline, rewrites the inference logic in pure PyTorch, and packages it as a FastAPI service. It supports FP16 half-precision and long audio segmentation, providing multiple noise reduction model switching and API interfaces.
Luce Spark is an open-source tool that enables running 35B MoE models on 16GB GPUs by intelligently caching hot experts on the GPU while keeping the rest in system RAM, using a calibrated placement and bounded async cache to maintain high throughput without the usual offload speed cliff.
A practical guide explaining how to calculate VRAM requirements for LLMs based on parameter count and quantization level, plus additional overhead from KV cache, activations, and batching.
The article highlights that the same AI model can exhibit different behaviors depending on the inference stack (e.g., scheduling, quantization, speculative decoding), especially in long sessions or agent workflows, making the serving method nearly as important as the model itself.
SIE is an open-source inference engine that serves over 100 AI models through a single API for agent tasks, featuring on-demand loading and integrations with popular frameworks like LangChain and LlamaIndex.