In-house LLM Inference on Kubernetes: A Production Runbook
Summary
The author shares a runbook for deploying in-house LLM inference on Kubernetes, based on their experience building the infrastructure at their organization.
Similar Articles
Your LLM inference benchmark is lying to you
The article explains why synthetic benchmarks for LLM inference can be misleading, as production traffic is bursty and variable, and recommends testing with real workloads to choose the right inference framework.
An open handbook on LLM inference at scale (GPU internals, KV cache, batching, vLLM/SGLang/TensorRT-LLM) [P]
An open, in-progress handbook explaining LLM inference internals including GPU memory hierarchy, KV cache, batching, and popular inference engines like vLLM and TensorRT-LLM.
@h100envy: Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference op…
An ex-vLLM core contributor explains how to reduce LLM inference cost by 10x using LMCache with KV cache offloading to CPU/SSD/remote storage, a technique used by production stacks like Bloomberg.
@keigohtr: From Netflix. Their in-house LLM serving method. In-House LLM Serving at Netflix https://netflixtechblog.com/in-house-l…
Netflix shares their in-house LLM serving method using vLLM and Triton inference server, providing an OpenAI-compatible API and detailing deployment strategies and production trade-offs.
Local LLM Inference Optimization: The Complete Guide
A comprehensive guide to optimizing local LLM inference on consumer hardware, covering tools like llama.cpp, vLLM, and LM Studio, with practical advice on memory hierarchy, layer placement, and common failure modes.