@pochenai: Inspired by @percyliang's CS336, I built a static performance model for LLM inference — no dynamic batching/chunking, j…
Summary
Inspired by CS336, a static performance model for LLM inference provides analytical bounds for VRAM, time-to-first-token, and throughput, covering various configurations and calibrated against public benchmarks.
View Cached Full Text
Cached at: 08/30/26, 12:02 AM
Inspired by @percyliang’s CS336, I built a static performance model for LLM inference — no dynamic batching/chunking, just clean analytical bounds.
Pick model × GPU × batch × seq length × parallelism(DP/TP/EP/PP) → VRAM check + TTFT + TPOT + prefill/decode breakdown + throughput vs batch size.
Covers 5 KV cache variants (GQA/MLA/SSM/sliding window/linear attention), speculative decoding, multiple quant precisions.
Calibrated against ~100 public benchmarks (TRT-LLM, Splitwise, MLPerf, Koyeb…).
Demo: …https://llm-inference-calculator-delta.vercel.app Repo: https://github.com/pochenai/llm-inference-calculator…
Similar Articles
@polynoamial: https://x.com/polynoamial/status/2064210146558136827
This article argues that LLM benchmark performance is increasingly a function of test-time compute, and that current evaluation methods fail to capture capability improvements when controlling for inference budget. It advocates for plotting performance vs. tokens, cost, or time, and discusses implications for safety evaluations.
Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)
A deep dive into vLLM's architecture and components for high-throughput LLM inference, covering scheduling, paged attention, continuous batching, advanced features, scaling, serving, and benchmarking.
Local LLM CPU users... How long is it taking you to do anything?
A discussion about the performance of running large language models locally on CPU, especially with large context sizes, and the challenges of VRAM constraints.
@athleticKoder: A 1600-word note on how llm inference work: Covering: 1. Attention - the only place tokens interact 2. KV caching - why…
A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.
@Alacritic_Super: The biggest bottleneck in LLM inference isn't arithmetic but it's moving data. A single multiply-accumulate operation i…
An educational thread explaining that the main bottleneck in LLM inference is data movement, not computation, and highlighting techniques like quantization, KV cache optimization, and FlashAttention to reduce memory traffic.