@h100envy: Ex-Berkeley PhD who leads SGLang at xAI explained how they serve Grok on 100K GPUs in 23 minutes - better than $2000 in…
Summary
A former Berkeley PhD who leads SGLang at xAI explains how they serve Grok on 100K GPUs using split prefill/decode, expert sharding, and communication/computation overlap to achieve DeepSeek-API-killing prices.
View Cached Full Text
Cached at: 07/06/26, 08:20 PM
Ex-Berkeley PhD who leads SGLang at xAI explained how they serve Grok on 100K GPUs in 23 minutes - better than $2000 inference-at-scale courses.
split prefill and decode -> shard experts across GPUs -> route tokens per expert -> overlap comm and compute -> serve at DeepSeek-API-killing prices.
That loop is why xAI runs Grok on SGLang and third parties beat DeepSeek’s own API by 5x on cost.
SGLang + prefill-decode disaggregation + expert parallelism + AMD MI300 - that’s the stack.
Watch and save it, then read the article below.
Similar Articles
@h100envy: Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs. She also built Fle…
Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs, achieving 5x cost cuts over DeepSeek's API; she also built FlexGen and helped build Chatbot Arena.
@VraserX: The xAI team really cooked with Grok 4.5. The price to performance ratio is honestly kind of insane. Getting frontier-l…
The tweet praises xAI's Grok 4.5 for its impressive price-to-performance ratio, making frontier-level AI inference more accessible.
@PyTorch: While SGLang provided Day-0 support for DeepSeek-V4, the collaboration between the @lmsysorg and @NVIDIAAI engineering …
SGLang provided Day-0 support for DeepSeek-V4, and collaboration between LMSys and NVIDIA engineering teams achieved up to 5x throughput increase in production, with improvements shown on the SemiAnalysis InferenceX dashboard.
@mark_k: Fascinating and very deep article about DeepSeek AI (@deepseek_ai). You would have never guessed what their strategy is…
An analysis of DeepSeek AI's unconventional strategy: prioritizing radical architecture innovations (MoE, MLA, engram, mHC) that drastically reduce compute and memory needs, enabling a long-term play to build a 10T Chinese AI hardware ecosystem and pursue a 1T valuation.
@Ex0byt: Update: the road to GLM-5.2: we're getting there, folks! non-quantized, non-pruned DeepSeek-v4-Flash. 11tok/s on a sing…
Update on running a non-quantized DeepSeek-v4-Flash model at 11 tok/s on a single DGX Spark using sglang inference and a custom mega-kernel, progressing towards GLM-5.2.