Tag
A detailed technical writeup on pairing an M5 Ultra 256 GB for prefill with an M3 Ultra 512 GB for decode, using disaggregated inference over Thunderbolt 5 RDMA with GLM-5.3-Flash on MLX, cutting time-to-first-token on long prompts from ~40s to ~18.5s.
A step-by-step, first-principles explainer on how prefill and decode actually work inside LLM inference, starting from how ChatGPT-style chats process prompts via tokenization and autoregressive token generation.
Eric Zhang shares a steady-state analysis arguing that PD disaggregation mainly improves mean per-token latency for prefill-heavy workloads (giving ~1/(decode-time fraction) interactivity gains) and is largely unnecessary for decode-bound, low-latency inference — validated in production, and echoed by Horace He's remark that few have really thought through PD disagg vs. mixed-chunk prefill.
ExFold is a unified training-free framework that accelerates MoE model inference by folding excluded expert contributions into retained experts, achieving up to 1.41× speedup while maintaining high quality.
Discusses the nuanced reality of prefill-decode disaggregation in LLM serving at scale, based on customer patterns and validated on AMD with vLLM.
ZCube is a new network architecture that flattens the topology and mixes single/multi-rail access to optimize KV Cache transmission in long-context and PD separation scenarios. In the GLM-5.1 production cluster, it achieved a 33% reduction in switch/optical module costs, a 15% increase in GPU inference throughput, and a 40.6% decrease in TTFT P99.
Researchers propose Prefill-as-a-Service (PrfaaS), a system that offloads long-context prefill to remote compute-dense clusters and streams KVCache over commodity Ethernet, enabling independent scaling and 32-54% higher throughput for a 1T-parameter hybrid model.