Tag
This article provides an in-depth interpretation of NVIDIA's newly released 'AI Model Co-Design' paper, pointing out that in AI inference scenarios, storage (memory bandwidth, weight reading) has replaced GPU compute as the primary bottleneck. It elaborates on the design strategies of TensorRT-LLM and Blackwell architecture around the Roofline model, emphasizing that reducing data movement is more critical than improving compute power.
Flash-KMeans is an IO-aware implementation of exact KMeans that redesigns the algorithm around modern GPU bottlenecks, achieving 33x speedup over cuML and 200x over FAISS by eliminating redundant memory reads and writes.
XCENA, a chip startup founded by Samsung and SK Hynix veterans, raised $135M to develop a memory-centric chip that handles AI inference tasks near DRAM, reducing costly data transfers between CPUs and GPUs. The company's MX1 chip is expected to improve efficiency and reduce infrastructure costs.
A hot take arguing that context windows are a distraction from the real problem of AI memory, which remains unsolved and leads to forgetting context and duplicating bad info.