memory-bound

Tag

Cards List
#memory-bound

Why is Speculative Decoding Fast? (4 minute read)

TLDR AI ↗ · 2026-10-09 Cached

一篇技术博客深入解释了投机解码为什么快:并非因为总FLOPs减少(实际上反而增加),而是因为它改变了GPU的工作类型——在低批量场景下把原本空闲的算力从内存瓶颈(memory-bound)转为计算瓶颈(compute-bound),用单序列的'深度'换取了批量解码的'广度'。

0 favorites 0 likes
#memory-bound

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

arXiv cs.AI ↗ · 2026-08-26 Cached

This paper compares tensor parallelism and KV-cache compression techniques for memory-bound LLM serving, finding that compression is generally cheaper and discusses decision rules based on model size relative to device memory.

0 favorites 0 likes
#memory-bound

@akshay_pachaar: how do you know whether your GPU is compute-bound or memory-bound? here's a simple explanation: your model weights sit …

X AI KOLs Following ↗ · 2026-08-16 Cached

The article explains how to determine if a GPU workload is compute-bound or memory-bound by analyzing operations per byte fetched from HBM, using NVIDIA's H100 as an example, and discusses how batching and prompt length affect performance.

0 favorites 0 likes
#memory-bound

TIL Why my dual 5060 Ti setup refuses to go past 50% usage and no, it's not broken.

Reddit r/LocalLLaMA ↗ · 2026-07-22

An investigation into why dual RTX 5060 Ti GPUs max out at ~50% utilization when running large LLMs like Qwen 27B reveals that memory bandwidth is the bottleneck and layer-by-layer splitting causes idle time, making it a relay race rather than parallel computation.

0 favorites 0 likes
#memory-bound

@_avichawla: Prefill & decode in LLM inference. Have you ever noticed that the first token from an LLM always takes a moment to appe…

X AI KOLs Timeline ↗ · 2026-06-29 Cached

Explains the two phases of LLM inference - prefill and decode - detailing how GPU bottlenecks shift from compute-bound during prefill to memory-bound during decode, and the importance of KV caching.

0 favorites 0 likes
#memory-bound

@HanGuo97: LLM training is built on fast MatMuls. But many surrounding ops still run as memory-bound kernels. CODA reparameterizes…

X AI KOLs Following ↗ · 2026-05-21 Cached

CODA reparameterizes memory-bound operations in LLM training to fuse them into the matmul epilogue, achieving near state-of-the-art performance with LLM-generated kernels.

0 favorites 0 likes
#memory-bound

@polydao: This Stanford lecture on AI inference will teach you more about how LLMs work in production than most ML courses > Clau…

X AI KOLs Timeline ↗ · 2026-05-13

A Stanford lecture on AI inference emphasizes practical bottlenecks like KV-cache and techniques like speculative decoding and continuous batching, offering more real-world insight than typical ML courses.

0 favorites 0 likes
← Back to home

Submit Feedback