@maxxfuu: Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unopti…

X AI KOLs Timeline News
cuda kernel paged-attention kv-cache gpu-memory inference-engineering

Summary

A developer shares their day 6 of inference engineering, writing a CUDA kernel for 1D convolution, explaining PagedAttention's memory efficiency, and providing an overview of GPU memory hierarchy (global, register, local, constant, shared).

Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unoptimized boring but correct CUDA! I also read PagedAttention (2312.07104), watched 1 short lecture on GPU Memory, and hack squatted 455lb x6 and 495lb x3 as a top set! Heres what I've learned about PagedAttention: PagedAttention reduces internal fragmentation and solves external fragmentation. During the decode phase, the KV cache manager actually hands out one cache block at a time to store output tokens. If the output tokens overflows the cache block, the KV cache manager assigns a new cache block to hold the remaining tokens that didn't fit into the previous cache block. What ends up happening is that the VRAM is used effectively such that each cache block is filled before new cache block is allocated on the VRAM. In a naive KV cache implementation, memory is reserved up front for the maximum sequence length, which might not ever be used, meaning there is just a massive chunk of VRAM that's not touched. PagedAttention reduces internal fragmentation by ensuring memory is never wasted between each decode sequence. To make usage of memory even more efficient, any unique prompts that share the same prefix tokens, the KV cache manager allows them to share the same cache block, however, it writes to a new cache block starting from where the new tokens differ within the two prompts. While each decode phase effectively writes to random parts of the VRAM, the Block table is able to provide an abstraction that makes KV cache seem like a contiguous memory to the model. Making the allocation process as simple as checking if a cache block is full or not. This continuous practice of using cache blocks to allocate just enough memory and using a block table to allocate non-contiguous free scattered memory on the VRAM reduces internal fragmentation and solves external fragmentation. --- Watched a video on GPU Memory: https://youtube.com/watch?v=Zrbw0zajhJM&list=PL5XwKDZZlwaY7t0M5OLprpkJUIrF8Lc9j&index=7… I read this blog that made absolutely cemented everything, written by (@hamzaelshafie): https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/…
Original Article
View Cached Full Text

Cached at: 07/12/26, 12:57 PM

Day 6/90 of Inference Engineering

I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unoptimized boring but correct CUDA! I also read PagedAttention (2312.07104), watched 1 short lecture on GPU Memory, and hack squatted 455lb x6 and 495lb x3 as a top set!

Heres what I’ve learned about PagedAttention:

PagedAttention reduces internal fragmentation and solves external fragmentation.

During the decode phase, the KV cache manager actually hands out one cache block at a time to store output tokens. If the output tokens overflows the cache block, the KV cache manager assigns a new cache block to hold the remaining tokens that didn’t fit into the previous cache block. What ends up happening is that the VRAM is used effectively such that each cache block is filled before new cache block is allocated on the VRAM.

In a naive KV cache implementation, memory is reserved up front for the maximum sequence length, which might not ever be used, meaning there is just a massive chunk of VRAM that’s not touched.

PagedAttention reduces internal fragmentation by ensuring memory is never wasted between each decode sequence.

To make usage of memory even more efficient, any unique prompts that share the same prefix tokens, the KV cache manager allows them to share the same cache block, however, it writes to a new cache block starting from where the new tokens differ within the two prompts.

While each decode phase effectively writes to random parts of the VRAM, the Block table is able to provide an abstraction that makes KV cache seem like a contiguous memory to the model. Making the allocation process as simple as checking if a cache block is full or not.

This continuous practice of using cache blocks to allocate just enough memory and using a block table to allocate non-contiguous free scattered memory on the VRAM reduces internal fragmentation and solves external fragmentation.

Watched a video on GPU Memory: https://youtube.com/watch?v=Zrbw0zajhJM&list=PL5XwKDZZlwaY7t0M5OLprpkJUIrF8Lc9j&index=7…

I read this blog that made absolutely cemented everything, written by (@hamzaelshafie): https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/…

TL;DR: 本集简要介绍CUDA中的五种内存类型——全局、寄存器、本地、常量、共享——及其片内/片外位置、读写权限、作用域和性能特点，为后续深入优化做铺垫。

GPU内存层次概述

GPU芯片（片上）通过PCB连接到VRAM（片外），但部分内存直接集成在芯片内部，访问速度更快。理解片内与片外内存的差异是编写高性能CUDA代码的基础。

全局内存

  • 位置：片外（GPU的VRAM）
  • 权限：所有线程均可读写
  • 分配方式：cudaMalloc或创建全局变量
  • 特点：容量最大，速度最慢
  • 作用域：全局可见，生命周期由主机控制

寄存器

  • 位置：片内
  • 权限：每个线程私有
  • 分配方式：内核中声明的局部变量默认存于寄存器
  • 特点：速度极快，数量有限
  • 占用率影响：使用过多寄存器可能导致占用率下降（后续集数详解）
  • 溢出问题：当寄存器不足时，变量会被自动溢出到本地内存（片外，速度慢），应避免。可通过编译标志（如增加PTX详细输出）或cuobjdump查看寄存器及本地内存使用情况。

示例：使用255个寄存器后，变量开始溢出到本地内存，导致2040字节的本地内存读写操作。

本地内存

  • 位置：片外（名称易混淆，仅因对线程私有而叫“本地”）
  • 权限：线程私有
  • 特点：访问速度慢，是寄存器溢出的结果，应尽量避免
  • 检测：通过编译标志可查看本地内存访问量

常量内存

  • 位置：片外（但配有缓存，只读）
  • 容量：仅64KB
  • 权限：所有线程只读
  • 访问特性：线程束内不同线程访问不同地址时会串行化；若多个线程访问同一地址，性能可能优于全局内存
  • 声明方式：__constant__关键字，用cudaMemcpyToSymbol从CPU复制数据

共享内存

  • 位置：片内
  • 权限：同一线程块内的所有线程可读写
  • 分配方式：内核中数组声明加__shared__关键字
  • 特点：访问速度远快于全局内存，适合多线程频繁访问同一地址的场景

五种内存类型总结

内存类型位置读写作用域生命周期
寄存器片内可读写线程线程
共享内存片内可读写线程块线程块
本地内存片外可读写线程线程
全局内存片外可读写全部主机控制
常量内存片外只读全局主机控制

后续集数将深入每种内存的优化技巧。欢迎订阅以避免错过更新，点赞、评论、分享你的反馈。

Source: https://www.youtube.com/watch?v=Zrbw0zajhJM

Similar Articles