Tag
Modal explains how it reduces AI inference cold starts by 40x using cloud buffers, a custom filesystem, checkpoint/restore, and CUDA checkpoint/restore, framing cloud buffer management as a linear optimization problem solved with GLOP.
Modal engineers detail their approach to achieving truly serverless GPUs for AI inference, combining cloud buffers, a custom content-addressed filesystem, and CPU/GPU checkpoint/restore to scale replicas in tens of seconds instead of minutes.