@lauriewired: How interesting. Samsung dropped a paper a few days ago with little fanfare: KV-cache offload over a CXL memory pool. R…
Summary
Samsung published a paper on offloading KV-cache over a CXL memory pool, demonstrating that even with early-gen CXL hardware, GPUs can be fed as effectively as with real DRAM, making the approach easily replicable.
View Cached Full Text
Cached at: 07/12/26, 11:02 PM
How interesting. Samsung dropped a paper a few days ago with little fanfare:
KV-cache offload over a CXL memory pool.
Really scrappy setup too. This is not future tech. Early-gen CXL stack (2.0), old switches, yet simple interleaving of the KV Cache across CXL modules kept the GPUs as well fed as real DRAM!
Not SOTA by any means, but should be easy to replicate!
of course, this (Jul 8th) whitepaper is in samsung’s best interest, fig 10 is a bit…misleading
(yes, 700gb of cache can’t fit in 512gb DRAM, of course having an additional CXL mem pool helps!)
however, it’s really showing some hints about how the most basic of software tricks can absorb much of the latency penalty, without major OS modifications
https://semiconductor.samsung.com/news-events/tech-blog/breaking-ai-memory-limits-with-cxl-memory-pooling/…
I wonder what the world would look like if IBM won instead of UNIX.
IBM’s i operating system is gorgeous (and still updated)! Everything is an object, no binaries are native (translated bytecode, almost JVM-y), and pointers all have an absurd amount of (useful) metadata.
Technically it hides a lot of the underlying system…you don’t even get to choose if an object lives on RAM or disk! But, interesting to think how much safer software would be.
Overengineered maybe, but beautiful. Kind of reminds me of Symbolics, like an alternate reality LISP machine where everything is an object.
Similar Articles
Xcena and Samsung's Near Memory Compute CXL Device
Xcena and Samsung announced MX1, a CXL memory expansion device with 3072 RISC-V cores for near-memory compute, designed to address memory needs for ML workloads at Hot Chips 2026.
@yukangchen_: We are excited to share a new technical article “KV Cache Compression and Its Infra Problems.” https://research.nvidia.…
NVIDIA Research publishes a technical blog post examining KV cache compression techniques and their infrastructure problems, including how FlashAttention and paged attention create practical obstacles for production deployment of long-context LLMs, with a proposed geometric solution using RoPE.
Maybe KV cache offload to RAM isn't bad
A user shares their experience offloading the KV cache to RAM in llama.cpp, achieving comparable speeds while freeing VRAM for larger models and context windows, suggesting this trade-off is often worthwhile.
@KVCache_AI: Mooncake now supports SSD Offloading for KV Cache. As agentic workloads become the norm, KV cache lifetimes are getting…
Mooncake announces support for SSD offloading of KV cache, enabling cost-effective scaling of KV cache capacity beyond DRAM for long-lived agentic workloads, with analysis showing bimodal reuse patterns that make tiered storage efficient.
@SKhynix: Meet CMM-Ax, @SKhynix's ASIC-based CXL-PNM solution developed with Marvell Technology. Designed to overcome memory bott…
SK Hynix introduces CMM-Ax, an ASIC-based CXL-PNM solution developed with Marvell Technology, designed to overcome memory bottlenecks in long-context LLM inference, achieving up to 5.5× higher throughput than GPU-only systems.