CachyLLama is a fork of llama.cpp that adds a persistent SSD-backed KV cache and multi-tier caching, dramatically reducing prompt reprocessing time for long local-agent sessions on slower hardware.
I’m not affiliated with this project, but I’ve been running it recently and I’m surprised it hasn’t received more attention here: https://github.com/fewtarius/CachyLLama CachyLLama is a fork of llama.cpp focused on a problem that matters a lot on slower hardware: repeated prompt processing. Not only does it have a new "SSD" based cache, but it also has some other improvements with caching, like a multi-tier KV cache. My local models generate at an acceptable speed once they get going. The painful part is using an agentic coding harness that sends a large system prompt, tool definitions, and most of the conversation back to the server on every request. A long session can spend far more time reprocessing familiar context than generating the answer. CachyLLama adds persistent SSD-backed KV checkpoints and a system-prompt cache. When the beginning of a request matches previously processed context, it can restore that state and evaluate only the changed tail rather than starting over. The checkpoints can also survive a server restart. On my older dual-MI50 setup, this has made repeated requests in long agent sessions substantially more responsive. I have not produced a controlled benchmark yet, so consider this an operator report rather than a scientific result, but the practical difference has been very noticeable. The project’s own 7840U/780M benchmark reports: ~1,243-token prompt: 9.3s cold, 0.41s warm ~5,409-token prompt: 43.3s cold, 0.57s warm ~15,700-token prompt: 143.1s cold, 0.99s warm The important distinction is that this does not claim to make generation faster. It avoids repeating prompt-evaluation work that has already been done. It also contains handling for hybrid architectures such as Qwen 3.5/3.6, Gemma 4, and GLM-4.7, where restoring recurrent state is more complicated than restoring a conventional attention-only KV cache. Has anyone else here tried it? It's been really helpful for me but I haven't seen any mention of it anywhere else.
This article details a critical bug in llama-server where restored KV cache is immediately discarded due to missing checkpoint metadata, and presents a 117-line fix using a sidecar file to persist checkpoints across restarts, achieving a ~720x speedup in prefill time.
A setting in llama.cpp's webUI re-sends generated tokens to the KV cache to significantly reduce prompt processing latency, improving responsiveness for long generations or tool calls without apparent trade-offs.
A troubleshooting guide showing how changing OpenClaw's contextInjection setting from 'always' to 'continuation-skip' fixes prompt cache reloading on every turn when using llama-server, resulting in a 100x speed improvement for long sessions.
User seeks advice on preventing llama.cpp from offloading KV cache to swap before RAM is fully exhausted, sharing their configuration on an M2 Max with 96GB RAM and a large Qwen model.
A user shares their experience offloading the KV cache to RAM in llama.cpp, achieving comparable speeds while freeing VRAM for larger models and context windows, suggesting this trade-off is often worthwhile.