CachyLLama is a fork of llama.cpp that adds a persistent SSD-backed KV cache and multi-tier caching, dramatically reducing prompt reprocessing time for long local-agent sessions on slower hardware.
I’m not affiliated with this project, but I’ve been running it recently and I’m surprised it hasn’t received more attention here: https://github.com/fewtarius/CachyLLama CachyLLama is a fork of llama.cpp focused on a problem that matters a lot on slower hardware: repeated prompt processing. Not only does it have a new "SSD" based cache, but it also has some other improvements with caching, like a multi-tier KV cache. My local models generate at an acceptable speed once they get going. The painful part is using an agentic coding harness that sends a large system prompt, tool definitions, and most of the conversation back to the server on every request. A long session can spend far more time reprocessing familiar context than generating the answer. CachyLLama adds persistent SSD-backed KV checkpoints and a system-prompt cache. When the beginning of a request matches previously processed context, it can restore that state and evaluate only the changed tail rather than starting over. The checkpoints can also survive a server restart. On my older dual-MI50 setup, this has made repeated requests in long agent sessions substantially more responsive. I have not produced a controlled benchmark yet, so consider this an operator report rather than a scientific result, but the practical difference has been very noticeable. The project’s own 7840U/780M benchmark reports: ~1,243-token prompt: 9.3s cold, 0.41s warm ~5,409-token prompt: 43.3s cold, 0.57s warm ~15,700-token prompt: 143.1s cold, 0.99s warm The important distinction is that this does not claim to make generation faster. It avoids repeating prompt-evaluation work that has already been done. It also contains handling for hybrid architectures such as Qwen 3.5/3.6, Gemma 4, and GLM-4.7, where restoring recurrent state is more complicated than restoring a conventional attention-only KV cache. Has anyone else here tried it? It's been really helpful for me but I haven't seen any mention of it anywhere else.
This article details a critical bug in llama-server where restored KV cache is immediately discarded due to missing checkpoint metadata, and presents a 117-line fix using a sidecar file to persist checkpoints across restarts, achieving a ~720x speedup in prefill time.
A setting in llama.cpp's webUI re-sends generated tokens to the KV cache to significantly reduce prompt processing latency, improving responsiveness for long generations or tool calls without apparent trade-offs.
The user proposes an automatic caching mechanism for long-context sessions in llama.cpp to avoid repeated prefills after restarts, enhancing usability on slower hardware.
A troubleshooting guide showing how changing OpenClaw's contextInjection setting from 'always' to 'continuation-skip' fixes prompt cache reloading on every turn when using llama-server, resulting in a 100x speed improvement for long sessions.
ReCache is a framework for efficient KV cache reuse and compression in tool-augmented LLM agents, achieving significant speedup and memory reduction while maintaining performance.