How do you deal with long-context sessions after restarting llama.cpp?
Summary
The user proposes an automatic caching mechanism for long-context sessions in llama.cpp to avoid repeated prefills after restarts, enhancing usability on slower hardware.
Similar Articles
CachyLLama’s: llama.cpp fork with persistent KV cache that makes long local-agent sessions much less painful
CachyLLama is a fork of llama.cpp that adds a persistent SSD-backed KV cache and multi-tier caching, dramatically reducing prompt reprocessing time for long local-agent sessions on slower hardware.
Self-hosting with llama-server? Fix for prompt cache reloading on every turn
A troubleshooting guide showing how changing OpenClaw's contextInjection setting from 'always' to 'continuation-skip' fixes prompt cache reloading on every turn when using llama-server, resulting in a 100x speed improvement for long sessions.
Maybe dumb question, but how do you serve multiple users with the full context length?
A user asks how llama.cpp can serve multiple users each with full context length, noting that it seems to only share the context pool rather than providing dedicated context per user.
llama.cpp has a clever trick for speeding up KV cache decode
A setting in llama.cpp's webUI re-sends generated tokens to the KV cache to significantly reduce prompt processing latency, improving responsiveness for long generations or tool calls without apparent trade-offs.
How do you keep long sessions from eating the whole context window?
A user shares a custom Plugin SDK hook that gradually compresses older turns while keeping recent ones raw to prevent context window exhaustion in long OpenClaw sessions, reducing re-sent context by 80%.