caching

A curated list of resources for mastering LLM cache management, including explanations, tutorials, and research papers on KV cache, prefix caching, and related techniques.

LaCache introduces a training-free acceleration framework for diffusion-based LLMs that uses lossless caching and precision-adaptive inference to eliminate redundant computation during denoising steps, achieving up to 40.2× end-to-end speedup while maintaining task accuracy.

Proposes Cache-Aware Prompt Compression (CAPC), a method that combines query-agnostic compression with caching to reduce LLM API costs, demonstrating significant savings over existing approaches on Anthropic's Sonnet API and production workloads.

IBM Research explains why model routing in agentic systems is more complex than a simple classification problem, highlighting how caching and hidden factors like actual workload cost and task difficulty estimation make routing a systems optimization challenge.

Simon Willison shares a cache-friendly approach to using uvx in GitHub Actions by setting an environment variable and incorporating it into the cache key to avoid repeated PyPI downloads.

A study comparing Claude Code and OpenCode reveals that Claude Code sends 33k tokens before reading the prompt while OpenCode sends only 7k, highlighting significant inefficiency in Claude Code's cache strategy and token usage.

llama.cpp b9966 introduces a fix for the -sm tensor mode that caches regex patterns, eliminating 29 recompilations per tensor per token on the decode thread, resulting in significantly reduced CPU overhead.

Reame is an LLM inference server built on llama.cpp that optimizes for CPU hardware by caching prompt prefixes and generated n-grams, becoming faster with repeated use. It is designed for cheap hardware like shared vCPUs and free tiers, targeting repetitive AI workloads such as document extraction and batch pipelines.

Speculative cache warming pre-processes the system prompt and tools array while the user types their prompt, saving 10-20 seconds of wait time on local LLM inference. This feature is part of the open-source OpenFox harness for local AI, improving interactivity without breaking cache consistency.

Sohu has open-sourced its internal Redis cloud management platform, CacheCloud, which supports standalone, sentinel, and cluster modes. It offers one-click setup, monitoring alerts, elastic scaling, and more. The project has garnered nearly 9,000 stars on GitHub and is licensed under Apache-2.0.

A practitioner's guide to KV cache management, introducing the open-source LMCache architecture that cuts input token costs by 90% and speeds up LLM inference by up to 14x by eliminating redundant context processing in agentic workflows.

The article explains why false sharing alignment on x64 should be 128 bytes instead of the typical 64 bytes, due to Intel's Sandy Bridge spatial prefetcher that loads cache lines in pairs. It provides reasoning and a benchmark demonstrating the improvement.

This paper proposes a hybrid Mamba-attention architecture for block diffusion language models that restricts reverse Mamba scans to the active denoising block, enabling exact caching across blocks and achieving high throughput for long-context generation.

A comprehensive argument that PostgreSQL alone is sufficient for most application needs, including caching, search, job queues, and document storage, before reaching for additional specialized systems.

A detailed technical report on Hermes Mixture-of-Agents (MoA) findings, including benchmark results, caching economics, GPU topology studies, and a roadmap for future development.

The author shares findings from Hermes Mixture-of-Agents experiments, including voter upgrades, GPU topology, and caching economics, showing that local prefix caching can make long agent sessions nearly free and that two independent GPU instances outperform a single partitioned one.

An article explaining why FreeBSD appears to use a lot of RAM, attributing it to disk caching and virtual memory management, similar to Linux's 'ate my RAM' phenomenon.

The VS Code team details recent optimizations to GitHub Copilot's agentic harness, such as prompt caching and tool search, to improve token efficiency and reduce costs under usage-based billing.

Discusses a blind spot in AI agent reliability: cached facts that were true when ingested but become stale by the time they are used, leading to coherent but incorrect actions. Proposes separating consistency (match with source) from currency (source still true now), and asks how the community handles this.

The author details a fully local, no-cloud-API web research stack for AI agents, using self-hosted SearXNG, a persistent cache, TLS-fingerprinted fetching, headless browser fallback, and a local reranker, inviting community discussion on similar setups.