caching

This paper introduces the Hydration Proxy Pattern, a three-tier architecture to manage conversational state for stateless LLM APIs, ensuring data sovereignty, optimizing caching, and improving semantic grounding in enterprise systems.

The author describes how a volatile request identifier in prompts disrupted prefix caching, causing increased costs and latency for high-volume users, and asks for validation methods for caching improvements.

aimake is an incremental build system for AI/ML pipelines that uses content-based fingerprints to rebuild only changed steps, optimizing expensive pipeline reruns.

El autor reporta los resultados de ejecutar un agente de IA durante 28 días sin alcanzar límites, procesando millones de tokens con alta tasa de caché y manteniendo objetivos verificables. Discute desafíos de ingeniería de contexto y pide comparaciones de ejecuciones similares.

The article explains why KV cache stores key and value vectors but not query vectors in autoregressive LLMs, detailing the inference process to optimize token generation.

Cloudflare optimized their 1.1.1.1 DNS cache to reduce memory usage by over 50%, saving 100 terabytes of memory while improving performance through refined Rust data structures.

The post critiques auto-routing companies in AI, emphasizing that frequent model switching invalidates prompt caches, resulting in higher token costs and inefficiencies.

A user clarifies that OMP has a 97.5% cache hit rate and discusses the use of self-hosted DeepSeek V4 Flash models, addressing concerns about caching performance in inference setups.

AI agents are consuming tokens at nearly 5 times the human rate, with usage exploding 14X since February, which may reduce the ability of routers like OpenRouter to leverage competition among model providers due to caching and task continuity.

The article discusses the overlooked timing gap in revoking access for AI agents, highlighting how cached credentials and side effects can lead to unintended actions after revocation, and proposes measuring this delay as a critical metric.

The user proposes an automatic caching mechanism for long-context sessions in llama.cpp to avoid repeated prefills after restarts, enhancing usability on slower hardware.

This paper presents a pre-registered negative result on training mixture-of-experts routers for cache locality against memory-bandwidth walls, showing that miss reduction trades off with language modeling quality despite training mechanisms.

This paper analyzes a one-year production trace from Chutes to study LLM serving workloads, revealing temporal evolution and user-model interactions to improve serving system benchmarking.

This paper introduces QV-PIC, a query-aware dual-resolution position-independent caching framework for efficient RAG serving, which improves F1 by 21.6 points over vanilla rendered-image PIC while reducing time-to-first-token by 83.8% relative to full prefill.

KGCache is an in-memory cache for one-hop knowledge graph neighborhoods that reduces redundant subgraph retrieval in KGQA systems with LLMs. Evaluated on WebQSP and CWQ, it achieves up to 1.91x faster KG retrieval and shows semantic caching further improves hit rates.

The article explains how to speed up running Haskell scripts with Magix on GitHub Actions by caching dependencies and compiled artifacts, reducing full build times from over 100 seconds to near-instant reruns.

The article argues that Nix evaluation is fundamentally a scheduling problem and introduces Evix, a library-first async Nix evaluation engine designed for persistent, structured evaluation results.

A technical essay exploring how the conventional web server deployment model (TLS termination, reverse proxying, static file serving, caching) breaks down for hobby-scale apps that must be easily self-hosted, leading to inefficiency and added complexity.

A llama.cpp PR adds heatmap-based caching of 'hot' MoE experts on the GPU, improving tok/s by ~1.7-2x for some models on 8GB VRAM, but with mixed results for others.

The article argues that model routing isn't the real problem to solve—token efficiency is. It advocates for a holistic closed-loop approach combining cheaper defaults, preference-aware routing, and better caching (e.g., Coinbase's 5%→60% cache hit improvement) to maximize useful intelligence per dollar.