caching

Anyscale demonstrates a 20x speedup in cross-region training data reads by using Alluxio NVMe caching with Ray Data, showing warm cache reads drop from 4,241 to 208 seconds for 1TB.

An open-source tool called Headroom compresses AI agent context by up to 90% using a reversible Compress-Cache-Retrieve architecture, enabling models to retrieve original details on demand instead of discarding them permanently.

Corbenic AI claims to offer lossless KV cache reuse for LLMs, allowing stored model memory to be restored bit-for-bit across machines and GPU generations, verified via public checksums. The project includes an open-sourced small model trained for ~600 EUR to make the full pipeline inspectable.

Trellis introduces RadixAttention to optimize LLM inference prefill phase by caching prefix tokens using a radix tree, reducing redundant computation in chat and agentic sessions.

The article discusses a C++/WinRT pattern for caching the result of a Windows Runtime IAsyncOperation, including handling failures, so that multiple coroutines can share the cached result or exception.

This paper presents a stateful inference architecture for multi-agent tool calling that reuses KV cache across turns and employs speculative decoding, achieving 2.1x-4.2x speedup over vLLM and SGLang on agentic workflows.

DeepSeek releases a native coding agent called DeepSeek reasonix, featuring high caching and low cost.

A comparison between Redis and Memcached covering data structures, performance, scalability, and operational considerations to help choose the right caching solution.

New research by Joshua Gu shows that AI agents perform better when they manage a small buffer in their context window as a cache for external context, challenging the common practice of pushing context entirely out of the prompt.

A discussion on effective FinOps strategies for managing costs in large-scale AI agent operations, covering tactics like model routing, prompt trimming, caching, and the need to track cost by agent, workflow, and customer.

This paper introduces PEEK, a system that caches orientation knowledge about recurring external contexts as a context map, enabling LLM agents to reuse context knowledge across invocations and significantly improving efficiency and accuracy on long-context reasoning and information aggregation tasks.

The author describes using HAProxy caching to reduce unnecessary load on snac threads in the FediMeteo service, following previous similar optimizations with nginx. The approach aims to keep the lightweight ActivityPub server efficient by having the reverse proxy absorb repeated public requests.

A thread sharing a structured install order for agentic projects: using direnv with a secrets manager for credential safety, litellm or portkey as a model proxy for cost and fallback management, uv+git commits on passing evals for reproducibility, and mitmproxy for full observability of LLM calls. Highlights common failure modes and security gaps.

The article discusses how the KV cache is evolving into a memory hierarchy for LLM inference, optimizing memory management during decoding.

A practical blueprint for designing a backend system capable of handling 1 million concurrent users, covering architecture decisions like language selection, load balancing, database sharding, multi-layer caching, and resilience patterns.

LM Studio announces a beta update to its MLX engine, introducing batching for vision models and improved caching for faster inference.

LMCache is an open-source KV cache management layer for LLM inference that reduces time-to-first-token and improves throughput by enabling persistent storage and reuse of KV cache across serving engines.