caching

A tweet listing 16 inference optimization techniques for achieving sub-second LLM responses, including KV-caching, speculative decoding, FlashAttention, and various parallelism methods.

The article questions the necessity of service workers in many web applications, arguing that simpler HTTP caching often suffices and that service workers introduce complexity and risk of stale caches.

This article benchmarks six Go in-memory cache designs under different workloads, finding that a sharded map with 256 locks outperforms single mutex and RWMutex approaches, especially on multi-core systems.

Coinbase reportedly reduced AI token spend by half through smart routing to cheaper models like GLM 5.2 and Kimi 2.7 and implementing caching, highlighting a trend in AI cost optimization.

LMCache is an open-source library that makes KV cache persistent and shareable across requests, eliminating recomputation in RAG and multi-turn chat workloads, achieving up to 15x throughput gain and 3-10x reduction in time-to-first-token.

Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.

This guest post explores the proposed Cross-Origin Storage API to improve caching of AI model resources in Transformers.js, enabling efficient reuse across origins while maintaining privacy and integrity for in-browser inference.

A guide on avoiding rate limits and reducing costs when using the GLM 5.2 model, covering prompt batching, caching, free model alternatives, effort levels, context window management, and self-hosting.

Venkat explains that unoptimized CPU work in the hot path can severely impact inference performance, and introduces his PR to mooncake that adds a memory arena for lock-free, allocation-free operations, benefiting vLLM and SGL projects.

Codex skills optimized for DeepSeek V4 Pro, saves 60-80% tokens by freezing skill files and minimal output, with cross-conversation persistent memory capability.

RFC 10008 defines the new HTTP QUERY method, allowing safe, idempotent, and cacheable requests with a body, addressing limitations of GET and POST for RPC-style APIs.

This tweet introduces various development features provided by Cloudflare, including object storage R2, backend API Workers, AI gateway AI Gateway, containers, cache KV, database D1, and PostgreSQL connection HyperDrive, emphasizing their low cost, rich features, and generous free tier.

CacheRL trains small agent foundation models for multi-step tool-calling tasks, achieving 92% process accuracy (approaching GPT-5's 94%) with 100x less compute using cached rollouts and hybrid reward shaping, with innovations in knowledge transfer, cache-aware rewards, and iterative SFT/GRPO training.

Practical guide on optimizing costs in Microsoft Agent Framework by using a gateway for caching, context compression, and model routing, ensuring each step uses only the necessary intelligence.

The article describes how building an intelligent caching gateway (Hawiyat Composer) saved significant AI API costs by eliminating repeated token waste through exact-match caching, semantic caching, model routing, and local routing.

Discusses token waste in AI agent workflows due to repeated context, introduces an open-source proxy called Badgr-auto for deduplication, and asks the community how they handle the issue.

This paper accelerates the NeurASP neurosymbolic AI framework by implementing vectorization, batch processing, and caching, achieving multiple orders of magnitude speedup on larger tasks.

MinLiBuilds released a true fork feature, allowing users to create a brand new child window based on conversation history with Command+F in the terminal, supports multi-level fork, and claims to be smoother than Claude Code, with 100% cache and free.

A programmer used the caches of 3.9 million open DNS resolvers to store file fragments, creating an absurd open-source project called dnsfs. The files disappear as the caches expire.

Anyscale demonstrates a 20x speedup in cross-region training data reads by using Alluxio NVMe caching with Ray Data, showing warm cache reads drop from 4,241 to 208 seconds for 1TB.