@LangChain: OpenAI's prompt cache makes a request 90% cheaper, but the cache key tops out around 15 requests per second. @HeggieCon…
Summary
OpenAI's prompt cache reduces request costs by 90% but has a cache key limit of 15 requests per second. Unify GTM built a custom routing solution to bypass this limit, achieving a 95% cache hit rate.
View Cached Full Text
Cached at: 09/02/26, 06:22 AM
OpenAI’s prompt cache makes a request 90% cheaper, but the cache key tops out around 15 requests per second. @HeggieConnor on how @unifygtm built its own routing around that limit, landing them close to a 95% cache hit rate. https://t.co/wLk0EaIiVb
Similar Articles
Prompt Caching in the API
OpenAI introduces Prompt Caching, an automatic feature that reduces API costs by 50% and improves latency by reusing recently cached input tokens on GPT-4o, GPT-4o mini, o1-preview, and o1-mini models. The feature automatically applies to prompts longer than 1,024 tokens without requiring developer integration changes.
Your Agentic Workflow's Cache Keepalive Costs 8x Too Much
A detailed measurement study across Anthropic, OpenAI, Gemini, and DeepSeek finds that the conventional 30-second prompt cache keepalive is 8x too frequent; a 4-minute interval is optimal, and only Anthropic's cache saves money at long idle gaps.
Does prompt caching actually save you meaningful money on AI agents?
A practical discussion questioning whether prompt caching delivers meaningful cost savings for AI agents in production, examining real-world factors like cache hit rates, routing strategies, and scale.
Stop shortening your prompts. Six agents, 97-99% cache hit rate - and why the standard advice is backwards.
The article argues that with prompt caching, longer, stable prompts can be cheaper than frequently changing short ones, sharing insights from running AI agents with high cache hit rates.
prompt caching, but for rl training - 7.5x speedup on long-prompt/short-response workloads
A new optimization technique for open-source RL training engines introduces prompt caching during training, achieving up to 7.5x speedup on long-prompt, short-response workloads by reducing redundant compute.