Tag
The author describes their agent workflow using asyncio.gather to batch vLLM API calls with shared prefixes and asks for best practices to optimize this process, proposing a pre-caching strategy for efficient decoding.
Henry Broomfield shares recent improvements in prompt caching at OpenAI, including tips to enhance cache hit rates for applications.
OpenAI's prompt cache reduces request costs by 90% but has a cache key limit of 15 requests per second. Unify GTM built a custom routing solution to bypass this limit, achieving a 95% cache hit rate.
An article discussing how prompt caching can significantly reduce LLM API costs, pointing out that providers under-explain it and offering a simple rule to structure prompts for maximum cache hits.
OpenAI details how WebSockets and API optimizations reduced latency by 40% for agentic workflows, enabling GPT-5.3-Codex-Spark to reach near 1,000 tokens per second.