HotKV cut Claude token use by 70% in our Banking77 benchmark

Hey everyone, I spent all my free time in the last year building HotKV. A multi-threaded, Redis-compatible in-memory store written from scratch in C++20, with a semantic cache for LLM responses built into the server. I benchmarked it on Claude first because this is where I expect the most useful feedback. Backstory, and how Claude Code was used A developer with over 20 years of commercial work. The engine design, the threading model and every architectural decision are mine, and I review, test and benchmark everything before it ships. Claude Code was the pair programmer on the rest, under my specifications: The benchmark behind this post: replay harness, lookup flow, token accounting, LLM judge pipeline A lock-contention fix in the vector search path, found while benchmarking against Redis 8 The MCP server that lets Claude Code read and write HotKV keys, prompts and agent memory, plus the LangGraph checkpointer and caching gateway SDK packaging for twelve languages, the docs site, the website, large parts of the cloud console Alone, next to my other work, that would have taken years. Used this way it is a tool, and the benchmark below is the check that tells a tool from "AI slop". The question How much does a cache in front of Claude save on realistic support traffic, and what does it cost in answer quality? Setup 3,000 support questions from Banking77 (PolyAI, CC BY 4.0), popular questions repeat the way real traffic does. Each request sent twice to each model: straight to Claude, and with HotKV in front Anthropic’s own prompt caching ON in both runs, so this is on top of it. Exact match first (hash of the normalized question), then semantic match (embedding cosine, threshold 0.81 fixed before the run). Only a miss on both calls the model. Results, 3,000 requests per model Model Calls Tokens Cost Fable 5.1 3,000 to 896 7.25M to 2.16M $30.31 to $9.18 Opus 5.5 3,000 to 898 7.34M to 2.20M $14.60 to $4.50 Sonnet 5.5 3,000 to 897 7.16M to 2.14M $5.51 to $1.66 Haiku 5.5 3,000 to 897 7.08M to 2.11M $0.27 to $0.08 About 70% fewer calls, tokens and dollars on every model: 39% exact repeats, 31% paraphrases. Median response fell from 1.4 to 2.9 s to about 127 ms. An exact hit takes 0.05 ms. The part cache vendors leave out A semantic hit means “a similar question was asked”, not “the same answer is right”. So an independent Claude model judged every semantic hit against a fresh answer to the exact question (no model graded its own output). At 0.81, cached answers fit 5.8 to 10.7 points less often than fresh ones. Sweeping the threshold on Sonnet 5.5: Threshold Saving Cached fits Fresh fits 0.81 70% 77.9% 86.1% 0.88 64% 89.5% 83.8% 0.90 61% 87.9% 86.5% From 0.88 up the gap disappears within the noise and you keep most of the saving. One model, one judge, one run per row, so a strong hint, not proof. We also built 119 deliberately nasty pairs (deposit vs withdraw, card vs bank transfer). At 0.81, 62% of the changed questions would still be served from cache. At 0.90, 25%. Raising the threshold helps, it does not make the problem go away. If you run Claude in production, what repeat rate do you see? Happy to answer anything about the method, and if we measured something wrong, say so. That is why the data is public.