@MaxForAI: I agree with Andrew — a major breakthrough in memory efficiency is coming, which is actually a direction that Infra has been working hard on for a long time. And there have already been significant results, such as cache hit. DeepSeek, as early as after the deployment of disk-based KV Cache, cut the input price for cache hits to 1/… of that for cache misses.
Summary
Technical experts discuss the upcoming major breakthrough in memory efficiency, mentioning that DeepSeek has reduced the input price for cache hits to 1/10 to 1/50 of the price for cache misses through KV cache optimization, and reveal that OpenAI engineers have used multiple optimization techniques to cut inference costs by more than half.
View Cached Full Text
Cached at: 07/01/26, 08:02 AM
I agree with Andrew — we’re on the verge of a major breakthrough in memory efficiency, something the infrastructure team has been working hard on for a long time.
And there have already been significant results, such as cache hits.
As early as after DeepSeek implemented hard drive–based KV Cache, they slashed the input price for cache hits to 1/10 to 1/50 of the miss price. This was essentially enabled by ongoing optimization in KV Cache — that’s the only reason they could price that aggressively. At the time, it was already a well-known precedent in the industry.
If the team Andrew mentioned — the one spun out of OpenAI — has truly made a new breakthrough in memory efficiency, they’ve likely pushed even further along this path. After all, making things cheap and user-friendly ultimately depends on the underlying architecture.
I also saw a report from The Information. One source revealed that earlier this month, OpenAI engineers told some colleagues that, by leveraging several newly developed optimization techniques, they had found a way to cut model inference costs by more than half.
When the engineers applied this new technology to ChatGPT scenarios accessed by visitors without free or paid accounts, they managed to reduce the number of required NVIDIA GPUs to just a few hundred.
Andrew Curran (@AndrewCurran_):
I’m posting this prediction now so I can quote it later. There has been a significant breakthrough in architecture - specifically around memory efficiency - not by one of the big labs, but by a team that was spun out of OpenAI (not SSI). They will probably announce it soon.
Similar Articles
@karminski3: Magic! DeepSeekV4 context memory compressed to 1/10! Everyone knows DeepSeekV4 supports 1M context and is heavily optimized. To actually use 1M context, VRAM usage is only about 10GB (compared to DeepSeek-V3.2 which needs about…
FlashMemory-DeepSeek-V4 proposes a novel inference paradigm called Lookahead Sparse Attention (LSA), which uses a neural memory indexer to actively predict future context needs, compressing physical KV cache usage to 13.5% of full context baseline while improving average accuracy by 0.6%. This method adopts a decoupled training strategy that allows independent training of the indexer without loading the base model, significantly reducing training cost.
@QingQ77: A terminal AI coding agent designed specifically for DeepSeek API prefix caching mechanism, maintaining ultra-low token costs in long sessions through a cache-first architecture. https://github.com/esengine/DeepSeek-Reasonix… Reaso…
Reasonix is a terminal AI coding agent designed specifically for DeepSeek API prefix caching mechanism, achieving ultra-low token costs in long sessions through a cache-first architecture. In testing, 435 million input tokens cost only about $12, with a cache hit rate of 99.82%.
@seclink: Chinese startup DeepSeek announced on Friday that its 75% discount on the DeepSeek-V4-Pro API will become permanent, with prices as low as $0.003625 per million cached input tokens and $0.87 per million output tokens—approximately 34 times cheaper than OpenAI's GPT-5.5. The model has 1.6 trillion...
DeepSeek has made its V4-Pro API price cut of 75% permanent, with per-million cached input tokens at just $0.003625 and output tokens at $0.87, about 34 times cheaper than OpenAI's GPT-5.5. The model has 1.6 trillion parameters but requires only 49 billion active parameters, supports a 1-million-token context, and leads in coding and reasoning benchmark tests.
@ZeroZ_JQ: https://x.com/ZeroZ_JQ/status/2066380476970103028
The article redefines KV Cache from an engineering perspective, pointing out that it is not just an inference optimization technique, but becomes a runtime infrastructure for reusing already computed results in the Agent era, helping AI avoid redundant reasoning.
@XAMTO_AI: One of the most draining aspects of AI development is this forced "context reset" — every time you switch environments, you have to re-explain what was done, where you were stuck, and the thought process at the time. To address this pain point, someone released Memanto, a working memory repository designed specifically for AI, currently supporting Claude...
Memanto is an active working memory repository designed for AI development environments. Without API keys or vector databases, it achieves zero-latency indexing and high-performance retrieval, supporting over 16 development environments including Claude Code, Codex, Cursor, LangGraph, CrewAI, and others. It scored 89.8% on the LongMemEval benchmark.