Pi published a blog post detailing their compaction mechanism design, providing an in-depth analysis of how context bloats, how compression is triggered, and the costs of caching. https://t.co/WTcRwcJKza

Understanding Pi’s Context Compression Mechanism

Anyone who has used a Coding Agent for an extended period encounters the same problem: the context grows and grows, and performance degrades. Every file the model reads, every search it runs, and every code modification it makes—processes and results alike—are appended to the conversation history. Each subsequent request carries this full history.

Lengthening context first degrades quality. While models score near-perfect on benchmarks like Needle-in-a-Haystack (NIAH)，and long-context capability seems solved，Chroma’s “context rot” research indicates such tests only assess literal retrieval—hiding a known sentence within irrelevant text and asking the model to find it. Real-world tasks, however，require semantic understanding and reasoning. When Chroma fixed task difficulty and only increased input length, the performance of all 18 models tested declined as input grew longer—and the decline was uneven.

Two conclusions from their research are particularly critical for the Coding Agent scenario:

• The first is distraction: content related to the topic but unable to answer the question. Even adding just one such item causes noticeable performance degradation，which worsens as input length increases. Long conversation histories are full of these semi-relevant fragments.

• The second is the comparison from LongMemEval: models perform well when given only relevant fragments，but performance drops significantly with the full 110，000-token history. This forces the model to do two things simultaneously in one turn: first, retrieve relevant parts from the long history，then reason based on what’s found. Retrieval and reasoning compete for the same processing space.

Just a few days ago, Pi published a blog post detailing their compaction mechanism design，providing an in-depth analysis of how context bloats, how compression is triggered, and the costs of caching.

How Context Bloats Step by Step

Each request Pi sends to the model includes the system prompt，AGENTS.md，tool definitions, and the full conversation history. Consider a concrete example: you ask the model to fix a bug. It first calls the read tool to view the file. After the Agent executes，the file content is appended to the request and sent again. After reading，the model might call grep. After a few back-and-forth turns, it finally provides an answer. Each request in a turn carries everything prior—your file contents，search results，and the model’s own intermediate thoughts.

History only grows；it never shrinks. Over time，it becomes more expensive and slower. Eventually，the next request will exceed the window limit and return an error like “Request exceeds the maximum size.” At that point，the conversation can no longer proceed.

Two Options After Context Is Full

One is to start a new session. History is cleared；previous decisions and unfinished tasks are all discarded. This sounds like a significant loss，but given the “context rot” mentioned earlier，a bad context is more harmful to the model’s state than no context at all. Sometimes, discarding history is actually the more correct choice.

The other is compaction: handing the old history to an LLM request to compress it into a summary. This summary replaces the original content, freeing up space for new messages and allowing the session to continue.

Pi’s Compression Workflow

Pi checks context usage after each turn. When it approaches the window limit，compression is triggered automatically. Manually typing /compact can also trigger it at any time. There’s also a fallback mechanism: if an overflow error occurs mid-turn, Pi compresses on the spot and continues，preventing the task from being aborted entirely.

Choosing to trigger compression at the end of a turn is deliberate. While a turn is still in progress, each request builds upon the previous one，and the LLM’s cache prefix can be reused continuously. Compressing at this point minimizes cache loss.

Compression isn’t a full deletion. Pi keeps the most recent segment of messages unchanged，with the length controlled by a configurable token budget (default: 20，000 tokens，roughly 5–20 turns). Everything before this segment is serialized and handed to a separate LLM request for summarization. This request differs from normal conversation in three ways:

• The system prompt changes，telling the model it’s a context summarization assistant.

• The user message changes，requesting a structured summary covering goals，progress，and key decisions.

• The request is independent of the session history，so a cheaper model can be used without incurring main conversation costs.

The summary is stored back in the session as plain text. This makes the summary portable—if you switch models in Pi，the new model continues using the same summary without re-compressing.

Pi itself is extensible. The author left an experiment entry at the end of the post, letting Pi write an extension to replace the default compression prompt，allowing you to test your own summarization strategies. I interpret this as the author’s attitude: there’s no one-size-fits-all approach to compaction；the current implementation is just the best practice as they see it.

The ideal outcome of a good summarization for a coding agent is like a handoff briefing from one shift to the next.

I strongly agree with this analogy. A good compressed summary is like a shift handoff—most of the old context is no longer relevant; you only need to convey the parts still in effect.

The Hidden Cost of Compression: Cache Invalidation

Prompt caching saves money through precise prefix matching. LLM providers offer steep discounts on cached portions—for example，DeepSeek V4 Flash is essentially free when a cache hit occurs. After compression, the retained segment of the turn remains unchanged，but its prefix has changed—it now follows directly after a summary. Once the prefix changes，all previously accumulated cache becomes invalid. The first request after compression requires full recomputation; subsequent requests then rebuild the cache.

Thus，compression has an implicit cost. The more frequently you compress，the higher the cache rebuild cost.

My Trade-offs

Three variables in context compression are at odds: information fidelity，context length，and cache hit rate. Frequent compression shortens context but repeatedly invalidates the cache and inevitably loses some details in summarization. Infrequent compression improves cache hit rate，but the model remains in a very long context for longer，making degradation and hallucination more pronounced.

My approach is to assess the relevance between earlier and later content. If what was discussed earlier is still needed later，compaction is worth doing. If the topic has already shifted，I simply clear and start a new session—no summary needed. The details lost in summarization are unknown at the time of compression; who knows which detail might be needed later? It’s fundamentally lossy compression, and detail loss is irreversible.

References

• earendil.com/posts/compaction-in-pi

• trychroma.com/research/context-rot

• pi.dev/docs/latest/compaction

• github.com/earendil-works/pi

🚀 Welcome to like and follow. Stay tuned for continuous updates on AI advancements.

LanLance@LanLance24·Aug 11 Article I replaced Claude Code with Pi Databricks and tested a batch of Coding Agents using their own codebase of millions of lines. With the same model and thinking settings，the only variable was the Harness. The output quality was essentially the same，but task costs varied by more than 2x. Pi feeds the model only as much context per turn as other agents feed…2374