The article introduces GCB and KRE as two layers to optimize token usage and context management in persistent multi-agent AI systems, reducing costs while maintaining capability.
If you’re building a persistent multi-agent AI system, eventually you run into the same problem: Usage gets insane. Agents are using expensive models for tasks that don’t require them, while simultaneously carrying around massive context windows full of information they don’t need. I’m approaching this from both sides by building two layers: GCB — Governed Capability Broker The GCB sits between the agent and the models. Instead of permanently assigning your orchestrator/researcher/coder/reviewer to one expensive model, it evaluates the task and routes it to the appropriate model + reasoning level + budget. Simple task? Small/cheap model. Complex architecture decision? Bigger model with higher reasoning. Critical task? Escalate. You stop paying maximum intelligence prices for minimum intelligence work. Then there’s the other half: KRE — Knowledge Retrieval Engine Think RAG, but expanded into an actual knowledge/context management layer. Instead of dumping huge conversation histories, project files, logs, and previous agent outputs into every prompt, the KRE determines what information is authoritative and actually relevant to the current task. It can retrieve hierarchical project state, distinguish current state from superseded state, compact older information, construct task-specific context, and only give each agent what it actually needs. So basically: GCB = stop wasting compute. KRE = stop wasting context. Put them together and you’re attacking token/usage costs from both directions. The important part is that this doesn’t necessarily mean making the AI less capable. Your strongest models are still there when they’re actually needed. Your knowledge is still there when it’s actually needed. You’re just not paying a frontier model to reread 50,000 tokens of irrelevant history before doing something a smaller model could have handled in 2,000. For persistent AI agents, I think this is a much more scalable approach than simply giving every agent a huge context window and the strongest model you can afford. RAG solves retrieval. A KRE manages knowledge/context. A GCB manages intelligence/compute. Put those together and you start getting something that feels less like “a bunch of LLMs talking to each other” and more like an actual AI operating architecture.
The article analyzes a 2026 paper by Bai et al. showing that subagents and context bloat cause token costs in long agent runs to be ~1000x higher than chat, and presents three practical fixes (PLAN.md, read budget, out-of-band notes) that reduce token usage by 70-90%.
This thread shares strategies to reduce token usage in AI agents, including prompt caching, context summarization, using smaller models, trimming tool outputs, subagents, RAG, and tight system prompts.
The article discusses how optimizing context structure for KV caching can significantly reduce the cost of running AI agents, based on insights from an OpenAI podcast during migration to GPT 5.6.