Jev Engineering for Coding Agents

Author: Independent compilation based on design notes by Diogo Almeida (Founder of TypeSafe) · September 2026 Work Notes

This 12-page document is worth reading in full. It’s not product promotion—it’s an engineering notebook that re-examines “how a coding agent should be designed” from first principles.

It begins with a provocative question: If language models had no KV cache, how would you design a coding agent?

The power of this question lies in the fact that KV cache is the fundamental reason all agents today are built as “append-only conversation logs.” Reusing a cache prefix is cheap; once you modify content early in the context, the cache is invalidated, and the model must reprocess everything after the modification. This single economic fact shapes almost every design decision in current agents—and it’s rarely stated explicitly.

The author calls this the “tyranny of the KV cache.” Imagining the cache gone does two things: it allows an architecture designed explicitly for Jev with a state that is assembled rather than accumulated, and it reveals why the intuitively correct idea of “routing simple tasks to cheaper models” fails in practice.

I. Where Jev Stands

Jev is not the model that writes code; it is the decision layer alongside it.

The harness feeds the current application state (goal, context, rules, available actions, previous actions) and a predefined question to Jev, which returns a typed answer: a choice, score, or noul, each with a probability. Then, the frontier model, sub-agents, tools, and deterministic code perform the actual work.

Because the output is typed rather than free text, the harness can validate it, apply thresholds, and branch based on it without parsing prose.

Every native feature in this document is, at its core, asking Jev a question at a high-frequency decision point. Each session is queried thousands of times—it’s the small decisions that are the leverage.

II. Six Symptoms: Inherited Designs Current Agents Accept Uncritically

Symptom 1: Routing Doesn’t Work. The intuitive approach is to have the frontier model plan, delegate execution to cheaper models, and then have the frontier model review the results. The author calculates using Opus ($5 input / 25 output per million tokens) and Sonnet (3 / $15) pricing. Let X be context tokens, Y generated output tokens, and Z additional tokens read during work:

• Path 1: Pure Opus: 25Y + 5Z
• Path 2: Opus → Sonnet → Opus:
  • Sonnet loading context: 3X
  • Sonnet generation: 15Y
  • Sonnet reading: 3Z
  • Opus reloading changes: 5(Y+Z)
  • Total: 3X + 20Y + 8Z

Plugging in reasonable session proportions (X=0.65, Y=0.12, Z=0.23), pure Opus costs 4.15, while the routing path costs 6.19.

The cost of staying on the frontier model is about two-thirds of the path that was supposed to save money.

The lesson isn’t “routing is wrong”—it’s that routing based on token pricing instead of context reconstruction pricing is wrong. Routing is only viable under two conditions: the harness can provide the cheap model with a small, specialized context (not the full transcript), and the return trip doesn’t force the frontier model to re-read everything the assistant produced.

Symptom 2: Tools Crowd Out Context. Tools must be declared upfront in the system message with their complete parameter schema—whether they’ll be used this turn or not. This consumes significant context and doesn’t particularly foster smart tool selection.

The author’s working hypothesis is that models struggle with some combination of “high cardinality” (too many tools at once) and “off-policy tool calls” (tool usage patterns not seen in training). This might be why skills (loading a short description, deferring details) often outperform raw tool lists and MCP servers.

Symptom 3: Compression Exists. If every future turn wants the same shared state, compression makes perfect sense. The notebook questions this assumption: compression attempts general compression, which is both hard and lossy. Query-aware compression is much easier—if you know the next question, you know what to keep. Summaries written before the question is known will inevitably discard what the question needs.

Symptom 4: Sub-Agents Are Mediocre. Model parallelism is less prevalent than expected. The suspected reason lies in state management: deciding which parts of the parent context to pass in and which findings from each sub-agent to merge back. When this decision is both expensive and error-prone, models avoid it.

Symptom 5: Restart Exists. Restarting a session is the standard remedy for conversation log drift or corruption, but it discards both bad state and good state.

With addressable state, the alternative is a clean start, selectively reloading only the still-relevant old blocks on demand.

Symptom 6: The Battery Debate. The ongoing argument about whether agents should have built-in capabilities is essentially a trade-off between “ease of use” and “power.” This trade-off exists only because every battery permanently consumes context. Remove that cost, and the debate dissolves.

III. Where Tokens Actually Go

This is the most practical table in the document (by processed token share, input-intensive view, each read counted):

• Reading file content: 30-40% The largest item; files are repeatedly read into context.
• Searching codebase: 10-18% grep, glob, lists; output is noisy.
• Command output: 10-20% Stack traces and logs explode on failure.
• System prompt + tool schema + AGENTS.md: 5-12% Fixed overhead paid every turn.
• Conversation replay amplifier: — Why every item above is counted repeatedly.
• Reasoning and planning: 5-15% Higher during difficult debugging.
• Writing and editing code: 4-10% diff and str_replace are compact.
• Explaining to user: 2-5% CLI agents default to brevity.

The most striking figure is the second-to-last: writing code—the reason coding agents exist—is one of the smallest cost items. Reading and searching dominate.

Independent analysis points in the same direction: Microsoft’s fastcontext project reports that in GPT-5.4 traces, reading and searching account for 56.2% of all tool-call turns and 46.5% of the main agent’s total tokens.

If this is generalizable, the biggest efficiency gain in coding agents isn’t better models or better diff formats—it’s smarter retrieval.

IV. The Foundation Layer: Permissions and Tool Routing (Can Be Bolted onto Any Existing Agent)

Programmable permissions. Every command an agent runs raises the question “should it actually run?” Claude’s auto mode uses a classifier for this. The notebook proposes going further: permissions expressed as programmable queries about “what is allowed, what is denied,” with deeper checks when risk is high—for example, reading the contents of a Python or shell file before execution, not just approving the command name.

policy "exec":
    deny   if command touches ~/.ssh or .env*
    deny   if script contents contain network egress
           and task.scope != "deploy"
    ask    if command writes outside repo root
    allow  if command in read_only_set
    allow  if tests/ and exit code is expected

Harness as a tool router. Instead of exposing every tool schema to the model, the harness sits between intent and invocation. The model describes in plain text what it wants to do, and the harness uses a series of typed Jev calls to select the most appropriate tool (or top candidates) and construct arguments. The model never has to hold hundreds of schemas in context, and wrong argument types become validation errors instead of silent failures.

V. Meta-Attention: Treating Context Itself as a Decision

This is the core proposal, eliminating the notion that “context is static.”

For every user query, the harness asks Jev two things:

1. How good is the previous context? Is reusing the existing KV cache correct, or would it be cheaper and better to rebuild from scratch? This is framed as an explicit, cost-aware decision, not a default behavior.
2. How to construct a new context that contains everything relevant and nothing extraneous.

The simplest form is: a noul for every context block—each tool call input, each tool call output, each internal reasoning trace, and possibly each user interaction. Later versions replace the score with a visibility level:

hidden / short-summary / long-summary / full

This is the Visibility Ladder: the same block can be hidden, briefly summarized, detailedly summarized, or fully revealed, depending on the current query. This is query-aware compression—the property compression lacked.

The benefit is that the idea behind compression survives, but its major flaw disappears. Compression compresses once before the question is known; the ladder compresses query-by-query after the question is known. A 2400-line grep output can be twelve relevant hits for one question, invisible for the next—without ever being deleted from the state.

(The author adds a notable visual idea: if the harness could show a heatmap of which parts of grep output are relevant, it could filter that output arbitrarily within the budget allowed. And, the notebook observes, it would also look cool.)

VI. Revisiting Routing and Sub-Agents

First-class dynamic context is what makes routing viable again. Once the harness can construct a small, relevant context for a sub-task, handing that sub-task to a cheaper or faster model doesn’t require the cheap model to load the entire session. The result can be merged back as a scored block, rather than a transcript the frontier model must re-read.

The same mechanism unlocks sub-agents. The notebook’s guess is that most of the cost of today’s sub-agents is the work of deciding what context to pass—compared to which, the user typing a simple instruction is easy. If constructing that context becomes cheap and automatic, sub-agents could be used much more frequently.

It also opens a user-facing control: pay more for faster or better results, or run conservatively to minimize spending.

(Two extending questions: Extreme parallelism—if dispatching tasks becomes cheap, many will run concurrently, and the harness inherits all the problems of concurrent systems: synchronization primitives, inter-agent communication, write conflicts when multiple agents share state. The notebook suggests shared state with locks, and making it tractable by explicitly distinguishing reads from writes, since read-only tasks never contend for locks. Goal deduplication—for goal-driven loops like /goal, registering it as a sub-goal and deduplicating against history before dispatching any sub-task ensures work already done or in flight isn’t started twice.)

VII. Tools and Skills, from First Principles

The model cannot propose an action it doesn’t know exists, so it needs short snippets describing what’s available. But these snippets don’t have to live in the system message—they can be loaded dynamically when relevant. Behind them is the ability to “dump the full schema of available actions when needed.”

The requirement tying this together is: once these things are no longer needed, they must not pollute the context.

Three-tier disclosure:

1. Tier 1: Snippets One line per capability, loaded when relevant, for hundreds of tools.
2. Tier 2: Schema Full parameters for the selected few, provided on demand.
3. Tier 3: Documentation Manuals provided for one-off queries.

If this works, the battery debate from Section II vanishes: when a built-in capability costs almost nothing until used, an agent can ship with hundreds of tools and thousands of pages of documentation.

The author points out a second benefit: near-zero-cost integrations are powerful co-marketing channels, and they make things “just work.”

VIII. Conditional Instructions: A New Form for AGENTS.md

Today’s AGENTS.md is loaded in full. The notebook proposes conditional loading.

Editing frontend code? Load the style guide. Working in a specific subdirectory? Load that directory’s gotchas file (and the notebook suggests every subdirectory should have one).

This is like skills, but with a distinction: skills typically mean “do this now”; conditional instructions mean “remember this for later.” The second type also requires a property skills lack: immunity to compression. A skill loaded early in a long session will eventually be compressed or summarized away; an instruction bound to a condition will be reloaded when the condition is met.

IX. Security-Aware Routing

Today’s routing is around difficulty and cost. The notebook adds a third axis: trust.

Some open-weight models available through low-cost providers are much cheaper than frontier APIs, but the concern raised is that data passing through certain endpoints may no longer be private. The proposal is to estimate “which files might be touched” for each sub-task, attach policies to file types, and route accordingly:

• Public docs, open-source dependencies → Open: Any model, cheapest first.
• Application code → Standard: Vetted providers.
• Secrets, env, infra config → Restricted: First-party frontier models only.
• Proprietary research code → Custom: Exclude specific providers.

The last line reflects a broader point: difficulty and cost aren’t the only reasons for routing. A team might avoid a provider’s models because they’re doing model research, or for security reasons. Once routing is policy-driven, these preferences become configuration, not discipline.

X. Background Processing: Where the Jev Argument Pays Off

The notebook identifies a common pattern in popular agent workflows: building HTML pages that update in parallel as work progresses; the argument that “understanding, not generation, is the new bottleneck”; generating evals in the background; using a few charts and minimal text for an ELI5 explanation of the system; and having the agent maintain a small, deployed progress page with screenshots and notes, viewable from a phone during long tasks. There’s also a production pattern: mirroring live traffic to a candidate model, automatically generating about a day’s worth of evals, then deciding whether to switch.

Their common thread is: they run in the background, as an extension of normal workflows, and are read-only functions of the current codebase state. Cross-model review (having one provider’s agent audit another’s output) fits this same pattern.

This is where the Jev argument pays off. A Jev-centric harness must know precisely what is in the context and whether each operation is a read or a write. Finding information relevant to a particular code change is non-trivial work. If this retrieval is shared across all background tasks instead of repeated for each, running them becomes much cheaper, allowing many more to be run economically. Given the findings in Section III—retrieval dominates—sharing it represents the single largest saving.

XI. Candidate Built-in Tool List

The notebook concludes by listing several open-source projects that could be natively integrated, each with a design note on “how a Jev-centric harness would use it”:

• headroom Context compressor. Uses classifiers to check if compression preserved needed facts.
• rtk Tool output compressor. First-party prompts to help the model understand it.
• ast-grep Structured search. Load the manual once, generate N queries, filter by relevance.
• ast-outline Structured outline. Hierarchical calls: select subtrees to inspect.
• fastcontext Repository exploration sub-agent. Route to it, or use its structure to replace its search.
• fff Path and content search. In-memory index, frequency-sorted; faster than ripgrep in long sessions.

XII. Conclusion

Coding agents are simple loops, and the leverage isn’t in the loop. The leverage is in what the harness puts in front of the model each turn—and today, that decision is made by defaults, by an append-only conversation log shaped around KV cache economics.

Removing the cache as a thought experiment changes six familiar behaviors. Routing fails because context is reprocessed, not because cheap models are weak. Tools crowd the window because they must be declared upfront. Compression loses information because it compresses before the question is known. Sub-agents are rare because passing state is hard. Restart discards good state along with bad. The battery debate exists solely because every built-in capability permanently consumes context.

None of this requires better models. It requires treating the context window as something intentionally assembled, not incidentally accumulated.

Links

Original document (Google Drive): https://drive.google.com/file/d/17h982xvsL3E7b80iGmOCfKp9qTOW9ohv/view

Jev Official Documentation: https://docs.typesafe.ai/

This publication’s Jev series and cascade tests: https://github.com/yibie/laya-jev-lab

#AgentEngineering #ContextEngineering #HarnessDesign