Harness Handbook to Map Agent Behavior to Code (28 minute read)

# Harness Handbook — Making Agent Harnesses Understandable, Auditable & Editable Source: [https://ruhan-wang.github.io/Harness-Handbook/](https://ruhan-wang.github.io/Harness-Handbook/) Agent Harness · Tencent HY LLM Frontier · June 2026 Making agent harnesses understandable, auditable, and editable\. Open an open\-source coding\-agent codebase and you may want to see how it actually runs, verify that it behaves as safely as the documentation claims, or adapt it into an agent of your own\. These goals sound different, but once you are in the code they all reduce to concrete questions about behavior\. Will the agent ask before deleting a file, for example? Answering that means finding the confirmation logic, tracing bypass paths, and identifying every implementation site a change would touch\. In a repository with thousands of files, searches for*delete*,*permission*, or*confirm*return scattered fragments—and piecing them back into a full behavior chain is hard work\. The problem is not missing code, but missing a path from behavior to implementation\. What is needed is not another code index, but a map that connects the two\. Harness Handbook organizes scattered implementation into a**behavior\-level manual**: it structures execution around system behavior and links every step to verifiable code evidence\. Users can ask directly what they want to understand, audit, or change, and the Handbook locates the relevant behavior units, implementation sites, and next steps\. As the harness evolves, this map keeps the system understandable and reviewable—and keeps humans in the loop throughout\. Ruhan Wang1,2,\*· Yucheng Shi1,†· Zongxia Li1,3· Zhongzhi Li1,4· Yue Yu2· Junyao Yang1,5· Kishan Panaganti1· Haitao Mi1· Dongruo Zhou2· Leoweiliang1 1Tencent HY LLM Frontier ·2Indiana University ·3University of Maryland ·4University of Georgia ·5National University of Singapore \*Corresponding author: Ruhan Wang \(ruhwang@iu\.edu\) †Lead project collaborator A single behavior request often spans multiple implementation sites\. Harness Handbook reorganizes those scattered locations into navigable behavior paths and links every step to verifiable code evidence—so understanding, auditing, and modification share one map\. ## At a glance 1. **The problem Harness Handbook addresses\.**A harness shapes how agent behavior unfolds, yet that behavior is often implicit and buried in complex code\. Harness Handbook organizes it into a navigable, verifiable behavior map\. 2. **Reading a harness\.**The Handbook explains how a harness runs in terms of system behavior, then links those explanations back to code evidence—so readers need not start from a file tree and disconnected source fragments\. 3. **More reliable harness changes with coding agents\.**The Handbook maps natural\-language change requests to the relevant behavior units and implementation sites, helping the agent skip irrelevant searches, miss fewer dependencies, and produce a tighter edit plan\. 4. **Building your own agent on an existing harness\.**By presenting the harness’s behavior and capabilities in natural language, the Handbook lets users understand and adjust the system without wading through low\-level code—and build an agent that fits their needs\. Part 01 ## Why does an agent harness need a behavior\-level manual? When people talk about AI agents, they often start with model capability\. Once an agent starts doing real work, however, the question quickly shifts from “what can the model do?” to “what will the system allow it to do?” Whether a command runs, whether the user is asked before a file is deleted, and how failures are handled are decided not only by the model, but also by theharnessthat surrounds it\. To understand how these behaviors arise, you have to examine the harness itself\. Doing so comprehensively is difficult in production\. Codex, for example, coordinates the model, tools, state, permissions, and execution environment to turn each user request into a sequence of real actions—but the relevant implementation is spread across**2,267 files, more than 34,000 functions, and nearly 160,000 code connections**\. At that scale, a file tree shows where code lives, but not how those pieces work together to produce a behavior\. Directories and search results alone cannot reconstruct a complete behavior\. What we need is a different way to read the system: start from behavior, then trace back to source code for verification\. That approach serves three goals\. Understand See how the harness runs Follow one request through the full flow: what the model receives, when tools are called, how state moves, and how the system responds to failure\. Audit Verify that behavior matches expectations Trace the actual execution path to check permissions, confirmation logic, sandboxing, and data flow, including unusual routes that might bypass those protections\. Adapt Build your own agent First see which capabilities the harness already provides and which behaviors and code support them, then extend or adjust the system for your own needs\. All three goals begin with a specificbehaviorand then return to thecodefor evidence\. The difficulty is that behavior and code do not map one\-to\-one: one behavior is often determined by implementation scattered across several modules\. Part 02 ## Why is one behavior scattered across so many locations? “Ask the user before deleting a file” sounds like a simple rule\. To verify how it works in code, however, you must follow a chain of decisions: does the model request confirmation, can the tool call be intercepted, where is the user’s choice recorded, and under what conditions is the deletion finally executed? Any link in that chain can change the outcome\. **One behavior, many implementation sites\.**“Ask the user before deleting a file” is not decided by a single function\. It is shaped jointly by prompts, tool wrappers, permission configuration, state management, sandbox execution, and fallback paths\. Understanding, auditing, and modifying the harness all begin by locating these sites\.Each implementation site controls only one part of the chain\. No single`confirmBeforeDelete\(\)`function represents the complete behavior\. You have to trace prompts, tool wrappers, permissions, and state through to sandbox execution and fallback paths before you can tell whether the request will run, be rejected, or enter another flow\. So when you ask, “Will it really ask me before deleting a file?”, you are not running a keyword search—you are reconstructing a behavior chain and finding code evidence for every decision along it\. We call this**behavior localization**\. How completely you reconstruct that chain shapes the explanation, the risk review, and the change boundary\. Harness Handbook turns behavior chains hidden in the codebase into a map you can browse layer by layer and verify against source code\. Part 03 ## Harness Handbook: a navigable behavior map Because a single behavior can span many implementation sites, the Handbook cannot simply repackage the file tree\. Harness Handbookrepresents the harnessin behavior\-centric terms, using L1, L2, and L3 to narrow the question step by step\. Every level preserves verifiablecode evidence, so readers can check each explanation and ground later edits in source\. **From system understanding to behavior evidence\.**L1 establishes an overall view of the harness\. L2 organizes the system into behavior units and describes their responsibilities, inputs, outputs, and dependencies\. L3 examines a single unit in detail, connecting triggers, state changes, exception paths, and code evidence\. Rather than exposing everything at once, the three levels turn a complex harness into a navigable behavior map\.**L1 · System overview**starts with a global view of the harness\. Instead of listing files and functions, it follows one request through the system: how the request enters, which stages it passes through, how state moves between them, and how model output becomes a real action\. It first answers:*How does this harness run as a whole?* **L2 · Behavior\-unit overview**then breaks that system flow into behavior units\. Each unit captures a coherent class of behavior and records its responsibilities, inputs and outputs, dependencies, and key state\. At this level, readers can see how complex behavior is decomposed and how the pieces reconnect within the end\-to\-end flow\. **L3 · Behavior\-unit detail**finally zooms into one unit: when a behavior is triggered, how it executes, how state changes, which path is taken after an exception or failure, and which files and functions provide the evidence\. Here, “confirm before deleting a file” is no longer a standalone rule—it is a behavior chain whose individual decisions can be verified one by one\. ### An example L3 behavior unit Return to the example of “confirm before deleting a file\.” The L3 entry below expands that one\-line rule into a behavior unit that can be checked item by item, covering both the normal execution path and edge cases that need separate review\. Behavior unit · Tool execution stage #### Confirm before file deletion When the agent requests file deletion, the harness does not execute the operation immediately\. It first checks the permission policy and user\-confirmation state, then determines whether to proceed, reject the request, or return an error\. TriggerThe model emits a deletion call such as`delete\_file\(path\)`\.Permission rulePermission configuration marks file deletion as a high\-risk operation that requires user confirmation before execution\.State changeThe harness records the confirmation request and user response, then uses them to determine whether the current execution may continue\.Execution pathUser approval → run in the`sandbox runner`; rejection or lack of authorization → abort the call and return an error\.Edge cases`headless`mode, auto\-approval policies, or fallback paths may change the confirmation flow and need separate checks\.Evidence`tools/file\_ops\.py`· L32–78`tools/wrapper\.py`· L84–128`policy/permissions\.py`· L15–66`runtime/sandbox\.py`· L40–112`state/manager\.py`· L40–61 **An L3 behavior unit\.**Instead of merely stating that file deletion requires confirmation, L3 breaks the decision into verifiable pieces: what triggers the request, how permission rules constrain it, how confirmation state is recorded, how execution proceeds or stops, and which code evidence supports each step\. L3 supports behavior understanding, bypass\-risk auditing, and locating the right implementation when the policy must change\. Part 04 ## How is the behavior map generated from code? The three\-level structure determines how the Handbook presents behavior, but the map is trustworthy only if every explanation traces back to real code\. It cannot be built by asking a model to summarize files one by one\. Generation follows a facts\-first rule instead: natural language explains behavior, while every claim stays anchored in code facts—not model guesswork\. **From codebase to behavior\-level manual\.**Static analysis extracts program facts; behavior\-centered organization maps those facts onto system behavior; and synthesis renders the three\-level handbook while preserving every evidence link\.1 ### Extract facts → program graph The first step extracts static program facts: files, functions, classes, call relationships, state reads and writes, configuration boundaries, and external API calls\. Together they form a**program graph**that connects the harness’s otherwise scattered implementation elements and provides the factual basis for organizing behavior\. 2 ### Organize by behavior → behavior map The second step reorganizes the program graph into a behavior map\. It sketches a coarse**execution skeleton**of the harness lifecycle, then maps functions, modules, and code regions to the corresponding stages and behavior units\. Because the first mapping may be imperfect, a**proposer–reviewer loop**repeatedly corrects it until the stages, unit boundaries, and code evidence align\. 3 ### Synthesize the handbook → handbook Finally, the converged map is rendered as the system overview, behavior\-unit overview, and behavior\-unit detail\. The explanations may be generated in natural language, but every source link, function reference, and code snippet must come from the extracted program facts\. In short,**prose explains; facts anchor**—and every behavior chain can be checked in the code\. Part 05 ## How do you get from a behavior question to code evidence? Once the behavior map exists, readers can start from a specific question and work toward code evidence layer by layer—without searching the entire repository first\. We call this**Behavior\-Guided Progressive Disclosure**\(BGPD\): L1 supplies system context, L2 locates the relevant behavior units, and L3 opens the verifiable implementation details\. **From question to evidence\.**BGPD turns a behavior question into a traceable evidence path: L1 establishes system context, L2 locates the relevant behavior units, and L3 exposes triggers, state changes, execution paths, and source links\. Understanding and auditing mean verifying that evidence; modification turns it into an edit plan\. The repository remains the source of truth, while the Handbook cuts untargeted search and leads readers more quickly to the sites that need verification or change\.Behavior question→L1 system overview→L2 behavior\-unit overview→L3 behavior\-unit detail→Code evidence ### One evidence path, three uses Understand: build a system\-level model - Start at L1 and L2: map the harness’s overall execution flow, state transitions, and dependencies among behavior units\. - Before opening any single file, establish the system context for the target behavior and how it connects to upstream and downstream stages\. - With that map in place, later code details stay tied to the full flow\. Audit: verify the behavior holds up - Auditing continues into L3, checking the unit’s triggers, permission rules, state changes, fallback paths, and code evidence one by one\. - Take file deletion: confirmation on the main path does not rule out bypasses\. Every real execution path still has to match the expected policy\. - The conclusion rests on verifiable implementation evidence along the full behavior path—not on what the documentation promises\. Adapt: locate the change boundary - When behavior must change, use the Handbook to find the relevant behavior units, implementation links, and dependency paths—then define the change boundary\. - That evidence path can then become an edit plan, reducing repository\-wide search and the risk of missing a prompt, permission rule, or fallback path\. - The plan cites Handbook evidence; the coding agent still verifies and applies the changes in the repository itself\. Part 06 ## Does the Handbook help a coding agent find relevant code more accurately? To answer that question, we run the same coding agent on the same harness change requests, varying only whether it may consult the Handbook before localization\. The experiment focuses on localization and planning before any edit is made\. The planner is a coding agent built on NexAU, with DeepSeek\-V4\-Pro as the planner LLM\. Harnesses evaluated Terminus\-2 & Codex Both harnesses come from real production code; behavior spans multi\-stage control flow, tool calls, state management, and multiple modules\. Comparison setup Agent with vs\. without the Handbook Both conditions use the same coding agent, planner LLM, and workflow\. The only difference is whether the planner reads the Handbook before localization\. Judging setup Three independent judge models We measure whether the planner finds the right implementation sites and proposes a reasonably scoped edit plan—not whether the final code passes on the first try\. GPT\-5\.5, Opus 4\.8, and DeepSeek\-V4\-Pro each judge every comparison independently\. Evaluation metrics Win rate & token cost Win rate records which condition is judged better more often\. Token cost records planner tokens consumed per case: one measures output quality, the other search cost\. Not all change requests are alike\. The evaluation groups them into three types:**Q \(Query\)**adjusts existing behavior, such as a trigger condition, execution timing, or control flow;**CF \(Cross\-file\)**adds a capability across files and must connect the schema, pipeline logic, runtime behavior, and external interface;**SH \(Search\-hostile\)**is hard to find with keyword search because the relevant code often hides in mirrored implementations, fallback paths, or rarely exercised execution paths\. ### Result 1More accurate localization at lower search cost On both harnesses, the Handbook\-assisted planner wins more often while using fewer tokens per case\. The gain does not come from feeding it more context: the Handbook steers search toward relevant code earlier, cuts irrelevant exploration, and yields a tighter plan\. #### Overall preference rate % of pairwise comparisons preferred · higher is better #### Planner token cost Average planner tokens per case \(M\) · lower is better BaselineWith handbook **Higher preference, lower search cost\.**On both harnesses, judges prefer the handbook\-assisted planner, which also uses fewer planner tokens per case\.#### Three independent judges, two harnesses Preference rate: baseline → with Handbook, by judge **The gain is not a single\-judge artifact\.**All three judge models prefer the handbook\-assisted planner on both harnesses, so the result does not depend on any one judge\.### Result 2The gain comes mainly from more accurate behavior localization Better headline results do not explain where the advantage comes from\. We therefore compare the planner’s proposed edit sites against reference plans from Opus 4\.8 and GPT\-5\.5, then measure localization at file and symbol level\. With the Handbook, recall, precision, and F1 rise in nearly every setting, while*Wrong*cases—where the planner lands entirely in the wrong subsystem—drop sharply\. The planner is not simply seeing more code; it is finding the sites that actually implement the target behavior\. HarnessGranularityMetricBaselineHandbookΔ **Localization scores against the reference plan**\(Opus 4\.8 reference\)\. For**Wrong**, lower is better\.### Result 3The advantage persists as tasks get harder Does the advantage vanish when localization gets harder? Splitting results by request type and difficulty, it does not\. Easy cases usually involve one main behavior unit or a few direct implementation sites; Medium cases require coordinating behavior units, state changes, or cross\-file dependencies; Hard cases require finding relationships buried in indirect execution paths\. The Handbook helps with both local edits and complex behavior spread across the harness\. BaselineWith handbook **Consistent gains across settings\.**On both harnesses, the handbook\-assisted planner achieves a higher preference rate across request types and localization difficulty levels; the improvement is not confined to one modification pattern or difficulty tier\.### What do these results tell us? 01 ### A file tree shows where code lives; a behavior map shows how the system runs The results suggest that behavior is the better organizing principle for harness work\. Whether the reader is human or agent, you need to see which sites jointly produce a behavior before you can understand, inspect, or change the system effectively\. 02 ### The Handbook improves relevance, not search breadth Recall and precision rise together while token cost falls—the planner is not searching more broadly\. It reaches relevant implementation regions sooner and covers more correct sites with less wasted exploration\. 03 ### Understanding, auditing, and modifying share one evidence path The Handbook connects different tasks to one path from behavior question to code evidence\. Along that path, a question can be explained and audited, then—when needed—turned into an evidence\-backed edit plan\. Part 07 ## Handbook Studio: turning the behavior map into an operational entry point The experiment shows that a behavior map improves localization\. The next step is to turn that map into an entry point for working with the harness directly\. Handbook Studio is that workbench: connect a repository, generate the three\-level Handbook, and read, verify, and propose changes on the same behavior map\. The repository remains the source of truth\. Read Handbook The three\-level handbook forms a navigable workbench: the system overview, execution stages, behavior units, and key concepts unfold layer by layer as needed\. Users can also ask questions directly and, with full context, see how a behavior is triggered, executes, and affects system state\. Cross\-check Handbook ⇄ Code Every behavior description links back to code evidence\. Click a behavior unit, state variable, or execution path and a split view opens the corresponding source on the right: the left explains what happens; the right shows the implementation behind each claim\. Modify Co\-Edit Changes start on the behavior map\. Select a behavior unit, describe what should change, and the system generates a reviewable edit plan and code diff\. Nothing writes to the repository—or updates the Handbook—until the user confirms\. ### A concrete example: temporary environment variables for one command Here is a concrete example\. A quantitative researcher is building a Codex\-based agent to run data experiments automatically—backtesting a trading strategy on several years of market data, for instance\. The experiments repeat with different parameters\. The researcher wants a single command to carry its own environment variables: a market\-data API key, the date range for this run, and nothing else\. The variables should live only for that command, disappear when it finishes, and never leak into later commands\. Stated simply:**“Let this command carry its own environment variables\.”** Return to the file tree and that one sentence fans out into many edits\. The command schema needs an`env`field; tool descriptions must tell the model it exists; the execution path must pass those variables to the process spawn point and keep them scoped to the current command; tests must verify both the field and runtime behavior\. In all, the change touches`ShellCommandToolCallParams`,`ExecCommandArgs`,`to\_exec\_params`,`ExecCommandRequest`,`process\_manager`, and three test files—**14 implementation sites across 10 files**\. Finding them by hand is tedious, and it is easy to miss a forwarding step or a mirrored test\. In Handbook Studio, the user never needs those filenames first\. The system maps the request to the “command execution and environment variables” behavior unit, follows the evidence chain to list every affected site, and generates a reviewable edit plan and diff\. On the map, it is a field\-level change; in the repository, it becomes the coordinated updates above\. The user stays in control: code and Handbook update together only after review and confirmation\. **One behavior\-level request, many implementation sites\.**The user states only the requirement—“let this command carry its own environment variables without affecting later commands\.” On the behavior map, this is a field\-level diff\. Once confirmed, the change expands into the parameter schema, two tool descriptions, the shell and unified\-exec execution paths, the spawn\-environment merge point, and the mirrored tests that validate the tool specifications—**14 coordinated code updates across 10 files**\.- **Start from a behavior question\.**No need to learn the repository layout first—the system places the question in the full execution flow and locates the relevant stages and behavior units\. - **Verify with code evidence\.**The Handbook explains; source code confirms\. Reading, cross\-checking, and tracing all happen in one place\. - **Modify on the behavior map\.**State the intended change at the relevant behavior unit, review the generated plan and diff, and confirm before anything is written to the repository or synced to the Handbook\. Part 08 ## Takeaways Open code does not make behavior self\-evident\. What a complex agent harness needs is a behavior\-level manual—organized around what the system does, with a path back to code evidence at every step\. Understanding Start from the system overview and behavior\-unit map to see the harness’s execution flow, state transitions, and behavior structure before diving into source\. Auditing Follow the same path through permission rules, confirmation logic, fallback routes, data flow, and rarely exercised branches—grounding every conclusion in verifiable implementation evidence\. Adapting When you need to change behavior, the Handbook maps the target to the relevant files, functions, and execution paths—sharpening localization before you edit and cutting down untargeted full\-repository search\. HANDBOOK STUDIO Handbook Studio turns the manual into an interactive entry point: users start with a behavior question, read and verify the answer, then review and confirm changes on the same map\.