System Design for Agent Systems (Part 1)

Put a frontier model into a badly designed agent system and you get a more articulate failure. Nearly all of the engineering sits outside the model, in the scaffolding that has come to be called the agent harness.

This part covers the five components you design whether or not you notice you are designing them: tools, prompts, memory, orchestration, and where a person sits in the loop. Then the evaluation framework that tells you whether any of it works, because none of this is testable the way ordinary code is.

Building agent systems requires intentional design across three key areas:

• Understanding the workload: Agent jobs are fundamentally different from web requests

• **Designing the harness: **The components you build that aren’t the model (tools, prompts, memory, orchestration)

• **Building evaluation frameworks: **How to measure quality and catch regressions

This guide covers all three. Everything is drawn from a real system: DevVoice, a five-agent pipeline that turns a GitHub README into a reviewed X thread, LinkedIn post, and dev.to article. These are the design decisions that determine whether your agent system is sound.

Note: Code samples are illustrative pseudocode showing patterns and concepts. Adapt them to your framework (LangChain, LlamaIndex, Anthropic SDK, etc.).

Agent Harness Components

Agent Harness Components

Part 1: Foundations

Agent Workloads Are Not Web Workloads

Start here, because almost every downstream decision follows from it.

A typical web request is 50–200ms, CPU-bound in bursts, deterministic, and costs approximately nothing per unit. An agent job is none of those things:

Four architectural consequences:

Duration. You cannot answer the request synchronously. Load balancers have idle timeouts. Holding a connection open for two minutes per active job is wasteful.

I/O-bound. Concurrency is not limited by cores. A worker spends 70–80% of its time waiting for the model API. A 2-core container can run dozens of concurrent jobs. Sizing for core count wastes most of what you pay for.

Non-determinism. The same input produces different output, so retries are not free. You cannot assert on output in CI which tests must target the harness, not the model.

Marginal cost. A bug is a bill. An infinite loop in an agent loop burns money at roughly a dollar a minute, silently, until someone notices.

Design for this shape from the first commit, and deployment is configuration. Design for the web shape, and deployment is a rewrite.

Part 2: The Agent Harness

What You Build That Isn’t The Model

The harness is everything you build that isn’t the model. Nine boxes of it, in the real system. Four are worth designing early.

Component 1: Tools

Tools are the agent’s only way to affect the world. Design them poorly and you waste model tokens on confusion; design them well and you cut context by 60%.

Tool definition is a contract:

python@tool def search_code( query: str, repository: str, max_results: int = 10 ) -> list[CodeMatch]: “”“Search for code patterns across a repository.

Args:
    query: The code pattern or keyword to search for
    repository: GitHub org/repo identifier
    max_results: Limit results (default 10, max 100)

Returns:
    List of matching code locations with context
"""

Three requirements:

• Typed inputs and outputs. The model sees the schema; ambiguity costs tokens and errors. Use Pydantic models for complex returns.

• Small surface area. One tool per discrete action. Five focused tools beat one with five optional parameters. Avoid tools that accept arbitrary queries the model will call them wrong.

• Deterministic and fast. A tool that takes 30 seconds is a context killer. If an operation is expensive, make it async and poll with a status tool, or move it into a task queue.

Tool management:

• Keep tools in a registry, not scattered through agent code

• Version tools separately from agents: agents call search_v2 while others still use search_v1

• Log every tool call with arguments and latency, you’ll debug half your issues here

• Add a timeout to every tool; a hung tool stalls the agent

**Component 2: Prompt Engineering **

Prompts are not prose, they are state machines. Structure matters as much as words

The basic shape:

markdown1. System prompt (static)

• Role and context
• Hard constraints and rules
• Output format specification

2. Static examples (static)

   • Few-shot examples showing desired behavior
   • Edge cases you’ve seen fail

3. Tools specification (static)

   • Tool definitions and usage examples
   • Error handling behavior

4. Dynamic context (dynamic)

   • Conversation history
   • Per-request data (user input, retrieved documents)
   • Job state

5. User turn (dynamic)

   • The actual request

The cardinal rule: Static content must precede dynamic content. Tool definitions, then system prompt, then examples, then history, then the user turn. Every dynamic value that leaks upward nukes the prompt cache and quietly doubles your token cost.

Versioning prompts:

pythonPROMPT_VERSION = “2024-08-19-v3”

def build_system_prompt(version: str) -> str: if version.startswith(“2024-08-19”): return “”“You are a code reviewer…”“” elif version.startswith(“2024-08-15”): return “”“You are a reviewer…”“” # Previous version

Version your response-cache keys on prompt version. A bad prompt gets cached and served for hours after you fix it. With versioning, rolling back is one env var change.

**Component 3: Memory **

Memory Layer

Memory Layer

Agents need two kinds of memory, and they are not interchangeable.

Working memory (short-term, in-context):

• Current conversation turn

• Last N messages (retrieved)

• Retrieved context for this job

• Token budget: whatever fits in context

Persistent memory (long-term, storage-backed):

• User preferences and history

• Learned facts from past interactions

• Evaluation results and failures

• Stored indefinitely

The mistake: treating working memory like persistent memory. Adding “remember X” to the system prompt on every turn burns tokens. The right pattern:

python# At job start: seed working memory working_memory = { “user_id”: job.user_id, “past_interactions”: fetch_last_n_interactions(job.user_id, n=5), “learned_preferences”: fetch_user_preferences(job.user_id), “retrieved_documents”: retrieve_relevant_docs(job.query), }

During job: work from working memory

prompt = build_prompt_with_context(working_memory)

At job end: update persistent memory

if agent.learned_something_new: store_user_fact(job.user_id, fact)

Avoiding context overflow:

• Truncate working memory intelligently — drop oldest messages, keep most recent and most relevant

• Store summaries of long conversations in persistent memory, retrieve the summary instead of the full history

• Use content-based chunking for documents: break at section boundaries, not token limits

Component 4: Multi-Agent Orchestration

Agent Orchestration

Agent Orchestration

Agents can coordinate in two ways: parallel (agents work simultaneously, results are merged) and sequential (agents work one after another, output of one feeds into the next).

Parallel agents sound more efficient. They’re not. They create a coordination problem: agent A produces output that agent B needs, but agents run at different speeds. You end up polling and merging, which is both slow and complex.

Sequential orchestration with subagents is better:

python# Bad: parallel agents that might conflict async def process_parallel(doc: str): results = await asyncio.gather( summarizer_agent(doc), analyzer_agent(doc) ) return merge_results(results)

Good: sequential subagents with clear data flow

def process_sequential(doc: str): summary = subagent_summarize(doc) analysis = subagent_analyze(doc, context=summary) final_review = subagent_review(doc, summary, analysis) return final_review

Why subagents win:

• Clear data flow. Agent B explicitly depends on Agent A’s output

• Early exit. If Agent A’s output is invalid, you stop before starting Agent B

• Simpler state. Working memory is threaded through agents in order; no merging logic

• Easier debugging. Each subagent has one clear input and output

Use parallel agents only when agents genuinely work on independent data and their outputs never interact (e.g., three different APIs for the same query). Even then, consider sequential agents with fallback.

The orchestration surface should be clear:

pythonclass Orchestrator: def run(self, job: Job) -> Result: extracted = self.extract_agent.run(job.content, job.schema) validated = self.validator_agent.run(extracted, job.rules) formatted = self.formatter_agent.run(validated, job.format) return Result(data=formatted, trace=self.trace)

Each subagent is built once per process and called many times. State flows through them in order.

Component 5: Human in the Loop

An agent that can only read is safe and not very interesting. The moment it can send, publish or spend, some of what it does stops being undoable and that is what a human in the loop is for. Not review for its own sake, but a person standing between the model and the actions it cannot take back.

Gate on irreversibility, not importance. The test is mechanical: can another tool call undo this? Reading, retrieving and drafting pass if the model gets them wrong, the next step fixes it. Publishing, sending, deleting and spending do not. Importance is the wrong axis because everything looks important to whoever asked for it.

Over-gating is the failure that looks like caution. A gate firing on everything trains the reviewer to click approve without reading you pay the latency and keep none of the safety. Gates keep their meaning by being rare.

One constraint: the gate is a state the run rests in, not a blocking call. The mechanics are Part 2.

Loop engineering vs graph engineering

Where that gate can sit depends on how the run is structured, and there are two ways to structure it.

• Loop engineering is one agentic node: one agent, one goal, a verifier, a stop condition.

• Graph engineering is the topology around those loops: which nodes exist, which transitions are legal, what state crosses each edge.

They are not rivals. A graph is what you build when one loop stops being enough, and every node in it is still a loop.

This decides what a gate can promise. In a loop, the gate is a tool the model chooses to call so it holds only as well as the model’s judgement. In a graph, it is a node on an edge, and the irreversible step has no path around it.

Start with a loop. Move a decision into the graph when its failure is expensive enough that you want it impossible rather than unlikely which is usually the irreversible ones, and usually first.

A model deciding it is finished is the least reliable stop condition you have. Add three more: a step budget, a wall-clock budget, a token budget. The model owns one exit; you own the other three.

A loop that stops making progress does not error. Every call succeeds, and it bills about a dollar a minute until someone notices.

Part 3: Evaluation and Metrics

Design evaluation before you design agents. Otherwise you’re flying blind.

The Evaluation Framework

Agent systems produce output that is plausibly correct but often wrong. You cannot test correctness in CI. You need a framework for evaluating quality, catching regressions, and comparing variants.

Three layers of evaluation:

Layer 1: Harness correctness (deterministic, CI-testable)

• Does the orchestration flow correctly?

• Do tools return the expected schema?

• Does state transition as expected?

• Does the result assemble correctly?

Layer 2: Agent output quality (non-deterministic, sampling-based)

• Does the output match the specification?

• Are the facts accurate?

• Is the reasoning sound?

• Is the format correct?

Layer 3: End-to-end regression (automated, periodic)

• Run a fixed test set through the system weekly

• Compare results to baseline with a judge model

• Flag regressions before they hit production

Standard Metrics

For the orchestration layer:

For output quality:

For cost and efficiency:

LLM as a Judge

Don’t ask humans to score agent output. Use another LLM.

The judge is a specialized agent with one job: evaluate output against a rubric.

pythonclass JudgeAgent: def evaluate(self, output: str, rubric: EvaluationRubric) -> Score: prompt = build_evaluation_prompt(output, rubric) response = self.model.invoke(prompt) score = parse_score(response)

    return Score(
        value=score.value,
        reasoning=score.reasoning,
        failures=[...],
        confidence=score.confidence
    )

The judge produces structured output:

python@dataclass class Score: value: float # 0-10 reasoning: str # why this score? failures: list[str] # what went wrong confidence: float # how sure is the judge? suggest_fix: str # what would improve it?

Use the judge to:

• Score a sample of every day’s output (100-200 jobs)

• Flag regressions: if today’s P50 judge score < baseline - 1σ, alert

• Compare variants: run both version A and version B on the same 100 test cases, compare judge scores

• Iteratively improve: when the judge finds a failure pattern, add it to the test set

Important: Test the judge itself by:

• Scoring known-good and known-bad outputs , the judge should differentiate them

• Checking judge agreement on edge cases, if two judge instances disagree, the rubric is ambiguous

• Comparing judge scores to human scores, they should correlate

Continuous Evaluation in Production

Weekly eval run:

python# Every Monday morning def eval_run(): sample = db.jobs.sample(200) scores = [] for job in sample: score = judge.evaluate(job.output, job.rubric) scores.append(score)

baseline_mean = 7.2
current_mean = np.mean([s.value for s in scores])

if current_mean < baseline_mean - 0.5:
    alert("Regression detected", current_mean)

db.metrics.insert(date=today, mean_score=current_mean)

A/B testing via judge:

pythondef compare_variants(prompt_old, prompt_new, test_set_size=100): test_jobs = db.jobs.sample(test_set_size)

scores_old = []
scores_new = []

for job in test_jobs:
    result_old = orchestrator.run(job, prompt_old)
    score_old = judge.evaluate(result_old)
    scores_old.append(score_old)
    
    result_new = orchestrator.run(job, prompt_new)
    score_new = judge.evaluate(result_new)
    scores_new.append(score_new)

mean_old = np.mean(scores_old)
mean_new = np.mean(scores_new)

if mean_new > mean_old + 0.3:
    return "ship"
else:
    return "revert"

The Scorecard

Not every system builds all of these out. Every system has all of them decided. The only question is whether you decided or defaulted.

• Tools: few, one action each, tightly typed, with a timeout and a log line

• **Prompts: **static before dynamic, versioned, and the cache key versioned with them

• **Memory: **working and persistent kept apart, seeded once rather than re-sent every turn

• **Orchestration: **sequential subagents over parallel agents; state flows one way

• Human in the loop: gate on irreversibility, not importance; rare enough that the gate still means something

• **Loop vs graph: **start with one loop, move a decision into the graph when its failure should be impossible rather than unlikely

• **Stop conditions: **four exits, only one of which the model owns

• **Evaluation: **the harness tested on every commit, the output sampled and judged out of band

Notice how little of that is about the model. Swap the model underneath and all eight survive unchanged, which is the argument for treating them as the actual engineering rather than the plumbing around it.

Part 2 is the backend this needs in order to run: the queue, the workers, the two stores, and why an agent workload breaks every instinct you have from web services. Part 3 puts it on AWS.

Further reading

• Awesome Harness Engineering: curated taxonomy across all of these areas

• Agent Harness for LLM Agents: A Survey: the 23-system comparison

• Graph Engineering vs Loop Engineering: What Actually Changed: the clearest treatment of that section

• Agentic AI System Design Explained in 27 mins by @aiwithaish

• 3 Years of Graph Engineering with LangGraph: the framework side of the same argument

• Context Engineering: Memory, Compaction, and Tool Clearing: more on

• The Agent Harness: Why the LLM Is the Smallest Part: the framing this article rests on

• My PoC: Agent Harness Ops

See you soon.

Follow @kmeanskaran