MIT OpenCourseWare 6.S950 Agency with AI

Lecture 5: The Harness handout has been updated: https://agencyai.mit.edu/w5/

Starting from a 300-line teaching Agent, it walks through observability, controllability, and “how much Harness do you need.” The specific best practices for Harnesses evolve quickly, but several systems principles from decades ago still hold. Understanding how a Harness works helps you collaborate better with Agents.

Agent = LLM + Harness: Deconstructing a Minimal Agent

The course team built their own teaching Agent called Baby Anigans: it answers questions about a text file and must cite the line numbers where the answers are found. It runs models locally via Ollama, with an outer shell of only about 300 lines of Python, two tools (search, read_lines), and no framework dependencies.

The core idea is that the LLM only makes requests — the loop decides what to execute: call the LLM, run a tool, inspect the result, then decide whether to continue or stop. This is the ReAct loop, and it’s fundamentally how mini-swe-agent and Claude Code work as well.

Demo trajectory: the model initially said “there is no file” and was prompted to search first; after finding something at line 125, it was asked to read neighboring lines before the answer was accepted. When a search fails, the system suggests trying shorter keywords; after two consecutive failed searches, Python forcibly returns a fixed “refusal” message, explaining that “not found doesn’t mean it’s not in the file.”

Observability: What You Can See — and What Vendors Hide

• Both answers passed the checks, but both missed the exception clause, with the same symptom but different causes: in one case line 127 never entered the model’s field of view (a retrieval problem); in the other, the exception content entered the view twice but was dropped from the answer (a model problem). The checks only verify “was there a search, is there a citation” — they don’t verify whether the answer is complete, because passing checks only tells you that the parts it checked are fine.

• Comparing what Baby Anigans and Claude Code can each observe: /context shows a context inventory, not the actual request being sent; a sub-Agent’s report is a summary written by the parent Agent and can contain omissions; git diff shows what was changed, but not the order of changes, nor whether the changes were correct.

• The trend is for vendors to wrap more of the Harness behind APIs: Codex’s context summaries have become encrypted content only the server can read; Claude’s reasoning process is encrypted, and although compressed summaries are readable, once modified, the API rejects them. Transparency has been traded for convenience, and the encapsulated parts may be exactly what’s most valuable (the handout cites an OpenAI article: tweaking two settings more than doubled ARC-AGI-3 scores).

Controllability: Three Long-Standing Principles Behind Various Specific Practices

• Lean AGENTS.md, on-demand loading of Skills and MCP, large inputs placed in files, sub-Agents returning only summaries, key rules surviving compression; principle: manage context — only let in what you’re actively using, and make sure it stays; origins: storage hierarchy (1965), working set (1968)
• Hooks that intercept, deny rules that take effect in any mode; principle: checkable rules should be enforced via mandatory interception; origin: reference monitor (1972)
• Permission controls, isolated MCP servers, two keys handed to two different people; principle: least privilege, separation of duties; origin: Saltzer & Schroeder (1975)

How Much Harness Do You Actually Need?

• Skepticism toward Harness engineering closely mirrors the skepticism once directed at prompt engineering.

• One research line lets the LLM manage the Harness itself: MemGPT moves data between memory layers on its own, ACE rewrites operating manuals, and Meta-Harness reads past run records and writes Harness code itself.

• There is no universal optimum: for 4 out of 5 models, the best-performing Harness differs across benchmarks; for the same component, it’s primarily a quality boost for weaker models and primarily a cost saver for stronger ones.

• Harnesses change fast: Manus says their Agent framework has already been rewritten four times.

Lec 5: The Harness

Source: https://agencyai.mit.edu/w5/ Search 6.S950 Agency with AI