Parthenon Law: A Self-Evolving Legal-Agent Framework
Summary
Parthenon is a self-evolving legal-agent framework that structures LLM agents into six auditable layers and uses an anti-leakage learning loop to improve performance on end-to-end legal matters without modifying model weights. A large-scale empirical study on Harvey LAB with 12,510 agent trajectories shows current frontier agents still struggle with strict matter completion, and Parthenon substantially improves results over state-of-the-art baselines.
arXiv:2606.04602v1 Announce Type: new
Abstract: As agents grow more capable, legal-domain LLM agents promise to turn document-heavy matters into reviewable work products -- yet reliable deployment faces three obstacles: no large-scale evidence on how today's strongest model-and-harness combinations behave on end-to-end legal matters; no agent architecture adapted to the legal vertical, only general-purpose harnesses; and, in a setting that keeps shifting with new facts, authorities, and deadlines, no mechanism for systems to learn from their own outcomes. We address each. A large-scale empirical study on Harvey LAB -- $12{,}510$ agent trajectories -- shows that even frontier agents remain far from completing matters in a single pass: per-criterion accuracy climbs with stronger models while strict matter completion stalls. We then introduce \textsc{Parthenon}, a self-evolving legal-agent framework that factors Model, Harness, Agent roles, legal Knowledge, deterministic Tools, and procedural Skills into auditable surfaces for source traceability, date and number grounding, deliverable compliance, and issue closure. Finally, an anti-leakage learning loop converts scored failures into task-agnostic edits to skills, tools, and knowledge, letting the system improve with experience -- as a firm refines its checklists and playbooks after each matter -- without touching model weights. Across our large-scale empirical analysis, \textsc{Parthenon} substantially improves the performance of state-of-the-art models and harnesses on legal-matter tasks.
View Cached Full Text
Cached at: 06/05/26, 02:08 AM
# Parthenon Law: A Self-Evolving Legal-Agent Framework
Source: [https://arxiv.org/html/2606.04602](https://arxiv.org/html/2606.04602)
###### Abstract
As agents grow more capable, legal\-domain LLM agents promise to turn document\-heavy matters into reviewable work products—yet reliable deployment faces three obstacles: no large\-scale evidence on how today’s strongest model\-and\-harness combinations behave on end\-to\-end legal matters; no agent architecture adapted to the legal vertical, only general\-purpose harnesses; and, in a setting that keeps shifting with new facts, authorities, and deadlines, no mechanism for systems to learn from their own outcomes\. We address each\. A large\-scale empirical study on Harvey LAB—12,51012\{,\}510agent trajectories—shows that even frontier agents remain far from completing matters in a single pass: per\-criterion accuracy climbs with stronger models while strict matter completion stalls\. We then introduceParthenon, a self\-evolving legal\-agent framework that factors Model, Harness, Agent roles, legal Knowledge, deterministic Tools, and procedural Skills into auditable surfaces for source traceability, date and number grounding, deliverable compliance, and issue closure\. Finally, an anti\-leakage learning loop converts scored failures into task\-agnostic edits to skills, tools, and knowledge, letting the system improve with experience—as a firm refines its checklists and playbooks after each matter—without touching model weights\. Across our large\-scale empirical analysis,Parthenonsubstantially improves the performance of state\-of\-the\-art models and harnesses on legal\-matter tasks\.
## 1Introduction
The field is moving from assistant\-style drafting toward agents that ingest a full matter record, reason across facts and law, and return a reviewable work product\. Errors in this setting are consequential: missed deadlines, unsupported calculations, omitted issues, invalid deliverables, or advice untraceable to the record\. Legal benchmarks evaluate reasoning, retrieval, and factuality\(Guhaet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib1); Pipitone and Houir Alami,[2024](https://arxiv.org/html/2606.04602#bib.bib22); Liet al\.,[2025](https://arxiv.org/html/2606.04602#bib.bib20)\); Harvey’s Legal Agent Benchmark \(LAB\) extends this to long\-horizon matter work with source documents, deliverables, and expert rubrics\(Grupenet al\.,[2026b](https://arxiv.org/html/2606.04602#bib.bib40)\)\. Yet we still lack a large\-scale empirical analysis of how today’s strongest models and harnesses actually perform on end\-to\-end legal matters, a framework that protects legal invariants, and a safe mechanism for turning scored failures into reusable improvements\. These gaps hinder the deployment of legal AI agents in real\-world legal practice\.
To measure this gap, we evaluate the major families of legal\-agent execution on the full Harvey LAB corpus: direct API prompting, a basic legal\-native harness, and the Codex and Claude Code workspace harnesses, each run across model tiers\. LAB’s all\-criteria\-must\-pass grading resembles human legal review: a memo that misses a material deadline, authority, risk, or requested deliverable remains incomplete\. Under this standard, per\-criterion scores improve with stronger models, but strict matter completion stays low\. The same failure modes recur across harnesses and model upgrades: incomplete source coverage, lost quantitative detail, malformed work products, unfinished issue analysis, and weak grounding\. The bottleneck is not the model alone; it is the absence of a structured legal work system around it\.
Parthenonaddresses these gaps as a six\-layer framework organized around attribution and auditability \(Figure[1](https://arxiv.org/html/2606.04602#S1.F1)\)\. The lower layers make execution explicit: Model \(LLM capability\), Harness \(the workspace runtime\), and Agent roles\. The upper layers hold legal expertise in editable artifacts: Knowledge for matter state, concepts, calendars, and schemas; Tools for deterministic inspection, search, transformation, and post\-draft audits; and Skills for work plans and release checks\. Separating these layers keeps model choice, legal memory, tool behavior, and procedural guidance from collapsing into one opaque prompt, while making each failure assignable to a surface that can actually be edited\.
Each scored matter is feedback on the upper harness rather than data the model must memorize: a solver produces the work product, a rubric\-isolated evaluator converts it into redacted feedback, and a learner proposes task\-agnostic edits to skills, tools, or knowledge\. This mirrors how legal practice firms update checklists, forms, and playbooks after poor outcomes\. An anti\-leakage protocol keeps rubric text, task identifiers, source facts, and answer keys out of the learner, so candidate edits must generalize beyond the batch that exposed the failure\. The system therefore adapts without fine\-tuning model weights or memorizing benchmark signal\.
We run12,51012\{,\}510agent trajectories on Harvey LAB across the Codex and Claude Code harnesses and three model tiers\. Current harnesses fall short: a stronger base model lifts per\-criterion accuracy but rarely passes a matter in full, and even the strongest baseline passes every criterion on barely one matter in eight\. With the model and agent harness fixed, addingParthenonraises pooled accuracy by\+13\.8/\+10\.2/\+7\.4\+13\.8/\+10\.2/\+7\.4pp—to82\.0/89\.9/90\.2%82\.0/89\.9/90\.2\\%, a gain comparable to a model upgrade, from the harness alone—and roughly triples strict all\-pass completion on the weaker solvers \(14→4214\\to 42,47→13747\\to 137\)\.
In summary, our contributions are: \(i\) a large\-scale empirical analysis of major agent execution modes on Harvey LAB; \(ii\)Parthenon, a six\-layer legal\-agent framework with auditable surfaces; and \(iii\) a self\-evolving learning loop that turns scored failures into task\-agnostic harness updates\.
Parthenon: Self\-Evolving Legal Agent FrameworkSkillsSubject TriageFailure ModesDoctrineLegal ScaffoldsRelationshipsOutput ShapeAnti\-LeakageData Privacy……ToolsDoc ReadersStatute SearchWindow SearchDeliverable SearchNumber AuditDate AuditCitation CheckDoc Builder……KnowledgeSchemasStatutesTimelineDoc TypeCalendars…AgentSolverEvaluatorLearner…HarnessCodexClaude CodeOpenClawOpenCodeOpenHands…ModelGPTClaudeGeminiDeepSeekKimi…
Figure 1:Parthenonas a six\-layer legal\-agent framework\.Each layer is a replaceable surface\. The Model and Harness layers enumerate compatible model families and workspace runtimes; the Agent layer fixes the solver–evaluator–learner roles; the Knowledge and Tools layers hold the legal knowledge bases and the auditable inspection, retrieval, computation, grounding, and validation operations over them; and the Skills layer holds the procedural scaffold edited by the learner\.
## 2Related Work
##### Legal benchmarks and legal\-agent evaluation\.
Legal NLP benchmarks mostly use bounded tasks: LegalBench\(Guhaet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib1)\), LexGLUE\(Chalkidiset al\.,[2021](https://arxiv.org/html/2606.04602#bib.bib8)\), CUAD\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.04602#bib.bib9)\), and LawBench\(Feiet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib10)\)probe reasoning, classification, and clause extraction\. Legal\-agent systems\(Cuiet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib19); Liet al\.,[2025](https://arxiv.org/html/2606.04602#bib.bib20); Mantravadi and others,[2025](https://arxiv.org/html/2606.04602#bib.bib21)\)and source\-grounding studies\(Pipitone and Houir Alami,[2024](https://arxiv.org/html/2606.04602#bib.bib22); Dahlet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib23); Kant and others,[2025](https://arxiv.org/html/2606.04602#bib.bib24); El Hamdaniet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib25)\)push toward grounded deliverables\. Closest to our setting, Harvey LAB\(Grupenet al\.,[2026b](https://arxiv.org/html/2606.04602#bib.bib40)\)scores matter\-style tasks with data rooms, deliverables, all\-pass rubrics, and a broad practice\-area/archetype mix \(Figure[2](https://arxiv.org/html/2606.04602#S2.F2)\)\. Yet a large\-scale empirical analysis of these systems is still missing, which leaves the true capabilities of state\-of\-the\-art agents on open\-ended legal matters poorly understood\.
Figure 2:Primary task\-archetype distribution by practice area in the Harvey LAB dataset\.Bars are normalized within each of Harvey LAB’s2424practice areas; colors denote the primary task clusters\.
##### Agent harnesses, externalization, and memory\.
The agent*harness*—the runtime mediating workspace, tools, traces, and output constraints—is the unit of long\-horizon evaluation\. SWE\-agent and SWE\-bench\(Yanget al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib71); Jimenezet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib72)\)make this explicit for software engineering; GAIA, WebArena, and AgentBench\(Mialonet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib69); Zhouet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib70); Liuet al\.,[2023a](https://arxiv.org/html/2606.04602#bib.bib68)\)provide settings where traces and tool use can be studied\. General\-purpose workspace agents \(Codex\-style, Claude Code, OpenCode\) externalize files, shell, search, and edits into an inspectable workspace\. Legal work additionally requires professional memory, provenance, deadlines, citation form, and review signals as versionable artifacts\. Yet none of these harnesses is built for the legal vertical, leaving such requirements as ad hoc prompt additions rather than first\-class architectural components\.
##### Non\-parametric agent learning\.
Non\-parametric agent improvement edits prompts, programs, and workflows rather than model weights\. Prompt and program optimization \(APE, OPRO, PromptBreeder, APO, TextGrad, PromptAgent, DSPy, GEPA;Zhouet al\.,[2022](https://arxiv.org/html/2606.04602#bib.bib56); Yanget al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib57); Fernandoet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib58); Pryzantet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib59); Yuksekgonulet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib34); Wanget al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib60); Khattabet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib33); Agrawalet al\.,[2025](https://arxiv.org/html/2606.04602#bib.bib61)\) and workflow search \(DyLAN, MetaGen, ADAS, AFlow, SEVerA;Liuet al\.,[2023b](https://arxiv.org/html/2606.04602#bib.bib45); Wanget al\.,[2026](https://arxiv.org/html/2606.04602#bib.bib38); Huet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib35); Zhanget al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib36); Banerjeeet al\.,[2026](https://arxiv.org/html/2606.04602#bib.bib48)\) treat the harness as a searchable optimization target, while multi\-agent systems\(Liet al\.,[2023a](https://arxiv.org/html/2606.04602#bib.bib62); Duet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib63); Honget al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib18); Qianet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib64); Wuet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib65); Chenet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib39)\)split roles across cooperating solvers\. Tool\-use work\(Schicket al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib17); Qianet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib54); Qinet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib55); Liet al\.,[2023b](https://arxiv.org/html/2606.04602#bib.bib67)\)and governance requirements\(Riedl and Desai,[2025](https://arxiv.org/html/2606.04602#bib.bib26); Mökanderet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib27); Wachteret al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib28)\)further motivate deterministic audit tools and human\-readable diffs\. None of this work, however, targets the expensive, leakage\-sensitive setting of legal rollouts, nor combines legal domain specialization with a self\-evolving framework\.
## 3TheParthenonFramework Architecture
### 3\.1Parthenonarchitecture
##### Design principles\.
Parthenonis built around three principles\.Compatibility: Codex, Claude Code, and related workspace harnesses already supply the long\-horizon execution needed for legal work, including file management, tool calling, iterative revision, and trace capture\. The framework therefore wraps them with legal state, deterministic tools, and procedural skills rather than replacing them\.Legal specialization: legal work carries hard professional invariants that general harnesses do not enforce\. A missed deadline can constitute malpractice, an ungrounded figure can invalidate a term, and an uncited authority is inadequate\.Parthenonturns these invariants into deterministic audit tools with mandatory execution, covering source traceability, date and number reconciliation, deliverable compliance, and issue\-lifecycle closure\. This replaces implicit model memory with an enforceable harness contract\.Dynamism: agent failures recur in the same procedural categories across models and practice areas, as documented in §[4](https://arxiv.org/html/2606.04602#S4)\. A self\-evolving loop therefore turns scored failures into reviewable harness diffs, mirroring how firms update checklists instead of retraining personnel\. This follows the lineage of tool\-using and self\-improving agents\(Schicket al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib17); Qinet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib55); Shinnet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib4); Madaanet al\.,[2023](https://arxiv.org/html/2606.04602#bib.bib5); Zhaoet al\.,[2024](https://arxiv.org/html/2606.04602#bib.bib31)\)but adds structural anti\-leakage\.
#### 3\.1\.1Model layer
The Model layer is apluggable capability provider, not a fixed binding\. Initial LAB results show no single frontier model dominates: different families lead in different legal sub\-domains\(Grupenet al\.,[2026a](https://arxiv.org/html/2606.04602#bib.bib41)\), and binding the framework to one model would convert those blind spots – weak temporal arithmetic, citation hallucination, thin practice\-area coverage – into system\-level failures\. A thin capability interface lets a deployment route matters by practice area, task shape, quality, or cost while the rest of the stack stays stable; because the model and agent harness stay fixed,Parthenon’s lift can be measured cleanly \(§[4\.3](https://arxiv.org/html/2606.04602#S4.SS3)\)\.
#### 3\.1\.2Harness layer
The Harness supplies theobservable execution contract: matter workspace, source/tool access, mediated read/search/execute/ write/validate/edit actions, and trace capture\. LAB trace analysis shows that in\-task behavior – searching before drafting, validating after – correlates with outcomes\(Grupenet al\.,[2026a](https://arxiv.org/html/2606.04602#bib.bib41)\), so the harness is where model capability becomes legal work, not merely a wrapper\.Parthenontreats it as pluggable: Codex, Claude Code, OpenCode, OpenHands, or basic legal\-native agents can occupy it so long as they expose workspace, tools, deliverable channel, and traces\. The legal contribution wraps that runtime, givingharness\-agnostic specialization: the legal layer runs across runtimes, and runtime\-specific behavior stays measurable\. Without this separation, an improvement could be a workspace\-ergonomic or a legal\-control change, and the harness could not migrate when the vendor changes\.
#### 3\.1\.3Agent layer
The Agent layer definesrole and information boundarieson top of the harness\. Reliability comes from separating three functions that must not share information – drafting the work product, evaluating it, and proposing harness edits – not from adding agents to drafting; the matter\-facing worker stays a single solver\. Collapsing these lets prompt\-optimization loops leak: a system that both drafts and grades against the rubric can eventually memorize it\. The split is therefore central to attribution and anti\-leakage\. Thesolverdrafts and self\-audits given task, sources, skill, and tools; theevaluatorscores the finished work against the rubric outside the solver’s context, so its judgments cannot leak into draft text; thelearnersees only redacted traces and aggregate signals and proposes task\-agnostic edits to Knowledge, Tools, or Skills, subject to automated rejection gates\. §[3\.2](https://arxiv.org/html/2606.04602#S3.SS2)specifies the loop over batches\.
SolverInput:TaskOutput:Legal Docs1LearnerInput:Traces\+\+ErrorsOutput:System Update3EvaluatorInput:Legal Docs\+\+RubricOutput:Score\+\+Analysis2TaskEnv\./Lawyer FeedbackCodebase
Figure 3:Self\-evolving loop inParthenon\.Thesolverdrafts legal documents, theevaluatorscores them against the rubric, and thelearnerturns redacted traces and errors into harness updates over the editable codebase \(tools, knowledge, skills\)\. Redaction between evaluator and learner \(§[3\.2](https://arxiv.org/html/2606.04602#S3.SS2)\) enforces single\-pass anti\-leakage\.
#### 3\.1\.4Knowledge layer
The Knowledge layer storesdurable legal memory as data, not prompt text, so a change to a deadline rule, authority, schema, or synonym is a reviewable harness edit rather than a hidden model behavior\. Knowledge artifacts do not execute checks; they give Tools structured content to retrieve and validate, and Skills stable legal objects to reference – institutional memory rather than ad hoc prompt content\. Table[1](https://arxiv.org/html/2606.04602#S3.T1)lists the six families used in our deployment\. The hard boundary is that Knowledge isnot an answer bank: a permissible edit adds a general authority, deadline rule, calendar, synonym, or work\-product schema, but never a task id, source fact, client identity, rubric phrase, or answer from the batch that exposed the failure\. Knowledge must be reusable across matters, otherwise it is benchmark leakage or a privacy risk\.
Table 1:Knowledge families inParthenon\.The Knowledge layer is organized into six reusable families totaling over 2,300 entries\. All entries are general\-purpose legal objects \(authorities, deadlines, schemas, calendars, synonyms, dispatch rules\) and must not encode matter\-specific facts, client identities, or benchmark\-derived answers\.
#### 3\.1\.5Tools layer
The Tools layer turns recurring legal requirements intodeterministic, interpretable operations\. Whenever a requirement reduces to inspection, retrieval, parsing, arithmetic, date logic, or citation and deliverable checks,Parthenonmakes it an executable tool rather than unstated prompt intent\. This matters because the most common failures \(§[4\.2](https://arxiv.org/html/2606.04602#S4.SS2)\) are mechanical but material – a wrong deadline, omitted source, unsupported citation, lost amount, malformed redline – each making an otherwise plausible answer unreleasable\. The surface follows the matter lifecycle \(Table[2](https://arxiv.org/html/2606.04602#S3.T2)\): pre\-draft tools canonicalize sources into typed matter state; retrieval tools query the Knowledge layer; inspectors parse spreadsheets, documents, emails, and summaries; computation tools reconcile numbers and dates; post\-draft release gates audit citation grounding, coverage, numeric/temporal consistency, and deliverable shape\. Skills decide when a tool is mandatory; Tools decide what can be checked without trusting model memory\.
Table 2:Tools\-layer:1414deterministic, agent\-callable capabilities, summarized here into the four legal\-work stages they serve and enumerated individually in Appendix[B\.1](https://arxiv.org/html/2606.04602#A2.SS1)\. Each stage enforces operational contracts that do not rely on model memory; the release gate consolidates citation, coverage, and deliverable checks into one reviewable report before any work product leaves the matter\.
#### 3\.1\.6Skills layer
The Skills layer is the most task\-facing content layer: it turns recurring legal work intorubric\-blind procedural plans\. A skill is selected for a matter class, not an answer, and specifies triage, required source coverage and issue lifecycle, mandatory tools, deliverable form, and the self\-audit that blocks finalization\. This layer is where empirical failures become reusable procedure: deliverable substitution→\\towork\-product identity checking; numeric/date loss→\\tomandatory reconciliation; single\-pass drafting→\\toexplicit decomposition; missing recommendations→\\toissue\-closure checklists\. Each skill follows a seven\-part scaffold \(Table[3](https://arxiv.org/html/2606.04602#S3.T3); an example appears in Appendix[12](https://arxiv.org/html/2606.04602#A2.T12)\) – triage, failure modes corrected, legal frameworks, analytical scaffolds, relationships, output structure, anti\-leakage checklist – and the final checklist forbids task ids, rubric phrases, client identities, deal amounts, dates, or private quotations from the batch that exposed the failure\. Skills are versioned artifacts the learner can edit, promoted only when the edit passes the acceptance gates\.
Table 3:Skills\-layer catalogue:1,2511\{,\}251task\-routed procedural skills\. A skill is selected by matter class, never by answer, and encodes a rubric\-blind plan – triage, coverage, mandatory tools, deliverable form, self\-audit gate\. Edits are versioned diffs, turning failure modes into reusable procedure rather than rubric memorization\.
### 3\.2Self\-Evolving Loop: Gated Harness Optimization
Parthenondoes not change the model; it changes the harness that each subsequent model call inhabits\. At steptt, the foundation modelMMand workspace harnessHHare fixed, while the editable harness
St=\(Kt,Tt,Gt,At\)S\_\{t\}=\(K\_\{t\},T\_\{t\},G\_\{t\},A\_\{t\}\)holds durable KnowledgeKtK\_\{t\}, the deterministic Tools surfaceTtT\_\{t\}, the Skills libraryGtG\_\{t\}, and role prompts \+ audit gatesAtA\_\{t\}\. A learning step is therefore a versioned, human\-readable harness commit, not a parameter update\.
#### 3\.2\.1Interfaces and update rule
A matter ism=\(q,X,d,C\)m=\(q,X,d,C\)with task briefqq, source setXX, deliverable specdd, and hidden rubricCC\(never given to the solver\)\. The three agent roles produce three signals:
Solver:\(y^,τ\)=SolveM\(q,X∣H,St,d\),\\text\{\{Solver:\}\}\\quad\(\\hat\{y\},\\tau\)=\\mathrm\{Solve\}\_\{M\}\(q,X\\mid H,S\_\{t\},d\),\(1\)Evaluator:z=EvaluateJ\(y^∣q,d,C\),\\text\{\{Evaluator:\}\}\\quad z=\\mathrm\{Evaluate\}\_\{J\}\(\\hat\{y\}\\mid q,d,C\),\(2\)Learner:Δ^t=LearnL\(Et,Zt,Repo\(St\)\),\\text\{\{Learner:\}\}\\quad\\widehat\{\\Delta\}\_\{t\}=\\mathrm\{Learn\}\_\{L\}\(E\_\{t\},Z\_\{t\},\\mathrm\{Repo\}\(S\_\{t\}\)\),\(3\)whereEtE\_\{t\}holds redacted failure trajectories andZtZ\_\{t\}the judge feedback over a without\-replacement minibatchBtB\_\{t\}; the traceτ\\tauserves as supervision signal and is never transcribed directly into the harness\. The harness then advances as
St\+1=St⊕Δ^t\.S\_\{t\+1\}=S\_\{t\}\\oplus\\widehat\{\\Delta\}\_\{t\}\.\(4\)The analogy to SGD is operational: the harness is discrete and reviewable, so⊕\\oplusdenotes the accepted effect of a bounded diff after rejection gates, not blind application of learner text\.
##### Anti\-leakage and rejection gates\.
Leakage protection is structural\. The solver receives task, sources, deliverable spec, and harness but never criterion ids, titles, or match criteria; the evaluator is post\-hoc and never feeds back during drafting; the learner sees only an abstract task shape, file names, bucketed tool\-use counts, a truncated final solver message, aggregate pass/fail counts, and de\-identified failure reasons\. The learner must emit matter\-independent edits\. A bounded learner diff is admitted only if \(i\) the feedback is general enough to be edited at all, \(ii\) any tool code compiles and passes static safety checks, and \(iii\) the candidate harness strictly improves the accepted per\-task pass rate\. Otherwise the previous harness is reinstated and the rejected candidate is logged\.
#### 3\.2\.2What the loop edits
EachΔ^t\\widehat\{\\Delta\}\_\{t\}is attributable to one harness surface:Tools\(Table[2](https://arxiv.org/html/2606.04602#S3.T2)\) turn mechanical legal invariants – dates, deadlines, amounts, coverage, citation presence, deliverable shape – into audited code rather than unstated prompt intent;Skills\(Table[3](https://arxiv.org/html/2606.04602#S3.T3)\) are rubric\-blind work plans naming inventory, issue lifecycle, mandatory tools, deliverable identity, and the finalization audit but never an answer;Knowledgestores durable legal objects \(authorities, windows, schemas, calendars, synonyms\) as data retrieved through tools\. The hard boundary is*procedure versus conclusion*: a skill that encodes a specific conclusion from one matter is a memorized answer and a client\-data leakage vector, so a rubric\-blind audit – given only task title and skill text – rewrites task\-specific content into generic procedure before commit\. A vocabulary check over the1,2511\{,\}251skills confirms that only domain\-general legal terms are shared with rubric criteria\. Empirical lift, concrete learner edits, and the hard\-1010optimization ledger appear in §[4\.4](https://arxiv.org/html/2606.04602#S4.SS4)\.
## 4Experiments and Analysis
### 4\.1Experimental setup and results
##### Benchmark and execution modes\.
Harvey LAB contains1,2511\{,\}251matters across2424practice areas \(median77source documents and5757criteria per matter; Figure[2](https://arxiv.org/html/2606.04602#S2.F2)\)\. Table[4](https://arxiv.org/html/2606.04602#S4.T4)reports four execution families: direct prompting \(API\), a basic legal\-native harness, and the Codex and Claude Code workspace harnesses\. Each baseline is paired with a same\-solverParthenoncolumn, so the lift isolates the legal harness rather than a model upgrade\. We report two metrics:*criterion accuracy*, the share of all rubric criteria passed, and*all\-pass*, the stricter share of matters that pass*every*criterion\.
##### Protocol\.
All full\-corpus cells share the task universe \(1,2511\{,\}251tasks\), judge, and deliverable\-shape conversion; solver settings are fixed within each paired comparison\. No solver reads judge outputs or failed\-criterion text during execution \(§[3\.2](https://arxiv.org/html/2606.04602#S3.SS2)’s anti\-leakage rules\)\. Each paired comparison holds the solver and its workspace fixed and adds only the optimizedParthenonharness\. All reasoning models run at themediumreasoning\-effort setting\.
Table 4:Baselines andParthenonon Harvey LAB\. All\-pass is strict matter pass rate; other rows are criterion accuracy\. Bold marks the better same\-solver paired cell; API/Basic are unpaired\. Domains are sorted by Codex/GPT\-5\.5 accuracy\.
##### Baseline pattern and framework effect\.
The first regularity is that a stronger base model raises*criterion*accuracy but not*matter*\-level completion\. Codex climbs from68\.268\.2to79\.8%79\.8\\%as the solver moves from GPT\-5\.4\-mini to GPT\-5\.5, and Claude Code with Sonnet 4\.6 reaches82\.8%82\.8\\%\. Yet even the strongest baseline passes every criterion on barely one matter in eight \(11\.8%11\.8\\%\)\. All\-pass is a conjunction over dozens of fact, authority, deadline, and deliverable conditions, so a few points of average accuracy cannot satisfy them jointly\. The bottleneck is therefore systemic rather than a matter of model scale\.
The second regularity is that, with the model and agent harness fixed,Parthenonalone delivers a lift comparable to a model upgrade at every tier:\+13\.8/\+10\.2/\+7\.4\+13\.8/\+10\.2/\+7\.4pp, reaching82\.0/89\.9/90\.2%82\.0/89\.9/90\.2\\%on mini, GPT\-5\.5, and Sonnet\. The gain is thus a property of the harness rather than of any base model\. The lift is largest where the base solver is weakest, which narrows the gap between solvers\. Strict completion roughly triples on the weaker solvers but barely moves on Sonnet\. §[4\.2](https://arxiv.org/html/2606.04602#S4.SS2)breaks that remaining error down by type\.
### 4\.2Baseline error analysis
##### Agents are active but incomplete\.
Outcome scores indicate whether a harness fails, not why\. We audit the full Codex/GPT\-5\.5 result cell:1,2511\{,\}251scored runs with paired native transcripts, final artifacts, and process metrics\. The cell passes about79\.8%79\.8\\%of all rubric criteria pooled, and the failed criteria anchor the Codex baseline column of Table[5](https://arxiv.org/html/2606.04602#S4.T5)\. We map judge rationales to ten scripted error classes and parse traces for shell activity, agent messages, and length\. The traces are dense: a typical run issues many shell commands and produces a long trajectory before emitting a finished artifact\. What the run lacks is closure\. Formulas, source\-specific facts, required sections, and rule conditions that were available in the data room do not reach the final work product, because the harness carries no release signal for “all required sources, numbers, authorities, and deliverables satisfied\.” Effort is not the bottleneck; verification is\.
##### Five classes carry most of the error\.
Five error classes dominate: missing facts, numbers/dates, legal\-rule use, deliverable form, and coverage\. Together they account for nearly two thirds of the failed criteria, with the remainder split across long\-tail or unnamed causes \(Table[5](https://arxiv.org/html/2606.04602#S4.T5), which gives the per\-cause rate for every cell\)\. These are mechanical but material misses, and they are diagnostic\. Workspace agents inspect many files yet keep no typed matter state, coverage manifest, or release gate, which is exactly what these classes demand\.Parthenonadds these controls and lowers the total error at every solver, with the form\- and grounding\-class errors falling fastest\.
Table 5:Error composition, baseline vs\.Parthenon\.Same columns as Table[4](https://arxiv.org/html/2606.04602#S4.T4)\. Each cell is the share of all criteria failing for a given cause \(judge\-rationale mapping\); cause rows sum to the*Total error*\(=100−=100\-pooled accuracy\)\. Bold marks the lower error rate in each same\-solver pair; API/Basic are unpaired\.Parthenonlowers total error at every solver and shrinks most causes\. The ten classes are defined in Appendix[B](https://arxiv.org/html/2606.04602#A2)\.
##### Difficulty varies by practice area; the kind of failure does not\.
The same Codex/GPT\-5\.5 error also splits by practice area\. The volume of failures varies widely across areas, tracking intrinsic matter difficulty such as document load and rubric density rather than any framework choice\. The composition of those failures, however, is stable\. The same five named classes dominate the error in every area, so the mix of error types is largely shared even where the difficulty is not\. A single set of cross\-area controls can therefore target most of the error, with per\-practice knowledge layered on top\.
### 4\.3Parthenon error reduction and comparison
With the model and agent harness unchanged, this subsection traces*how*Parthenonlowers the error rate – where it drops \(§[4\.3\.1](https://arxiv.org/html/2606.04602#S4.SS3.SSS1)\), what the agent does differently \(§[4\.3\.2](https://arxiv.org/html/2606.04602#S4.SS3.SSS2)–[4\.3\.3](https://arxiv.org/html/2606.04602#S4.SS3.SSS3)\), whether output length explains the gains \(§[4\.3\.4](https://arxiv.org/html/2606.04602#S4.SS3.SSS4)–[4\.3\.5](https://arxiv.org/html/2606.04602#S4.SS3.SSS5)\), and what the lift costs \(§[4\.3\.6](https://arxiv.org/html/2606.04602#S4.SS3.SSS6)\)\. Each effect is read across the four solver families: the headline tiers Codex/GPT\-5\.5 and Claude Code/Sonnet 4\.6, plus the lighter Codex/GPT\-5\.4\-mini and Claude Code/Haiku 4\.5\.
#### 4\.3\.1Error reduction by practice area
The harness lowers the error rate in almost every practice area, and the gains are largest where the baseline is weakest, so the cross\-area spread narrows \(Figure[4](https://arxiv.org/html/2606.04602#S4.F4)\)\. This holds across all four solver families, including the much stronger Claude Code/Sonnet 4\.6 baseline, which confirms the effect is not a low\-baseline artifact\. Because the failure modes are shared across practice areas, a small set of cross\-practice controls – source\-coverage closure, numeric and date reconciliation, deliverable validation, and authority grounding – handles most of the error, with practice\-specific knowledge layered on top\. This is exactly the divisionParthenonencodes as shared harness and tools versus per\-practice skills\.
Figure 4:Error rate by practice area, baseline vs\.Parthenon, four solver families\.Areas ordered hardest \(left\) to easiest \(right\) by Codex/GPT\-5\.5 baseline error rate\. One dumbbell per family per area, grouped Codex \(indigo, left\) and Claude Code \(terracotta, right\): the headline tiers Codex/GPT\-5\.5 \(circles\) and Claude Code/Sonnet 4\.6 \(squares\) in saturated color, the lighter tiers Codex/GPT\-5\.4\-mini \(faded diamonds\) and Claude Code/Haiku 4\.5 \(faded pentagons\) on the outer offsets; hollow = baseline, filled =Parthenon, connector = error drop\. Every family improves nearly every area \(Codex24/2424/24at both tiers, Claude23/2423/24\), most on the hardest, and the weaker base solvers \(mini, Haiku\) start higher and drop further\.
#### 4\.3\.2Action intensity per matter
The framework adds more structured actions per matter \(Figure[5](https://arxiv.org/html/2606.04602#S4.F5)\)\. The added events are mandatory inspections, retrievals, schema fills, and release\-gate audits, and the largest rises fall in the schema\-heavy domains that the Skills layer most expands\. The two solver families count actions on different scales: Codex issues many fine\-grained shell commands, while Claude Code issues fewer coarse tool calls, so the absolute counts are not comparable across families\. Read instead the within\-family rise: for example, mean native trace actions on the Claude Code/Sonnet 4\.6 family rise from6\.86\.8to20\.620\.6per matter\.
Figure 5:Actions per matter,2424practice areas, four solver families\.Same area order and marker scheme as Figure[4](https://arxiv.org/html/2606.04602#S4.F4): headline tiers Codex/GPT\-5\.5 \(circles\) and Claude Code/Sonnet 4\.6 \(squares\) in saturated color, lighter tiers Codex/GPT\-5\.4\-mini \(faded diamonds\) and Claude Code/Haiku 4\.5 \(faded pentagons\); hollow = baseline, filled =Parthenon, connector = rise in actions\. The two trace formats count actions differently \(an action is one tool call; Codex uses many fine\-grained shell commands – the indigo band near the top – whereas Claude Code issues fewer coarse tool calls – the terracotta band below\), so read the within\-family increase, not cross\-family magnitude\. Both tiers of each family rise underParthenon\. Buckets in Table[6](https://arxiv.org/html/2606.04602#S4.T6)\.
#### 4\.3\.3Action mix: a new tool/script bucket
The added actions form a new tool and script bucket rather than more writing\. Table[6](https://arxiv.org/html/2606.04602#S4.T6)decomposes each cell’s solver actions into four behavioral buckets recovered from the renderable trace\. The baseline is a near\-pure read trajectory dominated by*read/inspect*actions, with almost no*tool/script*use\. UnderParthenonthe new*tool/script*bucket accounts for essentially the entire increment: on Codex/GPT\-5\.5 it rises from0\.20\.2to16\.516\.5actions per matter, made up of Skills\-driven invocations ofsummarize\_docs,deliverable\_search, schema fillers, and audit utilities that the baseline does not have\. Over the same change, read/inspect edges down and*write/edit*barely moves, so the framework is not producing more text but a structured tool layer\. The pattern holds at every solver, the tool/script bucket appearing where the baseline had none, while write and edit stay a small fraction throughout\. The matter\-level shift is broadly positive, a clear majority of paired tasks improving\. The framework changes*what the solver does*, not how much text it emits\.
Table 6:Action mix, baseline vs\.Parthenon\.Mean number of solver trace actions*per matter*in each behavioral bucket, for the four solver families with renderable traces; the bottom row is the mean total actions per matter, and bold marks the larger value in each baseline/Parthenonpair\. Buckets:*read/inspect*\(rg/cat/sed/jq/wc\),*tool/script*\(Skills interpreter calls\),*write/edit*\(file changes\),*other*\. The framework adds a large*tool/script*bucket at every solver; per\-area variation is small, so counts are pooled over the2424practice areas\.
#### 4\.3\.4Deliverable length
Output length does not explain the gains\. Deliverables shrink for some solver families and grow for others, with the direction flipping by family, yet accuracy rises for all four \(Figure[6](https://arxiv.org/html/2606.04602#S4.F6)\)\. On Codex/GPT\-5\.5 deliverables get shorter where accuracy rises most, while the mini, Sonnet, and Haiku deliverables grow\. Since more text accompanies both larger and smaller accuracy gains, emitting more text is not the mechanism behind the improvement\.
Figure 6:Deliverable length by practice area, baseline vs\.Parthenon, four solver families \(1,0001\{,\}000words/matter\)\.Same layout and marker scheme as Figure[5](https://arxiv.org/html/2606.04602#S4.F5): Codex/GPT\-5\.5 \(circles\) and Sonnet 4\.6 \(squares\) in saturated color, faded Codex/GPT\-5\.4\-mini \(diamonds\) and Haiku 4\.5 \(pentagons\); hollow = baseline, filled =Parthenon, connector = change\. The direction flips by family: GPT\-5\.5 deliverables*shrink*while mini, Sonnet, and Haiku*grow*, yet accuracy rises for every family \(Table[4](https://arxiv.org/html/2606.04602#S4.T4)\), so length does not explain the gains\.
#### 4\.3\.5Agent\-trajectory length
The size of the lift tracks how hard the baseline found the matter, not how much text the agent produces\. Agent\-trajectory length, the total agent and tool text in the native trace, moves in mixed directions: the Codex tiers grow or hold while the Claude Code tiers shrink \(Figure[7](https://arxiv.org/html/2606.04602#S4.F7)\)\. This length is decoupled from the uniformly positive accuracy lift\. What the lift does track is baseline difficulty, with the largest gains falling on the matters the baseline handled worst and the matters it already passed gaining little\.
Figure 7:Agent\-trajectory length by practice area, baseline vs\.Parthenon, four solver families \(1,0001\{,\}000words/matter\)\.Total agent and tool text in the native trace; same layout and markers as Figure[6](https://arxiv.org/html/2606.04602#S4.F6)\. The direction is again mixed – the Codex tiers \(indigo\) grow or hold while the Claude Code tiers \(terracotta\) shrink – decoupled from the uniformly positive accuracy lift\.
#### 4\.3\.6What the lift costs
Whether the harness raises or lowers per\-matter cost depends on the base solver\. Figure[8](https://arxiv.org/html/2606.04602#S4.F8)prices each matter from logged token usage at public list rates \(cached reads free; judge and evaluation tokens excluded; App\.[A\.3](https://arxiv.org/html/2606.04602#A1.SS3)\)\. One cell is an exception: the Codex/GPT\-5\.5 baseline traces do not expose billable token counts, so its cost is the original full\-cell logged estimate rather than a token recomputation\. Read against the accuracy of Table[4](https://arxiv.org/html/2606.04602#S4.T4), the framework does not buy accuracy at a fixed token premium\. On the expensive, exploration\-heavy GPT\-5\.5 the harness is actually cheaper, at $1\.51→\\to$1\.29 per matter while accuracy rises79\.8→89\.9%79\.8\\to 89\.9\\%\. The reason is that the same audit loop that adds tool calls also yields shorter audited output, and on a solver whose output tokens dominate the bill that reduction more than offsets the extra calls\. The cheaper solvers show the opposite sign, spending a little more underParthenon, but the surcharge is small relative to the accuracy it buys\.Parthenon/mini reaches82\.0%82\.0\\%for $0\.66, clearing the GPT\-5\.5 baseline’s79\.8%79\.8\\%at under half that baseline’s cost\.Parthenon/Sonnet reaches the corpus\-best90\.2%90\.2\\%for $0\.81\. With per\-matter cost spanning more than an order of magnitude across families, the four pairings offer a range of operating points: the cheapest acceptable draft fromParthenon/mini, the highest accuracy fromParthenon/Sonnet, and, uniquely at the top Codex tier, accuracy and cost improving together\.
Figure 8:Cost per matter by practice area, baseline vs\.Parthenon, four solver families\.Mean USD per matter \(*log*scale\) from logged token usage×\\timespublic list prices, cached\-read\-free; same cell layout and area order as Figure[5](https://arxiv.org/html/2606.04602#S4.F5)\. Markers: Codex/GPT\-5\.5 \(circles\), Codex/GPT\-5\.4\-mini \(faded diamonds\), Claude Code/Sonnet 4\.6 \(squares\), Claude Code/Haiku 4\.5 \(faded pentagons\); hollow = baseline, filled =Parthenon\. Costs span more than an order of magnitude across families;Parthenon/GPT\-5\.5 is the only pair whose cost*falls*\($1\.51→\\to$1\.29\), while the cheaper solvers spend more under the framework\. All cells are recomputed from logged tokens with cached reads free, except the Codex/GPT\-5\.5 baseline, whose clean traces do not expose billable tokens and whose cost is the original logged estimate\. Judge and evaluation tokens are excluded\.##### What remains\.
Even with the strongest pairing \(Parthenon/Sonnet 4\.6\),∼10%\\sim 10\\%of criteria still fail, concentrated in recall and reasoning misses \(missing facts, numbers/dates\) and rationales the scripted taxonomy does not yet name \(Table[5](https://arxiv.org/html/2606.04602#S4.T5)\); the form\- and grounding\-class errors are nearly gone\. That remainder is the target of continued model progress and the continual\-learning loop \(§[3\.2](https://arxiv.org/html/2606.04602#S3.SS2)\), not of more workspace machinery\. As a caveat, per\-area accuracy is confounded with task mix, rubric density, and document count, so these are descriptive comparisons over a fixed task universe, not causal attributions\.
### 4\.4Ablation study
Three ablations probe*where*the framework’s gain comes from, and where it does not\. The first traces the self\-improvement loop that builds the harness; the second varies raw inference budget on a fixed harness; the third adds a representative tool, cached document summaries\. Read together they point the same way: the dependable source of improvement is retained, audited*procedure*, rather than additional compute or any single added tool\.
#### 4\.4\.1Harness optimization
The self\-improvement loop retains a transferable harness rather than the output of a single favorable run\. We run the loop \(§[3\.2](https://arxiv.org/html/2606.04602#S3.SS2)\) on the hardest ten tasks with two unrelated solver/learners – Codex \(GPT\-5\.4\-mini\) and Claude Code \(Haiku 4\.5\) – each starting from the same empty harness and using the same evaluation panel and scoring procedure\. The raw per\-step candidates never settle, swinging by several points between adjacent steps\. A strict best\-so\-far gate that keeps only audited gains converts this noise into a monotone, non\-decreasing climb; for Codex the gated accuracy rises from45\.3%45\.3\\%to55\.8%55\.8\\%over the run\. The central result is that the two solvers, despite different base models, converge to within0\.40\.4pp of one another\. The gated climb is stepwise rather than smooth: durable gains arrive as discrete jumps when an audited edit lands and then hold on a plateau – the signature of adding a missing control rather than tuning a model\. What accumulates across steps is therefore a transferable, inspectable harness, the product of selecting and retaining audited edits rather than a single stochastic sample from either model\.
Figure 9:Per\-task optimization trajectories\.One panel per hard\-1010task, ordered by final Codex accuracy\. Solid lines: gated best\-so\-far accuracy for Codex \(GPT\-5\.4\-mini\) and Claude Code \(Haiku 4\.5\) as joint solver/learner; faint dots: raw candidate pass rates\. Strict best\-so\-far gate makes the frontier monotone non\-decreasing\.
#### 4\.4\.2Reasoning effort
Raw inference budget helps on average but does so unreliably\. On the same ten tasks \(545545criteria\), increasing the coding\-agent’s effort setting raises mean accuracy, yet neither monotonically nor consistently across solvers\. For Codex/GPT\-5\.5 the medium baseline improves at thehighsetting and then regresses at the topxhighsetting\. Claude Code/Sonnet inverts this ordering: its medium baseline beats bothlowandhigh, and only themaxsetting edges ahead\. Per\-task winners scatter across settings rather than aligning on a single level, and on several matters the extra budget amplifies an unhelpful trajectory instead of correcting it\. The two solvers often move in opposite directions at the same setting, and at the highest setting a few matters collapse outright – accuracy falling by more than half from the medium baseline – as the extra reasoning entrenches a wrong approach\. Set against the monotone, transferable climb of the optimization loop \(Figure[9](https://arxiv.org/html/2606.04602#S4.F9)\), raw budget is the fragile lever and audited procedure the dependable one\. Effort is therefore a hyperparameter worth tuning, but not a substitute for the procedural controls and task\-specific harness learning that move accuracy more dependably\.
Figure 10:Reasoning\-effort ablation\.Same hard\-1010tasks as Figure[9](https://arxiv.org/html/2606.04602#S4.F9), run with Codex \(GPT\-5\.5\) and Claude Code \(Sonnet 4\.6\) while varying the coding\-agent CLI effort setting\. Hollow markers denote the medium\-effort baseline\. Thetoptick isxhighfor Codex andmaxfor Claude Code; other cells are matched low/high/top reruns\. Panels are ordered by best observed accuracy across effort levels\.
#### 4\.4\.3Document\-summary tool
The long\-input gap reflects retrieval discipline, not the absence of a compact summary\. A cached per\-document summary is the obvious remedy for matters with many sources, yet enabling it does not pay off\. Per\-task deltas center on zero and tilt slightly negative, so the cache modestly hurts short matters\. The reason is that the real handicap is input length itself: accuracy declines steadily as the number of source documents grows, and the cache leaves that decline essentially untouched\. If a compact overview were the fix, its benefit should grow with corpus size; instead it shows no trend with input length and gains and losses roughly cancel, so the tool fails exactly where long inputs hurt most\. The sharpest regressions all involve row\-level fact work \(identify,extract,compare,review,assess\), where the solver consumes the summary as compressed source text rather than using it as an index back into the originals\. The lift therefore comes from procedural controls that enforce disciplined retrieval and verification, not from shorter or orienting context\.
Figure 11:Document\-summary tool ablation\.Both panels use the same paired GPT\-5\.4\-mini Codex tasks\. \(A\) Baseline pass rate vs\. total source\-document chars \(log\); pass rate declines as inputs grow\. \(B\) Per\-task delta from enabling the cached document\-summary tool, centered near zero; the tool does not close the long\-input gap\.
### 4\.5Human\-lawyer reference: accuracy, time, and cost
The deployment target is to accelerate a supervising lawyer, not to replace one\. Harvey LAB has no controlled human\-lawyer run on the1,2511\{,\}251matters\. Rather than invent a human accuracy score, we treat the lawyer as the*release*standard that the benchmark approximates\. A work product is acceptable only when every material fact, authority, deadline, number, and deliverable requirement survives review, an all\-or\-nothing standard we denote100%100\\%\. Two readings of Table[7](https://arxiv.org/html/2606.04602#S4.T7)pull in opposite directions\. On accuracy, the system is still a draft assistant\. EvenParthenon’s strongest configuration \(Claude Code/Sonnet 4\.6\) passes90\.2%90\.2\\%of criteria, yet it clears*every*criterion on only150150matters, leaving1,1011\{,\}101that a lawyer must still finish\. On time and cost, the comparison already favors the system, though the two sides are measured differently\. Per\-matter cost is computed from logged tokens at $0\.81, thousands of times below the $4,3994\{,\}399blended human estimate\. Agent wall\-clock was not logged, so the time gap is an order\-of\-magnitude estimate rather than a measurement: against a human’s12\.612\.6h blended figure \(33–2020h by complexity\), a per\-complexity bucket estimate of roughly ten minutes per matter implies a speedup on the order of70×70\\times\. Both gaps precede any review\. The implication is a change of role rather than a replacement\. The framework converts the expensive part of a matter from drafting from scratch into reviewing a source\-grounded, audit\-flagged first draft, so the lawyer’s hours go to residual legal judgment rather than document assembly\. The accuracy gap says it is not autonomous; the time\-and\-cost gap says it is already useful\.
Table 7:Per\-practice\-area comparison against the release standard \(2424Harvey LAB domains, sorted by AI accuracy\)\.*Accuracy:*AI is the measuredParthenon\(Claude Code/Sonnet 4\.6, our strongest configuration\) criterion accuracy; human accuracy is the release standard LAB approximates\.*Time:*human times are matter\-complexity bucket estimates \(3 h screen / 10 h diligence / 20 h complex\); AI times are matter\-complexity bucket estimates\.*Cost:*human at $349/h\(Clio,[2026](https://arxiv.org/html/2606.04602#bib.bib44)\); AI cost is the measured per\-matter cost from logged tokens \(cached\-read\-free\)\.
## 5Conclusion
Stronger base models still fail legal work for a reason that is largely procedural rather than parametric: across models and practice areas, agents fail in the same professional categories – quantitative detail, missing facts, deliverable form, rule application, and source grounding – because the workspace enforces no contract for when a matter is complete\.Parthenoncloses this gap without retraining, wrapping the solver in an auditable legal harness: deterministic tools that enforce a matter’s invariants, procedural skills that encode how to work a matter class, and a self\-evolving loop that turns scored failures into reviewable harness diffs under strict anti\-leakage\. With the base model and agent runtime fixed, the harness delivers a lift comparable to a model upgrade at every solver tier and transfers across unrelated base models, so the gain is a property of the harness rather than of any one model\. The system is not an autonomous lawyer\. Against an all\-or\-nothing release bar it still clears every criterion on only a minority of matters, so its role is to accelerate a supervising lawyer: it converts drafting from scratch into reviewing a source\-grounded, audit\-flagged draft and leaves residual legal judgment to the attorney\. More broadly, when reliability is bounded by procedure rather than raw capability, an external harness that a human can inspect and a system can improve from its own failures may carry further than the next model\.
## References
- L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. Khattab \(2025\)GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.19457)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Banerjee, C\. Xu, E\. Ie, M\. Zhang, D\. Peng, C\. Lin, and G\. Singh \(2026\)SEVerA: Verified Synthesis of Self\-Evolving Agents\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2603.25111)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- I\. Chalkidis, A\. Jana, D\. Hartung, M\. J\. Bommarito, I\. Androutsopoulos, D\. M\. Katz, and N\. Aletras \(2021\)LexGLUE: A Benchmark Dataset for Legal Language Understanding in English\.SSRN Electronic Journal\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.3936759)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Chen, Y\. Su, J\. Zuo, C\. Yang, C\. Yuan, C\. Chan, H\. Yu, Y\. Lu, Y\. Hung, C\. Qian, Y\. Qin, X\. Cong, R\. Xie, Z\. Liu, M\. Sun, and J\. Zhou \(2023\)AgentVerse: Facilitating Multi\-Agent Collaboration and Exploring Emergent Behaviors\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.10848)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- Clio \(2026\)How much should i charge as a lawyer? compare average lawyer hourly rate by state\.Note:[https://www\.clio\.com/resources/legal\-trends/compare\-lawyer\-rates/](https://www.clio.com/resources/legal-trends/compare-lawyer-rates/)Updated March 2026; accessed 2026\-05\-26Cited by:[§A\.3](https://arxiv.org/html/2606.04602#A1.SS3.SSS0.Px2.p1.5),[Table 7](https://arxiv.org/html/2606.04602#S4.T7)\.
- J\. Cui, M\. Ning, Z\. Li, B\. Chen, Y\. Yan, H\. Li, B\. Ling, Y\. Tian, and L\. Yuan \(2023\)Chatlaw: A Multi\-Agent Collaborative Legal Assistant with Knowledge Graph Enhanced Mixture\-of\-Experts Large Language Model\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2306.16092)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Dahl, V\. Magesh, M\. Suzgun, and D\. E\. Ho \(2024\)Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models\.Journal of Legal Analysis\.External Links:[Document](https://dx.doi.org/10.1093/jla/laae003)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2023\)Improving Factuality and Reasoning in Language Models through Multiagent Debate\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2305.14325)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- R\. El Hamdani, T\. Bonald, F\. Malliaros, N\. Holzenberger, and F\. Suchanek \(2024\)The Factuality of Large Language Models in the Legal Domain\.Proceedings of the 33rd ACM International Conference on Information and Knowledge Management\.External Links:[Document](https://dx.doi.org/10.1145/3627673.3679961)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Fei, X\. Shen, D\. Zhu, F\. Zhou, Z\. Han, A\. Huang, S\. Zhang, K\. Chen, Z\. Yin, Z\. Shen, J\. Ge, and V\. Ng \(2024\)LawBench: Benchmarking Legal Knowledge of Large Language Models\.Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.452)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Fernando, D\. Banarse, H\. Michalewski, S\. Osindero, and T\. Rocktaschel \(2023\)PromptBreeder: Self\-Referential Self\-Improvement via Prompt Evolution\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2309.16797)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Grupen, G\. Pereyra, and J\. Pereyra \(2026a\)Initial results on legal agent benchmark\.Note:[https://www\.harvey\.ai/blog/legal\-agent\-benchmark\-initial\-results](https://www.harvey.ai/blog/legal-agent-benchmark-initial-results)Published May 26, 2026; accessed 2026\-05\-26Cited by:[§3\.1\.1](https://arxiv.org/html/2606.04602#S3.SS1.SSS1.p1.1),[§3\.1\.2](https://arxiv.org/html/2606.04602#S3.SS1.SSS2.p1.1)\.
- N\. Grupen, G\. Pereyra, and J\. Pereyra \(2026b\)Open\-sourcing Harvey’s long horizon legal agent benchmark\.Note:[https://www\.harvey\.ai/blog/introducing\-harveys\-legal\-agent\-benchmark](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark)Accessed 2026\-05\-25Cited by:[§1](https://arxiv.org/html/2606.04602#S1.p1.1),[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Guha, J\. Nyarko, D\. E\. Ho, C\. Ré, A\. Chilton, A\. Narayana, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. Rockmore, D\. Zambrano, D\. Talisman, E\. Hoque, F\. Surani, F\. Fagan, G\. Sarfaty, G\. M\. Dickinson, H\. Porat, J\. Hegland, J\. Wu, J\. Nudell, J\. Niklaus, J\. Nay, J\. H\. Choi, K\. Tobia, M\. Hagan, M\. Ma, M\. A\. Livermore, N\. Rasumov\-Rahe, N\. Holzenberger, N\. Kolt, P\. Henderson, S\. Rehaag, S\. Goel, S\. Gao, S\. Williams, S\. Gandhi, T\. Zur, V\. Iyer, and Z\. Li \(2023\)Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models\.SSRN Electronic Journal\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.4583531)Cited by:[§1](https://arxiv.org/html/2606.04602#S1.p1.1),[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- Harvey AI \(2026a\)Harvey Labs architecture\.Note:[https://github\.com/harveyai/harvey\-labs/blob/main/docs/architecture\.md](https://github.com/harveyai/harvey-labs/blob/main/docs/architecture.md)Accessed 2026\-05\-26Cited by:[§A\.1](https://arxiv.org/html/2606.04602#A1.SS1.SSS0.Px3.p1.1)\.
- Harvey AI \(2026b\)Harvey Labs harness directory\.Note:[https://github\.com/harveyai/harvey\-labs/tree/main/harness](https://github.com/harveyai/harvey-labs/tree/main/harness)Accessed 2026\-05\-26Cited by:[§A\.1](https://arxiv.org/html/2606.04602#A1.SS1.SSS0.Px3.p1.1)\.
- D\. Hendrycks, C\. Burns, A\. Chen, and S\. Ball \(2021\)CUAD: An Expert\-Annotated NLP Dataset for Legal Contract Review\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2103.06268)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber \(2023\)MetaGPT: Meta Programming for A Multi\-Agent Collaborative Framework\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.00352)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Hu, C\. Lu, and J\. Clune \(2024\)Automated Design of Agentic Systems\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2408.08435)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2023\)SWE\-bench: Can Language Models Resolve Real\-World GitHub Issues?\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.06770)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Kantet al\.\(2025\)Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2502.17638)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2023\)DSPy: Compiling Declarative Language Model Calls into Self\-Improving Pipelines\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.03714)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem \(2023a\)CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society\.Advances in Neural Information Processing Systems 36\.External Links:[Document](https://dx.doi.org/10.52202/075280-2264)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Li, J\. Chen, J\. Yang, Q\. Ai, W\. Jia, Y\. Liu, K\. Lin, Y\. Wu, G\. Yuan, Y\. Hu, W\. Wang, Y\. Liu, and M\. Huang \(2025\)LegalAgentBench: Evaluating LLM Agents in Legal Domain\.Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.116)Cited by:[§1](https://arxiv.org/html/2606.04602#S1.p1.1),[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li \(2023b\)API\-Bank: A Comprehensive Benchmark for Tool\-Augmented LLMs\.Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.187)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang \(2023a\)AgentBench: Evaluating LLMs as Agents\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.03688)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, Y\. Zhang, P\. Li, Y\. Liu, and D\. Yang \(2023b\)A Dynamic LLM\-Powered Agent Network for Task\-Oriented Agent Collaboration\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.02170)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-Refine: Iterative Refinement with Self\-Feedback\.Advances in Neural Information Processing Systems 36\.External Links:[Document](https://dx.doi.org/10.52202/075280-2019)Cited by:[§3\.1](https://arxiv.org/html/2606.04602#S3.SS1.SSS0.Px1.p1.1)\.
- A\. Mantravadiet al\.\(2025\)LegalWiz: A Multi\-Agent Generation Framework for Contradiction Detection in Legal Documents\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.03418)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom \(2023\)GAIA: A Benchmark for General AI Assistants\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2311.12983)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Mökander, J\. Schuett, H\. R\. Kirk, and L\. Floridi \(2023\)Auditing Large Language Models: A Three\-Layered Approach\.SSRN Electronic Journal\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.4361607)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Pipitone and G\. Houir Alami \(2024\)LegalBench\-RAG: A Benchmark for Retrieval\-Augmented Generation in the Legal Domain\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2408.10343)Cited by:[§1](https://arxiv.org/html/2606.04602#S1.p1.1),[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Pryzant, D\. Iter, J\. Li, Y\. T\. Lee, C\. Zhu, and M\. Zeng \(2023\)Automatic Prompt Optimization with “Gradient Descent” and Beam Search\.Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.494)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Qian, W\. Liu, H\. Liu, N\. Chen, Y\. Dang, J\. Li, C\. Yang, W\. Chen, Y\. Su, X\. Cong, J\. Xu, D\. Li, Z\. Liu, and M\. Sun \(2024\)ChatDev: Communicative Agents for Software Development\.Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.810)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Qian, C\. Han, Y\. R\. Fung, Y\. Qin, Z\. Liu, and H\. Ji \(2023\)CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models\.Findings of the Association for Computational Linguistics: EMNLP 2023\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.462)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2023\)ToolLLM: Facilitating Large Language Models to Master 16000\+ Real\-world APIs\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2307.16789)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2606.04602#S3.SS1.SSS0.Px1.p1.1)\.
- M\. O\. Riedl and D\. R\. Desai \(2025\)AI Agents and the Law\.Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society\.External Links:[Document](https://dx.doi.org/10.1609/aies.v8i3.36705)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: Language Models Can Teach Themselves to Use Tools\.Advances in Neural Information Processing Systems 36\.External Links:[Document](https://dx.doi.org/10.52202/075280-2997)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2606.04602#S3.SS1.SSS0.Px1.p1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems 36\.External Links:[Document](https://dx.doi.org/10.52202/075280-0377)Cited by:[§3\.1](https://arxiv.org/html/2606.04602#S3.SS1.SSS0.Px1.p1.1)\.
- S\. Wachter, B\. Mittelstadt, and C\. Russell \(2024\)Do large language models have a legal duty to tell the truth?\.Royal Society Open Science\.External Links:[Document](https://dx.doi.org/10.1098/rsos.240197)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Wang, C\. Li, Z\. Wang, F\. Bai, H\. Luo, J\. Zhang, N\. Jojic, E\. P\. Xing, and Z\. Hu \(2023\)PromptAgent: Strategic Planning with Language Models Enables Expert\-level Prompt Optimization\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.16427)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Wang, J\. Zhao, H\. Xie, H\. Ma, Y\. Lei, S\. Liu, X\. Song, Z\. Zhang, and H\. Zhang \(2026\)MetaGen: Self\-Evolving Roles and Topologies for Multi\-Agent LLM Reasoning\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.19290)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2023\)AutoGen: Enabling Next\-Gen LLM Applications via Multi\-Agent Conversation\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.08155)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen \(2023\)Large Language Models as Optimizers\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2309.03409)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)SWE\-agent: Agent\-Computer Interfaces Enable Automated Software Engineering\.Advances in Neural Information Processing Systems 37\.External Links:[Document](https://dx.doi.org/10.52202/079017-1601)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, Z\. Huang, C\. Guestrin, and J\. Zou \(2024\)TextGrad: Automatic "Differentiation" via Text\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2406.07496)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Zhang, J\. Xiang, Z\. Yu, F\. Teng, X\. Chen, J\. Chen, M\. Zhuge, X\. Cheng, S\. Hong, J\. Wang, B\. Zheng, B\. Liu, Y\. Luo, and C\. Wu \(2024\)AFlow: Automating Agentic Workflow Generation\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.10762)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024\)ExpeL: LLM Agents Are Experiential Learners\.Proceedings of the AAAI Conference on Artificial Intelligence\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i17.29936)Cited by:[§3\.1](https://arxiv.org/html/2606.04602#S3.SS1.SSS0.Px1.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2023\)WebArena: A Realistic Web Environment for Building Autonomous Agents\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2307.13854)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. Ba \(2022\)Large Language Models Are Human\-Level Prompt Engineers\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2211.01910)Cited by:[§2](https://arxiv.org/html/2606.04602#S2.SS0.SSS0.Px3.p1.1)\.
Appendix
## Appendix AEvaluation Protocol and Measurement
This appendix records the measurement details that the main text uses but does not spell out: the corpus and scoring definitions, how raw traces are turned into the action buckets of §[4\.3\.2](https://arxiv.org/html/2606.04602#S4.SS3.SSS2), and how the cost and time figures of §[4\.3\.6](https://arxiv.org/html/2606.04602#S4.SS3.SSS6)and §[4\.5](https://arxiv.org/html/2606.04602#S4.SS5)are computed\. It is deliberately not a per\-task data dump: the full scored outputs, native transcripts, and derived metrics are released with the project data pack \(App\.[B\.8](https://arxiv.org/html/2606.04602#A2.SS8)\)\.
### A\.1Corpus, cells, and metrics
##### Benchmark universe\.
The main results use the full Harvey LAB corpus, not the smaller hard\-task optimization subset:1,2511\{,\}251matters,2424practice areas, median5757criteria per matter \(range2323–194194\)\. Source\-file counts are reconstructed from the task corpus, which gives median77source files per matter\. The hard\-1010optimization experiment \(App\.[B\.6](https://arxiv.org/html/2606.04602#A2.SS6)\) is a separate harness\-learning panel and is not pooled into the full\-corpus tables\.
##### Scoring formulas\.
For scored run setRR, let taskiihavecic\_\{i\}criteria andpip\_\{i\}passed criteria\. The paper reports
PooledPass=∑ipi∑ici,AllPass=1\|R\|∑i∈R𝟏\[pi=ci\]\.\\mathrm\{PooledPass\}=\\frac\{\\sum\_\{i\}p\_\{i\}\}\{\\sum\_\{i\}c\_\{i\}\},\\qquad\\mathrm\{AllPass\}=\\frac\{1\}\{\|R\|\}\\sum\_\{i\\in R\}\\mathbf\{1\}\[p\_\{i\}=c\_\{i\}\]\.Pooled pass is criterion\-weighted; all\-pass is matter\-level and requires every criterion in a matter to pass\. The two diverge sharply because all\-pass is a conjunction: a single missed criterion fails the matter, so the metric is far more sensitive to errors than the pooled average\.
Table 8:Run inventory used by Table[4](https://arxiv.org/html/2606.04602#S4.T4)\. Listed cells cover the full1,2511\{,\}251\-matter corpus\.
##### Basic legal\-native harness\.
The Basic column is the benchmark\-provided native runner, not Harvey’s proprietary product environment\. It is the open\-source[Harvey AI](https://arxiv.org/html/2606.04602#bib.bib42)harness contract released with Harvey LAB\(Harvey AI,[2026b](https://arxiv.org/html/2606.04602#bib.bib42)\): tasks are filesystem workspaces, the harness loads prompts, state, model adapters, and workspace tools, and separate evaluation/reporting steps score outputs against rubric criteria\(Harvey AI,[2026a](https://arxiv.org/html/2606.04602#bib.bib43)\)\. We call it*Basic*to distinguish it from external workspace agents such as Codex and Claude Code\.
##### GPT\-5\.5 error extraction\.
The baseline error analysis \(§[4\.2](https://arxiv.org/html/2606.04602#S4.SS2)\) and the lift decomposition \(§[4\.3](https://arxiv.org/html/2606.04602#S4.SS3)\) use the recursive Codex/GPT\-5\.5 cell over the full corpus:1,2511\{,\}251scored runs,59,81959\{,\}819passed and15,17115\{,\}171failed criteria\. The failure taxonomy is assigned over those failed criteria, and the per\-area composition figures are computed from the same assignments\.
### A\.2From traces to action buckets
The per\-matter action counts in §[4\.3\.2](https://arxiv.org/html/2606.04602#S4.SS3.SSS2)–[4\.3\.3](https://arxiv.org/html/2606.04602#S4.SS3.SSS3)are recovered from the native execution traces and sorted into four buckets –*read/inspect*,*tool/script*,*write/edit*, and*other*– by a deterministic classifier, with unrecognized actions falling through to*other*\. The two solver families expose different trace formats, so the classifier is format\-aware\.
##### Codex \(shell grammar\)\.
Codex emits fine\-grained shell commands\. Each executed command is bucketed by its leading verb: file readers and search utilities \(rg,grep,cat,sed,jq,wc,head,ls,find\) count as*read/inspect*; interpreters and build commands \(python,node,bash,pytest,latexmk\) as*tool/script*; file mutations \(apply\_patch,tee,mv,cp,touch,mkdir\) and structured file\-change events as*write/edit*\.
##### Claude Code \(named\-tool grammar\)\.
Claude Code emits fewer, coarser named tool calls\. These are bucketed by tool name:Read,Glob,Grep,LSas*read/inspect*;Write,Edit,MultiEditas*write/edit*; and shell\- or interpreter\-style calls \(Bash,Skill,Agent\) as*tool/script*, with embedded shell commands routed through the same verb rules as above\. Because one Codex shell command and one Claude Code tool call are not the same unit of work, absolute counts are not comparable across families; the main text reads only the within\-family change\.
### A\.3Cost and time accounting
##### Cost\.
Per\-matter cost is logged token usage priced at public list rates \(Table[9](https://arxiv.org/html/2606.04602#A1.T9)\)\. Cached\-read tokens are treated as free, and judge and evaluation tokens are excluded, so the figures reflect solver inference only\. All cells are recomputed from logged token counts with one exception: the Codex/GPT\-5\.5*baseline*traces do not expose billable token fields, so its per\-matter cost \($1\.51\) is the original full\-cell logged estimate rather than a recomputation\. The correspondingParthenon/GPT\-5\.5 cell does expose tokens and is recomputed \($1\.29\); the “cost falls under the harness” comparison for this tier therefore weighs a recomputed value against a logged estimate and should be read as order\-of\-magnitude rather than exact\.
Table 9:Public list prices used for the cost figure, in USD per11M tokens \(accessed 2026\-06\-02\)\. Cached reads are treated as free in our accounting\. GPT\-5\.5 long\-context requests above the documented threshold are billed at higher rates; the recomputed cells apply the standard rates to logged token fields\.
##### Time\.
We did not log wall\-clock time for the agent runs, so the per\-matter*time*columns in Table[7](https://arxiv.org/html/2606.04602#S4.T7)are matter\-complexity bucket estimates, not measured run durations: human work is bucketed at33/1010/2020h by matter complexity, and the AI side uses a matched per\-complexity estimate\. The time comparison \(and the “∼\\sim70×70\\timesfaster” statement\) is therefore an order\-of\-magnitude claim about the scale of the gap, whereas the cost comparison is grounded in measured tokens\. The human cost column applies a single blended rate of $349/h\(Clio,[2026](https://arxiv.org/html/2606.04602#bib.bib44)\)to the same complexity buckets; the human accuracy column is the all\-criteria release standard that LAB approximates, not a controlled human run\.
### A\.4Information boundaries
Table 10:Anti\-leakage contract used by the continual\-learning loop\. The main full\-corpusParthenonresult uses the optimized harness with bounded revision disabled; the hard\-1010ledger \(App\.[B\.6](https://arxiv.org/html/2606.04602#A2.SS6)\) measures harness learning itself\.
## Appendix BFramework Internals, Failure Taxonomy, and Ablation Protocols
This appendix expands the parts of the framework that the main text states compactly: the concrete Tools surface \(§[B\.1](https://arxiv.org/html/2606.04602#A2.SS1)\), the structure of a Skill \(§[B\.2](https://arxiv.org/html/2606.04602#A2.SS2)\), the failure taxonomy and how rationales are mapped to it \(§[B\.3](https://arxiv.org/html/2606.04602#A2.SS3)\), and the protocols behind the ablations of §[4\.4](https://arxiv.org/html/2606.04602#S4.SS4)\(§[B\.6](https://arxiv.org/html/2606.04602#A2.SS6)–[B\.7](https://arxiv.org/html/2606.04602#A2.SS7)\)\. It reports protocols and representative cases rather than per\-run results, which are released in full with the data pack \(§[B\.8](https://arxiv.org/html/2606.04602#A2.SS8)\)\.
### B\.1The Tools surface in detail
Table[2](https://arxiv.org/html/2606.04602#S3.T2)groups the Tools layer into four lifecycle stages; Table[11](https://arxiv.org/html/2606.04602#A2.T11)lists the1414agent\-callable tools individually\. Each is a deterministic operation registered in a single tool registry, so adding a capability is one reviewable diff\. Several single\-file probes are prompt\-level capabilities that drive standard search utilities \(rg/sed\) against the canonicalized source text rather than carrying bespoke parsers\.
Table 11:The1414deterministic, agent\-callable tools, grouped by matter lifecycle stage\. Five carry a mandatory self\-audit step ordered into the release gate \(number then date grounding, skeleton compliance, then citation accuracy\); a trailing reasoning/completeness pass is always last\.
### B\.2Anatomy of a skill
Each of the1,2511\{,\}251skills is a versioned Markdown file routed to a matter by its task identifier, with a short YAML header \(name,task\_id,description, and the agent roles it activates for\) followed by the seven\-part scaffold of the Skills layer\. The scaffold is rubric\-blind by construction: it names*procedure*, not answers\. Before promotion, a separate rubric\-blind audit – given only the task title and the skill text – rewrites any residual task\-specific content into generic procedure, so deployed skills carry the procedural sections and delegate leakage control to that audit\. Table[12](https://arxiv.org/html/2606.04602#A2.T12)shows the scaffold instantiated, in abstracted form, for a litigation case\-assessment memorandum\.
Table 12:The seven\-part skill scaffold, abstracted to procedure\. A skill encodes how to work a matter class, never an answer; the example is rendered generically to show shape, not memorized content\.
### B\.3Error taxonomy and how rationales are mapped
Table[13](https://arxiv.org/html/2606.04602#A2.T13)gives the operational meaning of the ten error classes used in §[4\.2](https://arxiv.org/html/2606.04602#S4.SS2)and names theParthenonsurface expected to absorb each\. The mapping from a judge rationale to a class is deterministic, not an LLM call: a priority\-ordered keyword/regex matcher reads the concatenated criterion title and rationale and assigns the first class whose pattern fires \(quantitative/temporal terms first, then grounding, remediation, materiality, cross\-document, deliverable form, legal\-rule, coverage, and missing\-fact patterns\)\. Rationales that match no pattern fall through to*Other*, which is therefore a residual bucket to be split by future taxonomy work, not a named failure mode\. Counts are the criterion failures behind the Codex/GPT\-5\.5 column of Table[5](https://arxiv.org/html/2606.04602#S4.T5)\.
Table 13:Error taxonomy and corresponding framework controls\. Counts sum to15,17115\{,\}171failed criteria\.
### B\.4Trace defects behind the controls
Table 14:Qualitative trace\-defect map from Codex JSONL transcripts\. Rows are mirrored in the trace\-defect analysis ledger and serve as process evidence for the controls in Table[13](https://arxiv.org/html/2606.04602#A2.T13)\.### B\.5GPT\-5\.5 trajectory slices
Table[15](https://arxiv.org/html/2606.04602#A2.T15)gives concrete trace slices from the same Codex/GPT\-5\.5 result cell used in §[4\.2](https://arxiv.org/html/2606.04602#S4.SS2)–[4\.3](https://arxiv.org/html/2606.04602#S4.SS3): observable agent behavior, the scored miss, and the missing release gate\. They make the abstract failure classes concrete – in each case the agent reads the full source set yet omits a testable fact, formula, or identity check that a deterministic gate would have caught\.
Table 15:Representative GPT\-5\.5 trajectory slices\. Document counts are from run metrics; misses are from paired judge rationales\.### B\.6Self\-evolving optimization protocol \(hard\-1010\)
The hard\-1010panel \(Figure[9](https://arxiv.org/html/2606.04602#S4.F9)\) measures harness learning in isolation, separately from the full\-corpus results\. The ten tasks are selected for low baseline pass rate, non\-trivial document load, and enough rubric criteria to form a meaningful trajectory, with at most one task per practice area for cross\-domain coverage\. One optimization*step*runs all ten tasks \(solve then judge\), after which the learner proposes one batch of harness edits; the loop runs ten such steps from an empty starting harness\. Judging uses a two\-rung model ladder \(a small judge, escalating to a larger one only on context overflow or parse failure\)\. A candidate harness is accepted only if its scored per\-task pass rate strictly exceeds the last accepted harness \(*best\-so\-far gate*\); otherwise it is rolled back and archived, which is why the gated curve is monotone while the raw candidates are not\. The learner may edit Skills, Knowledge, and Tools under a fixed budget \(at most a few edits per step and at most one new tool file, which must be standard\-library Python under a small size limit and pass static safety checks\), and it sees only redacted signals – never task ids, party or document names, dates, amounts, clause numbers, or rubric text\. The headline outcome \(two unrelated solver/learners converging to within0\.40\.4pp\) is reported in §[4\.4](https://arxiv.org/html/2606.04602#S4.SS4); full per\-step ledgers are in the data pack\.
### B\.7Reasoning\-effort and document\-summary ablations
##### Reasoning effort\.
On the same ten tasks, we sweep the coding\-agent CLI effort setting in a full factorial:low,medium,high,xhighfor Codex/GPT\-5\.5, andlow,medium,high,xhigh,maxfor Claude Code/Sonnet, withmediumas the baseline\. The setting is a CLI\-level effort flag, not an API reasoning parameter, so it varies the agent’s exploration budget on a fixed harness\.
##### Document summaries\.
The cached document\-summary tool \(summarize\_docs\) writes a structured per\-document brief – named entities, operative terms, quantitative and date anchors, at roughly a quarter of the source length – intended as an index back into the originals\. The ablation pairs the same GPT\-5\.4\-mini Codex setting with and without the prewarmed summary directory, on the same task set\. Its conclusion \(§[4\.4](https://arxiv.org/html/2606.04602#S4.SS4)\) is that the cache does not close the long\-input gap; full paired outcomes are in the data pack\.
### B\.8Data availability
To keep this appendix to protocols and representative cases rather than exhaustive tables, the complete experimental record is released as an open data pack: per\-task scored outputs and judge rationales, the native solver transcripts, the derived trace and cell metrics behind every figure, the optimization and ablation ledgers, and the harness artifacts \(Skills, Tools, Knowledge\)\. Readers who need a specific number not printed here can recover it from that release\.Similar Articles
Investigating Multi-Agent Deliberation in Law
This paper investigates multi-agent deliberation methods for legal reasoning tasks using LLMs, introducing two novel frameworks inspired by courtroom procedures. The experiments show that multi-agent systems achieve comparable overall performance to monolithic LLMs but produce distinct answers and can solve cases that baselines fail, highlighting the potential of multi-agent approaches for legal AI.
When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval
This paper introduces a self-evolving framework that uses an LLM-based agent to iteratively create and refine query rewriting rules for BM25 in legal case retrieval, outperforming non-evolutionary baselines on the LeCaRD-v2 benchmark without any parameter training.
PolicyBank: Evolving Policy Understanding for LLM Agents
PolicyBank proposes a memory mechanism that enables LLM agents to autonomously refine their understanding of organizational policies through iterative interaction and corrective feedback, closing specification gaps that cause systematic behavioral divergence from true requirements. The work introduces a systematic testbed and demonstrates PolicyBank can close up to 82% of policy-gap alignment failures, significantly outperforming existing memory mechanisms.
Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents
This paper introduces the Hypothesis Evolution Protocol (HEP) for LLM agents, which makes hypothesis generation, testing, and belief updates explicit and auditable. Experiments on materials-science tasks show that HEP-equipped agents generalize across research questions and become more effective with stronger base LLMs.
Built an agent workstation where the environment does the structural reasoning so the LLM doesn't have to
Atlarix is a desktop environment that pre-parses codebases into a node/edge graph, allowing coding agents to navigate architecture via queries instead of reading raw text, which improves performance of smaller local models.