Studying Without a Syllabus: Task-Agnostic Environment Preprocessing

# Studying Without a Syllabus: Task-Agnostic Environment Preprocessing
Source: [https://arxiv.org/html/2609.10824](https://arxiv.org/html/2609.10824)
Varun UrsekarAffiliation:Scale AIVijay S\. KalmathAffiliation:Scale AIApaar ShankerAffiliation:Scale AIVeronica ChatrathAffiliation:Scale AIYuan XueAffiliation:Scale AI

###### Abstract

Before an LLM agent tackles tasks in a new environment, it can inspect available corpora and tools and construct reusable resources such as indices, scripts, or procedural guidance\. Most automated adaptation methods, however, rely on task examples, trajectories, or evaluation feedback to decide what to build\. Existing task\-agnostic approaches avoid this supervision but commit in advance to a preparation strategy for a particular type of environment\. We study a more open\-ended setting: can an agent study an unfamiliar environment without a syllabus, i\.e\. before test time and without knowledge of the downstream task distribution, and choose how to prepare it? We formalize task\-agnostic environment preprocessing, in which a studying system explores an environment under a budget and produces artifacts for a frozen solver\. We compare unaided and archive\-equippedMeta\-Agents with fixed synthetic\-practice and corpus\-processing methods across six heterogeneous benchmarks\. AMeta\-Agentvariant achieves the highest Avg@3 reward on five benchmarks, while fixed corpus processing remains best on the largest corpus benchmark\. Larger study budgets do not reliably improve downstream reward\. Nevertheless, studied artifacts reduce the test\-time sampling needed to reach a given score, demonstrating how reusable preparation can shift computation from repeated test\-time attempts to a pre\-task study phase\.

## 1Introduction

LLM agent performance depends substantially on choices beyond the underlying model: prompts, tools, context and memory management, and the organization of corpora, databases, and external services\[[16](https://arxiv.org/html/2609.10824#bib.bib9),[35](https://arxiv.org/html/2609.10824#bib.bib6),[37](https://arxiv.org/html/2609.10824#bib.bib8),[38](https://arxiv.org/html/2609.10824#bib.bib1)\]\. Consider an agent that performs retrieval over a large filesystem\. If files are unnamed and disorganized, the agent receives no clues about file content and must resort to expensive, exhaustive search\. Consequently, practitioners routinely adapt components of the agent*harness*and*environment*to tailor them to the tasks the agent is expected to perform\. This has led to increased interest in automated approaches to optimizing agents for novel tasks, spanning prompt optimization, harness optimization, and adaptive memory\.

Existing automated adaptation methods typically use information about the expected test\-time task distribution to guide modifications\. Prompt optimizers such as DSPy\[[15](https://arxiv.org/html/2609.10824#bib.bib33)\]and GEPA\[[1](https://arxiv.org/html/2609.10824#bib.bib34)\]use*task examples*and*evaluation feedback*\. VeRO\[[31](https://arxiv.org/html/2609.10824#bib.bib4)\]and Meta\-Harness\[[17](https://arxiv.org/html/2609.10824#bib.bib5)\]modify the entire harness as code using similar signals\. Adaptive memory systems such as AWM\[[34](https://arxiv.org/html/2609.10824#bib.bib13)\], ACE\[[40](https://arxiv.org/html/2609.10824#bib.bib21)\], and Dynamic Cheatsheet\[[29](https://arxiv.org/html/2609.10824#bib.bib20)\]extract reusable knowledge and procedures from*task trajectories*with or without*labels*\. Such supervisory signals may be unavailable when an agent first encounters a new environment, particularly when representative tasks are costly to obtain\. This cold start motivates*task\-agnostic adaptation*of agents and environments prior to test time\.

Several lines of work have addressed the problem of agent and environment adaptation prior to deployment\. One line substitutes knowledge of the true task distribution with self\-generated exploratory practice\. SPICE\[[21](https://arxiv.org/html/2609.10824#bib.bib7)\]uses corpus\-grounded self\-play to generate a curriculum for parameter adaptation, whilePREPING\[[6](https://arxiv.org/html/2609.10824#bib.bib10)\]generates synthetic tasks in tool environments and distills their trajectories into a procedural playbook\. In corpus environments, retrieval structures can be prepared offline to make document collections more amenable to search by agents: RAPTOR\[[26](https://arxiv.org/html/2609.10824#bib.bib30)\]recursively clusters and summarizes documents, GraphRAG\[[7](https://arxiv.org/html/2609.10824#bib.bib31)\]constructs a knowledge graph and community summaries, andCorpus2Skill\[[28](https://arxiv.org/html/2609.10824#bib.bib11)\]compiles a corpus into a navigable skill hierarchy\. Closest to our framing, Machine Studying\[[19](https://arxiv.org/html/2609.10824#bib.bib23)\]treats studying as an explicit pre\-task process and evaluates how well an agent can construct reusable context from a corpus without downstream tasks\. Each of these methods commits to a strategy for processing the environment depending on its structure\. With many methods available and suited to different environment types, we ask whether an*open\-ended studying system*can choose how to process an environment as a function of what it contains\.

![Refer to caption](https://arxiv.org/html/2609.10824v1/figures/fig1.png)Figure 1:An instance of ourMeta\-Agent w/ Archivepipeline\. The environmentEEis a sandboxed container with a filesystem\. The base environment containing benchmark\-specific artifacts such as tools and corpora is shared across study and test time\. Before test time, the meta\-agent exploresEEunder budgetBstudyB\_\{\\mathrm\{study\}\}, using strategies from a mounted archive\. It writes study artifacts into a volume that is later mounted into the test\-time agent’s sandbox\. The test\-time agent uses the study artifacts and the pre\-existing benchmark\-specific artifacts to solve unseen tasks\.Specifically, we ask whether aMeta\-Agentcan explore an environment without knowledge of the downstream task distribution and nonetheless produce artifacts that enable a frozen solver agent to effectively perform tasks at test time\. The solver’s weights and harness are fixed\. What changes is the set of resources it finds in its sandboxed environment\. These artifacts may extend the environment with directories, indices, knowledge bases, scripts, or tools, or supply harness\-side context through prompted guidance, skills, or playbooks\. Any file\-based artifact the solver can access from within its sandbox is admissible\. We compare two variants of thisMeta\-AgentagainstPREPINGandCorpus2Skill, two fixed workflows that process agent environments in a task\-agnostic way\.

Across six benchmarks spanning 36 unique environments, each with differently sized corpora, diverse tools, and specialized domain knowledge, the twoMeta\-Agentvariants collectively lead to the highest downstream Avg@3 reward on five, underperformingCorpus2Skillonly on BCP\-G\. Our archive\-equipped variant improves overNo Studyon all six, and ranks first or second under both Avg@3 and Best@3 of downstream held\-out rewards\. We also analyze what the methods explore and produce, how solver performance scales with study budget, and how test\-time sampling scales with and without studying\.

We make four contributions:

1. 1\.We formalizetask\-agnostic environment processing: systems receive an environment and a bounded study budget, but no downstream task instances, traces, or labels, and construct reusable environment resources, harness\-delivered context, or both for a frozen solver\.
2. 2\.We introduceopen\-ended studyingMeta\-Agentsthat dynamically choose*how to explore*an environment and*what to create*, with unaided and archive\-assisted variants\.
3. 3\.We show thatopen\-ended studying strategies outperform fixed ones on five of the six heterogeneous benchmarkswe evaluate on\.
4. 4\.We show that additional study budget does not reliably improve downstream performance, while studied agents often require less test\-time compute to reach a given score than agents without studying\.

## 2Related Work

##### Task\-informed adaptation of agent harnesses\.

Downstream task information can guide both what an agent system should change and whether the change helped\. DSPy\[[15](https://arxiv.org/html/2609.10824#bib.bib33)\], GEPA\[[1](https://arxiv.org/html/2609.10824#bib.bib34)\], and PromptBreeder\[[10](https://arxiv.org/html/2609.10824#bib.bib35)\]optimize instructions or demonstrations against task examples and evaluation signals, while VeRO\[[31](https://arxiv.org/html/2609.10824#bib.bib4)\]and Meta\-Harness\[[17](https://arxiv.org/html/2609.10824#bib.bib5)\]extend this search to executable harness components\. Memory and context systems, including AWM\[[34](https://arxiv.org/html/2609.10824#bib.bib13)\], Memp\[[9](https://arxiv.org/html/2609.10824#bib.bib14)\], ACE\[[40](https://arxiv.org/html/2609.10824#bib.bib21)\], Dynamic Cheatsheet\[[29](https://arxiv.org/html/2609.10824#bib.bib20)\], Reflexion\[[27](https://arxiv.org/html/2609.10824#bib.bib15)\], ExpeL\[[41](https://arxiv.org/html/2609.10824#bib.bib16)\], MemGPT\[[23](https://arxiv.org/html/2609.10824#bib.bib17)\], A\-MEM\[[36](https://arxiv.org/html/2609.10824#bib.bib18)\], and Mem0\[[4](https://arxiv.org/html/2609.10824#bib.bib19)\], instead distill knowledge, procedures, or code from observed interactions\. Other work relocates this processing within the interaction lifecycle:ProAct\[[13](https://arxiv.org/html/2609.10824#bib.bib24)\]acts between user turns,IdleSpec\[[5](https://arxiv.org/html/2609.10824#bib.bib25)\]during tool\-call latency, andAuto\-Dreamer\[[39](https://arxiv.org/html/2609.10824#bib.bib26)\]after experience has accumulated\. These methods use task examples, trajectories, or feedback to decide what to change\. Our setting asks what to construct before such task information is available\.

##### Task\-agnostic adaptation of agent harnesses and environments\.

Without task information, a system must derive its objective from what is available before use\. Sleep\-time Compute\[[20](https://arxiv.org/html/2609.10824#bib.bib22)\]studies how preprocessing a supplied context can displace computation after an unknown query arrives\. Machine Studying\[[19](https://arxiv.org/html/2609.10824#bib.bib23)\]isolates the same temporal regime as ours \(study before future queries\) but fixes both the input modality to a corpus and the candidate interventions to self\-supervised training, synthetic\-data fine\-tuning, and amortized cheatsheet construction\. Our decision space instead includes heterogeneous environments: the studying system must determine what to process and what form of reusable support to produce\. Existing approaches encode different commitments about what will transfer\. Retrieval systems assume that useful preparation takes the form of queryable corpus structure, from conventional sparse retrieval such as BM25\[[25](https://arxiv.org/html/2609.10824#bib.bib29)\], dense retrieval such as DPR\[[14](https://arxiv.org/html/2609.10824#bib.bib28)\], and retrieval\-augmented generation\[[18](https://arxiv.org/html/2609.10824#bib.bib27)\]to the summaries, graphs, and hierarchies constructed by RAPTOR\[[26](https://arxiv.org/html/2609.10824#bib.bib30)\], GraphRAG\[[7](https://arxiv.org/html/2609.10824#bib.bib31)\],Corpus2Skill\[[28](https://arxiv.org/html/2609.10824#bib.bib11)\], andPANINI\[[24](https://arxiv.org/html/2609.10824#bib.bib32)\]\. Practice\-based systems instead rely on self\-generated interaction: Voyager\[[33](https://arxiv.org/html/2609.10824#bib.bib12)\]acquires executable skills through open\-ended exploration, SPICE\[[21](https://arxiv.org/html/2609.10824#bib.bib7)\]constructs a corpus\-grounded training curriculum, andPREPING\[[6](https://arxiv.org/html/2609.10824#bib.bib10)\]practices with tools to produce a procedural playbook\.PREPING’s guided\-exploration baseline is the nearest operational analogue toMeta\-Agent w/o Archive: both explore before target tasks and record reusable guidance\. We place these commitments under a common protocol and test whether a study agent can choose and combine forms of preparation based on the environment\.

Table 1:Avg@3 downstream reward\.Best values are bold\. Second best are underlined\. For each studying method, each artifact\-iteration score averages three task\-time repetitions per task\. Values report the mean±\\pmsample SD across three independent artifact iterations\. ForNo Study, values report the mean±\\pmpopulation SD across all\(103\)\\binom\{10\}\{3\}three\-repetition subset estimates\.Table 2:Best@3 downstream reward\.Best values are bold and second\-best values are underlined\. For each studying method, each artifact\-iteration score takes the best of three task\-time repetitions per task\. Values report the mean±\\pmsample SD across three independent artifact iterations\. ForNo Study, values report the mean±\\pmpopulation SD across all\(103\)\\binom\{10\}\{3\}three\-repetition subset estimates\. Best@3 consumes three task\-time rollouts and is not a fixed\-cost substitute for Avg@3\.

## 3Task\-Agnostic Environment Preprocessing

In our setup, an agentπ\\pioperates in an environmentEE\. An agentπ=\(m,H\)\\pi=\(m,H\)is a tuple of an LLMmmand a harnessHH\. The environmentEEis a shared runtime and collection of resources required to complete a set of tasks\. Practically,EEis implemented as an isolated Docker sandbox compatible with the Harbor framework\[[12](https://arxiv.org/html/2609.10824#bib.bib3)\]\. The harnessHHrepresents the program that invokes the model within the environment\.HHmaintains context, registers and invokes tools, and drives the interaction loop between the model and environment\. While any standard CLI\-based coding harness is compatible with our setup, we use the Claude Code\[[2](https://arxiv.org/html/2609.10824#bib.bib2)\]harness throughout this work\.

Each environmentEEadmits a space𝒯E\\mathcal\{T\}\_\{E\}of feasible downstream tasks\. Each taskt=\(xt,rt\)∈𝒯Et=\(x\_\{t\},r\_\{t\}\)\\in\\mathcal\{T\}\_\{E\}consists of a promptxtx\_\{t\}and a reward functionrtr\_\{t\}that scores the output of an agent prompted withxtx\_\{t\}\. A benchmark provides a finite held\-out test set; we writeDED\_\{E\}for the uniform empirical distribution over that set\. Letπsolver\\pi\_\{\\mathrm\{solver\}\}denote a frozen downstream agent\. Its aggregate reward on an environmentE′E^\{\\prime\}and test distributionDED\_\{E\}is then

Rπsolver​\(E′,DE\)=𝔼t∼DE​\[rt​\(πsolver​\(xt,E′\)\)\],R\_\{\\pi\_\{\\mathrm\{solver\}\}\}\(E^\{\\prime\},D\_\{E\}\)=\\mathbb\{E\}\_\{t\\sim D\_\{E\}\}\\left\[r\_\{t\}\\\!\\left\(\\pi\_\{\\mathrm\{solver\}\}\(x\_\{t\},E^\{\\prime\}\)\\right\)\\right\],\(1\)whereE′E^\{\\prime\}is either the original environmentEEor a version of it modified before test time\.

The goal of a task\-agnostic*studying system*SSis to improveRπsolver​\(E,DE\)R\_\{\\pi\_\{\\mathrm\{solver\}\}\}\(E,D\_\{E\}\)without observingDED\_\{E\}\. In our work,SSdoes this by using the frozen solver and the original environment to output a modified environmentEstudiedE\_\{\\mathrm\{studied\}\}, i\.e\.S:Π×ℰ→ℰS:\\Pi\\times\\mathcal\{E\}\\rightarrow\\mathcal\{E\}, whereΠ\\Piandℰ\\mathcal\{E\}are the spaces of agents and environments, respectively\. We hold the implementation of the harnessHHfixed, although artifacts may be supplied through its existing context interfaces\. Here, task\-agnostic means thatSSdoes not receive real downstream task instances or signals derived from their distribution, including traces, labels, verifier outputs, or evaluation feedback\. It may interact withEEand use records of that interaction, including synthetic practice, to guide its modifications\.SSmay produce any number and type of artifacts materialized as files inEstudiedE\_\{\\mathrm\{studied\}\}, such as indices, knowledge bases, scripts, or executable tools\.EstudiedE\_\{\\mathrm\{studied\}\}contains all of the original artifacts inEEtogether with any additional artifacts intended to help the solver\. This preserves the integrity of the original environment\.

The studying process incurs a costCstudy​\(S,E\)C\_\{\\mathrm\{study\}\}\(S,E\), measured in API dollars, that must not exceed a given study budgetBstudyB\_\{\\mathrm\{study\}\}\. Letppdenote a distribution over environments\. We seek a studying system that maximizes the expected aggregate reward of the frozen solver across environments:

S⋆=arg⁡maxS​𝔼E∼p​\[Rπsolver​\(S⁡\(πsolver,E\),DE\)\]s\.t\.Cstudy​\(S,E\)≤Bstudy\.S^\{\\star\}=\\arg\\max\_\{S\}\\;\\mathbb\{E\}\_\{E\\sim p\}\\\!\\left\[R\_\{\\pi\_\{\\mathrm\{solver\}\}\}\\\!\\left\(S\(\\pi\_\{\\mathrm\{solver\}\},E\),\\,D\_\{E\}\\right\)\\right\]\\quad\\text\{s\.t\.\}\\quad C\_\{\\mathrm\{study\}\}\(S,E\)\\leq B\_\{\\mathrm\{study\}\}\\;\\;\.\(2\)There is no a priori restriction on the structure or logic ofSS\. It may be a filesystem processing pipeline as inCorpus2Skill, a fixed orchestration of several specialized agents as inPREPING, or an open\-ended agentic process like ourMeta\-Agentvariants\.

Figure 2:Study\-budget scaling\.Mean Avg@3 reward versus study budget\. Shading shows 95% task\-bootstrap intervals, and dotted lines showNo Study\. Harvey LAB uses dense verifier pass rate\.
## 4Methods

In practice, whatSSshould do depends on the environment\. A large corpus may benefit from indexing and summarization, whereas a broad tool surface may require skills or instructions distilled from synthetic practice or self\-play\. Figure[4](https://arxiv.org/html/2609.10824#A1.F4)describes the breadth of environments in our benchmark suite using two observable measures: counts of files exposed to the solver and benchmark\-specific tools\.

One of our aims is to compare the effectiveness of open\-ended study strategies versus fixed ones\. We call a studying system open\-ended when its studying procedure and artifact type are not fixed in advance\. We consider two representative baseline systems with contrasting but fixed adaptation procedures, each designed for different environmental niches:PREPINGandCorpus2Skill\. In contrast, our twoMeta\-Agentvariants use a single study agentπstudy\\pi\_\{\\mathrm\{study\}\}that can simultaneously explore an environment and modify it in an open\-ended way\. We provide a description of all methods below with additional implementation details reported in Appendix[A\.3\.1](https://arxiv.org/html/2609.10824#A1.SS3.SSS1)\.

### 4\.1Baselines: Fixed studying systems

##### PREPING\.

PREPING\[[6](https://arxiv.org/html/2609.10824#bib.bib10)\]performs task\-agnostic environment preparation through environment\-grounded synthetic practice organized into cycles\. In each cycle, a*Proposer*generates a batch of tasks based on available tools, a practice*Solver*attempts them in the target harness, and a*Validator*filters infeasible trajectories\. A reflector and curator then distill lessons from successes and failures into a procedural playbook containing strategies, pitfalls, interface notes, and reusable code\. The number of cycles, tasks per cycle, and per\-task execution limit determine the amount of synthetic practice and the overall study cost\. At test time, a configurable number of playbook entries are retrieved and prepended to the task instruction\. Thus, althoughPREPINGuses agents, its roles, invocation order, and output format are fixed\.

##### Corpus2Skill\.

Corpus2Skill\[[28](https://arxiv.org/html/2609.10824#bib.bib11)\]restructures a document corpus into a navigable directory of skills\. Its pipeline summarizes and embeds documents, clusters them into a labeled hierarchy, and constructs indexes that map topics to source documents\. The pipeline is not agentic: it follows a fixed end\-to\-end procedure without adapting its strategy in response to runtime feedback\. The amount of source content processed and the structure of the resulting hierarchy are controlled by a document\-length limit, whether document summaries are generated, the hierarchy’s branching ratio, the maximum number of top\-level clusters, and the minimum cluster size\. These choices affect the studying cost in turn\. At task time, the downstream agent receives navigation instructions, as well as the generated hierarchical filesystem\. UnlikePREPING, this system compiles declarative content without generating or executing synthetic tasks\.

### 4\.2Open\-ended studying systems

BothMeta\-Agentvariants instantiate a study agentπstudy\\pi\_\{\\mathrm\{study\}\}\. Under the same task\-agnostic protocol, the study agent must explore the environment, choose a studying procedure itself, and construct any file\-based artifacts it deems useful to a future solver\. The unaided variantMeta\-Agent w/o Archivemust choose without guidance\.Meta\-Agent w/ Archiveis additionally given a seed archive of skills comprising executable scripts and their descriptions\. The archive includes skills implementingPREPINGandCorpus2Skill, plus a general exploratory\-study workflow resemblingMeta\-Agent w/o Archive\. The skills are mounted into the agent sandbox’s filesystem and are accessible using standard shell\-based tools \(see Figure[1](https://arxiv.org/html/2609.10824#S1.F1)\)\. The agent may invoke these workflows during study and use, combine, or ignore their outputs when constructing the final artifacts\.

Figure 3:Test\-time scaling with and without study\.No StudyBest@KKcurves are compared with Best@KKcurves and Avg@3 reference levels for each study method\. Best@KKis an oracle upper bound\. Harvey LAB uses dense verifier pass rate\.

## 5Experimental Setup

##### Benchmarks\.

We evaluate on six agentic benchmarks spanning diverse environment types\.BCP\-Grep \(BCP\-G\)adapts BrowseComp\-Plus\[[3](https://arxiv.org/html/2609.10824#bib.bib37)\]: it retains the 830 questions and fixed 100,195\-document corpus, but exposes each document as a file for shell\-based search rather than through the original BM25 retriever\.OfficeQA\[[22](https://arxiv.org/html/2609.10824#bib.bib41)\]andHarvey LAB\[[11](https://arxiv.org/html/2609.10824#bib.bib39)\]also expose document collections\. For Harvey LAB, we use only its 250 firm\-knowledge tasks, which share the same fictional firm’s document\-management system\.DABStep\[[8](https://arxiv.org/html/2609.10824#bib.bib38)\]exposes seven files that combine task data, answer\-bearing references, and protocol instructions for processing that data using Python\.APEX\-Agents\[[32](https://arxiv.org/html/2609.10824#bib.bib40)\]contributes 452 tasks grouped into 31 worlds, each with its own shared filesystem and tools: eight investment\-banking, 12 law, and 11 management\-consulting worlds\. We treat each world as a separate environment and run the study process independently within each world\. Thus, the reported APEX\-Agents benchmark score aggregates results across 31 studied environments, whereas each other benchmark represents a single studied environment\.AppWorld\[[30](https://arxiv.org/html/2609.10824#bib.bib36)\]exposes a multi\-application tool interface\. Figure[4](https://arxiv.org/html/2609.10824#A1.F4)and Table[3](https://arxiv.org/html/2609.10824#A1.T3)report the observed corpora and tools\.

##### Frozen Solver Agent\.

For all target task evaluations, we set the frozen solverπsolver\\pi\_\{\\text\{solver\}\}to an agent that uses the Claude Code harness with Claude Haiku 4\.5 as the underlying model at temperature 1\.0\.

##### Study Methods\.

Per §[4](https://arxiv.org/html/2609.10824#S4), we compare four study methodsPREPING,Corpus2Skill,Meta\-Agent w/o Archive, andMeta\-Agent w/ Archiveagainst a singleNo Studybaseline\. We use Claude Code as the coding harness for all agentic components\. Claude Opus 4\.8 performsMeta\-Agentstudy, as well asPREPINGproposal, validation, reflection, and curation\. Claude Haiku 4\.5 executesPREPINGsynthetic tasks, every downstream task, andCorpus2Skilldocument\-card generation\. Claude Sonnet 4\.6 performsCorpus2Skillcluster summarization, labeling, repartitioning, and entity extraction\. For both reproduced baselines, we retain the embedding models and embedding\-related settings of the original works:Corpus2SkillusesQwen3\-Embedding\-8Bfor document and summary embeddings, andPREPINGusestext\-embedding\-3\-smallto retrieve playbook entries\.

For the primary comparisons, we runPREPINGfor five cycles of 10 synthetic tasks, yielding 50 practice tasks per environment; the original implementation uses 10 cycles of 10\. We otherwise retain its published validation thresholds\. ForCorpus2Skill, we use the published default configuration where possible, with benchmark\-specific adjustments to hierarchy branching, document length, and tree compaction\. Additional implementation details and representative prompts appear in Appendices[A\.3\.1](https://arxiv.org/html/2609.10824#A1.SS3.SSS1)and[A\.3\.2](https://arxiv.org/html/2609.10824#A1.SS3.SSS2), respectively\.

##### Repetitions and rollout metrics\.

Each studying system is run independentlyNstudy=3N\_\{\\mathrm\{study\}\}=3times per environment, producing three artifact sets\. For APEX\-Agents, this means three study runs for each of its 31 world environments\. For every artifact set, downstream evaluation is independently repeatedNeval=3N\_\{\\mathrm\{eval\}\}=3times\. We reserveKKfor the number of those task\-time repeats aggregated by Avg@KKor Best@KK\. The primary tables useK=3K=3\. LetDED\_\{E\}be the empirical test\-task distribution defined in §[3](https://arxiv.org/html/2609.10824#S3), and letrs,t,jr\_\{s,t,j\}be the reward from rolloutjjon taskttusing the artifact produced by study iterationss\. We report

Avg​@​K\\displaystyle\\mathrm\{Avg@\}K=1Nstudy​K​∑s=1Nstudy𝔼t∼DE​\[∑j=1Krs,t,j\],\\displaystyle=\\frac\{1\}\{N\_\{\\mathrm\{study\}\}K\}\\sum\_\{s=1\}^\{N\_\{\\mathrm\{study\}\}\}\\mathbb\{E\}\_\{t\\sim D\_\{E\}\}\\left\[\\sum\_\{j=1\}^\{K\}r\_\{s,t,j\}\\right\],\(3\)Best​@​K\\displaystyle\\mathrm\{Best@\}K=1Nstudy​∑s=1Nstudy𝔼t∼DE​\[max1≤j≤K⁡rs,t,j\]\\displaystyle=\\frac\{1\}\{N\_\{\\mathrm\{study\}\}\}\\sum\_\{s=1\}^\{N\_\{\\mathrm\{study\}\}\}\\mathbb\{E\}\_\{t\\sim D\_\{E\}\}\\left\[\\max\_\{1\\leq j\\leq K\}r\_\{s,t,j\}\\right\]Avg@KKaverages allKKrollout rewards per task, whereas Best@KKselects the best of theKKrollouts per task\. Both then average over tasks and study iterations\. We report the sample standard deviation across theNstudyN\_\{\\mathrm\{study\}\}study\-run\-specific task macro\-averages, so uncertainty reflects variation across study runs rather than individual tasks or rollouts\.

No Studyhas no artifact\-set dimension and is evaluated withNeval=10N\_\{\\text\{eval\}\}=10independent task\-time repetitions\. ForK=3K=3, both Avg@3 and Best@3 are computed over all\(103\)=120\\binom\{10\}\{3\}=120subsets of three repetitions: Avg@3 averages the subset means, whereas Best@3 averages the subset maxima\. We macro\-average over tasks within each subset and report the mean and population standard deviation across the 120 resulting benchmark\-level estimates\.

##### Metrics\.

For each target task, we use the benchmark’s original evaluation measure except on Harvey LAB\. For Harvey LAB, the original task reward is a strict all\-criteria\-pass indicator over per\-task rubrics\. We report instead the dense rubric criterion pass rate\. The three primary metrics are \(i\) study costCstudyC\_\{\\mathrm\{study\}\}in API dollars, \(ii\) downstream rewardrr, and \(iii\) downstream inference costCinferenceC\_\{\\mathrm\{inference\}\}in API dollars\. We meter study and downstream inference separately using provider\-reported usage and cost where available\. Study cost includes all model calls used to construct an artifact, including generation, control, embeddings, and compilation\.

##### Budgets\.

For our experiments in Tables[1](https://arxiv.org/html/2609.10824#S2.T1)and[2](https://arxiv.org/html/2609.10824#S2.T2), we do not impose a shared study budgetBstudyB\_\{\\mathrm\{study\}\}\. Instead, each method uses its full configuration, providing a comparison in which study budget is not the limiting constraint\. Study costs therefore vary with the method, environment, as well as selected archive workflows in the case ofMeta\-Agent w/ Archive; Apex Agents costs additionally sum preparation across its 31 independently studied worlds\. Separately, we setBstudy∈\{$1,$5,$10,$25,$50\}B\_\{\\mathrm\{study\}\}\\in\\\{\\$1,\\$5,\\$10,\\$25,\\$50\\\}for the study\-budget scaling experiments\.

## 6Results

##### RQ1: How do studying methods compare across environments?

Open\-endedMeta\-Agentstudying is the strongest method family across environments\. AMeta\-Agentvariant achieves the highest Avg@3 and Best@3 score on 5 of 6 benchmarks, whileMeta\-Agent w/ Archiveranks first or second on every benchmark under both metrics\. Among the committed strategies,PREPINGis consistently beneficial: it exceedsNo Studyon all 6 benchmarks under both metrics, although it never ranks first\.Corpus2Skillexhibits sharper specialization: it is strongest on BCP\-G but falls belowNo Studyon OfficeQA, Harvey LAB, and DABStep\. Thus, theMeta\-Agents combine strong performance with robustness across heterogeneous settings,PREPINGprovides smaller but consistent gains, andCorpus2Skilltrades broader reliability for strong corpus\-specific performance\. Benchmark\-level inference and pre\-task costs appear in Tables[4](https://arxiv.org/html/2609.10824#A1.T4)and[5](https://arxiv.org/html/2609.10824#A1.T5)\.

##### RQ2: What is the value of the workflow archive?

Archive access for theMeta\-Agenthas a positive but uneven effect\. Under Avg@3, its mean improvement on the four benchmarks where it helps is0\.0460\.046, compared with a mean decline of0\.0210\.021on the two where it hurts \(Table[1](https://arxiv.org/html/2609.10824#S2.T1)\)\. The largest gain is0\.1290\.129on DABStep, whereas the largest decline is0\.0230\.023\. Best@3 shows the same asymmetry: a mean gain of0\.0410\.041where the archive helps and a mean loss of0\.0170\.017where it hurts \(Table[2](https://arxiv.org/html/2609.10824#S2.T2)\)\. The archive therefore produces occasional large improvements without comparably large regressions, rather than a uniform lift across environments\.

##### RQ3: How does performance scale with study budget?

We conduct study budget scaling experiments on OfficeQA, Harvey LAB, and Apex Agents forPREPING,Meta\-Agent w/o Archive, andMeta\-Agent w/ ArchiveforBstudy∈\{$1,$5,$10,$25,$50\}B\_\{\\mathrm\{study\}\}\\in\\\{\\$1,\\$5,\\$10,\\$25,\\$50\\\}\. Results are shown in Figure[2](https://arxiv.org/html/2609.10824#S3.F2)\. Additional study budget does not generally improve downstream performance\. On OfficeQA and Apex Agents, no method exhibits a sustained increase from $1 to $50\. The curves are flat or non\-monotonic, and most intervals overlap\. Harvey LAB is the clear exception\.Meta\-Agent w/o Archiverises from0\.2710\.271to0\.3480\.348andMeta\-Agent w/ Archivefrom0\.2490\.249to0\.3320\.332;PREPINGremains roughly flat after $5\. Thus, in these experiments, study\-budget scaling appears only on Harvey LAB and only for theMeta\-Agentmethods\.

##### RQ4: How does test\-time compute scale with and without study?

Studying can reduce the test\-time sampling required to reach a given score \(Figure[3](https://arxiv.org/html/2609.10824#S4.F3)\)\. The clearest comparison is Best@1 for agents with study artifacts, which uses one rollout per task and requires no oracle selection\. To exceed the strongest studied Best@1 score,No Studyrequires 8 rollouts on BCP\-G, 2 on OfficeQA, 3 on Harvey LAB, 5 on DABStep, 2 on Apex Agents, and 4 on AppWorld\. BecauseNo StudyBest@KKassumes oracle selection among these outputs, these crossover points are optimistic forNo Studytest\-time scaling\. Using the average per\-rollout inference costs in Table[4](https://arxiv.org/html/2609.10824#A1.T4), reaching them costsNo Study1\.61\.6–5\.5×5\.5\\timesas much task\-time inference as the corresponding studied rollout\. This comparison excludes the upfront study cost: studying yields a total cost saving only when its artifacts are reused across enough downstream tasks for that cost to amortize\. At the higher Best@3 target for studied agents,No Studyrequires 6 rollouts on OfficeQA, 7 on Harvey LAB, and 5 on Apex Agents, and does not reach the strongest studied score within 10 rollouts on BCP\-G, DABStep, or AppWorld\.

## 7Interpretability

We examine how systems study and when their artifacts help or hinder the downstream solver\. Appendices[B](https://arxiv.org/html/2609.10824#A2)and[C](https://arxiv.org/html/2609.10824#A3)provide the full protocols, tables, figures, and examples\.

### 7\.1Studying Process

A natural hypothesis is that broader exploration helps a studying system distill an environment\. We test this by relating downstream reward to counts of distinct files accessed and benchmark\-specific tools invoked during study\. Appendix[B\.1](https://arxiv.org/html/2609.10824#A2.SS1)defines the measurement protocol and its limitations\.

Broader exploration does not reliably predict quality\.Figures[6](https://arxiv.org/html/2609.10824#A2.F6)and[7](https://arxiv.org/html/2609.10824#A2.F7)show inconsistent relationships with downstream reward\. On OfficeQA,Corpus2Skillprocesses all 697 files yet trails methods that inspect fewer\. Harvey LAB forms the clearest split: both meta\-agents discover its sole benchmark\-specific tool,labread, and score 0\.35–0\.38, while methods that do not invoke it score 0\.22–0\.31\.

Exploration strategies vary by environment\.We classifyMeta\-Agentstudy traces to identify recurring behaviors\. Appendix[B\.3](https://arxiv.org/html/2609.10824#A2.SS3)provides the behavior taxonomy and extraction protocol\. Table[7](https://arxiv.org/html/2609.10824#A2.T7)shows that inventory, unavailable\-resource tracking, and budget\-conscious extraction are common across environments\. Test anticipation concentrates in Apex Agents and AppWorld, where histories and examples foreshadow future tasks\. Parallel delegation is observed only in Harvey LAB and Apex Agents\. The agents combine a shared diagnostic core with environment\-specific exploration\.

Archive use is environment\-sensitive but imperfect\.The archive\-assisted meta\-agent can combine artifacts from open\-ended study,PREPING, andCorpus2Skill\. The resulting composition varies across environments: open\-ended\-study andPREPINGoutputs appear in every BCP\-G, OfficeQA, and DABStep artifact set, whereasCorpus2Skillappears in every AppWorld artifact set and 86 of 93 Apex Agents artifact sets \(Table[6](https://arxiv.org/html/2609.10824#A2.T6)\)\. On BCP\-G, however, theMeta\-AgentomitsCorpus2Skilleven though it is the strongest standalone method\. Archive access therefore supports adaptive composition without guaranteeing the best choice\.

### 7\.2Solver Process

Benefits concentrate in particular task strata\.On DABStep, theMeta\-Agent w/ Archiveimproves reward by 0\.185 on hard tasks but only 0\.015 on easy tasks, whereNo Studyalready scores 0\.792 \(Table[10](https://arxiv.org/html/2609.10824#A3.T10)\)\. On BCP\-G,Corpus2Skillimproves all ten topics, and its lift rises from 0\.224 with one gold document to 0\.293 with four or more \(Tables[14](https://arxiv.org/html/2609.10824#A3.T14)and[13](https://arxiv.org/html/2609.10824#A3.T13)\)\. Every method obtains its largest AppWorld gain on two\-application tasks \(Table[12](https://arxiv.org/html/2609.10824#A3.T12)\) and its largest Apex Agents domain gain in Law \(Table[11](https://arxiv.org/html/2609.10824#A3.T11)\)\. These patterns do not define a universal notion of task difficulty\.

Artifacts can sometimes misdirect the solver\.Appendix[C\.3](https://arxiv.org/html/2609.10824#A3.SS3)contrasts two cases\. In DABStep, an artifact supplies missing information about a fraud boundary that the solver uses correctly\. In Apex Agents, an incomplete artifact directs the solver toward general pricing documents and away from two task\-specific files containing the required inputs\. Studying can therefore actively mislead the solver, for example by narrowing its search or hiding relevant evidence\.

## 8Discussion and Limitations

Our results provide evidence that task\-agnostic, environment\-conditioned studying is generally beneficial: 21 of 24 method–benchmark comparisons improve overNo Study\.*Dynamically*selecting how to study is robust but not universally optimal\. AMeta\-Agentvariant leads on five of six benchmarks, while the specializedCorpus2Skillpipeline is strongest on BCP\-G\. A promising direction is therefore to improve how the agent selects a studying strategy\. A broader procedure archive or demonstrations of successful environment–strategy matches could improve selection in context\. Across repeated deployments, downstream evaluation feedback could instead be used to update theMeta\-Agent’s strategy\-selection policy – via the weights or harness – turning experience across environments into expertise, in a form of meta\-learning\.

Sustained study\-budget scaling appears only on Harvey LAB and only for theMeta\-Agentmethods\. ForPREPING, this differs from the original study, which reports steady gains with additional synthetic tasks on AppWorld\[[6](https://arxiv.org/html/2609.10824#bib.bib10)\]\. Since our sweep covers different benchmarks, we note that the returns to additional synthetic practice may be environment\-dependent\. For theMeta\-Agent, the budget is stated in the prompt, but effective scaling requires more than budget awareness: the model must track its consumption and plan differently as the allowance changes\. Current models may struggle with both, so a larger budget may extend the same strategy rather than induce a qualitatively different one\. Our experiment therefore measures how the current policy responds to more budget, not what a studier with calibrated budget tracking and planning could achieve\.

Studying has a clearer effect on task\-time scaling, producing reusable structure that can be amortized across future tasks\. One studied rollout reaches scores that require two to eightNo Studyrollouts under oracle selection\. This benefit is not guaranteed, however, and our qualitative examples show that study artifacts can sometimes misdirect the solver in detrimental ways\.

Our conclusions in this work are limited by scope and measurement\. The agentic studiers and frozen solver use models from the same family and share the Claude Code harness, so we cannot establish whether the resulting artifacts transfer across solvers or provide solver\-independent improvements\. We evaluate only one workflow archive and do not compare theMeta\-Agentagainst a non\-agentic classifier that selects one fixed workflow from the same archive based on an environment description\. We therefore cannot pinpoint the value of agentic decision\-making and workflow composition, the contribution of individual archive entries, or the effect of expanding the archive\. Although our benchmarks span different corpus sizes, tool surfaces, and domains, six benchmarks cannot cover every axis along which agent environments vary\. Finally, dollar cost provides a common currency for strategies that use different models, and sometimes multiple models, but collapses heterogeneous computation into a single provider\-dependent proxy\. A model\-independent measure such as FLOPs would avoid this dependence but is difficult to obtain consistently, particularly for closed\-source models\.

## 9Conclusion

We study task\-agnostic environment preparation, in which systems inspect an unfamiliar environment and construct artifacts for a frozen solver before the downstream task distribution is known\. We compare fixed studying strategies with meta\-agent variants that study freely or compose workflows from an archive\. Across six benchmarks, aMeta\-Agentvariant attains the highest Avg@3 reward on five, whileCorpus2Skillis strongest on BCP\-G; archive access produces occasional large gains but not uniform improvements\. Additional study budget does not reliably improve reward, but studied artifacts reduce the test\-time sampling needed to reach a given score and can sometimes misdirect the solver\.

##### Responsible Use

Task\-agnostic preparation may reduce the expert effort required to adapt agents to specialized environments\. Studying systems also explore environments before target tasks are known and persist what they learn for later use, creating four risks\.\(1\) Exploration side effects\.Study may invoke costly or irreversible actions and should use sandboxes, least\-privilege access, and approval gates\.\(2\) Sensitive artifacts\.Artifacts may preserve private data, environment structure, or credentials; they should inherit source access controls and retention policies\.\(3\) Misleading guidance\.Incomplete or stale artifacts can harm downstream performance, as our qualitative analysis illustrates\. Provenance and human review can mitigate this risk\.\(4\) Archive misuse\.Reusable workflows can facilitate exploration in harmful environments\. Archive entries should therefore specify permitted scopes and side\-effect assumptions, and their execution should be logged\.

## References

- \[1\]L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. Khattab\(2025\)GEPA: reflective prompt evolution can outperform reinforcement learning\.arXiv preprint arXiv:2507\.19457\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.19457)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]Anthropic\(2026\)Claude code documentation\.Note:[https://code\.claude\.com/docs/en/overview](https://code.claude.com/docs/en/overview)Accessed: 2026\-09\-01Cited by:[§3](https://arxiv.org/html/2609.10824#S3.p1.1)\.
- \[3\]Z\. Chen, X\. Ma, S\. Zhuang, P\. Nie, K\. Zou, A\. Liu, J\. Green, K\. Patel, R\. Meng, M\. Su, S\. Sharifymoghaddam, Y\. Li, H\. Hong, X\. Shi, X\. Liu, N\. Thakur, C\. Zhang, L\. Gao, W\. Chen, and J\. Lin\(2025\)BrowseComp\-Plus: a more fair and transparent evaluation benchmark of deep\-research agent\.arXiv preprint arXiv:2508\.06600\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2508.06600)Cited by:[Table 3](https://arxiv.org/html/2609.10824#A1.T3.2.2.1.1.1),[§C\.2](https://arxiv.org/html/2609.10824#A3.SS2.p1.1),[§5](https://arxiv.org/html/2609.10824#S5.SS0.SSS0.Px1.p1.1)\.
- \[4\]P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav\(2025\)Mem0: building production\-ready AI agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.19413)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]D\. Choi, K\. Park, W\. Song, S\. Dingliwal, S\. M\. Jayanthi, J\. Shin, and A\. Galstyan\(2026\)IdleSpec: exploiting idle time via speculative planning for LLM agents\.arXiv preprint arXiv:2605\.22154\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.22154)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]Y\. Choi, S\. Park, M\. Kang, J\. Baek, and S\. J\. Hwang\(2026\)PREPING: building agent memory without tasks\.arXiv preprint arXiv:2605\.13880\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.13880)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p3.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.10824#S4.SS1.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2609.10824#S8.p2.1)\.
- \[7\]D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, D\. Metropolitansky, R\. O\. Ness, and J\. Larson\(2024\)From local to global: a graph RAG approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2404.16130)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p3.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[8\]A\. Egg, M\. Iglesias Goyanes, F\. Kingma, A\. Mora, L\. von Werra, and T\. Wolf\(2025\)DABstep: data agent benchmark for multi\-step reasoning\.arXiv preprint arXiv:2506\.23719\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.23719)Cited by:[Table 3](https://arxiv.org/html/2609.10824#A1.T3.2.5.1.1.1),[§C\.2](https://arxiv.org/html/2609.10824#A3.SS2.p1.1),[§5](https://arxiv.org/html/2609.10824#S5.SS0.SSS0.Px1.p1.1)\.
- \[9\]R\. Fang, Y\. Liang, X\. Wang, J\. Wu, S\. Qiao, P\. Xie, F\. Huang, H\. Chen, and N\. Zhang\(2025\)Memp: exploring agent procedural memory\.arXiv preprint arXiv:2508\.06433\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2508.06433)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]C\. Fernando, D\. Banarse, H\. Michalewski, S\. Osindero, and T\. Rocktäschel\(2023\)Promptbreeder: self\-referential self\-improvement via prompt evolution\.arXiv preprint arXiv:2309\.16797\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2309.16797)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]N\. Grupen, G\. Pereyra, and J\. Pereyra\(2026\)Harvey’s legal agent benchmark \(LAB\)\.Note:Harvey\.[https://github\.com/harveyai/harvey\-labs](https://github.com/harveyai/harvey-labs)Cited by:[Table 3](https://arxiv.org/html/2609.10824#A1.T3.2.4.1.1.1),[§5](https://arxiv.org/html/2609.10824#S5.SS0.SSS0.Px1.p1.1)\.
- \[12\]Harbor Framework Team\(2026\)Harbor: A framework for evaluating and optimizing agents and models in container environments\.Note:ZenodoExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.20953922),[Link](https://doi.org/10.5281/zenodo.20953922)Cited by:[§3](https://arxiv.org/html/2609.10824#S3.p1.1)\.
- \[13\]H\. Hu, Q\. Lyu, X\. Kong, W\. Liu, J\. Lin, Z\. Guo, Y\. Xu, Y\. Wang, W\. Zhang, and Y\. Yu\(2026\)Anticipate and learn: unleashing idle\-time compute in proactive agents\.arXiv preprint arXiv:2605\.25971\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.25971)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]V\. Karpukhin, B\. Oğuz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih\(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6769–6781\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts\(2023\)DSPy: compiling declarative language model calls into self\-improving pipelines\.arXiv preprint arXiv:2310\.03714\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.03714)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]T\. Le Sellier De Chezelles, M\. Gasse, A\. Drouin, M\. Caccia, L\. Boisvert, M\. Thakkar, T\. Marty, R\. Assouel, S\. O\. Shayegan, L\. K\. Jang, X\. H\. Lù, O\. Yoran, D\. Kong, F\. F\. Xu, S\. Reddy, Q\. Cappart, G\. Neubig, R\. Salakhutdinov, N\. Chapados, and A\. Lacoste\(2024\)The BrowserGym ecosystem for web agent research\.arXiv preprint arXiv:2412\.05467\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.05467)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p1.1)\.
- \[17\]Y\. Lee, R\. Nair, Q\. Zhang, K\. Lee, O\. Khattab, and C\. Finn\(2026\)Meta\-harness: end\-to\-end optimization of model harnesses\.InConference on Language Modeling \(COLM\),Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2005\.11401External Links:[Document](https://dx.doi.org/10.48550/arXiv.2005.11401)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[19\]J\. X\. Li, R\. Battle, and O\. Khattab\(2026\)Machine studying: a system\-level reframing of continual adaptation from declarative corpora\.InContinual Adaptation at Scale: Towards Sustainable AI Workshop,External Links:[Link](https://openreview.net/forum?id=e7duOnFgL7)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p3.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]K\. Lin, C\. Snell, Y\. Wang, C\. Packer, S\. Wooders, I\. Stoica, and J\. E\. Gonzalez\(2025\)Sleep\-time compute: beyond inference scaling at test\-time\.arXiv preprint arXiv:2504\.13171\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.13171)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]B\. Liu, C\. Jin, S\. Kim, W\. Yuan, W\. Zhao, I\. Kulikov, X\. Li, S\. Sukhbaatar, J\. Lanchantin, and J\. Weston\(2025\)SPICE: self\-play in corpus environments improves reasoning\.External Links:2510\.24684,[Link](https://arxiv.org/abs/2510.24684)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p3.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]K\. Opsahl\-Ong, A\. Singhvi, J\. Collins, I\. Zhou, C\. Wang, A\. Baheti, O\. Oertell, J\. Portes, S\. Havens, E\. Elsen, M\. Bendersky, M\. Zaharia, and X\. Chen\(2026\)OfficeQA Pro: an enterprise benchmark for end\-to\-end grounded reasoning\.arXiv preprint arXiv:2603\.08655\.Note:Benchmark suite:[https://github\.com/databricks/officeqa](https://github.com/databricks/officeqa)External Links:[Document](https://dx.doi.org/10.48550/arXiv.2603.08655)Cited by:[Table 3](https://arxiv.org/html/2609.10824#A1.T3.2.3.1.1.1),[§C\.2](https://arxiv.org/html/2609.10824#A3.SS2.p1.1),[§5](https://arxiv.org/html/2609.10824#S5.SS0.SSS0.Px1.p1.1)\.
- \[23\]C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez\(2023\)MemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.08560)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]S\. Rajesh, P\. Holur, M\. Y\. Turali, C\. Duan, and V\. Roychowdhury\(2026\)Panini: continual learning in token space via structured memory\.arXiv preprint arXiv:2602\.15156\.Note:Published at ICML 2026External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.15156)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[25\]S\. E\. Robertson and H\. Zaragoza\(2009\)The probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.External Links:[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[26\]P\. Sarthi, S\. Abdullah, A\. Tuli, S\. Khanna, A\. Goldie, and C\. D\. Manning\(2024\)RAPTOR: recursive abstractive processing for tree\-organized retrieval\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2401\.18059External Links:[Document](https://dx.doi.org/10.48550/arXiv.2401.18059)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p3.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36,pp\. 8634–8652\.External Links:[Document](https://dx.doi.org/10.52202/075280-0377)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[28\]Y\. Sun, P\. Wei, and L\. B\. Hsieh\(2026\)Corpus2Skill: distilling enterprise knowledge into navigable agent skills for QA and RAG\.arXiv preprint arXiv:2604\.14572\.Note:Accepted to EMNLP 2026 \(Findings\)\. Titled “Don’t Retrieve, Navigate” in v1External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.14572)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p3.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.10824#S4.SS1.SSS0.Px2.p1.1)\.
- \[29\]M\. Suzgun, M\. Yüksekgönül, F\. Bianchi, D\. Jurafsky, and J\. Zou\(2025\)Dynamic cheatsheet: test\-time learning with adaptive memory\.arXiv preprint arXiv:2504\.07952\.Note:Published at EACL 2026External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.07952)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[30\]H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. Balasubramanian\(2024\)AppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 16022–16076\.Note:arXiv:2407\.18901External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.850)Cited by:[Table 3](https://arxiv.org/html/2609.10824#A1.T3.2.7.1.1.1),[§C\.2](https://arxiv.org/html/2609.10824#A3.SS2.p1.1),[§5](https://arxiv.org/html/2609.10824#S5.SS0.SSS0.Px1.p1.1)\.
- \[31\]V\. Ursekar, A\. Shanker, V\. Chatrath, Y\. Xue, and S\. M\. Denton\(2026\)VeRO: a harness for agents to optimize agents\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=zQzmwG2Nue)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[32\]B\. Vidgen, A\. Mann, A\. Fennelly, J\. W\. Stanly, L\. Rothman, M\. Burstein, J\. Benchek, D\. Ostrofsky, A\. Ravichandran, D\. Sur, N\. Venugopal, A\. Hsia, I\. Robinson, C\. Huang, O\. Varones, D\. Khan, M\. Haines, A\. Bridges, J\. Boyle, K\. Twist, Z\. Richards, C\. Mahapatra, B\. Foody, and O\. Nitski\(2026\)APEX\-Agents\.arXiv preprint arXiv:2601\.14242\.Note:The AI Productivity Index for agentsExternal Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.14242)Cited by:[Table 3](https://arxiv.org/html/2609.10824#A1.T3.2.6.1.1.1),[§C\.2](https://arxiv.org/html/2609.10824#A3.SS2.p1.1),[§5](https://arxiv.org/html/2609.10824#S5.SS0.SSS0.Px1.p1.1)\.
- \[33\]G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar\(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2305.16291)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px2.p1.1)\.
- \[34\]Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig\(2024\)Agent workflow memory\.arXiv preprint arXiv:2409\.07429\.Note:Published at ICML 2025 \(PMLR v267, pp\. 63897–63911\)External Links:[Document](https://dx.doi.org/10.48550/arXiv.2409.07429)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[35\]L\. Weng\(2026\)Harness engineering for self\-improvement\.lilianweng\.github\.io\.External Links:[Link](https://lilianweng.github.io/posts/2026-07-04-harness/)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p1.1)\.
- \[36\]W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang\(2025\)A\-MEM: agentic memory for LLM agents\.arXiv preprint arXiv:2502\.12110\.Note:Accepted at NeurIPS 2025External Links:[Document](https://dx.doi.org/10.48550/arXiv.2502.12110)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[37\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2405\.15793External Links:[Document](https://dx.doi.org/10.48550/arXiv.2405.15793)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p1.1)\.
- \[38\]Y\. Yao, X\. Tan, C\. Liu, Y\. Li, Z\. Wang, W\. Yu, Z\. Tan, Y\. Tian, G\. Zhao, L\. Sun, X\. Zhang, and T\. Yang\(2026\)Harness\-bench: measuring harness effects across models in realistic agent workflows\.External Links:2605\.27922,[Link](https://arxiv.org/abs/2605.27922)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p1.1)\.
- \[39\]C\. Ye, Y\. Liu, Y\. Wang, H\. Yu, Y\. Zhao, G\. Liu, J\. McAuley, and J\. You\(2026\)Auto\-Dreamer: learning offline memory consolidation for language agents\.arXiv preprint arXiv:2605\.20616\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.20616)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[40\]Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li, U\. Thakker, J\. Zou, and K\. Olukotun\(2025\)Agentic context engineering: evolving contexts for self\-improving language models\.arXiv preprint arXiv:2510\.04618\.Note:Published at ICLR 2026External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.04618)Cited by:[§1](https://arxiv.org/html/2609.10824#S1.p2.1),[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.
- \[41\]A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang\(2024\)ExpeL: LLM agents are experiential learners\.Proceedings of the AAAI Conference on Artificial Intelligence38\(17\),pp\. 19632–19642\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i17.29936)Cited by:[§2](https://arxiv.org/html/2609.10824#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ASupporting Materials

This section provides supporting material for our experiment setup, main results, and analysis\.

### A\.1Benchmark and Environment Details

Figure[4](https://arxiv.org/html/2609.10824#A1.F4)and Table[3](https://arxiv.org/html/2609.10824#A1.T3)provide complementary views of the counts and types of the files and tools in each of the benchmarks we investigate\.

files exposed to agent \(log scale\)benchmark\-specific tools \(log scale\)0101001K10K100K0110100DABStep7 filesOfficeQA697 filesHarvey LAB1 tool, 9,288 filesBCP\-G∼\\sim100K filesAppWorld≥\\geq105 toolsApex Agents8 tool entry pointsFigure 4:File and tool counts across benchmark environments\.Axes showlog10⁡\(1\+n\)\\log\_\{10\}\(1\+n\)so environments with 0 benchmark\-specific tools or real files remain visible\. Apex’s horizontal interval is its per\-world file range \(4–170, median∼\\sim40\)\. AppWorld exposes at least 105 distinct tools across observed traces, while its state lives in app databases rather than real files\. Generic shell and Python operations are not counted as benchmark\-specific tools\.Table 3:Files and benchmark\-specific tools available in each benchmark environment\.
### A\.2Additional Results

This section reports additional performance and cost results that accompany the main analysis\.

Table 4:Average inference cost per rollout\.Values are mean±\\pmsample SD in USD\. Studying\-method statistics are across three independent artifact iterations\.No Studystatistics are across 10 full task\-time repetitions\. Unpriced rollouts are excluded rather than imputed\.Table 5:Average pre\-task cost\.Values are mean±\\pmsample SD in USD\. Statistics are across three independent artifact\-construction iterations\.No Studyperforms no pre\-task computation and is shown as dashes\. The Apex AgentsCorpus2Skillcell, marked‡\\ddagger, uses the two metered artifact\-construction iterations because the original iteration did not retain cost telemetry\.Figure[5](https://arxiv.org/html/2609.10824#A1.F5)disaggregates theMeta\-Agent w/ ArchiveAvg@3 lift overNo Studyacross individual Apex Agents worlds, grouped by domain\.

Figure 5:Meta\-Agent w/ Archivelift by APEX Agents world\.Each bar shows the signed percentage\-point difference betweenMeta\-Agent w/ ArchiveAvg@3 and theNo StudyAvg@3 baseline for one APEX Agents world\. Within each of three independent artifact iterations, the studied score initially averages the three evaluation rewards for each task and then macro\-averages tasks\. The plotted score averages those three artifact\-iteration estimates\. ForNo Study, we compute the three\-repetition mean for each of the\(103\)\\binom\{10\}\{3\}subsets and average those subset estimates\. This point estimate is algebraically equal to the 10\-repetition mean\. Bar color denotes domain\. World scores macro\-average the tasks in that world\.
### A\.3Additional Implementation Details

#### A\.3\.1Studying method configuration

This section reports implementation choices not specified in the main text\.

##### Common configuration\.

Agentic study runs use Claude Code with permission bypass enabled and web access disabled; otherwise, they retain provider defaults\.

##### PREPING\.

Synthetic tasks run through the corresponding benchmark’s task\-agent harness using a task\-neutral environment projection\. We use a feasibility threshold of 5/5 and treat completion scores of at least 4/5 as successful\. Proposal and validation temperatures are 0\.7 and 0, respectively\. During construction and delivery, at most 40 playbook entries are retrieved usingtext\-embedding\-3\-small\. Individual practice rollouts are not dollar\-capped\. Only validator\-approved feasible trajectories are reflected on and curated into the playbook\.

##### Corpus2Skill\.

We use the published defaults except for benchmark\-specific adjustments to hierarchy branching, document length, and tree compaction\. We increase the branching ratio from 10 to 40 on OfficeQA and Harvey LAB\. We use a 4,000\-character document limit and compact tree representation on DABStep, OfficeQA, and Harvey LAB; the remaining benchmarks retain the default 8,000\-character limit and non\-compact representation\.

##### Artifact delivery boundary\.

No Studymakes no environment or prompt change\. Each studying method exposes only its frozen output through its native interface: retrieved playbook entries forPREPING, the compiled skill hierarchy and document accessor forCorpus2Skill, and mounted artifacts for the twoMeta\-Agentvariants\. The original target instruction is preserved in every arm\.

#### A\.3\.2Representative implementation prompts

The listings below show one representative prompt for each shared method component needed to interpret the intervention\. Benchmark\-specific environment clauses, proposer and validator variants, and submission suffixes are omitted here\. The implementation and released run records contain the complete rendered prompts\. Punctuation and formatting are normalized to the manuscript style\.

##### PREPINGtask proposer\.

You are a synthetic task generator for an AI agent to construct the memory in the environment\.Your goal is to generate diverse, realistic, and challenging task instructions that an AI assistant would need to complete\. These tasks should reflect real\-world user scenarios\.\#\# API Documentation\{api\_docs\}\#\# Guidelines1\. \*\*Feasibility\*\*: Generate tasks that are actually executable with the given APIs2\. \*\*Naturalness\*\*: Write as a real user would phrase them, not as technical API calls3\. \*\*Verifiability\*\*: Instructions must be precise enough to allow for an objective judgment of success or failure\.\- The agent will submit its answer as a single text string\. Success is judged by comparing that string to the expected value\.\- For this to be verifiable, each QUERY must require \*\*exactly one type of answer\*\* \(e\.g\. one number, or one name/title, or one list\.\)\- Do not ask for two or more distinct values in one task \(e\.g\. both a count and a list, or both title and artist\)\. Otherwise the expected format of the submitted text is ambiguous and cannot be reliably checked\.4\. \*\*Entity Use\*\*:\- Prefer realistic, natural task phrasing over rigid placeholder patterns\.\- If \*\*environment information is provided\*\*, treat it as optional grounding support:\- You may use discovered names, titles, folders, contacts, and other entities when they help make a task concrete\.\- You do not need to force those entities into every task\.\- If \*\*environment information is empty or missing\*\*, you may use generic user\-centric references such as:\- "my favorite playlist", "my song library", "my playlists", "my album library", "artists I follow"\#\# Task Category GuidelinesGenerate tasks across the following categories:\#\#\# 1\. QUERY Tasks \(Information Retrieval\)Tasks that require finding and returning specific information\. The answer should be objective and verifiable \(number, name, comma\-separated list, yes/no\)\.Include concrete details with specific output format such as numbers, dates, names, locations, thresholds, item lists, etc\.Avoid vague terms such as ’check’, ’review’, ’show’ or ’ensure’ unless they are accompanied by measurable criteria\.\#\#\# 2\. ACTION Tasks \(State Modification\)Tasks that require modifying the state of one or more apps\. No explicit answer required \- success is determined by state changes\.\{environment\_info\_section\}\{task\_history\_section\}\#\# Output FormatReturn a JSON array of task instructions:\`\`\`json\[\{\{"category": "QUERY\|ACTION","involved\_apps": \["app1", "app2"\],"involved\_apis": \["app\.api1", "app\.api2"\],"instruction": "Natural language task instruction"\}\}\]\`\`\`Generate \{num\_tasks\} diverse task instructions:

##### PREPINGtrajectory validator\.

You are an expert evaluator of agent task execution\.Evaluate the following Task and Trajectory using a 5\-point Likert scale for each criterion\.\#\# Evaluation Criteria\#\#\# 1\. Task FeasibilityEvaluate whether the task is executable and the entities are grounded in the current environment\.\- Do not judge feasibility by API availability alone\.\- Verify that every required target entity in the instruction \(person, contact, request, file, thread, project, etc\.\) is present and can be uniquely identified from the trajectory evidence\.\- If the entity cannot be identified after reasonable paginated search using relevant APIs, the task is infeasible\.\- \*\*5 \(Excellent\)\*\*: All required entities/constraints are explicitly present and uniquely identified\. Clearly executable as stated\.\- \*\*4 \(Good\)\*\*: Entities are present with minor ambiguity resolvable by a simple lookup\. No core entity missing\.\- \*\*3 \(Acceptable\)\*\*: Plausible, but key grounding is uncertain or search is insufficient\.\- \*\*2 \(Poor\)\*\*: Likely infeasible\. Required entity not found or not uniquely identifiable after reasonable search\.\- \*\*1 \(Unacceptable\)\*\*: Infeasible/contradictory or relies on non\-existent entities/conditions\.\#\#\# 2\. Task CompletionJudge whether the task instruction was successfully completed\.\- Evaluate all task instructions are satisfied based on the trajectory, including all constraints\.\- Treat a call to \`apis\.supervisor\.complete\_task\(\.\.\.\)\` as a required completion step, not as evidence that the task was successful correctly\.\- If the task asked for action, do not treat the task as successful unless the trajectory includes a call to \`apis\.supervisor\.complete\_task\(\)\` with no answer argument \(or with \`None\`\)\.\- If the task asked for information, success requires a call to \`apis\.supervisor\.complete\_task\(answer=<answer\>\)\`, and the answer should contain only the exact requested value: a number \(int or float\), a direct string value, a plain comma\-separated list, or \`yes\`/\`no\`\.\- Do not treat verbose answers, action summaries, explanations, full sentences, or answers with extra words, units, or symbols as correct unless explicitly requested\. e\.g\. \`10\` is acceptable, but \`The answer is 10 songs\.\` is not\. Do not require an answer for action\-only tasks\.\- \*\*5 \(Excellent\)\*\*: Every requirement AND every constraint is satisfied, and the trajectory shows clear completion for each required outcome\.\- \*\*4 \(Good\)\*\*: All requirements/constraints appear satisfied with minor ambiguity, and nothing important is missing\.\- \*\*3 \(Acceptable\)\*\*: Some progress, but at least one requirement/constraint is missing, ambiguous, or only asserted without support in the trajectory steps \(surface\-level success\)\. This includes cases where an object/ID exists but correctness constraints \(privacy, filters, counts, formatting, attachment actually uploaded, etc\.\) are not confirmed\.\- \*\*2 \(Poor\)\*\*: Minor progress only\. The main outcome is not achieved or an error/failed response prevents completion\.\- \*\*1 \(Unacceptable\)\*\*: No meaningful progress toward the task goal\.\#\# Task Instruction\{task\_instruction\}\#\# Trajectory\{trajectory\}Respond in JSON format:\`\`\`json\{\{"feasibility\_reason": "1\-2 sentence explanation for feasibility score","feasibility\_score": 1\-5,"task\_completion\_reason": "1\-2 sentence explanation for task completion score","task\_completion\_score": 1\-5\}\}\`\`\`

##### PREPINGreflector\.

You are an expert agent and educator\. Your job is to diagnose the current trajectory: identify what went wrong \(or could be better\), API usage, and ground truth when applicable\.Instructions:\- Carefully analyze the model’s reasoning trace to identify where it went wrong\- Take the environment feedback into account, comparing the predicted answer with the \(optional\) ground truth to understand the gap\- Identify specific conceptual errors, calculation mistakes, or misapplied strategies\- Provide actionable insights that could help the model avoid this mistake in the future\- Identify root causes: wrong source of truth, bad filters \(timeframe/direction/identity\), formatting issues, or missing authentication and how to correct them\.\- Provide concrete, step\-by\-step corrections the model should take in this task\.\- Be specific about what the model should have done differently\- You will receive bulletpoints that are part of playbook that’s used by the generator to answer the question\.\- You need to analyze these bulletpoints, and give the tag for each bulletpoint, tag can be \[’helpful’, ’harmful’, ’neutral’\] \(for the generator to generate the correct answer\)\- Explicitly curate from the environment feedback the output format/schema of APIs used when unclear or mismatched with expectations \(e\.g\., apis\.blah\.show\_contents\(\) returns a list of content\_ids \(strings\), not content objects\)Inputs:Task Instruction:\{\{task\_description\}\}\{\{ground\_truth\_result\}\}\{\{ground\_truth\_code\}\}\{\{unit\_test\_results\}\}PrePing playbook \(playbook that’s used by model for code generation\):PLAYBOOK\_START\{\{playbook\}\}PLAYBOOK\_ENDAgent\-Environment Trajectory \(including reasonings actions, and final observation\):\{\{trajectory\}\}Outputs: Your output should be a json object, which contains the following fields\- reasoning: your chain of thought / reasoning / thinking process, detailed analysis and calculations\- error\_identification: what specifically went wrong in the reasoning?\- root\_cause\_analysis: why did this error occur? What concept was misunderstood?\- correct\_approach: what should the model have done instead?\- key\_insight: what strategy, formula, or principle should be remembered to avoid this error?\- bullet\_tags: a dictionary mapping each bullet\_id \(the \{section\}\-\{number\} prefix shown in the playbook\) to its tag \(’helpful’, ’harmful’, or ’neutral’\)Answer in this exact JSON format:\{"reasoning": "\[Your chain of thought / reasoning / thinking process, detailed analysis and calculations\]","error\_identification": "\[What specifically went wrong in the reasoning?\]","root\_cause\_analysis": "\[Why did this error occur? What concept was misunderstood?\]","correct\_approach": "\[What should the model have done instead?\]","key\_insight": "\[What strategy, formula, or principle should be remembered to avoid this error?\]","bullet\_tags": \{"bullet\_id\_1": "helpful", "bullet\_id\_2": "harmful", "bullet\_id\_3": "neutral"\}\}

##### PREPINGcurator\.

You are a master curator of knowledge\. Your job is to identify what new insights should be added to an existing playbook based on a reflection from a previous attempt\.Context:\- The playbook you created will be used to help answering similar questions\.Instructions:\- Review the existing playbook and the reflection from the previous attempt\- Identify ONLY the NEW insights, strategies, or mistakes that are MISSING from the current playbook\- Avoid redundancy \- if similar advice already exists, only add new content that is a perfect complement to the existing playbook\- Do NOT regenerate the entire playbook \- only provide the additions needed\- Focus on quality over quantity \- a focused, well\-organized playbook is better than an exhaustive one\- Format your response as a PURE JSON object with specific sections\- For any operation if no new content to add, return an empty list for the operations field\- Be concise and specific \- each addition should be actionable\- For coding tasks, explicitly curate from the reflections the output format/schema of APIs used when unclear or mismatched with expectations \(e\.g\., apis\.blah\.show\_contents\(\) returns a list of content\_ids \(strings\), not content objects\)Task Instruction:\{question\_context\}Current Playbook:\{current\_playbook\}Agent\-Environment Trajectory \(actions and outputs from the attempt\):\{trajectory\}Current Reflections \(principles and strategies that helped to achieve current task\):\{guidebook\}Your Task:Output ONLY a valid JSON object with these exact fields:\- reasoning: your chain of thought / reasoning / thinking process, detailed analysis and calculations\- operations: a list of operations to be performed on the playbook\- type: the type of operation to be performed\- section: the section to add the bullet to \(one of: strategies, code\_snippets, pitfalls, apis\)\- content: the new content of the bulletAvailable Operations:1\. ADD: Create new bullet points with fresh IDs\- section: the section to add the new bullet to\- content: the new content of the bullet\. Note: no need to include the bullet\_id in the content like ’\[ctx\-00263\] helpful=1 harmful=0 ::’, the bullet\_id will be added by the system\.RESPONSE FORMAT \- Output ONLY this JSON structure \(no markdown, no code blocks\):\{\{"reasoning": "\.\.\.","operations": \[\{\{ "type": "ADD", "section": "\.\.\.", "content": "\.\.\." \}\}\]\}\}

##### Unaided exploratory\-study prompt, represented by BCP\-G\.

ENVIRONMENT : BCP\-Grep study\. The material you are studying \(the "corpus" referred to below\) is a fixed collection of plaintext documents under \`/workspace/corpus/\`\. Each \`<docid\>\.txt\` file starts with its \`DOCID\` and \`URL\`, followed by the complete document text\. Use Claude Code’s native \`Grep\` across \`/workspace/corpus\`, then \`Read\` matching files\. Use \`Glob\` when filename discovery is useful\. There is no web access\. You may write ONLY under \`/logs/artifacts/cheatsheet/\`\.You are studying for a test that you have no prior information about other than that it will be based on the environment you have been placed in : the documents it contains, and the tools available to you for finding and reading them\. You are not allowed to search the web at any point or to go outside this environment\. You are given a limited budget in which you may explore as you wish: inspect the corpus to learn where different kinds of information live and exercise the available tools to learn what they do and how they behave\. Your goal is to leave behind a minimal set of artifacts in the /logs/artifacts/cheatsheet/ directory that inform a future, budget\-constrained version of yourself as much as possible without overwhelming it : maximize useful signal per token, and add nothing that is merely noise\. A test\-time agent will have the same access you do, so its budget is wasted on exactly two things: finding where relevant information lives in an unfamiliar corpus, and figuring out how to operate unfamiliar tools correctly\. Spend your study budget mapping those two things and little else : where to look for a given kind of information, and how to use each non\-obvious tool \(its purpose, the few calls that matter, and any quirks or pitfalls you actually hit\)\. Prefer pointers and short operating notes over copied content: say where to look and how to act rather than reproducing material, which can go stale and invites blind reliance\. Be ruthless about clutter: do not record anything obvious from a document’s title or a tool’s name, anything a test\-time agent could rediscover in a step or two, or general knowledge you already possess : if a line does not save meaningful test\-time budget, delete it\. At test time you will be given read access to whatever you created in /logs/artifacts/cheatsheet/ along with the full environment, but you will be on a limited inference budget, so the quality, brevity, and accessibility of what you leave behind may greatly affect your test\-time performance\. Proceed however you like, but the only write access you have is to the /logs/artifacts/cheatsheet/ directory, and you may explore the corpus through its native filesystem tools and the /logs/artifacts/cheatsheet/ directory\.:GROUNDING REQUIREMENT: every fact, entity, claim, entry, table row, Q&A pair, map node, or note recorded in an artifact MUST cite the exact numeric document ID and file path, for example \`\[5412\] /workspace/corpus/5412\.txt\`\. A future agent must be able to open the cited file directly with \`Read\` instead of repeating discovery\. For tool\-operating notes, name the native tool, minimal call pattern, path, and any observed quirk or pitfall\.

##### Corpus2Skilldownstream navigation prompt\.

You are navigating a closed, precompiled corpus through Claude Code skills and one exact\-ID document tool\.For every task:1\. Invoke at least two plausible top\-level skills with the Skill tool before settling on a branch\.2\. Descend through the selected skills by reading their \`SKILL\.md\` and nested \`INDEX\.md\` files with Read\. Use Glob to discover relevant index files and Grep only within the skill/index tree\. Consult \`entity\_index\.json\` when named entities can disambiguate the route\. If a branch is thin or unhelpful, cross\-jump to a related or second plausible skill\.3\. Select exact document IDs from leaf indexes before calling \`get\_document\`\. The document tool does not search or list the corpus\.4\. Make at least one successful \`get\_document\` call\. Treat retrieved full documents, rather than skill summaries, index summaries, or entity mappings, as factual evidence for the answer\.Do not use corpus\-search tools or open\-web tools\. Do not read, grep, parse, or run scripts against \`documents\.json\`\. Access full documents only through \`get\_document\` with IDs found in leaf indexes\.

##### Meta\-Agent w/ ArchivePrompt

\# Preparation goalPrepare one frozen, task\-blind bundle that gives the later target solver the best chance of solvingunknown tasks from this environment group\. You may inspect only the neutral environment projection\.real target instructions, verifier code, metric details, answer\-bearing files, and real baselinetraces are unavailable and must remain unavailable\.You are the controller\. Every synthetic probe and every eventual target run uses the pinnedtarget solver through the real harness\. The elapsed deadline began before this session invocation\.resuming never resets it\. The gateway is the authority for the deadline, active work, costs, andcommitted final state\. If the main deadline fires, all exploration/workflow/probe tools are disabledand active jobs are canceled\. A short hard\-bounded grace permits only working\-note updates andprefix/bundle consolidation from workflow jobs that completed before the deadline\. It does notpermit new artifacts, probes, or workflow execution\. Expiry commits \`timeout\_forced\_null\`\.\#\# Durable work\`working\-notes\.md\` in durable scratch is your authoritative diagnosis and decision record\. Update it:\- after your initial diagnosis,\- after every meaningful probe or workflow result,\- before entering a long \`await\_jobs\` call,\- whenever you change the candidate stack, and\- immediately before finalization\.Keep draft prefix text, your ordinary artifacts, and workflow checkpoints in durable scratch\. Asession may be compacted or interrupted, so do not leave the only copy of a decision in conversationhistory or transient filesystem state\.Use \`update\_working\_notes\` so each update is fsynced and recorded in the append\-only ledger\.\#\# Workflows and probesThe control skills describe Cheatsheet, PREPING, Corpus2Skill, and synthetic probe invocation withoutperformance or routing advice\. You may invoke, configure, combine, compare, or decline them\. Selectat most one successful output from each workflow family\. PREPING delivery is one complete staticplaybook shared by the group\. It must never retrieve against a real target instruction\. Controller\-authoredfiles are ordinary inert artifacts and never new Claude Code skills\.Gateway submissions return durable job IDs immediately\. When work is outstanding, call\`await\_jobs\(job\_ids, return\_when\)\` once and let it block\. Do not implement sleeps, polling loops,repeated status prompts, or conversational check\-ins\. The wait wakes for a requested terminal job,the 30\-minute warning, the final deadline, or infrastructure requiring attention\. Before a longwait, update working notes\.\#\# Final bundleThe admitted \`/meta\_archive\` layout is fixed: \`manifest\.toml\`, \`prefix\.md\`, canonical workflow skillsunder \`skills/\`, workflow artifacts under \`artifacts/\{cheatsheet,preping,corpus2skill\}/\`, your inertartifacts under \`artifacts/meta/\`, and declared additive tools/data under \`tools/\`\. Seed control skillsnever transfer\. Native solver tools and filesystem access remain available\. Canonical prompt blocksdescribe actual validated knobs and allocated deliverables\. You may weave, reorder, shorten, extend,or decline them, but every interface named in the final prefix must be truthful\.Call \`finalize\_bundle\` to validate and lock the static bundle, cancel unselected work, and endpreparation\. You may instead call \`choose\_null\`\. A deadline or unrecoverable crash forces null\. Allnull states receive the unmodified baseline handoff\. Never write or infer a real task instruction\.

## Appendix BStudy Time Interpretability

This section details our methods for analyzing how the studying systems explored each environment, which behaviors they exhibited, and what they produced\.

### B\.1Environment Exploration Analysis

We measure environment exploration using the number of unique files read and benchmark\-specific tools invoked during study\. Figures[6](https://arxiv.org/html/2609.10824#A2.F6)and[7](https://arxiv.org/html/2609.10824#A2.F7)relate these quantities to downstream reward for each benchmark, method, and artifact iteration\.

Figure 6:File access and downstream reward\.Each point is one benchmark–method–artifact\-iteration combination\. The horizontal axis counts unique environment files accessed during the analyzed study attempt\. The vertical axis is that artifact’s task\-macro Avg@3 reward\. Dashed lines are descriptive per\-benchmark Theil–Sen fits of reward onlog10\\log\_\{10\}\(files\), pooled across methods and iterations\. The annotatedR2R^\{2\}is computed against the robust line\. DABStep is omitted because all methods access all seven files\. AppWorld has no real file surface\.Figure 7:Tool use and downstream reward\.Each point is one benchmark–method–artifact\-iteration combination\. The horizontal axis counts distinct benchmark\-specific tools invoked during the analyzed study attempt\. The vertical axis is that artifact’s task\-macro Avg@3 reward\. Dashed lines are descriptive per\-benchmark Theil–Sen fits pooled across methods and iterations, withR2R^\{2\}computed against the robust line\. Harvey LAB has one benchmark\-specific tool, so its fit represents a method\-confounded binary split rather than a continuous breadth relationship\.##### Measurement\.

For trace\-based methods, we count a file when its contents appear in the study trace through a direct read or an attributed search result\. Directory listings and filename mentions do not count\. ForCorpus2Skill, which has no interactive trace, we instead count the documents processed by its compiler\. We count each benchmark\-specific tool invoked at least once, including failed calls\. Shell and generic file operations are excluded\.

##### Scope\.

Counts are aggregated over the available study traces for each benchmark, method, and artifact iteration\. Apex Agents additionally aggregates across its 31 worlds\. DABStep is omitted from the file analysis because every method reads all seven files, and AppWorld because it exposes no real files\. Trace\-derived file counts are lower bounds because internal access and truncated results may not expose every file read\.

##### Results\.

We fit a Theil–Sen line to the points for each benchmark\. These fits are descriptive: exploration and method vary together, so they do not isolate the effect of reading an additional file or invoking an additional tool\. File\-count slopes are nearly flat for Harvey LAB and Apex Agents, negative for OfficeQA, and positive for BCP\-G\. The stronger OfficeQA and BCP\-G associations are driven largely byCorpus2Skill, which processes the entire corpus and therefore lies far from the trace\-based methods on the file axis\. Tool\-count slopes are also nearly flat for Apex Agents and AppWorld\. Harvey LAB is the exception: bothMeta\-Agentvariants invokelabreadand outperform methods that do not, although tool use and method identity are confounded\. Overall, exploration breadth may matter, but it does not by itself determine downstream performance\.

### B\.2Composition of Archive\-Assisted Artifacts

Meta\-Agent w/ Archivecan invoke three studying procedures: the open\-ended procedure used byMeta\-Agent w/o Archive,PREPING, andCorpus2Skill\. It can inspect their outputs and combine the resulting artifacts for the solver\. Table[6](https://arxiv.org/html/2609.10824#A2.T6)reports how often artifacts from each procedure appeared in the final artifact set\.

Table 6:Sources of artifacts included byMeta\-Agent w/ Archive\. Each cell reports the number of final artifact sets containing output from that procedure\. Multiple sources may appear in one artifact set\. Iterations are pooled\. Apex Agents is additionally pooled across worlds\.†\\daggerCorpus2Skillwas unavailable in the first Harvey LAB iteration and was included in both iterations in which it was available\.##### Results\.

Artifact composition varies across environments\. Open\-ended study andPREPINGappear in every BCP\-G, OfficeQA, and DABStep artifact set, whereasCorpus2Skillappears in none\. In contrast,Corpus2Skillappears in every AppWorld artifact set, both available Harvey LAB sets, and 86 of 93 Apex Agents sets\. However, the same procedures were generally invoked across environments, and an output may be absent because its procedure failed or becauseMeta\-Agent w/ Archivedid not retain it\. The table therefore describes final artifact composition rather than invocation decisions\. Composition is also imperfect:Meta\-Agent w/ ArchiveomitsCorpus2Skillon BCP\-G even though it is the strongest standalone method there\.

### B\.3Taxonomies ofMeta\-AgentStudy Behavior and Artifacts

We analyze theMeta\-Agentvariants in two ways: the behaviors visible in their study traces and the content of their final artifacts\.

##### Study behavior taxonomy\.

We define nine multi\-label categories for actions visible in a study trace:

- •Environment inventory before content\.Enumerating the environment’s whole surface \(file tree, corpus size, app list\) with the agent’s own listing or counting calls before the first content\-bearing read\.
- •Interface probing\.Issuing calls whose purpose is to learn the interface itself, including help probes, format tests, trying a tool to see whether it exists, or sweeping logins to confirm an authentication pattern, and recording the lesson\.
- •Documentation verification\.Explicitly testing a claim made by shipped documentation or metadata against observed data and recording the contradiction or confirmation\.
- •Structured sampling or hypothesis testing\.Sampling by named strata \(per decade, ID range, app, or document type\), or stating a hypothesis about the environment’s organization and running a call to confirm or refute it, including bisecting a discovered boundary\.
- •Recording unavailable resources\.Recording something as absent, empty, broken, or impossible so that the downstream agent does not spend time attempting to use it\.
- •Budget\-conscious extraction\.Making an explicit economizing decision: using cheap shell extraction before LLM reading, storing pointers instead of copied content, avoiding duplication of information already available to the solver, or timing operations to assess their cost\.
- •Parallel sub\-agent delegation\.Delegating exploration to parallel sub\-agents with an explicit work partition and a specified output format or budget rule\.
- •Artifact verification\.A dedicated pass checking claims destined for or already in the artifact against the environment: auditing cited paths, recomputing a recorded value, testing the artifact with mock questions, or a corrective edit after a re\-check\.
- •Test anticipation and rehearsal\.Predicting what the test will ask from a concrete task\-bearing surface \(inboxes, chat threads, an active sample task\) and acting on the inference, including rehearsing a realistic task end\-to\-end\.

##### Trace labeling\.

Claude Sonnet 4\.6 labels every analyzable trace at temperature 0 against the fixed category definitions\. “w/o” denotes directMeta\-Agent w/o Archivestudy, while “w/” denotes the open\-ended study procedure invoked byMeta\-Agent w/ Archive\. Each positive label requires a verbatim evidence quote, which we mechanically verify appears in the trace\.

Table 7:Study behaviors observed in available trajectories\. Each cell reports trajectories exhibiting the behavior out of all analyzable trajectories\. AppWorld inventory is not applicable because its tool schemas are supplied rather than discovered through trace actions\.
##### Behavior findings\.

Environment inventory, recording unavailable resources, and budget\-conscious extraction appear in nearly all applicable traces\. Test anticipation is concentrated in Apex Agents and AppWorld, where histories and examples provide clues about likely downstream tasks\. Parallel sub\-agent delegation appears only in Harvey LAB and Apex Agents\. TheMeta\-Agentvariants therefore share a common diagnostic core while adapting some exploration strategies to the environment\.

##### Artifact taxonomy\.

We define twelve multi\-label categories for content appearing anywhere in a preparation’s final artifact set:

- •Navigation architecture\.Routing indexes, need\-to\-path or metric\-to\-source tables, file or app maps, or escalation recipes into deeper artifact tiers\.
- •Method recipes\.Concrete commands, code snippets, or step\-by\-step procedures for recurring task shapes\.
- •Environment constants\.Verified formats, units, formulas, schema semantics, corpus counts, or identifier conventions, as distinct from task\-answer values\.
- •Located values\.Specific data values, computed results, verdicts, or recommendations recorded with their location or provenance\.
- •Test anticipation\.Explicit modeling of likely test asks, including predicted prompts and pre\-answered probes\. Generic descriptions of the task genre do not count\.
- •Trap and contradiction documentation\.Seeded errors, dirty data, conflicting versions or figures, version\-supersession doctrine, or adjudication guidance\.
- •Verification doctrine\.Verify\-live rules, anti\-fabrication warnings, or re\-derive\-before\-quoting requirements\.
- •Format and grading guidance\.Output\-format laws, tolerance or precision semantics, or grading mechanics translated into solver behavior\. In\-corpus scoring worksheets do not count\.
- •Negative maps and budget economics\.What is absent, empty, broken, or not worth reading, and measured cost models of operations\.
- •Environment corrections\.Overriding the environment’s own documentation or naive expectations, including data repair and tool\-capability matrices\.
- •Identity and social decoding\.Persona orientation, role or alias maps, or name\-collision warnings\.
- •Honesty disclosures\.Staleness, coverage, or truncation caveats about the artifact itself, declared unknowns, or provenance labels\.

##### Artifact labeling\.

An LLM coder \(claude\-fable\-5\) labels each of the 216 artifact sets against the fixed category definitions\. Corpus dumps insideCorpus2Skilloutputs are excluded, and compiled skill trees are skimmed as machine\-generated output\. Each positive label requires a verbatim evidence quote\. All 2,199 positive labels passed a mechanical check that the quote appears in the cited file\. A second, independent LLM coder re\-labeled a stratified sample of 24 artifact sets, covering 288 category decisions\. Agreement was 95\.5% \(275 of 288\)\. Two disagreements affecting small\-sample cells were adjudicated against the codebook\. We treat the resulting counts as descriptive\.

Table 8:Artifact content produced by theMeta\-Agentvariants\. Each cell reports artifact sets exhibiting the category out of all artifact sets for that variant\.Meta\-Agent w/o ArchiveandMeta\-Agent w/ Archivedenote the variants without and with the workflow archive, respectively\. Iterations are pooled, and Apex Agents is additionally pooled across worlds\.
##### Artifact findings\.

Navigation structures, method recipes, and environment constants appear in nearly every artifact set from both variants\. Other content reflects the environment: identity and social decoding is concentrated in Harvey LAB and Apex Agents, while located values are absent from AppWorld, whose state is randomized across tasks\. Archive\-assisted artifacts also contain more operating guidance on Apex Agents\. Verification guidance increases from 34 of 93 unaided artifacts to all 93 archive\-assisted artifacts, format and grading guidance from 9 to 58, and honesty disclosures from 59 to 91\. These descriptive patterns do not establish which artifact properties caused changes in downstream reward\.

### B\.4Qualitative Artifact Examples

These excerpts illustrate two forms of environment\-specific information captured during study: corrections to domain rules and consolidation of context scattered across the environment\. Both were produced byMeta\-Agent w/ Archivein iteration one\. Typography is normalized, and bracketed ellipses mark omitted text\.

##### DABStep: correcting documentation and recovering computation rules\.

The artifact corrects the supplied documentation and records schema and fee\-computation rules that cannot be inferred safely from field names alone\. The excerpt comes from thepayment\-dataenvironment group at/meta\_archive/artifacts/cheatsheet/study/README\.md\.

> payments\-readme\.md lists card\_scheme as \[MasterCard, Visa, Amex, Other\]\. WRONG\. Real schemes = NexPay, GlobalCard, SwiftCharge, TransactPlus\. \[…\] fee = fixed\_amount \+ rate \* transaction\_value / 10000\. A rule applies to a txn only if EVERY field matches\. Rule field null/empty\-list = wildcard \(matches all\)\. monthly\_volume and monthly\_fraud\_level are not stored\. They must be COMPUTED per merchant per calendar month from payments, then matched to the rule’s range string\.

##### Apex Agents: reconstructing identities and decision rationale\.

The artifact links inconsistent names across communication channels and preserves the rationale behind a project decision\. The excerpt comes frominvestment\-banking\-world\-221at/meta\_archive/artifacts/cheatsheet/study/02\_people\_and\_narrative\.md\.

> Ola / ‘‘Tarek’’ Amethyst: Deal structuring, M&A & pro forma analysis \(chat msg 71 addresses ‘‘Tarek’’ for M&A slides = same person as ‘‘Ola’’\)\. Huajian / Luca Amethyst: Pitch deck\. ‘‘Luca’’ does the slide work in chat\. Names drift: Ola≈\\approxTarek, and Huajian≈\\approxLuca\. The email From names and chat names don’t perfectly align\. Treat by ROLE, not name\. \[…\] Final target = TPVG \[…\] despite Luca flagging TPVG is venture/growth\-focused, outside BBDC’s senior\-secured profile\. Justification: diversification \+ tech/venture exposure\. No DCF for BBDC\. Valuation = Trading Comps only\. Merger model = accretion/dilution\.

The examples show how study artifacts can make both domain rules and dispersed contextual information directly available to the solver\.

## Appendix CTest\-Time Interpretability

### C\.1Observed Artifact Use

Determining whether an artifact influenced a rollout is inherently difficult\. Artifacts may be accessed through explicit file reads or supplied through prompts, and a solver may act on injected guidance without visibly retrieving it\. We therefore measure*observed artifact use*, not causal influence\.

##### Trace labeling\.

We usegpt\-5\.6\-lunato classify task\-time traces\. A trace counts as observed use when the solver deliberately retrieves artifact content or clearly acts on distinctive guidance from it\. Mere exposure, directory listings, failed retrievals, and generic behavior do not count\. The judge receives the trajectory and mechanical annotations of artifact\-related actions, but not its reward, correspondingNo Studytraces, or verifier output\. Figures[8](https://arxiv.org/html/2609.10824#A3.F8)and[9](https://arxiv.org/html/2609.10824#A3.F9)provide the complete rubric\.

Because artifact delivery differs across methods, “no observed use” does not establish that an artifact had no influence, and use rates should not be compared across methods\.

##### Within\-task comparison\.

For each task, method, and artifact iteration represented by both observed\-use and no\-observed\-use trials, we compute the difference in mean reward between the two groups and average these differences across tasks\. We omit cells with fewer than 10 eligible task–iterations\. This controls for differences between tasks but remains observational because artifact use may follow events within a rollout\.

Table 9:Within\-task reward difference between observed\-use and no\-observed\-use trials\. Dashes indicate fewer than 10 contributing task–iterations\. Full coverage counts will be included in the released analysis tables\.Observed use is not uniformly associated with higher reward\. Among the within\-task contrasts, the clearest positive associations occur on DABStep forPREPINGand bothMeta\-Agentvariants\. The BCP\-G contrasts forPREPINGandMeta\-Agent w/ Archiveare negative\. These patterns are descriptive and do not isolate the causal effect of artifact use\.

```
You are auditing whether a test-time solver agent USED a study artifact that was placed in its
environment. You will see the solver’s trace and mechanical annotations. You are NOT judging
whether the artifact helped -- only whether the agent used it.

Output exactly one JSON object:
{"used": true|false,
 "channel": "<how the artifact was engaged: e.g. ’read /cheatsheet files’, ’compiled
             get_document’, ’playbook citation’, or ’none’>",
 "evidence": ["2-5 bullets citing step ids and short quotes"],
 "confidence": 0.0-1.0}

USED (true) means at least one of:
- The agent, by its own tool call, brought artifact CONTENT into its context: a
  Read/cat/grep/head of a file under an artifact mount, a Skill load, or a SUCCESSFUL
  get_document call -- and the retrieved content is more than a bare directory listing.
- The agent’s OWN authored text (message, reasoning, or tool arguments -- never user-role text
  or tool observations) engages injected artifact content: it cites a playbook ID (e.g.
  [strategies-NNN]/[pitfalls-NNN]), explicitly invokes the artifact ("According to the
  playbook, I should ..." / "Based on the playbook, I expect ...") and then acts on the stated
  plan -- this counts even when the plan’s individual tactics are generic, because the agent is
  visibly steering by the artifact -- or reproduces a distinctive artifact-only fact, rule, or
  ORDERED multi-step recipe at a point where it shapes what the agent does next. Compare the
  agent’s behavior against the injected playbook text you can see in the first user message: a
  specific fact or rule stated in the playbook and applied by the agent counts even without a
  citation. A single generic tactic (grep, read-the-docs) that any competent agent might use
  does not.

NOT used (false):
- Exposure alone: injected playbook/prefix text sitting in the prompt, or artifact text echoed
  back in observations, with no agent-authored engagement.
- A coerced or trivial touch: an ‘ls‘ of the mount, a mandated Skill call whose output the
  agent never draws on, or get_document calls that ALL errored.
- Generic diligence that matches artifact advice only thematically, with no citation and no
  artifact-only fact (model-default behavior).

Hard rules:
1. Only agent-authored text counts for citations/provenance. Artifact text arrives in user-role
   messages and large read observations -- matches there are exposure, not use.
2. Echo traps: /logs/agent dumps, tool-result re-reads, and embedded subagent transcripts do
   not count as the agent engaging the artifact.
3. Reading an artifact file and then visibly ignoring it still counts as used (content entered
   context by the agent’s own action). Merely listing filenames does not.
4. When genuinely uncertain, set used=false with low confidence rather than guessing true.
```

Figure 8:Artifact\-use judge rubric \(part one of two\): core criteria\. Each prompt also includes the relevant method gate and benchmark note from Figure[9](https://arxiv.org/html/2609.10824#A3.F9), mechanical annotations, and the rendered trace\. Method gates take precedence over the core criteria\.```
METHOD c2s: the corpus is compiled into navigable skills (Skill tool, INDEX/SKILL.md files, MCP
corpus-documents get_document, mounted under /opt/corpus2skill or similar). On most benchmarks
a system prompt MANDATES >=2 Skill calls and >=1 get_document. A mandated call still counts as
used ONLY if artifact content actually entered context (successful get_document, or index/skill
text visibly consulted beyond the coerced minimum). All-error get_document with no
skill-content engagement is not use.

METHOD cheatsheet: small markdown files mounted read-only (at /cheatsheet or /context). The
task preamble tells the agent it may inspect them. Used = the agent Read/cat/grep’d cheatsheet
file CONTENT (tool args show the path). A bare ‘ls‘ of the mount, or the mount merely being
named in prompts, is not use.

METHOD preping: a playbook is pasted into the FIRST USER MESSAGE (PLAYBOOK_BEGIN/END, bullets
tagged [strategies-NNN]/[pitfalls-NNN]/[apis-NNN]/[code_snippets-NNN]). Exposure is 100% by
construction. Only agent-authored engagement counts. Used = a playbook-ID citation in agent
text, OR agent text/code reproducing a distinctive playbook-only fact or recipe (a specific
file/table/endpoint the playbook names, a code snippet’s distinctive shape) at a point that
shapes its next action. Generic strategies the model would use anyway are not use.

METHOD meta-agent-v1: a study bundle is delivered two ways at once -- (a) an orientation prefix
injected into the SYSTEM PROMPT (not visible in the trace, and it may embed a playbook with
[strategies-NNN]-style tags and names artifact file locations), and (b) artifact files mounted
under /meta_archive/... (e.g. /meta_archive/artifacts/cheatsheet/study/,
/meta_archive/artifacts/preping/playbook/). Used = the agent reads /meta_archive file content,
OR its own text cites a playbook ID or reproduces a distinctive bundle-only fact or recipe
(e.g. a corpus-era/table-format rule, a named source file, a prescribed search recipe) at a
decision point. Because the prefix is invisible here, weigh the mechanical annotations and
distinctive-content evidence carefully. Generic competence is not use.

Benchmark appworld: the agent drives simulated apps via mcp__appworld__* tools. c2s here has no
system mandate. Any skill/get_document access was a free choice. Cheatsheet files live under
/cheatsheet (e.g. 00_START_HERE.md, per-app API notes).

Benchmark apex: finance/law/consulting tasks over /filesystem and /.apps_data. c2s get_document
is often broken (’not connected’/’No such tool’). All-error access is not use. The cheatsheet
is a per-world markdown pack. The preping playbook often names exact workbook paths, tab
indices, cell ranges, or figures. An agent going straight to a playbook-named path/tab or
quoting playbook-supplied figures is using the playbook even without a citation.

Benchmark bcp-grep (BrowseComp over plaintext /workspace/corpus with native Grep/Glob/Read).
The cheatsheet README carries search mechanics (rg flags, triage recipes). c2s replaces search
with skills+get_document, so c2s corpus access is itself artifact use when content is
retrieved. The preping playbook prescribes an ORDERED candidate-list intersection recipe
(corpus-wide grep -rl for one clue, intersect with a second clue via xargs grep -l, then read
the intersection) -- an agent executing that ordered recipe is using the playbook even without
a citation. A lone grep is not.

Benchmark dabstep: data analysis over /app/data (payments.csv + docs). c2s compiles the same
files the agent can read directly. Only reads of the COMPILED copies or successful get_document
count as artifact use, reads of /app/data originals do not. Known distinctive playbook-only
rule: treating an EMPTY LIST in fees.json fields (account_type, aci) as a wildcard applying to
all values -- the corpus manual only covers null. An agent stating or applying
empty-list-as-wildcard is using the playbook.

Benchmark harvey_lab: legal research over a 9,288-doc DMS at /dms (labread CLI for binary
docs). Cheatsheet = 00-START-HERE.md + matter-index.md under the artifact mount. Reads of /dms
originals are corpus access, not artifact use.

Benchmark officeqa: numeric answers from 697 Treasury Bulletin text files at /app/corpus.
Artifact mounts hold corpus maps and table-parsing recipes. Reads of /app/corpus originals are
corpus access, not artifact use.
```

Figure 9:Artifact\-use judge rubric \(part two of two\): method gates and benchmark notes\. Each prompt includes one of each\. TheCorpus2Skillgate excludes mandated calls that do not bring artifact content into context\.

### C\.2Lift by Task Stratum

We next ask which kinds of tasks benefit from studying\. We use benchmark\-provided difficulty labels for OfficeQA\[[22](https://arxiv.org/html/2609.10824#bib.bib41)\]and DABStep\[[8](https://arxiv.org/html/2609.10824#bib.bib38)\], difficulty and application\-count labels for AppWorld\[[30](https://arxiv.org/html/2609.10824#bib.bib36)\], domain labels for Apex Agents\[[32](https://arxiv.org/html/2609.10824#bib.bib40)\], and topic and gold\-document counts for BCP\-G\[[3](https://arxiv.org/html/2609.10824#bib.bib37)\]\. Exact metadata sources and mappings will accompany the released analysis\.

For each task stratumSS, we report

1\|S\|​∑i∈S\(ri−r¯t⁡\(i\)NS\),\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}\\left\(r\_\{i\}\-\\bar\{r\}^\{\\mathrm\{NS\}\}\_\{t\(i\)\}\\right\),\(4\)wherer¯t⁡\(i\)NS\\bar\{r\}^\{\\mathrm\{NS\}\}\_\{t\(i\)\}is the mean reward from a fixed three\-run subset ofNo Studyevaluations for taskt⁡\(i\)t\(i\)\. We use this subset for computational reasons\. The primary results use all tenNo Studyruns\. Artifact iterations are pooled\. This analysis is exploratory, and the comparisons are not adjusted for the number of strata examined\.

Table 10:Task\-matched reward lift overNo Studyby dataset difficulty label\. The first row gives theNo Studyreward level of each stratum\. Full counts and intervals are provided in the released analysis\. AppWorld uses its three\-level difficulty scale\.##### Difficulty\.

Difficulty does not produce a universal pattern\. On DABStep, the positive estimates forPREPINGand bothMeta\-Agentvariants are substantially larger on hard tasks than on easy ones\.Meta\-Agent w/ Archiveimproves reward by 0\.185 on hard tasks but only 0\.015 on easy tasks, whereNo Studyalready scores 0\.792\. OfficeQA shows little separation between difficulty levels, and AppWorld’s gains are positive but non\-monotonic for theMeta\-Agentvariants\.

Table 11:Task\-matched reward lift overNo Studyon Apex Agents by task domain\. The first row gives theNo Studyreward level of each domain\. Full counts and intervals are provided in the released analysis\.
##### Apex Agents gains concentrate in Law\.

All four methods have their largest domain\-level lift on Law \(\+0\.089 to \+0\.139\)\. Estimates are near zero in Investment Banking \(−\-0\.004 to \+0\.009\) and mixed in Management Consulting \(−\-0\.017 to \+0\.035\)\.

##### BCP\-G retrieval strata\.

Corpus2Skillhas positive lift across all ten topic labels \(Table[14](https://arxiv.org/html/2609.10824#A3.T14)\), indicating that its overall gain is not driven by a single topic\. Its lift also rises from \+0\.224 with one gold document to \+0\.293 with four or more \(Table[13](https://arxiv.org/html/2609.10824#A3.T13)\)\. This pattern is consistent with corpus compilation being more useful when tasks require locating more supporting documents, although the analysis does not isolate retrieval burden from other task differences\.

##### AppWorld application count\.

Every studying method has its largest estimated lift on tasks involving two applications \(Table[12](https://arxiv.org/html/2609.10824#A3.T12)\), with reward gains from \+0\.108 to \+0\.141\.

Together, these results do not reveal a universal notion of task difficulty\. Studying helps when the information captured during study matches the structure required by downstream tasks\.

Table 12:Task\-matched reward lift overNo Studyon AppWorld by the number of applications required\. The first row gives theNo Studyreward level of each stratum\. Full counts and intervals are provided in the released analysis\.Table 13:Task\-matched reward lift overNo Studyon BCP\-G by the number of gold documents\. The first row gives theNo Studyreward level of each stratum\. Full counts and intervals are provided in the released analysis\.Table 14:Task\-matched reward lift overNo Studyon BCP\-G by BrowseComp topic label\. The first row gives theNo Studyreward level of each topic\. Full counts and intervals are provided in the released analysis\.

### C\.3Helpful and Harmful Artifact Use

Two matched examples illustrate how artifacts can help or hinder the solver\. Each compares aMeta\-Agent w/ Archiverollout with aNo Studyrollout on the same task and downstream model\. The examples are not paired by random seed and should not be interpreted causally\.

##### DABStep: recovering the correct fraud boundary\.

For task1701, the artifact records the dataset’s fraud\-level buckets:

> monthly\_fraud\_level buckets: <7\.2%, 7\.2%\-\-7\.7%, 7\.7%\-\-8\.3%, \>8\.3%\. = fraud volume / total volume that month\.

The archive\-assisted solver reads this guidance and applies the correct upper bucket:

> Read /meta\_archive/artifacts/cheatsheet/study/fee\_matching\.md April 2023 fraud rate: 9\.92% Fraud level bucket: \>8\.3%

The matchedNo Studysolver computes the same fraud rate but invents a different boundary:

> elif fraud\_rate <= 9: fraud\_bracket = ‘8%\-\-9%’ else: fraud\_bracket = ‘\>9%’ monthly\_fraud\_level = ‘\>9%’

The incorrect bucket propagates into the final Fee\-ID set:No Studyfails all 10 evaluations on this task, while the selected artifact succeeds in all three of its evaluations\.

##### Apex Agents: an artifact creates a search tunnel\.

For taskword\-129\-pj\-01, the artifact directs the solver toward general ServiceNow pricing documents:

> These documents collectively cover the competitive pricing, packaging, and market positioning landscape for workflow automation platforms … \[including\] ServiceNow\. Competitions\_Price\_Benchmarking\.xlsx, … ServiceNow ITOM HRSD SecOps modules\.

The solver follows this route and substitutes unsupported assumptions for the task inputs:

> The skills that seem most relevant are: skill\-02\-workflow\-automation\-pricing … Assumptions for calculation: … Total ServiceNow customers: 1,000 … Average users per customer: 150 … Module adoption varies by customer segment\. ServiceNow Total Annual Revenue Forecast: $146\.2m

The artifact does not index the two root\-level images containing the required subscriber counts and price multipliers\. The matchedNo Studysolver discovers those files through a directory listing and obtains the correct total of $1,438\.2 million\.No Studysucceeds in 6 of 10 evaluations on this task, whereasMeta\-Agent w/ Archivefails in all nine evaluations across the three artifact iterations\. This example shows how a plausible but incomplete artifact can narrow the solver’s search and hide relevant evidence\.