WANDR: A Benchmark for Wide and Deep Research

# WANDR: A Benchmark for Wide and Deep Research
Source: [https://arxiv.org/html/2608.14747](https://arxiv.org/html/2608.14747)
###### Abstract

WANDR \(Wide ANd Deep Research\) is a benchmark of 500 realistic and challenging agentic data\-collection tasks for knowledge work\. Each task asks a system to discover a large collection of members satisfying specified criteria \(the*wide*axis\) and establish specific facts about each member through orchestrated web searches \(the*deep*axis\)\. We represent a task as a qualification key hierarchy such ascompany \(nn\)→\\rightarrowemployee \(mm\)→\\rightarrowurl \(kk\), meaning “findnnqualifying companies,mmqualifying employees per company, andkkqualifying source pages per employee\.” This hierarchy defines a target ofn×m×kn\\times m\\times k*records*\. Instead of comparing submissions against a static gold answer set, a task\-specific judge re\-fetches the cited pages and verifies each record’s claims against the cited evidence, allowing the benchmark to cover current and changing facts\. Record\-level verdicts are aggregated into soft and hard precision, recall, and F1 at the task level\. Required record counts range from dozens to thousands, shifting the central challenge from finding a single correct answer or writing a polished report to open\-ended discovery at high volume without sacrificing per\-record factual correctness\. Tasks are derived from de\-identified product\-usage logs and produced by a semi\-automated pipeline\. Automated checks and empirical audits screen them, with human review where needed; the final set is curated to stress\-test recurring failure modes of frontier agentic search systems\. The benchmark remains far from saturated: at a high effort setting, the strongest system achieves only 0\.363 soft F1 and 0\.133 hard F1\. The tasks and evaluation harness are available at[https://github\.com/perplexityai/wandr](https://github.com/perplexityai/wandr)\.

## 1Introduction

Research agents are increasingly asked to produce not only a single answer or free\-form report, but also a structured collection of facts, each requiring in\-depth search to establish\. This task shape recurs across the professional knowledge work observed in our production data, including competitive\-landscape mapping, deal due diligence, literature review, market and product comparisons, and talent sourcing\. It has two orthogonal dimensions: wide and deep\. On the wide axis, the agent usually must discover a large set of entities not supplied in advance; some tasks instead specify a closed roster and require complete coverage\. On the deep axis, it must enrich each entity through multiple orchestrated web searches\. A representative request asks the agent to find at least 70 CEO or CFO appointments at US\-based companies, each first announced during March–April 2026, and provide an authoritative appointment source for every company–appointee pair\. A wide\-and\-deep research system returns such a structured collection, with one independently verifiable record per claim\. WANDR tasks require dozens to thousands of such records\. The central challenge is to achieve the requested volume through broad search while keeping every record factually correct\.

To capture the variety of these use cases while keeping every claim independently verifiable, we represent each WANDR task as a flexible*qualification key hierarchy*: a tree of identifying keys and filtering criteria\. The running example,ceo\_cfo\_appointments, has a primary hierarchycompany\(70\)→\\rightarrowcompany\_appointee\(1\)→\\rightarrowurl\(1\)\. More generally, each level may require several children—nncompanies,mmappointees per company, andkksources per appointee—so the hierarchy defines a target ofn×m×kn\\times m\\times krecords\. Each record, the atomic unit of evaluation, cites a live page and includes verbatim excerpts, making it independently checkable\.

At this scale, current agentic search systems often fail in predictable ways\. We observe several recurring modes in production agent runs, some of which are also documented in prior literature:*volume collapse*\(incomplete query decomposition and insufficient retrieval at scale, sometimes followed by hallucinated values\)\([9](https://arxiv.org/html/2608.14747#bib.bib13);[16](https://arxiv.org/html/2608.14747#bib.bib23)\);*snippet reliance*\(using a search\-result snippet without validating that its source supports the submitted claim\)\([16](https://arxiv.org/html/2608.14747#bib.bib23)\);*non\-systematic constraint application*\(applying a filter to the first few entities but silently dropping it thereafter\);*cross\-reference skips*\(asserting a claim without corroborating it across the required sources\);*premature stopping*\(giving up after an initial search yields insufficient information\)\([16](https://arxiv.org/html/2608.14747#bib.bib23)\);*missing enrichment*\(finding an entity but never gathering its required facts\); and*context overflow*\(long search trajectories exhausting the available context\)\([9](https://arxiv.org/html/2608.14747#bib.bib13)\)\. WANDR originated as an internal benchmark designed to distinguish systems by how well they avoid these failure modes\. In particular, it compares a*Search as Code*\(SaC\) system, in which agents compose programmable retrieval primitives\([13](https://arxiv.org/html/2608.14747#bib.bib21)\), with conventional systems that distribute fixed\-endpoint search\-and\-read loops across parallel subagents\. SaC, the strongest system evaluated, achieves only 0\.363 soft F1 and 0\.133 hard F1, making the benchmark challenging even for frontier research agents\.

The released task set instantiates recurring patterns from production data as self\-contained benchmark tasks\. Every admitted task must meet three top\-level criteria:*substantial volume*\(usually hundreds of records and sometimes thousands\);*feasibility*\(the requested volume is attainable\); and*difficulty and discrimination*\(simple internal baselines score low while stronger systems score meaningfully higher\)\. Together, these choices yield five defining characteristics whose combination distinguishes WANDR:

- •Task shapes and topics grounded in production usage\.The released tasks are derived from de\-identified requests observed in a real research product, preserving the domains, constraints, and wide\-then\-deep structures of professional knowledge work \(Section[4](https://arxiv.org/html/2608.14747#S4)\)\.
- •A compositional tree structure that flexibly encodes task shapes and makes failures localizable\.A flat list and a fixed\-schema table are both special cases of the qualification key hierarchy\. Failures can be localized across the tree: a system may fail to discover core task entities or members, enrich intermediate entities or keys, identify relevant pages, extract adequate evidence from those pages, or disambiguate identities when keys collide \(Section[3](https://arxiv.org/html/2608.14747#S3)\)\.
- •A semi\-automated task\-construction pipeline\.Automated authoring checks, an empirical feasibility audit, and a judge audit establish the conditions needed for a functional task\. Optional human review improves task quality but is not required for the task package to run and grade submissions\. With no gold answer set to annotate, human effort is quality review rather than exhaustive answer construction \(Section[4](https://arxiv.org/html/2608.14747#S4)\)\.
- •Reference\-free, evidence\-verified grading\.Every record is a citation\-backed claim \(URL plus verbatim excerpts\)\. A task\-specific judge re\-fetches the cited page\. The full verdict evaluates both the page and submitted excerpts, while the retrieval\-only verdict asks only whether the page satisfies every substantive task requirement\. These verdicts are then aggregated into precision \(accuracy among submitted members\), recall \(quality\-adjusted completion relative to the required member count\), and F1, each in soft \(partial credit\) and hard \(binary pass/fail\) forms \(Section[5](https://arxiv.org/html/2608.14747#S5), Appendix[E](https://arxiv.org/html/2608.14747#Ax1.SS5)\)\.
- •A scalable substrate for reinforcement learning \(RL\)\.The task pipeline can generate diverse training packages; per\-record verdicts and hierarchical metrics provide dense partial\-progress rewards; per\-level required counts provide an explicit difficulty gradient for curriculum learning; and the streaming, cache\-aware grader can amortize overlapping work across training batches \(Appendix[N](https://arxiv.org/html/2608.14747#Ax1.SS14)\)\.

The rest of the paper is organized as follows\. Section[2](https://arxiv.org/html/2608.14747#S2)situates WANDR among single\-answer, collection, and domain\-specialized benchmark families and compares how they establish ground truth\. Section[3](https://arxiv.org/html/2608.14747#S3)defines the task structure and its variations; Section[4](https://arxiv.org/html/2608.14747#S4)describes the task\-construction pipeline and reports summary statistics for the released set; Section[5](https://arxiv.org/html/2608.14747#S5)details how submissions are graded; Section[6](https://arxiv.org/html/2608.14747#S6)evaluates six production systems; and Section[7](https://arxiv.org/html/2608.14747#S7)summarizes the findings and discusses their implications, limitations, and future directions\.

## 2Related Work

We organize prior work on agentic information\-seeking benchmarks primarily by output shape—a compact answer or a collection of items—and by how that output is graded\. Domain specialization is a cross\-cutting distinction: both compact\-answer and collection benchmarks may restrict their tasks to one expert field\.

##### Closed\-ended browsing and question answering

The first family asks an agent to solve a bounded browsing or question\-answering problem and return a compact response\. BrowseComp poses deliberately obscure single\-answer questions\([15](https://arxiv.org/html/2608.14747#bib.bib5)\); Humanity’s Last Exam targets expert\-level, closed\-ended academic questions\([14](https://arxiv.org/html/2608.14747#bib.bib2)\); FRAMES and GAIA combine retrieval with multi\-hop reasoning\([7](https://arxiv.org/html/2608.14747#bib.bib3);[12](https://arxiv.org/html/2608.14747#bib.bib1)\); AssistantBench includes realistic web\-research requests but permits at most five answers\([18](https://arxiv.org/html/2608.14747#bib.bib4)\); and VeriWeb decomposes long\-horizon web tasks into chains of individually verifiable subtasks\([11](https://arxiv.org/html/2608.14747#bib.bib17)\)\. These benchmarks stress depth or end\-to\-end task completion, but their compact, gold\-answer\-based outputs do not test repeated depth across a large collection of discovered entities\. WANDR instead makes that width–depth composition the scored object: every member must be discovered, enriched, and independently evidenced\.

##### Wide and broad information seeking

Closest to WANDR are benchmarks whose deliverable is a collection of items graded individually\. WideSearch\([16](https://arxiv.org/html/2608.14747#bib.bib23)\)also sources tasks from real user queries and grades populated tables cell by cell, but primarily tests broad collection with comparatively shallow per\-item fields\. DeepSearchQA\([5](https://arxiv.org/html/2608.14747#bib.bib12)\)includes both single\- and set\-valued answers, graded against gold answer sets\. WideSeekBench\([6](https://arxiv.org/html/2608.14747#bib.bib15)\)uses a multi\-stage generation and quality\-control pipeline to produce table\-completion tasks with varied target volumes and logical constraints\. GISA\([22](https://arxiv.org/html/2608.14747#bib.bib16)\)spans item\-, set\-, list\-, and table\-valued answers and maintains a live subset through periodically updated answers\. These benchmarks expand the width axis, but generally do not require the same multi\-stage enrichment and corroboration process to be repeated for every discovered member\. Most similar to WANDR, DeepWideSearch\([9](https://arxiv.org/html/2608.14747#bib.bib13)\)explicitly combines depth and width, converting 220 tasks from existing deep\- and wide\-search datasets and grading against human\-verified ground\-truth tables\. WANDR differs in task shape and grading: its requests require wide\-then\-deep search to be repeated within one hierarchy, rather than combining separately sourced deep and wide task patterns; its grading verifies claims against submitted sources rather than relying on a gold solution\. The related*Table\-as\-Search*framework\([8](https://arxiv.org/html/2608.14747#bib.bib18)\)casts wide\-and\-deep search as table completion, though as a solving method rather than a benchmark\.

##### Domain\-specialized information seeking

Orthogonal to output format, a parallel line of work restricts search and research benchmarks to a single expert domain, testing domain knowledge alongside retrieval: medicine\([3](https://arxiv.org/html/2608.14747#bib.bib7)\), finance\([21](https://arxiv.org/html/2608.14747#bib.bib8);[2](https://arxiv.org/html/2608.14747#bib.bib11)\), law\([10](https://arxiv.org/html/2608.14747#bib.bib9)\), and academic literature\([20](https://arxiv.org/html/2608.14747#bib.bib10)\)\. AutoResearchBench is also literature\-specific: its*Deep Research*track identifies a target paper, while its*Wide Research*track collects all papers satisfying given conditions\([17](https://arxiv.org/html/2608.14747#bib.bib14)\)\. These benchmarks raise the expertise bar and often constrain the sources or tools used within one field\. WANDR is instead domain\-general: it applies a common hierarchical representation and grading framework to varied task structures across talent, finance, health, legal and regulatory work, and other professional domains\.

##### Grading paradigms

Most prior benchmarks establish ground truth in one of three ways: matching gold labels for single\-answer question answering; applying task\-specific rubric criteria with a large language model \(LLM\) judge to long\-form reports, as in our deep\-research benchmark DRACO\([19](https://arxiv.org/html/2608.14747#bib.bib22)\); or matching a gold collection in wide\-search benchmarks\. WANDR is the breadth\-oriented counterpart to DRACO\. Other work makes the evaluator itself agentic: Agent\-as\-a\-Judge\([23](https://arxiv.org/html/2608.14747#bib.bib19)\)evaluates code\-agent outputs and trajectories against hierarchical requirements, while Mind2Web 2\([4](https://arxiv.org/html/2608.14747#bib.bib6)\)assesses answer correctness and source attribution for citation\-backed responses\. WANDR brings source verification to collection scale: every record must carry its own citation \(URL plus verbatim excerpts\), and a task\-specific judge re\-fetches the cited page and verifies the claim against it\. Because the grader checks submitted claims rather than enumerating every eligible answer, this approach avoids exhaustive gold\-answer annotation\. Required\-volume targets, validated for feasibility during task construction, provide the denominator for recall without requiring an enumerated answer set\. This makes tasks about current and changing facts admissible and allows task production to scale through a semi\-automated pipeline rather than exhaustive expert annotation\. DeepWideSearch’s own limitations call for exactly this combination of “automated data generation techniques” and “reference\-free evaluation metrics”\([9](https://arxiv.org/html/2608.14747#bib.bib13)\)\.

Table[1](https://arxiv.org/html/2608.14747#S2.T1)summarizes WANDR’s position\. Relative to existing collection benchmarks, WANDR adds evidence\-backed records, reference\-free verification, and a compositional task structure: qualification key hierarchies in which discovery and per\-entity enrichment form flexible trees rather than a single flat table\. Its task shapes originate in real professional usage, and tasks are admitted only if they are high\-volume, feasible, and discriminative against internal baselines\. The resulting challenge is to sustain both breadth and per\-record accuracy across a large, structured collection—precisely the regime in which current agents struggle\.

Table 1:WANDR versus representative search and research benchmarks\. Rows are grouped by output format; domain specialization is orthogonal, so AutoResearchBench’s Wide track appears with collection benchmarks\.Real\-world tasks: task shapes derived from real user requests or professional workflows rather than hand\-curated puzzles or conversions of other benchmarks\.Open\-set discovery: the entities to report are not given and must be found at high recall\.Collection\-record grading: collection items, rows, or cells are a primary scoring unit, rather than a compact task\-level answer or holistic report\.At\-scale quality task generation: a scalable generation process couples task creation with automated quality controls rather than relying only on manual task\-by\-task authoring\.Reference\-free verification: grading checks each claim against its fetched source, with no pre\-annotated gold answer set to match\. For mixed\-format benchmarks, a check indicates that at least one named track or task family has the property\.

## 3Task Structure

### 3\.1Recurring Patterns in Production Requests

Wide\-and\-deep research requests vary widely in subject matter, but in the production data we observe a small number of recurring shapes\. In*entity discovery*, the system must find many instances of an entity class, such as companies, products, people, or events\. In*entity enrichment*, the entities are already known and the system must fill in the same facts for each one\. The most common shape combines the two: first discover the entities, then investigate each one in depth\. A fourth pattern is*multi\-condition research*, in which every candidate must satisfy several conditions, often supported by different pages or source types\.

These patterns also appear inside one another\. A request may ask for companies, several products per company, several facts per product, and one or more sources for every fact\. Other requests repeat a fixed set of facets for each entity, compare the same entities across time, or ask both sides of a relationship to document the connection\. The surface form changes, but the intent structure is the same: for each qualifying item, find a required set of qualifying children and carry the process through to source\-backed evidence\.

### 3\.2A Common Tree Representation

WANDR represents this repeated structure flexibly as a tree\. Each level names the kind of item being collected, and each edge means*for each parent, find these children*\. Every branch ends in one or more URLs, so each claim remains tied to the page that supports it\. This representation handles a wide range of real\-world requests without defining a new output format for every workflow\. The tree’s topology specifies which items to find, how they are related, and how many are required\. Concretely, it defines the keys, the parent–child edges, the required count at each level, and any subtasks whose top keys reuse keys in the parent hierarchy\.

### 3\.3Canonical Form

A basic WANDR hierarchy has the form

key1​\(r1\)→key2​\(r2\)→⋯→keyd​\(rd\)→url​\(ru\)\.\\texttt\{key\}\_\{1\}\(r\_\{1\}\)\\rightarrow\\texttt\{key\}\_\{2\}\(r\_\{2\}\)\\rightarrow\\cdots\\rightarrow\\texttt\{key\}\_\{d\}\(r\_\{d\}\)\\rightarrow\\texttt\{url\}\(r\_\{u\}\)\.A*key*is the field, or group of fields, that identifies an item at one level\. The value in parentheses is the minimum number of distinct children required at that level\. A count is therefore a coverage floor; a closed\-set level instead supplies the complete allowed roster\. The hierarchy always ends inurl\. The count is applied separately beneath each parent: the running example asks for one appointment under each of 70 companies, not 70 appointments distributed across fewer companies\. A key may be composite when one field is not globally unambiguous;company\_appointee\{company, appointee\}distinguishes appointments of the same person—or same\-named people—across different companies\.

A complete root\-to\-leaf path defines one record slot\{item, url, excerpts, answer\}:itemcontains the identifying key fields,urlsupplies the source page,excerptscontains verbatim passages selected from that page, andanswercontains flexible JSON content describing the claim\. A record is the atomic unit of grading\. For corpus\-level scale statistics, a*member*is the task’s designated core unit of coverage; in the running example, each company is one member and requires two leaf records, one in each branch\. Counts apply recursively\. A parent reaches its structural target when it has the required number of distinct children, and each child must in turn contain its own required descendants\.

### 3\.4Running Example

We use the sameceo\_cfo\_appointmentstask for illustration\.

ceo\_cfo\_appointmentsPrimary task:For at least 70 US\-based companies, identify at least one CEO or CFO appointment first announced between March 1 and April 30, 2026, and provide an authoritative appointment page\. Primary record:one company–appointee appointment backed by an authoritative announcement page\. Primary validity and requirements:the company is US\-based; the page identifies the company and appointee, establishes the CEO or CFO role and first public announcement within the target window, and comes from an authoritative issuer, filing, newswire, or directly attributed first\-hand business\-journalism surface\. Company\-listing subtask:For the same 70 or more companies, supply at least one URL on a recognized listing\-authority surface showing that the company has a primary or secondary listing on a US national securities exchange \(NYSE, NASDAQ, NYSE American, or NYSE Arca\) or is a US\-domiciled SEC Exchange Act reporting issuer\. Subtask record:the same company backed by a listing\-authority page\. Listing requirements: 1\. The page identifies the claimed company\. 2\. The page communicates recognized listing\-authority authorship, including through its URL when applicable\. 3\. The page shows the required US exchange listing or US\-domiciled Exchange Act reporting status\. Records required:140 \(70 appointment records plus 70 listing records\)\.

The primary hierarchy is

company\(70\)→company\_appointee\{company, appointee\}\(1\)→url\(1\)\.\\texttt\{company\(70\)\}\\rightarrow\\texttt\{company\\\_appointee\\lx@text@lbrace company, appointee\\lx@text@rbrace\(1\)\}\\rightarrow\\texttt\{url\(1\)\}\.A subtask,company\_listings, reuses the company key and adds one listing\-authority URL per company\. Its top key matches the primary root, so the two branches align at the company level and together require 70 appointment paths plus 70 listing paths\.

![Refer to caption](https://arxiv.org/html/2608.14747v1/ceo-cfo-appointments.png)Figure 1:Task viewer for a gradedceo\_cfo\_appointmentssubmission\. The primary hierarchy requires 70 company–appointee records, and thecompany\_listingsbranch attaches one listing\-authority URL to each shared company\. The expanded rows show both branches for 374Water and AI Era; the remaining companies are collapsed\.This one task demonstrates open\-set discovery, a composite key, recursive per\-parent counts, and a subtask\. The next subsection uses shorter examples to explain other structural variations\.

### 3\.5Structural Variations

The canonical hierarchy is intentionally small, but changing what a level represents, how many children it requires, or where a new branch attaches covers a broad range of research workflows\. Figure[2](https://arxiv.org/html/2608.14747#S3.F2)highlights six common patterns; the four groups below define these patterns and related variations\.

Figure 2:Six common hierarchy patterns\. Filled circles are entities, dashed circles are fixed labels, and squares are URL leaves\. In the matrix, repeated child labels denote the same child values under different parents\.##### Open versus closed entity sets

Most levels ask the system to discover a requested number of qualifying values\. A closed\-set level instead supplies the complete roster, changing “findnn” into “find all\.” The hierarchy and scoring rules remain the same, but canonicalization and deduplication map submitted names to the supplied entities and enforce complete coverage; Appendix[F](https://arxiv.org/html/2608.14747#Ax1.SS6)details these mechanisms\.

##### Organizing entities

- •*Flat list*: one entity level sits above the URL leaves, as incompany\(nn\)→\\rightarrowurl\(1\)\.
- •*Nesting*: levels follow a containment relationship, as in countries, cities within each country, and evidence for each city\.
- •*Fan\-out*: a parent connects to several related entities that are not contained within it\. Themisquotation\_instancestask asks for two attributed authors and five pages per quote–author pair; a startup may fan out to several investors\.
- •*Matrix*: the same kind of child is repeated across many parents, often with an encouraged or fixed shared set\. Examples include the same set of benchmarks for every model evaluated or the same set of products carried by every retailer\.

##### Controlling coverage

- •*Dispatch*: a level contains a fixed set of labels rather than discovered entities\. The labels can represent facets, such as product, market, and customer review for a company\. Requiring every label makes coverage explicit\.
- •*Anchor*: a count\-one intermediate key used to bind all downstream children to the same selected value and/or preserve the quota on its parent\. Incompany\(nn\)→\\rightarrowproduct\(1\)→\\rightarrowdistributor\(2\), every company contributes a product and both distributors refer to that product; omitting the product level loses the shared\-product guarantee, while flattening company and product loses per\-company coverage\.
- •*Partition*: a small top level divides the search space before the main entities are collected\.country\(3\)→\\rightarrowcompany\(mm\)enforces geographic spread that a flat company list would not\.

##### Structuring evidence

- •*Corroboration*: a URL count above one asks several distinct pages to support the same claim independently; each page must satisfy the full claim\. A task may separately require source or domain independence\.
- •*Triangulation*: several pages jointly describe a quantity or market surface without having to report the same value, such as three current retailer prices\.
- •*Data as keys*: an extracted value becomes its own level when values must be counted, kept distinct, or held fixed across later evidence\. Aprice\(1\)→\\rightarrowurl\(3\)branch requires all three pages to concern the same submitted price; asignal\(kk\)level requireskkdistinct signals\.

##### Adding subtasks

A subtask reuses a parent key but adds another hierarchy, with subtasks allowed to further recurse into subtasks of their own\. They usually serve one of four roles:

- •*Enrichment*: add facts that are useful but separate from the primary claim, such as pricing for each model or a release paper for each benchmark\.
- •*Another angle*: investigate a related question about the same entity, such as open roles alongside evidence of a recent company strategy change\.
- •*Another source family*: require evidence from a distinct source universe, as the running CEO/CFO task does by separating appointment evidence from listing\-authority evidence\.
- •*Chain*: attach a later branch to entities identified by an earlier one, such as approval→\\rightarrowactive molecule→\\rightarrowcredited scientist→\\rightarrowdissertation\.

These variations are building blocks, not mutually exclusive task classes\. A single hierarchy can combine nesting, fan\-out, fixed dispatch labels, and multiple\-source requirements, while subtasks add further branches at any shared key\. This composability allows one representation to cover diverse workflows\.

## 4Task Construction

Tasks are produced by a semi\-automated pipeline with four stages:*seeding*, iterative*authoring*,*admission*into a candidate pool, and*curation*into the released set \(Figure[3](https://arxiv.org/html/2608.14747#S4.F3)\)\.

*Seeding*identifies requests with a wide\-research shape in de\-identified product\-usage logs\. During*authoring*, an author agent drafts the design, task specification, and fixtures, and a separate*critic*reviews every draft; failures return to the stage that owns them\.*Admission*checks that the requested volume is attainable, audits the judge, and optionally adds human sign\-off\. Finally,*curation*labels admitted tasks and selects a subset matching the target distributions\. Each released task ships as a self\-contained*package*with its solver\-facing task, verifier, fixtures, and labels\. During construction, outputs from the authoring rollouts are merged into a best\-known solution that serves as an internal feasibility witness, not an answer key; grading never compares a submission against it\. Appendix[G](https://arxiv.org/html/2608.14747#Ax1.SS7)summarizes the public package format, while Appendix[H](https://arxiv.org/html/2608.14747#Ax1.SS8)expands the admission gates\. The subsections below describe seeding \(Section[4\.1](https://arxiv.org/html/2608.14747#S4.SS1)\), authoring \(Section[4\.2](https://arxiv.org/html/2608.14747#S4.SS2)\), admission \(Section[4\.3](https://arxiv.org/html/2608.14747#S4.SS3)\), and curation \(Section[4\.4](https://arxiv.org/html/2608.14747#S4.SS4)\); Section[4\.5](https://arxiv.org/html/2608.14747#S4.SS5)then summarizes the released set\.

Figure 3:The task\-construction pipeline has four stages\.Seedingmines product logs for wide\-research\-shaped requests and selects a seed\.Authoringuses an interleaved author–critic loop for design, task, and fixture drafts, plus a mechanical linter\.Admissionapplies a feasibility audit based on merged rollouts and a judge audit, followed by optional human sign\-off against the four\-axis rubric; automated runnability, cleanliness, difficulty, and discrimination gates are enforced during authoring\.Curationlabels admitted tasks and selects a subset matching the target distributions\. The output is 500 public task packages; system grading is a separate process \(Section[5](https://arxiv.org/html/2608.14747#S5)\)\.### 4\.1Seeding

Each task begins with a*seed*: a real request mined from de\-identified product\-usage logs\.111Two privacy safeguards apply\. First, seed mining follows the de\-identification methodology of the DRACO benchmark\([19](https://arxiv.org/html/2608.14747#bib.bib22)\): requests are sampled from a de\-identified pool and reworded to remove identifying details\. Second, authoring retains only the seed’s reusable pattern and re\-grounds the task in entities with a public web presence\. A released task therefore reflects a recurring form of professional work, not the content of any individual query or private record\.A first filter retains requests shaped like wide research—asking to compile a list, table, or directory—and favors sessions with heavy tool use\. A classifier then ranks candidates by whether their entities and sources are stable enough to build on, their answers can be checked, and discovery is genuinely difficult\. Rejected seeds remain as negative examples so they do not resurface\. The authoring stage claims seeds in batches under two controls:*deduplication*against previously used seeds and tasks, and*coverage steering*toward the target distributions, which favors diverse, economically useful domains and caps overrepresented or low\-value ones\.

### 4\.2Authoring

Authoring turns a seed’s reusable pattern into a fresh, self\-contained task through an iterative loop between two agents: an*author*drafts, a separate*critic*reviews, and the cycle repeats until the draft passes the relevant checks or returns to an earlier stage\. A structural problem can send the work back to seeding\. When a draft is too easy or exposes a shortcut, the author uses the topology patterns in Section[3\.5](https://arxiv.org/html/2608.14747#S3.SS5)and the criteria and identity mechanisms in Appendices[E](https://arxiv.org/html/2608.14747#Ax1.SS5)and[F](https://arxiv.org/html/2608.14747#Ax1.SS6)to raise its difficulty and adversarial robustness\.222For example, if a seed permits a solver to cite mirror pages—duplicate copies of one listing on different domains—for every submission, the task is tightened to require distinct source domains per entity and prohibit template\-substituted prose\.The work passes through three steps in order:333The mechanical linter runs between steps when only mechanical work remains\. It patches convention violations without changing the entity class, evidence bar, volume, or judge policy\. A meaning\-changing fix is returned to the Task or Design step instead\.

- •*Design*: the author sketches a few candidate directions for the task, with substantial web search to establish that the needed entities and pages actually exist; the critic selects one and sharpens it\.
- •*Task*: the author writes the task package—the key hierarchy, the description the solver reads, the judge specification, and the canonicalization/deduplication settings\.
- •*Fixture*: the author writes a few test records with known intended verdicts, grounded in real pages, and runs them through the judge to confirm the grader behaves as designed\.444Fixtures are unit tests for the task’s judge; they ship with the task but never grade the systems under test\. When a fixture the author trusts disagrees with the judge, the disagreement is triaged—usually the fixture is wrong and gets fixed, but a genuine judge defect sends the task back for repair\.

Throughout the loop, four automated bars require that the package runs, its solver\-facing description, judge specification, and schema remain aligned, primitive internal baselines do not already solve it, and stronger internal rollouts meaningfully outperform weaker ones\. These are the runnability, cleanliness, difficulty, and discrimination checks shown in Figure[3](https://arxiv.org/html/2608.14747#S4.F3)\.

The authoring context also contains a set of vetted guidance tasks\. Appendix[J](https://arxiv.org/html/2608.14747#Ax1.SS10)profiles a subset of five, chosen to expose the rationale behind matrix, open\-set, temporal\-panel, legal\-comparison, and reciprocal\-evidence structures\.

The pipeline can also create*sibling variants*by holding most of the hierarchy fixed while changing one controlled choice: a volume, time window, eligibility rule, or evidence bar\. For example,audio\_gearand itsaudio\_gear\_relaxedsibling share the same counts—50 products, two sentiment branches, and three URLs per branch—while the relaxed sibling accepts shorter and less dedicated opinion evidence\. Sibling variants expose practical tradeoffs, such as admitting a larger target pool while increasing the required volume\.

### 4\.3Admission

Admission begins after a task passes the automated runnability, cleanliness, difficulty, and discrimination gates during authoring\. It adds a feasibility audit, a judge audit, and optional human review\. A failed check returns the task to the stage that owns the problem\.

##### Feasibility audit

The pipeline merges everything found across 10–12 authoring rollouts into a single best\-known solution, which must recover a near\-full requested volume\. A large surplus indicates that the task asks for too little; a persistent shortfall indicates that it asks for too much\. This internal merged output is a feasibility witness, not an answer key\. It establishes that the requested volume is attainable, while grading still evaluates every submission against its own cited pages rather than against the witness\.

##### Judge audit

A reviewer samples graded records and regrades each one blindly, forming an independent verdict before seeing the automated judgment\. Disagreements are attributed to solver behavior, task design, or grading machinery\. Any major defect blocks the task\. After cases attributable to solver behavior are separated, an automated\-judge error rate above approximately10%10\\%of the remaining sample also blocks it\.

##### Optional human review

The author–critic loop automates much of the review; the human rubric provides a final checklist when sign\-off is used\. A reviewer scores four pass/fail axes: whether the*key hierarchy*has sensible keys, an attainable volume floor, and appropriate URL corroboration; whether the*task description*is complete and unambiguous; whether the*grader*correctly encodes the judge instructions, schema, validity gates, and requirement checks; and whether*identity handling*has appropriate canonicalization and deduplication settings\. A reviewed task must pass all four axes\. Appendix[H](https://arxiv.org/html/2608.14747#Ax1.SS8)gives the complete gate and review tables\.

### 4\.4Curation

Curation selects a balanced release from the admitted pool\. Because the pipeline deliberately overproduces, passing every check does not guarantee inclusion\. Each admitted task receives distribution labels, and selection matches the target distributions; a strong task may remain unused when its bucket is already full\. Selected tasks are then stamped into self\-contained release packages\. Across admission and curation, optional human work is budgeted at about 20 minutes per released task—approximately 15 minutes for rubric sign\-off and 5 minutes for labeling—plus a roughly five\-minute feasibility review for tasks that cannot be settled from merged rollouts\. This is roughly one sixth of WideSearch’s reported 2\.33\-hour average for one human completion; that study additionally used two independent annotators per task\([16](https://arxiv.org/html/2608.14747#bib.bib23)\)\. Omitting optional human review allows task production to scale further\.

### 4\.5Task Summary Statistics

We summarize the released benchmark by its subject areas, breadth, depth, and hierarchy\. In this section, we focus on the static structural properties of the benchmark; Section[I](https://arxiv.org/html/2608.14747#Ax1.SS9)complements this analysis by examining how these requirements manifest in practice across historical rollouts\.

##### Verticals

The release spans 13 vertical labels \(Table[2](https://arxiv.org/html/2608.14747#S4.T2)\)\. Each task has one to four labels: 294 tasks \(58\.8%\) have one label, and 206 \(41\.2%\) span multiple verticals\. The mean is 1\.50 labels per task\.

Table 2:Multi\-label vertical coverage over all 500 benchmark tasks\. Counts and shares are per label and therefore sum to more than 500 tasks and 100%\.
##### Breadth vs\. depth

For these corpus\-level statistics, a member is the task’s designated core unit of coverage, such as a company in the running example, whereas a record is one graded root\-to\-URL path\. The required member count multiplies root quotas through the last designated member key, including intervening partition keys\. Records per member is total required records divided by that count; their product therefore recovers required volume exactly \(Figure[4](https://arxiv.org/html/2608.14747#S4.F4)\)\. The medians are 100 members, 3\.00 records per member, and 245 records per task, respectively\. Required volume is below 100 records for 10\.8% of tasks, 100–299 for 44\.4%, 300–999 for 40\.4%, and at least 1,000 for 4\.4%; across all tasks, the total is 170,495 records\. The number of hierarchy levels counts distinct non\-member, non\-URL keys across the root and all subtasks: 12\.8% of tasks have none, 45\.0% have one, 31\.6% have two, and 10\.6% have three or more\.

Figure 4:Required scale and structure of the 500\-task benchmark set\. Top left: required members under the designated task\-level member keys\. Top right: required records per member\. Bottom left: required volume; for each task this is exactly the product of the two top\-panel quantities\. Bottom right: distinct non\-member, non\-URL keys across the root and all subtasks\.
##### Hierarchy and multiplicity

Figure[5](https://arxiv.org/html/2608.14747#S4.F5)places the three structural quantities on one view\. The number of hierarchy levels is not interchangeable with records per member: tasks with one, two, or at least three hierarchy levels all have a median of 3\.00 records per member, while flat tasks have a median of 1\.00\. The former counts distinct key dimensions; the latter measures required record multiplicity after fixing member coverage\.

Figure 5:Task structure across required members and records per member \(log–log; 500 benchmark tasks\)\. Point color denotes the number of hierarchy levels: distinct non\-member, non\-URL keys across the root and all subtasks\. Dashed guides mark the medians of 100 members and 3\.00 records per member\.Figures[4](https://arxiv.org/html/2608.14747#S4.F4)and[5](https://arxiv.org/html/2608.14747#S4.F5)report structural task statistics\.

Structure alone does not capture source accessibility, semantic ambiguity, or the practical effort required to complete each branch\. We additionally estimate complementary empirical statistics from the historical rollouts used during task construction; Appendix[I](https://arxiv.org/html/2608.14747#Ax1.SS9)is reserved for that analysis\.

## 5Grading

Every leaf record receives an independent binary verdict, and the verdicts are then aggregated to the task level\. The grader fetches the cited page and evaluates the submission\{item, url, excerpts, answer\}against universal and task\-specific checks; a record passes only when every applicable condition holds\. The judge prompt emits the repository fieldverdict; throughout the paper, we denote this complete\-record verdict asverdict\_full:

verdict\_full=\\displaystyle\\texttt\{verdict\\\_full\}=\{\}𝟏page\_content\_usable×𝟏answer\_intent\_clear×𝟏excerpts\_faithful⏟universal×\\displaystyle\\underbrace\{\\mathbf\{1\}\_\{\\texttt\{page\\\_content\\\_usable\}\}\\times\\mathbf\{1\}\_\{\\texttt\{answer\\\_intent\\\_clear\}\}\\times\\mathbf\{1\}\_\{\\texttt\{excerpts\\\_faithful\}\}\}\_\{\\text\{universal\}\}\\times\{\}\(1\)𝟏overall\_valid×𝟏requirements\_all\_satisfied×𝟏requirements\_all\_supported⏟task\-specific,\\displaystyle\\underbrace\{\\scalebox\{0\.88\}\{$\\displaystyle\\mathbf\{1\}\_\{\\texttt\{overall\\\_valid\}\}\\times\\mathbf\{1\}\_\{\\texttt\{requirements\\\_all\\\_satisfied\}\}\\times\\mathbf\{1\}\_\{\\texttt\{requirements\\\_all\\\_supported\}\}$\}\}\_\{\\text\{task\-specific\}\},
The universal block applies the same evidence mechanics to every task:

- •page\_content\_usablerequires substantive, on\-topic page content\.
- •answer\_intent\_clearrequires a specific, identifiable submitted claim\.
- •excerpts\_faithfulrequires verbatim or near\-verbatim, meaning\-preserving excerpts\.

The task\-specific block combines eligibility and evidence:

- •overall\_validrequires a well\-formed, in\-scope record\.
- •requirements\_all\_satisfiedasks whether the full page satisfies every substantive requirement\.
- •requirements\_all\_supportedasks whether the submitted excerpts alone support every requirement\.

For comparison, the scorer defines a retrieval\-only verdict:

verdict\_retrieval=𝟏requirements\_all\_satisfied\.\\texttt\{verdict\\\_retrieval\}=\\mathbf\{1\}\_\{\\texttt\{requirements\\\_all\\\_satisfied\}\}\.\(2\)The full verdict,verdict\_full, evaluates the complete submitted record using the full page and submitted excerpts\. The retrieval\-only verdict,verdict\_retrieval, asks only whether the fetched page satisfies every substantive task requirement, independently of the submitted excerpts\. Universal and validity fields are not mechanically multiplied intoverdict\_retrieval; instead, the judge is instructed to gate page\-level satisfiability on those preceding checks when they are relevant\. We useverdict\_fullfor the main results and compare the two verdicts in Section[6\.3](https://arxiv.org/html/2608.14747#S6.SS3)\. Appendix[E](https://arxiv.org/html/2608.14747#Ax1.SS5)details the criterion schema\.

The judge also reports confidence\. A leaf signal is used only at confidence 2 or 3; lower\-confidence leaves are treated as missing, so they do not enter precision but can reduce recall when their omission leaves fewer than the required number of members\. Because grading re\-fetches cited pages, solving and grading should occur close together: a URL that changes after submission can cause an otherwise correct record to fail\.

##### Identity resolution

Before aggregation, the grader resolves identity independently along each key axis\. Closed or predictable axes are canonicalized, while open\-ended axes are semantically deduplicated; this prevents cosmetic variants from satisfying a volume floor more than once and makes coverage well\-defined across parent tasks and subtasks\. In the running example, the company key is resolved across the appointment and listing branches, while the composite company–appointee key distinguishes appointments\. Appendix[F](https://arxiv.org/html/2608.14747#Ax1.SS6)details the canonicalization, deduplication, and entity\-scope policies\.

##### Metrics

Leaf scores aggregate*bottom\-up*\. At each level, precision averages every supplied child score\. Recall first collapses duplicate identities by retaining the worst score for each entity, then sorts the distinct entity scores, keeps the topkk, wherekkis the required count, and zero\-pads any shortfall before averaging\. The grader reports soft and hard versions of precision, recall, and F1:555More detail is provided by the stylized rollup\-algorithm implementation in Appendix[K](https://arxiv.org/html/2608.14747#Ax1.SS11)and by a rollup\-walkthrough example rendered via the repository\-shipped viewer in Appendix[L](https://arxiv.org/html/2608.14747#Ax1.SS12)\.

- •Precision— the mean score among submitted members\. Soft precision gives partial credit to incomplete members; hard precision assigns credit only when a submitted member’s required subtree is fully correct\.
- •Recall— the sum of retained member scores divided by the required member count, measuring quality\-adjusted completion relative to the target\. If a submission exceeds the target, only the topkkmembers by score are retained; if it falls short, the remainder is zero\-padded\. Soft recall gives partial credit, while hard recall assigns credit only to submitted members whose required subtrees are fully correct\.
- •F1— the harmonic mean of precision and recall within each task\. Soft F1 summarizes partial\-credit collection quality; hard F1 summarizes complete\-member performance\.

The benchmark reports the simple unweighted mean of each per\-task metric\. Every task therefore contributes equally to the headline score, regardless of its required record count\.

These metrics localize failure along several dimensions\.*Precision*measures the quality of submitted members, so low precision is consistent with a depth failure such as incorrect facts, unfaithful excerpts, or missing corroboration\. The gap between scores underverdict\_retrievalandverdict\_fullmeasures how much additional credit is lost when grading the complete submitted record rather than page\-level task satisfaction alone\. The drop from precision to recall is primarily a breadth signal: recall zero\-pads unmet quotas and collapses duplicate identities, whereas precision averages the submitted children\. A system that submits only a few perfect members can therefore earn high precision but low recall\. Poor identity management can inflate raw submitted\-member counts, but canonicalization and deduplication collapse aliases before recall is computed\. The soft–hard gap is a completeness signal: hard scores remove partial credit from submitted members whose required subtrees are incomplete\.

## 6Experiments and Results

### 6\.1Setup

Conceptually, evaluation uses a four\-stage record\-level pipeline—*solve*→\\rightarrow*fetch*→\\rightarrow*judge*→\\rightarrow*score*\(Figure[6](https://arxiv.org/html/2608.14747#S6.F6)\)\.*Solve*converts every system’s output into a common record format, ensuring identical downstream grading\.*Fetch*retrieves the cited pages and identifies broken or bot\-walled results, including login walls, paywall stubs, and unrendered JavaScript\. Triage routes these cases through a heavier JavaScript\-rendering browser before judgment, reducing crawler\-induced failures\. In parallel, the evaluation pipeline canonicalizes and deduplicates each key axis\.*Judge*issues one verdict per record\.*Score*aggregates those verdicts into task\-level metrics; during recall aggregation, duplicate identity variants collapse to one entity at their worst score\. Append\-only caching allows interrupted runs to resume rather than restart\. Every released task packages the same pipeline\.

Figure 6:The grading pipeline\.Solvenormalizes each system’s output into the common record format;fetchretrieves every cited page, with triage routing broken fetches to a browser retry;judgecanonicalizes and deduplicates identities and emits a structured judgment for each record;scoreaggregates verdicts into soft and hard precision, recall, and F1 scores at the task level \(Section[5](https://arxiv.org/html/2608.14747#S5)\)\. The evaluated solver interfaces are described under Setup \(Table[3](https://arxiv.org/html/2608.14747#S6.T3)\)\.We evaluate six production systems with different search and orchestration interfaces \(Table[3](https://arxiv.org/html/2608.14747#S6.T3)\)\. These external systems are evaluated only after the task set is frozen and are not used for admission or rejection\. The comparison includes task/search application programming interfaces \(APIs; Exa and Parallel\), general web\-agent APIs \(OpenAI’s Responses API with web search and a code interpreter; Anthropic’s managed agents\), a deep\-research agent \(Gemini Deep Research\), and a programmatic search\-orchestration system \(Perplexity Search as Code\)\. All systems use the same pinned GPT\-5\.4 evaluation configuration: low effort for triage and canonicalization, medium for judging, and high for deduplication\. Composite tasks multiply matching main\-task and subtask scores, so a weak subtask reduces the composed entity score\. By contrast, fan\-out over a dispatch axis is averaged: covering three of four evidence types earns0\.750\.75, while a missing subtask cell can zero the corresponding entity\.

Table 3:Systems evaluated in the main benchmark run\. Configurations, model settings, and output\-delivery mechanisms are the recorded run settings; when the interface exposes no effort override, we report the selected model’s documented default\. Output delivery records how the harness obtains the large JSON submission: through provider sandbox files, Perplexity’s file\-sharing tool, or model output tokens\. Solver\-surface labels describe how the benchmark invokes each system; they are not product tiers or capability classes\.
### 6\.2Main Results

We report two sets of main results\. On the full sample, we use the second\-highest available setting for Perplexity, OpenAI, Parallel, and Exa, Gemini’sspeedsetting,666We used Geminispeedinstead ofmaxon the full sample to save time and cost;maxis evaluated in a 45\-task subset\.and Anthropic’shighsetting,777Claude Managed Agents exposes no effort override in its agent configuration\([1](https://arxiv.org/html/2608.14747#bib.bib20)\), so we report effort from the selected model\. Claude Opus 4\.8 defaults theeffortparameter tohighacross all surfaces, including the Messages API\. See Anthropic’s May 28, 2026 release notes:[https://platform\.claude\.com/docs/en/release\-notes/overview\#may\-28\-2026](https://platform.claude.com/docs/en/release-notes/overview#may-28-2026)\.together with each system’s best available delivery method\. On a 45\-task subset, we evaluate every available effort setting\. Appendix[M](https://arxiv.org/html/2608.14747#Ax1.SS13)reports the delivery\-method ablation on the same subset\.

##### Full sample

We report one run per system over all 500 benchmark tasks\. Scores are simple, unweighted means of per\-task metrics\. Trials ending in terminal errors after repeated retries have no metric\-bearing verifier result and are zero\-filled in the aggregate\.888Retries target specific failure modes: provider/API retries handle request failures, Relay permits up to two full solver restarts, and verifier retries address incomplete judgments\. A trial still errors if every attempt ends in a terminal provider failure, timeout, or missing required output\.

Table 4:WANDR scores underverdict\_fullfor the main benchmark run\.Completedis the scheduled 500 tasks minus errored trials and therefore equals the number of metric\-bearing trials\. Soft and hard precision, recall, and F1 are copied directly from the recorded job\-level metrics; no score is reconstructed from another metric\. Bold marks the best value in each numeric column and underline marks the second\-best distinct value\.Table 5:Operational statistics for the same benchmark runs as Table[4](https://arxiv.org/html/2608.14747#S6.T4)\.Completedis the scheduled 500 tasks minus errored trials\. $/task is average solver cost\. Med\. min and P90 min are the median and 90th\-percentile solve\-stage latencies computed from first\-to\-last solve\-stage timestamps, excluding verifier time\. In M and Out k are average provider\-reported input and output tokens per scheduled task, in millions and thousands; Exa and Parallel do not expose token counters\. Bold marks the best value and underline marks the second\-best distinct value, using unrounded values; higher is better for completion and lower is better for resource use\.Taken together, Tables[4](https://arxiv.org/html/2608.14747#S6.T4)and[5](https://arxiv.org/html/2608.14747#S6.T5)and Figure[7](https://arxiv.org/html/2608.14747#S6.F7)show that no system dominates both performance and resource use\. Perplexity leads at0\.3630\.363soft F1 and0\.1330\.133hard F1 with midrange cost \($5\.20 per task,14\.914\.9\-minute median latency, and3\.823\.82M total reported tokens per task\)\. Anthropic ranks second \(0\.2490\.249,0\.0720\.072\) but has the highest cost, latency, and token use\. OpenAI and Exa cost about $0\.50 per task, and Exa is fastest at a5\.35\.3\-minute median, but the remaining systems reach at most0\.1210\.121soft F1 and0\.0350\.035hard F1\. Exa and Parallel do not expose token counts\. The benchmark remains unsaturated: the highest mean task\-level hard precision and recall are only0\.1500\.150and0\.1340\.134, corresponding to complete credit for roughly one in seven submitted members and one in seven required members, respectively, under the benchmark’s task\-level averaging\.

\(a\)Soft and hard precision and recall\.\(b\)Soft and hard F1\.
Figure 7:Performance\-cost views underverdict\_fullfor the main benchmark run\. Each column uses a conventional log x\-axis, so movement up and left is better\. In \(a\), circles and squares denote precision and recall, while white and solid fill denote soft and hard metrics; in \(b\), fill denotes soft and hard F1\. Small horizontal offsets are visual only; tables and prose report the unshifted values\. Scores are the task\-level means in Table[4](https://arxiv.org/html/2608.14747#S6.T4)\. Token use is the reported total per task; Exa and Parallel are omitted because they expose no token counters\.
##### 45\-task subset

We evaluate every system–setting pair under the systems’ best delivery channels, as in Table[3](https://arxiv.org/html/2608.14747#S6.T3)\. Tables[6](https://arxiv.org/html/2608.14747#S6.T6)and[7](https://arxiv.org/html/2608.14747#S6.T7)report the results\. Perplexity, OpenAI, Parallel, and Exa each have four settings; Gemini has two and Anthropic onehighconfiguration\.*Completed*is the number of metric\-bearing trials out of 45\.

Taken together, Tables[6](https://arxiv.org/html/2608.14747#S6.T6)and[7](https://arxiv.org/html/2608.14747#S6.T7)and Figure[8](https://arxiv.org/html/2608.14747#S6.F8)extend the comparison across all available effort settings\. Perplexity, Gemini, and Exa improve monotonically in soft and hard F1 across their available settings; Perplexityxhighreaches0\.4470\.447soft F1 and0\.2240\.224hard F1\. OpenAI peaks athigh, while Parallel’s soft F1 rises throughultra8xbut its hard F1 dips slightly betweenultra2xandultra4x\. Resource use spans more than four orders of magnitude in cost, from Exalowat $0\.03 per task to Geminimaxat $324\.83\.

Table 6:Matched 45\-task score sweep across all available system settings\. Every score is a simple zero\-filled mean over the same 45 scheduled tasks\. Bold marks the best value in each numeric column and underline marks the second\-best distinct value, using unrounded values\.SystemSettingCompletedCostLatencyToken usage$/taskMed\. minP90 minIn MOut kPerplexitylow450\.401\.23\.40\.103\.3Perplexitymedium452\.075\.819\.31\.0113\.7Perplexityhigh454\.7515\.636\.94\.1128\.2Perplexityxhigh447\.3218\.332\.37\.6943\.2Anthropichigh4549\.0978\.5132\.757\.04316\.6OpenAIlow450\.071\.22\.10\.024\.3OpenAImedium450\.325\.48\.00\.1019\.5OpenAIhigh450\.498\.813\.80\.1432\.1OpenAIxhigh450\.7410\.414\.40\.1655\.3Geminispeed4514\.0320\.658\.32\.6083\.1Geminimax45324\.8392\.9322\.773\.62896\.1Parallelultra400\.2726\.655\.3––Parallelultra2x450\.6023\.737\.8––Parallelultra4x451\.2034\.966\.2––Parallelultra8x452\.4031\.358\.1––Exalow450\.031\.25\.9––Examedium450\.106\.311\.5––Exahigh450\.505\.37\.2––Exaxhigh451\.009\.117\.1––Table 7:Operational statistics for the same matched 45\-task sweep as Table[6](https://arxiv.org/html/2608.14747#S6.T6)\. Cost and token totals are divided by 45 scheduled tasks; median and P90 are solve\-stage latencies over tasks with valid timestamps\. In M and Out k denote millions and thousands of tokens\. Exa and Parallel expose no token counters\. Bold marks the best value and underline marks the second\-best distinct value, using unrounded values; higher is better for completion and lower is better for resource use\.\(a\)Soft F1\.\(b\)Hard F1\.
Figure 8:F1 against cost, median solve latency, and total token use for the matched 45\-task effort sweep in Tables[6](https://arxiv.org/html/2608.14747#S6.T6)and[7](https://arxiv.org/html/2608.14747#S6.T7)\. Every point in \(a\) and \(b\) uses the same 45 tasks; for settings also run on the full benchmark, only the matched 45 tasks are included\. System names directly label the colored paths, and darker color indicates higher effort\. Exa and Parallel are absent from the token columns because their results expose no token counters\.

### 6\.3Failure Analysis

#### 6\.3\.1WANDR as a Retrieval Program

Solving a WANDR task amounts to executing a retrieval program over a required entity space: discover enough members, enrich every required branch, qualify the resulting pages, and render source\-backed records\. We analyze four observable components of that program\.*Breadth–depth–extract execution*carries candidates from discovery through enrichment to evidence extraction while tracking quotas and backfilling sparse branches\.*Identity disambiguation*canonicalizes keys, collapses duplicates, and attaches evidence to the correct entity\.*Semantic qualification*checks type, eligibility, source role, and page\-level support\.*Evidence rendering*selects faithful, sufficient excerpts and emits normalized records\. The layered verifier makes these components inspectable at structural, task, and record resolution\.

Search as Code \(SaC\) is well matched to this repeated horizontal structure\. It lets a model compose retrieval, ranking, filtering, fan\-out, and rendering primitives through generated Python, while the sandbox executes batching, retries, joins, aggregation, and deduplication without a separate model turn for every web operation\. Filesystem state can preserve candidate tables and quota deficits across turns, supporting systematic backfilling instead of repeated serial search\-and\-read cycles\([13](https://arxiv.org/html/2608.14747#bib.bib21)\)\. Perplexity’s observed profile is consistent with that advantage: it has the smallest soft precision–recall gap, the highest post\-discovery retention, and the strongest conditional evidence completion without the highest cost, latency, or token use \(Tables[4](https://arxiv.org/html/2608.14747#S6.T4),[5](https://arxiv.org/html/2608.14747#S6.T5), and[9](https://arxiv.org/html/2608.14747#S6.T9); Figure[9](https://arxiv.org/html/2608.14747#S6.F9)\)\.

#### 6\.3\.2Breadth–Depth–Extract Execution

##### Precision–recall gaps expose missing breadth

Underverdict\_full, soft recall is lower than soft precision for every system\. Precision averages the quality of submitted members, whereas recall also collapses duplicate identities and zero\-pads any shortfall against the required member count\. Perplexity retains0\.3570\.357recall from0\.3890\.389precision, the smallest gap; Anthropic falls from0\.3540\.354to0\.2220\.222, the largest\. Because measured identity collapse changes endpoint\-average full soft recall by at most0\.1270\.127percentage points, under\-delivery accounts for most of these gaps \(Tables[4](https://arxiv.org/html/2608.14747#S6.T4)and[8](https://arxiv.org/html/2608.14747#S6.T8)\)\.

##### Soft–hard gaps expose incomplete members

Soft metrics reward partial subtrees; hard metrics credit only members whose required descendants and checks are all complete\. Perplexity falls from0\.3630\.363soft F1 to0\.1330\.133hard F1, Anthropic from0\.2490\.249to0\.0720\.072, and the remaining systems from0\.0550\.055–0\.1210\.121to0\.0090\.009–0\.0350\.035\. The loss also appears in precision—0\.3890\.389to0\.1500\.150for Perplexity and0\.3540\.354to0\.1370\.137for Anthropic—so many submitted members are only partially complete\. SaC leads every hard metric, but coordinated identity, qualification, and rendering failures still prevent full\-member completion \(Table[4](https://arxiv.org/html/2608.14747#S6.T4)\)\.

##### Task structure separates discovery, enrichment, and extraction

We factor full soft recall into three successive retentions over the same submitted members\.*Discovery*is quota\-retained delivered members divided by required members;*enrichment*is their mean composed retrieval\-only soft recall; and*extraction*is their summed full soft recall divided by summed retrieval\-only soft recall\. The product reconstructs full soft recall after recognized identity effects are separated, making stage\-local and cumulative losses directly comparable \(Figure[9](https://arxiv.org/html/2608.14747#S6.F9)\)\.999A small endpoint\-level correction \(0\.01%0\.01\\%–0\.31%0\.31\\%\) reconciles the reconstructed product with reported full soft recall; restoring the identity effect in Table[8](https://arxiv.org/html/2608.14747#S6.T8)then reproduces the reported score exactly\.

Figure 9:Decomposition of full soft recall into discovery, enrichment, and extraction after factoring out recognized identity losses\. \(a\) Cumulative retention after each stage\. \(b\) Error at each stage conditional on reaching it\. \(c\) Each stage’s share of the total loss; segments sum to100%100\\%within an endpoint\. The small correction described in the text aligns the reconstructed product with reported full soft recall\. Endpoint order is Perplexity, Anthropic, OpenAI, Gemini, Parallel, and Exa, based on 496, 500, 498, 493, 350, and 490 scored submissions, respectively\.
##### Late extraction cliffs do not dominate total loss

Extraction removes30\.9%30\.9\\%–72\.9%72\.9\\%of the score that reaches it, yet contributes only7\.0%7\.0\\%–25\.3%25\.3\\%of total loss and is the smallest component for five systems\. Discovery and enrichment have already removed47\.7%47\.7\\%–89\.0%89\.0\\%of the initial opportunity\. OpenAI illustrates the converse: it retains83\.9%83\.9\\%through discovery, then loses68\.2%68\.2\\%of the remainder during enrichment, making enrichment64\.7%64\.7\\%of its total loss \(Figure[9](https://arxiv.org/html/2608.14747#S6.F9)\)\.

##### Target volume correlates primarily with pre\-extraction erosion

From the smallest to largest required\-volume bin, retrieval\-only hard recall falls from0\.3760\.376to0\.1720\.172for Perplexity,0\.2670\.267to0\.1170\.117for Anthropic, and0\.1910\.191to0\.0560\.056for OpenAI\. Full\-to\-retrieval conversion weakens less consistently: the first\-to\-last decline averages10\.410\.4percentage points for precision and13\.713\.7for recall across systems, while Gemini’s precision conversion rises\. Resource use does not explain a common scaling response: Anthropic’s latency, cost, and token use rise sharply, but Perplexity’s token use falls from4\.874\.87M to2\.952\.95M and OpenAI’s remains near0\.170\.17M \(Figure[10](https://arxiv.org/html/2608.14747#S6.F10), top block\)\.

##### More hierarchy levels compound both losses

From zero to three or more hierarchy levels, Perplexity’s retrieval\-only hard precision falls from0\.5780\.578to0\.1050\.105and hard recall from0\.5230\.523to0\.0720\.072; Anthropic falls from0\.5510\.551to0\.1410\.141and from0\.4070\.407to0\.0740\.074; and OpenAI falls from0\.2630\.263to0\.0590\.059and from0\.2150\.215to0\.0510\.051\. Unlike target volume, additional levels reduce both precision and recall conversion for every system, by33\.733\.7and35\.335\.3percentage points on average from the shallowest to deepest bin\. Resource use again has no shared monotonic response, so these observational curves localize the loss but do not isolate hierarchy as its cause \(Figure[10](https://arxiv.org/html/2608.14747#S6.F10), bottom block\)\.

Figure 10:Task\-level hard precision, hard recall, and resource scaling across target record volume \(top block\) and number of hierarchy levels \(bottom block\)\. Within each block, the upper\-left panels show retrieval\-only scores and the lower\-left panels show full\-to\-retrieval conversion\. Each conversion is the bin’s mean full score divided by its mean retrieval\-only score, so multiplying vertically reconstructs the mean full score exactly\. The shared resource panels span both score rows because operational statistics are unchanged\. Each line is one fixed main\-run system configuration; points are equal\-task aggregates within bins\. Scores, mean solver cost, and mean provider\-reported total tokens use the scheduled bin denominator, with missing values zero\-filled to match the benchmark aggregate within systems that expose each measure; latency is the median over tasks with valid solve timestamps\. Exa and Parallel are absent from the token panels because they expose no token counters\. The target\-volume bins contain 87, 103, 86, 117, and 107 tasks, and the hierarchy\-level groups contain 64, 225, 158, and 53 tasks;3\+3\+pools tasks with three to six distinct non\-member, non\-URL keys across the root and all subtasks\. Cost, latency, and token use have log y\-axes\.

#### 6\.3\.3Identity Disambiguation

##### Identity collapse is sparse globally but material when it changes score

The scorer merges submitted values that refer to the same entity, preventing aliases from inflating coverage\. Comparing full soft recall with and without the merge penalty captures both lost duplicate count and the take\-worst penalty when variants of one identity carry evidence of different quality \(Table[8](https://arxiv.org/html/2608.14747#S6.T8)\)\.

Table 8:Identity collapse in the six full runs\. Affected is the share of tasks with at least one merged entity identity; Values is the share of submitted entity\-key values merged into another identity\. All tasks is the mean full soft\-recall loss; When nonzero is the mean conditional on identity collapse lowering recall\. The performance columns use the scored submissions available for 496, 500, 498, 493, 350, and 490 tasks in displayed order\. One OpenAI submission moves in the opposite direction and remains in the all\-task net effect\.Across systems, only2\.4%2\.4\\%–8\.0%8\.0\\%of tasks contain a duplicate entity identity, and only0\.018%0\.018\\%–0\.426%0\.426\\%of submitted entity values collapse\. The mean full soft\-recall effect is therefore small,0\.0170\.017–0\.1270\.127percentage points, but rises to0\.770\.77–4\.524\.52points when collapse changes a task’s score\. Identity is not a major aggregate explanation of endpoint differences, but it can materially damage an affected task \(Table[8](https://arxiv.org/html/2608.14747#S6.T8); Appendix[L](https://arxiv.org/html/2608.14747#Ax1.SS12)\)\.

##### SaC concentrates rather than eliminates identity errors

Perplexity has duplicate identities on only4\.8%4\.8\\%of tasks, below Gemini, Exa, OpenAI, and Parallel, but it has both the largest collapsed\-value share \(0\.426%0\.426\\%\) and the largest conditional soft\-recall loss \(4\.524\.52points\)\. Parallel has the highest affected\-task share \(8\.0%8\.0\\%\), yet one of the smallest conditional effects \(0\.780\.78points\)\. SaC’s identity failures are therefore less frequent than in four systems but more concentrated when they occur \(Table[8](https://arxiv.org/html/2608.14747#S6.T8)\)\.

#### 6\.3\.4Semantic Qualification

##### Usable, valid pages are common; complete page support is not

For five systems, only3\.2%3\.2\\%–8\.9%8\.9\\%of submitted pages are unusable; OpenAI is the23\.1%23\.1\\%outlier \(Figure[11](https://arxiv.org/html/2608.14747#S6.F11)\)\. Invalid\-or\-wrong\-type failures are also comparatively limited at4\.2%4\.2\\%–17\.4%17\.4\\%\. The larger problem begins after a page is fetched and judged plausible:33\.6%33\.6\\%–68\.3%68\.3\\%of records cite a page that does not satisfy every substantive task requirement\. Anthropic performs best on this check at33\.6%33\.6\\%failure, with Perplexity second at41\.4%41\.4\\%\. Finding an accessible, broadly in\-scope page is therefore not the main semantic bottleneck; finding a page that establishes the complete requested claim is\.

![Refer to caption](https://arxiv.org/html/2608.14747v1/FailureFields.png)Figure 11:Measured row\-level failure probes from available verifier details \(lower is better\)\. Page req\. unmet means that the fetched page fails at least one substantive requirement; Excerpt req\. unmet means that the submitted excerpts fail at least one\. Final verdict denotes failure underverdict\_full\. Each cell reports the failure rate in percent, rounded to one decimal place; row labels give the number of available task details\. Coverage is 500, 500, 499, 493, 350, and 490 tasks for Perplexity, Anthropic, OpenAI, Gemini, Parallel, and Exa, respectively; Parallel’s field analysis is therefore less complete than its aggregate results\.

#### 6\.3\.5Evidence Rendering

##### Faithful excerpts usually support something, but not everything

We condition on records whose fetched page satisfies every substantive requirement, then measure whether the submitted excerpts are faithful, whether a faithful excerpt supports at least one requirement, and whether it supports all requirements \(Table[9](https://arxiv.org/html/2608.14747#S6.T9)\)\.

Table 9:Evidence rendering after conditioning on records whose fetched page satisfies all substantive requirements\. Values are equal\-task means\. Faithful uses all such records; the two support columns then condition on faithful excerpts\. Any supported is computed from the task\-specific support checks; All supported uses the judge’s aggregate all\-requirements field\. The initial condition is observed on 494, 499, 489, 476, 341, and 473 tasks in displayed order; the conditional columns have 488, 497, 464, 470, 336, and 457 tasks with at least one faithful record\.
##### Partial support is the dominant rendering failure

Once an excerpt is faithful, it supports something on96\.9%96\.9\\%–99\.4%99\.4\\%of records, but supports every requirement on only55\.6%55\.6\\%–80\.4%80\.4\\%\. The characteristic failure is therefore a faithful, partly useful selection that omits facts needed to verify the complete record\. OpenAI has low faithfulness \(57\.1%57\.1\\%\) but relatively strong completion once faithful \(75\.7%75\.7\\%\), whereas Gemini has high faithfulness \(79\.1%79\.1\\%\) but the weakest complete support \(55\.6%55\.6\\%; Table[9](https://arxiv.org/html/2608.14747#S6.T9)\)\.

Figure 12:Complete excerpt support after conditioning on a page that satisfies all requirements and on faithful excerpts\. \(a\) Equal\-task rates by the number of task\-specific support checks in the full runs\. \(b\) Rates by within\-task fetched\-page\-length quartile in a deterministic 12\-task\-per\-system cache sample\. Tick labels give the mean page length in each quartile, estimated at four characters per token\. The page\-length analysis covers 7,193 of 7,339 page\-satisfied records with joined content\.
##### More requirements widen the evidence\-selection gap

Complete support falls between tasks with two task\-specific support checks and those with five for all six systems, by6\.86\.8–21\.821\.8percentage points\. This association is descriptive because requirement count also varies with task content and structure, but it consistently exposes a broader selection burden after the page and excerpt pass the preceding checks \(Figure[12](https://arxiv.org/html/2608.14747#S6.F12)\(a\)\)\.

##### Long pages show the same selective\-completeness problem

In the cache sample, complete support falls from the shortest to longest within\-task page\-length quartile in five of six systems\. Manual inspection matches the pattern: failures omit a date from a ranking page, air\-burden language from a permit, or an individual review from a long review surface even when the selected text is faithful and partly relevant\. This analysis is exploratory because page length co\-varies with source type and record complexity \(Figure[12](https://arxiv.org/html/2608.14747#S6.F12)\(b\)\)\.

#### 6\.3\.6Cross\-System Synthesis

##### Failures compound across layers

Recall falls below precision, hard scores fall far below soft scores, and performance deteriorates as target volume and hierarchy levels increase\. The stage decomposition localizes the largest share of soft\-recall loss to enrichment for Perplexity, Anthropic, and OpenAI, but to discovery for Gemini, Parallel, and Exa\. Downstream, substantive page support fails much more often than page usability or basic validity, and faithful excerpts frequently omit part of the required proof \(Table[4](https://arxiv.org/html/2608.14747#S6.T4); Figures[9](https://arxiv.org/html/2608.14747#S6.F9),[10](https://arxiv.org/html/2608.14747#S6.F10), and[11](https://arxiv.org/html/2608.14747#S6.F11); Table[9](https://arxiv.org/html/2608.14747#S6.T9)\)\.

##### SaC’s advantage is broad but not uniform

Perplexity combines the strongest breadth retention with the highest conditional evidence completion, but Anthropic has the lowest page\-requirement failure and Perplexity has the largest conditional identity loss\. OpenAI is the page\-usability and excerpt\-faithfulness outlier; Gemini is usually faithful but has the weakest complete support once faithful; and Parallel has the highest identity\-affected task share, although its diagnostic coverage is limited to 350 tasks\. SaC’s distinctive signature is therefore stronger breadth retention and evidence construction, not uniformly better performance at every layer \(Figure[11](https://arxiv.org/html/2608.14747#S6.F11); Tables[8](https://arxiv.org/html/2608.14747#S6.T8)and[9](https://arxiv.org/html/2608.14747#S6.T9)\)\.

## 7Discussion

### 7\.1Summary

WANDR evaluates a form of research common in professional work but poorly represented by single\-answer and report\-writing benchmarks: constructing a large structured collection while independently verifying every member\. Its 500 tasks combine a large\-scale, often open\-ended breadth requirement with a per\-record depth requirement, and its qualification key hierarchy makes discovery, enrichment, and evidence failures separately observable\. Reference\-free grading accommodates changing facts without maintaining an exhaustive gold answer table; each submitted record instead carries the page and excerpt needed for verification\.

The evaluation shows that this problem remains far from solved\. Across six production systems, Perplexity Search as Code leads at0\.3630\.363soft F1 and0\.1330\.133hard F1, but even its hard recall is only0\.1340\.134\. Perplexity has the smallest soft precision–recall gap and the strongest evidence\-rendering funnel, the profile expected from programmable fan\-out, batching, filtering, joining, deduplication, and target\-aware backfilling\. Its depth performance is not uniformly best, however\. Anthropic has the lowest page\-requirement failure rate, and even for Perplexity only80\.4%80\.4\\%of faithful excerpts completely support the claim after conditioning on a page that does\. The common bottleneck is therefore not search alone but complete evidence construction at scale\.

The four\-layer framework measures failures at aggregate or task\-local resolution\. Breadth–depth–extract execution produces both precision–recall and soft–hard gaps; within soft recall, discovery, enrichment, and extraction show where continuous coverage is lost\. Complete performance also drops sharply with larger targets and more hierarchy levels\. Identity disambiguation has little aggregate effect but can cost several full soft\-recall points on affected submissions\. In semantic qualification, page usability and basic validity are usually not the main failures; substantive page support is\. Within evidence rendering, the dominant signature is partial proof: faithful excerpts usually support at least one requirement but often not all of them\. More generally, the results support treating wide research as a systems problem coupling retrieval, deterministic data processing, identity management, semantic qualification, state management, and evidence rendering\.

Even though WANDR is primarily an evaluation benchmark, the same structure makes it a plausible RL substrate\. The task pipeline can generate held\-out training packages, per\-level required counts support curriculum learning, record\-level verdicts and hierarchy\-level scores provide denser outcome rewards than a single exact match, and the streaming grader can amortize overlapping work across batched rollouts\. Appendix[N](https://arxiv.org/html/2608.14747#Ax1.SS14)discusses this design and its caveats, including the need to train on generated or separately held\-out tasks rather than the released benchmark split\.

### 7\.2Limitations

##### Scope and representativeness

The 500 released tasks are designed to represent recurring patterns in de\-identified production requests, but they are not a random sample of all research work\. Admission deliberately favors high\-volume, feasible, discriminative tasks with independently verifiable web evidence\. This makes WANDR useful as a stress test, but the resulting score distribution should not be interpreted as the difficulty of research requests in deployment\.

##### System comparison and causal attribution

The main evaluation reports one production run per system rather than repeated trials to save time and cost\. An earlier, separately configured evaluation reported the same qualitative ordering\([13](https://arxiv.org/html/2608.14747#bib.bib21)\), but it also used one run per system and therefore does not estimate run\-to\-run variance\. Systems differ simultaneously in model, search infrastructure, agent harness, tool interface, and provider\-side implementation\. The cross\-system ranking is therefore observational and does not isolate any individual component\.

##### Reference\-free grading

Evidence verification avoids a manual and static gold table but introduces judge and retrieval uncertainty\. The grader uses LLM judge calls and re\-fetches each cited page: the page may drift after solving, and extraction can omit dynamic, tabular, or JavaScript\-rendered content\. Canonicalization and deduplication introduce additional boundary decisions when identities are ambiguous\. We keep solving and grading close in time, and retry broken pages with a browser, but these controls reduce rather than eliminate grading error\.

##### Agentic fetch backend alignment

The grader first re\-fetches cited pages through an agentic fetch backend that is also available to the Perplexity solver; unsuccessful fetches are routed to a browser\. Because this backend is shared, the grading path may be better aligned with pages that the Perplexity solver can access than with pages reached through other systems’ retrieval stacks\. In internal grading ablations that disable the agentic fetch backend and use only the browser, scores decrease for every system while the ranking remains mostly stable\. The grading fetch path therefore affects absolute scores and may confer some alignment advantage, even though the comparative ordering appears less sensitive\. Future evaluations should report the grading fetch configuration and compare multiple retrieval paths\.

### 7\.3Future Research Directions

##### Controlled scaling studies

Figure[10](https://arxiv.org/html/2608.14747#S6.F10)identifies descriptive scale sensitivity in the current fixed\-configuration runs, but target volume, intermediate\-level count, content, and difficulty co\-vary\. A useful next experiment is a matched comparison that varies one factor at a time: workload scale, model, search interface, sandbox, skills, or retrieval infrastructure\.

##### Mechanism\-resolved failure analysis

The failure analysis is observational: score trees and verifier fields identify where submitted outputs lose credit, but they do not establish which model, retrieval, orchestration, or interface mechanism caused the loss\. Controlled ablations should hold the task sample and the rest of the system fixed while varying one component at a time, such as code execution, parallel fan\-out, target\-aware backfilling, context persistence, identity handling, or the fetcher\. Paired runs with repeated trials would distinguish causal effects from task composition and run\-to\-run variation\.

##### RL training experiments

Appendix[N](https://arxiv.org/html/2608.14747#Ax1.SS14)explains how WANDR can serve as a training environment; the next step is to test the claim\. Experiments can compare terminal F1 alone against denser, record\- and branch\-level rewards, measure whether quota curricula improve performance, and quantify the grading savings from shared queues and caches as task\-by\-rollout batch size grows\. Training should use generated sibling tasks or newly generated tasks from appropriate held\-out seeds, preserve a held\-out benchmark split, and test for reward hacking—especially over\-submission, evidence templating, and exploitation of judge or cache artifacts\.

### 7\.4Conclusion

WANDR makes large\-scale, evidence\-backed research measurable without an exhaustive gold collection\. Across 500 tasks, recall falls below precision, hard scores remain far below soft scores, and complete performance degrades sharply as target volume and the number of hierarchy levels increase\. Page usability and basic validity usually succeed, but substantive page support and especially complete excerpt support remain major bottlenecks\. It provides a basis for improving comprehensive search systems and testing whether those gains hold at scale\.

## References

- AnthropicClaude managed agents overview\.Note:[https://platform\.claude\.com/docs/en/managed\-agents/overview](https://platform.claude.com/docs/en/managed-agents/overview)Accessed: 2026\-07\-13Cited by:[footnote 7](https://arxiv.org/html/2608.14747#footnote7)\.
- Bigeardet al\.\(2025\)A\. Bigeard, L\. Nashold, R\. Krishnan, and S\. WuFinance agent benchmark: benchmarking LLMs on real\-world financial research tasks\.arXiv preprint arXiv:2508\.00828\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2025\)S\. Chen, P\. Moreira, Y\. Xiao, S\. Schmidgall, J\. Warner, H\. Aerts, T\. Hartvigsen, J\. Gallifant, and D\. S\. BittermanMedBrowseComp: benchmarking medical deep research and computer use\.arXiv preprint arXiv:2505\.14963\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px3.p1.1)\.
- Gouet al\.\(2025\)B\. Gou, Z\. Huang, Y\. Ning, Y\. Gu, M\. Lin,et al\.Mind2Web 2: evaluating agentic search with agent\-as\-a\-judge\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px4.p1.1)\.
- Guptaet al\.\(2026\)N\. Gupta, R\. Chatterjee, L\. Haas, C\. Tao, A\. Wang, C\. Liu, H\. Oiwa, E\. Gribovskaya, J\. Ackermann, J\. Blitzer,et al\.DeepSearchQA: bridging the comprehensiveness gap for deep research agents\.arXiv preprint arXiv:2601\.20975\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.12.1.1)\.
- Huanget al\.\(2026\)Z\. Huang, H\. Ren, X\. Yuan, J\. Wang, Z\. Jiang, K\. Xu, S\. He, J\. Zhao, and K\. LiuWideSeek: advancing wide research via multi\-agent scaling\.arXiv preprint arXiv:2602\.02636\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.14.1.1)\.
- Krishnaet al\.\(2025\)S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. FaruquiFact, fetch, and reason: a unified evaluation of retrieval\-augmented generation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 4745–4759\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.243),[Link](https://aclanthology.org/2025.naacl-long.243/)Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.6.1.1)\.
- Lanet al\.\(2026\)T\. Lan, F\. Henry, B\. Zhu, Q\. Jia, J\. Ren, Q\. Pu, H\. Li, L\. Wang, Z\. Xu, and W\. LuoTable\-as\-Search: formulate long\-horizon agentic information seeking as table completion\.arXiv preprint arXiv:2602\.06724\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px2.p1.1)\.
- Lanet al\.\(2025\)T\. Lan, B\. Zhu, Q\. Jia, J\. Ren, H\. Li, L\. Wang, Z\. Xu, W\. Luo, and K\. ZhangDeepWideSearch: benchmarking depth and width in agentic information seeking\.arXiv preprint arXiv:2510\.20168\.Cited by:[§1](https://arxiv.org/html/2608.14747#S1.p3.1),[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.11.1.1)\.
- Liet al\.\(2025\)H\. Li, J\. Chen, J\. Yang, Q\. Ai, W\. Jia, Y\. Liu, K\. Lin, Y\. Wu, G\. Yuan, Y\. Hu,et al\.LegalAgentBench: evaluating LLM agents in legal domain\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2322–2344\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.116),[Link](https://aclanthology.org/2025.acl-long.116/)Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)S\. Liu, M\. Liu, H\. Zhou, Z\. Cui, Y\. Zhou, Y\. Zhou, J\. Gao, H\. Zhou, Y\. Yang,et al\.VeriWeb: verifiable long\-chain web benchmark for agentic information\-seeking\.arXiv preprint arXiv:2508\.04026\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.8.1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. ScialomGAIA: a benchmark for general AI assistants\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.4.1.1)\.
- Perplexity \(2026\)PerplexityRethinking search as code generation\.Note:Perplexity Research\. Accessed: 2026\-06\-19External Links:[Link](https://research.perplexity.ai/articles/rethinking-search-as-code-generation)Cited by:[§1](https://arxiv.org/html/2608.14747#S1.p3.1),[§6\.3\.1](https://arxiv.org/html/2608.14747#S6.SS3.SSS1.p2.1),[§7\.2](https://arxiv.org/html/2608.14747#S7.SS2.SSS0.Px2.p1.1)\.
- Phanet al\.\(2025\)L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2025\)J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. GlaeseBrowseComp: a simple yet challenging benchmark for browsing agents\.arXiv preprint arXiv:2504\.12516\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.5.1.1)\.
- Wonget al\.\(2025\)R\. Wong, J\. Wang, J\. Zhao,et al\.WideSearch: benchmarking agentic broad info\-seeking\.arXiv preprint arXiv:2508\.07999\.Note:ByteDance SeedCited by:[§1](https://arxiv.org/html/2608.14747#S1.p3.1),[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.10.1.1),[§4\.4](https://arxiv.org/html/2608.14747#S4.SS4.p1.1)\.
- Xionget al\.\(2026\)L\. Xiong, K\. Luo, Z\. Xia, W\. Zhang, J\. Yao, Z\. Liu, J\. Shao, J\. Chen, H\. Qian,et al\.AutoResearchBench: benchmarking AI agents on complex scientific literature discovery\.arXiv preprint arXiv:2604\.25256\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.13.1.1)\.
- Yoranet al\.\(2024\)O\. Yoran, S\. J\. Amouyal, C\. Malaviya, B\. Bogin, O\. Press, and J\. BerantAssistantBench: can web agents solve realistic and time\-consuming tasks?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8938–8968\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.505),[Link](https://aclanthology.org/2024.emnlp-main.505/)Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.7.1.1)\.
- Zhonget al\.\(2026\)J\. Zhong, H\. Zhang, C\. Southern, J\. Yang, T\. Wang, K\. Jung, S\. Zhang, D\. Yarats, J\. Ho, and J\. MaDRACO: a cross\-domain benchmark for deep research accuracy, completeness, and objectivity\.arXiv preprint arXiv:2602\.11685\.External Links:[Link](https://arxiv.org/abs/2602.11685)Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px4.p1.1),[footnote 1](https://arxiv.org/html/2608.14747#footnote1)\.
- Zhouet al\.\(2025\)J\. Zhou, W\. Li, Y\. Liao, N\. Zhang, T\. Miao, Z\. Qi, Y\. Wu, and T\. YangScholarSearch: benchmarking scholar searching ability of LLMs\.arXiv preprint arXiv:2506\.13784\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px3.p1.1)\.
- Zhuet al\.\(2025\)F\. Zhu, X\. Y\. Ng, Z\. Liu, C\. Liu, X\. Zeng, C\. Wang, T\. Tan, X\. Yao, P\. Shao, M\. Xu,et al\.FinDeepResearch: evaluating deep research agents in rigorous financial analysis\.arXiv preprint arXiv:2510\.13936\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px3.p1.1)\.
- Zhuet al\.\(2026\)Y\. Zhu, X\. Zhang, M\. Zhang, J\. Jin, L\. Zhang, X\. Song, K\. Zhao, W\. Zeng, R\. Tang, H\. Li, J\. Wen, and Z\. DouGISA: a benchmark for general information\-seeking assistant\.arXiv preprint arXiv:2602\.08543\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.14747#S2.T1.3.15.1.1)\.
- Zhugeet al\.\(2024\)M\. Zhuge, C\. Zhao, D\. Ashley, W\. Wang, D\. Khizbullin, Y\. Xiong, Z\. Liu, E\. Chang, R\. Krishnamoorthi, Y\. Tian,et al\.Agent\-as\-a\-Judge: evaluate agents with agents\.arXiv preprint arXiv:2410\.10934\.Cited by:[§2](https://arxiv.org/html/2608.14747#S2.SS0.SSS0.Px4.p1.1)\.

## Appendices

### EPer\-Record Grading Criteria

Criteria define what one submitted record must prove\. The judge evaluates each leaf independently from the submitted item, answer, URL, and excerpts, together with the fetched page\.

#### E\.1Layered Checks

Each record is evaluated through universal checks, task\-specific validity checks, and substantive requirements\. Universal checks apply to every task\. Validity checks decide whether the record is eligible and well\-formed enough to judge\. Requirements state the facts that the cited evidence must establish\.

Every substantive requirement is evaluated twice\. A field ending in\_satisfiedasks whether the full page supports the requirement, regardless of which excerpts the solver selected\. Its paired\_supportedfield asks whether the submitted excerpts alone are sufficient for a careful reader to verify the same requirement\. The schema aggregates these pairs intorequirements\_all\_satisfiedandrequirements\_all\_supported\.

In the running CEO/CFO example, the appointment page must identify the company and appointee, establish the role, and place the first public announcement in the target window\. A page can satisfy those requirements even when the solver submits only an excerpt naming the appointee\. In that case, the page\-level fields pass, but the excerpt\-level date or company fields fail because the submitted evidence is incomplete\.

Three universal checks guard this requirement pair:

- •page\_content\_usablerequires a reachable, substantive, on\-topic page; dead links, stubs, and off\-topic pages fail here\.
- •answer\_intent\_clearrequires the record to state the specific claim being made\. A page dump with no localized answer, or excerpts that contradict one another, fails here\.
- •excerpts\_faithfulrequires every excerpt to appear verbatim or near\-verbatim on the page with its meaning preserved\. Paraphrase, fabricated text, misleading truncation, and sentence stitching fail this check\.

Faithfulness and completeness are separate\. An excerpt may quote the page exactly but omit the sentence that establishes the date, role, or source authority; it then passesexcerpts\_faithfulbut fails the relevant\_supportedfield\. Conversely, text stitched from separate passages may mention every needed fact but fail faithfulness because the page never states the combined claim\. This separation is especially important when a solver relies on search snippets, which can omit qualifiers or join nearby text out of context\.

Task\-specific validity checks cover eligibility conditions that do not fit a page/excerpt requirement pair\. Some are structural, such as a required excerpt length or allowed domain; others are broader sanity checks, such as whether a submitted person or company belongs to the requested class\. The schema summarizes them inoverall\_valid\. If a record is so invalid that substantive judgment would be meaningless, the judge short\-circuits the remaining fields to false; otherwise, it still reports the requirement\-level signals for diagnosis\.

Figure[13](https://arxiv.org/html/2608.14747#Ax1.F13)summarizes the resulting sequence\. As defined in Section[5](https://arxiv.org/html/2608.14747#S5),verdict\_retrievalasks whether the fetched page satisfies every substantive task requirement, independently of the submitted excerpts\. Universal and validity fields are not mechanically multiplied into this verdict, but the judge may use them to gate page\-level satisfiability when relevant\. The complete\-record verdict,verdict\_full, additionally requires validity, a usable page, a clear answer, faithful excerpts, and complete excerpt support\. These are the two record\-level verdicts aggregated by the task metrics\.

Figure 13:The record\-level checks\.verdict\_retrievalreads the full\-page requirement signal, shown by the solid bracket segment; its dashed continuation shows the preceding checks that the judge may use to gate satisfiability when relevant, not a mechanistic conjunction\.verdict\_fullalso requires a usable page, a clear answer, faithful excerpts, complete excerpt support, and task\-specific validity\. A task\-specific validity check can short\-circuit an ineligible record\.
#### E\.2Common Evidence Patterns

The exact requirements vary by task, but several evidence patterns recur across the benchmark:

- •*Plausible entity*— the submitted entity must be a sensible instance of the requested class\. A quote author should be a real person rather than a fictional character; an LLM producer should build models rather than merely resell access to them\.
- •*Verified eligibility*— some restrictions are key\-level validity checks, while others require cited evidence\. In the running example, the appointment branch checks that the company is US\-based, while the listing subtask requires listing\-authority evidence of public status\.
- •*Appropriate source*— the citation must come from a source suited to the claim, such as an official filing, first\-party product page, recognized listing authority, or practitioner forum\. The allowed source type can be established by domain, page authorship, or in\-page evidence\.
- •*Long\-tail source*— the qualifying evidence may sit outside the highest\-ranked results\. These tasks reward systematic search beyond prominent pages and aggregators\.
- •*Heavy source*— the evidence may live in a long report, registry, filing, or PDF\. Verifying the claim requires fetching the artifact and navigating to the relevant passage rather than relying on a snippet\.
- •*Dedicated source*— the page must be substantially about the submitted entity, not a directory row, search\-result page, or passing mention\. For example, a task may require a dedicated product listing rather than a buyer’s guide that names the product once\.
- •*Broad extraction*— the claim itself may require many pieces of evidence\. Showing that a governing body has at leastNNmembers, for example, requires excerpts that enumerate enough members and therefore yields empirically long evidence blocks even without an explicit word\-count floor\. A task can also impose such a floor directly\.
- •*Substantive extraction*— the page and excerpts must contain analysis or detail, not merely a rating, label, or short listing\. A narrative\-review requirement filters out bare aggregate scores\.
- •*Scattered extraction*— the required evidence appears in different parts of a page, such as a title, date, and substantive comment\. The solver must collect all of it without joining unrelated passages\.
- •*Structured extraction*— exact values sit in tables, filings, or specification sheets\. The main risk is reading the wrong cell, row, or column; GPU–game FPS results are a typical example\.

### FIdentity Resolution

Identity rules determine when two submitted values count as the same member\. Without them, cosmetic variants could satisfy a volume floor more than once: “Aleksandr Pushkin” and “Alexander Pushkin” should not count as two authors, nor should “NVIDIA” and “Nvidia Corp\.” count as two companies\.

Leaf verdicts remain independent, but scoring must compare values across the submission to count distinct members\. WANDR limits this cross\-record step to one sameness policy per key axis\. Once identity is resolved on each axis, required volumes, coverage, and score rollups are well defined\.

#### F\.1Canonicalization and Deduplication

WANDR uses two complementary mechanisms:

- •*Canonicalization*maps a submitted value to a standard form\. It is well suited to closed or predictable sets: “Mar\. 2025” and “March 2025” can both map to “2025\-03,” dispatch labels can map to a fixed vocabulary, and URLs can be normalized mechanically\. A closed\-set canonicalizer can also reject values outside the allowed set\.
- •*Deduplication*decides whether two values refer to the same entity\. Exact matching is sufficient after deterministic normalization for fields such as URLs or fixed labels\. Open\-ended entities such as people and companies usually require semantic comparison\.

Both mechanisms are configured per key axis\. Each key must contain enough information to identify its values without relying on an implicit parent\. Composite keys supply that context when one field is ambiguous\. In the running example,company\_appointeeincludes both the company and appointee fields, distinguishing appointments of the same person—or same\-named people—across different companies\. A single\-field key is sufficient when its values are already unambiguous within the task\.

When a subtask reuses a parent key, both branches share one identity axis\. In the running example, company names from the appointment and listing branches are resolved together before the two branch scores are composed\.

#### F\.2Common Identity Patterns

Identity policy is part of the task definition, not merely cleanup\. Changing what counts as the same entity changes the set the task asks the solver to cover\.

- •*Granularity*— the policy sets the level being counted\. Folding franchise locations and subsidiaries into one identity creates a brand\-level task; keeping them separate creates an outlet\- or company\-level task\. A closed set may deliberately mix levels when the task treats them as peers, such as a roster containing states, the District of Columbia, and selected county\-level regimes\.
- •*One referent across forms*— renames, abbreviations, and different stages of the same event can map to one underlying member\. A product can remain the same offering after a rebrand, and coverage of a transaction’s announcement and closing can refer to the same transaction even though the pages use different language\.
- •*Exact\-set coverage*— a finite canonical set changes “find at leastnn” into “cover every listed value\.” Aliases map to the same allowed member, and values outside the solver\-facing roster do not satisfy the axis\.

### GTask Package Format

Each released WANDR task is a self\-contained package\. Complete package files, prompts, fixtures, and verifier code live in the public repository\.

Table 10:WANDR task package format\.
### HTask Admission Gates and Review

The authoring prompts and linter instructions are implementation artifacts in the release repository\. A task must be runnable, convention\-clean, sufficiently difficult, discriminative, empirically feasible, and judgeable before it can enter the released set\.

Table 11:Admission and review gates for a task package\. Curation follows admission and selects from the admitted pool to match release\-level distributions\.Table 12:Human\-review rubric\.Reviewers also inspect the rollouts and trajectories accumulated during authoring\. This exposes degenerate solution strategies and shortcuts that may not be apparent from the package or aggregate scores alone, and lets reviewers return the task for hardening before admission\.

### IEmpirical Task Statistics

The structural statistics in Section[4\.5](https://arxiv.org/html/2608.14747#S4.SS5)describe what a task requires, but not how that requirement behaves in practice\. Historical rollouts accreted as tasks were authored, validated, and hardened: authors used them to test feasibility, critics inspected them for shortcuts and ambiguous obligations, and reviewers used them to decide whether a task needed another revision\. We aggregate that archive to characterize WANDR empirically at three resolutions\. The task\-ontology view measures where structured completion becomes difficult; the individual\-citation view measures source reuse and diversity; and the trajectory view measures exploration and effort\.

##### Task\-ontology difficulty

Figure[14](https://arxiv.org/html/2608.14747#Ax1.F14)decomposes structured attainment into discovery, enrichment, and extraction\. For each rollout, discovery difficulty is one minus the delivered\-member fraction; enrichment difficulty is one minus retrieval\-only soft recall averaged over delivered members; and extraction difficulty is one minus the conversion from retrieval\-only to full soft recall over the same members\. The plotted task values are plain means over available rollouts, so unsuccessful executions are not downweighted\. Overall difficulty is one minus the best soft F1 average observed during authoring\.

The extraction tertiles separate visibly: tasks with high extraction difficulty also tend to have higher enrichment difficulty\. Discovery and enrichment have a modest inverse association \(r=−0\.185r=\-0\.185in log\-odds space among positive pairs\): when discovery returns fewer members, the delivered subset can be easier to enrich\. Detached marginal strips retain the exact\-zero cases: 24 tasks for discovery, one for enrichment, and none for both\.

![Refer to caption](https://arxiv.org/html/2608.14747v1/EmpiricalDifficulty.png)Figure 14:Task\-ontology view of empirical difficulty over 500 tasks\. Positive discovery and enrichment difficulty are shown on log\-odds axes; exact\-zero discovery values appear in the detached vertical strips and the single exact\-zero enrichment value appears in the detached horizontal strip\. Panels are equal\-count extraction\-difficulty tertiles\. Color denotes overall difficulty\.
##### Citation volume and source ecology

A submitted collection can contain many records while repeatedly relying on the same pages\. For each rollout, URL nonreuse is the number of unique cited URLs divided by the number of judged records\. A value of one means that every record uses a distinct URL; a value of0\.10\.1corresponds to roughly one unique URL per ten records\. We average this ratio across available rollouts\.

We define effective citation volume as required records multiplied by the square root of URL nonreuse\. Counting every required record fully would ignore citation reuse, while counting only unique URLs would ignore the work required to attach and validate the same page against multiple records\. The square root offers a middle ground that we found well behaved in practice\. Domain dispersion is unique domains divided by unique URLs, averaged across the same rollouts\.

Figure[15](https://arxiv.org/html/2608.14747#Ax1.F15)places this effective volume against required members\. The ratio guides therefore read as effective records per member\. The median is1\.971\.97effective records per member, with the central90%90\\%of tasks spanning0\.730\.73–5\.925\.92\. Domain dispersion varies largely independently of scale: its median is53\.4%53\.4\\%, while the 5th and 95th percentiles are5\.0%5\.0\\%and86\.9%86\.9\\%\. The distribution\-shaped legend displays that concentration\.

![Refer to caption](https://arxiv.org/html/2608.14747v1/EffectiveCitationVolume.png)Figure 15:Individual\-citation view of effective volume and source ecology over 500 tasks \(log–log\)\. Effective records equal required records times the square root of average URL nonreuse\. Dashed guides mark constant effective records per required member\. Point color denotes average domain dispersion; the sidebar shows its smoothed task distribution\.
##### Trajectory effort and execution

Citation counts describe the submitted collection, not the search process that produced it\. We call a URL*surfaced*when the agent interacted with it through any tool call, whether or not the URL appeared in the final submission\. Figure[16](https://arxiv.org/html/2608.14747#Ax1.F16)uses one consistently instrumented authoring trajectory per task and retains the same 490 non\-stub trajectories used in the surfaced\-URL distribution analysis; ten trajectories with fewer than 80 surfaced URLs are excluded\. Task effort is the average cost across available rollouts\. Because cost is multiplicative and strongly skewed, the mean is taken in log space and exponentiated\.

The median retained trajectory surfaces 496 unique URLs, while the median task mean cost is $4\.00\. Exploration scope and effort are positively but noisily associated \(r=0\.280r=0\.280in log–log space\)\.

![Refer to caption](https://arxiv.org/html/2608.14747v1/EffortExploration.png)Figure 16:Trajectory view of exploration scope and effort over 490 non\-stub authoring trajectories \(log–log\)\. Surfaced URLs include every unique URL touched through agent tool calls, not only submitted citations\. Mean task cost is the geometric mean across available positive\-cost rollouts\. Point color denotes overall task difficulty; the dashed line is a least\-squares fit in log space\.

### JExemplar Task Coverage

The authoring pipeline supplies a broader set of vetted guidance tasks to the author, critic, and linter\. The five tasks profiled here are a deliberately varied subset that makes their design rationale explicit\. They provide orientation, not templates: during design, the author can map a seed to a nearby exemplar archetype but may not copy its structure\. Table[13](https://arxiv.org/html/2608.14747#Ax1.T13)summarizes the distinct benchmark shapes covered by this profiled subset, followed by task cards describing the requested records and claims\.

Table 13:Coverage summary for the five exemplar tasks\. The sketches intentionally abstract away the full package files, which are released in the repository\.gpu\_benchmarksTask:For at least 20 GPUs and 10 games per GPU, find a source\-backed FPS result \(one URL per GPU–game pair\) from a first\-hand benchmark or review page\. Two enrichment subtasks require each GPU’s current retail price and each game’s public review rating, with three URLs per item\. Record:an FPS figure for one GPU–game pair, backed by a benchmark page\. Records required:290 \(200 cells\+\+60 price\+\+30 review\)\. Claims per record: 1\. The benchmark is for the claimed GPU model —*proof: the benchmark page* 2\. The benchmark is for the claimed game title, a real released game —*same page* 3\. The page confirms a concrete FPS score —*same page* 4\. Per GPU, its current retail price; per game, its review rating —*proof: three retailer or review pages per item \(subtasks with corroboration\)*

bachir\_doha\_brand\_competitorsTask:Identify 100\+ dessert brands operating in Doha or elsewhere in Qatar \(ice cream, gelato, frozen dessert, sweets, bakery dessert, or dessert café\) and cover four analysis facets per brand—owned social identity, customer sentiment, delivery commerce, and market positioning—with one focused source per facet\. Record:one facet finding for one brand, backed by a facet\-appropriate page\. Records required:400 \(100 brands×\\times4 facets\)\. Claims per record: 1\. The brand is a real, locally operating, in\-scope Qatar brand —*validity* 2\. The page ties the brand to the Qatar market —*proof: the facet page* 3\. The page genuinely plays the claimed facet’s source role \(a review page cannot serve*market positioning*\) —*dispatch\-routed check* 4\. The page exposes a focused finding for that facet —*same page*

adult\_guardianship\_programsTask:For all 30 listed jurisdictions, including the District of Columbia, identify the public or last\-resort adult\-guardianship arrangement \(one URL each\)\. A legislation subtask then finds five legal signals in each of five comparison areas per jurisdiction: appointment basis, decision\-support alternatives, selection priority, oversight, and rights review\. Record \(root\):a jurisdiction’s institutional fallback arrangement, backed by a statute, agency, or program page\. Records required:780 \(30 root\+\+30×5×530\\times 5\\times 5subtask\)\. Claims per record: 1\. The named arrangement is the jurisdiction’s institutional fallback for adults needing a guardian of last resort —*proof: the cited page* 2\. The jurisdiction is one of the 30 items in the closed list —*closed\-set validity* 3\. \(subtask\) each legal signal’s cited section fits its comparison area, on an authoritative legal surface —*proof: own page per signal, dispatch\-specific*

devtools\_partnershipsTask:For at least 100 developer\-tools or DevOps companies, find three advertised partners whose relationship is acknowledged in both directions\. For every ordered \(company,other\_company\) pair, provide both a*quote*\(the company naming the partner\) and a*backquote*\(the partner meaningfully acknowledging the company\), with one URL for each direction\. Record:one direction of one partnership—a page on the referencing party’s own surface naming the counterpart\. Records required:600 \(100×3×2100\\times 3\\times 2\)\. Claims per record: 1\. The page is on a surface officially controlled by the referencing party —*proof: the page \(incl\. its host\)* 2\. It names the opposite party of the pair —*same page* 3\. The reference is substantive for its type—a backquote must show meaningful reciprocity, not a logo wall or name\-drop —*per\-type check* 4\. The counterpart is a genuine, distinct company \(not an alias or the same corporate family\) —*validity*

llm\_presenceTask:Identify 10\+ LLM\-producer companies and build a community\-sentiment panel: for each company and each month in a fixed 12\-month window, provide dedicated first\-hand discussion or impression pages from three different sites, with two URLs per \(company, month, site\) cell\. Record:one standalone community\-discussion page about one company in one month on one site\. Records required:720 \(10×12×3×210\\times 12\\times 3\\times 2\)\. Claims per record: 1\. The page is clearly dated to the target month \(in\-content dates, post timestamps, or thread markers\) —*proof: the page* 2\. It is on the claimed site —*same page \(incl\. host\)* 3\. It is a standalone, first\-hand impression or discussion of the company—not a landing page, bulk thread view, or search\-results display —*same page* 4\. Two independent such pages per cell —*corroboration \(k=2k\{=\}2\)*

### KStylized Metric Rollup Algorithm

Metric computation recursively follows the task structure \(Section[3](https://arxiv.org/html/2608.14747#S3)\)\.*Criteria*supply the leaf score selected from the two verdicts defined in Section[5](https://arxiv.org/html/2608.14747#S5)\.*Topology*specifies which levels exist, what volume each requires, and where subtasks compose\.*Identity*determines which submitted values represent the same entity within each level\. The rollup is parameterized by three choices: aggregation \(*precision*or*recall*\), continuity \(*soft*or*hard*\), and verdict\. A subtask score is multiplied into the matching parent entity’s score using the same choices and recursive procedure; a missing subtask value contributes zero\.

records=task\.records\.narrow\(identifiers\)

ifnotrecords:

return0

iflevel\+1==len\(task\.keys\):

returnrecords\[0\]\.judgment\[params\.verdict\]\#leafjudgment

next\_key=task\.keys\[level\+1\]\.name

values=\{r\[next\_key\]:r\["cluster\_id"\]forrinrecords\}

c\_scores=\[

score\(task,level\+1,identifiers\|\{next\_key:v\},params\)

forvinvalues

\]

ifparams\.aggregation=="recall":

volume=task\.keys\[level\+1\]\.volume

clusters=duplicates\(values,of=next\_key\)

c\_scores=pad\(top\(take\_worst\(c\_scores,clusters\),volume\),0,volume\)

s=mean\(c\_scores\)

iflevel==\-1:\#thresholdingandcompositiondon’tapplyatoverall\-taskavg

returns

ifparams\.continuity=="hard":

s=floor\(s\)

curr\_key=task\.keys\[level\]\.name

sub\_scores=\[

score\(sub,0,\{curr\_key:identifiers\[curr\_key\]\},params\)

forsubintask\.subtasks\(level\)

\]

returns\*product\(sub\_scores\)

metric=score\(root\_task,\-1,\{\},params\)

Stylized metric rollup\. The released task packages include the production implementation\.
### LWorked Rollup Walkthrough

We illustrate the rollup with a toy composite task scored end to end by the released grader and rendered in the submission viewer included with each task package\. The task requests three countries, two cities with more than one million inhabitants per country, and one supporting page per city—country \(3\)→\\rightarrowcity \(2\)→\\rightarrowurl \(1\)\. It adds three subtasks: a per\-cityheritage\_site \(2\)→\\rightarrowurl \(1\)branch requesting two UNESCO\-linked heritage sites, plus two per\-country URL\-only branches requesting evidence of current or historical female leadership and of hosting a worldwide sporting event\. Score indicators show hard and soft precision and recall underverdict\_full\. The submission covers the three countries at deliberately varied levels of completeness and obtains soft F10\.570\.57and hard F10\.330\.33\(Figure[17](https://arxiv.org/html/2608.14747#Ax1.F17)\)\. Figures[18](https://arxiv.org/html/2608.14747#Ax1.F18)–[20](https://arxiv.org/html/2608.14747#Ax1.F20)expand the three country subtrees\.

The Turkey subtree isolates the two identity effects summarized in Table[8](https://arxiv.org/html/2608.14747#S6.T8): duplicate collapse reduces the distinct count, and take\-worst can lower the retained cluster score before canonical identities match main\-task and subtask evidence branches\.

![Refer to caption](https://arxiv.org/html/2608.14747v1/figures/RollupToyOverview.png)Figure 17:Overall score for the toy composite task\. France is complete, India receives partial credit, and Turkey fails a required subtask, yielding soft F10\.570\.57and hard F10\.330\.33\.![Refer to caption](https://arxiv.org/html/2608.14747v1/figures/RollupToyFrance.png)Figure 18:France is fully complete: both qualifying cities have two heritage sites, and both country\-level subtasks pass\. All displayed precision and recall values equal one\.![Refer to caption](https://arxiv.org/html/2608.14747v1/figures/RollupToyIndia.png)Figure 19:India illustrates partial recall and surplus handling\. Delhi supplies one of two required heritage sites, so zero\-padding leaves soft recall at0\.50\.5while soft precision remains11\. Three cities are supplied for a required count of two; Bangalore’s empty heritage subtask gives it a zero composed score, so it is excluded when recall retains the two highest\-scoring cities\.![Refer to caption](https://arxiv.org/html/2608.14747v1/figures/RollupToyTurkey.png)Figure 20:Turkey illustrates identity collapse and subtask gating\. At the heritage level, take\-worst retains an ambiguous zero\-scoring citation when it merges the two Selimiye Mosque variants, reducing Istanbul’s recall\. The city\-level aliasesIstanbulandIstanbul, Türkiyethen merge, propagating the lower score\. The worldwide\-sporting\-event subtask also fails because its evidence concerns a continental final, so multiplicative composition sets the country score to zero\.
### MOutput\-Delivery Ablation

The output ablation compares solutions delivered through sandbox files, a file\-sharing tool, or output tokens\. All six rows use the same 45 tasks and the same score and operational columns as the full sweep\. These are independent rollouts rather than paired deterministic executions, so the differences are descriptive associations with both the delivery configuration and the particular rollout\.

Table 14:Score comparison for the matched 45\-task output\-delivery ablation at each system’s selected configuration\. Bold marks the best value in each numeric column and underline marks the second\-best distinct value\.Table 15:Operational statistics for the same matched output\-delivery ablation as Table[14](https://arxiv.org/html/2608.14747#Ax1.T14)\. Cost and token totals are divided by 45 scheduled tasks; median and P90 are solve\-stage latencies over tasks with valid timestamps\. In M and Out k denote millions and thousands of tokens\. Bold marks the best value and underline marks the second\-best distinct value, using unrounded values; higher is better for completion and lower is better for resource use\.Output\-token runs have lower soft F1 than file\-delivery runs for Perplexity \(0\.3970\.397versus0\.2380\.238\), Anthropic \(0\.2620\.262versus0\.2020\.202\), and OpenAI \(0\.1530\.153versus0\.1220\.122\); hard F1 shows the same ordering \(0\.1560\.156versus0\.1210\.121,0\.0990\.099versus0\.0800\.080, and0\.0730\.073versus0\.0680\.068\)\. The absolute soft F1 loss decreases monotonically with the file\-delivery score across these three systems \(0\.1590\.159,0\.0600\.060, and0\.0310\.031\)\. This three\-system association is descriptive rather than causal, but it suggests that output\-only systems may be somewhat disadvantaged while also making it unlikely that delivery alone explains the much larger gaps in the main ranking\. Operational differences vary by system: output\-token runs are slightly more expensive for Perplexity and OpenAI and less expensive per scheduled task for Anthropic, whose output\-token run also produces fewer metric\-bearing trials \(42/4542/45versus45/4545/45\)\.

### NSuitability for Reinforcement Learning

WANDR is released primarily as an evaluation benchmark\. Its structure nevertheless offers several properties relevant to future training splits and harnesses\.

##### Scalable task supply and diversity

The semi\-automated task pipeline avoids exhaustive answer annotation through a reusable author–critic–linter process and reference\-free verification\. It can produce new tasks and controlled siblings across domains, hierarchy shapes, evidence requirements, traps, and breadth/depth settings, while curation steers the resulting distribution \(Section[4](https://arxiv.org/html/2608.14747#S4)\)\. This creates a path to training\-scale data with broader variation than repeated sampling of the 500 released tasks\. For RL\-scale task supply, authoring and admission can run fully automatically, including feasibility and judge\-quality gates; periodic human audits can monitor distribution drift without sitting on the generation path\. To preserve evaluation validity, RL should use newly generated or separately held\-out task packages, not the released benchmark split\.

##### A natural curriculum

Workload is explicit in each task’s key hierarchy: every level declares a required count\. Holding the task’s semantic criteria fixed while lowering one or more counts produces a smaller instance—for example, requiring 5 qualifying entities rather than 20, or one corroborating source rather than three—and the counts can be increased across epochs toward the full task\. The same mechanism can stage breadth and depth separately, allowing a curriculum to move from reliable single\-record evidence construction to sustained collection building\. Count reduction controls workload rather than guaranteeing monotonic semantic difficulty, and each generated variant should still pass the normal feasibility and judge checks\.

##### Dense, decomposable reward

A long rollout need not receive only a terminal all\-or\-nothing score\. Each submitted record receivesverdict\_retrievalandverdict\_full, and the score tree aggregates those verdicts by entity and hierarchy level into soft and hard precision, recall, and F1\. Soft metrics reward correct partial progress when the requested collection is incomplete; hard metrics preserve pressure to complete whole members; and the gap between the two verdicts separates page\-level task satisfaction from complete record construction\. A trainer can therefore assign outcome rewards at record and branch granularity or use universal and task\-specific sub\-verdicts as auxiliary signals while retaining task\-level F1 as the objective\. This supervision is denser than a single exact\-match reward, though moving from outcome\-level to process supervision would require nontrivial trajectory backtracking and heuristics for assigning credit from individually graded final records to the actions responsible for them\.

##### Amortized batch grading

Reference\-free verification avoids constructing and maintaining exhaustive gold collections\. At runtime, expensive grading remains record\-local until canonicalization, deduplication, and score rollup\. The released grader accepts multiple task roots, flattens their records into one streaming queue graph, and executes fetch, triage, canonicalization, deduplication, and judgment with stage\-specific concurrency\. Fetches are batched, and queue nodes use both in\-memory request coalescing and persistent caches keyed to the smallest reusable unit\. Records citing the same URL can share one fetch result; identical triage, canonicalization, and judgment work can also be reused when task semantics and inputs match\. This structure aligns with RL batches, where multiple tasks and rollouts may revisit the same pages and entities\. Overlap among rollout records can therefore amortize grading work, while persistent caches limit resumed runs to missing work\.

Table 16:RL\-oriented view of WANDR’s task and grader design\.