Autoresearch for Marketplace Catalogs: From Legacy Forms to AI-Native Matching

# Autoresearch for Marketplace Catalogs:From Legacy Forms to AI-Native Matching
Source: [https://arxiv.org/html/2609.00274](https://arxiv.org/html/2609.00274)
## Autoresearch for Marketplace Catalogs: From Legacy Forms to AI\-Native MatchingNote:Accepted at the KDD Workshop on Evaluation and Trustworthiness of Agentic AI \(KDD 2026\)\.

Conference:KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI; August 2026; Jeju, Korea,Hojat Abdolanezhademail:[habdolanezhad@thumbtack\.com](mailto:[email protected])Affiliation:Thumbtack,San Francisco,CA,USA,Daniel Capoemail:[dcapo@thumbtack\.com](mailto:[email protected])Affiliation:Thumbtack,San Francisco,CA,USA,Sang Su Leeemail:[psulee@thumbtack\.com](mailto:[email protected])Affiliation:Thumbtack,San Francisco,CA,USA,Shishir Dashemail:[shishirdash@thumbtack\.com](mailto:[email protected])Affiliation:Thumbtack,San Francisco,CA,USAandVijay Anand Raghavanemail:[vraghavan@thumbtack\.com](mailto:[email protected])Affiliation:Thumbtack,San Francisco,CA,USA

© none

###### Abstract\.

Two\-sided service marketplaces that match consumers to service providers are transitioning from deterministic request\-form intake to an AI\-native probabilistic matching system, enabled by recent advances in large language models \(LLMs\) that can infer user intent, contextual preferences, and latent constraints from natural language instructions\. As marketplaces increasingly rely on inferred intent rather than fixed\-form fields, these platforms must regenerate the provider\-side preference taxonomy that underwrites matching, search, and pricing: a set of provider attributes and preferences interpretable to service providers while remaining useful signals for marketplace decision\-making\. We present an autoresearch loop that generates this taxonomy one occupation at a time\. The system has been deployed in production at a major U\.S\. consumer services marketplace since April 2026 across 132 occupations\. Instead of constructing a single global hierarchy, the loop treats each occupation as an independent generation problem and runs iterative propose\-evaluate\-keep refinement cycles\. Each candidate tag set is scored using a recalibrated six\-rubric LLM\-as\-judge framework, producing a composite score of up to 15\. A 7\-critic panel, each with distinct personas, contributes weighted penalties to produce an adjusted score, with no hard vetoes\. A separate LLM\-based parity\-mapping stage maps legacy request\-form Q&A pairs back to the generated taxonomy, producing both a coverage signal and a scalable interface for human quality assurance\. The loop migrates legacy structured Q&A by first inferring the underlying provider attribute that each question was intended to measure, rather than performing literal question\-to\-tag translation\. Unlike prior autonomous taxonomy\-generation systems, our approach \(i\) takes the per\-occupation preference\-tag catalog \(not the hierarchy of occupations\) as the unit of optimization, \(ii\) generates each occupation’s tags independently in parallel, \(iii\) evaluates with marketplace\-grounded critics contributing weighted penalties \(no hard vetoes\) to an adjusted score, and \(iv\) treats legacy\-Q&A\-to\-modern\-tag migration as a distinct pipeline component rather than a downstream side effect\. We report deployment results from a 14\-day post\-launch production cohort \(1,840 enrolled pros, 9\.3M filter evaluations\) that surfaces a concrete catalog\-hygiene gap that a global\-build approach would have masked\.

###### Keywords:

LLM autoresearch, taxonomy generation, marketplace catalog, multi\-critic evaluation, LLM\-as\-judge, deployed system

## 1\.Introduction

A marketplace catalog provides the shared representation through which consumer requests, provider preferences, and job attributes are interpreted\. Matching, search, and pricing all depend on this common language\. However, in service marketplaces that have historically relied on structured question\-and\-answer \(Q&A\) forms, the catalog is often implicit, distributed across hundreds of category\-specific schemas rather than represented as a unified structure\. When the marketplace transitions to AI\-native probabilistic matching \(Figure[1](https://arxiv.org/html/2609.00274#S1.F1)\), this implicit catalog becomes a liability\. Matching models require structured signals, pricing models require job attributes that transfer across categories, and marketplace participants need a coherent interface for expressing preferences without navigating hundreds of independent schemas\.

We frame this problem as a catalog*reconstruction*, not*creation*, since the catalog is already implicitly defined in the legacy system, albeit distributed, through Q&A pairs\. The task is to infer the latent attributes encoded across Q&A pairs and reorganize them into an explicit representation suitable for AI\-native marketplace interactions\. Because the implicit legacy catalog is organized around distinct service domains, each with its own vocabulary, preference structure, and matching semantics, we decompose catalog reconstruction into a collection of smaller*autoresearch*problems\([Karpathy, 2026](https://arxiv.org/html/2609.00274#bib.bib9)\)rather than a single global taxonomy\-generation task\. This design contrasts with recent hierarchical taxonomy\-generation systems, which optimize coherent occupation hierarchies\([Li et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib10)\)or corpus\-level research taxonomies\([Kargupta et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib8)\)\. Our goal is not to build a single hierarchy over all domains, but to reconstruct preference catalogs whose semantics remain meaningful within each service domain in a two\-sided marketplace\.

![Refer to caption](https://arxiv.org/html/2609.00274v1/legacy_rfqa_matching.png)\(a\)Legacy Q&A matching
![Refer to caption](https://arxiv.org/html/2609.00274v1/nl_search_matching.png)\(b\)AI\-native matching

Figure 1\.Evolution from legacy request\-form matching to AI\-native matching: Legacy matching \([1\(a\)](https://arxiv.org/html/2609.00274#S1.F1.sf1)\) relies on an implicit catalog embedded in occupation\-specific Q&A forms\. AI\-native matching \([1\(b\)](https://arxiv.org/html/2609.00274#S1.F1.sf2)\) requires an explicit catalog that translates consumer intent and provider preferences into a shared representation for downstream marketplace systems\.The key contributions of this paper are as follows:

1. \(1\)Catalogs for marketplaces \-We study a catalog\- reconstruction problem distinct from occupation\-hierarchy construction\([Li et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib10)\), text\-label taxonomy induction\([Wan et al\., 2024](https://arxiv.org/html/2609.00274#bib.bib13)\), research\-corpus taxonomy adaptation\([Kargupta et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib8)\), and product\-schema modeling\([Huang et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib6)\)\. The generated unit is a provider\-facing*preference tag*: a primitive that providers toggle with in production and that the downstream marketplace systems use\.
2. \(2\)Independent autoresearch loops \-We decompose reconstruction into independent autoresearch jobs, one per service occupation, each with its own*propose\-evaluate\-keep*loop\. This avoids forcing all service domains into a single global hierarchy and limits semantic interference between occupations with superficially similar but operationally distinct attributes\.
3. \(3\)Marketplace\-grounded automated evaluation \-We combine a recalibrated multi\-rubric LLM\-as\-judge\([Zheng et al\., 2023](https://arxiv.org/html/2609.00274#bib.bib15)\)with a seven\-persona critic panel that contributes weighted penalties to the generated catalog\. This aligns evaluation with marketplace objectives rather than generic measures of taxonomy quality\.
4. \(4\)Legacy Q&A parity mapping \-We introduce a parity\-mapping stage that maps the reconstructed catalog back to legacy request\-form Q&A pairs\. This stage measures whether the new catalog covers the operational distinctions encoded in the legacy system and produces a human\-reviewable artifact for validating migration from structured forms to provider\-preference tags\.

Success for this system means the marketplace can retire its legacy request forms without losing the preference structure they encoded\. The most direct evidence is reconstruction fidelity\. Across the reconstructed catalog,73\.3%73\.3\\%of legacy answers map directly to a tag in the regenerated catalog; most of the remainder are intake\-only answers \(sizes, ranges, “flexible”\) that a preference schema should not carry, excluded as acceptable non\-tags \(20\.8%20\.8\\%of all answers\)\. Only6\.0%6\.0\\%of legacy answers are*regrettable*misses, real preferences left uncovered\. Measured against the preferences that should map, pooled coverage is92\.5%92\.5\\%, the per\-occupation median is100%100\\%\(mean97\.8%97\.8\\%\)\. What the catalog drops is, by construction, intake detail; the screening preferences pros act on are almost entirely preserved\. Together with production scale \(132 occupations live, new ones onboarded on demand within hours\) and mandatory human sign\-off before deployment, this fidelity is what makes retiring the request forms defensible\.

The propose\-evaluate\-keep loop builds on prior work in iterative LLM refinement, principle\-guided critique, and autoresearch systems\([Madaan et al\., 2023](https://arxiv.org/html/2609.00274#bib.bib11);[Bai et al\., 2022](https://arxiv.org/html/2609.00274#bib.bib3);[Karpathy, 2026](https://arxiv.org/html/2609.00274#bib.bib9)\); here, we operationalize it for production catalog reconstruction in a two\-sided service marketplace\. We report results from a production deployment spanning 132 service occupations as of April 2026, with on\-demand onboarding of new occupations as the marketplace expands\. The remainder of the paper describes the marketplace setting \(§\\lx@sectionsign[2](https://arxiv.org/html/2609.00274#S2)\), other related works \(§\\lx@sectionsign[3](https://arxiv.org/html/2609.00274#S3)\), the autoresearch system used for catalog reconstruction \(§\\lx@sectionsign[4](https://arxiv.org/html/2609.00274#S4)\), its deployment in production \(§\\lx@sectionsign[5](https://arxiv.org/html/2609.00274#S5)\), and the empirical findings \(§\\lx@sectionsign[6](https://arxiv.org/html/2609.00274#S6)\) that emerge from operating the system at scale\.

## 2\.Background

In this section, we describe the marketplace setting and the production context in which the system is deployed \(§\\lx@sectionsign[2\.1](https://arxiv.org/html/2609.00274#S2.SS1)\), as well as the terminology needed to understand the catalog\-reconstruction problem \(§\\lx@sectionsign[2\.2](https://arxiv.org/html/2609.00274#S2.SS2)\)\.

### 2\.1\.Marketplace context

The deploying organization is a major U\.S\. online services marketplace that connects consumers with local service professionals \(*pros*\) across home, wellness, and event categories, including plumbing, house cleaning, photography, landscaping, and roughly 130 others\. Pros specify targeting preferences that determine which consumer leads they are eligible to receive and purchase\. The system described in this paper is part of a 2026 pro\-side marketplace redesign that replaces rigid binary targeting defined through structured Q&A with probabilistic matching driven by a structured provider\-side preference taxonomy\([Einav et al\., 2016](https://arxiv.org/html/2609.00274#bib.bib5)\)\.

### 2\.2\.Terminology

The legacy taxonomy is organized as a hierarchy of occupations, categories, and category\-specific request\-form Q&A schemas\. The reconstructed catalog contains two tag classes\.

1. \(1\)*Canonical tags*determine eligibility: a provider must possess the canonical tag associated with a category for the strict\-match filter to consider that provider eligible for leads in that category;
2. \(2\)*Specialty preference tags*allow providers to refine which leads they wish to receive within categories for which they are already eligible

Section[6](https://arxiv.org/html/2609.00274#S6)reports two production findings that depend on this distinction: a catalog\-hygiene gap in which deprecated canonical tags were still emitted by request\-time enrichment, and a deployment\-side seeding gap in which40\.66%40\.66\\%of*filter evaluations*failed because the canonical tag associated with the consumer request was not present on the provider profile\. A*filter evaluation*is a single matchmaker\-side check, for one candidate pro and one incoming consumer request, to determine whether the pro’s profile contains the request’s required canonical tag\.

## 3\.Related Work

Our work sits at the intersection of automated taxonomy construction, agentic schema generation, autoresearch systems, prompt optimization, and LLM\-based evaluation\.

#### Hierarchical taxonomy construction\.

CLIMB\([Li et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib10)\)built occupation hierarchies using a global semantic clustering to distill core occupations, followed by a reflection\-based multi\-agent system to iteratively build a coherent hierarchy\. Our work differs in both the unit of generation and the reconstruction objective: CLIMB generated occupations within a hierarchy, whereas we reconstruct provider\-preference tags within an occupation\. Furthermore, our catalogs are generated independently for each occupation and must support migration from legacy request\-form Q&A systems through parity mapping\. TnT\-LLM\([Wan et al\., 2024](https://arxiv.org/html/2609.00274#bib.bib13)\)used LLMs to induce and iteratively refine label taxonomies from unstructured text, while TaxoAdapt\([Kargupta et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib8)\)dynamically adapted multidimensional taxonomies to evolving scientific corpora through iterative hierarchical classification\. Both systems inform the generation\-and\-refinement paradigm we adopt\. However, their objective is taxonomy induction for corpus organization and classification, whereas ours is catalog reconstruction for marketplace decision\-making\.

#### Schema and catalog generation\.

AttributeForge\([Huang et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib6)\)automated end\-to\-end product\-schema modeling using a large collection of specialized LLM agents, together with automated evaluation and repair\. It is the closest prior work to ours in terms of production\-scale catalog reconstruction\. The key distinction is the object being modeled: AttributeForge generated product attributes for e\-commerce catalogs, whereas we reconstruct provider\-preference catalogs for service marketplaces, where matching depends on both provider capabilities and provider screening preferences\.

#### Autoresearch, self\-refinement, and prompt optimization\.

Our "propose, evaluate, keep" loop draws on a broader family of iterative LLM optimization methods, including Self\-Refine\([Madaan et al\., 2023](https://arxiv.org/html/2609.00274#bib.bib11)\), Constitutional AI\([Bai et al\., 2022](https://arxiv.org/html/2609.00274#bib.bib3)\), multi\-agent debate\([Du et al\., 2024](https://arxiv.org/html/2609.00274#bib.bib4)\), GEPA\([Agrawal et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib2)\), and prompt\-optimization frameworks such as DSPy and MIPROv2\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2609.00274#bib.bib12)\)\. Like DSPy, we treat prompts as optimizable artifacts and evaluation as the search signal\. However, rather than optimizing a single prompt against a fixed benchmark or held\-out metric, our framework optimizes occupation\-specific catalog\-generation prompts whose outputs are structured marketplace catalogs\.

Legacy RF dataDoD\_\{o\}GeneratorJudge \(score /15\)PMProTaxonomyAdversarialReasoningCoherenceParity7\-persona criticpanel \(parallel\)Aggregate penalties\(set\-level \+ critic\)Improved?Editor:revise promptiterateBest catalogT⋆T^\{\\star\}on plateauParity mapping\(tags↔\\leftrightarrowRF Q&A\)coverage \+ human QAFigure 2\.Per\-occupation autoresearch loop \(conceptual overview; see Algorithm[1](https://arxiv.org/html/2609.00274#alg1)for the precise procedure\)\. The occupation’s legacy RF dataDoD\_\{o\}and current best promptpop\_\{o\}feed a Generator that produces a candidate tag set\. Each candidate is scored by theℰ\\mathcal\{E\}six\-rubric LLM\-as\-judge; the seven\-persona critic panel then runs in parallel, and its feedback is aggregated\. The gate decides whether the editor’s single proposed prompt edit improves on the current best and is kept\. On a plateau or budget exhaustion, the best catalogT⋆T^\{\\star\}is handed to the post\-loop parity mapping, which links catalog tags to legacy RF Q&A \(many\-to\-many\) and produces the artifact humans review\.
#### LLM\-based evaluation and abstraction\.

Our system draws on recent work in both LLM\-based evaluation and abstraction\. The six\-rubric evaluation framework builds on LLM\-as\-judge methodologies\([Zheng et al\., 2023](https://arxiv.org/html/2609.00274#bib.bib15)\)and critique\-based evaluation\([Husain, 2024](https://arxiv.org/html/2609.00274#bib.bib7)\), adapting them to catalog reconstruction through marketplace\-specific criteria derived from production review\. Several rubrics were introduced to capture failure modes that generic taxonomy metrics overlook, such as tags that are semantically coherent but insufficiently specific or interpretable for provider\-facing use\. We also draw inspiration from Step\-Back Prompting\([Zheng et al\., 2024](https://arxiv.org/html/2609.00274#bib.bib14)\)\. Rather than translating legacy request\-form questions directly into tags, we first infer the underlying job attribute each question was intended to capture and use that abstraction as input to catalog generation\.

## 4\.System Design

We formulate catalog reconstruction as an iterative propose\-evaluate\-keep problem \(§\\lx@sectionsign[4\.1](https://arxiv.org/html/2609.00274#S4.SS1)\)\. Candidate catalogs are generated independently for each service domain, evaluated using a rubric\-based LLM judge \(§\\lx@sectionsign[4\.2](https://arxiv.org/html/2609.00274#S4.SS2)\), adjusted through a multi\-persona critic panel \(§\\lx@sectionsign[4\.4](https://arxiv.org/html/2609.00274#S4.SS4)\), and validated against legacy request\-form semantics through parity mapping \(§\\lx@sectionsign[4\.6](https://arxiv.org/html/2609.00274#S4.SS6)\)\.

### 4\.1\.Per\-Occupation Autoresearch Loop

Catalog reconstruction is a cold\-start problem: initially there is no per\-occupation generation prompt, so every occupationo∈Oo\\in O\(\|O\|=132\|O\|=132in the current production deployment\) is seeded with the same baseline promptp\(0\)p^\{\(0\)\}, i\.e\.po⋆←p\(0\)p\_\{o\}^\{\\star\}\\\!\\leftarrow\\\!p^\{\(0\)\}\. We run Algorithm[1](https://arxiv.org/html/2609.00274#alg1)\(illustrated in Figure[2](https://arxiv.org/html/2609.00274#S3.F2)\) independently per occupation\.

At iteration00, the generator𝒢\\mathcal\{G\}applies the baseline prompt to the occupation’s legacy dataDoD\_\{o\}\( RF Q&A schema for each category, % share of recently\-active pros who have currently enrolled a \(q, a\) pair as a preference\) to produce a candidate tag setTT\. The six\-rubric LLM\-as\-judge scores the set,s=ℰ⁡\(T\)∈\[0,15\]s=\\mathcal\{E\}\(T\)\\in\[0,15\], and the seven\-persona critic panel adds weighted penaltiesπk\\pi\_\{k\}, giving the composite scorea=s−∑kwk​πk​\(T\)a=s\-\\sum\_\{k\}w\_\{k\}\\pi\_\{k\}\(T\)\. This becomes the initial best\(po⋆,T⋆,s⋆,a⋆\)\(p\_\{o\}^\{\\star\},T^\{\\star\},s^\{\\star\},a^\{\\star\}\)\. The loop runs a cross\-family model stack: the generator𝒢\\mathcal\{G\}and judgeℰ\\mathcal\{E\}on one family \(GPT\-5\-4 / GPT\-5\-4\-mini\), the seven critics, editor, and parity mapper on another \(Claude Sonnet 4\.6\), so the models that propose and score do not share biases with the models that critique and mutate \(§[4\.2](https://arxiv.org/html/2609.00274#S4.SS2), §[7](https://arxiv.org/html/2609.00274#S7)\)\.

Each subsequent iteration proposes and tests a single prompt edit\. The editorℛ\\mathcal\{R\}makes one targeted change to the current best promptpo⋆p\_\{o\}^\{\\star\}, focused on a single weak quality dimension: it ranks the sixℰ\\mathcal\{E\}rubrics together with an injected parity\-coverage dimension by mean score and selects the lowest\-scoring dimension not targeted in the previous two iterations, avoiding fixation on any one rubric\. The editor conditions on the weakest tag examples in that dimension, the score trends across iterations, and the log of previously accepted and rejected edits\. The revised prompt is regenerated and rescored; as a per\-iteration cost saver, the critic panel is invoked only when the candidate’sℰ\\mathcal\{E\}score is not already below the best \(s≥s⋆s\\geq s^\{\\star\}\), since a lower\-ℰ\\mathcal\{E\}candidate is very unlikely to overcome the best composite score\. The edit is accepted, thus becoming the new best,*iff*its composite score improves,a\>a⋆a\>a^\{\\star\}; otherwise the prompt reverts topo⋆p\_\{o\}^\{\\star\}\. The loop runs for a fixed budget ofBBiterations \(default55\)\. The supplementary material traces one iteration end\-to\-end on the*Accounting*occupation\.

On termination, the parity mapping𝒫⁡\(T⋆,Qo\)\\mathcal\{P\}\(T^\{\\star\},Q\_\{o\}\)links the tags of the best catalogT⋆T^\{\\star\}to the legacy RF Q&AQoQ\_\{o\}as a many\-to\-many relationM⋆⊆T⋆×QoM^\{\\star\}\\subseteq T^\{\\star\}\\times Q\_\{o\}, leaving residual setsUQ⋆U\_\{Q\}^\{\\star\}\(RF Q&A covered by no tag—typically intake\-specific size/range/flexibility questions\) andUT⋆U\_\{T\}^\{\\star\}\(tags covered by no Q&A\)\. We emitM⋆M^\{\\star\}as a JSON mapping and a Google Sheet for human review before deployment\.

Algorithm 1Occupation\-Local Autoresearch1:legacy data

DoD\_\{o\}with RF Q&A

QoQ\_\{o\}, shared seed prompt

p\(0\)p^\{\(0\)\}, budget

BB, critic weights

ww
2:best prompt

po⋆p\_\{o\}^\{\\star\}, best catalog

T⋆T^\{\\star\}, parity relation

M⋆M^\{\\star\}, unmapped sets

UQ⋆,UT⋆U\_\{Q\}^\{\\star\},U\_\{T\}^\{\\star\}
3:

po⋆←p\(0\),T⋆←𝒢⁡\(po⋆,Do\),a⋆←ℰ⁡\(T⋆\)−ρ⁡\(T⋆\)−∑kwk​πk​\(T⋆\)p\_\{o\}^\{\\star\}\\\!\\leftarrow\\\!p^\{\(0\)\},\\;\\;T^\{\\star\}\\\!\\leftarrow\\\!\\mathcal\{G\}\(p\_\{o\}^\{\\star\},D\_\{o\}\),\\;\\;a^\{\\star\}\\\!\\leftarrow\\\!\\mathcal\{E\}\(T^\{\\star\}\)\-\\rho\(T^\{\\star\}\)\-\{\\textstyle\\sum\_\{k\}\}w\_\{k\}\\pi\_\{k\}\(T^\{\\star\}\)
4:for

t=1t=1to

BBdo

5:

po←ℛ⁡\(po⋆\),T←𝒢⁡\(po,Do\),a^←ℰ⁡\(T\)−ρ⁡\(T\)p\_\{o\}\\leftarrow\\mathcal\{R\}\(p\_\{o\}^\{\\star\}\),\\;\\;T\\leftarrow\\mathcal\{G\}\(p\_\{o\},D\_\{o\}\),\\;\\;\\hat\{a\}\\leftarrow\\mathcal\{E\}\(T\)\-\\rho\(T\)⊳\\trianglerightpre\-critic score;ρ\\rho= set\-level penalty \(parity coverage \+ proliferation\)

6:if

a^\>a⋆\\hat\{a\}\>a^\{\\star\}then⊳\\trianglerighta^≥a\\hat\{a\}\\geq a, so a candidate witha^≤a⋆\\hat\{a\}\\leq a^\{\\star\}provably cannot win; skip critics

7:

a←a^−∑kwk​πk​\(T\)a\\leftarrow\\hat\{a\}\-\{\\textstyle\\sum\_\{k\}\}w\_\{k\}\\pi\_\{k\}\(T\)
8:if

a\>a⋆a\>a^\{\\star\}then

9:

\(po⋆,T⋆,a⋆\)←\(po,T,a\)\(p\_\{o\}^\{\\star\},T^\{\\star\},a^\{\\star\}\)\\leftarrow\(p\_\{o\},T,a\)
10:endif

11:endif

12:endfor

13:

M⋆←𝒫⁡\(T⋆,Qo\)⊆T⋆×QoM^\{\\star\}\\leftarrow\\mathcal\{P\}\(T^\{\\star\},Q\_\{o\}\)\\subseteq T^\{\\star\}\\times Q\_\{o\}⊳\\trianglerightmany\-to\-many tag↔\\leftrightarrowQ&A relation

14:

UQ⋆←Qo∖πQ​\(M⋆\)U\_\{Q\}^\{\\star\}\\leftarrow Q\_\{o\}\\setminus\\pi\_\{Q\}\(M^\{\\star\}\)⊳\\trianglerightunmapped Q&A

15:

UT⋆←T⋆∖πT​\(M⋆\)U\_\{T\}^\{\\star\}\\leftarrow T^\{\\star\}\\setminus\\pi\_\{T\}\(M^\{\\star\}\)⊳\\trianglerighttags with no covering Q&A

16:return

\(po⋆,T⋆,M⋆,UQ⋆,UT⋆\)\(p\_\{o\}^\{\\star\},T^\{\\star\},M^\{\\star\},U\_\{Q\}^\{\\star\},U\_\{T\}^\{\\star\}\)

### 4\.2\.Six\-Rubric LLM\-as\-Judge

The judge modelℰ\\mathcal\{E\}scores every tag \(t∈Tt\\in T\) on six dimensions:

- •Screening vs\. Intake\(0–3\): Does the tag describe a job type the pro might want*more or less of*\(good\), or is it a customer\-intake detail the pro should not screen on \(bad\)? E\.g\. “Tax return preparation” \(screening\) vs\. “Travel range: 15 miles” \(intake\)\.
- •Tag Legibility\(0–3\): Would a working pro understand this tag from the tag text alone? E\.g\. “Appliance repair” \(clear\) vs\. the bare noun “Appliances”
- •Preference Variance\(0–3\): Do real pros split on this tag, or is everyone equally for\-or\-against \(degenerate\)? E\.g\. Accounting pros split on “QuickBooks” proficiency, but nearly all accountants serve the “healthcare industry” \(degenerate\)\.
- •Cross\-Category Consistency\(0–2\): When the same job\-attribute appears in two categories within the occupation, do the tags match? E\.g\. the client\-entity type tagged identically as “S\-Corp” wherever it recurs across the occupation’s categories, not “S\-Corp” in one and “S\-Corporation” in another\.
- •Canonical Coverage\(0 or 2\): Does every category in the occupation have a canonical tag covering it? E\.g\. “Payroll services” tag cleanly covers the*Payroll Services*category, whereas a tag backed by no category scores 0
- •Information Loss\(0–2\): Are the tag’s sources traceable to specific legacy Q&A answers, i\.e\. does it cite the legacy category and answers it consolidates? E\.g\. a tag citinglegacy\_category: Accountingand the “QuickBooks” answer scores 2 vs\. one with only a vague rationale \(00\)\.

ℰ\\mathcal\{E\}scores each tag independently on the six rubrics above, with all tags inTTscored in parallel; the per\-tag composite is the sum of the six rubric scores \(on a/15/15scale\)\. The set\-level score is the mean per\-tag composite reduced by small set\-level deductions for tag proliferation and category\-coverage gaps,s=ℰ⁡\(T\)∈\[0,15\]s=\\mathcal\{E\}\(T\)\\in\[0,15\]; the verdict mix summarizes the per\-tag outcomes\. The judge model is GPT\-5\-4\-mini at temperature zero; the generation model is GPT\-5\-4\. The seven\-persona critic panel, the editor agent, and the final parity\-mapping stage all run on Claude Sonnet 4\.6, a different model family from the generator and the judge\. The choice is deliberate: a same\-family stack would risk the loop reinforcing biases shared across mutation and evaluation, and the cross\-family arrangement is the strongest readily available defense at production scale\.ℰ\\mathcal\{E\}was recalibrated against product\-manager \(PM\) review: bare\-noun tags like “Appliances” originally scored a legibility of33but scored≤1\\leq 1after recalibration, since a working appliance\-repair pro reads “Appliances” as either trivially true or meaningless, not informative\.

### 4\.3\.Set\-Level Penalty \(ρ\\rho\)

Theℰ\\mathcal\{E\}judge scores each tag in isolation, so a catalog of individually high\-scoring tags can still be poor as a*set*\.TTcan be bloated with near\-duplicate siblings, or leave real pro preferences with no corresponding tag\. The set\-level penaltyρ⁡\(T\)\\rho\(T\)corrects for this by deducting from the composite two failures the per\-tag rubrics cannot see:*tag proliferation*\(an excess of sibling tags under a single parent, which fragments the catalog and the pro\-facing UI\) and*coverage gaps*\(legacy preferences inQoQ\_\{o\}which pros both enrolled and deselected recently that no tag captures, including whole categories left untagged;§\\lx@sectionsign[4\.6](https://arxiv.org/html/2609.00274#S4.SS6)\)\. Subtractingρ⁡\(T\)\\rho\(T\)before the critic penalties \(§\\lx@sectionsign[4\.4](https://arxiv.org/html/2609.00274#S4.SS4)\),a=s−ρ⁡\(T\)a=s\-\\rho\(T\), makes the loop optimize for a coherent, well\-covered catalog rather than a collection of locally excellent but globally redundant or incomplete tags\.

### 4\.4\.Seven\-Persona Critic Panel

Afterℰ\\mathcal\{E\}scoring, the full tag set is reviewed by seven critic\-personas running in parallel, each returning a verdict that is reduced to a weighted contribution to the composite score\. Most contributions are penalties; the Reasoning critic can instead award a small bonus for strong rationales\. The exact penalty formulas, caps, and bonus thresholds are tabulated in the supplementary material\.111An earlier design used a hard veto from the Pro Critic\. In early production runs, the veto dominated keep/discard decisions, and the loop stalled on occupations where one persona consistently rejected proposals that other critics rated favorably\. The current design replaces the veto with a weighted penalty capped at2\.52\.5on the composite\-of\-15 scale; low enough that a strong proposal can still be kept over Pro objections, high enough that consistent Pro disapproval pulls the adjusted score below the discard threshold\. The cap and rate were set empirically; the panel module records the change as “softened to weighted penalty, capped at 2\.5\.”The critics are:

- •PM critic: Would a product manager \(PM\) find this tag actionable? Surfaces design\-stage issues \(mutual\-exclusivity, awkward dimensionality\); penalized only when rejections exceed a small free allowance\.
- •Pro critic\(penalty capped at2\.52\.5\): Would a working pro in this occupation find this useful or confusing? The cap prevents a single skeptical persona from dominating the verdict\.
- •Taxonomy critic: Are tags at consistent levels of abstraction, are unions broken into components, are subtypes grouped? Penalized as structural consistency falls below the threshold\.
- •Adversarial critic: Where could the tag set be gamed or misinterpreted to extract leads that the pro cannot deliver? Penalty scales with the assessed risk level and the number of critical failures\.
- •Reasoning\-Quality critic\(bonus\-eligible\): Does the generation explain*why*each tag exists? Strong rationales \(score≥8\\geq 8\) earn a small bonus; weak ones incur a penalty\.
- •Occupation\-Coherence critic: Does the tag set, taken as a whole, describe*this occupation*rather than a generic union of services?
- •Parity critic: Do the generated tags map to a given occupation’s top\-30 most\-deselected legacy answers \(i\.e\., answers active pros have explicitly opted out of\), ranked by distinct\-pro count over a 90\-day window? Its penalty is small by design \(at most∼1\\sim\\\!1point\); the quantitative coverage enforcement lives in the set\-level penaltyρ\\rho\(§[4\.3](https://arxiv.org/html/2609.00274#S4.SS3)\)\. When an occupation has no deselection signal, the critic returns a neutral score and contributes nothing\.

### 4\.5\.Editor Agent

The editorℛ\\mathcal\{R\}is the loop’s mutation operator: at each iteration, it proposes exactly one targeted edit to the current best promptpo⋆p\_\{o\}^\{\\star\}and returns the revised prompt together with a one\-line description of the change\. A key design point is that the editor is driven by the judgeℰ\\mathcal\{E\}, not the critics: the critic penalties gate which prompts survive \(through the compositeaa\), whereas the editor decides*what to change*from theℰ\\mathcal\{E\}signal and the coverage signal alone\.

#### Target\-dimension selection\.

The editor ranks two complementary kinds of signals and targets the weakest\. The first is*per\-tag quality*: the sixℰ\\mathcal\{E\}rubrics, each averaged across the tag set to yield one mean per rubric\. The second is*set\-level coverage*: a single parity\-coverage score for the catalog as a whole\. Coverage cannot be expressed as a seventh rubric, sinceℰ\\mathcal\{E\}scores the tags that exist, whereas coverage is a property of the legacy preferences that have*no*tag, so there is nothing per\-tag to score\. It is therefore injected as a synthetic dimension, giving catalog incompleteness a seat in the ranking alongside tag\-quality weaknesses\. The coverage dimension is a*score*\(higher is better\),2​min⁡\(r\+,r−\)∈\[0,2\]2\\min\(r\_\{\+\},r\_\{\-\}\)\\in\[0,2\], wherer\+,r−∈\[0,1\]r\_\{\+\},r\_\{\-\}\\in\[0,1\]are the fractions of enrolled and deselected preferences the catalog covers; the00–22scaling puts it on the same footing as the rubric means\. This ranking score is used*only*to select the target dimension and is distinct from the set\-level penaltyρ\\rho\(§[4\.3](https://arxiv.org/html/2609.00274#S4.SS3)\), which turns the same coverage signal into points subtracted from the composite\. The editor then targets the lowest\-scoring dimension that has not already been addressed in the previous N iterations \(default N = 2\), a constraint that prevents fixation on a single persistent weakness and forces progress across both quality and coverage rather than over\-optimizing one axis\.

#### Conditioning\.

For the chosen dimension, the editor is given the tags scoring weakest on it \(ℰ\\mathcal\{E\}rationales with a score≤1\\leq 1, or the specific unmapped preferences when parity\-coverage is targeted\), the dimension’s score trend across iterations \(improving / stable / worsening\), and a log of prior edits annotated with their kept/discarded outcomes\. It is instructed to propose exactly one change, to try a fundamentally different approach when a dimension keeps worsening despite edits targeting it, and never to repeat a previously discarded change\. The revised promptpo\(t\+1\)p\_\{o\}^\{\(t\+1\)\}is then regenerated and rescored \(§[4\.1](https://arxiv.org/html/2609.00274#S4.SS1)\), and kept only if its composite score improves\.

### 4\.6\.Parity mapping

Once the loop terminates with the best catalogT⋆T^\{\\star\}, a final LLM pass migrates the occupation’s legacy RF Q&AQoQ\_\{o\}onto the new tags, producing the many\-to\-many relationM⋆⊆T⋆×QoM^\{\\star\}\\subseteq T^\{\\star\}\\times Q\_\{o\}returned by Algorithm[1](https://arxiv.org/html/2609.00274#alg1)\. The autoresearch design ensures that the generated catalog covers a good fraction of legacy taxonomies \(explicitly in§\\lx@sectionsign[4\.3](https://arxiv.org/html/2609.00274#S4.SS3), parity critic in§\\lx@sectionsign[4\.4](https://arxiv.org/html/2609.00274#S4.SS4), and set level coverage in§\\lx@sectionsign[4\.5](https://arxiv.org/html/2609.00274#S4.SS5)\)\.M⋆M^\{\\star\}, on the other hand, is*explicit and auditable*: it records exactly which legacy preferences map to which tags and which remain uncovered, producing the human\-reviewable artifact used to sign off the catalog before deployment\. For each legacy answer, the mapper finds*all*tags that cover the same real\-world concept; a single answer may map to several tags \(for example, a specific tag and the broader tag above it\), which is whyM⋆M^\{\\star\}is a many\-to\-many relation rather than a function\. Aggregated across the 132\-occupation production catalog,M⋆M^\{\\star\}is also what yields the reconstruction\-fidelity numbers quoted in §[1](https://arxiv.org/html/2609.00274#S1):73\.3%73\.3\\%of legacy answers map directly to a tag in the regenerated catalog;20\.8%20\.8\\%are intake\-only answers \(sizes, ranges, “flexible”\) excluded as acceptable non\-tags; the remaining6\.0%6\.0\\%are regrettable misses, real preferences left uncovered\. Measured against the answers that should map, pooled coverage is92\.5%92\.5\\%, with a per\-occupation median of100%100\\%\(mean97\.8%97\.8\\%\)\.

## 5\.Deployment

The reconstructed catalog has been running in production since April 2026\. This section describes how catalogs were generated at scale across the occupation set and how each catalog was staged for human review before going live\.

#### Catalog Generation

Each occupation is reconstructed independently by the per\-occupation loop of §[4\.1](https://arxiv.org/html/2609.00274#S4.SS1): starting from the shared baseline prompt, the generator, judge, critic panel, and editor iterate for the fixed budget, after which the parity mapping produces the occupation’s best catalogT⋆T^\{\\star\}, its tag↔\\leftrightarrowQ&A relationM⋆M^\{\\star\}, and the residual setsUQ⋆,UT⋆U\_\{Q\}^\{\\star\},U\_\{T\}^\{\\star\}\. Because the loop is occupation\-local, the 132 occupations are processed independently and in parallel\. Every run persists a full audit trail: a per\-occupation iteration log recording each kept change with its rationale and target dimension, a snapshot of the best tag set and its scores at each iteration, and the parity\-mapping outcome\. These artifacts are both retained per run and streamed to a shared reviewer workspace as iterations are kept \(§[5](https://arxiv.org/html/2609.00274#S5.SS0.SSS0.Px2)\), so the catalog and its rationale are available for inspection while a run is still in progress\.

#### Human QA at Scale

No catalog is deployed without human sign\-off, so each generated catalog is seeded into a shared Google Sheet for review\. Results were uploaded live during a run into managed per\-occupation tabs:*Best*\(the current tag set\),*Iteration Log*\(every kept change with its rationale\),*Tricky Parity Cases*\(ambiguous legacy mappings flagged for attention\), and*Comment Archive*\(resolved reviewer comments\)\. The parity tab includes a Verdict dropdown \(Keep/Update/Delete\) with conditional formatting, and reviewers can inline\-edit tag names\. Review is not a one\-way gate: unresolved reviewer comments are polled from the sheet \(via the Drive API\) and written to a per\-occupation feedback file, which the generator and editor read on the next run\. Human feedback, such as renamed tags, deletions, and free\-text objections, therefore feeds directly back into the automated loop, closing the gap between manual review and regeneration\.

#### Scale and Coverage

The system has been live across 132 occupations\. New occupations are onboarded on demand from a single configuration entry: a per\-occupation config file maps each occupation slug to its data\-warehouse table names and primary keys, and adding an occupation requires only that entry\.*Automotive Detailing*, for example, was added and taken end\-to\-end: baseline generation, autoresearch loop, parity mapping, and human QA review, within five hours on a single day in May 2026\.

## 6\.Evaluation and Operational Lessons

### 6\.1\.Catalog Quality vs\. Deployment Outcomes

The autoresearch loop produces a catalog whose quality is measured in one space \(theℰ\\mathcal\{E\}composite plus the seven critic penalties \), and the deployment is measured in another \(filter\-evaluation outcomes, occupation concentration\)\. Conflating the two muddles attribution: a missing\-canonical\-tag filter rate of40\.66%40\.66\\%in production is not a verdict on the catalog, it is a verdict on the deployment\-side seeding step that was supposed to attach canonical tags to existing pro profiles\. Likewise, the loop\-internal critic penalties say nothing about whether request\-time enrichment actually emits the right tags in production\. We separate the two strands explicitly in what follows\.

### 6\.2\.Catalog\-Quality Metrics

#### ℰ\\mathcal\{E\}composite distributions\.

Across the 132\-occupation production runs, the per\-tagℰ\\mathcal\{E\}composite \(out of 15\) lands in a relatively narrow band on the best kept catalog per occupation: the bulk of tags score in the 11–14 range, with low\-end outliers concentrated on Canonical Coverage \(the binary00\-or\-22rubric\) for occupations whose legacy categories do not have an obvious single\-tag canonical mapping\.

#### Critic\-panel score distributions\.

As designed, the Pro critic accumulates the largest aggregate penalty mass \(capped at2\.52\.5per iteration; supplementary material\); the Adversarial critic contributes a sparse but heavy\-tailed distribution dominated by thecriticalrisk tier; and the Reasoning bonus fires on roughly the iterations where the editor’s most recent change was a rationale upgrade\.

#### Cross\-occupation tag\-name drift\.

Because each occupation’s catalog is generated in isolation, two occupations whose tags should share a name could drift apart\. We measure this on a 14\-occupation, 189\-tag cross\-section at two layers:

\(i\) Pro\-facing tag strings \(the names shown to pros, e\.g\., Cabinet installation\) all 189 are unique to their occupation\. Occupation\-local vocabulary is preserved as intended: chess tutoring emits Adult chess tutoring, not a generic Adult that would also fit physical therapy\. This is an exact\-string check; semantic near\-duplicates like Installation vs Installations are a within\-occupation Taxonomy\-critic concern, not a cross\-occupation one\.

\(ii\) Structural category prefixes \(the part before the colon in each tag’s structured form, e\.g\., Work type in Work type: Cabinet installation — the axis the matchmaker and pricing model filter on\) 12 distinct prefixes across the cohort after case\-folding, distributed as shown in Table[1](https://arxiv.org/html/2609.00274#S6.T1)\. Four are widely shared: Capability and Work type in 9/14 occupations, Deliverable and Scope in 7/14, confirming partial standardization emerges without a global hierarchy\. Two residual drifts are both targets for the next autoresearch wave: \(a\) eight singleton prefixes, three of which are clearly the same concept under different names \(Client preference, Client type, Customer type\); and \(b\) case inconsistency on shared prefixes \(e\.g\., Deliverable vs DELIVERABLE\)\.

Table 1\.Cross\-occupation structural\-prefix \(KV\-key\) drift on then=14n=14occupation cross\-section \(189 tags\), case\-folded\. The four shared keys confirm partial standardization without a global hierarchy; the eight singletons and case variants are the residual drift targeted by the next autoresearch wave\.
#### Planned dual\-evaluator validation\.

A complementary validation we plan \(eval\-of\-eval\) compares the productionℰ\\mathcal\{E\}judge against an independently\-developed external evaluator \(different judge model, different rubric structure, different thresholds\) on the same per\-occupation tag set, structured around verdict\-agreement rate, per\-rubric Cohen’sκ\\kappaon the six shared dimensions, and a critic\-based qualitative analysis of disagreement tags\.

#### Critic ablation\.

A single\-occupation Parity\-removed critic ablation on*Accounting*\(supplementary material\) re\-aggregates the production iteration log under a six\-critic configuration\. On the observed Accounting run, the Parity critic returnedparity\_score=7/10=7/10in both kept iterations — above the panel’s penalty thresholds — and therefore contributed zero to the critic penalty\. Removing Parity from the panel leaves the adjusted\-score trajectory and the final best tag set unchanged on this occupation\. We read this as evidence that, on occupations where parity\-readiness is comfortably above the penalty floor, the Parity critic’s seat is insurance against the lower\-readiness tail rather than an active per\-iteration contributor; characterizing Parity’s marginal contribution across the 132\-occupation cohort is the next ablation\-program step\.

### 6\.3\.Deployment\-Side Metrics

The 14\-day production post\-launch cohort study \( 1,840 MVP\-enrolled pros, 9,324,798 filter evaluations over 2026\-04\-28 to 2026\-05\-12\) characterizes the catalog*as deployed*, distinct from the catalog as generated\.

#### Catalog hygiene\.

Three catalog tags markedstatus = 2\(deprecated\) were still being emitted by request\-time enrichment as*canonical category tags*, driving roughly486,000486\{,\}000filter events in the window\. The implicated tags—*Heavy lifting*,*Bathroom remodel*,*Wallpaper installation or repair*—came from a request\-side reverse\-mapping table that was never reconciled with the catalog’s active\-status flag\. The autoresearch loop did not surface this finding directly: it produced a per\-occupation catalog whose deployment, monitored at the filter\-evaluation event level, exposed the request\-side / pro\-side mismatch\. The*surfacing*came from per\-event deployment monitoring—any per\-event monitoring pipeline against a hierarchical or per\-occupation build could have detected the same SQL signal\. What per\-occupation independence specifically contributes is unambiguous*attribution*: in a unified hierarchy “Heavy lifting” has a defensible existence as a Moving\-and\-Lifting sub\-tag, so the deprecation flag is one signal among many and the diagnosis is “data\-quality nuisance”; under per\-occupation independence the deprecated tag has no occupation that claims it as canonical, so the request\-side emission is unambiguously a request\-side / pro\-side mismatch and the fix path is mechanical \(re\-activate the tag or remove from the reverse mapping\)\. The contribution is the diagnostic clarity, not the diagnostic visibility\.

#### Filter outcome distribution\.

For the same cohort, the strict\-match filter outcome distribution is shown in Table[2](https://arxiv.org/html/2609.00274#S6.T2)\. The dominant failure mode—missing canonical category tag,40\.66%40\.66\\%of all MVP evaluations—is a deployment\-stage gap, not a catalog\-quality gap: the catalog tag exists, request\-side enrichment fires it, but the pro’s profile was never seeded with it\. This motivates the deployment\-side intervention \(auto\-attach canonical category tags at MVP onboarding\) and validates the Parity critic’s instinct that catalog completeness without deployment\-side wiring is not production coverage\.

Table 2\.Strict\-match filter outcome distribution across 9\.3M MVP filter evaluations in the 14\-day post\-launch window\. The dominant failure mode—missing canonical category tag—accounts for 40\.66% of all MVP evaluations and is, by itself, larger than the sum of all other filter reasons combined\.
#### Occupation concentration\.

The same cohort surfaced a strong concentration of filter events:*Handyman*alone accounted for roughly70%70\\%of MVP filter events \(3\.773\.77M of5\.435\.43M filtered evaluations\)\. Four other occupations had structurally high trigger rates:*Roofing/siding*\(73\.5%73\.5\\%\),*Home technology*\(72\.8%72\.8\\%\),*Doors/windows*\(67\.0%67\.0\\%\),*Appliances*\(66\.5%66\.5\\%\)\. This suggests the canonical tag taxonomy for these occupations does not match how pros describe their work\. These are concrete targets for the next autoresearch wave, identifiable because per\-occupation results are not averaged into a global score\.

#### Directional post\-launch signals \(extended MVP window\)\.

We did an MVP analysis at88–1010weeks post\-launch \(June–July 2026\) and opened up enrollment from the1,8401\{,\}840\-pro cohort above to∼\\sim3,9003\{,\}900pros\. We got directional, though not causal, evidence consistent with the generated\-vs\-deployed attribution of §[6\.1](https://arxiv.org/html/2609.00274#S6.SS1)\. \(i\) About88%88\\%of enrolled pros remained enrolled, with attrition concentrated among high\-volume providers and attributed to price and lead\-mix economics, not catalog semantics\. \(ii\) Of4,7214\{,\}721“not what I do” lead\-declines,94%94\\%came from providers whose recorded preferences already excluded that work\. While early, we interpret this as an enforcement gap, not a vocabulary gap\. Additionally, a review from our category\-management team recommended seven point changes to the deployed taxonomy\. \(iii\) About35%35\\%of newly recorded provider limitations arrive through the LLM refinement flow built on the generated catalog\. Where no preference was adopted, around 75% trace to user\-experience gaps rather than rejection of a suggested tag \(∼\\sim21%21\\%explicit rejections\)\. \(iv\) Finally, churn and refunds seem to cluster amongst pros who had narrow legacy categories that mapped into broad new\-catalog occupations, and providers with no legacy\-category crosswalk link received∼\\sim35%35\\%fewer leads, consistent with the canonical\-assignment and parity lessons of §[6\.4](https://arxiv.org/html/2609.00274#S6.SS4)\. An outcome\-level comparison against legacy forms remains future work\.

### 6\.4\.Operational Lessons from the Loop

#### Canonical\-tag assignment is structurally different from holistic generation\.

An earlier version of the pipeline assigned canonical category tags inside the generation prompt\. This was migrated to a dedicated mapping\-phase LLM call that runs before each mapping batch and injects canonical context\. The architectural separation produced both better canonical coverage and more consistent generation \(the generator now focuses on holistic tag creation; canonical assignment is a structured decision with its own prompt and its own good/bad examples\)\.

#### Compound names are not unions\.

The canonical assignment rule originally rejected any category whose name contained “and” or “or” \(treating it as a union of distinct services\)\. This collapsed canonical coverage on*Electrical*from 10/10 to 1/10 because categories like*Circuit Breaker Panel or Fuse Box*are compound names for a single service, not unions\. The refined rule requires*actual RF Q&A disambiguation evidence*before rejecting a union category\.

#### Stopping is budget\-driven\.

The current loop stops at a fixed iteration budget \(default 5\) rather than at a convergence\-based criterion\. A confidence\-interval criterion is dominated in cost by the multi\-trial evaluations it would require; across production runs the kept\-iteration count typically saturates well before the budget is exhausted \(§[6\.5](https://arxiv.org/html/2609.00274#S6.SS5)\)\. The per\-iteration early\-skip described in §[4\.1](https://arxiv.org/html/2609.00274#S4.SS1)step 6 captures most of the available cost saving without committing to a plateau heuristic the implementation does not yet support\.

### 6\.5\.Convergence Behavior Across Production Occupations

To characterize how often the loop converges within budget, we aggregate the best\-iteration summary row for each of the132132production occupations from the autoresearch tracking logs\. Table[3](https://arxiv.org/html/2609.00274#S6.T3)summarizes the distribution\.

Table 3\.Convergence behavior across all132132production occupations\. The iteration index reported is the iteration that produced the best kept tag set per occupation \(index00denotes occupations whose baseline was never beaten\); “adjusted score” isℰ​composite−critic penalty\\mathcal\{E\}\\text\{ composite\}\-\\text\{critic penalty\}\. Quartiles use the standard linear\-interpolation convention \(NumPypercentiledefault\)\.Three observations hold across the full132132\-occupation cohort\. \(i\) The best kept tag set is reached by iteration33for every occupation—3636at iteration11,5353at iteration22,3838at iteration33, and55that never beat their baseline \(index00\)—with none reaching its best after iteration33, so the kept\-iteration count saturates well before the55\-iteration budget\. \(ii\) Theℰ\\mathcal\{E\}composite is tightly clustered \(median13\.84/1513\.84/15, mean13\.5313\.53, interquartile range13\.3213\.32–14\.2014\.20\), but the*adjusted*score spreads much wider \(median10\.1710\.17, IQR8\.638\.63–11\.4111\.41\):1919of132132occupations fall in the<8<8band that the production verdict thresholds \(§[6\.2](https://arxiv.org/html/2609.00274#S6.SS2)\) route to review or flag, almost entirely on critic penalty rather than on a weak judge score\. \(iii\) The critic penalty \(range0\.500\.50–9\.709\.70, median3\.623\.62\) is therefore the dominant source of variance in the adjusted score, consistent with the asymmetric panel design \(supplementary material\) in which the Adversarial critic alone can contribute up to2\.02\.0on thecriticalrisk tier\.

#### Loop vs\. single\-shot generation\.

Iteration 0 of every production run*is*single\-shot generation: the shared seed promptp\(0\)p^\{\(0\)\}applied once, with no critics and no editor, so the tracking logs contain the single\-pass baseline directly\. Across the 132\-occupation cohort, the loop lifts the per\-occupationEEcomposite from a median of 11\.36 at iteration 0 to 13\.84 on the best kept set, a median gain of\+2\.24\+2\.24points \(IQR 1\.60–2\.64\)\. It improves 119 of 132 occupations \(100 by at least one point, 87 by at least two\); 5 never beat their baseline, and 8 keep a set with equal\-or\-lower composite that wins on a smaller critic penalty, since keeps are decided on the adjusted score\. This comparison runs under the loop’s own judge, so it isolates the value of iterate\-and\-keep over single\-pass generation; whether the judge itself tracks human quality is assessed separately below\.

#### ℰ\\mathcal\{E\}\-only baseline \(vanilla LLM\-as\-judge contrast\)\.

To isolate the seven\-critic panel’s marginal contribution at the verdict\-decision boundary, we re\-bin all132132best\-iteration rows under anℰ\\mathcal\{E\}\-only configuration that drops the critic penalty entirely \(critic\_penalty:=0\\texttt\{critic\\\_penalty\}:=0\) and applies the production thresholds \(auto\_approve≥11\\geq 11,human\_review≥8\\geq 8,flag<8<8\)\. The two configurations disagree on8484of132132occupations \(63\.6%63\.6\\%\), and every disagreement is in the same direction—the panel’s verdict is the same or stricter, the expected sign because the critic penalty is non\-negative\. The distributions tell the story:ℰ\\mathcal\{E\}alone wouldauto\_approve125125of132132occupations \(94\.7%94\.7\\%\), leaving a human\-review queue of only77; the panel routes just4747toauto\_approve,6666tohuman\_review, and1919toflag\. The panel’s dominant operating mode is therefore to demote tag sets the judge alone would wave through—it is what creates the8585\-occupation human\-in\-the\-loop queue the production system relies on\. This is a verdict\-bucket comparison on the converged best\-iteration tag sets, not a re\-run of the loop with the panel disabled; the latter \(the panel’s effect on the kept\-iteration*trajectory*\) is the ablation scoped in §[6\.4](https://arxiv.org/html/2609.00274#S6.SS4)\.

#### ℰ↔\\mathcal\{E\}\\leftrightarrowPM\-consensus agreement\.

Table[4](https://arxiv.org/html/2609.00274#S6.T4)cross\-tabulates the model’sEEverdict against the production PM\-consensus action over the 4,112 tags that received both, under the verdict map Keep↔\\leftrightarrowauto\-approve, Update↔\\leftrightarrowhuman\-review, Delete↔\\leftrightarrowflag\. Raw agreement is 77\.2% \(3,174/4,112\), but this is almost entirely a base\-rate effect: the model auto\-approves 97\.3% of tags and reviewers keep 79\.1%, so chance agreement is already 77\.2% and Cohen’sκ≈0\\kappa\\approx 0\. The discreteEEverdict does not, on its own, predict which tags a human will reject\. We report this negative result deliberately: it is the empirical basis for using theEEcomposite as a ranking signal inside the loop rather than an acceptance gate, and for keeping human sign\-off mandatory before any catalog deploys\. One caveat is circularity on the legibility rubric, which was recalibrated from the same PM review \(§[4\.2](https://arxiv.org/html/2609.00274#S4.SS2)\)\.

Table 4\.Confusion matrix between the model’s E1 verdict \(columns\) and the human reviewer’s action \(rows\), over4,1124\{,\}112tags that received both\. Categories are aligned as auto\_approve↔\\leftrightarrowKeep, human\_review↔\\leftrightarrowUpdate, flag↔\\leftrightarrowDelete\.

## 7\.Discussion

Our contribution is not a single algorithm but a system architecture for reconstructing catalogs in marketplaces that carry substantial legacy structured data, where getting the pro\-side schema right has large downstream consequences\. The composition we defend has three elements: per\-occupation independence, the seven\-persona critic panel, and a separate parity\-mapping stage\. While each module has its own ancestry, what we add is their integration, the scale at which the system runs in production \(132 occupations, with new ones onboarded on demand\), and the operational lessons that scale has surfaced\. We see two clear extension paths as future work\. First, the search backend is currently a naive LLM proposal\. GEPA\-style reflective optimization\([Agrawal et al\., 2025](https://arxiv.org/html/2609.00274#bib.bib2)\)with execution traces is a natural upgrade once the critic\-panel output is structured enough to serve as a reflection signal\. Second, the production feedback loop is currently human\-in\-the\-loop \(PM comments flow back via the live sheet\)\. Routing matchmaker\-side production failures \(eg, 41% missing\-canonical\-tag filter rate\) into the next autoresearch iteration would close the loop between catalog generation and matchmaking outcomes\.

#### Trustworthiness considerations\.

The cross\-family arrangement described in §[4\.1](https://arxiv.org/html/2609.00274#S4.SS1)mitigates self\-reinforcement between the generator \(GPT\-5\-4 family\) and the seven critics, editor, and parity mapper \(Claude Sonnet 4\.6 family\), but two residual trust gaps remain\. \(i\) Within Claude Sonnet 4\.6, the editor and the seven critics share a model family; an editor that is systematically biased in a direction the critics also endorse will not be caught by the loop, even though it would be caught by a cross\-family critic seat\. A natural mitigation is to host one critic on a third model family \(e\.g\., a Gemini\- or Llama\-class model\) and treat its disagreement as an explicit signal to the editor\. \(ii\) Critic disagreement is currently silent: the editor sees the aggregate adjusted score, not the per\-critic decomposition, so a high\-variance verdict \(e\.g\., the Adversarial critic flags a critical failure that no other critic surfaces\) is averaged into the same scalar as a low\-variance verdict\. Surfacing per\-critic penalties and a disagreement summary as explicit editor inputs is the most direct upgrade to the loop’s trustworthiness profile we are aware of, and is a planned next iteration\.

## 8\.Conclusion

We presented a per\-occupation autoresearch system for generating marketplace catalogs, in production at a 132\-occupation consumer services marketplace\. The contributions we defend are compositional: the unit of optimization \(preference\-tag catalog within an occupation\), the per\-occupation parallelism, the seven\-persona critic panel with weighted penalties, the legacy\-Q&A\-to\-modern\-tag migration via Step\-Back abstraction, and the MEMORY\_ONLY typing primitive\. Production deployment at scale surfaced an operational catalog\-hygiene finding \(three deprecated tags driving 486K filter events\) that a global hierarchical build would have masked\. The algorithmic ancestry \(Karpathy, Self\-Refine, Step\-Back, Constitutional AI, GEPA\) is acknowledged; the integration and the deployment results are what we report\.

## References

- Agrawal et al\.\(2025\)Lakshya A\. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl\-Ong, Arnav Singhvi, Herumb Shandilya, Michael J\. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G\. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab\. 2025\.GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning\.arXiv:2507\.19457
- Bai et al\.\(2022\)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al\.2022\.Constitutional AI: Harmlessness from AI Feedback\.arXiv:2212\.08073 \[cs\.CL\]
- Du et al\.\(2024\)Yilun Du, Shuang Li, Antonio Torralba, Joshua B\. Tenenbaum, and Igor Mordatch\. 2024\.Improving Factuality and Reasoning in Language Models through Multiagent Debate\. In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*\.
- Einav et al\.\(2016\)Liran Einav, Chiara Farronato, and Jonathan Levin\. 2016\.Peer\-to\-Peer Markets\.*Annual Review of Economics*8 \(2016\), 615–635\.
- Huang et al\.\(2025\)Yunhan Huang, Klevis Ramo, Andrea Iovine, Melvin Monteiro, Sedat Gokalp, Arjun Bakshi, Hasan Turalic, Arsh Kumar, Jona Neumeier, Ripley Yates, Rejaul Monir, Simon Hartmann, Tushar Manglik, and Mohamed Yakout\. 2025\.AttributeForge: An Agentic LLM Framework for Automated Product Schema Modeling\. In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track*\. 2106–2121\.[doi:10\.18653/v1/2025\.emnlp\-industry\.148](https://doi.org/10.18653/v1/2025.emnlp-industry.148)
- Husain \(2024\)Hamel Husain\. 2024\.Critique Shadowing: Building Trustworthy LLM\-as\-Judge Systems\.Online essay,[https://hamel\.dev/blog/posts/llm\-judge/](https://hamel.dev/blog/posts/llm-judge/)\.Accessed 2026\-06\-09\.
- Kargupta et al\.\(2025\)Priyanka Kargupta, Nan Zhang, Yunyi Zhang, Rui Zhang, Prasenjit Mitra, and Jiawei Han\. 2025\.TaxoAdapt: Aligning LLM\-Based Multidimensional Taxonomy Construction to Evolving Research Corpora\. In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.[https://aclanthology\.org/2025\.acl\-long\.1442/](https://aclanthology.org/2025.acl-long.1442/)
- Karpathy \(2026\)Andrej Karpathy\. 2026\.autoresearch: AI agents running research on single\-GPU nanochat training automatically\.GitHub repository\.[https://github\.com/karpathy/autoresearch](https://github.com/karpathy/autoresearch)Released March 7, 2026\.
- Li et al\.\(2025\)Nan Li, Bo Kang, and Tijl De Bie\. 2025\.Building Data\-Driven Occupation Taxonomies: A Bottom\-Up Multi\-Stage Approach via Semantic Clustering and Multi\-Agent Collaboration\.arXiv:2509\.15786Introduces the CLIMB multi\-agent framework for occupation hierarchy construction\.
- Madaan et al\.\(2023\)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark\. 2023\.Self\-Refine: Iterative Refinement with Self\-Feedback\. In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Opsahl\-Ong et al\.\(2024\)Krista Opsahl\-Ong, Michael J\. Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab\. 2024\.Optimizing Instructions and Demonstrations for Multi\-Stage Language Model Programs \(MIPROv2\)\. In*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*\.
- Wan et al\.\(2024\)Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Xu, Subhabrata Mukherjee, et al\.2024\.TnT\-LLM: Text Mining at Scale with Large Language Models\. In*Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR ’24\)*\.
- Zheng et al\.\(2024\)Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng\-Tze Cheng, Ed H\. Chi, Quoc V\. Le, and Denny Zhou\. 2024\.Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models\. In*International Conference on Learning Representations \(ICLR\)*\.
- Zheng et al\.\(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\. 2023\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\. In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.