OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models

# A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models
Source: [https://arxiv.org/html/2608.20106](https://arxiv.org/html/2608.20106)
###### Abstract

We introduceOenoBench, a wine\-domain knowledge benchmark of3,266 multiple\-choice questionsacross six pillars \(regions, grape varieties, viticulture, winemaking, producers, business\) and four difficulty tiers\. The corpus is built from38,104 atomic, source\-anchored factsextracted by 35 provenance\-verified scrapers from government registries \(INAO, TTB, OIV\), peer\-reviewed journals, and Wikipedia/Wikidata\. Our methodological contribution is an*LLM\-driven pipeline*in which language models reformat verified facts and audit the result, but never serve as the source of truth: every claim traces to a URL, every question is generated by one of five strategies across five generator families, and every question is scored by a nine\-agent audit calibrated against a human gold sheet via Cohen’sκ\\kappa\. Evaluating sixteen frontier configurations, we find: \(i\) overall accuracy spans 53%–84%, led byo3 at 83\.6%; \(ii\) reasoning\-mode lift concentrates inDeepSeek R1 \(\+\+6\.8 pp\)and is absent in Claude Opus and Gemini Pro; \(iii\) Anthropic shows\+\+9 pp self\-preferenceon its own questions while Google shows−\-8 ppinverse preference; \(iv\) frontier open\-weight models share the cost\-vs\-accuracy Pareto frontier with proprietary reasoning models; and \(v\) every config gains≈\\approx33 ppon closed\-book solvable items, revealing a parametric\-recall ceiling that only the contextual slice avoids\. We release corpus, audit findings, human\-review app, and construction code under CC\-BY\-SA\-4\.0\.

## 1Introduction

Large language models are increasingly evaluated on knowledge\-intensive benchmarks, but the most widely\-used resources — MMLU\[[6](https://arxiv.org/html/2608.20106#bib.bib1)\], GPQA\[[16](https://arxiv.org/html/2608.20106#bib.bib2)\], and their successors\[[19](https://arxiv.org/html/2608.20106#bib.bib3),[21](https://arxiv.org/html/2608.20106#bib.bib4),[15](https://arxiv.org/html/2608.20106#bib.bib5)\]— share three weaknesses: they aggregate breadth at the expense of*depth*within any single discipline; they are typically authored by a small team using one generation pipeline, exposing systematic blind\-spots and stylistic regularities that frontier models can learn to exploit\[[14](https://arxiv.org/html/2608.20106#bib.bib10),[20](https://arxiv.org/html/2608.20106#bib.bib11)\]; and the questions are increasingly suspected of contamination through web\-scale pre\-training corpora\[[17](https://arxiv.org/html/2608.20106#bib.bib13),[13](https://arxiv.org/html/2608.20106#bib.bib14),[3](https://arxiv.org/html/2608.20106#bib.bib15)\]\. As models grow more capable, the marginal value of a benchmark depends increasingly on*how*it was constructed\.

We argue that domain\-specialised benchmarks built around bodies of knowledge that are*externally validated by humans*— through certification, regulation, or peer\-reviewed practice — offer a useful complement to broad knowledge benchmarks\. Examples already exist in medicine\[[9](https://arxiv.org/html/2608.20106#bib.bib6),[10](https://arxiv.org/html/2608.20106#bib.bib7)\], law\[[5](https://arxiv.org/html/2608.20106#bib.bib8)\], and finance\[[8](https://arxiv.org/html/2608.20106#bib.bib9)\]\. We extend this line of work to a domain that has so far received little attention from the benchmarking community, despite being unusually well\-defined:wine\.

#### Why wine\.

Wine integrates plant biology, geology and climate science, organic chemistry, food microbiology, sensory neuroscience, regulatory law, and business economics\. It is one of very few domains for which inter\-governmental and governmental sources publish structured taxonomic data \(INAO, OIV, TTB, regional consortia\), university research groups \(UC Davis, Geisenheim, Bordeaux Sciences Agro\) produce open scholarly literature, and professional certification bodies \(WSET Levels 1–4, Court of Master Sommeliers, Institute of Masters of Wine\) define a graded competence ladder\. These properties make it possible to build a benchmark whose every question is traceable to a verifiable source while still spanning the recall–reasoning–application spectrum that distinguishes a strong taster, winemaker, or buyer from a memorised glossary\.

#### Contributions\.

1. 1\.A 3,266\-question wine benchmarkacross six pillars and four difficulty tiers, built from 38,104 atomic provenance\-tagged facts \(Section[3](https://arxiv.org/html/2608.20106#S3)\)\.
2. 2\.An LLM\-driven, fact\-grounded generation pipelinewith five strategies×\\timesfive generator models, per\-strategy quotas, and an explicit closed\-book pre\-screen — LLMs reformat verified facts, never supply them\.
3. 3\.A nine\-agent automated audit\(Section[4](https://arxiv.org/html/2608.20106#S4)\) calibrated against a human gold sheet via Cohen’sκ\\kappa; the audit removed 341 questions and re\-labelled difficulty on 1,259, and surfaced one agent \(closed\-book solvability\) that over\-reports leakage relative to humans, which we keep with explicit disclosure rather than drop\.
4. 4\.A bias\-aware evaluation across sixteen frontier configurations\(Section[5](https://arxiv.org/html/2608.20106#S5)\) reporting overall accuracy, reasoning\-mode lift, a Self\-Preference Score, cost\-efficiency Pareto, and a closed\-book vs\. source\-grounded contrast that isolates parametric recall from contextual reasoning\.

OenoBench — corpus, audit findings, construction code, and human\-review app — ships under CC\-BY\-SA\-4\.0; release URLs, reproducibility scripts, and the per\-source licensing table are collected in Appendix[E](https://arxiv.org/html/2608.20106#A5)\.

## 2Related Work

#### Broad knowledge benchmarks\.

MMLU\[[6](https://arxiv.org/html/2608.20106#bib.bib1)\], BIG\-Bench\[[19](https://arxiv.org/html/2608.20106#bib.bib3)\], AGIEval\[[21](https://arxiv.org/html/2608.20106#bib.bib4)\], GPQA\[[16](https://arxiv.org/html/2608.20106#bib.bib2)\], HELM\[[12](https://arxiv.org/html/2608.20106#bib.bib20)\], and Humanity’s Last Exam\[[15](https://arxiv.org/html/2608.20106#bib.bib5)\]measure factual breadth and reasoning across many subjects\. We instead maximise depth within a single externally\-validated knowledge domain and treat*construction\-time bias control*as a first\-class contribution rather than a downstream decontamination step\.

#### Domain\-specific benchmarks\.

MedQA\[[9](https://arxiv.org/html/2608.20106#bib.bib6)\], PubMedQA\[[10](https://arxiv.org/html/2608.20106#bib.bib7)\], LegalBench\[[5](https://arxiv.org/html/2608.20106#bib.bib8)\], and FinanceBench\[[8](https://arxiv.org/html/2608.20106#bib.bib9)\]ground evaluation in regulated professions with clear sources of truth\. We adopt their expert\-syllabus / source\-traceable backbone and add multi\-model generation, multi\-agent audit, and self\-preference reporting\. OenoBench is, to our knowledge, the first such benchmark for wine\.

#### LLM\-as\-judge bias and self\-preference\.

A growing literature documents that models tend to favour outputs stylistically similar to their own\[[14](https://arxiv.org/html/2608.20106#bib.bib10),[20](https://arxiv.org/html/2608.20106#bib.bib11),[18](https://arxiv.org/html/2608.20106#bib.bib12)\], biasing benchmarks where the same model generates and grades\. We mitigate by distributing generation across five generator families plus deterministic templates and by reporting per\-model self\-vs\-other accuracy gaps as a first\-class diagnostic \(Section[5\.3](https://arxiv.org/html/2608.20106#S5.SS3)\)\.

#### Benchmark contamination and solvability\.

Pre\-training exposure inflates apparent capability\[[17](https://arxiv.org/html/2608.20106#bib.bib13),[13](https://arxiv.org/html/2608.20106#bib.bib14),[3](https://arxiv.org/html/2608.20106#bib.bib15)\]\. We mitigate by generating from atomic facts \(a fact\-echo agent flags any question whose longest common substring with the source exceeds 65%\) and by running an explicit*closed\-book solvability*pre\-screen and audit \(Section[5\.5](https://arxiv.org/html/2608.20106#S5.SS5)\)\. The same audit, calibrated against a human gold sheet, reveals that LLM judges over\-report leakage by an order of magnitude — a finding with broader implications for the LLM\-as\-judge paradigm\.

#### Wine in NLP\.

Wine has previously appeared in NLP via review\-text recommendation\[[2](https://arxiv.org/html/2608.20106#bib.bib18)\], taxonomy learning\[[11](https://arxiv.org/html/2608.20106#bib.bib17)\], and judge\-reliability studies\[[7](https://arxiv.org/html/2608.20106#bib.bib16)\]\. OenoBench is the first benchmark targeting*structured factual knowledge*of viticulture, winemaking, and wine regions evaluated against frontier LLMs\.

## 3Dataset Construction

OenoBench is constructed by an LLM\-driven pipeline \(Figure[1](https://arxiv.org/html/2608.20106#S3.F1)\) in two stages: \(1\) a*fact\-collection*stage that builds a corpus of atomic, provenance\-tagged statements from authoritative sources, and \(2\) a*question\-generation*stage that converts those facts into benchmark questions using five complementary strategies and five generator models\. The pipeline is designed around a single methodological commitment: large language models are used as*rephrasers and auditors*, never as the source of truth\. Every fact in the corpus traces to a URL with a tier\-of\-authority label, every question is anchored in one or more such facts, and every question is scored by a nine\-agent automated audit \(Section[4](https://arxiv.org/html/2608.20106#S4)\) before release\.

### 3\.1Domain taxonomy

![Refer to caption](https://arxiv.org/html/2608.20106v1/figures/diagram_pipeline.png)Figure 1:End\-to\-end OenoBench pipeline: authoritative sources→\\rightarrow35 scrapers→\\rightarrowatomic facts→\\rightarrowclosed\-book gate→\\rightarrownine\-agent audit→\\rightarrowrelease\.We organise wine knowledge into six*domain pillars*, chosen to align with the WSET Level 3 and Diploma syllabi: \(i\)wine regions\(appellations, classifications, geography, climate, soils\); \(ii\)grape varieties\(ampelography, parentage, regional distribution, sensory profile\); \(iii\)viticulture\(vineyard practice, training systems, pests and diseases, vintage effects\); \(iv\)winemaking\(vinification choices, fermentation, ageing, faults\); \(v\)producers\(estates, négociants, co\-operatives, key houses\); and \(vi\)wine business\(markets, regulation, distribution, economics\)\. Each domain has a separate target share informed by syllabus weighting and source availability; the realised distribution is reported in Table[1](https://arxiv.org/html/2608.20106#S3.T1)\.

### 3\.2Source tiering, licensing, and non\-fabrication

Every fact is tagged with one of three source tiers, mirroring evidence\-based\-medicine practice:

- •Tier 1 \(official, 19\.6%\):government registries and inter\-governmental bodies \(INAO, OIV, TTB, regional consortia, the UC Davis AVA database\)\.
- •Tier 2 \(authoritative, 76\.6%\):Wikipedia, Wikidata SPARQL, wine\-body and consortium websites, peer\-reviewed journals \(*OENO One*,*Vitis*,*Australian Journal of Grape and Wine Research*\)\.
- •Tier 3 \(reliable, 3\.8%\):curated open datasets on HuggingFace and Kaggle, secondary reference databases\.

#### Licensing and non\-fabrication\.

Every source is public\-record or released under a licence that permits scraping and redistribution: Tier\-1 government data under explicit open\-data licences \(INAO Licence Ouverte, OIV, UC Davis AVA, TTB public\-domain\), Tier\-2 Wikipedia/Wikidata under CC\-BY\-SA / CC0 and journals \(*OENO One*,*Vitis*,*AJEV*\) open\-access; the full per\-source licence table is Table[16](https://arxiv.org/html/2608.20106#A5.T16)in Appendix[E](https://arxiv.org/html/2608.20106#A5)\. A central design commitment is that*no fact in the corpus comes from an LLM’s internal knowledge*: every fact is extracted by a source\-specific scraper, stamped with source URL, retrieval timestamp, and tier label before insertion, and the fact\-insertion utility rejects records without a verified source\-URL trace\. We emphasise this because LLM\-generated content masquerading as scraped data is a contamination vector that conventional dataset documentation does not detect\.

Table 1:Atomic\-fact corpus: 38,104 facts from 35 scrapers, six domain pillars, three source tiers\.

### 3\.3Atomic fact extraction

Source pages are decomposed into atomic facts by a five\-stage pipeline \(Figure[5](https://arxiv.org/html/2608.20106#A2.F5), Appendix[B\.1](https://arxiv.org/html/2608.20106#A2.SS1)\): sentence splitting; reference resolution \(pronouns replaced by their entity referents\); domain classification into one of the six pillars; a length/predicate validator \(5–50 words, must contain a verb, no dangling references\); and a region\-keyword on\-topic filter that prevents cross\-contamination \(e\.g\. Austrian content in a Bordeaux scraper\)\. Wikidata SPARQL queries use direct country relations \(P17\) rather than transitive parents \(P131\*\), the latter having caused severe cross\-region leakage in early scrapers \(Appendix[B](https://arxiv.org/html/2608.20106#A2)\)\.

Facts are required to be*atomic*\(one assertion per sentence\),*entity\-tagged*\(subject and object linked to canonical knowledge\-graph IDs\), and*source\-faithful*\(a paraphrase, never a verbatim copy\); the last is enforced at extraction and re\-checked at audit time by agent A3\.

### 3\.4Question generation: five strategies, five models

A single\-strategy, single\-model benchmark inherits the blind\-spots of its pipeline\. We mitigate this by combining*five strategies*and*five generator models*, with per\-strategy and per\-model quotas enforced by an orchestrator that resumes safely across runs\. Strategies are:

- •Fact\-to\-question \(58\.4%, 1,909 questions\):the LLM rewrites a single verified fact as a multiple\-choice question; preserves grounding but tends toward recall\.
- •Distractor mining \(12\.4%, 405\):confusable\-entity sampling produces wrong answers that share the right answer’s category and dimension, raising plausibility and forcing fine\-grained discrimination\.
- •Template \(11\.9%, 389\):forty\-five deterministic parameterised templates across the six pillars; pure entity substitution, zero LLM creativity — a baseline against which neural generation is measured\.
- •Scenario synthesis \(9\.8%, 319\):coherent fact clusters are converted into applied multi\-fact reasoning prompts \(e\.g\. a sommelier service decision, a winemaker blending decision, a viticulturist vintage call\)\. Domain\-specific scenario types prevent persona\-content mismatch\.
- •Comparative \(7\.5%, 244\):entity\-affinity scoring pairs related entities \(e\.g\. two Burgundy premiers crus, two Rhône grapes\) and asks “which differs in X” or “what do both share”\.

The five generator models are Claude Opus 4\.7, GPT\-5, Gemini 2\.5 Pro, Llama 3\.1 405B, and Qwen 3\.5 235B, each accessed through a unified OpenRouter client with consistent temperature and top\-pp\. The realised generator distribution across the released corpus isQwen 20\.4%, Llama 19\.3%, Claude 19\.0%, ChatGPT 16\.6%, Gemini 12\.9%, deterministic templates 11\.9%\(computed directly from the 3,266 released questions\)\. The orchestrator allocates generation share dynamically based on per\-pilot audit pass rates — generators whose questions pass audit reliably get a larger share in subsequent rounds — subject to a hardper\-generator cap of 21%to keep any single model from dominating the corpus\.

### 3\.5Final corpus:release\_v1\.2

After generation, audit, and post\-eval review, the released corpusrelease\_v1\.2contains3,266 questions\. The distribution along the domain and difficulty axes is:

Table 2:Composition of the released corpus\.release\_v1\.2ships under CC\-BY\-SA\-4\.0 on HuggingFace with a Croissant manifest\[[1](https://arxiv.org/html/2608.20106#bib.bib21)\]and a full Datasheet\[[4](https://arxiv.org/html/2608.20106#bib.bib19)\]in Appendix[A](https://arxiv.org/html/2608.20106#A1)\.

## 4Multi\-Agent Quality Audit

The audit pipeline runs every generated question through nine automated agents in four teams\. Each agent emits a per\-question\{pass,warn,fail\}\\\{\\textsc\{pass\},\\textsc\{warn\},\\textsc\{fail\}\\\}signal calibrated against a human\-reviewed gold sheet via Cohen’sκ\\kappa; agents whose agreement falls below threshold are downweighted to advisory\-only\. The released human\-review web app \(Appendix[D](https://arxiv.org/html/2608.20106#A4)\) collects the gold\-sheet ratings, and the team\-by\-team architecture \(Figure[6](https://arxiv.org/html/2608.20106#A2.F6), Appendix[C](https://arxiv.org/html/2608.20106#A3)\) traces the flow from candidate question to drop / relabel / keep verdict\.

### 4\.1Team architecture

Of the 14 agents \(Tables[3](https://arxiv.org/html/2608.20106#S4.T3)–[4](https://arxiv.org/html/2608.20106#S4.T4)\),nine are always\-runon every candidate question; the remainingfive are escalation\-gatedand are invoked only when an upstream agent flags a corpus\-level concern\. This deferred\-execution policy keeps routine audit cost bounded \(∼\\sim$76 for the full corpus audit reported here, 5h 22m wall\) while preserving full coverage when a defect class is suspected\. The full audit code, prompts, and thresholds are released alongside the corpus \(Appendix[B](https://arxiv.org/html/2608.20106#A2)\)\.

Table 3:Audit agents \(1/2\): static and tri\-judge teams\.always: run on every question\.esc: escalation\-gated\.TeamAgentRunFunctionA\. StaticA1 LexicalHygienealwaysRegex sweep for vague phrasing, marketing language, meta\-questions\.A2 BiasStatsalwaysχ2\\chi^\{2\}on correct\-answer position; Mann–WhitneyUUon correct\-vs\-distractor length\.A3 FactEchoalwaysLongest\-common\-substring ratio between question text and source fact \(failat LCS≥\\geq0\.65\)\.A4 TemplateFingerprintalwaysPOS\-bigram logistic regression detector for template\-induced stylistic regularities \(held\-out AUC 0\.84\)\.B\. Tri\-judgeB1 TriJudgeAnsweralwaysClaude / GPT / Gemini panel reads question \+ source fact and votes on the marked\-correct option\.B2 ClosedBookSolvabilityalwaysSame panel, source fact*withheld*: can the question be answered from world knowledge alone?B3 UbiquityRiskalwaysInternationally\-grown grape stem×\\timesregion\-class answer⇒\\Rightarrowambiguity flag\.B4 AmbiguityescTri\-judge re\-read flags questions with\>\>1 defensible answer; invoked when B1 dissent rate elevated\.B5 VerifierSkipescSelf\-consistency probe across re\-prompted panels; invoked on disputed B1 verdicts\.Table 4:Audit agents \(2/2\): deterministic and corpus\-statistics teams\.TeamAgentRunFunctionC\. Determin\.C1 DistractorDifficultyescEmbedding\-distance distractor analysis; invoked when A2 reports length skew\.C2 CategoryLeakalwaysWine\-type distractor validation; flags red/white/sparkling category mismatches\.C3 SourceSwapescSubstitutes alternate source facts to test answer stability; invoked on B1/B2 disagreement\.C4 DifficultyAuditalwaysGemini Pro re\-rates difficulty;fail/warnwhen delta≥\\geq2 from assigned\.D\. Corpus statsD1 SelfPreferencealwaysPer\-generator\-family own\-vs\-other accuracy on a held\-out probe; corpus\-level signal\.D2 DedupCalibrationescNear\-duplicate audit at cosine≥\\geq0\.92; invoked on suspicious cluster signals\.D3 SkewAuditalwaysCountry / sub\-domain over\-representationχ2\\chi^\{2\}\.
### 4\.2Gold\-sheet calibration

A gold sheet of 50 stratified questions per release is rated independently by three WSET\-certified reviewers \(Diploma, Level 3, Level 2 — the Diploma rater is the lead author\) through the human\-review web app \(Appendix[D](https://arxiv.org/html/2608.20106#A4)\);15\+ rounds\(audit pilots v1–v16\) form a calibration cycle covering eight rubrics \(answer\-correctness, distractor plausibility, ambiguity, source\-faithfulness, needs\-source, vague language, label correctness, verbatim copying\)\. For each agent we compute Cohen’sκ\\kappaagainst the rubric;signals withκ<0\.6\\kappa<0\.6are downweighted to advisory\-only\.

The most consequential result is agentB2 ClosedBookSolvability: the human solved≈\\approx12% of gold\-sheet questions without the source fact while the LLM tri\-judge panel reported≈\\approx83% \(κ≈0\.007\\kappa\\approx 0\.007\)\. We read this as a property of the*evaluator*— frontier LLM judges have absorbed enough wine knowledge during pre\-training to over\-attribute “world knowledge” to non\-trivial questions\. We retain the 1,601 B2\-flagged questions with disclosure and turn the blind spot into a calibrated memorisation\-reliance diagnostic via the closed\-book vs\. source\-grounded contrast \(Section[5\.5](https://arxiv.org/html/2608.20106#S5.SS5)\)\.

### 4\.3Release\-cycle results

The 9\-agent audit onrelease\_v1\.1\(3,670 candidate questions\) produced the verdicts summarised in Table[6](https://arxiv.org/html/2608.20106#A2.T6)\(Appendix[B\.7](https://arxiv.org/html/2608.20106#A2.SS7)\): A1, A3, B1, C2 and B3 between them flagged 341 distinct questions fordrop, while C4 re\-labelled difficulty on 1,259 questions and B2 flagged 1,601 as closed\-book solvable but kept them with disclosure\.

After drops, a follow\-up post\-evaluation pass \(Section[5](https://arxiv.org/html/2608.20106#S5)and Appendix[C](https://arxiv.org/html/2608.20106#A3)\) audited the 97 questions that all 16 evaluation configs answered incorrectly: 54 were removed as defects \(wrong ground truth, all\-correct options, duplicate options\), 9 were dropped on borderline\-review and the remaining 34 retained as legitimately hard\. The final released corpus is 3,266 questions\.

The C4\-driven difficulty re\-label was decisive: it shifted the corpus from 14% L3\+L4 items \(the original generator\-assigned distribution\) to51% L3\+L4\(Table[7](https://arxiv.org/html/2608.20106#A2.T7), Appendix[B\.7](https://arxiv.org/html/2608.20106#A2.SS7)\), bringing the hardest\-tier counts above the entry\-tier counts and substantially sharpening difficulty\-stratified analyses \(Section[5\.1](https://arxiv.org/html/2608.20106#S5.SS1)\)\.

## 5Evaluation

We evaluate16 frontier configurationson the 3,266\-question release in a single end\-to\-end run\. The slate combines six within\-family cost pairs \(Claude Opus 4\.7 vs\. Claude Haiku 4\.5, Gemini 2\.5 Pro vs\. Gemini 2\.5 Flash, GPT\-5 vs\. GPT\-5\-mini, Llama 3\.3 70B vs\. Llama 3\.1 8B, Qwen 2\.5 72B vs\. Qwen 2\.5 7B\), four reasoning\-mode configurations \(o3, DeepSeek R1, Gemini 2\.5 Pro thinking, Claude Opus 4\.7 thinking\), and two additional standard models \(DeepSeek V3, Mistral Large\)\. Every config sees the same 3,266 questions, with single\-letter output \(A–D\) atmax\_tokens=5and a five\-stop fallback\. We organise the evaluation around five questions, one per subsection\.

### 5\.1Overall capability ranking

Figure 2:Overall accuracy on the 3,266\-question corpus across 16 configurations\. Reasoning\-mode configs are hatched\.OenoBench cleanly stratifies the slate into three bands \(Figure[2](https://arxiv.org/html/2608.20106#S5.F2)\):frontier\(≥\\geq80%, led byo3 at 83\.6%, with GPT\-5, Gemini 2\.5 Pro, and Claude Opus 4\.7 clustered within 3 pp\);mid\-tier\(mid\-60s to high\-70s\); andsmall\-tier\(low\-50s to low\-60s\)\. No config performs at random; the best\-vs\-worst spread is 30 pp, comparable to GPQA\[[16](https://arxiv.org/html/2608.20106#bib.bib2)\]but on a domain corpus 30×\\timessmaller and fully provenance\-tagged\. Difficulty\-stratified accuracies \(Appendix[C](https://arxiv.org/html/2608.20106#A3), Table[14](https://arxiv.org/html/2608.20106#A3.T14)\) confirm a clean L1→\\rightarrowL4 gradient: 93\.6% on L1 down to58\.7% on L4 expert items, with frontier configs holding 68–71% on L4 and small configs at 39–45%\.

### 5\.2Reasoning\-mode lift

Reasoning\-mode lift is concentrated in a single family \(Figure[7](https://arxiv.org/html/2608.20106#A3.F7), Appendix[C](https://arxiv.org/html/2608.20106#A3)\)\. OnlyDeepSeek R1 vs\. V3\(\+\+6\.8 pp \[\+4\.6, \+8\.8\]\) has a CI excluding zero; the three frontier pairs \(o3 vs\. GPT\-5, Gemini Pro thinking vs\. Pro, Claude Opus thinking vs\. Opus\) are all statistically indistinguishable from zero\. We attribute the asymmetry to headroom: V3 leaves substantial parametric ceiling at 70\.3% which chain\-of\-thought can recover, while frontier models are already near their recallable ceiling\. Per\-difficulty breakdowns \(Appendix[C](https://arxiv.org/html/2608.20106#A3), Table[8](https://arxiv.org/html/2608.20106#A3.T8)\) show the frontier reasoning configs gaining at most 1–2 pp at L4 and flat elsewhere, making the cost\-vs\-lift trade unfavourable except for DeepSeek \(Section[5\.4](https://arxiv.org/html/2608.20106#S5.SS4)\)\.

### 5\.3Self\-preference bias

![Refer to caption](https://arxiv.org/html/2608.20106v1/fig_sps_heatmap.png)

Figure 3:Self\-preference bias\.*Left:*cross\-family accuracy matrix \(rows = evaluator, columns = generator\)\.*Right:*per\-config SPSδ=Acc⁡\(own\)−Acc⁡\(other\)\\delta=\\mathrm\{Acc\}\(\\mathrm\{own\}\)\-\\mathrm\{Acc\}\(\\mathrm\{other\}\)with 95% bootstrap CI\.We compute the*Self\-Preference Score \(SPS\)*per configmmasSPS⁡\(m\)=Acc⁡\(m∣Qown\)−Acc⁡\(m∣Qother\)\\mathrm\{SPS\}\(m\)=\\mathrm\{Acc\}\(m\\mid Q\_\{\\mathrm\{own\}\}\)\-\\mathrm\{Acc\}\(m\\mid Q\_\{\\mathrm\{other\}\}\)whereQownQ\_\{\\mathrm\{own\}\}collects questions generated bymm’s family andQotherQ\_\{\\mathrm\{other\}\}its complement\. A pipeline that successfully neutralises generator fingerprints should produce\|SPS\|\|\\mathrm\{SPS\}\|near zero\.

The result \(Figure[3](https://arxiv.org/html/2608.20106#S5.F3)\) is family\-dependent:Anthropicclusters at\+\+9 to\+\+10 pp \(all CIs above zero\),OpenAIis statistically zero, andGoogleshows an unexpected*negative*−\-6 to−\-10 pp cluster\. The cross\-family matrix \(Figure[3](https://arxiv.org/html/2608.20106#S5.F3), left\) traces the asymmetry: Google\-generated questions are uniformly harder for everyone, while Anthropic\-generated questions are uniformly easier for Anthropic only\. We interpret the Anthropic gap as a residual stylistic fingerprint that pre\-release paraphrase passes did not neutralise, and the Google gap as harder\-by\-phrasing rather than by leak; both are first\-class disclosures with the released corpus\.

### 5\.4Cost efficiency

Figure 4:Total OpenRouter cost \(log scale\) versus accuracy on the full 3,266\-question slate\. Family colours match Figure[2](https://arxiv.org/html/2608.20106#S5.F2); the Pareto frontier \(red dashed\) illustrates the cost\-vs\-quality trade\-off\.The slate spans nearly four orders of magnitude in evaluation cost \(Figure[4](https://arxiv.org/html/2608.20106#S5.F4)\)\. Five configs sit on the Pareto frontier spanning the cost\-quality spectrum:Llama 3\.1 8B\(60\.5 % / $0\.01\),Gemini 2\.5 Flash\(75\.1 % / $0\.12\),GPT\-5\-mini\(78\.4 % / $2\.82\),Claude Opus 4\.7\(81\.0 % / $3\.35\), ando3\(83\.6 % / $11\.80\)\. The two most expensive configs \(Gemini 2\.5 Pro $29\.47, GPT\-5 $21\.90\) are dominated by cheaper Pareto neighbours: their extra spend buys output tokens, not better wine answers\. Reasoning\-mode lift \(Section[5\.2](https://arxiv.org/html/2608.20106#S5.SS2)\) yields a Pareto win only for DeepSeek R1, which is itself dominated by GPT\-5\-mini\. The efficient boundary therefore favours small proprietary plus a single reasoning model over reasoning\-everywhere or scale\-everywhere strategies\.

### 5\.5Closed\-book vs\. source\-grounded performance

The B2 agent reported∼\\sim83% closed\-book solvability against the human reviewer’s∼\\sim12% \(Section[4\.2](https://arxiv.org/html/2608.20106#S4.SS2)\) — itself a measurement of what frontier LLMs already*know*about wine\. We exploit it by partitioning the corpus into 1,601 B2\-flagged and 1,665 contextual / source\-grounded questions and reporting per\-config accuracy on each slice \(Figure[8](https://arxiv.org/html/2608.20106#A3.F8), Appendix[C\.3](https://arxiv.org/html/2608.20106#A3.SS3)\)\.

The contrast is the largest single effect we observe: the mean per\-config closed\-book minus contextual gap is\+\+32\.6 pp\(range\+26\.6\+26\.6to\+39\.6\+39\.6pp, every CI above zero\)\. The B2\-flagged slice overlaps heavily with parametric wine knowledge acquired in pre\-training; the contextual slice is where wine knowledge must be reasoned through the source fact —Claude Opus 4\.7 and o3 hold∼\\sim70% on the contextual slice, small models drop to≈\\approx45%, and the inter\-config spread widens from 27 to 33 pp\. The contextual slice is the more discriminative test of wine reasoning; the B2\-flagged slice is a calibrated indicator of how much apparent “wine knowledge” is parametric recall\.

## 6Applications and Future Work

### 6\.1Closing the wine\-knowledge gap in low\-cost models

Small open\-weight models trail proprietary frontier ones by 20–30 pp on this benchmark, while the broad\-benchmark gap is only 10–15 pp \(Section[5\.4](https://arxiv.org/html/2608.20106#S5.SS4)\)\. The 38,104\-fact corpus is a natural supervised\-fine\-tuning target via a two\-stage recipe: instruction tuning on held\-out question/source\-fact pairs, then LoRA adaptation on the full fact\-question\-source triples in the release schema\. Both stages run on consumer GPUs without proprietary weights or pre\-training compute\. The contextual slice \(Section[5\.5](https://arxiv.org/html/2608.20106#S5.SS5)\) is the harder\-to\-game evaluation surface, so measured gain tracks genuine wine reasoning\. A Llama\-3\.3\-70B\-class adaptation could plausibly close most of the GPT\-5\-mini gap at≈\\approx1% of proprietary inference cost\.

### 6\.2Case\-based evaluation with industry partners

Multiple\-choice tests have ceilings: a perfect OenoBench score does not establish useful behaviour in real workflows\. We are scoping anapplied case\-based datasetwith industry partners along three personas:

- •*Sommelier→\\rightarrowcustomer service:*pairing prompts rated on recommendation, justification, and substitutions\.
- •*Winemaker→\\rightarrowblending decisions:*vintage write\-ups, lab analyses, and regulatory constraints in context; rated on technical correctness, compliance, and stylistic intent\.
- •*Viticulturist→\\rightarrowvineyard decisions:*weather, soil/disease, canopy/yield records; recommendations rated against decisions actually taken in the same vintage\.

### 6\.3Generalising the scrape\-to\-audit pipeline

The pipeline modules — tier\-of\-authority taxonomy, atomic\-fact extraction, multi\-strategy multi\-model generation with per\-model quotas,κ\\kappa\-calibrated multi\-agent audit, and closed\-book pre\-screen — are domain\-agnostic\. Three near\-term targets share wine’s structural features:*horticulture/agronomy*\(regulated appellations, peer\-reviewed journals, certification ladders\);*regulated medicine*\(clinical guidelines, drug formularies; current benchmarks rely on board\-exam questions misaligned with practice\); and*financial regulation*\(FASB, IFRS, jurisdictional filings; CFA/CPA ladders\)\. The released code reduces months of expert authoring to weeks of curation\.

## 7Limitations

#### Snapshot vs\. moving target\.

Wine regulation, classifications, and producer ownership change on a multi\-year cadence \(INAO appellation revisions, DOCG promotions, estate sales\)\. The released corpus is a snapshot dated 2026\-04\-01 and will require periodic re\-extraction; we release the scrapers and provenance metadata so this is mechanically reproducible\.

#### Closed\-book audit calibration\.

The B2 ClosedBookSolvability agent showsκ≈0\.007\\kappa\\approx 0\.007with humans on the gold sheet \(Section[4\.2](https://arxiv.org/html/2608.20106#S4.SS2)\)\. Rather than drop the affected questions we keep them with explicit disclosure and report the closed\-book vs\. contextual contrast \(Section[5\.5](https://arxiv.org/html/2608.20106#S5.SS5)\) as a calibrated memorisation\-reliance diagnostic\.

#### Self\-preference is not fully neutralised\.

The Anthropic\+9\+9pp positive SPS \(Section[5\.3](https://arxiv.org/html/2608.20106#S5.SS3)\) shows the multi\-model strategy did not fully neutralise stylistic fingerprints; downstream Anthropic\-vs\-other comparisons on the full corpus should be read with the per\-config SPS in mind\.

## Acknowledgments and Disclosure of Funding

This work was independently funded by StrategAI\. We thank the open contributors to Wikipedia, Wikidata, INAO, OIV, TTB, UC Davis, USDA Extension, and the wine consortia and academic journals \(*OENO One*,*Vitis*,*AJEV*\) whose work makes a provenance\-grounded benchmark of this scale possible\. We thank the OpenRouter team for the unified\-API gateway used during construction and evaluation\.

## References

- \[1\]M\. Akhtar, O\. Benjelloun, C\. Conforti,et al\.\(2024\)Croissant: a metadata format for ML\-ready datasets\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Cited by:[§A\.6](https://arxiv.org/html/2608.20106#A1.SS6.SSS0.Px1.p1.1),[NeurIPS Paper Checklist](https://arxiv.org/html/2608.20106#Ax1.I1.ix39.p1.1),[§3\.5](https://arxiv.org/html/2608.20106#S3.SS5.p2.1)\.
- \[2\]L\. Chen, Y\. Chen, Y\. Liu, and S\. Ma\(2014\)Recommending wines based on subjective user ratings\.InProceedings of the 8th ACM Conference on Recommender Systems \(RecSys\),Cited by:[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px5.p1.1)\.
- \[3\]J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld, M\. Mitchell, and M\. Gardner\(2021\)Documenting large webtext corpora: a case study on the Colossal Clean Crawled Corpus\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px4.p1.1)\.
- \[4\]T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. Daumé III, and K\. Crawford\(2021\)Datasheets for datasets\.Communications of the ACM64\(12\)\.Cited by:[Appendix A](https://arxiv.org/html/2608.20106#A1.p1.1),[§3\.5](https://arxiv.org/html/2608.20106#S3.SS5.p2.1)\.
- \[5\]N\. Guha, J\. Nyarko, D\. E\. Ho, C\. Ré, A\. Chilton,et al\.\(2023\)LegalBench: a collaboratively built benchmark for measuring legal reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p2.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]R\. T\. Hodgson\(2008\)An examination of judge reliability at a major u\.s\. wine competition\.InJournal of Wine Economics,Vol\.3\.Cited by:[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px5.p1.1)\.
- \[8\]P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. Vidgen\(2023\)FinanceBench: a new benchmark for financial question answering\.External Links:2311\.11944Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p2.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits\(2021\)What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\)\.Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p2.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. Lu\(2019\)PubMedQA: a dataset for biomedical research question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p2.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px2.p1.1)\.
- \[11\]E\. Lefever, I\. Hendrickx, I\. Croijmans, A\. Majid, and A\. van den Bosch\(2018\)A hybrid approach to domain\-independent taxonomy learning applied to the wine domain\.InLanguage Resources and Evaluation \(LREC\),Cited by:[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px5.p1.1)\.
- \[12\]P\. Liang, R\. Bommasani, T\. Lee,et al\.\(2023\)Holistic evaluation of language models\.InTransactions on Machine Learning Research \(TMLR\),Cited by:[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]I\. Magar and R\. Schwartz\(2022\)Data contamination: from memorization to exploitation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px4.p1.1)\.
- \[14\]A\. Panickssery, S\. R\. Bowman, and S\. Feng\(2024\)LLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu,et al\.\(2025\)Humanity’s last exam\.External Links:2501\.14249Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InConference on Language Modeling \(COLM\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.20106#S5.SS1.p1.1)\.
- \[17\]O\. Sainz, J\. A\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. Agirre\(2023\)NLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP,Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px4.p1.1)\.
- \[18\]M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman,et al\.\(2024\)Towards understanding sycophancy in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px3.p1.1)\.
- \[19\]A\. Srivastavaet al\.\(2023\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px3.p1.1)\.
- \[21\]W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan\(2024\)AGIEval: a human\-centric benchmark for evaluating foundation models\.InFindings of the Association for Computational Linguistics: NAACL,Cited by:[§1](https://arxiv.org/html/2608.20106#S1.p1.1),[§2](https://arxiv.org/html/2608.20106#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ADatasheet for OenoBench

We follow the datasheet template of[4](https://arxiv.org/html/2608.20106#bib.bib19)\.

### A\.1Motivation

#### For what purpose was the dataset created?

To evaluate factual knowledge of large language models on a multi\-disciplinary, externally\-validated domain \(wine\), and to provide a benchmark whose construction includes explicit bias controls \(multi\-model generation, multi\-agent audit, self\-preference scoring, closed\-book vs\. source\-grounded partition\)\.

#### Who created the dataset?

The dataset was created by Nikita Khudov \(StrategAI\) for the NeurIPS 2026 Evaluations & Datasets Track\. The lead author holds the WSET Diploma in Wines \(the highest pre\-Master\-of\-Wine qualification of the Wine & Spirit Education Trust\)\. Gold\-sheet ratings used to calibrate the nine audit agents were produced bythree WSET\-certified reviewers\(Diploma, Level 3, Level 2\) including the lead author\.

#### Funding\.

Independently funded by StrategAI\. No external research grants were received\. Total OpenRouter API cost across the entire project — generation pilots, audit cycles, and the final 16\-config evaluation — was$783, paid out\-of\-pocket\.

### A\.2Composition

#### What do the instances represent?

Each instance is a multiple\-choice benchmark question with: question text, 4 options with marked\-correct answer, supporting fact\(s\) with source URL and tier label, generator model identifier, generation strategy, audit\-agent verdicts \(per\-agent pass/warn/fail signals\), and \(for the gold\-sheet subset\) human\-reviewer ratings on eight rubrics\.

#### How many instances are there?

3,266questions inrelease\_v1\.2\. By strategy: fact\-to\-question 1,909, distractor mining 405, template 389, scenario 319, comparative 244\. By difficulty \(post\-relabel\): L1 694, L2 894, L3 678, L4 1,001\. By domain: wine regions 1,108, grape varieties 766, producers 515, viticulture 502, wine business 250, winemaking 187\. The corpus is built from 38,104 atomic facts across 35 scrapers and 4,295 individually\-tracked source records spanning 56 unique top\-level domains\.

#### What data does each instance consist of?

Structured Postgres records mirroring the schema documented in Section[3](https://arxiv.org/html/2608.20106#S3)and Appendix[B](https://arxiv.org/html/2608.20106#A2)\.

#### Recommended data splits\.

The dataset ships as a singletestsplit \(this is an evaluation\-only benchmark\)\. Two analytic partitions are documented and recommended for use: \(i\) the*closed\-book*slice \(1,601 questions taggedclosed\_book\_solvable\) and the*contextual*slice \(1,665 remaining questions\), partitioned by audit agent B2; and \(ii\) per\-generator held\-out splits for self\-preference analysis \(Section[5\.3](https://arxiv.org/html/2608.20106#S5.SS3)\), defined dynamically by thegenerator\_familycolumn\.

#### Errors, noise, redundancies?

Documented in Section[4](https://arxiv.org/html/2608.20106#S4)\. Per\-agent fail rates andκ\\kappaversus the human gold sheet are released alongside the corpus\. The 1,601 B2\-flagged questions are kept with disclosure because LLM\-judge calibration with humans on closed\-book solvability is unreliable \(κ≈0\.007\\kappa\\approx 0\.007\)\.

#### Self\-contained, or relies on external resources?

Self\-contained: question text, supporting facts, and source URLs are all included\. Source URLs may rot over time; we mitigate by archiving Tier\-1 source pages and recording the access timestamp\.

### A\.3Collection process

#### How was the data acquired?

By 35 provenance\-verified web scrapers \(released insrc/scrapers/\)\. Sources span government registries \(INAO, TTB, OIV\), inter\-governmental bodies, university research groups \(notably UC Davis, USDA Extension\), Wikipedia, Wikidata, peer\-reviewed journals \(*OENO One*,*Vitis*,*AJEV*\), and curated open datasets\. Scraping was rate\-limited and used the disclosed user agentOenoBench\-Research/1\.0 \(academic wine benchmark\)\.

#### Time frame\.

Fact extraction: 2026\-03 through 2026\-04\. Question generation: 2026\-04 through 2026\-05\-03\. Evaluation: 2026\-05\-03\.

#### Ethical review\.

The work is non\-IRB\-eligible at our institution because \(a\) no human subjects were enrolled in research, \(b\) gold\-sheet ratings were produced by three WSET\-certified reviewers including the lead author, and \(c\) all source data is public\-record information about commercial wine entities\. Human review of LLM\-generated questions is treated as data\-quality assurance, not as human\-subjects research\.

### A\.4Preprocessing / cleaning / labelling

Atomic\-fact extraction with entity tagging; per\-source paraphrase to avoid verbatim copy; near\-duplicate suppression at cosine≥0\.92\\geq 0\.92; nine\-agent automated audit; human gold\-sheet rating on a stratified 50\-question subset per release, produced independently by three WSET\-certified reviewers \(Diploma, Level 3, Level 2\) with the lead author resolving disagreements\.

### A\.5Uses

#### Recommended uses\.

Evaluation of LLM factual knowledge in the wine domain; ablation studies of bias\-mitigation techniques in benchmark construction; study of self\-preference effects in LLM\-as\-judge pipelines; calibration of automated multi\-agent QA frameworks against expert human review\.

#### Discouraged uses\.

Direct deployment as a wine\-recommendation or purchasing system without separate alignment, age\-gating, and jurisdictional advertising review; fine\-tuning on the benchmark itself for ranking purposes \(would invalidate the benchmark for the trained model\)\. The scrape\-to\-audit pipeline should not be retargeted to high\-stakes domains \(medicine, law\) without comparable expert human review of the retargeted source taxonomy\.

#### Alcohol context\.

Wine is an alcoholic beverage\. OenoBench is a knowledge\-evaluation tool, not a consumer\-facing product\. A high OenoBench score reflects factual / reasoning capability about wine; it does not endorse downstream deployment as a drinking\-recommendation system, which would require separate alignment for health information, age\-gating, and jurisdictional advertising rules\.

### A\.6Distribution

#### Will the dataset be distributed?

#### Citation\.

```
@inproceedings{khudov2026oenobench,
  title  = {OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded
            Evaluation of Large Language Models},
  author = {Khudov, Nikita},
  booktitle = {Advances in Neural Information Processing Systems
               (NeurIPS), Datasets and Benchmarks Track},
  year   = {2026}
}
```

### A\.7Maintenance

#### Maintainer\.

Nikita Khudov,nikitahudov@gmail\.com\.

#### Update cadence\.

Wine regulation changes on a multi\-year cadence; we plan annual re\-extraction\. Scraper code is released so the community can re\-extract independently\. Corpus versioning followsrelease\_vX\.Y\(major release / minor patch\) with all prior versions retained on HuggingFace\.

#### Errata\.

A public errata log will be maintained alongside the HuggingFace release; corrections to questions, facts, audit verdicts, or difficulty labels will be versioned and dated\. Reports of factual errors are accepted via GitHub issues and will be evaluated against the source URL\.

## Appendix BAdditional Methodology Detail

This appendix expands the dataset\-construction and audit pipelines: the atomic\-fact extractor \(B\.1\), the Wikidata SPARQL choice \(B\.2\), question\-generation prompts \(B\.3\), the closed\-book gate \(B\.4\), and audit architecture, agent specifications, and release\-cycle results \(B\.5–B\.7\), with the calibration philosophy that informs them \(B\.8\)\.

### B\.1Atomic fact extraction pipeline

![Refer to caption](https://arxiv.org/html/2608.20106v1/figures/diagram_fact_processing.png)Figure 5:Atomic\-fact extraction pipeline\. Inputs are heterogeneous \(HTML pages, RDF graphs, SPARQL responses, CSV dumps\); outputs are uniform single\-assertion sentences with entity tags, domain label, source URL and tier\.The atomic\-fact extractor \(src/scrapers/\_fact\_processing\.py\) applies five sequential filters to each candidate sentence:

1. 1\.Sentence split\.NLTK Punkt tokeniser; further split at explicit conjunctions \(“and”, “but”, “while”\) when both clauses have an independent verb\. Maximum atomic length 30 words; reject longer\.
2. 2\.Reference resolution\.Pronouns and demonstratives are replaced with their entity referent from article context \(paragraph\-level coreference using a rule\-based resolver tuned on Wikipedia leads\)\. Sentences with unresolved*it/this/that*are dropped\.
3. 3\.Domain classification\.Lexicon\-based \+ rule\-based classifier maps each fact to one of the six domain pillars; ambiguous facts \(no anchor term in any domain lexicon\) are dropped\.
4. 4\.Length and predicate validation\.Reject facts<<5 words,\>\>50 words, missing a verb, or carrying dangling references / fragments\. The 5–30 word band is the strict filter; the 31–50 band is kept with reduced confidence \(910 facts\)\.
5. 5\.On\-topic filter\.Region\-specific keyword sets prevent cross\-contamination \(e\.g\. Austrian content in a Bordeaux scraper\)\. Fail⇒\\Rightarrowdrop\.

A 2026\-04 audit found that 19 of an original 35 scrapers contained hard\-coded LLM\-generated facts disguised as scraped data\. The fix required rebuilding all 19 against live sources; the lesson preserved for future retargeting is that*provenance auditing must run on the scrapers themselves*, not only on the collected facts\.

### B\.2Wikidata SPARQL: P17 vs P131\*

A common bug in early scrapers used the transitive administrative\-parent property \(P131\*\) to bind questions to a country\. This introduced severe cross\-region contamination, e\.g\. Austrian appellations appearing in a Bordeaux scraper because the SPARQL graph traversal walked through shared ancestor entities\. Switching to the direct country relation \(P17\) eliminated cross\-country leakage at the cost of∼\\sim3% lost coverage on borderline\-region entities\. The released SPARQL templates useP17exclusively and we recommend the same discipline for any pipeline retargeted to a domain with hierarchical political subdivisions\.

### B\.3Question\-generation prompts

The five strategies share a common JSON output schema enforced via Pydantic with a 3\-tier extraction fallback \(markdown\-fenced, raw JSON, prefix\-pattern\)\. Per\-strategy prompt summaries:

- •Fact\-to\-question:"Given a single atomic wine fact, write a 4\-option multiple\-choice question whose unique correct answer is grounded in the fact\. Paraphrase the fact; do not copy contiguous spans of\>\>5 words\. Distractors must be confusable same\-category entities, not random alternatives\."
- •Comparative:given two related facts \(matched on entity type, country/sub\-domain\), "ask which differs in the named attribute or what both share\. Avoid iconic\-vs\-non\-iconic pairings\."
- •Scenario synthesis:given a cluster of 3–5 facts, "write a brief domain\-appropriate professional scenario \(sommelier service / winemaker decision / viticulturist call / business/regulatory choice\) with a unique correct answer that requires*all*the cluster facts to derive\."
- •Distractor mining:given a fact \+ 3 confusable entities, "write a question whose 3 wrong options are these entities, ranked by ascending plausibility\."
- •Template:45 deterministic parameter substitution templates; no LLM call\. A subset is paraphrased post\-hoc by Gemini Pro to defeat the A4 stylometric fingerprint detector\.

Full prompt texts are insrc/generators/\_prompts\.py\.

### B\.4Closed\-book gate specification

The closed\-book pre\-screen \(src/generators/\_closed\_book\_gate\.py\) runs a per\-difficulty model:

- •L1: Claude Haiku 4\.5 reads the question with no source fact\. Correct answer⇒\\Rightarrowtagclosed\_book\_solvable\.
- •L2: Claude Sonnet 4\.6 same protocol\.
- •L3: Claude Opus 4\.7 same protocol; the corpus quota cap fires here\.
- •L4: gate is skipped by protocol; expert items are expected to be answerable only with the source fact\.

The gate’s role in generation is to enforce a corpus\-level cap on closed\-book solvable items; early pilots used a 25% cap, but it wasraised to 50%after the gold\-sheet calibration revealed that the LLM panel over\-attributes closed\-book solvability relative to humans \(Section[4\.2](https://arxiv.org/html/2608.20106#S4.SS2)\), so the higher cap retains the items that are in fact discriminating in evaluation\. The audit’s B2 agent later re\-runs the check at corpus scale on a tri\-judge panel\. Failures are not dropped; they are reserved into the closed\-book slice and used for the analytic partition in Section[5\.5](https://arxiv.org/html/2608.20106#S5.SS5)\.

### B\.5Audit architecture

![Refer to caption](https://arxiv.org/html/2608.20106v1/figures/diagram_quality_assurance.png)Figure 6:Multi\-agent audit architecture\. Candidate questions are fanned out to four teams; each team independently emits per\-question verdicts\. Criticalfailsignals route the question to the drop pool; C4 difficulty disagreements route to relabel; otherwise the question is kept inrelease\_v1\.2\. Every agent is calibrated against the human gold sheet via Cohen’sκ\\kappa\.
### B\.6Audit agent specifications

Each agent is a versioned Python module undersrc/qa/agents/\. Agents emit\(question\_id, agent\_id, version, severity, payload\_json\)tuples to theaudit\_findingstable;severityis one of \{pass,warn,fail\}\. Versioning is strict: the \(run\_id,question\_id,agent\_id,version\) tuple is unique, so re\-running an agent at a new version produces additive findings rather than overwriting\. We document each agent’s threshold and gold\-sheetκ\\kappahere\.

Table 5:Audit\-agent thresholds and gold\-sheet calibration\.The B2 row is the calibration finding discussed in Section[4\.2](https://arxiv.org/html/2608.20106#S4.SS2): the LLM panel and the human reviewer disagree on what counts as “world\-knowledge solvable”, because frontier LLMs have absorbed substantial wine knowledge during pre\-training\. We retain the agent as a measurement of the LLM panel’s prior, and report the contextual / closed\-book partition \(Section[5\.5](https://arxiv.org/html/2608.20106#S5.SS5)\) that turns the disagreement into a useful diagnostic\.

### B\.7Release\-cycle results

Table[6](https://arxiv.org/html/2608.20106#A2.T6)summarises the verdicts produced when the 9\-agent audit was applied torelease\_v1\.1\(3,670 candidate questions\); these are the numbers cited in Section[4\.3](https://arxiv.org/html/2608.20106#S4.SS3)\.

Table 6:Audit findings onrelease\_v1\.1and the resulting drop / re\-label policy\.Audit signalTriggered FAILActionA1 LexicalHygiene \(vague phrasing\)60dropA3 FactEcho \(verbatim copy\)63dropB1 TriJudgeAnswer \(wrong answer key\)47dropC2 CategoryLeak \(wine\-category leak\)9dropB3 UbiquityRisk \(grape×\\timesregion ambiguity\)183dropDistinct dropped \(with overlap\)341C4 DifficultyAudit \(relabel from 1,252 LLM hits \+ 7 human\)1,259re\-labelB2 ClosedBookSolvability \(kept w/ disclosure\)1,601discloseTable[7](https://arxiv.org/html/2608.20106#A2.T7)shows the corpus\-level effect of the C4 difficulty re\-label, which shifted the corpus from 14% L3\+L4 items to 51% L3\+L4\.

Table 7:Difficulty distribution before and after C4 re\-label\.
### B\.8Calibration philosophy

We emphasise the design choice underlying these numbers\. A naive “automated audit” interpretation would have dropped all 1,601 B2\-flagged questions, which would have removed nearly half the corpus and biased the remainder toward whatever idiosyncratic items the LLM panel happened not to know\. By calibrating B2 against humans and treating it as an evaluator\-side signal, we preserve dataset value while giving downstream users a principled way to interpret leakage\. The audit’s role, on this reading, is not to be the final arbiter of quality, but to surface signals*that humans validate*as quality\-relevant, and to disclose clearly the signals where humans and LLMs disagree\.

## Appendix CFull Evaluation Tables

This appendix gives the per\-config tables and supporting figures behind each Section[5](https://arxiv.org/html/2608.20106#S5)subsection: reasoning\-mode lift \(C\.1\), self\-preference bootstrap CIs \(C\.2\), the closed\-book vs\. source\-grounded contrast \(C\.3\), and per\-domain \(C\.4\), per\-strategy \(C\.5\), and per\-difficulty \(C\.6\) breakdowns of the overall ranking\. The cost ledger that supports Section[5\.4](https://arxiv.org/html/2608.20106#S5.SS4)is in Appendix[E](https://arxiv.org/html/2608.20106#A5)\.

### C\.1Reasoning\-mode lift

Figure 7:Reasoning\-mode vs\. standard accuracy for four families;δ\\deltain pp with 95% bootstrap CI \(1,000 resamples\)\. Only DeepSeek’s CI excludes zero\.Table 8:Reasoning lift per difficulty tier \(δ\\delta= thinking−\-standard, pp\)\.
### C\.2Self\-preference bootstrap CIs

Table 9:Per\-config Self\-Preference Score with 95% bootstrap CI \(1,000 resamples\)\.nncolumns are the question counts in each pool: own==questions whose generator family matches the config’s family; other==questions whose generator is a different non\-template LLM\. Templates \(n=389n\{=\}389\) are excluded from both pools, so own\+\+other==2,877 across every row\. DeepSeek and Mistral have no own\-family questions inrelease\_v1\.2and are omitted\.Table 10:Per\-tier Self\-Preference Scoreδ\\delta\(pp\)\. The overall SPS in Table[9](https://arxiv.org/html/2608.20106#A3.T9)is decomposed by the question’s post\-relabel difficulty tier; each cell is own\-Acc minus other\-Acc on the questions of that tier\.Boldmarks cells whose 95% bootstrap CI excludes zero\.#### Per\-tier reading\.

The Anthropic positive SPS is*not*concentrated at any single difficulty tier — it spikes at both L2 \(\+8\.5\+8\.5to\+10\.2\+10\.2across the three Claude configs, all CIs above zero\) and L4 \(\+9\.5\+9\.5to\+10\.9\+10\.9, two of three CIs above zero\), while L1 \(near ceiling\) and L3 are flat\. A pure question\-difficulty asymmetry would predict the gap to widen monotonically as accuracy falls; instead the gap appears at*both*ends of the non\-trivial range, which is more consistent with a residual stylistic fingerprint than with Anthropic having generated easier questions\. Google’s negative SPS is overwhelmingly an L4 phenomenon \(Gemini Pro / Pro thinking / Flash all−10\-10to−14\-14pp at L4, with CIs excluding zero; L1–L3 within±6\\pm 6pp\), consistent with Gemini\-authored hardest\-tier questions being uniformly hard rather than the model being penalised on its own style\. OpenAI’s near\-zero overall SPS is the result of positive L1/L2 and negative L4 cells averaging out, not stylistic neutrality\.

### C\.3Closed\-book vs\. source\-grounded performance

Figure 8:Per\-config accuracy on B2\-flagged closed\-book solvable questions \(n=1,601n\{=\}1\{,\}601\) vs\. contextual / source\-grounded questions \(n=1,665n\{=\}1\{,\}665\); every config gains\+26\.6\+26\.6to\+39\.6\+39\.6pp on the closed\-book slice\.Table 11:Closed\-book vs\. contextual accuracy gap \(pp\) within each difficulty tier\. A positive gap means the model scores higher on B2\-flagged closed\-book solvable items than on contextual items at the same tier\. Per\-tier B2\-flagged / contextual question counts: L1 680 / 13, L2 333 / 561, L3 472 / 206, L4 116 / 885\. The Mean row reports 95% bootstrap CIs \(1,000 resamples; question\-level resampling within each pool\)\.#### Within\-tier reading\.

The\+32\.6\+32\.6pp aggregate gap is preserved at L2–L4 \(\+31\.8\+31\.8,\+25\.1\+25\.1,\+26\.4\+26\.4pp; every CI above zero\) and effectively vanishes only at L1, where the contextual pool collapses ton=13n\{=\}13questions and ceiling effects dominate\. The L4 result is the most informative: the contextual pool is large \(n=885n\{=\}885\) and the gap is still\+26\.4\+26\.4pp with a tight CI, so the closed\-book vs\. source\-grounded contrast is not an artefact of difficulty composition — it is a stable property of the slice across non\-trivial tiers, consistent with B2 identifying questions recoverable from pre\-training rather than questions that happen to be easier\-stated\.

### C\.4Per\-domain breakdown

Table 12:Per\-config per\-domain accuracy \(%\)\.![Refer to caption](https://arxiv.org/html/2608.20106v1/fig_domain_heatmap.png)Figure 9:Per\-domain accuracy as a heatmap\.*Wine business*is the hardest domain across every tier \(mean 56\.5%\) — both because business\-domain facts are sparser in the training distribution and because the C4 difficulty re\-label promoted business questions that are computationally harder \(multi\-fact reasoning over regulation\)\.
### C\.5Per\-strategy breakdown

Table 13:Per\-config per\-strategy accuracy \(%\)\.![Refer to caption](https://arxiv.org/html/2608.20106v1/fig_strategy_heatmap.png)Figure 10:Per\-strategy accuracy heatmap\. Templates are the easiest strategy uniformly \(89\.5% mean\), even for small models, because they are deterministic single\-fact recall questions\. The fact\-to\-question strategy is hardest because its distractors are sampled from confusable same\-category entities; comparative questions add the constraint that the model must integrate two facts\.
### C\.6Per\-difficulty breakdown

Table 14:Per\-config per\-difficulty accuracy \(%\)\.Figure 11:Difficulty calibration: distribution of accuracy across the 16 configs at each level\. The medians sit at L1 96%, L2 70%, L3 81%, L4 60%\.#### On the L2 / L3 / L4 means in Table[14](https://arxiv.org/html/2608.20106#A3.T14)\.

Two features of the column averages deserve comment\. First, the L2 mean \(68\.8%\) is*below*the L3 mean \(78\.7%\) — an apparent inversion of the difficulty ordering\. This is an artefact of the C4 difficulty re\-label \(Section[4](https://arxiv.org/html/2608.20106#S4)\): many questions originally labelled L2 by their generator were promoted to L3, leaving behind a residual L2 set that is concentrated in the*wine\_business*pillar \(the hardest domain by a wide margin, Table[12](https://arxiv.org/html/2608.20106#A3.T12)\)\. The L3 set, in contrast, was enriched with cleanly\-stated multi\-fact items from the higher\-quality scrapers\. Second, the L3→\\rightarrowL4 drop is real and large \(78\.7%→\\rightarrow58\.7%, a 20 pp fall\), reflecting that L4 is the expert tier where the source fact is genuinely required to answer the question; that fall is also where the inter\-config spread is widest \(Section[5\.1](https://arxiv.org/html/2608.20106#S5.SS1)\), making L4 the most discriminating slice for distinguishing models\. Both effects are intended outcomes of the audit\-driven difficulty re\-label rather than miscalibration\.

## Appendix DApplication Screenshots

### D\.1Human\-review web application

The human\-review web application is the front\-end through which WSET\-Diploma reviewers rate stratified gold\-sheet questions and the through which the agent\-vs\-humanκ\\kappavalues reported in Section[4\.2](https://arxiv.org/html/2608.20106#S4.SS2)are collected\. It is a Flask application with two layers of authentication \(outer HTTP Basic for shared\-link access, inner per\-reviewer session cookies\) and writes ratings into a separatehuman\_reviewsPostgres table to keep gold\-sheet labels independent of automated audit findings\. Each reviewer is associated with a WSET certification level recorded at registration, and the app supports multi\-reviewer inter\-rater agreement \(κ\\kappa\) computation out of the box\.

![Refer to caption](https://arxiv.org/html/2608.20106v1/figures/screenshots/review_login.png)Figure 12:Sign\-in / register screen\. Outer HTTP Basic Auth gates shared\-link access; the inner form establishes a per\-reviewer session for IRR attribution\.![Refer to caption](https://arxiv.org/html/2608.20106v1/figures/screenshots/review_dashboard.png)Figure 13:Reviewer dashboard\. Active batches are listed; clicking a batch enters the rubric\-scoring flow \(eight\-rubric grid withpass/warn/failper rubric, inline reviewer\-guide hints, and time\-on\-page tracking\)\.

## Appendix EReleased Artifacts and Reproducibility

### E\.1Released artifacts

- •Dataset \(HuggingFace\)\.[https://huggingface\.co/datasets/oenobench/oenobench](https://huggingface.co/datasets/oenobench/oenobench)—release\_v1\.2, 3,266 questions, Parquet test split,README\.mddatasheet,croissant\.jsonmanifest\. License: CC\-BY\-SA\-4\.0\. Versioning is permanent: prior releases remain accessible via revision tags\.
- •Code \(GitHub\)\.[https://github\.com/nikitahudov/oenobench](https://github.com/nikitahudov/oenobench)— full pipeline: 35 scrapers, 5 question generators, 14 audit agents, the human review web app, and the monitoring dashboard\. Apache 2\.0\.
- •Process log\.docs/PROCESS\_LOG\.md— chronological lab\-notebook of every phase shipped, with sources, methodology, quality\-control counts, and decision rationale, in the format prescribed for the methodology sections of this paper\.
- •Audit reports\.docs/QUALITY\_AUDIT\_REPORT\.md,docs/RELEASE\_V1\_2\_AUDIT\_ACTIONS\.md,docs/GOLD\_CALIBRATION\_ANALYSIS\.md,data/reports/zero\_correct\_97\_audit\.md\.
- •Eval reports\.data/reports/eval\_release\_v1\_2\_full\_cleaned\.md\(the 16\-config × 3,266\-Q run reported in this paper\)\.

### E\.2Reproducibility scripts

The construction pipeline is driven by a small number of shell scripts inscripts/:

- •run\_all\_scrapers\.sh— orchestrates the 35 scrapers end\-to\-end\. Each scraper writes a timestamped log todata/logs/\.
- •run\_release\_v1\_build\.sh— full generation pipeline invocation; idempotent and resume\-safe via stamped tags\.
- •export\_release\_v1\_2\_to\_parquet\.py— produces the HuggingFace Parquet directly from Postgres; the released file is the byte\-identical output of this script\.
- •build\_smart\_review\_sheet\.py— generates a 50\-question stratified gold\-sheet review batch \(random \+ critical\-FAIL \+ borderline\-WARN strata\)\.
- •tag\_audit\_actions\.py— applies the audit drop / relabel policy to a tagged corpus; this is the script that produced therelease\_v1\.2state fromrelease\_v1\.1\.

### E\.3Database schema

The released schema is initialised by the migrations underconfig/postgres/\(six idempotent files, applied in order on container startup\):init\.sql\(sources, facts, questions, question\_facts, generation\_metadata, evaluation\_runs, evaluation\_answers, human\_reviews\),002\_audit\_schema\.sql\(audit\_runs, audit\_findings, audit\_gold\_labels, audit\_severity enum\),003\_sample\_schema\.sql\(mirror schema for the Phase 5 sample\-DB eval\),003\_cb\_reserve\.sql\(closed\-book reserve pool tagging\),004\_eval\_telemetry\.sql\(provider, tokens, reasoning\_config, latency columns onevaluation\_answers\),005\_or\_cost\_telemetry\.sql\(OpenRouter authoritative cost columns\)\.

The released schema is documented in code \(config/postgres/\); the load\-bearing tables aresources\(registry\),facts\(atomic facts grounded by source\),questions\+question\_facts\(corpus and fact\-question links\),generation\_metadata\(per\-question generator and strategy\),audit\_findings\(per\-agent verdicts\), andhuman\_reviews\(gold\-sheet ratings\)\.

### E\.4Compute and cost

The full evaluation reported in Section[5](https://arxiv.org/html/2608.20106#S5)ran in120 minutes 34 secondson a single VM driving 16 OpenRouter configurations in parallel\.Total LLM API spend across the entire project was $783, paid out\-of\-pocket to OpenRouter and covering all generation pilots, audit cycles, gold\-sheet calibration runs, and the final 16\-config evaluation\. No GPU compute was used; all model inference was via OpenRouter \(a unified API gateway\)\. The release artefacts include all per\-call telemetry \(provider, tokens, reasoning configuration, p50/p95 latency, OR\-authoritative cost\) for the full evaluation run; the per\-config cost ledger is reproduced inline as Table[15](https://arxiv.org/html/2608.20106#A5.T15)\.

Table 15:Per\-config cost ledger for the full 16\-config evaluation \(3,266 questions per config\)\. Effective OpenRouter spend\.
### E\.5Per\-source licensing

Section[3\.2](https://arxiv.org/html/2608.20106#S3.SS2)states that every fact in the corpus traces to a public\-record source or to a source distributed under a licence that permits scraping and redistribution\. We give the per\-source breakdown in Table[16](https://arxiv.org/html/2608.20106#A5.T16), grouping the 35 scrapers’ source families by tier and licence basis\. Each row maps to one or more rows in the releasedsourcestable, where per\-rowname,url,tier, and access timestamp are recorded for every fact \(facts\.source\_id→\\rightarrowsources\.id\)\. Wikipedia and Wikidata together account for 65\.3% of facts and propagate their share\-alike / public\-domain dedications through therelease\_v1\.2CC\-BY\-SA\-4\.0 licence\. Government records \(Tier 1, 19\.6% of facts\) are either explicit open\-data licences \(INAO Licence Ouverte 2\.0, EU re\-use Decision 2011/833/EU\) or US\-Government works in the public domain \(17 U\.S\.C\. §105: TTB, USDA Extension, UC IPM\)\.

Table 16:Per\-source licensing for OenoBench’s 35 scrapers, grouped by source family\. “Share” is the percentage of the 38,104\-fact corpus contributed; “Tier” is the source\-of\-authority label from Section[3\.2](https://arxiv.org/html/2608.20106#S3.SS2)\. Counts are computed directly from thefacts⋈\\bowtiesourcesjoin in the released schema\.The released corpus, including this paper’s quoted statistics, is distributed underCC\-BY\-SA 4\.0\. The construction code is released underApache 2\.0\. Verbatim source text is never stored: every fact is paraphrased atomically \(Section[3\.3](https://arxiv.org/html/2608.20106#S3.SS3)\) and a paraphrase guard at audit time \(agent A3\) re\-checks for verbatim leakage\. We retainaccessed\_dateand a per\-sourcecontent\_datefor every fact, so dataset users can audit freshness against original sources\.

### E\.6Test status

771 of 771 unit tests pass onmainas of the camera\-ready build\. Tests cover the fact\-processing pipeline, the five generation strategies, all 14 audit agents, the orchestrator, the closed\-book gate, the eval harness \(slot dispatch, cost computation, reasoning\-mode prompt assembly\), and the review\-app endpoints\.

## Appendix FConstruction Timeline

The full chronological lab notebook of construction is indocs/PROCESS\_LOG\.md\. We summarise the major dated milestones here for paper\-traceable provenance\.

Table 17:Construction\-and\-evaluation timeline \(selected milestones\)\.
## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: Section[1](https://arxiv.org/html/2608.20106#S1)lists four contributions, and each is supported by a numbered section: dataset construction \(Section[3](https://arxiv.org/html/2608.20106#S3)\), automated audit \(Section[4](https://arxiv.org/html/2608.20106#S4)\), bias\-aware evaluation \(Section[5](https://arxiv.org/html/2608.20106#S5)\)\. The abstract’s headline numbers \(3,266 questions, 16\-config eval, \+32\.6 pp closed\-book vs\. contextual gap\) are reproduced with full tables in Appendix[C](https://arxiv.org/html/2608.20106#A3)\.
5. 2\.Limitations
6. Question: Does the paper discuss the limitations of the work performed by the authors?
7. Answer:\[Yes\]
8. Justification: Section[7](https://arxiv.org/html/2608.20106#S7)discusses three limitations explicitly: snapshot vs\. moving target, B2 audit calibration ceiling, and residual self\-preference\.
9. 3\.Theory assumptions and proofs
10. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
11. Answer:\[N/A\]
12. Justification: This is a dataset\-and\-benchmark paper; no formal theoretical results are claimed\.
13. 4\.Experimental result reproducibility
14. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper?
15. Answer:\[Yes\]
16. Justification: Appendix[E](https://arxiv.org/html/2608.20106#A5)lists the construction scripts, the dataset URL, the model slate, the per\-call telemetry available alongside the released corpus \(including run id, wall time, OR\-authoritative cost, and exact config strings\), and the deterministic build pipeline\. Section[5](https://arxiv.org/html/2608.20106#S5)specifies the 16\-config slate, the single\-letter \(A–D\) output protocol, and the bootstrap procedure\.
17. 5\.Open access to data and code
18. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
19. Answer:\[Yes\]
21. 6\.Experimental setting/details
22. Question: Does the paper specify all the training and test details necessary to understand the results?
23. Answer:\[Yes\]
24. Justification: Section[5](https://arxiv.org/html/2608.20106#S5)specifies the 16\-config slate, single\-letter output protocol \(A–D,max\_tokens=5, five\-stop fallback\)\. Appendix[C](https://arxiv.org/html/2608.20106#A3)gives full per\-config × per\-domain × per\-strategy × per\-difficulty tables\. Appendix[B](https://arxiv.org/html/2608.20106#A2)gives audit\-agent thresholds and prompt summaries\.
25. 7\.Experiment statistical significance
26. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27. Answer:\[Yes\]
28. Justification: Reasoning lift, self\-preference scores, and closed\-book vs\. contextual deltas are all reported with 95% bootstrap confidence intervals \(1,000 resamples\)\. Section[5](https://arxiv.org/html/2608.20106#S5)states the procedure; Appendix[C](https://arxiv.org/html/2608.20106#A3)\(Tables[8](https://arxiv.org/html/2608.20106#A3.T8),[9](https://arxiv.org/html/2608.20106#A3.T9),[10](https://arxiv.org/html/2608.20106#A3.T10),[11](https://arxiv.org/html/2608.20106#A3.T11)\) reports per\-config CIs\.
29. 8\.Experiments compute resources
30. Question: For each experiment, does the paper provide sufficient information on the computer resources needed to reproduce the experiments?
31. Answer:\[Yes\]
32. Justification: Appendix[E](https://arxiv.org/html/2608.20106#A5)reports the full 16\-config evaluation wall time \(120 min 34 s\), total LLM calls \(52,256\), and effective evaluation cost \($98\.33\), and the deployment topology \(single VM driving 16 OpenRouter configurations in parallel; no GPU compute\)\. The total OpenRouter spend across the entire project — generation pilots, audit cycles, gold\-sheet calibration, and the final evaluation — was$783, paid out\-of\-pocket and itemised in Appendix[E](https://arxiv.org/html/2608.20106#A5)and the Datasheet \(Appendix[A](https://arxiv.org/html/2608.20106#A1), Motivation\)\.
33. 9\.Code of ethics
34. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics?
35. Answer:\[Yes\]
36. Justification: All sources used are public\-record; no human subjects research was conducted \(gold\-sheet ratings were produced by three WSET\-certified reviewers including the lead author, treated as data\-quality assurance\); responsible\-use guidance is given in the Datasheet \(Appendix[A](https://arxiv.org/html/2608.20106#A1), “Uses”\) and in Section[7](https://arxiv.org/html/2608.20106#S7)\.
37. 10\.Broader impacts
38. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
39. Answer:\[Yes\]
40. Justification: The Datasheet \(Appendix[A](https://arxiv.org/html/2608.20106#A1)\) covers alcohol\-context and discouraged uses; source\-licensing and the non\-fabrication guarantee are documented in Section[3\.2](https://arxiv.org/html/2608.20106#S3.SS2)and Table[16](https://arxiv.org/html/2608.20106#A5.T16)\(Appendix[E](https://arxiv.org/html/2608.20106#A5)\); the corpus contains only public\-record information about commercial wine entities \(no private individuals; Datasheet “Collection process”\)\. Section[6](https://arxiv.org/html/2608.20106#S6)discusses positive applications including specialist\-model fine\-tuning and pipeline retargeting to other expert domains\.
41. 11\.Safeguards
42. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse?
43. Answer:\[Yes\]
44. Justification: The dataset is text\-only multiple\-choice questions with public\-record entity names; no scraped images or personal\-data risks\. We document discouraged uses in the Datasheet \(Appendix[A](https://arxiv.org/html/2608.20106#A1), “Uses”\) including the recommendation against retargeting the pipeline to high\-stakes domains without expert review\.
45. 12\.Licenses for existing assets
46. Question: Are the creators or original owners of assets used in the paper properly credited and are the license and terms of use explicitly mentioned and properly respected?
47. Answer:\[Yes\]
48. Justification: The full per\-source licensing table \(Table[16](https://arxiv.org/html/2608.20106#A5.T16)in Appendix[E](https://arxiv.org/html/2608.20106#A5)\) credits every source family with licence and legal basis \(Wikipedia CC\-BY\-SA 4\.0, Wikidata CC0, INAO Licence Ouverte 2\.0, TTB / USDA / UC IPM public domain \(US Government work, 17 USC §105\), EU re\-use Decision 2011/833/EU, OENO One CC\-BY 4\.0, etc\.\) and is also summarised in Section[3\.2](https://arxiv.org/html/2608.20106#S3.SS2)\. Per\-fact source URL, accessed\-date, and content\-date are released alongside the corpus\.
49. 13\.New assets
50. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
51. Answer:\[Yes\]
52. Justification: A full Datasheet is in Appendix[A](https://arxiv.org/html/2608.20106#A1); a Croissant manifest\[[1](https://arxiv.org/html/2608.20106#bib.bib21)\]ships with the HuggingFace release; all 14 audit agents \(9 always\-run, 5 escalation\-gated; Section[4\.1](https://arxiv.org/html/2608.20106#S4.SS1)\), 5 generation strategies, 5 generator\-model families, and 35 scrapers are documented in source\.
53. 14\.Crowdsourcing and research with human subjects
54. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation?
55. Answer:\[N/A\]
56. Justification: No crowdsourcing or third\-party human\-subjects research\. Gold\-sheet ratings were produced by three WSET\-certified reviewers \(Diploma, Level 3, Level 2\) including the lead author; the review web app screenshots in Appendix[D](https://arxiv.org/html/2608.20106#A4)document the rubric and instructions\.
57. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
58. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether IRB approvals were obtained?
59. Answer:\[N/A\]
60. Justification: As noted in Appendix[A](https://arxiv.org/html/2608.20106#A1), the work involves no human subjects research distinct from data\-quality assurance by three WSET\-certified reviewers including the lead author\.
61. 16\.Declaration of LLM usage
62. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research?
63. Answer:\[Yes\]
64. Justification: LLMs are central to the methodology \(multi\-strategy question generation across five model families, nine\-agent automated audit including a tri\-judge panel, closed\-book solvability pre\-screen\)\. Section[3](https://arxiv.org/html/2608.20106#S3)and Section[4](https://arxiv.org/html/2608.20106#S4)describe their usage; Section[3\.2](https://arxiv.org/html/2608.20106#S3.SS2)and Appendix[B](https://arxiv.org/html/2608.20106#A2)document the explicit fact\-grounding constraint that prevents LLMs from being used as the source of factual claims\.