SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation

# SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation
Source: [https://arxiv.org/html/2607.07469](https://arxiv.org/html/2607.07469)
Andrea Scarinci1,Virginia Negri1,Brayan Impata1, Suleiman Khan1,Victor Martinez1,Marcello Federico1

1Amazon \{andscar, vrgngr, biimpata, suleimkh, vicmg, marcfede\}@amazon\.com

###### Abstract

Fine\-tuning large language models \(LLMs\) for e\-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages\. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly\. While recent work has demonstrated synthetic label generation using LLMs\(Negriet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib1)\), deploying such approaches at industrial scale requires integrated quality control mechanisms\. We presentSynthAVE, a large\-scale human\-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages \(Spanish, French, Italian, German\)\. To validate synthetic labels at scale, we introduce a multi\-LLM arena framework where samples are independently evaluated by 21 judge configurations \(7 model families×\\times3 prompts\), with final labels determined via majority voting\. The majority vote ensemble agrees with human experts at Cohen’sκ=0\.92\\kappa=0\.92\(95\.2% agreement\), while individual judges show substantial inter\-model agreement \(Fleiss’κ=0\.76\\kappa=0\.76\)\. This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost\-effective validation at scale while maintaining quality parity with human review\.

SynthAVE: Scalable Synthetic Labeling for E\-Commerce with LLM\-Arena Validation

Andrea Scarinci1, Virginia Negri1, Brayan Impata1,Suleiman Khan1,Victor Martinez1,Marcello Federico11Amazon\{andscar, vrgngr, biimpata, suleimkh, vicmg, marcfede\}@amazon\.com

## 1Introduction

Fine\-tuning and evaluating large language models \(LLMs\) for e\-commerce applications demands massive volumes of high\-quality labeled data\. Specifically, predicting and evaluating product attribute values from unstructured catalog text \(e\.g\., product titles, descriptions, bullet points\) requires labeled examples that maintain representativeness across three key dimensions: product categories \(e\.g\., electronics, clothing, furniture\), attributes \(e\.g\., color, material, dimensions\), and languages\. At catalog scale—thousands of product categories, thousands of attributes, and multiple languages—achieving reliable model performance requires hundreds of labeled examples per \(product category×\\timesattribute×\\timeslanguage\) triplet\. This combinatorial requirement translates to millions of annotations, making human labeling prohibitively costly\.

Prior work demonstrates the feasibility of using LLMs to generate synthetic labels for e\-commerce attribute value extraction\(Negriet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib1)\)\. However, deploying such approaches at industrial scale introduces a critical challenge: how to validate label quality efficiently across heterogeneous attributes and languages without exhaustive manual review\. A scalable solution must integrate synthetic label generation with automated quality control mechanisms\.

This paper presents SynthAVE \(Synthetic data for Attribute Value Extraction\), a large\-scale human\-validated benchmark augmenting prior generation methodology\(Negriet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib1)\)with scalable quality assurance\. We introduce a multi\-LLM auditing framework where each generated label is independently evaluated by 21 judge configurations \(7 model families×\\times3 prompts\), with final labels determined through majority voting\. To mitigate systematic biases, we employ diverse models from different providers alongside varied prompting strategies\. Ground truth for calibrating this system is established through human validation of 12,726 products\.

Empirical evaluation demonstrates that diverse models with different biases aggregate into highly reliable predictions: the ensemble achieves 95\.2% agreement with human experts \(Cohen’sκ=0\.92\\kappa=0\.92\) while enabling cost\-effective validation at scale\.

Our main contributions are:

- •SynthAVE: A human\-validated benchmark of 12,726 products spanning 229 product categories, 792 attributes, and 4 languages for multilingual attribute extraction research\.111The public release will additionally include English\-language annotations produced using the same pipeline, extending coverage to 5 languages\.
- •Multi\-LLM Validation Framework: An auditing approach employing majority voting across diverse judge configurations, achieving 95\.2% agreement with human evaluation while nabling cost\-effective validation at scale\.

## 2Related Work

Information extraction \(IE\) from unstructured web text is a foundational capability for large\-scale e\-commerce systems and knowledge base construction\. Common applications include product attribute extraction, entity population, catalog enrichment, and fact acquisition from heterogeneous online sources\(Zhanget al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib14); Martinez\-Rodriguezet al\.,[2020](https://arxiv.org/html/2607.07469#bib.bib13); Dagdelenet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib12)\)\. Recent advances in large language models \(LLMs\) have substantially lowered the barrier to deploying flexible extraction systems across domains and schemas\(Zhuet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib10)\)\. However, evaluating the quality of extracted information remains a central challenge, primarily due to the scarcity of high\-quality human\-annotated data and the scale at which modern IE systems operate\.

Classical evaluation methodologies for IE depend on manually annotated ground truth datasets and report performance using metrics such as precision, recall, and F1score\(Xuet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib11); Zhuet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib10)\)\. While effective in controlled research settings, this paradigm does not scale to real\-world web and e\-commerce applications\. Annotation is expensive, time\-consuming, and often requires domain expertise, particularly when extraction targets complex or evolving schemas\(Denget al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib2); Hsu and Roberts,[2025](https://arxiv.org/html/2607.07469#bib.bib3)\)\. As a result, many production pipelines operate with limited or no gold\-standard evaluation data, motivating the development of evaluation methods that function under weak or missing supervision\.

Existing benchmarks for product attribute extraction illustrate this tension between scale and quality\. The Multi\-source Attribute Value Extraction \(MAVE\) dataset\(Yanget al\.,[2022](https://arxiv.org/html/2607.07469#bib.bib44)\)provides 2\.2 million attribute\-value annotations across 1,257 attributes from Amazon product profiles—achieving impressive scale through automated pipelines\. However, MAVE’s quality assurance relies primarily on classifier\-based filtering rather than comprehensive human validation of attribute\-value annotations\. Additionally, MAVE is limited to English and addresses extraction rather than verification tasks\. These limitations motivate both the exploration of synthetic evaluation methods and the development of more rigorous validation frameworks\.

In response to these constraints, recent work has explored evaluation strategies that reduce or eliminate dependence on human\-labeled ground truth\.Seitlet al\.\([2024](https://arxiv.org/html/2607.07469#bib.bib40)\)propose an automatic IE evaluation framework based on synthetic ground truth generation, in which artificially constructed structured facts are injected into natural documents and systems are evaluated on their ability to recover them\. Despite their scalability, synthetic evaluation methods introduce limitations related to realism—artificially injected facts may not fully capture the ambiguity, inconsistency, and noise present in naturally occurring e\-commerce text\(Negriet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib1)\)\.

Another line of work investigates the use of LLMs to generate annotated datasets for downstream evaluation\. By leveraging LLMs as annotators, it becomes possible to rapidly construct large quantities of labeled data where human annotation is infeasible\(Mohtaet al\.,[2023](https://arxiv.org/html/2607.07469#bib.bib6); Tanet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib8)\)\. In e\-commerce contexts, this supports approximate evaluation of extraction pipelines across diverse product categories\(Nadaşet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib26)\)\. However, automatically generated annotations may reflect model\-specific biases or systematic errors, necessitating careful validation and selective human oversight\(Deet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib7)\)\. Pipeline design choices—decomposing extraction into multiple stages, structured output formats, and normalization rules—further improve performance\(Jaradehet al\.,[2023](https://arxiv.org/html/2607.07469#bib.bib9); Sabehet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib4); Satyadharmaet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib5)\), but complicate evaluation when standardized benchmarks are unavailable\.

## 3Task and Dataset

Attribute Value Extraction Task\.The attribute value extraction \(AVE\) task requires determining whether a given attribute value can be verified from unstructured product text\. Given a product’s unstructured text \(title, bullet points, description\) and an attribute\-value pair, the task is to assign one of three labels: CORRECT, INCORRECT, or UNKNOWN\.

We illustrate with a concrete example\. Given the product title‘‘Portable Laptop Stand, Adjustable Height, Aluminum Construction, Silver Finish, Compatible with 10\-17 inch Devices’’, the label depends on the attribute\-value pair:\(color,Silver\)\(\\texttt\{color\},\\texttt\{Silver\}\)iscorrectsince the text states “Silver Finish”;\(color,Black\)\(\\texttt\{color\},\\texttt\{Black\}\)isincorrectas it contradicts the text; and\(weight,1\.2kg\)\(\\texttt\{weight\},\\texttt\{1\.2kg\}\)isunknownsince no weight information is present\.

Training models to perform this three\-way classification reliably requires labeled examples of all cases across diverse product categories, attributes, and languages\. Our synthetic data generation pipeline produces such examples at scale, and our validation framework ensures their quality\.

Synthetic Data Generation\.We constructed the dataset in two stages\. First, we sampled real products from a large e\-commerce catalog spanning 4 languages \(Spanish, French, Italian, German\), ensuring categories exist in all languages with minimum 30 products per category\. Second, for each sampled product, we generated synthetic variants using the controlled generation methodology ofNegriet al\.\([2025](https://arxiv.org/html/2607.07469#bib.bib1)\)\. This process creates a strict one\-to\-one correspondence: each synthetic product instance is paired with exactly one target attribute and one value slot\. The generation pipeline produces all three label types:correct\(value explicitly or implicitly supported by text\),incorrect\(text contradicts the value\), andunknown\(attribute cannot be determined from available text\)\.

Table 1:Dataset statistics by languageThe synthetic dataset comprises 12,726 products across 229 product categories, covering 792 unique attributes and 2,607 distinct product\-attribute combinations\. The dataset spans major product domains including electronics, apparel, home & furniture, and sporting goods\. Each category contains a minimum of 30 products, with larger categories such as NOTEBOOK\_COMPUTER reaching 209 products\. Categories are evaluated on domain\-specific attributes \(e\.g\.,connectivity\_technologyfor electronics,closure\.typefor apparel\)\. Full category and attribute distributions are provided in Appendix[B](https://arxiv.org/html/2607.07469#A2)\.

Validation Challenge\.While the synthetic pipeline achieves 92\.6% accuracy in preliminary human analysis, this error rate is problematic for benchmark evaluation—metrics computed against noisy ground truth yield misleading conclusions\. Yet exhaustive human review of 12,726 products across 4 languages is prohibitively expensive\. Section[4](https://arxiv.org/html/2607.07469#S4)presents our solution: a multi\-LLM validation framework that achieves human\-level accuracy at a fraction of the cost\.

## 4Multi\-LLM Validation Framework

Having defined the attribute value extraction task and dataset construction process, we now address the core challenge: validating synthetic labels at scale without exhaustive human review\. Building upon established generation methods\(Negriet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib1)\), we present a multi\-LLM auditing framework that aggregates diverse model judgments into reliable quality estimates\.

Our framework treats LLM judges as independent evaluators whose collective judgment, when properly aggregated, can approximate human expert consensus\. This requires careful attention to two foundational aspects: ensuring independence across judges, and establishing qualification criteria that parallel traditional human auditor standards\.

Judge Selection and Configuration\.For aggregated judgments to be statistically meaningful, individual judges must provide genuinely independent assessments\. We enforce this through diversity along two dimensions\.

Model family diversity\.We selected 7 LLM families from different providers \(Table[2](https://arxiv.org/html/2607.07469#S4.T2)\): Claude\(Anthropic,[2024](https://arxiv.org/html/2607.07469#bib.bib39)\), Nova\(Amazon Artificial General Intelligence,[2024](https://arxiv.org/html/2607.07469#bib.bib36)\), GPT\(OpenAIet al\.,[2024](https://arxiv.org/html/2607.07469#bib.bib38)\), Mistral\(Mistral AI,[2025](https://arxiv.org/html/2607.07469#bib.bib43)\), DeepSeek\(Guoet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib41)\), Qwen\(Baiet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib45)\), and Gemma\(Gemma Teamet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib42)\)\. This ensures diversity across model scales and training objectives \(general\-purpose, reasoning\-focused, multilingual\), with outputs independent of family\-specific biases\.

Table 2:LLM models evaluated in our arena\.Prompt diversity\.For each model, we developed three substantially different prompt versions \(detailed in Appendix[D](https://arxiv.org/html/2607.07469#A4)\)\. This prevents model outputs from reflecting prompt over\-fitting or artificial consistency due to identical phrasing\.

This design yields7×3=217\\times 3=21independent judge configurations\. Each configuration evaluates all 12,726 products across 4 languages \(Spanish, French, Italian, German\), producing 267,246 total judgments\.

![Refer to caption](https://arxiv.org/html/2607.07469v1/data/llm_arena.png)Figure 1:Framework for multi\-LLM judge configurations\. Seven model families combined with three prompt variations yield 21 independent judges\.Qualification Criteria\.Beyond independence, each LLM judge must meet qualification criteria that parallel traditional standards for human auditors\.General competencyestablishes baseline capability: models must meet minimum performance thresholds on established benchmarks, such as Global Average≥70%\\geq 70\\%on LiveBench\(Whiteet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib34)\), paralleling education or certification requirements for human auditors\.Task\-specific competencyensures domain relevance: models must demonstrate sufficient understanding of attribute verification through internal benchmark evaluation, analogous to subject matter expertise\.Self\-consistencyaddresses output reliability: given known variability in LLM outputs due to floating\-point precision issues\(Yuanet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib35)\), judges must achieve minimum consistency when generating responses for identical prompts\.Instruction adherenceensures usable outputs: models must reliably follow specified output formats, paralleling structured reporting requirements for human evaluators\.

Evaluation Task\.Judges perform the attribute consistency verification task defined in Section[3](https://arxiv.org/html/2607.07469#S3), classifying each sample as CORRECT, INCORRECT, or UNKNOWN with brief justification\.

Aggregation through Majority Voting\.Given a panel of qualified, independent judges, we aggregate their outputs via majority voting across all 21 configurations\. This aggregation mechanism yields several important properties\. Higher inter\-judge agreement correlates with higher prediction accuracy, as consensus signals clearer cases\. Unanimous agreement indicates highest\-confidence predictions, while split decisions flag samples warranting closer inspection\. Most importantly, diverse judges with moderate individual agreement can achieve high ensemble accuracy through error cancellation\.

We therefore deliberately seek diversity among judges: moderate inter\-model agreement \(as measured by Fleiss’κ\\kappa\) combined with high ensemble\-human agreement \(Cohen’sκ\\kappa\) indicates successful diversity—independent judges with different biases combining into robust collective predictions\.

Ground Truth Establishment\.To evaluate our framework, we established ground truth labels through a disagreement\-based annotation strategy that leverages the LLM arena to minimize human effort\. Each synthetic sample has an original label assigned by the generation algorithm\. We compared this label against the LLM arena majority vote and applied the following protocol:

- •Agreement cases: When the majority vote matched the original synthetic label, the sample was accepted as correct without human review\.
- •Disagreement cases: When the majority vote contradicted the original label, the sample was flagged for manual review by domain experts, who determined the final ground truth label\.

To validate the agreement assumption, an expert annotator independently triaged a stratified random sample of 400 agreement cases \(100 per language, comprising 100 unanimous and 300 mixed\-agreement samples\)\. The annotator overturned the majority vote in only 12/400 cases \(3\.0%, 95% CI \[1\.3%, 4\.7%\]\)\. Critically, unanimous cases—where all 21 judges agreed—were never overturned \(0/100\), and all 12 disagreements occurred exclusively in mixed\-agreement samples \(4\.0% overturn rate\)\. Among majority labels, UNKNOWN showed the highest overturn rate \(5\.0%\) compared to CORRECT \(2\.2%\) and INCORRECT \(0\.0%\), consistent with UNKNOWN being the “hedge” vote that occasionally masks determinable cases\. This confirms that agreement between the generation algorithm and diverse LLM judges serves as a reliable signal of label correctness, with the majority panel achieving∼\\sim97% accuracy \(full triage analysis in Appendix[F](https://arxiv.org/html/2607.07469#A6)\)\.

## 5Results

We first establish that our validation framework achieves human\-level reliability, then present the final dataset features after cleaning\.

#### Overall Arena Performance\.

Table[3](https://arxiv.org/html/2607.07469#S5.T3)summarizes agreement between LLM Arena majority voting and human expert evaluation\. The arena achieves 95\.2% overall agreement with human evaluators, with Cohen’sκ=0\.92\\kappa=0\.92indicating almost perfect inter\-rater reliability\(Landis and Koch,[1977](https://arxiv.org/html/2607.07469#bib.bib46)\)\. Performance is consistent across languages \(94\.0–96\.4%\)\. Individual judges show substantial inter\-model agreement \(Fleiss’κ\\kappa= 0\.757\), yet the majority vote ensemble achieves almost perfect agreement with humans—validating that diverse models with different biases aggregate into highly accurate predictions\.

Table 3:LLM Arena agreement with human evaluation by language
#### Data Cleaning Performance\.

Table[4](https://arxiv.org/html/2607.07469#S5.T4)summarizes the Arena’s effectiveness as a data cleaning mechanism\. The original synthetic pipeline achieves 92\.6% accuracy \(946 errors across 12,726 samples\)\. The arena corrects 786 of these errors \(83\.1% error correction rate\), improving accuracy to 95\.0%\. When framed as binary bad\-label detection, the Arena achieves 98\.0% precision and 95\.2% recall \(F1 = 96\.6%\)—when it flags a label as incorrect, it is almost always right\.

Table 4:LLM Arena data cleaning performance
#### Per\-Class Performance\.

Table[5](https://arxiv.org/html/2607.07469#S5.T5)presents per\-label metrics\. The arena performs best on CORRECT \(F1 = 97\.3%\), followed by UNKNOWN \(F1 = 94\.5%\)\. INCORRECT labels prove most challenging \(F1 = 89\.2%\), primarily due to lower recall—the arena tends to classify borderline INCORRECT cases as UNKNOWN, a conservative behavior preferable in production where false negatives are less costly than false positives\.

Table 5:Per\-class performance metrics \(averaged across languages\)
#### Performance by Original Label Type\.

Stratifying by original synthetic label reveals where the arena adds most value \(Table[6](https://arxiv.org/html/2607.07469#S5.T6)\)\. Originally CORRECT labels already achieve 97\.3% accuracy, with modest arena improvement \(\+1\.2%\)\. Originally INCORRECT labels show the largest gap: synthetic accuracy is only 79\.3%, but the arena recovers to 88\.7% \(\+9\.4%\), demonstrating particular effectiveness at correcting the most error\-prone category\.

Table 6:Arena accuracy \(%\) by original synthetic label type
#### Agreement Level and Accuracy\.

Accuracy correlates strongly with agreement level: unanimous consensus \(36% of products\) yields 100% accuracy, very high agreement \(\>\>85%\) yields 98\.8%, while low\-agreement cases \(<<50%, 1\.4% of products\) achieve only 65\.1%, suggesting these warrant human review\. Full stratification is provided in Appendix[A](https://arxiv.org/html/2607.07469#A1)\.

#### Additional Analyses\.

No individual judge configuration outperformed the majority vote ensemble \(95\.0%\), and performance varied more across model families than across prompt variations, underscoring the importance of model diversity\. Figure[2](https://arxiv.org/html/2607.07469#S5.F2)illustrates this: certain model families consistently form high\-agreement clusters \(κ≈0\.80\\kappa\\approx 0\.80–0\.900\.90\) while others show lower mutual agreement \(κ≈0\.57\\kappa\\approx 0\.57–0\.650\.65\), with patterns invariant across languages\. Language\-specific cleaning shows Italian achieving the highest fix rate \(90\.2%\) and French the lowest \(76\.1%\)\. Error analysis reveals 80\.5% of errors involve confusion with UNKNOWN, while direct CORRECT↔\\leftrightarrowINCORRECT misclassification is rare \(19\.5%\)\. Full details are in Appendix[A](https://arxiv.org/html/2607.07469#A1)\.

![Refer to caption](https://arxiv.org/html/2607.07469v1/data/german_cohen_kappa_heatmap.png)Figure 2:Pairwise Cohen’sκ\\kappabetween judge configurations \(German\)\. Within\-family agreement is high \(diagonal blocks\), while cross\-family diversity drives ensemble accuracy\. Patterns hold across all languages \(Appendix[C](https://arxiv.org/html/2607.07469#A3)\)\.
#### Cost Analysis\.

Total validation cost was $290\.50 for 12,726 products \($22\.83 per 1,000 products, 267,246 API calls\)\.222Pricing based on commercial API rates as of June 2025\.Detailed breakdowns are in Appendix[G](https://arxiv.org/html/2607.07469#A7)\.

#### The SynthAVE Dataset\.

The resulting dataset comprises 12,726 human\-validated products spanning 229 product categories, 792 attributes, and 4 languages \(47\.7% CORRECT, 15\.8% INCORRECT, 36\.5% UNKNOWN\)\. All samples carry quality guarantees through LLM\-human consensus \(97% verified accuracy via expert triage of 400 agreement cases; Appendix[F](https://arxiv.org/html/2607.07469#A6)\) or direct expert annotation\. Unlike MAVE\(Yanget al\.,[2022](https://arxiv.org/html/2607.07469#bib.bib44)\), SynthAVE provides comprehensive human validation, explicit UNKNOWN labels, and multilingual coverage\. The dataset and per\-judge predictions will be publicly released\.

## 6Conclusion

We presented SynthAVE, a large\-scale human\-validated dataset and validation framework for synthetic e\-commerce data\. Building on prior work in synthetic data generation\(Negriet al\.,[2025](https://arxiv.org/html/2607.07469#bib.bib1)\), we introduced an LLM\-arena framework that enables scalable quality assurance through multi\-model evaluation\.

Our evaluation yields three key findings\. First, the LLM arena achieves 95% agreement with human evaluation \(Cohen’sκ\\kappa= 0\.92\) across 12,726 products, 229 product categories, 792 attributes, and 4 languages\. Second, diversity enables accuracy: individual judges show only substantial agreement \(Fleiss’κ\\kappa= 0\.76\), yet the ensemble outperforms any single model—with unanimous consensus reaching 100% accuracy\. Third, expert triage of 400 agreement cases confirms that the majority panel is overturned only 3% of the time \(0% for unanimous cases\), validating the disagreement\-based annotation strategy\. The arena corrects 83\.1% of synthetic labeling errors at $0\.02 per product, reserving human review only for low\-agreement cases\.

These results support adopting multi\-LLM evaluation protocols that incorporate prompt and model diversity, moving beyond single\-model approaches\. The SynthAVE dataset will be publicly released to support future research in scalable data quality assurance for attribute value extraction\.

## Limitations

Several limitations warrant discussion\. First, our evaluation covers four European languages with shared Latin script; generalization to languages with different scripts \(e\.g\., Chinese, Arabic\), morphological structures, or lower LLM proficiency requires additional validation\. Similarly, the framework was evaluated on e\-commerce attribute verification—a task with relatively objective ground truth—and may perform differently on more subjective evaluation tasks\.

Second, the framework depends on the availability of diverse, high\-capability LLMs\. As model families evolve, retire, or change pricing, the specific judge configurations may require updating\. The finding that some model families significantly underperform others \(accuracy ranging from 76% to 93%\) raises questions about minimum capability thresholds for judge inclusion\.

Third, our ground truth construction relied on a disagreement\-based annotation strategy where human review was triggered only when the LLM ensemble disagreed with synthetic labels\. Expert triage of 400 stratified agreement cases validated this approach \(3\.0% overturn rate, 0% for unanimous cases\), though it does not provide traditional inter\-annotator agreement metrics among human experts\.

Finally, while the framework is designed for offline validation rather than real\-time inference, the latency of running 21 judge configurations may still be prohibitive for rapid iteration cycles\. Future work could explore distilling ensemble judgments into smaller, specialized models for faster evaluation\.

## Ethics Statement

SynthAVE is constructed from synthetic product data generated using the methodology ofNegriet al\.\([2025](https://arxiv.org/html/2607.07469#bib.bib1)\), with seed products sampled from commercial e\-commerce catalogs\. All brand names, model identifiers, and potentially identifying product information were anonymized during generation to prevent privacy concerns and mitigate LLM biases from prior knowledge of specific products—no real customer data or personally identifiable information is present in the released dataset\. Human validation was performed by domain experts specialized in e\-commerce catalog systems, who established ground truth labels through careful review of disagreement cases between the synthetic generation pipeline and LLM arena predictions\. SynthAVE is designed as a test set optimized for evaluation coverage: it spans 2,598 product type\-attribute combinations with a minimum of 30 samples per combination, prioritizing breadth over the volume typically required for training\. We encourage researchers to leverage this benchmark for downstream applications including validating catalog quality assurance systems, improving attribute coverage in product catalogs, developing robust multilingual extraction methods, and advancing LLM\-as\-judge evaluation methodologies\. We hope SynthAVE contributes to more accurate and reliable e\-commerce information systems that benefit both businesses and consumers\.

## References

- The Amazon Nova family of models: technical report and model card\.Technical reportAmazon\.External Links:[Link](https://www.amazon.science/publications/the-amazon-nova-family-of-models-technical-report-and-model-card)Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- Anthropic \(2024\)The Claude 3 model family: Opus, Sonnet, Haiku\.Note:[https://www\.anthropic\.com/news/claude\-3\-family](https://www.anthropic.com/news/claude-3-family)Accessed: 2024\-03\-14Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-VL technical report\.External Links:2511\.21631Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- J\. Dagdelen, A\. Dunn, S\. Lee, N\. Walker, A\. S\. Rosen, G\. Ceder, K\. A\. Persson, and A\. Jain \(2024\)Structured information extraction from scientific text with large language models\.Nature Communications15\(1\),pp\. 1418\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p1.1)\.
- S\. De, D\. K\. Sanyal, and I\. Mukherjee \(2025\)Fine\-tuned encoder models with data augmentation beat ChatGPT in agricultural named entity recognition and relation extraction\.Expert Systems with Applications277,pp\. 127126\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- S\. Deng, Y\. Ma, N\. Zhang, Y\. Cao, and B\. Hooi \(2024\)Information extraction in low\-resource scenarios: survey and perspective\.In2024 IEEE International Conference on Knowledge Graph \(ICKG\),Vol\.,pp\. 33–49\.External Links:[Document](https://dx.doi.org/10.1109/ICKG63256.2024.00013)Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p2.1)\.
- Gemma Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière,et al\.\(2025\)Gemma 3 technical report\.External Links:2503\.19786Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- E\. Hsu and K\. Roberts \(2025\)Leveraging large language models for knowledge\-free weak supervision in clinical natural language processing\.Scientific Reports15\(1\),pp\. 8241\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p2.1)\.
- M\. Y\. Jaradeh, K\. Singh, M\. Stocker, A\. Both, and S\. Auer \(2023\)Information extraction pipelines for knowledge graphs\.Knowledge and Information Systems65\(5\),pp\. 1989–2016\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- J\. R\. Landis and G\. G\. Koch \(1977\)The measurement of observer agreement for categorical data\.Biometrics,pp\. 159–174\.Cited by:[§5](https://arxiv.org/html/2607.07469#S5.SS0.SSS0.Px1.p1.2)\.
- J\. L\. Martinez\-Rodriguez, A\. Hogan, and I\. Lopez\-Arevalo \(2020\)Information extraction meets the semantic web: a survey\.Semantic Web11\(2\),pp\. 255–335\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p1.1)\.
- Mistral AI \(2025\)Introducing Mistral 3: Mistral Large 3 and the Mistral 3 model family\.Note:[https://mistral\.ai/news/mistral\-3](https://mistral.ai/news/mistral-3)Accessed: 2026\-02\-04Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- J\. Mohta, K\. Ak, Y\. Xu, and M\. Shen \(2023\)Are large language models good annotators?\.InProceedings on “I Can’t Believe It’s Not Better: Failure Modes in the Age of Foundation Models” at NeurIPS 2023 Workshops,Proceedings of Machine Learning Research, Vol\.239,pp\. 38–48\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- M\. Nadaş, L\. Dioşan, and A\. Tomescu \(2025\)Synthetic data generation using large language models: advances in text and code\.External Links:2503\.14023Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- V\. Negri, V\. M\. Gómez, S\. A\. Balanya, and S\. Rajaram \(2025\)Attribute\-aware controlled product generation with LLMs for e\-commerce\.External Links:2601\.04200Cited by:[§1](https://arxiv.org/html/2607.07469#S1.p2.1),[§1](https://arxiv.org/html/2607.07469#S1.p3.1),[§2](https://arxiv.org/html/2607.07469#S2.p4.1),[§3](https://arxiv.org/html/2607.07469#S3.p4.1),[§4](https://arxiv.org/html/2607.07469#S4.p1.1),[§6](https://arxiv.org/html/2607.07469#S6.p1.1),[Ethics Statement](https://arxiv.org/html/2607.07469#Sx2.p1.1)\.
- OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2024\)GPT\-4 technical report\.External Links:2303\.08774Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p4.1)\.
- K\. Sabeh, M\. Kacimi, J\. Gamper, R\. Litschko, and B\. Plank \(2024\)Exploring large language models for product attribute value identification\.External Links:2409\.12695Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- S\. Satyadharma, F\. Sheikholeslami, S\. Kaul, A\. U\. Batur, and S\. A\. Khan \(2025\)Auto prompting without training labels: an LLM cascade for product quality assessment in e\-commerce catalogs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,S\. Potdar, L\. Rojas\-Barahona, and S\. Montella \(Eds\.\),Suzhou, China,pp\. 937–953\.External Links:[Link](https://aclanthology.org/2025.emnlp-industry.63/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.63)Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- F\. Seitl, T\. Kovářík, S\. Mirshahi, J\. Kryštůfek, R\. Dujava, M\. Ondreička, H\. Ullrich, and P\. Gronat \(2024\)Assessing the quality of information extraction\.External Links:2404\.04068Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p4.1)\.
- Z\. Tan, D\. Li, S\. Wang, A\. Beigi, B\. Jiang, A\. Bhattacharjee, M\. Karami, J\. Li, L\. Cheng, and H\. Liu \(2024\)Large language models for data annotation and synthesis: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 930–957\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.54/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.54)Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p5.1)\.
- C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey,et al\.\(2025\)LiveBench: a challenging, contamination\-limited LLM benchmark\.InProceedings of the International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p7.1)\.
- D\. Xu, W\. Chen, W\. Peng, C\. Zhang, T\. Xu, X\. Zhao, X\. Wu, Y\. Zheng, Y\. Wang, and E\. Chen \(2024\)Large language models for generative information extraction: a survey\.Frontiers of Computer Science18\(6\),pp\. 186357\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p2.1)\.
- L\. Yang, Q\. Wang, Z\. Yu, A\. Kulkarni, S\. Sanghai, B\. Shu, J\. Elsas, and B\. Kanagal \(2022\)MAVE: a product dataset for multi\-source attribute value extraction\.InProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining,WSDM ’22,New York, NY, USA,pp\. 1256–1265\.External Links:[Document](https://dx.doi.org/10.1145/3488560.3498377)Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p3.1),[§5](https://arxiv.org/html/2607.07469#S5.SS0.SSS0.Px8.p1.1)\.
- J\. Yuan, H\. Li, X\. Ding, W\. Xie, Y\. Li, W\. Zhao, K\. Wan, J\. Shi, X\. Hu, and Z\. Liu \(2025\)Understanding and mitigating numerical sources of nondeterminism in LLM inference\.External Links:2506\.09501Cited by:[§4](https://arxiv.org/html/2607.07469#S4.p7.1)\.
- Y\. Zhang, M\. Zhong, S\. Ouyang, Y\. Jiao, S\. Zhou, L\. Ding, and J\. Han \(2024\)Automated mining of structured knowledge from text in the era of large language models\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 6644–6654\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p1.1)\.
- Y\. Zhu, X\. Wang, J\. Chen, S\. Qiao, Y\. Ou, Y\. Yao, S\. Deng, H\. Chen, and N\. Zhang \(2024\)LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities\.World Wide Web27\(5\),pp\. 58\.Cited by:[§2](https://arxiv.org/html/2607.07469#S2.p1.1),[§2](https://arxiv.org/html/2607.07469#S2.p2.1)\.

## Appendix ADetailed Results

### A\.1Agreement Level and Accuracy

Table[7](https://arxiv.org/html/2607.07469#A1.T7)shows arena accuracy stratified by the proportion of judges agreeing on the majority vote\.

Table 7:LLM Arena accuracy by model agreement level
### A\.2Data Cleaning Performance by Language

Table[8](https://arxiv.org/html/2607.07469#A1.T8)presents detailed cleaning performance by language, including fix rates and regression rates\.

Table 8:Data cleaning performance by language
### A\.3Error Pattern Distribution

Table[9](https://arxiv.org/html/2607.07469#A1.T9)breaks down the arena’s 630 errors by confusion pattern\. The dominant pattern is INCORRECT→\\rightarrowUNKNOWN \(27\.5%\), indicating conservative abstention\. Confusion involving UNKNOWN accounts for 80\.5% of all errors, while direct CORRECT↔\\leftrightarrowINCORRECT misclassification is rare \(19\.5%\)\.

Table 9:Error pattern distribution \(Human→\\rightarrowArena\)

## Appendix BDataset Statistics

This appendix provides comprehensive statistics for the SynthAVE test set we release with this paper\.

### B\.1Overall Scale

- •Total products: 12,726 unique products
- •Product categories: 229 distinct types
- •Unique attributes: 792 across all languages
- •Product\-attribute combinations: 2,607 unique pairs
- •Average entries per combination: 4\.9
- •Languages: 4 \(Spanish, French, Italian, German\)
- •Products per language: 3,007–3,501
- •Judge configurations: 21 \(7 models×\\times3 prompts\)

### B\.2Label Distribution

Table 10:Overall label distribution \(majority vote\)Table[10](https://arxiv.org/html/2607.07469#A2.T10)presents the overall label distribution based on majority vote across all judges\.

### B\.3Product Category Distribution

Table[11](https://arxiv.org/html/2607.07469#A2.T11)presents the top 20 product categories by volume\.

Table 11:Top 20 product categories by count \(of 229 total\)
### B\.4Attribute Distribution

Table[12](https://arxiv.org/html/2607.07469#A2.T12)presents the top 20 attributes by frequency\.

Table 12:Top 20 attributes by count \(of 792 total\)
### B\.5Top Product\-Attribute Combinations

Table[13](https://arxiv.org/html/2607.07469#A2.T13)presents the most frequent product\-attribute combinations in the dataset\.

Table 13:Top 20 product\-attribute combinations \(of 2,607 total\)
### B\.6Per\-Language Statistics

Table[14](https://arxiv.org/html/2607.07469#A2.T14)provides detailed per\-language statistics including unique product types, attributes, and combinations\.

Table 14:Detailed statistics by language
### B\.7Ground Truth Distribution

For human validation, we collected ground truth labels for all entries\. Table[15](https://arxiv.org/html/2607.07469#A2.T15)shows the distribution\.

Table 15:Ground truth label distribution \(human\-verified\)

## Appendix CInter\-Judge Agreement Heatmaps by Language

This appendix presents pairwise agreement heatmaps for all four languages\. Figure[2](https://arxiv.org/html/2607.07469#S5.F2)in the main text shows German \(Cohen’sκ\\kappa\)\. Below we provide the complete set for all languages and both agreement metrics\.

### C\.1Cohen’s Kappa Heatmaps

Figures[3](https://arxiv.org/html/2607.07469#A3.F3),[4](https://arxiv.org/html/2607.07469#A3.F4),[5](https://arxiv.org/html/2607.07469#A3.F5)

![Refer to caption](https://arxiv.org/html/2607.07469v1/data/italian_cohen_kappa_heatmap.png)Figure 3:Pairwise Cohen’sκ\\kappabetween judge configurations for Italian\. Clustering patterns are consistent with other languages, with the same model families forming high\-agreement and low\-agreement groups\.![Refer to caption](https://arxiv.org/html/2607.07469v1/data/french_cohen_kappa_heatmap.png)Figure 4:Pairwise Cohen’sκ\\kappabetween judge configurations for French\. Despite French showing the lowest overall arena accuracy \(94\.0%\), inter\-judge agreement patterns remain consistent with other languages\.![Refer to caption](https://arxiv.org/html/2607.07469v1/data/spanish_cohen_kappa_heatmap.png)Figure 5:Pairwise Cohen’sκ\\kappabetween judge configurations for Spanish\.
### C\.2Raw Agreement Heatmaps

Figures[6](https://arxiv.org/html/2607.07469#A3.F6),[7](https://arxiv.org/html/2607.07469#A3.F7),[8](https://arxiv.org/html/2607.07469#A3.F8),[9](https://arxiv.org/html/2607.07469#A3.F9)

![Refer to caption](https://arxiv.org/html/2607.07469v1/data/spanish_llm_arena_agreement_heatmap.png)Figure 6:Pairwise raw agreement between judge configurations for Spanish\. Agreement ranges from 75% to 96% \(within\-family pairs\)\.![Refer to caption](https://arxiv.org/html/2607.07469v1/data/german_agreement_heatmap.png)Figure 7:Pairwise raw agreement between judge configurations for German\. Agreement ranges from approximately 75% \(cross\-family pairs with lower\-performing models\) to 96% \(within\-family pairs\)\.![Refer to caption](https://arxiv.org/html/2607.07469v1/data/italian_agreement_heatmap.png)Figure 8:Pairwise raw agreement between judge configurations for Italian\. Italian shows the highest overall agreement, consistent with its leading arena accuracy \(96\.4%\)\.![Refer to caption](https://arxiv.org/html/2607.07469v1/data/french_agreement_heatmap.png)Figure 9:Pairwise raw agreement between judge configurations for French\.

## Appendix DEvaluation Prompt Templates

This appendix provides the complete prompt templates used for our LLM Arena evaluation across all three prompt versions\. Each version maintains the same evaluation task \(attribute consistency verification\) while employing different structural approaches and reasoning strategies to ensure IID generation requirements\.

### D\.1Prompt Version 1: Direct Instruction

```
You are an expert product attribute evaluator. Your task is to determine whether a generated attribute label is correct,
incorrect, or unknown based on the provided product information.

PRODUCT INFORMATION:
ID: {id}  |  Category: {category}  |  Title: {title}
Product Description: {description}
Bullet Points: {bullet_points}
ATTRIBUTE: {attribute}  |  GENERATED LABEL: {generated_label}

TASK: Evaluate whether the generated label "{generated_label}" is accurate for the attribute "{attribute}" based on the product
information provided. Note that some product information (description or bullet points) may not be available.

RESPONSE OPTIONS:
- CORRECT: The generated label accurately reflects the product’s attribute based on the available information
- INCORRECT: The generated label is clearly wrong based on the product information
- UNKNOWN: Cannot determine if the label is correct or incorrect from the available product information
  (including cases where insufficient information is provided)

RESPONSE FORMAT: You must respond with exactly one of: "CORRECT", "INCORRECT", or "UNKNOWN".
After your choice, in a new line, provide a brief explanation (1-2 sentences) of your reasoning.

Example response format:
CORRECT
The product description explicitly mentions this attribute value.
```

Figure 10:Prompt Version 1: Direct instruction format with minimal structure\.
### D\.2Prompt Version 2: XML\-Structured with Examples

```
You are an e-Commerce data analyst specializing in product attribute verification and quality assurance.
Assess whether a generated attribute label correctly represents the specified product attribute based on available listing information.

<examples>
<example>
ID: B08N5WRWNW | Category: LUGGAGE
Title: <brandname> <makename> Hardside Expandable Luggage with Spinner Wheels, Brushed Anthracite, Carry-On 20-Inch
Description: Durable polycarbonate shell construction with scratch-resistant texture
Bullet Points: * 100% Polycarbonate shell * Four multidirectional spinner wheels * TSA-approved combination lock
Attribute Name: material_type | Generated Label: Polycarbonate
Your Response:
CORRECT
The description and bullet points explicitly confirm polycarbonate shell construction, validating the material type label.
</example>
<example>
ID: B07XYZ123A | Category: CLOTHING
Title: Men’s Cotton Blend Casual T-Shirt Blue Medium | Description: [Not provided] | Bullet Points: [Not provided]
Attribute Name: sleeve_length | Generated Label: Long Sleeve
Your Response:
UNKNOWN
Title mentions t-shirt but provides no sleeve length information, and additional product details are unavailable.
</example>
<example>
ID: B09ABC456D | Category: ELECTRONICS
Title: Wireless Bluetooth Headphones with Noise Cancellation
Description: Premium over-ear headphones featuring active noise cancellation technology
Bullet Points: * 30-hour battery life * Bluetooth 5.0 connectivity * Foldable design for portability
Attribute Name: connectivity_technology | Generated Label: USB-C
Your Response:
INCORRECT
Product clearly indicates Bluetooth connectivity, directly contradicting the USB-C label assignment.
</example>
</examples>

<inputs>
<id>{id}</id> <category>{category}</category> <item_name>{title}</item_name>
<description>{description}</description> <bullet_points>{bullet_points}</bullet_points>
<attribute_name>{attribute}</attribute_name> <generated_label>{generated_label}</generated_label>
</inputs>

<instructions>
Evaluate the generated attribute label by:
1. Reviewing all available product information for evidence relating to the specified attribute
2. Determining if the generated label matches, conflicts with, or cannot be verified from the product data
3. Handling incomplete information appropriately when description or bullet points are missing
Classify as:
- CORRECT: Label accurately reflects attribute based on available evidence
- INCORRECT: Label contradicts information in product details
- UNKNOWN: Insufficient data to verify label accuracy
Response format: Respond with exactly one of "CORRECT", "INCORRECT", or "UNKNOWN", followed by a brief explanation.
</instructions>
```

Figure 11:Prompt Version 2: XML\-structured format with in\-context examples\.
### D\.3Prompt Version 3: Role\-Based with Systematic Guidelines

```
## Role
You are an e-Commerce data analyst specializing in product attribute verification and quality assurance.
Your task is to assess whether a generated attribute label correctly represents the specified product attribute
based on available listing information.

## Task Overview
Evaluate the accuracy of a generated attribute label by analyzing product listing data and determining if the label
is supported, contradicted, or cannot be verified by the available information.

## Evaluation Guidelines
Follow these steps systematically:
1. **Review Available Evidence**: Examine all provided product information (title, description, bullet points)
   for any evidence relating to the specified attribute.
2. **Compare Label Against Evidence**: Determine whether the generated label:
   - Matches the product information (explicitly or implicitly)
   - Contradicts the product information
   - Cannot be verified due to insufficient data
3. **Handle Missing Information**: When description or bullet points are unavailable or incomplete,
   acknowledge the limitation in your assessment.

## Classification Categories
- **CORRECT**: The label accurately reflects the attribute based on available evidence in the product listing
- **INCORRECT**: The label contradicts or misrepresents information found in the product details
- **UNKNOWN**: Insufficient data exists to verify the label’s accuracy

## Examples
### Example 1: CORRECT
- ID: B08N5WRWNW | Category: LUGGAGE
- Title: <brandname> <makename> Hardside Expandable Luggage with Spinner Wheels, Brushed Anthracite, Carry-On 20-Inch
- Description: Durable polycarbonate shell construction with scratch-resistant texture
- Bullet Points: * 100% Polycarbonate shell * Four multidirectional spinner wheels * TSA-approved combination lock
- Attribute: material_type | Label: Polycarbonate
Assessment: CORRECT - Description and bullet points explicitly confirm polycarbonate shell construction.

### Example 2: UNKNOWN
- ID: B07XYZ123A | Category: CLOTHING
- Title: Men’s Cotton Blend Casual T-Shirt Blue Medium | Description: [Not provided] | Bullet Points: [Not provided]
- Attribute: sleeve_length | Label: Long Sleeve
Assessment: UNKNOWN - Title mentions t-shirt but provides no sleeve length information.

### Example 3: INCORRECT
- ID: B09ABC456D | Category: ELECTRONICS
- Title: Wireless Bluetooth Headphones with Noise Cancellation
- Description: Premium over-ear headphones featuring active noise cancellation technology
- Bullet Points: * 30-hour battery life * Bluetooth 5.0 connectivity * Foldable design for portability
- Attribute: connectivity_technology | Label: USB-C
Assessment: INCORRECT - Product clearly indicates Bluetooth connectivity, contradicting USB-C label.

## Product Information to Evaluate
- **ID**: {id} | **Category**: {category} | **Title**: {title}
- **Description**: {description}
- **Bullet Points**: {bullet_points}
- **Attribute Name**: {attribute} | **Generated Label**: {generated_label}

## Response Format
Provide your assessment as: CORRECT/INCORRECT/UNKNOWN followed by a brief explanation (1-2 sentences).
```

Figure 12:Prompt Version 3: Role\-based format with systematic guidelines and markdown structure\.

## Appendix EEvaluation Settings

### E\.1Input Format

```
{
  "id": "X",
  "category": "FILE_FOLDER",
  "text_fields": {
    "title": "<brand>, Lot de 100 pochettes...",
    "features": [
      "Les pochettes A4 en polythylne...",
      "Pochettes transparentes..."
    ],
    "description": "Ces chemises en plastique..."
  },
  "attribute_of_interest": {
    "name": "tab.position",
    "value": "Ct"
  }
}
```

Figure 13:Example input format for LLM judges\. Models receive structured product data with target attribute to verify\.Figure[13](https://arxiv.org/html/2607.07469#A5.F13)shows the input format provided to each judge\.

### E\.2Model Configuration

All models use a low temperature \(0\.1\) to ensure consistent, reproducible outputs\. We leave default settings for Top\-p and Top\-k parameters\.

For OpenAI’s GPT OSS reasoning model, we set reasoning\_effort="low"to balance computational cost with performance\. This parameter controls the depth of the model’s internal chain\-of\-thought reasoning before producing a final answer\. Even at the “low” setting, the model demonstrates strong performance on our classification task while significantly reducing inference costs compared to higher reasoning effort levels\.

Model\-Specific Formats\.Each model family requires different API message formats:

- •Claude: Uses Anthropic’s message format with anthropic\_versionspecification
- •Nova: Requires nested content structure with explicit text fields
- •OpenAI: Standard chat completion format with with reasoning\_effortparameter
- •Mistral: Uses instruction\-tuned format with\[INST\]tags
- •DeepSeek/Qwen3/Gemma: Standard chat message format

### E\.3Response Parsing

Models return responses in different formats requiring specialized parsing:

- •Standard format\(Claude, Nova, Mistral, DeepSeek, Qwen3, Gemma\): First line contains the classification label \(CORRECT, INCORRECT, or UNKNOWN\), followed by reasoning on subsequent lines\.
- •OpenAI reasoning format: Response contains <reasoning\>\.\.\.</reasoning\>tags encapsulating the model’s chain\-of\-thought, followed by the final classification label\. The parser extracts both components separately\.

Invalid or unparseable responses default to UNKNOWN with the full response content logged for debugging\.

### E\.4Evaluation Pipeline

For each product and judge configuration:

1. 1\.Input formatting: Product data formatted according to model\-specific API requirements
2. 2\.API invocation: Model called via AWS Bedrock with configured parameters
3. 3\.Response parsing: Extract classification and reasoning from model output, handling format variations
4. 4\.Validation: Verify response contains valid classification \(CORRECT, INCORRECT, or UNKNOWN\); invalid responses default to UNKNOWN
5. 5\.Error handling: API failures and malformed responses are caught and logged, with graceful degradation to UNKNOWN
6. 6\.Storage: Save classification, reasoning, model ID, prompt version, and timestamp

All models were accessed through AWS Bedrock’s unified API, ensuring consistent infrastructure across different model providers\. The pipeline processes 21 judge configurations \(7 models×\\times3 prompt variants\) per product systematically\.

## Appendix FHuman Triage of Agreement Cases

To validate the assumption that agreement between the synthetic generation pipeline and the LLM arena majority vote implies label correctness, an expert annotator independently triaged a stratified random sample of 400 agreement cases\. The sample was balanced across locales \(100 per language\) and stratified by agreement level \(100 unanimous, 300 mixed\-agreement\)\.

For each sample, the annotator reviewed the product text, attribute\-value pair, and majority vote label, then recorded whether theyagreewith the majority,disagree\(i\.e\., would overturn it\), or areunsure\. Only clear disagreements count as overturns in the analysis below; unsure cases \(6/400\) are excluded from the overturn rate\.

#### Overall Results\.

The annotator disagreed with the majority vote in only 12 out of 400 cases \(3\.0%\), with a 95% confidence interval of \[1\.3%, 4\.7%\] \(margin of error±\\pm1\.7%\)\.

Table 16:Expert triage overturn rates by slice
#### Breakdown by Locale\.

Table 17:Expert triage overturn rates by locale
#### Key Findings\.

- •The majority panel achieves a 97% accuracy rate \(12/400 overturns\) with a tight confidence interval of \[1\.3%, 4\.7%\]\.
- •Unanimous cases have a 0% overturn rate \(0/100\): when all 21 judges agree, the label is always correct in our sample\.
- •UNKNOWN is the least reliable majority label at 5\.0% overturn rate, compared to CORRECT at 2\.2% and INCORRECT at 0\.0%\. This is consistent with UNKNOWN being the conservative vote that can mask determinable cases\.
- •All 12 overturns came from mixed\-agreement samples \(4\.0% rate\); no unanimous sample was overturned\.
- •Locale differences range from 1\.0% \(FR\) to 5\.0% \(IT\), but withnn=100 per locale the margins of error \(2–4%\) overlap substantially\. Distinguishing locale\-level effects would require∼\\sim1,000\+ samples per locale\.

#### Implications\.

The 3%±\\pm1\.7% overturn rate supports the claim that the majority panel correctly labels∼\\sim97% of agreement cases\. Combined with the 0% overturn rate for unanimous cases, this validates the disagreement\-based annotation strategy: agreement between the generation pipeline and a diverse 21\-judge panel is a reliable signal of label correctness, with residual error concentrated in inherently ambiguous mixed\-agreement cases\.

## Appendix GCost Analysis Details

This appendix provides detailed cost breakdowns for the LLM Arena validation framework\. All costs are based on AWS Bedrock on\-demand pricing as of January 2026\.333[https://aws\.amazon\.com/bedrock/pricing/](https://aws.amazon.com/bedrock/pricing/)

### G\.1Token Statistics

Each API call consists of input tokens \(the prompt with product data\) and output tokens \(the model’s classification and reasoning\)\. Input token counts vary based on prompt version and product text length; output tokens are estimated at 50 tokens per response \(classification label plus brief reasoning\)\.

Table 18:Average input tokens per prompt by language and versionTable[18](https://arxiv.org/html/2607.07469#A7.T18)presents input token statistics\. The three prompt versions differ in complexity: V1 provides basic instructions, V2 adds attribute descriptions, and V3 includes worked examples\. This is reflected in token counts, with V3 prompts averaging 70% more tokens than V1\.

Token counts vary across languages due to differences in product text length and linguistic characteristics\. German \(DE\) has the highest average token counts, while Spanish \(ES\) has the lowest\. The standard deviation of approximately 285–308 tokens reflects the diversity in product description lengths across the dataset\.

### G\.2Full Arena Costs

Table 19:Full arena costs by languageTable[19](https://arxiv.org/html/2607.07469#A7.T19)shows the per\-language cost breakdown for running the full 21\-judge arena \(7 models×\\times3 prompt versions\)\. Costs scale linearly with dataset size, with French being most expensive due to having the most products\.

### G\.3Scaling Projections

Table[20](https://arxiv.org/html/2607.07469#A7.T20)projects arena costs at various scales\. At production scale \(1M products\), the full arena would cost approximately $22,825—substantially less than equivalent human annotation, which would cost $100,000–500,000 at typical rates of $0\.10–0\.50 per label\.

Table 20:Projected arena costs at scale
### G\.4Cost\-Efficient Configurations

For budget\-constrained deployments, reduced ensemble configurations offer significant savings with potential accuracy trade\-offs\.

Table 21:Alternative ensemble configurations \(per 1,000 products\)Table[21](https://arxiv.org/html/2607.07469#A7.T21)presents alternative ensemble configurations\. Cost varies substantially across model families, with open\-weight models typically 10–30×\\timescheaper than commercial APIs\. Using a single model family with all three prompt versions represents 3–49% of full arena cost depending on family choice\.

The trade\-off between cost and accuracy presents optimization opportunities\. Unanimous agreement cases \(36% of products\) could use a smaller initial ensemble, with full arena deployment reserved for uncertain cases\. Such adaptive strategies could reduce average cost by 30–50% while maintaining high accuracy on confident predictions\. However, ablation studies validating accuracy of reduced configurations are required before deployment\.