AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets

# AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets
Source: [https://arxiv.org/html/2607.15781](https://arxiv.org/html/2607.15781)
andPranav PaiThe University of MelbourneMelbourneVictoriaAustralia

\(2026\)

###### Abstract\.

Geospatial datasets support applications from urban planning to climate modeling, but their compliance with the FAIR principles \(Findability,Accessibility,Interoperability, andReusability\) is difficult to assess consistently\. Existing evaluators use different rubrics and evidence sources and can fail on JavaScript\-rendered pages or repository\-specific identifier schemes\. In a diagnostic study of 50 datasets from 10 repositories, the standard deviation of normalized scores across available tools averages 15\.0 percentage points and reaches 30\.3 points on one dataset\. Because these normalized outputs are not equivalent measurements, we use them to characterize disagreement and failure modes rather than comparative accuracy\.

We presentAgentFAIR, a multi\-agent framework that combines structured metadata extraction with 13 sub\-principle\-specific large language model \(LLM\) evaluators\. Each evaluator returns a 0–3 maturity score, cited evidence, and recommendations; a critic applies evidence and consistency checks and can request targeted re\-evaluation\. In the 50\-dataset sample, meanFindability,Accessibility,Interoperability, andReusabilityscores are 79\.7%, 70\.4%, 45\.3%, and 72\.0%, respectively\. Rank correlations betweenAgentFAIRand four baseline tools range fromρ=0\.31\\rho=0\.31to0\.610\.61, with the FAIR\-enough comparison not statistically significant\. On a 10\-dataset repeated\-run subset, sub\-principle agreement averages 89% \(reported standard deviation: 3 percentage points\); a critic ablation reports 71% agreement without the critic\. A preliminary 15\-dataset expert study yields Fleiss’κ=0\.71\\kappa=0\.71and 82% sub\-principle alignment with expert consensus\. The measured API cost is approximately USD $0\.054 per dataset\. These results support the framework’s auditability and feasibility, while the limited benchmark, incomplete component ablations, and single\-model\-family validation constrain claims about accuracy and generalization\.

Geospatial datasets, FAIR principles, Multi\-agent systems, Large language models, Metadata quality

††copyright:none††journalyear:2026††conference:Revised manuscript following peer review; 2026;††ccs:Information systems Data management systems Data quality††ccs:Computing methodologies Artificial intelligence Natural language processing Text mining††ccs:Information systems Data provenance## 1\.Introduction

The FAIR data principles—Findability,Accessibility,Interoperability, andReusability—provide widely adopted guidance for making digital research outputs more reusable and machine\-actionable\(Wilkinsonet al\.,[2016](https://arxiv.org/html/2607.15781#bib.bib1)\)\. In geospatial science, FAIRness \(FAIR compliance;*not*algorithmic fairness\) is especially consequential: spatial layers, rasters, and spatio\-temporal products are routinely integrated across agencies, jurisdictions, and scientific communities\. However, many geospatial datasets remain only partially FAIR, due to inconsistent metadata practices, heterogeneous formats \(e\.g\., Shapefile, GeoTIFF, NetCDF\), and incomplete spatial reference information\(Thompsonet al\.,[2023](https://arxiv.org/html/2607.15781#bib.bib2); Paprotny and Mengel,[2023](https://arxiv.org/html/2607.15781#bib.bib3)\)\. According to a European Commission cost\-benefit analysis, non\-FAIR practices are estimated to cost the EU research economy € 10\.2 Billion annually through duplicated effort and failed reuse\(PwC EU Services,[2018](https://arxiv.org/html/2607.15781#bib.bib30)\)\.

Existing automated FAIR assessment tools such as F\-UJI\(Devaraju and Huber,[2020](https://arxiv.org/html/2607.15781#bib.bib5)\)and FAIR\-Checker\(Gaignardet al\.,[2023](https://arxiv.org/html/2607.15781#bib.bib6)\)provide important mechanistic checks \(e\.g\., persistent identifiers, licenses, basic schema markup\)\. However, deterministic approaches face limitations that are amplified for geospatial resources: \(i\)Dynamic metadata exposure:Repository landing pages increasingly rely on JavaScript rendering and API\-driven content, which breaks traditional crawlers and causes partial or missing evidence collection; \(ii\)Repository heterogeneity:Identifier and metadata practices vary widely \(DOI, Handle, internal URIs; different schema profiles\), thus hard\-coded assumptions \(e\.g\., DataCite\-centric resolution patterns\) can lead to brittle failures; \(iii\)Limited semantic judgment:Rule\-based checks cannot reliably assess qualitative and domain\-specific aspects of metadata, such as whether a coordinate reference system is valid, whether access constraints are optional or mandatory, or whether terminology reflects community standards; \(iv\)Low score comparability:Recent analyses show that different tools operationalize FAIR differently, leading to substantial score variance and making stewardship decisions tool\-dependent\(Candelaet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib4); Kranset al\.,[2022](https://arxiv.org/html/2607.15781#bib.bib8)\)\.

Furthermore, geospatial FAIRness analysis depends on domain conventions that general\-purpose tools often treat as opaque strings\. Examples include ISO \(International Organisation for Standardisation\) 19115 metadata profiles\(International Organization for Standardization,[2014](https://arxiv.org/html/2607.15781#bib.bib9)\), coordinate reference system identifiers \(EPSG codes\), and OGC discovery and access conventions \(e\.g\., WMS, WFS, and STAC endpoints\)\. Because many geospatial vocabularies are not registered in common ontology registries, semantic validators may systematically underestimateInteroperability, even when datasets follow community practice\.

To address these challenges, we proposeAgentFAIR, a multi\-agent collaborative framework that combines: \(i\) deterministic extraction of machine\-readable metadata and identifiers, and \(ii\) sub\-principle\-specific LLM agents that perform evidence\-grounded semantic evaluation\. As shown in Figure[1](https://arxiv.org/html/2607.15781#S1.F1), the system orchestrates a pipeline that processes dataset landing pages, extracts metadata, evaluates each FAIR sub\-principle with specialized agents, and applies critic\-based quality control via running critic agents to ensure evidence\-backed scoring and consistency across principles\. When evidence is insufficient or internally inconsistent, the critic agent triggers a targeted re\-evaluation \(Retrymechanism\) with refined instructions, improving robustness while preserving traceability\. Overall,AgentFAIRevaluates all 13 FAIR sub\-principles \(F1–F4, A1\.1–A2, I1–I3, R1\.1–R1\.3\) on a 0–3 maturity scale and outputs a Markdown report, JSON assessment, SQLite evidence store, and local execution traces\. Remote LangSmith tracing is optional and disabled by default\.

We evaluateAgentFAIRon 50 geospatial datasets across 10 repositories, including Zenodo111[https://zenodo\.org](https://zenodo.org/)\(Open repository for research outputs with DOI assignment\.\), Dryad222[https://datadryad\.org](https://datadryad.org/)\(Curated repository for datasets supporting published scientific research\.\), Harvard Dataverse333[https://dataverse\.harvard\.edu](https://dataverse.harvard.edu/)\(General\-purpose repository for sharing and citing research data\.\), PANGAEA444[https://www\.pangaea\.de](https://www.pangaea.de/)\(Earth and environmental science repository for georeferenced datasets\.\), NOAA NCEI555[https://www\.ncei\.noaa\.gov](https://www.ncei.noaa.gov/)\(Global archive of climate, oceanic, and environmental data\.\), and domain\-specific portals\. We compare against four widely used FAIR evaluators \(F\-UJI\(Devaraju and Huber,[2020](https://arxiv.org/html/2607.15781#bib.bib5)\), FAIR\-Checker\(Gaignardet al\.,[2023](https://arxiv.org/html/2607.15781#bib.bib6)\), FAIRshake\(Clarkeet al\.,[2019](https://arxiv.org/html/2607.15781#bib.bib23)\), FAIR\-enough\(Emonet and Dumontier,[2022](https://arxiv.org/html/2607.15781#bib.bib21)\)\) and analyze \(i\) overall FAIR maturity patterns, \(ii\) cross\-tool agreement and failure modes, \(iii\) cost and runtime characteristics, and \(iv\) consistency across repeated evaluations\.

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/AgentGPT_framework.png)System diagram showing AgentFAIR’s pipeline: URL input, orchestration, agent execution with critic loop, persistence of audit trails, and reporting\.

Figure 1\.The proposedAgentFAIRframework for multi\-agent collaborative FAIRness evaluation\.To the best of our knowledge,AgentFAIRis the first multi\-agent LLM\-based framework specifically designed for FAIR compliance evaluation\. It integrates critic\-informed feedback loops with domain\-specific geospatial reasoning\. The main contributions of this work are as follows:

- •The first LLM\-driven multi\-agent FAIR evaluator\.We proposeAgentFAIR, a collaborative multi\-agent framework with critic\-based quality control and explicit awareness of geospatial standards\. To our knowledge, it is the first system to combine LLM reasoning with domain\-specific FAIR assessment\.
- •A transparent and fine\-grained scoring protocol\.We operationalize a 0–3 maturity rubric for each FAIR sub\-principle and explicitly map geospatial indicators to the assessment ofInteroperabilityandReusability\.
- •A cross\-repository empirical study\.We evaluate 50 geospatial datasets across 10 repositories\. The sample reveals strongFindability\(79\.7%\) and weakInteroperability\(45\.3%\), but is not intended to establish long\-tail generalization\.
- •A diagnostic analysis of evaluator disagreement and feasibility\.We quantify cross\-tool disagreement \(mean per\-dataset standard deviation 15\.0 points; maximum 30\.3\), document repository\-specific failure modes, and measure an average API cost of approximately USD $0\.054 per dataset\. These comparisons do not treat heterogeneous tool scores as equivalent accuracy measures\.

The rest of the paper is organized as follows\. Section 2 reviews related work on FAIR assessment tools, geospatial data quality, and multi\-agent LLM architectures\. Section 3 formalizes the task definition and scoring protocol\. Section 4 details theAgentFAIRframework design and implementation\. Section 5 presents the evaluation and its limitations, and Section 6 concludes the paper\. The software artifact, dataset list, prompt structure, and LangGraph orchestration details are provided in the Appendix\. The source code is available at[https://doi\.org/10\.5281/zenodo\.18529560](https://doi.org/10.5281/zenodo.18529560)and[https://github\.com/MingCHEN\-Github/AgentFAIR](https://github.com/MingCHEN-Github/AgentFAIR)\.

## 2\.Related Work

### 2\.1\.Automated FAIR Assessment Tools

Automated FAIR evaluators typically operationalize the principles as predefined, machine\-checkable indicators \(e\.g\., presence of PIDs, protocol support, and machine\-readable licensing\) and aggregate the results into per\-principle scores\. F\-UJI implements the FAIRsFAIR assessment metrics and collects evidence from dataset landing pages and exposed metadata, complemented by queries to external registries\(Devaraju and Huber,[2020](https://arxiv.org/html/2607.15781#bib.bib5)\)\. FAIR\-Checker emphasizes semantic validation by leveraging knowledge graphs and SHACL/SPARQL constraints\(Gaignardet al\.,[2023](https://arxiv.org/html/2607.15781#bib.bib6)\)\. FAIRshake provides community\-curated rubrics tailored to different digital resource types\(Clarkeet al\.,[2019](https://arxiv.org/html/2607.15781#bib.bib23)\), whereas FAIR\-enough offers a broader suite of largely binary tests, which can be conservative when evidence is partial or indirectly expressed\(Emonet and Dumontier,[2022](https://arxiv.org/html/2607.15781#bib.bib21)\)\. A recurring challenge is that tools differ in their operational definitions, evidence sources, and aggregation rules, leading to scores that are not directly comparable across evaluators\(Candelaet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib4)\)\.

Table[1](https://arxiv.org/html/2607.15781#S2.T1)summarizes representative capabilities of these systems relative toAgentFAIR\. For baselines,*coverage*denotes whether a tool—or our mapping of its outputs—supports assessment across all 13 FAIR sub\-principles used in this paper\.

Table 1\.Comparison of FAIR assessment systemsLLM=LLM\-based reasoning in the evaluation loop; Geo\-aware=explicit recognition of geospatial standards; Explain=evidence\-backed scoring rationales; Trace=persisted audit trails; Cross\-P=cross\-principle consistency checks; Coverage=ability to support assessment across all 13 sub\-principles used here\.

### 2\.2\.FAIRness and Quality for Geospatial Data

FAIRness assessment for geospatial datasets intersects with classic spatial data quality concerns, including spatial reference consistency, resolution/scale, temporal extent, and adherence to domain standards\. Metadata standards such as ISO 19115\(International Organization for Standardization,[2014](https://arxiv.org/html/2607.15781#bib.bib9)\)enable rich descriptions of spatial resources, but repositories vary substantially in how completely these metadata elements are captured and exposed in machine\-readable form\. As a result, automated checkers that rely on generic schemas, brittle pattern matching, or incomplete registry coverage may miss geospatial signals \(e\.g\., CRS \(Coordinate Reference System\) identifiers or geospatial service endpoints\), which can systematically depressInteroperability—and, downstream,Reusability—scores for spatial datasets\.

### 2\.3\.LLM Agent for Data Management and Evaluation

Large language models \(LLMs\) have shown promise for extracting, normalizing, and enriching metadata from heterogeneous sources\(Buschet al\.,[2025](https://arxiv.org/html/2607.15781#bib.bib13); Zhanget al\.,[2025](https://arxiv.org/html/2607.15781#bib.bib14)\)\. Recent systems combine LLM reasoning with workflow orchestration to automate data cleaning, validation, and quality assessment\(Liet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib16); Huang and Wu,[2024](https://arxiv.org/html/2607.15781#bib.bib27); Narayanet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib36)\)\. However, applying LLMs to FAIR evaluation introduces additional requirements—consistent rubrics, evidence\-backed justifications, and auditable trails—and systematic multi\-agent designs that explicitly address these requirements, especially under geospatial standards, remain limited\.

### 2\.4\.Multi\-agent LLM Architectures

Multi\-agent frameworks decompose complex tasks into coordinated subtasks executed by specialized agents, often improving robustness through structured interaction and self\-critique\(Xiet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib34); Wanget al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib38)\)\. AutoGen\(Wuet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib31)\)introduced conversational multi\-agent patterns enabling flexible agent interactions, while MetaGPT\(Honget al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib32)\)demonstrated role\-based collaboration for software engineering tasks\. Benchmarks such as AgentBench\(Liuet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib33)\)and TheAgentCompany\(Xuet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib35)\)evaluate agent capabilities on consequential real\-world tasks, providing evaluation ideas that we adapt for FAIR compliance assessment\. For geospatial domains, foundation model applications\(Maiet al\.,[2024](https://arxiv.org/html/2607.15781#bib.bib37)\)highlight both the potential and the domain\-specific challenges of applying LLMs to spatial information understanding\. Building on these architectural patterns,AgentFAIRcombines critic\-informed feedback loops with evidence\-grounded prompting and explicit geospatial standard awareness to support transparent, auditable FAIR scoring\.

Table 2\.Operationalization of the 13 FAIR sub\-principles used in this paper: evidence criteria, 0–3 maturity levels, and representative geospatial indicators\.

## 3\.Task Definition and Scoring Protocol

### 3\.1\.FAIR Evaluation Task and Principle Set

We formulate FAIR evaluation as an evidence\-grounded auditing task\(Wilkinsonet al\.,[2016](https://arxiv.org/html/2607.15781#bib.bib1)\)\. Given a dataset landing\-page URLuu\(the human\-facing entry point for a dataset\) and optional repository contextmm\(e\.g\., repository type, known metadata schemas, and API conventions\),AgentFAIRproduces three structured outputs: \(i\) per\-sub\-principle maturity scoresS=\{sp\}p∈𝒫S=\\\{s\_\{p\}\\\}\_\{p\\in\\mathcal\{P\}\}, \(ii\) per\-sub\-principle evidence bundlesE=\{ep\}p∈𝒫E=\\\{e\_\{p\}\\\}\_\{p\\in\\mathcal\{P\}\}that justify each score with provenance, and \(iii\) actionable recommendationsR=\{rp\}p∈𝒫R=\\\{r\_\{p\}\\\}\_\{p\\in\\mathcal\{P\}\}describing what is missing and how to improve compliance\.

Evidence model\.For each sub\-principlep∈𝒫p\\in\\mathcal\{P\}, the evidence bundleepe\_\{p\}is a structured record that includes \(a\) extracted metadata fields \(e\.g\., PID, license, protocol\), \(b\) supporting excerpts/snippets \(e\.g\., HTML fragments or metadata blocks\), and \(c\) provenance pointers \(e\.g\., source URL and the extraction route such as landing page, embedded JSON\-LD, downloadable metadata files, or repository API\)\. This design is critical for explainability and auditability: non\-zero scores must cite observable, traceable evidence\.

Evaluated principles\.We evaluate the 13 FAIR sub\-principles from the FAIR Guiding Principles\(Wilkinsonet al\.,[2016](https://arxiv.org/html/2607.15781#bib.bib1)\):

- •Findability:F1 \(globally unique persistent identifier\), F2 \(rich metadata\), F3 \(identifier in metadata\), F4 \(indexed in a searchable resource\)\.
- •Accessibility:A1\.1 \(retrieval protocol\), A1\.2 \(authentication/authorization\), A2 \(metadata persistence\)\.
- •Interoperability:I1 \(formal knowledge representation\), I2 \(FAIR vocabularies\), I3 \(qualified references\)\.
- •Reusability:R1\.1 \(license\), R1\.2 \(provenance\), R1\.3 \(community standards\)\.

Role of repository contextmm\.When available,mmprovides lightweight repository\-specific hints \(e\.g\., typical locations of machine\-readable metadata, common field names, or standardized API endpoints\)\. This improves recall of evidence without changing the rubric:AgentFAIRstill assigns scores strictly according to the same maturity criteria, and all inferences must be backed by retrievable evidence recorded inEE\.

### 3\.2\.Rubric\-Based Maturity Scoring and Aggregation

Maturity rubric\.Each sub\-principleppreceives an ordinal maturity scoresp∈\{0,1,2,3\}s\_\{p\}\\in\\\{0,1,2,3\\\}:

- •0 \(non\-compliant\):no supporting evidence is found, or evidence is contradictory/unverifiable from accessible resources\.
- •1 \(partial\):some relevant information exists but is incomplete, implicit, or primarily human\-readable \(weak machine actionability\)\.
- •2 \(substantial\):clear evidence exists in a machine\-actionable form for the core requirement, but with notable gaps \(e\.g\., missing qualifiers, incomplete standard alignment, or partial exposure across distributions\)\.
- •3 \(full\):strong, unambiguous evidence satisfies the requirement in a standard, machine\-actionable way with sufficient detail for automated reuse and validation\.

Table[2](https://arxiv.org/html/2607.15781#S2.T2)specifies the concrete indicators and decision rules used for each sub\-principle\. This explicit rubric is applied uniformly across repositories to reduce tool\-specific interpretation variance\(Wilkinsonet al\.,[2019b](https://arxiv.org/html/2607.15781#bib.bib7)\)\.

Dimension and overall aggregation\.Let𝒟=\{F,A,I,R\}\\mathcal\{D\}=\\\{\\text\{F\},\\text\{A\},\\text\{I\},\\text\{R\}\\\}denote the four FAIR dimensions, and let𝒫d⊂𝒫\\mathcal\{P\}\_\{d\}\\subset\\mathcal\{P\}be the proper subset of sub\-principles within dimensiondd\. We report scores as percentages for comparability with existing evaluators:

Score​\(d\)=13​\|𝒫d\|​∑p∈𝒫dsp×100,Score​\(FAIR\)=13​\|𝒫\|​∑p∈𝒫sp×100\.\\mathrm\{Score\}\(d\)=\\frac\{1\}\{3\|\\mathcal\{P\}\_\{d\}\|\}\\sum\_\{p\\in\\mathcal\{P\}\_\{d\}\}s\_\{p\}\\times 100,\\qquad\\mathrm\{Score\}\(\\mathrm\{FAIR\}\)=\\frac\{1\}\{3\|\\mathcal\{P\}\|\}\\sum\_\{p\\in\\mathcal\{P\}\}s\_\{p\}\\times 100\.Becausesps\_\{p\}is evidence\-based, missing machine\-readable signals directly translate into lower maturity and generate corresponding recommendationsrpr\_\{p\}\.

Geospatial operationalization\.A common failure mode of generic FAIR checkers for spatial datasets is missing domain\-specific signals that are essential forInteroperabilityandReusability\. To mitigate this,AgentFAIRexplicitly searches for and reasons over geospatial indicators when scoring I1–I3 and R1\.3, including \(non\-exhaustively\): coordinate reference system identifiers, spatial and temporal extent descriptors, ISO 19115\-1 elements, and OGC\-related service endpoints and encodings across different geospatial semantics\. These indicators are not “bonus points”; rather, they serve as admissible evidence under the rubric when they satisfy the corresponding sub\-principle requirements \(e\.g\., standard machine\-interpretable representations for I1 and community standards for R1\.3\)\.

Evaluation objectives supported by the protocol\.The above task formulation and rubric are designed to support: \(i\)Consistency\(controlled variability under repeated runs despite LLM stochasticity\), \(ii\)Explainability\(scores justified by explicit evidence and decision rules\), \(iii\)Auditability\(persisted evidence and traces enabling post\-hoc inspection\), and \(iv\)Domain awareness\(recognition of geospatial standards and metadata constructs\)\.

## 4\.Proposed AgentFAIR Framework

Figure[1](https://arxiv.org/html/2607.15781#S1.F1)summarizesAgentFAIRas a three\-stage, evidence\-first pipeline for evaluating all 13 FAIR sub\-principles on the 0–3 maturity rubric defined in Section 3\. Given a dataset input \(DOI/Handle/direct URL\), the system executes:\(Stage 1\) Data extraction→\\rightarrow\(Stage 2\) Parallel FAIR dimension evaluation→\\rightarrow\(Stage 3\) Quality assurance and reporting\.

Stage 1 \(Data extraction\)\.A Playwright\-based crawler resolves redirects and renders JavaScript\-heavy landing pages, then a hybrid extractor constructs a unified metadata recordMMthat merges structured fields \(e\.g\., JSON\-LD/Schema\.org, HTML meta tags, DCAT\-like fragments\) with provenance\-linked text snippets\. Agents must cite this record when assigning a non\-zero score\. The critic checks for missing evidence and conflicts with deterministic observations; this is a structural check on the returned record, not a claim that prompting alone guarantees factual correctness\.

Stage 2 \(Parallel dimension evaluation\)\.AgentFAIRdecomposes evaluation into four concurrent LangGraph state machines, one per FAIR dimension \(F/A/I/R\), each internally evaluating its sub\-principles in parallel\. For each sub\-principlep∈𝒫p\\in\\mathcal\{P\}, the corresponding agent outputs a tuple\(sp,ep,cp\)\(s\_\{p\},e\_\{p\},c\_\{p\}\)wheresp∈\{0,1,2,3\}s\_\{p\}\\in\\\{0,1,2,3\\\}is the maturity score,epe\_\{p\}is an evidence bundle \(fields/snippets \+ provenance pointers\), andcp∈\[0,1\]c\_\{p\}\\in\[0,1\]is a confidence estimate used for quality control\.

Stage 3 \(Quality assurance & reporting\)\.A critic\-driven feedback loop checks evidence sufficiency and cross\-sub\-principle consistency before scores are aggregated into dimension and overall percentages \(Section 3\.2\)\. The report service then materializes auditable artifacts: a human\-readable Markdown report, a machine\-readable JSON assessment, a SQLite evidence store, and local execution traces for post\-hoc inspection\. LangSmith export is an explicit opt\-in because prompts and evidence may contain sensitive content\.

Algorithm[1](https://arxiv.org/html/2607.15781#algorithm1)summarizes the control flow, including caching of deterministic signals to stabilize retries\.

Input :Dataset URL

uu, repository context

mm
Output :Sub\-principle scores

SS, evidence

EE, recommendations

RR
1

2

ℎ𝑡𝑚𝑙←Crawl​\(u\)\\mathit\{html\}\\leftarrow\\textsc\{Crawl\}\(u\);

/\*Playwright for JS rendering \*/

3

4

M←ExtractMetadata​\(ℎ𝑡𝑚𝑙,m\)M\\leftarrow\\textsc\{ExtractMetadata\}\(\\mathit\{html\},m\);

/\*Hybrid extraction & enrichment \*/

5

6

G←DeterministicChecks​\(u,M\)G\\leftarrow\\textsc\{DeterministicChecks\}\(u,M\);

/\*Resolver, protocol, content\-negotiation, and registry signals \*/

7

8foreach*sub\-principlep∈𝒫p\\in\\mathcal\{P\}*do

9

\(sp,ep,cp\)←Agentp​\(M,G\)\(s\_\{p\},e\_\{p\},c\_\{p\}\)\\leftarrow\\textsc\{Agent\}\_\{p\}\(M,G\);

/\*Evidence\-backed evaluation \*/

10if*cp<θpc\_\{p\}<\\theta\_\{p\}or*HardCheckFails*​\(p,sp,ep,G\)\\textsc\{HardCheckFails\}\(p,s\_\{p\},e\_\{p\},G\)*then

11

\(sp,ep,cp\)←CriticRepair​\(p,M,G,sp,ep\)\(s\_\{p\},e\_\{p\},c\_\{p\}\)\\leftarrow\\textsc\{CriticRepair\}\(p,M,G,s\_\{p\},e\_\{p\}\);

/\*Retry with feedback \*/

12

13end if

14

15end foreach

16

17

S←Aggregate​\(\{sp\}\)S\\leftarrow\\textsc\{Aggregate\}\(\\\{s\_\{p\}\\\}\);

/\*Dimension and overall scores \*/

18

19

R←GenerateRecommendations​\(S,E\)R\\leftarrow\\textsc\{GenerateRecommendations\}\(S,E\);

return*S,E,RS,E,R*

Algorithm 1AgentFAIRevaluation pipeline### 4\.1\.LangGraph orchestration

We use LangGraph\(Lang Chain Authors,[2026](https://arxiv.org/html/2607.15781#bib.bib18)\)to represent the multi\-agent workflow as a directed state machine\. Each FAIR dimension is expressed as a modular subgraph with nodes for crawling, extraction, sub\-principle evaluation, critic review, and report synthesis\. Conditional edges implement retry logic when confidence is low or evidence is missing \(Appendix[C](https://arxiv.org/html/2607.15781#A3)\)\. Checkpointing supports recovery, and trace integration enables inspection of agent decisions and intermediate state\.

### 4\.2\.Hybrid metadata extraction

The Extractor stage combines: \(1\)rule\-based parsingof structured metadata \(e\.g\., Schema\.org/JSON\-LD, DCAT\-like fragments, HTML meta tags\) and identifier patterns \(DOI/Handle/ORCID\); and \(2\)LLM\-assisted enrichmentfor normalization, deduplication, and resolving ambiguous natural\-language fields \(e\.g\., license statements and access conditions\)\. The output is a unified internal metadata recordMMthat retains both structured fields and provenance links to their sources \(e\.g\., landing page text, embedded JSON\-LD\)\.

### 4\.3\.Sub\-principle agents

AgentFAIRcomprises 13 specialized agents, one per FAIR sub\-principle\. Each agent receives the extracted metadata recordMMand returns: \(i\) a maturity scoresp∈\{0,1,2,3\}s\_\{p\}\\in\\\{0,1,2,3\\\}, \(ii\) evidenceepe\_\{p\}consisting of specific supporting snippets or fields, and \(iii\) a confidence scorecpc\_\{p\}used for quality control\. Each system prompt separates six elements: \(i\) the sub\-principle definition and scope; \(ii\) the 0–3 decision rubric; \(iii\) admissible deterministic and extracted evidence; \(iv\) geospatial indicators such as EPSG, ISO 19115, and OGC terms where relevant; \(v\) conservative instructions not to infer absent evidence; and \(vi\) a structured output schema requiring score, rationale, evidence, confidence, and recommendations\. Retry prompts append the critic’s identified contradiction or missing\-evidence requirement while retaining the original rubric and cached deterministic observations\.

### 4\.4\.Critic agent and consistency checks

The critic agent applies two forms of quality control:evidence sufficiency\(is the score supported by extracted evidence?\) andcross\-sub\-principle consistency\(do scores contradict each other given the evidence?\)\. For example, high A1\.1 \(open protocol\) paired with low A1\.2 \(authentication clarity\) can indicate ambiguous access conditions; the critic triggers a targeted re\-evaluation with refined instructions\. For Findability, seven deterministic checks include identifier presence, resolver success for maturity≥2\\geq 2, content\-negotiation evidence for maturity 3, score\-range validity, and evidence presence for non\-zero scores\. A retry can be triggered by a failed hard check or by confidence below 0\.5 \(0\.4 for I1 in the reported configuration\), subject to a retry budget\. Evaluator and critic use the same GPT\-4o\-mini backend with different roles and prompts; therefore the critic does not remove shared\-model bias\. All retries and revisions are logged as part of the audit trail\.

### 4\.5\.Implementation notes

AgentFAIRuses Playwright for JavaScript\-rendered access and Extruct, lxml, and rdflib for structured metadata parsing\. Geospatial standards and CRS identifiers are recognized from exposed metadata; the evaluated implementation does not perform GDAL\-based inspection of downloaded data files\. SQLite persists evidence and local traces, and FastAPI exposes a REST API for local integration\. The primary experiment uses GPT\-4o\-mini with temperature 0\.1\. The non\-zero setting was intended to permit limited lexical variation in rationales and evidence text; a temperature 0\.0 versus 0\.1 ablation was not conducted, so no performance benefit is claimed\.

### 4\.6\.Security boundary

Dataset pages and metadata are untrusted input\. The implementation restricts evaluation targets to public HTTP\(S\) destinations, uses hardened XML parsing, and requires structured outputs, but it has not been evaluated against prompt injection or poisoned metadata\. In particular, text retrieved from a page can still influence an LLM evaluator\. The released API is therefore intended for local, single\-user operation; adversarial prompt\-injection testing, redirect\-by\-redirect destination validation, authentication, rate limiting, and per\-request credential isolation are required before public deployment\.

Table 3\.FAIR maturity matrix for 50 datasets across 13 sub\-principles using the 0–3 rubric \(representative run\)\. Columns D1–D50 map to the dataset list in Appendix[E](https://arxiv.org/html/2607.15781#A5)\.Table 4\.Case study FAIR dimension scores \(%\) from the representative run in Table[3](https://arxiv.org/html/2607.15781#S4.T3)\.

## 5\.Evaluation

Experiments were conducted on an Apple M2 Pro system \(32 GB RAM, macOS 15\)\. The framework evaluated all 13 FAIR sub\-principles using the 0–3 maturity rubric in Section 3\. LLM calls usedgpt\-4o\-mini\(temperature 0\.1\)\. Playwright handled browser\-based crawling \(including JavaScript\-rendered pages\); SQLite stored evidence traces and audit logs\.

### 5\.1\.Datasets

We evaluated 50 geospatial datasets spanning 10 repositories to capture heterogeneous metadata practices and access patterns\. Our benchmark prioritizes*cross\-repository diversity*over single\-repository depth\. It samples heterogeneous repository and domain conditions but is too small to establish long\-tail generalization\.Repository distribution:Zenodo \(10\), Dryad \(8\), PANGAEA \(7\), Harvard Dataverse \(6\), NOAA NCEI \(6\), ScienceBase \(4\), Figshare \(4\), AURIN \(2\), EarthData \(2\), NASA SEDAC \(1\)\.Domains:Biodiversity/Ecology \(12\), Marine/Geoscience \(10\), Climate/Hydrology \(8\), Urban/Land Classification \(6\), Demographics \(4\), Disaster/Emissions \(3\), Other \(7\)\. Datasets are referenced as D1–D50 in Tables[3](https://arxiv.org/html/2607.15781#S4.T3)and[5](https://arxiv.org/html/2607.15781#S5.T5); the full mapping appears in Appendix[E](https://arxiv.org/html/2607.15781#A5)\.

### 5\.2\.Baselines and score normalization

We compare against four public FAIR evaluators: F\-UJI, FAIR\-Checker, FAIRshake, and FAIR\-enough\. These tools expose different rubrics and output schemas \(binary tests, rubric scores, or per\-dimension scores\), so we normalize each tool’s*overall*dataset score to a 0–100 scale\. For tools with multiple tests, the normalized score is the fraction of tests passed; for tools with per\-dimension scores, we use the mean across dimensions\. ForAgentFAIR, the overall score is computed from its 13 sub\-principle maturities \(0–3\), scaled to 0–100\. This normalization supports only dataset\-level disagreement and failure\-mode analysis; it does*not*make the underlying metrics equivalent and is not used as evidence that one evaluator is more accurate\.

Baseline tools also differ in metadata acquisition strategy\. F\-UJI relies primarily on API/identifier lookups, whileAgentFAIRuses browser\-based crawling via Playwright\. Browser\-based extraction better reflects end\-user access to JavaScript\-rendered metadata, but introduces different failure modes \(e\.g\., navigation timeouts on complex pages vs\. API endpoint unavailability\)\.

Run selection and table consistency\.BecauseAgentFAIRis LLM\-based, it exhibits run\-to\-run variability \(Section[5\.7](https://arxiv.org/html/2607.15781#S5.SS7)\)\. We report one representative run in Table[3](https://arxiv.org/html/2607.15781#S4.T3); theAgentFAIRrow in Table[5](https://arxiv.org/html/2607.15781#S5.T5)is the direct aggregation of those same 13 scores for each dataset\.

### 5\.3\.Results: FAIR maturity patterns

Table[3](https://arxiv.org/html/2607.15781#S4.T3)reports maturity scores for each dataset \(columns\) across the 13 FAIR sub\-principles \(rows\)\. Aggregating these sub\-principle scores,Findabilityperforms best on average \(79\.7%\), followed byReusability\(72\.0%\) andAccessibility\(70\.4%\);Interoperabilityis consistently weakest \(45\.3%\)\.

Sub\-principle bottlenecks\.F2 \(rich metadata\) is the strongest sub\-principle \(mean 2\.94/3; 47/50 datasets achieve full maturity\), reflecting widespread use of descriptive landing pages and repository\-generated metadata\. In contrast, A2 \(metadata should remain accessible even when the data are no longer available\) remains weak \(mean 1\.14/3; 0/50 at full maturity\), reflecting limited explicit preservation policies, certification, and machine\-actionable tombstone guarantees\. The sub\-principles I2 and I3 fromInteroperabilityare the dominant bottlenecks \(means 1\.00/3 and 1\.20/3; 0/50 at full maturity\), primarily because \(i\) widely used geospatial vocabularies \(CRS identifiers, ISO elements, OGC conventions\) are not consistently registered in the ontology registries expected by semantic validators, and \(ii\) explicit, typed links to related datasets and resources are rarely exposed\.

Repository effects\.The representative run suggests higher scores for repositories with stable identifiers and structured metadata exposure, including PANGAEA, Zenodo, Dryad, and Harvard Dataverse, and lower scores for several catalog\-oriented ScienceBase and AURIN records\. These are descriptive observations from an uneven, small sample rather than population estimates for each repository\.

Domain effects\.Across domains,Interoperabilityremains the principal limiting factor\. We observe a tendency for biodiversity/ecology datasets \(often benefiting from established community metadata norms\) to score higher overall, while datasets hosted in heterogeneous government catalog ecosystems \(e\.g\., ScienceBase\) score lower\.

### 5\.4\.Case studies: root\-cause analysis

Table[4](https://arxiv.org/html/2607.15781#S4.T4)analyzes three representative datasets spanning low, medium, and high maturity\. Dimension scores are computed as the mean of sub\-principles within each FAIR dimension, scaled to 0–100 \(using the same rubric as Table[3](https://arxiv.org/html/2607.15781#S4.T3)\)\.

AURIN OSM POIs \(D2, 43\.6%\)\.This dataset exhibits near\-zeroInteroperabilitydue to absent machine\-readable knowledge representations and unregistered vocabularies \(no JSON\-LD/RDF export, no typed relations\)\.Accessibilityis moderate due to HTTPS access, andReusabilityis moderate due to visible license/provenance statements\. The primary gap is the lack of machine\-actionable metadata exposure \(structured identifiers, vocabulary grounding, and typed relations\)\.

NASA GDIS \(D3, 71\.8%\)\.This dataset achieves highFindabilityvia resolvable persistent identifiers and strong indexing\.Interoperabilityremains moderate because metadata relies largely on generic schema markup without explicit domain vocabulary registration or typed links\.Reusabilityis constrained by incomplete provenance detail and limited machine\-readable linking between the dataset and related resources\.

Zenodo Crater Lake \(D1, 74\.4%\)\.This dataset achieves strong overall maturity, with highReusabilitydriven by explicit licensing and provenance\.Interoperabilityis constrained by I2/I3: the dataset relies primarily on generic vocabularies rather than registered domain vocabularies, and provides few typed links to related datasets/resources\.

### 5\.5\.Results: comparative tool analysis

Table 5\.Normalized dataset\-level overall scores \(0–100\) across five FAIR assessment tools \(single\-run per tool\)\. Columns D1–D50 map to the dataset list in Appendix[E](https://arxiv.org/html/2607.15781#A5)\. “–” indicates the tool did not return a score \(e\.g\., API timeout or FAIR\-enough database limit\)\.Table 6\.Spearman association betweenAgentFAIRand baseline normalized scores\.Table[5](https://arxiv.org/html/2607.15781#S5.T5)compares normalized overall scores across five tools, and Table[6](https://arxiv.org/html/2607.15781#S5.T6)reports their rank associations withAgentFAIR\. Across datasets, tools disagree substantially: the population standard deviation across available normalized tool scores averages 15\.0 points \(maximum 30\.3 points\), indicating that operationalizations of FAIR differ materially even after normalization\.

Failure and brittleness patterns\.F\-UJI exhibits sharp failures \(scores≤\\leq11\.5\) on 10/50 datasets \(20%\), concentrated on Harvard Dataverse and Figshare, consistent with brittle identifier and registry\-lookup dependencies\. FAIR\-Checker timed out on one dataset \(D34\)\. FAIR\-enough produced no score for 18 datasets \(36%\) due to database/runtime limits, and tends to score lowest when it does return output because its binary tests penalize partial compliance\.

Tool agreement\.AgentFAIRis most aligned with FAIRshake \(ρ=0\.61\\rho=0\.61,p<0\.001p<0\.001,n=50n=50\), followed by FAIR\-Checker \(ρ=0\.47\\rho=0\.47,p<0\.001p<0\.001,n=49n=49\) and F\-UJI \(ρ=0\.42\\rho=0\.42,p=0\.002p=0\.002,n=50n=50\)\. The FAIR\-enough association is weaker and not statistically significant \(ρ=0\.31\\rho=0\.31,p=0\.084p=0\.084,n=32n=32\)\. This pattern reflects both rubric differences \(e\.g\., limitedInteroperabilitycoverage in some baseline rubrics\) and extraction strategies \(API\-centric vs\. browser\-centric\)\.

As a deliberately stricter diagnostic, we also compareAgentFAIRwith the single highest baseline score available for each dataset\. Under the same±5\\pm 5\-point band,AgentFAIRis higher on 6/50 datasets, comparable on 8/50, and lower on 36/50\. This “best baseline” is a composite oracle rather than one deployable tool; the result is reported to prevent the average\-baseline analysis from being misread as a superiority claim\.

Qualitatively, three system properties contribute to differences:

- •Contextual resolution of identifiers and access conditions:LLM agents can disambiguate DOI\-like strings, reconcile multiple identifiers, and interpret whether authentication statements are optional or mandatory\.
- •Geospatial sensitivity:Prompts explicitly recognize CRS identifiers, ISO metadata elements, and OGC endpoint mentions, which informsInteroperabilityand R1\.3 scoring\.
- •Critic\-based reconciliation:Cross\-sub\-principle checks trigger re\-evaluation when evidence suggests inconsistency \(e\.g\., protocol openness vs\. authentication clarity\)\.

### 5\.6\.Results: cost and efficiency analysis

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/fig4_cost_efficiency_quadrant.png)Scatter plot positioning datasets by evaluation cost versus overall FAIR score, highlighting low\-cost high\-score clusters\.

Figure 2\.Cost\-efficiency quadrant: datasets positioned by evaluation cost \(x\-axis\) vs\. overall FAIR score \(y\-axis\)\.Table 7\.AgentFAIRefficiency metrics across 50 datasetsFigure[2](https://arxiv.org/html/2607.15781#S5.F2)and Table[7](https://arxiv.org/html/2607.15781#S5.T7)summarize computational characteristics\. Zenodo datasets cluster in a “low cost / high score” region, reflecting stable structured metadata exposure\. More complex repositories and datasets require additional crawling and semantic reasoning, increasing processing time and token usage\. A cost breakdown by FAIR dimension appears in Appendix \(Figure[6](https://arxiv.org/html/2607.15781#A4.F6)\)\.

### 5\.7\.Consistency and auditability analysis

Across repeated evaluations during development \(5 runs per dataset on a 10\-dataset subset, stratified by FAIR maturity quintile\), mean exact agreement on sub\-principle scores is 89%, with a reported run\-level standard deviation of 3 percentage points\. We omit the earlier confidence interval because the retained summary statistics are insufficient to reconstruct it correctly\. This analysis measures repeatability, not reproducibility in the deterministic sense\. AgentFAIR also retains deterministic extraction failure modes; browser rendering and retry logic reduce some failures but do not eliminate them\.

Failure analysis\.Of 50 datasets, 3 \(6%\) required manual URL correction due to redirect chains or domain changes\. The critic triggered re\-evaluation in 96% of sub\-principle assessments\. A trigger means that at least one hard logical check failed or confidence fell below its threshold; it does not by itself show that the initial score was wrong or that the revised score is more accurate\. In contrast, F\-UJI failed completely on 10 datasets \(20%\) due to identifier resolution errors, and FAIR\-enough encountered database errors on 18 datasets \(36%\)\.

Human evaluation protocol\.Three domain experts with 5\+ years of geospatial data management experience independently evaluated a stratified random sample of 15 datasets \(3 per maturity quintile\)\. After a calibration session with worked rubric examples, each expert independently assessed all 13 sub\-principles without access to automated tool outputs \(195 judgments per expert; 585 annotations in total\)\. Per\-dataset annotation time ranged from a few minutes for simple records to roughly 20 minutes for complex cases\. Inter\-rater agreement \(Fleiss’κ\\kappa=0\.71, 95% CI: \[0\.65, 0\.77\]\) indicates substantial agreement\.AgentFAIRachieved 82% alignment with expert consensus \(majority vote\); disagreements concentrate on I2 \(interpreting vocabulary FAIRness,κ\\kappa=0\.58\) and R1\.2 \(provenance completeness thresholds,κ\\kappa=0\.61\), indicating where rubric definitions can be further refined\.

Baseline comparison on the expert subset\.The baseline tools do not expose outputs directly compatible with all 13 ordinal expert labels, so a head\-to\-head sub\-principle accuracy comparison is not available\. As a coarser diagnostic, we aggregate the expert labels to an overall score and apply a±5\\pm 5\-point dataset\-level band\. F\-UJI and FAIR\-Checker each fall within the band on 2/15 datasets, FAIRshake on 4/15, and FAIR\-enough on 0/10 returned outputs \(five failures\)\. These figures are not commensurate withAgentFAIR’s 82% sub\-principle alignment and should not be interpreted as proof of superior accuracy\. A larger study with tool outputs mapped and judged at the same sub\-principle level remains necessary\.

### 5\.8\.Within\-family model transfer

To probe dependence on GPT\-4o\-mini, we reran the identical pipeline and prompts with GPT\-4o on the same stratified 10\-dataset subset\. Dataset rankings are strongly associated \(ρ=0\.877\\rho=0\.877,p<0\.001p<0\.001\); 92/130 sub\-principle scores are exact matches and 121/130 \(93\.1%\) differ by at most one maturity level\. The mean absolute difference in normalized score is 7\.1 percentage points\. This result is supportive within the GPT family, but it does not establish robustness to model version changes or transfer to a different model family\.

### 5\.9\.Ablation study: critic agent contribution

Table 8\.Ablation: critic agent contribution \(nn=10 datasets\)Consistency = agreement across 5 repeated runs; Re\-eval Rate = percentage of sub\-principle assessments triggering critic review\.

To isolate the critic agent’s contribution, we conducted an ablation study on 10 representative datasets \(2 from each FAIR maturity quintile, spanning Zenodo, Dryad, PANGAEA, Harvard Dataverse, and NOAA NCEI\)\. Table[8](https://arxiv.org/html/2607.15781#S5.T8)compares fullAgentFAIRagainst a variant with the critic agent disabled\. Without critic\-based quality control, consistency drops from 89±\\pm3% to 71±\\pm5% \(p<0\.01p<0\.01, pairedtt\-test\), with the largest degradation on I2 and R1\.2—sub\-principles requiring nuanced evidence interpretation\. The 96% trigger rate combines seven hard logical checks with confidence gates\. The experiment did not retain a trigger\-type breakdown, conditional score\-change distribution, latency contribution, or accuracy of the intermediate outputs; consequently, the ablation supports a repeatability benefit but does not fully characterize whether the critic corrects genuine errors\. A plain\-retry/majority\-vote baseline was not run\.

### 5\.10\.Limitations and future experiments

The study is exploratory and has several unresolved validity limits\. First, 50 datasets and 15 expert\-annotated datasets do not support broad claims across the long tail of geospatial repositories\. Second, the cross\-tool normalization conflates different constructs and is retained only for disagreement analysis; baseline runtimes were not systematically logged\. Third, the current evidence does not isolate browser engineering, LLM enrichment, geospatial prompt terms, multi\-agent decomposition, and critic retries\. Geospatial\-prompt, rule\-only extraction, temperature 0\.0 versus 0\.1, and plain\-retry ablations remain outstanding\. Fourth, GPT\-4o is a within\-family check rather than cross\-family validation, and proprietary model updates threaten longitudinal repeatability\. Fifth, the strict I2 registry criterion can under\-credit valid community vocabularies absent from generic registries; the low expert agreement on I2 reinforces this concern\. Sixth, prompt\-injection and poisoned\-metadata robustness have not been evaluated\. Finally, raw repeated\-run, human\-label, ablation, and model\-transfer outputs should be released with analysis scripts before the reported inferential statistics are treated as independently reproducible\.

## 6\.Conclusion

We introducedAgentFAIR, an LLM\-driven multi\-agent framework for auditable FAIRness evaluation of geospatial datasets\. The system combines browser\-based and structured metadata extraction, 13 rubric\-specific evaluators, deterministic checks, and critic\-guided retries\.

In a 50\-dataset, 10\-repository sample,Findabilityis strongest \(79\.7%\) andInteroperabilityweakest \(45\.3%\)\. Cross\-tool normalized scores show substantial disagreement \(mean per\-dataset standard deviation 15\.0 points\), but heterogeneous rubrics prevent interpreting those differences as comparative accuracy\. Preliminary repeated\-run and expert studies support further investigation of the critic and evidence model, while the high retry rate, small human sample, missing component ablations, and within\-family\-only model check leave important questions unresolved\. At approximately $0\.054 per dataset, the framework is economically feasible for larger validation studies; stronger accuracy or deployment claims require released raw evaluation artifacts, cross\-family testing, and adversarial security evaluation\.

## References

- C\. Bahlo, S\. Guru, N\. Car, and L\. Wyborn \(2024\)Advancing fair agricultural data: the agrefed fair assessment tool\.Data Science Journal23\(1\),pp\. 15\.External Links:[Document](https://dx.doi.org/10.5334/dsj-2024-018)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.11.10.1.1)\.
- L\. Busch, D\. Tebernum, and G\. Velarde \(2025\)Exploring llm capabilities in extracting dcat\-compatible metadata for data cataloging\.arXiv preprint arXiv:2507\.05282\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.05282)Cited by:[§2\.3](https://arxiv.org/html/2607.15781#S2.SS3.p1.1)\.
- L\. Candela, D\. Mangione, and G\. Pavone \(2024\)The fair assessment conundrum: reflections on tools and metrics\.Data Science Journal23\(1\)\.External Links:[Document](https://dx.doi.org/10.5334/dsj-2024-033)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.15781#S2.SS1.p1.1)\.
- D\. J\. B\. Clarke, L\. Wang, A\. Jones, M\. L\. Wojciechowicz, D\. Torre, K\. M\. Jagodnik, S\. L\. Jenkins, P\. McQuilton, Z\. Flamholz, M\. C\. Silverstein,et al\.\(2019\)FAIRshake: toolkit to evaluate the fairness of research digital resources\.Cell Systems9\(5\),pp\. 417–421\.External Links:[Document](https://dx.doi.org/10.1016/j.cels.2019.09.011)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.3.2.1.1),[§1](https://arxiv.org/html/2607.15781#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15781#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.15781#S2.T1.1.1.3.2.1)\.
- A\. Devaraju and R\. Huber \(2020\)F\-uji fair assessment tool\.Note:Accessed: 2026\-01\-08External Links:[Link](https://www.f-uji.net/)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.2.1.1.1),[§1](https://arxiv.org/html/2607.15781#S1.p2.1),[§1](https://arxiv.org/html/2607.15781#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15781#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.15781#S2.T1.1.1.2.1.1)\.
- V\. Emonet and M\. Dumontier \(2022\)FAIR\-enough: a community\-governed fair maturity assessment framework\.External Links:[Link](https://github.com/MaastrichtU-IDS/fair-enough)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.8.7.1.1),[§1](https://arxiv.org/html/2607.15781#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15781#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.15781#S2.T1.1.1.5.4.1)\.
- A\. Gaignard, T\. Rosnet, F\. De Lamotte, V\. Lefort, and M\. Devignes \(2023\)FAIR\-checker: supporting digital resource findability and reuse with knowledge graphs and semantic web standards\.Journal of Biomedical Semantics14\(1\),pp\. 7\.External Links:[Document](https://dx.doi.org/10.1186/s13326-023-00289-5)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.6.5.1.1),[§1](https://arxiv.org/html/2607.15781#S1.p2.1),[§1](https://arxiv.org/html/2607.15781#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15781#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.15781#S2.T1.1.1.4.3.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber \(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=VtmBAGCN7o)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- Z\. Huang and E\. Wu \(2024\)Cocoon: semantic table profiling using large language models\.InProceedings of the 2024 Workshop on Human\-In\-the\-Loop Data Analytics \(HILDA ’24\),pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1145/3665939.3665957),[Link](https://arxiv.org/abs/2404.12552)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.9.8.1.1),[§2\.3](https://arxiv.org/html/2607.15781#S2.SS3.p1.1)\.
- International Organization for Standardization \(2014\)ISO 19115\-1:2014 geographic information — metadata part 1: fundamentals\.Note:Published Edition 1, last reviewed and confirmed in 2019, with 2 amendmentsExternal Links:[Link](https://www.iso.org/standard/53798.html)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.15781#S2.SS2.p1.1)\.
- N\. A\. Krans, A\. Ammar, P\. Nymark, E\. L\. Willighagen, M\. I\. Bakker, and J\. T\. K\. Quik \(2022\)FAIR assessment tools: evaluating use and performance\.NanoImpact27,pp\. 100402\.External Links:[Document](https://dx.doi.org/10.1016/j.impact.2022.100402)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p2.1)\.
- Lang Chain Authors \(2026\)LangGraph: agent orchestration framework for reliable ai agents\.Note:[https://www\.langchain\.com/langgraph](https://www.langchain.com/langgraph)Accessed on: 2026\-02\-08Cited by:[Appendix C](https://arxiv.org/html/2607.15781#A3.p1.1),[§4\.1](https://arxiv.org/html/2607.15781#S4.SS1.p1.1)\.
- L\. Li, L\. Fang, and V\. I\. Torvik \(2024\)Autodcworkflow: llm\-based data cleaning workflow auto\-generation and benchmark\.arXiv preprint arXiv:2412\.06724\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.06724)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.4.3.1.1),[§2\.3](https://arxiv.org/html/2607.15781#S2.SS3.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang \(2024\)AgentBench: evaluating llms as agents\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=zAdUB0aCTQ)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- G\. Mai, C\. Cundy, K\. Choi, Y\. Hu, N\. Lao, and S\. Ermon \(2024\)On the opportunities and challenges of foundation models for geoai\.ACM Computing Surveys56\(10\),pp\. 1–41\.External Links:[Document](https://dx.doi.org/10.1145/3653070)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- A\. Narayan, I\. Chami, L\. Orr, S\. Arber, P\. Rong, M\. Shen, and C\. Ré \(2024\)Can foundation models wrangle your data?\.Proceedings of the VLDB Endowment17\(5\),pp\. 1212–1225\.External Links:[Document](https://dx.doi.org/10.14778/3641204.3641227)Cited by:[§2\.3](https://arxiv.org/html/2607.15781#S2.SS3.p1.1)\.
- Open Geospatial Consortium \(2022\)OGC disaster pilot 2022: integrating iso and ogc standards for fair geospatial data\.Note:OGC Innovation ProgramExternal Links:[Link](https://www.ogc.org/initiatives/disaster-pilot/)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.12.11.1.1)\.
- D\. Paprotny and M\. Mengel \(2023\)Population, land use and economic exposure estimates for europe at 100 m resolution from 1870 to 2020\.Scientific Data10\(1\),pp\. 372\.External Links:[Document](https://dx.doi.org/10.1038/s41597-023-02282-0)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p1.1)\.
- PwC EU Services \(2018\)Cost\-benefit analysis for fair research data\.Technical reportEuropean Commission\.External Links:[Link](https://op.europa.eu/en/publication-detail/-/publication/d375368c-1a0a-11e9-8d04-01aa75ed71a1)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p1.1)\.
- S\. N\. Sakib, K\. Naha, S\. Y\. Rubaiat, and H\. M\. Jamil \(2025\)A genai system for improved fair independent biological database integration\.ACM Journal of Data and Information Quality17\(4\),pp\. 1–29\.External Links:[Document](https://dx.doi.org/10.1145/3770753),[Link](https://arxiv.org/abs/2506.17934)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.10.9.1.1)\.
- P\. T\. Thompson, S\. Ojha, C\. D\. Powell, K\. G\. Pennell, and H\. N\. B\. Moseley \(2023\)A proposed fair approach for disseminating geospatial information system maps\.Scientific Data10\(1\),pp\. 389\.External Links:[Document](https://dx.doi.org/10.1038/s41597-023-02281-1)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p1.1)\.
- TKFDM \(2023\)FAIR data assessment tool\.Note:University of ZurichExternal Links:[Link](https://www.ubs.uzh.ch/de/TKFDM/Services/FAIR-Assessment.html)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.5.4.1.1)\.
- L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. Wen \(2024\)A survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\),pp\. 186345\.External Links:[Document](https://dx.doi.org/10.1007/s11704-024-40231-1)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- M\. D\. Wilkinson, M\. Dumontier, I\. J\. Aalbersberg, G\. Appleton, M\. Axton, A\. Baak, N\. Blomberg, J\. Boiten, L\. B\. da Silva Santos, P\. E\. Bourne,et al\.\(2016\)The fair guiding principles for scientific data management and stewardship\.Scientific Data3\(1\),pp\. 1–9\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1038/sdata.2016.18)Cited by:[§1](https://arxiv.org/html/2607.15781#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.15781#S3.SS1.p1.5),[§3\.1](https://arxiv.org/html/2607.15781#S3.SS1.p3.1)\.
- M\. D\. Wilkinson, M\. Dumontier, and L\. O\. Bonino da Silva Santos \(2019a\)FAIR evaluator: the fair evaluation services\.Note:GO FAIR FoundationExternal Links:[Link](https://fairsharing.github.io/FAIR-Evaluator-FrontEnd/)Cited by:[Appendix G](https://arxiv.org/html/2607.15781#A7.1.p1.1.1.7.6.1.1)\.
- M\. D\. Wilkinson, M\. Dumontier, S\. Sansone, L\. O\. Bonino da Silva Santos, M\. Prieto, D\. Batista, P\. McQuilton, T\. Kuhn, P\. Rocca\-Serra, M\. Crosas,et al\.\(2019b\)Evaluating fair maturity through a scalable, automated, community\-governed framework\.Scientific Data6\(1\),pp\. 174\.External Links:[Document](https://dx.doi.org/10.1038/s41597-019-0184-5)Cited by:[§3\.2](https://arxiv.org/html/2607.15781#S3.SS2.p1.3)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2024\)AutoGen: enabling next\-gen llm applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.Note:Microsoft ResearchExternal Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.08155)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- Z\. Xi, W\. Chen, X\. Guo, W\. He, Y\. Ding, B\. Hong, M\. Zhang, J\. Wang, S\. Jin, E\. Zhou, R\. Zheng, X\. Fan, X\. Wang, L\. Xiong, Y\. Zhou, W\. Wang, C\. Jiang, Y\. Zou, X\. Liu, Z\. Yin, S\. Dou, R\. Weng, W\. Cheng, Q\. Zhang, W\. Qin, Y\. Zheng, X\. Qiu, X\. Huang, and T\. Gui \(2024\)The rise and potential of large language model based agents: a survey\.arXiv preprint arXiv:2309\.07864\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2309.07864)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- F\. F\. Xu, Y\. Wang, B\. Sharma, H\. Peng, H\. H\. Liu, J\. Y\. Chen, S\. Lin, Y\. Yang, Z\. Sun, L\. D\. Zheng, S\. I\. Ding, N\. Hou, S\. Ni, J\. Hsu, R\. Ramanujan, Y\. Kim, T\. Bhasker, C\. Wu, and X\. Xu \(2024\)TheAgentCompany: benchmarking llm agents on consequential real world tasks\.External Links:[Link](https://arxiv.org/abs/2412.14161),[Document](https://dx.doi.org/10.48550/arXiv.2412.14161)Cited by:[§2\.4](https://arxiv.org/html/2607.15781#S2.SS4.p1.1)\.
- S\. Zhang, Z\. Huang, and E\. Wu \(2025\)Data cleaning using large language models\.In2025 IEEE 41st International Conference on Data Engineering Workshops \(ICDEW\),pp\. 28–32\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.15547)Cited by:[§2\.3](https://arxiv.org/html/2607.15781#S2.SS3.p1.1)\.

## Appendix AOpen\-source artifact and prompt specifications

### A\.1\.Zenodo release and contents

An archived software snapshot is available on Zenodo \([https://doi\.org/10\.5281/zenodo\.18529560](https://doi.org/10.5281/zenodo.18529560)\); the maintained source is intended for[https://github\.com/MingCHEN\-Github/AgentFAIR](https://github.com/MingCHEN-Github/AgentFAIR)\. The Zenodo deposit contains:AgentFAIR\.zip\(full source code\),DESCRIPTION\.md\(setup \+ artifact overview\), andapp\_demo\.mp4\(end\-to\-end walkthrough video\)\. The software is licensed under AGPL\-3\.0\-only\.

### A\.2\.Quick start and environment configuration

Prerequisites\.Python 3\.10\+ and an OpenAI API key\. The archived artifact is self\-contained \(no git required\)\.

One\-command launch\.

> unzip AgentFAIR\.zip cd AgentFAIR \./run\.sh

run\.shautomates setup and execution, including dependency installation, Playwright browser installation for web crawling, creation of\.envfrom\.env\.examplewhen needed, launching the FastAPI backend \(http://localhost:8000\), and opening the React/Vite frontend in a browser\.

API key configuration\.Keys can be provided either \(i\) via the frontend*Settings*modal at runtime \(recommended for quick evaluation\) or \(ii\) via a local\.envfile\. The artifact also supports optional keys for Brave Search \(BRAVE\_API\_KEY\), OpenRouter \(OPENROUTER\_API\_KEY\), and opt\-in LangSmith tracing \(LANGSMITH\_API\_KEY\)\. Remote tracing is disabled by default\.

### A\.3\.Repository layout and persisted outputs

UnzippingAgentFAIR\.zipproduces anAgentFAIR/directory\. Key components:

> fair\_agents/ core Python package \(agents, tools, LangGraph workflows, reporting\) fair\_agents/api/ FastAPI service \(local evaluation endpoints\) frontend/ React/Vite client \(run orchestration \+ results UI\) storage/ generated SQLite evidence store \(ignored by Git\) fair\_agents/reports/ generated Markdown/JSON reports \(ignored by Git\) tests/ URL\-safety regression tests run\.sh single\-command launcher

Audit trail outputs\.Each evaluation persists \(i\) per\-sub\-principle scores and confidence, \(ii\) evidence bundles with provenance pointers/snippets, and \(iii\) exported reports \(Markdown \+ JSON\) backed by a SQLite evidence store\. Optional LangSmith traces can be enabled when a tracing key is provided\.

### A\.4\.Representative prompt template and output schema

AgentFAIRprompts follow a uniform multi\-part structure: \(i\) sub\-principle definition and scope, \(ii\) maturity rubric \(0–3; operationalization in Table 2\), \(iii\) decision logic and edge cases, \(iv\) geospatial adaptations \(when applicable\), \(v\) evidence requirements, \(vi\) anti\-hallucination constraints, \(vii\) output JSON schema, and \(viii\) short worked examples\.

Below we show a simplified*illustrative*template for a sub\-principle agent\. In practice, each agent instantiates this template with the corresponding rubric criteria and geospatial indicators from the operationalization in Table 2\.

> Input:extracted metadata record M \(structured fields \+ landing page snippets\)\. Task:Evaluate sub\-principle \{F1/F2/\.\.\.\} using the rubric below\. Rubric:score 0/1/2/3 with explicit criteria\. Evidence rule:every claim must cite a field or quoted snippet from M\. Output JSON: \{ "principle": "\.\.\.", "score": 0\|1\|2\|3, "confidence": 0\.0\-\-1\.0, "evidence": \[\{"source": "\.\.\.", "snippet": "\.\.\."\}, \.\.\.\], "rationale": "\.\.\.", "recommendations": \["\.\.\.", \.\.\.\] \}

## Appendix BFurther Discussions ofAgentFAIR

### B\.1\.Key insights

Interoperabilityis the dominant bottleneck\.Across repositories, I2 and I3 consistently score lowest\. A key driver is that widely used geospatial vocabularies and standards are not consistently represented in the registries relied upon by semantic FAIR validators, which leads to systematic penalties\. This suggests that improving ecosystem\-level vocabulary registration \(not only dataset\-level metadata\) is critical for improving geospatial FAIRness\.

Rule\-based tools are brittle on modern repository interfaces\.We observe sharp baseline failures when crawlers cannot access machine\-readable metadata \(e\.g\., JavaScript\-rendered pages\) or when identifier resolution relies on assumptions that do not hold universally\. These failures can dominate overall tool disagreement and complicate the interpretation of FAIR scores as decision signals\.

LLM\-based evaluation can be useful but must be auditable\.LLM agents help interpret ambiguous natural language and recover domain cues, but they introduce stochasticity\. InAgentFAIR, evidence\-grounded prompts, explicit rubrics, and critic\-based consistency checks are essential to make LLM reasoning usable for compliance assessment\.

### B\.2\.Threats to validity

Dataset sampling\.Our benchmark emphasizes cross\-repository diversity, but 50 datasets is not exhaustive; results may shift under different sampling strategies or when focusing on a single community\.

Baseline normalization\.Normalizing heterogeneous tool outputs to a single 0–100 score enables comparison but necessarily abstracts away rubric differences\. We therefore interpret correlations and variance as indicators of*disagreement*, not as absolute correctness\.

Lack of ground\-truth FAIR labels\.FAIRness is not a single objective quantity; expert judgments may differ, and community standards evolve\. We address this only partially by reporting evidence and documenting disagreements; the preliminary expert sample is not a definitive ground truth\.

LLM\-specific threats\.The primary evaluation relies on GPT\-4o\-mini, which may change without a versioned model snapshot\. The GPT\-4o check provides only within\-family evidence\. Low temperature does not guarantee determinism, and no temperature ablation was run\. Prompt phrasing can affect score distributions, and the prompts were iteratively refined during development\. Evidence requirements and critic checks improve auditability but cannot prevent hallucination, prompt injection, or subtle misinterpretation of poisoned or ambiguous metadata\. Repository\-specific overfitting therefore remains possible\.

I2 construct validity\.The experimental I2 rubric places substantial weight on vocabulary registry presence\. Valid geospatial vocabularies may be documented and accepted in community practice without appearing in generic registries, so the rubric can systematically under\-credit them\. Future work should combine registry evidence with community governance, resolvability, versioning, licensing, and documentation rather than treating registry count as a normative definition of FAIRness\.

### B\.3\.Applications

AgentFAIRsupports: \(i\)repository FAIRification\(prioritized remediation roadmaps\), \(ii\)CI/CD integration\(continuous metadata quality monitoring\), \(iii\)funding compliance reporting\(evidence\-backed audit trails\), and \(iv\)training data curation\(filtering and ranking datasets for geospatial foundation models based on FAIR maturity\)\.

## Appendix CLangGraph workflow during evaluation process

The proposedAgentFAIRemploys LangGraph, an open\-source orchestration library in the LangChain ecosystem\(Lang Chain Authors,[2026](https://arxiv.org/html/2607.15781#bib.bib18)\), to design multi\-agent workflows as directed state machines\. As illustrated in Figure[3](https://arxiv.org/html/2607.15781#A3.F3), each FAIR dimension operates as an independent graph composed of modular nodes—crawling, metadata extraction, principle\-specific evaluation, critic review, and reporting—connected through conditional and retry\-based edges\. Shared state variables capture crawl results, agent confidence, and error counts, enabling traceable and auditable execution\.

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/LangGraph.png)Workflow diagram showing four FAIR\-dimension subgraphs with crawl, extract, evaluate, critic, and report nodes, including retry loops\.

Figure 3\.LangGraph state machine orchestrating FAIR evaluation workflows with conditional edges and retry logic\.
## Appendix DAdditional plots and tables

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/fig2_time_boxplot.png)Boxplots of processing time by FAIR dimension across datasets\.

Figure 4\.Processing time distribution by FAIR dimension\.Figure[4](https://arxiv.org/html/2607.15781#A4.F4)shows that processing time varies by FAIR dimension;Reusabilityis fastest, whileFindabilityandAccessibilitytake longest due to crawling and metadata extraction overhead\. Most evaluations cluster in the 150–350 second range per dimension, with outliers corresponding to metadata\-heavy datasets\.

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/fig3_token_time_scatter.png)Scatter plot showing the relationship between token usage and processing time\.

Figure 5\.Token usage vs\. processing time correlation\.Figure[5](https://arxiv.org/html/2607.15781#A4.F5)reveals a strong positive correlation between token usage and processing time, confirming that LLM API calls dominate wall\-clock time\. This relationship suggests that token\-reduction strategies can directly lower both evaluation cost and latency\.

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/fig6_cost_breakdown.png)Bar chart breaking down API cost by FAIR dimension\.

Figure 6\.API cost breakdown by FAIR dimension across 50 datasets\.Figure[6](https://arxiv.org/html/2607.15781#A4.F6)sorts datasets by total cost \(ascending\), withReusabilityconsistently contributing the largest cost share andInteroperabilitythe smallest\. Total evaluation cost for all 50 datasets remains under $3, demonstrating the economic viability of LLM\-based FAIR assessment at scale\.

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/visualization_3_methodology_radar.png)Radar plot comparing tools across FAIR dimensions and methodological attributes\.

Figure 7\.FAIR dimension coverage radar chart across tools\.Figure[7](https://arxiv.org/html/2607.15781#A4.F7)compares mean dimension\-level scores across five FAIR assessment tools\. AgentFAIR and F\-UJI show relatively balanced profiles across all four dimensions, whereas FAIRshake lacksInteroperabilitycoverage entirely and FAIR\-enough tends to score lower due to its binary test methodology\.

![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/fig5_principle_maturity_landscape.png)Scatter plot summarizing each FAIR principle by mean maturity and variance across datasets\.

Figure 8\.FAIR principle maturity landscape: mean score vs\. variance across datasets\.![Refer to caption](https://arxiv.org/html/2607.15781v1/figures/visualization_4_dataset_controversy.png)Bar chart showing per\-dataset score variance across tools, labeled as a controversy index\.

Figure 9\.Dataset controversy index: scoring variance across tools\.Figure[9](https://arxiv.org/html/2607.15781#A4.F9)ranks datasets by scoring variance \(standard deviation\) across tools, with high\-controversy datasets exhibiting ¿25% standard deviation\. These datasets expose fundamental differences in how each tool operationalizes FAIR principles and serve as valuable diagnostic benchmarks for tool comparison\.

Figure[8](https://arxiv.org/html/2607.15781#A4.F8)plots each FAIR sub\-principle by its mean maturity score and cross\-dataset variance\.FindabilityandAccessibilityprinciples cluster in the “Universal Strength” quadrant \(high mean, low variance\), whileInteroperabilitysub\-principles fall into the “High Discriminator” or “Systemic Gap” regions, confirming that I\-dimension compliance is the most variable and challenging aspect of FAIR maturity\.

## Appendix EDataset details

Table[9](https://arxiv.org/html/2607.15781#A5.T9)summarizes the key detailed information for all the evaluated 50 datasets\.

Table 9\.Full dataset summary \(50 datasets, 10 repositories\)\.
## Appendix FArtifact and auditability checklist

- •Hardware:Apple M2 Pro, 32 GB RAM, macOS 15
- •Software:Python 3\.11; current release requires Python 3\.10\+, LangChain Core 1\.x, and LangGraph 1\.x
- •Model:GPT\-4o\-mini via OpenAI API \(temperature=0\.1\)
- •Dependencies:Playwright, FastAPI, SQLite, Extruct, lxml, and rdflib
- •Logging:prompts, responses, evidence, and scores are persisted as an audit trail
- •
- •Missing research artifacts:raw repeated\-run outputs, expert labels, ablation runs, model\-transfer outputs, and analysis notebooks are not included in the current software checkout and are required for independent reproduction of the reported statistics

## Appendix GComparative analysis of FAIR assessment systems

Table 10\.Comparative analysis of FAIR assessment systems and this work’s contributions

## Appendix HDiagnostic comparison with the baseline average

For completeness, averaging the available baseline scores per dataset and applying a±5\\pm 5\-point band placesAgentFAIRabove that average on 20/50 datasets, within the band on 18/50, and below it on 12/50\. This statistic is sensitive to missing baseline outputs and mixes non\-equivalent rubrics\. It is therefore retained only as a descriptive disagreement summary; the stricter best\-baseline result is reported in Section 5, and neither comparison is treated as an accuracy ranking\. We omit the earlier per\-dataset “higher/lower” narratives because many large differences were attributable to crawler and redirect failures rather than the LLM or multi\-agent design\.