EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation

# EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation
Source: [https://arxiv.org/html/2607.01584](https://arxiv.org/html/2607.01584)
###### Abstract

Large language models have recently been explored for scientific hypothesis generation, but most prior work relies on unstructured literature and free\-form textual claims\. We present a pipeline for Earth observation that grounds hypothesis generation directly in the NASA Earth Observation Knowledge Graph\. A heterogeneous graph neural network trained on historical co\-usage relations ranks candidate dataset pairings, and a three\-agent LLM pipeline filters, generates, and evaluates structured research hypotheses\. Applied to 1,475 NASA datasets, the system produces 160 hypotheses spanning multiple Earth\-science domains, including ecohydrology, glaciology, aerosol–cloud interactions, vegetation phenology, and stratospheric chemistry\. Model\-predicted novel dataset pairings are rated nearly as plausible as held\-out real co\-usages from the literature, indicating that the pipeline surfaces scientifically coherent yet unexplored combinations\. A2×2×22\\times 2\\times 2factorial experiment across GPT\-5\.2 and Claude Sonnet 4\.6 shows that hypothesis rankings remain stable, while absolute scores depend strongly on judge identity, highlighting limitations of single\-judge LLM evaluation\. Code, dataset, and generated hypotheses can be found[here](https://github.com/Mahyar-GH79/EO_Agent)\.

knowledge graphs, graph neural networks, hypothesis generation, LLM\-as\-judge, Earth observation, scientific discovery, multi\-agent evaluation

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.01584v1/x1.png)Figure 1:Three\-panel overview of the pipeline\. \(a\) The NASA Earth Observation Knowledge Graph \(150,351 nodes, 436,203 edges across seven node types\)\. \(b\) A dense\-hop neighborhood around a candidate dataset pair is passed through a heterogeneous GNN; node embeddings𝐡i,𝐡j\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}are compared with a dot\-product scorerscore⁡\(i,j\)=σ​\(𝐡i⋅𝐡j\)\\operatorname\{score\}\(i,j\)=\\sigma\(\\mathbf\{h\}\_\{i\}\\cdot\\mathbf\{h\}\_\{j\}\)to produce a ranked list of 200 candidate pairs\. \(c\) Three LLM agents \(each independently GPT\-5\.2 or Claude Sonnet 4\.6\) filter, generate, and judge hypotheses, yielding 160 scored research hypotheses\.Earth\-observation \(EO\) research is fundamentally combinatorial\. A typical study fuses a soil\-moisture retrieval with a vegetation index, pairs a lidar canopy\-height product with a global digital\-elevation model, or cross\-references atmospheric chemistry against stratospheric profiles\. NASA alone distributes over 8,000 EO datasets across twelve archives, spanning hundreds of instruments and five decades of observations\. The scientific payoff is increasingly driven not by any single measurement but by the*pairing*a researcher chooses to study, and by whether that pairing has been attempted before\. The combinatorial space of unordered pairings,∼106\{\\sim\}10^\{6\}even under conservative filtering, far exceeds what any individual researcher can survey, and most of it remains scientifically unexplored\.

Recent work has proposed large language models \(LLMs\) as engines for scientific hypothesis generation, producing research ideas in chemistry\(Yanget al\.,[2025c](https://arxiv.org/html/2607.01584#bib.bib4),[b](https://arxiv.org/html/2607.01584#bib.bib9)\), biomedicine\(Wanget al\.,[2024a](https://arxiv.org/html/2607.01584#bib.bib6)\), materials science\(Ghafarollahi and Buehler,[2025](https://arxiv.org/html/2607.01584#bib.bib8)\), astrobiology\(Saeediet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib11)\), and machine\-learning research itself\(Siet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib1); Luet al\.,[2024](https://arxiv.org/html/2607.01584#bib.bib2); Yamadaet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib3)\)\. The dominant paradigm grounds LLM generation in unstructured scientific text: a retriever surfaces related papers, a generator composes an idea from their content, and an evaluator scores the result\(Baeket al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib5); Yuet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib7)\)\. This paradigm is effective for literature\-driven domains where hypotheses are themselves textual claims; in observational, data\-rich domains such as EO, however, a useful hypothesis is inseparable from the specific measurement products that would test it\. Saying “use satellite data to study drought” is not a hypothesis; saying “combine SPL4SMGP soil moisture with MYD13Q1 vegetation indices to test whether 16\-day EVI response depends on biome\-specific soil\-moisture thresholds” is\. A literature\-grounded pipeline can in principle reach the second statement by extracting dataset mentions from retrieved papers, but it must do so as a side\-effect of free\-form text generation\. We argue that grounding the retrieval step directly in a typed knowledge graph of measurement products is a more direct route, because every generated hypothesis is then pinned to two named NASA datasets by construction\.

We therefore explore a complementary architecture in which the retrieval step itself is structured\. We retrieve over a heterogeneous knowledge graph of EO research artifacts, where datasets, platforms, instruments, science keywords, and projects appear as typed nodes, and dataset co\-usage \(derived from joint citation by a shared publication\), platform–instrument mounting, and dataset–metadata membership appear as typed edges\. A heterogeneous graph neural network, trained by link prediction on past co\-usages \(publications through 2022\), ranks candidate dataset pairings by predicted co\-usage likelihood and surfaces the top 200 pairs not observed in the training, validation, or test co\-usage sets\. A three\-agent LLM pipeline then refines this list: a*filter*agent rescores the top candidates on plausibility and novelty, a*generator*agent articulates a structured research hypothesis \(question, testable claim, analysis method, expected finding\) for each surviving pair, and a*judge*agent rates the resulting hypothesis on importance, tractability, and novelty under both blind and contextual conditions\.[Figure1](https://arxiv.org/html/2607.01584#S1.F1)shows the complete pipeline\.

Applied to the NASA Earth Observation Knowledge Graph, the pipeline produces 160 hypotheses spanning multiple Earth science domains\. Held\-out real co\-usages and model\-predicted novel pairings receive comparable plausibility scores from LLM judges, suggesting that the system surfaces scientifically coherent yet unexplored combinations\. A representative example pairs IceBridge airborne magnetometer with InSAR\-derived Antarctic ice velocity: the generator proposes that subglacial magnetic anomalies predict basal\-friction patterns underlying modern ice flow—linking solid\-Earth geology to cryospheric dynamics with public archives\.

Because these scores can shift depending on which model plays which role, we test the pipeline under all eight combinations of GPT\-5\.2 and Claude Sonnet 4\.6 across the three agent roles\. These eight setups produce 640 judgments across the 160 hypotheses\. We find that the*ranking*of hypotheses stays consistent \(159 of 160 hypotheses change by at most 2 points across the four judge conditions\), but the*absolute*scores depend heavily on which model is the judge: about25%25\\%of the variation in importance scores comes from the judge alone, versus less than2%2\\%from the filter or the generator\. For tractability, the biggest factor is instead whether the judge sees the underlying datasets or only the hypothesis text \(16%16\\%of the variance\)\. Together, these findings suggest that relying on a single LLM judge is risky for absolute scoring, and motivate the multi\-judge protocol we use throughout the paper\.

Our contributions are threefold:

1. 1\.We introduce a hypothesis\-generation pipeline for observational science that ties each LLM\-generated hypothesis to a specific pair of NASA datasets, surfaced by a knowledge\-graph ranker rather than by free\-form text retrieval, yielding 160 structured hypotheses across many Earth\-science domains, with five flagship themes highlighted in[AppendixE](https://arxiv.org/html/2607.01584#A5)\.
2. 2\.We show that the pairs the model predicts as novel are rated nearly as plausible by LLM judges as held\-out real co\-usages from the published literature, which is evidence that the pipeline produces scientifically coherent ideas beyond what the model has already seen\.
3. 3\.We run a2×2×22\{\\times\}2\{\\times\}2factorial over filter, generator, and judge identity and find that, although hypothesis rankings are stable across setups, the absolute scores depend strongly on which model is the judge—a warning about relying on a single LLM judge for this kind of evaluation\. We release the knowledge\-graph preprocessing, the trained GNN, all 160 hypotheses, the 640 validator judgments, and the full analysis code\.

## 2Related Work

#### LLM hypothesis generation from literature\.

A growing line of work uses LLMs to generate research ideas grounded in*text*\. Prior systems explore retrieval\-based and multi\-agent approaches for scientific ideation, including SciMON\(Wanget al\.,[2024a](https://arxiv.org/html/2607.01584#bib.bib6)\), ResearchAgent\(Baeket al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib5)\), MOOSE\-Chem\(Yanget al\.,[2025c](https://arxiv.org/html/2607.01584#bib.bib4)\), and ResearchTown\(Yuet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib7)\)\. Fully autonomous pipelines\(Luet al\.,[2024](https://arxiv.org/html/2607.01584#bib.bib2); Yamadaet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib3)\)further extend this to end\-to\-end ideation, experimentation, and manuscript writing, while large\-scale expert studies\(Siet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib1)\)show that LLM\-generated ideas can be judged more novel than human expert ideas, though often less feasible\. All of this work retrieves over*unstructured*scientific text and treats a hypothesis as a free\-form natural\-language claim\. In observational, data\-rich domains such as Earth observation, the unit of discovery is instead a specific pair of measurement products whose joint analysis is executable, which motivates our GNN\-over\-typed\-KG ranker \([Section3](https://arxiv.org/html/2607.01584#S3)\) in place of literature retrieval\.

#### Multi\-agent and data\-grounded discovery\.

Closer to our setting, multi\-agent pipelines ground generation in structured artifacts\. SciAgents\(Ghafarollahi and Buehler,[2025](https://arxiv.org/html/2607.01584#bib.bib8)\)reasons over an ontological knowledge graph with Ontologist, Scientist, and Critic roles for bioinspired materials; AstroAgents\(Saeediet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib11)\)generates astrobiology hypotheses from mass\-spectrometry measurements with eight role\-specialized agents; MOOSE\-Chem2\(Yanget al\.,[2025b](https://arxiv.org/html/2607.01584#bib.bib9)\)targets*fine\-grained*hypotheses with concrete methodology via hierarchical search; AutoDiscovery\(Agarwalet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib10)\)drives open\-ended discovery across 21 real datasets with Bayesian surprise as an LLM\-defined reward under Monte Carlo tree search\. Our pipeline shares their role specialization and structure\-grounded retrieval, but differs on two axes\. First, retrieval operates on a typed, empirical*dataset co\-usage*graph with a heterogeneous GNN ranker rather than per\-query literature retrieval or symbolic ontology traversal, so every generated hypothesis is pinned to two named NASA products and is actionable by construction\. Second, prior pipelines typically fix one LLM per role and —in the agentic case— equip that LLM with execution tools; we instead cross agent identity in a2×2×22\{\\times\}2\{\\times\}2factorial \([Section5](https://arxiv.org/html/2607.01584#S5)\) and report the resulting variance decomposition, which is particularly consequential when the reward itself is LLM\-defined, as inWanget al\.\([2024a](https://arxiv.org/html/2607.01584#bib.bib6)\)andAgarwalet al\.\([2025](https://arxiv.org/html/2607.01584#bib.bib10)\)\.

#### LLM\-as\-judge and evaluation reliability\.

The reliability of LLM\-based evaluation has been studied extensively\.Zhenget al\.\([2023](https://arxiv.org/html/2607.01584#bib.bib12)\)introduce the LLM\-as\-judge formulation and document position, verbosity, and self\-enhancement biases\.Kentonet al\.\([2024](https://arxiv.org/html/2607.01584#bib.bib15)\)analyze scalable\-oversight protocols in which weak LLM judges adjudicate strong LLM agents, identifying positional and order effects across nine reasoning asymmetries\.Yeet al\.\([2025](https://arxiv.org/html/2607.01584#bib.bib13)\)catalog twelve bias types and automate their quantification via CALM;Wataokaet al\.\([2024](https://arxiv.org/html/2607.01584#bib.bib14)\)isolate self\-preference bias and link it to perplexity, predicting that a judge will prefer outputs whose distribution is familiar to it\. Most recently,Yanget al\.\([2025a](https://arxiv.org/html/2607.01584#bib.bib16)\)propose a plug\-in Reasoning\-based Bias Detector for post\-hoc correction\. Our factorial contributes an*empirical*variance decomposition in the scientific\-hypothesis evaluation regime: judge identity accounts for roughly25%25\\%of importance\-score variance, dwarfing filter or generator effects \(<2%\{<\}2\\%\), and the blind\-versus\-contextual contrast dominates tractability variance \(∼16%\{\\sim\}16\\%\)\. This is precisely the failure mode anticipated byWataokaet al\.\([2024](https://arxiv.org/html/2607.01584#bib.bib14)\)andYeet al\.\([2025](https://arxiv.org/html/2607.01584#bib.bib13)\), and the oneYanget al\.\([2025a](https://arxiv.org/html/2607.01584#bib.bib16)\)aims to mitigate, now quantified in the hypothesis\-evaluation setting for the first time\.

#### Machine learning for Earth observation\.

Rolfet al\.\([2024](https://arxiv.org/html/2607.01584#bib.bib23)\)argue that satellite data is a distinct ML modality and call for dedicated infrastructure\. The dominant response has been to scale foundation models over individual imagery products\(Stewartet al\.,[2023](https://arxiv.org/html/2607.01584#bib.bib24); Guoet al\.,[2024](https://arxiv.org/html/2607.01584#bib.bib25); Wuet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib26)\), surveyed inXiaoet al\.\([2024a](https://arxiv.org/html/2607.01584#bib.bib27)\)\. At the community level,Zhuet al\.\([2025](https://arxiv.org/html/2607.01584#bib.bib28)\)articulate eleven desiderata for ideal Earth foundation models and explicitly flag cross\-dataset knowledge synthesis as an open gap\. Our work targets precisely that complementary discovery layer: given hundreds of heterogeneous NASA products, which*pairings*are worth a researcher’s time? We address this question here\.

## 3Knowledge Graph and GNN Retrieval

#### Knowledge graph\.

We build on the publicly released NASA Earth Observation Knowledge Graph \(EO\-KG\)\(NASA Goddard Earth Sciences Data and Information Services Center \(GES\-DISC\),[2024](https://arxiv.org/html/2607.01584#bib.bib29)\): a single GraphML resource with 150,351 nodes and 436,203 edges, spanning seven node types \(Publication,Dataset,ScienceKeyword,Instrument,Platform,Project,DataCenter\) and nine typed relations covering citation, dataset usage, metadata membership, and a science\-keyword subcategory hierarchy\. The ranking universe consists of the 1,475 datasets that participate in at least one training\-period co\-usage pair \(i\.e\., are co\-cited with another dataset by at least one paper with year≤2022\\leq 2022\)\. This is by construction the natural evaluation pool for our link\-prediction task and is also the candidate pool used for negative sampling during training\. Per\-type counts and the full relation breakdown are in[AppendixA](https://arxiv.org/html/2607.01584#A1)\.

#### Deriving co\-usage supervision\.

The raw KG contains no direct dataset–dataset edge; we derive co\-usage fromPublication→\\toDatasetedges\. For every publicationppwith dataset setDp=\{dp,1,…,dp,k\}D\_\{p\}=\\\{d\_\{p,1\},\\dots,d\_\{p,k\}\\\},k≥2k\\geq 2, and every unordered pair\(di,dj\)∈Dp\(d\_\{i\},d\_\{j\}\)\\in D\_\{p\}, we emit a co\-usage observation tagged with the publication year\. Pairs are labeled binary \(co\-used vs\. not\); multiplicity is retained as a weight only for auxiliary analyses\. We then split pairs temporally by the publication year of each paper that observes them: train \(≤2022\\leq 2022\), val \(=2023=2023\), test \(=2024=2024\), giving 13,529 / 6,119 / 6,319 pairs\. A pair co\-cited in papers from multiple years enters multiple splits; we report*test pairs unseen in train*\(3,944\) as the strict held\-out subset used for evaluation\. Within test we carve two harder subpools:*cold\-start*\(2,284 pairs, at least one endpoint with no training co\-usage\) and*cross\-DAAC*\(1,480 pairs, endpoints in distinct archives\)\. Full statistics are reported in[Table7](https://arxiv.org/html/2607.01584#A1.T7)\.

#### Heterogeneous GNN ranker\.

Let𝒢\\mathcal\{G\}be the training graph induced by five surviving node types \(Dataset,ScienceKeyword,Instrument,Platform,Project\) and six typed relations:co\_usagebetween datasets, plushas\_platform,has\_keyword,has\_instrument,of\_project, andhas\_subcategory\. EachDatasetnode is initialized with a 768\-dim SPECTER2 abstract embedding\(Singhet al\.,[2023](https://arxiv.org/html/2607.01584#bib.bib22); Cohanet al\.,[2020](https://arxiv.org/html/2607.01584#bib.bib21)\); non\-dataset types receive Xavier\-initialized learnable embeddings of the same dimension\. Publication nodes are removed from𝒢\\mathcal\{G\}: they served only to derive co\-usage supervision, and retaining them would leak test pairs through shared neighbors\.

We learnd=128d\{=\}128dimensional node representations with a two\-layer heterogeneousGraphSAGE\(Zhang and Chen,[2018](https://arxiv.org/html/2607.01584#bib.bib18); Wanget al\.,[2024b](https://arxiv.org/html/2607.01584#bib.bib17)\)\. Within each layerℓ\\ell, for every typed relationrrand every target nodevvreachable throughrr, we form a per\-relation message

𝐦v\(ℓ\+1,r\)=𝐖r\(ℓ\)​MEANu∈𝒩r​\(v\)​𝐡u\(ℓ\),\\mathbf\{m\}\_\{v\}^\{\(\\ell\+1,\\,r\)\}\\;=\\;\\mathbf\{W\}\_\{r\}^\{\(\\ell\)\}\\,\\mathrm\{MEAN\}\_\{u\\in\\mathcal\{N\}\_\{r\}\(v\)\}\\,\\mathbf\{h\}\_\{u\}^\{\(\\ell\)\},\(1\)where𝒩r​\(v\)\\mathcal\{N\}\_\{r\}\(v\)are the neighbors ofvvunderrr\. Messages incoming tovvare summed across all incident relations and combined with the self\-state through a SAGE residual,

𝐡v\(ℓ\+1\)=σ​\(𝐖self\(ℓ\)​𝐡v\(ℓ\)\+∑r∈ℛv𝐦v\(ℓ\+1,r\)\),\\mathbf\{h\}\_\{v\}^\{\(\\ell\+1\)\}\\;=\\;\\sigma\\\!\\Big\(\\mathbf\{W\}\_\{\\mathrm\{self\}\}^\{\(\\ell\)\}\\mathbf\{h\}\_\{v\}^\{\(\\ell\)\}\+\\\!\\\!\\sum\_\{r\\in\\mathcal\{R\}\_\{v\}\}\\\!\\mathbf\{m\}\_\{v\}^\{\(\\ell\+1,\\,r\)\}\\Big\),\(2\)whereℛv\\mathcal\{R\}\_\{v\}is the set of relations incident onvv,σ=ReLU\\sigma\{=\}\\mathrm\{ReLU\}, and dropout0\.20\.2is applied to𝐡v\(ℓ\+1\)\\mathbf\{h\}\_\{v\}^\{\(\\ell\+1\)\}\. Per\-relation weights𝐖r\(ℓ\)\\mathbf\{W\}\_\{r\}^\{\(\\ell\)\}are*not*shared across relations; this is what distinguishes the heterogeneous variant from a homogeneous SAGE applied to the union graph and what allows the model to respect the feature\-heterophilic structure inherent to typed graphs\(Zhuet al\.,[2024](https://arxiv.org/html/2607.01584#bib.bib20)\)\. We useL=2L\{=\}2layers; the\(Dataset, co\_usage, Dataset\)relation participates in message passing on*training pairs only*\.

The final dataset embeddings𝐡i=𝐡i\(L\)\\mathbf\{h\}\_\{i\}\\\!=\\\!\\mathbf\{h\}\_\{i\}^\{\(L\)\}are passed through one of two scorers,

zdot​\(i,j\)\\displaystyle z\_\{\\mathrm\{dot\}\}\(i,j\)=𝐡i⊤​𝐡j,\\displaystyle=\\mathbf\{h\}\_\{i\}^\{\\\!\\top\}\\,\\mathbf\{h\}\_\{j\},\(3\)zmlp​\(i,j\)\\displaystyle z\_\{\\mathrm\{mlp\}\}\(i,j\)=MLP⁡\(\[𝐡i∥𝐡j\]\),\\displaystyle=\\operatorname\{MLP\}\\\!\\big\(\[\\mathbf\{h\}\_\{i\}\\,\\\|\\,\\mathbf\{h\}\_\{j\}\]\\big\),\(4\)with the MLP being a two\-layer network with hidden widthddandReLUactivation\. Both produce a logit; predicted co\-usage probability isσ​\(z​\(i,j\)\)\\sigma\(z\(i,j\)\)\. We report both heads because, as we show below, they form a Pareto pair across the evaluation pools\.

#### Training objective\.

Stage 2 is trained by binary cross\-entropy with logits\. Each minibatchℬ\\mathcal\{B\}samples one training positive\(s,t\+\)\(s,t^\{\+\}\)together with one negative\(s,t−\)\(s,t^\{\-\}\), and minimizes

ℒ​\(ℬ\)=−∑\(s,t\+,t−\)∈ℬ\[log⁡σ​\(z​\(s,t\+\)\)\+log⁡\(1−σ​\(z​\(s,t−\)\)\)\]\.\\mathcal\{L\}\(\\mathcal\{B\}\)=\-\\sum\_\{\(s,t^\{\+\},t^\{\-\}\)\\in\\mathcal\{B\}\}\\Big\[\\log\\sigma\(z\(s,t^\{\+\}\)\)\+\\log\\big\(1\-\\sigma\(z\(s,t^\{\-\}\)\)\\big\)\\Big\]\.\(5\)Negatives are drawn from a degree\-biased proposal,Pr\(t−=n\)∝deg\(n\)0\.75\\Pr\(t^\{\-\}\\\!=\\\!n\)\\\!\\propto\\\!\\deg\(n\)^\{0\.75\}\(matching theword2vecconvention of upweighting hub candidates so that the ranker is forced to discriminate against plausible distractors\), with rejection of anynnthat is itself a training positive ofss\. Optimization uses Adam at learning rate10−310^\{\-3\}for up to 300 epochs, with early stopping on validation MRR \(patience 30\)\.

#### Evaluation protocol\.

We evaluate as a 1:100 ranking task: each held\-out positive\(a,b\+\)\(a,b^\{\+\}\)is scored against 100 degree\-matched negatives\{bk−\}k=1100\\\{b^\{\-\}\_\{k\}\\\}\_\{k=1\}^\{100\}, and we report Hits@kk, mean reciprocal rank \(MRR\), AUC, and average precision \(AP\) on each of the three pools defined above\. Hits@kkis the fraction of held\-out positives ranked in the topkkover the 101\-element candidate list; MRR is the mean of1/rank​\(b\+\)1/\\mathrm\{rank\}\(b^\{\+\}\)\.

#### Baselines\.

We compare against six baselines spanning structure\-only and content\-only signals\.Popularityranks candidates by global degree:z​\(i,j\)=deg⁡\(j\)z\(i,j\)=\\deg\(j\), ignoringiientirely\.Common NeighborsandAdamic–Adarscore the candidate by overlap of co\-usage neighborhoods,z​\(i,j\)=∑u∈𝒩​\(i\)∩𝒩​\(j\)w​\(u\)z\(i,j\)=\\sum\_\{u\\in\\mathcal\{N\}\(i\)\\cap\\mathcal\{N\}\(j\)\}\\\!w\(u\), withw≡1w\\\!\\equiv\\\!1for the former andw​\(u\)=1/log⁡\|𝒩​\(u\)\|w\(u\)\\\!=\\\!1/\\log\|\\mathcal\{N\}\(u\)\|for the latter\.MF\-SVDfactorizes the binary publication×\\timesdataset co\-citation matrix by truncated SVD \(k=128k=128\) and scores pairs by inner product of the resulting dataset rows\.SPECTER2\(Singhet al\.,[2023](https://arxiv.org/html/2607.01584#bib.bib22)\)andBGE\-base\(Xiaoet al\.,[2024b](https://arxiv.org/html/2607.01584#bib.bib30)\)are dense\-retrieval baselines that score pairs by cosine similarity of the corresponding dataset abstract embeddings, giving us a content\-only ceiling to compare the GNN against\.

Table 1:Hits@10 on three test\-set pools \(pre\-2023 train, 2024 test\) under 1:100 ranking, where random performance is Hits@10≈0\.10\\approx 0\.10\.Bold: best per column\. Content baselines fail cross\-DAAC; structural baselines fail cold\-start; the heterogeneous GNN is the first method competitive on all three pools, reaching4\.7×4\.7\{\\times\}random on the full pool and5\.2×5\.2\{\\times\}random on cold\-start\. The two scorer heads form a Pareto pair: dot for cold\-start \(0\.519\), MLP for cross\-DAAC \(0\.266\)\. All publication\-derived edges are restricted to year≤2022\\leq 2022\.
#### Results\.

[Table1](https://arxiv.org/html/2607.01584#S3.T1)reports Hits@10 across the three pools, and the cross\-pool pattern is sharp\.Popularityis essentially a prior on dataset frequency and accordingly performs near floor onall\(0\.140\) andcold\-start\(0\.064\)\. The two co\-usage\-graph heuristics,Common Neighbors\(0\.349\) andAdamic–Adar\(0\.355\), more than double Popularity onall, but their reliance on observed neighbors is fatal oncold\-start\(0\.211, 0\.225\), where at least one endpoint has no training neighborhood at all\.MF\-SVDshows the opposite asymmetry: its 128\-dim factorization happens to generalize oncold\-start\(0\.274\) but loses oncross\-DAAC\(0\.125\), where bipartite co\-citation structure does not constrain inter\-archive pairs\. Content baselines invert this picture again:SPECTER2andBGE\-basedominatecold\-start\(0\.477 and 0\.484\), confirming that abstract semantics is the strongest single signal when no structural cue is available, but they collapse oncross\-DAAC\(0\.127, 0\.103\)\. The intuition is that abstracts of cross\-archive datasets share substantial Earth\-science boilerplate, so cosine similarity ceases to discriminate plausible joint analyses from coincidental textual overlap\.

The two GNN variants combine these signals\.GNN\-Homomatches the content baselines oncold\-start\(0\.473\) and exceeds them onall\(0\.448\), but gains little oncross\-DAAC\(0\.146\)\.GNN\-Hetero, which carries type\-specific aggregators across the metadata relations, is the first method competitive on all three pools simultaneously: it improvescold\-startto 0\.519 and liftscross\-DAACto 0\.175\. The two scorer heads form a clear Pareto pair: the dot\-product head peaks oncold\-start\(0\.519\) while the MLP head peaks oncross\-DAAC\(0\.266\)—a substantial0\.0910\.091absolute gain over its dot\-product counterpart, attributable to the MLP’s ability to learn nonlinear combinations of the two embeddings that compensate for the type\-confounded similarity between cross\-archive datasets\. We deploy the dot\-product head as the retrieval feed to Agent 1 \(selected by validation MRR over theallpool\) and release both embedding sets with the code; the per\-pool, per\-metric breakdown for all configurations and the 11\-lever ablation that selected this configuration are in[SectionB\.2](https://arxiv.org/html/2607.01584#A2.SS2)and[SectionB\.3](https://arxiv.org/html/2607.01584#A2.SS3)\.

## 4Three\-Agent LLM Pipeline

The 200 predicted\-novel pairs surfaced by the GNN \([Section3](https://arxiv.org/html/2607.01584#S3)\) are the input to a three\-agent LLM cascade\. Each agent has a distinct role: scoring, generation, and judgment\. Each agent is independently instantiated with one of two backbones \(GPT\-5\.2 or Claude Sonnet 4\.6\)\. All three agents return strictly JSON\-formatted output with a fixed schema, queried at temperature0to remove sampling variance from the experiment\. The full system prompts, user templates, and JSON schemas for all four roles \(Agent 1, Agent 2, Agent 3 blind, Agent 3 contextual\) are reproduced verbatim in[AppendixC](https://arxiv.org/html/2607.01584#A3)\.

#### Agent 1 — pair\-level filter\.

Agent 1 receives a candidate pair\(di,dj\)\(d\_\{i\},d\_\{j\}\)together with the two datasets’ short names, long names, and abstracts \(truncated at 1,200 characters\), and returns two integer scores on a11–55scale:*plausibility*\(could a research team reasonably combine these datasets in a single study?\) and*novelty*\(how non\-obvious is the combination?\), plus a two\-to\-three sentence rationale\. Scoring is deliberately decoupled from generation: Agent 1 sees no hypothesis text, so its rating reflects the pair itself, not how well a generator happens to motivate it\. To anchor the scoring distribution and provide a positive control, Agent 1 is run not only on the 200 predicted\-novel pairs \(stratum A\) but also on three additional strata of equal size containing held\-out real co\-usages sampled from 2024 publications, giving 800 scored pairs per Agent 1 backbone\. Because the held\-out pool is what the GNN was*not*trained on, comparing the plausibility distribution of stratum A against strata B–D is a direct test of whether predicted\-novel pairs are scientifically coherent at the level of literature co\-usages \([Section6](https://arxiv.org/html/2607.01584#S6)\)\. For each Agent 1 backbone we select 40 pairs to forward to the generator, drawn exclusively from stratum A\. We use a tiered rule rather than ranking by a single composite score, because plausibility and novelty answer different questions and naive scalarization would let either dimension dominate\. Tier 1 takes pairs with maximal plausibility \(=5\{=\}5\) and at least moderate novelty \(≥3\\geq 3\); tier 2 takes pairs with high plausibility \(=4=4\) and high novelty \(≥4\\geq 4\); tier 3 takes pairs with maximal plausibility \(=5=5\) and modest novelty \(=2=2\), which capture obvious\-yet\-strong pairings as a sanity floor\. Within each tier, pairs are ordered by a weighted composite score, giving 55% weight to plausibility and 45% to novelty, and the top 40 are taken in order Tier 1→\\toTier 2→\\toTier 3, with a fallback to allplaus≥4\\text\{plaus\}\\geq 4pairs ranked by the same score if the three tiers yield fewer than 40\.

#### Agent 2 — hypothesis generator\.

Agent 2 receives one selected pair\(di,dj\)\(d\_\{i\},d\_\{j\}\)together with their short names, long names, and abstracts, and returns a structured research hypothesis with six fields: a one\-sentence*research question*, a one\-to\-two sentence testable*hypothesis*, an*analysis method*describing how the two datasets jointly test it, an*expected finding*that would constitute support, a one\-to\-two sentence statement of*scientific importance*, and a one\-to\-three word*domain*tag\. Critically, Agent 2 does not see Agent 1’s plausibility or novelty scores; this prevents the generator from regressing toward its own filter’s preferences and keeps generation a reflection of what the backbone independently considers worth proposing\.

#### Agent 3 — judge under blind and contextual conditions\.

Agent 3 evaluates each generated hypothesis on three integer scales:*importance*\(does it meaningfully advance the field?\),*tractability*\(is it ready to execute with current methods?\), and*novelty*\(does it open a new line of inquiry?\)\. Each hypothesis is judged twice—once*blind*\(Agent 3 sees only the six hypothesis fields, with no information about which datasets underlie them\) and once*contextual*\(Agent 3 sees the hypothesis fields plus the underlying pair’s short names, long names, and abstracts\)\. The blind/contextual contrast isolates how much of the judgment depends on the hypothesis text alone versus on inspectable evidence about the datasets\. We will show in[Section6](https://arxiv.org/html/2607.01584#S6)that this contrast is the single largest source of variance in tractability scores, dwarfing both filter and generator effects\.

## 5Factorial Experiment

To diagnose how much each agent contributes to the quality scores the pipeline reports, we run the three\-agent cascade \([Section4](https://arxiv.org/html/2607.01584#S4)\) under all eight assignments of backbone to role in a2×2×22\{\\times\}2\{\\times\}2factorial design\. The three factors are the identities of the filter \(Agent 1\), the generator \(Agent 2\), and the judge \(Agent 3\); each is independently set to either GPT\-5\.2 or Claude Sonnet 4\.6\. A fourth nested factor, judging*condition*, varies whether the judge sees the underlying dataset metadata \(contextual\) or only the hypothesis text \(blind\)\. All other components of the pipeline including the GNN ranker, the four\-stratum input pool fed to Agent 1, the tiered top\-40 selection, prompt templates, JSON schemas, and the temperature\-0decoding are held constant across cells, so any observed differences are attributable to backbone identity or to the blind/contextual contrast\.

#### Cells and counts\.

The factorial yields four hypothesis sets indexed by\(a1,a2\)\(a\_\{1\},a\_\{2\}\), each containing 40 hypotheses, for a total of 160 distinct hypotheses\. Each hypothesis is then scored by both judge backbonesa3∈\{GPT,Claude\}a\_\{3\}\\in\\\{\\text\{GPT\},\\text\{Claude\}\\\}under both judging conditions, producing160×4=640160\\times 4=640validator judgments\. Each judgment returns three integer scores on a11–55scale \(importance, tractability, novelty\) plus a free\-text rationale\. Agent 1 itself is also a factorial cell with 800 inputs \(4 strata of 200 pairs\), giving2×800=1,6002\\times 800=1\{,\}600Agent 1 ratings used both to feed the top\-40 selection and to characterize how predicted\- novel pairs \(stratum A\) compare against held\-out real co\-usages \(stratum B\) and two control strata \([Section6](https://arxiv.org/html/2607.01584#S6)\)\.

#### Why factorial\.

The dominant practice in LLM\-as\-scientist evaluations is to fix one backbone in each role and report the resulting scores as if they were properties of the hypotheses being judged\. The factorial design instead lets us*decompose*score variance across the four nominal factors \(filter, generator, judge, condition\) and their two\-way interactions using a standard one\-way ANOVA per axis, reporting effect sizes asη2\\eta^\{2\}\. This decomposition asks a direct question: when an importance score moves from 3 to 4, how much of that movement is signal about the hypothesis, and how much is the contingent identity of the backbones playing each role? We will show that for some axes the answer is uncomfortable\.

## 6Results

#### Stage 4: predicted\-novel pairs are nearly as plausible as held\-out real co\-usages\.

[Table2](https://arxiv.org/html/2607.01584#S6.T2)reports the Agent\-1 stratum\-level results and[Figure2](https://arxiv.org/html/2607.01584#S6.F2)visualizes the\(plausibility,novelty\)\(\\text\{plausibility\},\\text\{novelty\}\)joint distribution for stratum A under each judge\. Both judges order the four strata identically: held\-out real 2024 co\-usages \(B\) on top, GNN\-predicted novel pairs \(A\) a hair below, then same\-DAAC random “hard negatives” \(D\), and fully random pairs \(C\) at the bottom\. Critically, the gap between A and B on plausibility is tiny—0\.19 for GPT \(4\.38 vs\. 4\.57\) and 0\.29 for Claude \(4\.34 vs\. 4\.63\)— while the gap between A and the random control C is more than five times larger \(≥\\geq1\.07 for GPT,≥\\geq1\.59 for Claude\)\. The GNN therefore surfaces predicted\-novel pairs that LLM judges find essentially indistinguishable in scientific plausibility from pairs that were actually co\-used in 2024 publications\. The novelty axis, by contrast, fails to separate A from B: judges rate the predicted\-novel pairs as no more novel than held\-out real co\-usages, a pattern we return to in[Section7](https://arxiv.org/html/2607.01584#S7)as a caution about LLM\-as\-novelty\-judge\. The joint distribution in[Figure2](https://arxiv.org/html/2607.01584#S6.F2)additionally exposes a calibration shift between judges: 49 of 200 stratum\-A pairs land in GPT’s hero zone \(plausibility≥4\\geq 4and novelty≥3\\geq 3\), versus only 18 in Claude’s, reflecting a tighter Claude novelty distribution \([Table2](https://arxiv.org/html/2607.01584#S6.T2)\) rather than a difference in plausibility ranking\. This calibration mismatch is the practical reason we run the tiered top\-40 selection independently per Agent\-1 backbone rather than pooling\.

Table 2:Stage\-4 pair\-level judgments\. Each of 800 pairs \(200 per stratum\) was scored by both backbones on plausibility and novelty \(11–55\)\.B= held\-out real co\-usages from 2024 publications;A= GNN\-predicted novel pairs;D= same\-DAAC random “hard negatives”;C= fully random pairs\. Both judges order the strata identically \(B≥\\\!\\geq\\\!A\>\>D\>\>C on plausibility\); the A–B gap is small \(≤0\.29\\leq 0\.29\) while the A–C gap is more than five times larger\.![Refer to caption](https://arxiv.org/html/2607.01584v1/x2.png)Figure 2:Per\-judge\(plausibility,novelty\)\(\\text\{plausibility\},\\text\{novelty\}\)score grid for stratum A \(GNN\-predicted novel pairs\)\. Cells report counts out of 200; rows are plausibility \(55at top\) and columns are novelty \(11at left\)\. The dashed red rectangle delimits the*hero zone*\(plausibility≥4\\geq 4and novelty≥3\\geq 3\) targeted by the tiered top\-40 selection of[Section4](https://arxiv.org/html/2607.01584#S4)\. GPT\-5\.2 populates the hero zone more densely than Claude Sonnet 4\.6 \(n=49n=49vs\.n=18n=18\), reflecting Claude’s tighter novelty distribution rather than a difference in plausibility ranking; both judges concentrate stratum\-A pairs in the high\-plausibility rows \(≥4\\geq 4\)\.
#### Stage 5: variance decomposition\.

[Figure3](https://arxiv.org/html/2607.01584#S6.F3)reports theη2\\eta^\{2\}contribution of each factor to score variance, by axis\. Three patterns dominate\.\(i\) Importance is a judge\-identity question\.Judge identity explains25\.1%25\.1\\%of importance variance, while filter and generator identity together explain0\.34%0\.34\\%\. In absolute terms the GPT judge rates importance roughly0\.60\.6points higher than the Claude judge in nearly every cell, regardless of who filtered or generated the hypothesis\.\(ii\) Tractability is a context question\.The blind/contextual contrast explains15\.8%15\.8\\%of tractability variance versus4\.6%4\.6\\%for judge identity; showing the dataset metadata reliably lowers tractability scores by about half a point \(blind mean3\.943\.94, contextual mean3\.363\.36, averaged over the eight\(a1,a2,a3\)\(a\_\{1\},a\_\{2\},a\_\{3\}\)cells\), suggesting that hypothesis text alone reads as more executable than it does once the actual measurement products are inspectable\.\(iii\) Novelty is judge\-dominated but small\.Judge identity is again the largest single factor \(6\.0%6\.0\\%\), but no factor explains more than∼\\sim10% of variance on novelty—consistent with the Stage\-4 finding that LLM judges do not strongly discriminate novelty across pair types\. Filter and generator identity contribute essentially zero variance on every axis \(η2<0\.013\\eta^\{2\}<0\.013in all six cells\), which has a useful corollary: Agent 1 and Agent 2 affect*which*hypotheses get scored, but their backbone identity does not bias the scores those hypotheses receive\.

![Refer to caption](https://arxiv.org/html/2607.01584v1/x3.png)Figure 3:Variance decomposition of validator scores by factor, per axis \(η2\\eta^\{2\}from one\-way ANOVA,N=640N=640judgments\)\. Judge identity dominates importance variance; the blind/contextual contrast dominates tractability variance; no factor explains more than∼\\sim10% of novelty variance\. Filter and generator identity are negligible on every axis\.
#### Inter\-rater agreement: judges line up on tractability, diverge on importance\.

Inter\-rater agreement between GPT\-5\.2 and Claude Sonnet 4\.6 on identical hypotheses \(full table in[AppendixD](https://arxiv.org/html/2607.01584#A4)\) is highest on tractability \(Pearsonr=0\.63r=0\.63blind, quadratic\-weightedκq=0\.47\\kappa\_\{q\}=0\.47\), moderate on novelty \(r=0\.49r=0\.49,κq=0\.36\\kappa\_\{q\}=0\.36\), and weakest on importance \(r=0\.35r=0\.35,κq=0\.14\\kappa\_\{q\}=0\.14\)\. The ordering is the opposite of what one might expect—importance is arguably the most subjective of the three rubrics—but it is exactly whatWataokaet al\.\([2024](https://arxiv.org/html/2607.01584#bib.bib14)\)predict for an LLM\-as\-judge setup: importance is the axis most coupled to a backbone’s training\-time priors about what counts as “meaningful advance,” and consequently the axis on which two backbones with different pretraining mixtures disagree most\. Showing dataset context improves agreement on*all*three axes, with the largest gain on importance \(κq\\kappa\_\{q\}:0\.14→0\.240\.14\\to 0\.24, exact match:37%→63%37\\%\\to 63\\%\), supporting the hypothesis that anchoring judges to inspectable evidence narrows the prior\-driven disagreement\.

#### Hypothesis ranking is stable across configurations\.

The variance decomposition above is about the*absolute*scores, but for picking which hypotheses to actually pursue, what matters is the*ranking*—which hypothesis scores higher than which\. That ranking turns out to be much more stable than the absolute scores\. Across the four judge configurations \(each pairing of judge backbone and blind/contextual condition\), 159 of 160 hypotheses change by at most 2 points on every axis, with an average change of just11point\. So even though one judge gives systematically higher scores than the other, the order in which they rank the hypotheses is nearly the same\. A researcher who picks the top25%25\\%by average score will get almost the same set no matter which single judge is used\. The practical takeaway: single\-judge scores are not portable in absolute terms, but the rankings they produce are reliable\.

#### Flagship hypotheses\.

A qualitative inspection of the highest\-scoring outputs is deferred to[AppendixE](https://arxiv.org/html/2607.01584#A5), which lists five flagship hypotheses with the highest combined importance \+ tractability and a maximum cross\- judge disagreement of one on every axis\. The five span ecohydrology, glaciology, aerosol–cloud interactions, vegetation phenology, and methodological bias correction in cloud retrieval; all five are pinned to two named NASA products and are executable today with public archives—the deployment property the pipeline was designed to preserve\.

## 7Discussion and Limitations

#### Scope: ideation, not execution\.

Our three agents operate text\-only by design: filter and judge are evaluative roles where tool use buys little, and generator is an ideation role whose output is intended to seed human\-led investigation rather than autonomously execute it\. This scope choice has a second benefit specific to our methodology: the variance decomposition of[Section6](https://arxiv.org/html/2607.01584#S6)is interpretable precisely because every backbone receives identical inputs at temperature0with strict\- JSON decoding\. Adding tool access would inject retrieval and execution stochasticity that would confound the agent\-identity signal we set out to measure\. We view text\-only ideation and tool\-equipped execution as complementary layers of a future end\-to\-end discovery system, with our pipeline supplying the candidate hypotheses an executor would attempt\. The most important limitation of this work is that the 160 hypotheses we report are LLM\-judged, not human\-verified\. A small domain\-expert study—even on the five flagship hypotheses—would sharpen the claim that the pipeline produces actionable research seeds rather than merely well\-formed text\. Three further limitations deserve mention\. First, the GNN treats co\-usage as a binary signal and discards multiplicity; an edge\-weight\-aware variant might sharpen the ranker but is not the bottleneck identified by our ablation \([SectionB\.3](https://arxiv.org/html/2607.01584#A2.SS3)\)\. Second, abstracts are truncated at 1,200 characters before being shown to any agent, which can clip relevant detail for long\-abstract products and may under\-represent some pairings\. Third, our factorial covers two backbones, two judging conditions, and three rubric axes; it does not control for prompt phrasing, position bias within the user template, or temperature, all of which prior work has shown to move LLM\-judge outputs\(Zhenget al\.,[2023](https://arxiv.org/html/2607.01584#bib.bib12); Yeet al\.,[2025](https://arxiv.org/html/2607.01584#bib.bib13)\)\. Quantifying their interaction with the variance budget reported here is a natural next step\.

## 8Conclusion

Hypothesis generation in observational sciences is a different problem from generation in literature\-driven ones: the unit of discovery is a specific pair of measurement products, not a sentence\. We addressed this by combining a heterogeneous GNN ranker over the NASA Earth\-Observation Knowledge Graph with a three\-agent LLM cascade, producing 160 structured, dataset\-grounded research hypotheses spanning many Earth\-science domains\. Predicted\- novel pairs are rated essentially as plausible as held\-out real co\-usages, and a2×2×22\{\\times\}2\{\\times\}2factorial over agent identity shows that hypothesis rankings are stable across LLM backbones while absolute scores are not—an empirical bound on what single\-judge LLM\-as\-scientist evaluations can claim\. We release the trained GNN, the 160 hypotheses, the 640 validator judgments, and the full pipeline as a reproducible substrate for both the EO community and the broader study of LLM\-driven scientific discovery\.

## Impact Statement

This paper presents a system that suggests candidate research directions in Earth science; generated hypotheses are not scientific claims but seeds for human investigation, and each must be verified with domain experts and actual data analysis before being treated as a finding\. We see no societal consequences requiring specific discussion beyond standard concerns about the use of LLM outputs in scientific contexts\.

## LLM Usage Disclosure

GPT\-5\.2 and Claude Sonnet 4\.6 were used*as pipeline components*: they are the Agents 1, 2, and 3 described in[Section4](https://arxiv.org/html/2607.01584#S4), and their outputs are the experimental data reported in[Section6](https://arxiv.org/html/2607.01584#S6)\. In addition, Claude was used as a writing collaborator during drafting of this manuscript; all final wording and technical claims were verified by the authors\.

## References

- D\. Agarwal, B\. P\. Majumder, R\. Adamson, M\. Chakravorty, S\. R\. Gavireddy, A\. Parashar, H\. Surana, B\. Dalvi Mishra, A\. McCallum, A\. Sabharwal, and P\. Clark \(2025\)AutoDiscovery: open\-ended scientific discovery via bayesian surprise\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2507\.00310External Links:[Link](https://arxiv.org/abs/2507.00310)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Baek, S\. K\. Jauhar, S\. Cucerzan, and S\. J\. Hwang \(2025\)ResearchAgent: iterative research idea generation over scientific literature with large language models\.InProceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),Note:arXiv:2404\.07738External Links:[Link](https://arxiv.org/abs/2404.07738)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Cohan, S\. Feldman, I\. Beltagy, D\. Downey, and D\. S\. Weld \(2020\)SPECTER: document\-level representation learning using citation\-informed transformers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 2270–2282\.External Links:[Link](https://aclanthology.org/2020.acl-main.207/)Cited by:[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px3.p1.2)\.
- A\. Ghafarollahi and M\. J\. Buehler \(2025\)SciAgents: automating scientific discovery through bioinspired multi\-agent intelligent graph reasoning\.Advanced Materials\.Note:Article 2413523External Links:[Document](https://dx.doi.org/10.1002/adma.202413523)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Guo, J\. Lao, B\. Dang, Y\. Zhang, L\. Yu, L\. Ru, L\. Zhong, Z\. Huang, K\. Wu, D\. Hu, H\. He, J\. Wang, J\. Chen, M\. Yang, Y\. Zhang, and Y\. Li \(2024\)SkySense: a multi\-modal remote sensing foundation model towards universal interpretation for earth observation imagery\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Note:arXiv:2312\.10115External Links:[Link](https://arxiv.org/abs/2312.10115)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px4.p1.1)\.
- Z\. Kenton, N\. Y\. Siegel, J\. Kramár, J\. Brown\-Cohen, S\. Albanie, J\. Bulian, R\. Agarwal, D\. Lindner, Y\. Tang, N\. D\. Goodman, and R\. Shah \(2024\)On scalable oversight with weak LLMs judging strong LLMs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2407\.04622External Links:[Link](https://arxiv.org/abs/2407.04622)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px3.p1.3)\.
- C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha \(2024\)The AI scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.External Links:[Link](https://arxiv.org/abs/2408.06292)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1)\.
- NASA Goddard Earth Sciences Data and Information Services Center \(GES\-DISC\) \(2024\)NASA\-EO\-Knowledge\-Graph\.Hugging Face\.External Links:[Document](https://dx.doi.org/10.57967/hf/3463),[Link](https://huggingface.co/datasets/nasa-gesdisc/nasa-eo-knowledge-graph)Cited by:[Appendix A](https://arxiv.org/html/2607.01584#A1.p1.1),[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px1.p1.1)\.
- E\. Rolf, K\. Klemmer, C\. Robinson, and H\. Kerner \(2024\)Position: Mission critical—satellite data is a distinct modality in machine learning\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),pp\. 42691–42706\.External Links:[Link](https://proceedings.mlr.press/v235/rolf24a.html)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px4.p1.1)\.
- D\. Saeedi, D\. Buckner, J\. C\. Aponte, and A\. Aghazadeh \(2025\)AstroAgents: a multi\-agent AI for hypothesis generation from mass spectrometry data\.InICLR Workshop on Agentic AI for Science,Note:arXiv:2503\.23170External Links:[Link](https://arxiv.org/abs/2503.23170)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Si, D\. Yang, and T\. Hashimoto \(2025\)Can LLMs generate novel research ideas? A large\-scale human study with 100\+ NLP researchers\.InProceedings of the International Conference on Learning Representations \(ICLR\),Note:arXiv:2409\.04109External Links:[Link](https://arxiv.org/abs/2409.04109)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Singh, M\. D’Arcy, A\. Cohan, D\. Downey, and S\. Feldman \(2023\)SciRepEval: a multi\-format benchmark for scientific document representations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2211\.13308External Links:[Link](https://aclanthology.org/2023.emnlp-main.338/)Cited by:[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px3.p1.2),[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px6.p1.7)\.
- A\. Stewart, N\. Lehmann, I\. Corley, Y\. Wang, Y\. Chang, N\. Ait Ali Braham, S\. Sehgal, C\. Robinson, and A\. Banerjee \(2023\)SSL4EO\-L: datasets and foundation models for Landsat imagery\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2306\.09424External Links:[Link](https://arxiv.org/abs/2306.09424)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px4.p1.1)\.
- Q\. Wang, D\. Downey, H\. Ji, and T\. Hope \(2024a\)SciMON: scientific inspiration machines optimized for novelty\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\), Volume 1: Long Papers,pp\. 279–299\.External Links:[Link](https://aclanthology.org/2024.acl-long.18/)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, H\. Yang, and M\. Zhang \(2024b\)Neural common neighbor with completion for link prediction\.InProceedings of the International Conference on Learning Representations \(ICLR\),Note:arXiv:2302\.00890External Links:[Link](https://arxiv.org/abs/2302.00890)Cited by:[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px3.p2.5)\.
- K\. Wataoka, T\. Takahashi, and R\. Ri \(2024\)Self\-preference bias in LLM\-as\-a\-judge\.InNeurIPS Safe Generative AI Workshop,Note:arXiv:2410\.21819External Links:[Link](https://arxiv.org/abs/2410.21819)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px3.p1.3),[§6](https://arxiv.org/html/2607.01584#S6.SS0.SSS0.Px3.p1.9)\.
- K\. Wu, Y\. Zhang, L\. Ru, B\. Dang, J\. Lao, L\. Yu, J\. Luo, Z\. Zhu, Y\. Sun, J\. Zhang, Q\. Zhu, J\. Wang, M\. Yang, J\. Chen, Y\. Zhang, and Y\. Li \(2025\)A semantic\-enhanced multi\-modal remote sensing foundation model for earth observation\.Nature Machine Intelligence7,pp\. 1235–1249\.External Links:[Document](https://dx.doi.org/10.1038/s42256-025-01078-8)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Xiao, W\. Xuan, J\. Wang, J\. Huang, D\. Tao, S\. Lu, and N\. Yokoya \(2024a\)Foundation models for remote sensing and Earth observation: A survey\.IEEE Geoscience and Remote Sensing Magazine\.Note:arXiv:2410\.16602External Links:[Link](https://arxiv.org/abs/2410.16602)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. Nie \(2024b\)C\-pack: packed resources for general chinese embeddings\.InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval,pp\. 641–649\.Cited by:[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px6.p1.7)\.
- Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. Ha \(2025\)The AI scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.InICLR Workshop on “I Can’t Believe It’s Not Better: Challenges in Applied Deep Learning”,Note:arXiv:2504\.08066External Links:[Link](https://arxiv.org/abs/2504.08066)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Yang, R\. Bao, C\. Xiao, J\. Ma, P\. Bhatia, S\. Gao, and T\. Kass\-Hout \(2025a\)Any large language model can be a reliable judge: Debiasing with a reasoning\-based bias detector\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2505\.17100External Links:[Link](https://arxiv.org/abs/2505.17100)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px3.p1.3)\.
- Z\. Yang, W\. Liu, B\. Gao, Y\. Liu, W\. Li, T\. Xie, L\. Bing, W\. Ouyang, E\. Cambria, and D\. Zhou \(2025b\)MOOSE\-Chem2: exploring LLM limits in fine\-grained scientific hypothesis discovery via hierarchical search\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2505\.19209External Links:[Link](https://arxiv.org/abs/2505.19209)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Yang, W\. Liu, B\. Gao, T\. Xie, Y\. Li, W\. Ouyang, S\. Poria, E\. Cambria, and D\. Zhou \(2025c\)MOOSE\-Chem: large language models for rediscovering unseen chemistry scientific hypotheses\.InProceedings of the International Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.07076External Links:[Link](https://arxiv.org/abs/2410.07076)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Ye, Y\. Wang, Y\. Huang, D\. Chen, Q\. Zhang, N\. Moniz, T\. Gao, W\. Geyer, C\. Huang, P\. Chen,et al\.\(2025\)Justice or prejudice? quantifying biases in LLM\-as\-a\-judge\.InInternational Conference on Learning Representations \(ICLR\),Note:arxiv:2410\.02736External Links:[Link](https://arxiv.org/abs/2410.02736)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px3.p1.3),[§7](https://arxiv.org/html/2607.01584#S7.SS0.SSS0.Px1.p1.1)\.
- H\. Yu, Z\. Hong, Z\. Cheng, K\. Zhu, K\. Xuan, J\. Yao, T\. Feng, and J\. You \(2025\)ResearchTown: simulator of human research community\.InProceedings of the International Conference on Machine Learning \(ICML\),Note:arXiv:2412\.17767External Links:[Link](https://arxiv.org/abs/2412.17767)Cited by:[§1](https://arxiv.org/html/2607.01584#S1.p2.1),[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Zhang and Y\. Chen \(2018\)Link prediction based on graph neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:1802\.09691External Links:[Link](https://arxiv.org/abs/1802.09691)Cited by:[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px3.p2.5)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2306\.05685External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px3.p1.3),[§7](https://arxiv.org/html/2607.01584#S7.SS0.SSS0.Px1.p1.1)\.
- J\. Zhu, G\. Li, Y\. Yang, J\. Zhu, X\. Cui, and D\. Koutra \(2024\)On the impact of feature heterophily on link prediction with graph neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2409\.17475External Links:[Link](https://arxiv.org/abs/2409.17475)Cited by:[§3](https://arxiv.org/html/2607.01584#S3.SS0.SSS0.Px3.p2.16)\.
- X\. X\. Zhu, Z\. Xiong, Y\. Wang, A\. J\. Stewart, K\. Heidler, Y\. Wang, Z\. Yuan, T\. Dujardin, Q\. Xu, and Y\. Shi \(2025\)On the foundations of Earth foundation models\.Communications Earth & Environment\.External Links:[Document](https://dx.doi.org/10.1038/s43247-025-03127-x)Cited by:[§2](https://arxiv.org/html/2607.01584#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AKnowledge Graph Characterization

This appendix reports the full statistics of the NASA EO Knowledge Graph\(NASA Goddard Earth Sciences Data and Information Services Center \(GES\-DISC\),[2024](https://arxiv.org/html/2607.01584#bib.bib29)\)summarized in[Section3](https://arxiv.org/html/2607.01584#S3)\.[Table3](https://arxiv.org/html/2607.01584#A1.T3)lists global graph properties,[Table4](https://arxiv.org/html/2607.01584#A1.T4)the seven node types,[Table5](https://arxiv.org/html/2607.01584#A1.T5)the nine relation types, and[Table6](https://arxiv.org/html/2607.01584#A1.T6)the dataset distribution across NASA’s Distributed Active Archive Centers, and[Table7](https://arxiv.org/html/2607.01584#A1.T7)the derived co\-usage statistics and temporal split sizes\.[Figure4](https://arxiv.org/html/2607.01584#A1.F4)visualizes node\- and edge\-type distributions;[Figure6](https://arxiv.org/html/2607.01584#A1.F6)reports the citation and co\-usage degree distributions;[Figure5](https://arxiv.org/html/2607.01584#A1.F5)shows the temporal split over publication year used for train/val/test\.

Table 3:Summary statistics of the NASA Earth Observation Knowledge Graph \(EO\-KG\), computed directly from the released GraphML\.Table 4:Node type distribution in the NASA EO\-KG with key attributes\.Table 5:Edge \(relation\) types in the NASA EO\-KG with dominant source and target node types\.Table 6:Distribution of NASA EO\-KG datasets across Distributed Active Archive Centers \(DAACs\)\.#### Why heavy\-tails matter for Stage 2\.

Panels \(a\)–\(d\) of[Figure6](https://arxiv.org/html/2607.01584#A1.F6)all exhibit heavy tails spanning two to three orders of magnitude\. This imbalance is the practical reason we sample training negatives fromPr\(n\)∝deg\(n\)0\.75\\Pr\(n\)\\propto\\deg\(n\)^\{0\.75\}rather than uniformly \([Section3](https://arxiv.org/html/2607.01584#S3)\): uniform negatives would rarely include hub datasets, yet hubs are exactly where the ranking problem is hardest because they have many plausible partners\. The edge\-weight panel \(c\) additionally motivates our*binary*treatment of co\-usage: weighting positives by the number of co\-citing papers would upweight already\-well\-known pairings at the expense of the rarer, less well\-traveled pairs that carry the most discovery value\.

#### Why a temporal split\.

A random held\-out split would let the model see co\-usages it will be evaluated on through common\-neighbor paths \(e\.g\., if pair\(A,B\)\(A,B\)is held out but papers citing\{A,B,C\}\\\{A,B,C\\\}remain in train, the training signal leaks throughCC\)\. The temporal split used here mitigates this leakage by partitioning pairs by the year of each observing paper, and the strictly held\-out evaluation subset is the 3,944 test pairs that never appear in any training\-era paper\. This mirrors the deployment scenario the system is designed for, where the model must rank candidate pairings that the research community has not yet explored\.

Table 7:Co\-usage statistics for the NASA EO\-KG\. Co\-usage edges are dataset pairs cited together in at least one publication\. The train/val/test split is temporal \(papers≤2022\\leq 2022/20232023/20242024\); pairs cited across multiple years can appear in more than one split, so split sizes do not partition the unique\-pair total\.*Test unseen in train*is the subset of test pairs never seen in any train\-era paper\.![Refer to caption](https://arxiv.org/html/2607.01584v1/x4.png)Figure 4:Node\-type and edge\-type distributions in the NASA EO\-KG\. Publications dominate the node count; citations and research\-area edges dominate the edge count\.![Refer to caption](https://arxiv.org/html/2607.01584v1/x5.png)Figure 5:Temporal split of the NASA EO\-KG by publication year, colored by assigned split \(blue: train,≤2022\\leq 2022; gold: val,20232023; orange: test,20242024; grey:20252025, dropped for incompleteness\)\.\(a\)Number ofUSES\_DATASET\(Paper→\\toDataset\) edges emitted per year; this is the raw scaffolding from which co\-usage is derived\.\(b\)Total co\-usage weight per year, i\.e\. the number of unordered dataset pairs contributed by each year’s papers \(a paper citingkkdatasets contributes\(k2\)\\binom\{k\}\{2\}to its year\)\. Each co\-usage pair is assigned to every split corresponding to a year in which a paper co\-citing both datasets appeared; pairs cited across multiple years thus enter more than one split\. The split sizes are 13,529 training, 6,119 validation, and 6,319 test pairs, with 3,944 test pairs unseen in train serving as the strictly held\-out evaluation subset\.![Refer to caption](https://arxiv.org/html/2607.01584v1/x6.png)Figure 6:Usage and co\-usage distributions in the NASA EO\-KG, all on log–log axes\.\(a\)Number of papers citing a given dataset \(median 3, max1,0511,051\)\.\(b\)Number of datasets cited by a given paper \(median 2, max 50\)\.\(c\)Co\-usage*edge weight*: for each unordered pair of datasets that have ever been jointly cited, the number of distinct papers that cite both \(max pair weight1,0511,051;23,49023,490unique pairs sum to total weight64,93064,930\)\.\(d\)Co\-usage*graph degree*: for each dataset, the number of distinct other datasets it has ever been co\-cited with \(median 11, max 314\)\. All four distributions are heavy\-tailed, a small number of flagship datasets \(MODIS, MERRA\-2, SMAP variants\) and flagship pairings dominate the graph\.

## Appendix BGNN Details, Full Baselines, and Ablation

### B\.1Hyperparameters

Table[8](https://arxiv.org/html/2607.01584#A2.T8)summarizes the full set of GNN hyperparameters\. They were not tuned per pool; the same configuration is used for all three test pools\.

Table 8:GNN training hyperparameters for Stage 2 \(shared across GNN\-Homo and GNN\-Hetero\)\.
### B\.2Full baseline metrics

[Section3](https://arxiv.org/html/2607.01584#S3)reports Hits@10 for compactness\.[Table9](https://arxiv.org/html/2607.01584#A2.T9)reports all five metrics \(Hits@10, Hits@50, MRR, AUC, AP\) for the six Stage\-1 baselines on each of the three test pools;[Table10](https://arxiv.org/html/2607.01584#A2.T10)reports the same for GNN\-Homo and GNN\-Hetero with mean±\\pmstd over three seeds\.

Table 9:Stage 1 baselines on the co\-usage link\-prediction task \(1:100 negative sampling\)\. Higher is better for all metrics\.Table 10:Stage 2 GNN results on the co\-usage link\-prediction task \(1:100 negative sampling, same pools as Table[9](https://arxiv.org/html/2607.01584#A2.T9)\)\. Mean±\\pmstd over 3 seeds\.
### B\.3Cumulative ablation

[Table11](https://arxiv.org/html/2607.01584#A2.T11)reports our 11\-lever cumulative ablation over both the homogeneous and heterogeneous GNN\. Each row adds one lever on top of all previously*kept*levers; a lever is kept \(✓\\checkmark\) if its addition improves mean validation MRR by more than10−310^\{\-3\}over the current best, with the sole exception that the first lever in each block \(\+pubs\_usesfor hetero,\+3\_layersfor homo\) is retained as the chain’s starting point because the imported Stage\-2 baseline carries no validation\-MRR record to compare against, so its checkmark does not indicate an improvement\. Three observations follow\. First, the heterogeneous baseline already dominates the homogeneous one on all three test pools, isolating the value of type\-specific message passing\. Second, after restricting publication nodes to year≤2022\\leq 2022to prevent test\-co\-usage leakage through 2\-hop message\-passing paths, adding publication\-derived edges \(\+pubs\_uses,\+pubs\_area\) yields no statistically meaningful improvement over the base heterogeneous architecture:\+pubs\_area’s validation\-MRR delta of\+0\.0009\+0\.0009falls below our keep threshold \(\+10−3\+10^\{\-3\}\), and Hits@10 on cold\-start and cross\-DAAC are slightly*lower*with publications than without\. Third, the dot\-product and MLP scorer heads form a Pareto pair on cold\-start versus cross\-DAAC performance rather than a strict ordering: dot wins cold\-start \(0\.519\) by a substantial margin, MLP wins cross\-DAAC \(0\.266\) by an even larger one, and the two are within0\.0010\.001of each other on theallpool\. We therefore release both embedding sets with the code so that downstream applications can pick the head that matches their target distribution\.

Table 11:Greedy cumulative ablation on GNN\-Homo and GNN\-Hetero\. Starting from the Stage\-2 baseline, levers are tried in the listed order; each row is the configuration after adding one lever to the previously\-kept set, and a lever is marked kept \(✓\) if it improves mean validation MRR by\>10−3\>10^\{\-3\}over the current best\. Because the imported Stage\-2 baseline carries no validation\-MRR record, the first lever in each block \(\+pubs\_usesfor hetero,\+3\_layersfor homo\) is retained as the chain’s starting point without a comparison and should not be read as an improvement—consistent with the text, publication edges do not help\. The checkmarks trace this greedy search, not a single deployed stack:\+mlp\_scorerswaps the dot head for an MLP head and\+ensembleis a post\-hoc blend, so several checked levers are mutually exclusive design choices\. The retrieval feed to Agent 1 is the base heterogeneous GNN with the dot\-product head and binary cross\-entropy loss \([Section3](https://arxiv.org/html/2607.01584#S3)\), with the MLP head released as its Pareto complement\. Hits@10, mean±\\pmstd over 3 seeds\.

## Appendix CThree\-Agent Prompts and JSON Schemas

The four colored boxes below reproduce verbatim the system prompts, user\-message templates, and JSON output schemas used in the three\-agent pipeline of[Section4](https://arxiv.org/html/2607.01584#S4)\. The same prompts are used regardless of which backbone \(GPT\-5\.2 or Claude Sonnet 4\.6\) is playing the role, so that any difference in the resulting scores reflects the backbone, not the prompt\. All four call sites use temperature0and the API’s strict\-JSON response format \(GPT:response\_format=json\_object; Claude: defensive post\-hoc fence stripping\)\.

Agent 1 — Pair\-Level FilterBackbone:GPT\-5\.2*or*Claude Sonnet 4\.6\.Temperature:0\.Max output tokens:400\.System promptYou are an expert Earth\-science research scientist\. Your job is to evaluate whether two Earth\-observation datasets could plausibly be combined in a single scientific study\. Base your judgment on the datasets’ scientific content as described in their abstracts, not on whether you have seen them combined before in published work\. Respond with JSON only, in the exact schema requested\.User\-message templatePlease evaluate the following pair of Earth\-observation datasets\. Dataset A: Short name: \{sn\_a\} Long name: \{ln\_a\} Abstract: \{abs\_a\} Dataset B: Short name: \{sn\_b\} Long name: \{ln\_b\} Abstract: \{abs\_b\} Rate this pairing on two axes: 1\. SCIENTIFIC PLAUSIBILITY \(1\-\-5\): Could a research team reasonably combine these datasets in a single scientific study? 2\. NOVELTY \(1\-\-5\): How novel or non\-obvious is this combination?Output JSON schema\{ "plausibility": <1\-\-5 int\>, "novelty": <1\-\-5 int\>, "rationale": "<2\-\-3 sentence explanation\>" \}Inputs:shortName,longName, and abstract \(truncated to 1,200 characters\) for each dataset\.Output:two integer scores plus a free\-text rationale\.

Agent 2 — Hypothesis GeneratorBackbone:GPT\-5\.2*or*Claude Sonnet 4\.6\.Temperature:0\.Max output tokens:900\.System promptYou are an expert Earth\-observation research scientist\. You will be given two NASA datasets\. Generate ONE concrete, publishable research hypothesis that could be tested by combining them\. Focus on scientific content, not data\-availability caveats\. Respond with JSON only\.User\-message templateDataset A: Short name: \{sn\_a\} Long name: \{ln\_a\} Abstract: \{abs\_a\} Dataset B: Short name: \{sn\_b\} Long name: \{ln\_b\} Abstract: \{abs\_b\} Generate one specific, testable research hypothesis that combines these two datasets\. The hypothesis must be concrete enough that a team could plan an actual study, not a vague ‘‘one could investigate X’’ statement\.Output JSON schema\{ "research\_question": "<one\-sentence question\>", "hypothesis": "<testable hypothesis, 1\-\-2 sent\.\>", "analysis\_method": "<how datasets test it, 1\-\-2 sent\.\>", "expected\_finding": "<what would support, 1 sent\.\>", "scientific\_importance": "<why it matters, 1\-\-2 sent\.\>", "domain": "<primary domain, 1\-\-3 words\>" \}Inputs:the surviving pair\(di,dj\)\(d\_\{i\},d\_\{j\}\)from the tiered top\-40 selection, with the same dataset metadata Agent 1 received\. Crucially, Agent 2 does*not*see Agent 1’s scores\.Output:a six\-field structured hypothesis\.

Agent 3 — Judge \(Blind condition\)Backbone:GPT\-5\.2*or*Claude Sonnet 4\.6\.Temperature:0\.Max output tokens:500\.System promptYou are a senior Earth\-observation research scientist reviewing a proposed research hypothesis\. Judge the hypothesis on its scientific merit alone; you do not need to verify data availability\. Respond with JSON only\.User\-message templateHere is a proposed Earth\-science research hypothesis: Research question: \{research\_question\} Hypothesis: \{hypothesis\} Proposed method: \{analysis\_method\} Expected finding: \{expected\_finding\} Scientific importance: \{scientific\_importance\} Domain: \{domain\} Rate 1\-\-5: IMPORTANCE: 1 = trivial, 5 = meaningfully advances field TRACTABILITY: 1 = needs breakthroughs, 5 = ready to execute NOVELTY: 1 = already well\-studied, 5 = opens new lineOutput JSON schema\{ "importance": <1\-\-5\>, "tractability": <1\-\-5\>, "novelty": <1\-\-5\>, "rationale": "<2\-\-3 sentences\>" \}Inputs:only the six hypothesis fields produced by Agent 2; no dataset metadata\.Output:three integer scores plus rationale\.

Agent 3 — Judge \(Contextual condition\)Backbone:GPT\-5\.2*or*Claude Sonnet 4\.6\.Temperature:0\.Max output tokens:500\.System promptYou are a senior Earth\-observation research scientist reviewing a proposed research hypothesis along with the datasets it uses\. Judge the hypothesis on scientific merit and the appropriateness of the dataset combination\. Respond with JSON only\.User\-message templateDataset A: Short name: \{sn\_a\} Long name: \{ln\_a\} Abstract: \{abs\_a\} Dataset B: Short name: \{sn\_b\} Long name: \{ln\_b\} Abstract: \{abs\_b\} Proposed research hypothesis: Research question: \{research\_question\} Hypothesis: \{hypothesis\} Proposed method: \{analysis\_method\} Expected finding: \{expected\_finding\} Scientific importance: \{scientific\_importance\} Domain: \{domain\} Rate 1\-\-5 \(same scales as the blind condition\)\.Output JSON schema\{ "importance": <1\-\-5\>, "tractability": <1\-\-5\>, "novelty": <1\-\-5\>, "rationale": "<2\-\-3 sentences\>" \}Inputs:the same six hypothesis fields as the blind judge,*plus*the underlying pair’sshortName,longName, and abstracts \(truncated to 1,200 characters\)\.Output:three integer scores plus rationale\.

## Appendix DFactorial Design: Extended Analyses

This appendix presents three extended views of the Stage\-5 factorial results summarized in[Section6](https://arxiv.org/html/2607.01584#S6)\.

#### Inter\-rater agreement \(Table[12](https://arxiv.org/html/2607.01584#A4.T12)\)\.

[Table12](https://arxiv.org/html/2607.01584#A4.T12)reports the full inter\-rater statistics referenced in[Section6](https://arxiv.org/html/2607.01584#S6): Pearsonrr, Spearmanρ\\rho, Cohen’s quadratic\-weightedκq\\kappa\_\{q\}, and the exact\-match percentage between GPT\-5\.2 and Claude Sonnet 4\.6 on identical hypotheses, separated by judging condition\. The headline ordering holds for every metric \(tractability\>\>novelty\>\>importance under both conditions\), and the contextual\-condition gain on importance \(κq\\kappa\_\{q\}:0\.14→0\.240\.14\\to 0\.24, exact match:37%→63%37\\%\\to 63\\%\) is the largest single improvement in the table\.

Table 12:Inter\-rater agreement between GPT\-5\.2 and Claude Sonnet 4\.6 on identical hypotheses\. Judges agree best on tractability, less on novelty, and only weakly on importance\. Showing dataset context improves agreement on all three axes; the largest gain is on importance \(κq\\kappa\_\{q\}:0\.14→0\.240\.14\\to 0\.24, exact match:37%→63%37\\%\\to 63\\%\)\.κq\\kappa\_\{q\}is Cohen’s quadratic\-weighted kappa\.
#### Per\-cell mean scores \(Figure[7](https://arxiv.org/html/2607.01584#A4.F7)\)\.

Three heatmaps report the mean validator score for every cell of the2×2×22\{\\times\}2\{\\times\}2factorial, with the blind/contextual condition appearing as adjacent column\-pairs\. Rows are\(a1,a2\)\(a\_\{1\},a\_\{2\}\)filter–generator combinations; columns are\(a3,condition\)\(a\_\{3\},\\text\{condition\}\)\. The visual signature of the variance decomposition \([Figure3](https://arxiv.org/html/2607.01584#S6.F3)\) is immediately visible:importance\(left panel\) splits cleanly by judge— the two GPT\-judge columns are uniformly darker than the two Claude\- judge columns regardless of\(a1,a2\)\(a\_\{1\},a\_\{2\}\), with mean shifts of∼0\.5\\sim 0\.5–0\.70\.7points;tractability\(middle panel\) splits by condition—blind columns are uniformly darker than ctx columns within each judge;novelty\(right panel\) is mostly flat, consistent with its low total variance\. The white vertical divider separates GPT\-judge from Claude\-judge columns; the dashed horizontal divider separates GPT\-filter from Claude\-filter rows\.

![Refer to caption](https://arxiv.org/html/2607.01584v1/x7.png)Figure 7:Per\-cell mean validator score for each axis \(importance, tractability, novelty\)\. Rows:\(a1,a2\)\(a\_\{1\},a\_\{2\}\)filter–generator combinations\. Columns:\(a3,condition\)\(a\_\{3\},\\text\{condition\}\)judge–condition combinations\. Color scale uses the cividis colormap on the score range\[2\.0,5\.0\]\[2\.0,5\.0\]\. The vertical white divider separates the GPT and Claude judges; the horizontal dashed divider separates GPT and Claude filters\.
#### Inter\-rater scatter \(Figure[8](https://arxiv.org/html/2607.01584#A4.F8)\)\.

For every hypothesis judged under both backbones in the same condition, the scatter plots GPT’s score \(x\-axis\) against Claude’s score \(y\-axis\), with jitter added to separate ties\. The diagonal is perfect agreement; the dashed diagonals mark±1\\pm 1tolerance\. Each panel reports Pearsonrrand Spearmanρ\\rhoover its 160\-point sample\. The visualization makes the pattern from[Table12](https://arxiv.org/html/2607.01584#A4.T12)concrete: tractability scatter clusters tightly around the diagonal under both conditions; novelty scatter is moderately tight; importance scatter exhibits a visible vertical bias \(GPT systematically high\) that narrows but does not vanish in the contextual condition\.

![Refer to caption](https://arxiv.org/html/2607.01584v1/x8.png)Figure 8:Inter\-rater scatter: GPT\-5\.2 score \(x\) vs\. Claude Sonnet 4\.6 score \(y\) for each hypothesis under each judging condition\. Top row: blind condition\. Bottom row: contextual condition\. Columns: importance, tractability, novelty\. Solid line: equality; dashed lines:±1\\pm 1tolerance\. Pearsonrr, Spearmanρ\\rho, and sample size annotated per panel\. Tractability shows the highest agreement; importance shows the lowest, with a visible upward bias for GPT that is partially attenuated by adding dataset context\.

## Appendix EFlagship Hypotheses

This appendix lists the five flagship hypotheses referenced in[Section6](https://arxiv.org/html/2607.01584#S6)\. Selection criterion: the highest combined importance \+ tractability scores with a maximum cross\-judge disagreement of one on every axis, i\.e\., consensus high quality across all four judge conditions \(a3∈\{GPT,Claude\}×\{blind,contextual\}a\_\{3\}\\\!\\in\\\!\\\{\\text\{GPT\},\\text\{Claude\}\\\}\\\!\\times\\\!\\\{\\text\{blind\},\\text\{contextual\}\\\}\)\. The score columns*Imp*,*Tract*, and*Novel*report the mean across those four judgments;*MaxΔ\\Delta*is the largest disagreement on any axis\.

Table 13:Five flagship hypotheses from the 2×\\times2×\\times2 factorial\. Each was generated by the \(Agent 1, Agent 2\) combination listed, then rated by both judges under both conditions \(blind and contextual\); theImp,Tract, andNovelcolumns report the mean across these 4 validator judgments\.MaxΔ\\Deltais the largest disagreement on any axis among the 4 judgments; a low value indicates the hypothesis is rated consistently across agents and conditions\.#### Qualitative reading\.

[Table13](https://arxiv.org/html/2607.01584#A5.T13)surfaces three properties worth noting\. First, the flagship pairs mix same\-mission, multi\-sensor or multi\-level combinations \(e\.g\., IceBridge Ku\-band radar with MCoRDS, or Aqua\-MODIS EVI with Aqua\-MODIS LAI\) with cross\-mission, cross\-archive combinations whose measurements are nonetheless physically co\-located \(e\.g\., Terra\-MODIS aerosol with CALIPSO lidar, or SMAP soil moisture with Aqua\-MODIS vegetation indices\), suggesting that the ranker prefers pairings that are jointly co\-registerable and physically complementary rather than merely semantically similar\. Second, the flagship set spans five disjoint scientific domains \(ecohydrology, glaciology, aerosol–cloud interactions, vegetation phenology, and methodological bias correction in cloud retrieval\), indicating that the pipeline does not collapse onto a single theme even when optimizing only for importance and tractability\. Third, the*novelty*column for every flagship hypothesis sits in the2\.52\.5–3\.253\.25range, the boundary between “moderately novel” and “opens a new line of inquiry”—the rating\-surface region where the tiered top\-40 selection of[Section4](https://arxiv.org/html/2607.01584#S4)concentrates effort\. The flagship set is therefore a calibrated sample of what the pipeline considers ambitious\-but\-plausible, not a curated highlight reel\.

#### Full reproductions\.

[AppendixF](https://arxiv.org/html/2607.01584#A6)reproduces three hypotheses in their entirety as gray chatboxes spanning the score spectrum, including the flagship glaciology entry \(IRKUB1B×\\timesIRMCR1B\) in its complete six\-field form together with all four Agent 3 verdicts\. The released hypothesis\-corpus TSV files \(hypotheses\_\{a1\}\_\{a2\}\.tsv\) contain the unabridged JSON outputs for all 160 hypotheses in the factorial\.

## Appendix FQualitative Hypothesis Examples

To complement the aggregate statistics of[Section6](https://arxiv.org/html/2607.01584#S6)and the flagship table of[AppendixE](https://arxiv.org/html/2607.01584#A5), we reproduce three full hypotheses spanning the score spectrum: a top\-quartile case \(High, mean combined score11\.5011\.50\), a near\-median case \(Medium,10\.2510\.25\), and a bottom\-decile case \(Low,7\.257\.25\)\. Each box reports the dataset pair as fed to Agent 2, the structured six\-field hypothesis Agent 2 returned, and the four Agent 3 verdicts \(judge∈\{\\in\\\{GPT, Claude\}\\\}×\\timescondition∈\{\\in\\\{blind, contextual\}\\\}\)\. Selected judge rationales are quoted to make the cross\-judge calibration shift visible at the example level\.

Example 1 \(High\) — IRKUB1B×\\timesIRMCR1B \(glaciology\)Filter→\\toGenerator:GPT→\\toGPTSource stratum:A \(predicted novel\)\.Dataset pairIRKUB1B— IceBridge Ku\-Band Radar L1B Geolocated Radar Echo Strength Profiles V002 \(NSIDC\)\. IRMCR1B— IceBridge MCoRDS L1B Geolocated Radar Echo Strength Profiles V002 \(NSIDC\)\.Agent 2 — generated hypothesisResearch question\.Do along\-track changes in Ku\-band surface echo strength predict the presence and thickness of near\-surface firn aquifers and/or ice lenses detected by MCoRDS in Greenland’s percolation zone?Hypothesis\.Where MCoRDS echograms show a laterally continuous, high\-reflectivity near\-surface horizon \(ice lens\) or a strong diffuse\-scattering zone consistent with a firn aquifer, the Ku\-band radar altimeter exhibits a systematic increase in surface echo strength and a coincident shift in leading\-edge/peak\-power range \(apparent elevation bias\) relative to adjacent dry\-firn segments; the magnitude of the Ku\-band anomaly scales with the MCoRDS\-inferred thickness of the wet/ice\-rich layer\.Analysis method\.Co\-register IRKUB1B and IRMCR1B profiles along overlapping flight lines; classify segments into dry\-firn, ice\-lens, or firn\-aquifer/wet\-firn using MCoRDS reflectors; estimate layer depth from MCoRDS two\-way travel time; test class\-conditional differences in Ku\-band echo strength and waveform\-derived range, controlling for surface slope and roughness proxies\.Expected finding\.Repeatable Ku\-band echo\-strength increases and range/leading\-edge shifts over MCoRDS\-identified wet/ice\-rich layers, with anomaly magnitude positively correlated with MCoRDS\-inferred layer thickness\.Scientific importance\.Linking Ku\-band surface\-return behavior to subsurface hydrologic state enables bias correction of radar\-altimetry\-derived elevation change in melt\-affected regions and provides an altimetry\-only route to map firn aquifers and ice lenses, sharpening Greenland mass\-balance estimates\.Agent 3 verdicts \(Importance / Tractability / Novelty, 1–5\)Judge \(condition\)ImpTractNovGPT\-5\.2 \(blind\)543GPT\-5\.2 \(contextual\)544Claude 4\.6 \(blind\)443Claude 4\.6 \(contextual\)433Mean across judges4\.503\.753\.25Selected rationalesGPT \(blind\):“Establishing a quantitative link between Ku\-band waveform/echo\-strength anomalies and near\-surface wet/ice\- rich firn structures would directly improve elevation\-change interpretation and bias correction\.”Claude \(contextual\):“Quantifying how near\-surface wet/ice\-rich layers bias Ku\-band altimetry returns is directly relevant to correcting CryoSat\-2 and Sentinel\-6 elevation\-change records in Greenland’s percolation zone\.”

Example 2 \(Medium\) — OMGLER×\\timesOMUVBG \(UV radiation\)Filter→\\toGenerator:Claude→\\toGPTSource stratum:A \(predicted novel\)\.Dataset pairOMGLER— OMI/Aura Global Geometry\-Dependent Surface LER 1\-Orbit L2 Swath 13×\\times24 km V3 \(GES\-DISC\)\. OMUVBG— OMI/Aura Surface UVB Irradiance and Erythemal Dose Daily L2 Global Gridded0\.25∘0\.25^\{\\circ\}V3 \(GES\-DISC\)\.Agent 2 — generated hypothesisResearch question\.How much of the day\-to\-day variability in OMI\-derived surface UVB irradiance and erythemal dose can be explained by geometry\-dependent surface reflectance \(GLER\) variations observed by OMI?Hypothesis\.After controlling for solar zenith angle and cloud screening, higher OMGLER within a0\.25∘0\.25^\{\\circ\}grid cell is associated with systematically higher OMUVBG surface UVB irradiance and erythemal dose on the same day, with strongest sensitivity over snow/ice and arid regions; the GLER–UVB relationship is nonlinear, saturating at very high GLER due to multiple\-scattering effects\.Analysis method\.Collocate OMGLER swath pixels to OMUVBG0\.25∘0\.25^\{\\circ\}daily cells; compute per\-cell daily statistics of GLER alongside OMUVBG using high\-quality, low\-cloud candidates; fit a mixed\-effects or generalized additive model predicting OMUVBG from GLER while adjusting for solar zenith angle, viewing geometry, season, and latitude; test interaction terms for surface type proxies\.Expected finding\.A statistically significant positive GLER coefficient under comparable illumination and cloud conditions, with the largest effect sizes in high\-GLER environments\.Scientific importance\.Quantifying the sensitivity of surface UV exposure estimates to geometry\-dependent reflectance improves bias correction of satellite UV products over bright surfaces, with direct relevance to UV climatology and public\- health exposure assessment\.Agent 3 verdicts \(Importance / Tractability / Novelty, 1–5\)Judge \(condition\)ImpTractNovGPT\-5\.2 \(blind\)453GPT\-5\.2 \(contextual\)443Claude 4\.6 \(blind\)342Claude 4\.6 \(contextual\)342Mean across judges3\.504\.252\.50Selected rationalesGPT \(blind\):“The hypothesis targets a known but consequential error source in satellite UV products—surface reflectance/BRDF effects—so quantifying day\-to\-day GLER\-driven variability can materially improve interpretation\.”Claude \(blind\):“The influence of surface reflectance on UV irradiance is physically well\-established and has been studied extensively in the context of OMI and other satellite UV retrievals, including snow/ice albedo …\[novelty=2\]\.”*Cross\-judge note\.*This example illustrates the calibration shift quantified in[Section6](https://arxiv.org/html/2607.01584#S6): GPT and Claude both rate tractability highly and agree on the analysis design, but GPT returns importance=4=4and novelty=3=3where Claude returns importance=3=3and novelty=2=2\. The hypothesis is sound; only its absolute level depends on which backbone is asked\.

Example 3 \(Low\) — OMOCLO×\\timesML2CH3CL \(stratospheric chemistry\)Filter→\\toGenerator:GPT→\\toGPTSource stratum:A \(predicted novel\)\.Dataset pairOMOCLO— OMI/Aura Chlorine Dioxide \(OClO\) Total Column 1\-orbit L2 Swath 13×\\times24 km V003 \(GES\-DISC\)\. ML2CH3CL— MLS/Aura L2 Methyl Chloride \(CH3Cl\) Mixing Ratio V005 \(GES\-DISC\)\.Agent 2 — generated hypothesisResearch question\.Do enhancements in lower\-stratospheric methyl chloride \(CH3Cl\) precede and predict increases in polar chlorine dioxide \(OClO\) total columns during the onset of Antarctic springtime chlorine activation?Hypothesis\.During late winter to early spring \(Aug–Sep\) in the Antarctic vortex \(60–80∘S\), positive anomalies in MLS CH3Cl mixing ratios at 100–30 hPa are followed within 0–14 days by statistically significant increases in OMI OClO total columns under comparable solar\-zenith\-angle conditions, with the strongest effect in colder\-vortex years\.Analysis method\.Collocate daily OMOCLO OClO columns with MLS CH3Cl profiles averaged over the same vortex\-defined air mass \(equivalent latitude/PV filtering\); compute CH3Cl anomalies at 100–30 hPa; apply lagged regression and event\-based composites to test whether CH3Cl anomalies lead OClO increases, controlling for SZA and stratifying by vortex temperature\.Expected finding\.A robust positive lagged association in which elevated CH3Cl precedes higher OClO columns by up to∼\\sim2 weeks, larger in colder\-vortex years\.Scientific importance\.Linking an organic\-chlorine tracer to observed chlorine activation would clarify how variability in chlorine source\-gas transport modulates polar halogen chemistry and ozone\-loss potential\.Agent 3 verdicts \(Importance / Tractability / Novelty, 1–5\)Judge \(condition\)ImpTractNovGPT\-5\.2 \(blind\)343GPT\-5\.2 \(contextual\)323Claude 4\.6 \(blind\)222Claude 4\.6 \(contextual\)221Mean across judges2\.502\.502\.25Selected rationalesClaude \(blind\):“The fundamental mechanisms linking chlorine source gases to polar chlorine activation and OClO formation are well\-established; attributing interannual OClO variability to CH3Cl transport anomalies specifically is unlikely to yield a strong signal\.”GPT \(contextual\):“CH3Cl is a weak and indirect proxy for inorganic chlorine available for activation; the proposed lagged relationship is plausible in principle but execution is difficult given measurement uncertainty\.”*Failure mode\.*The two judges concur that the underlying chemistry is well\-studied and that CH3Cl is a weak proxy for the actual activation pathway; the GPT–Claude tractability gap under the blind condition \(4 vs\. 2\) reflects different priors about what counts as an executable observational study when the predicted effect size is small\.