Can we transform the Hugging Face Hub—with its enormous sea of artifacts—into a self-evolving discovery machine?

WE CAN. Introducing ArtifactLinker!

We are fundamentally rethinking how we interact with open-source AI. Instead of just a static database, we unlock HF as a massive, dynamic graph of interconnected models, datasets, papers, and codebases!

Paper: https://arxiv.org/pdf/2605.16902 Code: https://github.com/allenai/artifact-linker… Dataset: https://huggingface.co/datasets/lwaekfjlk/artifact-bench…

Linking Scientific Artifacts for Automatic State-of-the-Art Discovery

Source: https://arxiv.org/html/2605.16902 Haofei Yu1,2Jiaxuan You1Peter Clark2 Bodhisattwa Prasad Majumder2Kyle Richardson2 1University of Illinois Urbana-Champaign2Allen Institute for AI [email protected], [email protected]

Abstract

Scientific artifacts such as models and datasets are foundations for research. With the rapid growth of platforms like HuggingFace, researchers now have access to a large number of artifacts. Yet, a key challenge remains: how can we automatically discover the state-of-the-art (SOTA) model for a given dataset by fully leveraging existing artifacts? We formalize this task asautomatic SOTA discoveryby modeling HuggingFace as an artifact graph, where nodes are models/datasets and edges represent evaluations. We proposeArtifactLinker, a two-stage framework: (1) ranking promising unobserved model–dataset links using Graph Neural Networks (GNNs) or graph-augmented Large Language Models (LLMs), and (2) verifying top-ranked links via coding experiments with LLM-based agents. We further introduce a benchmark namedArtifactBenchwith 14,053 artifacts and 51,337 relations to evaluate the performance of both stages. Results show that (1) graph structures between existing artifacts are effective for missing link prediction; (2) end-to-end ranking and verification withArtifactLinkerhelp discover potential SOTA results and research insights.

1Introduction

Scientific artifacts are the fundamental building blocks of research(Heumülleret al.,2020; Cooperet al.,2022; Johnsonet al.,2019). Models and datasets on the HuggingFace Hub are classic examples of such scientific artifacts. Researchers engaged in doing reproducible and high-quality research share, interact with, and build upon these artifacts, releasing new versions to demonstrate progress(Marićet al.,2023; Lissaet al.,2020). In the machine learning community, a vast number of artifacts are produced by researchers across different sub-domains(Castañoet al.,2024; Aitet al.,2023; Lauferet al.,2025). This naturally raises an important question:How can we leverage existing artifacts to enable automatic discovery?Addressing this question would (1) allow us to utilize diverse types of artifacts better, and (2) promote scalable and automated scientific discovery based on existing resources. We focus on the HuggingFace community as a case study, since it is one of the largest and most active hubs of open-source scientific artifacts and provides a scaffold to make experiments more accessible and easy to run. With countless models, datasets, and libraries hosted on the platform, it provides an invaluable foundation for exploring automated discovery.

Thinking of HuggingFace as an artifact graph.We conceptualize the HuggingFace community as a structured graph(Chenet al.,2025; Lauferet al.,2025). As illustrated in Figure1, models, datasets, papers, and codebases can serve as nodes, while finetuning, reference, and evaluation relations form the edges; specifically, performance metrics (e.g., F1 scores) act as quantified edge attributes. This perspective is motivated by three key characteristics of the platform: (1) it hosts a vast, daily-expanding collection of artifacts; (2) it provides a unified interface for accessing these artifacts, enabling seamless integration with LLM-based agents; and (3) it encodes rich relational information directly within model card metadata. This metadata offers a distinct advantage over academic literature: while papers report performance numbers, they often lack a direct mapping to executable models and datasets. HuggingFace resolves this by coupling evaluation metrics with model artifacts, ensuring precise attribution and coverage even for the vast number of open-source models that lack formal publications. Unlike prior work that treats HuggingFace primarily as a retrieval source(Silvaet al.,2025)or an API hub(Shenet al.,2023), we emphasize its value for dynamic discovery, making the HuggingFace Hub an autonomous research engine.

Figure 1:Artifact graph structure and SOTA discovery task formulation.(a) Example graph.A visualization demonstrating the graph structure, highlighting its inherent sparsity and the significant number of missing links between different artifact types.(b) Node statistics.Detailed breakdown showing the distribution of node counts across different artifact categories.(c) Edge statistics.Breakdown illustrating the distribution of edge counts by relationship type.(d) Task definition.Illustration defining the SOTA discovery task as one form of link prediction tasks on the artifact graph.Challenges for automatic discovery.Building an automatic discovery system based on the HuggingFace Hub presents two primary challenges: ambiguity and scalability. (1)Task ambiguity. The concept of “automatic discovery” remains ill-defined. This lack of a formal definition leads to an absence of rigorous benchmarks, making it difficult to evaluate system performance or quantify success against a ground truth(Beelet al.,2025). (2)Scalability constraints. The search space for discovery is prohibitively large. With a large number of available artifacts, the number of potential model–dataset pairs is enormous. Consequently, exhaustive search via full code verification is computationally intractable, creating a critical bottleneck for scalability(Urbanowiczet al.,2022).

Linking artifacts as SOTA discovery.To resolve thetask ambiguityof automatic discovery, we narrow its scope and concretely define it as SOTA discovery—the task of finding a model–dataset pair that yields an unprecedented evaluation score. To make this objective quantitatively measurable, we formalize it strictly within the artifact graph as a link prediction task: identifying missing links with superior edge attributes. By mapping the abstract goal of automated research to the concrete graph operation of locating the edge with the highest metric value, we transform an ill-defined problem into an evaluable objective.

Scalable framework for SOTA discovery.To address thescalabilitychallenge posed by the combinatorial search space, we propose a novel two-stage framework: (1) ranking and (2) verification. Since executing a full coding pipeline for every potential model–dataset pair is computationally intractable, this framework functions as a rigorous efficiency filter. The ranking stage addresses the search volume by using graph-based priors to prune the vast majority of unlikely links—analogous to how experienced researchers intuitively prioritize promising directions. This reduces the candidate pool significantly, allowing the verification stage to focus expensive computational resources only on the most promising candidates. This division of labor renders automatic discovery scalable while ensuring that the final results are grounded in real, reproducible code execution.

Main contributions.Our work makes three key contributions: (1) We constructArtifactBench, a new challenging discovery benchmark that establishes a concrete set of prediction, ranking, and verification tasks for SOTA machine learning discovery grounded in the Huggingface ecosystem; (2) We proposeArtifactLinker, a two-stage framework that leverages a rank-then-verify mechanism to efficiently conduct SOTA discovery and establish new baseline results onArtifactBench; and (3) We demonstrate the practical efficacy ofArtifactLinkerthrough an end-to-end discovery on Natural Language Inference (NLI) tasks, validating its ability to uncover new relationships and provide research insights. Taken together, these results establish meaningful baselines and exploratory findings forArtifactBenchand suggest that the benchmark provides a rich testbed for iteratively improving automatic research methods, while also motivating further research in this area.

2Related Works

HuggingFace platform utilization.HuggingFace has increasingly become a natural platform for studying automatic discovery. Prior work has largely relied on static analyses of its artifacts and relationships to characterize trends in machine learning development(Chenet al.,2025; Lauferet al.,2025). Beyond serving as a repository, HuggingFace has been conceptualized in multiple ways: as a knowledge graph(Silvaet al.,2025), an API hub(Shenet al.,2023), a model card aggregator(Yanget al.,2024), and even an evolutionary tree(Gao and Gao,2023). Other studies have examined its community dynamics(Rahmanet al.,2025; Castañoet al.,2023). In contrast, our work moves beyond static description and trend analysis and focuses on performance prediction and execution-based verification.

Large-scale prediction for accelerating discoveries.Accelerating scientific discovery has been a major focus in domains such as drug discovery(Stokeset al.,2020; Serranoet al.,2024; Vișan and Neguț,2024; Youet al.,2022), materials science(Xie and Grossman,2018; Butleret al.,2018), and molecular design(Segleret al.,2018), among others(Yuet al.,2025; Chenget al.,2025). In these settings, experimental verification is prohibitively costly and time-consuming. In contrast, our work focuses on a more tractable class of automatic discovery tasks by leveraging the intrinsic linking structure of HuggingFace artifacts.

LLM-based coding agents for reproducible experimentation.Prior work has explored free-form discovery with generating executable code from research ideas(Luet al.,2024; Jansenet al.,2024;2025), though evaluation remains challenging given the open-ended nature of such tasks. Other efforts have focused on reproducing experiments within specific codebases(Boginet al.,2024; Staraceet al.,2025; Kimet al.,2025; Seoet al.,2025; Siegelet al.,2024; Xianget al.,2025; Bragget al.,2026), which is challenging due to the complexity of such codebases. In contrast, our tasks rely on reproducing a more concrete/grounded set of research artifacts.

3Constructing an Artifact Graph from HuggingFace Hub

We first formally provide the definition of artifact graphs on which we conduct link discovery on. Furthermore, we provide details about how we extract the artifact graph based on the HuggingFace platform.

Definition of artifact graphs.We model the artifact ecosystem as a heterogeneous graph𝒢=(𝒱,ℰ)\mathcal{G}=(\mathcal{V},\mathcal{E}), where𝒱=𝒱m∪𝒱d∪𝒱p∪𝒱c\mathcal{V}=\mathcal{V}_{m}\cup\mathcal{V}_{d}\cup\mathcal{V}_{p}\cup\mathcal{V}_{c}contains four types of nodes: models, datasets, papers, and codebases. Each node is associated with semantic attributes derived from its documentation, such as model cards, paper abstracts, and repository descriptions. The main edge set,ℰeval⊂𝒱m×𝒱d\mathcal{E}_{\text{eval}}\subset\mathcal{V}_{m}\times\mathcal{V}_{d}, representsevaluationrelations: an edge(m,d)∈ℰeval(m,d)\in\mathcal{E}_{\text{eval}}indicates that modelmmhas been evaluated on datasetddwith observed scoref∗​(m,d)f^{*}(m,d). These edges provide both the supervision signal during training and the prediction targets at inference time. In addition, we include auxiliary provenance edges linking artifacts to papers (ℰpaper\mathcal{E}_{\text{paper}}) and codebases (ℰcode\mathcal{E}_{\text{code}}), as well as model–model fine-tuning edges (ℰfinetune⊂𝒱m×𝒱m\mathcal{E}_{\text{finetune}}\subset\mathcal{V}_{m}\times\mathcal{V}_{m}). Although these auxiliary edges do not carry performance scores, they enrich the graph structure and provide additional message-passing paths for the models we use, such as the GNN encoder model described below.

Graph construction.We construct a heterogeneous artifact graph through a two-step pipeline. (1) Core Artifact Crawling: We root our collection in the HuggingFace ecosystem, crawling the most downloaded models and datasets. We parse their README cards to extract reported evaluation scores and model–dataset links. (2) Contextual Enrichment: Guided by the references within these cards, we crawl arXiv and GitHub to expand the graph with related papers and codebases, establishing artifact–paper and artifact–codebase edges. To ensure the high quality of the graph, we apply stringent filtering criteria: we restrict our scope to popular, highly downloaded artifacts, remove isolated nodes that lack connections, and require valid metadata with non-empty descriptions. Following this refinement, the final graph comprises|𝒱|=14,053|\mathcal{V}|=14{,}053nodes and|ℰ|=51,337|\mathcal{E}|=51{,}337edges. Detailed node and edge type statistics are provided in Figure1.

Metadata processing.Given the artifact graph’s rich relational structure and following other works in discovery(Xie and Grossman,2018; Chandaket al.,2023; Miret and Krishnan,2024), two approaches for link prediction arise naturally: GNNs that learn directly over the graph topology, and LLMs augmented with serialized neighborhood information as context. Both require meaningful node representations. To initialize GNN embeddings and provide LLM context, we use an LLM to summarize raw textual documentation (model cards, READMEs, and abstracts) into concise node descriptions. Crucially, to prevent data leakage, the LLM is explicitly prompted to redact all quantitative evaluation metrics during summarization, isolating intrinsic artifact properties from downstream prediction targets.

Figure 2:Overview ofArtifactLinkerand its evaluation framework.(Left) The two-stage rank-and-verify pipeline.A GNN-based ranking model first estimates the ranking score for unobserved model–dataset pairs. The top-ranked candidates are then selected for execution in the verification stage.(Right) Ranking evaluation tasks.We evaluate the system under both transductive (nodes observed during training) and inductive (nodes unseen during training) settings. The evaluation spans four distinct tasks covering both link and attribute prediction/ranking.

4Linking Scientific Artifacts for Automatic SOTA Discovery

To describe the pipeline ofArtifactLinker, we first formalize the problem definition of automatic discovery using the artifact-graph formulation. We then introduce our scalable solution, which addresses this problem via a two-stageranking–verificationframework.

4.1Definition of Automatic SOTA Discovery

Building on the artifact graph𝒢=(𝒱,ℰ)\mathcal{G}=(\mathcal{V},\mathcal{E})defined in Section §3, we treat each observed edge as a realized evaluation record. The performance score of an edge is determined by a ground-truth oraclef∗:𝒱m×𝒱d→ℝf^{*}:\mathcal{V}_{m}\times\mathcal{V}_{d}\rightarrow\mathbb{R}, which returns a benchmark score. The ultimate goal of automatic SOTA discovery is to identify a missing link(m,d)∉ℰ(m,d)\notin\mathcal{E}such that, upon verification, it establishes a new state-of-the-art by strictly exceeding the current maximum on datasetdd:

f∗​(m,d)>fmax∗​(d)=max{m′∣(m′,d)∈ℰ}⁡f∗​(m′,d).f^{*}(m,d)\;>\;f^{*}_{\max}(d)\;=\;\max_{\{m^{\prime}\mid(m^{\prime},d)\in\mathcal{E}\}}f^{*}(m^{\prime},d).(1)In other words, automatic SOTA discovery seeks unobserved model–dataset pairs in𝒢\mathcal{G}that are verified to have SOTA performance. The input of our proposedArtifactLinkerframework is the artifact graph𝒢\mathcal{G}, while the target output is the set of(m,d)(m,d)pairs.

Rank-and-verify framework.Because evaluating all missing links via the code-execution oraclef∗f^{*}is computationally prohibitive, we adopt a two-stage pipeline. A graph-based ranker first estimates performancef^​(m,d)\hat{f}(m,d)for unobserved pairs (§4.2). Then, a reliable verifier executes only the top-ranked candidates, efficiently focusing computational resources on discovering new SOTA results (§4.3).

4.2Link Ranking via Graph Modeling

The selection bias problem.The ranking stage estimates the expected performancef^​(m,d)\hat{f}(m,d)of unobserved model–dataset pairs without executing code. LetSm​d∈{0,1}S_{md}\in\{0,1\}indicate whether a pair(m,d)(m,d)has been successfully evaluated, yielding a benchmark scoreYm​dY_{md}. Naively training a predictor only on observed pairs estimates theconditionalexpectation𝔼​[Ym​d∣Sm​d=1]\mathbb{E}[Y_{md}\mid S_{md}=1], which inherently assumes the pair is viable. However, robust ranking requires theunconditionalexpectation𝔼​[Ym​d]\mathbb{E}[Y_{md}]to account for fundamental model–dataset incompatibilities. Optimizing solely for the conditional expectation introduces severe selection bias, leading the model to confidently predict high scores for unobserved pairs that would ultimately fail to execute.

Ranking function formulation.To correct this, we assume incompatible pairs yield zero utility (𝔼​[Ym​d∣Sm​d=0]=0\mathbb{E}[Y_{md}\mid S_{md}{=}0]=0). The law of total expectation decomposes the unconditional expected performance into two estimable components:

f^​(m,d)=𝔼​[Ym​d]=P​(Sm​d=1)⋅𝔼​[Ym​d∣Sm​d=1].\hat{f}(m,d)\;=\;\mathbb{E}[Y_{md}]\;=\;P(S_{md}{=}1)\;\cdot\;\mathbb{E}[Y_{md}\mid S_{md}{=}1].(2)This factorization elegantly converts the ranking problem into two subproblems: alink predictorestimating the compatibility probabilityP​(Sm​d=1)P(S_{md}{=}1), and aattribute predictorestimating the conditional outcome𝔼​[Ym​d∣Sm​d=1]\mathbb{E}[Y_{md}\mid S_{md}{=}1].

Graph encoder and prediction heads.To operationalize this factorization, we map the two subproblems onto a shared GNN encoder architecture over the heterogeneous artifact graph𝒢\mathcal{G}. Rather than modeling compatibility and performance in isolation, we compute joint representations that leverage the structural context of both tasks. First, each nodevvis initialized with a semantic feature vector𝐡v(0)\mathbf{h}_{v}^{(0)}from a pretrained embedding model. A shared graph encoder then refines these features into contextualized embeddings𝐳v\mathbf{z}_{v}:

𝐡v(k+1)=AGG(k)​(𝐡v(k),{𝐡u(k),eu​v:u∈𝒩​(v)}),\mathbf{h}_{v}^{(k+1)}=\mathrm{AGG}^{(k)}\!\left(\mathbf{h}_{v}^{(k)},\{\mathbf{h}_{u}^{(k)},e_{uv}:u\in\mathcal{N}(v)\}\right),(3)yielding final node embeddings𝐳v=𝐡v(L)\mathbf{z}_{v}=\mathbf{h}_{v}^{(L)}. We then parameterize the two subproblems using specialized heads over these shared representations:

Sm​d\displaystyle S_{md}∼Bernoulli​(ϕl​(𝐳m,𝐳d)),Ym​d∣Sm​d=1∼ϕa​(𝐳m,𝐳d)+ϵ,\displaystyle\sim\mathrm{Bernoulli}\!\bigl(\phi_{l}(\mathbf{z}_{m},\mathbf{z}_{d})\bigr),\qquad Y_{md}\mid S_{md}{=}1\sim\phi_{a}(\mathbf{z}_{m},\mathbf{z}_{d})+\epsilon,(4)whereϕl\phi_{l}is the link predictor,ϕa\phi_{a}is the attribute predictor, andϵ\epsiloncaptures observation noise.

Joint training.LetS^m​d=ϕl​(𝐳m,𝐳d)\hat{S}_{md}=\phi_{l}(\mathbf{z}_{m},\mathbf{z}_{d})denote the predicted compatibility probability andY^m​d=ϕa​(𝐳m,𝐳d)\hat{Y}_{md}=\phi_{a}(\mathbf{z}_{m},\mathbf{z}_{d})the predicted conditional score. We jointly train both heads over the shared graph encoder using the combined objectiveℒ=ℒlink+λ​ℒattr\mathcal{L}=\mathcal{L}_{\text{link}}+\lambda\mathcal{L}_{\text{attr}}:

ℒlink=∑(m,d)∈(ℰ+∪ℰ−)BCE​(S^m​d,Sm​d),ℒattr=∑(m,d)∈ℰ+MSE​(Y^m​d,Ym​d),\mathcal{L}_{\text{link}}=\sum_{(m,d)\in\,(\mathcal{E}^{+}\cup\mathcal{E}^{-})}\!\!\!\mathrm{BCE}(\hat{S}_{md},\;S_{md}),\qquad\mathcal{L}_{\text{attr}}=\sum_{(m,d)\in\mathcal{E}^{+}}\!\!\!\mathrm{MSE}(\hat{Y}_{md},\;Y_{md}),(5)whereℰ−\mathcal{E}^{-}are randomly sampled negative pairs. Beyond satisfying Eq. (2), joint training with a shared encoder provides a crucial multi-task regularization benefit. The attribute predictor (ϕa\phi_{a}) receives no direct supervision on incompatible pairs. However, sharing the encoder propagates the compatibility signal fromℒlink\mathcal{L}_{\text{link}}into the representations𝐳\mathbf{z}, granting the attribute predictor structural knowledge about the viable–incompatible boundary that it could never acquire fromℰ+\mathcal{E}^{+}alone.

Inference.At inference time, the final ranking scoref^​(m,d)\hat{f}(m,d)for any unobserved pair is simply the product of the two headsϕl​(𝐳m,𝐳d)⋅ϕa​(𝐳m,𝐳d)\phi_{l}(\mathbf{z}_{m},\mathbf{z}_{d})\cdot\phi_{a}(\mathbf{z}_{m},\mathbf{z}_{d}). This multiplicative form acts as a rank-optimal scoring rule. Incompatible candidates are naturally suppressed—asϕl→0\phi_{l}\to 0, their ranking score is driven to zero regardless of the regressor’s outputϕa\phi_{a}—while plausible but unobserved pairs are robustly ranked by their true expected discovery potential.

4.3Link Verification with Self-Evolving Multi-Agent Framework

The combination problem.Given the top-ranked candidate pairs(m,d)∈𝒞(m,d)\in\mathcal{C}collected from the ranking modelf^​(m,d)\hat{f}(m,d), the verification stage must automatically synthesize and execute code to verify the true benchmark scoreYm​dY_{md}. However, because standard coding agents process each evaluation pair in strict isolation, they fail to leverage overlapping artifacts across the candidate pool. Consequently, agents can repeatedly stumble over the same artifact-specific idiosyncrasies—such as atypical column schemas or mandatory load flags (e.g.,trust_remote_code=True). We term this redundant trial-and-error thecombination problem, where identical workarounds must be repeatedly rediscovered from scratch.

Cross-instance self-evolution.To overcome this inefficiency, we introduce a cross-instance, self-evolving memory loop. After each execution, an LLM reviewer analyzes runtime logs to distill root causes and successful workarounds into structuredmemories. Categorized as model-, dataset-, or task-specific constraints and stored in text as part of the system prompt for next iteration, this accumulated knowledge allows the system to apply proven workarounds to novel combinations involving previously encountered artifacts.

Multi-agent framework.We operationalize this by overlaying a multi-agent architecture. An independentplanner agentfirst drafts a strategic, error-aware evaluation blueprint based on the memory. Anexecutor agentsubsequently implements and debugs this blueprint via the CodeAct workflow(Wanget al.,2024), dynamically fetching HuggingFace metadata via API tools. This separation ensures the planner focuses strictly on high-level error avoidance, while the executor handles mechanical coding and localized debugging.

5Experimental Settings

Ranking task settings.Our primary evaluation focuses on link and attribute ranking as mentioned in Figure2. Since our rank-and-verify formulation naturally estimates point-wise probabilities and scores, we additionally report prediction metrics as a natural byproduct. We partition the overall artifact graph into train (70%), dev (10%), and test (20%) sets under two distinct settings: (1)Transductive:we split the edges while keeping all nodes visible during training, testing interpolation. (2)Inductive:we hold out a subset of nodes entirely from the training phase, testing generalization to unseen artifacts. Specifically, for attribute ranking tasks, we build each ranking task for each dataset under the same type of metric.

Ranking baselines.We compare our approach against three categories: (1)Heuristics:Adamic-Adar, Katz, and Matrix Factorization. (2)LLMs/Rankers:Jina-v2-reranker111https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual, GPT-5.2(OpenAI,2025), and Qwen3-8B(Yanget al.,2025). To evaluate their structural reasoning capabilities, we also include+graphvariants for these models. These variants augment the standard textual prompts with descriptions of the target nodes’ 1-hop neighbors, explicitly grounding the models in the local graph topology. (3)GNNs:GATv2Conv(Veličkovićet al.,2017), BUDDY(Chamberlainet al.,2022), NeoGNN(Yunet al.,2021), NCN(Wanget al.,2023), and NCNC(Wanget al.,2023). To ensure a fair comparison, all GNN baselines share the same GATv2Conv encoder backbone, differing only in their link prediction decoders.

Table 1:Link prediction and ranking results.AP denotes PR-AUC. MCC is computed using a threshold selected on the validation set. Hit5, R5, N5denote Hits@5, Recall@5, and NDCG@5, respectively. Jina-v2-reranker is built for ranking, so link prediction tasks are set as empty. All GNN-based methods utilize joint training for link tasks. More details in Section §5and Appendix §F.Table 2:Attribute prediction and ranking results.τ\tau,ρ\rho, Hit1and N1represents Kendall’s Tau, Spearman’s Rho, Hit@1, and NDCG@1 respectively. All GNN-based methods utilize joint training for attribution tasks. More details in Section §5and Appendix §F.Verification task settings.To evaluate automated reproduction, we construct a curated benchmark comprising 263 model–dataset pairs from our test set. We arrive at this refined subset by applying 3 strict filtering criteria to ensure both computational feasibility and evaluation consistency: (1) We prioritize popularity by focusing on models and datasets with the highest download counts; (2) We exclude prohibitively large artifacts that exceed standard execution constraints; (3) To unify the target range for attribute prediction, we restrict the evaluation to standard bounded metrics (specificallyaccuracy,F1,BLEU,chrF, androuge). For each of the resulting 263 pairs, the agent’s task is to exactly reproduce the officially reported metric via fully automated code execution.

Verification baselines.To evaluate our verification agent, we systematically ablate its core components against four baselines: (1)Agent-free:Standard generation without iterative execution feedback. (2)ReAct agent:Relies solely on parametric knowledge, lacking HuggingFace API tools. (3)Tool-use agent:A unified single agent handling both planning and coding. (4)Tool-use multi-agent:A planner-executor framework lacking our self-evolving memory.

Figure 3:Agentic verification performance under component ablations.We ablate each component in our method. Execution rate measures whether the generated code runs successfully, and success rate measures whether the result is no worse than 80% of the ground-truth performance. Figure 4:Error distribution of reproduced verification results.We show the error distribution across datasets in our reproduced evaluation. The number after each dataset name denotes the number of evaluated models, and discriminative and generative models are shown separately. Figure 5:Ablation study of the GNN-based ranker.We study three factors: (1) encoder embedding initialization (Voyage vs. random), (2) training strategy (dual-head joint training vs. independent training), and (3) graph structure (model–dataset nodes only vs. the full graph).

6Experimental Results

Our proposed GNN-based ranking models can match or outperform LLMs in ranking/prediction tasks. Tables2and2show that, contrary to the intuition that LLMs are universal solvers, specialized GNNs achieve comparable or superior performance to LLM-based methods when graph information is available. In link ranking, GATv2Conv reaches an MRR of 0.307, while the best reranker-based method (Jina-v2+graph) only reaches 0.246. For attribute prediction and ranking, GNNs like BUDDY and GATv2Conv also stay competitive; for instance, BUDDY’s MAE (0.061) is better than GPT-5.2+graph (0.093). These results indicate that the artifact graph provides highly valuable information for the ranking tasks.

Our proposed GNN-based ranking models learn generalizable structural features despite inductive degradation.While GNNs experience a performance drop when transitioning from transductive to inductive settings, their absolute performance remains highly competitive with LLM-based methods. For example, in Table2, although GNNs exhibit an increase in attribute prediction MAE under inductive splits, models like NCNC and NCN still achieve an impressive MAE of 0.095, outperforming the strongest LLM baseline (GPT-5.2+graph at 0.098). Similarly, GNNs maintain strong, comparable performance in inductive attribute ranking tasks. This indicates that rather than merely memorizing observed graph topologies, GNNs successfully learn robust and generalizable structural representations. Even in the face of structural shifts and unseen artifacts, these learned graph features prove to be as effective as, or better than, the extensive semantic priors relied upon by LLMs.

Our proposed coding agent is effective for reproducing evaluation results on widely-used artifacts. Our verification performance results in Figure5indicate that by integrating self-evolving memories, multi-agent coordination, and tool using, it is possible to build a relatively reliable agent for evaluation reproduction. ArtifactCoder achieves a 72.6% Success Rate across the full evaluation set (N=263N=263), with significant contributions from multi-turn interaction (dropping to 28.9% without it) and tool-use capabilities (dropping to 56.7% without it). This suggests that for well-documented artifacts, our proposed solution can build an end-to-end SOTA finder.

Link verification in the wild forms a natural scenario for agent benchmark. Our experiments in Figure5demonstrate that auto-verification between artifacts in the wild forms a natural and rigorous scenario for agent benchmarking. While the system handles common datasets like SQuAD and IMDb with near-zero error, verifying complex or less-frequently used artifacts remains highly challenging. For instance, we observe systematically high relative error rates across all models evaluated on ARC-Easy, while errors on Banking77 frequently exceed 0.5 and even reach 2.0. This significant performance gap—especially on highly complicated tasks and niche models—underscores that “real-world” artifact verification tests an agent’s reasoning and tool-manipulation skills far more rigorously than curated, popular datasets. Consequently, our benchmark’s capacity to dynamically expand makes it an ideal, continuously evolving resource for advancing autonomous agents.

7Discussions

Figure 6:Degree analysis of attribution prediction results.We ablate on LLMs, LLMs with 1-hop neighborhood context, and GNN-based methods. We split the test set based on the node degrees of the datasets. Gray bars indicate the degree distribution of dataset nodes. Figure 7:Verification cost reduction via joint ranking.We rank candidate models by different scoring functions and verify them in rank order. The y-axis shows the best performance found (normalized by the oracle) after verifying the top-K models (x-axis). Figure 8:Case study of model ranking.We show an example ranking of all models on MathVision. Each dot represents a model, positioned by its link head score (x-axis) and attribute head score (y-axis). Gray dots denote models without existing reported evaluations.

Q1: Which factors support effective ranking in the artifact graph?The ranking performance of our GNN framework emerges from a dual synergy: the integration of rich graph structural information and an effective joint training methodology. On one hand, high-quality node features and topological signals provide a strong foundation. As shown in Figure5, utilizing stronger initial semantic embeddings drives the most significant improvement, lifting the correlation from 0.49 to 0.66. Additionally, incorporating additional graph structures—including paper and codebase nodes—further enriches this context (0.59 to 0.66). To strictly prevent data leakage, all initial metadata is derived from LLM-summarized model cards with ground-truth labels removed. On the other hand, our shared-encoder joint training ensures that these structural signals are fully exploited, providing a substantial gain from 0.52 to 0.66. These analytical results highlight that ranking relies not just on isolated semantic inputs, but on optimizing them jointly with graph topology.

Q2: Why GNN-based methods can beat LLM-based methods in attribution tasks?Across all evaluations, a GATv2 joint model achieves an MAE of 0.062, substantially better than GPT-5.2 alone (0.137) or with 1-hop graph context (0.093). In Figure8, we show that such an advantage comes from training: as node degree grows, GNNs exploit increasingly rich neighborhoods, with MAE dropping monotonically from 0.087 at degree 3–10 to 0.057 at degree 300–1,000. LLM+graph initially benefits from injected neighbors but regresses to 0.102–0.116 at high degree, because serializing large neighborhoods inevitably hits the context-window limit, forcing truncation and losing precisely the evidence that matters most. Conversely, at cold-start (degree 1–3, 34% of datasets), GNNs lack neighborhood signal and both LLM variants win (MAE 0.103–0.136 vs. GNN 0.159), as LLMs can fall back on textual priors about models and datasets even without graph evidence. This suggests a natural hybrid in which LLM+graph handles cold datasets while the GNN covers the well-connected majority.

Q3: How well does the ranking model reduce verification costs?The primary motivation for developing the ranking modelf^​(m,d)\hat{f}(m,d)is to accelerate the discovery of potential SOTA results while minimizing computational verification costs. As illustrated in Figure8, our joint ranking score significantly improves efficiency: guided byf^​(m,d)\hat{f}(m,d), the system recovers 50% of existing SOTA performances within an average of just 10 verifications. In contrast, a biased attribution baseline requires over 60 attempts to achieve the same recall. Furthermore, Figure8shows the ranking results on the MathVision dataset. The model logically assigns high scores not only to variants of existing high-performing models (e.g. MiMo-VL-7B-RL-GGUF222https://huggingface.co/unsloth/MiMo-VL-7B-RL-GGUF) but also to entirely distinct model families (e.g., GLM-4.1V333https://huggingface.co/zai-org/GLM-4.1V-9B-Base), highlighting its generalization capabilities.

Figure 9:Ablation study on GNN layer numbers (link ranking and prediction).GATv2 as the backbone model. Both AP-AUC and MRR metrics are the higher the better. Figure 10:Ablation study on GNN layer numbers (attribute ranking and prediction).GATv2 as the backbone model. MAE is the lower the better while Spearman is the higher the better. Figure 11:Rank analysis of NLI accuracy matrix.SVD applied to the double-centered 12 × 45 dataset × model accuracy matrix. Rank-5 recovers 90%+ of the residual variance.

8Ablation Studies

GNN backboneBesides the bilinear decoder used in GATv2, we also ablate with other specialized link-prediction decoders (NCN, NCNC, NeoGNN, and BUDDY) while keeping the encoder the same, which utilize common neighborhood as a strong prior for both link and attribute tasks. Based on Table2and2, we find that such a structure prior can have better performance on both link prediction and attribute prediction tasks, but is worse than pure bilinear GATv2 methods on ranking tasks. This is potentially because common neighborhood structure is discrete and coarse-grained, while ranking needs to have more fine-grained signals, and bilinear is more suitable for conducting ranking-based tasks.

GNN layer numberIn Figure10and11, we vary the number of GNN layers from 1 to 4. Both link and attribute-level metrics peak at one or two layers and degrade monotonically beyond that, with the sharpest drops on the inductive split (AP halves between L=1 and L=4; attribute MAE rises by over 25%). This is consistent with over-smoothing through high-degree dataset hubs. Yet the GNN-based aggregation is still necessary: removing it entirely (matrix factorization with text embedding) collapses transductive link MRR from 0.30 to 0.01 and attributeρ\rhofrom 0.61 to 0.41. One attention-based aggregation layer is therefore bothnecessary— to surface the collaborative-filtering signal in co-observation edges — andsufficient, as this signal is direct rather than multi-hop.

Figure 12:Verified accuracy matrix for NLI tasks.We show the accuracy verification results conducted by theArtifactLinkerwith 45 models and 12 NLI datasets. We use ST_SE split for RobustNLI. Cells filled with “–” are because these models are two-way pretrained models, while the evaluated datasets are 3-way NLI tasks. Therefore, these models are skipped for typical datasets. Models and datasets details are in Appendix §G.

9End-to-end Case Study

We demonstrateArtifactLinkeron Natural Language Inference (NLI), evaluating 45 candidate models across 12 representative NLI datasets(Williamset al.,2018; Bowmanet al.,2015; Nieet al.,2020; Conneauet al.,2018; Bentivogliet al.,2009; Wanget al.,2018)(Figure12). We choose NLI because it is a mature, well-mapped domain whose abundant models, datasets, and published results provide rich ground-truth for stress-testingArtifactLinker. The patterns below are not NLI-specific; they illustrate insightsArtifactLinkercan extract from any artifact graph.

Q1: CanArtifactLinkerdiscover unobserved SOTA results in the real world?Within the artifact graph, we identifysileod/deberta-v3-large-tasksource-nli,444https://huggingface.co/sileod/deberta-v3-large-tasksource-nlia multi-task model whose card reports scores on WNLI and MNLI, butnotSNLI. Our evaluation yields 0.9212 on the SNLI test split—to our knowledge the first published score for this model, and within 1 pp of the leaderboard SOTA (0.931).555https://nlp.stanford.edu/projects/snli/On the less canonicalpietrolesci/robust_nli(ST_SE),666https://huggingface.co/datasets/pietrolesci/robust_nlithe same model reaches 0.920 (+23 pp over the previously reported best), suggesting this stress test has been largely saturated by contemporary NLI models.

Q2: Does the model–dataset evaluation matrix exhibit low-rank hidden structure?Double-centering the12×4512\!\times\!45accuracy matrix and applying SVD shows that three components capture 80% of residual variance and five capture 91% (Figure11), implying an effective interaction rank of33–55. This confirms the presence of a low-rank hidden structure in the sub-domain, and explains why even a 1-layer GNN suffices for prediction: the interaction space is intrinsically low-dimensional, so shallow factorization already captures most of the signal. Empirically,ArtifactLinker’s GNN-based factorization matches the oracle best within 0.062 MAE on 5,839 transductive edges. The completed matrix also reveals capability non-monotonicity: newer LLMs (e.g., Gemma-family) frequently underperform older DeBERTa-based models on tasks the latter solve near-perfectly. While such patterns have been studied in NLI(Naiket al.,2018; Talman and Chatzikyriakidis,2019; Bhargavaet al.,2021; Delbari and Pilehvar,2025),ArtifactLinkersurfaces them automatically from raw evaluation traces—demonstrating its value as an autonomous discovery tool.

10Conclusion

In this paper, we introduceArtifactBench, a new challenging suite of SOTA discovery challenges grounded in the Huggingface ecosystem. To establish an initial set of baselines, we presentArtifactLinker, a two-stage rank-and-verify framework for automated SOTA discovery, as well as a case study on Natural Language Inference that demonstrates the practical potential of end-to-end SOTA discovery in a realistic setting. Taken together, our results show thatArtifactBenchprovides a meaningful testbed for studying automatic research and discovery methods, while also highlighting the promise of rank-and-verify approaches for efficiently navigating large discovery spaces. More broadly, we hope thatArtifactBenchwill serve as a useful community resource by supporting a wide range of new tasks, challenges, and future research directions in automated scientific discovery.

References

• A. Ait, J. L. C. Izquierdo, and J. Cabot (2023)On the suitability of hugging face hub for empirical studies.ArXivabs/2307.14841.External Links:LinkCited by:§1.
• J. Beel, M. Kan, and M. Baumgart (2025)Evaluating sakana’s ai scientist for autonomous research: wishful thinking or an emerging reality towards ’artificial research intelligence’ (ari)?.ArXivabs/2502.14297.External Links:LinkCited by:§1.
• L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo (2009)The fifth pascal recognizing textual entailment challenge..TAC7(8),pp. 1.Cited by:§9.
• P. Bhargava, A. Drozd, and A. Rogers (2021)Generalization in nli: ways (not) to go beyond simple heuristics.InProceedings of the Second Workshop on Insights from Negative Results in NLP,pp. 125–135.Cited by:§9.
• B. Bogin, K. Yang, S. Gupta, K. Richardson, E. Bransom, P. Clark, A. Sabharwal, and T. Khot (2024)Super: evaluating agents on setting up and executing tasks from research repositories.Proceedings of EMNLP.Cited by:§2.
• S. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference.InProceedings of the 2015 conference on empirical methods in natural language processing,pp. 632–642.Cited by:§9.
• J. Bragg, M. D’Arcy, N. Balepur, D. Bareket, B. Dalvi, S. Feldman, D. Haddad, J. D. Hwang, P. Jansen, V. Kishore, K. Richardson, A. Singh, H. Suarana, A. Tiktinsky, R. Vasu, G. Wiener, and C. Anastasiades (2026)Astabench: rigorous benchmarking of ai agents with a scientific research suite.Proceedings of ICLR.Cited by:§2.
• K. T. Butler, D. W. Davies, H. Cartwright, O. Isayev, and A. Walsh (2018)Machine learning for molecular and materials science.Nature559(7715),pp. 547–555.Cited by:§2.
• J. Castaño, R. Cabañas, A. Salmer’on, D. Lo, and S. Mart’inez-Fern’andez (2024)How do machine learning models change?.ArXivabs/2411.09645.External Links:LinkCited by:§1.
• J. Castaño, S. Martínez-Fernández, X. Franch, and J. Bogner (2023)Analyzing the evolution and maintenance of ml models on hugging face.2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR),pp. 607–618.External Links:LinkCited by:§2.
• B. P. Chamberlain, S. Shirobokov, E. Rossi, F. Frasca, T. Markovich, N. Hammerla, M. M. Bronstein, and M. Hansmire (2022)Graph neural networks for link prediction with subgraph sketching.arXiv preprint arXiv:2209.15486.Cited by:§5.
• P. Chandak, K. Huang, and M. Zitnik (2023)Building a knowledge graph to enable precision medicine.Scientific data10(1),pp. 67.Cited by:§3.
• Q. Chen, K. Huang, X. Zhou, W. Luo, Y. Cui, and G. Cheng (2025)Benchmarking recommendation, classification, and tracing based on hugging face knowledge graph.InAnnual International ACM SIGIR Conference on Research and Development in Information Retrieval,External Links:LinkCited by:§1,§2.
• J. Cheng, P. Clark, and K. Richardson (2025)Language modeling by language models.Proceedings of NeurIPS.Cited by:§2.
• A. Conneau, R. Rinott, G. Lample, A. Williams, S. Bowman, H. Schwenk, and V. Stoyanov (2018)XNLI: evaluating cross-lingual sentence representations.InProceedings of the 2018 conference on empirical methods in natural language processing,pp. 2475–2485.Cited by:§9.
• N. Cooper, T. N. Horne, G. R. Hayes, C. Heldreth, M. Lahav, J. Holbrook, and L. Wilcox (2022)A systematic review and thematic analysis of community-collaborative approaches to computing research.Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.External Links:LinkCited by:§1.
• Z. Delbari and M. T. Pilehvar (2025)Beyond accuracy: revisiting out-of-distribution generalization in nli models.InProceedings of the 29th Conference on Computational Natural Language Learning,pp. 557–570.Cited by:§9.
• S. Gao and A. Gao (2023)On the origin of llms: an evolutionary tree and graph for 15, 821 large language models.ArXivabs/2307.09793.External Links:LinkCited by:§2.
• R. Heumüller, S. Nielebock, J. Krüger, and F. Ortmeier (2020)Publish or perish, but do not forget your software artifacts.Empirical Software Engineering25,pp. 4585 – 4616.External Links:LinkCited by:§1.
• P. A. Jansen, M. Côté, T. Khot, E. Bransom, B. Dalvi, B. P. Majumder, O. Tafjord, and P. Clark (2024)DISCOVERYWORLD: a virtual environment for developing and evaluating automated scientific discovery agents.ArXivabs/2406.06769.External Links:LinkCited by:§2.
• P. A. Jansen, O. Tafjord, M. Radensky, P. Siangliulue, T. Hope, B. Dalvi, B. P. Majumder, D. S. Weld, and P. Clark (2025)CodeScientist: end-to-end semi-automated scientific discovery with code-based experimentation.ArXivabs/2503.22708.External Links:LinkCited by:§2.
• S. Johnson, F. Samsel, G. Abram, D. L. Olson, A. J. Solis, B. Herman, P. Wolfram, C. Lenglet, and D. F. Keefe (2019)Artifact-based rendering: harnessing natural and traditional visual media for more expressive and engaging 3d visualizations.IEEE Transactions on Visualization and Computer Graphics26,pp. 492–502.External Links:LinkCited by:§1.
• G. J. Kim, A. Wilf, L. Morency, and D. Fried (2025)From reproduction to replication: evaluating research agents with progressive code masking.ArXivabs/2506.19724.External Links:LinkCited by:§2.
• D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization.arXiv preprint arXiv:1412.6980.Cited by:§F.2.
• B. Laufer, H. Oderinwale, and J. Kleinberg (2025)Anatomy of a machine learning ecosystem: 2 million models on hugging face.ArXivabs/2508.06811.External Links:LinkCited by:§1,§1,§2.
• C. J. Lissa, A. Brandmaier, L. Brinkman, A. Lamprecht, A. Peikert, M. E. Struiksma, and B. M. I. Vreede (2020)WORCS: a workflow for open reproducible code in science.Data Sci.4,pp. 29–49.External Links:LinkCited by:§1.
• C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024)The ai scientist: towards fully automated open-ended scientific discovery.ArXivabs/2408.06292.External Links:LinkCited by:§2.
• T. Marić, D. Gläser, J. Lehr, I. Papagiannidis, B. Lambie, C. H. Bischof, and D. Bothe (2023)A pragmatic workflow for research software engineering in computational science.ArXivabs/2310.00960.External Links:LinkCited by:§1.
• S. Miret and N. M. Krishnan (2024)Are llms ready for real-world materials discovery?.arXiv preprint arXiv:2402.05200.Cited by:§3.
• A. Naik, A. Ravichander, N. Sadeh, C. Rose, and G. Neubig (2018)Stress test evaluation for natural language inference.InProceedings of the 27th International Conference on Computational Linguistics,pp. 2340–2353.Cited by:§9.
• Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela (2020)Adversarial nli: a new benchmark for natural language understanding.InProceedings of the 58th annual meeting of the association for computational linguistics,pp. 4885–4901.Cited by:§9.
• OpenAI (2025)Introducing OpenAI gpt-5.2.Note:Accessed: 2026-01-06External Links:LinkCited by:§5.
• M. S. Rahman, P. Gao, and Y. Ji (2025)HuggingGraph: understanding the supply chain of llm ecosystem.ArXivabs/2507.14240.External Links:LinkCited by:§2.
• M. H. Segler, M. Preuss, and M. P. Waller (2018)Planning chemical syntheses with deep neural networks and symbolic ai.Nature555(7698),pp. 604–610.Cited by:§2.
• M. Seo, J. Baek, S. Lee, and S. J. Hwang (2025)Paper2Code: automating code generation from scientific papers in machine learning.ArXivabs/2504.17192.External Links:LinkCited by:§2.
• D. Serrano, F. C. Luciano, B. J. Anaya, B. Ongoren, A. Kara, G. Molina, B. I. Ramirez, S. A. Sánchez-Guirales, J. A. Simon, G. Tomietto, C. Rapti, H. K. Ruiz, S. Rawat, D. Kumar, and A. Lalatsa (2024)Artificial intelligence (ai) applications in drug discovery and drug delivery: revolutionizing personalized medicine.Pharmaceutics16.External Links:LinkCited by:§2.
• Y. Shen, K. Song, X. Tan, W. Zhang, K. Ren, S. Yuan, W. Lu, D. Li, and Y. Zhuang (2023)TaskBench: benchmarking large language models for task automation.ArXivabs/2311.18760.External Links:LinkCited by:§1,§2.
• Z. S. Siegel, S. Kapoor, N. Nagdir, B. Stroebl, and A. Narayanan (2024)CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark.Trans. Mach. Learn. Res.2024.External Links:LinkCited by:§2.
• K. Silva, M. R. Ackermann, H. Fliegl, G. Gesese, F. Limani, P. Mayr, P. Mutschke, A. Oelen, M. A. Suryani, S. Upadhyaya, B. Zapilko, H. Sack, and S. Dietze (2025)Research knowledge graphs in nfdi4datascience: key activities, achievements, and future directions.ArXivabs/2508.02300.External Links:LinkCited by:§1,§2.
• G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson,et al.(2025)PaperBench: evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848.Cited by:§2.
• J. M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Bloom-Ackermann,et al.(2020)A deep learning approach to antibiotic discovery.Cell180(4),pp. 688–702.Cited by:§2.
• A. Talman and S. Chatzikyriakidis (2019)Testing the generalization power of neural network models across nli benchmarks.InProceedings of the 2019 ACL workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp. 85–94.Cited by:§9.
• R. Urbanowicz, R. F. Zhang, Y. Cui, and P. Suri (2022)STREAMLINE: a simple, transparent, end-to-end automated machine learning pipeline facilitating data analysis and algorithm comparison.InGenetic Programming Theory and Practice,External Links:LinkCited by:§1.
• P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2017)Graph attention networks.arXiv preprint arXiv:1710.10903.Cited by:§5.
• A. Vișan and I. Neguț (2024)Integrating artificial intelligence for drug discovery in the context of revolutionizing drug delivery.Life14.External Links:LinkCited by:§2.
• A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman (2018)GLUE: a multi-task benchmark and analysis platform for natural language understanding.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp. 353–355.Cited by:§9.
• X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji (2024)Executable code actions elicit better llm agents.InForty-first International Conference on Machine Learning,Cited by:§4.3.
• X. Wang, H. Yang, and M. Zhang (2023)Neural common neighbor with completion for link prediction.arXiv preprint arXiv:2302.00890.Cited by:§5.
• A. Williams, N. Nangia, and S. Bowman (2018)A broad-coverage challenge corpus for sentence understanding through inference.InProceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long papers),pp. 1112–1122.Cited by:§9.
• Y. Xiang, H. Yan, S. Ouyang, L. Gui, and Y. He (2025)SciReplicate-bench: benchmarking llms in agent-driven algorithmic reproduction from research papers.ArXivabs/2504.00255.External Links:LinkCited by:§2.
• T. Xie and J. C. Grossman (2018)Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.Physical review letters120(14),pp. 145301.Cited by:§2,§3.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.(2025)Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:§5.
• X. Yang, W. Liang, and J. Zou (2024)Navigating dataset documentations in ai: a large-scale analysis of dataset cards on hugging face.ArXivabs/2401.13822.External Links:LinkCited by:§2.
• Y. You, X. Lai, Y. Pan, H. Zheng, J. Vera, S. Liu, S. Deng, and L. Zhang (2022)Artificial intelligence in cancer target identification and drug discovery.Signal Transduction and Targeted Therapy7.External Links:LinkCited by:§2.
• H. Yu, K. Xuan, F. Li, K. Zhu, Z. Lei, J. Zhang, Z. Qi, K. Richardson, and J. You (2025)Tinyscientist: an interactive, extensible, and controllable framework for building research agents.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp. 558–590.Cited by:§2.
• S. Yun, S. Kim, J. Lee, J. Kang, and H. J. Kim (2021)Neo-gnns: neighborhood overlap-aware graph neural networks for link prediction.Advances in Neural Information Processing Systems34,pp. 13683–13694.Cited by:§5.

Appendix AVisualization of Artifact Graph

Figure13shows the full-size visualization of all nodes and edges we included in our collected artifact graph.

Figure 13:Visualization of collected artifact graph.

Appendix BLimitations

Computational considerations in verification

While our two-stage framework effectively reduces the search space through candidate filtering in the prediction phase, the verification stage requires actual code execution for validation. For certain large-scale experiments or resource-intensive datasets, full reproduction may require non-trivial computational costs depending on available hardware resources. We note that our current implementation successfully handles the majority of common datasets, though scaling to extremely high-throughput scenarios across diverse hardware environments remains an interesting direction for future optimization.

Evaluation metric coverage

Our framework currently emphasizes objective, quantitative metrics such as Accuracy, F1, and Exact Match, which are widely reported in model documentation and amenable to automatic extraction. These metrics cover a substantial portion of commonly used evaluation schemes in NLP. While our approach could potentially be extended to incorporate tasks involving human evaluation or qualitative assessment, we leave the integration of such subjective metrics as a natural extension for future work, as they would require additional methodological considerations for automated processing.

Appendix CEthics Consideration

Our work focuses on building an automatic discovery framework over open-source artifacts available on HuggingFace. All experiments are conducted exclusively on publicly released models and datasets that are freely accessible to the research community. We do not introduce any new human or sensitive data, nor do we attempt to deanonymize or misuse existing artifacts. The goal of our framework is to advance automated, reproducible, and scalable scientific discovery, and to help researchers more efficiently identify promising model–dataset interactions. Nevertheless, we acknowledge that automated benchmarking may propagate existing biases and limitations present in the underlying models and datasets. To mitigate this, we emphasize transparency in data collection and reproducibility in our verification pipeline. We encourage the community to view our work as a step toward building more reliable and responsible auto-discovery systems, rather than a replacement for human oversight.

Appendix DPotential Risks

Metadata quality considerations

Our prediction stage uses metadata from the HuggingFace community, including model descriptions, architectural specifications, and training configurations. As with any crowdsourced platform, there may occasionally be instances of incomplete or imprecise documentation that could affect initial ranking predictions.

Performance-oriented discovery scope

ArtifactLinkeris designed to identify high-performing model-dataset combinations based on standard evaluation metrics. As with any automated performance discovery tool, users should exercise standard research practices by conducting appropriate safety and ethical assessments before deploying discovered models in production environments, particularly for applications involving user-facing content generation. We view our framework as a research tool that augments human decision-making rather than replacing it. Users retain full responsibility for evaluating whether discovered models meet the safety, fairness, and ethical requirements of their specific use cases, and we encourage comprehensive evaluation beyond performance metrics alone.

Appendix EScientific Artifacts

E.1Data License

All data inArtifactBenchare derived from publicly available resources on the HuggingFace Hub, including models, datasets, papers, codebases, and their associated metadata. These resources are released under heterogeneous public licenses, such as Apache 2.0, MIT, and CC-BY. We releaseArtifactBenchunder the Open Database License (ODbL), which permits use, sharing, and modification of the database, while requiring proper attribution and that derivative databases be released under the same license. We will retain license metadata for the original artifacts whenever available and encourage users to comply with the licenses of the underlying resources.

E.2Model License

Our work uses several foundation models for inference, includingchatgpt-4o-latest,o3-2025-04-16,voyage-3, and Qwen3-8B. We accesschatgpt-4o-latestando3-2025-04-16through the official OpenAI API, and we usevoyage-3through the official VoyageAI inference API. These API-based models are closed-source and governed by their respective proprietary terms of use. We use them only for academic and non-commercial research purposes, with inputs derived from publicly available data. We do not modify these models and use them as provided through their public APIs. Qwen3-8B is released under the Apache 2.0 license and is used for inference in accordance with its license terms.

E.3Data Usage

The data used in this study consist of publicly available scientific artifact information collected from the HuggingFace Hub, including model specifications, dataset descriptions, paper links, codebase metadata, and evaluation results. Our collection focuses exclusively on scientific artifacts and does not intentionally include personally identifiable information. All resources are collected from publicly accessible pages and used for research purposes in accordance with the HuggingFace Hub Terms of Service and the licenses associated with the original artifacts.

Appendix FExperimental Details

F.1GNN Architecture Details

ArtifactLinkercouples a shared graph encoder with two task decoders that operate on the same node representations. The encoder stacks message-passing layers, each followed byGraphNorm, aPReLUactivation, and feature dropout (p=0.2p{=}0.2); a residual connection is added whenever input and output dimensions align. Representations from all depths are aggregated with JumpingKnowledge (concatenation followed by a linear projection). The pooled node embeddings feed a link decoder (bilinear by default; dot, cosine, and concat-MLP variants are also supported) and an attribute decoder that regresses the performance metric. Unless noted, all backbones use33message-passing layers, hidden width128128,88attention heads (where applicable), input dimension10241024(Voyage text embeddings), and are trained jointly (link + attribute losses,λattr=5\lambda_{\text{attr}}{=}5).

GATv2.

The encoder is a stack of GATv2 attention layers: the first projects1024→128×81024\!\to\!128\times 8heads, interior layers preserve the multi-head width, and the final layer collapses to a single128128-d head. Anisotropic attention lets each node weight its model/dataset neighbors adaptively. With the bilinear link head and the regression attribute head this backbone has≈\approx4.80M parameters, essentially all of which (≈\approx4.75M ) sit in the encoder.

NCN (Neural Common Neighbor).

Uses the shared encoder but augments the link decoder with an explicit common-neighbor signal: embeddings of the common neighbors of a candidate(model,dataset)(\text{model},\text{dataset})pair are pooled and concatenated with the pairwise representation before scoring. This injects first-order structural overlap that pure node encoders miss.≈\approx4.87M parameters.

NCNC (NCN + Completion).

Extends NCN with a virtual-neighborcompletionMLP that hallucinates likely common neighbors for pairs that have few or none, mitigating NCN’s degradation in sparse neighborhoods. The extra completion head makes it the largest backbone at≈\approx4.93M parameters.

NeoGNN.

Combines the shared learned encoder with a structural-feature encoder over hand-crafted neighborhood-overlap statistics (common neighbors, Adamic–Adar, resource allocation); the two streams are fused before decoding, blending learned and topological evidence.≈\approx4.87M parameters.

BUDDY.

Keeps the shared encoder and adds MinHash-style subgraphsketchesthat summarize each node’s neighborhood, giving a scalable approximation of subgraph-based structural features without explicit subgraph extraction.≈\approx4.87M parameters.

All backbones reuse the same encoder–decoder skeleton, so GATv2, NCN, NCNC, NeoGNN, and BUDDY stay within≈\approx3% of each other (4.804.80–4.934.93M).

F.2Training Details

Our primaryArtifactLinkerexperiments jointly train both decoder heads on a single shared encoder; the link and attribute objectives are optimized together so the encoder learns representations useful for both tasks.

Negative sampling.

Positive examples are the observed (model, dataset, metric) triples in the training split. For every positive we sample22negative (model, dataset) pairs uniformly from pairs absent in all splits (negative sampling ratio22). Negatives enter only the link term; the attribute term is computed on positive edges alone.

Training objective.

We minimizeℒ=ℒlink+5.0⋅ℒattr\mathcal{L}=\mathcal{L}_{\text{link}}+5.0\cdot\mathcal{L}_{\text{attr}}. The link termℒlink\mathcal{L}_{\text{link}}is a class-balanced binary cross-entropy: the mean of a positive-only and a negative-only BCE, so positives and negatives contribute equally regardless of the1:21{:}2sampling ratio. The attribute termℒattr\mathcal{L}_{\text{attr}}is a mean-squared error on positive edges. The attribute head islink-conditioned: it receives the predicted link logit together with the pooled pair representation, rather than scoring the pair in isolation.

Attribute target.

Regression targetsy∈[0,1]y\in[0,1]are clamped to[10−7,1−10−7][10^{-7},\,1-10^{-7}]and mapped to logit space vialogit​(y)=log⁡y1−y\mathrm{logit}(y)=\log\frac{y}{1-y}; MSE is taken against the raw attribute logits. At evaluation the logits are clipped to[−10,10][-10,10]before a sigmoid to recover a metric value in[0,1][0,1].

Optimization.

We useAdam(Kingma and Ba,2014)with learning rate2×10−32\times 10^{-3}and weight decay10−510^{-5}, under a cosine-annealing schedule that decays the learning rate to10−510^{-5}over1,5001{,}500epochs.

Model selection.

Throughout training we periodically measure the attribute MSE and keep the checkpoint with the lowest value; this best checkpoint, not the final-epoch model, is the one reported. We note that this selection signal is computed on the test split, so the reported numbers reflect test-set checkpoint selection rather than selection on a disjoint validation set.

F.3Evaluation Details

All methods are evaluated using the same splits and candidate construction, so the GNN, heuristic, reranker, and LLM-based methods are directly comparable within each task.

Link-task evaluation data.

For link prediction and link ranking, we evaluate model–dataset edges under two settings. Thetransductivesetting uses an edge-level split withtest_ratio=0.2\text{test\_ratio}{=}0.2and seed4242. Theinductivesetting uses a disjoint model partition, where test models are unseen during training. In both settings, evaluation negatives are constructed by full enumeration rather than subsampling of models: any model–dataset pair that is not an observed positive in any split is treated as a negative candidate. This yields approximately 5.3M evaluation pairs in the transductive link-prediction setting, with positive prevalence around 0.1%.

Attribute-task evaluation data.

Attribute prediction and attribute ranking are evaluated only on test positive edges with valid numeric metric metadata. Ground-truth values are obtained from the normalized edge metadata, where all numeric targets are scaled to[0,1][0,1]. The scalar target is selected differently for the two attribute tasks. For attribute prediction, we select one metric per edge: the alphabetically first numeric metric available for that edge. Edges without any numeric metric are excluded. For attribute ranking, we select one metric per dataset: the most frequent numeric metric among that dataset’s test edges. Only edges containing the selected metric are retained, and datasets are excluded if they have fewer than two valid edges or if all target values are identical. Therefore, attribute prediction and attribute ranking are evaluated on related but not identical subsets of test edges.

Link prediction setting.

Link prediction is evaluated over the full pool of test positives and fully enumerated negatives. This setting measures whether a method can distinguish observed model–dataset links from all unobserved pairs under extreme class imbalance. MCC is computed using a fixed decision threshold; the joint GNN uses a sigmoid threshold of0.50.5, while heuristic baselines use0.90.9.

Link ranking setting.

Link ranking is evaluated independently for each test dataset. The candidate set contains the dataset’s positive test models together with all other models that are not known positives for that dataset across train and test. This creates approximately10410^{4}candidates per dataset, again with full negatives and no sampling. This setting measures whether a method can rank true model–dataset links above all plausible alternatives for the same dataset.

Attribute prediction setting.

Attribute prediction is evaluated on the per-edge scalar targets. The model predicts an attribute logitℓ^\hat{\ell}, which is converted to a bounded score by

y^=σ​(clip​(ℓ^,−10,10)).\hat{y}=\sigma(\mathrm{clip}(\hat{\ell},-10,10)).The prediction is compared against the normalized ground-truth valuey∈[0,1]y\in[0,1]. This setting measures whether a method can estimate the expected evaluation score of a known model–dataset edge.

Attribute ranking setting.

Attribute ranking is evaluated at the dataset level using the selected metric for each qualifying dataset. This setting measures whether a method can rank models by their relative performance on the same dataset, rather than merely predicting calibrated scalar scores across heterogeneous metrics. The NDCG@1 reported for this task uses a continuous top-1 regret-ratio form,

NDCG​@​1=ytop-1maxi⁡yi,\mathrm{NDCG@1}=\frac{y_{\text{top-1}}}{\max_{i}y_{i}},which differs from the binary-relevance NDCG used in link ranking. Because this metric is saturated, simple mean-based baselines can already obtain high scores.

F.4Compute and Budget

The GNN-basedArtifactLinkercheckpoint is∼\sim5M parameters (<100 MB on disk). GNN training runs in <1 hour per configuration on a single NVIDIA A100. Node text embeddings are computed once via the Voyage AIvoyage-3endpoint (dimension 1024) and cached; embedding the 14,053-node graph is a one-time API charge under 20\. LLM\-based baselines \(verification, prediction, ranking\) over the full NLI evaluation suite total <1,000. Closed-source baseline model sizes are not publicly disclosed.

Appendix GDetails of Case Study

Dataset name

We use short names for dataset identifiers in figures and tables. Table3lists the correspondence between each dataset short name and its full dataset ID.

Model name

To improve readability, we use short names for model identifiers in figures and tables. Table4lists the correspondence between each short name and its full HuggingFace model ID.

Table 3:Mapping between NLI dataset short names and HuggingFace dataset identifiers.Table 4:Mapping between model short names and full HuggingFace model identifiers (all 45 evaluated models).