@allen_ai: Most models are only evaluated on a fraction of the benchmarks out there. ArtifactLinker, our new system, predicts whic…
Summary
Allen AI introduces ArtifactLinker, a system that predicts which AI models will achieve state-of-the-art results on HuggingFace benchmarks and then verifies by running evaluations.
View Cached Full Text
Cached at: 05/23/26, 12:03 PM
Most models are only evaluated on a fraction of the benchmarks out there.
ArtifactLinker, our new system, predicts which ones would set a new state-of-the-art on benchmarks hosted on @HuggingFace, then runs the evaluation to verify.
ArtifactLinker is built on a graph of HuggingFace data—models & datasets are nodes, and reported eval scores form the edges.
We trained a GNN for it to rank which models are likely to reach a new SOTA on which benchmarks, beating prompting-based LLMs.
In ArtifactLinker, an LLM coding agent writes and runs the evaluation code, with shared memory across runs.
We found that it comes within 80% of the officially reported score 72.6% of the time.
Using ArtifactLinker, we found cases where a strong model had never been evaluated on a benchmark it would set – or near-match – the SOTA on.
We also found that newer LLMs like Gemma often lose to older DeBERTa models on natural language inference tasks.
We’re releasing a dataset of 14K HuggingFace models, datasets, papers, & codebases linked by 51K evaluations, fine-tunings, & references, plus the ArtifactLinker code.
We hope it helps others find SOTA eval results.
Code: https://github.com/allenai/artifact-linker… Data: https://huggingface.co/datasets/lwaekfjlk/artifact-bench…
Similar Articles
@haofeiyu44: Can we transform the Hugging Face Hub—with its enormous sea of artifacts—into a self-evolving discovery machine? WE CAN…
Introduces ArtifactLinker, a framework that models HuggingFace as an artifact graph and uses GNNs and LLM agents to automatically discover state-of-the-art models and research insights.
I built a live ranking of every AI agent and foundation model (open source)
A developer launched AgentTape, a live ranking site that aggregates data from multiple sources (GitHub, Hugging Face, OpenRouter, etc.) to score and compare public AI agents and foundation models, aiming to provide a more holistic evaluation beyond benchmarks.
Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
Artifact-Bench is a comprehensive benchmark that evaluates multimodal large language models on detecting and analyzing artifacts in AI-generated videos, revealing significant limitations and misalignment with human perception.
@Lyubh22: Coding benchmarks are saturating. AI4Research is the next frontier. Thrilled to see our MLS-Bench (https://mls-bench.co…
Announcing MLS-Bench, the first AI4Research benchmark to gain broad community adoption, testing AI agents on 140 executable tasks across 12 domains to propose modular ML improvements. The post includes leaderboard scores for models like Claude Opus 4.6 and GPT-5.4.
@svpino: I don't trust benchmarks. We've all seen this movie: New model beats everyone else on a benchmark. People hype it. Then…
The tweet critiques traditional AI benchmarks and introduces TRACES, a new benchmark that evaluates AI's discovery process by focusing on how models reach answers, including tool usage, error correction, and evidence tracing.