SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

Hugging Face Daily Papers Papers
benchmark llm-agents scientific-discovery counterfactual-evaluation symbolic-regression language-models

Summary

SciGame 是一个将科研文献转化为结构化「科学卡片」的受控基准，通过原始与反事实环境配对来分离先验知识与实证证据，评估大模型智能体的科学发现能力，发现精确符号定律恢复仍是主要瓶颈。

Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface. SciGame contains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.
Original Article
View Cached Full Text

Cached at: 11/02/26, 11:29 AM

Paper page - SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

Source: https://huggingface.co/papers/2610.37919

Abstract

SciGame evaluates scientific discovery agents using structured cards with counterfactual environments to separate prior knowledge from empirical evidence, revealing that exact symbolic law recovery remains a key challenge.

Large language model agentsare increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduceSciGame, a controlled benchmark that converts scientific literature into structuredscientific cards. Each card yields a pair of Original and dimensionally consistentCounterfactual environmentswith the same task interface.SciGamecontains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second,adaptive experimentationhelps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exactsymbolic formremains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.37919

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.37919 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.37919 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.37919 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

arXiv cs.AI

SciGame is a controlled benchmark that converts scientific literature into structured cards with Original and Counterfactual environments to evaluate LLM agents on scientific law discovery. Experiments on five frontier models show that prior knowledge can hurt performance, adaptive experimentation helps only stronger models, and good explanations can mask quantitative failures.

[2610.08927] Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Reddit r/MachineLearning

该论文提出Station——一个多智能体开源环境，用于评估AI代理进行开放式科学发现的能力。通过添加Supervisor和Meta Reflection机制，Station在重现ICLR oral论文发现的基准上平均达到62.7%，显著超越Codex Multiagent-v2（15.4%）和AI Scientist-v2（14.4-20.6%）。

Rethinking Scientific Discovery in an Agentic Era

arXiv cs.CL

This paper presents SCION, an agentic scientific operating system that integrates AI tools for scientific discovery through a Research Execution Plan (REP) and hierarchical multi-agent execution. It demonstrates applications in materials analysis, molecule design, and protein screening, outperforming existing autonomous research-agent baselines.

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Hugging Face Daily Papers

该论文提出 Station 系统，让 AI 代理在禁用网络访问的条件下自主进行开放式科学发现，在基于三篇 ICLR 口头报告论文构建的任务上平均恢复了 62.7% 的预定义子发现，远超 AI Scientist-v2（20.6%）和 Codex Multiagent-v2（15.4%），并开源了代码与完整研究记录。

Scaling Scientific Discovery Environments for Turn-Level Agentic RL

arXiv cs.AI

This paper introduces SciDisco, a scalable framework for training scientific discovery agents via process-verifiable environments, DAG-grounded trajectory synthesis, and turn-level reinforcement learning. The proposed SciDisco-14B model achieves state-of-the-art on hypothesis-driven scientific data analysis benchmarks.