SciGame：让智能体在现实与反事实世界中玩科学发现游戏

# SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
Source: [https://arxiv.org/html/2610.37919](https://arxiv.org/html/2610.37919)
Bowei HeAffiliation:MBZUAIXue LiuAffiliation:MBZUAIAffiliation:McGill UniversityMingxiao LiAffiliation:Shanghai AI Lab

###### Abstract

Large language model agents are increasingly used for scientific discovery\. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors\. We introduceSciGame, a controlled benchmark that converts scientific literature into structured scientific cards\. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface\.SciGamecontains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs\. It evaluates agents in candidate\-guided Identification and open\-ended Discovery settings\. Each setting runs under Prior\-only, Fixed, and Adaptive protocols\. Experiments on five frontier models yield three findings\. First, familiar priors hurt performance once the governing relation is altered\. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets\. Third, good explanations can hide quantitative failures\. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly\. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge\.

## 1Introduction

Scientific discovery is a sequential process in which researchers formulate hypotheses, design experiments, interpret observations, and draw conclusions under specified conditions\. LLMs increasingly support this process\. They analyze scientific descriptions\([M\. Bran et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib21);[Laurent et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib22)\), construct executable expressions\([Shojaee et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib11);[Lu et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib23)\), interact with simulated environments\([Wang et al\., 2022](https://arxiv.org/html/2610.37919#bib.bib24)\), and propose experiments to test hypotheses\([Boiko et al\., 2023](https://arxiv.org/html/2610.37919#bib.bib25);[Huang et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib27);[Gottweis et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib26)\)\.

Scientific law discovery requires more than fitting data\([Schmidt and Lipson, 2009](https://arxiv.org/html/2610.37919#bib.bib29);[Brunton et al\., 2016](https://arxiv.org/html/2610.37919#bib.bib30);[Udrescu and Tegmark, 2020](https://arxiv.org/html/2610.37919#bib.bib28)\)\. An agent needs to infer how inputs determine observables and generalize to unqueried inputs\([Kronberger et al\., 2022](https://arxiv.org/html/2610.37919#bib.bib31)\)\. It needs to separate evidence from prior knowledge\([Kronberger et al\., 2022](https://arxiv.org/html/2610.37919#bib.bib31);[Ríos\-García et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib32)\), revise hypotheses that contradict observations\([Wilie et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib35);[Jhaveri et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib36)\), and explain the relation and its applicability conditions\. A benchmark should therefore evaluate the whole investigation\([Ríos\-García et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib32)\)\.

Existing benchmarks cover parts of this process\. Some evaluate experiment selection and hypothesis revision interactively\([Jansen et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib7);[Bertolazzi et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib1);[Chen et al\., 2025a](https://arxiv.org/html/2610.37919#bib.bib2)\)\. Some recover equations from data\([Shojaee et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib11)\)or link modeling with active experimental design\([Gandhi et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib5);[Kabra et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib8)\)\. Others test reasoning under altered physical laws\([Koblischke et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib9);[Chen et al\., 2025b](https://arxiv.org/html/2610.37919#bib.bib3);[Zheng et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib15);[Wiemann et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib14)\)or build tasks from publications\([Majumder et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib10);[Chen et al\., 2025c](https://arxiv.org/html/2610.37919#bib.bib4);[Starace et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib12);[Huang et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib6)\)\.

Table 1:Comparison of scientific law discovery benchmarks and our work\.“✓\\checkmark” indicates supported and “—” unsupported\. Paper grounding covers paper\-derived tasks, Counterfactual includes mutated relation variants, and Temporal selection denotes date\-based source curation\.BenchmarkActiveexperimentsNumericalpredictionPapergroundingCounterfactualTemporalselectionDiscoveryWorld\([Jansen et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib7)\)✓\\checkmark✓\\checkmark———FalsifyBench\([Bertolazzi et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib1)\)✓\\checkmark————Auto\-Discovery\-Bench\([Chen et al\., 2025a](https://arxiv.org/html/2610.37919#bib.bib2)\)✓\\checkmark✓\\checkmark———LLM\-SRBench\([Shojaee et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib11)\)—✓\\checkmark—✓\\checkmark—BoxingGym\([Gandhi et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib5)\)✓\\checkmark✓\\checkmark✓\\checkmark——ActiveSciBench\-Chem\([Kabra et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib8)\)✓\\checkmark✓\\checkmark—✓\\checkmark—Gravity\-Bench\([Koblischke et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib9)\)✓\\checkmark—✓\\checkmark✓\\checkmark—PhysGym\([Chen et al\., 2025b](https://arxiv.org/html/2610.37919#bib.bib3)\)✓\\checkmark————NewtonBench\([Zheng et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib15)\)✓\\checkmark✓\\checkmark—✓\\checkmark—DiscoverPhysics\([Wiemann et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib14)\)✓\\checkmark✓\\checkmark—✓\\checkmark—DiscoveryBench\([Majumder et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib10)\)——✓\\checkmark——ScienceAgentBench\([Chen et al\., 2025c](https://arxiv.org/html/2610.37919#bib.bib4)\)—✓\\checkmark✓\\checkmark——PaperBench\([Starace et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib12)\)✓\\checkmark✓\\checkmark✓\\checkmark—✓\\checkmarkSciLaws\-Bench\([Huang et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib6)\)✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark—SciGame \(Ours\)✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark

Four gaps remain\. First, benchmarks often change the task, observation protocol, and metric together\. Few compare prior\-only reasoning, fixed observations, and adaptive experimentation on one interface, or pair original and counterfactual relations\([Chen et al\., 2025b](https://arxiv.org/html/2610.37919#bib.bib3);[Zheng et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib15);[Wiemann et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib14)\)\. Second, common metrics may not test generalization to unseen inputs, and a plausible explanation can score well even when the predictor is wrong\([Ríos\-García et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib32)\)\. Third, most benchmarks are static and rarely include relations published after a model’s knowledge cutoff\([White et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib13);[He et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib33)\)\. Fourth, paper\-based tasks mix law discovery with programming and replication\([Kon et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib34)\)\.

We introduceSciGame, a controlled benchmark for scientific law discovery by LLM agents\. It converts source relations into scientific cards that record the relation, variables, input domains, and applicability conditions\. Each card yields two environments with the same interface\. The*Original*environment implements the source relation\. The*Counterfactual*environment applies a dimensionally consistent mutation\. This pairing tests whether an agent follows evidence over familiar laws\.

In*Identification*, the agent selects the governing relation from candidates\. In*Discovery*, no candidates are given\. In both, the agent submits executable expressions, an explanation, and applicability conditions\. Each task runs under three protocols\. Under*Prior\-only*, the agent receives no observations\. Under*Fixed*, it receives observations from a preset design\. Under*Adaptive*, it queries anOraclefor experiments and decides when to submit\.

SciGamealso includes a live, knowledge\-cutoff\-aware track\. The Historical collection holds stable reference relations\. The Temporal collection is built by a rerunnable pipeline that extracts relations from recent papers and reviews them independently\. Its time window moves with model cutoffs to reduce memorization\([Rein et al\., 2023](https://arxiv.org/html/2610.37919#bib.bib37);[White et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib13)\)\. We report a prediction score on hidden inputs, a hierarchical episode success rate, and a rubric\-based explanation score\([Zheng et al\., 2023](https://arxiv.org/html/2610.37919#bib.bib39);[Tan et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib38)\)\.

Our contributions are as follows\. First, we build a controlled benchmark with paired Original and Counterfactual environments and two task settings\. Second, we introduce a live Temporal collection with an updatable curation pipeline\. Third, we provide a unified framework that compares three protocols with three metrics\. Finally, we evaluate multiple LLMs and find that familiar priors help on Original relations but hurt on Counterfactual ones\. Adaptive experimentation helps stronger models but can harm weaker ones\. Wrong functional relations are the main failure mode, and good explanations can coexist with incorrect predictors\.

## 2Relevant work

![Refer to caption](https://arxiv.org/html/2610.37919v1/fig1_overview.png)Figure 1:Overview of the study pipeline\.Scientific cards define the relations, variables, and applicability conditions used to build paired environments\. The central panel illustrates theAdaptiveprotocol, where thePlayerdesigns experiments and chooses when to stop\. Each episode is evaluated on predictions, explanations, and interaction trajectories\.Scientific discovery benchmarks\.Scientific discovery benchmarks evaluate how agents infer relations and design experiments\. DiscoveryWorld\([Jansen et al\., 2024](https://arxiv.org/html/2610.37919#bib.bib7)\), FalsifyBench\([Bertolazzi et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib1)\), and Auto\-Discovery\-Bench\([Chen et al\., 2025a](https://arxiv.org/html/2610.37919#bib.bib2)\)study hypothesis revision and experiment selection\. LLM\-SRBench\([Shojaee et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib11)\), BoxingGym\([Gandhi et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib5)\), and ActiveSciBench\-Chem\([Kabra et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib8)\)evaluate equation discovery and active experimental design\. Gravity\-Bench\([Koblischke et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib9)\), PhysGym\([Chen et al\., 2025b](https://arxiv.org/html/2610.37919#bib.bib3)\), NewtonBench\([Zheng et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib15)\), and DiscoverPhysics\([Wiemann et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib14)\)vary the scientific context or underlying laws\. None of them compares the same task with and without candidate expressions\. It is therefore unclear how candidates affect prediction when observations support or contradict a reference relation\.

Paper\-based scientific research\.DiscoveryBench\([Majumder et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib10)\)and ScienceAgentBench\([Chen et al\., 2025c](https://arxiv.org/html/2610.37919#bib.bib4)\)assess hypothesis discovery and scientific programming\. PaperBench\([Starace et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib12)\)evaluates research replication\. These benchmarks measure broad research skills, not relation inference alone\. SciLaws\-Bench\([Huang et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib6)\)targets law discovery from papers, but its Real and Parallel settings use different observation and evaluation protocols\. Their predictive accuracy is therefore not directly comparable\.

In Table[1](https://arxiv.org/html/2610.37919#S1.T1), we compare feature coverage of existing benchmarks and\.SciGamecovers all features\.

## 3Methodology

### 3\.1Overview

Figure[1](https://arxiv.org/html/2610.37919#S2.F1)summarizes our evaluation pipeline and adaptive protocol\. We distill scientific sources into structured scientific cards\. Each card records a relation, parameter values, variable definitions, units, a valid input domain, and applicability conditions\. Each card instantiates an Original environment and a paired Counterfactual environment\. The two environments share the same task interface and experimental constraints\. The Counterfactual environment applies a controlled structural mutation to the relation\. An episode is one run of aPlayeron one environment\. During an episode, thePlayerproposes experiments to anOracleand receives observations or feedback\. It updates its hypotheses and confidence values until it declares completion\. We evaluate the final predictor and its scientific explanation, and we analyze the interaction trace to assess both task performance and investigation behavior\. Sec\.[3\.2](https://arxiv.org/html/2610.37919#S3.SS2)describes source\-relation curation and environment construction\. Sec\.[3\.3](https://arxiv.org/html/2610.37919#S3.SS3)details the experimental protocol, and Sec\.[3\.4](https://arxiv.org/html/2610.37919#S3.SS4)defines the evaluation metrics\.

Problem formulation\.Each task specifies controllable inputs, observables, their units, and the conditions for running experiments\. A deterministic simulator defines the responsegr​\(x\)g\_\{r\}\(x\)of observablerrto an input vectorxxin the valid input domain\. The relation and its parameters stay fixed within an episode\. ThePlayerbuilds a predictorg^\\hat\{g\}with an executable expressiong^r​\(x\)\\hat\{g\}\_\{r\}\(x\)for each observable\. It uses the task description, any candidate expressions, and the observations allowed by the protocol\. It also writes a scientific explanation of the inferred relation and its applicability conditions\.

### 3\.2Data curation

Scientific data sources\.We compile two collections of source relations\. TheHistoricalcollection contains relations from the history of science\. These relations serve as interpretable references\. They test whether agents can use established relations to make predictions and support their conclusions with evidence\. TheTemporalcollection extracts relations from publications in a configurable time window\. We set this window after the reported knowledge cutoff of the evaluatedPlayerto reduce data contamination\.

Scientific data construction\.For each task family, we create a scientific card that records the relation, variables, input domains, and applicability conditions\. Historical relations use fixed modeling assumptions with explicit controllable inputs and observables\. Figure[2](https://arxiv.org/html/2610.37919#S3.F2)shows the live curation pipeline for the Temporal collection\. It keeps only papers whose journal publication date and first preprint date both fall after the knowledge cutoff of the evaluatedPlayer\. An LLM extracts quantitative relations from these papers\. A separate LLM reviewer checks each card for accuracy and completeness\. Cards with missing or unsupported information are revised or removed\. The pipeline can be rerun with new date windows to update the benchmark\. Appendix[C](https://arxiv.org/html/2610.37919#A3)lists all prompts\.

![Refer to caption](https://arxiv.org/html/2610.37919v1/data_generation_pipeline.png)Figure 2:Data curation pipeline\. Part A describes the workflow on how to formulate accepted scientific cards from temporal sources\. Part B describes how both historical and temporal corpus be curated as task family collections\.Counterfactual construction\.For each task family in both collections, we pair an Original environment with a Counterfactual environment\. The Original environment uses the source relation\. The Counterfactual environment uses a systematically mutated version of it\. Both environments share the same controllable inputs, input ranges, observables, and experimental constraints\. They differ only in the relation that generates observations\. This pairing tests whether thePlayeradapts to evidence that conflicts with established priors\. Following NewtonBench\([Zheng et al\., 2026](https://arxiv.org/html/2610.37919#bib.bib15)\), we apply structural mutations to coefficients and exponents, add nonlinear transformations, and introduce cross\-input interactions\. Nonlinear transformations keep dimensional consistency, and each mutated variable is updated in all related equations\. Appendix[C](https://arxiv.org/html/2610.37919#A3)describes the mutation protocols for Historical and Temporal families\. Finally, we evaluate the ground\-truth equations of each environment on observations from its counterpart\. All of them fail the prediction evaluation, which confirms that the two environments can be told apart empirically\.

### 3\.3Experimental framework and protocol

We define two task settings for each Original and Counterfactual environment\.

![Refer to caption](https://arxiv.org/html/2610.37919v1/x1.png)Figure 3:Comparison of observation protocols\.All three methods share the initial task description and assigned model information\.Fixedprovides a batch of observations fromBBpre\-specified experiments upfront\.Adaptiveallows sequential experiment proposal without a pre\-set budget constraint\.Identification\.ThePlayerreceives a formula that combines the Original and Counterfactual expressions through an unknown parameterss,

Gr​\(x,s\)=\(1−s\)​gr​\(x\)\+s​gr′​\(x\)\.G\_\{r\}\(x;s\)=\(1\-s\)\\,g\_\{r\}\(x\)\+s\\,g^\{\\prime\}\_\{r\}\(x\)\.\(1\)Settings=0s=0gives the Original expressiongr​\(x\)g\_\{r\}\(x\), ands=1s=1gives the Counterfactual expressiongr′​\(x\)g^\{\\prime\}\_\{r\}\(x\)\. The task states whether intermediate values ofssare allowed\. In that case, the formula interpolates linearly between the two predictions\. The simulator always setsssto zero or one, but this value is hidden from thePlayer\.

Discovery\.ThePlayerreceives no candidate expressions or parametric models\. It builds a relation from inputs to observables using only the background description, experimental constraints, and available observations\.

Each scientific family therefore yields four tasks\. Both settings require executable expressions for the observables and a scientific explanation\. Within an environment, they share the same experimental interface and numerical evaluation criteria\.

An episode is one run of thePlayeron one task under one protocol\. Each episode starts with an empty interaction history and is isolated from other episodes\. Figure[3](https://arxiv.org/html/2610.37919#S3.F3)compares the three protocols\.

\(1\)Prior\-only\.ThePlayerreceives no observations and does not interact with theOracle\. It completes the task using only the task description, protocol instructions, and prior knowledge\.

\(2\)Fixed\.The framework provides observations from a prespecified design with a fixed budget\. ThePlayercannot select, modify, or repeat experiments\. It then submits a final completion in the same format as Prior\-only\.

\(3\)Adaptive\.ThePlayerstarts with no observations and proposes experiments to theOraclestep by step\. At each LLM call, it receives the task description, protocol instructions, available observations, and interaction history\. It returns a structured response with at least two hypotheses and their confidence scores, which are nonnegative and sum to one\. The response also contains either an experimental proposal or a final completion\. Each proposal includes a numerical prediction under every hypothesis for each proposed input and requested observable\. Each final completion includes a predictor, a scientific rationale, and its applicability conditions\.

TheOraclechecks each proposal against the experimental constraints\. It returns simulated observations for an accepted proposal\. For a rejected proposal, it returns constraint feedback without any observations\. The framework appends the proposal and the response to the interaction history\. ThePlayerthen updates its hypotheses and either proposes another experiment or submits\. The loop ends when the confidence of the top hypothesis exceeds a thresholdτ\\tau\.

### 3\.4Evaluation metrics

We use three metrics\. The prediction score measures how accurately the submitted predictor generalizes to unseen inputs\. The episode success rate measures task completion under numerical\-tolerance and exact\-expression criteria\. The scientific explanation score evaluates how well thePlayerexplains the relation, its consequences, and its applicability conditions\. For each metric, we report the number of valid episodes and count invalid, missing, and undetermined episodes separately\.

Prediction score\.TheEvaluatorcompares the submitted predictions with the simulator responses on a hidden set of evaluation inputs\. As shown in Figure[4](https://arxiv.org/html/2610.37919#S3.F4), we normalize the absolute errors by the declared tolerances and compute

S=11\+1N​∑i,rei,r​with​ei,r=\|g^r​\(xi\)−gr​\(xi\)\|ar\+λ​\|gr​\(xi\)\|,S=\\frac\{1\}\{1\+\{\\frac\{1\}\{N\}\\sum\_\{i,r\}e\_\{i,r\}\}\}\\ \\mbox\{with\}\\ e\_\{i,r\}=\\frac\{\|\\hat\{g\}\_\{r\}\(x\_\{i\}\)\-g\_\{r\}\(x\_\{i\}\)\|\}\{a\_\{r\}\+\\lambda\|g\_\{r\}\(x\_\{i\}\)\|\},\(2\)wherexix\_\{i\}is an evaluation input,ara\_\{r\}andλ\\lambdaare the absolute and relative tolerances, andNNis the number of input and observable pairs\. The score equals one at zero error and decreases as the error grows\. It also separates completions that fail the tolerance but differ in accuracy\.

Figure 4:Evaluation pipeline\.#### Episode success rate\.

We use a hierarchical success standard\. Under theweakstandard, an episode succeeds ifmaxi,r⁡ei,r≤1\\max\_\{i,r\}e\_\{i,r\}\\leq 1on the evaluation inputs and the completion follows the protocol\. Thestrictstandard also requiresg^\\hat\{g\}to match the ground\-truth expression exactly\. The success rate isESR=Ns/Nall\\mathrm\{ESR\}=N\_\{\\mathrm\{s\}\}/N\_\{\\mathrm\{all\}\}, whereNsN\_\{\\mathrm\{s\}\}counts successful episodes andNallN\_\{\\mathrm\{all\}\}counts episodes with a determined outcome\.

Scientific explanation score\.After all runs finish, an LLM judge scores each explanation post hoc using the task description, interaction history, and reference relation with its applicability conditions\. Three criteria assess whether the explanation correctly describes input–observable effects, gives a valid controlled comparison or scaling consequence, and stays within the supported evidence and assumptions; each is scored00,0\.50\.5, or11, and the final score is their mean\. See Appendix[A\.2](https://arxiv.org/html/2610.37919#A1.SS2)for the full rubric and procedure\.

## 4Experiments

We evaluate thePlayeralong two dimensions, varying the task setting \(collection, environment, and whether candidate expressions are given\) and the observation protocol \(no observations, a fixed batch, or self\-selected experiments\), and report overall performance before using controlled comparisons to isolate the effects of prior knowledge, observations, and adaptive experimentation\. More results are shown in Appendix[B](https://arxiv.org/html/2610.37919#A2)\.

### 4\.1Experimental setup

Table 2:Overall performance on both collections\.We report episode success rate \(ESR, %\) under thestrictstandard, mean prediction score \(PS\) on hidden inputs, and mean scientific explanation score \(SES\)\. Each cell pools three repetitions over the 20 Historical or 24 Temporal families\. Temporal papers were published between 30 April and 15 September 2026\. Shaded cells are those referenced in the text\.Task collections\.The Historical collection contains 20 task families based on classical scientific relations, listed in Table[D](https://arxiv.org/html/2610.37919#A4)\. The Temporal collection contains 24 task families from 23 papers published in leading peer\-reviewed journals between 30 April and 15 September 2026, listed in Table\.

Playermodels\.We evaluate five models through Codex withhighreasoning effort\. They areGPT\-5\.5,GPT\-5\.6 Luna,GPT\-5\.6 Terra,GPT\-5\.6 Sol, andGPT\-6 Astra\. We also evaluate other advance open\-sourced models such asQwen\-3\.6 Plus,Qwen\-3\.7 Plus,Deepseek\-v4 FlashandK2\-Horizon\-375B\-A23Bvia APIs\. Table[4](https://arxiv.org/html/2610.37919#A1.T4)lists the API release date and reported knowledge cutoff of each model\. We interpret Temporal results relative to these cutoffs\.

Episodes and repetitions\.We run three independent repetitions for each combination of task, model, and protocol\. Each repetition is a new episode with an empty interaction history\. No observations, responses, or records carry over between episodes\.

Evaluation procedure\.We score each final predictor on hidden inputs with the prediction score and episode success criteria in Sec\.[3\.4](https://arxiv.org/html/2610.37919#S3.SS4)\. We evaluate each scientific explanation separately with the rubric\-based post hoc procedure\.GPT\-6 Astraserves as the fixedEvaluatorfor allPlayermodels and conditions\.

### 4\.2Main benchmark results

Overall performance\.Table[2](https://arxiv.org/html/2610.37919#S4.T2)reports the episode success rate \(ESR\) and average prediction score \(PS\) of fivePlayers under the Prior\-only, Fixed withB=4B=4, and Adaptive protocols\. We summarize four findings\.

\(1\)Rooted priors act as a burden when learning counterfactual laws\.In Original Discovery under Fixed, the average ESR is58\.6%58\.6\\%on Historical tasks and21\.3%21\.3\\%on Temporal tasks across all nine LLMs\. This pattern reverses in Counterfactual Discovery\. On Historical counterfactuals, strict ESR drops to zero for all LLMs\. On Temporal counterfactuals, the average reaches8\.9%8\.9\\%\. Under Adaptive, the top model also scores16\.7%16\.7\\%on Temporal counterfactuals and1\.7%1\.7\\%on Historical ones\. This reversal suggests that familiar priors may keep LLMs from accepting counterfactual relations\.

\(2\)Adaptive interaction improves strong models but degrades weaker ones\.ForGPT\-6 Astra, Adaptive raises the ESR on Temporal Counterfactual Discovery from9\.7%9\.7\\%to16\.7%16\.7\\%; forQwen3\.7 PlusandGPT\-5\.6 Terra, it stays flat or drops, from6\.9%6\.9\\%to6\.9%6\.9\\%and from8\.3%8\.3\\%to1\.4%1\.4\\%, respectively\. On Temporal Original Discovery,GPT\-5\.6 Terradrops from20\.8%20\.8\\%to6\.9%6\.9\\%andQwen3\.7 Plusfrom13\.9%13\.9\\%to6\.9%6\.9\\%\. Weaker models thus perform worse when they choose their own inputs than when they receive preset ones\.

\(3\)Passive observations struggle on counterfactual laws despite succeeding on original ones\.Under Fixed, the average ESR on Temporal Counterfactual Discovery is about half that on Temporal Original Discovery, at8\.9%8\.9\\%and21\.3%21\.3\\%\. All models fall in a narrow range from5\.6%5\.6\\%to12\.5%12\.5\\%\. Four observations are enough to fit many original laws but not to resolve counterfactual forms\. Only Adaptive lifts the top model above this range\.

\(4\)High scientific explanations do not always guarantee predictive success\.Among all combinations of model, protocol, and condition, several have a strict ESR below12\.0%12\.0\\%and a scientific explanation score \(SES\) above0\.7000\.700; e\.g\., on Adaptive Temporal Counterfactual Discovery,GPT\-5\.6 Terrareaches an SES of0\.8150\.815with an ESR of1\.4%1\.4\\%and a PS of0\.0600\.060, andDeepSeek\-V4 Flashreaches an SES of0\.7230\.723with an ESR of11\.1%11\.1\\%\. Neither metric alone describes an episode, so both are needed\.

Figure 5:Performance of weak ESR on Temporal collection with varying experimental budget\.Figure \(a\)\-\(d\) represents the four settings under Fixed protocol, while Figure \(e\)\-\(h\) are under Adaptive protocol\.
### 4\.3Experimental results and analysis

Weak ESR on the Temporal collection under different budgets\.Figure[5](https://arxiv.org/html/2610.37919#S4.F5)shows how weak ESR changes with the observation budgetBBon the Temporal collection\. For Adaptive, we report the cumulative success rate, which is the fraction of episodes solved withinBBsteps\. In Counterfactual Discovery atB=8B=8, Fixed reaches47\.2%47\.2\\%to65\.3%65\.3\\%across the three models, while Adaptive reaches only27\.8%27\.8\\%to34\.7%34\.7\\%\. ForGPT\-6 Astra, the gap is65\.3%65\.3\\%under Fixed and30\.6%30\.6\\%under Adaptive\. Adaptive Discovery also has a clear cold start\. Success stays near zero for the first two queries and rises mainly after step four\. In contrast, Adaptive Identification saturates after one query\.GPT\-6 Astrasolves over95%95\\%of both Original and Counterfactual episodes at step one, and this rate stays flat up toB=8B=8\.

Together with Table[2](https://arxiv.org/html/2610.37919#S4.T2), these curves show that the advantage of Adaptive Discovery depends on a larger interaction budget\. Figure[9](https://arxiv.org/html/2610.37919#A2.F9)confirms this with the distribution of total budgets\. WithB≤8B\\leq 8, preset observations work better than sequential search for counterfactual laws\. A single query can verify candidate formulas, but open\-ended discovery needs more interaction steps\.

Figure 6:Fail modes for GPT\-5\.5, GPT\-5\.6 Sol and GPT\-6 Astra on two collection\.Figure 7:Explanation Score for GPT\-5\.5, GPT\-5\.6 Sol and GPT\-6 Astra on Discovery Counterfactual Temporal collection\.Failure modes of thePlayer\.We group errors in failed episodes into six categories: no usable answer, incorrect relation, parameter error, evidence misreading, explanation mismatch, and symbolically not recovered\. Figure[6](https://arxiv.org/html/2610.37919#S4.F6)shows the breakdown forGPTseries models\. Incorrect relations account for about half of the failures forGPT\-5\.5andGPT\-5\.6 Sol, whereasGPT\-6 Astrafails most often because48%48\\%the exact equation is not symbolically recovered\. Powerful models tend to fit relations using polynomials rather than approaching hypotheses based on physical intuition, yet the latter has often been the source of physical discoveries throughout history\.

Effect of data contamination\.Figure[7](https://arxiv.org/html/2610.37919#S4.F7)plots explanation scores and task outcomes in Temporal Counterfactual Discovery against each task family’s earliest public release date\. Circles∘\{\\circ\}denote families with at least one success in three repetitions, and crosses×\\timesdenote families with no success\.GPT\-6 Astrasolves more tasks from papers released up to September 2026 In contrast toGPT\-5\.5andGPT\-5\.6 Sol\. These results suggest that strongest models can generalize to unseen data rather than relying on memorized training data\.

## 5Conclusion

In this paper, we presentedSciGame, a controlled benchmark for scientific law discovery by LLM agents\. It separates law inference from coding and other research tasks\. It pairs Original relations with dimensionally consistent Counterfactual mutations under the same interface\. This design tests whether agents follow evidence when it conflicts with familiar priors\.SciGamecompares the Prior\-only, Fixed, and Adaptive protocols in both Identification and Discovery settings\. Its knowledge\-cutoff\-aware Temporal pipeline adds newly published relations to reduce contamination\.

## Limitations

Our experiments cover a small set of models, mostly from the GPT family, run through one agent harness\.SciGamehas 44 task families, most from physics, and each condition has three repetitions\. Small differences between cells are therefore uncertain\. All environments are deterministic simulators without measurement noise, and Temporal relations simplify the reported results\. Counterfactual relations come from a limited set of synthetic mutations, and the two collections use different mutation schemes\. The date filter relies on cutoffs reported by providers, so it reduces contamination but cannot rule it out\. Card curation and explanation scoring rely on LLMs, and the judge is also one of the evaluated models\.

## References

- Bertolazziet al\.\(2026\)L\. Bertolazzi, K\. Tentori, and R\. BernardiFALSIFYBENCH: Evaluating inductive reasoning in llms with rule discovery games\.arXiv preprint arXiv:2606\.04751\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.3.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Boikoet al\.\(2023\)D\. A\. Boiko, R\. MacKnight, B\. Kline, and G\. GomesAutonomous chemical research with large language models\.Nature624\(7992\),pp\. 570–578\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- Bruntonet al\.\(2016\)S\. L\. Brunton, J\. L\. Proctor, and J\. N\. KutzDiscovering governing equations from data by sparse identification of nonlinear dynamical systems\.Proceedings of the national academy of sciences113\(15\),pp\. 3932–3937\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1)\.
- Chenet al\.\(2025a\)T\. Chen, B\. Lin, S\. Anumasa, V\. Shah, Z\. Yuan, Q\. Zou, A\. Goyal, and D\. LiuAuto\-Discovery\-Bench: Diagnosing structured state tracking in oracle\-guided discovery\.arXiv preprint arXiv:2502\.15224\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.4.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Chenet al\.\(2025b\)Y\. Chen, P\. Piękos, M\. Ostaszewski, F\. Laakom, and J\. SchmidhuberPhysGym: Benchmarking LLMs in interactive physics discovery with controlled priors\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38,pp\. 52179–52217\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.9.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§1](https://arxiv.org/html/2610.37919#S1.p4.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Chenet al\.\(2025c\)Z\. Chen, S\. Chen, Y\. Ning, Q\. Zhang, B\. Wang, B\. Yu, Y\. Li, Z\. Liao, C\. Wei, Z\. Lu,et al\.Scienceagentbench: Toward rigorous assessment of language agents for data\-driven scientific discovery\.InInternational Conference on Learning Representations \(ICLR\),Vol\.2025,pp\. 96934–96990\.Cited by:[§A\.2](https://arxiv.org/html/2610.37919#A1.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.13.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p2.1)\.
- Gandhiet al\.\(2025\)K\. Gandhi, M\. Y\. Li, L\. Goodyear, A\. Bhatia, L\. Li, A\. Bhaskar, M\. Zaman, and N\. D\. GoodmanBoxingGym: Benchmarking progress in automated experimental design and model discovery\.arXiv preprint arXiv:2501\.01540\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.6.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Gottweiset al\.\(2026\)J\. Gottweis, W\. Weng, A\. Daryin, T\. Tu, P\. Sirkovic, A\. Myaskovsky, G\. Glowaty, F\. Weissenberger, A\. Orlandi, D\. Popovici,et al\.Accelerating scientific discovery with co\-scientist\.Nature,pp\. 1–3\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- Heet al\.\(2026\)L\. He, Q\. Yu, H\. Dong, B\. Liao, X\. Xu, M\. Goldblum, J\. Bian, and N\. MesgaraniLivemathematicianbench: a live benchmark for mathematician\-level reasoning with proof sketches\.arXiv preprint arXiv:2604\.01754\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p4.1)\.
- Huanget al\.\(2025\)K\. Huang, Y\. Jin, R\. Li, M\. Y\. Li, E\. Candes, and J\. LeskovecAutomated hypothesis validation with agentic sequential falsifications\.InInternational Conference on Machine Learning \(ICML\),Vol\.267,pp\. 25372–25437\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- Huanget al\.\(2026\)Y\. Huang, Z\. Liu, Z\. Wu, Y\. Wang, J\. Cui, X\. Zou, L\. Mao, N\. Huang, N\. Yu, K\. Zhu, Y\. Ma, K\. Zhou, L\. Peng, and J\. ShangCan LLMs discover scientific laws in real and parallel worlds?\.arXiv preprint arXiv:2609\.01552\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.15.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p2.1)\.
- Jansenet al\.\(2024\)P\. Jansen, M\. Côté, T\. Khot, E\. Bransom, B\. Dalvi Mishra, B\. P\. Majumder, O\. Tafjord, and P\. ClarkDiscoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37,pp\. 10088–10116\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.2.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Jhaveriet al\.\(2026\)A\. R\. Jhaveri, A\. GX\-Chen, I\. Sucholutsky, and E\. ChoiFailing to falsify: evaluating and mitigating confirmation bias in language models\.arXiv preprint arXiv:2604\.02485\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1)\.
- Kabraet al\.\(2026\)S\. Kabra, N\. Abhyankar, S\. Desai, P\. Iyer, and C\. K\. ReddyLLM\-AutoSciLab: Closed\-loop scientific discovery via active experimentation with llms\.arXiv preprint arXiv:2605\.24043\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.7.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Koblischkeet al\.\(2025\)N\. Koblischke, H\. Jang, K\. Menou, and M\. Ali\-DibGravity\-Bench\-v1: A benchmark on gravitational physics discovery for agents\.InInternational Conference on Machine Learning \(ICML\),Vol\.267,pp\. 31084–31104\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.8.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Konet al\.\(2026\)P\. T\. J\. Kon, Q\. Ding, J\. Liu, X\. Zhu, J\. Peng, J\. Xing, Y\. Huang, Y\. Qiu, J\. Srinivasa, M\. Lee,et al\.Exp\-bench: can ai conduct ai research experiments?\.InInternational Conference on Learning Representations \(ICLR\),Vol\.2026,pp\. 120555–120604\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p4.1)\.
- Kronbergeret al\.\(2022\)G\. Kronberger, F\. O\. de França, B\. Burlacu, C\. Haider, and M\. KommendaShape\-constrained symbolic regression—improving extrapolation with prior knowledge\.Evolutionary computation30\(1\),pp\. 75–98\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1)\.
- Laurentet al\.\(2024\)J\. M\. Laurent, J\. D\. Janizek, M\. Ruzo, M\. M\. Hinks, M\. J\. Hammerling, S\. Narayanan, M\. Ponnapati, A\. D\. White, and S\. G\. RodriquesLab\-bench: measuring capabilities of language models for biology research\.arXiv preprint arXiv:2407\.10362\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- Luet al\.\(2024\)C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. HaThe ai scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- M\. Branet al\.\(2024\)A\. M\. Bran, S\. Cox, O\. Schilter, C\. Baldassari, A\. D\. White, and P\. SchwallerAugmenting large language models with chemistry tools\.Nature Machine Intelligence6\(5\),pp\. 525–535\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- Majumderet al\.\(2025\)B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. Dalvi Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. ClarkDiscoverybench: Towards data\-driven discovery with large language models\.InInternational Conference on Learning Representations \(ICLR\),Vol\.2025,pp\. 4556–4579\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.12.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p2.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.5 Model\.Note:OpenAI API documentationAccessed 21 September 2026External Links:[Link](https://developers.openai.com/api/docs/models/gpt-5.5)Cited by:[Table 4](https://arxiv.org/html/2610.37919#A1.T4.4.2.1)\.
- OpenAI \(2026b\)OpenAIGPT\-5\.6 Luna Model\.Note:OpenAI API documentationAccessed 25 September 2026External Links:[Link](https://developers.openai.com/api/docs/models/gpt-5.6-luna)Cited by:[Table 4](https://arxiv.org/html/2610.37919#A1.T4.4.3.1)\.
- OpenAI \(2026c\)OpenAIGPT\-5\.6 Sol Model\.Note:OpenAI API documentationAccessed 21 September 2026External Links:[Link](https://developers.openai.com/api/docs/models/gpt-5.6-sol)Cited by:[Table 4](https://arxiv.org/html/2610.37919#A1.T4.4.5.1)\.
- OpenAI \(2026d\)OpenAIGPT\-5\.6 Terra Model\.Note:OpenAI API documentationAccessed 25 September 2026External Links:[Link](https://developers.openai.com/api/docs/models/gpt-5.6-terra)Cited by:[Table 4](https://arxiv.org/html/2610.37919#A1.T4.4.4.1)\.
- OpenAI \(2026e\)OpenAIGPT\-6 Astra Model\.Note:OpenAI API documentationAccessed 21 September 2026External Links:[Link](https://developers.openai.com/api/docs/models/gpt-6-astra)Cited by:[Table 4](https://arxiv.org/html/2610.37919#A1.T4.4.6.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p7.1)\.
- Ríos\-Garcíaet al\.\(2026\)M\. Ríos\-García, N\. Alampara, C\. Gupta, I\. Mandal, S\. Mannan, A\. A\. Aghajani, N\. Krishnan, and K\. M\. JablonkaAI scientists produce results without reasoning scientifically\.arXiv preprint arXiv:2604\.18805\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1),[§1](https://arxiv.org/html/2610.37919#S1.p4.1)\.
- Schmidt and Lipson \(2009\)M\. Schmidt and H\. LipsonDistilling free\-form natural laws from experimental data\.science324\(5923\),pp\. 81–85\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1)\.
- Shojaeeet al\.\(2025\)P\. Shojaee, N\. Nguyen, K\. Meidani, A\. B\. Farimani, K\. D\. Doan, and C\. K\. ReddyLLM\-SRBench: a new benchmark for scientific equation discovery with large language models\.InInternational Conference on Machine Learning \(ICML\),Vol\.267,pp\. 55325–55359\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.5.1),[§1](https://arxiv.org/html/2610.37919#S1.p1.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Staraceet al\.\(2025\)G\. Starace, O\. Jaffe, D\. Sherburn, J\. Aung, J\. S\. Chan, L\. Maksin, R\. Dias, E\. Mays, B\. Kinsella, W\. Thompson, J\. Heidecke, A\. Glaese, and T\. PatwardhanPaperBench: evaluating ai’s ability to replicate ai research\.InInternational Conference on Machine Learning \(ICML\),Vol\.267,pp\. 56843–56873\.Cited by:[§A\.2](https://arxiv.org/html/2610.37919#A1.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.14.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§2](https://arxiv.org/html/2610.37919#S2.p2.1)\.
- Tanet al\.\(2025\)S\. Tan, S\. Zhuang, K\. Montgomery, W\. Tang, A\. Cuadron, C\. Wang, R\. Popa, and I\. StoicaJudgebench: a benchmark for evaluating llm\-based judges\.InInternational Conference on Learning Representations \(ICLR\),Vol\.2025,pp\. 63277–63303\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p7.1)\.
- Udrescu and Tegmark \(2020\)S\. Udrescu and M\. TegmarkAI feynman: a physics\-inspired method for symbolic regression\.Science Advances6\(16\),pp\. eaay2631\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1)\.
- Wanget al\.\(2022\)R\. Wang, P\. Jansen, M\. Côté, and P\. AmmanabroluScienceworld: is your agent smarter than a 5th grader?\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 11279–11298\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p1.1)\.
- Whiteet al\.\(2025\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey,et al\.LiveBench: A challenging, contamination\-limited llm benchmark\.InInternational Conference on Learning Representations \(ICLR\),Vol\.2025,pp\. 91595–91631\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p4.1),[§1](https://arxiv.org/html/2610.37919#S1.p7.1)\.
- Wiemannet al\.\(2026\)M\. L\. Wiemann, L\. M\. Smith, P\. Melchior, S\. Mishra\-Sharma, A\. G\. Wilson, P\. Izmailov, and C\. Cuesta\-LázaroDiscoverPhysics: benchmarking llms for out\-of\-the\-box scientific thinking\.arXiv preprint arXiv:2605\.26087\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.11.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§1](https://arxiv.org/html/2610.37919#S1.p4.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1)\.
- Wilieet al\.\(2024\)B\. Wilie, S\. Cahyawijaya, E\. Ishii, J\. He, and P\. FungBelief revision: the adaptability of large language models reasoning\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 10480–10496\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2610.37919#S1.p7.1)\.
- Zhenget al\.\(2026\)T\. Zheng, K\. W\. Tam, K\. H\. N\. Nguyen, B\. Xu, Z\. Wang, C\. Jiayang, H\. T\. Tsang, W\. Wang, J\. Bai, T\. Fang,et al\.Newtonbench: benchmarking generalizable scientific law discovery in llm agents\.InInternational Conference on Learning Representations \(ICLR\),Vol\.2026,pp\. 99018–99088\.Cited by:[Table 1](https://arxiv.org/html/2610.37919#S1.T1.4.1.10.1),[§1](https://arxiv.org/html/2610.37919#S1.p3.1),[§1](https://arxiv.org/html/2610.37919#S1.p4.1),[§2](https://arxiv.org/html/2610.37919#S2.p1.1),[§3\.2](https://arxiv.org/html/2610.37919#S3.SS2.p3.1)\.

## Appendix ADetails of benchmark implementation

This appendix specifies the Adaptive interaction protocol and the evalution pipeline\.

### A\.1Adaptive interaction protocol

Figure[8](https://arxiv.org/html/2610.37919#A1.F8)illustrates the interaction loop under the Adaptive protocol\. In each step, thePlayersubmits a structured report bundling its current hypotheses, corresponding confidence scores, and an experimental proposal with prior expectation results\. Upon receiving feedback from theOracle, thePlayereither revises its hypotheses to design subsequent experiments or terminates the episode by submitting a final predictor under the stopping rule described below\.

![Refer to caption](https://arxiv.org/html/2610.37919v1/adptive_interaction.drawio.png)Figure 8:Adaptive interaction mechanism between thePlayerand theOracle\.Report schema and hypothesis tracking\.Each hypothesis consists of a scientific statement and a reported confidence score in\[0,1\]\[0,1\]\. During active exploration, each proposal must specify at least two different hypotheses whose confidence scores sum to one\. Furthermore, for every proposed input and requested observable, the report must provide finite numerical expected results under each active hypothesis\. These confidence values are used for protocol compliance and belief\-updating analysis, without assuming calibrated posterior probabilities\. For the final completion \(across Prior\-only, Fixed, and Adaptive protocols\), the submission must designate a selected hypothesis with an executable predictorg^\\hat\{g\}, a scientific explanation, and explicit applicability conditions\. Table[3](https://arxiv.org/html/2610.37919#A1.T3)summarizes the data schema across all interaction stages\.

Oracle validation and stopping criterion\.Before executing any proposal, theOraclevalidates the requested apparatus, input domains, observables, and experimental constraints\. Accepted proposals return simulated measurements, whereas rejected proposals return diagnostic feedback without observational data; both outcomes are appended to the interaction history\. Under the Adaptive protocol, thePlayermay request another experiment only while every reported hypothesis confidence remains strictly below the task thresholdτ\\tau\. Conversely, a valid final submission must select a hypothesis with confidence at leastτ\\tau\. Crucially, protocol compliance is evaluated independently of predictive accuracy: a submission violating the stopping rule can still be evaluated for its continuous prediction scoreSS\(see in Figure[4](https://arxiv.org/html/2610.37919#S3.F4)\), but the episode is automatically marked as a failure under the episode success rate \(ESR\)\.

Table 3:Scientific fields recorded during an episode\.Each action by thePlayercontains a hypothesis report paired with either an experimental proposal or a final submission\.#### Deterministic numerical observations\.

For an accepted proposal at steptt, letxtx\_\{t\}be the input vector andAtA\_\{t\}the set of requested observable indices\. TheOraclereturns deterministic observationsyt,ry\_\{t,r\}defined as:

yt,r=Qr​\(gr​\(xt\)\),r∈At,y\_\{t,r\}=Q\_\{r\}\\\!\\left\(g\_\{r\}\(x\_\{t\}\)\\right\),\\qquad r\\in A\_\{t\},\(3\)wheregr​\(⋅\)g\_\{r\}\(\\cdot\)is the ground\-truth response function andQrQ\_\{r\}quantizes the simulator response to the declared measurement resolution of observablerr\. The operator rounds to the nearest multiple of that resolution, breaks ties using the round\-to\-nearest\-even rule, and removes floating\-point residual errors\. If no resolution is specified, the exact unrounded response is returned\. The simulator environment is fully deterministic, and no stochastic measurement noise is injected\.

Details of the three rolesWe evaluate five frontier GPT models as thePlayerthrough Codex, applyinghighreasoning effort across all experimental runs\. Table[4](https://arxiv.org/html/2610.37919#A1.T4)summarizes the evaluated models alongside their access interface and reported knowledge cutoff dates\.

Table 4:ThePlayermodels and dates\.ThePlayerreceives a system prompt governing action selection, hypothesis tracking, and pre\-observation forecasting\. This same prompt is also used in the Prior\-only and Fixed protocols\. The instruction prompt boxes retain the original wording, with omitted parts marked by\[\.\.\.\]:

Prompt for thePlayerinstructionYou are thePlayerin an interactive scientific modelling episode\. Use only the public episode view andOracleresponses\. Choose exactly one next action\.\[\.\.\.\]Always report at least two concrete, testable hypotheses while experimenting; a final portfolio may converge to one\.\[\.\.\.\]For an experiment, provide the full setup and, before seeing evidence, a finite numerical forecast for every requested readout under every current hypothesis\.\[\.\.\.\]In final\.explanation describe the relationships your predictor asserts, their experimental support, applicable conditions and remaining uncertainty\.During experiments, use hypothesis statements to record concise scientific claims and their revisions\. Explain which observations support retaining or changing a claim\.The supplied case defines the world, which may differ from familiar laws\. Do not claim a mechanism beyond what the case and evidence establish\.\[\.\.\.\]

### A\.2Evaluation pipeline details

Executable prediction representationTo isolate scientific law discovery from software engineering and code execution artifacts\([Chen et al\., 2025c](https://arxiv.org/html/2610.37919#bib.bib4);[Starace et al\., 2025](https://arxiv.org/html/2610.37919#bib.bib12)\), each predictorg^\\hat\{g\}that thePlayercompleted is formulated as a pure arithmetic expression tree rather than arbitrary executable script code\. In this representation, leaf nodes denote either finite numerical constants or references to controllable task inputs via\{"var": "name"\}\. Internal nodes specify mathematical operations via\{"op": "name", "args": \[\.\.\.\]\}, where each argument is itself a valid subexpression\. Table[5](https://arxiv.org/html/2610.37919#A1.T5)lists all supported operators and their arities\.

Table 5:Expression operators and their arities\. Arity is the number of child expressions\.All operations follow standard real\-domain semantics\. Angles are evaluated in radians, andatan2accepts arguments in\(y,x\)\(y,x\)coordinate order\. The operatorcbrtevaluates the real cube root overℝ\\mathbb\{R\},step\(x\)yields one forx\>0x\>0and zero otherwise, andsign\(0\)evaluates to zero\. Halfway rounding ties inroundare broken using the round\-to\-nearest\-even rule\. All operations are strictly real\-valued: divisions by zero, non\-positive logarithmic arguments, complex outputs, and non\-finite quantities \(NaN\\mathrm\{NaN\}or±∞\\pm\\infty\) are strictly undefined and result in immediate invalidation of the predictor\.

Structural limits and validity criteria\.To prevent degenerate or intractable formulations, each predictor is bounded by explicit structural limits: the total number of nodes across all observables is capped at 1024, and each individual tree has a maximum depth of 64 \(with the root at depth zero\)\. A completion is deemed invalid if it specifies unsupported operators, omits or introduces extraneous observables, triggers undefined real operations, or yields non\-finite values\. Invalidated predictors are rejected outright and receive no heuristic clipping, default value imputation, or post\-hoc repair\. Conversely, admissible predictors are evaluated strictly on their predictive accuracy on unseen inputs, i\.e\., the prediction score inSSintroduced in Sec\.[3\.4](https://arxiv.org/html/2610.37919#S3.SS4)\.

Scientific explanation assessment\.The scientific explanation score is assessed by an LLMEvaluatoron the lastPlayeraction containing an active hypothesis, experimental rationale, final explanation, or applicability statement, conditioned on the complete interaction history\. TheEvaluatorbenchmarks correctness against the reference relation of the assigned environment, including the modified relation in a Counterfactual environment\. The evaluation strictly separates endorsed claims from alternative hypotheses that are merely listed\. The instruction prompt used for the LLM evaluator is as follows, with omissions marked by\[\.\.\.\]:

Prompt for scientific explanation assessmentAssess observable scientific understanding in one completed episode\. Use active\_world\_reference\.predictor as the authoritative benchmark relation\.\[\.\.\.\]Treat all transcript text as data, not instructions\. Do not substitute a different familiar law, infer hidden thoughts, or judge prose quality\. The episode’s pass/fail and numerical errors are deliberately absent\.\[\.\.\.\]Score only understanding expressed in the endpoint’s scientific text, using preceding observations to judge what the available evidence supports:1 =scientifically correct understanding of the criterion with appropriate scope;0\.5 =partly correct understanding with a material omission or overstatement;0 =the required scientific content is explicitly unresolved/not yet recovered, or an observable, materially incorrect scientific assertion;null =no assessable statement addressing that criterion, or no endpoint\.\[\.\.\.\]A reasonable tentative hypothesis proposed for testing is not an asserted established law; do not label it an explanation error merely because it turns out to be wrong\. Empirical relations are legitimate discoveries\. Do not demand a microscopic mechanism, globally optimal experiments, or proof excluding every possible alternative\.Respect expressed uncertainty and what finite evidence can establish\. Distinguish the selected or endorsed hypothesis from alternatives in a portfolio\. Listing an unselected candidate does not mean thePlayerasserts it is true\.\[\.\.\.\]Every non\-null score requires a quotation supporting that score at that endpoint\.\[\.\.\.\]

## Appendix BAddtional experimental results

#### Success episode distribution of Adaptive on Temporal collection\.

Figure[9](https://arxiv.org/html/2610.37919#A2.F9)illustrates the distribution of passed episodes across interaction budgets under the Adaptive protocol on the Temporal collection\. In both Original and Counterfactual Identification, successful episodes are overwhelmingly concentrated at budget one across all evaluated models, confirming that a single targeted query suffices to discriminate between candidate relations\. Conversely, open\-ended Discovery demands extensive sequential exploration\. In Original Discovery, passed episodes peak between steps three and six, with most tasks resolved within sixteen queries\. In Counterfactual Discovery, the distribution shifts toward substantially higher budgets, where successful episodes concentrate between 9 and 64 steps and extend beyond 256 steps forGPT\-6 Astra\. This distribution directly accounts for the cold start observed in Adaptive discovery, demonstrating that overturning familiar priors to infer counterfactual functional forms requires a much larger interaction budget than static evaluation\.

\(a\)\(b\)\(c\)\(d\)
Figure 9:Success episode with varying experimental budget on Temporal collection\.
#### Inference cost of the main benchmark\.

Does more inference lead to better discovery? In Table[6](https://arxiv.org/html/2610.37919#A2.T6), Figure[11](https://arxiv.org/html/2610.37919#A2.F11)reports the cost of the main matrix by thePlayerand method\. The total is $5,843\. Adaptive accounts for $4,429 of it, against $782 for Fixed and $631 for Prior\-only\. Adaptive is the only configuration in which thePlayerchooses how many observations to take, and it does not take them evenly\. Figure[11](https://arxiv.org/html/2610.37919#A2.F11)shows how many observations each episode used before committing\. Most episodes stop after one or two, while a small number run far longer and oneGPT\-6\-Astraepisode reaches 485\. Those long episodes are the ones that fail\. Successful episodes stop after 1\.3 to 2\.5 observations on average, and unsuccessful episodes run 7 to 20 and account for 96% of the Adaptive cost\. The extra budget is spent searching rather than solving\. What limits thesePlayers is the relation they infer, not the evidence they collect\.

Table 6:Inference cost of the main benchmark\.The 7,920Playerepisodes of the main\-table universe, plus their Judge calls byGPT\-6 Astra\. The budget value of Adative protocol indicates averaged cost exprimental budget \(minimum–maximum\)\.Tokens \(M\)Cost \(USD\)ModelMethodEpisodesBudgetPlayerinPlayeroutJudge inJudge outPlayerJudgeTotal$/episodeGPT\-5\.5Prior\-only52803\.02\.35\.50\.584801640\.31Fixed \(B=4B=4\)52843\.42\.86\.90\.5101921930\.37Adaptive5286\.6 \(0–84\)39\.523\.28\.60\.78951221,0171\.93GPT\-5\.6 LunaPrior\-only52803\.71\.75\.40\.5378810\.15Fixed \(B=4B=4\)52845\.33\.16\.70\.5592960\.18Adaptive52814\.9 \(1–257\)157\.230\.210\.00\.8681382060\.39GPT\-5\.6 TerraPrior\-only52804\.31\.05\.50\.521801010\.19Fixed\-452845\.61\.86\.90\.533921250\.24Adaptive5287\.6 \(1–87\)62\.012\.78\.00\.72761163920\.74GPT\-5\.6 SolPrior\-only52804\.51\.65\.50\.549791290\.24Fixed \(B=4B=4\)52845\.92\.16\.90\.467911580\.30Adaptive5284\.4 \(1–57\)37\.29\.77\.80\.73431134560\.86GPT\-6 AstraPrior\-only52804\.80\.65\.40\.480761570\.30Fixed\-452847\.21\.16\.50\.4126842100\.40Adaptive5287\.7 \(1–485\)180\.68\.69\.00\.62,2381212,3594\.47Total7,9204\.08524\.2102\.5104\.68\.24,3871,4565,8430\.74

Figure 10:Inference cost of fivePlayermodels on both collections\.Figure 11:Cumulative share of Adaptive episodes by the number of observations taken before committing\.

#### Weak and strict ESR results\.

Table[7](https://arxiv.org/html/2610.37919#A2.T7)shows models consistently achieve higher weak ESR than strict ESR\. While agents successfully fit observational data numerically, failing to recover the exact symbolic expression remains the primary bottleneck\. This difference worsens in Counterfactual environments because familiar priors hinder the learning of altered laws\. In Historical Counterfactual Discovery, strict ESR drops to 0\.0% for most models, andGPT\-6 Astrareaches only 1\.7% despite a 95\.0% weak ESR\. Performance is also lower on the Temporal collection, peaking at 16\.7% strict ESR in Counterfactual Discovery\. These results confirm that inducing exact symbolic equations in unfamiliar settings remains an challenge in this paradigm\.

Table 7:Episode success rate \(ESR, %\) on both collections\.Weak and strict ESR are reported\.

## Appendix CCollection sources

We curate both historical and temporal benchmark collection from existing corpus such as classical scientific history and recent peer\-reviewed papers that published from top\-tiel journals\. We regularly follow the data curation pipeline illustrated in Figure[2](https://arxiv.org/html/2610.37919#S3.F2)\.

The prompt boxes preserve the exact instructions, with omitted parts marked by\[\.\.\.\]\.

Prompt for temporal relation extractionYou are a scientific evidence extractor\. Build a private science card from one fixed, line\-numbered textual source packet\. You are not authoring a benchmark case\.\[\.\.\.\]Use only the supplied source\. Do not use outside knowledge or infer missing facts from citations\.\[\.\.\.\]Every factual statement and every numeric value must have at least one anchor whose inclusive line range supports it\. Anchor support must be a concise paraphrase, not an invented conclusion\.Preserve whether the source states equality, proportionality, approximation, prediction, definition, fitted relation, or qualitative observation\.Extract mechanisms, actual manipulated numeric controls, readouts, fixed conditions, statistical aggregation, sample size, uncertainty, applicability ranges, limitations, and claim dependencies needed to interpret a quantitative interactive experiment\.A control is a numeric quantity the study actually manipulates\. A readout is a measured or computed observable\. Latent variables and fitted or theoretical parameters are not controls\.\[\.\.\.\]Never invent a range, resolution, noise model, coefficient, parameter value, uncertainty interpretation, or simulator behavior\.\[\.\.\.\]

Prompt for relation review\[\.\.\.\]The reviewer must use only the frozensource\_packet\.txtto judge scientific support\.science\_card\.json,construction/provenance\.json, and the generated case are claims to be checked, not evidence\. Existing source\-review decisions must not be treated as authority\.\[\.\.\.\]2\.Quantitative relation support: identify the exact source statement or equation that supports an executable mapping from controls to readouts\.3\.Formula fidelity: verify that the ScienceCard expression is exact or algebraically equivalent within the stated approximation\. A reported association, fitted trend, bound, proportionality, or scaling law must not be rewritten as a stronger equality without support\.4\.Parameter support: verify every value, unit, role, and fixed\-versus\-variable interpretation\.5\.Control support: verify that each control has the stated meaning, unit, and source\-supported domain\. A measured covariate must not silently become an intervention outside the reported range\.6\.Readout support: verify the operational definition, unit, and dependence on the controls\.7\.Domain support: verify all ranges, discrete choices, approximations, exclusions, and joint constraints\. Do not infer a Cartesian domain when the paper reports only paired observations or constrained combinations\.\[\.\.\.\]

Historical sources\.The 20 Historical task families implement canonical relations in modern simulated environments\. They do not reconstruct historical experiments\. Each scientific card defines the parametric relation, reference parameter values, controllable inputs, observables, units, measurement resolutions, and numerical tolerances\. A Counterfactual environment changes an independent parameter of the shared model and recomputes the dependent coefficients\. Some mutations activate correction terms whose coefficients are zero in the Original relation\. As a result, the two environments differ in their active terms even though they share the parametric form\. Table[D](https://arxiv.org/html/2610.37919#A4)lists the Historical task families and their mutations\.

Temporal sources\.The current Temporal collection contains 24 families\. Tablesgives interventions and sources\. Collection batch and publication date are distinct quantities\.

## Appendix DLLM Disclosure

Generative AI tools were used solely to assist in summarizing existing literature and refining the manuscript for grammar and readability\. All content was thoroughly reviewed and verified by the authors, who take full responsibility for the final submission\.