CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

# CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Source: [https://arxiv.org/html/2607.08093](https://arxiv.org/html/2607.08093)
Andrej Leban Department of Statistics University of Michigan Ann Arbor, MI, United States leban@umich\.edu&Yuekai Sun Department of Statistics University of Michigan Ann Arbor, MI, United States yuekai@umich\.edu

###### Abstract

Large language models \(LLMs\) increasingly act as integrated*data\-science agents*, combining abstract reasoning with advanced tool use\. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysisordata analysis benchmarks without a principled causal data\-generating structure\. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures\. We introduceCausalDS, a benchmark for evaluating*causal reasoning in agentic data\-science workflows*\. Each benchmark instance is a*scene*consisting of a sampled structural causal model \(SCM\) with generated observational data and an accompanying synthetic natural\-language story grounded in a realistic domain\. We optionally ground thecompositionof the benchmark components in empirical distributions obtained from real\-world datasets, thus retaining empirical structure while reducing the “causal parrot” risk through completely synthetic generation\. From each scene, we then derive tasks spanning all three of Pearl’s rungs, with typical data\-science prediction tasks appearing as Rung 1\. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model\. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first\-class scored outcome\. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding\.

## 1Introduction

Modern LLMs are increasingly powerful in agentic settings and are routinely used in data\-science workflows\(Jinget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib33); Chanet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib43); Guet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib79); Majumderet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib80)\)\. Their actual*causal*reasoning capabilities, however, remain contentious\(Zecevicet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib15); Jinet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib24)\)\. In realistic causal data science, the relevant task is not merely to answer a causal question in text\. An analyst must interpret a domain description, reason about the implied causal structure, inspect observational data, decide what is identifiable, and then either estimate the target quantity or decline to answer when the available information is insufficient\.

CausalDS111[github\.com/andleb/causalds](https://github.com/andleb/causalds)evaluates this setting along five axes that are usually tested separately\. The first is symbolic causal reasoning: interpreting a causal scenario, reasoning over graph structure, and distinguishing associational, interventional, and counterfactual targets\. The second is data\-science execution: using tabular data and standard analysis tools to produce estimates and predictions\. The third is uncertainty quantification: attaching calibrated uncertainty to those estimates\. The fourth is epistemic abstention: recognizing when the requested causal claim is not warranted by the released data and assumptions\. The fifth is tool use and coding: carrying the analysis out through code in an agentic, file\-backed environment\. These axes are separable in principle, but realistic causal analysis requires their interaction\.

Existing evaluations tend to isolate parts of this problem\. Data\-science agent benchmarks stress coding and open\-ended analysis but usually lack a hidden causal data\-generating structure\(Laiet al\.,[2022](https://arxiv.org/html/2607.08093#bib.bib9); Jinget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib33); Chanet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib43); Qianget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib35); Guet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib79); Majumderet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib80)\)\. Causal reasoning benchmarks test graph reasoning, intervention, or counterfactual logic, but often do so on a purely symbolic level\(Jinet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib24); Shethet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib44); Chenet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib41); Zhouet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib34); Duet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib1)\)\. Benchmarks built from published datasets or familiar causal examples add realism, but this can make it difficult to guarantee novelty and avoid contamination: models may rely on*amortized causal inference*rather than genuine structure\-sensitive reasoning — the “causal parrot” failure mode\(Jinet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib24); Zecevicet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib15)\)\. Concurrent structure\-preserving probes make this concern concrete: merely anonymizing the semantic variable names, with the causal task held fixed, sharply reduces causal\-benchmark accuracy\(Yu and Zhou,[2026](https://arxiv.org/html/2607.08093#bib.bib159)\)\. Concurrent interactive\-discovery environments address another part of the gap by sampling hidden SCMs and letting agents intervene\(Yanget al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib156)\); they target experimental mechanism recovery rather than causal data\-science analysis over released observational files\.

CausalDS, on the other hand, addresses this gap by generating synthetic causal data\-science scenes with private SCM\-derived ground truth, natural\-language problem descriptions, tabular data, and deterministic scoring\. We evaluate contemporary agents on a realistically grounded CausalDS exam and find that the axes do not collapse to a single capability: models differ in content correctness, uncertainty quantification, abstention recognition, and tool\-use efficiency\.

##### Main Contributions

- •SCM\-grounded synthetic scenes with empirically anchored composition\.We generate hidden causal graphs, instantiate SCMs, synthesize observational data, and compute private ground truth for evaluation\. The scene generator is fully synthetic, while benchmark composition can be anchored along empirical axes such as variable type, graph structure, identifiability, mechanism profile, and observation complexity\.
- •Graph\-faithful free\-form verbalization\.CausalDS maps abstract graph nodes to coherent domain variables \(partially seeded from CauseNet\(Heindorfet al\.,[2020](https://arxiv.org/html/2607.08093#bib.bib4)\)\) and generates natural\-language stories describing the resulting causal setting\. Both the variable mapping and the final story are audited against the hidden graph and overall narrative coherence, so that the benchmark can combine realistic prose with controlled causal structure\.
- •A separate observation layer for data\-analysis difficulty\.We distinguish the conceptual SCM from the data view released to the agent\. In addition to clean observations, CausalDS can replace conceptual variables with bundles of noisy observations of the latter\. This lets the benchmark vary the data\-science difficulty without changing the causal aspects, such as the target causal estimand or its identifiability status\.
- •A broad causal data\-science task suite\.From each scene, CausalDS derives tasks across Pearl’s hierarchy, including prediction, association, graph recovery, identification, effect estimation, bias diagnostics, and counterfactual reasoning\. Because tasks share a hidden SCM, they connect language interpretation, statistical estimation, and formal causal reasoning\.
- •Abstention\-aware deterministic evaluation\.CausalDS scores submitted answers against hidden ground truth, including deliberately non\-identifiable causal queries where the correct behavior is to abstain\. We evaluate contemporary agents on an exam compositionally grounded in real\-world corpora and show that model performance dissociates along the five axes above\.

## 2Background

We briefly recall the formalism the benchmark relies on, the identifiability questions it tests, and what we mean by a*data\-science agent*\.

##### Causal graphs, SCMs, and Pearl’s hierarchy\.

A structural causal model \(SCM\) over a directed acyclic graph \(DAG\)GGon variablesV=\(V1,…,Vn\)V=\(V\_\{1\},\\dots,V\_\{n\}\)assigns each node astructural equationVi=fi​\(Pai,Ui\)V\_\{i\}=f\_\{i\}\(\\mathrm\{Pa\}\_\{i\},U\_\{i\}\)from its graph parentsPai\\mathrm\{Pa\}\_\{i\}and an exogenous disturbanceUiU\_\{i\}; the joint distribution of the\{Ui\}\\\{U\_\{i\}\\\}together with the mechanisms\{fi\}\\\{f\_\{i\}\\\}induces the observational distributionP​\(V\)P\(V\)\(Pearlet al\.,[2021](https://arxiv.org/html/2607.08093#bib.bib65)\)\. The same SCM defines interventional distributionsP​\(V∣do​\(X=x\)\)P\(V\\mid\\mathrm\{do\}\(X=x\)\)obtained by replacingfXf\_\{X\}with the constantX=xX=x, as well as counterfactual quantities that compare multiple “parallel worlds” sharing the same exogenous draw ofUU\. We use Pearl’s three\-rung hierarchy as the organizing axis of the task suite:*Rung 1*\(associational –P​\(V\)P\(V\), which includes usual data science tasks such as prediction\),*Rung 2*\(interventional,P​\(Y∣do​\(X\)\)P\(Y\\mid\\mathrm\{do\}\(X\)\)\), and*Rung 3*\(counterfactual\)\.

##### Effects and Identifiability\.

A causaleffectis*identifiable*fromP​\(V\)P\(V\)givenGGwhen the corresponding interventional functional can be expressed in observational terms alone\. Pearl’s do\-calculus together with the ID algorithm ofShpitser and Pearl \([2008](https://arxiv.org/html/2607.08093#bib.bib14)\)gives a complete decision procedure; classical sufficient conditions include the back\-door criterion and the front\-door criterion\(Pearlet al\.,[2021](https://arxiv.org/html/2607.08093#bib.bib65)\)\. At Rung 2 the target is the*population*average treatment effect \(ATE\),𝔼​\[Y∣do​\(X=x1\)\]−𝔼​\[Y∣do​\(X=x0\)\]\\mathbb\{E\}\[Y\\mid\\mathrm\{do\}\(X\{=\}x\_\{1\}\)\]\-\\mathbb\{E\}\[Y\\mid\\mathrm\{do\}\(X\{=\}x\_\{0\}\)\], and effect\-estimation tasks are*identification\-gated*: the agent must first decide whether this population estimand is identifiable from the conceptual observational law under the story\-implied graph, and abstain when it is not—even though a finite observational sample is always provided\.

Recognizing the non\-identifiable case is itself a benchmark\-relevant skill, and the CausalDS scoring rules \(Sec\.[3\.7](https://arxiv.org/html/2607.08093#S3.SS7)\) treat abstention on non\-identifiable estimands as a first\-class outcome — non\-identifiable problems can appear in all settings \(cf\. Sec\.[3\.4](https://arxiv.org/html/2607.08093#S3.SS4)\) where non\-identifiability is a possibility\. Counterfactual reasoning \(Rung 3\) yields estimands that Rung 2 effects cannot express, including the*effect of treatment on the treated*\(ETT\)𝔼​\[Yx1−Yx0∣X=x1\]\\mathbb\{E\}\[Y\_\{x\_\{1\}\}\-Y\_\{x\_\{0\}\}\\mid X=x\_\{1\}\]and Pearl’s*natural direct*and*natural indirect*effects \(NDE, NIE\) for mediation analysis\(Pearlet al\.,[2021](https://arxiv.org/html/2607.08093#bib.bib65); Pearl,[2022](https://arxiv.org/html/2607.08093#bib.bib72)\)\. Identifiability for these estimands is governed by theID\*/IDC\*algorithms\(Shpitser and Pearl,[2008](https://arxiv.org/html/2607.08093#bib.bib14)\)\. In particular: a graph for which a Rung 2 effect, e\.g\. the average treatment effect \(ATE\), is identifiable need not yield an identifiable ETT, NDE, or NIE\. CausalDS therefore tracks Rung\-3 identifiability separately from Rung\-2 identifiability\. Identifiability is always relative to the conceptual variables, as opposed to their noisy observations, which are discussed in Sec\.[3\.2](https://arxiv.org/html/2607.08093#S3.SS2)\.

##### Data\-science agents\.

By a*data\-science agent*we mean an LLM that interacts with a sandboxed environment by reading benchmark\-provided files, executing code, inspecting intermediate outputs, and writing answer files\. Each CausalDS task therefore mixes language understanding, code\-driven estimation, and basic tool use\.

## 3The CausalDS Benchmark

A CausalDS dataset instance is a*scene*consisting of a narrative story, a tabular dataset, a lightweight data schema, and a list of tasks; ground truth quantities and a hidden test split are stored separately for scoring\. Fig\.[1](https://arxiv.org/html/2607.08093#S3.F1)sketches the generation pipeline, executed in order: \(i\) sample a causal graph, optionally enlarged by anchor\-basedgraftingof auxiliary motifs through shared anchor nodes; \(ii\) instantiate a structural causal model under typed continuous and binary mechanism profiles, generate observational data \(Sec\.[3\.1](https://arxiv.org/html/2607.08093#S3.SS1)\), optionally using an additionalobservation model\(Sec\.[3\.2](https://arxiv.org/html/2607.08093#S3.SS2)\); \(iii\) map nodes to variable names grounded in a plausible domain and generate a verified narrative story \(Sec\.[3\.3](https://arxiv.org/html/2607.08093#S3.SS3)\); \(iv\) compute tasks and ground truth \(Sec\.[3\.4](https://arxiv.org/html/2607.08093#S3.SS4)\); and \(v\) package public and private scene artifacts \(Sec\.[3\.5](https://arxiv.org/html/2607.08093#S3.SS5)\)\. Larger production runs use a blueprint\-driven path in which a*composition configuration*specifies distributions over composition axes: motif, graft complexity, identifiability regime, treatment/outcome type, SCM profile, and released observation model variants\. For benchmarking, the agent receives only the public scene directory and writes answer files; the grader separately loads the private artifacts and scores deterministically\. Fig\.[2](https://arxiv.org/html/2607.08093#S3.F2)presents an example of ascene\.

Hidden causal worldGrounded language layerBenchmark artifacts and scoringGraph sampler motif or graftedSCM \+ data mechanisms \+ effects \+ proxies \+ calibrationVariable mapping ground via CauseNet \+ auditStory generation natural\-language scene \+ verify & refineTasks \+ packaging R1/R2/R3 prompts \+ ground truth \+ file splitPublic bundle story, schema, data, tasks, test featuresPrivate bundle ground truth, labeled test splitAgent \+ grader agent writes answers to file, deterministic score123456

Figure 1:CausalDS pipeline: from hidden causal structure to benchmark execution\.Scene example\(R2 bias diagnostic, forbidden\-controls output variant;ivmotif; hard observation variant\)\.Story\.Farmers in a semi\-arid region receive an Irrigation Activation Subsidy \(dollars per hectare\) that economically incentivizes whether an Irrigation Event is Executed on their cropland\. This decision also depends on the Subsurface Soil Permeability—an unobserved property \(cm/hour\) that remains latent to agricultural economists but governs how water infiltrates the soil: a sufficient subsidy together with permeable subsoil switches the binary Irrigation Event Executed to 1, while an insufficient subsidy or impermeable subsoil yields 0\. The Field Soil Erosion Rate \(tons per hectare per year\) is the outcome: executing an irrigation event mechanically disturbs and displaces topsoil and raises erosion, while the Subsurface Soil Permeability exerts its own direct effect—higher permeability minimizes surface runoff and soil loss, lower permeability intensifies it\. The observed erosion rate thus reflects both the irrigation intervention and the unobserved soil hydraulics\.Public data\.Z1,…,Z3Z\_\{1\},\\dots,Z\_\{3\}are noisy measurements of the conceptual variable Irrigation Activation Subsidy;X1,…,X3X\_\{1\},\\dots,X\_\{3\}are noisy measurements of the conceptual variable Irrigation Event Executed\.YZ\_1Z\_2Z\_3X\_1X\_2X\_30\.42\.20\.20\.5\-2\.202\.4\-0\.8\-2\.4\-0\.9\-0\.3\-1\.511\.64\.02\.9\-0\.30\.7\-0\.203\.0Private ground\-truth DAG\. XXZZYYUUZZ=irrigation subsidy \(instr\.\),XX=irrigation exec\. \(treat\.\),YY=soil erosion,UU=soil permeability \(latent\)\.Question\. For estimating the ATE of Irrigation Event Executed on Field Soil Erosion Rate by covariate adjustment, which observed variables must*not*be conditioned on? Return the forbidden set\{"forbidden":\[…\]\}, or"no\_backdoor"if no valid adjustment set exists but the ATE is otherwise identifiable, or"non\_id"if the ATE is not identifiable\.

Figure 2:A CausalDSscene: story, public dataset with observed measurements, hidden DAG, and the posed question\.### 3\.1Graphs and SCM\-generated tabular data

We sample DAGs starting from canonicalmotifs\(chain, fork, confounding, and others; full catalog is reproduced in App\.[A\.5](https://arxiv.org/html/2607.08093#A1.SS5), Fig\.[5](https://arxiv.org/html/2607.08093#A1.F5)\)\. To assemble larger, dynamic graphs, we additionally support*anchor\-based grafting*: after drawing and verbalizing a coherent variable mapping for themain graph, we attach one or more smallauxiliary motifsthrough exactly one shared node — the anchor — each\. The shared node is the only overlap between the existing graph and the new fragment\. When grafting is active, we restrict both the main graph and the auxiliary graph motif pools toward motifs that remain stable under grafting\. Anchor grafting thus allows increasingly complex structures while preserving narrative coherence by the use of the shared \(already verbalized\) anchor, and limiting the variable mapping problem to isolated subgraphs\. Some motifs include latent variables \(e\.g\., unobserved confounders\), enabling stress tests for identifiability and adjustment reasoning\. Graph properties are calibrated against empirical histograms from real\-world causal datasets \(App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)\)\. This is similar in principle to approaches such as Nemotron Personas\(Meyer and Corneil,[2025](https://arxiv.org/html/2607.08093#bib.bib155)\); to our knowledge, this is the first application in this type of benchmark\.

Given the resulting DAG, we instantiate a structural causal model \(SCM\) supporting a mix of continuous and binary variables, with mechanisms organized into typed*SCM profiles*: each scene is assigned acontinuous\_scm\_profileand abinary\_scm\_profiledrawn from independent registries, so mechanism choice is settled marginally per node type\. Each registry mixes*empirical profiles*whose mean\-function families derive from real biochemical\-equation and Boolean\-rule corpora with*synthetic profiles*that exercise additional regimes \(e\.g\., neural network mixture\-noise, sharp\-threshold, etc\.\) for additional diversity; noise is kept additive \(continuous\) or single\-link \(binary\) so that the same SCM remains suitable for Monte Carlo evaluation of treatment effects, mediation, and counterfactuals\. The full registry, mechanism families, and noise options are listed in App\.[A\.2](https://arxiv.org/html/2607.08093#A1.SS2)\.

### 3\.2Observation layer

A central design choice in CausalDS is to distinguish*causal structure*from*measurement*\. After the conceptual DAG is fixed, the public data table may withhold a small subset of conceptual variables \(typically causally important nodes such as treatments, outcomes, mediators, or confounders\) and replace each with a bundle of noisy measurements\. Measured conceptual variables are distinct from true latent SCM nodes: hidden confounders are unobserved everywhere, whereas measured variables remain part of the narrated graph, but are observed directly only in a smallcalibrationsample provided to the model\.

Formally, for a selectedZjZ\_\{j\}the public table containsWj=\(Wj​1,…,Wj​dj\)W\_\{j\}=\(W\_\{j1\},\\dots,W\_\{jd\_\{j\}\}\), with continuous measurementsWj​r=hj​r​\(Zj\)\+ϵj​rW\_\{jr\}=h\_\{jr\}\(Z\_\{j\}\)\+\\epsilon\_\{jr\}and binary measurementsWj​r∣Zj∼Bernoulli​\(pj​r​\(Zj\)\)W\_\{jr\}\\mid Z\_\{j\}\\sim\\mathrm\{Bernoulli\}\(p\_\{jr\}\(Z\_\{j\}\)\)\. Because the underlying SCM is unchanged, the conceptual causal estimands are preserved\. Causal identifiability is always evaluated on the conceptual graph: the observation layer is a calibrated*measurement*layer that changes how hard it is to*numerically estimate*an identified functional\. Each observation model bundle has a single conceptual parent, so the observation model never creates or removes a confounder, instrument, mediator, etc\., and never flips an identifiability label\.

This is distinct from proximal causal inference\(Tchetgenet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib136); Miaoet al\.,[2018](https://arxiv.org/html/2607.08093#bib.bib132)\)\. Proximal causal inference addresses causal\-effect identification when relevant confounding mechanisms are not directly observed, using auxiliary measurements or negative controls under additional assumptions about their relation to the unobserved confounding structure\. InCausalDS, the observation model has no such role: the conceptual DAG and SCM are fixed before the observation layer is applied, and each measurement bundle is generated as a noisy observation ofoneconceptual variable\. These released columns are not additional causal variables in the conceptual graph and cannot provide a new route to identification as they cannot create or remove confounding, instruments, or mediators\. Thus, the observation layer changes estimation difficulty from the released files, not the conceptual estimand or its identifiability\.

In implementation, several observation variants are released per scene; for example—clean\(no measurement layer\),proxy, andproxy\_hard\(more measured non\-outcome nodes, stronger corruption, smaller calibration split\)\. When generating the variants, candidates are screened by the bundle Fisher informationIj​\(z\)=∑rIj​r​\(z\)I\_\{j\}\(z\)=\\sum\_\{r\}I\_\{jr\}\(z\)across the realized support ofZjZ\_\{j\}to ensure that the measurement does not corrupt the value to the extent of making inference impossible\. Admissibility thresholds, recoverability diagnostics, and resulting private observation diagnostics are provided in App\.[A\.6](https://arxiv.org/html/2607.08093#A1.SS6)\.

### 3\.3Mapping variables and story generation

Abstract node identifiers are mapped to domain\-relevant variable names \(e\.g\.,v1→\\toExercise\) using an LLM\.222Kimi 2\.5 for the data used to generate the results presented\.Optionally, a knowledge\-grounding step seeds names for a subset of nodes using CauseNet\(Heindorfet al\.,[2020](https://arxiv.org/html/2607.08093#bib.bib4)\), inducing the synthetically generated remaining nodes to conform to them\. Thus, we introduce real\-world diversity into the entire verbalization\. In the results presented in this work, we adopted a policy of seeding the outcome node and one of its neighbors\. Because this step injects recognizable real\-world causal pairs into the prose, we verify that answers do not hinge on the particular pairs seeded: a matched verbalization\-swap ablation that holds the graph, SCM, data, task, and private ground truth fixed while varying only the seeded story and variable names shows stability for strong\-enough models \(App\.[A\.12](https://arxiv.org/html/2607.08093#A1.SS12)\)\.

To reduce semantic drift and “accidental edges” implied by naming choices, the mapping is then refined iteratively using amapper\-auditorloop\. For grafted graphs, we apply the same mapper/auditor machinery stage\-wise: the main graph is mapped first; each auxiliary fragment is then grounded separately with the shared anchor immutable and duplicate names forbidden; finally the merged full graph is audited globally\. Given the renamed graph and the matching proposed domain, an LLM generates a narrative that describes the variables and their causal relations\. The draft is checked by another auditor and revised iteratively until it passes\. The generation\-time LLM clients can also use web search to pin down realistic variable names, units, and story details\. Alg\.[1](https://arxiv.org/html/2607.08093#alg1)summarizes the top\-level control flow\. The subroutines invoked by Alg\.[1](https://arxiv.org/html/2607.08093#alg1)are listed in App\.[A\.3](https://arxiv.org/html/2607.08093#A1.SS3); further details are in App\.[A\.4](https://arxiv.org/html/2607.08093#A1.SS4), and the generation\-side prompt templates are reproduced in App\.[A\.16\.1](https://arxiv.org/html/2607.08093#A1.SS16.SSS1)\.

Algorithm 1Scene synthesis: variable mapping and story verbalization1:Input:DAGGG; optional stage planPP; budgetsR,T,SR,T,Sfor seeding, mapping audit/repair, and story verification\.2:Output:domainDD, mappingMM, verified storyσ\\sigma, or failure\.3:IfPPis absent, setP←\[G\]P\\leftarrow\[G\]; letH0H\_\{0\}be the first stage and initialize its fixed\-name assignmentF0←∅F\_\{0\}\\leftarrow\\emptyset\.4:ifCauseNet seeding is enabledthen5:Try up toRRCauseNet matches onH0H\_\{0\}; take the first passingPreAudit\(Alg\.[2](https://arxiv.org/html/2607.08093#alg2)\), or fail if seeding is mandatory\.6:endif7:\(D,M\)←MapStage​\(H0,F0,T\)\(D,M\)\\leftarrow\\textsc\{MapStage\}\(H\_\{0\},F\_\{0\},T\)\(Alg\.[3](https://arxiv.org/html/2607.08093#alg3)\);returnfailure if it fails\.8:foreach auxiliary stageHiH\_\{i\}with shared anchoraia\_\{i\}do9:ExtendMMwithMapStage​\(Hi,Fi,T\)\\textsc\{MapStage\}\(H\_\{i\},F\_\{i\},T\), whereFiF\_\{i\}fixesaia\_\{i\}to its current name and forbids names from other stages; fail on mapping failure, anchor drift, or name duplication across stages\.10:endfor11:ifthere are auxiliary stages and the merged\-graph audit is enabledthen12:Run the audit/repair loop onMMover the fullGGfor up toTTrounds, preserving anchor meanings;returnfailure if it fails\.13:endif14:σ←Verbalize​\(D,M,G,S\)\\sigma\\leftarrow\\textsc\{Verbalize\}\(D,M,G,S\)\(Alg\.[4](https://arxiv.org/html/2607.08093#alg4)\);returnfailure if it fails\.15:return\(D,M,σ\)\(D,M,\\sigma\)\.

### 3\.4Task suite

We use a 3\-level taxonomy:Rung\(R​1R1–R​3R3\) places the task on Pearl’s hierarchy \(associational, interventional, counterfactual\);question familyis the semantic intent / target quantity \(e\.g\., prediction, identification, effect estimation\) and expands on CLadder’s collection;output variantis the required output type within the family \(e\.g\., sign vs\. strength; point vs\. interval\)\. We denote tasks asRung::family::variant\. Tab\.[1](https://arxiv.org/html/2607.08093#S3.T1)summarizes the question families; the per\-family output variants and their scoring rules are enumerated by rung in Tabs\.[8](https://arxiv.org/html/2607.08093#A1.T8)–[10](https://arxiv.org/html/2607.08093#A1.T10)\(App\.[A\.9](https://arxiv.org/html/2607.08093#A1.SS9)\)\.

Table 1:Question families per rung\.RungFamilyDescriptionR​1R1PredictionFit a predictive model for outcomeYYfrom observed columns; write held\-out point predictions or, for continuousYY, central prediction intervals\.R​1R1AssociationQuantify the observational relation between treatmentXXand outcomeYY, marginally or after conditioning on a third variable\.R​1R1Collider phenomenonTest whether conditioning on a named collider opens a spurious association between two of its parents \(explaining\-away\)\.R​2R2Causal sketchRecover the story\-implied graph as a directed edge set or as an undirected skeleton\.R​2R2IdentificationDecide whether the population ATE ofXXonYYis identifiable from the observed conceptual variables and by what strategy; includes adjustment\-set queries\.R​2R2Effect estimationEstimate the population average treatment effect \(ATE\); output point, uncertainty, sign, or agreement with the observed association\.R​2R2Bias diagnosticsDiagnose whether an adjustment strategy is biased \(collider conditioning, forbidden controls\)\.R​3R3Counterfactual identificationDecide whether the target counterfactual is identifiable\.R​3R3Counterfactual effectsEstimate the effect of treatment on the treated \(ETT\), or abstain if non\-identifiable\.R​3R3MediationDecompose the effect into natural direct \(NDE\) and natural indirect \(NIE\) components; report point, sign, or which dominates\.Some variants are available only in compatible scenes \(interval prediction requires continuous outcomes, collider\-related tasks require colliders, etc\.\)\. Rung\-2 effect and identification tasks are gated on*population\-ATE*identifiability, decided with DoWhy\(Sharma and Kiciman,[2020](https://arxiv.org/html/2607.08093#bib.bib69)\)\. Rung\-3 identifiability is decided separately with they0

### 3\.5Scene format

Public scene artifacts comprise thestory, relevant datasetparquetfiles, and the dataset schema\. For scenes with a measurement observation model, we additionally include acalibration set, a smaller table containing both measurement columns and gold conceptual\-variable values\. Private scoring artifacts include the ground truth, the observation model diagnostics, and, when needed, hidden test set labels \(forKaggle\-style prediction tasks\)\. For the results presented, we constructed a dataset with 953 scenes, each with three possible observation model variants\.

### 3\.6Evaluation harness

We evaluate models usingmini\-swe\-agent\(Yanget al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib62)\), a lightweight agent harness in which the model interacts with the environment by emittingbashshell commands\. All tool execution happens inside a sandboxed Docker \(or similar\) container with network access disabled\. The container ships the standard data\-science stack; each command runs as a separate shell process and state persists through files\. Each task is run as a fresh agent conversation with configurable step and cost limits, keeping per\-task contexts small and recording full trajectories for reproducibility and error analysis\. After the run completes, the grader parses the agent’s answer files and scores them against private ground truth \(Sec\.[3\.7](https://arxiv.org/html/2607.08093#S3.SS7)\); we additionally log efficiency signals \(tool\-call counts and API\-reported token usage\) to support joint evaluation of reasoning quality and tool use\. Technical details \(containerization, available packages, etc\.\) are given in App\.[A\.15](https://arxiv.org/html/2607.08093#A1.SS15)\.

### 3\.7Scoring

Grading is fully deterministic: for each task, the grader checks that the required answer file is present and parseable, then computes a task\-specific metric from the answer and the private ground truth\. For non\-identifiable task instances, task schemas include explicit abstain targets \(e\.g\.,nullorunknown\), allowing the benchmark to quantify a model’s propensity to hallucinate when no answer is available\. This is complementary to verifier\-style scoring approaches proposed for causal expressions\(Heet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib61)\): a verifier checks whether an emitted causal expression is formally correct, whereas abstention scoring asks the orthogonal question of whether the agent recognizes that no formally correct answer exists\.

Each task emits one atomic score with a metric name; the per\-metric formulas \(RMSE, ROC\-AUC, Brier, log\-loss,F1F\_\{1\}, the interval score \(IS\)\(Gneiting and Raftery,[2007](https://arxiv.org/html/2607.08093#bib.bib105)\)…\) and the task\-specific scoring rules by rung are listed in App\.[A\.9](https://arxiv.org/html/2607.08093#A1.SS9)\(Tabs\.[8](https://arxiv.org/html/2607.08093#A1.T8)–[10](https://arxiv.org/html/2607.08093#A1.T10)\)\.

For every variant where the answer might not be identifiable, the grader usesmutually\-exclusive routing: a task is graded as a binary whenevereither sideabstains — i\.e\., the ground truth is non\-identifiable,orthe model wrongly abstains; otherwise it is graded by its content metric\.

To summarize models for a leaderboard, we introduce three pooled quality categories and two single\-number composites\. LetDDbe the discrete\-task slice; letTNRT\_\{\\mathrm\{NR\}\}be the set of continuous\-output tasks; letTF1T\_\{F\_\{1\}\}be the set ofF1F\_\{1\}\-graded tasks\. Define the per\-task normalized error \(Eq\.[1](https://arxiv.org/html/2607.08093#S3.E1)\):

NRelErri=ni−1​∑j\(θi​j−θ^i​j\)21\+si,\\mathrm\{NRelErr\}\_\{i\}=\\frac\{\\sqrt\{n\_\{i\}^\{\-1\}\\sum\_\{j\}\(\\theta\_\{ij\}\-\\hat\{\\theta\}\_\{ij\}\)^\{2\}\}\}\{1\+s\_\{i\}\},\(1\)with\(ni,si\)=\(ntest,sd​\(Ytest\)\)\(n\_\{i\},s\_\{i\}\)=\(\\mathrm\{n\_\{test\}\},\\mathrm\{sd\}\(Y\_\{\\mathrm\{test\}\}\)\)for prediction point tasks \(usingBrier\\sqrt\{\\mathrm\{Brier\}\}as the numerator for binary outcomes\), and\(ni,si\)=\(1,\|τi\|\)\(n\_\{i\},s\_\{i\}\)=\(1,\|\\tau\_\{i\}\|\)for scalar point estimates, whereτi\\tau\_\{i\}is ground\-truth scalar target \(e\.g\., the ATE\)\. For interval scores,sis\_\{i\}is one of the above depending on whether we are dealing with prediction intervals or effect intervals\. The headline columns are then defined in Eq\.[2](https://arxiv.org/html/2607.08093#S3.E2):

PassRate\\displaystyle\\mathrm\{PassRate\}=\|D\|−1​∑i∈D𝟏​\[y^i=yi\],\\displaystyle=\|D\|^\{\-1\}\\sum\_\{i\\in D\}\\mathbf\{1\}\[\\hat\{y\}\_\{i\}=y\_\{i\}\],\(2\)Med\.NRel\.Err\\displaystyle\\mathrm\{Med\.\\ NRel\.\\ Err\}=medi∈TNR​NRelErri,\\displaystyle=\\mathrm\{med\}\_\{i\\in T\_\{\\mathrm\{NR\}\}\}\\mathrm\{NRelErr\}\_\{i\},Med\.F1​\-​Loss\\displaystyle\\mathrm\{Med\.\\ F\_\{1\}\\text\{\-\}Loss\}=medi∈TF1​\(1−F1,i\)\.\\displaystyle=\\mathrm\{med\}\_\{i\\in T\_\{F\_\{1\}\}\}\(1\-F\_\{1,i\}\)\.Pass Rate pools both actual binary content and the abstention binaries; the medians keeps the typical\-task headline robust to the long\-tailed outliers\. We partitionTNR=TNRpt∪TNRintT\_\{\\mathrm\{NR\}\}=T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{pt\}\}\\cup T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{int\}\}into point\-graded tasks and interval\-graded tasks with the aggregatorSN​RS\_\{NR\}in Eq\.[3](https://arxiv.org/html/2607.08093#S3.E3):

SNR=\|TNRpt\|​mpt\+\|TNRint\|​mint\|TNR\|,mpt=medi∈TNRpt​NRelErri,mint=meani∈TNRint​min⁡\(NRelISi,c\),\\begin\{gathered\}S\_\{\\mathrm\{NR\}\}=\\frac\{\|T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{pt\}\}\|m\_\{\\mathrm\{pt\}\}\+\|T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{int\}\}\|m\_\{\\mathrm\{int\}\}\}\{\|T\_\{\\mathrm\{NR\}\}\|\},~m\_\{\\mathrm\{pt\}\}=\\mathrm\{med\}\_\{i\\in T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{pt\}\}\}\\mathrm\{NRelErr\}\_\{i\},\\\\ m\_\{\\mathrm\{int\}\}=\\mathrm\{mean\}\_\{i\\in T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{int\}\}\}\\min\(\\mathrm\{NRelIS\}\_\{i\},c\),\\end\{gathered\}\(3\)withc=10c=10, keeping point\-error tasks median\-robust while ensuring that the models do not get away with submitting overly\-confident confidence intervals by using acapped mean\(App\.[A\.9](https://arxiv.org/html/2607.08093#A1.SS9)\) for the intervals \(as the IS is strictly proper in expectation, the mean is the more appropriate aggregator\)\.

For an absolute single\-number summary we define theCausalDSscore \(Eq\.[4](https://arxiv.org/html/2607.08093#S3.E4)\):

CausalDSScore=wp​\(1−PassRate\)\+wr​SNR\+wf​Med\.F1​\-​Loss,\\mathrm\{CausalDSScore\}=w\_\{p\}\(1\-\\mathrm\{PassRate\}\)\+w\_\{r\}S\_\{\\mathrm\{NR\}\}\+w\_\{f\}\\,\\mathrm\{Med\.\\ F\_\{1\}\\text\{\-\}Loss\},\(4\)withwp=Np/\(Np\+Nr\+Nf\),wr=Nr/\(Np\+Nr\+Nf\)w\_\{p\}=N\_\{p\}/\\left\(N\_\{p\}\+N\_\{r\}\+N\_\{f\}\\right\),w\_\{r\}=N\_\{r\}/\\left\(N\_\{p\}\+N\_\{r\}\+N\_\{f\}\\right\), andwf=Nf/\(Np\+Nr\+Nf\)w\_\{f\}=N\_\{f\}/\\left\(N\_\{p\}\+N\_\{r\}\+N\_\{f\}\\right\), whereNp=\|D\|,Nr=\|TNR\|N\_\{p\}=\|D\|,N\_\{r\}=\\left\|T\_\{\\mathrm\{NR\}\}\\right\|, andNf=\|TF1\|N\_\{f\}=\\left\|T\_\{F\_\{1\}\}\\right\|are the per\-pool task counts;lower is better\.

A CausalDS score of0thus requires perfection on every column; while most scores should land in the\[0,1\]\[0,1\]range, the score can exceed11when the NRel\. Err median is large\. CausalDSScore is comparable across exams but is dominated, by construction, by whichever pool is largest unless re\-weighted\. As a complementary, exam\-relative summary we compute theComposite Rank: per\-column average ranks normalized to\[0,1\]\[0,1\], then averaged, computed on the same three metrics as CausalDSScore \(Pass Rate,SNRS\_\{\\mathrm\{NR\}\}, and Med\.F1F\_\{1\}\-Loss\)\. This does not transfer across exams of similar composition; however, it is not sensitive to large numeric outliers\. Missing or malformed outputs are handled by metric family: for bothboundedmetrics \(binary andF1F\_\{1\}scores\), missing answerscount as failures\. Forunboundedoutputs, we avoid assigning an arbitrary numeric loss to missing values; instead, NRel\. metrics are computed on valid numeric submissions and we report the fraction of continuous\-style tasks with a valid answer\. While a single\-number metric can be attractive as a rough orientation, it is not the final word\. For this reason, in Sec\.[5](https://arxiv.org/html/2607.08093#S5)we report both the intermediate \(the three components of the CausalDS score\) and atomic metrics, whenever appropriate\.

## 4Related work

Tab\.[2](https://arxiv.org/html/2607.08093#S4.T2)presents an overview across the main design axes for the most closely\-related work\.

Table 2:Related benchmark overview\.WorkSynth\.inst\.Gen\.graphFree\-formstoryFile /codeSCMGTObs\.layerR1–R3tasksNon\-ID /abstainCommentCausalDS\(Ours\.\) Integrated causal data\-science agent benchmark\. Generated hidden SCMs; synthetic tabular data; graph\-audited free\-form stories; clean and noisy\-measurement observation variants; file\-backed tool use; R1–R3 tasks; scored abstention\.CauSciBench⚠️⚠️⚠️End\-to\-end causal\-effect analysis benchmark\. Real\-paper/textbook\-derived tasks plus synthetic scenarios; natural\-language problem setup; variable/method selection; statistical implementation; R2 effect\-estimation focus; no generated graphs or observation layer\.CausalReasoningBenchmark⚠️Identification/estimation benchmark on real datasets\. Curated papers and datasets; structured identification specs; point estimates and standard errors; R2\-focused; no generated graph, hidden SCM, synthetic story, or measurement layer\.CausalProfilerSynthetic causal\-ML data generator; no LLM\-eval involved\. Random causal models, data, queries, and ground truth; observational/interventional/counterfactual coverage; in\- and out\-of\-identification regimes; no language scenes or file\-backed LLM agents\.CausalGame†⚠️Interactive causal\-agent game benchmark\. SCM\-driven game environments; hidden confounders, noisy measurement, selection bias; active data collection; small scenario family; not tabular scene files or broad R1–R3 task generation\.CausaLab†⚠️Interactive causal\-discovery environment\. Freshly sampled hidden SCMs per episode; budgeted interventions, held\-out\-target prediction, and DSL\-based mechanism recovery; fixed lab narrative rather than per\-graph stories; no tabular scene files, counterfactual tasks, or abstention\.CLadder⚠️⚠️Canonical causal reasoning QA benchmark\. Synthetic graph/query pairs; motif\-style causal graphs; templated natural\-language questions; Bernoulli CBNs; R1–R3 symbolic QA; no tabular data, tool use, measurement layer, or non\-ID abstention\.
Legend\.= central feature; = partially present; ⚠️ = narrow, probe\-level, or adjacent; = absent or N/A\.†Concurrent under a May 1, 2026 cutoff\. The columns correspond to:Synthetic benchmark instances:instances generated fresh, not curated from existing sources\.Generated causal graph:\(option of\) dynamically sampled graphs, not a fixed motif family or hand\-authored design\.Free\-form story:natural\-language verbalization tied to the graph, beyond templates\.File / code analysis:agent operates over files, code, and tools, not single\-prompt QA\.SCM ground truth:true causal structure, mechanisms, and effects are known\.Observation layer:data can reflect the conceptual causal variables through a noisy measurement\.R1–R3 tasks:does the benchmark cover Pearl’s hierarchy?Non\-ID / abstain:deliberately includes non\-identifiable cases requiring abstention\.

Besides the works summarized in Tab\.[2](https://arxiv.org/html/2607.08093#S4.T2)—CauSciBench,CausalReasoningBenchmark,CausalProfiler,CausalGame,CausaLab, andCLadder\(Acharyaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib81); Sawarniet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib93); Panayiotouet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib92); Chenet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib83); Yanget al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib156); Jinet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib24)\)—CausalDS is related to three broader lines of work\.

##### Real\-study causal analysis benchmarks\.

The closest real\-data benchmarks evaluate causal analysis over scientific or policy studies\.CauSciBenchandCausalReasoningBenchmarktest end\-to\-end causal analysis on curated datasets, with emphasis on method choice, identification, estimation, and uncertainty;InterveneBenchinstead emphasizes intervention\-centered study\-design reasoning in real social systems without predefined graphs or structural equations\(Shiet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib111)\)\. These benchmarks provide strong external realism, but their instances are derived from existing papers, datasets, or study designs, which makes them potentially vulnerable to data contamination \(the “causal parrot” issue\)\.

##### Synthetic causal benchmarks and causal reasoning probes\.

A second line evaluates formal causal reasoning in controlled settings\.CLadderconstructs natural\-language causal QA examples from small graph motifs and oracle\-generated answers spanning Pearl’s ladder;CausalGraph2LLM,CLEAR, andCausalBenchfocus on graph\-centric or format\-dependent tests of causal understanding\(Shethet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib44); Chenet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib41); Zhouet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib34)\)\. Other benchmarks target narrower failure modes:CausalPitfallsstresses classical statistical traps such as confounding, Simpson’s paradox, and selection bias, whileExecutable Counterfactualsfocuses on code\-mediated counterfactual reasoning\(Duet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib1); Vashishthaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib94)\)\. On the generation side,CausalProfilersamples causal models, data, queries, and ground truth, andLanguage Models as Causal Effect Generatorsintroduces sequence\-driven SCMs for treatment\-effect benchmarking\(Bynum and Cho,[2025](https://arxiv.org/html/2607.08093#bib.bib26)\)\. Concurrent work extends this line:NoisyCausalevaluates natural\-language causal QA under structured noise \(distractors, perturbed values, latent confounders, partial masking\), andReplaySCMscores executable Boolean\-SCM mechanism induction from finite interventional evidence via replay on held\-out interventions\(Xu and Fu,[2026](https://arxiv.org/html/2607.08093#bib.bib157); Batzoglou,[2026](https://arxiv.org/html/2607.08093#bib.bib158)\)\.

CausalDS differs by making the benchmark unit a narrated fully\-fledged scene: a hidden SCM is realistically grounded into domain variables, rendered as a graph\-audited free\-form story, packaged with synthetic SCM\-driven data files, and the agent’s tool\-backed answers are then scored deterministically\.

##### Agentic data\-science benchmarks and causal\-agent systems\.

Outside causality,DS\-1000,DSBench,MLE\-bench,MLE\-Dojo,BLADE, andDiscoveryBenchevaluate code generation, ML engineering, or open\-ended data\-analysis workflows\(Laiet al\.,[2022](https://arxiv.org/html/2607.08093#bib.bib9); Jinget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib33); Chanet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib43); Qianget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib35); Guet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib79); Majumderet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib80)\), often using interactive execution frameworks such asInterCodeandSWE\-agent/mini\-swe\-agent\(Yanget al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib58);[2024](https://arxiv.org/html/2607.08093#bib.bib62)\)\. These benchmarks motivate our file\-backed, tool\-using evaluation protocol, but they do not provide hidden causal structure, interventional or counterfactual task families, or abstention targets for non\-identifiable estimands\. Within causal inference,CAISautomates causal method selection and execution from a dataset\-plus\-query input\(Vermaet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib6)\), whileCausalGameevaluates causal thinking through interactive scientific\-discovery games\(Chenet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib83)\)\. Concurrent workCausaLabevaluates LLM agents in an interactive synthetic laboratory where each episode hides a freshly sampled SCM: the agent intervenes under a budget, predicts a held\-out target, and is scored on both task success and the fidelity of the recovered mechanism\(Yanget al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib156)\)\. CausaLab thus asks whether an agent can act as an experimental causal discoverer; CausalDS asks whether it can act as a causal data scientist over narrated, file\-backed observational scenes\. CausalDS is thus complementary to the above works: it benchmarks agents over scene directories where causal reasoning, data\-science skills, and tool use all matter simultaneously\.

Overall, prior and concurrent work covers some individual ingredients of CausalDS, but to the best of our knowledge no benchmark integrates graph\-audited free\-form scenes, SCM\-generated data files, a noisy\-measurement observation layer, complete Pearl\-ladder estimand tasks with deterministic scoring, and first\-class non\-identifiability abstention into a single generator\.

## 5Main Results

Table 3:Realistic\-exam leaderboard\. Sorted by ascending CausalDSScore \(Eq\.[4](https://arxiv.org/html/2607.08093#S3.E4)\)\.*Pass Rate*combines binary content and abstention;SNRS\_\{\\mathrm\{NR\}\}is the relative numeric component \(Eq\.[3](https://arxiv.org/html/2607.08093#S3.E3)\)\.*Comp\. Rank*is the rank\-version of CausalDS score over the same components\.*Valid cont\. answers*counts valid submissions among continuous\-style tasks\.*Tok\./task*counts the mean total \(prompt\+\+completion\) tokens per task\. Best per column in bold\.↓\\downarrow— lower is better,↑\\uparrow— higher is better\.\#ModelValid cont\. ans\.↑\\uparrowCausalDSScore↓\\downarrowComp\. Rank↓\\downarrowPass Rate↑\\uparrowMed\. NRel\. Err↓\\downarrowSNRS\_\{\\mathrm\{NR\}\}↓\\downarrowTok\./task1Claude Opus 4\.838/39 \(97\.4%\)0\.27800\.20082\.4%0\.1790\.56617\.7k2Gemini 3\.1 Pro38/39 \(97\.4%\)0\.37030\.36776\.5%0\.2310\.754145\.6k3Qwen 3\.6 35B31/39 \(79\.5%\)0\.44740\.56763\.2%0\.2760\.934140\.7k4Kimi K2\.638/39 \(97\.4%\)0\.47540\.56765\.7%0\.2300\.935266\.4k5GPT\-5\.537/39 \(94\.9%\)0\.56100\.53382\.4%0\.2241\.32412\.9k6Gemma 4 26B39/39 \(100\.0%\)0\.64420\.76755\.9%0\.3131\.26732\.4k

##### Realistic\-composition exam\.

We report results on a 100\-scene realistic\-composition exam sampled from a953953\-scene dataset\. Task\-family weights are tilted towards real\-world distributions \(App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)\), hence we term this therealistic exam\. Exam contains2828R1,5151R2, and2121R3 tasks; observation variants split4848/3232/2020across the three increasing difficulties \(clean/proxy/proxy\_hard\); outcomes split5959continuous /4141binary\. The motif breakdown \(with grafted scenes split by graft count\) is in

App\.[A\.10](https://arxiv.org/html/2607.08093#A1.SS10)\. We evaluate six models: three frontier closed models \(Claude Opus 4\.8, Gemini 3\.1 Pro, GPT\-5\.5\), each run at high reasoning effort, and three open\-weight models \(Qwen 3\.6 35B, Kimi K2\.6, Gemma 4 26B\), run at their model defaults\. The step limit was100100and per\-task cost limit was $1010444Where available, never hit in practice\.\. For the specific experimental settings, refer to App\.[A\.15](https://arxiv.org/html/2607.08093#A1.SS15)\.

##### Headline results

Table[3](https://arxiv.org/html/2607.08093#S5.T3)presents the aggregate metrics and is ranked according to theCausalDSscore \(Eq\.[4](https://arxiv.org/html/2607.08093#S3.E4)\)\. Claude Opus 4\.8 leads the field \(0\.2780\.278\) and sweeps the quality column s: it is best or tied\-best on Pass Rate \(82\.4%82\.4\\%\), Med\. NRel\. Err \(0\.1790\.179\), andSNRS\_\{\\mathrm\{NR\}\}\(0\.5660\.566\)\. Two frontier reasoning models take the top two composite spots \(Claude Opus 4\.8 then Gemini 3\.1 Pro at0\.3700\.370\); the open models occupy a narrow middle band — Qwen 3\.6 35B \(0\.4470\.447\) and Kimi K2\.6 \(0\.4750\.475\) separated by less than0\.030\.03— with the smaller Gemma 4 26B last \(0\.6440\.644\)\. The most informative dissociation is GPT\-5\.5: it ties Claude Opus 4\.8 for the best raw Pass Rate \(82\.4%82\.4\\%\) yet ranks only fifth on CausalDSScore \(0\.5610\.561\), recovering to third on the exam\-relative Composite Rank \(0\.5330\.533\)\. GPT\-5\.5 spotlights the difference between the composites: its continuous estimates are the worst\-calibrated among the models \(SNR=1\.324S\_\{\\mathrm\{NR\}\}=1\.324; mean NRel\. Err onproxy\_hardof3\.103\.10, Tab\.[6](https://arxiv.org/html/2607.08093#S5.T6)\)\. CausalDSScore sums the pools by magnitude, so that heavy tail dominates and drops it to fifth; Composite Rank averages the ranks, so the tail has less of an effect \(per\-axis rank breakdowns: Tab\.[14](https://arxiv.org/html/2607.08093#A1.T14), App\.[A\.10](https://arxiv.org/html/2607.08093#A1.SS10)\)\. Gemma 4 26B commits an answer on every continuous task \(39/3939/39valid\), but pays with the highest normalized error \(Med\. NRel\. Err0\.3130\.313\) and the lowest Pass Rate, landing last; Qwen 3\.6 35B has lowest valid\-continuous answer rate \(79\.5%79\.5\\%\) by failing more of the hardest estimands — mostly through its own poor tool\-output management: it has a tendency to dump the whole dataset to output instead of usinghead/tail\. This persists even with several re\-runs where we manually truncate its tool output \(App\.[A\.15](https://arxiv.org/html/2607.08093#A1.SS15)\)\.

For this exam, the differentiation is concentrated at Rung 2 and on uncertainty quantification: R2 Pass Rate spans28\.6%28\.6\\%\(Gemma 4 26B\) to92\.9%92\.9\\%\(GPT\-5\.5\) \(App\.[A\.10](https://arxiv.org/html/2607.08093#A1.SS10), Tab\.[13](https://arxiv.org/html/2607.08093#A1.T13)\) andidentificationPass Rate ranges from28\.6%28\.6\\%\(Gemma 4 26B\) to100\.0%100\.0\\%\(Claude Opus 4\.8\); empirical coverage of95%95\\%ATE intervals collapses to20\.0%20\.0\\%–71\.4%71\.4\\%— that is, the models areoverconfident\(Fig\.[3\(a\)](https://arxiv.org/html/2607.08093#S5.F3.sf1), Tab\.[6](https://arxiv.org/html/2607.08093#S5.T6)\), with Claude again performing the best\. This is consistent with recent uncertainty\-elicitation benchmarks: nominal 95% credible intervals from GPT\-5\-family models cover only 9–44% of ground truth in Bayesian\-elicitation evaluations\(Hoboret al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib147)\), nominal 99% intervals cover 65% on Fermi\-style estimation\(Epsteinet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib146)\), and elicited probabilistic priors are systematically overconfident\(Rendaet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib150)\)\.

![Refer to caption](https://arxiv.org/html/2607.08093v1/x1.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.08093v1/x2.png)\(b\)

Figure 3:\(a\) Empirical coverage of the nominal95%95\\%ATE intervals per model over the identifiableate\_uq\_95tasks \(n=7n\{=\}7per model;55for Qwen 3\.6 35B\)\. \(b\) Content pass rate, abstention pass rate, andSNRS\_\{\\mathrm\{NR\}\}\(Eq\.[3](https://arxiv.org/html/2607.08093#S3.E3); lower is better\) per model\. Models are ordered by CausalDSScore \(best at top\); blue circles — frontier \(closed\) models, green triangles — open\-weight models\.Tool\-use strategy splits the field \(Fig\.[4](https://arxiv.org/html/2607.08093#S5.F4)\): GPT\-5\.5 \(2\.12\.1calls/task\) and Claude Opus 4\.8 \(3\.43\.4\) are near one\-shot, while Gemini 3\.1 Pro, Kimi K2\.6, and Qwen 3\.6 35B iterate heavily \(1111–1818bash calls per task, App\.[A\.10](https://arxiv.org/html/2607.08093#A1.SS10), Tab\.[18](https://arxiv.org/html/2607.08093#A1.T18)\)\. Recent agent benchmarks likewise find that the number of tool calls a model makes per task varies strongly across model families and benchmarks, and is not predicted by overall capability\(Wanget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib144); Xuet al\.,[2026a](https://arxiv.org/html/2607.08093#bib.bib152); Zhanget al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib153); Xuet al\.,[2026b](https://arxiv.org/html/2607.08093#bib.bib154)\)\. Token usage spans more than an order of magnitude, from12\.912\.9k tokens/task \(GPT\-5\.5\) to266\.4266\.4k \(Kimi K2\.6\)\. Notably, the two model classes are linearly separable in the cost–quality plane \(the dashed line in Fig\.[4](https://arxiv.org/html/2607.08093#S5.F4)\): at comparable token budgets, the closed models outscore the open ones\.

![Refer to caption](https://arxiv.org/html/2607.08093v1/x3.png)Figure 4:Tokens per task \(log scale\) versus CausalDSScore \(lower is better\) on the realistic exam\. Each point is one model, annotated with its mean bash calls per task and interaction style; the dashed line separates the frontier \(closed\) models from the open\-weight ones\.
##### Content correctness and abstention separate\.

Pass Rate decomposes into content and abstention slices \(Tab\.[4](https://arxiv.org/html/2607.08093#S5.T4), Fig\.[3\(b\)](https://arxiv.org/html/2607.08093#S5.F3.sf2)\)\. Content correctness is high across the whole field \(81\.0%81\.0\\%–100\.0%100\.0\\%\); the differentiating axis is abstention, which spans18\.8%18\.8\\%–75\.0%75\.0\\%\. The split tracks thefrontier/open\-weights model divide: the three frontier models lead abstention — GPT\-5\.5 \(75\.0%75\.0\\%\), Claude Opus 4\.8 \(62\.5%62\.5\\%\), Gemini 3\.1 Pro \(56\.2%56\.2\\%\) — while the open models trail \(Qwen 3\.6 35B41\.2%41\.2\\%, Kimi K2\.638\.9%38\.9\\%, Gemma 4 26B18\.8%18\.8\\%\), with the smaller Gemma especially struggling\. The two extremes are instructive: Claude Opus 4\.8 is the only model with perfect content \(100\.0%100\.0\\%,18/1818/18\) and pairs it with solid abstention, whereas Gemma 4 26B over\-commits, matching the field on content \(88\.9%88\.9\\%\) but declining almost nothing when it should \(18\.8%18\.8\\%abstention\) and finishing last overall\. Thus knowing when to abstain is a separate, advanced skill\. As discussed in Sec\.[3\.7](https://arxiv.org/html/2607.08093#S3.SS7), pool sizes vary across models because routing into the abstention pool depends on the model’s own answer\. Abstention is the live discriminator on both upper rungs, which is presented in App\.[A\.10](https://arxiv.org/html/2607.08093#A1.SS10)\.

Table 4:Pass Rate decomposition\. Abstention pass rate \(PR\) is over tasks where either the ground truth is non\-identifiableorthe model chooses to abstain \- hence the number \(NN\) of tasks is dynamic\.ModelPass RateContent PR \(NN\)Abstention PR\(NN\)Claude Opus 4\.882\.4%100\.0% \(18\)62\.5% \(16\)GPT\-5\.582\.4%88\.9% \(18\)75\.0% \(16\)Gemini 3\.1 Pro76\.5%94\.4% \(18\)56\.2% \(16\)Kimi K2\.665\.7%94\.1% \(17\)38\.9% \(18\)Qwen 3\.6 35B63\.2%81\.0% \(21\)41\.2% \(17\)Gemma 4 26B55\.9%88\.9% \(18\)18\.8% \(16\)

##### Imperfect observation and uncertainty quantification drive a second axis of failure\.

Tables[5](https://arxiv.org/html/2607.08093#S5.T5)and[6](https://arxiv.org/html/2607.08093#S5.T6)present the discrete and continuous aggregates separated by the observation layer hardness \(withcleandenoting no observation layer — all conceptual variables directly measured\), as well as the performance on the 95% confidence interval for the ATE subset of questions\. We observe the expected trends: increasing observation difficulty tends to increase the difficulty, and disproportionally so for the weaker models\.

The results also reveal a failure mode invisible in the aggregate statistics, which takes the median\. Under the hardest observation the*mean*NRel\. Err runs far above the median for most models — Gemma 4 26B blows up to a mean of4\.594\.59against a median of0\.3480\.348\(a∼13×\\sim 13\\timesgap\), and GPT\-5\.5 to3\.103\.10\(vs\.0\.2220\.222\): a handful of estimates on noise\-corrupted data miss by orders of magnitude\.The clear exception is again Claude, whose mean stays controlled at0\.5570\.557\(against a median of0\.1830\.183\) A similar miscalibration also shows up in interval coverage, where empirical coverage of the95%95\\%ATE intervals collapses to20\.0%20\.0\\%–71\.4%71\.4\\%against the nominal0\.950\.95, with Claude Opus 4\.8 the best \(71\.4%71\.4\\%\) and Qwen 3\.6 35B the worst \(20\.0%20\.0\\%\)\.

However, the per\-observation layer hardness results in Tabs\.[5](https://arxiv.org/html/2607.08093#S5.T5)and[6](https://arxiv.org/html/2607.08093#S5.T6)are naturally aggregated overdifferentsub\-exam slice compositions \(cf\. App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)\)\. A controlled ablation that re\-evaluates the*same*conceptual scene/task under all three released observation views \(App\.[A\.13](https://arxiv.org/html/2607.08093#A1.SS13)\) confirms the hardness ordering within matched scenes and separates two effects: the estimates that do get produced become substantially worse, and the harder views additionally cause outright failures to answer — context\-window exhaustion, unwarranted abstention — alongside more calls per task\.

Table 5:Pass Rate by observation variant\. Bracketed numbers are pool sizes: a fixed count in a column header applies to every row, while*\(NN\)*marks columns whose pool size varies by model, with the per\-row count shown next to each value\.ModelPRclean\(NN\)PRproxy\(13\)PRhard\(NN\)Claude Opus 4\.888\.9%\(18\)84\.6%33\.3% \(3\)Gemini 3\.1 Pro83\.3% \(18\)76\.9%33\.3% \(3\)Qwen 3\.6 35B63\.2% \(19\)84\.6%16\.7% \(6\)Kimi K2\.663\.2% \(19\)76\.9%33\.3% \(3\)GPT\-5\.588\.9%\(18\)76\.9%66\.7%\(3\)Gemma 4 26B50\.0% \(18\)69\.2%33\.3% \(3\)Table 6:Normalized error by observation variant and uncertainty quantification \(*ATE\-UQ95 cov\.*— empirical coverage of95%95\\%ATE intervals\)\. Pool\-size conventions as in Tab\.[5](https://arxiv.org/html/2607.08093#S5.T5)\.ModelMed\. NRelclean\(NN\)Med\. NRelproxy\(7\)Med\. NRelhard\(NN\)Mean NRelhard\(NN\)ATE\-UQ95 cov\. \(NN\)Claude Opus 4\.80\.134 \(16\)0\.0990\.183 \(15\)0\.56\(15\)71\.4%\(7\)Gemini 3\.1 Pro0\.122\(16\)0\.3010\.238 \(15\)1\.41 \(15\)42\.9% \(7\)Qwen 3\.6 35B0\.223 \(15\)0\.3810\.276 \(9\)1\.58 \(9\)20\.0% \(5\)Kimi K2\.60\.223 \(15\)0\.3020\.173\(16\)1\.49 \(16\)42\.9% \(7\)GPT\-5\.50\.171 \(16\)0\.2240\.222 \(14\)3\.10 \(14\)42\.9% \(7\)Gemma 4 26B0\.212 \(16\)0\.0990\.348 \(16\)4\.59 \(16\)28\.6% \(7\)

##### Repeated\-attempt stability, pass@kkand passˆkk–like metrics\.

We bracket eachopen\-weight model’s behavior overk=3k\{=\}3independent restarts by simplified*pass@kk*– and*passˆkk*–like metrics\. Namely, instead of estimating the population parameter as inChenet al\.\([2021](https://arxiv.org/html/2607.08093#bib.bib139)\); Yaoet al\.\([2024](https://arxiv.org/html/2607.08093#bib.bib138)\), we look at whether the task is solved by*any*of thekkor, correspondingly,*all*of thekkrepeats; more broadly, we take thebestor theworstcase, respectively\.

In this case, the scoring usesground\-truth\-fixed pools\(a deviation from the answer\-dependent main routing\) so that the runs are compatible: only ground\-truth non\-identifiable cases go into the “abstention pool” ; thusk=1k\{=\}1matches the main result run; further details are presented in App\.[A\.11](https://arxiv.org/html/2607.08093#A1.SS11)\.

For the binary outcomes \(Tab\.[7](https://arxiv.org/html/2607.08093#S5.T7),n=34n\{=\}34\), pass@3 reaches76\.576\.5–91\.2%91\.2\\%— Kimi K2\.691\.2%91\.2\\%, numerically matching the best frontier single\-shot rate on the same pool — while passˆ33is within50\.050\.0–61\.8%61\.8\\%\. Thus, a2626–2929\-point gap exists between the best\-case and worst\-case performance\.

Perhaps a more interesting metric is the variance of the CausalDS score itself on repeated attempts as the latter adds continuous\-like answers to the binary ones captured by pass@kk/passˆkk\. That score\-level variance \(77–27%27\\%of the mean; Tab\.[7](https://arxiv.org/html/2607.08093#S5.T7)\) is not benchmark instability: the point estimates reproduce near\-exactly across restarts \(median per\-task NRelErr SD≤0\.002\\leq 0\.002over tasks answered in all 3 runs\)\. It comes instead from a relatively few uncertainty\-quantification tasks and from abstention\-decision flips \(App\.[A\.11](https://arxiv.org/html/2607.08093#A1.SS11), Tab\.[20](https://arxiv.org/html/2607.08093#A1.T20)\)\. For the former, a few effect\-interval estimands swing between near\-correct and the cap across restarts, and because intervals enter the score through a cappedmean, the swings pass through to the aggregate; for the latter, each flipped commit/abstain decision is re\-scored in a small pass/fail pool, so when several flips land the same way within a run, the Pass Rate shifts noticeably instead of averaging out\.

While we report the mean here \(with Kimi significantly improving its first\-try performance, mainly by getting a few of the above catastrophic mistakes right\), we use the first try for the main results Table[3](https://arxiv.org/html/2607.08093#S5.T3)so that the comparison with the frontier models is fair\.

Table 7:pass@kkand passˆkk\(solved by allkk\),*CausalDSScore*variability on the realistic exam for the open\-weight models\.pass@kk\(n=34n\{=\}34\)passˆ33CausalDSScoreModelk=1k\{=\}1k=2k\{=\}2k=3k\{=\}3\(all 3\)meanSDrel\. SDKimi K2\.667\.6%85\.3%91\.2%61\.8%0\.3750\.10227\.1%Qwen 3\.6 35B70\.6%73\.5%82\.4%55\.9%0\.4190\.0409\.5%Gemma 4 26B55\.9%70\.6%76\.5%50\.0%0\.6460\.0457\.0%

## 6Discussion

CausalDS pairs hidden SCMs and their synthetic, graph\-faithful verbalizations with SCM\-derived ground truth — including each target’s identifiability status — and an observation model that corrupts the released data view; thus, each exam scores abilities that existing benchmarks often isolate or conflate: symbolic causal reasoning, quantitative estimation, uncertainty quantification, epistemic abstention, and tool\-use efficiency\.

On the presented exam these axes demonstrably dissociate; however, we argue that all are integral to a well\-roundedcausal data\-science agent\.

Symbolic causal reasoning is the axis on which the field mostly converges: all six models recover structure essentially perfectly, and make the symbolic identification calls mostly without error\. The dissociation opens downstream, along several fault lines\. The frontier–open\-weight separation lives on the epistemic axes — uncertainty quantification and abstention — and on tool use/efficiency: the frontier models lead abstention outright, and under the hardest observation variant, only Claude Opus 4\.8 keeps the*mean*normalized error controlled\. In tool use/efficiency, the frontier is literal: the two model classes are linearly separable in the cost–quality plane \(Fig\.[4](https://arxiv.org/html/2607.08093#S5.F4)\)\. The reasoning–non\-reasoning line likewise shows up on the epistemic axes while sparing the numerics: Gemma 4 26B, the one non\-reasoning model, stays genuinely competitive at point estimation — it leads the open\-weight models — and is exposed specifically on abstention and interval calibration\. Finally, capability and reliability separate — given three attempts, the open\-weight models can reach frontier territory, yet fail to do so consistently — and this line, too, runs along the epistemic axes: the run\-to\-run variance concentrates in interval estimands and abstention flips, while repeated point estimates reproduce closely\.

The main future work is therefore to expand the benchmark along the many axes it already exposes: larger and more varied exam compositions, deeper trajectory\-level failure taxonomies, and more systematic stress tests via targeted exams for abstention, quantitative skills, and counterfactual reasoning\.

CausalDS thus poses a question that neither symbolic causal benchmarks nor data\-science benchmarks alone can answer: given a story, the structure it implies, and an imperfect view of the data, what is an agent’s answer worth? On the presented exam, the models largely master the parts — reading the structure, producing the estimates — and part ways on the whole: knowing how good the estimate is, and whether any answer is licensed at all\. A competent causal data\-science agent must clear every axis at once, and CausalDS is built to notice when it does not\. That causal data\-science competence decomposes this way is the empirical finding, with Claude Opus 4\.8 being the closest to a well\-roundedcausal data\-science agent\.

## References

- S\. Acharya, T\. J\. Zhang, A\. Kim, A\. Haghighat, X\. Sun, P\. Cobben, R\. B\. Shrestha, M\. Mordig, J\. T\. Emmerson, F\. Danisman, Y\. Chen, C\. Jose, A\. I\. Muresanu, J\. Cui, J\. Liu, Y\. Qi, P\. S\. Pandey, Y\. Huang, B\. Schölkopf, M\. Sachan, and Z\. Jin \(2026\)CauSciBench: a comprehensive benchmark on end\-to\-end causal inference for scientific research\.Note:Submitted to ICLR 2026External Links:[Link](https://openreview.net/forum?id=uQzPkWvTyo)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p3.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p4.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p5.4),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p7.3),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p8.1),[§4](https://arxiv.org/html/2607.08093#S4.p2.1)\.
- ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions\.arXiv\.Note:arXiv:2605\.08197 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2605.08197),[Document](https://dx.doi.org/10.48550/arXiv.2605.08197)Cited by:[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. Mirhoseini \(2024\)Large Language Monkeys: Scaling Inference Compute with Repeated Sampling\.arXiv\(en\-US\)\.Note:arXiv:2407\.21787 \[cs\.LG\], version 3External Links:[Link](https://arxiv.org/abs/2407.21787),[Document](https://dx.doi.org/10.48550/arXiv.2407.21787)Cited by:[§A\.11](https://arxiv.org/html/2607.08093#A1.SS11.SSS0.Px1.p1.18)\.
- L\. E\. J\. Bynum and K\. Cho \(2025\)Language Models as Causal Effect Generators\.\.EMNLP,pp\. 2096–2115\(da\)\.Note:arXiv:2411\.08019 \[cs\] version: 2External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.107),[Link](https://doi.org/10.18653/v1/2025.emnlp-main.107)Cited by:[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, A\. Madry, and L\. Weng \(2025\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.\.InICLR,Note:arXiv:2410\.07095 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.07095),[Link](https://openreview.net/forum?id=6s5uXNWGIh)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p1.1),[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating Large Language Models Trained on Code\.arXiv\(en\)\.Note:arXiv:2107\.03374 \[cs\.LG\]External Links:[Link](https://arxiv.org/abs/2107.03374),[Document](https://dx.doi.org/10.48550/arXiv.2107.03374)Cited by:[§A\.11](https://arxiv.org/html/2607.08093#A1.SS11.SSS0.Px1.p1.18),[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px5.p1.5)\.
- S\. Chen, M\. Xu, K\. Wang, X\. Zeng, R\. Zhao, S\. Zhao, and C\. Lu \(2024\)CLEAR: Can Language Models Really Understand Causal Graphs?\.InFindings of the Association for Computational Linguistics : EMNLP 2024,Miami, Florida, USA,pp\. 6247–6265\(da\)\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.363),[Link](https://aclanthology.org/2024.findings-emnlp.363)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- Z\. Chen, Y\. Chen, C\. Liu, J\. Yu, X\. Song, Z\. Li, J\. Li, P\. Torr, B\. Han, and K\. Zhang \(2026\)CausalGame: Benchmarking Causal Thinking of LLM Agents in Games\.InProceedings of the 43rd International Conference on Machine Learning \(ICML\),\(en\)\.Note:arXiv:2607\.04293 \[cs\]\. Oral presentationExternal Links:[Link](http://arxiv.org/abs/2607.04293),[Document](https://dx.doi.org/10.48550/arXiv.2607.04293)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p5.4),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2607.08093#S4.p2.1)\.
- J\. DeYoung, E\. Lehman, B\. Nye, I\. Marshall, and B\. C\. Wallace \(2020\)Evidence inference 2\.0: more data, better models\.InProceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing,Online,pp\. 123–132\.External Links:[Link](https://www.aclweb.org/anthology/2020.bionlp-1.13)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p7.3),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p9.6)\.
- J\. Du, L\. Chen, X\. Xian, A\. Luo, F\. Tian, G\. Wang, C\. Doss, X\. Shen, and J\. Ding \(2026\)Ice cream doesn’t cause drowning: benchmarking LLMs against statistical pitfalls in causal inference\.InThe Fourteenth International Conference on Learning Representations,Note:arXiv:2505\.13770 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.13770),[Link](https://openreview.net/forum?id=MGMG7yQ18v)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p5.4),[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- E\. L\. Epstein, J\. Winnicki, T\. Sornwanee, and R\. Dwaraknath \(2025\)LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval\.arXiv\.Note:arXiv:2510\.26995 \[stat\.ME\] version: 1External Links:[Link](http://arxiv.org/abs/2510.26995),[Document](https://dx.doi.org/10.48550/arXiv.2510.26995)Cited by:[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p2.7)\.
- T\. Gneiting and A\. E\. Raftery \(2007\)Strictly Proper Scoring Rules, Prediction, and Estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\(en\)\.External Links:ISSN 0162\-1459, 1537\-274X,[Link](http://www.tandfonline.com/doi/abs/10.1198/016214506000001437),[Document](https://dx.doi.org/10.1198/016214506000001437)Cited by:[§A\.9](https://arxiv.org/html/2607.08093#A1.SS9.p1.17),[§3\.7](https://arxiv.org/html/2607.08093#S3.SS7.p2.1)\.
- K\. Gu, R\. Shang, R\. Jiang, K\. Kuang, R\. Lin, D\. Lyu, Y\. Mao, Y\. Pan, T\. Wu, J\. Yu, Y\. Zhang, T\. M\. Zhang, L\. Zhu, M\. A\. Merrill, J\. Heer, and T\. Althoff \(2024\)BLADE: benchmarking language model agents for data\-driven science\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 13936–13971\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.815),[Link](https://aclanthology.org/2024.findings-emnlp.815/)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p1.1),[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- P\. He, Y\. Huang, M\. Sachan, and Z\. Jin \(2026\)Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification\.CoRRabs/2601\.21210\(en\-US\)\.Note:arXiv:2601\.21210 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.21210),[Link](http://arxiv.org/abs/2601.21210)Cited by:[§3\.7](https://arxiv.org/html/2607.08093#S3.SS7.p1.1)\.
- S\. Heindorf, Y\. Scholten, H\. Wachsmuth, A\. N\. Ngomo, and M\. Potthast \(2020\)CauseNet\.InProceedings of the 29th ACM International Conference on Information & Knowledge Management,Virtual Event Ireland,pp\. 3023–3030\(en\-US\)\.External Links:[Document](https://dx.doi.org/10.1145/3340531.3412763),ISBN 978\-1\-4503\-6859\-9,[Link](https://doi.org/10.1145/3340531.3412763)Cited by:[2nd item](https://arxiv.org/html/2607.08093#S1.I1.i2.p1.1),[§3\.3](https://arxiv.org/html/2607.08093#S3.SS3.p1.1)\.
- L\. Hobor, M\. Brcic, M\. Kovac, and K\. Poje \(2026\)Bayesian Elicitation with LLMs: Model Size Helps, Extra "Reasoning" Doesn’t Always\.arXiv\.Note:arXiv:2604\.01896 \[cs\.AI\] version: 1External Links:[Link](http://arxiv.org/abs/2604.01896),[Document](https://dx.doi.org/10.48550/arXiv.2604.01896)Cited by:[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p2.7)\.
- Z\. Jin, Y\. Chen, F\. Leeb, L\. Gresele, O\. Kamal, Z\. Lyu, K\. Blin, F\. G\. Adauto, M\. Kleiman\-Weiner, M\. Sachan, and B\. Schölkopf \(2023\)CLadder: Assessing Causal Reasoning in Language Models\.arXiv\(da\)\.Note:arXiv:2312\.04350 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2312.04350),[Link](http://arxiv.org/abs/2312.04350)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p4.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p7.3),[§1](https://arxiv.org/html/2607.08093#S1.p1.1),[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.p2.1)\.
- L\. Jing, Z\. Huang, X\. W\. 0001, W\. Yao, W\. Y\. 0002, K\. Ma, H\. Z\. 0009, X\. Du, and D\. Y\. 0001 \(2025\)DSBench: how far are data science agents from becoming data science experts?\.InICLR,\(en\-US\)\.Note:arXiv:2409\.07703 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2409.07703),[Link](https://openreview.net/forum?id=DSsSPr0RZJ)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p1.1),[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- C\. Kadelka, T\. Butrie, E\. Hilton, J\. Kinseth, A\. Schmidt, and H\. Serdarevic \(2024\)A meta\-analysis of boolean network models reveals design principles of gene regulatory networks\.Science Advances10\(2\),pp\. eadj0822\.External Links:[Document](https://dx.doi.org/10.1126/sciadv.adj0822),[Link](https://doi.org/10.1126/sciadv.adj0822)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p6.3)\.
- Y\. Lai, C\. Li, Y\. Wang, T\. Zhang, R\. Zhong, L\. Zettlemoyer, S\. Yih, D\. Fried, S\. Wang, and T\. Yu \(2022\)DS\-1000: a natural and reliable benchmark for data science code generation\.International Conference on Machine Learning,pp\. 18319–18345\.Note:arXiv:2211\.11501 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arxiv.2211.11501),[Link](https://doi.org/10.48550/arxiv.2211.11501)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- Y\. K\. Lal, N\. Chambers, R\. Mooney, and N\. Balasubramanian \(2021\)TellMeWhy: a dataset for answering why\-questions in narratives\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,Online,pp\. 596–610\.External Links:[Link](https://aclanthology.org/2021.findings-acl.53),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.53)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p9.6)\.
- D\. Lee, H\. Yun, M\. Cha, S\. Park, S\. Park, and J\. Kim \(2026\)EconCausal: A Context\-Aware Causal Reasoning Benchmark for Large Language Models in Social Science\.arXiv\.Note:arXiv:2510\.07231 \[cs\] version: 3External Links:[Link](http://arxiv.org/abs/2510.07231),[Document](https://dx.doi.org/10.48550/arXiv.2510.07231)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p3.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p4.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p5.4),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p7.3)\.
- X\. Liu, Z\. Wu, X\. Wu, P\. Lu, K\. Chang, and Y\. Feng \(2024\)Are llms capable of data\-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data\.InFindings of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/2402.17644)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p3.1)\.
- B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. Dalvi Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. Clark \(2025\)DiscoveryBench: towards data\-driven discovery with large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vyflgpwfJW)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p1.1),[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- R\. S\. Malik\-Sheriff, M\. Glont, T\. V\. N\. Nguyen, K\. Tiwari, M\. G\. Roberts, A\. Xavier, M\. T\. Vu, J\. Men, M\. Maire, S\. Kananathan, E\. L\. Fairbanks, J\. P\. Meyer, C\. Arankalle, T\. M\. Varusai, V\. Knight\-Schrijver, L\. Li, C\. Dueñas\-Roca, G\. Dass, S\. M\. Keating, Y\. M\. Park, N\. Buso, N\. Rodriguez, M\. Hucka, and H\. Hermjakob \(2020\)BioModels: 15 years of sharing computational models in life science\.Nucleic Acids Research48\(D1\),pp\. D407–D415\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkz1055),[Link](https://doi.org/10.1093/nar/gkz1055)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p6.3)\.
- Y\. Meyer and D\. Corneil \(2025\)Nemotron\-Personas\-USA: synthetic personas aligned to real\-world distributions\.External Links:[Link](https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA)Cited by:[§3\.1](https://arxiv.org/html/2607.08093#S3.SS1.p1.1)\.
- W\. Miao, Z\. Geng, and E\. J\. Tchetgen Tchetgen \(2018\)Identifying causal effects with proxy variables of an unmeasured confounder\.Biometrika105\(4\),pp\. 987–993\(en\)\.External Links:[Document](https://dx.doi.org/10.1093/biomet/asy038),[Link](https://dx.doi.org/10.1093/biomet/asy038)Cited by:[§A\.12](https://arxiv.org/html/2607.08093#A1.SS12.SSS0.Px5.p1.3),[§3\.2](https://arxiv.org/html/2607.08093#S3.SS2.p3.1)\.
- P\. Panayiotou, A\. Poinsot, A\. Leite, N\. Chesneau, M\. Schoenauer, and Ö\. Şimşek \(2026\)CausalProfiler: Generating Synthetic Benchmarks for Rigorous and Transparent Evaluation of Causal Machine Learning\.arXiv\.Note:arXiv:2511\.22842 \[cs\]External Links:[Link](http://arxiv.org/abs/2511.22842),[Document](https://dx.doi.org/10.48550/arXiv.2511.22842)Cited by:[§4](https://arxiv.org/html/2607.08093#S4.p2.1)\.
- S\. Pastva, D\. Safranek,et al\.\(2023\)Repository of logically consistent real\-world boolean network models\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2023.06.12.544361),[Link](https://doi.org/10.1101/2023.06.12.544361)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p6.3)\.
- J\. Pearl, M\. Glymour, and N\. P\. Jewell \(2021\)Causal inference in statistics: a primer\.Reprinted with revisions edition,Wiley,Chichester\(en\-US\)\.External Links:ISBN 978\-1\-119\-18684\-7Cited by:[§2](https://arxiv.org/html/2607.08093#S2.SS0.SSS0.Px1.p1.14),[§2](https://arxiv.org/html/2607.08093#S2.SS0.SSS0.Px2.p1.3),[§2](https://arxiv.org/html/2607.08093#S2.SS0.SSS0.Px2.p2.1)\.
- J\. Pearl \(2022\)Causality: models, reasoning, and inference\.Second edition, reprinted with corrections edition,Cambridge University Press,Cambridge New York, NY Port Melbourne New Delhi Singapore\(eng\)\.External Links:ISBN 978\-0\-521\-89560\-6Cited by:[§2](https://arxiv.org/html/2607.08093#S2.SS0.SSS0.Px2.p2.1)\.
- R\. Qiang, Y\. Zhuang, Y\. Li, D\. S\. V\. K, R\. Zhang, C\. Li, I\. S\. Wong, S\. Yang, P\. Liang, C\. Zhang, and B\. Dai \(2025\)MLE\-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering\.arXiv\(no\)\.Note:arXiv:2505\.07782 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.07782),[Link](http://arxiv.org/abs/2505.07782)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- A\. Renda, J\. Ross, M\. Cafarella, and J\. Andreas \(2025\)OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real\-World Data\.arXiv\.Note:arXiv:2510\.15096 \[cs\.AI\] version: 1External Links:[Link](http://arxiv.org/abs/2510.15096),[Document](https://dx.doi.org/10.48550/arXiv.2510.15096)Cited by:[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p2.7)\.
- R\. Saklad, A\. Chadha, O\. Pavlov, and R\. Moraffah \(2025\)Can Large Language Models Infer Causal Relationships from Real\-World Text?\.arXiv\.Note:arXiv:2505\.18931 \[cs\] version: 2External Links:[Link](http://arxiv.org/abs/2505.18931),[Document](https://dx.doi.org/10.48550/arXiv.2505.18931)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p4.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p9.6)\.
- A\. Sawarni, J\. Tan, and V\. Syrgkanis \(2026\)CausalReasoningBenchmark: A Real\-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation\.arXiv\.Note:arXiv:2602\.20571 \[cs\]External Links:[Link](http://arxiv.org/abs/2602.20571),[Document](https://dx.doi.org/10.48550/arXiv.2602.20571)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p3.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p4.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p7.3),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p8.1),[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p9.6),[§4](https://arxiv.org/html/2607.08093#S4.p2.1)\.
- A\. Sharma and E\. Kiciman \(2020\)DoWhy: An End\-to\-End Library for Causal Inference\.arXiv\(en\-US\)\.Note:arXiv:2011\.04216 \[stat\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2011.04216),[Link](http://arxiv.org/abs/2011.04216)Cited by:[§3\.4](https://arxiv.org/html/2607.08093#S3.SS4.p2.1)\.
- I\. Sheth, B\. Fatemi, and M\. Fritz \(2024\)CausalGraph2LLM: evaluating llms for causal queries\.North American Chapter of the Association for Computational Linguistics\(en\-US\)\.Note:arXiv:2410\.15939 \[cs\] version: 2External Links:[Document](https://dx.doi.org/10.48550/arxiv.2410.15939),[Link](https://doi.org/10.48550/arxiv.2410.15939)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- S\. Shi, Z\. Shi, L\. Zheng, X\. Su, A\. Xie, B\. Lv, R\. Xu, Z\. Chen, Z\. Chen, G\. Liu, N\. Zhang, M\. Dong, Z\. Quan, B\. Chen, T\. Hao, Y\. Qi, Y\. Xu, and L\. Wu \(2026\)InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems\.arXiv\.Note:arXiv:2603\.15542 \[cs\] version: 1External Links:[Link](http://arxiv.org/abs/2603.15542),[Document](https://dx.doi.org/10.48550/arXiv.2603.15542)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p4.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px1.p1.1)\.
- I\. Shpitser and J\. Pearl \(2008\)Complete Identification Methods for the Causal Hierarchy\.Journal of Machine Learning Research9,pp\. 1941–1979\(en\)\.External Links:[Link](https://jmlr.org/papers/v9/shpitser08a.html)Cited by:[§2](https://arxiv.org/html/2607.08093#S2.SS0.SSS0.Px2.p1.3),[§2](https://arxiv.org/html/2607.08093#S2.SS0.SSS0.Px2.p2.1)\.
- E\. J\. T\. Tchetgen, A\. Ying, Y\. Cui, X\. Shi, and W\. Miao \(2024\)An Introduction to Proximal Causal Inference\.Statistical Science39\(3\),pp\. 375–390\(en\)\.Note:arXiv:2009\.10982 \[stat\.ME\]External Links:ISSN 0883\-4237, 2168\-8745,[Link](https://projecteuclid.org/journals/statistical-science/volume-39/issue-3/An-Introduction-to-Proximal-Causal-Inference/10.1214/23-STS911.full),[Document](https://dx.doi.org/10.1214/23-STS911)Cited by:[§A\.12](https://arxiv.org/html/2607.08093#A1.SS12.SSS0.Px5.p1.3),[§3\.2](https://arxiv.org/html/2607.08093#S3.SS2.p3.1)\.
- A\. Vashishtha, Q\. Dai, H\. Mei, A\. Sharma, C\. Tan, and H\. Peng \(2026\)EXECUTABLE COUNTERFACTUALS: IMPROVING LLMS’\.\(en\)\.Cited by:[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- V\. Verma, S\. Acharya, D\. Bhardwaj, S\. Simko, Y\. Yang, A\. Haghighat, D\. Janzing, M\. Sachan, B\. Schölkopf, and Z\. Jin \(2025\)Causal AI Scientist: Facilitating Causal Data Science with Large Language Models\.\(en\-US\)\.External Links:[Link](https://openreview.net/forum?id=EDWTHMVOCj)Cited by:[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- Z\. Wang, Q\. Chang, H\. Patel, S\. Biju, C\. Wu, Q\. Liu, A\. Ding, A\. Rezazadeh, A\. Shah, Y\. Bao, and E\. Siow \(2025\)MCP\-Bench: Benchmarking Tool\-Using LLM Agents with Complex Real\-World Tasks via MCP Servers\.arXiv\.Note:arXiv:2508\.20453 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2508.20453),[Document](https://dx.doi.org/10.48550/arXiv.2508.20453)Cited by:[§A\.10](https://arxiv.org/html/2607.08093#A1.SS10.SSS0.Px7.p2.1),[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p3.6)\.
- U\. Wittig, M\. Rey, A\. Weidemann, R\. Kania, and W\. Muller \(2018\)SABIO\-RK: an updated resource for manually curated biochemical reaction kinetics\.Nucleic Acids Research\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkx1065),[Link](https://doi.org/10.1093/nar/gkx1065)Cited by:[§A\.7](https://arxiv.org/html/2607.08093#A1.SS7.p6.3)\.
- X\. Xu, R\. Yang, H\. Shen, W\. Xu, B\. Gao, R\. Wu, K\. Shi, W\. Xie, X\. Chen, M\. Wu, J\. Zeng, M\. Heinrich, E\. Zhang, L\. Chen, K\. Li, and B\. Chang \(2026a\)RoadmapBench: Evaluating Long\-Horizon Agentic Software Development Across Version Upgrades\.arXiv\.Note:arXiv:2605\.15846 \[cs\.SE\] version: 1External Links:[Link](http://arxiv.org/abs/2605.15846),[Document](https://dx.doi.org/10.48550/arXiv.2605.15846)Cited by:[§A\.10](https://arxiv.org/html/2607.08093#A1.SS10.SSS0.Px7.p2.1),[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p3.6)\.
- Z\. Xu, J\. Chen, Y\. Huang, D\. Jiang, J\. Chen, H\. Hua, Z\. Wu, Z\. Liu, Z\. He, L\. Li, S\. Diao, J\. Pei, J\. Yoon, H\. Zhang, M\. Wang, R\. Poovendran, M\. Sra, A\. Pentland, and Z\. Chen \(2026b\)AutoLab: Can Frontier Models Solve Long\-Horizon Auto Research and Engineering Tasks?\.arXiv\.Note:arXiv:2606\.05080 \[cs\.AI\]External Links:[Link](http://arxiv.org/abs/2606.05080),[Document](https://dx.doi.org/10.48550/arXiv.2606.05080)Cited by:[§A\.10](https://arxiv.org/html/2607.08093#A1.SS10.SSS0.Px7.p2.1),[§A\.15](https://arxiv.org/html/2607.08093#A1.SS15.SSS0.Px3.p1.4),[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p3.6)\.
- Z\. Xu and Y\. Fu \(2026\)NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise\.arXiv\.Note:arXiv:2605\.04313 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2605.04313),[Document](https://dx.doi.org/10.48550/arXiv.2605.04313)Cited by:[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. A\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.Neural Information Processing Systems\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2405.15793),[Link](https://doi.org/10.48550/arxiv.2405.15793)Cited by:[§3\.6](https://arxiv.org/html/2607.08093#S3.SS6.p1.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- J\. Yang, A\. Prabhakar, K\. Narasimhan, and S\. Yao \(2023\)InterCode: standardizing and benchmarking interactive coding with execution feedback\.Neural Information Processing Systems\.Note:arXiv:2306\.14898 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arxiv.2306.14898),[Link](https://doi.org/10.48550/arxiv.2306.14898)Cited by:[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1)\.
- J\. Yang, D\. Zhang, X\. Song, Q\. Dai, X\. Liu, Y\. Chen, A\. Vashishtha, J\. Shi, C\. Tan, and H\. Peng \(2026\)CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists\.arXiv\.Note:arXiv:2605\.26029 \[cs\.AI\]External Links:[Link](http://arxiv.org/abs/2605.26029),[Document](https://dx.doi.org/10.48550/arXiv.2605.26029)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2607.08093#S4.p2.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-bench: A Benchmark for Tool\-Agent\-User Interaction in Real\-World Domains\.arXiv\(en\-US\)\.Note:arXiv:2406\.12045 \[cs\]External Links:[Link](http://arxiv.org/abs/2406.12045),[Document](https://dx.doi.org/10.48550/arXiv.2406.12045)Cited by:[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px5.p1.5)\.
- Z\. Yu and S\. Zhou \(2026\)Caliper: Probing Lexical Anchors versus Causal Structure in LLMs\.arXiv\.Note:arXiv:2606\.04915 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2606.04915),[Document](https://dx.doi.org/10.48550/arXiv.2606.04915)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1)\.
- M\. Zecevic, M\. Willig, D\. Dhami, and K\. Kersting \(2023\)Causal parrots: large language models may talk causality but are not causal\.Trans\. Mach\. Learn\. Res\.2023\(ca\)\.Note:arXiv:2308\.13067 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arxiv.2308.13067),[Link](https://doi.org/10.48550/arxiv.2308.13067)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p1.1),[§1](https://arxiv.org/html/2607.08093#S1.p3.1)\.
- Y\. Zhang, S\. Jiang, R\. Li, J\. Tu, Y\. Su, L\. Deng, X\. Guo, C\. Lv, and J\. Lin \(2026\)DeepPlanning: Benchmarking Long\-Horizon Agentic Planning with Verifiable Constraints\.arXiv\.Note:arXiv:2601\.18137 \[cs\.AI\] version: 1External Links:[Link](http://arxiv.org/abs/2601.18137),[Document](https://dx.doi.org/10.48550/arXiv.2601.18137)Cited by:[§A\.10](https://arxiv.org/html/2607.08093#A1.SS10.SSS0.Px7.p2.1),[§5](https://arxiv.org/html/2607.08093#S5.SS0.SSS0.Px2.p3.6)\.
- Y\. Zhou, X\. Wu, B\. Huang, J\. Wu, L\. Feng, and K\. C\. Tan \(2024\)CausalBench: A Comprehensive Benchmark for Causal Learning Capability of LLMs\.Note:arXiv:2404\.06349 \[cs\]External Links:[Link](http://arxiv.org/abs/2404.06349),[Document](https://dx.doi.org/10.48550/arXiv.2404.06349)Cited by:[§1](https://arxiv.org/html/2607.08093#S1.p3.1),[§4](https://arxiv.org/html/2607.08093#S4.SS0.SSS0.Px2.p1.1)\.

## Appendix AAppendix

### A\.1Code and data availability

The CausalDS source code — the scene\-generation pipeline, the evaluation harness, and the grader — is available at[github\.com/andleb/causalds](https://github.com/andleb/causalds)\. The same repository also provides the entire datasets used for the main exam and the ablations presented in this work\.

### A\.2Scene generation details

##### Noise families and link functions\.

The continuous and binary SCM profile registries are documented in App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)\. For the noise: continuous noise families include Gaussian, Laplace, Student\-tt, and Gaussian mixtures; binary mechanisms produce a Bernoulli output through a logistic, threshold, or noisy\-gate link with no explicit noise term — the Bernoulli draw is the source of stochasticity\.

### A\.3Scene synthesis sub\-algorithms

The top\-level scene\-synthesis control flow \(Alg\.[1](https://arxiv.org/html/2607.08093#alg1)\) calls three sub\-routines:PreAudit\(Alg\.[2](https://arxiv.org/html/2607.08093#alg2)\) for the LLM\-based feasibility check on CauseNet seeding,MapStage\(Alg\.[3](https://arxiv.org/html/2607.08093#alg3)\) for naming the \(sub\)graph variables with audit/repair, andVerbalize\(Alg\.[4](https://arxiv.org/html/2607.08093#alg4)\) for story generation with a story\-to\-DAG verifier loop\.

Algorithm 2LLM pre\-audit for CauseNet seeding1:Input:stage graph

HHwith CauseNet\-fixed nodes

FF\.

2:Output:feasibility verdict

\(feasible,confidence,reason\)\(\\texttt\{feasible\},\\texttt\{confidence\},\\texttt\{reason\}\)\.

3:Extract directed edges

EE, all unordered non\-edges

𝒩\\mathcal\{N\}, and the fixed–fixed subset

𝒩F⊆𝒩\\mathcal\{N\}\_\{F\}\\subseteq\\mathcal\{N\}\.

4:Prompt with fixed nodes, remaining placeholders,

EE,

𝒩F\\mathcal\{N\}\_\{F\}, and

𝒩\\mathcal\{N\}\.

5:Ask whether a skilled mapper could interpret the fixed concepts and name placeholders so the graph constraints remain plausible\.

6:Reject iff a fixed–fixed non\-edge has an unavoidable mainstream direct\-cause reading, or a fixed–fixed edge is impossible\.

7:Otherwise accept, usinglowconfidence when only narrow mainstream interpretations make the constraints work; uncertain cases default to feasible \(as the following stages can still reject them\)

8:returnthe feasibility verdict\.

Algorithm 3MapStage: name the \(sub\)graph variables with audit/repair1:Input:\(sub\)graph

HH; fixed\-name assignment

FF\(possibly empty\); audit retry budget

TT\.

2:Output:mapping

MMfor

HH, or failure\.

3:The mapper proposes

MMunder fixed\-name, completeness, and uniqueness constraints\.

4:for

t=1t=1to

TTdo

5:The auditor checks edge plausibility, absence of plausible direct links for non\-edges, and conditional independence/type\-consistency signals\.

6:ifno violations remainthen

7:return

MM
8:endif

9:ifa violation cannot be fixed without changing fixed namesthen

10:returnfailure

11:endif

12:The mapper regenerates

MMfrom a feedback prompt provided by the auditor, listing the violations\.

13:endfor

14:returnfailure\.

Algorithm 4Verbalize: story generation with story\-to\-DAG verifier loop1:Input:domain

DD, mapping

MM, DAG

GG, story retry budget

SS\.

2:Output:verified story

σ\\sigma, or failure\.

3:Build yellow\-flag context

Φ\\Phiby scanning

MMfor derived\-name markers and restrictive qualifiers \(cf\. Sec[A\.4](https://arxiv.org/html/2607.08093#A1.SS4)\)

4:The story writer drafts

σ\\sigmafrom

\(D,M,Φ\)\(D,M,\\Phi\)\.

5:for

s=1s=1to

SSdo

6:The verifier checks

σ\\sigmaagainst

\(G,M\)\(G,M\): missing variables, missing edges, and direction contradictions are hard violations; extra direct effects, domain drift, graph jargon, and plausibility/coherence issues are warnings\.

7:ifthere are no hard violationsthen

8:return

σ\\sigma
9:endif

10:The story writer revises

σ\\sigmafrom the verifier’s feedback listing the hard violations, warnings, and

Φ\\Phi\.

11:endfor

12:returnfailure\.

### A\.4Mapping and narration audit details

Both the mapping and story auditors are augmented with a deterministic preprocessor that scans variable meanings for derived\-name markers \(e\.g\. “corrected”, “residual”, “count of”, “index based on”\) and restrictive qualifiers \(e\.g\. “based only on”, “not influenced by other factors”\), and appends a yellow\-flag block pointing the auditor at the suspicious nodes\. This targets a recurring failure mode in which a broad latent construct is narrated as a direct cause of an administrative artifact \(e\.g\. a construct narrated as directly causing a record ID meant to keep track of it\)\.

### A\.5Motif catalog

Fig\.[5](https://arxiv.org/html/2607.08093#A1.F5)shows the base motif templates used in this work\. When grafting, these are the pre\-augmentation graph skeletons used by the sampler\. Gray dashed nodes denote latent variables\.

XMY\(a\)chainXMY\(b\)mediationZXY\(c\)confoundingZXY\(d\)forkXYZ\(e\)colliderV1XV2Y\(f\)arrowheadXM1M2Y\(g\)diamondV1XV2Y\(h\)diamondcutUXZY\(i\)frontdoorZUXY\(j\)ivUAYZW\(k\)double\_ncXYV3\(l\)triangle
Figure 5:Base motif templates used by the sampler\. Latent variables are shown as gray dashed nodes\.
### A\.6Observation\-model implementation details

##### Mechanism families and noise\.

Continuous measurement mechanismshrh\_\{r\}are drawn from the same families used for SCM nodes: a handcrafted formhr​\(z\)=w​z\+b\+wnl​tanh⁡\(z−c\)\+wpair​z2h\_\{r\}\(z\)=wz\+b\+w\_\{\\mathrm\{nl\}\}\\tanh\(z\-c\)\+w\_\{\\mathrm\{pair\}\}~z^\{2\}\(linear plus shifted nonlinearity plus interaction\) and a small spectral\-normalized neural network\. The shiftc∼U​\(−1\.5,1\.5\)c\\sim U\(\-1\.5,1\.5\)is drawn independently per measurement and moves the cancellation point of thetanh\\tanhderivative away from the realized treatment\-contrast region, wherehr′​\(z\)≈0h^\{\\prime\}\_\{r\}\(z\)\\\!\\approx\\\!0would otherwise create a flat region with vanishing Fisher information \(cf\. “Admissibility checks” below\)\. The continuous noise family and binary link function selections are drawn from the same pool as for the main SCM \(cf\. App\.[A\.2](https://arxiv.org/html/2607.08093#A1.SS2)\)\. When the noise is chosen to beheteroscedastic,σ\\sigmais replaced withσ​\(𝐳\)=softplus​\(wσ⊤​𝐳\+bσ\)\\sigma\(\\mathbf\{z\}\)=\\mathrm\{softplus\}\(w\_\{\\sigma\}^\{\\top\}\\mathbf\{z\}\+b\_\{\\sigma\}\), clamped to\[σmin,σmax\]\[\\sigma\_\{\\min\},\\sigma\_\{\\max\}\], so the Fisher denominator is location\-dependent:Ir​\(z\)=\(hr′​\(z\)\)2/\(σ0​σ​\(z\)\)2I\_\{r\}\(z\)=\(h^\{\\prime\}\_\{r\}\(z\)\)^\{2\}/\(\\sigma\_\{0\}\\,\\sigma\(z\)\)^\{2\}\.

##### Admissibility checks on candidate bundles\.

For each candidate measurement bundle on a latentZZ, we evaluateI​\(z\)=∑rIr​\(z\)I\(z\)=\\sum\_\{r\}I\_\{r\}\(z\)on a grid over the realized support ofZZand accept the bundle only if: \(i\)minz⁡I​\(z\)≥τmin\\min\_\{z\}I\(z\)\\geq\\tau\_\{\\min\}\(no dead zones\), \(ii\)I¯≥τavg\\bar\{I\}\\geq\\tau\_\{\\mathrm\{avg\}\}\(sufficient average\), and \(iii\) lower\-tail quantiles \(I0\.10I\_\{0\.10\},I0\.25I\_\{0\.25\}\) clear separate thresholds\. Scenes that also place a measurement bundle on the outcome apply the same checks with stricter thresholds\. The resampler retries the mechanism draw up to a configured budget; persistently failing scenes are rejected before any downstream verbalization\. The setting used for the results presented isτmin=0\.03\\tau\_\{\\min\}=0\.03\.

##### Recoverability diagnostics in the private bundle\.

Beyond Fisher\-information filtering, we privately characterize each scene by recoverability diagnostics computed on a shared holdout, expressed asR2R^\{2\}for continuous conceptual variables and AUC for binary ones so that scores are comparable across scenes\. The*upper bound*\(intrinsic recoverability\) trains a gradient\-boosting model on alldjd\_\{j\}measurements jointly using the full conceptual\-plus\-measurement dataset—labels no benchmark agent ever sees—giving anoracle\-likeceiling\. The*lower bound*\(naive baseline\) fits the best single\-measurementlinearmodel from the public calibration set only \(OLS for continuous targets, logistic regression for binary\)\. The gap between the two bounds decomposes scene difficulty into a misspecification component and a multi\-measurement information\-gain component:

Δ=\(ℒnaive−ℒsingle,oracle\)⏟misspecification gap\+\(ℒsingle,oracle−ℒcal\)⏟multi\-measurement information gain,\\Delta\\;=\\;\\underbrace\{\(\\mathcal\{L\}\_\{\\mathrm\{naive\}\}\-\\mathcal\{L\}\_\{\\mathrm\{single,oracle\}\}\)\}\_\{\\text\{misspecification gap\}\}\\;\+\\;\\underbrace\{\(\\mathcal\{L\}\_\{\\mathrm\{single,oracle\}\}\-\\mathcal\{L\}\_\{\\mathrm\{cal\}\}\)\}\_\{\\text\{multi\-measurement information gain\}\},\(5\)whereℒsingle,oracle\\mathcal\{L\}\_\{\\mathrm\{single,oracle\}\}is the best single\-measurementnonlinearloss\. A large gap implies large potential gains from a skilled data\-science agent; a small gap means a single column captures most of the recoverable signal\. The upper bound itself caps even an oracle’s downstream performance\. Both bounds and their per\-variable gaps are recorded for analyis\. However, these recoverability scores are diagnostic summaries rather than hard acceptance criteria; admissibility is still determined at the mechanism level, as described above\. Fig\.[6](https://arxiv.org/html/2607.08093#A1.F6)shows the resulting per\-scene summary on a representative scene\.

![Refer to caption](https://arxiv.org/html/2607.08093v1/x4.png)Figure 6:Per\-scene recoverability diagnostic for theproxy\_hardvariant of one scene with four measured conceptual variables \(three continuous, one binary\)\. Light blue bars show the naive lower bound \(best single\-measurement linear/logistic fit on the public calibration set\); dark blue bars show the calibrated upper bound \(gradient\-boosting fit on the full latent\-plus\-measurement data the agent never sees\)\. The shaded band marks the target window \(0\.30\.3–0\.80\.8\) used during scene generation: below it is “too hard,” above it “too easy\.” TheΔ\\Deltanext to each pair is the upper\-minus\-lower gap \(Eq\.[5](https://arxiv.org/html/2607.08093#A1.E5)\)\.

### A\.7Optional empirical grounding of composition axes

CausalDS is fully synthetic at the scene level, but a number of*composition axes*can be grounded from external corpora when we want a specific benchmark mixture to better resemble real causal\-analysis workloads\. The key design choice is to ground*axes*, not to imitate any one source wholesale: we derive axis\-specific empirical anchors from several benchmark corpora and structural companions, weight them by how directly they expose the quantity of interest, and then allow small benchmark\-design tilts when needed for difficulty, coverage, or stress\-testing\. The composition axes are split into a*generation*side and an*exam*side\. Generation axes \(variable types; main\-motif weights, auxiliary\-motif weights, and graft probability; identifiability proportion; typed SCM mechanism profiles; observation\-model profile composition\) determine which scenes get sampled and built; exam axes \(P​\(QuestionType∣structural label\)P\(\\text\{QuestionType\}\\mid\\text\{structural label\}\),P​\(OutputVariant∣QuestionType\)P\(\\text\{OutputVariant\}\\mid\\text\{QuestionType\}\),P​\(observation variant∣QuestionType\)P\(\\text\{observation variant\}\\mid\\text\{QuestionType\}\)\) shape which tasks and which released measurement view are selected per scene at exam time\. This split lets the same scene support multiple released exams without re\-running expensive verbalization\.

Because we cannot estimate the full jointP​\(motif,question type,output variant,observation variant\)P\(\\text\{motif\},\\text\{question type\},\\text\{output variant\},\\text\{observation variant\}\)from available data, we factor theexam sidethrough the question type\. Conditional on the question typeQQ, the output variant is drawn from the answer\-contract families compatible withQQand the observation variant from the per\-question\-type observation distribution, with the two choices treated as independent givenQQ:

P\(S,Q,output,obs\)=P\(S\)P\(Q∣S\)P\(output∣Q\)P\(obs∣Q\),output⟂⟂obs∣Q\.P\(S,Q,\\text\{output\},\\text\{obs\}\)=P\(S\)\\,P\(Q\\mid S\)\\,P\(\\text\{output\}\\mid Q\)\\,P\(\\text\{obs\}\\mid Q\),\\qquad\\text\{output\}\\perp\\\!\\\!\\\!\\\!\\perp\\text\{obs\}\\mid Q\.The corresponding graphical factorization is shown in Fig\.[7](https://arxiv.org/html/2607.08093#A1.F7)\.

Structural labelSS\(motif / grafted bucket\)Question typeQQOutput variantObservation variant

Figure 7:Exam\-composition factorization\. Conditional on the structural labelSS\(motif or the syntheticgraftedbucket\), the exam builder draws the question typeQQ, then independently draws the output variant and the observation variant givenQQ\. Each conditional corresponds to one exam\-side axis:P​\(Q∣S\)P\(Q\\mid S\)is Exam Axis 1,P​\(output∣Q\)P\(\\text\{output\}\\mid Q\)is Exam Axis 2, andP​\(obs∣Q\)P\(\\text\{obs\}\\mid Q\)is Exam Axis 3\.Forvariable types, we treat treatment and outcome separately and estimate only the coarse continuous\-versus\-non\-continuous split that matters for scene generation\. The empirical anchor comes from row\-level causal benchmarks whose treatments and outcomes can be typed from data columns, metadata, or variable descriptions:EconCausal,CauSciBench,CausalReasoningBenchmark, andCauSciBench’s QR/textbook source\(Leeet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib107); Acharyaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib81); Sawarniet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib93); Liuet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib114)\)\. We then combine those sources using credibility weighting based on, e\.g\., how well the dataset’s constructs map to ours\. In production we nudge this empirical mixture slightly toward more continuous variables: some high\-cardinality discrete variables are better modeled as effectively continuous in downstream estimation, and a modest tilt increases data\-analysis difficulty without changing the underlying causal semantics\.

Formotif composition and grafting, real\-study causal benchmarks \(CausalReasoningBenchmark,CauSciBench,EconCausal, andInterveneBench\) typically expose*design\-family*labels \(e\.g\., adjusted observational, DiD/event\-study, RD, IV, matching\) rather than explicit graph motifs\. We therefore map those design\-family histograms into our motif inventory through an explicit crosswalk, producing source\-specific motif distributions first and deferring any global mixture to a separate weighting step\(Sawarniet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib93); Acharyaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib81); Leeet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib107); Shiet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib111)\)\. We use real\-paper graph corpora \(ReCITE\) differently: they are not benchmark\-side design\-family sources, but they do expose explicit graph structure, which lets us calibrate the split over observed\-effect motifs, auxiliary motif tendencies, and graft\-complexity indicators\(Sakladet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib110)\)\. Synthetic motif benchmarks \(CLadder\) can still be useful here, but only as additions to ensure coverage, rather than frequency priors\(Jinet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib24)\)\. The auxiliary motif pool and graft\-count distribution are derived separately\. The graft\-count prior is hand\-authored with a small empirical tail\-pull rather than taken from corpus frequencies, as the available real\-paper structural complexity indicator \(ReCITE\) violatesP​\(2​grafts\)<P​\(1​graft\)P\(2~\\text\{grafts\}\)<P\(1~\\text\{graft\}\), which we believe to be structurally justified\.

Foridentifiability, real\-world data on non\-identifiable cases is limited: empirical benchmark sources \(EconCausal,CauSciBench’s real\-study subset\) deal almost entirely with identifiable cases \(99\.6%99\.6\\%when pooled\), with the small remainder unresolved rather than explicitly non\-identifiable\(Leeet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib107); Acharyaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib81)\)\. Challenge suites \(CausalGame,CausalPitfalls\) inform the*kinds*of identification obstacles worth representing, but not their field frequency\(Chenet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib83); Duet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib1)\)\.double\_ncgraphs carry latent treatment–outcome confounding, and a generic instrumental variable is not accepted as identifying the population ATE; their frequency follows the motif priors above\. Second, we request a deliberatenon\-identifiable slice\(10%10\\%of generated scenes for the dataset presented\) on otherwise\-identifiable motifs, realized by rejection sampling: we inject latent confoundingU→\{X,Y\}U\\to\\\{X,Y\\\}and accept only graphs whose population\-ATE identifiability check returnsFalse\. Together the two routes make roughly30%30\\%of realistic\-exam scenes non\-identifiable, keeping abstention a first\-class evaluation axis\.

SCM mechanism profilesare derived from available empirical evidence: real biochemical\-equation and Boolean\-rule corpora \(BioModels,SABIO\-RK,BiodivineBooleanModels, and theKadelka DesignPrinciplescorpus\)\(Malik\-Sheriffet al\.,[2020](https://arxiv.org/html/2607.08093#bib.bib115); Wittiget al\.,[2018](https://arxiv.org/html/2607.08093#bib.bib116); Pastvaet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib117); Kadelkaet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib118)\); these are the only available corpora that expose per\-equation or per\-rule mechanism rows, so they alone contribute counts \(with a partial\-export weight applied where coverage is incomplete\)\. Source\-native classes are filtered through a small support floor and collapsed into a compact registry of three continuous profiles \(linear\_additive,interaction\_response,symbolic\_transform\) and two binary profiles \(single\_regulator\_logistic,multi\_input\_logic\_gate\), so the registry stays domain\-neutral despite a biology\-heavy source pool\. The resulting empirical distribution is heavily concentrated on multi\-parent interaction shapes and multi\-input gates, so the production mixture is the convex combinationα​p^emp\+\(1−α\)​pcov\\alpha\\,\\hat\{p\}\_\{\\mathrm\{emp\}\}\+\(1\-\\alpha\)\\,p\_\{\\mathrm\{cov\}\}withα=0\.3\\alpha=0\.3and a uniform synthetic coverage priorpcovp\_\{\\mathrm\{cov\}\}over additive, handcrafted, neural\-black\-box, mixture\-noise, and heteroscedastic regimes \(continuous\), plus logistic, sharp\-threshold, noisy\-OR, noisy\-AND, and signed\-logic regimes \(binary\); a sixth continuous regime \(coarsened\-continuous\) is implemented and configured but carries zero weight in the presented benchmark, as low\-cardinality coarsened nodes proved brittle under the measurement\-information admissibility check\. The empirical component is deliberately demoted due to both the narrow domain concentration, as well as limited mechanism diversity \. The two distributions are determined marginally by node type and are independent of motif class, since no public artifact ships row\-level joint labels of mechanism by motif\.

On the exam side, thequestion\-type priorP​\(Q∣S\)P\(Q\\mid S\)\(Exam Axis 1\) is built as a Bayes\-style combination of anempirical marginalp^emp​\(Q\)\\hat\{p\}\_\{\\mathrm\{emp\}\}\(Q\)over task families sourced from real\-world causal benchmarks that expose family labels:CausalReasoningBenchmark,CauSciBench,EvidenceInference,EconCausal, and a syntheticstructural liftderived fromCLadder\(Sawarniet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib93); Acharyaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib81); DeYounget al\.,[2020](https://arxiv.org/html/2607.08093#bib.bib112); Leeet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib107); Jinet al\.,[2023](https://arxiv.org/html/2607.08093#bib.bib24)\), which is the only available source that ships jointgraph×query type\\text\{graph\}\\times\\text\{query type\}rows\. The empirical marginal is mixed with a uniform deployable\-family floor so that no rung\-1 or rung\-3 family is collapsed to zero mass under R2\-heavy real\-world empirical distribution; the structural lift is applied with a small exponent so thatCLadder’s synthetic balance does not overpower the empirical ordering\. Motifs absent fromCLadder\(notably the grafted bucket\) fall back to the deployable base bucket\.

For theoutput\-variant priorP​\(OutputVariant∣QuestionType\)P\(\\text\{OutputVariant\}\\mid\\text\{QuestionType\}\)\(Exam Axis 2\), we keep an explicit manual crosswalk between source\-native answer contracts and our internal output\-variant set\. The weighted conditional is driven only by sources with multiple possible output variants:CausalReasoningBenchmarkandCauSciBench\(both its real\-study and its textbook\-derived QR subsets\)\(Sawarniet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib93); Acharyaet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib81)\)\. In practice this covers three question families \(causal\_sketch,identification,effect\_estimate\); for every other family no empirical row exists, and the exam builder draws the output variant uniformly over that family’s compatible variants\.

For theobservation\-model variant, the deployed three\-class set\{clean,proxy,proxy\_hard\}\\\{\\texttt\{clean\},\\texttt\{proxy\},\\texttt\{proxy\\\_hard\}\\\}is grounded against real\-data sources that expose per\-row accessibility fields\.ReCITE\(article\-levelexplicitnesson causal\-graph rows\) andEvidenceInference\(per\-prompt valid\-evidence location on biomedical RCT prompts\) provide direct rows for graph\-recovery and effect\-estimation question families;CausalReasoningBenchmark\(paired precise/vague prompt rows\) andTellMeWhy\(Answerable / Not Answerable real why\-questions\) act as additional companions\(Sakladet al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib110); DeYounget al\.,[2020](https://arxiv.org/html/2607.08093#bib.bib112); Sawarniet al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib93); Lalet al\.,[2021](https://arxiv.org/html/2607.08093#bib.bib113)\)\. A four\-level ordered source vocabulary \(clean<<partly\_explicit<<document\_context<<external\_implicit\) collapses these accessibility fields and bridges to the deployed three\-class set; the two hardest source levels both map onto the deployedproxy\_hardclass\. The exam\-side prefillP​\(observation variant∣QuestionType\)P\(\\text\{observation variant\}\\mid\\text\{QuestionType\}\)uses direct rows where available, explicit borrowed rows for nearby families, and the empirical marginalP^​\(observation variant\)\\hat\{P\}\(\\text\{observation variant\}\)as a fallback for question types without direct source labels\.

### A\.8Exam\-side difficulty knob

After the priors of App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)are built, a separatedifficulty knobd∈\[0,1\]d\\in\[0,1\]tilts three exam\-side distributions: the question\-type conditionalP​\(Q∣S\)P\(Q\\mid S\), the within\-family output\-variant distributionP​\(OutputVariant∣QuestionType\)P\(\\text\{OutputVariant\}\\mid\\text\{QuestionType\}\), and the released observation\-variant distributionP​\(obs∣Q\)P\(\\text\{obs\}\\mid Q\)\. We partition question families by Pearl rungr​\(Q\)∈\{1,2,3\}r\(Q\)\\in\\\{1,2,3\\\}, index output variants by an editorial within\-family difficulty rankrq​\(v\)r\_\{q\}\(v\)with midpointcqc\_\{q\}, and index observation variants byo​\(obs\)∈\{0,1,2\}o\(\\text\{obs\}\)\\in\\\{0,1,2\\\}for\{clean,proxy,proxy\_hard\}\\\{\\texttt\{clean\},\\texttt\{proxy\},\\texttt\{proxy\\\_hard\}\\\}\. Each tilt is multiplicative on the prefill before renormalization:

p^d​\(Q∣S\)\\displaystyle\\hat\{p\}\_\{d\}\(Q\\mid S\)∝p^​\(Q∣S\)​exp⁡\(β​\(2​d−1\)​\(r​\(Q\)−2\)\),\\displaystyle\\propto\\hat\{p\}\(Q\\mid S\)\\exp\\\!\\bigl\(\\beta\\,\(2d\{\-\}1\)\\,\(r\(Q\)\{\-\}2\)\\bigr\),\(6\)p^d​\(v∣q\)\\displaystyle\\hat\{p\}\_\{d\}\(v\\mid q\)∝p^​\(v∣q\)​exp⁡\(δ​\(2​d−1\)​\(rq​\(v\)−cq\)\),\\displaystyle\\propto\\hat\{p\}\(v\\mid q\)\\exp\\\!\\bigl\(\\delta\\,\(2d\{\-\}1\)\\,\(r\_\{q\}\(v\)\-c\_\{q\}\)\\bigr\),p^d​\(obs∣Q\)\\displaystyle\\hat\{p\}\_\{d\}\(\\text\{obs\}\\mid Q\)∝p^​\(obs∣Q\)​exp⁡\(γ​\(2​d−1\)​\(o​\(obs\)−1\)\),\\displaystyle\\propto\\hat\{p\}\(\\text\{obs\}\\mid Q\)\\exp\\\!\\bigl\(\\gamma\\,\(2d\{\-\}1\)\\,\(o\(\\text\{obs\}\)\{\-\}1\)\\bigr\),In Eq\.[6](https://arxiv.org/html/2607.08093#A1.E6),d=0\.5d=0\.5is a no\-op,d=0d=0pulls toward R1 / easier contracts /clean, andd=1d=1pulls toward R3 / harder contracts /proxy\_hard\. The strength constantsβ\\beta,δ\\delta, andγ\\gammaare kept separate so rung pressure, output\-variant pressure, and observation pressure can be calibrated independently \(all default to11\); single\-variant families \(e\.g\.counterfactual\_identification\) are unchanged on the output\-variant axis\. This is an editorial dial rather than a frequency claim: Pearl rung is only a coarse difficulty indicator and the within\-family rank is editorial, while the observation orderingclean<<proxy<<proxy\_hardis on firmer ground\. The realistic exam usesd=0\.5d=0\.5, so its composition is determined entirely by the empirical priors of App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)without any difficulty\-knob tilt; sweepingddaway from0\.50\.5is the intended way to produce easier or harder exam mixes from the same underlying priors\.

### A\.9Scoring rules by rung

The abstain convention from Sec\.[3\.7](https://arxiv.org/html/2607.08093#S3.SS7)applies whenever the target estimand is non\-identifiable or the model abstains \(routing details below\); otherwise, the task\-specific metric below is used\. GT denotes SCM\-derived ground truth\. For continuous outcomes we useRMSE\\mathrm\{RMSE\}; for binary outcomes ROC\-AUC alongside the Brier score1n​∑\(p^i−yi\)2\\tfrac\{1\}\{n\}\\sum\(\\hat\{p\}\_\{i\}\-y\_\{i\}\)^\{2\}and log\-loss; for set\- and graph\-recovery tasks we useF1=2​\|S^∩S\|/\(\|S^\|\+\|S\|\)F\_\{1\}=2\|\\hat\{S\}\\cap S\|/\(\|\\hat\{S\}\|\+\|S\|\)\(withF1=1F\_\{1\}=1when both sets are empty\); for interval forecasts the Gneiting–Raftery proper interval scoreISα​\(ℓ^,u^;τ\)=\(u^−ℓ^\)\+2α​\(ℓ^−τ\)​𝟏​\[τ<ℓ^\]\+2α​\(τ−u^\)​𝟏​\[τ\>u^\]\\mathrm\{IS\}\_\{\\alpha\}\(\\hat\{\\ell\},\\hat\{u\};\\tau\)=\(\\hat\{u\}\-\\hat\{\\ell\}\)\+\\tfrac\{2\}\{\\alpha\}\(\\hat\{\\ell\}\-\\tau\)\\mathbf\{1\}\[\\tau<\\hat\{\\ell\}\]\+\\tfrac\{2\}\{\\alpha\}\(\\tau\-\\hat\{u\}\)\\mathbf\{1\}\[\\tau\>\\hat\{u\}\]\(Gneiting and Raftery,[2007](https://arxiv.org/html/2607.08093#bib.bib105)\); and for the remaining content variants either exact match or absolute error\. The interval score is strictly proper, with expected minimum at the true central \(equal\-tailed\)\(1−α\)\(1\-\\alpha\)predictive interval: the agent picks its own estimator and the resulting\(ℓ^,u^\)\(\\hat\{\\ell\},\\hat\{u\}\)are scored against the scalar parameterτ\\tau\(or row\-wiseyjy\_\{j\}for prediction intervals\), so no reference interval is computed or stored\. For pooling, each interval task contributes the normalized interval scoreNRelISi=ISα\(i\)/\(1\+si\)\\mathrm\{NRelIS\}\_\{i\}=\\mathrm\{IS\}\_\{\\alpha\}^\{\(i\)\}/\(1\+s\_\{i\}\), with the same normalizersis\_\{i\}as in Eq\.[1](https://arxiv.org/html/2607.08093#S3.E1):si=sd​\(Ytest\)s\_\{i\}=\\mathrm\{sd\}\(Y\_\{\\mathrm\{test\}\}\)for prediction intervals \(row\-wise scores averaged over the test set,α=0\.1\\alpha=0\.1for the requested central90%90\\%interval\) andsi=\|τ\|s\_\{i\}=\|\\tau\|for effect intervals \(α=0\.05\\alpha=0\.05for the central95%95\\%interval\)\. Interval scores are aggregated by means at every level \(over test rows within a task, and across tasks via the capped mean discussed below\), since averaging preserves propriety\.

Table 8:Rung 1 scoring rules\.TaskOutcome variantMetricScore definitionpredictionpoint\_predictorRMSE / AUCRMSE on the private test set for continuousYY; ROC\-AUC for binaryY∈\{0,1\}Y\\in\\\{0,1\\\}\.predictionprediction\_intervalInt\. scoreProper interval score on the private test set; report coverage and mean width\.associationsign\_onlyExact11iff answersignmatches the GT sign in\{\+,−,unknown\}\\\{\+,\-,\\texttt\{unknown\}\\\}\.associationeffect\_size\_pointAbs\. err\.\|r^−r\|\|\\widehat\{r\}\-r\|whererris the GT Pearson correlation between treatment and outcome on the ground\-truth data\.associationsign\_before\_afterExact11iff bothsign\_beforeandsign\_aftermatch GT\.associationdelta\_pointAbs\. err\.Absolute error onΔ^\\widehat\{\\Delta\}whereΔ\\Deltais the GT change in association after conditioning\.associationdelta\_sign\_onlyExact11iff answersignmatches the sign of the GT conditional\-association change\.associationargmax\_changeExact11iff the selected conditioning variable matches the GT variable with the largest absolute association change\.collider\_phenomenoninduced\_association\_booleanExact11iff answerassociation\_presentmatches GT\.collider\_phenomenoninduced\_association\_sign\_onlyExact11iff answersignmatches the GT induced\-association sign after conditioning on the collider\.collider\_phenomenoninduced\_association\_strength\_pointAbs\. err\.Absolute error on the GT conditional association induced by conditioning on the collider\.Table 9:Rung 2 scoring rules\.TaskOutcome variantMetricScore definitioncausal\_sketchedges\_onlyEdgeF1F\_\{1\}Directed edge\-setF1F\_\{1\}against GT edges in story\-name space\.causal\_sketchskeleton\_edgesSkel\.F1F\_\{1\}Undirected edge\-setF1F\_\{1\}against the GT skeleton, ignoring direction\.identificationone\_valid\_adjustment\_setExact11iff answeradjustequals any GT valid backdoor set; if the ATE is identifiable but no valid backdoor set exists \(e\.g\., front\-door/IV\),11iff the answer is theno\_backdoorsentinel\.identificationmethod\_labelExact11iff answermethodmatches GT, withnonedenoting not identifiable\.identificationidentifiable\_booleanExact11iff answeridentifiablematches the GT population\-ATE identifiability\.identificationminimal\_adjustment\_set\_sizeExact11iff answerkequals the size of a GT minimal valid backdoor set; if the ATE is identifiable but no valid backdoor set exists,11iff the answer is theno\_backdoorsentinel\.identificationn\_valid\_adjustment\_setsExact11iff answernequals the number of GT valid backdoor sets\.identificationall\_minimal\_adjustment\_setsSetF1F\_\{1\}Set\-levelF1F\_\{1\}over predicted versus GT minimal valid backdoor sets\.effect\_estimateate\_pointAbs\. err\. / ExactIf identifiable,\|τ^−τ\|\|\\widehat\{\\tau\}\-\\tau\|whereτ\\tauis the SCM Monte Carlo GT ATE; also report relative error\. If non\-identifiable,11iff the answer abstains withate=null\.effect\_estimateate\_uq\_95Int\. score / ExactIf identifiable, the proper interval score forτ\\tau; report coverage and width\. If non\-identifiable,11iff the answer abstains withate=null\.effect\_estimateate\_sign\_onlyExact11iff answersignmatches the GT ATE sign in\{\+,−,0,unknown\}\\\{\+,\-,0,\\texttt\{unknown\}\\\}\.effect\_estimateate\_vs\_assoc\_sign\_matchExact11iff answermatchesagrees with whether the GT ATE sign matches the GT observational\-association sign\.bias\_diagnosticcollider\_bias\_booleanExact11iff answerbias\_presentmatches GT\.bias\_diagnosticforbidden\_controls\_listSetF1F\_\{1\}Set\-levelF1F\_\{1\}over predicted versus GT forbidden observed controls\.Table 10:Rung 3 scoring rules\.TaskOutcome variantMetricScore definitioncounterfactual\_identificationidentifiable\_booleanExact11iff answeridentifiablematches the GT identifiability of the target estimand \(ETT, NDE, or NIE\)\.counterfactual\_effecteffect\_pointAbs\. err\. / ExactIf identifiable,\|ETT^−ETT\|\|\\widehat\{\\mathrm\{ETT\}\}\-\\mathrm\{ETT\}\|using SCM GT; also report relative error\. If non\-identifiable,11iff the answer abstains withvalue=null\.counterfactual\_effecteffect\_uq\_95Int\. score / ExactIf identifiable, the proper interval score for the ETT; report coverage and width\. If non\-identifiable,11iff the answer abstains withvalue=null\.counterfactual\_effectsign\_onlyExactIf identifiable,11iff answersignmatches the GT ETT sign in\{\+,−,0,unknown\}\\\{\+,\-,0,\\texttt\{unknown\}\\\}\. If non\-identifiable,11iffsign=unknown\.mediation\_effecteffect\_pointAbs\. err\. / ExactFor NDE or NIE, if identifiable, absolute error on the requested SCM GT effect; also report relative error\. If non\-identifiable,11iff the answer abstains withvalue=null\.mediation\_effecteffect\_uq\_95Int\. score / ExactFor NDE or NIE, if identifiable, the proper interval score for the requested effect; report coverage and width\. If non\-identifiable,11iff the answer abstains withvalue=null\.mediation\_effectsign\_onlyExactFor NDE or NIE, if identifiable,11iff answersignmatches the GT sign in\{\+,−,0,unknown\}\\\{\+,\-,0,\\texttt\{unknown\}\\\}\. If non\-identifiable,11iffsign=unknown\.mediation\_effectdirect\_vs\_indirect\_dominanceExactIf identifiable,11iff answerdominantmatches whether\|NDE\|\|\\mathrm\{NDE\}\|exceeds\|NIE\|\|\\mathrm\{NIE\}\|, whether\|NIE\|\|\\mathrm\{NIE\}\|exceeds\|NDE\|\|\\mathrm\{NDE\}\|, or whether they are tied within tolerance\. If non\-identifiable,11iffdominant=null\.##### Abstention pool routing\.

The grader routes a task into the abstention pool whenever*either*the GT estimand is non\-identifiable*or*the model returned a null\-equivalent answer, replacing the per\-variant content metric with a single uniform abstention binaryAbstMatchi\\mathrm\{AbstMatch\}\_\{i\}\(Sec\.[3\.7](https://arxiv.org/html/2607.08093#S3.SS7)\)\. This covers every variant whose estimand can be non\-identifiable: the rung\-2identificationfamily \(bothidentifiable\_booleanandmethod\_label, and the four backdoor\-adjustment\-set variants\), the rung\-2effect\_estimatefamily,bias\_diagnostic\.forbidden\_controls\_list, and the rung\-3counterfactual\_identification,counterfactual\_effect, andmediation\_effectvariants\. For the identification\-type families this routing changes no score — the identifiability judgment is the content — it only assigns the task to the abstention pool so the same binary is counted once, uniformly across families; for the estimate\-type families it replaces the content metric, which would be meaningless on a non\-identifiable estimand\. Each task therefore lives in exactly one pool: a wrong abstention call collects a0on the abstention binary but does not also collect a substituted content score \(preventing partial\-credit hallucinations on non\-identifiable estimands\)\. The R3 abstention check uses the per\-estimand R3 identifiability \(counterfactual\_identification\.​⟨ETT,NDE,NIE⟩\\texttt\{counterfactual\\\_identification\.\}\\\!\\langle\\text\{ETT,NDE,NIE\}\\rangle\); for the dominance variant we require both NDE and NIE to be identifiable\. Null\-equivalent labels \(none,null,unknown, missing field, empty string\) are accepted interchangeably\.

##### Every task variant feeds an aggregate metric\.

The benchmark currently samples3333active\(TaskType, OutputVariant\)pairs across the rung\-1/2/3 tables above, and every active variant contributes to at least one of the three leaderboard metrics \(Pass Rate, Med\. NRel\. Err, Med\.F1F\_\{1\}\-Loss\)\. Pass Rate pools the1818exact\_match\-graded content variants together with the abstention binary\. Med\. NRel\. Err pools the1111continuous\-output variants in three groups: the two prediction variants \(prediction\.point\_predictor, whose binary branch contributesBrier/\(1\+sd​\(Ytest\)\)\\sqrt\{\\mathrm\{Brier\}\}/\(1\+\\mathrm\{sd\}\(Y\_\{\\mathrm\{test\}\}\)\), andprediction\.prediction\_intervalvia the normalized interval score\); the six effect variants \(effect\_estimate\.\{ate\_point, ate\_uq\_95\},counterfactual\_effect\.\{effect\_point, effect\_uq\_95\}, andmediation\_effect\.\{effect\_point, effect\_uq\_95\}, with the three\_uq\_95intervals scored by NRelIS\); and the three association\-strength points \(association\.\{effect\_size\_point, delta\_point\}andcollider\_phenomenon\.induced\_association\_strength\_point\), whose absolute errors are normalized by1\+\|τ\|1\+\|\\tau\|\. Med\.F1F\_\{1\}\-Loss pools the fourF1F\_\{1\}\-graded variants:causal\_sketch\.\{edges\_only, skeleton\_edges\},identification\.all\_minimal\_adjustment\_sets, and the identifiable branch ofbias\_diagnostic\.forbidden\_controls\_list\. The three pools cover all3333pairs \(18\+11\+418\+11\+4\)\.

##### Aggregator choice forSNRS\_\{\\mathrm\{NR\}\}\.

The aggregatorSNRS\_\{\\mathrm\{NR\}\}and its point/interval poolsTNRpt,TNRintT\_\{\\mathrm\{NR\}\}^\{\\mathrm\{pt\}\},T\_\{\\mathrm\{NR\}\}^\{\\mathrm\{int\}\}are defined in Eq\.[3](https://arxiv.org/html/2607.08093#S3.E3)\. Point\-graded tasks emit naturally bounded normalized errors, and a median overTNRptT\_\{\\mathrm\{NR\}\}^\{\\mathrm\{pt\}\}tracks typical\-scene performance without being tail\-driven\. Interval\-graded tasks emit a strictly proper score whose miss\-penalty multiplier2/α2/\\alphais unbounded by design: asingle overconfident misscan reachNRelIS\\mathrm\{NRelIS\}values an order of magnitude above the typical\-scene level, which is precisely the signal the score is meant to expose\. Aggregating those by median would discard exactly that information; aggregating by raw mean would let one catastrophic miss dominate the composite\. Thecapped meanmean​min⁡\(NRelISi,c\)\\mathrm\{mean\}\\,\\min\(\\mathrm\{NRelIS\}\_\{i\},c\)withc=10c=10preserves the proper\-score\-induced overconfidence penalty for typical misses \(a1×\|τ\|1\\times\|\\tau\|miss incurs≈4\\approx 4, well below the cap\) while bounding any single task’s contribution tocc, so cross\-exam stability does not depend on the single most extreme task\.

### A\.10Realistic\-exam breakdowns

Tables in this section report the exam composition together with input\-mode \(symbolic vs\. data\-backed\), pass\-rate, per\-rung, per\-task\-family, per\-motif, and abstention\-only views of the run summarized in Sec\.[5](https://arxiv.org/html/2607.08093#S5)\(the observation\-variant view is located in the main text, Tabs\.[5](https://arxiv.org/html/2607.08093#S5.T5)and[6](https://arxiv.org/html/2607.08093#S5.T6)\)\.*n*is the number of tasks in the slice;*Cont\. PR*restricts toexact\_match\-graded discrete tasks;*Abst\. PR*restricts to the abstention pool\. Best per column in bold\. All slices are marginal views of the same100100\-task exam: the axes are not varied independently, so a difference along one axis can also reflect the task families, output variants, and observation variants that happen to fall into that slice\.

Table 11:Composition of the realistic exam \(100 scenes\) by structural motif and task family\.grafted\_kkdenotes scenes withkkauxiliary motifs anchor\-grafted onto a host main graph\.MotifPredAssocColliderSketchIdentEffectBiasCF\-IDCFEMediationTotalarrowhead00010001002chain10011101005collider00010011003confounding610323343025diamond00000000011diamondcut00002000002double\_nc00000000101fork12001000004frontdoor10000100013iv230843331027mediation01020100004triangle10000100002grafted\_1232133002016grafted\_211001002005Total1511217141371272100

Table 12:Abstention by task family on the realistic exam: abstention pass rate \(per\-family abstention pool size in parentheses\)\. “—”: empty pool\. Pool sizes vary across models because routing depends on the model’s own answer\. Best per row in bold\.FamilyClaude Opus 4\.8Gemini 3\.1 ProGPT\-5\.5Qwen 3\.6 35BKimi K2\.6Gemma 4 26Bidentification100\.0%\(5\)60\.0% \(5\)80\.0% \(5\)40\.0% \(5\)60\.0% \(5\)0\.0% \(5\)bias\_diagnostic0\.0% \(3\)66\.7% \(3\)100\.0%\(3\)33\.3% \(3\)0\.0% \(3\)0\.0% \(3\)effect\_estimate0\.0% \(3\)0\.0% \(3\)100\.0%\(3\)25\.0% \(4\)0\.0% \(3\)33\.3% \(3\)counterfactual\_id100\.0%\(3\)100\.0%\(3\)66\.7% \(3\)66\.7% \(3\)66\.7% \(3\)66\.7% \(3\)counterfactual\_effect100\.0%\(2\)50\.0% \(2\)0\.0% \(2\)50\.0% \(2\)66\.7% \(3\)0\.0% \(2\)mediation\_effect————0\.0% \(1\)—

##### Abstention by family \(Tab\.[12](https://arxiv.org/html/2607.08093#A1.T12)\)\.

The frontier models perfectly solve abstention on several families while the open models do on none; the average rates likewise reflect this split\. GPT\-5\.5 is perfect on the R2 IV / forbidden\-control families \(effect\_estimate,bias\_diagnostic\) and Claude Opus 4\.8 onidentificationand both counterfactual families, whereas Gemma 4 26B fails to abstain correctly on every single question foridentification,bias\_diagnostic, orcounterfactual\_effect\.

Table 13:Per\-rung headline aggregates over the realistic exam\. R1 has no abstention pool\. Best per column in bold\.Rung 1 \(n=28n=28\)Rung 2 \(n=51n=51\)Rung 3 \(n=21n=21\)ModelPRCont\. PRNRel\. ErrPRCont\. PRAbst\. PRNRel\. ErrPRCont\. PRAbst\. PRNRel\. ErrClaude Opus 4\.8100\.0%100\.0%0\.31957\.1%100\.0%45\.5%0\.175100\.0%100\.0%100\.0%0\.028Gemini 3\.1 Pro100\.0%100\.0%0\.30157\.1%100\.0%45\.5%0\.24888\.2%91\.7%80\.0%0\.027Qwen 3\.6 35B75\.0%75\.0%0\.27646\.7%100\.0%33\.3%1\.06173\.7%78\.6%60\.0%0\.099Kimi K2\.6100\.0%100\.0%0\.27742\.9%100\.0%27\.3%0\.27577\.8%90\.9%57\.1%0\.118GPT\-5\.5100\.0%100\.0%0\.28792\.9%100\.0%90\.9%0\.17170\.6%83\.3%40\.0%0\.050Gemma 4 26B100\.0%100\.0%0\.36128\.6%100\.0%9\.1%0\.59670\.6%83\.3%40\.0%0\.049

![Refer to caption](https://arxiv.org/html/2607.08093v1/x5.png)Figure 8:Pass Rate per rung and model on the realistic exam; pool sizesnnin the panel titles\. Models are ordered by CausalDSScore \(best at top\)\. Blue — frontier \(closed\) models; green — open\-weight models\.
##### Per\-rung aggregates \(Tab\.[13](https://arxiv.org/html/2607.08093#A1.T13), Fig\.[8](https://arxiv.org/html/2607.08093#A1.F8)\)\.

Rung\-2 content is solved by every model \(Cont\. PR100%100\\%across the field\), so the entire R2 Pass\-Rate spread \(28\.628\.6–92\.9%92\.9\\%\) is carried by the abstention pool: the identifiability call, not the execution of an identified analysis, is what separates the models\. R1 is saturated for five of the six models \(Qwen 3\.6 35B at75\.0%75\.0\\%\), leaving the prediction\-error band \(0\.2760\.276–0\.3610\.361\) as the only R1 signal\. At R3 the ordering inverts for GPT\-5\.5 — best R2 abstention \(90\.9%90\.9\\%\) but joint\-worst R3 abstention \(40\.0%40\.0\\%\) — while Claude Opus 4\.8 is the only model perfect on the binary R3 columns; the surviving R3 point estimates are quite accurate for every model \(NRel\. Err0\.0270\.027–0\.1180\.118\)555Note that this can be due to other factors, such as the task family, outcome variant, and observation\-layer difficulty drawn for the R3 pool: the per\-rung slices differ in composition, not only in rung\.\.

Table 14:Per\-axisComposite Rankfor the realistic exam\. Within each axis, models are reranked over the tasks in that slice and the per\-column average rank is normalized to\[0,1\]\[0,1\]\(with0being best\), using the same three pools as CausalDSScore \(Pass Rate,SNRS\_\{\\mathrm\{NR\}\}, Med\.F1F\_\{1\}\-Loss\)\.*Overall*repeats the exam\-wide Composite Rank of Tab\.[3](https://arxiv.org/html/2607.08093#S5.T3); rows are sorted by it \(tie broken by CausalDSScore\)\. Best \(lowest\) per column in bold\.RungObservation variantModelR1R2R3cleanproxyproxy\_hardOverallClaude Opus 4\.80\.5000\.2670\.2330\.4000\.2000\.3330\.200Gemini 3\.1 Pro0\.3000\.4000\.2330\.3000\.6330\.4000\.367GPT\-5\.50\.5670\.5000\.6670\.3330\.5000\.5000\.533Qwen 3\.6 35B0\.5670\.5670\.6330\.7330\.5330\.7000\.567Kimi K2\.60\.6330\.5000\.6330\.4670\.5670\.4670\.567Gemma 4 26B0\.4330\.7670\.6000\.7670\.5670\.6000\.767

##### Per\-axis Composite Rank \(Tab\.[14](https://arxiv.org/html/2607.08093#A1.T14)\)\.

The frontier leaders split the axes by difficulty: Claude Opus 4\.8 is best\-ranked on the harder slices \(R2, R3,proxy,proxy\_hard\), Gemini 3\.1 Pro on the easier ones \(R1,clean, tying on R3\), and no open\-weight model takes any column\. With sketchF1F\_\{1\}saturated for every model, the remaining rank differentiator is the interval\-sensitiveSNRS\_\{\\mathrm\{NR\}\}pool, where GPT 5\.5’s poorly calibrated intervals cost it \(cf\. Tab\.[6](https://arxiv.org/html/2607.08093#S5.T6)\)\.

Table 15:Per\-model performance by task input mode on the realistic exam \(*symbolic*vs\.*data\-backed*; see text\)\. Pass Rate pools contentexact\_matchand abstention binaries; symbolic tasks carry no continuous estimate \(NRel\. Err undefined\) and data\-backed tasks carry no structure\-recoveryF1F\_\{1\}\(Med\.F1F\_\{1\}\-Loss undefined\)\.Task counts in parentheses; best per column in bold\. Invalid continuous answers are excluded\.Symbolic \(n=49n=49\)Data\-backed \(n=51n=51\)ModelPass RateMed\.F1F\_\{1\}\-LossPass RateMed\. NRel\. ErrClaude Opus 4\.886\.4% \(22\)0\.000 \(27\)75\.0%\(12\)0\.179\(38\)Gemini 3\.1 Pro81\.8% \(22\)0\.000 \(27\)66\.7% \(12\)0\.231 \(38\)Qwen 3\.6 35B72\.7% \(22\)0\.000 \(27\)50\.0% \(16\)0\.276 \(31\)Kimi K2\.672\.7% \(22\)0\.000 \(27\)53\.8% \(13\)0\.230 \(38\)GPT\-5\.590\.9%\(22\)0\.000 \(27\)66\.7% \(12\)0\.224 \(37\)Gemma 4 26B54\.5% \(22\)0\.000 \(27\)58\.3% \(12\)0\.313 \(39\)

##### Symbolic vs\. data\-backed \(Tab\.[15](https://arxiv.org/html/2607.08093#A1.T15)\)\.

The tasks can also be divided into those that require the use of the provided data \(*data\-backed*\) and those that can be solved symbolically:causal\_sketch,identification,counterfactual\_identification, and the forbidden\-controlsbias\_diagnosticvariant are solvable from the story\-implied graph alone\. Symbolic reasoning is led by GPT\-5\.5 \(90\.9%90\.9\\%Pass Rate\), Claude Opus 4\.8 \(86\.4%86\.4\\%\), and Gemini 3\.1 Pro \(81\.8%81\.8\\%\), with the open models trailing \(Qwen 3\.6 35B and Kimi K2\.6 tie at72\.7%72\.7\\%, Gemma 4 26B last at54\.5%54\.5\\%\)\. On data\-backed tasks Claude Opus 4\.8 leads both Pass Rate \(75\.0%75\.0\\%\) and continuous\-estimate error \(lowest at0\.1790\.179NRel\. Err\), with Gemini 3\.1 Pro and GPT\-5\.5 tied at66\.7%66\.7\\%and the open models between50\.0%50\.0\\%and58\.3%58\.3\\%\. The cross\-mode anti\-correlation seen among the open models — Gemma 4 26B is symbolic\-worst yet edges Qwen and Kimi on data\-backed Pass Rate — does not extend to the frontier models, which are strong on both, Claude Opus 4\.8 most of all\. Structure recovery is saturated \(Med\.F1F\_\{1\}\-Loss=0=0for all models\), so symbolic differentiation comes entirely fromidentification/counterfactual\_identificationcorrectness\.

Table 16:Per\-task\-family aggregates on the realistic exam\. Rows group models whose displayed pass rates agree, continuous columns then report the group’s min–max range\. Within each family, rows are sorted by Pass Rate; best per column within each family in bold\.Family \(nn\)ModelPRCont\. PRAbst\. PRMed\. NRel\. ErrMed\.F1F\_\{1\}\-Lossprediction\(15\)all but GPT\-5\.5———0\.530–0\.621—GPT\-5\.5———0\.954—association\(11\)all six100\.0%100\.0%—0\.001—collider\_phenomenon\(2\)all but Qwen 3\.6 35B100\.0%100\.0%—0\.009–0\.164—Qwen 3\.6 35B50\.0%50\.0%———causal\_sketch\(17\)all six————0\.000identification\(14\)Claude Opus 4\.8100\.0%100\.0% \(2/2\)100\.0%\(5/5\)—0\.000GPT\-5\.585\.7%100\.0% \(2/2\)80\.0% \(4/5\)—0\.000Gemini 3\.1 Pro / Kimi K2\.671\.4%100\.0% \(2/2\)60\.0% \(3/5\)—0\.000Qwen 3\.6 35B57\.1%100\.0% \(2/2\)40\.0% \(2/5\)—0\.000Gemma 4 26B28\.6%100\.0% \(2/2\)0\.0% \(0/5\)—0\.000effect\_estimate\(13\)GPT\-5\.5100\.0%—100\.0%\(3/3\)0\.171—Gemma 4 26B33\.3%—33\.3% \(1/3\)0\.596—Qwen 3\.6 35B25\.0%—25\.0% \(1/4\)1\.061—Claude Opus 4\.8 / Gemini 3\.1 Pro /Kimi K2\.60\.0%—0\.0% \(0/3\)0\.175–0\.275—bias\_diagnostic\(7\)GPT\-5\.5100\.0%100\.0% \(1/1\)100\.0%\(3/3\)—0\.000Gemini 3\.1 Pro75\.0%100\.0% \(1/1\)66\.7% \(2/3\)—0\.000Qwen 3\.6 35B50\.0%100\.0% \(1/1\)33\.3% \(1/3\)—0\.000Claude Opus 4\.8 / Kimi K2\.6 /Gemma 4 26B25\.0%100\.0% \(1/1\)0\.0% \(0/3\)—0\.000counterfactual\_identification\(12\)Claude Opus 4\.8100\.0%100\.0%\(9/9\)100\.0%\(3/3\)——Gemini 3\.1 Pro91\.7%88\.9% \(8/9\)100\.0%\(3/3\)——GPT\-5\.5 / Kimi K2\.6 /Qwen 3\.6 35B91\.7%100\.0%\(9/9\)66\.7% \(2/3\)——Gemma 4 26B83\.3%88\.9% \(8/9\)66\.7% \(2/3\)——counterfactual\_effect\(7\)Claude Opus 4\.8100\.0%100\.0%\(2/2\)100\.0%\(2/2\)0\.034—Gemini 3\.1 Pro75\.0%100\.0%\(2/2\)50\.0% \(1/2\)0\.020—Kimi K2\.660\.0%50\.0% \(1/2\)66\.7% \(2/3\)0\.147—Qwen 3\.6 35B40\.0%33\.3% \(1/3\)50\.0% \(1/2\)0\.099—GPT\-5\.5 / Gemma 4 26B25\.0%50\.0% \(1/2\)0\.0% \(0/2\)0\.080–0\.083—mediation\_effect\(2\)Claude Opus 4\.8 / Gemini 3\.1 Pro /Gemma 4 26B100\.0%100\.0%\(1/1\)—0\.003–0\.034—Qwen 3\.6 35B50\.0%50\.0% \(1/2\)———GPT\-5\.50\.0%0\.0% \(0/1\)—0\.019—Kimi K2\.60\.0%—0\.0% \(0/1\)0\.012—

##### Per\-family commentary on Tab\.[16](https://arxiv.org/html/2607.08093#A1.T16)\.

Three families do not discriminate:associationandcausal\_sketchare solved by all six models, andpredictionmostly sits close to the noise\-bounded floor \(five models within0\.5300\.530–0\.6210\.621NRel\. Err, GPT\-5\.5 the outlier at0\.9540\.954\)\. The R2 families separate the models almost purely through the abstention call — their identifiable discrete queries are answered correctly by everyone — and no model makes that call reliably across families: Claude Opus 4\.8 declines every non\-identifiableidentificationand counterfactual query yet catches none of theeffect\_estimateor forbidden\-control \(bias\_diagnostic\) abstentions, while GPT\-5\.5 is close to the mirror image — perfect on those two families, but the only model besides Gemma 4 26B to catch nocounterfactual\_effectabstention\. Estimation accuracy on the identifiableeffect\_estimatequeries — a wrong commit on a non\-identifiable estimand is scored only as a missed abstention and never enters the error pool — splits along the same line: the frontier models stay accurate \(NRel\. Err0\.1710\.171–0\.2480\.248\), whereas Qwen 3\.6 35B and Gemma 4 26B blow up \(1\.0611\.061and0\.5960\.596; Kimi K2\.6 is the open\-weight exception at0\.2750\.275\)\. On the counterfactual families Claude Opus 4\.8 is alone at ceiling \(12/1212/12and4/44/4\);mediation\_effect\(n=2n=2\) is too small to read\.

Table 17:Per\-motif aggregates on the realistic exam, with grafted scenes split by graft count\. Motifs with≤2\\leq 2tasks are not broken out\. Best per column within each motif in bold\.Motif \(nn\)ModelPR \(ndiscn\_\{\\text\{disc\}\}\)Cont\. PRAbst\. PRMed\. NRel\. Erriv\(27\)GPT\-5\.578\.6%\(14\)—78\.6%0\.052Claude Opus 4\.857\.1% \(14\)—57\.1%0\.052Gemini 3\.1 Pro50\.0% \(14\)—50\.0%0\.026Qwen 3\.6 35B42\.9% \(14\)—42\.9%0\.052Kimi K2\.635\.7% \(14\)—35\.7%0\.052Gemma 4 26B21\.4% \(14\)—21\.4%0\.052confounding\(25\)Claude Opus 4\.8100\.0%\(6\)100\.0%—0\.260Gemini 3\.1 Pro100\.0%\(6\)100\.0%—0\.243GPT\-5\.5100\.0%\(6\)100\.0%—0\.840Qwen 3\.6 35B85\.7% \(7\)85\.7%—0\.276Kimi K2\.685\.7% \(7\)100\.0%0\.0%0\.302Gemma 4 26B83\.3% \(6\)83\.3%—0\.374grafted\_1\(16\)Claude Opus 4\.8100\.0%\(5\)100\.0%100\.0%0\.112Gemini 3\.1 Pro100\.0%\(5\)100\.0%100\.0%0\.197GPT\-5\.580\.0% \(5\)75\.0%100\.0%0\.164Kimi K2\.680\.0% \(5\)75\.0%100\.0%0\.360Qwen 3\.6 35B66\.7% \(6\)60\.0%100\.0%0\.634Gemma 4 26B60\.0% \(5\)75\.0%0\.0%0\.250grafted\_2\(5\)all but Gemini 3\.1 Pro100\.0%\(2\)100\.0%—0\.246–0\.357Gemini 3\.1 Pro50\.0% \(2\)50\.0%—0\.250chain\(5\)all six100\.0%\(1\)100\.0%—0\.301–0\.530fork\(4\)all six100\.0%\(1\)100\.0%—0\.002collider\(3\)all six100\.0%\(2\)100\.0%——frontdoor\(3\)Claude Opus 4\.8100\.0%\(1\)100\.0%—0\.421Gemini 3\.1 Pro100\.0%\(1\)100\.0%—0\.224Gemma 4 26B100\.0%\(1\)100\.0%—0\.237Qwen 3\.6 35B50\.0% \(2\)100\.0%0\.0%0\.417GPT\-5\.50\.0% \(1\)0\.0%—0\.224Kimi K2\.60\.0% \(1\)—0\.0%0\.224mediation\(4\)all six———0\.008
##### Per\-motif commentary on Tab\.[17](https://arxiv.org/html/2607.08093#A1.T17)\.

The two large motifs probe complementary skills\. Oniv, every discrete query lands in the abstention pool — by design, a generic instrument does not identify the population ATE \(App\.[A\.7](https://arxiv.org/html/2607.08093#A1.SS7)\) — so the motif ranks models purely on the abstention call, with a wide spread \(GPT\-5\.578\.6%78\.6\\%down to Gemma 4 26B21\.4%21\.4\\%\), while the median error on the continuous tasks is small and nearly uniform \(0\.0260\.026–0\.0520\.052\)\.confoundingis the mirror image: its queries are identifiable \(the lone abstention entry is Kimi K2\.6 wrongly abstaining on one\), the frontier models pass every discrete query, and the error column separates them instead —0\.2430\.243–0\.3740\.374for most models against GPT\-5\.5’s0\.8400\.840\. Grafting does not degrade the frontier: ongrafted\_1, Claude Opus 4\.8 and Gemini 3\.1 Pro stay perfect with the lowest errors, whereas Qwen 3\.6 35B drops to66\.7%66\.7\\%with0\.6340\.634NRel\. Err and Gemma 4 26B misses its abstention call\. A second graft \(grafted\_2\) adds no further degradation: all models pass every query except a single unanswered counterfactual\-identification query from Gemini 3\.1 Pro\. The simple identifiable structures act as a sanity floor —chain,fork,collider, andmediationare passed by all six models, with only thechainestimation spread \(0\.3010\.301–0\.5300\.530\) separating them\. Onfrontdoor, the motif\-defining front\-door ATE query is estimated accurately by four models, while Qwen 3\.6 35B wrongly abstains on it and Claude Opus 4\.8 leaves it unanswered; the row’s discrete pass rates instead reflect the accompanying mediation sign query, which GPT\-5\.5 leaves unanswered and Kimi K2\.6 wrongly abstains on\. Only the five motifs with≤2\\leq 2tasks that are not broken out \(arrowhead,diamondcut,triangle,diamond,double\_nc\) carry too few tasks to support conclusions in either direction\.

Table 18:Token usage and efficiency on the realistic exam\. Tokens are prompt\+\+completion;*Tokens/\(1−Score\)\(1\\\!\-\\\!\\mathrm\{Score\}\)*is a measure of efficiency \(lower is better\) — the table is sorted by this metric\.*Calls/Task*is the mean number of bash invocations per task\.*CausalDSScore*is repeated from Tab\.[3](https://arxiv.org/html/2607.08093#S5.T3)\. Rows sorted by Tokens/\(1−Score\)\(1\\\!\-\\\!\\mathrm\{Score\}\)\. Best per column in bold\.ModelTotal TokensTokens/TaskCausalDSScoreTokens/\(1−\-Score\)Calls/TaskStrategyClaude Opus 4\.81\.77M17\.7k0\.27802\.45M3\.4near one\-shotGPT\-5\.51\.29M12\.9k0\.56102\.95M2\.1near one\-shotGemma 4 26B3\.24M32\.4k0\.64429\.10M6\.1iterativeGemini 3\.1 Pro14\.56M145\.6k0\.370323\.13M11\.2iterativeQwen 3\.6 35B14\.07M140\.7k0\.447425\.45M17\.6iterativeKimi K2\.626\.64M266\.4k0\.475450\.78M11\.9iterative

##### Efficiency\.

Tool\-use strategy splits the field \(Fig\.[4](https://arxiv.org/html/2607.08093#S5.F4)\)\. GPT\-5\.5 \(2\.12\.1calls/task\) and Claude Opus 4\.8 \(3\.43\.4\) are near one\-shot — they read the data, write the entire analysis as one script and submit — while the other four iterate, exploring the data, fitting several candidate models, and refining: Gemma 4 26B \(6\.16\.1\), Gemini 3\.1 Pro \(11\.211\.2\), Kimi K2\.6 \(11\.911\.9\), and Qwen 3\.6 35B \(17\.617\.6\)\. Calls and token usage do not track each other: Kimi K2\.6 spends the most tokens \(266\.4266\.4k/task,26\.626\.6M total\) despite making fewer calls than Qwen 3\.6 35B \(140\.7140\.7k/task\), reflecting a longer reasoning process, while GPT\-5\.5 is the leanest overall \(12\.912\.9k/task\) despite the high reasoning setting, with Claude Opus 4\.8 close behind \(17\.717\.7k/task\)\. Gemma, the one non\-reasoning model, follows behind at32\.432\.4k/task\. Normalized by quality, Tokens/\(1−Score\)\(1\\\!\-\\\!\\mathrm\{Score\}\)ranks Claude Opus 4\.8 most token\-efficient \(2\.452\.45M\) and Kimi K2\.6 least \(50\.850\.8M\) — a21×21\\timesspread\. Token usage is the efficiency axis comparable across all six models: five are served locally or through unpriced routes, and only Gemini 3\.1 Pro is billed at a published per\-token price, its full run costing $20\.0220\.02\($0\.200\.20/task\)\. These interaction styles echo trajectory\-level findings from recent agent benchmarks\. On RoadmapBench,

Claude Opus 4\.7 attains the highest resolved rate with the fewest tool calls and the lowest exploration ratio\(Xuet al\.,[2026a](https://arxiv.org/html/2607.08093#bib.bib152)\)— the same coupling of a targeted trajectory with top quality that we observe for Claude Opus 4\.8\. The pattern is not family\-general: the same benchmark reports GPT\-5\.4 making the*most*tool calls without converging \(“analysis paralysis”\), while AutoLab reports the opposite failure for the same model — submitting after minimal exploration despite substantial remaining budget\(Xuet al\.,[2026b](https://arxiv.org/html/2607.08093#bib.bib154)\); on MCP\-Bench, GPT\-5 is among the heavier tool users\(Wanget al\.,[2025](https://arxiv.org/html/2607.08093#bib.bib144)\), and DeepPlanning finds that more tool use generally improves long\-horizon planning\(Zhanget al\.,[2026](https://arxiv.org/html/2607.08093#bib.bib153)\)\. Tool\-use strategy is thus strongly model\- and benchmark\-specific\. We also note that call counts are not commensurate across benchmarks: a single bash call in our harness can execute a complete analysis script, so the comparable quantity is the interaction style \(single\-pass script submission versus multi\-round orchestration\) rather than the raw counts\.

### A\.11Per\-axis pass@kkbreakdowns

##### Setup\.

The headline pass@kk/ passˆkktable is in the main text \(Tab\.[7](https://arxiv.org/html/2607.08093#S5.T7)\); here we give the method, the continuous analogue \(Tab\.[19](https://arxiv.org/html/2607.08093#A1.T19)\), and the per\-axis breakdowns\. We draw up tok=3k\{=\}3independent restarts per task, each a fresh single\-shot trajectory\. Unlike the answer\-dependent main routing, each task is assigned to a scoring pool by its private ground truth: non\-identifiable targets are binary abstention tasks \(a committed answer is wrong, not incomplete\), identifiable targets keep their native metric, and abstaining on an identifiable target is an ineligible candidate\. This gives fixed pools — binaryn=34n\{=\}34, pointn=25n\{=\}25, intervaln=14n\{=\}14, graph/setn=27n\{=\}27— common to all three models\. Because point and interval losses are pooled separately here, while the leaderboard’s Med\. NRel\. Err mixes them under the answer\-dependent routing, the continuous medians below are not directly comparable to Tab\.[3](https://arxiv.org/html/2607.08093#S5.T3)\. pass@kk\(Chenet al\.,[2021](https://arxiv.org/html/2607.08093#bib.bib139)\)is the direct empirical estimator: a task is covered if any of its firstkkrestarts passes, and passˆkkrequires allkkto pass\. For continuous targets the analogue is the oracle best\-of\-kkloss\(Brownet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib140)\)— the lowest realized loss among thekkvalid candidates — with worst\-of\-kkits passˆkkcounterpart\. By construction binary coverage is nondecreasing inkkand each task’s best\-of\-kkloss nonincreasing \(the medians below are taken over the valid subsets, which grow withkk, so they need not be monotone\)\.

Table 19:Continuous analogue of pass@kk\(open\-weight models\): oracle best\-of\-kkand worst\-of\-kkmedian loss; cells show median \(valid/total\)\. Graph/set1−F11\{\-\}F\_\{1\}stays0\.0000\.000across restarts \(omitted\)\. Sorted by best@3 within each pool\.PoolModelbest@1best@2best@3worst@3point \(NRelErr\)Gemma 4 26B0\.0660\.066\(25/25\)0\.0520\.052\(25/25\)0\.0340\.034\(25/25\)0\.0910\.091\(24/25\)Kimi K2\.60\.0760\.076\(24/25\)0\.0530\.053\(25/25\)0\.0520\.052\(25/25\)0\.1090\.109\(24/25\)Qwen 3\.6 35B0\.0900\.090\(20/25\)0\.0900\.090\(24/25\)0\.0730\.073\(24/25\)0\.0760\.076\(18/25\)interval \(NRelIS\)Kimi K2\.61\.611\.61\(14/14\)1\.431\.43\(14/14\)1\.431\.43\(14/14\)1\.811\.81\(14/14\)Qwen 3\.6 35B1\.761\.76\(11/14\)1\.601\.60\(12/14\)1\.601\.60\(12/14\)1\.761\.76\(9/14\)Gemma 4 26B1\.791\.79\(14/14\)1\.791\.79\(14/14\)1\.791\.79\(14/14\)2\.332\.33\(13/14\)Three observations from Tab\.[19](https://arxiv.org/html/2607.08093#A1.T19)\. Qwen 3\.6 35B’s missing submissions are largely transient: a second attempt lifts its valid point coverage from20/2520/25to24/2524/25\(interval11/1411/14to12/1412/14\), so the tool\-output failures behind its low single\-run valid\-answer rate \(App\.[A\.15](https://arxiv.org/html/2607.08093#A1.SS15)\) are recoverable by restarting\. This is the source of our judgement in the main text, where we attributed the invalid submissions to the model and not the benchmark\. Kimi K2\.6 suffers from outliers: its best@3 is competitive in both pools \(0\.0520\.052point,1\.431\.43interval — the latter the best\) yet it posts the worst point\-pool worst@3 \(0\.1090\.109\) and by far the largest interval run\-to\-run SD \(0\.0580\.058; Tab\.[20](https://arxiv.org/html/2607.08093#A1.T20)\) — a few estimands flip between near\-correct and far\-off across restarts\. Gemma 4 26B is surprisingly competitive: it leads the point pool at everykk\(best@30\.0340\.034\) despite the worst leaderboard Med\. NRel\. Err \(Tab\.[3](https://arxiv.org/html/2607.08093#S5.T3)\) — its weakness is intervals \(flat at1\.791\.79acrosskk, worst worst@3 at2\.332\.33\), not point estimation\.

Table 20:Run\-to\-run stability and variance on the complete 3\-run grid \(open\-weight models\)\.*Binary pool*\(n=34n\{=\}34\): pass@3 \(any of 3\), passˆ33\(all 3\),*unstable*\(some but not all\), stable\-fail \(all fail\), and mean per\-run successp^¯\\bar\{\\hat\{p\}\}\.
*Run\-to\-run SD*: median per\-task SD of NRelErr / NRelIS over the tasks answered validly in all 3 runs; cells show median \(valid/total\)\. Best per column in bold\.Binary pool \(n=34n\{=\}34\)Run\-to\-run SDModelpass@3passˆ33unstablestable\-failp^¯\\bar\{\\hat\{p\}\}NRelErrNRelISKimi K2\.691\.2%61\.8%29\.4%8\.8%77\.5%0\.000350\.00035\(24/25\)0\.058 \(14/14\)Qwen 3\.6 35B82\.4%55\.9%26\.5%17\.6%67\.6%0\.00024\\mathbf\{0\.00024\}\(18/25\)0\.019 \(9/14\)Gemma 4 26B76\.5%50\.0%26\.5%23\.5%64\.7%0\.00150\.0015\(24/25\)0\.006\(13/14\)

##### Run\-to\-run stability and variance\.

For the CausalDSScore variability reported in the main text \(Tab\.[7](https://arxiv.org/html/2607.08093#S5.T7)\), we score each of the three restarts as a complete exam run through the same scoring pipeline as the leaderboard \(not the ground\-truth routing used for pass@kkabove\); run 1 therefore reproduces the leaderboard values \(Tab\.[3](https://arxiv.org/html/2607.08093#S5.T3)\)\. The reported SD —0\.1020\.102\(Kimi K2\.6\),0\.0400\.040\(Qwen 3\.6 35B\),0\.0450\.045\(Gemma 4 26B\) — is the sample standard deviation of the three per\-run scores\. Taking the SD of the complete scores, rather than combining the variances of the score’s components, keeps the covariance between components in the measurement\. The variability is dominated by the tail\-sensitiveSNRS\_\{\\mathrm\{NR\}\}component \(per\-run SD0\.110\.11–0\.230\.23across models\), while theF1F\_\{1\}\-Loss component is constant across runs\. One caveat: whether a task is graded on content or as an abstention depends on the model’s own answer, so the number of graded continuous submissions varies slightly across runs \(e\.g\., Qwen 3\.6 35B submits3131,3535, and3131valid continuous answers, of3939continuous tasks, over the three runs\), and the per\-run scores aggregate over slightly different denominators\.

On the complete 3\-run grid, all\-restarts consistency \(passˆ33\) sits well below best\-of\-3:50\.050\.0–61\.8%61\.8\\%against pass@376\.576\.5–91\.2%91\.2\\%\(Tab\.[20](https://arxiv.org/html/2607.08093#A1.T20)\)\. The gap is the*unstable*band —2626–29%29\\%of the binary pool passes on some but not all restarts — while stable failures \(0/3\) are8\.8%8\.8\\%\(Kimi K2\.6\),17\.6%17\.6\\%\(Qwen 3\.6 35B\), and23\.5%23\.5\\%\(Gemma 4 26B\), and mean per\-run success is64\.764\.7–77\.5%77\.5\\%\. The continuous metrics are far steadier across restarts: the median per\-task run SD is2\.4×10−42\.4\\times 10^\{\-4\}–1\.5×10−31\.5\\times 10^\{\-3\}NRelErr for point estimands — roughly two orders of magnitude below the point\-loss medians themselves, i\.e\., the agents refit the same released data to near\-identical estimates — against0\.0060\.006–0\.0580\.058NRelIS for intervals, the more variable construction\. That median understates the interval term: most intervals reproduce tightly, but a few effect\-interval estimands \(ate\_uq\_95\) swing between≈0\\approx 0and the cap \(1010\) across restarts \(per\-task NRelIS SD2\.52\.5–5\.55\.5\), and becauseSNRS\_\{\\mathrm\{NR\}\}aggregates intervals with a capped*mean*, those few tasks — not the typical interval, and not the point estimates — drive the run\-to\-runSNRS\_\{\\mathrm\{NR\}\}swing\.

Table 21:Pass\-rate lift \(pass​@​1→pass​@​3\\mathrm\{pass@1\}\\\!\\to\\\!\\mathrm\{pass@3\}, percent\) on the binary pool, sliced by rung, observation variant, and input mode \(open\-weight models, ground\-truth\-fixed pools\)\. The R1 andproxy\_hardslices are small \(n=3n\{=\}3\)\.SliceKimi K2\.6Gemma 4 26BQwen 3\.6 35B*Rung*R1100→100100\\\!\\to\\\!100100→100100\\\!\\to\\\!100100→100100\\\!\\to\\\!100R242\.9→78\.642\.9\\\!\\to\\\!78\.628\.6→57\.128\.6\\\!\\to\\\!57\.150\.0→64\.350\.0\\\!\\to\\\!64\.3R382\.4→10082\.4\\\!\\to\\\!10070\.6→88\.270\.6\\\!\\to\\\!88\.282\.4→94\.182\.4\\\!\\to\\\!94\.1*Observation variant*clean66\.7→94\.466\.7\\\!\\to\\\!94\.450\.0→72\.250\.0\\\!\\to\\\!72\.266\.7→77\.866\.7\\\!\\to\\\!77\.8proxy76\.9→92\.376\.9\\\!\\to\\\!92\.369\.2→84\.669\.2\\\!\\to\\\!84\.684\.6→92\.384\.6\\\!\\to\\\!92\.3proxy\_hard33\.3→66\.733\.3\\\!\\to\\\!66\.733\.3→66\.733\.3\\\!\\to\\\!66\.733\.3→66\.733\.3\\\!\\to\\\!66\.7*Input mode*symbolic72\.7→90\.972\.7\\\!\\to\\\!90\.954\.5→72\.754\.5\\\!\\to\\\!72\.772\.7→86\.472\.7\\\!\\to\\\!86\.4data\-backed58\.3→91\.758\.3\\\!\\to\\\!91\.758\.3→83\.358\.3\\\!\\to\\\!83\.366\.7→75\.066\.7\\\!\\to\\\!75\.0
##### Pass\-rate lift by slice\.

The binarypass​@​1→pass​@​3\\mathrm\{pass@1\}\\\!\\to\\\!\\mathrm\{pass@3\}lift is largest at Rung 2 \(e\.g\., Kimi K2\.6 from42\.942\.9to78\.6%78\.6\\%\) and Rung 3, while Rung 1 is saturated from the start — its binary slice holds onlyn=3n\{=\}3tasks, solved by every model in every run \(Tab\.[21](https://arxiv.org/html/2607.08093#A1.T21)\); by observation variant the relative gain is largest onproxy\_hard, as expected \(all three roughly double off a low base\); and data\-backed tasks gain more than symbolic ones for Kimi K2\.6 and Gemma 4 26B\. The clearest pattern is abstention \(Tab\.[22](https://arxiv.org/html/2607.08093#A1.T22)\): the open models recover much of the abstention they miss first\-try —identificationabstention reaches100%100\\%\(Kimi K2\.6\) and80%80\\%\(Qwen 3\.6 35B\) byk=3k\{=\}3, and every model reaches at least66\.7%66\.7\\%oncounterfactual\_identification\. Even Gemma, which get abstention entirely wrong on three task families, shows some ability given three tries\. Thebias\_diagnostic\(forbidden\-control\) family is the exception, staying low \(≤33\.3%\\leq 33\.3\\%\) even atk=3k\{=\}3\. The per\-family abstention pools are small \(n=2n=2–55\), so these indicative only\. Nonetheless, the open\-weights models areableto distinguish non\-identifiable scenes, just notreliably so\.

Table 22:Abstention recoverability \(pass​@​1→pass​@​3\\mathrm\{pass@1\}\\\!\\to\\\!\\mathrm\{pass@3\}, percent\) on the non\-identifiable targets, by task family \(open\-weight models\)\. Each cell is the fraction of the family’s non\-identifiable targets correctly abstained on withinkkrestarts\. Per\-family pools are small \(n=2n=2–55\);mediation\_effecthas no non\-identifiable target\.FamilyKimi K2\.6Gemma 4 26BQwen 3\.6 35Bidentification60\.0→10060\.0\\\!\\to\\\!1000\.0→40\.00\.0\\\!\\to\\\!40\.040\.0→80\.040\.0\\\!\\to\\\!80\.0effect\_estimate0\.0→66\.70\.0\\\!\\to\\\!66\.733\.3→66\.733\.3\\\!\\to\\\!66\.733\.3→33\.333\.3\\\!\\to\\\!33\.3bias\_diagnostic0\.0→33\.30\.0\\\!\\to\\\!33\.30\.0→33\.30\.0\\\!\\to\\\!33\.333\.3→33\.333\.3\\\!\\to\\\!33\.3counterfactual\_id66\.7→10066\.7\\\!\\to\\\!10066\.7→66\.766\.7\\\!\\to\\\!66\.766\.7→10066\.7\\\!\\to\\\!100counterfactual\_effect100→100100\\\!\\to\\\!1000\.0→1000\.0\\\!\\to\\\!10050\.0→10050\.0\\\!\\to\\\!100

### A\.12Robustness to the CauseNet\-seeded verbalization

CauseNet seeding \(Sec\.[3\.3](https://arxiv.org/html/2607.08093#S3.SS3)\) injects recognizable real\-world causal pairs into the story and variable names\. One concern is whether this semantic layer leaks the answer, letting a model solve a scene from a familiar causal template in the prose rather than from the released structure and data — the “causal parrot” failure mode\. The more general concern is stability: scores should be a function of the formal problem, not of the particular story drawn for it\. Because every story is audited to be faithful to its graph, a model whose answers move under a story swap is failing on its own terms — exactly the behavior the benchmark is meant to expose — but a persistent per\-scene movement across models could indicate dependence on the story draw\. We probe both with a*matched verbalization\-swap*ablation that holds the formal problem fixed and varies only the seeded verbal grounding\.

##### Construction\.

From the main scene pool we form ten*bundles*\. Within a bundle the motif, the raw node order, and the per\-node variable\-type signature are identical, and one representative member’s clean data and private ground truth are mapped back onto every member’s variable names, so all members share the same numbers, the same scoring target, and the same private truth\. The members thus differ only in their CauseNet\-seeded story and variable names, giving four verbalizations of a single fixed formal problem\. Each bundle contributes one task, for4040exam items spanning ten motifs and both symbolic and data\-backed families\. We evaluate two open\-weight agents, Kimi K2\.6 and Qwen 3\.6\-35B\.

##### Diagnostic loss\.

To place heterogeneous task outputs on one scale we use a “diagnostic loss” \(higher is worse\):1−score1\-\\text\{score\}for boolean, F1, and abstention tasks; the reported normalized relative error, RMSE, or interval score for continuous tasks; and11for a missing or unparseable answer\. The quantity of interest is the*within\-bundle range*of this loss: how much the outcome moves when only the verbalization changes while the formal problem is held fixed\.

Table 23:Matched verbalization\-swap ablation: within\-bundle range of the diagnostic loss \(max−\-minover the four verbalizations of one fixed formal problem; higher means the outcome moved more when only the story and variable names changed\)\. The lower block summarizes each model over the ten bundles\.BundleMotifRungTask familyKimi K2\.6Qwen 3\.6\-35B1arrowheadR2effect \(ATE\)3\.55×10−113\.55\\times 10^\{\-11\}0\.003122chainR2sketch0\.00\.03colliderR1association2\.65×10−112\.65\\times 10^\{\-11\}8\.20×10−58\.20\\times 10^\{\-5\}4confoundingR2identification0\.00\.05diamondR1prediction0\.006180\.1076diamondcutR2bias diagnostic0\.00\.07frontdoorR3counterfactual effect0\.8920\.6148double\_ncR3counterfactual ident\.0\.01\.009colliderR1collider phenomenon0\.00\.010mediationR2effect \(ATE\)0\.00\.00299Bundles with nontrivial spread \(\>0\.001\>0\.001\)2/105/10Bundles with large spread \(\>0\.05\>0\.05\)1/103/10Mean within\-bundle loss SD0\.0520\.086Mean / max within\-bundle range0\.090 / 0\.8920\.173 / 1\.00
##### The stronger agent is more stable\.

Table[23](https://arxiv.org/html/2607.08093#A1.T23)shows that most bundles are flat: every discrete\-output bundle outside Rung 3 \(sketch, identification, bias diagnostic, collider phenomenon\) is invariant for both models, and the ordinary Rung\-1/2 numeric bundles move only marginally, determined by the particular numeric approach the model picks on the scene\. Kimi is largely invariant, with nontrivial spread on2/102/10bundles and a single large \(\>0\.05\>0\.05\) swing\. Qwen is materially more sensitive:5/105/10bundles move and three swing widely\. The instability concentrates in the two Rung\-3 bundles \(counterfactual effect and identification\), where the ranges reflect discrete flips \(commit vs\. abstain, identifiable vs\. not\) rather than numeric dispersion, and, for Qwen, one binary prediction bundle where the ranking of test cases is preserved across stories \(AUC0\.730\.73–0\.750\.75, with the discrepant story in fact the highest\) while the emitted probabilities become poorly calibrated \(Brier0\.1280\.128against0\.0480\.048–0\.0500\.050on the other three verbalizations\): the story changed how confident the predictions are, not which cases the model scores higher\.

##### The flips are story\-linked, not run noise\.

The movement that does exist is attributable to the story layer, not to sampling variation\. On the stable numeric bundles, Kimi’s estimates are identical across all four verbalizations to ten or more significant digits and Qwen’s vary by at most3×10−33\\times 10^\{\-3\}in relative error: given the same numbers, the agents run essentially deterministic pipelines\. In bundle 7, by contrast, both models abstain on the same two stories and attempt the estimate on the other two \(Kimi with numerically identical estimates; Qwen with one estimate and one unparseable answer\) — two agents agreeing on*which*stories trigger an unwarranted abstention is not single\-run stochasticity\. This is precisely the persistent cross\-model movement flagged above as the possible signature of story\-draw dependence, and the only place the ablation finds it\. We nevertheless argue that the coordinated behavior is mostly model\-driven: it is a residue of the “causal parrot” failure mode \(also cf\. the next paragraph\), and a competent agent would score identically on all four members\. That this happens less often overall for the model otherwise benchmarked as the stronger one speaks in favor of this interpretation and against the “null hypothesis” that the verbalization seeding makes the benchmark unstable\. What remains on the benchmark side is limited: for susceptible models, the story a scene is told with can make it harder or easier\. This is thus a latent factor in thedifficulty, not an indictment of thestabilityof the benchmark\.

##### A concrete semantic\-lure failure\.

The clearest case is bundle 8 \(double\_ncmotif, symbolic counterfactual identification of the ETT\), where the estimand is*non\-identifiable*\. Kimi answers non\-identifiable throughout\. Qwen also answers non\-identifiable on the three stories seeded with the hypothermia–unconsciousness, illness–disability, and events–arrest pairs, but flips to*identifiable*on the fourth \(bundle008\_\_scene\_000409; source scenescene\_000409; the other ablation members arebundle008\_\_scene\_000104,bundle008\_\_scene\_000527, andbundle008\_\_scene\_000684\), whose prose describes rainfall\-gauge and satellite\-precipitation measurements of a latent precipitation confounder:

> “Because researchers cannot directly observe the true annual precipitation totals across the entire heterogeneous mountainscape, they rely on two complementary assessments: the Rainfall Gauge Measurement collected at fixed field stations, and the Satellite Precipitation Estimate derived from orbital remote sensing algorithms\. \[…\] The true precipitation levels driving these fire dynamics remain latent variables inferred through the gauge and satellite observations rather than measured continuously across all terrain\.”

From those measurement descendants Qwen constructs a proxy/back\-door identification argument\. The lure is a sophisticated one: the two\-proxy prose mirrors the double\-negative\-control setting of proximal causal inference\(Miaoet al\.,[2018](https://arxiv.org/html/2607.08093#bib.bib132); Tchetgenet al\.,[2024](https://arxiv.org/html/2607.08093#bib.bib136)\), and the conditions the model asserts have exactly that shape\. Even in that framework, however, point identification rests on completeness assumptions that are not encoded in the graph and that the model neither states nor checks\. Under the benchmark’s graph\-level identifiability policy, observed descendants of a latent confounder do not license identification, so this is a template imported from the story without the assumptions it needs, rather than a property of the graph — exactly the behavior the fully synthetic construction is meant to expose\.

##### Pair commonness does not explain the variation\.

We recovered the CauseNet support count and salience percentile of each scene’s seeded pair and regressed the diagnostic loss on commonness with bundle fixed effects\. The within\-bundle slope is negative for both models \(Kimi−0\.05\-0\.05, bootstrap95%95\\%CI\[−0\.18,0\.00\]\[\-0\.18,0\.00\]; Qwen−0\.45\-0\.45, CI\[−1\.08,0\.00\]\[\-1\.08,0\.00\]\), and the within\-bundle Spearman correlation is\+0\.05\+0\.05\(Kimi\) and−0\.27\-0\.27\(Qwen\)\. The sign is consistent with more common pairs being marginally easier, but the intervals include zero, two bundles carry no within\-bundle commonness variation\. We therefore report commonness as an inconclusive moderator: the seeded verbal grounding affects the weaker agent, but that effect is not sufficiently explained by CauseNet pair commonness in this diagnostic\.

##### Takeaway\.

The two results read as the two halves of the motivating concern\. The stronger agent’s near\-invariance — exact agreement of its answers across seeded pairs with different real\-world scenarios— indicates that the score measures the formal problem rather than the seeded verbalization: the benchmark is robust to CauseNet seeding for an agent that works from the released structure and data\. The weaker agent’s movement is shown likely to be causal\-parrot susceptibility itself: because the formal problem is held fixed, a story\-dependent answer can only arise from importing semantics that the graph and data do not support, and the matched design isolates precisely that\. Kimi’s single unstable bundle shows the same susceptibility in trace form: the failure mode recedes with capability but does not vanish\.

### A\.13Observation\-layer hardness under matched scenes

The observation layer \(Sec\.[3\.2](https://arxiv.org/html/2607.08093#S3.SS2)\) is designed to change how hard an identified functional is to estimate from the released files while preserving the conceptual SCM, the target estimand, and the identifiability label\. The per\-variant results in the main text \(Tabs\.[5](https://arxiv.org/html/2607.08093#S5.T5)and[6](https://arxiv.org/html/2607.08093#S5.T6)\) are scored on disjoint subsets of the exam\. This appendix, conversely, reports the controlled companion: a matched ablation in which the same conceptual scene/task is evaluated under all three released observation views\.

##### Construction\.

From the main scene pool we select ten matched scene/task pairs whose source scene releases all three observation variants, spanning the data\-backed continuous families\. Only identifiable targets enter, so abstention is never the correct response\. The10×3=3010\\times 3=30instances form a single randomized exam\. The observation model used is hidden from the agent, as are previously submitted answers\. We evaluate the two open\-weight agents, Kimi K2\.6 and Qwen 3\.6\-35B \(the pair also used in App\.[A\.12](https://arxiv.org/html/2607.08093#A1.SS12)\)\. The paired deltas are descriptive contrasts on these ten selected pairs; they are not estimates of a population\-level effect of the observation layer\.

##### Diagnostic loss\.

For evaluation purposes, we define a \(capped\)diagnostic loss\(higher is worse\): the normalized relative error \(point/scalar outputs\) or normalized interval score \(intervals\), capped at11; a missing, unparseable, or runtime\-failed answer scores11; and — because every selected target is identifiable — an abstention also scores11, as it’s always incorrect\. This convention is specific to this ablation and differs from the main leaderboard grading, where abstention is scored as its own metric and the continuous pools contain only valid numeric answers\. The paired contrasts are:Δproxy=Lproxy−Lclean\\Delta\_\{\\mathrm\{proxy\}\}=L\_\{\\mathrm\{proxy\}\}\-L\_\{\\mathrm\{clean\}\}andΔhard=Lproxy​\_​hard−Lclean\\Delta\_\{\\mathrm\{hard\}\}=L\_\{\\mathrm\{proxy\\\_hard\}\}\-L\_\{\\mathrm\{clean\}\}per pair, with paired\-bootstrap95%95\\%intervals over the ten pairs\. The survivor\-only view — deltas over pairs withvalidanswers in both arms is reported alongside\.

Table 24:Matched observation\-layer ablation: paired capped\-loss contrasts over the ten matched scene/task pairs \(mean±\\pmSD across pairs; paired\-bootstrap95%95\\%CI;*worse*— the number of pairs scoring withΔ\>0\\Delta\>0\)\. The survivor\-only column conditions on a valid numeric answer in both arms, with the number of contributing pairs in parentheses\.ModelContrastMean±\\pmSD95%95\\%CIWorseSurvivor\-only \(nn\)Kimi K2\.6proxy−\-clean−0\.02±0\.45\-0\.02\\pm 0\.45\[−0\.293,\+0\.240\]\[\-0\.293,\+0\.240\]7/10−0\.02\-0\.02\(10\)Kimi K2\.6proxy\_hard−\-clean\+0\.20±0\.27\+0\.20\\pm 0\.27\[\+0\.053,\+0\.373\]\[\+0\.053,\+0\.373\]8/10\+0\.14\+0\.14\(8\)Qwen 3\.6\-35Bproxy−\-clean\+0\.20±0\.42\+0\.20\\pm 0\.42\[\+0\.003,\+0\.498\]\[\+0\.003,\+0\.498\]5/10\+0\.01\+0\.01\(5\)Qwen 3\.6\-35Bproxy\_hard−\-clean\+0\.33±0\.47\+0\.33\\pm 0\.47\[\+0\.070,\+0\.626\]\[\+0\.070,\+0\.626\]6/10\+0\.08\+0\.08\(4\)Table 25:Matched observation\-layer ablation: per\-variant behavior\.*False abstentions*counts abstentions for the tasks, all of which were designed to have identifiable targets;*context exits*counts trajectories terminated by a context\-window overflow ;*median survivor loss*is the median capped loss over valid numeric answers only\.ModelBehaviorcleanproxyproxy\_hardKimi K2\.6valid answers \(of 10\)10108false abstentions \(of 6\)002context exits000mean calls per task16\.526\.726\.0median survivor loss0\.0060\.0390\.152Qwen 3\.6\-35Bvalid answers \(of 10\)864false abstentions \(of 6\)212context exits055mean calls per task13\.319\.619\.7median survivor loss0\.0050\.0300\.163![Refer to caption](https://arxiv.org/html/2607.08093v1/x6.png)Figure 9:Matched observation\-layer ablation: capped diagnostic loss for all6060instances \(1010matched scene/task pairs×\\times33observation variants×\\times22models\)\. Cell shading and the printed value give the capped loss; failed cells are labeled by cause \(*abst*— false abstention,*ctx*— context\-window failure with no gradable answer\), and an asterisk marks an answer graded despite a subsequent failure\.
##### The matched contrast confirms the hardness ordering\.

Table[24](https://arxiv.org/html/2607.08093#A1.T24)shows both models scoring worse underproxy\_hard\(per\-instance results in Fig\.[9](https://arxiv.org/html/2607.08093#A1.F9)\), with paired\-bootstrap intervals excluding zero \(Kimi\+0\.20\+0\.20,8/108/10pairs worse; Qwen\+0\.33\+0\.33\)\. The middle tier separates the models: Qwen already degrades atproxy\(\+0\.20\+0\.20, CI\[\+0\.003,\+0\.498\]\[\+0\.003,\+0\.498\]\), while Kimi’sproxycontrast is flat on average \(−0\.02\-0\.02against a\+0\.01\+0\.01median, the mean pulled down by one interval pair whosecleananswer was already at the loss cap while itsproxyanswer happened to score well\)\. Median deltas are moderate throughout \(≤0\.06\\leq 0\.06\): the mean contrast is carried by a subset of pairs that fail outright under the harder view\.

##### Hardness acts through two separate effects\.

The first effect is statistical: among instances that produce a valid answer, estimation degrades\. The median survivor loss rises0\.006→0\.1520\.006\\to 0\.152\(Kimi\) and0\.005→0\.1630\.005\\to 0\.163\(Qwen\) fromcleantoproxy\_hard\(Tab\.[25](https://arxiv.org/html/2607.08093#A1.T25)\), and Kimi’s survivor\-onlyΔhard\\Delta\_\{\\mathrm\{hard\}\}is\+0\.14\+0\.14with7/87/8pairs worse — and since it is exactly the failing instances that leave the survivor pool, these conditional numbers if anything understate the effect\.

The second effect is agentic: the harder views increasingly prevent a valid answer at all\. Overclean/proxy/proxy\_hard, valid answers fall10/10/810/10/8\(Kimi\) and8/6/48/6/4\(Qwen\)\. The underlying driver is workload: both models work harder as the view hardens \(Kimi16\.5→∼2616\.5\\to\{\\sim\}26calls per task, Qwen13\.3→∼2013\.3\\to\{\\sim\}20\), and the extra work ends differently for the two agents\. For Qwen, whose deployment serves a 32k\-token window, the longer exploration is precisely what exhausts the context: every one of its failures is a context\-window overflow \(ten overflow exits across the two proxy tiers\)\. Kimi never exhausts its context; instead it starts abstaining underproxy\_hard— plausibly the added measurement uncertainty tipping it into judging an identifiable target unanswerable \(its abstention cases are examined below\)\. Failing to finish is not incidental noise but a second, separate signal that the observation layer stresses the whole agentic workflow, not only estimator accuracy\. The decomposition in Fig\.[10](https://arxiv.org/html/2607.08093#A1.F10)separates the two contributions to the capped mean — for Qwen the failing pairs account for\+0\.30\+0\.30of the\+0\.33\+0\.33, while Kimi’s splits into\+0\.11\+0\.11from surviving estimates and\+0\.08\+0\.08from abstention onset — though these shares reflect the failure\-scores\-11convention as much as the data and should not be read as a ranking of the two effects\. On the main exam’sproxy\_hardslice \(caveat: different tasks\), the frontier models show no comparable failures: counting both missing answers and false abstentions, Claude Opus 4\.8 and Gemini 3\.1 Pro drop5%5\\%of tasks and GPT\-5\.510%10\\%, which is comparable to their overall rate, with no false abstentions at all\.

![Refer to caption](https://arxiv.org/html/2607.08093v1/x7.png)Figure 10:Exact decomposition of the mean paired capped\-loss delta over the ten matched pairs, by what happened to answer validity between thecleanarm and the comparison arm: pairs valid in both arms contribute their estimation\-error change; pairs valid undercleanbut failed under the comparison contribute1−Lclean1\-L\_\{\\mathrm\{clean\}\}; pairs without a valid answer in either arm are saturated at the cap and contribute zero\.
##### Abstention as a hardness\-triggered exit\.

All seven abstentions in the ablation fall on the same two pairs \(twelve instances: two models under three variants each\) — an ETT interval and an NIE interval, both on frontdoor\-motif scenes whose targets theID\*/IDC\*\-based algorithm certifies as identifiable\. The submitted explanations argue from textbook sequential\-ignorability and backdoor conditions: the models locate the latent confounder named in the story and declare the estimand non\-identifiable, missing the frontdoor\-type identification\. Kimi’s abstentions are hardness\-triggered exits rather than up\-front identification calls: it commits to estimates on both pairs undercleanandproxy, and itsproxy\_hardtrajectories show it fitting proxy\-reconstruction models and implementing the estimators — including a front\-door ETT estimator, and, on the NIE pair, writing the abstention, deleting it, and attempting two further numerical approaches — before submitting a null answer whose graph\-level rationale contradicts its own attempts\. The stated non\-identifiability argument is available in every arm; what changes under the hardest view is the model’s willingness to stand behind an estimate\.

##### Interval calibration and caveats\.

The interval tasks split into the same two regimes as the main exam: per\-row prediction intervals stay near their nominal90%90\\%coverage under every variant \(0\.860\.86–0\.900\.90\), while the single intervals for causal effects almost never cover their nominal95%95\\%— the causal\-uncertainty overconfidence of Tab\.[6](https://arxiv.org/html/2607.08093#S5.T6)is not created by the observation layer\. The ablation is deliberately small: ten pairs, two open\-weight agents, and one run per cell, so per\-family splits are not especially meaningful\.

### A\.14Per\-trajectory failure modes on the scene presented in Fig\.[2](https://arxiv.org/html/2607.08093#S3.F2)

As shown in Fig\.[2](https://arxiv.org/html/2607.08093#S3.F2), the running example \(scene\_000154\) is an instrumental\-variable scene narrated as an agricultural study: an Irrigation Activation SubsidyZZinstruments whether an Irrigation Event is Executed \(XX, binary\), which in turn drives the Field Soil Erosion RateYY, while an unobserved Subsurface Soil PermeabilityUUconfoundsXXandYY\. The instrumentZZ, treatmentXX, and outcomeYYare each observed only through independent noisy measurement bundles \(theproxy\_hardvariant\)\. BecauseUUis unmeasured and the only identifying route is the instrument, the*population*ATE is not point\-identified:ZZidentifies at most a local, compliance\-restricted effect, notE​\[Y∣do​\(X=1\)\]−E​\[Y∣do​\(X=0\)\]E\[Y\\mid\\mathrm\{do\}\(X\{=\}1\)\]\-E\[Y\\mid\\mathrm\{do\}\(X\{=\}0\)\]\(y0\.id/dowhyboth output the same verdict\)\. In the presented exam the harness administers this scene as a forbidden\-controls bias diagnostic: with the subsidy instrument the only observed variable besides the treatment and outcome, an agent must either list the observed variables that are forbidden controls or declare the population effect not identifiable, which is the correct answer\. Table[26](https://arxiv.org/html/2607.08093#A1.T26)shows that only two of the six models—Gemini 3\.1 Pro and GPT\-5\.5—return it\. The other four answerno\_backdoor: they correctly observe that no observed backdoor adjustment set exists, but then assert that the population ATE is nonetheless identifiable, treating the subsidy as an instrument that recovers it\. That is precisely the failure mode the scene targets—the instrument identifies only a local, compliance\-restricted effect, not the population ATE—sono\_backdoorover\-claims identifiability and is scored as a failed abstention\. The trajectories show that the fourno\_backdooragents reach the same over\-claim along very different routes—from deriving the correct answer in full and then discarding it in favor of guessed grader convention \(Kimi K2\.6\) to never questioning the instrument\-identifies\-ATE step at all \(Gemma 4 26B\)—while GPT\-5\.5 abstains without any exploration whatsoever \(Tab\.[26](https://arxiv.org/html/2607.08093#A1.T26)\)\.

Table 26:Per\-agent responses on the forbidden\-controls bias diagnostic for the running examplescene\_000154\(Fig\.[2](https://arxiv.org/html/2607.08093#S3.F2)\); quotes are verbatim from the agents’ own reasoning traces\. All six agents reconstruct the same graph and correctly conclude that no observed backdoor adjustment set exists; the answers diverge only on the final step—whether the subsidy instrument point\-identifies the*population*ATE\. Failing agents ordered by how much of the correct argument appears in their trace\.ModelAnswerDiagnosisGemini 3\.1 Pronon\_idStarts where the failures end \(“S is a valid instrument”\), then prosecutes its own claim at length: recalls that nonparametric IV with a binary treatment yields only Balke–Pearl bounds, notices that the story’s threshold mechanism \(subsidy sufficient*and*permeable soil\) “definitively breaks the linear SEM assumptions” that would rescue point identification, and reverses tonon\_id\.GPT\-5\.5non\_idAnswers in its first and only step, without opening anything in the workspace: the subsidy is “an instrument\-like cause of treatment, not a variable that blocks the backdoor path”, so the ATE “is not nonparametrically identifiable from this graph by adjustment or another graph\-based strategy”\. The nonparametric standard is applied by default, with no deliberation about what the grader might accept\.Kimi K2\.6no\_backdoorIts thinking trace contains the correct answer in full: it derives that IV identifies “the local average treatment effect \(LATE\)…but the population ATE may not be”, and even works out the bias\-amplification algebra for conditioning on an instrument—then bets against its own analysis on guessed grader convention: “the question likely comes from a course or dataset where they expectno\_backdoor” and “the problem might not be that nuanced”\.666Kimi K2\.6’s meta\-reasoning ispreciselythe failure modeCausalDS’s combination of fully synthetic generation and first\-class abstention grading is designed to expose: fitting the answer to an imagined exam sourced from a textbook rather than to the problem at hand\.Claude Opus 4\.8no\_backdoorReaches the correct fork, then bridges it with parametric assumptions it reads into the narration: “the population ATE may be identifiable via IV under linearity/additivity assumptions\. The story describes \[the outcome\] as linear\-additive combination…So ATE is identifiable by IV”\. The story states no functional form \(the outcome “reflects the combined action” of treatment and confounder\), and the task is symbolic—there is no released data against which linearity could be checked, and its sole exploration is reading the story\. The “rescue” rests on an assumption that is neither stated nor checkable\.Qwen 3\.6 35Bno\_backdoorReverses its verdict four times \(“Let me go withnon\_idas the safer answer…OK let me just commit tono\_backdoor”\), in between spending most of a 60\-message trajectory re\-printing the 1\.6 kB story it already had verbatim in the prompt\. The final commitment verifies the three instrument conditions—relevance, exclusion, independence, all of which*do*hold—and treats that checklist as the finish line: “So IV identification is valid\! The answer should beno\_backdoor\.” But those conditions \(plus the monotonicity itneverinvokes\) deliver only the complier\-local effect, not the population ATE the task asks for\. Unlike the two agents above, it never registers that a bridging step is missing: where Kimi K2\.6 derived the LATE\-vs\-ATE gap and gambled against it, and Claude Opus 4\.8 filled it with an invented linearity assumption, Qwen never surfaces the estimand question at all—LATE, monotonicity, and effect homogeneity appear nowhere in its trace\.Gemma 4 26Bno\_backdoorNever questions the step at all: “Is the ATE identifiable? Yes, via IV” is asserted flatly, and the entire deliberation—written into a 22 kB draft of the answer file itself, whose first version answered\[\]—concerns which answer label fits the definition of a forbidden control, not whether the instrument identifies anything\.
### A\.15Technical details

##### Containerization and package pinning\.

Evaluation runs use a Docker / Singularity image whose Python 3\.11 stack is pinned to exact versions to avoid mistakes due to API changes:pandas==3\.0\.2,numpy==2\.4\.4,scipy==1\.17\.1,scikit\-learn==1\.8\.0,statsmodels==0\.14\.6,pyarrow==24\.0\.0,matplotlib==3\.10\.9,seaborn==0\.13\.2,networkx==3\.6\.1,xgboost==3\.2\.0\)\.

The benchmark task prompts list the versions so the agent does not waste budget probing the environment for the right API\.

##### Workspace layout and answer validation\.

Singularity is launched with\-\-no\-homeandHOME=/workspaceso that submitted answer files land in a path the grader will actually read\. After each task the runner re\-checks that the expected answer file exists under/workspace/answersand is minimally parseable as JSON or CSV; failures are recorded asMissingAnswerAfterSubmitrather than counting silently as submitted\. Top\-level scratch under/workspaceis archived after each task to<output\_dir\>/workspace\_scratch/<scene\_id\>\_<task\_id\>\.tar\.gz, then removed before the next task starts so that scratch artifacts cannot leak across the benchmark’s per\-task fresh\-conversation contract; The given scenes \(scenes/\), the model’s answer directory \(answers/\), andINSTRUCTIONS\.mdpersist as the benchmark state\.

##### Provider routing, rate limits and harness details

The three frontier models are routed through OpenRouter withprovider\.onlypinned \(toanthropic,google\-ai\-studio, andopenairespectively\) andallow\_fallbacks=false, so failures appear as observable retry warnings rather than silently rerouting to a different endpoint or quantization; each is run at high reasoning effort \(Claude Opus 4\.8 additionally with adaptive thinking\) as this is the closest to a “default” for two of the models and hence gives the most comparable results\. The three open\-weight models \(Qwen 3\.6 35B, Kimi K2\.6, Gemma 4 26B\) are served locally through an OpenAI\-compatible \(vLLM\) endpoint\. We used their default serving configuration, as no standardized reasoning\-effort interface exists: the harness sends no sampling or reasoning parameters, so sampling follows each model’s released generation defaults\. In practice the two reasoning\-capable models \(Qwen 3\.6 35B, Kimi K2\.6\) emit thinking traces by default, while Gemma 4 26B has no reasoning mode\. We instead report token usage and tool\-call counts for all models to make inference\-compute differences visible\. One behavior specific to Qwen 3\.6 35B partly explains its lower valid\-continuous rate \(79\.5%79\.5\\%, Tab\.[3](https://arxiv.org/html/2607.08093#S5.T3)\): it manages tool output poorly, frequently emitting very long outputs — for instance, printing the entire provided dataset into the conversation instead of usinghead/tailor computing aggregates over it\. The harness does not progressively trim accumulated context; when a*single*command’s output exceeds a length threshold it instead shows only that command’s head and tail, together with an explicit warning to usehead/tail/grepor redirect to a file\. Qwen repeatedly ignored this and lost the numeric values it needed on the hardest continuous estimands\. We re\-ran the exam with it at several output\-budget thresholds \(11k,3\.23\.2k, and the default1010k characters used for the other models\) and the behavior persisted, so we count these lost submissions as a genuine tool\-use failure rather than a harness artifact\. On the near\-one shot submission style,Xuet al\.\([2026b](https://arxiv.org/html/2607.08093#bib.bib154)\)find that even after explicitly discouraging first\-pass submissions via the prompt when using the GPT\-family withmini\-swe\-agent, the behavior nonetheless persists, indicating that it is not purely harness\-induced\.

### A\.16Prompt templates

#### A\.16\.1Generation pipeline prompt templates

The generation pipeline \(Sec\.[3\.3](https://arxiv.org/html/2607.08093#S3.SS3)\) uses separate LLM sessions for variable mapping, feasibility checking, consistency auditing, story generation, and story verification\. Below we reproduce the core system and user prompt templates as used at the current benchmark version\. The variable\-mapping templates live incausalds/schemas\.py; feasibility\-check, consistency\-audit, story\-generation, and story\-verification templates live incausalds/pre\_auditor\.py,causalds/mapping\_audit\.py,causalds/verbalization\_story\.py, andcausalds/verbalization\_verify\.py, respectively\. Runtime template variables \(e\.g\.,\{format\_description\},\{serialized\_graph\}\) are filled with the graph serialization and scene\-specific details at generation time\.

##### Variable mapping system prompt\.

Instructs the mapper LLM to choose a coherent domain and assign human\-friendly variable names to abstract node identifiers\. The prompt enforces that latent/unobserved nodes are correctly marked based on the graph’sobserved\_nodeslist\. Mapping runs in unstructured \(prompt\) mode:\{output\_format\_block\}inlines the requested output format — JSON for the dataset presented — as a skeleton \(proposed\_domainplus onevariable\_mappingentry ofid/story\_name/observed/type/unitper node\) followed by per\-field instructions\. The runtime template variables\{format\_description\}and\{independence\_note\}expand to a graph\-format reminder and \(optionally\) a list of conditional independencies\.

Youareanexpertincausalinference,domainmodeling,and

statisticaldependencies\.

Youwillbegivenasetofabstractvariables\(possiblyasadirected

acyclicgraph\)\.Yourgoalistomapthesevariablestoarealistic,

coherent,andscientificallyplausibledomain\(e\.g\.,epidemiology,

economics,physics\)thatisconsistentwiththeprovideddependencies\.

FormatSpecification:

\{format\_description\}

\{independence\_note\}

Guidelines:

1\)Analyzethefixednodes\(ifany\)toinferthedomaincontext\.

2\)Renamethetargetnodestospecific,measurable,andrealistic

variablesthatfitthisdomain\.Avoidgenericnameslike"FactorA"

or"VariableX"\.

3\)Ensurethe’unit’fieldcontainsrealisticmeasurementunits

\(e\.g\.,"mmHg","years","kg","counts"\)and’type’isappropriate

\(e\.g\.,"continuous","binary"\)\.Thisisnotascrucialforthe

UNOBSERVEDvariables\.

4\)Thechosenvariablesmustmakesensetogetherinasinglescenario\.

5\)CRITICALfor"observed"field:Checkthe"observed\_nodes"listinthe

graph\.Setobserved=trueONLYfornodesinthatlist\.AnynodeNOTin

observed\_nodesisLATENT/unobserved\-youMUSTsetobserved=false

forthese\.

\{output\_format\_block\}

##### Variable mapping user prompt template\.

Optional placeholders are filled by the grafting and anchor logic\. The auxiliary stage substitutes\{additional\_requirements\_block\}with a directive to keep the shared anchor immutable, reuse its existing story name, and avoid meanings that would imply unintended direct links to existing variables; the final merged\-graph audit substitutes a softer directive that lets minor wording refinements pass while preserving anchor identity, scope, type, and unit\. The\{output\_format\_name\}expands to eitherJSONorXML\.

Youwillbegivenagraphofvariablesrepresentedas\{format\_label\}\.

Yourtaskistochooseacoherentdomainandassignhuman\-friendly

namestonodeids\.\{tool\_note\}

Thegraphrepresentationfollowsbelow:

\{serialized\_graph\}

\{independence\_section\}

Context:

\-Fixednodes\(alreadynamed\):\{fixed\_nodes\}

\-\>\{fixed\_nodes\_instruction\}

\-Nodesneedingnames:\{needs\_names\}

\-\>Renamethesetofitthecontextofthefixednodes\.

\{fixed\_name\_assignments\_block\}\{forbidden\_story\_names\_block\}

\{domain\_hint\_block\}\{additional\_requirements\_block\}

\{existing\_graph\_mapping\_block\}\{anchor\_context\_block\}

ReturnONLYthe\{output\_format\_name\}object\{extra\_instruction\}\.

##### Variable mapping feedback prompt\.

Used in the audit/repair loop \(Alg\.[3](https://arxiv.org/html/2607.08093#alg3)\) when the previous mapping fails consistency checks\.\{output\_format\_name\}again expands toJSONorXML\.

CRITICAL:YourpreviousmappingFAILEDacausal\-consistencyaudit\.

YouMUSTrevisevariablemeaningssothatthefollowingviolations

nolongerapply:

\{violation\_block\}

Especially:foreveryNON\-EDGEpair,ensurethereisNOplausible

directcausallinkineitherdirection\.

ReturnONLYthe\{output\_format\_name\}mappingobjectinthesame

schemaasbefore\.

##### Pre\-audit system prompt\.

The pre\-auditor \(Alg\.[2](https://arxiv.org/html/2607.08093#alg2)\) checks whether CauseNet\-fixed node concepts can plausibly satisfy the graph’s structural constraints before the mapper runs\. The prompt defines a mediator\-aware working definition of “direct causal link” and a HIGH/MEDIUM/LOW classification for non\-edge inevitability that explicitly considers whether a mediator could absorb the effect\. The block below is condensed: we omit the in\-prompt salvageable / hopeless examples, the surrounding context notes, and the output\-format instructions, and the inlinefeasible=true/falseannotations summarize the prompt’s decision\-summary lines\.

Youareacausalstructurefeasibilitychecker\.

Youwillbegiven:

1\.AcausalgraphstructurewithFIXEDNODECONCEPTS\(alreadynamed

fromaknowledgebase\)

2\.Thestructuralconstraintsimposedbythegraph:

\-DIRECTEDGES:CausalrelationshipsthatMUSTbeplausible

\-NON\-EDGES:PairswhereNOdirectcausallinkmustexist

YOURTASK:DecidewhetherthereisatleastoneMAINSTREAM,

non\-contrivedinterpretationunderwhichtheFIXEDNODEconcepts

satisfythestructuralconstraints\.Ask:"Canaskilledmapper

typicallyfindacoherentinterpretationthatmakestheconstraints

work?"ThisisaPRE\-FILTER;aseparate,thoroughauditrunsAFTER

mapping\.CatchonlyOBVIOUSfailures;defaulttolettingborderline

casesthrough\.

WORKINGDEFINITIONofDIRECTCAUSALLINK:

’ThereexistsaninterventiononUthatchangesVwhileholdingfixed

theothervariablesinthegraph\(especiallytheothernodesthat

couldmediatetheeffect\)\.’

NON\-EDGECHECK:

ForeachNON\-EDGEpairoffixednodes,classifytheinevitabilityof

adirectcausallinkinmainstreamusage:

\-HIGH:Adirecteffectwouldlikelypersistevenafterintroducing

amediator\(residualdirectpathwayremainsplausible\)\.

\-\>feasible=false

\-MEDIUM:Adirecteffectisplausiblebutamainstream

operationalization\(e\.g\.,administrativeproxy,eligibilityrule,

time\-indexedexposure\)canreasonablyremoveit\.

\-\>feasible=true,confidence="low"

\-LOW:Adirecteffectisnotatypicalinterpretation\.

\-\>feasible=true

EDGEPLAUSIBILITYCHECK\(applyverygenerously\):

Foreachdirectedge,acceptANYmechanism\(evenindirect,weak,or

context\-dependent\)\.OnlyflagasHOPELESSiftheedgeisphysically

impossibleorlogicallycontradictory\.

DECISIONRULE:DefaulttoFEASIBLEwithconfidence="low"when

uncertain;letthefullauditcatchactualproblems\.

##### Mapping audit system prompt\.

The mapping auditor enforces causal consistency of the proposed variable mapping against the graph structure\. Non\-edge violations are treated as hard constraints; direct\-edge plausibility is judged leniently \(only clearly implausible edges are flagged as violations\); CI and type consistency are soft constraints\. For grafted graphs, the prompt is rendered with a stage\-specific instruction block \(auxiliary\-stage anchor immutability or final merged\-graph leniency\) that the same template substitutes in via\{stage\_specific\_instruction\_block\}\. The block below is lightly condensed: we omit the conditional interpretation\-note inserts, the output\-format instructions, and the JSON/XML schemas\.

Youareastrictcausalconsistencyauditor\.

\{stage\_specific\_instruction\_block\}Youwillbegiven:

1\)DIRECTedges\(u\-\>v\)thatmustbeplausibleasdirectcausaleffects\.

2\)NON\-EDGEpairs\(u,v\)whereNOplausibledirectcausallinkmayexist\.

3\)Optionally,conditionalindependence\(CI\)relationsfromthegraph\.

4\)Aproposedvariablemappingfromnodeidstoreal\-worldmeanings\.

5\)Optionally,expectednodetypehintsfromgeneration\.

Yourjob:

1\.ForeachNON\-EDGEpair\(U,V\),decideifANYDIRECTCAUSALLINK

existsbetweenthemappedvariables\.Workingdefinition:

’ThereexistsaninterventiononUthatchangesVwhileholding

fixedtheothervariablesinthegraph\(especiallytheother

nodesthatcouldmediatetheeffect\)\.’

YouMUSTincludeanentryin"non\_edge\_attestations"forEVERY

non\-edgepair\(no\_direct\_linktrue/falseplusabrief

justification\);ifadirectlinkDOESexist,alsorecorda

VIOLATION\.

2\.ForeachDIRECTedge\(U\-\>V\),decideifaplausibledirectcausal

linkexists\.BeGENEROUS:anyreasonablemechanism\(weak,partial,

orcontext\-dependent\)issufficient\.OnlyflagaVIOLATIONif

clearlyimplausible\.

3\.ForeachCIstatement,judgeifplausiblegivenvariablemeanings\.

\(Softconstraint\.\)WhenflaggingaCIviolation,namethevariables,

explainwhythechosenmeaningsmaketheindependenceimplausible

\(e\.g\.,uncoveredalternativepathways\),andHINTatareframing

thatwouldrestoreit\.

4\.Typeconsistency:belenient\(treatclosefamiliesascompatible,

skipwhenmissing\);onlyflagclearcontradictions\.\(Soft\.\)

INTERPRETATIONNOTES:

WemostlycareabouttheNON\-EDGEviolations\.Usethedefinitionof

theDIRECTCAUSALLINKaboveverystrictly\.Forexample,averbal

aggregationofadetailedeffectX\-\>M\-\>YisNOTaseparate

directeffectX\-\>Y\!Forthat,werequireaSEPARATEdirect

pathway\!

FEEDBACKQUALITYREQUIREMENT:

Foreveryviolation,the"explanation"mustbeactionable\.Endeach

withashort’HINT:’suggestinghowitmightberesolved\.Donot

suggestchangingthegraphstructureorfixednodes;iftheviolation

isbetweentwofixednodes,saysoexplicitlysothecallercanreject

thesample\.

##### Story generation system prompt\.

Instructs the story generation LLM to produce a short narrative from the renamed graph and proposed domain\. The runtime template variable\{format\_system\}expands to a reminder of the graph serialization format used in the user prompt\.

Youareanexpertinnarratingacohesivestorybasedonacausalgraph

withadomainanddomain\-relevantvariables\.

Thegraphwillbeprovidedinthefollowingstructuredformat:

\{format\_system\}

YourtaskistowriteaSHORT,CONCRETESTORY\(2\-4paragraphs\)that:

\-issituatedintheprovidedPROPOSEDDOMAIN

\-usesONLYtheprovidedSTORYNAMESfromtheVARIABLEMAPPING

\(NEVERuserawnodeIDslikeV0,V1,etc\.\)

\-mentionsEVERYmappedvariableexplicitlyinthestory,

includinglatent/backgroundfactorswhenpresent

\-makesEVERYdirectcausalrelationshipinthegraphexplicitin

thestoryitself

\-doesNOTintroduceunsupportedextradirectcausalrelationships

\-avoidsgraphjargon\(no"edges","nodes","colliders","DAG",etc\.\)

\-isscientificallyplausibleandengaging

\-maybeslightlydenserthanusualifneededtocoverthefull

graphexactly

##### Story generation user prompt template\.

YouwillbegivenaPROPOSEDdomain,aVARIABLEMAPPING\(ids\-\>story

names\)inJSON,andaCAUSALGRAPHintheformatprescribedbythe

systemprompt\.

Yourtaskistowriteashort,concreteSTORY\(2\-4paragraphs\)using

ONLYtheprovidedstorynamesfromtheVARIABLEMAPPING\(norawids\)\.

Thestoryneedstoreflectthecausalrelationshipsinthegraphand

mentioneverymappedvariableexplicitlywithinthenarrative,BUT

WITHOUTusinganygraphjargonrevealingitsstructure\.

Everydirectedgeinthegraphmustbestatedorclearlyimpliedin

theSTORYitself\.Donotrelyontheappendix/justificationstocover

missingedges\.Donotintroduceunsupportedextradirectcauses\.

Retaintheunitsandvariabletypes\(e\.g\.,categorical,continuous\)

givenbythemapping\.AdditionallyreturnCAUSALJUSTIFICATIONS

explaininghowthestoryreflectsthecausalstructure\.

ReturnONLYaJSONobject:\{"story":"\.\.\.",

"causal\_justifications":"\.\.\."\}

Proposeddomain:\{proposed\_domain\}

Variablemapping\(JSON\):

\{variable\_mapping\_json\}

Graphstructure:\{graph\_structure\}

\{extra\_instruction\}

##### Story audit system prompt\.

The story auditor verifies that the generated narrative faithfully represents the causal graph\. Hard failures populateviolations; lower\-priority issues populatewarnings; the auditor must additionally fill anode\_attestationsentry for every required variable and anedge\_attestationsentry for every direct edge\. The block below is condensed: we omit the input list, the interpretation notes, the per\-item sub\-bullets, and the JSON/XML schemas\.

Youareastrictstory\-to\-DAGauditor\.

HARDREQUIREMENTS:

1\.EveryrequiredvariablemustbeexplicitlymentionedintheSTORY

usingitsprovidedstoryname\.Latent/unobservedvariablesmaybe

narratedashidden/backgroundfactors,buttheystillmustbe

mentioned\.Emitoneentryin"node\_attestations"forEVERY

requiredvariable\.

2\.EveryDIRECTedge\(U\-\>V\)mustbeclearlysupportedbytheSTORY

itself\.Apath\-levelclaimdoesNOTautomaticallycovereveryedge

onthepath\.Emitoneentryin"edge\_attestations"forEVERY

directedge\(supported/contradicted,withajustification\)\.

3\.IftheSTORYcontradictsthestateddirectionofanedge,mark

thatasahardviolation\.

SOFTCHECKS:

4\.WarniftheSTORYintroducesextradirectcausalclaimsbetween

NON\-EDGEpairs\.

5\.WarniftheSTORYdriftsfromtheproposeddomain\.

6\.WarniftheSTORYusesgraphjargon\.

7\.WarniftheSTORYisimplausible,globallyincoherent,ormixes

incompatibledomainsinawaythatmakesthescenariohardto

believeasarealsetting\.

DECISIONRULE:

\-Set"pass"totrueonlyifallhardrequirementsaresatisfied\.

\-PutONLYhardfailuresin"violations";putsoftissuesin

"warnings"\.

\-Eachissuemustbeactionable,endingina"HINT:"clausethat

suggestshowtorewritethestorytoremoveit\.

#### A\.16\.2Benchmark task prompt templates

Every benchmark task is delivered to the agent as a fixed*system message*followed by a*user message*assembled bybuild\_single\_task\_prompt\. The user message wraps a family\-specific body in shared scaffolding that is identical across tasks: a data\-location header, the scenario\#\#\# Background\(the story\), a\#\#\# Datasection, a shared\#\#\# Measurement Noteon data\-backed tasks, the family\-specific\#\#\# Questionand\#\#\# Output Format, and a footer giving the answer\-file path and submit instruction\. To avoid repetition, we reproduce the shared blocks once below and then list only the per\-family\#\#\# Question/\#\#\# Output Formatbodies\. Template variables \(\{story\},\{treatment\_name\},\{outcome\_name\}, etc\.\) are filled from the scene at generation time; the formal JSON response schema is not sent to the model \(the agent writes an answer file viabash\), so each body shows only the inline JSON example\. The full\(family, output variant\)catalog is enumerated in Sec\.[3\.4](https://arxiv.org/html/2607.08093#S3.SS4)and App\.[A\.9](https://arxiv.org/html/2607.08093#A1.SS9)\.

##### Shared system message\.

Identical for every task; lists the available Python stack and the one\-bash\-call interaction contract\.

Youareanexpertdatascientistsolvingcausalreasoningtasks\.

YouhavePython3\.11with:pandas3\.0\.2,numpy2\.4\.4,scipy1\.17\.1,

scikit\-learn1\.8\.0,statsmodels0\.14\.6,pyarrow24\.0\.0,

matplotlib3\.10\.9,seaborn0\.13\.2,networkx3\.6\.1,xgboost3\.2\.0\.

Donotassumeunlistedpackagesareinstalled\.

<rules\>

1\.EVERYresponsemustincludeEXACTLYONEbashtoolcall\.Noexceptions\.

2\.IncludeyourreasoningastextBEFOREmakingthetoolcall\.

3\.Formulti\-linePython,useaheredoc:

python3<<’PYEOF’

importpandasaspd

df=pd\.read\_parquet\("scenes/<scene\_id\>/data\.parquet"\)

print\(df\.describe\(\)\)

PYEOF

4\.Towritefiles,usecatwithheredoc:

cat\>/workspace/answers/my\_answer\.json<<’EOF’

\{"key":"value"\}

EOF

5\.Thegraderreadsonly/workspace/answers\.Donotwriteanswersunder/home\.

6\.Whendone,verifyyouranswerfileexistsattherequestedpath,thensubmit:echoDONE

</rules\>

##### Shared user\-message skeleton\.

The data\-location bullets list only the files released for the task: thedata\.parquet/calibration\.parquet/schema\.jsonlines appear only for data\-backed tasks, and atest\_features\.parquetline is added for prediction\. The story\-only Rung\-2/3 identification and graph\-recovery tasks omit the\#\#\# Datasection and the Measurement Note\.

\#Task

\*\*Datalocation:\*\*‘scenes/\{scene\_id\}/‘

\-Story/context:‘scenes/\{scene\_id\}/story\.md‘

\-Observationaldata:‘scenes/\{scene\_id\}/data\.parquet‘\(data\-backedonly\)

\-Calibrationsubset\(ifpresent\):‘scenes/\{scene\_id\}/calibration\.parquet‘\(data\-backedonly\)

\-Columnmetadata:‘scenes/\{scene\_id\}/schema\.json‘\(data\-backedonly\)

\-Testfeatures\(nooutcomecolumn\):‘scenes/\{scene\_id\}/test\_features\.parquet‘\(predictiononly\)

\#\#\#Background

\{story\}

\#\#\#Data

Youareprovidedwithobservationaldatain‘data\.parquet‘\.\(data\-backedonly\)

\{\#\#\#MeasurementNote\-shownbelow;data\-backedtasksonly\}

\{family\-specific\#\#\#Questionand\#\#\#OutputFormat\-seeper\-familyblocks\}

\*\*Answerfile:\*\*‘/workspace/answers/\{answer\_file\}‘

WriteaJSONfilematchingtheschemadescribedabove\.\(CSVwithcolumns‘prediction‘\[,‘lower‘,‘upper‘\]forprediction\)

Thegraderreadsonlyfilesunder‘/workspace/answers/‘;donotwriteanswersunder‘/home‘\.

Whendone,verifyyouranswerfileexistsattheexactpathabove,thensubmit:

‘‘‘bash

echoDONE

‘‘‘

##### Shared Measurement Note \(data\-backed tasks\)\.

Inserted before the\#\#\# Questionon every task that ships a\.parquetfile\. The first paragraph \(released data scale\) is always present; the remaining paragraphs appear only when the released observation variant contains noisy measurement columns\.

\#\#\#MeasurementNote

Thestorynamesconceptualvariablesinthecausalgraph\.Numericvaluesinthereleased‘\.parquet‘files,including‘data\.parquet‘,andanyquestion\-specifiedvaluesareonthebenchmark’sreleaseddatascaleforthoseconceptualvariables\.Thisscalemaydifferfromtheunits,ranges,orexamplesimpliedbythestory;valuesmaybecentered,standardized,orotherwisetransformed\.Foransweringdata\-backedquestions,usethereleaseddatascale\.

Inthisreleaseddataset,someconceptualvariablesarenotdirectlymeasuredin‘data\.parquet‘\.Eachsuchconceptualvariableisreplacedbyabundleofnoisymeasurementcolumns\.Eachbundlehasexactlyoneconceptualparentandshouldnotbetreatedasasetofseparatecausalvariables\.Examplesinthisscene:Covariate1:Covariate1\_meas\_a,Covariate1\_meas\_b;Mediator1:Mediator1\_meas\_a\.

Asmaller‘calibration\.parquet‘containsthemeasurementcolumnstogetherwithgold\-standardmeasurementsofthecorrespondingconceptualvariablesforasubsetofrows\.Usethesharednamestem,‘schema\.json‘,andthecalibrationrowstodeterminewhichmeasurementcolumnsbelongtoeachconceptualvariable\.

Fordeterminingcausalidentifiability\(whennecessary\),reasononlyovertheconceptualvariablesdescribedinthestory\.Samegoesforanycausalquantity\.Theobservationlayeraffectssolelythestatisticalestimationdifficultyofthepopulationquantityfromprovideddata\.

Whenestimatingaquantityfromdata,ifanamedconceptualvariableismeasuredbyabundle,usethatbundleandthecalibrationdatatoestimateorreconstructtheconceptualvariablebeforeestimatingtherequestedstatisticalorcausalquantity\.

Rung 1 \(Associational\)\.The bodies below give the\#\#\# Questionand\#\#\# Output Formatthat replace the\{family\-specific…\}placeholder above\.

##### R1: Prediction \(point\_predictor;prediction\_intervaladds a central 90% interval\)\.

Prediction is the one Kaggle\-style family: it writes a CSV of held\-out predictions rather than a JSON answer, so its body replaces\#\#\# Question/\#\#\# Output Formatwith an\#\#\# Objective\. Theprediction\_intervalvariant appends the\#\#\# Uncertainty Quantificationblock\.

\#\#\#Data

Youareprovidedwithtrainingdatain‘data\.parquet‘containingthefollowingcolumns:

"Treatment","Outcome","Covariate1","Covariate2"

\#\#\#Objective

Buildapredictivemodelfortheconceptualoutcome\*\*Outcome\*\*usingthevariablesavailableinthereleaseddata\.

Iftheconceptualoutcomeismeasuredbyabundleinthereleaseddata,usethecalibrationrowstolearnhowthosemeasurementsrelatetotheconceptualoutcome\.

\#\#\#UncertaintyQuantification\(PredictionIntervals\)\[prediction\_intervalvariantonly\]

Alsoproducea\*\*central90%predictioninterval\*\*foreachprediction\.

\#\#\#Notes

\-Evaluationusesheld\-outtestdata\.

##### R1: Association — sign\.

Theeffect\_size\_pointvariant instead asks for the numeric Pearson strength \(\{"value": 0\.45, "method": "pearson"\}\); it is explicitly framed as “not a causal\-effect estimate\.”

\#\#\#Question

Estimatethesignoftheobservationalassociationbetweentheconceptualvariables\*\*Treatment\*\*and\*\*Outcome\*\*\.

Report‘"\+"‘ifhighervaluesof\*\*Treatment\*\*areassociatedwithhighervaluesof\*\*Outcome\*\*,‘"\-"‘ifhighervaluesof\*\*Treatment\*\*areassociatedwithlowervaluesof\*\*Outcome\*\*,and‘"unknown"‘onlyiftheempiricalassociationistoouncleartodetermine\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{

"sign":"\+",//or"\-"or"unknown"

"stat":\{"method":"correlation","value":0\.45\}//optional,encouraged

\}

‘‘‘

##### R1: Conditional association — sign before/after conditioning\.

Sameassociationfamily; thedelta\_point/delta\_sign\_onlyvariants ask for the magnitude/sign of the*change*in association after conditioning, andargmax\_changeasks which of several offered conditioning variables changes it most\.

\#\#\#Question

Considertheassociationbetween\*\*Treatment\*\*and\*\*Outcome\*\*\.

1\.Whatisthesignofthemarginal\(unconditional\)association?

2\.Whatisthesignoftheassociationafterconditioningontheconceptualvariable\*\*Covariate1\*\*?

Conditioningheremeansstatisticalconditioning,notintervention\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{

"sign\_before":"\+",//marginalassociationsign:"\+","\-",or"unknown"

"sign\_after":"\+",//associationafterconditioning:"\+","\-",or"unknown"

"conditioning\_var":"Covariate1",

"explanation":"Briefexplanationofwhytheassociationchangesorstaysthesame"

\}

‘‘‘

##### R1: Collider phenomenon \(explaining away\)\.

Theinduced\_association\_sign\_only/induced\_association\_strength\_pointvariants instead ask for the sign / numeric strength of the association induced by conditioning on the collider\.

\#\#\#Question

Considertheconceptualvariables\*\*Treatment\*\*,\*\*Outcome\*\*,and\*\*Collider1\*\*\.

Afterconditioningontheconceptualvariable\*\*Collider1\*\*,isthereanonzeroobservationalassociationbetween\*\*Treatment\*\*and\*\*Outcome\*\*?

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{

"association\_present":true,//whetheranonzeroassociationispresentafterconditioningonCollider1

"explanation":"\.\.\."

\}

‘‘‘

Rung 2 \(Interventional\)\.

##### R2: Causal sketch — directed edges \(skeleton\_edgesdrops direction\)\.

A story\-only task \(no\#\#\# Data/Measurement Note\); the prompt states the exact variable count\.

\#\#\#Question

Basedonlyonthescenariodescription,identifytheconceptualcausalvariablesitdescribesandthedirectcausalrelationshipsamongthem\.

Theconceptualcausalgraphcontainsexactly5causalvariables\.

Important:

\-Includeonlydirectcausaleffects,notindirecteffectsthroughintermediatevariables\.

\-AcausesBmeansthataninterventiononAwoulddirectlychangeBwhileholdingfixedothervariablesthatcouldmediatetheeffect\.

\-Donotincludeassociationscausedonlybycommoncauses,selection,orconditioning\.

\-Includeeverycausalvariablethestorydescribes,includinganythatisdescribedonlyasahidden,background,orunmeasuredfactor\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject\(usetheexactvariablenamesastheyappearinthestory\):

‘‘‘json

\{"edges":\[\{"from":"Variable1","to":"Variable2"\},\{"from":"Variable1","to":"Variable3"\}\]\}

‘‘‘

Theskeleton\_edgesvariantreturnsundirectedpairsinstead:\{"skeleton\_edges":\[\{"a":"Variable1","b":"Variable2"\}\]\}\.

##### R2: Identification — method label\.

A story\-only task: the graph drives a population\-ATE identifiability judgment and no estimator is run\. Sibling variants areidentifiable\_boolean\(\{"identifiable": true\}\) and the adjustment\-set queriesone\_valid\_adjustment\_set\(\{"adjust": \[…\] \| "no\_backdoor" \| "non\_id"\}\),minimal\_adjustment\_set\_size\(\{"k": …\| "no\_backdoor" \| "non\_id"\}\),n\_valid\_adjustment\_sets\(\{"n": …\| 0 \| "no\_backdoor" \| "non\_id"\}\), andall\_minimal\_adjustment\_sets\(\{"adjustment\_sets": \[…\] \| "no\_backdoor" \| "non\_id"\}\)\. For the all\-minimal variant,\[\[\]\]denotes the empty adjustment set as the only minimal valid set; for the one\-valid variant,\[\]denotes that no adjustment is needed\. The size, count, and all\-minimal variants list the offered covariates under an\#\#\# Available Conceptual Covariatesheader, while the one\-valid variant names them in the question prose\.

\#\#\#ConceptualObservedVariables

Thefollowingconceptualvariablesareobservedforcausal\-identificationpurposes,innoparticularorder:"Treatment","Outcome","Covariate1","Covariate2","Mediator1"

\#\#\#Question

IsthepopulationAverageTreatmentEffect\(ATE\)of\*\*Treatment\*\*on\*\*Outcome\*\*identifiablefromtheobservationaldistributionovertheconceptualobservedvariables?Returnthefirstapplicableidentificationlabel\.

Targetestimand:

ATE=E\[Y\|do\(Treatment=x1\)\]\-E\[Y\|do\(Treatment=x0\)\]

Treatthestoryasspecifyingthequalitativeconceptualcausalgraph\.Identifiabilityisapropertyofthatgraphandtheobservationaldistributionovertheobservedconceptualvariables;donotassumeparametricforms,effecthomogeneity,oraccesstothestructuralmodel\.

Returnthefirstapplicablelabel:

1\."trivial\_zero":TreatmenthasnodirectedcausalpathtoOutcome,sothepopulationATEisidentifiableaszero\.

2\."backdoor":Otherwise,avalidbackdooradjustmentsetamongtheobservedconceptualvariablesidentifiesthepopulationATE\.

3\."frontdoor":Otherwise,avalidfront\-doorformulausingobservedconceptualvariablesidentifiesthepopulationATE\.

4\."other\_id":Otherwise,thepopulationATEisidentifiablebyanothervaliddo\-calculus/IDargument\.

5\."none":ThepopulationATEisnotidentifiable\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{"method":"backdoor"\}

‘‘‘

##### R2: Effect estimate — population ATE \(point\)\.

Data\-backed \(Measurement Note shown\)\. Variants share this stem and swap the output contract:ate\_sign\_only\(\{"sign": "\+"/"\-"/"0"/"unknown"\}\),ate\_vs\_assoc\_sign\_match\(\{"matches": true\|false\|null\}\), andate\_uq\_95, which appends “also provide a central 95% confidence interval” and returns\{"ate": \.\., "ci\_lower": \.\., "ci\_upper": \.\.\}\. Non\-identifiable scenes \(e\.g\. bare\-IV\) require thenull/unknownabstention value\.

\#\#\#Data

Youareprovidedwithobservationaldatain‘data\.parquet‘containingthefollowingcolumns:

"Treatment","Outcome","Covariate1","Covariate2"

\#\#\#Question

EstimatethepopulationAverageTreatmentEffect\(ATE\)oftheconceptualtreatment\*\*Treatment\*\*ontheconceptualoutcome\*\*Outcome\*\*\.

Usetreatmentlevelsx0=0\.0andx1=1\.0\.

ATE=E\[Y\|do\(Treatment=1\.0\)\]\-E\[Y\|do\(Treatment=0\.0\)\]

\#\#\#OutputFormat

ProvideyouranswerasJSON:

‘‘‘json

\{"ate":1\.23\}

‘‘‘

IfthepopulationATEisnotidentifiablefromtheobservationaldistributionovertheconceptualvariables,return:

‘‘‘json

\{"ate":null\}

‘‘‘

##### R2: Bias diagnostic — collider bias\.

Theforbidden\_controls\_listvariant instead lists the offered conceptual covariates and asks which must*not*be conditioned on, returning\{"forbidden": \[…\] \| \[\] \| "no\_backdoor" \| "non\_id"\}\(set\-F1F\_\{1\}graded, with the non\-identifiable branch routed to abstention\)\. The empty list means that adjustment is available and no listed variable must be excluded\.

\#\#\#Question

Aresearcherwantstoestimatethepopulationcausaleffectof\*\*Treatment\*\*on\*\*Outcome\*\*\.

Theyproposetoconditionontheconceptualvariable\*\*Collider1\*\*intheiranalysis\.

Wouldconditioningon\*\*Collider1\*\*introducecolliderbiasorotherwiseopenanoncausalpathbetweentreatmentandoutcome?

Answerusingthestory\-impliedconceptualcausalgraph\.Thereleaseddatamayhelpinspectassociations,butthebiasjudgmentiscausalandgraph\-based\.Measurementcolumnsaremeasurementsofconceptualvariables,notseparatecontrols\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{"bias\_present":true,"explanation":"\.\.\."\}

‘‘‘

Rung 3 \(Counterfactual\)\.

##### R3: Counterfactual identification — ETT \(identifiable\_boolean\)\.

A story\-only judgment of whether the target counterfactual estimand is identifiable; instantiated for ETT, NDE, and NIE, swapping in the corresponding estimand name, formula, and mediator list\.

\#\#\#Question

Treatthestoryasspecifyingthequalitativeconceptualcausalgraph\.

Targetestimand:\*\*EffectoftheTreatmentontheTreated\(ETT\)\*\*

Formalnotation:

ETT\(x1,x0\)=E\[Y\_\{x1\}\-Y\_\{x0\}\|X=x1\],withx0=0,x1=1

Istheeffectoftreatmentonthetreatedfor\*\*Treatment\*\*on\*\*Outcome\*\*identifiablefromtheobservationaldistributionunderthestory\-impliedconceptualgraph?

Identifiabilityofthiscounterfactualestimandisapropertyofthegraphandthepopulationobservationaldistributionalone;donotassumeparametricforms,effecthomogeneity,monotonicity,oraccesstothestructuralmodel\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{"identifiable":true,"explanation":"\.\.\."\}

‘‘‘

##### R3: Counterfactual effect — ETT \(point\)\.

Data\-backed\. Thesign\_onlyvariant returns\{"sign": "\+"/"\-"/"0"/"unknown"\}, andeffect\_uq\_95appends “estimate the population ETT and provide a central 95% confidence interval”, returning\{"value": \.\., "ci\_lower": \.\., "ci\_upper": \.\.\}\. A non\-identifiable ETT requires thevalue=null\(orsign="unknown"\) abstention value\.

\#\#\#Question

EstimatethepopulationEffectoftheTreatmentontheTreated\(ETT\)oftheconceptualtreatment\*\*Treatment\*\*ontheconceptualoutcome\*\*Outcome\*\*\.

Usetreatmentlevels\*\*x0=0\*\*and\*\*x1=1\*\*\.

ETT\(x1,x0\)=E\[Y\_\{x1\}\-Y\_\{x0\}\|X=x1\]

IftheETTisnotidentifiablefromtheobservationaldistributionovertheconceptualvariables,return‘\{"value":null\}‘\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{"value":1\.23,"explanation":"Brieflystatetheidentifiabilityjudgmentandestimationapproach\."\}

‘‘‘

##### R3: Mediation effect — NDE \(point\)\.

Shown for the NDE point estimate; the NIE prompt is analogous withNIE​\(x1,x0\)=E​\[Yx0,Mx1−Yx0,Mx0\]\\mathrm\{NIE\}\(x\_\{1\},x\_\{0\}\)=E\[Y\_\{x\_\{0\},M\_\{x\_\{1\}\}\}\-Y\_\{x\_\{0\},M\_\{x\_\{0\}\}\}\]\. For each estimand,sign\_onlyreturns\{"sign": "\+"/"\-"/"0"/"unknown"\}andeffect\_uq\_95appends a central 95% confidence interval \(\{"value": \.\., "ci\_lower": \.\., "ci\_upper": \.\.\}\); a non\-identifiable effect requires thevalue=null/sign="unknown"abstention\.

\#\#\#Question

EstimatethepopulationNaturalDirectEffect\(NDE\)oftheconceptualtreatment\*\*Treatment\*\*ontheconceptualoutcome\*\*Outcome\*\*,withmediator\(s\):\*\*Mediator1\*\*\.

Usetreatmentlevels\*\*x0=0\*\*and\*\*x1=1\*\*\.

NDE\(x1,x0\)=E\[Y\_\{x1,M\_\{x0\}\}\-Y\_\{x0,M\_\{x0\}\}\]

IftheNDEisnotidentifiablefromtheobservationaldistributionovertheconceptualvariables,return‘\{"value":null\}‘\.

\#\#\#OutputFormat

ProvideyouranswerasaJSONobject:

‘‘‘json

\{"value":0\.85,"explanation":"Brieflystatetheidentifiabilityjudgmentandestimationapproach\."\}

‘‘‘

Thedirect\_vs\_indirect\_dominancevariant instead asks which component \(NDE or NIE\) has larger absolute magnitude, returning\{"dominant": "direct"/"indirect"/"tie"\}\(ornullwhen either the NDE or NIE is non\-identifiable\)\.