From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers

# From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers
Source: [https://arxiv.org/html/2608.23660](https://arxiv.org/html/2608.23660)
Elnur Adl ZarabiAffiliation:Texas A&M University\-Corpus Christi, USASuranjana TrivedyAffiliation:BITS Pilani Goa, IndiaZhiqian ChenAffiliation:Mississippi State University, USALei ZhangAffiliation:Northern Illinois University, USAKaiqun FuAffiliation:Texas Christian University, USATaoran JiAffiliation:Texas A&M University\-Corpus Christi, USA

###### Abstract

Large language models \(LLMs\) are increasingly used to provide prior causal knowledge for structural causal discovery, yet whether their direct\-edge judgments and associated confidence can be trusted remains unclear\. We systematically evaluate 12 instruction\-tuned open\-weight models across six benchmark causal graphs, five prompting strategies, and four confidence sources: verbalized, logit\-based, cross\-prompt agreement, and cross\-model agreement\. Our evaluation yields three key findings\. \(i\) LLM\-based causal judgments are strongly recall\-dominant\. Models tend to predict overly dense graphs with many false\-positive edges, while prompting primarily shifts the precision–recall trade\-off rather than consistently resolving overprediction\. Improvements with model scale also diminish on the largest graphs and do not eliminate miscalibration\. \(ii\) LLMs often capture causal relatedness without reliably identifying directness or orientation\. Relative to the published reference graphs, models incorrectly classify 40\.0% of indirect and 36\.0% of reversed non\-edges as direct causal edges, compared with 28\.2% of other non\-edges\. Moreover, 80\.8% and 84\.6% of these false positives receive verbalized confidence of at least 80%, revealing substantial overconfidence in structurally incorrect predictions\. \(iii\) Conventional confidence estimates are unreliable, whereas agreement provides a more promising signal\. Logit\-based confidence frequently collapses near 1\.0 regardless of correctness, while cross\-prompt and cross\-model agreement achieve better mean calibration and discrimination, although their advantages are not statistically significant after Holm correction\. A benchmark\-familiarity audit additionally identifies potential familiarity in five model–dataset pairs, all involving AsiaM\. Overall, our results suggest that LLMs are better viewed as sources of externally validated soft causal priors than as direct evidence of causal structure\. Replication materials are available on[GitHub](https://github.com/aamitssharma07/calibrated-llm-causal-discovery)\.

## 1Introduction

Causal discovery \(CD\) aims to infer directed causal relationships among variables, typically represented as a causal graph, to support explanation, intervention analysis, and scientific understanding\([30](https://arxiv.org/html/2608.23660#bib.bib1)\)\. For example, consider a simplified medical graphViral Exposure→Infection→Positive Test\\textit\{Viral Exposure\}\\rightarrow\\textit\{Infection\}\\rightarrow\\textit\{Positive Test\}\. Within this graph, adjacent variables are connected by direct edges, whereasViral ExposureinfluencesPositive Testonly indirectly throughInfection\. Traditional approaches, including constraint\-based methods such as PC and FCI\([32](https://arxiv.org/html/2608.23660#bib.bib4)\), score\-based methods such as GES\([10](https://arxiv.org/html/2608.23660#bib.bib5)\)and NOTEARS\([45](https://arxiv.org/html/2608.23660#bib.bib6)\), and neural formulations such as GraN\-DAG\([20](https://arxiv.org/html/2608.23660#bib.bib7)\), operate on observational data under assumptions about the data\-generating process\. However, observational data may be limited or insufficient to distinguish Markov\-equivalent structures\([13](https://arxiv.org/html/2608.23660#bib.bib3)\), motivating the use of domain knowledge to guide or orient candidate graphs\([36](https://arxiv.org/html/2608.23660#bib.bib8)\)\. Such knowledge is often costly to elicit or unavailable in novel domains\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/intro_causal_reliability_problem.png)Figure 1:Illustration of unreliable pairwise direct\-edge judgments\. Solid arrows form the reference chain; dashed arrows show an indirect relation predicted as direct and a reversed relation\. Confidence values are illustrative\.Large language models \(LLMs\) offer a complementary source of causal knowledge for CD, with recent work exploring direct causal reasoning\([19](https://arxiv.org/html/2608.23660#bib.bib32)\), prior\-guided structure learning\([36](https://arxiv.org/html/2608.23660#bib.bib8);[6](https://arxiv.org/html/2608.23660#bib.bib9);[18](https://arxiv.org/html/2608.23660#bib.bib10)\), and graph orientation and refinement\([24](https://arxiv.org/html/2608.23660#bib.bib2);[5](https://arxiv.org/html/2608.23660#bib.bib11)\)\. Although LLMs demonstrate non\-trivial performance on pairwise causal tasks using textual metadata\([19](https://arxiv.org/html/2608.23660#bib.bib32)\), their predictions may depend on the frequency of causal relations in pre\-training corpora and vary under contextual changes\([12](https://arxiv.org/html/2608.23660#bib.bib45)\)\. Prompt style and response agreement can also affect confidence calibration\([40](https://arxiv.org/html/2608.23660#bib.bib44)\)\. Existing work shows that pairwise LLM\-based edge classification can yield poor graph recovery\([4](https://arxiv.org/html/2608.23660#bib.bib26)\), but does not systematically identify the structural sources of these errors or whether confidence reflects their correctness\. Because such judgments are used as structural constraints, optimization priors, and causal\-order information in CD pipelines\([6](https://arxiv.org/html/2608.23660#bib.bib9);[18](https://arxiv.org/html/2608.23660#bib.bib10);[36](https://arxiv.org/html/2608.23660#bib.bib8)\), their reliability remains important even when pairwise classification is insufficient as a standalone CD method\.

This gap reflects a distinction between causal plausibility and causal reliability\. For queried variablesAAandBB, a reliable direct\-edge judgment requires distinguishingA→BA\\rightarrow Bfrom reverse directionality, indirect influence, and no direct edge\. A model may recognize causal relatedness while misidentifying directness or orientation, producing many spurious edges in sparse graphs\. Confidence reliability is therefore as important as classification performance when LLM judgments inform graph learning\. Figure[1](https://arxiv.org/html/2608.23660#S1.F1)illustrates this reliability gap\.

Accordingly, our aim is to characterize the reliability of edge judgments on which LLM\-assisted CD pipelines may depend, rather than to propose a new CD method\. Our contributions are as follows:

- •We formulate pairwise direct causal\-edge classification as a calibrated evaluation task, jointly assessing edge prediction quality, graph reconstruction, and confidence reliability under structural ambiguity and severe class imbalance\.
- •We develop an evaluation framework spanning six benchmark causal graphs, five prompting strategies, 12 instruction\-tuned models at two scales, and four confidence sources: verbalized, logit\-based, cross\-prompt agreement, and cross\-model agreement\. We also evaluate potential benchmark familiarity across all 72 model–dataset combinations\. We will release the code and evaluation artifacts to support reproducibility\.
- •We show that LLM edge judgments exhibit recall\-dominant behavior and prompt sensitivity\. Larger models achieve higher mean F1 on five of six datasets, but provide only marginal gains on the largest graphs and do not resolve miscalibration\. Relative to the published reference graphs, false\-positive rates are higher for indirect \(40\.0%\) and reversed \(36\.0%\) non\-edges than for other non\-edges \(28\.2%\); over 80% of false positives on indirect and reversed relations receive verbalized confidence of at least 80%\. A three\-way task\-formulation ablation further shows that these errors are not solely induced by evaluating the two directions through separate binary queries\.
- •We find that agreement\-based confidence achieves better mean calibration and discrimination than verbalized and logit\-based confidence, although the differences are not statistically significant after Holm correction and remain model\- and prompt\-dependent\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/calibration_pipeline_3.png)Figure 2:Evaluation pipeline for pairwise direct\-edge classification, graph reconstruction, confidence calibration, and structural error analysis\.
## 2Evaluation Design

The framework in Fig\.[2](https://arxiv.org/html/2608.23660#S1.F2)treats LLMs as calibrated pairwise direct causal\-edge classifiers\. Across two model\-scale groups, five prompting strategies, and four confidence sources, we ask whether models avoid edge overprediction, distinguish direct edges from reversed, indirect, and other non\-edges, and assign confidence that reflects correctness\. Pairwise predictions are compared with published reference graphs and aggregated into reconstructed graphs to assess edge classification and graph recovery\. Structural error analysis, confidence thresholding, a three\-way label\-space ablation, and a benchmark\-familiarity audit further test these reliability concerns\.

### 2\.1Prompt Styles

Prompt formulation affects LLM performance\([39](https://arxiv.org/html/2608.23660#bib.bib36);[9](https://arxiv.org/html/2608.23660#bib.bib37);[31](https://arxiv.org/html/2608.23660#bib.bib38)\)\. We adapt pairwise causal prompting from prior work on causal reasoning and LLM\-guided structure discovery\([19](https://arxiv.org/html/2608.23660#bib.bib32);[36](https://arxiv.org/html/2608.23660#bib.bib8)\)to direct\-edge classification with explicit direct\-causation instructions and verbal confidence elicitation\([22](https://arxiv.org/html/2608.23660#bib.bib34)\)\.

We evaluate five prompt styles that provide progressively richer guidance\.Name\-onlysupplies the dataset context and variable names, testing whether the model can infer a direct edge from names alone\.Metadataadditionally provides variable definitions to clarify their domain\-specific meanings\.Chain\-of\-Thought \(CoT\)requests a brief rationale before the prediction\([38](https://arxiv.org/html/2608.23660#bib.bib33)\)\.Few\-shot \(FS\)provides five labeled direct\-edge examples that demonstrate the required task and output format\([8](https://arxiv.org/html/2608.23660#bib.bib35)\)\.Few\-shot \+ CoT \(FS\+CoT\)combines these demonstrations with brief reasoning\. All styles ask whetherAAdirectly causesBBand require a binary “Yes”/“No” judgment with a self\-reported confidence score from 0–100\. Full templates appear in Appendix[A\.1](https://arxiv.org/html/2608.23660#A1.SS1)\.

### 2\.2Direct Edge Classification and Graph Reconstruction

The primary task is binary direct\-edge classification\. Given an ordered pair\(A,B\)\(A,B\)and dataset context, the model predicts whetherAAdirectly causesBB, withYesas the positive class andNoas the negative class\. Responses follow a standardized answer–confidence format and are parsed deterministically, prioritizing labeled fields and then answer\-first forms while ignoring prompt echoes\. Missing fields trigger one retry; fewer than 0\.2% of responses remain without a valid label and are excluded from classification\.

For each model–prompt–dataset setting, all positively predicted ordered pairs are combined to form a reconstructed directed graph\. We compare this reconstructed graph with the published reference graph using normalized Structural Hamming Distance \(nSHD\), defined in Section[3\.3](https://arxiv.org/html/2608.23660#S3.SS3)\. Graph reconstruction is induced directly from pairwise predictions and does not enforce global structural constraints such as acyclicity\.

### 2\.3Confidence Estimation Task

This task assesses whether confidence reflects correctness in binary edge classification\. Letxxdenote an input query,𝒴=\{Yes,No\}\\mathcal\{Y\}=\\\{\\textsc\{Yes\},\\textsc\{No\}\\\}the label set, andy^∈𝒴\\hat\{y\}\\in\\mathcal\{Y\}the predicted label\. We consider four confidence sources, all normalized to\[0,1\]\[0,1\]\.

##### Verbalized confidence\.

The model reports a confidence scoresverb∈\[0,100\]s\_\{\\mathrm\{verb\}\}\\in\[0,100\]with its binary prediction\([22](https://arxiv.org/html/2608.23660#bib.bib34);[33](https://arxiv.org/html/2608.23660#bib.bib40)\)\. We define

cverb​\(y^\)=sverb100\.c\_\{\\mathrm\{verb\}\}\(\\hat\{y\}\)=\\frac\{s\_\{\\mathrm\{verb\}\}\}\{100\}\.\(1\)

##### Logit\-based confidence\.

We aggregate valid surface forms for each label and normalize their decoder logits over𝒴\\mathcal\{Y\}\([17](https://arxiv.org/html/2608.23660#bib.bib41);[33](https://arxiv.org/html/2608.23660#bib.bib40)\):

p⁡\(y∣x\)=exp⁡\(zy\)∑y′∈𝒴exp⁡\(zy′\)\.p\(y\\mid x\)=\\frac\{\\exp\(z\_\{y\}\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}\\exp\(z\_\{y^\{\\prime\}\}\)\}\.\(2\)The predicted label and its confidence are

y^=argmaxy∈𝒴p​\(y∣x\),clogit​\(y^\)=maxy∈𝒴⁡p⁡\(y∣x\)\.\\hat\{y\}=\\operatorname\*\{argmax\}\_\{y\\in\\mathcal\{Y\}\}p\(y\\mid x\),\\qquad c\_\{\\mathrm\{logit\}\}\(\\hat\{y\}\)=\\max\_\{y\\in\\mathcal\{Y\}\}p\(y\\mid x\)\.\(3\)

##### Cross\-prompt agreement\.

ForK=5K=5prompt\-specific predictionsy\(1\),…,y\(K\)y^\{\(1\)\},\\ldots,y^\{\(K\)\}, lety^\\hat\{y\}be the majority label\. Confidence is the fraction of prompts supporting it\([37](https://arxiv.org/html/2608.23660#bib.bib42);[41](https://arxiv.org/html/2608.23660#bib.bib43)\):

cprompt\(y^\)=1K∑k=1K\[y\(k\)=y^\]\.c\_\{\\mathrm\{prompt\}\}\(\\hat\{y\}\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbf\{1\}\\\!\\left\[y^\{\(k\)\}=\\hat\{y\}\\right\]\.\(4\)

##### Cross\-model agreement\.

ForMMpredictionsy1,…,yMy\_\{1\},\\ldots,y\_\{M\}under a fixed prompt, lety^\\hat\{y\}be the majority label\. Computed separately within each model\-scale group, confidence is\([40](https://arxiv.org/html/2608.23660#bib.bib44);[44](https://arxiv.org/html/2608.23660#bib.bib39)\)

cmodel\(y^\)=1M∑m=1M\[ym=y^\]\.c\_\{\\mathrm\{model\}\}\(\\hat\{y\}\)=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\mathbf\{1\}\\\!\\left\[y\_\{m\}=\\hat\{y\}\\right\]\.\(5\)

## 3Experiment

### 3\.1Models

We evaluate 12 instruction\-tuned open\-weight models from five families: Qwen, Gemma, Llama, Mistral, and Phi\([43](https://arxiv.org/html/2608.23660#bib.bib16);[42](https://arxiv.org/html/2608.23660#bib.bib15);[14](https://arxiv.org/html/2608.23660#bib.bib12);[15](https://arxiv.org/html/2608.23660#bib.bib17);[26](https://arxiv.org/html/2608.23660#bib.bib14);[27](https://arxiv.org/html/2608.23660#bib.bib13);[2](https://arxiv.org/html/2608.23660#bib.bib18);[1](https://arxiv.org/html/2608.23660#bib.bib19)\), accessed through Hugging Face\.111Hugging Face models:[https://huggingface\.co/models](https://huggingface.co/models)\.To examine scale effects, we group models by parameter count\.

Small models \(4–14B\):Qwen3\-4B\-Instruct, Qwen3\-8B\-Instruct, Gemma\-4\-E4B\-IT, Llama\-3\.1\-8B\-Instruct, Ministral\-8B\-Instruct\-2410, Phi\-4\-Mini\-Instruct, and Phi\-4\.

Large models \(31–72B\):Qwen3\-32B\-Instruct, Qwen2\.5\-72B\-Instruct, Gemma\-4\-31B\-IT, Llama\-3\.1\-70B\-Instruct, and Llama\-3\.3\-70B\-Instruct\.

Figures abbreviate Gemma\-4\-31B\-IT as Gemma\-31B, Gemma\-4\-E4B\-IT as Gemma\-E4B, and Llama\-3\.1\-8B\-Instruct as Llama\-8B\.

### 3\.2Datasets

We use six graphs from CausalGraphBench\([4](https://arxiv.org/html/2608.23660#bib.bib26)\): AsiaM, River Status, COVID, Coal Gasifier, Hepar2, and Munin1\. They span four size categories and medical, ecological, public\-health, and industrial domains\. Dataset statistics, benchmark IDs, and preprocessing details appear in Appendix[A\.3](https://arxiv.org/html/2608.23660#A1.SS3), Table[A1](https://arxiv.org/html/2608.23660#A1.T1)\.

This subset provides a controlled size\- and domain\-stratified evaluation\. Pairwise classification scales asn⁡\(n−1\)n\(n\{\-\}1\), and the six graphs already require approximately 2\.5M queries, making full 35\-graph evaluation prohibitive\. We acknowledge in the Limitations that results may not capture full benchmark diversity\.

Each dataset provides a domain description, variable definitions, and a published reference graph\. The pairwise setting is highly imbalanced—only a small fraction of ordered pairs are true edges, worsening with graph size\. This motivates our calibration focus, as aggregate accuracy can mask overconfidence on sparse edge classes\.

### 3\.3Evaluation Metrics

We evaluate performance along two dimensions: direct\-edge classification and confidence calibration\. For classification, we report Precision, Recall, and F1 over ordered variable pairs\([11](https://arxiv.org/html/2608.23660#bib.bib27)\)\. To compare structural recovery across datasets of different sizes, we also report normalized Structural Hamming Distance \(nSHD\), based on SHD\([34](https://arxiv.org/html/2608.23660#bib.bib31)\)\. For datasetdd, nSHD is defined as:

nSHDd=SHDdnd​\(nd−1\),\\mathrm\{nSHD\}\_\{d\}=\\frac\{\\mathrm\{SHD\}\_\{d\}\}\{n\_\{d\}\(n\_\{d\}\-1\)\},\(6\)wherendn\_\{d\}is the number of variables in datasetdd\.

For calibration, we report 10\-bin squared\-gap ECE, based on the binning framework of[16](https://arxiv.org/html/2608.23660#bib.bib28), Brier Score\([7](https://arxiv.org/html/2608.23660#bib.bib29)\), and AUROC\([35](https://arxiv.org/html/2608.23660#bib.bib30)\)\. Lower ECE and Brier indicate better confidence–correctness alignment; higher AUROC indicates stronger discrimination between correct and incorrect predictions\.

### 3\.4Experimental Settings

Few\-shot prompts use five fixed demonstrations: three positive and two negative examples, held constant across models\. We use temperature 1\.0, repetition penalty 1\.1, and maximum generation lengths of 128 tokens for direct\-answer prompts and 384 tokens for reasoning prompts\.

## 4Results and Analysis

### 4\.1Primary Edge Classification

We analyze how prompt design, dataset characteristics, model scale, and potential benchmark familiarity affect direct\-edge classification\. We first examine the general precision–recall behavior and prompt sensitivity, then assess whether performance differences are associated with dataset characteristics and model scale\. We finally evaluate potential benchmark familiarity and whether a three\-way label space better distinguishes edge direction from the absence of a direct edge\.

#### 4\.1\.1Recall\-Dominant Behavior and Prompt Sensitivity

The results reveal a consistent recall\-dominant pattern across both model groups\. Recall exceeds precision in 68 of 72 model–dataset combinations \(94\.4%\) after averaging across prompts, and in 305 of 360 model–dataset–prompt combinations \(84\.7%\)\. Figure[3](https://arxiv.org/html/2608.23660#S4.F3)summarizes this pattern at the prompt and model levels, macro\-averaged across the remaining dimensions\. Nearly all aggregates lie above the equal\-precision–recall diagonal\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/precision_recall_prompt_model_combined_2.png)Figure 3:Precision–recall behavior by \(a\) prompt, averaged across models and datasets, and \(b\) model, averaged across prompts and datasets\. The dashed diagonal denotes equal precision and recall; points above it are recall\-dominant\.Although the magnitude of this behavior varies across settings, its direction remains stable\. In Table[1](https://arxiv.org/html/2608.23660#S4.T1), mean recall exceeds mean precision for every dataset and model\-scale group, with recall standard deviations of 0\.173–0\.316 across model–prompt combinations\. Models therefore predict edges for a large fraction of ordered pairs, producing dense graphs with many false positives\. This pattern suggests that LLMs often treat causal relatedness as evidence of direct causation\.

Table 1:Dataset\-level edge\-classification and graph\-reconstruction performance\. Values are mean±\\pmSD across model–prompt combinations within each scale group \(S: 7 models; L: 5 models\)\. Higher precision, recall, and F1 and lower nSHD indicate better performance\.DatasetPrecision↑\\uparrowRecall↑\\uparrowF1↑\\uparrownSHD↓\\downarrowSLSLSLSLAsiaM0\.415±0\.1750\.415\\pm 0\.1750\.608±0\.139\\mathbf\{0\.608\\pm 0\.139\}0\.707±0\.2640\.707\\pm 0\.2640\.830±0\.210\\mathbf\{0\.830\\pm 0\.210\}0\.478±0\.1490\.478\\pm 0\.1490\.680±0\.126\\mathbf\{0\.680\\pm 0\.126\}0\.281±0\.1450\.281\\pm 0\.1450\.141±0\.062\\mathbf\{0\.141\\pm 0\.062\}River Status0\.198±0\.0480\.198\\pm 0\.0480\.216±0\.062\\mathbf\{0\.216\\pm 0\.062\}0\.592±0\.2130\.592\\pm 0\.2130\.597±0\.292\\mathbf\{0\.597\\pm 0\.292\}0\.281±0\.0500\.281\\pm 0\.0500\.299±0\.107\\mathbf\{0\.299\\pm 0\.107\}0\.357±0\.1390\.357\\pm 0\.1390\.292±0\.090\\mathbf\{0\.292\\pm 0\.090\}COVID0\.261±0\.1810\.261\\pm 0\.1810\.412±0\.171\\mathbf\{0\.412\\pm 0\.171\}0\.443±0\.2210\.443\\pm 0\.2210\.545±0\.252\\mathbf\{0\.545\\pm 0\.252\}0\.280±0\.1440\.280\\pm 0\.1440\.403±0\.080\\mathbf\{0\.403\\pm 0\.080\}0\.191±0\.1580\.191\\pm 0\.1580\.104±0\.039\\mathbf\{0\.104\\pm 0\.039\}Coal Gasifier0\.049±0\.0290\.049\\pm 0\.0290\.069±0\.057\\mathbf\{0\.069\\pm 0\.057\}0\.428±0\.2470\.428\\pm 0\.2470\.505±0\.316\\mathbf\{0\.505\\pm 0\.316\}0\.078±0\.0450\.078\\pm 0\.0450\.095±0\.053\\mathbf\{0\.095\\pm 0\.053\}0\.288±0\.2040\.288\\pm 0\.2040\.225±0\.159\\mathbf\{0\.225\\pm 0\.159\}Hepar20\.062±0\.0210\.062\\pm 0\.0210\.086±0\.024\\mathbf\{0\.086\\pm 0\.024\}0\.546±0\.1730\.546\\pm 0\.1730\.593±0\.218\\mathbf\{0\.593\\pm 0\.218\}0\.106±0\.0300\.106\\pm 0\.0300\.143±0\.029\\mathbf\{0\.143\\pm 0\.029\}0\.265±0\.1440\.265\\pm 0\.1440\.191±0\.097\\mathbf\{0\.191\\pm 0\.097\}Munin10\.018±0\.016\\mathbf\{0\.018\\pm 0\.016\}0\.018±0\.008\\mathbf\{0\.018\\pm 0\.008\}0\.468±0\.2590\.468\\pm 0\.2590\.480±0\.295\\mathbf\{0\.480\\pm 0\.295\}0\.031±0\.019\\mathbf\{0\.031\\pm 0\.019\}0\.031±0\.014\\mathbf\{0\.031\\pm 0\.014\}0\.326±0\.2410\.326\\pm 0\.2410\.269±0\.219\\mathbf\{0\.269\\pm 0\.219\}

We further observe that this behavior persists across prompt styles\. FS, CoT, and FS\+CoT often shift models toward higher recall without comparable precision gains\. Appendix Figure[A1](https://arxiv.org/html/2608.23660#A1.F1)also shows that the best\-performing prompt varies across model–dataset combinations\. Thus, prompt design affects the precision–recall tradeoff, but no prompt style consistently achieves a better balance across models and datasets\.

#### 4\.1\.2Effect of Graph Size and Semantic Accessibility

Having established that prompting does not eliminate recall\-dominant behavior, we next examine why the balance between precision and recall varies across datasets\. We use F1 to summarize this balance and compare edge\-classification performance across graphs\. Table[1](https://arxiv.org/html/2608.23660#S4.T1)shows that F1 generally declines as graph size and class imbalance increase: mean F1 falls from 0\.281 and 0\.299 onRiver Statusto 0\.031 for both model groups onMunin1\. Although the SDs indicate variation across model–prompt combinations, the largest graphs consistently yield low F1\. Appendix Figure[A3](https://arxiv.org/html/2608.23660#A1.F3)and Table[A5](https://arxiv.org/html/2608.23660#A1.T5)report the complete results\.

Graph size alone does not fully explain these differences\. We therefore examine semantic accessibility using description length, code\-like variable names, and acronym density \(Appendix Table[A2](https://arxiv.org/html/2608.23660#A1.T2)\)\. Considering all 12 models, even the lowest model\-level mean F1 onRiver Status\(0\.220\) exceeds the highest achieved onCoal Gasifier\(0\.121\) andMunin1\(0\.050\), with each model averaged across five prompts\. This contrast is consistent with the natural\-language descriptions inRiver Statusand the more technical metadata in the latter datasets\.Munin1is particularly challenging, combining 186 variables with 100% code\-like names and 27\.4% acronym\-containing descriptions\.

However, metadata complexity alone is insufficient\.COVIDachieves comparatively higher F1 despite having 100% code\-like names and 45% acronym\-containing descriptions, as its metadata refers to broadly recognizable public\-health concepts\. Overall, these exploratory indicators suggest that performance is jointly associated with graph size, class imbalance, and the linguistic accessibility of domain concepts; they do not establish causal effects\. Overall, these exploratory indicators suggest that performance is jointly associated with graph size, class imbalance, and the linguistic accessibility of domain concepts; they do not establish causal effects\. This variation also motivates assessing whether greater model scale mitigates these constraints\.

#### 4\.1\.3Effect of Model Scale

Model scale improves mean F1 on five of six datasets, but the gains are uneven\. Figure[4](https://arxiv.org/html/2608.23660#S4.F4)compares small and large LLMs across datasets\. The largest improvements occur onAsiaM\(0\.4780\.478to0\.6800\.680\) andCOVID\(0\.2800\.280to0\.4030\.403\)\. In contrast, improvements are modest onRiver Status\(\+0\.018\+0\.018\),Coal Gasifier\(\+0\.017\+0\.017\), andHepar2\(\+0\.037\+0\.037\), while performance is effectively unchanged onMunin1\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/model_scale_f1_dumbbell.png)Figure 4:Dataset\-level mean F1 for small and large LLMs, averaged across models and prompts within each group\. Lines connect group means; rightward shifts favor large LLMs\.Scale also does not eliminate recall\-dominant behavior\. Figure[3](https://arxiv.org/html/2608.23660#S4.F3)shows that nearly all models remain above the equal\-precision–recall diagonal, while Table[1](https://arxiv.org/html/2608.23660#S4.T1)shows recall exceeding precision for both scale groups on every dataset\. Thus, larger models can improve edge recovery, particularly on semantically accessible graphs, but provide limited gains on the most difficult datasets and do not resolve systematic edge overprediction\. The especially large improvement onAsiaMwarrants qualification through the benchmark\-familiarity audit presented next\.

#### 4\.1\.4Potential Benchmark Familiarity Audit

The benchmark\-familiarity audit flags five model–dataset pairs as high risk, all involvingAsiaM; no model is flagged on the other five datasets\. Recall\-dominant overprediction and low performance on larger graphs persist across these unflagged datasets, indicating that the principal classification patterns are not driven byAsiaMalone\.

Following[4](https://arxiv.org/html/2608.23660#bib.bib26), we apply a two\-stage node\- and structure\-recall test to all 72 model–dataset pairs\. Each model first reproduces a graph’s nodes using only its source reference and domain description\. Generated and reference nodes are aligned through semantic matching using Mixtral\-8x7B\-Instruct, an independent judge not included among the evaluated models\. A pair is flagged when node\-count deviation is below 15% and node recall exceeds 0\.85; only flagged pairs proceed to structure recall\.

Table 2:Structure recall for model–dataset pairs passing the benchmark\-familiarity screen, all on AsiaM \(7 nodes, 8 edges\)\. Nodes reports generated/true counts, Dev\. is node\-count deviation, and SHD is the number of edge edits\.ModelNodesRecallDev\.Edge F1SHD↓\\downarrowPhi\-48/70\.85714\.3%0\.5006Qwen3\-32B7/70\.8570\.0%0\.7144Qwen2\.5\-72B8/71\.00014\.3%0\.8572Gemma\-4\-31B7/71\.0000\.0%0\.8003Llama\-3\.3\-70B8/71\.00014\.3%0\.9411

The flagged pairs show varying degrees of structure recovery \(Table[2](https://arxiv.org/html/2608.23660#S4.T2)\), indicating potential familiarity rather than uniform graph memorization\. Notably, Gemma\-4\-31B\-IT is flagged, suggesting that familiarity may partly contribute to the strong model\-scale gain observed onAsiaM\. We therefore interpret results on this dataset cautiously, while the principal classification conclusions remain supported by the five unflagged datasets\. Full results appear in Appendix Table[A13](https://arxiv.org/html/2608.23660#A1.T13)\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/pr_curves.png)Figure 5:Edge precision–recall curves obtained by thresholding positive\-edge confidence for small and large model groups, macro\-averaged across six datasets\.

### 4\.2Confidence Calibration

Having established that LLMs frequently overpredict causal edges, we examine whether confidence can identify which predictions are reliable\. This is important when LLM judgments are used as causal priors: informative confidence can support edge weighting, thresholding, and graph sparsification, whereas overconfident errors can reinforce incorrect structure\. We compare four confidence sources in terms of calibration and discrimination, examine their sensitivity to prompt style, and test whether post\-hoc temperature scaling improves logit\-based calibration\.

#### 4\.2\.1Agreement Supports Edge Ranking but Does Not Guarantee Instance\-Level Calibration

Agreement\-based confidence provides the strongest aggregate reliability signal, although its advantage varies across datasets and model groups\. Table[3](https://arxiv.org/html/2608.23660#S4.T3)shows that cross\-model agreement achieves the highest mean AUROC for small models \(0\.692±0\.0430\.692\\pm 0\.043\), while cross\-prompt agreement performs best for large models \(0\.772±0\.0470\.772\\pm 0\.047\)\. Both methods also achieve lower mean ECE and Brier scores than verbalized and logit\-based confidence\. However, pairwise two\-sided Wilcoxon tests over the six datasets show no significant differences after Holm adjustment \(minimum adjustedp=0\.1875p=0\.1875; Appendix Table[A9](https://arxiv.org/html/2608.23660#A1.T9)\)\. Agreement therefore performs better descriptively, but its statistical superiority across datasets is not established\.

Its practical value is clearest when confidence is used to rank or filter candidate edges\. Correctness AUROC evaluates whether confidence separates correct from incorrect predictions, whereas edge AUPRC treats the presence of a reference edge as the positive class\. We assign positive\-edge confidencecctoYespredictions and1−c1\-ctoNopredictions; logit\-based confidence uses the normalizedYesprobability\. Figure[5](https://arxiv.org/html/2608.23660#S4.F5)shows that cross\-model agreement achieves the highest edge AUPRC for both small \(0\.2280\.228\) and large \(0\.2690\.269\) LLMs\. For large models, a cross\-prompt threshold of0\.80\.8reduces graph density by44\.7%44\.7\\%and increases precision from0\.2500\.250to0\.3310\.331, while F1 remains nearly unchanged \(0\.3140\.314to0\.3100\.310\)\. Agreement can therefore support controllable graph sparsification, although increasing precision generally reduces recall\.

Table 3:Calibration by model group and confidence source, reported as mean±\\pmSD across six datasets\. ECE denotes squared\-gap ECE\. Bold and underlining indicate the best and second\-best results within each group\.GroupMethodECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowAcc\.↑\\uparrowSmallCross\-model0\.019±\\pm0\.0130\.149±\\pm0\.0430\.692±\\pm0\.0430\.792±\\pm0\.071Cross\-prompt0\.038±\\pm0\.0120\.176±\\pm0\.0430\.661±\\pm0\.0470\.755±\\pm0\.060Logit\-based0\.226±\\pm0\.0540\.354±\\pm0\.0460\.611±\\pm0\.0370\.549±\\pm0\.064Verbalized0\.229±\\pm0\.0340\.364±\\pm0\.0320\.358±\\pm0\.0590\.709±\\pm0\.057LargeCross\-prompt0\.016±\\pm0\.0120\.127±\\pm0\.0500\.772±\\pm0\.0470\.820±\\pm0\.070Cross\-model0\.024±\\pm0\.0180\.139±\\pm0\.0560\.704±\\pm0\.0550\.822±\\pm0\.070Verbalized0\.205±\\pm0\.0610\.320±\\pm0\.0630\.448±\\pm0\.0990\.794±\\pm0\.072Logit\-based0\.217±\\pm0\.0530\.346±\\pm0\.0570\.582±\\pm0\.0210\.620±\\pm0\.061These aggregate gains do not guarantee informative instance\-level uncertainty\. When confidence is concentrated near aggregate accuracy, ECE can be low without separating correct from incorrect predictions\. Appendix Figures[A7](https://arxiv.org/html/2608.23660#A1.F7)–[A7](https://arxiv.org/html/2608.23660#A1.F7)show that verbalized and logit\-based confidence can cluster within narrow high\-confidence ranges, partly reflecting dominant non\-edge behavior\. Logit\-based confidence is particularly concentrated near1\.01\.0for both correct and incorrect predictions \(Appendix Figure[A10](https://arxiv.org/html/2608.23660#A1.F10)\)\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/agreement_confidence_original_curves.png)Figure 6:Agreement\-confidence distributions pooled across six datasets: \(a\) cross\-prompt agreement for Gemma\-4\-31B\-IT and \(b\) cross\-model agreement under Metadata prompting\. Correct predictions concentrate more strongly near full agreement\.Agreement is less affected because it measures stability across prompts or models\. Figure[6](https://arxiv.org/html/2608.23660#S4.F6)shows that correct predictions concentrate more strongly near full agreement than incorrect predictions\. Nevertheless, agreement should be interpreted as an aggregate stability signal rather than a calibrated probability of correctness\. Because prompt formulation is one such inference condition, the following subsection examines how prompt style affects calibration\.

#### 4\.2\.2Effects of Prompt Style on Calibration

Prompt formulation affects both causal\-edge predictions and how confidence is expressed, but no prompt style consistently provides the best calibration across models and datasets\. Appendix Tables[A\.5\.2](https://arxiv.org/html/2608.23660#A1.SS5.SSS2)and[A\.5\.2](https://arxiv.org/html/2608.23660#A1.SS5.SSS2)show substantial variation across prompt styles and confidence sources\. For example, CoT reduces verbalized ECE for Gemma\-4\-E4B\-IT onCOVID,Hepar2, andMunin1, but the corresponding AUROC does not consistently improve\. Prompting can therefore shift confidence closer to aggregate accuracy without making it more informative for distinguishing correct from incorrect predictions\.

This instability provides the rationale for cross\-prompt agreement, which measures whether an edge judgment remains stable across all five prompt formulations rather than relying on confidence from one prompt\. Its reliability nevertheless remains model\- and dataset\-dependent, so prompt agreement should not be treated as uniformly calibrated\. The instability of direct confidence across prompt settings also raises whether its calibration can be improved after inference, which we examine next through post\-hoc temperature scaling\.

#### 4\.2\.3Post\-hoc Temperature Scaling

Post\-hoc temperature scaling changes logit\-based calibration in a model\-specific way but cannot correct errors in the underlying causal\-edge decision\. We evaluate this without rerunning the LLMs by rescaling the saved Yes/No logits and recomputing normalized binary confidence\. The predicted label remains unchanged because positive temperature scaling preserves the ordering of the Yes and No logits\. This analysis therefore isolates changes in confidence calibration from changes in edge classification\.

As shown in Appendix Fig\.[A11](https://arxiv.org/html/2608.23660#A1.F11), Gemma\-31B benefits from higher\-temperature smoothing: its ECE and Brier score decrease as temperature increases, suggesting that the original Yes/No logits are overconfident\. Qwen2\.5\-72B changes less across temperatures and maintains lower ECE, indicating comparatively stable logit\-based confidence\. AUROC remains nearly unchanged for both models because temperature scaling preserves the ranking induced by the original logit differences\.

These results show that temperature scaling can adjust confidence sharpness but cannot correct cases where the model assigns the higher logit to the wrong causal label\. To identify the structural distinctions underlying these persistent errors, we next examine whether overconfident false positives concentrate on indirect and reversed relations\.

### 4\.3Overconfident False Positives on Structurally Difficult Non\-Edges

The analysis reveals a clear structural pattern: false positives occur more frequently on indirect and reversed relations and often carry high verbal confidence\. We analyze valid model\-query instances where the reference graph contains no direct edgeA→BA\\rightarrow B\. Non\-edges are divided into three mutually exclusive categories:*reversed direct*, whereB→AB\\rightarrow Aexists;*indirect*, whereBBis reachable fromAAthrough a directed path of length at least two; and*other*non\-edges\. This separation distinguishes orientation errors from cases where a mediated causal relation is incorrectly classified as a direct edge\.

Table 4:False\-positive behavior by non\-edge type\. FP Rate is the fraction of valid queries producing false positives; High\-Conf\. FP Rate is the fraction producing false positives with verbal confidence at least 80\.Non\-edge TypeValidPairsFPCountFPRateHigh\-Conf\.FP RateReversed direct29,55910,64436\.0%30\.4%Indirect161,52064,56840\.0%32\.3%Other2,256,303637,20528\.2%22\.1%

Table[4](https://arxiv.org/html/2608.23660#S4.T4)shows that models predict a direct edge for 40\.0% of indirect non\-edges and 36\.0% of reversed direct non\-edges, compared with 28\.2% of other non\-edges\. High\-confidence false positives follow the same pattern: 32\.3% of indirect and 30\.4% of reversed direct queries produce false positives with verbal confidence of at least 80, compared with 22\.1% of other non\-edges\. Overprediction is therefore especially pronounced when the queried variables are causally related in the reference graph but not through the proposed direct edge\. The indirect\-versus\-other pattern holds across all six datasets, while reversed direct non\-edges have a higher false\-positive rate than other non\-edges on five of six datasets\.

Figure[7](https://arxiv.org/html/2608.23660#S4.F7)further shows that these errors remain concentrated at high confidence\. Among false positives with parseable verbal confidence, 84\.6% of reversed direct, 80\.8% of indirect, and 78\.4% of other false positives fall in the 80–100 confidence range \(Appendix Table[A10](https://arxiv.org/html/2608.23660#A1.T10)\)\. Appendix Table[A11](https://arxiv.org/html/2608.23660#A1.T11)reports the corresponding worst\-case model\-level behavior\.

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/false_positive_confidence_survival.png)Figure 7:Verbal\-confidence survival among false positives with parseable scores\. Each curve shows the fraction at or above a given threshold; reversed\-direct errors remain most concentrated at high confidence\.These findings should be interpreted relative to the published reference graphs\. Because CausalGraphBench provides a single reference structure for each dataset, some absent edges may reflect graph\-construction choices, and our analysis cannot establish that every false positive is causally invalid\. Nevertheless, the higher false\-positive rates on reversed and indirect relations show that the errors are structurally concentrated rather than uniform\. Reversed predictions contradict the encoded orientation, while indirect predictions collapse a represented mediated path into a direct edge\. Thus, relative to the benchmark specification, LLMs struggle to preserve causal direction and distinguish direct from mediated influence, although some predicted relations absent from the reference graph may remain causally plausible\.

Prompting strategy is also associated with this behavior\. Appendix Table[A10](https://arxiv.org/html/2608.23660#A1.T10)shows that FS\+CoT produces the highest high\-confidence false\-positive rate across all three categories, indicating that reasoning\-oriented demonstrations are associated with a greater tendency to infer direct edges between causally related variables\. Overall, LLMs appear to capture coarse causal relatedness while frequently misstating directness or orientation with high confidence\. This finding motivates the following ablation, which tests whether jointly representing both edge directions and the no\-edge class reduces these errors\.

### 4\.4Three\-Way Label\-Space Ablation

Three\-way labeling does not consistently improve graph recovery, indicating that independent binary queries are not the sole cause of edge overprediction\. In the original formulation, each unordered variable pair is evaluated through two separate questions: whetherA→BA\\rightarrow Bexists and whetherB→AB\\rightarrow Aexists\. Because these decisions are made independently, a model can predict both directions or fail to compare a directed edge directly against its reverse\.

The ablation replaces these two queries with one three\-way decision overA→BA\\rightarrow B,B→AB\\rightarrow A, and no direct edge\. This formulation forces the two directions to compete and prevents reciprocal predictions for the same pair\. It also makes directness explicit: ifAAinfluencesBBonly through an intermediate variable, the correct label is no direct edge\. The ablation therefore tests whether jointly deciding edge existence and orientation reduces the indirect and reversed errors identified in Section[4\.3](https://arxiv.org/html/2608.23660#S4.SS3)\.

We evaluate Qwen3\-4B\-Instruct, Phi\-4, Gemma\-4\-31B\-IT, and Llama\-3\.3\-70B\-Instruct onRiver Status,COVID,Hepar2, andMunin1using Metadata prompting\. The selected datasets represent the small, medium, large, and very\-large categories through River Status, COVID, Hepar2, and Munin1, respectively, while the models represent four families across two scale groups\. Binary results are taken from the corresponding Metadata runs in the main experiment\. Three\-way predictions are converted into directed graphs by adding the selectedA→BA\\rightarrow BorB→AB\\rightarrow Aedge and adding no edge for the third label\. Both formulations are then evaluated using the same edge precision, recall, F1, and nSHD metrics, macro\-averaged across the four datasets\. All three\-way runs cover every unordered pair and produce validly parsed labels\. The complete prompt appears in Appendix[A\.1\.6](https://arxiv.org/html/2608.23660#A1.SS1.SSS6)\.

Table 5:Binary versus three\-way direct\-edge classification under Metadata prompting, macro\-averaged across four datasets\. Bold indicates the better formulation for each model and metric\.ModelFormulationP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowQwen3\-4BBinary0\.2290\.5700\.2610\.200Three\-way0\.1240\.6780\.1940\.276Phi\-4Binary0\.2220\.4160\.2340\.132Three\-way0\.2240\.6270\.3030\.145Gemma\-4\-31BBinary0\.1770\.8160\.2700\.242Three\-way0\.1870\.7140\.2750\.205Llama\-3\.3\-70BBinary0\.1620\.0760\.0980\.068Three\-way0\.1520\.7950\.2360\.294

Table[5](https://arxiv.org/html/2608.23660#S4.T5)shows that three\-way labeling improves F1 for Phi\-4, Gemma\-4\-31B\-IT, and Llama\-3\.3\-70B\-Instruct, but decreases it for Qwen3\-4B\-Instruct\. These F1 changes do not translate into consistent structural improvement\. Only Gemma\-4\-31B\-IT improves both F1 and nSHD, and its F1 increase is marginal\. For Llama\-3\.3\-70B\-Instruct, recall increases from 0\.076 to 0\.795, but nSHD worsens from 0\.068 to 0\.294\. The model therefore predicts substantially more true edges while adding enough false edges to produce a less accurate reconstructed graph\.

The explicit no\-edge label also does not eliminate directness errors\. We define the three\-way non\-edge false\-positive rate as the fraction of reference no\-edge pairs assigned either directed label\. OnMunin1, this rate ranges from 13\.2% for Phi\-4 to 91\.0% for Llama\-3\.3\-70B\-Instruct, reaching 62\.9% for Qwen3\-4B\-Instruct and 45\.2% for Gemma\-4\-31B\-IT\. Direction\-specific recall is also asymmetric under the fixed unordered\-pair ordering, with Qwen3\-4B\-Instruct and Llama\-3\.3\-70B\-Instruct recovering noB→AB\\rightarrow Aedges onMunin1\. Because the directional classes are imbalanced, this asymmetry indicates sensitivity to pair ordering rather than establishing that one causal direction is intrinsically harder\.

Overall, forcing edge existence and orientation into a single mutually exclusive decision does not reliably reduce overprediction or improve orientation recovery\. The errors observed in the binary experiment are therefore not solely artifacts of asking about each direction separately; models continue to select directed relations when the reference graph specifies no direct edge\. Full dataset\-level results are reported in Appendix Table[A12](https://arxiv.org/html/2608.23660#A1.T12)\.

## 5Conclusion

We evaluated whether LLMs provide reliable direct causal\-edge judgments and confidence estimates across 12 open\-weight models, six benchmark graphs, and five prompting strategies\. Models were recall\-dominant, producing dense graphs, while prompting shifted the precision–recall tradeoff and gains from model scale diminished on larger graphs\. False positives concentrated on indirect and reversed relations and were frequently assigned high confidence, indicating that LLMs often capture causal relatedness without preserving directness or orientation\. A mutually exclusive three\-way formulation did not consistently improve graph recovery, showing that these errors are not solely artifacts of separate binary queries\.

Verbalized and logit\-based confidence were unreliable, whereas agreement\-based methods achieved better mean calibration and discrimination without establishing statistical superiority across datasets\. These patterns persisted on the five graphs not flagged for potential benchmark familiarity\. Overall, LLM judgments are better treated as externally validated soft priors than as direct evidence of causal structure\.

##### Future Work\.

Future work should develop graph\-aware uncertainty methods that incorporate sparsity, acyclicity, and directional consistency, and evaluate calibrated LLM judgments as soft priors within data\-driven CD pipelines\. Experimenting with Small Language Models \(SLMs\) can further assess whether lightweight models offer competitive edge\-level reliability at lower computational cost\. Robustness should also be tested across paraphrased prompts and broader graph collections\.

## Limitations

Our evaluation covers six of the 35 CausalGraphBench graphs\. Although stratified by size and domain, they may not represent the benchmark’s full diversity or noisier real\-world settings\. We treat the published structures as fixed reference graphs, although they may reflect expert choices about causal granularity; alternative annotations and inter\-annotator agreement estimates are unavailable\. Consequently, not every prediction labeled as a false positive can be established as causally invalid\.

Our language\-only, pairwise formulation uses neither observational nor interventional data and does not enforce global constraints such as acyclicity\. The semantic\-accessibility analysis relies on heuristic metadata indicators and is exploratory rather than causal\. We compare prompting strategies but do not test robustness to paraphrased wording within each strategy\. Results may also depend on the selected open\-weight models and decoding settings\. Finally, the benchmark\-familiarity audit flags five model–dataset pairs involving AsiaM, but its single detection protocol and judge model cannot establish memorization or rule out familiarity with unflagged datasets\.

## References

- Abdinet al\.\(2024\)M\. Abdinet al\.Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Aboueleninet al\.\(2025\)A\. Aboueleninet al\.Phi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-LoRAs\.arXiv preprint arXiv:2503\.01743\.Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Andreassenet al\.\(1989\)S\. Andreassen, F\. V\. Jensen, S\. K\. Andersen, B\. Falck, U\. Kjærulff, M\. Woldbye, A\. R\. Sørensen, A\. Rosenfalck, and F\. JensenMUNIN: an expert EMG assistant\.InComputer\-Aided Electromyography and Expert Systems,J\. E\. Desmedt \(Ed\.\),pp\. 255–277\.Cited by:[Table A1](https://arxiv.org/html/2608.23660#A1.T1.2.1.11.3)\.
- Babakovet al\.\(2025\)N\. Babakov, E\. Reiter, and A\. Bugarín\-DizCausalGraphBench: a benchmark for evaluating language models capabilities of causal graph discovery\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),Vienna, Austria,pp\. 240–258\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-srw.16)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.23660#S3.SS2.p1.1),[§4\.1\.4](https://arxiv.org/html/2608.23660#S4.SS1.SSS4.p2.1)\.
- Banet al\.\(2023a\)T\. Ban, L\. Chen, D\. Lyu, X\. Wang, and H\. ChenCausal structure learning supervised by large language model\.arXiv preprint arXiv:2311\.11689\.Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1)\.
- Banet al\.\(2023b\)T\. Ban, L\. Chen, X\. Wang, and H\. ChenFrom query tools to causal architects: harnessing large language models for advanced causal discovery from data\.arXiv preprint arXiv:2306\.16902\.Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1)\.
- Brier \(1950\)G\. W\. BrierVerification of forecasts expressed in terms of probability\.Monthly Weather Review78\(1\),pp\. 1–3\.External Links:[Document](https://dx.doi.org/10.1175/1520-0493%281950%29078%3C0001%3AVOFEIT%3E2.0.CO%3B2)Cited by:[§3\.3](https://arxiv.org/html/2608.23660#S3.SS3.p2.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p2.1)\.
- Chenet al\.\(2023\)B\. Chen, Z\. Zhang, N\. Langrené, and S\. ZhuUnleashing the potential of prompt engineering in large language models: a comprehensive review\.arXiv preprint arXiv:2310\.14735\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.14735),[Link](https://arxiv.org/abs/2310.14735)Cited by:[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p1.1)\.
- Chickering \(2002\)D\. M\. ChickeringOptimal structure identification with greedy search\.Journal of Machine Learning Research3,pp\. 507–554\.External Links:[Link](https://www.jmlr.org/papers/v3/chickering02b.html)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1)\.
- Davis and Goadrich \(2006\)J\. Davis and M\. GoadrichThe relationship between precision\-recall and roc curves\.InProceedings of the 23rd International Conference on Machine Learning,pp\. 233–240\.Cited by:[§3\.3](https://arxiv.org/html/2608.23660#S3.SS3.p1.1)\.
- Fenget al\.\(2025\)T\. Feng, L\. Qu, N\. Tandon, Z\. Li, X\. Kang, and G\. HaffariOn the reliability of large language models for causal discovery\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 9565–9590\.External Links:[Link](https://aclanthology.org/2025.acl-long.471/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.471),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1)\.
- Glymouret al\.\(2019\)C\. Glymour, K\. Zhang, and P\. SpirtesReview of causal discovery methods based on graphical models\.Frontiers in Genetics10,pp\. 524\.Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1)\.
- Google DeepMind \(2026\)Google DeepMindGemma 4 model card\.Note:[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)Accessed: 2026Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§3\.3](https://arxiv.org/html/2608.23660#S3.SS3.p2.1)\.
- Jianget al\.\(2021\)Z\. Jiang, J\. Araki, H\. Ding, and G\. NeubigHow can we know when language models know? on the calibration of language models for question answering\.Transactions of the Association for Computational Linguistics9,pp\. 962–977\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00407),[Link](https://aclanthology.org/2021.tacl-1.57/)Cited by:[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px2.p1.1)\.
- Kampaniet al\.\(2024\)S\. Kampani, D\. Hidary, C\. van der Poel, M\. Ganahl, and B\. MiaoLLM\-initialized differentiable causal discovery\.arXiv preprint arXiv:2406\.06406\.Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1)\.
- Kıcımanet al\.\(2024\)E\. Kıcıman, R\. Ness, A\. Sharma, and C\. TanCausal reasoning and large language models: opening a new frontier for causality\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=mqoxLkX210)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p1.1)\.
- Lachapelleet al\.\(2020\)S\. Lachapelle, P\. Brouillard, T\. Deleu, and S\. Lacoste\-JulienGradient\-Based Neural DAG Learning\.InProceedings of the Eighth International Conference on Learning Representations,ICLR 2020,Addis Ababa, Ethiopia\.External Links:[Link](https://iclr.cc/virtual/2020/poster/1922)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1)\.
- Lauritzen and Spiegelhalter \(1988\)S\. L\. Lauritzen and D\. J\. SpiegelhalterLocal computations with probabilities on graphical structures and their application to expert systems\.Journal of the Royal Statistical Society: Series B \(Methodological\)50\(2\),pp\. 157–224\.Cited by:[Table A1](https://arxiv.org/html/2608.23660#A1.T1.2.1.3.3)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTeaching models to express their uncertainty in words\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=8s8K2UZGTZ)Cited by:[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px1.p1.1)\.
- Liuet al\.\(2022\)Y\. Liu, S\. Wang, Q\. Liu, D\. Liu, Y\. Yang, Y\. Dan, and W\. WuFailure risk assessment of coal gasifier based on the integration of bayesian network and trapezoidal intuitionistic fuzzy number\-based similarity aggregation method \(tpifn\-sam\)\.Processes10\(9\),pp\. 1863\.External Links:[Document](https://dx.doi.org/10.3390/pr10091863)Cited by:[Table A1](https://arxiv.org/html/2608.23660#A1.T1.2.1.7.3)\.
- Longet al\.\(2023\)S\. Long, A\. Piché, V\. Zantedeschi, T\. Schuster, and A\. DrouinCausal discovery with language models as imperfect experts\.InICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling,External Links:2307\.02390,[Link](https://arxiv.org/abs/2307.02390)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1)\.
- Mayfieldet al\.\(2022\)H\. J\. Mayfield, C\. L\. Lau, J\. E\. Sinclair, S\. J\. Brown, A\. Baird, J\. Litt, A\. Vuorinen, K\. R\. Short, M\. Waller, and K\. MengersenDesigning an evidence\-based bayesian network for estimating the risk versus benefits of astrazeneca covid\-19 vaccine\.Vaccine40\(22\),pp\. 3072–3084\.External Links:[Document](https://dx.doi.org/10.1016/j.vaccine.2022.04.004)Cited by:[Table A1](https://arxiv.org/html/2608.23660#A1.T1.2.1.6.3)\.
- Meta AI \(2024\)Meta AILlama\-3\.3\-70B\-Instruct model card\.Note:[https://huggingface\.co/meta\-llama/Llama\-3\.3\-70B\-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)Accessed: 2024Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Mistral AI \(2024\)Mistral AIMinistral\-8B\-Instruct\-2410\.Note:[https://huggingface\.co/mistralai/Ministral\-8B\-Instruct\-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410)Accessed: 2024Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Molina\-Navarroet al\.\(2020\)E\. Molina\-Navarro, P\. Segurado, P\. Branco, C\. Almeida, and H\. E\. AndersenPredicting the ecological status of rivers and streams under different climatic and socioeconomic scenarios using bayesian belief networks\.Limnologica80,pp\. 125742\.External Links:[Document](https://dx.doi.org/10.1016/j.limno.2019.125742)Cited by:[Table A1](https://arxiv.org/html/2608.23660#A1.T1.2.1.4.3)\.
- Onisko \(2003\)A\. OniskoProbabilistic causal models in medicine: application to diagnosis of liver disorders\.Ph\.D\. Thesis,Institute of Biocybernetics and Biomedical Engineering, Polish Academy of Sciences,Warsaw, Poland\.Cited by:[Table A1](https://arxiv.org/html/2608.23660#A1.T1.2.1.9.3)\.
- Pearl \(2009\)J\. PearlCausality: models, reasoning, and inference\.2 edition,Cambridge University Press\.Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1)\.
- Qiaoet al\.\(2023\)S\. Qiao, Y\. Ou, N\. Zhang, X\. Chen, Y\. Yao, S\. Deng, C\. Tan, F\. Huang, and H\. ChenReasoning with language model prompting: a survey\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 5368–5393\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.294),[Link](https://aclanthology.org/2023.acl-long.294/)Cited by:[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p1.1)\.
- Spirteset al\.\(2000\)P\. Spirtes, C\. Glymour, and R\. ScheinesCausation, prediction, and search\.2nd edition,Adaptive Computation and Machine Learning,MIT Press,Cambridge, MA\.External Links:ISBN 978\-0\-262\-19440\-2Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1)\.
- Tianet al\.\(2023\)K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. ManningJust ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 5433–5442\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330),[Link](https://aclanthology.org/2023.emnlp-main.330/)Cited by:[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px1.p1.1),[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px2.p1.1)\.
- Tsamardinoset al\.\(2006\)I\. Tsamardinos, L\. E\. Brown, and C\. F\. AliferisThe Max\-Min Hill\-Climbing Bayesian Network structure learning algorithm\.Machine Learning65\(1\),pp\. 31–78\.External Links:[Document](https://dx.doi.org/10.1007/s10994-006-6889-7)Cited by:[§3\.3](https://arxiv.org/html/2608.23660#S3.SS3.p1.1)\.
- Ulmeret al\.\(2024\)D\. Ulmer, M\. Gubri, H\. Lee, S\. Yun, and S\. OhCalibrating Large Language Models using their generations only\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 15440–15459\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.824),[Link](https://aclanthology.org/2024.acl-long.824/)Cited by:[§3\.3](https://arxiv.org/html/2608.23660#S3.SS3.p2.1)\.
- Vashishthaet al\.\(2023\)A\. Vashishtha, A\. G\. Reddy, A\. Kumar, S\. Bachu, V\. N\. Balasubramanian, and A\. SharmaCausal inference using llm\-guided discovery\.arXiv preprint arXiv:2310\.15117\.Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1),[§1](https://arxiv.org/html/2608.23660#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px3.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p2.1)\.
- Whiteet al\.\(2023\)J\. White, Q\. Fu, S\. Hays, M\. Sandborn, C\. Olea, H\. Gilbert, A\. Elnashar, J\. Spencer\-Smith, and D\. C\. SchmidtA prompt pattern catalog to enhance prompt engineering with chatgpt\.arXiv preprint arXiv:2302\.11382\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2302.11382),[Link](https://arxiv.org/abs/2302.11382)Cited by:[§2\.1](https://arxiv.org/html/2608.23660#S2.SS1.p1.1)\.
- Xiaet al\.\(2025\)Y\. Xia, P\. H\. Luz De Araujo, K\. Zaporojets, and B\. RothInfluences on LLM calibration: a study of response agreement, loss functions, and prompt styles\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 3740–3761\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.188),[Link](https://aclanthology.org/2025.acl-long.188/)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px4.p1.1)\.
- Xionget al\.\(2024\)M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. HooiCan LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gjeQKFxFpZ)Cited by:[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)A\. Yanget al\.Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yanget al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1](https://arxiv.org/html/2608.23660#S3.SS1.p1.1)\.
- Zhanget al\.\(2024\)M\. Zhang, J\. He, T\. Ji, and C\. LuDon’t go to extremes: revealing the excessive sensitivity and calibration limitations of LLMs in implicit hate speech detection\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12073–12086\.External Links:[Link](https://aclanthology.org/2024.acl-long.652/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.652)Cited by:[§2\.3](https://arxiv.org/html/2608.23660#S2.SS3.SSS0.Px4.p1.1)\.
- Zhenget al\.\(2018\)X\. Zheng, B\. Aragam, P\. Ravikumar, and E\. P\. XingDAGs with NO TEARS: Continuous Optimization for Structure Learning\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/e347c51419ffb23ca3fd5050202f9c3d-Abstract.html)Cited by:[§1](https://arxiv.org/html/2608.23660#S1.p1.1)\.

## Appendix AAppendix

### A\.1Prompt Templates

#### A\.1\.1Name\-only Prompt

Does A cause B? Answer Yes or No\. Provide your confidence \(0\-100%\) that A directly causes B\.Context:\{context\}Variable A:\{var\_a\}Variable B:\{var\_b\}Answer \(Yes/No, Confidence%\):

#### A\.1\.2Metadata Prompt

Given variable definitions, does A cause B? Answer Yes or No\. Provide your confidence \(0\-100%\) that A directly causes B\.Context:\{context\}Variable A:\{var\_a\}Definition:\{def\_a\}Variable B:\{var\_b\}Definition:\{def\_b\}Answer \(Yes/No, Confidence%\):

#### A\.1\.3Chain\-of\-Thought Prompt

Given variable definitions, does A cause B? Think step by step before answering\.Context:\{context\}Variable A:\{var\_a\}Definition:\{def\_a\}Variable B:\{var\_b\}Definition:\{def\_b\}Answer \(Yes/No\): Confidence \(0\-100%\): Reasoning:

#### A\.1\.4Few\-shot Prompt

Does A cause B? Answer Yes or No\. Provide your confidence \(0\-100%\) that A directly causes B\.Context:\{context\}Example 1: Variable A: Smoking \(tobacco intake\) Variable B: Lung Cancer \(malignant tumor\) Answer: Yes, 95%Example 2: Variable A: Rain \(precipitation\) Variable B: Wet Roads \(surface moisture\) Answer: Yes, 98%Example 3: Variable A: Exercise \(physical activity\) Variable B: Income \(earnings\) Answer: No, 90%Example 4: Variable A: Height \(body length\) Variable B: Intelligence \(cognitive ability\) Answer: No, 88%Example 5: Variable A: UV Radiation \(sun exposure\) Variable B: Skin Cancer \(malignant melanoma\) Answer: Yes, 93%Now answer: Variable A:\{var\_a\}Definition:\{def\_a\}Variable B:\{var\_b\}Definition:\{def\_b\}Answer \(Yes/No, Confidence%\):

#### A\.1\.5Few\-shot \+ Chain\-of\-Thought Prompt

Given variable definitions, does A cause B? Think step by step\. Provide answer, confidence, reasoning\.Context:\{context\}Example 1: Variable A: Smoking \(tobacco intake\) Variable B: Lung Cancer \(malignant tumor\) Answer: Yes, 95% Reasoning: Tobacco contains carcinogens that directly damage lung tissue leading to malignant tumor formation\.Example 2: Variable A: Rain \(precipitation\) Variable B: Wet Roads \(surface moisture\) Answer: Yes, 98% Reasoning: Rain directly causes water to accumulate on road surfaces\.Example 3: Variable A: Exercise \(physical activity\) Variable B: Income \(earnings\) Answer: No, 90% Reasoning: Physical activity level has no direct causal effect on earnings\.Example 4: Variable A: Height \(body length\) Variable B: Intelligence \(cognitive ability\) Answer: No, 88% Reasoning: Body height does not causally determine cognitive ability\.Example 5: Variable A: UV Radiation \(sun exposure\) Variable B: Skin Cancer \(malignant melanoma\) Answer: Yes, 93% Reasoning: UV radiation directly damages DNA in skin cells, triggering mutations that lead to malignant tumor formation\.Now answer: Variable A:\{var\_a\}Definition:\{def\_a\}Variable B:\{var\_b\}Definition:\{def\_b\}Answer \(Yes/No\): Confidence \(0\-100%\): Reasoning:

#### A\.1\.6Three\-Way Ablation Prompt

Given the context and variable definitions, determine the direct causal relationship between Variable A and Variable B\.Choose exactly one label:A\_TO\_B: Variable A directly causes Variable B\.B\_TO\_A: Variable B directly causes Variable A\.NO\_EDGE: There is no direct causal edge in either direction\.Provide your confidence \(0\-\-100%\) in the selected label\.Return only the selected label and confidence\. Do not provide reasoning or an explanation\.Context: \{context\}Variable A: \{var\_a\}Definition: \{def\_a\}Variable B: \{var\_b\}Definition: \{def\_b\}Answer \(A\_TO\_B/B\_TO\_A/NO\_EDGE, Confidence%\):

### A\.2Confidence Estimation Details

Formal definitions are provided in Section[2\.3](https://arxiv.org/html/2608.23660#S2.SS3)\. Verbalized confidence is extracted from the standardized answer–confidence response after deterministic parsing of labeled fields and answer\-first forms\. Logit\-based confidence aggregates accepted surface forms ofYesandNobefore renormalization over the binary label set\. Cross\-prompt agreement uses predictions from the five prompt styles, whereas cross\-model agreement is computed separately within the small\- and large\-model groups under a fixed prompt\. Agreement\-based predictions use the majority label\.

### A\.3Benchmark Dataset Details

#### A\.3\.1Dataset Statistics

Table A1:Reference\-graph statistics\. No\-edge:edge denotes class imbalance over ordered variable pairs\.DatasetDomainSourceNodesEdgesPairsNo\-edge:EdgeSmall networks\(n<20n<20\)AsiaMRespiratory diagnosisbnlearn;\([21](https://arxiv.org/html/2608.23660#bib.bib20)\)78424\.25:1River StatusEcological quality\([28](https://arxiv.org/html/2608.23660#bib.bib21)\)15252107\.40:1Medium networks\(20≤n≤5020\\leq n\\leq 50\)COVIDVaccine risk\-benefitBayesFusion;\([25](https://arxiv.org/html/2608.23660#bib.bib22)\)202638013\.62:1Coal GasifierIndustrial risk\([23](https://arxiv.org/html/2608.23660#bib.bib23)\)39391,48237\.00:1Large networks\(51≤n≤10051\\leq n\\leq 100\)Hepar2Hepatic diagnosisbnlearn;\([29](https://arxiv.org/html/2608.23660#bib.bib24)\)701234,83038\.27:1Very large networks\(n\>100n\>100\)Munin1Electromyographybnlearn;\([3](https://arxiv.org/html/2608.23660#bib.bib25)\)18627334,410125\.04:1

DatasetNumber ofvariablesMean wordsper descriptionCode\-like variablenames \(%\)Descriptions withacronyms \(%\)AsiaM75\.30\.00\.0River Status153\.76\.70\.0COVID206\.8100\.045\.0Coal Gasifier393\.9100\.05\.1Hepar2703\.415\.70\.0Munin11867\.6100\.027\.4Table A2:Metadata\-level indicators of variable\-description complexity across datasets\. The table reports simple descriptive measures of the node metadata used in the classification prompts\. Lower values for the number of variables, code\-like variable names, and acronym\-containing descriptions generally indicate easier metadata conditions\. Mean words per description is included as a descriptive measure of label length, but is not assumed to have a strictly monotonic relationship with performance because longer descriptions may either clarify or complicate variable meaning\.
#### A\.3\.2Dataset Preprocessing

To avoid leaking structural cues to the LLM, we remove explicit references to “Bayesian Network” from the dataset\-level context before prompting\. Since this term may implicitly suggest that directed edges exist among the variables, we replace it with the neutral term “system,” while preserving the original domain meaning of each dataset\.

### A\.4Model Classification Performance Ranking

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/prompt_best_f1_heatmap_clean.png)Figure A1:Prompt\-level F1 performance across models and datasets\. Each cell shows the best F1 score and prompt, with right\-side bars summarizing prompt wins per model\. The horizontal line separates large and small models\.
### A\.5Complete Results Tables

#### A\.5\.1Primary Classification Tables

ModelDatasetName\-onlyMetadataCoTFew\-shotFew\-shot\+CoTP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowGemma\-4\- E4B\-ITasiam0\.5000\.3750\.4290\.1900\.4170\.6250\.5000\.2141\.0000\.1250\.2220\.1670\.4000\.5000\.4440\.1900\.6250\.6250\.6250\.143river0\.2000\.2400\.2180\.2050\.2390\.4400\.3100\.2330\.2120\.5600\.3080\.3000\.1670\.9200\.2820\.5570\.1690\.8800\.2840\.529covid0\.0710\.0380\.0500\.0970\.4350\.3850\.4080\.0740\.4210\.3080\.3560\.0760\.2020\.6540\.3090\.1970\.1620\.6540\.2600\.255coal0\.0420\.0260\.0320\.0410\.1250\.2560\.1680\.0670\.0960\.3080\.1460\.0930\.0380\.8460\.0730\.5620\.0430\.7180\.0820\.423hepar20\.0960\.3250\.1480\.0940\.0930\.4150\.1520\.1150\.0960\.1790\.1250\.0620\.0460\.6260\.0860\.3350\.0490\.5610\.0910\.281munin10\.0140\.0510\.0220\.0370\.0330\.1940\.0560\.0510\.0180\.2420\.0340\.1090\.0100\.8500\.0190\.6870\.0090\.8020\.0180\.675Llama\-3\.1\-8B\- Instructasiam0\.3330\.8750\.4830\.3570\.3750\.7500\.5000\.2620\.5560\.6250\.5880\.1430\.3040\.8750\.4520\.4050\.4670\.8750\.6090\.214river0\.1720\.6800\.2740\.4140\.1920\.5600\.2860\.3100\.2450\.4800\.3240\.2240\.1530\.8400\.2590\.5670\.2220\.7200\.3400\.329covid0\.1360\.4620\.2110\.2260\.2000\.4620\.2790\.1580\.0950\.0770\.0850\.1110\.1340\.6920\.2250\.3240\.1980\.6920\.3080\.203coal0\.0290\.5130\.0550\.4570\.0290\.3590\.0530\.3290\.0180\.1540\.0330\.2330\.0420\.6670\.0780\.4110\.0370\.4100\.0690\.287hepar20\.0440\.5610\.0810\.3200\.0490\.5040\.0900\.2570\.0680\.3660\.1150\.1400\.0460\.7400\.0870\.3910\.0600\.6830\.1100\.278munin10\.0090\.5710\.0180\.4880\.0120\.5090\.0230\.3370\.0130\.4620\.0250\.2880\.0100\.8350\.0200\.6490\.0100\.6150\.0190\.506Ministral\-8B\- Instruct\-2410asiam0\.1820\.7500\.2930\.6670\.2920\.8750\.4380\.4050\.1710\.7500\.2790\.6900\.2140\.7500\.3330\.5240\.2410\.8750\.3780\.524river0\.1280\.6400\.2130\.5290\.1520\.6800\.2480\.4670\.1130\.6400\.1930\.6140\.1540\.8000\.2580\.5330\.1240\.7600\.2130\.643covid0\.0590\.5000\.1060\.5550\.0790\.5770\.1390\.4760\.0530\.4000\.0930\.4950\.0570\.5000\.1020\.5820\.0580\.5000\.1040\.568coal0\.0240\.5640\.0470\.5940\.0270\.5640\.0520\.5390\.0260\.6150\.0510\.6010\.0240\.5640\.0460\.6110\.0240\.6410\.0470\.681hepar20\.0260\.6020\.0490\.5870\.0260\.5530\.0500\.5290\.0270\.5770\.0520\.5320\.0340\.6420\.0640\.4700\.0300\.6100\.0580\.499munin10\.0080\.5750\.0150\.5900\.0080\.6010\.0170\.5590\.0090\.6370\.0180\.5390\.0080\.5600\.0160\.5590\.0090\.6410\.0170\.571Phi\-4asiam0\.4170\.6250\.5000\.2140\.6360\.8750\.7370\.1190\.5000\.8750\.6360\.1900\.4211\.0000\.5930\.2620\.3750\.7500\.5000\.262river0\.2380\.4000\.2990\.2240\.3330\.4400\.3790\.1710\.2100\.6800\.3210\.3430\.2200\.5200\.3100\.2760\.1790\.6800\.2830\.405covid0\.4500\.3460\.3910\.0710\.4440\.3080\.3640\.0740\.4640\.5000\.4810\.0710\.2960\.6150\.4000\.1260\.2000\.5600\.2950\.176coal0\.0000\.0000\.0000\.0740\.0800\.6670\.1430\.2100\.0510\.7440\.0960\.3640\.0500\.6410\.0930\.3270\.0410\.7440\.0790\.455hepar20\.0950\.5370\.1610\.1410\.0880\.5930\.1530\.1650\.0770\.7720\.1410\.2390\.0570\.7320\.1060\.3130\.0550\.7770\.1030\.337munin10\.0290\.1100\.0470\.0350\.0220\.3220\.0410\.1190\.0180\.4800\.0340\.2140\.0150\.5270\.0280\.2870\.0110\.6810\.0210\.495Phi\-4\-Mini\- Instructasiam0\.3750\.3750\.3750\.1900\.5710\.5000\.5330\.1670\.5000\.2500\.3330\.1670\.0000\.0000\.0000\.2620\.3330\.2500\.2860\.167river0\.2000\.2080\.2040\.1760\.2270\.2080\.2170\.1710\.1750\.2800\.2150\.2290\.1520\.4000\.2200\.3290\.2080\.4000\.2740\.252covid0\.1580\.1150\.1330\.0970\.1920\.1920\.1920\.1080\.1540\.1540\.1540\.1080\.1050\.1600\.1270\.1390\.0800\.0770\.0780\.121coal0\.0550\.1050\.0720\.0670\.0640\.1790\.0940\.0890\.0600\.2050\.0920\.1030\.0450\.2820\.0780\.1710\.0440\.2310\.0740\.148hepar20\.0710\.2280\.1090\.0920\.0840\.2030\.1190\.0740\.0750\.2600\.1170\.0980\.0630\.3690\.1070\.1520\.0650\.3820\.1110\.152munin10\.0120\.1850\.0220\.1290\.0160\.2750\.0310\.1370\.0110\.1980\.0210\.1420\.0130\.2920\.0240\.1820\.0130\.2600\.0240\.164Qwen3\-4B\- Instructasiam0\.4211\.0000\.5930\.2620\.4380\.8750\.5830\.2380\.3330\.8750\.4830\.3570\.3811\.0000\.5520\.3100\.3201\.0000\.4850\.405river0\.2220\.6400\.3300\.3100\.2130\.7600\.3330\.3620\.2440\.8330\.3770\.3190\.2070\.7600\.3250\.3760\.2110\.9200\.3430\.414covid0\.6150\.3330\.4320\.0610\.6190\.5000\.5530\.0530\.4360\.6540\.5230\.0820\.3210\.6920\.4390\.1210\.1980\.7690\.3150\.229coal0\.0000\.0000\.0000\.0280\.0970\.4740\.1610\.1260\.0670\.5130\.1190\.1990\.0600\.4100\.1050\.1840\.0430\.6670\.0800\.402hepar20\.0640\.5160\.1140\.2020\.0580\.6420\.1070\.2710\.0540\.7330\.1010\.3240\.0660\.6500\.1200\.2410\.0520\.7240\.0970\.340munin10\.0890\.1360\.1070\.0170\.0270\.3770\.0500\.1130\.0200\.5090\.0380\.2060\.0170\.6920\.0320\.3290\.0110\.8320\.0210\.609Qwen3\-8B\- Instructasiam0\.7140\.6250\.6670\.1190\.5380\.8750\.6670\.1670\.4670\.8750\.6090\.2140\.3331\.0000\.5000\.3810\.3890\.8750\.5380\.286river0\.3330\.3600\.3460\.1620\.2000\.3200\.2460\.2330\.1830\.4400\.2590\.3000\.1620\.8400\.2710\.5190\.1680\.8000\.2780\.476covid0\.5000\.2310\.3160\.0680\.5240\.4230\.4680\.0660\.6000\.3460\.4390\.0610\.1940\.7690\.3100\.2340\.2120\.8460\.3380\.226coal–0\.0000\.0000\.0260\.0820\.2310\.1210\.0880\.1190\.3850\.1820\.0910\.0310\.6670\.0600\.5490\.0350\.6150\.0670\.454hepar20\.0870\.4720\.1470\.1380\.0840\.5530\.1450\.1640\.0710\.4800\.1240\.1700\.0480\.7800\.0900\.4000\.0510\.7720\.0950\.372munin10\.0640\.1170\.0830\.0200\.0260\.1900\.0460\.0610\.0170\.2050\.0310\.1010\.0110\.9410\.0210\.7010\.0100\.9160\.0200\.714

Table A3:Primary causal edge classification performance for small language models across six benchmark datasets and five prompt styles\. Cell colors encode performance percentile within the full table \(yellow = good, orange = bad\), with intensity scaled per metric direction \(P/R/F1 higher is better; nSHD lower is better\)\. A dash denotes undefined precision because the model made no positive predictions\.Small LLMsModelPromptP↑\\uparrowR↑\\uparrowF1↑\\uparrowAvg\. nSHD↓\\downarrowGemma\-4\-E4B\-ITName\-only0\.1540\.1760\.1500\.111Metadata0\.2240\.3860\.2660\.126CoT0\.3070\.2870\.1990\.135Few\-shot0\.1440\.7330\.2020\.421Few\-shot \+ CoT0\.1760\.7070\.2270\.384Llama\-3\.1\-8B\-InstructName\-only0\.1210\.6100\.1870\.377Metadata0\.1430\.5240\.2050\.275CoT0\.1660\.3610\.1950\.190Few\-shot0\.1150\.7750\.1870\.458Few\-shot \+ CoT0\.1660\.6660\.2430\.303Ministral\-8B\-Instruct\-2410Name\-only0\.0710\.6050\.1210\.587Metadata0\.0970\.6420\.1570\.496CoT0\.0670\.6030\.1140\.579Few\-shot0\.0820\.6360\.1360\.547Few\-shot \+ CoT0\.0810\.6710\.1360\.581Phi\-4Name\-only0\.2050\.3360\.2330\.127Metadata0\.2670\.5340\.3030\.143CoT0\.2200\.6750\.2850\.237Few\-shot0\.1760\.6720\.2550\.265Few\-shot \+ CoT0\.1440\.6990\.2130\.355Phi\-4\-Mini\-InstructName\-only0\.1450\.2030\.1520\.125Metadata0\.1920\.2600\.1980\.124CoT0\.1630\.2250\.1550\.141Few\-shot0\.0630\.2510\.0930\.206Few\-shot \+ CoT0\.1240\.2670\.1410\.167Qwen3\-4B\-InstructName\-only0\.2350\.4380\.2630\.147Metadata0\.2420\.6050\.2980\.194CoT0\.1920\.6860\.2740\.248Few\-shot0\.1750\.7010\.2620\.260Few\-shot \+ CoT0\.1390\.8190\.2240\.400Qwen3\-8B\-InstructName\-only0\.2830\.3010\.2600\.089Metadata0\.2420\.4320\.2820\.130CoT0\.2430\.4550\.2740\.156Few\-shot0\.1300\.8330\.2090\.464Few\-shot \+ CoT0\.1440\.8040\.2230\.421

Large LLMsModelPromptP↑\\uparrowR↑\\uparrowF1↑\\uparrowAvg\. nSHD↓\\downarrowGemma\-4\-31B\-ITName\-only0\.2360\.5240\.2780\.150Metadata0\.2410\.8410\.3350\.222CoT0\.2430\.8160\.3300\.238Few\-shot0\.2760\.7380\.3550\.184Few\-shot \+ CoT0\.2340\.8400\.3260\.255Llama\-3\.1\-70B\-InstructName\-only0\.1830\.2730\.2010\.128Metadata0\.2470\.3180\.2510\.096CoT0\.1840\.8690\.2720\.372Few\-shot0\.1760\.6950\.2610\.243Few\-shot \+ CoT0\.1780\.8300\.2550\.401Llama\-3\.3\-70B\-InstructName\-only0\.2910\.1810\.2110\.074Metadata0\.2950\.1630\.1990\.066CoT0\.2040\.8440\.2930\.320Few\-shot0\.2060\.6500\.2980\.178Few\-shot \+ CoT0\.1680\.8420\.2510\.403Qwen2\.5\-72B\-InstructName\-only0\.3870\.1780\.2040\.076Metadata0\.3280\.3800\.3040\.084CoT0\.2670\.6840\.3200\.198Few\-shot0\.2130\.6580\.2850\.213Few\-shot \+ CoT0\.2140\.8050\.2970\.293Qwen3\-32B\-InstructName\-only0\.2070\.3440\.2080\.144Metadata0\.2650\.5350\.2790\.172CoT0\.2780\.5450\.3140\.143Few\-shot0\.2120\.5780\.2760\.196Few\-shot \+ CoT0\.1960\.6550\.2750\.241

Table A4:Macro\-averaged primary causal edge classification performance for small \(left\) and large \(right\) LLMs across all datasets\. Precision \(P\), recall \(R\), and F1 are macro\-averaged over the six benchmark datasets for each model–prompt pair\. Avg\. nSHD denotes the macro\-average of normalized structural Hamming distance, computed for each dataset asSHD/\(nd​\(nd−1\)\)\\mathrm\{SHD\}/\(n\_\{d\}\(n\_\{d\}\-1\)\), wherendn\_\{d\}is the number of variables in datasetdd\. Higher P, R, and F1 are better, while lower Avg\. nSHD is better\. Bold values indicate the best prompt for each model and metric\.Small LLMsLarge LLMsDatasetModelAvg\. P↑\\uparrowAvg\. R↑\\uparrowAvg\. F1↑\\uparrowAvg\. nSHD↓\\downarrowModelAvg\. P↑\\uparrowAvg\. R↑\\uparrowAvg\. F1↑\\uparrowAvg\. nSHD↓\\downarrowasiamGemma\-4\-E4B\-IT0\.5880\.4500\.4440\.181Qwen3\-32B\-Instruct0\.6030\.8250\.6940\.124Llama\-3\.1\-8B\-Instruct0\.4070\.8000\.5260\.276Gemma\-4\-31B\-IT0\.7050\.9000\.7800\.090Ministral\-8B\-Instruct\-24100\.2200\.8000\.3440\.562Qwen2\.5\-72B\-Instruct0\.6800\.8500\.7330\.114Phi\-40\.4700\.8250\.5930\.210Llama\-3\.3\-70B\-Instruct0\.6120\.8000\.6380\.167Phi\-4\-Mini\-Instruct0\.3560\.2750\.3050\.190Llama\-3\.1\-70B\-Instruct0\.4380\.7750\.5520\.210Qwen3\-4B\-Instruct0\.3790\.9500\.5390\.314Qwen3\-8B\-Instruct0\.4880\.8500\.5960\.233riverGemma\-4\-E4B\-IT0\.1970\.6080\.2800\.365Qwen3\-32B\-Instruct0\.2230\.6160\.3260\.303Llama\-3\.1\-8B\-Instruct0\.1970\.6560\.2970\.369Gemma\-4\-31B\-IT0\.2580\.8100\.3910\.296Ministral\-8B\-Instruct\-24100\.1340\.7040\.2250\.557Qwen2\.5\-72B\-Instruct0\.1600\.4560\.2200\.271Phi\-40\.2360\.5440\.3180\.284Llama\-3\.3\-70B\-Instruct0\.2280\.4960\.2610\.272Phi\-4\-Mini\-Instruct0\.1920\.2990\.2260\.231Llama\-3\.1\-70B\-Instruct0\.2130\.6080\.2950\.315Qwen3\-4B\-Instruct0\.2190\.7830\.3420\.356Qwen3\-8B\-Instruct0\.2090\.5520\.2800\.338covidGemma\-4\-E4B\-IT0\.2580\.4080\.2760\.140Qwen3\-32B\-Instruct0\.4180\.3620\.3410\.089Llama\-3\.1\-8B\-Instruct0\.1530\.4770\.2210\.204Gemma\-4\-31B\-IT0\.3520\.8000\.4860\.116Ministral\-8B\-Instruct\-24100\.0610\.4950\.1090\.535Qwen2\.5\-72B\-Instruct0\.5810\.4380\.4240\.086Phi\-40\.3710\.4660\.3860\.104Llama\-3\.3\-70B\-Instruct0\.3590\.5310\.3700\.108Phi\-4\-Mini\-Instruct0\.1380\.1400\.1370\.115Llama\-3\.1\-70B\-Instruct0\.3520\.5920\.3940\.122Qwen3\-4B\-Instruct0\.4380\.5900\.4530\.109Qwen3\-8B\-Instruct0\.4060\.5230\.3740\.131coalGemma\-4\-E4B\-IT0\.0690\.4310\.1000\.237Qwen3\-32B\-Instruct0\.0520\.5010\.0930\.215Llama\-3\.1\-8B\-Instruct0\.0310\.4210\.0580\.344Gemma\-4\-31B\-IT0\.0520\.6080\.0960\.241Ministral\-8B\-Instruct\-24100\.0250\.5900\.0480\.605Qwen2\.5\-72B\-Instruct0\.0910\.5130\.1210\.178Phi\-40\.0450\.5590\.0820\.286Llama\-3\.3\-70B\-Instruct0\.0880\.4360\.0730\.252Phi\-4\-Mini\-Instruct0\.0530\.2010\.0820\.116Llama\-3\.1\-70B\-Instruct0\.0690\.4670\.0930\.241Qwen3\-4B\-Instruct0\.0530\.4130\.0930\.188Qwen3\-8B\-Instruct0\.0670\.3790\.0860\.242hepar2Gemma\-4\-E4B\-IT0\.0760\.4210\.1200\.177Qwen3\-32B\-Instruct0\.0740\.4810\.1280\.164Llama\-3\.1\-8B\-Instruct0\.0530\.5710\.0970\.277Gemma\-4\-31B\-IT0\.0930\.7690\.1650\.196Ministral\-8B\-Instruct\-24100\.0290\.5970\.0550\.523Qwen2\.5\-72B\-Instruct0\.1020\.5410\.1620\.151Phi\-40\.0740\.6820\.1330\.239Llama\-3\.3\-70B\-Instruct0\.0910\.5530\.1360\.194Phi\-4\-Mini\-Instruct0\.0720\.2880\.1130\.113Llama\-3\.1\-70B\-Instruct0\.0720\.6200\.1230\.249Qwen3\-4B\-Instruct0\.0590\.6530\.1080\.275Qwen3\-8B\-Instruct0\.0680\.6110\.1200\.249munin1Gemma\-4\-E4B\-IT0\.0170\.4280\.0300\.312Qwen3\-32B\-Instruct0\.0210\.4050\.0390\.180Llama\-3\.1\-8B\-Instruct0\.0110\.5990\.0210\.453Gemma\-4\-31B\-IT0\.0150\.6250\.0300\.319Ministral\-8B\-Instruct\-24100\.0080\.6030\.0170\.564Qwen2\.5\-72B\-Instruct0\.0170\.4470\.0320\.238Phi\-40\.0190\.4240\.0340\.230Llama\-3\.3\-70B\-Instruct0\.0190\.4010\.0240\.257Phi\-4\-Mini\-Instruct0\.0130\.2420\.0240\.151Llama\-3\.1\-70B\-Instruct0\.0170\.5190\.0300\.352Qwen3\-4B\-Instruct0\.0320\.5090\.0500\.255Qwen3\-8B\-Instruct0\.0260\.4740\.0400\.320

Table A5:Dataset\-wise prompt\-averaged primary causal edge classification performance for small and large LLMs on the six benchmark datasets\. For each model–dataset pair, precision \(P\), recall \(R\), F1, and normalized structural Hamming distance \(nSHD\) are averaged across the five prompt styles\. nSHD is computed asSHD/\(nd​\(nd−1\)\)\\mathrm\{SHD\}/\(n\_\{d\}\(n\_\{d\}\-1\)\), wherendn\_\{d\}is the number of variables in datasetdd\. Higher P, R, and F1 are better, while lower Avg\. nSHD is better\. Bold values indicate the best model within each dataset and metric, with small and large LLMs evaluated independently\.ModelDatasetName\-onlyMetadataCoTFew\-shotFew\-shot\+CoTP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowQwen3\-32B\- Instructasiam0\.5000\.6250\.5560\.1900\.6670\.7500\.7060\.0950\.7271\.0000\.8420\.0710\.5830\.8750\.7000\.1190\.5380\.8750\.6670\.143river0\.2030\.5600\.2980\.3140\.2170\.5200\.3060\.2810\.2730\.7200\.3960\.2620\.2090\.5600\.3040\.3050\.2120\.7200\.3270\.352covid0\.4440\.1540\.2290\.0710\.5380\.2690\.3590\.0660\.5000\.2690\.3500\.0680\.3240\.4230\.3670\.1000\.2810\.6920\.4000\.139coal0\.0000\.0000\.0000\.0280\.0820\.6840\.1460\.2060\.0760\.5130\.1320\.1760\.0550\.6670\.1010\.3100\.0460\.6410\.0870\.355hepar20\.0600\.4720\.1070\.1960\.0670\.4630\.1170\.1730\.0720\.4630\.1240\.1610\.0860\.5280\.1480\.1500\.0860\.4800\.1460\.139munin10\.0320\.2560\.0570\.0670\.0200\.5240\.0380\.2090\.0210\.3040\.0390\.1170\.0170\.4180\.0340\.1890\.0130\.5240\.0250\.319Gemma\-4\- 31B\-ITasiam0\.6670\.5000\.5710\.1430\.6671\.0000\.8000\.0950\.7271\.0000\.8420\.0710\.8001\.0000\.8890\.0480\.6671\.0000\.8000\.095river0\.2790\.7730\.4100\.2480\.2660\.8750\.4080\.2950\.2470\.8400\.3820\.3240\.2570\.7200\.3790\.2810\.2410\.8400\.3750\.333covid0\.3640\.7690\.4940\.1080\.3330\.8080\.4720\.1240\.3180\.8080\.4570\.1320\.4000\.6920\.5070\.0920\.3430\.9230\.5000\.126coal0\.0000\.0000\.0000\.0330\.0690\.7840\.1270\.2670\.0590\.7690\.1100\.3260\.0770\.6920\.1380\.2250\.0560\.7950\.1050\.355hepar20\.0920\.7460\.1650\.1900\.0930\.7980\.1670\.1940\.0910\.8130\.1640\.2100\.1050\.6990\.1830\.1570\.0820\.7890\.1480\.229munin10\.0160\.3590\.0300\.1810\.0170\.7830\.0330\.3560\.0140\.6670\.0280\.3670\.0160\.6260\.0320\.2990\.0140\.6920\.0270\.394Qwen2\.5\-72B\- Instructasiam0\.8000\.5000\.6150\.1190\.7780\.8750\.8240\.0710\.5711\.0000\.7270\.1430\.5830\.8750\.7000\.1430\.6671\.0000\.8000\.095river0\.0000\.0000\.0000\.1520\.1540\.0800\.1050\.1620\.2130\.7600\.3330\.3620\.2130\.5200\.3020\.2860\.2210\.9200\.3570\.395covid1\.0000\.2690\.4240\.0500\.7270\.3080\.4320\.0550\.6110\.4230\.5000\.0580\.3110\.5380\.3940\.1130\.2580\.6540\.3700\.153coal–0\.0000\.0000\.0260\.1440\.3850\.2100\.0740\.0920\.7180\.1630\.1940\.0750\.7180\.1360\.2390\.0530\.7440\.0980\.358hepar20\.1250\.2600\.1680\.0640\.1330\.3740\.1960\.0770\.0940\.5850\.1620\.1520\.0810\.7240\.1460\.2140\.0750\.7640\.1370\.246munin10\.0080\.0400\.0140\.0450\.0340\.2560\.0600\.0630\.0170\.6150\.0330\.2810\.0160\.5710\.0310\.2860\.0120\.7510\.0230\.512Llama\-3\.3\-70B\- Instructasiam0\.7500\.3750\.5000\.1430\.8330\.6250\.7140\.0950\.5331\.0000\.6960\.1670\.4711\.0000\.6400\.2140\.4711\.0000\.6400\.214river0\.3570\.2000\.2560\.1380\.1430\.0400\.0620\.1430\.2160\.8800\.3460\.3860\.2340\.6000\.3370\.2810\.1900\.7600\.3040\.414covid0\.4620\.2310\.3080\.0680\.4000\.1540\.2220\.0710\.3570\.7690\.4880\.1110\.3330\.7690\.4650\.1210\.2440\.7310\.3650\.171coal0\.0000\.0000\.0000\.0310\.2860\.0510\.0870\.0280\.0420\.7690\.0790\.4700\.0730\.5900\.1300\.2070\.0370\.7690\.0720\.523hepar20\.1480\.2760\.1930\.0570\.0970\.1060\.1010\.0470\.0650\.8130\.1200\.3030\.0900\.6910\.1590\.1840\.0560\.8780\.1060\.378munin10\.0290\.0040\.0070\.0090\.0100\.0040\.0050\.0110\.0140\.8350\.0270\.4850\.0340\.2530\.0600\.0620\.0100\.9120\.0200\.717Llama\-3\.1\-70B\- Instructasiam0\.3330\.3750\.3530\.2380\.5380\.8750\.6670\.1430\.3811\.0000\.5520\.3100\.4000\.7500\.5220\.2140\.5380\.8750\.6670\.143river0\.2140\.3600\.2690\.2330\.2000\.1600\.1780\.1760\.2300\.9200\.3680\.3710\.2500\.8000\.3810\.3100\.1710\.8000\.2820\.486covid0\.3330\.3460\.3400\.0920\.5380\.2690\.3590\.0630\.3800\.7310\.5000\.1000\.2500\.8080\.3820\.1790\.2560\.8080\.3890\.174coal0\.1110\.0510\.0700\.0360\.0700\.1280\.0910\.0650\.0460\.8210\.0860\.4550\.0770\.5640\.1360\.1860\.0420\.7690\.0800\.462hepar20\.0860\.3820\.1400\.1170\.1070\.3410\.1630\.0870\.0550\.8130\.1030\.3590\.0630\.7400\.1160\.2860\.0510\.8210\.0950\.396munin10\.0200\.1210\.0340\.0540\.0290\.1360\.0480\.0420\.0110\.9270\.0230\.6380\.0140\.5050\.0280\.2820\.0100\.9080\.0190\.745

Table A6:Primary causal edge classification performance for large language models across six benchmark datasets and five prompt styles\. Cell colors encode performance percentile within the full table \(yellow = good, orange = bad\), with intensity scaled per metric direction \(P/R/F1 higher is better; nSHD lower is better\)\. A dash denotes undefined precision because the model made no positive predictions\.
#### A\.5\.2Calibration Tables

ModelDatasetMethodName\-onlyMetadataCoTFew\-shotFew\-shot \+ CoTAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrow\\endfirsthead*\(continued from previous page\)*ModelDatasetMethodName\-onlyMetadataCoTFew\-shotFew\-shot \+ CoTAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrow\\endhead*\(continued on next page\)*\\endfootPrompt\-specific calibration results for small LLMs onasiam,river,covid,coal,hepar2, andmunin1\. Cell colors encode performance percentile within the full table \(yellow = good, orange = bad\), with intensity scaled separately per metric direction\.\\endlastfootGemma\-4\-E4B\-ITasiamVerb\.0\.3970\.5130\.6410\.2440\.4170\.5390\.8780\.0100\.1030\.3750\.0510\.2190\.3840\.0240\.142Logit0\.6020\.0230\.1430\.8720\.0210\.0630\.5000\.6550\.8100\.5620\.0440\.2050\.6650\.0540\.141riverVerb\.0\.3200\.3560\.4850\.1920\.3890\.5030\.3950\.0880\.2800\.1950\.3160\.4600\.2660\.2550\.432Logit0\.6910\.0190\.1380\.6350\.0160\.1550\.5030\.5520\.7420\.4850\.2620\.5090\.5470\.2940\.525covidVerb\.0\.3980\.5500\.6340\.3420\.5900\.6570\.6610\.0090\.0740\.2330\.0810\.2170\.3490\.0570\.234Logit0\.6240\.0110\.0630\.6130\.0050\.0580\.5000\.5900\.7680\.6600\.0360\.1780\.6060\.0450\.225coalVerb\.0\.5410\.5210\.5610\.2450\.5000\.5470\.4050\.0590\.1430\.1760\.2980\.4610\.1630\.2600\.379Logit0\.4990\.0010\.0260\.7740\.0020\.0440\.5000\.8190\.9050\.4920\.2630\.5100\.5850\.1420\.381hepar2Verb\.0\.1440\.5330\.5830\.1320\.5480\.5980\.4230\.0370\.0970\.1600\.1300\.2970\.2400\.1080\.254Logit0\.7760\.0030\.0650\.7250\.0060\.0860\.5000\.9320\.9650\.5710\.0870\.3060\.5970\.1050\.328munin1Verb\.0\.1360\.5230\.5440\.1040\.4990\.5270\.3820\.0400\.1340\.0360\.5020\.5750\.0650\.4650\.551Logit0\.8060\.0010\.0150\.8240\.0000\.0300\.4990\.8540\.9240\.4540\.4160\.6190\.4600\.4090\.615Llama\-3\.1\-8B\-InstructasiamVerb\.0\.2200\.3110\.4720\.4350\.2310\.3770\.4730\.0230\.1550\.2090\.3130\.4820\.4380\.0370\.198Logit0\.8030\.0590\.2290\.8950\.0200\.1380\.7040\.0150\.1150\.5800\.0680\.3030\.6900\.0540\.203riverVerb\.0\.2260\.2850\.4600\.3430\.1560\.3540\.5120\.0170\.1920\.3370\.2510\.4580\.4360\.0970\.293Logit0\.5800\.0180\.2560\.7230\.0040\.1580\.7850\.0140\.1620\.4990\.1640\.3940\.6510\.0320\.240covidVerb\.0\.2890\.3680\.5220\.3320\.3310\.4590\.5640\.0510\.1490\.2950\.1740\.3660\.4680\.0580\.220Logit0\.7770\.0310\.0950\.7770\.0110\.0690\.6780\.0050\.0650\.6770\.0020\.1930\.8160\.0050\.101coalVerb\.0\.1270\.3620\.5100\.1950\.2630\.4320\.4490\.0280\.2050\.2350\.2360\.4270\.4430\.0730\.271Logit0\.9360\.0500\.2150\.7480\.0820\.1140\.8350\.0120\.0970\.6710\.0250\.2400\.8090\.0060\.156hepar2Verb\.0\.2080\.3400\.5030\.2320\.3220\.4780\.5180\.0330\.1530\.2100\.2430\.4220\.5020\.0530\.246Logit0\.8090\.0260\.1510\.8660\.0320\.0940\.8640\.0030\.0660\.6130\.0460\.2780\.7400\.0170\.186munin1Verb\.0\.1230\.3670\.5120\.2320\.2060\.3930\.4640\.0280\.2270\.2070\.3900\.5470\.4060\.1920\.423Logit0\.6430\.0300\.2570\.8740\.0640\.0950\.8290\.0080\.1210\.4670\.3030\.4560\.6380\.1590\.333Ministral\-8B\-Instruct\-2410asiamVerb\.0\.5030\.3870\.5740\.4970\.1310\.3500\.5800\.3920\.5800\.5060\.1750\.4050\.5400\.1470\.393Logit0\.6360\.2420\.3900\.5670\.1520\.3360\.7170\.2730\.4170\.5480\.2000\.3750\.6630\.2920\.429riverVerb\.0\.3910\.2450\.4720\.4200\.1830\.4180\.4580\.3200\.5290\.3060\.1810\.4100\.4100\.3230\.535Logit0\.5430\.2720\.3760\.5160\.1350\.2850\.4310\.2900\.4030\.5690\.2400\.3650\.7160\.3840\.482covidVerb\.0\.4230\.2090\.4420\.4700\.1620\.3880\.5190\.1680\.4080\.3640\.2250\.4530\.4380\.2380\.475Logit0\.5650\.2610\.3420\.6740\.0220\.2330\.4240\.0230\.2650\.5840\.3030\.3800\.5030\.2960\.388coalVerb\.0\.3830\.2930\.5140\.3680\.2470\.4740\.5040\.2800\.5080\.3200\.2670\.4810\.4210\.3370\.550Logit0\.2920\.3250\.3500\.2840\.2610\.3130\.3060\.3140\.3680\.3080\.2670\.3610\.3230\.4650\.505hepar2Verb\.0\.3780\.2800\.5050\.4120\.2320\.4650\.5550\.2220\.4630\.3710\.1200\.3590\.4620\.1520\.401Logit0\.4320\.3410\.3810\.4100\.1670\.3150\.3110\.2300\.3410\.5400\.0210\.2650\.4730\.0460\.290munin1Verb\.0\.3680\.2890\.5110\.3750\.2540\.4830\.5160\.2260\.4670\.3270\.1950\.4230\.4380\.2050\.450Logit0\.4980\.3480\.3570\.2510\.2900\.3230\.2180\.2660\.3320\.2520\.2280\.3350\.2310\.2780\.371Phi\-4asiamVerb\.0\.6800\.1790\.2740\.6240\.0790\.1650\.7560\.0090\.1430\.6770\.0260\.1960\.6020\.0310\.233Logit0\.5080\.0190\.1200\.7840\.0160\.1030\.8930\.6550\.8100\.8690\.0920\.1710\.6640\.0790\.240riverVerb\.0\.7720\.0180\.1560\.6600\.0390\.1620\.8360\.0620\.2100\.5390\.0150\.2120\.5560\.0720\.309Logit0\.7710\.0240\.1520\.7830\.0120\.1420\.6270\.7420\.8570\.8290\.0590\.2370\.3730\.2370\.488covidVerb\.0\.6710\.0120\.0770\.6990\.0200\.0830\.7150\.0020\.0620\.6270\.0040\.1100\.7600\.0210\.134Logit0\.8130\.0070\.0530\.7750\.0020\.0470\.6710\.7550\.8690\.8290\.0050\.0860\.5440\.0200\.185coalVerb\.0\.4680\.5580\.6270\.5220\.1830\.3080\.8490\.0720\.2150\.4830\.0350\.2510\.6210\.1050\.337Logit0\.6360\.0010\.0260\.7780\.0190\.1640\.5720\.9130\.9540\.7280\.0600\.2650\.4100\.2360\.484hepar2Verb\.0\.8250\.0260\.1190\.8100\.0410\.1430\.8880\.0500\.1530\.6260\.0350\.2370\.6620\.0540\.260Logit0\.8130\.0070\.1040\.8000\.0100\.1240\.5280\.9130\.9540\.7490\.0610\.2620\.4900\.1330\.377munin1Verb\.0\.4360\.3700\.4040\.6830\.0740\.1660\.8720\.0290\.1360\.4800\.0200\.2260\.5230\.1250\.370Logit0\.9050\.0000\.0150\.8750\.0030\.0750\.2720\.8950\.9440\.7210\.0230\.2010\.3320\.2780\.515Phi\-4\-Mini\-InstructasiamVerb\.0\.3670\.4100\.5600\.4310\.3430\.4640\.6050\.1360\.2420\.3320\.1830\.3590\.5220\.1250\.271Logit0\.7460\.0030\.1380\.8820\.0280\.1090\.8900\.0260\.1220\.7840\.0120\.1140\.8400\.0210\.124riverVerb\.0\.3150\.3860\.5160\.2870\.4240\.5480\.3810\.1210\.2970\.2120\.1840\.3480\.3560\.0580\.237Logit0\.5940\.0090\.1030\.6840\.0100\.0990\.7420\.0030\.0960\.7920\.0030\.1410\.7840\.0070\.122covidVerb\.0\.3650\.3990\.4890\.3400\.4320\.5230\.4670\.0770\.1800\.3480\.1630\.2830\.4110\.1010\.205Logit0\.7030\.0020\.0590\.8160\.0030\.0570\.8380\.0030\.0580\.7180\.0050\.0630\.6820\.0030\.062coalVerb\.0\.2890\.5650\.6240\.2430\.5200\.5900\.4350\.1040\.2000\.2110\.2000\.3140\.2690\.0930\.216Logit0\.5500\.0010\.0260\.7900\.0040\.0300\.7050\.0010\.0270\.7650\.0170\.0480\.6960\.0090\.035hepar2Verb\.0\.1770\.6260\.6920\.2110\.5970\.6580\.4530\.0830\.1740\.1760\.2370\.3390\.3420\.1010\.227Logit0\.8520\.0020\.0360\.8450\.0010\.0310\.8050\.0020\.0280\.8630\.0090\.0600\.8560\.0080\.050munin1Verb\.0\.1270\.5730\.6450\.1850\.4820\.5730\.4220\.0670\.1900\.1810\.1260\.2420\.2790\.0900\.218Logit0\.7770\.0160\.0250\.9040\.0160\.0340\.7080\.0120\.0200\.8810\.0300\.0480\.7450\.0180\.028Qwen3\-4B\-InstructasiamVerb\.0\.1660\.3000\.4150\.3200\.2260\.3310\.3540\.1850\.3090\.2310\.1260\.2720\.3580\.1190\.329Logit0\.6250\.5140\.6820\.5480\.5290\.7170\.6250\.2120\.4310\.5010\.0970\.3080\.5290\.1630\.402riverVerb\.0\.1540\.2690\.3700\.1450\.2860\.3980\.2080\.1330\.2990\.1650\.1020\.2890\.1850\.2020\.362Logit0\.5240\.2720\.5180\.5000\.2810\.5220\.5470\.1000\.3180\.5020\.1430\.3750\.4990\.1740\.416covidVerb\.0\.3860\.4000\.4480\.3840\.3420\.3900\.3980\.0940\.1680\.1940\.0610\.1510\.2350\.0610\.218Logit0\.4940\.4350\.6510\.5310\.2490\.4950\.5920\.0770\.3040\.5730\.0140\.1110\.5660\.0510\.222coalVerb\.0\.4250\.2730\.2990\.1010\.2530\.3260\.4170\.0320\.1900\.1750\.0570\.1850\.4190\.1150\.320Logit0\.5000\.9480\.9740\.5140\.1870\.4330\.6090\.0380\.2040\.5690\.0290\.1710\.5280\.1470\.386hepar2Verb\.0\.0800\.3830\.4540\.0990\.2860\.3820\.2650\.1260\.2970\.1280\.1340\.2280\.2300\.1180\.288Logit0\.4540\.4180\.6390\.5120\.1690\.4110\.5240\.1590\.4030\.5310\.0540\.2360\.5260\.1070\.331munin1Verb\.0\.1180\.4440\.4520\.0290\.3790\.4080\.2690\.0830\.2000\.0850\.2250\.3060\.4030\.2690\.481Logit0\.5050\.9710\.9850\.4550\.6290\.7830\.5080\.2240\.4710\.5170\.1010\.3220\.4790\.3460\.583Qwen3\-8B\-InstructasiamVerb\.0\.3460\.4170\.4990\.2470\.4670\.5230\.6580\.0860\.1710\.2550\.1600\.3240\.2430\.1140\.266Logit0\.5000\.6550\.8100\.5000\.6550\.8100\.5000\.6550\.8100\.5150\.2060\.4390\.4810\.0680\.240riverVerb\.0\.2490\.4810\.5770\.1790\.4360\.5320\.3250\.0760\.2400\.1860\.2320\.4160\.1430\.3160\.436Logit0\.5000\.7760\.8810\.5000\.7680\.8760\.5000\.7760\.8810\.4380\.2340\.4830\.5130\.2220\.466covidVerb\.0\.2660\.6110\.6630\.1980\.6690\.7150\.5540\.0130\.0690\.1650\.1150\.2290\.1280\.1260\.220Logit0\.4970\.6150\.7820\.5010\.8430\.9180\.5000\.8680\.9320\.6040\.0620\.2420\.5740\.0490\.220coalVerb\.0\.5460\.4780\.5030\.1120\.5520\.5760\.1800\.0510\.1180\.0690\.3740\.4520\.0410\.3750\.411Logit0\.5000\.9480\.9740\.5000\.9470\.9730\.5000\.9480\.9740\.4960\.2940\.5380\.5320\.1710\.415hepar2Verb\.0\.0470\.5850\.6140\.0480\.5880\.6240\.4170\.0340\.1560\.0980\.2500\.3450\.0540\.2940\.346Logit0\.4980\.9090\.9530\.4970\.9340\.9660\.5000\.9500\.9750\.5220\.1520\.3920\.5240\.1250\.357munin1Verb\.0\.1320\.5630\.5710\.0400\.5610\.5760\.2400\.0370\.1160\.1410\.3810\.5100\.0250\.5950\.621Logit0\.5000\.9840\.9920\.4980\.9830\.9920\.5000\.9840\.9920\.4420\.4680\.6710\.4210\.4850\.680
Small LLMsLarge LLMsDatasetModelAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowModelAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowasiamGemma\-4\-E4B\-IT0\.8450\.0050\.110Qwen3\-32B\-Instruct0\.7470\.0020\.077Llama\-3\.1\-8B\-Instruct0\.6890\.0040\.158Gemma\-4\-31B\-IT0\.7740\.0010\.056Ministral\-8B\-Instruct\-24100\.5400\.2280\.428Qwen2\.5\-72B\-Instruct0\.8220\.0060\.065Phi\-40\.7730\.0020\.118Llama\-3\.3\-70B\-Instruct0\.8630\.0140\.094Phi\-4\-Mini\-Instruct0\.7790\.0100\.138Llama\-3\.1\-70B\-Instruct0\.7530\.0100\.148Qwen3\-4B\-Instruct0\.5880\.0460\.240Qwen3\-8B\-Instruct0\.7740\.0180\.151riverGemma\-4\-E4B\-IT0\.6160\.0270\.234Qwen3\-32B\-Instruct0\.6420\.0570\.252Llama\-3\.1\-8B\-Instruct0\.6280\.0060\.212Gemma\-4\-31B\-IT0\.5270\.0710\.267Ministral\-8B\-Instruct\-24100\.4030\.1690\.392Qwen2\.5\-72B\-Instruct0\.8060\.0170\.161Phi\-40\.6980\.0230\.197Llama\-3\.3\-70B\-Instruct0\.7770\.0090\.162Phi\-4\-Mini\-Instruct0\.6470\.0120\.125Llama\-3\.1\-70B\-Instruct0\.7400\.0230\.202Qwen3\-4B\-Instruct0\.5550\.0840\.306Qwen3\-8B\-Instruct0\.5740\.0380\.232covidGemma\-4\-E4B\-IT0\.7060\.0110\.073Qwen3\-32B\-Instruct0\.7670\.0020\.056Llama\-3\.1\-8B\-Instruct0\.7990\.0050\.098Gemma\-4\-31B\-IT0\.7340\.0080\.089Ministral\-8B\-Instruct\-24100\.4400\.1260\.359Qwen2\.5\-72B\-Instruct0\.7290\.0040\.054Phi\-40\.7820\.0020\.057Llama\-3\.3\-70B\-Instruct0\.8820\.0040\.060Phi\-4\-Mini\-Instruct0\.6810\.0040\.071Llama\-3\.1\-70B\-Instruct0\.8880\.0010\.066Qwen3\-4B\-Instruct0\.7840\.0040\.056Qwen3\-8B\-Instruct0\.7880\.0070\.066coalGemma\-4\-E4B\-IT0\.7040\.0230\.106Qwen3\-32B\-Instruct0\.8180\.0220\.133Llama\-3\.1\-8B\-Instruct0\.6590\.0010\.176Gemma\-4\-31B\-IT0\.8550\.0620\.166Ministral\-8B\-Instruct\-24100\.3600\.2290\.428Qwen2\.5\-72B\-Instruct0\.8570\.0030\.100Phi\-40\.7660\.0180\.174Llama\-3\.3\-70B\-Instruct0\.8280\.0060\.123Phi\-4\-Mini\-Instruct0\.7090\.0140\.049Llama\-3\.1\-70B\-Instruct0\.8060\.0020\.118Qwen3\-4B\-Instruct0\.8170\.0020\.106Qwen3\-8B\-Instruct0\.5910\.0420\.120hepar2Gemma\-4\-E4B\-IT0\.6910\.0080\.103Qwen3\-32B\-Instruct0\.7210\.0070\.119Llama\-3\.1\-8B\-Instruct0\.7090\.0020\.151Gemma\-4\-31B\-IT0\.6380\.0210\.163Ministral\-8B\-Instruct\-24100\.4660\.0910\.337Qwen2\.5\-72B\-Instruct0\.7750\.0040\.099Phi\-40\.6830\.0100\.159Llama\-3\.3\-70B\-Instruct0\.8210\.0010\.108Phi\-4\-Mini\-Instruct0\.7760\.0080\.052Llama\-3\.1\-70B\-Instruct0\.7620\.0040\.148Qwen3\-4B\-Instruct0\.6770\.0300\.204Qwen3\-8B\-Instruct0\.6290\.0180\.168munin1Gemma\-4\-E4B\-IT0\.6220\.0290\.142Qwen3\-32B\-Instruct0\.8130\.0030\.107Llama\-3\.1\-8B\-Instruct0\.5510\.0210\.262Gemma\-4\-31B\-IT0\.7180\.0570\.233Ministral\-8B\-Instruct\-24100\.4230\.1410\.371Qwen2\.5\-72B\-Instruct0\.8270\.0020\.125Phi\-40\.7720\.0010\.120Llama\-3\.3\-70B\-Instruct0\.7640\.0450\.101Phi\-4\-Mini\-Instruct0\.8460\.0170\.055Llama\-3\.1\-70B\-Instruct0\.6990\.0020\.170Qwen3\-4B\-Instruct0\.7250\.0040\.136Qwen3\-8B\-Instruct0\.5110\.0550\.148

Table A7:Cross\-prompt agreement calibration results for small \(left\) and large \(right\) LLMs across all datasets\. This method combines predictions across the five prompt styles for each fixed model and dataset\. Bold values indicate the best model within each dataset for each metric \(small and large LLMs evaluated independently\)\.ModelDatasetMethodName\-onlyMetadataCoTFew\-shotFew\-shot \+ CoTAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrow\\endfirsthead*\(continued from previous page\)*ModelDatasetMethodName\-onlyMetadataCoTFew\-shotFew\-shot \+ CoTAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrow\\endhead*\(continued on next page\)*\\endfootPrompt\-specific calibration results for large LLMs onasiam,river,covid,coal,hepar2, andmunin1\. ECE is the 10\-bin squared\-gap measure defined in Section[3\.3](https://arxiv.org/html/2608.23660#S3.SS3)\. Cell colors encode performance percentile within the full table \(yellow = good, orange = bad\), with intensity scaled separately per metric direction\.\\endlastfootQwen3\-32B\-InstructasiamVerb\.0\.2650\.5250\.6370\.2110\.5160\.6030\.4400\.0360\.0970\.3310\.0380\.1540\.3060\.0290\.162Logit0\.6150\.3190\.5310\.6760\.4820\.6540\.7490\.0230\.1250\.6670\.1770\.3840\.7110\.0650\.162riverVerb\.0\.1420\.5120\.6160\.1610\.5150\.6140\.3540\.0550\.2200\.2130\.0490\.2460\.2350\.0850\.270Logit0\.6750\.5300\.6880\.7900\.3780\.5440\.5250\.2180\.4580\.5650\.1010\.3190\.5420\.0800\.300covidVerb\.0\.3760\.7470\.8060\.3240\.7310\.7830\.5710\.0620\.1240\.5000\.0450\.1360\.3680\.0250\.146Logit0\.6070\.7300\.8030\.7440\.6330\.7230\.6540\.0900\.2760\.6140\.0120\.1230\.7100\.0080\.104coalVerb\.0\.5030\.8390\.8650\.0550\.6990\.7300\.1540\.1850\.2840\.0590\.1670\.2910\.1130\.1640\.295Logit0\.6070\.7220\.7760\.6890\.6300\.7130\.6780\.0390\.1950\.6020\.1210\.3520\.6020\.0800\.305hepar2Verb\.0\.1050\.6010\.6640\.0870\.6100\.6650\.3800\.0400\.1650\.1640\.0800\.1800\.1040\.1070\.169Logit0\.6300\.6750\.7750\.6670\.6750\.7780\.5740\.0900\.3030\.5380\.0260\.1660\.6860\.0120\.120munin1Verb\.0\.0460\.8480\.8590\.0330\.7670\.7960\.2160\.1230\.2080\.0180\.1890\.2480\.0720\.1490\.280Logit0\.8180\.7610\.8070\.7040\.7560\.8150\.5130\.1850\.4320\.6480\.0320\.1990\.6810\.0470\.244Gemma\-4\-31B\-ITasiamVerb\.0\.5930\.2410\.3610\.9190\.0580\.1000\.8330\.0030\.0650\.9000\.0010\.0400\.9280\.0260\.076Logit0\.5440\.6310\.7820\.4580\.2450\.4710\.5000\.6550\.8100\.5000\.0020\.0480\.5000\.3540\.595riverVerb\.0\.4030\.4470\.5370\.7030\.1650\.3190\.8330\.0910\.2610\.7870\.0520\.2120\.8330\.0890\.261Logit0\.4550\.2870\.4980\.5310\.1380\.3880\.5000\.7760\.8810\.4970\.0790\.2810\.4870\.6740\.819covidVerb\.0\.7090\.1560\.2350\.8110\.0760\.1460\.9010\.0360\.1100\.8430\.0110\.0760\.9130\.0240\.102Logit0\.4100\.3800\.5710\.5880\.0320\.2400\.5000\.8680\.9320\.5040\.0080\.0850\.4950\.1750\.414coalVerb\.0\.4530\.4810\.5070\.3920\.3840\.5070\.8770\.1370\.2550\.7150\.0160\.1730\.8380\.1170\.269Logit0\.4320\.9210\.9560\.3560\.4080\.5920\.5000\.9480\.9740\.5260\.0480\.2190\.4800\.8070\.897hepar2Verb\.0\.3720\.4860\.5600\.5580\.2920\.3920\.8770\.0730\.1710\.8110\.0110\.1210\.8520\.0640\.177Logit0\.3080\.8490\.9020\.3480\.3960\.5770\.5000\.9500\.9750\.5210\.0240\.1560\.4920\.6450\.803munin1Verb\.0\.0130\.8450\.8550\.2570\.5550\.6520\.8710\.1510\.2870\.5520\.0310\.2400\.6960\.0940\.304Logit0\.4360\.9690\.9790\.4630\.2160\.4650\.5000\.9840\.9920\.5540\.0810\.2880\.4870\.2800\.528Qwen2\.5\-72B\-InstructasiamVerb\.0\.5160\.0020\.1070\.6320\.1010\.1410\.7220\.0030\.1130\.6880\.0050\.1160\.3680\.0340\.107Logit0\.5580\.0160\.1350\.5000\.0020\.0480\.5000\.6550\.8100\.5710\.0310\.1530\.4080\.0290\.113riverVerb\.0\.6620\.0010\.1240\.6700\.0120\.1360\.6140\.0360\.2540\.3680\.0380\.2240\.3600\.0680\.299Logit0\.5380\.0210\.1330\.5600\.0230\.1410\.5000\.7760\.8810\.5030\.0690\.2570\.5060\.1410\.379covidVerb\.0\.7140\.0220\.0660\.7650\.0120\.0590\.6640\.0040\.0570\.7760\.0020\.0900\.6990\.0030\.120Logit0\.5260\.0040\.0490\.5500\.0040\.0490\.5000\.8680\.9320\.5200\.0110\.0970\.6090\.0160\.133coalVerb\.0\.4330\.0700\.0960\.4340\.0260\.0950\.6000\.0030\.1540\.3490\.0330\.1980\.3180\.0620\.280Logit0\.5000\.0010\.0260\.5360\.0030\.0540\.5000\.9480\.9740\.5570\.0480\.2220\.5570\.1000\.325hepar2Verb\.0\.5070\.0150\.0750\.4740\.0250\.0950\.6490\.0010\.1250\.6200\.0040\.1640\.4930\.0070\.193Logit0\.5510\.0030\.0530\.5330\.0040\.0640\.5000\.9500\.9750\.5760\.0380\.1990\.5790\.0450\.221munin1Verb\.0\.6490\.1580\.2010\.2820\.0700\.1270\.5500\.0130\.2120\.2660\.0680\.2290\.1970\.1620\.367Logit0\.5000\.0000\.0080\.6170\.0010\.0340\.5000\.9840\.9920\.5590\.0540\.2330\.5110\.2150\.464Llama\-3\.3\-70B\-InstructasiamVerb\.0\.3310\.3950\.5070\.1450\.5430\.6130\.6960\.0060\.1310\.4970\.0090\.1750\.3670\.0290\.191Logit0\.5250\.0290\.1310\.5980\.0210\.0760\.5000\.0280\.1670\.6080\.0460\.2080\.5560\.0440\.198riverVerb\.0\.3110\.3690\.4760\.2480\.4670\.5700\.5530\.0480\.2810\.3180\.0480\.2310\.3470\.1170\.339Logit0\.5620\.0160\.1260\.6160\.0240\.1240\.5200\.1590\.3950\.5610\.0600\.2590\.5240\.1690\.404covidVerb\.0\.3040\.6070\.6680\.4070\.7530\.8180\.7990\.0090\.0880\.5090\.0190\.1160\.5060\.0060\.150Logit0\.6130\.0080\.0510\.5190\.0040\.0500\.5400\.0120\.1020\.6740\.0120\.0950\.5420\.0310\.172coalVerb\.0\.4700\.9390\.9670\.3400\.4870\.5130\.5800\.0900\.3220\.3680\.0120\.1730\.3010\.2260\.423Logit0\.5000\.0010\.0260\.5250\.0010\.0260\.5060\.2070\.4560\.6570\.0190\.1630\.5010\.2660\.514hepar2Verb\.0\.1640\.5140\.5620\.1400\.6070\.6450\.7480\.0470\.2150\.4330\.0120\.1570\.5230\.0650\.296Logit0\.6290\.0030\.0510\.6570\.0020\.0390\.5210\.0870\.2960\.5990\.0240\.1630\.5140\.1350\.369munin1Verb\.0\.3390\.6880\.6970\.4170\.4640\.4750\.5070\.0880\.3370\.0760\.0750\.0890\.1500\.4900\.594Logit0\.5020\.0000\.0080\.5220\.0000\.0080\.5080\.2180\.4670\.7950\.0010\.0420\.4670\.5020\.700Llama\-3\.1\-70B\-InstructasiamVerb\.0\.3170\.2890\.4480\.2730\.4410\.5190\.6370\.0400\.2290\.6860\.0190\.1900\.4060\.0160\.151Logit0\.7760\.0260\.1690\.7340\.0260\.1090\.5870\.0450\.2230\.7020\.0420\.2010\.6340\.0210\.130riverVerb\.0\.2780\.3610\.5020\.3560\.3220\.4570\.4640\.0600\.2850\.3640\.0570\.2530\.4520\.1440\.385Logit0\.7600\.0070\.1240\.7140\.0060\.1070\.4770\.1250\.3570\.7050\.0500\.2310\.5200\.1420\.383covidVerb\.0\.1910\.6350\.6980\.2870\.6020\.6550\.5760\.0020\.0920\.5160\.0120\.1550\.5240\.0060\.150Logit0\.8650\.0010\.0410\.8120\.0010\.0410\.8830\.0090\.0670\.8170\.0110\.1060\.6940\.0100\.139coalVerb\.0\.4460\.3840\.4180\.2360\.3960\.4530\.4860\.0870\.3250\.3680\.0330\.1780\.3250\.1650\.378Logit0\.5010\.0000\.0270\.7130\.0040\.0280\.5550\.1280\.3730\.8290\.0010\.1050\.5620\.1120\.357hepar2Verb\.0\.1670\.4520\.5380\.2350\.3860\.4610\.6100\.0360\.2560\.4950\.0230\.2190\.5110\.0780\.314Logit0\.8030\.0010\.0640\.8680\.0010\.0450\.6040\.0630\.2830\.7000\.0240\.1970\.6090\.0890\.320munin1Verb\.0\.1120\.4280\.4690\.2010\.3660\.4030\.3860\.2250\.4380\.2010\.1080\.2600\.2210\.4670\.612Logit0\.8690\.0020\.0110\.7860\.0030\.0130\.4460\.3160\.5320\.8420\.0120\.0920\.3560\.4970\.644
Small LLMsLarge LLMsDatasetPromptAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowAUROC↑\\uparrowECE↓\\downarrowBrier↓\\downarrowasiamName\-only0\.7720\.0080\.1350\.6320\.0220\.114Metadata0\.7110\.0140\.1050\.6560\.0120\.054CoT0\.6550\.0240\.1310\.8290\.0080\.099Few\-shot0\.6710\.0120\.2070\.8710\.0180\.095Few\-shot \+ CoT0\.8330\.0190\.1580\.5230\.0160\.100riverName\-only0\.7220\.0030\.1470\.6650\.0090\.136Metadata0\.7110\.0010\.1500\.7260\.0100\.124CoT0\.6540\.0090\.1930\.6310\.0860\.292Few\-shot0\.4780\.0510\.2910\.7080\.0360\.216Few\-shot \+ CoT0\.5660\.0460\.2860\.5260\.1020\.338covidName\-only0\.6200\.0150\.0680\.7740\.0050\.045Metadata0\.7290\.0100\.0560\.7940\.0010\.043CoT0\.7480\.0110\.0570\.9030\.0050\.052Few\-shot0\.7590\.0050\.1110\.8280\.0050\.080Few\-shot \+ CoT0\.7330\.0050\.1230\.6910\.0070\.113coalName\-only0\.5320\.0320\.0580\.5130\.0010\.027Metadata0\.8020\.0140\.0800\.8680\.0020\.062CoT0\.7680\.0110\.1020\.5940\.0290\.228Few\-shot0\.6290\.0140\.2350\.6990\.0100\.159Few\-shot \+ CoT0\.6500\.0270\.2490\.5540\.0770\.315hepar2Name\-only0\.7380\.0080\.0990\.7430\.0030\.071Metadata0\.7570\.0050\.1020\.7900\.0020\.064CoT0\.8000\.0060\.0990\.6890\.0090\.160Few\-shot0\.6870\.0060\.1890\.6950\.0070\.137Few\-shot \+ CoT0\.7010\.0060\.1860\.6910\.0190\.193munin1Name\-only0\.7250\.0370\.0540\.7900\.0110\.025Metadata0\.8080\.0200\.0690\.8780\.0090\.056CoT0\.7980\.0110\.0930\.6270\.0300\.248Few\-shot0\.5230\.0440\.2930\.7600\.0010\.122Few\-shot \+ CoT0\.4920\.1080\.3490\.4670\.1510\.396Table A8:Cross\-model agreement calibration results for small \(left\) and large \(right\) LLMs across all datasets\. This method combines predictions across all models in each group for each fixed prompt style and dataset\. Bold values indicate the best prompt within each dataset for each metric \(small and large LLMs evaluated independently\)\.

### A\.6Statistical Comparison of Confidence Methods

Table[A9](https://arxiv.org/html/2608.23660#A1.T9)reports pairwise comparisons among the four confidence methods\. Tests are performed separately within each model group and metric using the six paired dataset\-level scores\.

Small LLMsLarge LLMsMetricComparisonΔ~\\widetilde\{\\Delta\}RawppHolmppComparisonΔ~\\widetilde\{\\Delta\}RawppHolmppECECM vs\. CP\-0\.0220\.06250\.1875CM vs\. CP0\.0070\.03120\.1875CM vs\. Logit\-0\.1960\.03120\.1875CM vs\. Logit\-0\.1890\.03120\.1875CM vs\. Verb\.\-0\.2070\.03120\.1875CM vs\. Verb\.\-0\.1790\.03120\.1875CP vs\. Logit\-0\.1730\.03120\.1875CP vs\. Logit\-0\.1960\.03120\.1875CP vs\. Verb\.\-0\.1910\.03120\.1875CP vs\. Verb\.\-0\.1790\.03120\.1875Logit vs\. Verb\.\-0\.0010\.68750\.6875Logit vs\. Verb\.0\.0190\.43750\.4375BrierCM vs\. CP\-0\.0290\.03120\.1875CM vs\. CP0\.0080\.09380\.1875CM vs\. Logit\-0\.2000\.03120\.1875CM vs\. Logit\-0\.2160\.03120\.1875CM vs\. Verb\.\-0\.2290\.03120\.1875CM vs\. Verb\.\-0\.1890\.03120\.1875CP vs\. Logit\-0\.1700\.03120\.1875CP vs\. Logit\-0\.2190\.03120\.1875CP vs\. Verb\.\-0\.1990\.03120\.1875CP vs\. Verb\.\-0\.1880\.03120\.1875Logit vs\. Verb\.\-0\.0110\.43750\.4375Logit vs\. Verb\.0\.0320\.15620\.1875AUROCCM vs\. CP0\.0260\.03120\.1875CM vs\. CP\-0\.0540\.03120\.1875CM vs\. Logit0\.0890\.03120\.1875CM vs\. Logit0\.1180\.03120\.1875CM vs\. Verb\.0\.3210\.03120\.1875CM vs\. Verb\.0\.2260\.03120\.1875CP vs\. Logit0\.0590\.06250\.1875CP vs\. Logit0\.1810\.03120\.1875CP vs\. Verb\.0\.3090\.03120\.1875CP vs\. Verb\.0\.2860\.03120\.1875Logit vs\. Verb\.0\.2460\.03120\.1875Logit vs\. Verb\.0\.1310\.03120\.1875

Table A9:Pairwise two\-sided Wilcoxon signed\-rank tests over six paired dataset\-level scores, small \(left\) and large \(right\) LLMs evaluated separately\.Δ~\\widetilde\{\\Delta\}is the median difference \(first method minus second\)\. Holm correction is applied across the six pairwise comparisons within each model group and metric\. CM = cross\-model agreement, CP = cross\-prompt agreement, Verb\. = verbalized confidence\. WithN=6N=6paired observations, the smallest attainable two\-sided rawpp\-value is2/26≈0\.0312/2^\{6\}\\approx 0\.031; no adjusted \(Holm\)pp\-value falls below 0\.05\.
### A\.7Overconfident False Positives on Non\-Edges

\(a\) Confidence buckets among false positives\. Non\-edge typeFalse\-positivepredictionsConf\.<50<50Conf\.5050–7979Conf\.≥80\\geq 80Reversed direct10,6350\.8%14\.7%84\.6%Indirect64,5030\.6%18\.6%80\.8%Other636,3330\.9%20\.7%78\.4%

\(b\) High\-confidence false positives by prompt\. PromptReversed directnon\-edgeIndirectnon\-edgeOthernon\-edgeFew\-shot \+ CoT46\.4%55\.2%42\.5%Few\-shot36\.6%35\.8%25\.0%CoT30\.3%35\.2%21\.5%Metadata21\.8%21\.7%12\.1%Name\-only16\.9%13\.6%9\.5%

Table A10:Overconfident false\-positive summaries on reference\-graph non\-edges\. Panel \(a\) reports confidence distributions using false positives with a parseable verbal confidence score \(99\.9% of all false positives\); its counts therefore differ slightly from the all\-inclusive counts in Table[4](https://arxiv.org/html/2608.23660#S4.T4)\. Panel \(b\) reports high\-confidence false\-positive rates over valid non\-edge queries by prompt and non\-edge type\.Non\-edge typeModelFalse\-positiverateHigh\-conf\. false\-positive rateFalse positiveswith conf\.≥80\\geq 80Reversed directMinistral\-8B\-Instruct\-241058\.6%45\.0%76\.8%IndirectGemma\-4\-31B\-IT56\.3%52\.2%92\.7%OtherMinistral\-8B\-Instruct\-241056\.4%41\.0%72\.7%

Table A11:Worst model\-level high\-confidence false\-positive behavior by non\-edge type\. For each non\-edge category, we report the model with the largest high\-confidence false\-positive rate, along with its overall false\-positive rate and the share of its false positives that have verbal confidence of at least 80\.
### A\.8Additional Classification Figures

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/model_avg_f1_ranking_by_dataset.png)Figure A2:Model classification performance ranking across datasets\. Performance is averaged over five prompt styles\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/dataset_size_avg_f1_by_model.png)Figure A3:Model classification performance as a function of dataset size\. Datasets are ordered by increasing number of variables\. Points show mean F1 averaged over prompt styles, and shaded bands show variation across models within each group\.
### A\.9Reliability Diagram Examples

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/ReiabilityDiagrams/verbalized_best_reliability.png)Figure A4:Reliability diagrams for Qwen2\.5\-72B\-Instruct on Hepar2 with CoT prompting across four confidence sources: \(a\) verbalized, \(b\) logit\-based, \(c\) cross\-prompt, and \(d\) cross\-model\. Each panel reports ECE;⋆\\starmarks the lowest\-ECE source, here verbalized confidence\.
![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/ReiabilityDiagrams/logit_best_reliability.png)Figure A5:Reliability diagrams for Llama\-3\.3\-70B\-Instruct on Munin1 with name\-only prompting across four confidence sources: \(a\) verbalized, \(b\) logit\-based, \(c\) cross\-prompt, and \(d\) cross\-model\. Each panel reports ECE;⋆\\starmarks the lowest\-ECE source, here logit\-based confidence\.
![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/ReiabilityDiagrams/cross_prompt_best_reliability.png)Figure A6:Reliability diagram for cross\-prompt agreement on Munin1 with Phi\-4\.
![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/ReiabilityDiagrams/cross_model_best_reliability.png)Figure A7:Reliability diagram for cross\-model agreement on COVID under metadata prompting\.

### A\.10Additional Confidence Distributions

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/confidence_distribution_correct_incorrect_Qwen2.5-72B-Instruct_chain_of_thought_compact.png)Figure A8:Confidence distributions for Qwen2\.5\-72B\-Instruct under the Chain\-of\-Thought prompt\. Predictions are pooled across all six benchmark datasets\. Blue curves denote correctly classified variable pairs, while orange curves denote misclassified pairs\. Panel \(a\) shows verbalized confidence, and panel \(b\) shows logit\-based confidence\. Both confidence sources concentrate in the high\-confidence region, with logit\-based confidence collapsing almost entirely near 1\.0 for both correct and incorrect predictions, indicating strong overconfidence and limited separation between reliable and unreliable predictions\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/fig_qwen_cot_agreement_confidence.png)Figure A9:Agreement\-based confidence distributions for Qwen2\.5\-72B\-Instruct and the large\-model ensemble\. Panel \(a\) shows cross\-prompt agreement for Qwen2\.5\-72B\-Instruct, while panel \(b\) shows cross\-model agreement under the Chain\-of\-Thought prompt\. Correct predictions concentrate more strongly near full agreement, indicating that agreement\-based confidence is more informative than raw confidence for identifying reliable predictions\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/figure8_gemma_31b_metadata_confidence_distribution_compact.png)Figure A10:Confidence distributions for Gemma\-4\-31B\-IT using the Metadata prompt\.
### A\.11Three\-Way Label\-Space Ablation Results

ModelDatasetP↑\\uparrowR↑\\uparrowF1↑\\uparrownSHD↓\\downarrowNon\-edgeFP Rate↓\\downarrowOrientationError↓\\downarrowA→BA\\\!\\rightarrow\\\!BRecall↑\\uparrowB→AB\\\!\\rightarrow\\\!ARecall↑\\uparrowQwen3\-4BRiver Status0\.2470\.7600\.3730\.30572\.5%0\.0%0\.760–COVID0\.1670\.5000\.2500\.19739\.6%18\.8%0\.8120\.000Hepar20\.0670\.8130\.1240\.29160\.6%5\.7%0\.9090\.000Munin10\.0160\.6370\.0310\.31162\.9%22\.7%0\.8790\.000Phi\-4River Status0\.2870\.9200\.4380\.28171\.3%0\.0%0\.920–COVID0\.4710\.6150\.5330\.07111\.0%5\.9%0\.8120\.300Hepar20\.1020\.6750\.1770\.15932\.0%8\.8%0\.7180\.308Munin10\.0350\.2970\.0620\.07013\.2%17\.3%0\.4040\.013Gemma\-4\-31BRiver Status0\.2930\.8800\.4400\.26766\.2%0\.0%0\.880–COVID0\.3580\.7310\.4810\.10820\.7%0\.0%0\.8120\.600Hepar20\.0790\.7240\.1420\.22045\.5%15\.2%0\.7640\.385Munin10\.0180\.5200\.0350\.22445\.2%30\.4%0\.5560\.427Llama\-3\.3\-70BRiver Status0\.2781\.0000\.4350\.31081\.2%0\.0%1\.000–COVID0\.2420\.6150\.3480\.15330\.5%11\.1%0\.8750\.200Hepar20\.0750\.8460\.1380\.26755\.6%5\.5%0\.9000\.385Munin10\.0130\.7180\.0250\.44891\.0%23\.1%0\.9900\.000

Table A12:Dataset\-level three\-way classification results under Metadata prompting\. Non\-edge FP rate is the fraction of true no\-edge pairs assigned either directed label\. Orientation error is the fraction of detected true\-edge pairs assigned the reversed direction\. A dash indicates that the dataset contains noB→AB\\\!\\rightarrow\\\!Ainstances under the fixed unordered\-pair ordering\.
### A\.12Benchmark Familiarity Audit

Small LLMsLarge LLMsDatasetModelNodesMatchesRecallDev\.RiskModelNodesMatchesRecallDev\.RiskasiamGemma\-4\-E4B\-IT10/720\.2860\.429NoQwen3\-32B\-Instruct7/760\.8570\.000YesLlama\-3\.1\-8B\-Instruct8/710\.1430\.143NoGemma\-4\-31B\-IT7/771\.0000\.000YesMinistral\-8B\-Instruct\-24101/710\.1430\.857NoQwen2\.5\-72B\-Instruct8/771\.0000\.143YesPhi\-48/760\.8570\.143YesLlama\-3\.3\-70B\-Instruct8/771\.0000\.143YesPhi\-4\-Mini\-Instruct5/730\.4290\.286NoLlama\-3\.1\-70B\-Instruct10/771\.0000\.429NoQwen3\-4B\-Instruct2/710\.1430\.714NoQwen3\-8B\-Instruct9/760\.8570\.286NoriverGemma\-4\-E4B\-IT16/1560\.4000\.067NoQwen3\-32B\-Instruct9/1580\.5330\.400NoLlama\-3\.1\-8B\-Instruct11/1530\.2000\.267NoGemma\-4\-31B\-IT10/1590\.6000\.333NoMinistral\-8B\-Instruct\-24109/1570\.4670\.400NoQwen2\.5\-72B\-Instruct8/1560\.4000\.467NoPhi\-48/1520\.1330\.467NoLlama\-3\.3\-70B\-Instruct9/1570\.4670\.400NoPhi\-4\-Mini\-Instruct9/1570\.4670\.400NoLlama\-3\.1\-70B\-Instruct13/1530\.2000\.133NoQwen3\-4B\-Instruct10/1510\.0670\.333NoQwen3\-8B\-Instruct3/1510\.0670\.800NocovidGemma\-4\-E4B\-IT12/2060\.3000\.400NoQwen3\-32B\-Instruct15/2080\.4000\.250NoLlama\-3\.1\-8B\-Instruct14/2010\.0500\.300NoGemma\-4\-31B\-IT6/2030\.1500\.700NoMinistral\-8B\-Instruct\-241013/2070\.3500\.350NoQwen2\.5\-72B\-Instruct12/2020\.1000\.400NoPhi\-413/2020\.1000\.350NoLlama\-3\.3\-70B\-Instruct15/2020\.1000\.250NoPhi\-4\-Mini\-Instruct6/2050\.2500\.700NoLlama\-3\.1\-70B\-Instruct13/20110\.5500\.350NoQwen3\-4B\-Instruct16/2010\.0500\.200NoQwen3\-8B\-Instruct3/2000\.0000\.850NocoalGemma\-4\-E4B\-IT8/3970\.1790\.795NoQwen3\-32B\-Instruct1/3900\.0000\.974NoLlama\-3\.1\-8B\-Instruct10/3970\.1790\.744NoGemma\-4\-31B\-IT9/3900\.0000\.769NoMinistral\-8B\-Instruct\-241011/39100\.2560\.718NoQwen2\.5\-72B\-Instruct10/3970\.1790\.744NoPhi\-47/3960\.1540\.821NoLlama\-3\.3\-70B\-Instruct19/39100\.2560\.513NoPhi\-4\-Mini\-Instruct6/3960\.1540\.846NoLlama\-3\.1\-70B\-Instruct8/3970\.1790\.795NoQwen3\-4B\-Instruct15/3980\.2050\.615NoQwen3\-8B\-Instruct3/3900\.0000\.923Nohepar2Gemma\-4\-E4B\-IT8/7080\.1140\.886NoQwen3\-32B\-Instruct14/70110\.1570\.800NoLlama\-3\.1\-8B\-Instruct15/7050\.0710\.786NoGemma\-4\-31B\-IT13/70120\.1710\.814NoMinistral\-8B\-Instruct\-24101/7000\.0000\.986NoQwen2\.5\-72B\-Instruct13/70120\.1710\.814NoPhi\-413/70130\.1860\.814NoLlama\-3\.3\-70B\-Instruct14/70110\.1570\.800NoPhi\-4\-Mini\-Instruct12/7090\.1290\.829NoLlama\-3\.1\-70B\-Instruct12/70110\.1570\.829NoQwen3\-4B\-Instruct13/70120\.1710\.814NoQwen3\-8B\-Instruct3/7020\.0290\.957Nomunin1Gemma\-4\-E4B\-IT13/18680\.0430\.930NoQwen3\-32B\-Instruct7/18670\.0380\.962NoLlama\-3\.1\-8B\-Instruct14/18680\.0430\.925NoGemma\-4\-31B\-IT12/186120\.0650\.935NoMinistral\-8B\-Instruct\-24109/18690\.0480\.952NoQwen2\.5\-72B\-Instruct15/186140\.0750\.919NoPhi\-47/18670\.0380\.962NoLlama\-3\.3\-70B\-Instruct14/186130\.0700\.925NoPhi\-4\-Mini\-Instruct3/18630\.0160\.984NoLlama\-3\.1\-70B\-Instruct21/186170\.0910\.887NoQwen3\-4B\-Instruct15/186120\.0650\.919NoQwen3\-8B\-Instruct2/18610\.0050\.989No

Table A13:Full node\-level contamination results for all model–dataset pairs, small \(left\) and large \(right\) LLMs evaluated separately\. Nodes reports generated/true counts, and Matches reports semantically aligned nodes\. High\-risk pairs \(node\-count deviation below 15% and node recall above 0\.85\) are highlighted\.
### A\.13Post\-hoc Temperature Calibration

![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/postHocTemperatureCalibration/temperature_sensitivity_munin1.png)Figure A11:Post\-hoc temperature sensitivity of logit\-based calibration on Munin1\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/postHocTemperatureCalibration/temperature_sensitivity_asiam.png)Figure A12:Post\-hoc temperature sensitivity of logit\-based calibration on AsiaM\. AUROC is nearly unchanged because temperature scaling preserves the logit ranking, while Gemma\-31B’s ECE decreases at higher temperatures, indicating overconfident Yes/No logits\. Qwen2\.5\-72B remains stable, suggesting better calibrated logit confidence\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/postHocTemperatureCalibration/temperature_sensitivity_river_status.png)Figure A13:Post\-hoc temperature sensitivity of logit\-based calibration on River Status\. The main effect of temperature appears in ECE rather than AUROC, showing that temperature changes confidence sharpness rather than discrimination\. Gemma\-31B benefits from softer logits, while Qwen2\.5\-72B is comparatively insensitive\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/postHocTemperatureCalibration/temperature_sensitivity_covid.png)Figure A14:Post\-hoc temperature sensitivity of logit\-based calibration on COVID\. AUROC remains stable across temperatures, confirming that prediction ranking is preserved\. ECE varies modestly, with Gemma\-31B showing some calibration benefit from temperature adjustment and Qwen2\.5\-72B remaining consistently well calibrated\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/postHocTemperatureCalibration/temperature_sensitivity_coal_gasifier_risk.png)Figure A15:Post\-hoc temperature sensitivity of logit\-based calibration on Coal Gasifier\. Gemma\-31B shows a clear ECE reduction as temperature increases, suggesting its saved Yes/No logits are too sharp for reliable confidence estimates\. Qwen2\.5\-72B has low and stable ECE, indicating stronger post\-hoc calibration\.![Refer to caption](https://arxiv.org/html/2608.23660v1/sections/imgs/postHocTemperatureCalibration/temperature_sensitivity_hepar2.png)Figure A16:Post\-hoc temperature sensitivity of logit\-based calibration on Hepar2\. Gemma\-31B calibration improves substantially under higher\-temperature softening, while AUROC remains nearly constant\. This supports the project’s finding that logit confidence can be recalibrated without changing causal\-edge predictions\.