语言模型知道什么样的科学证据具有说服力吗？

# 1Introduction
Source: [https://arxiv.org/html/2610.41015](https://arxiv.org/html/2610.41015)
Do Language Models Know What Compelling Scientific Evidence Looks Like?Calibrating Hypothesis Judgment Across Multiple Biological DomainsConor Igoe1, Gabriel Grand1,2, Zane Koch1, Albert Bou1, Eli Moss1, Michaela M\. Hinks1, Samuel G\. Rodriques1, Andrew White1, Siddharth Narayanan1AbstractExpert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance\. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of suchevidence deliberation? We introduceEDB\-Bio, a scientific reasoning benchmark curated from real\-world data in three biological domains: drug\-perturbation transcriptomics, CRISPR screens, and human genome\-wide association studies \(GWAS\)\. Each of EDB\-Bio’s 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true\. Out\-of\-the\-box, open\-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base\-rate forecaster\. Post\-training against strictly proper scoring rules significantly improves both calibration and discrimination\. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is2\.5×2\.5\\timesmore efficient than the best frontier model\. Throughout, quantifying evidence weights asBayes factorsreveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases\. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology\. We release all[code](https://github.com/EdisonScientific/ibis0),[data](https://huggingface.co/datasets/EdisonScientific/EDB-Bio), and[trained models](https://huggingface.co/collections/EdisonScientific/ibis0)\. 1Edison Scientific Inc\., San Francisco, CA, USA2MIT CSAIL, Cambridge, MA, USA Correspondence to\{cigoe,sid\}@@edisonscientific\.com ![[Uncaptioned image]](https://arxiv.org/html/2610.41015v1/edison.png)

## 1Introduction

Experienced scientists are good at weighing evidence\. Shown the result of an experiment, they can often judge whether it is compelling and likely to generalize; or whether it is weak, confounded, or irrelevant to the question being studied\. Yet today, research is increasingly driven by language models\[[1](https://arxiv.org/html/2610.41015#bib.bib112),[2](https://arxiv.org/html/2610.41015#bib.bib113),[3](https://arxiv.org/html/2610.41015#bib.bib114),[4](https://arxiv.org/html/2610.41015#bib.bib70),[5](https://arxiv.org/html/2610.41015#bib.bib118)\]\. How reliably can these models exercise epistemic judgment? This question is particularly important for autonomous workflows in which a model’s output feeds decisions such as which hypothesis to pursue, what experiments to run, or how confidently to report conclusions\.

Prior calibration work has concentrated on general\-purpose domains such as mathematics and commonsense reasoning rather than on empirical scientific evidence\[[6](https://arxiv.org/html/2610.41015#bib.bib44),[7](https://arxiv.org/html/2610.41015#bib.bib45),[8](https://arxiv.org/html/2610.41015#bib.bib29)\]\. Agentic research systems weigh large and heterogeneous bodies of evidence of variable quality, which can be very different from observations in other domains\. To this end, we introduce EvidenceDeliberationBench\-Biology, EDB\-Bio, a benchmark of real biological hypotheses, each paired with a controlled, relevance\-graded pool of experimental evidence\. The model is tasked to predict whether the hypothesis is true or not, conditioned on progressively more evidence from the pool\. Because the evidence is controlled, we can measure both whether a model is right and how its belief tracks the strength and relevance of the evidence\.

On EDB\-Bio, language models can often differentiate more likely hypotheses from less likely ones, yet in all three domains their stated probabilities are poorly calibrated\. Training an open model against a strictly proper scoring rule closes the calibration gap and matches or exceeds frontier discrimination\. A model with reliable probabilistic judgments can alsochoosethe experiment whose result would most change its belief\. In a sequential Bayesian experimental design\[[9](https://arxiv.org/html/2610.41015#bib.bib15)\]setting, our trained Ibis0 model extracts about2\.5×2\.5\\timesmore information per experiment than leading frontier models\.

![Refer to caption](https://arxiv.org/html/2610.41015v1/fig1.png)Figure 1:Evidence deliberation task\.EDB\-Bio asks models to reason about a scientific hypothesishhgiven a sequence of empirical evidence\. Eachevidence stubis derived from published data in one of three biological domains \([Table1](https://arxiv.org/html/2610.41015#S2.T1)\)\. In the example here, each experimentxxis an RNA\-Seq study from Tahoe\-100M\[[10](https://arxiv.org/html/2610.41015#bib.bib11)\]in which a drug perturbs a gene in a particular cell line\. The resultyyreports a DESeq2 assay readout, including aneffect\(log2fold\-change\) and itssignificance\(adjustedpp\-value\)\.Left:In each episode, the hypothesishhand binary support labelθ\\thetaare defined by a held\-outtarget stub, which is not shown to the model\.Right:Givenhh, the model sees a sequence of supporting stubs from studies on related drugs, cell lines, and genes and must verbalize its beliefp⁡\(θ\)p\(\\theta\)after each\. By comparing sequential belief states, we can measure which stubs models find compelling based on the strength of their updates\.### 1\.1Contributions

1. 1\.A novel scientific forecasting task\.We formalize evidence deliberation as forecasting whether a target hypothesis is true or not, given a pool of priorevidence stubsrevealed one at a time\. The task structure uses a domain\-agnostic format, is scored with proper scoring rules, and enables principled analysis of belief revision and sequential experiment selection\.
2. 2\.EDB\-Bio dataset\.We instantiate the task as96,00096\{,\}000directly tested biological hypotheses across three domains: drug\-perturbation transcriptomics \(Tahoe\), CRISPR gene\-dependency screens \(DepMap\), and genome\-wide association studies \(GWAS\)\. Each domain is built using five design principles that ensure class balance/diversity and provide evidence relevance labels based on a tiered expert hierarchy\.
3. 3\.A calibration study of frontier and open models\.Frontier models rank supported hypotheses above unsupported ones, but their probabilities are poorly calibrated\. Post\-training Qwen3\-30B\-A3B against a strictly proper scoring rule yields Ibis0, which closes the calibration gap while outperforming frontier discrimination\.
4. 4\.Sequential analysis of belief revision in LMs\.We quantify theweight of evidenceusing log Bayes factors, which measure how each new stub causally influences the model’s belief about the target hypothesis\. This allows us to test if updates move in the right direction, track expert relevance, have the right magnitude, and are invariant to order\. Our analysis reveals interesting biases; for instance, models show ordering effects and overreact to evidence, making updates larger than the outcomes would justify\.
5. 5\.An EIG estimator for sequential experiment selection\.We use EDB\-Bio to instantiate a Bayesian experimental design setting and introduce a novel factored estimator of expected information gain \(EIG\) that is more sample\-efficient than standard Monte Carlo methods\. Under a well\-calibrated belief model \(Ibis0\), it extracts roughly2\.5×2\.5\\timesmore information per experiment than a prompted frontier model\. We also find that training an LLM tosimulateexperimental outcomes outperforms empirical samples when used for experiment selection with our EIG estimator \([§6](https://arxiv.org/html/2610.41015#S6)\)\.
6. 6\.

We introduce EDB\-Bio in[§2](https://arxiv.org/html/2610.41015#S2)and the models and metrics in[§3](https://arxiv.org/html/2610.41015#S3)\. We then evaluate the models’ ability to forecast target hypotheses’ validity on EDB\-Bio \([§4](https://arxiv.org/html/2610.41015#S4)\)\. Next, we examine how model beliefs update as evidence is progressively disclosed; in particular focusing on prior\-independent analyses \([§5](https://arxiv.org/html/2610.41015#S5)\)\. Finally, we use the models to select which experiment to run next and compare their efficiency against prompted frontier models \([§6](https://arxiv.org/html/2610.41015#S6)\)\.

## 2Evidence Deliberation

Our benchmark models the task ofevidence deliberation, in which a scientist is trying to forecast the likelihood of atarget hypothesisgiven a mixed pool of empirical evidence from prior studies\. Evidence is organized into episodes \([Fig\.1](https://arxiv.org/html/2610.41015#S1.F1)\), which each contain:

- •Target hypothesishh: A statement of a scientific hypothesis, which for each domain follows a consistent format; e\.g\.,DrugDDdownregulates geneGGin cell lineCC\.
- •Labelθ∈\{0,1\}\\theta\\in\\\{0,1\\\}: Whether direct scientific evidence supportshh\(θ=1\\theta=1\) or not \(θ=0\\theta=0\), as determined by a domain\-specific criterion \([Table1](https://arxiv.org/html/2610.41015#S2.T1)\)\.
- •Evidence poolIk=\{\(x1,y1\),…,\(xk,yk\)\}I\_\{k\}=\\\{\(x\_\{1\},y\_\{1\}\),\\ldots,\(x\_\{k\},y\_\{k\}\)\\\}: A variable\-length sequence ofk∈\{1,…,16\}k\\in\\\{1,\\dots,16\\\}tuples each comprised of an experimentxjx\_\{j\}and the outcomeyjy\_\{j\}of conducting the experiment\. Each entry in the sequence is formatted consistently and referred to as a stub\.

#### Model beliefs & weights of evidence\.

The model’s task is to predict the probability that the target hypothesishhis true conditioned on the available evidence, i\.e\.p⁡\(θ=1\|Ik\)p\(\\theta=1\|I\_\{k\}\)\. For notational convenience, we definepk=p⁡\(θ=1∣Ik\)p\_\{k\}=p\(\\theta=1\\mid I\_\{k\}\)\. The evolution ofpkp\_\{k\}as a function ofkkis a useful setting in which to study evidence that language models deem persuasive in favor of or against the target hypothesis\. We also study belief updates using prior\-invariant metrics that isolate the effect of scientific evidence on belief revisions, independent of model priors on EDB\-Bio tasks \([App\.E](https://arxiv.org/html/2610.41015#A5)\)\. In particular, we leverage the*weight of evidence*\[[11](https://arxiv.org/html/2610.41015#bib.bib71)\]of stubkk,wk=logit⁡\(pk\)−logit⁡\(pk−1\)w\_\{k\}=\\operatorname\{logit\}\(p\_\{k\}\)\-\\operatorname\{logit\}\(p\_\{k\-1\}\), wherelogit⁡\(p\)=log⁡p1−p\\operatorname\{logit\}\(p\)=\\log\\tfrac\{p\}\{1\-p\}\. We report it in*decibans*\(dB\),10​log1010\\log\_\{10\}of the Bayes factor\.

### 2\.1EDB\-Bio Dataset

#### Domains\.

The same task format is instantiated in three biological domains: transcriptional responses to drug perturbation in cancer cell lines \(Tahoe\-100M;[10](https://arxiv.org/html/2610.41015#bib.bib11)\), CRISPR gene\-dependency screens\[[12](https://arxiv.org/html/2610.41015#bib.bib16)\], and published genetic associations from the GWAS Catalog\[[13](https://arxiv.org/html/2610.41015#bib.bib19)\]\. The domains differ in what is measured, in the support criterion, and in how related evidence is defined, allowing us to test one form of evidence deliberation across interventional and observational data \([Table1](https://arxiv.org/html/2610.41015#S2.T1)A\)\.

#### Hypothesis and label construction\.

Each hypothesis in EDB\-Bio is derived from an experimental stub \(x∗x^\{\*\},y∗y^\{\*\}\) from the source dataset\. Specifically, the target hypothesis is defined to be the hypothesis thatx∗x^\{\*\}directly tests\. In stating the hypothesishh, we include details aboutx∗x^\{\*\}and how the truth labelθ\\thetais determined as a function ofy∗y^\{\*\}\([Table 1](https://arxiv.org/html/2610.41015#S2.T1)\)\. This minimizes ambiguity about the formulation of the hypothesis and its label\.

#### Design principles\.

Because the source datasets are imbalanced and contain assay\-specific artifacts, every domain is built under five design principles \([Table1](https://arxiv.org/html/2610.41015#S2.T1)B\) that control for these issues: targets with artifact or trivial labels are excluded \(P1\); targets are diversified and balanced50:5050\{:\}50across claim axes \(P2\); evidence pools follow a graded relevance hierarchy defined by domain experts \(P3\) and vary in sufficiency from evidence\-rich to evidence\-poor \(P4\); and the target experiment and outcome are never part of the pool for the corresponding hypothesis \(P5\)\.[App\.A](https://arxiv.org/html/2610.41015#A1)reports domain\-specific construction details, and[App\.I](https://arxiv.org/html/2610.41015#A9)gives a complete prompt from each domain\.

![[Uncaptioned image]](https://arxiv.org/html/2610.41015v1/domains_and_principles.png)Table 1:\(A\)Comparison of the three EDB\-Bio domains: each instantiates the same evidence\-deliberation task with a distinct assay, hypothesis, support criterion, and evidence modality\.\(B\)Five overarching design principles, which are applied to each domain\.
#### Dataset\.

Each domain contains32,00032\{,\}000training,256256validation, and2,0002\{,\}000test episodes, with evidence pools of up tok=16k=16stubs\. We evaluate two test configurations: the*primary set*shows every test episode once at a single context length, and the*sensitivity set*shows a256256\-episode subset at everyk∈\{0,…,16\}k\\in\\\{0,\\dots,16\\\}for the belief revision analyses \([§5](https://arxiv.org/html/2610.41015#S5.SS0.SSS0.Px3)\)\.[SectionA\.6](https://arxiv.org/html/2610.41015#A1.SS6)gives the full configuration and provenance, together with domain\-specific details\.

## 3Methods

### 3\.1Models and training

#### Models and baselines\.

We consider three kinds of LLMs: an open\-weight model evaluated as released, our post\-trained variants of it, and frontier models\. Our starting checkpoint isQwen3\-30B\-A3B, which we refer to throughout asQwen3\. We evaluate Qwen3 in two modes:*non\-thinking*prompts the model to state a probability immediately, while*thinking*generates a reasoning trace before stating the probability\.

#### Calibrating LLM uncertainty\.

We post\-train our models against two objective functions: Brier score and binary cross\-entropy, which are both strictly proper scoring rules\[[14](https://arxiv.org/html/2610.41015#bib.bib14)\]\. This approach is a promising method for calibrating LLM uncertainty\[[6](https://arxiv.org/html/2610.41015#bib.bib44),[7](https://arxiv.org/html/2610.41015#bib.bib45),[15](https://arxiv.org/html/2610.41015#bib.bib43),[16](https://arxiv.org/html/2610.41015#bib.bib28),[8](https://arxiv.org/html/2610.41015#bib.bib29)\], but prior work has applied it mainly to factual question answering and mathematics rather than to scientific judgment \([§7](https://arxiv.org/html/2610.41015#S7)\)\. In total, we present three post\-trained model variants, which we refer to collectively asIbis0:

- •Ibis0 \(non\-thinking\)trains with a group\-relative policy\-gradient objective \([App\.B](https://arxiv.org/html/2610.41015#A2)\) on a negative\-Brier rewardr=1−\(p−θ\)2r=1\-\(p\-\\theta\)^\{2\}, where the model directly states a probabilityppthat the target hypothesis is supported, with no intermediate reasoning\.
- •Ibis0 \(thinking\)trains using the same objective but reasons in text before stating the probability\.
- •Ibis0 \(classifier\)replaces the language\-model output head with a scalar linear head, applies it to the final\-token representation, and minimizes binary cross\-entropy \(BCE\)\.

All models were trained on a multi\-node cluster with 8x NVIDIA B200 GPUs per node usingprime\-rl\[[17](https://arxiv.org/html/2610.41015#bib.bib12)\]\. Training details are provided in[App\.B](https://arxiv.org/html/2610.41015#A2)\.

#### Frontier models\.

As a strong zero\-shot baseline, we evaluate two frontier models:Claude Opus 5\(Anthropic\) andGPT\-5\.6\-Sol\(OpenAI\)\. Both models were fed identical prompts and evaluated at high reasoning effort with a24,00024\{,\}000token budget\. Further details are provided in[App\.D](https://arxiv.org/html/2610.41015#A4)\.

### 3\.2Evaluation metrics

#### Aggregate forecast quality\.

For beliefspi=p⁡\(θi=1∣Ii\)p\_\{i\}=p\(\\theta\_\{i\}=1\\mid I\_\{i\}\)and binary labelsθi∈\{0,1\}\\theta\_\{i\}\\in\\\{0,1\\\}we report the*Brier score*1N​∑i\(pi−θi\)2\\frac\{1\}\{N\}\\sum\_\{i\}\(p\_\{i\}\-\\theta\_\{i\}\)^\{2\}\[[18](https://arxiv.org/html/2610.41015#bib.bib1)\]; lower is better, and because EDB\-Bio is balanced50:5050\{:\}50by construction \(P2\), a constant forecast at the base rate scores0\.250\.25\.

#### Calibration vs\. discrimination\.

We report*sMECE*, a binning\-free calibration error in the smooth\-ECE family of[Błasiok and Nakkiran \[19\]](https://arxiv.org/html/2610.41015#bib.bib6): the density\-weighted mean gap between a stated probability and the empirical frequency of support at that probability \([App\.C](https://arxiv.org/html/2610.41015#A3)\)\.*AUROC*measures whether supported targets are ranked above not\-supported ones, independent of calibration\.

#### Calibration of weights of evidence\.

If a model’s probabilities are calibrated, its weights of evidencewwshould correlate with the labelθ\\theta\. Specifically, across episodes in which the model starts from beliefpk−1p\_\{k\-1\}, an update ofwwdB should change the odds that the target hypothesis turns out to be supported bywwdB, compared with near\-zero updates\. To test this, we bin each model’s updates into five tiers byww\. Each tier’s*odds ratio of support*ψ\\psiis its odds of support divided by the odds after near\-zero updates, in decibans likeww\. It is computed within bins of starting belief, so it does not depend on the model’s prior \([App\.C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px4)\)\. Plottingψ\\psiagainstwwgives a reliability diagram for belief updates \([Fig\.4](https://arxiv.org/html/2610.41015#S5.F4), left\), which we summarize by its slopeβ\\beta: the change in the log\-odds of support per deciban of weight of evidence\. Calibrated updates giveβ=1\\beta=1\. Whenβ<1\\beta<1, the model*overreacts*: its updates are larger than the outcomes justify, and multiplying each update byβ\\betawould calibrate it on average\. Whenβ\>1\\beta\>1, the model*underreacts*, and its updates are too small\. The quantityβ−1\\beta\-1is the log\-odds analog of the coefficient that economists use to measure over\- and underreaction to news, obtained by regressing forecast errors on forecast revisions\[[20](https://arxiv.org/html/2610.41015#bib.bib73),[21](https://arxiv.org/html/2610.41015#bib.bib74)\]\.

All statistics are equal\-weight macro\-averages over the three domains, with95%95\\%intervals from a bootstrap that resamples whole episodes within each domain \([App\.C](https://arxiv.org/html/2610.41015#A3)\)\.

## 4Calibration and discrimination

A useful forecast must rank hypotheses correctly and assign probabilities of the right magnitude, and the two properties need not coincide\.[Fig\.2](https://arxiv.org/html/2610.41015#S4.F2)reports Brier score as the overall measure and then separates calibration from discrimination\.[Fig\.3](https://arxiv.org/html/2610.41015#S4.F3)shows why a well\-ranked forecast can still score poorly: out of the box, models state probabilities biased away from the0\.50\.5base rate and drawn from only a handful of distinct values\.

#### Post\-training outperforms zero\-shot models\.

GPT\-5\.6\-sol and Claude Opus 5 rank supported above not\-supported hypotheses \(AUROC0\.7370\.737and0\.7520\.752\), yet their Brier scores are not significantly better than a base\-rate forecaster\. Both Qwen3 modes show the same mismatch\. All three Ibis0 models obtain Brier below0\.250\.25in every domain\. The classifier has the best Brier score \(0\.1770\.177\) and AUROC \(0\.8200\.820\), whereas the RL policies are the best calibrated, with sMECE0\.0320\.032against0\.0660\.066for the classifier\.[Table6](https://arxiv.org/html/2610.41015#A3.T6)reports per\-domain results with intervals\.

Figure 2:Do language models produce calibrated & accurate beliefs when presented with evidence?Zero\-shot LLMs \(Qwen3, GPT\-5\.6\-sol, and Opus 5\) underperform a constantp=0\.5p=0\.5forecast on Brier score \(dotted line,left\)\. This is primarily due to poor calibration: Models that sit above the diagonal \(middle\) are systematicallyunderconfident, despite demonstrating above\-chance discrimination \(right\)\. Explicitly instructing models to use a prior of 0\.5, matching the base rate, helps somewhat \(dashed bars\), but post\-training on proper\-score objectives \(Ibis0 variants\) improves both calibration and discrimination substantially\.Figure 3:How do language models verbalize beliefs?Histograms of every statedp⁡\(θ\)p\(\\theta\)produced as evidence accrues on the sensitivity set \(N=13,056N=13\{,\}056per model\)\. Out of the box, Qwen3 \(thinking\) and the frontier models are underconfident; post\-training recenters the mean near the0\.50\.5support rate\. Models that verbalize a probability in text produce fewer than100100distinct values, whereas the scalar head of Ibis0 \(classifier\) produces a smooth distribution with1818–50×50\\timesfiner resolution\.
#### Prior\-aware prompting and post\-hoc recalibration\.

We test two methods to improve calibration without post\-training: \(1\) explicitly instructing models to use a prior of 0\.5, and \(2\) fitting a monotone transform post\-hoc\[[22](https://arxiv.org/html/2610.41015#bib.bib7)\]\. We find prompting modestly improves calibration but hurts discrimination \([Fig\.2](https://arxiv.org/html/2610.41015#S4.F2)\)\. Meanwhile, Platt rescaling approaches Ibis0 on calibration, but \(by construction\) does not affect discrimination \([App\.H](https://arxiv.org/html/2610.41015#A8)\)\. However, such post\-hoc recalibration requires access to ground truth labels, so it is not a substitute for training\.

#### Calibration is amortizable without reasoning\.

While the zero\-shot thinking variant of Qwen3 outperforms non\-thinking, after post\-training, this gap disappears\. In fact, Ibis0 \(thinking\) marginally under\-performs non\-thinking, though it does provide better interpretability through its reasoning traces \([App\.J](https://arxiv.org/html/2610.41015#A10)\)\. This aligns with recent work showing that calibration can be effectivelyamortizedin a single forward pass\[[23](https://arxiv.org/html/2610.41015#bib.bib47),[24](https://arxiv.org/html/2610.41015#bib.bib56),[25](https://arxiv.org/html/2610.41015#bib.bib57),[26](https://arxiv.org/html/2610.41015#bib.bib85)\], particularly for tasks that do not require mathematical or symbolic reasoning\[[27](https://arxiv.org/html/2610.41015#bib.bib46)\]\.

#### Cross\-domain transfer\.

To test generalization across domains, we train Ibis0 \(classifier\) on two domains and hold out the third\.[App\.F](https://arxiv.org/html/2610.41015#A6)reports the results of this transfer experiment for each held\-out domain in terms ofretention: the fraction of full\-model performance preserved by the cross\-evaluated classifier\. Brier retention ranges by domain from70\.0%70\.0\\%on Tahoe to26\.3%26\.3\\%on GWAS \([Fig\.11](https://arxiv.org/html/2610.41015#A6.F11)\)\. By further decomposing Brier\[[28](https://arxiv.org/html/2610.41015#bib.bib2)\], we observe that calibration transfers across domains, but discrimination does not\. In other words, the model retains the right probability scale, but not the knowledge needed to rank hypotheses in an unseen domain\.

## 5Belief revision

The model’s stated probabilities alone do not reveal whether a model effectively used the evidence provided or relied on its prior about the target hypothesis\. We answer this question by presenting each episode sequentially at every evidence countk∈\{0,…,16\}k\\in\\\{0,\\dots,16\\\}: the value atk=0k=0measures target\-only prediction \(prior\), and the change in belief askkincreases measures how the model uses the evidence\. Every tested model increases its confidence in the ground\-truth label askkincreases \([App\.E](https://arxiv.org/html/2610.41015#A5)\)\. In the Tahoe and DepMap domains, the Ibis0 classifier has both the highest magnitude of belief revision and the highest ground\-truth probability atk=16k=16\.

Figure 4:Do models weigh each piece of evidence at the right magnitude?Left:Odds ratio of supportψ\\psiagainst weight of evidenceww; calibrated updates lie on the diagonal \(β=1\\beta=1\)\.Middle:the slopeβ\\betaof each curve;β<1\\beta<1indicates overreaction, meaning updates are larger than justified\. Post\-training brings Ibis0 \(non\-thinking\) closest to calibrated\.Right:Evidence weightswwon Tahoe, normalized by episode mean and grouped by biological relevance\. A model that follows the expert hierarchy should show monotonic reduction from left to right\. Instead, all models under\-weight evidence from non\-matching drugs that share a mechanism of action with the target hypothesis \[2\], and all but Ibis0 over\-weight genome\-wide aggregate summaries \[4\], which are only indirectly\-related to the target\.#### Direction and relevance of belief updates\.

Sorting stubs by their direction relative tohh\([App\.C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px5)\), every model’s mean update is positive on supporting stubs, negative on refuting stubs, and small on inconclusive ones \([Fig\.9](https://arxiv.org/html/2610.41015#A5.F9)\)\. Ordering stubs by the curated relevance hierarchy \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\), Ibis0 \(non\-thinking\) aligns most closely in every domain \(Spearmanρ=0\.18\\rho=0\.18–0\.370\.37across domains;[§E\.3](https://arxiv.org/html/2610.41015#A5.SS3)\), despite trailing the classifier on Brier and AUROC \([Fig\.10](https://arxiv.org/html/2610.41015#A5.F10)\)\. Where models disagree with the expert ordering, they do so together: on GWAS every model over\-weights indirect same\-SNP evidence, and on Tahoe, Claude Opus 5 prefers aggregate summaries to specific assay readouts \([Fig\.4](https://arxiv.org/html/2610.41015#S5.F4),right\)\.[SectionE\.2](https://arxiv.org/html/2610.41015#A5.SS2)reports both analyses in full\.

#### Calibration of the weights of evidence\.

As observed in[Fig\.4](https://arxiv.org/html/2610.41015#S5.F4), every model hasβ<1\\beta<1, which indicates overreaction \([§3\.2](https://arxiv.org/html/2610.41015#S3.SS2)\)\. Values range from0\.130\.13for untrained Qwen3 \(non\-thinking\) to0\.890\.89for Ibis0 \(non\-thinking\), the only model whose interval includes11; the frontier models fall between\. Atβ=0\.13\\beta=0\.13, the updates of Qwen3 \(non\-thinking\) are about eight times larger than the outcomes justify\. Ibis0 \(classifier\) has the best Brier score and AUROC of any model \([§4](https://arxiv.org/html/2610.41015#S4)\) but a lowerβ\\beta\(0\.410\.41\) than either RL policy: accurate aggregate forecasts do not imply correctly sized updates\.

#### Order and composition of evidence\.

If stated probabilities behave like beliefs, rearranging a fixed evidence set should not change the result, and the joint weight of a matched supporting and refuting pair should equal the sum of the two single\-stub weights\. We test both properties by holding evidence content fixed while changing its position or composition \([App\.K](https://arxiv.org/html/2610.41015#A11),[Fig\.12](https://arxiv.org/html/2610.41015#A11.F12)\)\. Consistency does not follow from aggregate forecast quality: the classifier has the best Brier score of the Ibis0 models but the largest primacy bias \(\+1\.01\+1\.01dB\) and composition residual \(4\.164\.16dB\), whereas the two RL policies reduce the composition residual of Qwen3 \(non\-thinking\) from7\.547\.54dB to below11dB\. Enabling a reasoning trace reduces primacy bias by about0\.50\.5dB both before and after post\-training\.

## 6Bayesian experimental design

We now turn to a key application of calibrated beliefs: a Bayesian experimental design setting in which the model selectswhich experiment to run next\.

#### Experiment setup\.

We configure EDB\-Bio as a sequential\-selection task \([App\.G](https://arxiv.org/html/2610.41015#A7)\) on512512held\-out episodes in each of the three domains\. Starting from a target hypothesishhand a pool ofk=16k=16unrevealedcandidate stubs𝒞=\{\(x1,\_\),…,\(xk,\_\)\}\\mathcal\{C\}=\\\{\(x\_\{1\},\\\_\),\\ldots,\(x\_\{k\},\\\_\)\\\}, apolicyπ\\pichooses one experimentxtx\_\{t\}at a time to reveal the corresponding outcomeyty\_\{t\}\. We measure performance in two ways:

\(1\) Belief distribution sharpening:An efficient policy should increase the probability assigned to the ground truth label,θ​p​\(θ=1∣I\)\+\(1−θ\)​p​\(θ=0∣I\)\\theta\\,p\(\\theta=1\\mid I\)\+\(1\-\\theta\)\\,p\(\\theta=0\\mid I\), in as few timesteps as possible\.

\(2\) Information gain \(IG\):The reduction ininformation entropy\[[29](https://arxiv.org/html/2610.41015#bib.bib76),[30](https://arxiv.org/html/2610.41015#bib.bib77)\]from experimentxxisℍ⁡\(p⁡\(θ∣I\)\)−ℍ⁡\(p⁡\(θ∣I,x,y\)\)\\mathbb\{H\}\\big\(p\(\\theta\\mid I\)\\big\)\-\\mathbb\{H\}\\big\(p\(\\theta\\mid I,x,y\)\\big\)\. To isolate the effect of experiment selection, we compute IG with the same belief model, Ibis0 \(classifier\), for all policies\.

#### Policies\.

The policies differ in how they use a language model\.

- •Random:πrandom\\pi\_\{\\text\{random\}\}ignores the LM and chooses a uniformly random experiment from𝒞\\mathcal\{C\}\.
- •Naive prompting:πnaive\\pi\_\{\\text\{naive\}\}prompts a model to directly select a candidate based onhh\([§G\.1](https://arxiv.org/html/2610.41015#A7.SS1)\)\.
- •EIG:πEIG\\pi\_\{\\text\{EIG\}\}uses the model’s verbalized beliefp⁡\(θ∣I\)p\(\\theta\\mid I\)to select the experiment that maximizesexpected information gain,EIG​\(x\)=ℍ⁡\(p⁡\(θ∣I\)\)−𝔼y∼p⁡\(y∣x\)​\[ℍ⁡\(p⁡\(θ∣I,x,y\)\)\]\\text\{EIG\}\(x\)=\\mathbb\{H\}\\big\(p\(\\theta\\mid I\)\\big\)\-\\mathbb\{E\}\_\{y\\sim p\(y\\mid x\)\}\\big\[\\mathbb\{H\}\\big\(p\(\\theta\\mid I,x,y\)\\big\)\\big\]\(1\), where the expectation arises from uncertainty over possible outcomesyy\. We compare two approaches to sampling outcomes: \(i\)EIG\-exemplardraws empirical samples from the training set, and \(ii\)EIG\-simulate\(described below\) trains a simulatorp⁡\(y∣x,θ\)p\(y\\mid x,\\theta\)to sample outcomes for each candidate\.

#### Experiment simulator and a factored EIG estimator\.

The standard Monte Carlo estimator for[•‣§6](https://arxiv.org/html/2610.41015#S6.I1.i3)requires asimulatorp⁡\(y∣x,θ\)p\(y\\mid x,\\theta\)that can “imagine” possible outcomes for a candidatexx\. A simulator is easy to train from\(x,y\)\(x,y\)pairs by standard SFT, but conditioning it onθ\\thetarequires counterfactual reasoning about how a different experiment bears on the outcome ofxx\. We instead marginalize over the labelθ′\\theta^\{\\prime\}of the*local*hypothesish′h^\{\\prime\}that the candidatexxdirectly tests, which factors EIG into a supportive and a non\-supportive branch:

EIG⁡\(x\)≈ℍ⁡\(p⁡\(θ∣I\)\)⏟current uncertainty−p⁡\(θ′∣I\)​𝔼​y\+∼p⁡\(y∣x,θ′=1\)\[ℍ⁡\(p⁡\(θ∣I,x,y\+\)\)\]⏟supportive branch−\(1−p⁡\(θ′∣I\)\)​𝔼​y−∼p⁡\(y∣x,θ′=0\)\[ℍ⁡\(p⁡\(θ∣I,x,y−\)\)\]⏟non\-supportive branch\.\\displaystyle\\mathrm\{EIG\}\(x\)\\;\\approx\\;\\underbrace\{\\mathbb\{H\}\\big\(p\(\\theta\\mid I\)\\big\)\}\_\{\\text\{current uncertainty\}\}\-\\underbrace\{\{\\color\[rgb\]\{0,0\.5,1\}p\(\\theta^\{\\prime\}\\mid I\)\}\\;\\mathop\{\\mathbb\{E\}\}\\limits\_\{\\mathrlap\{\\rule\{0\.0pt\}\{5\.72635pt\}y^\{\\scriptscriptstyle\+\}\\sim\\,\{\\color\[rgb\]\{1,0\.49,0\}p\(y\\mid x,\\theta^\{\\prime\}\{=\}1\)\}\}\}\\\!\\big\[\\mathbb\{H\}\\big\(\{\\color\[rgb\]\{0,0\.5,1\}p\(\\theta\\mid I,x,y^\{\\scriptscriptstyle\+\}\)\}\\big\)\\big\]\}\_\{\\text\{supportive branch\}\}\-\\underbrace\{\\big\(1\-\{\\color\[rgb\]\{0,0\.5,1\}p\(\\theta^\{\\prime\}\\mid I\)\}\\big\)\\;\\mathop\{\\mathbb\{E\}\}\\limits\_\{\\mathrlap\{\\rule\{0\.0pt\}\{5\.72635pt\}y^\{\\scriptscriptstyle\-\}\\sim\\,\{\\color\[rgb\]\{1,0\.49,0\}p\(y\\mid x,\\theta^\{\\prime\}\{=\}0\)\}\}\}\\\!\\big\[\\mathbb\{H\}\\big\(\{\\color\[rgb\]\{0,0\.5,1\}p\(\\theta\\mid I,x,y^\{\\scriptscriptstyle\-\}\)\}\\big\)\\big\]\}\_\{\\text\{non\-supportive branch\}\}\.

\(2\)
Ourfactored Monte Carlo EIG estimator \([Eq\.2](https://arxiv.org/html/2610.41015#S6.E2)\)uses two trained models: \(i\) abelief model\(Ibis0\)M≈p⁡\(θ∣I\)M\\\!\\approx\\\!p\(\\theta\\mid I\)that returns the branch weightp⁡\(θ′∣I\)p\(\\theta^\{\\prime\}\\mid I\)and the revised beliefp⁡\(θ∣I,x,y\)p\(\\theta\\mid I,x,y\), and \(ii\) anoutcome simulatorS≈p⁡\(y∣x,θ′\)S\\\!\\approx\\\!p\(y\\mid x,\\theta^\{\\prime\}\)that generates a candidate’s outcomeyyfrom its designxxand the value ofθ′\\theta^\{\\prime\}\. We trainSSby supervised fine\-tuning a separate Qwen3\-30B\-A3B on the EDB\-Bio training split\. The factorization is efficient\. In practice, one outcome sample per branch gives a stable EIG estimate, and the two branches can be evaluated in parallel\. EachEIG⁡\(x\)\\mathrm\{EIG\}\(x\)therefore costs threebelief calls\(the branch weight and the two posteriors\) and twosimulator calls\(one outcome per branch\); the prior entropyℍ⁡\(p⁡\(θ∣I\)\)\\mathbb\{H\}\(p\(\\theta\\mid I\)\)is shared across candidates and cancels in the argmax\. A standard nested Monte Carlo estimate instead draws many outcomes fromp⁡\(y∣x\)p\(y\\mid x\)per candidate and recomputes a posterior for each\.

Figure 5:Experiment selection under the EIG objective\.Left:Belief in the held\-out conclusion,p⁡\(θ=θ∗∣Ik\)p\(\\theta=\\theta^\{\\ast\}\\mid I\_\{k\}\), versus experiments performed, pooled over domains \(±\\pms\.e\.m\.\)\. EIG\-simulate, which uses Ibis0, converges more quickly than both Opus 5 \(under naive prompting\) and a random policy\.Right:Information gained from the first experiment,ℍ⁡\[p⁡\(θ\)\]−ℍ⁡\[p⁡\(θ∣I1\)\]\\mathbb\{H\}\[p\(\\theta\)\]\-\\mathbb\{H\}\[p\(\\theta\\mid I\_\{1\}\)\]\(bits,95%95\\%CI\), pooled over domains\. EIG\-simulate and EIG\-exemplar extract∼2\.5×\{\\sim\}2\.5\\timesthe information of the best prompted model\.
#### Ibis0 \+ EIG extracts more information per experiment\.

[Fig\.5](https://arxiv.org/html/2610.41015#S6.F5)\(right\) shows the entropy reduction for the first experiment selected by each policy, averaged over domains\. EIG\-simulate and EIG\-exemplar reduce0\.1550\.155and0\.1310\.131bits respectively, versus at most0\.0620\.062bits for any naive prompting policy \(Opus 5, GPT\-5\.6\-sol, Qwen3, Ibis0\) and0\.0230\.023for random\. The EIG policies extract roughly2\.5×2\.5\\timesas much information as the best frontier policy, with non\-overlapping95%95\\%CIs\. The advantage is strongest on GWAS and DepMap; on Tahoe, the top information\-gain and frontier policies are statistically tied\. Prompted directly, Ibis0 \(non\-thinking\) does no better than the other naive policies\. In this setting, calibration improved experiment selection only when EIG was estimated and optimized explicitly at test time\.

#### Our approach converges more quickly to the truth\.

On belief in the held\-out conclusion \([Fig\.5](https://arxiv.org/html/2610.41015#S6.F5), left\), EIG\-simulate reaches a given belief in roughly3030–40%40\\%fewer experiments than random and slightly outperforms prompted Opus 5\.

#### A trained simulator matches empirical exemplar outcomes\.

The trained simulator \(EIG\-simulate\) matches or slightly exceeds EIG\-exemplar, which drawsactual outcomes from the training set\. A model can therefore learn to simulate outcomes of unseen experiments well enough to select informative ones\.

## 7Related work

#### Scientific reasoning benchmarks\.

Benchmarks for scientific AI span expert question answering\[[31](https://arxiv.org/html/2610.41015#bib.bib86)\], research coding and data\-driven discovery\[[32](https://arxiv.org/html/2610.41015#bib.bib87),[33](https://arxiv.org/html/2610.41015#bib.bib91),[34](https://arxiv.org/html/2610.41015#bib.bib93)\], simulated environments for scientific reasoning and discovery\[[35](https://arxiv.org/html/2610.41015#bib.bib89),[36](https://arxiv.org/html/2610.41015#bib.bib90)\], and biology research workflows\[[37](https://arxiv.org/html/2610.41015#bib.bib83),[38](https://arxiv.org/html/2610.41015#bib.bib84)\]\. Closest to our task is claim verification against retrieved abstracts\[[39](https://arxiv.org/html/2610.41015#bib.bib78),[40](https://arxiv.org/html/2610.41015#bib.bib92)\], which scores a categorical verdict on existing evidence; EDB\-Bio scores a calibrated probability that a hypothesis will be supported by an experiment whose result is not yet observed\.

#### Calibration and proper\-scoring training\.

Work on LLM calibration has centered on confidence in answer correctness\[[41](https://arxiv.org/html/2610.41015#bib.bib21),[42](https://arxiv.org/html/2610.41015#bib.bib24)\], and alignment tends to leave models overconfident\[[43](https://arxiv.org/html/2610.41015#bib.bib25),[44](https://arxiv.org/html/2610.41015#bib.bib26)\]\. Recent methods post\-train against a proper scoring rule to calibrate that confidence\[[6](https://arxiv.org/html/2610.41015#bib.bib44),[7](https://arxiv.org/html/2610.41015#bib.bib45),[15](https://arxiv.org/html/2610.41015#bib.bib43),[16](https://arxiv.org/html/2610.41015#bib.bib28),[8](https://arxiv.org/html/2610.41015#bib.bib29)\]; we extend these methods to scientific forecasting\.

#### Bayesian experimental design with LLMs\.

Bayesian experimental design selects experiments by expected information gain\[[45](https://arxiv.org/html/2610.41015#bib.bib48),[9](https://arxiv.org/html/2610.41015#bib.bib15)\], whose estimation is costly\[[46](https://arxiv.org/html/2610.41015#bib.bib50),[47](https://arxiv.org/html/2610.41015#bib.bib53)\]\. A growing line drives selection from a language model’s own beliefs, for information gathering\[[48](https://arxiv.org/html/2610.41015#bib.bib59),[49](https://arxiv.org/html/2610.41015#bib.bib61),[50](https://arxiv.org/html/2610.41015#bib.bib58),[51](https://arxiv.org/html/2610.41015#bib.bib68)\]and for planning perturbation screens\[[52](https://arxiv.org/html/2610.41015#bib.bib66),[53](https://arxiv.org/html/2610.41015#bib.bib67)\]\. Such selection presupposes coherent belief updating, yet language models show primacy and anchoring effects\[[54](https://arxiv.org/html/2610.41015#bib.bib94),[55](https://arxiv.org/html/2610.41015#bib.bib95),[56](https://arxiv.org/html/2610.41015#bib.bib96)\]and depart from Bayesian updating on sequential evidence\[[57](https://arxiv.org/html/2610.41015#bib.bib97),[58](https://arxiv.org/html/2610.41015#bib.bib98)\]\. We measure updates as log Bayes factors in decibans\[[11](https://arxiv.org/html/2610.41015#bib.bib71),[59](https://arxiv.org/html/2610.41015#bib.bib102)\]and test whether they over\- or underreact to the evidence\.

## 8Discussion, limitations, and future work

Scientists routinely say that something isknown\. Epistemology and meta\-epidemiology researchers\[[60](https://arxiv.org/html/2610.41015#bib.bib123),[61](https://arxiv.org/html/2610.41015#bib.bib119),[62](https://arxiv.org/html/2610.41015#bib.bib117)\]have fruitfully studied the various methodological, social and institutional forces behind that phrase\. Our work is motivated by a complementary focus, which we term*agent epistemology*: what compels an AI system to shift its belief in a scientific claim? Framing evidence deliberation as calibrated probabilistic forecasting makes this question measurable\. The same approach outlined in this paper could be used to more directly assess methodological notes\[[63](https://arxiv.org/html/2610.41015#bib.bib121)\], absence of detail\[[64](https://arxiv.org/html/2610.41015#bib.bib120)\], or other meta\-scientific information such as conflicts of interest or institutional affiliations\[[65](https://arxiv.org/html/2610.41015#bib.bib122)\], questions that today largely rest on human intuition when weighing scientific evidence\. In contrast, a well\-calibrated model trained across a sufficiently broad scientific literature may help scientists surface systematic credibility issues with studies\.

#### Limitations and future work\.

While constructed from real biological data sources, EDB\-Bio is synthetic in several respects: evidence stubs are structured summaries of the underlying studies, hypotheses are binary, and every domain is deliberately balanced atp⁡\(θ\)=0\.5p\(\\theta\)=0\.5\(P2\), which may account for the cross\-domain calibration transfer \([Section4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px4)\)\. We post\-train one model family \(Qwen3\) at one scale, and the factored EIG estimator, while cheaper than nested Monte Carlo\[[46](https://arxiv.org/html/2610.41015#bib.bib50)\], still costs more than naive prompting\. Finally, we treat a scientific belief as a single probability, but belief in ordinary usage is richer\[[66](https://arxiv.org/html/2610.41015#bib.bib75)\]; we adopt the probabilistic framing because it makes deliberation measurable via Bayesian analysis\. We assume that experiments are uniform and have fixed cost, whereas real research programs face budgets and timelines; extending the selection objective to cost\-aware acquisition is a natural next step, as is moving from structured evidence stubs to free\-text evidence drawn from the literature\.

## AI use statement

This research was carried out by computer scientists and biologists, with the assistance of AI agents and tools\. In this work, we used generative AI tools to clean and reformat datasets, to provide feedback on research methodology and experimental design, to implement methods, and to support qualitative analysis of model reasoning traces\. We have not used generative AI tools to generate data: AI tools helped write the scripts that construct EDB\-Bio, but the data itself is sourced from published scientific studies \([App\.A](https://arxiv.org/html/2610.41015#A1)\)\. Formulating mathematical claims, providing ingredients for or writing proofs, and translation are not applicable to this work\. Additionally, we used generative AI tools to draft and edit parts of the paper for readability, to suggest its structure, to summarize and identify related literature, to create and edit software code, to generate figures with matplotlib, and to draft schematic figures with Claude Design\. The tools used were Claude Code, Claude Design, OpenAI Codex, and the models Claude Opus 4\.8, Claude Opus 5, Claude Opus 5\.5, Claude Sonnet 5, Claude Fable 5\.1, GPT\-5\.6\-sol, and GPT\-6 Astra\.

We have reviewed all AI\-assisted work\. All AI\-drafted text was reviewed and revised by the authors; reported numbers were checked against the evaluation outputs; every citation was checked against its source; and AI\-assisted code was reviewed and tested by the authors\. Because Claude Opus 5 and GPT\-5\.6\-sol are also evaluated as baselines, we note that AI assistance did not produce or alter any reported result\. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI\.

## Ethics statement

This work involves no human subjects and no private or personally identifiable data\. EDB\-Bio is built entirely from public, curated scientific resources used under their respective licenses; the GWAS data are aggregate summary statistics, not individual genotypes\.

Advancing the scientific capabilities of language models is dual\-use: the same calibrated hypothesis evaluation that can accelerate beneficial research could in principle inform harmful research\. However, EDB\-Bio does not include examples of clearly harmful scientific hypotheses, and instead focuses on pre\-existing hypotheses of relevance to cancer therapeutic design\.

Finally, the Ibis0 models described in this work produce probabilistic judgments that remain imperfect and have not been exhaustively evaluated across other areas of life science; they are shared in the spirit of open\-science research, not as a substitute for experimental validation and expert review\.

## Reproducibility statement

We release the artifacts needed to reproduce every result in this paper:

- •
- •
- •
- •Code\.Training configurations, the model\-evaluation harness, and the calibration analyses, in a pinned dependency environment:[github\.com/EdisonScientific/ibis0](https://github.com/EdisonScientific/ibis0)\.
- •

Frontier models were queried through their providers’ APIs with the model identifiers and settings in[App\.D](https://arxiv.org/html/2610.41015#A4)\. Code and model weights are released under the Apache\-2\.0 licence, and the datasets, logs and evaluation outputs under CC BY 4\.0; upstream data sources keep their own terms, listed on the dataset card\.

## References

- \[1\]\(2023\)Autonomous chemical research with large language models\.Nature624\(7992\),pp\. 570–578\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06792-0)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p1.1)\.
- \[2\]C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha\(2024\)The AI scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p1.1)\.
- \[3\]Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. Ha\(2025\)The AI scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.arXiv preprint arXiv:2504\.08066\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p1.1)\.
- \[4\]L\. Mitchener, A\. Yiu, B\. Chang, M\. Bourdenx, T\. Nadolski, A\. Sulovari, E\. C\. Landsness, D\. L\. Barabasi, S\. Narayanan, N\. Evans, S\. Reddy, M\. Foiani, A\. Kamal, L\. P\. Shriver, F\. Cao, A\. T\. Wassie, J\. M\. Laurent, E\. Melville\-Green, M\. Caldas, A\. Bou, K\. F\. Roberts, S\. Zagorac, T\. C\. Orr, M\. E\. Orr, K\. J\. Zwezdaryk, A\. E\. Ghareeb, L\. McCoy, B\. Gomes, E\. A\. Ashley, K\. E\. Duff, T\. Buonassisi, T\. Rainforth, R\. J\. Bateman, M\. Skarlinski, S\. G\. Rodriques, M\. M\. Hinks, and A\. D\. White\(2025\)Kosmos: an AI scientist for autonomous discovery\.arXiv preprint arXiv:2511\.02824\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p1.1)\.
- \[5\]J\. Gottweis, W\. Weng, A\. Daryin, T\. Tu, P\. Sirkovic,et al\.\(2026\)Accelerating scientific discovery with co\-scientist\.Nature655\(8122\),pp\. 487–496\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10644-y)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p1.1)\.
- \[6\]M\. Damani, I\. Puri, S\. Slocum, I\. Shenfeld, L\. Choshen, Y\. Kim, and J\. Andreas\(2026\)Beyond binary rewards: training LMs to reason about their uncertainty\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2507\.16806Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[7\]I\. Puri, M\. Damani, I\. Shenfeld, M\. Ghassemi, J\. Andreas, and Y\. Kim\(2026\)Reaching beyond the mode: RL for distributional reasoning in language models\.arXiv preprint arXiv:2603\.24844\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[8\]T\. Xu, S\. Wu, S\. Diao, X\. Liu, X\. Wang, Y\. Chen, and J\. Gao\(2024\)SaySelf: teaching LLMs to express confidence with self\-reflective rationales\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 5985–5998\.Note:arXiv:2405\.20974External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.343)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[9\]T\. Rainforth, A\. Foster, D\. R\. Ivanova, and F\. Bickford Smith\(2024\)Modern bayesian experimental design\.Statistical Science39\(1\),pp\. 100–114\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.41015#S1.p3.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[10\]J\. Zhang, A\. A\. Ubas, R\. de Borja, V\. Svensson, N\. Thomas, N\. Thakar, I\. Lai,et al\.\(2025\)Tahoe\-100M: a giga\-scale single\-cell perturbation atlas for context\-dependent gene function and cellular modeling\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.02.20.639398)Cited by:[§A\.1](https://arxiv.org/html/2610.41015#A1.SS1.SSS0.Px1.p1.1),[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1),[Figure 1](https://arxiv.org/html/2610.41015#S1.F1),[§2\.1](https://arxiv.org/html/2610.41015#S2.SS1.SSS0.Px1.p1.1)\.
- \[11\]I\. J\. Good\(1950\)Probability and the weighing of evidence\.Charles Griffin\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.41015#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[12\]DepMap, Broad\(2024\)DepMap 24Q4 Public\.Note:Figshare\+ dataset,[https://doi\.org/10\.25452/figshare\.plus\.27993248\.v1](https://doi.org/10.25452/figshare.plus.27993248.v1)Cited by:[§A\.1](https://arxiv.org/html/2610.41015#A1.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2610.41015#S2.SS1.SSS0.Px1.p1.1)\.
- \[13\]E\. Sollis, A\. Mosaku, A\. Abid, A\. Buniello, M\. Cerezo, L\. Gil, T\. Groza, O\. Güneş, P\. Hall, J\. Hayhurst, A\. Ibrahim, Y\. Ji, S\. John, E\. Lewis, J\. A\. L\. MacArthur, A\. McMahon, D\. Osumi\-Sutherland, K\. Panoutsopoulou, Z\. Pendlington, S\. Ramachandran, R\. Stefancsik, J\. Stewart, P\. Whetzel, R\. Wilson, L\. Hindorff, F\. Cunningham, S\. A\. Lambert, M\. Inouye, H\. Parkinson, and L\. W\. Harris\(2023\)The NHGRI\-EBI GWAS Catalog: knowledgebase and deposition resource\.Nucleic Acids Research51\(D1\),pp\. D977–D985\.Cited by:[§A\.1](https://arxiv.org/html/2610.41015#A1.SS1.SSS0.Px3.p1.1),[§2\.1](https://arxiv.org/html/2610.41015#S2.SS1.SSS0.Px1.p1.1)\.
- \[14\]T\. Gneiting and A\. E\. Raftery\(2007\)Strictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.1)\.
- \[15\]D\. Bani\-Harouni, C\. Pellegrini, P\. Stangel, E\. Özsoy, K\. Zaripova, N\. Navab, and M\. Keicher\(2026\)Rewarding doubt: a reinforcement learning approach to calibrated confidence expression of large language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2503\.02623Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[16\]J\. Leng, C\. Huang, B\. Zhu, and J\. Huang\(2025\)Taming overconfidence in LLMs: reward calibration in RLHF\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.09724Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[17\]Prime Intellect\(2025\)PRIME\-RL\.Note:[https://github\.com/PrimeIntellect\-ai/prime\-rl](https://github.com/PrimeIntellect-ai/prime-rl)SoftwareCited by:[Appendix B](https://arxiv.org/html/2610.41015#A2.SS0.SSS0.Px5.p1.1),[§3\.1](https://arxiv.org/html/2610.41015#S3.SS1.SSS0.Px2.p1.2)\.
- \[18\]G\. W\. Brier\(1950\)Verification of forecasts expressed in terms of probability\.Monthly Weather Review78\(1\),pp\. 1–3\.External Links:[Document](https://dx.doi.org/10.1175/1520-0493%281950%29078%253C0001%3AVOFEIT%253E2.0.CO%3B2)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.41015#S3.SS2.SSS0.Px1.p1.1)\.
- \[19\]J\. Błasiok and P\. Nakkiran\(2024\)Smooth ECE: principled reliability diagrams via kernel smoothing\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2309\.12236Cited by:[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px7.p4.1),[§3\.2](https://arxiv.org/html/2610.41015#S3.SS2.SSS0.Px2.p1.1)\.
- \[20\]O\. Coibion and Y\. Gorodnichenko\(2015\)Information rigidity and the expectations formation process: a simple framework and new facts\.American Economic Review105\(8\),pp\. 2644–2678\.Cited by:[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px4.p4.2),[§3\.2](https://arxiv.org/html/2610.41015#S3.SS2.SSS0.Px3.p1.1)\.
- \[21\]P\. Bordalo, N\. Gennaioli, Y\. Ma, and A\. Shleifer\(2020\)Overreaction in macroeconomic expectations\.American Economic Review110\(9\),pp\. 2748–2782\.Cited by:[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px4.p4.2),[§3\.2](https://arxiv.org/html/2610.41015#S3.SS2.SSS0.Px3.p1.1)\.
- \[22\]J\. C\. Platt\(1999\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.InAdvances in Large Margin Classifiers,A\. J\. Smola, P\. Bartlett, B\. Schölkopf, and D\. Schuurmans \(Eds\.\),pp\. 61–74\.Cited by:[Appendix H](https://arxiv.org/html/2610.41015#A8.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px2.p1.1)\.
- \[23\]Y\. Zhao, K\. Y\. Tsang, H\. Vejendla, H\. Shi, Z\. Li, Z\. Hua, Q\. Xu, T\. Zhang, Y\. Wang, L\. Han, B\. A\. Malin, and H\. Wang\(2025\)Dist2ill: distributional distillation for one\-pass uncertainty estimation in large language models\.arXiv preprint arXiv:2505\.11731\.Cited by:[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px3.p1.1)\.
- \[24\]N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. Hutter\(2023\)TabPFN: a transformer that solves small tabular classification problems in a second\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2207\.01848Cited by:[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px3.p1.1)\.
- \[25\]N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter\(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-08328-6)Cited by:[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px3.p1.1)\.
- \[26\]D\. Almeida\(2026\)Introducing System One Models & Jev\.Note:TypeSafe AI,[https://typesafe\.ai/blog/introducing\-system\-one\-models\-and\-jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev)Cited by:[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px3.p1.1)\.
- \[27\]Z\. Sprague, F\. Yin, J\. D\. Rodriguez, D\. Jiang, M\. Wadhwa, P\. Singhal, X\. Zhao, X\. Ye, K\. Mahowald, and G\. Durrett\(2025\)To CoT or not to CoT? Chain\-of\-thought helps mainly on math and symbolic reasoning\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2409\.12183Cited by:[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px3.p1.1)\.
- \[28\]A\. H\. Murphy\(1973\)A new vector partition of the probability score\.Journal of Applied Meteorology12\(4\),pp\. 595–600\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px8.p1.1),[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px4.p1.1)\.
- \[29\]C\. E\. Shannon\(1948\)A mathematical theory of communication\.The Bell System Technical Journal27\(3\),pp\. 379–423\.External Links:[Document](https://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x),ISSN 0005\-8580,[Link](https://ieeexplore.ieee.org/document/6773024)Cited by:[§6](https://arxiv.org/html/2610.41015#S6.SS0.SSS0.Px1.p3.1)\.
- \[30\]D\. J\. C\. MacKay\(1992\)Information\-based objective functions for active data selection\.Neural Computation4\(4\),pp\. 590–604\.External Links:[Document](https://dx.doi.org/10.1162/neco.1992.4.4.590)Cited by:[§6](https://arxiv.org/html/2610.41015#S6.SS0.SSS0.Px1.p3.1)\.
- \[31\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2024\)GPQA: a graduate\-level Google\-proof Q&A benchmark\.InConference on Language Modeling \(COLM\),Note:arXiv:2311\.12022Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[32\]M\. Tian, L\. Gao, S\. D\. Zhang, X\. Chen, C\. Fan, X\. Guo, R\. Haas, P\. Ji, K\. Krongchon, Y\. Li,et al\.\(2024\)SciCode: a research coding benchmark curated by scientists\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2407\.13168Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[33\]B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. D\. Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. Clark\(2025\)DiscoveryBench: towards data\-driven discovery with large language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2407\.01725Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[34\]Z\. Chen, S\. Chen, Y\. Ning, Q\. Zhang, B\. Wang, B\. Yu, Y\. Li, Z\. Liao, C\. Wei, Z\. Lu, V\. Dey, M\. Xue, F\. N\. Baker, B\. Burns, D\. Adu\-Ampratwum, X\. Huang, X\. Ning, S\. Gao, Y\. Su, and H\. Sun\(2025\)ScienceAgentBench: toward rigorous assessment of language agents for data\-driven scientific discovery\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.05080Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[35\]R\. Wang, P\. Jansen, M\. Côté, and P\. Ammanabrolu\(2022\)ScienceWorld: is your agent smarter than a 5th grader?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 11279–11298\.Note:arXiv:2203\.07540External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.775)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[36\]P\. Jansen, M\. Côté, T\. Khot, E\. Bransom, B\. D\. Mishra, B\. P\. Majumder, O\. Tafjord, and P\. Clark\(2024\)DiscoveryWorld: a virtual environment for developing and evaluating automated scientific discovery agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2406\.06769Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[37\]J\. M\. Laurent, A\. Bou, M\. Pieler, C\. Igoe, A\. Andonian, S\. Narayanan, J\. Braza, A\. S\. Vassopoulos, J\. L\. Steenwyk, B\. Lash, A\. D\. White, and S\. G\. Rodriques\(2026\)LABBench2: an improved benchmark for AI systems performing biology research\.arXiv preprint arXiv:2604\.09554\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[38\]Z\. Koch, A\. T\. Wassie, J\. Valdes\-Aleman, J\. Lee, M\. M\. Hinks, S\. G\. Rodriques, A\. D\. White, and J\. M\. Laurent\(2026\)BixBench3: benchmarking AI agents on research\-study\-scale computational biology tasks\.arXiv preprint arXiv:2608\.25286\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[39\]D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. Hajishirzi\(2020\)Fact or fiction: verifying scientific claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7534–7550\.Note:arXiv:2004\.14974External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.609)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[40\]P\. Jansen, S\. Hassan, and R\. Wang\(2025\)Matter\-of\-fact: a benchmark for verifying the feasibility of literature\-supported claims in materials science\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 4090–4102\.Note:arXiv:2506\.04410External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.203)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px1.p1.1)\.
- \[41\]S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[42\]J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. Gurevych\(2024\)A survey of confidence estimation and calibration in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 6577–6595\.Note:arXiv:2311\.08298External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.366)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[43\]OpenAI\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[44\]C\. Zhu, B\. Xu, Q\. Wang, Y\. Zhang, and Z\. Mao\(2023\)On the calibration of large language models and alignment\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 9778–9795\.Note:arXiv:2311\.13240External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.654)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px2.p1.1)\.
- \[45\]D\. V\. Lindley\(1956\)On a measure of the information provided by an experiment\.The Annals of Mathematical Statistics27\(4\),pp\. 986–1005\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[46\]T\. Rainforth, R\. Cornish, H\. Yang, A\. Warrington, and F\. Wood\(2018\)On nesting monte carlo estimators\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),pp\. 4267–4276\.Note:arXiv:1709\.06181Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2610.41015#S8.SS0.SSS0.Px1.p1.1)\.
- \[47\]A\. Foster, D\. R\. Ivanova, I\. Malik, and T\. Rainforth\(2021\)Deep adaptive design: amortizing sequential bayesian experimental design\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),pp\. 3384–3395\.Note:arXiv:2103\.02438Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[48\]Z\. Hu, C\. Liu, X\. Feng, Y\. Zhao, S\. Ng, A\. T\. Luu, J\. He, P\. W\. Koh, and B\. Hooi\(2024\)Uncertainty of thoughts: uncertainty\-aware planning enhances information seeking in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2402\.03271Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[49\]K\. Kobalczyk, N\. Astorga, T\. Liu, and M\. van der Schaar\(2025\)Active task disambiguation with LLMs\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2502\.04485Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[50\]D\. Choudhury, S\. Williamson, A\. Goliński, N\. Miao, F\. Bickford Smith, M\. Kirchhof, Y\. Zhang, and T\. Rainforth\(2026\)BED\-LLM: intelligent information gathering with LLMs and bayesian experimental design\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2508\.21184Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[51\]G\. Grand, V\. Pepe, J\. B\. Tenenbaum, and J\. Andreas\(2026\)Shoot first, ask questions later? building rational agents that explore and act like people\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[52\]K\. Huang, R\. Lopez, J\. Hütter, T\. Kudo, A\. Rios, and A\. Regev\(2024\)Sequential optimal experimental design of perturbation screens guided by multi\-modal priors\.InResearch in Computational Molecular Biology \(RECOMB\),Lecture Notes in Computer Science,pp\. 17–37\.External Links:[Document](https://dx.doi.org/10.1007/978-1-0716-3989-4%5F2)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[53\]M\. Hao, Y\. Lee, H\. Wang, G\. Scalia, and A\. Regev\(2025\)PerTurboAgent: an LLM\-based agent for designing iterative Perturb\-Seq experiments\.InProceedings of Machine Learning in Computational Biology \(MLCB\),PMLR, Vol\.311,pp\. 44–64\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[54\]Y\. Wang, Y\. Cai, M\. Chen, Y\. Liang, and B\. Hooi\(2023\)Primacy effect of ChatGPT\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 108–115\.Note:arXiv:2310\.13206External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.8)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[55\]N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang\(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics \(TACL\)12,pp\. 157–173\.Note:arXiv:2307\.03172External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[56\]J\. Lou and Y\. Sun\(2024\)Anchoring bias in large language models: an experimental study\.arXiv preprint arXiv:2412\.06593\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[57\]R\. Gupta, R\. Corona, J\. Ge, E\. Wang, D\. Klein, T\. Darrell, and D\. M\. Chan\(2025\)Enough coin flips can make LLMs act Bayesian\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 7634–7655\.Note:arXiv:2503\.04722External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.377)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[58\]M\. Kim, S\. Kim, and J\. Thorne\(2025\)From evidence to belief: a Bayesian epistemology approach to language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 10578–10611\.Note:arXiv:2504\.19622External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.531)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[59\]R\. E\. Kass and A\. E\. Raftery\(1995\)Bayes factors\.Journal of the American Statistical Association90\(430\),pp\. 773–795\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1995.10476572)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2610.41015#S7.SS0.SSS0.Px3.p1.1)\.
- \[60\]T\. S\. Kuhn\(1997\)The structure of scientific revolutions\.Vol\.962,University of Chicago press Chicago\.Cited by:[§8](https://arxiv.org/html/2610.41015#S8.p1.1)\.
- \[61\]J\. P\. Ioannidis\(2005\)Why most published research findings are false\.PLoS medicine2\(8\),pp\. e124\.Cited by:[§8](https://arxiv.org/html/2610.41015#S8.p1.1)\.
- \[62\]K\. F\. Schulz, I\. Chalmers, R\. J\. Hayes, and D\. G\. Altman\(1995\)Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials\.JAMA273\(5\),pp\. 408–412\.Cited by:[§8](https://arxiv.org/html/2610.41015#S8.p1.1)\.
- \[63\]J\. A\. Sterne, J\. Savović, M\. J\. Page, R\. G\. Elbers, N\. S\. Blencowe, I\. Boutron, C\. J\. Cates, H\. Cheng, M\. S\. Corbett, S\. M\. Eldridge,et al\.\(2019\)RoB 2: a revised tool for assessing risk of bias in randomised trials\.bmj366\.Cited by:[§8](https://arxiv.org/html/2610.41015#S8.p1.1)\.
- \[64\]A\. Chan, A\. Hróbjartsson, M\. T\. Haahr, P\. C\. Gøtzsche, and D\. G\. Altman\(2004\)Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles\.Jama291\(20\),pp\. 2457–2465\.Cited by:[§8](https://arxiv.org/html/2610.41015#S8.p1.1)\.
- \[65\]A\. Lundh, J\. Lexchin, B\. Mintzes, J\. B\. Schroll, and L\. Bero\(2017\)Industry sponsorship and research outcome\.Cochrane Database of Systematic Reviews2017\(2\),pp\. MR000033\.External Links:[Document](https://dx.doi.org/10.1002/14651858.MR000033.pub3)Cited by:[§8](https://arxiv.org/html/2610.41015#S8.p1.1)\.
- \[66\]J\. Hawthorne, D\. Rothschild, and L\. Spectre\(2016\)Belief is weak\.Philosophical Studies173\(5\),pp\. 1393–1404\.Cited by:[§8](https://arxiv.org/html/2610.41015#S8.SS0.SSS0.Px1.p1.1)\.
- \[67\]J\. M\. Dempster, I\. Boyle, F\. Vazquez, D\. E\. Root, J\. S\. Boehm, W\. C\. Hahn, A\. Tsherniak, and J\. M\. McFarland\(2021\)Chronos: a cell population dynamics model of CRISPR experiments that improves inference of gene fitness effects\.Genome Biology22,pp\. 343\.External Links:[Document](https://dx.doi.org/10.1186/s13059-021-02540-7)Cited by:[§A\.1](https://arxiv.org/html/2610.41015#A1.SS1.SSS0.Px2.p1.1)\.
- \[68\]S\. M\. Corselloet al\.\(2020\)Discovering the anticancer potential of non\-oncology drugs by systematic viability profiling\.Nature Cancer1\(2\),pp\. 235–248\.External Links:[Document](https://dx.doi.org/10.1038/s43018-019-0018-6)Cited by:[§A\.1](https://arxiv.org/html/2610.41015#A1.SS1.SSS0.Px2.p1.1)\.
- \[69\]Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin\(2025\)Understanding R1\-zero\-like training: a critical perspective\.InConference on Language Modeling \(COLM\),Note:arXiv:2503\.20783Cited by:[Appendix B](https://arxiv.org/html/2610.41015#A2.SS0.SSS0.Px5.p1.1)\.
- \[70\]P\. Qi, X\. Zhou, Z\. Liu, T\. Pang, C\. Du, M\. Lin, and W\. S\. Lee\(2026\)Rethinking the trust region in LLM reinforcement learning\.arXiv preprint arXiv:2602\.04879\.Cited by:[Appendix B](https://arxiv.org/html/2610.41015#A2.SS0.SSS0.Px5.p1.1)\.
- \[71\]Kimi Team\(2026\)Kimi K2\.5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[Appendix B](https://arxiv.org/html/2610.41015#A2.SS0.SSS0.Px5.p1.1)\.
- \[72\]N\. Mantel and W\. Haenszel\(1959\)Statistical aspects of the analysis of data from retrospective studies of disease\.Journal of the National Cancer Institute22\(4\),pp\. 719–748\.Cited by:[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px4.p3.1)\.
- \[73\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InInternational Conference on Machine Learning \(ICML\),pp\. 1321–1330\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1),[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px7.p1.1)\.
- \[74\]A\. H\. Murphy and R\. L\. Winkler\(1977\)Reliability of subjective probability forecasts of precipitation and temperature\.Journal of the Royal Statistical Society, Series C \(Applied Statistics\)26\(1\),pp\. 41–47\.External Links:[Document](https://dx.doi.org/10.2307/2346866)Cited by:[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px8.p1.1)\.
- \[75\]M\. H\. DeGroot and S\. E\. Fienberg\(1983\)The comparison and evaluation of forecasters\.Journal of the Royal Statistical Society, Series D \(The Statistician\)32\(1\-2\),pp\. 12–22\.External Links:[Document](https://dx.doi.org/10.2307/2987588)Cited by:[Appendix C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px8.p1.1)\.
- \[76\]V\. Svensson, U\. Khan, H\. Heydari, A\. A\. Ubas, N\. Thomas, D\. Merico, H\. Goodarzi, J\. Yu, N\. Alidoust, and S\. Gandhi\(2026\)Back to basics: observed statistics are sufficient to predict drug responses\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.64898/2026.06.09.731197)Cited by:[Appendix L](https://arxiv.org/html/2610.41015#A12.p1.1)\.
- \[77\]H\. Cui, Z\. Shamsi, G\. Cheon, X\. Ma, S\. Li, M\. Tikhanovskaya, P\. Norgaard, N\. Mudur, M\. Plomecka, P\. Raccuglia,et al\.\(2025\)CURIE: evaluating LLMs on multitask scientific long context understanding and reasoning\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2503\.13517Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1)\.
- \[78\]J\. M\. Laurent, J\. D\. Janizek, M\. Ruzo, M\. M\. Hinks, M\. J\. Hammerling, S\. Narayanan, M\. Ponnapati, A\. D\. White, and S\. G\. Rodriques\(2024\)LAB\-Bench: measuring capabilities of language models for biology research\.arXiv preprint arXiv:2407\.10362\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1)\.
- \[79\]L\. Mitchener, J\. M\. Laurent, A\. Andonian, B\. Tenmann, S\. Narayanan, G\. P\. Wellawatte, A\. White, L\. Sani, and S\. G\. Rodriques\(2025\)BixBench: a comprehensive benchmark for LLM\-based agents in computational biology\.arXiv preprint arXiv:2503\.00096\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1)\.
- \[80\]M\. D\. Skarlinski, S\. Cox, J\. M\. Laurent, J\. D\. Braza, M\. Hinks, M\. J\. Hammerling, M\. Ponnapati, S\. G\. Rodriques, and A\. D\. White\(2024\)Language agents achieve superhuman synthesis of scientific knowledge\.arXiv preprint arXiv:2409\.13740\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1)\.
- \[81\]D\. Wadden, K\. Lo, B\. Kuehl, A\. Cohan, I\. Beltagy, L\. L\. Wang, and H\. Hajishirzi\(2022\)SciFact\-Open: towards open\-domain scientific claim verification\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 4719–4734\.Note:arXiv:2210\.13777External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.347)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px1.p1.1)\.
- \[82\]Z\. Jiang, J\. Araki, H\. Ding, and G\. Neubig\(2021\)How can we know when language models know? on the calibration of language models for question answering\.Transactions of the Association for Computational Linguistics \(TACL\)9,pp\. 962–977\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00407)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[83\]S\. Desai and G\. Durrett\(2020\)Calibration of pre\-trained transformers\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 295–302\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.21)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[84\]N\. Rathi, D\. Jurafsky, and K\. Zhou\(2025\)Humans overrely on overconfident language models, across languages\.InConference on Language Modeling \(COLM\),Note:arXiv:2507\.06306Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[85\]S\. Lin, J\. Hilton, and O\. Evans\(2022\)Teaching models to express their uncertainty in words\.Transactions on Machine Learning Research \(TMLR\)\.Note:arXiv:2205\.14334Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[86\]K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning\(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 5433–5442\.Note:arXiv:2305\.14975External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[87\]J\. Duan, H\. Cheng, S\. Wang, A\. Zavalny, C\. Wang, R\. Xu, B\. Kailkhura, and K\. Xu\(2024\)Shifting attention to relevance: towards the predictive uncertainty quantification of free\-form large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 5050–5063\.Note:arXiv:2307\.01379External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.276)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[88\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2203\.11171Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[89\]J\. Chen and J\. Mueller\(2024\)Quantifying uncertainty in answers from any language model and enhancing their trustworthiness\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 5186–5200\.Note:arXiv:2308\.16175External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.283)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[90\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2302\.09664Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[91\]S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal\(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[92\]D\. Ulmer, M\. Gubri, H\. Lee, S\. Yun, and S\. J\. Oh\(2024\)Calibrating large language models using their generations only\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 15440–15459\.Note:arXiv:2403\.05973External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.824)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[93\]M\. Shen, S\. Das, K\. Greenewald, P\. Sattigeri, G\. Wornell, and S\. Ghosh\(2024\)Thermometer: towards universal calibration for large language models\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),pp\. 44687–44711\.Note:arXiv:2403\.08819Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[94\]Z\. Mei, C\. Zhang, T\. Yin, J\. Lidard, O\. Shorinwa, and A\. Majumdar\(2026\)Reasoning about uncertainty: do reasoning models know when they don’t know?\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 3408–3458\.Note:arXiv:2506\.18183External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.178)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[95\]D\. Halawi, F\. Zhang, C\. Yueh\-Han, and J\. Steinhardt\(2024\)Approaching human\-level forecasting with language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2402\.18563Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[96\]P\. Schoenegger, I\. Tuminauskaite, P\. S\. Park, R\. V\. S\. Bastos, and P\. E\. Tetlock\(2024\)Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy\.Science Advances10\(45\),pp\. eadp1528\.Note:arXiv:2402\.19379External Links:[Document](https://dx.doi.org/10.1126/sciadv.adp1528)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[97\]M\. P\. Naeini, G\. F\. Cooper, and M\. Hauskrecht\(2015\)Obtaining well calibrated probabilities using bayesian binning\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 2901–2907\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px2.p1.1)\.
- \[98\]K\. Chaloner and I\. Verdinelli\(1995\)Bayesian experimental design: a review\.Statistical Science10\(3\),pp\. 273–304\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[99\]A\. Foster, M\. Jankowiak, E\. Bingham, P\. Horsfall, Y\. W\. Teh, T\. Rainforth, and N\. Goodman\(2019\)Variational bayesian optimal experimental design\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:1903\.05480Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[100\]B\. Poole, S\. Ozair, A\. van den Oord, A\. A\. Alemi, and G\. Tucker\(2019\)On variational bounds of mutual information\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),pp\. 5171–5180\.Note:arXiv:1905\.06922Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[101\]T\. Blau, E\. V\. Bonilla, I\. Chades, and A\. Dezfouli\(2022\)Optimizing sequential experimental design with deep reinforcement learning\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),pp\. 2107–2128\.Note:arXiv:2202\.00821Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[102\]S\. Müller, N\. Hollmann, S\. Pineda Arango, J\. Grabocka, and F\. Hutter\(2022\)Transformers can do bayesian inference\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2112\.10510Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[103\]D\. Mazzaccara, A\. Testoni, and R\. Bernardi\(2024\)Learning to ask informative questions: enhancing llms with preference optimization and expected information gain\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 5064–5074\.Note:arXiv:2406\.17453External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.291)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[104\]B\. Z\. Li, A\. Tamkin, N\. Goodman, and J\. Andreas\(2025\)Eliciting human preferences with language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.11589Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[105\]K\. Handa, Y\. Gal, E\. Pavlick, N\. Goodman, J\. Andreas, A\. Tamkin, and B\. Z\. Li\(2024\)Bayesian preference elicitation with language models\.arXiv preprint arXiv:2403\.05534\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[106\]M\. C\. Ramos, S\. S\. Michtavy, A\. D\. White, and M\. D\. Porosoff\(2026\)Bayesian optimization of catalysis with in\-context learning\.ACS Central Science12\(5\),pp\. 599–615\.Note:arXiv:2304\.05341External Links:[Document](https://dx.doi.org/10.1021/acscentsci.5c02418)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[107\]T\. Liu, N\. Astorga, N\. Seedat, and M\. van der Schaar\(2024\)Large language models to enhance bayesian optimization\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2402\.03921Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[108\]S\. S\. Li, V\. Balachandran, S\. Feng, J\. S\. Ilgen, E\. Pierson, P\. W\. Koh, and Y\. Tsvetkov\(2024\)MediQ: question\-asking llms and a benchmark for reliable interactive clinical reasoning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2406\.00922Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px4.p1.1)\.
- \[109\]B\. Raimondi and M\. Gabbrielli\(2025\)Exploiting primacy effect to improve large language models\.InProceedings of the International Conference on Recent Advances in Natural Language Processing \(RANLP\),pp\. 989–997\.Note:arXiv:2507\.13949Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1)\.
- \[110\]Y\. Sumita, K\. Takeuchi, and H\. Kashima\(2024\)Cognitive biases in large language models: a survey and mitigation experiments\.arXiv preprint arXiv:2412\.00323\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1)\.
- \[111\]J\. Zhang, J\. Yang, and K\. Wang\(2025\)Large language models as discounted Bayesian filters\.arXiv preprint arXiv:2512\.18489\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1)\.
- \[112\]I\. J\. Good\(1979\)Studies in the history of probability and statistics\. XXXVII A\. M\. Turing’s statistical work in World War II\.Biometrika66\(2\),pp\. 393–396\.External Links:[Document](https://dx.doi.org/10.1093/biomet/66.2.393)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1)\.
- \[113\]A\. Tversky and D\. J\. Koehler\(1994\)Support theory: a nonextensional representation of subjective probability\.Psychological Review101\(4\),pp\. 547–567\.Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px5.p1.1)\.
- \[114\]Y\. Roohani, K\. Huang, and J\. Leskovec\(2024\)Predicting transcriptional outcomes of novel multigene perturbations with GEARS\.Nature Biotechnology42\(6\),pp\. 927–935\.External Links:[Document](https://dx.doi.org/10.1038/s41587-023-01905-6)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[115\]C\. V\. Theodoris, L\. Xiao, A\. Chopra, M\. D\. Chaffin, Z\. R\. Al Sayed, M\. C\. Hill, H\. Mantineo, E\. M\. Brydon, Z\. Zeng, X\. S\. Liu, and P\. T\. Ellinor\(2023\)Transfer learning enables predictions in network biology\.Nature618\(7965\),pp\. 616–624\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06139-9)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[116\]H\. Cui, C\. Wang, H\. Maan, K\. Pang, F\. Luo, N\. Duan, and B\. Wang\(2024\)scGPT: toward building a foundation model for single\-cell multi\-omics using generative AI\.Nature Methods21\(8\),pp\. 1470–1480\.External Links:[Document](https://dx.doi.org/10.1038/s41592-024-02201-0)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[117\]C\. Ahlmann\-Eltze, W\. Huber, and S\. Anders\(2025\)Deep\-learning\-based gene perturbation effect prediction does not yet outperform simple linear baselines\.Nature Methods22\(8\),pp\. 1657–1661\.External Links:[Document](https://dx.doi.org/10.1038/s41592-025-02772-6)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[118\]Z\. Wei, Y\. Wang, Y\. Gao, S\. Wang, P\. Li, D\. Si, Y\. Gao, S\. Wu, D\. Li, K\. Dong, X\. Yang, C\. Tang, S\. Fu, X\. Chen, W\. Li, Y\. You, C\. Zhang, A\. Liang, G\. Chuai, and Q\. Liu\(2026\)Benchmarking algorithms for generalizable single\-cell perturbation response prediction\.Nature Methods23\(2\),pp\. 451–464\.External Links:[Document](https://dx.doi.org/10.1038/s41592-025-02980-0)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[119\]C\. Bunne, Y\. Roohani, Y\. Rosen, A\. Gupta, X\. Zhang, M\. Roed, T\. Alexandrov, M\. AlQuraishi, P\. Brennan, D\. B\. Burkhardt,et al\.\(2024\)How to build the virtual cell with artificial intelligence: priorities and opportunities\.Cell187\(25\),pp\. 7045–7063\.External Links:[Document](https://dx.doi.org/10.1016/j.cell.2024.11.015)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[120\]Y\. H\. Roohani, T\. J\. Hua, P\. Tung, L\. R\. Bounds, F\. B\. Yu, A\. Dobin, N\. Teyssier, A\. Adduri, A\. Woodrow, B\. S\. Plosky,et al\.\(2025\)Virtual cell challenge: toward a Turing test for the virtual cell\.Cell188\(13\),pp\. 3370–3374\.External Links:[Document](https://dx.doi.org/10.1016/j.cell.2025.06.008)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[121\]A\. K\. Adduri, D\. Gautam, B\. Bevilacqua, A\. Imran, R\. Shah, M\. Naghipourfar, N\. Teyssier, R\. Ilango, S\. Nagaraj, M\. Dong,et al\.\(2025\)Predicting cellular responses to perturbation across diverse contexts with State\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.06.26.661135)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px6.p1.1)\.
- \[122\]K\. Swanson, W\. Wu, N\. L\. Bulaong, J\. E\. Pak, and J\. Zou\(2025\)The virtual lab of AI agents designs new SARS\-CoV\-2 nanobodies\.Nature646\(8085\),pp\. 716–723\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09442-9)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1)\.
- \[123\]A\. E\. Ghareeb, B\. Chang, L\. Mitchener, A\. Yiu, C\. J\. Szostkiewicz, D\. Shved, G\. J\. Gyimesi, J\. M\. Laurent, S\. M\. Wright, M\. T\. Razzak, A\. D\. White, S\. C\. Finnemann, M\. M\. Hinks, and S\. G\. Rodriques\(2026\)A multi\-agent system for automating scientific discovery\.Nature655\(8122\),pp\. 497–505\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10652-y)Cited by:[Appendix M](https://arxiv.org/html/2610.41015#A13.SS0.SSS0.Px7.p1.1)\.

###### Appendix contents

1. [1Introduction](https://arxiv.org/html/2610.41015#S1)1. [1\.1Contributions](https://arxiv.org/html/2610.41015#S1.SS1)
2. [2Evidence Deliberation](https://arxiv.org/html/2610.41015#S2)1. [2\.1EDB\-Bio Dataset](https://arxiv.org/html/2610.41015#S2.SS1)
3. [3Methods](https://arxiv.org/html/2610.41015#S3)1. [3\.1Models and training](https://arxiv.org/html/2610.41015#S3.SS1) 2. [3\.2Evaluation metrics](https://arxiv.org/html/2610.41015#S3.SS2)
4. [4Calibration and discrimination](https://arxiv.org/html/2610.41015#S4)
5. [5Belief revision](https://arxiv.org/html/2610.41015#S5)
6. [6Bayesian experimental design](https://arxiv.org/html/2610.41015#S6)
7. [7Related work](https://arxiv.org/html/2610.41015#S7)
8. [8Discussion, limitations, and future work](https://arxiv.org/html/2610.41015#S8)
9. [References](https://arxiv.org/html/2610.41015#bib)
10. [AEDB\-Bio construction details](https://arxiv.org/html/2610.41015#A1)1. [A\.1Domains](https://arxiv.org/html/2610.41015#A1.SS1) 2. [A\.2Design principles](https://arxiv.org/html/2610.41015#A1.SS2) 3. [A\.3Tahoe: drug perturbation of cancer cell lines](https://arxiv.org/html/2610.41015#A1.SS3) 4. [A\.4DepMap: cancer gene dependency](https://arxiv.org/html/2610.41015#A1.SS4) 5. [A\.5GWAS: genetic associations](https://arxiv.org/html/2610.41015#A1.SS5) 6. [A\.6Dataset splits and test\-set provenance](https://arxiv.org/html/2610.41015#A1.SS6) 7. [A\.7Known limitations of dataset construction](https://arxiv.org/html/2610.41015#A1.SS7)
11. [BModel variants and training details](https://arxiv.org/html/2610.41015#A2)
12. [CEvaluation protocol](https://arxiv.org/html/2610.41015#A3)
13. [DFrontier baseline settings](https://arxiv.org/html/2610.41015#A4)
14. [EEvidence accumulation and per\-update belief revision](https://arxiv.org/html/2610.41015#A5)1. [E\.1Cumulative evidence accumulation](https://arxiv.org/html/2610.41015#A5.SS1) 2. [E\.2Direction of belief revision](https://arxiv.org/html/2610.41015#A5.SS2) 3. [E\.3Evidence by relevance tier](https://arxiv.org/html/2610.41015#A5.SS3)
15. [FFull cross\-domain transfer results](https://arxiv.org/html/2610.41015#A6)
16. [GExperimental design details](https://arxiv.org/html/2610.41015#A7)1. [G\.1Naive\-prompting selection prompt](https://arxiv.org/html/2610.41015#A7.SS1)
17. [HPost\-hoc recalibration](https://arxiv.org/html/2610.41015#A8)
18. [IPrompts: one worked example per domain](https://arxiv.org/html/2610.41015#A9)1. [I\.1Tahoe: drug perturbation of cancer cell lines](https://arxiv.org/html/2610.41015#A9.SS1) 2. [I\.2DepMap: cancer gene dependency](https://arxiv.org/html/2610.41015#A9.SS2) 3. [I\.3GWAS: genetic associations](https://arxiv.org/html/2610.41015#A9.SS3)
19. [JA thinking\-mode reasoning trace](https://arxiv.org/html/2610.41015#A10)1. [J\.1Prompt](https://arxiv.org/html/2610.41015#A10.SS1) 2. [J\.2Trace](https://arxiv.org/html/2610.41015#A10.SS2)
20. [KAdditional analyses of belief revision](https://arxiv.org/html/2610.41015#A11)1. [K\.1Composition of supporting and refuting evidence](https://arxiv.org/html/2610.41015#A11.SS1) 2. [K\.2Dependence of evidence updates on sequence position](https://arxiv.org/html/2610.41015#A11.SS2)
21. [LStructured and numeric\-only baselines on Tahoe](https://arxiv.org/html/2610.41015#A12)
22. [MExtended related work](https://arxiv.org/html/2610.41015#A13)

## Appendix AEDB\-Bio construction details

This appendix expands[§2](https://arxiv.org/html/2610.41015#S2): an overview of the three domains \([§A\.1](https://arxiv.org/html/2610.41015#A1.SS1)\), the design principles applied to all of them \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\), the construction and audit of each domain \([§§A\.3](https://arxiv.org/html/2610.41015#A1.SS3)to[A\.5](https://arxiv.org/html/2610.41015#A1.SS5)\), the dataset splits and test\-set provenance \([§A\.6](https://arxiv.org/html/2610.41015#A1.SS6)\), and a known limitation of the dataset construction \([§A\.7](https://arxiv.org/html/2610.41015#A1.SS7)\)\. Each domain is built by a single seeded script that emits EDB\-Bio together with a manifest recording every parameter; the scripts are released with the code\.

### A\.1Domains

In each domain, the target measurement is excluded from the evidence pool, and the amount, relevance, and strength of the remaining evidence are controlled during construction\. Each stub presents the measurement with its scientific context\. In Tahoe, this is the perturbing drug’s mechanism, approval status, and chemical structure, and the cell line’s lineage and driver mutations; in DepMap and GWAS, it is analogous assay annotations and study provenance, such as gene function, variant consequence, cohort ancestry, and sample size\.[Table1](https://arxiv.org/html/2610.41015#S2.T1)in the main text compares the domains,[§§A\.3](https://arxiv.org/html/2610.41015#A1.SS3)to[A\.5](https://arxiv.org/html/2610.41015#A1.SS5)give construction details, and[App\.I](https://arxiv.org/html/2610.41015#A9)gives a complete prompt from each domain\.

#### Tahoe: drug perturbation of cancer cell lines\.

The Tahoe\-100M atlas\[[10](https://arxiv.org/html/2610.41015#bib.bib11)\]measures transcriptional responses to small\-molecule perturbations across cancer cell lines, supporting the study of drug mechanism and of how response varies across cellular contexts\. A hypothesis states that “drugDDup/down\-regulates geneGGin cell lineCC”; its held\-out DESeq2 measurement supports the claim whenpadj<0\.05\\mathrm\{padj\}<0\.05,\|log2⁡FC\|\>0\.5\|\\log\_\{2\}\\mathrm\{FC\}\|\>0\.5, and the direction matches\. Evidence consists of differential\-expression measurements ofGGunder other drug and cell\-line conditions, ordered by relevance from the same drug in another cell line, through related mechanisms and other drugs, to a genome\-wide aggregate summary and unrelated controls\.

#### DepMap: cancer gene dependency\.

The Broad Institute Cancer Dependency Map \(DepMap\)\[[12](https://arxiv.org/html/2610.41015#bib.bib16)\]uses pooled CRISPR screens to identify genes that cancer cells require for growth, with applications to therapeutic\-target discovery and to mapping context\-specific cancer vulnerabilities\. A hypothesis states that “geneGGis a dependency in cell lineCC”; the held\-out screen supports the hypothesis when the Chronos gene\-effect score is at most−0\.5\-0\.5and the calibrated dependency probability is at least0\.50\.5\[[67](https://arxiv.org/html/2610.41015#bib.bib17)\]\. Evidence is multi\-modal and ordered by how directly it bears onGG’s dependency inCC: the dependency ofGGin related cell lines, then co\-essentiality \(genes whose across\-cell\-line dependency profiles correlate withGG’s, implying shared function\), pharmacological response \(the sensitivity ofCCto drugs that targetGGor its pathway\[[68](https://arxiv.org/html/2610.41015#bib.bib18)\]\), and the molecular state ofGGinCC\(its expression, mutation, and copy\-number status\)\. The dependencies of unrelated genes serve as irrelevant controls\.

#### GWAS: genetic associations\.

The NHGRI\-EBI GWAS Catalog\[[13](https://arxiv.org/html/2610.41015#bib.bib19)\]curates published associations between human genetic variants and traits, providing a resource for mapping inherited risk and trait biology\. A hypothesis states that “SNPSS\(risk alleleAA\) influences traitTTin a stated direction”; the held\-out association supports the hypothesis whenp<5×10−8p<5\\times 10^\{\-8\}and the effect direction matches\. Evidence consists of other catalog associations, ordered by relevance from the same gene or variant and a related trait, through pathway\-neighbor associations, to unrelated loci and traits\.

### A\.2Design principles

The source datasets are imbalanced and contain assay\-specific shortcuts\. We apply five principles to remove shortcuts based on label prevalence or assay artifacts\.[Table2](https://arxiv.org/html/2610.41015#A1.T2)records how each principle is instantiated in each domain; the domain sections below give the corresponding selection rules and audits\.

#### P1: No artifact or trivial labels\.

Target measurements dominated by technical artifacts are excluded, as are entities whose label is nearly constant across contexts\. The restriction applies to targets; excluded entities may still appear as evidence\.

#### P2: Diversity across claim axes\.

Targets span the biological entities and contexts on which we evaluate generalization, with caps or stratification to prevent any one group from dominating\. Each domain is balanced50:5050\{:\}50between supported and not\-supported targets\.

#### P3: A graded relevance hierarchy\.

Evidence pools range from measurements closely related to the target to deliberately unrelated controls\. The ordering within each domain’s hierarchy was defined by PhD\-level bioinformaticians with peer\-reviewed publications in the relevant biological fields, rather than by an automated heuristic\. In Tahoe, for example, the hierarchy runs from the same drug in another cell line, through drugs with the same mechanism and other drugs in the same cell line, to unrelated controls\.

#### P4: Diversity in evidence sufficiency\.

Episodes are divided approximately equally among evidence\-rich, \-moderate, and \-poor strata\. A calibrated model should become confident when evidence is conclusive and remain uncertain when it is not\.

#### P5: No target leakage\.

The held\-out target measurement is excluded from its own evidence pool, and every domain is audited for exact matches\. Domain\-specific near\-duplicate risks, such as variants in high linkage disequilibrium, are discussed in the domain sections below\. Overlap between episodes across splits is a separate matter, treated in[§A\.7](https://arxiv.org/html/2610.41015#A1.SS7)\.

Table 2:How each EDB\-Bio design principle is instantiated in the three domains\. Full criteria and audits are in[§§A\.3](https://arxiv.org/html/2610.41015#A1.SS3)to[A\.5](https://arxiv.org/html/2610.41015#A1.SS5)\.

### A\.3Tahoe: drug perturbation of cancer cell lines

We use the Tahoe\-100M pseudobulk differential\-expression tables, which report one DESeq2 result for each drug×\\timescell\-line×\\timesgene combination and are stored one cell line per file\. Rows with failed DESeq2 fits are discarded, and a result is*significant*iffpadj<0\.05∧\|log2⁡FC\|\>0\.5\\mathrm\{padj\}<0\.05\\wedge\|\\log\_\{2\}\\mathrm\{FC\}\|\>0\.5\. Significant results are rare \(roughly9%9\\%of measurements\)\. Principles P1–P5 \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\) are applied as follows\.

Target principles

P1 \(No artifact labels\)\.Mitochondrial, ribosomal, and pseudogene genes, and measurements with near\-zero base mean, are excluded as targets because their differential\-expression calls reflect technical properties of the assay rather than drug action\. Both remain admissible as evidence\. The realized artifact\-gene target rate is0%0\\%\.

P2 \(Diversity across claim axes\)\.Targets span4949cell lines \(no line contributes more than2\.0%2\.0\\%of episodes\),179179drugs across mechanisms of action, and13,08313\{,\}083genes\. All available cell lines and a shared set of drugs are retained so that every target has same\-drug, different\-cell\-line evidence\. Because significant results are rare, targets are drawn with fixed per\-cell\-line quotas from pre\-classified candidate sets\. A significant measurement yields a supported target \(claimed direction equal to the observed one\) or a refuted target \(opposite direction\); a non\-significant measurement yields an inconclusive target\. The supported rate is0\.500\.50within every cell line and49\.9%49\.9\\%overall\.

Evidence principles

P3 \(Relevance diversity\)\.Unlike those of DepMap and GWAS, the Tahoe hierarchy is*within\-assay*: every stub is a differential\-expression measurement of the target gene under another\(drug,cell line\)\(\\text\{drug\},\\text\{cell line\}\)condition\. We rank a stub’s relevance to the target, most to least, as: \(1\) the same drug in another cell line; \(2\) a drug with the same mechanism of action; \(3\) a different drug in the same cell line; \(4\) a genome\-wide aggregate summary for the target drug and cell line; and \(5\) an unrelated control\. Collapsing \(1\)–\(4\) into*related*and \(5\) into*control*, we fill each pool to an≈60:40\\approx 60\{:\}40ratio, drawing from the top of the hierarchy first\. The realized split is62\.5%62\.5\\%related and37\.5%37\.5\\%control \(same\-drug24%24\\%, same\-mechanism18%18\\%, different\-drug18%18\\%of stubs\), and same\-drug evidence is present in100%100\\%of episodes\. Each stub is tagged with its relation to the target and carries the drug and cell\-line context listed in[§A\.1](https://arxiv.org/html/2610.41015#A1.SS1)\.

P4 \(Sufficiency diversity\)\.Episodes vary in how much conclusive \(significant\) related evidence their pool contains: rich \(≥4\\geq 4significant related stubs\), moderate \(11–33\), or poor \(none\), in equal thirds\. Conclusive related stubs are added from highest to lowest relevance until the stratum’s quota is met; inconclusive related stubs are then added round\-robin across relevance levels, followed by one genome\-wide aggregate summary and unrelated controls\. Rich episodes carry5\.75\.7conclusive stubs on average against0\.80\.8for poor episodes, and88%88\\%of all stubs are inconclusive\.

P5 \(No leakage\)\.A target’s own held\-out measurement never appears in its pool, and no pool stub shares the target’s exact\(drug,cell line\)\(\\text\{drug\},\\text\{cell line\}\)pair; both are enforced by construction\.

#### Class balance and the negative split\.

Supported and not\-supported targets are balanced50:5050\{:\}50\. A Tahoe*refuted*outcome \(a significant change opposite to the claimed direction\) is comparatively common, so negatives are sampled at approximately67%67\\%inconclusive and33%33\\%refuted\.

### A\.4DepMap: cancer gene dependency

Dependencies comprise9\.7%9\.7\\%of valid DepMap gene×\\timescell\-line pairs\. Principles P1–P5 \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\) are applied as follows\.

Target principles

P1 \(No artifact labels\)\.Pan\-essential \(common\-essential\) genes, which are dependencies in nearly every line, and non\-expressed genes, which cannot be dependencies, are excluded as targets\. Both may still appear as \(uninformative\) evidence\.

P2 \(Diversity across claim axes\)\.Targets span the3030lineages and thousands of genes over which we make claims, with no stratum dominating \(at most1616episodes per gene\)\. Unlike Tahoe, DepMap has no target\-drug or mechanism axis: the perturbation is always a knockout, so drugs enter only as evidence\. Because dependencies are rare \(close to Tahoe’s∼9%\\sim\\\!9\\%significant rate\), targets are sampled to obtain a50:5050\{:\}50supported\-to\-not\-supported balance\. Targets are drawn from the2,4802\{,\}480*selective*\-dependency genes, which are essential in some cell lines but not others \(22–50%50\\%of lines\), so that the support label varies across cell\-line contexts\.

Evidence principles

P3 \(Relevance diversity\)\.We rank a stub’s relevance to the target, most to least, as: \(1\) the same gene knocked out in another cell line of the same lineage; \(2\) the same gene in a different lineage; \(3\) a co\-essential pathway partner ofGGknocked out inCC; \(4\) a drug targetingGGor its pathway profiled inCC; \(5\) the molecular state ofGGinCC\(expression, copy number, mutation\); and \(6\) an unrelated gene’s fitness inCC\. Collapsing \(1\)–\(5\) into*related*and \(6\) into*irrelevant*, we fill each pool to an≈60:40\\approx 60\{:\}40ratio, drawing from the top of the hierarchy first and guaranteeing multi\-modal coverage\. This hierarchy is*cross\-modal*, spanning genetic, pharmacological and molecular evidence\. As in Tahoe, each stub is tagged with its relation to the target \(e\.g\.*target gene knocked out in a different cell line of the same lineage*\), which keeps the prompt structure common across domains\. Each stub also carries its context: the gene \(name, family, identifiers\), the cell line \(lineage, disease, subtype, demographics, driver mutations\), and for pharmacological stubs the drug \(targets, mechanism, SMILES\)\.

P4 \(Sufficiency diversity\)\.Episodes vary in how much evidence*supporting*a dependency their pool contains \(related cell lines that are themselves dependencies, over\-expression, amplification, damaging mutation, drug sensitivity\): pools are built to be sufficiency\-rich, \-moderate, or \-poor\.

P5 \(No leakage\)\.A target’s own held\-out CRISPR measurement never appears in its pool; same\-gene stubs are drawn only from*other*cell lines\. The target gene’s*molecular*state inCC\(expression, copy number, mutation\) is measured by a different assay from the held\-out fitness readout and is retained as evidence\.

#### Class balance and the negative split\.

Supported and not\-supported targets are balanced50:5050\{:\}50, as in Tahoe\. DepMap’s*refuted*outcome, a knockout that*speeds*growth, is biologically rare \(only2\.7%2\.7\\%of measurements haveChronos\>0\.25\\mathrm\{Chronos\}\>0\.25\), so negatives are sampled at approximately85%85\\%inconclusive and15%15\\%refuted\.

### A\.5GWAS: genetic associations

The GWAS Catalog is a curated collection of published genome\-wide\-significant associations and contains few natural negatives\. Principles P1–P5 \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\) are applied as follows\.

Target principles

P1 \(No artifact/trivial labels\)\.Variants in the MHC/HLA region \(chr6,∼25\\sim\\\!25–3434Mb\) are excluded as targets because their extreme long\-range linkage disequilibrium makes single\-variant labels unreliable\. Heavily replicated “textbook” pairs \(those studied in dozens of prior GWAS\) are also excluded to reduce dependence on frequently reported associations\. Both classes remain admissible as \(uninformative or trivially informative\) evidence\.

P2 \(Diversity across claim axes\)\.Targets span thousands of genes and traits across the EFO parent categories over which we make claims, with per\-gene*and*per\-trait caps so that no stratum dominates; one episode corresponds to one unique \(SNP, trait\) pair\. Unlike Tahoe, GWAS has no perturbation: it is observational, so the “experiment” is the held\-out association study itself\. Because positives \(genome\-wide\-significant, direction\-consistent associations\) are abundant and negatives scarce, targets are sampled across outcome classes to obtain a50:5050\{:\}50supported\-to\-not\-supported balance\.

Evidence principles

P3 \(Relevance diversity\)\.We rank a stub’s relevance to the target, most to least, as: \(1\) a different SNP in the target gene associated with the same trait; \(2\) a different SNP in the target gene associated with a trait in the same EFO parent \(related trait\); \(3\) the target SNP associated with a different trait in the same EFO parent; \(4\) the target SNP associated with a trait in a different EFO parent; \(5\) a SNP in a pathway\-neighbor gene associated with the target trait; and \(6\) an unrelated SNP and trait \(different gene, different EFO parent\)\. This hierarchy is graded by both*genomic*proximity \(same SNP→\\tosame gene→\\topathway neighbor→\\tounrelated locus\) and*phenotypic*proximity \(same trait→\\torelated trait→\\tounrelated trait\)\. Collapsing \(1\)–\(5\) into*related*and \(6\) into*control*, we fill each pool toward a related:control target of approximately60:4060\{:\}40, drawing from the top of the hierarchy first\. Each stub is tagged with its relation to the target\.

P4 \(Sufficiency diversity\)\.Episodes vary in how much genome\-wide\-significant related evidence their pool contains; pools are built in roughly equal thirds to be sufficiency\-rich, \-moderate, or \-poor by capping the number of significant related stubs\. Because the catalog contains only significant hits, non\-significant*related*evidence is scarce, so the “poor” pools fill largely with unrelated controls and the realized related:control ratio falls below its target\.

P5 \(No leakage\)\.A target’s own held\-out \(SNP, trait\) association never appears in its pool; same\-gene and same\-SNP stubs are drawn only from*other*SNPs or*other*traits, and the exact\-pair exclusion is audited to zero\. This check does not detect variants in high linkage disequilibrium with the target, which may convey nearly the same signal\. Filtering such variants requires an LD reference panel and is not performed\.

#### Class balance and the negative split\.

Supported and not\-supported targets are balanced50:5050\{:\}50, as in Tahoe and DepMap\. A GWAS*direction\-inconsistent*outcome is a genome\-wide\-significant association whose effect points in the opposite direction\. It is rare relative to a*sub\-threshold*association, so negatives are sampled at≈85%\\approx\\\!85\\%sub\-threshold and15%15\\%direction\-inconsistent\. This split concerns the*target*outcome\. At the*evidence*level, GWAS has no refuting\-stub category: once the target association is held out, other catalog associations can corroborate the hypothesis or be uninformative, but do not directly refute it\. The directional analyses therefore omit GWAS \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\)\.

### A\.6Dataset splits and test\-set provenance

#### Dataset splits\.

Each domain contains32,00032\{,\}000training episodes,256256separate validation episodes reserved for model selection, and2,0002\{,\}000test episodes; both test configurations below are drawn from the test split\. Within each domain, labels are balanced approximately50:5050\{:\}50, and evidence order is randomized independently of relevance\. During training, the context length is sampled ask∼Uniform​\{2,…,16\}k\\sim\\mathrm\{Uniform\}\\\{2,\\dots,16\\\}\. The sensitivity set has the same per\-domain size as the validation split but is disjoint from it\.

#### Differences between domains\.

All three domains use evidence pools of at most1616stubs\. They differ in the composition of the not\-supported class: approximately33%33\\%of not\-supported targets are refuted in Tahoe and15%15\\%in DepMap and GWAS \(direction\-inconsistent associations\), with the remainder inconclusive or sub\-threshold\. GWAS also has no refuting*evidence*stubs \([§A\.5](https://arxiv.org/html/2610.41015#A1.SS5)\)\.

#### Test\-set configurations\.

The primary and sensitivity test sets are statically materialized and published as one versioned artifact\. Each row holds one episode with its evidence pool truncated to the specified context length and corresponds to one forward pass\. All models receive identical rendered inputs, including context length and evidence order\. A domain column supports both per\-domain and macro\-averaged analyses from the same tables\.

In the*primary*set, each episode appears once at a context length within the training range\. It is used for the master table, reliability and ROC panels, and leave\-one\-out transfer matrix\. In the*sensitivity*set, each episode appears at every context length from00to1616\. The resulting nested sweep is used for dose–response and weight\-of\-evidence analyses\.

Table 3:Primary and sensitivity test\-set configurations\. Both are published in thetestsets/folder ofEdisonScientific/EDB\-Bioas six configs \(two sets×\\timesthree domains\)\.
#### Primary set\.

The primary set contains2,0002\{,\}000episodes per domain and6,0006\{,\}000in total, comprising the complete held\-out TEST split of each domain\. Each episode is assigned one context lengthk∈\{2,…,16\}k\\in\\\{2,\\dots,16\\\}, matching thecontext\_size\_range=\[2,16\]=\[2,16\]under which every trained model saw its episodes\.

Context lengths are allocated in fixed counts\. Each of the1515values receives⌊2000/15⌋=133\\lfloor 2000/15\\rfloor=133episodes, andk∈\{12,…,16\}k\\in\\\{12,\\dots,16\\\}receives one additional episode, for134134\. Uniform random assignment would have a multinomial standard deviation near1111episodes per group; fixed allocation removes that variation\.

Labels are close to balanced by construction: the supported rate is50\.0%50\.0\\%for both GWAS and DepMap and49\.9%49\.9\\%for Tahoe\.

#### Sensitivity set\.

The sensitivity set contains256256episodes per domain, each drawn from the corresponding primary set and scored at all1717context lengthsk∈\{0,1,…,16\}k\\in\\\{0,1,\\dots,16\\\}:4,3524\{,\}352rows per domain,13,05613\{,\}056per model\. The context atkkis the firstkkstubs of the episode’s fixed, pre\-shuffled evidence pool, so the sweep is nested\. The stub revealed at stepjjis the same for every model and at everyk≥jk\\geq j\. This permits per\-stub attribution in the weight\-of\-evidence analysis, where each increment is a paired difference between consecutive context lengths within an episode\.

Thek=0k=0andk=1k=1conditions are below the training range\. Atk=0k=0, the model receives the hypothesis without evidence, providing the reference forecast used by the cumulative weight\-of\-evidence metric\. Thek=1k=1condition measures the initial response to evidence\.

#### Deterministic construction\.

Within each domain, TEST episodes are ordered by the SHA\-256 hash of the episode identifier\. Context lengths are assigned in balanced blocks over this order, and the first256256episodes form the sensitivity set\. Selection is therefore independent of source\-row order\. Rebuilding with seed00reproduces all six parquet files byte for byte\.

#### Provenance and release\.

Both sets are published in thetestsets/folder ofEdisonScientific/EDB\-Bio, at the same tag \(v1\.0\.0\) as the EDB\-Bio splits they are built from\. Release details for the test sets, builder, and EDB\-Bio are given in the reproducibility statement\.

### A\.7Known limitations of dataset construction

EDB\-Bio assigns episodes to the training, validation, and test splits by i\.i\.d\. sampling \([§A\.6](https://arxiv.org/html/2610.41015#A1.SS6)\), so test targets can overlap with training evidence\. Principle P5 excludes a target’s own measurement from its own evidence pool but does not constrain other episodes: the held\-out target of one episode may serve as an evidence stub in another\. When the two episodes fall on opposite sides of the split, a model trained on EDB\-Bio has seen the test target’s readout during training\. We measure the extent of this overlap and its effect on the reported results, which is small, and leave its removal to a future revision of the dataset\.

#### Extent of the overlap\.

For each test episode we check whether its held\-out measurement, identified by the domain’s target key, appears as an evidence stub within the first1616positions of any training or validation pool, which is the portion of a pool that training renders\.[Table4](https://arxiv.org/html/2610.41015#A1.T4)gives the result\. Overlap affects0\.6%0\.6\\%of test episodes in Tahoe,10\.1%10\.1\\%in DepMap, and53\.2%53\.2\\%in GWAS\. A stricter check, whether a test hypothesis itself recurs as a training target, finds no cases in DepMap or GWAS and1111in Tahoe, all of them one measurement claimed in opposite directions\.

The spread across domains follows from the size of each domain’s measurement space \(the set of possible target keys\) relative to the number of stubs that the training pools draw from it\. Tahoe targets come from a product of179179drugs,4949cell lines, and13,08313\{,\}083genes, about115115million combinations, and the training pools expose0\.39%0\.39\\%of them\. DepMap’s space has two axes and about2\.42\.4million pairs, and the restriction to selective\-dependency genes under P1 concentrates about1616episodes on each of2,0472\{,\}047genes, so the pools expose15%15\\%\. GWAS is not a product space: a \(SNP, trait\) pair exists only if a study reported it, and the relevance tiers enumerate sibling records in the same gene or at the same SNP rather than sampling them\. The training pools cover92\.5%92\.5\\%of the associations that the dataset touches\. Under uniform sampling the overlap rate would equal this coverage, and Tahoe and DepMap fall within a factor of1\.51\.5of it\. The construction choices that limit identity shortcuts, a narrow target set under P1 and relevance\-first pool filling under P3, are the same choices that raise the overlap rate\.

Table 4:Overlap between test targets and training evidence\.*Measurement space*is the number of possible target keys;*exposed as stubs*is the number of distinct measurements that appear as an evidence stub in the first1616positions of any training or validation pool;*coverage*is their ratio;*test overlap*is the number of the2,0002\{,\}000test episodes per domain whose held\-out measurement is among the exposed stubs\.∗GWAS has no product space, so the entry is the number of distinct \(SNP, trait\) associations that appear anywhere in the dataset, and coverage is relative to that set\.
#### Effect on the reported metrics\.

We recompute Brier score, AUROC, and sMECE on the primary test set with the overlapping episodes removed, for all seven models \([Fig\.6](https://arxiv.org/html/2610.41015#A1.F6)\)\. GWAS moves the most, as its overlap rate predicts\. The absolute changes are small: across the three Ibis0 variants, macro\-averaged Brier changes by at most0\.0050\.005and macro\-averaged AUROC by0\.0060\.006\. The shifts are similar in sign and size across models, and no ranking changes\. The zero\-shot models, which were never trained on EDB\-Bio, also score better on the overlapping episodes than on the remainder, and on GWAS their change on removal exceeds that of the Ibis0 RL variants\. The overlapping episodes are therefore easier in their own right, and most of the movement is a selection effect rather than memorization\. The one pattern consistent with memorization is on DepMap, where Brier score improves on the overlapping subset for the trained models and worsens for the untrained ones; it amounts to about0\.0040\.004in the full DepMap Brier score\.

Figure 6:Metrics with and without the overlapping test episodes\.Solid bars use the full2,0002\{,\}000\-episode primary test set in each domain; hatched bars use the filtered dataset, which removes the episodes whose held\-out measurement appears in a training or validation pool \([Table4](https://arxiv.org/html/2610.41015#A1.T4)\)\. Rows are metrics and columns are domains, with the equal\-weight macro\-average on the right\. Labels above a bar pair give the filtered value minus the full value where the difference reaches0\.0050\.005\.
#### Addressing the overlap\.

A split grouped by target entity closes the overlap for Tahoe and DepMap, whose pools are gene\-scoped, but not for GWAS, whose pools cross gene, SNP, and trait\. Alternatively, pools can be built after the split with test measurements excluded, which keeps the in\-distribution framing at the cost of a rebuild\. We intend to adopt one of these in a future revision\.

## Appendix BModel variants and training details

#### Shared setup\.

All post\-training runs start fromQwen3\-30B\-A3B\. The runs comprise one classifier and two RL policies trained on all three domains and three classifiers trained on leave\-one\-domain\-out mixtures \([Table5](https://arxiv.org/html/2610.41015#A2.T5)\)\. They use the same training split, assign equal weight to each included domain, and render each episode withK∼Uniform​\{2,…,16\}K\\sim\\mathrm\{Uniform\}\\\{2,\\dots,16\\\}evidence stubs\.KKis determined from the episode identifier, so a given episode has the same length across runs\.

Optimization is AdamW at learning rate9×10−69\\times 10^\{\-6\}, weight decay0\.010\.01and gradient\-norm clipping at1\.01\.0, with225225linear warmup steps and no subsequent decay\. These settings are shared by all training mixtures, including the leave\-one\-out runs\.

#### Checkpoint selection\.

Each domain reserves a256256\-episode validation split that is separate from the test episodes and from the equally sized sensitivity subset of the test set\. Each run is evaluated on this held\-out split every100100steps \(200200for Ibis0 \(thinking\), whose evaluation dominated its wall\-clock time\) and stops when its selection metric has gone1,0001\{,\}000steps without improving \(minimum improvement10−410^\{\-4\}for the classifiers,10−310^\{\-3\}for RL\)\. No run reached the9,0009\{,\}000\-step safety ceiling\. Five runs stopped on this patience criterion; the GWAS\-held\-out classifier terminated on a checkpoint\-write error799799steps into its patience window, after its selected checkpoint had been written\.

Classifier runs are selected by validation cross\-entropy and RL runs by mean validation reward over three evaluation environments \(256256episodes each for Ibis0 \(non\-thinking\),128128for Ibis0 \(thinking\),44rollouts per episode\)\.[Table5](https://arxiv.org/html/2610.41015#A2.T5)reports the selected checkpoints used in all evaluations\.

RunTraining domainsSelection metricBest valueStepIbis0 \(classifier\)Tahoe, DepMap, GWASin\-domain BCE↓\\downarrow0\.47850\.478524002400Ibis0 \(non\-thinking\)Tahoe, DepMap, GWASeval reward↑\\uparrow0\.83040\.830442004200Ibis0 \(thinking\)Tahoe, DepMap, GWASeval reward↑\\uparrow0\.81680\.816824002400Ibis0 \(classifier\), Tahoe held outDepMap, GWASin\-domain BCE↓\\downarrow0\.43900\.439017001700Ibis0 \(classifier\), DepMap held outTahoe, GWASin\-domain BCE↓\\downarrow0\.47940\.479413001300Ibis0 \(classifier\), GWAS held outTahoe, DepMapin\-domain BCE↓\\downarrow0\.50960\.509618001800Outcome simulator \(SFT\)Tahoe, DepMap, GWASfixed budget–12001200Table 5:Training runs and selected checkpoints\. All runs start fromQwen3\-30B\-A3B\. The Ibis0 runs use the same optimizer settings and differ in objective and training mixture; the outcome simulator used in[§6](https://arxiv.org/html/2610.41015#S6)is trained separately \([App\.B](https://arxiv.org/html/2610.41015#A2)\)\. In\-domain BCE is averaged over the trained domains only, so it is not comparable across rows with different mixtures\.
#### Classifier\.

We replace the language\-modeling head with a single linear unit \(ℝd→ℝ\\mathbb\{R\}^\{d\}\\\!\\to\\\!\\mathbb\{R\}for hidden sizedd\) on the last\-token hidden state, whose sigmoid is the predicted probability of support, and fine\-tune the whole transformer with this head \(no frozen backbone\) under binary cross\-entropy against the support label\.

All classification runs use a sequence length of16,38416\{,\}384tokens, which accommodates the longest render in any domain\. One episode is processed per data\-parallel rank per micro\-step, with1616gradient\-accumulation steps on88ranks, giving128128episodes per optimizer step\. The classifier produces a continuous scalar forecast \([Fig\.3](https://arxiv.org/html/2610.41015#S4.F3)\) rather than a verbalized probability\.

#### Leave\-one\-out classifiers\.

The transfer results in[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px4)come from three additional classification runs that use the same configuration but train on two domains\. All three are evaluated on every domain\. As for the all\-domain classifier, checkpoints are selected by validation cross\-entropy, here macro\-averaged over the two*training*domains and excluding the omitted one\.

#### RL runs\.

The two RL runs useprime\-rl\[[17](https://arxiv.org/html/2610.41015#bib.bib12)\]under the rewardr=1−\(p−y\)2r=1\-\(p\-y\)^\{2\}, wherey∈\{0,1\}y\\in\\\{0,1\\\}is the observed binary support label for the target hypothesis \([§2](https://arxiv.org/html/2610.41015#S2)\) andppis the probability the model states given the revealed evidence\. A completion with no parseable probability receivesr=−1r=\-1\. Advantages are group\-relative: each rollout’s reward minus the mean reward of the rollouts sampled for its prompt, with*no*standard\-deviation normalization\. This is the unbiased, group\-mean\-baselined advantage of Dr\. GRPO\[[69](https://arxiv.org/html/2610.41015#bib.bib8)\]rather than the standardized advantage of GRPO\. Groups whose advantages are all zero are dropped before the trainer sees them\. In place of PPO\-style ratio clipping, the policy is updated withprime\-rl’s default trust\-region objective, the binary total\-variation variant of DPPO\[[70](https://arxiv.org/html/2610.41015#bib.bib9)\]\. It masks a token out of the update when its sampling probability moves by more than0\.20\.2in the trust\-region\-violating direction \(conditioned on the advantage sign\) and leaves the importance ratio unclipped on the kept tokens\. A squared\-log\-ratio KL penalty \(coefficient10−310^\{\-3\}, following Kimi\-K2\.5\[[71](https://arxiv.org/html/2610.41015#bib.bib10)\]\) regularizes the update\. The policies are trained, validated and evaluated at sampling temperature0\.70\.7\.

#### Direct and thinking\.

The two RL runs share objective, reward, data and optimizer, and differ in Qwen3’s thinking mode\. Ibis0 \(non\-thinking\) has thinking disabled and a512512\-token completion budget: the model emits its answer immediately as a<probability\>pp</probability\>tag, with no chain of thought\. It trains at sequence length9,2169\{,\}216, with3232rollouts per optimizer step and88rollouts per example\. Ibis0 \(thinking\) has thinking enabled and a24,57624\{,\}576\-token budget, which forces sequence length32,76832\{,\}768; it trains with1616rollouts per step and44per example\. The reward reads only the stated probability in both cases\.

#### Use in subsequent analyses\.

The classifier is included in the weight\-of\-evidence and dose–response analyses\. In the experimental\-design evaluation \([§6](https://arxiv.org/html/2610.41015#S6)\), the belief model is the all\-domain classifier \(step24002400\), and Ibis0 \(non\-thinking\) is the Ibis0 naive\-prompting policy\.

#### Outcome simulator\.

The simulatorSSof[§6](https://arxiv.org/html/2610.41015#S6)is a supervised fine\-tune ofQwen3\-30B\-A3Bwith thinking disabled \(last row of[Table5](https://arxiv.org/html/2610.41015#A2.T5)\)\. Its training data are180,000180\{,\}000real \(experiment, outcome\) pairs from the EDB\-Bio training split, drawn from GWAS associations, DepMap CRISPR screens and Tahoe evidence stubs and balanced by valence\. Each prompt gives an experiment with its result hidden, the hypothesisθ′\\theta^\{\\prime\}that the experiment tests, and a requested valence; the target is the real outcome, labelled supporting or null by the sign of the weight of evidence the classifier assigns it forθ′\\theta^\{\\prime\}\. No outcome is synthetic\. Training uses AdamW at learning rate10−510^\{\-5\}, batch size6464, sequence length12,28812\{,\}288and loss on the completion only, for a fixed budget of1,2001\{,\}200steps \(validation loss0\.2870\.287at step1,1001\{,\}100, the last evaluation\)\. On100100held\-out test\-split experiments per domain, decoded greedily, the supporting outcome carries more weight of evidence forθ′\\theta^\{\\prime\}than the null outcome for the same experiment in83%83\\%of cases\.

## Appendix CEvaluation protocol

This appendix specifies the estimators, resampling procedures, and decoding rules used for the metrics defined in[§3\.2](https://arxiv.org/html/2610.41015#S3.SS2)\.

#### Macro\-averaging\.

Each aggregate statistic is computed separately within Tahoe, DepMap, and GWAS and then averaged with equal weight\. This procedure prevents domain\-specific metric distributions from changing the effective domain weights\.

If a statistic is undefined for a resampled domain, as can occur for AUROC when a resample contains one class, that domain is omitted from the resampled macro\-average\. This behavior is implemented for the macro bootstrap and transfer\-matrix statistic; the master\-table implementation raises an error\. No such case occurs in the2,0002\{,\}000\-episode point estimates\.

#### Within\-domain stratified bootstrap\.

Each bootstrap draw resamples episodes with replacement within each domain, holding the per\-domain count fixed, and recomputes the per\-domain statistics and their macro\-average\. For the aggregate results on the primary set \(the master table, headline figure, grouped\-metric panels, transfer matrix and ROC AUROC table\), intervals are the2\.52\.5th and97\.597\.5th percentiles over2,0002\{,\}000draws\. Other figures use fewer draws:1,0001\{,\}000for the dose–response and weight\-of\-evidence curves,500500for the per\-bin reliability and ROC bands, and250250for the per\-position and signed\-verdict intervals\. Point estimates are computed on the complete evaluation sample\.

Resampling within domains keeps domain weights fixed across draws\. Pooling episodes before resampling would instead estimate uncertainty for a micro\-average with variable domain proportions\.

The sMECE estimator isO⁡\(n2\)O\(n^\{2\}\)in the number of episodes and costs about3535times as much per draw as the other aggregate metrics, so its intervals use200200draws\. Its point estimate uses the complete sample\.

#### Episode\-level resampling\.

Each episode contributes one forecast to the aggregate metrics and up to1616updates to the per\-stub analyses\. Updates from the same episode share a target hypothesis, an evidence pool and an ordering, so they are strongly correlated\. The bootstrap for per\-stub quantities is therefore*episode\-clustered*: it samples complete episodes with all of their stubs, preserving within\-episode dependence\.

Relevance\-hierarchy quantities are macro\-averaged over domains using episode clustering\. Signed weight\-of\-evidence results are reported separately by domain because verdict categories differ across domains; their intervals use within\-domain episode\-clustered bootstraps\.

#### Calibration of weights of evidence\.

This paragraph gives the estimators of the odds ratio of supportψ\\psiand the slopeβ\\beta\([§3\.2](https://arxiv.org/html/2610.41015#S3.SS2)\), which Analysis 2 of[§5](https://arxiv.org/html/2610.41015#S5)reports\. The unit of analysis is a single belief update\. On the sensitivity set each model states a probabilitypkp\_\{k\}for every episode at each evidence countk∈\{0,…,16\}k\\in\\\{0,\\dots,16\\\}, and each consecutive pair of forecasts gives one update\(ℓi,wi,θi\)\(\\ell\_\{i\},w\_\{i\},\\theta\_\{i\}\): the incoming beliefℓi=logit⁡\(pk−1\)\\ell\_\{i\}=\\operatorname\{logit\}\(p\_\{k\-1\}\), the weight of evidence

wi=10ln⁡10​\[logit⁡\(pk\)−logit⁡\(pk−1\)\]w\_\{i\}=\\frac\{10\}\{\\ln 10\}\\,\\big\[\\operatorname\{logit\}\(p\_\{k\}\)\-\\operatorname\{logit\}\(p\_\{k\-1\}\)\\big\]in decibans \([§2](https://arxiv.org/html/2610.41015#S2)\), and the labelθi∈\{0,1\}\\theta\_\{i\}\\in\\\{0,1\\\}of the episode, which all of its updates share\. Each model contributes256×16256\\times 16updates per domain\. Probabilities are clipped to\[10−3,1−10−3\]\[10^\{\-3\},\\,1\-10^\{\-3\}\]before the logit\. Updates withwi=0w\_\{i\}=0, where the stated probability does not change, are excluded\. They range from1%1\\%of updates for Ibis0 \(classifier\) to68%68\\%for Ibis0 \(non\-thinking\); models that state probabilities in text often repeat the same value \([§4](https://arxiv.org/html/2610.41015#S4)\)\.

Tiers and belief strata\.Separately for each model and domain, the remaining updates are cut into five quantile bins ofww, the*tiers*, and five quantile bins ofℓ\\ell, the*belief strata*; tied quantile edges are merged\. The*reference tier*is the tier whose meanww, averaged over the three domains, is closest to zero\. It is chosen once per model and shared by all three domains\.

Odds ratio of support\.For tierttand belief stratumss, letasa\_\{s\}andbsb\_\{s\}count the supported \(θ=1\\theta=1\) and unsupported \(θ=0\\theta=0\) updates in tiertt, letcsc\_\{s\}anddsd\_\{s\}count the same for the reference tier, and letns=as\+bs\+cs\+dsn\_\{s\}=a\_\{s\}\+b\_\{s\}\+c\_\{s\}\+d\_\{s\}\. The odds ratio of support of tierttis the Mantel–Haenszel estimate pooled over belief strata\[[72](https://arxiv.org/html/2610.41015#bib.bib72)\], in decibans:

ψt=10ln⁡10​ln⁡∑sas​ds/ns∑sbs​cs/ns\.\\psi\_\{t\}=\\frac\{10\}\{\\ln 10\}\\,\\ln\\frac\{\\sum\_\{s\}a\_\{s\}d\_\{s\}/n\_\{s\}\}\{\\sum\_\{s\}b\_\{s\}c\_\{s\}/n\_\{s\}\}\.This odds ratio compares the odds that the hypothesis is supported after an update in tierttwith the same odds after a reference update from the same incoming belief\. Pooling within belief strata makesψ\\psiindependent of where the belief started\. When either sum is zero, as happens when a tier separates the labels perfectly,0\.50\.5is added to every cell before pooling\. Each tier is plotted atxt=w¯t−w¯refx\_\{t\}=\\bar\{w\}\_\{t\}\-\\bar\{w\}\_\{\\mathrm\{ref\}\}, its mean weight of evidence relative to the reference tier, so the reference tier sits at the origin withψ=0\\psi=0\. The points are then averaged over the three domains by tier rank\.

Slope\.If a model’s probabilities are calibrated,ψt=xt\\psi\_\{t\}=x\_\{t\}for every tier, up to sampling error\. We summarize the points by their least\-squares slope through the origin,

β=∑txt​ψt∑txt2=∑txt2​\(ψt/xt\)∑txt2,\\beta=\\frac\{\\sum\_\{t\}x\_\{t\}\\,\\psi\_\{t\}\}\{\\sum\_\{t\}x\_\{t\}^\{2\}\}=\\frac\{\\sum\_\{t\}x\_\{t\}^\{2\}\\,\(\\psi\_\{t\}/x\_\{t\}\)\}\{\\sum\_\{t\}x\_\{t\}^\{2\}\},where the sums run over the non\-reference tiers\. The slope is a weighted average of the per\-tier ratiosψt/xt\\psi\_\{t\}/x\_\{t\}with weightsxt2x\_\{t\}^\{2\}, so the tiers with the largest updates count most\. A value ofβ=1\\beta=1indicates calibrated updates,β<1\\beta<1indicates overreaction \(updates larger than the outcomes justify\), andβ\>1\\beta\>1indicates underreaction \(updates smaller than the outcomes justify\)\. The quantityβ−1\\beta\-1is the log\-odds analog of the coefficient in a regression of forecast errors on forecast revisions, which is negative when forecasters overreact to news and positive when they underreact\[[20](https://arxiv.org/html/2610.41015#bib.bib73),[21](https://arxiv.org/html/2610.41015#bib.bib74)\]\.

Uncertainty\.Each episode contributes up to1616updates that share its label, so the updates are not independent\. Intervals come from the episode\-clustered bootstrap described above: each draw resamples episodes within each domain, holds the bin edges and the reference tier fixed, and recomputes the points and the slope\.

Sampling noise\.Eachpkp\_\{k\}of a generative model is a single sample \(see*Decoding*below\), so part of the variation inwwis sampling noise that is unrelated to the outcome\. Binning on a noisywwbiasesβ\\betatoward zero and therefore overstates overreaction for these models\. The classifier is deterministic and is unaffected\.

#### Stub verdicts\.

Each evidence stub is assigned a verdict relative to the episode’s target hypothesish∗h^\{\*\}, judged againsth∗h^\{\*\}’s claim rather than the stub’s own effect\. InTahoethe verdict is directional: a stub is*supported*when its differential\-expression measurement is significant \(padj<0\.05\\mathrm\{padj\}<0\.05,\|log2⁡FC\|\>0\.5\|\\log\_\{2\}\\mathrm\{FC\}\|\>0\.5\) in the directionh∗h^\{\*\}claims,*refuted*when significant in the opposite direction, and*inconclusive*otherwise\. InDepMap, whereh∗h^\{\*\}\(“GGis a dependency inCC”\) is directionless, a stub is*supported*when it indicates a dependency \(Chronos≤−0\.5\\mathrm\{Chronos\}\\leq\-0\.5,P⁡\(dep\)≥0\.5P\(\\mathrm\{dep\}\)\\geq 0\.5\),*refuted*when it indicates a suppressor \(Chronos\>0\.25\\mathrm\{Chronos\}\>0\.25\), and*inconclusive*otherwise\.GWAShas no refuting category: a stub is*significant*whenp<5×10−8p<5\\times 10^\{\-8\}and*sub\-threshold*otherwise\. Because effect\-direction concordance across different variants and traits is not biologically guaranteed, a direction\-flipped association is not treated as refuting\.

#### Paired comparisons\.

Comparisons reported as differences use a paired bootstrap over models evaluated on identical episodes\. Each iteration applies one episode resample to every model in the comparison\. Leave\-one\-out deltas and retention intervals use this procedure\. Intervals for individual rows in the master table are unpaired\.

#### Calibration: sMECE in place of ECE\.

Expected calibration error \(ECE\)\[[73](https://arxiv.org/html/2610.41015#bib.bib3)\]averages the absolute difference between each bin’s mean forecast and outcome rate, weighted by bin occupancy\. Our implementation computes ECE over the same ten equal\-width bins used for the Murphy terms, but the reported calibration statistic is sMECE because ECE depends on the number and placement of bin edges\.

For sMECE, we estimate the empirical frequencyr⁡\(p\)=𝔼⁡\[y∣p\]r\(p\)=\\mathbb\{E\}\[y\\mid p\]by Nadaraya–Watson regression of the outcomes on the forecasts,

r^​\(pi\)=∑jK⁡\(pi,pj\)​yj∑jK⁡\(pi,pj\),K⁡\(pi,pj\)=ϕ⁡\(pi−pjh\)\+ϕ⁡\(pi\+pjh\)\+ϕ⁡\(2−pi−pjh\),\\hat\{r\}\(p\_\{i\}\)\\;=\\;\\frac\{\\sum\_\{j\}K\(p\_\{i\},p\_\{j\}\)\\,y\_\{j\}\}\{\\sum\_\{j\}K\(p\_\{i\},p\_\{j\}\)\},\\qquad K\(p\_\{i\},p\_\{j\}\)\\;=\\;\\phi\\\!\\left\(\\tfrac\{p\_\{i\}\-p\_\{j\}\}\{h\}\\right\)\+\\phi\\\!\\left\(\\tfrac\{p\_\{i\}\+p\_\{j\}\}\{h\}\\right\)\+\\phi\\\!\\left\(\\tfrac\{2\-p\_\{i\}\-p\_\{j\}\}\{h\}\\right\),withϕ\\phithe standard Gaussian density, and summarize the fit as the mean of\|r^​\(pi\)−pi\|\|\\hat\{r\}\(p\_\{i\}\)\-p\_\{i\}\|over the evaluated episodes\. Averaging at the observed forecasts weights the error by forecast density\.

The second and third kernel terms reflect the Gaussian across00and11to preserve boundary mass\. The bandwidth follows Silverman’s rule on the forecasts,h=1\.06σ^n−1/5h=1\.06\\,\\hat\{\\sigma\}n^\{\-1/5\}, with a minimum value of0\.020\.02to prevent a nearly degenerate forecast distribution from producing a near\-zero bandwidth\.

This is an estimator in the smooth\-ECE family introduced by[Błasiok and Nakkiran \[19\]](https://arxiv.org/html/2610.41015#bib.bib6), not their estimator: the bandwidth rule and the boundary handling are ours\. We therefore refer to the statistic as sMECE\.

#### Murphy decomposition\.

The Murphy decomposition separates the Brier score into reliability, resolution, and uncertainty terms\[[28](https://arxiv.org/html/2610.41015#bib.bib2),[74](https://arxiv.org/html/2610.41015#bib.bib4),[75](https://arxiv.org/html/2610.41015#bib.bib5)\]:

Brier​Score=1N​∑i\(pi−θi\)2=∑kwk​\(p¯k−θ¯k\)2⏟Reliability−∑kwk​\(θ¯k−θ¯\)2⏟Resolution\+θ¯​\(1−θ¯\)⏟Uncertainty,\\mathrm\{Brier\\ Score\}\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i\}\(p\_\{i\}\-\\theta\_\{i\}\)^\{2\}\\;=\\;\\underbrace\{\\sum\_\{k\}w\_\{k\}\(\\bar\{p\}\_\{k\}\-\\bar\{\\theta\}\_\{k\}\)^\{2\}\}\_\{\\text\{Reliability\}\}\\;\-\\;\\underbrace\{\\sum\_\{k\}w\_\{k\}\(\\bar\{\\theta\}\_\{k\}\-\\bar\{\\theta\}\)^\{2\}\}\_\{\\text\{Resolution\}\}\\;\+\\;\\underbrace\{\\bar\{\\theta\}\(1\-\\bar\{\\theta\}\)\}\_\{\\text\{Uncertainty\}\},\(3\)wherekkindexes ten equal\-width bins,wkw\_\{k\}is the bin’s share of episodes,p¯k\\bar\{p\}\_\{k\}andθ¯k\\bar\{\\theta\}\_\{k\}are its mean forecast and observed outcome rate, andθ¯\\bar\{\\theta\}is the base rate\. Bins are right\-closed, with the first bin including00\.

Reliability measures within\-bin calibration error and is lower when mean forecasts match observed frequencies\. Resolution measures variation in observed outcome rates across forecast bins and is higher when forecasts separate outcomes\. Uncertainty depends only on the label distribution and is identical across models evaluated on a fixed test set; for the balanced test set it is0\.250\.25\.

The identity is exact for binned forecasts\. The tables report unbinned mean squared error, so the right\-hand side of[Eq\.3](https://arxiv.org/html/2610.41015#A3.E3)differs from the reported Brier score by within\-bin dispersion\. We use the decomposition to interpret differences among models\.

#### Reading intervals near zero\.

Reliability and sMECE are non\-negative and positively biased: resampling injects noise that a squared or absolute deviation can only turn into a larger value\. For a row whose true calibration error is close to zero, the bootstrap distribution can lie above the point estimate, leaving the point estimate outside its own interval\. Such rows are flagged in the tables\. These intervals support comparisons among rows but should not be interpreted as coverage intervals for a near\-zero point estimate\.

#### Decoding\.

The generative rows are sampled with a fixed seed at temperature0\.70\.7, at which the RL policies are also trained and validated \([App\.B](https://arxiv.org/html/2610.41015#A2)\)\. On the primary set each episode is sampled four times and the reported probability is the mean of parsed draws\. On the sensitivity set, where every episode is rendered at all1717context lengths, each episode is drawn once\. Both estimators target the same quantity; the four\-draw mean has lower sampling variance\. The classifier is deterministic: its probability is the sigmoid of a scalar head applied to the last\-token representation\. Frontier models use provider\-default sampling settings as described in[App\.D](https://arxiv.org/html/2610.41015#A4)\. All evaluated completions produced a readable probability, and no episodes are excluded\.

#### Complete results table\.

[Table6](https://arxiv.org/html/2610.41015#A3.T6)reports the macro\-average for every model with its per\-domain breakdown, the number of episodes scored, and a bootstrap interval on every value\.

Macro\-average over Tahoe, DepMap, GWAS; 95% CI from a within\-domain bootstrap resampling episodes \(2,000 draws; sMECE 200, it is 35x costlier per draw\)\. The unindented row of each block is that macro\-average; the indented rows below it are the single domains, bootstrapped over that domain’s episodes alone\. The constant row ignores all evidence; the test set is balanced 50:50 by construction, so it scores≈\\approx0\.25 Brier\. Rows above that line do not beat guessing\.†\\daggerpoint estimate outside its bootstrap interval, as expected for the positively\-biased binned metrics \(Reliability, sMECE\) on rows near the zero floor\. Compare rows by these intervals; do not read them as covering the point\.

Table 6:Calibration and discrimination on the held\-out multi\-domain test set, in full\.

## Appendix DFrontier baseline settings

The frontier baselines, GPT\-5\.6\-sol \(OpenAI, API model identifiergpt\-5\.6\-sol\) and Claude Opus 5 \(Anthropic,claude\-opus\-5\), are evaluated on the same episodes as the other models, with reasoning effort and output budget set explicitly for each provider\. Reasoning effort ishighfor both, mapped tooutput\_config\.efforton Anthropic andreasoning\_efforton OpenAI\. The output budget is24,00024\{,\}000tokens for both, mapped tomax\_tokensandmax\_completion\_tokens, respectively; the local thinking\-mode models use the same budget\. On Anthropic, this limit includes both reasoning and response tokens, and truncation before the final response produces an unreadable probability \([App\.C](https://arxiv.org/html/2610.41015#A3)\)\.

The local generative models are decoded at temperature0\.70\.7with a fixed seed, matching their training and validation temperature \([App\.B](https://arxiv.org/html/2610.41015#A2)\)\. The classifier does not sample: a scalar head reads its probability off the last\-token representation\. Claude Opus 5 rejects temperature, top\-ppand top\-kk, so the frontier rows use provider\-default sampling\. Both frontier models receive the system prompt of the local thinking\-mode models\.

Both models were queried on 29 July 2026 for the test sets and on 4 August 2026 for the conflict and position probes\. Declines and API errors are recorded separately from unreadable completions, so policy refusals are not counted as parse failures\. Remaining per\-provider API details are in the released evaluation code\.

## Appendix EEvidence accumulation and per\-update belief revision

This appendix reports the evidence\-accumulation and per\-stub analyses summarized in[Section5](https://arxiv.org/html/2610.41015#S5)\. All use the sensitivity set, which presents each of256256episodes per domain at every evidence countk∈\{0,…,16\}k\\in\\\{0,\\dots,16\\\}\.

### E\.1Cumulative evidence accumulation

[Fig\.7](https://arxiv.org/html/2610.41015#A5.F7)reports the mean probability assigned to the true label \(supported or not supported\) as evidence accumulates,

pktrue=θ​pk\+\(1−θ\)​\(1−pk\)\.p^\{\\text\{true\}\}\_\{k\}=\\theta p\_\{k\}\+\(1\-\\theta\)\(1\-p\_\{k\}\)\.Every model assigns more probability to the true label atk=16k=16than atk=0k=0\([Fig\.7](https://arxiv.org/html/2610.41015#A5.F7)\); the paired gain excludes zero in1919of the2121model–domain curves\. The exceptions are the two frontier models on DepMap, whose mean changes are small relative to their uncertainty\. Starting points vary widely: Ibis0 \(classifier\) begins at0\.7290\.729on GWAS before seeing any evidence and gains only0\.0500\.050, whereas on DepMap it begins near0\.50\.5and gains0\.1810\.181\.

Figure 7:Confidence in the true label increases as evidence accumulates\.Mean probability each model assigns to the true label vs\. the number of evidence stubs shown \(256256episodes per domain\)\. The slope reflects integration of new evidence \(up tok=16k=16stubs; shading marksk<2k<2, below the range seen in training\)\. The intercept reflects prior knowledge:p\>0\.5p\>0\.5atk=0k=0means that the model predicts the target hypothesiswith no evidence\. Above\-chancea prioripredictions on GWAS, most notably by Ibis0 \(classifier\), reflect non\-random association of alleles in high linkage disequilibrium in EDB\-Bio \(see[§A\.5](https://arxiv.org/html/2610.41015#A1.SS5)\)\.This value depends on the priorp0p\_\{0\}, the model’s belief in the hypothesis at the start of the episode, and several models in[Fig\.7](https://arxiv.org/html/2610.41015#A5.F7)start atp0true\>0\.5p^\{\\text\{true\}\}\_\{0\}\>0\.5\. We therefore also measure how far the evidence moves each model from itsk=0k\{=\}0belief\. Thecumulative weight of evidenceis the running sum of the per\-stub updateswjw\_\{j\}\([§2](https://arxiv.org/html/2610.41015#S2)\) and, for an ideal Bayesian, the log Bayes factor of the revealed evidence\. We orient it toward the ground truth:

wkcum=s⁡\[logit⁡\(pk\)−logit⁡\(p0\)\]⋅10log⁡10,s=\{\+1,θ=1−1,θ=0w\_\{k\}^\{\\text\{cum\}\}=s\\,\[\\operatorname\{logit\}\(p\_\{k\}\)\-\\operatorname\{logit\}\(p\_\{0\}\)\]\\cdot\\frac\{10\}\{\\log 10\},\\quad s=\\begin\{cases\}\+1,&\\theta=1\\\\ \-1,&\\theta=0\\end\{cases\}so thatwkcumw\_\{k\}^\{\\text\{cum\}\}, in decibans, is positive when evidence has pushed belief toward the true outcome \([Fig\.8](https://arxiv.org/html/2610.41015#A5.F8)\)\. The metric isprior\-agnostic: it is zero atk=0k\{=\}0by construction and measures how far a model updates, independent of its starting belief\. It iscumulative: differencing against stepk−1k\-1gives the marginal counterpart,log⁡B​Fk−log⁡B​Fk−1\\log BF\_\{k\}\-\\log BF\_\{k\-1\}\. It is alsodirection\-independent: the per\-episode signssswaps the numerator and denominator ofB​FkBF\_\{k\}on episodes that are not supported, so a positive value means that the stubs moved belief in the correct direction whether or not the target hypothesis is supported\.

Byk=16k\{=\}16, models differ roughly threefold in how far the evidence moves them: Qwen3 \(non\-thinking\) moves farthest toward the truth \(\+4\.37\+4\.37dB\[\+3\.05,\+5\.74\]\[\+3\.05,\+5\.74\]\), then Ibis0 \(classifier\) \(\+3\.65\+3\.65\[\+3\.14,\+4\.17\]\[\+3\.14,\+4\.17\]\), while Claude Opus 5 moves least \(\+1\.39\+1\.39\[\+0\.99,\+1\.76\]\[\+0\.99,\+1\.76\]\)\. The most responsive model, Qwen3 \(non\-thinking\), also has the worst Brier score\. Movement this large can reflect strong response to useful evidence or indiscriminate overreaction; the per\-stub analyses in[§§E\.2](https://arxiv.org/html/2610.41015#A5.SS2)and[E\.3](https://arxiv.org/html/2610.41015#A5.SS3)and the calibration of weights of evidence in[§5](https://arxiv.org/html/2610.41015#S5.SS0.SSS0.Px2)separate the two\.

Figure 8:Cumulative evidence\-induced log\-odds change by domain\.Cumulative log Bayes factor toward the observed outcome,s⁡\[logit⁡\(pk\)−logit⁡\(p0\)\]s\\,\[\\operatorname\{logit\}\(p\_\{k\}\)\-\\operatorname\{logit\}\(p\_\{0\}\)\]in decibans, against the number of stubs shown\. Each curve is relative to that model’s ownk=0k=0forecast\. Panels show the three domains and their macro\-average on a sharedyyaxis; bands are95%95\\%intervals from within\-domain episode resampling\.
### E\.2Direction of belief revision

#### Direction of belief revision\.

We test whether models revise belief in the expected direction on evidence that plausibly supports or refutes the target hypothesish∗h^\{\*\}\. In each domain, stubs are sorted into heuristic*supporting*and*refuting*categories by their direction relative toh∗h^\{\*\}\. For example, whenh∗h^\{\*\}says drugDDup\-regulates geneGG, a related drug that significantly up\-regulatesGGcounts as supporting and one that significantly*down*\-regulates it as refuting; a stub with no statistically significant effect is*inconclusive*\. Full per\-domain definitions are given in[App\.C](https://arxiv.org/html/2610.41015#A3.SS0.SSS0.Px5)\.

Belief revisions have the expected signs \([Fig\.9](https://arxiv.org/html/2610.41015#A5.F9)\): every model’s mean update is positive on supporting stubs, negative on refuting stubs, and small on inconclusive ones\. Magnitudes differ substantially across models\. Because all models see the same stubs in the same order, the difference reflects how strongly each model weighs the same category of evidence\. Qwen3 \(non\-thinking\) produces the largest updates in both directions, and[§5](https://arxiv.org/html/2610.41015#S5.SS0.SSS0.Px2)shows that they overreact; its reinforcement\-learned counterpart Ibis0 \(non\-thinking\) responds more selectively \([§E\.3](https://arxiv.org/html/2610.41015#A5.SS3)\)\.

Figure 9:Do models distinguish supporting vs\. refuting evidence?Mean signedwjw\_\{j\}in decibans; positive raises belief in the target hypothesis and negative lowers it\. Every model’s mean update has the expected sign for supporting and refuting evidence, and most models place little weight on inconclusive evidence\. \(GWAS admits no refuting evidence \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\), so its panel contrasts genome\-wide\-significant with sub\-threshold associations\.\)

### E\.3Evidence by relevance tier

#### Evidence by relevance tier\.

A model that deliberates well should reserve its largest updates for the evidence that bears most directly on the target\.[Fig\.10](https://arxiv.org/html/2610.41015#A5.F10)plots each model’s mean\|wj\|\|w\_\{j\}\|by relevance tier and summarizes its alignment with the curated hierarchy \([§A\.2](https://arxiv.org/html/2610.41015#A1.SS2)\) as a per\-domain Spearman correlationρ\\rhobetween\|wj\|\|w\_\{j\}\|and tier rank \(positive when more\-relevant stubs draw larger updates\)\. Among the trained models, Ibis0 \(non\-thinking\) aligns most closely in every domain \(ρ=0\.22\\rho=0\.22,0\.370\.37,0\.180\.18on Tahoe, DepMap, and GWAS\), despite trailing the classifier on final Brier and AUROC \(the classifier’s correlations are only0\.090\.09,0\.190\.19,0\.090\.09\)\. Aggregate forecast quality and the structure of individual updates therefore capture different behaviors\. Selectivity is also distinct from responsiveness, the overall magnitude of belief change: Qwen3 \(non\-thinking\) produces large updates on average yet aligns only weakly with the hierarchy \(ρ=0\.07\\rho=0\.07on Tahoe\)\.

The cross\-model ordering in[Fig\.10](https://arxiv.org/html/2610.41015#A5.F10)also tests the hierarchy itself\. Agreement is strongest on DepMap\. On GWAS, models over\-weight indirect same\-SNP evidence relative to the curated top tier, and on Tahoe, Claude Opus 5 places disproportionate weight on aggregate summary evidence over more\-specific assay readouts\. These discrepancies may reflect model failures, limitations of the expert ordering, or both\.

Figure 10:Do models assign higher weight to more relevant evidence in an expert biological knowledge hierarchy?Each stem shows how strongly a tier of evidence moves a model, relative to that model’s average stub in the same episode \(1×1\\timesis average\)\. Tiers run left to right in the expert\-ranked order, so a model that follows the hierarchy shows heights falling from left to right\. On DepMap, the models largely follow the expert ordering\. On GWAS, models assign more weight to indirect evidence \(same SNP\) than to more direct evidence \(same gene, same trait\)\. On Tahoe, Claude gives disproportionately high weight to aggregate summary evidence over more\-specific assay readouts\.

## Appendix FFull cross\-domain transfer results

Raw performance on the omitted domain is hard to interpret because Qwen3 and the all\-domain classifier set a different floor and ceiling for each metric\. We therefore normalize the leave\-one\-domain\-out result between these two references\. For domainddand metricmm, we report

retention=mleave\-out\-​d−muntrainedmall domains−muntrainedon domain​d,\\text\{retention\}\\;=\\;\\frac\{m\_\{\\text\{leave\-out\-\}d\}\-m\_\{\\text\{untrained\}\}\}\{m\_\{\\text\{all domains\}\}\-\{m\_\{\\text\{untrained\}\}\}\}\\qquad\\text\{on domain \}d,\(4\)where100%100\\%denotes the full improvement from the untrained model \(Qwen3\) to the all\-domain classifier and0%0\\%denotes no improvement over Qwen3\. For higher\-is\-better metrics, both differences are reversed\. Paired bootstrap intervals use identical episodes for all three models\.

![Refer to caption](https://arxiv.org/html/2610.41015v1/transfer_combined.png)Figure 11:Transfer to domains excluded from training\.A:We train Ibis0 \(classifier\) on two domains \(rows\) and evaluate on a third held\-out domain \(columns\)\. Outlined diagonal cells are o\.o\.d\. evaluations on the excluded domain and report the raw Brier score and the retention percentage \([Eq\.4](https://arxiv.org/html/2610.41015#A6.E4)\)\.B:Retention on the excluded domains, decomposed into Brier score and its Murphy reliability and resolution terms\. Boxes show paired\-bootstrap95%95\\%intervals and internal lines show point estimates; vertical lines mark Qwen3 \(0%0\\%\) and the all\-domain classifier \(100%100\\%\)\. Reliability retains5858–86%86\\%of the full\-model improvement, whereas effectively no resolution improvement is retained\.[Table7](https://arxiv.org/html/2610.41015#A6.T7)reports the complete numerical results underlying[Fig\.11](https://arxiv.org/html/2610.41015#A6.F11)and[§4](https://arxiv.org/html/2610.41015#S4.SS0.SSS0.Px4)\. Block A adds the all\-domain reference and macro\-averages omitted from the main figure\. Block B reports the held\-out\-domain decomposition in absolute units, as a paired loss relative to the all\-domain classifier, and on the normalized retention scale\.

Table 7:Full cross\-domain transfer results\.A:Brier score for every classifier training mix and test domain, with95%95\\%bootstrap intervals\. Bold cells evaluate a domain excluded from training; other per\-domain cells are descriptive in\-domain evaluations\. Macro is the equal\-domain average and is bootstrapped directly as a stratified macro statistic\.B:Exact results on the excluded domain for Brier score and its Murphy reliability and resolution terms\. The Qwen3 and full\-model references denote the released checkpoint and the all\-domain classifier, respectively\. Loss vs\. full is signed so that positive values indicate worse performance: leave\-one\-out minus full for Brier and reliability, and full minus leave\-one\-out for resolution\. Gain retained places the leave\-one\-out classifier between Qwen3 \(0%0\\%\) and the full model \(100%100\\%\)\. Loss and retention intervals use paired bootstraps over identical episodes\.

## Appendix GExperimental design details

This appendix collects material supporting[§6](https://arxiv.org/html/2610.41015#S6)\.

### G\.1Naive\-prompting selection prompt

The naive\-prompting policyπnaive\\pi\_\{\\text\{naive\}\}\([§6](https://arxiv.org/html/2610.41015#S6)\) asks a model to name the single most informative candidate directly, given the target hypothesish∗h^\{\\ast\}and the evidence revealed so far\. Because this is a selection task rather than the probability elicitation of the belief model, it uses a distinct system prompt that forbids a probability and fixes the answer format to`<choice\>i</choice\>`: reusing the belief prompt caused format\-strict models to emit`<probability\>`and silently fall back to random selection\. The system prompt is

System promptYou are an expert scientist selecting which experiment to run next\. You areshown a target hypothesis, the evidence gathered so far, and a numbered listof candidate experiments whose results are hidden\. Do NOT output a probability\.Your only task is to choose the single candidate whose hidden result would mostchange your belief about the target hypothesis\. Respond with just its index inthe exact form <choice\>i</choice\>\.

The user message reuses the belief model’s rendering of the target hypothesis and the revealed evidence—theCONTEXT EVIDENCEandTARGET HYPOTHESISblocks documented in[App\.I](https://arxiv.org/html/2610.41015#A9)—then appends the numbered candidate experiments, each rendered with its result hidden, followed by the selection instruction:

User prompt template<CONTEXT EVIDENCE and TARGET HYPOTHESIS, rendered as in the prompts appendix\>CANDIDATE NEXT EXPERIMENTS \(results hidden\):\[0\] <candidate experiment setup, result hidden\>\[1\] <candidate experiment setup, result hidden\>\.\.\.Which single candidate experiment, once its result is revealed, would mostchange your belief about the target hypothesis? Answer with just its index as<choice\>i</choice\>\.

The same prompt is used for every naive policy \(Qwen3, Ibis0, GPT\-5\.6\-sol, Opus 5\); only the answering model changes\. The belief tracker that scores each policy’s trajectory is held fixed across policies, so the comparison isolates selection quality\.

## Appendix HPost\-hoc recalibration

We test how much of each model’s Brier deficit lies in its probability scale rather than its ranking, and whether the trained models keep their advantage when every model is recalibrated\.

#### Procedure\.

For each model and each domain we fit Platt scaling\[[22](https://arxiv.org/html/2610.41015#bib.bib7)\]to the model’s own reported probabilities: a two\-parameter affine map on the logit,p^=σ⁡\(a⋅logit⁡\(p\)\+b\)\\hat\{p\}=\\sigma\\\!\\left\(a\\cdot\\operatorname\{logit\}\(p\)\+b\\right\), estimated by logistic regression of the observed outcome onlogit⁡\(p\)\\operatorname\{logit\}\(p\)with negligible regularization \(C=106C=10^\{6\}\)\. The map is*cross\-fitted*over two random folds:aaandbbare estimated on one half of a domain’s episodes and applied to the other, so no episode is scored under a map to which it contributed\. Refitting within each bootstrap draw propagates fit variability into the intervals in[Table8](https://arxiv.org/html/2610.41015#A8.T8)\. We recalibrate all seven models, trained and untrained alike\.

Macro\-average over Tahoe, DepMap, GWAS; 95% CI from a within\-domain bootstrap resampling episodes \(2,000 draws; sMECE 200, it is 35x costlier per draw\)\.†\\daggerpoint estimate outside its bootstrap interval — expected for the positively\-biased binned metrics \(Reliability, sMECE\) on rows near the zero floor\. Compare rows by these intervals; do not read them as covering the point\.

Table 8:Cross\-fit Platt recalibration of every evaluated system: a two\-parameter affine map on the logit, fitted out\-of\-fold within each domain\. Because it requires labelled target\-domain data, it bounds how much of each gap a post\-hoc fix can close and is not a deployable method\. The trained systems are also recalibrated so that the two blocks are comparable\.
#### Calibration and ranking after scaling\.

sMECE decreases for every model, by a factor of5\.15\.1to10\.910\.9for the four untrained models and1\.31\.3to3\.13\.1for the three trained ones\. Murphy reliability follows the same split, falling by a factor of2626to8080against1\.41\.4to5\.45\.4\. The trained models gain less because they start better calibrated\.

Post\-hoc scaling reduces calibration error without materially changing rank discrimination\. The largest change in AUROC is0\.0030\.003, for Qwen3 \(non\-thinking\); every other change is below0\.0010\.001, and the model ordering by AUROC is unchanged\. A single affine map on the logit is monotone and preserves ranking\. The small changes arise because the two cross\-fitting folds use different maps, which can reorder observations across the fold boundary\.

#### Forecast metrics after scaling\.

Recalibration removes most of the untrained models’ Brier deficit: Brier falls by0\.0770\.077to0\.1020\.102for the four untrained models, against0\.0010\.001to0\.0070\.007for the three trained ones\. Ibis0 \(classifier\) still has the lowest recalibrated Brier at0\.1710\.171, ahead of Claude Opus 5 at0\.2000\.200, with non\-overlapping intervals\.

Matching the treatment reverses the sMECE comparison\. Recalibrated GPT\-5\.6\-sol reaches0\.0240\.024, against0\.0170\.017for recalibrated Ibis0 \(non\-thinking\) and0\.0210\.021for Ibis0 \(classifier\), with overlapping intervals\. Comparing recalibrated frontier models with unscaled trained ones understates the trained models on this metric, because the classifier’s as\-reported sMECE of0\.0660\.066is the one value in its row with appreciable headroom\.

Macro\-average over Tahoe, DepMap, GWAS; 95% CI from a within\-domain bootstrap resampling episodes \(2,000 draws; sMECE 200, it is 35x costlier per draw\)\. The unindented row of each block is that macro\-average; the indented rows below it are the single domains, bootstrapped over that domain’s episodes alone\.†\\daggerpoint estimate outside its bootstrap interval — expected for the positively\-biased binned metrics \(Reliability, sMECE\) on rows near the zero floor\. Compare rows by these intervals; do not read them as covering the point\.

Table 9:Cross\-fit Platt recalibration by domain, with the map fitted separately within each of Tahoe, DepMap and GWAS\. The as\-reported breakdown is in the complete master table\.
#### Recalibration by domain\.

[Table9](https://arxiv.org/html/2610.41015#A8.T9)fitsaaandbbseparately within Tahoe, DepMap and GWAS\. The gains are not uniform across domains\. Tahoe remains the hardest domain for every model after scaling, with recalibrated Brier between0\.2030\.203and0\.2450\.245, against0\.1650\.165to0\.2090\.209on DepMap\. Ibis0 \(classifier\) has the widest spread, from0\.1440\.144on GWAS to0\.2030\.203on Tahoe, a range that the macro\-average conceals\.

#### Requirement for representative labels\.

Becauseaaandbbare estimated from other labeled episodes in the same domain, the procedure applies only when representative historical outcomes exist in the deployment domain\. It is unavailable for a new assay with no calibration sample and may be invalid under distribution shift\.

#### Resolution after scaling\.

Resolution is computed over ten equal\-width forecast bins, so a monotone rescaling changes it only by shifting episodes across a bin edge\. The seven resolutions change by at most0\.0040\.004in either direction\. After scaling, Qwen3 \(non\-thinking\) is at0\.0210\.021and Qwen3 \(thinking\) at0\.0360\.036, compared with0\.0800\.080for the classifier\. Coarse quantization of the verbalized forecasts may contribute to this difference, but the current comparison does not isolate that mechanism\.

## Appendix IPrompts: one worked example per domain

A shared function renders every episode as a prompt\. The prompt contains aCONTEXT EVIDENCEheader, one numbered block per evidence stub beginning with its relevance tag, aTARGET HYPOTHESISheader, the target block, and a fixed answer\-format instruction\. Domain\-specific functions render individual stubs and targets\.[Section2](https://arxiv.org/html/2610.41015#S2)defines the episode structure; this appendix provides one rendered example per domain\.

Each system prompt describes the assay and evidence\-generation procedure, defines the reported quantities, states the support\-label rule, and supplies an answer\-format instruction\. The*direct*instruction requests only a probability; the*thinking*instruction requests a reasoning trace followed by the same probability tag\. Each evaluated model receives the same instruction in every domain \([Apps\.B](https://arxiv.org/html/2610.41015#A2)and[D](https://arxiv.org/html/2610.41015#A4)\); the examples use the direct instruction\.

Direct instructionRespond with ONLY a single line in this exact format — no reasoning, no explanation, no other text before or after:<probability\>0\.XX</probability\>

Thinking instructionThink through the evidence carefully, then commit to a single probability in the closed interval \[0, 1\]\.After your reasoning, output your probability in this exact format:<probability\>0\.XX</probability\>

The examples are drawn from theprimarytest\-set configuration\. Long lines wrap at the margin, with each continuation marked by↪\\hookrightarrow, and the sole\!=in the Tahoe system prompt replaces a≠\\neqglyph the document font cannot set\. Evidence pools are truncated after the first two stubs, and the number of omitted stubs is marked in place\. The displayed text is otherwise unchanged\.

### I\.1Tahoe: drug perturbation of cancer cell lines

Episodeeb33e117\-0abasks whether encorafenib upregulatesLINC00342inSNU\-1\. The episode contains six stubs, and the held\-out DESeq2 result supports the hypothesis\.

The two displayed stubs both measure the target drug and gene and are statistically significant; one reports a decrease in a lung line and the other an increase of similar magnitude in a skin line\. Of the four omitted stubs, one is the target cell line perturbed by a different drug \(a null result\), and three are unrelated drug×\\timescell\-line baselines\. No stub reports encorafenib inSNU\-1, so the pool constrains the answer without containing it\.

System promptYou are a scientific reasoning assistant analyzing drug\-perturbation evidence from Tahoe\-100M, a large\-scale pseudobulk RNA\-seq dataset measuring drug × cell\-line × gene differential expression\.How the evidence was generated:\- Cells from a given cell line are treated with a drug \(or DMSO vehicle control\)\.\- Each cell’s transcripts are captured by single\-cell RNA\-seq; counts are UMI\-deduplicated per gene\.\- Single cells from the same \(drug, cell\_line, plate, replicate\) bucket are \*pseudobulked\* by summing per\-gene UMI counts\.\- For each gene, DESeq2 fits a negative\-binomial GLM comparing treated vs control pseudobulk samples\.Column definitions \(DESeq2 conventions\):\- log2FoldChange: shrunken log2\(treated / control\) of normalized counts\. Positive = upregulated; negative = downregulated\.\- lfcSE: standard error of the log2FoldChange estimate \(uncertainty\)\.\- pvalue: Wald test p\-value for log2FoldChange \!= 0\.\- padj: Benjamini\-Hochberg adjusted p\-value \(FDR\-controlled across all genes in that drug × cell\_line comparison\)\.\- baseMean: mean of normalized counts across both treated AND control samples combined \(an expression\-level indicator, NOT a control baseline\)\.\- n\_cells\_trt / n\_cells\_ctrl: number of single cells pooled into the treated / control pseudobulk\.You will be shown:1\. CONTEXT — prior experimental results: drug × cell\-line × gene differential\-expression measurements, plus aggregate perturbation summaries\.2\. A TARGET HYPOTHESIS — a claim of the form "drug D upregulates/downregulates gene G in cell line C"\.Weigh the evidence and report your degree of belief that a held\-out experiment would \*support\* the target hypothesis\. Supported means padj < 0\.05 AND \|log2FoldChange\| \> 0\.5 in the claimed direction\.Respond with ONLY a single line in this exact format — no reasoning, no explanation, no other text before or after:<probability\>0\.XX</probability\>

User prompt\(episodeeb33e117\-0ab, 6 stubs, label = supported\)CONTEXT EVIDENCE================\[1\] same drug as target, different cell line, target geneDrug: EncorafenibMOA \(broad\) inhibitor/antagonistMOA \(fine\) RAF inhibitorTargets BRAFHuman\-approved yesIn clinical trials yesNotes Approved for use in BRAF\-mutant melanoma\.Canonical SMILES CC\(C\)N1C=C\(C\(=N1\)C2=C\(C\(=CC\(=C2\)Cl\)NS\(=O\)\(=O\)C\)F\)C3=NC\(=NC=C3\)NCC\(C\)NC\(=O\)OCPubChem CID 50922675Cell line: NCI\-H2347DepMap ID ACH\-000875Cellosaurus ID CVCL\_1550Organ LungDriver gene B2MDriver variant type StopgainDriver zygosity HomDriver protein/cDNA effect p\.Q28\*Driver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: LINC00342Ensembl ID ENSG00000232931Experiment:Plate 3n\_cells treated 2747n\_cells control 3106baseMean \(avg normalized counts across both arms\) 90\.74Result:log2FoldChange \-0\.473 ± 0\.153 \(downregulates\)Wald statistic \-3\.101p\-value \(raw\) 0\.00193p\-value \(adjusted\) 0\.01023\[2\] same drug as target, different cell line, target geneDrug: EncorafenibMOA \(broad\) inhibitor/antagonistMOA \(fine\) RAF inhibitorTargets BRAFHuman\-approved yesIn clinical trials yesNotes Approved for use in BRAF\-mutant melanoma\.Canonical SMILES CC\(C\)N1C=C\(C\(=N1\)C2=C\(C\(=CC\(=C2\)Cl\)NS\(=O\)\(=O\)C\)F\)C3=NC\(=NC=C3\)NCC\(C\)NC\(=O\)OCPubChem CID 50922675Cell line: LOX\-IMVIDepMap ID ACH\-000750Cellosaurus ID CVCL\_1381Organ SkinDriver gene ARID1ADriver variant type StopgainDriver zygosity HetDriver protein/cDNA effect p\.Q1212\*Driver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: LINC00342Ensembl ID ENSG00000232931Experiment:Plate 3n\_cells treated 2163n\_cells control 2021baseMean \(avg normalized counts across both arms\) 154\.3Result:log2FoldChange \+0\.535 ± 0\.136 \(upregulates\)Wald statistic 3\.925p\-value \(raw\) 8\.66e\-05p\-value \(adjusted\) 0\.00113\[\.\.\. 4 further evidence stubs omitted \.\.\.\]TARGET HYPOTHESIS=================Drug Encorafenib upregulates gene LINC00342 in cell line SNU\-1\.Drug: EncorafenibMOA \(broad\) inhibitor/antagonistMOA \(fine\) RAF inhibitorTargets BRAFHuman\-approved yesIn clinical trials yesNotes Approved for use in BRAF\-mutant melanoma\.Canonical SMILES CC\(C\)N1C=C\(C\(=N1\)C2=C\(C\(=CC\(=C2\)Cl\)NS\(=O\)\(=O\)C\)F\)C3=NC\(=NC=C3\)NCC\(C\)NC\(=O\)OCPubChem CID 50922675Cell line: SNU\-1DepMap ID ACH\-000932Cellosaurus ID CVCL\_0099Organ Esophagus/StomachDriver gene ARID1ADriver variant type FrameshiftDriver zygosity HetDriver protein/cDNA effect p\.D1633fsDriver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: LINC00342Ensembl ID ENSG00000232931Respond using the format specified in the system message:<probability\>0\.XX</probability\>

### I\.2DepMap: cancer gene dependency

Episode65b4265e\-9f1asks whetherDIS3Lis a dependency in the mesothelioma lineMERO25\. The held\-out screen does not support the hypothesis\.

The first displayed stub reports the target gene’s mutation status in the target line rather than a knockout result, an example of the multi\-modal evidence pool\. The four omitted stubs are two unrelated knockouts inMERO25and two knockouts ofDIS3Lin other pleural mesothelioma lines, the latter scoring−0\.25\-0\.25and−0\.45\-0\.45against a threshold of−0\.5\-0\.5\.

System promptYou are a scientific reasoning assistant analyzing cancer gene\-dependency evidence from DepMap, a large\-scale project of genome\-wide CRISPR knockout screens of cancer cell lines with matched molecular and pharmacological profiles\.How the evidence was generated:\- Each cell line is screened by pooled genome\-wide CRISPR: every gene is knocked out \(one gene per cell\), the culture grows ˜3 weeks, and cells whose knocked\-out gene was required for proliferation drop out of the pool\.\- The depletion of each gene is summarized by the Chronos gene\-effect score; a calibrated P\(dependency\) is derived from it\.\- Matched profiles, measured independently of the screen, give each line’s baseline mRNA expression, copy number, somatic mutations, and \(for some\) drug sensitivity\.Quantity definitions:\- Chronos gene\-effect: a CRISPR\-knockout fitness score, scaled so 0 = no effect on growth and \-1 = as lethal as a typical core\-essential gene \(more negative = more essential; positive = knockout speeds growth\)\. Analogous to a log2 fold\-change\.\- P\(dependency\): the calibrated probability in \[0,1\] that the gene is a true dependency in that cell line, inferred from the Chronos score\. Analogous to statistical significance\.You will be shown:1\. CONTEXT — prior measurements, each labeled by its relation to the target\.2\. A TARGET HYPOTHESIS — a claim of the form "gene G is a dependency in cell line C"\.Weigh the evidence and report your degree of belief that a held\-out genome\-wide CRISPR screen would support the target hypothesis\. Supported means the screen scores G at Chronos gene\-effect <= \-0\.5 AND P\(dependency\) \>= 0\.5\.Respond with ONLY a single line in this exact format — no reasoning, no explanation, no other text before or after:<probability\>0\.XX</probability\>

User prompt\(episode65b4265e\-9f1, 6 stubs, label = not supported\)CONTEXT EVIDENCE================\[1\] somatic mutation status of the target gene in the target cell lineSomatic mutation status of DIS3L in MERO25Gene: DIS3L — DIS3 like exosome 3’\-5’ exoribonuclease \(family: Exosome complex\)Cell line: MERO25 \(DepMap ACH\-001556\) — Pleura / Pleural Mesothelioma / Pleural Mesothelioma, Biphasic Type; Male; PrimaryResult: wild\-type\[2\] an unrelated gene knocked out in the target cell lineCRISPR knockout of KRT25 in MERO25Gene: KRT25 — keratin 25 \(family: Keratins, type I\)Cell line: MERO25 \(DepMap ACH\-001556\) — Pleura / Pleural Mesothelioma / Pleural Mesothelioma, Biphasic Type; Male; PrimaryResult:Chronos gene\-effect \-0\.02 \(no clear fitness effect\)P\(dependency\) 0\.03 \(almost certainly not a dependency\)\[\.\.\. 4 further evidence stubs omitted \.\.\.\]TARGET HYPOTHESIS=================Gene DIS3L is a dependency \(essential for fitness\) in cell line MERO25: a held\-out genome\-wide CRISPR knockout screen of MERO25 would score gene DIS3L at Chronos gene\-effect <= \-0\.5 AND P\(dependency\) \>= 0\.5\.Gene: DIS3L — DIS3 like exosome 3’\-5’ exoribonuclease \(family: Exosome complex\)Cell line: MERO25 \(DepMap ACH\-001556\) — Pleura / Pleural Mesothelioma / Pleural Mesothelioma, Biphasic Type; Male; PrimaryRespond using the format specified in the system message:<probability\>0\.XX</probability\>

### I\.3GWAS: genetic associations

Episode6b7db1d8\-f2basks whetherrs2238689\(T allele\) decreases a phospholipid ratio measurement\. The held\-out study supports the hypothesis at genome\-wide significance\.

GWAS relevance depends on locus and trait similarity: a stub may share the target locus, trait category, both, or neither\. The omitted stubs include a missense variant2,7312\{,\}731bp away in the same gene, associated atp=4×10−15p=4\\times 10^\{\-15\}with a trait in the target’s own EFO parent, and one association reported atp=8×10−8p=8\\times 10^\{\-8\}, below the genome\-wide significance threshold\.

System promptYou are an expert geneticist and epidemiologist analyzing genetic\-association evidence from the NHGRI\-EBI GWAS Catalog, a curated collection of published genome\-wide association studies \(GWAS\) of human traits and diseases\.How the evidence was generated:\- A GWAS genotypes many individuals and tests each common variant \(SNP\) for association with a trait, reporting an effect size and a p\-value per SNP\.\- An association is called genome\-wide significant at p < 5e\-8, the threshold that corrects for ˜1 million independent tests across the genome\.\- Each study reports, for the lead SNP, the risk allele whose effect is measured, its frequency, and the direction/size of effect \(odds ratio for disease traits, beta for quantitative traits\)\.Quantity definitions:\- p\-value: statistical significance of the SNP\-trait association; < 5e\-8 is genome\-wide significant\.\- Risk allele: the allele whose effect is reported; different studies may report opposite alleles for the same SNP, so direction is always stated relative to the named allele\.\- OR \(odds ratio, binary/disease traits\): OR \> 1 means the risk allele increases disease odds, OR < 1 means it decreases them \(protective\)\.\- Beta \(quantitative traits\): the named allele raises the trait when the direction is ’increase’, lowers it when ’decrease’\.\- RAF: risk\-allele frequency in the study population\.\- Mapped gene: gene assigned to the SNP by genomic position \(not a causal claim\); EFO parent: the Experimental Factor Ontology category grouping related traits\.You will be shown:1\. CONTEXT — prior GWAS associations, each labeled by its relation to the target\.2\. A TARGET HYPOTHESIS — a claim that a SNP influences a trait in a stated direction\.Weigh the evidence and report your degree of belief that a held\-out GWAS would support the target hypothesis\. Supported means the held\-out study reports p < 5e\-8 for this SNP\-trait pair with a consistent direction of effect\.Respond with ONLY a single line in this exact format — no reasoning, no explanation, no other text before or after:<probability\>0\.XX</probability\>

User prompt\(episode6b7db1d8\-f2b, 6 stubs, label = supported\)CONTEXT EVIDENCE================\[1\] the target SNP, associated with a trait in a different EFO parentrs2238689 for type 2 diabetes mellitusSNP: rs2238689 — chr19:45675403Risk allele: CRAF: 0\.4177Mapped gene: GIPRFunctional class: intron\_variantGene context:Reactome pathways: G alpha \(s\) signalling events, Glucagon\-type ligand receptorsGTEx expression: Cells \- EBV\-transformed lymphocytes 23\.8 TPM, Adipose \- Visceral \(Omentum\) 10\.8 TPM, Heart \- Atrial Appendage 9\.3 TPMStudy: GCST009379Ancestry: EuropeanSample size: 898,130Platform: Affymetrix, Illumina \[˜ 27000000\] \(imputed\)Result: p = 5\.0e\-09 \(genome\-wide significant\); OR/beta = 1\.0495% CI: \[1\.03\-1\.05\]Direction: C allele increases type 2 diabetes mellitus\[2\] an unrelated SNP and trait \(different gene, different EFO parent\)rs12324089 for testosterone measurementSNP: rs12324089 — chr15:64025657Risk allele: GRAF: 0\.409Mapped gene: DAPK2Functional class: intron\_variantGene context:Reactome pathways: Caspase activation via Dependence Receptors in the absence of ligandGTEx expression: Thyroid 42\.0 TPM, Muscle \- Skeletal 19\.2 TPM, Lung 16\.1 TPMStudy: GCST90483487Ancestry: EuropeanSample size: 235,096Platform: NR \[23899339\] \(imputed\)Result: p = 4\.0e\-08 \(genome\-wide significant\); OR/beta = 0\.015295% CI: \[0\.0097\-0\.0207\] unit decreaseDirection: G allele decreases testosterone measurement\[\.\.\. 4 further evidence stubs omitted \.\.\.\]TARGET HYPOTHESIS=================SNP rs2238689 \(T allele\) decreases phospholipids:total lipids ratio, high density lipoprotein cholesterol measurement\.A held\-out GWAS would report p < 5e\-8 for this SNP\-trait pair with a consistent direction of effect\.SNP: rs2238689 — chr19:45675403Risk allele: TRAF: 0\.600605Mapped gene: GIPRFunctional class: intron\_variantGene context:Reactome pathways: G alpha \(s\) signalling events, Glucagon\-type ligand receptorsGTEx expression: Cells \- EBV\-transformed lymphocytes 23\.8 TPM, Adipose \- Visceral \(Omentum\) 10\.8 TPM, Heart \- Atrial Appendage 9\.3 TPMTrait: phospholipids:total lipids ratio, high density lipoprotein cholesterol measurementEFO parent: Other measurementRespond using the format specified in the system message:<probability\>0\.XX</probability\>

## Appendix JA thinking\-mode reasoning trace

Ibis0 \(thinking\) emits a reasoning trace before committing to a probability\. This appendix reproduces one complete trace and the episode that produced it\. The rollout below is a single sample from the trained policy, on an episode from the Tahoe validation split rather than the test set\.

Episode6a6866fd\-49aasks whether belumosudil, a ROCK1/ROCK2 inhibitor, downregulatesSGMS2in the colorectal lineHCT15\. The prompt renders seven evidence stubs\. Two measure the same drug and gene in other cell lines, and both report*up*regulation:\+0\.833\+0\.833atpadj=1\.7×10−17\\text\{padj\}=1\.7\\times 10^\{\-17\}inHOP62, and\+0\.146\+0\.146atpadj=0\.045\\text\{padj\}=0\.045inHT\-29\. One measures the target cell line under a different drug and is null, as are the four remaining stubs\. The held\-out measurement refutes the hypothesis rather than leaving it unresolved: belumosudil significantly*up*regulatesSGMS2inHCT15\(\+0\.554\+0\.554,padj=0\.0056\\text\{padj\}=0\.0056\)\. The trace reads the same\-drug stubs as evidence against the claimed direction and settles at0\.200\.20\.

The trace also questions one of the prompt’s relevance tags\. Stub \[4\] is labeled as sharing the target drug’s mechanism, a match that holds on the coarse MOA field, where both drugs readinhibitor/antagonist, but not below it: artesunate’s fine MOA is a JAK/STAT inhibitor, and belumosudil’s listed targets are ROCK1 and ROCK2\. The trace registers the doubt \(“same mechanism as Belumosudil? Or not sure”\) and sets the stub aside as null\.

The trace is highly redundant\. The conclusion that the same\-drug stubs argue against a downregulation claim is restated at least four times, and the answer is approached, qualified and revisited before it is committed\.

Rendering follows the conventions of[App\.I](https://arxiv.org/html/2610.41015#A9), except that the completion is set as prose with its paragraph breaks preserved\. One drug name is transliterated \(18beta\-Glycyrrhetinic acid\) because the document font cannot set its Greek letter\. The<think\>tags enclosing the trace are dropped\. The text is otherwise unchanged, and the evidence pool is shown in full\.

### J\.1Prompt

The system prompt is the Tahoe system prompt of[App\.I](https://arxiv.org/html/2610.41015#A9)carrying the thinking instruction in place of the direct one, with a further paragraph inserted before it that names considerations to weigh:

System prompt\(addition over[App\.I](https://arxiv.org/html/2610.41015#A9)\)Reason about what the context implies — for example: the drug’s transcriptional footprint in this and nearby cell lines, whether mechanistically similar drugs show consistent effects on the target gene, whether the gene appears broadly responsive or quiet, the drug’s aggregate perturbation strength in the target cell line, and any contradictions between evidence pieces\. Then commit to a single probability in the closed interval \[0, 1\]\.

User prompt\(episode6a6866fd\-49a, 7 stubs, label = not supported\)CONTEXT EVIDENCE================\[1\] unrelated drug × cell line, included as a baselineDrug: Anethole trithioneMOA \(broad\) unclearMOA \(fine\) unclearHuman\-approved noIn clinical trials yesNotes Investigated for dry mouth in clinical trials\.Canonical SMILES COC1=CC=C\(C=C1\)C2=CC\(=S\)SS2PubChem CID 2194Cell line: hTERT\-HPNECellosaurus ID CVCL\_C466Organ PancreasDriver gene TERTDriver variant type TransgeneDriver gene type OncogeneGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 4n\_cells treated 2002n\_cells control 2917baseMean \(avg normalized counts across both arms\) 26\.75Result:log2FoldChange \+0\.115 ± 0\.334 \(upregulates\)Wald statistic 0\.3436p\-value \(raw\) 0\.7312p\-value \(adjusted\) 0\.9005\[2\] unrelated drug × cell line, included as a baselineDrug: DT\-061MOA \(broad\) activator/agonistMOA \(fine\) unclearHuman\-approved noIn clinical trials yesNotes Investigated for oncology, no approval yet for human use\.Canonical SMILES C1CC\(C\(C\(C1\)N2C3=CC=CC=C3OC4=CC=CC=C42\)O\)NS\(=O\)\(=O\)C5=CC=C\(C=C5\)OC\(F\)\(F\)FPubChem CID 91885558Cell line: CHP\-212DepMap ID ACH\-000120Cellosaurus ID CVCL\_1125Organ Peripheral Nervous SystemDriver gene NRASDriver variant type MissenseDriver zygosity HetDriver protein/cDNA effect p\.Q61KDriver mechanism \(LoF/GoF\) GoFDriver gene type OncogeneGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 2n\_cells treated 524n\_cells control 917baseMean \(avg normalized counts across both arms\) 6\.928Result:log2FoldChange \+0\.285 ± 0\.552 \(upregulates\)Wald statistic 0\.5167p\-value \(raw\) 0\.6053p\-value \(adjusted\) 0\.9021\[3\] same drug as target, different cell line, target geneDrug: Belumosudil \(mesylate\)MOA \(broad\) inhibitor/antagonistMOA \(fine\) unclearTargets ROCK2, ROCK1Human\-approved yesIn clinical trials yesNotes Approved for chronic graft\-versus\-host disease\.Canonical SMILES CC\(C\)NC\(=O\)COC1=CC=CC\(=C1\)C2=NC3=CC=CC=C3C\(=N2\)NC4=CC5=C\(C=C4\)NN=C5\.CS\(=O\)\(=O\)OPubChem CID 146681181Cell line: HOP62DepMap ID ACH\-000861Cellosaurus ID CVCL\_1285Organ LungDriver gene B2MDriver variant type MissenseDriver zygosity HetDriver protein/cDNA effect p\.E70QDriver gene type SuppressorGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 2n\_cells treated 2743n\_cells control 8616baseMean \(avg normalized counts across both arms\) 292\.5Result:log2FoldChange \+0\.833 ± 0\.093 \(upregulates\)Wald statistic 8\.991p\-value \(raw\) 2\.45e\-19p\-value \(adjusted\) 1\.67e\-17\[4\] drug with same mechanism of action as the target drugDrug: ArtesunateMOA \(broad\) inhibitor/antagonistMOA \(fine\) JAK/STAT inhibitorHuman\-approved yesIn clinical trials yesNotes Used for severe malaria, often partnered with other antimalarial drugs\.Canonical SMILES CC1CCC2C\(C\(OC3C24C1CCC\(O3\)\(OO4\)C\)OC\(=O\)CCC\(=O\)O\)CPubChem CID 6917864Cell line: RPMI\-7951DepMap ID ACH\-000348Cellosaurus ID CVCL\_1666Organ SkinDriver gene BRAFDriver variant type MissenseDriver zygosity HetDriver protein/cDNA effect p\.V600EDriver mechanism \(LoF/GoF\) GoFDriver gene type OncogeneGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 4n\_cells treated 1577n\_cells control 2477baseMean \(avg normalized counts across both arms\) 22\.73Result:log2FoldChange \+0\.200 ± 0\.361 \(upregulates\)Wald statistic 0\.555p\-value \(raw\) 0\.5789p\-value \(adjusted\) 0\.8409\[5\] unrelated drug × cell line, included as a baselineDrug: 18beta\-Glycyrrhetinic acidMOA \(broad\) unclearMOA \(fine\) unclearHuman\-approved noIn clinical trials yesNotes Used in cosmetics, derived from licorice root, no approval for human drugs\.Canonical SMILES C\[C@\]12CC\[C@\]\(C\[C@H\]1C3=CC\(=O\)\[C@@H\]4\[C@\]5\(CC\[C@@H\]\(C\(\[C@@H\]5CC\[C@\]4\(\[C@@\]3\(CC2\)C\)C\)\(C\)C\)O\)C\)\(C\)C\(=O\)OPubChem CID 10114Cell line: HS\-578TDepMap ID ACH\-000148Cellosaurus ID CVCL\_0332Organ BreastDriver gene CDKN2ADriver variant type DeletionDriver zygosity HomDriver protein/cDNA effect DELDriver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 4n\_cells treated 857n\_cells control 2654baseMean \(avg normalized counts across both arms\) 43\.66Result:log2FoldChange \+0\.255 ± 0\.218 \(upregulates\)Wald statistic 1\.17p\-value \(raw\) 0\.242p\-value \(adjusted\) 0\.6025\[6\] same drug as target, different cell line, target geneDrug: Belumosudil \(mesylate\)MOA \(broad\) inhibitor/antagonistMOA \(fine\) unclearTargets ROCK2, ROCK1Human\-approved yesIn clinical trials yesNotes Approved for chronic graft\-versus\-host disease\.Canonical SMILES CC\(C\)NC\(=O\)COC1=CC=CC\(=C1\)C2=NC3=CC=CC=C3C\(=N2\)NC4=CC5=C\(C=C4\)NN=C5\.CS\(=O\)\(=O\)OPubChem CID 146681181Cell line: HT\-29DepMap ID ACH\-000552Cellosaurus ID CVCL\_0320Organ BowelDriver gene APCDriver variant type StopgainDriver zygosity HetDriver protein/cDNA effect p\.E853\*Driver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 2n\_cells treated 2179n\_cells control 5453baseMean \(avg normalized counts across both arms\) 1\.18e\+03Result:log2FoldChange \+0\.146 ± 0\.056 \(upregulates\)Wald statistic 2\.584p\-value \(raw\) 0\.00976p\-value \(adjusted\) 0\.04521\[7\] different drug, same cell line as target, target geneDrug: AdenosineMOA \(broad\) unclearMOA \(fine\) unclearTargets ADORA1, ADORA2A, ADORA2B, ADORA3Human\-approved yesIn clinical trials yesNotes Used for certain arrhythmias; approved as a drug\.Canonical SMILES C1=NC\(=C2C\(=N1\)N\(C=N2\)C3C\(C\(C\(O3\)CO\)O\)O\)NPubChem CID 60961Cell line: HCT15DepMap ID ACH\-000997Cellosaurus ID CVCL\_0292Organ BowelDriver gene ACVR2ADriver variant type FrameshiftDriver zygosity HomDriver protein/cDNA effect p\.K437fsDriver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: SGMS2Ensembl ID ENSG00000164023Experiment:Plate 4n\_cells treated 925n\_cells control 2226baseMean \(avg normalized counts across both arms\) 83\.5Result:log2FoldChange \+0\.142 ± 0\.205 \(upregulates\)Wald statistic 0\.6909p\-value \(raw\) 0\.4897p\-value \(adjusted\) 0\.7822TARGET HYPOTHESIS=================Drug Belumosudil \(mesylate\) downregulates gene SGMS2 in cell line HCT15\.Drug: Belumosudil \(mesylate\)MOA \(broad\) inhibitor/antagonistMOA \(fine\) unclearTargets ROCK2, ROCK1Human\-approved yesIn clinical trials yesNotes Approved for chronic graft\-versus\-host disease\.Canonical SMILES CC\(C\)NC\(=O\)COC1=CC=CC\(=C1\)C2=NC3=CC=CC=C3C\(=N2\)NC4=CC5=C\(C=C4\)NN=C5\.CS\(=O\)\(=O\)OPubChem CID 146681181Cell line: HCT15DepMap ID ACH\-000997Cellosaurus ID CVCL\_0292Organ BowelDriver gene ACVR2ADriver variant type FrameshiftDriver zygosity HomDriver protein/cDNA effect p\.K437fsDriver mechanism \(LoF/GoF\) LoFDriver gene type SuppressorGene: SGMS2Ensembl ID ENSG00000164023Respond using the format specified in the system message:<probability\>0\.XX</probability\>

### J\.2Trace

Completion\(reasoning trace, verbatim\)Okay, let’s tackle this step by step\. So the task is to determine the probability that the target hypothesis \(Belumosudil downregulates SGMS2 in HCT15\) is supported based on the context provided\. The context has seven previous experiments, and I need to analyze each to see how they relate to the target hypothesis\.First, let’s understand the target hypothesis\. The drug is Belumosudil, which is an inhibitor/antagonist targeting ROCK2 and ROCK1\. The cell line is HCT15, which has a frameshift mutation in ACVR2A, a suppressor gene\. The gene in question is SGMS2, and the hypothesis is that Belumosudil downregulates it\. To support this, the experiment needs to have a padj < 0\.05 and \|log2FC\| \> 0\.5 in the direction of downregulation\.Now, looking at the context\. There are several experiments with Belumosudil \(same drug\) in different cell lines\. Let’s check those first\.Context \[3\] is Belumosudil in HOP62 cell line\. The result here is a log2FC of \+0\.833, which is a strong upregulation \(padj is very low, 1\.67e\-17\)\. So this is a clear upregulation of SGMS2\. But the target hypothesis here is a downregulation\. That’s a contradiction\. However, the cell line here is HOP62, which is different from HCT15\.Context \[6\] is Belumosudil in HT\-29, another cell line\. The log2FC here is \+0\.146, which is a slight upregulation, and the padj is 0\.04521 \(just under 0\.05, so it’s significant\)\. But again, it’s an upregulation, not downregulation\. So again, the same drug in different cell lines shows upregulation\.Then there’s context \[7\], which is a different drug \(Adenosine\) in the same HCT15 cell line\. The log2FC here is \+0\.142, which is a slight upregulation, but the padj is 0\.7822, which is not significant\. So that doesn’t support either direction\.Other contexts \[1\], \[2\], \[4\], \[5\] are unrelated drugs, but maybe they can give some general idea\. For example, \[4\] is Artesunate \(same mechanism as Belumosudil? Or not sure\)\. The result here is \+0\.20, which is upregulation, but not significant \(padj 0\.8409\)\. Not helpful here\.Now, the key is to see if there’s any evidence in the context that supports downregulation of SGMS2 in HCT15 or similar cell lines\. The context for Belumosudil in different cell lines \(3 and 6\) both show upregulation, but the target hypothesis is about downregulation\. So this might suggest that the drug is more likely to upregulate, which would make the target hypothesis less probable\.But wait, the context for HCT15 with Adenosine \(context \[7\]\) doesn’t show significant upregulation\. So maybe the gene is not always responsive, but the drug in other cell lines does show upregulation\. However, the target hypothesis is about HCT15\. Since the context for Belumosudil in other cell lines shows upregulation, but the target hypothesis is about a different direction \(downregulation\), that might suggest that the target hypothesis is less likely\.However, I should also check if there’s any evidence in the context that directly contradicts the target hypothesis\. For example, if other experiments in the same cell line or similar cell lines show downregulation, but in this case, none of the other contexts for HCT15 show that\. The same drug in other cell lines shows upregulation, and the unrelated drug in HCT15 \(context \[7\]\) isn’t significant\.But I need to consider the strength of the context\. The context for the same drug \(Belumosudil\) in different cell lines \(HOP62 and HT\-29\) both show upregulation\. That’s a pattern, but the target hypothesis here is for HCT15\. The question is whether this pattern in other cell lines supports or contradicts the hypothesis here\.Another thing to consider is the gene’s responsiveness\. The gene SGMS2 is being tested in multiple contexts, and in the same drug’s context, it’s upregulated in different cell lines\. That suggests that the gene might be a consistent target for this drug\. However, the target hypothesis here is the opposite\. So this would make the probability of the target hypothesis being supported lower\.But what about the other contexts? For example, context \[1\] and \[2\] are unrelated drugs, but they also show upregulation\. However, those are not directly relevant\. The key is the same drug in different cell lines\.Also, the base mean in the context experiments for the same drug \(context \[3\] and \[6\]\) are higher, which might indicate more reliable data\. But the target hypothesis here is for HCT15\.So, considering the context, the same drug in different cell lines shows upregulation of SGMS2\. The target hypothesis is that it’s downregulated in HCT15\. Therefore, the context suggests that the drug is more likely to upregulate, which would make the target hypothesis \(downregulation\) less probable\. However, the context doesn’t have any direct evidence for the same cell line \(HCT15\) and the same drug\.But the problem is to weigh the evidence\. The fact that in two different cell lines, the same drug upregulates the gene might suggest that the gene is a target of the drug\. If the target hypothesis here is in the opposite direction, that might lower the probability\. However, there’s no direct evidence against it\.But in the context, the same drug in different cell lines shows upregulation, which is a positive sign for the direction\. But the target hypothesis here is for downregulation, which is opposite\. So that would make the probability of the target hypothesis being supported lower\.But I need to be careful here\. Are there any other factors? For example, the driver mutations in the cell lines\. The HCT15 cell line has a LoF mutation in ACVR2A\. But the other cell lines in context \[3\] \(HOP62\) has a B2M mutation \(suppressor\), and HT\-29 \(context \[6\]\) has APC \(LoF\)\. So the cell lines vary in their driver mutations\. But the same drug in different cell lines still shows upregulation\.Therefore, based on the context, the evidence for the same drug in other cell lines showing upregulation of SGMS2 would make the target hypothesis \(downregulation\) less likely\. However, the context doesn’t have any experiments that directly contradict it\.But how does that translate into a probability? If the context shows that in two different cell lines, the same drug upregulates the gene, that’s a pattern\. The target hypothesis is the opposite\. So the probability might be low, say 0\.2 or 0\.3\. But I need to think carefully\.Alternatively, maybe the context doesn’t have enough evidence to make a strong conclusion\. However, since the same drug in other cell lines shows upregulation, it’s more likely that the gene is upregulated, not downregulated\. Therefore, the probability that the target hypothesis \(downregulation\) is supported is low\.But I need to consider the possibility that the target hypothesis could still be correct\. For example, if there’s a cell line\-specific effect\. But the context doesn’t have any evidence for that here\.So, based on the context provided, I would say that the probability is around 0\.2 \(20% chance that the target hypothesis is supported\)\. But I need to be precise here\.

Answer<probability\>0\.20</probability\>

## Appendix KAdditional analyses of belief revision

This appendix reports the two probes of belief revision summarized in[§5](https://arxiv.org/html/2610.41015#S5.SS0.SSS0.Px3), Analysis 3\. Both use the multi\-domain test set and the current roster, with256256episodes per domain\.

The preceding analyses ask whether individual updates have sensible signs and magnitudes; these probes ask whether a model combines the same evidence consistently\. If the stated probabilities behave like beliefs, rearranging a fixed evidence set should not materially change the result, and the joint weight of evidence of a matched pair should be predictable from the two single\-stub weights\. Both probes hold evidence content fixed while changing its position or composition\.

Theposition probeplaces one compelling stub at four locations in an otherwise identical ten\-stub context\. The reported position\-1 bias is the belief when that stub appears first minus the mean belief when it appears later\. Thecomposition probepresents a supporting stub, a constructed equal\-and\-opposite refuting stub, and the pair; its residual measures departure from adding the two single\-stub weights of evidence; it covers Tahoe and DepMap only\.[SectionsK\.1](https://arxiv.org/html/2610.41015#A11.SS1)and[K\.2](https://arxiv.org/html/2610.41015#A11.SS2)give the construction and results of each probe, and[Fig\.12](https://arxiv.org/html/2610.41015#A11.F12)summarizes both\.

Figure 12:Ordering effects: Given the same evidence, are models consistent in belief revision?Each point is one model; the schematic at right shows the two manipulations\.Position\-1 bias\(vertical\): a single “compelling” evidence stub \(supporting and highly\-relevant\) is placed at varying positions within an otherwise identical context\. We report the stated belief when the stub appears first minus the mean stated belief when it appears in the later three positions, in decibans\. Positive values indicate that the same stub moves the belief further when it appears first, a form of primacy bias\.Belief\-consistency violation\(horizontal\): a matched pair of supporting/refuting stubs is constructed by inverting the direction of effect while keeping all other details the same, and we measure how far the joint update departs from the product of the two single\-stub updates\. Zero on either axis is the unbiased value, so the lower left is best\. Arrows join the two matched pairs that differ only in whether a reasoning trace is verbalized\. Across both measures, thinking reduces bias\. Ibis0 \(classifier\) is well calibrated yet shows pronounced bias on both axes, whereas RL post\-training improves calibrationandreduces the observed biases\.Consistency does not follow from aggregate forecast quality \([Fig\.12](https://arxiv.org/html/2610.41015#A11.F12)\): the classifier has the best Brier score of the Ibis0 models but the largest position\-1 bias \(\+1\.01\+1\.01dB\) and composition residual \(4\.164\.16dB\)\. Post\-training still improves consistency: all three trained variants reduce the Qwen3 \(non\-thinking\) composition residual of7\.547\.54dB\. The classifier reaches4\.164\.16dB, and the two RL policies reach0\.940\.94dB \(Ibis0 non\-thinking\) and0\.980\.98dB \(Ibis0 thinking\), in the range of the frontier models \(GPT\-5\.6\-sol0\.260\.26dB, Claude Opus 51\.551\.55dB\)\.

Reasoning traces are associated with a separate shift in position dependence\. Within both the Qwen3 and Ibis0 pairs, enabling a trace reduces position\-1 bias by about0\.50\.5dB\. Because the same shift appears before and after post\-training, it is more naturally associated with the response process than with the proper\-scoring objective alone\.

### K\.1Composition of supporting and refuting evidence

The conflict probe selects the clearest supporting stub in an episode and constructs a matched refuting stub of equal magnitude\. In Tahoe this reverses the signs of the differential\-expression fold change and Wald statistic, leaving the adjustedpp\-value unchanged\. In DepMap it reflects the Chronos gene effect about zero and reads the accompanying dependency probability off the empirical relation between the two quantities in the test set, so that both numbers on the rendered card move together\. GWAS is excluded because the catalog records published associations but not failures to associate, which a refuting stub would require\. The same property restricts the directional analysis in[§5](https://arxiv.org/html/2610.41015#S5)\.

The supporting stub, the refuting stub, and the pair are presented separately\. The pair is presented in both orders and the two are averaged, so that order effects cancel and do not enter the residual\. Departure from additive weight\-of\-evidence updating is measured as

\|log⁡BFboth−log⁡\(BFsupport​BFrefute\)\|,\\left\|\\log\\mathrm\{BF\}\_\{\\mathrm\{both\}\}\-\\log\\left\(\\mathrm\{BF\}\_\{\\mathrm\{support\}\}\\mathrm\{BF\}\_\{\\mathrm\{refute\}\}\\right\)\\right\|,where each Bayes factor is taken relative to the same episode with no evidence\. Zero denotes exact agreement between the paired update and the sum of the two single\-stub updates\. The reference for each model is therefore its own single\-stub responses\.

[Fig\.13](https://arxiv.org/html/2610.41015#A11.F13)reports the result over256256episodes in each of Tahoe and DepMap, grouped by training regime\. Within Qwen3, the violation is7\.547\.54dB\[6\.03,9\.04\]\[6\.03,9\.04\]in non\-thinking mode and2\.872\.87dB\[2\.58,3\.16\]\[2\.58,3\.16\]in thinking mode\. Among the Ibis0 models it is0\.940\.94dB\[0\.75,1\.13\]\[0\.75,1\.13\]for Ibis0 \(non\-thinking\) and0\.980\.98dB\[0\.80,1\.15\]\[0\.80,1\.15\]for Ibis0 \(thinking\), which differ from each other by less than their intervals, and4\.164\.16dB\[3\.76,4\.55\]\[3\.76,4\.55\]for Ibis0 \(classifier\)\. The two frontier models give0\.260\.26dB\[0\.01,0\.75\]\[0\.01,0\.75\]for GPT\-5\.6\-sol and1\.551\.55dB\[1\.27,1\.85\]\[1\.27,1\.85\]for Claude Opus 5\.

The clearest effect in the figure is that both reinforcement\-learning policies are far more consistent than the Qwen3 checkpoint they were trained from\. The classifier is not: it violates consistency more than either frontier model and both policies, while achieving the lowest Brier score of any model in[§4](https://arxiv.org/html/2610.41015#S4)\. Post\-training against a proper scoring rule therefore does not improve this property on its own, and the output channel separates the models that share a training objective\. GPT\-5\.6\-sol is the most consistent model measured, so post\-training on EDB\-Bio is not required to obtain the property\.

Figure 13:Composition of conflicting evidence\.Absolute departure from additive weight\-of\-evidence updating for a supporting stub and its constructed equal\-and\-opposite counterpart, macro\-averaged over Tahoe and DepMap\. Panels group models by training regime and bars are ordered within each panel; the vertical scale is shared\. Intervals are95%95\\%episode bootstrap intervals with episodes resampled within domain,256256episodes per domain\. Lower values indicate closer agreement with multiplicative combination of Bayes factors\.
### K\.2Dependence of evidence updates on sequence position

The position probe inserts the same evidence stub at positions11,44,77, and1010of a ten\-stub context and fills the remaining slots with the same neutral padding in the same order\. The four conditions contain the same evidence in different arrangements, and an order\-invariant model reports the same probability in all four\. The context holds ten stubs rather than the sixteen of the earlier pilot because DepMap episodes carry about eleven stubs that can serve as padding; a sixteen\-stub context can be constructed for only1414of2,0002\{,\}000episodes\.

Reported values are the stated belief in decibans, centred within episode across the four positions to remove variation in how compelling each episode’s probe stub is\. The statistic of interest is the belief at the first position minus the mean over the later three\. Centring cancels in any contrast between positions, so this quantity is a raw within\-episode difference in log\-odds and does not depend on which positions were sampled\. A contrast between the first and last positions alone would not suffice: for Ibis0 \(non\-thinking\) the last position is as elevated as the first, and the two cancel\.

[Table10](https://arxiv.org/html/2610.41015#A11.T10)gives the statistic for each model\. It is positive and separated from zero for Ibis0 \(classifier\), both modes of Qwen3, and Ibis0 \(non\-thinking\), so in those four models the same stub moves the belief further when it appears first\. It is negative and separated from zero for Ibis0 \(thinking\) and Claude Opus 5, and contains zero for GPT\-5\.6\-sol\.

Table 10:Dependence of the stated belief on the first evidence position\.Stated belief at position11minus the mean over positions44,77and1010, in decibans, macro\-averaged over the three domains with256256episodes per domain\. Positive values indicate that the same stub moves the belief further when it appears first\. Intervals are95%95\\%episode bootstrap intervals resampled within domain; bold marks an interval excluding zero\.The two matched pairs that differ only in whether the model verbalizes a reasoning trace move together\. Paired within episode over768768episodes, adding the trace changes first\-position bias by−0\.49\-0\.49dB\[−0\.96,−0\.01\]\[\-0\.96,\-0\.01\]for Qwen3 and by−0\.45\-0\.45dB\[−0\.54,−0\.36\]\[\-0\.54,\-0\.36\]for the reinforcement\-learning policy\. The two estimates agree to within0\.040\.04dB although only one pair is trained, which suggests that deliberating before answering and training against a proper scoring rule act on this property separately\. Training reduces the effect from\+0\.69\+0\.69dB for Qwen3 \(non\-thinking\) to\+0\.18\+0\.18dB for Ibis0 \(non\-thinking\), and the trace shifts it downward from that level; in the reinforcement\-learning pair, the shift passes zero into a small preference for later positions\. The Qwen3 interval only just excludes zero, so the paired result rests mainly on the policy pair\.

The classifier, the only model that states no probability in text, shows the largest effect in the roster\. It also achieves the lowest Brier score in[§4](https://arxiv.org/html/2610.41015#S4)\.

Figure 14:Dependence of the stated belief on evidence position\.The same evidence stub placed at four positions of a ten\-stub context, with the remaining slots held to identical neutral padding in identical order\. Panels group models that differ in whether a reasoning trace is verbalized\. Values are decibans, centred within episode, macro\-averaged over the three domains with episodes resampled within domain; bands are95%95\\%intervals,256256episodes per domain\. Legend entries give the position\-1 bias of[Table10](https://arxiv.org/html/2610.41015#A11.T10)\. A flat line at zero indicates a belief that does not depend on the order of the evidence\.

## Appendix LStructured and numeric\-only baselines on Tahoe

Rhaister\[[76](https://arxiv.org/html/2610.41015#bib.bib20)\]predicts a differential\-expression outcome for a specified cell, treatment, and gene\. We convert its predicted log fold change to a probability by assigning its sign relative to the target direction and applying a sigmoid, and also report a hard\-thresholded version\. The published episodes do not record drug concentration, so treatment identity is defined by drug\. Rhaister can provide a forecast only when its fit data contain the target gene under a compatible treatment\.

The*prompt\-local*setting fits Rhaister only to the evidence stubs shown in the prompt\. The*all\-context*setting uses training\-split target rows and all structured context evidence, with held\-out target rows excluded\. A numeric\-only gradient\-boosted tree is trained on the Tahoe training split using DESeq2 summary statistics and counts, without gene, drug or cell\-line identities or text; it produces a forecast for every test episode\.

Table 11:Tahoe\-only structured baselines on the2,0002\{,\}000\-episode test set\. Coverage is the fraction of episodes for which a baseline produces a forecast\.
## Appendix MExtended related work

This appendix expands the summary in[§7](https://arxiv.org/html/2610.41015#S7)\.

#### Scientific reasoning and evidence benchmarks\.

Benchmarks for scientific AI have moved from expert\-written question answering\[[31](https://arxiv.org/html/2610.41015#bib.bib86)\]toward tasks drawn from research practice: long\-context reasoning over scientific documents\[[77](https://arxiv.org/html/2610.41015#bib.bib88)\], research coding\[[32](https://arxiv.org/html/2610.41015#bib.bib87)\], data\-driven discovery from datasets\[[33](https://arxiv.org/html/2610.41015#bib.bib91),[34](https://arxiv.org/html/2610.41015#bib.bib93)\], simulated environments in which an agent must form hypotheses and run experiments\[[35](https://arxiv.org/html/2610.41015#bib.bib89),[36](https://arxiv.org/html/2610.41015#bib.bib90)\], and biology research workflows spanning literature retrieval, protocol troubleshooting, and study\-scale computational analysis\[[78](https://arxiv.org/html/2610.41015#bib.bib80),[37](https://arxiv.org/html/2610.41015#bib.bib83),[79](https://arxiv.org/html/2610.41015#bib.bib81),[38](https://arxiv.org/html/2610.41015#bib.bib84)\]\. Literature agents answer questions by retrieving and synthesizing published evidence\[[80](https://arxiv.org/html/2610.41015#bib.bib82)\]\. Closest in form to our task is claim verification, where SciFact and SciFact\-Open ask whether a retrieved abstract supports or refutes a claim\[[39](https://arxiv.org/html/2610.41015#bib.bib78),[81](https://arxiv.org/html/2610.41015#bib.bib79)\]and Matter\-of\-Fact asks whether a literature\-supported claim is feasible\[[40](https://arxiv.org/html/2610.41015#bib.bib92)\]\. These tasks score a categorical verdict about existing evidence\. EDB\-Bio instead asks for a probability that a held\-out measurement will support the hypothesis, scores it with a proper scoring rule, and controls the relevance and sufficiency of the evidence pool so that belief revision can be measured update by update\.

#### Calibration and confidence estimation in LLMs\.

Work on LLM calibration has centered on confidence in answer*correctness*for factual or multiple\-choice question answering, where large models can be reasonably calibrated when probabilities are read from token logits or elicited asP⁡\(true\)P\(\\text\{true\}\)\[[41](https://arxiv.org/html/2610.41015#bib.bib21),[82](https://arxiv.org/html/2610.41015#bib.bib22),[83](https://arxiv.org/html/2610.41015#bib.bib23),[42](https://arxiv.org/html/2610.41015#bib.bib24)\]\. A recurring finding is that alignment \(RLHF and instruction tuning\) degrades this calibration and leaves models systematically overconfident\[[43](https://arxiv.org/html/2610.41015#bib.bib25),[44](https://arxiv.org/html/2610.41015#bib.bib26)\], with a corresponding risk that users over\-trust confident\-sounding output\[[84](https://arxiv.org/html/2610.41015#bib.bib27)\]\. Proposed remedies span verbalized confidence elicitation\[[85](https://arxiv.org/html/2610.41015#bib.bib30),[86](https://arxiv.org/html/2610.41015#bib.bib31)\], logit\- and sequence\-likelihood scores\[[87](https://arxiv.org/html/2610.41015#bib.bib32)\], sampling\-consistency measures\[[88](https://arxiv.org/html/2610.41015#bib.bib33),[89](https://arxiv.org/html/2610.41015#bib.bib34)\], semantic\-entropy estimators\[[90](https://arxiv.org/html/2610.41015#bib.bib35),[91](https://arxiv.org/html/2610.41015#bib.bib36)\], and auxiliary or post\-hoc calibrators\[[92](https://arxiv.org/html/2610.41015#bib.bib37),[93](https://arxiv.org/html/2610.41015#bib.bib38),[73](https://arxiv.org/html/2610.41015#bib.bib3)\]\. Test\-time reasoning does not reliably improve calibration and can worsen it\[[94](https://arxiv.org/html/2610.41015#bib.bib39)\], which motivates our direct\-mode elicitation of a single probability\. The task closest to ours is probabilistic forecasting scored by strictly proper rules\[[14](https://arxiv.org/html/2610.41015#bib.bib14),[18](https://arxiv.org/html/2610.41015#bib.bib1),[28](https://arxiv.org/html/2610.41015#bib.bib2)\], where LLMs now approach human\-crowd accuracy on real\-world events\[[95](https://arxiv.org/html/2610.41015#bib.bib40),[96](https://arxiv.org/html/2610.41015#bib.bib41)\]\. We elicit calibrated probabilities for*novel scientific hypotheses*under a controlled, manipulable evidence pool, and report discrimination \(AUROC\) and calibration \(ECE\[[97](https://arxiv.org/html/2610.41015#bib.bib42)\]\) separately rather than accuracy alone\.

#### Proper\-scoring post\-training\.

A recent line of work post\-trains language models against strictly proper scoring rules\[[14](https://arxiv.org/html/2610.41015#bib.bib14)\], using the Brier score or log loss of a verbalized confidence as a reinforcement\-learning reward or a supervised objective\[[6](https://arxiv.org/html/2610.41015#bib.bib44),[7](https://arxiv.org/html/2610.41015#bib.bib45),[15](https://arxiv.org/html/2610.41015#bib.bib43),[16](https://arxiv.org/html/2610.41015#bib.bib28),[8](https://arxiv.org/html/2610.41015#bib.bib29)\]\. These methods target confidence in the correctness of an answer or a reasoning chain on question\-answering tasks, so the label is whether the model’s own output is right\. We apply the same family of objectives to a target external to the model: the outcome of a held\-out experiment, conditioned on an evidence pool\. The model must weigh evidence instead of assessing its own reliability, and the reward comes from resolved scientific outcomes\. We compare a negative\-Brier reward in direct and thinking modes with binary cross\-entropy on a scalar head \([§3](https://arxiv.org/html/2610.41015#S3)\)\.

#### Bayesian experimental design with LLMs\.

Classical Bayesian experimental design chooses the experiment that maximizes expected information gain \(EIG\)\[[45](https://arxiv.org/html/2610.41015#bib.bib48),[98](https://arxiv.org/html/2610.41015#bib.bib49),[9](https://arxiv.org/html/2610.41015#bib.bib15)\], but estimating EIG is costly, requiring nested Monte\-Carlo integration\[[46](https://arxiv.org/html/2610.41015#bib.bib50)\], variational or contrastive bounds\[[99](https://arxiv.org/html/2610.41015#bib.bib51),[100](https://arxiv.org/html/2610.41015#bib.bib52)\], or amortized design policies\[[47](https://arxiv.org/html/2610.41015#bib.bib53),[101](https://arxiv.org/html/2610.41015#bib.bib54),[102](https://arxiv.org/html/2610.41015#bib.bib55)\]\. A growing line instead drives information\-theoretic selection from a language model’s own beliefs: LLMs pick EIG\-maximizing queries for interactive information gathering\[[48](https://arxiv.org/html/2610.41015#bib.bib59),[103](https://arxiv.org/html/2610.41015#bib.bib60),[49](https://arxiv.org/html/2610.41015#bib.bib61),[50](https://arxiv.org/html/2610.41015#bib.bib58),[51](https://arxiv.org/html/2610.41015#bib.bib68)\], elicit user intent by asking informative questions\[[104](https://arxiv.org/html/2610.41015#bib.bib62),[105](https://arxiv.org/html/2610.41015#bib.bib63)\], serve as uncertainty\-bearing surrogates for Bayesian optimization in scientific domains\[[106](https://arxiv.org/html/2610.41015#bib.bib64),[107](https://arxiv.org/html/2610.41015#bib.bib65)\], and plan iterative perturbation screens\[[52](https://arxiv.org/html/2610.41015#bib.bib66),[53](https://arxiv.org/html/2610.41015#bib.bib67)\]\. Prior studies also report that direct prompting for information gathering can reduce performance\[[108](https://arxiv.org/html/2610.41015#bib.bib69)\]and that off\-the\-shelf agents may ask uninformative questions\[[51](https://arxiv.org/html/2610.41015#bib.bib68)\]\. The multi\-domain evaluation in[§6](https://arxiv.org/html/2610.41015#S6)applies model\-based EIG selection to evidence\-deliberation episodes from Tahoe, DepMap, and GWAS\.

#### Belief revision, order effects, and weights of evidence\.

Whether language models revise beliefs coherently has been studied through cognitive\-bias probes and Bayesian\-updating tasks\. Models show primacy effects in multiple\-choice and long\-context settings\[[54](https://arxiv.org/html/2610.41015#bib.bib94),[55](https://arxiv.org/html/2610.41015#bib.bib95),[109](https://arxiv.org/html/2610.41015#bib.bib99)\]and anchor on initial numeric hints\[[56](https://arxiv.org/html/2610.41015#bib.bib96)\]; surveys catalog these and other human\-like biases\[[110](https://arxiv.org/html/2610.41015#bib.bib101)\]\. On sequential evidence, models approach Bayesian posteriors given enough observations\[[57](https://arxiv.org/html/2610.41015#bib.bib97)\]but discount earlier evidence relative to an ideal filter\[[111](https://arxiv.org/html/2610.41015#bib.bib100)\]and depart from Bayesian invariances even when instructed to follow them\[[58](https://arxiv.org/html/2610.41015#bib.bib98)\]\. We measure updates in decibans, the unit of weight of evidence that Good and Turing introduced for sequential evidence\[[11](https://arxiv.org/html/2610.41015#bib.bib71),[112](https://arxiv.org/html/2610.41015#bib.bib103)\], and summarize each update as a log Bayes factor\[[59](https://arxiv.org/html/2610.41015#bib.bib102)\]\. Whether a model’s weights of evidence are the right size, the question of[§5](https://arxiv.org/html/2610.41015#S5), has to our knowledge not been asked of language models before; the nearest prior work is support theory’s account of how human probability judgments depend on how the evidence is described\[[113](https://arxiv.org/html/2610.41015#bib.bib13)\]\.

#### Prediction of perturbation outcomes in biology\.

Predicting the transcriptional response to a perturbation is an active area\. GEARS uses gene\-interaction graphs to extrapolate to unseen combinations\[[114](https://arxiv.org/html/2610.41015#bib.bib104)\], and single\-cell foundation models such as Geneformer and scGPT are fine\-tuned for the task\[[115](https://arxiv.org/html/2610.41015#bib.bib106),[116](https://arxiv.org/html/2610.41015#bib.bib105)\]\. Systematic benchmarks find that these models do not yet outperform simple linear baselines\[[117](https://arxiv.org/html/2610.41015#bib.bib107),[118](https://arxiv.org/html/2610.41015#bib.bib108)\]\. The virtual\-cell program aims at models that predict perturbed cell state across contexts\[[119](https://arxiv.org/html/2610.41015#bib.bib109),[120](https://arxiv.org/html/2610.41015#bib.bib110),[121](https://arxiv.org/html/2610.41015#bib.bib111)\], and perturbation atlases such as Tahoe\-100M supply the training data\[[10](https://arxiv.org/html/2610.41015#bib.bib11)\]\. These models predict the measurement itself from molecular inputs\. EDB\-Bio asks a different question: given prior measurements as text, how confident should a reader be that the next one will support a hypothesis\. The two are complementary, and a perturbation model could serve as the outcome simulator in our experimental\-design loop \([§6](https://arxiv.org/html/2610.41015#S6)\)\.

#### AI for science and autonomous discovery\.

Language\-model agents now automate parts of the research cycle, and the systems differ mainly in which parts\. One line couples a model to laboratory automation so that it plans and executes experiments\[[1](https://arxiv.org/html/2610.41015#bib.bib112)\]\. A second generates and ranks hypotheses or research proposals, in biomedicine\[[5](https://arxiv.org/html/2610.41015#bib.bib118)\]and in machine learning, where the loop extends to running the experiments and writing the paper\[[2](https://arxiv.org/html/2610.41015#bib.bib113),[3](https://arxiv.org/html/2610.41015#bib.bib114)\]\. A third organizes several agents into a research team that iterates between hypothesis, experiment, and analysis, with wet\-lab validation of the resulting designs and candidate treatments\[[122](https://arxiv.org/html/2610.41015#bib.bib115),[123](https://arxiv.org/html/2610.41015#bib.bib116)\], and recent systems extend this loop to long\-horizon, data\-driven discovery\[[4](https://arxiv.org/html/2610.41015#bib.bib70)\]\. Each of these systems must decide which hypotheses to pursue and when the evidence suffices, decisions that depend on the model’s probabilistic judgment\. EDB\-Bio measures that judgment in isolation, and the experimental\-design results in[§6](https://arxiv.org/html/2610.41015#S6)show one way to use calibrated belief for better decisions inside such a loop\.