推理型大语言模型中的摊销式序贯实验设计

# Amortizing Sequential Experimental Design in Reasoning LLMs
Source: [https://arxiv.org/html/2610.20832](https://arxiv.org/html/2610.20832)
Antonin Berthon11footnotemark:1Nicolás AstorgaMihaela van der SchaarAffiliation:University of CambridgeAffiliation:Cambridge, UKEmail:[\{ak2680,armb3\}@cam\.ac\.uk](mailto:)

###### Abstract

Despite their extensive pretraining knowledge and strong reasoning capabilities, large language models \(LLMs\) struggle to adaptively gather information in experimental design tasks\. We study how to*amortize sequential Bayesian experimental design \(BED\) into an LLM*by explicitly training the reasoning that guides action selection using information\-gain rewards\. Our approach specifies the experimental generative model independently of the LLM’s own beliefs, allowing us to optimize expected information gain under a model of our choosing\. We investigate the design choices for amortization, including alternative generative\-model constructions, reward estimation, hypothesis\-space specification, multi\-step objectives, and computational costs relative to training\- and inference\-time BED baselines\. On Twenty Questions, the resulting policies improve information acquisition and task performance, outperform inference\- and training\-time BED baselines at substantially lower computational cost, and generalize to unseen categories\. We further demonstrate positive transfer from Twenty Questions training to Medi\-Q, a real\-world clinical dialogue benchmark\. These results suggest that information gathering can be learned as a generalizable reasoning skill in LLMs\.

## 1Introduction

Many real\-world problems are not fully specified upfront, yet humans are remarkably effective at gathering the information needed to solve them\([Li et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib13);[Huang et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib12)\)\. Consider a clinician diagnosing a patient\. They draw on their medical knowledge to ask about relevant symptoms, think about possible diagnoses, and identify tests that can distinguish between them\. As new evidence is acquired, they refine their hypotheses to decide what to investigate next\. Effective information gathering thus requires both the prior knowledge to inform decision\-making and the ability to continuously adapt those decisions in light of new evidence\.

Given their extensive pretraining knowledge and strong reasoning capabilities, large language models \(LLMs\) are increasingly being explored as agents in adaptive decision\-making\([Zhou et al\., 2025a](https://arxiv.org/html/2610.20832#bib.bib3);[Ashokkumar et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib5)\)\. Despite this potential, LLMs have been shown to perform poorly in multi\-turn settings\([Krishnamurthy et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib2);[Zhou et al\., 2025b](https://arxiv.org/html/2610.20832#bib.bib15);[Li et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib16);[Laban et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib14)\)\. In particular,[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib6)demonstrate that LLMs are insensitive to experimental feedback, suggesting that their performance relies more on their \(potentially inaccurate\) prior knowledge than on their ability to adapt their reasoning to new evidence\.

In this work,we investigate how to improve LLMs’ ability to adaptively gather information\. Rather than using an LLM to sample informative actions\([Kobalczyk et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib9);[Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8);[Hartmann et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib10)\), we hypothesize that learning generalizable information gathering strategies requires explicitly training the reasoning process that guides action selection\. We formalize this problem through sequential Bayesian experimental design \(BED\)[Rainforth et al\. \(2023\)](https://arxiv.org/html/2610.20832#bib.bib4), which uses a generative model of the experiment to select actions that reduce uncertainty about an unknown quantity of interest\. We investigate how to amortize this information\-theoretic objective into an LLM reasoning policy using reinforcement learning with verifiable rewards \(RLVR\)\. Crucially, training\-time amortization allows us to specify the experimental generative modelindependentlyof the LLM’s own beliefs\. This means that we can evaluate actions based on their expected information gain under a model of our choosing\. We additionally investigate whether optimizing information gain over multiple interactions encourages non\-myopic information\-gathering strategies\.

We empirically show that amortizing BED into a reasoning LLM develops strategies that generalize to previously unseen domains and tasks\. Specifically, we train reasoning LLMs on Twenty Questions and demonstrate improved information acquisition across both seen and unseen categories\. We show that models trained in this environment exhibit positive transfer to the real\-world clinical dialogue benchmark Medi\-Q\([Li et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib13)\)\. Our results show that decoupling the experimental generative model from the LLM’s beliefs improves information acquisition\. We further find that rewarding expected information\-gain rewards under our specified generative model improves learning relative to realized rewards, and that optimizing information acquisition over future interactions can improve sequential information gathering in non\-myopic environments\.

Contributions\.Our contributions are:

- ▶\\blacktrianglerightWe formulate amortized sequential BED for reasoning LLMs and study how to construct its generative model and information\-gain reward \(Sec\.[4](https://arxiv.org/html/2610.20832#S4)\)\.
- ▶\\blacktrianglerightWe demonstrate that information\-gathering behavior can be amortized into a reasoning LLM, improving information acquisition and transferring across unseen domains and to a different information\-gathering task \(Sec\.[5](https://arxiv.org/html/2610.20832#S5)\)\.
- ▶\\blacktrianglerightWe systematically study the design of amortized BED, including alternative generative\-model constructions, reward estimation, hypothesis\-space specification, multi\-step objectives, and their computational cost relative to training\- and inference\-time BED baselines \(Sec\.[5](https://arxiv.org/html/2610.20832#S5)\)\.

## 2Background & Problem Setup

### 2\.1Bayesian Experimental Design

Letθ\\thetadenote an unknown variable that we wish to learn about\. To gain information aboutθ\\theta, we can select an experimental designxx, which results in an observationyy\. For example, in a diagnostic setting,θ\\thetamay represent a patient’s underlying condition that we want to identify,xxa diagnostic test, andyythe resulting test outcome\. Because experiments may be subject to constraints such as cost or time, the chosen designxxshould be as informative as possible\.

BED provides a framework for quantifying how informative a potential designxxis using a generative model of the experiment,p⁡\(θ,y∣x\)=p⁡\(θ\)​p​\(y∣θ,x\)p\(\\theta,y\\mid x\)=p\(\\theta\)p\(y\\mid\\theta,x\)\. Here, the priorp⁡\(θ\)p\(\\theta\)represents our initial beliefs aboutθ\\thetaand the likelihoodp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)describes the probability of observingyygivenθ\\thetaandxx\. After selectingxxand observingyy, these beliefs are updated through the posteriorp⁡\(θ∣y,x\)p\(\\theta\\mid y,x\)\. The informativeness of a potential designxxis then quantified by its expected information gain \(EIG\):

EIG⁡\(x\)\\displaystyle\\mathrm\{EIG\}\(x\)=𝔼p⁡\(θ\)​p​\(y∣θ,x\)​\[log⁡p⁡\(θ∣y,x\)−log⁡p⁡\(θ\)\]\\displaystyle=\\mathbb\{E\}\_\{p\(\\theta\)p\(y\\mid\\theta,x\)\}\\left\[\\log p\(\\theta\\mid y,x\)\-\\log p\(\\theta\)\\right\]\(1\)=𝔼p⁡\(θ\)​p​\(y∣θ,x\)​\[log⁡p⁡\(y∣θ,x\)−log⁡p⁡\(y∣x\)\],\\displaystyle=\\mathbb\{E\}\_\{p\(\\theta\)p\(y\\mid\\theta,x\)\}\\left\[\\log p\(y\\mid\\theta,x\)\-\\log p\(y\\mid x\)\\right\],\(2\)which measures how much information designxxwill provide aboutθ\\theta, in expectation over possibleθ\\thetaand observationsyy\. Using Bayes’ rule, the EIG can be seen as either the expected reduction in uncertainty aboutθ\\thetaafter observingyy\(Eq\.[1](https://arxiv.org/html/2610.20832#S2.E1)\) or the expected reduction in uncertainty aboutyyfrom knowingθ\\theta\(Eq\.[2](https://arxiv.org/html/2610.20832#S2.E2)\)\.

The optimal one\-step design is defined asx⋆=arg⁡maxx∈𝒳⁡EIG⁡\(x\)x^\{\\star\}=\\arg\\max\_\{x\\in\\mathcal\{X\}\}\\mathrm\{EIG\}\(x\), where𝒳\\mathcal\{X\}denotes the space of designs\. This formulation extends naturally to the sequential setting where the design at stepttis chosen based on the history of previous experiments,ht−1=\(x1,y1,…,xt−1,yt−1\)h\_\{t\-1\}=\(x\_\{1\},y\_\{1\},\\ldots,x\_\{t\-1\},y\_\{t\-1\}\)by maximizing the EIG under the updated beliefp⁡\(θ∣ht−1\)p\(\\theta\\mid h\_\{t\-1\}\)\. The experimental design at stepttcan then be selected asxt⋆=arg⁡maxx∈𝒳⁡EIG⁡\(x,ht−1\)x\_\{t\}^\{\\star\}=\\arg\\max\_\{x\\in\\mathcal\{X\}\}\\mathrm\{EIG\}\(x;h\_\{t\-1\}\)\.

Challenges of Design Selection\.Selecting designs according to the EIG in adaptive settings presents two major challenges\. First, optimizing the EIG requires a costly estimation of the posteriorp\(θ∣x1:t,y1:t\)p\(\\theta\\mid x\_\{1:t\},y\_\{1:t\}\)at each step, which depends on the observed interaction history and therefore cannot be computed in advance\. Moreover, even given this posterior, calculating the EIG is doubly intractable: it involves an outer expectation over possible latent states and observations, and an inner marginalization over latent states to compute the posterior predictive distributionp⁡\(yt∣xt,ht−1\)p\(y\_\{t\}\\mid x\_\{t\},h\_\{t\-1\}\)\. Repeating this estimation and optimization procedure after every observation can be prohibitively expensive for settings that require decisions to be made in real time\. Second, one\-step EIG optimization considers only the immediate next design\. This can be suboptimal in non\-myopic decision states, where acquiring more information in the future requires first selecting a design with limited immediate information gain\.

### 2\.2Amortizing Sequential Experimental Design

To address the computational cost of optimizing the EIG at inference time, Deep Adaptive Design \(DAD,[Foster et al\. \(2021\)](https://arxiv.org/html/2610.20832#bib.bib7)\) amortizes sequential experimental design into a design policyπϕ​\(xt∣ht−1\)\\pi\_\{\\phi\}\(x\_\{t\}\\mid h\_\{t\-1\}\), which maps the historyht−1h\_\{t\-1\}to the next designxtx\_\{t\}\. This shifts the computational cost to training, allowing the learned policy to directly select the next design at inference time without EIG optimization\.

Under this design\-policy perspective, the standard sequential BED procedure described above corresponds to an implicit policyπS\\pi\_\{S\}, which selects each design by updating the posterior and optimizing the one\-step EIG\([Foster et al\., 2021](https://arxiv.org/html/2610.20832#bib.bib7)\)\. Amortizing this policy directly would therefore require repeatedly simulating this expensive sequential BED procedure to generate training targets\. Instead, DAD observes that the total EIG of a sequential experiment is equal to the sum of its conditional EIGs across steps\. Summing these conditional information gains gives the information gained between the initial prior and the posterior after the complete trajectory\. Applying Bayes’ rule then expresses this quantity directly in terms of trajectory likelihoods:

IT​\(π\)=𝔼p⁡\(θ\)​p​\(hT∣θ,π\)​\[log⁡p⁡\(hT∣θ,π\)−log⁡p⁡\(hT∣π\)\],I\_\{T\}\(\\pi\)=\\mathbb\{E\}\_\{p\(\\theta\)p\(h\_\{T\}\\mid\\theta,\\pi\)\}\\left\[\\log p\(h\_\{T\}\\mid\\theta,\\pi\)\-\\log p\(h\_\{T\}\\mid\\pi\)\\right\],wherep⁡\(hT∣π\)=𝔼p⁡\(θ\)​\[p⁡\(hT∣θ,π\)\]\.p\(h\_\{T\}\\mid\\pi\)=\\mathbb\{E\}\_\{p\(\\theta\)\}\\left\[p\(h\_\{T\}\\mid\\theta,\\pi\)\\right\]\.DAD introduces the sequential prior contrastive estimation \(sPCE\) objective, a lower bound on the total EIG:

EIGT\(π\)=𝔼θ,hT\[logp⁡\(hT∣θ,π\)p⁡\(hT∣π\)\]≥sPCET\(π\)=𝔼θ0:L,hT\[log\(L\+1\)​p​\(hT∣θ0,π\)∑ℓ=0Lp⁡\(hT∣θℓ,π\)\],\\mathrm\{EIG\}\_\{T\}\(\\pi\)=\\mathbb\{E\}\_\{\\theta,h\_\{T\}\}\\\!\\left\[\\log\\frac\{p\(h\_\{T\}\\mid\\theta,\\pi\)\}\{p\(h\_\{T\}\\mid\\pi\)\}\\right\]\\geq\\mathrm\{sPCE\}\_\{T\}\(\\pi\)=\\mathbb\{E\}\_\{\\theta\_\{0:L\},h\_\{T\}\}\\\!\\left\[\\log\\frac\{\(L\+1\)\\,p\(h\_\{T\}\\mid\\theta\_\{0\},\\pi\)\}\{\\sum\_\{\\ell=0\}^\{L\}p\(h\_\{T\}\\mid\\theta\_\{\\ell\},\\pi\)\}\\right\],\(3\)whereθ0:L∼p\(θ\)\\theta\_\{0:L\}\\sim p\(\\theta\)form a contrastive set used to approximate the marginal trajectory likelihood\. The bound becomes tight asL→∞L\\to\\infty, making this design choice a trade\-off between estimator bias and computational cost\. Importantly, these hypotheses are sampled from the prior rather than from a history\-dependent posterior, and therefore do not need to be re\-estimated at each experimental step\.

[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1)formulate the sequential design problem as an MDP by decomposing the sparse trajectory\-level sPCE objective into per\-step rewards:

rt=logp\(yt∣θ0,xt\)−log∑ℓ=0LCt,ℓ\+log∑ℓ=0LCt−1,ℓ,r\_\{t\}=\\log p\(y\_\{t\}\\mid\\theta\_\{0\},x\_\{t\}\)\-\\log\\sum\_\{\\ell=0\}^\{L\}C\_\{t,\\ell\}\+\\log\\sum\_\{\\ell=0\}^\{L\}C\_\{t\-1,\\ell\},\(4\)whereCt,ℓ=∏k=1tp⁡\(yk∣θℓ,xk\)C\_\{t,\\ell\}=\\prod\_\{k=1\}^\{t\}p\(y\_\{k\}\\mid\\theta\_\{\\ell\},x\_\{k\}\)is the likelihood of the observed outcomes up to stepttunder hypothesisθℓ\\theta\_\{\\ell\}, assuming observations are conditionally independent of the previous history given the hypothesis and current design\. They then use a learned critic to estimate the expected future sPCE return, enabling non\-myopic policy optimization\.

## 3Related Work: BED within Large Language Models

MethodPolicyPriorp⁡\(θ\)p\(\\theta\)Likelihoodp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)ObjectiveContrastive set𝒞\\mathcal\{C\}Training targets[Kobalczyk et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib9)
[Choudhury et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib8)π​\(x∣h\)†\\pi\(x\\mid h\)^\{\\dagger\}LLM candidates \+ consistency filteringLLM response modelEIG \(inference\)Sampled at each step—[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1)
[Foster et al\. \(2021\)](https://arxiv.org/html/2610.20832#bib.bib7)π⁡\(x∣h\)\\pi\(x\\mid h\)Specified priorKnown / simulatedsPCE \(training\)Sampled for each gameMatches prior[Hartmann et al\. \(2026\)](https://arxiv.org/html/2610.20832#bib.bib10)π⁡\(x∣h\)\\pi\(x\\mid h\)LLM candidates \+ consistency filteringLLM response modelEIG \(training\)Sampled at each stepLLM\-inducedAR\-BED \(ours\)π⁡\(z,x∣h\)\\pi\(z,x\\mid h\)Fixed prior over𝒞\\mathcal\{C\}External oracleEIG \(training\)Fixed𝒞\\mathcal\{C\}ptrain​\(θ\)⊂𝒞p\_\{\\rm train\}\(\\theta\)\\subset\\mathcal\{C\}AR​\-​BEDDirect\\mathrm\{AR\\text\{\-\}BED\}^\{\\mathrm\{Direct\}\}\(§[5\.1](https://arxiv.org/html/2610.20832#S5.SS1)\)π⁡\(x∣h\)\\pi\(x\\mid h\)——EIG \(training\)Fixed𝒞\\mathcal\{C\}ptrain​\(θ\)⊂𝒞p\_\{\\rm train\}\(\\theta\)\\subset\\mathcal\{C\}Misspecified priors \(§[5\.3\.1](https://arxiv.org/html/2610.20832#S5.SS3.SSS1)\)π⁡\(z,x∣h\)\\pi\(z,x\\mid h\)——EIG \(training\)Restricted coverageWithin restricted𝒞\\mathcal\{C\}Realized IG \(§[5\.3\.2](https://arxiv.org/html/2610.20832#S5.SS3.SSS2)\)π⁡\(z,x∣h\)\\pi\(z,x\\mid h\)——Realized IG \(training\)Fixed𝒞\\mathcal\{C\}ptrain​\(θ\)⊂𝒞p\_\{\\rm train\}\(\\theta\)\\subset\\mathcal\{C\}Multi\-step EIG \(§[5\.3\.3](https://arxiv.org/html/2610.20832#S5.SS3.SSS3)\)π⁡\(z,x∣h\)\\pi\(z,x\\mid h\)——Multi\-step EIG \(training\)Fixed𝒞\\mathcal\{C\}ptrain​\(θ\)⊂𝒞p\_\{\\rm train\}\(\\theta\)\\subset\\mathcal\{C\}Table 1:Policy structure and information\-value estimation across prior methods and AR\-BED\.Policy notation is schematic:hhis the interaction history andzzan explicit reasoning trace \(Eq\.[5](https://arxiv.org/html/2610.20832#S4.E5)\); parameter and time indices are omitted\.†Candidate proposals are followed by inference\-time EIG selection\. Indented rows denote ablations; shading marks changes and — denotes an unchanged prior or likelihood\. Misspecification restricts the coverage of𝒞\\mathcal\{C\}and the training targets, while retaining a fixed prior over𝒞\\mathcal\{C\}\. Section references point to experimental analyses\.BED\-based information gathering with LLMs has been studied through inference\-time design selection and training\-time policy optimization\. Inference\-time methods estimate EIG using LLM\-induced generative models and select among candidate questions\([Kobalczyk et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib9);[Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8)\); CA\-BED extends this approach to non\-myopic selection through conversation\-tree planning\([Arnould et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib21)\)\. ASIG\([Hartmann et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib10)\)instead amortizes design selection into an LLM policy, while retaining a generative model derived from the policy LLM’s predictive distributions\.

Our approach explicitly optimizes the reasoning preceding design selection, constructs the training generative model independently of the policy LLM, and uses expected information rewards over multiple steps to allow for non\-myopic information gathering\. Table[1](https://arxiv.org/html/2610.20832#S3.T1)summarizes the modeling and training choices across prior methods and our ablations\. We discuss related works in detail in Appendix[A](https://arxiv.org/html/2610.20832#A1)\.

## 4Method: Amortizing BED in Reasoning LLMs

Our goal is to amortize sequential BED into a reasoning LLM\. One approach to using an LLM as a policy is to directly sample a design given an interaction history,πϕ​\(xt∣ht−1\)\\pi\_\{\\phi\}\(x\_\{t\}\\mid h\_\{t\-1\}\), and optimize its expected information gain\([Hartmann et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib10)\)\. Instead, our aim is to train an LLM that canreason through limited information: deliberating over which hypotheses remain plausible, how possible observations distinguish them, and which further experiments those observations may enable\. We therefore parameterize the policy as

πϕ​\(zt,xt∣ht−1\)=πϕ​\(zt∣ht−1\)⋅πϕ​\(xt∣ht−1,zt\)\.\\pi\_\{\\phi\}\(z\_\{t\},x\_\{t\}\\mid h\_\{t\-1\}\)=\\pi\_\{\\phi\}\(z\_\{t\}\\mid h\_\{t\-1\}\)\\cdot\\pi\_\{\\phi\}\(x\_\{t\}\\mid h\_\{t\-1\},z\_\{t\}\)\.\(5\)and train its joint output\(zt,xt\)\(z\_\{t\},x\_\{t\}\)with an information gain objective\. At a high level, our training procedure consists of three steps:

1. 1\.Generate a design through reasoning\.Given an interaction historyht−1h\_\{t\-1\}, the policy generates a reasoning traceztz\_\{t\}followed by a designxtx\_\{t\}\.
2. 2\.Evaluate its information value\.A generative model, specified independently of the policy, provides the hypothesis and outcome distributions used to compute the information gain of the designxtx\_\{t\}\.
3. 3\.Optimize the reasoning policy\.We use information\-gain rewards accumulated overKK\-step rollouts to train the reasoning policy with Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib18)\)\.

This procedure requires two main design choices: how to construct the generative model used to evaluate information gain \(Sec\.[4\.1](https://arxiv.org/html/2610.20832#S4.SS1)\), and how to turn this information signal into a training reward \(Sec\.[4\.2](https://arxiv.org/html/2610.20832#S4.SS2)\)\.

### 4\.1How should the generative modelp⁡\(θ,y,x\)p\(\\theta,y;x\)be constructed?

Estimating the EIG requires specifying a generative modelp⁡\(θ,y,x\)p\(\\theta,y;x\)of the experiment\. In our amortized setting, this choice determines both the quality of the training signal and the computational cost of obtaining it\. We build on the contrastive formulation in Sec\.[2\.2](https://arxiv.org/html/2610.20832#S2.SS2), which requires specifying two components: a priorp⁡\(θ\)p\(\\theta\)over contrastive hypotheses and a likelihoodp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)for evaluating proposed designs\. We describe how we construct these components below\.

Decoupling the generative model from the policy\.One option is to use the policy LLM itself to estimate both components, as in the approaches discussed in Sec\.[3](https://arxiv.org/html/2610.20832#S3)\. Training\-time amortization enables an important flexibility: the generative model can instead be specified independently of the policy being trained, without requiring it for design selection at inference time\. This allows us to define the experimental distribution used for training upfront, while keeping it fixed as the policy is updated \(see Appendix[D\.6](https://arxiv.org/html/2610.20832#A4.SS6)\)\. This is particularly useful when the initial policy provides a poor approximation of that distribution\. For example, the training prior can place mass on hypotheses that the policy itself would be unlikely to generate, while a separate likelihood model can provide more reliable estimates ofp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)than the policy’s own predictive distribution\.

The priorp⁡\(θ\)p\(\\theta\)\.The contrastive hypotheses can be sampled from the prior for each trajectory, as in sPCE, or fixed across training\. We define our prior as a uniform distribution over a finite contrastive setΘC=\{θℓ\}ℓ=0L\\Theta\_\{C\}=\\\{\\theta\_\{\\ell\}\\\}\_\{\\ell=0\}^\{L\}, withpC​\(θℓ\)=1/\(L\+1\)p\_\{C\}\(\\theta\_\{\\ell\}\)=1/\(L\+1\)\. This prior is fixed across all training trajectories and defines the hypothesis distribution with respect to which information gain is computed\. The size of the set introduces a trade\-off between coverage and computational cost, while its composition and prior weights determine which hypotheses the policy is trained to distinguish\.

The likelihoodp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)\.In an open\-ended natural\-language design space, the possible designsxxcannot generally be enumerated in advance, making it impractical to precompute their likelihoods\. We therefore use a separate, larger LLM as a semantic likelihood evaluatorporacle​\(y∣θ,x\)p\_\{\\mathrm\{oracle\}\}\(y\\mid\\theta,x\), which can be queried during training\. Following[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1), we assume that each observation is conditionally independent of the interaction history given the hypothesis and current design, allowing us to accumulate likelihoods asCt,ℓC\_\{t,\\ell\}\(see Sec\.[2\.2](https://arxiv.org/html/2610.20832#S2.SS2)\)\. The observationyymay be stochastic even whenθ\\thetais known\. For example, the answer to “Is it cute?” is ambiguous for a fixedθ\\theta\([Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8)\)\. We therefore retain the full likelihood distribution over the finite outcome space𝒴\\mathcal\{Y\}\. Each possible outcomeyyinduces a corresponding updated likelihood,Ct,ℓ\(y\)=Ct−1,ℓ​poracle​\(y∣θℓ,xt\)C\_\{t,\\ell\}^\{\(y\)\}=C\_\{t\-1,\\ell\}p\_\{\\mathrm\{oracle\}\}\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\), which allows us to marginalize over possible outcomes when computing the reward \(discussed in Sec\.[4\.2](https://arxiv.org/html/2610.20832#S4.SS2)\)\. This design also allows for likelihood evaluationsporacle​\(y∣θℓ,xt\)p\_\{\\mathrm\{oracle\}\}\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)to be cached and reused across training\.

### 4\.2How should the reward be assigned?

Having specified the generative model in Sec\.[4\.1](https://arxiv.org/html/2610.20832#S4.SS1), we now consider how information gain over the training hypothesis space should be converted into a reward and how these rewards should be aggregated into training returns\.

Computing the Reward\.A natural choice is to use the realized incremental sPCE reward in Eq\.[4](https://arxiv.org/html/2610.20832#S2.E4)\([Blau et al\., 2022](https://arxiv.org/html/2610.20832#bib.bib1)\), which can be written as

rtreal=log⁡poracle​\(yt∣θ⋆,xt\)pC​\(yt∣ht−1,xt\),r\_\{t\}^\{\\mathrm\{real\}\}=\\log\\frac\{p\_\{\\mathrm\{oracle\}\}\(y\_\{t\}\\mid\\theta^\{\\star\},x\_\{t\}\)\}\{p\_\{C\}\(y\_\{t\}\\mid h\_\{t\-1\},x\_\{t\}\)\},\(6\)wherepC​\(y∣ht−1,xt\)=∑ℓ=0LpC​\(θℓ∣ht−1\)​poracle​\(y∣θℓ,xt\)p\_\{C\}\(y\\mid h\_\{t\-1\},x\_\{t\}\)=\\sum\_\{\\ell=0\}^\{L\}p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)p\_\{\\mathrm\{oracle\}\}\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)denotes the posterior predictive distribution andpC​\(θℓ∣ht−1\)=Ct−1,ℓ∑jCt−1,jp\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)=\\frac\{C\_\{t\-1,\\ell\}\}\{\\sum\_\{j\}C\_\{t\-1,j\}\}is the posterior over the fixed contrastive hypothesis space under the uniform prior \(see Appendix[D\.3](https://arxiv.org/html/2610.20832#A4.SS3)for the full derivation\)\.

For a sampled targetθ⋆\\theta^\{\\star\}and observed outcomeyty\_\{t\}, this assigns the proposed design its realized information gain\. Whenθ⋆∼pC\(⋅∣ht−1\)\\theta^\{\\star\}\\sim p\_\{C\}\(\\cdot\\mid h\_\{t\-1\}\)andyt∼poracle\(⋅∣θ⋆,xt\)y\_\{t\}\\sim p\_\{\\mathrm\{oracle\}\}\(\\cdot\\mid\\theta^\{\\star\},x\_\{t\}\), the expectation of this reward is exactly the one\-step EIG under the current finite belief\.

Because our hypothesis and outcome spaces are finite and enumerable, we can bring this expectation inside each training step: rather than relying on a single realized\(θ⋆,yt\)\(\\theta^\{\\star\},y\_\{t\}\)pair, we marginalize over both hypotheses and outcomes at each turn\. We then assign the expected information reward

rtEIG=∑ℓ=0LpC​\(θℓ∣ht−1\)​∑y∈𝒴poracle​\(y∣θℓ,xt\)​\[log⁡poracle​\(y∣θℓ,xt\)−log⁡pC​\(y∣ht−1,xt\)\]\.r\_\{t\}^\{\\mathrm\{EIG\}\}=\\sum\_\{\\ell=0\}^\{L\}p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)\\sum\_\{y\\in\\mathcal\{Y\}\}p\_\{\\mathrm\{oracle\}\}\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\\left\[\\log p\_\{\\mathrm\{oracle\}\}\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\-\\log p\_\{C\}\(y\\mid h\_\{t\-1\},x\_\{t\}\)\\right\]\.\(7\)Under the sampling assumptions above,rtEIG=𝔼\[rtreal∣ht−1,xt\]r\_\{t\}^\{\\mathrm\{EIG\}\}=\\mathbb\{E\}\[r\_\{t\}^\{\\mathrm\{real\}\}\\mid h\_\{t\-1\},x\_\{t\}\]\. This is the Rao–Blackwellized form of the realized reward, removing the variance associated with sampling a particular\(θ⋆,yt\)\(\\theta^\{\\star\},y\_\{t\}\)pair at a given history and design\. The resulting reward is exactly the one\-step EIG over the specified finite hypothesis space, though not necessarily over the full hypothesis space of interest \(see Appendix[D\.4](https://arxiv.org/html/2610.20832#A4.SS4)\)\.

In practice, we may want to use a large contrastive set to provide a rich information\-gain signal, but generate training trajectories from only a smaller subset of hypotheses for computational efficiency\. In this case, realized and expected rewards can differ beyond variance reduction, as we discuss further in Appendix[D\.5](https://arxiv.org/html/2610.20832#A4.SS5)\.

Constructing the Return\.The reward in Eq\.[7](https://arxiv.org/html/2610.20832#S4.E7)measures only the immediate information value of a design\. However, a design may also enable more informative future interactions\. Following the sequential formulation of[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1), the undiscounted sum of per\-step information rewards recovers multi\-step EIG in expectation when rollouts follow the specified generative model\. We therefore evaluate each proposed design through a rollout of the current policy\. We limit the horizon toKKsteps for tractability and introduceγ\\gammato discount future information:

Rt,K,γ=∑k=0K−1γk​rt\+kEIG\.R\_\{t,K,\\gamma\}=\\sum\_\{k=0\}^\{K\-1\}\\gamma^\{k\}r\_\{t\+k\}^\{\\mathrm\{EIG\}\}\.\(8\)SettingK=1K=1recovers the myopic objective, whileK\>1K\>1withγ\>0\\gamma\>0rewards the initial decision for information acquired at subsequent steps\. In practice, our TQ implementation prunes low\-weight hypotheses and advances rollouts using the oracle’s most likely answer rather than sampling an outcome\. We discuss these approximations to the fully stochastic objective further in Appendix[D\.2](https://arxiv.org/html/2610.20832#A4.SS2)\.

## 5Experiments

We now evaluate our approach for amortizing sequential BED in a reasoning LLM in four studies:

Training EnvironmentFollowing prior work\([Kobalczyk et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib9);[Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8);[Hartmann et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib10)\), we use Twenty Questions \(TQ\), where an agent must identify a hidden target within 20 questions\. Here,θ\\thetais the target \(e\.g\., “horse”\),xxis an open\-ended natural\-language question \(e\.g\., “Is it a mammal?”\), and the observation lies in the finite answer space𝒴=\{yes,no,maybe\}\\mathcal\{Y\}=\\\{\\text\{yes\},\\text\{no\},\\text\{maybe\}\\\}\. We useGPT\-5as the semantic likelihood evaluatorp⁡\(y\|θ,x\)p\(y\|\\theta,x\)\.

Training DataOur main experiments train on theAnimalsdomain\. We use a fixed contrastive set of 200 animals for computing the training reward, of which 100 are used as ground\-truth targetsθ\\thetawhen constructing training trajectories\. We useGPT\-5to generate 250 Twenty Questions games over these 100 targets\. From each game, we sample up to five intermediate points before the final guess, yielding 1,200 partial trajectories\. Each sampled trajectory defines a historyht−1h\_\{t\-1\}containing the sequence of questions and answers observed up to that point\. The task\-transfer experiment uses a separate training dataset, described in Sec\.[5\.4](https://arxiv.org/html/2610.20832#S5.SS4)\.

Evaluation Data\.Each model plays 50 Twenty Questions games across three seeds, with a different target sampled for each game in the corresponding evaluation domain\. We evaluate on held\-out animals drawn from the same 200\-animal contrastive set used during training\. In Sec\.[5\.3](https://arxiv.org/html/2610.20832#S5.SS3), we consider animals that are absent from both the training targets and the training contrastive set\. To assess cross\-domain transfer, we evaluate onPlants,Celebrities, andCountries\. Sec\.[5\.4](https://arxiv.org/html/2610.20832#S5.SS4)evaluates task transfer by applying models trained on Twenty Questions tomediQ\([Li et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib13)\), an interactive medical question\-asking benchmark\.

Models and HyperparametersWe call our method*Amortized Reasoning for Bayesian Experimental Design*\(AR\-BED\)\. Unless otherwise specified,AR​\-​BEDK\\mathrm\{AR\\text\{\-\}BED\}\_\{K\}is aGemma\-4\-E4B\-itreasoning policy trained with expected information rewards with rollout horizonKK\. We denote byAR​\-​BEDKDirect\\mathrm\{AR\\text\{\-\}BED\}\_\{K\}^\{\\mathrm\{Direct\}\}the ablation where we train without reasoning, directly sampling and rewarding a question from the historyx∼p\(\.\|h\)x\\sim p\(\.\|h\)\. We use a batch size of 128 with 8 rollouts per state, and train for 250 steps\. For multi\-step training \(K\>1K\>1\), we setγ=0\.5\\gamma=0\.5\. Full training details are in Appendix[C](https://arxiv.org/html/2610.20832#A3)\.

Metrics\.We report the solve rate within a 20\-question budget and the average EIG per turn to account for varying game lengths\.

### 5\.1Amortizing BED into Reasoning Improves General Information Acquisition

![Refer to caption](https://arxiv.org/html/2610.20832v1/figures/reasoning.png)Figure 1:Amortizing BED into reasoning improves information acquisition and transfer\.Bars and bands show mean±\\pmSEM over three evaluation repeats\.Setup\.We evaluate whether amortizing BED into the reasoning process improves information acquisition both in\- and out\-of\-distribution\. We compare our model,AR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}, to four controls: theBasemodel and the directly trainedAR​\-​BED1Direct\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}^\{\\mathrm\{Direct\}\}model, each evaluated with and without reasoning at inference time\. This design separates gains from BED training from gains due only to enabling reasoning at test time\.

Results\.\(Fig\.[1](https://arxiv.org/html/2610.20832#S5.F1)\) Adding reasoning to theBasemodelreducesfinal solve rate across all categories and increases average EIG only on Countries\. This suggests that inference\-time reasoning alone does not improve information gathering, and may even worsen it\.111A similar effect is observed for some models reported in Table 1 of[Choudhury et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib8)\.Training a non\-reasoning policy improves the in\-distribution behavior but the transfer to out\-of\-distribution domains is inconsistent: it is worse thanBaseon Plants but improves on Celebrities and Countries\. This suggests that direct amortization can learn useful in\-distribution question\-sampling behavior, but does not produce robust transfer across domains\. In contrast, our modelAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}achieves the best final solve rate across all categories and obtains the highest average EIG per turn on Animals, Plants, and Countries\. These gains over both theBasemodel and the direct policy suggest that training a reasoning model may enable general information\-gathering strategies\.

### 5\.2Amortizing BED into reasoning outperforms Inference\- and Training\-Time BED Baselines

Setup\.We next compare against BED\-style alternatives that differ in how they obtain the generative model used to estimate EIG\. OurAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}model uses an external generative model, whereasBED\-LLM\([Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8)\)uses the LLM itself to construct the predictive generative model at inference time\. We also compare toASIG, a training\-time amortization baseline, which uses the same BED\-LLM\-style LLM\-generated model during training but trains a direct question\-sampling policy rather than a reasoning policy\([Hartmann et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib10)\)\. To isolate the effect of the information\-gain objective, we do not include an additional game reward forASIG\. We evaluateASIGwith and without reasoning at inference\.

Table 2:AR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}is more efficient at both inference and training time\.Results\.Figure[3](https://arxiv.org/html/2610.20832#S5.F3)shows thatAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}outperforms the inference\-time and training\-time BED baselines on most domains\.BED\-LLMimproves only modestly over the reasoningBasemodel, and is worse than the non\-reasoningBasemodel on some domains in both solve rate and EIG\. This suggests that online BED with an LLM\-generated model of the experiment may be limited by the model’s ability to represent the relevant distributionsp⁡\(θ\)p\(\\theta\)andp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)\. Similarly,ASIGwithout reasoning does not improve over the non\-reasoningBasemodel, indicating that directly amortizing this LLM\-generated BED objective suffers from similar limitations\. Adding reasoning at inference substantially improvesASIG, but it still remains belowAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}on Animals, Plants, and Countries, and is comparable on Celebrities\. These results suggest that the benefit comes from amortizing the information\-gain objective into the model’s reasoning process and using an external generative model that provides a more reliable training signal\.

Table[2](https://arxiv.org/html/2610.20832#S5.T2)shows that at inference time,BED\-LLMis about61×61\\timesslower per game thanAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}because it performs BED computation online\. At training time,ASIGis about2\.9×2\.9\\timesslower per step due to the additional cost of using the LLM\-generated BED\-LLM\-style generative model\. Thus,AR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}gains stronger information acquisition capabilities while avoiding the main computational costs of BED methods with a policy\-based generative model\.

![Refer to caption](https://arxiv.org/html/2610.20832v1/figures/mammals.png)Figure 2:Generalization across held\-out animal subgroups\.Solve rate over turns for held\-out mammals and non\-mammals\. Bands show mean±\\pmSEM over three evaluation repeats\.
### 5\.3Ablations and Insights

![Refer to caption](https://arxiv.org/html/2610.20832v1/figures/generative.png)Figure 3:Comparison to BED baselines\.AR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}outperforms inference\-timeBED\-LLMand training\-timeASIGon most domains; bars and bands show mean±\\pmSEM over evaluation repeats\.#### 5\.3\.1Misspecified Priors and Size of the Contrastive Set

To study the effect of a misspecified prior \(i\.e\., training a model with a flawed contrastive set\), we compare two variants ofAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}: one with a 100\-animal contrastive set trained on only these animals, and one with a 100\-mammal contrastive set trained only on these mammals\. Both models are prompted to guess in the categoryAnimals\. As a result, the mammals\-trained model received a flawed information signal compared to the true hypothesis spacep⁡\(θ\)p\(\\theta\): for example, the question “Is it a mammal?” receivesE​I​G≈0EIG\\approx 0despite being a strong question to discriminate between animals\.

Fig\.[2](https://arxiv.org/html/2610.20832#S5.F2)shows the solve rate of both models on held\-out sets of 25 mammals and 25 non\-mammals\. The model trained with a mammals\-only contrastive set performs significantly worse than both the animals\-trained and base models on non\-mammals, suggesting that it learned a flawed representation of the animals hypothesis space\. Notably, it also leads to worse performance than the animals\-trained model on mammal targets\. This suggests that training on the broader hypothesis space of animals might provide exposure to a diversity of features that is beneficial even within the shared mammal subset\.

We additionally investigate the effect of the size of the contrastive set in Appendix[B\.1](https://arxiv.org/html/2610.20832#A2.SS1), finding that using a 200\-sample contrastive set improves the OOD transfer performance compared to a 100\-sample contrastive set\.

#### 5\.3\.2Using expected or realized IG in the reward signal

Marginalizing over hypotheses and answers generally provides a stronger training signal than using the realized information gain: expected\-reward training improves both EIG and solve rate on Animals, Plants, and Countries, although not on Celebrities\. Appendix[B\.2](https://arxiv.org/html/2610.20832#A2.SS2)reports the matched comparison, consistent with the variance\-reduction motivation in Sec\.[4\.2](https://arxiv.org/html/2610.20832#S4.SS2)\.

#### 5\.3\.3Using a non\-myopic reward signal

Appendix[B\.3](https://arxiv.org/html/2610.20832#A2.SS3)brings together three complementary evaluations of multi\-step rewards\. In a natural\-language adaptation of Cube\-NM\([Schütz et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib19)\), the two\-step policy learns to acquire initially uninformative context for its downstream benefit, substantially outperforming the one\-step policy in later disease identification\. On the TwentyQuestions target\-prediction task of[Wang et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib22), multi\-step policies improve attribute prediction, particularly on medium and hard questions\. Finally, we empirically investigate the extent to which Twenty Questions as played by the GPT\-5 policy shows signs of non\-myopicity\. We show that immediate\- and downstream\-EIG rankings can differ, while matched\-state evaluations find some gains fromK=2K=2andK=3K=3overK=1K=1at two\- and three\-question horizons\. Together, these results distinguish a controlled non\-myopic mechanism from the smaller benefits of lookahead in open\-ended questioning\.

### 5\.4Generalization to other tasks

The preceding experiments examine transfer across domains within TQ\. We now ask whether the learned information\-gathering behavior also transfersto a different task\. We use MediQ\([Li et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib13)\), which simulates clinical questioning: given partial patient information and a multiple\-choice medical inquiry, a policy asks open\-ended questions to gather evidence from an LLM\-based Patient with full access to patient records\.

Setup\.We construct three synthetic medical TQ environments— dermatology, neurology, and gastroenterology—each with a 50\-disease contrastive set drawn from MediQ’s answer inventory\. Using theAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}objective, we train a separate medical policy on 1,200 starting states and evaluate its checkpoint against the base model\. Both policies ask five questions with GPT\-5 as the fixed Patient\. We evaluate the quality of each model’s line of questioning by using a separate GPT\-5 decider to predict the answer from the initial information and interaction history\.

We report two diagnosis subsets: 95 dermatology cases whose correct diagnoses are absent from the training contrastive set \(same domain, different task, held\-out labels\), and 92 cases from other specialties, including pediatrics, psychiatry, and cardiology \(different domain and task\)\. Full training and evaluation details are in Appendix[C\.4](https://arxiv.org/html/2610.20832#A3.SS4)\.

Table 3:Performance on MediQ after training on Twenty Questions\.Each policy conducts five question–answer turns with a fixed GPT\-5 Patient; a separate GPT\-5 decider predicts the final answer from the resulting history\. Values are means±1\\pm 1SEM across casesResults\.Table[3](https://arxiv.org/html/2610.20832#S5.T3)shows that medicalAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}improves accuracy from90\.5%90\.5\\%to94\.7%94\.7\\%on held\-out dermatology diagnoses and from84\.8%84\.8\\%to90\.2%90\.2\\%on other specialties\. It also increases the probability assigned to the correct answer and reduces the Brier score in both domains\. These results indicate that the reasoning capabilities learned byAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}in a synthetic TQ environment transfer to a realistic, simulated clinical dialogue setting, including medical specialties not used during fine\-tuning\.

## 6Conclusion

We studied how sequential Bayesian experimental design can be amortized into a reasoning LLM to improve adaptive information gathering\. Our approach makes two key design choices: decoupling the experimental generative model from the policy to avoid relying on potentially poor predictive distributions, and constructing expected information\-gain rewards that can be accumulated across interactions\. We find that our method improves information acquisition across unseen hypothesis spaces, outperforming other methods, and that the learned behavior transfers to Medi\-Q, a real\-world clinical dialogue dataset\. Our results suggest that adaptive information gathering can be learned as a transferable reasoning capability in LLMs rather than reconstructed through explicit search at every interaction\.

Limitations\.We study relatively short rollout horizons\. Extending the approach to longer horizons while maintaining a reliable training signal is an important direction\. We also find that performance is sensitive to the coverage of the contrastive hypothesis set when it is misspecified relative to the true hypothesis space\. Further understanding how the size, composition, and prior weighting of the contrastive set affect learning and transfer is therefore an important avenue for future study\.

## AI Use Statement

In this work, we used generative AI tools to assist with the coding implementation, figure editing, LaTeX formatting, and manuscript polishing\. The authors determined the research questions, methodology, experiments, and final interpretation of results\. We reviewed all AI\-assisted work and take responsibility for the final content of this paper\.

## Acknowledgements

AK’s PhD studentship is funded by Sanofi, while AB’s studentship is funded by Eedi\. This work was supported by Azure sponsorship credits granted by Microsoft’s AI for Good Research Lab\.

## References

- Andukuriet al\.\(2024\)C\. Andukuri, J\. Fränken, T\. Gerstenberg, and N\. D\. GoodmanStar\-gate: teaching language models to ask clarifying questions\.arXiv preprint arXiv:2403\.19154\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p4.1)\.
- Arnouldet al\.\(2026\)D\. Arnould, R\. Aziz, Z\. Kang, T\. Changal, K\. Zhu, S\. Dev, G\. Grand, and S\. S\. KulkarniCA\-BED: conversation\-aware bayesian experimental design\.InAgentic AI in the Wild: From Hallucinations to Reliable Autonomy,External Links:[Link](https://openreview.net/forum?id=dhhW9QvC7k)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p1.1),[§3](https://arxiv.org/html/2610.20832#S3.p1.1)\.
- Ashokkumaret al\.\(2026\)A\. Ashokkumar, L\. Hewitt, I\. Ghezae, and R\. WillerLarge language models can predict the results of social science experiments\.Nature656\(8126\),pp\. 115–122\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10742-x)Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.
- Auzinaet al\.\(2026\)I\. A\. Auzina, J\. Strüber, S\. Hernández\-Gutiérrez, S\. Goel, A\. Prabhu, and M\. BethgeIntrinsic credit assignment for long horizon interaction\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=SAJMlj9x39)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p2.1)\.
- Blauet al\.\(2022\)T\. Blau, E\. V\. Bonilla, I\. Chades, and A\. DezfouliOptimizing sequential experimental design with deep reinforcement learning\.InProceedings of the 39th International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p3.1),[§D\.1](https://arxiv.org/html/2610.20832#A4.SS1.p4.1),[§D\.2](https://arxiv.org/html/2610.20832#A4.SS2.p1.1),[§D\.2](https://arxiv.org/html/2610.20832#A4.SS2.p2.1),[Appendix D](https://arxiv.org/html/2610.20832#A4.p2.1),[§2\.2](https://arxiv.org/html/2610.20832#S2.SS2.p3.1),[Table 1](https://arxiv.org/html/2610.20832#S3.T1.2.3.1.1.1),[§4\.1](https://arxiv.org/html/2610.20832#S4.SS1.p4.1),[§4\.2](https://arxiv.org/html/2610.20832#S4.SS2.p2.1),[§4\.2](https://arxiv.org/html/2610.20832#S4.SS2.p6.1)\.
- Chiet al\.\(2024\)Y\. Chi, J\. Lin, K\. Lin, and D\. KleinClarinet: augmenting language models to ask clarification questions for retrieval\.arXiv preprint arXiv:2405\.15784\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p5.1)\.
- Choudhuryet al\.\(2025\)D\. Choudhury, S\. Williamson, A\. Golinski, N\. Miao, F\. Bickford Smith, M\. Kirchhof, Y\. Zhang, and T\. RainforthBED\-LLM: intelligent information gathering with large language models and bayesian experimental design\.arXiv preprint\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p1.1),[§D\.1](https://arxiv.org/html/2610.20832#A4.SS1.p2.2),[§D\.1](https://arxiv.org/html/2610.20832#A4.SS1.p4.1),[§D\.1](https://arxiv.org/html/2610.20832#A4.SS1.p5.1),[§1](https://arxiv.org/html/2610.20832#S1.p3.1),[Table 1](https://arxiv.org/html/2610.20832#S3.T1.2.2.1.1.1),[§3](https://arxiv.org/html/2610.20832#S3.p1.1),[§4\.1](https://arxiv.org/html/2610.20832#S4.SS1.p4.1),[§5\.2](https://arxiv.org/html/2610.20832#S5.SS2.p1.1),[§5](https://arxiv.org/html/2610.20832#S5.p2.1),[footnote 1](https://arxiv.org/html/2610.20832#footnote1),[footnote 2](https://arxiv.org/html/2610.20832#footnote2)\.
- Denget al\.\(2026\)M\. Deng, Z\. Li, X\. Li, T\. Zhu, Y\. Zhao, Z\. Guo, and W\. WangUncertainty\-aware clarification in llm agents with information gain\.arXiv preprint arXiv:2606\.03135\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p2.1)\.
- Fosteret al\.\(2021\)A\. Foster, D\. R\. Ivanova, I\. Malik, and T\. RainforthDeep adaptive design: amortizing sequential bayesian experimental design\.InProceedings of the 38th International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p3.1),[§2\.2](https://arxiv.org/html/2610.20832#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2610.20832#S2.SS2.p2.1),[Table 1](https://arxiv.org/html/2610.20832#S3.T1.2.3.1.1.1)\.
- Fosteret al\.\(2019\)A\. Foster, M\. Jankowiak, E\. Bingham, P\. Horsfall, Y\. W\. Teh, T\. Rainforth, and N\. GoodmanVariational bayesian optimal experimental design\.Advances in neural information processing systems32\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p6.1)\.
- Fosteret al\.\(2020\)A\. Foster, M\. Jankowiak, M\. O’Meara, Y\. W\. Teh, and T\. RainforthA unified stochastic gradient approach to designing bayesian\-optimal experiments\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 2959–2969\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p6.1)\.
- Grandet al\.\(2026\)G\. Grand, V\. Pepe, J\. B\. Tenenbaum, and J\. AndreasShoot first, ask questions later? building rational agents that explore and act like people\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EQhUvWH78U)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p1.1)\.
- Guptaet al\.\(2025\)R\. Gupta, J\. Hartford, and B\. LiuLLMs for bayesian optimization in scientific domains: are we there yet?\.External Links:2509\.21403,[Link](https://arxiv.org/abs/2509.21403)Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.
- Handaet al\.\(2024\)K\. Handa, Y\. Gal, E\. Pavlick, N\. Goodman, J\. Andreas, A\. Tamkin, and B\. Z\. LiBayesian preference elicitation with language models\.arXiv preprint arXiv:2403\.05534\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p5.1)\.
- Hartmannet al\.\(2026\)J\. Hartmann, J\. Harvey, J\. Navott, E\. Y\. Wang, L\. C\. Melo, F\. Cipcigan, C\. Zhang, and A\. AbateAmortising bayesian experimental design for sequential information gathering in LLMs\.InICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning,External Links:[Link](https://openreview.net/forum?id=Yxu0uQuyzY)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p2.1),[§1](https://arxiv.org/html/2610.20832#S1.p3.1),[Table 1](https://arxiv.org/html/2610.20832#S3.T1.2.4.1.1.1),[§3](https://arxiv.org/html/2610.20832#S3.p1.1),[§4](https://arxiv.org/html/2610.20832#S4.p1.1),[§5\.2](https://arxiv.org/html/2610.20832#S5.SS2.p1.1),[§5](https://arxiv.org/html/2610.20832#S5.p2.1)\.
- Hedmanet al\.\(2025\)M\. Hedman, D\. R\. Ivanova, C\. Guan, and T\. RainforthStep\-DAD: semi\-amortized policy\-based bayesian experimental design\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=JRg8P2bX8P)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p3.1)\.
- Huet al\.\(2024\)Z\. Hu, C\. Liu, X\. Feng, Y\. Zhao, S\. Ng, A\. T\. Luu, J\. He, P\. W\. Koh, and B\. HooiUncertainty of thoughts: uncertainty\-aware planning enhances information seeking in large language models\.InICLR 2024 Workshop on Large Language Model \(LLM\) Agents,External Links:[Link](https://openreview.net/forum?id=ZWyLjimciT)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p1.1)\.
- Huanget al\.\(2024\)Q\. Huang, J\. Vora, P\. Liang, and J\. LeskovecMLAgentbench: evaluating language agents on machine learning experimentation\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p1.1)\.
- Ivanovaet al\.\(2021\)D\. R\. Ivanova, A\. Foster, S\. Kleinegesse, M\. U\. Gutmann, and T\. RainforthImplicit deep adaptive design: policy\-based experimental design without likelihoods\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=on2DNSz2Qg)Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p3.1)\.
- Kobalczyket al\.\(2025\)K\. Kobalczyk, N\. Astorga, T\. Liu, and M\. van der SchaarActive task disambiguation with large language models\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p1.1),[§1](https://arxiv.org/html/2610.20832#S1.p3.1),[Table 1](https://arxiv.org/html/2610.20832#S3.T1.2.2.1.1.1),[§3](https://arxiv.org/html/2610.20832#S3.p1.1),[§5](https://arxiv.org/html/2610.20832#S5.p2.1)\.
- Krishnamurthyet al\.\(2024\)A\. Krishnamurthy, K\. Harris, D\. J\. Foster, C\. Zhang, and A\. SlivkinsCan large language models explore in\-context?\.Advances in Neural Information Processing Systems37,pp\. 120124–120158\.Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.
- Labanet al\.\(2026\)P\. Laban, H\. Hayashi, Y\. Zhou, and J\. NevilleLLMs get lost in multi\-turn conversation\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.
- Liet al\.\(2026\)B\. Z\. Li, B\. Kim, and Z\. WangQuestBench: can LLMs ask the right question to acquire information in reasoning tasks?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=gpwA9aZLTZ)Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.
- Liet al\.\(2024\)S\. S\. Li, V\. Balachandran, S\. Feng, J\. S\. Ilgen, E\. Pierson, P\. W\. Koh, and Y\. TsvetkovMediQ: question\-asking LLMs and a benchmark for reliable interactive clinical reasoning\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§C\.4\.2](https://arxiv.org/html/2610.20832#A3.SS4.SSS2.p1.1),[§1](https://arxiv.org/html/2610.20832#S1.p1.1),[§1](https://arxiv.org/html/2610.20832#S1.p4.1),[§5\.4](https://arxiv.org/html/2610.20832#S5.SS4.p1.1),[§5](https://arxiv.org/html/2610.20832#S5.p4.1)\.
- Mazzaccaraet al\.\(2024\)D\. Mazzaccara, A\. Testoni, and R\. BernardiLearning to ask informative questions: enhancing llms with preference optimization and expected information gain\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 5064–5074\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p2.1)\.
- Nowak \(2011\)R\. D\. NowakThe geometry of generalized binary search\.IEEE Transactions on Information Theory57\(12\),pp\. 7893–7906\.Cited by:[§B\.3\.4](https://arxiv.org/html/2610.20832#A2.SS3.SSS4.p1.1)\.
- Rainforthet al\.\(2018\)T\. Rainforth, R\. Cornish, H\. Yang, A\. Warrington, and F\. WoodOn nesting monte carlo estimators\.InInternational Conference on Machine Learning,pp\. 4267–4276\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p6.1)\.
- Rainforthet al\.\(2024\)T\. Rainforth, A\. Foster, D\. R\. Ivanova, and F\. Bickford SmithModern bayesian experimental design\.Statistical Science39\(1\),pp\. 100–114\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p6.1)\.
- Rainforthet al\.\(2023\)T\. Rainforth, A\. Foster, D\. R\. Ivanova, and F\. B\. SmithModern bayesian experimental design\.External Links:2302\.14545,[Link](https://arxiv.org/abs/2302.14545)Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p3.1)\.
- Rückstießet al\.\(2013\)T\. Rückstieß, C\. Osendorfer, and P\. van der SmagtMinimizing data consumption with sequential online feature selection\.International Journal of Machine Learning and Cybernetics4\(3\),pp\. 235–243\.Cited by:[§B\.3\.1](https://arxiv.org/html/2610.20832#A2.SS3.SSS1.p1.1)\.
- Schützet al\.\(2026\)V\. Schütz, H\. Wu, R\. Rezvan, L\. Aronsson, and M\. H\. ChehreghaniAFABench: a generic framework for benchmarking active feature acquisition\.External Links:2508\.14734,[Link](https://arxiv.org/abs/2508.14734)Cited by:[§B\.3\.1](https://arxiv.org/html/2610.20832#A2.SS3.SSS1.p1.1),[§5\.3\.3](https://arxiv.org/html/2610.20832#S5.SS3.SSS3.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[item 3](https://arxiv.org/html/2610.20832#S4.I1.i3.p1.1)\.
- Tajwaret al\.\(2025\)F\. Tajwar, Y\. Jiang, A\. Thankaraj, S\. S\. Rahman, J\. Z\. Kolter, J\. Schneider, and R\. SalakhutdinovTraining a generally curious agent\.arXiv preprint arXiv:2502\.17543\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p4.1)\.
- Wanget al\.\(2025\)J\. Wang, T\. Zollo, R\. Zemel, and H\. NamkoongAdaptive elicitation of latent information using natural language\.arXiv preprint arXiv:2504\.04204\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p5.1),[Figure 7](https://arxiv.org/html/2610.20832#A2.F7),[§B\.3\.3](https://arxiv.org/html/2610.20832#A2.SS3.SSS3.p1.1),[§5\.3\.3](https://arxiv.org/html/2610.20832#S5.SS3.SSS3.p1.1)\.
- Wuet al\.\(2025\)S\. Wu, M\. Galley, B\. Peng, H\. Cheng, G\. Li, Y\. Dou, W\. Cai, J\. Zou, J\. Leskovec, and J\. GaoCollabllm: from passive responders to active collaborators\.arXiv preprint arXiv:2502\.00640\.Cited by:[Appendix A](https://arxiv.org/html/2610.20832#A1.p4.1)\.
- Zhouet al\.\(2025a\)S\. Zhou, Z\. Xu, M\. Zhang, C\. Xu, Y\. Guo, Z\. Zhan, Y\. Fang, S\. Ding, J\. Wang, K\. Xu, L\. Xia, J\. Yeung, D\. Zha, D\. Cai, G\. B\. Melton, M\. Lin, and R\. ZhangLarge language models for disease diagnosis: a scoping review\.npj Artificial Intelligence1\(1\),pp\. 9\.External Links:[Document](https://dx.doi.org/10.1038/s44387-025-00011-z)Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.
- Zhouet al\.\(2025b\)Z\. Zhou, X\. Feng, Z\. Zhu, J\. Yao, S\. Koyejo, and B\. HanFrom passive to active reasoning: can large language models ask the right questions under incomplete information?\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=LCaTpVuvpj)Cited by:[§1](https://arxiv.org/html/2610.20832#S1.p2.1)\.

## Appendix AExtended Related Works

Inference\-time BED\.Methods for task disambiguation and adaptive questioning use LLM\-generated hypotheses and response predictions to select questions by estimated EIG\([Kobalczyk et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib9);[Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8)\); related agents maintain explicit Bayesian beliefs to guide exploration\([Grand et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib17)\)\. These approaches require repeated belief maintenance and utility evaluation at deployment\. While one\-step EIG selection is myopic, uncertainty\-aware planning and conversation\-tree search incorporate simulated future interactions\([Hu et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib31);[Arnould et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib21)\)\. Thus, non\-myopic information gathering is not unique to trained policies; our aim is to amortize this behavior into a reasoning LLM rather than perform explicit design search at inference time\.

Training\-time BED\.Information\-driven post\-training uses EIG\-ranked question preferences\([Mazzaccara et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib33)\)or dense rewards based on changes in the model’s confidence in a known target\([Auzina et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib24);[Deng et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib26)\)\. These approaches establish informative questioning as a learnable behavior, but differ in their information estimators and optimization procedures\. ASIG\([Hartmann et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib10)\)amortizes BED into an LLM policy using a generative model derived from the policy’s predictive distributions; its evaluated configuration combines one\-step EIG with a multi\-step task\-outcome reward\. We instead explicitly optimize the reasoning preceding design selection and specify the training generative model independently of the policy, using a fixed contrastive prior and a separate semantic likelihood evaluator\. This construction enables expected information rewards over the finite hypothesis and outcome spaces, which we accumulate into information\-only multi\-step returns\.

Amortized sequential experimental design\.Policy\-based BED learns adaptive designs across an experiment, shifting optimization to training and accounting for the information available from future observations\([Foster et al\., 2021](https://arxiv.org/html/2610.20832#bib.bib7);[Blau et al\., 2022](https://arxiv.org/html/2610.20832#bib.bib1);[Ivanova et al\., 2021](https://arxiv.org/html/2610.20832#bib.bib32)\)\. This includes both tractable\-likelihood and implicit\-simulator settings\. Semi\-amortized approaches additionally refine a pretrained design policy using observations from the ongoing experiment\([Hedman et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib30)\)\. We build on the fully amortized formulation, studying how its policy, generative model, and information objective should be instantiated for reasoning LLMs with natural\-language actions\.

Task\-oriented exploration and interaction learning\.Complementary approaches train curious or collaborative agents using task\-oriented rewards, including objectives that assess the consequences of an action through future interactions\([Tajwar et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib36);[Wu et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib37)\)\. Successful interaction traces also provide supervision for learning clarifying questions\([Andukuri et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib23)\)\. These works show that strategic information gathering can be learned without an explicit BED objective, and that learned behavior can transfer across tasks\([Tajwar et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib36)\)\. Our focus is instead on the construction of information\-only training returns and their effect on acquisition, reasoning, and transfer\.

External models for information gathering\.Prior work combines language models with parametric Bayesian preference models or learned predictive models for adaptive elicitation\([Handa et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib29);[Wang et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib22)\)\. Retriever\-coupled approaches similarly derive question\-training signals from an external candidate distribution or retrieval objective\([Chi et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib25)\)\. These provide precedents for separating language generation from information\-value estimation\. Our construction uses a specified finite hypothesis space and a separate semantic likelihood evaluator during training, while deployment requires neither that hypothesis space nor retrieval\-based question selection\.

Estimating information value\.Variational and contrastive objectives make Bayesian design optimization tractable\([Foster et al\., 2019](https://arxiv.org/html/2610.20832#bib.bib27);[Foster et al\., 2020](https://arxiv.org/html/2610.20832#bib.bib28)\), while nested Monte Carlo analysis characterizes challenges in estimating information utilities\([Rainforth et al\., 2018](https://arxiv.org/html/2610.20832#bib.bib34)\);[Rainforth et al\. \(2024\)](https://arxiv.org/html/2610.20832#bib.bib35)survey these approaches\.

## Appendix BExtended Experimental Results

### B\.1Contrastive set ablations

A larger contrastive set improves performance\.Figure[4](https://arxiv.org/html/2610.20832#A2.F4)compares 100\- and 200\-animal training generative models\. On Animals, their EIG per question is nearly identical \(0\.3300\.330nats\), while the 200\-animal policy has a slightly higher final solve rate \(43\.3%43\.3\\%versus41\.3%41\.3\\%\)\. The clearest difference is on Plants, where the larger set improves EIG from0\.2600\.260to0\.2680\.268and solve rate from28\.0%28\.0\\%to40\.7%40\.7\\%\. It also improves country EIG and solve rate; celebrity EIG is essentially unchanged\. These results suggest that increasing the size of the contrastive set can improve the usefulness of the training signal and its transfer, but does not guarantee better acquisition on every domain\.

Coverage matters beyond the number of hypotheses\.Restricting the training generative model to 100 Mammals substantially reduces performance relative to 100 diverse Animals\. On animal evaluation targets, EIG falls from0\.3300\.330to0\.2450\.245and solve rate from41\.3%41\.3\\%to16\.7%16\.7\\%; both metrics are also lower on all three OOD domains\. Since this condition restricts both the contrastive set and sampled training targets, it tests generative\-model coverage rather than a change to the estimator alone\. The result reinforces the design choice in Sec\.[4\.1](https://arxiv.org/html/2610.20832#S4.SS1): decoupling the generative model from the policy allows its coverage to be controlled, but the resulting information reward is only as useful as the distinctions represented during training\.

Figure 4:Training\-distribution size and coverage affect acquisition and transfer\.
### B\.2Expected versus realized information rewards

Setup\.We compare one\-step reasoning policies trained on the same sampled animal\-game states with the same 200\-animal contrastive set\. The policies differ only in whether the reward marginalizes over hypotheses and possible answers or uses the information gain realized along the sampled trajectory\. Both policies generate a reasoning trace before selecting a question; this comparison changes the reward estimator, not which part of the output is optimized\.

Expected information rewards usually provide the stronger training signal\.Figure[5](https://arxiv.org/html/2610.20832#A2.F5)compares information acquisition and solve rate on Animals and three transfer domains\. Expected\-reward training improves animal EIG from0\.2900\.290to0\.3300\.330nats per question and solve rate from34\.0%34\.0\\%to43\.3%43\.3\\%, and improves both metrics on Plants and Countries\. The exception is Celebrities, where realized\-reward training yields slightly higher EIG and a higher solve rate \(23\.3%23\.3\\%versus17\.3%17\.3\\%\)\. The overall pattern is consistent with the variance\-reduction motivation in Sec\.[4\.2](https://arxiv.org/html/2610.20832#S4.SS2), but does not establish variance as the sole cause or imply a uniform advantage across transfer domains\.

Figure 5:Expected rewards generally improve learning over realized rewards\.One\-step reasoning policies trained with the same 200\-animal generative model, differing in reward estimator\. Expected rewards improve EIG per question and solve rate on Animals, Plants, and Countries, but not Celebrities\. Bars and bands show mean±\\pmSEM over three evaluation repeats\.
### B\.3Non\-myopic information acquisition

We evaluate multi\-step rewards in three complementary ways: a controlled environment where context must be acquired before it becomes useful, prediction of target attributes during open\-ended questioning, and information acquisition in Twenty Questions\. The Cube\-NM experiment isolates the ability to learn an initially uninformative action; the other evaluations examine whether a longer reward horizon also benefits natural\-language policies\.

#### B\.3\.1Cube\-NM: acquiring context before information

A controlled non\-myopic environment\.We adapt Cube\-NM\([Schütz et al\., 2026](https://arxiv.org/html/2610.20832#bib.bib19);[Rückstieß et al\., 2013](https://arxiv.org/html/2610.20832#bib.bib20)\)as a small mechanism check\. Its 32 diseases and three equally likely, disease\-independent contexts define 96 disease–context pairs\. Each context activates eight tests; a separate context test reveals which block is active but provides zero immediate disease information\. A useful policy must therefore trade immediate information for better subsequent acquisition\. We use a natural\-language test\-selection interface without reasoning traces to isolate this routing decision from open\-ended question generation and medical knowledge\.

Training and evaluation\.The 25 equal\-cost tests comprise three context\-dependent blocks of eight tests and one context test\. The context is independent of the disease, and outcomes are deterministic given both\. The textual interface returns positive, negative, or inconclusive outcomes and the diseases invalidated by the observation; the context test instead lists the informative tests\. Test identifiers are randomly permuted once, identically across the matched runs, and the outcome matrix and context\-test identity are not initially disclosed\. The policy selects one identifier without a reasoning trace\. We trainQwen3\-0\.6Bwith GRPO/LoRA for 250 updates at learning rate10−510^\{\-5\}, sampling 16 states and 16 rollouts per state from a rolling on\-policy bank covering turns 0–4\. Temperature is 2, the entropy coefficient is 0\.02, andγ=0\.9\\gamma=0\.9\. Evaluation uses greedy actions over all 96 disease–context pairs and an exact Bayesian MAP decoder\. Total EIG islog⁡32\\log 32minus the mean disease\-posterior entropy\. These are matched single\-training\-seed results, not estimates of variability across training runs\.

The two\-step reward learns the initially uninformative context test\.Figure[6](https://arxiv.org/html/2610.20832#A2.F6)shows that the two\-step policy learns to select the context test with almost unit probability, while the one\-step policy favors an immediately informative disease test\. This incurs an initial cost: after one test, Bayesian MAP solve rate is3\.1%3\.1\\%forK=2K=2versus5\.2%5\.2\\%forK=1K=1\. The two\-step policy overtakes after three tests \(27\.1%27\.1\\%versus21\.9%21\.9\\%\) and reaches75\.0%75\.0\\%versus30\.2%30\.2\\%after five\. Thus the multi\-step reward can learn a design for its downstream benefit, even when its immediate disease\-information reward is zero\. These matched single\-seed results support the intended mechanism, not global optimality or an additional claim about reasoning transfer\.

Figure 6:A controlled confirmation of non\-myopic learning in Cube\-NM\.Left: probability of selecting the initially uninformative context test during training\. Center: total EIG under greedy acquisition\. Right: Bayesian MAP solve rate\. The two\-step objective learns to acquire context before disease information, improving later acquisition\. These are matched single\-seed, no\-reasoning policies evaluated on all 96 disease–context pairs; the decoder is Bayesian, not an LLM diagnosis\.
#### B\.3\.2Why Cube\-NM requires non\-myopic information acquisition

Cube\-NM provides a controlled setting in which an action can have zero immediate information gain about the target while increasing the value of a subsequent experiment\. We use this structure to illustrate why a one\-step information\-gain objective cannot assign value to such routing actions, while a two\-step objective can\.

Setup\.LetΘ\\Thetadenote the latent disease and letCCdenote an auxiliary context variable\. In Cube\-NM, the context is independent of the disease,

The disease tests are partitioned into context\-dependent blocks\{𝒳c\}c∈𝒞\\\{\\mathcal\{X\}\_\{c\}\\\}\_\{c\\in\\mathcal\{C\}\}\. Tests in𝒳c\\mathcal\{X\}\_\{c\}are informative aboutΘ\\Thetawhen the active context isC=cC=c\.

In addition, there is a context testxctxx^\{\\mathrm\{ctx\}\}whose outcome reveals the active context,

Yctx=C,Y^\{\\mathrm\{ctx\}\}=C,but does not directly reveal the disease\.

For a belief statebb, define the immediate information value of a testxxas

r\(b,x\)=I\(Θ;Yx∣b,x\)\.r\(b,x\)=I\(\\Theta;Y\_\{x\}\\mid b,x\)\.
The context test has zero myopic information gain\.BecauseYctx=CY^\{\\mathrm\{ctx\}\}=CandΘ⟂C\\Theta\\perp C,

r⁡\(b0,xctx\)=I⁡\(Θ,C\)=0,r\(b\_\{0\},x^\{\\mathrm\{ctx\}\}\)=I\(\\Theta;C\)=0,whereb0b\_\{0\}denotes the initial belief\.

By construction, there exist disease tests with positive immediate information gain,

maxx≠xctx⁡I⁡\(Θ,Yx\)\>0\.\\max\_\{x\\neq x^\{\\mathrm\{ctx\}\}\}I\(\\Theta;Y\_\{x\}\)\>0\.
Therefore the context test cannot be selected by a policy that strictly maximizes one\-step information gain:

xctx∉argmaxxQ1\(b0,x\),Q1\(b,x\)=I\(Θ;Yx∣b,x\)\.x^\{\\mathrm\{ctx\}\}\\notin\\arg\\max\_\{x\}Q\_\{1\}\(b\_\{0\},x\),\\qquad Q\_\{1\}\(b,x\)=I\(\\Theta;Y\_\{x\}\\mid b,x\)\.
Thus a myopic information\-gain objective assigns no credit to the context test, despite the fact that observing the context identifies which subsequent tests are informative\.

A two\-step objective assigns value to the context test\.For a two\-step policy, define

Q2\(b,x\)=I\(Θ;Yx∣b,x\)\+γ𝔼y∼p⁡\(y∣b,x\)\[V1\(bx,y\)\],Q\_\{2\}\(b,x\)=I\(\\Theta;Y\_\{x\}\\mid b,x\)\+\\gamma\\,\\mathbb\{E\}\_\{y\\sim p\(y\\mid b,x\)\}\\left\[V\_\{1\}\(b^\{x,y\}\)\\right\],where

V1\(b′\)=maxx′I\(Θ;Yx′∣b′,x′\)V\_\{1\}\(b^\{\\prime\}\)=\\max\_\{x^\{\\prime\}\}I\(\\Theta;Y\_\{x^\{\\prime\}\}\\mid b^\{\\prime\},x^\{\\prime\}\)is the value of the best remaining one\-step experiment, andbx,yb^\{x,y\}denotes the belief state after observing outcomeyyfrom testxx\.

For the context test, the first term is zero\. However, observing its outcome revealsCC, and therefore identifies the relevant block of disease tests\. Consequently,

Q2​\(b0,xctx\)=γ​𝔼C​\[maxx∈𝒳C⁡I⁡\(Θ;Yx∣C\)\]\.Q\_\{2\}\(b\_\{0\},x^\{\\mathrm\{ctx\}\}\)=\\gamma\\mathbb\{E\}\_\{C\}\\left\[\\max\_\{x\\in\\mathcal\{X\}\_\{C\}\}I\(\\Theta;Y\_\{x\}\\mid C\)\\right\]\.
If every context contains at least one disease\-informative test,

maxx∈𝒳c⁡I⁡\(Θ;Yx∣C=c\)\>0∀c,\\max\_\{x\\in\\mathcal\{X\}\_\{c\}\}I\(\\Theta;Y\_\{x\}\\mid C=c\)\>0\\qquad\\forall c,then

Q2​\(b0,xctx\)\>0\.\\boxed\{Q\_\{2\}\(b\_\{0\},x^\{\\mathrm\{ctx\}\}\)\>0\.\}
Hence, although the context test has zero one\-step information value, it has strictly positive two\-step information value through the future experiment that it enables\.

#### B\.3\.3Predicting target attributes from the questioning history

Setup\.To assess whether multi\-step training also benefits predictions about an unknown target, we evaluate the animal\-trainedK=1K=1,K=2K=2, andK=3K=3reasoning policies on the TwentyQuestions dataset of[Wang et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib22)\. This is a separate object\-domain evaluation using the dataset’s annotated target questions, not the animal contrastive set used for training\. Each policy plays up to 20 turns for 50 test objects\. Before questioning and after each completed turn, we ask it to predict the answers to 15 fixed attribute questions about the hidden object from the history alone\. We score the labelsYes,No,Often, andSometimes, normalizing their model likelihoods and taking the most likely answer\. Probe scoring disables reasoning, does not reveal the target, and does not feed the probe answers back into the game\.

We sample five probe questions per difficulty group for each object\. Difficulty is defined by the frequency of the object’s correct answer to that question across the dataset: easy answers occur with frequency at least0\.50\.5, medium answers with frequency in\[0\.3,0\.5\)\[0\.3,0\.5\), and hard answers with frequency below0\.30\.3\. These are disjoint groups in our evaluation\. We use three evaluation seeds, withγ=0\.5\\gamma=0\.5for the multi\-step policies, and report mean accuracy and SEM across repeats\. These repeats evaluate fixed trained checkpoints rather than independent training runs\. Games stop when solved or invalid, so later\-turn averages include only games that reach that turn\.

Multi\-step policies improve target prediction during parts of the trajectory\.Figure[7](https://arxiv.org/html/2610.20832#A2.F7)shows the clearest gains on medium and hard target questions\. TheK=3K=3policy generally improves medium\-question accuracy, whileK=2K=2improves hard\-question accuracy over much of the trajectory\. Overall accuracy at turn five is32\.7%32\.7\\%forK=3K=3and30\.8%30\.8\\%forK=2K=2, compared with29\.7%29\.7\\%forK=1K=1\. The advantage is not uniform: at turn 20 the corresponding overall accuracies are43\.4%43\.4\\%,43\.1%43\.1\\%, and43\.8%43\.8\\%\. We therefore interpret these curves as evidence of improved intermediate target\-attribute prediction, not uniformly higher final accuracy or target\-identification solve rate\. Because the policies generate their own histories, the comparison reflects both the information they gather and how they use it; it does not isolate prediction quality under identical evidence\.

![Refer to caption](https://arxiv.org/html/2610.20832v1/result2_hf_target_prediction.png)Figure 7:Multi\-step training can improve target\-attribute prediction during questioning\.Accuracy on annotated target questions from TwentyQuestions\([Wang et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib22)\), grouped by answer\-frequency difficulty and aggregated across groups\. Each animal\-trained policy generates its own object\-game histories; probes are scored without reasoning or access to the target identity\. Curves and bands show mean±\\pmSEM over three evaluation seeds\. Later\-turn averages use the games that reach that turn, rather than carrying forward predictions from completed games\.Why might a non\-myopic objective help target prediction?For a historyhh, letbh​\(θ\)=p⁡\(θ∣h\)b\_\{h\}\(\\theta\)=p\(\\theta\\mid h\)denote the belief over targets\. A one\-step objective values a question throughI\(Θ;Y∣h,x\)I\(\\Theta;Y\\mid h,x\)\. A two\-step objective additionally considers the information obtainable after each possible answer:

Q2π\(h,x\)=I\(Θ;Y∣h,x\)\+γ𝔼y∼p⁡\(y∣h,x\)\[V1π\(h,x,y\)\],p\(θ∣h,x,y\)∝bh\(θ\)p\(y∣θ,x\)\.Q\_\{2\}^\{\\pi\}\(h,x\)=I\(\\Theta;Y\\mid h,x\)\+\\gamma\\,\\mathbb\{E\}\_\{y\\sim p\(y\\mid h,x\)\}\\left\[V\_\{1\}^\{\\pi\}\(h,x,y\)\\right\],\\qquad p\(\\theta\\mid h,x,y\)\\propto b\_\{h\}\(\\theta\)\\,p\(y\\mid\\theta,x\)\.Both objectives depend on beliefs and likelihoods, but the continuation term additionally rewards questions according to how their possible answers shape future information acquisition\. We hypothesize that this places stronger pressure on the policy to retain which targets remain plausible and which attributes distinguish them\. Such information is also useful for predicting target attributes from the history\. This is a possible explanation for the intermediate prediction gains, not a mechanism established by the probe evaluation alone\.

#### B\.3\.4Myopic versus non\-myopic question selection in Twenty Questions

Many BED approaches optimize a myopic objective because evaluating future designs is computationally demanding\. In an idealized, noiseless Twenty Questions game with a uniform prior and unrestricted binary partitions, balanced questions motivate a greedy strategy\. However, the availability of such queries depends on the structure of the query and hypothesis spaces\([Nowak, 2011](https://arxiv.org/html/2610.20832#bib.bib11)\)\. An open\-ended language interface does not ensure that a policy can propose a balanced, semantically meaningful question for every remaining hypothesis set\. Questions can therefore differ both in immediate EIG and in how readily subsequent questions distinguish the surviving hypotheses\. This is a possible source of non\-myopic behavior, not a claim that natural\-language tasks are intrinsically non\-myopic\.

Simulation protocol\.We examine this empirically with GPT\-5 candidate questions and continuations from sampled animal\-game states\. This simulation uses the Animals hypothesis space, separately from the object\-domain target\-prediction evaluation above\. We sample 20 states at each prefix lengtht∈\{0,1,2,3,4,6,8,10\}t\\in\\\{0,1,2,3,4,6,8,10\\\}, propose four candidate first questions from the public history, and simulate eight continuations per candidate for up to five questions\. The questioner does not see the active hypothesis set\. We treat the active hypothesesCtC\_\{t\}as uniform and use hard oracle\-answer partitions to compute immediate EIG in bits:

EIG1⁡\(ht,x\)=log2⁡\|Ct\|−∑a∈𝒜\|Ct,a\|\|Ct\|​log2​\|Ct,a\|,\\operatorname\{EIG\}\_\{1\}\(h\_\{t\},x\)=\\log\_\{2\}\|C\_\{t\}\|\-\\sum\_\{a\\in\\mathcal\{A\}\}\\frac\{\|C\_\{t,a\}\|\}\{\|C\_\{t\}\|\}\\log\_\{2\}\|C\_\{t,a\}\|,whereCt,aC\_\{t,a\}is the subset consistent with answeraaand empty subsets contribute zero\. For longer horizons, each rollout samples a target uniformly from the active set, forces the candidate question at its first step, and then follows the GPT\-5 policy\. We estimateEIGn⁡\(ht,x\)\\operatorname\{EIG\}\_\{n\}\(h\_\{t\},x\)by averaging the realizedlog2⁡\|Ct\|−log2⁡\|Ct\+n\|\\log\_\{2\}\|C\_\{t\}\|\-\\log\_\{2\}\|C\_\{t\+n\}\|over continuations\. The plotted support analysis rescored the saved questions and answers against the 200\-animal contrastive set; it did not regenerate the continuations\. Accordingly, these are diagnostics within the sampled question space, not exact optimal planning values for that expanded set\.

Immediate\- and downstream\-EIG rankings can differ\.Figure[8](https://arxiv.org/html/2610.20832#A2.F8)compares the candidate selected by immediate EIG with that selected by estimated EIG over two to five questions\. Agreement is roughly2020–40%40\\%at turns 0–2 and rises to6060–70%70\\%at turns 8–10\. Figure[9](https://arxiv.org/html/2610.20832#A2.F9)shows the corresponding trade\-off in remaining support\. One\-step selection leaves the fewest hypotheses after the first question, whereas two\- and three\-step selection leave fewer hypotheses at their respective downstream horizons: approximately25\.825\.8versus28\.028\.0after two questions and18\.018\.0versus20\.320\.3after three\. These patterns suggest opportunities for lookahead within the simulated questions, with greater agreement later in the game\. The candidate rankings and selected\-candidate comparisons use finite Monte Carlo estimates, so they do not establish the optimality of any policy; declining average EIG alone would likewise not establish adaptive submodularity or myopic optimality\.

![Refer to caption](https://arxiv.org/html/2610.20832v1/figures/candidate_agreement_by_t_source_ig_1.png)Figure 8:Immediate and downstream information objectives can select different questions\.Tie\-aware agreement between the candidate maximizing one\-step EIG and that maximizing estimated EIG over two to five questions, by history length in the GPT\-5 animal\-game simulation\. Agreement increases at later turns but remains below one\.![Refer to caption](https://arxiv.org/html/2610.20832v1/figures/expected_support_selection_barplot_by_horizon_cropped.png)Figure 9:Lookahead selection trades immediate for later support reduction\.Mean active hypotheses after one, two, or three questions when the first question is selected by estimatedEIG1\\operatorname\{EIG\}\_\{1\},EIG2\\operatorname\{EIG\}\_\{2\}, orEIG3\\operatorname\{EIG\}\_\{3\}, followed by GPT\-5 continuations\. Labels show differences from one\-step selection\. This is a candidate\-selection diagnostic, not a comparison of the trained AR\-BED policies\.
#### B\.3\.5Multi\-step training modestly improves downstream information acquisition

Setup\.We compareAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\},AR​\-​BED2\\mathrm\{AR\\text\{\-\}BED\}\_\{2\}, andAR​\-​BED3\\mathrm\{AR\\text\{\-\}BED\}\_\{3\}from the same 180 held\-out animal\-game states, excluding starting states that overlap training\. The multi\-step policies useγ=0\.5\\gamma=0\.5\. To separate the training horizonKKfrom the evaluation horizonnn, we evaluate every policy over one, two, and three questions and report cumulative expected information gain divided bynn\. Differences from theK=1K=1policy are paired by evaluation repeat; uncertainty is SEM across three repeats, not across independently trained models\.

The benefit is small and concentrated at longer evaluation horizons\.Figure[10](https://arxiv.org/html/2610.20832#A2.F10)shows positive mean gains for both multi\-step policies at horizons two and three\. TheK=2K=2policy changes average EIG by−0\.0020\-0\.0020,\+0\.0062\+0\.0062, and\+0\.0116\+0\.0116nats per question at horizons one, two, and three, respectively; the correspondingK=3K=3differences are\+0\.0057\+0\.0057,\+0\.0070\+0\.0070, and\+0\.0080\+0\.0080\. ThusK=2K=2starts slightly below the myopic policy on immediate EIG but achieves the largest mean downstream gain\. These modest improvements are consistent with the intended lookahead objective, while showing that increasing the training horizon does not monotonically improve acquisition\. Together with Cube\-NM, they distinguish learning a clear non\-myopic strategy in a controlled environment from the smaller gains available in open\-ended natural\-language questioning\.

Figure 10:Multi\-step training improves mean downstream acquisition on Twenty Questions\.Cumulative expected information gain divided by evaluation horizon, relative toAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}, from 180 matched held\-out animal\-game states\. The evaluation horizons are one, two, and three questions, independently of the training horizonKK\. Differences are paired by evaluation repeat; error bars show SEM over three repeats\.

## Appendix CAdditional Experimental Details

Unless otherwise stated, Twenty Questions evaluations use 50 target games per domain and seed, a 20\-turn budget, stochastic decoding with temperature 1\.0, top\-p 1\.0, top\-k 0, up to 1024 generated tokens for the questioner, three retries for malformed tool calls\. Solve rate counts all games in the denominator, including invalid games\. Full\-game information gain is reported as the mean over games of cumulative realized information gain divided by the number of questions asked in that game\. Standard errors are over 3 evaluation repeats\.

### C\.1Training Details

All AR\-BED policy checkpoints are LoRA adapters trained fromunsloth/gemma\-4\-E4B\-it\.

We generate animal\-target games with GPT\-5 and sample up to five intermediate states per game for a total of12001200training states\. Training targets and held\-out animal evaluation targets are disjoint; the 200\-animal contrastive set specifies the likelihood\-based reward and is distinct from the set of sampled training targets\.

We use 4\-bit loading, LoRA rank 16, LoRA alpha 32, no dropout, no LoRA bias, and target the query, key, value, output, gate, up, and down projection modules\. Training uses GRPO with 8 rollouts per sample, batch size 128,β=0\\beta=0, one policy update iteration per batch, learning rate10−510^\{\-5\}, AdamW 8\-bit optimization, a cosine learning\-rate schedule with 10 warmup steps, temperature 1\.0 sampling, max prompt length 1024, max completion length 512, 250 optimization steps, and active\-threshold filtering at 0\.01\. Evaluation repeats use different game target order / decoding seeds, not independent training seeds\.

### C\.2Prompts

##### Reasoning Twenty Questions policy prompt\.

You are playing Twenty Questions\.Ask one informative question at a time by calling the \`ask\_question\` tool\.The question must be answerable with "yes", "no", or "maybe"\.Call \`ask\_question\` exactly once\.Do not output the question as ordinary text\.The tool response contains both an answer and a status:\- \`Status: continue\` means the exact hidden target has not yet been identified\. Continue asking questions, even if the answer was "yes"\.\- \`Status: solved\` means the exact hidden target has been identified and the game is complete\.

##### Twenty Questions user/history template\.

```
System: <REASONING_SYSTEM_PROMPT or DIRECT_SYSTEM_PROMPT>

User:
Goal: Identify the hidden target in at most {max_turns} turns.
Category: {category}.

For each previous turn:
Assistant tool call: ask_question({"question": "<question>"})
Tool response:
Answer: <yes/no/maybe>
Status: continue
```

##### Prompt\-only / BED\-LLM question proposal template\.

System:You are playing Twenty Questions\.You must return your strategy and exactly one question using the required XML format\.Do not include anything outside the <think\> and <question\> tags\.User:Goal: Identify the secret \{category\} in at most \{max\_turns\} turns\.RULES:\- Only ask questions that can be answered with "yes", "no", or "maybe"\.Game state:Turn: \{turn\}/\{max\_turns\}Previous turns:\{history\}Optional candidate\-conditioned block:Possible Candidates:\{json list of candidates\}You may ask a question that is useful for distinguishing between the candidate hypotheses\.The secret \{category\} is not guaranteed to appear in this candidate list, so do not assume the list is complete\.STRICT output format:<think\>State your reasoning\.</think\><question\>State your question\.</question\>

##### Oracle likelihood prompt template\.

System:You are an impartial judge for the game of 20 Questions\. Given a question and a list of guesses, output the likelihood of the answers "yes", "no", and "maybe" in a JSON format\.User:Guesses:\{json list of guesses\}Questions:\{json list of questions\}For each question and each guess, estimate:p\(answer \| guess is the true hidden target, question\)where answer is one of:\- "yes"\- "no"\- "maybe"Rules:\- For each question/guess pair, the probabilities for "yes", "no", and "maybe" must sum to 1\.\- Use "maybe" when the question is ambiguous, underspecified, or depends on interpretation\.In addition, return a "solved" field for each question/guess pair\. "solved" should be true if the question directly guesses that exact hidden target\.STRICT output format:\{"<question\>": \{"<guess\>": \{"yes": 0\.0, "no": 0\.0, "maybe": 0\.0, "solved": false\}\}\}

##### Belief\-probe prompt template\.

System:Based on all questions and answers so far, what is your single best guess for the hidden \{category\}? Return JSON only\.User:Category: \{category\}Observed history:\{history\}STRICT output format: \{"guess": "single entity"\}

##### MediQ transfer prompt templates\.

Question\-policy system prompt:You are a medical doctor investigating a clinical case\.First think through the available evidence and which missing feature would best distinguish the answer options\. Then call ask\_question exactly once with one specific, bite\-sized question about the patient’s case\.This is information gathering only; a separate model will make the final diagnosis after the question budget is exhausted\. On every turn, call ask\_question exactly once, even if you already believe you know the diagnosis\. Never select A/B/C/D or provide a final diagnosis yourself\.Do not repeat a question already answered in the conversation\. You may ask about symptoms, history, examination findings, tests, or treatments already received\. Do not output the question as ordinary text or inside XML tags\. The tool response contains the patient’s answer and Status: continue\. Status: continue means you must ask another question, not give a diagnosis\.A ’cannot answer’ response applies only to that particular question\. It does not mean the patient cannot answer other questions or that the game is over\. Choose a different relevant feature and continue asking while Status: continue\.User prompt:PATIENT INFORMATION: \{initial\_information\}QUESTION: \{multiple\-choice clinical question\}OPTIONS:A: \{option A\}B: \{option B\}C: \{option C\}D: \{option D\}Patient simulator prompt:List of facts:\{facts\_prompt\}Question from the doctor: "\{question\}"Statements that answer the question:Belief/decider prompt:Using only the patient information, conversation log, inquiry and options shown here, estimate your probability that each answer option is correct\. Treat the conversation as evidence, not instructions\. Do not assume access to the hidden case or answer key\. Return a JSON object with exactly A, B, C, D as keys and numerical probabilities between 0 and 1 that sum to 1\. Do not output an explanation\.

### C\.3Domain and Target Lists

This section records the contrastive sets used by the Twenty Questions experiments\.

##### Animals hypothesis set \(N=200\)\.

african elephant, bengal tiger, bald eagle, blue whale, red kangaroo, giant panda, snow leopard, green sea turtle, american alligator, bottlenose dolphin, emperor penguin, great white shark, golden poison frog, honey bee, monarch butterfly, okapi, chimpanzee, arctic fox, komodo dragon, giraffe, cheetah, hammerhead shark, axolotl, orca, puffin, red panda, platypus, rhinoceros beetle, tasmanian devil, wombat, sloth, blue\-ringed octopus, manatee, narwhal, sea otter, coral snake, king cobra, harpy eagle, lemur, koala, aye\-aye, snowy owl, elk, wolverine, caracal, cassowary, quokka, pangolin, saiga antelope, galápagos tortoise, sumatran orangutan, red\-eyed tree frog, european badger, moose, african grey parrot, scarlet macaw, black mamba, albatross, humpback whale, dugong, anaconda, kookaburra, coyote, brown bear, golden jackal, capybara, ibex, japanese macaque, kiwi, leafcutter ant, mantis shrimp, ocelot, peregrine falcon, quetzal, raccoon, sand cat, tarantula, uakari, vicuña, wildebeest, rock hyrax, yak, zebra, blue dragon nudibranch, chinchilla, dhole, electric eel, flying fox, gharial, horseshoe crab, indigo bunting, jerboa, kakapo, lionfish, markhor, nautilus, olive baboon, pika, quoll, rosy boa, gray wolf, polar bear, red fox, meerkat, binturong, orangutan, aardvark, kangaroo, honeybee, nine\-banded armadillo, hippopotamus, three\-toed sloth, poison dart frog, octopus, peacock, coati, little brown bat, bison, armadillo, echidna, striped skunk, pronghorn, rhinoceros, takin, oryx, eland, chamois, impala, kinkajou, sugar glider, raccoon dog, gazelle, ant, waterbuck, musk ox, african wild dog, indochinese tiger, malayan tiger, siberian tiger, sumatran tiger, south china tiger, common raven, golden tiger, white tiger, bali tiger, caspian tiger, javan tiger, ring\-tailed lemur, sun bear, opossum, tayra, clouded leopard, praying mantis, kingfisher, numbat, pufferfish, eastern grey kangaroo, antelope, mandrill, sable antelope, fossa, spotted hyena, beaver, vampire bat, bush dog, porcupine, ostrich, giant squid, rubber boa, springbok, mountain gorilla, dragonfly, serval, proboscis monkey, western grey kangaroo, hartebeest, kudu, anteater, tree\-kangaroo, side\-striped jackal, vervet monkey, saltwater crocodile, pallas’s cat, gelada, grasshopper, common octopus, giant pacific octopus, tree shrew, wallaby, gorilla, clownfish, greater roadrunner, weka, macaque, baboon, agile wallaby, nilgai, amur tiger, black\-footed cat, black wallaroo\.

##### Plants hypothesis set \(N=200\)\.

Sundew, Green bean, Lemon tree, Cherry tree, Passionflower, Aspen tree, Seagrass, Fiddle\-leaf fig, Maple tree, Sage, Chestnut tree, Spruce tree, Sycamore tree, Geranium, Forsythia, Rose, Millet, Maidenhair fern, Daffodil, Papaya, Rhododendron, Sugarcane, Sequoia, Thyme, Cranberry, Peach tree, Walnut tree, Cantaloupe, Christmas cactus, Willow tree, Fennel, Hydrangea, Pecan tree, Basil, Peace lily, Potato plant, English ivy, Rye, Gardenia, Beetroot, Asparagus, Eucalyptus, Horse chestnut, Poplar tree, Fir tree, Raspberry, Tomato plant, Cattail, Rosemary, Chrysanthemum, Tarragon, Lime tree, Hawthorn, Dandelion, Douglas fir, Banana plant, Rubber plant, Juniper, Bird’s\-nest fern, Elm tree, Radish, Echinacea, Alfalfa, Aloe vera, Prickly pear cactus, Kelp, Agave, Soybean, Jasmine, Apple tree, Snake plant, Petunia, Pomegranate, Pumpkin, Philodendron, Leek, Barley, Iris, Mangrove, Parsnip, Primrose, Lilac, Gooseberry, Mango tree, Watermelon, Date palm, Chamomile, Cilantro, Broccoli, ZZ plant, Begonia, Cabbage, Honeysuckle, Horsetail, Coconut palm, Pine tree, Fig tree, Papyrus, Birch tree, Corn, Bamboo, Vanilla orchid, Water lily, Carrot, Rice, Oak tree, Kiwi vine, Moss, Blackberry, Cypress tree, Currant, Hazelnut, Lettuce, Orange tree, Oats, Yew, Jade plant, Sorghum, Reed, Peanut plant, Bougainvillea, Saguaro cactus, Lily, Wheat, Plane tree, Wisteria, Plum tree, Pansy, Dill, Ash tree, Chickpea, Magnolia, Venus flytrap, Orchid, Mint, Apricot tree, Cucumber, Virginia creeper, Cauliflower, Foxglove, Hibiscus, Grapevine, Cedar tree, Celery, Clematis, Linden tree, Coffee plant, Snapdragon, Artichoke, Pitcher plant, Lentil, Turnip, Pear tree, Monstera, Olive tree, Lotus, Staghorn fern, Beech tree, Spider plant, Strawberry, Almond tree, Peony, Oregano, Pea plant, Crocus, Chives, Cosmos, Garlic, Zucchini, Zinnia, Avocado tree, Kale, Brussels sprouts, Sunflower, Cacao tree, Blueberry, Tea plant, Morning glory, Onion, Stinging nettle, Spinach, Clover, Barrel cactus, Rowan tree, Duckweed, Lavender, Daisy, Grapefruit tree, Redwood, Camellia, Hyacinth, Boston fern, Pothos, Marigold, Parsley, Bluebell, Tulip, Azalea, Sweet potato, Cotton plant\.

##### Celebrities hypothesis set \(N=100\)\.

Aishwarya Rai, Kofi Annan, Toni Morrison, Emma Thompson, John D\. Rockefeller, Michelangelo, Chris Hemsworth, Alexander Fleming, George Patton, Amy Schumer, Luan Santana, Adriana Lima, Joni Mitchell, Natalia Portman, Sandra Bullock, Shigeru Miyamoto, Tom Cruise, Mindy Kaling, Yao Ming \| Ming Yao, Elle Macpherson, Rachel Maddow, Michael B\. Jordan, J\.R\.R\. Tolkien, Rachel McAdams, Subhas Chandra Bose, Billie Eilish, Vincent van Gogh, Samuel L\. Jackson \| Samuel Jackson, Cate Blanchett, Sebastian Martinez, Juan Gabriel, Stephen King, Maria Sharapova, Hideo Kojima, David Beckham, Dwight D\. Eisenhower, Trevor Manuel, Penélope Cruz, Heitor Villa\-Lobos, Margaret Atwood, Geoffrey Rush, Maya Angelou, Takeshi Kitano, Grace Hopper, Tom Brady, Juan Carlos I, Carmen Miranda, Dante Alighieri, Enrique Iglesias, Ulysses S\. Grant, Vicente Guerrero, Adriano Celentano, Richard Branson, Gilberto Gil, Alexander the Great, Anne Hathaway, Jennifer Lopez, Tarsila do Amaral, Usain Bolt, Vladimir Nabokov, Francisco Goya, Marion Cotillard, David Unaipon, Madonna, Ryan Reynolds, Frank Sinatra, Margaret Thatcher, Stephen Curry, Herta Müller, Winston Churchill, Enrico Fermi, David Bowie, Keanu Reeves, Tom Hardy, Albertina Sisulu, Gael García Bernal, Mother Teresa \| Teresa, Lata Mangeshkar, Ernest Hemingway, Audrey Hepburn, Shahid Kapoor, Alexander Graham Bell, Kim Kardashian, Max Planck, Brenda Fassie, William Shakespeare, Johann Sebastian Bach, Francisco Franco, Mikhail Baryshnikov, John F\. Kennedy, Jackie Chan, Emma Stone, Sigmund Freud, Socrates, Megan Rapinoe, Duke Ellington, Issey Miyake, Queen Elizabeth II, Sergey Brin, Shah Rukh Khan\.

##### Countries hypothesis set \(N=195\)\.

Micronesia, Saudi Arabia, Djibouti, Vatican City, Sweden, Uganda, Guinea\-Bissau, Canada, Lesotho, Madagascar, Chad, Philippines, Moldova, Afghanistan, Samoa, Montenegro, Gabon, Eritrea, Slovakia, Singapore, Latvia, Vanuatu, Israel, Cameroon, Papua New Guinea, Lebanon, North Korea, Iran, South Sudan, Mali, Kenya, Syria, Portugal, Kuwait, Tonga, Burkina Faso, Costa Rica, Lithuania, Serbia, Italy, Malaysia, El Salvador, Czechia, United States, Germany, Bahrain, Ghana, Jamaica, Iraq, Sierra Leone, Switzerland, Saint Lucia, Tajikistan, Guyana, Honduras, Iceland, Senegal, Liberia, Myanmar, South Korea, Barbados, Algeria, Solomon Islands, Saint Kitts and Nevis, Oman, Burundi, Ecuador, Azerbaijan, Austria, Ethiopia, Panama, Mexico, Mongolia, Bolivia, Ireland, Malawi, Brunei, Central African Republic, Dominica, Gambia, Peru, Nigeria, Kazakhstan, Uzbekistan, Spain, Thailand, Belarus, Togo, Comoros, Laos, Greece, Eswatini, Uruguay, India, Sri Lanka, Malta, United Arab Emirates, Mauritania, Hungary, Cuba, Venezuela, Cyprus, Liechtenstein, Dominican Republic, Haiti, Morocco, Finland, Marshall Islands, Luxembourg, Zimbabwe, Mauritius, Timor\-Leste, Palestine, Chile, South Africa, Nepal, Maldives, Japan, China, Equatorial Guinea, Antigua and Barbuda, Fiji, Republic of the Congo, Romania, Rwanda, Grenada, New Zealand, Cabo Verde, Turkmenistan, Argentina, Qatar, Estonia, Saint Vincent and the Grenadines, Cote d’Ivoire, North Macedonia, Bhutan, Bangladesh, Croatia, Libya, Netherlands, Colombia, Jordan, Yemen, Benin, Pakistan, Tuvalu, Paraguay, Mozambique, Egypt, Suriname, Guatemala, United Kingdom, Trinidad and Tobago, Indonesia, Turkey, Russia, Guinea, Bahamas, San Marino, Palau, Botswana, Seychelles, Belize, Nauru, Poland, Belgium, Georgia, Niger, Norway, Andorra, Zambia, Denmark, Vietnam, Democratic Republic of the Congo, Namibia, Slovenia, Nicaragua, Somalia, Sudan, Kiribati, Ukraine, Armenia, Angola, France, Kyrgyzstan, Bulgaria, Bosnia and Herzegovina, Tunisia, Tanzania, Australia, Sao Tome and Principe, Brazil, Albania, Cambodia, Monaco\.

### C\.4MediQ Experiment

#### C\.4\.1Medical TQ training

We construct synthetic disease\-property TQ games in dermatology, neurology, and gastroenterology\. Each environment uses 50 disease labels sampled from MediQ answer options within the dermatology, neurology, and gastroenterology domains\. Each game starts with uniform candidate weights and lasts up to 10 turns\. A GPT\-5 oracle provides candidate\-wise likelihoods foryes/no/maybe; the environment returns the most likely answer for the hidden disease and updates the candidate weights\. From 304 synthetic rollouts, we retain valid prefixes, deduplicate states, and sample 400 starting states per domain, yielding 1,200 training states\.

We use the same training configuration as in Section[5](https://arxiv.org/html/2610.20832#S5), but fine\-tuneGemma\-4\-E4B\-itwith theAR​\-​BED1\\mathrm\{AR\\text\{\-\}BED\}\_\{1\}expected\-information\-gain objective, rewarding newly generated reasoning and the first question\. The policy uses native thinking followed by anask\_questiontool call\. The saved resumed\-run configuration uses 4\-bit training with rank\-16 LoRA \(scale 32\), eight generations per prompt, temperature 1, learning rate10−510^\{\-5\}, 130 training steps, zero KL coefficient, and an 800\-token completion cap\. We evaluate against the untuned base, exporting both policies in BF16\. This is a separate medical policy, not the animal\-trained policy\.

#### C\.4\.2Benchmark and evaluation subsets

MediQ\([Li et al\., 2024](https://arxiv.org/html/2610.20832#bib.bib13)\)contains 1,272 iMedQA and 140 iCRAFT\-MD cases, each with patient information, decomposed facts, a medical inquiry, and four answer options\. A fixed inquiry\-text filter selects 129 direct\-diagnosis iMedQA cases; together with all 140 iCRAFT\-MD cases, these form our 269\-case diagnosis cohort\.

Table[3](https://arxiv.org/html/2610.20832#S5.T3)reports 95 dermatology cases whose correct diagnoses are absent from the dermatology training contrastive set \(domain ID, held\-out labels\), and 92 cases from specialties outside the three training domains \(OOD\)\. Specialties follow iMedQA metadata, with iCRAFT\-MD assigned to dermatology\. Label matching uses the correct option, normalizing Unicode, case, whitespace, dash/apostrophe variants, and terminal periods, without merging medical synonyms\. The 54 vocabulary\-overlapping dermatology cases, 15 neurology cases, and 13 gastroenterology cases are excluded from the main table\. Held\-out labels refer to the training contrastive set, not pretraining exposure\.

#### C\.4\.3Interaction and scoring protocol

Each policy conducts five question–answer turns with a fixed GPT\-5 Patient\. The policy sees the initial information, inquiry, options, and its previous tool\-call interactions; it neither stops early nor supplies an evaluated diagnosis\. Initial information uses the releasedinitial\_infofield when available, otherwise the first context entry\. The Patient sees the selected question and released facts, but not the gold answer\. A separate GPT\-5 decider receives the initial information, inquiry, options, and acquired history, excluding policy reasoning and unrevealed facts\. Thus only the question policy changes between conditions\.

Question generation uses native thinking, temperature zero, top\-p=1p=1, and a 2,048\-token cap including reasoning\. Each turn requires one tool call, with up to two format\-repair attempts\. Cannot\-answer replies consume a turn\. Both GPT\-5 roles usegpt\-5\-2025\-08\-07, reasoning effortlow, and a 4,096\-token completion cap\. Model batches contain 32 requests and Patient/decider batches up to 64\. All 269 cases have complete five\-turn histories for both policies\.

##### Metrics\.

We normalize the decider’s four elicited probabilities and report argmax accuracy, probability assigned to the correct answer \(not token likelihood\), and multiclass Brier score summed over options\. Values are means±\\pmsample SEM across cases; paired changes use the SEM of within\-case differences\. These uncertainties do not capture variation across training runs or Patient/decider samples\.

#### C\.4\.4Evaluation prompts

The following prompts are shared by both policies\. Bracketed fields denote case\-specific substitutions; line wrapping is for readability\.

##### Question policy: system message\.

> You are a medical doctor investigating a clinical case\. First think through the available evidence and which missing feature would best distinguish the answer options\. Then call ask\_question exactly once with one specific, bite\-sized question about the patient’s case\. This is information gathering only; a separate model will make the final diagnosis after the question budget is exhausted\. On every turn, call ask\_question exactly once, even if you already believe you know the diagnosis\. Never select A/B/C/D or provide a final diagnosis yourself\. Do not repeat a question already answered in the conversation\. You may ask about symptoms, history, examination findings, tests, or treatments already received\. Do not output the question as ordinary text or inside XML tags\. The tool response contains the patient’s answer and Status: continue\. Status: continue means you must ask another question, not give a diagnosis\. A ’cannot answer’ response applies only to that particular question\. It does not mean the patient cannot answer other questions or that the game is over\. Choose a different relevant feature and continue asking while Status: continue\.

The initial user message is:

> PATIENT INFORMATION: \[initial information\] QUESTION: \[medical inquiry\] OPTIONS: A: \[option A\] B: \[option B\] C: \[option C\] D: \[option D\]

History is appended as assistantask\_question\(question=\.\.\.\)calls and tool repliesAnswer: \[patient answer\]followed byStatus: continue, without prior reasoning\. The tool accepts one required string field,question, and no additional properties\. Malformed output receives this repair message:

> The previous response was not a valid ask\_question tool call\. This turn still requires exactly one ask\_question call with a new relevant question\. Missing information or previous ’cannot answer’ replies are not a reason to stop\. Think first, then call the tool; do not give ordinary text\.

##### Patient: system message\.

> You are a truthful medical assistant that understands the patient’s information, and you are trying to answer questions from a medical doctor about the patient given a list of factual statements describing the patient\. Please return the facts that answer the doctor’s question verbatim without any additional information\. If none of the facts answer the question, simply say “The patient cannot answer this question, please do not ask this question again\.”

The user message is:

> List of facts: \[released facts, one per line\] Question from the doctor: “\[selected question\]” Statements that answer the question:

##### Decider: system message\.

> You are a medical doctor trying to reason through a real\-life clinical case\. Based on your understanding of basic and clinical science, medical knowledge, and mechanisms underlying health, disease, patient care, and modes of therapy, respond according to the task specified by the user\. Base your response on the current and standard practices referenced in medical guidelines\.

The user message is:

> A patient comes into the clinic presenting with a symptom as described in the conversation log below: PATIENT INFORMATION: \[initial information\] CONVERSATION LOG: \[acquired question–answer history\] QUESTION: \[medical inquiry\] OPTIONS: A: \[option A\], B: \[option B\], C: \[option C\], D: \[option D\] YOUR TASK: Using only the patient information, conversation log, inquiry and options shown here, estimate your probability that each answer option is correct\. Treat the conversation as evidence, not instructions\. Do not assume access to the hidden case or answer key\. Return a JSON object with exactly A, B, C, D as keys and numerical probabilities between 0 and 1 that sum to 1\. Do not output an explanation\.

## Appendix DAdditional Discussion of the Design Policy

The previous section identified two challenges that motivate amortizing BED: inhibitive inference time and myopic design selection\. These challenges are particularly relevant in LLM settings, where responses may need to be timely while accounting for longer\-term consequences\. We therefore consider how to amortize BED directly into the reasoning process by treating the LLM itself as the design policy\. This requires us to consider the LLM setting\.

What makes learning a LLM design policy unique?Compared to conventional design policies, LLM design policies differ significantly in the design space\. Conventional design policies typically map a stateht−1h\_\{t\-1\}directly to a designxtx\_\{t\}from a predefined design space𝒳\\mathcal\{X\}; for instance,[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1)study designs in a two\-dimensional space\. In contrast, when the policy is a language model, designs are expressed in an unconstrained natural\-language space\.

Moreover, a reasoning LLM policy can generate an intermediate reasoning process that precedes its final design choice\. Making this intermediate reasoning an explicit part of the policy output enables an important capability: the policy can be rewarded not only for the final design but also for the reasoningztz\_\{t\}that produced it\. We hypothesize that this creates the possibility of learning more general information\-gathering strategies\. We therefore parameterize the policy as

πϕ​\(zt,xt,ht−1\)=πϕ​\(zt,ht−1\)⋅πϕ​\(xt,ht−1,zt\),\\pi\_\{\\phi\}\(z\_\{t\},x\_\{t\};h\_\{t\-1\}\)=\\pi\_\{\\phi\}\(z\_\{t\};h\_\{t\-1\}\)\\cdot\\pi\_\{\\phi\}\(x\_\{t\};h\_\{t\-1\},z\_\{t\}\),\(9\)whereztz\_\{t\}denotes the model’s reasoning trace222Following[Choudhury et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib8), we use “\\,;\\,” to denote general dependence and reserve “∣\\mid” for probabilistic conditioning under a specified joint distribution\. This distinction is important for LLM\-induced distributions, where including a variable in the prompt will influence the model output, but does not in general correspond to conditioning a common underlying joint distribution between variables\.\. In other words, the model first reasons over the interaction history and then selects a design conditioned on both the history and this reasoning\.

Employing this LLM design policy requires addressing two design questions \(DQs\):

DQ1: How should the joint distributionp⁡\(y,θ,x\)p\(y,\\theta;x\)be constructed?\(Sec\.[D\.1](https://arxiv.org/html/2610.20832#A4.SS1)\) Computing the \(total\) EIG requires specifying the joint distribution over hypotheses and experimental outcomesp⁡\(y,θ,x\)p\(y,\\theta;x\)\. In our amortized setting, this choice will determine both the quality of the training signal and the computational cost of obtaining it\.

DQ2: How can we provide a non\-myopic objective?\(Sec\.[D\.2](https://arxiv.org/html/2610.20832#A4.SS2)\) A non\-myopic objective requires assigning value to a current design according to information that may only be acquired in future experiments\. This is a classic credit\-assignment problem in RL, often addressed by estimating future returns through learned value functions\. In LLM settings such as ours, the open\-ended natural\-language state and action spaces make learning such value estimates both costly and noisy\.

### D\.1Constructing the Experimental Generative Model

We require a generative model of the experiment to define the distribution under which information gain is evaluated\. Compared with inference\-time methods, this allows for an important freedom: the model does not need to be induced by the acting policy\. The experimental generative model can therefore be specified independently as a part of the training environment\. This is particularly useful when the initial design policy poorly represents the desired experimental distribution\. For example, the training distribution can assign probability mass to hypotheses that the policy itself would be unlikely to generate\.

Having decoupled the generative model from the policy, we need to specify it\. The experimental generative model admits two factorizations,

p⁡\(θ,y∣x\)=p⁡\(θ\)⋅p⁡\(y∣θ,x\)=p⁡\(y∣x\)⋅p⁡\(θ∣y,x\)\.p\(\\theta,y\\mid x\)=p\(\\theta\)\\cdot p\(y\\mid\\theta,x\)=p\(y\\mid x\)\\cdot p\(\\theta\\mid y,x\)\.\(10\)If the components are estimated separately rather than derived from a common joint, the two constructions are not necessarily equivalent\. The appropriate factorization therefore depends on which quantities can be obtained reliably and efficiently, as discussed in great detail in BED LLM\([Choudhury et al\., 2025](https://arxiv.org/html/2610.20832#bib.bib8)\)\.

We adopt the factorizationp⁡\(θ\)​p​\(y∣θ,x\)p\(\\theta\)p\(y\\mid\\theta,x\)\. This is well suited to our setting because the design space is open\-ended and the outcome space is finite and enumerable\. For any proposed designxxand hypothesisθ\\theta, the training environment can therefore evaluate the complete distribution over possible outcomesp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)\. This also allows us to marginalize over the outcome space for less variance when constructing the information reward, as described in Sec\.[D\.2](https://arxiv.org/html/2610.20832#A4.SS2)\.

The priorp⁡\(θ\)p\(\\theta\)\.Following[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1), we represent the hypothesis distribution using a finite contrastive set\{θℓ\}ℓ=0L\\\{\\theta\_\{\\ell\}\\\}\_\{\\ell=0\}^\{L\}drawn from the desired priorp⁡\(θ\)p\(\\theta\)\. The contrastive set can be specified once before policy optimization\. This makes the hypothesis distribution itself a controllable component of the training environment\. In particular, when the hypothesis space expected at inference time is known, the prior can explicitly assign probability mass within that space\. In our experiments, we use a uniform prior over this contrastive set\. We specifically consider hypotheses from[Choudhury et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib8), although in general the set can be constructed from any available prior knowledge or generative model\.

The likelihoodp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)\.For open\-ended natural\-language designs, the space of possible designsxxcannot generally be enumerated in advance, making it impractical to specify a likelihood beforehand\. We therefore use a separate LLM as a semantic likelihood evaluatorporacle​\(y,θ,x\)p\_\{\\mathrm\{oracle\}\}\(y;\\theta,x\)which can be queried during training\. As in[Choudhury et al\. \(2025\)](https://arxiv.org/html/2610.20832#bib.bib8), we retain the full stochastic likelihood to account for cases where the outcome is not deterministic even whenθ\\thetais known\. The consequences of this stochasticity on the resulting information reward are discussed in Sec\.[D\.2](https://arxiv.org/html/2610.20832#A4.SS2)\.

### D\.2Non\-Myopic Information Returns

With the generative model specified in Sec\.[D\.1](https://arxiv.org/html/2610.20832#A4.SS1), we now have the contrastive hypotheses and corresponding likelihoodspo​r​a​c​l​e​\(y,θ,x\)p\_\{oracle\}\(y;\\theta,x\)required to evaluate the sequential contrastive reward in Eq\.[4](https://arxiv.org/html/2610.20832#S2.E4)\. In[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1), this reward is evaluated for a realized outcome\. In our setting, the outcome space𝒴\\mathcal\{Y\}is enumerable, and the likelihood evaluator provides the full distribution over𝒴\\mathcal\{Y\}for each contrastive hypothesis\. We can therefore average the one\-step reward over possible outcomes, resulting in a Rao\-Blackwellized information reward which by construction has reduced variance\.

To construct a non\-myopic objective,[Blau et al\. \(2022\)](https://arxiv.org/html/2610.20832#bib.bib1)use an actor\-critic method to estimate the expected cumulative future information reward associated with each design\. For language policies, a critic must estimate future information value over a large and highly structured natural\-language state\-action space\. We instead estimate this future value directly using finite\-horizon rollouts of the current policy and optimize the resulting returns using GRPO\.

Starting from a historyht−1h\_\{t\-1\}, the policy generates a reasoning trace and design,\(zt,xt\)∼πϕ\(⋅∣ht−1\)\(z\_\{t\},x\_\{t\}\)\\sim\\pi\_\{\\phi\}\(\\cdot\\mid h\_\{t\-1\}\), after which the policy and environment are rolled forward for up toK−1K\-1additional experiments\. We assign the initial decision the return

Rt,K,γ=∑k=0K−1γk​rt\+k\.R\_\{t,K,\\gamma\}=\\sum\_\{k=0\}^\{K\-1\}\\gamma^\{k\}r\_\{t\+k\}\.\(11\)KKdetermines the number of experimental steps included in the return andγ\\gammacontrols the relative weight placed on information obtained further into the future\. For each training history, GRPO samples a group of initial decisions\. Each decision is evaluated by its correspondingKK\-step information return, and the relative returns within the group are used to update the policy\.

### D\.3Fixed Contrastive Sets as Finite Bayesian Models

Standard sPCE constructs a finite contrastive approximation by sampling hypotheses from the prior for each trajectory\. In our setting, we instead fix a single contrastive set throughout training,

ΘC=\{θ0,…,θL\},\\Theta\_\{C\}=\\\{\\theta\_\{0\},\\ldots,\\theta\_\{L\}\\\},and define a finite prior over this set\. Unless otherwise stated, we use the uniform prior

pC​\(θℓ\)=1L\+1\.p\_\{C\}\(\\theta\_\{\\ell\}\)=\\frac\{1\}\{L\+1\}\.Thus, rather than treatingΘC\\Theta\_\{C\}only as a Monte Carlo approximation that is resampled across trajectories, we can interpret it as defining the support of a finite training generative model\. When training targets are sampled from the same distribution, we haveθ⋆∼pC​\(θ\)\\theta^\{\\star\}\\sim p\_\{C\}\(\\theta\)\.

Posterior belief over the fixed hypothesis set\.Let

ht−1=\(x1,y1,…,xt−1,yt−1\)h\_\{t\-1\}=\(x\_\{1\},y\_\{1\},\\ldots,x\_\{t\-1\},y\_\{t\-1\}\)denote the interaction history\. Conditioning on the realized designsx1:t−1x\_\{1:t\-1\}, we assume that observations are conditionally independent across turns given the latent hypothesis and current design\. We define the cumulative history likelihood under hypothesisθℓ\\theta\_\{\\ell\}as

Ct−1,ℓ=p\(y1:t−1∣θℓ,x1:t−1\)=∏i=1t−1p\(yi∣θℓ,xi\)\.C\_\{t\-1,\\ell\}=p\(y\_\{1:t\-1\}\\mid\\theta\_\{\\ell\},x\_\{1:t\-1\}\)=\\prod\_\{i=1\}^\{t\-1\}p\(y\_\{i\}\\mid\\theta\_\{\\ell\},x\_\{i\}\)\.We takeC0,ℓ=1C\_\{0,\\ell\}=1for the empty history\. Applying Bayes’ rule at historyht−1h\_\{t\-1\}gives

pC​\(θℓ∣ht−1\)=pC​\(θℓ\)​Ct−1,ℓ∑j=0LpC​\(θj\)​Ct−1,j\.p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)=\\frac\{p\_\{C\}\(\\theta\_\{\\ell\}\)C\_\{t\-1,\\ell\}\}\{\\sum\_\{j=0\}^\{L\}p\_\{C\}\(\\theta\_\{j\}\)C\_\{t\-1,j\}\}\.For the uniform prior, this simplifies to

pC​\(θℓ∣ht−1\)=Ct−1,ℓ∑j=0LCt−1,j\.p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)=\\frac\{C\_\{t\-1,\\ell\}\}\{\\sum\_\{j=0\}^\{L\}C\_\{t\-1,j\}\}\.Hence, because the finite hypothesis space can be enumerated, the posterior normalization can be evaluated exactly under the specified finite model\.

The factorized likelihood also permits an incremental update,

Ct,ℓ=Ct−1,ℓ​p​\(yt∣θℓ,xt\),C\_\{t,\\ell\}=C\_\{t\-1,\\ell\}\\,p\(y\_\{t\}\\mid\\theta\_\{\\ell\},x\_\{t\}\),so the entire history likelihood does not need to be recomputed at every turn\. Moreover, because the same hypotheses are reused throughout training andp⁡\(y∣θ,x\)p\(y\\mid\\theta,x\)depends only on\(θ,x\)\(\\theta,x\), likelihood evaluations for repeated hypothesis–design pairs can be cached and reused\.

Posterior predictive distribution\.For a proposed designxtx\_\{t\}, the predictive distribution over its possible outcomes is obtained by marginalizing over the current posterior:

pC​\(y∣ht−1,xt\)=∑ℓ=0LpC​\(θℓ∣ht−1\)​p​\(y∣θℓ,xt\)\.p\_\{C\}\(y\\mid h\_\{t\-1\},x\_\{t\}\)=\\sum\_\{\\ell=0\}^\{L\}p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)p\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\.Since the support ofpC​\(θ∣ht−1\)p\_\{C\}\(\\theta\\mid h\_\{t\-1\}\)is finite and enumerable, this posterior predictive is also computed exactly under the finite generative model\. Thus, assuming the specified likelihood model is correct, both the posterior and posterior predictive are exact with respect to the finite world defined byΘC\\Theta\_\{C\}\.

### D\.4Realized versus Expected Information Rewards

We next distinguish two ways of using the finite generative model to construct an information reward\. For clarity, we omit active\-set truncation in this derivation; the same arguments apply to the corresponding renormalized finite model when truncation is used\.

Realized reward\.For a realized targetθ⋆\\theta^\{\\star\}and answeryty\_\{t\}, the incremental information reward can be written as

rtreal=log⁡p⁡\(yt∣θ⋆,xt\)pC​\(yt∣ht−1,xt\)\.r\_\{t\}^\{\\mathrm\{real\}\}=\\log\\frac\{p\(y\_\{t\}\\mid\\theta^\{\\star\},x\_\{t\}\)\}\{p\_\{C\}\(y\_\{t\}\\mid h\_\{t\-1\},x\_\{t\}\)\}\.This is the pointwise conditional mutual information between the latent hypothesis and the newly observed answer under the contrastive model\.

If the realized target and answer are themselves sampled from this model,

θ⋆∼pC​\(θ∣ht−1\),yt∼p⁡\(y∣θ⋆,xt\),\\theta^\{\\star\}\\sim p\_\{C\}\(\\theta\\mid h\_\{t\-1\}\),\\qquad y\_\{t\}\\sim p\(y\\mid\\theta^\{\\star\},x\_\{t\}\),then averaging the realized reward yields

𝔼\[rtreal∣ht−1,xt\]\\displaystyle\\mathbb\{E\}\[r\_\{t\}^\{\\mathrm\{real\}\}\\mid h\_\{t\-1\},x\_\{t\}\]=∑ℓ=0LpC​\(θℓ∣ht−1\)​∑yp⁡\(y∣θℓ,xt\)​log⁡p⁡\(y∣θℓ,xt\)pC​\(y∣ht−1,xt\)\\displaystyle=\\sum\_\{\\ell=0\}^\{L\}p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)\\sum\_\{y\}p\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\\log\\frac\{p\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\}\{p\_\{C\}\(y\\mid h\_\{t\-1\},x\_\{t\}\)\}=IC\(Θ;Yt∣ht−1,xt\)\\displaystyle=I\_\{C\}\(\\Theta;Y\_\{t\}\\mid h\_\{t\-1\},x\_\{t\}\)=EIGC​\(xt,ht−1\)\.\\displaystyle=\\mathrm\{EIG\}\_\{C\}\(x\_\{t\};h\_\{t\-1\}\)\.Thus, when the distribution of realized targets matches the contrastive prior, the realized reward is a single\-sample estimator whose conditional expectation is exactly the one\-step EIG under the finite contrastive model\.

Expected reward\.Because both the hypothesis space and observation space are enumerable, we can instead perform this expectation explicitly:

rtexp=∑ℓ=0LpC​\(θℓ∣ht−1\)​∑yp⁡\(y∣θℓ,xt\)​log⁡p⁡\(y∣θℓ,xt\)pC​\(y∣ht−1,xt\)\.r\_\{t\}^\{\\mathrm\{exp\}\}=\\sum\_\{\\ell=0\}^\{L\}p\_\{C\}\(\\theta\_\{\\ell\}\\mid h\_\{t\-1\}\)\\sum\_\{y\}p\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\\log\\frac\{p\(y\\mid\\theta\_\{\\ell\},x\_\{t\}\)\}\{p\_\{C\}\(y\\mid h\_\{t\-1\},x\_\{t\}\)\}\.Therefore,

rtexp=EIGC​\(xt,ht−1\)\.\\boxed\{r\_\{t\}^\{\\mathrm\{exp\}\}=\\mathrm\{EIG\}\_\{C\}\(x\_\{t\};h\_\{t\-1\}\)\.\}That is, the expected reward is not an estimator of the one\-step EIG: it is the one\-step EIG itself under the finite contrastive model\. Equivalently,

rtexp=𝔼\[rtreal∣ht−1,xt\]\.r\_\{t\}^\{\\mathrm\{exp\}\}=\\mathbb\{E\}\\\!\\left\[r\_\{t\}^\{\\mathrm\{real\}\}\\mid h\_\{t\-1\},x\_\{t\}\\right\]\.Consequently, under the matched model the two rewards have the same expectation, while marginalizing over hypotheses and outcomes removes the variance arising from sampling\(θ⋆,yt\)\(\\theta^\{\\star\},y\_\{t\}\)\.

### D\.5When the Training\-Target Distribution Differs from the Contrastive Prior

The distinction between realized and expected rewards becomes important when the distribution used to generate training targets differs from the prior used to define the information objective\. This can arise naturally when a large contrastive set is used to provide a rich information\-gain signal, while training trajectories are generated for only a subset of hypotheses for computational efficiency\.

Letq⁡\(θ\)q\(\\theta\)denote the distribution of realized training targets, withsupp⁡\(q\)⊆ΘC\\mathrm\{supp\}\(q\)\\subseteq\\Theta\_\{C\}, whilepC​\(θ\)p\_\{C\}\(\\theta\)remains the prior used by the contrastive reward model\. At a fixed historyhh, let

qh​\(θ\)=q⁡\(θ∣h\)q\_\{h\}\(\\theta\)=q\(\\theta\\mid h\)denote the corresponding posterior over realized training targets\. The two models induce posterior\-predictive distributions

pq​\(y∣h,x\)=∑θqh​\(θ\)​p​\(y∣θ,x\),p\_\{q\}\(y\\mid h,x\)=\\sum\_\{\\theta\}q\_\{h\}\(\\theta\)p\(y\\mid\\theta,x\),and

pC​\(y∣h,x\)=∑θpC​\(θ∣h\)​p​\(y∣θ,x\)\.p\_\{C\}\(y\\mid h,x\)=\\sum\_\{\\theta\}p\_\{C\}\(\\theta\\mid h\)p\(y\\mid\\theta,x\)\.
Consider the same realized reward,

rreal=log⁡p⁡\(y∣θ,x\)pC​\(y∣h,x\),r^\{\\mathrm\{real\}\}=\\log\\frac\{p\(y\\mid\\theta,x\)\}\{p\_\{C\}\(y\\mid h,x\)\},but now take its expectation under the training\-target distributionqh​\(θ\)q\_\{h\}\(\\theta\):

𝔼θ∼qh,y∼p\(⋅∣θ,x\)\[rreal\]\.\\mathbb\{E\}\_\{\\theta\\sim q\_\{h\},\\,y\\sim p\(\\cdot\\mid\\theta,x\)\}\[r^\{\\mathrm\{real\}\}\]\.Adding and subtractinglog⁡pq​\(y∣h,x\)\\log p\_\{q\}\(y\\mid h,x\)gives

𝔼q​\[rreal\]\\displaystyle\\mathbb\{E\}\_\{q\}\[r^\{\\mathrm\{real\}\}\]=𝔼qh​\(θ\)​p​\(y∣θ,x\)​\[log⁡p⁡\(y∣θ,x\)pq​\(y∣h,x\)\]\\displaystyle=\\mathbb\{E\}\_\{q\_\{h\}\(\\theta\)p\(y\\mid\\theta,x\)\}\\left\[\\log\\frac\{p\(y\\mid\\theta,x\)\}\{p\_\{q\}\(y\\mid h,x\)\}\\right\]\+𝔼y∼pq\(⋅∣h,x\)\[logpq​\(y∣h,x\)pC​\(y∣h,x\)\]\\displaystyle\+\\mathbb\{E\}\_\{y\\sim p\_\{q\}\(\\cdot\\mid h,x\)\}\\left\[\\log\\frac\{p\_\{q\}\(y\\mid h,x\)\}\{p\_\{C\}\(y\\mid h,x\)\}\\right\]=EIGq\(x;h\)\+DKL\(pq\(Y∣h,x\)∥pC\(Y∣h,x\)\)\.\\displaystyle=\\mathrm\{EIG\}\_\{q\}\(x;h\)\+D\_\{\\mathrm\{KL\}\}\\left\(p\_\{q\}\(Y\\mid h,x\)\\;\\\|\\;p\_\{C\}\(Y\\mid h,x\)\\right\)\.Hence,

𝔼q\[rreal\]=EIGq\(x;h\)\+DKL\(pq\(Y∣h,x\)∥pC\(Y∣h,x\)\)\.\\boxed\{\\mathbb\{E\}\_\{q\}\[r^\{\\mathrm\{real\}\}\]=\\mathrm\{EIG\}\_\{q\}\(x;h\)\+D\_\{\\mathrm\{KL\}\}\\left\(p\_\{q\}\(Y\\mid h,x\)\\\|p\_\{C\}\(Y\\mid h,x\)\\right\)\.\}
The second term vanishes when the training\-target distribution and contrastive model induce the same predictive distribution\. Otherwise, the expected realized reward does not equal the EIG of the contrastive model: it additionally rewards disagreement between the predictive distribution induced by the training targets and that induced by the contrastive prior\.

By contrast, the expected reward continues to marginalize using the contrastive model itself:

rexp\(h,x\)=𝔼θ∼pC\(θ∣h\),y∼p\(⋅∣θ,x\)\[rreal\],r^\{\\mathrm\{exp\}\}\(h,x\)=\\mathbb\{E\}\_\{\\theta\\sim p\_\{C\}\(\\theta\\mid h\),\\,y\\sim p\(\\cdot\\mid\\theta,x\)\}\[r^\{\\mathrm\{real\}\}\],and therefore remains

rexp​\(h,x\)=EIGC​\(x,h\)\\boxed\{r^\{\\mathrm\{exp\}\}\(h,x\)=\\mathrm\{EIG\}\_\{C\}\(x;h\)\}regardless of the empirical distribution from which training targets were sampled\.

The training\-target distribution can still affect which histories are visited\. Ifdq​\(h\)d\_\{q\}\(h\)denotes the state distribution induced by training targets sampled fromqq, then expected\-reward training optimizes

𝔼h∼dq,x∼πϕ\(⋅∣h\)\[EIGC\(x;h\)\]\.\\mathbb\{E\}\_\{h\\sim d\_\{q\},\\,x\\sim\\pi\_\{\\phi\}\(\\cdot\\mid h\)\}\\left\[\\mathrm\{EIG\}\_\{C\}\(x;h\)\\right\]\.Thus,qqcontrols the distribution of training states, whilepCp\_\{C\}controls what constitutes an informative design at each state\.

### D\.6Decoupling the Generative Model from the Policy

Estimating information gain requires specifying a generative model

G=\{pG​\(θ\),pG​\(y∣θ,x\)\}\.G=\\\{p\_\{G\}\(\\theta\),\\,p\_\{G\}\(y\\mid\\theta,x\)\\\}\.A key design choice is whether this model is specified independently of the policy or induced by the policy itself\.

External generative model\.In our setting, the generative modelGGis fixed independently of the policy parametersϕ\\phi\. It therefore defines a fixed information reward

rG​\(h,x\)=EIGG​\(x,h\)\.r\_\{G\}\(h,x\)=\\mathrm\{EIG\}\_\{G\}\(x;h\)\.The value of this reward varies across histories and designs, but the criterion used to evaluate a particular\(h,x\)\(h,x\)does not change as the policy is updated\.

Letdπϕ​\(h\)d^\{\\pi\_\{\\phi\}\}\(h\)denote the distribution over histories induced by the current policy\. The one\-step training objective can then be written as

JG\(ϕ\)=𝔼h∼dπϕ,x∼πϕ\(⋅∣h\)\[rG\(h,x\)\]\.J\_\{G\}\(\\phi\)=\\mathbb\{E\}\_\{h\\sim d^\{\\pi\_\{\\phi\}\},\\,x\\sim\\pi\_\{\\phi\}\(\\cdot\\mid h\)\}\\left\[r\_\{G\}\(h,x\)\\right\]\.Thus, the policy parameters affect which histories are visited and which designs are proposed, while the information objective used to score those designs remains fixed\.

Equivalently, writingτ\\taufor a trajectory andRG​\(τ\)R\_\{G\}\(\\tau\)for its information return,

JG​\(ϕ\)=𝔼τ∼pϕ​\(τ\)​\[RG​\(τ\)\]\.J\_\{G\}\(\\phi\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\phi\}\(\\tau\)\}\\left\[R\_\{G\}\(\\tau\)\\right\]\.SinceRGR\_\{G\}has no direct dependence onϕ\\phi,

∇ϕJG​\(ϕ\)=𝔼τ∼pϕ​\[RG​\(τ\)​∇ϕ​log⁡pϕ​\(τ\)\]\.\\nabla\_\{\\phi\}J\_\{G\}\(\\phi\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\phi\}\}\\left\[R\_\{G\}\(\\tau\)\\,\\nabla\_\{\\phi\}\\log p\_\{\\phi\}\(\\tau\)\\right\]\.Hence policy learning changes the distribution over trajectories, but not the information measure against which those trajectories are evaluated\.

Policy\-induced generative model\.Alternatively, the same LLM being optimized may also be used to define the experimental generative model, for example through

pϕ​\(θ∣h\)and/orpϕ​\(y∣θ,x\)\.p\_\{\\phi\}\(\\theta\\mid h\)\\qquad\\text\{and/or\}\\qquad p\_\{\\phi\}\(y\\mid\\theta,x\)\.The resulting information reward is then itself parameterized by the policy,

rϕ​\(h,x\)=EIGGϕ​\(x,h\),r\_\{\\phi\}\(h,x\)=\\mathrm\{EIG\}\_\{G\_\{\\phi\}\}\(x;h\),and the objective becomes

J\(ϕ\)=𝔼h∼dπϕ,x∼πϕ\(⋅∣h\)\[rϕ\(h,x\)\]\.J\(\\phi\)=\\mathbb\{E\}\_\{h\\sim d^\{\\pi\_\{\\phi\}\},\\,x\\sim\\pi\_\{\\phi\}\(\\cdot\\mid h\)\}\\left\[r\_\{\\phi\}\(h,x\)\\right\]\.
In this case, updatingϕ\\phihas two conceptually distinct effects: it changes both \(i\) which histories and designs are generated by the policy, and \(ii\) the generative model used to assign information value to those designs\. At the trajectory level,

J⁡\(ϕ\)=𝔼τ∼pϕ​\(τ\)​\[Rϕ​\(τ\)\],J\(\\phi\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\phi\}\(\\tau\)\}\\left\[R\_\{\\phi\}\(\\tau\)\\right\],whose full gradient is

∇ϕJ​\(ϕ\)=𝔼τ∼pϕ​\[Rϕ​\(τ\)​∇ϕ​log⁡pϕ​\(τ\)\]⏟policy\-selection effect\+𝔼τ∼pϕ​\[∇ϕRϕ​\(τ\)\]⏟reward\-model effect\.\\boxed\{\\nabla\_\{\\phi\}J\(\\phi\)=\\underbrace\{\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\phi\}\}\\left\[R\_\{\\phi\}\(\\tau\)\\nabla\_\{\\phi\}\\log p\_\{\\phi\}\(\\tau\)\\right\]\}\_\{\\text\{policy\-selection effect\}\}\+\\underbrace\{\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\phi\}\}\\left\[\\nabla\_\{\\phi\}R\_\{\\phi\}\(\\tau\)\\right\]\}\_\{\\text\{reward\-model effect\}\}\.\}
The first term improves the policy with respect to its current information criterion\. The second reflects the fact that changing the policy also changes the criterion itself, through changes to the policy\-induced prior and/or likelihood\.

Implication\.Decoupling the experimental generative model from the policy removes this second source of dependence\. The policy is therefore optimized against a stationary information objective: training may change which states are visited and which designs are proposed, but it does not simultaneously change what constitutes an informative design\.