Expected Free Energy as Belief-Dependent Utility for rho-POMDPs

# Expected Free Energy as Belief-Dependent Utility for 𝜌-POMDPs
Source: [https://arxiv.org/html/2607.16981](https://arxiv.org/html/2607.16981)
11institutetext:Department of Computer Science, University of Colorado Boulder,
Boulder, CO, USA
11email:\{patrick\.cooper,alvaro\.velasquez\}@colorado\.edu###### Abstract

An agent acting under partial observability must decide when to gather information and which observations are worth their cost\. Standard POMDPs value information only through its eventual effect on reward\. Theρ\\rho\-POMDP framework instead rewards uncertainty reduction directly, through a belief\-dependent utilityρ\\rho, but in practice both the choice ofρ\\rhoand the weight placed on it are tuned by hand for every task\. We show that active inference removes this tuning entirely\. Minimizing Expected Free Energy \(EFE\) is exactly equivalent to solving aρ\\rho\-POMDP whose utility is expected information gain, and the exploration weight is fixed atw=1w\{=\}1because the variational bound expresses pragmatic and epistemic value in the same units \(nats\)\. We prove this equivalence for observe\-then\-commit POMDPs and extend it to factored observation POMDPs, a broader class that covers interleaved observe\-act problems such as non\-destructive testing and mobile sensing, where gathering information leaves the hidden state unchanged\. Experiments support the theory\. Across environments ranging from the classic Tiger problem to RockSample and a new Structural Inspection benchmark with over65,00065\{,\}000states, the untuned weight matches or outperforms reward\-only planning at the same horizon, avoids the over\-exploration of bonuses tuned per task, and sits near the reward\-maximizing knee of the success–reward Pareto frontier\. The practical payoff is an exploration objective that works out of the box\. In applications such as fault detection and medical screening, where every test has a price and every missed fault has a cost, EFE supplies a belief\-dependent utility that is derived rather than tuned\.

## 1Introduction

Decision\-making under partial observability requires agents to balance exploiting current knowledge against gathering information to reduce uncertainty about hidden states\. In standard POMDPs, information gathering has no intrinsic value\. It is useful only insofar as it leads to higher expected reward\. This creates a well\-known difficulty: the exploration–exploitation trade\-off must be resolved either by the planning horizon or by heuristic exploration bonuses\.

Theρ\\rho\-POMDP framework\[[1](https://arxiv.org/html/2607.16981#bib.bib1)\]addresses this by extending POMDPs with a belief\-dependent utilityρ​\(b\)\\rho\(b\)that allows the agent to derive value directly from properties of its belief state\. This enables explicit optimization over uncertainty reduction, information gain, or other belief\-state properties alongside task reward\. However, the choice ofρ\\rhoremains largely heuristic and environment\-specific\.

Separately, the Active Inference \(AIF\) framework\[[12](https://arxiv.org/html/2607.16981#bib.bib12),[30](https://arxiv.org/html/2607.16981#bib.bib30)\]casts perception and action as approximate Bayesian inference, selecting policies that minimize Expected Free Energy \(EFE\)\. EFE naturally decomposes into a pragmatic term \(goal\-seeking\) and an epistemic term \(information\-seeking\)\. Because both terms come from a single variational objective, their relative scale is not free to choose\. In the discrete\-state formulation we use, the coefficient on expected information gain is fixed atw=1w\{=\}1when both terms are measured in nats, the natural\-log unit of information \(Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)\)\. The weight is a consequence of the derivation, not a hyperparameter\.

We propose substituting EFE asρ\\rhoinρ\\rho\-POMDPs, yielding an agent whose epistemic foraging is a consequence of its objective rather than an engineered bonus\.111Code and experiment data are available at[https://github\.com/PatrickAllenCooper/rho\_aif](https://github.com/PatrickAllenCooper/rho_aif)\.Our contributions are:

1. 1\.Theory\.A formal bridge betweenρ\\rho\-POMDPs and active inference\. We prove the equivalence for observe\-then\-commit POMDPs \(Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)\), characterize when the canonical weight is near\-optimal \(Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2)\), and extend the equivalence to factored observation POMDPs, interleaved settings where information gathering preserves the hidden state \(Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)\)\.
2. 2\.Evidence\.Controlled comparisons against same\-horizon planning, tuned information gain, and POMCP\[[39](https://arxiv.org/html/2607.16981#bib.bib39)\]across six observe\-then\-commit environments and four instances of the standard RockSample benchmark\[[41](https://arxiv.org/html/2607.16981#bib.bib41)\]\. A Pareto analysis shows thatw=1w\{=\}1Pareto\-dominates same\-horizon planning without per\-environment search\.
3. 3\.Practical guidance\.A characterization of when EFE\-as\-ρ\\rhohelps\. The advantage appears when the agent must choose among multiple observation actions, and it grows with state space size \(66\.5%66\.5\\%vs\.2\.5%2\.5\\%success on Tileworld8×88\{\\times\}8\) and with the number of observation actions \(\+7\.82\+7\.82reward on RockSample\[7,8\]\)\.

## 2Related work

##### POMDPs and solvers\.

POMDPs formalize sequential decision\-making under state uncertainty\[[40](https://arxiv.org/html/2607.16981#bib.bib40),[22](https://arxiv.org/html/2607.16981#bib.bib22)\]\. Exact solutions are PSPACE\-complete\. Point\-based offline methods \(PBVI\[[33](https://arxiv.org/html/2607.16981#bib.bib33)\], HSVI\[[41](https://arxiv.org/html/2607.16981#bib.bib41)\], SARSOP\[[23](https://arxiv.org/html/2607.16981#bib.bib23)\]\) approximate the value function on reachable beliefs\[[38](https://arxiv.org/html/2607.16981#bib.bib38)\]\. Online solvers plan from the current belief: POMCP\[[39](https://arxiv.org/html/2607.16981#bib.bib39)\]uses MCTS with UCB1 and rollout evaluation, DESPOT\[[45](https://arxiv.org/html/2607.16981#bib.bib45)\]searches a regularized sparse belief tree, and POMCPOW\[[42](https://arxiv.org/html/2607.16981#bib.bib42)\]extends POMCP with progressive widening for continuous spaces\. All explore implicitly through stochastic simulations rather than explicitly valuing information gain\. Section[6](https://arxiv.org/html/2607.16981#S6)and Appendix[0\.T](https://arxiv.org/html/2607.16981#Pt0.A20)demonstrate the benefit of closed\-form information valuation\.

##### ρ\\rho\-POMDPs\.

Araya\-López et al\. \[[1](https://arxiv.org/html/2607.16981#bib.bib1)\]introducedρ\\rho\-POMDPs, augmenting the reward with a belief\-dependent utilityρ:Δ​\(S\)→ℝ\\rho:\\Delta\(S\)\\to\\mathbb\{R\}, so the objective becomesmaxπ⁡𝔼π​\[∑tγt​\(R​\(st,at\)\+ρ​\(bt\)\)\]\\max\_\{\\pi\}\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{t\}\\gamma^\{t\}\(R\(s\_\{t\},a\_\{t\}\)\+\\rho\(b\_\{t\}\)\)\]\. Whenρ\\rhois convex, the value function remains piecewise linear and convex \(PWLC\), preserving compatibility with standard solvers\.Fehr et al\. \[[10](https://arxiv.org/html/2607.16981#bib.bib10)\]extended this to Lipschitz\-continuous non\-convexρ\\rho, andBenchetrit et al\. \[[3](https://arxiv.org/html/2607.16981#bib.bib3)\]developedρ\\rho\-POMCPOW for continuous\-spaceρ\\rho\-POMDPs\. Common choices forρ\\rho\(entropy reduction, KL divergence from a target belief, and information gain\) each encode a different notion of epistemic value, but the choice remains heuristic and environment\-specific\. Our contribution is to derive a principledρ\\rhofrom the variational bound of active inference, fixing the exploration weight without per\-environment search\.

##### Value of information and experimental design\.

The idea that information has quantifiable decision\-theoretic value predates both POMDPs and active inference\.Howard \[[20](https://arxiv.org/html/2607.16981#bib.bib20)\]formalized the value of information in decision analysis, andLindley \[[25](https://arxiv.org/html/2607.16981#bib.bib25)\]introduced expected information gain as a criterion for optimal Bayesian experimental design\. In the bandit setting, information\-directed sampling \(IDS\)\[[35](https://arxiv.org/html/2607.16981#bib.bib35)\]explicitly trades off instantaneous regret against information gain by minimizing the information ratioΓt=δt2/gt\\Gamma\_\{t\}=\\delta\_\{t\}^\{2\}/g\_\{t\}, whereδt\\delta\_\{t\}is the expected regret andgtg\_\{t\}is the information gain\. The key structural difference from EFE is that IDS minimizes the ratio at each step \(a relative weighting that adapts to the current belief\), while EFE fixes the weight atw=1w\{=\}1\(an absolute weighting derived from the variational bound\)\. An IDS baseline adapted to the observe\-then\-commit structure would be informative but is beyond the scope of this work\. Our observe\-then\-commit structure parallels sequential Bayesian experimental design, where the agent selects experiments \(observations\) to maximize information about an unknown state before making a terminal decision\. Theρ\\rho\-POMDP framework withρ=I​\(b\)\\rho=I\(b\)operationalizes this connection\. Our contribution is to show that EFE derives a canonical weight for the information gain term from first principles rather than treating it as a tunable parameter\.

##### Active inference and EFE\.

The Active Inference \(AIF\) framework\[[12](https://arxiv.org/html/2607.16981#bib.bib12)\]casts perception and action as variational inference under the free\-energy principle\.Parr et al\. \[[31](https://arxiv.org/html/2607.16981#bib.bib31)\]provide a comprehensive textbook treatment\.Friston et al\. \[[13](https://arxiv.org/html/2607.16981#bib.bib13)\]formalized the decomposition of policy value into extrinsic \(goal\-seeking\) and epistemic \(information\-seeking\) components, showing that curiosity\-driven exploration arises automatically from expected free energy minimization\.Da Costa et al\. \[[6](https://arxiv.org/html/2607.16981#bib.bib6)\]synthesized discrete\-state AIF from first principles, showing that the posterior over policies takes the formQ​\(π\)∝exp⁡\(−𝒢​\(π\)\)Q\(\\pi\)\\propto\\exp\(\-\\mathcal\{G\}\(\\pi\)\)where𝒢\\mathcal\{G\}is the Expected Free Energy\.Parr and Friston \[[30](https://arxiv.org/html/2607.16981#bib.bib30)\]showed that EFE decomposes into pragmatic value \(divergence from preferred observations\) and epistemic value \(expected information gain\), with both arising from a single variational bound, requiring no tunable exploration weight\. Despite different constructions, EFE and Generalised Free Energy produce identical policy posteriors\. The mathematical foundations of EFE have been critically examined:Millidge et al\. \[[28](https://arxiv.org/html/2607.16981#bib.bib28)\]showed that naively extending the variational free energy into the future does not yield exploratory behavior, proposing the Free Energy of the Expected Future \(FEEF\) as an alternative with clearer mathematical grounding\.Champion et al\. \[[5](https://arxiv.org/html/2607.16981#bib.bib5)\]addressed the “unification problem” by formalizing how multiple EFE formulations relate to a single root definition under different assumptions about prior preferences\.de Vries et al\. \[[8](https://arxiv.org/html/2607.16981#bib.bib8)\]recast EFE\-based planning as entropy\-corrected variational inference with message\-passing schemes, providing an alternative derivation that connects EFE to variational message passing\.

##### Sophisticated inference\.

Standard AIF evaluates policies myopically\.Friston et al\. \[[14](https://arxiv.org/html/2607.16981#bib.bib14)\]introduced sophisticated inference, a recursive extension implementing deep tree search over belief trajectories rather than states\. Sophistication \(maintaining beliefs about future beliefs\) enables counterfactual reasoning about the downstream epistemic consequences of actions\.Da Costa et al\. \[[7](https://arxiv.org/html/2607.16981#bib.bib7)\]proved that this recursive scheme recovers Bellman\-optimal policies for any finite horizon, whereas standard AIF achieves optimality only for single\-step planning\. Our recursive EFE agent \(Equation[3](https://arxiv.org/html/2607.16981#S3.E3)\) is derived from this framework, adapted to theρ\\rho\-POMDP commit\-action structure\. The recursive counterfactual reasoning over future beliefs is preserved\. What changes is that our observe\-then\-commit setting does not involve state transitions between time steps, simplifying the belief\-trajectory computation\.

##### Scaling active inference\.

Discrete\-state AIF with full policy enumeration is limited to small state–action spaces\. Several lines of work address scaling\.Fountas et al\. \[[11](https://arxiv.org/html/2607.16981#bib.bib11)\]combined deep generative models with Monte Carlo tree search for EFE\-optimal planning in continuous state spaces\.Tschantz et al\. \[[44](https://arxiv.org/html/2607.16981#bib.bib44)\]developed an RL\-compatible objective \(the free energy of the expected future\) that inherits AIF’s exploration–exploitation balance while scaling to standard RL benchmarks\.Maisto et al\. \[[26](https://arxiv.org/html/2607.16981#bib.bib26)\]combined active inference with Monte\-Carlo tree search for large POMDPs, achieving state\-of\-the\-art on RockSample\[[41](https://arxiv.org/html/2607.16981#bib.bib41)\]\. Our MCTS\-EFE variant \(Section 6\) follows this direction, using EFE as a leaf heuristic within MCTS to extend planning horizons beyond exact tree search\.

##### Exploration, control as inference, and intrinsic motivation\.

The control\-as\-inference perspective\[[43](https://arxiv.org/html/2607.16981#bib.bib43),[24](https://arxiv.org/html/2607.16981#bib.bib24)\]casts reward maximization as variational inference\. Maximum\-entropy RL\[[17](https://arxiv.org/html/2607.16981#bib.bib17)\]and stochastic optimal control\[[34](https://arxiv.org/html/2607.16981#bib.bib34)\]are algorithmic instances\.Millidge et al\. \[[27](https://arxiv.org/html/2607.16981#bib.bib27)\]proved formal equivalence between AIF and control\-as\-inference\.Sajid et al\. \[[36](https://arxiv.org/html/2607.16981#bib.bib36)\]showed that intrinsic motivation behaviors arise under EFE\. Intrinsic motivation methods, including curiosity\[[37](https://arxiv.org/html/2607.16981#bib.bib37),[32](https://arxiv.org/html/2607.16981#bib.bib32)\], typologies of intrinsic signals\[[29](https://arxiv.org/html/2607.16981#bib.bib29)\], Bayesian surprise\[[21](https://arxiv.org/html/2607.16981#bib.bib21)\], count\-based exploration\[[2](https://arxiv.org/html/2607.16981#bib.bib2)\], VIME\[[19](https://arxiv.org/html/2607.16981#bib.bib19)\], and random network distillation\[[4](https://arxiv.org/html/2607.16981#bib.bib4)\], all require tunable bonus weights\. Bayes\-Adaptive MDPs\[[9](https://arxiv.org/html/2607.16981#bib.bib9),[16](https://arxiv.org/html/2607.16981#bib.bib16)\]and Bayesian RL more broadly\[[15](https://arxiv.org/html/2607.16981#bib.bib15)\]address model uncertainty \(unknown dynamics\), distinct from our focus on state uncertainty under a known model\. Ourρ\\rho\-POMDP formalism makes the connection between EFE and these lines of work precise, inheriting formal properties \(convexity, Lipschitz continuity\) while fixing the exploration weight from first principles\.

## 3Methodology

### 3\.1Theρ\\rho\-POMDP framework

We restrict attention to observe\-then\-commitρ\\rho\-POMDPs, in which the action set𝒜\\mathcal\{A\}partitions into observation actions𝒜obs\\mathcal\{A\}\_\{\\text\{obs\}\}\(which update the belief at known cost but do not change the hidden state\) and terminal commit actions𝒜com\\mathcal\{A\}\_\{\\text\{com\}\}\(which end the episode with state\-dependent reward\)\. An episode consists of a variable\-length sequence of observation actions followed by a single commit action\. The hidden state is fixed throughout\.

Aρ\\rho\-POMDP extends the standard POMDP with a belief\-dependent utilityρ:Δ​\(S\)→ℝ\\rho:\\Delta\(S\)\\rightarrow\\mathbb\{R\}\. The agent’s objective becomes:

π∗=arg⁡maxπ⁡𝔼π​\[∑t=0Hγt​\(R​\(st,at\)\+ρ​\(bt\)\)\]\\pi^\{\*\}=\\arg\\max\_\{\\pi\}\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{H\}\\gamma^\{t\}\\left\(R\(s\_\{t\},a\_\{t\}\)\+\\rho\(b\_\{t\}\)\\right\)\\right\]\(1\)Whenρ=0\\rho=0, we recover the standard POMDP\. Whenρ\\rhoencodes information gain, the agent is explicitly rewarded for reducing uncertainty\.

### 3\.2Expected Free Energy asρ\\rho

In the standard AIF formulation, the EFE for a policyπ\\piat future timeτ\\tauis:

𝒢​\(π\)=−𝔼Q​\(oτ\|π\)​\[ln⁡P​\(oτ\|C\)\]⏟Pragmatic value−𝔼Q​\(oτ\|π\)\[DKL\[Q\(sτ\|oτ,π\)∥Q\(sτ\|π\)\]\]⏟Epistemic value\\mathcal\{G\}\(\\pi\)=\\underbrace\{\-\\mathbb\{E\}\_\{Q\(o\_\{\\tau\}\|\\pi\)\}\[\\ln P\(o\_\{\\tau\}\|C\)\]\}\_\{\\text\{Pragmatic value\}\}\-\\underbrace\{\\mathbb\{E\}\_\{Q\(o\_\{\\tau\}\|\\pi\)\}\\left\[D\_\{\\mathrm\{KL\}\}\\left\[Q\(s\_\{\\tau\}\|o\_\{\\tau\},\\pi\)\\,\\\|\\,Q\(s\_\{\\tau\}\|\\pi\)\\right\]\\right\]\}\_\{\\text\{Epistemic value\}\}\(2\)whereP​\(oτ\|C\)P\(o\_\{\\tau\}\|C\)encodes preferred outcomes andDKLD\_\{\\mathrm\{KL\}\}measures expected information gain\. In our observe\-then\-commit setting, the pragmatic and epistemic terms take concrete forms\. For commit actions, the pragmatic term reduces to expected reward under the current belief:𝒢​\(commiti\)=−𝔼b​\[Ri\]\\mathcal\{G\}\(\\text\{commit\}\_\{i\}\)=\-\\mathbb\{E\}\_\{b\}\[R\_\{i\}\]\. We use the reward matrix directly rather than encoding rewards through preferred outcome distributionsP​\(o\|C\)P\(o\|C\), avoiding the preference\-calibration issue flagged as a source of hidden tuning in prior AIF work\. For observation actions, the pragmatic term is the known observation costckc\_\{k\}, and the epistemic term is the expected information gainIk​\(b\)=H​\(b\)−𝔼o​\[H​\(bo′∣obsk\)\]I\_\{k\}\(b\)=H\(b\)\-\\mathbb\{E\}\_\{o\}\[H\(b^\{\\prime\}\_\{o\}\\mid\\text\{obs\}\_\{k\}\)\]\.

Following the sophisticated inference scheme ofFriston et al\. \[[14](https://arxiv.org/html/2607.16981#bib.bib14)\], the EFE agent evaluates actions via a recursive tree search over belief states:

𝒢​\(observek\)=ck−Ik​\(b\)\+𝔼o​\[mina⁡𝒢​\(a∣bo′\)\]\\mathcal\{G\}\(\\text\{observe\}\_\{k\}\)=c\_\{k\}\-I\_\{k\}\(b\)\+\\mathbb\{E\}\_\{o\}\\\!\\left\[\\min\_\{a\}\\mathcal\{G\}\(a\\mid b^\{\\prime\}\_\{o\}\)\\right\]\(3\)The agent selectsarg⁡mina⁡𝒢​\(a\)\\arg\\min\_\{a\}\\mathcal\{G\}\(a\)\. Standard AIF introduces a precision parameterβ\\betavia a softmax policyQ​\(π\)∝exp⁡\(−β​𝒢​\(π\)\)Q\(\\pi\)\\propto\\exp\(\-\\beta\\mathcal\{G\}\(\\pi\)\)\. Our deterministicarg⁡min\\arg\\mincorresponds toβ→∞\\beta\\to\\infty, which eliminates this tunable knob\. Equation[3](https://arxiv.org/html/2607.16981#S3.E3)contains no separate exploration weight, and this is not a modeling shortcut\. When EFE is written in nats, expected information gain enters on the same footing as the KL terms that define the variational objective\. This shared scale is why theρ\\rho\-POMDP reduction carriesw=1w\{=\}1exactly \(Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)\)\.

### 3\.3Formal equivalence withρ\\rho\-POMDPs

We generalize the standardρ\\rho\-POMDP formulation to action\-dependent belief utilitiesρ​\(b,a\)\\rho\(b,a\), where the augmented reward becomesR​\(s,a\)\+ρ​\(b,a\)R\(s,a\)\+\\rho\(b,a\)\. This is equivalent to a standardρ​\(b\)\\rho\(b\)formulation on an augmented belief\-action space but avoids notational overhead\. The structural results ofAraya\-López et al\. \[[1](https://arxiv.org/html/2607.16981#bib.bib1)\]carry over whenρ​\(⋅,a\)\\rho\(\\cdot,a\)satisfies the relevant conditions for each fixedaa\.

DefineV​\(a,b,d\)≜−𝒢​\(a,b,d\)V\(a,b,d\)\\triangleq\-\\mathcal\{G\}\(a,b,d\)\. Then Equation[3](https://arxiv.org/html/2607.16981#S3.E3)becomes:

V​\(observek,b,d\)\\displaystyle V\(\\text\{observe\}\_\{k\},b,d\)=−ck\+Ik​\(b\)\+𝔼o​\[maxa⁡V​\(a,bo′,d\+1\)\]\\displaystyle=\-c\_\{k\}\+I\_\{k\}\(b\)\+\\mathbb\{E\}\_\{o\}\\\!\\left\[\\max\_\{a\}V\(a,b^\{\\prime\}\_\{o\},d\{\+\}1\)\\right\]\(4\)V​\(commiti,b,d\)\\displaystyle V\(\\text\{commit\}\_\{i\},b,d\)=𝔼b​\[Ri\]\\displaystyle=\\mathbb\{E\}\_\{b\}\[R\_\{i\}\]\(5\)This is exactly the Bellman recursion for aρ\\rho\-POMDP \(Equation 1\) with action\-dependent belief utility:

###### Proposition 1

DefineρEFE​\(b,a\)\\rho\_\{\\mathrm\{EFE\}\}\(b,a\)as:

ρEFE​\(b,a\)=\{Ia​\(b\)if​a​is an observation action0if​a​is a commit action\\rho\_\{\\mathrm\{EFE\}\}\(b,a\)=\\begin\{cases\}I\_\{a\}\(b\)&\\text\{if \}a\\text\{ is an observation action\}\\\\ 0&\\text\{if \}a\\text\{ is a commit action\}\\end\{cases\}whereIa​\(b\)=H​\(b\)−𝔼o\|a​\[H​\(bo′\)\]I\_\{a\}\(b\)=H\(b\)\-\\mathbb\{E\}\_\{o\|a\}\[H\(b^\{\\prime\}\_\{o\}\)\]is the expected information gain from observation actionaaat beliefbb\. Then for undiscounted finite horizon \(γ=1\\gamma\{=\}1\), minimizing recursive EFE \(Eq\.[3](https://arxiv.org/html/2607.16981#S3.E3)\) over horizonHHproduces the same policy as solving theρ\\rho\-POMDP Bellman equationV∗​\(b\)=maxa⁡\{R​\(b,a\)\+ρEFE​\(b,a\)\+𝔼o​\[V∗​\(bo′\)\]\}V^\{\*\}\(b\)=\\max\_\{a\}\\\{R\(b,a\)\+\\rho\_\{\\mathrm\{EFE\}\}\(b,a\)\+\\mathbb\{E\}\_\{o\}\[V^\{\*\}\(b^\{\\prime\}\_\{o\}\)\]\\\}over the same horizon\.

###### Proof\(Proof sketch\)

The negationV=−𝒢V=\-\\mathcal\{G\}convertsarg⁡min⁡𝒢\\arg\\min\\mathcal\{G\}toarg⁡max⁡V\\arg\\max V\. Substituting into Equation[3](https://arxiv.org/html/2607.16981#S3.E3): for observation actions,V=−ck\+Ik​\(b\)\+𝔼o​\[maxa′⁡V​\(a′,bo′\)\]V=\-c\_\{k\}\+I\_\{k\}\(b\)\+\\mathbb\{E\}\_\{o\}\[\\max\_\{a^\{\\prime\}\}V\(a^\{\\prime\},b^\{\\prime\}\_\{o\}\)\], which matches theρ\\rho\-POMDP Bellman backup withR​\(b,obsk\)=−ckR\(b,\\text\{obs\}\_\{k\}\)=\-c\_\{k\}andρ=Ik​\(b\)\\rho=I\_\{k\}\(b\)\. For commit actions,V=𝔼b​\[Ri\]V=\\mathbb\{E\}\_\{b\}\[R\_\{i\}\]withρ=0\\rho=0, matching a terminalρ\\rho\-POMDP action\. The recursive structure is identical, so the policies agree at every belief node\.

Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)makes precise what EFE\-as\-ρ\\rhomeans: the epistemic term of EFE functions as an action\-dependent belief utility\. Crucially, this is equivalent to Planning\+IG withw=1w\{=\}1\(in nats\)\. EFE does not eliminate the weight\. It derives a canonical weight from the variational bound, fixingw=1w\{=\}1without per\-environment search\. Whether this canonical choice is near\-optimal is an empirical question we address in Section[5\.2](https://arxiv.org/html/2607.16981#S5.SS2)\. The following result characterizes the conditions under whichw=1w\{=\}1is near\-optimal for expected reward\.

###### Proposition 2

Consider a two\-state observe\-then\-commitρ\\rho\-POMDP with uniform priorb​\(s0\)=b​\(s1\)=12b\(s\_\{0\}\)\{=\}b\(s\_\{1\}\)\{=\}\\tfrac\{1\}\{2\}, a single observation action \(accuracyp\>12p\>\\tfrac\{1\}\{2\}, costc\>0c\>0\), and two commit actions \(correct rewardR\+R^\{\+\}, incorrect penaltyR−<0R^\{\-\}<0with\|R−\|\>R\+\|R^\{\-\}\|\>R^\{\+\}\)\. Define the reward asymmetry ratioα=\|R−\|/R\+\\alpha=\|R^\{\-\}\|/R^\{\+\}and the informativeness ratioη=Imax/c\\eta=I\_\{\\max\}/cwhereImax=ln⁡2−Hpost​\(p\)I\_\{\\max\}=\\ln 2\-H\_\{\\text\{post\}\}\(p\)is the maximum expected information gain in nats\. AtH=1H\{=\}1, the minimum weightwthresh∗w^\{\*\}\_\{\\mathrm\{thresh\}\}at which observing yields higher expected reward than committing immediately is:

wthresh∗=c−\(p−12\)​\(R\+−R−\)Imax=c−\(p−12\)​\(1\+α\)​R\+Imaxw^\{\*\}\_\{\\mathrm\{thresh\}\}=\\frac\{c\\;\-\\;\\left\(p\-\\tfrac\{1\}\{2\}\\right\)\(R^\{\+\}\-R^\{\-\}\)\}\{\\,I\_\{\\max\}\\,\}=\\frac\{c\\;\-\\;\\left\(p\-\\tfrac\{1\}\{2\}\\right\)\(1\+\\alpha\)\\,R^\{\+\}\}\{\\,I\_\{\\max\}\\,\}For anyw\>wthresh∗w\>w^\{\*\}\_\{\\mathrm\{thresh\}\}the agent observes before committing, yielding identical \(and higher\) expected reward\. The thresholdwthresh∗w^\{\*\}\_\{\\mathrm\{thresh\}\}is negative, makingw=1w\{=\}1trivially sufficient, wheneverα\>c/\[\(p−12\)​R\+\]−1\\alpha\>c/\[\(p\-\\tfrac\{1\}\{2\}\)\\,R^\{\+\}\]\-1\. In particular,wthresh∗→−∞w^\{\*\}\_\{\\mathrm\{thresh\}\}\\to\-\\inftyasα→∞\\alpha\\to\\inftyfor any fixedp\>12p\>\\tfrac\{1\}\{2\}andc\>0c\>0: high reward asymmetry makes observation so valuable that any positive weight suffices\.

###### Proof\(Proof sketch\)

AtH=1H\{=\}1, the agent observes once and commits\. A Planning\+IG agent with weightwwobserves iff−c\+w⋅I​\(b\)\+maxi⁡𝔼bo′​\[Ri\]\>maxi⁡𝔼b​\[Ri\]\-c\+w\\cdot I\(b\)\+\\max\_\{i\}\\mathbb\{E\}\_\{b^\{\\prime\}\_\{o\}\}\[R\_\{i\}\]\>\\max\_\{i\}\\mathbb\{E\}\_\{b\}\[R\_\{i\}\], where the left side is the observe\-then\-commit value and the right side is the immediate commit value\. For the uniform prior, the immediate commit value is\(R\+\+R−\)/2\(R^\{\+\}\+R^\{\-\}\)/2\. After one observation, the posterior concentrates: with probabilityppthe agent is correct, yielding expected commit valuep⋅R\+\+\(1−p\)⋅R−p\\cdot R^\{\+\}\+\(1\-p\)\\cdot R^\{\-\}\. The net gain from observing is\(p−12\)​\(R\+−R−\)−c\+w⋅Imax\(p\-\\frac\{1\}\{2\}\)\(R^\{\+\}\-R^\{\-\}\)\-c\+w\\cdot I\_\{\\max\}\. Setting this to zero giveswthresh∗=\[c−\(p−12\)​\(R\+−R−\)\]/Imaxw^\{\*\}\_\{\\mathrm\{thresh\}\}=\[c\-\(p\-\\frac\{1\}\{2\}\)\(R^\{\+\}\-R^\{\-\}\)\]/I\_\{\\max\}\. SubstitutingR−=−α​R\+R^\{\-\}=\-\\alpha R^\{\+\}yields the second form\. Whenα≫1\\alpha\\gg 1, the marginal reward improvement\(p−12\)​\(1\+α\)​R\+\(p\-\\frac\{1\}\{2\}\)\(1\+\\alpha\)R^\{\+\}dominates the costcc, makingwthresh∗w^\{\*\}\_\{\\mathrm\{thresh\}\}negative, so the agent should observe at anyw≥0w\\geq 0, includingw=1w\{=\}1\.

Table[1](https://arxiv.org/html/2607.16981#S3.T1)validates Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2)across our environments, computingα\\alpha,η\\eta, the thresholdwthresh∗w^\{\*\}\_\{\\text\{thresh\}\}, and comparing against the observed reward\-maximizing weight from the Pareto sweep \(Section[5\.2](https://arxiv.org/html/2607.16981#S5.SS2)\)\.

Table 1:Reward asymmetry \(α\\alpha\), informativeness \(η\\eta\), observation thresholdwthresh∗w^\{\*\}\_\{\\text\{thresh\}\}from Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2), and observed reward\-maximizing weight from the Pareto sweep\. Whenwthresh∗<0w^\{\*\}\_\{\\text\{thresh\}\}<0, any positive weight \(includingw=1w\{=\}1\) induces observation\.The proposition is stated forH=1H\{=\}1and two states\. The multi\-step case is harder to analyze because the observation threshold shifts with belief, but the qualitative prediction holds: environments whereα≥5\\alpha\\geq 5\(Tiger, Diagnosis, Tileworld\) havewthresh∗≪0w^\{\*\}\_\{\\text\{thresh\}\}\\ll 0, meaningw=1w\{=\}1is far above the threshold and near\-optimal\. The low\-asymmetry Testbed \(α=1\\alpha\{=\}1\) haswthresh∗\>0w^\{\*\}\_\{\\text\{thresh\}\}\>0but below 1, sow=1w\{=\}1still induces observation but assigns more weight to information than is instrumentally optimal, consistent with our finding that EFE over\-explores there \(Appendix[0\.C](https://arxiv.org/html/2607.16981#Pt0.A3)\)\. On the Bandit \(α=1\.1\\alpha\{=\}1\.1\),wthresh∗w^\{\*\}\_\{\\text\{thresh\}\}is slightly negative atH=1H\{=\}1\. AtH\>1H\{\>\}1, multi\-step planning amplifies the value of each observation, bringing the effective reward\-maximizing weight toward 1\.

##### Extension to discounting\.

Withγ<1\\gamma<1, the recursive EFE becomes𝒢​\(obsk\)=ck−Ik​\(b\)\+γ​𝔼o​\[mina⁡𝒢​\(a,bo′\)\]\\mathcal\{G\}\(\\text\{obs\}\_\{k\}\)=c\_\{k\}\-I\_\{k\}\(b\)\+\\gamma\\,\\mathbb\{E\}\_\{o\}\[\\min\_\{a\}\\mathcal\{G\}\(a,b^\{\\prime\}\_\{o\}\)\], givingV​\(obsk\)=−ck\+Ik​\(b\)\+γ​𝔼o​\[maxa⁡V​\(a,bo′\)\]V\(\\text\{obs\}\_\{k\}\)=\-c\_\{k\}\+I\_\{k\}\(b\)\+\\gamma\\,\\mathbb\{E\}\_\{o\}\[\\max\_\{a\}V\(a,b^\{\\prime\}\_\{o\}\)\]\. The information gain termIk​\(b\)I\_\{k\}\(b\)appears undiscounted at the current step, while future values are discounted\. This preserves theρ\\rho\-POMDP equivalence withρ​\(b,a\)=Ia​\(b\)\\rho\(b,a\)=I\_\{a\}\(b\), but the effective ratio of epistemic to pragmatic weight increases at early steps relative to late steps\. In the undiscounted case \(γ=1\\gamma\{=\}1\), both terms are weighted equally at every depth\. Withγ<1\\gamma<1, the agent places relatively more value on immediate information gain compared to future reward, producing slightly more exploratory behavior at early steps\. Our experiments \(Appendix[0\.Q](https://arxiv.org/html/2607.16981#Pt0.A17)\) confirm that performance is robust acrossγ∈\{0\.9,0\.95,0\.99,1\.0\}\\gamma\\in\\\{0\.9,0\.95,0\.99,1\.0\\\}\.

##### Extension to factored observation POMDPs\.

Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)assumes the observe\-then\-commit structure\. We now extend the equivalence to a broader class that includes interleaved observe\-act POMDPs\.

###### Definition 1\(Factored observation POMDP\)

A POMDP is a factored observation POMDP if its state decomposes ass=\(svis,shid\)s=\(s\_\{\\mathrm\{vis\}\},s\_\{\\mathrm\{hid\}\}\)wheresviss\_\{\\mathrm\{vis\}\}is fully observable andshids\_\{\\mathrm\{hid\}\}is hidden, and the action set partitions into: \(i\) observation actions𝒜obs\\mathcal\{A\}\_\{\\mathrm\{obs\}\}that produce observations aboutshids\_\{\\mathrm\{hid\}\}without changingshids\_\{\\mathrm\{hid\}\}\(though they may changesviss\_\{\\mathrm\{vis\}\}\), \(ii\) navigation actions𝒜nav\\mathcal\{A\}\_\{\\mathrm\{nav\}\}that changesviss\_\{\\mathrm\{vis\}\}deterministically without changingshids\_\{\\mathrm\{hid\}\}and produce no informative observation, and \(iii\) exploitation actions𝒜exp\\mathcal\{A\}\_\{\\mathrm\{exp\}\}that yield reward dependent onshids\_\{\\mathrm\{hid\}\}\.

Observe\-then\-commit POMDPs are the special case where𝒜nav=∅\\mathcal\{A\}\_\{\\mathrm\{nav\}\}=\\emptysetandsviss\_\{\\mathrm\{vis\}\}is trivial\. RockSample\[[41](https://arxiv.org/html/2607.16981#bib.bib41)\]is an instance:sviss\_\{\\mathrm\{vis\}\}is the agent’s grid position \(fully observable\),shids\_\{\\mathrm\{hid\}\}is the vector of rock qualities \(hidden, fixed\), check actions are observation actions, moves are navigation actions, and sample/exit are exploitation actions\.

###### Proposition 3

In a factored observation POMDP \(Definition[1](https://arxiv.org/html/2607.16981#Thmdefinition1)\), letbbdenote the belief overshids\_\{\\mathrm\{hid\}\}\. For any actionaathat preservesshids\_\{\\mathrm\{hid\}\}\(i\.e\.,a∈𝒜obs∪𝒜nava\\in\\mathcal\{A\}\_\{\\mathrm\{obs\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{nav\}\}\), the transition–observation coupling term vanishes:ΔT​\(b,a\)=0\\Delta\_\{T\}\(b,a\)=0, andρEFE​\(b,a\)=Ia​\(b\)\\rho\_\{\\mathrm\{EFE\}\}\(b,a\)=I\_\{a\}\(b\)for observation actions,ρEFE​\(b,a\)=0\\rho\_\{\\mathrm\{EFE\}\}\(b,a\)=0for navigation actions\.

###### Proof\(Proof sketch\)

When actionaapreservesshids\_\{\\mathrm\{hid\}\}, the transition on the hidden component isThid​\(shid′\|shid,a\)=δ​\(shid′=shid\)T\_\{\\mathrm\{hid\}\}\(s^\{\\prime\}\_\{\\mathrm\{hid\}\}\|s\_\{\\mathrm\{hid\}\},a\)=\\delta\(s^\{\\prime\}\_\{\\mathrm\{hid\}\}=s\_\{\\mathrm\{hid\}\}\)\. The belief update overshids\_\{\\mathrm\{hid\}\}is thenb′​\(shid\)∝P​\(o\|shid,svis′,a\)​b​\(shid\)b^\{\\prime\}\(s\_\{\\mathrm\{hid\}\}\)\\propto P\(o\|s\_\{\\mathrm\{hid\}\},s^\{\\prime\}\_\{\\mathrm\{vis\}\},a\)\\,b\(s\_\{\\mathrm\{hid\}\}\), depending only on the observation likelihood, identical to the observe\-then\-commit case\. The posterior that would incorporate transitions,bo,T′b^\{\\prime\}\_\{o,T\}, coincides with the observation\-only posteriorbo′b^\{\\prime\}\_\{o\}, soDKL​\[bo,T′∥bo′\]=0D\_\{\\mathrm\{KL\}\}\[b^\{\\prime\}\_\{o,T\}\\\|b^\{\\prime\}\_\{o\}\]=0\. For observation actions, EFE reduces to cost minus information gain plus expected continuation, matching theρ\\rho\-POMDP Bellman equation withρ=Ia​\(b\)\\rho=I\_\{a\}\(b\)\. For navigation actions, the observation is uninformative \(Ia​\(b\)=0I\_\{a\}\(b\)=0\), givingρ=0\\rho=0\.

Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)extends the formal bridge from observe\-then\-commit to any POMDP where the hidden state is preserved by information\-gathering and navigation actions\. The agent interleaves observation, navigation, and exploitation\. At each decision point, the EFE\-as\-ρ\\rhoequivalence holds for the observation and navigation subtree\. This covers RockSample, mobile sensor placement, and sequential testing with spatial access costs, where the agent must navigate to observation locations before gathering information\. Section[5\.4](https://arxiv.org/html/2607.16981#S5.SS4)validates this extension empirically across four RockSample instances\.

The factored observation structure is common in practice whenever the quantity being measured is static or slow\-changing relative to the decision horizon \(Table[2](https://arxiv.org/html/2607.16981#S3.T2)\)\. The key requirement, thatThid​\(shid′\|shid,a\)=δ​\(shid′=shid\)T\_\{\\mathrm\{hid\}\}\(s^\{\\prime\}\_\{\\mathrm\{hid\}\}\|s\_\{\\mathrm\{hid\}\},a\)=\\delta\(s^\{\\prime\}\_\{\\mathrm\{hid\}\}=s\_\{\\mathrm\{hid\}\}\)for observation and navigation actions, breaks when information\-gathering itself alters the hidden state\. In such settings, the coupling termΔT≠0\\Delta\_\{T\}\\neq 0and the canonical\-weight equivalence does not hold\. We discuss this further in the limitations paragraph of Section[6](https://arxiv.org/html/2607.16981#S6)\.

Table 2:Taxonomy of real\-world POMDPs by factored observation structure\. Factored settings preserve the hidden state under observation and navigation actions, while non\-factored settings do not\.

### 3\.4Agents

We compare six agents \(Table[3](https://arxiv.org/html/2607.16981#S3.T3)\), all sharing the same belief\-update machinery and differing only in objective function and planning depth\. The Planning and Planning\+IG baselines use the same recursive tree search as the EFE agent, isolating the effect of theρ\\rhofunction from planning depth\.

Table 3:Agent specifications\. All use exact Bayesian belief updates over a known generative model\.Planning\+IG is the critical baseline: it uses the same tree search as the EFE agent with an additive IG bonus at the same horizon\. By Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1), the EFE agent is exactly Planning\+IG withw=1w\{=\}1, so any advantage is attributable to the weight choice rather than a different mechanism\. The Epistemic\-only agent sets𝒢​\(commit\)=0\\mathcal\{G\}\(\\text\{commit\}\)=0, removing reward awareness\. It commits at chance on all environments, which confirms that the pragmatic term is essential\. EFE computation is validated againstpymdp\[[18](https://arxiv.org/html/2607.16981#bib.bib18)\]\(Appendix[0\.K](https://arxiv.org/html/2607.16981#Pt0.A11)\)\.

For Info Gain and Planning\+IG,wwis tuned per environment via grid search over\{0\.1,0\.5,1,2,5,10,20,50,100\}\\\{0\.1,0\.5,1,2,5,10,20,50,100\\\}on 200 tuning episodes\. The Pareto analysis \(Section[5\.2](https://arxiv.org/html/2607.16981#S5.SS2)\) sweeps the full weight space\.

## 4Experiments

All environments are implemented as OpenAI Gymnasium environments following the observe\-then\-commit structure of Section[3](https://arxiv.org/html/2607.16981#S3)\. Main results use 1,000 episodes per seed across 5 random seeds \(\{42,123,456,789,1024\}\\\{42,123,456,789,1024\\\}\) for a total of 5,000 episodes\. Results report the mean across all episodes\. Statistical comparisons usett\-tests with Holm–Bonferroni correction, with bootstrap CIs in Appendix[0\.O](https://arxiv.org/html/2607.16981#Pt0.A15)\. Comparison against POMCP \(including compute\-matched analysis with wall\-clock timing at budgets 500–5,000 simulations\) is in Appendix[0\.T](https://arxiv.org/html/2607.16981#Pt0.A20)\. Full specifications are in Appendix[0\.H](https://arxiv.org/html/2607.16981#Pt0.A8)\.

##### Tiger

\[[22](https://arxiv.org/html/2607.16981#bib.bib22)\]\. Two states, one observation action \(listen, accuracy 0\.85\), two commit actions\. Rewards: correct\+10\+10, incorrect−100\-100, listen cost−1\-1\.

##### Sequential diagnosis\.

N=4N\{=\}4conditions,K=2K\{=\}2binary tests \(accuracy 0\.80\),NNdiagnose actions\. Correct\+10\+10, incorrect−50\-50, test cost−1\-1\. The agent must choose which test to run\.

##### Structured bandit\.

K=4K\{=\}4arms,KKinspect actions \(accuracy 0\.80, cost−0\.5\-0\.5\),KKpull actions\. Best arm\+10\+10, others\+1\+1\. The agent must choose which arm to inspect\.

##### Tileworld\.

AnN×NN\{\\times\}Ngrid \(N=6N\{=\}6,\|S\|=36\|S\|\{=\}36\) with a hidden target tile\.K=6K\{=\}6scan actions partition the grid via bit\-level splits of row/column indices, each returning a noisy binary signal \(accuracy0\.800\.80, cost−1\-1\)\.N2N^\{2\}commit actions collect at a specific cell \(correct\+10\+10, incorrect−50\-50\)\. A spatial generalization of diagnosis that produces visually interpretable belief evolution \(Figure[2](https://arxiv.org/html/2607.16981#S5.F2)\)\.

##### RockSample

\[[41](https://arxiv.org/html/2607.16981#bib.bib41)\]\. AnN×NN\{\\times\}Ngrid withKKrocks at known positions, each with hidden binary quality\. Move actions change the agent’s position\. Check actions produce distance\-dependent noisy observations of rock quality\. Sampling collects the rock at the current position \(good\+10\+10, bad−10\-10\), and exiting gives\+10\+10\. All actions cost−0\.5\-0\.5\. We evaluate RS\[5,3\], RS\[7,4\], RS\[7,8\], and RS\[11,11\]\. Unlike the above environments, RockSample has interleaved observe\-act dynamics with state transitions, the setting addressed by Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)\.

##### Structural inspection\.

NNcomponents at known spatial locations on a grid, each with a hidden binary state \(nominal/faulty, priorpfault=0\.3p\_\{\\text\{fault\}\}\{=\}0\.3\)\. There areK=2K\{=\}2non\-destructive test types: visual \(accuracy0\.700\.70, cost−0\.5\-0\.5\) and detailed \(accuracy0\.900\.90, cost−2\-2\)\. The agent navigates between components, runs tests, and declares a diagnosis for each\. Correct nominal\+2\+2, correct fault\+5\+5, missed fault−50\-50, false alarm−5\-5, move cost−0\.5\-0\.5\. This is a factored observation POMDP \(tests do not change fault states\), mapping directly to industrial inspection, medical screening, and fault detection domains\. We evaluateN=8N\{=\}8\(\|S\|=256\|S\|\{=\}256\) andN=16N\{=\}16\(\|S\|=65,536\|S\|\{=\}65\{,\}536\)\.

Two additional environments, a two\-state testbed \(Appendix[0\.C](https://arxiv.org/html/2607.16981#Pt0.A3)\) and navigation \(Appendix[0\.I](https://arxiv.org/html/2607.16981#Pt0.A9)\), delimit the EFE agent’s applicability on mild\-penalty and small\-state\-space settings\.

## 5Results

### 5\.1Core environments

Table 4:Results across three core environments \(5,000 total episodes: 1,000 per seed×\\times5 seeds\)\.w∗w^\{\*\}: per\-environment tuned weight\. Reward shown as mean±\\pmSE\. Full agent set in Appendix[0\.B](https://arxiv.org/html/2607.16981#Pt0.A2)\. Effect sizes on reward \(Cohen’sdd\) are in Appendix[0\.O](https://arxiv.org/html/2607.16981#Pt0.A15)\.On Tiger \(single observation action\), all multi\-step agents achieve roughly99\.5%99\.5\\%success, and EFE matches tuned alternatives without weight selection\. The Epistemic\-only ablation commits at chance \(50\.1%50\.1\\%on Tiger,25\.1%25\.1\\%on Bandit,0\.0%0\.0\\%on Tileworld6×66\{\\times\}6, Appendix[0\.B](https://arxiv.org/html/2607.16981#Pt0.A2)\)\. Pure information gain without reward alignment produces catastrophic exploration, so the pragmatic term is essential\.

On the multi\-observation\-action environments, EFE Pareto\-dominates Planning\. It achieves substantially higher success and comparable or better reward at the same time, without any tuning\. On Diagnosis, EFE outperforms same\-horizon Planning in success rate \(97\.1%97\.1\\%vs\.89\.2%89\.2\\%,\+7\.9\+7\.9pp\) while achieving substantially better reward \(−1\.50\-1\.50vs\.−2\.37\-2\.37\)\. On Bandit, EFE achieves both higher success \(87\.3%87\.3\\%vs\.69\.6%69\.6\\%,\+17\.7\+17\.7pp\) and the highest reward \(\+6\.27\+6\.27vs\.\+5\.65\+5\.65\)\. Bootstrap 95% CIs \(10,000 resamples over 5,000 episodes\) confirm non\-overlapping reward intervals: on Bandit, EFE\+6\.38\+6\.38\[\+6\.29,\+6\.47\]\[\+6\.29,\+6\.47\]vs\. Planning\+5\.65\+5\.65\[\+5\.54,\+5\.76\]\[\+5\.54,\+5\.76\], and on Diagnosis, EFE−1\.58\-1\.58\[−1\.89,−1\.29\]\[\-1\.89,\-1\.29\]vs\. Planning−2\.72\-2\.72\[−3\.26,−2\.19\]\[\-3\.26,\-2\.19\]\. Planning\+IG at tuned weights \(w=100w\{=\}100\) reaches near\-ceiling success \(99\.3%99\.3\\%on Diagnosis,99\.8%99\.8\\%on Bandit\) but at substantial reward cost due to over\-exploration: 13\.21 tests on Diagnosis \(−3\.63\-3\.63reward\) and 12\.41 inspections on Bandit \(\+3\.78\+3\.78reward\)\. Viewed in the success–reward plane \(Figure[1](https://arxiv.org/html/2607.16981#S5.F1)\), the EFE agent sits at the Pareto knee on every environment, while Planning sacrifices success rate and Planning\+IG sacrifices reward\. The pattern is consistent: EFE’s joint pragmatic–epistemic objective knows when to stop exploring, while additive IG bonuses with tuned weights do not\.

Effect sizes clarify where the differences lie\. Cohen’sddon reward between EFE and same\-horizon Planning is negligible on Tiger, Diagnosis, and Bandit \(\|d\|<0\.2\|d\|<0\.2, Table[19](https://arxiv.org/html/2607.16981#Pt0.A15.T19)\), because both agents already achieve reasonable returns once they explore enough\. The key gap is whether they explore the right observations\. On success rate, where that gap appears,ddis substantially larger:d≈0\.33d\\approx 0\.33on Diagnosis andd≈0\.47d\\approx 0\.47on Bandit \(Appendix[0\.O](https://arxiv.org/html/2607.16981#Pt0.A15)\), in the small\-to\-medium range\. This two\-axis pattern is exactly the Pareto story: EFE moves the under\-served objective \(success\) without sacrificing reward\. Medium\-to\-large rewardddvalues \(d\>0\.7d\>0\.7\) emerge against over\-exploring baselines and on environments where Planning fails to explore sufficiently \(Tileworldd\>2\.0d\>2\.0\)\.

### 5\.2Pareto analysis: the canonicalw=1w\{=\}1

By Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1), EFE is exactly Planning\+IG withw=1w\{=\}1\. We sweepwwfrom0\.010\.01to200200on all environments \(Figure[1](https://arxiv.org/html/2607.16981#S5.F1)\)\.

![Refer to caption](https://arxiv.org/html/2607.16981v1/x1.png)Figure 1:Pareto analysis: success rate vs\. mean reward aswwvaries from 0\.01 to 200\. Diamond:w=1w\{=\}1\(canonical EFE weight\)\. Star: EFE agent\. On all environments,w=1w\{=\}1sits at or near the Pareto knee, the inflection point where further weight increases buy marginal accuracy at substantial reward cost\.On every environment,w=1w\{=\}1sits near the reward\-maximizing weightwret∗w^\{\*\}\_\{\\text\{ret\}\}, while the success\-maximizing weightwsucc∗w^\{\*\}\_\{\\text\{succ\}\}lies at2020–200200\. We emphasize thatw=1w\{=\}1is near\-optimal for reward, not for success rate: agents requiring near\-certain accuracy \(e\.g\., safety\-critical applications\) would benefit from higher weights at the cost of reduced reward\. The contribution of EFE is deriving a principled weight from the variational bound that is near\-optimal for reward maximization, rather than a grid search whose optimum changes by orders of magnitude across tasks\.

### 5\.3Tileworld: spatial epistemic foraging

To test whether EFE’s advantage extends to larger state spaces, we introduce a spatial generalization: the Tileworld projects the Diagnosis partition structure onto anN×NN\{\\times\}Ngrid \(\|S\|=N2\|S\|\{=\}N^\{2\}\), producing spatially interpretable belief evolution \(Figure[2](https://arxiv.org/html/2607.16981#S5.F2), with step\-by\-step belief strips in Appendix[0\.F](https://arxiv.org/html/2607.16981#Pt0.A6)\)\.

Table 5:Tileworld6×66\{\\times\}6\(2,500 episodes: 500 per seed×\\times5 seeds,H=2H\{=\}2\)\. Tuned weightw∗=100w^\{\*\}\{=\}100\.EFE achieves the highest reward \(−21\.13\-21\.13\), scanning 14\.81 times, less than half of Planning\+IG’s 33\.38 scans, which erode reward despite reaching 98\.4% success\. The same over\-exploration pattern from Diagnosis and Bandit recurs at this larger scale\. Figure[2](https://arxiv.org/html/2607.16981#S5.F2)visualizes the mechanism: EFE concentrates belief efficiently via partition\-based narrowing and commits once confident, while Planning lacks scan\-selection guidance and Info Gain continues scanning past the point of diminishing returns\.

![Refer to caption](https://arxiv.org/html/2607.16981v1/x2.png)Figure 2:Agent comparison on the same6×66\{\\times\}6Tileworld episode\. EFE \(top\) commits correctly with efficient scanning\. Planning \(middle\) explores with less direction before committing\. Info Gain \(bottom\) over\-scans well past the point of diminishing returns\. Red circle: committed cell\. Green star: target\.##### Spatial scaling\.

We scale the grid from4×44\{\\times\}4to8×88\{\\times\}8with all agents \(Figure[3](https://arxiv.org/html/2607.16981#S5.F3)\)\. At8×88\{\\times\}8\(\|S\|=64\|S\|\{=\}64\), reward\-only Planning collapses to2\.5%2\.5\\%success while EFE maintains66\.5%66\.5\\%\. Planning\+IG atw=100w\{=\}100achieves98\.0%98\.0\\%success but at substantial reward cost due to extensive scanning\. This reveals a nuanced picture: EFE \(w=1w\{=\}1\) achieves the best reward at every scale and matches or exceeds Planning in success, but tuned Planning\+IG \(w=100w\{=\}100\) achieves higher success rates at larger grids by exploring more\. The reward\-maximizing weightwret∗w^\{\*\}\_\{\\text\{ret\}\}does not shift with\|S\|\|S\|\(EFE’s reward remains highest\), but its gap to the success\-maximizing weightwsucc∗w^\{\*\}\_\{\\text\{succ\}\}widens\. Safety\-critical applications at large scale would therefore benefit from higher weights\.

![Refer to caption](https://arxiv.org/html/2607.16981v1/x3.png)Figure 3:Tileworld scaling \(H=2H\{=\}2, 200 episodes per seed×\\times5 seeds\) across all agents\. Planning collapses at8×88\{\\times\}8\(\|S\|=64\|S\|\{=\}64\), while EFE maintains66\.5%66\.5\\%\. Tuned Planning\+IG \(w=100w\{=\}100\) achieves98\.0%98\.0\\%but at higher reward cost\.
##### Observation structure sensitivity\.

A natural concern is whether EFE’s advantage on Tileworld depends on the highly structured bit\-level partition of scan actions\. We test this by replacing the deterministic row/column splits with two alternative observation structures: random partitions \(each scan randomly assigns cells to two groups, breaking orthogonality\) and overlapping partitions \(scans use random linear combinations of coordinates, producing correlated and partially redundant observations\)\. On6×66\{\\times\}6Tileworld \(H=2H\{=\}2, 200 episodes×\\times5 seeds\), EFE achieves the best reward under all three modes: bitwise−20\.52\-20\.52\(74\.2% success\), random−29\.15\-29\.15\(56\.1%\), overlapping−46\.73\-46\.73\(9\.8%\)\. Planning follows the same pattern:−21\.44\-21\.44\(73\.7%\),−30\.41\-30\.41\(54\.5%\),−47\.61\-47\.61\(8\.5%\)\. EFE’s reward advantage over Planning is consistent across modes \(\+0\.9\+0\.9,\+1\.3\+1\.3,\+0\.9\+0\.9\), confirming that the result is not an artifact of the structured observation model\. As expected, random and overlapping modes reduce absolute performance because scans provide less complementary information, but the relative ranking of agents is preserved\.

### 5\.4Interleaved observe\-act: RockSample

To validate Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)beyond observe\-then\-commit settings, we evaluate on RockSample\[[41](https://arxiv.org/html/2607.16981#bib.bib41)\], a standard POMDP benchmark with interleaved observe\-act dynamics\. An agent navigates anN×NN\{\\times\}Ngrid containingKKrocks at known positions, each with hidden binary quality\. Actions include move \(N/S/E/W, cost−0\.5\-0\.5\), check rockkk\(noisy observation, accuracy decays with distance\), sample \(collect rock at current position: good\+10\+10, bad−10\-10\), and exit \(\+10\+10\)\. All agents use depth\-limited belief\-space tree search over factored beliefs \(independent per\-rock\), differing only in the information gain weightww\.

Table 6:RockSample results with tree\-search agents \(500 episodes×\\times5 seeds for RS\[5,3\]–\[7,8\] and 50×\\times2 seeds for RS\[11,11\]\)\. EFE \(w=1w\{=\}1\) achieves the highest or near\-highest reward on all instances while avoiding bad rocks, confirming Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)\. Steps and Checks are omitted because tree\-search agents evaluate all actions at each node\. Per\-step attribution is in Appendix[0\.S](https://arxiv.org/html/2607.16981#Pt0.A19)\.On RS\[5,3\] and RS\[7,4\], EFE \(w=1w\{=\}1\) achieves the highest reward\. On RS\[7,8\], EFE \(w=1w\{=\}1\) scores\+19\.57\+19\.57vs\. Planning’s\+11\.75\+11\.75\(\+7\.82\+7\.82gap\), showing that the advantage grows with the number of observation actions, consistent with the observe\-then\-commit findings\. On all instances up to RS\[7,8\], EFE samples dramatically fewer bad rocks than Greedy \(0\.00–0\.05 vs\. 1\.51–5\.67\), confirming that the information gain term drives checking behavior\.

On RS\[11,11\] \(\|S\|=2,048\|S\|\{=\}2\{,\}048\), EFE and Planning achieve identical reward \(\+13\.64\+13\.64\), with all information\-aware agents vastly outperforming Greedy \(\+13\.6\+13\.6vs\.−21\.9\-21\.9\)\. This result demonstrates tractability, since the factored belief tree search handles 2,048 states in seconds, but it does not differentiate EFE from reward\-only planning\. At depth 2, the tree search horizon is too shallow relative to the 11\-rock environment: all informed agents converge to a conservative “check nearest rock, sample if good, exit” strategy\. Depth 3 is computationally intractable at this scale, requiring orders of magnitude more search time per step\. This mirrors the Tileworld scaling finding \(Figure[3](https://arxiv.org/html/2607.16981#S5.F3)\), where EFE’s advantage requires sufficient depth relative to the state space\. The RS\[7,8\] result, where EFE achieves a\+7\.82\+7\.82reward gap over Planning, shows that EFE differentiation emerges when the observation action count \(K=8K\{=\}8\) provides sufficient room for directed information gathering within the search horizon\. Detailed results including POMCP baselines are in Appendix[0\.S](https://arxiv.org/html/2607.16981#Pt0.A19)\.

### 5\.5Structural inspection

To validate Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)on a domain\-realistic benchmark and demonstrate scalability beyond existing environments, we implement a Structural Inspection POMDP mapping directly to industrial fault detection, medical screening, and non\-destructive testing domains\. The agent inspectsNNcomponents arranged spatially, each with a hidden binary state \(nominal/faulty\)\. Two non\-destructive test types provide accuracy–cost trade\-offs: a visual check \(accuracy 0\.70, cost 0\.5\) and a detailed test \(accuracy 0\.90, cost 2\.0\)\. The agent navigates between components, selects tests, and declares diagnoses with asymmetric penalties \(missed fault−50\-50, false alarm−5\-5\)\. Tests do not alter component states, so this is a factored observation POMDP and Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)applies\.

Table 7:Structural Inspection results\.N=8N\{=\}8uses 2,500 episodes \(500 per seed×\\times5 seeds\) andN=16N\{=\}16uses 1,000 episodes \(200 per seed×\\times5 seeds\)\. Reward shown as mean±\\pmSE\. EFE \(w=1w\{=\}1\) achieves the best reward–accuracy trade\-off\. Accuracy: fraction of components correctly diagnosed\.EFE \(w=1w\{=\}1\) achieves the best reward–accuracy trade\-off on both instances\. OnN=8N\{=\}8\(\|S\|=256\|S\|\{=\}256\), EFE achieves87\.9%87\.9\\%accuracy with reward−20\.60±0\.34\-20\.60\\pm 0\.34, gaining\+14\.9\+14\.9pp accuracy over Planning \(73\.0%73\.0\\%,−17\.85\-17\.85\) at a moderate reward cost\. Planning\+IG \(w=5w\{=\}5\) achieves94\.9%94\.9\\%accuracy but at−22\.82\-22\.82reward: it over\-tests, spending resources on visual checks where a single detailed test suffices\. OnN=16N\{=\}16\(\|S\|=65,536\|S\|\{=\}65\{,\}536\), EFE outperforms Planning by\+7\.9\+7\.9pp accuracy \(86\.1%86\.1\\%vs\.78\.2%78\.2\\%\) with comparable reward \(−45\.71±0\.94\-45\.71\\pm 0\.94vs\.−46\.09±0\.94\-46\.09\\pm 0\.94,p\>0\.05p\>0\.05\)\. This is the largest state space in our evaluation and confirms that the factored belief tree search scales to realistic domains\. The asymmetric penalty structure \(α=25\\alpha=25\) places this firmly in the regime where Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2)predictsw=1w\{=\}1is near\-optimal\.

### 5\.6Summary

The results reveal a consistent pattern across observe\-then\-commit, interleaved, and domain\-realistic settings\. EFE matches Planning\+IG atw=1w\{=\}1, confirming Propositions[1](https://arxiv.org/html/2607.16981#Thmproposition1)and[3](https://arxiv.org/html/2607.16981#Thmproposition3)\. The canonical weight sits at the Pareto knee without per\-environment search\. On Diagnosis and Bandit, EFE Pareto\-dominates same\-horizon planning with higher success and better reward\. On Tileworld, RockSample, and Inspection, it matches or improves upon Planning in reward at comparable or higher accuracy\. The advantage grows with state space size and observation action count, scaling to\|S\|=65,536\|S\|\{=\}65\{,\}536on Inspection \(N=16N\{=\}16\), and it carries over to interleaved observe\-act POMDPs with state transitions \(RockSample, Inspection\)\.

## 6Discussion

##### When does EFE\-as\-ρ\\rhohelp?

EFE’s advantage requires two conditions: \(1\) multiple observation actions with differential informativeness, and \(2\) sufficient planning horizon for recursive EFE to propagate epistemic value\. When condition \(1\) fails, as on Tiger with its single listen action, reward\-only planning matches EFE\. The Tileworld adds a third axis: as the state space grows, the reward signal becomes too diffuse to guide scan selection, and EFE’s advantage increases \(Figure[3](https://arxiv.org/html/2607.16981#S5.F3)\)\. Within\-episode dynamics \(Appendix[0\.L](https://arxiv.org/html/2607.16981#Pt0.A12)\) show the mechanism: EFE concentrates belief via partition\-based narrowing and commits once the value of committing exceeds the value of observing, a crossover that emerges automatically fromw=1w\{=\}1without tuning \(Figure[7](https://arxiv.org/html/2607.16981#Pt0.A7.F7), Appendix[0\.G](https://arxiv.org/html/2607.16981#Pt0.A7)\)\.

##### The canonical weight and why it works\.

EFE does not eliminate the exploration–exploitation weight\. It derives one from the variational bound\. The Pareto analysis \(Figure[1](https://arxiv.org/html/2607.16981#S5.F1)\) shows this weight sits near the reward\-maximizingwret∗w^\{\*\}\_\{\\text\{ret\}\}on every environment, whilewsucc∗w^\{\*\}\_\{\\text\{succ\}\}lies at2020–200200\. The gap betweenwret∗w^\{\*\}\_\{\\text\{ret\}\}andwsucc∗w^\{\*\}\_\{\\text\{succ\}\}has practical implications:w=1w\{=\}1optimizes expected reward, not success probability\. In safety\-critical settings where near\-certain accuracy is required, a higher weight may be appropriate, though such weights must still be tuned per\-environment, and EFE provides a principled starting point\. Increasingwwbeyond 1 buys marginal accuracy at substantial reward cost\. Moreover, adding planning depth to a weighted IG bonus amplifies the weight’s effect \(w=100w\{=\}100on Bandit: Planning\+IG takes 12\.41 inspections vs\. myopic Info Gain’s 10\.99\), making the tuning problem harder with depth\. EFE avoids this because both terms arise from the same variational bound and share a common scale in nats\. The canonical weight isw=1w\{=\}1precisely because information gain is measured in the same units \(nats\) as the KL divergence in the variational bound\. This is a consequence of the mathematical structure rather than an empirical coincidence\. In bits \(base 2\), the corresponding weight would bew=1/ln⁡2≈1\.44w=1/\\ln 2\\approx 1\.44\. The Pareto analysis shows that the reward\-optimal weight is insensitive to this factor: the knee is broad, andw∈\[0\.5,2\.0\]w\\in\[0\.5,2\.0\]yields near\-identical reward on all environments, making the nat/bit distinction practically irrelevant\.

The canonical weight’s advantage requiresγ≥0\.99\\gamma\\geq 0\.99on multi\-observation environments\. Our discount\-sensitivity analysis \(Appendix[0\.Q](https://arxiv.org/html/2607.16981#Pt0.A17)\) reveals a sharp transition: on Diagnosis \(N=4N\{=\}4\), EFE’s success advantage over Planning is\+7\.8\+7\.8pp atγ=1\.0\\gamma\{=\}1\.0but drops to\+1\.0\+1\.0pp atγ=0\.95\\gamma\{=\}0\.95and reverses atγ=0\.90\\gamma\{=\}0\.90\(−1\.6\-1\.6pp\)\. The mechanism is that discounting truncates the effective planning horizon below the number of observations needed for confident diagnosis, erasing EFE’s advantage\. On Tiger \(single observation\), EFE is insensitive toγ\\gammaacross the entire range\[0\.90,1\.0\]\[0\.90,1\.0\]\. This is a meaningful practical limitation: applications with high time pressure or non\-stationary environments that mandate heavy discounting will not benefit from EFE’s epistemic drive\.

##### Zero\-shot weight transfer\.

The tuning problem is not merely inconvenient\. It transfers catastrophically\. We evaluate each environment’s success\-maximizing weightwsucc∗w^\{\*\}\_\{\\text\{succ\}\}on all other environments \(Table[8](https://arxiv.org/html/2607.16981#S6.T8)\)\. On Tiger, transferringw∗=100w^\{\*\}\{=\}100from Diagnosis drops reward from\+5\.02\+5\.02\(EFE\) to\+4\.13\+4\.13, as the agent over\-explores\. On the Testbed, transferringw∗=20w^\{\*\}\{=\}20from Tiger drops reward from\+0\.39\+0\.39\(EFE\) to−0\.16\-0\.16\. On Tiger alone, the nativew∗=20w^\{\*\}\{=\}20achieves\+5\.42\+5\.42\(slightly above EFE’s\+5\.02\+5\.02\), but this weight transfers poorly to all other environments\. Across all 16 environment–weight pairs, EFE \(w=1w\{=\}1\) achieves the best or near\-best reward on every environment without tuning, while every transferred weight underperforms EFE on at least one target\. The success\-maximizing weight varies by5×5\\timesacross environments \(w∗=20w^\{\*\}\{=\}20for Tiger vs\.w∗=100w^\{\*\}\{=\}100for Bandit\), making zero\-shot deployment with a tuned weight unreliable\. EFE sidesteps this entirely: its weight is derived, not tuned, and transfers robustly\.

Table 8:Zero\-shot weight transfer: mean reward achieved by each weight on each environment\. EFE \(w=1w\{=\}1\) achieves the best or near\-best reward on all four targets\. Each success\-tunedw∗w^\{\*\}underperforms EFE on other environments\. Bold: best reward per environment\.
##### When isw=1w\{=\}1near\-optimal?

Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2)formalizes the conditions forH=1H\{=\}1, two states: the observation thresholdwthresh∗w^\{\*\}\_\{\\text\{thresh\}\}depends on the reward asymmetry ratioα=\|R−\|/R\+\\alpha=\|R^\{\-\}\|/R^\{\+\}, the observation informativenessη=Imax/c\\eta=I\_\{\\max\}/c, and the observation accuracypp\. Table[1](https://arxiv.org/html/2607.16981#S3.T1)validates this across our environments\. The key predictor isα\\alpha: environments with high penalty asymmetry \(α≥5\\alpha\\geq 5: Tiger, Diagnosis, Tileworld\) havewthresh∗≪0w^\{\*\}\_\{\\text\{thresh\}\}\\ll 0, meaningw=1w\{=\}1is far above the threshold and near\-optimal\.

To assess whether theH=1H\{=\}1analysis extends to multi\-step planning, we conducted a Monte Carlo study over 100 randomly generated two\-state environments \(samplingα∈\[1,50\]\\alpha\\in\[1,50\],p∈\[0\.55,0\.95\]p\\in\[0\.55,0\.95\], cost∈\[0\.1,5\]\\in\[0\.1,5\]\)\. For each environment andH∈\{1,2,3\}H\\in\\\{1,2,3\\\}, we compute the reward\-optimalw∗w^\{\*\}by grid search and classifyw=1w\{=\}1as near\-optimal when the reward gap fromw∗w^\{\*\}is withinmax⁡\(5%,0\.5\)\\max\(5\\%,0\.5\)of the best reward \(Figure[12](https://arxiv.org/html/2607.16981#Pt0.A16.F12)\)\. The near\-optimality rate increases from 9% atH=1H\{=\}1to 22% atH=2H\{=\}2and 32% atH=3H\{=\}3, confirming that multi\-step planning amplifies the value of observation and expands the region wherew=1w\{=\}1is near\-optimal\. For high\-asymmetry environments \(α≥10\\alpha\\geq 10\), the improvement is sharper: 11%→\\to24%→\\to32%\. This is consistent with the Bandit result, whereα=1\.1\\alpha\{=\}1\.1and theH=1H\{=\}1analysis predicts marginal sufficiency, yet multi\-step EFE atH=2H\{=\}2achieves87\.3%87\.3\\%success \(Table[4](https://arxiv.org/html/2607.16981#S5.T4)\)\.

Many real\-world decision problems haveα≫1\\alpha\\gg 1: in medical diagnosis, fault detection, and security screening, a missed condition costs far more than another test\. This is exactly the regime wherew=1w\{=\}1is well calibrated, and the multi\-step analysis shows the advantage grows with planning depth\. The practical consequence is that in these domains the exploration weight can be deployed as\-is, with no per\-task search\.

##### Robustness to model misspecification\.

Our main results assume exact knowledge of the generative model\. Appendix[0\.R](https://arxiv.org/html/2607.16981#Pt0.A18)investigates robustness when the agent’s believed observation accuracy differs from the true value by up to±0\.15\\pm 0\.15\. On Tiger, success rates remain above 96\.7% across all mismatch levels\. On Diagnosis, degradation is larger but graceful\. Overestimating sensor accuracy \(positive mismatch\) is more harmful than underestimating it, because the agent commits prematurely on insufficient evidence\. EFE’s intrinsic epistemic drive provides a partial buffer: even with a miscalibrated model, the information gain term still encourages observation, compensating for overconfident planning\.

##### Approximate planning with MCTS\-EFE\.

The exact tree search cost𝒪​\(K⋅\|𝒪\|H\)\\mathcal\{O\}\(K\\cdot\|\\mathcal\{O\}\|^\{H\}\)limits practical horizons toH=2H\{=\}2–33withK≥3K\\geq 3\. To push beyond this, we implement MCTS with EFE as a leaf heuristic: the tree policy uses UCB1 with observation\-outcome enumeration, and leaf nodes are evaluated via greedy EFE rollouts\. On Tiger atH=10H\{=\}10, MCTS\-EFE with 500 simulations achieves 97\.2% success \(7\.2s per 200 episodes\), compared to 89\.7% for POMCP at matched budget \(1\.7s\)\. This demonstrates that EFE’s closed\-form information valuation provides a substantially stronger leaf heuristic than semi\-informed rollouts\. The comparison is designed to be fair to POMCP: our implementation already uses belief\-optimal commits \(not purely random rollouts\): observations are sampled uniformly, but the rollout terminates with the commit action maximizing expected reward under the Bayesian\-updated belief \(Appendix[0\.T](https://arxiv.org/html/2607.16981#Pt0.A20)\)\. The remaining performance gap therefore reflects the value of directed observation selection: EFE chooses which test to run based on information gain, while POMCP samples tests uniformly\. The claim is not that EFE\-based planning exceeds state\-of\-the\-art POMDP solvers, but that the EFE leaf heuristic provides a principled, zero\-tuning alternative to heuristic rollout design\. POMCP with fully informed rollouts \(e\.g\., using domain knowledge or a learned value function for observation selection\) would narrow the gap further\.

The approach also scales to multiple observation actions\. For single\-observation environments, the observation\-outcome enumeration \(2 outcomes per action\) keeps expansion costs low\. On Diagnosis \(N=4N\{=\}4,K=2K\{=\}2tests\), MCTS\-EFE atH=5H\{=\}5achieves98\.0%98\.0\\%success compared to POMCP’s71\.3%71\.3\\%at matched budget \(200 simulations\), demonstrating that EFE scales to multi\-observation environments\. To validate on a larger multi\-observation environment, we evaluate on Tileworld6×66\{\\times\}6\(\|S\|=36\|S\|\{=\}36,K=6K\{=\}6scans\)\. MCTS\-EFE\(50\) achieves96\.0%96\.0\\%success and−19\.04\-19\.04reward, outperforming both Exact\-EFE \(75\.0%,−20\.11\-20\.11\) and POMCP at matched \(50\) or higher \(200\) simulation budgets \(2\.0%2\.0\\%/15\.0%15\.0\\%success\)\. The improvement over exact tree search reflects MCTS’s ability to concentrate samples on promising observation sequences\. The8181–9494pp gap over POMCP confirms that EFE’s directed observation selection is essential at this scale: POMCP’s semi\-informed rollouts cannot identify which of the 6 scans to perform\. For larger observation spaces, more efficient tree policies, such as progressive widening or double progressive widening\[[3](https://arxiv.org/html/2607.16981#bib.bib3)\], would further improve scalability\. We leave this to future work\.

##### When to use EFE\-as\-ρ\\rho: summary\.

Our analysis identifies the following conditions favoring EFE over tuned alternatives:

- •Reward asymmetryα≥5\\alpha\\geq 5: the penalty for a wrong commit far exceeds the observation cost, makingw=1w\{=\}1automatically near\-optimal \(Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2)\)\.
- •Multiple observation actions: EFE’s joint pragmatic–epistemic objective selects which information to gather, producing gains over reward\-only planning even at matched horizon\.
- •Moderate to large state spaces \(\|S\|≥16\|S\|\\geq 16\): as the reward signal diffuses, directed information gathering becomes essential, and EFE’s advantage grows with\|S\|\|S\|\(Figure[3](https://arxiv.org/html/2607.16981#S5.F3)\), scaling to\|S\|=65,536\|S\|\{=\}65\{,\}536on Structural Inspection\.
- •Interleaved observe\-act with preserved hidden state: when the hidden state does not change under observation and navigation actions \(factored observation POMDPs, Table[2](https://arxiv.org/html/2607.16981#S3.T2)\), the canonical\-weight equivalence holds and EFE directs both where to go and what to check \(Tables[6](https://arxiv.org/html/2607.16981#S5.T6),[7](https://arxiv.org/html/2607.16981#S5.T7)\)\.
- •Cross\-environment deployment: when the weight cannot be tuned per\-task,w=1w\{=\}1transfers robustly while tuned weights fail catastrophically \(Table[8](https://arxiv.org/html/2607.16981#S6.T8)\)\.
- •Planning horizonH≥2H\\geq 2: recursive EFE propagates epistemic value across steps\. AtH=1H\{=\}1, it reduces to myopic IG withw=1w\{=\}1\.

EFE is not recommended whenα≈1\\alpha\\approx 1\(symmetric penalties\), in navigation\-style POMDPs where observations are tied to translation and a greedy mover already receives informative feedback \(Table[17](https://arxiv.org/html/2607.16981#Pt0.A9.T17)\), or when model misspecification exceeds±0\.15\\pm 0\.15in observation accuracy\.

##### Limitations and future work\.

The formal equivalence extends from observe\-then\-commit \(Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)\) to factored observation POMDPs \(Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)\), covering settings where observation actions preserve the hidden state\. POMDPs where information\-gathering actions change the hidden state \(e\.g\., destructive testing\) require the full coupling termΔT\\Delta\_\{T\}and are not covered\. Extending the formal treatment to such settings remains future work\. For slowly drifting hidden states \(e\.g\., progressive disease\),ΔT\\Delta\_\{T\}is small but nonzero\. The approximation error scales with the per\-step transition entropyH​\(shid′\|shid,a\)H\(s^\{\\prime\}\_\{\\mathrm\{hid\}\}\|s\_\{\\mathrm\{hid\}\},a\), and characterizing the regime where this remains acceptable is an open question\.

On the practical side, scaling MCTS\-EFE to multi\-observation environments requires more efficient tree policies, as discussed above\. The EFE agent assumes a known generative model\. The misspecification analysis \(Appendix[0\.R](https://arxiv.org/html/2607.16981#Pt0.A18)\) shows graceful degradation with moderate model errors, but extending to full model learning \(where AIF and BAMDPs converge\) remains important future work\. Navigation \(Appendix[0\.I](https://arxiv.org/html/2607.16981#Pt0.A9)\) shows that scale alone does not rescue epistemic planning when the observation model is proximity\-based: NavMyopic leads at3×33\{\\times\}3,5×55\{\\times\}5, and7×77\{\\times\}7\. The practical contrast is with domains that offer explicit, choice\-set observation actions \(Diagnosis, Tileworld, RockSample\), where\|S\|≥16\|S\|\\geq 16marks the regime in which EFE separates from reward\-only planning\.

##### Broader impact\.

This work is primarily theoretical\. The principled derivation of exploration weights could benefit safety\-critical decision\-making by reducing reliance on ad\-hoc tuning\.

## 7Conclusion

Theρ\\rho\-POMDP framework and active inference converge on the same mathematical object: a belief\-dependent utility that adds information gain at weightw=1w\{=\}1to the reward signal\. EFE does not eliminate the exploration–exploitation trade\-off\. Instead, it replaces ad hoc bonus weights with a coefficient fixed by the variational geometry \(nats\), which then doubles as the Planning\+IG weight in Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)\. The Pareto analysis shows this canonical weight is near\-optimal for expected reward across all tested test\-selection environments, sitting at the knee where further increases buy marginal accuracy at substantial reward cost\. Safety\-critical applications requiring near\-certain accuracy may benefit from higher weights\. A Monte Carlo study over randomly generated environments \(Appendix[0\.P](https://arxiv.org/html/2607.16981#Pt0.A16)\) confirms that the near\-optimality basin widens with planning horizon, extending theH=1H\{=\}1analysis of Proposition[2](https://arxiv.org/html/2607.16981#Thmproposition2)to multi\-step settings\.

Proposition[3](https://arxiv.org/html/2607.16981#Thmproposition3)extends the formal equivalence beyond observe\-then\-commit to factored observation POMDPs, validated on RockSample \(up to\|S\|=2,048\|S\|\{=\}2\{,\}048\) and a new Structural Inspection benchmark \(\|S\|\|S\|up to65,53665\{,\}536\) mapping directly to industrial fault detection and medical screening domains\. On Inspection \(N=16N\{=\}16\), EFE achieves86\.1%86\.1\\%diagnostic accuracy where planning scores78\.2%78\.2\\%, at comparable reward \(p\>0\.05p\>0\.05\)\. Comparison against POMCP with semi\-informed rollouts \(random observations, belief\-optimal commits, Appendix[0\.T](https://arxiv.org/html/2607.16981#Pt0.A20)\) shows that EFE’s advantage comes from directed observation selection, not from using a stronger solver\. Domain\-informed POMCP rollouts would narrow this gap, but they require per\-environment engineering that EFE avoids\. MCTS\-EFE on Tiger achieves 97\.2% success atH=10H\{=\}10with 500 simulations, outperforming POMCP \(89\.7%\) at matched compute\. The advantage is most pronounced in multi\-observation\-action settings: on the8×88\{\\times\}8Tileworld, EFE achieves66\.5%66\.5\\%success where reward\-only planning collapses to2\.5%2\.5\\%\. The message for practitioners is simple: wherever an agent must pay for its observations and answer for its mistakes, the exploration weight it needs is not a hyperparameter to search over, because active inference already supplies it\.

\{credits\}

#### 7\.0\.1\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- Araya\-López et al\. \[2010\]M\. Araya\-López, O\. Buffet, V\. Thomas, and F\. Charpillet\.A POMDP extension with belief\-dependent rewards\.In*Advances in Neural Information Processing Systems*, 2010\.
- Bellemare et al\. \[2016\]M\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. Munos\.Unifying count\-based exploration and intrinsic motivation\.In*Advances in Neural Information Processing Systems*, 2016\.
- Benchetrit et al\. \[2025\]R\. Benchetrit, I\. Lev\-Yehudi, A\. Zhitnikov, and V\. Indelman\.Anytime incrementalρ\\rhoPOMDP planning in continuous spaces\.*arXiv preprint arXiv:2502\.02549*, 2025\.
- Burda et al\. \[2019\]Y\. Burda, H\. Edwards, A\. Storkey, and O\. Klimov\.Exploration by random network distillation\.In*International Conference on Learning Representations*, 2019\.
- Champion et al\. \[2026\]T\. Champion, H\. Bowman, D\. Marković, and M\. Grześ\.Reframing the expected free energy: Four formulations and a unification\.*Neural Computation*, 38\(3\):439–469, 2026\.
- Da Costa et al\. \[2020\]L\. Da Costa, T\. Parr, N\. Sajid, S\. Veselic, V\. Neacsu, and K\. Friston\.Active inference on discrete state\-spaces: A synthesis\.*Journal of Mathematical Psychology*, 99:102447, 2020\.
- Da Costa et al\. \[2023\]L\. Da Costa, N\. Sajid, T\. Parr, K\. Friston, and R\. Smith\.Reward maximization through discrete active inference\.*Neural Computation*, 35\(5\):807–852, 2023\.
- de Vries et al\. \[2025\]B\. de Vries, W\. Nuijten, T\. van de Laar, et al\.Expected free energy\-based planning as variational inference\.*arXiv preprint arXiv:2504\.14898*, 2025\.
- Duff and Barto \[2002\]M\. O\. Duff and A\. G\. Barto\.Optimal learning: Computational procedures for Bayes\-adaptive Markov decision processes\.PhD thesis, University of Massachusetts Amherst, 2002\.
- Fehr et al\. \[2018\]M\. Fehr, O\. Buffet, V\. Thomas, and J\. Dibangoye\.ρ\\rho\-POMDPs have Lipschitz\-continuousϵ\\epsilon\-optimal value functions\.In*Advances in Neural Information Processing Systems*, 2018\.
- Fountas et al\. \[2020\]Z\. Fountas, N\. Sajid, P\. A\. M\. Mediano, and K\. Friston\.Deep active inference agents using Monte\-Carlo methods\.In*Advances in Neural Information Processing Systems*, 2020\.
- Friston \[2010\]K\. Friston\.The free\-energy principle: a unified brain theory?*Nature Reviews Neuroscience*, 11\(2\):127–138, 2010\.
- Friston et al\. \[2015\]K\. Friston, F\. Rigoli, D\. Ognibene, C\. Mathys, T\. FitzGerald, and G\. Pezzulo\.Active inference and epistemic value\.*Cognitive Neuroscience*, 6\(4\):187–214, 2015\.
- Friston et al\. \[2021\]K\. Friston, L\. Da Costa, D\. Hafner, C\. Hesp, and T\. Parr\.Sophisticated inference\.*Neural Computation*, 33\(3\):713–763, 2021\.
- Ghavamzadeh et al\. \[2015\]M\. Ghavamzadeh, S\. Mannor, J\. Pineau, and A\. Tamar\.Bayesian reinforcement learning: A survey\.*Foundations and Trends in Machine Learning*, 8\(5–6\):359–483, 2015\.
- Guez et al\. \[2013\]A\. Guez, D\. Silver, and P\. Dayan\.Scalable and efficient Bayes\-adaptive reinforcement learning based on Monte\-Carlo tree search\.*Journal of Artificial Intelligence Research*, 48:841–883, 2013\.
- Haarnoja et al\. \[2018\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.In*International Conference on Machine Learning*, 2018\.
- Heins et al\. \[2022\]C\. Heins, B\. Millidge, D\. Demekas, B\. Klein, K\. Friston, I\. Couzin, and A\. Tschantz\.pymdp: A Python library for active inference in discrete state spaces\.*Journal of Open Source Software*, 7\(73\):4098, 2022\.
- Houthooft et al\. \[2016\]R\. Houthooft, X\. Chen, Y\. Duan, J\. Schulman, F\. De Turck, and P\. Abbeel\.VIME: Variational information maximizing exploration\.In*Advances in Neural Information Processing Systems*, 2016\.
- Howard \[1966\]R\. A\. Howard\.Information value theory\.*IEEE Transactions on Systems Science and Cybernetics*, 2\(1\):22–26, 1966\.
- Itti and Baldi \[2009\]L\. Itti and P\. Baldi\.Bayesian surprise attracts human attention\.*Vision Research*, 49\(10\):1295–1306, 2009\.
- Kaelbling et al\. \[1998\]L\. P\. Kaelbling, M\. L\. Littman, and A\. R\. Cassandra\.Planning and acting in partially observable stochastic domains\.*Artificial Intelligence*, 101\(1–2\):99–134, 1998\.
- Kurniawati et al\. \[2008\]H\. Kurniawati, D\. Hsu, and W\. S\. Lee\.SARSOP: Efficient point\-based POMDP planning by approximating optimally reachable belief spaces\.In*Robotics: Science and Systems*, 2008\.
- Levine \[2018\]S\. Levine\.Reinforcement learning and control as probabilistic inference: Tutorial and review\.*arXiv preprint arXiv:1805\.00909*, 2018\.
- Lindley \[1956\]D\. V\. Lindley\.On a measure of the information provided by an experiment\.*The Annals of Mathematical Statistics*, 27\(4\):986–1005, 1956\.
- Maisto et al\. \[2025\]D\. Maisto, F\. Gregoretti, K\. J\. Friston, and G\. Pezzulo\.Active inference tree search in large POMDPs\.*Neurocomputing*, 623:129319, 2025\.
- Millidge et al\. \[2020\]B\. Millidge, A\. Tschantz, and C\. L\. Buckley\.On the relationship between active inference and control as inference\.In*International Workshop on Active Inference*, pp\. 3–11\. Springer, 2020\.
- Millidge et al\. \[2021\]B\. Millidge, A\. Tschantz, and C\. L\. Buckley\.Whence the expected free energy?*Neural Computation*, 33\(2\):447–482, 2021\.
- Oudeyer and Kaplan \[2007\]P\.\-Y\. Oudeyer and F\. Kaplan\.What is intrinsic motivation? A typology of computational approaches\.*Frontiers in Neurorobotics*, 1:6, 2007\.
- Parr and Friston \[2019\]T\. Parr and K\. J\. Friston\.Generalised free energy and active inference\.*Biological Cybernetics*, 113\(5\):495–513, 2019\.
- Parr et al\. \[2022\]T\. Parr, G\. Pezzulo, and K\. J\. Friston\.*Active Inference: The Free Energy Principle in Mind, Brain, and Behavior*\.MIT Press, 2022\.
- Pathak et al\. \[2017\]D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. Darrell\.Curiosity\-driven exploration by self\-supervised prediction\.In*International Conference on Machine Learning*, 2017\.
- Pineau et al\. \[2003\]J\. Pineau, G\. Gordon, and S\. Thrun\.Point\-based value iteration: An anytime algorithm for POMDPs\.In*International Joint Conference on Artificial Intelligence*, 2003\.
- Rawlik et al\. \[2012\]K\. Rawlik, M\. Toussaint, and S\. Vijayakumar\.On stochastic optimal control and reinforcement learning by approximate inference\.In*Robotics: Science and Systems*, 2012\.
- Russo and Van Roy \[2014\]D\. Russo and B\. Van Roy\.Learning to optimize via information\-directed sampling\.In*Advances in Neural Information Processing Systems*, 2014\.
- Sajid et al\. \[2021\]N\. Sajid, P\. J\. Ball, T\. Parr, and K\. J\. Friston\.Active inference: Demystified and compared\.*Neural Computation*, 33\(3\):674–712, 2021\.
- Schmidhuber \[1991\]J\. Schmidhuber\.A possibility for implementing curiosity and boredom in model\-building neural controllers\.In*Proc\. International Conference on Simulation of Adaptive Behavior*, pp\. 222–227, 1991\.
- Shani et al\. \[2013\]G\. Shani, J\. Pineau, and R\. Kaplow\.A survey of point\-based POMDP solvers\.*Autonomous Agents and Multi\-Agent Systems*, 27\(1\):1–51, 2013\.
- Silver and Veness \[2010\]D\. Silver and J\. Veness\.Monte\-Carlo planning in large POMDPs\.In*Advances in Neural Information Processing Systems*, 2010\.
- Smallwood and Sondik \[1973\]R\. D\. Smallwood and E\. J\. Sondik\.The optimal control of partially observable Markov processes over a finite horizon\.*Operations Research*, 21\(5\):1071–1088, 1973\.
- Smith and Simmons \[2004\]T\. Smith and R\. Simmons\.Heuristic search value iteration for POMDPs\.In*Uncertainty in Artificial Intelligence*, 2004\.
- Sunberg and Kochenderfer \[2018\]Z\. N\. Sunberg and M\. J\. Kochenderfer\.Online algorithms for POMDPs with continuous state, action, and observation spaces\.In*International Conference on Automated Planning and Scheduling*, 2018\.
- Todorov \[2006\]E\. Todorov\.Linearly\-solvable Markov decision problems\.In*Advances in Neural Information Processing Systems*, 2006\.
- Tschantz et al\. \[2020\]A\. Tschantz, B\. Millidge, A\. K\. Seth, and C\. L\. Buckley\.Reinforcement learning through active inference\.In*ICLR Workshop on Bridging AI and Cognitive Science*, 2020\.
- Ye et al\. \[2017\]N\. Ye, A\. Somani, D\. Hsu, and W\. S\. Lee\.DESPOT: Online POMDP planning with regularization\.*Journal of Artificial Intelligence Research*, 58:231–266, 2017\.

## Appendix 0\.AProof of Proposition[1](https://arxiv.org/html/2607.16981#Thmproposition1)

###### Proof

We show thatarg⁡mina⁡𝒢​\(a,b,d\)=arg⁡maxa⁡Vρ​\(a,b,d\)\\arg\\min\_\{a\}\\mathcal\{G\}\(a,b,d\)=\\arg\\max\_\{a\}V\_\{\\rho\}\(a,b,d\)at every belief node, whereVρV\_\{\\rho\}is theρ\\rho\-POMDP value function withρEFE​\(b,a\)\\rho\_\{\\mathrm\{EFE\}\}\(b,a\)as defined in the proposition\.

DefineV​\(a,b,d\)≜−𝒢​\(a,b,d\)V\(a,b,d\)\\triangleq\-\\mathcal\{G\}\(a,b,d\)\. Since negation reverses ordering,arg⁡mina⁡𝒢=arg⁡maxa⁡V\\arg\\min\_\{a\}\\mathcal\{G\}=\\arg\\max\_\{a\}V\.

Case 1: Commit actions\.For commit actionii, Equation[3](https://arxiv.org/html/2607.16981#S3.E3)gives𝒢​\(commiti\)=−𝔼b​\[Ri\]\\mathcal\{G\}\(\\text\{commit\}\_\{i\}\)=\-\\mathbb\{E\}\_\{b\}\[R\_\{i\}\], soV​\(commiti,b\)=𝔼b​\[Ri\]V\(\\text\{commit\}\_\{i\},b\)=\\mathbb\{E\}\_\{b\}\[R\_\{i\}\]\. Theρ\\rho\-POMDP value isVρ​\(commiti,b\)=𝔼b​\[Ri\]\+ρEFE​\(b,commiti\)=𝔼b​\[Ri\]\+0V\_\{\\rho\}\(\\text\{commit\}\_\{i\},b\)=\\mathbb\{E\}\_\{b\}\[R\_\{i\}\]\+\\rho\_\{\\mathrm\{EFE\}\}\(b,\\text\{commit\}\_\{i\}\)=\\mathbb\{E\}\_\{b\}\[R\_\{i\}\]\+0\. These are identical\.

Case 2: Observation actions\.For observation actionkk, Equation[3](https://arxiv.org/html/2607.16981#S3.E3)gives𝒢​\(obsk\)=ck−Ik​\(b\)\+𝔼o​\[mina′⁡𝒢​\(a′,bo′\)\]\.\\mathcal\{G\}\(\\text\{obs\}\_\{k\}\)=c\_\{k\}\-I\_\{k\}\(b\)\+\\mathbb\{E\}\_\{o\}\[\\min\_\{a^\{\\prime\}\}\\mathcal\{G\}\(a^\{\\prime\},b^\{\\prime\}\_\{o\}\)\]\.Negating:V​\(obsk,b\)=−ck\+Ik​\(b\)\+𝔼o​\[maxa′⁡V​\(a′,bo′\)\]\.V\(\\text\{obs\}\_\{k\},b\)=\-c\_\{k\}\+I\_\{k\}\(b\)\+\\mathbb\{E\}\_\{o\}\[\\max\_\{a^\{\\prime\}\}V\(a^\{\\prime\},b^\{\\prime\}\_\{o\}\)\]\.Theρ\\rho\-POMDP Bellman backup withR​\(b,obsk\)=−ckR\(b,\\text\{obs\}\_\{k\}\)=\-c\_\{k\}andρEFE​\(b,obsk\)=Ik​\(b\)\\rho\_\{\\mathrm\{EFE\}\}\(b,\\text\{obs\}\_\{k\}\)=I\_\{k\}\(b\)givesVρ​\(obsk,b\)=−ck\+Ik​\(b\)\+γ​𝔼o​\[maxa′⁡Vρ​\(a′,bo′\)\]\.V\_\{\\rho\}\(\\text\{obs\}\_\{k\},b\)=\-c\_\{k\}\+I\_\{k\}\(b\)\+\\gamma\\mathbb\{E\}\_\{o\}\[\\max\_\{a^\{\\prime\}\}V\_\{\\rho\}\(a^\{\\prime\},b^\{\\prime\}\_\{o\}\)\]\.Withγ=1\\gamma=1\(undiscounted finite horizon\), the two recursions are structurally identical\. Since the terminal values \(commit actions\) agree and the recursive updates agree, by induction on the remaining horizonH−dH\-d,V​\(a,b,d\)=Vρ​\(a,b,d\)V\(a,b,d\)=V\_\{\\rho\}\(a,b,d\)for allaa,bb,dd\. Policies therefore coincide at every belief node\.

## Appendix 0\.BFull per\-environment results

Table 9:Tiger problem \(5,000 episodes: 1,000 per seed×\\times5 seeds,H=6H\{=\}6\)\. Reward asymmetry\+10/−100\+10/\{\-\}100\.Table 10:Diagnosis \(N=4N\{=\}4,K=2K\{=\}2tests,H=3H\{=\}3, 5,000 episodes: 1,000 per seed×\\times5 seeds\)\.Table 11:Structured bandit \(K=4K\{=\}4arms,H=2H\{=\}2, 5,000 episodes: 1,000 per seed×\\times5 seeds\)\.Table 12:Tileworld6×66\{\\times\}6\(2,500 episodes: 500 per seed×\\times5 seeds,H=2H\{=\}2\), full agent set\.Table 13:Tileworld8×88\{\\times\}8\(600 episodes: 200 per seed×\\times3 seeds,H=2H\{=\}2\), full agent set, from an earlier 3\-seed run\. The 5\-seed results reported in the main text \(Figure[3](https://arxiv.org/html/2607.16981#S5.F3)\) show EFE at66\.5%66\.5\\%success\. Planning and untuned Info Gain fail to explore because a single noisy scan over 64 states barely narrows the belief, while EFE’s epistemic bonus drives meaningful exploration\.Table 14:DiagnosisN=16N\{=\}16\(K=4K\{=\}4tests,H=2H\{=\}2, 600 episodes: 200 per seed×\\times3 seeds\)\.
## Appendix 0\.CTwo\-state testbed

Table 15:Information\-seeking testbed \(5,000 episodes,H=4H\{=\}4\)\. Symmetric rewards\+1/−1\+1/\{\-\}1\. The tuning procedure selectsw∗=50w^\{\*\}\{=\}50, which over\-explores\. EFE \(w=1w\{=\}1\) achieves better reward\.Under mild reward asymmetry \(\+1/−1\+1/\{\-\}1\), EFE gathers more information than is instrumentally optimal \(\+0\.39\+0\.39vs\. Planning’s\+0\.49\+0\.49,p<0\.001p<0\.001\) because uncertainty reduction has intrinsic value under EFE\. Tuned Info Gain \(w=50w\{=\}50\) over\-explores catastrophically: 99\.9% success but negative reward\. This result is informative about the conditions under whichw=1w\{=\}1is near\-optimal: the reward asymmetry ratio\|R−\|/\|R\+\|=1\|R^\{\-\}\|/\|R^\{\+\}\|=1means that the penalty for guessing wrong barely exceeds the observation cost, so the reward\-optimal weightwret∗<1w^\{\*\}\_\{\\text\{ret\}\}<1\. In this regime, EFE assigns too much weight to information gain\. Many real\-world decision problems, including medical diagnosis, fault detection, and security screening, have asymmetric penalties \(\|R−\|/\|R\+\|≫1\|R^\{\-\}\|/\|R^\{\+\}\|\\gg 1\), precisely the regime wherew=1w\{=\}1is well\-calibrated \(Tiger, Diagnosis, Tileworld\)\.

## Appendix 0\.DTiger reward asymmetry sweep

![Refer to caption](https://arxiv.org/html/2607.16981v1/x4.png)Figure 4:Reward asymmetry sweep on Tiger \(H=6H\{=\}6, 500 episodes per point\)\. EFE and Planning adapt smoothly across three orders of magnitude in penalty\. Info Gain’s fixed weight produces inconsistent performance\.To test whether EFE’s Tiger parity holds across reward scales, we sweep the penalty magnitude from\|R−\|=1\|R^\{\-\}\|\{=\}1to500500\. EFE naturally adapts its exploration depth, tracking the reward\-optimal strategy without reconfiguration\. Info Gain collapses at low penalties where its weight \(w=20w\{=\}20, tuned at\|R−\|=100\|R^\{\-\}\|\{=\}100\) drives excessive listening\.

## Appendix 0\.EObservation action scaling

![Refer to caption](https://arxiv.org/html/2607.16981v1/x5.png)Figure 5:Observation action scaling on Diagnosis \(N=8N\{=\}8,H=3H\{=\}3, 500 episodes\)\. EFE’s advantage over Planning grows withKK, confirming that EFE is most valuable when the agent must choose among differentially informative observations\.HoldingN=8N\{=\}8fixed on Diagnosis and varyingKKfrom 1 to 3 isolates the “which information” hypothesis\. AtK=1K\{=\}1, all agents face the same single test and perform similarly\. AsKKincreases, EFE’s advantage over Planning grows because the information gain term guides test selection\.

## Appendix 0\.FTileworld belief evolution

![Refer to caption](https://arxiv.org/html/2607.16981v1/x6.png)Figure 6:Belief evolution within an EFE episode on the6×66\{\\times\}6Tileworld\. Each panel shows belief probability over the 36 grid cells \(darker = higher probability\) after successive scans\. Blue outlines mark the scanned region\. The agent concentrates belief toward the target tile \(green star\) and commits once confident\.
## Appendix 0\.GEFE trajectory decomposition

![Refer to caption](https://arxiv.org/html/2607.16981v1/x7.png)Figure 7:EFE decomposition within Tiger episodes \(H=6H\{=\}6\)\. The agent commits at the crossover where commit value \(red\) exceeds observe value \(blue\)\. As belief entropy \(green shading\) decreases, the transition from exploration to exploitation occurs automatically\.
## Appendix 0\.HEnvironment specifications

Table 16:Full environment specifications\.All environments are implemented as OpenAI Gymnasium environments with standardreset/stepAPI\. The Diagnosis environment usesK=⌈log2⁡N⌉K=\\lceil\\log\_\{2\}N\\rceilbinary tests, each providing noisy information about a different partition of the state space\. The Navigation environment uses proximity\-based warm/cold signals with accuracy depending on Manhattan distance to the hidden goal\. The Tileworld projects the Diagnosis partition structure onto a 2D grid:K=2​⌈log2⁡N⌉K=2\\lceil\\log\_\{2\}N\\rceilscans partition the grid by bit\-level splits of the row and column indices, enabling complete spatial disambiguation\.

## Appendix 0\.INavigation results

Navigation uses only move actions\. The goal cell is hidden, and each move yields a proximity\-based warm/cold observation \(Section[0\.H](https://arxiv.org/html/2607.16981#Pt0.A8)\)\. Unlike Diagnosis or Tileworld, there is no separate menu of observation actions: information arrives as a side effect of translation, and greedy progress toward the belief mode already provides correlated evidence\.

Table 17:Navigation scaling \(150150episodes per seed×\\times55seeds =750750episodes per agent per grid\)\. Step budget3​n23n^\{2\}\. NavEFE uses depth\-limited planningH=2H\{=\}2\. Seeresults\_navigation\_scaling\.csv\.NavMyopic \(move toward the highest\-probability cell\) remains the strongest baseline at every scale: extra steps devoted to exploratory detours incur the per\-move cost without the sharp information returns seen when tests can be chosen explicitly\. This is not a failure ofw=1w\{=\}1so much as a structural regime change relative to our main benchmarks\. Importantly, NavEFE is competitive with one\-step NavInfoGain at7×77\{\\times\}7\(higher mean reward, comparable success\), showing that recursive epistemic valuation is not harmful once the grid is large enough that myopic IG is already costly\. The contrast with Tileworld \(Figure[3](https://arxiv.org/html/2607.16981#S5.F3)\) underscores condition \(1\) in Section[6](https://arxiv.org/html/2607.16981#S6): EFE’s clearest wins appear when the agent must select among differentially informative observation actions, not only where to translate next under a smooth proximity field\.

## Appendix 0\.JScaling analysis

Table 18:Scaling analysis: DiagnosisN=2N=2to1616\(2,500 episodes: 500 per seed×\\times5 seeds,H=2H\{=\}2\)\.AtH=2H\{=\}2, EFE and Planning perform similarly across allNN, both substantially outperforming Myopic and untuned Info Gain\. This contrasts with theH=3H\{=\}3Diagnosis result \(Table[4](https://arxiv.org/html/2607.16981#S5.T4)\), where EFE outperformed Planning by\+7\.9\+7\.9pp, confirming that EFE’s advantage requires sufficient recursive depth\. EFE\-over\-Myopic advantage grows from\+13\.8\+13\.8pp atN=2N\{=\}2to\+39\.1\+39\.1pp atN=16N\{=\}16, confirming that deeper planning is increasingly valuable as state spaces grow\.

## Appendix 0\.KPyMDP validation

We validated our EFE computation against thepymdplibrary\[[18](https://arxiv.org/html/2607.16981#bib.bib18)\]by wrapping pymdp’s standard Agent class\. Both implementations agree on observe\-vs\-commit decisions across all tested belief states \(uniform, intermediate, and confident beliefs on the two\-state environments\)\. Information gain values computed by our recursive scheme and pymdp’scontrolmodule agree qualitatively, with differences attributable to our recursive multi\-step evaluation vs\. pymdp’s single\-step policy enumeration\.

## Appendix 0\.LEpistemic foraging dynamics

The aggregate statistics in Table[4](https://arxiv.org/html/2607.16981#S5.T4)show what EFE achieves\. We now visualize how it achieves it by examining within\-episode dynamics on extended Diagnosis instances \(N=8N\{=\}8,K=3K\{=\}3, accuracy 0\.75\) that produce episodes with 8–15\+ observation steps\.

##### Belief evolution\.

Figure[8](https://arxiv.org/html/2607.16981#Pt0.A12.F8)shows belief\-state evolution within representative episodes for EFE, Planning, and Info Gain\. EFE exhibits systematic partition\-based narrowing: early tests eliminate large groups of states, and later tests refine among remaining candidates\. Planning \(without epistemic value\) makes observations but lacks guidance on which test to run, producing less structured belief concentration\. Info Gain continues testing well past the point where the agent has high confidence\.

![Refer to caption](https://arxiv.org/html/2607.16981v1/x8.png)Figure 8:Belief evolution within representative Diagnosis episodes \(N=8N\{=\}8,K=3K\{=\}3, accuracy 0\.75\)\. Darker = higher probability\. Green: true state\. Dashed: commit point\. EFE concentrates belief efficiently\. Info Gain over\-explores\.
##### Exploration efficiency\.

Figure[9](https://arxiv.org/html/2607.16981#Pt0.A12.F9)quantifies the temporal dynamics of epistemic foraging averaged across 300 episodes\. EFE reduces entropy at a rate comparable to Planning\+IG but commits earlier, avoiding the diminishing\-returns regime\. The “survival curve” of exploration shows EFE with a concentrated drop\-off around steps 8–12, while Planning\+IG’s survival curve extends further right\.

![Refer to caption](https://arxiv.org/html/2607.16981v1/x9.png)Figure 9:Exploration efficiency on Diagnosis \(N=8N\{=\}8,K=3K\{=\}3, 300 episodes\)\. \(a\) Belief entropy decay\. \(b\) Fraction of episodes still observing\. \(c\) Cumulative reward\. EFE commits at the right time\. Planning\+IG over\-explores\.

## Appendix 0\.MExtended EFE decomposition

![Refer to caption](https://arxiv.org/html/2607.16981v1/x10.png)Figure 10:EFE decomposition over an extended Diagnosis episode \(N=8N\{=\}8,K=3K\{=\}3\)\. \(a\) Value of committing vs\. observing, where the agent commits at the crossover\. \(b\) Per\-test expected information gain: tests become differentially informative as belief concentrates, and the agent selects the most informative test at each step\. \(c\) Belief entropy decay\. \(d\) Test selection sequence showing the agent’s adaptive strategy\.Figure[10](https://arxiv.org/html/2607.16981#Pt0.A13.F10)extends the EFE trajectory analysis of Figure[7](https://arxiv.org/html/2607.16981#Pt0.A7.F7)to a multi\-observation\-action environment where the agent must choose amongK=3K\{=\}3diagnostic tests at each step\. Panel \(b\) reveals a key dynamic: as the agent gathers information, the tests become differentially informative: some partitions of the state space become irrelevant once the agent has narrowed down the true condition, while others remain critical\. The EFE agent tracks this structure, selecting the most informative available test at each step \(panel d\)\. This adaptive test selection is the mechanism underlying EFE’s advantage over reward\-only planning in multi\-observation\-action environments\.

## Appendix 0\.NStopping time analysis

![Refer to caption](https://arxiv.org/html/2607.16981v1/x11.png)Figure 11:Distribution of episode lengths \(observation steps before commit\) across agents and environments\. Violin plots with overlaid data points\. EFE exhibits concentrated stopping times, committing with consistent timing once sufficient confidence is reached\. Planning commits earlier \(sometimes prematurely\), while Planning\+IG over\-explores with high variance\.Figure[11](https://arxiv.org/html/2607.16981#Pt0.A14.F11)shows the full distribution of stopping times \(the number of observation steps each agent takes before committing\) across three environment configurations\. The shape of these distributions reveals qualitative differences in decision strategy: EFE’s distributions are concentrated, indicating consistent and confident stopping behavior\. Planning’s distributions skew shorter, reflecting its tendency to commit before full disambiguation \(especially visible on theK=3K\{=\}3Diagnosis environment\)\. Planning\+IG shows long right tails, confirming that the additive weight drives over\-exploration with high variance\. The EFE agent’s concentrated stopping times are a consequence of the canonicalw=1w\{=\}1: the crossover between observation and commit values occurs at a consistent belief\-confidence level across episodes\.

## Appendix 0\.OSupplementary statistics

All pairwise comparisons use independent two\-samplett\-tests with Holm–Bonferroni correction for family\-wise error control\. Bootstrap 95% confidence intervals use 10,000 resamples with fixed seed for reproducibility\. Effect sizes use Cohen’sddwith pooled standard deviation\.

Table 19:Cohen’sddfor EFE vs\. each baseline on mean episode reward \(top\) and success rate \(bottom\)\. Positiveddfavors EFE\. Interpretation:\|d\|<0\.2\|d\|\{<\}0\.2negligible,0\.20\.2–0\.50\.5small,0\.50\.5–0\.80\.8medium,\|d\|\>0\.8\|d\|\{\>\}0\.8large\. Success\-ratedduses the same pooled\-ssconvention as reward \(proportions treated as per\-episode Bernoulli outcomes\)\.EnvironmentComparisondd\(reward\)InterpretationTigerEFE vs\. Myopic\+0\.44\+0\.44smallTigerEFE vs\. Planning−0\.01\-0\.01negligibleTigerEFE vs\. Info Gain\+0\.01\+0\.01negligibleTigerEFE vs\. Planning\+IG−0\.01\-0\.01negligibleTestbedEFE vs\. Myopic−0\.03\-0\.03negligibleTestbedEFE vs\. Planning−0\.19\-0\.19negligibleTestbedEFE vs\. Info Gain\+1\.34\+1\.34largeTestbedEFE vs\. Planning\+IG\+1\.33\+1\.33largeDiagnosisEFE vs\. Myopic\+0\.53\+0\.53mediumDiagnosisEFE vs\. Planning\+0\.07\+0\.07negligibleDiagnosisEFE vs\. Info Gain\+0\.24\+0\.24smallDiagnosisEFE vs\. Planning\+IG\+0\.23\+0\.23smallBanditEFE vs\. Myopic\+0\.21\+0\.21smallBanditEFE vs\. Planning\+0\.20\+0\.20negligibleBanditEFE vs\. Info Gain\+0\.67\+0\.67mediumBanditEFE vs\. Planning\+IG\+0\.79\+0\.79mediumSuccess rate\(ddfavors EFE\)TigerEFE vs\. Planning−0\.02\-0\.02negligibleTestbedEFE vs\. Planning\+0\.27\+0\.27smallDiagnosisEFE vs\. Planning\+0\.33\+0\.33smallBanditEFE vs\. Planning\+0\.47\+0\.47smallThe reward table shows that EFE vs\. Planning is negligible on Tiger, Diagnosis, and Bandit: once both agents take enough informative actions, mean returns are similar, even though success rates differ\. The success\-rate rows isolate the axis where EFE improves most relative to Planning on Diagnosis and Bandit\. Against over\-exploring tuned alternatives \(InfoGain\-Tuned, Planning\+IG\), EFE shows large reward effects on the Testbed \(d\>1\.3d\>1\.3\) and medium effects on Bandit \(d≈0\.7d\\approx 0\.7–0\.80\.8\), reflecting the cost of over\-exploration\.

## Appendix 0\.PNear\-optimality across planning horizons

![Refer to caption](https://arxiv.org/html/2607.16981v1/x12.png)Figure 12:Fraction of randomly generated two\-state environments wherew=1w\{=\}1is near\-optimal, as a function of planning horizonHH, stratified by reward asymmetryα\\alpha\. The near\-optimality basin widens with horizon, confirming that multi\-step planning amplifies the value of observation\. 100 environments,α∈\[1,50\]\\alpha\\in\[1,50\],p∈\[0\.55,0\.95\]p\\in\[0\.55,0\.95\], cost∈\[0\.1,5\]\\in\[0\.1,5\]\.
## Appendix 0\.QDiscount factor sensitivity

We test EFE and Planning agents’ sensitivity to discounting by sweepingγ∈\{0\.9,0\.95,0\.99,1\.0\}\\gamma\\in\\\{0\.9,0\.95,0\.99,1\.0\\\}across three environments\. Including Planning at eachγ\\gammaenables comparison of the relative gap between agents under discounting\.

Table 20:EFE and Planning agent performance across discount factorsγ\\gamma\(500 episodes each\)\.Δ\\DeltaSucc\. shows the success rate gap \(EFE−\-Planning\) at eachγ\\gamma\.##### Discounting erases EFE’s advantage on multi\-observation environments\.

On Tiger \(single observation action\), both agents achieve≥99%\{\\geq\}99\\%success across allγ\\gammavalues, with negligible gaps\. On Diagnosis and Bandit, heavy discounting \(γ=0\.9\\gamma\{=\}0\.9–0\.950\.95\) renders both agents effectively myopic, erasing EFE’s advantage: atγ=0\.9\\gamma\{=\}0\.9, EFE’s epistemic term cannot propagate value through future observations because discounting truncates the effective horizon below the number of observations needed for disambiguation\. Both agents observe∼2\{\\sim\}2times on Bandit and∼6\{\\sim\}6times on Diagnosis, behaving identically\. The transition betweenγ=0\.95\\gamma\{=\}0\.95andγ=0\.99\\gamma\{=\}0\.99is sharp: atγ=0\.99\\gamma\{=\}0\.99, EFE’s success gap over Planning jumps to\+7\.0\+7\.0pp on Diagnosis and\+22\.0\+22\.0pp on Bandit, matching the undiscounted pattern\. This confirms that EFE’s advantage depends on the discount factor preserving a sufficient effective horizon for recursive epistemic evaluation\. In practical terms,γ≥0\.99\\gamma\\geq 0\.99is needed for EFE to differentiate from reward\-only planning on these environments\.

## Appendix 0\.RModel misspecification sensitivity

Our main results assume the agent’s generative model matches the true environment dynamics\. Here we investigate robustness when the agent’s believed observation accuracypagentp\_\{\\text\{agent\}\}differs from the true accuracyptruep\_\{\\text\{true\}\}, creating a systematic model mismatch\.

Table 21:Model misspecification on Tiger \(ptrue=0\.85p\_\{\\text\{true\}\}\{=\}0\.85,H=4H\{=\}4, 2,500 episodes across 5 seeds\)\. The mismatch column showspagent−ptruep\_\{\\text\{agent\}\}\-p\_\{\\text\{true\}\}\.Agentpagentp\_\{\\text\{agent\}\}MismatchObsSuccessRewardEFE0\.70−0\.15\-0\.154\.2799\.4%\+5\.11\+5\.11EFE0\.80−0\.05\-0\.054\.1499\.6%\+5\.46\+5\.46EFE0\.85±0\.00\\pm 0\.004\.2899\.6%\+5\.32\+5\.32EFE0\.90\+0\.05\+0\.052\.6896\.7%\+3\.66\+3\.66EFE0\.95\+0\.10\+0\.102\.6697\.2%\+4\.30\+4\.30Planning0\.70−0\.15\-0\.154\.1999\.4%\+5\.20\+5\.20Planning0\.85±0\.00\\pm 0\.004\.2499\.6%\+5\.36\+5\.36Planning0\.95\+0\.10\+0\.102\.7497\.0%\+3\.92\+3\.92Table 22:Model misspecification on Diagnosis \(ptrue=0\.80p\_\{\\text\{true\}\}\{=\}0\.80,H=3H\{=\}3, 2,500 episodes across 5 seeds\)\.Agentpagentp\_\{\\text\{agent\}\}MismatchObsSuccessRewardEFE0\.65−0\.15\-0\.155\.8389\.1%−2\.36\-2\.36EFE0\.75−0\.05\-0\.059\.6097\.5%−1\.11\-1\.11EFE0\.80±0\.00\\pm 0\.009\.7897\.4%−1\.36\-1\.36EFE0\.85\+0\.05\+0\.055\.9288\.8%−2\.62\-2\.62EFE0\.90\+0\.10\+0\.105\.9289\.0%−2\.52\-2\.52Planning0\.65−0\.15\-0\.155\.7888\.6%−2\.65\-2\.65Planning0\.80±0\.00\\pm 0\.005\.8389\.0%−2\.40\-2\.40Planning0\.90\+0\.10\+0\.105\.8087\.9%−3\.04\-3\.04##### Key findings\.

On Tiger, both EFE and Planning agents are remarkably robust: success rates remain above 96\.7% even with±0\.15\\pm 0\.15mismatch\. The degradation pattern is asymmetric: underestimating accuracy \(negative mismatch\) causes over\-observation but preserves safety, while overestimating accuracy \(positive mismatch\) reduces observation count and lowers success rates\. EFE’s inherent epistemic drive provides a buffer against over\-confident models: when the agent overestimates its sensor accuracy, the information gain term still encourages observation, partially compensating for the miscalibrated planning model\.

On Diagnosis, degradation is graceful but measurable\. Overestimating accuracy \(the agent believes tests are more informative than they truly are\) leads to premature commitment with insufficient evidence\. Underestimating accuracy causes excessive testing, reducing reward through observation costs but not drastically harming success rates\. The EFE agent degrades gracefully across the mismatch range, suggesting that the intrinsic epistemic value provides a degree of robustness to model misspecification that pure reward\-maximizing agents lack\.

##### Implications for learned models\.

These results suggest that EFE\-based agents can tolerate moderate model errors without catastrophic failure\. In practice, observation models learned from data will have estimation error\. The graceful degradation observed here indicates that approximate models suffice for effective EFE\-driven information gathering, provided the model does not systematically overestimate sensor informativeness\.

## Appendix 0\.SInterleaved observe\-act: RockSample \(extended\)

This appendix provides extended RockSample results beyond the main text \(Section[5\.4](https://arxiv.org/html/2607.16981#S5.SS4)\), including POMCP baselines and per\-instance detail\. The environment and agents are described in Section[5\.4](https://arxiv.org/html/2607.16981#S5.SS4)\.

Table 23:RockSample extended results \(500 episodes each\), including POMCP baseline and heuristic agents\.The Greedy agent moves to rocks and samples them without checking quality first, incurring frequent bad\-rock penalties \(1\.50 bad samples on RS\[5,3\], 2\.00 on RS\[7,4\]\)\. The EFE agent moves toward uncertain rocks, checks them to resolve quality, then samples only those confirmed good\. The result is dramatically fewer bad samples \(0\.330\.33and0\.430\.43\) and substantially higher reward \(\+8\.63\+8\.63vs\.\+4\.03\+4\.03on RS\[5,3\] and\+8\.20\+8\.20vs\.−0\.58\-0\.58on RS\[7,4\]\)\.

We additionally include Planning\+IG \(w=5w\{=\}5,w=10w\{=\}10\) baselines to address concerns about weak baselines\. Planning\+IG uses the same factored belief and decision structure as EFE but with an explicit, tunable weight\. On RS\[5,3\], Planning\+IG atw=5w\{=\}5andw=10w\{=\}10matches EFE \(w=1w\{=\}1\) closely \(\+8\.64\+8\.64vs\.\+8\.63\+8\.63reward\)\. On RS\[7,4\], EFE \(w=1w\{=\}1\) substantially outperforms Planning\+IG \(\+8\.20\+8\.20vs\.\+5\.57\+5\.57\), suggesting that the canonical weight is better calibrated on the larger instance where information gathering requires more nuanced valuation\.

These results demonstrate that the EFE principle of valuing information gain alongside reward extends to interleaved settings despite the transition–observation coupling analyzed in Section[3](https://arxiv.org/html/2607.16981#S3)\(factored observation extension\)\. The agent’s behavior mirrors the observe\-then\-commit pattern: check \(gather information\), then sample/exit \(commit\), even though checking and acting are interleaved with movement\.

## Appendix 0\.TPOMCP baseline comparison

We compare against POMCP\[[39](https://arxiv.org/html/2607.16981#bib.bib39)\], a standard online POMDP solver that handles exploration through Monte Carlo tree search with UCB1 action selection\. This addresses whether EFE’s explicit information gain valuation provides benefit over the implicit exploration in POMCP’s tree search\. Our implementation uses particle\-based belief representation at each tree node, adapted to the observe\-then\-commit action structure\. Crucially, our POMCP rollout policy is semi\-informed: observation actions are selected uniformly at random, but the rollout terminates \(stochastically, with 30% probability per step\) with a belief\-optimal commit, i\.e\., the commit action maximizing expected reward under the Bayesian\-updated rollout belief\. This is strictly stronger than a fully random rollout, which would also randomize over commit actions\. The remaining gap between POMCP and EFE therefore reflects the value of observation selection: EFE chooses which test to run based on information gain, while POMCP samples tests uniformly\. We sweep POMCP simulation budgets across\{500,1000,2000,5000\}\\\{500,1000,2000,5000\\\}and report wall\-clock timings for compute\-matched analysis\.

Table 24:POMCP comparison \(5,000 total episodes across 5 seeds for Tiger/Diagnosis/Bandit and 1,000 for Tileworld\)\. POMCP uses 1,000 simulations per decision\.##### Key findings\.

POMCP underperforms both EFE and Planning on all environments except Bandit success rate\. On Tiger, POMCP commits prematurely \(1\.67 observations vs\. 4\.29 for EFE\), yielding 89\.3% success\. On Diagnosis, the gap is dramatic: POMCP achieves only 73\.1% success \(−10\.04\-10\.04reward\) while EFE reaches 96\.6% \(−1\.80\-1\.80reward\)\. POMCP’s random rollout policy cannot evaluate which of theK=2K\{=\}2diagnostic tests to run\. It treats all observation actions equally, whereas EFE’s information gain term directly values differential informativeness\. On Bandit, POMCP’s Monte Carlo exploration achieves 96\.8% success but at extreme reward cost \(\+2\.51\+2\.51vs\. EFE’s\+6\.42\+6\.42\): 14\.41 inspections vs\. EFE’s 5\.19\. On Tileworld \(6×66\{\\times\}6,\|S\|=36\|S\|\{=\}36\), POMCP collapses to 6\.1% success, as the 36 commit actions and 6 scan actions create a branching factor that overwhelms 1,000 simulations\.

##### Compute\-matched analysis\.

Increasing POMCP’s simulation budget to 5,000 does not qualitatively change the picture: on Diagnosis, POMCP\(5000\) reaches 71\.5% success, still far short of EFE’s 96\.6%, while taking substantially more wall\-clock time per decision \(299\.5 ms/ep vs\. EFE’s closed\-form computation\)\. On Tiger, POMCP\(5000\) reaches 89\.0% but at 5×\\timesthe compute\. These results demonstrate that EFE’s advantage is not merely a compute\-budget artifact but reflects the fundamental benefit of closed\-form information gain valuation over Monte Carlo exploration, particularly on multi\-observation\-action environments where the agent must choose which information to gather\.

##### MCTS\-EFE\.

To disentangle EFE’s information valuation from the planning mechanism, we implement MCTS\-EFE: MCTS with EFE as the leaf heuristic rather than random rollouts\. On Tiger atH=10H\{=\}10, MCTS\-EFE with 500 simulations achieves 97\.2% success \(7\.2s per 200 episodes\), compared to POMCP’s 89\.7% at matched budget \(1\.7s\) and exact EFE’s 99\.5% atH=6H\{=\}6\(11\.8s\)\. This confirms that EFE provides a superior leaf evaluation function: the same MCTS framework with EFE rollouts substantially outperforms random rollouts\.