Not all uncertainty is alike: volatility, stochasticity, and exploration

# Not all uncertainty is alike: volatility, stochasticity, and exploration
Source: [https://arxiv.org/html/2605.19215](https://arxiv.org/html/2605.19215)
###### Abstract

Adaptive decision\-making in biological and artificial intelligence requires balancing the exploitation of known outcomes with the exploration of uncertain alternatives\. Although prior work suggests that uncertainty generally promotes exploration, it has typically treated distinct sources of environmental uncertainty as equivalent\. We consider environments with latent reward states that drift over time \(volatility\) and are observed through noisy outcomes \(stochasticity\)\. Both increase posterior uncertainty, yet we show they drive optimal exploration in opposite directions: volatility enhances it, stochasticity suppresses it\. We establish this asymmetry formally by extending the Gittins index framework to Gaussian state\-space bandits with latent dynamics\. We further derive Cause\-Aware Uncertainty\-Sensitive Exploration \(CAUSE\), a closed\-form exploration bonus obtained via control\-as\-inference that inherits the same monotonicities\. CAUSE outperforms standard exploration strategies in environments with heterogeneous noise structure, and also improves on a Gittins\-per\-arm policy whose rested\-bandit optimality does not transfer to restless settings\. Learning and exploration are governed by the same noise\-inference asymmetry, and the framework predicts that pathological noise inference produces*reversed*rather than merely impaired exploration, with implications for computational accounts of psychiatric conditions\.

## 1Introduction

Across computational accounts of exploration, a recurring principle is that uncertainty drives behavior: greater uncertainty, more exploration\. This logic underlies a broad family of exploration strategies: upper confidence bound \(UCB\) methods explore by acting optimistically under uncertainty\[[2](https://arxiv.org/html/2605.19215#bib.bib53),[18](https://arxiv.org/html/2605.19215#bib.bib52)\]; Thompson sampling explores by sampling from the posterior over rewards\[[34](https://arxiv.org/html/2605.19215#bib.bib51)\]; information\-directed sampling balances expected regret against expected information gain\[[31](https://arxiv.org/html/2605.19215#bib.bib50)\]; and neuroscience work links exploration to neural and behavioral signatures of uncertainty\[[38](https://arxiv.org/html/2605.19215#bib.bib49),[12](https://arxiv.org/html/2605.19215#bib.bib48),[8](https://arxiv.org/html/2605.19215#bib.bib47),[10](https://arxiv.org/html/2605.19215#bib.bib46)\]\. In each case, more uncertainty prescribes more exploration\.

We show that this prescription is incomplete\. Not all uncertainty is alike, and some forms of uncertainty should*decrease*exploration\. Specifically, we consider environments with latent reward states that change over time \(volatility\) and are observed through noisy outcomes \(stochasticity\)\. Both volatility and stochasticity increase posterior uncertainty about rewards, and standard methods such as UCB and Thompson sampling treat them equivalently, prescribing more exploration in both cases\. We prove that they drive optimal exploration in opposite directions: volatility enhances exploration, while stochasticity suppresses it\.

The intuition is that uncertainty matters for exploration only insofar as it reflects potential information gain\. Volatility creates information gain: latent states are changing, so new observations reveal something new\. Stochasticity destroys it: observations are noisy, so each sample is less informative\. An agent that simply explores more whenever it is more uncertain conflates these two cases, exploring vigorously precisely when observations are least useful\.

We establish this result formally by extending the Gittins index framework from independent and identically distributed \(iid\) Gaussian bandits to Gaussian state\-space bandits, where each arm has a latent reward state that evolves with volatility and is observed with stochasticity\. In this richer setting, we prove that the optimal exploration bonus, defined as the value of information that justifies pulling an arm beyond its expected reward, is monotonically increasing in volatility and decreasing in stochasticity\. This result follows from the structure of the Gittins retirement problem and requires no approximations\.

The Gittins index is provably optimal only for rested bandits\. In restless settings, where unselected arms also evolve, Gittins loses its optimality guarantee and the truly optimal policy is intractable in general\. To obtain a practical exploration strategy, we turn to the control\-as\-inference framework\[[19](https://arxiv.org/html/2605.19215#bib.bib43),[35](https://arxiv.org/html/2605.19215#bib.bib42)\], which casts action selection as posterior inference in a probabilistic graphical model\. Within this framework, we derive*Cause\-Aware Uncertainty\-Sensitive Exploration*\(CAUSE\), a closed\-form index policy for Gaussian state\-space bandits\. CAUSE decomposes cleanly into exploitation and exploration components, and its exploration component inherits the same qualitative dependence on volatility and stochasticity established by the Gittins analysis\. To our knowledge, this is the first application of control\-as\-inference to derive an exploration index for restless bandits\.

These opposing effects parallel a known result in the learning literature\. Optimal learning rates increase with volatility and decrease with stochasticity, because the same information\-theoretic logic applies: volatility makes new observations more relevant, stochasticity makes them less reliable\[[26](https://arxiv.org/html/2605.19215#bib.bib45)\]\. The parallel is not coincidence: both are normative properties of the same generative model\.

This unified view yields a testable prediction about pathological behavior\. Volatility and stochasticity must be jointly inferred from observations, since both increase experienced noise and can only be disentangled by explaining away\. A priori insensitivity to one source leads to systematic misattribution of the other\[[26](https://arxiv.org/html/2605.19215#bib.bib45)\]: an agent with hyposensitive stochasticity priors sees the world as more volatile than it is, and vice versa\. In the learning setting, such miscalibration produces*reversed*learning\-rate modulation, a pattern observed in human behavior\[[27](https://arxiv.org/html/2605.19215#bib.bib44)\]\. We show that the analogous failure in exploration produces reversed exploratory behavior: stochasticity\-blind agents over\-explore precisely when observations are least informative, while volatility\-blind agents under\-explore precisely when the environment is changing fastest\. Miscalibrated noise inference has been proposed as a candidate mechanism in several psychiatric conditions, making these reversals a concrete behavioral target for clinical investigation\.

Our contributions are: \(i\) we extend the Gittins framework from iid Gaussian bandits to Gaussian state\-space bandits with latent dynamics, proving that the optimal exploration bonus increases monotonically with volatility and decreases monotonically with stochasticity; \(ii\) within the control\-as\-inference framework, we derive CAUSE, a closed\-form index policy for Gaussian state\-space bandits, which to our knowledge is the first closed\-form index for restless bandits with continuous\-state Gaussian dynamics and the first application of control\-as\-inference to derive such an index; \(iii\) we show that the opposing effects of volatility and stochasticity on exploration parallel those on learning, both following from the normative properties of the same generative model; \(iv\) empirically, CAUSE outperforms Thompson sampling, UCB, predictive sampling, and a numerical Gittins\-per\-arm baseline across volatility and stochasticity regimes, and matches the optimal Gittins reference in the rested limit; \(v\) lesioned inference produces reversed, not merely impaired, patterns of exploration, with the direction of reversal diagnostic of which latent inference has failed\. Code will be released upon publication\.

## 2Related work

#### Exploration and uncertainty in neuroscience\.

Behavioral and neural studies have linked exploratory behavior to uncertainty signals, including relative and total uncertainty\[[38](https://arxiv.org/html/2605.19215#bib.bib49)\], expected and unexpected uncertainty\[[7](https://arxiv.org/html/2605.19215#bib.bib14)\], and information value\[[8](https://arxiv.org/html/2605.19215#bib.bib47),[11](https://arxiv.org/html/2605.19215#bib.bib37)\]\. These accounts generally predict that exploration increases with uncertainty, without distinguishing between the sources of that uncertainty\. Our framework predicts that volatility and stochasticity should have opposing effects on exploratory behavior, and that selective failures of inference about these quantities should produce characteristic reversals rather than uniform deficits\. To our knowledge, these qualitative predictions have not been tested experimentally\.

#### Volatility, stochasticity, and adaptive learning\.

Adaptive learning under uncertainty has been modeled using hierarchical Bayesian frameworks that track volatility\[[4](https://arxiv.org/html/2605.19215#bib.bib36),[22](https://arxiv.org/html/2605.19215#bib.bib35)\]or detect change\-points\[[24](https://arxiv.org/html/2605.19215#bib.bib34),[23](https://arxiv.org/html/2605.19215#bib.bib4)\]\. These approaches typically estimate one source of uncertainty while treating the other as known, overlooking the computational challenge of their simultaneous estimation\.Piray and Daw \[[26](https://arxiv.org/html/2605.19215#bib.bib45)\]introduced a framework for jointly estimating volatility and stochasticity, showing that although both increase outcome variance, they can be distinguished through temporal structure and exert opposing effects on learning rate\. The same work showed that selective failures to represent one source produce reversed, rather than merely impaired, learning\-rate modulation, offering a candidate account of pathological learning in psychiatric and neurological conditions\. This framework was extended to binary outcomes and validated in large\-scale behavioral experiments\[[27](https://arxiv.org/html/2605.19215#bib.bib44)\]\. Our work builds on this foundation, showing that the same dissociation drives exploration as a normative result, with analogous lesion\-model predictions for exploratory behavior\.

#### Exploration in bandits\.

The explore/exploit dilemma has been studied extensively in the multi\-armed bandit literature\. UCB methods\[[2](https://arxiv.org/html/2605.19215#bib.bib53),[18](https://arxiv.org/html/2605.19215#bib.bib52)\]and Thompson sampling\[[34](https://arxiv.org/html/2605.19215#bib.bib51),[30](https://arxiv.org/html/2605.19215#bib.bib9)\]both increase exploration with uncertainty, the former through optimistic reward estimates and the latter through posterior sampling; a Bayesian variant of UCB\[[17](https://arxiv.org/html/2605.19215#bib.bib6)\]replaces frequentist confidence bounds with posterior credible intervals, the version we use as a baseline in our experiments\. The Gittins index\[[13](https://arxiv.org/html/2605.19215#bib.bib41)\]provides an optimal decomposition for discounted rested bandits, and is well characterized for Gaussian iid rewards with an unknown mean estimated from noisy observations\[[39](https://arxiv.org/html/2605.19215#bib.bib39)\]\. Whittle’s index\[[37](https://arxiv.org/html/2605.19215#bib.bib40)\]extends index\-based policies to restless settings where latent states evolve over time, though it is typically computable only through approximation, and exact optimal policies for restless bandits are PSPACE\-hard\[[25](https://arxiv.org/html/2605.19215#bib.bib3)\]\. Information\-directed sampling\[[31](https://arxiv.org/html/2605.19215#bib.bib50)\]explicitly targets uncertainty reduction, allocating exploration to maximize information gain per unit of regret; unlike index\-based approaches, however, it requires solving an auxiliary optimization problem at each step and does not yield comparative statics in closed form\.Russo and Van Roy \[[32](https://arxiv.org/html/2605.19215#bib.bib24)\]propose satisficing Thompson sampling for time\-sensitive bandit learning, which samples against a relaxed near\-optimal target rather than the exact optimum, andLiuet al\.\[[20](https://arxiv.org/html/2605.19215#bib.bib23)\]extend Thompson sampling to bandits with mean\-reverting non\-stationary rewards via predictive sampling, which down\-weights information that loses value due to drift\. Both extend the posterior\-sampling framework rather than producing closed\-form indices\. Predictive sampling is the closer methodological neighbor to our work, but it operates in a different setting \(mean\-reverting AR\(1\) rewards\) and does not yield the closed\-form asymmetry between volatility and stochasticity that we establish for random\-walk dynamics\. We extend the Gittins analysis to Gaussian state\-space bandits with latent random\-walk dynamics, and show that the relationship between uncertainty and optimal exploration is not monotonic: volatility promotes exploration while stochasticity suppresses it\. We further derive CAUSE, a tractable closed\-form index policy that inherits this structure\.

#### Control as inference\.

The connection between optimal control and probabilistic inference has a long history\[[35](https://arxiv.org/html/2605.19215#bib.bib42),[16](https://arxiv.org/html/2605.19215#bib.bib31),[19](https://arxiv.org/html/2605.19215#bib.bib43)\]\. In this framework, optimal actions are obtained by conditioning on high reward in a probabilistic graphical model, yielding policies that are regularized by their divergence from a prior policy\. This formulation has been applied in reinforcement learning\[[1](https://arxiv.org/html/2605.19215#bib.bib29),[14](https://arxiv.org/html/2605.19215#bib.bib28)\], robot planning\[[36](https://arxiv.org/html/2605.19215#bib.bib1)\], and human planning\[[5](https://arxiv.org/html/2605.19215#bib.bib27)\]\. We leverage this framework to derive a tractable closed\-form index for restless bandits with volatility and stochasticity, a setting where the Gittins index characterizes optimal behavior but does not yield a computable policy\.

#### Pathological inference in computational psychiatry\.

Miscalibrated learning\-rate adjustment in response to volatility has been proposed as a candidate mechanism in several psychiatric conditions\. In anxiety, overestimation of volatility or environmental instability has been linked to inappropriately high learning rates and over\-reactivity to noise\[[6](https://arxiv.org/html/2605.19215#bib.bib21),[15](https://arxiv.org/html/2605.19215#bib.bib22)\], although a recent study has produced inconsistent results\[[33](https://arxiv.org/html/2605.19215#bib.bib16)\]\. In psychosis, miscalibrated precision weighting has been linked to delusional inference\[[28](https://arxiv.org/html/2605.19215#bib.bib19)\]\. In depression, insensitivity to environmental change has been linked to impaired updating of beliefs about reward\[[29](https://arxiv.org/html/2605.19215#bib.bib17)\]\. Our framework contributes to this program by providing the exploration\-side analogue, with the reversal pattern characterized in Section[6\.4](https://arxiv.org/html/2605.19215#S6.SS4)\.

## 3Problem setting

We consider a restless multi\-armed bandit in which each arm’s reward is driven by an independent latent statextx\_\{t\}that drifts over time as a Gaussian random walk with innovation variancev\>0v\>0\(*volatility*\)\. When pulled at timett, the arm yields a noisy rewardrt=xt\+ϵtr\_\{t\}=x\_\{t\}\+\\epsilon\_\{t\}withϵt∼𝒩​\(0,s\)\\epsilon\_\{t\}\\sim\\mathcal\{N\}\(0,s\)and observation noise variances\>0s\>0\(*stochasticity*\)\. The innovation and observation noises are independent across time, and the prior isx0∼𝒩​\(m0,P0\)x\_\{0\}\\sim\\mathcal\{N\}\(m\_\{0\},\\,P\_\{0\}\)\. The setting is restless: all arms drift regardless of whether they are pulled\. The agent’s goal is to maximize the expected discounted return𝔼​\[∑t=1∞γt−1​rt\]\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=1\}^\{\\infty\}\\gamma^\{t\-1\}r\_\{t\}\\right\], whereγ∈\(0,1\)\\gamma\\in\(0,1\)\. Throughout Sections[4](https://arxiv.org/html/2605.19215#S4)and[5](https://arxiv.org/html/2605.19215#S5),vvandssare treated as known parameters of each arm; Section[6\.4](https://arxiv.org/html/2605.19215#S6.SS4)considers the case in which they are inferred from observations\.

To make informed decisions, the agent must track the latent states from noisy observations\. The posterior over each arm’s state,p​\(xt∣r1:t\)=𝒩​\(mt,Pt\)p\(x\_\{t\}\\mid r\_\{1:t\}\)=\\mathcal\{N\}\(m\_\{t\},\\,P\_\{t\}\), is updated via the Kalman filter:

mt\\displaystyle m\_\{t\}=mt−1\+Kt​\(rt−mt−1\),\\displaystyle=m\_\{t\-1\}\+K\_\{t\}\\,\(r\_\{t\}\-m\_\{t\-1\}\),\(1\)Pt\\displaystyle P\_\{t\}=\(1−Kt\)​\(Pt−1\+v\),\\displaystyle=\(1\-K\_\{t\}\)\(P\_\{t\-1\}\+v\),\(2\)whereKt=\(Pt−1\+v\)/\(Pt−1\+v\+s\)K\_\{t\}=\(P\_\{t\-1\}\+v\)/\(P\_\{t\-1\}\+v\+s\)is the Kalman gain\. For arms not pulled,mt=mt−1m\_\{t\}=m\_\{t\-1\}andPt=Pt−1\+vP\_\{t\}=P\_\{t\-1\}\+v: the mean is unchanged but uncertainty grows byvvat every time step\.

## 4Volatility, stochasticity, and optimal exploration

We now establish the paper’s central theoretical claims: that the exploration bonus depends oppositely on stochasticity and volatility\. We first set up the retirement problem and decompose the index, then prove monotonicity in each direction\.

### 4\.1Exploration bonus via the retirement problem

To quantify the value of exploring a single arm, we consider a classical thought experiment\[[13](https://arxiv.org/html/2605.19215#bib.bib41)\]\. An agent faces a single arm and, at each period, chooses between pulling it \(receiving a noisy rewardrtr\_\{t\}and updating her belief via the Kalman filter\) or*retiring*to collect a fixed salaryλ\\lambdain perpetuity\. The agent may pull as many times as she wishes before retiring, but cannot return once she leaves\. The*index*of the arm is the break\-even salary: the smallestλ\\lambdaat which immediate retirement is optimal\. Intuitively, this is the agent’s indifference price for giving up future access to the arm, and therefore measures the arm’s total value\. Formally,

λ1−γ=supτ≥0𝔼​\[∑n=1τγn−1​rn\+γτ​λ1−γ\],\\frac\{\\lambda\}\{1\-\\gamma\}\\;=\\;\\sup\_\{\\tau\\geq 0\}\\;\\mathbb\{E\}\\\!\\left\[\\;\\sum\_\{n=1\}^\{\\tau\}\\gamma^\{n\-1\}\\,r\_\{n\}\\;\+\\;\\gamma^\{\\tau\}\\,\\frac\{\\lambda\}\{1\-\\gamma\}\\;\\right\],\(3\)where the supremum is over all stopping timesτ\\tauadapted to the observations\. This construction isolates the value of a single arm, independent of the rest of the bandit\. In rested bandits, pulling the arm with the highest index is optimal\[[13](https://arxiv.org/html/2605.19215#bib.bib41)\]; in the restless setting, the retirement\-problem index remains a well\-defined per\-arm quantity, and its dependence onssandvvis the subject of our theoretical analysis\.

The Gaussian structure of the model permits a decomposition of the index into exploitation and exploration components\.

###### Proposition 1\(Index Decomposition\)\.

The indexλ\\lambdaadmits the decomposition:

λ​\(m,P,s,v,γ\)=m\+B​\(P,s,v,γ\),\\lambda\(m,\\,P,\\,s,\\,v,\\,\\gamma\)=m\+B\(P,\\,s,\\,v,\\,\\gamma\),\(4\)whereB​\(P,s,v,γ\)≔λ​\(0,P,s,v,γ\)≥0B\(P,s,v,\\gamma\)\\coloneqq\\lambda\(0,P,s,v,\\gamma\)\\geq 0is theexploration bonus: the option value of the information gained by pulling\.

Proven in Appendix[A](https://arxiv.org/html/2605.19215#A1)via shift\-coupling of the latent state, the decomposition isolates an exploration termBBthat depends only on the uncertainty structure\. The rest of this section analyzesBB\.

For iid Gaussian bandits with an unknown mean and known observation noise,Yao \[[39](https://arxiv.org/html/2605.19215#bib.bib39)\]established that the exploration bonus is nonincreasing in the observation variance and nondecreasing in the prior variance\. This setting is thev=0v=0special case of our model\. We show below that both monotonicities continue to hold once drift is introduced, and we establish a third monotonicity invvitself, which has no iid counterpart\. The opposing effects ofssandvvon optimal exploration, which is the paper’s central structural claim, cannot be drawn within iid models\.

### 4\.2Optimal exploration decreases with stochasticity

###### Theorem 2\(Monotonicity inss\)\.

The exploration bonusB​\(P,s,v,γ\)B\(P,s,v,\\gamma\)is nonincreasing in the observation noisess\.

The proof, given in Appendix[A](https://arxiv.org/html/2605.19215#A1), proceeds by coupling: a noisier arm is constructed from a cleaner one by adding independent Gaussian noise to each observation\. Any stopping rule based on noisy observations can be simulated by the clean\-arm agent \(who can draw the extra noise internally\), but not conversely, so the noisy agent optimizes over a strictly smaller class of decision rules\. Since the two arms share the same latent state and reward process, both agents are optimizing the same objective; the noisy agent simply has access to fewer decision rules, so the supremum is no larger, and the monotonicity follows\. Because stochasticity enters only through the observation channel, which is active only when the arm is pulled, this argument does not rely on whether unselected arms drift; the result holds in both rested and restless settings\.

### 4\.3Optimal exploration increases with volatility

###### Theorem 3\(Monotonicity invv\)\.

The exploration bonusB​\(P,s,v,γ\)B\(P,s,v,\\gamma\)is nondecreasing in the innovation variancevv\.

Where stochasticity degrades the signal, volatility creates opportunity\. The proof, given in Appendix[A](https://arxiv.org/html/2605.19215#A1), rests on a supporting lemma that is of independent interest:

###### Lemma 4\(Monotonicity inPP\)\.

The exploration bonusB​\(P,s,v,γ\)B\(P,s,v,\\gamma\)is nondecreasing in the posterior variancePP\.

The mechanism is convexity\. Higher posterior variancePPproduces a wider distribution over the next\-period posterior mean, because more of the remaining uncertainty will be resolved by the next observation\. Since the agent can exploit favorable posterior means and retire to avoid unfavorable ones, the value function is convex in the posterior mean, and Jensen’s inequality converts wider spread into higher expected value\. More uncertainty about the arm therefore yields a larger option value of pulling\.

Volatility enters this logic through the belief dynamics\. Each unit ofvvadds directly to the next\-period posterior variance \(P′P^\{\\prime\}grows withvv\) and, by Lemma[4](https://arxiv.org/html/2605.19215#Thmtheorem4), raises the bonus\. The effect compounds over time: in a volatile environment, each future pull operates on an arm whose uncertainty has been replenished by drift, so the option value of pulling is sustained rather than decaying toward zero\.

## 5CAUSE: a closed\-form index via control\-as\-inference

The Gittins analysis of Section[4](https://arxiv.org/html/2605.19215#S4)characterizes optimal exploration but does not yield a computable policy for the restless setting\. We now derive CAUSE, a tractable closed\-form index for Gaussian state\-space bandits, by casting action selection as posterior inference under an optimality constraint within the control\-as\-inference framework\[[19](https://arxiv.org/html/2605.19215#bib.bib43),[35](https://arxiv.org/html/2605.19215#bib.bib42)\]\. The derivation is methodologically distinct from standard routes to exploration bonuses \(optimism, posterior sampling, dynamic programming on belief\-state MDPs\) and yields, to our knowledge, the first closed\-form index for a natural restless bandit setting\. The complete derivation is provided in Appendix[B](https://arxiv.org/html/2605.19215#A2); here we summarize the setup, state the closed\-form result, and discuss its structural properties\.

### 5\.1Control as inference for the state\-space bandit

Following the control\-as\-inference framework, we place a binary optimality variableot∈\{0,1\}o\_\{t\}\\in\\\{0,1\\\}at each time step with

p​\(ot=1∣rt\)=σ​\(γt−1​rt\),p\(o\_\{t\}=1\\mid r\_\{t\}\)=\\sigma\(\\gamma^\{t\-1\}\\,r\_\{t\}\),\(5\)whereσ\\sigmais the logistic sigmoid and the discountγt−1\\gamma^\{t\-1\}encodes the diminishing importance of future rewards\. The agent’s action preference is governed by the posteriorp​\(xt∣ot:∞=1,r1:t−1\)p\(x\_\{t\}\\mid o\_\{t:\\infty\}=1,r\_\{1:t\-1\}\), which weighs the predictive belief aboutxtx\_\{t\}against the backward messagebt​\(xt\)≡p​\(ot:∞=1∣xt\)b\_\{t\}\(x\_\{t\}\)\\equiv p\(o\_\{t:\\infty\}=1\\mid x\_\{t\}\)carrying information from future optimality constraints\. Exploration arises not from an externally added bonus but from this posterior: states that make future optimality more likely are upweighted, and the corresponding arms preferred\.

A note on what the inference reformulation buys: in the optimality formulation, restless bandits are notoriously difficult because optimal actions depend on the joint state of all arms\[[37](https://arxiv.org/html/2605.19215#bib.bib40)\]\. Under the inference formulation, this difficulty does not arise: each arm’s posterior under future optimality depends only on its own latent process and backward message, and action selection reduces to comparing per\-arm scalars\. The restless structure, hard in the optimality formulation, becomes per\-arm in the inference formulation, which is what makes a closed\-form index possible\.

### 5\.2The closed\-form index

The backward message admits a closed\-form recursion in the Gaussian state\-space setting\. We restrict the backward message to a sigmoid ansatzbt​\(xt\)≈σ​\(αt​xt\)b\_\{t\}\(x\_\{t\}\)\\approx\\sigma\(\\alpha\_\{t\}x\_\{t\}\), natural because the local optimality likelihood is itself sigmoid \(Eq\.[5](https://arxiv.org/html/2605.19215#S5.E5)\), and track the coefficientαt\\alpha\_\{t\}through the recursion using probit approximations for the Gaussian\-sigmoid convolutions involved \(Appendix[B](https://arxiv.org/html/2605.19215#A2)\)\. This yields an infinite\-horizon precisionSSthat depends transparently onγ\\gamma,ss, andvv:

S≈2\(ln⁡D\+1−γ\)​\(1\+1\+ϕ​s\),D=12​\(1\+1\+ϕ​v​\(α∗\)2\),S\\approx\\frac\{2\}\{\(\\ln D\+1\-\\gamma\)\\bigl\(1\+\\sqrt\{1\+\\phi\\,s\}\\bigr\)\},\\qquad D=\\tfrac\{1\}\{2\}\\bigl\(1\+\\sqrt\{1\+\\phi\\,v\\,\(\\alpha^\{\*\}\)^\{2\}\}\\bigr\),\(6\)withϕ=π/8\\phi=\\pi/8andα∗=2/\[\(1−γ\)​\(1\+1\+ϕ​s\)\]\\alpha^\{\*\}=2/\[\(1\-\\gamma\)\(1\+\\sqrt\{1\+\\phi s\}\)\]the undamped \(v=0v=0\) limit\. The structure is interpretable: stochasticityssdampsSSthrough the1\+ϕ​s\\sqrt\{1\+\\phi s\}factor, while volatilityvventers throughDD, which shortens the effective planning horizon by attenuating the backward flow of precision\. CombiningSSwith the predictive belief and approximating the resulting skew\-normal posterior mean \(Appendix[B](https://arxiv.org/html/2605.19215#A2)\) gives the CAUSE index:

λCAUSE=m\+c\(P\+v\)α~,\\boxed\{\\;\\lambda\_\{\\mathrm\{CAUSE\}\}=m\+c\\,\(P\+v\)\\,\\tilde\{\\alpha\},\\;\}\(7\)where

α~=S1\+ϕ​\(P\+v\)​S2,\\tilde\{\\alpha\}=\\frac\{S\}\{\\sqrt\{1\+\\phi\\,\(P\+v\)\\,S^\{2\}\}\},\(8\)mmandPPare the current Kalman posterior moments, andc∈\(0,1\)c\\in\(0,1\)is a tunable scale parameter\. The agent selects the arm with the largestλCAUSE\\lambda\_\{\\mathrm\{CAUSE\}\}\. The parametercccontrols the overall magnitude of the exploration bonus and does not affect its dependence onssorvv; we fixc=1/2c=1/2throughout this paper, a value that emerges naturally from one route through the derivation \(Appendix[B](https://arxiv.org/html/2605.19215#A2)\)\.

### 5\.3Structure of the optimal bonus

StochasticityssdampsSSthrough the\(1\+1\+ϕ​s\)−1\(1\+\\sqrt\{1\+\\phi s\}\)^\{\-1\}factor, soSS,α~\\tilde\{\\alpha\}, and the bonus all decrease withss\. In the high\-volatility \(or high\-PP\) limit\(P\+v\)→∞\(P\+v\)\\to\\infty,α~→1/ϕ​\(P\+v\)\\tilde\{\\alpha\}\\to 1/\\sqrt\{\\phi\\,\(P\+v\)\}and the bonus scales asc​8/π​\(P\+v\)c\\sqrt\{8/\\pi\}\\sqrt\{\(P\+v\)\}, recovering canonical UCB\-styleP\+v\\sqrt\{P\+v\}scaling\. More generally, volatility raises the bonus through the predictive varianceP\+vP\+vwhile dampingSSthroughDD, acting as a dual discount that shortens the effective planning horizon\. The CAUSE index thus inherits, in closed form, the structural properties of the optimal Gittins bonus: exploration rises with volatility that makes new observations relevant and falls with stochasticity that makes them unreliable\.

## 6Experiments

We evaluate CAUSE in four experiments: cumulative regret in heterogeneous\-arms restless bandits, comparison of CAUSE’s exploration bonus to the optimal Gittins bonus across the noise\-parameter grid, verification against the rested\-optimal Gittins reference atv=0v=0, and a lesion analysis that uses CAUSE to characterize the behavioral consequences of miscalibrated noise inference\.

### 6\.1Regret in heterogeneous\-arms restless bandits

Standard exploration strategies treat uncertainty as a single quantity and prescribe more exploration whenever uncertainty is high\. The structural analysis of Section[4](https://arxiv.org/html/2605.19215#S4)predicts this is suboptimal whenever volatility and stochasticity vary across arms: uncertainty arising from volatility warrants exploration; uncertainty arising from stochasticity does not\. We test this in three regimes\. In the*mixed*regime, arms span the four cells of the\(v,s\)\(v,s\)grid, with both volatility and stochasticity varying across arms\. In the*s\-dominant*regime, stochasticity varies substantially across arms while volatility is uniform\. In the*v\-dominant*regime, volatility varies while stochasticity is uniform\. We compare CAUSE against Thompson sampling\[[34](https://arxiv.org/html/2605.19215#bib.bib51)\], UCB\[[17](https://arxiv.org/html/2605.19215#bib.bib6)\], predictive sampling\[[20](https://arxiv.org/html/2605.19215#bib.bib23)\], a myopic baseline, and a Gittins\-per\-arm policy whose bonus is computed by numerical value iteration on the rested retirement problem \(Appendix[D](https://arxiv.org/html/2605.19215#A4)\)\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/regret.png)Figure 1:Cumulative discounted regret overT=200T=200steps in three regimes \(K=4K=4,γ=0\.95\\gamma=0\.95,10001000Monte Carlo runs\)\. CAUSE achieves the lowest regret in all three regimes\.In the*mixed*regime \(Figure[1](https://arxiv.org/html/2605.19215#S6.F1), left\), CAUSE achieves the lowest regret, ahead of all baselines including Gittins\-per\-arm\. This is not surprising: the Gittins index is optimal for rested bandits, but its rested\-optimality does not transfer to the restless setting we study, in which the latent state evolves under volatility\. In the*s\-dominant*regime \(Figure[1](https://arxiv.org/html/2605.19215#S6.F1), middle\), where stochasticity varies substantially across arms but volatility is shared, we expect both CAUSE and Gittins\-per\-arm to perform well\. High\-ssarms accumulate high posterior variance, but exploring such arms is unproductive: the variance arises from stochasticity rather than from drift in the latent state, so additional observations reduce posterior uncertainty only slowly\. Both CAUSE and Gittins\-per\-arm correctly down\-weight exploration of high\-ssarms, and the figure confirms this: the two policies tie at the lowest regret\. UCB and Thompson sampling, which treat posterior variance as a single quantity, incur substantially higher regret, worse than the myopic baseline that ignores uncertainty entirely\. In the*v\-dominant*regime \(Figure[1](https://arxiv.org/html/2605.19215#S6.F1), right\), CAUSE outperforms Gittins\-per\-arm by a substantial margin\. Both policies allocate more exploration to high\-volatility arms, as the structural analysis prescribes; the difference is that CAUSE’s bonus additionally incorporates a horizon\-shortening damping that accounts for the diminishing value of distant\-future information when latent states drift \(Section[5](https://arxiv.org/html/2605.19215#S5)\)\. Gittins\-per\-arm, derived for rested bandits, lacks this damping and over\-explores in this regime\.

Robustness across problem sizes \(KK\), discount factors \(γ\\gamma\), reward configurations, and the UCB scaling parameterccis reported in Appendix[E](https://arxiv.org/html/2605.19215#A5)\.

### 6\.2Closed\-form bonus tracks the optimal Gittins bonus

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/bonus_s.png)Figure 2:Exploration bonus as a function of stochasticityss, normalized to a common range\. CAUSE tracks the Gittins shape; UCB is insensitive toss\.CAUSE is derived independently of the Gittins framework, via control\-as\-inference under an optimality constraint\. Whether its closed form captures the structural shape of the optimal Gittins bonus is an empirical question\. We compute the Gittins bonus by value iteration on the retirement problem \(Appendix[C\.2](https://arxiv.org/html/2605.19215#A3.SS2)\) and compare it to the bonuses of CAUSE and UCB at a fixed posterior variance \(Thompson sampling and predictive sampling do not produce explicit per\-arm bonuses\)\.

Figure[2](https://arxiv.org/html/2605.19215#S6.F2)shows each policy’s bonus along the stochasticity axis\. CAUSE tracks the Gittins shape closely across two orders of magnitude, while UCB is insensitive tossby construction, allocating exploration based on posterior variance alone\. The convergence of two structurally distinct derivations \(the Gittins retirement problem and the control\-as\-inference posterior\) on the samess\-dependence supports the structural account of Section[4](https://arxiv.org/html/2605.19215#S4)\. Bonus dependence on volatility \(Appendix[D\.1](https://arxiv.org/html/2605.19215#A4.SS1)\) is less discriminative between policies; the differences along that axis emerge in regret rather than in bonus shape\.

### 6\.3Matching the rested optimum

CAUSE’s derivation involves several analytical approximations \(Appendix[B](https://arxiv.org/html/2605.19215#A2)\)\. Quantifying the suboptimality these approximations introduce is intractable in general, since formal regret bounds for index policies on restless Gaussian bandits remain an open problem\. At the rested limit \(v=0v=0\), however, the optimal policy is the numerically computable Gittins index, provably optimal for infinite\-horizon discounted reward in iid bandits\[[13](https://arxiv.org/html/2605.19215#bib.bib41)\]\. This regime admits a direct empirical comparison of CAUSE against optimal performance\. Atv=0v=0across both moderate \(s∈\{9,25\}s\\in\\\{9,25\\\}\) and extreme \(s∈\{9,900\}s\\in\\\{9,900\\\}\) heterogeneity in stochasticity, CAUSE’s discounted cumulative regret is statistically indistinguishable from the numerical Gittins reference\. Ats∈\{9,25\}s\\in\\\{9,25\\\}overT=200T=200steps and10001000Monte Carlo runs, CAUSE achieves22\.61±0\.8822\.61\\pm 0\.88and Gittins\-per\-arm achieves22\.50±0\.7522\.50\\pm 0\.75\(mean±\\pmSEM\), with the difference of0\.110\.11well within the sampling error\. The closeness is preserved at extreme stochasticity \(s∈\{9,900\}s\\in\\\{9,900\\\}, Appendix[D\.2](https://arxiv.org/html/2605.19215#A4.SS2)\)\.

Combined with the restless results of Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1), this provides a layered empirical bound on CAUSE’s suboptimality: where the optimal policy is known, CAUSE matches it within Monte Carlo precision; where the optimal policy is intractable, CAUSE exceeds the strongest tractable reference\.

### 6\.4Pathological noise inference produces opposing reversals

We compare three agents that differ only in their inference of the environmental noise structure\. The*healthy*agent jointly infers\(s^,v^\)\(\\hat\{s\},\\hat\{v\}\)from the observation stream by explaining away\[[26](https://arxiv.org/html/2605.19215#bib.bib45)\]: the two sources compete to account for observed variance, with their relative contributions disentangled by their distinct temporal signatures\. The*stochasticity\-blind*agent is insensitive toss: itss^\\hat\{s\}estimate is pinned at its initial value, so its inference attributes the observed variance almost entirely to volatility, producing a smalls^\\hat\{s\}and an inflatedv^\\hat\{v\}\. The*volatility\-blind*agent is the symmetric case: insensitive tovv, withv^\\hat\{v\}pinned at its initial value, so the increased variance is absorbed intos^\\hat\{s\}, producing a smallv^\\hat\{v\}and an inflateds^\\hat\{s\}\. Each agent uses its own inferred\(s^,v^\)\(\\hat\{s\},\\hat\{v\}\)to compute the CAUSE bonus\. The posterior variance is held at the prior valueP0P\_\{0\}across all agents, so that the bonus differences across agents reflect their inferred\(s^,v^\)\(\\hat\{s\},\\hat\{v\}\)alone, without compounding through differences in belief updating\.

Figure[3](https://arxiv.org/html/2605.19215#S6.F3)shows the resulting bonus surfaces alongside the learning\-rate measure\[[26](https://arxiv.org/html/2605.19215#bib.bib45)\]\. Both signatures reverse similarly under each lesion\. The stochasticity\-blind agent’s bonus is increasing inss, a direct reversal of the healthy monotonicity: as the truessgrows, the agent’s inference absorbs the variance intov^\\hat\{v\}rather thans^\\hat\{s\}, and the inflatedv^\\hat\{v\}raises the bonus through the same channel that would, in a healthy agent, register genuine drift\. The agent therefore explores more in environments where exploration is least informative, treating noise as if it were drift\. The volatility\-blind agent shows the symmetric reversal: bonus decreases invvbecause the increased variance is absorbed intos^\\hat\{s\}, suppressingα~t\\tilde\{\\alpha\}\_\{t\}through the stochasticity channel\. The agent under\-explores in environments where exploration is most informative, treating drift as if it were noise\. Each lesion reverses only its corresponding monotonicity, leaving the orthogonal axis intact\. The cross\-channel parallel between learning rate and exploration bonus is a direct consequence of both quantities being downstream of the same joint inference of\(s^,v^\)\(\\hat\{s\},\\hat\{v\}\): when the inference fails through misattribution, both the belief\-update signal \(learning rate\) and the action\-selection signal \(exploration bonus\) inherit the same misweighting\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/lesion.png)Figure 3:Learning rate \(top row\) and CAUSE exploration bonus \(bottom row\) for healthy, stochasticity\-blind, and volatility\-blind agents, averaged across multiple seeds\. Both signatures reverse identically under each lesion: stochasticity\-blindness reverses thess\-monotonicity; volatility\-blindness reverses thevv\-monotonicity\.

## 7Discussion

We have shown that volatility and stochasticity drive optimal exploration in opposite directions, that this asymmetry is captured in closed form by the CAUSE index, and that miscalibrated noise inference produces dissociable, axis\-specific reversals in exploratory behavior\. The structural result extends the Gittins framework from iid Gaussian bandits to Gaussian state\-space bandits with latent dynamics, providing, to our knowledge, the first closed\-form exploration index for restless bandits with continuous\-state Gaussian dynamics\. Empirically, CAUSE achieves the lowest regret across the mixed, s\-dominant, and v\-dominant restless regimes; in the rested limit, where Gittins is provably optimal, CAUSE’s regret matches it within Monte Carlo precision\. Together these results provide a layered empirical bound on CAUSE’s suboptimality: where the optimal policy is known, CAUSE matches it; where it is intractable, CAUSE exceeds the strongest tractable reference\. The lesion analysis additionally shows that miscalibrated noise inference produces a specific behavioral signature whose direction identifies the failed inferential channel\.

The framework is developed for Gaussian state\-space models with random\-walk latent dynamics, and the closed\-form CAUSE index inherits this scope; extension to non\-Gaussian distributions or change\-point dynamics is an open question\. The directional predictions should extend on general informational grounds \(uncertainty arising from drift is reducible by exploration, uncertainty arising from stochasticity is not, regardless of the parametric form\), but the closed form itself relies on Gaussian conjugacy and the probit approximation to Gaussian\-sigmoid convolutions, and does not transfer mechanically\. The control\-as\-inference framework we use to derive CAUSE, however, is general: it casts exploration as a backward message\-passing problem that applies wherever a Bellman recursion admits this form\. To our knowledge, this is the first application of control\-as\-inference to derive an exploration index for restless bandits, and the framework provides a route to closed\-form policies in restless bandit classes beyond linear\-Gaussian dynamics\. Within the present setting, our derivation tracks the backward message in the sigmoid family using probit approximations and trapezoidal linearization of the recursion \(Appendix[B](https://arxiv.org/html/2605.19215#A2)\); an exponential\-family alternative admits exact Gaussian recursions but, in exploratory tests, produced over\-exploration at high uncertainty since it does not damp the second\-moment contribution to the message\. The sigmoid choice trades exact tractability for calibrated behavior\. The behavioral predictions are evaluated in simulation; the predicted reversal patterns offer concrete empirical targets for behavioral testing, complementing the learning\-rate signatures already established in this literature\.

The framework is most directly relevant to computational accounts of psychiatric conditions in which inference about environmental noise has been implicated, including anxiety, psychosis, and depression\. The empirical exploration literature in these conditions is mixed: studies report increased exploration, reduced exploration, or context\-dependent effects, with the inconsistencies typically attributed to task heterogeneity or population differences\[[3](https://arxiv.org/html/2605.19215#bib.bib8),[9](https://arxiv.org/html/2605.19215#bib.bib7)\]\. Our framework offers a different reading\. If a condition involves miscalibrated noise inference, the direction of the exploratory abnormality should depend on which inferential channel is affected and on the relative weight of volatility and stochasticity in the task: hyposensitive stochasticity inference predicts over\-exploration in high\-stochasticity tasks, while hyposensitive volatility inference predicts under\-exploration in high\-volatility tasks\. The apparent inconsistency in this literature should therefore resolve when tasks are stratified by their volatility\-stochasticity composition and individuals by their inferential profile\.

## References

- \[1\]A\. Abdolmaleki, J\. T\. Springenberg, Y\. Tassa, R\. Munos, N\. Heess, and M\. Riedmiller\(2018\)Maximum a posteriori policy optimisation\.arXiv preprint arXiv:1806\.06920\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1)\.
- \[2\]P\. Auer, N\. Cesa\-Bianchi, and P\. Fischer\(2002\)Finite\-time analysis of the multiarmed bandit problem\.Machine learning47\(2\),pp\. 235–256\.Cited by:[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px4),[§1](https://arxiv.org/html/2605.19215#S1.p1.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1)\.
- \[3\]J\. Aylward, V\. Valton, W\. Ahn, R\. L\. Bond, P\. Dayan, J\. P\. Roiser, and O\. J\. Robinson\(2019\)Altered learning under uncertainty in unmedicated mood and anxiety disorders\.Nature human behaviour3\(10\),pp\. 1116–1123\.Cited by:[§7](https://arxiv.org/html/2605.19215#S7.p3.1)\.
- \[4\]T\. E\. Behrens, M\. W\. Woolrich, M\. E\. Walton, and M\. F\. Rushworth\(2007\)Learning the value of information in an uncertain world\.Nature neuroscience10\(9\),pp\. 1214–1221\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px2.p1.1)\.
- \[5\]M\. Botvinick and M\. Toussaint\(2012\)Planning as inference\.Trends in cognitive sciences16\(10\),pp\. 485–488\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1)\.
- \[6\]M\. Browning, T\. E\. Behrens, G\. Jocham, J\. X\. O’reilly, and S\. J\. Bishop\(2015\)Anxious individuals have difficulty learning the causal statistics of aversive environments\.Nature neuroscience18\(4\),pp\. 590–596\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px5.p1.1)\.
- \[7\]J\. D\. Cohen, S\. M\. McClure, and A\. J\. Yu\(2007\)Should i stay or should i go? how the human brain manages the trade\-off between exploitation and exploration\.Philosophical Transactions of the Royal Society B: Biological Sciences362\(1481\),pp\. 933–942\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px1.p1.1)\.
- \[8\]N\. D\. Daw, J\. P\. O’doherty, P\. Dayan, B\. Seymour, and R\. J\. Dolan\(2006\)Cortical substrates for exploratory decisions in humans\.Nature441\(7095\),pp\. 876–879\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p1.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]H\. Fan, S\. J\. Gershman, and E\. A\. Phelps\(2023\)Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty\.Nature Human Behaviour7\(1\),pp\. 102–113\.Cited by:[§7](https://arxiv.org/html/2605.19215#S7.p3.1)\.
- \[10\]M\. J\. Frank, B\. B\. Doll, J\. Oas\-Terpstra, and F\. Moreno\(2009\)Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation\.Nature neuroscience12\(8\),pp\. 1062–1068\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p1.1)\.
- \[11\]S\. J\. Gershman\(2018\)Deconstructing the human algorithms for exploration\.Cognition173,pp\. 34–42\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]S\. J\. Gershman\(2019\)Uncertainty and exploration\.\.Decision6\(3\),pp\. 277\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p1.1)\.
- \[13\]J\. C\. Gittins\(1979\)Bandit processes and dynamic allocation indices\.Journal of the Royal Statistical Society Series B: Statistical Methodology41\(2\),pp\. 148–164\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2605.19215#S4.SS1.p1.3),[§4\.1](https://arxiv.org/html/2605.19215#S4.SS1.p1.6),[§6\.3](https://arxiv.org/html/2605.19215#S6.SS3.p1.12)\.
- \[14\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1)\.
- \[15\]H\. Huang, W\. Thompson, and M\. P\. Paulus\(2017\)Computational dysfunctions in anxiety: failure to differentiate signal from noise\.Biological psychiatry82\(6\),pp\. 440–446\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px5.p1.1)\.
- \[16\]H\. J\. Kappen, V\. Gómez, and M\. Opper\(2012\)Optimal control as a graphical model inference problem\.Machine learning87\(2\),pp\. 159–182\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1)\.
- \[17\]E\. Kaufmann, O\. Cappé, and A\. Garivier\(2012\)On bayesian upper confidence bounds for bandit problems\.InArtificial intelligence and statistics,pp\. 592–600\.Cited by:[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px4),[§E\.3](https://arxiv.org/html/2605.19215#A5.SS3.p1.7),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.19215#S6.SS1.p1.1)\.
- \[18\]T\. L\. Lai and H\. Robbins\(1985\)Asymptotically efficient adaptive allocation rules\.Advances in applied mathematics6\(1\),pp\. 4–22\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p1.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1)\.
- \[19\]S\. Levine\(2018\)Reinforcement learning and control as probabilistic inference: tutorial and review\.arXiv preprint arXiv:1805\.00909\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p5.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2605.19215#S5.p1.1)\.
- \[20\]Y\. Liu, B\. Van Roy, and K\. Xu\(2023\)Nonstationary bandit learning via predictive sampling\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 6215–6244\.Cited by:[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px3),[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px3.p1.10),[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px3.p1.3),[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px3.p2.2),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.19215#S6.SS1.p1.1)\.
- \[21\]D\. J\. MacKay\(2003\)Information theory, inference and learning algorithms\.Cambridge university press\.Cited by:[§B\.2](https://arxiv.org/html/2605.19215#A2.SS2.p1.3)\.
- \[22\]C\. Mathys, J\. Daunizeau, K\. J\. Friston, and K\. E\. Stephan\(2011\)A bayesian foundation for individual learning under uncertainty\.Frontiers in human neuroscience5,pp\. 39\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px2.p1.1)\.
- \[23\]M\. R\. Nassar, K\. M\. Rumsey, R\. C\. Wilson, K\. Parikh, B\. Heasly, and J\. I\. Gold\(2012\)Rational regulation of learning dynamics by pupil\-linked arousal systems\.Nature neuroscience15\(7\),pp\. 1040–1046\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]M\. R\. Nassar, R\. C\. Wilson, B\. Heasly, and J\. I\. Gold\(2010\)An approximately bayesian delta\-rule model explains the dynamics of belief updating in a changing environment\.Journal of Neuroscience30\(37\),pp\. 12366–12378\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px2.p1.1)\.
- \[25\]C\. H\. Papadimitriou and J\. N\. Tsitsiklis\(1999\)The complexity of optimal queuing network control\.Mathematics of Operations Research24\(2\),pp\. 293–305\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]P\. Piray and N\. D\. Daw\(2021\)A model for learning based on the joint estimation of stochasticity and volatility\.Nature communications12\(1\),pp\. 6587\.Cited by:[§C\.3](https://arxiv.org/html/2605.19215#A3.SS3.p1.8),[§1](https://arxiv.org/html/2605.19215#S1.p6.1),[§1](https://arxiv.org/html/2605.19215#S1.p7.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px2.p1.1),[§6\.4](https://arxiv.org/html/2605.19215#S6.SS4.p1.13),[§6\.4](https://arxiv.org/html/2605.19215#S6.SS4.p2.9)\.
- \[27\]P\. Piray and N\. D\. Daw\(2024\)Computational processes of simultaneous learning of stochasticity and volatility in humans\.Nature communications15\(1\),pp\. 9073\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p7.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]A\. R\. Powers, C\. Mathys, and P\. R\. Corlett\(2017\)Pavlovian conditioning–induced hallucinations result from overweighting of perceptual priors\.Science357\(6351\),pp\. 596–600\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px5.p1.1)\.
- \[29\]E\. Pulcu and M\. Browning\(2017\)Affective bias as a rational response to the statistics of rewards and punishments\.Elife6,pp\. e27879\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px5.p1.1)\.
- \[30\]D\. Russo, B\. Van Roy, A\. Kazerouni, I\. Osband, and Z\. Wen\(2018\)A tutorial on thompson sampling\.Foundations and Trends® in Machine Learning11\(1\),pp\. 1–99\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1)\.
- \[31\]D\. Russo and B\. Van Roy\(2014\)Learning to optimize via information\-directed sampling\.Advances in neural information processing systems27\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p1.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1)\.
- \[32\]D\. Russo and B\. Van Roy\(2018\)Satisficing in time\-sensitive bandit learning\.arXiv preprint arXiv:1803\.02855\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1)\.
- \[33\]M\. H\. Satti, K\. Wille, M\. R\. Nassar, R\. M\. Cichy, N\. W\. Schuck, P\. Dayan, and R\. Bruckner\(2025\)Absence of systematic effects of internalizing psychopathology on learning under uncertainty\.bioRxiv,pp\. 2025–05\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px5.p1.1)\.
- \[34\]W\. R\. Thompson\(1933\)On the likelihood that one unknown probability exceeds another in view of the evidence of two samples\.Biometrika25\(3/4\),pp\. 285–294\.Cited by:[§C\.1](https://arxiv.org/html/2605.19215#A3.SS1.SSS0.Px2),[§1](https://arxiv.org/html/2605.19215#S1.p1.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.19215#S6.SS1.p1.1)\.
- \[35\]E\. Todorov\(2008\)General duality between optimal control and estimation\.In2008 47th IEEE conference on decision and control,pp\. 4286–4292\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p5.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2605.19215#S5.p1.1)\.
- \[36\]M\. Toussaint\(2009\)Robot trajectory optimization using approximate inference\.InProceedings of the 26th annual international conference on machine learning,pp\. 1049–1056\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px4.p1.1)\.
- \[37\]P\. Whittle\(1988\)Restless bandits: activity allocation in a changing world\.Journal of applied probability25\(A\),pp\. 287–298\.Cited by:[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2605.19215#S5.SS1.p2.1)\.
- \[38\]R\. C\. Wilson, A\. Geana, J\. M\. White, E\. A\. Ludvig, and J\. D\. Cohen\(2014\)Humans use directed and random exploration to solve the explore–exploit dilemma\.\.Journal of experimental psychology: General143\(6\),pp\. 2074\.Cited by:[§1](https://arxiv.org/html/2605.19215#S1.p1.1),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px1.p1.1)\.
- \[39\]Y\. Yao\(2006\)Some results on the gittins index for a normal reward process\.Lecture Notes\-Monograph Series,pp\. 284–294\.Cited by:[§A\.3](https://arxiv.org/html/2605.19215#A1.SS3.1.p1.7),[§A\.4](https://arxiv.org/html/2605.19215#A1.SS4.1.p1.11),[Appendix A](https://arxiv.org/html/2605.19215#A1.p1.4),[§2](https://arxiv.org/html/2605.19215#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2605.19215#S4.SS1.p4.4)\.

## Appendix AProofs of monotonicity results

This appendix provides proofs of the three monotonicity results stated in Section[4](https://arxiv.org/html/2605.19215#S4)of the main text: the index decomposition \(Proposition[1](https://arxiv.org/html/2605.19215#Thmtheorem1)\), the monotonicity of the exploration bonus in the observation noisess\(Theorem[2](https://arxiv.org/html/2605.19215#Thmtheorem2)\), and the monotonicity in the innovation variancevv\(Theorem[3](https://arxiv.org/html/2605.19215#Thmtheorem3)\)\. The monotonicity in the posterior variancePP\(Lemma[4](https://arxiv.org/html/2605.19215#Thmtheorem4)\) follows fromYao \[[39](https://arxiv.org/html/2605.19215#bib.bib39)\]and is proved here for use in thevv\-proof\. Notation, the generative model, and the Kalman recursion follow Section[3](https://arxiv.org/html/2605.19215#S3)of the main text\.

### A\.1Preliminaries

A*stopping rule*for an observation sequencer1,r2,…r\_\{1\},r\_\{2\},\\ldotsis a rule that, at each timenn, decides whether to stop using only the observationsr1,…,rnr\_\{1\},\\ldots,r\_\{n\}seen so far\. We writeξ\\xifor the time at which the rule stops\.

###### Definition 5\(Value function of the retirement problem\)\.

Given retirement salaryλ\\lambda, the value function of the retirement problem is

V​\(m,P;s,v,γ\)=supξ≥0E​\[∑n=1ξγn−1​rn\+γξ​λ1−γ\|x0∼N​\(m,P\)\],V\(m,P;s,v,\\gamma\)=\\sup\_\{\\xi\\geq 0\}\\;E\\\!\\left\[\\left\.\\sum\_\{n=1\}^\{\\xi\}\\gamma^\{n\-1\}\\,r\_\{n\}\+\\gamma^\{\\xi\}\\,\\frac\{\\lambda\}\{1\-\\gamma\}\\;\\right\|\\;x\_\{0\}\\sim N\(m,P\)\\right\],where the supremum is over stopping rulesξ\\xifor the observation sequence\.

The value function satisfies the Bellman equation

V​\(m,P\)=max⁡\(λ1−γ,m\+γ​Em′​\[V​\(m′,P′\)\]\),V\(m,P\)=\\max\\\!\\left\(\\frac\{\\lambda\}\{1\-\\gamma\},\\;\\;m\+\\gamma\\,E\_\{m^\{\\prime\}\}\[V\(m^\{\\prime\},P^\{\\prime\}\)\]\\right\),\(9\)whereP′=\(P\+v\)​s/\(P\+v\+s\)P^\{\\prime\}=\(P\+v\)s/\(P\+v\+s\)is the updated posterior variance \(deterministic\) andm′∼N​\(m,σ2​\(P\)\)m^\{\\prime\}\\sim N\(m,\\sigma^\{2\}\(P\)\)withσ2​\(P\)=\(P\+v\)2/\(P\+v\+s\)=\(P\+v\)−P′\\sigma^\{2\}\(P\)=\(P\+v\)^\{2\}/\(P\+v\+s\)=\(P\+v\)\-P^\{\\prime\}is the next\-step posterior mean\.

### A\.2Index decomposition

Rearranging the retirement equation \([3](https://arxiv.org/html/2605.19215#S4.E3)\) using∑n=1ξγn−1=\(1−γξ\)/\(1−γ\)\\sum\_\{n=1\}^\{\\xi\}\\gamma^\{n\-1\}=\(1\-\\gamma^\{\\xi\}\)/\(1\-\\gamma\)yields the equivalent ratio form

λ=supξ≥1E​\[∑n=1ξγn−1​rn\]E​\[∑n=1ξγn−1\],\\lambda\\;=\\;\\sup\_\{\\xi\\geq 1\}\\;\\frac\{E\\\!\\left\[\\sum\_\{n=1\}^\{\\xi\}\\gamma^\{n\-1\}\\,r\_\{n\}\\right\]\}\{E\\\!\\left\[\\sum\_\{n=1\}^\{\\xi\}\\gamma^\{n\-1\}\\right\]\},\(10\)which we use in the proof of Theorem[2](https://arxiv.org/html/2605.19215#Thmtheorem2)below\.

###### Proof of Proposition[1](https://arxiv.org/html/2605.19215#Thmtheorem1)\.

Replacing the prior meanmmbym\+am\+aand writingx~t=xt−a\\tilde\{x\}\_\{t\}=x\_\{t\}\-agivesx~0∼N​\(m,P\)\\tilde\{x\}\_\{0\}\\sim N\(m,P\), with the same innovations and observation noise, andrt=x~t\+a\+ϵtr\_\{t\}=\\tilde\{x\}\_\{t\}\+a\+\\epsilon\_\{t\}\. The filtration is unchanged\. Substitutingrn=r~n\+ar\_\{n\}=\\tilde\{r\}\_\{n\}\+ainto \([3](https://arxiv.org/html/2605.19215#S4.E3)\):

E​\[∑n=1ξγn−1​rn\+γξ​λ1−γ\]=E​\[∑n=1ξγn−1​r~n\+γξ​λ−a1−γ\]\+a1−γ\.E\\\!\\left\[\\sum\_\{n=1\}^\{\\xi\}\\gamma^\{n\-1\}r\_\{n\}\+\\gamma^\{\\xi\}\\frac\{\\lambda\}\{1\-\\gamma\}\\right\]=E\\\!\\left\[\\sum\_\{n=1\}^\{\\xi\}\\gamma^\{n\-1\}\\tilde\{r\}\_\{n\}\+\\gamma^\{\\xi\}\\frac\{\\lambda\-a\}\{1\-\\gamma\}\\right\]\+\\frac\{a\}\{1\-\\gamma\}\.The retirement equation at salaryλ\\lambdafor the shifted arm is thus the retirement equation at salaryλ−a\\lambda\-afor the unshifted arm, after subtractinga/\(1−γ\)a/\(1\-\\gamma\)from both sides\. Henceλ​\(m\+a,P,s,v,γ\)=a\+λ​\(m,P,s,v,γ\)\\lambda\(m\+a,P,s,v,\\gamma\)=a\+\\lambda\(m,P,s,v,\\gamma\), and settingm=0m=0gives \([4](https://arxiv.org/html/2605.19215#S4.E4)\)\. ∎

### A\.3Proof of Theorem[2](https://arxiv.org/html/2605.19215#Thmtheorem2)

###### Proof\.

The argument adapts Yao’s randomization proof for iid Gaussian bandits\[[39](https://arxiv.org/html/2605.19215#bib.bib39), Lemma 1\]to the state\-space setting\. Fixs1<s2s\_\{1\}<s\_\{2\}, and let\{xt\}\\\{x\_\{t\}\\\}be a single realization of the latent state\. Letϵn∼𝒩​\(0,s1\)\\epsilon\_\{n\}\\sim\\mathcal\{N\}\(0,s\_\{1\}\)be the observation noise of the clean arm,rn=xn\+ϵnr\_\{n\}=x\_\{n\}\+\\epsilon\_\{n\}, andηn∼𝒩​\(0,s2−s1\)\\eta\_\{n\}\\sim\\mathcal\{N\}\(0,s\_\{2\}\-s\_\{1\}\)independent of everything else\. Settingrn′=rn\+ηnr\_\{n\}^\{\\prime\}=r\_\{n\}\+\\eta\_\{n\}gives a valid observation sequence for ans2s\_\{2\}\-arm driven by the same latent process\.

Any stopping ruleξ′\\xi^\{\\prime\}on the noisier sequence\{rn′\}\\\{r\_\{n\}^\{\\prime\}\\\}can be viewed as a randomized stopping rule on the clean sequence\{rn\}\\\{r\_\{n\}\\\}: the agent operating on\{rn\}\\\{r\_\{n\}\\\}can simply draw the extra noise\{ηn\}\\\{\\eta\_\{n\}\\\}internally and stop according toξ′\\xi^\{\\prime\}\. The supremum in the ratio form \(Eq\. \([10](https://arxiv.org/html/2605.19215#A1.E10)\)\) over stopping rules for\{rn′\}\\\{r\_\{n\}^\{\\prime\}\\\}is therefore bounded above by the supremum over \(possibly randomized\) stopping rules for\{rn\}\\\{r\_\{n\}\\\}, givingB​\(P,s2,v,γ\)≤B​\(P,s1,v,γ\)B\(P,s\_\{2\},v,\\gamma\)\\leq B\(P,s\_\{1\},v,\\gamma\)\.

The randomization argument extends from the iid setting to the state\-space setting without modification because the observation channel is still additive Gaussian noise and the latent process\{xt\}\\\{x\_\{t\}\\\}is shared between the clean and noisy versions of the bandit; the latent dynamics never enter the coupling\. ∎

### A\.4Proof of Lemma[4](https://arxiv.org/html/2605.19215#Thmtheorem4)

###### Proof\.

The result follows from a standard convexity argument\. The retirement\-problem value functionV​\(m,P\)V\(m,P\)is convex inmmfor eachPP\(a standard backward\-induction property of Bellman operators with linear and constant alternatives, used also in the proof of Theorem[3](https://arxiv.org/html/2605.19215#Thmtheorem3)\)\. ThePP\-dependent Bellman operator at fixedvvsatisfies: increasingPPincreases the predictive varianceP\+vP\+v, hence increases the spread of the next\-step posterior meanm′m^\{\\prime\}, hence by Jensen’s inequality increases𝔼​\[V​\(m′,P′\)\]\\mathbb\{E\}\[V\(m^\{\\prime\},P^\{\\prime\}\)\]at fixed continuation value\. The break\-evenλ\\lambdainherits the monotonicity\. In the iid limit \(v=0v=0\), this reduces to a result ofYao \[[39](https://arxiv.org/html/2605.19215#bib.bib39)\]\(Theorem 1\)\. ∎

### A\.5Proof of Theorem[3](https://arxiv.org/html/2605.19215#Thmtheorem3)

#### Intuition\.

When the latent state drifts faster, two things happen\. First, each pull is more informative relative to the growing uncertainty, so exploring is more rewarding\. Second, the arm keeps generating fresh uncertainty even after you observe it, effectively keeping the posterior variance elevated, and higher variance increases the bonus by Lemma[4](https://arxiv.org/html/2605.19215#Thmtheorem4)\.

###### Proof\.

We track thevv\-dependence of the value function explicitly, writingV​\(m,P;v\)V\(m,P;v\)\. Recall

P′​\(v\)=\(P\+v\)​sP\+v\+s,σ2​\(v\)=\(P\+v\)2P\+v\+s,P^\{\\prime\}\(v\)=\\frac\{\(P\+v\)s\}\{P\+v\+s\},\\qquad\\sigma^\{2\}\(v\)=\\frac\{\(P\+v\)^\{2\}\}\{P\+v\+s\},both strictly increasing invv\.

Convexity ofVVin its first argument follows from a standard backward\-induction argument: the continuation valuem\+γ​E​\[V​\(m′,P′;v\)\]m\+\\gamma\\,E\[V\(m^\{\\prime\},P^\{\\prime\};v\)\]withm′∼N​\(m,σ2\)m^\{\\prime\}\\sim N\(m,\\sigma^\{2\}\)is a Gaussian convolution of a convex function plus a linear term, hence convex, and the Bellman operator preserves convexity through the pointwise maximum with the retirement value\.

The proof structure is:vventers the continuation value at three places, through the spreadσ2​\(v\)\\sigma^\{2\}\(v\)of the next\-step posterior mean, through the next\-step posterior varianceP′​\(v\)P^\{\\prime\}\(v\), and through the next\-period value functionV​\(⋅,⋅;v\)V\(\\cdot,\\cdot;v\)itself\. Each contributes a monotonicity\-preserving inequality, which we chain\.

*Base case\.*V=λ/\(1−γ\)V=\\lambda/\(1\-\\gamma\), independent ofvv\.

*Inductive step\.*Joint hypothesis: at all future stages,V​\(⋅,P;v\)V\(\\cdot,P;v\)is convex in its first argument for every\(P,v\)\(P,v\), andV​\(m,P;v\)V\(m,P;v\)is nondecreasing invvfor everyPP\. Takev1<v2v\_\{1\}<v\_\{2\}and consider the continuation valueC​\(0,P;v\)=γ​E​\[V​\(mv′,P′​\(v\);v\)\]C\(0,P;v\)=\\gamma\\,E\[V\(m^\{\\prime\}\_\{v\},P^\{\\prime\}\(v\);v\)\]wheremv′∼N​\(0,σ2​\(v\)\)m^\{\\prime\}\_\{v\}\\sim N\(0,\\sigma^\{2\}\(v\)\)\. We chain three inequalities:

1. 1\.*Wider spread ofm′m^\{\\prime\}helps \(Jensen\)\.*σ2​\(v2\)\>σ2​\(v1\)\\sigma^\{2\}\(v\_\{2\}\)\>\\sigma^\{2\}\(v\_\{1\}\)andV​\(⋅,P′​\(v2\);v2\)V\(\\cdot,P^\{\\prime\}\(v\_\{2\}\);v\_\{2\}\)is convex, so spreading the zero\-mean distribution ofm′m^\{\\prime\}increases its expected value: E​\[V​\(mv2′,P′​\(v2\);v2\)\]≥E​\[V​\(mv1′,P′​\(v2\);v2\)\]\.E\[V\(m^\{\\prime\}\_\{v\_\{2\}\},P^\{\\prime\}\(v\_\{2\}\);v\_\{2\}\)\]\\geq E\[V\(m^\{\\prime\}\_\{v\_\{1\}\},P^\{\\prime\}\(v\_\{2\}\);v\_\{2\}\)\]\.
2. 2\.*LargerP′P^\{\\prime\}helps \(Lemma[4](https://arxiv.org/html/2605.19215#Thmtheorem4)\)\.*P′​\(v2\)\>P′​\(v1\)P^\{\\prime\}\(v\_\{2\}\)\>P^\{\\prime\}\(v\_\{1\}\)andVVis nondecreasing in its second argument: E​\[V​\(mv1′,P′​\(v2\);v2\)\]≥E​\[V​\(mv1′,P′​\(v1\);v2\)\]\.E\[V\(m^\{\\prime\}\_\{v\_\{1\}\},P^\{\\prime\}\(v\_\{2\}\);v\_\{2\}\)\]\\geq E\[V\(m^\{\\prime\}\_\{v\_\{1\}\},P^\{\\prime\}\(v\_\{1\}\);v\_\{2\}\)\]\.
3. 3\.*Largervvhelps \(inductive hypothesis\)\.*v2\>v1v\_\{2\}\>v\_\{1\}andVVis nondecreasing invv: E​\[V​\(mv1′,P′​\(v1\);v2\)\]≥E​\[V​\(mv1′,P′​\(v1\);v1\)\]\.E\[V\(m^\{\\prime\}\_\{v\_\{1\}\},P^\{\\prime\}\(v\_\{1\}\);v\_\{2\}\)\]\\geq E\[V\(m^\{\\prime\}\_\{v\_\{1\}\},P^\{\\prime\}\(v\_\{1\}\);v\_\{1\}\)\]\.

Chaining:C​\(0,P;v2\)≥C​\(0,P;v1\)C\(0,P;v\_\{2\}\)\\geq C\(0,P;v\_\{1\}\), and therefore

V​\(0,P;v2\)=max⁡\(λ1−γ,C​\(0,P;v2\)\)≥V​\(0,P;v1\)\.V\(0,P;v\_\{2\}\)=\\max\\\!\\left\(\\frac\{\\lambda\}\{1\-\\gamma\},\\;C\(0,P;v\_\{2\}\)\\right\)\\geq V\(0,P;v\_\{1\}\)\.By Proposition[1](https://arxiv.org/html/2605.19215#Thmtheorem1), the bonus inherits this monotonicity at allmm, henceB​\(P,s,v2,γ\)≥B​\(P,s,v1,γ\)B\(P,s,v\_\{2\},\\gamma\)\\geq B\(P,s,v\_\{1\},\\gamma\)\. ∎

## Appendix BDerivation of the CAUSE index

We derive the closed\-form exploration index used in the main text\. The derivation proceeds in four steps: \(i\) marginalize the reward out of the per\-step optimality likelihood, \(ii\) solve the backward recursion under a sigmoid ansatz to obtain a closed\-form infinite\-horizon message, \(iii\) form the posterior over the latent state under future optimality, and \(iv\) extract the index as the posterior mean, replacing a single gating term by a tunable scale\.

### B\.1Generative model

Each arm is governed by a linear\-Gaussian state\-space model with latent dynamics and observation model

p​\(xt\+1∣xt\)=𝒩​\(xt,v\),p​\(rt∣xt\)=𝒩​\(xt,s\)\.p\(x\_\{t\+1\}\\mid x\_\{t\}\)=\\mathcal\{N\}\(x\_\{t\},v\),\\qquad p\(r\_\{t\}\\mid x\_\{t\}\)=\\mathcal\{N\}\(x\_\{t\},s\)\.\(11\)Following the control\-as\-inference framework, we place a binary optimality variable at each step,

p​\(ot=1∣rt\)=σ​\(γt−1​rt\),p\(o\_\{t\}=1\\mid r\_\{t\}\)=\\sigma\(\\gamma^\{t\-1\}r\_\{t\}\),\(12\)whereσ\\sigmais the logistic sigmoid andγ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor\. The agent acts under the eventot:∞=1o\_\{t:\\infty\}=1\.

### B\.2Per\-step optimality after marginalizing the reward

Marginalizing the unobserved reward against the observation model and using the standard probit approximation for the Gaussian\-sigmoid convolution\[[21](https://arxiv.org/html/2605.19215#bib.bib5)\]yields

p​\(ot=1∣xt\)=∫σ​\(γt−1​rt\)​𝒩​\(rt∣xt,s\)​𝑑rt≈σ​\(βt​xt\),p\(o\_\{t\}=1\\mid x\_\{t\}\)\\;=\\;\\int\\sigma\(\\gamma^\{t\-1\}r\_\{t\}\)\\,\\mathcal\{N\}\(r\_\{t\}\\mid x\_\{t\},s\)\\,dr\_\{t\}\\;\\approx\\;\\sigma\(\\beta\_\{t\}\\,x\_\{t\}\),\(13\)with

βt=γt−11\+ϕ​γ2​\(t−1\)​s,ϕ=π8\.\\beta\_\{t\}=\\frac\{\\gamma^\{t\-1\}\}\{\\sqrt\{1\+\\phi\\,\\gamma^\{2\(t\-1\)\}\\,s\}\},\\qquad\\phi=\\frac\{\\pi\}\{8\}\.\(14\)Observation noisessdampens the local optimality signal: largerssweakens the coupling between latent state and optimality at that step\.

### B\.3Backward message recursion

The backward message is the probability of future optimality conditioned on the current latent state,

bt​\(xt\)≡p​\(ot:∞=1∣xt\),b\_\{t\}\(x\_\{t\}\)\\equiv p\(o\_\{t:\\infty\}=1\\mid x\_\{t\}\),\(15\)which satisfies

bt​\(xt\)=p​\(ot=1∣xt\)​∫p​\(xt\+1∣xt\)​bt\+1​\(xt\+1\)​𝑑xt\+1\.b\_\{t\}\(x\_\{t\}\)=p\(o\_\{t\}=1\\mid x\_\{t\}\)\\int p\(x\_\{t\+1\}\\mid x\_\{t\}\)\\,b\_\{t\+1\}\(x\_\{t\+1\}\)\\,dx\_\{t\+1\}\.\(16\)Exact evaluation of this recursion is intractable\. We adopt the ansatz

bt​\(xt\)≈σ​\(αt​xt\),b\_\{t\}\(x\_\{t\}\)\\;\\approx\\;\\sigma\(\\alpha\_\{t\}\\,x\_\{t\}\),\(17\)whereαt\\alpha\_\{t\}is a scalar coefficient to be determined, and track its value through time\. This is in the spirit of fixed\-form variational inference: we restrict the backward message to a parametric family and track the parameter that best represents the true message within that family\.

#### Base case\.

At the terminal horizonTT, the future message is trivial and Eq\. \([16](https://arxiv.org/html/2605.19215#A2.E16)\) reduces to the local likelihood, givingbT​\(xT\)=σ​\(βT​xT\)b\_\{T\}\(x\_\{T\}\)=\\sigma\(\\beta\_\{T\}x\_\{T\}\)and thusαT=βT\\alpha\_\{T\}=\\beta\_\{T\}\.

#### Inductive step\.

Assumebt\+1​\(xt\+1\)≈σ​\(αt\+1​xt\+1\)b\_\{t\+1\}\(x\_\{t\+1\}\)\\approx\\sigma\(\\alpha\_\{t\+1\}x\_\{t\+1\}\)\. Applying the probit approximation to the Gaussian\-sigmoid convolution induced by the latent transition yields

∫𝒩​\(xt\+1∣xt,v\)​σ​\(αt\+1​xt\+1\)​𝑑xt\+1≈σ​\(αt\+1​xt1\+ϕ​v​αt\+12\)\.\\int\\mathcal\{N\}\(x\_\{t\+1\}\\mid x\_\{t\},v\)\\,\\sigma\(\\alpha\_\{t\+1\}x\_\{t\+1\}\)\\,dx\_\{t\+1\}\\;\\approx\\;\\sigma\\\!\\left\(\\frac\{\\alpha\_\{t\+1\}\\,x\_\{t\}\}\{\\sqrt\{1\+\\phi v\\alpha\_\{t\+1\}^\{2\}\}\}\\right\)\.\(18\)The right\-hand side of Eq\. \([16](https://arxiv.org/html/2605.19215#A2.E16)\) is then the product of two sigmoids: the local likelihoodσ​\(βt​xt\)\\sigma\(\\beta\_\{t\}x\_\{t\}\)and the propagated future message\. We project this product back onto the ansatz \([17](https://arxiv.org/html/2605.19215#A2.E17)\) by matching the linear coefficient ofxtx\_\{t\}at the origin, giving the recursion

αt=βt\+αt\+11\+ϕ​v​αt\+12\.\\alpha\_\{t\}\\;=\\;\\beta\_\{t\}\\;\+\\;\\frac\{\\alpha\_\{t\+1\}\}\{\\sqrt\{1\+\\phi v\\alpha\_\{t\+1\}^\{2\}\}\}\.\(19\)Iterating Eq\. \([19](https://arxiv.org/html/2605.19215#A2.E19)\) backward from the terminal valueαT=βT\\alpha\_\{T\}=\\beta\_\{T\}determinesαt\\alpha\_\{t\}at all earlier steps\. Two structural consequences follow\. First, observation noisessentersβt\\beta\_\{t\}and dampens the per\-step contribution to the backward precision\. Second, process noisevvdampens the propagated precisionαt\+1\\alpha\_\{t\+1\}by the factor1\+ϕ​v​αt\+12\\sqrt\{1\+\\phi v\\alpha\_\{t\+1\}^\{2\}\}, so volatility acts as a dual discount factor: it reduces the effective planning horizon by attenuating the backward flow of precision through highly volatile transitions, independently of the geometric discountγ\\gamma\.

### B\.4Closed\-form infinite\-horizon precision

To obtain a closed form, we linearize the recursion by replacing the dynamic denominator by its trapezoidal\-rule average over the horizon:

D=12​\(1\+1\+ϕ​v​\(α∗\)2\),D=\\tfrac\{1\}\{2\}\\left\(1\+\\sqrt\{1\+\\phi v\(\\alpha^\{\*\}\)^\{2\}\}\\right\),\(20\)whereα∗\\alpha^\{\*\}is the maximum accumulated precision\. The recursion linearizes toαt≈βt\+αt\+1/D\\alpha\_\{t\}\\approx\\beta\_\{t\}\+\\alpha\_\{t\+1\}/D, which is geometric and unrolls toα1=∑k=1Tβk/Dk−1\\alpha\_\{1\}=\\sum\_\{k=1\}^\{T\}\\beta\_\{k\}/D^\{k\-1\}\. Therefore, we have

S≡limT→∞α1≈∑k=1∞11\+ϕ​s​γ2​\(k−1\)​\(γD\)k−1\.S\\;\\equiv\\;\\lim\_\{T\\to\\infty\}\\alpha\_\{1\}\\;\\approx\\;\\sum\_\{k=1\}^\{\\infty\}\\frac\{1\}\{\\sqrt\{1\+\\phi s\\,\\gamma^\{2\(k\-1\)\}\}\}\\left\(\\frac\{\\gamma\}\{D\}\\right\)^\{k\-1\}\.\(21\)The summand is slowly varying forγ\\gammanear11, so the sum is well\-approximated by the corresponding integral, which after the substitutionu=\(γ/D\)x−1u=\(\\gamma/D\)^\{x\-1\}reduces to

S≈1ln⁡\(D/γ\)​∫01d​u1\+ϕ​s​u2\.S\\approx\\frac\{1\}\{\\ln\(D/\\gamma\)\}\\int\_\{0\}^\{1\}\\frac\{du\}\{\\sqrt\{1\+\\phi s\\,u^\{2\}\}\}\.\(22\)This integral has a closed form in terms of the inverse hyperbolic sine, but a simpler algebraic approximation suffices for our purposes\. We approximate the integrand’s denominator by its trapezoidal\-rule average over\[0,1\]\[0,1\],12​\(1\+1\+ϕ​s\)\\frac\{1\}\{2\}\(1\+\\sqrt\{1\+\\phi s\}\), accurate since the denominator is nearly linear on this interval\. We further use−ln⁡γ≈1−γ\-\\ln\\gamma\\approx 1\-\\gamma, valid forγ\\gammanear 1, so thatln⁡\(D/γ\)≈ln⁡D\+\(1−γ\)\\ln\(D/\\gamma\)\\approx\\ln D\+\(1\-\\gamma\)\. Substituting into Eq\. \([22](https://arxiv.org/html/2605.19215#A2.E22)\) gives:

S≈2\(ln⁡D\+1−γ\)​\(1\+1\+ϕ​s\)\.S\\approx\\frac\{2\}\{\(\\ln D\+1\-\\gamma\)\(1\+\\sqrt\{1\+\\phi s\}\)\}\.\(23\)EvaluatingDDin turn requiresα∗\\alpha^\{\*\}\. The recursion’s maximum precision is achieved att=1t=1, so self\-consistency would requireα∗=S\\alpha^\{\*\}=S, leading to a fixed\-point equation\. To avoid solving it, we approximateα∗\\alpha^\{\*\}by its undamped \(v=0v=0\) limit, whereD=1D=1and Eq\. \([23](https://arxiv.org/html/2605.19215#A2.E23)\) reduces to

α∗≈2\(1−γ\)​\(1\+1\+ϕ​s\)\.\\alpha^\{\*\}\\approx\\frac\{2\}\{\(1\-\\gamma\)\(1\+\\sqrt\{1\+\\phi s\}\)\}\.\(24\)Substituting Eq\. \([24](https://arxiv.org/html/2605.19215#A2.E24)\) into the definition ofDDcloses the form\.

### B\.5The CAUSE index

At decision time, the predictive belief over the latent state isp​\(xt∣r1:t−1\)=𝒩​\(mt−1,Σt\)p\(x\_\{t\}\\mid r\_\{1:t\-1\}\)=\\mathcal\{N\}\(m\_\{t\-1\},\\Sigma\_\{t\}\)with predictive varianceΣt=Pt−1\+v\\Sigma\_\{t\}=P\_\{t\-1\}\+v\. Conditioning on infinite\-horizon optimality multiplies this prior by the backward message:

p​\(xt∣ot:∞=1,r1:t−1\)∝𝒩​\(xt∣mt−1,Σt\)​σ​\(S​xt\)\.p\(x\_\{t\}\\mid o\_\{t:\\infty\}=1,r\_\{1:t\-1\}\)\\;\\propto\\;\\mathcal\{N\}\(x\_\{t\}\\mid m\_\{t\-1\},\\Sigma\_\{t\}\)\\,\\sigma\(Sx\_\{t\}\)\.\(25\)The product of a Gaussian and a sigmoid is a skew\-normal distribution\. Its mean is obtained from the exponential\-family identity𝔼​\[xt\]=mt−1\+Σt​∂mt−1log⁡Zt\\mathbb\{E\}\[x\_\{t\}\]=m\_\{t\-1\}\+\\Sigma\_\{t\}\\,\\partial\_\{m\_\{t\-1\}\}\\log Z\_\{t\}, whereZtZ\_\{t\}is the normalizer\. The probit approximation gives

Zt≈σ​\(S​mt−11\+ϕ​Σt​S2\),Z\_\{t\}\\;\\approx\\;\\sigma\\\!\\left\(\\frac\{S\\,m\_\{t\-1\}\}\{\\sqrt\{1\+\\phi\\Sigma\_\{t\}S^\{2\}\}\}\\right\),\(26\)and differentiating yields

𝔼​\[xt∣ot:∞=1,r1:t−1\]=mt−1\+Σt​α~t​σ​\(−α~t​mt−1\),\\mathbb\{E\}\[x\_\{t\}\\mid o\_\{t:\\infty\}=1,r\_\{1:t\-1\}\]\\;=\\;m\_\{t\-1\}\+\\Sigma\_\{t\}\\,\\tilde\{\\alpha\}\_\{t\}\\,\\sigma\(\-\\tilde\{\\alpha\}\_\{t\}\\,m\_\{t\-1\}\),\(27\)with uncertainty\-adjusted precision

α~t=S1\+ϕ​Σt​S2\.\\tilde\{\\alpha\}\_\{t\}\\;=\\;\\frac\{S\}\{\\sqrt\{1\+\\phi\\Sigma\_\{t\}S^\{2\}\}\}\.\(28\)The sigmoid factor in Eq\. \([27](https://arxiv.org/html/2605.19215#A2.E27)\) depends onmt−1m\_\{t\-1\}and breaks the additive exploitation\-plus\-bonus decomposition that the optimal index admits\. We replace this gate by a tunable scale parameterc∈\(0,1\)c\\in\(0,1\), recovering an additivem\+Bm\+Bform and yielding the CAUSE index:

λCAUSE=mt−1\+c​Σt​α~t\.\\lambda\_\{\\mathrm\{CAUSE\}\}\\;=\\;m\_\{t\-1\}\+c\\,\\Sigma\_\{t\}\\,\\tilde\{\\alpha\}\_\{t\}\.\(29\)The parametercccontrols the overall magnitude of the exploration bonus and does not affect its monotonicities inssorvv\. One principled default emerges from the gate itself: under the symmetric prior and dynamics \(Section[3](https://arxiv.org/html/2605.19215#S3)\), the marginal distribution ofmt−1m\_\{t\-1\}across histories is symmetric about zero, and the sigmoid satisfiesσ​\(−u\)\+σ​\(u\)=1\\sigma\(\-u\)\+\\sigma\(u\)=1, so the prior\-averaged value of the gate is12\\tfrac\{1\}\{2\}\. We fixc=12c=\\tfrac\{1\}\{2\}throughout this paper\.

## Appendix CExperimental setup

This appendix provides implementation details for the experiments of Section[6](https://arxiv.org/html/2605.19215#S6)\. The scale parametercc\(Eq\.[7](https://arxiv.org/html/2605.19215#S5.E7)\) was held at 0\.5 across all experiments and not tuned per\-condition; reported regret reflects this fixed setting\.

### C\.1Baseline policies

All baselines use the same Kalman tracker as CAUSE, with arm\-specific known\(vk,sk\)\(v\_\{k\},s\_\{k\}\)for filtering\. We letmkm\_\{k\}andPkP\_\{k\}denote the agent’s posterior mean and variance for armkkat the current decision step\.

#### Myopic\.

Selectsarg⁡maxk⁡mk\\arg\\max\_\{k\}m\_\{k\}at each step\. No exploration bonus\.

#### Thompson sampling\[[34](https://arxiv.org/html/2605.19215#bib.bib51)\]\.

At each step, samplesx~k∼𝒩​\(mk,Pk\+vk\)\\tilde\{x\}\_\{k\}\\sim\\mathcal\{N\}\(m\_\{k\},P\_\{k\}\+v\_\{k\}\)from each arm’s predictive distribution over the next latent state and selectsarg⁡maxk⁡x~k\\arg\\max\_\{k\}\\tilde\{x\}\_\{k\}\. The variancePk\+vkP\_\{k\}\+v\_\{k\}is the posterior uncertainty over the arm’s mean reward at the next step, including the contribution of latent drift\.

#### Predictive sampling\[[20](https://arxiv.org/html/2605.19215#bib.bib23)\]\.

Predictive sampling differs from Thompson sampling in its learning target: rather than sampling from the posterior over the current latent state, it samples from a distribution that deprioritizes information whose value will decay due to drift\. Proposition 3 ofLiuet al\.\[[20](https://arxiv.org/html/2605.19215#bib.bib23)\]reduces the AR\(1\) Gaussian case to Gaussian sampling with the same predictive meanmkm\_\{k\}but a shrunken sampling varianceσ~k2\\tilde\{\\sigma\}^\{2\}\_\{k\}\. Specializing their formula to the random\-walk dynamics \(γAR=1\\gamma\_\{\\mathrm\{AR\}\}=1\) used in this paper gives

σ~k2=\(Pk\+vk\)2\(Pk\+vk\)\+xk∗,xk∗=12​\(vk\+vk2\+4​vk​sk\),\\tilde\{\\sigma\}^\{2\}\_\{k\}=\\frac\{\(P\_\{k\}\+v\_\{k\}\)^\{2\}\}\{\(P\_\{k\}\+v\_\{k\}\)\+x^\{\*\}\_\{k\}\},\\qquad x^\{\*\}\_\{k\}=\\tfrac\{1\}\{2\}\\bigl\(v\_\{k\}\+\\sqrt\{v\_\{k\}^\{2\}\+4\\,v\_\{k\}\\,s\_\{k\}\}\\bigr\),\(30\)and the agent samplesx~k∼𝒩​\(mk,σ~k2\)\\tilde\{x\}\_\{k\}\\sim\\mathcal\{N\}\(m\_\{k\},\\tilde\{\\sigma\}^\{2\}\_\{k\}\)and selectsarg⁡maxk⁡x~k\\arg\\max\_\{k\}\\tilde\{x\}\_\{k\}\. The shrinkage factorxk∗x^\{\*\}\_\{k\}is increasing in bothvkv\_\{k\}andsks\_\{k\}, soσ~k2≤Pk\+vk\\tilde\{\\sigma\}^\{2\}\_\{k\}\\leq P\_\{k\}\+v\_\{k\}with equality only in the stationary limitvk→0v\_\{k\}\\to 0, in which predictive sampling and Thompson sampling coincide\[[20](https://arxiv.org/html/2605.19215#bib.bib23), Theorem 1\]\.

Liuet al\.\[[20](https://arxiv.org/html/2605.19215#bib.bib23)\]state their AR\(1\) results under the assumptionγAR<1\\gamma\_\{\\mathrm\{AR\}\}<1, ensuring the latent process admits a steady\-state distribution\. The random\-walk dynamics studied here correspond to theγAR→1\\gamma\_\{\\mathrm\{AR\}\}\\to 1limit, in which Eq\. \([30](https://arxiv.org/html/2605.19215#A3.E30)\) remains well\-defined as the natural extension of their formula, although the accompanying Bayesian regret bound is not directly applicable in this limit\.

#### UCB\[[2](https://arxiv.org/html/2605.19215#bib.bib53),[17](https://arxiv.org/html/2605.19215#bib.bib6)\]\.

Selectsarg⁡maxk⁡\[mk\+c​Pk\+vk\]\\arg\\max\_\{k\}\\bigl\[m\_\{k\}\+c\\sqrt\{P\_\{k\}\+v\_\{k\}\}\\bigr\]withc=2c=2\. This is a Bayesian adaptation of UCB in which the posterior standard deviation over the next\-step latent state plays the role of the confidence radius; withc=2c=2, the bonus corresponds to the upper bound of an approximately 97\.7% one\-sided Gaussian credible interval\. The bonus depends onPk\+vkP\_\{k\}\+v\_\{k\}but not onsks\_\{k\}, so this UCB variant cannot differentiate informative drift from uninformative observation noise\.

#### Oracle\.

Has access to the true latent statesxkx\_\{k\}and selectsarg⁡maxk⁡xk\\arg\\max\_\{k\}x\_\{k\}at each step\. Provides a lower bound on regret\.

### C\.2Computing the optimal Gittins bonus

The optimal Gittins bonus is computed by solving the retirement problem of Section[4](https://arxiv.org/html/2605.19215#S4)numerically, via value iteration with bisection on the retirement salaryλ\\lambda\.

#### Value iteration\.

For a fixedλ\\lambda, the value functionV​\(m,P\)V\(m,P\)satisfies the Bellman equation

V​\(m,P\)=max⁡\{λ1−γ,m\+γ​𝔼m′​\[V​\(m′,P′\)\]\},V\(m,P\)=\\max\\left\\\{\\frac\{\\lambda\}\{1\-\\gamma\},\\;m\+\\gamma\\,\\mathbb\{E\}\_\{m^\{\\prime\}\}\[V\(m^\{\\prime\},P^\{\\prime\}\)\]\\right\\\},\(31\)where\(m′,P′\)\(m^\{\\prime\},P^\{\\prime\}\)are the posterior moments after one Kalman update\. The next\-step posterior varianceP′=\(P\+v\)​s/\(P\+v\+s\)P^\{\\prime\}=\(P\+v\)s/\(P\+v\+s\)is computed in closed form via the Kalman recursion\. The expectation overm′m^\{\\prime\}is evaluated via Gauss–Hermite quadrature with 15 nodes, withm′m^\{\\prime\}centered atmmand standard deviation\(P\+v\)/P\+v\+s\(P\+v\)/\\sqrt\{P\+v\+s\}\. Value iteration is run on a grid of 251 evenly\-spacedmmvalues spanning\[−mrange,mrange\]\[\-m\_\{\\text\{range\}\},m\_\{\\text\{range\}\}\], withmrange=6​Pmax\+v/1−γm\_\{\\text\{range\}\}=6\\sqrt\{P\_\{\\max\}\+v\}/\\sqrt\{1\-\\gamma\}, wherePmaxP\_\{\\max\}is the largest value in thePP\-grid\. ThePP\-grid is logarithmically spaced from0\.010\.01tomax⁡\{50\+10​H​vmax,5​\(vmax\+smax\)\}\\max\\\{50\+10Hv\_\{\\max\},\\;5\(v\_\{\\max\}\+s\_\{\\max\}\)\\\}with 30 points, whereH=1/\(1−γ\)H=1/\(1\-\\gamma\)is the effective horizon\. The constant5050is set conservatively above the bandit’s prior variance \(P0=25P\_\{0\}=25\) to ensure the grid spans the simulation’s posterior range\. Iteration terminates when the maximum value\-function change between iterations falls below10−410^\{\-4\}, or after 300 iterations\.

#### Bisection onλ\\lambda\.

For each value ofPP, the bonus is the smallestλ\\lambdasuch that immediate retirement atm=0m=0is optimal:V​\(0,P\)≤λ/\(1−γ\)V\(0,P\)\\leq\\lambda/\(1\-\\gamma\)\. We find thisλ\\lambdavia bisection on the interval\[0,λmax\]\[0,\\lambda\_\{\\max\}\], withλmax=10​Pmax\+v/1−γ\\lambda\_\{\\max\}=10\\sqrt\{P\_\{\\max\}\+v\}/\\sqrt\{1\-\\gamma\}\. Bisection runs for 20 iterations or until the bracket width falls below10−310^\{\-3\}\. The reported bonus is the midpoint of the final bracket\.

#### Bonus comparison figure\.

For the bonus shape comparison \(Section[6\.2](https://arxiv.org/html/2605.19215#S6.SS2)\), which evaluates the bonus only at fixedPrefP\_\{\\text\{ref\}\}, we use a tighter localPP\-grid \(linearly spaced from0\.5​Pref0\.5P\_\{\\text\{ref\}\}to2​Pref2P\_\{\\text\{ref\}\}with 20 points\), 121mm\-grid points, 11 quadrature nodes, and at most 200 value\-iteration steps; the other tolerances are unchanged\. The reduced settings preserve shape accuracy atPrefP\_\{\\text\{ref\}\}while reducing computation per\(v,s\)\(v,s\)point\.

### C\.3Lesion analysis

The joint inference of\(v^,s^\)\(\\hat\{v\},\\hat\{s\}\), the agent’s running estimates of volatility and stochasticity, uses the framework ofPiray and Daw \[[26](https://arxiv.org/html/2605.19215#bib.bib45)\], with each agent differing only in two sensitivity parametersλv,λs∈\[0,1\]\\lambda\_\{v\},\\lambda\_\{s\}\\in\[0,1\]that control the update rate of the corresponding estimate\. FollowingPiray and Daw \[[26](https://arxiv.org/html/2605.19215#bib.bib45)\], we setλv=λs=0\.1\\lambda\_\{v\}=\\lambda\_\{s\}=0\.1for the healthy agent\. Blind agents are defined by setting the corresponding parameter to0, removing the agent’s ability to attribute experienced noise to that source and pinning the corresponding estimate \(s^\\hat\{s\}orv^\\hat\{v\}\) at its initial value\. Initial values for bothv^\\hat\{v\}ands^\\hat\{s\}are set to the midpoint of the true values, followingPiray and Daw \[[26](https://arxiv.org/html/2605.19215#bib.bib45)\]\.

For each lesion variant and each cell of the\(v,s\)\(v,s\)grid, the agent observes 200 trials of reward data generated under the true noise parameters, runs the joint inference, and reports the inferred\(v^,s^\)\(\\hat\{v\},\\hat\{s\}\)at the end of the trial sequence\. These estimates are then used to compute the CAUSE bonus at the fixed reference variance\. The framework’s reference implementation is available under the MIT license\.

### C\.4Bandit configuration

All experiments use Gaussian state\-space bandits with latent dynamicsxt\+1∣xt∼𝒩​\(xt,v\)x\_\{t\+1\}\\mid x\_\{t\}\\sim\\mathcal\{N\}\(x\_\{t\},v\)and observation modelrt∣xt∼𝒩​\(xt,s\)r\_\{t\}\\mid x\_\{t\}\\sim\\mathcal\{N\}\(x\_\{t\},s\), with arm\-specific noise parameters\(vk,sk\)\(v\_\{k\},s\_\{k\}\)\. Latent states are initialized asx0∼𝒩​\(0,P0\)x\_\{0\}\\sim\\mathcal\{N\}\(0,P\_\{0\}\)independently for each arm, withP0=25P\_\{0\}=25\. Each arm’s true initial mean is therefore drawn from the agent’s prior, providing nontrivial reward differentiation across arms regardless of volatility\. The discount factor in all main analyses isγ=0\.95\\gamma=0\.95\. Sensitivity toγ\\gammais reported in Appendix[E](https://arxiv.org/html/2605.19215#A5)\.

#### Heterogeneous\-arms regret experiment\.

The experiments of Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)useK=4K=4arms overT=200T=200timesteps in three regimes\. The*mixed*regime partitions arms equally across the four cells of the\(v,s\)\(v,s\)grid withvlow=1v\_\{\\text\{low\}\}=1,vhigh=4v\_\{\\text\{high\}\}=4,slow=9s\_\{\\text\{low\}\}=9,shigh=25s\_\{\\text\{high\}\}=25\. The*s\-dominant*regime fixesv=4v=4across all arms, withs=9s=9for two arms ands=900s=900for two arms\. The*v\-dominant*regime fixess=25s=25across all arms, withv=1v=1for two arms andv=100v=100for two arms\. Sensitivity to the\(v,s\)\(v,s\)scale, the discount factorγ\\gamma, and the number of armsKKis reported in Appendix[E](https://arxiv.org/html/2605.19215#A5)\.

#### Bonus comparison experiment\.

The experiment of Section[6\.2](https://arxiv.org/html/2605.19215#S6.SS2)sweeps one noise parameter while holding the other fixed at the high value of the mixed regime \(v=4v=4when sweepingss;s=25s=25when sweepingvv\)\. Each axis is swept over the range\[10,1000\]\[10,1000\]at 14 logarithmically spaced points\. The posterior variance is held atPrefP\_\{\\text\{ref\}\}, set separately for each axis to the median, across the swept range, of the stationary posterior varianceP∞​\(v,s\)=\(v2\+4​v​s−v\)/2P\_\{\\infty\}\(v,s\)=\(\\sqrt\{v^\{2\}\+4vs\}\-v\)/2implied by the Kalman recursion\.

#### Rested verification atv=0v=0\.

The verification of Section[6\.3](https://arxiv.org/html/2605.19215#S6.SS3)setsv=0v=0for allK=4K=4arms and tests two stochasticity configurations:s∈\{9,25\}s\\in\\\{9,25\\\}\(thessvalues from the mixed regime\) ands∈\{9,900\}s\\in\\\{9,900\\\}\(from the s\-dominant regime\), with two arms at eachsslevel\. Other settings \(T=200T=200,γ=0\.95\\gamma=0\.95,10001000Monte Carlo runs\) match Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)\.

#### Lesion analysis\.

The lesion analysis of Section[6\.4](https://arxiv.org/html/2605.19215#S6.SS4)uses the same\(v,s\)\(v,s\)grid as the mixed regime \(v∈\{1,4\}v\\in\\\{1,4\\\},s∈\{9,25\}s\\in\\\{9,25\\\}, one arm per cell,K=4K=4\)\. The reference posterior variancePrefP\_\{\\text\{ref\}\}is the median across the four arms of the stationary posterior varianceP∞​\(v,s\)=\(v2\+4​v​s−v\)/2P\_\{\\infty\}\(v,s\)=\(\\sqrt\{v^\{2\}\+4vs\}\-v\)/2implied by the Kalman recursion\. Other settings \(T=200T=200,γ=0\.95\\gamma=0\.95,10001000Monte Carlo runs\) match Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)\.

#### Monte Carlo procedure\.

All experiments are averaged over 1000 independent runs with different random seeds\. Reported quantities are mean±\\pmstandard error of the mean \(SEM\)\.

### C\.5Compute resources

All experiments were run on CPUs\. Each independent run completes in seconds, and full experiments \(1000 Monte Carlo runs across all conditions\) complete in a few minutes per experiment on a standard laptop\. Total compute for all experiments and preliminary explorations is well under one CPU core day\.

## Appendix DExtended main empirical results

This appendix presents supplementary results extending the main empirical claims of Sections[6\.2](https://arxiv.org/html/2605.19215#S6.SS2)and[6\.3](https://arxiv.org/html/2605.19215#S6.SS3)\.

### D\.1Bonus shape on the volatility axis

Figure[4](https://arxiv.org/html/2605.19215#A4.F4)extends the bonus comparison of Section[6\.2](https://arxiv.org/html/2605.19215#S6.SS2)to the volatility axis: each policy’s bonus as a function ofvv, withssfixed at2525andPrefP\_\{\\text\{ref\}\}as described in Appendix[C\.4](https://arxiv.org/html/2605.19215#A3.SS4)\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/bonus_v.png)Figure 4:Exploration bonus as a function of volatilityvvat fixeds=25s=25, normalized to a common range\. CAUSE, UCB, and Gittins all show the correct increasing direction; CAUSE lies modestly below Gittins\.All three policies increase withvv, as the structural analysis prescribes\. CAUSE lies modestly below Gittins, reflecting the additional volatility\-aware damping \(theDD\-factor; see Appendix[B](https://arxiv.org/html/2605.19215#A2)\) that accounts for the diminishing value of distant\-future information when latent states drift\. UCB tracks Gittins closely along this axis since UCB’sP\+v\\sqrt\{P\+v\}scaling implicitly captures the volatility dependence, even though it does not capture stochasticity dependence \(Section[6\.2](https://arxiv.org/html/2605.19215#S6.SS2)\)\. The differences along the volatility axis are subtle in bonus shape but produce substantial differences in regret in the v\-dominant regime \(Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)\)\.

### D\.2Rested verification at extreme stochasticity

Section[6\.3](https://arxiv.org/html/2605.19215#S6.SS3)reports the rested verification \(v=0v=0\) at moderate stochasticity heterogeneity \(s∈\{9,25\}s\\in\\\{9,25\\\}\)\. Figure[5](https://arxiv.org/html/2605.19215#A4.F5)extends the verification to extreme stochasticity heterogeneity \(s∈\{9,900\}s\\in\\\{9,900\\\}\), matching the high\-ssvalue of the s\-dominant regime\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/rested.png)Figure 5:Cumulative discounted regret atv=0v=0forK=4K=4arms overT=200T=200steps,γ=0\.95\\gamma=0\.95,10001000Monte Carlo runs\.*Left*:s∈\{9,25\}s\\in\\\{9,25\\\}\.*Right*:s∈\{9,900\}s\\in\\\{9,900\\\}\. CAUSE and Gittins\-per\-arm overlap within Monte Carlo precision in both configurations\.At extreme stochasticity, CAUSE achieves42\.71±2\.4042\.71\\pm 2\.40and Gittins\-per\-arm achieves41\.36±2\.0441\.36\\pm 2\.04\(mean±\\pmSEM, 1000 runs\); the difference of1\.351\.35is well within the combined sampling error of∼3\.15\\sim 3\.15\. The closeness of CAUSE to the rested\-optimal Gittins reference is therefore preserved across the stochasticity range tested in the main experiments\.

## Appendix ERobustness

### E\.1Discount factor

We replicate the mixed\-regime experiment at three additional discount factors,γ∈\{0\.8,0\.9,0\.98\}\\gamma\\in\\\{0\.8,0\.9,0\.98\\\}\. The qualitative ordering of policies is preserved across all values: CAUSE achieves the lowest regret, followed by Gittins\-per\-arm, with UCB and the posterior\-sampling baselines further behind\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/discount.png)Figure 6:Cumulative discounted regret in the mixed regime across discount factorsγ∈\{0\.8,0\.9,0\.98\}\\gamma\\in\\\{0\.8,0\.9,0\.98\\\}\(K=4K=4,T=200T=200,10001000Monte Carlo runs\)\.
### E\.2Number of arms

We replicate the mixed\-regime experiment atK∈\{8,12,16\}K\\in\\\{8,12,16\\\}arms, with arms distributed equally across the four cells of the\(v,s\)\(v,s\)grid\. The CAUSE\-Gittins gap, which favors CAUSE atK=4K=4\(Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)\), narrows asKKgrows: atK=8K=8, CAUSE remains modestly ahead; atK=12K=12, the two policies are statistically tied; atK=16K=16, Gittins\-per\-arm is modestly ahead\. CAUSE uses a fixedc=0\.5c=0\.5\(Equation[7](https://arxiv.org/html/2605.19215#S5.E7)\) across all conditions; the narrowing trend suggests that allowingccto vary withKKcould preserve the lead at largerKK, but we do not pursue this tuning\. The qualitative advantages of CAUSE over UCB and the posterior\-sampling baselines are preserved at all values ofKK\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/regret_K.png)Figure 7:Cumulative discounted regret in the mixed regime atK∈\{8,12,16\}K\\in\\\{8,12,16\\\}\(T=200T=200,γ=0\.95\\gamma=0\.95,10001000Monte Carlo runs\)\.
### E\.3UCB exploration constant

The UCB baseline in Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)uses the canonicalc=2c=2scaling, corresponding to the standard2​σ2\\sigmaupper confidence bound of Bayesian UCB\[[17](https://arxiv.org/html/2605.19215#bib.bib6)\]\. UCB allocates exploration in proportion toc​P\+vc\\sqrt\{P\+v\}, scaling with predictive variance and not with stochasticityss\. This structural insensitivity tossmeans that no choice ofcccan produce the s\-axis discounting that the optimal exploration bonus exhibits \(Sections[4](https://arxiv.org/html/2605.19215#S4)and[6\.2](https://arxiv.org/html/2605.19215#S6.SS2)\); UCB therefore over\-explores high\-ssarms regardless of the global exploration scale\.

We verify this structurally by sweepingc∈\{0\.5,1,2,3\}c\\in\\\{0\.5,1,2,3\\\}across the three regimes of Section[6\.1](https://arxiv.org/html/2605.19215#S6.SS1)\. Two patterns emerge\. First, the optimalccfor UCB varies across regimes: the best\-performing value is smaller in s\-dominant \(c=0\.5c=0\.5\) than in v\-dominant or mixed regimes \(c=1c=1\)\. A single fixedcctherefore cannot be globally optimal for UCB\. Second, even at its best per\-regimecc, UCB does not match CAUSE in any regime, with the gap largest in s\-dominant\. The structural insensitivity is visible directly in the bonus comparison \(Figure[2](https://arxiv.org/html/2605.19215#S6.F2), left\): UCB’s bonus is exactly flat as a function ofss, while CAUSE and Gittins decrease monotonically\. This is a structural feature of UCB’s allocation rule and cannot be repaired by rescalingcc\.

![Refer to caption](https://arxiv.org/html/2605.19215v1/figures/regret_ucb.png)Figure 8:UCB regret across the three regimes forc∈\{0\.5,1,2,3\}c\\in\\\{0\.5,1,2,3\\\}, with CAUSE shown for comparison \(K=4K=4,T=200T=200,γ=0\.95\\gamma=0\.95,10001000Monte Carlo runs\)\.