A Bellman Optimality Equation for Plasticity

# A Bellman Optimality Equation for Plasticity
Source: [https://arxiv.org/html/2609.10776](https://arxiv.org/html/2609.10776)
###### Abstract

In continual reinforcement learning, carefully managing the stability\-plasticity tradeoff remains a core challenge\. Recent work by[Abel et al\. \(2025\)](https://arxiv.org/html/2609.10776#bib.bib1)formalized this dilemma by defining plasticity as the generalized directed information from an agent’s observations to its actions, and empowerment as the generalized directed information from its actions to its observations\. This formulation successfully reframes the traditional stability\-plasticity tradeoff as an empowerment\-plasticity tradeoff\. However, while extensive literature exists on optimizing for empowerment, there is currently no research addressing the optimization of plasticity under this new definition\. This paper presents preliminary work toward optimizing plasticity within Markov decision processes\. We show that there exists a Bellman optimality equation for optimizing plasticity similar to previous work for empowerment\.

## 1Introduction

Conceptually, empowerment is interpreted as an agent’s ability to control its future, whereas plasticity represents its capacity to remain malleable by the environment\. Recently,[Abel et al\. \(2025\)](https://arxiv.org/html/2609.10776#bib.bib1)connected these two domains by formalizing plasticity as the mirror of empowerment\. While empowerment describes controllability by maximizing the information flowing from an agent’s actions to its subsequent observations,[Abel et al\. \(2025\)](https://arxiv.org/html/2609.10776#bib.bib1)defined plasticity as the information flowing in the opposite direction\. They theorize that designing robust continual learning agents requires maintaining both empowerment and plasticity above critical thresholds, and this claim is the motivation for this work\.

To build algorithms that address this claim, both empowerment and plasticity must be actively optimized and controlled\. The literature presents numerous methods for maximizing empowerment, ranging from its introduction by[Klyubin et al\. \(2005\)](https://arxiv.org/html/2609.10776#bib.bib2)to the Bellman formulation of[Leibfried et al\. \(2019\)](https://arxiv.org/html/2609.10776#bib.bib3)and the deep variational approaches of[Mohamed and Rezende \(2015\)](https://arxiv.org/html/2609.10776#bib.bib4)\. However, plasticity optimization remains unexplored\. Plasticity has primarily been studied through the lens of loss of plasticity in deep architectures\([Lyle et al\., 2023](https://arxiv.org/html/2609.10776#bib.bib5);[Dohare et al\., 2024](https://arxiv.org/html/2609.10776#bib.bib7)\), and is often addressed using architectural or training methods\([Dohare et al\., 2021](https://arxiv.org/html/2609.10776#bib.bib8)\)\. Consequently, no existing work has formalized the direct optimization of the new information\-theoretic definition\. In this paper, we bridge this gap by deriving a Bellman optimality equation for plasticity\.

The remainder of this paper is structured as follows: Section 2 establishes the necessary mathematical preliminaries; Section 3 details the formal derivation of our Bellman optimality equation; Section 4 introduces our benchmark environment; and Section 5 presents empirical evaluations across both the benchmark and two scaled\-up environments\.

## 2Background

### 2\.1Empowerment and Plasticity via Directed Information

From[Klyubin et al\. \(2005\)](https://arxiv.org/html/2609.10776#bib.bib2)the formal definition of empowerment is the channel capacity between an agent’s action channel and the next state that follows these actions\.

ℰt​\(St\)=maxp⁡\(Atn\)⁡I⁡\(Atn;St\+n∣St\)\\mathcal\{E\}\_\{t\}\(S\_\{t\}\)=\\max\_\{p\(A\_\{t\}^\{n\}\)\}I\(A\_\{t\}^\{n\};S\_\{t\+n\}\\mid S\_\{t\}\)\(1\)
This requires an agent with an open loop system that can produce and executennactions every state\.[Capdepuy \(2011\)](https://arxiv.org/html/2609.10776#bib.bib6)extended this framework by incorporating a feedback loop\. They redefined empowerment as the directed information from the agent’s actions to its subsequent observations\.

I⁡\(AtN→Ot\+1N\)=∑n=0N−1I⁡\(Atn\+1;Ot\+1\+n∣Ot\+1n\)I\(A\_\{t\}^\{N\}\\rightarrow O\_\{t\+1\}^\{N\}\)=\\sum\_\{n=0\}^\{N\-1\}I\(A\_\{t\}^\{n\+1\};O\_\{t\+1\+n\}\\mid O\_\{t\+1\}^\{n\}\)\(2\)
More recently,[Abel et al\. \(2025\)](https://arxiv.org/html/2609.10776#bib.bib1)built off of the definition and ideas from[Capdepuy \(2011\)](https://arxiv.org/html/2609.10776#bib.bib6)and defined empowerment through general directed information\. They define the empowerment of an agent architectureΛ\\Lambdawithin an environmenteerelative to an action window\[a:b\]\[a:b\]and observation window\[c:d\]\[c:d\]as:

𝔈c:da:b\(Λ,e\)≜maxλ∈ΛI\(Aa:b→Oc:d\)\\mathfrak\{E\}\_\{c:d\}^\{a:b\}\(\\Lambda,e\)\\triangleq\\max\_\{\\lambda\\in\\Lambda\}I\(A\_\{a:b\}\\rightarrow O\_\{c:d\}\)\(3\)
Similar to empowerment,[Abel et al\. \(2025\)](https://arxiv.org/html/2609.10776#bib.bib1)define the plasticity of an agentλ\\lambdawithin a set of environmentsℰ\\mathcal\{E\}relative to an observation window\[a:b\]\[a:b\]and action window\[c:d\]\[c:d\]as:

𝔅c:da:b\(λ,ℰ\)≜maxe∈ℰI\(Oa:b→Ac:d\)\\mathfrak\{B\}\_\{c:d\}^\{a:b\}\(\\lambda,\\mathcal\{E\}\)\\triangleq\\max\_\{e\\in\\mathcal\{E\}\}I\(O\_\{a:b\}\\rightarrow A\_\{c:d\}\)\(4\)
In this paper we are interested in their definition with a single agentλ\\lambdainteraction with a specific environmentee\.

### 2\.2Value\-Based Information Maximization

While directed information provides a complete formalism to analyze these concepts, it is very difficult to optimize directly in Markov decision processes\. Value based methods proposed in[Leibfried et al\. \(2019\)](https://arxiv.org/html/2609.10776#bib.bib3);[Tiomkin and Tishby \(2018\)](https://arxiv.org/html/2609.10776#bib.bib9)formulate generalized Bellman equations to make the optimization for empowerment tractable\.[Leibfried et al\. \(2019\)](https://arxiv.org/html/2609.10776#bib.bib3)introduced a unified framework for the maximization of reward and empowerment\. They introduce two parameters to control focus on reward versus empowerment\. They define an empowerment Bellman optimality equation given as the following:

V∗\(s\)=maxπbehave,q𝔼a∼πbehave\(⋅∣s\)\[αR\(s,a\)\+𝔼s′∼P\(⋅∣s,a\)\[βlog\(q⁡\(a∣s′,s\)πbehave​\(a∣s\)\)\+γV∗\(s′\)\]\]V^\{\*\}\(s\)=\\max\_\{\\pi\_\{\\text\{behave\}\},q\}\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\text\{behave\}\}\(\\cdot\\mid s\)\}\\left\[\\alpha R\(s,a\)\+\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\\left\[\\beta\\log\\left\(\\frac\{q\(a\\mid s^\{\\prime\},s\)\}\{\\pi\_\{\\text\{behave\}\}\(a\\mid s\)\}\\right\)\+\\gamma V^\{\*\}\(s^\{\\prime\}\)\\right\]\\right\]\(5\)
We can now develop a Bellman optimality equation for plasticity similar to this formulation\. We focus on the scenario where there are no external rewards \(orα\\alphais set to00\) to isolate and examine the information\-theoretic value of plasticity and the resulting policies without the influence of external rewards\.

## 3The Bellman Optimality Equation for Plasticity

### 3\.1Plasticity and Empowerment Value Channels

In[Leibfried et al\. \(2019\)](https://arxiv.org/html/2609.10776#bib.bib3), they create a value function that isolates the present and future information\. GivenStS\_\{t\}, they useI⁡\(At,St\+1∣St\)I\(A\_\{t\},S\_\{t\+1\}\\mid S\_\{t\}\)as the step reward andV⁡\(St\+1\)V\(S\_\{t\+1\}\)as the future reward which looks like the following\.

I⁡\(At;St\+1∣St\)I\(A\_\{t\};S\_\{t\+1\}\\mid S\_\{t\}\)γ​V​\(St\+1\)\\gamma V\(S\_\{t\+1\}\)StS\_\{t\}St\+1S\_\{t\+1\}St\+2S\_\{t\+2\}…\\dotsSt\+nS\_\{t\+n\}AtA\_\{t\}At\+1A\_\{t\+1\}At\+2A\_\{t\+2\}At\+n−1A\_\{t\+n\-1\}

In this paper we formulate the equivalent counterpart for plasticity\. We use the step rewardI\(St\+1,At\+1\|St,At\)I\(S\_\{t\+1\},A\_\{t\+1\}\|S\_\{t\},A\_\{t\}\)and the future contextV⁡\(St\+1,At\+1\)V\(S\_\{t\+1\},A\_\{t\+1\}\)as the future value\.

I\(St\+1;At\+1∣St,At\)I\(S\_\{t\+1\};A\_\{t\+1\}\\mid S\_\{t\},A\_\{t\}\)γ​V​\(St\+1,At\+1\)\\gamma V\(S\_\{t\+1\},A\_\{t\+1\}\)StS\_\{t\}St\+1S\_\{t\+1\}St\+2S\_\{t\+2\}…\\dotsSt\+nS\_\{t\+n\}AtA\_\{t\}At\+1A\_\{t\+1\}At\+2A\_\{t\+2\}At\+n−1A\_\{t\+n\-1\}

### 3\.2The Bellman Optimality Equation Definition

We now introduce the Bellman optimality equation for plasticity\. We establish its theoretical foundations by proving that the associated Bellman operator is a contraction, with the full proof deferred to Appendix A\.

V∗​\(st−1,at−1\)=maxπ∈Π⁡\(st−1,at−1\)⁡𝔼st∼p​\[𝔼at∼π​\[log⁡π⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γ​V∗​\(st,at\)\]\]V^\{\*\}\(s\_\{t\-1\},a\_\{t\-1\}\)=\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V^\{\*\}\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\(6\)
Here,Π⁡\(st−1,at−1\)\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)denotes the set of policies that match the current policy everywhere except for the non\-zero values ofp\(⋅∣st−1,at−1\)p\(\\cdot\\mid s\_\{t\-1\},a\_\{t\-1\}\)\. Theqqmarginal is defined below:

q⁡\(at∣st−1,at−1\)=∑stp⁡\(st∣st−1,at−1\)⋅π⁡\(at∣st−1,at−1,st\)q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)=\\sum\_\{s\_\{t\}\}p\(s\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\\cdot\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\(7\)
To apply this Bellman value function in a tabular learning setting, the policy must be augmented with the historical state and action\. For each history context, the action distributions for all next statessts\_\{t\}that can arise fromp⁡\(st∣st−1,at−1\)p\(s\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)must be optimized jointly due to the coupling from theqqmarginal\. A tractable computation for this is described below\.

### 3\.3Algorithm Optimization

The Bellman operator can be restricted to deterministic policies, as proven in the appendix\. Therefore, the search set for each history context consists of the max over the distinct combinations of deterministic action distributions\. This requires searching through\|A\|\|S\|\|\{\}A\|\{\}^\{\|\{\}S\|\{\}\}combinations, where\|A\|\|\{\}A\|\{\}is the number of actions and\|S\|\|\{\}S\|\{\}is the number of distinct states arising from each history context\. This search space becomes rapidly intractable and a further optimization must be made\.

### 3\.4Multi\-Weight Knapsack Dynamic Programming

Because the marginal distributionq⁡\(at∣st−1,at−1\)q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)is shared, the action choices across all next\-statessts\_\{t\}are coupled and cannot be optimized individually\. However, this joint search space can be significantly reduced by framing the optimization as a Multi\-Weight Knapsack problem\.

In this formulation, each next\-statests\_\{t\}acts as an item for which we must select exactly one actionata\_\{t\}\. Choosing actionata\_\{t\}contributes a "weight" equal to the transition probabilityp⁡\(st∣st−1,at−1\)p\(s\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)to the corresponding coordinate of theqqmarginal, while yielding a linear future value ofγ​V​\(st,at\)\\gamma V\(s\_\{t\},a\_\{t\}\)\. The dimensions of our knapsack track the accumulated weight distribution across the action space\. By using dynamic programming, we track identical accumulated weight states across action\-selection paths, allowing us to prune suboptimal combinations and stop redundant computation\.

Algorithm 1Tabular Value Iteration for PlasticityInput:State space

𝒮\\mathcal\{S\}, Action space

𝒜\\mathcal\{A\}, transition probabilities

p⁡\(st∣st−1,at−1\)p\(s\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\), discount factor

γ\\gamma, convergence threshold

θ\>0\\theta\>0
Output:Optimal value function

V∗​\(s,a\)V^\{\*\}\(s,a\)and optimal policy

π∗​\(at∣st−1,at−1,st\)\\pi^\{\*\}\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)
1Initialize

V⁡\(s,a\)←0V\(s,a\)\\leftarrow 0for all

\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}
2Initialize

π⁡\(at∣st−1,at−1,st\)←𝟏​\(at=a∗\)\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\\leftarrow\\mathbf\{1\}\(a\_\{t\}=a^\{\*\}\)where

a∗∼U⁡\(𝒜\)a^\{\*\}\\sim U\(\\mathcal\{A\}\), for all

\(st−1,at−1,st,at\)∈𝒮×𝒜×𝒮×𝒜\(s\_\{t\-1\},a\_\{t\-1\},s\_\{t\},a\_\{t\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\times\\mathcal\{A\}
3repeat

4

Δ←0\\Delta\\leftarrow 0
5foreach*\(st−1,at−1\)∈𝒮×𝒜\(s\_\{t\-1\},a\_\{t\-1\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}*do

6

v←V⁡\(st−1,at−1\)v\\leftarrow V\(s\_\{t\-1\},a\_\{t\-1\}\)
7

V\(st−1,at−1\),π∗\(⋅∣st−1,at−1,⋅\)←Solve Multi\-Weight Knapsack DP\(π,V,p,γ\)V\(s\_\{t\-1\},a\_\{t\-1\}\),\\pi^\{\*\}\(\\cdot\\mid s\_\{t\-1\},a\_\{t\-1\},\\cdot\)\\leftarrow\\text\{Solve Multi\-Weight Knapsack DP\}\(\\pi,V,p,\\gamma\)
8

Δ←max⁡\(Δ,\|v−V⁡\(st−1,at−1\)\|\)\\Delta\\leftarrow\\max\(\\Delta,\|v\-V\(s\_\{t\-1\},a\_\{t\-1\}\)\|\)
9end foreach

10until*Δ<θ\\Delta<\\theta*

## 4The Plasticity\-Empowerment Landscape

To build intuition regarding the dynamics of plasticity and empowerment, we introduce a minimal benchmark environment designed to isolate unique policy interactions between the two metrics\. The simplest architecture identified that retains a rich, non\-trivial plasticity\-empowerment landscape is the Markov Decision Process illustrated in Figure[1](https://arxiv.org/html/2609.10776#S4.F1)\.

S0S\_\{0\}S1S\_\{1\}a0,1:0\.5a\_\{0,1\}:0\.5a0,1:0\.5a\_\{0,1\}:0\.5a0:1\.0a\_\{0\}:1\.0a1:1\.0a\_\{1\}:1\.0Figure 1:The Control\-Gated MDP Benchmark\.We designate this environment as theControl\-Gated MDP\. Despite its minimal size, it exhibits several compelling properties for analysis:

1. 1\.Computational Efficiency:Consisting of only two states and two actions, it allows for exact, rapid numerical optimization and complete policy enumeration\.
2. 2\.Asymmetric Controllability:It features a highly empowering state \(S1S\_\{1\}\) alongside a non\-empowering state \(S0S\_\{0\}\)\.

By grid\-searching across all valid stochastic policiesπ⁡\(a∣s\)\\pi\(a\\mid s\)at a granularity of0\.010\.01, we map the discretized, empirical plasticity\-empowerment rate landscape shown in Figure[2](https://arxiv.org/html/2609.10776#S4.F2)\. The plasticity and empowerment is a measure of total directed information from states to actions and vice\-versa for the given policy and environment\. Although both metrics are dependent on the environment, the maximum information rate \(in bits\) for plasticity is bounded bylog2⁡\|A\|\\log\_\{2\}\|A\|while for empowerment it is bounded bylog2⁡\|S\|\\log\_\{2\}\|S\|\.

![Refer to caption](https://arxiv.org/html/2609.10776v1/figures/env.png)Figure 2:The empirical plasticity\-empowerment frontier mapped via policy enumeration\.The plasticity\-maximizing policyπplastic∗\\pi^\{\*\}\_\{\\text\{plastic\}\}selects actions deterministically, withπplastic∗​\(1∣0\)=1\\pi^\{\*\}\_\{\\text\{plastic\}\}\(1\\mid 0\)=1andπplastic∗​\(0∣1\)=1\\pi^\{\*\}\_\{\\text\{plastic\}\}\(0\\mid 1\)=1\. Conversely, the empowerment\-maximizing policyπempower∗\\pi^\{\*\}\_\{\\text\{empower\}\}exhibits more distributed behavior: it acts uniformly in state00, yieldingπempower∗​\(a∣0\)=12\\pi^\{\*\}\_\{\\text\{empower\}\}\(a\\mid 0\)=\\frac\{1\}\{2\}fora∈\{0,1\}a\\in\\\{0,1\\\}, while in state11it selects actions with probabilitiesπempower∗​\(0∣1\)=13\\pi^\{\*\}\_\{\\text\{empower\}\}\(0\\mid 1\)=\\frac\{1\}\{3\}andπempower∗​\(1∣1\)=23\\pi^\{\*\}\_\{\\text\{empower\}\}\(1\\mid 1\)=\\frac\{2\}\{3\}\. Visually, this divergence is reflected in the policy landscape: highly plastic policies occupy the blue regions of the plot, whereas empowering policies are characterized by a mixture of blue and red regions\.

## 5Experiments

### 5\.1Control\-Gated MDP

We evaluate our Bellman optimality equation in the Control\-Gated MDP\. We record each history context value and the current plasticity and empowerment at each iteration\.

Figure 3:Results from the Control\-Gated MDP BenchmarkFrom our results, it is clear that the value iteration is monotonically non\-decreasing and converges to the optimal value function across all history contexts\. Intuitively, the history contexts with deterministic transitions,\(1,0\)\(1,0\)and\(1,1\)\(1,1\), yield the lowest values\. Among these,\(1,1\)\(1,1\)results in the minimum value because it transitions back to state11, where the agent again faces low\-value prospects, while\(1,0\)\(1,0\)leads to state00\.

The empowerment and plasticity results show that the policy ultimately converges to approximately0\.920\.92bits of plasticity and00bits of empowerment\. The non\-zero standard deviation observed in the early iterations is a consequence of the random selection among multiple optimal policies that share the same value but vary in their total plasticity\. This highlights the discrepancy between our Bellman optimality equation and the directed information definition of plasticity\.

### 5\.2Road Environment

The Road Environment consists of 5 states\{−2,−1,0,1,2\}\\\{\-2,\-1,0,1,2\\\}, signifying the five lanes of a road, and 3 actions\{−1,0,1\}\\\{\-1,0,1\\\}, describing the directions the agent can turn\. At each step, the agent selects an action and the environmental wind is sampled uniformly from\{−1,0,1\}\\\{\-1,0,1\\\}\. The agent’s action and the wind are added together to determine the next state of the agent\. The transition dynamics are bounded such that neither the agent’s action nor the wind can cause the agent to leave the road\. The agent starts in a random lane\.

Figure 4:Road EnvironmentFigure 5:Results from Road EnvironmentFrom these results, attempting to leave the road—corresponding to history contexts\(0,0\)\(0,0\)and\(4,2\)\(4,2\)—yields the least amount of value\. This is followed by staying straight in the outer lanes,\(0,1\)\(0,1\)and\(4,1\)\(4,1\), along with moving to the outside from the second inner lanes,\(1,0\)\(1,0\)and\(3,2\)\(3,2\)\. Interestingly, the initial "None" history, where no steps have yet occurred, exhibits a very slightly less plastic value than the inner dynamics of the environment\.

### 5\.3Two\-Room Environment

This final environment is a5×75\\times 7gridworld containing3535states and a44\-directional action space, consisting of two distinct rooms\. The first room is fully noisy, where all actions result in a transition to a random state within that same room\. The second room is fully deterministic, allowing the agent to navigate freely\. At the beginning of the episode, the agent must choose which bridge to cross to enter its respective room\. If the agent fails to cross a bridge, they fall into a river that continuously flows downward toward the bottom of the grid, where the episode terminates\. While the agent is in the river, all state transitions are completely deterministic, and the agent’s actions have no effect on their downward flow\.

Figure 6:Results from the Two\-Room Environment![Refer to caption](https://arxiv.org/html/2609.10776v1/simulations.png)Figure 7:Policy Simulation in the Two\-Room EnvironmentIn this gridworld environment, there are too many history contexts to analyze individually, but the policy simulation clearly demonstrates that the agent navigates toward the bridge to the noisy room to capture the maximum plastic value\.

## 6Future Work

A natural progression is to develop an online, sample\-based algorithm to scale effectively to continuous, high\-dimensional environments by parameterizing the policy and value function with deep neural networks\. Finally, this framework can be integrated alongside existing empowerment\-maximization algorithms which will open the door to agents that can dynamically balance plasticity, empowerment, and external reward during continual learning\.

#### Broader Impact Statement

This work introduces a foundational Bellman optimality equation focused on maximizing an agent’s plasticity in Markov decision processes\. Because it operates at a fundamental algorithmic level, it does not carry immediate negative societal impacts or direct applications to sensitive domains\.

## Acknowledgements

We would like to thank Zihan Wang for inspiration for the Road Environment\.

## References

- Abelet al\.\(2025\)D\. Abel, M\. Bowling, A\. Barreto, W\. Dabney, S\. Dong, S\. Hansen, A\. Harutyunyan, K\. Khetarpal, C\. Lyle, R\. Pascanu, G\. Piliouras, D\. Precup, J\. Richens, M\. Rowland, T\. Schaul, and S\. SinghPlasticity as the mirror of empowerment\.arXiv preprint arXiv:2505\.10361\.Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.10776#S2.SS1.p5.1),[§2\.1](https://arxiv.org/html/2609.10776#S2.SS1.p7.1),[Abstract](https://arxiv.org/html/2609.10776#abstract1.1)\.
- Capdepuy \(2011\)P\. CapdepuyInformational principles of perception\-action loops and collective behaviours\.Ph\.D\. Thesis,University of Hertfordshire\.Cited by:[§2\.1](https://arxiv.org/html/2609.10776#S2.SS1.p3.1),[§2\.1](https://arxiv.org/html/2609.10776#S2.SS1.p5.1)\.
- Dohareet al\.\(2024\)S\. Dohare, J\. F\. Hernandez\-Garcia, Q\. Lan, P\. Rahman, A\. R\. Mahmood, and R\. S\. SuttonLoss of plasticity in deep continual learning\.Nature632\(8026\),pp\. 768–774\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07711-7)Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p2.1)\.
- Dohareet al\.\(2021\)S\. Dohare, R\. S\. Sutton, and A\. R\. MahmoodContinual backprop: stochastic gradient descent with persistent randomness\.arXiv preprint arXiv:2108\.06325\.Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p2.1)\.
- Klyubinet al\.\(2005\)A\. S\. Klyubin, D\. Polani, and C\. L\. NehanivEmpowerment: a universal agent\-centric measure of control\.InProceedings of the IEEE Congress on Evolutionary Computation,Vol\.1,pp\. 214–221\.Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.10776#S2.SS1.p1.1)\.
- Leibfriedet al\.\(2019\)F\. Leibfried, S\. Pascual\-Díaz, and J\. Grau\-MoyaA unified Bellman optimality principle combining reward maximization and empowerment\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 7867–7878\.Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.10776#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2609.10776#S3.SS1.p1.1)\.
- Lyleet al\.\(2023\)C\. Lyle, Z\. Zheng, E\. Nikishin, B\. A\. Pires, R\. Pascanu, and W\. DabneyUnderstanding plasticity in neural networks\.InInternational Conference on Machine Learning,pp\. 23193–23211\.Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p2.1)\.
- Mohamed and Rezende \(2015\)S\. Mohamed and D\. J\. RezendeVariational information maximisation for intrinsically motivated reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.28,pp\. 2125–2133\.Cited by:[§1](https://arxiv.org/html/2609.10776#S1.p2.1)\.
- Tiomkin and Tishby \(2018\)S\. Tiomkin and N\. TishbyA unified bellman equation for causal information and value in markov decision processes\.arXiv preprint arXiv:1703\.01585\.External Links:1703\.01585Cited by:[§2\.2](https://arxiv.org/html/2609.10776#S2.SS2.p1.1)\.

## Appendix AProofs and Theoretical Derivations

### A\.1Contraction Mapping Property of the Bellman Operator

Let𝒱\\mathcal\{V\}be the Banach space of bounded, real\-valued functions mapping from history state\-action space toℝ\\mathbb\{R\}\. The supremum norm∥⋅∥∞\\\|\\cdot\\\|\_\{\\infty\}is defined for anyV∈𝒱V\\in\\mathcal\{V\},‖V‖∞=sups,a\|V⁡\(s,a\)\|\\\|V\\\|\_\{\\infty\}=\\sup\_\{s,a\}\|V\(s,a\)\|\. Letγ∈\(0,1\)\\gamma\\in\(0,1\)be the discount factor\.

We define the Bellman operatorT:𝒱→𝒱T:\\mathcal\{V\}\\to\\mathcal\{V\}associated with the value function as:

\(T​V\)​\(st−1,at−1\)=maxπ∈Π⁡\(st−1,at−1\)⁡𝔼st∼p​\[𝔼at∼π​\[log⁡π⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γ​V​\(st,at\)\]\]\(TV\)\(s\_\{t\-1\},a\_\{t\-1\}\)=\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\(8\)
###### Theorem A\.1\.

The operatorTTis aγ\\gamma\-contraction mapping under the supremum norm, i\.e\., for anyV1,V2∈𝒱V\_\{1\},V\_\{2\}\\in\\mathcal\{V\}:

‖T​V1−T​V2‖∞≤γ​‖V1−V2‖∞\\\|TV\_\{1\}\-TV\_\{2\}\\\|\_\{\\infty\}\\leq\\gamma\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\(9\)

###### Proof\.

LetV1,V2∈𝒱V\_\{1\},V\_\{2\}\\in\\mathcal\{V\}be two arbitrary value functions\. For any state\-action context\(st−1,at−1\)\(s\_\{t\-1\},a\_\{t\-1\}\), we evaluate the absolute difference\|\(T​V1\)​\(st−1,at−1\)−\(T​V2\)​\(st−1,at−1\)\|\|\(TV\_\{1\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\-\(TV\_\{2\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\|\.

Using the algebraic property of the maximum,\|maxx⁡f⁡\(x\)−maxx⁡g⁡\(x\)\|≤maxx⁡\|f⁡\(x\)−g⁡\(x\)\|\|\\max\_\{x\}f\(x\)\-\\max\_\{x\}g\(x\)\|\\leq\\max\_\{x\}\|f\(x\)\-g\(x\)\|, we can upper\-bound the difference as follows:

\|\(TV1\)\(st−1,at−1\)−\(TV2\)\(st−1,at−1\)\|=\|maxπ∈Π⁡\(st−1,at−1\)⁡𝔼st∼p​\[𝔼at∼π​\[log⁡π⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γ​V1​\(st,at\)\]\]−maxπ∈Π⁡\(st−1,at−1\)𝔼st∼p\[𝔼at∼π\[logπ⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γV2\(st,at\)\]\]\|≤maxπ∈Π⁡\(st−1,at−1\)\|𝔼st∼p​\[𝔼at∼π​\[log⁡π⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γ​V1​\(st,at\)\]\]−𝔼st∼p\[𝔼at∼π\[logπ⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γV2\(st,at\)\]\]\|\\begin\{split\}\\big\|\(TV\_\{1\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)&\-\(TV\_\{2\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\\big\|\\\\ &=\\Bigg\|\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V\_\{1\}\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\\\\ &\\quad\-\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V\_\{2\}\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\\Bigg\|\\\\ &\\leq\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\Bigg\|\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V\_\{1\}\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\\\\ &\\quad\-\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V\_\{2\}\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\\Bigg\|\\end\{split\}\(10\)
The identical log terms cancel out within the absolute value:

\|\(T​V1\)​\(st−1,at−1\)−\(T​V2\)​\(st−1,at−1\)\|\\displaystyle\|\(TV\_\{1\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\-\(TV\_\{2\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\|≤maxπ∈Π⁡\(st−1,at−1\)⁡\|𝔼st∼p​\[𝔼at∼π​\[γ⁡\(V1​\(st,at\)−V2​\(st,at\)\)\]\]\|\\displaystyle\\leq\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\Bigg\|\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\gamma\\big\(V\_\{1\}\(s\_\{t\},a\_\{t\}\)\-V\_\{2\}\(s\_\{t\},a\_\{t\}\)\\big\)\\right\]\\right\]\\Bigg\|\(11\)
By definition of the supremum norm,\|V1​\(st,at\)−V2​\(st,at\)\|≤‖V1−V2‖∞\|V\_\{1\}\(s\_\{t\},a\_\{t\}\)\-V\_\{2\}\(s\_\{t\},a\_\{t\}\)\|\\leq\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}holds for all\(st,at\)\(s\_\{t\},a\_\{t\}\)\.

\|\(T​V1\)​\(st−1,at−1\)−\(T​V2\)​\(st−1,at−1\)\|\\displaystyle\|\(TV\_\{1\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\-\(TV\_\{2\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\|≤γ​maxπ∈Π⁡\(st−1,at−1\)​𝔼st∼p​\[𝔼at∼π​\[‖V1−V2‖∞\]\]\\displaystyle\\leq\\gamma\\max\_\{\\pi\\in\\Pi\(s\_\{t\-1\},a\_\{t\-1\}\)\}\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\\right\]\\right\]
Because valid probability distributions integrate to 1, we can pull the norm out of the expectations:

\|\(T​V1\)​\(st−1,at−1\)−\(T​V2\)​\(st−1,at−1\)\|\\displaystyle\|\(TV\_\{1\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\-\(TV\_\{2\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\|≤γ​‖V1−V2‖∞\\displaystyle\\leq\\gamma\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\(13\)
Since this upper bound is uniform and independent of the choice of\(st−1,at−1\)\(s\_\{t\-1\},a\_\{t\-1\}\), taking the supremum over all state\-action contexts on the left\-hand side completes the proof:

‖T​V1−T​V2‖∞=supst−1,at−1\|\(T​V1\)​\(st−1,at−1\)−\(T​V2\)​\(st−1,at−1\)\|≤γ​‖V1−V2‖∞\\\|TV\_\{1\}\-TV\_\{2\}\\\|\_\{\\infty\}=\\sup\_\{s\_\{t\-1\},a\_\{t\-1\}\}\|\(TV\_\{1\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\-\(TV\_\{2\}\)\(s\_\{t\-1\},a\_\{t\-1\}\)\|\\leq\\gamma\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\(14\)
Givenγ∈\(0,1\)\\gamma\\in\(0,1\), the operatorTTis aγ\\gamma\-contraction mapping\. By the Banach Fixed\-Point Theorem,TTpossesses a unique fixed pointV∗V^\{\*\}\. ∎

### A\.2Limiting the Bellman Operator to Policy Endpoints

We now show that the optimization domain of the Bellman operatorTTcan be restricted entirely to deterministic policies without altering the operator itself\.

###### Theorem A\.2\.

For any value functionV∈𝒱V\\in\\mathcal\{V\}, the objective function inside the Bellman operator optimization problem:

f⁡\(π\)=𝔼st∼p​\[𝔼at∼π​\[log⁡π⁡\(at∣st−1,at−1,st\)q⁡\(at∣st−1,at−1\)\+γ​V​\(st,at\)\]\]f\(\\pi\)=\\underset\{s\_\{t\}\\sim p\}\{\\mathbb\{E\}\}\\left\[\\underset\{a\_\{t\}\\sim\\pi\}\{\\mathbb\{E\}\}\\left\[\\log\\frac\{\\pi\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)\}\{q\(a\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\}\+\\gamma V\(s\_\{t\},a\_\{t\}\)\\right\]\\right\]\(15\)
is convex with respect toπ\\pi\. Consequently, the maximum off⁡\(π\)f\(\\pi\)over the convex compact setΠ\\Piis attained at an extreme point of the policy space, corresponding to a deterministic action selection\.

###### Proof\.

LetV∈𝒱V\\in\\mathcal\{V\}be an arbitrary value function\. To analyze the convexity off⁡\(π\)f\(\\pi\)with respect toπ\\pi, we can expand the inner expectation for a fixed next\-statests\_\{t\}\. For structural clarity, letpi=π⁡\(ai∣st−1,at−1,st\)p\_\{i\}=\\pi\(a\_\{i\}\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\),qi=q⁡\(ai∣st−1,at−1\)q\_\{i\}=q\(a\_\{i\}\\mid s\_\{t\-1\},a\_\{t\-1\}\), andVi=γ​V​\(st,ai\)V\_\{i\}=\\gamma V\(s\_\{t\},a\_\{i\}\)whereiiindexes the finite action space𝒜\\mathcal\{A\}\.

We can rewrite the core objective function as a sum of three distinct components mapping from the probability simplexΔ⁡\(𝒜\)→ℝ\\Delta\(\\mathcal\{A\}\)\\to\\mathbb\{R\}:

ϕ⁡\(𝐩\)=∑i=1\|𝒜\|pi​log⁡pi−∑i=1\|𝒜\|pi​log⁡qi\+∑i=1\|𝒜\|pi​Vi\\phi\(\\mathbf\{p\}\)=\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}\\log p\_\{i\}\-\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}\\log q\_\{i\}\+\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}V\_\{i\}\(16\)
We will establish the convexity ofϕ⁡\(𝐩\)\\phi\(\\mathbf\{p\}\)by analyzing each term independently:

##### 1\. Convexity of the negative entropy term:

Letg⁡\(pi\)=pi​log⁡pig\(p\_\{i\}\)=p\_\{i\}\\log p\_\{i\}defined on the domainpi∈\(0,1\]p\_\{i\}\\in\(0,1\]\. We prove convexity by showing that the second derivative ofpi​log⁡pip\_\{i\}\\log p\_\{i\}is non\-negative forpi\>0p\_\{i\}\>0\. Evaluating the first and second derivatives with respect topip\_\{i\}yields:

g′​\(pi\)=log⁡pi\+1,andg′′​\(pi\)=1pig^\{\\prime\}\(p\_\{i\}\)=\\log p\_\{i\}\+1,\\quad\\text\{and\}\\quad g^\{\\prime\\prime\}\(p\_\{i\}\)=\\frac\{1\}\{p\_\{i\}\}\(17\)Sincepi\>0p\_\{i\}\>0for any valid probability vector inside the simplex, the second derivative is strictly positive \(g′′​\(pi\)\>0g^\{\\prime\\prime\}\(p\_\{i\}\)\>0\)\. This guarantees thatg⁡\(pi\)g\(p\_\{i\}\)is strictly convex\. Because the sum of convex functions preserves convexity, the aggregate term∑i=1\|𝒜\|pi​log⁡pi\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}\\log p\_\{i\}is strictly convex\.

##### 2\. Convexity of the cross\-entropy baseline term:

Leth⁡\(pi\)=−pi​log⁡qih\(p\_\{i\}\)=\-p\_\{i\}\\log q\_\{i\}\. Because the baseline distributionqqis fixed and independent of the optimization policyπ\\pi, the value−log⁡qi\-\\log q\_\{i\}treats itself as a constant coefficient\. Taking the derivatives with respect topip\_\{i\}results in:

h′​\(pi\)=−log⁡qi,andh′′​\(pi\)=0h^\{\\prime\}\(p\_\{i\}\)=\-\\log q\_\{i\},\\quad\\text\{and\}\\quad h^\{\\prime\\prime\}\(p\_\{i\}\)=0\(18\)Since the second derivative is zero everywhere,h⁡\(pi\)h\(p\_\{i\}\)is an affine \(linear\) function\. All affine functions are convex, implying the sum−∑i=1\|𝒜\|pilogqi\-\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}\\log q\_\{i\}is a convex function of𝐩\\mathbf\{p\}\.

##### 3\. Convexity of the value function expectation term:

Similarly, the value function parametersViV\_\{i\}are independent of𝐩\\mathbf\{p\}\. The sum∑i=1\|𝒜\|pi​Vi\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}V\_\{i\}forms a linear combination of the coordinates of𝐩\\mathbf\{p\}, which makes it an affine function, and therefore it is inherently convex\.

In conclusion, becauseϕ⁡\(𝐩\)\\phi\(\\mathbf\{p\}\)is a sum consisting of a strictly convex function and two affine functions, the entire objective functionϕ⁡\(𝐩\)\\phi\(\\mathbf\{p\}\)is strictly convex across the action distribution vector space\. Since the outer expectation𝔼st∼p\\mathbb\{E\}\_\{s\_\{t\}\\sim p\}acts as a linear operator it preserves convexity\. Therefore, the global objective functionf⁡\(π\)f\(\\pi\)is convex with respect toπ\\pi\.

By Bauer’s Maximum Principle, a convex function optimized over a compact, convex set \(the probability simplexΔ⁡\(𝒜\)\\Delta\(\\mathcal\{A\}\)\) must attain its maximum at one of its extreme points\. The extreme points of a probability simplex are the vertices where the entire probability mass is allocated to a single choice:

π∗​\(a∣st−1,at−1,st\)=δ⁡\(a−a∗\)\\pi^\{\*\}\(a\\mid s\_\{t\-1\},a\_\{t\-1\},s\_\{t\}\)=\\delta\(a\-a^\{\*\}\)\(19\)whereδ\\deltais the Kronecker delta anda∗∈𝒜a^\{\*\}\\in\\mathcal\{A\}represents a deterministic action choice\. Consequently, limiting the policy search space exclusively to these deterministic endpoints yields an identical value under the operator\. ∎

## Appendix BExperimental Details

Table 1:Hyperparameter configurations for the Control\-Gated MDP, Road Environment and Two\-Room Environment\.
## Appendix CCode and Reproducibility