IncentRL：偏好引导与任务性能的权衡

# IncentRL: The Trade-Off Between Preference Guidance and Task Performance
Source: [https://arxiv.org/html/2609.21525](https://arxiv.org/html/2609.21525)
Yanlan KangAffiliation:Fudan UniversityShenqin YinAffiliation:Fudan University

###### Abstract

Preference\-based reward shaping can guide reinforcement learning, but adding preference signals to the reward may unintentionally change the task being optimized\. We address this problem with IncentRL, a framework that introduces preference guidance while explicitly characterizing its effect on external\-task performance\. IncentRL adds a Kullback–Leibler \(KL\) penalty between a specified outcome distribution and a preferred distribution\. For finite discounted Markov decision processes with bounded shaping costs, we derive an external\-value perturbation bound, establish a sufficient strict\-action\-gap condition for preserving the original optimal policy, and characterize the large\-weight regime through discounted cumulative preference cost\. Exact examples clarify the limits of these guarantees, including tied optima and support mismatch\. We study a practical implementation using a hand\-designed, distance\-based outcome proxy, a fixed preference distribution, and score\-weighted coefficient search\. On MiniGrid DoorKey\-8x8, the reported three\-seed mean success rate after two million training steps reaches 98% with coefficient 0\.01, compared with 90\.5% for the reported zero\-coefficient baseline, while the search progressively shifts toward smaller coefficients\. Together, these results provide a principled view of the central trade\-off in preference\-based RL: using additional guidance to improve learning without excessively distorting the original task objective\. The current experiments remain descriptive and do not yet isolate KL shaping from simpler alternatives\.

Keywords:reinforcement learning; reward shaping; outcome preferences; KL divergence; policy preservation; adaptive parameter search\.

## 1Introduction

External reward and desired outcomes need not be expressed at the same level\. A task may provide reward only at completion while a designer can specify which intermediate outcomes are preferable\. Curiosity and information\-gain methods offer one response to limited reward feedback by introducing additional learning signals\[[7](https://arxiv.org/html/2609.21525#bib.bib7),[5](https://arxiv.org/html/2609.21525#bib.bib5)\]\. A different response is to encode task preferences directly and use them to modify the reward\. Such a modification can guide behavior, but it can also change what the agent is being asked to optimize\. A concrete question is when local preferences support long\-horizon goal\-directed behavior and when they obstruct it\. For example, moving away from a goal may be necessary to reach a door or take a feasible route\. Here, preferences are designer\-specified mathematical quantities; no human preference measurements or cognitive model are assumed\.

IncentRL uses the shaped reward

rβ\(s,a\)=r\(s,a\)−βDKL\(p\(⋅∣s,a\)∥q\(⋅∣s\)\),β≥0,r\_\{\\beta\}\(s,a\)=r\(s,a\)\-\\beta D\_\{\\mathrm\{KL\}\}\\bigl\(p\(\\cdot\\mid s,a\)\\\|q\(\\cdot\\mid s\)\\bigr\),\\qquad\\beta\\geq 0,\(1\)wherepprepresents outcomes associated with an action andqqspecifies preferred outcomes\. This expression raises several questions before any learning algorithm is chosen\. Is the KL finite? Does the penalty preserve externally optimal behavior? What is optimized whenβ\\betabecomes large? Does an implemented quantity actually represent predicted outcomes, or does it encode a hand\-designed state score?

We address these questions for a restricted, explicit setting\. The theoretical analysis fixes the outcome and preference distributions and studies their induced bounded cost in a finite discounted Markov decision process \(MDP\)\. We give a value\-loss bound, a sufficient policy\-preservation condition, and a large\-weight limit statement\. The proofs use standard bounded\-reward perturbation and optimality arguments; they are included to establish the scope of this formulation, not as new general results in MDP theory\. Exact examples demonstrate why weaker assumptions are insufficient\.

We also examine an available MiniGrid implementation\[[3](https://arxiv.org/html/2609.21525#bib.bib3)\]\. It constructs a two\-outcome vector from the agent’s distance to a target after a transition\. Its adaptation routine fits a distribution over coefficient candidates using scores from separate training runs\. These mechanisms can be described precisely without identifying the distance vector with a calibrated prediction model or the fitted proposal with a Bayesian posterior\.

This paper combines a well\-defined preference\-penalty objective, a self\-contained analysis with counterexamples, and a characterization of the distance proxy and coefficient\-search rule\. Section[5](https://arxiv.org/html/2609.21525#S5)evaluates fixed\-coefficient shaping on MiniGrid and examines the behavior of coefficient search\. A separate prospective study asks whether the observed benefit persists against information\-matched shaping controls\. The existing comparison does not establish a KL\-specific mechanism, biological validity, or tuning\-free adaptation\.

## 2Related Work

#### Reward shaping\.

Potential\-based shaping addsγ​Φ​\(s′\)−Φ⁡\(s\)\\gamma\\Phi\(s^\{\\prime\}\)\-\\Phi\(s\)to reward and preserves optimal policies under the corresponding assumptions\[[6](https://arxiv.org/html/2609.21525#bib.bib6)\]\. The penalty in Equation \([1](https://arxiv.org/html/2609.21525#S1.E1)\) is generally not of this form\. IncentRL instead permits a change in the optimal policy when preferences conflict with external reward\. Its small\-weight analysis quantifies a perturbation of the objective; it does not supply the general invariance property of potential\-based shaping\. Whether preserving the original policy is desirable depends on whether preferences are intended as guidance or as an additional task requirement\.

#### Intrinsic motivation\.

Prediction\-error curiosity\[[7](https://arxiv.org/html/2609.21525#bib.bib7)\]and information\-gain exploration\[[5](https://arxiv.org/html/2609.21525#bib.bib5)\]construct learning signals with different meanings from a preferred\-outcome penalty\. In particular, a KL between predicted and preferred outcomes is not automatically information gain about unknown dynamics\. A preference distribution supplied by a designer encodes additional task information\. The comparisons needed to evaluate its effect should control for access to that information\.

#### Policy regularization and maximum\-entropy RL\.

TRPO constrains policy change using a KL\-based trust region\[[9](https://arxiv.org/html/2609.21525#bib.bib9)\]; PPO includes a clipped surrogate formulation and a KL\-penalty alternative\[[10](https://arxiv.org/html/2609.21525#bib.bib10)\]\. The clipped formulation does not impose a hard KL constraint\. Soft actor\-critic augments its objective with action\-policy entropy\[[4](https://arxiv.org/html/2609.21525#bib.bib4)\]\. IncentRL’s KL is over outcomes, not successive policies\. Its outcome\-entropy term, derived below, is likewise distinct from action entropy\. A shared divergence notation does not make these objectives equivalent\.

#### Preferences and probabilistic control\.

Active inference uses prior preferences over outcomes in a generative model and policy evaluation\[[2](https://arxiv.org/html/2609.21525#bib.bib2)\]\. It need not modify those preferences during each decision\. Our distinction is the addition of a local outcome penalty to an external\-reward MDP; we do not derive a full expected\-free\-energy objective or its epistemic terms\. The analogy to cognitive motivation provides context rather than a biological prediction\.

#### Adaptive search distributions\.

Updating a sampling distribution from candidate performance is an established optimization approach, exemplified by the cross\-entropy method\[[8](https://arxiv.org/html/2609.21525#bib.bib8)\]\. The available IncentRL coefficient search belongs broadly to this family of adaptive sampling procedures\. Its particular score\-weighted moment update and prescribed concentration schedule are not identified here with the standard cross\-entropy update, nor are they presented as a new general search algorithm\. Sampling from a Beta distribution does not by itself constitute Bayesian inference\.

## 3Method

### 3\.1Setting and support conditions

Let𝒮\\mathcal\{S\}and𝒜⁡\(s\)\\mathcal\{A\}\(s\)be finite, with transition kernelPP, bounded expected external rewardrr, discount0≤γ<10\\leq\\gamma<1, and initial distributionρ\\rho\. Let𝒪\\mathcal\{O\}be a finite outcome space\. Fix normalized distributionsp\(⋅∣s,a\)p\(\\cdot\\mid s,a\)andq\(⋅∣s\)q\(\\cdot\\mid s\)\. Ifppis called predictive, its meaning must be specified, for example as the distribution of an outcome observation produced by the next transition\. The theoretical results below concern the induced cost and do not establish that an estimatedppis accurate\.

Define, using natural logarithms,

c⁡\(s,a\)=∑o∈𝒪p⁡\(o∣s,a\)​log⁡p⁡\(o∣s,a\)q⁡\(o∣s\)\.c\(s,a\)=\\sum\_\{o\\in\\mathcal\{O\}\}p\(o\\mid s,a\)\\log\\frac\{p\(o\\mid s,a\)\}\{q\(o\\mid s\)\}\.\(2\)We use0​log⁡\(0/q\)=00\\log\(0/q\)=0\. A positivep⁡\(o∣s,a\)p\(o\\mid s,a\)paired with zeroq⁡\(o∣s\)q\(o\\mid s\)makes the cost infinite\. A convenient sufficient condition for boundedness is

q⁡\(o∣s\)≥qmin\>0for all​s,o,0≤c⁡\(s,a\)≤log⁡\(1/qmin\)=:Cmax\.q\(o\\mid s\)\\geq q\_\{\\min\}\>0\\quad\\text\{for all \}s,o,\\qquad 0\\leq c\(s,a\)\\leq\\log\(1/q\_\{\\min\}\)=:C\_\{\\max\}\.\(3\)The upper bound follows because∑opo​log⁡po≤0\\sum\_\{o\}p\_\{o\}\\log p\_\{o\}\\leq 0and−∑opologqo≤log\(1/qmin\)\-\\sum\_\{o\}p\_\{o\}\\log q\_\{o\}\\leq\\log\(1/q\_\{\\min\}\)\. Support inclusion alone is sufficient for a finite KL on each finite outcome space; Equation \([3](https://arxiv.org/html/2609.21525#S3.E3)\) supplies a transparent uniform bound\.

For a preferenceq0q\_\{0\}with zeros, one explicit modification is

qε​\(o∣s\)=\(1−ε\)​q0​\(o∣s\)\+ε​u​\(o\),0<ε<1,q\_\{\\varepsilon\}\(o\\mid s\)=\(1\-\\varepsilon\)q\_\{0\}\(o\\mid s\)\+\\varepsilon u\(o\),\\qquad 0<\\varepsilon<1,\(4\)whereuuhas full support\. The smoothing level is an additional design parameter and changes the objective\. It must be reported\. In continuous outcome spaces, absolute continuity alone does not ensure finite or uniformly bounded KL; the finite\-space result cannot be transferred without further assumptions\.

### 3\.2Returns and interpretation

For a policyπ\\pi, write

J0,ρ​\(π\)\\displaystyle J\_\{0,\\rho\}\(\\pi\)=𝔼ρ,π​\[∑t=0∞γt​r​\(St,At\)\],\\displaystyle=\\mathbb\{E\}\_\{\\rho,\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(S\_\{t\},A\_\{t\}\)\\right\],\(5\)Cρ​\(π\)\\displaystyle C\_\{\\rho\}\(\\pi\)=𝔼ρ,π​\[∑t=0∞γt​c​\(St,At\)\],\\displaystyle=\\mathbb\{E\}\_\{\\rho,\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}c\(S\_\{t\},A\_\{t\}\)\\right\],\(6\)Jβ,ρ​\(π\)\\displaystyle J\_\{\\beta,\\rho\}\(\\pi\)=J0,ρ​\(π\)−β​Cρ​\(π\)\.\\displaystyle=J\_\{0,\\rho\}\(\\pi\)\-\\beta C\_\{\\rho\}\(\\pi\)\.\(7\)For finitecc,β=0\\beta=0recovers the external objective\. A numerical implementation of that baseline should bypass penalty computation so that an invalid penalty cannot generate0×∞0\\times\\inftyor a NaN\.

The identity

−DKL\(p∥q\)=H\(p\)\+𝔼p\[logq\(O\)\]\-D\_\{\\mathrm\{KL\}\}\(p\\\|q\)=H\(p\)\+\\mathbb\{E\}\_\{p\}\[\\log q\(O\)\]\(8\)separates outcome entropy from preference log\-probability\. At a fixed expected log preference, the entropy term favors higher, not lower, outcome entropy\. Maximizing this expression is not generally an objective for making outcomes predictable\. For uniformqqovermmoutcomes,c=log⁡m−H⁡\(p\)c=\\log m\-H\(p\); this special case makes the distinction explicit\. Neither outcome entropy nor proximity to a desired outcome establishes information gain about an unknown environment\.

The value ofβ\\betadepends on the reward scale\. Multiplyingrrandβ\\betaby the same positive constant multiplies the entire objective by that constant and leaves its maximizing policies unchanged\. Consequently,β=1\\beta=1has no universal interpretation as a balanced regime\. The preference distribution and its smoothing also determine the effective scale of the penalty\.

### 3\.3Use with a learning algorithm

For fixedβ\\beta, a generic implementation can computec⁡\(s,a\)c\(s,a\), formr−β​c​\(s,a\)r\-\\beta c\(s,a\), and pass that reward to its RL update\. This specifies a new reward function, not a new policy optimizer\. For instance, a tabular Q\-learning target would be

rt−β​c​\(st,at\)\+γ​maxa′⁡Q⁡\(st\+1,a′\),r\_\{t\}\-\\beta c\(s\_\{t\},a\_\{t\}\)\+\\gamma\\max\_\{a^\{\\prime\}\}Q\(s\_\{t\+1\},a^\{\\prime\}\),with zero bootstrap at a true terminal state\. Time\-limit truncation must be distinguished from task termination according to the intended MDP\. This expression alone gives no convergence guarantee for a function approximator\.

Ifpporqqis learned during training, the reward can change over time\. Updating a predictor to fit observed transitions has a different purpose from changing it merely to make its predictions resemble preferences\. The latter can reduce measured KL without changing actual outcomes\. The analysis below fixes these distributions and therefore does not cover this nonstationarity\. A partially observed implementation also needs a state, belief, or history representation appropriate to its problem; a raw observation is not automatically Markov\.

### 3\.4Distance\-based outcome proxy

The following construction instantiates the preference penalty using goal distance and selects its coefficient through an outer\-loop search\. The formulas describe the public implementation\[[3](https://arxiv.org/html/2609.21525#bib.bib3)\]; the correspondence between that code version and the experimental runs has not been independently established\. Version\-specific behavior and reproducibility details are collected in Appendix[C](https://arxiv.org/html/2609.21525#A3)\.

For a transition to states′s^\{\\prime\}, letddbe the Euclidean distance from the successor agent position to the target\. A distance\-based outcome proxy is

u⁡\(d\)=e−d1\+e−d,p~d=\(u⁡\(d\),1−u⁡\(d\)\),q=\(0\.999,0\.001\)\.u\(d\)=\\frac\{e^\{\-d\}\}\{1\+e^\{\-d\}\},\\qquad\\widetilde\{p\}\_\{d\}=\(u\(d\),1\-u\(d\)\),\\qquad q=\(0\.999,0\.001\)\.\(9\)The public implementation evaluates the following numerical approximation to the KL penalty:

fη​\(d\)=∑i=12p~d,i​log⁡p~d,i\+ηqi\+η,η=10−8\.f\_\{\\eta\}\(d\)=\\sum\_\{i=1\}^\{2\}\\widetilde\{p\}\_\{d,i\}\\log\\frac\{\\widetilde\{p\}\_\{d,i\}\+\\eta\}\{q\_\{i\}\+\\eta\},\\qquad\\eta=10^\{\-8\}\.\(10\)Equation \([10](https://arxiv.org/html/2609.21525#S3.E10)\) records the actual numerical adjustment; it is not the exact KL in Equation \([2](https://arxiv.org/html/2609.21525#S3.E2)\)\. For interpretation, letf0f\_\{0\}be the exact KL with the same two normalized distributions\.

The proxy is hand\-designed, not trained from transition observations\. Ford≥0d\\geq 0, its first component is at most1/21/2, including at the goal\. Consequently, it should not be described as a calibrated probability of goal attainment\. Its exact penalty satisfies

f0​\(0\)≈2\.76123,limd→∞f0​\(d\)=log⁡1000≈6\.90776\.f\_\{0\}\(0\)\\approx 2\.76123,\\qquad\\lim\_\{d\\to\\infty\}f\_\{0\}\(d\)=\\log 1000\\approx 6\.90776\.\(11\)Forq=\(q1,q2\)q=\(q\_\{1\},q\_\{2\}\), the derivative of the binary KL with respect touuis

dd​uDKL\(\(u,1−u\)∥q\)=logu​q2\(1−u\)​q1\.\\frac\{d\}\{du\}D\_\{\\mathrm\{KL\}\}\(\(u,1\-u\)\\\|q\)=\\log\\frac\{uq\_\{2\}\}\{\(1\-u\)q\_\{1\}\}\.It is negative for0<u≤1/20<u\\leq 1/2whenq1=0\.999q\_\{1\}=0\.999, whileu′​\(d\)<0u^\{\\prime\}\(d\)<0\. Thusf0f\_\{0\}increases with distance\. The transformation is a nonlinear distance penalty with a nonzero constant component, not an independently learned distributional signal\. No superiority over a simpler distance penalty follows from writing it as a KL\.

Both agent coordinates and target information should be available to information\-matched baselines\.

Since the penalty is computed after the action, its natural form isr⁡\(s,a,s′\)−β​f​\(s′\)r\(s,a,s^\{\\prime\}\)\-\\beta f\(s^\{\\prime\}\)\. If this is a fixed bounded Markov reward, its expected penaltyc\(s,a\)=𝔼\[f\(S′\)∣s,a\]c\(s,a\)=\\mathbb\{E\}\[f\(S^\{\\prime\}\)\\mid s,a\]falls within Section[4](https://arxiv.org/html/2609.21525#S4)\. This mapping does not turnp~d\\widetilde\{p\}\_\{d\}into an action\-conditioned prediction model\. In episodic tasks the nonzero component is also consequential: paying a constant penalty until termination can favor earlier termination\. By contrast, the same constant paid forever in a continuing discounted task adds a policy\-independent value shift\.

### 3\.5Coefficient selection by adaptive distribution search

The search procedure samples coefficient candidates from a Beta proposal, trains a fresh PPO agent for each candidate and training seed, and uses final evaluation success to update the next round’s proposal\. The coefficient is fixed within each such training run\. The procedure is therefore an outer\-loop search, not a within\-agent online coefficient update\.

Letiiindex candidate–seed pairs with positive scoreyiy\_\{i\}in a round\. The procedure computes

wi=yi∑jyj,m=∑iwi​βi,v=max⁡\{∑iwi​\(βi−m\)2,10−6\}\.w\_\{i\}=\\frac\{y\_\{i\}\}\{\\sum\_\{j\}y\_\{j\}\},\\quad m=\\sum\_\{i\}w\_\{i\}\\beta\_\{i\},\\quad v=\\max\\left\\\{\\sum\_\{i\}w\_\{i\}\(\\beta\_\{i\}\-m\)^\{2\},10^\{\-6\}\\right\\\}\.\(12\)It then moment\-matches Beta shape parameters

t=m⁡\(1−m\)v−1,atarget=m​t,btarget=\(1−m\)​t,t=\\frac\{m\(1\-m\)\}\{v\}\-1,\\qquad a\_\{\\rm target\}=mt,\\quad b\_\{\\rm target\}=\(1\-m\)t,and rescales their sum to

κr=min\{2\(1\.618\)r\+3,100\},r=0,1,…\.\\kappa\_\{r\}=\\min\\\{2\(1\.618\)^\{r\+3\},100\\\},\\qquad r=0,1,\\ldots\.\(13\)Whenever the moment match has positive shape parameters, the final distribution simplifies to

Beta⁡\(κr​m,κr​\(1−m\)\)\.\\operatorname\{Beta\}\\bigl\(\\kappa\_\{r\}m,\\kappa\_\{r\}\(1\-m\)\\bigr\)\.\(14\)The empirical variance in Equation \([12](https://arxiv.org/html/2609.21525#S3.E12)\) no longer controls the final concentration\. Valid moment matching requires0<v<m⁡\(1−m\)0<v<m\(1\-m\)and0<m<10<m<1; the variance floor alone does not guarantee these conditions near a boundary\. A robust implementation must check them before sampling again\.

If there are no positive scores, the script leaves the first shape parameter unchanged, increments the second by ten, and rescales to the prescribed concentration\. These are specified search heuristics\. No likelihood for the update is defined, and a shrinking proposal is not a calibrated uncertainty statement\. Even in a valid Bayesian model, posterior concentration would not by itself demonstrate better task performance or the removal of tuning choices\.

## 4Theoretical Analysis

The results in this section hold for any fixed cost0≤c≤Cmax0\\leq c\\leq C\_\{\\max\}in the finite discounted MDP above\. Their applicability to a particular KL follows from establishing that bound\. These are statements about optimal objectives, not rates of learning\.

###### Proposition 1\(External\-value perturbation\)\.

Letπ0\\pi\_\{0\}maximizeJ0,ρJ\_\{0,\\rho\}andπβ\\pi\_\{\\beta\}maximizeJβ,ρJ\_\{\\beta,\\rho\}\. Then

0≤J0,ρ​\(π0\)−J0,ρ​\(πβ\)≤β⁡\(Cρ​\(π0\)−Cρ​\(πβ\)\)≤β​Cmax1−γ\.0\\leq J\_\{0,\\rho\}\(\\pi\_\{0\}\)\-J\_\{0,\\rho\}\(\\pi\_\{\\beta\}\)\\leq\\beta\\bigl\(C\_\{\\rho\}\(\\pi\_\{0\}\)\-C\_\{\\rho\}\(\\pi\_\{\\beta\}\)\\bigr\)\\leq\\frac\{\\beta C\_\{\\max\}\}\{1\-\\gamma\}\.\(15\)

###### Proof\.

Shaped optimality givesJ0,ρ​\(πβ\)−β​Cρ​\(πβ\)≥J0,ρ​\(π0\)−β​Cρ​\(π0\)J\_\{0,\\rho\}\(\\pi\_\{\\beta\}\)\-\\beta C\_\{\\rho\}\(\\pi\_\{\\beta\}\)\\geq J\_\{0,\\rho\}\(\\pi\_\{0\}\)\-\\beta C\_\{\\rho\}\(\\pi\_\{0\}\)\. Rearranging proves the middle inequality\. External optimality proves the lower bound, and0≤Cρ​\(π\)≤Cmax/\(1−γ\)0\\leq C\_\{\\rho\}\(\\pi\)\\leq C\_\{\\max\}/\(1\-\\gamma\)proves the last inequality\. ∎

This result does not require a unique external optimum\. If a learned policy is onlyϵ\\epsilon\-optimal for the shaped objective, the same argument yields the weaker external\-value boundβ​Cmax/\(1−γ\)\+ϵ\\beta C\_\{\\max\}/\(1\-\\gamma\)\+\\epsilon\. It does not supply a value ofϵ\\epsilonfor an actual training run\.

###### Proposition 2\(Policy preservation under a strict action gap\)\.

Assume every state has a unique externally optimal actiona∗​\(s\)a^\{\*\}\(s\)\. Over the states with alternative actions, define

Δ=mins:\|𝒜⁡\(s\)\|\>1mina≠a∗​\(s\)\[Q0∗\(s,a∗\(s\)\)−Q0∗\(s,a\)\]\>0\.\\Delta=\\min\_\{s:\\,\|\\mathcal\{A\}\(s\)\|\>1\}\\min\_\{a\\neq a^\{\*\}\(s\)\}\\bigl\[Q\_\{0\}^\{\*\}\(s,a^\{\*\}\(s\)\)\-Q\_\{0\}^\{\*\}\(s,a\)\\bigr\]\>0\.\(16\)IfCmax\>0C\_\{\\max\}\>0andβ​Cmax/\(1−γ\)<Δ\\beta C\_\{\\max\}/\(1\-\\gamma\)<\\Delta, thens↦a∗​\(s\)s\\mapsto a^\{\*\}\(s\)remains optimal from every state for the shaped reward\. IfCmax=0C\_\{\\max\}=0, or there is no state with alternative actions, the conclusion holds without this threshold\.

###### Proof\.

LetB=β​Cmax/\(1−γ\)B=\\beta C\_\{\\max\}/\(1\-\\gamma\)\. For each initial state\-action pair and every continuation policy, subtracting the discounted penalty reduces return by an amount in\[0,B\]\[0,B\]\. Taking suprema over continuation policies gives

Q0∗​\(s,a\)−B≤Qβ∗​\(s,a\)≤Q0∗​\(s,a\)\.Q\_\{0\}^\{\*\}\(s,a\)\-B\\leq Q\_\{\\beta\}^\{\*\}\(s,a\)\\leq Q\_\{0\}^\{\*\}\(s,a\)\.For every alternative action,

Qβ∗​\(s,a∗​\(s\)\)−Qβ∗​\(s,a\)\\displaystyle Q\_\{\\beta\}^\{\*\}\(s,a^\{\*\}\(s\)\)\-Q\_\{\\beta\}^\{\*\}\(s,a\)≥Q0∗​\(s,a∗​\(s\)\)−Q0∗​\(s,a\)−B\\displaystyle\\geq Q\_\{0\}^\{\*\}\(s,a^\{\*\}\(s\)\)\-Q\_\{0\}^\{\*\}\(s,a\)\-B≥Δ−B\>0\.\\displaystyle\\geq\\Delta\-B\>0\.Thus the maximizing action at each state is unchanged and defines an optimal stationary policy\. ∎

The strict gap matters\. Multiple external optima can have different preference costs and be separated by any positive penalty weight\. Moreover, the collection of stochastic policies need not have a positive gap between optimal and suboptimal returns, even in a finite MDP\. Proposition[2](https://arxiv.org/html/2609.21525#Thmproposition2)is a sufficient condition, not a practical certificate unless the gap and cost bound are known\.

###### Proposition 3\(Large\-weight limit\)\.

Suppose\|r⁡\(s,a\)\|≤Rmax\|r\(s,a\)\|\\leq R\_\{\\max\}, and letCmin=minπ⁡Cρ​\(π\)C\_\{\\min\}=\\min\_\{\\pi\}C\_\{\\rho\}\(\\pi\)\. Forβ\>0\\beta\>0, every maximizerπβ\\pi\_\{\\beta\}ofJβ,ρJ\_\{\\beta,\\rho\}satisfies

0≤Cρ​\(πβ\)−Cmin≤2​Rmaxβ⁡\(1−γ\)\.0\\leq C\_\{\\rho\}\(\\pi\_\{\\beta\}\)\-C\_\{\\min\}\\leq\\frac\{2R\_\{\\max\}\}\{\\beta\(1\-\\gamma\)\}\.\(17\)Restricting the sequence to stationary randomized policies, every accumulation point asβ→∞\\beta\\to\\inftyminimizesCρC\_\{\\rho\}and maximizesJ0,ρJ\_\{0,\\rho\}within that minimum\-cost set\. A unique policy limit is not asserted\.

###### Proof\.

Choose a cost minimizerπC\\pi\_\{C\}\. Optimality implies

β⁡\(Cρ​\(πβ\)−Cmin\)≤J0,ρ​\(πβ\)−J0,ρ​\(πC\)≤2​Rmax1−γ\.\\beta\\bigl\(C\_\{\\rho\}\(\\pi\_\{\\beta\}\)\-C\_\{\\min\}\\bigr\)\\leq J\_\{0,\\rho\}\(\\pi\_\{\\beta\}\)\-J\_\{0,\\rho\}\(\\pi\_\{C\}\)\\leq\\frac\{2R\_\{\\max\}\}\{1\-\\gamma\}\.This proves the bound\. The space of stationary randomized policies is a finite product of probability simplices and is therefore compact\. Its discounted reward and cost values are continuous\. For example, these values have the form\(I−γ​Pπ\)−1​rπ\(I\-\\gamma P\_\{\\pi\}\)^\{\-1\}r\_\{\\pi\}and\(I−γ​Pπ\)−1​cπ\(I\-\\gamma P\_\{\\pi\}\)^\{\-1\}c\_\{\\pi\}, and the inverse exists for every policy\. Any convergent subsequence therefore has a limit with costCminC\_\{\\min\}\. For any minimum\-cost policyπ\\pi,

J0,ρ​\(πβ\)≥J0,ρ​\(π\)\+β⁡\(Cρ​\(πβ\)−Cmin\)≥J0,ρ​\(π\)\.J\_\{0,\\rho\}\(\\pi\_\{\\beta\}\)\\geq J\_\{0,\\rho\}\(\\pi\)\+\\beta\\bigl\(C\_\{\\rho\}\(\\pi\_\{\\beta\}\)\-C\_\{\\min\}\\bigr\)\\geq J\_\{0,\\rho\}\(\\pi\)\.Passing to the same subsequence proves external\-return maximization within the minimum\-cost set\. ∎

The quantity minimized in this limit is a cumulative control cost\. Its optimum depends on dynamics, discounting, the outcome model, preferences, and the initial distribution\. A minimum\-cost statement for oneρ\\rhoimposes no constraint on behavior at states that cannot affect its value\. It is not a prescription to greedily minimize immediate KL at every state\.

### 4\.1An illustrative reward–preference trade\-off

The following exact example illustrates the policy change permitted by the bounds above\. It is a deterministic calculation, not a learning experiment\.

###### Example 1\(An explicit reward–preference trade\-off\)\.

Consider a one\-step decision, equivalently a problem withγ=0\\gamma=0\. The aligned action has reward zero andpa=q=\(3/4,1/4\)p\_\{a\}=q=\(3/4,1/4\), hence zero cost\. The reward\-seeking action has reward one andpb=\(0,1\)p\_\{b\}=\(0,1\), hence costlog⁡4\\log 4\. Their shaped values are zero and1−β​log⁡41\-\\beta\\log 4, so the maximizing action switches at

βc=1log⁡4≈0\.72135\.\\beta\_\{c\}=\\frac\{1\}\{\\log 4\}\\approx 0\.72135\.Below the threshold the external return is one; above it the external return is zero\. At the threshold both actions, and their mixtures, are optimal\. HereΔ=1\\Delta=1andCmax=log⁡4C\_\{\\max\}=\\log 4, so the sufficient threshold in Proposition[2](https://arxiv.org/html/2609.21525#Thmproposition2)matches the switch\. The example shows why strong preferences can deliberately sacrifice external reward\.

Additional examples in Appendix[B](https://arxiv.org/html/2609.21525#A2)examine support failure, ties among externally optimal policies, and the difference between immediate and cumulative cost\. Table[3](https://arxiv.org/html/2609.21525#A2.T3)in that appendix gives numerical values for Example[1](https://arxiv.org/html/2609.21525#Thmexample1)\.

## 5Experiments

Our experiments examine two aspects of IncentRL: the effect of a fixed shaping coefficient on MiniGrid learning, and the evolution of sampled coefficients across search rounds\. The fixed\-coefficient comparison measures task success and episode length, while the search analysis characterizes the coefficient distribution\. Together they connect the preference\-penalty formulation to observed learning behavior\. The six\-group mechanism study in Section[6\.1](https://arxiv.org/html/2609.21525#S6.SS1)is a separate, planned extension and is not part of the results presented here\.

### 5\.1Fixed\-coefficient learning curves

We use MiniGrid DoorKey\-8x8, requiring the agent to obtain a key, unlock a door, and reach the goal\. We compare coefficientsβ=0\\beta=0andβ=0\.01\\beta=0\.01, with the same architecture and training hyperparameters across conditions, two million training steps per agent, and three independent training seeds\. Success rate and mean episode length are tracked during training\. The correspondence between the experiment\-generating code and the public version examined in Appendix[C](https://arxiv.org/html/2609.21525#A3)is not established by the available records\. The implementation issues in Appendix[C](https://arxiv.org/html/2609.21525#A3)therefore cannot be assumed to have affected this comparison\.

Table[1](https://arxiv.org/html/2609.21525#S5.T1)summarizes the experimental settings documented for these figures and the remaining reporting gaps\. The public implementation provides candidate configuration details, but without a run\-to\-code link those details cannot establish the settings used to generate these results\. In particular, the prospective protocol in Section[6\.1](https://arxiv.org/html/2609.21525#S6.SS1)is not used to fill these gaps\.

Table 1:Documented settings and reporting gaps for the MiniGrid experiments\. Unavailable details are not inferred from library defaults or from the prospective protocol\.![Refer to caption](https://arxiv.org/html/2609.21525v1/mini-grid.png)Figure 1:Learning curves on MiniGrid DoorKey\-8x8 forβ=0\\beta=0andβ=0\.01\\beta=0\.01, averaged over three seeds with two million training steps per agent\. The shared vertical axis combines success rate in percent and episode length in steps\. No uncertainty bands are shown; the plot supports a descriptive comparison of the mean trajectories\.At the final reported checkpoint, the mean success rate is 98% forβ=0\.01\\beta=0\.01and 90\.5% for the reportedβ=0\\beta=0baseline, an absolute difference of 7\.5 percentage points\. The reported mean episode lengths are 29 and 75 steps, respectively\. These endpoints and the trajectories in Figure[1](https://arxiv.org/html/2609.21525#S5.F1)support a descriptive advantage in this particular comparison\. They do not establish statistical significance, asymptotic superiority, or a general improvement across tasks\. The available aggregate statistics do not specify whether episode length is averaged over all episodes or only successful ones, so the length difference is not interpreted as a controlled measure of path efficiency\. Run\-level data would be needed to quantify uncertainty and analyze learning\-curve area under a defined evaluation protocol\.

Both conditions improve late in training, with substantial success gains after roughly 1\.25 million steps\. No causal attribution to exploration or policy\-update stabilization follows from these curves alone\. Unshaped reward is the only displayed control: the comparison does not separate distance information, the nonlinear transformation, and a constant per\-step cost\. It also does not establish that coefficient search outperforms fixed\-coefficient or random search at equal total cost\.

### 5\.2Coefficient\-search observations

Table[2](https://arxiv.org/html/2609.21525#S5.T2)summarizes sampled coefficients across search rounds, and Figure[2](https://arxiv.org/html/2609.21525#S5.F2)shows their mean and standard deviation\. The mean decreases from 0\.1057 in round one to 0\.0173 in round four\. Dispersion decreases through round three, then increases from 0\.0098 to 0\.0182 in round four\. The spread of sampled coefficients is not a confidence interval for an optimal coefficient or calibrated posterior uncertainty\.

Table 2:Summary statistics of sampled coefficients across four search rounds\. Values are shown to four decimal places\. The available aggregates do not specify the exact pooling or averaging rule across seeds\.![Refer to caption](https://arxiv.org/html/2609.21525v1/bayesian_line.png)Figure 2:Mean and standard deviation of sampled coefficients across search rounds, using the values in Table[2](https://arxiv.org/html/2609.21525#S5.T2)\. Bars denote the reported standard deviation of sampled coefficients, not uncertainty in the mean or a Bayesian credible interval\. A symmetric mean\-plus/minus\-SD bar can extend below zero even when all samples are nonnegative\. No raw samples are reconstructed\.These summaries document coefficient\-search behavior\. Supplementary distributions for individual seeds are provided in Appendix[E](https://arxiv.org/html/2609.21525#A5)\(Figure[3](https://arxiv.org/html/2609.21525#A5.F3)\)\. Interpreting them as an outer\-loop adaptive search is consistent with the available procedure in Section[3\.5](https://arxiv.org/html/2609.21525#S3.SS5); a Bayesian interpretation would require a specified likelihood and posterior update\. A numerical resemblance between later sampled values and a useful fixed coefficient is not a held\-out performance comparison or a demonstration that tuning has been eliminated\.

## 6Discussion and Limitations

The evidence combines mathematical analysis, exact examples, implementation characterization, and MiniGrid experiments\. The empirical comparison is descriptive: the available records do not establish exact experiment\-generating versions, per\-seed evaluation results, or all aggregation details\. These gaps limit reproducibility and uncertainty quantification\. Appendix[A](https://arxiv.org/html/2609.21525#A1)summarizes the scope of the claims\.

The theory assumes a fixed bounded penalty, exact optimization when referring toπβ\\pi\_\{\\beta\}, and a finite discounted MDP\. It does not establish convergence for neural function approximation, learned or changing preferences, or adaptiveβt\\beta\_\{t\}\. The worst\-case bound can be loose, especially whenγ\\gammais close to one or preference probabilities are very small\. The strict action gap may also be unknown or absent\. These limitations constrain both theoretical interpretation and practical parameter selection\.

Preference quality is an independent issue\. A task\-informed penalty can be useful guidance or a source of bias\. Euclidean distance can conflict with necessary detours, unlocking a door, or building momentum\. A preference penalty does not guarantee exploration, safety, or transfer to a new task\. Learned outcome models introduce additional approximation and calibration questions that the distance proxy does not answer\.

### 6\.1A prospective test of local guidance and task completion

The next empirical question is whether a nonlinear distance penalty improves learning beyond direct distance guidance when observations, coefficient\-selection opportunities, and training budgets are matched\. The associated mechanism question is whether any difference depends on a constant cost paid until termination\. These questions follow from the proxy analysis; they do not assume that the KL construction is beneficial\. A locally frozen design, dated 18 September 2026, specifies the following study\. Its implementation and training results are pending, and it has not been externally preregistered\.

Letf0f\_\{0\}denote the exact proxy KL from Section[3\.4](https://arxiv.org/html/2609.21525#S3.SS4),DDa fixed upper distance bound,K=f0​\(D\)K=f\_\{0\}\(D\), andc0=f0​\(0\)/Kc\_\{0\}=f\_\{0\}\(0\)/K\. The planned penalties are normalized for an explicit coefficient grid, and differ from the unnormalized coefficients in Section[5](https://arxiv.org/html/2609.21525#S5)\. Withd′d^\{\\prime\}measured at the actual successor state, the six groups are:

HereΦ\(s\)=−d/D\\Phi\(s\)=\-d/D, with zero successor potential at every task terminal, including the finite horizon\. Thus the discounted sum of the weighted shaping terms telescopes to−λ​Φ​\(s0\)\-\\lambda\\Phi\(s\_\{0\}\)\[[6](https://arxiv.org/html/2609.21525#bib.bib6)\]\. The other penalties apply on the final real transition but not after termination\. Full and centered KL differ by exactly−λ​c0\-\\lambda c\_\{0\}per transition; this algebraic identity does not imply an additive decomposition of learning performance\.

A small finite\-horizon MDP will contrast a direct route with a condition requiring an initial detour, using exact dynamic programming to compare optimal action sets and task outcomes\. This planned model is distinct from the completed exact examples in this note\. The learning experiment will use fully observed symbolic states in DoorKey\-8x8, a 640\-step task horizon, and the same PPO architecture and goal information in all groups\. This planned study introduces six groups, symbolic observations, normalized coefficients, and independent confirmation seeds\. These settings differ from the two\-condition experiment in Section[5](https://arxiv.org/html/2609.21525#S5)and should not be used to describe its results\. Real goal coordinates and terminal observations must be checked before training\. Potential\-based invariance concerns the underlying objective, not equality of finite\-budget PPO learning curves\.

The primary comparison is full KL versus direct distance, each selected using the same gridλ∈\{0,0\.003,0\.01,0\.03,0\.1\}\\lambda\\in\\\{0,0\.003,0\.01,0\.03,0\.1\\\}and three tuning seeds\. The centered group inherits the full\-KL coefficient and is diagnostic rather than an independently tuned competitor\. Ten fresh training seeds will provide paired differences in normalized external\-success learning\-curve area at 17 fixed checkpoints, each evaluated on the same 100 held\-out reset seeds\. Each full run has 2,097,152 training transitions\. The analysis uses a paired training\-seed bootstrap with 20,000 resamples and a 95% interval; the prespecified practical margin is 0\.05 AUC\. Inference is conditional on the chosen coefficients and fixed evaluation bank\. Distinct reset seeds do not guarantee distinct layouts\. This comparison estimates the difference between tuned learning procedures, not the pure effect of KL at a matched penalty amplitude\. All selection costs and technical failures must be recorded\. The protocol fixes the detailed selection, failure, and interpretation rules before confirmation results are examined\.

### 6\.2Extensions beyond the planned mechanism study

The frozen study does not test adaptive coefficient search, learned outcome predictions, or generalization across task families\. Evaluating the adaptive search would require grid and random\-search controls at matched total environment interactions, with final evaluation separate from selection scores\. A learnedppwould require an explicit outcome horizon, training data and loss, and calibration assessment\. Any additional benchmark must specify environment versions, reward transformations, termination handling, and run\-level provenance\. For example, standard MountainCar assigns−1\-1per step\[[1](https://arxiv.org/html/2609.21525#bib.bib1)\]; a sparse\+1/0\+1/0version must be labeled as a reward\-transformed task\.

A full Bayesian extension would need to define the uncertain quantity, prior, observation model, and likelihood, and derive an update consistent with them\. It would also need to distinguish posterior inference from the decision rule that chooses a coefficient\. A Beta\-shaped sampling distribution is insufficient\. Such an extension should be evaluated against the explicit heuristic characterized here, with matching tuning budgets and held\-out performance measures\.

## 7Conclusion

IncentRL expresses a preference\-guided modification of an external\-reward objective through an outcome KL penalty\. Making its support and boundedness assumptions explicit yields a value\-loss bound, a sufficient strict\-gap condition for policy preservation, and a cumulative\-cost interpretation of the large\-weight limit\. Exact examples show why unconditional preservation and immediate\-KL minimization are not valid substitutes for these statements\. The available implementation uses a distance\-derived proxy and an outer\-loop adaptive search distribution\. The prospective study isolates local distance guidance, episode\-length costs, and conflicts with task completion\. It supplies a route from this analysis to a mechanism\-focused empirical evaluation without presupposing a performance gain\. The MiniGrid experiment shows higher endpoint success for the nonzero coefficient, while the search summaries show movement toward smaller coefficients\. Neither establishes a KL\-specific advantage or Bayesian adaptation; those stronger claims remain separate work\.

## References

- \[1\]Farama Foundation\.Mountain car: Gymnasium documentation, n\.d\.URL[https://gymnasium\.farama\.org/environments/classic\_control/mountain\_car/](https://gymnasium.farama.org/environments/classic_control/mountain_car/)\.Accessed 17 September 2026\.
- \[2\]Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pezzulo\.Active inference: A process theory\.*Neural Computation*, 29\(1\):1–49, 2017\.doi:10\.1162/NECO\_a\_00912\.URL[https://doi\.org/10\.1162/NECO\_a\_00912](https://doi.org/10.1162/NECO_a_00912)\.
- \[3\]gravitywavelet\.IncentRL public implementation\.GitHub repository, 2025\.URL[https://github\.com/gravitywavelet/incentive\-RL\-anon](https://github.com/gravitywavelet/incentive-RL-anon)\.Commit f512c32b835fee07832e3044280a00a2381ad8ba; accessed 17 September 2026\.
- \[4\]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.In*Proceedings of the 35th International Conference on Machine Learning*, volume 80 of*Proceedings of Machine Learning Research*, pages 1861–1870, 2018\.URL[https://proceedings\.mlr\.press/v80/haarnoja18b\.html](https://proceedings.mlr.press/v80/haarnoja18b.html)\.
- \[5\]Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel\.VIME: Variational information maximizing exploration\.arXiv:1605\.09674, 2016\.URL[https://arxiv\.org/abs/1605\.09674](https://arxiv.org/abs/1605.09674)\.
- \[6\]Andrew Y\. Ng, Daishi Harada, and Stuart Russell\.Policy invariance under reward transformations: Theory and application to reward shaping\.In*Proceedings of the Sixteenth International Conference on Machine Learning*, pages 278–287, 1999\.URL[https://people\.eecs\.berkeley\.edu/~russell/papers/icml99\-shaping\.pdf](https://people.eecs.berkeley.edu/~russell/papers/icml99-shaping.pdf)\.
- \[7\]Deepak Pathak, Pulkit Agrawal, Alexei A\. Efros, and Trevor Darrell\.Curiosity\-driven exploration by self\-supervised prediction\.In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*, pages 2778–2787, 2017\.URL[https://proceedings\.mlr\.press/v70/pathak17a\.html](https://proceedings.mlr.press/v70/pathak17a.html)\.
- \[8\]Reuven Rubinstein\.The cross\-entropy method for combinatorial and continuous optimization\.*Methodology and Computing in Applied Probability*, 1\(2\):127–190, 1999\.doi:10\.1023/A:1010091220143\.URL[https://doi\.org/10\.1023/A:1010091220143](https://doi.org/10.1023/A:1010091220143)\.
- \[9\]John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz\.Trust region policy optimization\.In*Proceedings of the 32nd International Conference on Machine Learning*, volume 37 of*Proceedings of Machine Learning Research*, pages 1889–1897, 2015\.URL[https://proceedings\.mlr\.press/v37/schulman15\.html](https://proceedings.mlr.press/v37/schulman15.html)\.
- \[10\]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.arXiv:1707\.06347, 2017\.URL[https://arxiv\.org/abs/1707\.06347](https://arxiv.org/abs/1707.06347)\.

## Appendix AScope and interpretation

The paper studies a preference\-penalized objective, its mathematical properties, a distance\-based implementation, and MiniGrid learning and coefficient\-search behavior\. The empirical results support a descriptive comparison in the evaluated setting\. They do not establish an online Bayesian update, elimination of coefficient tuning, or a general advantage across tasks\. The six\-group study in Section[6\.1](https://arxiv.org/html/2609.21525#S6.SS1)is a future evaluation designed to distinguish candidate mechanisms\.

The theoretical guarantees depend on their stated assumptions\. When external optima are tied, an arbitrary externally optimal policy need not remain optimal at every sufficiently small positive weight\. Proposition[2](https://arxiv.org/html/2609.21525#Thmproposition2)gives a sufficient strict\-gap condition, while Proposition[1](https://arxiv.org/html/2609.21525#Thmproposition1)bounds external\-value loss without that condition\. At large weight, the relevant objective is discounted cumulative cost\. Support conditions such as those illustrated in Example[2](https://arxiv.org/html/2609.21525#Thmexample2)are necessary before finite\-return arguments can be applied\.

The public implementation inspected here is fixed by the full commit identifier

f512c32b835fee07832e3044280a00a2381ad8ba\.

Static analysis of this version does not establish the exact code used for each experimental run\. Reproduction requires run identifiers, generating code versions, per\-seed records, and aggregation code\. The supplied figures and coefficient summaries do not replace those records\.

## Appendix BAdditional examples and numerical values

These fully specified examples illustrate the assumptions and limits of Section[4](https://arxiv.org/html/2609.21525#S4)\. They are analytic calculations, not RL training experiments\.

###### Example 2\(A preference support failure\)\.

With predicted outcomesp=\(0\.3,0\.7\)p=\(0\.3,0\.7\)and preferencesq=\(1,0\)q=\(1,0\), the second outcome has positive predicted mass and zero preferred mass\. HenceDKL\(p∥q\)=∞D\_\{\\mathrm\{KL\}\}\(p\\\|q\)=\\infty\. Replacingqqwith\(0\.999,0\.001\)\(0\.999,0\.001\)gives a finite value of approximately4\.224864\.22486\. This replacement defines a different objective; it cannot justify finite numerical results from an implementation whose support handling is unspecified\.

###### Example 3\(Ties need not survive shaping\)\.

A one\-step decision has two actions with external reward one\. Their outcome distributions arepa=\(1,0\)p\_\{a\}=\(1,0\)andpb=\(0,1\)p\_\{b\}=\(0,1\), andq=\(3/4,1/4\)q=\(3/4,1/4\)\. The corresponding costs arelog⁡\(4/3\)\\log\(4/3\)andlog⁡4\\log 4\. Both actions are externally optimal, but actionbbis strictly suboptimal under everyβ\>0\\beta\>0\. This does not contradict Proposition[1](https://arxiv.org/html/2609.21525#Thmproposition1); the external return can remain unchanged even though the set of optimal policies shrinks\.

###### Example 4\(Immediate mismatch can be misleading\)\.

An initial actionaaincurs cost zero and leads to an absorbing state whose cost is one at every subsequent step\. Actionbbincurs cost one and leads to a zero\-cost absorbing state\. All external rewards are zero\. Atγ=0\.9\\gamma=0\.9, the discounted costs are nine foraaand one forbb\. Immediate\-cost minimization chooses the wrong action for cumulative preference alignment\. These costs can be realized with a fixed binary preferenceq=\(e−1,1−e−1\)q=\(e^\{\-1\},1\-e^\{\-1\}\), usingp=qp=qwherever the cost is zero andp=\(1,0\)p=\(1,0\)wherever it is one\. The outcome variable may be an auxiliary observation emitted on a transition; it need not identify the next state\.

### B\.1Values for the main\-text trade\-off example

Table[3](https://arxiv.org/html/2609.21525#A2.T3)gives values obtained directly from the two action values in Example[1](https://arxiv.org/html/2609.21525#Thmexample1)\.

Table 3:Exact values for Example[1](https://arxiv.org/html/2609.21525#Thmexample1)\. No agent is trained\. The external reward is sacrificed once the weight exceeds1/log⁡41/\\log 4\.

## Appendix CImplementation and reproducibility details

The public MiniGrid scriptKL\_minigrid\_mp\_bayes\.pyis inspected at commitf512c32\[[3](https://arxiv.org/html/2609.21525#bib.bib3)\]\. The observations below concern that version and do not establish which code generated individual experimental runs\.

### C\.1Position lookup and target coordinates

After stepping the environment, the wrapper attempts to read the agent position and computes distance to\(W−2,H−2\)\(W\-2,H\-2\)\. If the position is unavailable, the script uses\(0\.5,0\.5\)\(0\.5,0\.5\)instead, giving a constant penalty\. The relevant wrapper\-attribute behavior depends on installed library versions and requires runtime validation\. The target coordinate must also be verified against the generated environment\. Both agent coordinates and target information are additional task information and should be available to matched baselines\.

### C\.2Baseline handling and search accounting

Several details prevent interpreting the inspected script as a completed controlled comparison\. A zero coefficient appended as a baseline is clipped to10−510^\{\-5\}; this search baseline is therefore not exactly unshaped\. The result loader uses a wildcard across rounds and selects its first matching file, allowing an earlier result to be attributed to a later round\. These are identifiable code paths, not evidence that the experimental results were affected\. Exact run identifiers and an independent zero\-penalty baseline are needed before reproducing a comparison\.

The default configuration specifies five rounds, ten candidates per round, one additional final\-round baseline, three training seeds, and two million target training steps per run\. This amounts nominally to

\(5⋅10\+1\)⋅3⋅2,000,000=306,000,000\(5\\cdot 10\+1\)\\cdot 3\\cdot 2\{,\}000\{,\}000=306\{,\}000\{,\}000training steps, excluding evaluation and rollout\-size rounding\. This is configuration arithmetic, not a measured execution total\. Comparing the search procedure against a single two\-million\-step fixed\-coefficient run would omit the cost of selection\. Held\-out final evaluation must also be separated from the scores used to fit the proposal\.

## Appendix DDetails of the exact calculations

The exact\-example table uses coefficient values00,0\.250\.25,0\.500\.50,0\.750\.75, and11\. For each coefficient, the optimal action is determined by comparing the shaped values00and1−β​log⁡41\-\\beta\\log 4\. Selecting the aligned action gives external return and preference cost both equal to zero; selecting the reward\-seeking action gives external return11and preference costlog⁡4\\log 4\. These are deterministic calculations, not estimates from training runs; no training\-seed uncertainty or confidence interval is associated with them\. In the support\-failure example, the KL is infinite because an outcome with positive probability underpphas zero probability underqq\.

## Appendix ESupplementary coefficient\-search distributions

Figure[3](https://arxiv.org/html/2609.21525#A5.F3)shows the sampled\-coefficient distributions for seeds 42 and 43 across four search rounds, supplementing the aggregate summaries in Section[5](https://arxiv.org/html/2609.21525#S5)\. The two seeds displayed here differ in number from the three seeds used for the fixed\-coefficient learning curves\. These distributions illustrate variation between seeds and a shift toward smaller sampled coefficients; they do not establish posterior inference or a performance advantage from coefficient search\.

![Refer to caption](https://arxiv.org/html/2609.21525v1/bayesian_box.png)Figure 3:Sampled\-coefficient distributions for seeds 42 and 43 across four search rounds\. The red line marksβ=0\.1\\beta=0\.1as a visual reference, not an empirically established optimum\. The plot shows a shift toward smaller sampled values and variation between seeds; it does not establish Bayesian posterior concentration\. Sample counts and the precise boxplot convention are not available in the accompanying aggregate records\.