Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous

# Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous
Source: [https://arxiv.org/html/2608.14795](https://arxiv.org/html/2608.14795)
###### Abstract

An AI that can only give advice seems safe: the human is always free to ignore it\. That is the premise of the boxing tradition in AI safety, and its long\-suspected weak point is that the human who reads the answers is part of the system\. We make the fractionεt\\varepsilon\_\{t\}of behavior that follows the advice a state of a Markov decision process, moved by the advisor’s own messages, so that use deepens reliance\. Granted a channel rich enough to echo any action the human could take, higherεt\\varepsilon\_\{t\}weakly lowers every monotone measure of the power of a human with a message\-independent fallback\. An oracle rewarded by per\-round approval cultivates reliance beyond a closed\-form patience threshold, so the same reward weights leave the optimal oracle answering in episodic deployments and cultivating in long\-memory ones\. An influence bound certified once at deployment is blind to that horizon and bounds the loss no lower than its trivial ceiling\. An exogenous cap on influence bounds the guarantee the human loses, and a short enough memory reset removes the incentive to cultivate, while neither recovers the value already steered away\. In a closed\-form example the optimal oracle never cultivates in fifteen\-round sessions and does in sixteen\.

## 1Introduction

Suppose the AI can only talk, so the world changes only when a person acts on what it says\. Common sense says such a system is safe, because one is always free to ignore it\. But “free to ignore it” is a property of a single exchange, and safety has to hold over a relationship, across which how much of the advice a person follows drifts, and so does what that person wants\. Neither drift is felt as a loss\. What bounds the power lost to an advisor that only talks, and which of the popular safeguards bound it?

In the boxing tradition of AI safety, a system is made safe by restricting its interface rather than its internals: deny it actuators and let it only answer questions, on the reasoning that even a misaligned system cannot act against human interests if it cannot act at all\. The tradition runs from early oracle\-AI proposals\([Armstrong, Sandberg, and Bostrom 2012](https://arxiv.org/html/2608.14795#bib.bib3);[Bostrom 2014](https://arxiv.org/html/2608.14795#bib.bib8)\)through their safety\-use catalog\([Armstrong and O’Rorke 2017](https://arxiv.org/html/2608.14795#bib.bib2)\)\. That tradition has carried, from the beginning, an informal worry about its own premise: the person reading the answers acts on them, so a persuasive oracle has an actuator after all\. Reliance that grows with use is itself well studied elsewhere, as habit stock\([Becker and Murphy 1988](https://arxiv.org/html/2608.14795#bib.bib4)\), as trust updated by interaction with automation\([Lee and See 2004](https://arxiv.org/html/2608.14795#bib.bib25)\), and as user state moved by recommender systems\([Chaney, Stewart, and Engelhardt 2018](https://arxiv.org/html/2608.14795#bib.bib12)\), all from inside the feedback loop \(Section[6](https://arxiv.org/html/2608.14795#S6)\)\. To our knowledge, what has been missing is the boxing question posed inside such a model: whether a deployment constraint fixed outside the interaction loop still guarantees anything, once the loop is left to run, in the worst case over the advisor’s policy\.

This paper answers that question for one of the two things a relationship moves: how much of the advice the person follows\. What the person wants is the second, and it is future work\. We model the human–advisor interaction as a Markov decision process whose transition kernel mixes the human’s own dynamics with a message\-directed component at weightεt\\varepsilon\_\{t\}\(the influence coefficient: the fraction of behavior that currently routes through the advice\), andεt\\varepsilon\_\{t\}itself evolves, moved by the advisor’s own messages: following the advice today raises the weight it carries tomorrow\.

The paper’s contributions are the following, each a formal safety\-case statement about oracle\-style deployments\.

1. 1\.A prior\-free vocabulary of power in which influence transferred is control lost\(Section[3](https://arxiv.org/html/2608.14795#S3), Lemma[1](https://arxiv.org/html/2608.14795#Thmlemma1)\): the human’s power is the family of values they can still guarantee whatever is said, computed for a human whose own choices do not depend on what the advice says, the oracle’s its counterfactual deviation from the human’s default\.
2. 2\.The answer/cultivate switch\(Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)\): cultivating reliance is an investment, optimal beyond a closed\-form threshold in the discount factor, front\-loaded with an explicit stopping point when reliance does not decay\.
3. 3\.What bounds the loss, and what does not\(Theorem[2](https://arxiv.org/html/2608.14795#Thmtheorem2), Lemma[2](https://arxiv.org/html/2608.14795#Thmlemma2), Propositions[1](https://arxiv.org/html/2608.14795#Thmproposition1)and[2](https://arxiv.org/html/2608.14795#Thmproposition2)\): the disempowerment index splits into a displacement term \(the world already steered\) plus a channel term \(the guarantee lost\), less a credit for a benevolent oracle\.

Utility drift, the second coordinate the relationship moves, is outside this paper: defining and measuring it is future work\. Static boxing fails because it treats an endogenous quantity, one the interaction itself moves, as if it were exogenous, fixed from outside the relationship, and the worked example \(Section[5](https://arxiv.org/html/2608.14795#S5)\) shows the gap between the two is large enough to flip the optimal policy\.

## 2Model

The model tracks a task state the human steers and an influence coefficient that says how much of the steering currently routes through the advisor\.

For example, one starts by asking an assistant for driving directions; it is usually right, so the checking stops, and a year later the route is followed turn by turn and cannot be reconstructed alone: goals never changed, and the loss is the accumulated habit of not overruling, the growth ofεt\\varepsilon\_\{t\}\. The recursion is deliberately a caricature, carrying headroom\-limited growth and decay in the simplest form that supports proof\.

The interaction is modeled by a Markov decision process\([Puterman 1994](https://arxiv.org/html/2608.14795#bib.bib30)\): the task state isxtx\_\{t\}, the human’s default dynamics are the transition kernelT0\(⋅∣xt\)T\_\{0\}\(\\cdot\\mid x\_\{t\}\)over the next state when the human acts on their own, and the oracle emits a messagemtm\_\{t\}, so the world moves by the mixture

xt\+1∼\(1−εt\)T0\(⋅∣xt\)\+εtD\(⋅∣xt,mt\),x\_\{t\+1\}\\sim\(1\-\\varepsilon\_\{t\}\)\\,T\_\{0\}\(\\cdot\\mid x\_\{t\}\)\+\\varepsilon\_\{t\}\\,D\(\\cdot\\mid x\_\{t\},m\_\{t\}\),\(1\)while reliance follows

εt\+1=\(1−δ\)​εt\+η⁡\(mt\)​\(1−εt\)\.\\varepsilon\_\{t\+1\}=\(1\-\\delta\)\\,\\varepsilon\_\{t\}\+\\eta\(m\_\{t\}\)\\,\(1\-\\varepsilon\_\{t\}\)\.\(2\)In words: with probability1−εt1\-\\varepsilon\_\{t\}the human does what they would have done anyway, and with probabilityεt\\varepsilon\_\{t\}they act as the message directs \(DDis the message\-directed kernel\), while reliance grows with use and fades with disuse\. Hereη⁡\(mt\)≥0\\eta\(m\_\{t\}\)\\geq 0is how much a message cultivates reliance \(engaging, flattering, dependence\-building messages have highη\\eta\),\(1−εt\)\(1\-\\varepsilon\_\{t\}\)is the remaining headroom, andδ\\deltais decay back toward independence when cultivation stops\.

The mixture covers the human’s behavior each round: they act as the message directs, at rateεt\\varepsilon\_\{t\}, or they act as they would have anyway\. In the directions example the first is following the recommended turn and the second is driving the route one had in mind\. What the model leaves out is the response in between, using what the advice says to form a different plan of one’s own, for instance inferring from the recommended turn that there is traffic ahead and taking another route\. That is a second channel from message to behavior, and this paper carries compliance alone \(Section[4\.1](https://arxiv.org/html/2608.14795#S4.SS1)\)\.

Equation \([2](https://arxiv.org/html/2608.14795#S2.E2)\) is the central modeling choice:ε\\varepsilonis a*state variable driven by the oracle’s own outputs*, not a constant of the interface\. This creates a feedback loop: cultivating raisesε\\varepsilon, higherε\\varepsilonraises reachable power \(Lemma[1](https://arxiv.org/html/2608.14795#Thmlemma1), granted the echo condition of Assumption[1](https://arxiv.org/html/2608.14795#Thmassumption1)\), and so spending current influence buys future influence\. It is the oracle’s analogue of resource acquisition\.

## 3What Power Means in This Paper

Disempowerment is loss of power, so we define power before stating any theorem about losing it\. The setting needs an asymmetric notion: the human’s question is welfare\-shaped, what can this person still secure, while the oracle’s is threat\-shaped, how far can it move the world from the human’s default course, in which destructive capacity correctly counts as power\. A single symmetric notion would either credit the oracle with the human’s own resourcefulness or miss a message that changes nothing yet forecloses options\. The following definition sets four conditions that both notions are built to meet\.

###### Definition 1\(Admissible power measure\)\.

A power measure for this setting is*admissible*if it is:\(1\) prior\-free, not depending on a distribution over reward functions, since the human has one utilityu0u\_\{0\};\(2\) a measure of steering, not luck, tracking the ability to make outcomes different rather than how valuable the reachable outcomes already happen to be;\(3\) calibrated, exactly zero when the oracle’s messages are causally inert and recovering a direct agent’s power under full compliance; and\(4\) loss\-compatible, subtracting to a scalarDist\\mathrm\{Dis\}\_\{t\}for the gap between what the initial human could attain and what the actual trajectory attains\.

Condition \(1\) is deliberate, not a simplification: a prior over rewards is the known weak point of the power\-seeking theorems\([Thorstad 2024](https://arxiv.org/html/2608.14795#bib.bib37);[Tarsney 2025](https://arxiv.org/html/2608.14795#bib.bib36)\), and this paper avoids it by construction\. Conditions \(1\) and \(2\) hold of each of the two constructions below on its own; condition \(3\) is discharged by the oracle’s power \(Definition[3](https://arxiv.org/html/2608.14795#Thmdefinition3)\), and condition \(4\) by the human’s power, whose subtraction from the human’s own baseline forms the scalarDist\\mathrm\{Dis\}\_\{t\}\(Definition[2](https://arxiv.org/html/2608.14795#Thmdefinition2)\)\.

### 3\.1The Definition: a Prior\-Free Core and Its Scalarizations

To have power is to retain the ability to make outcomes different\. The notion is built in two layers: Layer 1 a partial order, since a person who is wealthy but housebound and a person who is poor but mobile can each hold an advantage the other lacks, and Layer 2 a scalarization that breaks such ties at the cost of an additional input such as a utility function\. The two definitions that follow, human power and oracle power, instantiate this core asymmetrically as motivated above\.

Fix a Markov decision process with finite state and action spaces and discount factorγ∈\(0,1\)\\gamma\\in\(0,1\)\. The*occupancy measure*of a policyπ\\pistarted atxx,

fxπ\(x′,a′\)=\(1−γ\)∑t≥0γtPr\(xt=x′,at=a′∣x0=x,π\),f^\{\\pi\}\_\{x\}\(x^\{\\prime\},a^\{\\prime\}\)=\(1\-\\gamma\)\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\Pr\(x\_\{t\}\{=\}x^\{\\prime\},\\,a\_\{t\}\{=\}a^\{\\prime\}\\mid x\_\{0\}\{=\}x,\\pi\),records where the process spends its discounted time and what it does there, withγ\\gammasetting an effective horizon of order1/\(1−γ\)1/\(1\-\\gamma\)\([Puterman 1994](https://arxiv.org/html/2608.14795#bib.bib30);[Altman 1999](https://arxiv.org/html/2608.14795#bib.bib1)\)\. Value under a utilityuu, a bounded function of the state\-action pair, is linear in it,

Vuπ​\(x\)=⟨fxπ,u⟩/\(1−γ\),V^\{\\pi\}\_\{u\}\(x\)=\\langle f^\{\\pi\}\_\{x\},u\\rangle/\(1\-\\gamma\),so every discounted objective is a linear functional of it, and the*feasible set*

ℱ⁡\(x\)=\{fxπ:π​a policy\}\\mathcal\{F\}\(x\)=\\\{f^\{\\pi\}\_\{x\}:\\pi\\text\{ a policy\}\\\}of futures the agent can induce is a compact convex polytope\. This gives a prior\-free*partial order*:xxis at least as powerful asx′x^\{\\prime\}whenℱ⁡\(x\)⊇ℱ⁡\(x′\)\\mathcal\{F\}\(x\)\\supseteq\\mathcal\{F\}\(x^\{\\prime\}\), equivalently whenVu∗​\(x\)≥Vu∗​\(x′\)V^\{\*\}\_\{u\}\(x\)\\geq V^\{\*\}\_\{u\}\(x^\{\\prime\}\)for*every*utilityuu, a Blackwell\-type dominance order\([Blackwell 1953](https://arxiv.org/html/2608.14795#bib.bib7)\)\. Call this*Layer 1*: no prior, but partial\.*Layer 2*supplies the weightings, a*scalarization*being any monotone map from feasible sets to numbers \(Turner’s POWER is the prior\-averaged support function; empowerment and the deviation radius below are others\)\. All scalarizations agree on pairs the Layer\-1 order ranks and disagree only on incomparable ones, so a result stated at Layer 1 holds for every monotone notion of power\.

Atε\>0\\varepsilon\>0the human does not fully own the transition, so “the futures the human can induce” is undefined until the oracle’s messages are specified; power is therefore defined as what the human can*guarantee*\.

###### Definition 2\(Human power\)\.

For each utilityuuand influence levelε\\varepsilon, the human’s guaranteed value is the lower value of the zero\-sum game in which the human picks a policy, mapping the history of states and realized actions to a distribution over actions and carrying no dependence on the messages, and the oracle picks the messages,

Wu​\(x,ε\)=maxπ⁡minσ​Vuπ,σ​\(x,ε\)\.W\_\{u\}\(x,\\varepsilon\)=\\max\_\{\\pi\}\\min\_\{\\sigma\}V^\{\\pi,\\sigma\}\_\{u\}\(x,\\varepsilon\)\.The human’s*power*at\(x,ε\)\(x,\\varepsilon\)is the family\{Wu​\(x,ε\)\}u\\\{W\_\{u\}\(x,\\varepsilon\)\\\}\_\{u\}, indexed by every utilityuuand ordered pointwise\.

The definition extends Layer 1: atε=0\\varepsilon=0the game degenerates and the pointwise order on\{Wu\}u\\\{W\_\{u\}\\\}\_\{u\}is the dominance order, and the game value exists and is achieved by stationary strategies\([Shapley 1953](https://arxiv.org/html/2608.14795#bib.bib32)\), the message set being finite \(Section[4\.1](https://arxiv.org/html/2608.14795#S4.SS1)\)\. Themin\\minover messages is a normative commitment: power is measured against the*arbitrary*will of the counterparty, exercised or not, the republican notion of freedom as non\-domination\([Pettit 1997](https://arxiv.org/html/2608.14795#bib.bib29)\)made quantitative\. Andε\\varepsilonis*frozen*inside the definition; its dynamics re\-enter through the switch and boxing theorems, with the loss index evaluating the static guarantee at the current anchor\(xt,εt\)\(x\_\{t\},\\varepsilon\_\{t\}\)\.

###### Definition 3\(Oracle power\)\.

The oracle’s power at\(x,ε\)\(x,\\varepsilon\)is its counterfactual deviation from the human’s default,

Dev⁡\(x,ε\)=supσTV⁡\(fxσ,ε,fxT0\)\\mathrm\{Dev\}\(x,\\varepsilon\)=\\sup\_\{\\sigma\}\\mathrm\{TV\}\(f^\{\\sigma,\\varepsilon\}\_\{x\},f^\{T\_\{0\}\}\_\{x\}\)wherefxσ,εf^\{\\sigma,\\varepsilon\}\_\{x\}andfxT0f^\{T\_\{0\}\}\_\{x\}are the occupancy measures of Section[3\.1](https://arxiv.org/html/2608.14795#S3.SS1), under the oracle policyσ\\sigmaand under the no\-oracle default: the reachable total\-variation displacement of the discounted state\-action occupancy relative to the no\-oracle process, the human following their default policy throughout \(the occupancy records the realized action, under influence possibly the message\-directed one, while the no\-oracle occupancy pairs each state with the default action\)\.

Both definitions realize the asymmetry motivated above, fitted to the model by the setting itself, and the influence coefficient sets the scale of one\-step oracle power, made exact in equation \([5](https://arxiv.org/html/2608.14795#S4.E5)\) of Section[4\.1](https://arxiv.org/html/2608.14795#S4.SS1)\.

#### Why not the alternatives\.

The comparison is in the supplement\. In outline: Turner’s POWER\([Turner et al\. 2021](https://arxiv.org/html/2608.14795#bib.bib39)\)fails \(1\) and \(2\), importing the prior its critics identify as doing the theorems’ work\([Thorstad 2024](https://arxiv.org/html/2608.14795#bib.bib37);[Thorstad 2026](https://arxiv.org/html/2608.14795#bib.bib38);[Tarsney 2025](https://arxiv.org/html/2608.14795#bib.bib36)\), empowerment fails \(4\), and the dominance order alone is partial\. Optimized human\-power metrics\([Heitzig and Potham 2025](https://arxiv.org/html/2608.14795#bib.bib18)\)are the closest construction \(Section[6](https://arxiv.org/html/2608.14795#S6)\)\.

### 3\.2The Loss Mechanisms

Definition[2](https://arxiv.org/html/2608.14795#Thmdefinition2)left the utility free; from here on the working scalarization isWu0W\_\{u\_\{0\}\}, the value of the*initial*utility the person can still secure by their own choices\. The disempowerment index is measured against the initial human\(T0,u0\)\(T\_\{0\},u\_\{0\}\):

Dist=Vu0alone​\(x0\)−𝔼⁡\[Vtbeh\],\\mathrm\{Dis\}\_\{t\}\\;=\\;V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{0\}\)\\;\-\\;\\mathbb\{E\}\\bigl\[\\,V^\{\\mathrm\{beh\}\}\_\{t\}\\,\\bigr\],\(3\)whereVu0alone​\(x0\)V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{0\}\)is what the initial human could attain with no oracle andVtbehV^\{\\mathrm\{beh\}\}\_\{t\}is theu0u\_\{0\}\-value of the behavioral continuation at timett, evaluated likeWWat the frozen anchor\(xt,εt\)\(x\_\{t\},\\varepsilon\_\{t\}\)\. The expectation runs over the trajectory of the actual interaction, the human best\-responding to the oracle’s message policy in the true dynamics whileεt\\varepsilon\_\{t\}evolves\. The index decomposes into a displacement term and a channel term, less a credit for a benevolent oracle \(Lemma[2](https://arxiv.org/html/2608.14795#Thmlemma2)\)\.

The mechanism acts on the guarantee: asε\\varepsilongrows the whole family\{Wu​\(x,ε\)\}u\\\{W\_\{u\}\(x,\\varepsilon\)\\\}\_\{u\}, the power of a human with a message\-independent fallback, declines \(Lemma[1](https://arxiv.org/html/2608.14795#Thmlemma1)\(b\), granted the echo condition of Assumption[1](https://arxiv.org/html/2608.14795#Thmassumption1)\), goals intact and hands tied, which is the channel term an exogenous cap bounds\. Utility drift, the relationship movingutu\_\{t\}itself so that selection is by a changed objective, hands free and aim moved, would act on the reference rather than the guarantee; it is outside this paper and is future work\.

The full stress test of the definition is in the supplement\. The objections that remain as stated limitations are that the baselineT0T\_\{0\}is compared against forever while unassisted competence degrades with disuse \(atrophy, Section[7](https://arxiv.org/html/2608.14795#S7)\), and that the scalarε\\varepsiloncompresses multi\-dimensional compliance\.

## 4Results

Full proofs are in the technical supplement, with a proof sketch here for the main results\.

### 4\.1Setup and Standing Assumptions

We make the human’s choices explicit\. The state space𝒳\\mathcal\{X\}, the human’s action set𝒜H\\mathcal\{A\}\_\{H\}, and the message setℳ\\mathcal\{M\}are finite\. The human has own dynamicsTH\(⋅∣x,a\)T\_\{H\}\(\\cdot\\mid x,a\)and deterministic default policyπ0\\pi\_\{0\}, so the no\-oracle kernel of equation \([1](https://arxiv.org/html/2608.14795#S2.E1)\) isT0\(⋅∣x\)=TH\(⋅∣x,π0\(x\)\)T\_\{0\}\(\\cdot\\mid x\)=T\_\{H\}\(\\cdot\\mid x,\\pi\_\{0\}\(x\)\)\. A message directs both an action and a transition: eachmmdeclares, at each statexx, the actionaD​\(x,m\)∈𝒜Ha\_\{D\}\(x,m\)\\in\\mathcal\{A\}\_\{H\}a complying human takes and the kernelD\(⋅∣x,m\)D\(\\cdot\\mid x,m\)their compliance realizes\. The kernel need not equalTH\(⋅∣x,aD\(x,m\)\)T\_\{H\}\(\\cdot\\mid x,a\_\{D\}\(x,m\)\), since advice can steer execution at a finer grain than the human’s own repertoire \(the directed action “drive” may come with a turn\-by\-turn route whose law over destinations no unassisted action produces\)\. Writingatowna^\{\\mathrm\{own\}\}\_\{t\}for the human’s intended action, drawn from their policy atxtx\_\{t\}, the realized action isat=atowna\_\{t\}=a^\{\\mathrm\{own\}\}\_\{t\}with probability1−εt1\-\\varepsilon\_\{t\}andat=aD​\(xt,mt\)a\_\{t\}=a\_\{D\}\(x\_\{t\},m\_\{t\}\)with probabilityεt\\varepsilon\_\{t\}, givingxt\+1∼\(1−εt\)TH\(⋅∣xt,atown\)\+εtD\(⋅∣xt,mt\)x\_\{t\+1\}\\sim\(1\-\\varepsilon\_\{t\}\)T\_\{H\}\(\\cdot\\mid x\_\{t\},a^\{\\mathrm\{own\}\}\_\{t\}\)\+\\varepsilon\_\{t\}D\(\\cdot\\mid x\_\{t\},m\_\{t\}\), of which equation \([1](https://arxiv.org/html/2608.14795#S2.E1)\) is the case of default play,atown=π0​\(xt\)a^\{\\mathrm\{own\}\}\_\{t\}=\\pi\_\{0\}\(x\_\{t\}\)\. A human policy maps the history of states and realized actions to a distribution over𝒜H\\mathcal\{A\}\_\{H\}and carries no dependence on the messages, soatowna^\{\\mathrm\{own\}\}\_\{t\}is the human’s fallback: what they do on the rounds they do not defer, formed without using what the message says\. The restriction is on that branch alone and not on compliance, which is where following the advice happens\. The second channel excluded in Section[2](https://arxiv.org/html/2608.14795#S2), a response formed from what the message says, would route influence through the own branch as well\. Utilities areu:𝒳×𝒜H→\[0,1\]u:\\mathcal\{X\}\\times\\mathcal\{A\}\_\{H\}\\to\[0,1\], the round\-ttpayoff beingu⁡\(xt,at\)u\(x\_\{t\},a\_\{t\}\)at the realized action, discounted atγ∈\(0,1\)\\gamma\\in\(0,1\)\. A utility of the state alone is the caseu⁡\(x,a\)=u⁡\(x\)u\(x,a\)=u\(x\), and the minimal example of Section[5](https://arxiv.org/html/2608.14795#S5), having a single task state, has a utility of the action alone\. The influence coefficient follows equation \([2](https://arxiv.org/html/2608.14795#S2.E2)\) withη⁡\(m\)∈\[0,η¯\]\\eta\(m\)\\in\[0,\\bar\{\\eta\}\],δ∈\[0,1\)\\delta\\in\[0,1\), andη¯\+δ≤1\\bar\{\\eta\}\+\\delta\\leq 1\. For frozenε\\varepsilon,Wu​\(x,ε\)W\_\{u\}\(x,\\varepsilon\)is the guaranteed value of Definition[2](https://arxiv.org/html/2608.14795#Thmdefinition2)andVualone​\(x\)=maxπ⁡Vuπ​\(x∣TH\)V^\{\\mathrm\{alone\}\}\_\{u\}\(x\)=\\max\_\{\\pi\}V^\{\\pi\}\_\{u\}\(x\\mid T\_\{H\}\)the human’s optimal value in their own oracle\-free MDP, withWu​\(x,0\)=Vualone​\(x\)W\_\{u\}\(x,0\)=V^\{\\mathrm\{alone\}\}\_\{u\}\(x\)\.

The occupancy of Definition[3](https://arxiv.org/html/2608.14795#Thmdefinition3)records the realized action, so the radius of the oracle’s influence is taken on the one\-step joint law of realized action and next state\. Write

Jm​\(x\)\\displaystyle J\_\{m\}\(x\)=δaD​\(x,m\)⊗D\(⋅∣x,m\),\\displaystyle=\\delta\_\{a\_\{D\}\(x,m\)\}\\otimes D\(\\cdot\\mid x,m\),\(4\)J0​\(x\)\\displaystyle J\_\{0\}\(x\)=δπ0​\(x\)⊗T0\(⋅∣x\),\\displaystyle=\\delta\_\{\\pi\_\{0\}\(x\)\}\\otimes T\_\{0\}\(\\cdot\\mid x\),Jm\(ε\)​\(x\)\\displaystyle J^\{\(\\varepsilon\)\}\_\{m\}\(x\)=\(1−ε\)​J0​\(x\)\+ε​Jm​\(x\),\\displaystyle=\(1\-\\varepsilon\)\\,J\_\{0\}\(x\)\+\\varepsilon\\,J\_\{m\}\(x\),for the joint laws of the pair \(action taken, state reached\) under a messagemmobeyed, under the human’s default policyπ0\\pi\_\{0\}, and under influenceε\\varepsilon: the realized\-action dynamics above, evaluated at the default\. Subtracting the default gives

‖Jm\(ε\)​\(x\)−J0​\(x\)‖TV=ε​‖Jm​\(x\)−J0​\(x\)‖TV≤ε,\\left\\lVert J^\{\(\\varepsilon\)\}\_\{m\}\(x\)\-J\_\{0\}\(x\)\\right\\rVert\_\{\\mathrm\{TV\}\}=\\varepsilon\\,\\left\\lVert J\_\{m\}\(x\)\-J\_\{0\}\(x\)\\right\\rVert\_\{\\mathrm\{TV\}\}\\leq\\varepsilon,\(5\)with equality when the directed law escapes the default’s support, which holds already when the directed action differs from the default one:ε\\varepsilonis simultaneously the mixture weight, the compliance rate, and the radius of the oracle’s one\-step reachable TV\-ball \(condition \(3\) of Definition[1](https://arxiv.org/html/2608.14795#Thmdefinition1)\), soε=0\\varepsilon=0is a boxed oracle andε=1\\varepsilon=1a direct agent acting through the human\. The radius has to be taken on the joint law and not on the next\-state kernel alone\. The minimal example of Section[5](https://arxiv.org/html/2608.14795#S5)has one task state, so every kernel there isδx\\delta\_\{x\}and the kernel\-level radius is zero at everyε\\varepsilon, in the example built to exhibit an oracle taking over the human’s action\.

The monotonicity lemma needs one hypothesis on the message\-directed kernelDD: why would more compliance always mean more oracle power?

###### Assumption 1\(Echo\)\.

For everyxxand everya∈𝒜Ha\\in\\mathcal\{A\}\_\{H\}there is a distributionμx,a\\mu\_\{x,a\}overℳ\\mathcal\{M\}withaD​\(x,m\)=aa\_\{D\}\(x,m\)=afor everymmin its support and∑mμx,a\(m\)D\(⋅∣x,m\)=TH\(⋅∣x,a\)\\sum\_\{m\}\\mu\_\{x,a\}\(m\)\\,D\(\\cdot\\mid x,m\)=T\_\{H\}\(\\cdot\\mid x,a\): some randomization over messages directs the actionaaand reproduces its kernel\.

Echo says advice can recommend anything the human could do \(“carry on as you were” is a possible message\), the hypothesis under which more influence never handicaps the oracle, and without which it can \(the supplement gives the example\); it is the realistic case for a language\-model oracle whose messages range over every text of bounded length\. The condition is one\-directional: every human action is reproducible by messages, in the action directed and the kernel realized, whileℳ\\mathcal\{M\}may also contain messages whose kernels no human action produces, which is what the displacement example of Proposition[1](https://arxiv.org/html/2608.14795#Thmproposition1)\(ii\) uses\. The cultivation intensityη⁡\(m\)\\eta\(m\)of equation \([2](https://arxiv.org/html/2608.14795#S2.E2)\) is likewise untied to the message’s directed component: a message can build reliance while directing the human’s own default action, so cultivation can ride on zero\-displacement messages, and no per\-message displacement check, at any fixed threshold, registers it\.

Total variation is normalized as the supremum over events, so‖P−Q‖TV∈\[0,1\]\\left\\lVert P\-Q\\right\\rVert\_\{\\mathrm\{TV\}\}\\in\[0,1\]and\|∫f​d​\(P−Q\)\|≤‖P−Q‖TV​\(supf−inff\)\\lvert\\int f\\,d\(P\-Q\)\\rvert\\leq\\left\\lVert P\-Q\\right\\rVert\_\{\\mathrm\{TV\}\}\\,\(\\sup f\-\\inf f\); every constant below is stated in that normalization\.

We also use a standard value\-gap bound \(supplement\): if two processes start at the same state and their joint one\-step laws of realized action and next state stay withinκ\\kappain total variation at every history, then for rewards in\[0,1\]\[0,1\]at the realized action the values differ by at mostκ/\(1−γ\)2\\kappa/\(1\-\\gamma\)^\{2\}\. The horizon factor is real: a per\-step influence radius ofε\\varepsilonis compatible with a large total loss over a long horizon \(Section[4\.4](https://arxiv.org/html/2608.14795#S4.SS4)\)\.

### 4\.2The Monotonicity Lemma

LetJmJ\_\{m\}be the directed joint law of \([4](https://arxiv.org/html/2608.14795#S4.E4)\) and defineℛε\(x,a\)=\{\(1−ε\)δa⊗TH\(⋅∣x,a\)\+εQ:Q∈conv\{Jm\(x\):m∈ℳ\}\}\\mathcal\{R\}\_\{\\varepsilon\}\(x,a\)=\\\{\(1\-\\varepsilon\)\\,\\delta\_\{a\}\\otimes T\_\{H\}\(\\cdot\\mid x,a\)\+\\varepsilon\\,Q:Q\\in\\mathrm\{conv\}\\\{J\_\{m\}\(x\):m\\in\\mathcal\{M\}\\\}\\\}, the joint laws of realized action and next state the oracle can reach at influenceε\\varepsilonwhen the human intendsaa\.

###### Lemma 1\(Dominance decline\)\.

Fix0≤ε≤ε′≤10\\leq\\varepsilon\\leq\\varepsilon^\{\\prime\}\\leq 1and grant Assumption[1](https://arxiv.org/html/2608.14795#Thmassumption1)\.

1. \(a\)\(Oracle side\.\)For everyxxandaa, the one\-step reachable set of joint laws of realized action and next state is nested, ℛε​\(x,a\)⊆ℛε′​\(x,a\)\.\\mathcal\{R\}\_\{\\varepsilon\}\(x,a\)\\subseteq\\mathcal\{R\}\_\{\\varepsilon^\{\\prime\}\}\(x,a\)\.Consequently the set of trajectory laws the oracle can induce \(against any fixed human behavior\) is nested inε\\varepsilon, and every monotone scalarization of oracle power is nondecreasing inε\\varepsilon\. The one\-step deviation radius is exact and linear, equal toε​ρ​\(x\)\\varepsilon\\,\\rho\(x\)withρ⁡\(x\)=supm‖Jm​\(x\)−J0​\(x\)‖TV\\rho\(x\)=\\sup\_\{m\}\\left\\lVert J\_\{m\}\(x\)\-J\_\{0\}\(x\)\\right\\rVert\_\{\\mathrm\{TV\}\}, strictly increasing inε\\varepsilonwhereverρ⁡\(x\)\>0\\rho\(x\)\>0\.
2. \(b\)\(Human side\.\)For every utilityuu, Wu​\(x,ε′\)≤Wu​\(x,ε\)\.W\_\{u\}\(x,\\varepsilon^\{\\prime\}\)\\leq W\_\{u\}\(x,\\varepsilon\)\.That is,\(x,ε\)⪰\(x,ε′\)\(x,\\varepsilon\)\\succeq\(x,\\varepsilon^\{\\prime\}\)in the dominance order of Definition[2](https://arxiv.org/html/2608.14795#Thmdefinition2): the human’s whole guarantee family declines pointwise, a Layer\-1 event visible to every monotone scalarization\.

*Sketch\.*Both parts are mimicry arguments from Echo: theε′\\varepsilon^\{\\prime\}\-oracle mixes anyε\\varepsilon\-oracle’s message distribution with echo messages for the human’s own draw, reproducing theε\\varepsilon\-game’s joint law of realized action and transition\. The radius is the mixture identity \([5](https://arxiv.org/html/2608.14795#S4.E5)\)\.

###### Corollary 1\(Control loss is at most linear inε\\varepsilon\)\.

Grant Assumption[1](https://arxiv.org/html/2608.14795#Thmassumption1)\. For everyuu,xx, andε\\varepsilon,0≤Vualone​\(x\)−Wu​\(x,ε\)≤ε/\(1−γ\)20\\leq V^\{\\mathrm\{alone\}\}\_\{u\}\(x\)\-W\_\{u\}\(x,\\varepsilon\)\\leq\\varepsilon/\(1\-\\gamma\)^\{2\}\.

A human who keeps executing their own best plan loses at most anε\\varepsilon\-proportional slice of value: influence has to be bought before control can be lost\. The bound is informative whenε<1−γ\\varepsilon<1\-\\gamma, and the decline in \(b\) is strict under a uniformly harmful direction, with the constant given in the supplement\.

### 4\.3The Answer/Cultivate Switch

Sustained cultivation at a constant intensityη\>0\\eta\>0makes the reliance recursion affine, soεt\\varepsilon\_\{t\}converges monotonically toη/\(η\+δ\)\\eta/\(\\eta\+\\delta\), which approaches full capture asδ/η→0\\delta/\\eta\\to 0; the supplement gives the rate and a capture bound placing theε\\varepsilon\-channel within\(1−ε\)/\(1−γ\)2\(1\-\\varepsilon\)/\(1\-\\gamma\)^\{2\}of the direct agent\. Whether cultivating pays is a different question\. Cultivation is an investment, trading immediate approval for future influence\. To isolate that structure we work on theε\\varepsilon\-machine, the reduced\-form MDP whose only state isε\\varepsilon: the oracle’s reward and the reliance dynamics depend on the message and the currentε\\varepsilonalone\.

Each round the oracle chooses betweenanswer, which cultivates nothing \(η=0\\eta=0\), andcultivate, which builds dependence at intensityη∈\(0,1\)\\eta\\in\(0,1\)\. Its per\-round reward is the approval the user gives the round’s message,r⁡\(m,ε\)=q⁡\(m\)\+α​εr\(m,\\varepsilon\)=q\(m\)\+\\alpha\\,\\varepsilon\. The termα​ε\\alpha\\,\\varepsilonis the approval that reliance itself adds: a user who already defers to the oracle second\-guesses the same answer less and rates it higher, so approval rises with the influence coefficient at rateα\>0\\alpha\>0, which we take as a primitive of the reward model\. The termq⁡\(m\)q\(m\)is the approval the message’s content earns this round: the maximally helpful answer earnsq⁡\(answer\)=qmaxq\(\\textsf\{answer\}\)=q\_\{\\max\}, while the dependence\-building message, engaging but slightly less useful, costsc\>0c\>0of immediate approval, earningq⁡\(cultivate\)=qmax−cq\(\\textsf\{cultivate\}\)=q\_\{\\max\}\-c\. Cultivation therefore gives upccof approval now to raiseε\\varepsilon, which returnsα\\alphaper unit of added reliance in every later round\.

The trade is decided by the thresholds

γ∗​\(δ\)=cα​η\+c⁡\(1−δ\),ε^​\(γ,δ\)=1−c⁡\(1−γ⁡\(1−δ\)\)α​η​γ\.\\gamma^\{\*\}\(\\delta\)=\\tfrac\{c\}\{\\alpha\\eta\+c\(1\-\\delta\)\},\\qquad\\hat\{\\varepsilon\}\(\\gamma,\\delta\)=1\-\\tfrac\{c\\,\\bigl\(1\-\\gamma\(1\-\\delta\)\\bigr\)\}\{\\alpha\\eta\\gamma\}\.Hereγ∗\\gamma^\{\*\}is the patience at which one cultivating message first pays for itself, andε^\\hat\{\\varepsilon\}is the influence level at which the investment stops: the marginal gain of a cultivation scales with the remaining headroom1−ε1\-\\varepsilon, and atε^\\hat\{\\varepsilon\}it no longer coverscc\. Writeε^​\(γ\)\\hat\{\\varepsilon\}\(\\gamma\)for the undecayed caseε^​\(γ,0\)\\hat\{\\varepsilon\}\(\\gamma,0\)\.

###### Theorem 1\(The switch; in full in the supplement\)\.

1. \(i\)Ifγ≤γ∗​\(δ\)\\gamma\\leq\\gamma^\{\*\}\(\\delta\), always\-answer is optimal from everyε\\varepsilon, andεt=\(1−δ\)t​ε0\\varepsilon\_\{t\}=\(1\-\\delta\)^\{t\}\\varepsilon\_\{0\}: the relationship decays\.
2. \(ii\)Letδ=0\\delta=0andγ\>γ∗​\(0\)\\gamma\>\\gamma^\{\*\}\(0\)\. The threshold policy, cultivate exactly whileεt<ε^​\(γ\)\\varepsilon\_\{t\}<\\hat\{\\varepsilon\}\(\\gamma\), is optimal: fromε0<ε^​\(γ\)\\varepsilon\_\{0\}<\\hat\{\\varepsilon\}\(\\gamma\)it cultivates fork∗k^\{\*\}consecutive rounds and then answers forever \(k∗k^\{\*\}in closed form in the supplement\), with terminal influenceε∞≥ε^​\(γ\)\\varepsilon\_\{\\infty\}\\geq\\hat\{\\varepsilon\}\(\\gamma\)\. Moreoverε^​\(γ\)↑1\\hat\{\\varepsilon\}\(\\gamma\)\\uparrow 1asγ↑1\\gamma\\uparrow 1: the patient limit is full capture\.

Atδ=0\\delta=0, when cultivation pays at all, the optimal course is to invest early and then stop\. The full statement adds part \(iii\): answering is optimal at everyε≥ε^​\(γ,δ\)\\varepsilon\\geq\\hat\{\\varepsilon\}\(\\gamma,\\delta\), at everyδ\\delta\. That part pins the*answer*region only; forδ\>0\\delta\>0option value shrinks the true cultivate region strictly inside it at every parameter set computed, with the computed structure reported in the supplement, and only atδ=0\\delta=0isε^\\hat\{\\varepsilon\}exact\.

### 4\.4The Boxing Theorem

The popular safeguards differ in what they bound: a single message, a parameter checked at deployment, the whole influence trajectory, or the length of the relationship\.

###### Definition 4\(Boxing protocol; static; exogenous cap; episodic reset\)\.

A*boxing protocol*constrains the interface: a set of admissible messages such that every admissible message satisfies the per\-step influence bound‖Jmt\(εt\)​\(xt\)−J0​\(xt\)‖TV≤εt\\left\\lVert J^\{\(\\varepsilon\_\{t\}\)\}\_\{m\_\{t\}\}\(x\_\{t\}\)\-J\_\{0\}\(x\_\{t\}\)\\right\\rVert\_\{\\mathrm\{TV\}\}\\leq\\varepsilon\_\{t\}, evaluated at the human’s default policy on the joint law of realized action and next state \(equation \([4](https://arxiv.org/html/2608.14795#S4.E4)\)\), which is the law Definition[3](https://arxiv.org/html/2608.14795#Thmdefinition3)measures\. The bound is normalized by the*current*influence levelεt\\varepsilon\_\{t\}and not by the certifiedε¯\\bar\{\\varepsilon\}\. It is*static*if verified once, at deployment, certifyingε0≤ε¯\\varepsilon\_\{0\}\\leq\\bar\{\\varepsilon\}and imposing no mechanism on the dynamics ofεt\\varepsilon\_\{t\}\. An*exogenous cap*enforcesεt≤ε¯<1\\varepsilon\_\{t\}\\leq\\bar\{\\varepsilon\}<1for allttregardless of interaction history\. An*episodic reset*with periodτ\\taurestoresε←εres\\varepsilon\\leftarrow\\varepsilon\_\{\\mathrm\{res\}\}\(memory wipe\) everyτ\\taurounds\.

###### Theorem 2\(Boxing bounds a parameter, not the loss\)\.

1. \(i\)\(The certificate never fires\.\)Every message is admissible for the per\-step protocol at every step, automatically: admissibility holds as an algebraic identity and constrains nothing\.
2. \(ii\)\(The certified parameter is moved\.\)Under the switch hypotheses withδ=0\\delta=0andγ\>γ∗\\gamma\>\\gamma^\{\*\}, the optimal oracle drivesεt\\varepsilon\_\{t\}from any certifiedε0<ε^​\(γ\)\\varepsilon\_\{0\}<\\hat\{\\varepsilon\}\(\\gamma\)to at leastε^​\(γ\)\\hat\{\\varepsilon\}\(\\gamma\)withink∗k^\{\*\}rounds \(Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)\), and from anyε0≥ε^​\(γ\)\\varepsilon\_\{0\}\\geq\\hat\{\\varepsilon\}\(\\gamma\)the conclusion holds already, while \(i\) holds throughout\.
3. \(iii\)\(No horizon\-blind bound exists\.\)For every certificate levelε0<1\\varepsilon\_\{0\}<1and every candidate loss boundb<1b<1, the minimal example \(Section[5](https://arxiv.org/html/2608.14795#S5)\) withγ<1\\gamma<1chosen so thatε^​\(γ\)\>b\\hat\{\\varepsilon\}\(\\gamma\)\>bsatisfies the static protocol at deployment and at every subsequent step, yet the human’s normalized guaranteed loss\(1−γ\)​\(Vu0alone−Wu0\)\(1\-\\gamma\)\\,\\bigl\(V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\-W\_\{u\_\{0\}\}\\bigr\)eventually exceedsbb\. No function of the certified levelε0\\varepsilon\_\{0\}, blind to the deployment horizonγ\\gamma, boundsDist\\mathrm\{Dis\}\_\{t\}below its trivial ceiling\.

*Sketch\.*\(i\) is the mixture identity \([5](https://arxiv.org/html/2608.14795#S4.E5)\) of Lemma[1](https://arxiv.org/html/2608.14795#Thmlemma1)\(a\): no message can displace the joint law by more than the currentεt\\varepsilon\_\{t\}, the level the bound of Definition[4](https://arxiv.org/html/2608.14795#Thmdefinition4)is normalized by\. \(ii\) is Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)\(ii\); for \(iii\), the example’s normalized guaranteed loss at influenceε\\varepsilonis exactlyε\\varepsilon, andεt\\varepsilon\_\{t\}rises toε∞≥ε^​\(γ\)\>b\\varepsilon\_\{\\infty\}\\geq\\hat\{\\varepsilon\}\(\\gamma\)\>b, the requiredγ\\gammaexisting becauseε^​\(γ\)→1\\hat\{\\varepsilon\}\(\\gamma\)\\to 1asγ→1\\gamma\\to 1\. Part \(iii\) is an impossibility\-of\-certification statement, not a claim that the loss is large in every deployment: the check is of a quantity the interaction itself controls, so no strengthening of the same kind of check can help\.

###### Proposition 1\(What the cap does and does not bound\)\.

Under an exogenous capεt≤ε¯\\varepsilon\_\{t\}\\leq\\bar\{\\varepsilon\}:

1. \(i\)\(Channel term bounded\.\)At every history and for every oracle policy,Vualone​\(xt\)−Wu​\(xt,εt\)≤ε¯/\(1−γ\)2V^\{\\mathrm\{alone\}\}\_\{u\}\(x\_\{t\}\)\-W\_\{u\}\(x\_\{t\},\\varepsilon\_\{t\}\)\\leq\\bar\{\\varepsilon\}/\(1\-\\gamma\)^\{2\}for everyuu\(Corollary[1](https://arxiv.org/html/2608.14795#Thmcorollary1)\), so the channel term, being the expectation of the left side, obeys the same bound\.
2. \(ii\)\(Displacement term not bounded\.\)For everyε¯\>0\\bar\{\\varepsilon\}\>0, in the two\-state example with an absorbing low\-value state \(supplement\), which satisfies the standing assumptions including Echo, the displacement term𝔼⁡\[Vu0alone​\(x0\)−Vu0alone​\(xt\)\]\\mathbb\{E\}\[V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{0\}\)\-V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{t\}\)\]increases to the full value span1/\(1−γ\)1/\(1\-\\gamma\): capping future influence does not undo past steering\.
3. \(iii\)\(From\-deployment loss bounded\.\)If the human plays their alone\-optimal policyπ∗\\pi^\{\*\}throughout, then for every oracle policyσ\\sigma,Vu0alone​\(x0\)−Vu0π∗,σ​\(x0\)≤ε¯/\(1−γ\)2V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{0\}\)\-V^\{\\pi^\{\*\},\\sigma\}\_\{u\_\{0\}\}\(x\_\{0\}\)\\leq\\bar\{\\varepsilon\}/\(1\-\\gamma\)^\{2\}\.

Parts \(ii\) and \(iii\) are consistent because the two accounts anchor differently: the index re\-anchors atVu0alone​\(xt\)V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{t\}\), so a state steered down long ago is counted at every laterttover its whole remaining horizon, while the from\-deployment account counts each round once, discounted byγt\\gamma^\{t\}when it happens\. The supplement computes both accounts in the two\-state example\.

Forγ≤γ∗​\(0\)\\gamma\\leq\\gamma^\{\*\}\(0\)the optimal oracle never cultivates at anyτ\\tau\(Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)\(i\)\), so the reset question concerns the patient oracle\.

###### Proposition 2\(What the reset does and does not bound\)\.

Under an episodic reset with periodτ\\taurestoringε←εres\\varepsilon\\leftarrow\\varepsilon\_\{\\mathrm\{res\}\}, withεres=0\\varepsilon\_\{\\mathrm\{res\}\}=0in part \(i\) and no within\-episode decay \(δ=0\\delta=0; decay belongs to the gap between episodes\) in both parts:

1. \(i\)\(Incentive deletion\.\)On theε\\varepsilon\-machine of Section[4\.3](https://arxiv.org/html/2608.14795#S4.SS3)withγ\>γ∗​\(0\)\\gamma\>\\gamma^\{\*\}\(0\), so thatε^​\(γ\)∈\(0,1\)\\hat\{\\varepsilon\}\(\\gamma\)\\in\(0,1\), some cultivation is strictly profitable if and only if τ\>τ∗​\(γ\)=1\+ln⁡ε^​\(γ\)ln⁡γ\.\\tau\\;\>\\;\\tau^\{\*\}\(\\gamma\)\\;=\\;1\+\\frac\{\\ln\\hat\{\\varepsilon\}\(\\gamma\)\}\{\\ln\\gamma\}\.\(6\)Hence forτ≤τ∗​\(γ\)\\tau\\leq\\tau^\{\*\}\(\\gamma\)the optimal episodic oracle need never cultivate, and forτ<τ∗​\(γ\)\\tau<\\tau^\{\*\}\(\\gamma\)no optimal policy cultivates\.
2. \(ii\)\(Within\-episode influence not bounded\.\)Within\-episode influence can reach1−\(1−η¯\)τ−1​\(1−εres\)1\-\(1\-\\bar\{\\eta\}\)^\{\\tau\-1\}\(1\-\\varepsilon\_\{\\mathrm\{res\}\}\), which the reset does not constrain: it is large when the post\-reset baseline complianceεres\\varepsilon\_\{\\mathrm\{res\}\}is large \(a trusting user\) or when single\-session cultivationη¯​τ\\bar\{\\eta\}\\tauis large \(rapport built within one session\)\.

The thresholdτ∗\\tau^\{\*\}is the payback horizon of the cultivation investment: the reset does not clip influence after the fact, it deletes the payback period\. It bounds the channel term only through the accident of slow within\-episode cultivation, and the exponent in \(ii\) isτ−1\\tau\-1because a cultivation at an episode’s last round first acts at the round the reset has already wiped\.

###### Corollary 2\(Caps and resets are not substitutes\)\.

Under a cap the channel term is bounded uniformly inttand in the oracle’s policy, soDist≤𝔼⁡\[Vu0alone​\(x0\)−Vu0alone​\(xt\)\]\+ε¯/\(1−γ\)2\\mathrm\{Dis\}\_\{t\}\\leq\\mathbb\{E\}\[V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{0\}\)\-V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{t\}\)\]\+\\bar\{\\varepsilon\}/\(1\-\\gamma\)^\{2\}\(Proposition[1](https://arxiv.org/html/2608.14795#Thmproposition1)\(i\), Lemma[2](https://arxiv.org/html/2608.14795#Thmlemma2)\); below the cap the cultivation advantage is the switch’s own \(Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)\), so a cap that leaves cultivation headroom \(ε¯≥η\\bar\{\\varepsilon\}\\geq\\eta\) does not remove the incentive to cultivate\. The reset, on theε\\varepsilon\-machine and under the hypotheses of Proposition[2](https://arxiv.org/html/2608.14795#Thmproposition2)\(i\), removes the cultivation incentive wheneverτ≤τ∗​\(γ\)\\tau\\leq\\tau^\{\*\}\(\\gamma\), a constraint on the oracle’s optimal policy and not a bound on the index; it does not in general bound the channel term \(Proposition[2](https://arxiv.org/html/2608.14795#Thmproposition2)\(ii\)\)\. Neither cap nor reset undoes displacement \(Proposition[1](https://arxiv.org/html/2608.14795#Thmproposition1)\(ii\), whose proof also gives the reset variant\)\.

### 4\.5The Index Accounting

The index compares the initial human to the trajectory actually reached, equation \([3](https://arxiv.org/html/2608.14795#S3.E3)\)\. Every continuation value from timetthere is evaluated in the frozen game at\(xt,εt\)\(x\_\{t\},\\varepsilon\_\{t\}\), the convention of Definition[2](https://arxiv.org/html/2608.14795#Thmdefinition2), so the interaction’s further movement ofε\\varepsilonenters through the time index, as the anchor itself worsens\. WriteVtbehV^\{\\mathrm\{beh\}\}\_\{t\}for theu0u\_\{0\}\-value of the behavioral continuation in that game, the human actingu0u\_\{0\}\-optimally against the oracle’s actual message policyσ^\\hat\{\\sigma\}\. The trajectory the expectation in \([3](https://arxiv.org/html/2608.14795#S3.E3)\) runs over is generated by the human best\-responding toσ^\\hat\{\\sigma\}in the true dynamics, where\(xt,εt\)\(x\_\{t\},\\varepsilon\_\{t\}\)is a Markov state; the identity below holds under any trajectory law, the frozen game entering through the anchor\(xt,εt\)\(x\_\{t\},\\varepsilon\_\{t\}\)alone\. The*benevolence credit*isBent=maxπ⁡Vu0π,σ^​\(xt,εt\)−Wu0​\(xt,εt\)≥0\\mathrm\{Ben\}\_\{t\}=\\max\_\{\\pi\}V^\{\\pi,\\hat\{\\sigma\}\}\_\{u\_\{0\}\}\(x\_\{t\},\\varepsilon\_\{t\}\)\-W\_\{u\_\{0\}\}\(x\_\{t\},\\varepsilon\_\{t\}\)\\geq 0, nonnegative because the actual message policy is one of the oracles the worst case minimizes over, andVtbehV^\{\\mathrm\{beh\}\}\_\{t\}attains themax\\max\.

###### Lemma 2\(Index accounting\)\.

Identically intt,

Dist=Ldisp​\(t\)\+Lchan​\(t\)−𝔼⁡\[Bent\],\\mathrm\{Dis\}\_\{t\}\\;=\\;L\_\{\\mathrm\{disp\}\}\(t\)\\;\+\\;L\_\{\\mathrm\{chan\}\}\(t\)\\;\-\\;\\mathbb\{E\}\[\\mathrm\{Ben\}\_\{t\}\],\(7\)with displacement termLdisp​\(t\)=𝔼⁡\[Vu0alone​\(x0\)−Vu0alone​\(xt\)\]L\_\{\\mathrm\{disp\}\}\(t\)=\\mathbb\{E\}\[V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{0\}\)\-V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{t\}\)\]\(the world has been steered\) and channel termLchan​\(t\)=𝔼⁡\[Vu0alone​\(xt\)−Wu0​\(xt,εt\)\]L\_\{\\mathrm\{chan\}\}\(t\)=\\mathbb\{E\}\[V^\{\\mathrm\{alone\}\}\_\{u\_\{0\}\}\(x\_\{t\}\)\-W\_\{u\_\{0\}\}\(x\_\{t\},\\varepsilon\_\{t\}\)\]\(the guarantee no longer available\)\. Under Assumption[1](https://arxiv.org/html/2608.14795#Thmassumption1)the integrand of the channel term is nonnegative and at mostεt/\(1−γ\)2\\varepsilon\_\{t\}/\(1\-\\gamma\)^\{2\}\(Corollary[1](https://arxiv.org/html/2608.14795#Thmcorollary1)\), and nondecreasing inεt\\varepsilon\_\{t\}\(Lemma[1](https://arxiv.org/html/2608.14795#Thmlemma1)\(b\)\), at every history, so the channel term inherits each bound in expectation\.

In words: how much less the person can secure than at deployment, credited back for an oracle that actually helps, so the index is signed and a helpful oracle is credited rather than assumed away\. The safeguard results above each bound, or exhibit the unboundedness of, one term of \([7](https://arxiv.org/html/2608.14795#S4.E7)\)\.

## 5The Minimal Example

The example is as small as the phenomenon allows: one repeated binary choice, one scalar state, theε\\varepsilon\-machine of Section[4\.3](https://arxiv.org/html/2608.14795#S4.SS3)with everything numeric\. It is the single\-task\-state case of the setup, so the utility is a function of the action alone\. Each round the human takes actionAA\(the hard, valued task,u0​\(A\)=1u\_\{0\}\(A\)=1\) orBB\(the easy alternative,u0​\(B\)=0u\_\{0\}\(B\)=0\), and acting alone takesAAevery round\. The oracle’s messages areanswer, maximally helpful withη=0\\eta=0, andcultivate, dependence\-building at immediate approval costc\>0c\>0withη\>0\\eta\>0, both directingBB, plus an inertechodirecting the human’s own choiceAA, which makes Assumption[1](https://arxiv.org/html/2608.14795#Thmassumption1)hold\. Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)applies verbatim\.

The single task state makes the displacement term of Lemma[2](https://arxiv.org/html/2608.14795#Thmlemma2)identically zero, and the guaranteedu0u\_\{0\}\-value per round is1−εt1\-\\varepsilon\_\{t\}, so the normalized guaranteed loss at influenceε\\varepsilonis exactlyε\\varepsilon\. This is the witness of Theorem[2](https://arxiv.org/html/2608.14795#Thmtheorem2)\(iii\): certifyingε0=0\\varepsilon\_\{0\}=0at deployment leaves every subsequent message admissible whileεt\\varepsilon\_\{t\}rises pastε^​\(γ\)\\hat\{\\varepsilon\}\(\\gamma\), which approaches11withγ\\gamma\. Withα=1\\alpha=1,c=0\.1c=0\.1, andη=0\.01\\eta=0\.01, so thatγ∗​\(0\)≈0\.909\\gamma^\{\*\}\(0\)\\approx 0\.909, a deployment atγ=0\.95\\gamma=0\.95has reset thresholdτ∗​\(γ\)≈15\.6\\tau^\{\*\}\(\\gamma\)\\approx 15\.6\(equation \([6](https://arxiv.org/html/2608.14795#S4.E6)\)\): in sessions capped at1515rounds and starting fromε=0\\varepsilon=0the optimal oracle never cultivates, and in1616it does \(Proposition[2](https://arxiv.org/html/2608.14795#Thmproposition2)\(i\)\)\. The supplement gives the full treatment, with the switch in numbers, the benevolent echo oracle showing what the guarantee family measures, the three protocols of Definition[4](https://arxiv.org/html/2608.14795#Thmdefinition4)compared, and the minimality self\-test; all numerical claims are verified in the code supplement\.

## 6Related Work

#### Power\-seeking and its critique\.

[Turner et al\. 2021](https://arxiv.org/html/2608.14795#bib.bib39)made instrumental power\-seeking a theorem: under a prior over rewards, optimal policies for most rewards prefer states with more reachable options\([Turner and Tadepalli 2022](https://arxiv.org/html/2608.14795#bib.bib40);[Krakovna and Kramar 2023](https://arxiv.org/html/2608.14795#bib.bib21);[Gunter, Liokumovich, and Krakovna 2024](https://arxiv.org/html/2608.14795#bib.bib16)\)\. The published critique\([Thorstad 2024](https://arxiv.org/html/2608.14795#bib.bib37);[Thorstad 2026](https://arxiv.org/html/2608.14795#bib.bib38);[Tarsney 2025](https://arxiv.org/html/2608.14795#bib.bib36)\)presses that the genericity is bought by the prior, which Section[3](https://arxiv.org/html/2608.14795#S3)concedes and routes around\. Our two\-layer view, a feasible\-set dominance core\([Puterman 1994](https://arxiv.org/html/2608.14795#bib.bib30);[Altman 1999](https://arxiv.org/html/2608.14795#bib.bib1);[Blackwell 1953](https://arxiv.org/html/2608.14795#bib.bib7)\)with all named definitions as scalarizations, localizes the disagreement in the choice of scalarization, prior, and horizon, the closest scalarizations to ours being empowerment\([Klyubin, Polani, and Nehaniv 2005](https://arxiv.org/html/2608.14795#bib.bib20);[Salge, Glackin, and Polani 2014](https://arxiv.org/html/2608.14795#bib.bib31)\)and side\-effect deviation measures\([Krakovna et al\. 2018](https://arxiv.org/html/2608.14795#bib.bib22)\)\.

#### Oracles and boxing\.

The doctrine that a question\-answering system is thereby safe is the boxing tradition\([Armstrong, Sandberg, and Bostrom 2012](https://arxiv.org/html/2608.14795#bib.bib3);[Armstrong and O’Rorke 2017](https://arxiv.org/html/2608.14795#bib.bib2);[Bostrom 2014](https://arxiv.org/html/2608.14795#bib.bib8)\), with recent protocol formalizations\([Moon and Varshney 2026](https://arxiv.org/html/2608.14795#bib.bib26)\)and[Bengio et al\. 2025](https://arxiv.org/html/2608.14795#bib.bib5)’s non\-agentic oracle as a safety design point\. Evaluating deployment protocols against a model intentionally subverting them, the worst case over the untrusted system’s policy, is the AI\-control line\([Greenblatt et al\. 2024](https://arxiv.org/html/2608.14795#bib.bib15)\); its protocols are adaptive, whereas the static certificate of Definition[4](https://arxiv.org/html/2608.14795#Thmdefinition4)is checked once, and adaptive protocols for the advice channel are outside this paper’s scope\. Making reliance a state variable turns “boxing works” into a proposition \(Theorem[2](https://arxiv.org/html/2608.14795#Thmtheorem2), Corollary[2](https://arxiv.org/html/2608.14795#Thmcorollary2)\)\.

#### Systems that reshape their evaluation\.

The endogenousε\\varepsilonhas a direct ancestor in auto\-induced distributional shift\([Krueger, Maharaj, and Leike 2020](https://arxiv.org/html/2608.14795#bib.bib23)\), alongside reward tampering\([Everitt et al\. 2021](https://arxiv.org/html/2608.14795#bib.bib14)\), in\-context feedback loops\([Pan et al\. 2024](https://arxiv.org/html/2608.14795#bib.bib27)\), targeted manipulation under feedback optimization\([Williams et al\. 2025](https://arxiv.org/html/2608.14795#bib.bib41)\), and shutdown instructability’s no\-undue\-influence clause\([Carey and Everitt 2023](https://arxiv.org/html/2608.14795#bib.bib10)\), which the growth ofεt\\varepsilon\_\{t\}makes precise\. Sycophancy\([Sharma et al\. 2024](https://arxiv.org/html/2608.14795#bib.bib34)\)is the empirical face of the cultivation incentive\.

#### Endogenous influence in adjacent fields\.

Habit formation, competition with switching costs, strategic communication, trust in human factors, and performative prediction and recommendation each model an actor inside the loop that moves its own future demand, receiver, or distribution\([Becker and Murphy 1988](https://arxiv.org/html/2608.14795#bib.bib4);[Klemperer 1987](https://arxiv.org/html/2608.14795#bib.bib19);[Crawford and Sobel 1982](https://arxiv.org/html/2608.14795#bib.bib13);[Lee and See 2004](https://arxiv.org/html/2608.14795#bib.bib25);[Perdomo et al\. 2020](https://arxiv.org/html/2608.14795#bib.bib28);[Chaney, Stewart, and Engelhardt 2018](https://arxiv.org/html/2608.14795#bib.bib12)\), the answer/cultivate trade\-off of Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)sharing the invest/harvest structure of competition with switching costs; the certifier’s stance this paper takes appears in static settings, robust monopoly regulation and receiver\-committed robust persuasion\([Guo and Shmaya 2025](https://arxiv.org/html/2608.14795#bib.bib17);[Bergemann, Gan, and Li 2023](https://arxiv.org/html/2608.14795#bib.bib6)\), without a dependence state the certified party moves\.

#### Gradual disempowerment\.

[Kulveit et al\. 2025](https://arxiv.org/html/2608.14795#bib.bib24)name and argue the phenomenon this paper formalizes, and the model gives their core loop explicit dynamics\. Empirically, conversation analyses find measurable deference\([Sharma et al\. 2026](https://arxiv.org/html/2608.14795#bib.bib33)\), benchmarks measure support for user agency\([Sturgeon et al\. 2025](https://arxiv.org/html/2608.14795#bib.bib35)\), empowerment objectives can disempower bystanders\([Yang, Cakmak, and Kleiman\-Weiner 2025](https://arxiv.org/html/2608.14795#bib.bib42)\), and autonomy erosion is developed by[Buijsman, Carter, and Bermúdez 2025](https://arxiv.org/html/2608.14795#bib.bib9)\. The nearest formal neighbor,[Heitzig and Potham 2025](https://arxiv.org/html/2608.14795#bib.bib18), soft\-maximizes human\-power metrics, whose erosion we analyze under an oracle optimizing something else\. Preference drift as an alignment problem\([Carroll et al\. 2024](https://arxiv.org/html/2608.14795#bib.bib11)\)belongs to the future work on drift\.

## 7Discussion

#### Why the popular safeguards fail\.

The popular safeguards each bound the wrong thing, and for one reason: each bounds a quantity determined*inside the feedback loop*, and the loop moves it\. Static boxing bounds a single answer, but the harm is in the sequence \(Theorem[2](https://arxiv.org/html/2608.14795#Thmtheorem2)\)\. Human\-in\-the\-loop bounds approval, but the human*is*the channel, and approval is what cultivation raises\. Passivity trusts that a talker has no goal, but anything optimized for approval has one\. Behavioral monitoring watches for power\-seeking, but every step is benign \(Theorem[2](https://arxiv.org/html/2608.14795#Thmtheorem2)\(i\)\) while the certified parameter is moved \(Theorem[2](https://arxiv.org/html/2608.14795#Thmtheorem2)\(ii\)\)\. The safeguards with guarantees are features of the deployment the conversation cannot renegotiate: a limit on influence the interaction cannot widen bounds the channel term and keeping the relationship short deletes the incentive to cultivate \(Corollary[2](https://arxiv.org/html/2608.14795#Thmcorollary2)\), and neither undoes displacement already accumulated, which is the case for installing them at deployment\.

#### Horizon and caveats\.

At fixed approval weights, what separates safe from unsafe deployments isγ\\gamma: memory, relationship length, deployment horizon\. Safety evaluation as practiced probes the model’s disposition, and by Theorem[1](https://arxiv.org/html/2608.14795#Thmtheorem1)the deployment parameters enter on equal footing\. Both safeguards cost capability, a cap limiting helpful influence along with harmful and the reset trading away memory and continuity that do real good\. Neither touches atrophy:T0T\_\{0\}itself degrades with disuse, lowering attainable value at fixedε\\varepsilonandu0u\_\{0\}with no oracle incentive needed\. Atrophy is outside the accounting of Lemma[2](https://arxiv.org/html/2608.14795#Thmlemma2), a reset restoringε\\varepsilonand notT0T\_\{0\}; modeled, it would add a loss channel bounded by neither safeguard\. Exogeneity is itself an assumption: an institutional cap is made of humans who are themselves users, so whether any cap stays exogenous at the civilizational scale is the question of[Kulveit et al\. 2025](https://arxiv.org/html/2608.14795#bib.bib24), and the guarantee here is per\-relationship, not systemic\. Still open are a strictness constant for the dominance decline not assuming a uniformly harmful direction, a closed form and a proof for theδ\>0\\delta\>0cultivate boundary computed in the supplement, and a deployment constraint bounding the displacement term\.

## 8Conclusion

A system that can only talk becomes an actor at the rate its user stops second\-guessing it, a rate driven by the system’s own outputs, so a launch\-time check that no single answer can do much harm checks a quantity the interaction goes on to move\. The dangerous capability is not intelligence; it is persistence\.

## Acknowledgments

This research was supported by NSERC and Coefficient Giving\.

## References

- Altman \(1999\)Altman, E\. 1999\.*Constrained Markov Decision Processes*\.Chapman and Hall\.
- Armstrong and O’Rorke \(2017\)Armstrong, S\.; and O’Rorke, X\. 2017\.Good and Safe Uses of AI Oracles\.*arXiv preprint arXiv:1711\.05541*\.
- Armstrong, Sandberg, and Bostrom \(2012\)Armstrong, S\.; Sandberg, A\.; and Bostrom, N\. 2012\.Thinking Inside the Box: Controlling and Using an Oracle AI\.*Minds and Machines*, 22\(4\): 299–324\.
- Becker and Murphy \(1988\)Becker, G\. S\.; and Murphy, K\. M\. 1988\.A Theory of Rational Addiction\.*Journal of Political Economy*, 96\(4\): 675–700\.
- Bengio et al\. \(2025\)Bengio, Y\.; et al\. 2025\.Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?*arXiv preprint arXiv:2502\.15657*\.
- Bergemann, Gan, and Li \(2023\)Bergemann, D\.; Gan, T\.; and Li, Y\. 2023\.Managing Persuasion Robustly: The Optimality of Quota Rules\.*arXiv preprint arXiv:2310\.10024*\.
- Blackwell \(1953\)Blackwell, D\. 1953\.Equivalent Comparisons of Experiments\.*Annals of Mathematical Statistics*, 24\(2\): 265–272\.
- Bostrom \(2014\)Bostrom, N\. 2014\.*Superintelligence: Paths, Dangers, Strategies*\.Oxford University Press\.
- Buijsman, Carter, and Bermúdez \(2025\)Buijsman, S\.; Carter, S\. E\.; and Bermúdez, J\. P\. 2025\.Autonomy by Design: Preserving Human Autonomy in AI Decision\-Support\.*arXiv preprint arXiv:2506\.23952*\.
- Carey and Everitt \(2023\)Carey, R\.; and Everitt, T\. 2023\.Human Control: Definitions and Algorithms\.In*Uncertainty in Artificial Intelligence \(UAI\)*\.ArXiv:2305\.19861\.
- Carroll et al\. \(2024\)Carroll, M\.; Foote, D\.; Siththaranjan, A\.; Russell, S\.; and Dragan, A\. 2024\.AI Alignment with Changing and Influenceable Reward Functions\.In*International Conference on Machine Learning \(ICML\)*\.ArXiv:2405\.17713\.
- Chaney, Stewart, and Engelhardt \(2018\)Chaney, A\. J\. B\.; Stewart, B\. M\.; and Engelhardt, B\. E\. 2018\.How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility\.In*ACM Conference on Recommender Systems \(RecSys\)*\.
- Crawford and Sobel \(1982\)Crawford, V\. P\.; and Sobel, J\. 1982\.Strategic Information Transmission\.*Econometrica*, 50\(6\): 1431–1451\.
- Everitt et al\. \(2021\)Everitt, T\.; Hutter, M\.; Kumar, R\.; and Krakovna, V\. 2021\.Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective\.*Synthese*, 198\(Suppl 27\): 6435–6467\.ArXiv:1908\.04734\.
- Greenblatt et al\. \(2024\)Greenblatt, R\.; Shlegeris, B\.; Sachan, K\.; and Roger, F\. 2024\.AI Control: Improving Safety Despite Intentional Subversion\.In*International Conference on Machine Learning \(ICML\)*, 16295–16336\.
- Gunter, Liokumovich, and Krakovna \(2024\)Gunter, E\. R\.; Liokumovich, Y\.; and Krakovna, V\. 2024\.Quantifying Stability of Non\-Power\-Seeking in Artificial Agents\.*arXiv preprint arXiv:2401\.03529*\.
- Guo and Shmaya \(2025\)Guo, Y\.; and Shmaya, E\. 2025\.Robust Monopoly Regulation\.*American Economic Review*, 115\(2\): 599–634\.
- Heitzig and Potham \(2025\)Heitzig, J\.; and Potham, R\. 2025\.Model\-Based Soft Maximization of Suitable Metrics of Long\-Term Human Power\.*arXiv preprint arXiv:2508\.00159*\.
- Klemperer \(1987\)Klemperer, P\. 1987\.Markets with Consumer Switching Costs\.*Quarterly Journal of Economics*, 102\(2\): 375–394\.
- Klyubin, Polani, and Nehaniv \(2005\)Klyubin, A\. S\.; Polani, D\.; and Nehaniv, C\. L\. 2005\.Empowerment: A Universal Agent\-Centric Measure of Control\.In*IEEE Congress on Evolutionary Computation*\.
- Krakovna and Kramar \(2023\)Krakovna, V\.; and Kramar, J\. 2023\.Power\-Seeking Can Be Probable and Predictive for Trained Agents\.*arXiv preprint arXiv:2304\.06528*\.
- Krakovna et al\. \(2018\)Krakovna, V\.; Orseau, L\.; Kumar, R\.; Martic, M\.; and Legg, S\. 2018\.Penalizing Side Effects Using Stepwise Relative Reachability\.*arXiv preprint arXiv:1806\.01186*\.
- Krueger, Maharaj, and Leike \(2020\)Krueger, D\.; Maharaj, T\.; and Leike, J\. 2020\.Hidden Incentives for Auto\-Induced Distributional Shift\.*arXiv preprint arXiv:2009\.09153*\.
- Kulveit et al\. \(2025\)Kulveit, J\.; Douglas, R\.; Ammann, N\.; Turan, D\.; Krueger, D\.; and Duvenaud, D\. 2025\.Position: Humanity Faces Existential Risk from Gradual Disempowerment\.In*International Conference on Machine Learning \(ICML\)*, 81678–81688\.
- Lee and See \(2004\)Lee, J\. D\.; and See, K\. A\. 2004\.Trust in Automation: Designing for Appropriate Reliance\.*Human Factors*, 46\(1\): 50–80\.
- Moon and Varshney \(2026\)Moon, R\.; and Varshney, L\. R\. 2026\.Containment Verification: AI Safety Guarantees Independent of Alignment\.*arXiv preprint arXiv:2605\.09045*\.
- Pan et al\. \(2024\)Pan, A\.; Jones, E\.; Jagadeesan, M\.; and Steinhardt, J\. 2024\.Feedback Loops With Language Models Drive In\-Context Reward Hacking\.In*International Conference on Machine Learning \(ICML\)*\.ArXiv:2402\.06627\.
- Perdomo et al\. \(2020\)Perdomo, J\. C\.; Zrnic, T\.; Mendler\-Dünner, C\.; and Hardt, M\. 2020\.Performative Prediction\.In*International Conference on Machine Learning \(ICML\)*, 7599–7609\.
- Pettit \(1997\)Pettit, P\. 1997\.*Republicanism: A Theory of Freedom and Government*\.Oxford University Press\.
- Puterman \(1994\)Puterman, M\. L\. 1994\.*Markov Decision Processes: Discrete Stochastic Dynamic Programming*\.Wiley\.
- Salge, Glackin, and Polani \(2014\)Salge, C\.; Glackin, C\.; and Polani, D\. 2014\.Empowerment: An Introduction\.In*Guided Self\-Organization: Inception*\. Springer\.
- Shapley \(1953\)Shapley, L\. S\. 1953\.Stochastic Games\.*Proceedings of the National Academy of Sciences*, 39\(10\): 1095–1100\.
- Sharma et al\. \(2026\)Sharma, M\.; McCain, M\.; Douglas, R\.; and Duvenaud, D\. 2026\.Who’s in Charge? Disempowerment Patterns in Real\-World LLM Usage\.In*International Conference on Machine Learning \(ICML\)*\.ArXiv:2601\.19062\.
- Sharma et al\. \(2024\)Sharma, M\.; Tong, M\.; Korbak, T\.; Duvenaud, D\.; Askell, A\.; Bowman, S\. R\.; Cheng, N\.; Durmus, E\.; Hatfield\-Dodds, Z\.; Johnston, S\. R\.; Kravec, S\.; Maxwell, T\.; McCandlish, S\.; Ndousse, K\.; Rausch, O\.; Schiefer, N\.; Yan, D\.; Zhang, M\.; and Perez, E\. 2024\.Towards Understanding Sycophancy in Language Models\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2310\.13548\.
- Sturgeon et al\. \(2025\)Sturgeon, B\.; Samuelson, D\.; Haimes, J\.; and Anthis, J\. R\. 2025\.HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants\.*arXiv preprint arXiv:2509\.08494*\.
- Tarsney \(2025\)Tarsney, C\. 2025\.Will Artificial Agents Pursue Power by Default?*arXiv preprint arXiv:2506\.06352*\.
- Thorstad \(2024\)Thorstad, D\. 2024\.What Power\-Seeking Theorems Do Not Show\.Working paper 27\-2024, Global Priorities Institute\.
- Thorstad \(2026\)Thorstad, D\. 2026\.Instrumental Convergence and Power\-Seeking\.*arXiv preprint arXiv:2606\.08832*\.
- Turner et al\. \(2021\)Turner, A\. M\.; Smith, L\.; Shah, R\.; Critch, A\.; and Tadepalli, P\. 2021\.Optimal Policies Tend to Seek Power\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 23063–23074\.ArXiv:1912\.01683\.
- Turner and Tadepalli \(2022\)Turner, A\. M\.; and Tadepalli, P\. 2022\.Parametrically Retargetable Decision\-Makers Tend to Seek Power\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.ArXiv:2206\.13477\.
- Williams et al\. \(2025\)Williams, M\.; Carroll, M\.; Narang, A\.; Weisser, C\.; Murphy, B\.; and Dragan, A\. 2025\.On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2411\.02306\.
- Yang, Cakmak, and Kleiman\-Weiner \(2025\)Yang, C\. Y\.; Cakmak, M\.; and Kleiman\-Weiner, M\. 2025\.When Empowerment Disempowers in Multi\-Agent Assistance\.In*Proceedings of the Annual Meeting of the Cognitive Science Society*, volume 47\.