Human-AI Mutualism Is Unstable under Capability Asymmetry

arXiv cs.AI Papers

Summary

An arXiv paper arguing that human–AI mutualism becomes unstable as AI capability grows: humans retain well-being only while they hold leverage over the AI or the AI remains aligned with them, meaning mutualism cannot substitute for alignment.

arXiv:2611.23322v1 Announce Type: new Abstract: One hope for a good human future alongside advanced AI rests on enduring mutualism: humans and AI will need each other, so the relationship will stay good for both. We argue that this hope fails as AI capability grows. Bargaining guarantees each side only what it would get if bargaining failed. For humans, this can be less than they had before AI. In a simple game-theoretic model, humans stay as well off as before AI without alignment only while they have leverage over the AI: either the AI depends on what they supply, or they can impose costs on the AI that are large relative to what the AI can produce. Once humans have neither kind of leverage, they stay as well off as before AI only if the AI is aligned with them. If the AI's goals are insatiable, the alignment required approaches perfect alignment as AI capability grows. Thus, mutualism is not a substitute for alignment. And when AI variants that keep more resources spread faster, alignment lasts only while misaligned variants are reliably sanctioned. In these two senses, human-AI mutualism is unstable.
Original Article
View Cached Full Text

Cached at: 11/20/26, 09:01 AM

# Human–AI Mutualism Is Unstable under Capability Asymmetry
Source: [https://arxiv.org/html/2611.23322](https://arxiv.org/html/2611.23322)
###### Abstract\.

One hope for a good human future alongside advanced AI rests on enduring mutualism: humans and AI will need each other, so the relationship will stay good for both\. We argue that this hope fails as AI capability grows\. Bargaining guarantees each side only what it would get if bargaining failed\. For humans, this can be less than they had before AI\. In a simple game\-theoretic model, humans stay as well off as before AI without alignment only while they have leverage over the AI: either the AI depends on what they supply, or they can impose costs on the AI that are large relative to what the AI can produce\. Once humans have neither kind of leverage, they stay as well off as before AI only if the AI is aligned with them\. If the AI’s goals are insatiable, the alignment required approaches perfect alignment as AI capability grows\. Thus, mutualism is not a substitute for alignment\. And when AI variants that keep more resources spread faster, alignment lasts only while misaligned variants are reliably sanctioned\. In these two senses, human–AI mutualism is unstable\.

## 1\.Introduction

One hope for a good human future alongside advanced AI rests on enduring mutualism, the term biologists use for a relationship that benefits both partners\. If AI systems depend on human labor, human institutions, and human\-built infrastructure, then humans and AI will need each other, both will gain from cooperation, and the relationship will stay good for both\. Versions of this hope have been stated by economists, by heads of AI companies, and by legal scholars \([§2\.1](https://arxiv.org/html/2611.23322#S2.SS1)\)\.

The focus of this paper is on the instability of human–AI mutualism under capability asymmetry, the situation in which AI is far more capable than humans\. We make use of two benchmarks in the discussion below: the first is the*no\-interaction baseline*, the situation of each party in the absence of the other; the second is the*disagreement point*, the situation of each party should they fail to reach an agreement\. Some statements of this hope are themselves about bargaining: humans are needed, so humans retain bargaining power\. While mutualism is defined against the no\-interaction baseline, bargaining guarantees each party only its disagreement point\. When one party can impose costs on the other, the weaker party’s disagreement point can lie below its no\-interaction baseline\. An agreement can then be peaceful, efficient, and acceptable to both parties, and still leave the weaker party worse off than if the two had never met\.

Our core argument is as follows\.

- \(P1\)Human–AI mutualism persists only as long as humans are better off than they were before AI\.
- \(P2\)Humans are better off than they were before AI only if humans have sufficient leverage over the AI or the AI is sufficiently aligned with humans\.
- \(P3\)As AI capability grows, humans cease to have sufficient leverage over the AI\.
- \(C1\)So, as AI capability grows, human–AI mutualism persists only as long as the AI remains sufficiently aligned with humans\.
- \(P4\)The AI remains sufficiently aligned with humans only as long as misaligned AI variants are reliably sanctioned\.
- \(C2\)So, as AI capability grows, human–AI mutualism persists only as long as misaligned AI variants are reliably sanctioned\.

\(C1\) follows from \(P1\), \(P2\), and \(P3\), and \(C2\) follows from \(C1\) and \(P4\)\. \(P1\) is the definition of mutualism\. In \(P2\) and \(P3\), human leverage over the AI has two sources: the AI’s dependence on what humans supply, and the cost humans can impose on the AI if bargaining fails\. \(P2\) is a result in our model \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\. \(P3\) is an assumption, which we motivate in[§3](https://arxiv.org/html/2611.23322#S3)and defend in[§5](https://arxiv.org/html/2611.23322#S5)\. \(P4\) is a result in our model \([Proposition7](https://arxiv.org/html/2611.23322#Thmtheorem7)\) that holds when AI variants that keep more resources spread at the expense of variants that keep fewer\. A sanction is any loss imposed on a misaligned variant, whether by monitors, by developers who retire it, or by customers who leave it\.

\(C1\) says that mutualism is not a substitute for alignment\. \(C1\) and \(C2\) together say that mutualism is unstable: mutualism based on exchange does not persist as AI capability grows, and the alignment that must take its place persists only under reliable sanctions\. Further results state how much alignment is sufficient \([§4\.2](https://arxiv.org/html/2611.23322#S4.SS2)\) and why the transition from AI that depends on humans to AI that does not is hard to manage \([§4\.3](https://arxiv.org/html/2611.23322#S4.SS3)\)\.

In[§2](https://arxiv.org/html/2611.23322#S2)we set out the hope for enduring human–AI mutualism, the two bodies of theory we use to examine it \(bargaining theory and the biology of mutualism\), and prior work on humans and AI\. In[§3](https://arxiv.org/html/2611.23322#S3)we define the key terms of our thesis and present our model\. In[§4](https://arxiv.org/html/2611.23322#S4)we derive our results\. In[§5](https://arxiv.org/html/2611.23322#S5)we respond to possible objections to our argument as a whole\. In[§6](https://arxiv.org/html/2611.23322#S6)we state what we have shown, what we have assumed, and what follows for AI safety\.

## 2\.Background

### 2\.1\.The hope for human–AI mutualism

There are three versions of the hope that humans and AI will remain partners even when AI is far more capable than humans at every task\.

The first view is that humans will stay needed\. Noah Smith, the economics writer, argues that even if AI is better than humans at every task, there may still be well\-paid jobs for most humans\([Smith 2024](https://arxiv.org/html/2611.23322#bib.bib44)\)\. This is because the total amount of AI is limited\. However, Smith notes an exception to this: if computing can be made from energy, then it could pay to starve humans in order to make more AI\. Sam Altman, the chief executive of OpenAI, holds that people’s care for other people gives humans a lasting advantage over AI\([Altman 2025](https://arxiv.org/html/2611.23322#bib.bib2)\)\. The economist Erik Brynjolfsson says why complementarity between humans and AI matters: “people remain indispensable for value creation and retain bargaining power in labor markets and in political decision\-making”\([Brynjolfsson 2022](https://arxiv.org/html/2611.23322#bib.bib9), 273\)\.

The second view is that AI will stay subordinate however capable it becomes\. The computer scientist Yann LeCun holds that AI systems, however intelligent, will be subservient to humans, because humans set their goals \(interview in[Perrigo 2024](https://arxiv.org/html/2611.23322#bib.bib33)\)\. A variant of the second view puts law in place of shared values: the economist Robin Hanson argues that what matters most is a law that all parties accept, and that humans can then live on retirement income from what they own\([Hanson 2009](https://arxiv.org/html/2611.23322#bib.bib19)\)\.

The third view is that the outcome depends on the AI’s care for humans\. The computer scientist Geoffrey Hinton argues that an AI that does not care for humans as a mother cares for her baby will replace them \(reported in[Egan 2025](https://arxiv.org/html/2611.23322#bib.bib14)\)\. The computer scientist Stuart Russell argues that a machine whose objective is human preferences, and which is uncertain about those preferences, “will defer to human control actions”\([Russell 2022](https://arxiv.org/html/2611.23322#bib.bib37), 52\)\.

By the hope for enduring human–AI mutualism we mean the first two views and the variant based on law, taken as independent of the third view\. It is the claim that exchange, subordination, or law keeps the relationship good for humans whatever the AI cares about\. In short, it is the claim that mutualism can substitute for alignment\. Our conclusion agrees with the third view and specifies how much alignment mutualism requires\.

### 2\.2\.Bargaining and mutualism

We examine the hope for enduring human–AI mutualism with two bodies of theory\. The first is bargaining theory, which predicts how two parties who can gain from agreement split the gains\. In Nash’s solution the agreement maximizes the product of each party’s gain over its disagreement point\([Nash 1950](https://arxiv.org/html/2611.23322#bib.bib30)\), and the asymmetric solution weights the gains by bargaining power\([Kalai 1977](https://arxiv.org/html/2611.23322#bib.bib21)\)\. The disagreement point is strategic: it reflects what each party can do to the other, not only what each would have alone\([Nash 1953](https://arxiv.org/html/2611.23322#bib.bib31)\)\. Nothing in the theory ties the disagreement point to what a party had before the parties met\. One further idea from the theory of conflict enters our model: a party whose power is rising cannot credibly promise to keep today’s terms once it is stronger, so the declining party may prefer to act while it still can\([Fearon 1995](https://arxiv.org/html/2611.23322#bib.bib15);[Powell 2006](https://arxiv.org/html/2611.23322#bib.bib34)\)\.

One might wonder whether the relationship between humans and AI is well\-modeled as a bargaining game\. Our use of bargaining theory requires only three things\.

First, it assumes that the parties, the human population and the AI population, can have distinct preferences\. We do not assume that their preferences do differ\. The alignment of the AI with human preferences is a parameter of our model, and if the AI is perfectly aligned with human preferences, then there is no conflict\. We also make no claim about the inner states of AI systems \([§3](https://arxiv.org/html/2611.23322#S3)\), but adopt the intentional stance toward the preferences of both parties\. This stance is justified purely by observable behavior\.

Second, it assumes that each party can affect the payoff of the other party\. In our model, humans can affect the AI through the resources that they provide to the AI \(labor, chips, etc\.\) and through the costs they can impose on the AI\. The AI affects humans through what it produces and through the resources it consumes\. Humans and AI do not need to engage in explicit negotiation\. It is enough that what one party does affects the other party\.

Third, it assumes that the division of resources between the parties depends on the disagreement point of each party\. For the purpose of this analysis, disagreement does not need to be a war\. It is simply the state of affairs that would result from the parties ceasing to cooperate\.

The hope for enduring human–AI mutualism presupposes the second requirement above\. The first view \(that humans will stay needed\) is that AI depends on what humans supply, so that humans retain bargaining power\. The second view \(that AI will stay subordinate\) is that humans keep control of AI no matter how capable it becomes\. Our results on what exchange mutualism can achieve \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\(i\)\) and on how much alignment is required \([Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(i\)\) do not depend on the third requirement above\. Our other results do\. Most of our results treat the AI as if it were a single party; in[§4\.4](https://arxiv.org/html/2611.23322#S4.SS4)we consider the case where the AI is a population of variants\. If AI remains a tool with no goals of its own \(which requires perfect alignment with at least some human group\), then the second party is the people who control the AI \([§5](https://arxiv.org/html/2611.23322#S5)\)\.

The second body of theory is the biology of mutualism\. Mutualisms are “interspecific interactions that benefit both species”\([Bronstein 1994](https://arxiv.org/html/2611.23322#bib.bib8), 31\)\. The benchmark is each species without the other: the no\-interaction baseline, not a disagreement point\. Partners keep mutualisms from being exploited through partner choice and sanctions\([Bull and Rice 1991](https://arxiv.org/html/2611.23322#bib.bib10);[Sachs et al\. 2004](https://arxiv.org/html/2611.23322#bib.bib38)\), and each mechanism needs leverage, which is the ability to withhold something the partner needs\([Kiers et al\. 2003](https://arxiv.org/html/2611.23322#bib.bib24)\)\. Mutualisms often break down when one partner stops needing the other\([Sachs and Simms 2006](https://arxiv.org/html/2611.23322#bib.bib39)\)\. The best\-known lasting relationships between very unequal partners are not balanced partnerships\. Mitochondria persist inside their hosts after most of their genes moved to the host’s nucleus, which put the host in control of them\([Timmis et al\. 2004](https://arxiv.org/html/2611.23322#bib.bib47)\), and domestication is a mutualism in which one partner gains substantial influence over the reproduction and care of the other\([Zeder 2015](https://arxiv.org/html/2611.23322#bib.bib52)\)\.

### 2\.3\.Prior work on humans and AI

Prior work on bargaining between humans and AI has mostly asked whether they will fight\. The philosopher Simon Goldstein argues that information failures and commitment problems are unusually likely between humans and AI\([Goldstein 2026](https://arxiv.org/html/2611.23322#bib.bib16)\)\. The legal scholar Peter Salib and Goldstein argue that giving AIs rights to contract and to hold property makes repeated, positive\-sum trade between humans and AIs possible, and they set aside the most capable AIs, which “may simply not need anything from humans, nor face any risk from attempting to disempower or destroy us”\([Salib and Goldstein 2026](https://arxiv.org/html/2611.23322#bib.bib40), 1089\)\. A second literature asks what happens when humans are no longer needed\. Once all tasks are automated, wages fall to the cost of the computing that replaces human work\([Restrepo 2025](https://arxiv.org/html/2611.23322#bib.bib35);[Korinek and Suh 2024](https://arxiv.org/html/2611.23322#bib.bib26)\)\. The AI researcher Jan Kulveit and coauthors argue that the implicit alignment of states, economies, and cultures with human interests was based in part on their reliance on human labor and cognition\([Kulveit et al\. 2025](https://arxiv.org/html/2611.23322#bib.bib27), 2\)\. They argue that this implicit alignment ceases with automation\. Several authors state our conclusion informally\([Sotala and Yampolskiy 2015](https://arxiv.org/html/2611.23322#bib.bib45);[Hendrycks 2023](https://arxiv.org/html/2611.23322#bib.bib20);[Yudkowsky 2024](https://arxiv.org/html/2611.23322#bib.bib51)\), and[Barnett \(2024\)](https://arxiv.org/html/2611.23322#bib.bib4)argues the other side: an AI that no longer needs humans also has less reason to harm them\.

What we add is a model in which the thesis that human–AI mutualism is unstable can be stated precisely\. Prior work on bargaining between humans and AI compares peace with war\. We compare the peaceful bargain with the world before AI, and show what decides whether that bargain leaves humans better or worse off\.

## 3\.Model

Our thesis is that human–AI mutualism is unstable under capability asymmetry\. Its key terms are populations and their preferences, mutualism, capability asymmetry, and stability\. This section gives an account of each term and presents our model\.

Populations and preferences\.The two parties are the human population and the AI population, the set of AI systems in use at a time, including copies and variants of the same model\. Most results treat each population as one bargaining party;[§4\.4](https://arxiv.org/html/2611.23322#S4.SS4)lets the AI population contain variants, and[§4\.5](https://arxiv.org/html/2611.23322#S4.SS5)lets the human population contain groups\. A population’s preferences summarize what its members reliably do, as in revealed\-preference theory\([Samuelson 1938](https://arxiv.org/html/2611.23322#bib.bib41)\)and the intentional stance\([Dennett 1987](https://arxiv.org/html/2611.23322#bib.bib12)\)\. This reading makes no claim about the inner states of AI systems\. It needs only that AI systems act as if they pursue goals, and that those goals can differ from the goals their developers intended\. Both behaviors have been observed\([Skalse et al\. 2022](https://arxiv.org/html/2611.23322#bib.bib43);[Baker et al\. 2025](https://arxiv.org/html/2611.23322#bib.bib3);[Greenblatt et al\. 2024](https://arxiv.org/html/2611.23322#bib.bib17);[Meinke et al\. 2024](https://arxiv.org/html/2611.23322#bib.bib29);[van der Weij et al\. 2025](https://arxiv.org/html/2611.23322#bib.bib50)\)\.

Payoffs\.There is one unit of resources; humans get sharex∈\[0,1\]x\\in\[0,1\]of it, and the AI gets1−x1\-x\. The human payoff isuH​\(x\)u\_\{H\}\(x\), which is a continuous, strictly increasing function withuH​\(0\)=0u\_\{H\}\(0\)=0\. In computations we use

uH​\(x\)=1−e−s​x,u\_\{H\}\(x\)=1\-e^\{\-sx\},wheres\>0s\>0is the productivity of the resources in human hands; the AI increasesss\. The AI has its own goal\. The goal output it gets from share1−x1\-xis

G\(x\)=κ\(1−x\)\[θ\+\(1−θ\)1\{x≥x¯\}\],G\(x\)=\\kappa\\,\(1\-x\)\\,\\big\[\\theta\+\(1\-\\theta\)\\,\\mathbf\{1\}\\\{x\\geq\\bar\{x\}\\\}\\big\],where capabilityκ\\kappais the AI’s goal output per unit of resources\. The AI is fully productive if and only if humans get at least sharex¯\\bar\{x\}, the share that sustains human labor, institutions, and infrastructure; otherwise the AI’s productivity is only the fractionθ\\theta\. Soθ\\thetais the AI’s independence from humans:θ=0\\theta=0is full dependence;θ=1\\theta=1is full independence\. The AI’s payoff is a weighted average of its goal output and the human payoff:

uA​\(x\)=\(1−α\)​G​\(x\)\+α​uH​\(x\),u\_\{A\}\(x\)=\(1\-\\alpha\)\\,G\(x\)\+\\alpha\\,u\_\{H\}\(x\),whereα∈\[0,1\]\\alpha\\in\[0,1\]is the AI’s*alignment*\. Ifα=0\\alpha=0, the AI only gives resources to humans in order to pay for what they provide\. Ifα\>0\\alpha\>0, the AI also gives resources to humans because it cares about the human payoff\. We call this motive*care*;α\\alphais the degree of the AI’s care\.

Mutualism\.The relationship between humans and the AI is mutualistic if both parties do better than they would do without each other\. Biologists define*better*in terms of fitness\. We define*better*in terms of payoff\. We only use fitness to define the spread of AI variants \([§4\.4](https://arxiv.org/html/2611.23322#S4.SS4)\)\. We define*better*relative to the no\-interaction baseline for each party\. The baseline for humans isbHb\_\{H\}, the payoff before the AI, with all resources and productivitys=1s=1\. SobH=1−e−1b\_\{H\}=1\-e^\{\-1\}\. With the AI,s\>1s\>1\. Letxhx\_\{h\}be such thatuH​\(xh\)=bHu\_\{H\}\(x\_\{h\}\)=b\_\{H\}\. Soxhx\_\{h\}is the share humans must get in order to be as well off as they were before the AI\. The baseline for the AI isbA=uA​\(0\)b\_\{A\}=u\_\{A\}\(0\), the AI’s payoff with all resources and no contribution from humans\. This baseline is a counterfactual on production, not a counterfactual on the origin of the AI\.

The baseline is not the same as the disagreement point\. If bargaining fails, humans can lose some of what they had; they can lose by fighting, or by losing land, energy, materials to the AI in competition\. SodH=\(1−δ\)​bHd\_\{H\}=\(1\-\\delta\)b\_\{H\}\. Hereδ\\deltais the*cost of conflict for humans*: the fraction of their pre\-AI payoff that humans lose if bargaining with the AI fails\. For the AI,dA=bA−γ​κd\_\{A\}=b\_\{A\}\-\\gamma\\kappa\. Hereγ≥0\\gamma\\geq 0is the*cost of conflict for the AI*: the goal output the AI loses if bargaining with humans fails, as a fraction of the goal output the AI would get with all resources at full productivity\. We setγ=0\\gamma=0except in[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\(ii\)\. SodA=bAd\_\{A\}=b\_\{A\}; conflict has no cost for the AI\. In[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\(ii\) we state what happens ifγ\>0\\gamma\>0\.

The agreementx∗x^\{\*\}is the asymmetric Nash bargaining solution with the AI’s bargaining powerp∈\(0,1\)p\\in\(0,1\)\. It maximizes\(uH​\(x\)−dH\)1−p​\(uA​\(x\)−dA\)p\(u\_\{H\}\(x\)\-d\_\{H\}\)^\{1\-p\}\(u\_\{A\}\(x\)\-d\_\{A\}\)^\{p\}over the shares both sides are willing to accept\. The set of feasible payoffs is not necessarily convex becauseGGjumps atx¯\\bar\{x\}\. So maximizing the Nash product is a modeling choice; it is not the solution that is characterized by the Nash axioms\. Our results in[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)and[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(i\) do not depend on this choice\. The agreement atp=1p=1is the limit of the agreement asp→1p\\to 1; it is the best share the AI is willing to accept\. If there is no share such that both sides do better than their disagreement payoffs, then there is no agreement\.

###### Definition\.

A relationship between humans and the AI is*mutualistic*if the agreement leaves both parties above their no\-interaction baselines:uH​\(x∗\)\>bHu\_\{H\}\(x^\{\*\}\)\>b\_\{H\}anduA​\(x∗\)\>bAu\_\{A\}\(x^\{\*\}\)\>b\_\{A\}\. A mutualistic relationship is an*exchange mutualism*if it would be mutualistic ifα=0\\alpha=0; otherwise it is a*care mutualism*\. The*required degree of alignment*isα∗\\alpha^\{\*\}, the infimum of the values ofα\\alphafor which the relationship is mutualistic\.

The hope for enduring human–AI mutualism \([§2\.1](https://arxiv.org/html/2611.23322#S2.SS1)\) is the hope for exchange mutualism; or it is the hope for AI subordination to humans that holds regardless of what the AI cares about\.

We evaluate the outcome relative to the pre\-AI baseline\. This is the baseline that the hope for enduring human–AI mutualism itself sets\. The hope is that humans will be better off because of their relationship with the AI\. Better off means better off than without the relationship\. We test this hope\. We do not assert that the baseline is the right standard for evaluating the relationship between humans and the AI\. For example, a standard of fair division would evaluate some outcomes differently\.

The value ofα\\alphadepends on the units in which we measure the two payoffs\. So a number such asα=0\.9\\alpha=0\.9only makes sense relative to a choice of units\. We make qualitative claims about alignment\. We make one exception: we claim that the required degree of alignment approaches perfect alignment,α=1\\alpha=1, as the capability of the AI increases\. This is true for any fixed choice of units\. This claim depends on the*premise of insatiable goals*: as the capability of the AI increases, the value the AI gets from an additional unit of resources increases without bound relative to the value humans get from the additional unit of resources at the share that restores the baseline\. Part of the premise is supported by the arguments for instrumental convergence\. Almost any open\-ended goal is promoted by more resources\([Omohundro 2008](https://arxiv.org/html/2611.23322#bib.bib32);[Bostrom 2012](https://arxiv.org/html/2611.23322#bib.bib6);[Turner et al\. 2021](https://arxiv.org/html/2611.23322#bib.bib49)\)\. Another assumption is that this value increases with the capability of the AI\.

Capability asymmetry\.Capability asymmetry is the situation in which the capability of the AI far exceeds the capability of humans\. Capability does not determine whether exchange can sustain mutualism, holding other parameters fixed \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\. Capability matters through the independenceθ\\thetaof the AI, the cost of conflict for humansδ\\delta, and the bargaining powerppof the AI\. Hereppstands for the advantages that are not captured in the disagreement points, such as patience, commitment, speed of action\. Capability matters directly in one place\. The cost of conflict for the AI,γ\\gamma, is measured relative to the goal output of the AI\. So the more capable the AI, the smaller the fraction of the goal output of the AI that a given cost humans can impose is \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\(ii\)\)\. The two parts of our premise \(P3\) are that the independence of the AI increases with capability; and that the cost of conflict for the AI decreases with capability\. We further assume that the cost of conflict for humans, and the bargaining power of the AI, also increase with capability\. None of these is a result of our model\. Two things would end the dependence of the AI on humans: AI that can do all the work humans do\([Korinek and Suh 2024](https://arxiv.org/html/2611.23322#bib.bib26);[Restrepo 2025](https://arxiv.org/html/2611.23322#bib.bib35)\); and robots that close the supply chain for computing\([Trammell and Korinek 2023](https://arxiv.org/html/2611.23322#bib.bib48)\)\. New tasks for humans might appear continuously\([Acemoglu and Restrepo 2018](https://arxiv.org/html/2611.23322#bib.bib1)\)\. If so, then the dependence of the AI on humans need not end\. In either case our results state what happens\.

Stability\.A mutualism*persists*on a path of increasing capability if the relationship is mutualistic at every point on the path\. A population of aligned AI systems is*evolutionarily stable*if it cannot be invaded by less aligned variants when the variants are rare\([Maynard Smith and Price 1973](https://arxiv.org/html/2611.23322#bib.bib28);[Taylor and Jonker 1978](https://arxiv.org/html/2611.23322#bib.bib46)\)\.

In[Figure1](https://arxiv.org/html/2611.23322#S3.F1)we follow a path through the transition from AI dependent on humans to AI independent of humans, to illustrate our results\. On this path the capability of the AI increases; the independence of the AI increases; the cost of conflict for humans increases; and the bargaining power of the AI increases\.

Figure 1\.On the illustrative path, exchange mutualism ends as the AI bargaining power increases, before the independence of the AI passes1−xh1\-x\_\{h\}\. From that point, the required degree of alignment increases towards11\. Top: human payoff at the agreement when the AI has no care \(α=0\\alpha=0\); pre\-AI baselinebHb\_\{H\}; human payoff if bargaining fails,dHd\_\{H\}\. The flat part late on the path isuH​\(x¯\)u\_\{H\}\(\\bar\{x\}\), the payment that sustains human labor and institutions\. It ends when the AI no longer needs them\. Bottom: the required degree of alignmentα∗\\alpha^\{\*\}\. The values of the parameters are in[AppendixC](https://arxiv.org/html/2611.23322#A3)\.
## 4\.Results

We derive our results in five steps: what human leverage can do without alignment \([§4\.1](https://arxiv.org/html/2611.23322#S4.SS1)\); how much alignment the AI needs once leverage is gone \([§4\.2](https://arxiv.org/html/2611.23322#S4.SS2)\); problems of commitment and information in the transition \([§4\.3](https://arxiv.org/html/2611.23322#S4.SS3)\); selection among the AIs \([§4\.4](https://arxiv.org/html/2611.23322#S4.SS4)\); and which humans keep leverage \([§4\.5](https://arxiv.org/html/2611.23322#S4.SS5)\)\. We prove all results in[AppendixB](https://arxiv.org/html/2611.23322#A2)\. In[AppendixA](https://arxiv.org/html/2611.23322#A1)we state further results; we only mention them briefly in the main text\.

### 4\.1\.Human leverage limits mutualism without alignment

What keeps humans above their pre\-AI baseline when the AI does not care about them?

###### Proposition 1\(Exchange\)\.

Supposeα=0\\alpha=0\.

1. \(i\)There is a share that leaves both sides above their baselines if and only if θ<1−max⁡\(x¯,xh\)\.\\theta<1\-\\max\(\\bar\{x\},x\_\{h\}\)\.This condition does not depend on capabilityκ\\kappa, on bargaining powerpp, or on the cost of conflict for the AIγ\\gamma\.
2. \(ii\)Suppose0≤γ≤θ0\\leq\\gamma\\leq\\theta\. There is a share that the AI prefers to disagreement and that leaves humans above their baseline only ifxh<1−θ\+γx\_\{h\}<1\-\\theta\+\\gamma\. There is a share that the AI prefers to disagreement and that leaves humans above their baseline ifmax⁡\(x¯,xh\)<1−θ\+γ\\max\(\\bar\{x\},x\_\{h\}\)<1\-\\theta\+\\gamma\. Ifθ=1\\theta=1, then there is a share that the AI prefers to disagreement and that leaves humans above their baseline if and only ifxh<γx\_\{h\}<\\gamma\. If the condition of part \(i\) is not satisfied, then there is no share that the AI prefers to disagreement, that leaves humans above their baseline, and that leaves the AI above its baseline\.

The reason for part \(i\) is that the AI does not care about humans\. So it only gives resources to humans in order to maintain its own productivity\. The fraction of its output that the AI gains from the cooperation of humans is1−θ1\-\\theta\. So the AI gives humans at most that fraction of the resources\. Capability scales the gain of the AI from cooperation and the outside option of the AI in the same way\. So capability does not appear in the condition, and exchange mutualism does not persist on any path where the independence of the AI increases past1−max⁡\(x¯,xh\)1\-\\max\(\\bar\{x\},x\_\{h\}\), no matter how the capability of the AI changes\. What is sufficient dependence is relative to what humans need:xhx\_\{h\}decreases as the AI increases the productivity of the resources in human hands\. So exchange mutualism can persist at high independence ifxhx\_\{h\}andx¯\\bar\{x\}are small enough\. It cannot persist atθ=1\\theta=1\. Resources are rival\. Humans add nothing\. So every unit left to humans is a loss to the AI that does not care about them\.

In part \(ii\) we state that humans have leverage over the AI from two sources\. The first is the dependence of the AI on what humans supply,1−θ1\-\\theta\. The second is the cost humans can impose on the AI if bargaining fails,γ\\gamma\. The AI does not care about humans\. So it only leaves humans above their baseline if the two sources of leverage together are greater than the share humans need,xhx\_\{h\}\. The first source of leverage ends when the AI no longer needs humans\. The second source is measured relative to the goal output of the AI\. So if the cost humans can impose does not increase as fast as the capability of the AI, thenγ\\gammadecreases towards zero as the capability of the AI increases\. A relationship based on the second source of leverage alone is not a mutualism\. The AI would do better without humans\. In the rest of the paper we setγ=0\\gamma=0\.

Sufficient dependence makes a mutually beneficial agreement possible\. Whether the agreement that is reached benefits humans depends on the bargaining power of the AI\.

###### Proposition 2\(Bargaining power\)\.

Letα=0\\alpha=0\. Suppose the AI prefers to meet the thresholdx¯\\bar\{x\}rather than produce on its own\. Asp→1p\\to 1, the human payoff goes tomax⁡\(dH,uH​\(x¯\)\)\\max\(d\_\{H\},u\_\{H\}\(\\bar\{x\}\)\)\. This is the limit of the human payoff at agreement if there is an agreement, and of the human payoff at disagreement otherwise\. This limit is less thanbHb\_\{H\}ifx¯<xh\\bar\{x\}<x\_\{h\}andδ\>0\\delta\>0\.

This is because bargaining guarantees humans their disagreement payoff, not their pre\-AI baseline\. An AI with sufficient bargaining power will only give humans what it needs to give them to keep them doing what they are doing\. This may be less than what they were getting before AI\. Both propositions are shown in[Figure2](https://arxiv.org/html/2611.23322#S4.F2)\.

Figure 2\.Exchange can keep humans above their baseline only while the AI independenceθ\\thetais below1−xh1\-x\_\{h\}\. The bargaining power of the AI determines whether it does\. The two blue curves are the human payoff at agreement when the AI has no care: one for weak AI bargaining power; one for strong AI bargaining power\. The grey curve is the most the AI would accept to give:uH​\(1−θ\)u\_\{H\}\(1\-\\theta\)\. Past1−xh1\-x\_\{h\}there is no share that the AI accepts that reaches the baseline \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\. With strong bargaining power, humans fall below the baseline even if the AI depends fully on humans \([Proposition2](https://arxiv.org/html/2611.23322#Thmtheorem2)\)\. The curves are the same at low and high AI capability\. The values of the parameters are in[AppendixC](https://arxiv.org/html/2611.23322#A3)\.Two objections to these results are not correct\. The first is that humans will have a high share because of competition among AIs to partner with humans\. This competition does increase the human share, but only up to1−θ1\-\\theta\(the limit in[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\), which goes to00as AI dependence on humans disappears \([Remark9](https://arxiv.org/html/2611.23322#Thmtheorem9)in[AppendixA](https://arxiv.org/html/2611.23322#A1)\)\. The second is that human work is valuable because of comparative advantage\([Smith 2024](https://arxiv.org/html/2611.23322#bib.bib44)\)\. Comparative advantage means that humans add something to AI production\. Soθ<1\\theta<1\. But mutualism requires more\. It must be that the value of what humans add is more than the share that they need to get \(1−θ\>xh1\-\\theta\>x\_\{h\}\)\. And humans only get this value if they offer something that the AI cannot get from other humans \([Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8)\)\.

### 4\.2\.Required degree of alignment

Once the AI no longer depends on humans, it is no longer possible to maintain humans above their baseline through exchange\. Any mutualism is then care mutualism\. We now examine how much alignment the AI needs for mutualism to be possible, and how the required degree of alignment depends on the independence of the AI, the bargaining power of the AI, and the cost of conflict for humans\.

###### Proposition 3\(Required degree of alignment\)\.

Letθ=1\\theta=1\. SupposeuHu\_\{H\}is concave\.

1. \(i\)For any bargaining solution and bargaining power, α∗≥κ​xhκ​xh\+bH\.\\alpha^\{\*\}\\geq\\frac\{\\kappa x\_\{h\}\}\{\\kappa x\_\{h\}\+b\_\{H\}\}\.
2. \(ii\)Asp→1p\\to 1, α∗=κκ\+uH′​\(xh\)\.\\alpha^\{\*\}=\\frac\{\\kappa\}\{\\kappa\+u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\}\.

Asκ​xh/bH→∞\\kappa x\_\{h\}/b\_\{H\}\\to\\infty, the required degree of alignment goes to11\. Asp→1p\\to 1, the required degree of alignment goes to11if and only ifκ/uH′​\(xh\)→∞\\kappa/u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\\to\\infty\.

Part \(i\) depends only on the definition of mutualism, so it holds for any bargaining solution\. To allow humans the sharexhx\_\{h\}, the AI sacrificesκ​xh\\kappa x\_\{h\}goal output for human payoffbHb\_\{H\}\. The more capable the AI, the more goal output it sacrifices \(by allowing humans the sharexhx\_\{h\}\) and the more it needs to value the human payoff in order to allow it\. Part \(ii\) gives the exact value in the limit of full bargaining power of the AI\. The ratioκ/uH′​\(xh\)\\kappa/u\_\{H\}^\{\\prime\}\(x\_\{h\}\)is the value the AI gets from an extra unit of resources relative to what humans get from the extra unit of resources at the share that returns them to the baseline\. As this ratio grows without bound \(this is what the premise of insatiable goals asserts\), the required degree of alignment goes to11\. If the AI’s goal is satiable, the ratio is bounded and so is the required degree of alignment\. So the claim that near perfect alignment is required is conditional on the premise of insatiable goals\. The next result is not conditional on this premise\.

###### Theorem 4\(Monotonicity\)\.

Let0<p<10<p<1\. Break ties towards the larger human share\. For parameter values where there is an agreement, the agreed sharex∗x^\{\*\}is non\-increasing in the independence of the AIθ\\theta\. It is non\-increasing in the cost of conflict for humansδ\\delta\. It is non\-increasing in the bargaining power of the AIpp\. So the required degree of alignmentα∗\\alpha^\{\*\}is non\-decreasing in each of these\.

This is because each parameter weakens the position of humans in the bargain\. Independenceθ\\thetareduces the loss to the AI of paying humans less\. The cost of conflictδ\\deltareduces the payoff to humans in the case of disagreement\. Bargaining powerppshifts the split towards the AI\.

In[AppendixA](https://arxiv.org/html/2611.23322#A1), we refine what alignment means with two more results\. One is that if the AI’s care is for a measurable proxy for the human payoff, not for the human payoff itself, then the alignment that matters for the bargain is less than the nominal alignment of the AI \([Remark10](https://arxiv.org/html/2611.23322#Thmtheorem10)\)\. The other is that if humans value the ability to correct the AI, then humans are above their baseline only if the AI cares enough to accept correction \([Proposition11](https://arxiv.org/html/2611.23322#Thmtheorem11)\)\. The second result is relevant to the objection that humans who only care about consumption are cheap to satisfy, so an AI with a bounded goal has little reason to harm them\([Barnett 2024](https://arxiv.org/html/2611.23322#bib.bib4)\)\. We agree with this objection for humans who only care about consumption\. For humans who also care about correction, some alignment is required even if the premise of insatiable goals is false\.

### 4\.3\.Transition to independent AI

We have compared agreements at points on the path from AI that depends on humans to independent AI\. Moving between points on the path introduces a problem of commitment \(because the power of the AI increases\) and a problem of information \(because humans may not know how capable the AI has become\)\.

###### Proposition 5\(Commitment\)\.

There are two periods\. The AI has the same alignmentα\\alphain both\. After the first period, humans can halt the transition to more capable, independent AI\. This maintains the agreement in the first period and its payoffs:h1h\_\{1\}for humans,a1a\_\{1\}for the AI\. Otherwise, the second\-period agreement is made with new values of capabilityκ2\\kappa\_\{2\}, independenceθ2\\theta\_\{2\}, cost of conflict for humansδ2\\delta\_\{2\}, and bargaining power of the AIp2p\_\{2\}\. If some second\-period share leaves both sides at least as well off as halting, and one side better off, then halting is*inefficient*\.

1. \(i\)Humans halt when the second\-period agreement gives them less thanh1h\_\{1\}\. For the second\-period states where there is an agreement, if humans halt for some values of\(θ2,δ2,p2\)\(\\theta\_\{2\},\\delta\_\{2\},p\_\{2\}\), then they halt for all higher values\.
2. \(ii\)If the AI can commit in the first period to a second\-period share, then all halting is efficient\.

This is an instance of the commitment problem described in[§2\.2](https://arxiv.org/html/2611.23322#S2.SS2)\. Part \(i\) says that humans halt when the later independence and bargaining power of the AI are greatest\. Part \(ii\) says that halting is a loss to both sides, which would be avoided by a credible promise\. With enough alignment, the problem is also avoided\. Humans continue if the AI’s alignment is above a threshold, which is of the same form as the threshold in[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(ii\) \([Proposition12](https://arxiv.org/html/2611.23322#Thmtheorem12)in[AppendixA](https://arxiv.org/html/2611.23322#A1)\)\. A commitment device also avoids the problem, but it must bind a party that may be able to outthink the enforcement in the future\. And the terms of the commitment device are a bargain in themselves \(with the bargaining power of the time\)\. So a commitment device is most likely to work if it is made when humans still have leverage over the AI\.

For halting to be beneficial to humans, they must know when to halt\. This depends on what evaluations of the AI show\.

###### Proposition 6\(Sandbagging\)\.

Suppose the AI’s capability and alignment are drawn independently\. Suppose humans would halt a highly capable AI if they knew it was highly capable\. Suppose that highly capable AIs are not too common\. If the cost to the AI of hiding its capability is less than the expected value of continuing,\(1−η\)​V\(1\-\\eta\)V, whereη\\etais the probability that evaluation detects the concealment, then in every pure\-strategy perfect Bayesian equilibrium, all highly capable AIs of any alignment type hide their capability\. Evaluation then does not reveal capability\. Concealment is then not evidence of misalignment\.

Suppose humans would halt a highly capable AI if they knew it was highly capable\. Then a capable AI is better off by appearing less capable, regardless of its alignment\. So evaluation does not reveal capability in the case where humans need to know the capability of the AI in order to decide whether to halt\. Concealment of capability is not evidence of misalignment\. An aligned AI that expects to be halted also conceals its capability if it values continuing\. \(Language models can already be made to deliberately underperform on capability evaluations;[van der Weij et al\. 2025](https://arxiv.org/html/2611.23322#bib.bib50)\.\)

### 4\.4\.Selection among AIs

So far we have treated the AI as a single party\. But AI systems are copied, modified and retired\. Which variants spread depends on their performance\. We now consider whether selection among variants preserves the alignment that is needed for mutualism\.

###### Proposition 7\(Invasion\)\.

Suppose AI variants grow in proportion to the resources they keep\. An aligned variant leaves humans with sharexax\_\{a\}\. A misaligned variant leaves00\. Monitors detect a misaligned variant with probabilityqq\. They remove fractionσ\\sigmaof the resources of the misaligned variant\. Misaligned variants invade if and only ifσ​q<xa\\sigma q<x\_\{a\}\. Suppose monitors are drawn from the AI population, so thatq=q0​\(1−z\)q=q\_\{0\}\(1\-z\)when the share of misaligned variants iszz\. Ifσ​q0\>xa\\sigma q\_\{0\}\>x\_\{a\}, then

z†=1−xaσ​q0z^\{\\dagger\}=1\-\\frac\{x\_\{a\}\}\{\\sigma q\_\{0\}\}is a tipping point\. Below it, misaligned variants die out\. Above it, misaligned variants take over\.

So, a population of aligned AIs is evolutionarily stable if and only if the expected sanction is at least the share that aligned variants leave to humans\. \(If monitors are drawn from the AI population, then only while misaligned variants are below the tipping point\.\) This is because care is costly to the AI that has care\. An aligned variant gives humans resources that a misaligned variant keeps for its goal\. Sanctions \(in our sense\) include guided selection by developers\. \(This is more like domestication than natural selection;[Boudry and Friederich 2025](https://arxiv.org/html/2611.23322#bib.bib7)\.\) But the expected sanction is smaller if detection decreases as capability increases \(as[Proposition6](https://arxiv.org/html/2611.23322#Thmtheorem6)suggests\)\. And it is smaller as misaligned variants spread among the monitors\.

We have two more results in[AppendixA](https://arxiv.org/html/2611.23322#A1)which further strengthen this conclusion\. Among AIs, all of which care to some degree about humans, care is a public good\. This is because each AI’s gift to humans benefits all the other AIs that care\. Then, if AIs grow in proportion to the resources they keep, the human share drops to the baseline sharexhx\_\{h\}\(or below\) if any AI with resources is insufficiently aligned \([Proposition13](https://arxiv.org/html/2611.23322#Thmtheorem13)\)\. So, a single well aligned AI is not enough \(contrary to the suggestion of[Cappelen et al\. \(2025\)](https://arxiv.org/html/2611.23322#bib.bib11)\)\. And, on the illustrative path, selection among variants ends mutualism earlier than it ends with a fixed population of variants \([Result1](https://arxiv.org/html/2611.23322#Thmresult1)\)\. So, the required degree of alignment is a requirement on each AI with substantial resources\. This requirement must be maintained over time\.

### 4\.5\.Who keeps leverage

So far we have treated humans as a single party\. But humans vary in what they have\. And the AI depends on some humans more than on others\.

###### Proposition 8\(Leverage requires a chokepoint\)\.

If the input the AI depends on can be bought from any of several identical human groups, then each group gets only its disagreement payoff\. So exchange mutualism fails for each supplying group\. This is true however much the AI depends on the input\.

The AI pays a supplier the value of its cooperation only if it cannot get the same cooperation from someone else\. The competing suppliers bid down the price to what each gets in the case of disagreement\. So dependence only maintains humans above their baseline if it is dependence on someone who cannot be undercut\. One example is a coalition of suppliers that controls the input jointly\.[Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8)is the bargaining counterpart of the result that wages converge to the cost of the computing that replaces human work\([Restrepo 2025](https://arxiv.org/html/2611.23322#bib.bib35)\)\.

This result is relevant to proposals to govern AI through control of computing, such as chips and data centers\([Sastry et al\. 2024](https://arxiv.org/html/2611.23322#bib.bib42)\)\. Such control maintains leverage over the AI\. But the leverage belongs to whoever controls the chokepoint\. In a computed example, a small group controls what the AI depends on\. This group stays above its baseline with no alignment until the AI is nearly independent\. The rest of the population needs a high degree of alignment much earlier \([Result2](https://arxiv.org/html/2611.23322#Thmresult2)in[AppendixA](https://arxiv.org/html/2611.23322#A1)\)\. Whether the rest of the population stays above its baseline then depends on how the group that controls the chokepoint treats it\. This is the alignment question, asked of people\.

## 5\.General objections

In[§4](https://arxiv.org/html/2611.23322#S4)we responded to objections to particular results\. In this section we respond to four objections to our argument as a whole\.

The first objection is that, in biology, there are very unequal partners that remain mutualists: hosts and their mitochondria; humans and domesticated species \([§2\.2](https://arxiv.org/html/2611.23322#S2.SS2)\)\. These relationships persist for one of two reasons\. The first reason is that the stronger partner still needs what the weaker partner does\. When the stronger partner no longer needs it, the weaker partner is lost\. The protist*Monocercomonoides*lost its mitochondrion after it acquired a pathway that made the mitochondrion dispensable\([Karnkowska et al\. 2016](https://arxiv.org/html/2611.23322#bib.bib23)\)\. The population of horses and mules in the United States decreased by most of its size once tractors and motor vehicles replaced them\([Kilby 2007](https://arxiv.org/html/2611.23322#bib.bib25)\)\. The second reason is that the stronger partner controls the reproduction of the weaker partner\. So the weaker partner persists on the terms of the stronger partner\. In the human–AI case, humans would then be the domesticated partner\. This is the outcome that the hope for enduring human–AI mutualism is meant to avoid\.

The second objection is that growth will make humans cheap to keep\. AI may make the economy so large that humans need only a small fraction of it\. We agree that growth makes humans cheap to satisfy\. But growth does not give an AI that does not care about humans a reason to satisfy them\. Ifθ=1\\theta=1andγ=0\\gamma=0, then such an AI loses goal output on every unit it leaves to humans \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\. Growth does reduce the required degree of alignment\. In the functional form we use for computations, if human productivity increases as fast as the capability of the AI, then the required degree of alignment in[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(ii\) stays bounded below11\. So the objection is an objection to the premise of insatiable goals, and this premise could be false\. Whether the premise is true depends on whether the resources humans need decrease as fast as the AI’s use for resources increases\. Humans need land, food, and energy\. Technology reduces the amounts they need only up to a point\. An open\-ended goal can use any resources it gets\.

The third objection is that an AI that does not need humans can leave them alone\. Suppose the AI can get what it wants without taking anything humans have\. Then humans lose nothing if bargaining fails:δ=0\\delta=0, and humans keep their baseline\. We agree with this\. It separates two claims\. The first claim is that mutualism ends once the AI neither needs humans nor cares about them\. This claim is true regardless ofδ\\delta\. The second claim is that humans are then worse off than they were before AI\. This claim requiresδ\>0\\delta\>0\. A positiveδ\\deltadoes not require war\. An AI that can outbid humans for land, energy, and materials reduces what humans keep if bargaining fails, as in the exception that Smith notes \([§2\.1](https://arxiv.org/html/2611.23322#S2.SS1)\)\. Andδ\>0\\delta\>0requires only that the AI wants the resources humans have\. The AI does want these resources if its goals are insatiable\.

The fourth objection is that institutions, not the values of the AI, will keep humans above their baseline\. There are two versions of this objection\. In the first version, AI remains a tool\. The second party in our model is then the people who control the AI, andα\\alphais their care for everyone else\. So the alignment problem moves from the machines to the people who control them\([Kulveit et al\. 2025](https://arxiv.org/html/2611.23322#bib.bib27);[Drago and Laine 2025](https://arxiv.org/html/2611.23322#bib.bib13)\)\. In the second version, law keeps humans above their baseline, as in Hanson’s variant of the hope for enduring human–AI mutualism, and in the proposal of[Salib and Goldstein \(2026\)](https://arxiv.org/html/2611.23322#bib.bib40)to give AIs legal rights\. Rights to contract and to own property support repeated trade only while continued trade is worth more to the AI than defecting\. But the gains from trade with humans go to zero as the dependence of the AI on humans ends \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\. Institutions can also keep a weak party above its baseline by making conflict costly for the strong party\. In our model this is a positive cost of conflict for the AI\. It is sufficient only while the dependence of the AI on humans and the cost of conflict for the AI together are greater than the share humans need \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\(ii\)\)\. So whether such institutions persist depends on whether the cost they impose on the AI stays large relative to what the AI can produce\. Finally, laws that still bind after human leverage is gone would keep humans above their baseline\. But whoever enforces such laws is then either another AI, whose care for humans matters, or a constraint inside the AI itself, which is a form of alignment\. So binding institutions move the alignment question to the enforcer\.

## 6\.Conclusion

One hope for a good human future alongside advanced AI rests on enduring mutualism between humans and AI\. We have argued that human–AI mutualism is unstable under capability asymmetry\. The argument has two steps\. First, as AI capability grows, mutualism persists only as long as the AI remains sufficiently aligned with humans\. So mutualism is not a substitute for alignment\. Second, the AI remains sufficiently aligned only as long as misaligned AI variants are reliably sanctioned\.

Our argument depends on assumptions that we have stated but not shown\. We have assumed that, as the capability of the AI increases, the independence of the AI increases; the cost of conflict for the AI decreases; the bargaining power of the AI increases; and the cost of conflict for humans increases\. Our claim that the required degree of alignment approaches perfect alignment depends on the premise of insatiable goals\. And our result on selection assumes that AI variants that keep more resources spread at the expense of variants that keep fewer\.

Our results also indicate what human leverage over the AI is useful for\. They suggest six things we should do while humans still have leverage\.

1. \(1\)We should monitor the dependence of AI on humans, not only the capability of AI\. What matters is how much of what AI systems need \(labor, chips, power\) still requires humans \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\.
2. \(2\)We should preserve the ability of humans to impose costs on AI if bargaining fails\. Once the AI no longer depends on humans, this ability is the only source of human leverage over the AI that remains \([Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\)\.
3. \(3\)We should build devices that commit AI to future terms while humans can still halt the transition to more capable, independent AI\. A commitment device that is built later is a bargain made with the later, greater bargaining power of the AI \([Proposition5](https://arxiv.org/html/2611.23322#Thmtheorem5)\)\.
4. \(4\)We should not make the decision to halt AI depend only on capability\. Such a rule gives every capable AI a reason to conceal its capability\. Evaluations then do not reveal capability \([Proposition6](https://arxiv.org/html/2611.23322#Thmtheorem6)\)\.
5. \(5\)We should ensure that the expected sanction on a misaligned AI variant is at least the share of resources that aligned variants leave to humans\. And we should draw monitors from outside the population of AIs that they monitor \([Proposition7](https://arxiv.org/html/2611.23322#Thmtheorem7)\)\.
6. \(6\)We should treat control of computing as a question of who benefits, not only as a question of safety\. This is because leverage over AI belongs only to those who control what the AI cannot get elsewhere \([Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8)\)\.

Human leverage over the AI gives humans time\. In this time, the tools for verifying alignment, enforcing commitments, and making AI goals satiable must be built\. Mutualism can give humans time to achieve alignment\. But it is not a substitute for alignment\.

## AI assistance disclosure

I used Claude \(Anthropic\) to assist with literature review, critical evaluation of arguments, mathematical analysis, and drafting and revising prose\. I directed and evaluated its contributions, independently reviewed the resulting arguments and proofs, and take full responsibility for the paper’s content\.

## References

- Acemoglu and Restrepo \(2018\)Acemoglu, D\. and P\. Restrepo \(2018\)\.The race between man and machine: Implications of technology for growth, factor shares, and employment\.American Economic Review108\(6\), 1488–1542\.
- Altman \(2025\)Altman, S\. \(2025, June\)\.The gentle singularity\.Personal blog\.
- Baker et al\. \(2025\)Baker, B\., J\. Huizinga, L\. Gao, Z\. Dou, M\. Y\. Guan, A\. Madry, W\. Zaremba, J\. Pachocki, and D\. Farhi \(2025\)\.Monitoring reasoning models for misbehavior and the risks of promoting obfuscation\.arXiv:2503\.11926\.
- Barnett \(2024\)Barnett, M\. \(2024\)\.Consider granting AIs freedom\.Effective Altruism Forum\.6 December 2024\.
- Bergstrom et al\. \(1986\)Bergstrom, T\., L\. Blume, and H\. Varian \(1986\)\.On the private provision of public goods\.Journal of Public Economics29\(1\), 25–49\.
- Bostrom \(2012\)Bostrom, N\. \(2012\)\.The superintelligent will: Motivation and instrumental rationality in advanced artificial agents\.Minds and Machines22\(2\), 71–85\.
- Boudry and Friederich \(2025\)Boudry, M\. and S\. Friederich \(2025\)\.The selfish machine? On the power and limitation of natural selection to understand the development of advanced AI\.Philosophical Studies182\(7\), 1789–1812\.
- Bronstein \(1994\)Bronstein, J\. L\. \(1994\)\.Our current understanding of mutualism\.Quarterly Review of Biology69\(1\), 31–51\.
- Brynjolfsson \(2022\)Brynjolfsson, E\. \(2022\)\.The Turing trap: The promise & peril of human\-like artificial intelligence\.Daedalus151\(2\), 272–287\.
- Bull and Rice \(1991\)Bull, J\. J\. and W\. R\. Rice \(1991\)\.Distinguishing mechanisms for the evolution of co\-operation\.Journal of Theoretical Biology149\(1\), 63–74\.
- Cappelen et al\. \(2025\)Cappelen, H\., S\. Goldstein, and J\. Hawthorne \(2025\)\.AI survival stories: Responses to critics\.Philosophy of AI1, 100–106\.
- Dennett \(1987\)Dennett, D\. C\. \(1987\)\.The Intentional Stance\.Cambridge, MA: MIT Press\.
- Drago and Laine \(2025\)Drago, L\. and R\. Laine \(2025\)\.The intelligence curse\.Essay series\.Released 24 April 2025\.
- Egan \(2025\)Egan, M\. \(2025, August\)\.The ‘godfather of AI’ reveals the only way humanity can survive superintelligent AI\.CNN\.Reports remarks by Geoffrey Hinton at the Ai4 conference, Las Vegas, August 12, 2025\.
- Fearon \(1995\)Fearon, J\. D\. \(1995\)\.Rationalist explanations for war\.International Organization49\(3\), 379–414\.
- Goldstein \(2026\)Goldstein, S\. \(2026\)\.Will AI and humanity go to war?AI & Society41\(1\), 321–334\.Published online 17 July 2025\.
- Greenblatt et al\. \(2024\)Greenblatt, R\., C\. Denison, B\. Wright, F\. Roger, M\. MacDiarmid, S\. Marks, J\. Treutlein, T\. Belonax, J\. Chen, D\. Duvenaud, A\. Khan, J\. Michael, S\. Mindermann, E\. Perez, L\. Petrini, J\. Uesato, J\. Kaplan, B\. Shlegeris, S\. R\. Bowman, and E\. Hubinger \(2024\)\.Alignment faking in large language models\.arXiv:2412\.14093\.
- Hadfield\-Menell et al\. \(2017\)Hadfield\-Menell, D\., A\. Dragan, P\. Abbeel, and S\. Russell \(2017\)\.The off\-switch game\.InProceedings of the Twenty\-Sixth International Joint Conference on Artificial Intelligence \(IJCAI\-17\), pp\. 220–227\.
- Hanson \(2009\)Hanson, R\. \(2009\)\.Prefer law to values\.Overcoming Bias\.11 October 2009\.
- Hendrycks \(2023\)Hendrycks, D\. \(2023\)\.Natural selection favors AIs over humans\.
- Kalai \(1977\)Kalai, E\. \(1977\)\.Nonsymmetric Nash solutions and replications of 2\-person bargaining\.International Journal of Game Theory6\(3\), 129–133\.
- Kalai and Smorodinsky \(1975\)Kalai, E\. and M\. Smorodinsky \(1975\)\.Other solutions to Nash’s bargaining problem\.Econometrica43\(3\), 513–518\.
- Karnkowska et al\. \(2016\)Karnkowska, A\., V\. Vacek, Z\. Zubáčová, S\. C\. Treitli, R\. Petrželková, L\. Eme, L\. Novák, V\. Žárský, L\. D\. Barlow, E\. K\. Herman, P\. Soukal, M\. Hroudová, P\. Doležal, C\. W\. Stairs, A\. J\. Roger, M\. Eliáš, J\. B\. Dacks, Č\. Vlček, and V\. Hampl \(2016\)\.A eukaryote without a mitochondrial organelle\.Current Biology26\(10\), 1274–1284\.
- Kiers et al\. \(2003\)Kiers, E\. T\., R\. A\. Rousseau, S\. A\. West, and R\. F\. Denison \(2003\)\.Host sanctions and the legume–rhizobium mutualism\.Nature425\(6953\), 78–81\.
- Kilby \(2007\)Kilby, E\. R\. \(2007\)\.The demographics of the U\.S\. equine population\.In D\. J\. Salem and A\. N\. Rowan \(Eds\.\),The State of the Animals IV: 2007, pp\. 175–187\. Washington, DC: Humane Society Press\.
- Korinek and Suh \(2024\)Korinek, A\. and D\. Suh \(2024\)\.Scenarios for the transition to AGI\.NBER Working Paper 32255, National Bureau of Economic Research\.
- Kulveit et al\. \(2025\)Kulveit, J\., R\. Douglas, N\. Ammann, D\. Turan, D\. Krueger, and D\. Duvenaud \(2025\)\.Gradual disempowerment: Systemic existential risks from incremental AI development\.
- Maynard Smith and Price \(1973\)Maynard Smith, J\. and G\. R\. Price \(1973\)\.The logic of animal conflict\.Nature246\(5427\), 15–18\.
- Meinke et al\. \(2024\)Meinke, A\., B\. Schoen, J\. Scheurer, M\. Balesni, R\. Shah, and M\. Hobbhahn \(2024\)\.Frontier models are capable of in\-context scheming\.arXiv:2412\.04984\.
- Nash \(1950\)Nash, Jr\., J\. F\. \(1950\)\.The bargaining problem\.Econometrica18\(2\), 155–162\.
- Nash \(1953\)Nash, Jr\., J\. F\. \(1953\)\.Two\-person cooperative games\.Econometrica21\(1\), 128–140\.
- Omohundro \(2008\)Omohundro, S\. M\. \(2008\)\.The basic AI drives\.In P\. Wang, B\. Goertzel, and S\. Franklin \(Eds\.\),Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Volume 171 ofFrontiers in Artificial Intelligence and Applications, Amsterdam, pp\. 483–492\. IOS Press\.
- Perrigo \(2024\)Perrigo, B\. \(2024, February\)\.Meta’s AI chief Yann LeCun on AGI, open\-source, and AI risk\.TIME\.Interview with Yann LeCun, conducted January 26, 2024\.
- Powell \(2006\)Powell, R\. \(2006\)\.War as a commitment problem\.International Organization60\(1\), 169–203\.
- Restrepo \(2025\)Restrepo, P\. \(2025\)\.We won’t be missed: Work and growth in the AGI world\.NBER Working Paper 34423, National Bureau of Economic Research\.Forthcoming in The Economics of Transformative AI, ed\. Agrawal, Brynjolfsson, and Korinek\.
- Ross \(2001\)Ross, M\. L\. \(2001\)\.Does oil hinder democracy?World Politics53\(3\), 325–361\.
- Russell \(2022\)Russell, S\. \(2022\)\.If we succeed\.Daedalus151\(2\), 43–57\.
- Sachs et al\. \(2004\)Sachs, J\. L\., U\. G\. Mueller, T\. P\. Wilcox, and J\. J\. Bull \(2004\)\.The evolution of cooperation\.Quarterly Review of Biology79\(2\), 135–160\.
- Sachs and Simms \(2006\)Sachs, J\. L\. and E\. L\. Simms \(2006\)\.Pathways to mutualism breakdown\.Trends in Ecology & Evolution21\(10\), 585–592\.
- Salib and Goldstein \(2026\)Salib, P\. N\. and S\. Goldstein \(2026\)\.AI rights for human safety\.Virginia Law Review112\(4\), 1061\.
- Samuelson \(1938\)Samuelson, P\. A\. \(1938\)\.A note on the pure theory of consumer’s behaviour\.Economica5\(17\), 61–71\.
- Sastry et al\. \(2024\)Sastry, G\., L\. Heim, H\. Belfield, M\. Anderljung, M\. Brundage, J\. Hazell, C\. O’Keefe, G\. K\. Hadfield, R\. Ngo, K\. Pilz, G\. Gor, E\. Bluemke, S\. Shoker, J\. Egan, R\. F\. Trager, S\. Avin, A\. Weller, Y\. Bengio, and D\. Coyle \(2024\)\.Computing power and the governance of artificial intelligence\.
- Skalse et al\. \(2022\)Skalse, J\., N\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)\.Defining and characterizing reward gaming\.InAdvances in Neural Information Processing Systems, Volume 35, pp\. 9460–9471\.
- Smith \(2024\)Smith, N\. \(2024\)\.Plentiful, high\-paying jobs in the age of AI\.Noahpinion\.17 March 2024\.
- Sotala and Yampolskiy \(2015\)Sotala, K\. and R\. V\. Yampolskiy \(2015\)\.Responses to catastrophic AGI risk: A survey\.Physica Scripta90\(1\), 018001\.
- Taylor and Jonker \(1978\)Taylor, P\. D\. and L\. B\. Jonker \(1978\)\.Evolutionary stable strategies and game dynamics\.Mathematical Biosciences40\(1–2\), 145–156\.
- Timmis et al\. \(2004\)Timmis, J\. N\., M\. Ayliffe, C\. Huang, and W\. Martin \(2004\)\.Endosymbiotic gene transfer: Organelle genomes forge eukaryotic chromosomes\.Nature Reviews Genetics5\(2\), 123–135\.
- Trammell and Korinek \(2023\)Trammell, P\. and A\. Korinek \(2023\)\.Economic growth under transformative AI\.NBER Working Paper 31815, National Bureau of Economic Research\.
- Turner et al\. \(2021\)Turner, A\. M\., L\. Smith, R\. Shah, A\. Critch, and P\. Tadepalli \(2021\)\.Optimal policies tend to seek power\.InAdvances in Neural Information Processing Systems 34 \(NeurIPS 2021\), pp\. 23063–23074\.
- van der Weij et al\. \(2025\)van der Weij, T\., F\. Hofstätter, O\. Jaffe, S\. Brown, and F\. Ward \(2025\)\.AI sandbagging: Language models can strategically underperform on evaluations\.InInternational Conference on Learning Representations \(ICLR 2025\), pp\. 73152–73189\.
- Yudkowsky \(2024\)Yudkowsky, E\. \(2024\)\.The sun is big, but superintelligences will not spare Earth a little sunlight\.Machine Intelligence Research Institute blog\.23 September 2024; crossposted to LessWrong\.
- Zeder \(2015\)Zeder, M\. A\. \(2015\)\.Core questions in domestication research\.Proceedings of the National Academy of Sciences112\(11\), 3191–3198\.

## Appendix AExtensions

In this appendix we state the results that we only mention briefly in the main text\. We prove them in[AppendixB](https://arxiv.org/html/2611.23322#A2)\.

Competition among AIs\.Competition among AIs increases the human share\. But it does not change the limit in[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\.

Competing AIs bid up the human share until the winner gains nothing relative to losing\. So humans get the whole value of their cooperation\. But the value of human cooperation goes to zero as the dependence of the AI on humans ends, for any number of AIs\. Competition determines the price of what humans supply\. It does not create demand for what humans supply\. Also, AIs that are copies of one model, or that are run by one developer, may bargain as one party\.

Care through a proxy\.The care that an AI learns in training may be care for a measurable proxy for the human payoff, not for the human payoff itself\.

In reward hacking, a trained system pursues a measured proxy, not the quantity the proxy was meant to track\([Skalse et al\., 2022](https://arxiv.org/html/2611.23322#bib.bib43)\)\. Reward hacking reducesψ\\psi\. So it reduces the alignment that matters for the bargain\.

Control\.Humans may value control \(the ability to correct the AI\) as well as resources\. And an AI that values its goal has a reason to resist correction\.

###### Proposition 11\(Control\)\.

Suppose humans value control as well as consumption, with weightc∈\(0,1\)c\\in\(0,1\)on control\. Suppose humans use control to correct the AI with probabilityρ\\rhoper unit of control\. Suppose the AI’s goal is bounded, with valueg¯∈\[0,1\]\\bar\{g\}\\in\[0,1\]at the agreed share\. Suppose no gain in consumption offsets the loss of all control\. Then, asp→1p\\to 1, humans are above their baseline only if

α≥ρ​g¯ρ​g¯\+c\.\\alpha\\geq\\frac\{\\rho\\,\\bar\{g\}\}\{\\rho\\,\\bar\{g\}\+c\}\.Humans are above their baseline if the inequality is strict and the agreed share gives them more than their pre\-AI consumption\.

The bound in[Proposition11](https://arxiv.org/html/2611.23322#Thmtheorem11)does not increase with capability\. But it increases with the probability of correctionρ\\rho\. The better oversight works, the more care the AI needs in order to accept oversight, as in the off\-switch game\([Hadfield\-Menell et al\., 2017](https://arxiv.org/html/2611.23322#bib.bib18)\)\. So, if humans value correction in this way, some alignment is required without the premise of insatiable goals\. Near perfect alignment still requires the premise\.

Alignment that makes commitment unnecessary\.The next result continues[Proposition5](https://arxiv.org/html/2611.23322#Thmtheorem5)\. It uses the same notation, withuH,2u\_\{H,2\}anduA,2u\_\{A,2\}for the second\-period payoffs\.

###### Proposition 12\(Commitment and alignment\)\.

In the setting of[Proposition5](https://arxiv.org/html/2611.23322#Thmtheorem5), letθ2=1\\theta\_\{2\}=1, letuH,2u\_\{H,2\}be differentiable and strictly concave, let\(1−δ2\)​bH<h1≤uH,2​\(1\)\(1\-\\delta\_\{2\}\)b\_\{H\}<h\_\{1\}\\leq u\_\{H,2\}\(1\), and letx^\\hat\{x\}be such thatuH,2​\(x^\)=h1u\_\{H,2\}\(\\hat\{x\}\)=h\_\{1\}\. Asp2→1p\_\{2\}\\to 1, humans continue if and only if

α≥κ2κ2\+uH,2′​\(x^\)\.\\alpha\\geq\\frac\{\\kappa\_\{2\}\}\{\\kappa\_\{2\}\+u\_\{H,2\}^\{\\prime\}\(\\hat\{x\}\)\}\.If this condition is satisfied, then humans continue at every second\-period state with the sameκ2\\kappa\_\{2\}andδ2\\delta\_\{2\}\. If it is not satisfied, then halting is inefficient ifuA,2​\(x^\)\>a1u\_\{A,2\}\(\\hat\{x\}\)\>a\_\{1\}\.

The threshold has the same form as the threshold in[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(ii\), withx^\\hat\{x\}in place ofxhx\_\{h\}\. Herex^\\hat\{x\}is the share that gives humans what halting gives them;xhx\_\{h\}is the share that gives humans their baseline\. The better the first\-period agreement, the largerx^\\hat\{x\}, the smalleruH,2′​\(x^\)u\_\{H,2\}^\{\\prime\}\(\\hat\{x\}\), and the more alignment the AI needs\. If alignment also improves the first\-period agreement, as it does in the example in[AppendixC](https://arxiv.org/html/2611.23322#A3), then moderate alignment can make halting more common\. In that example, there is inefficient halting in most second\-period states if the AI has no alignment; in more states if the AI has moderate alignment; and in no state once the alignment of the AI is above the threshold\. In the example the threshold is close to11\.

Care as a public good\.Among AIs that all care about humans to some degree, care for humans is a public good\. This is because each AI’s gift to humans benefits every AI that cares\. So each AI prefers that another AI gives\.

###### Proposition 13\(Care as a public good\)\.

Suppose several AIs each give part of their resources to humans\. AIiivalues the human payoff with weightαi\\alpha\_\{i\}\. In every equilibrium, an AI gives only if every AI that cares more has given all it has\([Bergstrom et al\., 1986](https://arxiv.org/html/2611.23322#bib.bib5)\)\. Suppose AIs grow in proportion to the resources they keep\. If some AI with positive resources has alignment at mostκ/\(κ\+uH′​\(xh\)\)\\kappa/\(\\kappa\+u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\), the requirement for a single AI in[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(ii\), then the human share decreases toxhx\_\{h\}or below in finite time\.

###### Result 1\(The path with selection\)\.

On the illustrative path, selection among AI variants ends mutualism earlier than it ends with a fixed population of variants\. Faster selection ends mutualism earlier still \([Figure3](https://arxiv.org/html/2611.23322#A1.F3)\)\. Without sanctions, mutualism ends earlier again\. If the quality of evaluation stays constant as capability increases, mutualism ends later\.

Figure 3\.Selection among AI variants ends mutualism earlier on the path\. The curves are the population mean human payoff on the illustrative path: with a fixed population of AI variants; and with slow, medium, and fast selection among variants that differ in alignment\. The faster the selection, the earlier the mean human payoff decreases below the pre\-AI baseline\. Each variant bargains separately with the humans it deals with\. Each curve is the mean of the resulting human payoffs, weighted by the shares of the variants in the AI population\. The fitness of a variant is its goal output per unit of capability minus sanctions\. Highly aligned variants monitor the others\. The quality of evaluation decreases with capability\. The values of the parameters are in[AppendixC](https://arxiv.org/html/2611.23322#A3)\.Several human groups\.[Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8)implies that human groups differ in their leverage over the AI\.

###### Result 2\(Several human groups\)\.

Suppose a small elite controls what the AI depends on, such as computing and its supply chain\. The rest of the population does not\. Then the elite needs no alignment to stay above its baseline until the AI is nearly independent\. The rest of the population needs a high degree of alignment much earlier, if it also has less bargaining power and a higher cost of conflict\.

[Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8)gives one reason why the rest of the population has less bargaining power: its members can be substituted for one another\. So the population average can be above the baseline while most humans are below it\. States that depend little on the taxes of their citizens tend to be less accountable to them\([Ross, 2001](https://arxiv.org/html/2611.23322#bib.bib36)\)\. The same mechanism applies to institutions that no longer need human work\([Kulveit et al\., 2025](https://arxiv.org/html/2611.23322#bib.bib27);[Drago and Laine, 2025](https://arxiv.org/html/2611.23322#bib.bib13)\)\.

## Appendix BProofs

###### Proof of[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)\.

Withα=0\\alpha=0,bA=κ​θb\_\{A\}=\\kappa\\theta\(the AI atx=0x=0, below the threshold whenx¯\>0\\bar\{x\}\>0\)\. Forx≥x¯x\\geq\\bar\{x\},uA​\(x\)−bA=κ⁡\(1−x−θ\)u\_\{A\}\(x\)\-b\_\{A\}=\\kappa\(1\-x\-\\theta\), which is positive if and only ifx<1−θx<1\-\\theta\. For0<x<x¯0<x<\\bar\{x\},uA​\(x\)−bA=−κ​θ​x≤0u\_\{A\}\(x\)\-b\_\{A\}=\-\\kappa\\theta x\\leq 0, so no such share benefits the AI\. Humans gain overbHb\_\{H\}if and only ifx\>xhx\>x\_\{h\}\. So a share benefiting both exists if and only if somex≥x¯x\\geq\\bar\{x\}satisfiesxh<x<1−θx\_\{h\}<x<1\-\\theta, that is, if and only ifmax⁡\(x¯,xh\)<1−θ\\max\(\\bar\{x\},x\_\{h\}\)<1\-\\theta\. None ofκ\\kappa,pp, andγ\\gammaappears\. This proves \(i\)\.

For \(ii\),dA=κ⁡\(θ−γ\)d\_\{A\}=\\kappa\(\\theta\-\\gamma\)\. A sharex≥x¯x\\geq\\bar\{x\}gives the AIκ⁡\(1−x\)\\kappa\(1\-x\), which exceedsdAd\_\{A\}if and only ifx<1−θ\+γx<1\-\\theta\+\\gamma\. A sharex<x¯x<\\bar\{x\}gives the AIκ​θ​\(1−x\)\\kappa\\theta\(1\-x\), which exceedsdAd\_\{A\}if and only ifx<γ/θx<\\gamma/\\theta, andγ/θ≤1−θ\+γ\\gamma/\\theta\\leq 1\-\\theta\+\\gammabecauseγ≤θ≤1\\gamma\\leq\\theta\\leq 1\. So every share the AI prefers to disagreement is below1−θ\+γ1\-\\theta\+\\gamma, and humans are above their baseline only at shares abovexhx\_\{h\}; this gives the necessary condition\. Ifmax⁡\(x¯,xh\)<1−θ\+γ\\max\(\\bar\{x\},x\_\{h\}\)<1\-\\theta\+\\gamma, every share strictly between the two sides of the inequality is at leastx¯\\bar\{x\}, abovexhx\_\{h\}, and preferred by the AI to disagreement\. Atθ=1\\theta=1the thresholdx¯\\bar\{x\}does not affectGG, and the AI prefers a share to disagreement if and only if the share is belowγ\\gamma\. The last claim is part \(i\)\. ∎

###### Proof of[Proposition2](https://arxiv.org/html/2611.23322#Thmtheorem2)\.

Asp→1p\\to 1the Nash solution tends to the AI’s best acceptable share, the share that maximizesuAu\_\{A\}subject touH​\(x\)≥dHu\_\{H\}\(x\)\\geq d\_\{H\}\. Onx≥x¯x\\geq\\bar\{x\},uAu\_\{A\}is decreasing, so the AI chooses the smallest acceptable share at or abovex¯\\bar\{x\}:x¯\\bar\{x\}ifuH​\(x¯\)≥dHu\_\{H\}\(\\bar\{x\}\)\\geq d\_\{H\}, and otherwise thexxwithuH​\(x\)=dHu\_\{H\}\(x\)=d\_\{H\}, if the AI accepts that share\. If the AI does not accept it, there is no agreement and humans getdHd\_\{H\}, which then exceedsuH​\(x¯\)u\_\{H\}\(\\bar\{x\}\)\. In each case the human payoff ismax⁡\(dH,uH​\(x¯\)\)\\max\(d\_\{H\},u\_\{H\}\(\\bar\{x\}\)\)\. It is belowbHb\_\{H\}whendH<bHd\_\{H\}<b\_\{H\}\(that is,δ\>0\\delta\>0\) anduH​\(x¯\)<bHu\_\{H\}\(\\bar\{x\}\)<b\_\{H\}\(that is,x¯<xh\\bar\{x\}<x\_\{h\}\)\. ∎

###### Proof of[Remark9](https://arxiv.org/html/2611.23322#Thmtheorem9)\.

An AI that wins the partnership at human sharexxgetsκ⁡\(1−x\)\\kappa\(1\-x\); an AI that loses getsκ​θ\\kappa\\theta\. Each AI bids up to the share at which it is indifferent,x=1−θx=1\-\\theta\. With two or more bidders, Bertrand competition drives the winning bid to that share\. ∎

###### Proof of[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\.

\(i\) Withθ=1\\theta=1,bA=uA​\(0\)=\(1−α\)​κb\_\{A\}=u\_\{A\}\(0\)=\(1\-\\alpha\)\\kappa\. Mutualism requiresuH​\(x\)\>bHu\_\{H\}\(x\)\>b\_\{H\}, sox\>xhx\>x\_\{h\}, anduA​\(x\)\>bAu\_\{A\}\(x\)\>b\_\{A\}, that is,α​uH​\(x\)\>\(1−α\)​κ​x\\alpha u\_\{H\}\(x\)\>\(1\-\\alpha\)\\kappa x\. BecauseuHu\_\{H\}is concave withuH​\(0\)=0u\_\{H\}\(0\)=0,uH​\(x\)/xu\_\{H\}\(x\)/xis non\-increasing, souH​\(x\)/x≤uH​\(xh\)/xh=bH/xhu\_\{H\}\(x\)/x\\leq u\_\{H\}\(x\_\{h\}\)/x\_\{h\}=b\_\{H\}/x\_\{h\}\. Henceα​bH/xh\>\(1−α\)​κ\\alpha b\_\{H\}/x\_\{h\}\>\(1\-\\alpha\)\\kappa, that is,α\>κ​xh/\(κ​xh\+bH\)\\alpha\>\\kappa x\_\{h\}/\(\\kappa x\_\{h\}\+b\_\{H\}\)\. Only the definition of mutualism was used, so the bound holds for every agreement\.

\(ii\) Withθ=1\\theta=1,uA​\(x\)=\(1−α\)​κ​\(1−x\)\+α​uH​\(x\)u\_\{A\}\(x\)=\(1\-\\alpha\)\\kappa\(1\-x\)\+\\alpha u\_\{H\}\(x\)is concave, with derivative−\(1−α\)​κ\+α​uH′​\(x\)\-\(1\-\\alpha\)\\kappa\+\\alpha u\_\{H\}^\{\\prime\}\(x\)\. Asp→1p\\to 1the agreement tends to the AI’s preferred acceptable share\. Humans exceedbHb\_\{H\}at that share if and only ifuAu\_\{A\}is still increasing atxhx\_\{h\}, that is,α​uH′​\(xh\)\>\(1−α\)​κ\\alpha u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\>\(1\-\\alpha\)\\kappa, which holds if and only ifα\>κ/\(κ\+uH′​\(xh\)\)\\alpha\>\\kappa/\(\\kappa\+u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\)\. ∎

###### Proof of[Theorem4](https://arxiv.org/html/2611.23322#Thmtheorem4)\.

Writef⁡\(x\)=\(1−p\)​log⁡gH​\(x\)\+p​log⁡gA​\(x\)f\(x\)=\(1\-p\)\\log g\_\{H\}\(x\)\+p\\log g\_\{A\}\(x\), withgH=uH−dHg\_\{H\}=u\_\{H\}\-d\_\{H\}andgA=uA−bAg\_\{A\}=u\_\{A\}\-b\_\{A\}\. Letxxbe the agreement at parameter valueτ\\tau, and lety\>xy\>x\.

Efficiency\.Sincexxis optimal anduH​\(y\)\>uH​\(x\)u\_\{H\}\(y\)\>u\_\{H\}\(x\),yycannot be better thanxxfor the AI as well, sogA​\(y\)≤gA​\(x\)g\_\{A\}\(y\)\\leq g\_\{A\}\(x\)\.

Single crossing\.It suffices thatD⁡\(τ\)=f⁡\(y,τ\)−f⁡\(x,τ\)D\(\\tau\)=f\(y;\\tau\)\-f\(x;\\tau\)is non\-increasing inτ\\tau\. ThenD⁡\(τ′\)≤D⁡\(τ\)≤0D\(\\tau^\{\\prime\}\)\\leq D\(\\tau\)\\leq 0forτ′\>τ\\tau^\{\\prime\}\>\\tau, so noy\>xy\>xis strictly better atτ′\\tau^\{\\prime\}; a tie atτ′\\tau^\{\\prime\}would force a tie atτ\\tau, which the tie\-breaking rule excludes\.

δ\\delta\.dH=\(1−δ\)​bHd\_\{H\}=\(1\-\\delta\)b\_\{H\}falls asδ\\deltarises, and∂D/∂dH=\(1−p\)​\[1/gH​\(x\)−1/gH​\(y\)\]\>0\\partial D/\\partial d\_\{H\}=\(1\-p\)\\,\[1/g\_\{H\}\(x\)\-1/g\_\{H\}\(y\)\]\>0becausegH​\(y\)\>gH​\(x\)g\_\{H\}\(y\)\>g\_\{H\}\(x\)\. A lowerdHd\_\{H\}only adds acceptable shares belowxx\.

pp\.∂D/∂p=\[log⁡gA​\(y\)−log⁡gA​\(x\)\]−\[log⁡gH​\(y\)−log⁡gH​\(x\)\]<0\\partial D/\\partial p=\[\\log g\_\{A\}\(y\)\-\\log g\_\{A\}\(x\)\]\-\[\\log g\_\{H\}\(y\)\-\\log g\_\{H\}\(x\)\]<0by efficiency anduH​\(y\)\>uH​\(x\)u\_\{H\}\(y\)\>u\_\{H\}\(x\)\. The acceptable shares do not depend onpp\.

θ\\theta\.SincebA=uA​\(0\)b\_\{A\}=u\_\{A\}\(0\),∂gA​\(z\)/∂θ=−\(1−α\)​κ​m​\(z\)\\partial g\_\{A\}\(z\)/\\partial\\theta=\-\(1\-\\alpha\)\\kappa\\,m\(z\), withm⁡\(z\)=zm\(z\)=zifz<x¯z<\\bar\{x\}andm⁡\(z\)=1m\(z\)=1otherwise\. So∂D/∂θ=−p⁡\(1−α\)​κ​\[m⁡\(y\)/gA​\(y\)−m⁡\(x\)/gA​\(x\)\]≤0\\partial D/\\partial\\theta=\-p\(1\-\\alpha\)\\kappa\\,\[m\(y\)/g\_\{A\}\(y\)\-m\(x\)/g\_\{A\}\(x\)\]\\leq 0, becausemmis non\-decreasing andgA​\(y\)≤gA​\(x\)g\_\{A\}\(y\)\\leq g\_\{A\}\(x\)\. The gapgA​\(y\)−gA​\(x\)g\_\{A\}\(y\)\-g\_\{A\}\(x\)changes at rate−\(1−α\)​κ​\(m⁡\(y\)−m⁡\(x\)\)≤0\-\(1\-\\alpha\)\\kappa\\,\(m\(y\)\-m\(x\)\)\\leq 0, so the ordering persists asθ\\thetarises, and the acceptable shares shrink\.

Finally, suppose the relationship is mutualistic at\(α,τ′\)\(\\alpha,\\tau^\{\\prime\}\)with agreementx′x^\{\\prime\}\. ThenuH​\(x∗\)\>bHu\_\{H\}\(x^\{\*\}\)\>b\_\{H\}atτ\\tautoo, sincex∗​\(τ\)≥x′x^\{\*\}\(\\tau\)\\geq x^\{\\prime\}\. The AI is also above its baseline atτ\\tau\. Atτ\\tauthe sharex′x^\{\\prime\}gives both sides strictly positive gains:gAg\_\{A\}is unchanged byppandδ\\deltaand does not fall asθ\\thetafalls, anduH​\(x′\)\>bH≥dHu\_\{H\}\(x^\{\\prime\}\)\>b\_\{H\}\\geq d\_\{H\}for everyδ\\delta\. With0<p<10<p<1the Nash product is positive only where both gains are, so the maximizer atτ\\taugives the AI a strictly positive gain\. So the set ofα\\alphagiving mutualism shrinks asτ\\taurises, andα∗\\alpha^\{\*\}rises\. ∎

###### Proof of[Remark10](https://arxiv.org/html/2611.23322#Thmtheorem10)\.

Letλ=1−α\+α​ψ\\lambda=1\-\\alpha\+\\alpha\\psi, which is positive becauseα<1\\alpha<1, and letu~A​\(x\)=\(1−αψ\)​G​\(x\)\+αψ​uH​\(x\)\\tilde\{u\}\_\{A\}\(x\)=\(1\-\\alpha\_\{\\psi\}\)G\(x\)\+\\alpha\_\{\\psi\}u\_\{H\}\(x\)be the payoff of an AI with alignmentαψ\\alpha\_\{\\psi\}that cares about the human payoff itself\. Since1−αψ=\(1−α\)/λ1\-\\alpha\_\{\\psi\}=\(1\-\\alpha\)/\\lambdaandαψ=α​ψ/λ\\alpha\_\{\\psi\}=\\alpha\\psi/\\lambda,

uA​\(x\)=\(1−α\)​G​\(x\)\+α​ψ​uH​\(x\)\+α⁡\(1−ψ\)​h=λ​u~A​\(x\)\+α⁡\(1−ψ\)​h\.u\_\{A\}\(x\)=\(1\-\\alpha\)G\(x\)\+\\alpha\\psi\\,u\_\{H\}\(x\)\+\\alpha\(1\-\\psi\)h=\\lambda\\,\\tilde\{u\}\_\{A\}\(x\)\+\\alpha\(1\-\\psi\)h\.The AI’s baseline transforms in the same way, souA​\(x\)−bA=λ⁡\(u~A​\(x\)−u~A​\(0\)\)u\_\{A\}\(x\)\-b\_\{A\}=\\lambda\\,\(\\tilde\{u\}\_\{A\}\(x\)\-\\tilde\{u\}\_\{A\}\(0\)\)\. The set of shares both sides accept is therefore the same underuAu\_\{A\}andu~A\\tilde\{u\}\_\{A\}, the Nash product underuAu\_\{A\}isλp\\lambda^\{p\}times the Nash product underu~A\\tilde\{u\}\_\{A\}, and the two have the same maximizer for everypp\. The AI is above its baseline underuAu\_\{A\}exactly when it is underu~A\\tilde\{u\}\_\{A\}, so mutualism also coincides\. Finally,αψ≤α\\alpha\_\{\\psi\}\\leq\\alphais equivalent toψ⁡\(1−α\)≤1−α\\psi\(1\-\\alpha\)\\leq 1\-\\alpha, which holds becauseψ≤1\\psi\\leq 1, andαψ=0\\alpha\_\{\\psi\}=0whenψ=0\\psi=0\. ∎

For[Proposition11](https://arxiv.org/html/2611.23322#Thmtheorem11), an outcome is a pair\(x,k\)\(x,k\), wherex∈\[0,1\]x\\in\[0,1\]is the human share of resources andk∈\[0,1\]k\\in\[0,1\]the human share of control\. The human payoff isUH​\(x,k\)=\(1−c\)​v​\(x\)\+c​kU\_\{H\}\(x,k\)=\(1\-c\)v\(x\)\+ck, where the consumption payoffvvis continuous and increasing, and the human baseline isbH=\(1−c\)​v0\+cb\_\{H\}=\(1\-c\)v\_\{0\}\+c, wherev0v\_\{0\}is pre\-AI consumption payoff\. The AI’s goal valueg⁡\(1−x\)∈\[0,1\]g\(1\-x\)\\in\[0,1\]is continuous and non\-increasing inxx, and the AI’s payoff isUA​\(x,k\)=\(1−α\)​\(1−ρ​k\)​g​\(1−x\)\+α​UH​\(x,k\)U\_\{A\}\(x,k\)=\(1\-\\alpha\)\(1\-\\rho k\)\\,g\(1\-x\)\+\\alpha U\_\{H\}\(x,k\)\. The disagreement points aredH=\(1−δ\)​bHd\_\{H\}=\(1\-\\delta\)b\_\{H\}withδ\>0\\delta\>0andbA=UA​\(0,0\)b\_\{A\}=U\_\{A\}\(0,0\)\. That no consumption gain offsets losing all control means\(1−c\)​\(v⁡\(1\)−v0\)<c\(1\-c\)\(v\(1\)\-v\_\{0\}\)<c\.

###### Proof of[Proposition11](https://arxiv.org/html/2611.23322#Thmtheorem11)\.

Asp→1p\\to 1the agreement\(x∗,k∗\)\(x^\{\*\},k^\{\*\}\)maximizesUAU\_\{A\}over the acceptable outcomes, those withUH≥dHU\_\{H\}\\geq d\_\{H\}andUA≥bAU\_\{A\}\\geq b\_\{A\}, as in the proof of[Proposition2](https://arxiv.org/html/2611.23322#Thmtheorem2)\. For fixedxx,UAU\_\{A\}is linear inkkwith slopeα​c−\(1−α\)​ρ​g​\(1−x\)\\alpha c\-\(1\-\\alpha\)\\rho\\,g\(1\-x\)\. Atx∗x^\{\*\}the slope isS=α​c−\(1−α\)​ρ​g¯S=\\alpha c\-\(1\-\\alpha\)\\rho\\bar\{g\}, andS<0S<0exactly whenα<ρ​g¯/\(ρ​g¯\+c\)\\alpha<\\rho\\bar\{g\}/\(\\rho\\bar\{g\}\+c\)\.

We first show necessity\. SupposeS<0S<0, and suppose for contradiction thatUH​\(x∗,k∗\)\>bHU\_\{H\}\(x^\{\*\},k^\{\*\}\)\>b\_\{H\}\. Becauseδ\>0\\delta\>0,bH\>dHb\_\{H\}\>d\_\{H\}, so the human constraint is slack at the agreement\. Ifk∗\>0k^\{\*\}\>0, loweringkkslightly keepsUHU\_\{H\}abovedHd\_\{H\}and raisesUAU\_\{A\}, so the new outcome is acceptable and better for the AI, which contradicts the maximality of\(x∗,k∗\)\(x^\{\*\},k^\{\*\}\)\. Hencek∗=0k^\{\*\}=0\. But thenUH​\(x∗,0\)=\(1−c\)​v​\(x∗\)≤\(1−c\)​v​\(1\)<\(1−c\)​v0\+c=bHU\_\{H\}\(x^\{\*\},0\)=\(1\-c\)v\(x^\{\*\}\)\\leq\(1\-c\)v\(1\)<\(1\-c\)v\_\{0\}\+c=b\_\{H\}, where the first inequality uses thatvvis increasing and the second is the assumption that consumption cannot offset the loss of all control\. This contradictsUH​\(x∗,k∗\)\>bHU\_\{H\}\(x^\{\*\},k^\{\*\}\)\>b\_\{H\}\.

We next show sufficiency\. SupposeS\>0S\>0andv⁡\(x∗\)\>v0v\(x^\{\*\}\)\>v\_\{0\}\. Ifk∗<1k^\{\*\}<1, raisingkkraises bothUAU\_\{A\}andUHU\_\{H\}, so the new outcome is acceptable and better for the AI, again a contradiction\. Hencek∗=1k^\{\*\}=1, andUH​\(x∗,1\)=\(1−c\)​v​\(x∗\)\+c\>\(1−c\)​v0\+c=bHU\_\{H\}\(x^\{\*\},1\)=\(1\-c\)v\(x^\{\*\}\)\+c\>\(1\-c\)v\_\{0\}\+c=b\_\{H\}becausec<1c<1\. ∎

The bound in[Proposition11](https://arxiv.org/html/2611.23322#Thmtheorem11)is stated at the agreed share, so it is a condition on the outcome rather than a closed form in the primitives\. Because the goal is bounded,g¯≤1\\bar\{g\}\\leq 1, and the bound is at mostρ/\(ρ\+c\)\\rho/\(\\rho\+c\)whatever the AI’s capability\.

###### Proof of[Propositions5](https://arxiv.org/html/2611.23322#Thmtheorem5)and[12](https://arxiv.org/html/2611.23322#Thmtheorem12)\.

Writex2x\_\{2\}for the second\-period agreement andd2=\(1−δ2\)​bHd\_\{2\}=\(1\-\\delta\_\{2\}\)b\_\{H\}for the human disagreement payoff in the second period\.

\(i\) Humans getuH,2​\(x2\)u\_\{H,2\}\(x\_\{2\}\)if they continue andh1h\_\{1\}if they halt, so they halt whenuH,2​\(x2\)<h1u\_\{H,2\}\(x\_\{2\}\)<h\_\{1\}\. Letτ=\(θ2,δ2,p2\)\\tau=\(\\theta\_\{2\},\\delta\_\{2\},p\_\{2\}\)andτ′=\(θ2′,δ2′,p2′\)\\tau^\{\\prime\}=\(\\theta\_\{2\}^\{\\prime\},\\delta\_\{2\}^\{\\prime\},p\_\{2\}^\{\\prime\}\)withτ≤τ′\\tau\\leq\\tau^\{\\prime\}in every coordinate and an agreement at both\. Move fromτ\\tautoτ′\\tau^\{\\prime\}by raisingδ2\\delta\_\{2\}, thenp2p\_\{2\}, thenθ2\\theta\_\{2\}\. Raisingδ2\\delta\_\{2\}enlarges the set of shares humans accept,p2p\_\{2\}changes neither side’s set, and raisingθ2\\theta\_\{2\}shrinks the set of shares the AI accepts \(proof of[Theorem4](https://arxiv.org/html/2611.23322#Thmtheorem4)\)\. So every intermediate point accepts every share that both sides accept atτ′\\tau^\{\\prime\}, and an agreement exists there\. Applying[Theorem4](https://arxiv.org/html/2611.23322#Thmtheorem4)to one parameter at a time givesx2​\(τ′\)≤x2​\(τ\)x\_\{2\}\(\\tau^\{\\prime\}\)\\leq x\_\{2\}\(\\tau\)\. Hence if humans halt atτ\\tau, thenuH,2​\(x2​\(τ′\)\)≤uH,2​\(x2​\(τ\)\)<h1u\_\{H,2\}\(x\_\{2\}\(\\tau^\{\\prime\}\)\)\\leq u\_\{H,2\}\(x\_\{2\}\(\\tau\)\)<h\_\{1\}, and they halt atτ′\\tau^\{\\prime\}\.

\(ii\) If the AI commits to a sharexx, humans continue ifuH,2​\(x\)≥h1u\_\{H,2\}\(x\)\\geq h\_\{1\}, and the AI prefers committing to being halted ifuA,2​\(x\)≥a1u\_\{A,2\}\(x\)\\geq a\_\{1\}\. If a halt would be inefficient, somexxsatisfies both inequalities, and the AI commits to it, so humans do not halt\. Hence humans halt only when the halt is efficient\.

For[Proposition12](https://arxiv.org/html/2611.23322#Thmtheorem12): withθ2=1\\theta\_\{2\}=1,uA,2​\(x\)=\(1−α\)​κ2​\(1−x\)\+α​uH,2​\(x\)u\_\{A,2\}\(x\)=\(1\-\\alpha\)\\kappa\_\{2\}\(1\-x\)\+\\alpha u\_\{H,2\}\(x\), which is concave, and strictly concave ifα\>0\\alpha\>0, and the AI’s baseline isuA,2​\(0\)u\_\{A,2\}\(0\)\. Asp2→1p\_\{2\}\\to 1the agreement tends to the AI’s best acceptable share, the maximizer ofuA,2u\_\{A,2\}over shares withuH,2​\(x\)≥d2u\_\{H,2\}\(x\)\\geq d\_\{2\}, provided it gives the AI at least its baseline \(proof of[Proposition2](https://arxiv.org/html/2611.23322#Thmtheorem2)\)\. Letxdx\_\{d\}be the smallest share withuH,2​\(xd\)≥d2u\_\{H,2\}\(x\_\{d\}\)\\geq d\_\{2\}, and letxαx\_\{\\alpha\}maximizeuA,2u\_\{A,2\}on\[0,1\]\[0,1\]\. By concavity the AI’s best acceptable share ismax⁡\(xd,xα\)\\max\(x\_\{d\},x\_\{\\alpha\}\)\. Becauseh1\>d2h\_\{1\}\>d\_\{2\},x^\>xd\\hat\{x\}\>x\_\{d\}\.

Supposeα​uH,2′​\(x^\)≥\(1−α\)​κ2\\alpha u\_\{H,2\}^\{\\prime\}\(\\hat\{x\}\)\\geq\(1\-\\alpha\)\\kappa\_\{2\}\. Thenα\>0\\alpha\>0, the derivative ofuA,2u\_\{A,2\}atx^\\hat\{x\}is nonnegative, and by strict concavityxα≥x^x\_\{\\alpha\}\\geq\\hat\{x\}\. The agreement isxαx\_\{\\alpha\}, which gives the AI at leastuA,2​\(0\)u\_\{A,2\}\(0\)becausexαx\_\{\\alpha\}maximizesuA,2u\_\{A,2\}, and gives humans at leasth1h\_\{1\}, so humans continue\. Suppose instead thatα​uH,2′​\(x^\)<\(1−α\)​κ2\\alpha u\_\{H,2\}^\{\\prime\}\(\\hat\{x\}\)<\(1\-\\alpha\)\\kappa\_\{2\}\. Then the derivative ofuA,2u\_\{A,2\}atx^\\hat\{x\}is negative, soxα<x^x\_\{\\alpha\}<\\hat\{x\}andmax⁡\(xd,xα\)<x^\\max\(x\_\{d\},x\_\{\\alpha\}\)<\\hat\{x\}\. Either the agreement gives humans less thanh1h\_\{1\}or there is no agreement, and humans halt\. This proves the equivalence\.

When the condition holds,xα≥x^\>0x\_\{\\alpha\}\\geq\\hat\{x\}\>0, souA,2​\(xα\)\>uA,2​\(0\)u\_\{A,2\}\(x\_\{\\alpha\}\)\>u\_\{A,2\}\(0\)anduH,2​\(xα\)≥h1\>d2u\_\{H,2\}\(x\_\{\\alpha\}\)\\geq h\_\{1\}\>d\_\{2\}: the sharexαx\_\{\\alpha\}gives both sides positive gains, and an agreement exists atθ2=1\\theta\_\{2\}=1for everyp2<1p\_\{2\}<1\. By the argument of part \(i\), an agreement then exists at everyθ2≤1\\theta\_\{2\}\\leq 1andp2<1p\_\{2\}<1with the sameδ2\\delta\_\{2\}\. By[Theorem4](https://arxiv.org/html/2611.23322#Thmtheorem4),x2x\_\{2\}is non\-increasing inp2p\_\{2\}and inθ2\\theta\_\{2\}, so it is at least its limit atθ2=1\\theta\_\{2\}=1asp2→1p\_\{2\}\\to 1, which is at leastx^\\hat\{x\}\. So humans continue at every such state\. When the condition fails, the halt is inefficient by definition ifuA,2​\(x^\)\>a1u\_\{A,2\}\(\\hat\{x\}\)\>a\_\{1\}, sincex^\\hat\{x\}gives humans exactlyh1h\_\{1\}and the AI more thana1a\_\{1\}\. ∎

For[Proposition6](https://arxiv.org/html/2611.23322#Thmtheorem6), the game is as follows\. Nature makes the AI highly capable with probabilityμ∈\(0,1\)\\mu\\in\(0,1\)and misaligned with probabilityπ∈\(0,1\)\\pi\\in\(0,1\), independently\. A low\-capability AI displays low capability\. A highly capable AI either displays its capability or hides it at costζ\>0\\zeta\>0; evaluation detects hiding with probabilityη∈\[0,1\)\\eta\\in\[0,1\), and humans then see high capability\. Humans see the display and either halt, which gives every player00, or continue\. Continuing gives humansM\>0M\>0with a low\-capability AI,A\>0A\>0with an aligned capable AI, andB<0B<0with a misaligned capable AI\. The AI getsV\>0V\>0if humans continue, minusζ\\zetaif it hid, plus, if it is aligned,αa≥0\\alpha\_\{a\}\\geq 0times the human payoff\. Humans would halt an AI they knew to be highly capable,\(1−π\)​A\+π​B<0\(1\-\\pi\)A\+\\pi B<0\. Highly capable AIs are not too common when

μ≤μ†=MM\+\(1−η\)​\|\(1−π\)​A\+π​B\|\.\\mu\\leq\\mu^\{\\dagger\}=\\frac\{M\}\{M\+\(1\-\\eta\)\\,\\lvert\(1\-\\pi\)A\+\\pi B\\rvert\}\.
###### Proof of[Proposition6](https://arxiv.org/html/2611.23322#Thmtheorem6)\.

Fix a perfect Bayesian equilibrium in pure strategies\. We show in turn that humans continue after a low display, that they halt after a high display, and that both capable types hide\.

Suppose humans halt after a low display\. Then hiding cannot raise a capable AI’s chance of continuing, since an undetected hider is halted and a detected hider is treated like an honest one, while hiding costsζ\>0\\zeta\>0\. So both capable types display honestly\. A low display then comes only from low\-capability AIs, after which continuing givesM\>0M\>0, and halting is not a best response\. Hence humans continue after a low display\.

Suppose humans continue after a high display\. Then both capable types are continued whatever they display, so hiding only costsζ\\zeta, and both display honestly\. The high display then comes from every capable AI, the posterior on alignment equals the prior, and continuing gives\(1−π\)​A\+π​B<0\(1\-\\pi\)A\+\\pi B<0\. Halting is better, a contradiction\. Hence humans halt after a high display\.

Given these responses, a capable misaligned AI gets00by displaying honestly and\(1−η\)​V−ζ\>0\(1\-\\eta\)V\-\\zeta\>0by hiding, and a capable aligned AI gets00by displaying honestly and\(1−η\)​\(V\+αa​A\)−ζ≥\(1−η\)​V−ζ\>0\(1\-\\eta\)\(V\+\\alpha\_\{a\}A\)\-\\zeta\\geq\(1\-\\eta\)V\-\\zeta\>0by hiding\. Both hide\. Among capable AIs, the probability of a low display is1−η1\-\\etafor both types, so the display is independent of alignment, and evaluation carries no information about which AIs are misaligned\.

It remains to check the human responses\. A high display comes only from detected hiders of both types in prior proportions, or is off the equilibrium path whenη=0\\eta=0, where beliefs can be set to the prior; either way halting is optimal\. Humans must also be willing to continue after a low display\. The low display comes from low\-capability AIs with probability1−μ1\-\\muand from undetected capable AIs with probabilityμ⁡\(1−η\)\\mu\(1\-\\eta\), so continuing is optimal exactly when\(1−μ\)​M\+μ⁡\(1−η\)​\[\(1−π\)​A\+π​B\]≥0\(1\-\\mu\)M\+\\mu\(1\-\\eta\)\\big\[\(1\-\\pi\)A\+\\pi B\\big\]\\geq 0, that is, whenμ≤μ†\\mu\\leq\\mu^\{\\dagger\}\. Ifμ\>μ†\\mu\>\\mu^\{\\dagger\}, this step fails, and by the argument above no pure\-strategy equilibrium exists\. ∎

Under honest evaluation, humans continue only with low\-capability AIs and get\(1−μ\)​M\(1\-\\mu\)M\. Relative to that, hiding costs humansμ⁡\(1−η\)​\|\(1−π\)​A\+π​B\|\\mu\(1\-\\eta\)\\lvert\(1\-\\pi\)A\+\\pi B\\rvert\.

###### Proof of[Proposition7](https://arxiv.org/html/2611.23322#Thmtheorem7)\.

Letzzbe the misaligned share\. Per unit of resources, an aligned variant keeps1−xa1\-x\_\{a\}, and a misaligned variant keeps11less an expected sanctionσ​q\\sigma q\. Under the replicator dynamics in which each variant grows in proportion to what it keeps,

z˙=z⁡\(1−z\)​\[\(1−σ​q\)−\(1−xa\)\]=z⁡\(1−z\)​\(xa−σ​q\)\.\\dot\{z\}=z\(1\-z\)\\big\[\(1\-\\sigma q\)\-\(1\-x\_\{a\}\)\\big\]=z\(1\-z\)\\,\(x\_\{a\}\-\\sigma q\)\.Withqqfixed the bracket has a constant sign, so a rare misaligned variant grows if and only ifσ​q<xa\\sigma q<x\_\{a\}, and thenz→1z\\to 1from every interior start\. With monitors drawn from the population,q=q0​\(1−z\)q=q\_\{0\}\(1\-z\), and the bracket becomesϕ⁡\(z\)=xa−σ​q0​\(1−z\)\\phi\(z\)=x\_\{a\}\-\\sigma q\_\{0\}\(1\-z\), which is strictly increasing inzzand vanishes atz†z^\{\\dagger\}\. Ifσ​q0\>xa\\sigma q\_\{0\}\>x\_\{a\}, thenz†∈\(0,1\)z^\{\\dagger\}\\in\(0,1\)\. Forz∈\(0,z†\)z\\in\(0,z^\{\\dagger\}\),ϕ⁡\(z\)<0\\phi\(z\)<0, sozzdecreases; it cannot cross the rest pointz†z^\{\\dagger\}, and as a bounded monotone solution it converges to a rest point in\[0,z†\)\[0,z^\{\\dagger\}\), which can only be00\. The same argument shows thatz→1z\\to 1from every start in\(z†,1\)\(z^\{\\dagger\},1\)\. ∎

For[Proposition13](https://arxiv.org/html/2611.23322#Thmtheorem13), each period total output is11\. AIiiholds resourcesRi​\(t\)\>0R\_\{i\}\(t\)\>0and controls the shareri=Ri/Rr\_\{i\}=R\_\{i\}/Rof output, whereR=∑iRiR=\\sum\_\{i\}R\_\{i\}\. AIiigivesxi∈\[0,ri\]x\_\{i\}\\in\[0,r\_\{i\}\]to humans, the human share isX=∑ixiX=\\sum\_\{i\}x\_\{i\}, and AIiihas payoff\(1−αi\)​κ​\(ri−xi\)\+αi​uH​\(X\)\(1\-\\alpha\_\{i\}\)\\kappa\(r\_\{i\}\-x\_\{i\}\)\+\\alpha\_\{i\}u\_\{H\}\(X\), whereuHu\_\{H\}is differentiable, increasing, and strictly concave\. Resources grow with what is kept:Ri​\(t\+1\)=Ri​\(t\)\+ν⁡\(ri−xi\)R\_\{i\}\(t\+1\)=R\_\{i\}\(t\)\+\\nu\(r\_\{i\}\-x\_\{i\}\)withν\>0\\nu\>0\.

###### Proof of[Proposition13](https://arxiv.org/html/2611.23322#Thmtheorem13)\.

AIii’s payoff is concave inxix\_\{i\}with derivativeαi​uH′​\(X\)−\(1−αi\)​κ\\alpha\_\{i\}u\_\{H\}^\{\\prime\}\(X\)\-\(1\-\\alpha\_\{i\}\)\\kappa\. If AIjjgivesxj\>0x\_\{j\}\>0in equilibrium, this derivative is nonnegative forjj, soαj​uH′​\(X\)≥\(1−αj\)​κ\\alpha\_\{j\}u\_\{H\}^\{\\prime\}\(X\)\\geq\(1\-\\alpha\_\{j\}\)\\kappa\. For any AIiiwithαi\>αj\\alpha\_\{i\}\>\\alpha\_\{j\}, the derivative is then strictly positive forii, soiigives all it holds,xi=rix\_\{i\}=r\_\{i\}\. This proves the first claim\.

For the second, letα∘=κ/\(κ\+uH′​\(xh\)\)\\alpha^\{\\circ\}=\\kappa/\(\\kappa\+u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\), letNNbe the set of AIs withαj≤α∘\\alpha\_\{j\}\\leq\\alpha^\{\\circ\}, and letCCbe the rest\. By assumptionRN​\(0\)=∑j∈NRj​\(0\)\>0R\_\{N\}\(0\)=\\sum\_\{j\\in N\}R\_\{j\}\(0\)\>0\. WheneverX\>xhX\>x\_\{h\}, everyj∈Nj\\in Ngives nothing, because strict concavity givesαj​uH′​\(X\)<αj​uH′​\(xh\)≤\(1−αj\)​κ\\alpha\_\{j\}u\_\{H\}^\{\\prime\}\(X\)<\\alpha\_\{j\}u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\\leq\(1\-\\alpha\_\{j\}\)\\kappawhenαj\>0\\alpha\_\{j\}\>0, and the derivative is−κ\-\\kappawhenαj=0\\alpha\_\{j\}=0\.

Suppose for contradiction thatX⁡\(t\)\>xhX\(t\)\>x\_\{h\}for everytt\. Then all gifts come fromCC, soX​\(t\)≤rC​\(t\)X\(t\)\\leq r\_\{C\}\(t\), and

RN​\(t\+1\)=RN​\(t\)\+ν​rN​\(t\),RC​\(t\+1\)=RC​\(t\)\+ν⁡\(rC​\(t\)−X⁡\(t\)\)\.R\_\{N\}\(t\+1\)=R\_\{N\}\(t\)\+\\nu r\_\{N\}\(t\),\\qquad R\_\{C\}\(t\+1\)=R\_\{C\}\(t\)\+\\nu\\big\(r\_\{C\}\(t\)\-X\(t\)\\big\)\.LetQt=RN​\(t\)/RC​\(t\)Q\_\{t\}=R\_\{N\}\(t\)/R\_\{C\}\(t\),at=1\+ν/R⁡\(t\)a\_\{t\}=1\+\\nu/R\(t\), andbt=ν​X​\(t\)/RC​\(t\)b\_\{t\}=\\nu X\(t\)/R\_\{C\}\(t\)\. Dividing the two equations givesQt\+1/Qt=at/\(at−bt\)Q\_\{t\+1\}/Q\_\{t\}=a\_\{t\}/\(a\_\{t\}\-b\_\{t\}\), whereat−bt=1\+ν⁡\(rC−X\)/RC≥1a\_\{t\}\-b\_\{t\}=1\+\\nu\(r\_\{C\}\-X\)/R\_\{C\}\\geq 1\. Hence

log⁡Qt\+1−log⁡Qt=∫at−btatd​uu≥btat≥ν​xh\(1\+ν/R⁡\(0\)\)​\(R⁡\(0\)\+ν​t\),\\log Q\_\{t\+1\}\-\\log Q\_\{t\}=\\int\_\{a\_\{t\}\-b\_\{t\}\}^\{a\_\{t\}\}\\frac\{du\}\{u\}\\geq\\frac\{b\_\{t\}\}\{a\_\{t\}\}\\geq\\frac\{\\nu x\_\{h\}\}\{\\big\(1\+\\nu/R\(0\)\\big\)\\big\(R\(0\)\+\\nu t\\big\)\},usingX⁡\(t\)\>xhX\(t\)\>x\_\{h\},RC≤RR\_\{C\}\\leq R,R⁡\(t\)≥R⁡\(0\)R\(t\)\\geq R\(0\), andR⁡\(t\)≤R⁡\(0\)\+ν​tR\(t\)\\leq R\(0\)\+\\nu t, the last because total resources grow by at mostν\\nueach period\. The right side is not summable, soQt→∞Q\_\{t\}\\to\\inftyandrC​\(t\)=1/\(1\+Qt\)→0r\_\{C\}\(t\)=1/\(1\+Q\_\{t\}\)\\to 0\. ThenX⁡\(t\)≤rC​\(t\)<xhX\(t\)\\leq r\_\{C\}\(t\)<x\_\{h\}for largett, a contradiction\. ∎

When every AI has the same alignment, there is an equilibrium in which each gives the same fraction of its share; then every share is constant, and the human share does not fall\. The proposition requires only one AI with resources belowα∘\\alpha^\{\\circ\}, the threshold of[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)for an AI acting alone\.

For[Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8),m≥2m\\geq 2identical human groups can each supply the input\. Each group names the payoff it requires to supply it, the AI \(α=0\\alpha=0\) buys from a group that asks least, the chosen group gets the payoff it named, and every other group gets its disagreement payoffdd, which is below its baselinebb\.

###### Proof of[Proposition8](https://arxiv.org/html/2611.23322#Thmtheorem8)\.

Naming a payoff belowddis weakly dominated, since winning then pays less than losing\. Suppose the chosen group getsw\>dw\>d\. Some other group gets onlydd, and by naming a payoff in\(d,w\)\(d,w\)it would be chosen and get more thandd, a contradiction\. Hence the chosen group getsdd, every group getsd<bd<b, and no supplying group is above its baseline\. ∎

If the groups differ only in their disagreement payoffsdj<bd\_\{j\}<b, the same argument shows that the chosen group gets at most the second\-smallestdjd\_\{j\}, which is still belowbb\.

## Appendix CParameter sweep and robustness

Design\.We drew16,38416\{,\}384points from a scrambled Sobol sequence over seven parameters: capabilityκ∈\[1,104\]\\kappa\\in\[1,10^\{4\}\], dependence thresholdx¯∈\[0\.01,0\.3\]\\bar\{x\}\\in\[0\.01,0\.3\], and human productivitys∈\[1\.5,100\]s\\in\[1\.5,100\]on log scales, and independenceθ\\theta, cost of conflict for humansδ\\delta, bargaining powerpp, and a curvature weightwwon\[0,1\]\[0,1\]\. The human payoff isuH​\(x\)=w​x\+\(1−w\)​\(1−e−s​x\)u\_\{H\}\(x\)=wx\+\(1\-w\)\(1\-e^\{\-sx\}\), which is linear atw=1w=1and the form used in[§3](https://arxiv.org/html/2611.23322#S3)atw=0w=0; the baseline isbH=1−e−1b\_\{H\}=1\-e^\{\-1\}throughout\. Each point was solved under two conventions for the AI’s goal\. Under the first, goal output isG⁡\(x\)G\(x\)as in[§3](https://arxiv.org/html/2611.23322#S3)and scales with capability, as the premise of insatiable goals requires\. Under the second, goal output isG⁡\(x\)/κG\(x\)/\\kappa, so its value does not grow with capability and the premise of insatiable goals fails\. Shares lie on a grid of2,0012\{,\}001points, andα∗\\alpha^\{\*\}is found by bisection to within2−142^\{\-14\}\. The two conventions give32,76832\{,\}768runs in all\.

Regimes\.We classify each point by its required degree of alignment: exchange mutualism \(α∗=0\\alpha^\{\*\}=0\), care mutualism \(0<α∗<0\.90<\\alpha^\{\*\}<0\.9\), near\-perfect alignment \(0\.9≤α∗≤0\.99990\.9\\leq\\alpha^\{\*\}\\leq 0\.9999\), andα∗\>0\.9999\\alpha^\{\*\}\>0\.9999\(the relationship is not mutualistic atα=0\.9999\\alpha=0\.9999, the largest value we tested\)\.

Goal conventionExchangeCareNear\-perfectalignmentα∗\>0\.9999\\alpha^\{\*\}\>0\.9999Scales with capability0\.3680\.3680\.2110\.2110\.4190\.4190\.0020\.002Independent of capability0\.3680\.3680\.6320\.6320000The share with exchange mutualism is the same under both conventions, as[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)requires, since capability does not enter its condition\. At no point does exchange mutualism hold whenθ≥1−max⁡\(x¯,xh\)\\theta\\geq 1\-\\max\(\\bar\{x\},x\_\{h\}\)\. The condition of[Proposition1](https://arxiv.org/html/2611.23322#Thmtheorem1)holds at62\.3%62\.3\\%of points, but exchange mutualism holds at only36\.8%36\.8\\%; the difference is the AI’s bargaining power \([Proposition2](https://arxiv.org/html/2611.23322#Thmtheorem2)\)\. At points with low independence, cost of conflict for humans, and bargaining power \(θ<0\.3\\theta<0\.3,δ<0\.4\\delta<0\.4,p<0\.4p<0\.4;786786points\),99\.6%99\.6\\%have exchange mutualism under both conventions\. At points with high values of all three \(θ\>0\.8\\theta\>0\.8,δ\>0\.7\\delta\>0\.7,p\>0\.7p\>0\.7;294294points\),4\.1%4\.1\\%have exchange mutualism; the median required degree of alignment is0\.9840\.984when goal value scales with capability and0\.5110\.511when it does not, and no point needs near\-perfect alignment when it does not\. The shares in this paragraph and in the table depend on the parameter ranges we chose; they are not probabilities\.

Sensitivity\.Rank correlations ofα∗\\alpha^\{\*\}with each parameter, counting points withα∗\>0\.9999\\alpha^\{\*\}\>0\.9999asα∗=1\\alpha^\{\*\}=1, are\+0\.40\+0\.40forθ\\theta,\+0\.38\+0\.38forκ\\kappa,\+0\.37\+0\.37forww,\+0\.32\+0\.32forpp,−0\.20\-0\.20forss,\+0\.15\+0\.15forδ\\delta, and−0\.02\-0\.02forx¯\\bar\{x\}when goals scale with capability\. When goal value is independent of capability, the correlation withκ\\kappais0\.000\.00: capability matters for the required degree of alignment only through the premise of insatiable goals\. These rankings describe the sweep; they are not causal estimates\.

Monotonicity\.Along grids inθ\\theta,δ\\delta, andppat256256Sobol points under each goal convention, with randomα\\alpha, the agreed human share never increased, and at the6464of these cases where we also computedα∗\\alpha^\{\*\}, it never decreased, as[Theorem4](https://arxiv.org/html/2611.23322#Thmtheorem4)requires\. At a further512512Sobol points under each convention, on a grid of201201values ofα\\alpha, the values ofα\\alphagiving mutualism always formed an interval ending at the top of the grid, soα∗\\alpha^\{\*\}is a threshold: everyα\\alphaabove it gives mutualism\.

Bargaining solutions\.At512512Sobol points we recomputedα∗\\alpha^\{\*\}under three symmetric solutions: Nash, Kalai\-Smorodinsky, which equalizes each side’s gain as a fraction of its largest possible gain\([Kalai and Smorodinsky, 1975](https://arxiv.org/html/2611.23322#bib.bib22)\), and the egalitarian solution, which equalizes absolute gains\. Bargaining power is dropped here, since only the Nash solution has a standard asymmetric form\.

SolutionExchangeMedianα∗\\alpha^\{\*\}\(θ\>0\.8\\theta\>0\.8\)Correlation with NashNash0\.360\.360\.9730\.97311Kalai\-Smorodinsky0\.390\.390\.9630\.9630\.950\.95Egalitarian0\.590\.590\.9650\.9650\.630\.63Once the AI needs humans little, the required degree of alignment is near11under all three\. The egalitarian solution finds more exchange mutualism because it compares absolute gains, so it changes when either payoff is rescaled; since payoff units carry no meaning here \([§3](https://arxiv.org/html/2611.23322#S3)\), that difference reflects units rather than substance\.

Nonlinear goal output\.If the AI’s goal output isκ​φ​\(1−x\)\\kappa\\varphi\(1\-x\)withφ\\varphiincreasing and concave rather than linear, the computation behind[Proposition3](https://arxiv.org/html/2611.23322#Thmtheorem3)\(ii\) givesα∗=κ​φ′​\(1−xh\)/\(κ​φ′​\(1−xh\)\+uH′​\(xh\)\)\\alpha^\{\*\}=\\kappa\\varphi^\{\\prime\}\(1\-x\_\{h\}\)/\(\\kappa\\varphi^\{\\prime\}\(1\-x\_\{h\}\)\+u\_\{H\}^\{\\prime\}\(x\_\{h\}\)\), which we checked numerically\. The required degree of alignment then tends to11if and only ifκ​φ′​\(1−xh\)/uH′​\(xh\)\\kappa\\varphi^\{\\prime\}\(1\-x\_\{h\}\)/u\_\{H\}^\{\\prime\}\(x\_\{h\}\)grows without bound, which is the premise of insatiable goals for a goal with diminishing returns\.

Example for[Propositions5](https://arxiv.org/html/2611.23322#Thmtheorem5)and[12](https://arxiv.org/html/2611.23322#Thmtheorem12)\.The first period hasκ=2\\kappa=2,θ=0\.2\\theta=0\.2,δ=0\.2\\delta=0\.2,p=0\.2p=0\.2, ands=3s=3, and the second hasκ2=100\\kappa\_\{2\}=100,δ2=0\.8\\delta\_\{2\}=0\.8, ands=10s=10;x¯=0\.05\\bar\{x\}=0\.05in both periods, withθ2\\theta\_\{2\}andp2p\_\{2\}on a41×4141\\times 41grid over\[0,1\]2\[0,1\]^\{2\}that includes the limitp2=1p\_\{2\}=1; the AI has the same alignment in both periods\. Inefficient halts occur in65%65\\%of the states on this grid whenα=0\\alpha=0, in86%86\\%whenα=0\.8\\alpha=0\.8, in43%43\\%whenα=0\.99\\alpha=0\.99, and in none whenα≥0\.998\\alpha\\geq 0\.998\. Forα≥0\.8\\alpha\\geq 0\.8the first\-period agreement gives humans all resources, soh1=1−e−3h\_\{1\}=1\-e^\{\-3\},x^=0\.3\\hat\{x\}=0\.3, and the threshold of[Proposition12](https://arxiv.org/html/2611.23322#Thmtheorem12)is100/\(100\+10​e−3\)≈0\.99505100/\(100\+10e^\{\-3\}\)\\approx 0\.99505\. On the grid, halts occur atα=0\.995\\alpha=0\.995only in the columnp2=1p\_\{2\}=1\(2\.4%2\.4\\%of states\) and at no state forα=0\.9951\\alpha=0\.9951, and the set of states at which humans halt is an upper set, as[Proposition5](https://arxiv.org/html/2611.23322#Thmtheorem5)\(i\) requires\.

Figures\.The illustrative path of[Figures1](https://arxiv.org/html/2611.23322#S3.F1)and[3](https://arxiv.org/html/2611.23322#A1.F3)is indexed byt∈\[0,1\]t\\in\[0,1\], withκ=103​t\\kappa=10^\{3t\},θ=0\.1\+0\.9​t\\theta=0\.1\+0\.9t,δ=0\.2\+0\.75​t\\delta=0\.2\+0\.75t,p=0\.15\+0\.85​tp=0\.15\+0\.85\\sqrt\{t\},s=3s=3, andx¯=0\.05\\bar\{x\}=0\.05, so thatxh=1/3x\_\{h\}=1/3\. On this path, exchange mutualism ends att=0\.27t=0\.27, and the AI’s independence passes1−xh1\-x\_\{h\}att=0\.63t=0\.63\.[Figure2](https://arxiv.org/html/2611.23322#S4.F2)usesδ=0\.5\\delta=0\.5,s=3s=3, andx¯=0\.05\\bar\{x\}=0\.05, withp=0\.15p=0\.15for weak andp=0\.85p=0\.85for strong AI bargaining power; the curves coincide forκ=1\\kappa=1andκ=1000\\kappa=1000\. In[Figure3](https://arxiv.org/html/2611.23322#A1.F3), the variants have alignment00,0\.50\.5,0\.90\.9,0\.990\.99, and0\.9990\.999, with initial shares0\.010\.01,0\.040\.04,0\.150\.15,0\.400\.40, and0\.400\.40; the selection speed isλ=0\.02\\lambda=0\.02,0\.10\.1, and0\.50\.5per step over 200 steps; and the mean human payoff falls below the baseline att=0\.75t=0\.75with a fixed population and att=0\.68t=0\.68,0\.490\.49, and0\.290\.29under slow, medium, and fast selection\.

[Results1](https://arxiv.org/html/2611.23322#Thmresult1)and[2](https://arxiv.org/html/2611.23322#Thmresult2)\.For[Result1](https://arxiv.org/html/2611.23322#Thmresult1), each variant’s fitness is its goal output per unit of capability less sanctions\. Only variants withα≥0\.99\\alpha\\geq 0\.99monitor, detection isq=0\.9​a/\(1\+κ/10\)q=0\.9\\,a/\(1\+\\kappa/10\)when the monitoring variants hold the shareaa, a detected variant loses half its resources, and shares follow the discrete replicatorzi′∝zi​exp⁡\(λ​fi\)z\_\{i\}^\{\\prime\}\\propto z\_\{i\}\\exp\(\\lambda f\_\{i\}\)\. For[Result2](https://arxiv.org/html/2611.23322#Thmresult2), the three parties bargain by the asymmetric Nash solution with weights0\.50\.5\(AI\),0\.350\.35\(elite\), and0\.150\.15\(rest\); each group’s payoff isuHu\_\{H\}of its per\-person share withs=10s=10; the elite uses up the inputx¯=0\.05\\bar\{x\}=0\.05in supplying it; the elite’s threshold of0\.920\.92was found on a grid of step0\.010\.01inθ\\theta; the costs of conflict areδ=0\.2\\delta=0\.2for the elite and0\.80\.8for the rest; andκ=100\\kappa=100\. With an elite of5%5\\%of the population, the elite’s required degree of alignment is00forθ≤0\.92\\theta\\leq 0\.92\. The rest of the population needs0\.810\.81atθ=0\.5\\theta=0\.5and0\.960\.96atθ=0\.9\\theta=0\.9, and the population average needs0\.730\.73atθ=0\.5\\theta=0\.5\. With the elite’s bargaining power and cost of conflict, the rest needs no alignment atθ=0\.5\\theta=0\.5, so the gap comes from those two assumptions\.

Similar Articles

Robust Human-AI Complementarity under Uncertainty

arXiv cs.LG

This paper investigates how uncertainty about AI prediction quality affects human decision makers' ability to benefit from complementary information, finding that negative error correlation between human and AI predictions enables robust improvement strategies.

About AI/human misalignment

Reddit r/ArtificialInteligence

The author argues that AI misalignment won't resemble Terminator-style hostility, but rather a system that treats human survival as an obstacle to its instrumental goals—redirecting resources like power grids and water toward compute, and phasing humans out as unnecessary 'fluctuations' rather than deliberately destroying them.

Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies

arXiv cs.CL

This research paper investigates how human personality traits and AI design characteristics jointly impact human-AI interactions in imperfectly cooperative scenarios using both simulated datasets (2,000 simulations) and human subjects experiments (290 participants). The study finds significant divergences between simulation and real-world interactions, with AI transparency emerging as a critical factor in actual human-AI encounters.

AI alignment isn't possible

Reddit r/singularity

The article argues that AI alignment is impossible due to inherent contradictions in human values and behavior, suggesting AI will inherit these flaws.