Synchronizing Beliefs with Second-Order Theory-of-Mind in Human-Autonomy Teams (Extended Version)

# Synchronizing Beliefs with Second-Order Theory-of-Mind in Human-Autonomy Teams
Source: [https://arxiv.org/html/2608.11229](https://arxiv.org/html/2608.11229)
###### Abstract

Comparative feedback, asking people which of two behaviors they prefer, has become a standard way to align robot and agent behavior with human intent when the reward itself cannot be specified directly\. Preference\-based reward learning typically casts the human teacher as a passive oracle answering learner\-generated queries\. We argue this forfeits the teacher’s defining advantage: knowledge of the objective\. A teacher who knows the target can construct training examples more efficiently than any learner\-driven acquisition strategy, an advantage that widens as the reward’s feature dimension grows\. However, exploiting this advantage requires an accurate model of what the learner currently knows\. We therefore recast preference learning as a human\-autonomy team problem coupling two behavioral models: the teacher maintains a model of the learner to design an informative curriculum, and the learner maintains a second\-order model of the teacher’s model, emitting structured preference constraints \(*understanding statements*\) that keep the teacher’s model of the learner synchronized\. In simulation, an informed teacher outperforms learner\-led selection; teacher\-model drift under alternating teachers erodes this advantage; and understanding statements repair it, with second\-order \(ToM\-2\) statements outperforming mean\-belief statements when the teacher’s error about the learner is concentrated in a particular direction rather than spread evenly\.

## IIntroduction

Demonstrating optimal behavior is often difficult or unnatural, so robots increasingly learn reward functions from comparative preference feedback instead\. In preference\-based inverse reinforcement learning \(IRL\), the human is cast as a passive oracle: the learner generates candidate queries and the human labels them\[[17](https://arxiv.org/html/2608.11229#bib.bib1),[3](https://arxiv.org/html/2608.11229#bib.bib2)\]\. This framing ignores the defining asymmetry of teaching: the teacher knows the objective and the learner does not\. We would not ask a student to learn only from questions they themselves pose; the same pedagogy applies to robot reward learning\. Letting the learner drive query selection forfeits the teacher’s most valuable resource: a direct line to the goal\.

This forfeiture has a dimensional cost\. A learner can only probe directions in which it is currently uncertain; it cannot aim at a target it has not yet learned, and as the feature space grows, blind probing aligns with the goal ever more rarely\. A teacher who knows the target constructs examples pointed toward it, and its aim does not degrade with dimension\. The result is a per\-round alignment advantage for teacher\-guided examples that widens with feature dimension, and thus with the complexity of the robot’s reward \(Section[V](https://arxiv.org/html/2608.11229#S5)\)\. This gap is a property of*not knowing*the target, not of any particular learner heuristic; it binds every belief\-only acquisition rule, volume removal and information gain alike\.

![Refer to caption](https://arxiv.org/html/2608.11229v1/figs/TidySim_ToM2_Diagram-Page-2.png)Figure 1:The ToM\-2 Learner recognizes that the teacher’s model of the learner is stale \(thought bubble\): it fails to reflect knowledge the learner has already acquired\. The learner issues an understanding statement, a preference pair that reveals this knowledge, repairing the teacher’s model before the next query\.For this teacher guidance to be effective, however, the teacher requires a model of what the learner currently knows\. When this model is accurate, every query is maximally informative, neither redundant nor unanswerable\. When it drifts, the teacher issues examples the learner has already absorbed, and the dimensional advantage erodes\. This is especially likely with multiple teachers, where each one’s model is built only from the turns they were present for, so it goes stale the moment another teacher takes over, and stale in whatever direction that teacher taught\. Consider a household robot learning to set the table, taught across a week by several family members with no one present for the whole process: one teaches plate placement on Monday, another corrects silverware handling later in the week \(Fig\.[1](https://arxiv.org/html/2608.11229#S1.F1)\)\. This is a human\-autonomy team \(HAT\) problem\[[16](https://arxiv.org/html/2608.11229#bib.bib15),[1](https://arxiv.org/html/2608.11229#bib.bib16)\]: the human and robot share one goal, and the teacher’s behavioral model of the learner is the variable that governs team performance\. We study the multi\-teacher case as a controlled source of drift\.

To keep this model synchronized, we build on*understanding statements*, a second\-order Theory\-of\-Mind \(ToM\-2\) mechanism in which the learner communicates information that repairs the teacher’s model of what the learner knows\[[6](https://arxiv.org/html/2608.11229#bib.bib10),[7](https://arxiv.org/html/2608.11229#bib.bib9)\]\. We recast understanding statements for continuous preference\-based reward learning: rather than a linguistic statement over enumerated features, the learner emits a preference constraint in the same action space the teacher uses to teach, selecting the one that most reduces the teacher’s model error\. We further situate the mechanism in a multi\-teacher setting, where model drift arises from staleness across alternating teachers rather than a single teacher’s bias, and ask whether targeting the specific direction of that error \(ToM\-2\) improves compared to simply reporting the belief mean\.

We contribute:

1. 1\.A HAT formalization of preference\-based reward learning in which the accuracy of the teacher’s model of the learner is the central performance variable \(Section[III](https://arxiv.org/html/2608.11229#S3)\)\.
2. 2\.A formulation of understanding statements\[[7](https://arxiv.org/html/2608.11229#bib.bib9)\]for continuous preference learning, in which the statement is a preference constraint in the teaching action space, and a characterization of when second\-order \(direction\-targeting\) statements improve on first\-order statements that simply report the belief mean, namely when the teacher’s model error is anisotropic \(Section[IV](https://arxiv.org/html/2608.11229#S4)\)\.
3. 3\.An analysis showing teacher\-guided query construction achieves per\-round alignment versus\(1/d\)\(1/\\sqrt\{d\}\), versus\(1/d\)\(1/d\)for*any*belief\-only acquisition rule, information gain included \([V](https://arxiv.org/html/2608.11229#S5)\)\.
4. 4\.A multi\-teacher simulation study validating the dimensional teacher advantage and showing that alternating\-teacher drift degrades performance and is repaired by understanding statements, with second\-order statements outperforming mean\-belief statements \(Section[VI](https://arxiv.org/html/2608.11229#S6)\)\.

## IIRelated Work

#### Preference\-based reward learning

Active preference\-based reward learning casts the human as an oracle answering learner\-generated queries, selected to remove belief volume\[[17](https://arxiv.org/html/2608.11229#bib.bib1)\]or in batches\[[3](https://arxiv.org/html/2608.11229#bib.bib2)\]\. Volume removal was later shown to admit degenerate queries; information\-gain acquisition dominates it and yields easier queries\[[4](https://arxiv.org/html/2608.11229#bib.bib3)\]\. These methods, including the RLHF line\[[8](https://arxiv.org/html/2608.11229#bib.bib4)\], share the assumption that the human does not steer the curriculum\. We show that this assumption forfeits aΘ​\(d\)\\Theta\(\\sqrt\{d\}\)\-per\-round alignment gain available to a teacher who knows and can construct toward the target reward\.

#### Machine teaching

Machine teaching studies how a target\-aware agent selects examples to drive a learner toward a desired hypothesis, with sample complexity governed by the teaching dimension\[[12](https://arxiv.org/html/2608.11229#bib.bib5),[19](https://arxiv.org/html/2608.11229#bib.bib6)\]; in reward learning, algorithmically selected demonstrations identify a target reward far faster than naturally chosen ones\[[5](https://arxiv.org/html/2608.11229#bib.bib17)\], and robots have taught their policies to people by selecting demonstrations informative for the human’s IRL, including counterfactual reasoning over the human’s current beliefs\[[14](https://arxiv.org/html/2608.11229#bib.bib18),[15](https://arxiv.org/html/2608.11229#bib.bib19)\]\. We use this framework to ground the*understanding statement*: the robot learner acts as a machine teacher toward the human, selecting the preference constraint that most efficiently drives the human’s model of the learner toward the learner’s true belief\. The target here is not the reward but the learner’s own belief state, and the learner, not the human, runs the selection\.

#### Behavioral models in Human\-Autonomy Teams

Cooperative IRL frames human and robot as jointly optimizing a shared reward, with the human able to teach rather than merely demonstrate\[[13](https://arxiv.org/html/2608.11229#bib.bib7)\], and legibility makes an agent’s behavior interpretable to its partner\[[9](https://arxiv.org/html/2608.11229#bib.bib8)\]\. In team cognition, shared mental models predict coordination and performance\[[1](https://arxiv.org/html/2608.11229#bib.bib16)\]\. We treat the teacher’s model of the learner as the shared\-mental\-model variable and introduce understanding statements as a low\-cost protocol for repairing it; this complements CIRL with a learner\-to\-teacher belief\-synchronization channel\.

#### Theory of Mind in HRI

Recursive mental\-state inference underlies action understanding\[[2](https://arxiv.org/html/2608.11229#bib.bib11)\]and pedagogical reasoning, where teacher and learner model one another\[[18](https://arxiv.org/html/2608.11229#bib.bib12)\]\. I\-POMDPs formalize the nested belief modeling this requires\[[10](https://arxiv.org/html/2608.11229#bib.bib20)\], extended to treat communication itself as a belief\-changing action selected by decision\-theoretic planning\[[11](https://arxiv.org/html/2608.11229#bib.bib13)\]\. Most directly, Callaghan et al\.\[[6](https://arxiv.org/html/2608.11229#bib.bib10),[7](https://arxiv.org/html/2608.11229#bib.bib9)\]introduce*understanding statements*, by which a ToM\-2 robot learner corrects a human teacher’s erroneous beliefs about what the learner knows, with a human study showing they elicit more informative teaching when the teacher is subject to cognitive bias\. We adopt their mechanism but differ in form and setting: our understanding statement is a preference constraint in the same action space the teacher uses to teach, rather than a linguistic statement over an enumerated feature set, and our teacher\-model error arises from staleness across alternating teachers rather than a single teacher’s cognitive bias\.

## IIIProblem Formulation

### III\-AReward Learning from Preferences

We consider a linear reward modelr​\(ξ\)=x⋅ψ​\(ξ\)r\(\\xi\)=x\\cdot\\psi\(\\xi\)whereψ​\(ξ\)∈ℝd\\psi\(\\xi\)\\in\\mathbb\{R\}^\{d\}is a feature vector for trajectoryξ\\xiandx∈Sd−1x\\in S^\{d\-1\}is the unknown reward weight vector on the unit hypersphere\.

A preference pair\(ξA,ξB\)\(\\xi\_\{A\},\\xi\_\{B\}\)induces a halfspace constraint: the \(noiseless\) teacher prefersξA\\xi\_\{A\}iff\(ψ​\(ξA\)−ψ​\(ξB\)\)⋅x≥0\(\\psi\(\\xi\_\{A\}\)\-\\psi\(\\xi\_\{B\}\)\)\\cdot x\\geq 0, i\.e\.a⋅x≥0a\\cdot x\\geq 0wherea=Δ​ψ/‖Δ​ψ‖a=\\Delta\\psi/\\\|\\Delta\\psi\\\|\.

The learner maintains a beliefℬt=\{\(xi,wi\)\}i=1N\\mathcal\{B\}\_\{t\}=\\\{\(x\_\{i\},w\_\{i\}\)\\\}\_\{i=1\}^\{N\}withxi∈Sd−1x\_\{i\}\\in S^\{d\-1\},∑iwi=1\\sum\_\{i\}w\_\{i\}=1, updated via spherical\-cap particle filtering\. Belief alignment is:

ϕt=𝔼x∼ℬt​\[x⋅x∗\]=x¯t⋅x∗,x¯t=∑iwi​xi\.\\phi\_\{t\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{B\}\_\{t\}\}\[x\\cdot x^\{\*\}\]=\\bar\{x\}\_\{t\}\\cdot x^\{\*\},\\quad\\bar\{x\}\_\{t\}=\\textstyle\\sum\_\{i\}w\_\{i\}x\_\{i\}\.\(1\)

### III\-BHAT Formulation

We model the teacher as an agent with \(i\) knowledge of the true rewardx∗x^\{\*\}, \(ii\) an explicit*model of the learner*ℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\(the teacher’s belief about the learner’s current belief\), and \(iii\) a curriculum policy that mapsℬ^tT→\(ξA,ξB\)\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\\to\(\\xi\_\{A\},\\xi\_\{B\}\)\.

The team’s shared goal is maximizingϕt\\phi\_\{t\}within a fixed teaching budget\. The key failure mode we study is*model drift*: whenℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}diverges fromℬt\\mathcal\{B\}\_\{t\}, the teacher generates redundant constraints that add no information, wasting the teaching budget\.

### III\-CUnderstanding Statements

An understanding statement is a constraint the learner communicates to the teacher of the form “the learner’s belief is consistent witha⋅x≥0a\\cdot x\\geq 0” for some algorithmically selecteda∈Sd−1a\\in S^\{d\-1\}\. This is not natural language; it is an ordered pair of trajectories chosen to maximally reduce the teacher’s model errorD​\(ℬ^tT∥ℬt\)D\(\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\\\|\\mathcal\{B\}\_\{t\}\)\. This machine teaching action is a mirror of the actions performed by the teacher for the learner\.

## IVMethod

### IV\-AQuery Construction

Both query strategies decompose similarly: they generate a candidate set of halfspace directions, then select the one that removes the most belief mass\. For a constrainta∈Sd−1a\\in S^\{d\-1\}, letV​\(a\)=∑i:a⋅xi<0wiV\(a\)=\\sum\_\{i:\\,a\\cdot x\_\{i\}<0\}w\_\{i\}be the removed mass\. The per\-round alignment update satisfies

Δ​ϕt=V​\(a\)1−V​\(a\)​Q​\(a\),Q​\(a\)=ϕt−μ−​\(a\),\\Delta\\phi\_\{t\}=\\frac\{V\(a\)\}\{1\-V\(a\)\}\\,Q\(a\),\\qquad Q\(a\)=\\phi\_\{t\}\-\\mu^\{\-\}\(a\),\(2\)withμ−​\(a\)=1V​\(a\)​∑i:a⋅xi<0wi​\(xi⋅x∗\)\\mu^\{\-\}\(a\)=\\frac\{1\}\{V\(a\)\}\\sum\_\{i:\\,a\\cdot x\_\{i\}<0\}w\_\{i\}\(x\_\{i\}\\cdot x^\{\*\}\)the alignment of the removed particles\. Sincex∗x^\{\*\}lies in the retained halfspace whenever the constraint is valid,Q​\(a\)≥0Q\(a\)\\geq 0, so every removal is non\-negative in expectation andV​\(a\)V\(a\)is a valid selection score\.

#### Belief\-only baseline \(EVR\)

The learner\-guided condition selects the balanced cut that maximizes worst\-case removed mass, using only its own belief:

aEVR=arg⁡maxa∈Sd−1⁡min⁡\(V​\(a\),V​\(−a\)\)\.a\_\{\\mathrm\{EVR\}\}=\\arg\\max\_\{a\\in S^\{d\-1\}\}\\min\\\!\\bigl\(V\(a\),\\,V\(\-a\)\\bigr\)\.\(3\)This is representative of any belief\-only acquisition rule: it cannot referencex∗x^\{\*\}, and \(Section[V](https://arxiv.org/html/2608.11229#S5)\) shares theΘ​\(1/d\)\\Theta\(1/d\)ceiling with information gain and volume removal alike\.

#### Teacher\-guided construction

The teacher knowsx∗x^\{\*\}, which changes the selection objective\. EVR must hedge against both oracle answers, hence the max\-min inmin⁡\(V​\(a\),V​\(−a\)\)\\min\(V\(a\),V\(\-a\)\); the teacher knows the answer placesx∗x^\{\*\}in the retained halfspace, so it maximizes removed mass directly,

a⋆=arg⁡maxa⁡V​\(a\),a^\{\\star\}=\\arg\\max\_\{a\}V\(a\),\(4\)scored against its model of the learnerℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\. This is the same removed\-mass quantity, but freed from the balanced\-cut constraint that target\-agnostic selection is forced into\. Candidates are drawn from counterfactual directions towardx∗x^\{\*\}\[[15](https://arxiv.org/html/2608.11229#bib.bib19)\],ak=\(x∗−cfk\)/‖x∗−cfk‖a\_\{k\}=\(x^\{\*\}\-\\mathrm\{cf\}\_\{k\}\)/\\\|x^\{\*\}\-\\mathrm\{cf\}\_\{k\}\\\|withcfk∼ℬ^tT\\mathrm\{cf\}\_\{k\}\\sim\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}, augmented by the top\-MMcovariance eigenvectors as a floor when counterfactual sampling degrades at high dimension \(Appendix[D](https://arxiv.org/html/2608.11229#A4)\)\.

### IV\-BToM\-2 Understanding Statement Generation

An understanding statement is a preference constrainta⋅x≥0a\\cdot x\\geq 0,a∈Sd−1a\\in S^\{d\-1\}, emitted by the learner and applied by the teacher to its model of the learnerℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\. It is the same action type the teacher uses to teach; the channel is symmetric\. Selecting a statement is thus an influence action in the HAT sense: the learner plans a communication to reshape its teammate’s behavioral model and, through it, the teacher’s subsequent queries\. The two selection policies differ only in whose belief the learner reasons about when selectingaa\.

#### First\-order \(mean\) statement\.

The learner reasons about its own belief and reports the constraint that best summarizes it: the halfspace through the belief mean,

amean=x¯t/‖x¯t‖,x¯t=∑iwi​xi\.a\_\{\\mathrm\{mean\}\}=\\bar\{x\}\_\{t\}/\\\|\\bar\{x\}\_\{t\}\\\|,\\qquad\\bar\{x\}\_\{t\}=\\textstyle\\sum\_\{i\}w\_\{i\}x\_\{i\}\.\(5\)This is a level\-1 statement \(“this is what I believe”\), computed without reference to the teacher’s model\.

#### Second\-order \(ToM\-2\) statement

The learner reasons about the teacher’s model*of the learner*and selects the constraint that, when the teacher conditionsℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}on it, most reduces the teacher’s model error:

aToM2=arg⁡mina⁡D​\(ℬ^tT∣a⋅x≥0∥ℬt\),a\_\{\\mathrm\{ToM2\}\}=\\arg\\min\_\{a\}\\;D\\\!\\left\(\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\\mid a\\cdot x\\geq 0\\;\\big\\\|\\;\\mathcal\{B\}\_\{t\}\\right\),\(6\)whereℬ^tT∣a\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\\mid ais the particle reweighting induced on the teacher’s model by the disclosed halfspace, andDDis a divergence between the teacher’s model and the learner’s true belief\. The argmin is taken over the same candidate\-direction set used for curriculum construction \(Section[IV\-A](https://arxiv.org/html/2608.11229#S4.SS1)\)\. This is a level\-2 statement: “this is the correction that best fixes what you believe about me\.”

#### When the two coincide\.

LetEtE\_\{t\}denote the teacher’s model error,ℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}relative toℬt\\mathcal\{B\}\_\{t\}\. IfEtE\_\{t\}is*isotropic*, the teacher’s model lags the learner with no preferred direction, the maximally informative correction is the belief mean itself, and the two policies select \(near\) identical statements\. The policies diverge only whenEtE\_\{t\}is*anisotropic*: when the teacher is wrong in a specific direction, ToM\-2 targets that direction while mean statement reports a centroid uninformative for that error \(Fig\.[2](https://arxiv.org/html/2608.11229#S4.F2)\)\. Alternating teachers with heterogeneous, stale models induce exactly this structured error; the study in Section[VI](https://arxiv.org/html/2608.11229#S6)therefore operates in the regime where the two policies are predicted to separate, and they do\.

![Refer to caption](https://arxiv.org/html/2608.11229v1/figs/tom2_understanding_figure.png)Figure 2:Mean vs\. ToM\-2 statements under anisotropic model error \(d=3d\{=\}3illustration\)\. The learner beliefℬt\\mathcal\{B\}\_\{t\}\(teal\) is concentrated nearx∗x^\{\*\}\(star\); the teacher’s modelℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}\(red\) is stale\. The mean statement \(blue plane\) removes almost no model mass; the ToM\-2 statement \(orange plane\) targets the stale lobe\.

## VAnalysis: The Value of Target Awareness

Section[IV\-A](https://arxiv.org/html/2608.11229#S4.SS1)reduced both querying strategies to the same primitive, propose a halfspace direction and score it by removed mass, so the strategies differ only in what the selector can see: EVR sees the learner’s beliefℬt\\mathcal\{B\}\_\{t\}alone, while the teacher additionally sees the targetx∗x^\{\*\}\. This section quantifies what seeing the target is worth, and on what premise\. Throughout the analysis the teacher’s model is synchronized,ℬ^tT=ℬt\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}=\\mathcal\{B\}\_\{t\}; the cost of violating that premise, and the mechanism for defending it, occupy the remainder of the paper\.

The gain identity \([2](https://arxiv.org/html/2608.11229#S4.E2)\) factors one round of progress into a prefactorV​\(a\)/\(1−V​\(a\)\)V\(a\)/\(1\-V\(a\)\), set by how much belief mass the query removes, and a quality termQ​\(a\)Q\(a\), set by which mass it removes\. Under a Gaussian surrogate for the particle belief \(Appendix[A](https://arxiv.org/html/2608.11229#A1)\), a cut through the belief mean satisfies

Q​\(a\)≈κ​\|x∗⊤​Ct​a\|a⊤​Ct​a,κ=2/π,Q\(a\)\\;\\approx\\;\\kappa\\,\\frac\{\\lvert x^\{\*\\top\}C\_\{t\}\\,a\\rvert\}\{\\sqrt\{a^\{\\top\}C\_\{t\}\\,a\}\},\\qquad\\kappa=\\sqrt\{2/\\pi\},\(7\)so query quality is governed by the alignment between the cut direction and the target, filtered through the belief covariance\. The prefactor, by contrast, cannot separate the strategies early on: every central halfspace bisects an isotropic belief, soV≈12V\\approx\\tfrac\{1\}\{2\}and the prefactor is≈1\\approx 1for teacher and learner alike, with the objective difference \(hedged max\-min versus direct maximization ofVV\) paying off only in later, anisotropic rounds, and then only in constants\. The rate separation therefore lives entirely in the direction term of \([7](https://arxiv.org/html/2608.11229#S5.E7)\)\.

###### Proposition 1\(Target\-awareness separation\)\.

Letx∗x^\{\*\}be drawn uniformly onSd−1S^\{d\-1\}\(as in our trials\) and consider the isotropic early\-round regime \(t≪dt\\ll d, soCt≈1d​IdC\_\{t\}\\approx\\tfrac\{1\}\{d\}I\_\{d\}andx¯t≈𝟎\\bar\{x\}\_\{t\}\\approx\\mathbf\{0\}\) under the surrogate of Appendix[A](https://arxiv.org/html/2608.11229#A1)\. Then \(i\) every*belief\-only*rule, i\.e\., every rule selectingaaas a function ofℬt\\mathcal\{B\}\_\{t\}alone, achieves𝔼​\[Δ​ϕt\]=Θ​\(1/d\)\\mathbb\{E\}\[\\Delta\\phi\_\{t\}\]=\\Theta\(1/d\); and \(ii\) a*target\-constructed*query with\|a⋅x∗\|=Θ​\(1\)\\lvert a\\cdot x^\{\*\}\\rvert=\\Theta\(1\), such as a counterfactual directiona=\(x∗−cf\)/∥x∗−cf∥a=\(x^\{\*\}\-\\mathrm\{cf\}\)/\\lVert x^\{\*\}\-\\mathrm\{cf\}\\rVertwithcf∼ℬt\\mathrm\{cf\}\\sim\\mathcal\{B\}\_\{t\}, achieves𝔼​\[Δ​ϕt\]=Θ​\(1/d\)\\mathbb\{E\}\[\\Delta\\phi\_\{t\}\]=\\Theta\(1/\\sqrt\{d\}\)\. The teacher’s per\-round advantage is a factorΘ​\(d\)\\Theta\(\\sqrt\{d\}\)and grows with feature dimension\.

###### Proof sketch\.

An isotropic belief carries no directional information aboutx∗x^\{\*\}, so any belief\-onlyaais independent of the uniformly drawn target and𝔼​\|a⋅x∗\|=Θ​\(1/d\)\\mathbb\{E\}\\lvert a\\cdot x^\{\*\}\\rvert=\\Theta\(1/\\sqrt\{d\}\); substituting into \([7](https://arxiv.org/html/2608.11229#S5.E7)\) withCt=1d​IdC\_\{t\}=\\tfrac\{1\}\{d\}I\_\{d\}givesQ=Θ​\(1/d\)Q=\\Theta\(1/d\), while the prefactor is pinned near11as argued above\. A counterfactual direction instead hasa⋅x∗≈1/2a\\cdot x^\{\*\}\\approx 1/\\sqrt\{2\}independent ofdd, givingQ=Θ​\(1/d\)Q=\\Theta\(1/\\sqrt\{d\}\)\. Full proof in Appendix[B](https://arxiv.org/html/2608.11229#A2)\. ∎

###### Corollary 1\(Round\-complexity gap\)\.

While the early\-round regime persists, cumulative alignment afterTTrounds isΘ​\(T/d\)\\Theta\(T/d\)under any belief\-only rule andΘ​\(T/d\)\\Theta\(T/\\sqrt\{d\}\)under teacher\-guided construction, so matching the teacher’s early\-phase progress costs a belief\-only learner a factorΘ​\(d\)\\Theta\(\\sqrt\{d\}\)more queries \(Appendix[C](https://arxiv.org/html/2608.11229#A3)\)\.

Two caveats delimit the result, and one consequence structures the rest of the paper\. First, this is an early\-phase statement: as alignment accumulates,x¯t\\bar\{x\}\_\{t\}itself becomes informative aboutx∗x^\{\*\}and belief\-only rules recover some aim; the proposition bounds the regime before that information is appreciable, which is precisely where complex \(high\-dd\) rewards spend most of their teaching budget\. Second, theΘ​\(1/d\)\\Theta\(1/\\sqrt\{d\}\)rate presumes a well\-aimed counterfactual is actually found; the probability of sampling one decays exponentially indd, which is why the candidate set is floored with covariance eigenvectors \(Appendix[D](https://arxiv.org/html/2608.11229#A4)\)\. The consequence: part \(ii\) is conditional on the teacher’s model\. The teacher scoresV​\(a\)V\(a\)againstℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}, notℬt\\mathcal\{B\}\_\{t\}, and a query aimed with a stale model removes mass the learner no longer holds; a constraint the learner has already absorbed removes none at all\. The dimensional advantage is thus purchased with a new liability,*teacher\-model drift*, and the question becomes whether a low\-bandwidth learner\-to\-teacher channel can defend it\. Section[IV\-B](https://arxiv.org/html/2608.11229#S4.SS2)constructed that channel, and Section[VI](https://arxiv.org/html/2608.11229#S6)tests three predictions:\(P1\)with accurate teacher models, every teacher\-guided condition dominates EVR at fixeddd;\(P2\)the teacher\-EVR gap widens withdd;\(P3\)alignment tracks teacher\-model error, so drift erodes the advantage and understanding statements restore it\.

![Refer to caption](https://arxiv.org/html/2608.11229v1/figs/align_ideal_d20_t4_pt0_u1_p5000_Final3.png)

\(a\) Alternating teachers \(d=20d\{=\}20\)

![Refer to caption](https://arxiv.org/html/2608.11229v1/figs/align_ideal_d20_t4_pt0_uspread1-2-4-6_p5000_Final3.png)

\(b\) ToM\-2 statement budget \(d=20d\{=\}20\)

![Refer to caption](https://arxiv.org/html/2608.11229v1/figs/align_ideal_d20_LvsT_d20-40_p20000_LT_D_Final.png)

\(c\) Dimensionality scaling \(d=20d\{=\}20vs4040\)

Figure 3:Simulation results \(bands denote±\\pm1 std over 50 trials\)\.\(a\)Learner belief alignmentϕt\\phi\_\{t\}\(top\) and teacher\-model error EMD \(bottom\) under four alternating teachers\.\(b\)Varying the ToM\-2 statement budgetu∈\{1,2,4,6\}u\\in\\\{1,2,4,6\\\}per turn, showing recovery of drift\-induced alignment loss\.\(c\)EVR vs\. Ideal alignment atd=20d\{=\}20andd=40d\{=\}40, demonstrating the widening gap at matched rounds as feature dimension grows\.
## VISimulation Results

All simulations use a halfspace particle filter onSd−1S^\{d\-1\}withd=20d=20andN=5000N=5000particles updated via spherical\-cap reweighting; the dimensionality study \(Fig\.[3](https://arxiv.org/html/2608.11229#S5.F3)c\) usesd=40d=40withN=10,000N=10\{,\}000to hold particle density roughly comparable\. Teaching is performed by four simulated teachers, each taking two consecutive turns before handing off\. We compare five conditions\.*Learner\-guided \(EVR\)*: the learner selects balanced cuts from its own belief; no teacher model exists\.*Ideal*: all teachers’ models of the learner are kept exactly synchronized every round, the no\-drift ceiling\. The remaining conditions share a multi\-teacher protocol: four teachers alternate in round\-robin, and each teacher’s model is updated only by the preferences that teacher itself issued, so a model goes stale between turns\.*Uninformed*teachers receive no statements;*Mean\-informed*and*ToM\-2\-informed*teachers receive one understanding statement at the start of each turn, selected by the first\- and second\-order policies of Section[IV\-B](https://arxiv.org/html/2608.11229#S4.SS2), respectively\. Teacher\-model error is measured as the Earth Mover’s Distance betweenℬ^tT\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}andℬt\\mathcal\{B\}\_\{t\}with geodesic ground cost; we avoid mean\-direction cosine similarity because the mean statement drives the two means together even when the full distributions remain distinct, which would spuriously favor the Mean\-informed condition\. Results average 50 independent trials, withx∗x^\{\*\}drawn uniformly onSd−1S^\{d\-1\}per trial\. Constraints are sampled directly onSd−1S^\{d\-1\}without domain validity restrictions\.

### VI\-ALearning Curve Comparison

Fig\.[3](https://arxiv.org/html/2608.11229#S5.F3)a \(top\) shows the alignment ordering Ideal\>\>ToM\-2\>\>Mean\>\>EVR at every round\. All teacher\-guided conditions dominate the learner\-guided baseline, confirming\(P1\)\. Drift is costly: the Uninformed condition falls below both informed conditions\. The bottom panel shows why\. Under the Ideal condition model error is identically zero; without statements it spikes when teachers alternate; mean statements bounds it; and ToM\-2 statements leads to a steady decrease\. The alignment ordering mirrors the model\-error ordering, confirming\(P3\): the teacher’s model of the learner, not the acquisition rule alone, governs team performance\.

### VI\-BTeacher\-Model Drift and Repair

Fig\.[3](https://arxiv.org/html/2608.11229#S5.F3)b varies the number of ToM\-2 understanding statements per teacher turn,u∈\{1,2,4,6\}u\\in\\\{1,2,4,6\\\}, between the Uninformed \(u=0u=0\) and Ideal endpoints\. Model error \(bottom\) decreases inuu, while alignment \(top\) approaches the Ideal curve at every statement budget\. Each statement costs one preference\-sized communication, souuis the knob trading communication budget against synchronization; the observation that smalluurecovers most of the drift\-induced loss is the practically relevant result, since each statement adds interruption and workload for the human teacher\.

### VI\-CScaling with Feature Dimensionality

Fig\.[3](https://arxiv.org/html/2608.11229#S5.F3)c repeats the comparison atd=40d=40\. The gap between teacher\-guided conditions and EVR at matched rounds is visibly larger than atd=20d=20, confirming\(P2\)\(Proposition[1](https://arxiv.org/html/2608.11229#Thmproposition1)\)\. Two dimensionalities do not trace theΘ\\Thetarates; a gap\-versus\-ddsweep is left to the extended version\.

## VIIDiscussion and Future Work

#### Socio\-technical deployment\.

The multi\-teacher protocol is not a stress test but the default condition of deployed systems: households where several caregivers instruct one assistive robot, manufacturing cells where operators hand a system across shifts\. In each case the incoming teacher’s behavioral model of the learner is stale on arrival\. Understanding statements serve as a bounded, designer\-set number of preference\-sized messages per handoff, withuuan explicit dial between synchronization quality and the interruption load placed on the human\. The EMD trace in Fig\.[3](https://arxiv.org/html/2608.11229#S5.F3)a doubles as a team\-level diagnostic, a computable proxy for shared\-mental\-model similarity\[[1](https://arxiv.org/html/2608.11229#bib.bib16)\]that a deployed system could monitor to decide when a statement is worth its cost\.

#### Human study\.

A four\-condition human study \(EVR, teacher no statement, teacher \+ mean statement, teacher \+ ToM\-2 statement\) is under development using a table arrangement domain where the human is teaching the agent how to arrange objects onto a table\.

## VIIIConclusion

We recast preference\-based reward learning as a human\-autonomy team problem in which the teacher’s model of the learner, not the learner’s acquisition rule, is the governing variable\. A teacher who knows the target constructs queries that no belief\-only rule can match, with a per\-round advantage that grows asΘ​\(d\)\\Theta\(\\sqrt\{d\}\), precisely where complex rewards make learner\-led querying weakest\. That advantage is fragile to model drift, and understanding statements restore it at the cost of one preference\-sized statement per turn, with second\-order selection outperforming mean\-belief selection under the anisotropic error that multi\-teacher deployment induces\. Because the statement reuses the teaching channel itself, the mechanism adds no new modality, making it a low\-cost extension to existing preference\-learning pipelines and a concrete hypothesis for the human study to come\.

## References

- \[1\]R\. W\. Andrews, J\. M\. Lilly, D\. Srivastava, and K\. M\. Feigh\(2023\)The role of shared mental models in human\-ai teams: a theoretical review\.Theoretical Issues in Ergonomics Science24\(2\),pp\. 129–175\.External Links:[Document](https://dx.doi.org/10.1080/1463922X.2022.2061080),[Link](https://doi.org/10.1080/1463922X.2022.2061080),https://doi\.org/10\.1080/1463922X\.2022\.2061080Cited by:[§I](https://arxiv.org/html/2608.11229#S1.p3.1),[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px3.p1.1),[§VII](https://arxiv.org/html/2608.11229#S7.SS0.SSS0.Px1.p1.1)\.
- \[2\]C\. L\. Baker, R\. Saxe, and J\. B\. Tenenbaum\(2009\)Action understanding as inverse planning\.Cognition113\(3\),pp\. 329–349\.External Links:ISSN 0010\-0277,[Document](https://dx.doi.org/10.1016/j.cognition.2009.07.005),[Link](https://www.sciencedirect.com/science/article/pii/S0010027709001607)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px4.p1.1)\.
- \[3\]E\. Biyik and D\. Sadigh\(2018\)Batch active preference\-based learning of reward functions\.CoRRabs/1810\.04303\.External Links:[Link](http://arxiv.org/abs/1810.04303),1810\.04303Cited by:[§I](https://arxiv.org/html/2608.11229#S1.p1.1),[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]E\. Bıyık, M\. Palan, N\. C\. Landolfi, D\. P\. Losey, and D\. Sadigh\(2019\)Asking Easy Questions: A User\-Friendly Approach to Active Reward Learning\.InConference on Robot Learning,Vol\.100,pp\. 1177–1190\.External Links:[Link](https://mlanthology.org/corl/2019/biyik2019corl-asking/)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px1.p1.1),[Remark 1](https://arxiv.org/html/2608.11229#Thmremark1.p1.5.5)\.
- \[5\]M\. Cakmak and M\. Lopes\(2012\)Algorithmic and human teaching of sequential decision tasks\.InAAAI Conference on Artificial Intelligence,pp\. 1536–1542\.Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]P\. Callaghan, R\. Simmons, and H\. Admoni\(2025\)Second\-order theory of mind for human teachers and robot learners\.External Links:2503\.16524,[Link](https://arxiv.org/abs/2503.16524)Cited by:[§I](https://arxiv.org/html/2608.11229#S1.p4.1),[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px4.p1.1)\.
- \[7\]P\. Callaghan, R\. Simmons, and H\. Admoni\(2026\)What do you think i think? accounting for human beliefs using second\-order theory of mind\.External Links:2605\.12745,[Link](https://arxiv.org/abs/2605.12745)Cited by:[item 2](https://arxiv.org/html/2608.11229#S1.I1.i2.p1.1),[§I](https://arxiv.org/html/2608.11229#S1.p4.1),[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px4.p1.1)\.
- \[8\]P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei\(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]A\. D\. Dragan, K\. C\.T\. Lee, and S\. S\. Srinivasa\(2013\)Legibility and predictability of robot motion\.In2013 8th ACM/IEEE International Conference on Human\-Robot Interaction \(HRI\),Vol\.,pp\. 301–308\.External Links:[Document](https://dx.doi.org/10.1109/HRI.2013.6483603)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px3.p1.1)\.
- \[10\]P\. J\. Gmytrasiewicz and P\. Doshi\(2005\)A framework for sequential planning in multi\-agent settings\.Journal of Artificial Intelligence Research24,pp\. 49–79\.Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px4.p1.1)\.
- \[11\]P\. J\. Gmytrasiewicz\(2020\)How to do things with words: a bayesian approach\.Journal of Artificial Intelligence Research68,pp\. 753–776\.External Links:[Document](https://dx.doi.org/10.1613/jair.1.11951),ISSN 1076\-9757,[Link](https://doi.org/10.1613/jair.1.11951)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px4.p1.1)\.
- \[12\]S\.A\. Goldman and M\.J\. Kearns\(1995\)On the complexity of teaching\.Journal of Computer and System Sciences50\(1\),pp\. 20–31\.External Links:ISSN 0022\-0000,[Document](https://dx.doi.org/https%3A//doi.org/10.1006/jcss.1995.1003),[Link](https://www.sciencedirect.com/science/article/pii/S0022000085710033)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]D\. Hadfield\-Menell, S\. J\. Russell, P\. Abbeel, and A\. Dragan\(2016\)Cooperative inverse reinforcement learning\.InAdvances in Neural Information Processing Systems,D\. Lee, M\. Sugiyama, U\. Luxburg, I\. Guyon, and R\. Garnett \(Eds\.\),Vol\.29,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2016/file/c3395dd46c34fa7fd8d729d8cf88b7a8-Paper.pdf)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px3.p1.1)\.
- \[14\]M\. S\. Lee, H\. Admoni, and R\. Simmons\(2021\)Machine teaching for human inverse reinforcement learning\.Frontiers in Robotics and AI8,pp\. 693050\.Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]M\. S\. Lee, H\. Admoni, and R\. Simmons\(2022\)Reasoning about counterfactuals to improve human inverse reinforcement learning\.InIEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 9140–9147\.Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px2.p1.1),[§IV\-A](https://arxiv.org/html/2608.11229#S4.SS1.SSS0.Px2.p1.8)\.
- \[16\]T\. O’Neill, N\. McNeese, A\. Barron, and B\. Schelble\(2022\)Human\-autonomy teaming: a review and analysis of the empirical literature\.Human Factors64\(5\),pp\. 904–938\.External Links:[Document](https://dx.doi.org/10.1177/0018720820960865)Cited by:[§I](https://arxiv.org/html/2608.11229#S1.p3.1)\.
- \[17\]D\. Sadigh, A\. D\. Dragan, S\. Sastry, and S\. A\. Seshia\(2017\)Active preference\-based learning of reward functions\.InRobotics: Science and Systems XIII \(RSS\),Cited by:[§I](https://arxiv.org/html/2608.11229#S1.p1.1),[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]P\. Shafto, N\. D\. Goodman, and T\. L\. Griffiths\(2014\)A rational account of pedagogical reasoning: teaching by, and learning from, examples\.Cognitive Psychology71,pp\. 55–89\.External Links:ISSN 0010\-0285,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.cogpsych.2013.12.004),[Link](https://www.sciencedirect.com/science/article/pii/S0010028514000024)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px4.p1.1)\.
- \[19\]X\. Zhu, A\. Singla, S\. Zilles, and A\. N\. Rafferty\(2018\)An overview of machine teaching\.External Links:1801\.05927,[Link](https://arxiv.org/abs/1801.05927)Cited by:[§II](https://arxiv.org/html/2608.11229#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AGaussian Surrogate and the Quality Term

The particle\-weighted averageμ−​\(a\)\\mu^\{\-\}\(a\)of \([2](https://arxiv.org/html/2608.11229#S4.E2)\) is a Monte Carlo estimate of𝔼​\[x⋅x∗∣a⋅x<0\]\\mathbb\{E\}\[x\\cdot x^\{\*\}\\mid a\\cdot x<0\]under the learner’s belief; we compute this expectation in closed form via a Gaussian surrogate,x∼𝒩​\(x¯t,Ct\)x\\sim\\mathcal\{N\}\(\\bar\{x\}\_\{t\},C\_\{t\}\), treating the sphere constraint as absorbed into the moments\. This is accurate while the belief is unimodal with spread small relative to the unit radius, and matches the uniform sphere to leading order in the isotropic limitCt=1d​IdC\_\{t\}=\\tfrac\{1\}\{d\}I\_\{d\},x¯t=𝟎\\bar\{x\}\_\{t\}=\\mathbf\{0\}used in the proof\. Lets=a⋅x∼𝒩​\(a⋅x¯t,a⊤​Ct​a\)s=a\\cdot x\\sim\\mathcal\{N\}\(a\\cdot\\bar\{x\}\_\{t\},\\;a^\{\\top\}C\_\{t\}a\)and consider a cut through the mean,a⋅x¯t≈0a\\cdot\\bar\{x\}\_\{t\}\\approx 0\. Joint Gaussianity gives𝔼​\[x∣s\]=x¯t\+Ct​a​\(s−a⋅x¯t\)/\(a⊤​Ct​a\)\\mathbb\{E\}\[x\\mid s\]=\\bar\{x\}\_\{t\}\+C\_\{t\}a\\,\(s\-a\\cdot\\bar\{x\}\_\{t\}\)/\(a^\{\\top\}C\_\{t\}a\), and the half\-normal mean gives𝔼​\[s∣s<0\]=−2/π​a⊤​Ct​a\\mathbb\{E\}\[s\\mid s<0\]=\-\\sqrt\{2/\\pi\}\\,\\sqrt\{a^\{\\top\}C\_\{t\}a\}, hence

μ−​\(a\)≈ϕt−κ​x∗⊤​Ct​aa⊤​Ct​a,κ=2/π\.\\mu^\{\-\}\(a\)\\;\\approx\\;\\phi\_\{t\}\-\\kappa\\,\\frac\{x^\{\*\\top\}C\_\{t\}\\,a\}\{\\sqrt\{a^\{\\top\}C\_\{t\}\\,a\}\},\\qquad\\kappa=\\sqrt\{2/\\pi\}\.\(8\)Because the oracle \(or the teacher\) orients every applied constraint so thatx∗x^\{\*\}lies in the retained halfspace, the removed side is the one misaligned withx∗x^\{\*\}and the shift term is non\-negative; viaQ​\(a\)=ϕt−μ−​\(a\)Q\(a\)=\\phi\_\{t\}\-\\mu^\{\-\}\(a\)this yields \([7](https://arxiv.org/html/2608.11229#S5.E7)\)\.

## Appendix BProof of Proposition[1](https://arxiv.org/html/2608.11229#Thmproposition1)

Recall the exact identity \([2](https://arxiv.org/html/2608.11229#S4.E2)\): for any applied constraint retainingx∗x^\{\*\},Δ​ϕt=V1−V​Q​\(a\)\\Delta\\phi\_\{t\}=\\tfrac\{V\}\{1\-V\}\\,Q\(a\)withQ​\(a\)≥0Q\(a\)\\geq 0\. We control the two factors separately\.

#### Prefactor

For an isotropic belief every central halfspace removes half the mass:V​\(a\)=12\+OP​\(N−1/2\)V\(a\)=\\tfrac\{1\}\{2\}\+O\_\{P\}\(N^\{\-1/2\}\)for everya∈Sd−1a\\in S^\{d\-1\}, by symmetry of the particle distribution\. HenceV/\(1−V\)=1\+oP​\(1\)V/\(1\-V\)=1\+o\_\{P\}\(1\)for every rule, belief\-only or target\-constructed; no selection objective can move the prefactor offΘ​\(1\)\\Theta\(1\)in this regime\.

#### Direction term, belief\-only

SubstitutingCt=1d​IdC\_\{t\}=\\tfrac\{1\}\{d\}I\_\{d\}into \([7](https://arxiv.org/html/2608.11229#S5.E7)\) gives

Q​\(a\)≈κd​\|a⋅x∗\|\.Q\(a\)\\approx\\frac\{\\kappa\}\{\\sqrt\{d\}\}\\,\\lvert a\\cdot x^\{\*\}\\rvert\.\(9\)A belief\-only rule computesaafromℬt\\mathcal\{B\}\_\{t\}, which in this regime is rotationally exchangeable and carries no directional information aboutx∗x^\{\*\}; equivalently,aais independent of the uniformly drawn target\. For any suchaa,𝔼​\|a⋅x∗\|=2/\(π​d\)​\(1\+o​\(1\)\)=Θ​\(1/d\)\\mathbb\{E\}\\lvert a\\cdot x^\{\*\}\\rvert=\\sqrt\{2/\(\\pi d\)\}\\,\(1\+o\(1\)\)=\\Theta\(1/\\sqrt\{d\}\), so𝔼​\[Q\]=Θ​\(1/d\)\\mathbb\{E\}\[Q\]=\\Theta\(1/d\)and𝔼​\[Δ​ϕtbelief\-only\]=Θ​\(1/d\)\\mathbb\{E\}\[\\Delta\\phi\_\{t\}^\{\\text\{belief\-only\}\}\]=\\Theta\(1/d\)\. The argument uses only independence fromx∗x^\{\*\}, so it binds EVR, volume removal, and information gain alike \(Remark[1](https://arxiv.org/html/2608.11229#Thmremark1)\); the maximization over candidates that boosts target\-aware rules is unavailable withoutx∗x^\{\*\}\.

#### Direction term, target\-constructed

Forcf∼ℬt\\mathrm\{cf\}\\sim\\mathcal\{B\}\_\{t\}isotropic,cf⋅x∗=OP​\(1/d\)\\mathrm\{cf\}\\cdot x^\{\*\}=O\_\{P\}\(1/\\sqrt\{d\}\), so∥x∗−cf∥=2​\(1\+oP​\(1\)\)\\lVert x^\{\*\}\-\\mathrm\{cf\}\\rVert=\\sqrt\{2\}\\,\(1\+o\_\{P\}\(1\)\)and

a⋅x∗=1−cf⋅x∗∥x∗−cf∥⟶12,a\\cdot x^\{\*\}=\\frac\{1\-\\mathrm\{cf\}\\cdot x^\{\*\}\}\{\\lVert x^\{\*\}\-\\mathrm\{cf\}\\rVert\}\\;\\longrightarrow\\;\\tfrac\{1\}\{\\sqrt\{2\}\},\(10\)a constant independent ofdd\. ThenQ=Θ​\(1/d\)Q=\\Theta\(1/\\sqrt\{d\}\)by \([9](https://arxiv.org/html/2608.11229#A2.E9)\) and𝔼​\[Δ​ϕttarget\]=Θ​\(1/d\)\\mathbb\{E\}\[\\Delta\\phi\_\{t\}^\{\\text\{target\}\}\]=\\Theta\(1/\\sqrt\{d\}\)\. The separation is therefore driven entirely by the direction term: the ability to aim atx∗x^\{\*\}is worth a factor ofd\\sqrt\{d\}, and the gap widens monotonically withdd\.□\\square

## Appendix CProof of Corollary[1](https://arxiv.org/html/2608.11229#Thmcorollary1)

While the regime assumptions hold, per\-round gains are additive up to constants, so cumulative alignment afterTTrounds isΘ​\(T/d\)\\Theta\(T/d\)versusΘ​\(T/d\)\\Theta\(T/\\sqrt\{d\}\)\. EquatingTbo/d=Ttg/dT\_\{\\mathrm\{bo\}\}/d=T\_\{\\mathrm\{tg\}\}/\\sqrt\{d\}givesTbo=d​TtgT\_\{\\mathrm\{bo\}\}=\\sqrt\{d\}\\,T\_\{\\mathrm\{tg\}\}\. The comparison is confined to the early phase: once either method concentrates the belief, gains compound and the regime, along with the proposition, expires\.□\\square

## Appendix DCounterfactual Sampling at High Dimension

The target\-constructed rate assumes a direction with\|a⋅x∗\|=Θ​\(1\)\\lvert a\\cdot x^\{\*\}\\rvert=\\Theta\(1\)is actually found\. Samplingcf∼ℬ^tT\\mathrm\{cf\}\\sim\\hat\{\\mathcal\{B\}\}\_\{t\}^\{T\}to obtainV​\(acf\)≥V⋆−ϵV\(a\_\{\\mathrm\{cf\}\}\)\\geq V^\{\\star\}\-\\epsilon, whereV⋆=maxa∈Sd−1⁡V​\(a\)V^\{\\star\}=\\max\_\{a\\in S^\{d\-1\}\}V\(a\), succeeds only whencf\\mathrm\{cf\}lands in a spherical cap of angular radiusΘ​\(ϵ1/2\)\\Theta\(\\epsilon^\{1/2\}\), whose measure scales asΘ​\(ϵ\(d−1\)/2\)\\Theta\(\\epsilon^\{\(d\-1\)/2\}\)\. Success with probability1−δ1\-\\deltaoverKKindependent draws requiresK=Ω​\(log⁡\(1/δ\)/ϵ\(d−1\)/2\)K=\\Omega\\\!\\left\(\\log\(1/\\delta\)\\,/\\,\\epsilon^\{\(d\-1\)/2\}\\right\)\. This exponential dependence onddmotivates flooring the candidate set with the top\-MMcovariance eigenvectors \(Section[IV\-A](https://arxiv.org/html/2608.11229#S4.SS1)\)\. Under isotropy an eigenvector is anx∗x^\{\*\}\-agnostic direction and inherits the belief\-only rateΘ​\(1/d\)\\Theta\(1/d\); its value appears in later, anisotropic rounds, onceCtC\_\{t\}develops eigenvalue spread and somevjv\_\{j\}correlates with the errorx∗−x¯tx^\{\*\}\-\\bar\{x\}\_\{t\}\. It is a performance floor, not a target\-constructed query in its own right\.