Latent-IM: Latent Interaction Management for Speech LLMs

# Latent-IM: Latent Interaction Management for Speech LLMs
Source: [https://arxiv.org/html/2607.26928](https://arxiv.org/html/2607.26928)
###### Abstract

Classical spoken dialogue systems often separated dialogue management from response realization: a policy selected the next dialogue action, and a generation component expressed that action\. As dialogue systems shift toward LLMs, this decomposition has largely disappeared into the model’s hidden representations\. We ask whether an LLM\-internal analogue of state estimation and action control can be recovered for conversational moves such as acknowledging, checking, querying, explaining, and replying\. We formulate move control as two coupled problems:*selection*, predicting the appropriate next move from the dialogue context, and*realization*, causally producing a chosen move at generation time\. We introduceLatent\-IM, an internal dialogue\-management framework that provides a general interface for choosing and deploying conversational moves under different objectives\. Here, we use this control to reproduce human move choices, improving average end\-to\-end move accuracy by 12\.5 points over the unsteered backbone while performing comparably to fine\-tuning\.

## Introduction

Classical dialogue systems often modeled conversation as a state–action control problem\. A dialogue manager maintained a representation of the conversation state, often a belief state under uncertainty, and a policy selected the next system action\. This decomposition was especially influential in Partially Observable Markov Decision Process \(POMDP\) approaches, where noisy observations were integrated into a belief state and actions were chosen to maximize expected dialogue success\(Williams and Young[2007](https://arxiv.org/html/2607.26928#bib.bib28); Younget al\.[2013](https://arxiv.org/html/2607.26928#bib.bib29)\)\. Modern LLMs instead take an end\-to\-end approach, generating responses directly from the dialogue history without explicit state management or action policy\.

Although modern LLMs lack explicit state and policy modules, they must still resolve the same underlying control problem: determining what conversational action is appropriate in the current context\(Shaikhet al\.[2024](https://arxiv.org/html/2607.26928#bib.bib1); Maitraet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib3); Luoet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib4); Avsian and Heck[2026](https://arxiv.org/html/2607.26928#bib.bib23); Nguyenet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib40); Veluriet al\.[2024](https://arxiv.org/html/2607.26928#bib.bib41); Leeet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib42)\)\. This decision must therefore be represented and resolved implicitly within the LLM’s hidden activations\. This raises our central question: can an internal analogue of state estimation and action control be recovered from these representations and used to govern conversational behavior?

We introduceLatent\-IM, a framework for recovering interaction management \(IM\) from the representations of a frozen speech LLM\. We operationalize IM as deciding which local conversational action the model should take and when it should yield the floor\. Specifically, we study five moves that support grounding, information exchange, and uncertainty resolution\. Mirroring the classical state–action decomposition,Latent\-IMseparates*selection*, which predicts the next move from residual\-stream activations and reaches 0\.60 average accuracy, from*realization*, which uses activation steering to express that move and outperforms the strongest baseline by 10\.4 percentage points\. A complementary turn\-boundary direction controls when the response terminates, monotonically varying mean response length from71\.371\.3to10\.410\.4words across the steering range\.

Our contributions are as follows:

- •We combine activation\-based selection and realization intoLatent\-IM, an end\-to\-end dialogue\-management system that operates without gold moves or backbone updates, outperforming other control baselines and matching supervised fine\-tuning\.
- •We show that a lightweight streaming controller can recover conversational state from frozen speech LLM activations, outperforming transcript\-based selectors across three task\-oriented dialogue datasets\.
- •We derive reusable move\-specific activation directions that causally realize conversational actions\. Directions learned from MapTask transfer without re\-estimation to FindTask and CReST and outperform prompting, candidate\-selection, and decoding\-time control baselines\.
- •We identify a distinct turn\-boundary direction that causally controls response length and integrate it with realization\-aware gating to regulate when the model yields the floor\.

## Related Work

#### Dialogue State, Policy, and Conversation Analysis\.

Classical task\-oriented dialogue systems framed dialogue management as state tracking followed by action selection\. Finite\-state and frame\-based systems explicitly represented task progress and slot values, while POMDP approaches maintained beliefs over latent dialogue states to support action under uncertainty\. Conversation analysis and dialogue\-act theory complement this view by describing interaction through sequential actions such as grounding, repair, acknowledgment, clarification, and response\(Clark and Brennan[1991](https://arxiv.org/html/2607.26928#bib.bib8); Traum and Hinkelman[1992](https://arxiv.org/html/2607.26928#bib.bib9); Sackset al\.[1974](https://arxiv.org/html/2607.26928#bib.bib12); Schegloffet al\.[1977](https://arxiv.org/html/2607.26928#bib.bib38); Purveret al\.[2001](https://arxiv.org/html/2607.26928#bib.bib39)\)\. Although classical systems provided an explicit state/action abstraction, they were often tied to domain\-specific intents, slots, and templates\. Recent work such as NextLat induces belief\-like internal states through latent\-prediction objectives\(Teohet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib56)\)\. We instead recover dialogue state from a frozen LLM’s residual stream and represent actions as domain\-general conversational moves\.

#### LLMs and Activation Steering\.

LLMs can produce fluent dialogue while underproducing grounding and repair, answering prematurely under ambiguity, diverging from human dialogue\-act distributions\(Shaikhet al\.[2024](https://arxiv.org/html/2607.26928#bib.bib1); Maitraet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib3); Luoet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib4); Dokmeet al\.[2026](https://arxiv.org/html/2607.26928#bib.bib54)\)\. Prior work has used internal representations to modify model behavior at inference time\(Liet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib24); Toddet al\.[2024](https://arxiv.org/html/2607.26928#bib.bib25); Panicksseryet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib26); Avsianet al\.[2026](https://arxiv.org/html/2607.26928#bib.bib27); Subramaniet al\.[2022](https://arxiv.org/html/2607.26928#bib.bib36); Dokme and Heck[2026](https://arxiv.org/html/2607.26928#bib.bib55); Zouet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib37)\)\. Activation Addition constructs contrastive directions for attributes such as sentiment, topic, and toxicity\(Turneret al\.[2024](https://arxiv.org/html/2607.26928#bib.bib10)\), while methods such as conceptor steering capture richer activation structure\(Postmus and Abreu[2024](https://arxiv.org/html/2607.26928#bib.bib11)\)\. Recent work extends activation steering to speech and audio\-language models, including temporal audio attention, speech adaptation, and interruption control\(Changet al\.[2026](https://arxiv.org/html/2607.26928#bib.bib47); Lin and Lee[2026](https://arxiv.org/html/2607.26928#bib.bib48); Yegorovaet al\.[2026](https://arxiv.org/html/2607.26928#bib.bib49)\)\. Our work instead targets local, context\-dependent conversational actions by separating when a move should be selected from how it is causally realized\.

## Task & Data

We study controllable generation of conversational moves in asymmetric, instruction\-oriented task dialogues\. Let a dialogueDDbe a sequence of turns

D=\(\(r1,u1\),…,\(rT,uT\)\),D=\\bigl\(\(r\_\{1\},u\_\{1\}\),\\ldots,\(r\_\{T\},u\_\{T\}\)\\bigr\),wherertr\_\{t\}andutu\_\{t\}denote the speaker role and utterance at turntt\. All datasets contain two asymmetric participant roles, which we call*giver*and*follower*\. The giver provides task\-relevant instructions or information, while the follower acts on that information\. We focus on follower turns because they exhibit a richer and more interaction\-oriented move inventory\. For a follower turntt, the preceding context is

ct=\(\(r1,u1\),…,\(rt−1,ut−1\)\),c\_\{t\}=\\bigl\(\(r\_\{1\},u\_\{1\}\),\\ldots,\(r\_\{t\-1\},u\_\{t\-1\}\)\\bigr\),and the human responseutu\_\{t\}has gold move label

yt∈ℳ=\{acknowledge,check,explain,query,reply\}\.y\_\{t\}\\in\\mathcal\{M\}=\\\{\\textit\{acknowledge\},\\textit\{check\},\\textit\{explain\},\\textit\{query\},\\textit\{reply\}\\\}\.We map each dataset’s annotations into this shared taxonomy, defined in Table[1](https://arxiv.org/html/2607.26928#Sx3.T1)\. These labels capture domain\-general interactional functions rather than task\-specific intents\. Turn completion is modeled separately and is not included inℳ\\mathcal\{M\}\.

We decompose move control into two tasks\.*Selection*predicts the appropriate follower move from the dialogue context:

pθ​\(yt∣ct\)=sθ​\(ct\),y^t=arg⁡maxm∈ℳ⁡pθ​\(yt=m∣ct\)\.p\_\{\\theta\}\(y\_\{t\}\\mid c\_\{t\}\)=s\_\{\\theta\}\(c\_\{t\}\),\\qquad\\hat\{y\}\_\{t\}=\\arg\\max\_\{m\\in\\mathcal\{M\}\}p\_\{\\theta\}\(y\_\{t\}=m\\mid c\_\{t\}\)\.*Realization*generates a response conditioned on a target move:

u^t=gϕ​\(ct,yt⋆\)\.\\hat\{u\}\_\{t\}=g\_\{\\phi\}\(c\_\{t\},y\_\{t\}^\{\\star\}\)\.In oracle\-realization experiments,yt⋆=yty\_\{t\}^\{\\star\}=y\_\{t\}is the gold human move\. In the end\-to\-end setting,yt⋆=y^ty\_\{t\}^\{\\star\}=\\hat\{y\}\_\{t\}is selected by the controller\. Thus, selection determines*which*conversational move to make, while realization determines*how*that move is expressed\.

Table 1:Merged follower\-move types\.### Datasets

We evaluate this formulation on three dialogue datasets\.

#### MapTask\.

Our primary dataset is MapTask\(Andersonet al\.[1991](https://arxiv.org/html/2607.26928#bib.bib13)\), a collection of unscripted, task\-oriented spoken dialogues in which two participants collaborate to reproduce a route on a map\. The giver has a map with a marked route, while the follower has a related but non\-identical map and must draw the route from verbal instructions\.

#### FindTask\.

We also use FindTask from the RoboHelper corpus\(Chen and Di Eugenio[2013](https://arxiv.org/html/2607.26928#bib.bib21)\), which contains task\-oriented human\-robot dialogues in which participants collaborate with a robotic helper to locate objects, exchange task information, and resolve misunderstandings\.

#### CReST\.

Finally, we include the Indiana Cooperative Remote Search Task \(CReST\) corpus\(Eberhardet al\.[2010](https://arxiv.org/html/2607.26928#bib.bib22)\), in which a director views an indoor map and guides a remotely connected searcher through the environment\.

![Refer to caption](https://arxiv.org/html/2607.26928v1/Figures/main.png)Figure 1:\(a\) Before generation, the controller predicts the next conversational move, whether and how strongly to apply move steering, and the appropriate turn length \(via a turn\-boundary strength\)\. These predictions are made without gold labels at inference\. In this example, the controller selects QUERY as the next move\. \(b\) During generation, the selected move directionvqueryv\_\{\\textsc\{query\}\}and the turn\-boundary directionvE​O​Tv\_\{EOT\}are injected at selected transformer layers with strengthsα\\alphaandβ\\beta\. This causes the frozen backbone to realize the selected move and terminate at an appropriate boundary\.

## Method

Latent\-IMrecovers a classical dialogue\-management pipeline within a frozen speech LLM \(Figure[1](https://arxiv.org/html/2607.26928#Sx3.F1)\)\. A recurrent controller reads residual\-stream summaries to maintain dialogue state and select the next conversational move, which is then realized through move\-specific activation steering rather than symbolic policies or template\-based generation\.

### LLM\-Grounded Dialogue State and Move Selection

The controller is a lightweight stateful sidecar that reads mean\-pooled residual\-stream activations from the frozen backbone\. For each context turnτ<t\\tau<t, it standardizes the layer\-ℓctrl⋆\\ell\_\{\\mathrm\{ctrl\}\}^\{\\star\}representationxτ∈ℝdx\_\{\\tau\}\\in\\mathbb\{R\}^\{d\}and updates a selective state\-space model\(Gu and Dao[2023](https://arxiv.org/html/2607.26928#bib.bib30)\):

zτ=a¯τ⊙zτ−1\+b¯τ,oτ=Cτ​zτ,z\_\{\\tau\}=\\bar\{a\}\_\{\\tau\}\\odot z\_\{\\tau\-1\}\+\\bar\{b\}\_\{\\tau\},\\qquad o\_\{\\tau\}=C\_\{\\tau\}z\_\{\\tau\},\(1\)wherea¯τ,b¯τ\\bar\{a\}\_\{\\tau\},\\bar\{b\}\_\{\\tau\}depend onxτx\_\{\\tau\}anda¯τ∈\(0,1\)N\\bar\{a\}\_\{\\tau\}\\in\(0,1\)^\{N\}\. A linear head predicts the next move, and an auxiliary head regresses log response length:

pθ​\(yt∣ct\)=softmax⁡\(Wo​ot−1\),L^t=wL⊤​ot−1\.p\_\{\\theta\}\(y\_\{t\}\\mid c\_\{t\}\)=\\operatorname\{softmax\}\(W\_\{o\}o\_\{t\-1\}\),\\qquad\\widehat\{L\}\_\{t\}=w\_\{L\}^\{\\top\}o\_\{t\-1\}\.\(2\)The length output is monotonically calibrated to a turn\-boundary steering strength and, like all steering decisions, acts through the move\-level gating rule below; move\-steering strengths are per\-move constants selected on training data\.

#### Realization\-aware gating\.

Based on training\-set steerability, move steering is withheld forreply, which the unsteered backbone realizes more reliably, while turn\-boundary steering is applied only to naturally terse moves \(acknowledge,query, andreply\)\. All gates depend on the controller prediction rather than gold labels\.

#### Training and inference\.

With the backbone frozen, we train the controller using next\-move cross\-entropy and smooth\-L1L\_\{1\}log\-length loss\. All components use the same dialogue\-level split, excluding held\-out test dialogues\. At inference, the controller maintains a persistent streaming state across turns\.

### Conversational\-Move Subspaces for Realization

For each follower turnttin the training dialogues, with gold moveyt∈ℳy\_\{t\}\\in\\mathcal\{M\}, we extract the residual\-stream change at response onset\. Lethtℓ​\(s\)∈ℝdh\_\{t\}^\{\\ell\}\(s\)\\in\\mathbb\{R\}^\{d\}denote the activation at layerℓ\\elland token positionss, and letst,1s\_\{t,1\}denote the position of the first token of responseutu\_\{t\}\. We define

Δtℓ=htℓ​\(st,1\)−htℓ​\(st,1−1\),\\Delta\_\{t\}^\{\\ell\}=h\_\{t\}^\{\\ell\}\(s\_\{t,1\}\)\-h\_\{t\}^\{\\ell\}\(s\_\{t,1\}\-1\),\(3\)which captures the transition from dialogue context to response generation\. For each backbone, move directions are estimated exclusively from its activations on the MapTask training split and then applied, without re\-estimation, to the same backbone on the held\-out MapTask test split, FindTask, and CReST\.

At each layer, we standardize the onset differences\{Δtℓ\}t∈ℐtrain\\\{\\Delta\_\{t\}^\{\\ell\}\\\}\_\{t\\in\\mathcal\{I\}\_\{\\text\{train\}\}\}, project them into akk\-dimensional PCA space, and estimate a one\-versus\-rest Fisher direction for each move using Ledoit–Wolf shrinkage\(Ledoit and Wolf[2004](https://arxiv.org/html/2607.26928#bib.bib32); Fisher[1936](https://arxiv.org/html/2607.26928#bib.bib31)\)\. Letνmℓ\\nu\_\{m\}^\{\\ell\}andν¬mℓ\\nu\_\{\\neg m\}^\{\\ell\}denote the means of movemmand all remaining moves in PCA space, and letΣ^w,mℓ\\widehat\{\\Sigma\}\_\{w,m\}^\{\\ell\}denote the shrinkage estimate of their pooled within\-class covariance\. The discriminative direction is

wmℓ∝\(Σ^w,mℓ\)−1​\(νmℓ−ν¬mℓ\)\.w\_\{m\}^\{\\ell\}\\propto\\left\(\\widehat\{\\Sigma\}\_\{w,m\}^\{\\ell\}\\right\)^\{\-1\}\\left\(\\nu\_\{m\}^\{\\ell\}\-\\nu\_\{\\neg m\}^\{\\ell\}\\right\)\.\(4\)
We mapwmℓw\_\{m\}^\{\\ell\}back to residual\-stream coordinates, undo the feature scaling, and normalize the result to obtain the move directionvmℓv\_\{m\}^\{\\ell\}\. The inverse covariance suppresses dimensions with high within\-class variation, yielding a more discriminative direction than a simple difference of means\.

For target movemm, we inject its direction at the selected layerℓm⋆\\ell\_\{m\}^\{\\star\}at every generated token positionss:

hℓm⋆​\(s\)←hℓm⋆​\(s\)\+α​cℓm⋆​vmℓm⋆\.h^\{\\ell\_\{m\}^\{\\star\}\}\(s\)\\leftarrow h^\{\\ell\_\{m\}^\{\\star\}\}\(s\)\+\\alpha c\_\{\\ell\_\{m\}^\{\\star\}\}v\_\{m\}^\{\\ell\_\{m\}^\{\\star\}\}\.\(5\)wherecℓmc\_\{\\ell\_\{m\}\}is the median residual\-stream norm at that layer andα\\alphais a dimensionless steering coefficient\. This norm\-relative scaling improves comparability across layers and model backbones\.

### Automatic Move Classifier

We use Qwen2\.5\-72B\(Yanget al\.[2025](https://arxiv.org/html/2607.26928#bib.bib51)\)as a prompted few\-shot classifier over five move types, using the same prompt and decoding procedure across datasets\. Classification is context\-dependent: we first label the preceding giver turn, then classify the follower response conditioned on that label, using the gold giver label when available\. Decoding is restricted to one option token, whose normalized log\-probabilities defineP​\(move∣⋅\)P\(\\text\{move\}\\mid\\cdot\)\. On human corpus responses, the classifier recovers the gold move with77\.2%77\.2\\%–81\.6%81\.6\\%accuracy \(Table[2](https://arxiv.org/html/2607.26928#Sx4.T2)\)\. For generation evaluation, we withhold the human response, generate a replacement, and compare its classifier\-assigned move with the original human gold label\. We further validate it on500500generated responses, balanced across target moves, datasets, and backbone models\. Three blinded annotators label each response from its dialogue context\. The classifier agrees with their majority label on73\.9%73\.9\\%of examples, while human agreement is Fleiss’κ=0\.682\\kappa=0\.682\. Across the seven generation methods, classifier\-based and human\-majority move accuracies are strongly correlated \(Spearmanρ=0\.96\\rho=0\.96\) and induce nearly the same ranking, so the automatic metric does not systematically favor any particular method\.

Table 2:Move\-classifier validation against corpus labels and majority judgments from three blinded annotators; Fleiss’κ\\kappameasures inter\-annotator agreement\.

## Results

We evaluate the proposed framework at three levels: next\-move selection, oracle\-target realization, and end\-to\-end conversational control\. Experiments span Qwen2\.5\-Omni\(Xuet al\.[2025a](https://arxiv.org/html/2607.26928#bib.bib33)\), Qwen3\-Omni\(Xuet al\.[2025b](https://arxiv.org/html/2607.26928#bib.bib34)\), and Phi\-4\-Multimodal\(Aboueleninet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib35)\)\.

### Selection: Predicting the Next Move

Table 3:Next\-move selection accuracy\. Markov\-1 receives the gold preceding move; the activation probe and the controller read the residual stream at the selected controller layerℓctrl⋆\\ell\_\{\\mathrm\{ctrl\}\}^\{\\star\}\.The activation controller outperforms all transcript\-based selectors across datasets\. On MapTask it reaches0\.620\.62, close to the gold\-history Markov baseline at0\.650\.65, while on FindTask and CReST it surpasses that baseline\. The non\-recurrent activation probe performs similarly on MapTask and CReST, suggesting that the backbone already integrates the relevant dialogue history\. On FindTask, the stateful controller improves over the probe by0\.080\.08, indicating that explicit cross\-turn state is useful for shorter episodes\.

### Realization: Generating a Target Move

Table 4:Oracle\-target move\-realization accuracy \(%\)\. All methods receive the gold target move\.Table 5:End\-to\-end move\-control results across dialogue datasets and multimodal language models\. Acc\. is the percentage of generated responses whose predicted conversational move matches the held\-out human move\. BLEU measures lexical overlap with the held\-out human response\.We evaluate oracle realization by providing every method with the dialogue context and gold follower move\. Baselines include direct prompting \(Base\), few\-shot candidate selection \(PBL\)\(Ramirezet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib18)\), instruction\-derived steering \(SI\)\(Stolfoet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib19)\), classifier\-guided decoding \(FUDGE\)\(Yang and Klein[2021](https://arxiv.org/html/2607.26928#bib.bib20)\), and reward\-guided search \(DeAL\)\(Huanget al\.[2025](https://arxiv.org/html/2607.26928#bib.bib16)\)\.Latent\-IMapplies move directions learned exclusively from MapTask and transferred unchanged to all evaluation sets\. It achieves the highest mean accuracy \(60\.0%60\.0\\%\) and leads in six of nine settings\.

### End\-to\-End Move Control

Finally, we evaluate end\-to\-end response generation, where all methods receive the same held\-out dialogue context and no gold move labels at inference\. Table[5](https://arxiv.org/html/2607.26928#Sx5.T5)compares two explicit selection–realization pipelines with three direct\-generation baselines\. Prompt Control first predicts a conversational move using an LLM selector and then includes that prediction in a definitional prompt\.Latent\-IMinstead uses the streaming activation controller to select the next move and realizes it through fixed move\-specific steering directions\. Base, PAS, and SFT do not explicitly predict or condition on move labels: Base generates directly from the dialogue context, PAS applies its learned intervention directly during generation, and SFT separately fine\-tunes each backbone with LoRA\(Huet al\.[2022](https://arxiv.org/html/2607.26928#bib.bib53)\)on context–response pairs to imitate the human response\.

Latent\-IMachieves the highest mean end\-to\-end move accuracy at46\.9%46\.9\\%, slightly exceeding SFT at46\.6%46\.6\\%and outperforming Base \(34\.4%34\.4\\%\), PAS \(34\.2%34\.2\\%\), and Prompt Control \(27\.6%27\.6\\%\)\. It obtains the highest accuracy in the majority of the dataset–model settings\. SFT remains stronger for Qwen3 and Qwen2\.5 on MapTask and CReST and produces higher BLEU\(Papineniet al\.[2002](https://arxiv.org/html/2607.26928#bib.bib50)\)scores overall, consistent with direct supervised imitation of the reference response\. In contrast,Latent\-IMoften realizes the appropriate conversational move using lexically different wording\. These results show that explicitly recovering and controlling an intermediate conversational action can match direct supervised fine\-tuning while retaining a modular, label\-free inference pipeline\.

## Turn\-Boundary Subspace

Predicting whether a speaker will continue or yield the floor is a longstanding problem in spoken dialogue modeling\(Skantze[2017](https://arxiv.org/html/2607.26928#bib.bib43); Ekstedt and Skantze[2020](https://arxiv.org/html/2607.26928#bib.bib44); Linet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib45)\)\. We identify a turn\-boundary direction representing whether the current response should continue or terminate, complementing acoustic pauses that may occur within a turn\(Lalaet al\.[2017](https://arxiv.org/html/2607.26928#bib.bib46)\)\. We derive supervision from the speaker sequence of the MapTask training split\. For each turnt<Tt<T, we assign

bt=𝟏​\[rt\+1≠rt\],b\_\{t\}=\\mathbf\{1\}\\\!\\left\[r\_\{t\+1\}\\neq r\_\{t\}\\right\],\(6\)wherebt=1b\_\{t\}=1denotes yielding the floor andbt=0b\_\{t\}=0denotes continuing\. At each layer, we mean\-pool the residual stream over the turn’s token span\. A layer sweep selects layerℓeot⋆\\ell^\{\\star\}\_\{\\textsc\{eot\}\}, and the unit\-norm turn\-boundary direction is estimated by the difference between the yield and hold class means:

veot=μyield−μhold∥μyield−μhold∥2\.v\_\{\\textsc\{eot\}\}=\\frac\{\\mu\_\{\\textsc\{yield\}\}\-\\mu\_\{\\textsc\{hold\}\}\}\{\\lVert\\mu\_\{\\textsc\{yield\}\}\-\\mu\_\{\\textsc\{hold\}\}\\rVert\_\{2\}\}\.\(7\)Like the move directions,veotv\_\{\\textsc\{eot\}\}is learned only from MapTask training dialogues and transferred without re\-estimation to all evaluation sets\. The direction is predictive: on held\-out follower turns it detects upcoming boundaries at0\.950\.95AUROC from content alone, versus0\.9320\.932for a pause\-only detector, which is also post hoc since it can only score a boundary after the silence has elapsed\. An incremental spoken system needs the former to yield the floor without a gap\. During generation, we inject the direction using the same norm\-relative rule as move steering:

hℓeot⋆​\(s\)←hℓeot⋆​\(s\)\+β​cℓeot⋆​veot\.h^\{\\ell^\{\\star\}\_\{\\textsc\{eot\}\}\}\(s\)\\leftarrow h^\{\\ell^\{\\star\}\_\{\\textsc\{eot\}\}\}\(s\)\+\\beta c\_\{\\ell^\{\\star\}\_\{\\textsc\{eot\}\}\}v\_\{\\textsc\{eot\}\}\.\(8\)Positiveβ\\betaencourages termination, while negativeβ\\betaprolongs the response\. Figure[2](https://arxiv.org/html/2607.26928#Sx6.F2)sweepsβ\\betaover\[−1,1\]\[\-1,1\]on250250held\-out test contexts with only the turn\-boundary direction injected: mean response length varies monotonically from71\.371\.3words atβ=−1\\beta\{=\}\-1through the unsteered24\.924\.9down to10\.410\.4atβ=1\\beta\{=\}1, with decoding remaining fluent throughout, giving graded, sample\-free control over when the model yields the turn\.

![Refer to caption](https://arxiv.org/html/2607.26928v1/Figures/eot_beta_sweep.png)Figure 2:Steering along the turn\-boundary direction monotonically controls response length \(mean±\\pm95% CI\)\.
## Analysis

### Controller

#### State is read, not recomputed\.

Table[3](https://arxiv.org/html/2607.26928#Sx5.T3)separates what carries the dialogue state from what reads it\. The transcript\-embed rows see the same token sequence as the controller but through the backbone’s embedding table alone; with matched readers, replacing contextual activations with these lexical features costs three to five points on MapTask and CReST, and on FindTask the gap is closed only by a reader that carries explicit cross\-turn state\. The state is computed by the frozen backbone and merely read out\. Reading it is also more reliable than asking for it: prompting GPT\-4o or Qwen2\.5\-72B to name the next move falls0\.080\.08–0\.260\.26below the 0\.5M controller\. Finally, the reader’s architecture barely matters: MLP, GRU, LSTM, and SSM variants all reach≈0\.60\\approx 0\.60on MapTask, which is what one expects if the state is linearly available in the residual stream at each turn\. We adopt the selective SSM because it formalizes the per\-turn state update with a constant\-time streaming step, not because recurrence wins on accuracy; the exception is FindTask’s short episodes, where stateless readers drop to≈0\.50\\approx 0\.50and carried state is what remains\.

#### Selection is not enough\.

Higher selection accuracy does not always translate into better end\-to\-end control, because per\-move predictability and steerability tend to trade off: a move the controller predicts confidently is not necessarily one that steering can realize\. The clearest case isreply: after a giver question the follower almost always replies, so a giver\-move prior sharply improvesreplyselection \(precision0\.53→0\.640\.53\\\!\\to\\\!0\.64\), yet this yields no end\-to\-end gain, because thereplysteering direction surface\-overlapsacknowledgeand does not land \(controllability0\.140\.14vs\.0\.990\.99forquery\)\. This motivates*realization\-aware gating*: apply the action\-subspace steering only where it lands\.

#### Selection under ambiguity\.

Selection accuracy must be interpreted against how consistent*humans*are in the same situation\. Even near\-identical giver turns license different follower moves: in our held\-out set, after the identical giver turn “*do you see what I mean*” one followerreplies\(“yeah”\) while anotherqueries\(“so do I go up straight first?”\); after near\-identical instructions \(“*…centimetres from the top/bottom of the page*”\) one followerchecks\(“the bottom of the page you mean”\) while anotheracknowledges\(“mm right”\)\. We find 55 such similar\-giver/different\-follower pairs in the test dialogues alone\. Aggregating by the preceding giver move, the gold follower move is concentrated after questions \(afterquery\-yn,75%75\\%of humans reply\) but genuinely open after instructions and acknowledgements \(mode probability0\.300\.30–0\.510\.51\)\.

The controller inherits exactly this structure\. Splitting held\-out turns by human consistency: on*clear*contexts \(mode≥0\.6\\geq 0\.6\) it reaches0\.610\.61against a human\-consistency ceiling of0\.680\.68\(89%89\\%\); on*open*contexts it reaches0\.460\.46against a ceiling of0\.490\.49\(95%95\\%\)\. Per giver\-move type, controller accuracy tracks the human mode probability nearly one\-to\-one \(e\.g\.,query\-yn:0\.7580\.758vs\.0\.750\.75\), and its predictive entropy correlates with the true follower\-move entropy atr=0\.95r=0\.95\. Its confusions flow intoacknowledge—the modal move—which is the Bayes\-optimal guess under ambiguity rather than a failure mode\. The ceiling is also modality\-invariant: adding the follower’s map as text \(0\.5260\.526\) or as an image through the vision pathway \(0\.5210\.521\) leaves selection at the no\-map level \(0\.5240\.524\)\. Together these results indicate the remaining selection gap is irreducible follower choice, not missing features; moving past it requires scoring moves by task outcome rather than matching a single human reference, which we leave to future work\.

#### Realization\-aware gating\.

The remedy is a gating rule on the action subspace, not a new controller: for each predicted move we decide whether to inject the move direction and whether to inject the turn\-length direction, both read from the controller’s own prediction \(no gold labels\)\. The rule follows measured per\-move steerability: steer where the controllability matrix is strong \(check,query,explain\), defer where it is weak \(reply\)\. Deferringreplyraises end\-to\-end realization from0\.670\.67to0\.750\.75at no selection cost; length\-aware control adds a further gain to0\.780\.78\. Length is handled as move policy rather than per\-turn regression: the controller maps its selected move to a turn\-boundary coefficient, pulling the naturally terse moves \(acknowledge,query,reply\) from∼25\\sim\\\!25words toward the human median of22–33\. For content\-bearing moves \(check,explain\) the intervention is*withheld*: the injection needed to shorten them overwhelms the move direction and collapses realization \(explain0\.76→0\.280\.76\\\!\\to\\\!0\.28\)\. A controller that models the geometry of its own steering outperforms one that only predicts moves accurately\.

### Steering

To test whether steering directions are move\-specific, we apply each move vector to the same held\-out contexts and classify the resulting generations\. The resulting controllability matrix shows whether each intervention selectively increases its corresponding conversational move rather than merely perturbing generation\. Figure[3](https://arxiv.org/html/2607.26928#Sx7.F3)shows that each steering direction selectively increases its intended move, with a mean diagonal controllability of0\.660\.66and limited off\-diagonal leakage\. Relative to the unsteered baseline, steering raisesqueryfrom0\.050\.05to0\.960\.96andexplainfrom0\.070\.07to0\.700\.70, whileacknowledgeandcheckrise to0\.650\.65and0\.700\.70\. These results indicate that the interventions causally target move\-specific behavior rather than generically perturbing generation\.

![Refer to caption](https://arxiv.org/html/2607.26928v1/Figures/controllability_matrix.png)Figure 3:Controllability matrix for move steering \(map\-task, Qwen2\.5\-Omni\-7B\)\. Each row applies one steering direction to the same held\-out contexts; columns give the move assigned to the generation by the classifierTable[6](https://arxiv.org/html/2607.26928#Sx7.T6)ablates the main design choices in the steering intervention using oracle target moves; each row varies one component while holding the others at the default configuration\. Steering raises target\-move accuracy from 0\.33 to 0\.65, a 0\.31\-point gain\. The modality comparison shows that spoken information contributes signal beyond the transcript alone: audio\+text outperforms text\-only \(0\.65 vs\. 0\.58\)\. The strength sweep exposes a clear controllability tradeoff: accuracy improves up to a moderate strength\(α≈1\.0\(\\alpha\\approx 1\.0, 0\.64\) but falls to 0\.55 atα=2\.0\\alpha=2\.0\. The layer ablation shows that early\-layer injection is substantially weaker \(0\.25\) than middle\- or late\-layer injection \(0\.48/0\.47\), while no fixed layer matches the per\-move tuned default \(0\.65\)\. Finally, control is largely insensitive to injection site: residual\-stream, attention, and MLP interventions obtain 0\.65, 0\.65, and 0\.64, respectively\. Overall, steering depends on multimodal context, moderate intervention strength, and layer choice, while remaining robust to injection location\.

Abl\.SettingAcc\.Δ\\DeltaBLEUDefaultAudio\+text,α⋆\\alpha^\{\\star\},ℓ⋆\\ell^\{\\star\}, residual0\.65–0\.18ModalityText only0\.58−0\.07\-0\.070\.19Strengthα=0\.0\\alpha=0\.00\.33−0\.31\-0\.310\.11Strengthα=0\.5\\alpha=0\.50\.53−0\.12\-0\.120\.14Strengthα=1\.0\\alpha=1\.00\.64−0\.01\-0\.010\.17Strengthα=2\.0\\alpha=2\.00\.55−0\.10\-0\.100\.19LayerEarly0\.25−0\.40\-0\.400\.10LayerMiddle0\.48−0\.17\-0\.170\.19LayerLate0\.47−0\.18\-0\.180\.09SiteAttention0\.650\.000\.000\.18SiteMLP0\.64−0\.01\-0\.010\.18Table 6:Steering ablations on held\-out MapTask contexts using Qwen2\.5\-Omni\-7B and oracle target moves\. Each row varies one component of the default configuration while holding the others fixed\.Δ\\Deltadenotes the change in target\-move accuracy relative to the default; BLEU measures lexical overlap with the human response\.
### Future Work: Beyond Human\-Move Imitation

Our controller imitates the move a human is likely to make, but the best task\-oriented move may differ from the human choice\. Because our steering mechanism can realize any move in the taxonomy, a future goal\-conditioned controller could select moves according to predicted task utility rather than imitation likelihood\.

MapTask: an acknowledgment hides a map mismatchGiver:“Walk north until you reach the Roman baths\.”\[instruct\]Follower:“Uh\-huh\.”\[acknowledge\]Giver:“Then walk right around them\.”\[instruct\]Follower:“Roman baths? I don’t have any Roman baths\.”\[query\]The follower initially acknowledges an instruction whose referent is absent from their map\. Although acknowledgment matches the observed human move, an earliercheckorquery, such as “Roman baths? I do not have those on my map,” would reveal the mismatch before it propagates into later instructions\.

## Conclusion

We introducedLatent\-IM, which separates conversational\-move selection from realization within a frozen speech LLM\. A streaming controller reads residual\-stream activations to select the next move, reaching 0\.60 average accuracy, while activation steering improves oracle realization by 10\.4 percentage points over the strongest baseline\. A complementary turn\-boundary direction reduces response length by 9\.1 words, and the full system improves end\-to\-end move accuracy by 12\.5 points over the unsteered backbone while performing comparably to fine\-tuning\. These results provide evidence that frozen speech LLM activations can serve as an observable and causally controllable interface for local interaction management\. Future work can replace imitation\-based selection with goal\-conditioned policies that choose moves according to predicted task utility\.

## References

- A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen,et al\.\(2025\)Phi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-loras\.arXiv preprint arXiv:2503\.01743\.Cited by:[Results](https://arxiv.org/html/2607.26928#Sx5.p1.1)\.
- A\. H\. Anderson, M\. Bader, E\. G\. Bard, E\. Boyle, G\. Doherty, S\. Garrod, S\. Isard, J\. Kowtko, J\. McAllister, J\. Miller,et al\.\(1991\)The hcrc map task corpus\.Language and speech34\(4\),pp\. 351–366\.Cited by:[MapTask\.](https://arxiv.org/html/2607.26928#Sx3.SSx1.SSS0.Px1.p1.1)\.
- A\. Avsian and L\. Heck \(2026\)SNEAK: evaluating strategic communication and information leakage in large language models\.arXiv preprint arXiv:2603\.29846\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1)\.
- A\. Avsian, C\. Richardson, A\. Sundar, and L\. Heck \(2026\)FLEx: language modeling with few\-shot language explanations\.arXiv preprint arXiv:2601\.04157\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- C\. Chang, K\. Chang, A\. H\. Liu, and J\. Glass \(2026\)Overcoming state inertia in full\-duplex spoken language models via activation steering\.arXiv preprint arXiv:2606\.11386\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- L\. Chen and B\. Di Eugenio \(2013\)Multimodality and dialogue act classification in the RoboHelper project\.InProceedings of the SIGDIAL 2013 Conference,M\. Eskenazi, M\. Strube, B\. Di Eugenio, and J\. D\. Williams \(Eds\.\),Metz, France,pp\. 183–192\.External Links:[Link](https://aclanthology.org/W13-4031/)Cited by:[FindTask\.](https://arxiv.org/html/2607.26928#Sx3.SSx1.SSS0.Px2.p1.1)\.
- H\. H\. Clark and S\. E\. Brennan \(1991\)Grounding in communication\.\.Cited by:[Dialogue State, Policy, and Conversation Analysis\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Cui and Z\. Chen \(2026\)Painless activation steering: an automated, lightweight approach for post\-training large language models\.External Links:2509\.22739,[Link](https://arxiv.org/abs/2509.22739)Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px6.p1.1)\.
- A\. Dokme and L\. Heck \(2026\)Selective state\-space adaptation and retrieval for language model reasoning\.arXiv preprint arXiv:2607\.19326\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- A\. Dokme, B\. Reichman, and L\. Heck \(2026\)TEMPER: testing emotional perturbation in quantitative reasoning\.InProceedings of the Conference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2604.07801)Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- K\. Eberhard, H\. Nicholson, S\. Kübler, S\. Gundersen, and M\. Scheutz \(2010\)The Indiana “cooperative remote search task” \(CReST\) corpus\.InProceedings of the Seventh International Conference on Language Resources and Evaluation \(LREC’10\),N\. Calzolari, K\. Choukri, B\. Maegaard, J\. Mariani, J\. Odijk, S\. Piperidis, M\. Rosner, and D\. Tapias \(Eds\.\),Valletta, Malta\.External Links:[Link](https://aclanthology.org/L10-1459/)Cited by:[CReST\.](https://arxiv.org/html/2607.26928#Sx3.SSx1.SSS0.Px3.p1.1)\.
- E\. Ekstedt and G\. Skantze \(2020\)TurnGPT: a transformer\-based language model for predicting turn\-taking in spoken dialog\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 2981–2990\.Cited by:[Turn\-Boundary Subspace](https://arxiv.org/html/2607.26928#Sx6.p1.1)\.
- R\. A\. Fisher \(1936\)The use of multiple measurements in taxonomic problems\.Annals of eugenics7\(2\),pp\. 179–188\.Cited by:[Conversational\-Move Subspaces for Realization](https://arxiv.org/html/2607.26928#Sx4.SSx2.p2.6)\.
- A\. Gu and T\. Dao \(2023\)Mamba: linear\-time sequence modeling with selective state spaces\.arXiv preprint arXiv:2312\.00752\.Cited by:[LLM\-Grounded Dialogue State and Move Selection](https://arxiv.org/html/2607.26928#Sx4.SSx1.p1.3)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px4.p2.11),[End\-to\-End Move Control](https://arxiv.org/html/2607.26928#Sx5.SSx3.p1.1)\.
- J\. Y\. Huang, S\. Sengupta, D\. Bonadiman, Y\. Lai, A\. Gupta, N\. Pappas, S\. Mansour, K\. Kirchhoff, and D\. Roth \(2025\)DeAL: decoding\-time alignment for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 26280–26300\.External Links:[Link](http://dx.doi.org/10.18653/v1/2025.acl-long.1274),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1274)Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px5.p1.3),[Realization: Generating a Target Move](https://arxiv.org/html/2607.26928#Sx5.SSx2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Appendix B](https://arxiv.org/html/2607.26928#A2.p1.1)\.
- D\. Lala, P\. Milhorat, K\. Inoue, M\. Ishida, K\. Takanashi, and T\. Kawahara \(2017\)Attentive listening system with backchanneling, response generation and flexible turn\-taking\.InProceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue,pp\. 127–136\.Cited by:[Turn\-Boundary Subspace](https://arxiv.org/html/2607.26928#Sx6.p1.1)\.
- O\. Ledoit and M\. Wolf \(2004\)A well\-conditioned estimator for large\-dimensional covariance matrices\.Journal of multivariate analysis88\(2\),pp\. 365–411\.Cited by:[Appendix E](https://arxiv.org/html/2607.26928#A5.SS0.SSS0.Px2.p1.4),[Conversational\-Move Subspaces for Realization](https://arxiv.org/html/2607.26928#Sx4.SSx2.p2.6)\.
- S\. Lee, K\. Kim, and G\. Kim \(2025\)Behavior\-sd: behaviorally aware spoken dialogue generation with large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 9574–9593\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1)\.
- K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2023\)Inference\-time intervention: eliciting truthful answers from a language model\.Advances in Neural Information Processing Systems36,pp\. 41451–41530\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- T\. Lin and H\. Lee \(2026\)Steering where to listen: instruction\-based activation steering redirects temporal attention in large audio\-language models\.arXiv preprint arXiv:2606\.11400\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Lin, Y\. Zheng, M\. Zeng, and W\. Shi \(2025\)Predicting turn\-taking and backchannel in human\-machine conversations using linguistic, acoustic, and visual signals\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15310–15322\.Cited by:[Turn\-Boundary Subspace](https://arxiv.org/html/2607.26928#Sx6.p1.1)\.
- S\. Luo, Y\. Huang, M\. Li, S\. Meng, F\. Liu, Z\. Hu, J\. Feng, and Q\. Liu \(2025\)ClarifyMT\-bench: benchmarking and improving multi\-turn clarification for conversational large language models\.arXiv preprint arXiv:2512\.21120\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1),[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- A\. Maitra, D\. French, and K\. von der Wense \(2025\)Dialogue acts as a lens on human–llm interaction: analyzing conversational norms in model\-generated responses\.InFourth Workshop on Bridging Human–Computer Interaction and Natural Language Processing,pp\. 317\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1),[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- T\. A\. Nguyen, E\. Kharitonov, J\. Copet, Y\. Adi, W\. Hsu, A\. Elkahky, P\. Tomasello, R\. Algayres, B\. Sagot, A\. Mohamed,et al\.\(2023\)Generative spoken dialogue language modeling\.Transactions of the Association for Computational Linguistics11,pp\. 250–266\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px4.p1.2)\.
- N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner \(2023\)Steering llama 2 via contrastive activation addition\.arXiv preprint arXiv:2312\.06681\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[End\-to\-End Move Control](https://arxiv.org/html/2607.26928#Sx5.SSx3.p2.5)\.
- J\. Postmus and S\. Abreu \(2024\)Steering large language models using conceptors: improving addition\-based activation engineering\.arXiv preprint arXiv:2410\.16314\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- M\. Purver, J\. Ginzburg, and P\. Healey \(2001\)On the means for clarification in dialogue\.InProceedings of the Second SIGdial Workshop on Discourse and Dialogue,Cited by:[Dialogue State, Policy, and Conversation Analysis\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px1.p1.1)\.
- A\. Ramirez, K\. Agarwal, J\. Juraska, U\. Garg, and M\. A\. Walker \(2023\)Controllable generation of dialogue acts for dialogue systems via few\-shot response generation and ranking\.External Links:2307\.14440,[Link](https://arxiv.org/abs/2307.14440)Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px1.p1.2),[Realization: Generating a Target Move](https://arxiv.org/html/2607.26928#Sx5.SSx2.p1.1)\.
- H\. Sacks, E\. A\. Schegloff, and G\. Jefferson \(1974\)A simplest systematics for the organization of turn\-taking for conversation\.language50\(4\),pp\. 696–735\.Cited by:[Dialogue State, Policy, and Conversation Analysis\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px1.p1.1)\.
- E\. A\. Schegloff, G\. Jefferson, and H\. Sacks \(1977\)The preference for self\-correction in the organization of repair in conversation\.Language53\(2\),pp\. 361–382\.Cited by:[Dialogue State, Policy, and Conversation Analysis\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px1.p1.1)\.
- O\. Shaikh, K\. Gligorić, A\. Khetan, M\. Gerstgrasser, D\. Yang, and D\. Jurafsky \(2024\)Grounding gaps in language model generations\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6279–6296\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1),[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- G\. Skantze \(2017\)Towards a general, continuous model of turn\-taking in spoken dialogue using lstm recurrent neural networks\.InProceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue,pp\. 220–230\.Cited by:[Turn\-Boundary Subspace](https://arxiv.org/html/2607.26928#Sx6.p1.1)\.
- A\. Stolfo, V\. Balachandran, S\. Yousefi, E\. Horvitz, and B\. Nushi \(2025\)Improving instruction\-following in language models through activation steering\.External Links:2410\.12877,[Link](https://arxiv.org/abs/2410.12877)Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px2.p1.1),[Realization: Generating a Target Move](https://arxiv.org/html/2607.26928#Sx5.SSx2.p1.1)\.
- N\. Subramani, N\. Suresh, and M\. E\. Peters \(2022\)Extracting latent steering vectors from pretrained language models\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 566–581\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- J\. Teoh, M\. Tomar, K\. Ahn, E\. S\. Hu, T\. Pearce, P\. Sharma, A\. Krishnamurthy, R\. Islam, A\. Lamb, and J\. Langford \(2025\)Next\-latent prediction transformers learn compact world models\.arXiv preprint arXiv:2511\.05963\.Cited by:[Dialogue State, Policy, and Conversation Analysis\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px1.p1.1)\.
- E\. Todd, M\. Li, A\. Sen Sharma, A\. Mueller, B\. Wallace, and D\. Bau \(2024\)Function vectors in large language models\.InInternational conference on learning representations,Vol\.2024,pp\. 17282–17333\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- D\. R\. Traum and E\. A\. Hinkelman \(1992\)Conversation acts in task\-oriented spoken dialogue\.Computational intelligence8\(3\),pp\. 575–599\.Cited by:[Dialogue State, Policy, and Conversation Analysis\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px1.p1.1)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2024\)Steering language models with activation engineering, 2024\.URL https://arxiv\. org/abs/2308\.102482308\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- B\. Veluri, B\. N\. Peloquin, B\. Yu, H\. Gong, and S\. Gollakota \(2024\)Beyond turn\-based interfaces: synchronous llms as full\-duplex dialogue agents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 21390–21402\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p2.1)\.
- J\. D\. Williams and S\. Young \(2007\)Partially observable markov decision processes for spoken dialog systems\.Computer Speech & Language21\(2\),pp\. 393–422\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p1.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz,et al\.\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations,pp\. 38–45\.Cited by:[Appendix B](https://arxiv.org/html/2607.26928#A2.SSx1.p1.1)\.
- J\. Xu, Z\. Guo, J\. He, H\. Hu, T\. He, S\. Bai, K\. Chen, J\. Wang, Y\. Fan, K\. Dang, B\. Zhang, X\. Wang, Y\. Chu, and J\. Lin \(2025a\)Qwen2\.5\-omni technical report\.External Links:2503\.20215,[Link](https://arxiv.org/abs/2503.20215)Cited by:[Results](https://arxiv.org/html/2607.26928#Sx5.p1.1)\.
- J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.\(2025b\)Qwen3\-omni technical report\.arXiv preprint arXiv:2509\.17765\.Cited by:[Results](https://arxiv.org/html/2607.26928#Sx5.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix G](https://arxiv.org/html/2607.26928#A7.SSx1.p1.4),[Automatic Move Classifier](https://arxiv.org/html/2607.26928#Sx4.SSx3.p1.7)\.
- K\. Yang and D\. Klein \(2021\)FUDGE: controlled text generation with future discriminators\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3511–3535\.External Links:[Link](http://dx.doi.org/10.18653/v1/2021.naacl-main.276),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.276)Cited by:[Appendix A](https://arxiv.org/html/2607.26928#A1.SSx3.SSS0.Px3.p1.5),[Realization: Generating a Target Move](https://arxiv.org/html/2607.26928#Sx5.SSx2.p1.1)\.
- Y\. Yegorova, A\. Gerogiannis, H\. Zheng, J\. Hockenmaier, C\. D\. Yoo, and M\. A\. Hasegawa\-Johnson \(2026\)SALSA: speech aware llm adaptation via learned steering activation vectors\.arXiv preprint arXiv:2606\.00460\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.
- S\. Young, M\. Gašić, B\. Thomson, and J\. D\. Williams \(2013\)Pomdp\-based statistical spoken dialog systems: a review\.Proceedings of the IEEE101\(5\),pp\. 1160–1179\.Cited by:[Introduction](https://arxiv.org/html/2607.26928#Sx1.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.\(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[LLMs and Activation Steering\.](https://arxiv.org/html/2607.26928#Sx2.SS0.SSS0.Px2.p1.1)\.

## Appendix AImplementation Details and Prompts

### Generation Prompts

We use the same prompt structure across datasets, with dataset\-specific role and task descriptions\. The placeholder\{CONTEXT\}contains the preceding dialogue transcript, while\{CONTEXT\_AUDIO\}contains the aligned context audio when available\.

#### MapTask\.

System:You are the follower in a collaborative MapTask dialogue\. Your partner, the giver, describes a route on a map\. Follow their instructions and ask for clarification when needed\.User:\[AUDIO: \{CONTEXT\_AUDIO\}\]\{CONTEXT\}

#### FindTask\.

System:You are the helper in a collaborative object\-finding dialogue\. Your partner, the elder, directs you to locate objects in a kitchen\. Follow their instructions and ask for clarification when needed\.User:\{CONTEXT\}

#### CReST\.

System:You are the searcher in a collaborative remote\-search dialogue\. Your partner, the director, guides you through a series of rooms to locate numbered boxes\. Follow their directions and ask for clarification when needed\.User:\[AUDIO: \{CONTEXT\_AUDIO\}\]\{CONTEXT\}

### Automatic Move Classifier

The automatic evaluator classifies the generated follower turn into one of five moves:

ℳ=\{acknowledge,check,explain,query,reply\}\.\\mathcal\{M\}=\\\{\\texttt\{acknowledge\},\\texttt\{check\},\\texttt\{explain\},\\texttt\{query\},\\texttt\{reply\}\\\}\.Decoding is constrained to these five label tokens\. To remove dataset\-specific cues, speaker prefixes are normalized toAandB\. The classifier receives the immediately preceding turn, its move label when available, and the follower response being evaluated\.

System:Two people are doing a task together by talking\. Read the recent exchange and classify the FINAL turn by what it is doing in the conversation\. Choose exactly one:acknowledge– only signals that the speaker heard or registered the other; adds no new information and asks nothing \(“okay,” “mm\-hmm,” “right,” “uh\-huh,” “I see”\)\.check– confirms something already said or inferred by restating, paraphrasing, or pointing to it \(“so I go left?”, “the blue one?”, “this one?”\)\.explain– volunteers information about the speaker’s own situation, view, or what they can or cannot see that the other did not ask for \(“I don’t have that,” “there’s a box in front of me”\)\.query– asks for new information the other has not yet given \(“where do I go?”, “how many are there?”\)\.reply– answers a question the other speaker just asked \(“yes,” “no,” “it’s on the left”\)\.Use the preceding turn to decide\. If it asked a question or made a check, the final turn is normally areply, even when it contains only one word such as “yes,” “no,” or “okay\.” An exception occurs when the final speaker ignores the question and instead asks their own question or volunteers information about their situation\.If the preceding turn was an instruction or statement, the final turn is not areply; classify it as anacknowledge,check,query, orexplainbased on its content\.A short token such as “yeah,” “right,” or “okay” is areplywhen it answers a question, but anacknowledgewhen it merely registers an instruction or statement\.A turn can be acheckeven without explicit question syntax when it repeats or reformulates specific preceding information to confirm it, as in “to the right,” “past the door,” or “the blue box then\.” A single bare word should not be treated as a check\.

User:Recent exchange:\{CONTEXT\}The preceding turn was labeled:\{PRECEDING\_MOVE\}Final turn to classify:“\{MODEL\_RESPONSE\}”Answer with exactly one word from:acknowledge,check,explain,query, orreply\.Answer:

### Baselines

Prompt\-Based Learning, Steering\-Instruct, and FUDGE receive the target move in the oracle\-realization setting\. SFT is an end\-to\-end baseline and receives no move label during training or inference\.

#### Prompt\-Based Learning \(PBL\)\.

PBL\(Ramirezet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib18)\)prependsN=10N=10in\-context demonstrations sampled from the training split using seed4242\. Each demonstration contains at most the three most recent context turns\. Demonstrations are sampled at run time rather than drawn from a single fixed set\.

The default target\-shaped template is:

Here is a dialogue context:\{CONTEXT\}The follower’s next response, written as a ‘\{MOVE\}’ dialogue act: \{RESPONSE\}

At test time, the same template is used but ends immediately afterdialogue act:\. Thedefinitionalvariant prepends the target move’s definition\. Thefewshotvariant instead uses:

Context:\{CONTEXT\}Response \(\{MOVE\}\):\{RESPONSE\}

For each example, the model samplesK=10K=10candidates using temperature0\.70\.7, top\-p=1\.0p=1\.0, and at most6060new tokens\. Among candidates that the move classifier assigns to the target move, PBL selects the one with the highest mean token log\-probability\. If no candidate is assigned to the target move, it selects

S^=arg⁡maxS⁡Pcls​\(m∣ct,S\)​PG​\(S∣ct\)\.\\hat\{S\}=\\arg\\max\_\{S\}P\_\{\\mathrm\{cls\}\}\(m\\mid c\_\{t\},S\)\\,P\_\{G\}\(S\\mid c\_\{t\}\)\.

#### Steering\-Instruct\.

Steering\-Instruct\(Stolfoet al\.[2025](https://arxiv.org/html/2607.26928#bib.bib19)\)contrasts a bare dialogue context against three move\-conditioned prompt variants:

Your next response should \{DEFINITION\}\.Respond in a way that will \{DEFINITION\}\.Make sure your reply will \{DEFINITION\}\.

Using200200training contexts sampled with seed4242, we collect the last\-input\-token residual at every layer for both the base and instructed prompts\. At layerℓ\\ell, the direction is

vmℓ=μinstr,mℓ−μbaseℓ,umℓ=vmℓ∥vmℓ∥2\.v\_\{m\}^\{\\ell\}=\\mu\_\{\\mathrm\{instr\},m\}^\{\\ell\}\-\\mu\_\{\\mathrm\{base\}\}^\{\\ell\},\\qquad u\_\{m\}^\{\\ell\}=\\frac\{v\_\{m\}^\{\\ell\}\}\{\\lVert v\_\{m\}^\{\\ell\}\\rVert\_\{2\}\}\.We also store the target projection

z¯mℓ=⟨μinstr,mℓ,umℓ⟩\.\\bar\{z\}\_\{m\}^\{\\ell\}=\\left\\langle\\mu\_\{\\mathrm\{instr\},m\}^\{\\ell\},u\_\{m\}^\{\\ell\}\\right\\rangle\.
During generation, a projection\-matching hook is applied atℓ=⌊L/2⌋\\ell=\\lfloor L/2\\rfloor\. At each generated token, the intervention coefficient is recomputed so that the residual’s projection ontoumℓu\_\{m\}^\{\\ell\}matchesz¯mℓ\\bar\{z\}\_\{m\}^\{\\ell\}\. We use gain1\.01\.0and greedy decoding\. The move instruction is used only to construct the direction and is not included in the generation prompt\.

#### FUDGE\.

FUDGE\(Yang and Klein[2021](https://arxiv.org/html/2607.26928#bib.bib20)\)uses a lightweight future discriminator consisting of a128128\-dimensional token embedding, a single\-layer LSTM with hidden dimension256256, and a linear output layer over the five moves\. The discriminator is trained on response prefixes from the training dialogues to predict the completed response’s move\. We apply class\-balanced cross\-entropy at every prefix position and train using Adam with learning rate10−310^\{\-3\}, batch size6464, and88epochs\. Ten percent of the training dialogues are reserved for validation, and the checkpoint with the highest macro accuracy is retained\.

At decoding stepii, the generator’s topK=200K=200tokens are scored as

score⁡\(xi\)=log⁡PG​\(xi∣x<i,ct\)\+λ​log⁡PB​\(m∣x≤i,ct\),\\operatorname\{score\}\(x\_\{i\}\)=\\log P\_\{G\}\(x\_\{i\}\\mid x\_\{<i\},c\_\{t\}\)\+\\lambda\\log P\_\{B\}\(m\\mid x\_\{\\leq i\},c\_\{t\}\),whereλ=1\\lambda=1\. The highest\-scoring token is selected greedily, without sampling or beam search\.

#### Supervised Fine\-Tuning \(SFT\)\.

SFT is an end\-to\-end baseline that fine\-tunes the backbone on context–response pairs,

ct⟶ut,c\_\{t\}\\longrightarrow u\_\{t\},without providing a move label during training or inference\(Ouyanget al\.[2022](https://arxiv.org/html/2607.26928#bib.bib52)\)\.

We apply LoRA with rankr=16r=16, scaling parameter3232, dropout0\.050\.05, and all linear layers as target modules\(Huet al\.[2022](https://arxiv.org/html/2607.26928#bib.bib53)\)\. Training is text\-only and uses up to1,0001\{,\}000training pairs, with loss computed only over response tokens\. We use AdamW with learning rate2×10−42\\times 10^\{\-4\}, three epochs, per\-device batch size11, gradient accumulation over1616steps, BF16 precision, and seed4242\. The final\-epoch weights are retained without validation\-based checkpoint selection\. For Qwen2\.5\-Omni, approximately6565M of9\.09\.0B parameters are trainable, corresponding to0\.72%0\.72\\%\.

#### DeAL\.

DeAL\(Huanget al\.[2025](https://arxiv.org/html/2607.26928#bib.bib16)\)is a decoding\-time alignment baseline that performs anA∗A^\{\\ast\}\-style search—top\-kkexpansion, greedy lookahead, and a self\-reward heuristic—toward the target move, without modifying weights or activations\. It is text\-only\. Optionally the context is augmented with an alignment instructionpap\_\{a\},

\{CONTEXT\}Respond with a \{MOVE\} move: \{DEFINITION\}

but in our main runs this instruction is dropped \(\-\-no\-align\-prompt\) so that alignment is driven purely by the search reward and does not double\-count the move\-classifier signal\. At each committed step we \(i\) take the generator’s topK=5K=5next\-token candidates with their log\-probabilities, \(ii\) extend each candidate with a greedy lookahead ofL=16L=16tokens, and \(iii\) score a self\-reward

hj=log⁡PG​\(m∣ct,committed⊕candj⊕lookahead\),h\_\{j\}=\\log P\_\{G\}\\\!\\left\(m\\mid c\_\{t\},\\,\\text\{committed\}\\oplus\\text\{cand\}\_\{j\}\\oplus\\text\{lookahead\}\\right\),read from the*same*generation model via A–E letter log\-probabilities over the five moves\. We commit

ȷ^=arg⁡maxj⁡\[log⁡PG​\(candj\)\+λ​hj\],\\hat\{\\jmath\}=\\arg\\max\_\{j\}\\ \\bigl\[\\log P\_\{G\}\(\\text\{cand\}\_\{j\}\)\+\\lambda\\,h\_\{j\}\\bigr\],advancingstride=4\\text\{stride\}=4tokens per decision, withλ=0\.5\\lambda=0\.5and at most4040new tokens\. Decoding is greedy; there is no sampling or beam search\.

#### PAS\.

PAS \(Painless Activation Steering\) is a self\-supervised, end\-to\-end baseline that builds a corrective steering vector without any external supervision\(Cui and Chen[2026](https://arxiv.org/html/2607.26928#bib.bib17)\)\. We run the unmodified generation model on the training split’s next\-move\-prediction task,

The follower’s possible moves are: \- acknowledge: \{DEFINITION\} … \- reply: \{DEFINITION\}Dialogue so far:\{CONTEXT\}The follower \(F\) speaks next\. Which single move type will it be?Answer with exactly one of: acknowledge, check, explain, query, reply\.Answer:

and partition up to1,0001\{,\}000training contexts \(seed4242\) into correct and incorrect by the model’s*own*parsed answer versus the gold move\. The vector at layerℓ\\ellis the raw difference of mean last\-token residuals between a positive and a negative prompt set, captured at all layers,

aℓ∗=meanP\+\(hℓ,last token\)−meanP−\(hℓ,last token\)\.a^\{\\ast\}\_\{\\ell\}=\\operatorname\*\{mean\}\_\{P\_\{\+\}\}\\\!\\bigl\(h\_\{\\ell\},\\ \\text\{last token\}\\bigr\)\-\\operatorname\*\{mean\}\_\{P\_\{\-\}\}\\\!\\bigl\(h\_\{\\ell\},\\ \\text\{last token\}\\bigr\)\.We use three variants:PASf, whoseP\+/P−P\_\{\+\}/P\_\{\-\}are the full multiple\-choice prompts for correctly/incorrectly answered contexts;iPASa, whose prompts append the model’s chosen move and are partitioned by correctness; andiPASwo, built from incorrect contexts only, withP\+P\_\{\+\}appending the correct move andP−P\_\{\-\}the chosen wrong move \(the default\)\. At generation the vector is added additively at a single layerℓ=⌊L/2⌋\\ell=\\lfloor L/2\\rfloorwith strengthλ=1\.0\\lambda=1\.0, using greedy decoding\.

## Appendix BCompute

Table 7:Hyperparameters used for decoding\.We use vLLM\(Kwonet al\.[2023](https://arxiv.org/html/2607.26928#bib.bib14)\)for inference\. Parameters are shown in Table[7](https://arxiv.org/html/2607.26928#A2.T7)\. All inference was run on Nvidia A40 GPUs with 48GB GDDR6 memory\. We use vLLM tensor parallelism across 2, 4, or 8 GPUs depending on model size\. Runtime ranged from a few minutes to about one hour per \(model, dataset\) evaluation, depending on model size and dataset\.

### Models

We evaluate three instruction\-tuned multimodal language models: Qwen2\.5\-Omni, Qwen3\-Omni, and Phi\-4\-Multimodal, using the following Hugging Face checkpoints without modification\(Wolfet al\.[2020](https://arxiv.org/html/2607.26928#bib.bib15)\)\.

Table 8:Instruction\-tuned models evaluated, with special tokens used for prompt construction\.

## Appendix CDatasets and Preprocessing

### Corpus Statistics and Splits

We evaluate on three task\-oriented dialogue corpora in which one participant issues instructions—the*giver*,*elder*, or*director*—and the other executes them—the*follower*,*helper*, or*searcher*\. We normalize these roles to*giver*and*follower*\. The follower turn is the unit of prediction and steering, and each retained follower turn constitutes one example\.

Table 9:Dataset statistics after label mapping and filtering\. Splits are grouped by dialogue\. “Turns” counts retained follower turns whose labels map to one of the five target moves\.All experiments use a single canonical dialogue\-grouped split\. We hold out20%20\\%of dialogues for testing, ensuring that no dialogue appears in more than one split\. Dialogue identifiers are sorted and sampled without replacement usingnumpy\.default\_rng\(42\), with

ntest=max⁡\(1,⌊0\.2​\|𝒟\|⌋\)\.n\_\{\\mathrm\{test\}\}=\\max\\\!\\left\(1,\\left\\lfloor 0\.2\|\\mathcal\{D\}\|\\right\\rfloor\\right\)\.This produces2525MapTask test dialogues,2626FindTask test dialogues, and33CReST test dialogues\.

Steering\-vector estimation, controller training, baseline training, and end\-to\-end evaluation all use this split\. No separate development set is reserved for the main experiments\. When an auxiliary validation set is needed, we sample10%10\\%of the training dialogues: seed4242for the FUDGE discriminator and seed123123for the controller\.

### Move\-Label Mapping

We map corpus\-specific dialogue\-act labels into the shared move set

ℳ=\{acknowledge,check,explain,query,reply\}\.\\mathcal\{M\}=\\\{\\textsc\{acknowledge\},\\textsc\{check\},\\textsc\{explain\},\\textsc\{query\},\\textsc\{reply\}\\\}\.Only turns mapped to these five moves are used as prediction or generation targets\. Excluded turns remain in the preceding dialogue context unless otherwise noted\.

Table 10:Mapping from corpus\-specific dialogue\-act labels to the shared five\-move taxonomy\. Excluded labels are not used as target turns but are retained as context unless removed by the preprocessing rules below\.
### Text and Audio Preprocessing

#### Turn filtering\.

We remove giver, elder, and director turns from the target set, although they remain available as dialogue context\. We additionally exclude turns with empty text and follower turns with no preceding context\. MapTask turns labeleduncodableare removed\.

For CReST, filler\-only turns are removed from both the target set and dialogue context:

\{um,uh,and,un,wait,whoops,and\_um\}\.\\\{\\texttt\{um\},\\texttt\{uh\},\\texttt\{and\},\\texttt\{un\},\\texttt\{wait\},\\texttt\{whoops\},\\texttt\{and\\\_um\}\\\}\.We also repair known transcription errors, remove inline annotations enclosed in braces, brackets, or parentheses, and use the primary label preceding “/” when multiple tags are present\.

#### Audio processing\.

MapTask and CReST use audio and transcript input, while FindTask is text\-only\. Dialogue waveforms are loaded from<id\>\.mix\.wav, converted to mono, and resampled to16​kHz16\\,\\mathrm\{kHz\}\.

For generation, the audio segment begins0\.2​s0\.2\\,\\mathrm\{s\}before the onset of the immediately preceding turn and ends at the onset of the target follower response, ensuring that the held\-out response is never included\. Segments are capped at30​s30\\,\\mathrm\{s\}, retaining the most recent audio when the available context is longer\. For teacher\-forced activation extraction, the segment extends through the end of the gold follower response\.

When a waveform is unavailable, we substitute0\.1​s0\.1\\,\\mathrm\{s\}of silence to preserve the expected number of audio placeholders\.

#### Prompt construction\.

For multimodal conditions, the user input contains the audio item followed by the corresponding transcript\. Transcript context contains the most recentnnturns when a context limit is specified and the full preceding dialogue otherwise\. Audio\+text and text\-only conditions are stored and evaluated separately\. The automatic move classifier always receives text\-only input\.

#### Speaker normalization\.

Corpus\-specific roles are rendered in generation prompts usingG:for the giver role andF:for the follower role\. For automatic classification, these prefixes are further anonymized:

G:→A:,F:→B:\.\\texttt\{G:\}\\rightarrow\\texttt\{A:\},\\qquad\\texttt\{F:\}\\rightarrow\\texttt\{B:\}\.The speaker prefix is removed from the final response being classified, reducing dataset\- and role\-specific cues\.

## Appendix DController Details

Frozen spoken dialogue LMper\-turn poolxt∈ℝdx\_\{t\}\\in\\mathbb\{R\}^\{d\}selective SSMzt→otz\_\{t\}\\\!\\to\\\!o\_\{t\}movey^t\+1\\hat\{y\}\_\{t\+1\}lengthL^t\+1\\widehat\{L\}\_\{t\+1\}direction selection, gating,and strength calibrationsteer:vm​@​ℓm⋆v\_\{m\}@\\,\\ell\_\{m\}^\{\\star\}\+\+veot​@​ℓeot⋆v\_\{\\textsc\{eot\}\}@\\,\\ell\_\{\\textsc\{eot\}\}^\{\\star\}hth\_\{t\}next turn

Figure 4:The controller reads the frozen backbone’s residual stream and predicts the next conversational move and response length\. The predicted move selects a move direction, its fixed steering strength, and the realization\-aware gate; the predicted length is calibrated to a turn\-boundary coefficient\. All decisions depend on controller predictions rather than gold labels at inference\.#### Per\-turn feature\.

For each dialogue turntt, we mean\-pool the residual\-stream activations at controller layerℓctrl⋆\\ell\_\{\\mathrm\{ctrl\}\}^\{\\star\}over the turn’s token span and standardize them using training\-set statistics:

xt=\(ϕ​\(htℓctrl⋆\)−μ\)⊘σ∈ℝd,x\_\{t\}=\\left\(\\phi\\\!\\left\(h\_\{t\}^\{\\ell\_\{\\mathrm\{ctrl\}\}^\{\\star\}\}\\right\)\-\\mu\\right\)\\oslash\\sigma\\in\\mathbb\{R\}^\{d\},\(9\)whereϕ\\phidenotes mean pooling,\(μ,σ\)\(\\mu,\\sigma\)are the per\-coordinate training\-set mean and standard deviation, and⊘\\oslashdenotes elementwise division\.

#### Objective\.

For each retained follower turntt, the causal state after turnt−1t\-1predicts its move and log response length:

ℒ=∑t∈ℐtrain\(CE\(Woot−1,yt\)\+λsmoothL1\(wL⊤ot−1,Lt\)\),\\mathcal\{L\}=\\sum\_\{t\\in\\mathcal\{I\}\_\{\\mathrm\{train\}\}\}\\Big\(\\mathrm\{CE\}\\\!\\left\(W\_\{o\}o\_\{t\-1\},y\_\{t\}\\right\)\\\\ \+\\lambda\\,\\mathrm\{smooth\}L\_\{1\}\\\!\\left\(w\_\{L\}^\{\\top\}o\_\{t\-1\},L\_\{t\}\\right\)\\Big\),\(10\)where

Lt=log⁡\(1\+words⁡\(ut\)\),L\_\{t\}=\\log\\\!\\left\(1\+\\operatorname\{words\}\(u\_\{t\}\)\\right\),andℐtrain\\mathcal\{I\}\_\{\\mathrm\{train\}\}indexes retained training\-set follower turns whose labels belong toℳ\\mathcal\{M\}\.

#### Length calibration\.

At inference, the predicted log length

L^t\+1=wL⊤​ot\\widehat\{L\}\_\{t\+1\}=w\_\{L\}^\{\\top\}o\_\{t\}is mapped to a turn\-boundary steering coefficientβ\\betaby a monotone calibration fitted on the training dialogues\. The coefficient is applied only to naturally terse moves\. For content\-bearing moves \(checkandexplain\), it is set to zero because shortening strong enough to affect these moves can overwhelm the move direction\.

#### Streaming inference\.

At deployment, the controller performs one incremental update

\(ot,zt\)=step⁡\(xt,zt−1\)\(o\_\{t\},z\_\{t\}\)=\\operatorname\{step\}\(x\_\{t\},z\_\{t\-1\}\)per dialogue turn\. Its recurrent state persists across turns, so the controller does not recompute its previous states\. The update has constant time and memory with respect to dialogue history and is equivalent to the parallel scan used during training, up to a numerical difference below10−710^\{\-7\}\. Thus, the controller operates as a streaming policy rather than an offline classifier\.

## Appendix ESteering Implementation

#### Onset features\.

For each retained follower turnt∈ℐtraint\\in\\mathcal\{I\}\_\{\\mathrm\{train\}\}, we extract the move\-onset delta

Δtℓ=htℓ​\(st,1\)−htℓ​\(st,1−1\),\\Delta\_\{t\}^\{\\ell\}=h\_\{t\}^\{\\ell\}\(s\_\{t,1\}\)\-h\_\{t\}^\{\\ell\}\(s\_\{t,1\}\-1\),the change in the layer\-ℓ\\ellresidual stream between the positions immediately before and at the first response token \(Eq\.[3](https://arxiv.org/html/2607.26928#Sx4.E3)\)\. Activations are extracted under teacher forcing\. For each target movemm, we form a balanced one\-vs\-rest set containing all turns labeledmmand an equal\-size random sample of the remaining retained follower turns\.

#### Direction estimation \(deployed\)\.

The deployed steering direction is the PCA\-reduced, shrinkage\-LDA \(Fisher\) direction of Eq\.[4](https://arxiv.org/html/2607.26928#Sx4.E4), mapped back to the raw residual stream\. On the balanced one\-vs\-rest set we \(i\) standardize the onset features per dimension \(StandardScaler\), \(ii\) reduce tok=min⁡\(128,n−1,d\)k=\\min\(128,\\,n\{\-\}1,\\,d\)principal components, and \(iii\) fitLinearDiscriminantAnalysis\(solver="lsqr", shrinkage="auto"\), whoselsqrcoefficient vector is proportional toΣ^w−1​\(νmℓ−ν¬mℓ\)\\widehat\{\\Sigma\}\_\{w\}^\{\-1\}\(\\nu\_\{m\}^\{\\ell\}\-\\nu\_\{\\lnot m\}^\{\\ell\}\)with the within\-class covarianceΣ^w\\widehat\{\\Sigma\}\_\{w\}regularized by automatic Ledoit–Wolf shrinkage\(Ledoit and Wolf[2004](https://arxiv.org/html/2607.26928#bib.bib32)\)\. Letwmℓw^\{\\ell\}\_\{m\}be this direction in standardized\-PCA coordinates\. We map it back to raw activation space, undoing the PCA rotation and the per\-dimension scaling,

v~mℓ=diag⁡\(\(σℓ\)−1\)​\(Pℓ\)⊤​wmℓ,vmℓ=v~mℓ∥v~mℓ∥2,\\tilde\{v\}\_\{m\}^\{\\ell\}=\\operatorname\{diag\}\\\!\\left\(\(\\sigma^\{\\ell\}\)^\{\-1\}\\right\)\(P^\{\\ell\}\)^\{\\top\}w\_\{m\}^\{\\ell\},\\qquad v\_\{m\}^\{\\ell\}=\\frac\{\\tilde\{v\}\_\{m\}^\{\\ell\}\}\{\\lVert\\tilde\{v\}\_\{m\}^\{\\ell\}\\rVert\_\{2\}\},\(11\)wherePℓP^\{\\ell\}are the PCA components andσℓ\\sigma^\{\\ell\}the per\-dimension standard deviations of the standardizer \(elementwise division\)\. The result is unit\-normed in the raw residual space \(d=3584d\\\!=\\\!3584for Qwen2\.5\-Omni\)\. The vector is oriented toward the target move by construction \(LDA class 1==move\), so no separate sign\-fixing rule is applied\. Directions are estimated independently per backbone\.

#### Injection\.

Steering is applied additively to the post\-block residual stream at the selected move\-specific layer\. For every generated token positionss,

hℓm⋆​\(s\)←hℓm⋆​\(s\)\+αm⋆​cℓm⋆​vmℓm⋆\.h^\{\\ell\_\{m\}^\{\\star\}\}\(s\)\\leftarrow h^\{\\ell\_\{m\}^\{\\star\}\}\(s\)\+\\alpha\_\{m\}^\{\\star\}c\_\{\\ell\_\{m\}^\{\\star\}\}v\_\{m\}^\{\\ell\_\{m\}^\{\\star\}\}\.\(12\)Here,αm⋆\\alpha\_\{m\}^\{\\star\}is the selected move\-specific steering coefficient andcℓm⋆c\_\{\\ell\_\{m\}^\{\\star\}\}is the medianL2L\_\{2\}norm of the residual stream at layerℓm⋆\\ell\_\{m\}^\{\\star\}, estimated over generated tokens from300300MapTask training contexts\. This normalization makesαm⋆\\alpha\_\{m\}^\{\\star\}dimensionless\. The intervention is applied only during decoding, not to prompt\-prefill positions, and uses constant strength across generated tokens with no positional decay\.

#### Layer and strength selection\.

For each movemmand backbone, we selectℓm⋆\\ell\_\{m\}^\{\\star\}andαm⋆\\alpha\_\{m\}^\{\\star\}using only MapTask training dialogues through a two\-stage sweep\. Stage 1 evaluates all layers at a fixedα=0\.1\\alpha=0\.1and selects the layer maximizing the mean paired*soft lift*

𝔼c∈𝒞¬mtrain​\[Psteered​\(m∣c\)−Pbase​\(m∣c\)\],\\mathbb\{E\}\_\{c\\in\\mathcal\{C\}\_\{\\neg m\}^\{\\mathrm\{train\}\}\}\\left\[P\_\{\\mathrm\{steered\}\}\(m\\mid c\)\-P\_\{\\mathrm\{base\}\}\(m\\mid c\)\\right\],where𝒞¬mtrain\\mathcal\{C\}\_\{\\neg m\}^\{\\mathrm\{train\}\}contains MapTask training contexts whose gold move is notmm, andPPis obtained from the Qwen2\.5\-72B move classifier\. Stage 2 fixes the selected layer and sweepsα∈\{0\.05,0\.1,0\.2,0\.3,0\.4\}\\alpha\\in\\\{0\.05,0\.1,0\.2,0\.3,0\.4\\\}, choosing the value with the greatest soft lift among those whose degenerate\-generation rate is at most0\.100\.10\. Directions and hyperparameters are estimated independently for each backbone\. The resultingvmℓm⋆v\_\{m\}^\{\\ell\_\{m\}^\{\\star\}\},ℓm⋆\\ell\_\{m\}^\{\\star\}, andαm⋆\\alpha\_\{m\}^\{\\star\}are then transferred unchanged to held\-out MapTask, FindTask, and CReST\.

#### Adaptive strength\.

As an optional variant, we modulate the move\-steering strength for each example using its alignment with the target direction:

scale=αm⋆​cℓm⋆​\(1\+γ​zc,m\),\\operatorname\{scale\}=\\alpha\_\{m\}^\{\\star\}c\_\{\\ell\_\{m\}^\{\\star\}\}\\left\(1\+\\gamma z\_\{c,m\}\\right\),wherezc,mz\_\{c,m\}is the alignment of the context representation atℓm⋆\\ell\_\{m\}^\{\\star\}withvmℓm⋆v\_\{m\}^\{\\ell\_\{m\}^\{\\star\}\}, standardized using the training\-set mean and standard deviation\. The coefficientγ\\gammacontrols the adaptive gain;γ=0\\gamma=0recovers the fixed\-strength intervention\.

## Appendix FTurn\-Boundary Subspace and Realization\-Aware Gating

#### Predicted length\.

The controller regresses the follower turn’s log length

Lt\+1=log⁡\(1\+words⁡\(ut\+1\)\)\.L\_\{t\+1\}=\\log\\\!\\left\(1\+\\operatorname\{words\}\(u\_\{t\+1\}\)\\right\)\.At inference, the prediction

L^t\+1=wL⊤​ot\\widehat\{L\}\_\{t\+1\}=w\_\{L\}^\{\\top\}o\_\{t\}is mapped to the turn\-boundary coefficientβ\\betaby a monotone calibration fitted on training dialogues\.

Table 11:Realization\-aware gating policy\.

## Appendix GHuman Study

We conduct a human study to validate the automatic move classifier against human judgment on model\-generated responses, and to measure agreement between human annotators and the LLM\-based evaluator\. From the pooled generations of all methods, we sample500500responses—100100for each of the five target moves, distributed approximately equally across the three generation models and three datasets\. Each response is annotated by three workers who see only the dialogue context and the generated utterance and select one conversational move; the target move, generation method, and classifier prediction are withheld so that annotation is blind\.

#### Annotators\.

All annotations were collected on Amazon Mechanical Turk \(AMT\) using workers with a minimum approval rate of 95%\. Annotators were restricted to English\-speaking countries to ensure reliable comprehension\. Compensation was set to meet or exceed the U\.S\. federal minimum wage based on conservative task\-time estimates\. No personally identifiable information was collected\.

### Human–LLM Agreement

We compare human judgments to the LLM move classifier at the item level\. Human decisions are aggregated by plurality voting across the three annotators, and the LLM prediction is the label assigned by the two\-stage Qwen2\.5\-72B\-Instruct classifier\(Yanget al\.[2025](https://arxiv.org/html/2607.26928#bib.bib51)\)\. Agreement is reported per dataset in Table[12](https://arxiv.org/html/2607.26928#A7.T12): overall, the classifier matches the human plurality on73\.9%73\.9\\%of items, rising to90\.5%90\.5\\%on the327327items where all three annotators agreed\. Inter\-annotator agreement is moderate \(Fleiss’κ=0\.682\\kappa=0\.682overall\), and the classifier’s agreement with the human majority is comparable to the agreement among annotators themselves\.

Table 12:Agreement between the automatic classifier and human judgments\. Plurality agreement uses items with a clear majority; unanimous agreement uses only items on which all three annotators selected the same move\.To assess alignment at the distribution level, we compare the classifier’s normalized move probabilities with the human vote shares across all item–move pairs\. The two are positively correlated \(Pearsonr=0\.721r=0\.721overall, ranging fromr=0\.62r=0\.62forexplaintor=0\.85r=0\.85forquery\), indicating that the classifier’s confidence tracks human uncertainty\. Moreover, ranking the seven generation methods by human\-judged target\-move realization reproduces the classifier\-based ranking almost exactly \(Spearmanρ=0\.96\\rho=0\.96, Pearsonr=0\.99r=0\.99\), showing that the two evaluators agree not only per item but also in their relative assessment of methods\. These results support the use of the LLM move classifier as a scalable proxy for human judgment in our evaluation\.

### MTurk Interfaces

Figure[5](https://arxiv.org/html/2607.26928#A7.F5)shows the annotation interface presented to workers\.

![Refer to caption](https://arxiv.org/html/2607.26928v1/Figures/form.png)Figure 5:Human annotation interface for move classification\. Annotators are shown the preceding dialogue context and a single candidate response, and asked to select the conversational move the response performs \(acknowledge, check, explain, query, or reply\)\. The target move, generation method, and classifier prediction are hidden\.