TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

# TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue
Source: [https://arxiv.org/html/2607.01345](https://arxiv.org/html/2607.01345)
Hao Zhang1,†\\dagger, Thomas Thebaud1, Georgi Tinchev2, Venkatesh Ravichandran3, Laureano Moro\-Velázquez1,†\\dagger

###### Abstract

Turn\-taking naturalness is central to full\-duplex spoken dialogue systems, yet its automatic evaluation remains limited\. Existing evaluations often rely on human judgments or behavior\-specific timing metrics, making it difficult to compare heterogeneous timing failures within a unified framework\. We propose TurnNat, a likelihood\-based framework for automatic turn\-taking naturalness evaluation in two\-channel spoken dialogue\. A causal turn\-taking prediction model trained on natural conversations estimates future two\-speaker voice\-activity states, and the negative log\-likelihood \(NLL\) of the observed future activity measures timing atypicality\. TurnNat pools frame\-level NLLs over turn\-taking boundary units \(TBUs\) extracted from utterance onsets and offsets, and aggregates mean and tail TBU scores into a dialogue\-level naturalness score\. We further construct a controlled perturbation benchmark of paired natural and perturbed dialogue clips, validated by human naturalness judgments\. Experiments on this benchmark show that TurnNat successfully identifies unnatural turn\-taking perturbations across heterogeneous timing failures\.

††footnotetext:Codes and Dataset are released:https://github\.com/TedZhangHao/turn\-taking\-naturalness## IIntroduction

Natural spoken dialogue relies on fluent turn\-taking and appropriate conversational timing\. Human conversations exhibit remarkably precise temporal coordination, with turn transitions often occurring after only a few hundred milliseconds of silence\. Such coordination requires participants to continuously interpret conversational cues, anticipate upcoming turn completions, and decide whether to hold, yield, backchannel, or take the conversational floor\. In contrast, many spoken dialogue systems still rely on heuristic turn\-taking strategies, such as fixed silence thresholds, often resulting in delayed responses or premature interruptions that reduce conversational naturalness\[[4](https://arxiv.org/html/2607.01345#bib.bib2),[25](https://arxiv.org/html/2607.01345#bib.bib10)\]\.

As spoken dialogue systems move from command\-style voice interfaces toward real\-time speech\-to\-speech interaction, their success increasingly depends on how naturally they participate in the temporal flow of conversation\[[13](https://arxiv.org/html/2607.01345#bib.bib5),[6](https://arxiv.org/html/2607.01345#bib.bib3),[9](https://arxiv.org/html/2607.01345#bib.bib6),[33](https://arxiv.org/html/2607.01345#bib.bib7),[32](https://arxiv.org/html/2607.01345#bib.bib8)\]\. Recent full\-duplex spoken models aim to support low\-latency responses, overlapping speech, interruptions, and backchannels rather than treating dialogue as a sequence of isolated turns\[[6](https://arxiv.org/html/2607.01345#bib.bib3),[17](https://arxiv.org/html/2607.01345#bib.bib39)\]\. These interactional behaviors are not merely surface\-level timing details: response latency affects perceived responsiveness and conversational naturalness\[[20](https://arxiv.org/html/2607.01345#bib.bib32),[23](https://arxiv.org/html/2607.01345#bib.bib37)\], while the ability to handle interruptions and backchannels can influence users’ sense of control, trust, and willingness to rely on voice assistants\[[19](https://arxiv.org/html/2607.01345#bib.bib33),[3](https://arxiv.org/html/2607.01345#bib.bib34)\]\. Such properties are especially important for deployed dialogue systems in settings such as healthcare support, education, and customer service, where user engagement and trust are central to effective interaction\[[5](https://arxiv.org/html/2607.01345#bib.bib35),[26](https://arxiv.org/html/2607.01345#bib.bib36)\]\. Recent benchmarks have begun to evaluate interactive spoken dialogue behaviors such as pause handling, backchanneling, turn\-taking, and interruption management\[[17](https://arxiv.org/html/2607.01345#bib.bib39),[2](https://arxiv.org/html/2607.01345#bib.bib25)\]\. These efforts provide valuable diagnostics for spoken dialogue models, but their metrics are typically tied to specific interaction tasks or behavior categories\. This motivates a complementary form of evaluation: an automatic metric for turn\-taking naturalness within a single scoring framework, rather than evaluating each interactional behavior with a separate task\-specific metric\.

Turn\-taking prediction models provide a natural starting point for this form of evaluation\. This line of work estimates upcoming conversational behavior from dialogue context, including turn shifts, pauses, overlap, and backchannels\[[29](https://arxiv.org/html/2607.01345#bib.bib9),[7](https://arxiv.org/html/2607.01345#bib.bib21),[8](https://arxiv.org/html/2607.01345#bib.bib12)\]\. Models differ in the signals they use, ranging from linguistic context to speech and multimodal cues\[[7](https://arxiv.org/html/2607.01345#bib.bib21),[8](https://arxiv.org/html/2607.01345#bib.bib12),[18](https://arxiv.org/html/2607.01345#bib.bib23),[30](https://arxiv.org/html/2607.01345#bib.bib24)\]\. Recent work such as DualTurn has also explored dual\-channel generative speech pretraining, where models learn conversational dynamics by predicting both speakers’ future audio\[[22](https://arxiv.org/html/2607.01345#bib.bib29),[31](https://arxiv.org/html/2607.01345#bib.bib30)\]\. Such models are useful for naturalness evaluation because they are trained to learn regularities of natural turn\-taking dynamics\. In particular, Voice Activity Projection’s \(VAP\) formulation of turn\-taking as future two\-speaker voice\-activity prediction provides a direct probabilistic target for scoring conversational timing\[[8](https://arxiv.org/html/2607.01345#bib.bib12)\]\. We therefore take a likelihood\-based view of turn\-taking naturalness\. A model trained only on natural conversations defines a distribution over future two\-speaker voice\-activity states\. Locally unnatural timing patterns should make the observed future activity less likely under this distribution\. We define turn\-taking naturalness as the likelihood of observed local two\-speaker activity patterns under natural conversational dynamics\.

Operationally, our evaluation framework TurnNat computes frame\-level negative log\-likelihood over future two\-speaker voice\-activity states\. We then aggregate these values over automatically extracted turn\-taking boundary units \(TBUs\) to obtain an overall naturalness score\. Because the score does not require event\-type labels at test time, the same procedure can be applied across different unnatural events, such as delayed responses, early entries, floor\-transfer errors, and excessive backchannels within a shared likelihood\-based evaluation framework\. We construct a human\-validated turn\-taking perturbation benchmark consisting of paired natural and locally perturbed dialogue clips\. We then instantiate TurnNat with turn\-taking prediction models and show that it successfully identifies unnatural turn\-taking perturbations across heterogeneous timing failures\.

Our contributions are threefold:

- •We propose TurnNat, a unified likelihood\-based framework for automatic turn\-taking naturalness evaluation, using future two\-speaker voice\-activity likelihood aggregated over TBUs\.
- •We construct a human\-validated turn\-taking perturbation benchmark covering five localized turn\-taking perturbations in natural human\-to\-human dyadic spoken dialogue\.
- •We instantiate the framework with VAP and DualTurn\-based predictors, showing that future\-activity likelihood successfully distinguishes natural from perturbed timing patterns across heterogeneous turn\-taking failures\.

## IIRelated Work

### II\-ATurn\-Taking Prediction

Turn\-taking has long been studied as a central mechanism of spoken interaction, involving turn shifts and holds, pauses, interruptions, overlap, and backchannels\[[29](https://arxiv.org/html/2607.01345#bib.bib9)\]\. Earlier computational approaches often focused on specific turn\-taking decisions, such as whether a pause should lead the current speaker to hold the floor or yield it to another speaker\[[27](https://arxiv.org/html/2607.01345#bib.bib13),[21](https://arxiv.org/html/2607.01345#bib.bib14),[14](https://arxiv.org/html/2607.01345#bib.bib15)\]\. Skantze later proposed a more general continuous formulation, using recurrent neural networks to predict future speech activity over time rather than only making local hold\-or\-shift decisions\[[28](https://arxiv.org/html/2607.01345#bib.bib16)\]\. Building on this predictive view, recent models use richer dialogue context and input signals for turn\-taking prediction\. TurnGPT predicts turn shifts from linguistic context and shows that syntactic and pragmatic completeness provide useful cues for turn\-taking prediction\[[7](https://arxiv.org/html/2607.01345#bib.bib21)\]\. Speech\-based approaches instead model conversational timing directly from the audio signal\. VAP introduced future two\-speaker voice\-activity prediction as a self\-supervised target for turn\-taking modeling using voice activity detection \(VAD\) labels\[[8](https://arxiv.org/html/2607.01345#bib.bib12)\]\.

Subsequent work has extended turn\-taking prediction models across languages, interaction settings, and input modalities\. VAP\-style models have been applied to multilingual turn\-taking, prompt\-guided conversational timing, and real\-time prediction from streaming stereo audio\[[11](https://arxiv.org/html/2607.01345#bib.bib20),[10](https://arxiv.org/html/2607.01345#bib.bib22),[12](https://arxiv.org/html/2607.01345#bib.bib27)\]\. Multimodal and audio\-text models further incorporate visual, acoustic, and lexical cues for turn\-taking and backchannel prediction\[[24](https://arxiv.org/html/2607.01345#bib.bib18),[18](https://arxiv.org/html/2607.01345#bib.bib23),[30](https://arxiv.org/html/2607.01345#bib.bib24)\]\. More recently, DualTurn has explored dual\-channel generative speech pretraining for turn\-taking prediction, learning dyadic conversational dynamics from two\-speaker audio\[[22](https://arxiv.org/html/2607.01345#bib.bib29)\]\.

Taken together, these studies show that turn\-taking prediction models can learn temporally structured expectations about two\-speaker conversational behavior\. Our work repurposes these models for evaluation, using future\-activity likelihood as an automatic signal for turn\-taking naturalness\. We instantiate this idea with VAP and with DualTurn adapted to the same future\-activity prediction target\.

### II\-BEvaluation of Turn\-Taking and Spoken Dialogue Naturalness

Conversational timing has often been evaluated through human listening tests or user studies, especially when the goal is to determine whether a system response occurs at an appropriate time\. For example, Roddy and Harte\[[23](https://arxiv.org/html/2607.01345#bib.bib37)\]model dialogue response offsets and evaluate generated timings with both offline experiments and human listening tests, showing that perceived timing naturalness depends on dialogue context\. Such evaluations provide direct perceptual evidence, but they are expensive to collect and difficult to use as a development\-stage metric\.

Recent benchmarks for full\-duplex spoken dialogue systems introduce automatic diagnostics for interactive turn\-taking behavior\. Full\-Duplex\-Bench\[[17](https://arxiv.org/html/2607.01345#bib.bib39)\]evaluates specific capabilities such as pause handling, backchanneling, smooth turn\-taking, and interruption management using task\-specific metrics, including takeover rate, backchannel frequency, timing distribution, and response latency\. Its follow\-up benchmark extends this setup to overlap\-heavy scenarios, measuring how systems respond to user interruptions, backchannels, side conversations, and ambient speech\[[16](https://arxiv.org/html/2607.01345#bib.bib40)\]\. Talking Turns\[[2](https://arxiv.org/html/2607.01345#bib.bib25)\]takes a timing\-centric event\-decision approach: it trains a supervised judge on human\-human conversations to predict discrete turn\-taking event labels, and then applies event\-specific thresholds to evaluate whether a dialogue system speaks up, continues, backchannels, interrupts, or yields at appropriate moments during human\-AI interaction\.

These benchmarks provide useful diagnostics of system capabilities, but their metrics are organized around separate behaviors, event decisions, or interaction scenarios\. Our work instead evaluates overall turn\-taking naturalness in a shared continuous scoring space\. Rather than defining a different thresholded decision rule for each event type, we score heterogeneous timing failures using the same future two\-speaker voice\-activity likelihood formulation\.

## IIIMethod

![Refer to caption](https://arxiv.org/html/2607.01345v1/x1.png)Figure 1:Overview of the TurnNat framework\. TurnNat first extracts VAD\-based turn\-taking boundary units from the two\-channel dialogue, then uses a causal turn\-taking prediction model to assign likelihoods to future two\-speaker voice\-activity states at frames inside these units\. The resulting frame\-level NLLs are aggregated through mean and tail terms to obtain a dialogue\-level turn\-taking naturalness score, which is then calibrated to human preference judgments\.### III\-AProblem Formulation

We study automatic evaluation of turn\-taking naturalness in two\-channel spoken dialogue\. Letx=\(x\(1\),x\(2\)\)x=\(x^\{\(1\)\},x^\{\(2\)\}\)denote a dialogue segment, wherex\(1\)x^\{\(1\)\}andx\(2\)x^\{\(2\)\}correspond to the audio channels of speakers 1 and 2, respectively\.

Letfθf\_\{\\theta\}denote a turn\-taking prediction model with parametersθ\\theta\. For each framett, the model uses the available dialogue contextx≤tx\_\{\\leq t\}to estimate the likelihood of the observed future two\-speaker voice\-activity statectc\_\{t\}\. TurnNat converts this likelihood into frame\-level negative log\-likelihood values and aggregates them into a dialogue\-level turn\-taking naturalness scoremθ​\(x\)∈ℝm\_\{\\theta\}\(x\)\\in\\mathbb\{R\}, where higher values indicate more natural turn\-taking\. TurnNat does not require human judgments, perturbation labels, or manually annotated turn\-taking events at inference time\. Fig\.[1](https://arxiv.org/html/2607.01345#S3.F1)shows the overall TurnNat framework in detail\.

### III\-BTurn\-Taking Boundary Units

TurnNat scores local timing behavior through turn\-taking boundary units \(TBUs\), which are extracted from the cleaned two\-speaker voice\-activity sequence\. We first identify contiguous active regions in each speaker channel and treat them as utterance candidates\. To remove spurious VAD fragments while retaining short feedback events, we keep only regions whose duration is at least 200ms to preserve brief backchannels\.

For each retained utterance candidate, we define two TBUs: one associated with the utterance onset and one associated with the utterance offset\. For a boundary timeτj\\tau\_\{j\}, the corresponding TBUuju\_\{j\}contains frames in the pre\-boundary interval\[τj−L,τj\]\[\\tau\_\{j\}\-L,\\tau\_\{j\}\], whereL=2​sL=2\\mathrm\{s\}\.

For a dialogue segmentxx, we extract all TBUs𝒰​\(x\)=\{uj\}j=1J\\mathcal\{U\}\(x\)=\\\{u\_\{j\}\\\}\_\{j=1\}^\{J\}, whereJ\{J\}is the total amount\. Let𝒯​\(uj\)\\mathcal\{T\}\(u\_\{j\}\)denote the set of frames contained in unituju\_\{j\}\. We define the set of all TBU frames as follows:

𝒯TBU​\(x\)=⋃uj∈𝒰​\(x\)𝒯​\(uj\)\.\\mathcal\{T\}\_\{\\mathrm\{TBU\}\}\(x\)=\\bigcup\_\{u\_\{j\}\\in\\mathcal\{U\}\(x\)\}\\mathcal\{T\}\(u\_\{j\}\)\.\(1\)
Although the scoring frames are selected before the boundary, each frame is later evaluated through a two\-speaker voice\-activity prediction over a future horizon\. Thus, a TBU covers the local turn\-taking region around an onset or offset boundary, including possible responses, gaps, overlaps, holds, or backchannels following the boundary, while preserving a causal scoring setup\.

### III\-CFuture Voice\-Activity Prediction

Future Voice\-Activity Prediction Target\.TurnNat uses a causal future voice\-activity prediction model as its likelihood source\. Following VAP\[[8](https://arxiv.org/html/2607.01345#bib.bib12)\], for each framett, the model predicts both speakers’ voice activity over a future horizon ofH=2​sH=2\\mathrm\{s\}, divided intoK=4K=4non\-uniform bins:\[0,200\]\[0,200\],\[200,600\]\[200,600\],\[600,1200\]\[600,1200\], and\[1200,2000\]\[1200,2000\]ms\\mathrm\{ms\}\. This non\-uniform discretization assigns finer temporal resolution to the near future and coarser resolution to farther future activity, reflecting the increasing uncertainty of longer\-horizon prediction while keeping the joint two\-speaker future\-activity state space tractable\. This is well suited to turn\-taking evaluation, where local timing differences around upcoming speech activity are often perceptually important\. Let

𝐛t=\[bt,1\(1\),…,bt,K\(1\),bt,1\(2\),…,bt,K\(2\)\]∈\{0,1\}2​K\\mathbf\{b\}\_\{t\}=\\left\[b^\{\(1\)\}\_\{t,1\},\\ldots,b^\{\(1\)\}\_\{t,K\},b^\{\(2\)\}\_\{t,1\},\\ldots,b^\{\(2\)\}\_\{t,K\}\\right\]\\in\\\{0,1\\\}^\{2K\}\(2\)denote the future voice\-activity pattern after framett\. For speakerssand future binkk,bt,k\(s\)=1b^\{\(s\)\}\_\{t,k\}=1if more than50%50\\%of the frames in that bin are active according to the VAD, andbt,k\(s\)=0b^\{\(s\)\}\_\{t,k\}=0otherwise\. Each possible binary pattern𝐛t\\mathbf\{b\}\_\{t\}is indexed as one categorical future voice\-activity state:

ct=index​\(𝐛t\),ct∈\{1,…,22​K\}\.c\_\{t\}=\\mathrm\{index\}\(\{\\mathbf\{b\}\_\{t\}\}\),\\quad c\_\{t\}\\in\\\{1,\\ldots,2^\{2K\}\\\}\.\(3\)
WithK=4K=4, this yields28=2562^\{8\}=256possible joint future voice\-activity states\.

Turn\-Taking Prediction Model\.The causal prediction modelfθf\_\{\\theta\}consists of a turn\-taking modeling backbone followed by a 256\-way future voice\-activity head, as illustrated in Fig\.[1](https://arxiv.org/html/2607.01345#S3.F1)\. Given the available dialogue contextx≤tx\_\{\\leq t\}, the model outputs a probability distribution𝐩t\\mathbf\{p\}\_\{t\}over the 256 joint future\-activity states:

𝐩θ​\(t;x\)=Softmax​\(fθ​\(x≤t\)\),𝐩θ​\(t;x\)∈\[0,1\]256\.\\mathbf\{p\}\_\{\\theta\}\(t;x\)=\\mathrm\{Softmax\}\\left\(f\_\{\\theta\}\(x\_\{\\leq t\}\)\\right\),\\quad\\mathbf\{p\}\_\{\\theta\}\(t;x\)\\in\[0,1\]^\{256\}\.\(4\)Here,θ\\thetadenotes the parameters of the turn\-taking backbone and the future voice\-activity head\. The specific backbone instantiations are described in Section[IV\-B](https://arxiv.org/html/2607.01345#S4.SS2)\.

Future\-Activity Likelihood Training Objective\.The turn\-taking prediction models are trained only on natural human\-to\-human dialogues \(Section[IV\-A](https://arxiv.org/html/2607.01345#S4.SS1)\)\. Let𝒟nat\\mathcal\{D\}\_\{\\mathrm\{nat\}\}denote the natural training set\. We minimize the weighted future\-activity negative log\-likelihood over prediction frames:

ℒtrain​\(θ\)=−∑x∈𝒟nat∑t∈𝒯pred​\(x\)wt​\(x\)​log⁡𝐩θ​\(t;x\)​\[ct\]∑x∈𝒟nat∑t∈𝒯pred​\(x\)wt​\(x\),\\mathcal\{L\}\_\{\\mathrm\{train\}\}\(\\theta\)=\-\\frac\{\\sum\_\{x\\in\\mathcal\{D\}\_\{\\mathrm\{nat\}\}\}\\sum\_\{t\\in\\mathcal\{T\}\_\{\\mathrm\{pred\}\}\(x\)\}w\_\{t\}\(x\)\\log\\mathbf\{p\}\_\{\\theta\}\(t;x\)\[c\_\{t\}\]\}\{\\sum\_\{x\\in\\mathcal\{D\}\_\{\\mathrm\{nat\}\}\}\\sum\_\{t\\in\\mathcal\{T\}\_\{\\mathrm\{pred\}\}\(x\)\}w\_\{t\}\(x\)\},\(5\)where𝒯pred​\(x\)\\mathcal\{T\}\_\{\\mathrm\{pred\}\}\(x\)denotes the prediction\-frame set for segmentxx, and

wt​\(x\)=\{α,t∈𝒯TBU​\(x\),1,otherwise\.w\_\{t\}\(x\)=\\begin\{cases\}\\alpha,&t\\in\\mathcal\{T\}\_\{\\mathrm\{TBU\}\}\(x\),\\\\ 1,&\\text\{otherwise\}\.\\end\{cases\}\(6\)
Settingα=1\\alpha=1gives the standard uniformly weighted objective, whileα\>1\\alpha\>1emphasizes frames inside TBUs\.

### III\-DTurnNat Scoring

As depicted in Fig\.[1](https://arxiv.org/html/2607.01345#S3.F1), TurnNat converts the future\-activity likelihood into frame\-level NLL during evaluation time:

ℓθ​\(t;x\)=−log⁡𝐩θ​\(t;x\)​\[ct\]\.\\ell\_\{\\theta\}\(t;x\)=\-\\log\\mathbf\{p\}\_\{\\theta\}\(t;x\)\[c\_\{t\}\]\.\(7\)
A largerℓθ​\(t;x\)\\ell\_\{\\theta\}\(t;x\)indicates that the observed future two\-speaker activity is less typical under natural conversational dynamics\.

For each TBUuju\_\{j\}, TurnNat computes the unit\-level NLL:

sθ​\(uj\)=1\|𝒯​\(uj\)\|​∑t∈𝒯​\(uj\)ℓθ​\(t;x\)\.s\_\{\\theta\}\(u\_\{j\}\)=\\frac\{1\}\{\|\\mathcal\{T\}\(u\_\{j\}\)\|\}\\sum\_\{t\\in\\mathcal\{T\}\(u\_\{j\}\)\}\\ell\_\{\\theta\}\(t;x\)\.\(8\)Given theJJTBU scores\{sθ​\(uj\)\}j=1J\\\{s\_\{\\theta\}\(u\_\{j\}\)\\\}\_\{j=1\}^\{J\}, TurnNat aggregates them using mean and tail terms:

MeanNLLθ​\(x\)=1J​∑j=1Jsθ​\(uj\),TailNLLθ​\(x\)=AvgTopK​\(\{sθ​\(uj\)\}j=1J\),\\begin\{gathered\}\\mathrm\{MeanNLL\}\_\{\\theta\}\(x\)=\\frac\{1\}\{J\}\\sum\_\{j=1\}^\{J\}s\_\{\\theta\}\(u\_\{j\}\),\\\\ \\mathrm\{TailNLL\}\_\{\\theta\}\(x\)=\\mathrm\{AvgTopK\}\(\\\{s\_\{\\theta\}\(u\_\{j\}\)\\\}\_\{j=1\}^\{J\}\),\\end\{gathered\}\(9\)whereAvgTopK​\(⋅\)\\mathrm\{AvgTopK\}\(\\cdot\)averages the top fraction of highest\-NLL TBUs, so that strongly unnatural local turn\-taking events are not diluted by the global mean\. The final dialogue\-level naturalness score is:

mθ​\(x\)=−\[λ​MeanNLLθ​\(x\)\+\(1−λ\)​TailNLLθ​\(x\)\]\.m\_\{\\theta\}\(x\)=\-\[\\lambda\\mathrm\{MeanNLL\}\_\{\\theta\}\(x\)\+\(1\-\\lambda\)\\mathrm\{TailNLL\}\_\{\\theta\}\(x\)\]\.\(10\)The negative sign converts NLL\-oriented atypicality into naturalness, so highermθ​\(x\)m\_\{\\theta\}\(x\)indicates more natural turn\-taking\.

## IVExperiments

### IV\-ADataset

Data Source\.We use English two\-channel dialogue recordings from the naturalistic portion of the Seamless Interaction dataset\[[1](https://arxiv.org/html/2607.01345#bib.bib41)\]\. Following the released train, development, and test partitions, we select controlled subsets and focus on ordinary prompted dyadic conversations based on the Interpersonal Circumplex \(IPC\) framework\. We exclude task\-oriented interaction types, such as collaborative storytelling, grounded gesture, and charades, because their turn\-taking patterns are more strongly constrained by the task format than by open\-ended conversational dynamics\. Table[I](https://arxiv.org/html/2607.01345#S4.T1)summarizes the resulting subsets and their use in the experiments\.

TABLE I:Speaker\-disjoint natural dialogue splits\.Split\# Dyads\# SpeakersHoursMean Dur\.UsageTrain4,2631,140250\.18211\.27sModel trainingDev3454920\.44213\.26sModel selectionTest2,251287129\.32206\.82sPerturbation benchmark

Turn\-taking Perturbation Benchmark\.To evaluate whether TurnNat identifies localized turn\-taking degradations, we construct paired natural–perturbed dialogue clips from the held\-out test subset\. Each pair preserves speaker identity, recording condition, and most surrounding dialogue context, while locally altering one targeted turn\-taking behavior\.

We sample 20–25 s dialogue segments\. Candidate regions are identified using the Silero voice activity detector111Silero VAD:https://github\.com/snakers4/silero\-vad\., with transcript text used only for manual verification\. Each crop is centered around a turn\-taking event targeted by one of our perturbation types, and its start and end points are selected from regions where both speakers are silent to avoid truncating ongoing speech\.

The perturbation magnitudes are chosen to be clearly outside typical human–human response timing while remainingplausible enoughfor listening evaluation\. Prior work reports that many human responses begin within a few hundred milliseconds of the previous turn ending, for example from about \-200 to 400 ms\[[25](https://arxiv.org/html/2607.01345#bib.bib10)\]\. We therefore use larger timing shifts to create controlled degradations rather than borderline timing variation\. For hold and shift perturbations, we select clean candidate regions using the VAP shift/hold criterion\[[8](https://arxiv.org/html/2607.01345#bib.bib12)\]: a mutual\-silence interval is bounded by a single active speaker in a 1s pre\-offset region and a single active speaker in a 1s post\-onset region\. If the active speaker before and after the silence differs, the region is treated as a speaker shift; if the speaker is the same, it is treated as a speaker hold\. Backchannels are selected using the transcript\-based criterion of Wang et al\.\[[30](https://arxiv.org/html/2607.01345#bib.bib24)\]: isolated one\- or two\-word listener responses, such as “yeah” or “mmhmm”, during the other speaker’s turn\.

- •Late response: We delay a responding utterance by 1\.2–2\.0s, creating an abnormally long response gap\.
- •Early entry: We advance a responding utterance by 1\.2–2\.5s, creating premature overlap with the current speaker\.
- •Hold instead of shift: We start from a clean speaker\-shift region and remove the responding turn, so the original speaker appears to retain the floor instead of shifting it\.
- •Shift instead of hold: We start from a clean speaker\-hold region and insert a complete turn from the other speaker, so the dialogue appears to shift speakers where the original speaker would normally continue\.
- •Excessive backchanneling: We insert non\-repetitive two to three additional listener\-feedback events into regions where the listener is originally silent, using naturally occurring backchannels from the same speaker and conversation\.

### IV\-BPredictive Model Comparison

We compare TurnNat scorers instantiated with VAP and DualTurn backbones\. The comparison includes released checkpoints and fully fine\-tuned variants, using either each backbone’s native future\-activity target or the shared 256\-way categorical target used by TurnNat\.

VAP\.VAP uses a CPC speech encoder, a causal Transformer, and a 256\-way classification head that predicts the joint future voice activity of both speakers over a 2s horizon\[[8](https://arxiv.org/html/2607.01345#bib.bib12)\]\. We evaluate the released VAP checkpoint directly and a fully fine\-tuned VAP variant trained with the same 256\-way categorical future\-activity target\.

DualTurn\.DualTurn learns causal two\-speaker representations through dual\-channel generative speech pretraining with a frozen Mimi encoder and a Qwen backbone\[[22](https://arxiv.org/html/2607.01345#bib.bib29)\]\. Its original training includes multiple turn\-taking signals, including VAD, future activity, end\-of\-turn, hold, beginning\-of\-turn, and backchannel prediction\. The Mimi encoder is kept frozen in all experiments\.

For comparison with the 256\-way categorical TurnNat scorer, we evaluate two DualTurn scoring forms\. The first uses DualTurn’s native eight Bernoulli future\-activity targets, defined by 4 horizon bins per speaker at 0\.24, 0\.48, 0\.96, and 2\.00 s\. The second replaces this output with the shared 256\-way categorical head used by TurnNat\.

Training Details\.We train each model for up to five epochs with AdamW and a batch size of 8, and select the checkpoint with the lowest developmentfuture\-activity loss\. Early stopping is used to reduce overfitting\. Every experiment is conducted on a single NVIDIA A100 80GB GPU\.

TABLE II:Human evaluation summary\.StatisticValueStatisticValueAnnotators18Formal pairs150Formal judgments450Annotations / pair3Natural pref\. rate0\.680±\\pm0\.043Mean rating diff\.0\.564±\\pm0\.129Majority agreement0\.780Kripp\.\-ord\.0\.341Artifact rating nat\./pert\.2\.067 / 2\.300Artifact diff\.0\.233±\\pm0\.317
*Note\.*Nat\. pref\. rate is the fraction of pairwise judgments preferring the natural clip, where values above 0\.5 favor natural timing\. Majority agreement is the fraction of judgments matching the pair\-level majority vote, where higher values indicate more consistent pairwise choices\. Mean rating diff\. is natural minus perturbed rating, where positive values favor natural clips\. Kripp\.\-ord\. is Krippendorff’s ordinal alpha over five\-point naturalness ratings, where higher values indicate stronger rating agreement\[[15](https://arxiv.org/html/2607.01345#bib.bib28)\]\. Artifact diff\. is perturbed minus natural artifact rating; lower artifact ratings and smaller differences indicate better artifact control\.

### IV\-CHuman Evaluation

Human judgments are used only to assess the perceptual validity of the perturbation benchmark and are never used for model training\. In each formal trial, annotators \(native English speakers\) choose the clip with more natural turn\-taking and rate both clips on a five\-point naturalness scale\. We also include artifact\-control trials to check whether audible waveform artifacts could confound judgments\.

We summarize pairwise judgments using natural\-preference rates and majority\-vote statistics\. For ratings, we report mean rating differences and ordinal agreement\. Aggregate statistics are shown in Table[II](https://arxiv.org/html/2607.01345#S4.T2)\.

### IV\-DMetric Evaluation

For each natural–perturbed pair\(xinat,xipert\)\(x\_\{i\}^\{\\mathrm\{nat\}\},x\_\{i\}^\{\\mathrm\{pert\}\}\), an automatic metric produces dialogue\-level NLL scores:

zinat=zθ​\(xinat\),zipert=zθ​\(xipert\),z\_\{i\}^\{\\mathrm\{nat\}\}=z\_\{\\theta\}\(x\_\{i\}^\{\\mathrm\{nat\}\}\),\\qquad z\_\{i\}^\{\\mathrm\{pert\}\}=z\_\{\\theta\}\(x\_\{i\}^\{\\mathrm\{pert\}\}\),\(11\)where lower NLL indicates more typical, and therefore more natural, turn\-taking behavior\. We define the paired NLL difference as

Δ​zi=zipert−zinat\.\\Delta z\_\{i\}=z\_\{i\}^\{\\mathrm\{pert\}\}\-z\_\{i\}^\{\\mathrm\{nat\}\}\.\(12\)A positiveΔ​zi\\Delta z\_\{i\}indicates that the perturbed dialogue receives higher surprisal than its natural counterpart\.

Concordance Index\.The concordance index measures global separability between natural and perturbed clips:

C​\-​index=∑i=1N∑j=1N𝕀​\[zipert\>zjnat\]∑i=1N∑j=1N𝕀​\[zipert≠zjnat\]\.\\mathrm\{C\\mbox\{\-\}index\}=\\frac\{\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{N\}\\mathbb\{I\}\\left\[z\_\{i\}^\{\\mathrm\{pert\}\}\>z\_\{j\}^\{\\mathrm\{nat\}\}\\right\]\}\{\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{N\}\\mathbb\{I\}\\left\[z\_\{i\}^\{\\mathrm\{pert\}\}\\neq z\_\{j\}^\{\\mathrm\{nat\}\}\\right\]\}\.\(13\)
Pairwise Accuracy\.Pairwise accuracy measures whether the metric ranks each matched pair in the expected direction:

Accpair=1N​∑i=1N𝕀​\[Δ​zi\>0\]\.\\mathrm\{Acc\}\_\{\\mathrm\{pair\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\left\[\\Delta z\_\{i\}\>0\\right\]\.\(14\)

## VResults

We first verify that the constructed turn\-taking perturbation benchmark produces human\-perceived naturalness differences\. We then evaluate whether TurnNat distinguishes natural from perturbed clips on the same benchmark\.

TABLE III:Automatic discrimination results for dialogue\-level TurnNat scores\.ArchitectureTrainingOverallPair Acc\. by perturbation type↑\\uparrowIDBackboneOutputAdapt\.α\\alphaΔ​mθ\\Delta m\_\{\\theta\}↑\\uparrowC\-index↑\\uparrowPair Acc\. \(%\)↑\\uparrowLate resp\.Early entryHold→\\rightarrowshiftShift→\\rightarrowholdExcess BCV0VAP256\-way cat\.None–0\.60±\\pm0\.060\.63380\.690\.091\.066\.082\.074\.0D0DualTurnindep\. Bern\.None–0\.47±\\pm0\.040\.64577\.566\.085\.083\.073\.080\.5V1VAP256\-way cat\.FT10\.36±\\pm0\.040\.64180\.279\.585\.082\.574\.080\.0D1DualTurnindep\. Bern\.FT10\.47±\\pm0\.040\.66381\.275\.582\.091\.573\.084\.0D2DualTurnindep\. Bern\. \+ aux\.FT10\.40±\\pm0\.040\.65781\.578\.086\.092\.071\.580\.0D3DualTurn256\-way cat\.FT10\.44±\\pm0\.040\.66083\.382\.090\.082\.078\.084\.5D4DualTurn256\-way cat\. \+ aux\.FT10\.45±\\pm0\.040\.67086\.293\.593\.579\.580\.084\.5D2DualTurnindep\. Bern\. \+ aux\.FT30\.47±\\pm0\.040\.66981\.778\.581\.592\.075\.581\.0D4DualTurn256\-way cat\. \+ aux\.FT30\.46±\\pm0\.040\.67687\.394\.092\.082\.083\.585\.0D4DualTurn256\-way cat\. \+ aux\.FT80\.45±\\pm0\.040\.67688\.095\.092\.581\.084\.587\.0
*Note\.*mθ​\(x\)m\_\{\\theta\}\(x\)is the dialogue\-level naturalness score, where higher values indicate more natural turn\-taking\.Δ​mθ=mθ​\(xn​a​t\)−mθ​\(xp​e​r​t\)\\Delta m\_\{\\theta\}=m\_\{\\theta\}\(x^\{nat\}\)\-m\_\{\\theta\}\(x^\{pert\}\)\. C\-index measures global natural–perturbed separability\. Pair Acc\. measures matched\-pair discrimination accuracy\. Per\-type columns report matched\-pair accuracy\.α\\alphais the training weight for TBUs\. FT denotes full fine\-tuning\. “indep\. Bern\.” denotes DualTurn’s native eight Bernoulli future\-activity targets over four horizons per speaker\.

### V\-AHuman Validation of the Perturbation Benchmark

As shown in Table[II](https://arxiv.org/html/2607.01345#S4.T2), human judgments consistently favor natural turn\-taking\. Annotators preferred the natural clip in 68\.0% of pairwise comparisons, and natural clips received higher five\-point naturalness ratings by 0\.564 points on average\. Pairwise choices were also stable, with 78\.0% of annotations matching the pair\-level majority vote\. Agreement on the five\-point ratings was moderate, with Krippendorff’s ordinal alpha of 0\.341, indicating non\-trivial consistency despite the finer\-grained rating scale\. Artifact ratings were similar between natural and perturbed clips, suggesting that the observed preferences are not primarily explained by waveform artifacts\.

### V\-BTurnNat Evaluation on the Perturbation Benchmark

Overall discrimination\.Table[III](https://arxiv.org/html/2607.01345#S5.T3)reports automatic discrimination results on the turn\-taking perturbation benchmark\. We focus primarily on C\-index and matched\-pair accuracy, since these metrics are scale\-free and directly measure whether a scorer assigns higher naturalness to natural clips than to their perturbed counterparts\. The best configuration, D4 withα=8\\alpha=8, achieves 88\.0% matched\-pair accuracy and a C\-index of 0\.676\. This improves over the VAP baselines \(V0: 80\.6%, V1: 80\.2%\) by about 7–8 absolute points in pair accuracy, and over the unadapted DualTurn Bernoulli scorer \(D0: 77\.5%\) by 10\.5 points\. The Wilson 95% confidence interval for D4 withα=8\\alpha=8is 85\.8–89\.9%, compared with 78\.0–82\.9% for V0 and 77\.6–82\.6% for V1\.

Architecture ablation\.The architecture ablation shows that improvements come from the combination of DualTurn representations, categorical future\-activity prediction, and auxiliary turn\-taking supervision\. Full fine\-tuning alone does not substantially improve the VAP\-based scorer: V1 remains close to V0 in both pair accuracy and C\-index\. DualTurn\-based Bernoulli scorers improve some perturbation types, especially hold\-to\-shift, but remain less balanced overall\. Replacing the Bernoulli output with the shared 256\-way categorical future\-activity target improves overall coverage: D3 reaches 83\.3% pair accuracy, and adding auxiliary supervision in D4 further improves accuracy to 86\.2% atα=1\\alpha=1\.

Effect of TBU weighting\.Emphasizing TBUs during training further improves the final scorer\. Increasingα\\alphafrom 1 to 3 and 8 improves D4 from 86\.2% to 87\.3% and 88\.0% pair accuracy, respectively, while C\-index increases from 0\.670 to 0\.676\. The two weighted D4 variants are close, suggesting that TBU weighting is beneficial but not highly sensitive within this range\. Applying the same weighting to the Bernoulli auxiliary scorer yields only a small gain, from 81\.5% to 81\.7%, suggesting that the strongest results require both the categorical future\-activity target and auxiliary DualTurn supervision\.

Perturbation\-type analysis\.Per\-type results show that the final scorer is especially effective for late responses, early entries, shift\-to\-hold errors, and excessive backchanneling\. D4 withα=8\\alpha=8achieves 95\.0% accuracy on late responses, 92\.5% on early entries, 84\.5% on shift\-to\-hold perturbations, and 87\.0% on excessive backchanneling\. The main exception is hold\-to\-shift, where the Bernoulli\-based D2 scorer performs best\. This suggests that different future\-activity parameterizations emphasize different aspects of turn\-taking: Bernoulli horizon targets are particularly sensitive to missing speaker transitions, while the categorical target with auxiliary supervision provides stronger overall discrimination across heterogeneous timing failures\.

## VIConclusion

We introduced TurnNat, a likelihood\-based framework for automatic turn\-taking naturalness evaluation in two\-channel spoken dialogue\. TurnNat uses a causal future\-activity prediction model trained on natural conversations, pools NLLs over turn\-taking boundary units, and aggregates mean and tail scores into a dialogue\-level naturalness score\. We also constructed a human\-validated turn\-taking perturbation benchmark covering five localized timing failures\. Experiments show that TurnNat reliably distinguishes natural from perturbed clips, with the strongest DualTurn\-based scorer improving over VAP and Bernoulli\-output baselines\. These results suggest that future two\-speaker voice\-activity likelihood provides a useful unified signal for evaluating turn\-taking naturalness across heterogeneous timing failures\.

## VIILimitations

This work evaluates controlled perturbations of natural human\-to\-human dialogue\. While this paired design isolates turn\-taking timing differences, it does not cover all failures produced by deployed spoken dialogue systems, such as ASR errors, semantic misunderstandings, prosodic mismatch, or system\-side latency patterns\. TurnNat also focuses on future two\-speaker voice activity, so it may miss cases where naturalness depends on lexical content, discourse intent, speaker relationship, or task context\. Finally, human judgments are used to validate the perturbation benchmark rather than to calibrate TurnNat scores to subjective ratings\. Future work should extend the benchmark to real human–AI conversations, more languages and domains, and larger\-scale human judgments for studying score calibration and model–human agreement\.

## References

- \[1\]V\. Agrawal, A\. Akinyemi, K\. Alvero, M\. Behrooz, J\. Buffalini, F\. M\. Carlucci, J\. Chen, J\. Chen, Z\. Chen, S\. Cheng,et al\.\(2025\)Seamless interaction: dyadic audiovisual motion modeling and large\-scale dataset\.arXiv preprint arXiv:2506\.22554\.Cited by:[§IV\-A](https://arxiv.org/html/2607.01345#S4.SS1.p1.1)\.
- \[2\]S\. Arora, Z\. Lu, C\. Chiu, R\. Pang, and S\. Watanabe\(2025\)Talking turns: benchmarking audio foundation models on turn\-taking dynamics\.arXiv preprint arXiv:2503\.01174\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1),[§II\-B](https://arxiv.org/html/2607.01345#S2.SS2.p2.1)\.
- \[3\]A\. Baughan, X\. Wang, A\. Liu, A\. Mercurio, J\. Chen, and X\. Ma\(2023\)A mixed\-methods approach to understanding user trust after voice assistant failures\.InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems,pp\. 1–16\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[4\]G\. Castillo\-López, G\. de Chalendar, and N\. Semmar\(2025\)A survey of recent advances on turn\-taking modeling in spoken dialogue systems\.InProceedings of the 15th international workshop on spoken dialogue systems technology,pp\. 254–271\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p1.1)\.
- \[5\]K\. E\. Cevasco, R\. E\. Morrison Brown, R\. Woldeselassie, and S\. Kaplan\(2024\)Patient engagement with conversational agents in health applications 2016–2022: a systematic review and meta\-analysis\.Journal of medical systems48\(1\),pp\. 40\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[6\]A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour\(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[7\]E\. Ekstedt and G\. Skantze\(2020\)TurnGPT: a transformer\-based language model for predicting turn\-taking in spoken dialog\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 2981–2990\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1)\.
- \[8\]E\. Ekstedt and G\. Skantze\(2022\)Voice activity projection: self\-supervised learning of turn\-taking events\.InProc\. Interspeech 2022,pp\. 5190–5194\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1),[§III\-C](https://arxiv.org/html/2607.01345#S3.SS3.p1.8),[§IV\-A](https://arxiv.org/html/2607.01345#S4.SS1.p4.1),[§IV\-B](https://arxiv.org/html/2607.01345#S4.SS2.p2.1)\.
- \[9\]Q\. Fang, S\. Guo, Y\. Zhou, Z\. Ma, S\. Zhang, and Y\. Feng\(2025\)Llama\-omni: seamless speech interaction with large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 57607–57624\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[10\]K\. Inoue, M\. Elmers, Y\. Fu, Z\. H\. Pang, D\. Lala, K\. Ochi, and T\. Kawahara\(2025\)Prompt\-guided turn\-taking prediction\.InProceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue,pp\. 146–151\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1)\.
- \[11\]K\. Inoue, B\. Jiang, E\. Ekstedt, T\. Kawahara, and G\. Skantze\(2024\)Multilingual turn\-taking prediction using voice activity projection\.InProceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation \(lrec\-coling 2024\),pp\. 11873–11883\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1)\.
- \[12\]K\. Inoue, B\. Jiang, E\. Ekstedt, T\. Kawahara, and G\. Skantze\(2024\)Real\-time and continuous turn\-taking prediction using voice activity projection\.arXiv preprint arXiv:2401\.04868\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1)\.
- \[13\]S\. Ji, Y\. Chen, M\. Fang, J\. Zuo, J\. Lu, H\. Wang, Z\. Jiang, L\. Zhou, S\. Liu, X\. Cheng,et al\.\(2024\)Wavchat: a survey of spoken dialogue models\.arXiv preprint arXiv:2411\.13577\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[14\]M\. Johansson and G\. Skantze\(2015\)Opportunities and obligations to take turns in collaborative multi\-party human\-robot interaction\.InProceedings of the 16th annual meeting of the special interest group on discourse and dialogue,pp\. 305–314\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1)\.
- \[15\]K\. Krippendorff\(2018\)Content analysis: an introduction to its methodology\.Sage publications\.Cited by:[TABLE II](https://arxiv.org/html/2607.01345#S4.T2.4.2)\.
- \[16\]G\. Lin, S\. S\. Kuan, Q\. Wang, J\. Lian, T\. Li, S\. Watanabe, and H\. Lee\(2026\)Full\-duplex\-bench v1\. 5: evaluating overlap handling for full\-duplex speech models\.InICASSP 2026\-2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 19447–19451\.Cited by:[§II\-B](https://arxiv.org/html/2607.01345#S2.SS2.p2.1)\.
- \[17\]G\. Lin, J\. Lian, T\. Li, Q\. Wang, G\. Anumanchipalli, A\. H\. Liu, and H\. Lee\(2025\)Full\-duplex\-bench: a benchmark to evaluate full\-duplex spoken dialogue models on turn\-taking capabilities\.arXiv preprint arXiv:2503\.04721\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1),[§II\-B](https://arxiv.org/html/2607.01345#S2.SS2.p2.1)\.
- \[18\]Y\. Lin, Y\. Zheng, M\. Zeng, and W\. Shi\(2025\)Predicting turn\-taking and backchannel in human\-machine conversations using linguistic, acoustic, and visual signals\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15310–15322\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1)\.
- \[19\]C\. Liu, M\. Su, Y\. Xiang, Y\. Huang, Y\. Yang, K\. Zhang, and M\. Fan\(2025\)Toward enabling natural conversation with older adults via the design of llm\-powered voice agents that support interruptions and backchannels\.InProceedings of the 2025 CHI conference on human factors in computing systems,pp\. 1–22\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[20\]M\. Maslych, M\. Katebi, C\. Lee, Y\. Hmaiti, A\. Ghasemaghaei, C\. Pumarada, J\. Palmer, E\. Segarra Martinez, M\. Emporio, W\. Snipes,et al\.\(2025\)Mitigating response delays in free\-form conversations with llm\-powered intelligent virtual agents\.InProceedings of the 7th ACM Conference on Conversational User Interfaces,pp\. 1–15\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[21\]R\. Meena, G\. Skantze, and J\. Gustafson\(2014\)Data\-driven models for timing feedback responses in a map task dialogue system\.Computer Speech & Language28\(4\),pp\. 903–922\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1)\.
- \[22\]S\. Rajaa\(2026\)DualTurn: learning turn\-taking from dual\-channel generative speech pretraining\.arXiv preprint arXiv:2603\.08216\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1),[§IV\-B](https://arxiv.org/html/2607.01345#S4.SS2.p3.1)\.
- \[23\]M\. Roddy and N\. Harte\(2020\)Neural generation of dialogue response timings\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 2442–2452\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1),[§II\-B](https://arxiv.org/html/2607.01345#S2.SS2.p1.1)\.
- \[24\]S\. O\. Russell and N\. Harte\(2025\-07\)Visual cues enhance predictive turn\-taking for two\-party human interaction\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 209–221\.External Links:[Link](https://aclanthology.org/2025.findings-acl.12/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.12),ISBN 979\-8\-89176\-256\-5Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1)\.
- \[25\]J\. Sakuma, S\. Fujie, and T\. Kobayashi\(2023\)Response timing estimation for spoken dialog systems based on syntactic completeness prediction\.In2022 IEEE Spoken Language Technology Workshop \(SLT\),pp\. 369–374\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p1.1),[§IV\-A](https://arxiv.org/html/2607.01345#S4.SS1.p4.1)\.
- \[26\]R\. Sanjeewa, R\. Iyer, P\. Apputhurai, N\. Wickramasinghe, and D\. Meyer\(2024\)Empathic conversational agent platform designs and their evaluation in the context of mental health: systematic review\.JMIR Mental Health11,pp\. e58974\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[27\]D\. Schlangen\(2006\)From reaction to prediction: experiments with computational models of turn\-taking\.Proceedings of Interspeech 2006, Panel on Prosody of Dialogue Acts and Turn\-Taking\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1)\.
- \[28\]G\. Skantze\(2017\)Towards a general, continuous model of turn\-taking in spoken dialogue using lstm recurrent neural networks\.InProceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue,pp\. 220–230\.Cited by:[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1)\.
- \[29\]G\. Skantze\(2021\)Turn\-taking in conversational systems and human\-robot interaction: a review\.Computer Speech & Language67,pp\. 101178\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p1.1)\.
- \[30\]J\. Wang, L\. Chen, A\. Khare, A\. Raju, P\. Dheram, D\. He, M\. Wu, A\. Stolcke, and V\. Ravichandran\(2024\)Turn\-taking and backchannel prediction with acoustic and large language model fusion\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 12121–12125\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1),[§II\-A](https://arxiv.org/html/2607.01345#S2.SS1.p2.1),[§IV\-A](https://arxiv.org/html/2607.01345#S4.SS1.p4.1)\.
- \[31\]Q\. Wang, Z\. Meng, W\. Cui, Y\. Zhang, P\. Wu, B\. Wu, I\. King, L\. Chen, and P\. Zhao\(2025\)Ntpp: generative speech language modeling for dual\-channel spoken dialogue via next\-token\-pair prediction\.arXiv preprint arXiv:2506\.00975\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p3.1)\.
- \[32\]X\. Wang, Y\. Li, C\. Fu, Y\. Zhang, Y\. Shen, L\. Xie, K\. Li, X\. Sun, and L\. Ma\(2025\)Freeze\-omni: a smart and low latency speech\-to\-speech dialogue model with frozen llm\.InInternational Conference on Machine Learning,pp\. 63345–63354\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.
- \[33\]Z\. Xie and C\. Wu\(2024\)Mini\-omni: language models can hear, talk while thinking in streaming\.arXiv preprint arXiv:2408\.16725\.Cited by:[§I](https://arxiv.org/html/2607.01345#S1.p2.1)\.