Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning

# Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning
Source: [https://arxiv.org/html/2604.16622](https://arxiv.org/html/2604.16622)
Gabriel Skantze Department for Speech, Music and Hearing KTH Royal Institute of Technology, Sweden [liviaq@kth\.se](https://arxiv.org/html/2604.16622v1/[email protected]),[skantze@kth\.se](https://arxiv.org/html/2604.16622v1/[email protected])

###### Abstract

Backchannels \(e\.g\., ‘yeah’, ‘mhm’, and ‘right’\) are short, non\-interruptive feedback signals whose lexical form and prosody jointly convey pragmatic meaning\. While prior computational research has largely focused on predicting backchannel timing, the relationship between lexico\-prosodic form and meaning remains underexplored\. We propose a two\-stage framework: first, fine\-tuning large language models on dialogue transcripts to derive rich contextual representations; and second, learning a joint embedding space for dialogue contexts and backchannel realizations\. We evaluate alignment with human perception via triadic similarity judgments \(prosodic and cross\-lexical\) and a context–backchannel suitability task\. Our results demonstrate that the learned projections substantially improve context\-backchannel retrieval compared to previous methods\. In addition, they reveal that backchannel form is highly sensitive to extended conversational context and that the learned embeddings align more closely with human judgments than raw WavLM features\.

Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine\-Tuning

Livia Qian and Gabriel SkantzeDepartment for Speech, Music and HearingKTH Royal Institute of Technology, Sweden[liviaq@kth\.se](https://arxiv.org/html/2604.16622v1/[email protected]),[skantze@kth\.se](https://arxiv.org/html/2604.16622v1/[email protected])

## 1Introduction

Conversational feedback refers to short, non\-interrupting responses signaling, e\.g\., attention, understanding, and surpriseAllwoodet al\.\([1992](https://arxiv.org/html/2604.16622#bib.bib1)\)\. These responses streamline communication by establishingcommon groundClark and Schaefer \([1989](https://arxiv.org/html/2604.16622#bib.bib17)\); Fusaroliet al\.\([2017](https://arxiv.org/html/2604.16622#bib.bib34)\)and replacing explicit metalinguistic exchanges — for instance, substituting full answers to the question “Did you understand?”, such as “Yes, continue” or “No, please repeat”, with simple tokens like “yeah\!” or “sorry?”, respectively\. Feedback is typically multimodal, involving vocalizations, gaze, and gesturesBertrandet al\.\([2007](https://arxiv.org/html/2604.16622#bib.bib8)\); Truonget al\.\([2011](https://arxiv.org/html/2604.16622#bib.bib96)\); Ferré and Renaudier \([2017](https://arxiv.org/html/2604.16622#bib.bib29)\)\. Modeling these signals is crucial for building rapport in conversational systemsAxelssonet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib4)\)\. Vocal instances, such as ‘yeah’, ‘uh\-huh’ or ‘wow\!’, are commonly termedbackchannelsYngve \([1970](https://arxiv.org/html/2604.16622#bib.bib106)\)\.

![Refer to caption](https://arxiv.org/html/2604.16622v1/x1.png)Figure 1:Architecture of the joint context\-backchannel model\. In this version of the model, both the context transcript and the context audio are passed through their respective encoders \(fine\-tuned LLM and pre\-trained WavLM\) before being concatenated and projected to a lower\-dimensional space to form a context embedding\.Computational work has prioritized backchannel placement and timingHeldneret al\.\([2013](https://arxiv.org/html/2604.16622#bib.bib41)\); Ruedeet al\.\([2019](https://arxiv.org/html/2604.16622#bib.bib87)\); Ortegaet al\.\([2020](https://arxiv.org/html/2604.16622#bib.bib74)\); Ishiiet al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib47)\)but the link betweenformandmeaningis often overlooked\. Prior workBeňušet al\.\([2007](https://arxiv.org/html/2604.16622#bib.bib7)\)shows that lexical choice and prosody both shape the pragmatic interpretation of backchannels — e\.g\., when distinguishing ‘yeah\!’ from ‘yeah?’\. Consequently, inadequate use of backchannel forms risks resulting in pragmatically inappropriate responses\.

To use and interpret backchannels effectively, robust representations are required\. One promising approach embeds backchannel signals in a continuous space where distance reflects similarity\. Our earlier workQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\)showed that this is feasible using a contrastive learning framework that projects past context and feedback into a shared space\. However, that work relied on simplistic mean\-pooled text embeddings \(e\.g\., BERT byDevlinet al\.,[2019](https://arxiv.org/html/2604.16622#bib.bib24)\) to encode dialogue context, limited to the last 4 seconds of the previous turn\.

We extend this method by fine\-tuning an autoregressive large language model \(LLM\) on dialogue transcripts \(Section[4\.1](https://arxiv.org/html/2604.16622#S4.SS1)\); the hidden state preceding a backchannel serves as a dense semantic representation of the context\. This is fused with a WavLMChenet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib14)\)encoding of the last second of the corresponding speech \(Section[4\.2](https://arxiv.org/html/2604.16622#S4.SS2)\)\. The resulting architecture is shown in Figure[1](https://arxiv.org/html/2604.16622#S1.F1)\.

Our contributions are fourfold\. First, we show that fine\-tuning an autoregressive LLM on spoken dialogue data substantially improves context encoding, which is needed for effective backchannel representations\. Second, we highlight the importance of context length for these representations, indicating that the choice of backchannel form is a pragmatically complex phenomenon\. Third, we bridge model and human semantics via downstream evaluation tasks grounded in perception data, showing that the learned representations align with human perception\. Finally, we demonstrate that interpretable affective dimensions — Energy, Surprisal, and Polarity — can be recovered from the learned representations through linear projections\.

## 2Related work

Research on conversational feedback and backchannels has historically focused on placement and timing, utilizing concepts such as “feedback relevance spaces”Heldneret al\.\([2013](https://arxiv.org/html/2604.16622#bib.bib41)\); Howes and Eshghi \([2021](https://arxiv.org/html/2604.16622#bib.bib44)\)and identifying backchannel\-inviting cuesGravano and Hirschberg \([2011](https://arxiv.org/html/2604.16622#bib.bib38)\)\. While early approaches relied on rule\-based acoustic feature extractionKoisoet al\.\([1998](https://arxiv.org/html/2604.16622#bib.bib52)\); Bertrandet al\.\([2007](https://arxiv.org/html/2604.16622#bib.bib8)\); Heldneret al\.\([2010](https://arxiv.org/html/2604.16622#bib.bib40)\); Poppeet al\.\([2010](https://arxiv.org/html/2604.16622#bib.bib77)\), more recent computational work has shifted toward neural methods, diversifying the potential tasks that can be solved, i\.e\., those based on backchannel form and meaning\. Neural methods include prediction and classification tasksRuedeet al\.\([2017](https://arxiv.org/html/2604.16622#bib.bib86)\); Wanget al\.\([2024](https://arxiv.org/html/2604.16622#bib.bib98)\); Parket al\.\([2024](https://arxiv.org/html/2604.16622#bib.bib75)\); Inoueet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib46)\); Fukunagaet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib33)\), contrastive learning frameworksQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\), textless generationMai and Carson\-Berndsen \([2025](https://arxiv.org/html/2604.16622#bib.bib66)\), and multi\-task learning approachesJanget al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib48)\); Liermannet al\.\([2023](https://arxiv.org/html/2604.16622#bib.bib61)\)\.

Recent efforts have also targeted the collection, annotation, and analysis of backchannel data\.Boudinet al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib11)\)labeled feedback attributes \(expectedness, valence, and specificity\);Figueroaet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib30)\)provided annotations of feedback functions for SwitchboardGodfreyet al\.\([1992](https://arxiv.org/html/2604.16622#bib.bib36)\);Mülleret al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib70)\)focused on backchannels in multimodal groups; andLinet al\.\([2025b](https://arxiv.org/html/2604.16622#bib.bib64)\)created a tri\-modal dataset with backchannel and turn shift labels\. Studies have also examined relative feedback perceptionQianet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib80)\)and virtual agent backchanneling behaviorPoppeet al\.\([2013](https://arxiv.org/html/2604.16622#bib.bib78)\)\.

Transformer\-based large language models \(LLMs\) have been shown to capture long\-range context and pragmatic cues essential for turn\-takingEkstedt and Skantze \([2020](https://arxiv.org/html/2604.16622#bib.bib28)\)and response generationZhanget al\.\([2020](https://arxiv.org/html/2604.16622#bib.bib107)\)\. While modern LLMs power sophisticated chatbots \(OpenAI’s[ChatGPT](https://chatgpt.com/), Google’s[Gemini](https://gemini.google.com/), etc\.\), these systems typically operate sequentially with clear turn demarcations, often lacking the naturalness required for real\-time conversational feedback\. Furthermore, these models are text\-based and therefore cannot capture the prosodic aspects of contextual cues and backchannel forms\.

In parallel, neural speech models have advanced the representation of prosody, which is critical for pragmaticsWardet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib101)\)\. Models like wav2vec 2\.0Baevskiet al\.\([2020](https://arxiv.org/html/2604.16622#bib.bib5)\), HuBERTHsuet al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib45)\), and WavLMChenet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib14)\)are trained in a self\-supervised manner to learn continuous representations of speech\. Recently, generative spoken language models \(SLMs\) have emerged as systems capable of processing and generating audio directlyNguyenet al\.\([2023](https://arxiv.org/html/2604.16622#bib.bib71)\); Défossezet al\.\([2024](https://arxiv.org/html/2604.16622#bib.bib21)\), sometimes in a “full\-duplex” fashion\. Some fused SLMs combine text and speech embeddingsAroraet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib2)\)\. While promising, current SLMs often struggle to generate frequent, semantically controlled backchannels, and it is unclear whether they understand the nuances of backchannels coming from the userLinet al\.\([2025a](https://arxiv.org/html/2604.16622#bib.bib63)\)\.

Our work sits at the intersection of these fields\. As the potential space of backchannel forms \(considering both lexical choice and prosody\) is very large, it is hard to discretize them in a meaningful way for pure next\-token prediction\. While previous research has shown how contrastive learning can be used to project context and backchannels into a shared space, it relied on simplistic context representationsQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\)\. Our approach leverages the semantic depth of LLMs to capture the long\-range pragmatic constraints of the preceding context while using audio embeddings to capture the prosodic nuances of both the end of the context and the backchannel response, directly addressing the gap between form and meaning in backchannels\.

Learned context and backchannel embeddings enable several practical applications\. First, cosine similarity between contexts and backchannels can be used to rank candidate responses\. Second, aligning the embedding space with interpretable dimensions \(e\.g\., Energy, Surprisal, and Polarity\) potentially enables semantic control for backchannel synthesis\. Finally, projecting user backchannels into the same space allows for inferring their pragmatic meaning in a dialogue system setting\.

## 3Datasets

We use Fisher Part 1111[https://catalog\.ldc\.upenn\.edu/LDC2004T19](https://catalog.ldc.upenn.edu/LDC2004T19),[https://catalog\.ldc\.upenn\.edu/LDC2004S13](https://catalog.ldc.upenn.edu/LDC2004S13)Cieriet al\.\([2004](https://arxiv.org/html/2604.16622#bib.bib16)\)— a prevalent dataset consisting of 5,850, 3\-to\-10\-minute\-long telephone calls between native U\.S\. English speakers — for training, evaluation, and the perception study, utilizing its time\-aligned transcripts for backchannel extraction\. The other dataset forming the source of backchannels is FiCaFigueroaet al\.\([2024](https://arxiv.org/html/2604.16622#bib.bib32)\); Carol Figueroa \([2024](https://arxiv.org/html/2604.16622#bib.bib112)\), which is a set of reenacted and spontaneously produced backchannels uttered by a single native English speaker, thereby eliminating speaker\-dependent effects\. FiCa was used for evaluation and comparison, in the form presented inQianet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib80)\)\.

The set of backchannels included was chosen based on the frequency of their lexical form in the Fisher corpus: ‘absolutely’, ‘ah’, ‘cool’, ‘definitely’, ‘exactly’, ‘good’, ‘mhm’, ‘mm’, ‘oh’, ‘okay’, ‘really’, ‘right’, ‘sure’, ‘uh\-huh’, ‘wow’, ‘yeah’, ‘yep’, ‘yes’\. UnlikeQianet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib80)\), we do not include responses in the form of ‘no’ and direct displays of non\-understanding \(‘pardon’, ‘sorry’, ‘what’\) because these, under certain circumstances, are not regarded as backchannels but rather as full turns, answers, and clarification requests\.

## 4Models

Model training consisted of two stages\. First, LLMs were fine\-tuned on transcripts of Fisher conversations to learn textual context representations \(Section[4\.1](https://arxiv.org/html/2604.16622#S4.SS1)\)\. Second, a joint context–backchannel architecture was fine\-tuned, leveraging text and speech forcontextmodeling and speech alone forbackchannelgeneration \(Section[4\.2](https://arxiv.org/html/2604.16622#S4.SS2)\)\.

The first stage corresponds to fine\-tuning theLLM encoder, as illustrated in Figure[1](https://arxiv.org/html/2604.16622#S1.F1), while the second stage involves fine\-tuning theprojection layers\. Although the LLM fine\-tuning stage does not capture prosodic variations of backchannels, we assume it learns how different conversational contexts shape expectations over the probability distribution of lexical backchannel forms \(which carry distinct semantics\)\. This enables the joint training stage to associate these lexical expectations with a richer representation of both prosodic and lexical realizations\.

### 4\.1Fine\-tuning LLMs for contextual semantic features

In this step, we compared the capability of open\-weight, state\-of\-the\-art LLMs to model context semantics and to see how the length of prior context \(expressed innumber of turns\) affects this\. We compared Gemma 3Gemma Teamet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib35)\); Google Deepmind \([2025](https://arxiv.org/html/2604.16622#bib.bib110)\), LLaMA 3\.1Grattafioriet al\.\([2024](https://arxiv.org/html/2604.16622#bib.bib37)\); Meta AI \([2024](https://arxiv.org/html/2604.16622#bib.bib108)\), Qwen2\.5Yanget al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib104)\); Qwen \([2024](https://arxiv.org/html/2604.16622#bib.bib109)\)and MistralJianget al\.\([2023](https://arxiv.org/html/2604.16622#bib.bib49)\); Mistral AI \([2024](https://arxiv.org/html/2604.16622#bib.bib111)\)\. All models were fine\-tuned for causal language modeling with a fixed set of hyperparameters \(batch size = 2, max token length = 1024\), using QLoRA \(attention dimension = 32, alpha = 64, dropout = 0\.05\)\.

We split the 83,047 Fisher transcript snippets into training and test sets of equal size\. First, we trained the models on full transcripts in the training set, with each transcript consisting of up to 50 turns, including both backchannel and non\-backchannel turns\. To evaluate the models, we compared their average perplexity over all backchannels found in the test set by feeding in different numbers of preceding turns \(1, 3, and 5\)\. For comparison, this was also applied to the corresponding pre\-trained models \(with 5 preceding turns\)\. The formatting of the transcripts and the calculations are described in Appendix[A](https://arxiv.org/html/2604.16622#A1)\. The results of our experiments are shown in Table[1](https://arxiv.org/html/2604.16622#S4.T1)\.

Average perplexity was computed based on the first token of each backchannel; the relative results were comparable when using a weighted average over all tokens, including those in multi\-token words\. Perplexity consistently decreased as context length increased, highlighting the importance of rich context for determining backchannel form\. The three larger fine\-tuned models exhibited similar performance, whereas the fine\-tuned Gemma 3 4B yielded slightly higher perplexity, likely due to its smaller size\. The pre\-trained models showed considerably higher perplexity, demonstrating the necessity of fine\-tuning; this was anticipated given the specific formatting of the data\.

Model1355 \(pre\)Gemma 3 4B11\.459\.889\.3054\.35LLaMA 3\.1 8B10\.198\.688\.1932\.83Qwen2\.5 7B10\.378\.878\.4055\.33Mistral 7B10\.879\.308\.7630\.91Table 1:Average perplexity on the backchannels in the test set using 1, 3, and 5 preceding turns as context, across fine\-tuned LLMs\. For comparison, the perplexity scores by the corresponding pre\-trained models are also included \(for 5 turns\)\. Perplexity is calculated on the first token of each backchannel word\. For more details, see Appendix[A](https://arxiv.org/html/2604.16622#A1)\.
### 4\.2Joint contrastive learning of context and backchannel embeddings

For contrastive learning, we created a joint architecture that projects context and backchannel vector representations into a shared space, formingjoint embeddings, as illustrated in Figure[1](https://arxiv.org/html/2604.16622#S1.F1)\. Forcontext, we take the final layer’s hidden representation that is used by the LLM head to predict the next token \(the first token of the backchannel\)\. This is concatenated with the last second of audio from the interlocutor’s channel \(ending at the onset of the backchannel\), encoded with WavLMChenet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib14)\)and mean\-pooled over its final layer\. For ablation, we also compare this with using only the LLM or WavLM embeddings\. These embeddings are then projected with an MLP to a context embedding \(Zc​t​xZ\_\{ctx\}\)\. For thebackchannelembedding \(Zb​cZ\_\{bc\}\), we simply used the mean\-pooled WavLM encoding of the audio with a linear projection\.

#### Loss

We use a symmetric InfoNCE\-style contrastive loss, similar to the objective introduced inOordet al\.\([2018](https://arxiv.org/html/2604.16622#bib.bib73)\)and later adopted in a symmetric form byRadfordet al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib81)\)\. Given a set ofNNpairs \(a batch\), the loss maximizes the cosine similarity between the representations of the matching \(positive\) pairs and minimizes the similarity for the non\-matching \(negative\) pairs:

ℒ\\displaystyle\\mathcal\{L\}=12​\(ℒc​o​n​t​e​x​t\+ℒb​a​c​k​c​h​a​n​n​e​l\)\\displaystyle=\\frac\{1\}\{2\}\\Big\(\\mathcal\{L\}\_\{context\}\+\\mathcal\{L\}\_\{backchannel\}\\Big\)ℒc​o​n​t​e​x​t\\displaystyle\\mathcal\{L\}\_\{context\}=1N​∑i=1NC​E​\(Si,:,i\)\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}CE\(S\_\{i,:\},i\)ℒb​a​c​k​c​h​a​n​n​e​l\\displaystyle\\mathcal\{L\}\_\{backchannel\}=1N​∑j=1NC​E​\(S:,j,j\)\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}CE\(S\_\{:,j\},j\)
C​ECEis the cross\-entropy loss, andSSis a temperature\-scaledN×NN\\times Ncosine similarity matrix containing similarity scores between the joint embeddings of all contexts \(rows\) and all backchannels \(columns\)\. Considering that the goal is to maximize the diagonal, i\.e\., the similarity score of the ground truth pairs, this matrix can also be viewed as logits from the joint model; in this case, the task can be framed as a multiclass classification problem where the number of classes is equal to the number of true pairs \(NN\)\. For each context, the model maximizes the score of the true backchannel amongNNcandidates; the same applies to the backchannels trying to optimize the score of their own contexts\. As can be seen, this method is largely affected by the batch sizeNN\.

#### Dataset

Using the backchannel forms listed in Section[3](https://arxiv.org/html/2604.16622#S3), 105,209 samples, i\.e\., context\-backchannel pairs, were found in the Fisher dataset\. Ground truth context\-backchannel pairs form positive samples, while all other combinations serve as negative samples\. Batches \(sets of pairs\) are shuffled and include multiple speakers for robustness and generalizability\. We used an 80\-10\-10% train\-validation\-test split with mutually exclusive speakers and dialogues and ensured that the validation and test data were not used for LLM fine\-tuning\.

#### Hyperparameters

With a predefined temperature of 0\.07 \(as used inRadfordet al\.,[2021](https://arxiv.org/html/2604.16622#bib.bib81)\), we tuned the following hyperparameters within each category oftext embedding\(fine\-tuned Gemma, LLaMA, Qwen, and Mistral, as well as the baseline GTELiet al\.\([2023](https://arxiv.org/html/2604.16622#bib.bib59)\)\):the number of layersin the context encoder’s MLP \(1, 2, 3, 4\),embedding size\(64, 128, 256\), andbatch size\(1024, 2048, 4096, 8192\)\. GTE \(General Text Embedding\) operates with mean\-pooled deep contextualized embeddings; it was used as a baseline due to its performance inQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\)\. We also found that projecting the feedback with a linear layer is sufficient, and larger MLPs decrease performance due to the simplicity of the feedback embeddings; therefore, no hyperparameter search was needed on the feedback side\. We checked the best validation results within2020epochs, based ontop\-10%accuracy \(seeMetricbelow\)\. The best hyperparameter configurations can be found in Appendix[B](https://arxiv.org/html/2604.16622#A2)\.

#### Metric

Optimizing cosine similarity scores can be viewed as a ranking task; for each context, the joint model provides a list of predictions in order of similarity\. As mentioned before, this is equivalent toNN\-class classification, where the classes are the backchannels in the given batch, making accuracy a fitting metric\. However, since backchannels are often very similar and batches are large, we considertop\-kaccuracy\. Since this metric is relative to batch size, we usetop\-k%accuracy, similarly toQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\), which yields a random baseline ofk%k\\%\. In this paper, we reportk=10k=10\.

#### Best hyperparameters

Models favored mid\-to\-large batch sizes \(2048, 4096\) and generally smaller embedding sizes \(64, 128\), with no consistent pattern in number of layers\. This suggests that dense, sometimes shallow embeddings suffice for modeling the context\-backchannel relationships in our dataset\. The preference for larger batch sizes was expected, as they improve discrimination between positive and negative pairs \(Radfordet al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib81)\), for example, used a batch size of 32,768\), but the question of why the largest chosen batch size \(8192\) is relatively underrepresented remains\.

#### Results

The top\-10% test accuracy results are shown in Table[2](https://arxiv.org/html/2604.16622#S4.T2)\. Autoregressive LLMs clearly outperform the SSL models inQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\), where HuBERT combined with GTE achieved 36\.45%\. Our GTE baseline performs slightly worse, likely due to a somewhat different selection of backchannels\. Consistent with prior findings, combined modalities yield the best representations \(with some fluctuations\)\. There were no significant differences among the LLMs, suggesting that small LLMs are sufficient for this problem\.

Context modalityTextembeddingsAudio \+ textTextAudioGemma 3 4B45\.643\.329\.4LLaMA 3\.1 8B49\.844\.2Qwen2\.5 7B46\.339\.7Mistral 7B44\.746\.4GTE baseline33\.421\.2\-random10\.010\.010\.0Table 2:Top\-10% test accuracy on the test set, in relation to context embedding modality and text embedding type \(irrelevant for the last column\)\. Here, the context consists of 5 turns\.For ablation, we also examined the impact of context length \(1, 3, and 5 turns\) and LLM fine\-tuning, as shown in Table[3](https://arxiv.org/html/2604.16622#S4.T3)\. These results reflect the perplexities of the LLMs reported in Table[1](https://arxiv.org/html/2604.16622#S4.T1), reinforcing the finding that LLM fine\-tuning is an important first step, and that longer contexts are important to consider in order to predict backchannel forms\. The best\-performing architecture is henceforth referred to asthe joint model\.

Model1355 \(pre\)Gemma 3 4B40\.140\.345\.640\.9LLaMA 3\.1 8B39\.946\.549\.841\.4Qwen2\.5 7B39\.742\.446\.340\.9Mistral 7B37\.343\.644\.741\.3Table 3:Top\-10% test accuracy on the test set, for the full \(audio \+ text\) model, depending on how many preceding turns were given to the LLM\. Pre\-trained baselines with a context length of 5 turns are also provided\.

## 5Downstream tasks

While the results from the contrastive learning demonstrate internal alignment between context and backchannel embeddings, they do not guarantee that the learned embeddings correspond to human perception of semantic similarity, neither within or across the embedding spaces\. To assess external validity, we collected human perception data for downstream evaluation\.

### 5\.1Human perception data

First, we used the dataset presented byQianet al\.\([2025](https://arxiv.org/html/2604.16622#bib.bib80)\), where participants had compared triplets of feedback responses with identical lexical forms and without context \(from the multi\-speakerFisherand the single\-speakerFiCadatasets\), selecting the two most similar in terms of prosody and semantics\. Using this dataset, we assessed representational quality by expecting the selected pair to be the closest in the embedding space, measured using cosine similarity\. We call this theprosodic backchannel similarity task, as the lexical forms are identical\.

We collected another perception dataset usingFishersamples that were not included in the training set for the LLM fine\-tuning or the joint model\. Participants evaluated a continuous, uninterrupted single\-turn context \(auditory and textual, 4\-80 s\) against three audio\-only backchannels:one ground truthandtwo distractorsfrom different speakers in different dialogues\. Unlike the previously mentioned experiment, the candidate responses differed in lexical form\. The participants were given three different tasks for each combination of context and three responses, henceforth referred to asstimulus set\(for details, see Appendix[C](https://arxiv.org/html/2604.16622#A3),[D](https://arxiv.org/html/2604.16622#A4)and[E](https://arxiv.org/html/2604.16622#A5)\):

#### Cross\-lexical backchannel similarity task

This is a triadic comparison task in which participants select the most similar pair of backchannels\. It differs from theprosodic backchannel similarity taskin that different lexical forms are presented\. Here, too, perceived similarity is expected to align with proximity in the embedding space\. Results of this task and theprosodic backchannel similarity taskare reported in Section[5\.2](https://arxiv.org/html/2604.16622#S5.SS2)\.

#### Context\-backchannel matching task

Participants rate each backchannel’s suitability and naturalness in the given context \(on a scale of 1\-5\)\. We leverage the ground truth to compare human accuracy against the model, evaluating the alignment of context and backchannel embeddings\. Results are reported in Section[5\.3](https://arxiv.org/html/2604.16622#S5.SS3)\.

#### Affective rating task

Participants rate backchannels \(1–5\) onEnergy\(how energetic the response sounds\),Polarity\(how positive the response sounds\), andSurprisal\(how surprised the backchannel speaker sounds\)\. These names were chosen to be fairly easy for annotators to understand\. Polarity and Energy correspond to the notions ofvalenceandarousal, commonly used in psychology, psycholinguistics, and affective computingKuppenset al\.\([2013](https://arxiv.org/html/2604.16622#bib.bib54)\); Warrineret al\.\([2013](https://arxiv.org/html/2604.16622#bib.bib102)\); Mollahosseiniet al\.\([2019](https://arxiv.org/html/2604.16622#bib.bib68)\)\. Surprisal \(the level of surprise\) was chosen due to its importance as a backchannel functionFigueroaet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib30)\)and because it is not clearly captured by the other two dimensions — although they are slightly correlated, as we show later\. We evaluate how well linear probes on projected backchannel embeddings predict these dimensions\. Results are in Section[5\.4](https://arxiv.org/html/2604.16622#S5.SS4)\.

In total, we collected 2100 data points \(consisting of replies to all three tasks\), with 6300 individual ratings of 1260 unique backchannels in 420 unique contexts\.

### 5\.2Backchannel similarity tasks

For the similarity tasks, we selected samples \(stimulus sets\) with≥80%\\geq 80\\%rater agreement on which backchannels are the most similar\. We applied the best joint model’s backchannel encoder to obtain embeddings \(Zb​cZ\_\{bc\}\), identified the most similar pair via cosine similarity, and calculated the agreement rate with human consensus\. As a baseline, we use the mean\-pooled WavLM embeddings fed into the backchannel projection layer, allowing us to isolate and quantify how much the projection improves the representational space\.

The results, shown in Table[4](https://arxiv.org/html/2604.16622#S5.T4), indicate that the projection indeed makes the space more discriminative for backchannel similarity\. In particular, embeddings produced by the joint model’s projection layer yield substantially higher agreement with human judgments than the mean\-pooled WavLM baseline, demonstrating that the projection reshapes the acoustic representations in a way that better aligns with perceptual similarity\. This suggests that the improvements are not merely due to the underlying pre\-trained speech features, but rather to the task\-specific structures learned during joint training\. Overall, these results support the effectiveness of the backchannel projection in capturing fine\-grained similarities that are salient to human listeners\.

ProsodicCross\-lexicalFisherFiCaFisherJoint model69\.775\.966\.3WavLM61\.770\.856\.6Random33\.333\.333\.3Table 4:Proportions of correct selections in the triadic backchannel similarity tasks \(%\)\.
### 5\.3Context\-backchannel matching task

For the context–backchannel matching task, we computed the cosine similarity between the context and backchannel embeddings produced by our context and backchannel encoders \(Zc​t​xZ\_\{ctx\}andZb​cZ\_\{bc\}\), selecting the backchannel with the highest similarity score for a given context\. We measured the proportion of cases where the model correctly identified the ground truth backchannel, using 1 or 5 context turns\. To compare with human performance, we averaged rater scores for each backchannel option, selecting the backchannel with the highest mean assigned score\. We then calculated how often this choice matched the ground truth\.

Results \(Table[5](https://arxiv.org/html/2604.16622#S5.T5)\) show that the model, surprisingly, substantially outperforms humans, even when it only has access to 1 context turn \(just like humans had in the experiment\)\. An alternative interpretation is that the relatively low human “performance” indicates that the same conversational context can allow for multiple valid backchannel responses\. Thus, treating the actual backchannel as ground truth should be done with caution\.

Rater typeScoreJoint model \(5 turns\)72\.3Joint model \(1 turn\)62\.0Human \(1 turn\)47\.4Random33\.3Table 5:Proportions of correct selections in the context\-backchannel matching task \(%\)\.
### 5\.4Affective rating task

Figure[2](https://arxiv.org/html/2604.16622#S5.F2)displays the distributions of median ratings for each backchannel: Energy is centered but Surprisal skews lower, which is consistent with how many continuers \(e\.g\., ‘mhm’, ‘yeah’, and ‘right’\) do not express surprise\. Polarity is predominantly neutral or positive, reflecting the fact that backchannels are less frequently used to express negative sentimentsJurafskyet al\.\([1998](https://arxiv.org/html/2604.16622#bib.bib50)\)\. The correlation between these dimensions is also shown in Figure[2](https://arxiv.org/html/2604.16622#S5.F2); this indicates that Surprisal and Polarity correlate the least, while Energy captures some aspects of both\. For a more detailed analysis grouped by lexical category, see Appendix[F](https://arxiv.org/html/2604.16622#A6)\.

To measure how well the backchannel embeddings \(Zb​cZ\_\{bc\}\) capture these dimensions, we split the samples \(50/50 train/test\) at random and fitted a linear ridge regression probe \(α=1\.0\\alpha=1\.0\)\. We compared this against four baselines: raw mean\-pooled WavLM embeddings, one\-hot encoded lexical tokens, basic prosodic features \(pitch rangein semitones,durationin number of voiced frames\), and the combination of lexical categories and prosodic features\. Pitch was extracted using[Reaper](https://github.com/google/REAPER)\.

The results \(Table[6](https://arxiv.org/html/2604.16622#S5.T6)\) show that our learned embeddings capture these perceived dimensions better than WavLM, while simple prosodic features perform the worst\. This indicates that the distributed representations integrate lexical and prosodic information more effectively than low\-level acoustics or lexical identity alone\. Consistent with our earlier findings in this paper, the backchannel projection layer further reshapes the embedding space in a way that improves alignment with human judgments\.

![Refer to caption](https://arxiv.org/html/2604.16622v1/fig_affective.png)R2R^\{2\}Energy \- Polarity0\.45Energy \- Surprisal0\.39Surprisal \- Polarity0\.09Figure 2:Distribution of median affective ratings over backchannels and correlation between mean ratings\.EnergyPolaritySurprisalJoint model0\.4650\.3410\.552WavLM0\.4060\.2870\.502Lexical0\.1930\.2040\.432Prosody0\.1180\.0280\.156Lex\. \+ Pros\.0\.2400\.2370\.473Table 6:Linear probe fit\(R2R^\{2\}\)on the test set in the affective rating task\.

## 6Analysis of backchannel embeddings

The downstream tasks demonstrate the utility and validity of the learned context and backchannel embedding spaces\. To investigate this further, we developed a tool to visualize backchannel embedding projections in different ways and listen to their prosodic realizations for both the Fisher and FiCa datasets222[https://qianlivia\.github\.io/Aligning\-Backchannel\-and\-Dialogue\-Context\-Representations/](https://qianlivia.github.io/Aligning-Backchannel-and-Dialogue-Context-Representations/)\.

Figure[3](https://arxiv.org/html/2604.16622#S6.F3)shows a scatterplot of the representation space created by the tool\. The backchannel embeddings are projected onto theSurprisalandPolarityaffective dimensions \(the least correlated dimensions\) using the learned probes from Section[5\.4](https://arxiv.org/html/2604.16622#S5.SS4)\. For clarity, we only show ‘yeah’, ‘mm’, ‘exactly’, and ‘really’\. The tool can also provide prosodic analysis over a region of backchannels, which is shown as rectangles in the plot\. The reported metrics are average duration and average pitch range, as defined in Section[5\.4](https://arxiv.org/html/2604.16622#S5.SS4)\.

The four lexical tokens exhibit distinct semantic tendencies: ‘really’ conveys high Polarity and Surprisal, whereas ‘exactly’ signals strong Polarity but low Surprisal\. Both ‘mm’ and ‘yeah’ show weaker Polarity \(with ‘yeah’ slightly stronger\), while ‘mm’ generally expresses higher Surprisal than ‘yeah’\. Despite these lexical tendencies, there is substantial variation driven by prosodic realization, leading to considerable overlap between lexical clusters\. Although higher Surprisal and Polarity are broadly associated with longer durations and wider pitch ranges, Table[6](https://arxiv.org/html/2604.16622#S5.T6)shows that simple prosodic features alone are insufficient predictors, suggesting that the relevant prosodic cues are more nuanced\. A more detailed analysis of the central tendencies and dispersion of lexical forms with respect to the affective dimensions is provided in Appendix[F](https://arxiv.org/html/2604.16622#A6)\.

![Refer to caption](https://arxiv.org/html/2604.16622v1/x2.png)Figure 3:Backchannel samples from the Fisher corpus, embedded through the model, and projected to the Surprisal and Polarity affective dimensions\. Rectangles show average pitch length \(in voiced frames, 100 Hz\) and pitch range \(in semitones\) for a certain region and for a specific lexical form\.
## 7Discussion

The findings in this work reinforce the view of backchannels as pragmatically conditioned signals whose lexical and prosodic forms are important for their appropriateness and nuanced meaning\. The strong effect of context length suggests that backchannel choice depends on discourse\-level structure and expectations rather than solely on local acoustic or lexical cues, and that such dependencies can be effectively captured by fine\-tuned autoregressive language models\.

The joint contrastive framework provides a useful abstraction for modeling backchannels in continuous space\. Improvements over raw acoustic representations in perceptual similarity tasks indicate that the learned embedding space emphasizes pragmatically salient variation while down\-weighting speaker\- and token\-specific idiosyncrasies\. This is particularly evident in the cross\-lexical similarity results, where different lexical forms are grouped according to perceived functional similarity rather than surface form alone\.

As discussed earlier, while current generative SLMs should, in principle, be able to handle backchannels, they often struggle to generate and understand them correctlyLinet al\.\([2025a](https://arxiv.org/html/2604.16622#bib.bib63)\)\. SLMs are trained to predict or synthesize the next segment of speech end\-to\-end, which entangles backchannel behavior with many other factors \(lexical content, speaker identity, channel conditions\) and makes fine\-grained control of short feedback signals challenging\. In contrast, we learn pragmatically meaningful representations in a joint embedding space that aligns dialogue contexts with backchannel realizations via a contrastive objective\. This representation\-centric formulation supports efficient retrieval/ranking, interpretable analysis \(e\.g\., affective axes\), and direct validation against human perceptual similarity — capabilities that are difficult to obtain from generation\-focused objectives alone\. Our approach could also be complementary to generative SLMs and could serve as a lightweight backchannel selection/control module within a more modular spoken dialogue system\.

## 8Conclusion

In this paper, we address the undermodeled link between the lexical–prosodic form of backchannels and their pragmatic appropriateness and interpretation in context\. We introduced a two\-stage approach that \(i\) fine\-tunes an autoregressive LLM on spoken dialogue transcripts to obtain richer contextual representations and \(ii\) learns a shared embedding space that aligns dialogue contexts with acoustic backchannel realizations via contrastive learning\. In retrieval\-style evaluations, longer conversational context consistently improved performance, indicating that backchannel choice depends on pragmatics that extend beyond the immediately preceding turn\.

Crucially, the learned projections produced backchannel embeddings that better match human perceptual structure than raw WavLM features: agreement increased in both prosodic and cross\-lexical triadic similarity judgments\. Linear probes further showed that the embedding space supports interpretable affective dimensions \(Energy, Polarity, Surprisal\), suggesting a path toward controllable feedback generation and pragmatic inference of user feedback in conversational systems\.

## Limitations

Modeling backchannels remains challenging due to their personality\-dependentWarrineret al\.\([2013](https://arxiv.org/html/2604.16622#bib.bib102)\)and idiosyncratic natureBlomsma\_Vaitonyté\_Skantze\_Swerts\_2024, as well as their dependence on specific dyadic dynamicsCavalcanti and Skantze \([2025](https://arxiv.org/html/2604.16622#bib.bib13)\)\. Furthermore, usage varies by languageHeinz \([2003](https://arxiv.org/html/2604.16622#bib.bib39)\); Beňuš \([2016](https://arxiv.org/html/2604.16622#bib.bib6)\); Liesenfeld and Dingemanse \([2022](https://arxiv.org/html/2604.16622#bib.bib62)\), dialectWong and Peters \([2007](https://arxiv.org/html/2604.16622#bib.bib103)\); Kraaz and Bernaisch \([2022](https://arxiv.org/html/2604.16622#bib.bib53)\), and speaker proficiencyCutrone \([2005](https://arxiv.org/html/2604.16622#bib.bib19)\); Shelley and Gonzalez \([2013](https://arxiv.org/html/2604.16622#bib.bib91)\); Cutrone \([2014](https://arxiv.org/html/2604.16622#bib.bib20)\); Lee \([2020](https://arxiv.org/html/2604.16622#bib.bib57)\); Heinz \([2003](https://arxiv.org/html/2604.16622#bib.bib39)\)\. In this work, we disregard these factors, experimenting with data collected from native speakers of United States English, irrespective of dialect\. In the Fisher dataset, most participants did not know each other, making their dynamics more general and less clouded by previous acquaintance; however, we believe that future work needs to address individual, dyadic, and demographic differences to create more universally adaptive conversational agents\.

While we have previously investigated the effect of different speech encodersQian and Skantze \([2024](https://arxiv.org/html/2604.16622#bib.bib79)\), in this paper we consistently used WavLM over alternatives such as wav2vec 2\.0 and HuBERT, due to its superior performance on the SUPERB benchmarkYanget al\.\([2021](https://arxiv.org/html/2604.16622#bib.bib113)\)and its robust capacity for modeling paralinguistic featuresChenet al\.\([2022](https://arxiv.org/html/2604.16622#bib.bib14)\)\. We also kept the audio context length short \(1 second\), given that the simple temporal mean\-pooling we used cannot capture complex temporal dynamics or longer\-term acoustic dependencies anyway\. Furthermore, we kept the WavLM weights frozen throughout the training process to focus specifically on the alignment mechanism\. While fine\-tuning the audio backbone or employing different self\-supervised speech models \(or layers\) might yield higher absolute performance metrics, we assume that the relative trends and the efficacy of the contrastive framework observed in this study would likely remain consistent\.

Due to resource constraints, we fine\-tuned smaller LLMs \(4B and 7B\), requiring 1\-2 days on 8 NVIDIA RTX 3090 GPUs per model\. We avoided LLM hyperparameter tuning, as the performance differences between model sizes were negligible\. Focusing on context representations, we did not heavily optimize backchannel encoding; however, we found that a linear projection of audio\-only encodings performed best\.

## Ethical considerations

As far as we are aware, our methods do not have any harmful effects, biases, or risks, apart from what can generally be expected from machine learning\-based models \(such as introducing demographic or linguistic biases\)\. We did not develop new foundation models, nor have we created large\-scale datasets that can be mass\-deployed to cause intentional harm\.

For data collection, participants were informed that we do not intend to store personal or sensitive data and that the information they provide is anonymized\. No ethics approval was needed\.

All coding was done in Python using[Pytorch](https://pytorch.org/), and pre\-trained models were downloaded from[Hugging Face](https://huggingface.co/)\. AI assistants \([ChatGPT](https://chatgpt.com/)and[Gemini](https://gemini.google.com/)\) were used to correct grammar and reformulate sentences in the paper\. Additionally, they were used for coding in some instances\.

## Acknowledgements

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program \(WASP\) funded by the Knut and Alice Wallenberg Foundation, as well as the Swedish Research Council \(VR\) project 2020\-03812\.

## Code and other resources

## References

- J\. Allwood, J\. Nivre, and E\. Ahlsén \(1992\)On the semantics and pragmatics of linguistic feedback\.Journal of Semantics9\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1093/jos/9.1.1)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- S\. Arora, K\. Chang, C\. Chien, Y\. Peng, H\. Wu, Y\. Adi, E\. Dupoux, H\. Lee, K\. Livescu, and S\. Watanabe \(2025\)On the landscape of spoken language models: a comprehensive survey\.arXiv preprint arXiv:2504\.08528\.External Links:2504\.08528,[Link](https://arxiv.org/abs/2504.08528)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1)\.
- A\. Axelsson, H\. Buschmeier, and G\. Skantze \(2022\)Modeling feedback in interaction with conversational agents: a review\.Frontiers in Computer Science4,pp\. 744574\.External Links:[Document](https://dx.doi.org/10.3389/fcomp.2022.744574)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli \(2020\)wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.Advances in Neural Information Processing Systems 33 \(NeurIPS 2020\)33,pp\. 12449–12460\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1)\.
- Š\. Beňuš, A\. Gravano, and J\. B\. Hirschberg \(2007\)The prosody of backchannels in American English\.Proceedings of the 16th International Congress of Phonetic Sciences \(ICPhS 2007\),pp\. 1065–1068\.External Links:[Document](https://dx.doi.org/10.7916/D8WW7S1P)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p2.1)\.
- Š\. Beňuš \(2016\)The prosody of backchannels in Slovak\.InProceedings of the 8th International Conference on Speech Prosody,pp\. 75–79\.External Links:[Document](https://dx.doi.org/10.21437/SpeechProsody.2016-85)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- R\. Bertrand, G\. Ferré, P\. Blache, R\. Espesser, and S\. Rauzy \(2007\)Backchannels revisited from a multimodal perspective\.InAuditory\-Visual Speech Processing,pp\. 1–5\.External Links:[Link](https://hal.science/hal-00244490)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1),[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- A\. Boudin, R\. Bertrand, S\. Rauzy, M\. Ochs, and P\. Blache \(2021\)A multimodal model for predicting conversational feedbacks\.InText, Speech, and Dialogue,K\. Ekštein, F\. Pártl, and M\. Konopík \(Eds\.\),pp\. 537–549\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-83527-9%5F46)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1)\.
- Carol Figueroa \(2024\)FiCa speech dataset\.Note:Accessed: 2026\-01\-02External Links:[Link](https://carolfigphd.github.io/FiCa-speech-dataset/)Cited by:[§3](https://arxiv.org/html/2604.16622#S3.p1.1)\.
- J\. C\. Cavalcanti and G\. Skantze \(2025\)“Dyadosyncrasy”, idiosyncrasy and demographic factors in turn\-taking\.arXiv preprint arXiv:2505\.24736\.External Links:[Link](https://arxiv.org/abs/2505.24736)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao, J\. Wu, L\. Zhou, S\. Ren, Y\. Qian, Y\. Qian, J\. Wu, M\. Zeng, X\. Yu, and F\. Wei \(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing16\(6\),pp\. 1505–1518\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2022.3188113)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p4.1),[§2](https://arxiv.org/html/2604.16622#S2.p4.1),[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.p1.2),[Limitations](https://arxiv.org/html/2604.16622#Sx1.p2.1)\.
- C\. Cieri, D\. Miller, and K\. Walker \(2004\)The Fisher corpus: a resource for the next generations of speech\-to\-text\.InProceedings of the Fourth International Conference on Language Resources and Evaluation \(LREC’04\),Vol\.4,pp\. 69–71\.External Links:[Link](https://aclanthology.org/L04-1500/)Cited by:[§3](https://arxiv.org/html/2604.16622#S3.p1.1)\.
- H\. H\. Clark and E\. F\. Schaefer \(1989\)Contributing to discourse\.Cognitive Science13\(2\),pp\. 259–294\.External Links:[Document](https://dx.doi.org/10.1016/0364-0213%2889%2990008-6)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- P\. Cutrone \(2005\)A case study examining backchannels in conversations between Japanese\-British dyads\.Multilingua, Journal of Cross\-Cultural and Interlanguage Communication24\(3\),pp\. 237–274\.External Links:[Document](https://dx.doi.org/10.1515/mult.2005.24.3.237)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- P\. Cutrone \(2014\)A cross\-cultural examination of the backchannel behavior of Japanese and Americans: considerations for Japanese EFL learners\.Intercultural Pragmatics11\(1\),pp\. 83–120\.External Links:[Document](https://dx.doi.org/10.1515/ip-2014-0004)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour \(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.External Links:[Link](https://arxiv.org/abs/2410.00037)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of NAACL\-HLT,Vol\.1,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/V1/N19-1423)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p3.1)\.
- E\. Ekstedt and G\. Skantze \(2020\)TurnGPT: a transformer\-based language model for predicting turn\-taking in spoken dialog\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 2981–2990\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.268)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p3.1)\.
- G\. Ferré and S\. Renaudier \(2017\)Unimodal and bimodal backchannels in conversational English\.InProceedings of the 21st Workshop on the Semantics and Pragmatics of Dialogue \(SemDial 2017\),pp\. 27–37\.External Links:[Document](https://dx.doi.org/10.21437/SemDial.2017-3)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- C\. Figueroa, A\. Adigwe, M\. Ochs, and G\. Skantze \(2022\)Annotation of communicative functions of short feedback tokens in Switchboard\.InProceedings of the Thirteenth Language Resources and Evaluation Conference \(LREC\),pp\. 1849–1859\.External Links:[Link](https://aclanthology.org/2022.lrec-1.197/)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1),[§5\.1](https://arxiv.org/html/2604.16622#S5.SS1.SSS0.Px3.p1.1)\.
- C\. Figueroa, M\. de Korte, M\. Ochs, and G\. Skantze \(2024\)Mhm… Yeah? Okay\! evaluating the naturalness and communicative function of synthesized feedback responses in spoken dialogue\.InProceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue \(SIGdial\),pp\. 544–553\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.sigdial-1.46)Cited by:[§3](https://arxiv.org/html/2604.16622#S3.p1.1)\.
- Y\. Fukunaga, R\. Nishimura, K\. Ohta, and N\. Kitaoka \(2025\)Backchannel prediction for natural spoken dialog systems using general speaker and listener information\.InProceedings of Interspeech 2025,pp\. 1078–1082\.External Links:[Document](https://dx.doi.org/10.21437/interspeech.2025-1316)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- R\. Fusaroli, K\. Tylén, K\. Garly, J\. Steensig, M\. H\. Christiansen, and M\. Dingemanse \(2017\)Measures and mechanisms of common ground: backchannels, conversational repair, and interactive alignment in free and task\-oriented social interactions\.InProceedings of the Annual Meeting of the Cognitive Science Society \(CogSci 2017\),pp\. 2055–2060\.External Links:[Link](https://www.mpi.nl/publications/item2449737/measures-and-mechanisms-common-ground-backchannels-conversational-repair)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- Gemma Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- J\.J\. Godfrey, E\.C\. Holliman, and J\. McDaniel \(1992\)SWITCHBOARD: telephone speech corpus for research and development\.InProceedings of ICASSP,Vol\.1,pp\. 517–520\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP.1992.225858)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1)\.
- Google Deepmind \(2025\)Gemma 3 4B\.Note:Accessed: 2026\-01\-02External Links:[Link](https://huggingface.co/google/gemma-3-4b-pt)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- A\. Gravano and J\. Hirschberg \(2011\)Turn\-taking cues in task\-oriented dialogue\.Computer Speech & Language25\(3\),pp\. 601–634\.External Links:[Document](https://dx.doi.org/10.1016/j.csl.2010.10.003)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- B\. Heinz \(2003\)Backchannel responses as strategic responses in bilingual speakers’ conversations\.Journal of Pragmatics35\(7\),pp\. 1113–1142\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S0378-2166%2802%2900190-X)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- M\. Heldner, J\. Edlund, and J\. Hirschberg \(2010\)Pitch similarity in the vicinity of backchannels\.InProceedings of Interspeech 2010,pp\. 3054–3057\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2010-58)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- M\. Heldner, A\. Hjalmarsson, and J\. Edlund \(2013\)Backchannel relevance spaces\.InNordic Prosody XI,pp\. 137–146\.Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p2.1),[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- C\. Howes and A\. Eshghi \(2021\)Feedback relevance spaces: interactional constraints on processing contexts in dynamic syntax\.Journal of Logic, Language and Information30\(2\),pp\. 331–362\.External Links:[Document](https://dx.doi.org/10.1007/s10849-020-09328-1)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed \(2021\)HuBERT: self\-supervised speech representation learning by masked prediction of hidden units\.IEEE/ACM Transactions on Audio, Speech, and Language Processing29,pp\. 3451–3460\.External Links:[Document](https://dx.doi.org/10.1109/TASLP.2021.3122291)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1)\.
- K\. Inoue, D\. Lala, G\. Skantze, and T\. Kawahara \(2025\)Yeah, Un, Oh: continuous and real\-time backchannel prediction with fine\-tuning of voice activity projection\.InProceedings of NAACL \(Volume 1: Long Papers\),pp\. 7171–7181\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.367)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- R\. Ishii, X\. Ren, M\. Muszynski, and L\. Morency \(2021\)Multimodal and multitask approach to listener’s backchannel prediction: can prediction of turn\-changing and turn\-management willingness improve backchannel modeling?\.InIVA ’21: Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents,pp\. 131–138\.External Links:[Document](https://dx.doi.org/10.1145/3472306.3478360)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p2.1)\.
- J\. Y\. Jang, S\. Kim, M\. Jung, S\. Shin, and G\. Gweon \(2021\)BPM\_MT: enhanced backchannel prediction model using multi\-task learning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 3447–3452\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.277)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch,et al\.\(2023\)Mistral 7B\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- D\. Jurafsky, E\. Shriberg, B\. Fox, and T\. Curl \(1998\)Lexical, prosodic, and syntactic cues for dialog acts\.InDiscourse Relations and Discourse Markers,External Links:[Link](https://api.semanticscholar.org/CorpusID:31039)Cited by:[§5\.4](https://arxiv.org/html/2604.16622#S5.SS4.p1.1)\.
- H\. Koiso, Y\. Horiuchi, S\. Tutiya, A\. Ichikawa, and Y\. Den \(1998\)An analysis of turn\-taking and backchannels based on prosodic and syntactic features in Japanese map task dialogs\.Language and Speech41\(3\-4\),pp\. 295–321\.External Links:[Document](https://dx.doi.org/10.1177/002383099804100404)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- M\. Kraaz and T\. Bernaisch \(2022\)Backchannels and the pragmatics of South Asian Englishes\.World Englishes41\(2\),pp\. 224–243\.External Links:[Document](https://dx.doi.org/10.1111/weng.12522)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- P\. Kuppens, F\. Tuerlinckx, J\. A\. Russell, and L\. F\. Barrett \(2013\)The relation between valence and arousal in subjective experience\.Psychological Bulletin139\(4\),pp\. 917\.External Links:[Document](https://dx.doi.org/10.1037/a0030811)Cited by:[§5\.1](https://arxiv.org/html/2604.16622#S5.SS1.SSS0.Px3.p1.1)\.
- K\. Lee \(2020\)Backchannels as a cooperative strategy in ELF communications\.Korean Journal of English Language and Linguistics20,pp\. 257–281\.External Links:[Document](https://dx.doi.org/10.15738/kjell.20..202008.257)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- Z\. Li, X\. Zhang, Y\. Zhang, D\. Long, P\. Xie, and M\. Zhang \(2023\)Towards general text embeddings with multi\-stage contrastive learning\.arXiv preprint arXiv:2308\.03281\.External Links:[Link](https://arxiv.org/abs/2308.03281)Cited by:[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px3.p1.1)\.
- W\. Liermann, Y\. Park, Y\. Choi, and K\. Lee \(2023\)Dialogue act\-aided backchannel prediction using multi\-task learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 15073–15079\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.1006)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- A\. Liesenfeld and M\. Dingemanse \(2022\)Bottom\-up discovery of structure and variation in response tokens \(‘backchannels’\) across diverse languages\.InProceedings of Interspeech 2022,pp\. 1126–1130\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2022-11288)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- G\. Lin, J\. Lian, T\. Li, Q\. Wang, G\. Anumanchipalli, A\. H\. Liu, and H\. Lee \(2025a\)Full\-Duplex\-Bench: a benchmark to evaluate full\-duplex spoken dialogue models on turn\-taking capabilities\.arXiv preprint arXiv:2503\.04721\.External Links:[Link](https://arxiv.org/abs/2503.04721)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1),[§7](https://arxiv.org/html/2604.16622#S7.p3.1)\.
- Y\. Lin, Y\. Zheng, M\. Zeng, and W\. Shi \(2025b\)Predicting turn\-taking and backchannel in human\-machine conversations using linguistic, acoustic, and visual signals\.InProceedings of ACL \(Volume 1: Long Papers\),pp\. 15310–15322\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.743)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1)\.
- L\. Mai and J\. Carson\-Berndsen \(2025\)Real\-time textless dialogue generation\.arXiv preprint arXiv:2501\.04877\.External Links:[Link](https://arxiv.org/abs/2501.04877)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- Meta AI \(2024\)LLaMA 3\.1 8B\.Note:Accessed: 2026\-01\-02External Links:[Link](https://huggingface.co/meta-llama/Llama-3.1-8B)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- Mistral AI \(2024\)Mistral 7B v0\.3\.Note:Accessed: 2026\-01\-02External Links:[Link](https://huggingface.co/mistralai/Mistral-7B-v0.3)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- A\. Mollahosseini, B\. Hasani, and M\. H\. Mahoor \(2019\)AffectNet: a database for facial expression, valence, and arousal computing in the wild\.IEEE Transactions on Affective Computing10\(1\),pp\. 18–31\.External Links:[Document](https://dx.doi.org/10.1109/TAFFC.2017.2740923)Cited by:[§5\.1](https://arxiv.org/html/2604.16622#S5.SS1.SSS0.Px3.p1.1)\.
- P\. Müller, M\. Dietz, D\. Schiller, D\. Thomas, H\. Lindsay, P\. Gebhard, E\. André, and A\. Bulling \(2022\)MultiMediate’22: backchannel detection and agreement estimation in group interactions\.InProceedings of the 30th ACM International Conference on Multimedia,pp\. 7109–7114\.External Links:[Document](https://dx.doi.org/10.1145/3503161.3551589)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1)\.
- T\. A\. Nguyen, E\. Kharitonov, J\. Copet, Y\. Adi, W\. Hsu, A\. Elkahky, P\. Tomasello, R\. Algayres, B\. Sagot, A\. Mohamed, and E\. Dupoux \(2023\)Generative spoken dialogue language modeling\.Transactions of the Association for Computational Linguistics11,pp\. 250–266\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00545)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1)\.
- A\. v\. d\. Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.External Links:[Link](https://arxiv.org/abs/1807.03748)Cited by:[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px1.p1.1)\.
- D\. Ortega, C\. Li, and N\. T\. Vu \(2020\)Oh, Jeez\! or uh\-huh? a listener\-aware backchannel predictor on ASR transcriptions\.InProceedings of ICASSP,pp\. 8064–8068\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP40776.2020.9054223)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p2.1)\.
- Y\. Park, W\. Liermann, Y\. Choi, and K\. J\. Lee \(2024\)Improving backchannel prediction leveraging sequential and attentive context awareness\.InFindings of the Association for Computational Linguistics: EACL 2024,pp\. 1689–1694\.External Links:[Link](https://aclanthology.org/2024.findings-eacl.118/)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- R\. Poppe, K\. P\. Truong, and D\. Heylen \(2013\)Perceptual evaluation of backchannel strategies for artificial listeners\.Autonomous Agents and Multi\-Agent Systems27\(2\),pp\. 235–253\.External Links:[Document](https://dx.doi.org/10.1007/s10458-013-9219-z)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1)\.
- R\. Poppe, K\. P\. Truong, D\. Reidsma, and D\. Heylen \(2010\)Backchannel strategies for artificial listeners\.InInternational Conference on Intelligent Virtual Agents,pp\. 146–158\.External Links:[Document](https://dx.doi.org/10.1007/978-3-642-15892-6%5F16)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- L\. Qian, C\. Figueroa, and G\. Skantze \(2025\)Representation of perceived prosodic similarity of conversational feedback\.InProceedings of Interspeech 2025,pp\. 374–378\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2025-1771)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p2.1),[§3](https://arxiv.org/html/2604.16622#S3.p1.1),[§3](https://arxiv.org/html/2604.16622#S3.p2.1),[§5\.1](https://arxiv.org/html/2604.16622#S5.SS1.p1.1)\.
- L\. Qian and G\. Skantze \(2024\)Joint learning of context and feedback embeddings in spoken dialogue\.InProceedings of Interspeech 2024,pp\. 2955–2959\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-1082)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p3.1),[§2](https://arxiv.org/html/2604.16622#S2.p1.1),[§2](https://arxiv.org/html/2604.16622#S2.p5.1),[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px4.p1.3),[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px6.p1.1),[Limitations](https://arxiv.org/html/2604.16622#Sx1.p2.1)\.
- Qwen \(2024\)Qwen2\.5 7B\.Note:Accessed: 2026\-01\-02External Links:[Link](https://huggingface.co/Qwen/Qwen2.5-7B)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),Vol\.139,pp\. 8748–8763\.External Links:[Link](https://dblp.org/rec/conf/icml/RadfordKHRGASAM21.html)Cited by:[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2604.16622#S4.SS2.SSS0.Px5.p1.1)\.
- R\. Ruede, M\. Müller, S\. Stüker, and A\. Waibel \(2017\)Enhancing backchannel prediction using word embeddings\.InProceedings of Interspeech 2017,pp\. 879–883\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2017-1606)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- R\. Ruede, M\. Müller, S\. Stüker, and A\. Waibel \(2019\)Yeah, right, uh\-huh: a deep learning backchannel predictor\.InAdvanced Social Interaction with Agents: 8th International Workshop on Spoken Dialog Systems,pp\. 247–258\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-92108-2%5F25)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p2.1)\.
- L\. Shelley and F\. Gonzalez \(2013\)Back channeling: function of back channeling and L1 effects on back channeling in L2\.Linguistic Portfolios2\(1\),pp\. 9\.External Links:[Link](https://repository.stcloudstate.edu/stcloud_ling/vol2/iss1/9/)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- K\. P\. Truong, R\. W\. Poppe, I\.A\. de Kok, and D\. K\.J\. Heylen \(2011\)A multimodal analysis of vocal and visual backchannels in spontaneous dialogs\.InProceedings of Interspeech 2011,pp\. 2973–2976\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2011-744)Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- J\. Wang, L\. Chen, A\. Khare, A\. Raju, P\. Dheram, D\. He, M\. Wu, A\. Stolcke, and V\. Ravichandran \(2024\)Turn\-taking and backchannel prediction with acoustic and large language model fusion\.InProceedings of ICASSP,pp\. 12121–12125\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP48485.2024.10447196)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p1.1)\.
- N\. G\. Ward, D\. Marco, and O\. Fuentes \(2025\)Which prosodic features matter most for pragmatics?\.InProceedings of ICASSP,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP49660.2025.10889382)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p4.1)\.
- A\. B\. Warriner, V\. Kuperman, and M\. Brysbaert \(2013\)Norms of valence, arousal, and dominance for 13,915 English lemmas\.Behavior Research Methods45\(4\),pp\. 1191–1207\.External Links:[Document](https://dx.doi.org/10.3758/s13428-012-0314-x)Cited by:[§5\.1](https://arxiv.org/html/2604.16622#S5.SS1.SSS0.Px3.p1.1),[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- D\. Wong and P\. Peters \(2007\)A study of backchannels in regional varieties of English, using corpus mark\-up as the means of identification\.International Journal of Corpus Linguistics12\(4\),pp\. 479–510\.External Links:[Document](https://dx.doi.org/10.1075/ijcl.12.4.03won)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p1.1)\.
- A\. Yang, B\. Yu, C\. Li, D\. Liu,et al\.\(2025\)Qwen2\.5\-1M technical report\.arXiv preprint arXiv:2501\.15383\.External Links:[Link](https://arxiv.org/abs/2501.15383)Cited by:[§4\.1](https://arxiv.org/html/2604.16622#S4.SS1.p1.1)\.
- S\. Yang, P\. Chi, Y\. Chuang, C\. J\. Lai, K\. Lakhotia, Y\. Y\. Lin, A\. T\. Liu, J\. Shi, X\. Chang, G\. Lin,et al\.\(2021\)SUPERB: Speech Processing Universal PERformance Benchmark\.Proceedings of Interspeech 2021,pp\. 1194–1198\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1775)Cited by:[Limitations](https://arxiv.org/html/2604.16622#Sx1.p2.1)\.
- V\. H\. Yngve \(1970\)On getting a word in edgewise\.InCLS\-70,pp\. 567–577\.Cited by:[§1](https://arxiv.org/html/2604.16622#S1.p1.1)\.
- Y\. Zhang, S\. Sun, M\. Galley, Y\. Chen, C\. Brockett, X\. Gao, J\. Gao, J\. Liu, and B\. Dolan \(2020\)DialoGPT: large\-scale generative pre\-training for conversational response generation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations,pp\. 270–278\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-demos.30)Cited by:[§2](https://arxiv.org/html/2604.16622#S2.p3.1)\.

## Appendix ATranscripts for LLM fine\-tuning

### A\.1Spoken dialogue transcripts

For LLM fine\-tuning, we used unpunctuated lowercase transcripts based on the ones provided with the Fisher dataset\. A set of special characters were used to indicate speaker shifts and overlapping speech, however these were treated as ordinary characters from the perspective of the language model, so that no new tokens had to be introduced:

- •<A\>and<B\>: the beginning of Speaker A or B’s turn, respectively
- •Slash\(//\): turn shift
- •Braces\(\{\}\): the part of a given turn \(Turn A\) that overlaps with parts of the next turn \(Turn B\)\. There is always a corresponding part surrounded by square brackets in the next turn \(Turn B\), unless the current turn \(Turn A\) is the last one in the transcript\.
- •Square brackets\(\[\]\): the part of a given turn \(Turn B\) that overlaps with parts of the previous turn \(Turn A\)\. There is always a corresponding part surrounded by braces in the previous turn \(Turn A\), unless the current turn \(Turn B\) is the first one in the transcript\.

Backchannels were located by searching for complete turns that consistedonlyof the backchannels we are investigating \(see Section[3](https://arxiv.org/html/2604.16622#S3)\) andonly once, i\.e\., we identified them only if they were surrounded by two turn shifts and a speaker token \(with occasional braces and brackets\)\.

Here are two example transcripts, with backchannels marked in bold:

### Example transcript 1

<B\> hi / <A\> hello / <B\> hi / <A\> oh okay um my name is brandon um you / <B\> right um my name’s rajul mm / <A\> okay okay so uh what do you think / <B\> um i didn’t quite catch the topic i mean i got the gist of it but um can you hear the topic of the day / <A\> okay basically um how has corporate scandals affected you or do you think um what is what is the affect of corporate scandals on america um do you believe it’s responsible for the m the mild recession that we we been having lately mm / <B\>right/ <A\>mhm/ <B\> um well uh personally for me to the the solution and uh you know the aftermath of the uh uh the uh tech boom and the uh bubble thereafter because i think um probably um because uh i think because of my ignorance at that point in time about / <A\> mhm mhm / <B\> um about probably what i would call uh the truth as i see it wh what actually goes on in in the um in in the stock market as well as uh uh the biggest uh financial institutions uh in america um but initially e i i i really could not believe uh um things that were coming out uh in in the newspapers you know arthur anderson and uh citibank merrill lynch you know just n name a all those groups were like uh um being charged for um um for misguiding and misleading uh the investors / <A\> mhm mhm / <B\> and and thereby um in my words uh duping people uh of their money uh uh pretty blatantly \{ knowing \} / <A\> \[mhm\] /

### Example transcript 2

<A\> and you know we we uh all / <B\>yeah/ <A\> of us hurt uh because of that and i believe it’s contributed to the recession it’s i mean the recess recessions happen because they happen i mean they they come in waves but / <B\>right/ <A\> i don’t i don’t think that me i don’t think that america takes white collar white collar crimes seriously and \{ it \} / <B\> \[right\] / <A\> doesn’t it doesn’t try to it doesn’t try to stop it at all i don’t i think i don’t know \{ h \} / <B\> \[right\] / <A\> uh if you have a theory behind that because also i think the main people who are running the / <B\>mhm/ <A\> the country the main people who are actually contributing to politicians who run the countr the country are all these companies \{ who are \} / <B\> \[right\] / <A\> so rich and have all this money and so if they you know if they embezzle a few million dollars it doesn’t hurt anyone cause we’re still in power \{ th \} / <B\> \[right\] / <A\> these are the um the people in power who are talking you know we’re st we’re still in power and like they’re still paying us off so it doesn’t really matter but everyone \{ but \} / <B\> \[ right right \] / <A\> everyone else hurts in the long run so i don’t know /

InExample transcript 1, the penultimate turn overlaps with the last turn, which means that\{ knowing \}is uttered at the same time as\[ mhm \]in the original audio\. Other vocalized dialogue phenomena are included, e\.g\., repetitions and repairs, without special markers\.

### A\.2Defining the context

When restricting the number of past turns to, e\.g\., 2, the contexts corresponding to some of the above examples appear as follows \(where everything before the backchannel in bold is treated as context\):

<B\> um i didn’t quite catch the topic i mean i got the gist of it but um can you hear the topic of the day / <A\> okay basically um how has corporate scandals affected you or do you think um what is what is the affect of corporate scandals on america um do you believe it’s responsible for the m the mild recession that we we been having lately mm / <B\>right

<A\> okay basically um how has corporate scandals affected you or do you think um what is what is the affect of corporate scandals on america um do you believe it’s responsible for the m the mild recession that we we been having lately mm / <B\> right / <A\>mhm

<A\> mhm mhm / <B\> and and thereby um in my words uh duping people uh of their money uh uh pretty blatantly \{ knowing \} / <A\> \[mhm

Thus, when computing the perplexity of the backchannel token, or the LLM context embedding, all text up to the backchannel \(marked in bold\) is used, including the current turn’s speaker token and potential opening brackets\.

## Appendix BBest hyperparameters

The hyperparameters of the best models for each configuration in Section[4\.2](https://arxiv.org/html/2604.16622#S4.SS2)are shown in Table[7](https://arxiv.org/html/2604.16622#A2.T7)and[8](https://arxiv.org/html/2604.16622#A2.T8)\.

Context modalityTextembeddingsAudio \+ textTextGemma 3 4B4, 128, 20482, 128, 4096LLaMA 3\.1 8B3, 64, 40963, 128, 4096Qwen2\.5 7B4, 64, 20481, 128, 2048Mistral 7B3, 64, 20481, 64, 8192Table 7:Hyperparameters for the models \(5 turns\) in Table[2](https://arxiv.org/html/2604.16622#S4.T2), listed in the ordernumber of layers,embedding size,batch size\. For audio\-only, the optimal configuration was 2, 64, and 2048\.Model13Gemma 3 4B2, 64, 20483, 64, 1024LLaMA 3\.1 8B4, 256, 20482, 64, 4096Qwen2\.5 7B3, 64, 20483, 64, 2048Mistral 7B3, 64, 20484, 64, 4096Table 8:Hyperparameters for the models \(text \+ audio, fine\-tuned LLM context encoder\) in Table[3](https://arxiv.org/html/2604.16622#S4.T3)in the ordernumber of layers,embedding size, andbatch size\. The number of past turns is 1 and 3\.
## Appendix CPerception study

In the perception study, participants were given three tasks, with the last one consisting of three separate questions regarding Energy, Polarity and Surprisal\. In total, five questions were asked for each stimulus set\. The participants were shown a range of examples at the beginning of the study\.

The perception study was conducted on 100 native speakers of North American \(U\.S\.\) English who reported English as their primary — most frequently used — language and the United States as their primary place of residence during their first 18 years\. The participants had no hearing difficulties\. Each participant received 21 stimulus sets and two additional sets for attention checks\. Each stimulus set was seen by at least three subjects\.

Participants were recruited via[Prolific](https://www.prolific.com/)and the experiment was hosted on[cognition\.run](https://www.cognition.run/)\. The median completion time was approximately 56 minutes, and the participants were compensated with Prolific’s default reward per hour\.

## Appendix DOutline of perception study questions

#### General instruction:

Listen to the context and the feedback responses\. The feedback responses are repeated at the beginning of each question where you have to rate them individually\.

\[context audio\]

\[context transcript\]

\[Feedback 1, Feedback 2, Feedback 3 \(only audio\)\]

#### Question 1:

Please rate the feedback responses based on how well they match the context \(min: 1, max: 5\)\.

#### Question 2:

Choose the two responses that are the most similar to each other\.

#### Question 3:

Rate the energy level: how energetic is the response \(min: 1, max: 5\)?

#### Question 4:

Rate polarity: how positive is the response \(min: 1, max: 5\)?

#### Question 5:

Rate surprisal: how surprised does the feedback speaker sound \(min: 1, max: 5\)?

## Appendix EInformed consent and general information \(verbatim\)

For each question, you will hear:

- •onecontextclip \(Speaker 1\)
- •three shortfeedbackclips \(Speaker 2\), labeled as 1, 2 and 3

### Your task

1. 1\.Rate compatibility - •For each feedback, rate how well it fits the context as a possible response - •Scale:1=1=not at all,5=5=extremely - •Consider factors like naturalness, appropriateness and expectedness
2. 2\.Choose similar feedback responses - •Pick the two feedback responses most similar in role, attitude, intention and emotion \(not just meaning\!\), given the current context - •Options: 1–2, 2–3 or 1–3 - •Consider factors like how replaceable they are with each other\! Take into account both sound and meaning\.
3. 3\.Rate energy - •How energetic does the feedback speaker sound? - •Scale:1=1=not at all,5=5=extremely
4. 4\.Rate polarity - •How positive is what the feedback speaker said? - •Scale:1=1=not at all,5=5=extremely
5. 5\.Rate surprisal - •How surprised does the feedback speaker sound? - •Scale:1=1=not at all,5=5=extremely

### Instructions

- •You will see some examples on the first six pages\. Please, read them carefully\.
- •You have to wait a few seconds for the survey to load\.
- •Evaluate feedback together with context, not in isolation\. For better judgment, play each feedbackimmediately after the context\.
- •Consider tone, emotion, and purpose — butignoredifferences in recording quality and loudness\.

### Listening requirements

- •Useheadphonesin a calm environment\.
- •Replay samples as needed, but youcannot returnto earlier ones\.
- •The volume of the context can be adjusted\.
- •If an audio file doesn’t load or doesn’t play, wait a few seconds\. Once loaded, it should be possible to play it without delay\. If it still does not work, please take a note of where the issue arose and contact us\.

### Attention checks

- •Very short or long completion times may lead to rejection\.
- •Failing attention checks results in rejection\. Listening activityis recordedfor validation\.
- •The survey will not end if you fail the attention checks, and your answers, including the checks, will be evaluated after your submission\.

### Other notes

- •Answers will help study how people perceivebackchannelsin conversation\.
- •Audio clips are from a dataset and do not represent our views\.
- •Responses are anonymized; no personal/sensitive data is collected\.
- •You may stop anytime or contact us via Prolific if needed\.

By continuing, you confirm voluntary participation and acceptance of these conditions\.

Thank you\!

## Appendix FDistribution of affective ratings

The mean and standard deviation over the median scores for each backchannel, grouped by lexical category, are shown in Table[9](https://arxiv.org/html/2604.16622#A6.T9)\. These provide insight into the typical human perception of each category and how it varies within each group\. The data suggests a hierarchy from high\-arousal, expressive words \(top of the table\) to low\-arousal, passive continuers \(bottom of the table\)\.

#### Highest Energy:

The token “wow” \(3\.74±0\.703\.74\\pm 0\.70\) has the highest average Energy, indicating that it is perceived as the most intense or aroused response in general\. It is closely followed by strong, occasionally enthusiastic agreements like “absolutely” \(3\.673\.67\) and “exactly” \(3\.673\.67\)\.

#### Lowest Energy:

The tokens “mhm” \(2\.70±0\.682\.70\\pm 0\.68\), “yeah” \(2\.832\.83\), and “right” \(2\.872\.87\) are at the bottom\. This aligns with their typical linguistic function as “passive” continuers, i\.e\., simple signals that the listener is still present, without adding significant emotion\.

#### Most positive:

“Absolutely” \(4\.03±0\.674\.03\\pm 0\.67\) and “definitely” \(3\.98±0\.883\.98\\pm 0\.88\) are the most positive tokens\. This suggests that multi\-syllabic, explicit agreement words carry more positive weight than short sounds\.

#### Least positive \(most neutral\):

The lowest Polarity scores belong to “mm” \(2\.712\.71\) and “mhm” \(2\.922\.92\)\. These scores are likely lower not because they are negative, but because they are highly neutral \(our selection does not, for the most part, include backchannels that are usually perceived as expressing negative sentiment\)\.

#### Most surprised:

Words like “wow” \(3\.923\.92\), “really” \(3\.843\.84\), “ah” \(3\.823\.82\), and “oh” \(3\.693\.69\) all have very high Surprisal scores\. These are typically used when the listener is reacting to new, shocking, or interesting information\.

#### Least surprised:

Words like “mhm” \(2\.042\.04\), “yep” \(2\.212\.21\), “right” \(2\.222\.22\), and “sure” \(2\.302\.30\) have low Surprisal scores\. These are used to confirm known information or simply to agree\.

#### High variance/variability:

Certain tokens have consistently high standard deviations, suggesting that they can be pronounced in various ways and that their meaning depends heavily on how they are said \(tone/prosody\) rather than just on the word itself\. Examples include “oh”, “mm”, and “cool”\.

Table 9:Mean ± standard deviation per backchannel lexical token\. Calculated on the median rating of each backchannel\.tokenenergypolaritysurprisalwow3\.74±\\pm0\.703\.64±\\pm0\.613\.92±\\pm0\.71absolutely3\.67±\\pm0\.734\.03±\\pm0\.672\.45±\\pm0\.56exactly3\.67±\\pm0\.613\.96±\\pm0\.572\.41±\\pm0\.66ah3\.57±\\pm0\.743\.27±\\pm0\.673\.82±\\pm0\.78really3\.56±\\pm0\.743\.22±\\pm0\.723\.84±\\pm0\.74definitely3\.51±\\pm0\.703\.98±\\pm0\.882\.34±\\pm0\.44oh3\.45±\\pm0\.863\.17±\\pm0\.873\.69±\\pm0\.89good3\.34±\\pm0\.713\.55±\\pm0\.592\.84±\\pm0\.63yes3\.11±\\pm0\.733\.39±\\pm0\.652\.21±\\pm0\.62cool3\.07±\\pm0\.833\.49±\\pm0\.692\.65±\\pm0\.81okay3\.02±\\pm0\.643\.21±\\pm0\.582\.54±\\pm0\.69uh\-huh3\.00±\\pm0\.653\.07±\\pm0\.692\.38±\\pm0\.67yep2\.95±\\pm0\.623\.30±\\pm0\.572\.21±\\pm0\.61sure2\.89±\\pm0\.713\.34±\\pm0\.642\.30±\\pm0\.71mm2\.88±\\pm0\.822\.71±\\pm0\.702\.82±\\pm0\.95right2\.87±\\pm0\.663\.30±\\pm0\.662\.22±\\pm0\.52yeah2\.83±\\pm0\.623\.14±\\pm0\.552\.20±\\pm0\.60mhm2\.70±\\pm0\.682\.92±\\pm0\.612\.04±\\pm0\.58