Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating

# Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating
Source: [https://arxiv.org/html/2607.23037](https://arxiv.org/html/2607.23037)
\(2026\)

###### Abstract\.

Large language models \(LLMs\) can predict interpersonal attraction from conversation transcripts, but it remains unclear what a speech predictor can add beyond transcript\-only LLM prediction\. Using Japanese speed\-dating conversations, we combine predictions from a transcript\-only LLM and a supervised speech predictor to estimate participants’ reported liking of their partners\. We show that speech can complement transcript\-only LLM prediction, but that this complementarity is conditional rather than universal\. Combining the two predictions significantly improves pairwise ranking accuracy over the transcript\-only LLM alone in all evaluated conditions\. By contrast, gains in per\-participant Pearsonrrvary across conversation rounds and rating directions, with none significant after correction\. Retrospectively, theserrgains are concentrated among participants for whom the speech predictor is more accurate\. Speech can therefore retain predictive value even when an LLM predicts attraction from transcripts\. The relevant question is not simply whether speech helps, but where its complementarity emerges\.111Code is available at[github\.com/yurikomium/speech\-llm\-complementarity](https://github.com/yurikomium/speech-llm-complementarity)\.

interpersonal attraction; speed dating; large language models; speech processing; multimodal fusion

††journalyear:2026††copyright:cc††conference:INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION; October 05–09, 2026; Napoli, Italy††booktitle:INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION \(ICMI ’26\), October 05–09, 2026, Napoli, Italy††doi:10\.1145/3776574\.3831151††isbn:979\-8\-4007\-2318\-6/2026/10††ccs:Human\-centered computing Empirical studies in collaborative and social computing††ccs:Applied computing Psychology††ccs:Computing methodologies Natural language processing## 1\.Introduction

Interpersonal attraction formed during an initial encounter can shape whether a relationship continues to develop\(Berscheid and Reis,[1998](https://arxiv.org/html/2607.23037#bib.bib8)\)\. Speed dating provides a naturalistic setting for studying how initial attraction forms: participants meet multiple potential partners and report how much they like each one\. Conversational behavior can be informative in this setting\. Pitch convergence, for example, varies with how speakers perceive their partners’ visual attractiveness and overall likability\(Michalsky and Schoormann,[2017](https://arxiv.org/html/2607.23037#bib.bib2)\)\. These findings motivate computational models of attraction from interaction data\.

Automatic prediction of interpersonal outcomes from conversations has developed along two largely separate lines\. One uses large language models \(LLMs\) to infer outcomes from transcripts\. In a direct study of speed dates, LLMs predicted objective and subjective indicators of interaction success from conversation transcripts\(Matzet al\.,[2026](https://arxiv.org/html/2607.23037#bib.bib17)\)\. The other uses supervised models with lexical, speech, dialogue, and visual signals to predict social judgments and outcomes\(Ranganathet al\.,[2009](https://arxiv.org/html/2607.23037#bib.bib13); Veenstra and Hung,[2011](https://arxiv.org/html/2607.23037#bib.bib14)\)\. Together, these lines raise a broad question about which conversational signals remain useful beyond transcript\-only LLM prediction\. We address a narrower question for speed\-dating attraction: can a supervised speech predictor add predictive value beyond a transcript\-only LLM, and where does that value emerge?

We study this question using the Multi\-Modal Speed Dating corpus\(Ishiiet al\.,[2023](https://arxiv.org/html/2607.23037#bib.bib15)\), which contains Japanese speed\-dating conversations with transcripts, speaker\-specific audio, and post\-conversation liking scores\. Our task is to predict each participant’s reported liking of their partners, with an emphasis on differentiation within the participant’s partner set\. We define*speech\-derived complementarity*as the additional predictive value contributed by a supervised speech predictor beyond the evaluated transcript\-only LLM’s predictions, without claiming that the speech predictor captures information fundamentally absent from transcripts\.

For the primary analysis, the transcript\-only predictor is Claude Sonnet 4\.6 with extended thinking disabled \(hereafter Claude\), evaluated using zero\-shot prompting\. The supervised speech predictor is built on frozen HuBERT\-Large representations \(hereafter HuBERT\)\. We combine their scalar predictions through weighted score\-level late fusion, as shown in Figure[1](https://arxiv.org/html/2607.23037#S1.F1)\. This deliberately simple combination serves as a diagnostic for additional predictive value rather than as a new fusion architecture\. We evaluate complementarity across two conversation rounds, two rating directions, and participants\. Two multimodal LLMs that receive transcripts and audio provide auxiliary direct\-input comparisons\.

In summary, we clarify where a supervised speech predictor adds predictive value beyond the evaluated transcript\-only LLM in speed\-dating attraction prediction\. We do so by examining this added value across evaluation metrics, conversation rounds and rating directions, and participants, using score\-level late fusion only as a diagnostic\. Supporting analyses assess whether the transcript\-only LLM and speech predictor make redundant predictions by quantifying their within\-rater prediction overlap and mutual incremental associations\. Together, these analyses shift the question from whether speech helps on average to where its complementarity emerges\.

Relative to Claude, fusion significantly improves pairwise ranking accuracy in all four round\-by\-rating\-direction settings\. Gains in per\-participant Pearsonrr, however, vary across these settings, and no paired gain is statistically significant after correction\. Retrospectively, participant\-levelrrgains are larger where HuBERT is more accurate, although this association is partly expected because the HuBERT prediction enters the fused score\. The Claude and HuBERT predictions share little within\-rater variance, with each explaining comparable incremental in\-sample variance beyond the other\. As supporting context, Claude has higher per\-participantrrthan every tested supervised text model in all four settings\.

Taken together, these findings do not support a blanket claim that adding speech uniformly improves attraction prediction\. Instead, they show that speech can retain predictive value even when an LLM predicts attraction from transcripts\. For multimodal social prediction, the relevant question is therefore not simply whether speech helps, but where its complementarity emerges\.

![Refer to caption](https://arxiv.org/html/2607.23037v1/)Figure 1\.Prediction pipeline for F2M \(female rating male\) and M2F \(male rating female\)\. Claude Sonnet 4\.6, used with extended thinking disabled and zero\-shot prompting, predicts liking from the full two\-speaker transcript, while frozen HuBERT and a supervised head predict from rater\-side speech\. Weighted score\-level fusion combines the predictions using a fold\-selected weightww\.A prediction pipeline for liking after a speed\-dating conversation\. F2M denotes a female participant rating a male partner, and M2F denotes a male participant rating a female partner\. The conversation feeds two branches\. The upper branch sends the full two\-speaker transcript to Claude Sonnet 4\.6 with zero\-shot prompting and extended thinking disabled, yielding the transcript\-only prediction\. The lower branch sends rater\-side speech through frozen HuBERT and a supervised prediction head, yielding the HuBERT prediction\. The two predictions are combined using weights $w$ and $1\-w$ to produce the fused predicted liking score\.
## 2\.Related Work

### 2\.1\.LLMs for Interpersonal Inference

Beyond attraction, LLMs have been evaluated as judges of empathic communication in text\-based conversations\. When given criteria for each framework and expert\-annotated examples, LLM–expert agreement approached expert–expert agreement across four evaluative frameworks\(Kumaret al\.,[2026](https://arxiv.org/html/2607.23037#bib.bib16)\)\.

Most directly related to the present paper, Matz et al\.\(Matzet al\.,[2026](https://arxiv.org/html/2607.23037#bib.bib17)\)applied ChatGPT to speed\-dating transcripts to predict mutual contact exchange as well as subjective outcomes and experiences\. ChatGPT’s correlations with these criteria were modest \(r=\.12r=\.12–\.23\.23\)\. For mutual contact exchange specifically, its performance was comparable to that of transcript\-only human judges \(r=\.12r=\.12vs\.\.13\.13\)\. Building on this work, we test whether a supervised speech predictor adds predictive value beyond transcript\-only LLM predictions for continuous liking and within\-participant partner ranking\.

### 2\.2\.Multimodal Prediction of Interpersonal Outcomes

Interpersonal outcomes have long been studied computationally using verbal and nonverbal features from conversation\. Brief observations of expressive behavior can support accurate interpersonal judgments, and nonverbal communication encompasses cues in the face, voice, body, touch, and interpersonal space\(Ambady and Rosenthal,[1992](https://arxiv.org/html/2607.23037#bib.bib7); Hallet al\.,[2019](https://arxiv.org/html/2607.23037#bib.bib4)\)\. In speed dating, supervised models have used prosodic, dialogue, and lexical features to detect flirtation\(Ranganathet al\.,[2009](https://arxiv.org/html/2607.23037#bib.bib13)\)and position, proximity, and motion features from video to predict contact exchange and physical attraction\(Veenstra and Hung,[2011](https://arxiv.org/html/2607.23037#bib.bib14)\)\. Complementary statistical analyses have linked conversational behavior to perceived connection in speed\-dating encounters\(McFarlandet al\.,[2013](https://arxiv.org/html/2607.23037#bib.bib12)\)and function\-word language style matching to mutual romantic interest\(Irelandet al\.,[2011](https://arxiv.org/html/2607.23037#bib.bib11)\)\. Speakers’ pitch convergence also varies with their perception of the partner’s visual attractiveness and overall likability\(Michalsky and Schoormann,[2017](https://arxiv.org/html/2607.23037#bib.bib2)\)\. On Japanese speed\-dating data, Ishii et al\.\(Ishiiet al\.,[2023](https://arxiv.org/html/2607.23037#bib.bib15)\)predicted post\-interaction liking scores from information available before the interaction, including profiles, facial features, and psychometric measures\. More recently, frozen self\-supervised speech representations, including HuBERT, have transferred effectively to paralinguistic tasks such as emotion recognition\(Hsuet al\.,[2021](https://arxiv.org/html/2607.23037#bib.bib27); Yanget al\.,[2021](https://arxiv.org/html/2607.23037#bib.bib38)\)\.

Santana et al\.\(Santanaet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib19)\)provide the closest architectural precedent for our supervised branches\. Their Speech\-to\-Joy framework trains separate predictors over frozen pretrained text and audio embeddings, using additive attention pooling and modality\-specific regression heads\. The framework then averages the two predictors’ output scores for enjoyment prediction in human–robot dialogue\. We adapt this supervised\-branch architecture, but our main comparison pairs the speech predictor with a transcript\-only LLM prediction rather than a supervised text prediction\.

### 2\.3\.LLM Integration in Multimodal Prediction

Multimodal fusion methods are conventionally partitioned into feature\-level early fusion and score\-level late fusion\. Late fusion is particularly well suited when the predictors are trained under different regimes, since each can be optimized independently\. Recent work has integrated LLMs into multimodal prediction pipelines in several ways\. Some approaches textualize nonverbal modalities in the prompt\(Maet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib20); Hasanet al\.,[2023](https://arxiv.org/html/2607.23037#bib.bib21)\)or use an LLM for post hoc refinement of conventional model outputs\(Singh and Villalba,[2025](https://arxiv.org/html/2607.23037#bib.bib22)\)\. Others use an LLM to integrate modality\-specific predictions in natural language rather than through a weighted score\-level combination\(Demirelet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib24)\)\.

At the task level, Pereira et al\.\(Pereiraet al\.,[2024](https://arxiv.org/html/2607.23037#bib.bib18)\)combined an LLM\-derived enjoyment score with temporal, visual, and acoustic features in supervised models for human–robot conversations, and separately compared text\-only and video\-plus\-audio LLM inputs\. Their supervised target was third\-party enjoyment annotations\. The LLM score entered the supervised model alongside modality descriptors rather than being fused at score level with the output of a separately trained modality\-specific predictor\.

Two prior works more specifically combine score\-level late fusion with LLMs, but neither treats a transcript\-only LLM prediction as an independent branch\. In Amin et al\.\(Aminet al\.,[2023](https://arxiv.org/html/2607.23037#bib.bib23)\), ChatGPT serves as an auxiliary text generator whose output is featurized alongside the raw text, so all fused branches take text input\. In ZS\-Fuse\(Kataria and Hu,[2026](https://arxiv.org/html/2607.23037#bib.bib25)\), whose “zero\-shot large\-scale model plus supervised specialist model” structure parallels ours, the zero\-shot branch is itself audio\-driven rather than transcript\-driven\. A separate direction integrates audio directly into LLMs: audio\-native models such as SALMONN\(Tanget al\.,[2024](https://arxiv.org/html/2607.23037#bib.bib32)\)integrate pre\-trained speech and audio encoders with an LLM and let it reason over audio inputs directly\. We include this direction as an auxiliary comparison using the MLLM baselines Gemini 2\.5 Flash and GPT\-audio\-mini with text\-plus\-audio input\.

Together, these studies leave unresolved whether a separately trained speech predictor improves interpersonal\-attraction prediction beyond a transcript\-only LLM\. We treat a transcript\-only LLM prediction obtained through zero\-shot prompting and a separately trained supervised speech predictor as independent branches, combining their outputs through weighted score\-level late fusion\. We use this conventional fusion method to diagnose the speech predictor’s additional predictive value beyond the transcript\-only prediction\. Audio\-native MLLMs form a separate direct\-input route, which we include only as an auxiliary comparison in the present study\.

## 3\.Corpus and Task Definition

### 3\.1\.Speed\-Dating Corpus

We use the Multi\-Modal Speed Dating corpus\(Ishiiet al\.,[2023](https://arxiv.org/html/2607.23037#bib.bib15)\)\. It was collected in the speed\-dating paradigm, a methodologically validated setting for observing initial romantic attraction under controlled yet ecologically valid conditions\(Finkelet al\.,[2007](https://arxiv.org/html/2607.23037#bib.bib10)\)\. The corpus records face\-to\-face conversations between male\-female pairs involving 147 native Japanese speakers: 75 female and 72 male participants\. Their ages ranged from 19 to 60 years, with a mean of 31\.9 years and a standard deviation of 8\.6 years\. All participants had indicated a strong interest in meeting future partners\.

Participants were assigned to age\-balanced groups of 5 female and 5 male members\. Each group held all 25 possible male\-female pairwise conversations; some participants joined multiple groups\. The corpus is organized into two conversation rounds with nominal durations of 5 minutes \(Session 1\) and 10 minutes \(Session 2\)\. Each session contains 624 pairwise conversations for a total of 1,248\. The same pairs conversed in both rounds, and Session 2 was held after Session 1\. Thus, each Session 2 conversation followed a prior encounter with the same partner\. Each conversation is released with a human\-made transcript and per\-speaker audio recorded via headset microphones\.

### 3\.2\.Liking Scores and Evaluation Conditions

After each conversation, both participants rated their partner using a Japanese translation of the 13\-item Rubin’s Liking Scale\(Rubin,[1970](https://arxiv.org/html/2607.23037#bib.bib9); Ishiiet al\.,[2023](https://arxiv.org/html/2607.23037#bib.bib15)\)\. Each item used a 1–9 Likert scale, and the scale had a Cronbach’sα\\alphaof\.94\.94\. The prediction target is the*liking score*, defined as the mean of the 13 items\. The items span multiple facets of liking, including empathy, adaptability, trust in judgment, likability, perceived similarity, intelligence, and admiration\.

We treat the two rating directions independently\. F2M denotes predictions of liking scores assigned by female participants to their male partners, and M2F denotes the reverse\. In both abbreviations, the direction runs from the rater to the rated partner\. Within each conversation group, we exclude a rater’s conversations if that rater assigned the same liking score to every partner, because correlation is undefined for that subset\. After this exclusion, the Session 1 samples contain 604 F2M pairs from 73 raters and 619 M2F pairs from 72 raters\. The Session 2 samples contain 619 pairs in each direction, from 75 female raters for F2M and 72 male raters for M2F\. Crossing the two conversation rounds with the two rating directions yields the four conditions analyzed throughout\. Mean liking scores ranged from 4\.36 to 4\.94 across the four conditions, with standard deviations near 1\.5\.

### 3\.3\.Task Definition

Let𝒟i=\{\(Xi​j,yi​j\)\}j=1Ni\\mathcal\{D\}\_\{i\}=\\\{\(X\_\{ij\},y\_\{ij\}\)\\\}\_\{j=1\}^\{N\_\{i\}\}denote raterii’s set of conversations across multiple partners, the central unit of analysis\. For each partnerjj,Xi​jX\_\{ij\}denotes the transcript and/or speech information available from the interaction, andyi​j∈\[1,9\]y\_\{ij\}\\in\[1,9\]denotes the liking score that rateriiassigns to partnerjjafter the conversation\. Our task is to develop a predictorffthat mapsXi​jX\_\{ij\}to a scalar scorey^i​j=f​\(Xi​j\)\\hat\{y\}\_\{ij\}=f\(X\_\{ij\}\)\. The predictor should approximate the absolute liking score \(y^i​j≈yi​j\\hat\{y\}\_\{ij\}\\approx y\_\{ij\}\) while preserving the within\-participant ordering of non\-tied partner pairs\. Preserving this ordering means that, for each non\-tied pair within a participant’s partner set, the predicted and observed score differences have the same sign\.

Recovering within\-participant differentiation is our primary inferential target: it captures which partners a participant reported liking more relative to their own baseline, independent of between\-participant differences in response\-scale use\. Absolute score agreement nevertheless remains relevant when predictions are interpreted on the original 1–9 scale, motivating a calibration\-sensitive optimization objective\. The supervised predictors are therefore trained on the concordance correlation coefficient \(CCC\)\(Lin,[1989](https://arxiv.org/html/2607.23037#bib.bib36)\)\. CCC rewards association only when the predicted and observed scores also agree in location and scale\.

## 4\.Models, Fusion, and Evaluation Setup

The primary comparison combines a transcript\-only LLM prediction from the full two\-speaker transcript with a supervised prediction from rater\-side speech\. We use conventional weighted score\-level late fusion to diagnose the speech predictor’s additional value, not as a new architecture\. The supervised branches adapt Santana et al\.’s architecture\(Santanaet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib19)\): frozen representations, additive attention pooling, and modality\-specific regression heads\.

### 4\.1\.Supervised Single\-Modality Predictors

We train separate supervised text and speech predictors rather than a joint encoder\. Each places a small learnable module on a frozen pretrained encoder, so only the downstream aggregation and regression are learned from the speed\-dating corpus\. Both predictors share the same downstream architecture: attention pooling over the rater’s own utterances, followed by a scalar regression head\. Holding the downstream architecture constant helps isolate differences in what the encoders extract from the conversation\. It also makes the two predictors directly compatible inputs to score\-level late fusion\. In what follows, we refer to each supervised predictor by the name of its underlying encoder \(e\.g\., HuBERT denotes the supervised predictor built on frozen HuBERT features, not the encoder alone\)\.

#### Text embeddings

Consecutive same\-speaker transcript segments are merged before utterance\-level encoding\. Each utterance is encoded with Sentence\-T5\-large \(335M parameters\)\(Niet al\.,[2022](https://arxiv.org/html/2607.23037#bib.bib26)\), yielding an L2\-normalized 768\-dimensional sentence embedding\.

#### Speech embeddings

We use pretrained HuBERT\-Large \(317M parameters\)\(Hsuet al\.,[2021](https://arxiv.org/html/2607.23037#bib.bib27)\)as the speech encoder, a self\-supervised speech model pretrained on Libri\-Light\(Kahnet al\.,[2020](https://arxiv.org/html/2607.23037#bib.bib28)\)\. All parameters are frozen\. For each utterance we feed the raw waveform \(16 kHz\) through HuBERT and average\-pool the final hidden states to obtain a 1,024\-dimensional embedding \(L2\-normalized\)\. Utterances are cut by transcript timestamps with silence removed via WebRTC voice activity detection\.

#### Architecture details

The encoder stays frozen throughout training because each cross\-validation training split contains only on the order of10210^\{2\}training conversations, a regime in which fine\-tuning a10810^\{8\}\-parameter model is prone to overfit\. Utterance\-level embeddings are aggregated by additive attention pooling, letting the model emphasize utterances that carry rater\-side signal about liking rather than weighting all utterances equally\. Pooling is restricted to the rater’s own utterances; prior speed\-dating work shows that speakers modulate voice pitch with their own mate preferences, providing precedent for rater\-side acoustic signal\(Pisanskiet al\.,[2018](https://arxiv.org/html/2607.23037#bib.bib3)\)\. The pooled representation passes through a two\-layer head\. A linear layer projects it to hidden dimensionmax⁡\(⌊d/5⌋,16\)\\max\(\\lfloor d/5\\rfloor,16\)\(whereddis the encoder output, 768 for text and 1024 for audio\), followed by ReLU and dropout 0\.2; a final linear layer maps it to a scalar\. Additional supervised baselines—BERT\(Devlinet al\.,[2019](https://arxiv.org/html/2607.23037#bib.bib30)\), the Japanese Linguistic Inquiry and Word Count \(J\-LIWC\)\(Igarashiet al\.,[2022](https://arxiv.org/html/2607.23037#bib.bib29)\), and openSMILE\(Eybenet al\.,[2010](https://arxiv.org/html/2607.23037#bib.bib31)\)—use the same architecture, withddset to each feature’s dimension\.

#### Training settings

All supervised models use CCC loss \(1−CCC¯1\-\\overline\{\\text\{CCC\}\}, whereCCC¯\\overline\{\\text\{CCC\}\}is the mean per\-participant CCC across raters in a batch\)\. We optimize them with Adam \(learning rate10−310^\{\-3\}, no weight decay or learning\-rate scheduling\)\. Batches include up to six raters and all of their conversations\. Training runs for a maximum of 500 epochs with early stopping \(patience 20 on validation CCC\) and random seed 42\. Targets are standardized using the training\-fold mean and standard deviation; predictions are inverse\-transformed and clipped to the valid\[1,9\]\[1,9\]range\.

### 4\.2\.LLM\-Based Prediction

Transcript\-only LLM predictions provide the comparison point for testing the speech predictor’s additional value\. We obtain them using zero\-shot prompting, without task\-specific fine\-tuning or in\-context examples\. Each model receives the complete chronological transcript, including utterances from both the rater and the partner, with speaker labels and timestamps; no rater\-side filtering is applied\. The prompt identifies the rater \(female for F2M, male for M2F\)\. Adapting Santana et al\.’s item\-wise questionnaire prediction strategy\(Santanaet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib19)\)to Rubin’s Liking Scale, the model predicts the rater’s response to each of the scale’s 13 items on a 1–9 integer scale using each API’s structured\-output mechanism\. The mean is the predicted liking score \(∈\[1,9\]\\in\[1,9\]\)\. This input scope differs from the supervised text and speech branches, whose pooling is restricted to the rater’s own utterances\. We use temperature 0 for all configurations reported in the main analysis\. The prompt structure, Japanese and English versions, and placeholder substitutions are documented in Section S2\.4222In section, table, and figure numbers, “S” denotes the Supplementary Appendix\.; the released code contains the exact executable templates, including the full item wording\.

#### Transcript\-only LLMs

We evaluate proprietary LLMs \(Claude Sonnet 4\.6 and GPT\-5\.4\) and an open\-weight model \(Gemma 3 12B\-IT\) in a transcript\-only \(T\) input configuration, where each model receives this complete two\-speaker transcript without audio\. The selected Claude configuration disables extended thinking\.

#### Multimodal LLMs

We also evaluate an alternative route that feeds audio directly to the language model alongside the transcript\. As auxiliary baselines, we use Gemini 2\.5 Flash and GPT\-audio\-mini\. Each model receives the full transcript and the full conversation audio in a mixed\-mono input configuration created from the two headset recordings\. Gemini receives lossless FLAC at 16 kHz, whereas GPT\-audio\-mini receives 128 kbps MP3 passed as base64\. For Gemini, we report matched transcript\-only \(T\) and text\-plus\-audio \(T\+A\) input configurations\. The two prompts are identical in transcript content, system role, and item\-wise scoring instruction, so the within\-model difference isolates the addition of the audio stream under this input configuration\. For GPT\-audio\-mini, we report the T\+A configuration as a direct\-input baseline and compare it with the transcript\-only LLM baselines, but do not make a within\-model audio ablation claim\. Here, transcript\-only LLM and direct\-input MLLM describe the evaluated input configurations, not a model’s underlying multimodal capability\.

### 4\.3\.Representative Selection and Score\-Level Late Fusion

We perform score\-level fusion of predictions from two unimodal representatives, one per input branch\. Score\-level fusion is necessary for Claude because the branch exposes only a scalar score, not an intermediate embedding, making feature\-level concatenation inapplicable\. The supervised unimodal predictors likewise produce per\-conversation scalar predictions\. For the supervised text\+speech pair \(Sentence\-T5 \+ HuBERT\), where feature\-level concatenation is technically feasible, Table[1](https://arxiv.org/html/2607.23037#S5.T1)reports score\-level late and utterance\-level early fusion as reference points\. For the main fusion, we choose one representative per input branch*a priori*, before inspecting any results: Claude for transcript\-only prediction and HuBERT for speech\. Claude permits the common temperature\-0 setting, unlike extended\-thinking mode\. The broader comparison pool additionally includes BERT, openSMILE, and J\-LIWC as supervised baselines, alongside Sentence\-T5 and the evaluated LLM and MLLM variants\. The retrospective complementarity landscape of this pool \(Figure S1\) is consistent with this choice: across all four conditions, HuBERT occupies a favorable region of relatively high utility and low prediction correlation with Claude\.

Given predictionsy^1\\hat\{y\}\_\{1\}andy^2\\hat\{y\}\_\{2\}from the two branch representatives, the fused prediction is

\(1\)y^fusion=w​y^1\+\(1−w\)​y^2,0≤w≤1\.\\hat\{y\}\_\{\\text\{fusion\}\}=w\\,\\hat\{y\}\_\{1\}\+\(1\-w\)\\,\\hat\{y\}\_\{2\},\\quad 0\\leq w\\leq 1\.For each test fold, the weightwwis selected by a 21\-point grid search overw∈\{0,0\.05,…,1\.0\}w\\in\\\{0,0\.05,\\dots,1\.0\\\}to maximize per\-participant CCC over the pooled held\-out predictions of the remaining folds\. The selected weight is then applied to the held\-out test fold without further tuning\. Because participants can appear in multiple conversation groups, the primary weight\-selection pool is not strictly participant\-disjoint; excluding all pairs involving test\-fold participants yields the same qualitative and inferential conclusions \(Section S2\.3\)\.

### 4\.4\.Evaluation Procedure

We evaluate predictions by 25\-fold Leave\-One\-Group\-Out cross\-validation, holding out each of the 25 conversation groups as the test fold\. For the supervised models, all pairs involving participants of the held\-out group are removed from training, so the trained predictor never sees a test\-fold participant\. A separate validation fold, participant\-disjoint from the test fold, is used for early stopping\. Raters with fewer than three partners or near\-constant ratings are additionally excluded from the training set only\.

For each participant, we compute Pearson’srrbetween predicted and true liking scores across their partners, then averagerracross participants\. Per\-participantrris the primary evaluation metric because our central research question concerns whether a model recovers each participant’s relative differentiation among partners\. Its invariance to positive affine transformations factors out participant\-specific location and scale use\(Cronbach,[1955](https://arxiv.org/html/2607.23037#bib.bib6)\)\. Optimization and primary evaluation use different criteria\. CCC is calibration\-sensitive: unlikerr, it penalizes differences in prediction mean and variance\. We therefore use CCC for both supervised training and fusion\-weight selection, discouraging models or weights that preserve association while producing poorly calibrated scores on the original 1–9 scale\. We compute CCC per participant and average it across participants, reporting it as a secondary measure of absolute agreement \(Table S2\)\. We also report per\-participant Top\-1 accuracy and pairwise accuracy \(PW\), the fraction of same\-participant partner pairs whose predicted order matches ground truth\. For Top\-1, all partners tied at the ground\-truth maximum are valid; predicted ties receive fractional credit equal to the proportion belonging to that set\. For PW, pairs with tied ground\-truth liking are excluded and tied predictions are counted as incorrect\. PW depends only on ordering and is therefore invariant to strictly increasing transformations\.

## 5\.Results

### 5\.1\.Individual Models and Fusion Performance

Table[1](https://arxiv.org/html/2607.23037#S5.T1)reportsrrfor four transcript\-only LLMs, two text\-plus\-audio MLLMs, five supervised unimodal models, and three fusion variants: two score\-level late\-fusion variants and one utterance\-level early\-fusion variant\.

Table 1\.Per\-participant Pearson correlation \(rr\) for transcript\-only LLMs, direct\-input text\-plus\-audio MLLMs, supervised unimodal predictors, and fusion variants\. S1/S2 denote Sessions 1/2; F2M denotes female participants rating male partners, and M2F the reverse\.Rows compare transcript\-only LLMs, direct\-input text\-plus\-audio MLLMs, supervised text and speech predictors, supervised fusion, and the primary LLM\-plus\-speech fusion; columns give per\-participant r for the four session\-by\-direction conditions\.

Note\.Values are mean per\-participantr±r\\pmstandard error of the mean \(SE\); undefinedrrvalues are excluded\.Boldmarks the highest point estimate in each column\.†denotesrrundefined for openSMILE in Session 1 because of near\-constant predictions\. L and E denote weighted score\-level late fusion and utterance\-level early fusion, respectively\.

Models\.Claude: Claude Sonnet 4\.6 with extended thinking disabled; Sentence\-T5: Sentence\-T5\-large; HuBERT: HuBERT\-Large\.

#### LLMs and auxiliary MLLM comparisons\.

The best\-performing LLM configuration in each condition is transcript\-only: Claude leads in Session 2 and GPT\-5\.4 leads in Session 1, although the Session 1 differences are smaller than the SE reported for either model\. Under our mixed\-mono input configuration, Gemini’s T\+A input does not consistently improve on its matched transcript\-only input, and GPT\-audio\-mini with T\+A input does not surpass the strongest transcript\-only baselines\.

#### Supervised baselines and fusion variants\.

Claude has higher per\-participantrrthan each of the three supervised text models, Sentence\-T5, BERT, and J\-LIWC, in all four conditions \(Wilcoxon signed\-rank tests; rank\-biserialrrb=\.24r\_\{\\text\{rb\}\}=\.24–\.62\.62over the twelve Claude vs\. supervised\-text comparisons\)\. HuBERT is condition\-dependent: its performance is closer to Claude’s in Session 2 than in Session 1, where it is substantially weaker\. The openSMILE predictor produces near\-constant outputs for a majority of Session 1 raters\. Consequently, per\-participantrris undefined on the common rater set; Table[1](https://arxiv.org/html/2607.23037#S5.T1)marks these results as undefined with†\\dagger\. The score\-level late fusion of Sentence\-T5 and HuBERT is below the Claude–HuBERT fusion in all four conditions, so the Claude prediction is not interchangeable with the Sentence\-T5 prediction under the shared fusion setup\. The utterance\-level early\-fusion variant is lower than the score\-level variant in three conditions; the two are essentially tied in Session 1 F2M\. These results establish Claude as the transcript\-only comparison point and HuBERT as a speech predictor whose performance varies by condition\.

### 5\.2\.Fusion Analysis

#### Condition\-dependent fusion gain\.

Throughout, we define fusion gain relative to Claude asΔ=Fusion−Claude\\Delta=\\text\{Fusion\}\-\\text\{Claude\}, usingΔ​r\\Delta rfor per\-participantrrandΔ\\DeltaPW for pairwise accuracy\. Relative to Claude, the Claude–HuBERT fusion significantly improves pairwise accuracy in all four conditions, with Holm\-correctedp<\.05p<\.05in each \(Table[2](https://arxiv.org/html/2607.23037#S5.T2)\)\. By contrast,Δ​r\\Delta ris largest in Session 2 F2M, smaller in Session 1 F2M, and essentially zero in both M2F conditions \(Table[1](https://arxiv.org/html/2607.23037#S5.T1)\); no pairedΔ​r\\Delta rreaches significance after Holm correction\. The fused model also exceeds HuBERT alone in pairwise accuracy in three of four conditions, with Holm\-correctedp<\.05p<\.05in each\. Session 1 M2F is the exception: HuBERT is the stronger single predictor, and fusion does not improve on HuBERT\. The selected weights are comparable between Claude and HuBERT in Session 2 but concentrate on Claude in Session 1 \(Figure S2\)\.

Replacing Claude with GPT\-5\.4 yields a similar four\-conditionΔ​r\\Delta rpattern \(Table S7\)\. This robustness does not extend to the tested alternative representatives: substituting Sentence\-T5 for Claude or openSMILE for HuBERT lowers fusion per\-participantrrin every condition \(Table S1\)\. Table[3](https://arxiv.org/html/2607.23037#S5.T3)further shows little shared within\-rater explained variance between Claude and HuBERT, with each providing a similar positive increment beyond the other\.

#### Pairwise accuracy by rank distance\.

Aggregate PW shows that fusion improves pairwise ordering over Claude, but not where the gains occur\. Table[2](https://arxiv.org/html/2607.23037#S5.T2)therefore breaks down fusion gains by within\-participant rank distance using Near \(11–22\), Mid \(33–55\), and Far \(6\+6\+\) bins, and reports Top\-1 change separately\. In Session 2 F2M, where therrgain is largest,Δ\\DeltaPW increases with rank distance; this pattern is less clear elsewhere\. Tie\-aware Top\-1 changes are small, decreasing slightly in Session 1 and increasing slightly in Session 2, so aggregate PW gains do not correspond to uniform improvements in identifying the top\-ranked partner\.

Table 2\.Pairwise accuracy \(PW\) and fusion gains over Claude, reported overall and by rank distance, together with changes in Top\-1 accuracy\. S1/S2 denote Sessions 1/2; F2M denotes female participants rating male partners, and M2F the reverse\.Rows give the four session\-by\-direction conditions; columns report aggregate pairwise accuracy for Claude, HuBERT, and fusion, the fusion gain and effect size, rank\-distance\-specific gains, and the change in Top\-1 accuracy\.

Aggregate PWFusion comparison with ClaudeConditionClaudeHuBERTFusionOverall PW gainrrbr\_\{\\text\{rb\}\}Near PW gainMid PW gainFar PW gainTop\-1 changeS1 F2M\.549\.531\.603\+\.054\+\.054\.594∗⁣∗∗\.594^\{\*\*\*\}\+\.044\+\.044\+\.076\\mathbf\{\+\.076\}\+\.074\+\.074−\.025\-\.025S1 M2F\.518\.560\.543\+\.025\+\.025\.623∗∗\.623^\{\*\*\}\+\.026\+\.026\+\.039\\mathbf\{\+\.039\}\+\.031\+\.031−\.007\-\.007S2 F2M\.579\.597\.628\+\.049\+\.049\.299∗\.299^\{\*\}\+\.031\+\.031\+\.055\+\.055\+\.091\\mathbf\{\+\.091\}\+\.013\+\.013S2 M2F\.545\.564\.594\+\.049\+\.049\.374∗\.374^\{\*\}\+\.044\+\.044\+\.042\+\.042\+\.050\\mathbf\{\+\.050\}\+\.028\+\.028
Note\.Claude denotes Claude Sonnet 4\.6 with extended thinking disabled\. Aggregate PW \(left\) is macro\-averaged across participants; rank\-distance bins \(Near==1–2, Mid==3–5, Far==6\+\) are micro\-averaged over all within\-participant pairs \(rank distance is a pair\-level attribute\), so bins need not average to the aggregate\. PW gain and Top\-1 change are computed as Fusion−\-Claude\.rrbr\_\{\\text\{rb\}\}is the rank\-biserial effect size for the Fusion vs\. Claude Wilcoxon signed\-rank test\.Boldmarks the bin with the largest PW gain per condition\.p∗<\.05\{\}^\{\*\}p<\.05,p∗∗<\.01\{\}^\{\*\*\}p<\.01,p∗⁣∗∗<\.001\{\}^\{\*\*\*\}p<\.001\(Holm\-corrected,m=4m=4\)\.

Table 3\.Mutual incremental within\-rater variance between Claude and HuBERT across the four conditions\.Rows give Session 1 and Session 2 for female\-to\-male and male\-to\-female ratings; columns report the incremental within\-rater explained variance of HuBERT beyond Claude, of Claude beyond HuBERT, and their shared explained variance\.

Note\.Claude denotes Claude Sonnet 4\.6 with extended thinking disabled\. “HuBERT beyond Claude” is the increase inR2R^\{2\}obtained by adding the HuBERT prediction to the Claude prediction; “Claude beyond HuBERT” is defined symmetrically\. SharedR2R^\{2\}is the overlap in explained variance\. Values are within\-rater, in\-sample averages \(hence the increments are≥0\\geq 0\); brackets show 95% CIs based on 2,000 participant bootstraps\.

## 6\.Discussion

### 6\.1\.When Does Speech Complement the Transcript\-Only LLM?

Speech\-derived complementarity is conditional across evaluation metrics, conditions, and participants\. Relative to Claude, fusion significantly improves PW in all four conditions \(Table[2](https://arxiv.org/html/2607.23037#S5.T2)\), indicating more accurate ordering of partner pairs\. By contrast, no pairedΔ​r\\Delta rreaches significance after correction, so the results do not show a uniform improvement in participants’ overall differentiation among partners\. Absolute per\-participantrralso remains modest for all models; the gains indicate complementarity, not deployment\-ready accuracy\.

At the condition level, HuBERT is closer to Claude in Session 2 than in Session 1, and the selected fusion weights are correspondingly more balanced in Session 2 \(Figure S2\)\. Stronger HuBERT performance does not, however, map uniformly onto largerrrgains: the largest numerical gain occurs in Session 2 F2M, but the two M2F conditions show essentially no change\. HuBERT’s stronger Session 2 performance may reflect both prior\-encounter exposure and longer within\-encounter observation, which this corpus cannot disentangle\. Pitch convergence in speed dating varies with perceived partner attractiveness\(Michalsky and Schoormann,[2017](https://arxiv.org/html/2607.23037#bib.bib2)\), and longer interaction may permit richer partner representations\(Altman and Taylor,[1973](https://arxiv.org/html/2607.23037#bib.bib1)\)\. The observed condition pattern therefore locates, but does not explain, where complementarity emerges\.

Figure[2](https://arxiv.org/html/2607.23037#S6.F2)shows the participant\-level relationship between HuBERT’srrand fusion gainΔ​r\\Delta r\. Across participants, Claude’srrand HuBERT’srrare weakly associated \(\|ρ\|<\.30\|\\rho\|<\.30in all conditions\), whereas HuBERT’srris strongly associated withΔ​r\\Delta rin every condition \(ρ≥\.70\\rho\\geq\.70, allp<\.0001p<\.0001\)\. BecauseΔ​r\\Delta ris computed from a fused prediction that already contains the HuBERT score, the correlation between HuBERT’srrandΔ​r\\Delta ris partly structurally expected and is not an independent test of a participant\-level mechanism\. The correlation instead provides a descriptive localization of the observed gains: participants for whom HuBERT is weak also show little fusion benefit\. Prior multimodal affect research has emphasized aggregate fusion gains over unimodal baselines\(D’Mello and Kory,[2015](https://arxiv.org/html/2607.23037#bib.bib5); Santanaet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib19)\), leaving such participant\-level variation largely unexplored\.

A within\-rater variance decomposition further characterizes the overlap between the Claude and HuBERT predictions \(Table[3](https://arxiv.org/html/2607.23037#S5.T3)\)\. The two predictions are weakly correlated within raters \(meanrrbetween−\.08\-\.08and\+\.13\+\.13\), and their sharedR2R^\{2\}is at most\.04\.04in every condition\. The incremental associations are comparable in both directions, indicating mutual rather than one\-sided association\. The low sharedR2R^\{2\}and comparable bidirectional increments are consistent with limited overlap between the two predictions, but the decomposition does not by itself distinguish distinct information from variance reduction under noisy prediction\.

![Refer to caption](https://arxiv.org/html/2607.23037v1/x2.png)Figure 2\.Across all four conditions, participants with higher HuBERT per\-participant Pearsonrrshow larger Claude–HuBERT fusion gainΔ​r\\Delta rover Claude, although gain magnitude varies by condition\. Panels show Sessions 1/2 and female\-to\-male \(F2M\)/male\-to\-female \(M2F\) ratings\. Points are participants; solid lines and bands show OLS fits and 95% CIs, and dashed lines mark zero\. Because HuBERT enters the fused score, this association is descriptive rather than an independent mechanism\.Four scatter panels show Session 1 and Session 2 for female\-to\-male and male\-to\-female ratings\. Each panel plots participants with HuBERT’s per\-participant Pearson correlation on the horizontal axis and the gain from weighted score\-level late fusion of Claude Sonnet 4\.6 with extended thinking disabled and HuBERT over Claude alone on the vertical axis\. Dashed horizontal and vertical reference lines mark zero\. Each panel includes an upward\-sloping ordinary least squares line and its shaded 95 percent confidence band\. Panel titles report Spearman rank correlations of \+0\.76 and \+0\.70 in Session 1 for female\-to\-male and male\-to\-female ratings, respectively, and \+0\.80 and \+0\.72 in Session 2\. All p\-values are below \.0001, and the panels contain 72 to 75 participants\.
### 6\.2\.Late Fusion vs\. Direct Multimodal Input

The auxiliary MLLM comparison examines a different route for incorporating speech\. Under the tested mixed\-mono configuration, Gemini’s text\-plus\-audio input does not consistently improve on its matched transcript\-only input, and GPT\-audio\-mini does not surpass the strongest transcript\-only baselines\. By contrast, combining independently obtained Claude and HuBERT predictions yields a pairwise\-ordering benefit in the present configuration\. The observed contrast between direct multimodal input and score\-level fusion does not establish a general advantage of score\-level late fusion over direct multimodal inference; the comparison is specific to the evaluated models, prompts, and audio processing\.

Score\-level fusion instead provides a modular way to evaluate the speech predictor’s additional predictive value relative to a transcript\-only LLM\. The selected weights place less weight on HuBERT in Session 1, where it is less accurate, and relatively more weight on it in Session 2\. As transcript\-only LLMs evolve, the magnitude of speech\-derived complementarity remains an empirical question across models and conditions\.

### 6\.3\.Limitations and Future Work

#### Evaluation target and partner\-level consensus

Per\-participantrrevaluates within\-participant differentiation among partners but does not distinguish rater\-specific partner preference from partner\-level consensus shared across raters\. To contextualize this limitation, we construct a descriptive partner\-level consensus reference using the mean score each partner received from other raters\. The reference reaches per\-participantrrof\.36\.36–\.48\.48, above the Claude–HuBERT fusion in all four conditions \(Section S1\.6\)\. These ground\-truth ratings are unavailable at inference, so the reference provides context rather than a deployable baseline\.

#### Mechanism of speech\-derived complementarity

Figure[2](https://arxiv.org/html/2607.23037#S6.F2)shows that fusion gains are concentrated among participants for whom HuBERT has higherrr, but this association is retrospective: it requires all conversations from a participant and cannot predict fusion benefit in advance\. Table[3](https://arxiv.org/html/2607.23037#S5.T3)characterizes the mutual incremental associations between Claude and HuBERT\. Replacing HuBERT with the hand\-crafted openSMILE representation does not improve on Claude in any condition \(Table S1\)\. These analyses do not identify which acoustic information HuBERT captures\. HuBERT represents sequential speech structure\(Hsuet al\.,[2021](https://arxiv.org/html/2607.23037#bib.bib27)\)and transfers well to paralinguistic tasks\(Yanget al\.,[2021](https://arxiv.org/html/2607.23037#bib.bib38)\)\. Which cues contribute—laughter, pauses, backchannels, or turn\-taking—remains open\. Future work should test whether participant\-adaptive or hierarchical models can prospectively estimate speech\-branch reliability\.

#### Direction asymmetry and conversation round

NumericalΔ​r\\Delta rdiffers across rating directions and is largest in Session 2 F2M\. Because Session 2 combines prior\-encounter exposure with a longer nominal duration, this corpus cannot separate their contributions\. Future studies should vary these factors independently\. Modeling actor and partner effects, interpersonal synchrony, and turn\-level dynamics such as prosodic adaptation may also help explain the observed condition dependence\.

#### Corpus, language, and inclusivity

The evidence comes from a single Japanese speed\-dating corpus with 147 participants, restricted to opposite\-sex pairs\. Whether the condition\-dependent complementarity we observe generalizes across languages and pair compositions or to other short, dyadic, first\-encounter interactions with an explicit evaluative goal remains an open empirical question\.

#### Model and modality coverage

The main analysis focuses on text and speech, although video cues can also predict interpersonal outcomes in speed dating\(Veenstra and Hung,[2011](https://arxiv.org/html/2607.23037#bib.bib14)\)\. The direct\-input MLLM comparison covers two models under one prompting and audio\-processing pipeline; audio\-LLM capabilities are evolving rapidly\(Sakshiet al\.,[2025](https://arxiv.org/html/2607.23037#bib.bib33)\)\. We reuse the transcript\-only prompt, mix the headset recordings to mono, and supply GPT\-audio\-mini with compressed MP3 audio\. The supervised branches likewise use HuBERT and Sentence\-T5 rather than exhaustively comparing alternatives such as wav2vec 2\.0\(Baevskiet al\.,[2020](https://arxiv.org/html/2607.23037#bib.bib34)\), WavLM\(Chenet al\.,[2022](https://arxiv.org/html/2607.23037#bib.bib35)\), or Japanese and multilingual encoders\. These choices bound the comparison: the experiments neither establish the general performance of direct\-input MLLMs nor determine which speech encoder best captures Japanese paralinguistic cues\. Future work should evaluate video\-aware extensions, audio\-specific prompts, speaker\-separated audio, and alternative encoders\.

## 7\.Conclusion

In Japanese speed dating, we examined whether a supervised speech predictor complements a transcript\-only LLM in predicting how much participants reported liking each partner\. The results show that speech can complement transcript\-only LLM prediction, but not as a uniform multimodal gain\. Across both conversation rounds and rating directions, fusing speech and transcript\-only LLM predictions significantly improves pairwise ranking accuracy over the transcript\-only LLM alone\. By contrast, gains in per\-participant Pearsonrrvary across these settings, and none is significant after correction\. Retrospectively, theserrgains are larger for participants whose liking is more accurately predicted by the speech predictor\.

Speech can therefore retain predictive value even when an LLM predicts interpersonal attraction from conversation transcripts\. For multimodal social prediction, the relevant question is not simply whether speech helps, but how its complementarity varies across evaluation metrics, conversation settings, and participants\.

## Safe and Responsible Innovation Statement

Inferring interpersonal attraction from conversational data risks nonconsensual profiling \(e\.g\., dating\-app scoring or workplace surveillance\) and overreliance in high\-stakes matchmaking\. The speed\-dating corpus was collected under institutional ethics approval and written informed consent, with access restricted to contract\-based research\. The main analysis uses transcripts and audio; supplementary checks also use overhead video\. These data remain identifying and sensitive\. Predictors may reflect pretraining biases, and findings apply only to Japanese opposite\-sex speed dating\. Responsible deployment requires explicit consent, bias auditing, human oversight, and validation across broader gender identities and cultures\.333The authors used Claude \(Anthropic\) for drafting text, verifying numerical results against raw outputs, reviewing code, and suggesting the analysis plan\. All study decisions and manuscript content remain the authors’ responsibility\.

###### Acknowledgements\.

This work was partially supported by JSPS KAKENHI Grant Number 26K03016, JST CREST Grant Number JPMJCR2563, JST CRONOS Grant Number JPMJCS24K7, and AMED under Grant Number JP256f0137001\.

## References

- I\. Altman and D\. A\. Taylor \(1973\)Social penetration: the development of interpersonal relationships\.Holt, Rinehart and Winston,New York\.External Links:ISBN 9780030766350Cited by:[§6\.1](https://arxiv.org/html/2607.23037#S6.SS1.p2.1)\.
- N\. Ambady and R\. Rosenthal \(1992\)Thin slices of expressive behavior as predictors of interpersonal consequences: a meta\-analysis\.Psychological Bulletin111\(2\),pp\. 256–274\.External Links:[Document](https://dx.doi.org/10.1037/0033-2909.111.2.256)Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1)\.
- M\. M\. Amin, E\. Cambria, and B\. W\. Schuller \(2023\)Can ChatGPT’s responses boost traditional natural language processing?\.IEEE Intelligent Systems38\(5\),pp\. 5–11\.External Links:[Document](https://dx.doi.org/10.1109/MIS.2023.3305861)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p3.1)\.
- A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli \(2020\)wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InProc\. Conference on Neural Information Processing Systems \(NeurIPS\),H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 12449–12460\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf)Cited by:[§6\.3](https://arxiv.org/html/2607.23037#S6.SS3.SSS0.Px5.p1.1)\.
- E\. Berscheid and H\. T\. Reis \(1998\)Attraction and close relationships\.InThe Handbook of Social Psychology,D\. T\. Gilbert, S\. T\. Fiske, and G\. Lindzey \(Eds\.\),Vol\.2,pp\. 193–281\.Cited by:[§1](https://arxiv.org/html/2607.23037#S1.p1.1)\.
- S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao, J\. Wu, L\. Zhou, S\. Ren, Y\. Qian, Y\. Qian, J\. Wu, M\. Zeng, X\. Yu, and F\. Wei \(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing16\(6\),pp\. 1505–1518\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2022.3188113)Cited by:[§6\.3](https://arxiv.org/html/2607.23037#S6.SS3.SSS0.Px5.p1.1)\.
- L\. J\. Cronbach \(1955\)Processes affecting scores on “understanding of others” and “assumed similarity”\.Psychological Bulletin52\(3\),pp\. 177–193\.External Links:[Document](https://dx.doi.org/10.1037/h0044919)Cited by:[§4\.4](https://arxiv.org/html/2607.23037#S4.SS4.p2.4)\.
- S\. K\. D’Mello and J\. Kory \(2015\)A review and meta\-analysis of multimodal affect detection systems\.ACM Computing Surveys47\(3\),pp\. 43:1–43:36\.External Links:[Document](https://dx.doi.org/10.1145/2682899)Cited by:[§6\.1](https://arxiv.org/html/2607.23037#S6.SS1.p3.12)\.
- I\. Demirel, K\. Thakkar, B\. Elizalde, M\. Espi Marques, A\. Sarathy, Y\. Bai, U\. Srinivas, J\. Xu, S\. Ren, and J\. Narain \(2025\)Using LLMs for late multimodal sensor fusion for activity recognition\.External Links:2509\.10729,[Link](https://arxiv.org/abs/2509.10729)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProc\. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px3.p1.5)\.
- F\. Eyben, M\. Wöllmer, and B\. Schuller \(2010\)openSMILE: the munich versatile and fast open\-source audio feature extractor\.InProc\. ACM International Conference on Multimedia \(ACM MM\),MM ’10,New York, NY, USA,pp\. 1459–1462\.External Links:ISBN 9781605589336,[Link](https://doi.org/10.1145/1873951.1874246),[Document](https://dx.doi.org/10.1145/1873951.1874246)Cited by:[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px3.p1.5)\.
- E\. J\. Finkel, P\. W\. Eastwick, and J\. Matthews \(2007\)Speed\-dating as an invaluable tool for studying romantic attraction: a methodological primer\.Personal Relationships14\(1\),pp\. 149–166\.External Links:[Document](https://dx.doi.org/10.1111/j.1475-6811.2006.00146.x)Cited by:[§3\.1](https://arxiv.org/html/2607.23037#S3.SS1.p1.1)\.
- J\. A\. Hall, T\. G\. Horgan, and N\. A\. Murphy \(2019\)Nonverbal communication\.Annual Review of Psychology70\(1\),pp\. 271–294\.External Links:[Document](https://dx.doi.org/10.1146/annurev-psych-010418-103145)Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1)\.
- M\. K\. Hasan, M\. S\. Islam, S\. Lee, W\. Rahman, I\. Naim, M\. I\. Khan, and E\. Hoque \(2023\)TextMI: textualize multimodal information for integrating non\-verbal cues in pre\-trained language models\.External Links:2303\.15430,[Link](https://arxiv.org/abs/2303.15430)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p1.1)\.
- W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed \(2021\)HuBERT: self\-supervised speech representation learning by masked prediction of hidden units\.IEEE/ACM Transactions on Audio, Speech, and Language Processing29,pp\. 3451–3460\.External Links:[Document](https://dx.doi.org/10.1109/TASLP.2021.3122291)Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px2.p1.1),[§6\.3](https://arxiv.org/html/2607.23037#S6.SS3.SSS0.Px2.p1.1)\.
- T\. Igarashi, S\. Okuda, and K\. Sasahara \(2022\)Development of the japanese version of the linguistic inquiry and word count dictionary 2015\.Frontiers in Psychology13,pp\. 841534\.External Links:[Document](https://dx.doi.org/10.3389/fpsyg.2022.841534)Cited by:[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px3.p1.5)\.
- M\. E\. Ireland, R\. B\. Slatcher, P\. W\. Eastwick, L\. E\. Scissors, E\. J\. Finkel, and J\. W\. Pennebaker \(2011\)Language style matching predicts relationship initiation and stability\.Psychological Science22\(1\),pp\. 39–44\.External Links:[Document](https://dx.doi.org/10.1177/0956797610392928)Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1)\.
- R\. Ishii, F\. Nihei, Y\. Ishii, A\. Otsuka, K\. Matsuo, N\. Nomoto, A\. Fukayama, and T\. Nakamura \(2023\)Prediction of love\-like scores after speed dating based on pre\-obtainable personal characteristic information\.InHuman\-Computer Interaction – INTERACT 2023,J\. Abdelnour Nocera, M\. Kristín Lárusdóttir, H\. Petrie, A\. Piccinno, and M\. Winckler \(Eds\.\),Lecture Notes in Computer Science, Vol\.14145,Cham,pp\. 551–556\.External Links:ISBN 978\-3\-031\-42293\-5,[Document](https://dx.doi.org/10.1007/978-3-031-42293-5%5F71)Cited by:[§1](https://arxiv.org/html/2607.23037#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2607.23037#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.23037#S3.SS2.p1.2)\.
- J\. Kahn, M\. Rivière, W\. Zheng, E\. Kharitonov, Q\. Xu, P\.E\. Mazaré, J\. Karadayi, V\. Liptchinsky, R\. Collobert, C\. Fuegen, T\. Likhomanenko, G\. Synnaeve, A\. Joulin, A\. Mohamed, and E\. Dupoux \(2020\)Libri\-light: a benchmark for ASR with limited or no supervision\.InProc\. IEEE International Conference on Acoustics, Speech & Signal Processing \(ICASSP\),pp\. 7669–7673\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP40776.2020.9052942)Cited by:[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px2.p1.1)\.
- S\. Kataria and X\. Hu \(2026\)Prompt amplification and zero\-shot late fusion in audio\-language models for speech emotion recognition\.External Links:2603\.23057,[Link](https://arxiv.org/abs/2603.23057)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p3.1)\.
- A\. Kumar, N\. Poungpeth, D\. Yang, E\. Farrell, B\. L\. Lambert, and M\. Groh \(2026\)When large language models are reliable for judging empathic communication\.Nature Machine Intelligence8,pp\. 173–185\.External Links:[Document](https://dx.doi.org/10.1038/s42256-025-01169-6)Cited by:[§2\.1](https://arxiv.org/html/2607.23037#S2.SS1.p1.1)\.
- L\. I\. Lin \(1989\)A concordance correlation coefficient to evaluate reproducibility\.Biometrics45\(1\),pp\. 255–268\.External Links:ISSN 0006341X, 15410420,[Document](https://dx.doi.org/10.2307/2532051),[Link](https://www.jstor.org/stable/2532051)Cited by:[§3\.3](https://arxiv.org/html/2607.23037#S3.SS3.p2.1)\.
- C\. Ma, K\. H\. Joo, A\. K\. Vail, S\. Bhattacharya, Á\. Fernández García, K\. Baker\-Matsuoka, S\. Mathew, L\. L\. Holt, and F\. De La Torre \(2025\)Multimodal fusion with LLMs for engagement prediction in natural conversation\.InCompanion Proc\. ACM International Conference on Multimodal Interaction \(ICMI\),ICMI Companion ’25,New York, NY, USA,pp\. 244–259\.External Links:ISBN 9798400720765,[Document](https://dx.doi.org/10.1145/3747327.3764904)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p1.1)\.
- S\. C\. Matz, H\. Peters, M\. Cerf, E\. Grunenberg, P\. W\. Eastwick, M\. Back, and E\. J\. Finkel \(2026\)Large language models can detect verbal indicators of romantic attraction\.Scientific Reports16\(1\),pp\. 21441\.External Links:[Document](https://dx.doi.org/10.1038/s41598-026-52308-x)Cited by:[§1](https://arxiv.org/html/2607.23037#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.23037#S2.SS1.p2.4)\.
- D\. A\. McFarland, D\. Jurafsky, and C\. Rawlings \(2013\)Making the connection: social bonding in courtship situations\.American Journal of Sociology118\(6\),pp\. 1596–1649\.External Links:[Document](https://dx.doi.org/10.1086/670240)Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1)\.
- J\. Michalsky and H\. Schoormann \(2017\)Pitch convergence as an effect of perceived attractiveness and likability\.InProc\. Annual Conference of the International Speech Communication Association \(INTERSPEECH\),pp\. 2253–2256\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2017-1520)Cited by:[§1](https://arxiv.org/html/2607.23037#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1),[§6\.1](https://arxiv.org/html/2607.23037#S6.SS1.p2.1)\.
- J\. Ni, G\. Hernandez Abrego, N\. Constant, J\. Ma, K\. Hall, D\. Cer, and Y\. Yang \(2022\)Sentence\-T5: scalable sentence encoders from pre\-trained text\-to\-text models\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 1864–1874\.External Links:[Link](https://aclanthology.org/2022.findings-acl.146/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.146)Cited by:[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Pereira, L\. Marcinek, J\. Miniota, S\. Thunberg, E\. Lagerstedt, J\. Gustafson, G\. Skantze, and B\. Irfan \(2024\)Multimodal user enjoyment detection in human\-robot conversation: the power of large language models\.InProc\. ACM International Conference on Multimodal Interaction \(ICMI\),ICMI ’24,New York, NY, USA,pp\. 469–478\.External Links:ISBN 9798400704628,[Document](https://dx.doi.org/10.1145/3678957.3685729)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p2.1)\.
- K\. Pisanski, A\. Oleszkiewicz, J\. Plachetka, M\. Gmiterek, and D\. Reby \(2018\)Voice pitch modulation in human mate choice\.Proceedings of the Royal Society B: Biological Sciences285\(1893\),pp\. 20181634\.External Links:[Document](https://dx.doi.org/10.1098/rspb.2018.1634)Cited by:[§4\.1](https://arxiv.org/html/2607.23037#S4.SS1.SSS0.Px3.p1.5)\.
- R\. Ranganath, D\. Jurafsky, and D\. McFarland \(2009\)It’s not you, it’s me: detecting flirting and its misperception in speed\-dates\.InProc\. Conference on Empirical Methods in Natural Language Processing \(EMNLP\),P\. Koehn and R\. Mihalcea \(Eds\.\),Singapore,pp\. 334–342\.External Links:[Link](https://aclanthology.org/D09-1035/)Cited by:[§1](https://arxiv.org/html/2607.23037#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1)\.
- Z\. Rubin \(1970\)Measurement of romantic love\.Journal of Personality and Social Psychology16\(2\),pp\. 265–273\.External Links:[Document](https://dx.doi.org/10.1037/h0029841)Cited by:[§3\.2](https://arxiv.org/html/2607.23037#S3.SS2.p1.2)\.
- Sakshi, U\. Tyagi, S\. Kumar, A\. Seth, R\. Selvakumar, O\. Nieto, R\. Duraiswami, S\. Ghosh, and D\. Manocha \(2025\)MMAU: a massive multi\-task audio understanding and reasoning benchmark\.InProc\. International Conference on Learning Representations \(ICLR\),External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/d36f208919582785db965fe648b9fe59-Abstract-Conference.html)Cited by:[§6\.3](https://arxiv.org/html/2607.23037#S6.SS3.SSS0.Px5.p1.1)\.
- R\. Santana, B\. Irfan, E\. Lagerstedt, G\. Skantze, and A\. Pereira \(2025\)Speech\-to\-joy: self\-supervised features for enjoyment prediction in human\-robot conversation\.InProc\. ACM International Conference on Multimodal Interaction \(ICMI\),ICMI ’25,New York, NY, USA,pp\. 238–248\.External Links:ISBN 9798400714993,[Document](https://dx.doi.org/10.1145/3716553.3750747)Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p2.1),[§4\.2](https://arxiv.org/html/2607.23037#S4.SS2.p1.1),[§4](https://arxiv.org/html/2607.23037#S4.p1.1),[§6\.1](https://arxiv.org/html/2607.23037#S6.SS1.p3.12)\.
- P\. Singh and J\. Villalba \(2025\)EmoJudge: LLM based post\-hoc refinement for multimodal speech emotion recognition\.InProc\. Annual Conference of the International Speech Communication Association \(INTERSPEECH\),pp\. 4703–4707\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2025-1141)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p1.1)\.
- C\. Tang, W\. Yu, G\. Sun, X\. Chen, T\. Tan, W\. Li, L\. Lu, Z\. Ma, and C\. Zhang \(2024\)SALMONN: towards generic hearing abilities for large language models\.InProc\. International Conference on Learning Representations \(ICLR\),External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/476ab8f369e489c04187ba84f68cfa68-Abstract-Conference.html)Cited by:[§2\.3](https://arxiv.org/html/2607.23037#S2.SS3.p3.1)\.
- A\. Veenstra and H\. Hung \(2011\)Do they like me? using video cues to predict desires during speed\-dates\.InProc\. IEEE International Conference on Computer Vision Workshops \(ICCVW\),pp\. 838–845\.External Links:[Document](https://dx.doi.org/10.1109/ICCVW.2011.6130339)Cited by:[§1](https://arxiv.org/html/2607.23037#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1),[§6\.3](https://arxiv.org/html/2607.23037#S6.SS3.SSS0.Px5.p1.1)\.
- S\. Yang, P\. Chi, Y\. Chuang, C\. J\. Lai, K\. Lakhotia, Y\. Y\. Lin, A\. T\. Liu, J\. Shi, X\. Chang, G\. Lin, T\. Huang, W\. Tseng, K\. Lee, D\. Liu, Z\. Huang, S\. Dong, S\. Li, S\. Watanabe, A\. Mohamed, and H\. Lee \(2021\)SUPERB: Speech Processing Universal PERformance Benchmark\.InProc\. Annual Conference of the International Speech Communication Association \(INTERSPEECH\),pp\. 1194–1198\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1775),ISSN 2958\-1796Cited by:[§2\.2](https://arxiv.org/html/2607.23037#S2.SS2.p1.1),[§6\.3](https://arxiv.org/html/2607.23037#S6.SS3.SSS0.Px2.p1.1)\.