基于双向音频-文本交叉注意力的多模态矛盾与犹豫状态识别

arXiv cs.CL 论文
multimodal audio-text cross-attention speech-recognition affective-computing bert wav2vec2

摘要

该论文提出一种基于双向音频-文本交叉注意力的多模态框架，用于视频级别的矛盾与犹豫状态识别，在BAH数据集上取得77.95%的A/H F1分数。

arXiv:2610.26868v1 Announce Type: new Abstract: Ambivalence and hesitancy (A/H) are subtle behavioural states that can be expressed through both what a person says and how it is spoken. This paper presents an audio--text framework for video-level A/H recognition. Full-video transcripts and speech waveforms are encoded using pre-trained multilingual BERT and wav2vec~2.0, respectively, and projected into a shared representation space. Two stacked bidirectional cross-attention blocks allow linguistic tokens to attend to acoustic frames and acoustic frames to attend to linguistic context without requiring explicit temporal alignment. The resulting sequences are aggregated through masked mean pooling and combined using concatenation, element-wise products, and absolute differences for binary classification. We evaluate the model on the subject-independent split of the Behavioural Ambivalence/Hesitancy (BAH) dataset. The proposed system achieves 72.19\% accuracy, 75.00\% precision, 81.13\% recall, 77.95\% A/H F1-score, and 70.16\% Macro-F1 on the test set. These results demonstrate the potential of jointly modelling linguistic and acoustic information for automatic A/H recognition.code at:https://github.com/DorriG/ECCV.
查看原文
查看缓存全文

缓存时间: 2026/10/21 06:58 

# Multimodal Ambivalence and Hesitancy Recognition with Bidirectional Audio--Text Cross-Attention
Source: [https://arxiv.org/abs/2610.26868](https://arxiv.org/abs/2610.26868)
[View PDF](https://arxiv.org/pdf/2610.26868)

> Abstract:Ambivalence and hesitancy \(A/H\) are subtle behavioural states that can be expressed through both what a person says and how it is spoken\. This paper presents an audio\-\-text framework for video\-level A/H recognition\. Full\-video transcripts and speech waveforms are encoded using pre\-trained multilingual BERT and wav2vec~2\.0, respectively, and projected into a shared representation space\. Two stacked bidirectional cross\-attention blocks allow linguistic tokens to attend to acoustic frames and acoustic frames to attend to linguistic context without requiring explicit temporal alignment\. The resulting sequences are aggregated through masked mean pooling and combined using concatenation, element\-wise products, and absolute differences for binary classification\. We evaluate the model on the subject\-independent split of the Behavioural Ambivalence/Hesitancy \(BAH\) dataset\. The proposed system achieves 72\.19\\% accuracy, 75\.00\\% precision, 81\.13\\% recall, 77\.95\\% A/H F1\-score, and 70\.16\\% Macro\-F1 on the test set\. These results demonstrate the potential of jointly modelling linguistic and acoustic information for automatic A/H[this http URL](http://recognition.code/)at:[this https URL](https://github.com/DorriG/ECCV)\.

## Submission history

From: Jincheng L \[[view email](https://arxiv.org/show-email/20c53bda/2610.26868)\] **\[v1\]**Thu, 16 Jul 2026 13:48:35 UTC \(364 KB\)

相似文章

面向任务的认知障碍检测多模态学习

arXiv cs.LG

该论文提出了一种接触式多模态筛查框架，利用与 Amazon Alexa 交互过程中采集的视频、音频和文本信号来检测轻度认知障碍（MCI），实验表明多模态融合模型在 Command-Reading 和 Question-Answering 任务上均达到 0.76 的准确率，优于单一模态模型。