Multimodal Ambivalence and Hesitancy Recognition with Bidirectional Audio--Text Cross-Attention
Summary
该论文提出一种基于双向音频-文本交叉注意力的多模态框架，用于视频级别的矛盾与犹豫状态识别，在BAH数据集上取得77.95%的A/H F1分数。
View Cached Full Text
Cached at: 10/21/26, 06:58 AM
# Multimodal Ambivalence and Hesitancy Recognition with Bidirectional Audio--Text Cross-Attention Source: [https://arxiv.org/abs/2610.26868](https://arxiv.org/abs/2610.26868) [View PDF](https://arxiv.org/pdf/2610.26868) > Abstract:Ambivalence and hesitancy \(A/H\) are subtle behavioural states that can be expressed through both what a person says and how it is spoken\. This paper presents an audio\-\-text framework for video\-level A/H recognition\. Full\-video transcripts and speech waveforms are encoded using pre\-trained multilingual BERT and wav2vec~2\.0, respectively, and projected into a shared representation space\. Two stacked bidirectional cross\-attention blocks allow linguistic tokens to attend to acoustic frames and acoustic frames to attend to linguistic context without requiring explicit temporal alignment\. The resulting sequences are aggregated through masked mean pooling and combined using concatenation, element\-wise products, and absolute differences for binary classification\. We evaluate the model on the subject\-independent split of the Behavioural Ambivalence/Hesitancy \(BAH\) dataset\. The proposed system achieves 72\.19\\% accuracy, 75\.00\\% precision, 81\.13\\% recall, 77\.95\\% A/H F1\-score, and 70\.16\\% Macro\-F1 on the test set\. These results demonstrate the potential of jointly modelling linguistic and acoustic information for automatic A/H[this http URL](http://recognition.code/)at:[this https URL](https://github.com/DorriG/ECCV)\. ## Submission history From: Jincheng L \[[view email](https://arxiv.org/show-email/20c53bda/2610.26868)\] **\[v1\]**Thu, 16 Jul 2026 13:48:35 UTC \(364 KB\)
Similar Articles
Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition
The paper introduces the Modality Discrepancy Transformer (MDT), a novel multimodal fusion framework that enhances cross-modal discrepancy modeling for recognizing ambivalence and hesitancy in clinical videos, outperforming baselines on the BAH dataset.
SZED at ArGuard Shared Tasks: Multi-Task Label-Attention and Cross-Modal Fusion for Arabic Hateful Meme Detection
SZED 在 ArGuard 2026 Track A 提交的方案，融合 MARBERTv2 与 CLIP ViT-L/14 嵌入做二分类，并用多任务标签注意力与级联校准处理细粒度多标签标注，二分类 macro-F1 达 0.7597。论文还报告了跨模态注意力与图文不一致特征未带来提升的负结果。
Task-Centric Multimodal Learning for Cognitive Impairment Detection
该论文提出了一种接触式多模态筛查框架，利用与 Amazon Alexa 交互过程中采集的视频、音频和文本信号来检测轻度认知障碍（MCI），实验表明多模态融合模型在 Command-Reading 和 Question-Answering 任务上均达到 0.76 的准确率，优于单一模态模型。
HierBias: Context-Conditioned Hierarchical Media Bias Detection with Multi-Task Type Classification
HierBias introduces a hierarchical context-conditioned model for media bias detection that leverages document context to improve sentence-level classification, achieving state-of-the-art F1 and MCC on the BABE and BASIL datasets.
Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts
The paper proposes a multimodal solution for audio sentiment polarity classification that integrates audio and multilingual text transcripts via cross-modal transformers, and uses knowledge distillation to enhance an audio-only model without computational overhead during inference.