MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions

# A Multimodal Dataset for Humour Styles and Emotions
Source: [https://arxiv.org/html/2609.11322](https://arxiv.org/html/2609.11322)
## MultiHuSE: A Multimodal Dataset for Humour Styles and EmotionsThanks:This research was supported by the Petroleum Technology Development Fund \(PTDF\) of Nigeria\.

1stMary Ogbuka KennethAffiliation:Algorithmic Human Development group, Department of Computing Imperial College London London, United Kingdom m\.kenneth22@imperial\.ac\.uk2ndFoaad KhosmoodAffiliation:Computer Engineering Department California Polytechnic State University San Luis Obispo, United States foaad@calpoly\.edu3rdAbbas EdalatAffiliation:Algorithmic Human Development group, Department of Computing Imperial College London London, United Kingdom a\.edalat@imperial\.ac\.uk

###### Abstract

Computational recognition of verbal humour remains a challenging task, requiring an understanding of language, delivery style, emotions, and cultural context\. Most existing approaches focus on binary classification and lack datasets that capture psychological dimensions of humour alongside variations in expression\. We introduce MultiHuSE, a multimodal dataset comprising 2,407 high\-definition videos of 50 demographically diverse actors performing 1,463 text samples across four psychological humour styles \(affiliative, aggressive, self\-enhancing, and self\-deprecating\), as well as neutral content\. A subset is additionally annotated for underlying emotions\. The dataset uniquely captures multiple actor interpretations of the same texts, enabling systematic analysis of expressive diversity\. Baseline experiments show that multimodal fusion outperforms unimodal approaches \(80\.1% vs\. 77\.4% accuracy\) in humour style classification, with particularly strong gains for affiliative humour \(66% to 74%\)\. While text provides the strongest individual signal, fusion models deliver meaningful improvements\. We hope that MultiHuSE provides empirical support for psychological theories linking humour and emotion, while also opening new avenues for research in human communication, well\-being, and AI\-driven interaction\. The dataset is available for academic use under an End\-User Licence Agreement\.

###### Index Terms:

Computational humour, Multimodal emotion recognition, Humour style classification, Facial expression analysis, Affective computing

## IIntroduction

Humour recognition presents significant challenges for artificial intelligence systems, particularly in multimodal contexts that require integration of verbal, visual, and acoustic cues\[[1](https://arxiv.org/html/2609.11322#bib.bib7),[2](https://arxiv.org/html/2609.11322#bib.bib3),[3](https://arxiv.org/html/2609.11322#bib.bib10)\]\. While research in multimodal humour detection has advanced\[[4](https://arxiv.org/html/2609.11322#bib.bib32),[5](https://arxiv.org/html/2609.11322#bib.bib24),[6](https://arxiv.org/html/2609.11322#bib.bib35)\], most approaches remain limited to binary classification \(humour vs\. non\-humour\)\[[7](https://arxiv.org/html/2609.11322#bib.bib30)\], overlooking deeper psychological dimensions of humour identified by Martin et al\.\[[8](https://arxiv.org/html/2609.11322#bib.bib16)\]: self\-enhancing, self\-deprecating, affiliative, and aggressive styles\. These categories have significant implications for psychological well\-being: affiliative and self\-enhancing styles generally foster positive outcomes\[[9](https://arxiv.org/html/2609.11322#bib.bib29),[10](https://arxiv.org/html/2609.11322#bib.bib33),[11](https://arxiv.org/html/2609.11322#bib.bib12),[12](https://arxiv.org/html/2609.11322#bib.bib9)\], while aggressive and self\-deprecating styles can be detrimental to mental health\[[13](https://arxiv.org/html/2609.11322#bib.bib27),[14](https://arxiv.org/html/2609.11322#bib.bib22),[15](https://arxiv.org/html/2609.11322#bib.bib13)\]\.

The growing adoption of laughter\-based interventions in therapeutic settings for addressing depression and anxiety\[[16](https://arxiv.org/html/2609.11322#bib.bib34),[17](https://arxiv.org/html/2609.11322#bib.bib18)\], combined with evidence that specific humour styles differentially impact mental health outcomes\[[18](https://arxiv.org/html/2609.11322#bib.bib19),[19](https://arxiv.org/html/2609.11322#bib.bib11),[8](https://arxiv.org/html/2609.11322#bib.bib16)\], underscores the need for computational systems that detect humour styles and their associated emotional expressions\. Unlike conventional emotion datasets capturing well\-defined emotional states, our approach examines nuanced emotional undertones within humour delivery, emotions that may be subtly expressed or filtered through the comedic performance context\[[20](https://arxiv.org/html/2609.11322#bib.bib31)\]\. This integration enables computational validation of psychological theories and the development of emotionally\-aware systems for mental health applications\.

Existing multimodal humour datasets present several limitations for analysing humour styles\. UR\-FUNNY\[[21](https://arxiv.org/html/2609.11322#bib.bib36)\]focuses on binary humour detection using TED talks; MHD\[[5](https://arxiv.org/html/2609.11322#bib.bib24)\]uses sitcom recordings with laughter tracks; and Passau\-SFCH\[[22](https://arxiv.org/html/2609.11322#bib.bib25)\]provides style annotations but is restricted to German football press conferences\. Additionally, these datasets typically capture only a single performance per instance, limiting the study of expressive variation\.

To address these gaps, we introduceMultiHuSE, a new multimodal dataset of over 2,400 recordings featuring fifty diverse performers executing humour across four psychological categories and neutral content, with annotations of underlying emotion\. The dataset uniquely captures multiple interpretations of the same texts, enabling systematic analysis of expression variation and multimodal humour perception\. Our contributions include:

1. 1\.The first English\-language multimodal dataset annotated with psychological humour styles and enriched with emotion labels\.
2. 2\.Multiple video performances of the same text by different actors, enabling analysis of expressive and subjective variation in humour delivery
3. 3\.Benchmark experiments showing that multimodal fusion outperforms unimodal models \(80% vs\. 77% F1\-score\)

## IIRelated Work

Recent datasets have advanced multimodal humour detection, but with significant limitations \(see Table[I](https://arxiv.org/html/2609.11322#S2.T1)for comparison\)\. UR\-FUNNY\[[21](https://arxiv.org/html/2609.11322#bib.bib36)\]contains 16,514 video segments from TED Talks with binary classification, while MHD\[[5](https://arxiv.org/html/2609.11322#bib.bib24)\]provides 13,633 TV sitcom clips with laughter\-based annotations\. Both datasets offer valuable scale and naturalistic settings but focus solely on binary humour detection, overlooking psychologically grounded humour styles that have different implications for well\-being\[[8](https://arxiv.org/html/2609.11322#bib.bib16),[2](https://arxiv.org/html/2609.11322#bib.bib3)\]\.

The Passau\-SFCH dataset\[[22](https://arxiv.org/html/2609.11322#bib.bib25)\]addresses this gap by incorporating four humour styles across 39,682 German football press conference segments, though only 6% contains actual humour content\. However, it is limited to male German speakers in a single domain and lacks controlled recording conditions, limiting its generalisability across diverse populations and contexts\. Similarly, YouTube stand\-up datasets\[[23](https://arxiv.org/html/2609.11322#bib.bib23)\]capture authentic performances but focus on laughter detection rather than humour style classification\.

Multimodal emotion datasets provide methodological foundations for affective analysis but are limited for humour research\. MELD\[[24](https://arxiv.org/html/2609.11322#bib.bib20)\]includes 13,708 video segments fromFriendswith rich emotion annotations, but only 38% contain humorous content and lacks humour\-specific style labels\. SHEMuD\[[25](https://arxiv.org/html/2609.11322#bib.bib2)\]combines binary humour detection with emotion labels across 6,191 TV dialogue segments, but its humour classification remains binary without distinguishing psychologically meaningful humour styles\. Both datasets capture only single expressions of content, preventing analysis of how performers might express the same humorous material with varying emotional undertones\.

TABLE I:Comparison of existing multimodal humour and emotion datasets\. The table shows key characteristics, including data source, number of video samples, average video duration in seconds, modalities used, annotation granularity, and whether datasets capture single or multiple performances of the same contentDatasetContent Source\# VideosAvg Dur\.\[s\]ModalitiesAnnotation TypeKey Limitations/FeaturesUR\-FUNNY\[[21](https://arxiv.org/html/2609.11322#bib.bib36)\]TED Talks16,5144\.58Video, audio, textBinary humourSingleEnglish; No humour style differentiationMHD\[[5](https://arxiv.org/html/2609.11322#bib.bib24)\]TV sitcoms\(Big Bang Theory\)13,6333\.83Video, audioBinary humourSingleEnglish; Producer\-added laughter; no stylesPassau\-SFCH\[[22](https://arxiv.org/html/2609.11322#bib.bib25)\]Football press conferences39,682N/AVideo, audio, textFour humour stylesSingleGerman; Limited to male speakers in single domainMELD\[[24](https://arxiv.org/html/2609.11322#bib.bib20)\]TV series \(Friends\)13,7083\.59Video, audio, textEmotions onlySingleEnglish; No humour style annotationsSHEMuD\[[26](https://arxiv.org/html/2609.11322#bib.bib4)\]TV dialogues6,191N/AVideo, audio, textBinary humour \+ emotionsSingleHindi/English; Single source; no style annotationsYouTube Stand\-up\[[23](https://arxiv.org/html/2609.11322#bib.bib23)\]YouTube stand\-up comedy39,127N/AVideo, audio, textLaughter\-basedSingleRussian/English; Focus on laughter, not humour stylesMultiHuSE \(Ours\)Controlled studio2,4076\.66Video, audio, textFour humour styles\+ emotionsMultipleEnglish; Diverse performers; consistent quality;943 alternative performances ofidentical jokes with varied expressions

### II\-ADataset Gaps and Research Motivation

The analysis of existing datasets reveals three critical gaps: \(1\) predominant focus on binary classification rather than psychologically meaningful humour styles, \(2\) single\-expression capture that overlooks interpretative diversity, and \(3\) limited integration with psychological theories linking humour to emotional expression and well\-being\. We developed MultiHuSE with the goal of addressing these gaps and supporting research at the intersection of computation and psychological theory\.

## IIIDataset Creation

We present MultiHuSE, an unpublished multimodal dataset for humour style and emotion recognition\. The collection comprises 2,407 high\-definition recordings \(1080p at 25fps\) of fifty demographically diverse participants performing nearly 1,500 text instances\. The texts were sourced from the humour style corpus of Kenneth et al\.\[[2](https://arxiv.org/html/2609.11322#bib.bib3)\], which includes psychologically grounded humour style annotations\. While the text annotations are from this prior work, all video recordings, actor performances, and emotion annotations are novel contributions of our dataset\.

### III\-AData Collection

##### Recording Setup

We employed a professional, controlled recording environment in a dedicated room at Imperial College London with a Canon EOS Rebel DSLR camera and Rode VideoMic Pro directional microphone\. All recordings were conducted under identical setup conditions—white backdrop, consistent lighting, and standardised participant positioning \(head to knee\)—to ensure both facial expressions and body gestures were clearly visible while minimising environmental variability that could interfere with model training\.

##### Participant Demographics

We recruited 50 actors \(54% female, 40% male, 6% other genders; ages 18\-\-69 years; 72% White, 12% Asian/Asian British, 6% Black/African/Caribbean, 10% other backgrounds\) through a structured recruitment process\.111Detailed recruitment process available at:https://sites\.google\.com/view/laughterai\-com/recruitementParticipants received a £50 compensation for their∼2\.5\\sim 2\.5hour commitment, including preparation and recording\.

##### Participant Preparation

Participants received detailed instructions explaining each humour style \(full instructions available online222https://sites\.google\.com/view/laughterai\-com/dataset\), along with the following definitions:

- •Affiliative: Humour used to build connections and social bonds, focusing on shared experiences\.
- •Self\-enhancing: Humour involving finding amusement in adversity; a positive, adaptive style\.
- •Aggressive: Humour used to attack or belittle others through sarcasm or put\-downs\.
- •Self\-deprecating: Humour involving making fun of oneself, highlighting personal weaknesses\.
- •Neutral: Statements that are not jokes or do not fit other humour styles\.

Participants were provided with the text samples prior to the recording day to familiarise themselves with the content, practise delivery with different emotions, and opt out of any samples they felt uncomfortable performing\. They were instructed to consider six emotional categories \(joy, empathy, anger, neutral, embarrassment, and superiority\) while preparing their performances\.

##### Dataset Composition

Our video collection contains over 2,400 samples distributed across five categories with near\-balanced representation: self\-enhancing \(500 videos, 20\.8%\), aggressive \(529 videos, 22\.0%\), neutral \(495 videos, 20\.6%\), self\-deprecating \(457 videos, 19\.0%\), and affiliative \(426 videos, 17\.7%\) instances\.

The dataset uniquely features 943 re\-performed videos \(approximately 40% of the dataset\), capturing multiple interpretations of identical content for direct comparison of expression variations\. This approach enables systematic analysis of the subjective nature of humour delivery, where performers bring unique interpretations to the same textual content\. Fig\.[1](https://arxiv.org/html/2609.11322#S3.F1)illustrates this interpretative diversity through an example of the aggressive humour sample“I hear you were born on April 2, one day too late”\. The first performer \(Fig\.[1\(a\)](https://arxiv.org/html/2609.11322#S3.F1.sf1)\) delivers the joke with visible anger, whilst the second performer \(Fig\.[1\(b\)](https://arxiv.org/html/2609.11322#S3.F1.sf2)\) expresses a sense of superiority—demonstrating how identical text can elicit distinct emotional expressions and performance styles\. Such variation is particularly valuable for studying the nuanced relationship between humour styles, emotional expression, and delivery techniques\.

![Refer to caption](https://arxiv.org/html/2609.11322v1/images/ammar_2.png)\(a\)Performer 1 Expressing Anger
![Refer to caption](https://arxiv.org/html/2609.11322v1/images/jonathan_2.png)\(b\)Performer 2 Expressing Superiority

Fig\. 1:Expression variability for the same aggressive joke: \(a\) anger versus \(b\) superiority delivery of joke:“I hear you were born on April 2, one day too late”\.
##### Recording Duration

Videos range from 2 to 74\.96 seconds \(mean = 6\.66 s\), with 91\.3% under 10 seconds, 6\.7% \(10–20 s\), 1\.2% \(20–30 s\), and 0\.8% \(\>30\>30s\)\. Brief videos include concise jokes such as“Do your thing and don’t care if they like it”\(self\-enhancing, 2 seconds\),“They say good things take time so that’s why I am always late”\(self\-enhancing, 3 seconds\), and“I feel there is no more trash to burn but myself”\(self\-deprecating, 2 seconds\)\. These brief clips capture complete but concise humorous expressions, while longer videos contain more elaborate joke narratives\. This variation in duration reflects the natural diversity of humour delivery, from quick one\-liners to extended anecdotes\.

### III\-BAnnotation and Performance Approach

##### Humour Style and Text Categories

We utilised the labels from the existing text dataset\[[2](https://arxiv.org/html/2609.11322#bib.bib3)\], which contains 1,463 instances across five categories: 1,129 jokes aligned with four humour styles framework \(self\-enhancing, self\-deprecating, affiliative, and aggressive\)\[[8](https://arxiv.org/html/2609.11322#bib.bib16)\], and 334 neutral non\-jokes\.

##### Performance Guidance and Interpretative Freedom

Actors received a preparation document333Available at:https://sites\.google\.com/view/laughterai\-com/datasetwith brief definitions of each humour style, along with the actual jokes and their associated category labels\. Crucially, the instructions explicitly stated:“Remember, there are no ‘correct’ interpretations\. We’re interested in your natural portrayal of these jokes across different emotional spectrums\.”Participants were encouraged to practise delivering each joke with different emotions from a provided list \(joy, anger, neutral, superiority, embarrassment, and empathy\) to explore varied interpretations\. This balanced approach–providing category understanding while emphasising interpretative freedom– resulted in substantial performance diversity, even for identical text samples, as evidenced by the varying emotions displayed across performances \(see Fig\.[2](https://arxiv.org/html/2609.11322#S3.F2), analysed in detail in Section[III\-C](https://arxiv.org/html/2609.11322#S3.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2609.11322v1/images/paired_humour_emotion_heatmaps.png)Fig\. 2:Relationship between humour styles and emotions visualised through heatmaps\. Left: original performance annotations; Right: re\-performance annotations\. Darker colours represent stronger associations between specific emotions and humour styles
##### Emotion Annotation

Immediately after performing each text sample, performers provided self\-reports of their own emotional state during delivery, selecting from six emotion categories \(joy, anger, neutral, superiority, embarrassment, and empathy\)\. These self\-reported emotions represent the performers’ internal experience while expressing the humour, not anticipated audience reactions or observer perceptions\. This approach aligns with psychological research\[[10](https://arxiv.org/html/2609.11322#bib.bib33),[27](https://arxiv.org/html/2609.11322#bib.bib17),[28](https://arxiv.org/html/2609.11322#bib.bib8),[29](https://arxiv.org/html/2609.11322#bib.bib15)\]that examines humour styles in relation to the expresser’s emotional state\. These categories enable analysis of humour\-embedded emotional states distinct from general emotion datasets, supporting psychological theory validation \(Section III\-C, Figure 2\)\. This performer\-centred emotional assessment provides insight into the psychological experience of humour production rather than reception\.

##### Emotion Inter\-annotator Agreement

We evaluated agreement between emotion annotations from original performances and re\-performances using paired data where the same jokes were performed by different actors or the same actor with different interpretations\. Our analysis used 687 paired instances representing jokes with both original and re\-performance emotion annotations\.

Our analysis employed two key metrics:

1. 1\.Simple Agreement Rate \(0\.440\): The proportion of instances where both annotators assigned the same emotion, calculated as the number of perfect matches \(302\) divided by the total paired instances \(687\)\.
2. 2\.Cohen’s Kappa \(0\.325\): A robust measure accounting for chance agreement, indicating fair agreement above random chance\[[30](https://arxiv.org/html/2609.11322#bib.bib6),[31](https://arxiv.org/html/2609.11322#bib.bib21)\]\.

While all original performances include emotion annotations, only 687 of the 943 re\-performances have emotion labels, allowing for paired analysis of these 687 jokes\.

### III\-CHumour Styles\-Emotion Relationships

Figure[2](https://arxiv.org/html/2609.11322#S3.F2)demonstrates consistent emotion\-humour associations across original and re\-performances using 687 paired instances\. The stable patterns \(e\.g\., aggressive humour predominantly paired with anger/superiority, self\-deprecating humour paired with embarrassment\) validate that these emotion\-humour relationships are robust across different performers, supporting psychological theories that link specific humour styles to corresponding emotional states during expression\. Key patterns include:

- •Neutral humour: Strong association with neutral emotion \(Original: 90; Re\-performance: 98\)
- •Aggressive humour: Predominantly paired withAnger\(Original: 60; Re\-performance: 74\) andSuperiority\(Original: 42; Re\-performance: 40\), reflecting consistent assertive or mocking interpretations\.
- •Self\-deprecating humour: Strongly aligns withEmbarrassment\(Original: 70; Re\-performance: 55\), supporting its association with vulnerability and self\-directed awkwardness\.
- •Self\-enhancing humour: Co\-occurs withEmpathy\(Original: 39; Re\-performance: 31\) andJoy\(Original: 36; Re\-performance: 45\), indicating reflective and positive self\-regard\.
- •Affiliative humour: Primarily linked toJoy\(Original: 59; Re\-performance: 67\), supporting its role in positive social bonding\.

The consistent patterns observed between original and re\-performance data indicate emotion\-humour associations that align with established psychological theories\[[10](https://arxiv.org/html/2609.11322#bib.bib33),[27](https://arxiv.org/html/2609.11322#bib.bib17),[28](https://arxiv.org/html/2609.11322#bib.bib8),[29](https://arxiv.org/html/2609.11322#bib.bib15)\]and offer potential support for computational approaches to recognising emotional aspects of humour styles\.

### III\-DEthical Considerations and Data Access

The MultiHuSE dataset was created with careful attention to ethical principles\. All procedures received approval from the Science, Engineering and Technology Research Ethics Committee at Imperial College London\. Participants provided informed consent, were compensated £50 for their time, and had the option to opt out of performing any content they found uncomfortable\. While personal identifiers were pseudonymised, facial features remained visible as required for research purposes, with participants explicitly informed of this necessity\.

The MultiHuSE dataset is available to researchers via a dedicated section of our project website444Dataset available at:https://sites\.google\.com/view/laughterai\-com/dataset, under a custom End\-User Licence Agreement \(EULA\) permitting use for academic and research purposes only\. The website currently provides information about the project and access procedures\.

To mitigate potential risks associated with offensive content in certain aggressive humour samples, the dataset documentation explicitly states that inclusion does not imply endorsement\. Use of the dataset is restricted to research purposes aligned with the advancement of mental health understanding and computational humour analysis\.

## IVExperiments

To demonstrate the viability and research utility of our multimodal collection, we performed baseline experiments evaluating humour style and emotion recognition across single and combined modalities\. Direct comparison with existing datasets is not feasible as MultiHuSE represents the first English\-language multimodal dataset with psychological humour style annotations, making our baselines foundational benchmarks for future research\.

### IV\-AFeature Extraction

We employed state\-of\-the\-art encoders to extract robust multimodal features:

Text:BERT\-base\-uncased embeddings \(768\-D\), using the \[CLS\] token from padded sequences \(max 128 tokens\)\[[32](https://arxiv.org/html/2609.11322#bib.bib5)\]

Audio:Dasheng\-0\.6B \(1280D\)\[[33](https://arxiv.org/html/2609.11322#bib.bib28)\], a powerful audio encoder pretrained on diverse audio data for capturing nuanced speech and sound features\.

Video:MC3\-18 \(768D\)\[[34](https://arxiv.org/html/2609.11322#bib.bib1)\], a spatiotemporal convolutional encoder optimised for short video clips and fine\-grained motion representation\.

### IV\-BData Splits and Preprocessing

We used an 80:20 train\-test split stratified by humour style for humour style classification\. Training data underwent 5\-fold stratified cross\-validation\.

### IV\-CBaseline Models

We evaluated XGBoost classifiers and Cross\-Modal Attention Transformer:

XGBoost:XGBClassifier\(use\_label\_encoder =False, random\_state=42, eval\_metric=’mlogloss’\)\[[35](https://arxiv.org/html/2609.11322#bib.bib37)\]\.

Transformer:Cross\-modal attention network with multi\-head attention \(8 heads\) projecting text, audio, and visual features to a common 1024D space, trained end\-to\-end for humour style and emotions classification\.

### IV\-DFusion Strategies

Single Modality:Individual XGBoost models trained for text, audio, and visual encoder features\.

Exponential Weighted Fusion:Predictions from separate XGBoost modality models combined using exponential weighting \(α=2\\alpha=2\) based on individual modality performance, resulting in dynamic weights \(humour: text 0\.54, audio 0\.31, visual 0\.15; emotion: text 0\.40, audio 0\.32, visual 0\.27\)\.

Cross\-Modal Attention:Uses an attention mechanism to weigh the relevance of features from one modality when processing another\. This allows the model to learn complex relationships between modalities directly\.

### IV\-EEvaluation

We report accuracy, precision, recall, and F1\-score \(weighted averages\)\. Accuracy measures overall prediction correctness, while precision reflects the proportion of true positives among predicted positives\. Recall assesses the model’s ability to identify all actual positives, and the F1\-score provides the harmonic mean of precision and recall, balancing both metrics\.

## VResults

### V\-AModality Effectiveness Analysis

Table[II](https://arxiv.org/html/2609.11322#S5.T2)shows baseline results\. Text features yielded the strongest performance \(77\.4%\), confirming linguistic content as the primary signal for humour style classification\. Audio \(58\.5%\) and visual \(40\.0%\) lagged behind, reflecting the subtle and subjective nature of emotional and humorous expression, which makes these cues harder to model\. Still, they add complementary prosodic and expressive information\.

Multimodal fusion outperformed unimodal baselines: exponential weighted fusion reached 80\.1% accuracy and cross\-attention 79\.7%\. Though gains over text\-only are modest \(\+2\.7%, \+2\.3%\), they highlight meaningful cross\-modal contributions and demonstrate the dataset’s compatibility with modern architectures\.

TABLE II:Comprehensive Multimodal Results on MultiHuSE DatasetTable[III](https://arxiv.org/html/2609.11322#S5.T3)shows humour\-style variation: neutral content was most reliable \(85 \- 90% F1\), while affiliative humour benefited most from multimodality \(66% to 74% F1 with exponential fusion\)\. Text remained dominant across all fusions, with audio\-visual alone achieving only 58\.3%\. Thus, humour style recognition is semantically driven but enriched by multimodal expression, underscoring the capacity of the data set to provide a complete understanding of humour\.

TABLE III:F1\-Scores for Individual Humour Styles
### V\-BDataset Quality and Validation

To assess the reliability and research utility of the dataset, we conducted validation across three key dimensions: feature extraction reliability, re\-performance variation, and annotation coverage\.

#### V\-B1Feature Extraction Reliability

To validate technical data quality, we assessed computational feature extraction success rates across all video samples using state\-of\-the\-art encoders\. Table[IV](https://arxiv.org/html/2609.11322#S5.T4)presents the success rates for each modality\.

TABLE IV:Feature extraction success ratesAll modalities achieved perfect extraction rates \(100%\) using modern pre\-trained encoders, ensuring comprehensive coverage for multimodal fusion experiments\.

#### V\-B2Re\-performance Variation

A central strength of our dataset is its capture of diverse interpretations of identical humorous content\. We analysed 1,872 re\-performance videos corresponding to 927 unique jokes, using 17 facial features extracted via MediaPipe Face Mesh\[[36](https://arxiv.org/html/2609.11322#bib.bib26)\]\. For each joke performed by multiple actors, we computed pairwise Euclidean distances between facial feature vectors and compared them to random baseline distances\.

TABLE V:Re\-performance variation analysisWithin\-joke distances were significantly smaller than random baseline distances, indicating that re\-performances of identical content maintain meaningful similarity whilst exhibiting substantial expressive diversity\. This statistically significant difference suggests that re\-performances capture both content consistency and interpretative variation, supporting MultiHuSE’s ability to study the subjective nature of humour expression\.

#### V\-B3Annotation Coverage

All 2,407 videos include complete humour style labels inherited from the source text dataset\. Self\-reported emotion annotations are available for all 1,463 original performances and 687 re\-performances \(72\.7% of re\-performed instances\), providing 89\.3% overall emotion coverage \(2,150/2,407 instances\)\. The partial emotion coverage resulted from methodological evolution during data collection, as the emotion annotation protocol for re\-performance was introduced partway through the study after recognising its value for understanding expression variation\.

## VIApplications of the MultiHuSE Dataset

MultiHuSE enables recognition of psychologically grounded humour styles with established links to mental health outcomes\. Meta\-analytic evidence shows affiliative and self\-enhancing humour protects against depression and anxiety\[[19](https://arxiv.org/html/2609.11322#bib.bib11),[18](https://arxiv.org/html/2609.11322#bib.bib19)\], while aggressive and self\-deprecating styles increase mental health risks\[[15](https://arxiv.org/html/2609.11322#bib.bib13),[37](https://arxiv.org/html/2609.11322#bib.bib14)\]\. Recent laughter\-based interventions for treating depression\[[17](https://arxiv.org/html/2609.11322#bib.bib18),[16](https://arxiv.org/html/2609.11322#bib.bib34)\]highlight the therapeutic potential of humour style differentiation\.

### VI\-AMental Health Applications

Digital Monitoring: Apps that track users’ humour expressions during therapy sessions–detecting predominant self\-deprecating humour \(depression risk indicator\) versus affiliative humour \(protective factor\)–enabling personalised intervention recommendations\. For example, a system could alert therapists when patients consistently use self\-deprecating humour, prompting targeted cognitive behavioural interventions\.

Clinical Assessment: Tools supporting mental health professionals in evaluating humour\-based coping mechanisms and therapeutic progress through automated analysis of patient expressions\.

Content Moderation: Systems detecting harmful humour while preserving adaptive styles, reducing cyberbullying and enhancing digital well\-being\.

Emotionally Aware AI: Virtual assistants that differentiate supportive from harmful humour, enabling psychologically sensitive responses in therapeutic contexts\.

## VIIConclusion

We present MultiHuSE, a multimodal dataset for humour style and emotion recognition comprising over 2,400 high\-definition recordings of fifty diverse performers executing a balanced set of textual content across four psychologically\-grounded categories\. The dataset includes 943 re\-performed samples, enabling direct comparison of expression variations for identical content\.

Our validation shows 100% feature extraction success with state\-of\-the\-art encoders and significant performance variation reflecting interpretative diversity \(p<0\.000001p<0\.000001\)\. Multimodal fusion outperformed single modalities, with exponential fusion achieving 80\.1% accuracy vs\. 77\.4% for text\-only; cross\-attention reached 79\.7%, confirming compatibility with modern deep learning\. Ablation highlights text dominance while showing complementary contributions from audio and visual cues\. We have observed relationships between humour styles and specific emotions that appear consistent with psychological theories\.

## VIIILIMITATIONS AND FUTURE DIRECTIONS

We acknowledge several limitations\. First, the moderate inter\-annotator agreement \(Cohen’sκ=0\.325\\kappa=0\.325\) reflects the inherent subjectivity of emotional experiences in humour contexts\. Second, despite demographic diversity within our sample 72% White, 12% Asian/Asian British, 6% Black/African/Caribbean, 10% other backgrounds\), recruitment was geographically constrained to London, which may limit the dataset’s representation of global humour expression patterns and cultural interpretations of humour styles\. This geographical limitation reinforces Western humour perspectives and may introduce cultural bias in computational models trained on this data\.

Third, while our dataset contains 2,407 videos compared to larger collections, our controlled design emphasises quality and balance–79\.4% humorous content with even distribution across styles, compared to Passau\-SFCH’s 6\.02% humorous content from a single domain\. Direct comparison with existing datasets is challenging as MultiHuSE represents the first English\-language multimodal dataset with psychological humour style annotations\.

Future research will address these limitations through multiple recruitment centers across different countries and cultural contexts to capture diverse humour expression patterns and cultural interpretations of humour styles\. Additional directions include exploring advanced vision\-language models, longitudinal studies of humour style changes for intervention applications, and investigating relationships between automatically detected and human\-perceived emotions in humour contexts\.

## References

- \[1\]\(2011\)Computational Humour\.InEmotion\-Oriented systems,pp\. 609–634\.Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[2\]M\. O\. Kenneth, F\. Khosmood, and A\. Edalat\(2024\)A Two\-Model Approach for Humour Style Recognition\.InProceedings of the 4th International Conference on Natural Language Processing for Digital Humanities,Miami,pp\. 259–274\.External Links:[Link](https://aclanthology.org/2024.nlp4dh-1.25/)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1),[§II](https://arxiv.org/html/2609.11322#S2.p1.1),[§III\-B](https://arxiv.org/html/2609.11322#S3.SS2.SSS0.Px1.p1.1),[§III](https://arxiv.org/html/2609.11322#S3.p1.1)\.
- \[3\]C\. Shani, N\. Borenstein, and D\. Shahaf\(2021\)How Did This Get Funded?\! Automatically Identifying Quirky Scientific Achievements\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,pp\. 14–28\.External Links:[Link](http://arxiv.org/abs/2106.03048),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.2)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[4\]Y\. Kayatani, Z\. Yang, M\. Otani, N\. Garcia, C\. Chu, Y\. Nakashima, and H\. Takemura\(2021\)The laughing machine: Predicting humor in video\.InProceedings \- 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021,pp\. 2072–2081\.External Links:ISBN 9780738142661,[Document](https://dx.doi.org/10.1109/WACV48630.2021.00212)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[5\]B\. N\. Patro, M\. Lunayach, D\. Srivastava, H\. Singh, and V\. P\. Namboodiri\(2021\)Multimodal Humor Dataset: Predicting Laughter tracks for Sitcoms\.Computer Vision fundation,pp\. 576–585\.External Links:[Link](https://delta-lab-iitk.github.io/)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1),[§I](https://arxiv.org/html/2609.11322#S1.p3.1),[TABLE I](https://arxiv.org/html/2609.11322#S2.T1.4.1.3.1),[§II](https://arxiv.org/html/2609.11322#S2.p1.1)\.
- \[6\]S\. Castro, D\. Hazarika, V\. Pérez\-Rosas, R\. Zimmermann, R\. Mihalcea, and S\. Poria\(2019\)Towards Multimodal Sarcasm Detection \(An Obviously Perfect Paper\)\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4619–4629\.External Links:[Link](https://github./)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[7\]M\. O\. Kenneth, F\. Khosmood, and A\. Edalat\(2024\)Systematic Literature Review: Computational Approaches for Humour Style Classification\.ArXiv\.External Links:[Link](https://arxiv.org/abs/2402.01759),[Document](https://dx.doi.org/https%3A//doi.org/10.48550/arXiv.2402.01759)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[8\]R\. A\. Martin, P\. Puhlik\-Doris, G\. Larsen, J\. Gray, and K\. Weir\(2003\)Individual differences in uses of humor and their relation to psychological well\-being: Development of the Humor Styles Questionnaire\.Journal of Research in Personality37,pp\. 48–75\.External Links:[Link](https://www.elsevier.com/locate/jrp)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1),[§I](https://arxiv.org/html/2609.11322#S1.p2.1),[§II](https://arxiv.org/html/2609.11322#S2.p1.1),[§III\-B](https://arxiv.org/html/2609.11322#S3.SS2.SSS0.Px1.p1.1)\.
- \[9\]A\. Edalat\(2023\)Self\-initiated humour protocols: An algorithmic approach for learning to laugh\.PsyArXiv5,pp\. 1–14\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.31234/osf.io/w9cvx)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[10\]W\. P\. Hampes\(2007\)The Relation Between Humor Styles and Empathy\.Europe’s Journal of Psychology6\(3\),pp\. 34–45\.External Links:[Link](https://www.ejop.org/)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1),[§III\-B](https://arxiv.org/html/2609.11322#S3.SS2.SSS0.Px3.p1.1),[§III\-C](https://arxiv.org/html/2609.11322#S3.SS3.p3.1)\.
- \[11\]C\. Y\. Plessen, F\. R\. Franken, C\. Ster, R\. R\. Schmid, C\. Wolfmayr, A\. M\. Mayer, M\. Sobisch, M\. Kathofer, K\. Rattner, E\. Kotlyar, R\. J\. Maierwieser, and U\. S\. Tran\(2020\)Humor styles and personality: A systematic review and meta\-analysis on the relations between humor styles and the Big Five personality traits\.Personality and Individual Differences154\.External Links:[Document](https://dx.doi.org/10.1016/j.paid.2019.109676),ISSN 01918869Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[12\]M\. O\. Kenneth, F\. Khosmood, and A\. Edalat\(2025\)Explaining Humour Style Classifications: An XAI Approach to Understanding Computational Humour Analysis\.Journal of Data Mining & Digital Humanities, NLP4DH\.External Links:[Link](http://arxiv.org/abs/2501.02891)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[13\]L\. Veselka, J\. A\. Schermer, R\. A\. Martin, and P\. A\. Vernon\(2010\)Relations between humor styles and the Dark Triad traits of personality\.Personality and Individual Differences48\(6\),pp\. 772–774\.External Links:[Document](https://dx.doi.org/10.1016/j.paid.2010.01.017),ISSN 01918869Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[14\]I\. I\. Khramtsova and T\. S\. Chuykova\(2016\)Mindfulness and self\-compassion as predictors of humor styles in US and Russia\.Social Psychology and Society7\(2\),pp\. 93–108\.External Links:[Document](https://dx.doi.org/10.17759/sps.2016070207),ISSN 23117052Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1)\.
- \[15\]N\. Kuiper and N\. McHale\(2009\)Humor styles as mediators between self\-evaluative standards and psychological well\-being\.Journal of Psychology: Interdisciplinary and Applied143\(4\),pp\. 359–376\.External Links:[Document](https://dx.doi.org/10.3200/JRLP.143.4.359-376),ISSN 00223980Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p1.1),[§VI](https://arxiv.org/html/2609.11322#S6.p1.1)\.
- \[16\]J\. E\. Yim\(2016\)Therapeutic benefits of laughter in mental health: A theoretical review\.Vol\.239,Tohoku University Medical Press\.External Links:[Document](https://dx.doi.org/10.1620/TJEM.239.243),ISSN 13493329Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p2.1),[§VI](https://arxiv.org/html/2609.11322#S6.p1.1)\.
- \[17\]N\. S\. Akimbekov and M\. S\. Razzaque\(2021\)Laughter therapy: A humor\-induced hormonal intervention to reduce stress and anxiety\.Vol\.4,Elsevier B\.V\.\.External Links:[Document](https://dx.doi.org/10.1016/j.crphys.2021.04.002),ISSN 26659441Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p2.1),[§VI](https://arxiv.org/html/2609.11322#S6.p1.1)\.
- \[18\]T\. E\. Ford, S\. K\. Lappi, E\. C\. O’Connor, and N\. C\. Banos\(2017\)Manipulating humor styles: Engaging in self\-enhancing humor reduces state anxiety\.Humor30\(2\),pp\. 169–191\.External Links:[Document](https://dx.doi.org/10.1515/humor-2016-0113),ISSN 16133722Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p2.1),[§VI](https://arxiv.org/html/2609.11322#S6.p1.1)\.
- \[19\]Á\. Menéndez\-Aller, Á\. Postigo, P\. Montes\-Álvarez, F\. J\. González\-Primo, and E\. García\-Cueto\(2020\)Humor as a protective factor against anxiety and depression\.International Journal of Clinical and Health Psychology20\(1\),pp\. 38–45\.External Links:[Document](https://dx.doi.org/10.1016/j.ijchp.2019.12.002),ISSN 16972600Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p2.1),[§VI](https://arxiv.org/html/2609.11322#S6.p1.1)\.
- \[20\]G\. Ritchie\(2018\)The Comprehension of Jokes: A Cognitive Science Framework\.1 edition,Routledge Taylor & Francis Group,London\(English\)\.Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p2.1)\.
- \[21\]M\. Kamrul Hasan, W\. Rahman, A\. Zadeh, J\. Zhong, M\. Iftekhar Tanveer, L\. Morency, and M\. Hoque\(2019\)UR\-FUNNY: A Multimodal Language Dataset for Understanding Humor\.InProceedings ofthe 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2046–2056\.External Links:[Link](https://www.ted.com/)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p3.1),[TABLE I](https://arxiv.org/html/2609.11322#S2.T1.4.1.2.1),[§II](https://arxiv.org/html/2609.11322#S2.p1.1)\.
- \[22\]L\. Christ, S\. Amiriparian, A\. Kathan, N\. Müller, A\. König, and B\. W\. Schuller\(2022\)Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results\.Transactions on Affective Computing20\(10\),pp\. 1–15\.External Links:[Link](http://arxiv.org/abs/2209.14272)Cited by:[§I](https://arxiv.org/html/2609.11322#S1.p3.1),[TABLE I](https://arxiv.org/html/2609.11322#S2.T1.4.1.4.1),[§II](https://arxiv.org/html/2609.11322#S2.p2.1)\.
- \[23\]A\. Kuznetsova and C\. Strapparava\(2024\)Multimodal and Multilingual Laughter Detection in Stand\-Up Comedy Videos\.InThe 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation,pp\. 11884–11889\.External Links:[Link](https://github.com/MontrealCorpusTools/)Cited by:[TABLE I](https://arxiv.org/html/2609.11322#S2.T1.4.1.7.1),[§II](https://arxiv.org/html/2609.11322#S2.p2.1)\.
- \[24\]S\. Poria, D\. Hazarika, N\. Majumder, G\. Naik, E\. Cambria, and R\. Mihalcea\(2019\)MELD: A Multimodal Multi\-Party Dataset for Emotion Recognition in Conversations\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 527–536\(English\)\.Cited by:[TABLE I](https://arxiv.org/html/2609.11322#S2.T1.4.1.5.1),[§II](https://arxiv.org/html/2609.11322#S2.p3.1)\.
- \[25\]D\. Singh Chauhan, G\. Vikram Singh, A\. Arora, A\. Ekbal, and P\. Bhattacharyya\(2022\)A Sentiment and Emotion aware Multimodal Multiparty Humor Recognition in Multilingual Conversational Setting\.InProceedings of the 29th International Conference on Computational Linguistics,,pp\. 6752–6761\(English\)\.External Links:[Link](https://huggingface.co/)Cited by:[§II](https://arxiv.org/html/2609.11322#S2.p3.1)\.
- \[26\]D\. S\. Chauhan, G\. V\. Singh, A\. Arora, A\. Ekbal, and P\. Bhattacharyya\(2022\)An emoji\-aware multitask framework for multimodal sarcasm detection\.Knowledge\-Based Systems257\.External Links:[Document](https://dx.doi.org/10.1016/j.knosys.2022.109924),ISSN 09507051Cited by:[TABLE I](https://arxiv.org/html/2609.11322#S2.T1.4.1.6.1)\.
- \[27\]J\. Torres\-Marín, G\. Navarro\-Carrillo, and H\. Carretero\-Dios\(2018\)Is the use of humor associated with anger management? The assessment of individual differences in humor styles in Spain\.Personality and Individual Differences120,pp\. 193–201\.External Links:[Document](https://dx.doi.org/10.1016/j.paid.2017.08.040),ISSN 01918869Cited by:[§III\-B](https://arxiv.org/html/2609.11322#S3.SS2.SSS0.Px3.p1.1),[§III\-C](https://arxiv.org/html/2609.11322#S3.SS3.p3.1)\.
- \[28\]G\. E\. Weisfeld and M\. B\. Weisfeld\(2014\)Does a humorous element characterize embarrassment?\.Humor27\(1\),pp\. 65–85\.External Links:[Document](https://dx.doi.org/10.1515/humor-2013-0050),ISSN 09331719Cited by:[§III\-B](https://arxiv.org/html/2609.11322#S3.SS2.SSS0.Px3.p1.1),[§III\-C](https://arxiv.org/html/2609.11322#S3.SS3.p3.1)\.
- \[29\]M\. Billig\(2001\)Humour and Embarrassment: Limits of ‘Nice\-Guy’ Theories of Social Life\.Theory, Culture & Society \(SAGE, London, Thousand oaks and new Delhi\)18\(5\),pp\. 23–43\(English\)\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1177/02632760122051959)Cited by:[§III\-B](https://arxiv.org/html/2609.11322#S3.SS2.SSS0.Px3.p1.1),[§III\-C](https://arxiv.org/html/2609.11322#S3.SS3.p3.1)\.
- \[30\]C\. L\. Sabharwal\(2021\)Cohen’s Kappa Statistic and newKappaStatistic for Measuring and Interpreting Inter\-Rater Agreement\.International Journal of Research in Engineering and Science \(IJRES\) ISSN9\(7\),pp\. 23–28\.External Links:[Link](https://www.ijres.org/)Cited by:[item 2](https://arxiv.org/html/2609.11322#S3.I2.i2.p1.1)\.
- \[31\]S\. Sun\(2011\)Meta\-analysis of Cohen’s kappa\.Health Services and Outcomes Research Methodology11\(3\-4\),pp\. 145–163\.External Links:[Document](https://dx.doi.org/10.1007/s10742-011-0077-3),ISSN 13873741Cited by:[item 2](https://arxiv.org/html/2609.11322#S3.I2.i2.p1.1)\.
- \[32\]J\. Devlin, M\. Chang, K\. Lee, K\. T\. Google, and A\. I\. Language\(2019\)BERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding\.InProceedings of NAACL\-HLT,Minnesota,pp\. 4171–4186\.External Links:[Link](https://github.com/tensorflow/tensor2tensor)Cited by:[§IV\-A](https://arxiv.org/html/2609.11322#S4.SS1.p2.1)\.
- \[33\]H\. Dinkel, Z\. Yan, Y\. Wang, J\. Zhang, Y\. Wang, and B\. Wang\(2024\)Scaling up masked audio encoder learning for general audio classification\.InProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH,pp\. 547–551\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-246),ISSN 19909772Cited by:[§IV\-A](https://arxiv.org/html/2609.11322#S4.SS1.p3.1)\.
- \[34\]D\. Tran, H\. Wang, L\. Torresani, J\. Ray, Y\. Lecun, and M\. Paluri\(2018\)A Closer Look at Spatiotemporal Convolutions for Action Recognition\.InProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,pp\. 6450–6459\.External Links:ISBN 9781538664209,[Document](https://dx.doi.org/10.1109/CVPR.2018.00675),ISSN 10636919Cited by:[§IV\-A](https://arxiv.org/html/2609.11322#S4.SS1.p4.1)\.
- \[35\]T\. Chen and C\. Guestrin\(2016\)XGBoost: A scalable tree boosting system\.InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Vol\.13\-17\-August\-2016,pp\. 785–794\.External Links:ISBN 9781450342322,[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[§IV\-C](https://arxiv.org/html/2609.11322#S4.SS3.p2.1)\.
- \[36\]Y\. Kartynnik, A\. Ablavatski, I\. Grishchenko, and M\. Grundmann\(2019\)Real\-time Facial Surface Geometry from Monocular Video on Mobile GPUs\.arXiv\(English\)\.External Links:[Link](http://arxiv.org/abs/1907.06724)Cited by:[§V\-B2](https://arxiv.org/html/2609.11322#S5.SS2.SSS2.p1.1)\.
- \[37\]A\. Amjad and R\. Dasti\(2022\)Humor styles, emotion regulation and subjective well\-being in young adults\.Current Psychology41\(9\),pp\. 6326–6335\.External Links:[Document](https://dx.doi.org/10.1007/s12144-020-01127-y),ISSN 19364733Cited by:[§VI](https://arxiv.org/html/2609.11322#S6.p1.1)\.