BanglaMemeEvidence: A Multimodal Benchmark Dataset for Explanatory Evidence Detection in Bengali Memes

# BanglaMemeEvidence: A Multimodal Benchmark Dataset for Explanatory Evidence Detection in Bengali Memes
Source: [https://arxiv.org/html/2607.03981](https://arxiv.org/html/2607.03981)
11institutetext:Ahsanullah University of Science and Technology, Dhaka\-1208, Bangladesh11email:fatema\.faria142@gmail\.com, mukaffi28@gmail\.com, pronaydebnath99@gmail\.com, mahim1066@gmail\.com, fahimthescientist@gmail\.com, faisal\.cse@aust\.eduMukaffi Bin MoinMd\. Mahfuzur RahmanPronay DebnathAsif Iftekher FahimFaisal Muhammad Shah

###### Abstract

Memes have become influential communication tools on social media, combining viral visuals with concise messaging to convey impactful ideas\. While substantial research has examined the affective dimensions of memes, key challenges such as detecting harmful content, identifying cyberbullying, and performing accurate sentiment analysis remain critical, largely due to the need for deeper contextual understanding\. In this paper, we introduceMemeEvidenceDetect, a hybrid task aimed at analyzing a meme and its contextual information to identify specific sentences that explain or elucidate its meaning and humor\. To support this task, we presentBanglaMemeEvidence, a curated dataset of 2,917 Bengali memes, emphasizing its significance as a resource for the Bangla language\. Each meme is annotated with natural language explanations, including Meme OCR, Meme Context, and Evidence Sentences, alongside relevance scores that reflect the relationship between a meme and its corresponding annotations\. To address the gap in dynamically inferring a meme’s context, we proposeBengaliMemeEvidenceNet, a hybrid multimodal framework that integrates textual and visual features for comprehensive meme representation\. Our experiments demonstrate the effectiveness of BengaliMemeEvidenceNet, achieving an F1 score of 0\.74\. To the best of our knowledge, this is the first study to focus on evidence detection in Bengali memes, marking a notable step forward in the analysis of memes in low\-resource languages\.

## 1Intoduction

Social media has become a key way for people to communicate, changing how we interact in society\. The content shared on social media comes in many forms, including text, audio, and images, or a mix of them\. Memes, often shared with humor or sarcasm, are a common example\. While memes help spread complex ideas about society, culture, or politics, they often lack the context needed for full understanding, which is important for both humans and computers to interpret them accurately\.\[[21](https://arxiv.org/html/2607.03981#bib.bib1)\]\[[20](https://arxiv.org/html/2607.03981#bib.bib2)\]\.

![Refer to caption](https://arxiv.org/html/2607.03981v1/x1.png)Figure 1:An illustration of MemeEvidenceDetect, a hybrid task where, given a meme, manually transcribed Bengali text, and a description of the meme’s context, the meme is supported by evidence sentences\. These sentences are tagged with relevance scores, where a score of 2 denotes “Relevant”\.The recent surge in meme dissemination has led to a growing body of research on meme analysis, focusing on tasks such as multimodal hate speech detection from Bengali memes and texts\[[12](https://arxiv.org/html/2607.03981#bib.bib3)\], the development of datasets for Bengali abusive meme classification\[[7](https://arxiv.org/html/2607.03981#bib.bib4)\], multimodal cyberbullying meme detection from social media using deep learning approaches\[[1](https://arxiv.org/html/2607.03981#bib.bib5)\], and multimodal analysis of memes for sentiment extraction\[[3](https://arxiv.org/html/2607.03981#bib.bib6)\]\. Other studies have worked on multimodal offensive meme classification using transformers\[[15](https://arxiv.org/html/2607.03981#bib.bib7)\], sarcasm detection in typographic memes\[[13](https://arxiv.org/html/2607.03981#bib.bib8)\], and multi\-task learning frameworks for multi\-modal sarcasm, sentiment, and emotion analysis\[[5](https://arxiv.org/html/2607.03981#bib.bib9)\]\.

Despite the extensive work done in this field, there has been a noticeable lack of emphasis on multimodal evidence detection, especially concerning Bangla memes, which poses a significant challenge\. To address this gap, we present innovative solutions aimed at automating the extraction of contextual evidence, thereby enhancing meme accessibility\. Introducing a pioneering framework called BengaliMemeEvidenceNet, our approach focuses on identifying sentences within the provided context that can potentially elucidate the meme’s meaning\.

Figure[1](https://arxiv.org/html/2607.03981#S1.F1)presents a meme categorized under Politics, featuring Bengali text that translates to “But you are my opposing candidate\.” This meme contextualizes the 2024 elections, depicting a complex political scenario in which the Awami League leadership strategically nominates candidates from both its own ranks and rival parties\. This strategic move seeks to influence world opinion and strengthen domestic authority by creating the perception of democratic choice while concealing underlying political strategies\. The goal of this research is to analyze this multimodal meme in order to uncover and investigate evidence of political interference during electoral processes\. The embedded evidence sentence in the meme provides direct insight into the government’s strategy: intentional candidate nominations across the political spectrum to shape global perceptions and consolidate local power\. This study seeks to explore methodologies for identifying such evidence within related contextual documents\. By doing so, it aims to elucidate the complex dynamics of communication in modern environments across diverse subjects and themes depicted in memes\. To summarize, our main contributions are as follows:

- ∘\\circMemeEvidenceDetectis an hybrid task that predicts and analyzes the relevance of sentences to a given meme, aiming to clarify its meaning and humor\. This task involves detecting relevance scores, with 0 indicating “Not relevant,” 1 representing “Partially relevant,” and 2 signifying “Relevant\.”
- ∘\\circTo bridge the gap in Bangla meme evidence research, we introduce a novel dataset calledBanglaMemeEvidence, designed to facilitate the automatic detection of contextual evidence in such memes\. The dataset comprises gold standard human annotations for 2,917 Bengali memes\. The Bengali text from the meme image is manually transcribed to ensure accurate content representation\.
- ∘\\circWe introduceBengaliMemeEvidenceNet, a hybrid multimodal framework aimed at identifying evidence for memes from their related contexts\. To achieve this, we employ various fusion techniques to integrate text and image features\.

## 2Related Work

### 2\.1Meme Analysis

The study by Karim et al\.\[[12](https://arxiv.org/html/2607.03981#bib.bib3)\]constructed a dataset integrating textual and visual elements to tackle hate speech detection\. Their study showcased the efficacy of Conv\-LSTM for textual analysis and DenseNet\-161 for visual interpretation, alongside highlighting the effectiveness of XLM\-RoBERTa in textual analysis\. In a parallel study\[[7](https://arxiv.org/html/2607.03981#bib.bib4)\], delved into abusive memes with the BanglaAbuseMeme dataset, favoring multimodal strategies, particularly CLIP\. While XLMR showed promise in textual analysis and ViT in visual, it was CLIP that achieved the highest macro score\. Transitioning to cyberbullying detection, Ahmed et al\.\[[1](https://arxiv.org/html/2607.03981#bib.bib5)\]proposed a VGG16\-BiLSTM model, achieving notable accuracy in detecting harmful content within Bengali memes\. Meanwhile, Alluri and Krishna\[[3](https://arxiv.org/html/2607.03981#bib.bib6)\]explored meme classification using the Memotion dataset, employing ViT for visuals and RoBERTa for textual analysis, alongside a transformer\-based image captioning model for contextual understanding\.

### 2\.2Visual Question Answering \(VQA\)

Recent advancements in Bengali Visual Question Generation \(VQG\) and Visual Question Answering \(VQA\) underscore the need for further exploration in these areas\. Hasan et al\.\[[10](https://arxiv.org/html/2607.03981#bib.bib11)\]have made significant strides in Bengali VQG by employing a transformer\-based approach and introducing novel models, such as image\-category and image\-answer\-category variants, which have demonstrated superior question generation capabilities\. In contrast, Islam et al\.\[[11](https://arxiv.org/html/2607.03981#bib.bib12)\]focused on Bengali VQA by adapting existing datasets and employing a hybrid model that combines CNN and bi\-LSTM layers\. Meanwhile, Rafi et al\.\[[17](https://arxiv.org/html/2607.03981#bib.bib13)\]contributed to Bengali VQA by developing a human\-annotated Bengali VQA dataset and proposing a Top\-Down Attention\-based model\. Collectively, these studies highlight the need for continued research in VQG and VQA for underrepresented languages, showcasing the potential of modern CNN architectures and pretrained models in advancing image\-based question generation and answering\.

## 3BanglaMemeEvidence Dataset Annotation

### 3\.1Meme Collection

Our meme collection process involved a comprehensive search across a diverse range of sources, ensuring a rich and varied dataset\. We scoured popular social media platforms such as Facebook, Instagram, and X \(formerly known as Twitter\) to capture the latest trends and viral content\. Additionally, we delved into various online websites, forums, and communities known for their vibrant meme cultures\. To ensure the dataset’s breadth and relevance, we cast our net wide, encompassing memes spanning multiple thematic categories\. We explored content related to Entertainment, where memes often riff on celebrity culture, movies, and television shows, providing a lighthearted and humorous take on popular media\. In the realm of Politics, we curated memes that reflect the dynamic landscape of political discourse, capturing satirical commentary, political humor, and memes that encapsulate key events and figures in the political sphere\. Our collection extended to Sports, where memes capture the passion, drama, and camaraderie of sporting events, as well as the lighter side of sports culture through memes celebrating iconic moments and poking fun at sporting rivalries\. Education\-themed memes offered insights into the world of academia, student life, and the challenges and quirks of learning environments, resonating with students and educators alike\. In the realm of Technology, memes provided a humorous lens through which to explore the ever\-evolving world of gadgets, software, and digital culture, offering witty observations on the quirks and frustrations of technology users\.

![Refer to caption](https://arxiv.org/html/2607.03981v1/x2.png)Figure 2:Overview of BanglaMemeEvidence dataset across different meme categories
### 3\.2Context Document Curation

In curating the contextual corpus corresponding to the memes collected, we employed a multifaceted approach\. For some topics, we relied on Bangla Wikipedia111[https://bn\.wikipedia\.org/](https://bn.wikipedia.org/)as a primary source, leveraging its wealth of information to provide evidence for the background of the memes\. Additionally, we explored community\-based discussion forums and question\-answering websites such as Quora222[https://bn\.quora\.com/](https://bn.quora.com/), as well as other general\-purpose websites\. Our thorough search spanned traditional media and digital publications, including newspapers333[https://www\.prothomalo\.com/](https://www.prothomalo.com/), online article portals444[https://blog\.muktomona\.com/](https://blog.muktomona.com/), and specialized media outlets555[https://ekattor\.tv/](https://ekattor.tv/), where we gathered in\-depth analysis and expert commentary on meme\-related events and phenomena\. This comprehensive process ensured that each context document captured the essence of meme culture, offering valuable insights into its social, cultural, and political underpinnings\.

### 3\.3Dataset Description

- ∘\\circMeme\_ID:Each meme is assigned a unique image ID for identification purposes\.
- ∘\\circMeme\_Category:We have taken Politics, Sports, Entertainment, Education, Technology, and Others as meme categories\.
- ∘\\circMeme\_OCR:The Bengali text from the meme image is manually transcribed, ensuring accurate representation of the content\.
- ∘\\circMeme\_Context:A concise description of the main Bengali context or theme depicted in the meme is provided\.
- ∘\\circEvidence\_Sentence:This section includes a sentence or short text extracted from contextual documents, serving as evidence to support the interpretation or understanding of the meme\.
- ∘\\circRelevance\_Score:Each evidence sentence in the dataset is tagged with a relevance score, indicating its relationship to the meme\. This score operates on a scale where\(0\)denotes “Not relevant,”\(1\)signifies “Partially relevant,” and\(2\)represents “Relevant\.”

![Refer to caption](https://arxiv.org/html/2607.03981v1/x3.png)Figure 3:An illustration showcasing the diversity of meme categories, including Politics, Sports, Entertainment, and more from the BanglaMemeEvidence dataset\.
### 3\.4Annotation Guideline

We provided the following annotation guideline to the annotators:

a\) Comprehensive Comprehension:We ensure understanding of both the meme and its associated context before annotation\. This comprehensive comprehension is crucial for accurate and insightful annotation\.

b\) Semantics Alignment:We let the semantics of the meme guide the annotation process\. By aligning with the intended meaning of the meme, our annotations maintain relevance and fidelity\.

c\) Unit of Information:We recognize that self\-contained, minimal units of information can serve as evidence\. Each piece of evidence contributes to our understanding of the meme’s background and significance\.

d\) Non\-Contiguous Evidence:We acknowledge that valid evidence may not always occur contiguously\. This understanding allows us to identify and annotate relevant information regardless of its spatial arrangement\.

e\) Completeness Assurance:In cases where the context document does not support a meme, we diligently search for corroborating evidence from other established sources\. This ensures the completeness and accuracy of our annotations\.

f\) Caution with Ambiguity:We exercise caution with ambiguous cases, opting to skip them to maintain the integrity of the annotation process\. This approach prevents potential misinterpretations and ensures the reliability of our dataset\.

### 3\.5Annotation Process

In our annotation process, we engaged the expertise of six male annotators, all hailing from Bangladesh and currently pursuing undergraduate studies\. With ages ranging between 21 to 25 years, these individuals possessed not only a knack for meme creation but also an intricate understanding of meme culture, coupled with proficiency in navigating various social media platforms\. Acknowledging the invaluable contribution of our annotators, we ensured fair compensation in accordance with Bangladeshi standards, recognizing the time and expertise they dedicated to the task\. Equipped with a comprehensive set of guidelines, our annotators embarked on their mission: to scour context documents in search of succinct sentences that provided essential background information for each meme\. These identified sentences, termed “evidence sentences,” served as the cornerstone of our dataset, offering profound insights into the genesis, significance, and cultural context of each meme\. To ensure the reliability and consistency of the annotations, we implemented a rigorous process for addressing disagreements between annotators\. Whenever annotators encountered differing interpretations of a meme or evidence sentences, they engaged in discussions to reach a consensus, guided by the annotation guidelines\. In cases where disagreements persisted, a senior annotator, with deeper expertise in meme culture and contextual analysis, reviewed the conflicting annotations and made the final decision\.

Table 1:Dataset Distribution Across Train, Test, and Validation Subsets Based on Relevance Score\.
### 3\.6Annotation Quality Maintenance

In our annotation process, we utilized Fleiss Kappa\[[2](https://arxiv.org/html/2607.03981#bib.bib26)\]to ensure annotation quality\. Fleiss Kappa assesses agreement among annotators, considering chance occurrences\. It provides a score indicating agreement beyond chance\. By monitoring Fleiss Kappa regularly, we maintained consistency and reliability in annotations\. This practice ensured the dataset’s accuracy and reliability for research and analysis\. We achieved a Fleiss Kappa score of0\.87, indicating strong agreement among annotators\.

### 3\.7Dataset Statistics

We partitioned the dataset into three subsets: training \(80%\), validation \(10%\), and testing \(10%\)\. This distribution ensures a balanced and representative split that supports effective model training, tuning, and evaluation\. By reserving distinct portions of the data for each stage, we aim to promote generalization and minimize overfitting\. Table[1](https://arxiv.org/html/2607.03981#S3.T1)presents the data split by relevance score across train, test, and validation sets\. Figure[2](https://arxiv.org/html/2607.03981#S3.F2)illustrates representative examples from the dataset\. The first meme critiques the superficial display of religious identity on social media through the adoption of Arabic names\. The second highlights issues in the education system, where material provisions are present, but effective learning often relies on external tutoring\. The third meme provides a humorous observation on technology usage, contrasting the frequent use of the left Ctrl key with the neglect of its counterpart on the right\. The fourth meme celebrates the consistent athletic performance of footballer Emiliano Martinez at both the national and club levels\. The fifth meme offers a pointed critique of electoral integrity, emphasizing low voter participation and the questionable fairness of the process\. The sixth brings a lighthearted moment from sports, depicting Bangladeshi cricketers happily claiming window seats during air travel, capturing a relatable and simple joy\. Figure[3](https://arxiv.org/html/2607.03981#S3.F3)showcases its category\-wise diversity\.

## 4Implementation Details

In our approach,BengaliMemeEvidenceNet, for detecting explanatory evidence in memes, we explore multiple fusion techniques, including Early Fusion, Late Fusion, and Intermediate Fusion\. By investigating and comparing the performance of these individual approaches, we develop a hybrid framework that combines the strengths of each fusion method\. This exploration allows us to propose the best\-performing fusion strategy, leveraging the advantages of each technique to achieve superior results in detecting explanatory evidence in memes\.

![Refer to caption](https://arxiv.org/html/2607.03981v1/x4.png)Figure 4:Illustration of BengaliMemeEvidenceNet, a hybrid multimodal framework designed for identifying evidence in Bengali memes by leveraging related contexts and predicting relevance scores\.Step 1\) Text Preprocessing:Before delving into analysis, we meticulously preprocess textual data extracted from various sources within memes, includingMeme\_OCR,Meme\_Context, andEvidence\_Sentence\. This comprehensive preprocessing involves several steps aimed at ensuring the cleanliness and standardization of the text:

- ∘\\circPunctuation Removal: Removing punctuation marks such as exclamation points, question marks, underscores, and quotation marks\. This can be mathematically represented as: Tcleaned=remove\_punctuation​\(T\)T\_\{\\text\{cleaned\}\}=\\text\{remove\\\_punctuation\}\(T\)\(1\)whereTTis the original text, andTcleanedT\_\{\\text\{cleaned\}\}is the text with punctuation removed\.
- ∘\\circWhitespace Elimination: Systematically eliminating any extraneous white spaces\. This step is represented as: Tcleaned=remove\_whitespace​\(Tcleaned\)T\_\{\\text\{cleaned\}\}=\\text\{remove\\\_whitespace\}\(T\_\{\\text\{cleaned\}\}\)\(2\)where we remove any unnecessary whitespace from the text\.
- ∘\\circEmoji Removal: Meticulously scanning and removing emojis to prevent noise and ambiguity\. This is expressed as: Tcleaned=remove\_emoji​\(Tcleaned\)T\_\{\\text\{cleaned\}\}=\\text\{remove\\\_emoji\}\(T\_\{\\text\{cleaned\}\}\)\(3\)where emojis are identified and removed from the text\.
- ∘\\circRemoving Non\-Textual Content: Remove any URLs, HTML tags, and special characters that are not relevant to the text analysis\. This can be represented as: Tcleaned=remove\_non\_textual​\(Tcleaned\)T\_\{\\text\{cleaned\}\}=\\text\{remove\\\_non\\\_textual\}\(T\_\{\\text\{cleaned\}\}\)\(4\)where any non\-relevant content is stripped from the text\.
- ∘\\circSpelling Correction: Correct misspelled words to standardize the text\. This is typically done by applying a spelling correction algorithm such as: Tcorrected=spell\_check​\(Tcleaned\)T\_\{\\text\{corrected\}\}=\\text\{spell\\\_check\}\(T\_\{\\text\{cleaned\}\}\)\(5\)whereTcorrectedT\_\{\\text\{corrected\}\}is the text with corrected spelling\.

Step 2\) Image Preprocessing:To ensure uniformity and high quality across our dataset, we standardized the size of images extracted from memes to224×224224\\times 224pixels, facilitating reliable and comparable analysis\. Our image preprocessing pipeline includes several critical steps aimed at enhancing image quality and preparing the visual data for robust feature extraction and analysis:

- ∘\\circNormalizing the images to maintain consistency in pixel values\. The normalization process is expressed as: Inormalized=I−μσI\_\{\\text\{normalized\}\}=\\frac\{I\-\\mu\}\{\\sigma\}\(6\)whereIIis the original image,μ\\muis the mean pixel value, andσ\\sigmais the standard deviation of the pixel values\.
- ∘\\circApplying edge detection algorithms \(such as the Sobel operator\) to highlight the boundaries within the images\. This is represented as: Iedges=Sobel​\(I\)I\_\{\\text\{edges\}\}=\\text\{Sobel\}\(I\)\(7\)whereIedgesI\_\{\\text\{edges\}\}contains the detected edges of the image\.
- ∘\\circApplying noise reduction techniques, such as Gaussian smoothing, to reduce unwanted artifacts\. This can be expressed as: Idenoised=Gaussian\_filter​\(I\)I\_\{\\text\{denoised\}\}=\\text\{Gaussian\\\_filter\}\(I\)\(8\)whereIdenoisedI\_\{\\text\{denoised\}\}is the filtered image with reduced noise\.
- ∘\\circAdjusting the contrast of the images to highlight significant features\. This adjustment is expressed as: Icontrast\_adjusted=adjust\_contrast​\(Idenoised\)I\_\{\\text\{contrast\\\_adjusted\}\}=\\text\{adjust\\\_contrast\}\(I\_\{\\text\{denoised\}\}\)\(9\)whereIcontrast\_adjustedI\_\{\\text\{contrast\\\_adjusted\}\}has improved contrast to better emphasize visual details\.

Step 3\) Feature Extraction:To capture the rich and intricate nuances of explanatory information within memes, we employ advanced pre\-trained language models and cutting\-edge image analysis techniques\. Our feature extraction process is two\-fold, focusing on both textual and visual data:

- ∘\\circTextual Feature Extraction:We utilize state\-of\-the\-art \(SOTA\) pre\-trained language models to extract semantic insights from various textual components of memes\. The models we employ include mBERT\[[8](https://arxiv.org/html/2607.03981#bib.bib14)\], XLM\-RoBERTa\[[6](https://arxiv.org/html/2607.03981#bib.bib15)\], and distilBERT\[[18](https://arxiv.org/html/2607.03981#bib.bib21)\]\. These models excel in understanding context and semantics, allowing us to accurately capture the subtleties and nuances inherent in the textual data of memes\. We preferred mBERT, XLM\-RoBERTa, and distilBERT due to their excellent multilingual capabilities, which are helpful when performing cross\-lingual tasks and data collection from related languages\.
- ∘\\circVisual Feature Extraction:For the visual aspect, we utilize advanced models specifically tailored for meme analysis to extract relevant features from images\. These models include Vision Transformers \(ViTs\)\[[9](https://arxiv.org/html/2607.03981#bib.bib16)\], Swin Transformer\[[14](https://arxiv.org/html/2607.03981#bib.bib17)\], SwiftFormer\[[19](https://arxiv.org/html/2607.03981#bib.bib18)\], PoolFormer\[[22](https://arxiv.org/html/2607.03981#bib.bib19)\]\. Each model is trained to detect and emphasize significant visual elements that contribute to the explanatory content of memes\.

Step 4\) Fusion Techniques:After extracting text and image features, we employ fusion techniques tailored to each approach: Early Fusion\[[4](https://arxiv.org/html/2607.03981#bib.bib23)\]and Late Fusion\[[16](https://arxiv.org/html/2607.03981#bib.bib24)\]\.

a\) Early Fusion for Evidence Detection:In early fusion, we integrate features from both text and image modalities at a raw level and then feed them into a joint representation function\. This joint representation is then used for further analysis\. The equation for early fusion can be mathematically represented as:

𝐙joint=fearly​\(𝐗text,𝐗image\)\\mathbf\{Z\}\_\{\\text\{joint\}\}=f\_\{\\text\{early\}\}\(\\mathbf\{X\}\_\{\\text\{text\}\},\\mathbf\{X\}\_\{\\text\{image\}\}\)\(10\)
Where:

- ∘\\circ𝐙joint\\mathbf\{Z\}\_\{\\text\{joint\}\}represents the joint representation combining features from text and image\.
- ∘\\circfearlyf\_\{\\text\{early\}\}is the early fusion function\.
- ∘\\circ𝐗text\\mathbf\{X\}\_\{\\text\{text\}\}denotes the feature representation extracted from text sources such as Meme\_OCR, Meme\_Context, and Evidence\_Sentence\.
- ∘\\circ𝐗image\\mathbf\{X\}\_\{\\text\{image\}\}represents the feature representation extracted from the meme image\.

b\) Late Fusion for Evidence Detection:In late fusion, predictions from text and image classification models are aggregated at a later stage to make the final decision\. Each modality produces its prediction, and these predictions are combined using a fusion function\. The equation for late fusion can be mathematically represented as:

y^final=flate​\(y^text,y^image\)\\hat\{y\}\_\{\\text\{final\}\}=f\_\{\\text\{late\}\}\(\\hat\{y\}\_\{\\text\{text\}\},\\hat\{y\}\_\{\\text\{image\}\}\)\(11\)
Where:

- ∘\\circy^final\\hat\{y\}\_\{\\text\{final\}\}represents the final prediction\.
- ∘\\circflatef\_\{\\text\{late\}\}is the late fusion function\.
- ∘\\circy^text\\hat\{y\}\_\{\\text\{text\}\}denotes the prediction obtained from the text classification model based on sources like Meme\_OCR, Meme\_Context, and Evidence\_Sentence\.
- ∘\\circy^image\\hat\{y\}\_\{\\text\{image\}\}represents the prediction obtained from the image classification model\.

c\) Intermediate Fusion for Evidence Detection:In intermediate fusion, features from both text and image modalities are fused at an earlier stage, before the final classification decision is made\. Rather than aggregating the predictions directly, the features from both modalities are combined and passed through a shared model to produce the final prediction\. The fusion function in this case operates on the feature vectors of both modalities\.

Let the feature vectors from the text and image models be denoted as𝐟text\\mathbf\{f\}\_\{\\text\{text\}\}and𝐟image\\mathbf\{f\}\_\{\\text\{image\}\}, respectively\. The intermediate fusion can be mathematically represented as:

y^final=fintermediate​\(𝐟text,𝐟image\)=𝐟text\+𝐟image\\hat\{y\}\_\{\\text\{final\}\}=f\_\{\\text\{intermediate\}\}\\left\(\\mathbf\{f\}\_\{\\text\{text\}\},\\mathbf\{f\}\_\{\\text\{image\}\}\\right\)=\\mathbf\{f\}\_\{\\text\{text\}\}\+\\mathbf\{f\}\_\{\\text\{image\}\}\(12\)
Where:

- ∘\\circy^final\\hat\{y\}\_\{\\text\{final\}\}represents the final prediction after intermediate fusion\.
- ∘\\circfintermediatef\_\{\\text\{intermediate\}\}is the intermediate fusion function, which in this case is the element\-wise sum of the feature vectors from both modalities\.
- ∘\\circ𝐟text\\mathbf\{f\}\_\{\\text\{text\}\}denotes the feature vector obtained from the text classification model, derived from sources such as Meme\_OCR, Meme\_Context, and Evidence\_Sentence\.
- ∘\\circ𝐟image\\mathbf\{f\}\_\{\\text\{image\}\}represents the feature vector obtained from the image classification model\.

In this implementation, the intermediate fusion functionfintermediatef\_\{\\text\{intermediate\}\}combines the feature vectors from the text and image modalities by performing an element\-wise sum of the two feature vectors\.

Step 5\) Hyperparameter Tuning:Hyperparameter tuning plays a vital role in optimizing the performance of fusion strategies such as Early, Late, and Intermediate Fusion\. For Early Fusion, the key hyperparameters include the learning rate \(η\\eta\), fusion weight \(wfusionw\_\{\\text\{fusion\}\}\), and dropout rate \(dd\)\. The optimal learning rateηopt\\eta\_\{\\text\{opt\}\}is determined by minimizing the loss function:

ηopt=arg⁡minη⁡L​\(θ;η\)\\eta\_\{\\text\{opt\}\}=\\arg\\min\_\{\\eta\}L\(\\theta;\\eta\)\(13\)The fusion weightwfusionw\_\{\\text\{fusion\}\}is tuned to balance the contribution of each modality in the fused feature vector, with the optimal value found as:

wfusion=arg⁡minwfusion⁡Lfusion​\(Xtext,Ximage,wfusion\)w\_\{\\text\{fusion\}\}=\\arg\\min\_\{w\_\{\\text\{fusion\}\}\}L\_\{\\text\{fusion\}\}\(X\_\{\\text\{text\}\},X\_\{\\text\{image\}\},w\_\{\\text\{fusion\}\}\)\(14\)The dropout ratedoptd\_\{\\text\{opt\}\}is selected to prevent overfitting:

dopt=arg⁡mind⁡L​\(θ;d\)d\_\{\\text\{opt\}\}=\\arg\\min\_\{d\}L\(\\theta;d\)\(15\)For Late Fusion, the model combines predictions from different modalities, and key hyperparameters include the learning rate \(η\\eta\), fusion weight \(wfusionw\_\{\\text\{fusion\}\}\), and batch size \(bb\)\. The learning rateηopt\\eta\_\{\\text\{opt\}\}is tuned similarly to Early Fusion, while the optimal fusion weightwfusionw\_\{\\text\{fusion\}\}is selected by minimizing:

wfusion=arg⁡minwfusion⁡Lfusion​\(ytext,yimage,wfusion\)w\_\{\\text\{fusion\}\}=\\arg\\min\_\{w\_\{\\text\{fusion\}\}\}L\_\{\\text\{fusion\}\}\(y\_\{\\text\{text\}\},y\_\{\\text\{image\}\},w\_\{\\text\{fusion\}\}\)\(16\)The batch sizeboptb\_\{\\text\{opt\}\}is optimized to ensure efficient training:

bopt=arg⁡minb⁡L​\(θ;b\)b\_\{\\text\{opt\}\}=\\arg\\min\_\{b\}L\(\\theta;b\)\(17\)For Intermediate Fusion, which combines features after extraction but before final decision\-making, the learning rate \(η\\eta\), number of layers \(LL\), fusion weight \(wfusionw\_\{\\text\{fusion\}\}\), and dropout rate \(dd\) are crucial\. The learning rateηopt\\eta\_\{\\text\{opt\}\}and dropout ratedoptd\_\{\\text\{opt\}\}are optimized in the same way as in Early Fusion\. The number of layersLoptL\_\{\\text\{opt\}\}is determined by minimizing the loss function:

Lopt=arg⁡minL⁡L​\(θ;L\)L\_\{\\text\{opt\}\}=\\arg\\min\_\{L\}L\(\\theta;L\)\(18\)The fusion weightwfusionw\_\{\\text\{fusion\}\}is optimized similarly to Early Fusion:

wfusion=arg⁡minwfusion⁡Lfusion​\(Xtext,Ximage,wfusion\)w\_\{\\text\{fusion\}\}=\\arg\\min\_\{w\_\{\\text\{fusion\}\}\}L\_\{\\text\{fusion\}\}\(X\_\{\\text\{text\}\},X\_\{\\text\{image\}\},w\_\{\\text\{fusion\}\}\)\(19\)Bayesian Optimization is employed to efficiently search for the optimal hyperparameters by modeling the loss function and predicting the best combination of hyperparameters\. The overall optimization problem for all fusion methods can be formalized as:

θopt,ηopt,wfusion,\\displaystyle\\theta\_\{\\text\{opt\}\},\\eta\_\{\\text\{opt\}\},w\_\{\\text\{fusion\}\},dopt,bopt,Lopt\\displaystyle\\ d\_\{\\text\{opt\}\},b\_\{\\text\{opt\}\},L\_\{\\text\{opt\}\}\(20\)=arg⁡minθ,η,w,d,b,L\\displaystyle=\\arg\\min\_\{\\theta,\\eta,w,d,b,L\}L​\(θ;η,w,d,b,L\)\\displaystyle\\ L\(\\theta;\\eta,w,d,b,L\)
whereLLrepresents the loss function\. This approach ensures that the optimal hyperparameters are selected to minimize the overall loss across different fusion strategies\.

Step 6\) Evaluation of Meme Relevant Score Detection:Evaluation involves assessing model performance in detecting explanatory evidence within memes using metrics such as accuracy, precision, recall, and F1\-score\. Precision measures the proportion of true positives among predicted positives, while recall evaluates the proportion of true positives among actual positives\. The F1\-score is crucial as it balances precision and recall into a single metric, which is particularly useful for handling imbalanced datasets where certain relevance scores are less frequent\. This task includes classifying evidence into relevance scores: 0 for “Not relevant,” 1 for “Partially relevant,” and 2 for “Relevant,” making it challenging due to the imbalanced distribution of these scores\.

## 5Result Analysis

Table 2:Performance Metrics of Various Fusion Approaches for Relevance Score Detection in Bengali Memes\.Table[2](https://arxiv.org/html/2607.03981#S5.T2)present a comparative analysis of early and late fusion approaches for relevance score detection in Bengali memes, using F1\-score as the primary evaluation metric\. Among the early fusion models, Swin\-m achieves the highest F1\-score \(0\.71\), while other models exhibit moderate performance\. However, the late fusion method outperforms both strategies, with our proposed model,BengaliMemeEvidenceNet, achieving the highest F1\-score of 0\.74, surpassing all other architectures\. This result underscores the effectiveness of BengaliMemeEvidenceNet in capturing multimodal interactions, demonstrating its superiority in Bengali meme relevance detection over traditional fusion techniques\.

## 6Limitations

Although our approach, BengaliMemeEvidenceNet, empirically outperforms several competitive baselines, we observe certain limitations in the modeling capacity towards BanglaMemeEvidence\. As depicted in Table 6, there are three possible scenarios of ineffective detection: \(a\) no predictions, \(b\) partial match, and \(c\) incorrect predictions\. The key challenges stem from the limitations in modeling the complex level of abstractions that a meme exhibits\. These are primarily encountered in the following scenarios:

- ∘\\circIntegration of Visual and Factual Knowledge:A critical yet cryptic piece of information within memes often comes from the visuals, which typically require systematic integration of factual knowledge\. BengaliMemeEvidenceNet currently lacks this capability, making it difficult to accurately interpret and explain the visual elements in memes\.
- ∘\\circLexical Bias and Spurious Evidence:The model is prone to picking up potentially spurious pieces of evidence due to lexical biasing within the related context\. This can lead to incorrect predictions, as the model may focus on irrelevant or misleading textual features rather than the intended meaning of the meme\.
- ∘\\circComplex Abstractions:Memes often exhibit a complex level of abstraction that is challenging for the model to capture\. This includes subtle humor, cultural references, and context\-specific nuances that require advanced reasoning capabilities beyond the current scope of BengaliMemeEvidenceNet\.
- ∘\\circInadequate Contextual Understanding:The model’s current contextual understanding is often inadequate for fully grasping the meaning conveyed by memes\. This limitation is particularly evident when memes rely on intricate social or cultural contexts that the model has not been trained to recognize\.

## 7Future Works

There are several areas where further research could significantly enhance the robustness and applicability of our models\. Below, we outline future work that will build on our current findings:

- ∘\\circDialect\-Based Meme Annotation:We will explore dialect\-based meme annotation and subsequent performance analysis to account for variations in the Bengali language\. This approach will improve model accuracy and effectiveness in diverse linguistic contexts, ensuring that regional dialects and linguistic variations are accurately represented and understood\.
- ∘\\circBanglish Meme Detection:We aim to develop capabilities for detecting Banglish memes, which involve a mixture of Bengali and English, particularly prevalent in spoken communication by Bangladeshis\. We will create methods to handle language switching within sentences or conversations, enhancing the model’s ability to accurately interpret and explain Banglish memes\.
- ∘\\circFine\-Grained Semantic Role Analysis:We will conduct a fine\-grained analysis of the semantic roles in memes to explore nuances in connotation and interpretation\. We will investigate how factors such as image composition, text placement, and meme format influence the perceived roles of entities like heroes, villains, and victims\. This detailed analysis will provide deeper insights into how different elements of a meme contribute to its overall message\.
- ∘\\circApplications Beyond Memes:We will investigate how the concept of visual semantic role labeling and natural language explanation generation can be applied to other forms of visual communication, such as advertisements, political cartoons, and social media posts\. Expanding the application of these techniques will provide valuable tools for analyzing a wide range of visual media, enhancing our understanding of multimodal communication across different contexts\.

## 8Conclusion

We have introduced and explored the hybrid task of meme evidence detection, emphasizing the need for an in\-depth understanding of the visual and textual semantics embedded in memes\. By presenting BanglaMemeEvidence, a unique dataset of 2,917 annotated Bengali memes, we address a significant gap in meme analysis for low\-resource languages\. Our dataset includes rich annotations and relevance scores, providing a comprehensive resource for further studies in this domain\. To effectively detect explanatory evidence within memes, we developed BengaliMemeEvidenceNet, a hybrid approach that integrates textual and visual features\. This pioneering work not only sets a benchmark in the field of meme analysis for Bengali but also opens avenues for future research in other low\-resource languages\.

## References

- \[1\]M\. Ahmed, N\. Akter, M\. Rahman, A\. Islam, D\. Das, and Md\. G\. Rashed\(2023\-08\)Multimodal cyberbullying meme detection from social media using deep learning approach\.International Journal of Computer Science and Information Technology15,pp\. 27–37\.External Links:[Document](https://dx.doi.org/10.5121/ijcsit.2023.15403)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.03981#S2.SS1.p1.1)\.
- \[2\]I\. Albakkosh\(2024\-01\)Using fleiss’ kappa coefficient to measure the intra and inter\-rater reliability of three ai software programs in the assessment of efl learners’ story writing\.International Journal of Educational Sciences and Arts3\(1\),pp\. 69–96\.External Links:ISSN 2976\-7237,[Link](http://dx.doi.org/10.59992/ijesa.2023.v3n1p4),[Document](https://dx.doi.org/10.59992/ijesa.2023.v3n1p4)Cited by:[§3\.6](https://arxiv.org/html/2607.03981#S3.SS6.p1.1)\.
- \[3\]N\. Alluri and N\. Krishna\(2021\-11\)Multi modal analysis of memes for sentiment extraction\.pp\. 213–217\.External Links:[Document](https://dx.doi.org/10.1109/ICIIP53038.2021.9702696)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.03981#S2.SS1.p1.1)\.
- \[4\]G\. Barnum, S\. Talukder, and Y\. Yue\(2020\)On the benefits of early fusion in multimodal representation learning\.External Links:2011\.07191,[Link](https://arxiv.org/abs/2011.07191)Cited by:[§4](https://arxiv.org/html/2607.03981#S4.p7.1)\.
- \[5\]D\. Chauhan, D\. R, A\. Ekbal, and P\. Bhattacharyya\(2020\-01\)Sentiment and emotion help sarcasm? a multi\-task learning framework for multi\-modal sarcasm, sentiment and emotion analysis\.pp\. 4351–4360\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.401)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1)\.
- \[6\]A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov\(2020\)Unsupervised cross\-lingual representation learning at scale\.External Links:1911\.02116,[Link](https://arxiv.org/abs/1911.02116)Cited by:[1st item](https://arxiv.org/html/2607.03981#S4.I3.i1.p1.1)\.
- \[7\]M\. Das and A\. Mukherjee\(2023\)BanglaAbuseMeme: a dataset for bengali abusive meme classification\.External Links:2310\.11748,[Link](https://arxiv.org/abs/2310.11748)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.03981#S2.SS1.p1.1)\.
- \[8\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2018\)BERT: pre\-training of deep bidirectional transformers for language understanding\.CoRRabs/1810\.04805\.External Links:[Link](http://arxiv.org/abs/1810.04805),1810\.04805Cited by:[1st item](https://arxiv.org/html/2607.03981#S4.I3.i1.p1.1)\.
- \[9\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.External Links:2010\.11929,[Link](https://arxiv.org/abs/2010.11929)Cited by:[2nd item](https://arxiv.org/html/2607.03981#S4.I3.i2.p1.1)\.
- \[10\]M\. Hasan, L\. Islam, J\. F\. Ruma, T\. T\. Mayeesha, and R\. M\. Rahman\(2023\)Visual question generation in bengali\.arXiv preprint arXiv:2310\.08187\.Cited by:[§2\.2](https://arxiv.org/html/2607.03981#S2.SS2.p1.1)\.
- \[11\]S\. S\. Islam, R\. A\. Auntor, M\. Islam, M\. Y\. H\. Anik, A\. A\. A\. Islam, and J\. Noor\(2022\)Note: towards devising an efficient vqa in the bengali language\.InProceedings of the 5th ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies,pp\. 632–637\.Cited by:[§2\.2](https://arxiv.org/html/2607.03981#S2.SS2.p1.1)\.
- \[12\]Md\. R\. Karim, S\. K\. Dey, T\. Islam, Md\. Shajalal, and B\. R\. Chakravarthi\(2022\)Multimodal hate speech detection from bengali memes and texts\.External Links:2204\.10196,[Link](https://arxiv.org/abs/2204.10196)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.03981#S2.SS1.p1.1)\.
- \[13\]A\. Kumar and G\. Garg\(2019\-01\)Sarc\-m: sarcasm detection in typo\-graphic memes\.SSRN Electronic Journal,pp\.\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.3384025)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1)\.
- \[14\]Z\. Liu, Y\. Lin, Y\. Cao, H\. Hu, Y\. Wei, Z\. Zhang, S\. Lin, and B\. Guo\(2021\)Swin transformer: hierarchical vision transformer using shifted windows\.In2021 IEEE/CVF International Conference on Computer Vision \(ICCV\),Vol\.,pp\. 9992–10002\.External Links:[Document](https://dx.doi.org/10.1109/ICCV48922.2021.00986)Cited by:[2nd item](https://arxiv.org/html/2607.03981#S4.I3.i2.p1.1)\.
- \[15\]R\. Nayak, U\. Kannantha, K\. S, and C\. Gururaj\(2022\-02\)Multimodal offensive meme classification using transformers and bilstm\.International Journal of Engineering and Advanced Technology11,pp\. 96–102\.External Links:[Document](https://dx.doi.org/10.35940/ijeat.C3392.0211322)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p2.1)\.
- \[16\]Y\. R\. Pandeya and J\. Lee\(2020\-09\)Deep learning\-based late fusion of multimodal information for emotion classification of music video\.Multimedia Tools and Applications80\(2\),pp\. 2887–2905\.External Links:ISSN 1573\-7721,[Link](http://dx.doi.org/10.1007/s11042-020-08836-3),[Document](https://dx.doi.org/10.1007/s11042-020-08836-3)Cited by:[§4](https://arxiv.org/html/2607.03981#S4.p7.1)\.
- \[17\]M\. H\. Rafi, S\. Islam, S\. H\. I\. Labib, S\. S\. Hasan, F\. M\. Shah, and S\. Ahmed\(2022\)A deep learning\-based bengali visual question answering system\.In2022 25th International Conference on Computer and Information Technology \(ICCIT\),pp\. 114–119\.Cited by:[§2\.2](https://arxiv.org/html/2607.03981#S2.SS2.p1.1)\.
- \[18\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2020\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.External Links:1910\.01108,[Link](https://arxiv.org/abs/1910.01108)Cited by:[1st item](https://arxiv.org/html/2607.03981#S4.I3.i1.p1.1)\.
- \[19\]A\. Shaker, M\. Maaz, H\. Rasheed, S\. Khan, M\. Yang, and F\. S\. Khan\(2023\)SwiftFormer: efficient additive attention for transformer\-based real\-time mobile vision applications\.External Links:2303\.15446,[Link](https://arxiv.org/abs/2303.15446)Cited by:[2nd item](https://arxiv.org/html/2607.03981#S4.I3.i2.p1.1)\.
- \[20\]S\. Sharma, S\. Agarwal, T\. Suresh, P\. Nakov, Md\. S\. Akhtar, and T\. Chakraborty\(2022\)What do you meme? generating explanations for visual semantic role labelling in memes\.External Links:2212\.00715,[Link](https://arxiv.org/abs/2212.00715)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p1.1)\.
- \[21\]S\. Sharma, R\. S, U\. Arora, Md\. S\. Akhtar, and T\. Chakraborty\(2023\)MEMEX: detecting explanatory evidence for memes via knowledge\-enriched contextualization\.External Links:2305\.15913,[Link](https://arxiv.org/abs/2305.15913)Cited by:[§1](https://arxiv.org/html/2607.03981#S1.p1.1)\.
- \[22\]W\. Yu, M\. Luo, P\. Zhou, C\. Si, Y\. Zhou, X\. Wang, J\. Feng, and S\. Yan\(2022\)MetaFormer is actually what you need for vision\.External Links:2111\.11418,[Link](https://arxiv.org/abs/2111.11418)Cited by:[2nd item](https://arxiv.org/html/2607.03981#S4.I3.i2.p1.1)\.