Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges

# Methods, Datasets, Evaluation, and Challenges
Source: [https://arxiv.org/html/2607.19011](https://arxiv.org/html/2607.19011)
1\]Case Western Reserve University 2\]The Hong Kong Polytechnic University

## Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges

###### Abstract

Multimodal humor in memes, cartoons, and comics remains difficult for AI systems because intended meaning depends on non\-literal mechanisms, shared cultural knowledge, and communicative intent rather than literal scene description\. This survey focuses on visual humor understanding in single\-image and multi\-panel artifacts, while treating humor generation as an emerging downstream frontier\. We position the literature against prior humor, sarcasm, and general MLLM surveys and organize it using a capability\-centric hierarchy spanning recognition, interpretation and reasoning, and generation\. Under this lens, we synthesize benchmark design, evaluation protocols, and modeling paradigms, tracing the field’s shift from task\-specific fusion models to large\-model approaches based on multimodal alignment, evidence\-grounded reasoning, and controlled generation\. We conclude by highlighting the main barriers to progress: shortcut\-prone evaluation, limited cultural and narrative coverage, weak evidence grounding, and unresolved safety and ownership concerns\.

11footnotetext:These authors contributed equally\.22footnotetext:Corresponding author\.## 1Introduction

Recent advances in AI have enabled models to jointly process text and images with unprecedented scale and performance\. Multimodal Large Language Models \(MLLMs\) now achieve strong results on tasks such as image captioning, visual question answering, and cross\-modal retrievalyin2024survey;comanici2025gemini;bai2025qwen3vltechnicalreport, which primarily emphasize the recognition, grounding, and reasoning of explicit visual and textual content\. However, this paradigm remains limited when handling content whose meaning extends beyond literal perceptionhwang2023memecap;chen2024we;nayak2024benchmarking;xu2025visulogic\. Human communication frequently relies on expressive artifacts such as memes, comics, cartoons, and satirical images, where meaning arises from humor, cultural reference, symbolic association, or intentional incongruity between modalities\. Interpreting such media requires reasoning about implicit meaning and communicative intent rather than merely recognizing observable content\.

We refer to this problem setting asmultimodal humor understanding: interpreting image\-based artifacts whose humorous, satirical, or ironic meaning emerges from the interaction of visual content and associated text\. We focus on single\-image and multi\-panel image artifacts, and we treat humor generation as an emerging downstream frontier that is useful for probing whether models have internalized the mechanisms of humorous interpretation, rather than as an independent capability to be evaluated in isolation\.

Why this gap matters\.The gap between perceptual recognition and communicative interpretation is not a minor residual error; it is a structural blind spot in how MLLMs are built and evaluated\. A model can correctly identify a burning house, a courtroom, or a smiling face, and still fail to recover the metaphor, the social target, or the punchline that those elements jointly encode\. This is because the core difficulty is not multimodal alignment in the usual sense, but reasoning about why a juxtaposition is funny, what background knowledge it presupposes, and what stance it communicates to an audiencesaakyan2024v;ryan2025humor\. As MLLMs are increasingly deployed in applications such as content moderation, social media analysis, and creative assistance, systematically understanding their capabilities, limitations, and failure modes in these tasks becomes increasingly urgent\.

The gap in existing surveys\.Despite growing interest, existing surveys only cover this space partially\. Surveys of text\-based computational humoramin2020survey;kalloniatis2024computational;loakman2025s;lemmens2026computationalcenter on linguistic phenomena such as puns, wordplay, and verbal irony, but do not address visual grounding or cross\-modal incongruity\. Multimodal surveys exist, but each narrows to a single task: sarcasm detectionfarabi2024survey;gao2025spokenor meme classificationafridi2020multimodal;ren2026survey\. Broader MLLM surveysyin2024survey, meanwhile, typically subsume creative and figurative understanding as one capability under general multimodal reasoning, without dedicated systematic treatment\.

This survey is designed to fill the space\. Rather than grouping work by task name or model architecture, we organize the literature around three progressively demanding capabilities:*recognition*, where systems detect or classify humorous phenomena;*interpretation and reasoning*, where systems explain the mechanism, target, or implicit meaning; and*generation*, where models must produce humor\-consistent outputs grounded in the input\. This framing clarifies not only what different benchmarks measure, but also which modeling assumptions are needed to move from label prediction to evidence\-grounded interpretation\.

Concretely, this survey makes the following contributions:

- •We propose a capability\-centric organization that aligns background concepts, benchmark design, and modeling paradigms\.
- •We synthesize datasets and evaluation protocols with particular attention to what current benchmarks do and do not measure about interpretive understanding\.
- •We identify the technical and socio\-technical bottlenecks that currently limit progress, including shortcut\-prone evaluation, sparse mechanism\-level annotation, weak cultural grounding, and safety and ownership concerns\.

Survey scope\.We include Multimodal Visual Humor, such as*multimodal humor*,*meme understanding*,*visual sarcasm*,*satire detection*,*comic understanding*,*humorous captioning*, and*visual figurative language*\. We focus on contemporary MLLMs work, and briefly introduce task\-specific multimodal models before the MLLM era in Appendix[C](https://arxiv.org/html/2607.19011#A3)We include papers that study image\-based artifacts and where success depends on reasoning beyond literal grounding, typically through visual–text interaction, non\-literal mechanisms, or communicative intent\. We exclude audio\- and video\-centric humor, generic aesthetic modeling, and persuasive media unless they are directly used as evaluation resources for image\-based humor understanding\. We additionally used backward and forward snowballing from benchmark and survey papers to recover closely related work\. Our goal is not an exhaustive catalog of every humor\-adjacent dataset, but a principled synthesis of the resources and modeling strategies that most directly test multimodal visual humor understanding\.

![Refer to caption](https://arxiv.org/html/2607.19011v1/imgs/overview.png)Figure 1:Overview of the survey\.We define*multimodal humor*and its two*representation forms*: StaVT \(single image \+ short text\) and SeqVN \(multi\-panel sequence\)\. Three progressively demanding*capabilities*— Recognition, Interpretation & Reasoning, and Generation—form the columns \(§[3](https://arxiv.org/html/2607.19011#S3)\)\. Row 3 gives the dominant*modeling paradigm*for each capability: cross\-modal alignmentpθ​\(y∣x\)p\_\{\\theta\}\(y\\mid x\), grounded reasoningpθ​\(y,r1:K∣x,𝒦x\)p\_\{\\theta\}\(y,r\_\{1:K\}\\mid x,\\mathcal\{K\}\_\{x\}\), and controllable generationpθ​\(ygen∣x,c\)p\_\{\\theta\}\(y\_\{\\mathrm\{gen\}\}\\mid x,c\)\(§[4](https://arxiv.org/html/2607.19011#S4), Table[1](https://arxiv.org/html/2607.19011#S4.T1)\)\. Row 4 lists the*benchmark family and evaluation protocol*, tagged by representation form \(§[5](https://arxiv.org/html/2607.19011#S5), Table[2](https://arxiv.org/html/2607.19011#S5.T2)\); Row 5 \(accented\) highlights the*persistent gaps*exposed by our cross\-benchmark analysis \(§[6](https://arxiv.org/html/2607.19011#S6), §[7](https://arxiv.org/html/2607.19011#S7)\)\. Horizontal arrows mark capability progression; vertical arrows trace each column top\-to\-bottom from task to open problem\.
## 2Background

### 2\.1Definition of Multimodal Humor

In this survey, we usemultimodal humorto refer to communicative artifacts that intentionally convey meaning beyond literal depiction through the coordinated use of visual content and associated text, such as internet memes, editorial cartoons, comic strips, and satirical images\.

Unlike natural images or descriptive text, human creative media are designed to communicate expressive, humorous, narrative, or satirical meaning, requiring interpretation of what is implied rather than what is directly shown\. Their interpretation relies not only on perceptual recognition but also on inference of implicit intent and shared social knowledge\. We focus on single\-image and multi\-panel image artifacts where meaning emerges from visual–text interaction and from non\-literal reasoning\.

### 2\.2Representation Forms

Multimodal humor spans a wide range of image–text data forms, including single images with captions and multi\-panel sequences with dialogue\. Modeling multimodal creative understanding can be formulated as learning a conditional mapping:p​\(y\|x\)p\(y\|x\), wherexxdenotes multimodal inputs andyyrepresents a task\-specific output, such as a humor label, explanation, or generated continuation\. The structure ofxxdetermines the perceptual encoding, alignment mechanisms, and architectural components required to model this conditional distribution\.

Unlike conventional vision or language tasks, creative media often distribute meaning across modalities and \(for comics\) across panels\. Therefore, representation form defines not only the input space but also the model’s required integration capacity\. We categorize representation forms into two modeling\-oriented classes:

![Refer to caption](https://arxiv.org/html/2607.19011v1/x1.png)Figure 2:We categorize visual humor data into Static Visual–Textual Artifacts \(StaVT\), such as comicshessel2023androids, memes, and humorous imagesjain2025humordb, and Sequential Visual Narratives \(SeqVN\), such as cartoon stripsliang2025yes, mangaikuta2025mangaub, and multi\-panel memes\. Common annotations include captions, image text, questions, object bounding boxes, humor or intensity scores, and task\-specific labels\.\(1\) Static Visual–Textual Artifacts \(StaVT\)include memes, editorial cartoons, and captioned images, wherex=\{xi​m​g,xt​e​x​t\}x=\\\{x^\{img\},x^\{text\}\\\}consists of a single image optionally paired with short text\. Meaning frequently arises from cross\-modal incongruity or implicit symbolic alignment\. Modeling requires joint embedding spaces, multimodal alignment, and sensitivity to social knowledge priorsshifman2013memes;sharma2020semeval\. Architectures typically rely on vision–language encoders or MLLMs with cross\-attention mechanisms\.

\(2\) Sequential Visual Narratives \(SeqVN\)encompass comic strips and multi\-panel memes, where meaning emerges from progression across panels, withx=\{x1,x2,…,xT\}x=\\\{x\_\{1\},x\_\{2\},\\ldots,x\_\{T\}\\\}representing an ordered visual sequence\. Understanding these forms requires tracking entities and events, inferring causal relations, and integrating information across a visual sequencepaval\-etal\-2025\-comicscene154;wang\-etal\-2025\-beyond\-single\.

Across all categories, data form shapes not only perceptual requirements but also the types of reasoning and alignment needed for understanding and generating creative meaning\.

### 2\.3Creative Meaning Construction

Beyond representation form, creative artifacts convey meaning through non\-literal mechanisms\. For AI systems, interpretation therefore requires more than recognizing visual and textual patterns: models must also infer why those patterns are usedhu2023language\. In this survey, we treat creative meaning construction as the interaction between recurrent expressive mechanisms\.

Across data forms, four mechanisms recur\.Incongruitycreates meaning through a mismatch between expectation and observation, often through cross\-modal conflict in multimodal humorforabosco1992cognitive;veale2004incongruity;schifanella2016detecting;farabi2024survey\.Analogy and Conceptual Mappingproject structure from a familiar source domain onto a target concept, as in metaphor and symbolic representationlakoff2024metaphors;refaie2003understanding;foss2004theory\.Exaggeration and Hyperboleamplify attributes against implicit norms for emphasis or affectkreuz1996figurative;zhang2024image, whileNarrative Structureorganizes setup, payoff, and causal progression over timegenette1980narrative;bruner1991narrative;paval\-etal\-2025\-comicscene154\. Together, these mechanisms explain how creative artifacts encode meaning beyond surface semantics and why they remain challenging for literalist AI systems\.

### 2\.4Why Multimodal Humor Is Hard for AI

Multimodal humor is fundamentally challenging for AI models because the intended meaning often cannot be directly inferred from observable inputs\. Unlike literal multimodal tasks where answers are grounded in explicit visual or textual evidence, creative understanding requires reasoning over latent variables such as implied metaphors, violated expectations, cultural references, and communicative intent\. Models must detect incongruity, infer implicit norms, integrate external socio\-cultural knowledge, and reason about why an artifact was produced and how it is meant to be interpreted\. As a result, creative understanding goes beyond multimodal feature fusion, demanding the integration of perceptual alignment, structured reasoning, and pragmatic inference within a unified modeling framework\.

![Refer to caption](https://arxiv.org/html/2607.19011v1/x2.png)Figure 3:Capability\-centric task hierarchy for multimodal humor\. We organize tasks into three levels—Recognition, Interpretation and Reasoning, and Generation—reflecting increasing requirements for non\-literal grounding, mechanism\-aware inference, and evaluation beyond discriminative labels\.

## 3Task Hierarchy: Recognition, Interpretation, and Generation

We organize existing work into three capability levels and pair each with the evaluation signal it most directly demands \(Figure[3](https://arxiv.org/html/2607.19011#S2.F3)\)\. The hierarchy is not purely chronological: recent benchmarks often mix levels, but the distinction is useful because gains in label prediction do not automatically transfer to explanation or generation\.

### 3\.1Level 1: Recognition

Recognition tasks ask whether a multimodal input contains a humorous, sarcastic effect and, in more fine\-grained settings, which components instantiate it\. Typical problems include binary or multi\-class classification, target or role extraction, and intensity estimation\. These tasks dominate early work on multimodal sarcasm, meme classification, and visual humor detectioncai2019multi;hasan2021humor;zhang2024image\. They require reliable visual\-text alignment and sensitivity to local cues, but they do not by themselves verify whether a model has captured the mechanism that makes an artifact funny or critical\.

Recognition tasks are usually evaluated with discriminative metrics such as accuracy, F1, AUROC, or correlation with human ratings\. These metrics are appropriate for testing cue sensitivity and class separation, yet they remain weak proxies for genuine understanding: a model can predict a correct label by exploiting recurrent surface patterns without being able to explain the intended meaning\.

### 3\.2Level 2: Interpretation and Reasoning

Interpretation tasks ask models to explain*why*an artifact is humorous, satirical, or ironic\. A useful distinction is betweendescriptive explanation, which verbalizes salient cues or paraphrases the joke, andmechanism\-grounded reasoning, which identifies the specific conflict, analogy, target, or narrative step that produces the effecthu2024cracking;saakyan2024v;wang2024mementos\. These tasks require explicit cross\-modal grounding, abstraction over non\-literal meaning, and often external knowledge or context to resolve implicit references\. For multi\-panel inputs, they also require temporal and narrative reasoning across panels rather than within a single imagepaval\-etal\-2025\-comicscene154;wang\-etal\-2025\-beyond\-single\.

Evaluation at this level must combine language quality with interpretive faithfulness\. Automatic metrics such as BLEU or BERTScorepapineni2002bleu;zhang2019bertscoreare useful for checking lexical or semantic overlap, but they remain insufficient when several explanations are plausible\. Stronger protocols ask whether a model identifies the relevant cues, names the right mechanism, and stays consistent with available evidence, often through rubric\-based human judgment or LLM\-assisted evaluationliu2023g;hu2024cracking\.

### 3\.3Level 3: Generation

Generation tasks require models to produce humor\-consistent outputs such as captions, punchlines, explanations, or comic continuations conditioned on an input artifact\. In this survey, we treat generation as an emerging downstream frontier rather than the core of the field: successful generation presupposes at least partial understanding of incongruity, target selection, tone, and narrative setuphwang2023memecap;li2023oxfordtvg;tanaka\-etal\-2024\-content\. The challenge is therefore not only to produce fluent text, but also to maintain faithfulness to the source image and control over the mechanism being realized\.

Because valid outputs are diverse, evaluation at this level relies heavily on human preference judgments or rubric\-based assessment\. Reference\-based metrics remain weak proxies for humor quality and novelty, so the most informative benchmarks combine generation quality with tests of faithfulness to the source image, intended target, and rhetorical device\.

## 4Modeling Paradigms in the Large\-Model Era

With the rise of MLLMs, multimodal humor modeling has shifted from handcrafted fusion toward alignment\-driven representation learning, explicit reasoning, and evidence\-grounded generation\. We organize the modeling literature by thecapabilityit primarily supports rather than by model family, because the same backbone behaves very differently when optimized forrecognition,interpretation, orcontrolled generation\. Table[1](https://arxiv.org/html/2607.19011#S4.T1)summarizes the three resulting paradigms along their input formulation, technical core, and characteristic failure modes; the strongest recent systems combine them rather than treating them as substitutes\. We detail each paradigm below and close with a cross\-paradigm analysis\.

Table 1:Modeling paradigms for multimodal humor in the large\-model era\.x=\(xv,xt\)x=\(x^\{v\},x^\{t\}\)denotes the image–text input;yydenotes a recognition or interpretation output;r1:Kr\_\{1:K\}denotes an intermediate rationale chain;𝒦x\\mathcal\{K\}\_\{x\}denotes input\-specific external knowledge retrieved or selected forxx; andccdenotes a creative control signal\.### 4\.1Recognition\-Oriented Multimodal Alignment

Recognition\-oriented systems learn a joint encoderfθf\_\{\\theta\}that maps visual and textual inputs into a shared semantic space and minimizes a task lossℒ=𝔼​\[ℓ​\(gϕ​\(fθ​\(xv,xt\)\),y\)\]\\mathcal\{L\}=\\mathbb\{E\}\[\\ell\(g\_\{\\phi\}\(f\_\{\\theta\}\(x^\{v\},x^\{t\}\)\),\\;y\)\]\. The critical design choices are \(i\)*how*fθf\_\{\\theta\}is trained—via contrastive pre\-training, instruction tuning, or modular routing—and \(ii\)*at what granularity*visual features are extracted\. Three strategies span this design space\.

Creativity\-oriented instruction tuningadapts a pre\-trained MLLM with creativity\-specific\(xv,xt,instruction,y\)\(x^\{v\},x^\{t\},\\text\{instruction\},y\)supervision\. The principle is that humor, sarcasm, and meme communication have statistical regularities absent from generic VL corpora, so domain\-specific fine\-tuning consistently outperforms general\-purpose checkpoints on recognition benchmarkszhang2024somelvlm\. A contrastive alternative optimizes an InfoNCE loss over image–text embeddings, exposing cross\-modal correspondences for multi\-task meme classificationshah2024memeclip\. Across both strategies, gains are most pronounced on surface\-level labels; performance on nuanced intent lags behind, suggesting that alignment alone captures*what co\-occurs*but not*why*\.

Modular and text\-centric designsdecouple perception from reasoning\. Mixture\-of\-expert routing assigns vision and language tokens to specialized sub\-networks, improving robustness when humor depends on only one modalityyu2024mmoe\. A more radical approach converts all non\-textual modalities into text via captioning, reducing multimodal fusion to unified self\-attentionhasan2023textmi;baluja2025text\. This text\-centric strategy is surprisingly effective for humor understanding—even without architectural changes to the LLM—but incurs information loss on visually dense inputs where spatial layout or fine\-grained detail carries the joke\.

Multi\-panel and region\-aware alignmentaddresses sequential visual narratives \(x=\{x1v,…,xTv\}x=\\\{x\_\{1\}^\{v\},\\ldots,x\_\{T\}^\{v\}\\\}\), where meaning emerges across panels\. The key technical challenge is to capture both intra\-panel content and inter\-panel relations \(entity co\-reference, causal flow, setup–punchline structure\)\. Current approaches range from multi\-panel instruction tuningliang2025yes, to panel\-selection tasks that test narrative coherencevivoli2025comicspap, to RL\-trained region\-level encoders that attend to character expressions and speech bubbles within each panelchen2025zooming\. These extensions are necessary because entity continuity and visual salience across panels are prerequisites for downstream interpretation\.

Collectively, alignment methods are scalable and strong on classification, but the humor mechanism remains implicit: a model can predict the correct label while failing to recover the violated expectation or social target that drives the humor—a gap confirmed by the recognition\-to\-interpretation drop in Section[6](https://arxiv.org/html/2607.19011#S6)\.

### 4\.2Interpretation\-Oriented Reasoning and Grounding

![Refer to caption](https://arxiv.org/html/2607.19011v1/x3.png)Figure 4:\(a\) Recognition\-oriented multimodal alignment maps visual and textual inputs into a shared representation for scalable humor detection and classification, but leaves the underlying humor mechanism implicit\. \(b\) Interpretation\-oriented reasoning and grounding introduces intermediate rationales and optional external knowledge to explain why an artifact is humorous\. \(c\) Generation and creative control conditions humorous output generation on control signals such as mechanism, target, tone, and style, with evaluation and safety constraints\. \(d\) Cross\-paradigm integration combines alignment, reasoning, retrieval, and generation into a unified framework, highlighting open questions about alignment–reasoning interaction, retrieval timing, and generation as a diagnostic of understanding\.Once tasks demand explanation, alignment alone is insufficient because it captures*what co\-occurs*but not*why it is funny*\. Interpretation\-oriented methods address this by conditioning the prediction on a chain of intermediate reasoning steps:p​\(y∣x\)≈∏k=1Kp​\(rk∣r<k,x\)⋅p​\(y∣r1:K,x\)p\(y\\mid x\)\\approx\\prod\_\{k=1\}^\{K\}p\(r\_\{k\}\\mid r\_\{<k\},x\)\\cdot p\(y\\mid r\_\{1:K\},x\), where eachrkr\_\{k\}is a natural\-language rationale that exposes part of the humor mechanism\. The design space varies along two axes: \(i\)*how the reasoning trace is obtained*—prompted at inference, learned from human annotations, or imposed by theory—and \(ii\)*where missing knowledge comes from*—the model’s own parameters or an external retriever\.

Prompted vs\. supervised reasoning\.The cheapest approach elicits chain\-of\-thought reasoning at inference time via carefully constructed promptswei2022chain\. Prompting models to first describe each panel and then articulate the conflict yields clear gains on multi\-panel humor, where the contradiction is cross\-panel rather than within a single framehu2024cracking; similar staged prompts decompose conversational jokes into setup, incongruity, and resolutionchen2024talk\. However, prompted reasoning is inherently unstable: output quality varies with prompt phrasing, and models can hallucinate plausible\-sounding but factually wrong rationales\. Training\-time rationale supervision is more robust\. When models are jointly trained to generate the reasoning chain*and*the label—i\.e\.,p​\(y,r1:K∣x\)p\(y,r\_\{1:K\}\\mid x\)—both accuracy and interpretability improve substantially over pattern\-based baselines, as demonstrated at scale on harmful\-meme datasetsgu2025mememind\. An alternative is to apply an information bottleneck objective that compresses joke representations to retain only mechanism\-relevant features before explanationhwang2025bottlehumor\. The practical trade\-off is clear: prompted CoT is zero\-cost but fragile; rationale supervision is strong but requires expensive human annotation and risks overfitting to annotator phrasing\.

Theory\-guided decomposition\.A deeper commitment to structure anchors the reasoning pipeline in established humor theories, imposing fixed stages rather than free\-form chains\. The dominant template follows the incongruity\-resolution model: \(i\)*setup extraction*—identify the expected scenario; \(ii\)*conflict detection*—localize the violated expectation; \(iii\)*resolution*—explain how the conflict produces humortikhonov2024humor;zhang2025humorchain\. This staged design generalizes better than single\-pass prediction because each stage is independently evaluable, and theory labels \(incongruity, superiority, relief\) can steer the decomposition\. The same principle of explicit intermediate targets appears in multimodal QA frameworks that chain evidence retrieval with step\-by\-step reasoningagarwal2024mememqa, in figurative\-language benchmarks that frame understanding as natural\-language inference over non\-literal hypothesessaakyan2024v;saakyan2025understanding, and in probes showing that visual metaphor and symbolic\-graphics comprehension collapses without intermediate decompositionkundu2025looking;qiu2024can\. The limitation is that theory\-guided pipelines can over\-analyze simple humor or introduce reasoning drift when the mechanism does not neatly fit the assumed template\.

External knowledge retrieval and grounding\.When the missing information is social, cultural, or temporal rather than perceptual, reasoning over the input alone is insufficient\. Knowledge\-grounded systems augment the prediction with a retrieval step:p​\(y∣x,𝒦x\)p\(y\\mid x,\\mathcal\{K\}\_\{x\}\), where𝒦x=ℛ​\(x;𝒟\)\\mathcal\{K\}\_\{x\}=\\mathcal\{R\}\(x;\\mathcal\{D\}\)is evidence selected from an external corpus𝒟\\mathcal\{D\}\. Three integration strategies have emerged\.*Dense passage retrieval*encodes the meme as a query and retrieves background documents that explain implicit cultural referencessharma2023memex\.*Knowledge\-graph and commonsense injection*augments feature representations with structured world knowledge, which is especially valuable for detecting offensiveness that cannot be inferred from surface cuesgarg2025just;kumari2025memedetoxnet\.*Feature\-space adaptation*fuses domain\-specific attributes \(e\.g\., brand, persuasion technique\) directly into the visual encoder via lightweight adapters, avoiding the latency of an explicit retrieverjia2023kafa\. A related technique retrieves similar labeled examples at inference time for in\-context learning, enabling domain\-shift robustness without retrainingtang2024leveraging; reasoning\-knowledge distillation from large LLMs into smaller students offers another route to groundinglin2023beneath\. Retrieval is critical for satire, time\-sensitive memes, and culturally loaded references, but it introduces noise: irrelevant evidence can mislead the model and obscure the actual creative mechanism\.

### 4\.3Generation and Creative Control

Generation can be framed as conditional decodingp​\(ygen∣x,c\)p\(y\_\{\\text\{gen\}\}\\mid x,c\), wherexxis the source artifact andccencodes a creative control signal \(intended mechanism, target concept, tone\)\. The central difficulty is that effectiveccpresupposes interpretive competence: generating a funny caption for an image requires understanding*what is funny about*that image\. Current work separates along three design principles\.

Mechanism\-explicit captioningadopts a two\-stage*analyze\-then\-generate*pipeline\. The analysis stage identifies the incongruity, salient objects, or violated expectation; the generation stage decodes a caption conditioned on that analysis\. This architecture—instantiated through incongruity\-resolution CoT promptingtanaka\-etal\-2024\-content, grounded humor captioning benchmarksli2023oxfordtvg, and cascaded describe\-explain\-caption pipelineshwang2023memecap—consistently outperforms direct prompting on image\-specificity\. A practical finding is that decoding\-time interventions such as logit bias and negative sampling further suppress generic outputs, confirming that the bottleneck is not fluency but groundedness\. The cascaded design has its own cost: errors in early stages propagate, making the overall quality sensitive to the weakest link\.

Creative artifact synthesisextends generation beyond text to novel images or multimodal artifacts, typically by separating semantic planning \(LLM\) from visual rendering \(diffusion model\)\. Visual metaphor generation pipelines first extract source and target domains, then compose a concrete scene description, and finally render it—allowing each module to be evaluated and improved independentlychakrabarty2023spy;shahmohammadi2023vipe\. Meme generation adds a stance or safety dimension: unconstrained generation produces more novel outputs but also higher rates of harmful contentwang2024memecraft\. A distinct thread explores “creative leaps”—humor requiring associative jumps beyond logical entailment\. The emerging finding is that moderate\-distance leaps between setup and resolution are rated funniest by human judges; too close is predictable, too far is nonsensicalzhong2024let;wang2024innovative\.

Human\-AI co\-creationaddresses a key limitation of fully autonomous systems: the absence of audience feedback during creative production\. Interactive tools that suggest incongruity\-based options for human refinement show that even novice users produce higher\-quality humor when the system handles mechanism generation while the human curates tone and relevancekariyawasam2024appropriate\. Extending generation to temporal media such as short\-video commentary further raises the bar, requiring both cross\-modal alignment and temporal reasoningouyang2025laugh\.

Generation remains bottlenecked by controllability and evaluation\. Models often regress toward generic captions or reuse familiar templates\. Current metrics—both reference\-based \(BLEU, BERTScore\) and human preference—cannot reliably distinguish genuine novelty from paraphrasing\. Generation is therefore best viewed as a stress test of interpretive competence: without understanding the mechanism, faithful creative output is unlikely\.

### 4\.4Cross\-Paradigm Analysis

The three paradigms are complementary: alignment provides scalable perception, reasoning adds interpretive depth, and retrieval supplies missing context\. The strongest recent systems already combine elements—e\.g\., theory\-guided reasoning over alignment\-tuned featureszhang2025humorchain, CoT supervision with commonsense groundinggu2025mememind, or reasoning decomposition feeding into generationtanaka\-etal\-2024\-content—but integration remains ad hoc rather than principled\.

Three open questions emerge from this landscape\. First,*alignment–reasoning interaction*: does finer\-grained visual alignment \(region\-level, panel\-level\) reduce the burden on downstream reasoning, or does it introduce redundant detail that increases reasoning drift? Second,*retrieval timing*: should evidence be injected before reasoning \(to inform the trace\) or after an initial reasoning pass \(to fill identified gaps\)? Current systems use fixed timing, but adaptive strategies could improve efficiency\. Third,*generation as a diagnostic for understanding*: the field treats generation as a downstream application, yet generation quality could serve as a stronger probe of interpretive competence than MCQ accuracy—if appropriate evaluation protocols existed\. Addressing these questions will require joint benchmarks that evaluate alignment, reasoning, and generation on the same artifacts, a direction we revisit in Section[6](https://arxiv.org/html/2607.19011#S6)\.

## 5Datasets and Benchmarks

Table 2:Capability\-aligned summary of benchmark resources\. Detailed dataset inventories appear in Appendix Tables[4](https://arxiv.org/html/2607.19011#A2.T4)and[5](https://arxiv.org/html/2607.19011#A2.T5)\.In this section, we align benchmarks with the same capability hierarchy used for tasks and models\. This makes it easier to distinguish resources that only test label prediction from those that require explanation, cross\-panel inference, or image\-grounded generation\. Detailed inventories are provided in Appendix Tables[4](https://arxiv.org/html/2607.19011#A2.T4)and[5](https://arxiv.org/html/2607.19011#A2.T5)\.

Recognition Resources\.Recognition benchmarks remain the most abundant and the most standardized\. In the image\-based setting, they cover humorous cartoons, sarcastic image–text pairs, and multi\-panel comics through labels for humor presence, target type, dialogue act, or intensityshahaf2015inside;tang2024leveraging;jain2025humordb;martinek2024comicorda\. These resources are useful for training perception and alignment modules, but they remain classification\-oriented and therefore provide only indirect evidence about whether a model understands the underlying joke, target, or narrative conflict\.

Interpretation and Reasoning Resources\.More recent benchmarks explicitly probe interpretive depth by pairing image\-based inputs with explanations, rationales, multiple\-choice reasoning questions, or cross\-panel inference targets\. Datasets such asDo Androids Laugh at Electric Sheep?, V\-FLUTE, PixelHumor, and the YesBut series move beyond surface labels and ask whether models can identify why a caption is funny, what contradiction drives a panel sequence, or how an implicit target should be resolvedhessel2023androids;saakyan2025understanding;hu2024cracking;ryan2025humor\. These resources are far more diagnostic than recognition datasets, but they remain smaller, culturally narrower, and more expensive to annotate\.

Generation Resources\.Dedicated generation benchmarks are fewer and often repurpose understanding data as supervised targets\. MEMECAP and OxfordTVG\-HIC center humorous caption generation from static images, while Oogiri\-GO and XMeCap extend the space toward joke completion, meme rewriting, or continuation over structured inputshwang2023memecap;li2023oxfordtvg;zhong2024let;chen2024xmecap\. Their main value is diagnostic: they reveal whether a model can transform an interpretation of the source image into a controlled, novel output\. At present, however, generation benchmarks remain sparse, and their evaluation protocols are still less mature than those used for recognition or explanation\.

## 6Cross\-Benchmark Empirical Analysis

Table 3:Results on humor understanding benchmarks\. All numbers are accuracy \(%\)\. Model results are obtained by evaluating a broader and more recent set of MLLMs using the benchmark\-specific prompts and task definitions reported in the original papers\. Human performance figures, where available, are taken from the corresponding benchmark papers\. Task abbreviations: forYesBut\-v2liang2025yes,MoralandTitledenote choosing the correct moral or title from multiple choices;NYCChessel2023androidsdenotes selecting the most suitableCaptionfor a New Yorker cartoon from options A–E;MemeQAnguyen2025memeqadenotes multiple\-choiceFill\-in\-the\-blank question answering on memes;ExHVVsharma2023youdenotesRoleselection for entities from \{hero, villain, victim\} in memes;DarkHumorkasu2025ddenotes binaryDetection of dark humor \(Yes/No\);HumorDBjain2025humordbdenotes binaryDetection of humor \(Yes/No\)\. ForMangaUBikuta2025mangaub,RecBg,CharCnt,PanelLoc,NextInf, andOnomdenoterecognition\_background,character\_count,panel\_localization,next\_panel\_inference, andonomatopoeia\_scene, respectively\.To move beyond qualitative synthesis, we extend the original evaluation protocols of the surveyed benchmarks to a broader and more recent set of MLLMs, while retaining the benchmark\-specific prompts reported in the corresponding papers\. We organize the resulting scores according to the capability hierarchy introduced in Section 3\. This cross\-benchmark empirical analysis serves two purposes: \(1\) it provides an updated snapshot of current MLLM performance across different humor\-understanding capabilities, and \(2\) it reveals recurring patterns that are difficult to observe from individual benchmarks alone, particularly the contrast between recognition\-oriented tasks and interpretation\- or reasoning\-oriented tasks\. Human performance figures, where available, are taken from the original benchmark papers\.

#### Evaluation protocol\.

For each benchmark, we follow the prompt format and task definition reported in the original paper\. We use the official evaluation split whenever available and evaluate a broader set of recent MLLMs under the same benchmark\-specific protocol\. Because prompts differ across benchmarks by design, the results support within\-benchmark model comparison and descriptive cross\-benchmark analysis, rather than a strictly controlled comparison of task difficulty\.

### 6\.1Cross\-Level Performance Landscape

Table[3](https://arxiv.org/html/2607.19011#S6.T3)summarizes MLLM results across seven humor\-understanding benchmarks and twelve task settings\. Following our capability hierarchy, we group ExHVV, DarkHumor, HumorDB, and MangaUB underrecognition, since these tasks primarily require models to identify roles, detect humor categories, or recognize visual and structural elements\. We group YesBut\-v2, NYCC, and MemeQA underinterpretation & reasoning, since these tasks require models to infer morals, select appropriate titles or captions, and fill in missing semantic content\.

Several patterns emerge from this cross\-benchmark view\.

#### Recognition\-oriented tasks generally yield higher reported accuracies, but remain far from uniformly solved\.

Models generally perform strongly on visually grounded recognition tasks, especially the MangaUB subtasks\. GPT\-4o achieves 97\.60% on background recognition, 99\.21% on panel localization, 64\.30% on next\-panel inference, and 95\.80% on onomatopoeia\-scene recognition, obtaining the best score on four of the five MangaUB subtasks\. Open\-source models are also competitive on several recognition tasks: Qwen3\.5\-27B reaches 95\.21% on character counting and 74\.71% on HumorDB, while Qwen3\.5\-35B\-A3B achieves the best ExHVV role\-classification accuracy at 80\.03%\. However, recognition is not uniformly easy\. DarkHumor remains difficult for most models, with the best score reaching only 66\.43% with Qwen3\-VL\-8B\-Thinking\. This suggests that recognition remains challenging when it depends on implicit social norms, taboo framing, or pragmatic cues\.

#### Interpretation and reasoning remain the central bottleneck\.

Compared with recognition\-oriented tasks, interpretation\-oriented benchmarks show larger and more consistent gaps from human performance\. On YesBut\-v2, the best model score is 84\.72% for moral selection and 83\.28% for title selection, both achieved by Qwen3\.5\-27B, while human performance reaches 91\.30% and 97\.50%, respectively\. On NYCC caption selection, GPT\-4o substantially outperforms all other models with 82\.30%, but still trails the human score of 94\.00%\. MemeQA shows a similar gap: the best model, Qwen3\.5\-35B\-A3B, reaches 60\.41%, compared with 81\.90% for humans\. These results indicate that current MLLMs can often recognize salient entities or visual structures, but still struggle to infer the intended humorous mechanism, implicit punchline, or culturally appropriate interpretation\.

#### Model ranking varies substantially across humor capabilities\.

No single model dominates all task types\. GPT\-4o is strongest on NYCC and most MangaUB subtasks, suggesting strong visual recognition and caption\-matching ability\. Qwen3\.5\-27B performs best on both YesBut\-v2 subtasks and HumorDB, indicating stronger performance on moral and title inference as well as general humor detection\. Qwen3\.5\-35B\-A3B achieves the highest ExHVV and MemeQA scores, while Qwen3\-VL\-8B\-Thinking performs best on DarkHumor\. This fragmented ranking suggests that humor understanding is not a single monolithic ability\. Different benchmarks emphasize different combinations of visual recognition, social knowledge, narrative inference, and pragmatic reasoning\.

#### Reasoning\-oriented variants do not consistently improve humor understanding\.

The comparison between Qwen3\-VL\-8B and Qwen3\-VL\-8B\-Thinking is particularly revealing\. The Thinking variant substantially improves DarkHumor detection from 49\.43% to 66\.43% and improves ExHVV from 76\.24% to 78\.64%, suggesting that deliberative reasoning can help when the task requires social or normative judgment\. However, it decreases performance on several interpretation tasks, including YesBut\-v2 Moral, NYCC, and MemeQA, and also drops sharply on MangaUB next\-panel inference\. This mixed pattern indicates that explicit reasoning does not automatically translate into better humor understanding\. In some settings, extended reasoning may help identify implicit intent, while in others it may divert the model from direct visual–semantic matching or introduce unnecessary intermediate steps\.

#### Human performance remains substantially higher on interpretation\-heavy tasks\.

Where human results are available, the largest gaps appear in tasks requiring semantic or pragmatic interpretation\. Humans outperform the best model by 6\.58 points on YesBut\-v2 Moral, 14\.22 points on YesBut\-v2 Title, 11\.70 points on NYCC, and 21\.49 points on MemeQA\. The gap is also large on HumorDB, where the best model reaches 74\.71% compared with 85\.00% for humans\. These gaps reinforce the main finding of this section: current MLLMs have made considerable progress on visual and categorical recognition, but still fall short in recovering the intended meaning, communicative function, and incongruity structure that make visual humor understandable to humans\.

## 7Challenges and Future Directions

Despite rapid progress in MLLMs on literal scene perception, a substantial gap remains between recognizing*what is depicted*and interpreting*what is meant*in multimodal visual humor\. Closing this gap requires moving beyond physical description toward socio\-cultural, rhetorical, and value\-laden interpretation\. We synthesize this gap into interconnected challenges and outline concrete directions for the community\.

### 7\.1The Evaluation Crisis

Most benchmarks reduce humor understanding to multiple\-choice questions \(MCQs\) or binary classificationhessel\-etal\-2023\-androids;hu2024cracking;yang\-etal\-2024\-large\. This is scalable but poorly matched to the phenomenon: discriminative accuracy is inflated by shortcut learning, and creative interpretation is inherently open\-ended\.

Future directions\.A promising direction is*rubric\-based generative evaluation*, where models produce free\-form interpretations that are assessed along disentangled dimensions such as incongruity detection, target identification, and cultural grounding, rather than a single aggregate labelgunjal2025rubrics;liang2025yes\. This can be coupled with LLM\-as\-judge frameworks to operationalize humor\-specific rubrics, rewarding plausible novel interpretations while penalizing unsupported or hallucinated reasoningliu2025inference;liu2023g\. Such a paradigm better aligns evaluation with the inherently open\-ended and interpretive nature of humor understanding\.

### 7\.2Social and Cultural Grounding

Creative artifacts depend on shared background knowledge, implicit norms, and audience beliefs that current MLLMs reason about poorlyjiang2025can;hu2023language;chiu2025culturalbench\. First, current large models still lack theory\-of\-mind–like reasoning to infer whose perspective is expressed, what belief is being challenged, how an audience is meant to react, and often default to flat literal description insteadchen\-etal\-2025\-theory\. In addition, parametric knowledge has a fixed cutoff, so memes tied to breaking news or ephemeral trends become opaquekasai2024realtime, while training data remains predominantly English and Western, producing systematic blind spots on non\-Western visual symbols and humor conventionsyu2025seeing;park2025evaluating\.

Early efforts such as CHumor for Chinese humorhe2024chumor, Chinese multimodal sarcasmgao2025multimodal, and StandUp4AI for multilingual stand\-upbarriere2025standup4aibroaden coverage, but still capture only a small slice of global humor traditions\. Retrieval\-augmented generation \(RAG\)lewis2020retrievaloffers a promising path: retrieval\-guided learning improves hateful\-meme detectionmei2024improvingand contextualized meme explanationsharma2023memex, yet humor demands retrieval of not only topical knowledge but also the social norms and cultural contexts that make jokes intelligible\.

Future directions\.\(i\)*Multicultural, per\-culture benchmarks*with annotations of the background knowledge each item requires, rather than a single aggregate accuracy; \(ii\)*humor\-aware RAG*: retrieval pipelines that surface community norms, trending discourse, and event timelines alongside factual context, paired with periodic knowledge refresh to mitigate temporal decay; \(iii\)*explicit intent and stance modeling*\(critique, mockery, self\-deprecation\) as a structured, trainable proxy for theory\-of\-mind reasoning\.

### 7\.3Narrative Reasoning in Sequential Humor

Our cross\-benchmark analysis \(Section[6](https://arxiv.org/html/2607.19011#S6)\) shows that sequential, multi\-panel humor exposes the widest model–human gap on panel\-sequencing and temporal reorderingryan2025humor;wang\-etal\-2025\-beyond\-single\. Unlike single\-image humor, where the incongruity sits within one frame, multi\-panel humor requires maintaining entity identity across panels, building an expectation from setup panels, and localizing the exact point where the narrative violates it\. Current architectures largely process panels in isolation, missing these temporal and causal dependencies\.

Future directions\.\(i\)*Panel\-aware architectures*that explicitly encode panel order and cross\-frame entity co\-reference, borrowing temporal\-attention and state\-tracking inductive biases from video understanding; \(ii\)*setup–punchline decomposition*as an explicit training objective rather than holistic pattern matching; \(iii\)*diagnostic benchmarks*isolating individual narrative skills \(next\-panel prediction, swapped\-panel detection\) for targeted evaluation beyond aggregate accuracy\.

### 7\.4Toward Unified, Safe Systems

Recognition, interpretation, and generation are currently treated as independent tasks with separate pipelines, despite being deeply intertwined\. A system that can explain why a meme is funny should, in principle, be better at generating one\. This fragmentation is compounded by a safety problem specific to humor: the same non\-literal mechanisms that make humor effective \(incongruity, exaggeration, irony\) are also what makes harmful content hard to detect\. The Hateful Memes Challenge showed that neither unimodal classifier can reliably catch cross\-modally\-constructed hatekiela2020hateful, and follow\-up work shows detecting*who*is targeted is as important as detecting*that*harm occurssharma2022disarm;pramanick2021momenta\. As generation models improve, this becomes dual\-use: the same system that writes witty captions can be steered toward harassment or stereotype\-reinforcing contentweidinger2022taxonomy;wang2024memecraft, and current VLMs remain vulnerable to adversarial meme\-based attackslee2025vision\. Overly conservative filters, in turn, risk censoring legitimate satire and self\-deprecating humor — an unresolved safety–creativity trade\-offjha2024memeguard\. Separately, creative datasets are frequently scraped without creator consent, raising unresolved data\-provenance and “right to style” concerns as models move toward style\-imitative generation\.

Future directions\.\(i\)*Joint training across the capability hierarchy*\(recognition, interpretation, generation\) to test whether gains at one level transfer to the others, and*self\-consistency checks*\(e\.g\., a model that rates a meme funny but explains it blandly\) as a diagnostic for genuine understanding; \(ii\)*context\-sensitive, rhetoric\-aware moderation*: safety classifiers conditioned on whether an artifact critiques versus promotes a harmful stance, paired with humor\-specific red\-teaming rather than keyword filtering; \(iii\)*provenance\-aware training*: datasets and models that record content origin, licensing, and consent, auditable back to their training sources\.

## 8Conclusion

Multimodal visual humor challenges AI systems as its meaning extends beyond surface perception\. Achieving robust understanding and generation therefore requires moving toward socially and rhetorically grounded reasoning\. This survey introduces a capability\-centric task hierarchy that clarifies how creative understanding progresses from recognition to interpretation and generation\. We further outline future directions for meaningful, reliable, and responsible engagement with human\-created media\.

## Limitations

Despite growing interest in multimodal humor, this survey has several limitations\. First, most existing research and datasets focus on Western, internet\-centric visual humor forms \(e\.g\., memes, cartoons\), leaving many cultural traditions and non\-mainstream media underexplored\. Second, as multimodal models evolve rapidly, some observations may not fully generalize to future architectures or training paradigms\.

## Ethical considerations

Understanding creative expression poses distinct ethical challenges\. Creative media may potentially convey harmful or sensitive content implicitly through humor, irony, or symbolism, increasing the risk of misinterpretation, bias amplification, or over\-censorship\. Dataset bias and cultural imbalance further threaten fairness and robustness, particularly for marginalized communities\. In addition, many creative datasets raise unresolved copyright and ownership concerns, especially as models increasingly transition from understanding to generation and style imitation\. Addressing these issues requires context\-aware evaluation, transparent dataset practices, and greater emphasis on interpretability and human oversight when deploying such systems\.

## References

## Appendix AEvaluation Details

We evaluate all models in a zero\-shot setting\. For each benchmark, we retain the task definition, prompt format, answer format, evaluation split, and scoring procedure reported in the corresponding original paper\. Except for GPT\-4o, all evaluated models use publicly available checkpoints hosted on Hugging Face\. For GPT\-4o, we use the fixed API versiongpt\-4o\-2024\-05\-13\. All experiments were completed by May 29, 2026\.

Each model is queried once per question\. For open\-source models, decoding is performed withdo\_sample=true; all remaining benchmark\-specific generation and evaluation settings follow the corresponding original papers\. When an official evaluation split is available, we use it directly; otherwise, we follow the split or sampling procedure described in the benchmark paper\. Because prompts and evaluation protocols differ across benchmarks by design, the reported results are intended primarily for within\-benchmark model comparison and descriptive cross\-benchmark analysis, rather than for a strictly controlled comparison of task difficulty\. Since each question is evaluated with a single sampled generation, small score differences may be affected by decoding stochasticity and should not be interpreted as statistically significant\.

## Appendix BOverview of Datasets and Benchmarks

Here we provide a comprehensive tabular overview of datasets and benchmarks studied in this survey, organized according to the capability hierarchy introduced in the main paper\.

Table 4:Overview of multimodal visual humor datasets focused onRecognitiontasks\. Recognition datasets typically use binary labels \(e\.g\., humorous vs\. non\-humorous\), element\-level labels \(e\.g\., punchlines or rhetorical roles\), and intensity scores \(e\.g\., degree of funniness or offensiveness\)\. In Data Forms, StaVT denotes Static Visual–Textual Artifacts; SeqVN denotes Sequential Visual Narratives\. In Mechanism, Multi denotes datasets that involve multiple mechanisms defined in Sec\.[2](https://arxiv.org/html/2607.19011#S2)\. The Availability column provides links to publicly accessible datasets, and "N/A" indicates unpublished datasets\.DatasetVenueData FormsMechanismSizeAvail\.StaVT: MemeGoal: Humor & Entertainment, Satire & Social CritiqueD\-HUMORkasu2025dICDM’25StaVT: MemeMulti4,379[Link](https://github.com/Sai-Kartheek-Reddy/D-Humor-Dark-Humor-Understanding-via-Multimodal-Open-ended-Reasoning)MemeMindgu2025mememindArxiv’25StaVT: MemeMulti43,223N/ATOXICN MMlu2024towardsNeurIPS’24StaVT: MemeMulti12K[Link](https://github.com/DUT-lujunyu/ToxiCN_MM)PrideMMshah2024memeclipEMNLP’24StaVT: MemeMulti5,063[Link](https://drive.google.com/file/d/17WozXiXfq44Z6kkWsPPDHRzqIH2daUaQ/view)BHMhossain2024decipheringACL’24StaVT: MemeMulti7,148N/AExt\-Harm\-Psharma2022disarmNAACL’23StaVT: MemeMulti4,446[Link](https://github.com/LCS2-IIITD/DISARM)HVVMemessharma2023characterizingEACL’23StaVT: MemeMulti6,933[Link](https://codalab.lisn.upsaclay.fr/competitions/906)RESTOREyadav2023towardsACL’23StaVT: MemeMulti4,664[Link](https://docs.google.com/forms/d/e/1FAIpQLSeK8h2TvDt-XzZzvy0vWvjauDAX8eUXJ6OUuoAfd4ZqWQPqLA/viewform)Dank or not?Barnes2020DankONApp\. Net\. Sci\.’21StaVT: MemeMulti70KN/AThe Hateful Memes Challenge Setkiela2020hatefulNeurIPS’20StaVT: MemeMulti10K[Link](https://www.drivendata.org/competitions/64/hateful-memes/)StaVT: Sarcastic ImageGoal: Satire & Social CritiqueRedEvaltang2024leveragingNAACL’24StaVT: Sarcastic ImageSarcasm1,004[Link](https://github.com/TangBinghao/naacl2024)SPMSDguo2025multiCOLING’24StaVT: Sarcastic ImageSarcasm1KN/AMSTI datasetwang2022multimodalACL’22StaVT: Sarcastic ImageSarcasm5,015[Link](https://github.com/wjq-learning/MSTI)Multi\-Modal Sarcasm Detection in Twittercai2019multiACL’19StaVT: Sarcastic ImageSarcasm24,635N/ASarcasm in Multimodal Social Platformsschifanella2016detectingACM MM’16StaVT: Sarcastic ImageSarcasm10KN/AStaVT: Humorous Image / CartoonGoal: Humor & EntertainmentHumorDBjain2025humordbICCV’25StaVT: Humorous ImageMulti3,542[Link](https://github.com/kreimanlab/HumorDB)Inside Jokesshahaf2015insideKDD’15StaVT: CartoonMulti76,928N/ASeqVN: Comic StripGoal: Humor & EntertainmentCOMICORDAmartinek2024comicordaCOLING’24SeqVN: Comic StripNarrative1,438N/AAVH & FORchandrasekaran2016weCVPR’16SeqVN: Comic StripNarrative7,150[Link](https://github.com/GT-Vision-Lab/abstract_scenes_v002)Table 5:Overview of multimodal visual humor datasets focused onUnderstandingandGeneration\. These datasets typically pair multimodal inputs with explanations or rationales—often augmented with contextual or external knowledge—to justify intended meaning \(e\.g\., humor, irony, or satire\), alongside target outputs that support coherent and controllable generation \(e\.g\., memes or comics with captions/rationales\)\. StaVT denotes Static Visual–Textual Artifacts; SeqVN denotes Sequential Visual Narratives\. In Mechanism, Multi denotes datasets that involve multiple mechanisms defined in Sec\.[2](https://arxiv.org/html/2607.19011#S2)\. The Availability column provides links to publicly accessible datasets, and "N/A" indicates unpublished datasets\.DatasetVenueMechanismSizeAvail\.StaVT: Meme\|Goal:Humor & Entertainment, Satire & Social CritiqueMemeReaConzhao2025memereaconEMNLP’25Multi1,565N/AMEMESAFETY\-BENCHlee2025visionEMNLP’25Multi50,430[Link](https://huggingface.co/datasets/oneonlee/Meme-Safety-Bench)MemeMindgu2025mememindArxiv’25Multi43,223N/AMemeQAnguyen2025memeqaACL’25Multi9K[Link](https://github.com/npnkhoi/memeqa)SEMANTICMEMESzhou2024socialNAACL’24Multi3\.8M[Link](https://github.com/naitian/social-memeing)MMDkhan2024hopeEMNLP\-Fdgs’24Multi13,494[Link](https://github.com/anas2908/MeSum)MultiBully\-Exjha2024memeEACL’24Multi5,854[Link](https://github.com/Jhaprince/MemeExplanation)Oogiri\-GOzhong2024letCVPR’24Multi130K[Link](https://huggingface.co/datasets/zhongshsh/CLoT-Oogiri-GO)MemeMQACorpusagarwal2024mememqaACL\-Fdgs’24Multi1,880N/AICMMjha2024memeguardACL’24Multi1K[Link](https://github.com/Jhaprince/MemeGuard)OxfordTVG\-HICli2023oxfordtvgICCV’23Multi2\.9M[Link](https://drive.google.com/drive/folders/1BDuUcMeaWrFD8TwgHLhFPkuAwmoHaVNQ)MEMECAPhwang2023memecapEMNLP’23Multi6,300[Link](https://github.com/eujhwang/meme-cap)MCCsharma2023memexACL’23Multi3\.4K[Link](https://github.com/LCS2-IIITD/MEMEX_Meme_Evidence)HatReDhee2023hatredIJCAI’24Multi3,228[Link](https://github.com/Social-AI-Studio/HatReD)ExHVVsharma2023youAAAI’22Multi3K[Link](https://github.com/LCS2-IIITD/LUMEN-Explaining-Memes)StaVT: Cartoon\|Goal:Humor & EntertainmentHumorBenchnarad2025llmsArxiv’25Multi300N/AHumor in AIzhang2024humorNeurIPS’24Incongruity2\.2M[Link](https://huggingface.co/datasets/yguooo/newyorker_caption_ranking)Do Androids Laugh at Electric Sheep?hessel2023androidsACL’23Multi24,048N/AStaVT: Social Media Image\|Goal:Humor & Entertainment, Satire & Social Critique, Emotion & Aesthetic ExperienceV\-FLUTEsaakyan2025understandingNAACL’25Multi6,027[Link](https://huggingface.co/datasets/ColumbiaNLP/V-FLUTE)SoMeLVLMzhang2024somelvlmACL’24Multi653\.8K[Link](https://huggingface.co/datasets/Lishi0905/SoMeData)StaVT: Humorous Image\|Goal:Humor & EntertainmentHumorDBjain2025humordbICCV’25Multi3,542[Link](https://github.com/kreimanlab/HumorDB)VisualPun\_UNPIEchung2024canEMNLP’24Incongruity1K[Link](https://huggingface.co/datasets/jiwan-chung/VisualPun_UNPIE)SeqVN: Comic Strip\|Goal:Humor & EntertainmentMangaUBikuta2025mangaubIEEE MM’25Narrative18,179[Link](http://www.manga109.org/en/mangaub.html)AI4VA\-FGchen2025zoomingArxiv’25Narrative16,264N/APixelHumorryan2025humorEMNLP\-Fdgs’25Multi2\.8K[Link](https://github.com/Social-AI-Studio/PixelHumor)YesBut\-v2liang2025yesArxiv’25Multi1,262[Link](https://huggingface.co/datasets/zhehuderek/YESBUT_Benchmark)YesButhu2024crackingNeurIPS’24Multi348[Link](https://huggingface.co/datasets/zhehuderek/YESBUT_Benchmark)YesBut \(synth\. 3D stick\)nandy2024yesbutEMNLP’24Multi2,547 \(syn\.\)[Link](https://huggingface.co/datasets/bansalaman18/yesbut)ComicsPAPvivoli2025comicspapArxiv’25Narrative103,933[Link](https://huggingface.co/datasets/VLR-CVC/ComicsPAP)SeqVN: Meme\|Goal:Humor & EntertainmentXMeCapchen2024xmecapACM MM’24Multi12,320N/A
## Appendix CTask Specific Models Before MLLM Era

Prior to MLLMs, multimodal visual humor understanding was dominated by task\-specific discriminative architectures tightly coupled with individual tasks such as sarcasm, humor, or metaphor detection\.

Early models focused on effective fusion mechanisms for heterogeneous features\.schifanella2016detectingfirst incorporated visual cues into sarcasm detection via separate encoders and concatenation, whilecai2019multishowed hierarchical fusion of text, images, and attributes better captures cross\-modal interactions\.zhang2021multiintroduced contrastive attention to explicitly model inter\-modal incongruity for finer detection of cross\-modal discrepancies\. These approaches improved reasoning on specific tasks, but they also introduced stronger task assumptions and often depended on curated knowledge sources or structures\.

Beyond fusion design, later work emphasized implicit knowledge and commonsense\. HKThasan2021humorinjected humor\-related knowledge into Transformer architecture for deeper incongruity modeling\.lee2021disentanglingandliang2022multileveraged object\-level visual representations to construct richer contextual embeddings and cross\-modal graphs, facilitating localized reasoning over visual–textual conflicts\.liu2022towardsincorporated external commonsense and semantic knowledge into hierarchical congruity modeling, improving implicit intent interpretation\.

Overall, non\-MLLM approaches made important progress by improving fusion design, structured representations, and knowledge injection\. However, their gains were typically task\-dependent, and their scalability and cross\-domain generalization remained limited, as also noted in prior surveyssharma2022detecting;farabi2024survey\. This limitation motivates the later shift toward more general multimodal foundation\-model paradigms\.