NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection

# Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection
Source: [https://arxiv.org/html/2608.19212](https://arxiv.org/html/2608.19212)
Sanjeev Khatiwada Independent Researcher Kathmandu, Nepal skhatiwada558@gmail\.com

###### Abstract

Out\-of\-context \(OOC\) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making detection a problem of multimodal alignment rather than image forensics\. Despite the prevalence and consequences of OOC misinformation in Nepal, no public benchmark exists for Nepali\. We introduceNepOOC, the first publicly available Nepali\-dominant multilingual OOC benchmark, comprising 1,090 image–caption pairs \(545 pristine, 545 OOC\) annotated across five typologies \(fabricated, miscaptioned, temporal mismatch, geographic mismatch, identity mismatch\) with inter\-annotator agreementκ=0\.84\\kappa=0\.84\. Systematic evaluation of five multimodal architectures alongside text\-only and image\-only baselines reveals that caption semantics appear sufficient for strong performance at the current dataset scale\. A text\-only mBERT model achieves94\.65±0\.20%94\.65\\pm 0\.20\\%Macro\-F1, statistically equivalent to the best multimodal system \(ResNet\-50\+mBERT,94\.65±0\.20%94\.65\\pm 0\.20\\%; McNemar medianp=1\.000p=1\.000, 0/5 seeds significant atα=0\.05\\alpha=0\.05\)\. Image\-only models perform near chance \(33–50%\), while training\-size scaling suggests that dataset expansion is a more direct path to progress than architectural sophistication or regional specialisation\.

## IIntroduction

Out\-of\-context \(OOC\) misinformation exploits authentic, unmanipulated images by pairing them with false or misleading captions to construct plausible\-looking but semantically inaccurate claims\[[2](https://arxiv.org/html/2608.19212#bib.bib2),[3](https://arxiv.org/html/2608.19212#bib.bib3),[4](https://arxiv.org/html/2608.19212#bib.bib4)\]\. Unlike synthetic or doctored imagery, OOC content passes visual plausibility checks and is consequently difficult for both humans and automated systems to detect\[[5](https://arxiv.org/html/2608.19212#bib.bib5)\]\. The core challenge is not whether the image is authentic, but whether the image and caption together accurately represent a real event or situation\. This distinction has motivated a research agenda for OOC detection that is separate from deepfake or image tampering detection\.

In Nepal, OOC misinformation is both prevalent and consequential, spanning elections, public health, natural disasters, and communal affairs\. Recurring peaks occur during major events, where authentic photographs are recirculated with fabricated captions designed to reframe the depicted situation\[[27](https://arxiv.org/html/2608.19212#bib.bib27)\]\. A representative case involved a Madhesh Province official who used photographs of Delhi to falsely claim provincial infrastructure development\[[28](https://arxiv.org/html/2608.19212#bib.bib28)\]\. In another incident, an old photograph was paired with an inflammatory caption tied to the 2023 Dharan controversy, nearly inciting communal conflict before fact\-checkers intervened\[[27](https://arxiv.org/html/2608.19212#bib.bib27)\]\. These cases share a common structure that distinguishes Nepali OOC from English\-language counterparts: the critical evidence lies in caption claims about regional actors, locations, and events that require localised knowledge to evaluate\.

Despite the severity of this problem, OOC detection research remains concentrated in high\-resource, English\-language settings\. Benchmarks such as NewsCLIPpings\[[2](https://arxiv.org/html/2608.19212#bib.bib2)\], COSMOS\[[3](https://arxiv.org/html/2608.19212#bib.bib3)\], and VERITE\[[4](https://arxiv.org/html/2608.19212#bib.bib4)\]have driven methodological progress, yet none addresses challenges unique to a low\-resource, bilingual environment: code\-switching between Devanagari and Latin scripts, sparse named\-entity coverage for Nepali figures and locations, and the absence of multimodal OOC tools adapted to regional contexts\. We refer to this as the*regional context gap*: a vision\-language model that performs well on English news may still fail on Nepali content because it lacks the script\-level and cultural grounding needed to identify image–caption inconsistencies in this domain\.

This paper addresses the regional context gap through three contributions:

1. 1\)NepOOCDataset\.The first publicly available Nepali\-dominant multilingual OOC benchmark, comprising 1,090 image–caption pairs annotated across five typologies with Cohen’sκ=0\.84\\kappa=0\.84, drawn from real\-world Nepali misinformation incidents\.
2. 2\)Comparative Architecture Study\.Systematic evaluation of five multimodal architectures with dedicated text\-only and image\-only baselines, designed to identify which inductive biases and modalities transfer effectively to this low\-resource, bilingual setting\.
3. 3\)Text Sufficiency Finding\.Multi\-metric evaluation indicates that caption semantics are sufficient for high performance within this benchmark: text\-only mBERT is statistically equivalent to the best multimodal system, while image\-only models perform at chance\. This challenges common assumptions about multimodal fusion benefits in low\-resource settings\.

The central empirical findings are: \(1\) ResNet\-50\+mBERT achieves the highest Macro\-F1 among multimodal systems \(94\.65±0\.20%94\.65\\pm 0\.20\\%\); \(2\) text\-only mBERT matches this performance, indicating that visual features contribute minimally at the current dataset scale; and \(3\) training\-size scaling suggests that dataset expansion is the primary lever for future progress\.

## IIRelated Work

We organise the literature along four axes: OOC detection methods, vision\-language architectures, the low\-resource and Nepali language landscape, and the gap thatNepOOCaddresses\.

### II\-AOut\-of\-Context Misinformation Detection

OOC misinformation differs from image forgery or deepfake detection: the image is authentic, but its caption displaces it from its original context\[[2](https://arxiv.org/html/2608.19212#bib.bib2),[3](https://arxiv.org/html/2608.19212#bib.bib3)\]\. Detection requires cross\-modal semantic alignment rather than pixel\-level forensics\.

Early work established the task infrastructure\. NewsCLIPpings\[[2](https://arxiv.org/html/2608.19212#bib.bib2)\]introduced automatic pipelines for constructing large\-scale OOC benchmarks from news corpora\. COSMOS\[[3](https://arxiv.org/html/2608.19212#bib.bib3)\]proposed self\-supervised consistency objectives for grounded image–caption representations, while VERITE\[[4](https://arxiv.org/html/2608.19212#bib.bib4)\]incorporated unimodal\-bias controls to ensure models perform genuine cross\-modal reasoning\. Alongside these benchmarks, entity\-enhanced fusion\[[8](https://arxiv.org/html/2608.19212#bib.bib8)\], transformer\-based architectures\[[7](https://arxiv.org/html/2608.19212#bib.bib7)\], and context\-aware reasoning\[[9](https://arxiv.org/html/2608.19212#bib.bib9)\]improved detection in high\-resource settings, and open\-domain methods extended the task by retrieving web evidence\[[5](https://arxiv.org/html/2608.19212#bib.bib5)\]\.

More recently, the field has moved toward explainability and external knowledge\. RED\-DOT\[[10](https://arxiv.org/html/2608.19212#bib.bib10)\]showed that retrieved evidence documents substantially improve OOC detection, establishing evidence\-aware pipelines as a standard paradigm\. COVE\[[11](https://arxiv.org/html/2608.19212#bib.bib11)\]incorporated contextual verification signals, and SNIFFER\[[37](https://arxiv.org/html/2608.19212#bib.bib37)\]demonstrated that multimodal large language models can produce human\-readable explanations alongside binary verdicts\. Graph\-based architectures have also emerged as a promising direction\[[38](https://arxiv.org/html/2608.19212#bib.bib38)\]\. A comprehensive survey\[[33](https://arxiv.org/html/2608.19212#bib.bib33)\]charts the transition from early fusion architectures\[[1](https://arxiv.org/html/2608.19212#bib.bib1)\]toward retrieval\-augmented systems\. Modality contribution studies\[[24](https://arxiv.org/html/2608.19212#bib.bib24)\]document that text often provides sufficient signal in certain settings—a finding this work corroborates and extends to a low\-resource, non\-English context\.

### II\-BVision\-Language Architectures and Fusion Strategies

On the visual side, convolutional networks—particularly ResNet\[[22](https://arxiv.org/html/2608.19212#bib.bib22)\]—remain competitive baselines for spatial feature extraction, while Vision Transformers \(ViT\)\[[16](https://arxiv.org/html/2608.19212#bib.bib16)\]extend patch\-based representation learning to visual inputs\. CLIP\[[17](https://arxiv.org/html/2608.19212#bib.bib17)\]yields highly transferable image–text representations via large scale contrastive pretraining, though this objective optimises global similarity ranking rather than supervised binary classification, which may limit applicability to small, domain\-specific benchmarks\.

On the text side, BERT\-style encoders\[[14](https://arxiv.org/html/2608.19212#bib.bib14)\]and their multilingual extensions\[[15](https://arxiv.org/html/2608.19212#bib.bib15)\]provide strong cross\-lingual transfer baselines\. Fusion strategies span late concatenation, cross\-attention over patch tokens, and graph\-based relational modelling\[[38](https://arxiv.org/html/2608.19212#bib.bib38)\]\. A recurring observation across the literature and a finding of our own ablations is that architectural complexity does not consistently improve performance when labelled data are scarce; strong pretrained text encoders often suffice\[[24](https://arxiv.org/html/2608.19212#bib.bib24)\]\.Yet whether architectural complexity improves OOC detection with code\-switched, script\-diverse captions remains empirically untested\.

### II\-CLow\-Resource and Nepali Language Processing

Work on Nepali NLP has grown, though multimodal misinformation detection remains underexplored\. MuRIL\[[18](https://arxiv.org/html/2608.19212#bib.bib18)\], trained on 17 Indian and South Asian languages including Devanagari, provides the strongest off\-the\-shelf foundation for Nepali text processing\. Nepali\-specific language models—NepBERTa\[[20](https://arxiv.org/html/2608.19212#bib.bib20)\]and NepaliBERT\[[21](https://arxiv.org/html/2608.19212#bib.bib21)\]together with transformer\-based classifiers\[[19](https://arxiv.org/html/2608.19212#bib.bib19),[30](https://arxiv.org/html/2608.19212#bib.bib30),[31](https://arxiv.org/html/2608.19212#bib.bib31),[32](https://arxiv.org/html/2608.19212#bib.bib32)\]demonstrate that script\-aware pretraining improves text classification\. These models, however, address text\-only tasks and do not extend to multimodal or OOC settings\.

At the multimodal frontier, recent Indic\-language work has begun combining MuRIL with Vision Transformers for multilingual fake news detection\[[39](https://arxiv.org/html/2608.19212#bib.bib39)\], though without a dedicated OOC evaluation or typology schema\. A bilingual Nepali–English misinformation tool\[[40](https://arxiv.org/html/2608.19212#bib.bib40)\]addresses code\-switching but is limited to text and lacks image–caption pair classification\. The prevalence and consequences of OOC manipulation in Nepal are documented in\[[27](https://arxiv.org/html/2608.19212#bib.bib27),[28](https://arxiv.org/html/2608.19212#bib.bib28)\], motivating a dedicated detection resource\.

No prior work provides a multimodal OOC benchmark with typology\-level annotations in Nepali or any closely related low\-resource South Asian language\. This is the gap thatNepOOCfills\.

### II\-DPositioningNepOOC

[TableI](https://arxiv.org/html/2608.19212#S2.T1)situatesNepOOCrelative to the most relevant existing benchmarks\. NewsCLIPpings\[[2](https://arxiv.org/html/2608.19212#bib.bib2)\]and COSMOS\[[3](https://arxiv.org/html/2608.19212#bib.bib3)\]provide large\-scale English OOC resources but lack typology labels and multilingual coverage\. VERITE\[[4](https://arxiv.org/html/2608.19212#bib.bib4)\]addresses unimodal bias but is English\-dominant\. MuMiN\[[6](https://arxiv.org/html/2608.19212#bib.bib6)\]covers multiple languages in a low\-resource regime but does not focus on OOC detection or provide typology annotations\.NepOOCis the first resource to simultaneously offer OOC\-specific focus, fine\-grained typology labels \(κ=0\.84\\kappa=0\.84\), genuine bilingual content reflecting real code\-switching patterns, and pairs drawn from actual Nepali misinformation incidents\.

TABLE I:Benchmark comparison\. Y=Yes, N=No\. Typol\.=typology labels; Multi\.=multilingual; Low\-Res\.=low\-resource\.BenchmarkOOCTypol\.Multi\.Low\-Res\.SizeNewsCLIPpings\[[2](https://arxiv.org/html/2608.19212#bib.bib2)\]YNNN71kCOSMOS\[[3](https://arxiv.org/html/2608.19212#bib.bib3)\]YNNN200kVERITE\[[4](https://arxiv.org/html/2608.19212#bib.bib4)\]YNPartialN1kMuMiN\[[6](https://arxiv.org/html/2608.19212#bib.bib6)\]NNYPartial22kNepOOC\(ours\)YYYY1,090![Refer to caption](https://arxiv.org/html/2608.19212v1/nepooc_representative_figures.png)Figure 1:Representative pristine and OOC pairs across five typologies\. Red boxes highlight manipulated phrases\.

## IIINepOOC Dataset

NepOOCreflects three core design principles: all pairs are drawn from real\-world misinformation instances \(ecological validity\); each OOC instance carries a typology annotation \(typological granularity\); and the benchmark encompasses Nepali, English, and code\-switched captions \(multilingual realism\)\.

### III\-AData Collection and Sources

NepOOCcomprises 545 unique image–source pairs, each yielding one pristine and one OOC image–caption pair, for a total of 1,090 samples\. The dataset covers a broad range of domains in Nepali digital media, including public affairs, elections, health crises, natural disasters, infrastructure claims, and social controversy\. Sources were drawn from three categories: fact\-checking organisations operating in Nepal, Nepali online news portals, and social media archives flagged for potential misinformation\. For all source types, we documented both the false claim and the verified true context from the same evidentiary reference, enabling direct derivation of OOC and pristine labels without post\-hoc re\-annotation\.[TableII](https://arxiv.org/html/2608.19212#S3.T2)provides the source breakdown\. Fact\-checking organisations constitute the dominant source \(86\.1%\), reflecting a deliberate prioritisation of expert\-verified ground truth over scale\. We acknowledge this introduces a*source bias*toward professionally curated content; see[SectionIX](https://arxiv.org/html/2608.19212#S9)\.

TABLE II:Source distribution \(n=545n=545unique images\)\.Source TypeCount%Fact\-checking organisations46986\.1%Social media archives499\.0%Online news portals275\.0%Total545100\.0%
### III\-BAnnotation Protocol and Inter\-Annotator Agreement

Binary OOC labels for fact\-checker\-sourced pairs were established by the fact\-checkers themselves as part of their published verdicts, constituting expert\-verified ground truth\. For pairs sourced from news portals and social media archives \(n=76n=76; 14\.0% of unique sources\), two trained annotators independently confirmed the binary OOC/Pristine classification; inter\-annotator agreement on this binary task reached Cohen’sκ=0\.81\\kappa=0\.81\. All annotators independently assigned one of five typology labels \([SectionIII\-C](https://arxiv.org/html/2608.19212#S3.SS3)\) to each OOC instance, using published fact\-checker analyses as reference material\. Disagreements were adjudicated by a third expert; pairs with unresolvable disagreement or insufficient provenance were discarded\. Typology inter\-annotator agreement reached Cohen’sκ=0\.84\\kappa=0\.84, indicating strong reliability\.

### III\-COOC Typology Schema

Each OOC instance is annotated with one of five typology labels reflecting distinct classes of media manipulation observed in Nepali discourse:

1. 1\)Fabricated— the caption describes an event or situation that did not occur; the authentic image lends false visual credibility to the invented claim\.
2. 2\)Miscaptioned— the caption is factually inaccurate but does not involve a systematic mismatch of time, location, or identity\.
3. 3\)Temporal mismatch— an authentic image from one time period is recirculated with a caption implying a different, typically more recent, time\.
4. 4\)Geographic mismatch— an authentic image captured at one location is presented as depicting a different location\.
5. 5\)Identity mismatch— an authentic image of one person or group is captioned as depicting a different person or group\.

[TableIII](https://arxiv.org/html/2608.19212#S3.T3)shows the typology distribution\. Fabricated instances are the largest category \(54\.9%\), reflecting the prevalence of entirely invented captions in Nepali misinformation\. Identity mismatch is the rarest subtype \(1\.8%,n=10n=10\), creating substantial within\-OOC class imbalance\.

TABLE III:Typology distribution \(n=545n=545OOC samples\)\.TypologyCount% of OOCFabricated29954\.9%Miscaptioned13625\.0%Temporal mismatch5610\.3%Geographic mismatch448\.1%Identity mismatch101\.8%Total OOC545100\.0%
### III\-DDataset Statistics and Splits

The final benchmark contains 1,090 image–caption pairs partitioned into train/validation/test splits \(754/108/228\) using stratified sampling\.[TableIV](https://arxiv.org/html/2608.19212#S3.T4)summarises the full dataset statistics\. Captions are predominantly Nepali \(78\.5%\), with English \(14\.5%\) and code\-switched \(7\.0%\) varieties\. Both IAA scores \(κ=0\.84\\kappa=0\.84for typology,κ=0\.81\\kappa=0\.81for binary non\-FC sources\) confirm annotation reliability\.

TABLE IV:Dataset statistics\.AttributeCategoryCount \(%\)LabelPristine545 \(50\.0%\)OOC545 \(50\.0%\)LanguageNepali856 \(78\.5%\)English158 \(14\.5%\)Code\-switched76\(7\.0%\)SplitTrain754 \(69\.2%\)Validation108\(9\.9%\)Test228 \(20\.9%\)Typology IAA \(Cohen’sκ\\kappa\)0\.84Binary IAA, non\-FC \(Cohen’sκ\\kappa\)0\.81
### III\-ERepresentative Samples and Collection Pipeline

[Figure1](https://arxiv.org/html/2608.19212#S2.F1)illustrates representative pristine and OOC pairs across all five typologies\.[Figure2](https://arxiv.org/html/2608.19212#S3.F2)depicts the three\-stage data collection and annotation pipeline\.

![Refer to caption](https://arxiv.org/html/2608.19212v1/DataPipeline.png)Figure 2:Data collection and annotation pipeline: acquisition, verification, annotation, and benchmark output\.
### III\-FEvent\-Cluster Analysis and Leakage Validation

To assess event\-level leakage risk—where related images from the same real\-world incident appear in both train and test—event clusters were constructed in three stages: \(1\) temporal anchor extraction from caption dates; \(2\) named\-entity recognition on captions to group actors and locations; and \(3\) image\-feature clustering using ResNet\-50 embeddings with a cosine similarity threshold of 0\.85\. This procedure identified 36 distinct event clusters \(mean cluster size: 15\.1 samples\)\.

For each model, we compare accuracy under the standard random split against an event\-cluster\-aware split in which all samples from the same cluster are assigned exclusively to either train or test\.[TableV](https://arxiv.org/html/2608.19212#S3.T5)reports the per\-model accuracy differentialΔ=Accrandom−Acccluster\\Delta=\\mathrm\{Acc\}\_\{\\mathrm\{random\}\}\-\\mathrm\{Acc\}\_\{\\mathrm\{cluster\}\}\. All\|Δ\|\|\\Delta\|values fall below1%1\\%, confirming the absence of systematic leakage\.

TABLE V:Leakage validation:Δ=Accrandom−Acccluster\\Delta=\\mathrm\{Acc\}\_\{\\text\{random\}\}\-\\mathrm\{Acc\}\_\{\\text\{cluster\}\}\(mean over 5 seeds\)\. All\|Δ\|<1%\|\\Delta\|<1\\%indicates no leakage\.ModelTypeΔ\\Delta\(%\)StatusmBERTText\-only\+0\.66\+0\.66CleanCNN\+LSTMMultimodal−0\.39\-0\.39CleanViT\+TCNMultimodal\+0\.70\+0\.70CleanResNet\-50\+mBERTMultimodal−0\.66\-0\.66CleanViT\+MuRILMultimodal\+0\.48\+0\.48Clean
### III\-GEthical Considerations

All data were collected from publicly available sources\. No private communications or non\-public information are included\. The dataset will be released under a research\-only non\-commercial licence upon acceptance\.

![Refer to caption](https://arxiv.org/html/2608.19212v1/architecture_overview.png)Figure 3:Five multimodal architectures organised by visual encoder, text encoder, and fusion mechanism\. Double border indicates the best\-performing multimodal model \(ResNet\-50\+mBERT; 94\.65% Macro\-F1\)\.

## IVMethodology

### IV\-ATask Formulation and Preprocessing

Given an image–caption pair\(I,T\)\(I,T\), the task is binary classification into Pristine \(label 0\) or OOC \(label 1\)\. The training objective minimises standard binary cross\-entropy loss overNNtraining samples, whereyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}is the ground truth label andp^i\\hat\{p\}\_\{i\}is the model’s predicted probability for class 1\. Images are resized to model specific resolutions and channel normalised using ImageNet statistics\.111ImageNet normalisation:μ=\[0\.485,0\.456,0\.406\]\\mu=\[0\.485,0\.456,0\.406\],σ=\[0\.229,0\.224,0\.225\]\\sigma=\[0\.229,0\.224,0\.225\]\.No data augmentation is applied\. Captions are tokenised using each model’s native tokeniser: mBERT \(bert\-base\-multilingual\-cased; vocabulary 119,547\), MuRIL \(google/muril\-base\-cased; vocabulary 197,285\), and CLIP’s BPE tokeniser\. All tokenisations are padded or truncated to a maximum sequence length of 128 tokens\.

### IV\-BModel Selection and Architecture Overview

Five models are selected across three distinct architectural dimensions: \(i\) visual encoder type—from\-scratch CNN \(CNN\+LSTM\), supervised convolutional pretraining \(ResNet\-50\), contrastive vision\-language pretraining \(CLIP ViT\-B/32\), and supervised ViT pretraining \(ViT\-B/16\); \(ii\) text encoder type—generic multilingual BERT \(mBERT\), Devanagari\-aware multilingual BERT with LoRA adaptation \(MuRIL\), and lightweight temporal convolution \(TCN, included for stable convergence under limited data\[[41](https://arxiv.org/html/2608.19212#bib.bib41)\]\); and \(iii\) fusion mechanism—late concatenation \(CNN\+LSTM, ResNet\-50\+mBERT\), cross\-attention over patch tokens \(ViT\+TCN, ViT\+MuRIL\), and similarity\-based head \(CLIP\)\. This design enables direct comparison of architectural choices under identical splits, training protocols, and evaluation metrics\.

[Figure3](https://arxiv.org/html/2608.19212#S3.F3)provides a schematic overview of all five architectures\.

### IV\-CArchitecture Specifications

[TableVI](https://arxiv.org/html/2608.19212#S4.T6)summarises key architectural specifications\. CNN\+LSTM uses randomly initialised embeddings and bidirectional LSTM \(256 dimensions per direction\)\. ViT\+TCN employs single\-head cross\-attention over ViT patch tokens\. ResNet\-50\+mBERT uses separate learning rates for visual \(1×10−41\\times 10^\{\-4\}\) and text \(2×10−52\\times 10^\{\-5\}\) encoders\. CLIP keeps both encoders frozen, training only a classification head\. ViT\+MuRIL applies LoRA with rankr=8r=8,α=16\\alpha=16to MuRIL, contributing∼2\.5\{\\sim\}2\.5M trainable parameters\.

TABLE VI:Architecture specifications\. VE=visual encoder, TE=text encoder, Params=trainable parameters\.ModelVETEFusionTE VocabParamsCNN\+LSTM5\-layer CNN \(512\)LSTM \(256/dir\)Late concat119k∼16\{\\sim\}16MViT\+TCNViT\-B/16 \(frozen\)TCN \(3 blocks\)Cross\-attention119k∼90\{\\sim\}90MResNet\-50\+mBERTResNet\-50 \(2048→\\to768\)mBERT \(768\)Late concat119k134MCLIPViT\-B/32 \(frozen\)CLIP BPE \(frozen\)Similarity head49k∼10\{\\sim\}10M \(head\)ViT\+MuRILViT\-B/16 \(frozen\)MuRIL\+LoRA \(r=8r\{=\}8\)Cross\-attention197k∼2\.5\{\\sim\}2\.5M \(LoRA\)
### IV\-DTraining Configuration

All models are trained on the 754/108/228 splits across 5 independent random seeds\{42,123,456,789,2024\}\\\{42,123,456,789,2024\\\}\.[TableVII](https://arxiv.org/html/2608.19212#S4.T7)reports the complete training configuration\. Early stopping is applied based on validation Macro\-F1 with patience 10–12 epochs depending on the model\. Total computational cost was approximately 120 GPU\-hours on NVIDIA T4 GPUs\.

TABLE VII:Training configuration\. LR=learning rate, WD=weight decay, BS=batch size, EP=epochs, Pat\.=early\-stopping patience\.†\\daggerAdaptive LR:1​e−41\\mathrm\{e\}\{\-4\}at≤50%\\leq 50\\%,5​e−55\\mathrm\{e\}\{\-5\}at\>50%\>50\\%\.‡\\ddaggerStepLR: step 10,γ=0\.5\\gamma=0\.5\.ModelOpt\.LRWDBSEPPat\.Sched\.CNN\+LSTMAdam1​e−41\\mathrm\{e\}\{\-4\}1​e−51\\mathrm\{e\}\{\-5\}328010StepLR‡ViT\+TCNAdamW5​e−55\\mathrm\{e\}\{\-5\}1​e−41\\mathrm\{e\}\{\-4\}3210010Cos\+WUResNet\+mBERTAdamWvis:1​e−41\\mathrm\{e\}\{\-4\}txt:2​e−52\\mathrm\{e\}\{\-5\}1​e−41\\mathrm\{e\}\{\-4\}325010Cos\+WUCLIPAdamW†\\dagger0\.05810012—ViT\+MuRILAdamW†\\dagger0\.05810012Cos\+WU

## VExperimental Setup

### V\-AEvaluation Metrics

Accuracy, Macro\-F1, and AUC \(area under the ROC curve\) are reported across all experiments, with Macro\-F1 as the primary metric\. OOC\-class precision and recall are additionally reported, derived from per\-seed average confusion matrices\. All metrics are computed on the held\-out test set \(n=228n=228; 114 Pristine, 114 OOC\), averaged across 5 random seeds with standard deviation \(ddof=1\\mathrm\{ddof\}=1\)\. For the best\-performing models, McNemar’s test with Yates continuity correction is applied to per\-seed predictions;pp\-values, medianppacross seeds, and per\-seed discordant pair counts are reported in[TableIX](https://arxiv.org/html/2608.19212#S6.T9)\.

### V\-BTraining\-Size Scaling Experiment

All five models are trained at four fractions of the training set—\{25%,50%,75%,100%\}\\\{25\\%,50\\%,75\\%,100\\%\\\}—with validation and test sets held fixed\. Each model–fraction combination is repeated across 5 seeds\.

## VIResults

### VI\-AMain Benchmark Results

[TableVIII](https://arxiv.org/html/2608.19212#S6.T8)reports benchmark results averaged across 5 seeds, including dedicated text\-only baselines\. ResNet\-50\+mBERT and text\-only mBERT both achieve94\.65±0\.20%94\.65\\pm 0\.20\\%Macro\-F1\. Statistical equivalence was assessed using McNemar’s test with Yates continuity correction: medianp=1\.000p=1\.000across five seeds \(range\[0\.479,1\.000\]\[0\.479,1\.000\]; 0/5 seeds significant atα=0\.05\\alpha=0\.05\)\. The low mean discordant pair count \(0\.6 per seed\) indicates the two systems make nearly identical errors\.

Similarly, ViT\+MuRIL vs\. ResNet\-50\+mBERT yielded medianp=1\.000p=1\.000\(range\[0\.074,1\.000\]\[0\.074,1\.000\]; 0/5 seeds significant\), and text\-only MuRIL vs\. text\-only mBERT also yielded medianp=1\.000p=1\.000\(range\[0\.074,1\.000\]\[0\.074,1\.000\]; 0/5 seeds significant\)\. Full per\-seed results are in[TableIX](https://arxiv.org/html/2608.19212#S6.T9)\. All results use the random split, whose validity was confirmed in[SectionIII\-F](https://arxiv.org/html/2608.19212#S3.SS6)\(all\|Δ\|<1%\|\\Delta\|<1\\%\)\.

TABLE VIII:Main results on test split \(n=228n=228, mean±\\pmstd over 5 seeds\)\.ModelTypeAcc\. \(%\)Macro\-F1 \(%\)AUCmBERTText\-only94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}0\.9697±0\.0126\\mathbf\{0\.9697\\pm 0\.0126\}MuRILText\-only94\.38±0\.3594\.38\\pm 0\.3594\.38±0\.3594\.38\\pm 0\.350\.9567±0\.01580\.9567\\pm 0\.0158ResNet\-50\+mBERTMultimodal94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}0\.9662±0\.0142\\mathbf\{0\.9662\\pm 0\.0142\}ViT\+MuRILMultimodal93\.33±0\.3793\.33\\pm 0\.3793\.33±0\.3793\.33\\pm 0\.370\.9505±0\.00570\.9505\\pm 0\.0057ViT\+TCNMultimodal92\.11±1\.3592\.11\\pm 1\.3592\.10±1\.3692\.10\\pm 1\.360\.9616±0\.00640\.9616\\pm 0\.0064CNN\+LSTMMultimodal78\.16±10\.9778\.16\\pm 10\.9778\.15±10\.9778\.15\\pm 10\.970\.8548±0\.12030\.8548\\pm 0\.1203CLIPMultimodal69\.39±0\.7269\.39\\pm 0\.7269\.00±0\.7269\.00\\pm 0\.720\.7127±0\.00280\.7127\\pm 0\.0028TABLE IX:McNemar’s test with Yates correction \(α=0\.05\\alpha=0\.05\)\. See text \(§5\.1\) for comparison pairs and discordant counts\.Model AModel BSeedDisc\.bbccχ2\\chi^\{2\}ppSig\.text\-only mBERTResNet\-50\+mBERT421010\.0001\.000×\\times1230000\.0001\.000×\\times4560000\.0001\.000×\\times7892200\.5000\.480×\\times20240000\.0001\.000×\\timesSummary0\.6 meanmedianp=1\.000p=1\.000; 0/5 sig\.ViT\+MuRILResNet\-50\+mBERT421010\.0001\.000×\\times1230000\.0001\.000×\\times4563120\.0001\.000×\\times7891010\.0001\.000×\\times20245053\.2000\.074×\\timesSummary2\.0 meanmedianp=1\.000p=1\.000; 0/5 sig\.text\-only MuRILtext\-only mBERT422110\.5000\.480×\\times1233120\.0001\.000×\\times4560000\.0001\.000×\\times7895053\.2000\.074×\\times20241010\.0001\.000×\\timesSummary2\.2 meanmedianp=1\.000p=1\.000; 0/5 sig\.
### VI\-BOOC\-Class Precision and Recall

[TableX](https://arxiv.org/html/2608.19212#S6.T10)extends the main results with OOC\-class precision and recall\. ResNet\-50\+mBERT achieves the highest OOC Precision \(95\.37±0\.38%95\.37\\pm 0\.38\\%\) and OOC F1 \(94\.61±0\.19%94\.61\\pm 0\.19\\%\)\. The 0\.00% standard deviation reported for OOC Recall in ResNet\-50\+mBERT reflects rounding to two decimal places; unrounded values confirm non\-zero variance\. CLIP shows severely imbalanced precision–recall \(Precision 74\.97%, Recall 58\.25%\), indicating systematic misclassification of OOC instances as Pristine\. CNN\+LSTM exhibits high variance across all OOC metrics, with per\-seed F1 ranging from approximately 57% to 93%, rendering it unsuitable for reliable deployment\.

TABLE X:OOC\-class metrics at 100% training data \(mean±\\pmstd over 5 seeds\)\.ModelMacro\-F1 \(%\)OOC Prec\. \(%\)OOC Rec\. \(%\)OOC F1 \(%\)CNN\+LSTM78\.15±10\.9778\.15\\pm 10\.9779\.23±11\.2779\.23\\pm 11\.2776\.32±11\.4276\.32\\pm 11\.4277\.72±11\.1977\.72\\pm 11\.19ViT\+TCN92\.10±1\.3692\.10\\pm 1\.3692\.20±2\.8192\.20\\pm 2\.8192\.11±1\.9692\.11\\pm 1\.9692\.11±1\.2492\.11\\pm 1\.24CLIP69\.00±0\.7269\.00\\pm 0\.7274\.97±1\.3474\.97\\pm 1\.3458\.25±1\.3358\.25\\pm 1\.3365\.54±0\.8765\.54\\pm 0\.87ResNet\-50\+mBERT94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}95\.37±0\.38\\mathbf\{95\.37\\pm 0\.38\}93\.86±0\.00†\\mathbf\{93\.86\\pm 0\.00\}^\{\\dagger\}94\.61±0\.19\\mathbf\{94\.61\\pm 0\.19\}ViT\+MuRIL93\.33±0\.3793\.33\\pm 0\.3794\.14±1\.2294\.14\\pm 1\.2292\.46±1\.7192\.46\\pm 1\.7193\.27±0\.4393\.27\\pm 0\.43
### VI\-CTraining\-Size Scaling

[TableXI](https://arxiv.org/html/2608.19212#S6.T11)and[Figure4](https://arxiv.org/html/2608.19212#S6.F4)report Macro\-F1 at each training fraction\. ResNet\-50\+mBERT achieves the highest Macro\-F1 at every fraction and exhibits a narrow scaling gain \(93\.15%→94\.65%93\.15\\%\\to 94\.65\\%from 25% to 100%\), indicating its pretrained representations transfer well at low data volumes\. ViT\+TCN is highly sensitive to training set size, dropping sharply to69\.25±11\.31%69\.25\\pm 11\.31\\%at 25%\. CLIP shows the smallest absolute gain across the full range \(62\.01%→69\.00%62\.01\\%\\to 69\.00\\%\), consistent with its contrastive objective being poorly aligned with supervised binary classification on this domain\.

TABLE XI:Scaling results: Macro\-F1 \(mean±\\pmstd over 5 seeds per fraction\)\.Model25%50%75%100%CNN\+LSTM55\.22±4\.8055\.22\\pm 4\.8061\.65±4\.9961\.65\\pm 4\.9967\.67±9\.5167\.67\\pm 9\.5178\.15±10\.9778\.15\\pm 10\.97ViT\+TCN69\.25±11\.3169\.25\\pm 11\.3189\.12±1\.0589\.12\\pm 1\.0590\.87±2\.5790\.87\\pm 2\.5792\.10±1\.3692\.10\\pm 1\.36CLIP62\.01±1\.2562\.01\\pm 1\.2563\.45±2\.2763\.45\\pm 2\.2766\.14±1\.4266\.14\\pm 1\.4269\.00±0\.7269\.00\\pm 0\.72ResNet\-50\+mBERT93\.15±0\.91\\mathbf\{93\.15\\pm 0\.91\}93\.77±1\.10\\mathbf\{93\.77\\pm 1\.10\}94\.38±0\.79\\mathbf\{94\.38\\pm 0\.79\}94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}ViT\+MuRIL90\.61±0\.6790\.61\\pm 0\.6792\.72±0\.6792\.72\\pm 0\.6793\.33±0\.7293\.33\\pm 0\.7293\.33±0\.3793\.33\\pm 0\.37![Refer to caption](https://arxiv.org/html/2608.19212v1/fig4_scaling_curves.png)Figure 4:Macro\-F1 vs\. training fraction with standard deviation bands over 5 seeds\.

## VIIAnalysis

### VII\-AModality Ablation

[TableXII](https://arxiv.org/html/2608.19212#S7.T12)reports Macro\-F1 for text\-only, image\-only, and multimodal configurations\. Image\-only performance falls at or below chance \(33\.3333\.33–49\.83%49\.83\\%\) across all models\. Text\-only configurations approach or match multimodal performance: ViT\+MuRIL’s text\-only result \(93\.95±0\.84%93\.95\\pm 0\.84\\%\) surpasses its multimodal counterpart \(93\.33±0\.37%93\.33\\pm 0\.37\\%\) by 0\.62 pp, and ResNet\-50\+mBERT gains zero from visual features\.

TABLE XII:Modality ablation: Macro\-F1 at 100% training data \(mean±\\pmstd over 5 seeds\)\.ModelText\-only\(%\)Image\-only\(%\)Multimodal\(%\)CNN\+LSTM74\.91±16\.8074\.91\\pm 16\.8049\.80±0\.2749\.80\\pm 0\.2778\.15±10\.9778\.15\\pm 10\.97ViT\+TCN90\.87±1\.8090\.87\\pm 1\.8033\.33±0\.0033\.33\\pm 0\.0092\.10±1\.3692\.10\\pm 1\.36CLIP67\.48±1\.0667\.48\\pm 1\.0649\.20±0\.4849\.20\\pm 0\.4869\.00±0\.7269\.00\\pm 0\.72ResNet\+mBERT94\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}49\.00±0\.9649\.00\\pm 0\.9694\.65±0\.20\\mathbf\{94\.65\\pm 0\.20\}ViT\+MuRIL93\.95±0\.8493\.95\\pm 0\.8449\.83±0\.1849\.83\\pm 0\.1893\.33±0\.3793\.33\\pm 0\.37
### VII\-BConfusion Matrix Analysis and Typology\-Level Performance

[TableXIII](https://arxiv.org/html/2608.19212#S7.T13)presents per\-seed average confusion matrices\. ResNet\-50\+mBERT misclassifies on average only 5\.2 Pristine samples as OOC and 7\.0 OOC samples as Pristine, achieving the best balance\.

TABLE XIII:Per\-seed mean confusion matrices on test split \(n=228n=228; 114 Pristine, 114 OOC\)\.ModelActualPred:PristinePred:OOCCNN\+LSTMPristine91\.222\.8OOC27\.087\.0ViT\+TCNPristine105\.09\.0OOC9\.0105\.0CLIPPristine91\.822\.2OOC47\.666\.4ResNet\-50\+mBERTPristine108\.85\.2OOC7\.0107\.0ViT\+MuRILPristine107\.46\.6OOC8\.6105\.4[TableXIV](https://arxiv.org/html/2608.19212#S7.T14)reports per\-typology OOC Macro\-F1\. Models perform strongly on dominant typologies: ResNet\-50\+mBERT achieves98\.18%98\.18\\%on Miscaptioned and95\.80%95\.80\\%on Fabricated\. Due to severe class imbalance, Temporal, Geographic, and Identity mismatches are grouped as “Other Mismatches” \(n=24n=24\); individual Identity Mismatch results \(n=2n=2test instances\) are not reliable\. ViT\+TCN shows elevated variance on Other Mismatches due to its sensitivity to training set size\.

TABLE XIV:Per\-typology OOC Macro\-F1 at 100% training data \(mean±\\pmstd over 5 seeds\)\.nn=OOC test instances per typology\.TypologynnCNN\+LSTMViT\+TCNCLIPResNet\-50\+mBERTViT\+MuRILFabricated6283\.93±7\.5783\.93\\pm 7\.5796\.14±0\.9996\.14\\pm 0\.9974\.49±1\.0674\.49\\pm 1\.0695\.80±1\.00\\mathbf\{95\.80\\pm 1\.00\}95\.97±0\.3995\.97\\pm 0\.39Miscaptioned2887\.92±8\.9287\.92\\pm 8\.9295\.09±2\.2395\.09\\pm 2\.2368\.53±1\.7068\.53\\pm 1\.7098\.18±0\.00†\\mathbf\{98\.18\\pm 0\.00\}^\{\\dagger\}95\.89±1\.3795\.89\\pm 1\.37Other Mismatches‡2489\.01±9\.5089\.01\\pm 9\.5091\.43±4\.2191\.43\\pm 4\.2174\.81±4\.4274\.81\\pm 4\.4297\.79±0\.00†\\mathbf\{97\.79\\pm 0\.00\}^\{\\dagger\}96\.38±3\.1696\.38\\pm 3\.16
†\\dagger0\.00% std reflects rounding; unrounded values confirm variance\.‡\\ddaggerIncludes Temporal, Geographic, and Identity mismatches \(n=2n=2Identity test samples; highly unstable\)\.

### VII\-CPrecision–Recall and ROC Curves

[Figure5](https://arxiv.org/html/2608.19212#S7.F5)presents Precision–Recall and ROC curves for all five models using the best\-performing seed per model\. The divergence between AUC and Macro\-F1 for CLIP \(AUC=0\.7127\\mathrm\{AUC\}=0\.7127,F1=69\.00%\\mathrm\{F1\}=69\.00\\%\) illustrates the importance of reporting both metrics; AUC alone would overstate CLIP’s suitability\. The PR curves also suggest that threshold tuning could recover additional recall for CLIP at the cost of precision\. Notably, text\-only mBERT achieves the highest AUC overall \(0\.9697±0\.01260\.9697\\pm 0\.0126\), exceeding ResNet\-50\+mBERT \(0\.9662±0\.01420\.9662\\pm 0\.0142\), providing further metric\-level evidence for text sufficiency\.

![Refer to caption](https://arxiv.org/html/2608.19212v1/pr_roc_curves.png)Figure 5:Precision–Recall and ROC curves for all five models on test split \(n=228n=228\), best seed per model\.
### VII\-DFailure Case Analysis

ResNet\-50\+mBERT produces 13 unique failure cases across five seeds \(5\.7% error rate\), spanning three dominant patterns: Fabricated claims \(5 cases\), Geographic mismatches \(1 case\), and Miscaptioned content \(1 case\), plus 6 false positives\. All failures share a common root cause: the closed\-pair constraint prevents access to external fact\-checking, knowledge bases, and visual grounding beyond the dataset scale\.

Fabricated Claims \(5/13\):Captions employ plausible administrative or political language \(US–China grants, infrastructure projects, leader accusations\), while images depict authentic, unmanipulated contexts\. Without an external knowledge base or evidence corpus, mBERT cannot refute invented specific claims\.

Geographic Mismatches \(1/13\):An image from Location A is paired with a caption attributing it to Location B\. The location name is internally consistent and syntactically plausible, preventing text\-only detection\. With only 8\.1% geographic mismatch representation in training and no geographic metadata available, the visual encoder cannot learn location\-specific grounding\.

Miscaptioned Content \(1/13\):Subtle attribution errors where image and caption are domain\-related but the specific event claim is false\. mBERT finds the caption semantically plausible; resolving such cases requires fine\-grained entity grounding and knowledge of Nepali institutional context\.

False Positives \(6/13\):Pristine fact\-checking articles that re\-state false claims during debunking are misclassified as OOC\. Without access to source context or article structure, the model cannot distinguish “reporting a false claim” from “making one\.”

Representative qualitative examples are provided in[Representative Failure Case Examples](https://arxiv.org/html/2608.19212#Ax1)\.

## VIIIDiscussion

### VIII\-AText\-Sufficiency Finding

Results onNepOOCdemonstrate that caption text alone achieves strong performance on this benchmark\. Text\-only mBERT attains94\.65±0\.20%94\.65\\pm 0\.20\\%Macro\-F1, matching the best multimodal system exactly\. McNemar’s test confirms statistical equivalence across all five seeds \(medianp=1\.000p=1\.000, 0/5 seeds significant atα=0\.05\\alpha=0\.05\)\. The low mean discordant pair count \(0\.6 per seed\) indicates the systems make nearly identical errors rather than complementary ones\. These results suggest that caption semantics carry sufficient signal for high performance within the current dataset scale\.

### VIII\-BWhy Multimodal Gains Remain Limited

Image\-only baselines perform at or below chance \(33–50%\) across all architectures, indicating that visual features alone provide negligible discriminative signal\. The ablation results show that adding visual information to text\-only models yields no measurable benefit: text\-only mBERT and ResNet\-50\+mBERT differ by 0\.00 percentage points, and text\-only ViT\+MuRIL actually surpasses its multimodal variant by 0\.62 pp\. This pattern holds across both traditional concatenation\-based fusion \(ResNet\) and sophisticated cross\-attention mechanisms \(ViT\+MuRIL\)\.

Among multimodal systems, ResNet\-50\+mBERT achieves the highest Macro\-F1 \(94\.65%94\.65\\%\), possibly because its deep convolutional features and late fusion with separate learning rates stabilise training on limited data\. More complex alternatives \(ViT\+TCN, ViT\+MuRIL, CLIP\) show either reduced performance or higher variance\. This hierarchy suggests that fusion sophistication does not compensate for limited training data\.

### VIII\-CDataset Scaling vs\. Architectural Scaling

Training\-size scaling experiments reveal that data volume drives performance more directly than architectural novelty\. ResNet\-50\+mBERT achieves the highest Macro\-F1 at every data fraction \(25–100%\) and shows minimal growth \(93\.15% to 94\.65%\), indicating strong transfer from pretraining\. By contrast, ViT\+TCN drops to 69\.25% at 25% data, showing dramatic data sensitivity\. CLIP exhibits the flattest curve \(62\.01% to 69\.00%\), reflecting misalignment between its contrastive pretraining objective and supervised binary classification\.

These findings support the conclusion that expanding the dataset is a more promising research direction than refining model architecture for this task\. At 754 training samples, pretrained text encoders already saturate performance; additional improvements are more likely to come from larger, more diverse training sets than from architectural innovation\.

### VIII\-DResource Limitations and Future Implications

A key secondary finding is that Devanagari\-specialised pretraining via LoRA\-adapted MuRIL provides no advantage over general\-purpose mBERT\. Both text\-only encoders differ by only 0\.27 pp, and the multimodal ViT\+MuRIL trails ResNet\-50\+mBERT by 1\.32 pp\. This suggests that strong generic multilingual encoders already cover Nepali text adequately, even without script\-specific adaptation\.

The closed\-pair constraint \(no external knowledge or evidence retrieval\) limits performance on minority typologies: Temporal, Geographic, and Identity mismatches comprise 20% of OOC cases but require world knowledge unavailable within the image–caption pair alone\. Fabricated claims \(55% of OOC\) similarly demand external fact\-checking\. Addressing these challenges requires retrieval\-augmented architectures, as demonstrated by recent work \(RED\-DOT, EXCLAIM\)\. This motivates future work toward evidence\-grounded detection pipelines that integrate knowledge bases and retrieved documents\.

## IXLimitations

Dataset scale\.At 1,090 samples,NepOOCis substantially smaller than high\-resource benchmarks\. A single misclassification on the 228\-sample test set shifts Macro\-F1 by approximately 0\.4 percentage points, limiting statistical power to detect small differences\. Typology\-level findings for Identity mismatch \(n=2n=2OOC test samples\) are not reliable\.

Text\-dominant signal\.Caption semantics appear to carry nearly all discriminative information at the current dataset scale, limiting conclusions about cross\-modal fusion\. Whether this finding holds at larger scales or with different source distributions remains to be tested\.

Source and topical bias\.Sources are concentrated among fact\-checking organisations \(86\.1%\) covering salient events, introducing skew toward professionally curated content\. High overall performance may partly reflect lexical patterns specific to this source distribution\. Future work should quantify per\-source error rates and test generalisation to organically\-produced social media content\.

Closed\-pair constraint\.All models observe only the image–caption pair, with no access to external evidence, retrieval, knowledge bases, or metadata\.

Explainability\.Attention\-based and gradient\-based visualisations are diagnostic heuristics rather than faithful causal explanations\[[25](https://arxiv.org/html/2608.19212#bib.bib25)\]\.

Temporal validity\.OOC manipulation strategies evolve; tactics prevalent at data collection may be superseded\. Periodic dataset refresh should be considered as the misinformation landscape changes\.

## XConclusion

We introduceNepOOC, the first publicly available Nepali\-dominant multilingual OOC benchmark, with 1,090 image–caption pairs annotated across five typologies \(Cohen’sκ=0\.84\\kappa=0\.84\) and partitioned into train/validation/test splits of 754/108/228\.

Systematic evaluation yields three principal contributions\. First, a text\-sufficiency finding: caption semantics prove sufficient for strong classification performance, as text\-only mBERT is statistically indistinguishable from the best multimodal system\. Second, image\-only models perform at chance, indicating visual features provide minimal signal at this data scale\. Third, training\-size scaling experiments identify dataset expansion, rather than architectural refinement, as the primary lever for progress\.

These results challenge two common assumptions in low\-resource vision\-language research: that multimodal fusion consistently benefits text\-only encoders at small data scales, and that script\-specialised encoders are necessary for non\-Latin scripts\. Within theNepOOCsetting, they establish text\-only baselines as an essential evaluation practice and highlight the importance of data\-centric approaches in low\-resource misinformation detection\.

Future work should prioritise dataset expansion to 5,000–10,000 samples with broader event and source coverage, retrieval\-augmented architectures to address knowledge\-dependent typologies, and cross\-lingual extension to neighbouring Devanagari languages to assess generalisation\.

## XIFuture Work

Three gaps remain critical to address\.

Dataset expansion and diversity\.ScalingNepOOCto 5,000–10,000 samples with broader event, domain, and source coverage would stress\-test predicted scaling advantages and address minority typology sparsity\. Expansion should prioritise underrepresented typologies and sources beyond fact\-checking organisations to reduce source bias and test whether text\-sufficiency holds for organically\-produced social media content\.

Retrieval\-augmented detection\.Building a Nepali evidence corpus and integrating retrieval into the detection pipeline, following RED\-DOT\[[10](https://arxiv.org/html/2608.19212#bib.bib10)\]and EXCLAIM\[[12](https://arxiv.org/html/2608.19212#bib.bib12)\], would address open problems identified in[SectionVII\-D](https://arxiv.org/html/2608.19212#S7.SS4)\. Targeted experiments for minority typologies using entity linking, face recognition, and geolocation augmentation are needed\.

Cross\-lingual extension and generalisation\.Extending the benchmark to neighbouring low\-resource South Asian languages sharing Devanagari would test whether text\-sufficiency and encoder\-adequacy findings generalise beyond Nepali\. Cross\-dataset evaluation on English OOC benchmarks and human baseline measurement would further establish the scope of transfer\.

## Data Availability

NepOOCwill be publicly released upon publication under a research\-only non\-commercial licence, including image–caption pairs, binary and typology labels, train/validation/test splits, event\-cluster assignments, and inter\-annotator agreement logs\.

Code and Implementation:The benchmark evaluation code, model implementations, and preprocessing scripts are available at:[https://github\.com/SanjeevKCodes/nepooc](https://github.com/SanjeevKCodes/nepooc)

## References

- \[1\]Y\. Wang*et al\.*, “EANN: Event adversarial neural networks for multi\-modal fake news detection,” in*Proc\. ACM SIGKDD*, 2018\.
- \[2\]G\. Luo*et al\.*, “NewsCLIPpings: Automatic generation of out\-of\-context multimodal media,” in*Proc\. EMNLP*, 2021\.
- \[3\]S\. Aneja*et al\.*, “COSMOS: Catching out\-of\-context misinformation with self\-supervised learning,”*IEEE Trans\. Multimedia*, vol\. 24, pp\. 3518–3529, 2022\.
- \[4\]S\. Papadopoulos*et al\.*, “VERITE: A robust benchmark for multimodal misinformation detection accounting for unimodal bias,”*Int\. J\. Multimedia Inf\. Retrieval*, vol\. 13, no\. 2, 2024\.
- \[5\]S\. Abdelnabi*et al\.*, “Open\-domain, content\-based, multi\-modal fact\-checking of out\-of\-context images via online resources,” in*Proc\. IEEE/CVF CVPR*, 2022\.
- \[6\]D\. S\. Nielsen and R\. McConville, “MuMiN: A large\-scale multilingual multimodal fact\-checked misinformation social network dataset,” in*Proc\. SIGIR*, 2022\.
- \[7\]P\. Yang*et al\.*, “Multi\-modal transformer for fake news detection,”*Math\. Biosci\. Eng\.*, vol\. 20, no\. 8, pp\. 14699–14717, 2023\.
- \[8\]P\. Qi*et al\.*, “Improving fake news detection by using an entity\-enhanced framework to fuse diverse multimodal clues,” in*Proc\. ACM Multimedia*, 2021\.
- \[9\]Y\. Zhang*et al\.*, “Interpretable detection of out\-of\-context misinformation with neural\-symbolic\-enhanced large multimodal model,”*arXiv:2304\.07633*, 2023\.
- \[10\]S\.\-I\. Papadopoulos*et al\.*, “RED\-DOT: Multimodal fact\-checking via relevant evidence detection,” 2024\.
- \[11\]J\. Tonglet*et al\.*, “COVE: Context and veracity prediction for out\-of\-context images,” 2025\.
- \[12\]Y\. Wu*et al\.*, “EXCLAIM: Exposing out\-of\-context misinformation via multi\-agent retrieval,”*arXiv preprint \(not peer\-reviewed\)*, 2025\.
- \[13\]A\. Vaswani*et al\.*, “Attention is all you need,” in*Adv\. Neural Inf\. Process\. Syst\.*, 2017\.
- \[14\]J\. Devlin*et al\.*, “BERT: Pre\-training of deep bidirectional transformers for language understanding,” in*Proc\. NAACL\-HLT*, 2019\.
- \[15\]A\. Conneau*et al\.*, “Unsupervised cross\-lingual representation learning at scale,” 2020\.
- \[16\]A\. Dosovitskiy*et al\.*, “An image is worth16×1616\{\\times\}16words: Transformers for image recognition at scale,” in*Proc\. ICLR*, 2021\.
- \[17\]A\. Radford*et al\.*, “Learning transferable visual models from natural language supervision,” in*Proc\. ICML*, 2021\.
- \[18\]S\. Khanuja*et al\.*, “MuRIL: Multilingual representations for Indian languages,”*arXiv:2103\.10730*, 2021\.
- \[19\]U\. Maskey*et al\.*, “Nepali encoder transformers: Analysis of auto\-encoding transformer models for Nepali text classification,” 2022\.
- \[20\]S\. Timilsina*et al\.*, “NepBERTa: Nepali language model trained on a large corpus,” 2022\.
- \[21\]S\. Pudasaini*et al\.*, “NepaliBERT: Pre\-training of masked language model in Nepali corpus,” 2023\.
- \[22\]K\. He*et al\.*, “Deep residual learning for image recognition,” in*Proc\. IEEE CVPR*, 2016\.
- \[23\]S\. Tahmasebi*et al\.*, “Multimodal misinformation detection using large vision\-language models,” 2024\.
- \[24\]N\. A\. S\. Eldien*et al\.*, “Interpreting multimodal fake news detection models: Performance factors and modality contributions,” 2026\.
- \[25\]S\. Jain and B\. C\. Wallace, “Attention is not explanation,” in*Proc\. NAACL\-HLT*, 2019\.
- \[26\]E\. J\. Hu*et al\.*, “LoRA: Low\-rank adaptation of large language models,” in*Proc\. ICLR*, 2022\.
- \[27\]S\. Adhikari, “Fake news, misinformation and social media: New security challenges in Nepali context,” 2025\.
- \[28\]R\. Dahal and U\. Acharya, “An anatomy of information manipulation in Nepal,” in*Nepal’s Misinformation Landscape*, 2025\.
- \[29\]R\. R\. Selvaraju*et al\.*, “Grad\-CAM: Visual explanations from deep networks via gradient\-based localization,” in*Proc\. IEEE ICCV*, 2017\.
- \[30\]T\. B\. Shahi and A\. K\. Pant, “Nepali news classification using Naive Bayes, SVM, and neural networks,” 2018\.
- \[31\]S\. S\. Wagle and S\. Thapa, “Comparative analysis of Nepali news classification using LSTM, Bi\-LSTM, and transformer models,” 2021\.
- \[32\]P\. Adhikari and A\. Timalsina, “Nepali news classification using LSTM\-CNN hybrid model,” 2023\.
- \[33\]J\. Lv*et al\.*, “Multimodal fake news detection: A comprehensive survey on deep learning advances and challenges,” 2025\.
- \[34\]A\. Wilson*et al\.*, “Multimodal analysis of dis\-/misinformation,”*Royal Society Open Science*, 2023\.
- \[35\]B\. Anupam, “Nepal’s misinformation domain: Comparing mainstream journalists and content creators,”*PCN Journal*, 2025\.
- \[36\]S\. Duval*et al\.*, “Evidence\-grounded multimodal misinformation detection with attention\-based GNNs,”*arXiv:2505\.18221 \(not peer\-reviewed\)*, 2025\.
- \[37\]P\. Qi*et al\.*, “SNIFFER: Multimodal large language model for explainable out\-of\-context misinformation detection,”*arXiv:2403\.03170*, 2024\.
- \[38\]M\. Kananian*et al\.*, “GraMuFeN: Graph\-based multi\-modal fake news detection in social media,”*arXiv:2310\.07668*, 2023\.
- \[39\]M\. Vedaksha*et al\.*, “Multimodal and multilingual fake news detection using MuRIL and vision transformers with explainable AI,”*Preprints \(not peer\-reviewed\)*, 2026, doi:10\.20944/preprints202602\.0333\.v1\.
- \[40\]P\. Giri*et al\.*, “AI\-powered fake news detection tool for Nepali media,”*Research Square preprint*, doi:10\.21203/rs\.3\.rs\-6778598/v1, 2025\.
- \[41\]S\. Bai, J\. Z\. Kolter, and V\. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,”*arXiv:1803\.01271*, 2018\.

## Representative Failure Case Examples

OOC instances misclassified as Pristine by ResNet\-50\+mBERT\.

Case 1 \[ FABRICATED \]

![[Uncaptioned image]](https://arxiv.org/html/2608.19212v1/KH_202003_fakenews_us_china.jpg)Post IDKH\_202003\_fakenews\_us\_chinaTypologyFabricatedGT / PredOOC→\\;\\rightarrow\\;Pristine\(conf\. 0\.58\)Caption\\nepalifontअमेरिकाले नेपाललाई ५० करोड डलर सहायता दिने\! उद्देश्य चीनलाई घेर्ने। यो कति ठूलो षड्यन्त्र हो\! सबैलाई शेयर गर्नुहोस्।\(US to give Nepal $500M aid\! Purpose: to surround China\. What a massive conspiracy\! Share with everyone\.\)Why it failsThe caption uses emotionally provocative geopolitical language typical of conspiracy\-themed posts\. mBERT cannot refute the invented grant figure or foreign policy claim without an external knowledge base\.
Case 2 \[ GEOGRAPHIC MISMATCH \]

![[Uncaptioned image]](https://arxiv.org/html/2608.19212v1/Dept_road_image.jpeg)Post IDQT\_20240303\_1TypologyGeographic mismatchGT / PredOOC→\\;\\rightarrow\\;Pristine\(conf\. 0\.59\)Caption\\nepalifontयो हो भारतको विकास\! सडक विभागकै अगाडि यस्तो सडकको अवस्था। शेयर गरौं सबैलाई देखाउन।\(This is India’s development\! Road condition like this right in front of the Road Department\.\)Why it failsThe caption’s location attribution is syntactically plausible\. With only 8\.1% geographic mismatch in training, the visual encoder cannot learn location\-specific grounding\.
Case 3 \[ MISCAPTIONED \]

![[Uncaptioned image]](https://arxiv.org/html/2608.19212v1/NFC_chhabi_dabi_kantipur.png)Post IDNFC\_chhabi\_dabi\_kantipurTypologyMiscaptionedGT / PredOOC→\\;\\rightarrow\\;Pristine\(conf\. 0\.61\)Caption\\nepalifontब्रेकिंग: सर्वोच्च सहकारी घोटाला मुद्दामा ठूलो निर्णय\! उच्च अदालत तुलसीपुर\-बुटवलले मुद्दा लगाउमा राख्ने आदेश दिएको कान्तिपुरले जनायो।\(Breaking: Big decision in Supreme Cooperative scam\! Kantipur reports High Court Tulsipur\-Butwal ordered case placed\.\)Why it failsSubtle legal terminology differences are imperceptible to text\-only models without institutional knowledge of Nepali court systems\.
Note:GT = ground truth; Pred = predicted label; conf\. = prediction confidence\.