The Role of Natural Language Understanding in Multimodal Video-Based Dengue Diagnosis
Summary
This paper proposes a YOLO- and CLIP-based vision-language framework to classify mosquito flight frames for Dengue virus detection, achieving 98.54% accuracy and 99.91% sensitivity at frame level, with complete video-level performance after temporal aggregation.
View Cached Full Text
Cached at: 08/14/26, 09:27 AM
# The Role of Natural Language Understanding in Multimodal Video-Based Dengue Diagnosis Source: [https://arxiv.org/abs/2608.12677](https://arxiv.org/abs/2608.12677) [View PDF](https://arxiv.org/pdf/2608.12677) > Abstract:Detecting infection\-related behavioral changes in mosquitoes from video data is challenging because mosquitoes are small, move rapidly and irregularly, and are affected by environmental factors such as background, lighting, and shadows, which can make reliable feature extraction difficult\. In this study, a YOLO\- and Contrastive Language\-Image Pre\-training \(CLIP\)\-based vision\-language framework is proposed to classify mosquito flight frames of uninfected and Dengue virus serotype 2 \(DENV2\)\-infected mosquitoes\. First, YOLO is used to isolate mosquito regions from the background\. Then, visual features extracted from video frames are aligned with biologically meaningful textual prompts in a shared embedding space\. The multimodal model was fine\-tuned using supervised bidirectional contrastive learning and evaluated through frame\-level image\-text similarity\-based classification\. The results show that the proposed method achieved 98\.54% accuracy and 99\.91% sensitivity at the frame level\. After temporal aggregation of frame\-level information, the model achieved complete video\-level performance\. The ablation results showed that fine\-tuning and CLIP\-based representations were essential for this domain, while the textual branch provided semantic image\-text alignment rather than an accuracy advantage over the vision\-only model\. These findings suggest that vision\-language models can provide a useful framework for analyzing infection\-related biological behaviors from video data\. ## Submission history From: Danial Sharifrazi \[[view email](https://arxiv.org/show-email/bb15516b/2608.12677)\] **\[v1\]**Thu, 13 Aug 2026 00:27:22 UTC \(531 KB\)
Similar Articles
Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models
Ultralytics YOLO26 introduces a unified real-time vision model family with NMS-free inference, improved training strategies, and multi-task capabilities for detection, segmentation, and pose estimation, achieving state-of-the-art accuracy-latency trade-offs.
When Vision Speaks for Sound
This paper identifies that video-capable multimodal LLMs often appear to understand audio but actually rely on visual cues, a failure mode termed the audio-visual Clever Hans effect. It introduces Thud, an intervention-driven probing framework to diagnose this issue, and proposes an alignment recipe that improves audio-visual consistency by 28 percentage points.
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
This paper introduces LaViD, a framework that transfers semantic knowledge from a language-only LLM to a vision student model by generating multiple-choice questions as conceptual signatures, achieving superior fine-grained classification performance and robustness.
Leveraging Vision-Language Models to Detect Attention in Educational Videos
This paper explores using a Vision-Language Model (VLM) to detect attention loss in educational videos by combining gaze data with video content, but finds that VLM approaches do not outperform traditional machine learning baselines.
Large Language Models as Unified Multimodal Learners for Clinical Prediction
The paper proposes converting multimodal patient data (text, labs, vitals) into a single natural language sequence and fine-tuning LLMs for clinical prediction, achieving comparable or better performance than specialized fusion architectures across three tasks.