面向任务的认知障碍检测多模态学习

# Task-Centric Multimodal Learning for Cognitive Impairment Detection
Source: [https://arxiv.org/html/2610.20906](https://arxiv.org/html/2610.20906)
Sankalp Vaish1, Rishank Singh1, Sumukh Veeramalla1, Caroline Summerour2, John A\. Batsis2, Xiaohui Liang1Affiliation:Affiliation:1Department of Computer Science, University of Massachusetts Boston, MA, USAAffiliation:2School of Medicine, University of North Carolina at Chapel hill, NC, USAAffiliation:Email: \{Sankalp\.Vaish001, Rishank\.Singh001, Sumukh\.Veeramalla001, Xiaohui\.Liang\}@umb\.edu, Caroline\_Summerour@med\.unc\.edu, John\.Batsis@unc\.edu

September 2025

###### Abstract

Dementia affects millions of older adults worldwide, yet early detection remains difficult outside clinical settings\. Traditional models primarily rely on text or voice\-based responses, capturing only a limited view of human cognition despite the inherently multimodal nature of human expression\. This work proposes a contactless, multimodal screening framework that leverages video, audio, and text signals collected during structured voice\-based interactions with Amazon Alexa\. By incorporating these modalities, our approach captures a more holistic representation of human behavior encompassing speech patterns, vocal characteristics, facial expressions, and linguistic structure, which may enable more accurate detection of cognitive decline\. Different participants completed three conversational tasks designed to probe spontaneous speech, comprehension, and recall\. We pursued three objectives: \(1\) evaluate which of the three Alexa tasks provides the most informative cognitive signals, \(2\) compare the effectiveness of unimodal models, and \(3\) assess the benefits of multimodal fusion models\. Task\-level analysis shows that the video model achieves its best performance on Command\-Reading, with an accuracy of 0\.65 \(AUC 0\.69\), while the audio model performs best on Question\-Answering, reaching an accuracy of 0\.67 \(AUC 0\.73\)\. In contrast, the multimodal model demonstrates consistently stable performance across tasks, achieving an accuracy of 0\.76 \(AUC 0\.75\) on Command\-Reading and 0\.76 \(AUC 0\.73\) on Question\-Answering\. These findings demonstrate that multimodal behavioral cues captured through naturalistic interactions can reliably distinguish MCI from healthy aging, highlighting the limitations of traditional diagnostic paradigms and supporting the shift toward multimodal, real\-world cognitive assessment frameworks\.

© 2026 IEEE\. Personal use of this material is permitted\. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works\.

Accepted for publication at the 2026 IEEE Global Communications Conference \(GLOBECOM\)\.

## IIntroduction

Dementia is a growing global health concern, particularly among older adults, and its prevalence continues to increase with aging populations\. Early detection is crucial, as timely intervention can significantly improve quality of life and inform effective care planning\. Currently, diagnosis is heavily based on standardized clinical assessments, such as the Montreal Cognitive Assessment \(MoCA\)\. Mild cognitive impairment \(MCI\)\[[23](https://arxiv.org/html/2610.20906#bib.bib7)\]is a condition in which people have more memory or thinking problems than other people their age\. Individuals with MCI face an elevated risk of progressing to Alzheimer’s disease or related dementias, with studies estimating that approximately one to two out of every ten adults aged 65 or older with MCI develop dementia each year\.

Despite the growing success of cognitive impairment\[[18](https://arxiv.org/html/2610.20906#bib.bib11),[11](https://arxiv.org/html/2610.20906#bib.bib22)\]detection, most existing studies primarily focus on improving feature representations or fusion strategies, while treating the underlying interaction context as secondary\. This leads to a gap in understanding how different tasks influence the quality and discriminative power of behavioral signals\. To address this, there is a need for a systematic, task\-centric investigation that explicitly analyzes how different interaction paradigms contribute to cognitive assessment\. In real\-world conversational settings, individuals do not produce uniform responses; instead, their speech, language, and behavioral cues vary significantly depending on the cognitive demands imposed by the task\. To account for this variability, we design our experimental framework to explicitly model task\-dependent behavior\. Specifically, instead of treating participant data as homogeneous, we consider three distinct interaction paradigms: Command\-Reading \(task\-1\), Command\-Generation \(task\-2\), and Question\-Answering \(task\-3\), each designed to elicit different cognitive and linguistic processes\.

A substantial amount of prior work has focused on speech\-based dementia detection, examining acoustic, prosodic, and linguistic biomarkers extracted from conversational speech\[[4](https://arxiv.org/html/2610.20906#bib.bib3),[7](https://arxiv.org/html/2610.20906#bib.bib4)\]\. Other studies have extended this line of research by incorporating transcript\-derived linguistic cues alongside audio features\[[8](https://arxiv.org/html/2610.20906#bib.bib1)\]\. While these methods demonstrate the utility of speech for cognitive assessment, they largely overlook additional behavioral channels\. To address this limitation, we incorporate visual behavioral cues by analyzing detailed facial emotion patterns\[[9](https://arxiv.org/html/2610.20906#bib.bib2)\]and Action Unit\[[6](https://arxiv.org/html/2610.20906#bib.bib24)\]derived from video recordings to characterize differences between cognitively healthy and MCI participants\.

In this work, we argue that the nature of the collection task itself plays a central role in determining the sensitivity of behavioral signals to early cognitive decline\. We further investigate whether such early signals can be detected through natural facial behavior, speech, and language use\. In addition, we analyze how these behavioral cues vary across tasks, providing insight into which interaction paradigm is most informative\. Beyond facial cues, we extract audio biomarkers and textual linguistic features, building on prior speech\-driven dementia studies\[[4](https://arxiv.org/html/2610.20906#bib.bib3)\]\. Each modality is first evaluated independently, followed by integration within a unified multimodal framework to assess the benefits of fusion over individual signals\. Our findings show that such multimodal models can detect early cognitive impairment even from short, casual interactions\. This work makes the following key contributions:

1. 1\.A conversational dataset with structured interaction tasks\.We created a dataset collected across multiple sessions through three distinct Alexa\-based tasks\. Unlike traditional clinical datasets, these tasks are designed to capture diverse cognitive behaviors, including natural conversation, semi\-structured interaction, and memory\-based question answering\. This diversity enables the analysis of task\-specific cognitive signals, providing a more realistic setting for home\-based screening\.
2. 2\.Unified multimodal preprocessing pipeline:We develop an end\-to\-end pipeline for extracting standardized features across all three modalities, including \(i\) facial Action Units and frame\-level emotion probabilities from video, \(ii\) acoustic and prosodic biomarkers from participant speech, and \(iii\) linguistic, semantic, and cognitive\-linguistic features from transcripts\. These then act as a input to transformer encoder architecture with multi\-head self\-attention to capture time\-series data\.
3. 3\.Task\-level analysis:We observe that unimodal models are highly task\-sensitive across the three interaction paradigms\. The video model performs better on task\-1 and task\-2, achieving an accuracy of 0\.65 \(AUC 0\.69\) on Task 1, but degrades on Task 3, where it attains an accuracy of 0\.59 \(AUC 0\.59\)\. In contrast, the audio\-based model performs best on task\-2 and task\-3, reaching its highest performance on task\-3 with an accuracy of 0\.67 \(AUC 0\.73\)\.
4. 4\.Comprehensive evaluation of unimodal and multimodal models:We treat each task as a distinct data point and observe consistent trends across modalities\. The video model produces a higher number of Type I errors \(41\) compared to Type II errors \(22\), whereas the audio model exhibits the opposite behavior, with more Type II errors \(46\) than Type I errors \(14\)\. In contrast, the multimodal model reduces both error types to 21 \(Type I\) and 16 \(Type II\), indicating that multimodal fusion effectively leverages complementary predictive strengths of different modalities\.

## IIRelated Work

Early detection of dementia traditionally relies on standardized clinical assessments, which require trained examiners and controlled environments, limiting scalability and frequency of testing\. As a result, automated and contactless approaches have gained significant attention\. Facial behavior, like emotional expressions and action units, offers another rich modality for cognitive health analysis\. Prior studies have demonstrated that facial expressions may help differentiate individuals with dementia from healthy controls\[[9](https://arxiv.org/html/2610.20906#bib.bib2)\]\. However, such work typically treats facial cues as a stand\-alone modality, without integrating them with speech or linguistic features\. To date, no existing study has combined audio, text, and video into a unified multimodal framework for dementia detection, despite increasing evidence that cognitive decline manifests across all three domains\. Recent advances in affective computing and sentiment analysis further highlight the value of multimodal behavioral cues in machine learning systems\[[22](https://arxiv.org/html/2610.20906#bib.bib8),[3](https://arxiv.org/html/2610.20906#bib.bib9)\]\. Building on this, Large Language Models \(LLMs\) have rapidly emerged in healthcare applications, supporting clinical decision\-making, diagnostics, and multimodal reasoning\. These systems leverage domain\-specific pretraining, fine\-tuning, and prompt engineering techniques to interpret diverse data types\[[14](https://arxiv.org/html/2610.20906#bib.bib10)\]\. Despite this progress, the potential of LLMs to reason over multimodal behavioral biomarkers for cognitive impairment remains largely unexplored\. Some research has also incorporated neuroimaging modalities such as MRI and PET scans for dementia diagnosis\[[2](https://arxiv.org/html/2610.20906#bib.bib5),[12](https://arxiv.org/html/2610.20906#bib.bib6)\]\. Although effective, these methods are costly, inaccessible, and unsuitable for home\-based or longitudinal monitoring\. In contrast, behavioral signals from everyday interactions \(speech, facial expressions, and language\) offer a scalable alternative that can be captured unobtrusively through common consumer devices\.

A recent study, Cog\-TiPRO\[[15](https://arxiv.org/html/2610.20906#bib.bib15)\], demonstrated the feasibility of using voice\-assistant\-based speech and linguistic data for longitudinal detection of cognitive decline \[cite\]\. However, that work, like most prior research\[[13](https://arxiv.org/html/2610.20906#bib.bib23),[11](https://arxiv.org/html/2610.20906#bib.bib22)\], omits video\-derived behavioral cues\. Our work advances this direction in three important ways: \(1\) by adding facial expression and Action Unit analysis, capturing non\-verbal indicators that may manifest earlier than speech changes; \(2\) by combining video, audio, and text in a unified multimodal pipeline; and \(3\) by evaluating across multiple structured interaction types and sessions, enabling a more comprehensive and ecologically valid evaluation for real\-world, home\-based screening\.

## IIIMethodology

### III\-APreprocessing

![Refer to caption](https://arxiv.org/html/2610.20906v1/Processing_Pipeline1.png)Fig\. 1:Processing PipelineThe dataset consisted of synchronized audio–video recordings of participant interactions with Alexa, complemented by automatic transcripts generated via Rev\.ai\. All three modalities were preprocessed to extract meaningful behavioral features before their integration into the unimodal and multimodal models\. Figure[1](https://arxiv.org/html/2610.20906#S3.F1)illustrates the pipeline from raw recordings to modality specific feature embeddings and their mapping to various modeling architectures\.

Video Encoder:To address the presence of multiple individuals in each recording \(participant, interviewer, Alexa device, questionnaire holder\), we first applied MiVOLO\[[10](https://arxiv.org/html/2610.20906#bib.bib13)\]to every frame for face detection and age estimation\. This allowed us to isolate the older adult participant across all sessions\.Emotion Features:We employed ResEmoteNet\[[19](https://arxiv.org/html/2610.20906#bib.bib14)\], to estimate frame\-level emotional states \(happy, sad, angry, surprise, neutral, disgust, fear\)\. For each speaking segment, we aggregated these predictions into quantitative descriptors\[[9](https://arxiv.org/html/2610.20906#bib.bib2)\], including: emotion counts, emotion distribution, emotion variability, and frequency of emotion shifts\.Action Unit Features:To capture more granular facial behavior, we used OpenFace\[[1](https://arxiv.org/html/2610.20906#bib.bib16)\]to extract Facial Action Units \(AUs\), which quantify the activation of specific facial muscles \(e\.g\., AU1: Inner Brow Raise, AU12: Lip Corner Pull\)\. These AUs provide a physiologically grounded representation of subtle facial movements\[[6](https://arxiv.org/html/2610.20906#bib.bib24)\]that may correlate with cognitive decline\.

Audio EncoderUsing Parselmouth and Librosa, we analyzed only the participant’s speech segments\[[18](https://arxiv.org/html/2610.20906#bib.bib11),[5](https://arxiv.org/html/2610.20906#bib.bib12)\]\. Two families of features were extracted:Acoustic Features:MFCC statistics, Spectral centroid, Spectral flux, Harmonics\-to\-noise ratio\.Prosodic and Speech Biomarkers:Pitch \(F0\) metrics, Intensity contours, Jitter and shimmer, Speaking rate, Pause and Articulation rate\.

Text Encoder:Transcripts generated by Rev\.ai were processed using NLTK, TextBlob, and Sentence Transformers\[[24](https://arxiv.org/html/2610.20906#bib.bib17),[17](https://arxiv.org/html/2610.20906#bib.bib18)\], to derive these features:Linguistic and Semantic Features:Lexical richness, Syntactic complexity, Semantic coherence, Use of filler words, hesitations, and discourse markers\.Semantic EmbeddingsSentence\-level embeddings were extracted using a Sentence Transformer model, followed by Principal Component Analysis \(PCA\) to retain the top 10 components representing the semantic structure of speech\.

### III\-BModeling

Our modeling framework systematically evaluates the contribution of different tasks using a diverse set of models, ranging from traditional machine learning classifiers to multimodal transformer architectures that capture temporal dynamics in the extracted features from the above\-described pipeline\. We use a standard transformer encoder architecture with multi\-head self\-attention for sequence modeling\[[16](https://arxiv.org/html/2610.20906#bib.bib19)\],\[[21](https://arxiv.org/html/2610.20906#bib.bib20),[20](https://arxiv.org/html/2610.20906#bib.bib21)\]\. We first trained separate models on each of the three tasks to identify which task provides the most informative signals for dementia detection\. We then evaluated unimodal models for each task and compared their performance with a multimodal model that integrates information across tasks and modalities, enabling us to isolate the cognitive and behavioral cues\.

## IVDataset

The dataset was collected from 35 older adult participants through a series of structured interactions with Amazon Alexa\. Each experimental session was divided into three distinct tasks, designed to elicit different forms of verbal engagement:

Command\-Reading \(task\-1\): Participants were provided with a list of 30 predefined questions \(e\.g\., “What is the weather outside?”, “What is today’s date?”\) and instructed to ask Alexa directly\. This task primarily assessed verbal clarity and recall when following straightforward instructions\. Command\-Generation \(task\-2\): Participants were given 30 intent prompts \(e\.g\., weather, date, news\) and asked to independently formulate appropriate questions to pose to Alexa\. This task introduced a higher cognitive load, evaluating language generation, flexibility, and planning\. Question\-Answering \(task\-3\): Alexa presented participants with 30 pre\-recorded questions \(e\.g\., “Name some animals you would find in a zoo”\), and participants were required to respond verbally\. This task targeted comprehension, memory recall, and spontaneous speech production\.

![Refer to caption](https://arxiv.org/html/2610.20906v1/DataTable.png)Fig\. 2:DataFigure[2](https://arxiv.org/html/2610.20906#S4.F2)provides an overview of the dataset structure, illustrating the duration and composition of each experimental session\. One session lasted approximately 35–75 minutes and included all three tasks and was repeated 7 times over a period of 18 months\. The figure also displays the MoCA score serves as the primary ground\-truth labels for cognitive status estimation\. Participants with MoCA scores between 18 and 25 were categorized as having Mild Cognitive Impairment \(MCI\), while those scoring 26 or above were considered cognitively healthy\. Finally, the figure lists all participant IDs and their corresponding session numbers\. No personal or identifiable information is included, ensuring full anonymity and compliance with privacy standards\.

## VResults

### V\-AEmotion Analysis

![Refer to caption](https://arxiv.org/html/2610.20906v1/EmotionCount.png)Fig\. 3:Relationship between Emotion Count and EmotionFigure[3](https://arxiv.org/html/2610.20906#S5.F3)presents a boxplot illustrating the distribution of emotion counts across participants\. The results indicate that individuals in the healthy group \(orange\) exhibit a higher frequency of positive emotions, such as happy and neutral, whereas participants with Mild Cognitive Impairment \(MCI\) \(blue\) display a greater occurrence of negative emotions, including sadness, disgust, and surprise\. This trend suggests that emotional expressivity and valence patterns may serve as behavioral indicators of cognitive decline\.

![Refer to caption](https://arxiv.org/html/2610.20906v1/part_2015_across_acts.png)Fig\. 4:Emotions’ distribution for a MCI ParticipantFigure[4](https://arxiv.org/html/2610.20906#S5.F4)illustrates emotion variation across tasks for a participant with MCI\. Negative affect, particularly sadness \(blue\), is prevalent across tasks, while happiness \(orange\) appears infrequently, suggesting reduced emotional variability and diminished positive affect\. Notably, task 3 shows a marked increase in sadness, likely corresponding to moments when the participant was searching for or formulating an answer, potentially reflecting increased cognitive effort or task\-related frustration\. Overall, the emotional profile appears more constrained and skewed toward negative affect\.

![Refer to caption](https://arxiv.org/html/2610.20906v1/part_2024_across_acts.png)Fig\. 5:Emotions’ distribution for a healthy ParticipantIn contrast, Figure[5](https://arxiv.org/html/2610.20906#S5.F5)shows emotion variation for a cognitively healthy participant, where positive emotions, especially happiness, are more consistently expressed across tasks\. This predominance of positive affect, along with broader emotional variability, may reflect stronger emotional regulation and preserved cognitive functioning\. Even during more cognitively demanding tasks, the participant maintained positive emotional expression\. In particular, task 3 exhibits richer emotional variation compared to the MCI participant, suggesting greater affective flexibility in response to task demands\.

Task 1 shows a notable difference in how each participant initially perceives and engages with the task\. As task complexity increases across tasks, the participant with MCI exhibits reduced emotional expressiveness and diminished variability in affective responses, whereas the healthy participant maintains richer emotional expression\.

### V\-BAudio Analysis

![Refer to caption](https://arxiv.org/html/2610.20906v1/Normalized_Audio_Features_Distributions_by_Cognitive_Status.png)Fig\. 6:Audio Features Distributions by Cognitive StatusThe figure[6](https://arxiv.org/html/2610.20906#S5.F6)illustrates the normalized distributions of key acoustic features for speakers with Mild Cognitive Impairment \(MCI\) and cognitively healthy controls\. Individuals with MCI show lower pitch variability, reduced speaking and articulation rates, and longer pauses, indicating less dynamic prosody and slower, more effortful speech production\. Measures of voice stability \(jitter, shimmer, HNR\) also suggest less stable phonation in the MCI group\. Together, these patterns highlight consistent differences across prosodic, temporal, and voice\-quality features, supporting their potential relevance as markers of early cognitive decline\.

![Refer to caption](https://arxiv.org/html/2610.20906v1/AudioFDistP2015.png)Fig\. 7:Audio Feature distribution for a MCI ParticipantFigure[7](https://arxiv.org/html/2610.20906#S5.F7)illustrates the distribution of some normalized acoustic features across sessions, enabling comparison of different tasks for participant VASP2015 with MCI\. In task\-1, the participant exhibits elevated pitch measures and moderate vocal instability, suggesting increased cognitive load during task initiation\. Further, task\-2 reveals a partial normalization of pitch but continued irregularity in temporal features such as speaking rate and articulation rate, indicating difficulty maintaining consistent speech rhythm during sustained interaction\. By task\-3, pitch variability increases again and hesitation markers, including pauses become more pronounced, maybe because of potential fatigue or diminished cognitive control as the tasks progress\.

![Refer to caption](https://arxiv.org/html/2610.20906v1/AudioFDistP2024.png)Fig\. 8:Audio Feature distribution for a healthy ParticipantFigure[8](https://arxiv.org/html/2610.20906#S5.F8)displays the same acoustic features for Participant VASP2024, a cognitively healthy control\. This participant exhibits more compact and consistent distributions across nearly all measures\. Task\-1 shows controlled pitch values and balanced pause behavior, consistent with efficient task engagement from the outset\. Task\-2 maintains this stability, with articulation and speaking rates remaining tightly clustered and voice\-quality measures \(jitter, shimmer, HNR\) indicating sustained phonatory control\. Task\-3 introduces more expressive variation such as slightly increased pause frequency and wider pitch range, but these shifts remain within the bounds of healthy adaptive modulation rather than cognitive strain\.

Utterance duration during task 3 suggests that this task elicits longer verbal responses for both groups\. However, the healthy participant exhibits more sustained and consistently longer utterances, particularly in task 3, with tighter distributions indicating more stable response patterns\. In contrast, the cognitively impaired participant shows greater variability in utterance duration, especially in the later task, suggesting less consistent speech production\. The cognitively impaired participant \(VASP2015\) generally exhibits higher variability in jitter and shimmer, particularly in tasks 2 and 3\. Since jitter captures cycle\-to\-cycle frequency instability and shimmer reflects amplitude instability, elevated or more dispersed values may indicate less stable phonation and greater vocal irregularity\. In contrast, VASP2024 exhibits controlled, stable, and predictable acoustic behavior across sessions\. The contrast between the two distributions supports evidence that MCI is associated with increased acoustic variability, disrupted speech timing, and reduced vocal stability, highlighting the potential of speech\-based measures as sensitive indicators for early cognitive decline\. Moreover, the MCI participant displays task\-dependent destabilization, with greater variability and performance decline from task\-1 to task\-3, while the healthy participant maintains consistent control with slight changes\.

### V\-CModel Analysis

![Refer to caption](https://arxiv.org/html/2610.20906v1/comparison_of_accuracy_and_Auc_Across_Models.png)Fig\. 9:Comparison of Accuracy and AUC across tasksFigure[9](https://arxiv.org/html/2610.20906#S5.F9)presents a comparative analysis of four models across three experimental tasks\. The plot displays Accuracy \(solid lines with circular, diamond, and triangular markers\) and AUC \(dashed lines with square, plus, and inverted triangle markers\) for each task\. The Video model performs particularly well on task\-1 and task\-2, suggesting that these tasks provide richer and more discriminative facial expressions associated with cognitive state\. In contrast, the Audio model achieves stronger performance on task\-3, indicating that question answering better exposes verbal inconsistencies and speech\-based cognitive markers\. Notably, the Video \+ Audio \+ Text model demonstrates the most consistent performance across all tasks, maintaining relatively stable Accuracy and AUC values\. Across models, tasks 2 and 3 consistently provide more informative signals, reflected in higher Accuracy and AUC scores, whereas task\-1 appears comparatively less discriminative, exhibiting slightly lower performance and greater variability\.

![Refer to caption](https://arxiv.org/html/2610.20906v1/Confusion_MatrixN.png)Fig\. 10:Comparison across different modelsFigure[10](https://arxiv.org/html/2610.20906#S5.F10)illustrates how classification errors evolve as additional modalities are incorporated \(0 = MCI\)\. For this analysis, each task was treated as an independent data point, enabling an across\-task rather than within\-task analysis\. This allows the model to capture variations in behavioral and emotional patterns across different tasks\. The unimodal models exhibit a tendency toward dominant\-class bias, leading to asymmetric misclassification patterns\. The Video Model provides a baseline performance, correctly identifying 36 MCI participants, but suffers from substantial false negatives, misclassifying 41 healthy participants as MCI, indicating limited discriminative capability when relying on visual emotional cues alone\. The Audio Model improves class separation by reducing false negatives from 41 to 14 and substantially increasing correct healthy classifications \(72\), though it introduces a high number of false positives for the MCI class, suggesting a bias toward predicting healthy controls\. The combined Video\+Audio\+Text model achieves the most balanced and reliable detection, correctly identifying 37 MCI participants while reducing overall misclassifications to 16 false negatives and 21 false positives, demonstrating that integrating complementary behavioral signals mitigates dominant\-class bias and improves class separability\.

## VIDiscussion

The findings of this study demonstrate that natural conversational behavior contains meaningful signals for the early detection of cognitive impairment\. Facial emotion patterns and AU dynamics reveal clear distinctions between cognitively healthy and MCI participants, with the latter exhibiting a higher prevalence of negative affect and reduced emotional variability\. In addition, audio\-based biomarkers and linguistic features provide complementary indicators of impairment, capturing disruptions in prosody, speech fluency, and semantic coherence\. Together, these results highlight the importance of integrating visual, acoustic, and linguistic cues to obtain a more comprehensive characterization of early cognitive decline in naturalistic interaction settings\. Task\-wise comparisons showed that tasks requiring question\-generation or spontaneous responses produced more informative video signals while predefined question answering gives more informative audio signals\. Looking ahead, several directions can strengthen and extend this work\. Expanding the dataset with more participants and diverse demographics will improve generalizability\. We will also focus on longitudinal and cross\-session modeling using Transformer\-based sequence models to capture subtle behavioral changes across sessions\. Additionally, we will focus on improving multimodal fusion strategies and exploring LLMs to provide more interpretable and clinically meaningful reasoning over behavioral signals\.

## VIIConclusion

This work introduces a multimodal preprocessing pipeline that integrates facial behavior, audio biomarkers, and linguistic features to distinguish cognitively healthy individuals from those with MCI\. In contrast to traditional models our framework leverages the inherently multimodal nature of human communication to capture a richer and more comprehensive representation of cognitive function\. By analyzing emotional dynamics, we show that meaningful cognitive signals emerge even during casual conversational tasks, with Command\-Reading and Command\-Generation revealing richer behavioral markers than structured question\-answering tasks\. Moreover, acoustic and linguistic cues provide complementary information, particularly in tasks such as Command\-Generation and Question\-Answering, highlighting the value of modeling task\-dependent multimodal behavior across tasks\. Our results demonstrate that combining all three modalities substantially improves predictive performance \(accuracy = 0\.75, AUC = 0\.80\), outperforming any unimodal system\. This approach enables scalable, continuous monitoring through natural interactions without clinical instrumentation, reflecting a real\-world environment\. This framework can be embedded into telehealth workflows, smart\-home devices, video\-calling platforms, and digital wellness applications, enabling continuous, remote monitoring and early alerts for cognitive decline\.

## VIIIAcknowledgement

This research was funded by the US National Institutes of Health National Institute on Aging R01AG067416\.

## References

- \[1\]\(2016\)OpenFace: an open source facial behavior analysis toolkit\.In2016 IEEE Winter Conference on Applications of Computer Vision \(WACV\),pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1109/WACV.2016.7477553)Cited by:[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p2.1)\.
- \[2\]S\. Bottani, N\. Burgos, A\. Maire, and et al\.\(2023\)Evaluation of mri\-based machine learning approaches for computer\-aided diagnosis of dementia in a clinical data warehouse\.Medical Image Analysis89,pp\. 102903\.External Links:[Document](https://dx.doi.org/10.1016/j.media.2023.102903)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p1.1)\.
- \[3\]Y\. Cai, X\. Li, Y\. Zhang, and et al\.\(2025\)Multimodal sentiment analysis based on multi\-layer feature fusion and multi\-task learning\.Scientific Reports15,pp\. 2126\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-85859-6)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p1.1)\.
- \[4\]K\. Chlasta and K\. Wołk\(2021\)Towards computer\-based automated screening of dementia through spontaneous speech\.Frontiers in Psychology11\.External Links:[Document](https://dx.doi.org/10.3389/fpsyg.2020.623237)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p3.1),[§I](https://arxiv.org/html/2610.20906#S1.p4.1)\.
- \[5\]H\. Ding, A\. Lister, C\. Karjadi, and et al\.\(2024\)Detection of mild cognitive impairment from non\-semantic, acoustic voice features: the framingham heart study\.JMIR Aging7,pp\. e55126\.External Links:[Document](https://dx.doi.org/10.2196/55126)Cited by:[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p3.1)\.
- \[6\]A\. Filali Razzouki, L\. Jeancolas, S\. Sambin, and et al\.\(2025\)Explaining facial action units’ correlation with hypomimia and clinical scores in parkinson’s disease\.npj Parkinson’s Disease11,pp\. 53\.External Links:[Document](https://dx.doi.org/10.1038/s41531-025-00895-3)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p3.1),[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p2.1)\.
- \[7\]Z\. Jahan, S\. Bhatia, and M\. Saraee\(2024\)Early dementia detection with speech analysis and machine learning techniques\.Discover Sustainability5\.External Links:[Document](https://dx.doi.org/10.1007/s43621-024-00217-2)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p3.1)\.
- \[8\]A\. Ksibi, A\. Walha, M\. Zakariah, and et al\.\(2025\)Multimodal siamese networks for dementia detection from speech in women\.Scientific Reports15\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-13902-7)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p3.1)\.
- \[9\]P\. Kundu Maji, S\. Acharya, P\. Paul, and et al\.\(2025\)Deep learning inspired game\-based cognitive assessment for early dementia detection\.Engineering Applications of Artificial Intelligence142,pp\. 109901\.External Links:[Document](https://dx.doi.org/10.1016/j.engappai.2024.109901)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p3.1),[§II](https://arxiv.org/html/2610.20906#S2.p1.1),[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p2.1)\.
- \[10\]M\. Kuprashevich and I\. Tolstykh\(2023\)MiVOLO: multi‑input transformer for age and gender estimation\.arXiv preprint\.External Links:arXiv:2307\.04616,[Document](https://dx.doi.org/10.48550/arXiv.2307.04616)Cited by:[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p2.1)\.
- \[11\]E\. Kurtz, Y\. Zhu, T\. Driesse, and et al\(2023\)Early detection of cognitive decline using voice assistant commands\.InICASSP 2023 \- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p2.1),[§II](https://arxiv.org/html/2610.20906#S2.p2.1)\.
- \[12\]M\. Leming and H\. Im\(2025\)Differential dementia detection from multimodal brain images in a real\-world dataset\.Alzheimer’s & Dementia21\(7\),pp\. e70362\.External Links:[Document](https://dx.doi.org/10.1002/alz.70362)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p1.1)\.
- \[13\]N\. Lin, Y\. Zhu, X\. Liang, and et al\.\(2024\)Analyzing multimodal features of spontaneous voice assistant commands for mild cognitive impairment detection\.InProc\. Interspeech,pp\. 3030–3034\.External Links:[Document](https://dx.doi.org/10.21437/interspeech.2024-2288)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p2.1)\.
- \[14\]S\. Maity and M\. J\. Saikia\(2025\)Large language models in healthcare and medical applications: a review\.Bioengineering12\(6\),pp\. 631\.External Links:[Document](https://dx.doi.org/10.3390/bioengineering12060631)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p1.1)\.
- \[15\]K\. Qi, Y\. Zhu, C\. Summerour, and et al\(2025\)Cog\-tipro: iterative prompt refinement with llms to detect cognitive decline via longitudinal voice assistant commands\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.17137)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p2.1)\.
- \[16\]N\. Rahim, T\. Abuhmed, S\. Mirjalili, and et al\.\(2023\)Time\-series visual explainability for alzheimer’s disease progression detection for smart healthcare\.Alexandria Engineering Journal82,pp\. 484–502\.External Links:[Document](https://dx.doi.org/10.1016/j.aej.2023.09.050)Cited by:[§III\-B](https://arxiv.org/html/2610.20906#S3.SS2.p1.1)\.
- \[17\]L\. Rasmy, Y\. Xiang, Z\. Xie, and et al\.\(2021\)Med\-bert: pretrained contextualized embeddings on large\-scale structured electronic health records for disease prediction\.npj Digital Medicine4,pp\. 86\.External Links:[Document](https://dx.doi.org/10.1038/s41746-021-00455-y)Cited by:[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p4.1)\.
- \[18\]N\. Rezaii, B\. Wong, P\. Aisen, and et al\.\(2025\)Voiceprints of cognitive impairment: analyzing digital voice for early detection of alzheimer’s and related dementias\.npj Dementia1,pp\. 35\.External Links:[Document](https://dx.doi.org/10.1038/s44400-025-00040-0)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p2.1),[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p3.1)\.
- \[19\]A\. K\. Roy, H\. K\. Kathania, A\. Sharma, and et al\.\(2024\)ResEmoteNet: bridging accuracy and loss reduction in facial emotion recognition\.arXiv preprint\.External Links:arXiv:2409\.10545,[Document](https://dx.doi.org/10.48550/arXiv.2409.10545)Cited by:[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p2.1)\.
- \[20\]R\. Singh, S\. Vaish, X\. Liang, and et al\.\(2024\)Clinical geriatric functional assessment using wearable sensing and machine learning\.In2024 IEEE International Conference on E\-health Networking, Application & Services \(HealthCom\),Vol\.,pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1109/HealthCom60970.2024.10880842)Cited by:[§III\-B](https://arxiv.org/html/2610.20906#S3.SS2.p1.1)\.
- \[21\]R\. A\. Taylor, A\. Gilson, L\. Chi, and et al\.\(2023\)Dementia risk analysis using temporal event modeling on a large real\-world dataset\.Scientific Reports13,pp\. 22618\.External Links:[Document](https://dx.doi.org/10.1038/s41598-023-49330-8)Cited by:[§III\-B](https://arxiv.org/html/2610.20906#S3.SS2.p1.1)\.
- \[22\]G\. Tuerhong, F\. Fu, and M\. Wushouer\(2025\)Adaptive multimodal transformer based on exchanging for multimodal sentiment analysis\.Scientific Reports15,pp\. 27265\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-11848-4)Cited by:[§II](https://arxiv.org/html/2610.20906#S2.p1.1)\.
- \[23\]U\.S\. Department of Health and Human Services\(2025\)Mild cognitive impairment\.Alzheimers\.gov\.Note:\[Online; accessed 7\-October\-2025\]External Links:[Link](https://www.alzheimers.gov/alzheimers-dementias/mild-cognitive-impairment)Cited by:[§I](https://arxiv.org/html/2610.20906#S1.p1.1)\.
- \[24\]X\. Zhang, G\. Zhao, Y\. Ren, and et al\.\(2025\)Contrastive text embeddings with effective sample mining for enhanced disease diagnosis\.Array28,pp\. 100578\.External Links:[Document](https://dx.doi.org/10.1016/j.array.2025.100578)Cited by:[§III\-A](https://arxiv.org/html/2610.20906#S3.SS1.p4.1)\.