CogDrift-4D：用于轻度认知障碍检测的时间行为衰退分层多模态建模

# CogDrift-4D: Hierarchical Multimodal Modeling of Temporal Behavioral Decay for MCI Detection
Source: [https://arxiv.org/html/2610.20877](https://arxiv.org/html/2610.20877)
Sankalp Vaish1, Sumukh Veeramalla1, Rishank Singh1, Caroline Summerour2, John A\. Batsis2, Xiaohui Liang1Affiliation:Affiliation:1Department of Computer Science, University of Massachusetts Boston, MA, USAAffiliation:2School of Medicine, University of North Carolina at Chapel hill, NC, USAAffiliation:Email: \{Sankalp\.Vaish001, Sumukh\.Veeramalla001, Rishank\.Singh001, Xiaohui\.Liang\}@umb\.edu, Caroline\_Summerour@med\.unc\.edu, John\.Batsis@unc\.edu

September 2025

###### Abstract

Early detection of mild cognitive impairment remains a critical bottleneck in dementia care, demanding scalable tools capable of capturing subtle behavioral decay\. To capture the subtle temporal changes of cognitive decline, we introduce a behavioral dataset collected across seven unscripted multimodal interactions over an 18\-month horizon\. This dataset formalizes continuous, multi\-session interactions into a dense 4D tensor \(Batch×Session×Time×Feature\\text\{Batch\}\\times\\text\{Session\}\\times\\text\{Time\}\\times\\text\{Feature\}\) that explicitly preserves concurrent local dynamics alongside long\-term changes\. To effectively process these complex behavioral patterns, we proposeCogDrift\-4D: a novel, hierarchical 4D Transformer framework\. To capture the complex trajectory of cognitive decline, CogDrift\-4D implements a dual\-level behavioral fatigue and drift modeling architecture\. At the intra\-session level, the model computes localized least\-squares regression slopes to quantify the acute “fading” of frame\-level facial action units and acoustic energy\. At the inter\-session level, a recursive cross\-attention feedback encoder acts as a cumulative fatigue accumulator, mapping the patient’s continuous behavioral drift across successive sessions\. To mitigate environmental noise and track failures across sessions, a localized Quality Gate dynamically filters low\-integrity features before final fusion, also an early fusion that integrates semantic insights with raw physical dynamics\. Evaluated on structured verbal and linguistic interactions, our architecture successfully detects MCI with a classification accuracy of0\.733±0\.130\.733\\pm 0\.13and an F1\-score of0\.716±0\.140\.716\\pm 0\.14, outperforming advanced baselines\. The framework provides a mathematically rigorous, scalable, and robust foundation for home\-based, real\-time dementia screening\.

© 2026 IEEE\. Personal use of this material is permitted\. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works\.

Accepted for publication at the 2026 IEEE International Conference on E\-health Networking, Application & Services \(Healthcom\)\.

## IIntroduction

Dementia is a major global public health challenge whose prevalence is rising with aging populations\. Mild Cognitive Impairment \(MCI\) marks the transitional stage between normal age\-related decline and early dementia, involving measurable deficits in memory, language, or executive function while daily independence is largely preserved\. Individuals diagnosed with MCI are at increased risk of developing Alzheimer’s disease, with studies estimating that approximately 10% to 20% of adults aged 65 or older with MCI transition to dementia annually\[[28](https://arxiv.org/html/2610.20877#bib.bib7)\]\. However, standard early screening tools such as the MoCA and MMSE depend on in\-person clinical settings and trained administrators, limiting accessibility, longitudinal monitoring, and scalable early screening\.

Motivated by the need for longitudinal monitoring approaches, we investigated the feasibility of tracking MCI progression by modeling long\-term behavioral dynamics over an 18\-month longitudinal horizon, in which participants completed standardized speech interaction tasks at consecutive 3\-month intervals \(7 sessions\)\. Leveraging the continuous nature of human expression, this study captures physical facial dynamics, acoustic characteristics, and linguistic patterns\. Fine\-grained facial affect signals\[[10](https://arxiv.org/html/2610.20877#bib.bib2)\]and facial Action Unit \(AU\) activations\[[7](https://arxiv.org/html/2610.20877#bib.bib24)\]are extracted from video recordings to identify variations between cognitively healthy individuals and those with MCI\. We track these behavioral and temporal signatures across successive sessions to evaluate longitudinal decline\.

In recent years, several techniques\[[14](https://arxiv.org/html/2610.20877#bib.bib28),[22](https://arxiv.org/html/2610.20877#bib.bib30),[17](https://arxiv.org/html/2610.20877#bib.bib25),[26](https://arxiv.org/html/2610.20877#bib.bib27)\]have proven highly successful in extracting localized diagnostic markers from isolated, single\-session recordings of spontaneous speech, facial micro\-expressions, and linguistic features\[[25](https://arxiv.org/html/2610.20877#bib.bib29)\]\. However, most paradigms fail to capture long\-term changes because they are restricted to static snapshots or flat sequence processing\. Frame\-level facial landmarks, acoustic features\[[5](https://arxiv.org/html/2610.20877#bib.bib3)\], and text embeddings\[[27](https://arxiv.org/html/2610.20877#bib.bib8),[3](https://arxiv.org/html/2610.20877#bib.bib9)\]exhibit substantial temporal variation, multi\-scale dynamics, and complex cross\-modal dependencies that conventional mean pooling or static classifiers cannot adequately model\. To address these data\-specific limitations, our pipeline introduces a context\-aware dual\-stage fusion to resolve multi\-resolution temporal dynamics, adaptive quality gating to dynamically suppress low\-integrity or missing sessions, and localized least\-squares regression to explicitly isolate vocal fatigue from true cognitive drift \(gradual trajectory of decline in a patient’s cognitive functions\)\. Our dual\-stage fusion architecture first performs context\-aware early fusion by integrating sentence embeddings with acoustic and expressive features within each session\. Following session processing, a secondary inter\-session fusion stage processes cross\-modal representations over the 18 months \(7 sessions\) to identify cognitive decline\.

The goal of this research is to lay the groundwork for an early\-warning home\-based, non\-invasive cognitive screening ecosystem capable of operating robustly within unconstrained, everyday environments\. The system is highly adaptable, allowing for seamless integration into existing telehealth platforms, smart home virtual assistants, and mobile health applications\. Unlike prior diagnostic methodologies that restrict their scope to isolated, single\-session evaluations, our framework synthesizes longitudinal interactions, capable of tracking parallel cognitive and behavioral fluctuations over time\. This work makes the following key contributions:

- •Hierarchical 4D Transformer \(CogDrift\-4D\):We introduce a novel, end\-to\-end hierarchical architecture designed specifically to capture multi\-scale temporal degradation in behavioral tracking data\. The model operates on unified four\-dimensional spatiotemporal tensors \(Batch×Session×Time×Feature\\text\{Batch\}\\times\\text\{Session\}\\times\\text\{Time\}\\times\\text\{Feature\}\) generated by a dedicated multi\-modal alignment pipeline\. The model leverages lower\-level intra\-session attention blocks to align asynchronous multi\-modal features, coupled with an upper\-level inter\-session encoder to track macro\-level longitudinal trajectory\. CogDrift\-4D achieves a peak classification accuracy of 0\.733 \( Outperforms advanced benchmarks by∼\\sim8%\)\.
- •A Longitudinal Conversational Dataset:We introduce a novel, multi\-session longitudinal behavioral dataset collected using Amazon Alexa\. Moving beyond traditional clinical datasets that rely on rigid, highly structured laboratory tests, our tasks are explicitly designed for high ecological validity, capturing unscripted natural conversations alongside diverse cognitive behavioral tasks\.
- •Temporal analysis across seven sessions\.We examine temporal stability in emotional expressions and behavioral cues across repeated sessions\. MCI participants exhibit greater acoustic variability, disrupted timing, reduced vocal stability, reduced emotional diversity, and persistent negative affect, while cognitively healthy participants show stable speech patterns, greater emotional variability, and more positive affect\.

## IIRelated Work

Conventional dementia screening depends on standardized clinical assessments administered by trained professionals in controlled settings, limiting repeated testing and large\-scale deployment\. This has increased interest in automated, contactless screening methods, with prior work largely focusing on speech\-based biomarkers such as acoustic, prosodic, and linguistic features from conversational speech\[[5](https://arxiv.org/html/2610.20877#bib.bib3),[8](https://arxiv.org/html/2610.20877#bib.bib4),[12](https://arxiv.org/html/2610.20877#bib.bib22)\], as well as transcript\-derived linguistic cues \[\[[9](https://arxiv.org/html/2610.20877#bib.bib1)\]\]\. While these approaches demonstrate the value of speech for cognitive assessment, they often overlook nonverbal behavioral signals\. Facial expressions and muscle activations provide complementary cognitive\-health information, and prior studies suggest that facial cues can distinguish individuals with dementia\[[10](https://arxiv.org/html/2610.20877#bib.bib2)\]\.

Recent progress in affective computing and sentiment analysis shows that multimodal behavioral cues can improve machine learning\-based interpretation of human states\[[27](https://arxiv.org/html/2610.20877#bib.bib8),[3](https://arxiv.org/html/2610.20877#bib.bib9)\]\. Although LLMs benefit from domain\-specific pretraining, fine\-tuning, and prompt engineering\[[16](https://arxiv.org/html/2610.20877#bib.bib10),[4](https://arxiv.org/html/2610.20877#bib.bib12),[30](https://arxiv.org/html/2610.20877#bib.bib11)\], their computational cost and text\-centered design make them less suitable for direct analysis of continuous audio and video streams\. Neuroimaging methods such as MRI and PET have also been studied for dementia diagnosis\[[2](https://arxiv.org/html/2610.20877#bib.bib5),[13](https://arxiv.org/html/2610.20877#bib.bib6)\], but their cost, limited accessibility, and clinical dependence restrict their use for home\-based longitudinal monitoring\. In contrast, everyday behavioral signals from speech, facial expressions, and language can be captured unobtrusively through consumer devices\. Building on Cog\-TiPRO\[[18](https://arxiv.org/html/2610.20877#bib.bib17)\]and prior work\[[15](https://arxiv.org/html/2610.20877#bib.bib23)\], our study extends voice\-assistant\-based cognitive monitoring by incorporating a multimodal Transformer\-based pipeline\.

## IIIMethodology

### III\-APreprocessing

The dataset consisted of synchronized audio–video recordings of participant interactions with Alexa\. All modalities were preprocessed to extract meaningful behavioral features as:

Video Encoder:To address the presence of multiple individuals in each recording \(participant, interviewer, Alexa device, questionnaire holder\), we first applied MiVOLO\[[11](https://arxiv.org/html/2610.20877#bib.bib15)\]to every frame for face detection and age estimation\. This allowed us to isolate the older adult participant across all sessions\. Video models are trained using facial Action Units or emotion features\. We employed ResEmoteNet\[[21](https://arxiv.org/html/2610.20877#bib.bib16)\], to estimate frame\-level emotional states \(happy, sad, angry, surprise, neutral, disgust, fear\)\. For each speaking segment, we aggregated these predictions into quantitative descriptors\[[10](https://arxiv.org/html/2610.20877#bib.bib2)\]\. To characterize detailed facial behavior, we used OpenFace\[[1](https://arxiv.org/html/2610.20877#bib.bib18)\]to extract Facial Action Units \(AUs\), which encode the activation of individual facial muscles \(e\.g\., AU1: Inner Brow Raiser, AU12: Lip Corner Puller\)\. As physiologically grounded descriptors of facial movement\[[7](https://arxiv.org/html/2610.20877#bib.bib24)\], AUs can capture subtle expression changes associated with cognitive decline\.

Audio EncoderUsing Parselmouth and Librosa, participant speech segments were first isolated and subsequently analyzed to extract both acoustic and prosodic characteristics\[[20](https://arxiv.org/html/2610.20877#bib.bib13),[6](https://arxiv.org/html/2610.20877#bib.bib14)\]\. Acoustic features included MFCC statistics, harmonics\-to\-noise ratio, etc\., while prosodic and speech biomarkers comprised fundamental frequency \(F0\) measures, intensity contours, jitter, shimmer, speaking rate, pause count, and articulation rate\. These features were then encoded into fixed\-length embeddings for downstream multimodal modeling\.

Text Encoder:Transcripts generated using Rev\.ai were encoded into embeddings using a Sentence Transformer model\.

![Refer to caption](https://arxiv.org/html/2610.20877v1/Model_Pipeline.png)Fig\. 1:CogDrift\-4D Framework
### III\-BCogDrift\-4D Framework

The CogDrift\-4D architecture is designed to capture the longitudinal decay of behavioral signals across multiple assessment tasks\. The model processes data through a hierarchical pipeline of temporal encoding, divergence modeling, and multi\-session fusion\. Figure[1](https://arxiv.org/html/2610.20877#S3.F1)presents the proposed Transformer\-based Multimodal architecture, using extracted features\[[19](https://arxiv.org/html/2610.20877#bib.bib19),[24](https://arxiv.org/html/2610.20877#bib.bib20),[23](https://arxiv.org/html/2610.20877#bib.bib21)\]\.

#### III\-B1Intra\-Session Temporal Encoding

Each session is first processed at the frame level to capture fine\-grained behavioral dynamics\.

Feedback\-Driven Session EncoderTo model the longitudinal progression across theSSassessment tasks, we employ a recursive feedback mechanism\. Given the position\-aware input tensor𝒳∈ℝB×S×T×D\\mathcal\{X\}\\in\\mathbb\{R\}^\{B\\times S\\times T\\times D\}, let𝐦s−1∈ℝD\\mathbf\{m\}\_\{s\-1\}\\in\\mathbb\{R\}^\{D\}represent the memory state \(summary\) of the preceding session\. The current session features𝐇s\\mathbf\{H\}\_\{s\}are updated via a cross\-attention operation, where𝐇s\\mathbf\{H\}\_\{s\}acts as the query and𝐦s−1\\mathbf\{m\}\_\{s\-1\}provides the contextual key and value:

𝐇^s=MultiHeadAttn​\(Q=𝐇s,K=𝐦s−1,V=𝐦s−1\)\\hat\{\\mathbf\{H\}\}\_\{s\}=\\text\{MultiHeadAttn\}\(Q=\\mathbf\{H\}\_\{s\},K=\\mathbf\{m\}\_\{s\-1\},V=\\mathbf\{m\}\_\{s\-1\}\)\(1\)This architecture allows the model to utilize the cumulative ”behavioral history” of the participant as a prior for interpreting the current task, effectively capturing inter\-session fatigue and cognitive fluctuations\. We apply a Transformer encoder whereMask\-aware attentionensures the model attends only to valid frames𝒱\\mathcal\{V\}within each session:

Attn​\(Q,K,V,𝐌\)=softmax​\(Q​K⊤dk\+𝐌\)​V\\text\{Attn\}\(Q,K,V,\\mathbf\{M\}\)=\\text\{softmax\}\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\mathbf\{M\}\\right\)V\(2\)
where𝐌\\mathbf\{M\}is a binary mask preventing information leakage from padding\. This process yields the modality\-specific enriched representations𝐇^s,aud\\hat\{\\mathbf\{H\}\}\_\{s,\\text\{aud\}\}and𝐇^s,vid\\hat\{\\mathbf\{H\}\}\_\{s,\\text\{vid\}\}\.

Dynamic Divergence:To capture the “rhythm” of talking, we compute finite differences at multiple temporal offsetsτ∈\{1,2,4\}\\tau\\in\\\{1,2,4\\\}\. For a feature sequence𝐗\\mathbf\{X\}, the delta is defined as:Δτ​𝐗t=𝐗t−𝐗t−τ\\Delta\_\{\\tau\}\\mathbf\{X\}\_\{t\}=\\mathbf\{X\}\_\{t\}\-\\mathbf\{X\}\_\{t\-\\tau\}The original signal is augmented with these multi\-scale deltas\. We define the Dynamic Divergence Score as the cosine similarity between the encoded temporal signatures of the visual and acoustic streams:

𝒟d​y​n=cos⁡\(Enc​\(Δ​𝐗v​i​d\),Enc​\(Δ​𝐗a​u​d\)\)\\mathcal\{D\}\_\{dyn\}=\\cos\\left\(\\text\{Enc\}\(\\Delta\\mathbf\{X\}\_\{vid\}\),\\text\{Enc\}\(\\Delta\\mathbf\{X\}\_\{aud\}\)\\right\)

Static Divergence:While dynamic divergence tracks frame\-to\-frame fluctuations, Static Divergence identifies persistent affective mismatches throughout the session\. Using the attention\-pooled session descriptors𝐡v​i​d\\mathbf\{h\}\_\{vid\}and𝐡a​u​d\\mathbf\{h\}\_\{aud\}, the score is computed as the complement of their similarity:

𝒟s​t​a​t=1−cos⁡\(𝐡v​i​d,𝐡a​u​d\)\\mathcal\{D\}\_\{stat\}=1\-\\cos\\left\(\\mathbf\{h\}\_\{vid\},\\mathbf\{h\}\_\{aud\}\\right\)

Dynamic \(Rhythm\): The patient’s face and voice are ”out of sync” in time \(e\.g\., the smile comes 1 second after the laugh\)\. Static \(State\): The patient’s face and voice are ”out of sync” in emotion \(e\.g\., the face looks sad, but the voice sounds angry\)\.

#### III\-B2Neuro\-Symbolic Insights

By utilizing a sentence transformer model to directly process raw transcripts, we map textual inputs into dense, context\-aware semantic embeddings\. These embeddings serve as high\-level linguistic anchors within our early\-fusion layer, allowing the network to jointly correlate subtle shifts in word choice and conversational structure with concurrent acoustic and expressive features\.𝐡i​n​s​i​g​h​t=SentenceTansformer​\(Transcript\)\\mathbf\{h\}\_\{insight\}=\\text\{SentenceTansformer\}\(\\text\{Transcript\}\)

#### III\-B3Context\-Aware Early Fusion and Quality Gating

To bridge the gap between cognitive intent and multi\-modal behaviors, we implement a context\-aware early fusion layer\. We enrich the frame\-level physical representations by projecting the concatenated multimodal features into a shared latent space𝐇t∈\{Vt,At\}\\mathbf\{H\}\_\{t\}\\in\\\{V\_\{t\},A\_\{t\}\\\}andHi​n∈Proj⁡\(hinsight\)H\_\{in\}\\in\\mathrm\{Proj\}\(h\_\{\\text\{insight\}\}\):

𝐇~t=ReLU​\(Wf​\[𝐇t;𝐇i​n\]\+bf\)\\tilde\{\\mathbf\{H\}\}\_\{t\}=\\text\{ReLU\}\\left\(W\_\{f\}\[\\mathbf\{H\}\_\{t\};\\mathbf\{H\}\_\{in\}\]\+b\_\{f\}\\right\)\(3\)
This joint modeling allows the architecture to learn cross\-modal temporal dependencies, where the linguistic context acts as a semantic prior for interpreting facial and acoustic dynamics\. Concurrently, to safeguard against noisy or missing streams, we introduce a session\-level Quality Gate mechanism𝐆s\\mathbf\{G\}\_\{s\}that dynamically assesses the signal reliability of each input channel:𝐆s=σ⁡\(Wg​𝐡s\+bg\)\\mathbf\{G\}\_\{s\}=\\sigma\(W\_\{g\}\\mathbf\{h\}\_\{s\}\+b\_\{g\}\)where𝐡s\\mathbf\{h\}\_\{s\}represents the aggregate session\-level representation, and𝐆s=\[gvid,gaud,gfused\]∈ℝ3\\mathbf\{G\}\_\{s\}=\[g\_\{\\text\{vid\}\},g\_\{\\text\{aud\}\},g\_\{\\text\{fused\}\}\]\\in\\mathbb\{R\}^\{3\}denotes the channel\-specific gating coefficients bounded between 0 and 1 via the sigmoid \(σ\\sigma\) activation\. These coefficients are then applied element\-wise to scale the respective representations:

𝐇^s,vid\\displaystyle\\hat\{\\mathbf\{H\}\}\_\{s,\\text\{vid\}\}=gvid⋅𝐇s,vid,\\displaystyle=g\_\{\\text\{vid\}\}\\cdot\\mathbf\{H\}\_\{s,\\text\{vid\}\},\(4\)𝐇^s,aud\\displaystyle\\hat\{\\mathbf\{H\}\}\_\{s,\\text\{aud\}\}=gaud⋅𝐇s,aud,\\displaystyle=g\_\{\\text\{aud\}\}\\cdot\\mathbf\{H\}\_\{s,\\text\{aud\}\},\(5\)𝐇^t\\displaystyle\\hat\{\\mathbf\{H\}\}\_\{t\}=gfused⋅𝐇~t\.\\displaystyle=g\_\{\\text\{fused\}\}\\cdot\\tilde\{\\mathbf\{H\}\}\_\{t\}\.\(6\)
This scalar acts as a soft filter; sessions identified as ”noisy” or ”uninformative” are scaled toward zero, ensuring that the model’s diagnostic decision is grounded primarily in high\-fidelity behavioral data\.

#### III\-B4Behavioral Drift & Fatigue Modeling

A primary innovation of CogDrift\-4D is the quantification of longitudinal performance decay\. We model fatigue as a multi\-level hierarchical process to capture both acute, task\-specific decay and cumulative cognitive exhaustion\.

Level 1: Intra\-Session Decay\.For each sessionss, we quantify the ”fading” of behavioral intensity by calculating the least\-squares regression slopeβ\\betaof the feature magnitude\. Letyt=‖𝐡t‖2y\_\{t\}=\\\|\\mathbf\{h\}\_\{t\}\\\|\_\{2\}denote theL2L\_\{2\}norm of the latent features at framett\. The slope is computed against the normalized time indicesτ∈\[0,1\]\\tau\\in\[0,1\]as follows:

β=∑t=1T\(τt−τ¯\)​\(yt−y¯\)∑t=1T\(τt−τ¯\)2\\beta=\\frac\{\\sum\_\{t=1\}^\{T\}\(\\tau\_\{t\}\-\\bar\{\\tau\}\)\(y\_\{t\}\-\\bar\{y\}\)\}\{\\sum\_\{t=1\}^\{T\}\(\\tau\_\{t\}\-\\bar\{\\tau\}\)^\{2\}\}\(7\)This scalar serves as a proxy for acute exhaustion, where a negativeβ\\betaindicates a significant decrease in expressive or acoustic energy during a single assessment task\.

Level 2: Inter\-Session Accumulation\.To track global clinical progression across the interview, we aggregate intra\-session descriptors via a recursive feedback loop\. Let𝐀s\\mathbf\{A\}\_\{s\}be the accumulated fatigue state at sessionss, defined as:𝐀s=GRU​\(𝐡s,𝐀s−1\)\\mathbf\{A\}\_\{s\}=\\text\{GRU\}\(\\mathbf\{h\}\_\{s\},\\mathbf\{A\}\_\{s\-1\}\)where the hidden state𝐀\\mathbf\{A\}serves as aCumulative Fatigue Accumulator\. By comparing states𝐀1\\mathbf\{A\}\_\{1\}through𝐀S\\mathbf\{A\}\_\{S\}, the model quantifies the inter\-session ”behavioral drift,” identifying patients whose performance degrades holistically over the duration of the 4D assessment\.

#### III\-B5Hierarchical Global Attention and Pooling

Following temporal encoding, the modality\-specific sequences𝐇^s,vid\\hat\{\\mathbf\{H\}\}\_\{s,\\text\{vid\}\},𝐇^s,aud\\hat\{\\mathbf\{H\}\}\_\{s,\\text\{aud\}\}, and the context\-aware temporal stream𝐇^t\\hat\{\\mathbf\{H\}\}\_\{t\}are processed by aGlobal Encoder\. This module captures high\-level dependencies across modalities before aggregating the data into session\-level descriptors\. To transition from 4D frame\-level tensors to a unified representation, we employ a multi\-branch attention pooling mechanism\.

For each streamk∈\{vid, aud, temp\}k\\in\\\{\\text\{vid, aud, temp\}\\\}, we compute scalar importance scoresαtk\\alpha\_\{t\}^\{k\}via a learned linear projection𝐰a\\mathbf\{w\}\_\{a\}\. The normalized weightsα~tk\\tilde\{\\alpha\}\_\{t\}^\{k\}are obtained via a softmax operation over the set of valid \(non\-padded\) frames𝒱\\mathcal\{V\}:

αtk=𝐰a⊤​𝐇^k,t,α~tk=exp⁡\(αtk\)∑i∈𝒱exp⁡\(αik\)\\alpha\_\{t\}^\{k\}=\\mathbf\{w\}\_\{a\}^\{\\top\}\\hat\{\\mathbf\{H\}\}\_\{k,t\},\\quad\\tilde\{\\alpha\}\_\{t\}^\{k\}=\\frac\{\\exp\(\\alpha\_\{t\}^\{k\}\)\}\{\\sum\_\{i\\in\\mathcal\{V\}\}\\exp\(\\alpha\_\{i\}^\{k\}\)\}\(8\)
The final session\-level embeddings,𝐡vid\\mathbf\{h\}\_\{\\text\{vid\}\},𝐡aud\\mathbf\{h\}\_\{\\text\{aud\}\}, and𝐡ef\\mathbf\{h\}\_\{\\text\{ef\}\}, are computed as the attention\-weighted sum of their respective frame\-level representations:𝐡k=∑t∈𝒱α~tk​𝐇^k,t\\mathbf\{h\}\_\{k\}=\\sum\_\{t\\in\\mathcal\{V\}\}\\tilde\{\\alpha\}\_\{t\}^\{k\}\\hat\{\\mathbf\{H\}\}\_\{k,t\}

#### III\-B6Late Fusion

The final patient descriptor𝐡f​i​n​a​l\\mathbf\{h\}\_\{final\}is constructed via late fusion, integrating all specialized heads:𝐡f​i​n​a​l=\[𝐡v​i​d;𝐡a​u​d;𝐡e​f;𝒟d​y​n,s​t​a​t;βi​n​t​r​a,i​n​t​e​r\]⊙𝐆s\\mathbf\{h\}\_\{final\}=\[\\mathbf\{h\}\_\{vid\};\\mathbf\{h\}\_\{aud\};\\mathbf\{h\}\_\{ef\};\\mathcal\{D\}\_\{dyn,stat\};\\beta\_\{intra,inter\}\]\\odot\\mathbf\{G\}\_\{s\}The final outputy^\\hat\{y\}is obtained via a linear classifier:y^=σ⁡\(Wo​u​t​𝐡f​i​n​a​l\+b\)\\hat\{y\}=\\sigma\(W\_\{out\}\\mathbf\{h\}\_\{final\}\+b\)

Network Depth and Regularization:The SessionTransformer employs a hidden dimension ofd=128d=128and consists ofL=8L=8Transformer encoder layers withH=8H=8attention heads per layer\. Dropout with ratep=0\.3p=0\.3is applied with the AdamW optimizer with a learning rate of3×10−53\\times 10^\{\-5\}, weight decay of1×10−41\\times 10^\{\-4\}and cross\-entropy loss for 50 epochs\.

## IVDataset

The dataset was collected from 35 older adult participants through a series of structured interactions with Amazon Alexa\. Each experimental session was divided into three distinct tasks, designed to elicit different forms of verbal and cognitive engagement\. These included question responses, participant\-generated queries, and system\-prompted answering scenarios, collectively capturing variations in language production, comprehension, and recall\. To capture longitudinal variations, the entire experiment was repeated seven times with each participant at different time intervals\. All sessions were recorded with synchronized audio and video, and transcripts were generated for subsequent analysis\. In total, this resulted in \(35∗\*3 = 105\) data points, each sample is structured as a 4\-D tensor spanning up to 7 sessions containing multiple intra\-session time frames\. Data completeness was preserved, with the exception of 2 participants who completed only 6 sessions\.

![Refer to caption](https://arxiv.org/html/2610.20877v1/Cog4D_data.png)Fig\. 2:DataFigure[2](https://arxiv.org/html/2610.20877#S4.F2)provides an overview of the dataset structure, illustrating the duration and composition of each experimental session\. Each session lasted approximately 40–75 minutes and included all three activities\. Participants with MoCA scores between 18 and 25 were categorized as having Mild Cognitive Impairment \(MCI\), while those scoring 26 or above were considered cognitively healthy\. The figure lists all participant IDs and their corresponding session numbers\. No personal information is included, ensuring strict privacy compliance\.

## VResults

### V\-ASession Analysis

![Refer to caption](https://arxiv.org/html/2610.20877v1/AudioFeatures_across_sessions.png)Fig\. 3:Fatigue across sessionsFigure[3](https://arxiv.org/html/2610.20877#S5.F3)shows the temporal fatigue variation across seven recorded sessions for a participant diagnosed with MCI\. To quantify user exhaustion, we derive an acoustic fatigue metric by capturing the relationship between vocal energy and temporal processing delays\. The MCI \(blue\) dominates across sessions, reflecting a consistently higher fatigue, while the healthy \(orange\) have lower fatigue\.

![Refer to caption](https://arxiv.org/html/2610.20877v1/part_15_across_ses.png)Fig\. 4:Emotions across sessions of a person with MCIFigure[4](https://arxiv.org/html/2610.20877#S5.F4)shows the temporal variation of emotions across seven recorded sessions for a participant diagnosed with MCI\. The sad emotion \(blue\) dominates across sessions, reflecting a consistently negative affective state, while the happy emotion \(orange\) appears infrequently, indicating reduced emotional variability and positivity\.

![Refer to caption](https://arxiv.org/html/2610.20877v1/part_24_across_ses.png)Fig\. 5:Emotions across sessions of a healthy personFigure[5](https://arxiv.org/html/2610.20877#S5.F5)depicts emotion variation across seven sessions for a cognitively healthy participant\. In contrast to the MCI case, happy \(orange\) and neutral \(gray\) emotions occur more frequently and consistently over time, while negative emotions, such as sadness, appear less often\. This stability and predominance of positive affect suggest a potential link between emotional regulation and preserved cognitive functioning\.

### V\-BModel Analysis

Our CogDrift\-4D framework was implemented in PyTorch and optimized on an NVIDIA RTX GPU\. We used binary classification, where11denotes MCI and00represents healthy controls, using the standard clinical threshold of MoCA<26<26\. To establish a rigorous evaluation benchmark, three state\-of\-the\-art clinical architectures were adapted to accept our uniform 4D behavioral tensors:M\-BEHRT\[[17](https://arxiv.org/html/2610.20877#bib.bib25)\], which models consecutive tasks as sequential electronic health record visits;AAMLA\[[26](https://arxiv.org/html/2610.20877#bib.bib27)\], which utilizes a parameterized bilinear product for cross\-modal affinity token alignment; andPRA\-PoE\[[29](https://arxiv.org/html/2610.20877#bib.bib26)\], which handles incomplete data streams using a closed\-form product\-of\-experts rule\. To evaluate the specific impact of multi\-scale temporal modeling, we implemented two internal structural baselines:Baseline 1 \(Static Late\-Fusion MLP\)andBaseline 2 \(Isolated Multimodal Transformer\)\. Baseline 1 compresses the entire 18\-month timeline using mask\-aware mean pooling to merge the longitudinal sessions into a single static array, fusing the modalities via concatenation before routing them through a multi\-layer perceptron\. Baseline 2 isolates each 3\-month session, utilizing an intra\-session transformer encoder to mix facial, acoustic, and linguistic tokens independently per task milestone\. The resulting unaligned session representations are then globally averaged\.

TABLE I:Comparision study results \(5\-fold cross\-validation, mean±\\pmstd\)\.To prevent data leakage, we evaluate our proposed framework using a participant\-grouped k\-fold cross\-validation, benchmarking performance across accuracy, F1\-AUC, and ROC\-AUC\. Table[I](https://arxiv.org/html/2610.20877#S5.T1)presents the comparative evaluation using 5\-fold cross\-validation with 50 epochs of our proposed framework against the above architectures\. In comparison, Baseline 1 and Baseline 2 yield near\-chance accuracies of0\.5429±0\.090\.5429\\pm 0\.09and0\.5524±0\.090\.5524\\pm 0\.09, respectively, confirming that flattening the 18\-month longitudinal timeline or isolating single sessions obliterates the vital temporal indicators of cognitive decay\. Furthermore, advanced public SOTA clinical benchmarks like AAMLA and PRA\-PoE stall at ceiling accuracies of0\.5714±0\.070\.5714\\pm 0\.07and0\.5810±0\.070\.5810\\pm 0\.07, demonstrating that optimization focused purely on single\-session cross\-modal alignment or probabilistic fusion cannot decipher long\-term degradation\. Crucially, even when measured against M\-BEHRT—a powerful longitudinal sequence baseline that achieved a respectable accuracy of0\.6571±0\.130\.6571\\pm 0\.13and F1\-score of0\.6549±0\.110\.6549\\pm 0\.11, our framework yields a decisive∼\\sim8% improvement in overall accuracy\. This proves that token\-sequencing transformers are structurally ill\-equipped to isolate the subtle cross\-session behavioral drift and micro\-fatigue slopes embedded within human expression and behavior, underscoring the necessity of CogDrift\-4D’s dual\-stage design\.

TABLE II:Ablation study results \(10\-fold cross\-validation, mean±\\pmstd\)\.Table[II](https://arxiv.org/html/2610.20877#S5.T2)evaluates the contribution of different components of the proposed multimodal framework using 10\-fold cross\-validation with 50 epochs\. The full model achieved the best overall performance with an accuracy of 0\.742, F1\-score of 0\.733, and AUC of 0\.724, demonstrating the effectiveness of combining all proposed components\. Removing the fatigue\-related features led to a reduction in performance across all metrics, indicating that fatigue cues provide important information for classification\. Similarly, excluding the quality module slightly decreased performance, suggesting that quality\-aware modeling contributes to more robust predictions\. A significant performance drop was observed when the fatigue component was removed, with f1\-score decreasing to 0\.69 and AUC to 0\.68, highlighting the importance of fatigue\-based representations in capturing discriminative patterns\.

## VIDiscussion

The empirical findings of this study demonstrate that continuous, natural conversational behavior contains highly sensitive digital biomarkers capable of identifying early\-stage cognitive impairment\. While our 18\-month, 7\-session longitudinal modeling framework successfully captured micro\-fatigue slopes and macro\-level behavioral drift, expanding the cohort to encompass a more prominent, multi\-center participant demographic will be crucial to solidify the generalizability of these digital biomarkers against diverse geographic speech variations\. We will also integrate lightweight LLMs to extract deep semantic and contextual insights directly from patient transcripts without compromising data privacy\. A promising future direction is to enable on\-device deployment of the model, allowing privacy\-preserving and low\-latency cognitive assessment and supporting practical use in mHealth, telehealth, and smart\-home monitoring applications\.

## VIIConclusion

This study introduces a fully non\-invasive multimodal deep learning framework that integrates facial behavior dynamics, acoustic biomarkers, and linguistic features to distinguish cognitively healthy individuals from those experiencing MCI\. By utilizing naturalistic interactions with Alexa, our proposed CogDrift\-4D architecture successfully tracks fatigue, multi\-scale temporal dynamics, and long\-term longitudinal cognitive drift across an 18\-month study\. CogDrift\-4D achieves a peak classification accuracy of0\.733±0\.130\.733\\pm 0\.13and an F1\-score of0\.716±0\.150\.716\\pm 0\.15, yielding a performance margin \(∼\\sim8% accuracy improvement\) over advanced sequence benchmarks like M\-BEHRT, and outperforming static late\-fusion models by over 20%\. These results validate that early neurodegeneration manifests as a synchronized, multi\-modal degradation across expressive, acoustic, and linguistic channels that can be effectively captured via dual\-stage hierarchical self\-attention\. Because the system operates entirely without specialized clinical instrumentation and requires only brief video recordings, it eliminates significant operational bottlenecks\. Ultimately, this framework provides a highly scalable asset for seamless deployment within telehealth workflows, smart\-home devices, and digital wellness platforms, enabling continuous, remote monitoring and low\-cost early detection for cognitive decline\.

## VIIIAcknowledgement

This research was funded by the US National Institutes of Health National Institute on Aging R01AG067416\.

## References

- \[1\]\(2016\)OpenFace: an open source facial behavior analysis toolkit\.In2016 IEEE Winter Conference on Applications of Computer Vision \(WACV\),pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1109/WACV.2016.7477553)Cited by:[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p2.1)\.
- \[2\]S\. Bottani, N\. Burgos, A\. Maire, and et al\.\(2023\)Evaluation of mri\-based machine learning approaches for computer\-aided diagnosis of dementia in a clinical data warehouse\.Medical Image Analysis89,pp\. 102903\.External Links:[Document](https://dx.doi.org/10.1016/j.media.2023.102903)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[3\]Y\. Cai, X\. Li, Y\. Zhang, and et al\.\(2025\)Multimodal sentiment analysis based on multi\-layer feature fusion and multi\-task learning\.Scientific Reports15,pp\. 2126\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-85859-6)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1),[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[4\]X\. Chen, J\. Xiang, S\. Lu, and et al\.\(2025\)Evaluating large language models and agents in healthcare: key challenges in clinical applications\.Intelligent Medicine5\(2\),pp\. 151–163\.External Links:[Document](https://dx.doi.org/10.1016/j.imed.2025.03.002)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[5\]K\. Chlasta and K\. Wołk\(2021\)Towards computer\-based automated screening of dementia through spontaneous speech\.Frontiers in Psychology11\.External Links:[Document](https://dx.doi.org/10.3389/fpsyg.2020.623237)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1),[§II](https://arxiv.org/html/2610.20877#S2.p1.1)\.
- \[6\]H\. Ding, A\. Lister, C\. Karjadi, and et al\.\(2024\)Detection of mild cognitive impairment from non\-semantic, acoustic voice features: the framingham heart study\.JMIR Aging7,pp\. e55126\.External Links:[Document](https://dx.doi.org/10.2196/55126)Cited by:[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p3.1)\.
- \[7\]A\. Filali Razzouki, L\. Jeancolas, S\. Sambin, and et al\.\(2025\)Explaining facial action units’ correlation with hypomimia and clinical scores in parkinson’s disease\.npj Parkinson’s Disease11,pp\. 53\.External Links:[Document](https://dx.doi.org/10.1038/s41531-025-00895-3)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p2.1),[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p2.1)\.
- \[8\]Z\. Jahan, S\. Bhatia, and M\. Saraee\(2024\)Early dementia detection with speech analysis and machine learning techniques\.Discover Sustainability5\.External Links:[Document](https://dx.doi.org/10.1007/s43621-024-00217-2)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p1.1)\.
- \[9\]A\. Ksibi, A\. Walha, M\. Zakariah, and et al\.\(2025\)Multimodal siamese networks for dementia detection from speech in women\.Scientific Reports15\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-13902-7)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p1.1)\.
- \[10\]P\. Kundu Maji, S\. Acharya, P\. Paul, and et al\.\(2025\)Deep learning inspired game\-based cognitive assessment for early dementia detection\.Engineering Applications of Artificial Intelligence142,pp\. 109901\.External Links:[Document](https://dx.doi.org/10.1016/j.engappai.2024.109901)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p2.1),[§II](https://arxiv.org/html/2610.20877#S2.p1.1),[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p2.1)\.
- \[11\]M\. Kuprashevich and I\. Tolstykh\(2023\)MiVOLO: multi‑input transformer for age and gender estimation\.arXiv preprint\.External Links:arXiv:2307\.04616,[Document](https://dx.doi.org/10.48550/arXiv.2307.04616)Cited by:[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p2.1)\.
- \[12\]E\. Kurtz, Y\. Zhu, T\. Driesse, and et al\(2023\)Early detection of cognitive decline using voice assistant commands\.InICASSP 2023 \- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p1.1)\.
- \[13\]M\. Leming and H\. Im\(2025\)Differential dementia detection from multimodal brain images in a real\-world dataset\.Alzheimer’s & Dementia21\(7\),pp\. e70362\.External Links:[Document](https://dx.doi.org/10.1002/alz.70362)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[14\]Y\. Liao, T\. Wai, T\. Liao, H\. Chang, Y\. Chang, and L\. Fu\(2026\)Mild cognitive impairment detection system based on unstructured spontaneous speech: longitudinal dual\-modal framework\.JMIR Medical Informatics14,pp\. e80883\.Note:PMID: 41538772, PMCID: PMC12807404External Links:[Document](https://dx.doi.org/10.2196/80883)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1)\.
- \[15\]N\. Lin, Y\. Zhu, X\. Liang, and et al\.\(2024\)Analyzing multimodal features of spontaneous voice assistant commands for mild cognitive impairment detection\.InProc\. Interspeech,pp\. 3030–3034\.External Links:[Document](https://dx.doi.org/10.21437/interspeech.2024-2288)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[16\]S\. Maity and M\. J\. Saikia\(2025\)Large language models in healthcare and medical applications: a review\.Bioengineering12\(6\),pp\. 631\.External Links:[Document](https://dx.doi.org/10.3390/bioengineering12060631)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[17\]N\. M\. Mbaye, M\. M\. Danziger, M\. Rosen\-Zvi, A\. Toussaint, and et al\.\(2025\)Multimodal behrt: transformers for multimodal electronic health records to predict breast cancer prognosis\.Frontiers in Oncology15,pp\. 1496215\.External Links:[Document](https://dx.doi.org/10.3389/fonc.2025.1496215)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1),[§V\-B](https://arxiv.org/html/2610.20877#S5.SS2.p1.1)\.
- \[18\]K\. Qi, Y\. Zhu, C\. Summerour, and et al\(2025\)Cog\-tipro: iterative prompt refinement with llms to detect cognitive decline via longitudinal voice assistant commands\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.17137)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[19\]N\. Rahim, T\. Abuhmed, S\. Mirjalili, and et al\.\(2023\)Time\-series visual explainability for alzheimer’s disease progression detection for smart healthcare\.Alexandria Engineering Journal82,pp\. 484–502\.External Links:[Document](https://dx.doi.org/10.1016/j.aej.2023.09.050)Cited by:[§III\-B](https://arxiv.org/html/2610.20877#S3.SS2.p1.1)\.
- \[20\]N\. Rezaii, B\. Wong, P\. Aisen, and et al\.\(2025\)Voiceprints of cognitive impairment: analyzing digital voice for early detection of alzheimer’s and related dementias\.npj Dementia1,pp\. 35\.External Links:[Document](https://dx.doi.org/10.1038/s44400-025-00040-0)Cited by:[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p3.1)\.
- \[21\]A\. K\. Roy, H\. K\. Kathania, A\. Sharma, and et al\.\(2024\)ResEmoteNet: bridging accuracy and loss reduction in facial emotion recognition\.arXiv preprint\.External Links:arXiv:2409\.10545,[Document](https://dx.doi.org/10.48550/arXiv.2409.10545)Cited by:[§III\-A](https://arxiv.org/html/2610.20877#S3.SS1.p2.1)\.
- \[22\]J\. Silva, M\. Santos, and A\. Fonseca\(2026\)Multimodal deep learning applications in neurodegenerative disease screening: a systematic review\.MDPI Diagnostics16\(2\),pp\. 142\.External Links:[Document](https://dx.doi.org/10.3390/diagnostics16020142)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1)\.
- \[23\]R\. Singh, S\. Vaish, X\. Liang, and et al\.\(2024\)Clinical geriatric functional assessment using wearable sensing and machine learning\.In2024 IEEE International Conference on E\-health Networking, Application & Services \(HealthCom\),Vol\.,pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1109/HealthCom60970.2024.10880842)Cited by:[§III\-B](https://arxiv.org/html/2610.20877#S3.SS2.p1.1)\.
- \[24\]R\. A\. Taylor, A\. Gilson, L\. Chi, and et al\.\(2023\)Dementia risk analysis using temporal event modeling on a large real\-world dataset\.Scientific Reports13,pp\. 22618\.External Links:[Document](https://dx.doi.org/10.1038/s41598-023-49330-8)Cited by:[§III\-B](https://arxiv.org/html/2610.20877#S3.SS2.p1.1)\.
- \[25\]C\. U\. R\. Team\(2025\)LLMCARE: early detection of cognitive impairment via transformer models enhanced by llm\-generated synthetic data\.PubMed Central \(PMC\)Preprint,pp\. PMC12631619\.Note:PMID: 41280882Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1)\.
- \[26\]W\. Tsai, C\. Lee, and Y\. Tung\(2026\)Cross\-modal affinity\-aligned multimodal learning analytics for predicting student collaboration satisfaction in game\-based learning\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.16806)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1),[§V\-B](https://arxiv.org/html/2610.20877#S5.SS2.p1.1)\.
- \[27\]G\. Tuerhong, F\. Fu, and M\. Wushouer\(2025\)Adaptive multimodal transformer based on exchanging for multimodal sentiment analysis\.Scientific Reports15,pp\. 27265\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-11848-4)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p3.1),[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.
- \[28\]U\.S\. Department of Health and Human Services\(2025\)Mild cognitive impairment\.Alzheimers\.gov\.Note:\[Online; accessed 7\-October\-2025\]External Links:[Link](https://www.alzheimers.gov/alzheimers-dementias/mild-cognitive-impairment)Cited by:[§I](https://arxiv.org/html/2610.20877#S1.p1.1)\.
- \[29\]G\. Yang, Y\. Du, W\. Hou, Q\. Niu, and S\. Wang\(2026\)PRA\-poe: robust alzheimer’s diagnosis with arbitrary missing modalities\.pp\.\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.13081)Cited by:[§V\-B](https://arxiv.org/html/2610.20877#S5.SS2.p1.1)\.
- \[30\]S\. Zhou, Z\. Xu, M\. Zhang, and et al\.\(2025\)Large language models for disease diagnosis: a scoping review\.npj Artificial Intelligence1\(1\),pp\. 9\.External Links:[Document](https://dx.doi.org/10.1038/s44387-025-00011-z)Cited by:[§II](https://arxiv.org/html/2610.20877#S2.p2.1)\.