Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection

# Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection
Source: [https://arxiv.org/html/2606.24985](https://arxiv.org/html/2606.24985)
Louis Simon Institut des Systèmes Intelligents et de Robotique Sorbonne University Paris, France louis\.simon@isir\.upmc\.fr &Mohamed Chetouani Institut des Systèmes Intelligents et de Robotique Sorbonne University Paris, France mohamed\.chetouani@isir\.upmc\.fr

###### Abstract

Personalization in wearable\-based stress detection remains challenging due to substantial inter\-individual variability in physiological and behavioral responses\. While traditional approaches rely on user\-specific fine\-tuning or costly self\-supervised pre\-training on large datasets, we propose a lightweight alternative based on retrieval\-augmented personalization\. Our method leverages frozen, out\-of\-domain foundation models to retrieve similar patterns from a target user’s history and encode them into a compact personalized embedding that modulates representations extracted by a lightweight transformer network\. We evaluate our approach on the WESAD stress detection dataset with N=15 users, comprising wrist\-worn physiological \(EDA, BVP, temperature\) and activity \(accelerometer\) signals, and report gains of \+3\.92% in accuracy and \+4\.76% in macro F1\-score over a non\-personalized transformer baseline, approaching supervised fine\-tuning performance without requiring any labeled user data\. We further show that temporal retrieval, where only prior user samples are available, achieves performance close to full intra\-user retrieval, demonstrating robustness to limited user history\. Finally, we explore personalization in a cross\-dataset retrieval setting, leveraging embeddings from the K\-Emocon dataset to personalize representations for stress detection on the WESAD dataset\.

*Keywords*Personalization, Foundation Models, Affective Computing

## 1Introduction

Personalization refers to the process of tailoring a model’s output to an individual’s profile, preferences, or behaviors\. In the context of affective computing, personalization pertains to accounting for individual behavioral differences that exist when expressing emotions or experiencing stressLiet al\.\([2023](https://arxiv.org/html/2606.24985#bib.bib163)\); Hanet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib20)\)\. Such individual differences can stem from various factors, e\.g\., age, gender, culture, personality, but also from behavioral or physiological differences, that are much more complex to unveil\. Personalization techniques in Affective Computing often leverage small subsets of labeled data from target users to train, fine\-tune or select modelsBehinaeinet al\.\([2021](https://arxiv.org/html/2606.24985#bib.bib95)\); Tayloret al\.\([2020](https://arxiv.org/html/2606.24985#bib.bib92)\); Yuet al\.\([2021](https://arxiv.org/html/2606.24985#bib.bib111)\)\. While efficient, such approaches require labeled user data, which may be unavailable in some scenarios\. With the advent of transformers and foundation models, more researchers are shifting towards generalized models to tackle datasets with high inter\-individual variabilityErturket al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib14)\); Pillaiet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib11)\); Yuanet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib10)\)\. Such approach builds on largely pre\-trained models’ ability to represent patterns and capture invariance to reduce the variability induced by individual differences\. Several studies have demonstrated the efficacy of self\-supervised pre\-training on physiological data on various affective computing downstream tasksWuet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib137)\); Dissanayakeet al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib138)\)\. Pre\-training such model can, however, be costly, requiring large datasets, which are still small compared to modern image or text pretraining dataset\. Finally, studies often fail to report meaningful metrics of personalization\. In particular, when it comes to evaluating individual users, most of the metrics report average performance across all users\. In this work, we propose methods that leverage learned representations from foundation models\. Numerous studies have used conditioning on user demographics, which can raise ethical questions, or score, either from clinical or personality tests, to personalize modelRudovicet al\.\([2018a](https://arxiv.org/html/2606.24985#bib.bib50)\); Yang and Lee \([2019](https://arxiv.org/html/2606.24985#bib.bib99)\)\. Acquiring such information could be tedious and the knowledge of the relation between high level individual traits and physiological responses is still unknown\. Therefore, we focus on exploiting abstract representations from a generalist foundation model to condition our model for personalization\. Such techniques have been extensively studied in domains such as recommendation systemsDhelimet al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib126)\); Heet al\.\([2023](https://arxiv.org/html/2606.24985#bib.bib119)\), personalized chatbotsZhanget al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib125)\), or healthcareLianget al\.\([2019](https://arxiv.org/html/2606.24985#bib.bib759)\)\. We draw inspiration from recent work on personalized LLMDoddapaneniet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib135)\); Ninget al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib109)\), and adapt it to wearable\-based stress detection\. Our contributions are as follows: \(1\) a personalized model trained end\-to\-end that conditions predictions on a user representation, \(2\) a personalization module that builds this representation by retrieving similar patterns from the user’s own history using a frozen pre\-trained model, requiring no labeled user data, \(3\) a study of cold\-start personalization, evaluating how our method behaves as the user’s history grows over time, and \(4\) insights on the role of pre\-trained models as retrieval backbones\.

## 2RELATED WORKS

### 2\.1Personalization in Affective Computing

Personalization is a key challenge when working with human behavioral data, as subjects usually exhibit high inter\-individual variability in both physiological responses and emotional expression\. Following Li et al\. surveyLiet al\.\([2023](https://arxiv.org/html/2606.24985#bib.bib163)\), personalization techniques in affective computing can be broadly split into two approaches: data\-based, which involve selecting, grouping, or modifying data prior to training, and model\-based, which consist of specific training procedures designed to account for individual differences\.

The most straightforward approach consists in training one model per individualHanet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib20)\); Caoet al\.\([2015](https://arxiv.org/html/2606.24985#bib.bib962)\)or groupsSalamet al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib963)\), as opposed to a single population\-level model\. This is however rarely feasible for deep learning models, which require substantial amounts of data per user\. A natural relaxation is to share a common feature encoder while training separate classification heads per user, i\.e\., multitask learningTayloret al\.\([2020](https://arxiv.org/html/2606.24985#bib.bib92)\); Saeedet al\.\([2018](https://arxiv.org/html/2606.24985#bib.bib91)\)\. When a small amount of labeled data is available for new users, fine\-tuning offers an efficient way to adapt a generalized model, although it is prone to overfitting with very few samplesHanet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib20)\); Behinaeinet al\.\([2021](https://arxiv.org/html/2606.24985#bib.bib95)\)\. A principled solution to this low\-data regime is meta\-learning, where users are treated as tasks and the objective is to learn an initialization that quickly adapts to new onesYuet al\.\([2021](https://arxiv.org/html/2606.24985#bib.bib111)\); Kambaleet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib964)\)\. However, all of the above approaches require at least some labeled data per user, which can be difficult and costly to acquire in practice\.

An alternative line of work avoids per\-sample annotations entirely by conditioning models on user attributes or personal information, allowing the model to learn the underlying correlation between signals and user characteristics\. For instance, Yang and LeeYang and Lee \([2019](https://arxiv.org/html/2606.24985#bib.bib99)\)condition a variational autoencoder on Big Five personality scores to learn attribute\-invariant representations of physiological signals for emotion recognition\. In a more advanced approach, Gerczuk et al\.Gerczuket al\.\([2023](https://arxiv.org/html/2606.24985#bib.bib148)\)proposed hypernetworks that generate model parameters conditioned on user metadata, enabling zero\-shot adaptation to new users without requiring any of their signal data\. In a study on personalized robot interaction with children with ASD, Rudovic et al\.Rudovicet al\.\([2018b](https://arxiv.org/html/2606.24985#bib.bib772)\)combine conditioning on a clinical severity score \(CARS\) with a multi\-branch architecture that captures group\-level variation across culture, gender, and individual, demonstrating the benefit of integrating both user attributes and structured population priors\. However, such attributes are often difficult to acquire, requiring either self\-report questionnaires or clinical diagnosis, and may introduce bias when demographic information is used as a proxy for individual physiological differences\. More broadly, these methods depend on the availability of user\-specific side information that may be impractical to collect in passive sensing scenarios\.

One way to avoid any dependence on personal information and annotation builds explicit user representations from the data only\. For instance, Barros et al\.Barroset al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib114)\)proposed an affective memory system that encodes user\-specific responses to stimuli as slots in a dynamic memory structure, updated in a continual fashion as the user interacts with the system\. Triantafyllopoulos et al\.Triantafyllopoulos and Schuller \([2024](https://arxiv.org/html/2606.24985#bib.bib97)\)proposed to encode a small set of annotated speech samples into a user embedding that conditions a speech emotion recognition model\. While their approach leverage annotated samples, they are connected at enrollment, i\.e\., only once and by user themselves, therefore enabling adaptation with minimal user\-specific information and without full model retraining\. Most closely related to our work, Tran et al\.Tranet al\.\([2023](https://arxiv.org/html/2606.24985#bib.bib960)\)learn a user embedding by jointly optimizing a supervised emotion loss regularized by a HuBERT self\-supervised objective, and perform label distribution calibration by retrieving users with similar embeddings in the HuBERT latent space\. While sharing our intuition of leveraging pretrained representations for user\-aware inference, their approach retrieves similarusersrather than similarsamples, requires labeled emotion annotations to train the user embedding, and operates on speech with a domain\-specific pretrained model\.

In contrast, our method constructs a personalization embedding from unlabeled physiological recordings via frozen, out\-of\-domain foundation models, requiring no affect annotations and no physiological pretraining, making it well\-suited to passive wearable sensing scenarios where rich unlabeled user history is available but annotation is impractical\.

### 2\.2Foundation models for wearable sensing

The rise of self\-supervised transformers has enabled the emergence of generalizable representations that transfer across tasks and domains\. In the context of affective computing and personal sensing, several studies have demonstrated that large pretrained models can serve as effective feature extractors for wearable data without task\-specific pretraining, either by using it as input features or by direct linear probing of the foundation modelsSchulleret al\.\([2026](https://arxiv.org/html/2606.24985#bib.bib967)\); Lianet al\.\([2026](https://arxiv.org/html/2606.24985#bib.bib966)\); Zhaoet al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib968)\)\. A parallel line of work has explored training domain\-specific SSL models directly on physiological signals\. For instance, Wu et al\.Wuet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib137)\)pretrained a multimodal transformer on a closed\-source dataset of 4M samples using a signal transformation pretext task, achieving strong performance on WESAD and K\-EmoCon\. Similarly inDissanayakeet al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib138)\), authors pretrained a transformer model on a collection of wearable physiology datasets\. While effective, such approaches require substantial computational resources and large\-scale physiological datasets that remain scarce compared to vision or text domains\.

More recently, foundation models specifically designed for wearable and physiological data have begun to emerge\. PaPaGeiPillaiet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib11)\)offers pretrained representations for optical physiological signals, while Erturk et al\.Erturket al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib14)\)demonstrated the value of large\-scale behavioral wearable data for health prediction\. However, these models typically assume fixed input modalities or formats, PPG\-only inputs, fixed channel configurations, or patch\-based tokenization, limiting their applicability across heterogeneous wearable setups\. Furthermore, efficient fine\-tuning strategies such as LoRA, while promising for adapting large language models, remain difficult to apply in wearable affective computing due to the absence of sufficiently large physiological foundation models and the limited dataset sizes typical of this domain\.

An alternative is to leverage out\-of\-domain foundation models whose representations transfer to physiological signals without any physiological pretraining\. Two recent studies have shown that speech foundation models can extract meaningful representations from physiological time seriesPhukanet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib6)\); Ahmed Al Dossaryet al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib8)\), achieving strong performance despite the apparent domain mismatch, though the reasons for this transferability remain an open question\. Building on this line of work, we extend the use of out\-of\-domain foundation models to time series and speech foundation models, and evaluate both families as retrieval backbones for personalized physiological affect recognition\. Crucially, we use these models in a fully frozen setting, requiring no physiological data for pretraining and supporting flexible multimodal input by processing each modality independently\.

![Refer to caption](https://arxiv.org/html/2606.24985v1/x1.png)Figure 1:CNN \+ Transformer Backbone: Wearable input signals are processed individually with Inception\-like CNNs \(left\), fused and jointly process by a Transformer\. The full architecture with max pooling and a prediction \(right\) serves as a baseline\.

## 3Methods

In this work, we draw inspiration from UserLLMNinget al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib109)\), which uses a user embedding module and interleaved cross\-attention to condition a frozen LLM on learned user embeddings\. Adapting this to wearable\-based stress detection faces three challenges: \(1\) the absence of large\-pretrained foundation models for wearable with flexible input modalities, \(2\) limited data availability and high inter\-individual heterogeneity that prevent training complex deep architectures, and \(3\) the smaller number of users in the WESAD dataset \(N=15N=15\) compared to MovieLens \(N=138N=138k in UserLLM\), making user embedding pre\-training impractical\. Instead, we propose to train the predictive branch, analogous to the LLM in UserLLM, from scratch as a lightweight transformer, while leveraging a general\-purpose out\-of\-domain Foundation Model \(FM\) to build contextualized user representations through similarity\-based pattern retrieval, as depicted in[Figure˜3](https://arxiv.org/html/2606.24985#S3.F3)\. Given an input multimodal signalxx, we extract non\-personalized representationsf​\(x\)f\(x\)from the transformer encoder, retrieve similar user patterns in the FM embedding space, and pool them into a compact embedding that modulatesf​\(x\)f\(x\)via FiLM layers\. In the following sections, we describe in detail each design steps of our approach, emphasizing on its relevance for small datasets with high inter\-individual differences\.

### 3\.1Encoder

Transformers have recently shown great potential for time series and physiological signals\. However, it can be difficult to efficiently train them with a limited amount of data, especially without pre\-training\. Therefore, we adopt a rather simple and lightweight architecture inspired by prior workDissanayakeet al\.\([2022](https://arxiv.org/html/2606.24985#bib.bib138)\)\. Each of theMMmodalities are first down\-sampled and processed by a CNN encoder built with inception\-like blocks\. After modality\-wise encoding, convolutional features are then concatenated feature\-wise, projected with a single layer feedforward neural network and passed to a shallow Transformer\. This CNN\-Transformer backbone produces a non\-personalized representationf​\(x\)f\(x\)and serves as the basis of our personalized model \(see[Figure˜1](https://arxiv.org/html/2606.24985#S2.F1)\)\.

### 3\.2Pattern Retrieval with a Frozen Foundation Model

Rather than encoding the full user history as in UserLLM, we retrieve a small set of users’ wearable patterns similar to the current input and extract a single vector from them\. This enables context\-aware personalization under limited data by capturing variability in an individual’s physiological and activity responses, thereby reducing overfitting\. To this date, only a few studies have proposed to leverage out\-of\-domain Foundation model for physiological data, using either Speech Foundation Model \(SFM\)Phukanet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib6)\); Ahmed Al Dossaryet al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib8)\), or TSFMChristensonet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib5)\)\. Here, we experiment with two TSFMs, namelyMOMENT\-1\-base\(109M parameters\)Goswamiet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib9)\), a T5\-based foundation model known for its zero\-shot performances in a variety of tasks,Chronos\-2\(120M parameters\)Ansariet al\.\([2025](https://arxiv.org/html/2606.24985#bib.bib969)\), an encoder\-only model with time and multivariate time series attention, as well asHuBERT\-base\-ls960\(90M parameters\)Hsuet al\.\([2021](https://arxiv.org/html/2606.24985#bib.bib970)\), a popular SFM which here stands as a pure out\-of\-domain encoder\.

For a given useruu, we feed each of their samples into the FM, extract embeddings and build a retrieval database𝒟uR\\mathcal\{D\}^\{R\}\_\{u\}, withNuN\_\{u\}user sampleszuR=FM​\(x\)z^\{R\}\_\{u\}=\\mathrm\{FM\}\(x\)\. Although Chronos\-2 implements group\-attention between multiple input time series, the embeddings are of the same size as HuBERT and MOMENT, i\.e\.,\(B,T,M,dF​M\)\(B,T,M,d\_\{FM\}\)withBBbeing the batch size,TTthe number of temporal tokens,MMthe number of modalities, anddF​M=768d\_\{FM\}=768the dimension of FM’s embeddings\. In order to have eachBBsamples in the form of a vector, we choose to \(a\) apply temporal pooling\(B,T,M,dF​M\)→\(B,M,dF​M\)\(B,T,M,d\_\{FM\}\)\\rightarrow\(B,M,d\_\{FM\}\), and \(b\) concatenate all modality representations\(B,T,M,dF​M\)→\(B,dF​M×M\)\(B,T,M,d\_\{FM\}\)\\rightarrow\(B,d\_\{FM\}\\times M\)\. The resulting sample embeddingszRz^\{R\}are therefore of sizeM×768=4608M\\times 768=4608\. We provide UMAP visualizations of each foundation model’s embeddings, color\-coded by label and user, in the supplementary materials\. These visualizations reveal that, despite being trained on out\-of\-domain data, all three models produce embeddings in which samples from the same user form loosely consistent clusters, providing qualitative support for the intra\-user cosine similarity retrieval used in our approach\. Then given an inputxi,ux\_\{i,u\}sample from useruu, top\-K elements from the same user are retrieved using cosine similarity from the user’s database𝒟uR\\mathcal\{D\}^\{R\}\_\{u\}, excluding the corresponding embeddingzi,uR=FM​\(xi,u\)z^\{R\}\_\{i,u\}=\\mathrm\{FM\}\(x\_\{i,u\}\)\. The resulting set of retrieved embeddings form the context set ofxi,ux\_\{i,u\}, denoted asci,uc\_\{i,u\}\. For simplicity, index⋅i,u\\cdot\_\{i,u\}will be dropped in the following section\.

![Refer to caption](https://arxiv.org/html/2606.24985v1/x2.png)Figure 2:TheCustom SetTransformerpools the set𝐜\\mathbf\{c\}ofKKretrieved samples into a smaller set by attending to learned tokensII\.
### 3\.3Retrieval\-augmented personalization

The set of retrieved pattern𝐜\\mathbf\{c\}is then used to condition representation from the main predictive transformerf​\(x\)f\(x\)through a personalized decoder\. While UserLLM learns interleaved cross\-attention modules in a frozen LLM to condition prediction on user embedding, we choose a much lighter yet efficient approach for conditioning, namely Feature\-wise Modulation Layer \(FiLM\)Perezet al\.\([2018](https://arxiv.org/html/2606.24985#bib.bib1)\)\. We first pool theKKretrieved sample into a smaller set with lower dimension elements using a custom and simplified version of the SetTransformerLeeet al\.\([2019](https://arxiv.org/html/2606.24985#bib.bib2)\)where a set learnable inducing pointsI∈ℝK′×ds​e​tI\\in\\mathbb\{R\}^\{K^\{\\prime\}\\times d\_\{set\}\}, withK′<<KK^\{\\prime\}<<K, are cross\-attended with retrieved contextcc\(see[Figure˜2](https://arxiv.org/html/2606.24985#S3.F2)\)\. This allow us to effectively reduce both the number of elements in the context as well as the retrieved embeddings’ dimension\. Then, we further reduce the set size to obtain a single condition vectorc¯i=AvgPool​\(𝐜\)∈ℝds​e​t\\bar\{c\}\_\{i\}=\\mathrm\{AvgPool\}\(\\mathbf\{c\}\)\\in\\mathbb\{R\}^\{d\_\{set\}\}through average pooling of theK′K^\{\\prime\}elements of the reduced set\. This conditioning vector is then used as input to a FiLM layer generating the modulating parametersγ\\gammaandβ\\beta\. We apply such conditioning in a modified self attention block described as:

fF​i​L​M​\(x\)\\displaystyle f\_\{FiLM\}\(x\)=γ​f​\(x\)\+β\\displaystyle=\\gamma f\(x\)\+\\beta\(1\)fM​H​A​\(x\)\\displaystyle f\_\{MHA\}\(x\)=MHA​\(Norm​\(fF​i​L​M​\(x\)\)\)\+f​\(x\)\\displaystyle=\\mathrm\{MHA\}\(\\mathrm\{Norm\}\(f\_\{FiLM\}\(x\)\)\)\+f\(x\)\(2\)fp​e​r​s​o​\(x\)\\displaystyle f\_\{perso\}\(x\)=FFN​\(Norm​\(fM​H​A​\(x\)\)\)\+fM​H​A​\(x\)\\displaystyle=\\mathrm\{FFN\}\(\\mathrm\{Norm\}\(f\_\{MHA\}\(x\)\)\)\+f\_\{MHA\}\(x\)\(3\)wherefp​e​r​s​o​\(x\)f\_\{perso\}\(x\)is the final personalized representation, and FFN, MHA stands for fully connected network and multi\-head attention respectively\. Personalized prediction is then obtained with a 2\-layer FFN withzp​e​r​s​o​\(x\)=MaxPool​\(fp​e​r​s​o​\(x\)\)\+MaxPool​\(f​\(x\)\)z\_\{perso\}\(x\)=\\mathrm\{MaxPool\}\(f\_\{perso\}\(x\)\)\+\\mathrm\{MaxPool\}\(f\(x\)\)as input\.

To prevent non\-personalized representation collapse in early stages due to poor performance of the personalized branch, we choose to perform multi\-task learning and predict both the base labely^b​a​s​e\\hat\{y\}\_\{base\}fromf​\(x\)f\(x\)and the personalized predictiony^p​e​r​s​o\\hat\{y\}\_\{perso\}fromfp​e​r​s​o​\(x\)f\_\{perso\}\(x\)\. The model is trained on the sum of base and personalized Cross\-Entropy losses\. In order to dynamically balance these losses, we apply exponential moving average such that the final loss at stepssis equal to:

ℒt​o​ts=CE​\(y^b​a​s​e,y\)/αb​a​s​es\+CE​\(y^p​e​r​s​o,y\)/αp​e​r​s​os\\mathcal\{L\}\_\{tot\}^\{s\}=\\mathrm\{CE\}\(\\hat\{y\}\_\{base\},y\)/\\alpha\_\{base\}^\{s\}\+\\mathrm\{CE\}\(\\hat\{y\}\_\{perso\},y\)/\\alpha\_\{perso\}^\{s\}\(4\)
where the normalization terms are updated as:

αp​e​r​s​os\\displaystyle\\alpha\_\{perso\}^\{s\}=0\.95⋅αp​e​r​s​os−1\+0\.05⋅CE​\(y^p​e​r​s​o,y\),\\displaystyle=0\.95\\cdot\\alpha\_\{perso\}^\{s\-1\}\+0\.05\\cdot\\mathrm\{CE\}\(\\hat\{y\}\_\{perso\},y\),\(5\)αb​a​s​es\\displaystyle\\alpha\_\{base\}^\{s\}=0\.95⋅αb​a​s​es−1\+0\.05⋅CE​\(y^b​a​s​e,y\)\\displaystyle=0\.95\\cdot\\alpha\_\{base\}^\{s\-1\}\+0\.05\\cdot\\mathrm\{CE\}\(\\hat\{y\}\_\{base\},y\)\(6\)
These terms act as adaptive scaling factors for each loss at every gradient step \(i\.e\., each batch\)\. For instance, ifCE​\(y^p​e​r​s​o,y\)\\mathrm\{CE\}\(\\hat\{y\}\_\{perso\},y\)increases sharply at a given step,αp​e​r​s​os\\alpha\_\{perso\}^\{s\}only partially reflects this increase due to the momentum term \(0\.95\), effectively reducing the relative contribution of the personalized branch\. This ensures that, during early training when the personalized branch is unstable, its influence is automatically dampened, thereby preserving the base representation\.

![Refer to caption](https://arxiv.org/html/2606.24985v1/x3.png)Figure 3:Our proposed personalized modelconditions non\-personalized representationf​\(x\)f\(x\)with intra\-user contextual representations built via foundation model embedding retrieval\.
### 3\.4Datasets

In this work, we focus on wearable\-based data, collecting physiological and accelerometer data, applied to stress\. We use the WESAD datasetSchmidtet al\.\([2018](https://arxiv.org/html/2606.24985#bib.bib25)\)which consists of wrist\- and chest\-worn physiological and activity data ofN=15N=15subjects who go through various stressful or calming situations during a lab study\. For each user, we extract electro\-dermal activity \(EDA\), temperature \(TEMP\), blood volume pressure \(BVP\), and accelerometer \(ACC\)\. In the original WESAD study, windows with a stride of 0\.3 seconds are extracted\. To avoid redundant sampling in the FM embedding space while preserving enough information for encoding, we extract 60\-second windows using a higher stride of 3 seconds\. We then select windows with only one corresponding signal, either baseline \(0\), stress \(1\), or amusement \(2\)\. We used code from thedl\-4\-tscGithub repository111https://github\.com/Emognition/dl\-4\-tscfor data extraction and modified it to perform our pre\-processing\. We employ similar preprocessing as inWuet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib137)\), applying a low\-pass Butterworth filter with a cutoff frequency of 0\.5 Hz to the EDA signals, and a similar one with a cutoff frequency of 2 Hz for the BVP signal\. For the accelerometry, we apply a low\-pass filter with cutoff frequency of 10 Hz\. Each user’s signals are then standardized using z\-score normalization, and downsampled to 4 Hz\. After extracting windows of 60 seconds with 3 seconds stride, we obtain 11044 samples, with a label distribution of 54 / 30 / 16 for baseline, stress, and amusement respectively\. Each user’s dataset comprises 736 samples on average and label distributions are consistent with the full dataset\.

Along with the main intra\-user context\-aware personalization, we explore cross\-dataset retrieval by using the K\-EmoCon datasetParket al\.\([2020](https://arxiv.org/html/2606.24985#bib.bib89)\)as our retrieval database𝒟uR=FM​\(𝒟K​e​m​o​C​o​n\)\\mathcal\{D\}\_\{u\}^\{R\}=\\mathrm\{FM\}\(\\mathcal\{D\}\_\{KemoCon\}\)\. K\-Emocon is a multimodal dataset of naturalistic conversations, with comprehensive annotations of continuous emotions\. The dataset contains multimodal measurements, including audiovisual recordings, EEG, and peripheral physiological signals, acquired with off\-the\-shelf devices from 16 sessions of approximately 10\-minute long paired debates on a social issue\. Similarly to WESAD, the K\-EmoCon dataset utilizes the Empatica E4222https://www\.empatica\.com/en\-eu/research/e4/\. We build on the original preprocessing code333https://github\.com/Kaist\-ICLab/K\-EmoCon\_SupplementaryCodesto remove baseline calibration and extract debate sections and the corresponding wearable signals\. We then adopt the same filtering and standardization as above\.

### 3\.5Evaluation

In order to evaluate general performance of our approach in a robust way, we perform a repeated Leave\-One\-Subject\-Out Cross\-Validation \(LOSO CV\) with S = 5 different random seeds\. We present three different analysis to \(a\) analyze overall performances with respect to baselines, \(b\) assess cross\-user capabilities in cold start scenarios, and \(c\) investigate cross\-dataset retrieval\.

#### General performances

We first report average and standard deviation of accuracy and macro F1\-score across the 5 random seeds\. To further validate our model’s ability to guarantee uniform improvement for most users, we report the median of per user differenceMedian\(Δ\\Delta\) in % of accuracy between the Transformer baseline and our personalized model\. Looking at this median of differences rather than the difference of average between the baseline and the personalized model helps us identify if the personalization is beneficial for all users or not\. We also report the personalization ratio % corresponding to the percentage of users who got their performance increased with personalization out ofN=15N=15individuals w\.r\.t the Transformer baseline\. Finally, to ensure the efficiency of our method, we perform a Linear Mixed Model \(LMM\) analysis on the accuracy metric to compare our model to baselines, accounting for both random effects \(seed\) and dependent fixed effects \(users\), and verify that the Gaussian assumption is satisfied\.

#### Cold\-start

Personalization at deployment requires adapting to new users with limited or no prior data, commonly referred to as the cold\-start problem\. Our model retrieves intra\-user patterns to condition predictions, which requires unlabeled samples from the target user and a foundation model to compute their embeddings\. We investigate two retrieval configurations that simulate the cold\-start problem under realistic deployment constraints\. In the first setting, we simulate a user whose history grows over time by applying a chronological mask to target user data, retaining only samples recorded prior to each query window\. The resulting target user embeddings are merged with training user embeddings to form a hybrid retrieval database, progressively shifting from inter\-user to intra\-user retrieval as more data accumulates\. We refer to this setting ashybrid retrieval\. In the second setting, no target user history is available and retrieval relies exclusively on pre\-stored embeddings from training users\. Beyond simulating the absence of target user data, this setting also addresses a practical constraint: by pre\-storing training user embeddings, the foundation model need not run on\-device, reducing memory and privacy overhead\. We denote this configuration asinter user retrieval\. We report average for all personalized model in both cold start settings in[Table˜2](https://arxiv.org/html/2606.24985#S4.T2)\.

#### Cross\-dataset retrieval

We further investigate cross\-dataset retrieval, where the retrieval database is populated with embeddings from a different dataset rather than the target user’s own history\. Specifically, query embeddings are computed from WESAD samples as usual, but the retrieval database consists exclusively of K\-EmoCon physiological recordings\. When a new sample arrives, its foundation model embedding is matched against K\-EmoCon embeddings, and the top\-K most similar are pooled via the SetTransformer to form the conditioning vector, following the same pipeline as intra\-user retrieval\. While both datasets share the same sensing device and physiological modalities, their underlying contexts differ\. WESAD involves passive participants exposed to controlled stress stimuli, characterized by phase\-dependent, low\-frequency trends due to controlled stress stimulus\. On the other hand, K\-EmoCon captures active individuals engaged in naturalistic debates with continuous arousal and valence annotations, with less predictable physiological responses\. This cross\-dataset retrieval setting therefore evaluates whether foundation model embeddings can capture physiological patterns that generalize across datasets despite differences in affective context and sensing conditions\. We report cross\-dataset results in[Table˜3](https://arxiv.org/html/2606.24985#S4.T3)\. Along with accuracy and macro F1\-score, we report the mean normalized concentration index across users, denoted asCuC\_\{u\}\. This is computed as\(Hm​a​x−Hu\)/Hm​a​x\(H\_\{max\}\-H\_\{u\}\)/H\_\{max\}, whereHm​a​x=log⁡NH\_\{max\}=\\log Nis the maximum entropy under uniform retrieval across theN=26N=26K\-EmoCon subjects, andHuH\_\{u\}is the empirical entropy of useruu’s retrieval distribution over those subjects\. A value of 0 indicates that retrieval is uniformly distributed across all subjects in the pool, while a value of 1 indicates that all retrieved samples originate from a single subject\.

### 3\.6Baselines

We first compare our model to fully supervised approaches, namely a Random Forest with statistical functionals as inputs, and a baseline Transformer, depicted in[Figure˜1](https://arxiv.org/html/2606.24985#S2.F1)\. To assess the representational quality of out\-of\-domain foundation models without task\-specific adaptation, we perform classification using a simple RBF\-kernel SVM on frozen foundation model embeddings, following the linear probing protocol proposed inGoswamiet al\.\([2024](https://arxiv.org/html/2606.24985#bib.bib9)\)\. The objective is not to maximize classification performance through approaches such as parameter\-efficient fine\-tuning, but rather to evaluate the intrinsic quality of representations produced by each foundation model\. Finally, we report results for a fine\-tuned Transformer baseline using 1% and 5% of each target user’s data\. Unlike data\-level personalization approachesLiet al\.\([2023](https://arxiv.org/html/2606.24985#bib.bib163)\)that incorporate target user samples directly into the training set, we first train the model on the N\-1 training users and then fine\-tune the whole model, including the CNN encoder and the transformer, on a small subset of the held\-out user’s samples, as inBehinaeinet al\.\([2021](https://arxiv.org/html/2606.24985#bib.bib95)\)\. This simulates personalization at deployment time for a model pre\-trained on users it has never encountered\. The model is fine\-tuned for 5 epochs with a learning rate of 1e\-5, a batch size of 16, and no weight decay\.

Table 1:Main results on WESADare reported with average and standard deviation of LOSO metrics across 5 random seeds\. Along with Accuracy and Macro F1, personalization metrics are reported forFine\-tunedandRetrieval\-based Personalizedmodels \(w\.r\.t to the baseline Transformer\)\.
### 3\.7Implementation details

#### CNN\+Transformer encoder

We implement the CNN encoder with 64 filters per modality, yielding concatenated features of size\(L,F×M\)=\(240,64×6=384\)\(L,F\\times M\)=\(240,64\\times 6=384\)\. These convolutional features are then fed to a 6\-layer Transformer withdm​o​d​e​l=128d\_\{model\}=128, 4 attention heads, and a feedforward dimension of4×dm​o​d​e​l=5124\\times d\_\{model\}=512\.

#### Foundation model embeddings

We employ two strategies for extracting foundation models’ embeddings\. Given that Chronos\-2 supports variable\-length multivariate input, we therefore feed the downsampled sequence of size\(L,M\)=\(240,6\)\(L,M\)=\(240,6\)directly to the model\. For MOMENT and HuBERT, we apply linear interpolation to upsample the wearable signals to the required input length\. MOMENT expects sequences of length 512, while HuBERT has no fixed length requirement but is designed for audio sampled at 16kHz, to which we upsample accordingly\. Each modality is processed independently by the foundation model, yielding embeddings of size\(L,M,dF​M\)=\(240,6,768\)\(L,M,d\_\{FM\}\)=\(240,6,768\), which are then average\-pooled alongLLand concatenated across modalities to produce a single vector of sizeM×dF​M=4608M\\times d\_\{FM\}=4608per sample\.

#### Personalization modules

We retrieveK=16K=16samples per query, corresponding to approximately 2% of each user’s available history, providing sufficient diversity while limiting noise\. The retrieved context is pooled by a custom SetTransformer with hidden dimensionds​e​t=128d\_\{set\}=128, 4 heads for both self\- and cross\-attention, andK′=4K^\{\\prime\}=4inducing points\. The resulting average\-pooled conditioning vectorc¯\\bar\{c\}is projected into a 64\-dimensional space before being passed to 2 blocks of FiLM \+ Self\-Attention with the same hidden dimension\.

#### Full model and training

Both the personalized and base prediction heads consist of a linear layer with ReLU activation mapping to 64 dimensions, followed by a final linear layer producing class logits\. To limit overfitting under LOSO, we adopt a compact architecture with aggressive regularization, namely high weight decay and high dropout, combined with learning rate scheduling\. Both the baseline Transformer and the retrieval\-based personalized model are trained for 20 epochs with a learning rate of10−410^\{\-4\}, a weight decay of10−210^\{\-2\}, and a batch size of 128, with a minimum learning rate of10−610^\{\-6\}\. All models were trained on an NVIDIA RTX A6000, with training times ranging from 35 minutes to 1 hour for the full LOSO evaluation depending on the configuration444Code used for experiments will be made available upon acceptance\.

## 4Results

### 4\.1Main results

We first analyze the overall performance of our personalized model and report results in[Table˜1](https://arxiv.org/html/2606.24985#S3.T1)\. The Random Forest baseline achieves 80\.73% accuracy and 75\.44% F1, while the baseline Transformer improves upon it at 83\.99% accuracy and 77\.07% F1\. Linear probing of frozen foundation model embeddings performs below both supervised baselines across all models, with the exception of Chronos which marginally exceeds the Random Forest in accuracy\. This confirms that out\-of\-domain foundation models do not produce representations sufficiently discriminative for direct stress classification, and that naive transfer of these embeddings to physiological affect recognition is insufficient\.

On the other hand, all three variants of our retrieval\-based personalization approach consistently outperform the non\-personalized baseline Transformer, with MOMENT achieving the highest accuracy \(87\.91%±\\pm0\.56%\) and macro F1\-score \(81\.83%±\\pm0\.84%\), corresponding to gains of \+3\.92% and \+4\.76% respectively\. The LMM analysis confirms that these gains are statistically significant \(β\\beta= 3\.915, 95% CI \[2\.590, 5\.241\], z = 5\.789, p < 0\.001 for MOMENT,β\\beta= 3\.490, 95% CI \[2\.402, 4\.578\], z = 6\.287, p < 0\.001 for Chronos, andβ\\beta= 2\.950, 95% CI \[1\.776, 4\.124\], z = 4\.926, p < 0\.001\)\. When it comes to personalization gain, Chronos achieves the highest median per\-user gain \(Median\(Δ\\Delta\) = 3\.49\) and win rate \(78\.67%\), suggesting more consistent improvements across individuals than MOMENT despite slightly lower aggregate performance\. Finally, our method approaches the performance of supervised fine\-tuning with 1% of target user data \(88\.08% / 84\.85%\) without requiring any labeled user samples, closing most of the gap between the non\-personalized baseline and the fine\-tuned model\. It remains however well below 5% fine\-tune, indicating that our model still fails at capturing some users’ specificity\. A detailed per\-user breakdown of accuracy and macro F1\-score for baseline Transformer and our proposed model is provided in the supplementary materials\.

### 4\.2Cold\-start

![Refer to caption](https://arxiv.org/html/2606.24985v1/x4.png)Figure 4:The temporal evolution of the cross\-user retrieval ratio in hybrid cold\-start\.We report in[Table˜2](https://arxiv.org/html/2606.24985#S4.T2)results under both cold\-start configurations\. The hybrid retrieval setting, which combines chronologically prior samples from the target user with training user embeddings, achieves performance close to full intra\-user retrieval \(see[Table˜1](https://arxiv.org/html/2606.24985#S3.T1)\) across all foundation models \(MOMENT: 87\.27% accuracy, 81\.20% F1; Chronos: 87\.04% / 81\.33%; HuBERT: 85\.33% / 78\.92%\)\. This demonstrates that even a partial, temporally ordered user history is sufficient for effective personalization\. On the other hand, inter\-user retrieval, i\.e\., where no target user data is available and retrieval relies exclusively on training user embeddings, yields substantially lower performance across all models \(MOMENT: 78\.90% / 63\.23%; Chronos: 78\.79% / 68\.54%; HuBERT: 77\.92% / 69\.04%\), falling below the non\-personalized baseline\. This suggests that the personalization gain partially stems from user\-specific physiological patterns rather than generalized, user\-invariant representations of wearable signals, validating our intra\-user retrieval approach\.

Table 2:Cold\-start resultsin hybrid and inter\-user only settings\.To further analyze hybrid cold\-start, we represent in[Figure˜4](https://arxiv.org/html/2606.24985#S4.F4)the temporal evolution of the cross\-user retrieval ratio, averaged across all users and seeds with±1\\pm 1std\. At the beginning of each session, the temporal mask ensures that all retrieved embeddings originate from other users, yielding a cross\-user ratio of 100%\. This ratio decreases to approximately 30% as the target user accumulates history, with a few notable peaks corresponding to transitions between protocol phases\. The WESAD protocol alternates between baseline, amusement, and stress phases, with the order of the last two permuted across participants\. We represent this in the figure with Group 1→\\rightarrowbaseline, amusement, stress and Group 2→\\rightarrowbaseline, stress, amusement\. At each phase transition, the target user has no prior embeddings from the new phase, causing the model to temporarily fall back on other users’ embeddings, with cross\-user retrieval rising to approximately 70% on average at the first transition\. This behavior suggests that the model adaptively leverages inter\-user context when intra\-user history is insufficient, rather than relying on potentially irrelevant recent samples from a different phase\. The first transition yields the highest peak, which we attribute to the frequent confusion between baseline and amusement phases observed in our predictions, as these two states are the most physiologically similar in the WESAD protocol\.

### 4\.3Cross\-dataset retrieval

Table 3:Cross\-dataset retrievalresults on the K\-EmoCon dataset\.Cross\-dataset retrieval results are reported in[Table˜3](https://arxiv.org/html/2606.24985#S4.T3), where the retrieval database exclusively contains K\-EmoCon embeddings\. All three foundation models achieve performance close to the non\-personalized baseline Transformer, which is notably higher than inter\-user retrieval reported in[Table˜2](https://arxiv.org/html/2606.24985#S4.T2)\. This suggests that K\-EmoCon physiological recordings, despite originating from a different affective context, provide more useful retrieval context than other WESAD users’ embeddings\. However, the mean normalized concentration indexCuC\_\{u\}remains low across users, indicating that retrieval is broadly distributed across K\-EmoCon subjects rather than concentrating on a few physiologically similar ones\. Drawing on our observations in[Section˜4\.2](https://arxiv.org/html/2606.24985#S4.SS2), where consistent intra\-user retrieval was shown to be key for effective personalization, the low concentration index suggests that the model cannot identify sufficiently consistent cross\-dataset patterns to build a meaningful conditioning embedding\. We hypothesize that in this setting, the model may be discarding personalized context via the residual connection in the personalized branch, effectively falling back on the base representation\.

To verify this, we measure the personalization magnitude, defined as the normalized difference between the base and personalized representations in the final prediction:‖Δ​z‖2/‖zb​a​s​e‖2\|\|\\Delta z\|\|\_\{2\}/\|\|z\_\{base\}\|\|\_\{2\}, wherezb​a​s​e=MaxPool​\(f​\(x\)\)z\_\{base\}=\\mathrm\{MaxPool\}\(f\(x\)\)andΔ​z=zb​a​s​e−MaxPool​\(fp​e​r​s​o​\(x\)\)\\Delta z=z\_\{base\}\-\\mathrm\{MaxPool\}\(f\_\{perso\}\(x\)\)\. A value close to zero indicates that the personalized branch contributes little beyond the base representation\.

![Refer to caption](https://arxiv.org/html/2606.24985v1/img/residual_dist_Personalized_vs_Cross-dataset.png)Figure 5:Personalization magnitudemeasures the contribution of personalized representation in the final prediction \(the closer to zero, the weaker the personalization\)\.[Figure˜5](https://arxiv.org/html/2606.24985#S4.F5)shows the distribution of this quantity for the full intra\-user model and the cross\-dataset variant, both using MOMENT\. The intra\-user model exhibits a broad, high\-magnitude distribution, reflecting active personalization across users and labels\. In contrast, the cross\-dataset model shows a sharp peak at low magnitude, confirming that the model suppresses the personalized context when it originates from K\-EmoCon, relying predominantly on the base representation\. This may indicate why cross\-dataset performance approaches the non\-personalized baseline rather than full intra\-user personalization\.

### 4\.4Ablation studies

Table 4:Ablation study: Results are averaged over 5 seeds\. The†non\-personalized baseline and our∗personalized model \(MOMENT version\) constitute lower and upper bound respectively\.Table[4](https://arxiv.org/html/2606.24985#S4.T4)reports ablation results averaged over 5 seeds\. Replacing the foundation model retrieval backbone with a VAE trained on WESAD physiological signals substantially degrades performance \(81\.48% accuracy, 75\.28% F1\), falling below the non\-personalized baseline \(83\.99% / 77\.07%\)\. This confirms that the pretrained out\-of\-domain foundation model’s representations are essential for meaningful retrieval\. A domain\-specific unsupervised model such as a VAE trained on limited physiological data does not provide sufficient embedding quality to support effective personalization\. Replacing FiLM conditioning with simple concatenation yields a modest reduction \(87\.33% / 81\.21% vs\. 87\.91% / 81\.83%\), suggesting that our method is robust to the choice of conditioning mechanism, and that the primary source of personalization gain lies in the quality of the retrieved embeddings rather than the conditioning architecture\.

## 5Conclusion

In this study, we introduced a personalization approach that retrieves similar patterns from a user’s own history using frozen, out\-of\-domain foundation models to condition a lightweight stress detection model\. Our method, trained end\-to\-end without any labeled user data from target users, consistently outperforms a non\-personalized baseline and approaches the performance of supervised fine\-tuning\. We further show that even with limited user history, our method remains effective, and that the model naturally discards uninformative retrieval context when it is not useful\. These results suggest that frozen foundation models, despite being trained on unrelated data, can serve as effective retrieval backbones for personalization in domains where large\-scale physiological data is scarce\. There are, however, few limitations to this work\. While our approach eliminates the need for labeled user data, it still requires sufficient unlabeled samples from each user, limiting applicability in true cold\-start scenarios\. Additionally, despite uniform gains, some users still show modest improvements, and the foundation model’s size may constrain deployment in resource\-limited settings\. Finally, despite exploring cross\-dataset retrieval, we evaluated our approach only on one dataset\. Future work will explore model compression, investigate other out\-of\-domain foundation models and their fine\-tuning, e\.g\., through LoRA, and extension to other multimodal affect recognition settings\.

## Safe and Responsible Innovation Statement

In this study, we propose a novel personalization method and evaluate it on two publicly available datasets, namely WESAD and K\-EmoCon, collected in laboratory settings with participants’ informed consent\. No demographic attributes, e\.g\., age, gender, or ethnicity, were used in order to limit potential bias\. However, our proposed method relies on unlabeled physiological data collected from target users, which raises privacy considerations in deployment and calls for explicit informed consent from participants\. We emphasize that the proposed method is not intended for clinical or diagnostic use, as the laboratory\-controlled conditions used for data collection may not reflect real\-world scenarios\.

## References

- From Speech and PPG to EDA: Stress Detection Based on Cross\-Modal Fine\-Tuning of Foundation Models\.InProceedings of the 27th International Conference on Multimodal Interaction,Canberra Australia,pp\. 87–95\(en\)\.External Links:ISBN 9798400714993,[Link](https://dl.acm.org/doi/10.1145/3716553.3750753),[Document](https://dx.doi.org/10.1145/3716553.3750753)Cited by:[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p3.1),[§3\.2](https://arxiv.org/html/2606.24985#S3.SS2.p1.1)\.
- A\. F\. Ansari, O\. Shchur, J\. Küken, A\. Auer, B\. Han, P\. Mercado, S\. S\. Rangapuram, H\. Shen, L\. Stella, X\. Zhang, M\. Goswami, S\. Kapoor, D\. C\. Maddix, P\. Guerron, T\. Hu, J\. Yin, N\. Erickson, P\. M\. Desai, H\. Wang, H\. Rangwala, G\. Karypis, Y\. Wang, and M\. Bohlke\-Schneider \(2025\)Chronos\-2: From Univariate to Universal Forecasting\.arXiv\(en\)\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2510.15821),[Document](https://dx.doi.org/10.48550/ARXIV.2510.15821)Cited by:[§3\.2](https://arxiv.org/html/2606.24985#S3.SS2.p1.1)\.
- P\. Barros, E\. Barakova, and S\. Wermter \(2022\)Adapting the Interplay Between Personalized and Generalized Affect Recognition Based on an Unsupervised Neural Framework\.IEEE Transactions on Affective Computing13\(3\),pp\. 1349–1365\.External Links:ISSN 1949\-3045,[Link](https://ieeexplore.ieee.org/document/9117172),[Document](https://dx.doi.org/10.1109/TAFFC.2020.3002657)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p4.1)\.
- B\. Behinaein, A\. Bhatti, D\. Rodenburg, P\. Hungler, and A\. Etemad \(2021\)A Transformer Architecture for Stress Detection from ECG\.InProceedings of the 2021 ACM International Symposium on Wearable Computers,ISWC ’21,New York, NY, USA,pp\. 132–134\.External Links:ISBN 978\-1\-4503\-8462\-9,[Link](https://dl.acm.org/doi/10.1145/3460421.3480427),[Document](https://dx.doi.org/10.1145/3460421.3480427)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1),[§3\.6](https://arxiv.org/html/2606.24985#S3.SS6.p1.1)\.
- H\. Cao, R\. Verma, and A\. Nenkova \(2015\)Speaker\-sensitive emotion recognition via ranking: Studies on acted and spontaneous speech\.Computer Speech & Language28\(1\),pp\. 186–202\(eng\)\.External Links:ISSN 0885\-2308,[Document](https://dx.doi.org/10.1016/j.csl.2014.01.003)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- M\. Christenson, C\. Geary, B\. Locke, P\. Koirala, and W\. W\. Pettine \(2024\)Assessing Foundation Models’ Transferability to Physiological Signals in Precision Medicine\.arXiv\.Note:arXiv:2412\.03427 \[cs\]External Links:[Link](http://arxiv.org/abs/2412.03427),[Document](https://dx.doi.org/10.48550/arXiv.2412.03427)Cited by:[§3\.2](https://arxiv.org/html/2606.24985#S3.SS2.p1.1)\.
- S\. Dhelim, N\. Aung, M\. A\. Bouras, H\. Ning, and E\. Cambria \(2022\)A Survey on Personality\-Aware Recommendation Systems\.Artificial Intelligence Review55\(3\),pp\. 2409–2454\.Note:arXiv:2101\.12153 \[cs\]External Links:ISSN 0269\-2821, 1573\-7462,[Link](http://arxiv.org/abs/2101.12153),[Document](https://dx.doi.org/10.1007/s10462-021-10063-7)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- V\. Dissanayake, S\. Seneviratne, R\. Rana, E\. Wen, T\. Kaluarachchi, and S\. Nanayakkara \(2022\)SigRep: Toward Robust Wearable Emotion Recognition With Contrastive Representation Learning\.IEEE Access10,pp\. 18105–18120\.External Links:ISSN 2169\-3536,[Link](https://ieeexplore.ieee.org/document/9706192),[Document](https://dx.doi.org/10.1109/ACCESS.2022.3149509)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2606.24985#S3.SS1.p1.2)\.
- S\. Doddapaneni, K\. Sayana, A\. Jash, S\. Sodhi, and D\. Kuzmin \(2024\)User Embedding Model for Personalized Language Prompting\.InProceedings of the 1st Workshop on Personalization of Generative AI Systems \(PERSONALIZE 2024\),A\. Deshpande, E\. Hwang, V\. Murahari, J\. S\. Park, D\. Yang, A\. Sabharwal, K\. Narasimhan, and A\. Kalyan \(Eds\.\),St\. Julians, Malta,pp\. 124–131\.External Links:[Link](https://aclanthology.org/2024.personalize-1.12/)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- E\. Erturk, F\. Kamran, S\. Abbaspourazad, S\. Jewell, H\. Sharma, Y\. Li, S\. Williamson, N\. J\. Foti, and J\. Futoma \(2025\)Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 15516–15541\(en\)\.Note:ISSN: 2640\-3498External Links:[Link](https://proceedings.mlr.press/v267/erturk25a.html)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p2.1)\.
- M\. Gerczuk, A\. Triantafyllopoulos, S\. Amiriparian, A\. Kathan, J\. Bauer, M\. Berking, and B\. W\. Schuller \(2023\)Zero\-shot personalization of speech foundation models for depressed mood monitoring\.Patterns4\(11\),pp\. 100873\.External Links:ISSN 2666\-3899,[Link](https://www.sciencedirect.com/science/article/pii/S2666389923002635),[Document](https://dx.doi.org/10.1016/j.patter.2023.100873)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p3.1)\.
- M\. Goswami, K\. Szafer, A\. Choudhry, Y\. Cai, S\. Li, and A\. Dubrawski \(2024\)MOMENT: A Family of Open Time\-series Foundation Models\.InProceedings of the 41st International Conference on Machine Learning,pp\. 16115–16152\(en\)\.Note:ISSN: 2640\-3498External Links:[Link](https://proceedings.mlr.press/v235/goswami24a.html)Cited by:[§3\.2](https://arxiv.org/html/2606.24985#S3.SS2.p1.1),[§3\.6](https://arxiv.org/html/2606.24985#S3.SS6.p1.1)\.
- Y\. Han, P\. Zhang, M\. Park, and U\. Lee \(2024\)Systematic Evaluation of Personalized Deep Learning Models for Affect Recognition\.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies8\(4\),pp\. 1–35\(en\)\.External Links:ISSN 2474\-9567,[Link](https://dl.acm.org/doi/10.1145/3699724),[Document](https://dx.doi.org/10.1145/3699724)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- Z\. He, W\. Liu, W\. Guo, J\. Qin, Y\. Zhang, Y\. Hu, and R\. Tang \(2023\)A Survey on User Behavior Modeling in Recommender Systems\.arXiv\.Note:arXiv:2302\.11087 \[cs\]External Links:[Link](http://arxiv.org/abs/2302.11087),[Document](https://dx.doi.org/10.48550/arXiv.2302.11087)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed \(2021\)HuBERT: Self\-Supervised Speech Representation Learning by Masked Prediction of Hidden Units\.arXiv\.Note:arXiv:2106\.07447 \[cs\]External Links:[Link](http://arxiv.org/abs/2106.07447),[Document](https://dx.doi.org/10.48550/arXiv.2106.07447)Cited by:[§3\.2](https://arxiv.org/html/2606.24985#S3.SS2.p1.1)\.
- W\. V\. Kambale, D\. D’Ambrosi, M\. El Bahnasawi, M\. Salem, and K\. Kyamakya \(2024\)Analyzing the Potential Contribution of a Meta\-Learning Approach to Robust and Effective Subject\-Independent, Emotion\-Related Time Series Analysis of Bio\-signals\.InRecent Advances in Machine Learning Techniques and Sensor Applications for Human Emotion, Activity Recognition and Support,K\. Kyamakya, F\. Al Machot, H\. Ullah, and F\. Demrozi \(Eds\.\),pp\. 139–187\(en\)\.External Links:ISBN 978\-3\-031\-71821\-2,[Link](https://doi.org/10.1007/978-3-031-71821-2_5),[Document](https://dx.doi.org/10.1007/978-3-031-71821-2%5F5)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- J\. Lee, Y\. Lee, J\. Kim, A\. R\. Kosiorek, S\. Choi, and Y\. W\. Teh \(2019\)Set Transformer: A Framework for Attention\-based Permutation\-Invariant Neural Networks\.arXiv\.Note:arXiv:1810\.00825 \[cs\]External Links:[Link](http://arxiv.org/abs/1810.00825),[Document](https://dx.doi.org/10.48550/arXiv.1810.00825)Cited by:[§3\.3](https://arxiv.org/html/2606.24985#S3.SS3.p1.10)\.
- J\. Li, A\. Waleed, and H\. Salam \(2023\)A Survey on Personalized Affective Computing in Human\-Machine Interaction\.arXiv\.Note:arXiv:2304\.00377 \[cs\]External Links:[Link](http://arxiv.org/abs/2304.00377),[Document](https://dx.doi.org/10.48550/arXiv.2304.00377)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p1.1),[§3\.6](https://arxiv.org/html/2606.24985#S3.SS6.p1.1)\.
- Z\. Lian, L\. Sun, Y\. Ren, H\. Gu, H\. Sun, L\. Chen, B\. Liu, and J\. Tao \(2026\)MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition\.IEEE Transactions on Pattern Analysis and Machine Intelligence48\(5\),pp\. 5793–5810\.External Links:ISSN 1939\-3539,[Link](https://ieeexplore.ieee.org/document/11347465),[Document](https://dx.doi.org/10.1109/TPAMI.2026.3653457)Cited by:[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p1.1)\.
- Y\. Liang, X\. Zheng, and D\. D\. Zeng \(2019\)A survey on big data\-driven digital phenotyping of mental health\.Information Fusion52,pp\. 290–307\(en\)\.External Links:ISSN 1566\-2535,[Link](https://www.sciencedirect.com/science/article/pii/S1566253518305244),[Document](https://dx.doi.org/10.1016/j.inffus.2019.04.001)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- L\. Ning, L\. Liu, J\. Wu, N\. Wu, D\. Berlowitz, S\. Prakash, B\. Green, S\. O’Banion, and J\. Xie \(2025\)User\-LLM: Efficient LLM Contextualization with User Embeddings\.InCompanion Proceedings of the ACM on Web Conference 2025,Sydney NSW Australia,pp\. 1219–1223\(en\)\.External Links:ISBN 979\-8\-4007\-1331\-6,[Link](https://dl.acm.org/doi/10.1145/3701716.3715463),[Document](https://dx.doi.org/10.1145/3701716.3715463)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§3](https://arxiv.org/html/2606.24985#S3.p1.5)\.
- C\. Y\. Park, N\. Cha, S\. Kang, A\. Kim, A\. H\. Khandoker, L\. Hadjileontiadis, A\. Oh, Y\. Jeong, and U\. Lee \(2020\)K\-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations\.Scientific Data7\(1\),pp\. 293\(en\)\.Note:Publisher: Nature Publishing GroupExternal Links:ISSN 2052\-4463,[Link](https://www.nature.com/articles/s41597-020-00630-y),[Document](https://dx.doi.org/10.1038/s41597-020-00630-y)Cited by:[§3\.4](https://arxiv.org/html/2606.24985#S3.SS4.p2.1)\.
- E\. Perez, F\. Strub, H\. d\. Vries, V\. Dumoulin, and A\. Courville \(2018\)FiLM: Visual Reasoning with a General Conditioning Layer\.Proceedings of the AAAI Conference on Artificial Intelligence32\(1\) \(en\)\.External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/11671),[Document](https://dx.doi.org/10.1609/aaai.v32i1.11671)Cited by:[§3\.3](https://arxiv.org/html/2606.24985#S3.SS3.p1.10)\.
- O\. C\. Phukan, S\. R\. Behera, Girish, M\. M\. Akhtar, A\. B\. Buduru, and R\. Sharma \(2024\)Beyond Speech and More: Investigating the Emergent Ability of Speech Foundation Models for Classifying Physiological Time\-Series Signals\.arXiv\.Note:arXiv:2410\.12645 \[eess\]External Links:[Link](http://arxiv.org/abs/2410.12645),[Document](https://dx.doi.org/10.48550/arXiv.2410.12645)Cited by:[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p3.1),[§3\.2](https://arxiv.org/html/2606.24985#S3.SS2.p1.1)\.
- A\. Pillai, D\. Spathis, F\. Kawsar, and M\. Malekzadeh \(2024\)PaPaGei: Open Foundation Models for Optical Physiological Signals\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),\(en\)\.External Links:[Link](https://openreview.net/forum?id=kYwTmlq6Vn)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p2.1)\.
- O\. Rudovic, J\. Lee, M\. Dai, B\. Schuller, and R\. W\. Picard \(2018a\)Personalized machine learning for robot perception of affect and engagement in autism therapy\.Science Robotics3\(19\),pp\. eaao6760\(eng\)\.External Links:ISSN 2470\-9476,[Document](https://dx.doi.org/10.1126/scirobotics.aao6760)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- O\. Rudovic, Y\. Utsumi, J\. Lee, J\. Hernandez, E\. C\. Ferrer, B\. Schuller, and R\. W\. Picard \(2018b\)CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism\.In2018 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 339–346\.Note:ISSN: 2153\-0866External Links:[Document](https://dx.doi.org/10.1109/IROS.2018.8594177)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p3.1)\.
- A\. Saeed, T\. Ozcelebi, J\. Lukkien, J\. B\.F\. van Erp, and S\. Trajanovski \(2018\)Model Adaptation and Personalization for Physiological Stress Detection\.In2018 IEEE 5th International Conference on Data Science and Advanced Analytics \(DSAA\),pp\. 209–216\.External Links:[Link](https://ieeexplore.ieee.org/document/8631447),[Document](https://dx.doi.org/10.1109/DSAA.2018.00031)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- H\. Salam, V\. Manoranjan, J\. Jiang, and O\. Celiktutan \(2022\)Learning Personalised Models for Automatic Self\-Reported Personality Recognition\.InUnderstanding Social Behavior in Dyadic and Small Group Interactions,pp\. 53–73\(en\)\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v173/salam22a.html)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- P\. Schmidt, A\. Reiss, R\. Duerichen, C\. Marberger, and K\. Van Laerhoven \(2018\)Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection\.InProceedings of the 20th ACM International Conference on Multimodal Interaction,ICMI ’18,New York, NY, USA,pp\. 400–408\.External Links:ISBN 978\-1\-4503\-5692\-3,[Link](https://doi.org/10.1145/3242969.3242985),[Document](https://dx.doi.org/10.1145/3242969.3242985)Cited by:[§3\.4](https://arxiv.org/html/2606.24985#S3.SS4.p1.1)\.
- B\. Schuller, A\. Mallol\-Ragolta, A\. P\. Almansa, I\. Tsangko, M\. M\. Amin, A\. Semertzidou, L\. Christ, and S\. Amiriparian \(2026\)Affective computing has changed: the foundation model disruption\.npj Artificial Intelligence2\(1\),pp\. 16\(en\)\.External Links:ISSN 3005\-1460,[Link](https://www.nature.com/articles/s44387-025-00061-3),[Document](https://dx.doi.org/10.1038/s44387-025-00061-3)Cited by:[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p1.1)\.
- S\. Taylor, N\. Jaques, E\. Nosakhare, A\. Sano, and R\. Picard \(2020\)Personalized Multitask Learning for Predicting Tomorrow’s Mood, Stress, and Health\.IEEE transactions on affective computing11\(2\),pp\. 200–213\.External Links:ISSN 1949\-3045,[Link](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7266106/),[Document](https://dx.doi.org/10.1109/TAFFC.2017.2784832)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- M\. Tran, Y\. Yin, and M\. Soleymani \(2023\)Personalized Adaptation with Pre\-trained Speech Encoders for Continuous Emotion Recognition\.InProc\. Interspeech 2023,pp\. 636–640\.External Links:[Link](https://www.isca-archive.org/interspeech_2023/tran23c_interspeech.html),[Document](https://dx.doi.org/10.21437/Interspeech.2023-2170)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p4.1)\.
- A\. Triantafyllopoulos and B\. Schuller \(2024\)Enrolment\-based personalisation for improving individual\-level fairness in speech emotion recognition\.InProc\. Interspeech 2024,pp\. 3729–3733\.External Links:[Link](https://www.isca-archive.org/interspeech_2024/triantafyllopoulos24c_interspeech.html),[Document](https://dx.doi.org/10.21437/Interspeech.2024-98)Cited by:[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p4.1)\.
- Y\. Wu, M\. Daoudi, and A\. Amad \(2024\)Transformer\-Based Self\-Supervised Multimodal Representation Learning for Wearable Emotion Recognition\.IEEE Transactions on Affective Computing15\(1\),pp\. 157–172\.External Links:ISSN 1949\-3045,[Link](https://ieeexplore.ieee.org/document/10091193),[Document](https://dx.doi.org/10.1109/TAFFC.2023.3263907)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p1.1),[§3\.4](https://arxiv.org/html/2606.24985#S3.SS4.p1.1)\.
- H\. Yang and C\. Lee \(2019\)An Attribute\-invariant Variational Learning for Emotion Recognition Using Physiology\.InICASSP 2019 \- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1184–1188\.Note:ISSN: 2379\-190XExternal Links:[Link](https://ieeexplore.ieee.org/document/8683290),[Document](https://dx.doi.org/10.1109/ICASSP.2019.8683290)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p3.1)\.
- R\. Yu, Y\. Gong, X\. He, Y\. Zhu, Q\. Liu, W\. Ou, and B\. An \(2021\)Personalized Adaptive Meta Learning for Cold\-start User Preference Prediction\.Proceedings of the AAAI Conference on Artificial Intelligence35\(12\),pp\. 10772–10780\(en\)\.Note:Number: 12External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/17287),[Document](https://dx.doi.org/10.1609/aaai.v35i12.17287)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.24985#S2.SS1.p2.1)\.
- H\. Yuan, S\. Chan, A\. P\. Creagh, C\. Tong, A\. Acquah, D\. A\. Clifton, and A\. Doherty \(2024\)Self\-supervised learning for human activity recognition using 700,000 person\-days of wearable data\.npj Digital Medicine7\(1\),pp\. 91\(en\)\.Note:Publisher: Nature Publishing GroupExternal Links:ISSN 2398\-6352,[Link](https://www.nature.com/articles/s41746-024-01062-3),[Document](https://dx.doi.org/10.1038/s41746-024-01062-3)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- Z\. Zhang, R\. A\. Rossi, B\. Kveton, Y\. Shao, D\. Yang, H\. Zamani, F\. Dernoncourt, J\. Barrow, T\. Yu, S\. Kim, R\. Zhang, J\. Gu, T\. Derr, H\. Chen, J\. Wu, X\. Chen, Z\. Wang, S\. Mitra, N\. Lipka, N\. Ahmed, and Y\. Wang \(2025\)Personalization of Large Language Models: A Survey\.arXiv\.Note:arXiv:2411\.00027 \[cs\]External Links:[Link](http://arxiv.org/abs/2411.00027),[Document](https://dx.doi.org/10.48550/arXiv.2411.00027)Cited by:[§1](https://arxiv.org/html/2606.24985#S1.p1.1)\.
- Z\. Zhao, Y\. Wang, and Y\. Wang \(2022\)Multi\-level Fusion of Wav2vec 2\.0 and BERT for Multimodal Emotion Recognition\.InProc\. Interspeech 2022,pp\. 4725–4729\.External Links:[Link](https://www.isca-archive.org/interspeech_2022/zhao22k_interspeech.html),[Document](https://dx.doi.org/10.21437/Interspeech.2022-10230)Cited by:[§2\.2](https://arxiv.org/html/2606.24985#S2.SS2.p1.1)\.

\(a\)Baseline
\(b\)Personalised \(Moment\)
\(c\)Personalised \(HuBERT\)
\(d\)Personalised \(Chronos\)

Figure 6:Per\-user accuracy and macro F1\(mean±\\pmstd across seeds\)\. Cells highlighted ingreen\(resp\.red\) indicate improvements \(resp\. degradations\) exceeding the median absolute difference with respect to the baseline\.![Refer to caption](https://arxiv.org/html/2606.24985v1/x5.png)Figure 7:UMAP projections of raw foundation\-model embeddings on WESAD\.Top row: colored by stress label \(baseline,amusement,stress\)\.Bottom row: colored by participant\.