A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

# A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset
Source: [https://arxiv.org/html/2606.27886](https://arxiv.org/html/2606.27886)
11institutetext:University of Siegen, Germany
11email:engahmedmohamady23@gmail\.com
[https://ubicomp\.eti\.uni\-siegen\.de/](https://ubicomp.eti.uni-siegen.de/)
\*These authors contributed equally to this work\.###### Abstract

Recent advances in Human Activity Recognition \(HAR\) from wearable sensors have shown that multi\-modal deep learning models consistently outperform their uni\-modal counterparts\. Modalities can include IMUs, RGB cameras, audio signals, and others\. One important aspect of multi\-modal deep learning is the sensor fusion approach we apply\. Over recent years, multiple fusion paradigms have been proposed for multi\-modal HAR\. However, to the best of our knowledge, no head\-to\-head comparison of these paradigms exists on a common multi\-modal HAR benchmark dataset\. To address this research gap, we systematically compare seven state\-of\-the\-art sensor fusion methods on the recently released HARMES dataset, which comprises 61 hours of fully labeled IMU, audio, and ambient humidity data\. The chosen dataset focuses on 15 household and personal hygiene activities of daily living \(ADLs\)\. By applying the seven different fusion techniques to a state\-of\-the\-art multi\-modal model architecture, we show that Gated Multi\-modal Fusion achieves the highest macro F1\-score \(0\.82\), surpassing the concatenation\-based late fusion HARMES paper baseline of 0\.76 by \+6pp under leave\-one\-participant\-out evaluation\. All code used in our experiments is made publicly available on GitHub111https://github\.com/AhmedMohamady98/A\-Comparison\-of\-Fusion\-Techniques\-for\-Multi\-Modal\-Human\-Activity\-Recognition\-on\-the\-HARMES\-Dataset\.

## 1Introduction

Human Activity Recognition \(HAR\) from wearable and ambient sensors is a foundational task for healthcare monitoring, assistive living, smart\-home automation, and activity\-aware computing\[[6](https://arxiv.org/html/2606.27886#bib.bib56)\]\. The past decade has seen rapid progress driven by deep learning architectures applied to inertial measurement unit \(IMU\) data\[[41](https://arxiv.org/html/2606.27886#bib.bib333),[30](https://arxiv.org/html/2606.27886#bib.bib257)\]\. These models now routinely achieve high classification accuracy on scripted activity corpora, but they operate on a single sensory channel and remain fragile to real\-world confounds such as handedness, sensor placement variability, and activities with similar motion signatures\. A growing body of work shows that the limitations of unimodal HAR can be overcome by combining multiple sensing modalities\[[36](https://arxiv.org/html/2606.27886#bib.bib286),[3](https://arxiv.org/html/2606.27886#bib.bib25)\]\. For example, audio captures characteristic acoustic signatures of water\-based and mechanical activities \(e\.g\., dishwashing, vacuuming, brushing teeth\) that are difficult to disambiguate from IMU sensors alone\. Environmental sensors such as humidity provide contextual priors: high humidity near a sink is indicative of water\-related tasks\[[10](https://arxiv.org/html/2606.27886#bib.bib64),[7](https://arxiv.org/html/2606.27886#bib.bib62)\]\. The HARMES dataset\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]offers a public, multi\-participant, long\-duration benchmark with synchronised IMU, audio, and humidity streams recorded from activities of daily living \(ADLs\), enabling multi\-modal fusion research on ecologically valid data\. The literature on multi\-modal fusion is diverse\. Architectures range from simple late concatenation of modality embeddings to gated units that learn per\-sample modality weights\[[2](https://arxiv.org/html/2606.27886#bib.bib18)\], bottleneck transformers routing information through shared latent tokens\[[29](https://arxiv.org/html/2606.27886#bib.bib233)\], directional cross\-attention\[[38](https://arxiv.org/html/2606.27886#bib.bib319)\], low\-rank tensor factorization\[[23](https://arxiv.org/html/2606.27886#bib.bib206)\], and decision\-level weighted averaging\. While these approaches beat the respective baselines, each approach has been validated on different datasets with different encoder backbones and evaluation protocols, making direct comparison across many fusion methods impossible\. A researcher choosing a fusion strategy for a new multi\-modal HAR system currently has no direct empirical guidance\. In this work, we provide such a comparison\. Using HARMES as a shared testbed, we train and evaluateseven fusion strategiesunder identical conditions: the same three encoder backbones, the same 10\-second window size, the same training hyperparameters, and the same evaluation metrics\. We report results under 3\-fold group cross\-validation and 20\-fold LOPO evaluation, matching the HARMES benchmark protocol\. We further conduct a modality ablation and a subgroup analysis of left\-handed participants\.

Our contribution is threefold\.

1. 1\.We present a systematic head\-to\-head comparison of seven prominent multi\-modal fusion paradigms on a uniform HAR benchmark, providing an empirical ranking independent of backbone or protocol variation\.
2. 2\.We identify the practical drivers of fusion performance on the HARMES dataset by identifying the per\-class\-performance contribution of each modality\.
3. 3\.We show that multi\-modal fusion partially mitigates the handedness gap by relying on handedness\-invariant audio cues when IMU signals become unreliable\.

## 2Related Work

To position this work in the context of current research, we discuss existing HAR model architectures, fusion methods, and fusion method comparison studies\.

### 2\.1Encoder Architectures for Multi\-Modal Wearable HAR

Deep learning for IMU\-based activity recognition has progressed through several generations of architecture\. Ordóñez et al\.\[[30](https://arxiv.org/html/2606.27886#bib.bib257)\]introduced DeepConvLSTM, pairing convolutional feature extraction with LSTM temporal modelling into the baseline most later work is measured against, while Ha and Choi\[[17](https://arxiv.org/html/2606.27886#bib.bib136)\]apply CNNs that can generalise across sensor placements via partial weight sharing\. The trend has since shifted toward lightweight, deployable designs\. Zhou et al\.\[[46](https://arxiv.org/html/2606.27886#bib.bib389)\]proposed TinyHAR, which chains depthwise separable convolutions with transformer encoded blocks, LSTM, and temporal attention, while remaining small enough for execution on mobile devices\. Bian et al\.\[[4](https://arxiv.org/html/2606.27886#bib.bib44)\]published an even smaller, transformer\-free model with TinierHAR, which is built around bi\-directional GRUs and temporal aggregation instead, reducing parameter count and computational cost significantly while maintaining competitive accuracy\.

Audio is a powerful complementary channel for a multitude of activities\. Gong et al\.\[[16](https://arxiv.org/html/2606.27886#bib.bib121)\]adapted the Vision Transformer to audio by treating log\-mel spectrograms as patch sequences, producing the Audio Spectrogram Transformer \(AST\), which remains a strong and widely used audio classification backbone\. Mollyn et al\.\[[26](https://arxiv.org/html/2606.27886#bib.bib221)\]showed that IMU\-triggered, privacy\-preserving audio sampling recognizes 26 daily activities without continuous recording, and Lee et al\.\[[21](https://arxiv.org/html/2606.27886#bib.bib191)\]confirmed that audio is highly effective for detecting activities with minimal limb motion\. For general timeseries signals, Ekambaram et al\.\[[13](https://arxiv.org/html/2606.27886#bib.bib101)\]proposed TSMixer, an all\-MLP time\-series architecture\.

Several multi\-modal HAR systems combine these channels in different ways: E\.g\., AttnSense’s multi\-level attention\[[25](https://arxiv.org/html/2606.27886#bib.bib213)\], Cosmo’s quality\-guided contrastive fusion\[[31](https://arxiv.org/html/2606.27886#bib.bib260)\], and Mamba\-MHAR’s two\-branch selective state space models\[[20](https://arxiv.org/html/2606.27886#bib.bib195)\]\.

### 2\.2Multi\-Modal Fusion Strategies

Multi\-modal fusion can be placed along a spectrum defined by how early in the processing pipeline modalities are combined\[[3](https://arxiv.org/html/2606.27886#bib.bib25),[36](https://arxiv.org/html/2606.27886#bib.bib286)\]\. At one extreme, early fusion merges raw or low\-level features before any modality\-specific learning has taken place\. At the other extreme, decision\-level fusion combines the outputs of fully independent per\-modality classifiers\. Between these two there exists a multitude of intermediate approaches that operate on learned modality embeddings\. Ramachandram and Taylor\[[36](https://arxiv.org/html/2606.27886#bib.bib286)\]identified at least five distinct architectural families within this space, and the field has continued to diversify since\. Below, we list and discuss the most prominent fusion approaches in current literature\.

Early fusionconcatenates raw sensor readings into a single input vector before any model processing\. It is simple but assumes all modalities share the same temporal resolution and scale, which rarely holds for heterogeneous sensors such as IMUs, microphones, and environmental probes\. Yilmaz\. et al\.\[[44](https://arxiv.org/html/2606.27886#bib.bib374)\]show that audio and IMU data can be processed into images, which leads to beneficial results with their early fusion approach\. In practice, early fusion is often outperformed by methods that encode each modality separately\[[36](https://arxiv.org/html/2606.27886#bib.bib286),[3](https://arxiv.org/html/2606.27886#bib.bib25),[28](https://arxiv.org/html/2606.27886#bib.bib229)\]\.

Late embedding fusion, oftentimes called feature\-level or representation\-level fusion, concatenates the output embeddings of per\-modality encoders and feeds the combined vector to a shared classifier\. We refer to embedding\-level feature fusion as late embedding fusion to distinguish it from decision\-level late fusion\. It is often applied in multi\-modal HAR because it decouples encoder training from fusion: each encoder can be pre\-trained independently, and the fusion head stays lightweight\. Münzner et al\.\[[28](https://arxiv.org/html/2606.27886#bib.bib229)\]found embedding concatenation outperformed decision\-level fusion on PAMAP2, and it has since been used for HAR tasks, e\.g\., household activity recognition\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]and exercise counting\[[21](https://arxiv.org/html/2606.27886#bib.bib191)\]\.

Gated fusionaddresses the fact that not all modalities are equally informative for every sample: a static concatenation head cannot down\-weight a noisy input\. Arevalo et al\.\[[2](https://arxiv.org/html/2606.27886#bib.bib18)\]introduced the Gated Multi\-modal Unit \(GMU\), where a sigmoid gate computed from all modalities controls how much each contributes to the fused vector\. Because the gate is conditioned on the combined context, the model can learn to suppress modalities adaptively, when they are less informative, e\.g\., due to background noise\. As the weights for each modality can be inspected, GMU produces interpretable modality “importances”\.

Tensor fusionmodels cross\-modal interactions explicitly via the outer product of modality embeddings, capturing pairwise and higher\-order combinations in one operation\. The Tensor Fusion Network \(TFN\) of Zadeh et al\.\[[45](https://arxiv.org/html/2606.27886#bib.bib379)\]showed this improves over additive baselines on sentiment analysis, but the outer\-product tensor grows multiplicatively with the number and dimensionality of modalities, making it expensive\. Liu et al\.\[[23](https://arxiv.org/html/2606.27886#bib.bib206)\]addressed this with Low\-Rank multi\-modal Fusion \(LMF\), which decomposes the weight tensor into modality\-specific low\-rank factors combined by an element\-wise product\. LMF matches or beats TFN with far fewer parameters, showing the full\-rank tensor is largely redundant\.

Cross\-modal attentionbuilds on the Transformer\[[40](https://arxiv.org/html/2606.27886#bib.bib323)\], using one modality as the query and another as the value so the query can attend to the most relevant parts of the other modality\. Tsai et al\.\[[38](https://arxiv.org/html/2606.27886#bib.bib319)\]systematised this as the multi\-modal Transformer \(MulT\), applying it in both directions for every modality pair, which gives six cross\-attention streams per layer with three modalities\. MulT is commonly used for language\-vision\-acoustic fusion\. The same idea shows up in ViLBERT’s co\-attentional layers\[[24](https://arxiv.org/html/2606.27886#bib.bib212)\]and in RGB\-D detection\[[42](https://arxiv.org/html/2606.27886#bib.bib332)\]\. Within HAR, related attention\-based fusion ideas appear in AttnSense\[[25](https://arxiv.org/html/2606.27886#bib.bib213)\], which applies multi\-level attention over inertial sensors, implicitly weighting both on\-body placement channels and time steps

Bottleneck transformer fusionis a compute\-efficient alternative to full cross\-attention\. Instead of letting every token attend to every other, Nagrani et al\.\[[29](https://arxiv.org/html/2606.27886#bib.bib233)\]route all cross\-modal information through a small set of shared bottleneck tokens\. In the multi\-modal Bottleneck Transformer \(MBT\), each layer processes a modality’s tokens together with the bottleneck, but never mixes two modalities directly\. The per\-modality bottleneck copies are then averaged into a shared state\. On AudioSet and VGGSound, it beat late fusion and full cross\-attention with fewer parameters, and the bottleneck may also act as an implicit regularizer by constraining cross\-modal information flow to a limited set of shared tokens\. Bralina et al\.\[[5](https://arxiv.org/html/2606.27886#bib.bib55)\]extended it with the Adaptive Bottleneck Transformer \(AMBT\), adding factorised contrastive learning to prevent modality collapse\.

Decision\-level fusionnever builds a shared representation\. Each modality has its own classifier producing a class distribution, and these get combined by a fixed rule or a learned weighted sum\. Atrey et al\.\[[3](https://arxiv.org/html/2606.27886#bib.bib25)\]formalised this as one of the three canonical fusion levels\. The approach is modular and transparent, since the weights directly quantify per\-modality confidence, but it loses interactions at the representation level: the model has no way to discover that a humid environment combined with repetitive arm motion is more diagnostic than either signal on its own\.

CLS\-token fusiontreats each per\-modality embedding as a token in a shared sequence with a prepended, learnable\[CLS\]token, processed by standard transformer encoder layers\[[40](https://arxiv.org/html/2606.27886#bib.bib323)\]following the classification convention of BERT \(Devlin et al\.\[[11](https://arxiv.org/html/2606.27886#bib.bib97)\]\) and ViT \(Dosovitskiy et al\.\[[12](https://arxiv.org/html/2606.27886#bib.bib99)\]\)\. The\[CLS\]state aggregates all modality tokens through unrestricted self\-attention\. There is no bottleneck constraint as in MBT, and no directional structure as in MulT\. All tokens take part in one unified attention operation\.

Contrastive self\-supervised fusionis a term for self\-supervised fusion methods that align modalities using the natural pairing structure of the data rather than merging them into one representation\. Tian et al\.\[[37](https://arxiv.org/html/2606.27886#bib.bib316)\]introduced Contrastive Multiview Coding \(CMC\), pulling together views of the same scene and pushing apart different scenes, and Radford et al\.\[[34](https://arxiv.org/html/2606.27886#bib.bib283)\]scaled this to image\-text pairs with CLIP, enabling zero\-shot transfer via similarity retrieval\. Girdhar et al\.\[[15](https://arxiv.org/html/2606.27886#bib.bib119)\]extended it to six modalities, IMU included, with ImageBind, showing that pairing each modality with images alone suffices to align them all\. Closest to our setting, Moon et al\.\[[27](https://arxiv.org/html/2606.27886#bib.bib226)\]introduced IMU2CLIP, aligning wrist\-worn IMU with video and text, and Ouyang et al\.\[[31](https://arxiv.org/html/2606.27886#bib.bib260)\]applied contrastive fusion to wearable HAR with Cosmo\. These approaches suit wearable sensing well, where labels are expensive but paired unlabeled recordings are cheap to collect, but do not apply to our fully supervised modality fusion problem\.

Feature conditioninguses one modality to modulate the internal computation of a network processing another, instead of merging them at a fixed layer\. Perez et al\.\[[32](https://arxiv.org/html/2606.27886#bib.bib268)\]introduced Feature\-wise Linear Modulation \(FiLM\), where a conditioning network predicts per\-channel scale and shift parameters applied to a target network’s feature maps\. It was proposed for visual question answering and has since been used wherever one signal provides context for another\. Rahman et al\.\[[35](https://arxiv.org/html/2606.27886#bib.bib284)\]adapted this to pretrained transformers with the multi\-modal Adaptation Gate \(MAG\), injecting a learned shift into BERT’s representations from visual and acoustic signals while leaving the backbone intact\.

Latent bottleneck architecturesgeneralise the MBT bottleneck into a modality\-agnostic framework\. Jaegle et al\.\[[18](https://arxiv.org/html/2606.27886#bib.bib163)\]introduced the Perceiver, which maps inputs of arbitrary modality and size into a fixed\-size latent array via asymmetric cross\-attention, then processes that array with standard Transformer layers\. Decoupling compute cost from input size lets one architecture handle images, audio, point clouds, and video with minimal modality\-specific architectural changes\. Where MBT still feeds per\-modality token sequences in separately, the Perceiver compresses the entire input into a shared latent space from the first step\.

Feature recalibrationlets each modality adjust the internal features of the others while the per\-modality streams stay separate\. Joze et al\.\[[39](https://arxiv.org/html/2606.27886#bib.bib322)\]introduced the multi\-modal Transfer Module \(MMTM\), which uses squeeze\-and\-excitation to recalibrate the channel\-wise features of each convolutional stream from a joint summary of all modalities\. Because it slots between existing branches, each one can keep its pretrained weights\. Within HAR, Gao et al\.\[[14](https://arxiv.org/html/2606.27886#bib.bib114)\]proposed MMTSA, which encodes inertial signals as Gramian Angular Field images and fuses RGB and IMU streams via inter\-segment attention, reaching strong cross\-participant accuracy\.

### 2\.3Fusion Comparison Studies

The literature on multi\-modal fusion is large, but systematic head\-to\-head comparisons of multiple fusion strategies under controlled conditions are surprisingly rare\. Most papers introduce a single new method and compare it against a concatenation baseline, making it difficult to draw conclusions about the relative merit of different architectural families\.

At the survey level, Atrey et al\.\[[3](https://arxiv.org/html/2606.27886#bib.bib25)\]provide a foundational taxonomy of fusion approaches, distinguishing feature\-level, decision\-level, and hybrid fusion and categorising methods by their combination rule\. Ramachandram and Taylor\[[36](https://arxiv.org/html/2606.27886#bib.bib286)\]later surveyed the deep multi\-modal learning landscape and identified five architectural families, but their treatment is conceptual rather than empirical\. In the HAR domain specifically, Qiu et al\.\[[33](https://arxiv.org/html/2606.27886#bib.bib281)\]reviewed over 300 multi\-sensor papers and mapped out dominant design patterns, and Yadav et al\.\[[43](https://arxiv.org/html/2606.27886#bib.bib367)\]concluded that multi\-modal fusion consistently outperforms unimodal approaches\. Neither study, however, evaluates two or more fusion methods under the same experimental conditions\.

One of the few HAR papers to directly compare fusion strategies is Münzner et al\.\[[28](https://arxiv.org/html/2606.27886#bib.bib229)\], who tested sensor\-level CNN fusion against decision\-level fusion on the PAMAP2 dataset and reported higher performance for feature\-level fusion than decision\-level fusion\. Aguileta et al\.\[[1](https://arxiv.org/html/2606.27886#bib.bib8)\]similarly surveyed multi\-sensor fusion for activity recognition and noted that no consensus had emerged on the best strategy, with results varying substantially across datasets and sensor combinations\.

Outside HAR, more controlled comparisons do exist but remain tied to their own domains and encoder choices\. Tsai et al\.\[[38](https://arxiv.org/html/2606.27886#bib.bib319)\]evaluated MulT against late fusion and tensor fusion on language\-vision\-acoustic sentiment datasets\. Nagrani et al\.\[[29](https://arxiv.org/html/2606.27886#bib.bib233)\]compared MBT against late fusion and full cross\-attention on video\-audio classification, finding that the bottleneck constraint does not hurt performance while reducing computational cost\. Koutoupis et al\.\[[19](https://arxiv.org/html/2606.27886#bib.bib181)\]evaluated contrastive fusion objectives for higher\-order multi\-modal alignment across several non\-HAR settings\. These studies are valuable but their findings do not transfer directly to wearable sensor streams, where modality characteristics, sampling rates, and temporal structure differ fundamentally from video, language, and audio\-visual pairs\.

To the best of our knowledge, we found no prior study that compares such a broad range of fusion paradigms under identical encoder backbones on a public multi\-modal HAR benchmark\.

## 3Methodology

### 3\.1Dataset

We use the HARMES dataset\[[9](https://arxiv.org/html/2606.27886#bib.bib59),[8](https://arxiv.org/html/2606.27886#bib.bib60)\], a public multi\-modal corpus for wearable HAR\. It contains recordings from 20 participants \(10 female, 10 male, ages 18–67, mean 37\.8, SD 14\.4\) performing 15 activities of daily living in their own homes\. Each participant contributed about three hours of fully labeled data across three recording sessions\. We exclude the fourth, free\-form, session, leaving 61 hours of labelled data in total\.

#### 3\.1\.1Modalities

Three synchronized streams are provided\. TheIMUstream includes dual\-wrist accelerometer and gyroscope data at 50 Hz, from a Puck\.js device on the left wrist and a smartwatch on the right, totaling 12 channels \(3\-axis acceleration and 3\-axis angular velocity per wrist\)\. Theaudiostream is an ambient recording at 44\.1 kHz from a wrist\-worn microphone \(in the smartwatch\), with no speech contained in the dataset for privacy reasons\. Theatmosphericstream is humidity from a wrist\-mounted BME280 sensor at 1 Hz, up\-sampled to the IMU grid as a 1D time series\.

#### 3\.1\.2Activity classes

The 15 labelled activities fall into three groups: self\-care \(washing hands, brushing teeth, applying hand cream, disinfecting hands\), household \(floor cleaning, window cleaning, vacuum cleaning, washing dishes, putting away dishes, cleaning table, cleaning out dishwasher\), and feeding \(making tea, cutting vegetables, drinking, watering plants\)\. The Null class covers background and transition periods between activities, leading to a total of 16 classes\. At 10 s granularity, the Null class is the most common label \(20\.5% of all windows\)\.

#### 3\.1\.3Left\-handed participants

Three participants \(P07, P10, P14\) are left\-handed, a 15% prevalence in line with the real\-world estimate of 9–18%\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]\. This leads to a generalization challenge: IMU signals depend on which hand performs the dominant motion\. While better IMU sensor placement generalization can likely be achieved with data augmentation, audio is expected to be mostly handedness\-invariant\.

### 3\.2Encoder Architectures

We process each modality with its own encoder that maps the raw stream to a 128\-dimensional embedding\. We fix this embedding size across all three encoders so that every fusion method operates on a common representation, which lets us swap fusion strategies without changing the encoders\. The IMU and humidity encoders are trained from scratch, while the audio encoder uses a frozen pretrained backbone with only a small trainable head\.

#### 3\.2\.1IMU encoder

For the dual\-wrist inertial stream, we use TinyHAR\[[46](https://arxiv.org/html/2606.27886#bib.bib389)\]\. It first applies lightweight convolutions across the sensor channels to extract local motion patterns, then a temporal self\-attention block to capture how those patterns relate over the length of a window\. The classic baseline here is DeepConvLSTM\[[30](https://arxiv.org/html/2606.27886#bib.bib257)\], which pairs convolutions with a recurrent layer, but its LSTM processes time steps sequentially and is comparatively heavy to train\. TinyHAR reaches similar or better accuracy on standard HAR benchmarks at a fraction of the parameters, and its attention block handles long\-range temporal structure without the sequential bottleneck of a recurrent network\. Recent state\-space models such as HARMamba\[[22](https://arxiv.org/html/2606.27886#bib.bib202)\]are promising but less established, so we favour TinyHAR as a proven, compact backbone well suited to wrist\-worn data\.

#### 3\.2\.2Audio encoder

For the ambient audio stream, we use the Audio Spectrogram Transformer \(AST\)\[[16](https://arxiv.org/html/2606.27886#bib.bib121)\], which treats a spectrogram as a sequence of patches and applies transformer attention over them\. We keep its backbone frozen and train only a small projection head\. The reason is transfer learning: AST is pretrained on AudioSet, a very large and diverse sound corpus, so it already carries a strong general audio representation\. Our labelled HAR audio is far too small to train a comparable model from scratch, and fine\-tuning the whole backbone on it would risk overfitting\. Convolutional audio models such as VGGish or PANNs were possible alternatives, but AST’s attention gives a stronger pretrained starting point and a clean way to freeze the backbone and adapt with a single linear layer\.

#### 3\.2\.3Humidity encoder

For the atmospheric stream, we use TSMixer\[[13](https://arxiv.org/html/2606.27886#bib.bib101)\], an all\-MLP architecture that alternates between mixing information along the time axis and along the feature axis\. Humidity is a slow, single\-channel signal, so it does not need the heavy machinery of attention or recurrence to be modelled well\. A transformer would be overkill and prone to overfitting on such a simple stream, while an LSTM adds sequential cost for little benefit\. TSMixer stays lightweight and trains quickly, which keeps the humidity branch cheap relative to the IMU and audio encoders, an appropriate balance given that humidity likely carries the least activity\-discriminative information of the three\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]\.

### 3\.3Fusion Architectures

We compare seven fusion methods drawn from the strategies surveyed in Section[2\.2](https://arxiv.org/html/2606.27886#S2.SS2)\. Our aim is not to propose a new fusion method but to compare existing ones under identical conditions, so the selection follows a single principle: every method must operate on the fixed 128\-dimensional embeddings produced by our encoders, treat the three modalities symmetrically, and require no modality\-specific pretraining or input format\. This allows us to use the same encoders for every fusion method and vary only the fusion step, so any difference in performance is attributable to the fusion mechanism alone\.

This criterion rules out several families discussed in Section[2\.2](https://arxiv.org/html/2606.27886#S2.SS2)\. Contrastive and joint\-embedding methods such as CMC, CLIP, ImageBind, and IMU2CLIP need large\-scale paired data and a separate pretraining stage, which does not fit a controlled supervised comparison on a single dataset\. Methods tied to a particular input structure, such as the Perceiver’s raw\-input bottleneck or MMTM and MMTSA’s convolutional streams, would bind the comparison to a specific encoder format\. Conditioning methods such as FiLM and MAG assume an asymmetric relationship in which one modality modulates another, which does not match three co\-equal sensor streams\.

The seven retained methods cover the remaining intermediate and decision\-level paradigms that do fit this setting \(cf\. Section[2\.2](https://arxiv.org/html/2606.27886#S2.SS2)for the exact descriptions\):

- •Late Fusion: embedding concatenation, followed by a simple MLP classification head\.
- •Gated Multi\-modal Fusion \(GMF\): learned per\-modality gating\.
- •Low\-Rank multi\-modal Fusion \(LMF\): tensor interaction with a low\-rank factorisation\.
- •Cross\-Modal Attention \(CMA\): directional attention between every modality pair\.
- •Multi\-modal Bottleneck Transformer \(MBT\): cross\-modal exchange through shared bottleneck tokens\.
- •CLS\-Token Transformer: a learnable token aggregating all modalities under unrestricted self\-attention\.
- •Decision Fusion: a learned weighted sum of per\-modality class distribution predictions\.

![Refer to caption](https://arxiv.org/html/2606.27886v1/x1.png)Figure 1:Overview of the multi\-modal pipeline\. Each modality is encoded into a 128\-dimensional embedding by a dedicated encoder\. The three embeddings are then combined by one of seven interchangeable fusion methods to predict the activity class\. The fusion block is held generic here\. The seven concrete architectures are detailed in Section[3\.3](https://arxiv.org/html/2606.27886#S3.SS3)and Appendix[0\.A](https://arxiv.org/html/2606.27886#Pt0.A1)\.Together, these span the breadth of the fusion taxonomy, from the simplest concatenation to tensor, gated, attention\-based, and bottleneck designs\. Six operate at the intermediate \(embedding\) level and one, Decision Fusion, at the decision level, so the set also covers two of the three canonical fusion stages\. Early \(raw\-signal\) fusion is excluded by construction, since all methods operate on encoder embeddings\. The general structure of our multi\-modal classification pipeline is displayed in Fig\.[1](https://arxiv.org/html/2606.27886#S3.F1)\. Full architectural details for each method are given in the respective original publications and in graph\-based visual representations in Appendix[0\.A](https://arxiv.org/html/2606.27886#Pt0.A1)\.

### 3\.4Training and Evaluation Protocol

#### 3\.4\.1Windowing

We segment the continuous recordings into 10\-second windows \(leading to 500 samples per window at 50 Hz\)\. For 3\-fold cross\-validation, we use a step of 250 samples \(50% overlap\), which gives roughly 43,760 windows\. For LOPO we, use non\-overlapping windows \(step = 500, matching the HARMES paper\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]\), yielding 21,897 windows\. Each window takes the activity label that covers most of its span \(simple majority voting\)\. Because the sliding window runs across the entire recording, including background and transition periods, windows without majority activity labels are assigned to the “Null” class\.

#### 3\.4\.2Cross\-validation splits

We use two cross\-validation protocols: 3\-fold group cross\-validation for all fusion methods and 20\-fold Leave\-One\-Participant\-Out for the best method from the 3\-fold CV evaluation, for comparison with the HARMES baseline\. In3\-fold group cross\-validation, participants are split into three disjoint folds \(see Table[1](https://arxiv.org/html/2606.27886#S3.T1)\)\. Within each fold, the smallest\-ID participant outside the test set is held out for validation, used for early stopping and checkpoint selection, and the rest form the training pool\. Train, validation, and test sets are disjoint at the participant level in every fold\. In20\-fold Leave\-One\-Participant\-Out \(LOPO\), each participant is held out once as the test set\. For each test participantPkP\_\{k\}, the smallest\-ID participant among the remaining 19 is used for validation, and the other 18 are used for training\. The LOPO paradigm matches the evaluation in the original HARMES paper\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\], allowing a direct comparison\. We use three folds rather than a larger fold count for the main comparison for experiment runtime reasons\. As every fusion method is trained and evaluated on each fold, the total training cost scales with the number of methods times the number of folds\. Three folds still leave roughly seven held\-out participants per fold, enough for a stable estimate, while keeping the seven\-way comparison tractable\. We reserve the more expensive LOPO protocol for the single best\-performing method, where a direct comparison with the original HARMES paper is most informative\.

Table 1:Participant assignment for the three cross\-validation folds\.
#### 3\.4\.3Training configuration

We apply an almost identical training configuration to all model configurations\. We use Adam as the optimizer with cosine annealing \(no restarts\), a maximum of 50 epochs, and a batch size of 32\. The learning rate is10−310^\{\-3\}for every method except LMF, which uses5×10−35\\times 10^\{\-3\}to compensate for vanishing gradients through its three\-way tensor product\. Early stopping \(patience 10 epochs\) monitors validation macro F1, and the best checkpoint is loaded for a single final evaluation on the held\-out test set, which is never used for model selection\. TinyHAR and TSMixer are trained from scratch, while the AST backbone, which is available pre\-trained, stays frozen\. We train all methods using cross\-entropy loss except Decision Fusion, for which we use negative log\-likelihood on log\-probabilities\.

#### 3\.4\.4Evaluation

For our comparison, we compute metric scores on ground truth labels and predictions by each model\. The primary metric is macro\-averaged F1, which weights all 16 classes equally regardless of their frequency\. This is relevant, as the class distribution in the original dataset is imbalanced\. We also report accuracy for comparability with prior work, along with per\-class and per\-participant results, and confusion matrices for diagnostic analysis of confounded classes/samples\.

## 4Results

We compare the seven fusion methods against the unimodal baselines and the HARMES reference using the protocols of Section[3](https://arxiv.org/html/2606.27886#S3)\. Our analysis, and thus, the results are organised around two questions: how much does combining modalities help, and which fusion strategy makes the best use of them\. Table[2](https://arxiv.org/html/2606.27886#S4.T2)summarises the aggregated results for each fusion strategy, and the subsections that follow break them down by class, by participant, and by handedness\.

Fusion strategies improve over the strongest single modality across the board\. The best unimodal model is AST on audio at 0\.734 macro F1, and the best fusion method, GMF, reaches 0\.827, an improvement of 9\.3 points\. Audio is by far the most informative single stream, while humidity on its own is close to the chance level at 0\.09 macro F1, which is consistent with the modality ablation reported in the HARMES paper\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]\. The 3\-fold and leave\-one\-participant\-out protocols agree closely for GMF \(0\.827 and 0\.819\), so the comparison is stable across evaluation schemes, and GMF under LOPO also exceeds the HARMES multi\-modal baseline of 0\.760 by 5\.9 points\.

Table 2:Results across modality combination, fusion strategies, and cross\-validation modes\. F1 is the macro F1 score, averaged over folds \(3\-fold CV\) or participants \(LOPO\), Acc\. is the prediction accuracy, averaged in the same way\. All fusion methods use the three modalities jointly \(Modality = All\)\.Boldfont marks the best result within each section\. The HARMES baseline is the multimodal leave\-one\-participant\-out macro F1 reported by Burchard et al\.\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\]### 4\.1Fusion Method Comparison

Figure[2](https://arxiv.org/html/2606.27886#S4.F2)ranks the seven fusion methods on 3\-fold cross\-validation\. Every method clears the best unimodal baseline \(AST, 0\.734 macro F1\), confirming that combining modalities helps regardless of the tested fusion strategy\. The margin, however, varies\. GMF leads at 0\.827, with Late Fusion close behind at 0\.817\. The two simplest mechanisms, gated summation and plain concatenation, are also the two strongest\. The attention\- and transformer\-based methods form a middle cluster \(CMA 0\.795, CLS\-Token Transformer 0\.793, MBT 0\.788\) that lands several points below GMF\. LMF and Decision Fusion trail at 0\.747, beating the audio\-based unimodal baseline only by a smaller margin\.

![Refer to caption](https://arxiv.org/html/2606.27886v1/x2.png)Figure 2:Fusion method comparison on 3\-fold group cross\-validation \(macro F1\)\. All methods use the three modalities jointly\. The dashed line marks the best unimodal baseline \(AST\)\.
### 4\.2Class performance analysis

![Refer to caption](https://arxiv.org/html/2606.27886v1/x3.png)Figure 3:Per\-class confusion\-matrix difference, GMF minus AST \(best unimodal\)\. Note the interpretation: Red, positive values on the diagonal mark classes recognised more reliably under fusion\. Blue, negative off\-diagonal entries mark confusions that fusion removes\. Blue, negative values on the diagonal, or off\-diagonal red values mean that the GMF model performed worse than the unimodal AST for the specific confusion matrix entry\.To analyze on which classes the performance is improved by multimodel models, Figure[3](https://arxiv.org/html/2606.27886#S4.F3)shows the per\-class confusion\-matrix difference between the best multi\-modal model \(GMF\) and the best unimodal one \(AST\)\. Each diagonal entry shows how much more often GMF classifies that activity correctly, and the blue, negative off\-diagonal entries are the confusions GMF removes in comparison\. The improvement is concentrated in activities that are quiet or acoustically ambiguous but involve distinctive hand motion\. Notably,Putting away dishesis recognized correctly \+25pp more often, disinfecting hands \+24pp more, and drinking \+19pp more\. These are exactly the classes where audio alone is weak, as there is little characteristic sound\. The overall confounding pair ofApply hand creamandDisinfecting handsis handled better by GMF as well, although notably, while the confusion ofDisinfecting handswithApply hand creamis reduced by \-24pp, vice\-versa, it is increased by \+8pp\.

We additionally report per\-modality absolute confusion matrices, together with those of GMF and the GMF\-minus\-TinyHAR \(multimodal\-minus\-IMU\-only\) difference, in Appendix[0\.B\.1](https://arxiv.org/html/2606.27886#Pt0.A2.SS1)\.

![Refer to caption](https://arxiv.org/html/2606.27886v1/x4.png)Figure 4:Per\-class macro F1 scores for each fusion strategy\. Results presented are averaged over the three\-fold CV\. Models are sorted in descending order by their global performance, from left to right\.In Fig\.[4](https://arxiv.org/html/2606.27886#S4.F4), we display the per\-class F1\-scores of all fusion strategies in a bar chart\. For each class on the x\-axis, the fusion strategies are ordered by their previously calculated macro performance\. With minor exceptions, the global ordering of fusion strategy performances also applies to the per\-class performance, where GMF and Late Fusion by concatenation perform best\. Disinfecting hands is consistently the most challenging class for all fusion strategies\.

### 4\.3Generalisation to unseen participants

#### 4\.3\.1Leave\-one\-participant\-out

The 3\-fold protocol mixes participants across folds\. To approximate the performance on a single, unseen participant and to be comparable to the HARMES baseline\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\], we re\-evaluate the best model, GMF, under the stricter leave\-one\-participant\-out \(LOPO\) scheme\. GMF reaches 0\.819 macro F1 under LOPO, within one point of its 3\-fold score of 0\.827\. The drop is negligible despite LOPO being the more demanding test\. GMF under LOPO also exceeds the multi\-modal HARMES baseline of 0\.760 by 5\.9 points, on the exact same evaluation protocol\. To the best of our knowledge, our multi\-modal classification model with GMF is therefore also the currently highest performing model on the HARMES dataset\.

The per\-class picture reinforces this\. We display the GMF LOPO confusion matrix in Fig\.[5](https://arxiv.org/html/2606.27886#S4.F5)\. When comparing the LOPO confusion matrix with the 3\-fold matrix in Appendix[0\.B\.1](https://arxiv.org/html/2606.27886#Pt0.A2.SS1)\(Figure[0\.B\.1](https://arxiv.org/html/2606.27886#Pt0.A2.SS1)\), LOPO matches or slightly exceeds the 3\-fold result on most activities, gaining a little on vacuum cleaning, brushing teeth, and window cleaning\. The model is almost a 100 % accurate on the activities with a clear acoustic and motion signature: vacuum cleaning \(0\.97\), cutting vegetables \(0\.94\), making tea and brushing teeth \(0\.91\), and washing dishes and window cleaning \(0\.90\)\. Where LOPO falls slightly behind, the gap is confined to the low\-support self\-care classes, disinfecting hands and applying hand cream, which have the fewest windows and the least distinctive signatures and are therefore the most sensitive to an unseen participant’s personalized motion\. These differences are small, and the bulk of the activities are recognized cleanly, confirming that the multi\-modal model generalizes well to new users rather than overfitting to the training participants\.

![Refer to caption](https://arxiv.org/html/2606.27886v1/x5.png)Figure 5:Pooled confusion matrix for GMF under leave\-one\-participant\-out evaluation, aggregated over all 20 held\-out participants\.
#### 4\.3\.2Per\-participant results and left\-handed participants

The results are mostly stable across participants, but deviations exist\. Fig\.[6](https://arxiv.org/html/2606.27886#S4.F6)shows a heatmap of the macro F1 scores per model and per participant\. The largest variations exist in the unimodal IMU\-model TinyHAR, with left\-handed participants scoring significantly lower\. No model strictly beats all other models over all participants\. Rather, the results are close, and GMF performs best for most participants, closely followed by Late fusion by concatenation\.

![Refer to caption](https://arxiv.org/html/2606.27886v1/x6.png)Figure 6:Heatmap table showing per\-participant macro F1 scores for each participant and model \(3\-fold CV\)\. Left\-handed participants are marked in red font\. The three leftmost models are unimodal: TinyHAR \(IMU\), AST \(Audio\), TSMixer \(humidity\)\. The worst results on each left\-handed participant are marked in red, and the best results on them are marked in green\.![Refer to caption](https://arxiv.org/html/2606.27886v1/x7.png)Figure 7:3\-Fold CV macro F1 performance per model, split by dominant hand into left\-handers and right\-handers\. The plot shows both unimodal \(AST: audio, TSMixer: humidity, TinyHAR: IMU\) and multi\-modal methods \(all others\), sorted by performance gap between groups of left\-handers and right\-handers, descending from left to right\.Three participants \(P07, P10, P14\) are left\-handed, which negatively affects the IMU modality because the dominant\-hand motion appears on the opposite wrist from the right\-handed majority\. Audio and humidity are largely unaffected by handedness, so any modality\-induced robustness gap likely comes from the IMU\. We show the gap between the groups of left\-handers and right\-handers in Fig\.[7](https://arxiv.org/html/2606.27886#S4.F7), for each fusion strategy and for the unimodal models\. The IMU\-based unimodal TinyHAR by far shows the largest gap of 0\.18 \(0\.54 vs 0\.72\)\.

Humidity is excluded from this comparison because it carries almost no discriminative signal for any participant\.

Audio behaves in the opposite way\. AST is essentially handedness\-invariant and in fact performs marginally better on the left\-handers \(0\.74 versus 0\.73\), since a microphone picks up the same activity regardless of which hand performs it\. Fusion inherits this robustness and recovers what the IMU alone loses\. For every left\-hander, the most informative model is a fusion method\.

## 5Discussion

##### Simple fusion outperforms complex fusion

From the performance results, we infer that two of the simplest fusion methods score highest\. GMF \(0\.827\) and Late Fusion \(0\.817\) outrank the attention\- and transformer\-based methods \(CMA, CLS Transformer, MBT, all near 0\.79\), and the tensor and decision\-level methods \(LMF, Decision Fusion\) trail at 0\.75\. This runs against the intuition that richer cross\-modal interaction should help\. We attribute it to the setting rather than to a flaw in those methods\. With only three modalities, one of which \(humidity\) is of limited use in 10s windows, and with a frozen, already\-strong audio encoder, there is little structure left for elaborate attention to exploit, and the extra parameters are harder to fit on a dataset of 20 participants, despite the comparatively large size of 61 labeled hours\. Especially for HAR, where datasets are often not that large, it seems that these simpler methods already deliver the best performance\.

##### Fusion improves performance by resolving specific confusions, not by a uniform lift

The per\-class analysis \(Section[4\.2](https://arxiv.org/html/2606.27886#S4.SS2)\) shows the performance gain is concentrated in activities that are ambiguous in a unimodal IMU or audio setting, but have an additional distinctive acoustic or motion pattern\. This applies, e\.g\., to putting away dishes, disinfecting hands, and drinking\. These improve because fusion untangles confusion pairs the audio or IMU model could not separate on its own, for example, putting away dishes versus cleaning out the dishwasher, or disinfecting hands versus applying hand cream\. Activities with a loud, unmistakable acoustic signature \(vacuum cleaning, cutting vegetables, making tea\) are already solved by the acoustic model alone, and their performance thus does not benefit from other modalities\. The gain of the fusion models over the best unimodal model \(AST\) is moderate in total, and fusion adds value precisely on the subset of activities where a microphone is blind\. Overall, the results of this work strengthen pre\-existing findings that showed the benefits of multi\-modal HAR\.

##### Humidity contribution is minimal

In our 10 s\-window model setting, across the fusion methods, removing the humidity stream changes macro F1 by a fraction of a percentage point, and TSMixer \(0\.088\) performs near the chance level\. Only watering plants and cleaning out the dishwasher, where moisture genuinely changes, benefit at the class level\. The practical implication is that a two\-sensor IMU\-plus\-audio system captures essentially all of the achievable performance on this dataset for 10 s windows, and the humidity sensor can be dropped without cost\. We report this as a negative result worth recording rather than a limitation of the fusion methods\. As acknowledged by Burchard et al\. in the dataset publication\[[8](https://arxiv.org/html/2606.27886#bib.bib60)\], the humidity sensor reacts to the environment changes slowly and should thus be treated differently from the accurate high\-frequency measurements of IMU and microphone\.

##### The models generalise to unseen participants

Under the stricter leave\-one\-participant\-out protocol, where every test participant is entirely unseen and evaluated alone, GMF reaches 0\.819, within one point of its 3\-fold score and 5\.9 points above the already multi\-modal HARMES baseline of 0\.760 on the same protocol\. The per\-class behaviour is stable between the two protocols, with the small residual drops confined to the low\-support classesApply hand creamanddisinfecing hands\. This indicates the multi\-modal model learns activity signatures that transfer across individuals rather than fitting the particular participants it was trained on\. While the literature sees LOPO\-validation as the gold standard, the small performance difference between 3\-fold CV and LOPO for GMF hints at the validity of the tradeoff between cross\-validation meaningfulness and training time/energy consumption we chose\.

##### Fusion improves robustness to handedness

The IMU\-only unimodal model \(TinyHAR\) degrades sharply for the three left\-handed participants, since the dominant\-hand motion appears on the opposite wrist, giving it the largest right\-hand\-versus\-left\-hand gap of any model \(0\.72 versus 0\.54\)\. Audio is handedness\-invariant and is unaffected\. Because fusion combines the two, the multimodal models inherit the audio model’s robustness and recover most of what the IMU loses: GMF narrows the gap to 0\.027 and is the best model for the left\-handers in nearly every case\. Beyond raw accuracy, this is a fairness argument for multi\-modal sensing: a single\-modality IMU system would systematically underperform for a minority of users, and adding audio removes most of that disparity\. Likely, this issue can also be overcome by unimodal IMU models to a certain degree, e\.g\., by applying sophisticated data augmentation techniques\.

##### Limitations

Several caveats bound these conclusions\. The dataset has twenty participants and only three left\-handers, so the handedness finding, while consistent, rests on a small sample\. All results come from a single dataset of activities of daily living, and the audio backbone is frozen, so the ceiling we observe partly reflects a fixed pretrained representation rather than end\-to\-end optimization\. Due to the design of the encoder networks, the conclusions we draw are specific to this architecture and should be tested against other encoder architectures and datasets in the future\.

The ordering of fusion methods may also shift with more modalities or larger data, where the more expressive architectures could have room to pay off\. Finally, performance on the rare self\-care classes remains modest under cross\-participant evaluation, which is the most likely target for future improvement\. Our selection of fusion methods is also limited to methods fusing at a late intermediary stage in the model, after passing the data through completely separate encoder heads\. Early fusion approaches are excluded from this particular work, but should also be considered by practitioners when developing fusion models\.

## 6Conclusion

We compared seven fusion strategies for multi\-modal activity recognition on the HARMES dataset under a leakage\-free, participant\-grouped protocol\. Every method improved on the best unimodal model, and the simplest mechanisms were the strongest: gated multimodal fusion \(GMF\) led at 0\.827 macro F1, with late concatenation fusion close behind, while the heavier attention\- and tensor\-based methods trailed\. The multi\-modal gains came from resolving confusions between activities that sound alike but display different movement patterns and vice\-versa, with audio and IMU proving complementary\. The GMF\-based model generalized well to unseen participants \(F1 of 0\.819 under leave\-one\-participant\-out, 5\.9 points above the dataset baseline\) and was far more robust to handedness than the IMU\-only model, suggesting that a lightweight gated fusion of IMU and audio is both the most accurate and the most practical choice on the HARMES dataset\.

When considering future applications and enhancements, a natural next step is wearable deployment: quantising and pruning the IMU\-plus\-audio model for on\-device use, with streaming inference for continuous rather than window\-level recognition\. On the other hand, while our study already compares seven fusion strategies, more methods, especially from the area of early fusion, could be explored\. Our analysis should also be extended to other multi\-modal datasets, to strengthen and compare these findings and rule out the influence of a single dataset’s specific properties\. For sensing, a longer context or multi\-rate fusion would better match humidity’s slow dynamics, and adding further sensors or more data would test whether the preference for simple fusion persists as the modality count grows or the dataset size increases\.\{credits\}

#### 6\.0\.1Acknowledgements

The authors would like to thank the participants of the HARMES dataset and all researchers involved in creating it\.

#### 6\.0\.2\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

#### 6\.0\.3Contribution Statement\.

AM and RB contributed equally to this work\. AM co\-designed the experiments, executed the experiments, collected all results, and co\-wrote the manuscript\. RB co\-designed the experiments and co\-wrote the manuscript\. All authors reviewed the manuscript\.

## References

- \[1\]A\. A\. Aguileta, R\. F\. Brena, O\. Mayora, E\. Molino\-Minero\-Re, and L\. A\. Trejo\(2019\-01\)Multi\-Sensor Fusion for Activity Recognition—A Survey\.Sensors19\(17\),pp\. 3808\.External Links:ISSN 1424\-8220,[Document](https://dx.doi.org/10.3390/s19173808)Cited by:[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p3.1)\.
- \[2\]J\. Arevalo, T\. Solorio, M\. Montes\-y\-Gómez, and F\. A\. González\(2020\-07\)Gated multimodal networks\.Neural Computing and Applications32\(14\),pp\. 10209–10228\.External Links:ISSN 0941\-0643, 1433\-3058,[Document](https://dx.doi.org/10.1007/s00521-019-04559-1)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p4.1)\.
- \[3\]P\. K\. Atrey, M\. A\. Hossain, A\. El Saddik, and M\. S\. Kankanhalli\(2010\-11\)Multimodal fusion for multimedia analysis: a survey\.Multimedia Systems16\(6\),pp\. 345–379\.External Links:ISSN 1432\-1882,[Document](https://dx.doi.org/10.1007/s00530-010-0182-0)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p2.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p8.1),[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p2.1)\.
- \[4\]S\. Bian, M\. Liu, V\. F\. Rey, D\. Geißler, and P\. Lukowicz\(2025\-10\)TinierHAR: Towards Ultra\-Lightweight Deep Learning Models for Efficient Human Activity Recognition on Edge Devices\.InProceedings of the 2025 ACM International Symposium on Wearable Computers,Espoo Finland,pp\. 163–169\.External Links:[Document](https://dx.doi.org/10.1145/3715071.3750410),ISBN 979\-8\-4007\-1481\-8Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p1.1)\.
- \[5\]S\. Bralina, A\. Yazici, C\. Guan, and M\. Lee\(2026\-05\)Adaptive bottleneck transformer for multimodal EEG, audio, and vision fusion\.Expert Systems with Applications312,pp\. 131487\.External Links:ISSN 09574174,[Document](https://dx.doi.org/10.1016/j.eswa.2026.131487)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p7.1)\.
- \[6\]A\. Bulling, U\. Blanke, and B\. Schiele\(2014\-01\)A tutorial on human activity recognition using body\-worn inertial sensors\.ACM Comput\. Surv\.46\(3\),pp\. 33:1–33:33\.External Links:ISSN 0360\-0300,[Document](https://dx.doi.org/10.1145/2499621)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1)\.
- \[7\]R\. Burchard, H\. Ali, and K\. Van Laerhoven\(2026\)Improved Strategies for Multi\-modal Atmospheric Sensing to Augment Wearable IMU\-Based Hand Washing Detection\.InSensor\-Based Activity Recognition and Artificial Intelligence,Ö\. Durmaz Incel, J\. Qin, G\. Bieber, and A\. Kuijper \(Eds\.\),Vol\.16292,pp\. 308–323\.External Links:[Document](https://dx.doi.org/10.1007/978-3-032-13312-0%5F18),ISBN 978\-3\-032\-13311\-3 978\-3\-032\-13312\-0Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1)\.
- \[8\]R\. Burchard, P\. Brückner, M\. Bock, J\. Gall, and K\. V\. Laerhoven\(2026\-05\)HARMES: A Multi\-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound\.arXiv\.External Links:2605\.02596,[Document](https://dx.doi.org/10.48550/arXiv.2605.02596)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p3.1),[§3\.1\.3](https://arxiv.org/html/2606.27886#S3.SS1.SSS3.p1.1),[§3\.1](https://arxiv.org/html/2606.27886#S3.SS1.p1.1),[§3\.2\.3](https://arxiv.org/html/2606.27886#S3.SS2.SSS3.p1.1),[§3\.4\.1](https://arxiv.org/html/2606.27886#S3.SS4.SSS1.p1.1),[§3\.4\.2](https://arxiv.org/html/2606.27886#S3.SS4.SSS2.p1.1),[§4\.3\.1](https://arxiv.org/html/2606.27886#S4.SS3.SSS1.p1.1),[Table 2](https://arxiv.org/html/2606.27886#S4.T2),[Table 2](https://arxiv.org/html/2606.27886#S4.T2.3.16.16.1),[§4](https://arxiv.org/html/2606.27886#S4.p2.1),[§5](https://arxiv.org/html/2606.27886#S5.SS0.SSS0.Px3.p1.1)\.
- \[9\]R\. Burchard, P\. Brückner, M\. Bock, and K\. Van Laerhoven\(2026\-04\)HARMES: A Multi\-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.19425719)Cited by:[§3\.1](https://arxiv.org/html/2606.27886#S3.SS1.p1.1)\.
- \[10\]R\. Burchard and K\. Van Laerhoven\(2025\)Multi\-modal Atmospheric Sensing to Augment Wearable IMU\-Based Hand Washing Detection\.InSensor\-Based Activity Recognition and Artificial Intelligence,O\. Konak, B\. Arnrich, G\. Bieber, A\. Kuijper, and S\. Fudickar \(Eds\.\),Cham,pp\. 55–68\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-80856-2%5F4),ISBN 978\-3\-031\-80856\-2Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1)\.
- \[11\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\-05\)BERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding\.arXiv\.External Links:1810\.04805,[Document](https://dx.doi.org/10.48550/arXiv.1810.04805)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p9.1)\.
- \[12\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\-06\)An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\.arXiv\.External Links:2010\.11929,[Document](https://dx.doi.org/10.48550/arXiv.2010.11929)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p9.1)\.
- \[13\]V\. Ekambaram, A\. Jati, N\. Nguyen, P\. Sinthong, and J\. Kalagnanam\(2023\-08\)TSMixer: Lightweight MLP\-Mixer Model for Multivariate Time Series Forecasting\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 459–469\.External Links:2306\.09364,[Document](https://dx.doi.org/10.1145/3580305.3599533)Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p2.1),[§3\.2\.3](https://arxiv.org/html/2606.27886#S3.SS2.SSS3.p1.1)\.
- \[14\]Z\. Gao, Y\. wang, J\. Chen, J\. Xing, S\. Patel, X\. Liu, and Y\. Shi\(2023\-09\)MMTSA: Multi\-Modal Temporal Segment Attention Network for Efficient Human Activity Recognition\.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies7\(3\),pp\. 96:1–96:26\.External Links:[Document](https://dx.doi.org/10.1145/3610872)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p13.1)\.
- \[15\]R\. Girdhar, A\. El\-Nouby, Z\. Liu, M\. Singh, K\. V\. Alwala, A\. Joulin, and I\. Misra\(2023\-06\)ImageBind One Embedding Space to Bind Them All\.In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 15180–15190\.External Links:ISSN 2575\-7075,[Document](https://dx.doi.org/10.1109/CVPR52729.2023.01457)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p10.1)\.
- \[16\]Y\. Gong, Y\. Chung, and J\. Glass\(2021\-08\)AST: Audio Spectrogram Transformer\.InInterspeech 2021,pp\. 571–575\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-698)Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p2.1),[§3\.2\.2](https://arxiv.org/html/2606.27886#S3.SS2.SSS2.p1.1)\.
- \[17\]S\. Ha and S\. Choi\(2016\-07\)Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors\.In2016 International Joint Conference on Neural Networks \(IJCNN\),Vancouver, BC, Canada,pp\. 381–388\.External Links:[Document](https://dx.doi.org/10.1109/IJCNN.2016.7727224),ISBN 978\-1\-5090\-0620\-5Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p1.1)\.
- \[18\]A\. Jaegle, F\. Gimeno, A\. Brock, O\. Vinyals, A\. Zisserman, and J\. Carreira\(2021\-07\)Perceiver: General Perception with Iterative Attention\.InProceedings of the 38th International Conference on Machine Learning,pp\. 4651–4664\.External Links:ISSN 2640\-3498Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p12.1)\.
- \[19\]S\. Koutoupis, M\. A\. Zervou, K\. Kontras, M\. D\. Vos, P\. Tsakalides, and G\. Tsagkatakis\(2026\-04\)The More, the Merrier: Contrastive Fusion for Higher\-Order Multimodal Alignment\.arXiv\.External Links:2511\.21331,[Document](https://dx.doi.org/10.48550/arXiv.2511.21331)Cited by:[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p4.1)\.
- \[20\]T\. Le, T\. K\. Nguyen, T\. Le, M\. Delalandre, K\. T\. Trung, T\. Tran, and C\. Pham\(2025\-09\)Mamba\-MHAR: An efficient multimodal framework for human action recognition\.Journal of Computer Science and Cybernetics,pp\. 245–264\.External Links:ISSN 2815\-5939,[Document](https://dx.doi.org/10.15625/1813-9663/22770)Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p3.1)\.
- \[21\]S\. Lee, Y\. Lim, and K\. Lim\(2024\-04\)Multimodal sensor fusion models for real\-time exercise repetition counting with IMU sensors and respiration data\.Information Fusion104,pp\. 102153\.External Links:ISSN 15662535,[Document](https://dx.doi.org/10.1016/j.inffus.2023.102153)Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p3.1)\.
- \[22\]S\. Li, T\. Zhu, F\. Duan, L\. Chen, H\. Ning, C\. Nugent, and Y\. Wan\(2025\-02\)HARMamba: Efficient and Lightweight Wearable Sensor Human Activity Recognition Based on Bidirectional Mamba\.IEEE Internet of Things Journal12\(3\),pp\. 2373–2384\.External Links:ISSN 2327\-4662,[Document](https://dx.doi.org/10.1109/JIOT.2024.3463405)Cited by:[§3\.2\.1](https://arxiv.org/html/2606.27886#S3.SS2.SSS1.p1.1)\.
- \[23\]Z\. Liu, Y\. Shen, V\. B\. Lakshminarasimhan, P\. P\. Liang, A\. Zadeh, and L\. Morency\(2018\-05\)Efficient Low\-rank Multimodal Fusion with Modality\-Specific Factors\.arXiv\.External Links:1806\.00064,[Document](https://dx.doi.org/10.48550/arXiv.1806.00064)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p5.1)\.
- \[24\]J\. Lu, D\. Batra, D\. Parikh, and S\. Lee\(2019\-08\)ViLBERT: Pretraining Task\-Agnostic Visiolinguistic Representations for Vision\-and\-Language Tasks\.arXiv\.External Links:1908\.02265,[Document](https://dx.doi.org/10.48550/arXiv.1908.02265)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p6.1)\.
- \[25\]H\. Ma, W\. Li, X\. Zhang, S\. Gao, and S\. Lu\(2019\)AttnSense: Multi\-level Attention Mechanism For Multimodal Human Activity Recognition\.pp\. 3109–3115\.Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p6.1)\.
- \[26\]V\. Mollyn, K\. Ahuja, D\. Verma, C\. Harrison, and M\. Goel\(2022\-09\)SAMoSA: Sensing Activities with Motion and Subsampled Audio\.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies6\(3\),pp\. 1–19\.External Links:ISSN 2474\-9567,[Document](https://dx.doi.org/10.1145/3550284)Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p2.1)\.
- \[27\]S\. Moon, A\. Madotto, Z\. Lin, A\. Saraf, A\. Bearman, and B\. Damavandi\(2023\-12\)IMU2CLIP: Language\-grounded Motion Sensor Translation with Multimodal Contrastive Learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13246–13253\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.883)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p10.1)\.
- \[28\]S\. Münzner, P\. Schmidt, A\. Reiss, M\. Hanselmann, R\. Stiefelhagen, and R\. Dürichen\(2017\-09\)CNN\-based sensor fusion techniques for multimodal human activity recognition\.InProceedings of the 2017 ACM International Symposium on Wearable Computers,Maui Hawaii,pp\. 158–165\.External Links:[Document](https://dx.doi.org/10.1145/3123021.3123046),ISBN 978\-1\-4503\-5188\-1Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p2.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p3.1),[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p3.1)\.
- \[29\]A\. Nagrani, S\. Yang, A\. Arnab, A\. Jansen, C\. Schmid, and C\. Sun\(2022\-11\)Attention Bottlenecks for Multimodal Fusion\.arXiv\.External Links:2107\.00135,[Document](https://dx.doi.org/10.48550/arXiv.2107.00135)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p7.1),[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p4.1)\.
- \[30\]F\. Ordóñez and D\. Roggen\(2016\-01\)Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition\.Sensors16\(1\),pp\. 115\.External Links:ISSN 1424\-8220,[Document](https://dx.doi.org/10.3390/s16010115)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p1.1),[§3\.2\.1](https://arxiv.org/html/2606.27886#S3.SS2.SSS1.p1.1)\.
- \[31\]X\. Ouyang, X\. Shuai, J\. Zhou, I\. W\. Shi, Z\. Xie, G\. Xing, and J\. Huang\(2022\-10\)Cosmo: contrastive fusion learning with small data for multimodal human activity recognition\.InProceedings of the 28th Annual International Conference on Mobile Computing And Networking,MobiCom ’22,New York, NY, USA,pp\. 324–337\.External Links:[Document](https://dx.doi.org/10.1145/3495243.3560519),ISBN 978\-1\-4503\-9181\-8Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p10.1)\.
- \[32\]E\. Perez, F\. Strub, H\. De Vries, V\. Dumoulin, and A\. Courville\(2018\-04\)FiLM: Visual Reasoning with a General Conditioning Layer\.Proceedings of the AAAI Conference on Artificial Intelligence32\(1\)\.External Links:ISSN 2374\-3468, 2159\-5399,[Document](https://dx.doi.org/10.1609/aaai.v32i1.11671)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p11.1)\.
- \[33\]S\. Qiu, H\. Zhao, N\. Jiang, Z\. Wang, L\. Liu, Y\. An, H\. Zhao, X\. Miao, R\. Liu, and G\. Fortino\(2022\-04\)Multi\-sensor information fusion based on machine learning for real applications in human activity recognition: State\-of\-the\-art and research challenges\.Information Fusion80,pp\. 241–265\.External Links:ISSN 1566\-2535,[Document](https://dx.doi.org/10.1016/j.inffus.2021.11.006)Cited by:[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p2.1)\.
- \[34\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\(2021\-07\)Learning Transferable Visual Models From Natural Language Supervision\.InProceedings of the 38th International Conference on Machine Learning,pp\. 8748–8763\.External Links:ISSN 2640\-3498Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p10.1)\.
- \[35\]W\. Rahman, M\. K\. Hasan, S\. Lee, A\. Bagher Zadeh, C\. Mao, L\. Morency, and E\. Hoque\(2020\-07\)Integrating Multimodal Information in Large Pretrained Transformers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 2359–2369\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.214)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p11.1)\.
- \[36\]D\. Ramachandram and G\. W\. Taylor\(2017\-11\)Deep Multimodal Learning: A Survey on Recent Advances and Trends\.IEEE Signal Processing Magazine34\(6\),pp\. 96–108\.External Links:ISSN 1558\-0792,[Document](https://dx.doi.org/10.1109/MSP.2017.2738401)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p2.1)\.
- \[37\]Y\. Tian, D\. Krishnan, and P\. Isola\(2020\)Contrastive Multiview Coding\.InComputer Vision – ECCV 2020,A\. Vedaldi, H\. Bischof, T\. Brox, and J\. Frahm \(Eds\.\),Cham,pp\. 776–794\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-58621-8%5F45),ISBN 978\-3\-030\-58621\-8Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p10.1)\.
- \[38\]Y\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Z\. Kolter, L\. Morency, and R\. Salakhutdinov\(2019\-06\)Multimodal Transformer for Unaligned Multimodal Language Sequences\.arXiv\.External Links:1906\.00295,[Document](https://dx.doi.org/10.48550/arXiv.1906.00295)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p6.1),[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p4.1)\.
- \[39\]H\. R\. Vaezi Joze, A\. Shaban, M\. L\. Iuzzolino, and K\. Koishida\(2020\-06\)MMTM: Multimodal Transfer Module for CNN Fusion\.In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Seattle, WA, USA,pp\. 13286–13296\.External Links:[Document](https://dx.doi.org/10.1109/CVPR42600.2020.01330),ISBN 978\-1\-7281\-7168\-5Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p13.1)\.
- \[40\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. ukasz Kaiser, and I\. Polosukhin\(2017\)Attention is All you Need\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1706.03762)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p6.1),[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p9.1)\.
- \[41\]J\. Wang, Y\. Chen, S\. Hao, X\. Peng, and L\. Hu\(2019\-03\)Deep learning for sensor\-based activity recognition: A survey\.Pattern Recogn\. Lett\.119\(C\),pp\. 3–11\.External Links:ISSN 0167\-8655,[Document](https://dx.doi.org/10.1016/j.patrec.2018.02.010)Cited by:[§1](https://arxiv.org/html/2606.27886#S1.p1.1)\.
- \[42\]K\. Wang, C\. Liu, and R\. Zhang\(2025\-05\)CMA\-SOD: cross\-modal attention fusion network for RGB\-D salient object detection\.The Visual Computer41\(7\),pp\. 5135–5151\.External Links:ISSN 1432\-2315,[Document](https://dx.doi.org/10.1007/s00371-024-03712-9)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p6.1)\.
- \[43\]S\. K\. Yadav, K\. Tiwari, H\. M\. Pandey, and S\. A\. Akbar\(2021\-07\)A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions\.Knowledge\-Based Systems223,pp\. 106970\.External Links:ISSN 0950\-7051,[Document](https://dx.doi.org/10.1016/j.knosys.2021.106970)Cited by:[§2\.3](https://arxiv.org/html/2606.27886#S2.SS3.p2.1)\.
- \[44\]T\. A\. Yılmaz, H\. Y\. Yatbaz, E\. Ever, and A\. Yazici\(2025\-12\)Hierarchical human activity recognition with fusion of audio and multiple inertial sensor modalities\.Scientific Reports16\(1\),pp\. 382\.External Links:ISSN 2045\-2322,[Document](https://dx.doi.org/10.1038/s41598-025-29801-w)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p2.1)\.
- \[45\]A\. Zadeh, M\. Chen, S\. Poria, E\. Cambria, and L\. Morency\(2017\-09\)Tensor Fusion Network for Multimodal Sentiment Analysis\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,M\. Palmer, R\. Hwa, and S\. Riedel \(Eds\.\),Copenhagen, Denmark,pp\. 1103–1114\.External Links:[Document](https://dx.doi.org/10.18653/v1/D17-1115)Cited by:[§2\.2](https://arxiv.org/html/2606.27886#S2.SS2.p5.1)\.
- \[46\]Y\. Zhou, H\. Zhao, Y\. Huang, T\. Riedel, M\. Hefenbrock, and M\. Beigl\(2022\-09\)TinyHAR: A Lightweight Deep Learning Model Designed for Human Activity Recognition\.InProceedings of the 2022 ACM International Symposium on Wearable Computers,Cambridge United Kingdom,pp\. 89–93\.External Links:[Document](https://dx.doi.org/10.1145/3544794.3558467),ISBN 978\-1\-4503\-9424\-6Cited by:[§2\.1](https://arxiv.org/html/2606.27886#S2.SS1.p1.1),[§3\.2\.1](https://arxiv.org/html/2606.27886#S3.SS2.SSS1.p1.1)\.

## Appendix 0\.AFusion Strategy Visualization

The following diagrams depict the seven fusion strategies we employed for this comparison study\. They replace the block “Fusion Method” in Fig\.[1](https://arxiv.org/html/2606.27886#S3.F1)\.

![Refer to caption](https://arxiv.org/html/2606.27886v1/x8.png)Figure 8:Gated Multi\-modal Fusion \(GMF\)![Refer to caption](https://arxiv.org/html/2606.27886v1/x9.png)Figure 9:Late Fusion \(concatenation\)![Refer to caption](https://arxiv.org/html/2606.27886v1/x10.png)Figure 10:Cross\-Modal Attention \(CMA\)![Refer to caption](https://arxiv.org/html/2606.27886v1/x11.png)Figure 11:CLS\-Token Transformer![Refer to caption](https://arxiv.org/html/2606.27886v1/x12.png)Figure 12:Multi\-modal Bottleneck Transformer![Refer to caption](https://arxiv.org/html/2606.27886v1/x13.png)Figure 13:Low\-Rank multi\-modal Fusion![Refer to caption](https://arxiv.org/html/2606.27886v1/x14.png)Figure 14:Decision Fusion
## Appendix 0\.BAdditional Machine Learning Results

### 0\.B\.1Confusion Matrices

In this section, we show additional confusion matrices\. We include one for each unimodal model \(AST, TinyHAR, TSMixer\), as well as the 3\-Fold confusion matrix of the best performing model \(GMF\), and the confusion difference between TinyHAR and GMF

![[Uncaptioned image]](https://arxiv.org/html/2606.27886v1/x15.png)![[Uncaptioned image]](https://arxiv.org/html/2606.27886v1/x16.png)![[Uncaptioned image]](https://arxiv.org/html/2606.27886v1/x17.png)![[Uncaptioned image]](https://arxiv.org/html/2606.27886v1/x18.png)![[Uncaptioned image]](https://arxiv.org/html/2606.27886v1/x19.png)