A Graph Neural Network Model for Real-Time Gesture Recognition Based on sEMG Signals

# A Graph Neural Network Model for Real-Time Gesture Recognition Based on sEMG Signals ††thanks: This work is based on research supported by the National Science Foundation Grant #2123635 and the University of Miami Provost Fellowship in Interdisciplinary Computing.
Source: [https://arxiv.org/html/2607.07850](https://arxiv.org/html/2607.07850)
###### Abstract

For seemless control of advanced hand prostheses and augmented reality, accurate and immediate hand gestures recognition is essential\. Surface electromyography \(sEMG\) signals obtained from the forearm are commonly employed for this purpose\. In this paper, we present a novel approach for sEMG representation that utilizes graph networks which contain information about muscle activation patterns in the forearm\. Based on these graph networks, we have developed a machine learning algorithm capable of real\-time hand gesture recognition using a graph neural network\. The algorithm’s performance was evaluated using sEMG signals acquired from myoband, which has 8 electrodes placed around the forearm, involving 8 healthy subjects\. The proposed method demonstrated an average classification accuracy of 99%, surpassing the performance of state\-of\-the\-art techniques\. The average time for both graph construction and prediction stood at 48ms utilizing a M1 pro CPU, rendering the approach well\-suited for real\-time applications\.

## IIntroduction

In the realm of human\-computer interaction, real\-time gesture recognition plays a pivotal role in enabling seamless communication between humans and machines\. The ability to interpret human gestures accurately and swiftly holds immense potential for applications ranging from virtual reality \(VR\) and augmented reality \(AR\) systems to healthcare and robotics\. Some of these systems utilize surface electromyogram \(sEMG\) signals which are captured from the forearm\[[17](https://arxiv.org/html/2607.07850#bib.bib918),[6](https://arxiv.org/html/2607.07850#bib.bib919)\]\. The prevalent approach for gesture recognition involves extracting temporal and frequency domain features from the acquired sEMG recordings and subsequently classifying them using various learning algorithms, such as support vector machines \(SVMs\), linear discriminant analysis \(LDA\), and neural networks \(NNs\)\[[5](https://arxiv.org/html/2607.07850#bib.bib920),[21](https://arxiv.org/html/2607.07850#bib.bib430),[4](https://arxiv.org/html/2607.07850#bib.bib921),[15](https://arxiv.org/html/2607.07850#bib.bib429),[8](https://arxiv.org/html/2607.07850#bib.bib427)\]\.

While numerous studies have been conducted to accurately classify hand gestures from pre\-recorded sEMG signals\[[21](https://arxiv.org/html/2607.07850#bib.bib430),[14](https://arxiv.org/html/2607.07850#bib.bib432),[11](https://arxiv.org/html/2607.07850#bib.bib431)\], only a limited number of research works have concentrated on*real\-time*hand gesture recognition using sEMG signals from the forearm\[[7](https://arxiv.org/html/2607.07850#bib.bib426),[9](https://arxiv.org/html/2607.07850#bib.bib915),[16](https://arxiv.org/html/2607.07850#bib.bib433)\]\. The majority of existing methods utilize binning of sEMG signals and extracting specific features of the wave within each bin for classification purposes\[[20](https://arxiv.org/html/2607.07850#bib.bib434),[11](https://arxiv.org/html/2607.07850#bib.bib431),[7](https://arxiv.org/html/2607.07850#bib.bib426),[9](https://arxiv.org/html/2607.07850#bib.bib915)\]\. Thus, the issue of computational complexity arising from continuous binning poses a challenge for real\-time gesture prediction\. In response to these challenges, this paper introduces an innovative approach harnessing the power of graph theory for real\-time gesture recognition\. The motivation for this research is rooted in the need for robust and efficient gesture recognition systems that can operate in real\-time, allowing for natural and intuitive interactions between humans and digital interfaces\.

Graph\-based methods offer a versatile framework for modeling complex relationships and patterns within data, making them ideally suited for capturing the spatial and temporal dynamics of gestures\[[24](https://arxiv.org/html/2607.07850#bib.bib917)\]\. By representing gestures as dynamic graphs, we bridge the gap between raw sensor data and meaningful gesture representations\. Each gesture is conceptualized as a graph, with nodes representing the electrodes on the human body and edges capturing the temporal dependencies and spatial correlations between the nodes\[[9](https://arxiv.org/html/2607.07850#bib.bib915)\]\. Traditional graph kernel based methods such as shortest path kernel\[[2](https://arxiv.org/html/2607.07850#bib.bib438)\], graphlet kernel\[[19](https://arxiv.org/html/2607.07850#bib.bib435)\], random walk kernel\[[22](https://arxiv.org/html/2607.07850#bib.bib922)\], Weisfelier\-Lehman subtree kernel\[[18](https://arxiv.org/html/2607.07850#bib.bib923)\], propagation kernel\[[12](https://arxiv.org/html/2607.07850#bib.bib436)\], and deep graph kernels\[[23](https://arxiv.org/html/2607.07850#bib.bib437)\]often fall short in terms of performance with smaller graph sizes, when compared with NN based methods\[[25](https://arxiv.org/html/2607.07850#bib.bib924)\]\.

In this paper, we propose a graph\-based approach which builds upon recent advancements in graph neural networks \(GNNs\), graph convolution networks \(GCNs\) and graph signal processing \(GSP\) to enable the recognition of intricate gestures with minimal latency\[[10](https://arxiv.org/html/2607.07850#bib.bib428),[3](https://arxiv.org/html/2607.07850#bib.bib916)\]\. We employ the novel idea of constructing a graph network from time series sEMG signals that can represent the individual and mutual activation patterns of forearm muscles\. We examine how this graph network could be employed to detect the time of gesture onset and also to classify different gesture types \(for which we utilize a GNN\)\. We explore its computational efficiency, real\-time functionality, and accuracy, and demonstrate that that it surpasses the performance of state\-of\-the\-art real\-time gesture classification algorithms\.

## IIMethodology

### II\-AOverview

In our study, we leverage the insights from previous research which has highlighted the crucial role that correlation between sEMG signal play in understanding complex electrical and chemical processes that occur in the nervous system dynamics\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\]\. Building upon this foundation, we adopt a graph\-based approach to model the intricate relationships within our time series data\. Specifically, we construct a graph representation where electrodes serve as nodes and the calculated correlations between the corresponding time series act as edges\. By transforming our time series data into a graph structure based on these correlations, we gain a holistic perspective on the individual electrode activities and their inter\-dependencies\. This graph\-based representation provides a powerful framework for analyzing the dynamic interactions and the correlations between sEMG signals and the gestures and enables us to unravel intricate patterns and gain deeper insights into the neuromuscular communication\.

### II\-BGraph Generation

Denote by\{X1​\(t\),X2​\(t\),…,XM​\(t\)\},t=0,1,2,…\\\{X\_\{1\}\(t\),\\,X\_\{2\}\(t\),\\,\\ldots,\\,X\_\{M\}\(t\)\\\},\\;t=0,1,2,\\ldots\\,, the surface sEMG signal time series recorded fromMMchannels\. In the dataset that we explored \(see Section[III\-A](https://arxiv.org/html/2607.07850#S3.SS1)\), these time series correspond to sEMG signals produced by different muscle groups of human subjects\. Here,ttis a non\-negative integer that represents the sampling instant\. Fig\.[1](https://arxiv.org/html/2607.07850#S2.F1)depicts the pipeline through which graphs are extracted from these sEMG time series for gesture classification\.

![Refer to caption](https://arxiv.org/html/2607.07850v1/x1.png)Figure 1:Pipeline of graph extraction from sEMG time series for gesture classification\. Graphs are extracted only from those windows where a gusture is detected\.First, each continuous sEMG time series signalXi​\(⋅\)X\_\{i\}\(\\cdot\)is partitioned into windowsW​\[n\],n=0,1,2,…W\[n\],\\;n=0,1,2,\\ldots, each window being of lengthLLsamples and having an overlap percentage ofδ\\deltasamples with the next consecutive window\. These windowed time series allow us to capture fine\-grained temporal patterns within the signal\. Let use denote the time series signalXiX\_\{i\}restricted to within the windowW​\[n\]W\[n\]byXi​\[n\]X\_\{i\}\[n\], i\.e\.,

Xi​\[n\]=\{Xi​\[n\]​\(t\)=Xi​\(n​L\+t\),t=0,1,…,L−1\}\.X\_\{i\}\[n\]=\\\{X\_\{i\}\[n\]\(t\)=X\_\{i\}\(nL\+t\),\\;t=0,1,\\ldots,L\-1\\\}\.\(1\)
Next, within each windowW​\[n\]W\[n\], we calculate the Pearson correlation coefficient between each pair of time series signalsXi​\[n\]X\_\{i\}\[n\]andXj​\[n\]X\_\{j\}\[n\]\. This produces a\(M×M\)\(M\\times M\)\-sized correlation matrixP¯​\[n\]\\underline\{P\}\[n\]whose\(i,j\)\(i,j\)\-th element is the Pearson correlation coefficient between the time seriesXi​\[n\]X\_\{i\}\[n\]andXj​\[n\]X\_\{j\}\[n\]given by

P¯​\[n\]i,j=∑t=0L−1\(Xi​\[n\]​\(t\)−Xi​\[n\]¯\)​\(Xj​\[n\]​\(t\)−Xj​\[n\]¯\)∑t=0L−1\(Xi​\[n\]​\(t\)−Xi​\[n\]¯\)2​∑t=0L−1\(Xj​\[n\]​\(t\)−Xj​\[n\]¯\)2,\\underline\{P\}\[n\]\_\{i,j\}\\\\ =\\frac\{\\displaystyle\\sum\_\{t=0\}^\{L\-1\}\(X\_\{i\}\[n\]\(t\)\-\\overline\{X\_\{i\}\[n\]\}\)\\,\(X\_\{j\}\[n\]\(t\)\-\\overline\{X\_\{j\}\[n\]\}\)\}\{\\sqrt\{\\displaystyle\\sum\_\{t=0\}^\{L\-1\}\(X\_\{i\}\[n\]\(t\)\-\\overline\{X\_\{i\}\[n\]\}\)^\{2\}\}\\sqrt\{\\displaystyle\\sum\_\{t=0\}^\{L\-1\}\(X\_\{j\}\[n\]\(t\)\-\\overline\{X\_\{j\}\[n\]\}\)^\{2\}\}\},\(2\)whereXi¯​\[n\]\\overline\{X\_\{i\}\}\[n\]denotes the average signal amplitude of the time seriesXi​\[n\]​\(t\),t=0,1,…,L−1X\_\{i\}\[n\]\(t\),\\;t=0,1,\\ldots,L\-1\. The value ofP¯​\[n\]i,j\\underline\{P\}\[n\]\_\{i,j\}measures the linear relationship betweenXi​\[n\]X\_\{i\}\[n\]andXj​\[n\]X\_\{j\}\[n\]; it takes values within the interval\[−1,\+1\]\[\-1,\\,\+1\],\+1\+1indicating a perfect positive correlation,−1\-1indicating a perfect negative correlation, and0indicating no correlation\.

Finally, we use this correlation matrixP¯​\[n\]\\underline\{P\}\[n\], with its diagonal elements pinned at zero \(so thatP¯​\[n\]i,i=0,i=1,…,M\\underline\{P\}\[n\]\_\{i,i\}=0,\\;i=1,\\ldots,M\), as the adjacency matrix of a graph𝒢​\[n\]=\(𝒱​\[n\],ℰ​\[n\]\)\\mathcal\{G\}\[n\]=\(\\mathcal\{V\}\[n\],\\,\\mathcal\{E\}\[n\]\)\. Here,𝒱​\[n\]\\mathcal\{V\}\[n\]represents the set of nodes andℰ​\[n\]\\mathcal\{E\}\[n\]represents the set of edges with the calculated Pearson correlation coefficients constituting the edge weights\. Note that,𝒢​\[n\]\\mathcal\{G\}\[n\]is undirected \(becauseP​\[n\]i,j=P​\[n\]j,iP\[n\]\_\{i,j\}=P\[n\]\_\{j,i\}\) and is free of self\-loops \(becauseP​\[n\]i,i=0P\[n\]\_\{i,i\}=0\)\. Thus, it is a simple, fully connected undirected graph\[[13](https://arxiv.org/html/2607.07850#bib.bib3)\]associated with theMMtime series falling within the windowW​\[n\]W\[n\]\.

### II\-CEvent Detection

With the undirected graph sequence𝒢​\[n\],n=0,1,2,…\\mathcal\{G\}\[n\],\\;n=0,1,2,\\ldots, generated from the time series in hand, our objective is to identify meaningful temporal segments where significant gestures had occurred\. Akin to what is used in\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\], we achieve this by employing the following event detection measure:

Δ​\[n,τ\]=∑i=1M∑t=0L−1\(Xi​\[n\]​\(t\)−Xi​\[n−τ\]​\(t\)\)2,\\Delta\[n,\\tau\]=\\sqrt\{\\sum\_\{i=1\}^\{M\}\\sum\_\{t=0\}^\{L\-1\}\\left\(X\_\{i\}\[n\]\(t\)\-X\_\{i\}\[n\-\\tau\]\(t\)\\right\)^\{2\}\},\(3\)whereXi​\[n−τ\]​\(t\)X\_\{i\}\[n\-\\tau\]\(t\)is the time series falling within the earlier windowW​\[n−τ\]W\[n\-\\tau\], for some positive integerτ\\tau\. Note that, one may view this as the Fröbenius norm of a certain ‘error’ matrix\. To be specific, consider the\(M×L\)\(M\\times L\)\-sized matrix

X¯\[n\]\(0:L−1\)=\[X1​\[n\]​\(0\)X1​\[n\]​\(1\)⋯X1​\[n\]​\(L−1\)X2​\[n\]​\(0\)X2​\[n\]​\(1\)⋯X2​\[n\]​\(L−1\)⋮⋮⋱⋮XM​\[n\]​\(0\)XM​\[n\]​\(1\)⋯XM​\[n\]​\(L−1\)\],\\underline\{X\}\[n\]\(0:L\-1\)\\\\ =\\begin\{bmatrix\}X\_\{1\}\[n\]\(0\)&X\_\{1\}\[n\]\(1\)&\\cdots&X\_\{1\}\[n\]\(L\-1\)\\\\ X\_\{2\}\[n\]\(0\)&X\_\{2\}\[n\]\(1\)&\\cdots&X\_\{2\}\[n\]\(L\-1\)\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ X\_\{M\}\[n\]\(0\)&X\_\{M\}\[n\]\(1\)&\\cdots&X\_\{M\}\[n\]\(L\-1\)\\end\{bmatrix\},\(4\)forn=0,1,…n=0,1,\\ldots\. Then,

Δ\[n,τ\]=∥X¯\[n\]\(0:L−1\)−X¯\[n−τ\]\(0:L−1\)∥F\.\\Delta\[n,\\tau\]=\\\|\\underline\{X\}\[n\]\(0:L\-1\)\-\\underline\{X\}\[n\-\\tau\]\(0:L\-1\)\\\|\_\{F\}\.\(5\)Theii\-th row ofX¯\[n\]\(0:L−1\)\\underline\{X\}\[n\]\(0:L\-1\)corresponds to the samples0,1,…,L−10,1,\\ldots,L\-1of the time seriesXiX\_\{i\}that lie within windowW​\[n\]W\[n\]; theii\-th row ofX¯\[n−τ\]\(0:L−1\)\\underline\{X\}\[n\-\\tau\]\(0:L\-1\)corresponds to the time seriesXiX\_\{i\}that lie within windowW​\[n−τ\]W\[n\-\\tau\]\.

We flag the occurrence of a gesture event within only those windowsW​\[n\]W\[n\]whereΔ​\[n,τ\]\\Delta\[n,\\tau\]exceeds a predefined threshold\. Similar to the strategy employed in\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\], for event detection purposes, we consider only theLL\-length time seriesXi​\[n\],i=1,…,NX\_\{i\}\[n\],\\;i=1,\\ldots,N, falling within these windowsW​\[n\]W\[n\]where a gesture event had occurred\. This focused approach ensures that our predictions are based on meaningful temporal segments characterized by significant events thus avoiding regions of low signal\-to\-noise ratio\.

### II\-DClassification

#### II\-D1Training

The graphs associated with the windows where an event had occurred, together with the corresponding hand gesture labels, serve as the training data for our classification algorithm\. They encapsulate the correlation between signal channels thus reflecting muscle activity related to specific hand gestures\. A straightforward GNN architecture, which includes two fully connected layers with a ReLU activation function, was then trained on this training data set\.

#### II\-D2Testing

The trained GNN is then used to classify hand gestures\. It is worthwhile mentioning that leveraging the graph structure to propagate and sustain the information between channels allows even a simple network to understand the context and relationship in the data\[[1](https://arxiv.org/html/2607.07850#bib.bib914)\]\.

## IIIExperiment

### III\-ADataset

\.

![Refer to caption](https://arxiv.org/html/2607.07850v1/x2.png)

Figure 2:Top row left\-to\-right: sEMG signals of the dataset captures the 5 hand gestures, pointer, middle finger flexion, ring finger flexion, v\-flexion, and hand closure\. Bottom row: the hand in the neutral position with Myo band placed in forearm\.The dataset, sourced from\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\], captures sEMG signal activity ofC=5C=5gestures, pointer, middle finger flexion, ring finger flexion, v\-flexion, and hand closure \(see Fig\.[2](https://arxiv.org/html/2607.07850#S3.F2)\)\. The signals were recorded at a sampling rate of200​Hz200\\,\\textrm\{Hz\}fromK=8K=8healthy subjects \(4 males and 4 females, aged25±225\\pm 2years\) using a Myo band \(Thalamic Labs, Canada\)\. During data collection, the subjects were asked to perform 20 repetitions of each gesture with their right hand\. A single time series \(roughly 20,000 samples\) is generated by one subject repeatedly holding one gesture for a period of5​s5\\,\\textrm\{s\}followed by a resting period of5​s5\\,\\textrm\{s\}\. Each subject performs this task with 8 electrodes for each gesture, thus producing 40 time series\. WithK=8K=8subjects participating in the study, we thus have40​K=32040K=320time series in total\.

### III\-BParameters

To extract the graphs from the time series, we used a window width ofL=80​samplesL=80\\,\\textrm\{samples\}\(corresponding to0\.4​s0\.4\\,\\textrm\{s\}\), an overlap ofδ=95%\\delta=95\\%between consecutive windows, and for computing the event detection measure in \([3](https://arxiv.org/html/2607.07850#S2.E3)\) a window delay ofτ=5\\tau=5windows \(corresponding to 20 samples\)\. To select the event detection threshold, we followed the procedure that is utilized in\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\]where the threshold employed is subject\-dependant\.

Let us denote byTkT\_\{k\}the threshold employed for the time series produced by thekk\-th, wherek=1,…,K=8k=1,\\ldots,K=8\. To getTkT\_\{k\}, we first compute the event detection measureΔk,c​\[n,τ\]\\Delta\_\{k,c\}\[n,\\tau\]separately for each gesture type\. Here,k,k=1,…,K=8k,\\;k=1,\\ldots,K=8, identifies the subject andc,c=1,…,C=5c,\\;c=1,\\ldots,C=5, identifies the gesture\. Letσ2​\(Δk,c​\[τ\]\)\\sigma^\{2\}\(\\Delta\_\{k,c\}\[\\tau\]\)denote the variance ofΔk,c​\[n,τ\]\\Delta\_\{k,c\}\[n,\\tau\]over the windows, i\.e\.,

σ2​\(Δk,c​\[τ\]\)=1N​∑n=0N−1\(Δk,c​\[n,τ\]−Δ¯k,c​\[n,τ\]\)2,\\sigma^\{2\}\(\\Delta\_\{k,c\}\[\\tau\]\)=\\frac\{1\}\{N\}\\sum\_\{n=0\}^\{N\-1\}\(\\Delta\_\{k,c\}\[n,\\tau\]\-\\overline\{\\Delta\}\_\{k,c\}\[n,\\tau\]\)^\{2\},\(6\)whereNNdenotes the number of windows andΔ¯k,c​\[n,τ\]\\overline\{\\Delta\}\_\{k,c\}\[n,\\tau\]denotes the average over the windows\. These variances are then used to determine the subject\-dependant event detection threshold as\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\]

Tk=4C​∑c=1Cσ2​\(Δk,c​\[τ\]\),k=1,…,K=8\.T\_\{k\}=\\frac\{4\}\{C\}\\sum\_\{c=1\}^\{C\}\\sqrt\{\\sigma^\{2\}\(\\Delta\_\{k,c\}\[\\tau\]\)\},\\;k=1,\\ldots,K=8\.\(7\)For subjectkk, an event is considered to occur within a specific window ifΔ​\[n,τ\]≥Tk\\Delta\[n,\\tau\]\\geq T\_\{k\}, i\.e\., the event detection metric in \([3](https://arxiv.org/html/2607.07850#S2.E3)\) exceeds the thresholdTkT\_\{k\}\. The entire window, in which an event is detected, is then included in the training dataset, while segments where no event is detected are ignored\.

### III\-CArchitecture

After extracting a training dataset of graphs for each of the subjects and their respective hand gestures, the GNN \(implemented using PyTorch\) was trained with the stochastic gradient descent optimizer using the following parameters: learning rate of 0\.01 for 100 epochs, and 64 and 2048 neurons in the first fully and second fully connected layers, respectively\.

### III\-DTraining and Real\-Time Prediction

The GNN model was evaluated using a randomly selected 20% extracted from the original dataset\. This testing dataset had an average of 927 graphs per class for an overlap ofδ=90%\\delta=90\\%\.

The classifier, after training, was employed in real\-time for predicting gestures\. The pipeline through which the gestures are classified using the trained model appears in Fig\.[1](https://arxiv.org/html/2607.07850#S2.F1)\. The pseudo\-code of the prediction algorithm is in Algorithm[1](https://arxiv.org/html/2607.07850#alg1)\.

Algorithm 1Real\-time gesture detection pipelines​u​b​j​e​c​t→ksubject\\to k

w​i​n​d​o​w→nwindow\\to n

Fetch previous

LLsamples

X¯\[n\]\(0:L−1\)\\underline\{X\}\[n\]\(0:L\-1\);

Calculate event detection

Δ​\[n,τ\]←\\Delta\[n,\\tau\]\\leftarrowevent likelihood metric based on \([3](https://arxiv.org/html/2607.07850#S2.E3)\)

if

Δ​\[n,τ\]≥\\Delta\[n,\\tau\]\\geqevent threshold \(

TkT\_\{k\}\)then

Generate graph network

𝒢​\[n\]←\\mathcal\{G\}\[n\]\\leftarrowgraph network based on matrix

P¯​\[n\]\\underline\{P\}\[n\]
prediction

←GNN​\(𝒢​\[n\]\)\\leftarrow\\mathrm\{GNN\}\(\\mathcal\{G\}\[n\]\)
endif

Wait fort0t\_\{0\}number of new samples

n←n\+1n\\leftarrow n\+1

Initially, the framework begins with the acquisition of new data, which is seamlessly captured from the Myo band\. The framework awaits the accumulation oft0=20t\_\{0\}=20\(equivalent toF=0\.1​sF=0\.1\\,\\textrm\{s\}\) new samples before initiating the subsequent phase of processing\. This intentional waiting period ensures that the framework captures gestures during transitions, therefore enhancing the precision and efficacy of our process\.

The model prediction pipeline was configured to activate upon detection of a gesture event\. Once an event was detected, the window of samples was utilized to generate a graph, utilizing the preprocessing described in Section[II\-B](https://arxiv.org/html/2607.07850#S2.SS2)\. The generated graph is inferenced using the trained classifier to identify the gesture\. The time taken for the end\-to\-end process of detecting a gesture from the data is denoted asdd\. We demonstrate that the time required to process and classify a gesture is significantly shorter than the time the framework awaits the accumulation oft0t\_\{0\}samples, thereby ensuring real\-time applicability\.

## IVResults and Discussion

The classification accuracies obtained from the proposed real\-time gesture recognition algorithm and a comparison with state\-of\-the\-art methods described in\[[7](https://arxiv.org/html/2607.07850#bib.bib426),[9](https://arxiv.org/html/2607.07850#bib.bib915),[8](https://arxiv.org/html/2607.07850#bib.bib427)\]appear in Table[I](https://arxiv.org/html/2607.07850#S4.T1)\. Crepin, et al\.\[[7](https://arxiv.org/html/2607.07850#bib.bib426)\]employed a continuous binning approach, applying a LDA classifier to each bin while treating channels as independent entities\. In contrast, Furui, et al\.\[[9](https://arxiv.org/html/2607.07850#bib.bib915)\]utilized a muscle synergy\-based approach, conducting classification for each bin through a recurrent log\-linearized Gaussian mixture network\. On the other hand, Ashwin, et al\.\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\]utilized a temporal muscle activation \(TMA\) image based approach, which utilized a CNN for classification\.

Our method appears to offer superior accuracy in recognizing gestures compared to these techniques\. In comparison to Crepin, et al\.\[[7](https://arxiv.org/html/2607.07850#bib.bib426)\], our method exhibited higher classification accuracy across all gestures, possibly attributed to our consideration of both individual and mutual muscle activations\. When compared with Ashwin, et al\.\[[8](https://arxiv.org/html/2607.07850#bib.bib427)\], utilizing correlation between time series instead of the TMA mapping appears to yield better results\. Additionally, the data involves complex relationships and dependencies that are not easily captured by regular grid structures\. It is easier to capture these dependencies via a graph network based approach \(a la GNN\) than a CNN based approach\.

In Fig[3](https://arxiv.org/html/2607.07850#S4.F3), we report the confusion matrix corresponding to our approach\. The proposed method exhibited an average classification accuracy above 98%\. Notably, the model achieved this performance even in the presence of a significant class imbalance problem\.

![Refer to caption](https://arxiv.org/html/2607.07850v1/x3.png)Figure 3:Confusion matrix associated with window overlap parameterδ=90%\\delta=90\\%\.To systematically test the robustness of the algorithm, we studied the model’s performance with different window overlap \(i\.e,δ\\delta\) values\. With decreasingδ\\delta, the number of graphs being produced is reduced meaning that the GNN has less training data from which to learn\. Fig\.[4](https://arxiv.org/html/2607.07850#S4.F4)shows the training sample size and the test accuracies which were yielded\. The proposed method demonstrates significant performance even in limited training data scenarios \(i\.e\., lowerδ\\deltavalues\)\. In contrast\. the work in Ashwin, et al\. require a larger overlap of consecutive windows \(δ=97\.5%\\delta=97\.5\\%\) to maximize their training dataset\. This highlights the GNN’s capacity to generalize effectively even with limited training data\. It could be noted that our approach surpasses testing accuracy of 99% when training dataset was maximized via the appropriate selection ofδ\\delta\(δ=97\.5%\\delta=97\.5\\%\)\.

![Refer to caption](https://arxiv.org/html/2607.07850v1/x4.png)Figure 4:Variation of number of training samples \(left\) and accuracy \(right\) with respect to window non overlap parameter1−δ1\-\\delta\.To test the model’s ability to distinguish between classes and to illustrate the trade\-off between the true positive rate \(sensitivity\) and the false positive rate, we employed micro average Receiver Operating Characteristic \(ROC\) analysis with respect to the window overlap parameterδ\\delta\. The Area Under the Curve \(AUC\) is seen to be affected only slightly with decreasingδ\\delta\. This provides a clear measure of the GNN’s responsiveness to changes in overlap, offering insights into model robustness\. It is noteworthy that, whenδ\\deltawas reduced to 70%, only 15% of the maximum attainable training data samples \(i\.e\., graphs\) were produced\. Despite this significant reduction, an AUC score of 0\.967 was attained, highlighting the GNN’s capacity to sustain high performance levels even with a diminished training dataset\.

![Refer to caption](https://arxiv.org/html/2607.07850v1/x5.png)Figure 5:ROC curves with respect to window overlap parameterδ\\delta\.The average computation time of the proposed method was48​ms48\\,\\textrm\{ms\}\(3​ms3\\,\\textrm\{ms\}for graph generation and45​ms45\\,\\textrm\{ms\}for prediction\)\. The CPU usage was 39% for gesture prediction when executed on a computer featuring a M1 pro CPU\. Considering that this computation time was notably shorter than the duration between two successive events \(denoted asFFin Fig\.[1](https://arxiv.org/html/2607.07850#S2.F1)\), the algorithm appears to support real\-time execution\.

TABLE I:Classification Accuracy \(%\)
## VConclusion

In this study, we have presented a novel technique for generating undirected weighted graphs based on the interrelationships between sEMG signals\. Our method was applied to identify the onset of gestures and successfully recognize five specific hand gestures in real\-time\. The accuracy achieved by our approach seems to be higher than the performance offered by state\-of\-the\-art methods, thus showcasing the effectiveness of graph\-based approaches\. The method also showcases robustness across training datasets of various sizes\.

Looking forward, it might be possible to develop real\-time fluid gesture recognition, representing a significant leap forward for individuals with disabilities\. One may also explore regions where events went unnoticed to improve gesture detection, even in situations with a sub\-optimal signal\-to\-noise ratio\.

## References

- \[1\]P\. W\. Battaglia, J\. B\. Hamrick, V\. Bapst, A\. Sanchez\-Gonzalez, V\. Zambaldi, M\. Malinowski, A\. Tacchetti, D\. Raposo, A\. Santoro, R\. Faulkner,et al\.\(2018\)Relational inductive biases, deep learning, and graph networks\.arXiv preprint arXiv:1806\.01261\.Cited by:[§II\-D2](https://arxiv.org/html/2607.07850#S2.SS4.SSS2.p1.1)\.
- \[2\]K\. M\. Borgwardt and H\. Kriegel\(2005\)Shortest\-path kernels on graphs\.InFifth IEEE international conference on data mining \(ICDM’05\),pp\. 8–pp\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[3\]H\. Chen, H\. Zhao, B\. Qi, S\. Zhang, and Z\. Yu\(2023\)Gesture recognition of graph convolutional neural network based on spatial domain\.Neural Computing and Applications35\(3\),pp\. 2157–2167\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p4.1)\.
- \[4\]X\. Chen and Z\. J\. Wang\(2013\)Pattern recognition of number gestures based on a wireless surface EMG system\.Biomedical Signal Processing and Control8\(2\),pp\. 184–192\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1)\.
- \[5\]J\. Chu, I\. Moon, Y\. Lee, S\. Kim, and M\. Mun\(2007\)A supervised feature\-projection\-based real\-time EMG pattern recognition for multifunction myoelectric hand control\.IEEE/ASME Transactions on mechatronics12\(3\),pp\. 282–290\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1)\.
- \[6\]R\. Clement, K\. E\. Bugler, and C\. W\. Oliver\(2011\)Bionic prosthetic hands: a review of present technology and future aspirations\.The surgeon9\(6\),pp\. 336–340\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1)\.
- \[7\]R\. Crepin, C\. L\. Fall, Q\. Mascret, C\. Gosselin, A\. Campeau\-Lecours, and B\. Gosselin\(2018\)Real\-time hand motion recognition using sEMG patterns classification\.In2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society \(EMBC\),pp\. 2655–2658\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p2.1),[TABLE I](https://arxiv.org/html/2607.07850#S4.T1.1.2.1.2),[§IV](https://arxiv.org/html/2607.07850#S4.p1.1),[§IV](https://arxiv.org/html/2607.07850#S4.p2.1)\.
- \[8\]A\. De Silva, M\. V\. Perera, K\. Wickramasinghe, A\. M\. Naim, T\. D\. Lalitharatne, and S\. L\. Kappel\(2020\)Real\-time hand gesture recognition using temporal muscle activation maps of multi\-channel sEMG signals\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1299–1303\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.07850#S2.SS1.p1.1),[§II\-C](https://arxiv.org/html/2607.07850#S2.SS3.p1.1),[§II\-C](https://arxiv.org/html/2607.07850#S2.SS3.p2.5),[§III\-A](https://arxiv.org/html/2607.07850#S3.SS1.p1.8),[§III\-B](https://arxiv.org/html/2607.07850#S3.SS2.p1.4),[§III\-B](https://arxiv.org/html/2607.07850#S3.SS2.p2.11),[TABLE I](https://arxiv.org/html/2607.07850#S4.T1.1.2.1.4),[§IV](https://arxiv.org/html/2607.07850#S4.p1.1),[§IV](https://arxiv.org/html/2607.07850#S4.p2.1)\.
- \[9\]A\. Furui, S\. Eto, K\. Nakagaki, K\. Shimada, G\. Nakamura, A\. Masuda, T\. Chin, and T\. Tsuji\(2019\)A myoelectric prosthetic hand with muscle synergy–based motion determination and impedance model–based biomimetic control\.Science Robotics4\(31\),pp\. eaaw6339\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p2.1),[§I](https://arxiv.org/html/2607.07850#S1.p3.1),[TABLE I](https://arxiv.org/html/2607.07850#S4.T1.1.2.1.3),[§IV](https://arxiv.org/html/2607.07850#S4.p1.1)\.
- \[10\]X\. Gao, K\. Li, Y\. Zhang, Q\. Miao, L\. Sheng, J\. Xie, and J\. Xu\(2019\)3D skeleton\-based video action recognition by graph convolution network\.In2019 IEEE International Conference on Smart Internet of Things \(SmartIoT\),Vol\.,pp\. 500–501\.External Links:[Document](https://dx.doi.org/10.1109/SmartIoT.2019.00093)Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p4.1)\.
- \[11\]R\. N\. Khushaba and S\. Kodagoda\(2012\)Electromyogram \(EMG\) feature reduction using mutual components analysis for multifunction prosthetic fingers control\.In2012 12th International Conference on Control Automation Robotics & Vision \(ICARCV\),pp\. 1534–1539\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p2.1)\.
- \[12\]M\. Neumann, N\. Patricia, R\. Garnett, and K\. Kersting\(2012\)Efficient graph kernels by randomization\.InMachine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24\-28, 2012\. Proceedings, Part I 23,pp\. 378–393\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[13\]M\. E\. J\. Newman\(2018\)Networks\.second edition,Oxford University Press,Oxford, UK\.Cited by:[§II\-B](https://arxiv.org/html/2607.07850#S2.SS2.p4.10)\.
- \[14\]A\. Phinyomark, S\. Hirunviriya, C\. Limsakul, and P\. Phukpattaranont\(2010\)Evaluation of EMG feature extraction for hand movement recognition based on euclidean distance and standard deviation\.InECTI\-CON2010: The 2010 ECTI International Confernce on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology,pp\. 856–860\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p2.1)\.
- \[15\]S\. Raurale, J\. McAllister, and J\. M\. del Rincon\(2018\)EMG acquisition and hand pose classification for bionic hands from randomly\-placed sensors\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1105–1109\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1)\.
- \[16\]S\. Raurale, J\. McAllister, and J\. M\. del Rincon\(2019\)EMG wrist\-hand motion recognition system for real\-time embedded platform\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1523–1527\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p2.1)\.
- \[17\]P\. Shenoy, K\. J\. Miller, B\. Crawford, and R\. P\. Rao\(2008\)Online electromyographic control of a robotic prosthesis\.IEEE transactions on biomedical engineering55\(3\),pp\. 1128–1135\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1)\.
- \[18\]N\. Shervashidze, P\. Schweitzer, E\. J\. Van Leeuwen, K\. Mehlhorn, and K\. M\. Borgwardt\(2011\)Weisfeiler\-lehman graph kernels\.\.Journal of Machine Learning Research12\(9\)\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[19\]N\. Shervashidze, S\. Vishwanathan, T\. Petri, K\. Mehlhorn, and K\. Borgwardt\(2009\)Efficient graphlet kernels for large graph comparison\.InArtificial intelligence and statistics,pp\. 488–495\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[20\]K\. Shima and T\. Tsuji\(2010\)Classification of combined motions in human joints through learning of individual motions based on muscle synergy theory\.In2010 IEEE/SICE International Symposium on System Integration,pp\. 323–328\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p2.1)\.
- \[21\]F\. Tenore, A\. Ramos, A\. Fahmy, S\. Acharya, R\. Etienne\-Cummings, and N\. V\. Thakor\(2007\)Towards the control of individual fingers of a prosthetic hand using surface EMG signals\.In2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society,pp\. 6145–6148\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p1.1),[§I](https://arxiv.org/html/2607.07850#S1.p2.1)\.
- \[22\]S\. V\. N\. Vishwanathan, N\. N\. Schraudolph, R\. Kondor, and K\. M\. Borgwardt\(2010\)Graph kernels\.The Journal of Machine Learning Research11,pp\. 1201–1242\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[23\]P\. Yanardag and S\. Vishwanathan\(2015\)Deep graph kernels\.InProceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining,pp\. 1365–1374\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[24\]B\. Yu, H\. Yin, and Z\. Zhu\(2017\)Spatio\-temporal graph convolutional networks: a deep learning framework for traffic forecasting\.arXiv preprint arXiv:1709\.04875\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.
- \[25\]Y\. Zhou, H\. Huo, Z\. Hou, and F\. Bu\(2023\)A deep graph convolutional neural network architecture for graph classification\.Plos one18\(3\),pp\. e0279604\.Cited by:[§I](https://arxiv.org/html/2607.07850#S1.p3.1)\.