Domain adaptation for handwriting trajectory reconstruction from IMU sensors

# Domain adaptation for handwriting trajectory reconstruction from IMU sensors
Source: [https://arxiv.org/html/2607.26736](https://arxiv.org/html/2607.26736)
11institutetext:IRISA, Universite de Rennes, INSA Rennes, Rennes, France22institutetext:IRISA, Universite Rennes 2, Rennes, France
22email:florent\.imbert@irisa\.fr
###### Abstract

Digital pens are commonly used to write on digital devices, providing the handwriting trace and enhancing human\-computer interation\. This study focuses on a digital pen equipped with kinematic sensors, allowing users to write on any surface while simultaneously preserving a digital trajectory of handwriting\. This technology holds significant potential as a valuable educational tool, particularly in classrooms where it can facilitate the process of learning to write\. A major issue is based on the difference in captured signals between adults and children\. For similar handwriting trace, we have large differences in sensor signals due to differences in speed and confidence in the handwriting gesture of children\. To address this, we investigate a domain adaptation approach to build a unified intermediate feature representation aimed at facilitating the trajectory reconstruction\. We demonstrate the interest of domain adaptation methods in leveraging existing knowledge for application in different contexts\. Specifically, we compare our domain adaptation approach with two other methods: training the model from scratch and fine\-tuning the model\.

## 1Introduction

This work focuses on reconstructing digital handwriting trajectories using the Digipen stylus\[[3](https://arxiv.org/html/2607.26736#bib.bib10)\]\. The Digipen is equipped with 2 accelerometers, 1 gyroscope and a force sensor, and it can be used to write on any surface\. While Inertial Measurement Units \(IMU\) are commonly used in tracking systems due to their cost\-effectiveness, their signals are often imprecise due to high noise levels, presenting challenges\. This issue is particularly prominent in the context of handwriting trajectory reconstruction, where precision is crucial, especially for e\-learning purposes that require accurate feedback\.

Although the Digipen stylus can be used on any surface and by different users, state\-of\-the art works\[[9](https://arxiv.org/html/2607.26736#bib.bib3),[10](https://arxiv.org/html/2607.26736#bib.bib1)\]have experimented handwriting reconstruction approaches from data written by adults on tablets\. Using the Digipen in another experimental context, e\.g\. on data written by children, or on another surface, e\.g\. on paper, leads to very different input signals\. On the one hand, children’s handwriting are of variable speed depending of the assertiveness in the handwriting gesture\. On the other hand, handwriting on paper produces noisier signals due to friction than when the user writes on a tablet\. This leads us to consider a domain adaptation method for dealing with the different domains of data\.

In this context, we present a domain adaptation approach designed to enhance the adaptability of our model to deal with the different data sources\. Initially trained on data acquired from tablets by adults, our model aims to effectively handle a additional types of data: handwriting acquired from tablets by children\. In fact, while collecting labelled adult data on a tablet is not a problem, it’s more complicated to collect data from children \(contact a school\)\. This approach is expected to broaden the applicability of handwriting trajectory reconstruction, making it more versatile and robust across different user groups\. To our knowledge, no domain adaptation method addresses handwriting trajectory reconstruction from different sources, e\.g\. from adults vs children\.

## 2Related works

### 2\.0\.1Handwriting Trajectory Reconstruction

Traditional approaches\[[5](https://arxiv.org/html/2607.26736#bib.bib5),[7](https://arxiv.org/html/2607.26736#bib.bib4)\], which do not rely on deep learning, often struggle with sensor noise and error accumulation\. These methods typically involve a series of preprocessing steps, such as applying low\-pass filters to remove high\-frequency noise and using coordinate transformation matrices to adjust for sensor orientation\. However, the cumulative effect of noise and small errors during these preprocessing steps can lead to significant inaccuracies in the reconstructed trajectories over time\.

Deep learning methods offer a more advanced approach to handling IMU data by learning and adapting to noise patterns, potentially providing more accurate trajectory reconstructions\. For instance,\[[6](https://arxiv.org/html/2607.26736#bib.bib2)\]applied multi\-task learning for joint classification and trajectory reconstruction\.\[[4](https://arxiv.org/html/2607.26736#bib.bib6)\]explored magnetic signals, while\[[10](https://arxiv.org/html/2607.26736#bib.bib1)\]addressed the Stabilo Digipen, using linear interpolation to align pen and tablet signals as preprocessing in training\. This alignment is crucial for matching sequences of variable sizes and having a point\-to\-point matching for training the network\. A Convolutional Neural Network is used for online handwriting trajectory reconstruction\.

Recently,\[[9](https://arxiv.org/html/2607.26736#bib.bib3)\]improved on\[[10](https://arxiv.org/html/2607.26736#bib.bib1)\]by proposing a complete pipeline for online handwriting trajectory reconstruction \(Fig[1](https://arxiv.org/html/2607.26736#S2.F1)\)\. Specifically, they used a specific preprocessing based on the Dynamic Time Warping \(DTW\) to align the ground truth with the input signal, which has the advantage of preserving the dynamics of handwriting, unlike the linear interpolation used in\[[10](https://arxiv.org/html/2607.26736#bib.bib1)\]and use a TCN model to extract local context information\.

![Refer to caption](https://arxiv.org/html/2607.26736v1/x1.png)Figure 1:Pipeline of handwriting reconstruction\.
### 2\.0\.2Domain adaptation methods

Domain adaptation \(DA\) adjusts a model trained on one domain \(source\) to perform well on a different but related domain \(target\), mainly when labeled data are rare or unavailable in the target domain\. The goal is to minimize domain shift, or differences in data distribution, between the source and target domains\. DA can be:

- •Supervised: Labeled data from the target domain are available\.
- •Semi\-Supervised: Both labeled and unlabeled data from the target domain are available\.
- •Unsupervised: Only unlabeled data from the target domain are available\.

Domain adaptation approaches can be broadly categorized into divergence\-based, adversarial\-based, and reconstruction\-based methods\. Divergence\-based DA minimizes statistical differences between source and target distributions using measures like Maximum Mean Discrepancy \(MMD\) or Kullback\-Leibler \(KL\) divergence\[[8](https://arxiv.org/html/2607.26736#bib.bib8)\]\. These methods focus on reducing the distribution gap to improve model performance on the target domain\. Adversarial\-based DA, inspired by Generative Adversarial Networks \(GAN\), employs a discriminator and feature extractor to create domain\-indistinguishable features through adversarial training\[[1](https://arxiv.org/html/2607.26736#bib.bib7)\]\. This approach leverages the power of adversarial learning to align features from different domains, making them indistinguishable to a domain classifier\. Reconstruction\-based DA, uses autoencoders to maintain reconstruction ability across domains, thereby creating robust, domain\-agnostic feature representations\[[2](https://arxiv.org/html/2607.26736#bib.bib9)\]\. By ensuring that features can be accurately reconstructed regardless of the domain, these methods enhance the model’s robustness and adaptability\. Together, these approaches help models generalize better to target domains by aligning data distributions, creating domain\-invariant features, or ensuring robust feature representations, thereby improving performance and reliability in diverse settings\.

## 3DANN\-based method for handwriting reconstruction

To get a shared feature space for different data sources, we explore a an adversarial\-based DA method, named Domain\-Adversarial Training of Neural Networks \(DANN\)\.

### 3\.0\.1Motivation

The variability of handwriting sources complicates the task of handwriting reconstruction\. Children handwriting poses a unique challenge due to the ongoing development of graphomotor skills, resulting in dynamic and inconsistent handwriting patterns as children learn to write \(Fig\. 2\)\. This translates into longer signal sequences than adults and a wider range of possible values \(Fig\.[3](https://arxiv.org/html/2607.26736#S3.F3)\)\. This difference makes it difficult the handwriting reconstruction using a model trained on tablet\-acquired data\.

![Refer to caption](https://arxiv.org/html/2607.26736v1/Images/visu_data_3_v2.png)

Figure 2:Visualization of the x component of the Digipen’s rear accelerometer over time in seconds, from left to the right: a F from adult on tablet, a F from children on tablet\.We notice that the same pattern is written but not with the same intensity due to the different level of automation of handwriting \(adult vs child\)![Refer to caption](https://arxiv.org/html/2607.26736v1/Images/visu_data_2_v2.png)

Figure 3:Data visualization with the Multidimensional Scaling \(MDS\) method, we can see that children’s data on tablets \(in red\) takes on a wider range of values due to their handwriting which is still being learned\.
### 3\.0\.2DANN as a solution

Domain\-Adversarial Training of Neural Networks\[[1](https://arxiv.org/html/2607.26736#bib.bib7)\]are specialized neural network architectures designed to address the challenge of domain shift, where a model trained on one domain \(source\) is expected to perform well on a different but related domain \(target\)\. These networks operate by learning features that are domain\-invariant, meaning they are useful and generalizable across both domains\. This is typically achieved through a shared feature extractor on which additional components are built: a domain classifier and a task\-specific classifier\. The domain classifier is trained to determine the domain of the input data, whereas the task\-specific classifier focuses on predicting the label of the dedicated task\. In our context, the task\-specific classifier is trained to reconstruct the handwriting trajectory\.

Training involves a twist \(Fig\.[4](https://arxiv.org/html/2607.26736#S3.F4)\): the domain classifier’s gradients are reversed during backpropagation, which encourages the feature extractor to generate features that are indistinguishable between domains, thus fooling the domain classifier\. This technique, known as adversarial training, helps the network to minimize the representation gap between the source and target domains, leading to better performance on the target domain without requiring extensive labeled data from it\.

### 3\.0\.3Application

In this work, the reconstruction part of the DANN is the TCN\-based network from\[[9](https://arxiv.org/html/2607.26736#bib.bib3)\]which is currently the state\-of\-the art model for handwriting reconstruction from the Digipen sensors \(Fig\.[5](https://arxiv.org/html/2607.26736#S3.F5)\)\. We named it baseline model \(BM\) in the following\. Then we have slice the baseline model as follows: the 4 blocks of the non\-causal TCN is the feature extractor \(in green in Fig\.[4](https://arxiv.org/html/2607.26736#S3.F4)\)\. Each TCN block is composed of 2 convolutions with dilation 1 and 2 respectively and a kernel size of 3\. The next two dense layers refer to the label predictor \(in blue in Fig\.[4](https://arxiv.org/html/2607.26736#S3.F4)\) which in our context corresponds to the trajectory reconstruction\. The domain classifier \(in pink in the Figure[4](https://arxiv.org/html/2607.26736#S3.F4)\), is made up of a max polling layer followed by two dense layers\.

In our case study, we aim to transition from adult handwriting on a tablet \(source\) to children handwriting on a tablet \(target\)\. During training, we provide mixed matches to two model branches: \(1\) the feature extractor and the label predictor \(green \+ blue in Fig\.[4](https://arxiv.org/html/2607.26736#S3.F4)\) pre\-trained on adult data, and \(2\) a feature extractor and domain classifier \(green \+ pink in Fig\.[4](https://arxiv.org/html/2607.26736#S3.F4)\)\.

![Refer to caption](https://arxiv.org/html/2607.26736v1/x2.png)Figure 4:DANN from\[[1](https://arxiv.org/html/2607.26736#bib.bib7)\]\.![Refer to caption](https://arxiv.org/html/2607.26736v1/x3.png)Figure 5:TCN model from\[[9](https://arxiv.org/html/2607.26736#bib.bib3)\]\. Also named baseline model \(BM\)\.

## 4Experimental results

We experiment our approach on two datasets, one for adult tablet data \(9629 samples\), another for children tablet data \(3910 samples\)\. Each datasets contains characters, words, word groups, equations, and shapes\. Following\[[9](https://arxiv.org/html/2607.26736#bib.bib3)\], we compute the Fréchet distance to evaluate the quality of reconstruction on children’s characters\. We trained a DANN with children and adult data, and compared it qualitatively \(Fig\.[6](https://arxiv.org/html/2607.26736#S4.F6)\) and quantitatively \(Table[1](https://arxiv.org/html/2607.26736#S4.T1)\) to the following methods: baseline model trained on adult data and fine\-tuned on children data and the baseline model trained from scratch on children data\.

Table 1:Comparison of reconstruction methods between training from scratch, fine tuning and domain adaptation method on children test data with the Fréchet distance\.![Refer to caption](https://arxiv.org/html/2607.26736v1/Images/Result_2.png)Figure 6:Comparison of reconstruction methods on children test data, on the first line the ground truth, on the second the Baseline Model trained on adult data, on the third the Baseline Model trained on child data, then the Baseline Model trained on adult data and fine\-tuned on child data, on the last line the DANN pretrain on adult data\.Table 2:Comparison of reconstruction methods between training from scratch, fine tuning and domain adaptation method on adult test data with the Fréchet distance\.Table[1](https://arxiv.org/html/2607.26736#S4.T1)shows that methods integrating children data perform the best overall\. Specifically, all the methods trained on children’s data improve the Baseline Model trained only on adult data\. The advantage of DANN is that it keeps a common representation of adult and children’s features, and table[2](https://arxiv.org/html/2607.26736#S4.T2)shows that it performs well in both areas, making it a 2\-in\-1 solution\. Unlike from scratch and fine tuning models \(Table[1](https://arxiv.org/html/2607.26736#S4.T1)\), which lose out on performance in the domain where they are not learned \(Table[2](https://arxiv.org/html/2607.26736#S4.T2)\)\. Regarding the qualitative analysis \(Fig\.[6](https://arxiv.org/html/2607.26736#S4.F6)\), we observe that the trajectory reconstruction using the DANN is quite satisfactory and it seems closest to the ground truth than using the other approaches on those examples\. The unique representation shared by the two data sources seems help the model the model in the trajectory reconstruction, especially on the hovering part\. Additionally, this work is still in progress and the DANN has potential for further improvement\.

## 5Conclusion

This paper shows the benefits of retaining knowledge from one domain \(adults on tablet\) and moving on to a second \(children on tablet\)\. Our areas for improvement include optimizing the lambda parameter to adaptively manage the weight of each branch over time, which could enhance DANN performance\. Another area for improvement involves creating DANN batches and studying factors such as padding\. We will also investigate domain adaptation from data acquired on tablet to data written on paper\.

\{credits\}

### 5\.0\.1Acknowledgements

This project is financed by the KIHT French\-German bilateral ANR\-21\-FAI2\-0007\-01 project and these four partners, IRISA, KIT, Learn & Go and Stabilo\. This work was performed using HPC resources from GENCI\-IDRIS \(Grant 2021\-AD011013148\)

### 5\.0\.2\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]Y\. Ganin, E\. Ustinova, H\. Ajakan, P\. Germain, H\. Larochelle, F\. Laviolette, M\. Marchand, and V\. Lempitsky\(2016\)Domain\-adversarial training of neural networks\.External Links:1505\.07818Cited by:[§2\.0\.2](https://arxiv.org/html/2607.26736#S2.SS0.SSS2.p1.2),[Figure 4](https://arxiv.org/html/2607.26736#S3.F4),[§3\.0\.2](https://arxiv.org/html/2607.26736#S3.SS0.SSS2.p1.1)\.
- \[2\]M\. Ghifary, W\. B\. Kleijn, M\. Zhang, D\. Balduzzi, and W\. Li\(2016\)Deep reconstruction\-classification networks for unsupervised domain adaptation\.External Links:1607\.03516Cited by:[§2\.0\.2](https://arxiv.org/html/2607.26736#S2.SS0.SSS2.p1.2)\.
- \[3\]T\. Harbaum, A\. Serdyuk, F\. Kreß, T\. Hamann, J\. Barth, P\. Kämpf, F\. Imbert, Y\. Soullard, R\. Tavenard, E\. Anquetil, and J\. Delahaie\(2024\-03\)KIHT: Kaligo\-based Intelligent Handwriting Teacher\.InDATE 2024,Valencia, Spain\.Cited by:[§1](https://arxiv.org/html/2607.26736#S1.p1.1)\.
- \[4\]Y\. Liu, K\. Huang, X\. Song,et al\.\(2020\)MagHacker: eavesdropping on stylus pen writing via magnetic sensing from commodity mobile devices\.InProceedings of the 18th International Conference on Mobile Systems, Applications, and Services,MobiSys ’20\.External Links:ISBN 9781450379540,[Link](https://doi.org/10.1145/3386901.3389030),[Document](https://dx.doi.org/10.1145/3386901.3389030)Cited by:[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p2.1)\.
- \[5\]T\. Miyagawa, Y\. Yonezawa, K\. Itoh, and M\. Hashimoto\(2000\)Handwritten pattern reproduction using pen acceleration and angular velocity\.IEICE Trans\.\.Cited by:[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p1.1)\.
- \[6\]F\. Ott, D\. Rügamer, L\. Heublein,et al\.\(2022\)Joint classification and trajectory regression of online handwriting using a multi\-task learning approach\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 266–276\.Cited by:[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p2.1)\.
- \[7\]T\-Y\. Pan, C\-H\. Kuo, and M\-C\. Hu\(2016\)A noise reduction method for imu and its application on handwriting trajectory reconstruction\.In2016 IEEE International Conference on Multimedia & Expo Workshops \(ICMEW\),Vol\.,pp\. 1–6\.External Links:[Document](https://dx.doi.org/10.1109/ICMEW.2016.7574685)Cited by:[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p1.1)\.
- \[8\]J\. Shen, Y\. Qu, W\. Zhang, and Y\. Yu\(2018\)Wasserstein distance guided representation learning for domain adaptation\.External Links:1707\.01217Cited by:[§2\.0\.2](https://arxiv.org/html/2607.26736#S2.SS0.SSS2.p1.2)\.
- \[9\]W\. Swaileh, F\. Imbert, Y\. Soullard, R\. Tavenard, and É\. Anquetil\(2023\)Online handwriting trajectory reconstruction from kinematic sensors using temporal convolutional network\.International Journal on Document Analysis and Recognition \(IJDAR\),pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2607.26736#S1.p2.1),[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p3.1),[Figure 5](https://arxiv.org/html/2607.26736#S3.F5),[§3\.0\.3](https://arxiv.org/html/2607.26736#S3.SS0.SSS3.p1.1),[§4](https://arxiv.org/html/2607.26736#S4.p1.1)\.
- \[10\]M\. Wehbi, D\. Luge, T\. Hamann,et al\.\(2022\)Surface\-free multi\-stroke trajectory reconstruction and word recognition using an imu\-enhanced digital pen\.Sensors\.Cited by:[§1](https://arxiv.org/html/2607.26736#S1.p2.1),[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p2.1),[§2\.0\.1](https://arxiv.org/html/2607.26736#S2.SS0.SSS1.p3.1)\.