On-Device Neural Architecture Search

# On-Device Neural Architecture Search
Source: [https://arxiv.org/html/2606.24900](https://arxiv.org/html/2606.24900)
Andrea Mattia Garavagno123, Edoardo Ragusa1, Paolo Gastaldo1, Antonio Frisoli23, and Claudio Loconsole43

###### Abstract

This paper proposes a new approach to near\-sensor computing, in which a lightweight Neural Architecture Search \(NAS\) is performed directly on the deployment device to find the best tiny neural architecture for analyzing the real\-time data acquired through sensors\. This new adaptation capability can be particularly useful in the case of human\-machine interfaces for which the neural network analyzing the biometrical data can be re\-designed each time the user changes, after a guided data collection procedure, fighting the typical data variations between individuals on a new level\. To implement the proposed approach a new NAS has been designed and then validated on the Italian Sign Language dataset \(ISL\), a collection of surface electromyography \(sEMG\) signals of the signs of the Italian alphabet, using several embedded systems\. Moreover, further validation on the Case Western Reserve University dataset \(CWRU\), a benchmark for intelligent fault diagnosis, is presented to suggest another possible application of the proposed approach\. When run on a Raspberry Pi 4, the proposed NAS performs beyond the state of the art proposing a tiny neural architecture having 0\.63 times less RAM occupancy and 5\.96 percentage points of more accuracy in the case of the ISL dataset; and 0\.44 times less RAM occupancy and 0\.2 percentage points of more accuracy in the case of the CWRU dataset\.

###### Index Terms:

Neural Architecture Search, Human\-Machine Interfaces, Intelligent Fault Diagnosis, TinyML

This work has been accepted for publication in the proceedings of the 2025 IEEE International Conference on Consumer Electronics \(ICCE\)\. © 2025 IEEE\. Personal use of this material is permitted\. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works\. Final publication is available at[https://doi\.org/10\.1109/ICCE63647\.2025\.10929801](https://doi.org/10.1109/ICCE63647.2025.10929801)\.

## IIntroduction

The present work proposes a new approach to near\-sensor computing\[[21](https://arxiv.org/html/2606.24900#bib.bib1)\], in which a lightweight Neural Architecture Search \(NAS\) is performed directly on the deployment device to find the best tiny neural architecture for analyzing the real\-time data acquired through sensors\. In this scenario, the sensor node can collect labelled data from human feedback, find a tiny neural architecture tailored to the collected data thanks to a lightweight search procedure, and then use it to analyze the data from the attached sensors in real\-time\. This novel near\-sensor computing approach has the potential to enhance continual learning\[[5](https://arxiv.org/html/2606.24900#bib.bib2)\], allowing not only weights but also the architecture itself to co\-adapt with the data drift\. The new adaptive capability has the potential to enhance a next\-generation of Human\-Machine Interfaces \(HMIs\) that analyze real\-time data from biometric sensors\. Each time the user changes, the tiny neural architecture analyzing biometric data is redesigned after a guided data collection procedure, addressing the typical data variations between individuals\[[12](https://arxiv.org/html/2606.24900#bib.bib16)\]on a new level\.

The same approach could support applications typical of near\-sensor computing such as e\-health, personalized advertisement, structural health monitoring, and intelligent fault diagnosis\. In the latter use case, an application\-agnostic intelligent fault diagnosis system \(IFDS\) could be designed in\-house, using the proposed approach, and then deployed on the field\. Personnel could manually tamper the machine to be monitored to gather fault data with the available sensors; then collected data feed the application\-agnostic IFDS, which automatically designs a custom neural architecture for classifying faults in real\-time\. In this way, a single IFDS can satisfy the requirements of multiple applications, as the neural architecture can adapt to the specific set of sensors used to acquire fault data\.

## IIRelated Work

The quest towards implementing deep learning on edge devices started with the inference phase\[[6](https://arxiv.org/html/2606.24900#bib.bib3),[9](https://arxiv.org/html/2606.24900#bib.bib4),[8](https://arxiv.org/html/2606.24900#bib.bib5)\], proceeded with the training phase\[[10](https://arxiv.org/html/2606.24900#bib.bib8),[7](https://arxiv.org/html/2606.24900#bib.bib6),[19](https://arxiv.org/html/2606.24900#bib.bib7)\], and is leaning towards the design phase\[[4](https://arxiv.org/html/2606.24900#bib.bib10),[3](https://arxiv.org/html/2606.24900#bib.bib9)\]given the recent results provided by hardware\-aware NAS \(HW\-NAS\)\[[14](https://arxiv.org/html/2606.24900#bib.bib11),[16](https://arxiv.org/html/2606.24900#bib.bib12),[15](https://arxiv.org/html/2606.24900#bib.bib13),[20](https://arxiv.org/html/2606.24900#bib.bib14)\]\.

Garavagno et al\.\[[4](https://arxiv.org/html/2606.24900#bib.bib10)\]reformulated the optimization problem of HW\-NAS by including in the set of constraints the resources available in the device performing the search procedure\. Such constraints add to the constraints imposed by the deployment device, which usually characterize the classical versions of HW\-NAS; this in turn further reduces the search space\. The goal is to avoid the evaluation of architectures not fitting the resources available in the device performing the search procedure; as such, one can enable practical implementation of HW\-NAS on consumer devices\. In\[[3](https://arxiv.org/html/2606.24900#bib.bib9)\], Garavagno et al\. further developed the idea of running the search procedure on embedded devices, proposing an adaptive search strategy, which tunes the length of the evaluation process according to the available computational resources\.

The objective of\[[4](https://arxiv.org/html/2606.24900#bib.bib10)\]and\[[3](https://arxiv.org/html/2606.24900#bib.bib9)\]was to automatically design custom neural architectures inside a gateway, one for each type of sensor node connected to it\. Such approach preserves privacy, as each device cab exploit the locally acquired data, without involving any third party\. Conversely, this paper aims to propose a different approach, where the NAS is expected to run on the deployment device\. Another main difference is that the NAS implemented in\[[4](https://arxiv.org/html/2606.24900#bib.bib10)\]and\[[3](https://arxiv.org/html/2606.24900#bib.bib9)\]produce models meant to analyze images, while the NAS proposed in the present document produces models meant to analyze time series\.

## IIIProposed Neural Architecture Search

### III\-ASearch Space

The proposed NAS organizes the input time series from different sensors as a bi\-dimensional matrix where each row contains the samples belonging to one sensor\. Hence obtaining a matrix havingns​e​n​s​o​r​sn\_\{sensors\}rows andns​a​m​p​l​e​sn\_\{samples\}columns\. In such a way the resulting matrix can be analyzed by a bi\-dimensional convolutional neural network built according to the following rules: the input is a base cell, composed of a bi\-dimensional convolutional layer havingkkkernels, followed by a batch normalization layer and a ReLu activation\. Thencccells, composed of a bi\-dimensional max pooling layer followed by a bi\-dimensional convolutional layer, a batch normalization layer, and a ReLu activation, are stacked upon the base cell\. The number of kernels used in each cell is computed using the formula presented in \([1](https://arxiv.org/html/2606.24900#S3.E1)\)\. The extracted features are then reduced by a global average pooling layer and finally classified by a fully connected layer, having Softmax as the activation function\.

nc=\{ki​fc=0⌈\(2−∑i=1c−12−i\)⋅nc−1⌉i​fc≥1\\displaystyle n\_\{c\}=\(1\)
To preserve the input matrix height, kernels ofns​e​n​s​o​r​sn\_\{sensors\}height are used, and stride 1 is always applied to the height dimension\. Instead, on the rows dimension, a stride equal to two and a kernel as wide as two samples are always applied in the case of pooling layers, this is done to halve the time series length each time, whereas, in the case of convolution, a stride equal to one is applied alongside a kernel width of three\. All the convolutional layers employ zero padding to maintain the input’s resolution intact\.

### III\-BOptimization Problem

Classical HW\-NAS aims to find the best neural architecture for the target hardware in the search space\. This leads to a constrained optimization problem where the objective function describes the metric adopted to evaluate candidate solutions and the constraints set the boundaries of the search according to the deployment hardware chosen\. Garavagno et al\.\[[4](https://arxiv.org/html/2606.24900#bib.bib10)\]proposed to include in the constraints the resources available on the hardware performing the search, which usually were considered unbounded\. As a result, HW\-NAS can run on a resource\-constrained device\.

In the case of the approach proposed in the present paper, the device performing the search procedure coincides with the device used for deployment\. Therefore, the constraints refer to a single target hardware\. Indeed, as the goal is to deal with real\-time applications \(see Sec\.[I](https://arxiv.org/html/2606.24900#S1)\), a constraint on the inference time is also added\. The network’s input size is considered fixed during the search since it depends on the user’s needs\. Hence, the search variablex=\(k,c\)x=\(k,c\)is defined without including the network’s input size\. This leads to problem formulationPP

P:\{max⁡f​\(x\)ϕM​\(x\)≤ξM,ϕT​\(x\)≤ξTξM,ξT\>0\\displaystyle P:\(2\)
where functionffreturns the maximum validation accuracy obtained during the evaluation phase, functionϕM\\phi\_\{M\}returns the memory occupancy during training and functionϕT\\phi\_\{T\}returns the time consumed by one inference\. ParametersξM\\xi\_\{M\}andξT\\xi\_\{T\}respectively represent the upper bounds for the candidate’s memory occupancy during training and time per inference\.

### III\-CSearch Strategy

In the proposed approach the developed NAS runs on constrained devices; thus, there is a need for a lightweight search strategy\. Accordingly, the derivative\-free search strategy proposed in\[[2](https://arxiv.org/html/2606.24900#bib.bib15)\]has been reformulated to solve the optimization problem presented in section[III\-B](https://arxiv.org/html/2606.24900#S3.SS2)\. In detail, the phase in which a candidate network is evaluated has been reorganized as a child process\. As such, its parent can monitor if the child process has been killed by the operative system because of memory depletion; this in turn means that the candidate network violated the constraint on the available memory\. Moreover, the evaluation phase now measures the inference time of the candidate network to determine if such solution satisfies the time constraint\.

## IVExperimental Validation

To provide experimental validation of the proposed approach, the developed NAS has been run on three different embedded devices, namely: the Raspberry Pi 4, the Raspberry Pi 3, and the Raspberry Pi Zero 2 W, which are often used in wearable and intelligent systems\[[11](https://arxiv.org/html/2606.24900#bib.bib17)\]\. Table[I](https://arxiv.org/html/2606.24900#S4.T1)provides the hardware specifications for each device\.

Embedded DeviceSoCRAMRaspberry PiBroadcom\[GiB\]4 Model BBCM271143 Model BBCM28371Zero 2 WBCM2710A10\.5TABLE I:Specifications of the embedded devices targeted by this work\.TABLE II:Resulting Architectures VS Reference ArchitecturesTwo datasets have been selected according to the use cases proposed in section[I](https://arxiv.org/html/2606.24900#S1): the Italian Sign Language \(ISL\) dataset\[[17](https://arxiv.org/html/2606.24900#bib.bib18)\]to demonstrate that the proposed approach can be effectively used to automatically design neural architectures for human\-machine interfaces, and the Case Western Reserve University \(CWRU\) dataset\[[18](https://arxiv.org/html/2606.24900#bib.bib19)\]to demonstrate that the proposed approach can be also applied to design neural architectures for intelligent fault diagnosis systems\.

The ISL dataset\[[17](https://arxiv.org/html/2606.24900#bib.bib18)\]contains surface electromyography \(sEMG\) and inertial measurements regarding the 26 letters of the hand sign Italian alphabet\. 8 electrodes of the Myo Gesture Control Armband have been used to acquire the sEMG data at 200 Hz for 2 seconds\. 30 samples for each sign have been collected, for a total of 780\. To compare the resulting architecture with the one proposed by Pau et al\.\[[13](https://arxiv.org/html/2606.24900#bib.bib20)\], only the sEMG data regarding A, C, H, J, K, M, N, P, S, W, X signs were used, for a total of 11 classes and 330 samples\. The test split of 20% which brought the best results in terms of accuracy score in\[[13](https://arxiv.org/html/2606.24900#bib.bib20)\]has been adopted\.

The CWRU dataset\[[18](https://arxiv.org/html/2606.24900#bib.bib19)\]contains vibration data for normal and faulty bearings\. To compare the resulting architecture with the work proposed by Chen et al\.\[[1](https://arxiv.org/html/2606.24900#bib.bib21)\], the vibration data from the acceleration sensor at the driving end were selected at 12 kHz sampling frequency\. Absence of faults and 9 types of faults, namely ball and inner raceway at 0\.007, 0\.014, 0\.021 inches, and outer raceway oriented at 3, 6, and 12 o’clock at 0\.007 inches were selected at 1797 rpm, i\.e\. at 0 HP\. 100 of samples of length 1024 are collected for each class, for a total of 1000 samples\.

In both cases, an upper bound of 0\.33 seconds, equivalent to 30 inferences per second has been set to the latency, as the constraint of real\-time\. All models are trained using quantization\-aware training and then quantized to 8\-bit\. Architectures are evaluated for500500epochs with a learning rate of10−310^\{\-3\}and a batch size of256256\.

Table[II](https://arxiv.org/html/2606.24900#S4.T2)compares the resulting architectures with the state\-of\-the\-art work for each dataset, and also reports the time and the energy needed by the NAS to obtain the result\. RAM and Flash occupancies have been measured using X\-CUBE\-AI’s stm32tflm executable, which reports resource usage when using Tensorflow Lite for Microcontrollers as runtime\. Whereas, latency is measured by running the model 1000 times on the chosen hardware using Tensorflow Lite runtime\.

As can be seen, the resulting architectures’ footprint decrease according to the resources available to the embedded device performing the search\. The main cause is the decrease in the memory available for training, which further restricts the search space, not allowing the algorithm to explore solutions with larger footprints\. The latter decrease also causes a loss in test accuracy\. However, it does not cause an increase in the latency: even if the processor’s computational power decreases, the footprint reduction overcomes the decrease, providing lower latencies\. In any case, the proposed approach provides significantly smaller footprints than the reference work for both datasets, whatever execution platform is considered\. Still, the test accuracy provided surpasses the one offered by the reference works in the case of the Raspberry Pi 4 as an execution platform while remaining comparable in the case of the Raspberry Pi 3\. Only in the case of Raspberry Pi Zero 2, the accuracy score obtained is drastically lower, especially in the case of the ISL dataset, in which no candidate architecture provides satisfactory results within such a tight memory constraint for training\. Even if the total power consumption could be provided by a common laptop battery, the search cost of the proposed approach is high, suggesting a run overnight\.

## VConclusion

The present document proposed a new approach to NAS where the search procedure is performed directly on the deployment device\. This novel approach enables a new paradigm where an edge device can collect labelled data from human feedback, find a neural architecture tailored to the collected data, and use it to perform inference in real time\. The latter paradigm has the potential to enhance continual learning, allowing not only weights but also architectural changes to co\-adapt with the data drift\.

The proposed approach has been validated on two different datasets: the Italian Sign Language dataset and the Case Western Reserve University dataset; using three different embedded devices: the Raspberry Pi 4 Mobel B, the Raspberry Pi 3 Mobel B, and the Raspberry Pi Zero 2 W; demonstrating a promising capacity of delivering state\-of\-the\-art architectures not only in the domain of biometrical signals for human\-machine interfaces, but also in the domain of intelligent fault diagnosis\.

The search cost measured during experiments suggests overnight runs\. In the latter scenario, the human\-machine interface can adapt to the new owner overnight, delivering a tailored neural architecture for the real\-time analysis of the user’s biometrical signals\. However, considering that the amount of data collected from a single user would be less than the one of the datasets adopted as benchmark, the search cost could be lower\. For the latter use case, further experiments are needed the asses the real search cost\.

## Acknowledgment

Project funded under the National Recovery and Resilience Plan \(NRRP\), Mission 4 Component 2 Investment 1\.1 \- Call for tender No\. 1409 published on Sept 14, 2022 by the Italian Ministry of University and Research \(MUR\) funded by the European Union – NextGenerationEU \- Project Title ”LEARN \- muLtimodal Edge computing\-bAsed weaRable exoskeletoNs for assistance in daily life” – CUP: D53D23016190001, D53D23016200001, J53D23014090001 \- Grant Assignment Decree No\. 1383 adopted on September 01, 2023 by the Italian Ministry of University and Research \(MUR\)\.

![[Uncaptioned image]](https://arxiv.org/html/2606.24900v1/images/ack2.png)

## References

- \[1\]\(2020\)An improved fault diagnosis using 1d\-convolutional neural network model\.Electronics10\(1\),pp\. 59\.Cited by:[TABLE II](https://arxiv.org/html/2606.24900#S4.T2.8.8.11.1),[§IV](https://arxiv.org/html/2606.24900#S4.p4.1)\.
- \[2\]A\. M\. Garavagno, D\. Leonardis, and A\. Frisoli\(2024\)ColabNAS: obtaining lightweight task\-specific convolutional neural networks following occam’s razor\.Future Generation Computer Systems152,pp\. 152–159\.Cited by:[§III\-C](https://arxiv.org/html/2606.24900#S3.SS3.p1.1)\.
- \[3\]A\. M\. Garavagno, E\. Ragusa, A\. Frisoli, and P\. Gastaldo\(2024\)An affordable hardware\-aware neural architecture search for deploying convolutional neural networks on ultra\-low\-power computing platforms\.IEEE Sensors Letters\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1),[§II](https://arxiv.org/html/2606.24900#S2.p2.1),[§II](https://arxiv.org/html/2606.24900#S2.p3.1)\.
- \[4\]A\. M\. Garavagno, E\. Ragusa, A\. Frisoli, and P\. Gastaldo\(2024\)Running hardware\-aware neural architecture search on embedded devices under 512mb of ram\.In2024 IEEE International Conference on Consumer Electronics \(ICCE\),pp\. 1–2\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1),[§II](https://arxiv.org/html/2606.24900#S2.p2.1),[§II](https://arxiv.org/html/2606.24900#S2.p3.1),[§III\-B](https://arxiv.org/html/2606.24900#S3.SS2.p1.1)\.
- \[5\]R\. Hadsell, D\. Rao, A\. A\. Rusu, and R\. Pascanu\(2020\)Embracing change: continual learning in deep neural networks\.Trends in cognitive sciences24\(12\),pp\. 1028–1040\.Cited by:[§I](https://arxiv.org/html/2606.24900#S1.p1.1)\.
- \[6\]B\. Jacob, S\. Kligys, B\. Chen, M\. Zhu, M\. Tang, A\. Howard, H\. Adam, and D\. Kalenichenko\(2018\)Quantization and training of neural networks for efficient integer\-arithmetic\-only inference\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 2704–2713\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[7\]J\. Lee and H\. Yoo\(2021\)An overview of energy\-efficient hardware accelerators for on\-device deep\-neural\-network training\.IEEE Open Journal of the Solid\-State Circuits Society1,pp\. 115–128\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[8\]J\. Lin, W\. Chen, H\. Cai, C\. Gan, and S\. Han\(2021\)Mcunetv2: memory\-efficient patch\-based inference for tiny deep learning\.arXiv preprint arXiv:2110\.15352\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[9\]J\. Lin, W\. Chen, Y\. Lin, C\. Gan, S\. Han,et al\.\(2020\)Mcunet: tiny deep learning on iot devices\.Advances in Neural Information Processing Systems33,pp\. 11711–11722\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[10\]J\. Lin, L\. Zhu, W\. Chen, W\. Wang, C\. Gan, and S\. Han\(2022\)On\-device training under 256kb memory\.Advances in Neural Information Processing Systems35,pp\. 22941–22954\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[11\]S\. E\. Mathe, H\. K\. Kondaveeti, S\. Vappangi, S\. D\. Vanambathina, and N\. K\. Kumaravelu\(2024\)A comprehensive review on applications of raspberry pi\.Computer Science Review52,pp\. 100636\.Cited by:[§IV](https://arxiv.org/html/2606.24900#S4.p1.1)\.
- \[12\]J\. Ortega\-Garcia, J\. Bigun, D\. Reynolds, and J\. Gonzalez\-Rodriguez\(2004\)Authentication gets personal with biometrics\.IEEE signal processing magazine21\(2\),pp\. 50–62\.Cited by:[§I](https://arxiv.org/html/2606.24900#S1.p1.1)\.
- \[13\]D\. P\. Pau and M\. D\. Randriatsimiovalaza\(2023\)Electromyography gestures sensing with deeply quantized neural networks\.In2023 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering \(MetroXRAINE\),pp\. 711–716\.Cited by:[TABLE II](https://arxiv.org/html/2606.24900#S4.T2.2.2.11.1),[§IV](https://arxiv.org/html/2606.24900#S4.p3.1)\.
- \[14\]E\. Ragusa, F\. Zonzini, P\. Gastaldo, and L\. De Marchi\(2024\)Combining compressed sensing and neural architecture search for sensor\-near vibration diagnostics\.IEEE Transactions on Industrial Informatics\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[15\]J\. Rala Cordeiro, A\. Raimundo, O\. Postolache, and P\. Sebastião\(2021\)Neural architecture search for 1d cnns—different approaches tests and measurements\.Sensors21\(23\),pp\. 7990\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[16\]M\. Risso, A\. Burrello, F\. Conti, L\. Lamberti, Y\. Chen, L\. Benini, E\. Macii, M\. Poncino, and D\. J\. Pagliari\(2022\)Lightweight neural architecture search for temporal convolutional networks at the edge\.IEEE Transactions on Computers72\(3\),pp\. 744–758\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[17\]P\. Sernani, I\. Pacifici, N\. Falcionelli, S\. Tomassini, A\. F\. Dragoni,et al\.\(2021\)Italian sign language alphabet recognition from surface emg and imu sensors with a deep neural network\.\.InRTA\-CSIT,pp\. 74–83\.Cited by:[§IV](https://arxiv.org/html/2606.24900#S4.p2.1),[§IV](https://arxiv.org/html/2606.24900#S4.p3.1)\.
- \[18\]W\. A\. Smith and R\. B\. Randall\(2015\)Rolling element bearing diagnostics using the case western reserve university data: a benchmark study\.Mechanical systems and signal processing64,pp\. 100–131\.Cited by:[§IV](https://arxiv.org/html/2606.24900#S4.p2.1),[§IV](https://arxiv.org/html/2606.24900#S4.p4.1)\.
- \[19\]Y\. Yang, G\. Li, and R\. Marculescu\(2023\)Efficient on\-device training via gradient filtering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 3811–3820\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[20\]L\. L\. Zhang, Y\. Yang, Y\. Jiang, W\. Zhu, and Y\. Liu\(2020\)Fast hardware\-aware neural architecture search\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,pp\. 692–693\.Cited by:[§II](https://arxiv.org/html/2606.24900#S2.p1.1)\.
- \[21\]F\. Zhou and Y\. Chai\(2020\)Near\-sensor and in\-sensor computing\.Nature Electronics3\(11\),pp\. 664–671\.Cited by:[§I](https://arxiv.org/html/2606.24900#S1.p1.1)\.