Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization

# Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization
Source: [https://arxiv.org/html/2607.06922](https://arxiv.org/html/2607.06922)
###### Abstract

Deep learning applications have been widely adopted on edge devices, to mitigate the privacy and latency issues of accessing cloud servers\. Deciding the number of neurons during the design of a deep neural network to maximize performance is not intuitive\. Particularly, many application scenarios are real\-time and have a strict latency constraint, while conventional neural network optimization methods do not directly change the temporal cost of model inference for latency\-critical edge systems\. In this work, we propose a latency\-oriented neural network learning method to optimize models for high accuracy while fulfilling the latency constraint\. For efficiency, we also introduce a universal hardware\-customized latency predictor to optimize this procedure to learn a model that satisfies the latency constraint by only a one\-shot training process\. The experiment results reveal that, compared to state\-of\-the\-art methods, our approach can well\-fit the ‘hard’ latency constraint and achieve high accuracy\. Under the same training settings as the original model and satisfying a 34ms latency constraint on the ImageNet\-100 dataset, we reduce GoogLeNet’s latency from 40\.32ms to 34ms with a 0\.14% accuracy reduction on the NVIDIA Jetson Nano\. When coupled with quantization, our method can be further improved to only 0\.04% drop for GoogLeNet\. On the NVIDIA Jetson TX2, we compress VGG\-19 from 119\.98ms to 34ms and even improve its accuracy by 0\.5%, and we scale GoogLeNet up from 20\.27ms to 34ms and achieve higher accuracy by 0\.78%\. We also open source this framework at[https://github\.com/ntuliuteam/ZeroBN](https://github.com/ntuliuteam/ZeroBN)\.

###### keywords:

deep neural network , latency optimization , edge device , latency prediction , neural network learning , high performance

††journal:Future Generation Computer Systems\\affiliation

\[label1\]organization=School of Computer Science and Engineering, addressline=Nanyang Technological University, postcode=639798, country=Singapore\\affiliation\[label2\]organization=HP\-NTU Digital Manufacturing Corporate Lab, addressline=Nanyang Technological University, postcode=639798, country=Singapore\\affiliation\[label4\]organization=Department of Computer Science, addressline=Norwegian University of Science and Technology, postcode=7491, city=Trondheim, country=Norway\\affiliation\[label3\]organization=HP Inc\., city=Palo Alto, postcode=94304, state=California, country=United States

## 1Introduction

Deep Neural Networks \(DNNs\) have achieved striking success in a variety of applications, including image recognition and object detection\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]\. Meanwhile, as accessing cloud servers hurts data confidentiality and cannot guarantee latency owing to unstable network bandwidth, many DNN applications have been increasingly deployed on edge devices such as autonomous vehicles, healthcare devices and so on\[[2](https://arxiv.org/html/2607.06922#bib.bib2)\]\. Although various model architectures are designed by manual\[[3](https://arxiv.org/html/2607.06922#bib.bib3)\]or by neural architecture search \(NAS\)\[[4](https://arxiv.org/html/2607.06922#bib.bib4)\]to increase the accuracy and improve the efficiency of DNN applications, the emergence of diverse edge devices with different computational capabilities and resources results in a mismatch between existing models and the majority of edge devices\.

Many applications have rigorous response latency requirements, such as self\-driving vehicles and drone tracking\. But complicated models significantly degrade the performance of edge devices, leading to them being unacceptable for latency\-critical systems\. On the other hand, simple models sacrifice too much accuracy to reduce the latency\. Thus, to implement an efficient edge DNN system with the best accuracy and guaranteed latency, DNN models need to contain efficient architectures and appropriate computational costs \(floating\-point operations per second, FLOPs\) to fully utilize the computational resource provided by the underlying hardware\.

Although some works use NAS\[[4](https://arxiv.org/html/2607.06922#bib.bib4)\]to solve this problem, designing new DNN models for each edge device is very time\-consuming and requires significant engineering efforts\[[4](https://arxiv.org/html/2607.06922#bib.bib4),[5](https://arxiv.org/html/2607.06922#bib.bib5)\]\. Therefore, optimizing existing DNN models is more promising and versatile\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]\.Neural network pruning\[[1](https://arxiv.org/html/2607.06922#bib.bib1),[6](https://arxiv.org/html/2607.06922#bib.bib6),[7](https://arxiv.org/html/2607.06922#bib.bib7),[8](https://arxiv.org/html/2607.06922#bib.bib8),[9](https://arxiv.org/html/2607.06922#bib.bib9)\]and Neural network scaling\[[10](https://arxiv.org/html/2607.06922#bib.bib10),[11](https://arxiv.org/html/2607.06922#bib.bib11)\]can change the model complexity by removing redundant parameters from DNNs or extending DNNs with more channels and layers, respectively\. Thus, we borrow the idea from them to improve the model architecture and change the inference latency while keeping high accuracy\.

Nonetheless, these existing methods suffer fromthreeflaws\. Firstly, these methods adopt indirect performance metrics, for example, FLOPs or the number of channels and layers as the optimization target\. These metrics are not hardware\-specific and they cannot accurately reflect the real latency of a DNN model on edge devices\. These present technologies are challenging to optimize models for latency\-critical edge applications, thus a direct DNN optimization strategy for latency is more critical to efficient edge intelligence systems\.

Secondly, when considering latency as the optimization objective, these methods need to deploy models to the device for measurement after making any modifications throughout the whole optimization process\. As demonstrated in Fig\.[1](https://arxiv.org/html/2607.06922#S1.F1)\(green lines\), this measurement process is quite time\-consuming because the interaction with hardware to obtain the latency usually takes minutes\[[12](https://arxiv.org/html/2607.06922#bib.bib12)\], which is prohibitively expensive in a large design space\. Furthermore, the indirect performance targets \(e\.g\., FLOPs\) used by these methods\[[1](https://arxiv.org/html/2607.06922#bib.bib1),[6](https://arxiv.org/html/2607.06922#bib.bib6),[7](https://arxiv.org/html/2607.06922#bib.bib7),[8](https://arxiv.org/html/2607.06922#bib.bib8),[9](https://arxiv.org/html/2607.06922#bib.bib9),[10](https://arxiv.org/html/2607.06922#bib.bib10)\]must be adjusted iteratively to satisfy the latency requirement upon the target edge system\. This aggravates the overhead issue\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x1.png)Figure 1:The workflow of hardware\-aware neural network optimization\. With a latency predictor, the host can avoid time\-consuming on\-device measurements during the process, extremely improving efficiency\.Thirdly, some methods are very time\-consuming and are mainly divided into three parts: pre\-training, optimizing, and re\-training\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]\. A pre\-trained model is needed to identify the importance of different parts of the model, which is used for optimizing procedures to eliminate or add parameters\. Re\-training fine\-tunes the new model to recover accuracy\. As the training procedure is complicated and time\-consuming, the high overhead of network optimization is mostly due to the procedures of pre\-training and re\-training\. Meanwhile, in order to derive a model that meets the optimization target, some prior methods\[[1](https://arxiv.org/html/2607.06922#bib.bib1),[13](https://arxiv.org/html/2607.06922#bib.bib13)\]must do the optimizing\-retraining procedure for several rounds\. This further exacerbates the overhead issue\.

However, previous work\[[14](https://arxiv.org/html/2607.06922#bib.bib14)\]has demonstrated that pre\-training is not a necessary step to obtain an efficient and competitive model\. It has also stated that the architecture, rather than the inherited weight value of significant elements, is more crucial to the efficiency of the final model\. Therefore, an efficient learning method, which directly optimizes the model architecture without a pre\-trained model and re\-training process, is necessary to reduce the time cost\.

This paper aims to develop a comprehensive and hardware\-aware DNN optimization framework which effectively and efficiently fits various DNN models on diverse edge systems\. We propose a neural network optimizing framework to obtain as high accuracy as possible under the latency constraint by only one learning process\. We first design a compact learning scheme, which can condense redundant DNNs to meet a ‘hard’ latency requirement on targeted devices by dynamically zeroizing and recovering Batch Normalization \(BN\) layers\. Based on it, we propose a scaling scheme for simple models to improve their accuracy and avoid the expensive scaling factor search process\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\]\. Thus, this framework is a comprehensive and adaptive learning method that can optimize both complex and simple models\. The primary contributions of this paper are as follows:

- •We introduce a compact model learning method that can extract the optimal DNN architecture to fulfil the ‘hard’ latency constraint and maintain high accuracy\. It applies only aone\-shottraining procedure \(named compact learning\) to avoid the expensive pre\-training/re\-training cost\. The global importance rank of channels and the dynamic Zero\-Recovery process are proposed to extend the exploration space for better architecture\.\(see Section[3\.1](https://arxiv.org/html/2607.06922#S3.SS1)\)
- •We present a machine learning\-based latency predictor, which is embedded into our adaptive learning framework to provide a guideline for each Zero training phase so as to avoid time\-expensive on\-device measurements\. This method can be also generalized to build other hardware metrics predictors\.\(see Section[3\.2](https://arxiv.org/html/2607.06922#S3.SS2)\)
- •We involve a model scaling scheme to better deploy simple models on high\-end edge devices, thereby improving the accuracy and still meeting the latency constraint\. Benefiting from our compact learning can optimize models from both channel and layer, the scaling method does not require the time\-consuming search process for scaling factors\.\(see Section[3\.3](https://arxiv.org/html/2607.06922#S3.SS3)\)
- •We demonstrate that our adaptive learning framework can effectively and efficiently optimize redundant or simple models to suit various edge devices under the specific latency constraint through a one\-shot training process by numerous experiments, surpassing previous methods in terms of both accuracy and efficiency\.\(see Section[4](https://arxiv.org/html/2607.06922#S4)\)

The rest of this paper is organized as follows\. Section[2](https://arxiv.org/html/2607.06922#S2)introduces the background and some related works\. Section[3](https://arxiv.org/html/2607.06922#S3)describes the details of our proposed method, and the experimental results are presented in Section[4](https://arxiv.org/html/2607.06922#S4)\. Finally, we conclude this paper in Section[5](https://arxiv.org/html/2607.06922#S5)\.

## 2Related Work & Background

In this section, we introduce the preliminaries of model optimization and latency prediction, including achievements and some flaws of these related works\.

### 2\.1Neural Network Compression

Many researchers have proposed methods for reducing the high complexity of current complicated DNN models to suit the emerging embedded systems\. These methods include neural network pruning\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]– reducing the number of weights in a model, quantization\[[15](https://arxiv.org/html/2607.06922#bib.bib15)\]– reducing the number of weight bits in a model, knowledge distillation\[[16](https://arxiv.org/html/2607.06922#bib.bib16)\]– modifying the model’s architecture to simplify it\.

#### 2\.1\.1Neural Network Pruning

Recent efforts have been made either on unstructured pruning \(i\.e\., weight pruning\)\[[13](https://arxiv.org/html/2607.06922#bib.bib13),[17](https://arxiv.org/html/2607.06922#bib.bib17)\]or structured pruning \(i\.e\., channel/layer pruning\)\[[1](https://arxiv.org/html/2607.06922#bib.bib1),[6](https://arxiv.org/html/2607.06922#bib.bib6),[7](https://arxiv.org/html/2607.06922#bib.bib7)\]\. Unstructured pruning methods directly delete individual unimportant elements in weight tensors, which can achieve less accuracy loss\. However, they need specific hardware and libraries that support sparse to run pruned models\. In contrast, structured pruning methods are coarse\-grained, which remove entire regular regions \(e\.g\., channel/layer\) of weight tensors, no extra supports are required\. Thus, structure pruning methods are easy to deploy on different kinds of processors\. Since this paper aims to optimize models for universal edge devices, we design an efficient compact model learning framework by eliminating channels and layers\. As shown in Fig\.[2](https://arxiv.org/html/2607.06922#S2.F2), removing layers or channels can reduce the latency\. However, the latency reduction of different model architectures has different tendencies to layer pruning or channel pruning under the same FLOPs\. Thus, we reasonably include both channel pruning and layer pruning into our framework to determine the optimal approach through the learning process\.

Conventional pruning strategies include a typical 3\-stage process: 1\)pre\-training, 2\)pruning, 3\)re\-training\. However, the pre\-training stage is unnecessary to get the efficient final compact model\[[14](https://arxiv.org/html/2607.06922#bib.bib14)\]\. Meanwhile, it is the model architecture, rather than the inherited weight value of essential elements, that is more crucial to the efficiency and accuracy of the final model\[[14](https://arxiv.org/html/2607.06922#bib.bib14)\]\. Therefore, this paper aims to propose an efficient method to find the optimal compact architecture without the pre\-trained model and re\-training step\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x2.png)Figure 2:The relationship between the number of FLOPs and the inference latency by changing the number of channels \(width\) or layers \(depth\) measured on NVIDIA Jetson TX2\.To obtain an efficient model architecture, we should rank the channel importance across layers \(i\.e\., global channel importance ranking\) to preserve important channels throughout the whole model, rather than pre\-defining pruning ratios of each layer\. Among global channel importance ranking methods, using batch normalization \(BN\)\[[18](https://arxiv.org/html/2607.06922#bib.bib18)\]as the scaling factor is more efficient\. Network slimming \(NS\)\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]leverages BN to remove the redundant channels and achieves some good results\. However, there are some limitations within this work\. First, it employs a ‘hard’ method to remove channels, which means that if the channels are eliminated during the pruning process, they will not be recovered in the afterwards training process\. Another drawback is that after the pre\-training step, the importance rank of all channels is fixed, implying that the compact model obtained from a large compression ratio must be a sub\-model of a small compression ratio\. The fact is there are many different settings for each layer’s channel number, hence their method blinds the design space to obtain a more efficient architecture for the specific compression ratio\. In this paper, we aim to design a dynamic method to make the compression decision\.

Soft Filter Pruning \(SFP\)\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], similar to our method, exploits a learning scheme in which some unimportant channels are set to zero according to theLpL\_\{p\}norm\. Unfortunately, this method uses local \(intra\-layer\-wise\) channel importance ranking, so they need to initialize the pruning ratio for each layer manually by experience\. Thus, this method only tries to find the ‘significant’ weights but not the ‘optimal’ architecture\. However, previous work\[[14](https://arxiv.org/html/2607.06922#bib.bib14)\]has demonstrated that the pruned architecture is more crucial to the efficiency of the final model, rather than the inherited weight value of important elements\. Furthermore, it is observed that in SFP, zeroized filters are difficult to be recovered\. Meanwhile, SFP does not have an initial training process so it cannot effectively discriminate the channel importance, resulting in low accuracy \(similar tos=0s=0in Section[3\.4](https://arxiv.org/html/2607.06922#S3.SS4)\)\. The flaws of NS and SFP lead to more accuracy loss of the compact model seen in Section[4](https://arxiv.org/html/2607.06922#S4)\.

Some existing works also compact models by one training process without the typical 3\-stage process, for example, Prior Gradient Mask Guided Pruning\-Aware Fine\-Tuning \(PGMPF\)\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\], which selectively suppresses the gradient of those ”unimportant” parameters via a prior gradient mask generated by the pruning criterion\. It elegantly unifies pruning and fine\-tuning\. However, the training time overhead of PGMPF is huge, even larger than that of methods with pre\-training and re\-training, as shown in Section[4\.2](https://arxiv.org/html/2607.06922#S4.SS2)\. Thus, it is also inefficient in finding the optimal compact model architecture\.

#### 2\.1\.2Neural Network Quantization

Quantization is used to map a large set of input to a smaller set of output\. In DNN model quantization, instead of adopting the 32\-bit floating point \(FP\-32\) format to represent weights, quantized models use more compact formats such as integers or 16\-bit float point \(FP\-16\) format for weights\. Existing methods on DNN model quantization mainly include training from scratch\[[19](https://arxiv.org/html/2607.06922#bib.bib19),[20](https://arxiv.org/html/2607.06922#bib.bib20)\], fine\-tuning from a full\-precision model\[[21](https://arxiv.org/html/2607.06922#bib.bib21),[22](https://arxiv.org/html/2607.06922#bib.bib22)\]or post\-training quantization\[[23](https://arxiv.org/html/2607.06922#bib.bib23),[24](https://arxiv.org/html/2607.06922#bib.bib24)\]\. The first two methods introduce another training during the quantization process and these methods reduce accuracy drop significantly only when quantized to very low bits \(e\.g\., below 4\-bits\)\. Meanwhile, post\-training methods have already been used in commercial scenarios and perform very well in common\-used 16/8\-bits, like NVIDIA’s TensorRT\[[25](https://arxiv.org/html/2607.06922#bib.bib25)\]\. As these post\-training methods do not bring extra operations in the training stage and have enough accuracy for commercial usage, we can adopt the post\-training methods\[[15](https://arxiv.org/html/2607.06922#bib.bib15)\]to obtain an efficient framework in this paper\. After quantization, we can reduce the latency of a model, which indicates the quantized model can keep more channels or layers than the full\-precision one under the same specific latency constraint, hence the quantized model owns higher accuracy\.

#### 2\.1\.3Knowledge distillation

Knowledge distillation\[[16](https://arxiv.org/html/2607.06922#bib.bib16)\]is also promising to design compact models\. However, the compact model \(student\) has to be designed manually, whereas our approach is an end\-to\-end method to automatically learn a compact model from the redundant model \(teacher\)\. In future work, we are like to explore combining knowledge distillation and our method to derive more competitive models\.

### 2\.2Neural Network Scaling

Model scaling\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\]will increase the model complexity to improve the prediction accuracy and hardware utilization, and we can use it for simple models to obtain higher accuracy and still meet the latency constraint\. Model scaling can scale a model up from three dimensions, including the depth, the width and the resolution\. Scaling network depth means inserting more layers into the model, scaling network width means expend channels in some layers of the model and scaling the resolution means applying the input images with a higher resolution\. EfficientNet\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\]has demonstrated the results of scaling combined with these three dimensions\. They use an expensive search process to find scaling factors for these three dimensions respectively to determine the expanded amount of each dimension for an efficient architecture and high accuracy\. And they find that larger resolutions can get higher predicted accuracy, but the accuracy improvement is limited without contributing to the model architecture\. Previous works\[[26](https://arxiv.org/html/2607.06922#bib.bib26)\]also show that the depth and width of a model are both important for high accuracy, and Fig\.[2](https://arxiv.org/html/2607.06922#S2.F2)illustrates that scaling depth and width can lead to different increase trends of latency for different models\. In the paper, we introduce model width scaling and depth scaling into our framework to improve the utilization of simple models on edge devices for higher accuracy\. Our scaling method is based on our compact learning, and we can also get the scaled model with only one training process without pre\-training and re\-training steps\. Also, we do not need the expensive pre\-searched depth and width scaling factors\[[10](https://arxiv.org/html/2607.06922#bib.bib10),[11](https://arxiv.org/html/2607.06922#bib.bib11)\]\.

### 2\.3Latency Prediction

To speed up inference, previous model optimizing works focus on changing FLOPs or channels/layers of models\. However, these metrics cannot reflect the actual latency\. Optimization based on real latency can better explore hardware features and hence offer additional advantages\[[27](https://arxiv.org/html/2607.06922#bib.bib27)\]\. Only a relatively small amount of work has been done to design model compression/scaling methods considering latency as the direct optimization goal\. Currently, there are some works using look\-up tables \(LUTs\)\[[12](https://arxiv.org/html/2607.06922#bib.bib12),[27](https://arxiv.org/html/2607.06922#bib.bib27),[28](https://arxiv.org/html/2607.06922#bib.bib28)\]or hardware simulators\[[29](https://arxiv.org/html/2607.06922#bib.bib29)\]to predict the latency of neural networks on devices, and they have achieved good works\. Yang et al\.\[[27](https://arxiv.org/html/2607.06922#bib.bib27)\]build layer\-wise LUTs with pre\-measured latency of each layer and sum up the layer\-wise results to estimate the network\-wise latency\. Dai et al\.\[[12](https://arxiv.org/html/2607.06922#bib.bib12)\]and Wu et al\.\[[28](https://arxiv.org/html/2607.06922#bib.bib28)\]construct operator\-level latency LUTs by measuring the latency of each operator on real devices and computing the overall latency of models by adding up the latency of each operator in the corresponding model\.

Nonetheless, a LUT can only predict the latency of some models/layers whose structures are already in its table, hence it is only suitable for that all structures are known, and it needs rebuilding when a new structure is added\. During the learning process, the number of input channels and output channels of each layer and other size\-related parameters will be changed continuously\. Thus, the LUT method is not available\. And unlike FPGA, most hardware is a black box to users, hence it is very hard to simulate hardware by analysing its resource and scheduling algorithms, indicating that the hardware simulator is also unavailable\. Therefore, a universal hardware latency predictor is essential to guide the learning algorithm to find a better architecture for a specific edge device\. Justus et al\.\[[30](https://arxiv.org/html/2607.06922#bib.bib30)\]proposed a latency predictor based on machine learning, which can predict the execution time of the whole model\. But its training data is randomly set, lowering the accuracy of predicting real models\. Zhang et al\.\[[31](https://arxiv.org/html/2607.06922#bib.bib31)\]proposed nn\-Meter to use a more fine\-grained LUT approach which characterizes models at the kernel level, but their approach involves exhaustive testing of all possible combinations of operations \(kernels\) which can take anywhere up to 4\.4 days\. But our predictor is layer\-wise and operation\-level which consumes much less time \(a few hours\)\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x3.png)Figure 3:The overview of our model learning framework\. The input is a mismatched original model and this framework outputs a model with high accuracy under the latency constraint\.

## 3Latency\-constraint Architecture Learning Framework

In this section, we show the process of our learning framework, which trains DNNs for an edge device under a latency constraint and optimizes complex/simple models to suit edge devices\. This framework is based on our compact learning method and it reasonably uses model scaling to expand the applicability and improve accuracy, which is shown in Fig\.[3](https://arxiv.org/html/2607.06922#S2.F3)\. It generates a latency predictor from the specific edge device in advance and then executes the whole algorithm on a high\-performance host machine\. After evaluating the latency of the original model, if the latency is smaller than the system latency constraint, we first scale the model up to obtain a new model that exceeds the constraint\. Otherwise, we keep the model unchanged\. Thus, the current model is a redundant complex model that needs to be compressed to satisfy the latency constraint\. For simple models, scaling up first and then compacting can eliminate the expensive scaling factors search process and improve the accuracy111For overfitting considerations, see Section 3\.3\.under the latency constraint with one training process\. The latency predictor also guides the compression process to find out the appropriate compact ratio for the specific latency constraint\. If quantization inference is supported, we can further improve the model accuracy under the current latency constraint by integrating quantization\. To use quantization, a quantized latency predictor is generated in the first step\.

In the following, we first introduce the compact learning scheme, then we show how to build the universal latency predictor, and the next is our model scaling method\. Finally, we introduce how to set the hyperparameters introduced by our algorithm and its further potential\.

Input:model, training data and settings, start epoch:ss, zero interval:kk,latency constraint:ll,layer pruning:l​plp\.

Output:the compact model and its parameters

1:Initialize model and its parameters;

2:Preprocess model as Fig\.[5](https://arxiv.org/html/2607.06922#S3.F5)if

l​plp;

3:for

e←1e\\leftarrow 1to

s−1s\-1do

4:

t​r​a​i​n​\_​w​i​t​h​\_​s​p​a​r​s​i​t​y​\(\)train\\\_with\\\_sparsity\(\);

5:endfor

6:for

e←se\\leftarrow sto

e​p​o​c​hm​a​xepoch\_\{max\}do

7:

t​r​a​i​n​\_​w​i​t​h​\_​s​p​a​r​s​i​t​y​\(\)train\\\_with\\\_sparsity\(\);

8:if

e%k==s%ke\\%k==s\\%kor

e==epochm​a​xe==epoch\_\{max\}then

9:

I​m​p​\_​r​a​n​k←Imp\\\_\{rank\}\\leftarrowSort channels by

\|γ\|\|\\gamma\|;

10:

C​o​m​p​\_​r​a​t​i​o←P​r​e​d​i​c​t​o​r​\(l,I​m​p​\_​r​a​n​k,l​p\)Comp\\\_\{ratio\}\\leftarrow Predictor\(l,Imp\\\_\{rank\},lp\);

11:Find threshold

γt​h​r​e\\gamma\_\{thre\}of

γ\\gammaby

I​m​p​\_​r​a​n​kImp\\\_\{rank\},

C​o​m​p​\_​r​a​t​i​oComp\\\_\{ratio\};

12:Zeroize

γi\\gamma\_\{i\}and

βi\\beta\_\{i\}if

\|γi\|\|\\gamma\_\{i\}\|<<γt​h​r​e\\gamma\_\{thre\};

13:if

l​plpthen

14:Calculate the sum of importance values in each layer

vi=∑\|γi\|v\_\{i\}=\\sum\|\\gamma\_\{i\}\|;

15:Remove the layer if

vi<vt​h​r​ev\_\{i\}<v\_\{thre\};

16:Remove the added branch if

vi≥vt​h​r​ev\_\{i\}\\geq v\_\{thre\};

17:endif

18:endif

19:endfor

20:Remove all channels with zero

γi\\gamma\_\{i\}and

βi\\beta\_\{i\};

Algorithm 1Latency\-Critical Compact Learning![Refer to caption](https://arxiv.org/html/2607.06922v1/x4.png)Figure 4:The process of our compact learning approach\. The input is a redundant model and the output is a compact model that meets the system latency constraint\. The gray dotted boxes represent the process of quantization\. Each oblique rectangle represents a channel\.### 3\.1Latency\-Critical Compact Learning

Traditional neural network pruning procedures include 3 stages:pre\-training,pruning, andre\-training\. The pre\-trained over\-parameterized model, however, is not necessary to obtain an effective compact model, and it is the network architecture that is more important to the efficiency and accuracy of the final compact model\[[14](https://arxiv.org/html/2607.06922#bib.bib14)\]\. Thus, the compression approach presented in this paper intends to directly learn an efficient compact architecture under the specified latency constraint during the training stage of the seed model, avoiding the time\-consuming three steps\. The key steps in our compact learning technique are depicted in Fig\.[4](https://arxiv.org/html/2607.06922#S3.F4), where the input seed model is a large redundant DNN model and the output is a compact model that meets the latency constraint\. The seed model does not need to be well\-trained, which means that all we need is only the architecture of the DNN model being worked with\. Finally, the framework will produce a trained compact model with high accuracy under the latency constraint\.

In the latency\-critical compact model learning process, there are two critical challenges: one is determining an efficient compact architecture extracted from the original model, and the other is determining the compression ratio under a latency constraint\. To address the first challenge, We present a novel technique to automatically cultivate an efficient sub\-model architecture while training the seed model from scratch\. For the second problem, we design a machine learning\-based latency predictor to directly determine the number of channels or layers to eliminate\. As illustrated in Algorithm[1](https://arxiv.org/html/2607.06922#alg1), the proposed compression approach only requires one training procedure, mainly including 3 phases: 1\)Initial Training; 2\)Zero Training; 3\)Recovery Training\.The input of this algorithm ismodel– the model architecture without trained parameters \(weights\);training data and settings– the training dataset and training settings, like epochs, learning rate, and so on;start epoch– the epoch at which starts Zero training;zero interval– the epoch interval between Zero training and Recovery training;latency constraint– the latency constraint that the model being trained needs to satisfy;layer pruning– whether layer pruning is used in the training process\. More details regarding these inputs are introduced in the following sections\. The output of this algorithm is a well\-trained model that satisfies the latency constraint\.

#### 3\.1\.1Initial Training

Initial training \(Line 3\-5\) is the first step of our compact learning process and it is similar to that of conventional DNN model training, with the goal of obtaining trained weights for the seed model\. In order to improve the accuracy and efficiency of the final compact model, we need an explicit learning medium to find out the optimal architecture, such as the number of layers in this model and the number of channels of each layer\. According to previous work\[[32](https://arxiv.org/html/2607.06922#bib.bib32)\], we can leverage the trainable parameterγ\\gammaof the BN layer to offer useful instruction for compact architecture learning without incurring any overhead\. As shown in Eq\. \([1](https://arxiv.org/html/2607.06922#S3.E1)\),xxis the input of the BN layer andyyis the output\.ϵ\\epsilonis a number that is artificially set to keep the denominator from becoming zero\. The mean and standard deviation of a mini\-batchBBareμB\\mu\_\{B\}andσB\\sigma\_\{B\}, respectively\.γ\\gammaandβ\\betaare affine transformation parameters that may be trained to translate normalized activation to any scale linearly\. Since each channel has a correspondingγ\\gamma, they can be utilized as explicit medium directly to reflect the importance of channels\. Theseγ\\gammais collaboratively optimized with the network weights during the training phase, thus, the network can automatically recognize the importance of each channel, and channels with smallγ\\gammacan be safely removed\.

x¯=x−μBσB2\+ϵ;y=γ⋅x¯\+β\\overline\{x\}=\\frac\{x\-\\mu\_\{B\}\}\{\\sqrt\{\{\\sigma^\{2\}\_\{B\}\}\+\\epsilon\}\};\\quad y=\\gamma\\cdot\\overline\{x\}\+\\beta\(1\)
To determine the optimal architecture of the compact model, the seed model should be trained for several epochs, allowing all explicit mediumγ\\gammato have informative values that represent the importance of channels \(Line 3\-5\) rather than randomly initialized numbers \(Line 1\)\. Meanwhile, the Initial training process also guarantees that the updating directions are not 0 \(used in Recovery training\)\.We use sparsity training\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]throughout the training phase to impose sparsity regularization on these explicit mediumγ\\gamma\. Although sparsity training was also applied in the pre\-training phase of NS’s\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], our initial training requires fewer epochs, as shown in Section[4](https://arxiv.org/html/2607.06922#S4)\. Meanwhile, compared to the pre\-training stage in other works\[[33](https://arxiv.org/html/2607.06922#bib.bib33),[34](https://arxiv.org/html/2607.06922#bib.bib34)\], which simply apply traditional DNN model training, the sparsity training used in our initial training can better distinguish the importance of these explicit mediumγ\\gammafor better compact DNN architecture\.After Initial training epochs, we can formally enter the two most essential iterative phases of our method: Zero and Recovery \(Line 6\-19\)\. Our approach can extract an optimal compact architecture by dynamically adjusting it according to the trainedγ\\gammaduring these two iterative phases\.

#### 3\.1\.2Zero Training

Zero training \(Line 6\-19\) is used to compress the model and extract a compact architecture\. We sort all theγ\\gammaaccording to their absolute value first to obtain the global importance rank of channels across all layers \(Line 9\)\. Then our proposed latency predictor reports a compression ratio to satisfy the specified latency constraint \(Line 10\)\. The details about our machine learning\-based latency predictor are described in Section[3\.2](https://arxiv.org/html/2607.06922#S3.SS2)\. The compression ratio and the global importance rank are used to determine the threshold ofγi\\gamma\_\{i\}\(Line 11\)\. Then, for insignificant channelcic\_\{i\}whoseγi\\gamma\_\{i\}is less than the threshold, we zeroize out those correspondingγi\\gamma\_\{i\}andβi\\beta\_\{i\}\(Line 12\) to let these channels in BN layers output zero\. When the BN layer is applied after a convolutional layer, each channel of the BN layer corresponds to one channel of the previous layer, implying that the Zero process is equivalent to removing those channels from the convolutional layer\. Therefore, our compression method has extracted a compact architecture through the Zero process to fulfil the certain latency constraint\. Meanwhile, it retains all of the trained weights of convolutional layers in this model and only simply zeroizes out the associatedγi\\gamma\_\{i\}andβi\\beta\_\{i\}\. Each channel contains aγi\\gamma\_\{i\}value and aβi\\beta\_\{i\}value, thus, this Zero training just removes a small number of values from the trained model, making it easier to recover\.

Layer Pruning:As shown in Fig\.[2](https://arxiv.org/html/2607.06922#S2.F2), different model architectures have different latency trends by changing the number of channels or layers\. Thus, in order to derive the best model architecture, layer pruning is as important as channel pruning\. However, eliminating layers directly during the training process results in a disconnected model architecture\. Thus, for supporting layer pruning, before training the model, we need to pre\-process it \(Line 2\)\. As illustrated in Fig\.[5](https://arxiv.org/html/2607.06922#S3.F5), we add a directly\-connected layer as a branch to each convolutional layer and we aggregate the results from the convolutional layer and its associated branch layer to get the input of the next layer222If the output size is different to that of the branch layer, only aggregate outputs in corresponding positions\.\. In the Zero training, the latency predictor only needs to predict one branch latency as we only retain one branch for inference\.To eliminate layers, after zeroizingγi\\gamma\_\{i\}andβi\\beta\_\{i\}corresponding to unimportant channelcic\_\{i\}in each layer \(Line 12\), we calculate the sum of each layer’s channel importance values and remove a layer \(zeroize the correspondingλ\\lambda\) if its total importance values are less than the threshold \(Line 14\-15\)\. Otherwise, we remove the corresponding added branch \(Line 16\)\. The larger the important value, the more “information” the corresponding channel retains\. And the sum of important values in a layer means the “total information” this layer keeps\. In the layer pruning, we only eliminate those layers with tiny total important values to maintain more “information” for high accuracy\. Thus, during the compact training, if the sum of important values in each layer is larger than the thresholdvt​h​r​ev\_\{thre\}, we do not remove any layer in this model, implying that layer pruning does not occur in this training process\. This is why some models did not have the experimental result of layer pruning in Section[4](https://arxiv.org/html/2607.06922#S4)\.

#### 3\.1\.3Recovery Training

After the Zero training phase, the Recovery training process \(Line 6\-19\) is executed fork−1k\-1epochs to allow the zeroized parameters \(Line 8\), such asγi\\gamma\_\{i\}andβi\\beta\_\{i\}, to recover themselves rather than being permanently removed\. If there are channels that are eliminated in the previous Zero training phase but are potentially essential, the Recovery training can help them jump out of zero and play a crucial part in the subsequent training process and even the inference processes\. The Recovery training gives our method a chance to learn more efficient architecture with higher accuracy\. Eq\. \([2](https://arxiv.org/html/2607.06922#S3.E2)\) indicates the common calculation formula in a convolutional layer with BN and activation function, whereLLis the index of layer,ALA^\{L\}is the output of layerLL,WLW^\{L\}is the weight between layerL−1L\-1andLL, andσL\\sigma^\{L\}is the activation function at layerLL\.∗\\astrepresents the convolution operation and other parameters are the same as Eq\. \([1](https://arxiv.org/html/2607.06922#S3.E1)\)\.

AL=σL​\(γL​\(AL−1∗WL¯\)\+βL\)A^\{L\}=\\sigma^\{L\}\(\\gamma^\{L\}\(\\overline\{A^\{L\-1\}\\ast W^\{L\}\}\)\+\\beta^\{L\}\)\(2\)In the Recovery training, we focus more on updatingγL\\gamma^\{L\}andβL\\beta^\{L\}that are zeroized during the Zero training \(theλ\\lambdaused in layer pruning is similar\)\. Eq\. \([3](https://arxiv.org/html/2607.06922#S3.E3)\)\-\([4](https://arxiv.org/html/2607.06922#S3.E4)\) show the gradients ofγL\\gamma^\{L\}andβL\\beta^\{L\}to the final loss function\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x5.png)Figure 5:The preprocessing for layer pruning by adding branch layers\.∂l​o​s​s∂γL=∂l​o​s​s∂AL⋅σL′⋅\(AL−1∗WL¯\)\\frac\{\\partial loss\}\{\\partial\\gamma^\{L\}\}=\\frac\{\\partial loss\}\{\\partial A^\{L\}\}\\cdot\{\\sigma^\{L\}\}^\{\{\}^\{\\prime\}\}\\cdot\(\\overline\{A^\{L\-1\}\\ast W^\{L\}\}\)\(3\)∂l​o​s​s∂βL=∂l​o​s​s∂AL⋅σL′\\frac\{\\partial loss\}\{\\partial\\beta^\{L\}\}=\\frac\{\\partial loss\}\{\\partial A^\{L\}\}\\cdot\{\\sigma^\{L\}\}^\{\{\}^\{\\prime\}\}\(4\)Since we have zeroized some values inγL\\gamma^\{L\}andβL\\beta^\{L\}, and the most often used activation function isReLU, the corresponding output ofσL\\sigma^\{L\}is likewise zero underReLU\. AlthoughReLUis not differentiable at zero, it is widely accepted that its derivative is also zero\. Thus, we can easily derive Eq\. \([5](https://arxiv.org/html/2607.06922#S3.E5)\)\.

γL,βL=0→σL′=0→∂l​o​s​s∂γL,∂l​o​s​s∂βL=0\\gamma^\{L\},\\beta^\{L\}=0\\rightarrow\{\\sigma^\{L\}\}^\{\{\}^\{\\prime\}\}=0\\rightarrow\\frac\{\\partial loss\}\{\\partial\\gamma^\{L\}\},\\frac\{\\partial loss\}\{\\partial\\beta^\{L\}\}=0\(5\)Until now, we may need to offer these zeroizedγL\\gamma^\{L\}andβL\\beta^\{L\}some small values to break this deadlock\. However, most training algorithms employ momentum in the optimizer to speed up convergence, as the momentum accumulates the gradients of the past steps to determine the direction to go, rather than using only the gradient of the current step\[[35](https://arxiv.org/html/2607.06922#bib.bib35)\]\. Eq\. \([6](https://arxiv.org/html/2607.06922#S3.E6)\)\-\([7](https://arxiv.org/html/2607.06922#S3.E7)\) illustrates the updating rules of weight with momentum, in whichl​rlris the learning rate,zkz\_\{k\}is the updated value of the last step andmmis the accumulation coefficient\. Following these rules,wkw\_\{k\}are updated towk\+1w\_\{k\+1\}by combining current and past gradients\.

zk\+1=m⋅zk\+∂l​o​s​s∂wkz\_\{k\+1\}=m\\cdot z\_\{k\}\+\\frac\{\\partial loss\}\{\\partial w\_\{k\}\}\(6\)wk\+1=wk−l​r⋅zk\+1w\_\{k\+1\}=w\_\{k\}\-lr\\cdot z\_\{k\+1\}\(7\)As a result, even though the current gradients ofγL\\gamma^\{L\}andβL\\beta^\{L\}are zero, they can be updated to recover from zero by following the last directions \(This is also why we need the Initial training\)\. This process is fully automated and efficient, as no additional steps are required\.

wk,∂l​o​s​s∂wk=0→wk\+1=−l​r⋅m⋅zkw\_\{k\},\\frac\{\\partial loss\}\{\\partial w\_\{k\}\}=0\\rightarrow w\_\{k\+1\}=\-lr\\cdot m\\cdot z\_\{k\}\(8\)
![Refer to caption](https://arxiv.org/html/2607.06922v1/x6.png)Figure 6:Channel importance changes during our learning process\. Zeroized channels \(white blocks\) can be easily recovered\.Specifically, as the previous training phase is Zero training, the updating rule ofwk\+1w\_\{k\+1\}is shown in Eq\.[8](https://arxiv.org/html/2607.06922#S3.E8), implying thatwk\+1w\_\{k\+1\}are automatically set to non\-zero small values \(l​rlrandzkz\_\{k\}are small\)\. In the Recovery training, the network first tries to predict results under thoseγL\\gamma^\{L\}andβL\\beta^\{L\}are small\. And channels corresponding to these smallγL\\gamma^\{L\}andβL\\beta^\{L\}have little effect on final results\. Just like knowledge distillation, the training process tries to use a small network architecture to cover all the knowledge learned by the large network architecture\. There are two situations in this process: 1\) if the loss is small, then the network can exclude these channels safely for finishing compression; 2\) if the loss is huge, then the optimizer updates these parameters significantly \(see Eq\.[6](https://arxiv.org/html/2607.06922#S3.E6)and Eq\.[7](https://arxiv.org/html/2607.06922#S3.E7)\)\. In the second condition, these channels zeroized in previous Zero training may become significant and we will not zeroize them in the future Zero training process, therefore the architecture of the final compact model is changed\. Especially, as the architecture changes, the latency predictor will report a different compression ratio in the next Zero training phase\. The training process tries to minimize loss and improve accuracy, allowing the compact architecture to improve within the specific latency constraint\. Compared to\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], our method can identify a more efficient and accurate model with better architecture, as shown in Section[4](https://arxiv.org/html/2607.06922#S4)\. Furthermore, we do not need to retrain the compact model further as our learning process already obtains high accuracy, reducing the training overhead\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x7.png)Figure 7:The framework of our latency predictor\. The left process represents the building of the predictor and the right part reveals how to predict the latency of a real model with the predictor\.Fig\.[6](https://arxiv.org/html/2607.06922#S3.F6)shows the changes in the importance of all channels during the learning process on an example model, where the start epoch of Zero trainingssis66, the interval epochs of Zero trainingkkis 2, and the compression ratio is set to a constant0\.30\.3\. We can derive from this figure that zeroized channels are easy to recover and even some channels, like those framed in the red rectangle, may become very essential\. We also discovered from our experiments that if we zeroize filter weights, which are utilized in SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], it is difficult for these filters to avoid being zeroized in the next pruning step by employing momentum\. The reason may be gradients are also zero and each filter has a large number of weights, then the momentum is a little small for a fully zeroized filter to win other non\-zeroized filters byLpL\_\{p\}norm\. Thus, zerorizingγ\\gammaandβ\\betain the BN layer is an effective way to cultivate the optimal sub\-model\. This is why our accuracy is higher than that of other methods\.

After some repetitions of Zero and Recovery training epochs, removing all channels corresponding to zeroizedγi\\gamma\_\{i\}from the channel\-sparsity model \(Line 20\), we can obtain an efficient compact architecture fulfilling the specific system latency constraint\. Also, we remove layers \(branches\) with zeroizedλ\\lambda\. This model keeps weights from the training process, hence it does not need to be re\-trained\.

Training Overhead per Epoch:Compared to the traditional training process, we only introduce overhead in the zeroizing process\. As we remove channels in convolution layers, we mainly focus on convolution\. The time complexity of convolution layers for an input isO​\(∑l=1LMl2​Kl2​Cl−1​Cl\)O\(\\sum\_\{l=1\}^\{L\}M\_\{l\}^\{2\}K\_\{l\}^\{2\}C\_\{l\-1\}C\_\{l\}\), whereLLis the number of convolution layers,ClC\_\{l\}is the number of output channels in layerll,MlM\_\{l\}is the output size of layerllandKlK\_\{l\}represents the kernel size of layerll\. The zeroizing operation includes a sorting step with a time complexity ofO​\(n​l​o​g​n\),n=∑l=1LClO\(nlogn\),n=\\sum\_\{l=1\}^\{L\}C\_\{l\}\. Generally,l​o​g​nlognis much smaller thanMl2​Kl2​Cl−1M\_\{l\}^\{2\}K\_\{l\}^\{2\}C\_\{l\-1\}of each layerll, so its complexity is much lower than the convolution layers\. The latency predictor only includes hundreds of multiplication and addition operations, which is much smaller than convolution operations\. Thus, the overhead of our compact training can be neglected, demonstrating the high efficiency of our method\.The evaluation results of overhead are shown in Section[4\.2](https://arxiv.org/html/2607.06922#S4.SS2)\.

### 3\.2ML\-based Latency Prediction Methodology

Previous researches have used hardware simulator\-based\[[29](https://arxiv.org/html/2607.06922#bib.bib29)\]or look\-up table \(LUT\)\-based\[[12](https://arxiv.org/html/2607.06922#bib.bib12),[27](https://arxiv.org/html/2607.06922#bib.bib27),[28](https://arxiv.org/html/2607.06922#bib.bib28)\]predictors to estimate the latency of DNN on a certain device\. However, the limitations of these methods become obvious throughout the model learning process\. Since most devices are black boxes to users, it is difficult to emulate hardware by analyzing its resources and scheduling algorithms\. As a result, most commercial edge devices are incompatible with hardware simulators\. Second, LUT calculates the overall latency of DNN by summing up the recorded latency of each layer, thus, it can only provide the latency of a model that only includes pre\-defined layer structures\. However, during the learning process, the number of layers and the number of input channels, output channels, and other size\-related parameters of each layer will be updated continuously\. It is impracticable for LUT to preserve the whole design space, and LUT needs to rebuild anytime a new layer structure is introduced\. Thus, LUT is not suitable for our framework, either\. There are several latency predictors based on Neural Network\[[30](https://arxiv.org/html/2607.06922#bib.bib30),[31](https://arxiv.org/html/2607.06922#bib.bib31),[36](https://arxiv.org/html/2607.06922#bib.bib36)\], but their training data is randomly chosen and network architectures are not well designed, therefore the accuracy of these predictors is lower\.

In this model learning framework, we need a predictor which can provide any model latency on the specific hardware\. There are three challenges in building such a latency predictor: 1\) for each operation, such as convolution, each variable parameter has tens to hundreds of common choices, hence the search space is nearly infinite; 2\) no hardware design details or scheduling schemes are available for reference; and 3\) this predictor is should be accurate enough, as the latency is a ‘hard’ constraint\. Therefore, only methods that can extract patterns from limited samples are suitable for our framework\. As illustrated in Fig\.[7](https://arxiv.org/html/2607.06922#S3.F7), we propose a hardware\-customized latency predictor based on Backpropagation \(BP\) neural network\[[37](https://arxiv.org/html/2607.06922#bib.bib37)\]\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x8.png)Figure 8:The latency distribution before/after transform\. The normal distribution can improve the accuracy of the latency predictor\.g​\(x\)=σL​\(WL​σL−1​\(WL−1​⋯​σ1​\(W1​x\)​⋯\)\)g\(x\)=\\sigma^\{L\}\(W^\{L\}\\sigma^\{L\-1\}\(W^\{L\-1\}\\cdot\\cdot\\cdot\\sigma^\{1\}\(W^\{1\}x\)\\cdot\\cdot\\cdot\)\)\(9\)BP network is a lightweight neural network used to map the input space to the output space without expert\-designed mathematical functions\. Eq\.[9](https://arxiv.org/html/2607.06922#S3.E9)shows the forward propagation of the BP network, wherexxis the vector of features,LLis the number of layers,WlW^\{l\}is the weight between layerl−1l\-1andll, andσl\\sigma^\{l\}is the activation function at layerll\. According to this equation, after training this BP network, hundreds of multiplication and addition operations are enough to obtain the latency, which just takes a few milliseconds, while interacting with the hardware takes several minutes\. Also, it is easy to find that the updating scheme of the BP network is also simple, thus training it is not time\-consuming\. The BP network we used includes three layers, which are the input layer \(16 neurons\), the hidden layer \(12 neurons\), and the output layer \(1 neuron\)\. And there is an activation function between input features and every two connected layers, here we uset​a​n​s​i​gtansig,t​a​n​s​i​gtansig, andp​u​r​e​l​i​npurelin, respectively\.

To train the BP\-based latency predictor, we first need to build a dataset, which includes many parameterized models\. The parameterized model refers to the single\-layer model with various operations like Convolution, BN and Pooling configured with different size\-related parameters \(configurations\)\. The design space of size\-related parameters is determined by the commonly\-used range of each parameter\. Then we undertake regular and random sampling over the whole design space of parameters\. Regular sampling splits the design space into several grids, whereas random sampling supplements extra details to these grids\. After constructing parameterized models with these sampled parameters, we measure them on the specific device to obtain precise latency\. Then we train the BP neural network with configurations of these sampling models and their corresponding precise latency \(labels\), trying to let the BP network output the latency by inputting a configuration\. To improve the accuracy of the BP network, the training labels must correspond to the normal distribution\[[38](https://arxiv.org/html/2607.06922#bib.bib38)\], however, the latency of most single\-layer models is small\. Thus, as illustrated in Fig\.[8](https://arxiv.org/html/2607.06922#S3.F8), we need to transform these latency data labels to approximate the normal distribution before the BP network training process\.

Although the number of single\-layer models in our sampling is limited, the trained BP network can predict the latency of any configuration in the design space\[[37](https://arxiv.org/html/2607.06922#bib.bib37)\]\. So, the trained BP network is a single\-layer predictor\. To predict the overall latency of the entire model, we first use the single\-layer predictor to obtain the latency of each layer in the model\. Then, we analyse the model’s topological information to find the latency\-weighted longest path and add all latency on this path to obtain the whole model latency\.Table[1](https://arxiv.org/html/2607.06922#S3.T1)shows the accuracy of the proposed BP\-based latency predictor on widely\-used models and the average error is about 6\.12%, whereas the error of\[[30](https://arxiv.org/html/2607.06922#bib.bib30)\]on real models is about 13\.52%\. Without loss of generality, the BP\-based predictor is a common approach that is easily applied to other neural network design frameworks when considering the latency constraint\. Also, this method can be also generalized to other hardware metrics\.

Table 1:Results of latency predictor on various models\.
### 3\.3Latency\-critical Model Scaling

For accelerating the inference, some lightweight model architectures are proposed\. The latency of a simple model may be smaller than the constraint of a high\-end system, indicating that this model can be extended for more accuracy but still meet the latency requirement\. Previous works\[[10](https://arxiv.org/html/2607.06922#bib.bib10),[11](https://arxiv.org/html/2607.06922#bib.bib11),[39](https://arxiv.org/html/2607.06922#bib.bib39)\]have proposed some methods to scale up a model, demonstrating the efficiency of model scaling\. In this work, we first scale a DNN model up uniformly \(unified scaling\) and then use our compact learning method to ensure that the latency constraint is satisfied, eliminating the expensive search process of scaling factors\.

To undertake model scaling with our framework, we need to expand the model before feeding it into the compact learning scheme, as shown in Fig\.[9](https://arxiv.org/html/2607.06922#S3.F9), where the input seed model is a simple DNN model and the output is an extended model with higher accuracy under the latency constraint\. We also do not need to train the seed model and only its model architecture is needed\. As illustrated in Algorithm[2](https://arxiv.org/html/2607.06922#alg2), we first scale the model up by a factorδ\\delta, which increases the number of channels in each layer and the total number of layers in the model by this factor \(Line 1\-2\), thus it is named “unified scaling”\. The scaling factorδ\\deltais set by the latency margin from the constraint and makes sure the current latency is larger than the latency constraint\. After this step, we obtain a redundant model again\. Then we preprocess this model following Fig\.[5](https://arxiv.org/html/2607.06922#S3.F5)to add branch layers \(Line 3\), allowing layer pruning \(Line 4\)\. And our compact learning can compress this extended model \(Line 5\), and the final scaled\-up model meets the latency requirement with higher accuracy than the original model\. As our compact learning can remove channels and layers and it trains for high accuracy, we do not need the expensive search\[[10](https://arxiv.org/html/2607.06922#bib.bib10),[11](https://arxiv.org/html/2607.06922#bib.bib11)\]to identify the optimal scaling factors for layers and channels, respectively\.

However, for small datasets such as CIFAR\-10\[[40](https://arxiv.org/html/2607.06922#bib.bib40)\], commonly designed models may already be redundant\. In this instance, model scaling cannot lead to higher accuracy; for example, the VGG\-19\[[41](https://arxiv.org/html/2607.06922#bib.bib41)\]model with 70% of channels removed can achieve higher accuracy than the full VGG\-19 model on the Cifar\-10 dataset \(see Fig\. 11\)\. Then we can derive that, for redundant models, model scaling diminishes rather than increases accuracy\. On the other hand, for large datasets such as ImageNet\[[42](https://arxiv.org/html/2607.06922#bib.bib42)\], model scaling generally improves accuracy\[[10](https://arxiv.org/html/2607.06922#bib.bib10),[11](https://arxiv.org/html/2607.06922#bib.bib11),[43](https://arxiv.org/html/2607.06922#bib.bib43)\]\. Thus, we can safely scale the model up for better accuracy on large datasets\. Meanwhile, without loss of generality, we can evaluate and compare the accuracy of the original model and the scaled model to choose the better one\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x9.png)Figure 9:The process of our latency\-critical model scaling approach\.Input:model, training data and settings, start epoch:

ss, zero interval:

kk,latency constraint:ll\.

Output:the scaled model and its parameters

1:Extend channels of each layer by

δ\\deltatimes; \(Width\)

2:Duplicate each layer by

δ\\deltatimes; \(Depth\)

3:Preprocess the extended model according to Fig\.[5](https://arxiv.org/html/2607.06922#S3.F5);

4:Set the value of

l​plptoTrue;

5:CallLatency\-Critical Compact Learning;

Algorithm 2Latency\-critical Model Scaling
### 3\.4Automatic Parameter Setting

Although our framework is efficient for optimizing models to meet the latency constraint of an edge system by only one training process, it introduces two hyperparameters \(i\.e\.,ssandkkin Algorithm[1](https://arxiv.org/html/2607.06922#alg1)and[2](https://arxiv.org/html/2607.06922#alg2)\) into the training process, which brings some difficulty to developers\. We will give a reference on how to determine these hyperparameters by our experimental data\. First, for thekk, as shown in the left part of Fig\.[10](https://arxiv.org/html/2607.06922#S3.F10), we can set the reference value to22in most instances, which implies we should alternatively execute one Zero epoch and one Recovery epoch after Initial epochs\. Second, for thess, according to our experiments, it is a little difficult to find the optimal value, as the optimal value is not fixed but in a range\. Therefore, after analyzing the results shown in the right part of Fig\.[10](https://arxiv.org/html/2607.06922#S3.F10), the simple way is to set the reference value of the start epoch for our framework to half of all training epochs \(e​p​o​c​hm​a​xepoch\_\{max\}\)\.

Particularly, we can conclude from Fig\.[10](https://arxiv.org/html/2607.06922#S3.F10)that ifssis close to the optimal value, the accuracy is similar to the optimum\. And we can also deduce that it prefers a smallerssto a largerss, thus, we round down the result to 10 multiples to guarantee thessvalue is not larger than the optimal one\. The other settings related to Fig\.[10](https://arxiv.org/html/2607.06922#S3.F10)will be introduced in Section[4](https://arxiv.org/html/2607.06922#S4)\.

### 3\.5Model Quantization

In this paper, the edge devices we used \(i\.e\., Nvidia Jetson TX2 and Nvidia Jetson Nano\) only support half\-precision floating\-point \(FP16\) quantized inference\. For FP16 quantization, the post\-training quantization method can achieve high accuracy and does not bring extra training overhead\[[25](https://arxiv.org/html/2607.06922#bib.bib25)\]\. Quantizing the trained model from full\-precision floating\-point \(FP32\) format to FP16 format can reduce the model complexity and lower latency\. Under the specific latency constraint, the FP16 model can retain more weight than the FP32 model, resulting in higher accuracy\. We use the TensorFlow Converter\[[44](https://arxiv.org/html/2607.06922#bib.bib44)\]to transform models from FP32 format to FP16 format and finish the quantization process\. Without loss of generality, other quantification methods\[[15](https://arxiv.org/html/2607.06922#bib.bib15),[23](https://arxiv.org/html/2607.06922#bib.bib23)\]and other quantization bits \(e\.g\., INT8\) also suit our framework\.

### 3\.6Further discussion

As shown in previous sections, our learning is an efficient and effective method to train suitable models for a variety of hardware\. Given that BN is included in most DNN models, we believe that our approach has a significant potential for learning efficient and accurate networks for the emerging edge era\. Meanwhile, if a DNN model does not have the BN layer, we can simply introduce a valueγ′\\gamma^\{\{\}^\{\\prime\}\}into each convolution layer as a mask\. After training, we can remove all channels with zeroγ′\\gamma^\{\{\}^\{\\prime\}\}and multiply nonzeroγ′\\gamma^\{\{\}^\{\\prime\}\}to corresponding channel weights to obtain the final model, implying that this model does not include extra operations in the inference phase\.

Meanwhile, not only the pre\-verified models \(e\.g\., state\-of\-the\-art models\) but also models that have not been validated \(e\.g\., searched from NAS\), can be fed into our framework to optimize for more suitable architectures\. As NAS methods\[[4](https://arxiv.org/html/2607.06922#bib.bib4)\]are progressively replacing the hand\-crafted model design, our framework can be integrated into NAS to design more hardware\-efficient, competitive models\. This is what we are like to investigate in the future\.

Furthermore, our framework provides a method to trade\-off between latency and accuracy for deploying machine learning algorithms better to black\-box edge devices\. Furthermore, the constraints of edge systems include not only latency but also energy, memory, and so on\. Our framework is also easy to modify for optimizing these goals, for example, we can build a power predictor or a memory predictor by the BP network\. Then the same framework can balance accuracy and energy or memory\. This paper just takes latency as an example, but our framework is not limited to optimizing latency\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x10.png)Figure 10:The accuracy w\.r\.t the variation ofkkandssvalues\.

## 4Experiments

In this section, we first introduce the details of our experimental environment including hardware platforms, benchmark models, datasets, and training settings\.Then we show the training overhead of our compact learning\.Next, we show the accuracy of our proposed learning method in comparison to state\-of\-the\-art methods\. Finally, we present the accuracy and efficiency of our framework under a hard latency constraint\.

### 4\.1Experimental Setup

Hardware Platforms:The target hardware platforms we used include NVIDIA®Jetson™TX2 and NVIDIA®Jetson Nano™\. Jetson TX2 is equipped with an NVIDIA Pascal™GPU with 256 CUDA®cores and 8GB LPDDR4 memory\. Jetson Nano is equipped with an NVIDIA Maxwell™GPU with 128 CUDA®cores and 4GB LPDDR4 memory\. These two are widely\-used embedded machine learning edge devices\. To maximize the performance, we put both TX2 and Nano in the “Max\-N” mode and lock their clocks to maximums\.

Datasets:The datasets used for benchmark are CIFAR\-10\[[40](https://arxiv.org/html/2607.06922#bib.bib40)\]and ILSVRC\-2012\[[42](https://arxiv.org/html/2607.06922#bib.bib42)\], which are commonly used for evaluating model optimization approaches\. The CIFAR\-10 dataset consists of 10 classes of RGB images with a resolution of 32×\\times32\. There are a total of 50K and 10K images in the training and testing sets, respectively\. The ILSVRC\-2012 dataset consists of RGB images in 1K classes with higher resolutions\. The training set contains around 1\.28M images, whereas the validation set has approximately 50K images\. Following previous works\[[45](https://arxiv.org/html/2607.06922#bib.bib45)\]\[[46](https://arxiv.org/html/2607.06922#bib.bib46)\], we randomly select 100 classes from the ILSVRC\-2012 dataset to set up a subset named ImageNet\-100 for more efficient comparison\. All images of the selected classes in the training set and validation set are used to train and test the models, respectively\.

Baseline Methods and Models:We compare our framework against seven state\-of\-the\-art methods, which are Soft Filter Pruning \(SFP\)\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], Filter Pruning via Geometric Median \(FPGM\)\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\], Network Slimming \(NS\)\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], Prior Gradient Mask Guided Pruning\-Aware Fine\-Tuning \(PGMPF\)\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\], Only Train Once \(OTO\)\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\], EfficientNet Compound Scaling \(Eff\-Compd\)\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\]and HACScale\[[11](https://arxiv.org/html/2607.06922#bib.bib11)\],on four representative DNN architectures: VGGNet\[[41](https://arxiv.org/html/2607.06922#bib.bib41)\], Res\-Net\[[47](https://arxiv.org/html/2607.06922#bib.bib47)\], DenseNet\[[48](https://arxiv.org/html/2607.06922#bib.bib48)\], and Goo\-gLeNet\[[3](https://arxiv.org/html/2607.06922#bib.bib3)\]\. The reason for using these networks is that their architectures use different kinds of network modules, which are popular and widely used\. VGGNet has a simple sequential structure\. ResNet, DenseNet, and GoogLeNet are the representatives of using Residual modules, Dense modules, and Inception modules, respectively\. The architectures of other models are composed by these modules or their variants\. On the CIFAR\-10 dataset, we choose VGG\-19, ResNet\-164, and Dens\-eNet\-40 networks, and on the ImageNet\-100 data\-set, we choose VGG\-19, GoogLeNet, and ResNet\-50444In this paper, we keep the order of BN layers and convolution layers in the Residual modules\.networks\.

Implementation Details:The proposed framework is implemented using PyTorch framework\[[49](https://arxiv.org/html/2607.06922#bib.bib49)\]\. We set the start epoch automatically and set the Zero\-Recovery interval to22\. The training scheme for the CIFAR\-10 dataset follows\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], where the initial learning rate is0\.10\.1with a decrease by the factor of1010at50%50\\%and75%75\\%of the total epochs\. And the batch size is set to 64\. For the ImageNet\-100 dataset, following\[[49](https://arxiv.org/html/2607.06922#bib.bib49)\], the batch size is 128, the initial learning rate starts at0\.10\.1for ResNet\-50 and GoogLeNet, and 0\.01 for VGG\-19, and the learning rate decays by a factor of1010every1/31/3of all epochs\. And for both datasets, the optimizer is SGD with a momentum of0\.90\.9and weight decay of1​e−41e^\{\-4\}\. These settings are also used in Fig\.[10](https://arxiv.org/html/2607.06922#S3.F10)\.Due to comparison methods – SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\]and PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]train models for 160 epochs on the CIFAR\-10 dataset and 90 epochs on the ImageNet\-100 dataset, respectively, we also choose these two settings of epochs for CIFAR\-10 and ImageNet\-100\. And the comparison method – NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]trains models for 320 epochs on the CIFAR\-10 dataset and 180 epochs on the ImageNet\-100 dataset totally, we choose these settings for comparison to NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]\. For OTO\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\], the model is gradually compressed during training, which means that when the pre\-defined maximum compression ratio is not reached, the ratio increases as the training epoch increases, and when the pre\-defined maximum compression ratio is reached, the ratio no longer increases\. Thus, we first use OTO to train models for 160 epochs on the CIFAR\-10 dataset and 90 epochs on the ImageNet\-100 dataset\. If it meets the pre\-defined compression ratio, we call it OTO\-160/OTO\-90 and compare it to methods that used the same training epochs\. If it fails to meet the pre\-defined compression ratio, we double its training epochs \(named OTO\-320/OTO\-180\) and compare it to methods with the currently same training epochs\. Finally, if it still does not meet the pre\-defined compression ratio with the doubled epochs, its result is discarded\.Moreover, we need to compare the accuracy results under a specific latency constraint, hence we use open\-source codes of these comparison methods to evaluate their performance\.

Table 2:Training time of VGG\-19 model on ImageNet\-100 via NVIDIA Quadro GV100 GPU\.MethodEpochsTime/Epoch\(s\)Total Time \(h\)Overhead\(%\)Trad\.Training\[[49](https://arxiv.org/html/2607.06922#bib.bib49)\]9080720\.1750\.00SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]9086821\.7007\.56FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\]9089622\.40011\.03PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]90193348\.325139\.53OTO\-90\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]90118029\.50046\.22Our\-909081620\.4001\.12NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]18078939\.45095\.54OTO\-180\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]180118059\.000192\.44Our\-18018081640\.800102\.23
### 4\.2Training Overhead Evaluation

To demonstrate the high efficiency of our method, we evaluate the overhead of our method and other model pruning methods\[[1](https://arxiv.org/html/2607.06922#bib.bib1),[6](https://arxiv.org/html/2607.06922#bib.bib6),[7](https://arxiv.org/html/2607.06922#bib.bib7),[8](https://arxiv.org/html/2607.06922#bib.bib8),[9](https://arxiv.org/html/2607.06922#bib.bib9)\]\. As indicated in Table[2](https://arxiv.org/html/2607.06922#S4.T2), we train the VGG\-19 model on the NVIDIA®Quadro®GV100 GPU using different methods, with the dataset ImageNet\-100 and batch size set to128128\.Trad\. Trainingrefers to traditional training that does not use a compact algorithm\. NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]first pre\-trains a full VGG\-19 model before pruning this model\. Then, it trains the small model after pruning for better accuracy\. Thus, the average time cost per epoch of NS is less than that of Trad\. Training\. According to Table[2](https://arxiv.org/html/2607.06922#S4.T2), the overhead ofOur\-90is only1\.12%1\.12\\%\. Thus, under the same training epochs, the overhead of our compact learning can be neglected\. However, the training overhead of other methods is much larger than that of our method\. For example, the training overhead of PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]is about139\.53%139\.53\\%, despite having the same training epochs as Trad Training\. Moreover, its training overhead is even higher than that ofOur\-180, whose training epochs are double that of Trad\. Training\. In the following experiments, we place methods according to the number of training epochs\. However, we need to note that even though PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]achieves the highest accuracy among all methods with the same training epochs in some cases, its training time is about double that of other methods\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x11.png)Figure 11:Comparison of compact models on CIFAR\-10\. Baseline means original models without compression\. OmitOurLPif no layer pruning\.
### 4\.3Accuracy Evaluation

We first evaluate our core compact learning in terms of accuracy on these datasets\. This method can also compress a model by a specific compact ratio, only changing theC​o​m​p​\_​r​a​t​i​oComp\\\_ratioto a constant \(Line 10of Algorithm[1](https://arxiv.org/html/2607.06922#alg1)\)\. Since the edge systems have a strict latency constraint, unlike other latency\-agnostic approaches, larger compression ratios are required to verify the resilience of all methods\. We employ a two aggressive compression ratio of 50% \(denoted as Comp\-50%\) and 70% \(denoted as Comp\-70%\) for evaluation\.

Results on the CIFAR\-10 Dataset:Fig\.[11](https://arxiv.org/html/2607.06922#S4.F11)indicates that the our compression method achieves the highest accuracy \(Acc\.\)compared to NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\], PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]and OTO\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]under different compression ratios for each model\. In comparison to SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\], PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]and OTO\-160\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\],all algorithms train for 160 epochs without fine\-tuning, and our method, marked asOur\-160, can minimize the loss of accuracy for both 50% and 70% compression ratio\. Compared to NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], which needs 160 epochs of pre\-training and 160 epochs of fine\-tuning,and OTO\-320\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\],Our\-160can obtain a compact model with half the time cost and maintain a smaller loss of accuracy under both compression ratios\. In addition, if our method trains for the same number of epochs as NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]and OTO\-320\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]do, denoted asOur\-320, it can achieve higher accuracy than that of the original model, even at a compression ratio of 70%, which shows the superiority of our framework\. For layer pruning, although, in general, the accuracy of layer pruning is lower than that of channel pruning under the same compression ratio\[[50](https://arxiv.org/html/2607.06922#bib.bib50)\], our method, denoted asOurLP, can achieve similar accuracy asOurmethods\. Also, compared to other channel pruning methods\[[1](https://arxiv.org/html/2607.06922#bib.bib1),[6](https://arxiv.org/html/2607.06922#bib.bib6),[7](https://arxiv.org/html/2607.06922#bib.bib7),[8](https://arxiv.org/html/2607.06922#bib.bib8)\], its accuracy is higher than theirs\.

Table 3:Comparison of compact models on ImageNet\-100\. OmitOurLPif no layer pruning\.Results on the ImageNet\-100 Dataset:Table[3](https://arxiv.org/html/2607.06922#S4.T3)indicates that on a large\-scale dataset, the proposed frameworkcan also obtain the highest accuracy compared to NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\], PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]and OTO\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]for most network architectures\.All lowest accuracy reduction \(or highest accuracy increase\) are marked inboldface\.When applied to large models like VGG\-19 and compared to SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\], and PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\], all methods are trained for 90 epochs without fine\-tuning, our method, denoted asOur\-90, can achieve the highest accuracy under both 50% and 70% compression ratios\. In the case of middle models such as ResNet\-50, the average performance ofOur\-90at 50% and 70% compression ratios is comparable to that of PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]and is better than SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]and FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\]\. But the training time ofOur\-90is much less than that of PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]\(see Section[4\.2](https://arxiv.org/html/2607.06922#S4.SS2)\)\. And for small models like GoogLeNet, under the small compression ratio like 50%,Our\-90can still minimize the loss of accuracy\. But if 70% of channels are pruned, the accuracy drop with all methods is large, as the redundancy of GoogLeNet is low\. Compared to NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]with fine\-tuning and OTO\-180\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\], when trained for the same number of epochs as theirs, our method, denoted asOur\-180, can further minimize the loss of accuracy, and even outperform that of the original model, like VGG\-19 and ResNet\-50 under a compression ratio of 50%\. When considering the training time,Our\-180can achieve higher accuracy than that of PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]with less time\.Meanwhile, combined with layer pruning, our method, denoted asOurLP, can achieve similar accuracy as originalOurmethods, which means our layer pruning also works well for large datasets\.

![Refer to caption](https://arxiv.org/html/2607.06922v1/x12.png)Figure 12:Model architecture from layer pruning of VGG\-19\.Model ais for Cifar\-10 andModel bis for ImageNet\-100\. The vertical line represents the pooling layer, the rectangle represents the Conv\-BN\-ReLU layer, and the number in the rectangle represents the number of channels\.Layer Pruning Architecture:We visualize the layer pruned model architecture \(eliminating 50% channels\) of VGG\-19 by our method under different datasets in Fig\.[12](https://arxiv.org/html/2607.06922#S4.F12), from which we observe that our engine encourages to train the desired architecture that goes deeper for a larger dataset\.

### 4\.4Latency\-Accuracy Evaluation

To demonstrate the effectiveness of our method for latency\-critical edge systems, we evaluate the accuracy of our proposed framework and state\-of\-the\-art methods under the34​m​s34mslatency constraint, which is equal to the recognition speed of human eyes –3030frames per second \(FPS\)\. Table[4](https://arxiv.org/html/2607.06922#S4.T4)presents the results of proposed learning method compared with NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\], SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\], FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\],PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\], OTO\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\], Eff\-Compd\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\], and HACScale\[[11](https://arxiv.org/html/2607.06922#bib.bib11)\],in whichQ\-flag means with quantization,S\-flag means with model scaling andLP\-flag means with layer pruning\.The originally proposed compact learning method, denoted asOur, under the same training epoch as the original model, already achieves similar accuracy to the most recent works for all these three DNN models with much less training time \(See Section[4\.2](https://arxiv.org/html/2607.06922#S4.SS2)\)\. When equipped with layer pruning, denoted asOurLP, its accuracy is better, and it can achieve the highest accuracy for most models\. Layer pruning gives our framework an opportunity to find a more efficient architecture under the latency constraint\. Particularly, when our method is trained for the same epoch as NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]and OTO\-180\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\], our method can exceed the accuracy of the uncompressed model a lot, whether equipped with layer pruning or not\.

For theOurQmethod, quantization can reduce the latency, hence we can use a smaller compression ratio in our compact learning\. And after quantization, the compact model still meets the same latency constraint\. It can achieve higher accuracy than the originalOurmethod and obtain the best accuracy for all the three DNN architectures on different hardware platforms\.

For theOurSmethod, in these three DNN models, only the uncompressed GoogLeNet inference on NVIDIA Jetson TX2 consumes less time than the latency constraint, hence we uniformly scale up the model from width and depth, then compact it to meet the latency constraint by our compact learning\. With the same training epochs to the unified scaling, Eff\-Compd\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\],and HACScale\[[11](https://arxiv.org/html/2607.06922#bib.bib11)\],we can achieve the highest accuracy\.Layer pruning also provides our model scaling method to endue different implicit scaling factors \(the number of non\-zeroγ\\gammain Zero training\) to depth and width, and these scaling factors are dynamically optimized during the learning process, this is the reason that our accuracy is higher than other methods\.

Additionally, each model in Table[4](https://arxiv.org/html/2607.06922#S4.T4)achieves a latency in the range of33\.2±0\.7​m​s33\.2\\pm 0\.7ms, however, the compression ratio \(negative ratios indicate scaling\) of each model is not proportional to the reduction of latency\. Meanwhile, the latency drop only has a weak relationship to the FLOPs drop\. For example, the FLOPs drop of compact VGG\-19 model under SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]method is about81\.06%81\.06\\%, which that of NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]method is about76\.59%76\.59\\%, but they achieve the same latency\. This is good proof of the necessity of the proposed framework to directly optimize latency for latency\-critical edge systems\. Furthermore, different methods have different attempt times to meet the latency constraint\. For other methods, we used dichotomy to find the approximate compression/scaling ratio, but they still need about6−76\-7timesto meet the specific latency constraint\. However, with the assistance of the latency predictor, our method only needs one time\. Therefore, a latency predictor is also critical in the DNN optimization for latency\-critical edge systems\.

Table 4:Comparison of optimized models on ImageNet\-100 \(s\.t\. 34ms\)\. OmitOurLPif no layer pruning\.ModelAccuracy \(%\)DevicesMethodTop\-1Top\-5Comp\.Ratio \(%\)SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]81\.224\.0294\.941\.2257\.00FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\]82\.173\.0795\.200\.9657\.00PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]82\.902\.3394\.981\.1855\.00Our\-9083\.182\.0695\.380\.7860\.69OurQ\-9083\.441\.8095\.400\.7656\.78OurLP\-9085\.74\-0\.5096\.66\-0\.5052\.14NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]80\.145\.1094\.221\.9456\.00Our\-18085\.78\-0\.5496\.68\-0\.5260\.90OurQ\-18086\.64\-1\.4096\.66\-0\.5055\.98VGG\-19119\.98OurLP\-18087\.10\-1\.8696\.74\-0\.5853\.16SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]83\.601\.0095\.121\.0642\.00FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\]83\.920\.6895\.960\.2242\.00PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]84\.040\.5695\.280\.9040\.00Our\-9083\.800\.8095\.580\.6041\.02OurQ\-9084\.180\.4296\.000\.1838\.99OurLP\-9083\.740\.8695\.181\.0037\.44NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]80\.084\.5294\.042\.1445\.00OTO\-180\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]84\.70\-0\.1096\.26\-0\.0837\.00Our\-18086\.82\-2\.2296\.36\-0\.1840\.89ResNet\-5049\.06OurQ\-18087\.04\-2\.4496\.80\-0\.6239\.27Unified85\.44\-0\.1296\.42\-0\.04\-37\.51Eff\-compd\[[10](https://arxiv.org/html/2607.06922#bib.bib10)\]85\.52\-0\.2096\.45\-0\.07N\.A\.HACScale\[[11](https://arxiv.org/html/2607.06922#bib.bib11)\]85\.62\-0\.3096\.43\-0\.05N\.A\.OurSLP\-9085\.64\-0\.3296\.48\-0\.10\-23\.76OurSQLP\-9086\.10\-0\.7896\.48\-0\.10\-33\.76OurSLP\-18087\.46\-2\.1497\.02\-0\.64\-24\.28NVIDIAJetsonTX2GoogLeNet20\.27OurSQLP\-18088\.10\-2\.7897\.12\-0\.74\-35\.30SFP\[[6](https://arxiv.org/html/2607.06922#bib.bib6)\]84\.241\.0895\.800\.5816\.00FPGM\[[7](https://arxiv.org/html/2607.06922#bib.bib7)\]84\.960\.3695\.920\.4616\.00PGMPF\[[8](https://arxiv.org/html/2607.06922#bib.bib8)\]85\.200\.1296\.220\.1617\.00OTO\-90\[[9](https://arxiv.org/html/2607.06922#bib.bib9)\]84\.360\.9695\.820\.5618\.00Our\-9085\.180\.1496\.66\-0\.2817\.08OurQ\-9085\.280\.0496\.44\-0\.0615\.86NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]85\.060\.2695\.900\.4817\.00Our\-18086\.86\-1\.5496\.94\-0\.5617\.10NVIDIAJetsonNanoGoogLeNet40\.32OurQ\-18086\.92\-1\.6096\.88\-0\.5016\.11

## 5Conclusion

In this work, we have presented a neural network learning framework for latency\-critical edge systems to directly optimize the latency of DNN on a specific edge device\. This framework mainly includes three parts: a compact learning scheme, a latency predictor and a scaling scheme\. From our experiments, the latency predictor is accurate enough with only a 6\.12% error\. The proposed Zero\-Recovery compact learning scheme can achieve less accuracy drop even under a large compression ratio, and it can increase the accuracy if it is trained with the same epochs as NS\[[1](https://arxiv.org/html/2607.06922#bib.bib1)\]on both datasets\. Compared to the typical 3\-stage pipeline pruning methods, our compact learning method can achieve higher accuracy with less time consumption\. In addition, we evaluated our full framework under a latency constraint of3434ms, and the results show the efficiency and necessity of our method\. Equipped with model scaling, our method can effectively adjust all DNN models to satisfy the system latency constraint\. It solves the mismatch problem between state\-of\-the\-art models and most edge devices\. For example, we can compact VGG\-19 model from119\.98119\.98ms to3434ms and we can also scale GoogLeNet model from20\.2720\.27ms to3434ms\. Compared to other methods, our approach can obtain the highest accuracy with less training time\. Meanwhile, for the conventional hardware\-agnostic optimization methods, generally, they need to adjust the indirect parameters iteratively and evaluate on the real platform repeatedly to satisfy the latency constraint, which causes a large overhead\. Our latency predictor greatly speeds up this process\. Furthermore, our framework provides an optimization method for deploying machine learning on edge systems, it is not only suitable for latency but also suitable for energy and memory\.

## CRediT authorship contribution statement

Shuo Huai:Conceptualization, Methodology, Software, Writing – original draft\.Di Liu:Validation, Data curation, Writing \- Review & Editing\.Hao Kong:Data curation, Writing \- Review & Editing\.Ravi Subramaniam:Writing \- Review & Editing, Resources\.Christian Makaya:Writing \- Review & Editing, Resources\.Qian Lin:Writing \- Review & Editing, Resources\.Weichen Liu:Writing \- Review & Editing, Supervision, Project administration\.

## Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Weichen Liu reports financial support was provided by National Research Foundation \(NRF\) Singapore\.

## Acknowledgment

This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects \(IAF\-ICP\) Funding Initiative, as well as cash and in\-kind contribution from the industry partner, HP Inc\., through the HP\-NTU Digital Manufacturing Corporate Lab \(I1801E0028\)\.

## References

- \[1\]Z\. Liu, J\. Li, Z\. Shen, G\. Huang, S\. Yan, C\. Zhang, Learning efficient convolutional networks through network slimming, in: ICCV, 2017\.
- \[2\]S\. Pouyanfar, S\. Sadiq, Y\. Yan, H\. Tian, Y\. Tao, M\. P\. Reyes, M\.\-L\. Shyu, S\.\-C\. Chen, S\. S\. Iyengar, A survey on deep learning: Algorithms, techniques, and applications, ACM Comput\. Surv\. 51 \(5\) \(2018\)\.
- \[3\]C\. Szegedy, W\. Liu, Y\. Jia, P\. Sermanet, S\. Reed, D\. Anguelov, D\. Erhan, V\. Vanhoucke, A\. Rabinovich, Going deeper with convolutions, in: CVPR, 2015\.
- \[4\]B\. Zoph, Q\. V\. Le, Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611\.01578 \(2016\)\.
- \[5\]A\. Wan, X\. Dai, P\. Zhang, Z\. He, Y\. Tian, S\. Xie, B\. Wu, M\. Yu, T\. Xu, K\. Chen, et al\., Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp\. 12965–12974\.
- \[6\]Y\. He, G\. Kang, X\. Dong, Y\. Fu, Y\. Yang, Soft filter pruning for accelerating deep convolutional neural networks, in: IJCAI, 2018, pp\. 2234–2240\.
- \[7\]Y\. He, P\. Liu, Z\. Wang, Z\. Hu, Y\. Yang, Filter pruning via geometric median for deep convolutional neural networks acceleration, in: CVPR, 2019\.
- \[8\]L\. Cai, Z\. An, C\. Yang, Y\. Yan, Y\. Xu, Prior gradient mask guided pruning\-aware fine\-tuning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol\. 1, 2022\.
- \[9\]T\. Chen, B\. Ji, T\. Ding, B\. Fang, G\. Wang, Z\. Zhu, L\. Liang, Y\. Shi, S\. Yi, X\. Tu, Only train once: A one\-shot neural network training and pruning framework, Advances in Neural Information Processing Systems 34 \(2021\) 19637–19651\.
- \[10\]M\. Tan, Q\. Le, Efficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning, PMLR, 2019, pp\. 6105–6114\.
- \[11\]H\. Kong, D\. Liu, X\. Luo, W\. Liu, R\. Subramaniam, Hacscale: Hardware\-aware compound scaling for resource\-efficient dnns, in: 2022 27th Asia and South Pacific Design Automation Conference \(ASP\-DAC\), IEEE, 2022, pp\. 708–713\.
- \[12\]X\. Dai, P\. Zhang, B\. Wu, H\. Yin, F\. Sun, Y\. Wang, M\. Dukhan, Y\. Hu, Y\. Wu, Y\. Jia, P\. Vajda, M\. Uyttendaele, N\. K\. Jha, Chamnet: Towards efficient network design through platform\-aware model adaptation, in: CVPR, 2019\.
- \[13\]S\. Han, J\. Pool, J\. Tran, W\. Dally, Learning both weights and connections for efficient neural network, in: NIPS, Curran Associates, Inc\., 2015, pp\. 1135–1143\.
- \[14\]Z\. Liu, M\. Sun, T\. Zhou, G\. Huang, T\. Darrell, Rethinking the value of network pruning, in: ICLR, 2019\.
- \[15\]R\. Zhao, Y\. Hu, J\. Dotzel, C\. De Sa, Z\. Zhang, Improving neural network quantization without retraining using outlier channel splitting, in: International Conference on Machine Learning, 2019\.
- \[16\]G\. Hinton, O\. Vinyals, J\. Dean, Distilling the knowledge in a neural network, arXiv preprint arXiv:1503\.02531 \(2015\)\.
- \[17\]C\. Louizos, M\. Welling, D\. P\. Kingma, Learning sparse neural networks throughl0l\_\{0\}regularization, in: ICLR, 2018\.
- \[18\]N\. Bjorck, C\. P\. Gomes, B\. Selman, K\. Q\. Weinberger, Understanding batch normalization, in: NIPS, 2018, pp\. 7694–7705\.
- \[19\]J\. Choi, P\. I\.\-J\. Chuang, Z\. Wang, S\. Venkataramani, V\. Srinivasan, K\. Gopalakrishnan, Bridging the accuracy gap for 2\-bit quantized neural networks \(qnn\), arXiv preprint arXiv:1807\.06964 \(2018\)\.
- \[20\]S\. Wu, G\. Li, F\. Chen, L\. Shi, Training and inference with integers in deep neural networks, arXiv preprint arXiv:1802\.04680 \(2018\)\.
- \[21\]A\. Zhou, A\. Yao, Y\. Guo, L\. Xu, Y\. Chen, Incremental network quantization: Towards lossless cnns with low\-precision weights, arXiv preprint arXiv:1702\.03044 \(2017\)\.
- \[22\]S\. Han, H\. Mao, W\. J\. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510\.00149 \(2015\)\.
- \[23\]W\. Sung, S\. Shin, K\. Hwang, Resiliency of deep neural networks under quantization, arXiv preprint arXiv:1511\.06488 \(2015\)\.
- \[24\]S\. Shin, K\. Hwang, W\. Sung, Fixed\-point performance analysis of recurrent neural networks, in: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\), IEEE, 2016, pp\. 976–980\.
- \[25\]S\. Migacz, 8\-bit inference with tensorrt, in: GPU technology conference, Vol\. 2, 2017, p\. 5\.
- \[26\]M\. Raghu, B\. Poole, J\. Kleinberg, S\. Ganguli, J\. Sohl\-Dickstein, On the expressive power of deep neural networks, in: international conference on machine learning, PMLR, 2017, pp\. 2847–2854\.
- \[27\]T\.\-J\. Yang, A\. Howard, B\. Chen, X\. Zhang, A\. Go, M\. Sandler, V\. Sze, H\. Adam, Netadapt: Platform\-aware neural network adaptation for mobile applications, in: ECCV, 2018\.
- \[28\]B\. Wu, X\. Dai, P\. Zhang, Y\. Wang, F\. Sun, Y\. Wu, Y\. Tian, P\. Vajda, Y\. Jia, K\. Keutzer, Fbnet: Hardware\-aware efficient convnet design via differentiable neural architecture search, in: CVPR, 2019\.
- \[29\]A\. Gholami, K\. Kwon, B\. Wu, Z\. Tai, X\. Yue, P\. Jin, S\. Zhao, K\. Keutzer, Squeezenext: Hardware\-aware neural network design, in: CVPR Workshop, 2018\.
- \[30\]D\. Justus, J\. Brennan, S\. Bonner, A\. S\. McGough, Predicting the computational cost of deep learning models, in: 2018 IEEE international conference on big data \(Big Data\), IEEE, 2018, pp\. 3873–3882\.
- \[31\]L\. L\. Zhang, S\. Han, J\. Wei, N\. Zheng, T\. Cao, Y\. Yang, Y\. Liu, nn\-meter: towards accurate latency prediction of deep\-learning model inference on diverse edge devices, in: Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, 2021, pp\. 81–93\.
- \[32\]S\. Huai, L\. Zhang, D\. Liu, W\. Liu, R\. Subramaniam, Zerobn: Learning compact neural networks for latency\-critical edge systems, in: 2021 58th ACM/IEEE Design Automation Conference \(DAC\), IEEE, 2021, pp\. 151–156\.
- \[33\]H\. Li, A\. Kadav, I\. Durdanovic, H\. Samet, H\. P\. Graf, Pruning filters for efficient convnets, arXiv preprint arXiv:1608\.08710 \(2016\)\.
- \[34\]P\. Molchanov, S\. Tyree, T\. Karras, T\. Aila, J\. Kautz, Pruning convolutional neural networks for resource efficient inference, arXiv preprint arXiv:1611\.06440 \(2016\)\.
- \[35\]G\. Goh, Why momentum really works\. distill \(2017\)\.
- \[36\]L\. Dudziak, T\. Chau, M\. Abdelfattah, R\. Lee, H\. Kim, N\. Lane, Brp\-nas: Prediction\-based nas using gcns, Advances in Neural Information Processing Systems 33 \(2020\) 10480–10490\.
- \[37\]R\. Hecht\-Nielsen, Theory of the backpropagation neural network, in: Neural networks for perception, Elsevier, 1992, pp\. 65–93\.
- \[38\]J\. De Villiers, E\. Barnard, Backpropagation neural nets with one and two hidden layers, IEEE transactions on neural networks 4 \(1\) \(1993\) 136–141\.
- \[39\]E\. Lee, C\.\-Y\. Lee, Neuralscale: Efficient scaling of neurons for resource\-constrained deep neural networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp\. 1478–1487\.
- \[40\]A\. Krizhevsky, Learning multiple layers of features from tiny images \(2009\)\.
- \[41\]K\. Simonyan, A\. Zisserman, Very deep convolutional networks for large\-scale image recognition \(2015\)\.[arXiv:1409\.1556](http://arxiv.org/abs/1409.1556)\.
- \[42\]O\. Russakovsky, J\. Deng, H\. Su, J\. Krause, S\. Satheesh, S\. Ma, Z\. Huang, A\. Karpathy, A\. Khosla, M\. Bernstein, A\. C\. Berg, L\. Fei\-Fei, ImageNet Large Scale Visual Recognition Challenge, IJCV 115 \(3\) \(2015\) 211–252\.
- \[43\]S\. Bianco, R\. Cadene, L\. Celona, P\. Napoletano, Benchmark analysis of representative deep neural network architectures, IEEE access 6 \(2018\) 64270–64277\.
- \[44\]M\. Abadi, et al\., TensorFlow: Large\-scale machine learning on heterogeneous systems, software available from tensorflow\.org \(2015\)\.
- \[45\]Z\. Cao, M\. Long, J\. Wang, P\. S\. Yu, Hashnet: Deep learning to hash by continuation, in: ICCV, 2017\.
- \[46\]F\. Cakir, K\. He, S\. Sclaroff, Hashing with binary matrix pursuit, in: ECCV, 2018\.
- \[47\]K\. He, X\. Zhang, S\. Ren, J\. Sun, Deep residual learning for image recognition, in: CVPR, 2016\.
- \[48\]G\. Huang, Z\. Liu, L\. van der Maaten, K\. Q\. Weinberger, Densely connected convolutional networks, in: CVPR, 2017\.
- \[49\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Kopf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, S\. Chintala, Pytorch: An imperative style, high\-performance deep learning library, in: NIPS, 2019, pp\. 8024–8035\.
- \[50\]H\. Mao, et al\., Exploring the granularity of sparsity in convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp\. 13–20\.