Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs

# Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs
Source: [https://arxiv.org/html/2607.22786](https://arxiv.org/html/2607.22786)
###### Abstract

In this work, we explore how the inference time of a Transformer Neural Network can be efficiently optimized with applications to real\-time anomaly detection in financial time series\. The financial time series are price series such as asset prices\. Unfortunately, the data is often with errors or outliers that make the downstream data processing tasks useless, unstable or even harmful\[[1](https://arxiv.org/html/2607.22786#bib.bib1)\]\[[2](https://arxiv.org/html/2607.22786#bib.bib2)\]\. Moreover, the amount of financial time\-series data has been significantly increasing\[[3](https://arxiv.org/html/2607.22786#bib.bib3)\]\. Hence, there is a need for better data\-cleaning methods in terms of accuracy and in terms of processing speed\.

Transformers as a neural network architecture have achieved superior performances in many tasks such as Natural Language Processing and Computer Vision\[[4](https://arxiv.org/html/2607.22786#bib.bib4)\]\. Time series modelling and especially anomaly detection tasks can benefit from the features of transformers architecture in multiple ways, including the capacity to capture long\-range dependencies and interactions\[[5](https://arxiv.org/html/2607.22786#bib.bib5)\]\.

Increasingly powerful hardware, such as field\-programmable gate arrays \(FPGAs\), have seen increasing usage in recent years due to their reconfigurability and high performance\[[6](https://arxiv.org/html/2607.22786#bib.bib6)\]\. They can be efficiently utilized to speed up the computations of the Transformer architecture\.

We explore different Transformer architectures for time series modelling and how they can be efficiently implemented on an FPGA board \(PYNQ\-Z2\)\. In particular, we examine the application of Transformers to detect anomalies in time series and we show how they can be efficiently implemented on an FPGA board to minimize latency\.

The code is available at[https://github\.com/thxi/icl\_thesis](https://github.com/thxi/icl_thesis)

## 1Introduction

The rapid growth of financial time series data has given rise to significant challenges in maintaining data quality and processing speed\. In this study, we address these challenges by proposing an implementation of Transformer\-based Neural Networks which achieve good performance in anomaly detection tasks while having a low inference time\.

Financial time series data often suffer from data inaccuracies, errors, and outliers, rendering downstream data processing tasks ineffective or even detrimental to decision\-making processes\. The urgency of this issue is underscored by the ever\-increasing volume of financial time\-series data available\. Consequently, there is a pressing demand for more accurate and faster data\-cleaning methods\.

Transformers, as a powerful neural network architecture, have consistently demonstrated exceptional performance across various domains, including Natural Language Processing \(NLP\) and Computer Vision\. Leveraging the capabilities of Transformer architectures for time series modeling, especially in the realm of anomaly detection, offers numerous advantages, including the ability to capture long\-range dependencies and intricate interactions within the data\.

The evolution of hardware, for example, the growing utilization of field\-programmable gate arrays \(FPGAs\), has brought about increased computational capabilities due to their reconfigurability, high performance and deterministic latency\. These FPGAs can be effectively used to accelerate computations associated with Transformer architectures\.

### 1\.1Outline

In Section[2](https://arxiv.org/html/2607.22786#S2), we will describe the problem statement of anomaly detection task\. In Section[3](https://arxiv.org/html/2607.22786#S3), we will describe the main concepts and ideas used in the Transformer architecture\. In Section[4](https://arxiv.org/html/2607.22786#S4), we will describe the main concepts of programming an FPGA and the specific optimizations that can be applied to speed up the computations\. In Section[5](https://arxiv.org/html/2607.22786#S5), we will describe the experiments that we conducted to evaluate the performance of the proposed architectures\. Section[6](https://arxiv.org/html/2607.22786#S6)will conclude this work and will provide some ideas for future work\.

### 1\.2Contributions

The main contributions of this paper are as follows:

1. 1\.Implementation of vanilla Transformer architecture on an FPGA board\.
2. 2\.Implementation of a Linear Attention Transformer on an FPGA board\.
3. 3\.Analysis of the latency/resource utilization tradeoff for the proposed architectures\.
4. 4\.Comparative analysis of the performance, related to accuracy, of the proposed architectures on the anomaly detection task\.

## 2Problem statement

In this section, we will describe the problem statement of anomaly detection task\.

### 2\.1Definitions and examples

###### Definition 2\.1\.

We consider atime\-series𝒯\\mathcal\{T\}which is simply a timestamped sequence of observationsxi∈Rnx\_\{i\}\\in R^\{n\}, potentially with some metadatayiy\_\{i\}\.

Most of the times we will consider univariate case, i\.e\.n=1n=1\. An example of this is a price time\-series of a single stock\. However, the multivariate case is also important and we will consider it in the experiments\. For example, one can consider a time\-series of prices of multiple stocks to get a multivariate time series or one can also create features \(for example, by calculating thedifferenced time series\) on the original time\-series to get a multivariate time\-series\.

###### Definition 2\.2\.

Thedifferenced time seriesis a time\-series𝒯\\mathcal\{T\}where each observationxix\_\{i\}is simply the difference between the current and the previous observation\. That is,xi=xi−1−xix\_\{i\}=x\_\{i\-1\}\-x\_\{i\}\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/x1.png)Figure 1:Sample mutli\-variate time\-series of Apple stock price\. The blue line are the close prices and the red line is the differenced close prices\.###### Definition 2\.3\.

TheAnomaly Detectiontask: for any time\-series𝒯^\\hat\{\\mathcal\{T\}\}of lengthnn, we need to predict𝒴=\{y1,…,yn\},yi∈\{0,1\}\\mathcal\{Y\}=\\\{y\_\{1\},\.\.\.,y\_\{n\}\\\},y\_\{i\}\\in\\\{0,1\\\}, whether the datapoint at theii\-th timestamp anomalous \(where by convention we will use11as anomaly and0as not an anomaly\)\.

In this work, we will restrict ourselves to thesupervised casewhere the labelsyiy\_\{i\}are known for the*seen*\(or training\) part of the dataset\.

### 2\.2Different Anomalies Types

Although there is no universally accepted definition of an anomaly, there exists a common classification ofsyntheticanomalies into different types\. For a broad comprehensive review, reader is referred to\[[8](https://arxiv.org/html/2607.22786#bib.bib8)\],\[[9](https://arxiv.org/html/2607.22786#bib.bib9)\]\.

In this work, we will only considerPoint anomaliestypes since the main focus of this work is not to show the superior performance of the Transformer architecture on different types of anomalies but to show that it can be efficiently implemented on an FPGA board\.

###### Definition 2\.5\.

Point anomaliesare individual data points that are considered anomalous with respect to the rest of the data\. For example, a point anomaly can be a sudden spike in the price of a stock\. See Figure[1](https://arxiv.org/html/2607.22786#S2.F1)\.

Mathematically, we could define a point anomaly as follows:

xi=\{xioriginal\+Syi=1xoriginalotherwisex\_\{i\}=\\begin\{cases\}x^\{\\text\{original\}\}\_\{i\}\+S&y\_\{i\}=1\\\\ x^\{\\text\{original\}\}&\\text\{otherwise\}\\end\{cases\}\(2\.1\)wherexoriginalx^\{\\text\{original\}\}is the original time\-series \(as if there was no outliers\) andSSis the spike\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/x2.png)Figure 2:Apple stock modified time series with injectedPoint anomalies\. Top panel: price time series and its distribution\. Bottom panel: differenced price time series and its distribution\.

## 3Transformers

In this section, we will describe the main concepts and ideas used in the Transformer architecture\. and how to use them for anomaly detection time series task\. We will describe the main building blocks of the Transformer architecture and will give a special treatment to the attention mechanism firstly introduced in\[[10](https://arxiv.org/html/2607.22786#bib.bib10)\]\.

### 3\.1General architecture

In\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\], authors introduced thevanillaTransformer architecture which a neural network architecture which is the architecture that is dominantly used in Natural Language Processing tasks\. The architecture’s main feature was reliance on the attention mechanism and the complete elimination of recurrent and convolutional layers\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/attention_is_all_you_need.png)Figure 3:Model architecture of the Transformer\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\]Figure[3](https://arxiv.org/html/2607.22786#S3.F3)presents the main architecture of the transformer\. The architecture consists of anencoderand adecoderblocks\. For the purpose of this work we will only consider theencoderpart of the architecture \(and in that regard the idea is similar to BERT model for language processing\[[12](https://arxiv.org/html/2607.22786#bib.bib12)\]which only uses a transformer encoder and this work which uses BERT for anomaly detection\[[13](https://arxiv.org/html/2607.22786#bib.bib13)\]\)\. Theencoderis preceded by apositional encodinglayer which is used to*inject*the positional information to the input vectorsxix\_\{i\}because the attention mechanism is permutation invariant, this will be explained in Section[3\.2](https://arxiv.org/html/2607.22786#S3.SS2)\.

#### 3\.1\.1Positional encoding

Since the attention mechanism is permutation invariant \(i\.e\., it does not take into account the order of the input, refer to Section[3\.2](https://arxiv.org/html/2607.22786#S3.SS2)\), we need to inject the positional information into the input vectorsxix\_\{i\}\. In short, a positional encoding is usually a vector of the same dimension as the input vectorsxix\_\{i\}which can either be fixed \(for example, we have the same vector for different positions\) or learned\.

For the comprehensive overview of different positional encoding techniques, reader is referred to\[[14](https://arxiv.org/html/2607.22786#bib.bib14)\]and\[[15](https://arxiv.org/html/2607.22786#bib.bib15)\]\.

The vanilla Transformer architecture uses a sinusoidal positional encoding technique which is a fixed positional encoding scheme\. Suppose we have a time\-series of lengthTT\. Then the positional encoding for theii\-th input vectorxix\_\{i\}of dimensionddis defined as follows:

P​Ei,2​j=sin⁡\(i100002​j/d\)P​Ei,2​j\+1=cos⁡\(i100002​j/d\)\\begin\{array\}\[\]\{rll\}PE\_\{i,2j\}&=\\sin\(\\frac\{i\}\{10000^\{2j/d\}\}\)\\\\ PE\_\{i,2j\+1\}&=\\cos\(\\frac\{i\}\{10000^\{2j/d\}\}\)\\end\{array\}\(3\.1\)Refer to Figure[4](https://arxiv.org/html/2607.22786#S3.F4)for an example of the embedding vectors that are obtained using this method\.

This combination of sine and cosine functions allows the positional encoding to generate the embedding of a position in a unique way\. The embedding is then added to the input vectorxix\_\{i\}, however, it can also be simply stacked with the input vectorxix\_\{i\}to get a vector of dimension2​d2d\. In this work, we will use the former approach\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/x3.png)Figure 4:Positional encoding example embeddings for a sequence of max length of 100 and embedding dimension of 48\. The embeddings are generated using the sinusoidal positional encoding technique\. The columns represent the embeddings that we would add to the input vectorxix\_\{i\}\.
#### 3\.1\.2Encoder Block

The encoder consists ofNNidentical layers\. Each layer has two sub\-layers which are amulti\-head self\-attentionlayer \(described in detail in Section[3\.2\.1](https://arxiv.org/html/2607.22786#S3.SS2.SSS1)\) and afeed\-forward \(FF\)layer\. Thefeed\-forward \(FF\)layerFFN​\(⋅\)\\text\{FFN\}\(\\cdot\)is a simple neural network which comprises of 2 fully\-connected \(FC\) linear layers and an activation function in between them\. Specifically, authors of\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\]used anFClayer with the ReLU activation function

ReLU​\(x\)=max⁡\(0,x\)\\text\{ReLU\}\(x\)=\\max\(0,x\)followed by anotherFClayer without activation function, i\.e\.

FFN​\(x\)=W2⋅R​e​L​U​\(W1⋅x\+b1\)\+b2\\text\{FFN\}\(x\)=W\_\{2\}\\cdot ReLU\(W\_\{1\}\\cdot x\+b\_\{1\}\)\+b\_\{2\}whereW1,W2,b1,b2W\_\{1\},W\_\{2\},b\_\{1\},b\_\{2\}are the weight matrices and bias vectors respectively for theFClayers\.

TheAdd & Normlayer is a residual connection\[[16](https://arxiv.org/html/2607.22786#bib.bib16)\]followed by a layer normalization layers\[[17](https://arxiv.org/html/2607.22786#bib.bib17)\]\. That is,

Add & Norm​\(x\)=LayerNorm​\(x\+Sublayer​\(x\)\)\\text\{Add \\& Norm\}\(x\)=\\text\{LayerNorm\}\(x\+\\text\{Sublayer\}\(x\)\)whereSublayer​\(x\)\\text\{Sublayer\}\(x\)is either the multi\-head self\-attention layer or the feed\-forward layer \(refer to Figure[3](https://arxiv.org/html/2607.22786#S3.F3)\)\.

Residual connections \(i\.e\., adding the input to the output of the layer\), also known as skip connections, are commonly used to avoid the vanishing gradient problem\[[16](https://arxiv.org/html/2607.22786#bib.bib16)\]\. In this case, the residual connection is used to avoid the degradation of the model performance when the model is deep \(i\.e\., when the number of sequential layersNNis large\)\. While we do not use very deep models in this work, we still use residual connections as in the original architecture for more stable training \(refer to Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1)\)\. Moreover, the residual connection is used to propagate the positional information to the next layers as the attention mechanism is permutation invariant \(refer to Section[3\.2](https://arxiv.org/html/2607.22786#S3.SS2)\)\.

The layer normalization layer\[[17](https://arxiv.org/html/2607.22786#bib.bib17)\],LayerNormis used to normalize the output of the residual connection\. The normalization is used to stabilize the training process \(however, it is not strictly necessary, refer to Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1)for our empirical findings\)\.

This constitutes the main building block of the Transformer encoder architecture\. The next section will describe the attention mechanism in detail\.

### 3\.2Attention mechanism

This section will describe the attention mechanism, its variations and the intuition behind it\. Moreover, we will compare different attention mechanism implementations in terms of their computational complexity which is important for fast calculations\. Most importantly, we will describe themulti\-head attentionmechanism which is used in the vanilla Transformer architecture as a continuation of Section[3\.1](https://arxiv.org/html/2607.22786#S3.SS1)\.

#### 3\.2\.1Dot\-Product Attention and Multi\-Head Attention

The attention mechanism introduced in the Transformer architecture\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\]used ascaled dot\-product attention\.

The main idea of thedot\-product attentionmechanism is to compute the mapping of a queryqiq\_\{i\}for each input vectorxix\_\{i\}to a set of key\-value pairs\(kj,vj\)\(k\_\{j\},v\_\{j\}\)\. The queryqiq\_\{i\}, keykik\_\{i\}and valueviv\_\{i\}vectors are simply linear transformations of the input vectorsxix\_\{i\}, i\.e\.,qi=WQ⋅xiq\_\{i\}=W\_\{Q\}\\cdot x\_\{i\},ki=WK⋅xik\_\{i\}=W\_\{K\}\\cdot x\_\{i\},vi=WV⋅xiv\_\{i\}=W\_\{V\}\\cdot x\_\{i\}whereWQW\_\{Q\},WKW\_\{K\}andWVW\_\{V\}are the weight matrices\. The attention mechanism is a weighted sum of the valuesvjv\_\{j\}where the weights are computed as a function of the queryqiq\_\{i\}and the keykjk\_\{j\}\. That is,A​t​t​e​n​t​i​o​n​\(xi\)=∑jαi​j​vjAttention\(x\_\{i\}\)=\\sum\_\{j\}\\alpha\_\{ij\}v\_\{j\}whereαi​j=softmax​\(qi⋅vi\)\\alpha\_\{ij\}=\\text\{softmax\}\(q\_\{i\}\\cdot v\_\{i\}\)is the weight of thejj\-th valuevjv\_\{j\}\. In practice, the attention mechanism is computed for all the queriesqiq\_\{i\}at the same time by utilizing the following expression in matrix form:

Q=WQ⋅X,K=WK⋅X,V=WV⋅X\\begin\{array\}\[\]\{rll\}Q&=W\_\{Q\}\\cdot X,\\\\ K&=W\_\{K\}\\cdot X,\\\\ V&=W\_\{V\}\\cdot X\\end\{array\}\(3\.2\)Attention​\(Q,K,V\)=softmax​\(Q​KT⋅1dk\)​V\\text\{Attention\}\(Q,K,V\)=\\text\{softmax\}\(QK^\{T\}\\cdot\\frac\{1\}\{\\sqrt\{d\_\{k\}\}\}\)V\(3\.3\)
Figure[3](https://arxiv.org/html/2607.22786#S3.F3)visualizes the attention mechanism introduced in\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\]\.

Themulti\-head attentionmechanism is simply a concatenation of multiple attention mechanisms\. That is, we can computehhdifferent attention mechanisms in parallel and then concatenate the results\. The main idea behind this is that different attention mechanisms can learn different features of the input vectors\. After the concatenation, we apply a linear transformation \(i\.e\., a fully\-connected layer\) to the concatenated vectors to get the final output with a desired dimensionality which is usually the same as the input vectorsxix\_\{i\}\.

However, in the main paper\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\], authors implemented the multi\-head attention mechanism in a more efficient way compared to the naive concatenation of multiple attention heads of sizedkd\_\{k\}\. They, instead, linearly transformed the input vectors to the dimension ofdk​hd\_\{k\}\\/hwherehhis the number of attention heads and applied the attention mechanism to the transformed vectors\. This is more efficient because we require less parameters and can benefit from the increased accuracy of having multiple attention heads\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/scaled_dot_product_attention.png)Figure 5:Scaled Dot\-Product Attention and Multi\-Head Attention\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\]
#### 3\.2\.2Linear attention

In\[[18](https://arxiv.org/html/2607.22786#bib.bib18)\], authors propose an extension to the dot\-product attention mechanism calledlinear attention\. This significantly reduces the computational complexity of the attention mechanism by eliminating the need to compute the softmax function\.

Notice that in[3\.3](https://arxiv.org/html/2607.22786#S3.E3), the softmax function is applied rowwise to the matrixQ​KTQK^\{T\}\. The softmax function can be substituted with a general similarity functionsim​\(⋅,⋅\)\\text\{sim\}\(\\cdot,\\cdot\)between a queryqiq\_\{i\}and a keykjk\_\{j\}\. The equation[3\.3](https://arxiv.org/html/2607.22786#S3.E3)for output valuevi′v^\{\\prime\}\_\{i\}can then be rewritten as follows:

vi′=Attention​\(Q,K,V\)i=∑jsim​\(qi,kj\)​vj∑jsim​\(qi,kj\)v^\{\\prime\}\_\{i\}=\\text\{Attention\}\(Q,K,V\)\_\{i\}=\\frac\{\\sum\_\{j\}\\text\{sim\}\(q\_\{i\},k\_\{j\}\)v\_\{j\}\}\{\\sum\_\{j\}\\text\{sim\}\(q\_\{i\},k\_\{j\}\)\}\(3\.4\)
The only constrained imposed on the similarity functionsim​\(⋅,⋅\)\\text\{sim\}\(\\cdot,\\cdot\)is that it should be non\-negative for it to define an attention function\. This conveniently includes all kernels\. That issim​\(qi,kj\)=ϕ​\(qi\)T​ϕ​\(kj\)\\text\{sim\}\(q\_\{i\},k\_\{j\}\)=\\phi\(q\_\{i\}\)^\{T\}\\phi\(k\_\{j\}\)whereϕ​\(⋅\)\\phi\(\\cdot\)is a feature map\.

So that given a kernel with a feature mapϕ​\(⋅\)\\phi\(\\cdot\), the attention mechanism can be computed as follows:

vi′=Attention​\(Q,K,V\)i=∑jϕ​\(qi\)T​ϕ​\(kj\)​vj∑jϕ​\(qi\)T​ϕ​\(kj\)v^\{\\prime\}\_\{i\}=\\text\{Attention\}\(Q,K,V\)\_\{i\}=\\frac\{\\sum\_\{j\}\\phi\(q\_\{i\}\)^\{T\}\\phi\(k\_\{j\}\)v\_\{j\}\}\{\\sum\_\{j\}\\phi\(q\_\{i\}\)^\{T\}\\phi\(k\_\{j\}\)\}\(3\.5\)
And we can rewrite the attention mechanism in matrix form as follows:

Attention​\(Q,K,V\)=ϕ​\(Q\)T​ϕ​\(K\)​Vϕ​\(Q\)T​ϕ​\(K\)\\text\{Attention\}\(Q,K,V\)=\\frac\{\\phi\(Q\)^\{T\}\\phi\(K\)V\}\{\\phi\(Q\)^\{T\}\\phi\(K\)\}\(3\.6\)
Regrouping the terms, we get the following expression for the attention mechanism:

Attention​\(Q,K,V\)=ϕ​\(Q\)T​ϕ​\(K\)​Vϕ​\(Q\)T​ϕ​\(K\)\\text\{Attention\}\(Q,K,V\)=\\phi\(Q\)^\{T\}\\frac\{\\phi\(K\)V\}\{\\phi\(Q\)^\{T\}\\phi\(K\)\}\(3\.7\)
which makes it evident that we can compute∑jϕ​\(kj\)​vj\\sum\_\{j\}\\phi\(k\_\{j\}\)v\_\{j\}once and reuse them for all the queriesqiq\_\{i\}which reduces the computational complexity fromO​\(N2\)O\(N^\{2\}\)toO​\(N\)O\(N\)whereNNis the number of input vectors in the attention layer\.

### 3\.3Transformers for time series modelling

Originally, the Transformer architecture was introduced for Natural Language Processing tasks\[[11](https://arxiv.org/html/2607.22786#bib.bib11)\]but has since found applications in other domains and time\-sereis modelling specifically\[[5](https://arxiv.org/html/2607.22786#bib.bib5)\]\. The motivation behind using the Transformer architecture for time series modelling is that they show good performance in many sequence modelling tasks\. While many specialized Transformer architectures were proposed for time series modelling \(e\.g\.,\[[19](https://arxiv.org/html/2607.22786#bib.bib19)\],\[[20](https://arxiv.org/html/2607.22786#bib.bib20)\]\) and specifically deployed in asset management setting\[[21](https://arxiv.org/html/2607.22786#bib.bib21)\], in the current work, we will use the least tuned architecture, i\.e\., the vanilla Transformer architecture and the architecture with linear attention\[[18](https://arxiv.org/html/2607.22786#bib.bib18)\]\. The reason for using the not so specialized architectures is that we want to show that the Transformer can achieve good out of the box performance on the anomaly detection task with minimal tuning and that it can be efficiently implemented on an FPGA board\.

## 4FPGA design

In this section, the main concepts of programming an FPGA will be introduced and the specific optimizations that can be applied to speed up the computations\.

Readers will be introduced to the common optimization techniques and how they are achieved\. An matrix multiplication example will be provided to illustrate the concepts and how the optimizations affect the latency and resource utilization\.

Lastly, the analysis of the specific optimizations that can be applied to the Transformer architecture will be provided\.

### 4\.1Introduction to FPGA programming and development

In this section, we will introduce the main concepts of FPGA programming and development tools\.

#### 4\.1\.1Common Terms

In this section, common terms will be introduced\. The terms will be used throughout Section[4](https://arxiv.org/html/2607.22786#S4)\. It is not required to read all of them at once, but it is recommended to refer to this section when a term is not clear \(i\.e\., only when necessary\)\.

###### Definition 4\.1\.

LUT \(Look\-Up Table\)is a small, fast memory that stores the output of a Boolean function of its inputs\. The LUT is the basic building block of an FPGA and is capable of implementing any logic function of N Boolean variables\.

###### Definition 4\.2\.

BRAM \(Block RAM\)is a dedicated two\-port memory that can be used to store data\.

###### Definition 4\.3\.

DSP \(Digital Signal Processing\)is a specialized hardware unit that is optimized for performing mathematical operations\.

###### Definition 4\.4\.

Clock cycleis the time between two consecutive rising edges of the clock signal\. It is the amount of time between two pulses of an oscillator, a single increment of the central processing unit \(CPU\) clock during which the smallest unit of processor activity is carried out\.

###### Definition 4\.5\.

Latencyis the time between the start of an operation and the moment its results become available or the number of clock cycles required to complete an operation\.Latency of a loopis the number of clock cycles required to complete one iteration of the loop\.

###### Definition 4\.6\.

Throughputis the number of operations that can be completed in a given amount of time\.

###### Definition 4\.7\.

Initiation Interval \(II\)is the number of clock cycles between the start of two consecutive iterations of a loop\. That is, it is the maximum rate \(in clock cycles\) at which loop iterations can be initiated\. In the ideal case, the II is equal to 1 so that we can start a new iteration of the loop every clock cycle\. Initiation interval is different from latency\. The reason for this is pipelining which will be described in Section[4\.1\.2](https://arxiv.org/html/2607.22786#S4.SS1.SSS2.Px1)\. For a visual explanation, see Figure[6](https://arxiv.org/html/2607.22786#S4.F6)\.

###### Definition 4\.8\.

Trip countis simply the number of iterations of a loop\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/iiandlatency.png)Figure 6:Latency vs Initiation Interval illustration\. Source:\[[22](https://arxiv.org/html/2607.22786#bib.bib22)\]##### HLS synthesis

In this section, HLS synthesis will be described\[[23](https://arxiv.org/html/2607.22786#bib.bib23)\]\. It is now the common workflow in the FPGA development because it significantly improves the productivity when working with design\.

HLS synthesis is a methodology that bridges the gap between high\-level programming languages, such as C, C\+\+, or OpenCL, and the low\-level hardware description languages typically used in FPGA design, like Verilog or VHDL\. This approach enables developers to describe their algorithms and functionality at a higher level of abstraction, allowing them to focus on the problem\-solving aspect rather than the intricacies of hardware implementation\.

The HLS process starts with a high\-level description of the desired functionality\. This description can be written in familiar programming languages \(in this case, C\+\+ HLS\), taking advantage of their abstractions and concise syntax\. The HLS tool then performs a series of transformations on this high\-level description to generate an optimized hardware implementation that can be deployed on an FPGA\. The high\-level description is transformed by the HLS tool into a RTL \(Register Transfer Level\) representation, which is the low\-level hardware description that defines the behavior of the FPGA\.

##### Simulation, Cosimulation

In this section, the processes ofsimulationandcosimulationwill be described\. In the context of developing for Field\-Programmable Gate Arrays \(FPGAs\), simulation and cosimulation are two crucial techniques for verifying and testing the functionality of your design before actually programming it onto the FPGA hardware\.

###### Definition 4\.9\.

Simulationis the process of running a software\-based model of your FPGA design on a computer to simulate its behavior\. The process is very similar to running a software program on a computer for the purpose of unit testing certain parts of functionality of your code\[[24](https://arxiv.org/html/2607.22786#bib.bib24)\]\. That is, the simulation does not involve any RTL code and is simply a software simulation of the high\-level description\.

###### Definition 4\.11\.

Cosimulationis a technique that combines simulation of the high\-level description with simulation of the generated RTL description\. This means that the simulations of both the original high\-level code and the RTL representation in parallel, comparing their behavior\. The purpose of cosimulation is to ensure that the high\-level synthesis tool accurately transformed the high\-level description into the desired RTL behavior\.\[[24](https://arxiv.org/html/2607.22786#bib.bib24)\]\.

#### 4\.1\.2Common optimizations

In this section, common optimization techniques and how they are achieved will be introduced\.

It is quite common to process data blocks \(for example, a sequence of samples in anomaly detection\) using for loops\. For loops are usually the main bottleneck in the performance of the design and it is the area where most of the optimizations are applied first\[[23](https://arxiv.org/html/2607.22786#bib.bib23)\]\.

##### Loop Pipelining

Loop pipelining is a technique used in FPGAs programming to optimize the performance of sequential operations within a loop\. It improves the throughput of loops by breaking them down into multiple stages that can execute concurrently\. That is, it allows to start the next iteration of a loop before the current iteration has finished\[[25](https://arxiv.org/html/2607.22786#bib.bib25)\]\. Refer to Figure[7](https://arxiv.org/html/2607.22786#S4.F7)and Figure[6](https://arxiv.org/html/2607.22786#S4.F6)for a visual illustration\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/loop_pipelining.png)Figure 7:Loop pipelining illustration\. Source:\[[25](https://arxiv.org/html/2607.22786#bib.bib25)\]Consider example of pipelining a simple for loop which adds 2 vectors \(\[[25](https://arxiv.org/html/2607.22786#bib.bib25)\]\):

voidtoplevel\(int\*a,int\*b,int\*c,intlen\)\{

vadd:for\(inti=0;i<len;i\+\+\)\{

\#pragmaHLSPIPELINE

c\[i\]=a\[i\]\+b\[i\];

\}

\}

Taking the reference numbers from\[[25](https://arxiv.org/html/2607.22786#bib.bib25)\], assume that len is 20 and that one loop iteration takes 3 clock cycles, then the total latency of the loop is 60 clock cycles without pipelining\. The pipelining pragma\#pragma HLS PIPELINEallows to start the next iteration of the loop before the current iteration has finished\. By default, the pipelining pragma will try to achieve an II of 1 but this can be specified manually by using theIIparameter\. So, the pipelining pragma reduces the latency of the loop to 22 clock cycles\.

In general, the latency of a loop with pipelining is given by the following formula\[[26](https://arxiv.org/html/2607.22786#bib.bib26)\]:

Latency=II⋅\(Trip Count−1\)\+Loop Body Latency\\text\{Latency\}=\\text\{II\}\\cdot\(\\text\{Trip Count\}\-1\)\+\\text\{Loop Body Latency\}\(4\.1\)

##### Loop Unrolling

Loop unrolling is an optimization technique that involves expanding or unwinding loops in code to potentially enable better utilization of hardware resources and/or to minimize control flow \(branching\) in loop iterations\. This technique is not limited to FPGA development but can be particularly useful in optimizing code for FPGA implementations\.

Loop unrolling works by duplicating loop iterations \(read ascopy\-pasting\)\. This allows utilizing more hardware as the loop body is duplicated multiple times and loop iterations will utilize different hardware resources\. This increase in performance \(i\.e\., throughput\) comes at the cost of increased resource utilization\[[25](https://arxiv.org/html/2607.22786#bib.bib25)\]\.

Consider a simple example of a function which multiplies the input vector of length 4 by a constant, 2:

voidtoplevel\(int\*a,int\*b\)\{

smult:for\(inti=0;i<4;i\+\+\)\{

\#pragmaUNROLL

b\[i\]=2\*a\[i\];

\}

\}

The unroll pragma\#pragma UNROLLallows to unroll the loop and execute the loop iterations in parallel\.

The unrolled version of the loop will be equivalent to the following code:

voidtoplevel\(int\*a,int\*b\)\{

b\[0\]=2\*a\[0\];

b\[1\]=2\*a\[1\];

b\[2\]=2\*a\[2\];

b\[3\]=2\*a\[3\];

\}

##### Loop Reordering

Loop reordering optimization is an optimization used to improve the performance by changing the order in which loops are executed\. This optimization is not strictly related to FPGA development and it involves altering the nesting order of loops in a way that improves data locality, cache utilization \(for example, on modern CPU\) and enables usage of SIMD resources\.

In the context of FPGA, loop reordering can be used achieve betterIIwhen we are dealing with pipelining and nested loops \(see Section[4\.1\.2](https://arxiv.org/html/2607.22786#S4.SS1.SSS2.Px1)and Section[4\.1\.3](https://arxiv.org/html/2607.22786#S4.SS1.SSS3.Px3)for an example\)\.

##### Function Inlining

Function inlining optimization technique is not specific to FPGA development only\. The technique is used to improve the performance of a program by reducing the overhead associated with function calls\. Calling a function incurs some overhead in terms of memory and execution time due to the need to set up the function call stack, pass arguments, and jump to the function’s code\. Function inlining aims to eliminate this overhead by replacing a function call with the actual body of the function at the call site\. In other words, the compiler takes the contents of the called function and inserts it directly into the location where the function is called\. This, however, increases the size of the code and, specifically in the case of FPGAs, the resource utilization \(LUTs and FF\)\.

##### Array Partitioning and Reshaping

Partitioning arrays in an FPGA \(Field\-Programmable Gate Array\) refers to the process of dividing a large memory block \(e\.g\., one array\) into smaller sections, often referred to as memory banks or partitions\.

Partitioning allows accessing different parts of the array in parallel so that bottlenecks caused by a single memory interface being overwhelmed with requests can be avoided\.

There are 3 different types of partitioning that can be applied to an array \(Refer to Figure[8](https://arxiv.org/html/2607.22786#S4.F8)\):

1. 1\.Cyclic\. In a cyclic partition, the array is divided blocks of interleaved elements of the original array\.
2. 2\.Block\. In a block partition, the array is divided into non\-overlapping blocks of sequential elements in the original array\. Each block is assigned to a separate memory bank\.
3. 3\.Complete\. The array is split intio intividual elements

![Refer to caption](https://arxiv.org/html/2607.22786v1/array_partition_illust.png)Figure 8:Array partitioning illustration\.The disadvantage of partitioning is that it increases the number of memory interfaces which leads to the increased resource utilization \(i\.e\., more FFs, LUTs are used because each memory block requires separate control logic\)\. This can be partly mitigated by using thearray reshaping\. The difference between partitioning and reshaping is that partitioning creates multiple memory interfaces while reshaping still uses a single memory interface \(i\.e\., all partitions are merged to a single physical memory\)\.

#### 4\.1\.3Example of optimizing matrix multiplication

In this section, we will describe the process of optimizing a matrix multiplication using the techniques described in Section[4\.1\.2](https://arxiv.org/html/2607.22786#S4.SS1.SSS2)\. This section can be treated as a tutorial on how to optimize a simple matrix multiplication\.

Full source code for with all the files can be located invitis\_hls/matmul\_naive\.

##### Naive implementation

The naive implementation of matrix multiplication is simply a triple for loop which directly implements the definition ofC=A⋅BC=A\\cdot Bwhere A, B and C are the matrices andCi,jC\_\{i,j\}are defined as in Equation[4\.2](https://arxiv.org/html/2607.22786#S4.E2)\.

Ci,j=∑k=0NAi,k⋅Bk,jC\_\{i,j\}=\\sum\_\{k=0\}^\{N\}A\_\{i,k\}\\cdot B\_\{k,j\}\(4\.2\)
\#include"matrixmul\.h"

voidmatmul\(mat\_a\_ta\[MAT\_A\_ROWS\]\[MAT\_A\_COLS\],

mat\_b\_tb\[MAT\_B\_ROWS\]\[MAT\_B\_COLS\],

result\_tres\[MAT\_A\_ROWS\]\[MAT\_B\_COLS\]\)\{

loop\_i:

for\(inti=0;i<MAT\_A\_ROWS;i\+\+\)\{

loop\_j:

for\(intj=0;j<MAT\_B\_COLS;j\+\+\)\{

res\[i\]\[j\]=0;

loop\_k:

for\(intk=0;k<MAT\_B\_ROWS;k\+\+\)\{

\#pragmaHLSPIPELINEoff

res\[i\]\[j\]\+=a\[i\]\[k\]\*b\[k\]\[j\];

\}

\}

\}

\}

The Vitis HLS Synthesis Report for the naive implementation is presented in Figure[9](https://arxiv.org/html/2607.22786#S4.F9)\. The data types for matrices used are int32\_t\. Matrix A is of size 3x4 and matrix B has size 4x3\. For this baseline implementation, we are reaching a latency of 205 clock cycles, using 327 FF and 282 LUTs\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/matmul_naive_synth.png)Figure 9:Naive matrix multiplication synthesis report
##### Loop pipelining and unrolling

The baseline code can be optimized by pipelining\. There are three loops that can be pipelined \(loop\_i,loop\_jandloop\_k\)\.

Case 1: Pipeliningloop\_k\.A simple pipelining of the innermost loop leads to the pipelining of Multiply and Accumulate operation \(MAC\) which is the main operation in the loop\. There is no need to partition arrays a and b as memory in arrays a and b only need to supply 1 element per cycle\.

The pipelining leads to decrease in latency to 42 clock cycles at the cost of using 526 FF and 501 LUT which is almost twice as many resources as in the naive implementation\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/matmul_pipek_synth.png)Figure 10:Matrix multiplication with pipelined looploop\_kCase 2: Pipeliningloop\_j\.The pipelining of the middle loop requires partitioning of arrays a byMAT\_A\_COLSand b byMAT\_B\_ROWS\. There areMAT\_A\_COLS\(orMAT\_B\_ROWS\) MAC operations per cycle so that memory of a and b needs to be sufficiently divided to supplyMAT\_B\_ROWSelements per cycle\. Array a is partitioned along the second dimension and array b is partitioned along the first dimension because of the access patterns\. In this example, we use thecompletepartitioning which divides the memory into individual registers\.

\#include"matrixmul\.h"

voidmatmul\(mat\_a\_ta\[MAT\_A\_ROWS\]\[MAT\_A\_COLS\],

mat\_b\_tb\[MAT\_B\_ROWS\]\[MAT\_B\_COLS\],

result\_tres\[MAT\_A\_ROWS\]\[MAT\_B\_COLS\]\)\{

\#pragmaHLSARRAY\_PARTITIONvariable=acompletedim=2

\#pragmaHLSARRAY\_PARTITIONvariable=bcompletedim=1

loop\_i:

for\(inti=0;i<MAT\_A\_ROWS;i\+\+\)\{

\#pragmaHLSPIPELINEoff

loop\_j:

for\(intj=0;j<MAT\_B\_COLS;j\+\+\)\{

inttmp=0;

\#pragmaHLSPIPELINE

loop\_k:

for\(intk=0;k<MAT\_B\_ROWS;k\+\+\)\{

\#pragmaHLSUNROLL

tmp\+=a\[i\]\[k\]\*b\[k\]\[j\];

\}

res\[i\]\[j\]=tmp;

\}

\}

\}

With this addition, the latency is down to 15 clock cycles and the resource usage has increased to 1221 FF and 511 LUTs \(Figure[11](https://arxiv.org/html/2607.22786#S4.F11)\)\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/matmul_pipej_synth.png)Figure 11:Matrix multiplication with pipelined looploop\_jCase 3: Pipeliningloop\_i\.Pipelining the outermost loop requires additional partitioning of array res byMAT\_B\_COLSand full partitioning of array b asloop\_kandloop\_jare getting unrolled\.

\#include"matrixmul\.h"

voidmatmul\(mat\_a\_ta\[MAT\_A\_ROWS\]\[MAT\_A\_COLS\],

mat\_b\_tb\[MAT\_B\_ROWS\]\[MAT\_B\_COLS\],

result\_tres\[MAT\_A\_ROWS\]\[MAT\_B\_COLS\]\)\{

\#pragmaHLSARRAY\_PARTITIONvariable=acompletedim=2

\#pragmaHLSARRAY\_PARTITIONvariable=bcompletedim=0

\#pragmaHLSARRAY\_PARTITIONvariable=rescompletedim=2

loop\_i:

for\(inti=0;i<MAT\_A\_ROWS;i\+\+\)\{

\#pragmaHLSPIPELINE

loop\_j:

for\(intj=0;j<MAT\_B\_COLS;j\+\+\)\{

\#pragmaHLSUNROLL

inttmp=0;

loop\_k:

for\(intk=0;k<MAT\_B\_ROWS;k\+\+\)\{

\#pragmaHLSUNROLL

tmp\+=a\[i\]\[k\]\*b\[k\]\[j\];

\}

res\[i\]\[j\]=tmp;

\}

\}

\}

The latency is down to 9 clock cycles and the resource usage is up to 3819 FF and 1383 LUTs \(Figure[12](https://arxiv.org/html/2607.22786#S4.F12)\)\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/matmul_pipei_synth.png)Figure 12:Matrix multiplication with pipelined looploop\_i
##### Loop reordering

We can also compare how the loop reordering affects the performance of the matrix multiplication and specifically how it compares to the pipelining of theloop\_jin terms of latency and resource usage\.

The loop reordering technique allows us to avoid partitioning matrix a compared to theloop\_jpipelining solution\. However, we now have to partition the output matrix byMAT\_B\_COLS\.

\#include"matrixmul\.h"

voidmatmul\(mat\_a\_ta\[MAT\_A\_ROWS\]\[MAT\_A\_COLS\],

mat\_b\_tb\[MAT\_B\_ROWS\]\[MAT\_B\_COLS\],

result\_tres\[MAT\_A\_ROWS\]\[MAT\_B\_COLS\]\)\{

inttemp\_sum\[MAT\_B\_COLS\];

\#pragmaHLSARRAY\_PARTITIONvariable=bdim=2complete

\#pragmaHLSARRAY\_PARTITIONvariable=resdim=2complete

\#pragmaHLSARRAY\_PARTITIONvariable=temp\_sumdim=1complete

loop\_i:

for\(inti=0;i<MAT\_A\_ROWS;i\+\+\)\{

loop\_k:

for\(intk=0;k<MAT\_B\_ROWS;k\+\+\)\{

\#pragmaHLSPIPELINE

loop\_j:

for\(intj=0;j<MAT\_B\_COLS;j\+\+\)\{

\#pragmaHLSUNROLL

intresult=\(k==0\)?0:temp\_sum\[j\];

result\+=a\[i\]\[k\]\*b\[k\]\[j\];

temp\_sum\[j\]=result;

if\(k==MAT\_B\_ROWS\-1\)\{

res\[i\]\[j\]=result;

\}

\}

\}

\}

\}

From Figure[13](https://arxiv.org/html/2607.22786#S4.F13), the latency is 17 clock cycles and the resource usage is 1031 FF and 611 LUTs\. The performance is worse than the pipelined looploop\_jbut the resource usage is lower for FF and higher for LUTs\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/matmul_loop_reordering.png)Figure 13:Matrix multiplication with loop reorderingSummaryThe pipelining of the outermost loop leads to the best performance with the lowest latency\. However, this comes at the big cost of using the most resources as array b needs to be fully partitioned\. In this work, we will use the pipelined looploop\_jas it provides a good trade\-off between latency and resource usage\.

### 4\.2Transformer architecture optimizations

In this section, we present the specific optimizations that were applied to the Transformer architecture\. For HLS code \(i\.e\., implementation\), seevitis\_hlsdirectory\. The comparison of the performance and resource utilization of the FPGA optimized architecture and the CPU implementation is presented in Section[5\.4](https://arxiv.org/html/2607.22786#S5.SS4)\.

#### 4\.2\.1Vanilla Transformer

In this section, we will describe the specific optimizations that can be applied to the Transformer architectures and present the HLS synthesis reports for the optimized architecture\.

For the purpose of the analysis, we will use the architectures with the hyperparameters as described in Section[5\.1](https://arxiv.org/html/2607.22786#S5.SS1)\. The models achieve reasonable performance while still being small enough to fit the board\.

For allmatrix multiplicationsperformed in the architecture, we will use the pipelined looploop\_jas described in Section[4\.1\.3](https://arxiv.org/html/2607.22786#S4.SS1.SSS3.Px2)because it provides a reasonable performance\-resource utilization tradeoff\.

We utilizefloatdata typefor all the matrices in the architecture\. We found that usingdoubledata type does not affect the accuracy performance of the model, while it significantly increases the resource utilization \(DSP\) and makes the model not fit into the board\.

We also use the following numerical optimization technique applied tosoftmaxto make it numerically stable\. Notice that

softmax​\(xi\)\\displaystyle\\text\{softmax\}\(x\_\{i\}\)=exi∑j=1Nexj\\displaystyle=\\frac\{e^\{x\_\{i\}\}\}\{\\sum\_\{j=1\}^\{N\}e^\{x\_\{j\}\}\}=exi−max⁡\(x\)∑j=1Nexj−max⁡\(x\)\\displaystyle=\\frac\{e^\{x\_\{i\}\-\\max\(x\)\}\}\{\\sum\_\{j=1\}^\{N\}e^\{x\_\{j\}\-\\max\(x\)\}\}=softmax​\(xi−max⁡\(x\)\)\\displaystyle=\\text\{softmax\}\(x\_\{i\}\-\\max\(x\)\)Sinceeixe^\{x\}\_\{i\}term can be very large, subtracting the maximum value from all the elements bounds the values ofeixe^\{x\}\_\{i\}\(the largest element will be 0\) so that the sum of exponents will never become large\.

For theReLUactivation function, we which needs to be applied to a 2 dimensional matrix \(see Feed Forward layer of the encoder block in Section[3\.1\.2](https://arxiv.org/html/2607.22786#S3.SS1.SSS2)\) fully unroll the loops and use thecompletepartitioning of the matrix\.

All the loops in the architecture are automatically pipelined unless it is impossible to do so \(e\.g\., because of the dependencies between the iterations\)\.

The optimized architecture reaches latency of 3714 cycles \(37\.14 microseconds\) utilizes 5% BRAM, 30% FFs and 90% LUTs\. See full report invitis\_hls/transformer/csynth\.rpt\.

For comparison, the architecture with all optimization turned off reaches latency of 34745 cycles \(347\.45 microseconds, 10 times as many\) and utilizes only 2% BRAM, 2% FFs and 10% LUTs\. See full report invitis\_hls/transformer\_slow/csynth\.rpt\.

#### 4\.2\.2Linear transformer

We also implement the linear transformer architecture \(i\.e\., linear attention\) as described in Section[3\.2\.2](https://arxiv.org/html/2607.22786#S3.SS2.SSS2)on the board\.

Similarly to the vanilla transformer implementation \(described in Section[4\.2\.1](https://arxiv.org/html/2607.22786#S4.SS2.SSS1)\), we also utilizefloatdata type for all the matrices in the architecture, useloop\_jpipelining for matrices, fully unroll the loops for ReLU activation function\.

The optimized version achieves 2986 cycles \(29\.86 microseconds\) latency and utilizes 18% BRAM, 43% FFs and 99% LUTs The full report is invitis\_hls/linear\_transformer/csynth\.rpt\.

For comparison, the architecture with all optimization turned off reaches latency of 68894 cycles \(688\.94 microseconds\) and utilizes 17% BRAM, 6% FFs and 24% LUTs\. The full report is invitis\_hls/linear\_transformer\_slow/csynth\.rpt\.

## 5Experiments

In this section, we will describe the experiments that were performed to evaluate the performance of the proposed architecture\. While the main focus of this work is the inference performance of the models \(refer to Section[5\.4](https://arxiv.org/html/2607.22786#S5.SS4)\), we will also describe the training procedure and the accuracy performance of the models\.

### 5\.1Architecture and hyperparameters

Here we will describe the model architecture and the hyperparameters used for the experiments\.

In the experiments 3 models were used for comparison:

- •Linear Regression\- a simple linear regression model on handcrafted features
- •Transformer Encoder\- a transformer encoder model on raw time series data
- •Linear Transformer Encoder\- a linear transformer model on raw time series data

The reason for using the linear regression model is to provide a baseline for the performance of the transformer models to show that even simple untuned models can outperform the linear regression model\.

For both transformer models, we used the encoder architectures as described in Section[3\.2](https://arxiv.org/html/2607.22786#S3.SS2)with a linear layer on top of the output of the transformer encoder to get the final prediction \(i\.e\., if a sample is an anomaly\)\.

The transformer hyperparameters used for the experiments are presented in Table[1](https://arxiv.org/html/2607.22786#S5.T1)\.

Table 1:HyperparametersThe learning rate was chosen using the learning rate finder\[[27](https://arxiv.org/html/2607.22786#bib.bib27)\]\(see Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1.Px3)\) and the batch size was chosen to be the maximum value that fitted the dataset in memory \(2132^\{13\}samples\), we also usedAdamoptimizer\[[28](https://arxiv.org/html/2607.22786#bib.bib28)\]\(see Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1.Px2)\)\.

#### 5\.1\.1Model Fitting

In this section, we will describe the training procedure, the main tools used and issues that we encountered and how they were addressed\.

##### General procedure

The General process of training a neural network involves iteratively adjusting its parameters to minimize a specified loss function\. This is typically achieved through an optimization algorithm, such as gradient descent\.

That is, at each iteration, the parameters of the modelθ\\thetaare updated as follows:

θt\+1=θt−α​∇θℒ​\(θt\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\alpha\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{t\}\)\(5\.1\)
whereα\\alphais the learning rate andℒ​\(θt\)\\mathcal\{L\}\(\\theta\_\{t\}\)is the loss function at thett\-th iteration\.

While the general procedure is simple there are multiple methods to improve the convergence of the optimization algorithm, which will be described in the following sections\.

##### Optimizer

The optimizer is the algorithm that is used to update the parameters of the model\. The general procedure of updating the parameters as described in Equation[5\.1](https://arxiv.org/html/2607.22786#S5.E1), can be improved by using more sophisticated optimization algorithms\.

TheAdamoptimizer is a popular optimization algorithm used in training artificial neural networks and is a reasonable baseline choice for many problems\. It stands for ”Adaptive Moment Estimation” and combines ideas from two other optimization techniques: RMSprop \(Root Mean Square Propagation\) and Momentum\. The Adam optimizer is known for its efficiency and fast convergence, and robustness to various types of neural network architectures and problem domains\.

The optimization procedure performed by Adam is as follows:

1. 1\.Initialize 2 moving average accumulators,mmandvvwith 0\. They will store the exponentially decaying average of past gradients and squared gradients respectively for each parameterθ\\theta
2. 2\.At each iteration, compute the gradient of the loss function with respect to the parameters,∇θℒ​\(θt\)\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{t\}\)
3. 3\.Update the moving averagesmmandvvas follows: m\\displaystyle m=β1​m\+\(1−β1\)​∇θℒ​\(θt\)\\displaystyle=\\beta\_\{1\}m\+\(1\-\\beta\_\{1\}\)\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{t\}\)v\\displaystyle v=β2​v\+\(1−β2\)​∇θℒ​\(θt\)2\\displaystyle=\\beta\_\{2\}v\+\(1\-\\beta\_\{2\}\)\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{t\}\)^\{2\}whereβ1\\beta\_\{1\}andβ2\\beta\_\{2\}are the hyperparameters that control the decay rate of the moving averages\. The usual values forβ1\\beta\_\{1\}andβ2\\beta\_\{2\}are 0\.9 and 0\.999 respectively\.
4. 4\.Compute the bias\-corrected moving averagesm^\\hat\{m\}andv^\\hat\{v\}as follows: m^\\displaystyle\\hat\{m\}=m1−β1t\\displaystyle=\\frac\{m\}\{1\-\\beta\_\{1\}^\{t\}\}v^\\displaystyle\\hat\{v\}=v1−β2t\\displaystyle=\\frac\{v\}\{1\-\\beta\_\{2\}^\{t\}\}The bias correction is necessary because the moving averages are initialized with 0 \(see original paper for derivations\[[28](https://arxiv.org/html/2607.22786#bib.bib28)\]\)\.
5. 5\.Update the network parameters as follows: θt\+1=θt−α​m^v^\+ϵ\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\alpha\\frac\{\\hat\{m\}\}\{\\sqrt\{\\hat\{v\}\}\+\\epsilon\}whereϵ\\epsilonis a small constant used for numerical stability \(i\.e\., to avoid division by zero\)\. The update is similar to the gradient descent update rule \(Equation[5\.1](https://arxiv.org/html/2607.22786#S5.E1)\) except that the gradient is divided by the square root of the moving average of squared gradients\.

We found that the optimizer with default parameters \(β1\\beta\_\{1\}andβ2\\beta\_\{2\}\) works well for our task, and we did not experiment with other optimizers\.

##### Learning rate finder

The learning rateα\\alphais one of the most important hyperparameters that determines the step size in parameter space during gradient descent optimization\. An appropriate learning rate is essential for model convergence\. The problem of choosing the learning rate is a well\-known problem in machine learning and badly chosen learning rate can lead to either underfitting where the model learns too slowly or it can lead to divergence where the parameters are updated too abruptly\.

In\[[27](https://arxiv.org/html/2607.22786#bib.bib27)\], authors proposed a simple method to find an appropriate learning rate automatically by plotting the loss function against the learning rate\.

The procedure is performed as follows:

1. 1\.Start with a very small learning rateα\\alphaand increase it at each iteration
2. 2\.At each iteration, train the model for a few epochs and compute the loss function
3. 3\.Plot the loss function against the learning rate This plot is crucial in identifying the ”sweet spot” in the learning rate range where the loss is decreasing effectively\.
4. 4\.Choose the point on the learning rate vs\. loss curve where the loss starts to decrease most steeply\. This point indicates that the model is making the most significant progress towards convergence\.

Instead of manually plotting the loss function against the learning rate, we used the implementation provided in Pytorch Lightning\[[29](https://arxiv.org/html/2607.22786#bib.bib29)\]\.

##### Gradient explosion and Gradient clipping

A different challenge of training neural networks is the phenomenon known as the ”gradient explosion problem\.” We have found that the gradient explosion problem is especially prevalent in the proposed architectures\.

The gradient explosion problem occurs when the gradient of the loss function with respect to the parameters becomes too large and the parameters are updated too abruptly \(e\.g\., as in Equation[5\.1](https://arxiv.org/html/2607.22786#S5.E1)\)\. This can lead to the model diverging and the loss function increasing instead of decreasing\. An example of the loss function diverging is presented in Figure[14](https://arxiv.org/html/2607.22786#S5.F14)\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/x4.png)Figure 14:Example of gradient explosion at around epoch 40 which leads to the loss function diverging for a few following epochs\.To solve the problem of gradient explosion, we used the gradient clipping technique introduced in\[[30](https://arxiv.org/html/2607.22786#bib.bib30)\]\. The idea behind gradient clipping is to clip the gradient to a maximum valuegm​a​xg\_\{max\}\. This remediates the problem of gradient explosion because the gradient is bounded and the parameters are updated more smoothly\.

##### Class imbalance and loss functions

In anomaly detection task the dataset is often imbalanced, i\.e\., the number of normal samples is much larger than the number of anomalous samples \(see Section[5\.2](https://arxiv.org/html/2607.22786#S5.SS2)\)\.

This poses a problem for the training of the model because the model can simply learn to predict the majority class \(i\.e\., normal samples\) and achieve a high accuracy without having good recall \(refer to Section[5\.3\.2](https://arxiv.org/html/2607.22786#S5.SS3.SSS2)for the description to the metrics\)\.

The loss functions that we use for training is the binary cross\-entropy loss function\[[31](https://arxiv.org/html/2607.22786#bib.bib31)\]which is computed as follows:

ℒ​\(θ\)=−1N​∑i=1Nyi​log⁡\(y^i\)\+\(1−yi\)​log⁡\(1−y^i\)\\mathcal\{L\}\(\\theta\)=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}y\_\{i\}\\log\(\\hat\{y\}\_\{i\}\)\+\(1\-y\_\{i\}\)\\log\(1\-\\hat\{y\}\_\{i\}\)\(5\.2\)whereyiy\_\{i\}is the true label andy^i\\hat\{y\}\_\{i\}is the predicted label\.

While the binary cross\-entropy loss function is a good choice for the anomaly detection task, it does not take into account the class imbalance problem\.

A way to solve the class imbalance problem is to use a weigh positive samples \(i\.e\., anomalous samples\) more than negative samples in the loss function\.

So we can modify the loss function as follows:

ℒ​\(θ\)=−1N​∑i=1Nwi​\(yi​log⁡\(y^i\)\+\(1−yi\)​log⁡\(1−y^i\)\)\\mathcal\{L\}\(\\theta\)=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}w\_\{i\}\(y\_\{i\}\\log\(\\hat\{y\}\_\{i\}\)\+\(1\-y\_\{i\}\)\\log\(1\-\\hat\{y\}\_\{i\}\)\)\(5\.3\)wherewiw\_\{i\}is the weight of theii\-th sample\. In the experiments, we found that weighting the positive samples 5 times more than the negative samples yielded good results for most of the datasets which we used throughout the experiments\.

##### Training stability: Layer normalization and positional encoding

While the base transformer encoder architecture uses layer normalization and positional encoding layers, we found that they lead to unstable training and worse performance of the model on most of the datasets\. Hence, we disabled them for the experiments and replaced them with identity layers\.

##### Data Preprocessing

Although, this is not a part of the training procedure, before fitting a neural network, it is usually a good idea to preprocess the data by scaling it\.

For the experiments, we used theMinMaxScalerfrom thescikit\-learnlibrary\[[32](https://arxiv.org/html/2607.22786#bib.bib32)\], to scale the input features to the range of\[0,1\]\[0,1\]for a more stable training\.

### 5\.2Datasets

In this section, the datasets used for model training and performance evaluation will be described\.

#### 5\.2\.1Numenta Anomaly Benchmark \(NAB\)

To assess the accuracy of predictions, we use the Numenta Anomaly Benchmark\[[33](https://arxiv.org/html/2607.22786#bib.bib33)\]dataset, which contains various real\-world labeled time series of temperature sensor readings, CPU utilization of cloud machines, service request latencies, and taxi demands in New York City\. It is commonly used to assess the performance of anomaly detection models on time\-series data\.

The reason why we use this dataset is that it is a standard benchmark dataset for anomaly detection in time series and because it has a large number of labeled time series\.

A sample time series of NYC taxi demand is presented in Figure[15](https://arxiv.org/html/2607.22786#S5.F15)\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/NAB_example_nyc_taxi.png)Figure 15:NYC Taxi demand \- anomalies highlighted in red\. Left: Time series of observations, Right: Distribution of observations\.In this work, we only used the NYC taxi demand dataset as in the\[[19](https://arxiv.org/html/2607.22786#bib.bib19)\]\. The dataset consists of 10320 samples with 10% of anomalous samples\.

#### 5\.2\.2KPI Anomaly Detection Dataset

The other labeled dataset that we use is the KPI Anomaly Detection Dataset \(KPI AIOps\)\[[34](https://arxiv.org/html/2607.22786#bib.bib34)\]\. This dataset alongside the NAB dataset will be used to evaluate the predictive performance of the anomaly detection models\.

The dataset consists of KPI \(key performace index\) time series data from many real scenarios of Internet companies with ground truth label\. KPIs fall into two broad categories: service KPIs and machine KPIs\. Service KPIs are performance metrics that reflect the size and quality of a Web service, such as page response time, page views, and number of connection errors\. Machine KPIs are performance indicators that reflect the health of the machine \(server, router, switch\), such as CPU utilization, memory utilization, disk IO and network card throughput\.

A sample time series of a sensor readings is presented in Figure[16](https://arxiv.org/html/2607.22786#S5.F16)\. We can clearly see the outliers for some of the observations \(colored in red\)\.

![Refer to caption](https://arxiv.org/html/2607.22786v1/KPIAIOps_example.png)Figure 16:Sensor data from a machine in a data center\. The red dots indicate the anomalies\. Left: Time series of observations, Right: Distribution of observations\.
#### 5\.2\.3FI2010

In\[[35](https://arxiv.org/html/2607.22786#bib.bib35)\], authors described the first publicly available benchmark dataset of high\-frequency limit order markets for mid\-price prediction\. The dataset contains 10\-day limit order book data from June 2010 for five stocks that are listed on the Helsinki exchange\. Each entry in the time series provides price details and aggregate order sizes for the top ten levels on both the bid and offer sides of the market, totaling forty data points\. The time series consists of approximately four million messages, representing incoming buy/sell orders or cancellations\. The dataset features order book snapshots taken after every 10 messages, resulting in approximately 400,000 records for the five stocks\.

A number of versions of the dataset are available using different normalization schemes\. We used the not normalized version of the dataset\.

For the purpose of this work, we only extract only the mid price from the dataset which will be used for anomaly detection task\.

##### Synthetic outliers

Since the dataset is not labeled, we have to inject synthetic anomalies into the dataset\. We employ the approach similar to\[[36](https://arxiv.org/html/2607.22786#bib.bib36)\]with a slight modification\. The algorithm can be summarized as follows:

1. 1\.Selectnnsamples from the time series which will be contaminated \(i\.e\., anomalous\)
2. 2\.Replace the sampleSiS\_\{i\}withS^i=Si​\(1\+δ\)\\hat\{S\}\_\{i\}=S\_\{i\}\(1\+\\delta\)whereδ\\deltais the injected outlier in the return space\.

Authors modelδ\\deltaas a uniformly distributed random variable𝒰​\[0,ρ\]\\mathcal\{U\}\[0,\\rho\]\. We instead use the uniform distribution𝒰​\[−ρ,ρ\]\\mathcal\{U\}\[\-\\rho,\\rho\]\(i\.e\., just increase the range\) to allow for positive and negative outliers\.

An example of the injected outliers is presented in Figure[17](https://arxiv.org/html/2607.22786#S5.F17)

![Refer to caption](https://arxiv.org/html/2607.22786v1/FI2010_example_outliers_injected.png)Figure 17:Example of the injected outliers in the FI2010 dataset\.

### 5\.3Accuracy

In this section, we will describe the main metrics used to evaluate the performance of the anomaly detection models\. The Transformer encoder model will be compared with the simple linear regression model on handcrafted features and with the Linear Transformer model\[[18](https://arxiv.org/html/2607.22786#bib.bib18)\]\. The inference procedure will be described and the results will be presented\.

#### 5\.3\.1Inference

After training the model on the training set as described in Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1), we can use the model to make predictions on the test set\.

#### 5\.3\.2Metrics

In this section the metrics used to evaluate the performance of the anomaly detection models will be described\.

Before we describe the metrics, we need to introduce the confusion matrix and the following notation:

- •TP\- True Positive calculated as the number of correctly predicted anomalies
- •TN\- True Negative calculated as the number of correctly predicted non\-anomalies
- •FP\- False Positive which is the number of incorrectly predicted anomalies
- •FN\- False Negative which is the number of incorrectly predicted non\-anomalies
- •P\- Number of positive samples, i\.e\.,P=TP\+FN\\textbf\{P\}=\\textbf\{TP\}\+\\textbf\{FN\}
- •N\- Number of negative samples, i\.e\.,N=TN\+FP\\textbf\{N\}=\\textbf\{TN\}\+\\textbf\{FP\}

The confusion matrix is a table with two rows and two columns that reports the number of false positives, false negatives, true positives, and true negatives\.

TruepositiveTP𝐏′\\mathbf\{P^\{\\prime\}\}𝐏\\mathbf\{P\}FalsenegativeFNntotalP′\\mathrm\{P\}^\{\\prime\}FalsepositiveFP𝐍′\\mathbf\{N^\{\\prime\}\}totalPTruenegativeTNN′\\mathrm\{N\}^\{\\prime\}Nactual
valueprediction outcomeFigure 18:Confusion matrix descriptionThe matrix summarizes the predictions from a classification model, i\.e\., how well the model performed when predicting the class labels for positive and negative samples\. While the matrix presents the most informative view of the performance of the model, we still need to summarize the information in the matrix into a single number\(s\) that can be used to compare different models\.

In this paper, we will use the following metrics to compare the performance of the anomaly detection models:

- •Accuracyis the fraction of predictions that the model got right\. It is defined as follows: Accuracy=TP\+TNTP\+TN\+FP\+FN\\text\{Accuracy\}=\\frac\{\\textbf\{TP\}\+\\textbf\{TN\}\}\{\\textbf\{TP\}\+\\textbf\{TN\}\+\\textbf\{FP\}\+\\textbf\{FN\}\}While this metric is easy to understand, it is not very informative when the dataset is imbalanced which is the case for the anomaly detection task where the proportion of positive samples is low\.
- •Precisionis the fraction of positive predictions that were correct\. Precision=TPTP\+FP\\text\{Precision\}=\\frac\{\\textbf\{TP\}\}\{\\textbf\{TP\}\+\\textbf\{FP\}\}This metric is useful when the cost of false positives is high\. For example, in the case of anomaly detection, we want to have a high precision so that we do not have to manually check many false positives or trigger any downstream filtering task too often\.
- •Recallis the fraction of positive samples that were correctly predicted\. Recall=TPTP\+FN\\text\{Recall\}=\\frac\{\\textbf\{TP\}\}\{\\textbf\{TP\}\+\\textbf\{FN\}\}This metric is useful when the cost of false negatives is high\. For example, in the case of anomaly detection, we want to minimize the number of false negatives because we do not want to miss any anomalies\.
- •F1is the harmonic mean of precision and recall which ranges from 0 to 1\. F1=2⋅Precision⋅RecallPrecision\+Recall\\text\{F1\}=2\\cdot\\frac\{\\text\{Precision\}\\cdot\\text\{Recall\}\}\{\\text\{Precision\}\+\\text\{Recall\}\}This metric is useful when we want to balance the precision and recall\. For example, in the case of anomaly detection, we want to have a high precision so that we do not have to manually check many false positives or trigger any downstream filtering task too often\. At the same time, we want to minimize the number of false negatives because we do not want to miss any anomalies\. The advantage of using this metric instead of the accuracy is that it can be used even when the dataset is highly imbalanced and it would detect if the model performs poorly in terms of precision and/or recall\.

#### 5\.3\.3Comparison

Table 2:Final metrics\. The reported value are for the train and validation set\.The final metrics for the models are presented in Table[2](https://arxiv.org/html/2607.22786#S5.T2)\. Refer to the Appendix[A](https://arxiv.org/html/2607.22786#A1)for the training plots for the models\.

The Transformer model outperforms the other models on all datasets\. And that is expected because the Transformer model is more complex \(i\.e\., has more parameters\) than the other models\. However, the Linear Transformer model performs almost as well and it provides better performance\-resource utilization tradeoff \(see Section[5\.4](https://arxiv.org/html/2607.22786#S5.SS4)\)\.

In terms of training robustness \(i\.e\., how stable the training is, refer to Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1)\), a simple Linear Regression model is the most stable model, as expected\. Both Transformer models suffer from the gradient explosion problem which leads to unstable training which can be partially remediated \(see Section[5\.1\.1](https://arxiv.org/html/2607.22786#S5.SS1.SSS1.Px4)\)\.

However, the performance of a simple Linear regression depends on the dataset\. For example, the Linear regression is not able to learn the patterns in the NAB dataset and is stuck at predicting only the positive samples \(i\.e\., anomalies\)\. As for the KPI dataset, the performance on the KPI can be tweaked by training the model for a different number of epochs to achieve the required balance between precision and recall \(refer to Figure[23](https://arxiv.org/html/2607.22786#A1.F23)\)\. Both Transformer models, on the other hand, perform equally well on both datasets as long as there is no gradient explosion problem\.

In short, Transformer models outperform the simple linear regression but can have unstable training\. Linear Transformer model is slightly behind the vanilla model\. This which shows that untuned Transformer models provide a better baseline for anomaly detection than the simple linear regression model\.

### 5\.4Performance/Speed

The performance of FPGA implementation of the models will be compared with the pytorch CPU and GPU implementation of the models\. The CPU used for the experiments is Intel\(R\) Core\(TM\) i7\-8750H CPU @ 2\.20GHz clocked to 4\.20GHz and the GPU is NVIDIA GeForce GTX 1050 Ti\. The latency on CPU and GPU was measured empirically by running the inference multiple times \(10,000 runs\)\. Refer to the Appendix[B](https://arxiv.org/html/2607.22786#A2)for the histograms of inference times\.

Table 3:Inference time for the models on different devices\. For CPU and GPU we report the mean and standard deviation\. For FPGA, the deterministic latency is reported \(see Section[4\.2](https://arxiv.org/html/2607.22786#S4.SS2)\)\.According to the results \(Table[3](https://arxiv.org/html/2607.22786#S5.T3)\), the FPGA implementation of the models outperforms the CPU implementation by a large margin \(achieving a speedup of 10x\)\. The reason for this is that the FPGA implementation is highly parallelized compared to the CPU implementation\. Moreover, the benefit of using the FPGA implementation is that it has deterministic latency \(i\.e\., the inference time is constant\) which is not the case for the CPU and GPU implementation where the inference time varies depending on the load of the CPU/GPU \(see the standard deviation for the inference times of CPU and GPU\)\.

Regarding the resource utilization, the Linear Transformer model uses more resources than the Transformer model \(see Table[4](https://arxiv.org/html/2607.22786#S5.T4)\) and achieves a slightly better runtime performance \(see Table[3](https://arxiv.org/html/2607.22786#S5.T3)\)\.

Table 4:Relative resource utilization for the PYNQ\-Z2 board\.

## 6Conclusion

### 6\.1Summary

In this work, we have examined the performance of the Transformer model for the anomaly detection task and how it can be effectively implemented on FPGA\.

We have shown how different Transformer Architectures \(and especially the attention module\) affect the performance of the model both in terms of accuracy and inference speed\. It was shown that the Transformer model outperforms the simple linear regression model without too much hyperparameter tuning\. The only downside of the Transformer model that was observed was the unstable training which can be partially remediated by using gradient clipping\.

We have also demonstrated how different optimizations affect the performance and resource utilization of the model on FPGA\. We also provided an optimized implementation of the Transformer model on FPGA which can be used as a reference\.

This work can be used as a starting point for further research on the topic of anomaly detection using Transformer models and for further research on the topic of efficient implementation of Transformer models on FPGA in different domains \(e\.g\., natural language processing\)\.

### 6\.2Future work

For future work, we propose the following:

1. 1\.To try bigger FPGA boards to push the limit of maximum performance\. The current implementations hit the limit of the PYNQ\-Z2 board\.
2. 2\.To evaluate the performance on more recentlabeledfinancial market data\. Currently, there is no publicly available labeled financial market dataset with anomalies\. The FI2010 dataset is not labeled and we had to inject synthetic anomalies\.
3. 3\.To evaluate the performance Transformer on FPGA board for other tasks \(e\.g\., natural language processing\)\. The current implementation is only for the anomaly detection task\. The other domains might involve a slightly different architecture \(e\.g\., different number of layers, different number of heads, etc\.\), it can be valuable to have those models on edge devices because of their recent surge in popularity and increased use\. Currently, the language models are too big to fit on the FPGA board \(especially, PYNQ\-Z2\), however, the smaller models can still be implemented and evaluated on bigger boards\.

## References

- \[1\]Thomas Neil Falkenberry CFA\.High frequency data filtering\.https://www\.tickdata\.com/whitepaper/high\-frequency\-data\-filtering, Sep 2008\.
- \[2\]Owen Vallis, Jordan Hochenbaum, and Twitter\.Introducing practical and robust anomaly detection in a time series\.[https://blog\.twitter\.com/engineering/en\_us/a/2015/introducing\-practical\-and\-robust\-anomaly\-detection\-in\-a\-time\-series](https://blog.twitter.com/engineering/en_us/a/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series)\.
- \[3\]Mohiuddin Ahmed, Nazim Choudhury, and Shahadat Uddin\.Anomaly detection on big data in financial markets\.In2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining \(ASONAM\), pages 998–1001, 2017\.
- \[4\]Anthony Gillioz, Jacky Casas, Elena Mugellini, and Omar Abou Khaled\.Overview of the transformer\-based models for nlp tasks\.In2020 15th Conference on Computer Science and Information Systems \(FedCSIS\), pages 179–183, 2020\.
- \[5\]Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun\.Transformers in time series: A survey\.2022\.
- \[6\]Andreea\-Ingrid Funie, Liucheng Guo, Xinyu Niu, Wayne Luk, and Mark Salmon\.Custom framework for run\-time trading strategies\.In Stephan Wong, Antonio Carlos Beck, Koen Bertels, and Luigi Carro, editors,Applied Reconfigurable Computing, pages 154–167, Cham, 2017\. Springer International Publishing\.
- \[7\]Julio\-Omar Palacio\-Niño and Fernando Berzal\.Evaluation metrics for unsupervised learning algorithms, 2019\.
- \[8\]Georg Steinbuss and Klemens Böhm\.Generating artificial outliers in the absence of genuine ones — a survey\.ACM Transactions on Knowledge Discovery from Data \(TKDD\), 15:1 – 37, 2020\.
- \[9\]Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock\.Anomaly detection in time series: A comprehensive evaluation\.Proc\. VLDB Endow\., 15:1779–1797, 2022\.
- \[10\]Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio\.Neural machine translation by jointly learning to align and translate, 2014\.
- \[11\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N\. Gomez, Lukasz Kaiser, and Illia Polosukhin\.Attention is all you need, 2017\.
- \[12\]Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\.Bert: Pre\-training of deep bidirectional transformers for language understanding\.ArXiv, abs/1810\.04805, 2019\.
- \[13\]Yungi Jeong, Eu\-Hui Yang, Jung Hyun Ryu, Imseong Park, and Myung joo Kang\.Anomalybert: Self\-supervised transformer for time series anomaly detection using data degradation scheme\.ArXiv, abs/2305\.04468, 2023\.
- \[14\]Philipp Dufter, Martin Schmitt, and Hinrich Schütze\.Position information in transformers: An overview\.Computational Linguistics, 48:733–763, 2021\.
- \[15\]Lilian Weng\.The transformer family version 2\.0\.[https://lilianweng\.github\.io/posts/2023\-01\-27\-the\-transformer\-family\-v2](https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2), Jan 2023\.
- \[16\]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Deep residual learning for image recognition, 2015\.
- \[17\]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E\. Hinton\.Layer normalization, 2016\.
- \[18\]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret\.Transformers are rnns: Fast autoregressive transformers with linear attention, 2020\.
- \[19\]Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings\.TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data\.Proceedings of VLDB, 15\(6\):1201–1214, 2022\.
- \[20\]Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu\.Are transformers effective for time series forecasting?, 2022\.
- \[21\]Edmond Lezmi and Jiali Xu\.Time series forecasting with transformer models and application to asset management\.SSRN Electronic Journal, 2023\.
- \[22\]University of York\.Vitis HLS Knowledge Base \- Real\-Time Systems \- York Wiki Service\.[https://wiki\.york\.ac\.uk/display/RTS/Vitis\+HLS\+Knowledge\+Base](https://wiki.york.ac.uk/display/RTS/Vitis+HLS+Knowledge+Base), July 2020\.
- \[23\]Xilinx Inc\.Vitis High\-Level Synthesis User Guide\.[https://docs\.xilinx\.com/r/en\-US/ug1399\-vitis\-hls/Design\-Principles](https://docs.xilinx.com/r/en-US/ug1399-vitis-hls/Design-Principles), May 2023\.
- \[24\]Xilinx Inc\.C/RTL Co\-Simulation in Vitis HLS\.[https://docs\.xilinx\.com/r/en\-US/ug1399\-vitis\-hls/C/RTL\-Co\-Simulation\-in\-Vitis\-HLS](https://docs.xilinx.com/r/en-US/ug1399-vitis-hls/C/RTL-Co-Simulation-in-Vitis-HLS), July 2023\.
- \[25\]Xilinx Inc\.Vitis hls: Pipelining loops\.[https://docs\.xilinx\.com/r/en\-US/ug1399\-vitis\-hls/Design\-Principles](https://docs.xilinx.com/r/en-US/ug1399-vitis-hls/Design-Principles), May 2023\.
- \[26\]Intel\.Intel high level synthesis: Best practices guide\.[https://www\.intel\.com/content/www/us/en/docs/programmable/683152/21\-3/pipeline\-loops\.html](https://www.intel.com/content/www/us/en/docs/programmable/683152/21-3/pipeline-loops.html)\.
- \[27\]Leslie N\. Smith\.Cyclical learning rates for training neural networks, 2015\.
- \[28\]Diederik P\. Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.CoRR, abs/1412\.6980, 2014\.
- \[29\]lightning\.ai\.Learningratefinder — pytorch lightning 2\.0\.7 documentation\.[https://lightning\.ai/docs/pytorch/stable/api/lightning\.pytorch\.callbacks\.LearningRateFinder\.html](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.LearningRateFinder.html), Aug 2023\.
- \[30\]Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio\.On the difficulty of training recurrent neural networks, 2012\.
- \[31\]I\. J\. Good\.Rational decisions\.Journal of the Royal Statistical Society: Series B \(Methodological\), 14\(1\):107–114, Jan 1952\.
- \[32\]F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and E\. Duchesnay\.Scikit\-learn: Machine learning in Python\.Journal of Machine Learning Research, 12:2825–2830, 2011\.
- \[33\]Subutai Ahmad, Alexander Lavin, Scott Purdy, and Zuha Agha\.Unsupervised real\-time anomaly detection for streaming data\.Neurocomputing, 262:134–147, 11 2017\.\[Online; accessed 2023\-07\-19\]\.
- \[34\]Zeyan Li, Nengwen Zhao, Shenglin Zhang, Yongqian Sun, Pengfei Chen, Xidao Wen, Minghua Ma, and Dan Pei\.Constructing large\-scale real\-world benchmark datasets for aiops, 2022\.
- \[35\]Adamantios Ntakaris, Martin Magris, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis\.Benchmark dataset for mid\-price forecasting of limit order book data with machine learning methods\.2017\.
- \[36\]Stéphane Crépey, Noureddine Lehdili, Nisrine Madhar, and Maud Thomas\.Anomaly Detection in Financial Time Series by Principal Component Analysis and Neural Networks\.Algorithms, 15\(10\):385, oct 19 2022\.\[Online; accessed 2023\-07\-19\]\.

## Appendix ATraining plots

The training plots are presented in Figures[19](https://arxiv.org/html/2607.22786#A1.F19),[20](https://arxiv.org/html/2607.22786#A1.F20),[21](https://arxiv.org/html/2607.22786#A1.F21),[22](https://arxiv.org/html/2607.22786#A1.F22),[23](https://arxiv.org/html/2607.22786#A1.F23),[24](https://arxiv.org/html/2607.22786#A1.F24)\.

Figure 19:Training metrics for different epochs for Transformer model on KPI dataset![Refer to caption](https://arxiv.org/html/2607.22786v1/Transformer_KPI.png)Figure 20:Training metrics for different epochs for Transformer model on NAB dataset![Refer to caption](https://arxiv.org/html/2607.22786v1/Transformer_NAB.png)Figure 21:Training metrics for different epochs for Linear Transformer model on KPI dataset![Refer to caption](https://arxiv.org/html/2607.22786v1/LinearTransformer_KPI.png)Figure 22:Training metrics for different epochs for Linear Transformer model on NAB dataset![Refer to caption](https://arxiv.org/html/2607.22786v1/LinearTransformer_NAB.png)Figure 23:Training metrics for different epochs for Linear Regression model on KPI dataset![Refer to caption](https://arxiv.org/html/2607.22786v1/LinearRegression_KPI.png)Figure 24:Training metrics for different epochs for Linear Regression model on NAB dataset![Refer to caption](https://arxiv.org/html/2607.22786v1/LinearRegression_NAB.png)
## Appendix BEmpirical latency

The empirical latency plots are presented in Figures[25](https://arxiv.org/html/2607.22786#A2.F25),[26](https://arxiv.org/html/2607.22786#A2.F26),[27](https://arxiv.org/html/2607.22786#A2.F27),[28](https://arxiv.org/html/2607.22786#A2.F28)\.

Figure 25:Empirical latency for Transformer model on CPU\.![Refer to caption](https://arxiv.org/html/2607.22786v1/x5.png)Figure 26:Empirical latency for Transformer model on GPU\.![Refer to caption](https://arxiv.org/html/2607.22786v1/x6.png)Figure 27:Empirical latency for Linear Transformer model on CPU\.![Refer to caption](https://arxiv.org/html/2607.22786v1/x7.png)Figure 28:Empirical latency for Linear Transformer model on GPU\.![Refer to caption](https://arxiv.org/html/2607.22786v1/x8.png)