RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory

# RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
Source: [https://arxiv.org/html/2607.21731](https://arxiv.org/html/2607.21731)
###### Abstract

Transformers are widely used across many domains, including natural language processing, computer vision, web search, and DNA sequence analysis\. Given their broad applicability, improving the performance of transformer models is critical\. However, the high volume of data movement between processing units and memory during attention operations significantly limits their efficiency\. Processing\-In\-Memory \(PIM\) mitigates this issue by performing computations directly inside memory\. While prior work has proposed PIM\-based transformer implementations, they suffer from costly inter\-bank communication, and struggle to scale due to the limited capacity of memory banks\. As a result, attention\-related data must be split across banks, diminishing the potential benefits of PIM\.

In this work, we propose RED\-PIM, an algorithm\-architecture co\-design that reduces attention latency by minimizing inter\-bank data movement fromO​\(N2\)O\(N^\{2\}\)toO​\(N\)O\(N\)and shrinking intermediate attention matrices fromN×NN\\times Ntod×dd\\times d\. By reorganizing matrix operations, performing computations locally, and employing an optimized data transfer strategy, RED\-PIM significantly reduces computation cost and interconnect traffic\. Compared to baseline PIM implementation, RED\-PIM achieves inference time reductions ranging from 16\.05% to 99\.99% \(geometric mean of 66\.42%\), with the largest gains on longer sequences\. On real\-world datasets, RED\-PIM improves performance by 99\.60% for long documents and 13\.44% for shorter ones, while maintaining or improving accuracy\. These results demonstrate RED\-PIM’s effectiveness for scalable and efficient transformer inference\.

## IIntroduction

Traditional CPU and GPU\-based architectures face significant challenges in meeting the computational demands of large\-scale neural networks due to inefficiencies caused by data movement between memory and processing units\. This back\-and\-forth transfer of massive amounts of data creates bottlenecks that degrade performance and increase energy consumption\.

Among various deep learning models, this issue becomes more significant in Transformer\-based architectures\[[42](https://arxiv.org/html/2607.21731#bib.bib1)\], mainly due to their self\-attention module\. The reason is that self\-attention involves computing attention scores between all pairs of input tokens\. To this end, after the initial calculation of the query, key, and value vectors from input tokens, the model has to calculate the similarity between all queries and keys to generate an attention map\. This requiresO​\(N2\)O\(N^\{2\}\)data movements for \(query, key\) comparisons, whereNNis the sequence length\. Subsequently, multiplying this attention map with the value vectors to obtain the final result also involvesO​\(N2\)O\(N^\{2\}\)movements\.

Due to the large volume of data needed in large\-scale transformers, typical CPU or GPU caches are ineffective\. Self\-attention computations generate extremely large intermediate data structures that far exceed typical cache capacities, leading to frequent cache evictions and increased latency\. To illustrate, consider a sequence with 8,000 tokens\. The resulting attention map consists of 64 million values, requiring approximately 32MB of memory even when using the lowest\-precision format \(i\.e\., FP16\), a much larger size than typical CPU or GPU cache capacities\.

Processing In/Near Memory \(PIM\)\[[48](https://arxiv.org/html/2607.21731#bib.bib10),[49](https://arxiv.org/html/2607.21731#bib.bib11),[40](https://arxiv.org/html/2607.21731#bib.bib15),[23](https://arxiv.org/html/2607.21731#bib.bib16),[27](https://arxiv.org/html/2607.21731#bib.bib17),[26](https://arxiv.org/html/2607.21731#bib.bib18),[18](https://arxiv.org/html/2607.21731#bib.bib20),[25](https://arxiv.org/html/2607.21731#bib.bib21)\]offers an alternative solution to reduce the data movement between the memory and processing units\. PIM addresses this challenge by integrating computation directly within memory arrays, thereby eliminating the need for frequent data transfers between the processor\(s\) and memory\. However, even with PIM, the limited capacity of memory banks remains a major barrier for scaling transformer models\. For instance, a typical HBM bank has only 32MB of storage, which supports up to 8,000 tokens in FP16 precision\. In contrast, modern large language models \(LLMs\) such as GPT‑4\.1\[[31](https://arxiv.org/html/2607.21731#bib.bib56)\], Gemini 2\[[7](https://arxiv.org/html/2607.21731#bib.bib57)\], and Grok‑3\[[47](https://arxiv.org/html/2607.21731#bib.bib58)\]support much longer sequences, up to 1 million tokens\. Attempting to store full attention maps across multiple banks introduces significant inter\-bank communication overhead, which undermines the performance and energy\-efficiency advantages that PIM is meant to provide\.

In this paper, we propose RED\-PIM that reduces the latency of the attention operation by leveraging PIM and reducing the inter\-bank data movement\. RED\-PIM achieves this by adopting an alternative algorithm for attention computation which eliminates the need to construct the fullN×NN\\times Nattention map\. Instead, it operates on and stores only a compactd×dd\\times dmatrix, whereddis the latent dimension in the self\-attention module andd≪Nd\\ll N\. Specifically, RED\-PIM employs efficient attention\[[36](https://arxiv.org/html/2607.21731#bib.bib29)\]to significantly reduce memory usage while maintaining model performance\. We reorganize the attention operations to keep computations more local, reduce data movement, and create smaller intermediate results that can fit in a single memory bank\.

Along with reorganizing the attention operations, we introduce architectural optimizations, including near\-bank Processing Control Units \(PCUs\) that perform local computations with optimized implementations\. We propose architectural enhancements to optimize inter\-bank data flow via a hierarchical aggregation strategy among memory banks, which results in lower data transfer and latency\.

In this paper, we make the following contributions:

- •Bottleneck Analysis of Standard Self\-Attention: We show that the standard Query\-Key multiplication in self\-attention introduces two key inefficiencies: \(1\) it causes significant data movement due to the all\-to\-all token comparisons across the input sequence, and \(2\) it produces a large intermediate matrix \(attention map\) that often exceeds the storage capacity of a single HBM bank\. As a result, the subsequent Attention\-Value multiplication requires additional inter\-bank communication to access the full score matrix\.
- •Algorithmic Optimizations for Reduced Data Movement: By restructuring Scaled Dot\-Product Attention \(SDPA\), we optimize data locality, minimizing redundant memory accesses and reducing data movement between HBM banks fromO​\(N2\)O\(N^\{2\}\)toO​\(N\)O\(N\)\.
- •Architectural Design for In\-HBM Processing: We integrate our architecture with FIMDRAM\[[21](https://arxiv.org/html/2607.21731#bib.bib30)\], leveraging PIM capabilities to execute self\-attention computations directly within memory banks with optimized implementation\. A specialized network topology facilitates efficient inter\-bank communication\. We introduce a hierarchical aggregation strategy that progressively refines attention computations and reduces redundant data transfers between memory banks\. We design a softmax function optimized for hardware using bitwise operations and lookup tables to accelerate exponentiation\. This approach maintains accuracy while reducing computational complexity and memory usage in transformer models\.
- •Customized Simulation Framework for Data Movement Analysis: Building on the DAMOV simulator\[[30](https://arxiv.org/html/2607.21731#bib.bib49)\], we develop a tailored simulation environment to evaluate and model inter\-bank data movement in HBM architectures, enabling detailed analysis of communication patterns for attention mechanisms\.
- •Extensive Evaluation Demonstrating Significant Performance Gains: Our evaluation shows inference time reductions ranging from 16\.05% to 99\.99% \(geometric mean of 66\.42%\) compared to a baseline PIM implementation, with the greatest improvements observed on longer sequences\. On real\-world datasets, we achieve a geometric mean improvement of 99\.56% for long documents \(IMDB, PubMed, Arxiv, WikiHop, and GovReport\[[6](https://arxiv.org/html/2607.21731#bib.bib45),[28](https://arxiv.org/html/2607.21731#bib.bib54),[41](https://arxiv.org/html/2607.21731#bib.bib53),[15](https://arxiv.org/html/2607.21731#bib.bib52)\]\) and 13\.44% for shorter documents \(GLUE benchmark\[[43](https://arxiv.org/html/2607.21731#bib.bib38)\]\), while maintaining or improving accuracy\.

In the remainder of this paper, we discuss some background on transformer models, processing\-in\-memory, and High\-Bandwidth Memory in Section[II](https://arxiv.org/html/2607.21731#S2)\. We present our proposed algorithmic changes and architecture in Section[III](https://arxiv.org/html/2607.21731#S3)\. We discuss our methodology and results in Section[IV](https://arxiv.org/html/2607.21731#S4)\. Section[V](https://arxiv.org/html/2607.21731#S5)discusses related work, and we conclude in Section[VI](https://arxiv.org/html/2607.21731#S6)\.

## IIBackground

### II\-ATransformers

Transformers, introduced by Vaswani et al\. in 2017\[[42](https://arxiv.org/html/2607.21731#bib.bib1)\], are a class of deep learning models that have become widely used in various applications\. Their popularity was a result of their ability to process sequential data in parallel, and effectively capture long\-range dependencies\. Transformer\-based architectures are typically built using a stack of encoder blocks, decoder blocks, or both\. Each block consists of several key components: Multi\-Head Attention \(MHA\), Feed\-Forward Neural Networks \(FFN\), and Layer Normalization, as illustrated in Figure[1](https://arxiv.org/html/2607.21731#S2.F1)\[[42](https://arxiv.org/html/2607.21731#bib.bib1)\]\.

TheMHA blockis designed to capture complex relationships between tokens in a sequence\. Each attention head computes attention scores that quantify the influence of one token on another\. The outputs from multiple attention heads are then combined and processed to generate a comprehensive contextual representation of the input sequence\. To generate such representation, the MHA block first derives the Query \(Q\), Key \(K\), and Value \(V\) matrices by applying linear transformations to the input sequenceX=\[x1,x2,…,xN\]X=\[x\_\{1\},x\_\{2\},\.\.\.,x\_\{N\}\]\. These transformations use trainable weight matrices: the Query matrix \(WQW\_\{Q\}\), Key matrix \(WKW\_\{K\}\), and Value matrix \(WVW\_\{V\}\):

Q=X\.WQ,V=X\.WV,K=X\.WKQ=X\.W\_\{Q\},\\quad V=X\.W\_\{V\},\\quad K=X\.W\_\{K\}\(1\)
The Scaled Dot\-Product Attention mechanism then computes attention scores as follows:

A​t​t​e​n​t​i​o​n​\(K,Q,V\)=s​o​f​t​m​a​x​\(Q\.KTd\)\.VAttention\(K,Q,V\)=softmax\(\\frac\{Q\.K^\{T\}\}\{\\sqrt\{d\}\}\)\.V\(2\)
whereddis the dimensionality of the key vectors\. The scaling factord\\sqrt\{d\}prevents the dot product from becoming too large, which could push the softmax function into regions with very small gradients that would hinder effective learning\. The outputs of multiple attention heads are concatenated and passed through another linear layer to combine their results and generate the final representation\.

TheFFN layerreceives the output of the MHA block and transforms it into a more expressive representation\. This transformation typically involves applying two linear layers with an intermediate activation function \(e\.g\., ReLU\) as follows:

FFN\(x\)=max\(0,x\.W1\+b1\)\.W2\+b2FFN\(x\)=max\(0,x\.W\_\{1\}\+b\_\{1\}\)\.W\_\{2\}\+b\_\{2\}\(3\)
Layer normalization and residual connectionsare crucial for enhancing stability and improving training efficiency of transformers\. Residual connections bypass the attention and feed\-forward layers, while layer normalization ensures consistent input scaling, which improves gradient flow\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x1.png)Figure 1:The Transformer model architecture\[[42](https://arxiv.org/html/2607.21731#bib.bib1)\]\.In recent years, transformers have grown significantly\. Current transformer models contain billions of parameters\[[44](https://arxiv.org/html/2607.21731#bib.bib22)\]\. In addition to the size of the model, the sequence length has also increased to accommodate more complex tasks, further amplifying the computational and memory demands of these models\[[8](https://arxiv.org/html/2607.21731#bib.bib23),[11](https://arxiv.org/html/2607.21731#bib.bib24)\]\. Although these increases in size and sequence length have improved performance across various domains, they have also led to significantly longer execution times and higher computational costs\.

### II\-BProcessing In/Near Memory Architectures

The concept of PIM has emerged as a response to the increasing computational bottlenecks caused by data movement between memory and processing unit in traditional von Neumann architectures\[[12](https://arxiv.org/html/2607.21731#bib.bib25)\]\. In these architectures, the separation of memory and compute units leads to significant energy and time costs, especially in data\-intensive applications such as machine learning, graph processing, and high\-performance computing\[[1](https://arxiv.org/html/2607.21731#bib.bib26),[3](https://arxiv.org/html/2607.21731#bib.bib27),[12](https://arxiv.org/html/2607.21731#bib.bib25)\]\.

PIM addresses this challenge by integrating computational capabilities directly within the memory arrays, enabling data to be processed close to where it is stored\. This approach eliminates the need for frequent data transfers between the processor\(s\) and memory, significantly reducing both execution time and energy consumption\[[29](https://arxiv.org/html/2607.21731#bib.bib28)\]\.

### II\-CPrior PIM\-Based Accelerators for Transformers

Prior research on PIM\-based accelerators for Transformers can be broadly classified into two categories\.

The first category comprises hybrid architectures that integrate host\-based accelerators with near\-memory processors to enhance transformer performance\[[9](https://arxiv.org/html/2607.21731#bib.bib12),[14](https://arxiv.org/html/2607.21731#bib.bib14),[37](https://arxiv.org/html/2607.21731#bib.bib13)\]\. While these approaches aim to combine the advantages of traditional and PIM systems, they often suffer from high communication overhead between the host and memory\. Additionally, they introduce substantial architectural complexity, which limits scalability and energy efficiency\.

The second category consists of fully PIM\-based transformer accelerators, which execute computations entirely within memory\. These include both analog PIM designs\[[48](https://arxiv.org/html/2607.21731#bib.bib10),[40](https://arxiv.org/html/2607.21731#bib.bib15),[23](https://arxiv.org/html/2607.21731#bib.bib16),[27](https://arxiv.org/html/2607.21731#bib.bib17)\]and DRAM\-based solutions\[[49](https://arxiv.org/html/2607.21731#bib.bib11),[37](https://arxiv.org/html/2607.21731#bib.bib13)\]\. Despite their promise, these designs still face key challenges: They struggle with the large intermediate data sizes generated by self\-attention, and inter\-bank communication remains a major bottleneck, particularly for long sequences\. Furthermore, many analog PIM approaches face limitations related to precision, reliability, and non\-ideal analog behavior, which hinder their practicality and accuracy in real\-world deployments\.

We further discuss these proposals and their limitations in Section[V](https://arxiv.org/html/2607.21731#S5)\.

### II\-DHigh Bandwidth Memory

HBM, illustrated in Figure[2](https://arxiv.org/html/2607.21731#S2.F2)\.a, is a type of 3D\-stacked DRAM designed to deliver significantly higher memory bandwidth compared to traditional memory technologies like DDR or GDDR\. HBM achieves this by vertically stacking multiple memory dies, interconnecting them using through\-silicon vias, and placing the stack close to the processor on the same substrate\[[39](https://arxiv.org/html/2607.21731#bib.bib31)\]\. This architecture provides higher capacity, reduced latency, and increased effective data bandwidth\[[16](https://arxiv.org/html/2607.21731#bib.bib32)\]\. HBM is widely employed in high\-performance computing \(HPC\), networking, and graphics applications\[[50](https://arxiv.org/html/2607.21731#bib.bib34),[17](https://arxiv.org/html/2607.21731#bib.bib33),[16](https://arxiv.org/html/2607.21731#bib.bib32)\], where memory bandwidth and latency are critical factors\.

An example of PIM\-based HBM is Function\-In\-Memory DRAM \(FIMDRAM\)\[[21](https://arxiv.org/html/2607.21731#bib.bib30)\]that was introduced by Samsung in 2021\. FIMDRAM is a real PIM prototype implementation based on the HBM2 memory standard \(Figure[2](https://arxiv.org/html/2607.21731#S2.F2)\.b\)\. FIMDRAM \(also known as HBM\-PIM\) incorporates 16\-wide single\-instruction multiple\-data \(SIMD\) engines into memory banks, leveraging bank\-level parallelism while maintaining the optimized memory subarray\. Each pair of memory banks includes a programmable computing unit \(PCU\) as shown in Figure[2](https://arxiv.org/html/2607.21731#S2.F2)\.c, enhancing parallelism and processing bandwidth\. By performing computations directly within memory, FIMDRAM significantly reduces data movement, improving efficiency for memory\-intensive tasks\.

Figure[2](https://arxiv.org/html/2607.21731#S2.F2)\.d illustrates the internal architecture of a PCU\. Each PCU includes two sets of SIMD floating\-point units \(FPUs\): one dedicated to addition and the other to multiplication\. Each set contains 16 FPUs, where each FPU is 16 bits wide\.

The PCU is equipped with three kind of registers for managing computations:

- •Command Register File \(CRF\)which serves as an instruction buffer capable of holding up to 32 instructions, where each instruction is 32 bits\.
- •General Register Files \(GRFs\)which consist of 16 registers, each 256\-bits wide\. These registers are used for intermediate data storage\.
- •Scalar Register Files \(SRFs\)designed to store up to 16 scalar values, each 16\-bits wide\.

The PCU’s instruction set includes nine 32\-bit RISC instructions, categorized into three groups: Control flow instructions:NOP,JUMP, andEXIT; Arithmetic instructions:ADD,MUL,MAC, andMAD; and Data movement instructions:MOVandFILL\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x2.png)Figure 2:a\) HBM architecture; b\) Conventional HBM2; c\) FIMDRAM architecture; d\) Computing unit block\.

## IIIRED\-PIM: Proposed Design

### III\-ABaseline Implementation of Attention in PIM Architecture

One baseline implementation of the attention mechanism in PIM architectures is to directly apply the standard Scaled Dot\-Product Attention mechanism \(SDPA\), as shown in Figure[4](https://arxiv.org/html/2607.21731#S3.F4)\. For load balancing, tokens are distributed across memory banks, where each bank storesmmtokens, with each token represented as a1×d1\\times dvector\. We assume that the weight matrices \(WQW\_\{Q\},WKW\_\{K\},WVW\_\{V\}\) are locally stored in each bank, following previous works such as\[[49](https://arxiv.org/html/2607.21731#bib.bib11)\], as illustrated in Figure[3](https://arxiv.org/html/2607.21731#S3.F3)\. Later in Section[III\-D](https://arxiv.org/html/2607.21731#S3.SS4), we further analyze the scalability implications of this setup\.

Each bank computes the Query, Key, and Value vectors \(QQ,KK,VV\) for itsmmlocal tokens using the stored weight matrices\. Thus, after this step, each bank holdsmmcomputed Q/K/V vectors, as shown in Figure[3](https://arxiv.org/html/2607.21731#S3.F3)\. The attention score computation will be discussed in detail in Section[III\-C2](https://arxiv.org/html/2607.21731#S3.SS3.SSS2)\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x3.png)Figure 3:Token and weight matrix mapping across banks\.As depicted in Figure[4](https://arxiv.org/html/2607.21731#S3.F4), the first step in attention calculation is multiplyingQQbyKTK^\{T\}\. For illustration, considermmrows of theQQandKKmatrices that are stored in the same bank \(highlighted in red\)\. Each row ofQQmust be multiplied by every column ofKTK^\{T\}, which involves both local and non\-local computations\. For instance, multiplying the first row ofQQwith themmcolumns ofKTK^\{T\}that reside in the same bank can be performed locally\. However, for the remaining columns, theQQvector must be transferred from its source bank to the destination bank that holds the correspondingKKvectors\. EachQQvector needs to be transferred toNm\\frac\{N\}\{m\}banks, whereNNis the total sequence length\. This implies that computingQ​KTQK^\{T\}requiresNm\\frac\{N\}\{m\}movements perQQvector, resulting in a total ofN2m\\frac\{N^\{2\}\}\{m\}vector movements \(each of size1×d1\\times d\)\. Additionally, note that the resulting attention mapSSis of sizeN×NN\\times N, and cannot fit entirely in a single bank\. When each bank stores am×mm\\times msubmatrix ofSS, the storage overflow requires further movement of these partialSSmatrices to new banks, which introduces additional movement not captured in this estimate but accounted for in our evaluation\.

Each bank now holds am×mm\\times mportion of the attention mapSS\. As shown in Equation[2](https://arxiv.org/html/2607.21731#S2.E2), the scaling operation is applied locally\. However, the softmax operation requires computing the sum of exponentials for each row ofSS\. Since onlymmelements of each row are stored in a single bank, and each full row spans acrossNm\\frac\{N\}\{m\}banks, the computation must involve inter\-bank communication\.

After computingexp⁡\(x\)\\exp\(x\)locally for each value inSS, these partial results need to be aggregated acrossNm\\frac\{N\}\{m\}banks and then broadcast back for normalization\. This results inNm\\frac\{N\}\{m\}movements per row and a total ofN2m\\frac\{N^\{2\}\}\{m\}movements\. Even with optimization—such as sending aggregated sums ofmmvalues at once—the total movement would still beN2m2\\frac\{N^\{2\}\}\{m^\{2\}\}\.

The final step involves multiplying the attention mapSSby the Value matrixVV\. This operation, like theQ​KTQK^\{T\}step, includes both local and non\-local computation\. For instance, multiplying the first row ofSSwith the correspondingmmcolumns ofVVcan be performed locally, assuming they are co\-located in the same bank\. However, if parts of matrixSSwere already moved to other banks due to storage limitations, additional data movement will be required even for these nominally local computations\. However, for the remaining parts of the row, data must again move acrossNm\\frac\{N\}\{m\}banks to align with the relevant columns ofVV\. Thus, the total movement is againN2m\\frac\{N^\{2\}\}\{m\}or optimallyN2m2\\frac\{N^\{2\}\}\{m^\{2\}\}\. Also, since not all intermediate results can be stored in the same bank due to limited storage, additional data movement is required \(included in our evaluation but not in the above rough estimate\)\.

Finally, all these intermediate products must be aggregated to generate the final output of the attention layer\. Because the results are distributed across multiple banks, additional data movement \(up toN2m2\\frac\{N^\{2\}\}\{m^\{2\}\}\) is incurred during accumulation\. Note that all these movement estimates are rough approximations and do not account for factors such as physical bank distance, contention, or memory interconnect architecture\. However, our evaluation \(Section[IV](https://arxiv.org/html/2607.21731#S4)\) incorporates these details for a precise and realistic analysis\.

In summary, this baseline approach using standard SDPA in a PIM system incursO​\(N2m\)O\\left\(\\frac\{N^\{2\}\}\{m\}\\right\)data movement, making it inefficient and non\-scalable for long sequences or high\-throughput workloads\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x4.png)Figure 4:Regular Scaled Dot\-Product Attention mechanism\. Here, N denotes the sequence length, d is the embedding dimension, and m represents the maximum number of tokens that can be stored as anm×dm\\times dmatrix within a single memory bank\. Parts highlighted in red indicate portions stored locally within the same bank\.
### III\-BRED\-PIM Implementation of Attention in PIM Architecture

RED\-PIM adopts an alternative attention algorithm that significantly reduces data movement\. Instead of explicitly computing the fullQ​KTQK^\{T\}attention matrix, the algorithm first applies the softmax function separately toQQandKTK^\{T\}\. It then multipliesσ​\(KT\)\\sigma\(K^\{T\}\)with the Value matrixVV, followed by a multiplication withσ​\(Q\)\\sigma\(Q\)to produce the final result, whereσ\\sigmadenotes the softmax operation\.

This algorithm, introduced by Zhuoran et al\.\[[36](https://arxiv.org/html/2607.21731#bib.bib29)\], was originally designed to improve computational efficiency\. However, it does not explicitly address the critical bottleneck of data movement in memory—one of the main contributors to latency in LLMs\. In this work, we adapt and extend their method for in\-memory execution\. Our version restructures the attention computation to reduce inter\-bank communication and improve data locality, making it highly suitable for PIM architectures\.

The first step in RED\-PIM is to computeσ​\(Q\)\\sigma\(Q\)andσ​\(KT\)\\sigma\(K^\{T\}\)\. For the softmax operation, we only need to calculate the∑exp⁡\(x\)\\sum\\exp\(x\)for each row, wherexxrepresents the values in that row\. Computingσ​\(Q\)\\sigma\(Q\)is entirely local because each row ofQQcorresponds to a single query vector, which is fully stored within one bank \(as illustrated in Figure[5](https://arxiv.org/html/2607.21731#S3.F5)\)\.

However, computingσ​\(KT\)\\sigma\(K^\{T\}\)involves some data movement\. For each row ofKTK^\{T\}, onlymmvalues are stored locally in a bank, and the remaining values are distributed across the otherNm\\frac\{N\}\{m\}banks\. As a result, partial sums from these banks need to be transferred and accumulated, leading toNm\\frac\{N\}\{m\}data movements\.

Next, we compute the multiplicationσ​\(KT\)⋅V\\sigma\(K^\{T\}\)\\cdot V\. Notably, theii\-th block ofmmcolumns inσ​\(KT\)\\sigma\(K^\{T\}\)is stored in the same bank as the correspondingmmrows ofVVas shown in Figure[5](https://arxiv.org/html/2607.21731#S3.F5), allowing this portion of the multiplication to be done locally\. For example, when multiplying the first row ofσ​\(KT\)\\sigma\(K^\{T\}\)toVV, its values are distributed acrossNm\\frac\{N\}\{m\}banks\. Each of these banks holdsmmelements from the row and the correspondingmmrows fromVV\. The result of each local multiplication is a1×d1\\times dvector, and doing this across all rows produces a locald×dd\\times dmatrix in each bank\.

These local matrices are then aggregated across banks to form the final result ofσ​\(KT\)⋅V\\sigma\(K^\{T\}\)\\cdot V\. Since intermediate results are distributed acrossNm\\frac\{N\}\{m\}banks, the accumulation step also incursNm\\frac\{N\}\{m\}data movements\.

We refer to the resulting matrix asLL, which is a compactd×dd\\times dmatrix, significantly smaller than theN×NN\\times Nattention matrixSSused in the standard attention approach \(as shown in Figure[4](https://arxiv.org/html/2607.21731#S3.F4)\)\.

Finally,LLmust be multiplied byσ​\(Q\)\\sigma\(Q\)\. In the worst\-case scenario, ifLLis stored in a single bank without replication, each row ofσ​\(Q\)\\sigma\(Q\)must be transferred to that bank\. Sinceσ​\(Q\)\\sigma\(Q\)is distributed acrossNm\\frac\{N\}\{m\}banks, this step requires up toNm\\frac\{N\}\{m\}data movements\.

As mentioned earlier, these estimations do not take into account the physical distance between memory banks or the specific characteristics of the bank architecture\. Additionally, they do not reflect any architectural optimizations we introduce later in this paper, which are designed to further reduce data movement\.

Overall, this approach reduces data movement toO​\(Nm\)O\\left\(\\frac\{N\}\{m\}\\right\), which is significantly lower than that of the conventional implementation\. It also requires substantially less memory capacity, as the matrixLLis much smaller than the matrixSS\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x5.png)Figure 5:RED\-PIM Scaled Dot\-Product Attention\. Parts highlighted in red indicate portions stored locally within the same bank\.σ\\sigmadenotes the softmax operation\.
### III\-CArchitecture Design in RED\-PIM

In this section, we present the architectural details of our proposed RED\-PIM design\.

#### III\-C1Memory Architecture

We adopt FIMDRAM\[[21](https://arxiv.org/html/2607.21731#bib.bib30)\]as our baseline PIM architecture to leverage in\-bank processing capabilities for computation\. Each DRAM bank is equipped with local compute logic, enabling vector multiplications to be performed directly within the memory banks\.

#### III\-C2In\-bank Multiplication

As discussed in the previous sections, a large number of vector multiplications are executed locally within each bank\. As an example, we provide the pseudocode for multiplying vectorsxxandyyin Algorithm[1](https://arxiv.org/html/2607.21731#alg1)\.

Algorithm 1FP16 Dot Product of Vectorsxxandyy\(lengthdd\)1:MOVESRF\_M, 0⊳\\trianglerightClear accumulator

2:forchunk = 0 to

\(d/128\)−1\(d/128\)\-1do⊳\\trianglerightProcess 128 FP16 values per chunk \(64 fromxx, 64 fromyy\)

3:for

i=0i=0to

33do

4:FILLDRAM\[

xx\+ chunk

×\\times128 \+

i×16i\\times 16\]

→\\rightarrowGRF\[

2​i2i\]

5:FILLDRAM\[

yy\+ chunk

×\\times128 \+

i×16i\\times 16\]

→\\rightarrowGRF\[

2​i\+12i\+1\]

6:endfor

7:for

i=0i=0to

33do

8:for

j=0j=0to

1515do

9:FP16\_MACGRF\[

2​i2i\]\[

jj\], GRF\[

2​i\+12i\+1\]\[

jj\], SRF\_M

10:endfor

11:endfor

12:endfor

13:MOVESRF\_M

→\\rightarrowGRF⊳\\trianglerightMove final result to GRF

14:MOVEGRF

→\\rightarrowDRAM⊳\\trianglerightStore final result to memory

#### III\-C3Softmax

For softmax computation on a vectorxx, we need to calculate:

σ​\(xi\)=exp⁡\(xi\)∑jexp⁡\(xj\)\\sigma\(x\_\{i\}\)=\\frac\{\\exp\(x\_\{i\}\)\}\{\\sum\_\{j\}\\exp\(x\_\{j\}\)\}\(4\)
Thus, computingexp⁡\(x\)\\exp\(x\)efficiently is critical\. There are two common approaches for this calculation: a simple lookup table \(LUT\)\[[48](https://arxiv.org/html/2607.21731#bib.bib10),[13](https://arxiv.org/html/2607.21731#bib.bib19)\], and a Taylor series expansion\[[49](https://arxiv.org/html/2607.21731#bib.bib11)\]\.

The LUT approach requires a large table to store precomputed values and is restricted to a limited range ofxx\. The Taylor series method is accurate near zero but suffers from a significant decline in accuracy asxxdeviates from zero\. Moreover, it incurs higher computational complexity\.

To overcome these limitations, we designed a hardware\-friendly implementation that maintains both high accuracy and low complexity across a wider input range\.

Our method is based on the identity:

ex=2α=2⌊α⌋\+\{α\},whereα=x⋅log2⁡ee^\{x\}=2^\{\\alpha\}=2^\{\\lfloor\\alpha\\rfloor\+\\\{\\alpha\\\}\},\\quad\\text\{where\}\\quad\\alpha=x\\cdot\\log\_\{2\}e\(5\)
Here,log2⁡e\\log\_\{2\}eis a constant, so computingα\\alphais fast\. The integer part,2⌊α⌋2^\{\\lfloor\\alpha\\rfloor\}, can be computed efficiently via bit shifting\. The fractional part,2\{α\}2^\{\\\{\\alpha\\\}\}, lies in the range\[0,1\]\[0,1\]and is approximated using a small LUT with linear interpolation\. This approach significantly reduces memory usage and improves performance while retaining good accuracy\.

To evaluate the effectiveness of our method, we conducted experiments using 10 million random values in the range\[−200,0\]\[\-200,0\]\. Negative range is used, since frameworks like PyTorch\[[19](https://arxiv.org/html/2607.21731#bib.bib60)\]first subtract the maximum value in the vector before applying the exponential function\. We compared the average relative error of our method against both a simple LUT and the Taylor series expansion\.

Our method, using only 0\.5 KB of LUT storage, achieves an average relative error of approximately2\.5×10−32\.5\\times 10^\{\-3\}, which is comparable to the simple LUT method that uses 312\.5 KB \(625× larger table\)\. In contrast, the Taylor series method yields a significantly higher average relative error of2\.6×10772\.6\\times 10^\{77\}\.

We also evaluated all three methods on 10 million random numbers in the range\[−1,0\]\[\-1,0\], where values are close to zero\. In this case, the Taylor series achieves better accuracy, with an error of about5\.9×10−45\.9\\times 10^\{\-4\}, while the performance of our method and the LUT method remains consistent with previous results\.

These experiments demonstrate that our proposed implementation offers competitive accuracy compared to existing methods, while significantly reducing both memory usage and computational complexity\.

After computing eachexp⁡\(xi\)\\exp\(x\_\{i\}\), the next step is to accumulate the results\. If the values are stored locally, this accumulation can also be performed locally without requiring any data movement\. However, for non\-local values, a hierarchical accumulation approach is used, as described in Section[III\-C5](https://arxiv.org/html/2607.21731#S3.SS3.SSS5)\.

#### III\-C4Network

For inter\-bank communication, we adopt a network topology as shown in Figure[3](https://arxiv.org/html/2607.21731#S3.F3), inspired by the design in\[[33](https://arxiv.org/html/2607.21731#bib.bib59)\]\. In this architecture, each channel is composed of multiple banks, and inter\-bank data communication is supported through horizontal and diagonal links\. There is also links between channels\.

To facilitate communication, a small buffer is included in each bank to temporarily store data being sent or received over the interconnect\. These buffers help in decoupling the communication latency from computation\.

Furthermore, a local controller \(Ctrl\) in each bank is responsible for routing the data\. It uses a multiplexer to select which data should be forwarded to the arrays\.

#### III\-C5Hierarchical Transfer

A naive approach to data transfer involves sending all data directly to a centralized unit for processing\. Although conceptually simple, this method suffers from significant drawbacks, most notably high latency and congestion\. Centralizing all communication creates a bottleneck, as data from multiple sources must traverse shared links, leading to increased traffic and longer delays\.

To address this, RED\-PIM adopts a more scalable strategy called hierarchical transfer, where computation and communication are distributed across multiple levels and executed in parallel\. In our design, this method is used for operations such as accumulation during matrix multiplication and softmax computation\.

In the first step, within each bank group, accumulation is performed locally\. For example, in each bankgroup, Bank 0 and Bank 1, sharing the same PCU, accumulate their results in parallel with Bank 2 and Bank 3\. After this step, the results from each pair are further accumulated, e\.g\., the partial sum from Bank 1 is sent to Bank 3 for the next level of accumulation\. This calculation under bankgroup is parallel among all bankgroups, so at the end each of them has a fully accumulated value stored in a designated bank \(denoted asbi​jb\_\{ij\}, whereiiis the channel index andjjis the bank group index\)\.

In the second step, this process continues in parallel across all channels\. Specifically, the designated accumulation banks \(e\.g\.,bib\_\{i\}from each bank group within a channel\) exchange data with their neighboring bank groups, and further accumulation is performed\. Since each channel consists of four bank groups, an additional level of transfer and accumulation is required to combine the intermediate results\. After this final stage, the complete accumulated result for each channel is stored in a single designated bank, denoted asbib\_\{i\}\.

If accumulation across multiple channels or dies is required, the same hierarchical mechanism is extended across dies\.

Figure[6](https://arxiv.org/html/2607.21731#S3.F6)illustrates the hierarchical transfer process using different colors: red arrows represent transfers at the bank group level, green arrows indicate transfers at the channel level, and purple arrows correspond to transfers at the die level\.

Compared to the naive approach, this method offers several key advantages:

- •Reduced communication overhead: By performing early local and group\-wise aggregation, the amount of interconnect traffic is significantly reduced\.
- •Lower latency: Parallel accumulation at each stage allows different parts of the memory hierarchy to work concurrently, avoiding the bottlenecks associated with serialized summation\.
- •Scalability: The structured, multi\-level design naturally scales with the number of banks and channels, maintaining efficiency even in large memory configurations\.

Overall, this hierarchical mechanism accelerates the computation, significantly reducing communication overhead, overall latency, and traffic\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x6.png)Figure 6:Hierarchical data transfer architecture\. Each colored arrow represents a different level of transfer: red for bank group level, green for channel level, and purple for die level\.

### III\-DScalability

In the previous sections, we assumed \(for simplicity of explanation\) that the weight matricesWKW\_\{K\},WQW\_\{Q\}, andWVW\_\{V\}, each of sized×dd\\times d, could fit entirely within a single bank\. This assumption holds only for models with relatively small parameter sizes\. However, RED\-PIM is designed to also handle cases where these weight matrices are too large to fit in a single bank, unlike prior approaches\.

In this scenario, as before, we distribute the tokens across banks, such that each bank holds at mostmmtokens\. The weight matrices are then*tiled column\-wise*, where each bank holds a tile ofkkcolumns\. As a result, each weight matrix is distributed acrossdk\\frac\{d\}\{k\}banks, and the tokens are distributed acrossNm\\frac\{N\}\{m\}banks\.

To compute thekk,QQ, andVVmatrices, we move the weight tiles to the banks where the tokens reside\. Specifically, each bank storing tokens must receive all column vectors from thedk\\frac\{d\}\{k\}weight banks\. This requires data movement ofddcolumn vectors to each of theNm\\frac\{N\}\{m\}token banks for each of the three matrices\. Therefore, the total data movement will be3​N​d2m\\frac\{3Nd^\{2\}\}\{m\}\.

## IVEvaluation Methodology and Results

In this section, we present a comprehensive evaluation of RED\-PIM using simulation of several workloads\.

### IV\-AWorkloads and Datasets

We evaluated RED\-PIM on a diverse set of benchmarks, including the GLUE benchmark suite\[[43](https://arxiv.org/html/2607.21731#bib.bib38)\], which contains datasets designed to assess a model’s performance across various natural language understanding tasks\. The datasets used in our evaluation are described below:

- •MRPC \(Microsoft Research Paraphrase Corpus\):The MRPC dataset\[[10](https://arxiv.org/html/2607.21731#bib.bib36)\]consists of sentence pairs extracted from online news sources, annotated to indicate whether the sentences in each pair are semantically equivalent\. This dataset evaluates a model’s ability to determine semantic similarity between sentence pairs\.
- •QQP \(Quora Question Pairs\):The QQP dataset\[[43](https://arxiv.org/html/2607.21731#bib.bib38)\]comprises pairs of questions sourced from the Quora platform\. The objective of this task is to identify whether two questions are semantically equivalent\.
- •STS\-B \(Semantic Textual Similarity Benchmark\):STS\-B\[[5](https://arxiv.org/html/2607.21731#bib.bib42)\]is a dataset that evaluates a model’s ability to predict the semantic similarity between sentence pairs\. Each pair is annotated with a similarity score ranging from 1 to 5, representing the degree of semantic equivalence\.
- •MNLI \(Multi\-Genre Natural Language Inference\):The MNLI dataset\[[46](https://arxiv.org/html/2607.21731#bib.bib40),[43](https://arxiv.org/html/2607.21731#bib.bib38)\]is a large\-scale corpus for textual entailment tasks\. It includes sentence pairs from various genres annotated with entailment labels \(entailment, contradiction, or neutral\)\. We used both the matched \(in\-domain\) and mismatched \(cross\-domain\) validation splits, containing 19,647 pairs each\.
- •QNLI \(Question Natural Language Inference\):QNLI\[[32](https://arxiv.org/html/2607.21731#bib.bib41),[43](https://arxiv.org/html/2607.21731#bib.bib38)\]is a binary classification task derived from the Stanford Question Answering Dataset \(SQuAD\)\. The goal is to determine if a given context sentence answers a corresponding question\.
- •RTE \(Recognizing Textual Entailment\):The RTE dataset\[[2](https://arxiv.org/html/2607.21731#bib.bib39),[43](https://arxiv.org/html/2607.21731#bib.bib38)\]evaluates textual entailment, requiring models to predict whether a hypothesis logically follows from a given premise\.
- •WNLI \(Winograd NLI\):The WNLI dataset\[[24](https://arxiv.org/html/2607.21731#bib.bib37),[43](https://arxiv.org/html/2607.21731#bib.bib38)\]is adapted from the Winograd Schema Challenge\. It presents challenging reasoning tasks where the model must determine whether a hypothesis follows logically from a given premise\.
- •CoLA \(Corpus of Linguistic Acceptability\):The CoLA dataset\[[45](https://arxiv.org/html/2607.21731#bib.bib43)\]assesses a model’s ability to distinguish between grammatically acceptable and unacceptable sentences\.
- •SST\-2 \(Stanford Sentiment Treebank\):The SST\-2 dataset\[[38](https://arxiv.org/html/2607.21731#bib.bib44)\]contains sentences from movie reviews annotated with sentiment labels\. The task is to classify each sentence as expressing positive or negative sentiment\.

We also utilized additional datasets featuring longer contexts, including thePubMedandArXivdatasets\[[6](https://arxiv.org/html/2607.21731#bib.bib45)\], which contain scientific papers from the biomedical and general scientific domains, respectively\. These datasets are valuable for a wide range of tasks, such as information retrieval, document classification, and other advanced natural language processing applications\. Additionally, we incorporatedGovReport\[[15](https://arxiv.org/html/2607.21731#bib.bib52)\], which comprises reports authored by government research agencies, andWikiHop\[[41](https://arxiv.org/html/2607.21731#bib.bib53)\], a dataset consisting of paragraphs from Wikipedia\. Lastly, we utilized theIMDBdataset\[[28](https://arxiv.org/html/2607.21731#bib.bib54)\], providing movie reviews that serve as a benchmark for sentiment analysis and text classification tasks\.

### IV\-BExperimental Setup

We modify the DAMOV simulation framework\[[30](https://arxiv.org/html/2607.21731#bib.bib49)\]to implement and evaluate our proposed architecture\. DAMOV seamlessly integrates the ZSim\[[34](https://arxiv.org/html/2607.21731#bib.bib50)\]CPU simulator and the Ramulator\[[20](https://arxiv.org/html/2607.21731#bib.bib51)\]memory simulator\. This framework allows us to simulate a configurable number of traditional CPU cores or PIM cores with various memory technologies, including HBM\. In our experiments, we simulate our RED\-PIM architecture on top of FIMDRAM \(i\.e\., HBM\-PIM\) memory with 8 channels\. Additional system configurations for RED\-PIM are detailed in Table[I](https://arxiv.org/html/2607.21731#S4.T1)\.

To evaluate our method in Section[IV\-D](https://arxiv.org/html/2607.21731#S4.SS4), except for the accuracy evaluation, we conducted experiments on a standard transformer block consisting of MHA, FFN, and Add & Normalization layers, as shown in Figure[1](https://arxiv.org/html/2607.21731#S2.F1)\. This block serves as a fundamental component in transformer architectures\.

For the accuracy evaluation \(presented in Section[IV\-D](https://arxiv.org/html/2607.21731#S4.SS4)\), we used thebert\-base\-uncasedmodel and replaced all self\-attention layers with our proposed implementation\.

TABLE I:Architectural Parameters of RED\-PIM\.
### IV\-CBaseline

As the baseline, we consider the standard transformer implementation described in Section[III\-A](https://arxiv.org/html/2607.21731#S3.SS1)and illustrated in Figure[4](https://arxiv.org/html/2607.21731#S3.F4), using FIM\-DRAM for in\-memory computation\. For the softmax operation, we use a Taylor\-series approximation, and for inter\-bank communication, the naive broadcast mechanism is employed, as discussed in Section[III\-C5](https://arxiv.org/html/2607.21731#S3.SS3.SSS5)\. The HBM structure is kept consistent with our optimized implementation to ensure a fair comparison\.

This design represents a straightforward in\-memory implementation without any specific optimizations for data movement or bank\-level parallelism\.

### IV\-DPerformance Analysis

Evaluation on different sequence lengths:To evaluate the efficiency of our proposed method, we conducted experiments across a range of sequence lengths, from 128 tokens up to 1 million\. Figure[7](https://arxiv.org/html/2607.21731#S4.F7)presents a performance comparison between RED\-PIM and the baseline\. The results show that the performance gains of RED\-PIM increase with sequence length\. Specifically, RED\-PIM achieves improvements ranging from 16\.05% to 99\.99%, with a geometric mean speedup of 66\.42% over the baseline\. These results demonstrate that although RED\-PIM provides noticeable improvements even for shorter sequences, it is particularly effective for longer sequences, where the benefits of reduced data movement and improved parallelism become more significant\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x7.png)Figure 7:Execution time improvement vs\. baseline for sequence lengths ranging from 128 to 1M\.Evaluation on long\-context datasets:We further evaluated our approach on five datasets that consist of long sequences, such as full documents and articles\. Specifically, we used IMDB, PubMed, ArXiv, GovReport, and WikiHop\. Figure[8](https://arxiv.org/html/2607.21731#S4.F8)illustrates the distribution of sequence lengths in these datasets, where darker blue shades represent longer sequences\. As shown in the figure, the sequence length distributions vary significantly across datasets, for example, IMDB contains relatively shorter sequences\. The figure also presents performance improvements, indicated by red dots\. Our approach achieves a geometric mean performance improvement of 99\.60%, demonstrating a substantial reduction in execution time\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x8.png)Figure 8:Execution time improvement vs\. baseline across different datasets with longer documents \(indicated by red points, left y\-axis\), along with the sequence length distribution for each dataset \(depicted by the bars, right y\-axis\)\.Evaluation on short\-context datasets:We also tested our approach on datasets with shorter sequence lengths to determine whether our method still provides benefits in such scenarios\. We evaluated RED\-PIM’s performance on the GLUE benchmark, which consists of several natural language understanding tasks with varying sequence lengths\. Figure[9](https://arxiv.org/html/2607.21731#S4.F9)shows the distribution of sequence lengths for each dataset under GLUE separately\. As this figure shows, datasets contain short sequences\.

This figure also demonstrate performance improvement by red dots\.RED\-PIM achieved a performance improvement ranging from 8\.47% to 22\.61%, with a geometric mean improvement of 13\.44%\. While the improvements are smaller compared to those achieved from longer sequences, the consistent performance gain highlights the generality of our approach across different sequence lengths\.

Additionally, the distribution of sequence lengths across the GLUE datasets has a clear correlation with improvement: datasets with a higher proportion of longer sequences exhibit greater performance improvements\. This is expected intuitively since data movement overhead of the baseline grows quadratically with sequence length, whereas our proposal mitigates this effect more effectively\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x9.png)Figure 9:Execution time improvement on the GLUE benchmark for our method compared to baseline \(represented by dots, left y\-axis\), and sequence length distribution in each dataset \(represented by bars, right y\-axis\)\.Accuracy evaluation:Since our algorithm approximates the standard Scaled Dot\-Product Attention, it is important to evaluate its impact on model accuracy\. On the GLUE benchmark, we observed an average performance drop of 0\.58%, while on IMDB, the drop was only 0\.32%\. Interestingly, for GovReport, PubMed, and ArXiv, the model achieved performance improvements of 0\.72%, 4\.29%, and 1\.37%, respectively\.

These results indicate that our approximation introduces negligible changes in accuracy, making it well\-suited for efficient inference in transformer models\.

### IV\-EComparison to Prior Transformer Accelerators

We compared the performance improvement of RED\-PIM against previous transformer accelerators\.

Comparison on long\-context datasets:Figure[10](https://arxiv.org/html/2607.21731#S4.F10)presents the performance improvements on datasets with larger document sizes\. RED\-PIM achieves the highest speedup, with a geometric mean improvement of 99\.60%, significantly outperforming all other methods\. PACT\-3D \(60\.53%\) and HAIMA \(53\.30%\) also deliver notable improvements, though their performance is lower on average than that of RED\-PIM\. TransPIM, while effective in certain scenarios, shows only modest improvements on datasets such as IMDB\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x10.png)Figure 10:Performance improvement analysis of RED\-PIM, TransPIM\[[49](https://arxiv.org/html/2607.21731#bib.bib11)\], HAIMA\[[9](https://arxiv.org/html/2607.21731#bib.bib12)\], and PACT\-3D\[[37](https://arxiv.org/html/2607.21731#bib.bib13)\]on datasets with larger documents \(all improvements are relative to the baseline\)\.Comparison on short\-context datasets:Figure[11](https://arxiv.org/html/2607.21731#S4.F11)illustrates performance gains on the GLUE benchmark, where RED\-PIM consistently outperforms all other methods across all tasks\. TransPIM\[[49](https://arxiv.org/html/2607.21731#bib.bib11)\]achieves moderate improvements, particularly in MRPC and WNLI, while PACT\-3D and HAIMA exhibit more modest gains across most benchmarks\. The geometric mean results highlight RED\-PIM as the most effective approach, delivering an overall performance improvement of 13\.44%, compared to 5\.98% for PACT\-3D, 3\.47% for HAIMA, and 0\.15% for TransPIM \(all relative to the baseline\)\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x11.png)Figure 11:Performance improvement comparison of RED\-PIM, TransPIM\[[49](https://arxiv.org/html/2607.21731#bib.bib11)\], HAIMA\[[9](https://arxiv.org/html/2607.21731#bib.bib12)\], and PACT\-3D\[[37](https://arxiv.org/html/2607.21731#bib.bib13)\]on the GLUE benchmark \(all performance improvements are relative to the baseline\) \.
### IV\-FSensitivity Analysis

#### IV\-F1Sensitivity to embedding size \(dd\)

In the previous experiments, we evaluated RED\-PIM using different input sequence lengths\. Here, we analyze the effect of embedding size \(dd\) on performance\. The embedding size impacts both the amount of data transferred and the number of computations\. For instance, as discussed in Section[III\-B](https://arxiv.org/html/2607.21731#S3.SS2), each query vector that must be moved to the bank storing matrixLLhas a size of1×d1\\times d, so increasingddresults in greater data movement\. Additionally, the matrixLLhas dimensionsd×dd\\times d, meaning that a largerddleads to increased computational cost during attention calculation\.

Figure[12](https://arxiv.org/html/2607.21731#S4.F12)shows the performance improvement of RED\-PIM across varying sequence lengths for different embedding sizes \(64, 128, and 256\)\. The results indicate that performance generally improves with longer sequences, as they benefit more from our optimization strategy\. Additionally, configurations with smallerddvalues show greater improvements due to reduced data transfer and lower computational cost\.

The geometric mean results further suggest that our method remains effective across different embedding sizes, with the differences diminishing as sequence length increases\. This is because, in longer sequences, the effect of the number of tokens \(NN\) on number of data movement becomes dominant, thereby reducing the relative impact ofdd\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x12.png)Figure 12:Sensitivity of model performance to embedding size \(d\) across varying sequence lengths, including the geometric mean performance aggregated over all sequence lengths\.
#### IV\-F2Sensitivity to the number of banks

The number of memory banks per pseudo channel\(sub channel\) can also affect performance\. Increasing the number of banks within a single die reduces inter\-die communication overhead, which is generally more costly than intra\-die communication\. Therefore, having more banks per pseudo channel may lead to better performance compared to configurations with fewer banks and more dies\.

Figure[13](https://arxiv.org/html/2607.21731#S4.F13)illustrates the impact of varying the number of memory banks \(8, 16, and 32\) on performance improvement across different sequence lengths\. The results show a similar trend as in previous experiments: performance increases with longer sequences\. The geometric mean results reveal a slight improvement when using 32 banks\. Interestingly, the configuration with 16 banks shows slightly lower performance than the 8\-bank setup\. This behavior is attributed to interconnect bottlenecks: as the number of banks increases, simultaneous communication between them can saturate internal data paths or memory controller bandwidth\. In such cases, the added parallelism is not fully utilized and instead leads to increased latency, offsetting the benefits of additional banks\.

Overall, these findings demonstrate that RED\-PIM performs effectively across varying embedding sizes and memory bank configurations, ensuring robust performance under diverse workload characteristics\.

![Refer to caption](https://arxiv.org/html/2607.21731v1/x13.png)Figure 13:Sensitivity of model performance to varying numbers of memory banks\.

## VRelated Work

PIM implementations for transformers have attracted attention for reducing data movement and improving efficiency\[[48](https://arxiv.org/html/2607.21731#bib.bib10),[49](https://arxiv.org/html/2607.21731#bib.bib11),[9](https://arxiv.org/html/2607.21731#bib.bib12),[37](https://arxiv.org/html/2607.21731#bib.bib13),[40](https://arxiv.org/html/2607.21731#bib.bib15),[23](https://arxiv.org/html/2607.21731#bib.bib16),[18](https://arxiv.org/html/2607.21731#bib.bib20)\]\. Early work like ReTransformer\[[48](https://arxiv.org/html/2607.21731#bib.bib10)\]used ReRAM\-based vector\-matrix and matrix\-matrix multiplication and lookup\-based softmax to accelerate QKV and attention computations\. However, such designs suffer from low precision and reliability, as well as limitations of ReRAM technology, which faces practical obstacles to large\-scale manufacturing and has not yet been commercialized\.

TransPIM\[[49](https://arxiv.org/html/2607.21731#bib.bib11)\]uses an HBM\-based architecture that distributes tokens across banks and shifts keys in a Ring Broadcast and Compute pattern for attention\. It employs Ambit\[[35](https://arxiv.org/html/2607.21731#bib.bib46)\]for bit\-serial point\-wise multiplications and includes an Auxiliary Compute Unit with an adder tree and buffer in each bank\. Exponentiation is approximated using a fifth\-order Taylor expansion via PIM operations\. While the ring pattern reduces data movement, serial processing introduces high latency, each 8\-bit multiplication requires hundreds of cycles due to repeated Activate\-Activate\-Precharge \(AAP\) operations\. AAP also disrupts DRAM timing and requires decoder changes for parallel row activation\. As a result, performance gains are limited, particularly for long sequences\.

HAIMA\[[9](https://arxiv.org/html/2607.21731#bib.bib12)\]optimizes computation by coordinating SRAM, DRAM, and the host\. The host handles softmax and add\-normalization, while other operations are distributed across memory hierarchies\. However, frequent data transfers between SRAM, DRAM, and the host introduce high energy consumption and latency, with bus transactions becoming a performance bottleneck\.

PACT\-3D\[[37](https://arxiv.org/html/2607.21731#bib.bib13)\]employs NMP\-Units with Neuron Processing Elements \(NPEs\) in each DRAM layer for in\-memory computation, while RISC\-V processors in the HBM logic layer handle normalization and softmax\. However, integrating NPEs and RISC\-V cores increases hardware overhead and chip complexity, and high\-precision computations with NPEs result in longer latencies\. Additionally, frequent data movement between DRAM banks and the RISC\-V processors introduces further latency and data movement overhead\.

Moreover, there have been efforts to improve the computational efficiency of self\-attention mechanisms using locality\-sensitive hashing and bucketing methods\. Notable examples include IMCAT\[[22](https://arxiv.org/html/2607.21731#bib.bib48)\]and Cai et al\[[4](https://arxiv.org/html/2607.21731#bib.bib47)\]\. These approaches reduce computation cost but increase complexity due to the bucketing and hashing operations, potentially increasing latency and memory overhead\. Furthermore, IMCAT hashes keys into binary signatures and computes self\-attention using only themmclosest keys by Hamming distance\. This risks losing important information by ignoring many relevant token relationships\.

In summary, while prior works have explored various Processing\-in\-Memory solutions for transformer models, they often target small\-scale models or rely on impractical or overly complex hardware mechanisms\. These designs struggle to scale to larger models with high\-dimensional weights and long input sequences\. Most importantly, they do not address the significant challenge of data movement\.

## VIConclusion

In this work, we addressed a key bottleneck in transformer models, the excessive data movement incurred during self\-attention operations, by proposing RED\-PIM, a PIM\-based solution tailored for High\-Bandwidth Memory \(HBM\) architectures\. RED\-PIM restructures attention computation to operate on a compact intermediate representation, significantly reducing memory usage and inter\-bank communication\. Our algorithmic and architectural co\-design leverages local computations within memory banks, efficient inter\-bank communication strategies, and hardware\-friendly softmax approximation to achieve scalable and low\-latency transformer inference\.

To evaluate our approach, we developed a simulation framework based on DAMOV, which enables detailed analysis of data movement in HBM architectures, showing up to a 99\.99% improvement for long sequences and an average improvement of 66\.42% over a simple PIM baseline with little impact on model accuracy\.

Along with these improvements, some limitations remain to be explored in future work:

- •Reduced Usable Bank Capacity:The use of FIMDRAM involves allocating portions of each memory bank for PCUs, which reduces the available storage\.
- •Simulation\-Based Evaluation:Due to the lack of available commercial or open\-source hardware support, our PIM architecture is evaluated through simulation, which models data movement and in\-memory computation\.
- •Focus on Inference:Our work focuses on inference\. Adapting RED\-PIM for training introduces challenges like gradient computation and intermediate data storage, which require further analysis and optimization\.

## References

- \[1\]J\. Ahn, S\. Hong, S\. Yoo, O\. Mutlu, and K\. Choi\(2015\)A scalable processing\-in\-memory accelerator for parallel graph processing\.InProceedings of the 42nd Annual International Symposium on Computer Architecture,pp\. 105–117\.Cited by:[§II\-B](https://arxiv.org/html/2607.21731#S2.SS2.p1.1)\.
- \[2\]L\. Bentivogli, P\. Clark, I\. Dagan, and D\. Giampiccolo\(2009\)The fifth pascal recognizing textual entailment challenge\.\.TAC7\(8\),pp\. 1\.Cited by:[6th item](https://arxiv.org/html/2607.21731#S4.I1.i6.p1.1)\.
- \[3\]A\. Boroumand, S\. Ghose, Y\. Kim, R\. Ausavarungnirun, E\. Shiu, R\. Thakur, D\. Kim, A\. Kuusela, A\. Knies, P\. Ranganathan,et al\.\(2018\)Google workloads for consumer devices: mitigating data movement bottlenecks\.InProceedings of the Twenty\-Third International Conference on Architectural Support for Programming Languages and Operating Systems,pp\. 316–331\.Cited by:[§II\-B](https://arxiv.org/html/2607.21731#S2.SS2.p1.1)\.
- \[4\]J\. Cai, M\. A\. Kaleem, R\. Genov, M\. R\. Azghadi, and A\. Amirsoleimani\(2024\)In\-memory transformer self\-attention mechanism using passive memristor crossbar\.In2024 IEEE International Symposium on Circuits and Systems \(ISCAS\),pp\. 1–5\.Cited by:[§V](https://arxiv.org/html/2607.21731#S5.p5.1)\.
- \[5\]D\. Cer, M\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. Specia\(2017\)Semeval\-2017 task 1: semantic textual similarity\-multilingual and cross\-lingual focused evaluation\.arXiv preprint arXiv:1708\.00055\.Cited by:[3rd item](https://arxiv.org/html/2607.21731#S4.I1.i3.p1.1)\.
- \[6\]A\. Cohan, F\. Dernoncourt, D\. S\. Kim, T\. Bui, S\. Kim, W\. Chang, and N\. Goharian\(2018\)A discourse\-aware attention model for abstractive summarization of long documents\.arXiv preprint arXiv:1804\.05685\.Cited by:[5th item](https://arxiv.org/html/2607.21731#S1.I1.i5.p1.1),[§IV\-A](https://arxiv.org/html/2607.21731#S4.SS1.p3.1)\.
- \[7\]G\. DeepMind\(2024\)Gemini 2 models\.Note:Available at[https://deepmind\.google/technologies/gemini](https://deepmind.google/technologies/gemini)Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1)\.
- \[8\]J\. Ding, S\. Ma, L\. Dong, X\. Zhang, S\. Huang, W\. Wang, N\. Zheng, and F\. Wei\(2023\)Longnet: scaling transformers to 1,000,000,000 tokens\.arXiv preprint arXiv:2307\.02486\.Cited by:[§II\-A](https://arxiv.org/html/2607.21731#S2.SS1.p10.1)\.
- \[9\]Y\. Ding, C\. Liu, M\. Duan, W\. Chang, K\. Li, and K\. Li\(2023\)HAIMA: a hybrid sram and dram accelerator\-in\-memory architecture for transformer\.In2023 60th ACM/IEEE Design Automation Conference \(DAC\),pp\. 1–6\.Cited by:[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p2.1),[Figure 10](https://arxiv.org/html/2607.21731#S4.F10),[Figure 11](https://arxiv.org/html/2607.21731#S4.F11),[§V](https://arxiv.org/html/2607.21731#S5.p1.1),[§V](https://arxiv.org/html/2607.21731#S5.p3.1)\.
- \[10\]B\. Dolan and C\. Brockett\(2005\)Automatically constructing a corpus of sentential paraphrases\.InThird international workshop on paraphrasing \(IWP2005\),Cited by:[1st item](https://arxiv.org/html/2607.21731#S4.I1.i1.p1.1)\.
- \[11\]A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§II\-A](https://arxiv.org/html/2607.21731#S2.SS1.p10.1)\.
- \[12\]S\. Ghose, A\. Boroumand, J\. S\. Kim, J\. Gómez\-Luna, and O\. Mutlu\(2019\)Processing\-in\-memory: a workload\-driven perspective\.IBM Journal of Research and Development63\(6\),pp\. 3–1\.Cited by:[§II\-B](https://arxiv.org/html/2607.21731#S2.SS2.p1.1)\.
- \[13\]W\. Han, H\. Cho, D\. Kim, and J\. Kim\(2024\)SAL\-pim: a subarray\-level processing\-in\-memory architecture with lut\-based linear interpolation for transformer\-based text generation\.arXiv preprint arXiv:2401\.17005\.Cited by:[§III\-C3](https://arxiv.org/html/2607.21731#S3.SS3.SSS3.p3.1)\.
- \[14\]G\. Heo, S\. Lee, J\. Cho, H\. Choi, S\. Lee, H\. Ham, G\. Kim, D\. Mahajan, and J\. Park\(2024\)Neupims: npu\-pim heterogeneous acceleration for batched llm inferencing\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 722–737\.Cited by:[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p2.1)\.
- \[15\]L\. Huang, S\. Cao, N\. Parulian, H\. Ji, and L\. Wang\(2021\)Efficient attentions for long document summarization\.arXiv preprint arXiv:2104\.02112\.Cited by:[5th item](https://arxiv.org/html/2607.21731#S1.I1.i5.p1.1),[§IV\-A](https://arxiv.org/html/2607.21731#S4.SS1.p3.1)\.
- \[16\]H\. Jun, J\. Cho, K\. Lee, H\. Son, K\. Kim, H\. Jin, and K\. Kim\(2017\)Hbm \(high bandwidth memory\) dram technology and architecture\.In2017 IEEE International Memory Workshop \(IMW\),pp\. 1–4\.Cited by:[§II\-D](https://arxiv.org/html/2607.21731#S2.SS4.p1.1)\.
- \[17\]A\. K\. Kabat, S\. Pandey, and V\. T\. Gopalakrishnan\(2022\)Performance evaluation of high bandwidth memory for hpc workloads\.In2022 IEEE 35th International System\-on\-Chip Conference \(SOCC\),Vol\.,pp\. 1–6\.External Links:[Document](https://dx.doi.org/10.1109/SOCC56010.2022.9908071)Cited by:[§II\-D](https://arxiv.org/html/2607.21731#S2.SS4.p1.1)\.
- \[18\]M\. Kang, H\. Shin, and L\. Kim\(2021\)A framework for accelerating transformer\-based language model on reram\-based architecture\.IEEE Transactions on Computer\-Aided Design of Integrated Circuits and Systems41\(9\),pp\. 3026–3039\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1),[§V](https://arxiv.org/html/2607.21731#S5.p1.1)\.
- \[19\]N\. Ketkar, J\. Moolayil, N\. Ketkar, and J\. Moolayil\(2021\)Introduction to pytorch\.Deep learning with python: learn best practices of deep learning models with PyTorch,pp\. 27–91\.Cited by:[§III\-C3](https://arxiv.org/html/2607.21731#S3.SS3.SSS3.p9.1)\.
- \[20\]Y\. Kim, W\. Yang, and O\. Mutlu\(2016\)Ramulator: a fast and extensible dram simulator\.IEEE Computer Architecture Letters15\(1\),pp\. 45–49\.External Links:[Document](https://dx.doi.org/10.1109/LCA.2015.2414456)Cited by:[§IV\-B](https://arxiv.org/html/2607.21731#S4.SS2.p1.1)\.
- \[21\]Y\. Kwon, S\. H\. Lee, J\. Lee, S\. Kwon, J\. M\. Ryu, J\. Son, O\. Seongil, H\. Yu, H\. Lee, S\. Y\. Kim,et al\.\(2021\)25\.4 a 20nm 6gb function\-in\-memory dram, based on hbm2 with a 1\.2 tflops programmable computing unit using bank\-level parallelism, for machine learning applications\.In2021 IEEE International Solid\-State Circuits Conference \(ISSCC\),Vol\.64,pp\. 350–352\.Cited by:[3rd item](https://arxiv.org/html/2607.21731#S1.I1.i3.p1.1),[§II\-D](https://arxiv.org/html/2607.21731#S2.SS4.p2.1),[§III\-C1](https://arxiv.org/html/2607.21731#S3.SS3.SSS1.p1.1)\.
- \[22\]A\. F\. Laguna, A\. Kazemi, M\. Niemier, and X\. S\. Hu\(2021\)In\-memory computing based accelerator for transformer networks for long sequences\.In2021 Design, Automation & Test in Europe Conference & Exhibition \(DATE\),pp\. 1839–1844\.Cited by:[§V](https://arxiv.org/html/2607.21731#S5.p5.1)\.
- \[23\]A\. F\. Laguna, M\. M\. Sharifi, A\. Kazemi, X\. Yin, M\. Niemier, and X\. S\. Hu\(2022\)Hardware\-software co\-design of an in\-memory transformer network accelerator\.Frontiers in Electronics3,pp\. 847069\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p3.1),[§V](https://arxiv.org/html/2607.21731#S5.p1.1)\.
- \[24\]H\. Levesque, E\. Davis, and L\. Morgenstern\(2012\)The winograd schema challenge\.InThirteenth international conference on the principles of knowledge representation and reasoning,Cited by:[7th item](https://arxiv.org/html/2607.21731#S4.I1.i7.p1.1)\.
- \[25\]C\. Li, Z\. Zhou, S\. Zheng, J\. Zhang, Y\. Liang, and G\. Sun\(2024\)SpecPIM: accelerating speculative inference on pim\-enabled system via architecture\-dataflow co\-exploration\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 950–965\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1)\.
- \[26\]W\. Li, M\. Manley, J\. Read, A\. Kaul, M\. S\. Bakir, and S\. Yu\(2023\)H3datten: heterogeneous 3\-d integrated hybrid analog and digital compute\-in\-memory accelerator for vision transformer self\-attention\.IEEE Transactions on Very Large Scale Integration \(VLSI\) Systems\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1)\.
- \[27\]Z\. Lu, X\. Wang, M\. T\. Arafin, H\. Yang, Z\. Liu, J\. Zhang, and G\. Qu\(2023\)An rram\-based computing\-in\-memory architecture and its application in accelerating transformer inference\.IEEE Transactions on Very Large Scale Integration \(VLSI\) Systems\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p3.1)\.
- \[28\]A\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. Potts\(2011\)Learning word vectors for sentiment analysis\.InProceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies,pp\. 142–150\.Cited by:[5th item](https://arxiv.org/html/2607.21731#S1.I1.i5.p1.1),[§IV\-A](https://arxiv.org/html/2607.21731#S4.SS1.p3.1)\.
- \[29\]O\. Mutlu, S\. Ghose, J\. Gómez\-Luna, and R\. Ausavarungnirun\(2022\)A modern primer on processing in memory\.InEmerging computing: from devices to systems: looking beyond Moore and Von Neumann,pp\. 171–243\.Cited by:[§II\-B](https://arxiv.org/html/2607.21731#S2.SS2.p2.1)\.
- \[30\]G\. F\. Oliveira, J\. Gómez\-Luna, L\. Orosa, S\. Ghose, N\. Vijaykumar, I\. Fernandez, M\. Sadrosadati, and O\. Mutlu\(2021\)DAMOV: A new methodology and benchmark suite for evaluating data movement bottlenecks\.IEEE Access9,pp\. 134457–134502\.Cited by:[4th item](https://arxiv.org/html/2607.21731#S1.I1.i4.p1.1),[§IV\-B](https://arxiv.org/html/2607.21731#S4.SS2.p1.1)\.
- \[31\]OpenAI\(2025\)Introducing gpt\-4\.1 in the api\.Note:OpenAI blogExternal Links:[Link](https://openai.com/)Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1)\.
- \[32\]P\. Rajpurkar\(2016\)Squad: 100,000\+ questions for machine comprehension of text\.arXiv preprint arXiv:1606\.05250\.Cited by:[5th item](https://arxiv.org/html/2607.21731#S4.I1.i5.p1.1)\.
- \[33\]S\. H\. S\. Rezaei, M\. Modarressi, R\. Ausavarungnirun, M\. Sadrosadati, O\. Mutlu, and M\. Daneshtalab\(2020\)NoM: network\-on\-memory for inter\-bank data transfer in highly\-banked memories\.IEEE Computer Architecture Letters19\(1\),pp\. 80–83\.Cited by:[§III\-C4](https://arxiv.org/html/2607.21731#S3.SS3.SSS4.p1.1)\.
- \[34\]D\. Sanchez and C\. Kozyrakis\(2013\)ZSim: fast and accurate microarchitectural simulation of thousand\-core systems\.InProceedings of the 40th Annual International Symposium on Computer Architecture,New York, NY, USA,pp\. 475–486\.External Links:ISBN 9781450320795,[Document](https://dx.doi.org/10.1145/2485922.2485963)Cited by:[§IV\-B](https://arxiv.org/html/2607.21731#S4.SS2.p1.1)\.
- \[35\]V\. Seshadri, D\. Lee, T\. Mullins, H\. Hassan, A\. Boroumand, J\. Kim, M\. A\. Kozuch, O\. Mutlu, P\. B\. Gibbons, and T\. C\. Mowry\(2017\)Ambit: in\-memory accelerator for bulk bitwise operations using commodity dram technology\.InProceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture,pp\. 273–287\.Cited by:[§V](https://arxiv.org/html/2607.21731#S5.p2.1)\.
- \[36\]Z\. Shen, M\. Zhang, H\. Zhao, S\. Yi, and H\. Li\(2021\)Efficient attention: attention with linear complexities\.InProceedings of the IEEE/CVF winter conference on applications of computer vision,pp\. 3531–3539\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p5.4),[§III\-B](https://arxiv.org/html/2607.21731#S3.SS2.p2.1)\.
- \[37\]G\. Singh and S\. Vrudhula\(2024\)A dram\-based near\-memory architecture for accelerated and energy\-efficient execution of transformers\.InProceedings of the Great Lakes Symposium on VLSI 2024,pp\. 57–62\.Cited by:[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p2.1),[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p3.1),[Figure 10](https://arxiv.org/html/2607.21731#S4.F10),[Figure 11](https://arxiv.org/html/2607.21731#S4.F11),[§V](https://arxiv.org/html/2607.21731#S5.p1.1),[§V](https://arxiv.org/html/2607.21731#S5.p4.1)\.
- \[38\]R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Y\. Ng, and C\. Potts\(2013\)Recursive deep models for semantic compositionality over a sentiment treebank\.InProceedings of the 2013 conference on empirical methods in natural language processing,pp\. 1631–1642\.Cited by:[9th item](https://arxiv.org/html/2607.21731#S4.I1.i9.p1.1)\.
- \[39\]K\. Sohn, W\. Yun, R\. Oh, C\. Oh, S\. Seo, M\. Park, D\. Shin, W\. Jung, S\. Shin, J\. Ryu,et al\.\(2016\)A 1\.2 v 20 nm 307 gb/s hbm dram with at\-speed wafer\-level io test scheme and adaptive refresh considering temperature distribution\.IEEE Journal of Solid\-State Circuits52\(1\),pp\. 250–260\.Cited by:[§II\-D](https://arxiv.org/html/2607.21731#S2.SS4.p1.1)\.
- \[40\]S\. Sridharan, J\. R\. Stevens, K\. Roy, and A\. Raghunathan\(2023\)X\-former: in\-memory acceleration of transformers\.IEEE Transactions on Very Large Scale Integration \(VLSI\) Systems31\(8\),pp\. 1223–1233\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p3.1),[§V](https://arxiv.org/html/2607.21731#S5.p1.1)\.
- \[41\]M\. Tu, G\. Wang, J\. Huang, Y\. Tang, X\. He, and B\. Zhou\(2019\)Multi\-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs\.arXiv preprint arXiv:1905\.07374\.Cited by:[5th item](https://arxiv.org/html/2607.21731#S1.I1.i5.p1.1),[§IV\-A](https://arxiv.org/html/2607.21731#S4.SS1.p3.1)\.
- \[42\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 6000–6010\.External Links:ISBN 9781510860964Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p2.3),[Figure 1](https://arxiv.org/html/2607.21731#S2.F1),[§II\-A](https://arxiv.org/html/2607.21731#S2.SS1.p1.1)\.
- \[43\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.arXiv preprint arXiv:1804\.07461\.Cited by:[5th item](https://arxiv.org/html/2607.21731#S1.I1.i5.p1.1),[2nd item](https://arxiv.org/html/2607.21731#S4.I1.i2.p1.1),[4th item](https://arxiv.org/html/2607.21731#S4.I1.i4.p1.1),[5th item](https://arxiv.org/html/2607.21731#S4.I1.i5.p1.1),[6th item](https://arxiv.org/html/2607.21731#S4.I1.i6.p1.1),[7th item](https://arxiv.org/html/2607.21731#S4.I1.i7.p1.1),[§IV\-A](https://arxiv.org/html/2607.21731#S4.SS1.p1.1)\.
- \[44\]H\. Wang, S\. Ma, L\. Dong, S\. Huang, D\. Zhang, and F\. Wei\(2024\)Deepnet: scaling transformers to 1,000 layers\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§II\-A](https://arxiv.org/html/2607.21731#S2.SS1.p10.1)\.
- \[45\]A\. Warstadt, A\. Singh, and S\. R\. Bowman\(2019\)Neural network acceptability judgments\.Transactions of the Association for Computational Linguistics7,pp\. 625–641\.Cited by:[8th item](https://arxiv.org/html/2607.21731#S4.I1.i8.p1.1)\.
- \[46\]A\. Williams, N\. Nangia, and S\. R\. Bowman\(2017\)A broad\-coverage challenge corpus for sentence understanding through inference\.arXiv preprint arXiv:1704\.05426\.Cited by:[4th item](https://arxiv.org/html/2607.21731#S4.I1.i4.p1.1)\.
- \[47\]xAI\(2024\)Grok by xai\.Note:Available at[https://x\.ai](https://x.ai/)Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1)\.
- \[48\]X\. Yang, B\. Yan, H\. Li, and Y\. Chen\(2020\)ReTransformer: reram\-based processing\-in\-memory architecture for transformer acceleration\.InProceedings of the 39th International Conference on Computer\-Aided Design,pp\. 1–9\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p3.1),[§III\-C3](https://arxiv.org/html/2607.21731#S3.SS3.SSS3.p3.1),[§V](https://arxiv.org/html/2607.21731#S5.p1.1)\.
- \[49\]M\. Zhou, W\. Xu, J\. Kang, and T\. Rosing\(2022\)Transpim: a memory\-based acceleration via software\-hardware co\-design for transformer\.In2022 IEEE International Symposium on High\-Performance Computer Architecture \(HPCA\),pp\. 1071–1085\.Cited by:[§I](https://arxiv.org/html/2607.21731#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.21731#S2.SS3.p3.1),[§III\-A](https://arxiv.org/html/2607.21731#S3.SS1.p1.5),[§III\-C3](https://arxiv.org/html/2607.21731#S3.SS3.SSS3.p3.1),[Figure 10](https://arxiv.org/html/2607.21731#S4.F10),[Figure 11](https://arxiv.org/html/2607.21731#S4.F11),[§IV\-E](https://arxiv.org/html/2607.21731#S4.SS5.p3.1),[§V](https://arxiv.org/html/2607.21731#S5.p1.1),[§V](https://arxiv.org/html/2607.21731#S5.p2.1)\.
- \[50\]M\. Zhu, Y\. Zhuo, C\. Wang, W\. Chen, and Y\. Xie\(2018\)Performance evaluation and optimization of hbm\-enabled gpu for data\-intensive applications\.IEEE Transactions on Very Large Scale Integration \(VLSI\) Systems26\(5\),pp\. 831–840\.Cited by:[§II\-D](https://arxiv.org/html/2607.21731#S2.SS4.p1.1)\.