An MLIR-Based Compilation Method for Large Language Models

# An MLIR-Based Compilation Method for Large Language Models
Source: [https://arxiv.org/html/2607.15865](https://arxiv.org/html/2607.15865)
Pengchao Hu Zhibin Xin Yifan Chen Yangyang Zhou Liang Wang \{pengchao\.hu,zhibin\.xin,yifan\.chen,yangyang\.zhou,liang\.wang01\}@sophgo\.com Sophgo Inc

###### Abstract

Large Language Models \(LLMs\) have become the dominant workload on modern AI accelerators, yet deploying them on specialized hardware still faces two core challenges: how to import a trained model into a compiler\-friendly intermediate representation, and how to efficiently schedule the autoregressive inference loop under limited on\-chip memory\. This paper presents an MLIR \(Multi\-Level Intermediate Representation\) based compilation method for large language models, illustrated using two dialects of operators, TopOp and TpuOp\. TopOp serves as a high\-level graph dialect that is independent of both the source framework and the target chip, and is responsible for expressing model semantics; TpuOp serves as the target hardware dialect, carrying chip\-related decisions such as quantization, layer groups, and memory layout\. A model is first represented as TopOp, then lowered layer by layer to TpuOp, and finally a deployable binary is generated\. In addition, each Transformer layer is split into three stages for static compilation: prefill, prefill\_kv \(prefill with historical key\-value cache\), and decode, so as to accommodate the different computational characteristics of prompt\-parallel processing and per\-token generation\. The method has been implemented in the TPU\-MLIR compiler111https://github\.com/sophgo/tpu\-mlirand the LLM\-TPU deployment project222https://github\.com/sophgo/LLM\-TPU, supporting a variety of generative models including the Qwen, Llama, InternVL, and MiniCPM\-V series, as well as multiple quantization and deployment forms such as GPTQ, AWQ, and AutoRound\.

## 1Introduction

Transformer\-based large language models\[[5](https://arxiv.org/html/2607.15865#bib.bib3)\]have evolved from research prototypes into production\-grade services\. Frameworks such as PyTorch and HuggingFace Transformers\[[6](https://arxiv.org/html/2607.15865#bib.bib4)\]greatly simplify training and experimentation, but the resulting model weights usually cannot run directly on specialized AI accelerators\. The traditional approach of hand\-writing operator libraries for each hardware is costly and hard to keep pace with model evolution\. Therefore, the industry increasingly relies on domain\-specific compilers to automate the transformation from model description to hardware instructions\.

MLIR\[[4](https://arxiv.org/html/2607.15865#bib.bib1)\]provides a reusable and extensible compiler infrastructure that allows developers to express computation at different levels of abstraction through custom dialects\. This paper presents an MLIR compilation method for large language models, illustrated concretely with TopOp and TpuOp: TopOp first captures framework\-agnostic model semantics, which is then converted to target\-specific TpuOp through a standard lowering pipeline\. To address the difference between prompt processing and token generation in autoregressive inference, each Transformer layer is statically compiled into three variants: prefill, prefill\_kv, and decode\.

This paper focuses on two topics: how to import a large language model with TopOp and deploy it to an accelerator through TpuOp; and why an LLM is split into the three stages of prefill, prefill\_kv, and decode, and how each stage is represented in the compilation flow\. We intentionally keep the discussion at the methodological level; concrete implementation details \(such as chip\-level operator fusion, quantization tuning, and layer\-group slicing\) can be adjusted flexibly according to the target hardware\.

## 2Background

### 2\.1MLIR and Multi\-Level Dialects

MLIR\[[4](https://arxiv.org/html/2607.15865#bib.bib1)\]is a novel compiler infrastructure that emphasizes reusable and extensible intermediate representations\. Its core abstractions include operations, values, types, attributes, and dialects\. A dialect logically organizes a set of related operations, types, and attributes, allowing the same program to coexist at different levels of abstraction and to be progressively lowered to the target representation\.

When targeting deep learning compilation, two levels of dialects are typically defined:

- •High\-level graph dialect: independent of the source framework and the target chip, expressing the semantics of the neural network graph, such asMatMul,RMSNorm,Rope,FAttention\(short for Full Attention\),Reshape,Concat,MLP, etc\.
- •Target dialect: after determining the quantization mode, data type, memory layout, and hardware instructions, expressing the same computation\. This dialect directly targets code generation\.

A standard pass pipeline is responsible for lowering the high\-level graph dialect operators to the target dialect operators, performing calibration when needed, running layer\-group and memory planning, and finally generating the executable binary for the target hardware\.

### 2\.2TopOp and TpuOp Examples

For ease of exposition, this paper uses TopOp and TpuOp as concrete examples of the two dialects of operators\.

TopOpis the set of operators in the high\-level graph dialect, used to encode the semantics of deep learning graphs\. It is independent of source frameworks such as PyTorch and TensorFlow, and also independent of any specific accelerator\. Typical TopOps include:

- •top\.MatMul: matrix multiplication;
- •top\.RMSNorm: RMS normalization;
- •top\.Gather: fetching data based on indices;
- •top\.Rope: RoPE operation for LLMs;
- •top\.Reshape: tensor reshape;
- •top\.Concat: concatenating tensors along a specified axis;
- •top\.FAttention: Full Attention \(FAttention\) for LLMs, encapsulating scaled dot\-product attention and the causal mask;
- •top\.MLP: the MLP operator for LLMs;
- •top\.Weight: referencing weight data\.

TopOp usually operates on tensor values ofRankedTensorType, maintaining a high level of abstraction that facilitates hardware\-independent graph transformations and equivalent rewrites\. Among them,top\.MLPandtop\.FAttentionare kept as fused high\-level operators rather than expanded into basic operators such asMatMul\+ activation \+ residual, because their target\-chip implementations usually require dedicated fused instruction sequences \(such as attention fusion kernels\[[1](https://arxiv.org/html/2607.15865#bib.bib6)\]and MLP fusion kernels\)\. Keeping the fusion boundary at the high level allows the lowering stage to choose the optimal fusion strategy for different chips, rather than passively matching at the granularity of basic operators\.

TpuOpis the set of operators in the target hardware dialect, generated by the lowering pass according to the target chip characteristics after TopOp has expressed the semantics\. Compared with TopOp, TpuOp additionally carries chip\-related attributes, such as:

- •data type \(F32, BF16, F16, INT8, etc\.\);
- •quantization format support, including symmetric/asymmetric INT8, and direct\-through compilation of weight\-quantized models such as AWQ / GPTQ / AutoRound;
- •layer\-group information \(group\_info\) for on\-chip memory scheduling;
- •operator splitting information \(e\.g\., splitting by the number of cores\);
- •address information \(required memory size and storage offset\);
- •target chip instruction encapsulation\.

The relationship between TopOp and TpuOp is shown in Figure[1](https://arxiv.org/html/2607.15865#S2.F1)\.

![Refer to caption](https://arxiv.org/html/2607.15865v1/dialects.png)Figure 1:Relationship between TopOp and TpuOp\.
### 2\.3Overview of LLM Inference

The rough execution process of an LLM is as follows:

1. 1\.Convert the input text into input tokens;
2. 2\.Obtain hidden states through word embedding;
3. 3\.Pass throughnum\_layersblocks to produce hidden states;
4. 4\.Obtain logits through the LM head;
5. 5\.Output a token from the logits via a sampling algorithm;
6. 6\.Repeat steps 2–5 for that token until an end\-of\-sequence symbol is encountered\.

The block computation mainly consists of RoPE, Full Attention, MLP, etc\., and each block generates the corresponding K cache and V cache for each token, collectively referred to as the KV Cache\. The overall execution flow of the large model is shown in Figure[2](https://arxiv.org/html/2607.15865#S2.F2)\.

For ease of understanding, the figure assumes:num\_layersis 32, the number of input tokens is 100,hidden\_sizeis 4096,kv\_headis 2, andhead\_dimis 128; o0, o1, and o2 represent the sequentially output tokens\.

![Refer to caption](https://arxiv.org/html/2607.15865v1/llm_flow.png)Figure 2:LLM execution flow\.

## 3Building Large Models with TopOp

The first topic of this paper is the complete path from a trained checkpoint to a target hardware binary\. Takingllm\_convert\.pyin TPU\-MLIR\[[2](https://arxiv.org/html/2607.15865#bib.bib2)\]as an example, the script loads the HuggingFace model configuration and weight shards, dispatches to the corresponding converter according to the model family \(e\.g\., for dense decoder\-only Transformers, grouped\-query attention variants, and multimodal language models, respectively\), and then sequentially completes TopOp MLIR generation, lowering of each module to TpuOp, compilation, and result merging\.

### 3\.1Model Import and TopOp Module Generation

Below is a TopOp pseudocode example for one block \(some parameters and shape annotations are omitted for brevity\)\.

1

2input=top\.InputOp\("input\_states",0\)

3pos\_ids=top\.InputOp\("position\_ids",1\)

4

5x=top\.RMSNorm\(input\)

6q=top\.MatMul\(x,w\_q\)

7k=top\.MatMul\(x,w\_k\)

8v=top\.MatMul\(x,w\_v\)

9cos=top\.Gather\(rotary\_cos\_w,pos\_ids\)

10sin=top\.Gather\(rotary\_sin\_w,pos\_ids\)

11q=top\.Rope\(q,cos,sin\)

12k=top\.Rope\(k,cos,sin\)

13attn=top\.FAttention\(q,k,v\)

14h=top\.Add\(input,top\.MatMul\(attn,w\_o\)\)

15o=top\.Add\(h,top\.MLP\(top\.RMSNorm\(h\)\)\)

16returntop\.Return\(o,k,v\)

Listing 1: A block \(without historical KV\)\.1

2input=top\.InputOp\("input\_states",0\)

3pos\_ids=top\.InputOp\("position\_ids",1\)

4k\_cache=top\.InputOp\("history\_k",2\)

5v\_cache=top\.InputOp\("history\_v",3\)

6

7x=top\.RMSNorm\(input\)

8q=top\.MatMul\(x,w\_q\)

9k=top\.MatMul\(x,w\_k\)

10v=top\.MatMul\(x,w\_v\)

11cos=top\.Gather\(rotary\_cos\_w,pos\_ids\)

12sin=top\.Gather\(rotary\_sin\_w,pos\_ids\)

13q=top\.Rope\(q,cos,sin\)

14k=top\.Rope\(k,cos,sin\)

15k\_all=top\.Concat\(k\_cache,k\)

16v\_all=top\.Concat\(v\_cache,v\)

17attn=top\.FAttention\(q,k\_all,v\_all\)

18h=top\.Add\(input,top\.MatMul\(attn,w\_o\)\)

19o=top\.Add\(h,top\.MLP\(top\.RMSNorm\(h\)\)\)

20returntop\.Return\(o,k,v\)

Listing 2: A block \(with historical KV\)\.Heretop\.Gatheris used to fetch the precomputed cos/sin values according toposition\_ids, rather than computing RoPE directly at runtime; the reason is explained in Section[4\.6](https://arxiv.org/html/2607.15865#S4.SS6)on performance optimization\.

After the frontend construction, the generated MLIR file format is roughly as follows:

1func\.func@block\_0\(%arg0:tensor<1x2048x4096xf32\>,\.\.\.\)\-\>\.\.\.\{

2%0="top\.RMSNorm"\(%arg0,%weight0\)\.\.\.

3%1="top\.MatMul"\(%0,%weight\_q\)\.\.\.

4%2="top\.MatMul"\(%0,%weight\_k\)\.\.\.

5%3="top\.MatMul"\(%0,%weight\_v\)\.\.\.

6\.\.\.

7%i="top\.FAttention"\(%q,%k\_cache,%v\_cache,\.\.\.\)\.\.\.

8\.\.\.

9%n="top\.MLP"\(%m\)\.\.\.

10\.\.\.

11return%hidden,%k\_cache,%v\_cache:\.\.\.

12\}

Listing 3: Generated TopOp MLIR for a block\.
### 3\.2Lowering from TopOp to TpuOp

After the TopOp module is generated, the compiler converts it to TpuOp through the lowering pipeline\. During lowering, each TopOp is replaced by the corresponding TpuOp according to the target chip’s quantization mode, data type, and memory constraints, and chip\-related attributes are attached\. For example:

- •Aftertop\.MatMulis lowered totpu\.MatMul, weight quantization parameters and scaling factors are attached under W4A16 / INT8 quantization modes; for already\-quantized weights such as GPTQ / AWQ, the compiler can directly parse their compressed format and generate the corresponding TpuOp;
- •Aftertop\.FAttentionis lowered to the fused attention TpuOp of the corresponding chip, the backend generates the concrete TPU instruction sequence;
- •All TpuOps may also carrygroup\_info\(layer\-group information\), physical addresses, and operator splitting information, for use by subsequent layer\-group scheduling, memory allocation, and code generation\.

TPU\-MLIR supports direct\-through compilation of F32, BF16, F16, INT8 \(symmetric/asymmetric\), and quantized models such as AWQ / GPTQ / AutoRound\. For the INT8 mode, a calibration pass is inserted to determine the scale and zero\-point of each tensor using a small amount of sample data\.

After lowering is complete, the compiler continues to run standard optimization and code generation passes: layer\-group slicing, on\-chip/off\-chip memory allocation, and codegen, finally outputting a per\-module target binary\. Since a large language model cannot be processed as a single compilation unit, the frontend splits the model into multiple small TopOp modules—one module per Transformer layer variant, plus embedding, the language model head, and auxiliary heads—and compiles each module independently\. The final binary is obtained by merging all per\-module results\. This modular approach has two advantages: compilation can be parallelized across layers; and at runtime the functions can be invoked in the order required by autoregressive generation\.

The overall compilation pipeline from a trained checkpoint to a deployment binary is shown in Figure[3](https://arxiv.org/html/2607.15865#S3.F3)\.

![Refer to caption](https://arxiv.org/html/2607.15865v1/compile_flow.png)Figure 3:Overall compilation pipeline \(checkpoint→\\toTopOp→\\toTpuOp→\\tobinary\)\.The process of modular splitting, parallel compilation, and merging is shown in Figure[4](https://arxiv.org/html/2607.15865#S3.F4)\. The frontend splits the model into multiple small TopOp modules by layer and by stage; each module independently completes TopOp→\\toTpuOp→\\tocodegen, and the results are merged into the final deployment binary\.

![Refer to caption](https://arxiv.org/html/2607.15865v1/parallel.png)Figure 4:Modular splitting, parallel compilation, and merging\.

## 4Three\-Stage Splitting: Prefill, Prefill\_kv, and Decode

The second topic of this paper is the execution model\. Autoregressive LLM inference is not a single forward pass, but alternates between processing a \(possibly long\) prompt and generating tokens one by one\. Therefore, we split each Transformer layer into three TopOp module variants for static compilation: prefill, prefill\_kv, and decode, and then lower them to the corresponding TpuOp variants respectively\.

### 4\.1Why Split into Three Stages?

- •Prefill: processes all tokens in the input prompt in parallel\. The query length equals the prompt length, and a Key/Value tensor is generated for each token\.
- •Decode: processes only the latest generated token, but must attend to all previously computed Keys and Values\. The query length is 1, while the Key/Value length grows with the generation sequence\.
- •Prefill\_kv: used when the runtime wants to continue a conversation and needs to merge the historical Key/Value cache with a new prompt\. Its behavior is similar to prefill, but it first concatenates the historical cache before the new tokens\.

The main reason for splitting into three stages is that the tensor shapes and memory layouts of these three cases differ significantly; if compiled into a single function, a large amount of padding or dynamic recompilation would be forced\. Most specialized AI accelerators are parallel architectures oriented toward tensor operations and rely heavily on static compilation; dynamic compilation not only increases runtime compilation overhead, but also reduces the determinism of memory allocation and operator scheduling\. To ensure that as many parts as possible can be statically compiled, prefill and prefill\_kv are usually precompiled at a fixed maximum length \(e\.g\., 8K\)\. By generating independent TopOp modules, each stage can be statically optimized for its own tensor shapes, then lowered to a dedicated TpuOp implementation, with memory allocated statically and codegen more deterministic, thereby reducing dynamic overhead at runtime\.

Splitting into three stages can simultaneously support three typical scenarios:

1. 1\.When historical context is not considered, usePrefill\+Decode;
2. 2\.When historical context is supported, the first session usesPrefill\+Decode, and subsequent sessions usePrefill\_kv\+Decode;
3. 3\.When the prompt length is very large, e\.g\., prefill and prefill\_kv are both precompiled at 8K while the prompt reaches 80K, it can be completed in segments using 1 Prefill \+ 9 Prefill\_kv\.

The main differences among the three stages are summarized in Table[1](https://arxiv.org/html/2607.15865#S4.T1)\.

Table 1:Comparison of the prefill, prefill\_kv, and decode stages\. Lethh= hidden\_states,pp= position\_ids,h​k/vhk/v= history\_k/v; all three stages output\(h,k/v​cache\)\(h,k/v\\ \\text\{cache\}\)\.
### 4\.2Prefill Stage

Prefill and prefill\_kv are actually generated from the same parameterized TopOp template, distinguished only by awith\_historyflag: whenwith\_history=False, the historical KV inputs are empty tensors, yielding the prefill moduleblock\_\{i\}; whenwith\_history=True, the historical KV inputs carry the real cache, yielding the prefill\_kv moduleblock\_kv\_\{i\}\. The two share the same dataflow and lowering path, avoiding duplicate implementation\. This subsection first describes prefill, whose signature is:

Input:\[hidden\_states,position\_ids\]

Output:\[hidden\_states,key\_cache,value\_cache\]

wherehidden\_stateshas shape\[1, max\_input\_length, hidden\_size\], andhistory\_k/history\_vare empty tensors in pure prefill and carry the historical cache in prefill\_kv\. The Key/Value cache returned by this stage is then passed to the decode stage\.

### 4\.3Prefill\_kv Stage

The prefill\_kv moduleblock\_kv\_\{i\}is the instance of the above template withwith\_history=True; its TopOp dataflow is identical to prefill, and the only difference is that the inputhistory\_k/history\_vcarry the historical cache:

Input:\[hidden\_states,position\_ids,

history\_k,history\_v\]

Output:\[hidden\_states,key\_cache,value\_cache\]

Inside the module, the newly computed Key/Value tensors are concatenated with the historical cache along the sequence dimension viatop\.Concat\. The effective Key/Value length is thereforeseq\_length \+ max\_input\_length\. This stage is generated only when historical KV reuse is enabled, which is a common setting for long\-context conversation applications\.

### 4\.4Decode Stage

The decode moduleblock\_cache\_\{i\}processes one new token per forward call:

Input:\[hidden\_states,position\_ids,

history\_k,history\_v\]

Output:\[hidden\_states,key\_cache,value\_cache\]

Herehidden\_stateshas shape\[batch, 1, hidden\_size\], and the Key/Value of the new token are concatenated into the history\. Since the context length keeps growing during generation, the compiler may optionally generate multiple decode variants for different cache lengths \(i\.e\., a “decode chunk list”\), and at runtime select the variant whose static shape best matches the current context size\.

### 4\.5Runtime Orchestration

The three stages are not mutually exclusive, but are used in sequence\. A typical inference loop is shown in Figure[5](https://arxiv.org/html/2607.15865#S4.F5)\.

![Refer to caption](https://arxiv.org/html/2607.15865v1/runtime.png)Figure 5:Three\-stage runtime orchestration of autoregressive inference\.A textual description of the typical inference loop is as follows:

1. 1\.Run the embedding module on the prompt token ids\.
2. 2\.For each layer, run theprefill\(orprefill\_kvif history exists\) module to obtain the Key/Value cache and the hidden state at the end of the prompt\.
3. 3\.Run the language model head to obtain the logits of the last prompt token\.
4. 4\.Sample to obtain the next token\.
5. 5\.Run thedecodemodule with the ever\-growing Key/Value cache to obtain the hidden state of the new token\.
6. 6\.Repeat steps 3–5 until an end\-of\-sequence symbol is generated\.

Since each stage is an independently compiled function in the merged deployment binary, the runtime can freely combine them\. The static shapes inside the functions keep the generated hardware instructions simple, while the coarse\-grained split between stages captures the dynamic nature of autoregressive generation\.

### 4\.6Typical Performance Optimizations

Taking a specialized AI accelerator as an example, we present several common optimization ideas; the specific parameters need to be adjusted according to the target chip and model size\.

#### 4\.6\.1Causal Mask Support

By algorithmic logic, the attention of an 8K\-length input requires an8​K×8​K8K\\times 8Kcausal mask, and loading it directly wastes a lot of bandwidth\. A fixed small mask \(e\.g\.,128×128128\\times 128\) can be used; as long as the tile size of the full attention \(e\.g\.,64×6464\\times 64, determined by the number of accelerator cores and the on\-chip cache size\) is fixed, this small mask can be reused, significantly reducing mask bandwidth\. This is also one of the reasons why the prefill / prefill\_kv modules do not need to takeattention\_maskas an explicit input\.

#### 4\.6\.2Implementation of Positional Encoding

By algorithmic logic, positional encoding involves the cos/sin series operations required by RoPE, and such element\-wise operations are inefficient on specialized accelerators\. We can precompute the position vectors from0toseq\_lengthat compile time and store them as weights, and at runtime fetch the corresponding values viatop\.Gatheraccording toposition\_ids, thereby turning runtime computation into memory access\. This scheme trades a constant storage ofseq\_length×\\timeshead\_dim×\\times2floating\-point numbers for the elimination of element\-wise computation, and is usually a net gain in the decode stage where on\-chip compute is more likely to be the bottleneck than memory access\.

#### 4\.6\.3KV Cache Management

The compiler needs to specially plan the memory addresses of the KV Cache\. Whereas mainstream serving systems such as vLLM\[[3](https://arxiv.org/html/2607.15865#bib.bib5)\]manage the KV cache dynamically with paged attention, our static\-compilation setting requires the cache layout to be fixed at compile time\. In particular, when usingtop\.Concatto concatenate the KV of a new token into the historical cache, it should be implemented through address reuse or in\-place updates to avoid extra bandwidth consumption from physical copies\. In the decode implementation of LLM\-TPU, the correspondingblock\_cache\_\*variant is also selected according to the current context length, so that the address allocation and access pattern of the KV Cache remain statically predictable\.

With the above optimizations, a specialized AI accelerator can fully exploit compute in the prefill stage and fully utilize off\-chip bandwidth in the decode stage, thereby achieving high hardware utilization for overall inference\.

## 5Discussion

The two topics of this paper are tightly coupled\. TopOp provides a common expression language for the model, while the three\-stage splitting is the strategy that cuts this common representation into hardware\-specific TpuOp modules that can be executed efficiently at runtime\.

The following design decisions are worth emphasizing:

- •Framework independence\.TopOp modules contain no PyTorch or HuggingFace concepts, only tensor operations\. This allows the same TopOp→\\toTpuOp lowering pipeline to apply to dense decoder\-only Transformers, grouped\-query attention variants, mixture\-of\-experts variants, and vision\-language models\.
- •Modular compilation\.Splitting the model into TopOp modules by layer and by stage keeps each compilation unit small, supports parallel compilation, and allows the runtime to reuse compiled layers under multiple sequence lengths\.
- •Per\-stage static shapes\.Rather than compiling a fully dynamic Transformer, we compile several static\-shape variants and schedule among them at runtime\. This aligns with MLIR’s design philosophy of encoding transformation validity preconditions directly into the IR\.

Future work includes extending the three\-stage model to speculative decoding, supporting longer contexts through more aggressive cache tiling, and applying the same splitting method to multimodal language models whose vision tower output feeds into the same prefill/decode loop\.

## 6Conclusion

This paper presented an MLIR\-based compilation method for large language models, illustrated with TopOp and TpuOp\. A model is first imported into the framework\-agnostic TopOp, then lowered to the target\-hardware\-specific TpuOp, and compiled into deployment functions partitioned by stage\. To handle autoregressive inference, each Transformer layer is split into three static\-shape stages: prefill, prefill\_kv, and decode\. This design maintains the compiler’s generality across Transformer families while providing the runtime with dedicated functions for efficiently handling prompts and per\-token generation\.

The above method has been validated in a real compiler project\. Based on this method, we implemented LLM support in the reference project TPU\-MLIR333https://github\.com/sophgo/tpu\-mlir, and provided LLM\-TPU444https://github\.com/sophgo/LLM\-TPU, a demo project supporting a variety of large models, to demonstrate the complete flow from model import to deployment\.

## References

- \[1\]\(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.arXiv preprint arXiv:2205\.14135\.Note:https://arxiv\.org/abs/2205\.14135Cited by:[§2\.2](https://arxiv.org/html/2607.15865#S2.SS2.p4.1)\.
- \[2\]P\. Hu, M\. Lu, L\. Wang, and G\. Jiang\(2022\)TPU\-MLIR: a compiler for TPU using MLIR\.arXiv preprint arXiv:2210\.15016\.Note:https://arxiv\.org/abs/2210\.15016Cited by:[§3](https://arxiv.org/html/2607.15865#S3.p1.1)\.
- \[3\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Lian, Y\. Yu, J\. Gonzalez, H\. Zhang,et al\.\(2023\)vLLM: easy, fast, and cheap LLM serving with PagedAttention\.arXiv preprint arXiv:2309\.06180\.Note:https://arxiv\.org/abs/2309\.06180Cited by:[§4\.6\.3](https://arxiv.org/html/2607.15865#S4.SS6.SSS3.p1.1)\.
- \[4\]C\. Lattner, M\. Amini, U\. Bondhugula, A\. Cohen, A\. Davis, J\. Pienaar, R\. Riddle, T\. Shpeisman, N\. Vasilache, and O\. Zinenko\(2020\)MLIR: a compiler infrastructure for the end of moore’s law\.InarXiv preprint arXiv:2002\.11054,Note:http://arxiv\.org/abs/2002\.11054Cited by:[§1](https://arxiv.org/html/2607.15865#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.15865#S2.SS1.p1.1)\.
- \[5\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.arXiv preprint arXiv:1706\.03762\.Note:https://arxiv\.org/abs/1706\.03762Cited by:[§1](https://arxiv.org/html/2607.15865#S1.p1.1)\.
- \[6\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi,et al\.\(2020\)HuggingFace Transformers\.Note:https://huggingface\.co/docs/transformersCited by:[§1](https://arxiv.org/html/2607.15865#S1.p1.1)\.