LLM-Based Embeddings for Program Analysis and Optimization

# LLM-Based Embeddings for Program Analysis and Optimization
Source: [https://arxiv.org/html/2608.07894](https://arxiv.org/html/2608.07894)
###### Abstract

Recent advances have highlighted the potential of machine learning, particularly Large Language Models \(LLMs\), for analyzing and optimizing programs\. We present the first application of program embeddings from LLMCompiler—an LLM massively pretrained on intermediate representation \(IR\) code—to representative program analysis and optimization tasks\. We generate program embeddings directly from source and IR code using a simple approach: split programs into chunks, independently embed each chunk with pretrained LLMs, and then aggregate the chunk embeddings into a single program embedding\. Our experiments show that combining source and IR code embeddings achieves an error rate of 1\.54% in algorithm classification, a 12% improvement over the current state\-of\-the\-art, and a competitive accuracy on heterogeneous device mapping\. These findings suggest that training a performance\-aware LLM for embedding IR code might yield state\-of\-the\-art results in code optimization tasks\.

## IIntroduction

Software developers are producing code at an unprecedented rate\. In 2024, GitHub saw 5\.2 billion new contributions and reported widespread adoption of generative AI coding tools111https://octoverse\.github\.com\. Scaling both program analysis \(e\.g\. algorithm classification\) and optimization tasks \(e\.g\. heterogeneous device mapping\) accordingly requires automated tooling\.

One current machine learning approach constructs semantic program embeddings and applies them to downstream program analysis and optimization tasks\. Instead of embedding source or machine code directly, many methods remain language and architecture\-agnostic by embedding intermediate representation \(IR\) code\. This IR can be viewed as a virtual assembly language: a standardized, low\-level abstraction of program behavior that is independent of source language and machine architecture\[[14](https://arxiv.org/html/2608.07894#bib.bib95)\]\. An alternative approach, rather than operating on textual IR directly, constructs graph representations from IR code and then applies Graph Neural Networks \(GNNs\)\[[8](https://arxiv.org/html/2608.07894#bib.bib91),[17](https://arxiv.org/html/2608.07894#bib.bib90),[13](https://arxiv.org/html/2608.07894#bib.bib84)\]\.

Although there has been some work on text\-based IR models, such as OSCAR\[[19](https://arxiv.org/html/2608.07894#bib.bib17)\]graph\-based and multimodal methods have largely outperformed textual models\[[8](https://arxiv.org/html/2608.07894#bib.bib91),[17](https://arxiv.org/html/2608.07894#bib.bib90),[13](https://arxiv.org/html/2608.07894#bib.bib84)\]\. Graph\-structured inductive biases have proven highly effective for learning program embeddings\. However, until recently, no large\-scale text\-based IR model had employed massive pretraining like their source code counterparts\. In\[[7](https://arxiv.org/html/2608.07894#bib.bib83)\], Cummins et al\. introduced LLMCompiler, a 70\-billion parameter Large Language Model \(LLM\) pretrained on 546 billion tokens of LLVM IR and assembly code\. LLVM IR is an IR defined by the LLVM compiler infrastructure\[[14](https://arxiv.org/html/2608.07894#bib.bib95)\]\. To date, program embeddings from LLMCompiler have not been applied to standard downstream program analysis and optimization tasks\.

Inspired by LLMCompiler, we employ pretrained LLMs to generate program embeddings from source and IR code\. These two views complement each other: source code captures high\-level algorithmic intent, while IR code exposes low\-level data and control flow\. To embed programs, we first divide them into chunks, embed each chunk independently, and then aggregate them into a program embedding with a Long\-Short Term Memory \(LSTM\) network\.

The main contributions of this work are:

1. 1\.A lightweight embedding framework that feeds pretrained LLMs chunked source and IR code, while fine\-tuning only a LSTM head\.
2. 2\.The first empirical evaluation of LLMCompiler\-based embeddings on representative downstream tasks in program analysis and optimization\.
3. 3\.State\-of\-the\-art performance on algorithm classification and competitive accuracy on heterogeneous device mapping\.

These results indicate that massive pretraining can close the performance gap between graph\-based and text\-based models\. Performance\-aware pretraining of LLMCompiler and improved embedding methods remain an open path to further gains on program analysis and optimization tasks\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2608.07894#S2)provides an overview of related work in machine learning for program analysis and optimization, followed by a description of our methodology in Section[III](https://arxiv.org/html/2608.07894#S3)\. Sections[IV](https://arxiv.org/html/2608.07894#S4)and[V](https://arxiv.org/html/2608.07894#S5)present our experimental setup, results and discuss key findings, and finally, Section[VI](https://arxiv.org/html/2608.07894#S6)summarizes contributions and potential future directions\.

## IIRelated Work

Graph\-based representations are central to modern program analysis and optimization, particularly Control Flow Graphs \(CFGs\) and Data Flow Graphs \(DFGs\)\. CFGs model a program’s execution paths: each node is a basic block of instructions, and directed edges indicate possible control flow between blocks\. DFGs capture how values propagate through a program: nodes denote operations or value definitions, and directed edges connect every producer to the operations that consume its results\.

Although plain CFGs and DFGs are effective, they omit crucial semantics such as operand order, memory aliasing and interprocedural context, motivating a rich lineage of enhanced graph representations\. ConteXual Flow Graphs \(XFGs\), used by inst2vec, merge CFGs and DFGs into a single graph\[[1](https://arxiv.org/html/2608.07894#bib.bib43)\]\. Control Data Flow Graphs \(CDFGs\) extend XFGs with explicit memory dependency edges between load and store instructions\[[2](https://arxiv.org/html/2608.07894#bib.bib49)\]\. ProGraML unifies control, data and call flow while encoding operand positions and literal values\[[5](https://arxiv.org/html/2608.07894#bib.bib66)\]\. PerfoGraph extends ProGraML with aggregate data type information\[[13](https://arxiv.org/html/2608.07894#bib.bib84)\]\. Finally, FAIR constructs separate CFGs and DFGs augmented with call flow information\[[17](https://arxiv.org/html/2608.07894#bib.bib90)\]\.

Although these enhanced graphs improve downstream performance, they require careful feature engineering\. To avoid this, some methods operate directly on textual IR to automatically extract relevant features but without the inductive biases of graph representations they generally lag behind graph\-based models\. OSCAR, a hierarchical Transformer trained with textual IR, delivers competitive results but still falls short of the best graph\-based models\[[19](https://arxiv.org/html/2608.07894#bib.bib17)\]\. MIREncoder adopts a multimodal approach, incorporating textual IR as well as data, control and call flow graphs, achieving strong performance\[[9](https://arxiv.org/html/2608.07894#bib.bib93)\]\. Both models, however, are orders of magnitude smaller than today’s large\-scale source\-code LLMs, which limits their capacity to generalize\.

LLMCompiler narrows the scale gap with a 70\-billion parameter LLM pretrained on 546B LLVM IR and assembly tokens\[[7](https://arxiv.org/html/2608.07894#bib.bib83)\]\. Yet its embeddings have never been evaluated on representative program analysis or optimization tasks, so we still do not know whether massive pretraining on textual IR can rival graph\-based embedding methods\. This paper fills that gap: we extract program embeddings from LLMCompiler and deliver the first evaluation on algorithm classification and heterogeneous device mapping\.

## IIIMethods and Approach

This work presents a comparative analysis of program embeddings from pretrained LLMs and GNNs\. The evaluation covers two classification tasks: heterogeneous device mapping and algorithm classification\. This section describes the embedding generation methods\.

![Refer to caption](https://arxiv.org/html/2608.07894v1/x1.png)Figure 1:The program embedding procedure\. We use the same pretrained LLM to generate chunk embeddings for the source and LLVM IR code\. The chunk embeddings are aggregated by separate LSTMs and concatenated to produce a program\-level embedding\. Note that while the LSTMs are trainable, the LLM is not fine\-tuned\.### III\-AProgram Graphs

For GNN approach, programs are modeled as control flow graphs \(CFGs\)\. A CFG is a directed graphG=\(V,E\)G=\(V,E\), whereVVis the set of basic blocks \(nodes\) andE⊆V×VE\\subseteq V\\times Vdenotes control flow transitions between basic blocks \(edges\)\. Every node inVVcan be labeled with the following feature vectors:

#### Instruction counts

A histogram of the number of instructions in the basic block\.

#### Autophase features

A vector with feature counts for the following autophase features\[[12](https://arxiv.org/html/2608.07894#bib.bib64)\]:ArgsPhi, BBPhi, BeginPhi, BinaryConstArg, Const32Bit, Const64Bit, NumConstOnes, NumConstZeroes, TestUnary, TotalInsts, TotalMemInst\.

### III\-BGraph Neural Networks

We experiment with two GNN architectures that have shown state\-of\-the\-art performance on various tasks:

#### Graph Attention Network

The GAT\[[3](https://arxiv.org/html/2608.07894#bib.bib96)\]formulation incorporates learned attention mechanisms over node neighborhoods through the following message passing scheme:

hi′=σ​\(∑j∈𝒩iαi​j​W​hj\)h^\{\\prime\}\_\{i\}=\\sigma\\left\(\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}\}\\alpha\_\{ij\}Wh\_\{j\}\\right\)whereαi​j\\alpha\_\{ij\}are learned attention coefficients,WWare network parameters, and𝒩i\\mathcal\{N\}\_\{i\}is the neighborhood of nodeii\.

#### Principal Neighborhood Aggregation

PNA\[[4](https://arxiv.org/html/2608.07894#bib.bib97)\]extends traditional GNNs with continuous features and degree\-aware aggregators, enabling the network to adjust signals based on the degree of each node\. Empirical results support PNA’s effectiveness in capturing graph structures and outperforming other GNN architectures\.

### III\-CPretrained LLMs

Transfer learning leverages models pretrained on large datasets to extract features for downstream tasks\. We use the following pretrained LLMs to generate program embeddings\.

#### CodeBERT\[[10](https://arxiv.org/html/2608.07894#bib.bib99)\]

CodeBERT is an encoder\-only LLM that uses masked language modeling and replaced token detection to capture syntactic and semantic nuances\. Its dual pretraining on natural and programming languages enhances contextual understanding and performance in tasks like code summarization and search\.

#### CodeLlama\[[20](https://arxiv.org/html/2608.07894#bib.bib98)\]

Building on Llama2, CodeLlama is a decoder\-only LLM pretrained and fine\-tuned for tasks like code synthesis and analysis\.

#### LLMCompiler\[[7](https://arxiv.org/html/2608.07894#bib.bib83)\]

Building on CodeLlama, LLMCompiler is a decoder\-only LLM designed for code optimization\. LLMCompiler’s pretraining and fine\-tuning on 546B tokens of LLVM IR and assembly code, improves its understanding of compiler IRs, assembly language, and optimization techniques\.

#### Llama3

Llama3222https://ai\.meta\.com/blog/meta\-llama\-3/is a standard decoder\-only LLM with an efficient tokenizer, leading to improved model performance\. Grouped query attention is used to reduce inference cost\.

### III\-DGenerating Code Embeddings from LLMs

Given the challenges of processing large programs with LLMs, we adopt a chunk\-based approach, illustrated in Fig\.[1](https://arxiv.org/html/2608.07894#S3.F1)to generate the embeddings\. First we tokenize the input program using the LLM’s default tokenizer\. As programs may exceed the LLM’s context window \(the maximum number of tokens the LLM can process at once\), we divide input programs into fixed\-size, non\-overlapping chunks\. Although overlapping chunks offer additional context, we opt for non\-overlapping chunks to reduce the number of forward passes through the LLM and improve computational efficiency\.

Each chunk of tokens is processed independently by the LLM to generate token\-level embeddings\. The embedding for each token is extracted from the layer just before the LLM head\. To obtain a chunk\-level embedding, we either use the last token embedding in the chunk or compute the mean of all token embeddings in the chunk\. These chunk\-level embeddings are then aggregated into a single program\-level embedding using a long\-short\-term memory \(LSTM\) network\. We choose an LSTM over simple aggregation strategies, such as mean aggregation, to capture dependencies between chunks, particularly when important relationships in the code may span multiple chunks\.

For our experiments, we compute program embeddings from both the source and LLVM IR code\. The source and LLVM IR code are processed independently by two separate LSTMs\. This provides an embedding of both the high\-level and low\-level program representations\. Having both embeddings offers several advantages: the high\-level source code captures the program’s original intent, structure, and logic, making it easier to understand the program’s functionality, while the low\-level LLVM IR offers a more detailed view of how the code will be executed on the hardware, including optimizations and instruction\-level nuances\. Combining these two perspectives ensures that the model benefits from both abstract and concrete representations of the program\. For downstream classification tasks, both program\-level embeddings are concatenated with any additional global program features and passed to a classifier head, a multi\-layer perceptron \(MLP\), to produce the final classification\.

## IVHeterogeneous Device Mapping

This section evaluates the proposed models on heterogeneous device mapping, a representative program optimization task, comparing GNNs, LLMs, and state\-of\-the\-art approaches\.

### IV\-ADataset

Heterogeneous device mapping involves classifying whether an OpenCL kernel runs faster on a CPU or a GPU\. The dataset contains 680 labeled examples derived from 256 distinct kernels across seven benchmark suites \(AMD SDK, NPB, NVIDIA SDK, Parboil, Polybench, Rodinia, SHOC\)\[[11](https://arxiv.org/html/2608.07894#bib.bib100)\]\. Some kernels appear multiple times with varied runtime parameters, including data transfer size, workgroup size \(threads per workgroup\), and chosen dataset \(e\.g\. sorted or unsorted\), as optimal device placement depends on both the code and runtime settings\. Labels are provided for AMD GPU \(Tahiti 7970\) vs\. Intel CPU \(i7\-3820\) and NVIDIA GPU \(GTX 970\) vs\. Intel CPU \(i7\-3820\) performance\.

### IV\-BGNN Preprocessing

CFGs are built from the LLVM IR code with instruction count and autophase node features\. Global features include transfer size, workgroup size, their log\-transformed values \(log⁡\(x\+1\)\\log\(x\+1\)\) and a one\-hot encoding of the chosen dataset\. Global features with zero variance are removed, and the remaining features are standardized to have zero mean and unit variance\.

### IV\-CLLM Preprocessing

The OpenCL source code is preprocessed as described in\[[6](https://arxiv.org/html/2608.07894#bib.bib33)\], while only the instructions and labels from the LLVM IR code are retained\. Both are tokenized and chunked\. For decoder\-only LLMs, the first chunk begins with a beginning\-of\-sequence token \(BOS\) and the last chunk ends with an end\-of\-sequence token \(EOS\)\. For encoder\-only LLMs, each chunk begins with a classification token \(CLS\) and ends with an EOS token\. The last chunk is always padded with EOS tokens which are ignored during token embedding aggregation\. Global features are preprocessed as described in Section[IV\-B](https://arxiv.org/html/2608.07894#S4.SS2)and passed to the classifier head\.

### IV\-DStratified Nested Cross Validation

Instead of the standard 10\-fold stratified cross\-validation \(CV\), we use 5\-fold outer and 10\-fold inner stratified nested cross\-validation \(stratified NCV\) to reduce variance\. In the inner sweep, models are trained with early stopping \(patience 10 epochs, minimum delta 0\.0001\) for up to 250 epochs\. The inner hyperparameter combination with maximum validation Matthews correlation coefficient \(MCC\) is retrained for a fixed number of epochs on the combined training and validation sets\. The number of training epochs is scaled according to the ratio of the combined dataset size \(training \+ validation\) to the original training set size\. The retrained model is evaluated on the test set, and its predictions are saved\. Test metrics for outer hyperparameter combinations are calculated by aggregating these predictions from each inner fold, with the best outer hyperparameter combination selected according to maximum test accuracy\.

Models are trained with the AdamW optimizer \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,λ=0\.01\\lambda=0\.01\), a batch size of 256, and cosine learning rate warmup for 10 steps\. The classifier head is a multi\-layer perceptron \(MLP\) with two hidden layers: the first is twice the input size and the second matches the input size\. Batch normalization and ReLU activation are applied between layers, except for the last\. Other hyperparameters are selected via hyperparameter sweeps\.

#### GNN Hyperparameters

We evaluate the GAT and PNA architectures\. The outer sweep tunes data transfer and workgroup size features \(enabled/disabled\), chosen dataset feature \(enabled/disabled\), instruction counts \(enabled/disabled\), autophase features \(enabled/disabled\), and GNN aggregator \(Max/Min/MLP\)\. The inner sweep tunes learning rate \(0\.005/0\.0005\), dropout rate \(0\.0/0\.25\), GNN hidden size \(32/64\), and GNN layers \(1/2/4\)\.

#### LLM Hyperparameters

We evaluate six decoder\-only models \(CodeLlama\-7B, CodeLlama\-70B, Llama3\-8B, Llama3\-70b, LLM\-Compiler\-7B, LLM\-Compiler\-13B\) and one encoder\-only model \(CodeBERT\-125M\)\. For decoder\-only models, the outer sweep tunes chunk size \(4096/16384\) and aggregator \(Last/Mean\)\. For encoder\-only models, it tunes chunk size \(512\) and aggregator \(First/Last/Mean\)\. The inner sweep tunes learning rate \(0\.005/0\.0005\), dropout rate \(0\.5\), LSTM hidden size \(32/64\), LSTM layers \(1/2\), and LSTM bidirectional \(enabled/disabled\)\.

#### Results

Examining Table[I](https://arxiv.org/html/2608.07894#S4.T1), we observe that the decoder\-only LLMs outperform the GNN baselines for both the AMD and NVIDIA labels\. Surprisingly, LLMCompiler, the only model pretrained on both source and LLVM IR code, achieves similar accuracy to the generic Llama variants\.

A likely reason lies in the mismatch between LLMCompiler’s pretraining tasks and the demands of heterogeneous device mapping\. LLMCompiler is initialized from CodeLlama then finetuned with autoregressive language modeling on a large corpus of LLVM IR and assembly\. LLMCompiler is also trained to emulate LLVM passes: given unoptimized program, starting code size and an optimization flag list, output the optimized program and code size\. The sole explicit performance cue is code size\. Critical hardware\-dependent factors such as latency, bandwidth and instruction throughput never appear in the loss\. As a result, LLMCompiler learns syntactic and optimization patterns but might not fully capture code runtime in its embeddings\.

TABLE I:Performance with stratified NCV\. Numbers in parentheses are the weighted standard deviation of the per\-fold accuracies\.ModelAMD Acc\.NVIDIA Acc\.PNA88\.38 \(2\.48\)87\.65 \(1\.50\)GAT88\.68 \(1\.51\)86\.32 \(0\.59\)CodeLlama\-7B92\.06 \(1\.99\)88\.38 \(2\.44\)CodeLlama\-70B91\.03 \(2\.39\)85\.74 \(2\.40\)Llama3\-8B91\.18 \(2\.28\)86\.76 \(2\.67\)Llama3\-70B91\.32 \(1\.94\)85\.00 \(2\.31\)LLMCompiler\-7B91\.18 \(2\.98\)87\.50 \(1\.32\)LLMCompiler\-13B91\.03 \(1\.88\)86\.62 \(1\.89\)CodeBERT87\.50 \(3\.05\)82\.21 \(4\.09\)

### IV\-ELeave\-One\-Group\-Out Nested Cross Validation

Stratified CV presents challenges for this dataset, as identical source code may appear in both the training and test sets because kernels are repeated with varied runtime parameters\. Additionally, kernels from the same benchmark suite, which might be similar due to co\-evolution, can also appear in both sets\. This risks test set leakage, potentially inflating results from prior literature using stratified CV\.

To illustrate this, we reinterpret our 5\-fold outer, 10\-fold inner stratified NCV results as five different 10\-fold stratified CV runs\. We take the validation accuracy of the best model as the “test” accuracy\. Histograms of the test accuracies for AMD and NVIDIA labels are shown in Fig\.[2](https://arxiv.org/html/2608.07894#S4.F2)and Fig\.[3](https://arxiv.org/html/2608.07894#S4.F3)\.

65657070757580808585909095950100100200200300300AMD Test AccuracyFrequencyFigure 2:Accuracies across five different 10\-fold stratified CV splits with the AMD labels\.606065657070757580808585909005050100100150150200200250250NVIDIA Test AccuracyFrequencyFigure 3:Accuracies across five different 10\-fold stratified CV splits with the NVIDIA labels\.Notably, when compared to the results from stratified NCV, the maximum AMD test accuracy increased from 92\.06% to 95\.04%, and the maximum NVIDIA test accuracy rose from 88\.38% to 92\.28%\. By considering five random splits, our 10\-fold stratified CV results meet the state\-of\-the\-art, underscoring the impact of test\-set leakage and the need for grouped splits\.

To mitigate these issues, we use leave\-one\-group\-out nested cross\-validation \(LOGO NCV\)\. With seven groups, this results in seven outer folds and six inner folds\. As the group sizes are highly imbalanced, we also report the weighted standard deviation of the per\-fold accuracies\. The training and model selection procedures are otherwise consistent with Section[IV\-D](https://arxiv.org/html/2608.07894#S4.SS4)\.

#### Results

In Table[II](https://arxiv.org/html/2608.07894#S4.T2), we observe that the LLMs are competitive with the GNN baselines on AMD labels, although the GNNs outperform the LLMs on NVIDIA labels\. While this could be due to the high variance caused by imbalanced folds, its likely caused by overfitting, as discussed in Section[IV\-G](https://arxiv.org/html/2608.07894#S4.SS7)\.

TABLE II:Performance with LOGO NCV\. Numbers in parentheses are the weighted standard deviation of the per\-fold accuracies\.ModelAMD Acc\.NVIDIA Acc\.PNA72\.65 \(5\.87\)60\.59 \(17\.52\)GAT75\.44 \(4\.68\)56\.76 \(8\.80\)CodeLlama\-7B75\.29 \(7\.79\)46\.62 \(6\.94\)CodeLlama\-70B71\.47 \(7\.00\)44\.12 \(5\.01\)Llama3\-8B74\.85 \(5\.14\)50\.88 \(5\.57\)Llama3\-70B74\.12 \(4\.82\)45\.88 \(8\.33\)LLMCompiler\-7B76\.03 \(5\.15\)46\.76 \(12\.53\)LLMCompiler\-13B72\.65 \(7\.65\)50\.74 \(14\.66\)CodeBERT73\.24 \(8\.68\)43\.82 \(10\.39\)

### IV\-FState\-of\-the\-Art Approaches

To compare our approach with the state\-of\-the\-art, we select the LLM and GNN models that achieved the highest test accuracy in stratified NCV\. As shown in Table[III](https://arxiv.org/html/2608.07894#S4.T3), while our LLM method is competitive, it doesn’t achieve the state\-of\-the\-art\. However, it’s important to note that the other methods report results using 10\-fold CV or 10\-fold stratified CV whereas we employ a more robust 5\-fold outer, 10\-fold inner stratified NCV\. This approach reduces the risk of performance being skewed by a favorable data split\.

TABLE III:Comparison with other methods\.ModelAMD Acc\.NVIDIA Acc\.CV MethodGNN \(ours\)88\.6887\.65Stratified NCVLLM \(ours\)92\.0688\.38Stratified NCVLLM \(ours\)95\.0492\.28Stratified CVinst2vec\-imm\[[1](https://arxiv.org/html/2608.07894#bib.bib43)\]88\.0986\.62Stratified CVOSCAR\[[19](https://arxiv.org/html/2608.07894#bib.bib17)\]88\.889\.7Stratified CVProGraML\[[5](https://arxiv.org/html/2608.07894#bib.bib66)\]86\.680\.0CVPerfoGraph\[[13](https://arxiv.org/html/2608.07894#bib.bib84)\]9490CVFAIR\[[17](https://arxiv.org/html/2608.07894#bib.bib90)\]96\.5291\.61CVMIREncoder\[[9](https://arxiv.org/html/2608.07894#bib.bib93)\]93\.693\.7Stratified CVAdditionally, the potential for test\-set leakage is significant without a grouped split, raising concerns about the reliability of stratified CV results\. In Section[IV\-E](https://arxiv.org/html/2608.07894#S4.SS5), we demonstrate this effect and show that our method meets the state\-of\-the\-art simply by trying five different 10\-fold stratified CV splits\. Moreover, the high variance of stratified split results is already reflected in the existing literature\. For example, the authors of\[[17](https://arxiv.org/html/2608.07894#bib.bib90)\]report that ProGraML achieves 92\.60% AMD accuracy and 88\.13% NVIDIA accuracy on their split while the original ProGraML paper reports only 86\.6% AMD accuracy and 80\.0% NVIDIA accuracy on a different split\[[5](https://arxiv.org/html/2608.07894#bib.bib66)\]\.

Unfortunately only\[[2](https://arxiv.org/html/2608.07894#bib.bib49)\]and\[[18](https://arxiv.org/html/2608.07894#bib.bib75)\]have considered a grouped split\. The authors of\[[2](https://arxiv.org/html/2608.07894#bib.bib49)\]also evaluate several existing but older methods shown in Table[IV](https://arxiv.org/html/2608.07894#S4.T4)\. Of the results available in the literature, ours are the best, although we suspect modern methods would achieve similar results\.

TABLE IV:Comparison with other methods\. Results marked with∗\*are from\[[2](https://arxiv.org/html/2608.07894#bib.bib49)\]and not the original paper\.ModelAMD Acc\.NVIDIA Acc\.CV MethodGNN \(ours\)75\.4460\.59LOGO NCVLLM \(ours\)76\.0350\.88LOGO NCVGrewe et al\.\[[11](https://arxiv.org/html/2608.07894#bib.bib100)\]56∗56^\{\*\}38∗38^\{\*\}LOGO CVDeepTune\[[6](https://arxiv.org/html/2608.07894#bib.bib33)\]48∗48^\{\*\}48∗48^\{\*\}LOGO CVinst2vec\[[1](https://arxiv.org/html/2608.07894#bib.bib43)\]49∗49^\{\*\}41∗41^\{\*\}LOGO CVGNN\-CDFG\[[2](https://arxiv.org/html/2608.07894#bib.bib49)\]5146LOGO CVGNN\-AST\[[2](https://arxiv.org/html/2608.07894#bib.bib49)\]6458LOGO CVCNN\[[18](https://arxiv.org/html/2608.07894#bib.bib75)\]49\.244\.3LOGO CVGiven this, it’s uncertain which method truly outperforms the others, as the potential for bias in the reported results makes direct comparison difficult\. However, the results in Table[III](https://arxiv.org/html/2608.07894#S4.T3)and Table[IV](https://arxiv.org/html/2608.07894#S4.T4)show that our method is at least competitive with the state\-of\-the\-art\.

TABLE V:Input features ablation with LOGO NCV
### IV\-GAblation

After removing the source code, accuracy decreases across all models, indicating that it contains relevant information\. Likewise, after removing the LLVM IR code, accuracy decreases for most models, although a few become slightly more accurate\. In general, the results in Table[V](https://arxiv.org/html/2608.07894#S4.T5), indicate that including both the source and LLVM IR code improves performance\. That is, the high\-level intent captured by the source code and the low level details captured by the LLVM IR code are both relevant to heterogeneous device mapping\.

We also tried replacing the LSTMs with unweighted mean aggregation\. As shown in Table[VI](https://arxiv.org/html/2608.07894#S4.T6), accuracy decreased for AMD labels but increased for NVIDIA labels\. Notably, the LLMs achieve NVIDIA accuracy comparable to PNA, the best GNN model\. This suggests that the LSTMs might be overfitting the NVIDIA labels due to limited training data in some imbalanced folds\.

TABLE VI:LSTM ablation with LOGO NCV

## VAlgorithm Classification

This section evaluates the proposed models on algorithm classification, a representative program analysis task, comparing GNNs, LLMs, and state\-of\-the\-art approaches\.

### V\-ADataset

We use the POJ\-104 dataset\[[16](https://arxiv.org/html/2608.07894#bib.bib101)\]for a classification task, where a given program is classified as a solution to one of 104 programming problems\. Each problem has 500 corresponding C/C\+\+ solutions, for a total of 52,000 programs\. To generate LLVM IR code from the programs, we add include directives, replacevoid mainwithint main, and compile usingclang14 \-O3as C, retrying as C\+\+ if needed\.

### V\-BGNN Preprocessing

CFGs are built from the LLVM IR with instruction count and autophase features as in Section[IV\-B](https://arxiv.org/html/2608.07894#S4.SS2), but no graph\-level features are included\.

### V\-CLLM Preprocessing

The original source code is used without preprocessing\. The first five lines of the LLVM IR are removed along with string literals containing the filename to prevent label leakage\. Embeddings are generated as in Section[IV\-C](https://arxiv.org/html/2608.07894#S4.SS3)but no global features are added\.

### V\-DTrain\-Validation\-Test Split

We use the train\-validation\-test split introduced in\[[1](https://arxiv.org/html/2608.07894#bib.bib43)\]for consistency with prior work\. After excluding programs that do not compile, there are 26,911 training, 8,921 validation, and 9,008 test examples\.

Each hyperparameter combination is trained using early stopping \(patience 10 epochs, minimum delta 0\.0001\) for up to 250 epochs on the training set\. The model with maximum validation MCC is retrained for a fixed number of epochs on the combined training and validation sets\. The number of training epochs is scaled according to the ratio of the combined dataset size \(training \+ validation\) to the original training set size\. The model is evaluated on the test set, and accuracy is reported\. The training procedure is otherwise consistent with Section[IV\-D](https://arxiv.org/html/2608.07894#S4.SS4)\.

#### GNN Hyperparameters

We evaluate the same GNN models as in Section[IV\-D](https://arxiv.org/html/2608.07894#S4.SS4.SSS0.Px1)using an identical hyperparameter grid, except for the dropout rate \(0\.5\) and GNN hidden size \(128/512\)\.

#### LLM Hyperparameters

We evaluate the same LLM models as in Section[IV\-D](https://arxiv.org/html/2608.07894#S4.SS4.SSS0.Px2)using an identical hyperparameter grid, except for the LSTM hidden size \(128/512\)\.

#### Results

LLM\-based models consistently outperform GNN\-based models in algorithm classification, as shown in Table[VII](https://arxiv.org/html/2608.07894#S5.T7)\. Unlike in heterogeneous device mapping, where decoder\-only LLMs performed similarly, LLMCompiler surpasses the other LLMs in algorithm classification\.

One possible explanation is that LLMCompiler’s pretraining on a compiler emulation task, which involves translating one LLVM IR program into an equivalent LLVM IR program, helps it generalize better to algorithm classification\. Since this task requires grouping programs that compute the same output, LLMCompiler’s ability to learn structural similarities between functionally equivalent programs makes it more adept at recognizing algorithmic equivalence\.

In contrast, the LLMCompiler models do not outperform on heterogeneous device mapping, where programs within a class are not necessarily functionally equivalent\. In this context, functionally equivalent programs might be categorized into different classes based on performance differences, such as one being faster on the CPU while another excels on the GPU\. This observation further suggests that LLMCompiler’s pretraining is particularly effective for algorithm classification, where recognizing functional equivalence is crucial\. It also implies that an LLM trained with specialized code optimization objectives could generate more effective embeddings\.

TABLE VII:Performance on algorithm classification

### V\-EState\-of\-the\-Art Approaches

To compare to the state\-of\-the\-art, we select the LLM and GNN models that achieved the best test error rates\. The top\-performing LLM model, LLMCompiler\-13B, achieves a new state\-of\-the\-art error rate of 1\.54%, as shown in Table[VIII](https://arxiv.org/html/2608.07894#S5.T8)\.

TABLE VIII:Comparison with other methods
### V\-FAblation

Removing the source code degrades all models, especially those not trained on LLVM IR code, while LLMCompiler is less affected\. Likewise, removing LLVM IR code harms most models, with LLMCompiler impacted the most\. Table[IX](https://arxiv.org/html/2608.07894#S5.T9)shows the error rates for this ablation experiment\.

TABLE IX:Input features ablationThese results suggest that incorporating both source and LLVM IR code is crucial for optimal performance\. LLMCompiler\-13B, trained on both, performs best, highlighting their complementary roles: source code conveys high\-level algorithmic intent, while LLVM IR code captures low\-level structural patterns\. This underscores the benefit of training LLMs on both code representations\.

We also experimented with replacing LSTM aggregation with unweighted mean aggregation, which led to a significant increase in error rates across all models, as shown in Table[X](https://arxiv.org/html/2608.07894#S5.T10)\. This suggests modeling long range dependencies across chunks is required for optimal results\.

TABLE X:LSTM ablation

## VIConclusion and Future Work

In this work, we proposed a simple yet effective approach for embedding both source code and LLVM IR using pretrained LLMs\. To address the constraints of fixed LLM context windows, we split programs into non\-overlapping chunks, independently embedded each chunk, and aggregated the resulting embeddings into a single program embedding using an LSTM network\. Despite its simplicity, our method achieved an error rate of 1\.54% in algorithm classification, or a 12% improvement over the state\-of\-the\-art, as well as competitive results in heterogeneous device mapping\. Remarkably, this was accomplished without engineered graph representations, highlighting the potential of LLMCompiler\-based embeddings for code analysis and optimization\.

Our findings suggest several promising directions for future research\. First, our ablation studies showed that combining source and LLVM IR code embeddings can yield better performance compared to using either embedding alone\. Moreover, LLMCompiler performed exceptionally well on algorithm classification, an analysis task aligned with its source\-to\-source pretraining objectives, but did not outperform on heterogeneous device mapping, an optimization task\. This suggests training or fine\-tuning a specialized LLM with performance\-aware objectives for embedding LLVM IR code\. We plan to train such an LLM on a large corpus of LLVM IR code in future work\.

Second, the simplicity of our approach affords numerous opportunities for enhancing embedding quality\. Future research could explore alternative methods for constructing chunk embeddings, such as those discussed in\[[15](https://arxiv.org/html/2608.07894#bib.bib104)\], or investigate replacing LSTM\-based chunk aggregation with other context extending mechanisms like RoPE\[[21](https://arxiv.org/html/2608.07894#bib.bib103)\]\. Additional preprocessing, such as cleaning the source and LLVM IR code, trying multiple compiler flags or even including explicit prompts, could yield further performance gains\. In short, our method is poised to benefit from ongoing advancements in techniques for extracting embeddings from LLMs\.

Finally, while our current approach is computationally expensive, its cost could be significantly reduced through methods like pruning, quantization, or knowledge distillation\. These optimizations would make our method more practical for real\-world applications\. The cost\-effectiveness of our approach will continue to scale with reductions in LLM inference costs\.

## Acknowledgment

This work was partially funded by an unrestricted gift from Google\. We also thank Edniel Campos for his work on data preprocessing and hyperparameter sweeps\.

ChatGPT333https://openai\.com/chatgptwas used for editing assistance throughout all sections of this paper\.

## References

- \[1\]T\. Ben\-Nun, A\. S\. Jakobovits, and T\. HoeflerCited by:[§II](https://arxiv.org/html/2608.07894#S2.p2.1),[TABLE III](https://arxiv.org/html/2608.07894#S4.T3.4.5.5.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.8.6.3),[§V\-D](https://arxiv.org/html/2608.07894#S5.SS4.p1.1),[TABLE VIII](https://arxiv.org/html/2608.07894#S5.T8.4.4.3.1)\.
- \[2\]A\. Brauckmann, A\. Goens, S\. Ertel, and J\. CastrillonCited by:[§II](https://arxiv.org/html/2608.07894#S2.p2.1),[§IV\-F](https://arxiv.org/html/2608.07894#S4.SS6.p3.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.2.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.8.10.4.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.8.11.5.1)\.
- \[3\]S\. Brody, U\. Alon, and E\. Yahav\(2022\)How attentive are graph attention networks?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§III\-B](https://arxiv.org/html/2608.07894#S3.SS2.SSS0.Px1.p1.5)\.
- \[4\]G\. Corso, L\. Cavalleri, D\. Beaini, P\. Liò, and P\. Veličković\(2020\)Principal neighbourhood aggregation for graph nets\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§III\-B](https://arxiv.org/html/2608.07894#S3.SS2.SSS0.Px2.p1.1)\.
- \[5\]C\. Cummins, Z\. V\. Fisches, T\. Ben\-Nun, T\. Hoefler, M\. F\. P\. O’Boyle, and H\. LeatherCited by:[§II](https://arxiv.org/html/2608.07894#S2.p2.1),[§IV\-F](https://arxiv.org/html/2608.07894#S4.SS6.p2.1),[TABLE III](https://arxiv.org/html/2608.07894#S4.T3.4.7.7.1),[TABLE VIII](https://arxiv.org/html/2608.07894#S5.T8.4.6.5.1)\.
- \[6\]C\. Cummins, P\. Petoumenos, Z\. Wang, and H\. LeatherCited by:[§IV\-C](https://arxiv.org/html/2608.07894#S4.SS3.p1.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.6.4.3)\.
- \[7\]C\. Cummins, V\. Seeker, D\. Grubisic, B\. Roziere, J\. Gehring, G\. Synnaeve, and H\. Leather\(2025\)LLM compiler: foundation language models for compiler optimization\.Cited by:[§I](https://arxiv.org/html/2608.07894#S1.p3.1),[§II](https://arxiv.org/html/2608.07894#S2.p4.1),[§III\-C](https://arxiv.org/html/2608.07894#S3.SS3.SSS0.Px3)\.
- \[8\]A\. Dutta, J\. Alcaraz, A\. Tehranijamsaz, E\. Cesar, A\. Sikora, and A\. JannesariCited by:[§I](https://arxiv.org/html/2608.07894#S1.p2.1),[§I](https://arxiv.org/html/2608.07894#S1.p3.1)\.
- \[9\]A\. Dutta and A\. Jannesari\(2024\)MIREncoder: multi\-modal ir\-based pretrained embeddings for performance optimizations\.InParallel Architectures and Compilation Techniques \(PACT\),Cited by:[§II](https://arxiv.org/html/2608.07894#S2.p3.1),[TABLE III](https://arxiv.org/html/2608.07894#S4.T3.4.10.10.1)\.
- \[10\]Z\. Feng, D\. Guo, D\. Tang, N\. Duan, X\. Feng, M\. Gong, L\. Shou, B\. Qin, T\. Liu, D\. Jiang, and M\. Zhou\(2020\)CodeBERT: a pre\-trained model for programming and natural languages\.Cited by:[§III\-C](https://arxiv.org/html/2608.07894#S3.SS3.SSS0.Px1)\.
- \[11\]D\. Grewe, Z\. Wang, and M\. F\. P\. O’Boyle\(2013\)Portable mapping of data parallel programs to opencl for heterogeneous systems\.InInternational Symposium on Code Generation and Optimization \(CGO\),Cited by:[§IV\-A](https://arxiv.org/html/2608.07894#S4.SS1.p1.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.4.2.3)\.
- \[12\]A\. Haj\-Ali, Q\. \(\. Huang, J\. Xiang, W\. Moses, K\. Asanovic, J\. Wawrzynek, and I\. StoicaCited by:[§III\-A](https://arxiv.org/html/2608.07894#S3.SS1.SSS0.Px2.p1.1)\.
- \[13\]A\. T\. Jamsaz, Q\. I\. Mahmud, L\. Chen, N\. K\. Ahmed, and A\. Jannesari\(2024\)PERFOGRAPH: a numerical aware program graph representation for performance optimization and program analysis\.Cited by:[§I](https://arxiv.org/html/2608.07894#S1.p2.1),[§I](https://arxiv.org/html/2608.07894#S1.p3.1),[§II](https://arxiv.org/html/2608.07894#S2.p2.1),[TABLE III](https://arxiv.org/html/2608.07894#S4.T3.4.8.8.1),[TABLE VIII](https://arxiv.org/html/2608.07894#S5.T8.4.8.7.1)\.
- \[14\]C\. Lattner and V\. AdveLLVM: a compilation framework for lifelong program analysis & transformation\.InInternational Symposium on Code Generation and Optimization \(CGO\),Cited by:[§I](https://arxiv.org/html/2608.07894#S1.p2.1),[§I](https://arxiv.org/html/2608.07894#S1.p3.1)\.
- \[15\]C\. Lee, R\. Roy, M\. Xu, J\. Raiman, M\. Shoeybi, B\. Catanzaro, and W\. Ping\(2025\)NV\-embed: improved techniques for training LLMs as generalist embedding models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§VI](https://arxiv.org/html/2608.07894#S6.p3.1)\.
- \[16\]L\. Mou, G\. Li, L\. Zhang, T\. Wang, and Z\. Jin\(2016\)Convolutional neural networks over tree structures for programming language processing\.InAAAI Conference on Artificial Intelligence \(AAAI\),Cited by:[§V\-A](https://arxiv.org/html/2608.07894#S5.SS1.p1.1)\.
- \[17\]C\. Niu, C\. Li, V\. Ng, D\. Lo, and B\. LuoCited by:[§I](https://arxiv.org/html/2608.07894#S1.p2.1),[§I](https://arxiv.org/html/2608.07894#S1.p3.1),[§II](https://arxiv.org/html/2608.07894#S2.p2.1),[§IV\-F](https://arxiv.org/html/2608.07894#S4.SS6.p2.1),[TABLE III](https://arxiv.org/html/2608.07894#S4.T3.4.9.9.1),[TABLE VIII](https://arxiv.org/html/2608.07894#S5.T8.4.7.6.1)\.
- \[18\]E\. Parisi, F\. Barchi, A\. Bartolini, and A\. AcquavivaCited by:[§IV\-F](https://arxiv.org/html/2608.07894#S4.SS6.p3.1),[TABLE IV](https://arxiv.org/html/2608.07894#S4.T4.8.12.6.1)\.
- \[19\]D\. Peng, S\. Zheng, Y\. Li, G\. Ke, D\. He, and T\. Liu\(2021\)How could neural networks understand programs?\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§I](https://arxiv.org/html/2608.07894#S1.p3.1),[§II](https://arxiv.org/html/2608.07894#S2.p3.1),[TABLE III](https://arxiv.org/html/2608.07894#S4.T3.4.6.6.1),[TABLE VIII](https://arxiv.org/html/2608.07894#S5.T8.4.5.4.1)\.
- \[20\]B\. Rozière, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez, J\. Rapin, A\. Kozhevnikov, I\. Evtimov, J\. Bitton, M\. Bhatt, C\. C\. Ferrer, A\. Grattafiori, W\. Xiong, A\. Défossez, J\. Copet, F\. Azhar, H\. Touvron, L\. Martin, N\. Usunier, T\. Scialom, and G\. Synnaeve\(2024\)Code llama: open foundation models for code\.External Links:2308\.12950Cited by:[§III\-C](https://arxiv.org/html/2608.07894#S3.SS3.SSS0.Px2)\.
- \[21\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\(2023\)RoFormer: enhanced transformer with rotary position embedding\.InNeurocomputing,Cited by:[§VI](https://arxiv.org/html/2608.07894#S6.p3.1)\.