LLM-Powered Predictive Decision-Making for Sustainable Data Center Operations

# LLM-Powered Predictive Decision-Making for Sustainable Data Center Operations
Source: [https://arxiv.org/html/2608.18503](https://arxiv.org/html/2608.18503)
Hanzhao Wang\*1Jingxuan Wu\*2Yumeng Li3Yu Pan1Guanting Chen†2 1![[Uncaptioned image]](https://arxiv.org/html/2608.18503v1/logo/usyd.png)The University of Sydney Business School, The University of Sydney 2![[Uncaptioned image]](https://arxiv.org/html/2608.18503v1/logo/unc.png)Department of Statistics and Operations Research, University of North Carolina at Chapel Hill 3![[Uncaptioned image]](https://arxiv.org/html/2608.18503v1/logo/Purdue.png)Department of Industrial Engineering, Purdue University \*Equal contribution†Corresponding authorguanting@unc\.edu

###### Abstract

The growing demand for AI\-driven workloads, particularly from Large Language Models \(LLMs\), has raised concerns about the significant energy and resource consumption in data centers\. This work introduces a novel LLM\-based predictive scheduling system designed to enhance operational efficiency while reducing the environmental impact of data centers\. Our system utilizes an LLM to predict key metrics such as execution time and energy consumption from source code, and it has the potential to extend to other sustainability\-focused metrics like water usage for cooling and carbon emissions, provided the data center can track such data\. The predictive model is followed by a real\-time scheduling algorithm that allocates GPU resources, aiming to improve sustainability by optimizing both energy consumption and queuing delays\. With fast inference times, the ability to generalize across diverse task types, and minimal data requirements for training, our approach offers a practical solution for data center scheduling\. This framework demonstrates strong potential for advancing sustainability objectives in AI\-driven infrastructure\. Through our collaboration with a data center, we achieved a 32% reduction in energy consumption and a 30% decrease in waiting time\.

## 1Introduction

The rapid advancement of Machine Learning \(ML\), especially Large Language Models \(LLMs\), has brought about groundbreaking capabilities, yet it has also raised significant social and environmental concerns\[[47](https://arxiv.org/html/2608.18503#bib.bib19),[4](https://arxiv.org/html/2608.18503#bib.bib18),[42](https://arxiv.org/html/2608.18503#bib.bib17)\]\. One of the most pressing issues is the substantial energy and water resources consumed during ML model training and serving, which has sparked widespread concerns\[[8](https://arxiv.org/html/2608.18503#bib.bib22),[15](https://arxiv.org/html/2608.18503#bib.bib21),[56](https://arxiv.org/html/2608.18503#bib.bib39)\]\. The computationally intensive ML jobs can demand hundreds of GPU\-hours and consume substantial amounts of energy for power and water for cooling\. As a result, the growing demand for ML\-based applications has significantly amplified the environmental impact of data centers, highlighting the need for modern infrastructure that is both more efficient and sustainable specifically for AI\-driven workloads\[[7](https://arxiv.org/html/2608.18503#bib.bib45),[39](https://arxiv.org/html/2608.18503#bib.bib44),[33](https://arxiv.org/html/2608.18503#bib.bib46)\]\. This paper aims to enhance data center operation efficiency and reduce the environmental footprint of modern data centers by devising an LLM\-based predictive scheduling system for allocating data center resources \(e\.g\., GPUs\) across a stream of ML jobs efficiently\. We harness the contextual understanding and predictive abilities of LLMs and combine the predictive power with sequential decision\-making that optimizes data\-center operations\.

![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/outline.png)Figure 1:Our Proposed PipelineAs illustrated in Figure[1](https://arxiv.org/html/2608.18503#S1.F1), our proposed method first employs an LLM that takes the ML job’s source code as input and outputs estimates for the required resources, including execution time and GPU energy consumption\. We believe that the approach has great potential in predicting other metrics, including water consumption and carbon emissions, if these data are available from the data center\. By leveraging these estimates, the data center can make more informed and efficient real\-time decisions for GPU resource allocation, leading to improvements such as reduced task queue waiting times and lower overall energy consumption\.

This work aims to address a key limitation in the current operational pipelines of small\-scale to large\-scale data centers\. We focus on the prevalent practice where users submit tasks and request resources, and the data center allocates resources based on heuristic algorithms that rely on user\-provided estimates, such as predicted task duration and resource needs\[[60](https://arxiv.org/html/2608.18503#bib.bib16)\]\. The optimal resource allocation, in hindsight, depends on the actual time and energy consumed by the tasks\. However, accurately predicting these factors is nearly impossible due to the inherent complexity of modern computing systems\. Consequently, neither the user nor the data center has precise knowledge of the optimal resources required for a given task, leading to inefficiencies and resource wastage\. Our approach introduces a unified, effective predictive model, followed by a real\-time, data\-driven scheduling system that enables more efficient and sustainable resource allocation decisions\.

![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/unified0.png)Figure 2:Illustration of our unified predictive approach\. Notably, previous methods were unable to handle tasks that had not been seen before, or composite tasks \(e\.g\., training a CNN followed by an LSTM\)\. However, the generalization capacity of LLMs allows our model to effectively manage such cases, making it adaptable to a wider variety of task types and combinations\.### 1\.1Our Contribution

We present a prototype pipeline that provides both methodological and practical contributions to various aspects of data center operations, specifically tailored for AI training and inference applications\. Our approach provides methodological insights while opening up new possibilities for applications, especially within today’s sustainability and AI\-driven context\. We defer more discussions on the related works to Appendix[A\.1](https://arxiv.org/html/2608.18503#A1.SS1)\.

Methodological Contributions:Our work advances the fields of data center predictive modeling and sequential decision\-making\.

- •LLM\-based Versatile Predictive Model:While many predictive methods exist for data center operations, to our best knowledge, we are the first to offer a comprehensive, end\-to\-end solution that takes source code as input and outputs estimations of interest\. Our model has the potential to be compatible withanyuser\-submitted task and can predictanymeasurable metric the data center requires\. The strength of our approach lies in the fact that LLM\-generated representations are more informative and generalizable than handcrafted features, leading to improved predictive performance and enhanced operational efficiency\.
- •Fully Automated Predictive Scheduling System:This approach enables the possibility of a fully automated predictive scheduling system\. By adopting our unified framework, the system is capable of handling any user\-submitted tasks and providing estimations of interest directly\. This was previously unattainable \(see Figure[3](https://arxiv.org/html/2608.18503#S2.F3)\), as traditional prediction methods were highly task\-specific \(e\.g\., CNN\-only or LLM\-only\), and could only cover a limited number of task types\. Moreover, each type of task required separate prediction models and handcrafted features, significantly limiting the flexibility and scalability of these methods\.
- •Sequential Decision\-Making Algorithms:Our predictive framework is complemented by a sequential decision\-making algorithm for optimizing resource allocation\. This problem arises from the area of reusable resource allocation and queuing control, where no analytical solution exists\. We propose a data\-driven decision\-making algorithm and implement it in collaboration with a small data center\. Our algorithm significantly outperforms the baseline scheduling rule, reducing energy consumption by32%32\\%and queuing delays by30%30\\%\. This approach leads to more effective data\-center management, especially given the growing emphasis on sustainability\.

Application\-Level Contributions:The unique features of our methodology—an end\-to\-end predictive model and its compatibility with diverse task types and estimation targets—offer substantial potential for developing next\-generation AI\-driven infrastructures in data centers\.

- •Practical Deployment:Our pipeline executes within one second, enabling a cost\-effective, fully automated predictive scheduling system\. Additionally, training and deploying the model within data centers is straightforward and can be effective even with a limited amount of data, making it particularly practical for real\-world data center operations\.
- •Multi\-Purpose Sustainable Data Center Operations:Our method is versatile enough to predict a wide range of metrics, including time and energy, and we believe it can also predict carbon emissions and water consumption\. Unlike previous target\-specific approaches, which were limited to particular models, our unified framework offers an all\-in\-one solution that addresses a broad array of predictive needs, providing key insights for building more sustainable and environmentally friendly data center infrastructures\.

## 2Problem Formulation

The Prediction Problem\.Consider a data center equipped with various types of GPUs, where machine learning tasks arrive sequentially\. The data center must decide which type of GPU to allocate to each arriving task \(noting that the number of GPUs is often specified in the user\-submitted code or command input\)\. Let𝒙\\bm\{x\}represent the source code submitted by the user, andzzrepresent the data center’s decision, which corresponds to the type of GPU to allocate for executing the task\. A decisionzzresults in outcomes including the time required to complete the task, denoted bytt, and the energy consumption using configurationzzto complete task𝒙\\bm\{x\}, denoted byee\. The relationship between the outcomes and the decision is modeled by functionsffandgg, such that

t:=f⁡\(𝒙,z\),e:=g⁡\(𝒙,z\)\.\\displaystyle t:=f\(\\bm\{x\},z\),\\,\\,\\,\\,\\,e:=g\(\\bm\{x\},z\)\.Here we note that these functionsffandggare data\-center\-specific, they depend on the infrastructure of the data center, for example, bandwidth, CPUs, memory, storage, communication, system setup, software stack, etc\. In general, characterizing the functionsf⁡\(𝒙,z\)f\(\\bm\{x\},z\)andg⁡\(𝒙,z\)g\(\\bm\{x\},z\)is highly challenging due to the complexity of modern computing systems\.

Prediction using Large Language Models\.To tackle the prediction problem, we employ Large Language Models \(LLMs\)\[[43](https://arxiv.org/html/2608.18503#bib.bib41),[52](https://arxiv.org/html/2608.18503#bib.bib40),[1](https://arxiv.org/html/2608.18503#bib.bib10)\]\. For simplicity, we refer readers to\[[62](https://arxiv.org/html/2608.18503#bib.bib43)\]for a detailed explanation of decoder\-based Transformers and LLM architectures\. In this paper, we usefθf\_\{\\theta\}andgθg\_\{\\theta\}as shorthand notations to represent LLMs, whereθ\\thetaencapsulates all the model parameters\.fθf\_\{\\theta\}andgθg\_\{\\theta\}take two inputs: the task’s source code𝒙\\bm\{x\}and the data center’s decisionzz\. The rationale is that the LLM can effectively analyze and interpret the task’s source code, transforming it into meaningful representations; these representations, when combined with the GPU configurationzz, allow the model to produce estimates such that:

fθ​\(𝒙,z\)≈f⁡\(𝒙,z\),gθ​\(𝒙,z\)≈g⁡\(𝒙,z\)\.f\_\{\\theta\}\(\\bm\{x\},z\)\\approx f\(\\bm\{x\},z\),\\,\\,\\,\\,\\,\\,g\_\{\\theta\}\(\\bm\{x\},z\)\\approx g\(\\bm\{x\},z\)\.
To justify our LLM\-based approach, we highlight two key features of LLMs that make them particularly appealing in our context:

- •Contextual comprehension and feature extraction\.LLMs are renowned for their superior ability to comprehend the context in the source code and extract features, referred to as representations\. With a large corpus of pre\-training data, LLMs and other natural language models \(NLP\) can discern essential parts of the code with relevant information, transform them into representations containing meaningful information\[[58](https://arxiv.org/html/2608.18503#bib.bib54),[48](https://arxiv.org/html/2608.18503#bib.bib55)\], and leverage these vectors to predict time, energy consumption, and other metrics\.
- •Generalization\.Another remarkable capability of LLMs is their ability to generalize\[[67](https://arxiv.org/html/2608.18503#bib.bib57)\]\. Even if the pre\-training dataset does not cover all possible examples within the function domains, LLMs can still recognize patterns in the dataset and extend their learned knowledge to achieve reasonably accurate approximations\.

### 2\.1Model Architecture \- Prediction

In this section, we formally describe the architecture of our prediction model\. Our approach utilizes a pre\-trained LLM for extracting source code representations and applies a probe\[[2](https://arxiv.org/html/2608.18503#bib.bib58),[51](https://arxiv.org/html/2608.18503#bib.bib59),[64](https://arxiv.org/html/2608.18503#bib.bib60),[71](https://arxiv.org/html/2608.18503#bib.bib56)\]to train a supervised model that predicts the quantity of interest\. Figure[3](https://arxiv.org/html/2608.18503#S2.F3)illustrates the architecture and the different performance compared to previous methods\.

![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/fig3.png)Figure 3:Illustration of our model architecture\. Our streamlined design is easy and fast to implement, highly flexible, and generalizable across a variety of tasks\. Notice that to estimate other quantities of interest, we simply need to add additional probes\. In contrast, previous methods required different handcrafted features and separate prediction models for each task, making them far less flexible and scalable\. The lower graph illustrates the performance difference, where points closer to the straight line indicate more accurate predictions\. The performance is evaluated under test sets with additional adversarially generated data, and our method outperforms the previous approach \(\[[9](https://arxiv.org/html/2608.18503#bib.bib70)\]and\[[32](https://arxiv.org/html/2608.18503#bib.bib50)\]\) in both accuracy and robustness\. Notably, the previous method shows consistent systematic bias, likely due to its reliance on “physical” features like the number of forward/backward passes and layer depth, which lack generalizability\.- •Extracting Representations:We adopt a pre\-trained LLM to extract task representations, modeled as a functionlθ1​\(⋅\):𝒱→ℝdl\_\{\\theta\_\{1\}\}\(\\cdot\):\\mathcal\{V\}\\to\\mathbb\{R\}^\{d\}, where𝒱\\mathcal\{V\}denotes the space of source code, andℝd\\mathbb\{R\}^\{d\}is thedd\-dimensional embedding space for mapped representations\. Here,θ1\\theta\_\{1\}encapsulates the LLM’s parameters\. Note thatlθ1​\(⋅\)l\_\{\\theta\_\{1\}\}\(\\cdot\)refers to the LLM without its final linear and softmax layers, and for the last layer, it outputs the last token’s representation, which is add\-dimensional vector\.
- •Probing:Probing involves taking the generated representations and predicting the target value using a linear regression or shallow neural network\. We denote this probe ashθ2​\(⋅,⋅\):ℝd×𝒵→ℝh\_\{\\theta\_\{2\}\}\(\\cdot,\\cdot\):\\mathbb\{R\}^\{d\}\\times\\mathcal\{Z\}\\to\\mathbb\{R\}, whereθ2\\theta\_\{2\}represents the parameters of the probe and𝒵\\mathcal\{Z\}denotes the decision space \(e\.g\., the types of GPUs\)\.

Thus, our predictive model for the execution time can be represented asfθ​\(𝒙,z\)=hθ2​\(lθ1​\(𝒙\),z\)f\_\{\\theta\}\(\\bm\{x\},z\)=h\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\),z\)\. For the energy estimation functiongθ​\(𝒙,z\)g\_\{\\theta\}\(\\bm\{x\},z\), we use the same representation but train a separate probe, denotedqθ2q\_\{\\theta\_\{2\}\}, such thatgθ​\(𝒙,z\)=qθ2​\(lθ1​\(𝒙\),z\)g\_\{\\theta\}\(\\bm\{x\},z\)=q\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\),z\)\. Note thatθ\\theta,θ1\\theta\_\{1\}, andθ2\\theta\_\{2\}refer to classes of parameters, indicating that the parameter dimensions are the same across functions, though these functions do not necessarily share the same parameters\.

Advantage over Previous Methods\.We identify the key factor that could explain the advantage of our method\. As mentioned earlier, the challenging part of data\-center operations arises from the complexity of modern computer systems\. We first give a notation that characterizes the previous approaches\.

- •For estimating execution timet=f⁡\(𝒙,z\)t=f\(\\bm\{x\},z\)and energy consumptione=g⁡\(𝒙,z\)e=g\(\\bm\{x\},z\), previous methods rely on handcrafted features, denoted by𝒖t=lt​\(𝒙\)\\bm\{u\}\_\{t\}=l\_\{t\}\(\\bm\{x\}\)for time and𝒖e=le​\(𝒙\)\\bm\{u\}\_\{e\}=l\_\{e\}\(\\bm\{x\}\)for energy, wherelt​\(⋅\)l\_\{t\}\(\\cdot\)andle​\(⋅\)l\_\{e\}\(\\cdot\)represent the feature extraction functions\. Importantly, the design oflt​\(⋅\)l\_\{t\}\(\\cdot\)andle​\(⋅\)l\_\{e\}\(\\cdot\)varies significantly depending on the target variable and prediction context, making them different for each prediction task\.
- •Separate models are then trained for each target, such asht​\(⋅,⋅\)h\_\{t\}\(\\cdot,\\cdot\)for time andhe​\(⋅,⋅\)h\_\{e\}\(\\cdot,\\cdot\)for energy, resulting in the following approximations:t≈ht​\(lt​\(𝒙\),z\)t\\approx h\_\{t\}\(l\_\{t\}\(\\bm\{x\}\),z\)ande≈he​\(le​\(𝒙\),z\)e\\approx h\_\{e\}\(l\_\{e\}\(\\bm\{x\}\),z\)\. To account for the dependence onzz, which represents the GPU choices, several estimation techniques attempt to model the architecture differences between GPUs\. While this approach makes sense physically, it suffers from a lack of flexibility and scalability\. Modeling the dependence onzzis typically restricted to specific types of source code𝒙\\bm\{x\}and GPU choiceszz, rendering the models unable to generalize to unseen tasks or GPU configurations\.

Our approach is better in terms of the architecture for various reasons\.

- •Better Representation\.It has been demonstrated in various domains that representations extracted by pre\-trained models are significantly more effective than handcrafted features\. In this context, our prediction task can also be viewed as a form of feature extraction akin to Natural Language Processing \(NLP\), where pre\-trained models capture richer and more comprehensive information\. These representations are more generalizable and universal, making them applicable to a wide variety of prediction tasks related to the characteristics of the source code\.
- •Data\-Driven Predictive Modeling\.Many traditional predictive models rely on the intrinsic characteristics of the underlying machine learning model, such as CNNs, where predictions are made by explicitly calculating factors like the number of forward/backward passes and the layer depth\. While these “physical” models can achieve high accuracy when the test environment perfectly matches the assumptions, they often fall short in data\-center operations\. For instance, even tasks that train the same CNN architecture may exhibit variations in execution time due to differences in the way the code is written\. Additionally, many modern tasks involve training and inference across multiple models, alongside various other function implementations\. In such settings, physically based predictive models lack the flexibility and adaptability needed for accurate predictions\. In contrast, our data\-driven approach is flexible enough to learn the inherent relationships between the source code, the characteristics of computer systems and GPUs, and a variety of targeted metrics\.
- •Less Data Hungry\.Traditional methods often require large amounts of source code examples to train predictive models effectively\. In contrast, by leveraging a pretrained LLM, which has already been trained on terabytes of data, we capitalize on the model’s superior understanding of contextual knowledge\. This allows us to obtain a highly effective feature extractor with far less training data, improving both data efficiency and performance\. Indeed, our predictive model is trained using only a little more than 500 source code examples, each paired with corresponding results for energy consumption and execution time across different GPUs in the server\.
- •Generalization for GPU Dependence\.Another advantage of our approach is its ability to maximize the generalization power of the LLM’s representation across different downstream tasks\. By utilizing the same representation for every prediction task and maintaining a consistent structure for the downstream probe, our method facilitates the discovery of patterns across different GPU configurations\. This consistency preserves the model’s generalization capacity, ensuring its adaptability to various GPU architectures\. Such a factor is missing in previous methods, which followed a more restricted modeling approach, limiting their flexibility and ability to generalize across diverse hardware setups\.

Training Process\.To train our predictive model, for simplicity we assume the parameters of the pretrained LLMlθ1​\(⋅\)l\_\{\\theta\_\{1\}\}\(\\cdot\)remain fixed, and we will discuss the practical training procedure in a follow\-up remark\. We first collect a dataset𝒟\\mathcal\{D\}consisting of the representations of the source codelθ1​\(𝒙\)l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\)and the corresponding execution results, including timettand energyee, across different GPU configurationszz\. Each task𝒙\\bm\{x\}is executed on multiple GPUs to measure both time and energy consumption\. Let\|𝒵\|\|\\mathcal\{Z\}\|denote the cardinality of the decision space, i\.e\., the number of available GPUs\. The training set, with sizenn, can be written as:

𝒟=\{\(lθ1​\(𝒙i\),\{ti​j,ei​j\}j=1\|𝒵\|\)\}i=1n,\\mathcal\{D\}=\\left\\\{\\left\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{i\}\),\\\{t\_\{ij\},e\_\{ij\}\\\}\_\{j=1\}^\{\|\\mathcal\{Z\}\|\}\\right\)\\right\\\}\_\{i=1\}^\{n\},whereti​j=f⁡\(𝒙i,j\)t\_\{ij\}=f\(\\bm\{x\}\_\{i\},j\)andei​j=g⁡\(𝒙i,j\)e\_\{ij\}=g\(\\bm\{x\}\_\{i\},j\)are the time and energy required for task𝒙i\\bm\{x\}\_\{i\}when executed on GPUz=jz=j\. Using this dataset, we train two separate probes:hθ2​\(⋅\)h\_\{\\theta\_\{2\}\}\(\\cdot\)for time prediction , andqθ2​\(⋅\)q\_\{\\theta\_\{2\}\}\(\\cdot\)for energy prediction\. The goal is to minimize the error between the actual valuesti​j,ei​jt\_\{ij\},e\_\{ij\}and the predictionshθ2​\(lθ1​\(𝒙i\),j\)h\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{i\}\),j\)andqθ2​\(lθ1​\(𝒙i\),j\)q\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{i\}\),j\)respectively\. Since, for simplicity, we assumeθ1\\theta\_\{1\}is fixed, the loss for time prediction isℒt​\(θ2\)=1n​\|𝒵\|​∑i=1n∑j=1\|𝒵\|\(ti​j−hθ2​\(lθ1​\(𝒙i\),j\)\)2,\\mathcal\{L\}\_\{t\}\(\\theta\_\{2\}\)=\\frac\{1\}\{n\|\\mathcal\{Z\}\|\}\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{\|\\mathcal\{Z\}\|\}\\left\(t\_\{ij\}\-h\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{i\}\),j\)\\right\)^\{2\},and the loss for energy prediction is:ℒe​\(θ2\)=1n​\|𝒵\|​∑i=1n∑j=1\|𝒵\|\(ei​j−qθ2​\(lθ1​\(𝒙i\),j\)\)2\.\\mathcal\{L\}\_\{e\}\(\\theta\_\{2\}\)=\\frac\{1\}\{n\|\\mathcal\{Z\}\|\}\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{\|\\mathcal\{Z\}\|\}\\left\(e\_\{ij\}\-q\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{i\}\),j\)\\right\)^\{2\}\.

We remark that the training process described above represents the simplest version for illustrative purposes\. In practice, the training process can be more complex, and we will address this here\.

- •One limitation of the approach presented is that the data center would need to run all submitted code to obtain execution time and energy consumption estimates\. However, data centers can leverage their existing user\-submitted codebase and operational data\. By recording execution time, energy consumption, and even metrics like carbon emissions and water usage \(if measurable\), data centers can accumulate vast amounts of data to train the probes effectively\. This method allows data centers to continuously refine their predictive models without the need for additional computational overhead\.
- •Another limitation of the current approach is that we fix the parametersθ1\\theta\_\{1\}of the pretrained LLM and only train the probe parametersθ2\\theta\_\{2\}\. While this approach simplifies the training process, it may not fully utilize the potential of the model\. In practice, we can enhance performance by fine\-tuning the LLM or incorporating more advanced techniques like Reinforcement Learning from Human Feedback \(RLHF\)\[[43](https://arxiv.org/html/2608.18503#bib.bib41)\]or incorporating a reward model\[[53](https://arxiv.org/html/2608.18503#bib.bib8)\]\. Based on the potential benefits of fine\-tuning, RLHF, and reward models, these approaches offer even greater possibilities for improving predictive performance\. These enhancements will be considered as part of future work\.

### 2\.2Problem Formulation: Decision\-Making

Constraints\.Consider a data center equipped with\|𝒵\|=M\|\\mathcal\{Z\}\|=Mtypes of GPUs, where tasksiiwith source code𝒙i\\bm\{x\}\_\{i\}arrive sequentially in time, and the data center must decide which GPU typezi∈𝒵z\_\{i\}\\in\\mathcal\{Z\}to allocate \(notice that the task’s source code𝒙i\\bm\{x\}\_\{i\}also specify its preference on GPUs, and we can model this by constraining the action set𝒵\\mathcal\{Z\}\)\. The GPU resources are limited and are represented by𝒄=\[c1,⋯,cM\]⊤\\bm\{c\}=\[c\_\{1\},\\cdots,c\_\{M\}\]^\{\\top\}, wherecjc\_\{j\}is the total number of GPUs of typejj\. If no GPUs are available, the task is placed in a waiting queue\. The total available GPU resources at any timessare represented by𝒄⁡\(s\)=\[c1​\(s\),⋯,cM​\(s\)\]⊤\\bm\{c\}\(s\)=\[c\_\{1\}\(s\),\\cdots,c\_\{M\}\(s\)\]^\{\\top\}, wherecj​\(s\)c\_\{j\}\(s\)is the available resource at timessfor GPU typejj\.

Dynamics\.Since all the tasks arrive sequentially, and the task arrival follows a stochastic process, let us denote byN⁡\(T\)N\(T\)the total number of tasks arrived during time\[0,T\]\[0,T\], and for each taskii, letsis\_\{i\}represent the arrival time\. If there are sufficient non\-occupied GPUs available, and we assignzi=jz\_\{i\}=jto taskii, thenai​ja\_\{ij\}number of corresponding inventory will be temporarily occupied, hence taken out fromcj​\(si\)c\_\{j\}\(s\_\{i\}\)\. Onceai​ja\_\{ij\}of GPUjjare assigned to𝒙j\\bm\{x\}\_\{j\}, the task will be running forti=f⁡\(𝒙i,zi\)t\_\{i\}=f\(\\bm\{x\}\_\{i\},z\_\{i\}\)amount of time\. If we denote by𝒜⁡\(s\)\\mathcal\{A\}\(s\)theactive setthat contains all tasks that are still running at timess, theni∈𝒜⁡\(s\)i\\in\\mathcal\{A\}\(s\)fors∈\(si,si\+ti\)s\\in\(s\_\{i\},s\_\{i\}\+t\_\{i\}\)\.

Waiting Queue\.LetQ⁡\(s\)Q\(s\)denote the set of waiting tasks at timess\. If a task𝒙i\\bm\{x\}\_\{i\}cannot be immediately processed due to insufficient GPU resources, it is placed in the setQ⁡\(s\)Q\(s\)\. If task𝒙i\\bm\{x\}\_\{i\}has been put in the queue, we denote bywiw\_\{i\}the time it spent in the queue, which depends on factors including other tasks in the waiting queue, the current tasks that are active with their corresponding GPU allocations, and the decision\-making model\.

Decision\-Making Model\.We aim to find a decision\-making policyπ\\pithat takes 4 inputs: the current task feature𝒙\\bm\{x\}, the current timess, the current inventory level𝒄⁡\(s\)\\bm\{c\}\(s\), and the historical information up to the current timeℋs\\mathcal\{H\}\_\{s\}\. The policy can be deterministic or stochastic, and the range of the policyπ\\piis𝒵∪∅\\mathcal\{Z\}\\cup\\emptyset\. Ifπ⁡\(𝒙,s,𝒄⁡\(s\),ℋ\)∈𝒵\\pi\(\\bm\{x\},s,\\bm\{c\}\(s\),\\mathcal\{H\}\)\\in\\mathcal\{Z\}, we assign the task GPU of typeπ⁡\(𝒙,s,𝒄⁡\(s\),ℋ\)\\pi\(\\bm\{x\},s,\\bm\{c\}\(s\),\\mathcal\{H\}\), and ifπ⁡\(𝒙,s,𝒄⁡\(s\),ℋ\)=∅\\pi\(\\bm\{x\},s,\\bm\{c\}\(s\),\\mathcal\{H\}\)=\\emptyset, we put this task in the waiting queue\. We denote byΠ\\Pithe space of policies that satisfy the condition described above\. Our goal is to solve the following objective function, which is a combination of the task completion time, the waiting time, and energy cost\. \(Notice that these times are directly affected by the policyπ\\pi, and we denote byS=maxi∈N⁡\(T\)⁡\{ti\+wi\}S=\\max\_\{i\\in N\(T\)\}\\\{t\_\{i\}\+w\_\{i\}\\\}the time at which all the tasks are finished\.\)

minπ∈Π​∑i=1N⁡\(T\)\(α​ti\+β​wi\+γ​ei\),\\displaystyle\\hskip 85\.35826pt\\text\{min\}\_\{\\pi\\in\\Pi\}\\,\\,\\,\\sum\_\{i=1\}^\{N\(T\)\}\\left\(\\alpha t\_\{i\}\+\\beta w\_\{i\}\+\\gamma e\_\{i\}\\right\),s\.t\.​∑i=1N⁡\(T\)ai​j​𝕀​\(i∈𝒜⁡\(s\)​and​zi=j\)≤cj,for alls∈\[0,S\]andj∈𝒵,\\displaystyle\\text\{s\.t\.\}\\,\\,\\,\\sum\_\{i=1\}^\{N\(T\)\}a\_\{ij\}\\mathbb\{I\}\(i\\in\\mathcal\{A\}\(s\)\\,\\,\\text\{and\}\\,\\,z\_\{i\}=j\)\\leq c\_\{j\},\\,\\,\\,\\,\\,\\,\\text\{for all $s\\in\[0,S\]$ and $j\\in\\mathcal\{Z\}$,\}whereα\\alpha,β\\beta, andγ\\gammaare the weights assigned to execution time, waiting time, and energy cost\. This formulation indeed resembles an online allocation problem with reusable resources, as discussed in\[[13](https://arxiv.org/html/2608.18503#bib.bib14),[72](https://arxiv.org/html/2608.18503#bib.bib13)\]\. However, two key factors differentiate our work from existing studies in the literature: \(i\) Multi\-purpose objective: while traditional allocation problems typically focus on optimizing a single reward or objective, our approach involves multiple goals\. This multi\-purpose objective requires balancing multiple criteria rather than relying on a single reward function, adding a layer of complexity not addressed by standard models of online allocation\. \(ii\) Waiting queue: In our problem, there is a waiting queue of tasks that influences decision\-making\. The policyπ\\pimust account for tasks already in the queue \(as inferred from the historyℋ\\mathcal\{H\}\) in addition to handling new arrivals\. This contrasts with standard resource allocation models, which usually make decisions based solely on newly arriving tasks, without considering previously queued tasks\. This interaction between the queue and decision\-making introduces an additional layer of complexity\. In summary, these challenges highlight that existing works can not provide \(near\) optimal solutions for our problem\. This necessitates the development of algorithms to effectively address these complexities\.

We note that for each task𝒙i\\bm\{x\}\_\{i\}, with the predictive model, the data center can incorporate the prediction results of time and energy,t^i:=fθ​\(𝒙i,zi\)\\hat\{t\}\_\{i\}:=f\_\{\\theta\}\(\\bm\{x\}\_\{i\},z\_\{i\}\)ande^i:=gθ​\(𝒙i,zi\),\\hat\{e\}\_\{i\}:=g\_\{\\theta\}\(\\bm\{x\}\_\{i\},z\_\{i\}\),into their decision\-making process\. This is because this information can be made available within 1 second after accessing the source code𝒙i\\bm\{x\}\_\{i\}, and does not rely on future information\. Notice that this decision\-making model is also applicable to other objectives, for example, water consumption and carbon emission, if corresponding data is provided by the data center\.

We conducted our experiment in collaboration with a data center, collecting two months of operational data from 07/01/2024 to 09/01/2024\. As shown in Figure[4](https://arxiv.org/html/2608.18503#S2.F4)\(a\), the task arrival times exhibit high non\-stationarity\. During this period, we recorded the characteristics of tasks, including execution time and energy consumption across different GPUs\. This dataset enabled us to backtest our decision\-making algorithms that rely on predictions from the LLM\-based model\.

We propose two algorithms: Greedy \(Algorithm[1](https://arxiv.org/html/2608.18503#algorithm1)\), where we follow a first\-come\-first\-served rule and the GPU type is selected greedily based on the smallest estimated objective valueα​t^i​j\+β​wi\+γ​e^i​j\\alpha\\hat\{t\}\_\{ij\}\+\\beta w\_\{i\}\+\\gamma\\hat\{e\}\_\{ij\}, and value\-based \(Algorithm[2](https://arxiv.org/html/2608.18503#algorithm2)\), which is inspired by the algorithm for the multiple knapsack problem\[[34](https://arxiv.org/html/2608.18503#bib.bib9)\]\. We compare the performance of these algorithms against the current baseline allocation policy of the data center, with the results presented in Figure[4](https://arxiv.org/html/2608.18503#S2.F4)\. Both algorithms outperform the baseline, with the Value\-based algorithm achieving a 32% reduction in energy consumption and a 30% reduction in task waiting time\. The Value\-based method outperforms the Greedy method because it considers all tasks in the waiting queue and accordingly assigns them based on their “values” of reducing the waiting time and energy cost, rather than following a simple first\-come\-first\-served rule as in the Greedy algorithm\. These results are consistent with theoretical insights from[57](https://arxiv.org/html/2608.18503#bib.bib6),[65](https://arxiv.org/html/2608.18503#bib.bib20)\. More experimental details can be found in Appendix[A\.3](https://arxiv.org/html/2608.18503#A1.SS3)\.

ModelTWT \(s\)TDTCRT\(s\)TEC \(kWh\)Simple3,135,824\.0129\.7916,924,533\.71470\.69Value\-based2,186,166\.4922\.5016,565,627\.33322\.07Improvement \(%\)30\.28%24\.47%2\.12%31\.58%Table 1:Performance gain for our algorithm implemented in data centers\. Here, TWT stands for total waiting time, TDT stands for total delayed tasks, the tasks that have to wait, CRT stands for cumulative running time, and TEC stands for the total energy cost\.![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/fig4-arrival.png)\(a\)Task Arrival time
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/fig4-waiting.png)\(b\)Number of Waiting Tasks
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/fig4-power.png)\(c\)Real\-time Energy Consumption

Figure 4:\(a\) Task arrival pattern and \(b\), \(c\) performance comparison among the proposed predictive decision\-making algorithms, Value\-based and Greedy \(see Appendix[A\.3\.1](https://arxiv.org/html/2608.18503#A1.SS3.SSS1)\), and the benchmark algorithm, Simple Rule\. The Simple Rule assigns the available GPU type to the first task in the waiting queue, following a first\-come\-first\-served policy when sufficient GPUs are available\. If multiple GPU types are available, the most powerful one \(e\.g\., A100\) is selected\.

## 3Other Extension

In this section, we discuss relevant extensions to our pipeline that align with data center practices\. Our prototype is designed using a pre\-trained LLM with strong capabilities in code comprehension and completion, coupled with a probe trained on 500 source\-code files\. Although we are able to achieve good performance with a relatively small dataset of 500 source\-code files, this approach still risks poor out\-of\-sample performance\. Due to variations in the coding practices of machine learning engineers—such as differences in variable names, function abstractions, and comments—the contextual information represented in the source code can vary significantly\. Even if two codes are functionally equivalent, resulting in identical execution times and energy consumption, the representations generated by the LLM can be quite different\.

This issue reflects an inconsistency problem\. Formally, there may exist two tasks,𝒙1\\bm\{x\}\_\{1\}and𝒙2\\bm\{x\}\_\{2\}, such that the representations differ noticeably \(lθ1​\(𝒙1\)≠lθ1​\(𝒙2\)l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{1\}\)\\neq l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{2\}\)\), but the execution times \(and energy consumption\) are identical:hθ2​\(lθ1​\(𝒙1\),z\)=hθ2​\(lθ1​\(𝒙2\),z\)h\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{1\}\),z\)=h\_\{\\theta\_\{2\}\}\(l\_\{\\theta\_\{1\}\}\(\\bm\{x\}\_\{2\}\),z\)\. We adopt another LLM, referred to as theAlign\-LLM, to address this issue and mitigate potential out\-of\-distribution \(OOD\) estimation errors, which could otherwise degrade model performance\.

![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/extension0.png)Figure 5:Architecture of the extended pipeline to resolve out\-of\-distribution tasksAs illustrated in Figure[5](https://arxiv.org/html/2608.18503#S3.F5), while data centers can gradually accumulate a code base to improve prediction performance for both in\-distribution and out\-of\-distribution tasks, we propose a novel algorithm that partially resolves the OOD issue, showcasing the flexibility and practicality of our pipeline\. The architecture and experimental details can be found in Appendix[A\.2](https://arxiv.org/html/2608.18503#A1.SS2)

The extended architecture first takes the source code as input and extracts representations using the pre\-trained LLM\. These representations are compared against representation clusters of the tasks from the local codebase to assess whether the current task’s representation deviates significantly from those in the training set\. If not, the task is passed to the probe for estimation\. Otherwise, the Align\-LLM assists by extracting key information from the source code, allowing the data center to identify similar code from its codebase\. Next, Align\-LLM rewrites the source code based on the style of the similar sample\. The rewritten code, although functionally identical to the original, adopts a more familiar written style to the pre\-trained LLM, aligning better with the LLM’s representations in the training dataset and enhancing the performance of the downstream probe\.

We highlight key insights from our experiments\. First, by computing representation distances for the source code in the probe’s training dataset, we observe that different task types tend to form distinct clusters \(Figure[6](https://arxiv.org/html/2608.18503#S3.F6)\(a\)\)\. Second, as shown in Figures[6](https://arxiv.org/html/2608.18503#S3.F6)\(b\) and \(c\), when source code with the same functionality is rewritten by different users or engineers, its embedding diverges from the original cluster, leading to significantly reduced predictive accuracy due to the probe encountering out\-of\-distribution inputs\. However, when we apply Align\-LLM to rewrite the code while following the style of the original task, the resulting representation is closer to the original distribution \(see indexes 4 vs\. 5 and 6 vs\. 7 in Figure[6](https://arxiv.org/html/2608.18503#S3.F6)\(b\)\)\. Although the rewritten representation is not identical, possibly due to the Align\-LLM’s style differing slightly from human coding styles, the probe can predict the rewritten code more accurately thanks to the generalizability of our predictive model\.

![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/pic/heatmap/heatmap_compare.png)\(a\)Distance of Task Clusters
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/pic/heatmap/heatmap_vit_class.png)\(b\)Distance before/after rewrite
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/pic/contradiction/test_change.png)\(c\)Prediction before rewrite
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/pic/contradiction/test_modify.png)\(d\)Prediction after rewrite

Figure 6:Figure \(a\) displays the heat map of embedding distances between two classes\. Indexes 0–3 represent tasks training the ViT model, while the remaining indexes correspond to tasks training the GAN Model\. In Figure \(b\), indexes 0 and 1 represent ViT model training tasks, indexes 2 and 4 represent out\-of\-distribution ViT model tasks, and indexes 3 and 6 are LLM\-rewritten versions of the tasks of index 2 and 4\. Figures \(c\) and \(d\) compare the prediction performance of the execution time for the out\-of\-distribution tasks before\(c\) and after\(d\) the LLM rewrite\.
## 4Conclusion

We propose an LLM\-based predictive scheduling system for data center operations aimed at enhancing the efficiency and sustainability of AI training\. Our system leverages a pre\-trained LLM specialized in code comprehension to extract representations of source code and utilizes sequential decision\-making algorithms to optimize the scheduling of computational resources\. Compared to traditional prediction methods, our architecture design and the use of LLMs offer significantly improved generalizability, flexibility, and practicality\. Complemented by the decision\-making scheduling algorithms, our approach achieves a 32% reduction in energy consumption and a 30% reduction in waiting times in real data centers\. These results demonstrate the strong potential of our method in advancing AI infrastructure\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§2](https://arxiv.org/html/2608.18503#S2.p2.1)\.
- \[2\]\(2016\)Understanding intermediate layers using linear classifier probes\.arXiv preprint arXiv:1610\.01644\.Cited by:[§2\.1](https://arxiv.org/html/2608.18503#S2.SS1.p1.1)\.
- \[3\]G\. Ali, M\. Side, S\. Bhalachandra, N\. J\. Wright, and Y\. Chen\(2023\)Performance\-aware energy\-efficient gpu frequency selection using dnn\-based models\.InProceedings of the 52nd International Conference on Parallel Processing,pp\. 433–442\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[4\]Amazon\(2024\)Our carbon footprint\.Note:[https://sustainability\.aboutamazon\.com/climate\-solutions](https://sustainability.aboutamazon.com/climate-solutions)Amazon SustainabilityCited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[5\]T\. Anderson, A\. Belay, M\. Chowdhury, A\. Cidon, and I\. Zhang\(2023\)Treehouse: a case for carbon\-aware datacenter software\.ACM SIGENERGY Energy Informatics Review3\(3\),pp\. 64–70\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[6\]L\. Bai, W\. Ji, Q\. Li, X\. Yao, W\. Xin, and W\. Zhu\(2022\)Dnnabacus: toward accurate computational cost prediction for deep neural networks\.arXiv preprint arXiv:2205\.12095\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[7\]R\. Bianchini, C\. Belady, and A\. Sivasubramaniam\(2024\)Datacenter power and energy management: past, present, and future\.IEEE Micro\.Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[8\]K\. Blunt and J\. Hiller\(2024\)Big Tech’s latest obsession is finding enough energy\.The Wall Street Journal\.External Links:[Link](https://www.wsj.com/business/energy-oil/big-techs-latest-obsession-is-finding-enough-energy-f00055b2)Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[9\]E\. Cai, D\. Juan, D\. Stamoulis, and D\. Marculescu\(2017\)Neuralpower: predict and deploy energy\-efficient convolutional neural networks\.InAsian Conference on Machine Learning,pp\. 622–637\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1),[Figure 3](https://arxiv.org/html/2608.18503#S2.F3),[Figure 3](https://arxiv.org/html/2608.18503#S2.F3.4)\.
- \[10\]Q\. Cao, Y\. K\. Lal, H\. Trivedi, A\. Balasubramanian, and N\. Balasubramanian\(2021\)IrEne: interpretable energy prediction for transformers\.arXiv preprint arXiv:2106\.01199\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[11\]J\. S\. Chase, D\. C\. Anderson, P\. N\. Thakar, A\. M\. Vahdat, and R\. P\. Doyle\(2001\)Managing energy and server resources in hosting centers\.ACM SIGOPS operating systems review35\(5\),pp\. 103–116\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[12\]G\. Chen, X\. Li, and Y\. Ye\(2024\)An improved analysis of lp\-based control for revenue management\.Operations Research72\(3\),pp\. 1124–1138\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[13\]Y\. Chen, R\. Levi, and C\. Shi\(2017\)Revenue management of reusable resources with advanced reservations\.Production and Operations Management26\(5\),pp\. 836–859\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1),[§2\.2](https://arxiv.org/html/2608.18503#S2.SS2.p4.2)\.
- \[14\]E\. Cortez, A\. Bonde, A\. Muzio, M\. Russinovich, M\. Fontoura, and R\. Bianchini\(2017\)Resource central: understanding and predicting workloads for improved resource management in large cloud platforms\.InProceedings of the 26th Symposium on Operating Systems Principles,pp\. 153–167\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[15\]C\. Criddle and K\. Bryan\(2024\)AI boom sparks concern over Big Tech’s water consumption\.The Financial Times\.External Links:[Link](https://www.ft.com/content/6544119e-a511-4cfa-9243-13b8cf855c13)Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[16\]M\. Daraghmeh, A\. Agarwal, and Y\. Jararweh\(2023\)A multilevel learning model for predicting cpu utilization in cloud data centers\.In2023 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress \(DASC/PiCom/CBDCom/CyberSciTech\),pp\. 1016–1023\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[17\]D\. Ding, X\. Fan, Y\. Zhao, K\. Kang, Q\. Yin, and J\. Zeng\(2020\)Q\-learning based dynamic task scheduling for energy\-efficient cloud computing\.Future Generation Computer Systems108,pp\. 361–371\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[18\]R\. Evans and J\. Gao\(2016\)DeepMind ai reduces google data centre cooling bill by 40%\.Google DeepMind blog\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[19\]X\. Fan, W\. Weber, and L\. A\. Barroso\(2007\)Power provisioning for a warehouse\-sized computer\.ACM SIGARCH computer architecture news35\(2\),pp\. 13–23\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[20\]J\. Feldman, M\. Henzinger, N\. Korula, V\. S\. Mirrokni, and C\. Stein\(2010\)Online stochastic packing applied to display ad allocation\.InEuropean Symposium on Algorithms,pp\. 182–194\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[21\]L\. J\. Gadhavi and M\. D\. Bhavsar\(2022\)Adaptive cloud resource management through workload prediction\.Energy Systems13\(3\),pp\. 601–623\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[22\]S\. Garg, R\. Ahuja, R\. Singh, and I\. Perl\(2023\)GMM\-lstm: a component driven resource utilization prediction model leveraging lstm and gaussian mixture model\.Cluster Computing26\(6\),pp\. 3547–3563\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[23\]X\. Y\. Geoffrey, Y\. Gao, P\. Golikov, and G\. Pekhimenko\(2021\)Habitat: a\{\\\{runtime\-based\}\\\}computational performance predictor for deep neural network training\.In2021 USENIX Annual Technical Conference \(USENIX ATC 21\),pp\. 503–521\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[24\]E\. Gianniti, L\. Zhang, and D\. Ardagna\(2018\)Performance prediction of gpu\-based deep learning applications\.In2018 30th International Symposium on Computer Architecture and High Performance Computing \(SBAC\-PAD\),pp\. 167–170\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[25\]S\. Gollapudi and D\. Panigrahi\(2019\)Online algorithms for rent\-or\-buy with expert advice\.InInternational Conference on Machine Learning,pp\. 2319–2327\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[26\]J\. Guerreiro, A\. Ilic, N\. Roma, and P\. Tomás\(2019\)GPU static modeling using ptx and deep structured learning\.IEEE Access7,pp\. 159150–159161\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[27\]G\. N\. Güğül, F\. Gökçül, and U\. Eicker\(2023\)Sustainability analysis of zero energy consumption data centers with free cooling, waste heat reuse and renewable energy systems: a feasibility study\.Energy262,pp\. 125495\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[28\]M\. H\. Hilman, M\. A\. Rodriguez, and R\. Buyya\(2018\)Task runtime prediction in scientific workflows using an online incremental learning approach\.In2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing \(UCC\),pp\. 93–102\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[29\]C\. Hsu, P\. Indyk, D\. Katabi, and A\. Vakilian\(2019\)Learning\-based frequency estimation algorithms\.\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[30\]L\. Huang, J\. Jia, B\. Yu, B\. Chun, P\. Maniatis, and M\. Naik\(2010\)Predicting execution time of computer programs using sparse polynomial regression\.Advances in neural information processing systems23\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[31\]E\. Jafarnejad Ghomi, A\. M\. Rahmani, and N\. N\. Qader\(2019\)Applying queue theory for modeling of cloud computing: a systematic review\.Concurrency and Computation: Practice and Experience31\(17\),pp\. e5186\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[32\]D\. Justus, J\. Brennan, S\. Bonner, and A\. S\. McGough\(2018\)Predicting the computational cost of deep learning models\.In2018 IEEE international conference on big data \(Big Data\),pp\. 3873–3882\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1),[Figure 3](https://arxiv.org/html/2608.18503#S2.F3),[Figure 3](https://arxiv.org/html/2608.18503#S2.F3.4)\.
- \[33\]L\. H\. Kaack, P\. L\. Donti, E\. Strubell, G\. Kamiya, F\. Creutzig, and D\. Rolnick\(2022\)Aligning artificial intelligence with climate change mitigation\.Nature Climate Change12\(6\),pp\. 518–527\.Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[34\]A\. R\. Kan, L\. Stougie, and C\. Vercellis\(1993\)A class of generalized greedy algorithms for the multi\-knapsack problem\.Discrete applied mathematics42\(2\-3\),pp\. 279–290\.Cited by:[§2\.2](https://arxiv.org/html/2608.18503#S2.SS2.p7.1)\.
- \[35\]A\. Katal, S\. Dahiya, and T\. Choudhury\(2023\)Energy efficiency in cloud computing data centers: a survey on software technologies\.Cluster Computing26\(3\),pp\. 1845–1875\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[36\]A\. Lacoste, A\. Luccioni, V\. Schmidt, and T\. Dandres\(2019\)Quantifying the carbon emissions of machine learning\.arXiv preprint arXiv:1910\.09700\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[37\]Y\. Lei and S\. Jasin\(2020\)Real\-time dynamic pricing for revenue management with reusable resources, advance reservation, and deterministic service time requirements\.Operations Research68\(3\),pp\. 676–685\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[38\]X\. Li and Y\. Ye\(2021\)Online linear programming: dual convergence, new algorithms, and regret bounds\.Operations Research70\(5\),pp\. 2948–2966\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[39\]Y\. Li, M\. Mughees, Y\. Chen, and Y\. R\. Li\(2024\)The unseen ai disruptions for power grids: llm\-induced transients\.arXiv preprint arXiv:2409\.11416\.Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[40\]T\. Lykouris and S\. Vassilvitskii\(2021\)Competitive caching with machine learned advice\.Journal of the ACM \(JACM\)68\(4\),pp\. 1–25\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[41\]C\. A\. Metz, M\. Goli, and R\. Drechsler\(2022\)ML\-based power estimation of convolutional neural networks on gpgpus\.In2022 25th International Symposium on Design and Diagnostics of Electronic Circuits and Systems \(DDECS\),pp\. 166–171\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[42\]M\. Nakagawa and B\. Smith\(2023\)On the road to 2030: our 2022 environmental sustainability report\.Note:[https://blogs\.microsoft\.com/on\-the\-issues/2023/05/10/2022\-environmental\-sustainability\-report/](https://blogs.microsoft.com/on-the-issues/2023/05/10/2022-environmental-sustainability-report/)Microsoft BlogCited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[43\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems35,pp\. 27730–27744\.Cited by:[2nd item](https://arxiv.org/html/2608.18503#S2.I5.i2.p1.1),[§2](https://arxiv.org/html/2608.18503#S2.p2.1)\.
- \[44\]K\. O’Neal and P\. Brisk\(2018\)Predictive modeling for cpu, gpu, and fpga performance and power consumption: a survey\.In2018 IEEE Computer Society Annual Symposium on VLSI \(ISVLSI\),pp\. 763–768\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[45\]P\. Patel, E\. Choukse, C\. Zhang, Í\. Goiri, B\. Warrier, N\. Mahalingam, and R\. Bianchini\(2024\)Characterizing power management opportunities for llms in the cloud\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 207–222\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[46\]T\. Pham, J\. J\. Durillo, and T\. Fahringer\(2017\)Predicting workflow task execution time in the cloud using a two\-stage machine learning approach\.IEEE Transactions on Cloud Computing8\(1\),pp\. 256–268\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[47\]S\. Pichai\(2024\)Climate change is humanity’s next big moonshot\.Note:[https://blog\.google/outreachinitiatives/sustainability/dear\-earth/](https://blog.google/outreachinitiatives/sustainability/dear-earth/)Google BlogCited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[48\]M\. T\. Pilehvar and J\. Camacho\-Collados\(2020\)Embeddings in natural language processing: theory and advances in vector representations of meaning\.Morgan & Claypool Publishers\.Cited by:[1st item](https://arxiv.org/html/2608.18503#S2.I1.i1.p1.1)\.
- \[49\]E\. Pinheiro, R\. Bianchini, E\. V\. Carrera, and T\. Heath\(2001\)Load balancing and unbalancing for power and performance in cluster\-based systems\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[50\]M\. Purohit, Z\. Svitkina, and R\. Kumar\(2018\)Improving online algorithms via ml predictions\.Advances in Neural Information Processing Systems31\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[51\]A\. Radford, R\. Jozefowicz, and I\. Sutskever\(2017\)Learning to generate reviews and discovering sentiment\.arXiv preprint arXiv:1704\.01444\.Cited by:[§2\.1](https://arxiv.org/html/2608.18503#S2.SS1.p1.1)\.
- \[52\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§2](https://arxiv.org/html/2608.18503#S2.p2.1)\.
- \[53\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2024\)Direct preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[2nd item](https://arxiv.org/html/2608.18503#S2.I5.i2.p1.1)\.
- \[54\]P\. Ranganathan, P\. Leech, D\. Irwin, and J\. Chase\(2006\)Ensemble\-level power management for dense blade servers\.ACM SIGARCH computer architecture news34\(2\),pp\. 66–77\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[55\]C\. Rodriguez, L\. Degioanni, L\. Kameni, R\. Vidal, and G\. Neglia\(2024\)Evaluating the energy consumption of machine learning: systematic literature review and experiments\.arXiv preprint arXiv:2408\.15128\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p1.1)\.
- \[56\]O\. Solon\(2021\)Drought\-stricken communities push back against data centers\.NBC News\.Note:[https://www\.nbcnews\.com/tech/internet/drought\-stricken\-communities\-push\-back\-against\-data\-centers\-n1271344](https://www.nbcnews.com/tech/internet/drought-stricken-communities-push-back-against-data-centers-n1271344)Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p1.1)\.
- \[57\]J\. Spencer, M\. Sudan, and K\. Xu\(2014\)Queueing with future information\.ACM SIGMETRICS Performance Evaluation Review41\(3\),pp\. 40–42\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1),[§2\.2](https://arxiv.org/html/2608.18503#S2.SS2.p7.1)\.
- \[58\]I\. Tenney, P\. Xia, B\. Chen, A\. Wang, A\. Poliak, R\. T\. McCoy, N\. Kim, B\. Van Durme, S\. R\. Bowman, D\. Das,et al\.What do you learn from context? probing for sentence structure in contextualized word representations\.InInternational Conference on Learning Representations,Cited by:[1st item](https://arxiv.org/html/2608.18503#S2.I1.i1.p1.1)\.
- \[59\]K\. Thonglek, K\. Ichikawa, K\. Takahashi, H\. Iida, and C\. Nakasan\(2019\)Improving resource utilization in data centers using an lstm\-based prediction model\.In2019 IEEE international conference on cluster computing \(CLUSTER\),pp\. 1–8\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[60\]M\. Tirmazi, A\. Barker, N\. Deng, M\. E\. Haque, Z\. G\. Qin, S\. Hand, M\. Harchol\-Balter, and J\. Wilkes\(2020\)Borg: the next generation\.InProceedings of the Fifteenth European Conference on Computer Systems,pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2608.18503#S1.p3.1)\.
- \[61\]S\. Tuli, S\. Ilager, K\. Ramamohanarao, and R\. Buyya\(2020\)Dynamic scheduling for stochastic edge\-cloud computing environments using a3c learning and residual recurrent neural networks\.IEEE transactions on mobile computing21\(3\),pp\. 940–954\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[62\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2608.18503#S2.p2.1)\.
- \[63\]J\. Vilaplana, F\. Solsona, I\. Teixidó, J\. Mateo, F\. Abella, and J\. Rius\(2014\)A queuing theory model for cloud computing\.The Journal of Supercomputing69,pp\. 492–507\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[64\]I\. Vulić, E\. M\. Ponti, R\. Litschko, G\. Glavaš, and A\. Korhonen\(2020\)Probing pretrained language models for lexical semantics\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7222–7240\.Cited by:[§2\.1](https://arxiv.org/html/2608.18503#S2.SS1.p1.1)\.
- \[65\]B\. Wagner, A\. Kohn, and T\. Neumann\(2021\)Self\-tuning query scheduling for analytical workloads\.InProceedings of the 2021 International Conference on Management of Data,pp\. 1879–1891\.Cited by:[§2\.2](https://arxiv.org/html/2608.18503#S2.SS2.p7.1)\.
- \[66\]H\. Wang and Y\. Cao\(2015\)Predicting power consumption of gpus with fuzzy wavelet neural networks\.Parallel Computing44,pp\. 18–36\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[67\]J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler,et al\.\(2022\)Emergent abilities of large language models\.Transactions on Machine Learning Research\.Cited by:[2nd item](https://arxiv.org/html/2608.18503#S2.I1.i2.p1.1)\.
- \[68\]Q\. Wu, Q\. Deng, L\. Ganesh, C\. Hsu, Y\. Jin, S\. Kumar, B\. Li, J\. Meza, and Y\. J\. Song\(2016\)Dynamo: facebook’s data center\-wide power management system\.ACM SIGARCH Computer Architecture News44\(3\),pp\. 469–480\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[69\]Q\. Xie and Y\. Lu\(2015\)Priority algorithm for near\-data scheduling: throughput and heavy\-traffic optimality\.In2015 IEEE Conference on Computer Communications \(INFOCOM\),pp\. 963–972\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p3.1)\.
- \[70\]K\. Xu and C\. W\. Chan\(2016\)Using future information to reduce waiting times in the emergency department via diversion\.Manufacturing & Service Operations Management18\(3\),pp\. 314–331\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1)\.
- \[71\]L\. Zhang, M\. Wang, L\. Chen, and W\. Zhang\(2022\)Probing gpt\-3’s linguistic knowledge on semantic tasks\.InProceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,pp\. 297–304\.Cited by:[§2\.1](https://arxiv.org/html/2608.18503#S2.SS1.p1.1)\.
- \[72\]X\. Zhang and W\. C\. Cheung\(2022\)Online resource allocation for reusable resources\.arXiv preprint arXiv:2212\.02855\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p4.1),[§2\.2](https://arxiv.org/html/2608.18503#S2.SS2.p4.2)\.
- \[73\]Q\. Zhao, H\. Yang, Z\. Luan, and D\. Qian\(2013\)POIGEM: a programming\-oriented instruction level gpu energy model for cuda program\.InAlgorithms and Architectures for Parallel Processing: 13th International Conference, ICA3PP 2013, Vietri sul Mare, Italy, December 18\-20, 2013, Proceedings, Part I 13,pp\. 129–142\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.
- \[74\]M\. Zhou, J\. Chen, H\. Hu, J\. Yu, Z\. Li, and H\. Hu\(2019\)Deeptle: learning code\-level features to predict code performance before it runs\.In2019 26th Asia\-Pacific Software Engineering Conference \(APSEC\),pp\. 252–259\.Cited by:[§A\.1](https://arxiv.org/html/2608.18503#A1.SS1.p2.1)\.

Content of Appendix

## Appendix AAppendix

### A\.1Discussion on related works

Prediction based on device features\.A body of literature focuses on predicting relevant metrics, such as execution time and energy consumption, using features that summarize the characteristics of the hardware\. This approach assumes that analyzing hardware parameters and runtime data can uncover patterns that influence these metrics\. For example,\[[46](https://arxiv.org/html/2608.18503#bib.bib48)\]combines GPU runtime parameters with static hardware features, applying regression models to predict execution time\. Similarly,\[[16](https://arxiv.org/html/2608.18503#bib.bib66)\]and\[[22](https://arxiv.org/html/2608.18503#bib.bib65)\]utilize various clustering techniques to identify operational patterns in machines based on hardware metrics, followed by sequence modeling for accurate predictions\. Conversely,\[[28](https://arxiv.org/html/2608.18503#bib.bib51)\]takes an alternative approach by first predicting hardware behavior during code execution and then employing the KNN clustering method to forecast execution time\. In general, this stream of literature is not very related to our approach and we refer the readers to\[[44](https://arxiv.org/html/2608.18503#bib.bib38),[55](https://arxiv.org/html/2608.18503#bib.bib37),[3](https://arxiv.org/html/2608.18503#bib.bib67)\]for a literature survey\.

Prediction based on code features\.Regarding code characteristics, early work by\[[30](https://arxiv.org/html/2608.18503#bib.bib47)\]used feature engineering, extracting elements like loop counts and conditional branches, and applying sparse polynomial regression for time prediction\. Recent approaches have shifted towards deep learning, where two main strategies dominate\. One approach treats models as composed of atomic operations, with works like\[[66](https://arxiv.org/html/2608.18503#bib.bib69),[9](https://arxiv.org/html/2608.18503#bib.bib70),[23](https://arxiv.org/html/2608.18503#bib.bib63),[32](https://arxiv.org/html/2608.18503#bib.bib50)\]using program slicing and MLPs to predict time and energy by decomposing models into layers\. The second approach leverages graph\-based techniques, as in\[[10](https://arxiv.org/html/2608.18503#bib.bib72)\]and\[[6](https://arxiv.org/html/2608.18503#bib.bib64)\], which use graphs to represent layer dependencies and employ machine learning to learn these representations\. Some methods are similar to ours in extracting code representations for prediction\. For example,\[[26](https://arxiv.org/html/2608.18503#bib.bib71)\]transforms PTX instructions into embeddings for LSTM inputs, while\[[74](https://arxiv.org/html/2608.18503#bib.bib52)\]uses attention\-based Bi\-LSTMs and graph convolutional networks to automatically extract code semantics and structure\. While these methods focus on extracting features, our model generalizes across a wider variety of task types and prediction metrics using a unified, high\-level representation approach\. See\[[24](https://arxiv.org/html/2608.18503#bib.bib49),[41](https://arxiv.org/html/2608.18503#bib.bib73),[73](https://arxiv.org/html/2608.18503#bib.bib68)\]for more related work\.

Data center operations\.Energy management in data centers has been a longstanding area of interest, with foundational work by\[[49](https://arxiv.org/html/2608.18503#bib.bib34),[11](https://arxiv.org/html/2608.18503#bib.bib35),[54](https://arxiv.org/html/2608.18503#bib.bib23),[19](https://arxiv.org/html/2608.18503#bib.bib24)\]\. As cloud computing and AI technologies emerged, software\-based approaches have been developed to improve data center operations\[[68](https://arxiv.org/html/2608.18503#bib.bib33),[18](https://arxiv.org/html/2608.18503#bib.bib61),[14](https://arxiv.org/html/2608.18503#bib.bib62),[35](https://arxiv.org/html/2608.18503#bib.bib26)\]\. In today’s AI and sustainability\-driven era, there has been growing interest in carbon emissions related to AI\[[36](https://arxiv.org/html/2608.18503#bib.bib25),[5](https://arxiv.org/html/2608.18503#bib.bib32),[27](https://arxiv.org/html/2608.18503#bib.bib31),[45](https://arxiv.org/html/2608.18503#bib.bib36)\]\. However, most efforts focus on software or infrastructure\-level operations, whereas our approach specifically targets algorithmic\-level improvements\. While there is a substantial body of literature on optimal scheduling and queuing policies for service systems, much of this work is highly theoretical and relies on numerous assumptions\[[63](https://arxiv.org/html/2608.18503#bib.bib28),[69](https://arxiv.org/html/2608.18503#bib.bib11),[31](https://arxiv.org/html/2608.18503#bib.bib27)\]\. Additionally, many scheduling algorithms that avoid theoretical assumptions, such as reinforcement learning\-based approaches\[[17](https://arxiv.org/html/2608.18503#bib.bib29),[61](https://arxiv.org/html/2608.18503#bib.bib30)\], primarily focus on scheduling tasks but lack task\-level predictive inference capabilities, which is a core strength of our method\. Additionally, we note that our focus is specifically on the operations of data centers for AI\-driven workloads, which exhibit distinct characteristics compared to traditional data center operations considered in previous studies\.

Predictive decision\-makingOur work is also related to the area of decision\-making with future predictions as side information\. With such predictions, well\-established decision algorithms, originally designed without the benefit of foresight, can be improved\[[50](https://arxiv.org/html/2608.18503#bib.bib1)\]\. Examples include problems such as caching\[[40](https://arxiv.org/html/2608.18503#bib.bib2)\], rent\-or\-buy \(also known as the ski rental problem\)\[[25](https://arxiv.org/html/2608.18503#bib.bib3)\], frequency estimation\[[29](https://arxiv.org/html/2608.18503#bib.bib4)\], and queuing control\[[57](https://arxiv.org/html/2608.18503#bib.bib6),[70](https://arxiv.org/html/2608.18503#bib.bib5)\]\. Among these, the most closely related topic to our work is \(online\) resource allocation\[[20](https://arxiv.org/html/2608.18503#bib.bib15),[38](https://arxiv.org/html/2608.18503#bib.bib42),[12](https://arxiv.org/html/2608.18503#bib.bib7),[72](https://arxiv.org/html/2608.18503#bib.bib13)\]\. Specifically,[37](https://arxiv.org/html/2608.18503#bib.bib12),[13](https://arxiv.org/html/2608.18503#bib.bib14)examine online allocation with reusable resources \(like GPUs in data centers\), where each arriving request occupies resources for only a limited period before releasing them\.\[[59](https://arxiv.org/html/2608.18503#bib.bib53)\]and\[[21](https://arxiv.org/html/2608.18503#bib.bib74)\]focus on CPU and memory utilization, adopting LSTM and other time series models to optimize resource allocation\.

### A\.2Experiment: Predictive model details

#### A\.2\.1Experiment Setup

##### Data Generation

The training \(and testing\) data for the probes includes embedding features and corresponding label values\.

- •Embedding Features: The input features for the probe model are generated through Starcoder\-7B\(LLM\)\. Specifically, we input 500 source code files as prompts into Starcoder\-7B\(LLM\) and extract 4608\-dimensional embedding vectors from each inference’s penultimate layer, i\.e\., the output of the last transformer block\. Following standard practices for sequence classification tasks, we then use the last token’s embedding vector to represent the features of the entire code file\. The 500 code files are either carefully selected or handcrafted to ensure that each file can run independently to complete a full computational process\. These files cover a diverse set of tasks and structures, including ResNet, BERT, GAN, ViT, VGG, and more\. The minimum, average, and maximum number of tokens in the code files are 1037, 1582\.26, and 2151, respectively\. We ensure that the LLM’s context length is sufficient to process the entire code file without truncation\.
- •Label Values: The label values, which the probes aim to predict, are generated by running the 500 code files on two different types of GPUs, NVIDIA A100 and NVIDIA A6000\. We record the running time and energy consumption using the official toolnvidia\-smiby NVIDIA\. For each code run, we open an independent process runningnvidia\-smi \-\-query\-gpu=power\.drawto record the real\-time power consumption with a logging interval of 1 second and compute the average power consumption\. For each code file, we run each experiment for at least twice and make sure the gap of recorded values is less than 10% of the average\. We also make sure the GPUs are exclusively used by our experiments\.

##### Probe Architecture

In our experiments, the probe model is a 3\-layer dense neural network, utilizing ReLU as the activation function, with batch normalization applied to each layer\. The input dimension of 4608 aligns with the dimensions of the \(input\) embedding vectors\. The embedding dimensions for each layer are 1024, 30, and 1, respectively\.

##### Probe Training

We randomly separate the generated data into training data and testing data with a ratio of 9:1\. All the data are further normalized by StandardScaler\. The probe models are trained using the following configurations: a batch size of 8, 2000 training epochs, a learning rate of 1e\-4, weight decay of 0\.001, andL​1L1regularization with a penalty parameter of 1e\-5\. We use the Mean Squared Error \(MSE\) as the loss function and AdamW as the optimizer\. Additionally, we apply early stopping when the epoch loss decreases by no more than 0\.001 for 30 consecutive epochs\.

##### Testing and Inference

To test the time needed for prediction, in the testing phase, we let the LLM Starcoder take the source code as input, outputting the representation, and use the probe to predict the estimated value\. The inference time for 48 source codes takes 32 seconds, averaging0\.650\.65seconds per task\. The testing phase is carried out on 1 Nvidia RTX 6000 \(Ampere Version\) GPU\.

##### Predictive pipeline for OOD

For the experiments in Section[3](https://arxiv.org/html/2608.18503#S3), we take gpt\-4o\-2024\-05\-13 as the Align\-LLM\. With the Align\-LLM, the rewrite time for the source code of 7 tasks takes 68 seconds, averaging 9\.7 seconds per task\.

### A\.3Experiment: Decision\-making model details

In this section, we first provide detailed predictive decision\-making algorithms applied in[A\.3](https://arxiv.org/html/2608.18503#A1.SS3)\. Further, due to data privacy restrictions imposed by the collaborating data center, we present additional numerical results across various settings using a simulation system\. These results can offer managerial insights for the data center operator and validate our choice of multi\-criteria optimization\.

#### A\.3\.1Predictive Decision\-Making Algorithms

In this section, we provide detailed implementations of the two types of predictive decision\-making algorithms used for GPU allocation\.

Input:Current time

ss; GPU types

𝒵\\mathcal\{Z\}; waiting queue

Q⁡\(s\)Q\(s\); active set

𝒜⁡\(s\)\\mathcal\{A\}\(s\); GPU occupation information

ai​ja\_\{ij\}; available GPUs

𝒄⁡\(s\)\\bm\{c\}\(s\); prediction models

fθf\_\{\\theta\}and

gθg\_\{\\theta\}; weights

α,γ\\alpha,\\gamma\.

Estimate the running time and energy of each task

i∈𝒜⁡\(s\)∪Q⁡\(s\)i\\in\\mathcal\{A\}\(s\)\\cup Q\(s\)for each GPU type

j∈𝒵j\\in\\mathcal\{Z\}:

t^i​j=fθ​\(𝒙i,j\),e^i​j=gθ​\(𝒙i,j\)\.\\hat\{t\}\_\{ij\}=f\_\{\\theta\}\(\\bm\{x\}\_\{i\},j\),\\qquad\\hat\{e\}\_\{ij\}=g\_\{\\theta\}\(\\bm\{x\}\_\{i\},j\)\.
//First\-come\-first\-served rule

Sort

Q⁡\(s\)Q\(s\)in ascending order by arrival time

sis\_\{i\};

Assign the first task to

z1=arg⁡minj∈𝒵⁡\(α​t^1​j\+γ​e^1​j\)\.z\_\{1\}=\\arg\\min\_\{j\\in\\mathcal\{Z\}\}\\left\(\\alpha\\hat\{t\}\_\{1j\}\+\\gamma\\hat\{e\}\_\{1j\}\\right\)\.
Schedule task

11when GPU type

z1z\_\{1\}has sufficient available GPUs, i\.e\., at

min⁡\{s′≥s∣cz1​\(s′\)≥a1​z1\}\.\\min\\left\\\{s^\{\\prime\}\\geq s\\mid c\_\{z\_\{1\}\}\(s^\{\\prime\}\)\\geq a\_\{1z\_\{1\}\}\\right\\\}\.

Algorithm 1Predictive Decision\-Making: GreedyInput:Current time

ss; GPU types

𝒵\\mathcal\{Z\}; waiting queue

Q⁡\(s\)Q\(s\); active set

𝒜⁡\(s\)\\mathcal\{A\}\(s\); GPU occupation information

ai​ja\_\{ij\}; available GPUs

𝒄⁡\(s\)\\bm\{c\}\(s\); prediction models

fθf\_\{\\theta\}and

gθg\_\{\\theta\}; hyperparameter

κ\\kappa\.

Estimate the running time and energy of each task

i∈𝒜⁡\(s\)∪Q⁡\(s\)i\\in\\mathcal\{A\}\(s\)\\cup Q\(s\)for each GPU type

j∈𝒵j\\in\\mathcal\{Z\}:

t^i​j=fθ​\(𝒙i,j\),e^i​j=gθ​\(𝒙i,j\)\.\\hat\{t\}\_\{ij\}=f\_\{\\theta\}\(\\bm\{x\}\_\{i\},j\),\\qquad\\hat\{e\}\_\{ij\}=g\_\{\\theta\}\(\\bm\{x\}\_\{i\},j\)\.
Compute the value

vi​jv\_\{ij\}for each task

i∈Q⁡\(s\)i\\in Q\(s\)and GPU type

j∈𝒵j\\in\\mathcal\{Z\}:

vi​j=1ai​j​t^i​j−κ​e^i​j\.v\_\{ij\}=\\frac\{1\}\{a\_\{ij\}\\hat\{t\}\_\{ij\}\}\-\\kappa\\hat\{e\}\_\{ij\}\.
Construct the value set

𝒱=\{\(i,j,vi​j\)∣i∈Q\(s\),j∈𝒵\},\\mathcal\{V\}=\\left\\\{\(i,j,v\_\{ij\}\)\\mid i\\in Q\(s\),\\,j\\in\\mathcal\{Z\}\\right\\\},and sort

𝒱\\mathcal\{V\}in descending order by

vi​jv\_\{ij\};

foreach*\(i,j,vi​j\)∈𝒱\(i,j,v\_\{ij\}\)\\in\\mathcal\{V\}*do

if*ai​j≤cj​\(s\)a\_\{ij\}\\leq c\_\{j\}\(s\)*then

//Assign taskiito GPU typejj

Set

zi←jz\_\{i\}\\leftarrow j;

Remove all tuples associated with task

ii:

𝒱←𝒱∖\{\(i′,j′,vi′​j′\)∈𝒱∣i′=i\}\.\\mathcal\{V\}\\leftarrow\\mathcal\{V\}\\setminus\\left\\\{\(i^\{\\prime\},j^\{\\prime\},v\_\{i^\{\\prime\}j^\{\\prime\}\}\)\\in\\mathcal\{V\}\\mid i^\{\\prime\}=i\\right\\\}\.
Update the available GPUs:

cj​\(s\)←cj​\(s\)−ai​j\.c\_\{j\}\(s\)\\leftarrow c\_\{j\}\(s\)\-a\_\{ij\}\.else

Skip;

Algorithm 2Predictive Decision\-Making: Value BasedThe Greedy algorithm focuses on a first\-come\-first\-served approach to allocate tasks to GPUs\. At each time stepss, the algorithm estimates the running time and energy consumption for each taskiion each GPU typejjusing the prediction models\. Once these estimates are made, the waiting queue is sorted in ascending order of task arrival times\. The algorithm assigns the first task in the queue to the GPU type selected based on the smallest estimated value ofα​t^1​j\+γ​e^1​j\\alpha\\hat\{t\}\_\{1j\}\+\\gamma\\hat\{e\}\_\{1j\}, reflecting a weighted combination of running time and energy consumption\. This algorithm’s simplicity makes it efficient for quick decision\-making, but it may not always optimize resource allocation across the entire queue\.

The Value\-Based algorithm extends beyond the Greedy approach by considering all tasks in the waiting queue and selecting the GPU allocation based on a more strategic optimization\. At each time step, the algorithm first estimates the running time and energy consumption for each task across GPU types\. However, the next step involves computing a cost valuevi​jv\_\{ij\}for each task\-GPU pair, which is a combination of the inverse of the estimated running timet^i​j\\hat\{t\}\_\{ij\}adjusted by the GPU occupancy requirementai​ja\_\{ij\}and a penalty termκ​e^i​j\\kappa\\hat\{e\}\_\{ij\}representing the energy consumption\. The algorithm then constructs a set of task\-GPU pairs, sorting them in descending order based on the valuevi​jv\_\{ij\}\. It allocates the GPUs to tasks based on this sorted list, prioritizing higher values, and updates the available GPU resources accordingly\. This method is inspired by the algorithms in multiple knapsack problems and balances multiple objectives, such as reducing energy consumption and running time with limited available GPUs\.

#### A\.3\.2More Simulation Experiments

##### Simulation Environment

We built the simulation environment using data collected from the cooperating data center\. Specifically, we first estimated a heterogeneous Poisson process to model task arrivals\. And we build a task simulator that generates the running timeti​jt\_\{ij\}and the energy consumptionei​je\_\{ij\}for each arriving taskiiacross GPU typejj\. The values are randomly sampled from truncated normal distributions \(truncated above 0\), with the mean and variance estimated from the collected data\. Further, we assume there are two types of GPUs \(\|𝒵\|=2\|\\mathcal\{Z\}\|=2\) with the number of each type randomly sampled uniformly from the intervals\[10,20\]\[10,20\]and\[20,40\]\[20,40\], respectively\.

##### Performance under different criteria

We evaluate the performance of the proposed Algorithm[2](https://arxiv.org/html/2608.18503#algorithm2)under different settings, with varying emphasis on the optimization criteria\. Specifically, we tune the hyperparameterκ\\kappain Algorithm[2](https://arxiv.org/html/2608.18503#algorithm2)using independently sampled validation data, based on the following metrics: \(i\) waiting time only \(α=γ=0\\alpha=\\gamma=0andβ=1\\beta=1\) \(ii\) running time only \(β=γ=0\\beta=\\gamma=0andα=1\\alpha=1\), and \(iii\) energy only \(α=β=0\\alpha=\\beta=0andγ=1\\gamma=1\)\. These settings prioritize different objectives, allowing us to compare the results and validate the effectiveness of the proposed Algorithm[2](https://arxiv.org/html/2608.18503#algorithm2)under various criteria\. We compare the performance of Algorithm[2](https://arxiv.org/html/2608.18503#algorithm2)with a simple rule, which assigns the available GPU type to the first task in the waiting queue, following a first\-come\-first\-served policy, provided there are sufficient GPUs\. When multiple GPU types are available, the most powerful type \(e\.g\., A100\) is selected\. The reported results are based on100100testing samples, and the validation data contains5050samples\.

Table[2](https://arxiv.org/html/2608.18503#A1.T2)summarizes the experimental results\. First, it showcases that the proposed value\-based rule can outperform the benchmark simple rule consistently in all tuning methods\. In addition, it demonstrates that the performance of the value\-based rule aligns with the specific objective being emphasized during tuning\. Specifically, when tuned to minimize waiting time, it achieves the shortest waiting time \(and the fewest tasks with positive wait times\)\. Similarly, when tuned to minimize running time, it achieves the shortest running time\. Finally, when tuned to minimize energy consumption, it achieves the lowest energy usage during testing\. We also provide visualizations of sample path levels in Figure[7](https://arxiv.org/html/2608.18503#A1.F7)\.

MetricSimple ruleValue\-based ruleWaiting timeRunning timeEnergyTotal Waiting Time \(s\)2,466,062\.561,810,123\.922,045,399\.252,112,773\.56\-Tasks with Wait Time27\.3821\.1622\.1322\.80Total Running Time \(s\)15,333,868\.4715,215,721\.2115,001,371\.1515,291,965\.57Total Energy Cost \(kWh\)440\.98245\.42243\.97240\.07Table 2:Comparison of testing total waiting time, tasks with wait time, cumulative running time, and energy cost for the benchmark algorithm \(simple rule\) and value\-based models tuned under different emphasized objectives\.![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/Appendix/Exp1/Waiting.png)\(a\)Number of Waiting Tasks \(Waiting time\)
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/Appendix/Exp1/Energy.png)\(b\)Real\-time Energy Consumption \(Waiting time\)
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/Appendix/Exp2/Waiting.png)\(c\)Number of Waiting Tasks \(Running time\)
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/Appendix/Exp2/Energy.png)\(d\)Real\-time Energy Consumption \(Running time\)
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/Appendix/Exp3/Waiting.png)\(e\)Number of Waiting Tasks \(Energy\)
![Refer to caption](https://arxiv.org/html/2608.18503v1/figures/Appendix/Exp3/Energy.png)\(f\)Real\-time Energy Consumption \(Energy\)

Figure 7:Performance comparison across different metrics for the benchmark algorithm \(simple rule\) and value\-based models tuned under different emphasized objectives\.