量子云编排中成本与延迟权衡的量子强化学习

# Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration
Source: [https://arxiv.org/html/2609.27446](https://arxiv.org/html/2609.27446)
Dang Van HuynhMuhammad UsmanHoa T\. Nguyen††thanks:A\. N\. H\. Phan and D\. V\. Huynh are with the Faculty of Computer Networks and Communications, University of Information Technology, Vietnam National University, Ho Chi Minh City, Quarter 34, Linh Xuan Ward, Ho Chi Minh City, Vietnam \(e\-mails: 23520029@gm\.uit\.edu\.vn, danghv@uit\.edu\.vn\)\.††thanks:M\. Usman is with the Faculty of Information Technology, Monash University, Clayton, Victoria, Australia\. The work was done while the author was at CSIRO \(e\-mail: muhammad\.usman@monash\.edu\)\.††thanks:H\. T\. Nguyen is with the Quantum Systems Team, CSIRO, Clayton, Victoria, Australia \(e\-mail: hoa\.nguyen@csiro\.au\)\.

###### Abstract

Quantum cloud computing, delivered through the quantum\-as\-a\-service \(QaaS\) model, provides access to quantum computing resources\. However, applying uniform time\-based pricing across fundamentally heterogeneous quantum resources significantly complicates task orchestration, particularly when addressing the tradeoff between execution costs and system performance\. While heuristic methods rely on predefined scheduling rules, classical deep reinforcement learning \(DRL\) models may require more trainable parameters in this setting\. Motivated by the potential of parameterised quantum circuits \(PQCs\) as compact function approximators, we propose QRLQ, a cost\-delay\-aware quantum cloud scheduling framework integrating PQCs with a dueling double deep Q\-network \(D3QN\) to dynamically account for both cost and delay\. Our simulation results show that QRLQ achieves lower mean cost and delay than the heuristic baselines, achieving a 5\-11% lower mean cost relative to availability\-based and rotation\-based heuristics and reducing mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively, while retaining execution fidelity within 2% of a fidelity\-greedy policy\. Compared with the classical DRL baseline, QRLQ achieves comparable scheduling performance while using 72% fewer trainable parameters\. This work explores the feasibility of using QRL for task orchestration in quantum cloud environments and demonstrates its potential for cost\-delay\-aware quantum resource management\.

###### Index Terms:

Cost\-aware scheduling, parameterised quantum circuits, quantum cloud computing, quantum reinforcement learning, quantum resource management\.

## IIntroduction

Quantum cloud computing is an emerging computing model that integrates quantum processing units \(QPUs\) into conventional cloud infrastructure, enabling on\-demand access to quantum hardware through the quantum\-as\-a\-service \(QaaS\) model\[[1](https://arxiv.org/html/2609.27446#bib.bib1)\]\. By reducing the need for individual organisations to acquire and maintain physical QPUs, QaaS lowers the barrier to quantum exploration and application development\[[2](https://arxiv.org/html/2609.27446#bib.bib2)\]\. Given its impact on accessibility and scalability, quantum cloud computing is expected to become one of the main ways to access quantum computing resources over the next decade\[[3](https://arxiv.org/html/2609.27446#bib.bib3)\]\.

Despite this potential, orchestrating QPU resources remains challenging due to hardware heterogeneity\. Each QPU has different technical specifications, from high\-level system metrics such as the number of qubits, quantum volume, and circuit layer operations per second\[[4](https://arxiv.org/html/2609.27446#bib.bib4)\]to low\-level physical properties such as gate error rates and execution duration\[[5](https://arxiv.org/html/2609.27446#bib.bib5),[6](https://arxiv.org/html/2609.27446#bib.bib6)\]\. This heterogeneity directly affects execution fidelity, delay, and cost, making quantum task placement a multi\-objective scheduling problem\.

In the current noisy intermediate\-scale quantum \(NISQ\) era\[[7](https://arxiv.org/html/2609.27446#bib.bib7)\], maximising execution fidelity remains a primary objective for obtaining meaningful computational results\[[5](https://arxiv.org/html/2609.27446#bib.bib5)\]\. However, under the commercial QaaS model\[[8](https://arxiv.org/html/2609.27446#bib.bib8),[9](https://arxiv.org/html/2609.27446#bib.bib9)\], cost is also an important constraint\. For example, advanced classical GPU instances typically cost tens of dollars per hour, whereas reserving dedicated access to the QuEra Aquila through Amazon Braket Direct costs $2,500 per hour111https://aws\.amazon\.com/braket/pricing \(accessed: July 2026\)\. the IBM Quantum Pay\-as\-you\-go plan charges $96 per QPU\-minute222https://www\.ibm\.com/quantum/products \(accessed: July 2026\)\. These prices show that quantum resource allocation should consider both cost and execution fidelity\.

QaaS platforms use different pricing models that affect resource allocation decisions\[[10](https://arxiv.org/html/2609.27446#bib.bib10)\]\. Providers such as AWS Braket employ heterogeneous pricing based on the selected quantum device or request type, making cost differences among available quantum devices an important factor in resource allocation decisions\. In contrast, platforms such as IBM Quantum operate under a homogeneous time\-based pricing model, where different quantum resources are charged at the same rate and billed based on execution time\. Because assigning requests to lower\-performance hardware increases execution time and consequently cost, while concentrating workloads on high\-performance hardware creates queues, scheduling under uniform pricing involves a direct cost\-delay tradeoff\. These queues increase delay\[[11](https://arxiv.org/html/2609.27446#bib.bib11)\], which is also understood as the total completion time\[[2](https://arxiv.org/html/2609.27446#bib.bib2)\]\. Effective quantum cloud scheduling therefore needs to reduce execution time to lower cost while bounding delay\.

Given that cost is directly proportional to QPU execution time, minimising execution time also minimises cost\. Existing scheduling strategies often consider execution time as a secondary metric and prioritise execution fidelity as the primary objective\[[5](https://arxiv.org/html/2609.27446#bib.bib5),[6](https://arxiv.org/html/2609.27446#bib.bib6)\]\. This approach suggests a tradeoff between execution time and execution fidelity\. However, across the evaluated IBM backends, shorter execution time did not coincide with lower execution fidelity\. We evaluate three representative circuit families from the MQT Bench library\[[12](https://arxiv.org/html/2609.27446#bib.bib12)\], namely GHZ, QAOA, and EfficientSU2, with varying circuit depths across selected IBM Quantum simulated backends\. Fig\.[1](https://arxiv.org/html/2609.27446#S1.F1)shows that the evaluated backends with shorter execution times generally exhibit relatively high execution fidelity, while those with longer execution times do not consistently exhibit higher fidelity\. Therefore, selecting a higher\-performance QPU may reduce both execution time and cost without reducing execution fidelity\.

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig01_execution_time_fidelity.png)Fig\. 1:Average execution time and execution fidelity across selected IBM Quantum simulated backends for the evaluated circuits with varying depths\.Existing work on quantum cloud resource orchestration relies mainly on heuristic strategies\[[3](https://arxiv.org/html/2609.27446#bib.bib3)\]\. Although some studies now consider cost as an optimisation objective\[[10](https://arxiv.org/html/2609.27446#bib.bib10)\], heuristic methods often lack the adaptability required to handle dynamic and stochastic environments\. To overcome the limitations of static policies, recent work has adopted deep reinforcement learning \(DRL\) for adaptive scheduling in dynamic quantum cloud environments\[[2](https://arxiv.org/html/2609.27446#bib.bib2),[13](https://arxiv.org/html/2609.27446#bib.bib13)\]\. By optimising sequential scheduling decisions, DRL provides a suitable approach to address this problem\. Building on advances in both DRL and quantum computing, quantum reinforcement learning \(QRL\) replaces the classical neural network of a DRL agent with parameterised quantum circuits \(PQCs\)\[[14](https://arxiv.org/html/2609.27446#bib.bib14)\]\. By leveraging the potential parameter efficiency of PQCs, QRL provides a parameter\-efficient function approximator for DRL and has demonstrated competitive performance in classical cloud resource allocation\[[15](https://arxiv.org/html/2609.27446#bib.bib15)\], suggesting its potential for the more constrained quantum cloud setting\.

To the best of our knowledge, no prior work applies QRL to quantum cloud resource orchestration under a uniform time\-based pricing model\. Moreover, current NISQ hardware is constrained by noise and gate errors, limiting the practical deployment of deep quantum circuits\. Therefore, shallow PQCs are practically important for stable training on current quantum hardware\[[16](https://arxiv.org/html/2609.27446#bib.bib16),[17](https://arxiv.org/html/2609.27446#bib.bib17)\]\. Motivated by this research gap and the limitations of current NISQ hardware, we propose QRLQ, aQRL\-based cost\-delay\-aware resource orchestration framework forQuantum cloud environments with uniform time\-based pricing\. We also evaluate the proposed architecture under parameter perturbations in the context of current NISQ hardware, with promising results across the evaluated perturbation levels\. This study explores the feasibility of applying QRL to quantum cloud resource orchestration, targeting future fault\-tolerant and error\-corrected environments, where the impact of operational noise is expected to be substantially reduced\. The primary contributions of this study are as follows\.

- •We formulate cost\-delay\-aware quantum task orchestration as an online sequential decision problem to jointly minimise execution and waiting times\. To address the inherently stochastic dynamics of the environment, we model the orchestration process as a Markov Decision Process \(MDP\), thereby enabling reinforcement learning\-based scheduling in dynamic quantum cloud settings\.
- •We propose QRLQ, a scheduling framework built on a quantum duelling double deep Q\-network, for cost\-delay\-aware resource orchestration in quantum cloud environments\. Our framework addresses the tradeoff between cost and delay by jointly minimising execution time and waiting time\.
- •We present the design of the hybrid quantum\-classical network in our QRLQ framework, including the PQC configuration, quantum state encoding, and quantum\-to\-classical feature readout for duelling network\-based scheduling across heterogeneous QPUs\.
- •We show that QRLQ outperforms heuristic strategies by reducing costs by 5–11% and delay by 17–82%, while retaining execution fidelity within 2% of a fidelity\-greedy policy\. Compared with classical DRL, QRLQ lowers the number of model parameters by approximately 72% while achieving comparable scheduling performance\.

Paper structure and notations:Section[II](https://arxiv.org/html/2609.27446#S2)reviews quantum resource orchestration approaches from static heuristics to DRL and discusses the potential of QRL for quantum cloud management\. Section[III](https://arxiv.org/html/2609.27446#S3)formalises the system model and problem statement for heterogeneous quantum cloud orchestration with multiple objectives\. Section[IV](https://arxiv.org/html/2609.27446#S4)details the core methodology driving the hybrid QRLQ framework\. Section[V](https://arxiv.org/html/2609.27446#S5)evaluates this framework alongside a discussion of simulation results\. Section[VI](https://arxiv.org/html/2609.27446#S6)concludes the study with key insights and future research directions\. Key mathematical notations are summarised in Table[I](https://arxiv.org/html/2609.27446#S1.T1)\.

TABLE I:Summary of key notations\.SymbolDescriptionSymbolDescription𝒯\\mathcal\{T\}Set of QTasks\.𝒩\\mathcal\{N\}Set of QNodes\.τi\\tau\_\{i\}ii\-th QTask\.njn\_\{j\}jj\-th QNode\.tiat\_\{i\}^\{a\}Arrival time ofτi\\tau\_\{i\}\.tjrt\_\{j\}^\{r\}Next ready time ofnjn\_\{j\}\.TjT\_\{j\}Operation\-time mapping\.EjE\_\{j\}Operation\-error mapping\.𝒯i,jexec\\mathcal\{T\}\_\{i,j\}^\{\\mathrm\{exec\}\}Execution time onnjn\_\{j\}\.𝒯i,jwait\\mathcal\{T\}\_\{i,j\}^\{\\mathrm\{wait\}\}Waiting time onnjn\_\{j\}\.𝒯i,j\\mathcal\{T\}\_\{i,j\}Total delay onnjn\_\{j\}\.ℱi,j\\mathcal\{F\}\_\{i,j\}Execution fidelity onnjn\_\{j\}\.ℛe\\mathcal\{R\}^\{\\mathrm\{e\}\}Execution\-time score\.ℛw\\mathcal\{R\}^\{\\mathrm\{w\}\}Waiting\-time score\.Φi,j\\Phi\_\{i,j\}Overall scheduling score\.CiC\_\{i\}Abstract circuit ofτi\\tau\_\{i\}\.
## IIRelated Work

### II\-AHeuristic\-based Orchestration Approaches

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig02_system_architecture.png)Fig\. 2:System\-level workflow of the proposed QRLQ framework for cost\-delay\-aware resource orchestration\.Early studies mainly used heuristics and mathematical optimisation\. For example, adaptive scheduling based on historical IBM Quantum statistics aims to maximise execution fidelity under low\-load conditions and minimise waiting time under high\-load conditions\[[3](https://arxiv.org/html/2609.27446#bib.bib3)\]\. To improve execution fidelity, QuSplit uses a genetic algorithm to decompose quantum jobs based on backend noise levels\[[18](https://arxiv.org/html/2609.27446#bib.bib18)\]\.

Several studies in this category have considered cost\. To handle demand uncertainty, two\-stage stochastic programming methods have been applied to minimise the total cost, including reservation cost, usage cost, and penalties for exceeding the maximum waiting time\[[19](https://arxiv.org/html/2609.27446#bib.bib19)\]\. Quantum\-inspired particle swarm optimisation algorithms have also been proposed to minimise latency, energy consumption, and cost under execution\-time pricing models\[[10](https://arxiv.org/html/2609.27446#bib.bib10)\]\. Similarly, multi\-criteria decision\-making methods have been integrated into systems such as Qonductor to provide resource plans that balance cost and execution fidelity\[[11](https://arxiv.org/html/2609.27446#bib.bib11)\]\. However, rule\-based, history\-based, and mathematical programming approaches often struggle to adapt to dynamic quantum cloud environments, which are characterised by unpredictable workload arrivals and changing queue states\.

### II\-BLearning\-based Orchestration Approaches

DRL has been used for quantum cloud orchestration to address the limitations of static heuristics, as it can learn scheduling policies from changing system states\. DRLQ represents an early DRL\-based framework in this direction, where Rainbow deep Q\-networks \(DQNs\) are used to minimise total completion time and the number of rescheduling events\[[2](https://arxiv.org/html/2609.27446#bib.bib2)\]\. In the context of serverless quantum functions using the function\-as\-a\-service model, graph convolutional networks combined with proximal policy optimisation \(PPO\) have been used to improve the distribution of quantum functions\[[13](https://arxiv.org/html/2609.27446#bib.bib13)\]\. To address the challenge of executing large\-scale circuits that exceed the capacity of a single QPU, PPO algorithms have also been applied to allocate sub\-tasks across multiple QPUs\[[6](https://arxiv.org/html/2609.27446#bib.bib6)\]\. This method maximises expected execution fidelity while learning its tradeoff with execution time and network communication latency\. More recently, QFOR combines PPO with hardware calibration data to balance execution fidelity and execution time based on custom weights\[[5](https://arxiv.org/html/2609.27446#bib.bib5)\]\. Although these DRL\-based approaches are promising for quantum cloud orchestration, they mainly focus on maximising execution fidelity and minimising total completion time\. These methods do not optimise cost during orchestration, even though cost is an important constraint for commercial QaaS platforms\.

While DRL is a suitable approach for this orchestration problem, recent advances in quantum machine learning have made QRL a promising direction\. Instead of relying on parameter\-intensive dense neural networks, QRL employs PQCs as function approximators, providing a compact parameterisation with substantially fewer trainable parameters\. Although QRL has been studied for multi\-objective optimisation in other domains\[[15](https://arxiv.org/html/2609.27446#bib.bib15),[16](https://arxiv.org/html/2609.27446#bib.bib16),[20](https://arxiv.org/html/2609.27446#bib.bib20)\], its specific application to quantum cloud resource orchestration has not yet been extensively investigated\.

## IIISystem Model and Problem Formulation

### III\-ASystem Architecture

Fig\.[2](https://arxiv.org/html/2609.27446#S2.F2)illustrates the system\-level workflow of the proposed QRLQ framework within a heterogeneous QaaS environment\. The architecture consists of a centralised broker managed by the QRLQ orchestrator and a set of heterogeneous quantum nodes \(QNodes\) to process user\-submitted quantum tasks \(QTasks\)\. Each QTask includes an abstract circuit with specific technical characteristics\. Initially, a feature extractor processes this circuit to extract key task features\. By evaluating these task features alongside the dynamic QNode states, the QRLQ core allocates each QTask to the QNode with the highest estimated action value\. Then, the circuit is passed to the preparation stage before execution on the assigned QNode\[[13](https://arxiv.org/html/2609.27446#bib.bib13)\]\.

In the NISQ era, the heterogeneity of quantum hardware results in different execution times for the same abstract circuit\[[21](https://arxiv.org/html/2609.27446#bib.bib21)\]\. This variation can increase cost when a QTask is allocated to a QNode with a longer execution time\. As a result, orchestrators may tend to assign more QTasks to a small number of high\-performance QNodes\. This concentration of tasks, together with the limited availability of high\-performance quantum resources, can increase the waiting time of QTasks in queues before execution\. In this study, we aim to jointly minimise execution time and waiting time using an equal\-weight multi\-objective formulation, thereby reducing both cost and delay\.

### III\-BQuantum Task and Resource Models

#### III\-B1Quantum Task Model

In a quantum cloud environment, QTasks represent execution requests submitted by users to the system\[[10](https://arxiv.org/html/2609.27446#bib.bib10)\]\. Each QTask contains the technical parameters required for execution, such as the number of shots necessary for statistical sampling, and typically includes one or more gate\-based quantum circuits\. These circuits serve as low\-level representations of a quantum algorithm\[[3](https://arxiv.org/html/2609.27446#bib.bib3)\]\. In this work, we model a QTask as a computational workload that requires adaptive resource orchestration to optimise execution performance\. Furthermore, we assume each QTask consists of a single, independent abstract circuit\.

As shown in the system architecture in Fig\.[2](https://arxiv.org/html/2609.27446#S2.F2), when a QTask arrives, the task feature extractor module analyses the abstract circuit to gather the essential parameters required for the scheduler’s decision\-making process\. Formally, we consider a set of independent arriving QTasks, denoted as𝒯=\{τ1,τ2,…,τN\}\\mathcal\{T\}=\\\{\\tau\_\{1\},\\tau\_\{2\},\\dots,\\tau\_\{N\}\\\}, whereN=\|𝒯\|N=\|\\mathcal\{T\}\|\. Each QTaskτi∈𝒯\\tau\_\{i\}\\in\\mathcal\{T\}is formulated as

τi=\(tia,Ci,Wi,Di,Gi\(1​q\),Gi\(2​q\),Mi,Si\),\\tau\_\{i\}=\(t\_\{i\}^\{a\},C\_\{i\},W\_\{i\},D\_\{i\},G\_\{i\}^\{\(1q\)\},G\_\{i\}^\{\(2q\)\},M\_\{i\},S\_\{i\}\),\(1\)
wheretiat\_\{i\}^\{a\}denotes the arrival time of QTaskτi\\tau\_\{i\}\. The abstract circuitCiC\_\{i\}is represented as a directed acyclic graph \(DAG\)\. Before hardware\-specific compilation,CiC\_\{i\}requires a circuit widthWiW\_\{i\}and has a pre\-compilation depthDiD\_\{i\}\. Furthermore, the gate\-level execution cost is parameterised by the number of single\-qubit and two\-qubit gates, denoted asGi\(1​q\)G\_\{i\}^\{\(1q\)\}andGi\(2​q\)G\_\{i\}^\{\(2q\)\}, respectively, along with the final measurement operationsMiM\_\{i\}\. Finally,SiS\_\{i\}specifies the number of shots required to construct a statistically reliable probability distribution for the measurement outcomes\.

#### III\-B2Quantum Resource Model

In quantum cloud infrastructure, quantum computing resources are represented as QNodes, each abstracting a QPU together with its execution queue and hardware characteristics\. Because of physical constraints in the NISQ era, these characteristics vary substantially across QNodes, including qubit capacity and qubit connectivity\[[21](https://arxiv.org/html/2609.27446#bib.bib21)\]\. In modern hardware implementations, a physical quantum resource may include multiple QPUs and support multi\-programming for parallel circuit execution\[[22](https://arxiv.org/html/2609.27446#bib.bib22)\]\. In this study, we focus on the single\-QPU case and assume that each QNode abstracts a single dedicated QPU\. Furthermore, a node provides exclusive access to one independent quantum circuit at any given time, provided that the required number of qubits does not exceed the available capacity of the target node\. Execution is strictly non\-preemptive\[[5](https://arxiv.org/html/2609.27446#bib.bib5),[2](https://arxiv.org/html/2609.27446#bib.bib2)\]\.

Formally, we consider a heterogeneous set of QNodes available in the quantum cloud infrastructure, denoted as𝒩=\{n1,n2,…,nM\}\\mathcal\{N\}=\\\{n\_\{1\},n\_\{2\},\\dots,n\_\{M\}\\\}, whereM=\|𝒩\|M=\|\\mathcal\{N\}\|\. Each QNodenj∈𝒩n\_\{j\}~\\in~\\mathcal\{N\}is modelled as the parameter tuple

nj=\(tjr,Qj,Bj,Tj,Ej\),n\_\{j\}=\(t\_\{j\}^\{r\},Q\_\{j\},B\_\{j\},T\_\{j\},E\_\{j\}\),\(2\)
wheretjrt\_\{j\}^\{r\}denotes the expected next available time when the QNodenjn\_\{j\}completes its currently scheduled tasks and becomesready for new allocation\. The parameterQjQ\_\{j\}represents the set of physical qubits available at the QNode, andBjB\_\{j\}specifies the native gate set supported by the hardware\. Finally, hardware speed characteristics are described by the time mapping functionTj:Bj×𝒜j→ℝ\>0T\_\{j\}:B\_\{j\}\\times\\mathcal\{A\}\_\{j\}\\rightarrow\\mathbb\{R\}\_\{\>0\}, where𝒜j\\mathcal\{A\}\_\{j\}represents the set of all hardware\-valid qubit operand configurations for executing supported operations onnjn\_\{j\}\. Similarly, the error functionEj:Bj×𝒜j→\[0,1\]E\_\{j\}:B\_\{j\}\\times\\mathcal\{A\}\_\{j\}\\rightarrow\[0,1\]represents the corresponding error probability of that execution\.

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig03_dag_circuit_transpilation.png)Fig\. 3:Example of quantum circuit transpilation and its DAG representation for execution time and execution fidelity estimation\.Figure[3](https://arxiv.org/html/2609.27446#S3.F3)illustrates the processing of a QTask by quantum computing resources and the associated circuit compilation process\. The abstract circuit, shown in the left panel, must be compiled and mapped to the physical qubit topology of the target quantum hardware, such as theibm\_sydneysystem depicted in the centre panel\. This procedure produces a transpiled circuit, represented as a DAG in the right panel\. Notably, the DAG of the transpiled circuit enables execution time estimation via the critical path using operation timesTjT\_\{j\}, and supports fidelity evaluation through error probabilitiesEjE\_\{j\}\.

### III\-COrchestration Performance Model

As discussed in Section[I](https://arxiv.org/html/2609.27446#S1), under the homogeneous pricing model, the cost is directly proportional to the actual execution time of the QPU\. Letρ\\rhodenote the unit price per unit time\. For each accepted assignment, the cost of executing a QTaskτi\\tau\_\{i\}on a QNodenjn\_\{j\}is formally expressed as

Cost​\(τi,nj\)=ρ​𝒯i,jexec,\\text\{Cost\}\(\\tau\_\{i\},n\_\{j\}\)=\\rho\\mathcal\{T\}\_\{i,j\}^\{\\text\{exec\}\},\(3\)
where the execution time𝒯i,jexec\\mathcal\{T\}\_\{i,j\}^\{\\text\{exec\}\}is estimated directly by analysing the critical path of the transpiled circuit\. We assume serial shot execution without reset or readout overheads, establishing a deterministic and consistent execution time model for training scheduling policies across heterogeneous backends, similar to\[[5](https://arxiv.org/html/2609.27446#bib.bib5)\]\. Formally, the execution time can be calculated as

𝒯i,jexec=Si​∑g∈C​P​\(Ci,j′\)Tj​\(g,q→g\),\\mathcal\{T\}\_\{i,j\}^\{\\text\{exec\}\}=S\_\{i\}\\sum\_\{g\\in CP\(C\_\{i,j\}^\{\\prime\}\)\}T\_\{j\}\(g,\\vec\{q\}\_\{g\}\),\(4\)
whereCi,j′C\_\{i,j\}^\{\\prime\}is the transpiled circuit ofτi\\tau\_\{i\}mapped ontonjn\_\{j\}, andC​P​\(Ci,j′\)CP\(C\_\{i,j\}^\{\\prime\}\)denotes its critical path\.Tj​\(g,q→g\)T\_\{j\}\(g,\\vec\{q\}\_\{g\}\)represents the execution time of operationggon the target qubit setq→g\\vec\{q\}\_\{g\}along this path, whileSiS\_\{i\}is the number of shots requested by the user\. In a quantum cloud environment, simple circuits with shallow depth and few qubits require significantly less execution time than complex circuits\. This difference introduces the risk that scheduling strategies based mainly on execution time may favour simple circuits over complex circuits\. To reduce this effect, we introduce a complexity bonusκi∈\(0,1\]\\kappa\_\{i\}\\in\(0,1\]for QTasks with higher circuit complexity\[[5](https://arxiv.org/html/2609.27446#bib.bib5)\], which is formulated as

κi=wd​DiDmax\+wg​OiOmax\+ws​SiSmax,\\kappa\_\{i\}=w\_\{d\}\\frac\{D\_\{i\}\}\{D\_\{\\max\}\}\+w\_\{g\}\\frac\{O\_\{i\}\}\{O\_\{\\max\}\}\+w\_\{s\}\\frac\{S\_\{i\}\}\{S\_\{\\max\}\},\(5\)
whereDiD\_\{i\}is the depth of the abstract circuit andOiO\_\{i\}is its total number of gates, calculated asOi=Gi\(1​q\)\+Gi\(2​q\)\+MiO\_\{i\}=G\_\{i\}^\{\(1q\)\}\+G\_\{i\}^\{\(2q\)\}\+M\_\{i\}\.SiS\_\{i\}represents the number of shots, whilewdw\_\{d\},wgw\_\{g\}, andwsw\_\{s\}act as adjustment coefficients wherewd\+wg\+ws=1w\_\{d\}\+w\_\{g\}\+w\_\{s\}=1\. The valuesDmaxD\_\{\\max\},OmaxO\_\{\\max\}, andSmaxS\_\{\\max\}denote the corresponding normalisation limits\. Accordingly, the execution time performance score is determined as

ℛe=\(1−η\)​\(1−𝒯~i,jexec\)\+η​κi,\\mathcal\{R\}^\{\\text\{e\}\}=\(1\-\\eta\)\(1\-\\tilde\{\\mathcal\{T\}\}\_\{i,j\}^\{\\text\{exec\}\}\)\+\\eta\\kappa\_\{i\},\(6\)
where𝒯~i,jexec\\tilde\{\\mathcal\{T\}\}\_\{i,j\}^\{\\text\{exec\}\}denotes the execution time normalised by the maximum execution time observed in the dataset to ensure training stability, andη\\etais the complexity adjustment coefficient, set by default to0\.20\.2\. Consequently, when a complex task receives a low performance score,κi\\kappa\_\{i\}increases the total scoreℛe\\mathcal\{R\}^\{\\text\{e\}\}to reduce the disadvantage of high\-complexity circuits, while execution time remains the main factor guiding the scheduler toward cost\-efficient resource allocation\.

The limited availability and high cost of quantum computing resources create competition for access to execution resources\. We assume that each QNode maintains a queue where each QTask waits for execution\. The waiting time𝒯i,jwait\\mathcal\{T\}\_\{i,j\}^\{\\text\{wait\}\}of QTaskτi\\tau\_\{i\}, calculated from its arrival at the system until the start of execution, is defined as

𝒯i,jwait=max⁡\(0,tjr−tia\),\\mathcal\{T\}\_\{i,j\}^\{\\text\{wait\}\}=\\max\(0,t\_\{j\}^\{r\}\-t\_\{i\}^\{a\}\),\(7\)
wheretjrt\_\{j\}^\{r\}is the time when QNodenjn\_\{j\}is ready for the next execution, andtiat\_\{i\}^\{a\}is the arrival time of QTaskτi\\tau\_\{i\}\. If𝒯i,jwait=0\\mathcal\{T\}\_\{i,j\}^\{\\text\{wait\}\}=0, QTaskτi\\tau\_\{i\}can commence execution immediately\. Subsequently, the next ready timetjrt\_\{j\}^\{r\}of QNodenjn\_\{j\}is updated totjr′=max⁡\(tia,tjr\)\+𝒯i,jexect\_\{j\}^\{r^\{\\prime\}\}=\\max\(t\_\{i\}^\{a\},t\_\{j\}^\{r\}\)\+\\mathcal\{T\}\_\{i,j\}^\{\\text\{exec\}\}\. We define the overall delay experienced by QTaskτi\\tau\_\{i\}as its delay, formulated as

𝒯i,j=𝒯i,jexec\+𝒯i,jwait\.\\mathcal\{T\}\_\{i,j\}=\\mathcal\{T\}\_\{i,j\}^\{\\text\{exec\}\}\+\\mathcal\{T\}\_\{i,j\}^\{\\text\{wait\}\}\.\(8\)
Finally, to evaluate the impact of delay on overall scheduling performance, let𝒯~i,jwait\\tilde\{\\mathcal\{T\}\}\_\{i,j\}^\{\\text\{wait\}\}denote the waiting time normalised in the same manner as the execution time\. The waiting time performance score is then defined as

ℛw=1−𝒯~i,jwait\.\\mathcal\{R\}^\{\\text\{w\}\}=1\-\\tilde\{\\mathcal\{T\}\}\_\{i,j\}^\{\\text\{wait\}\}\.\(9\)
We approximate the execution fidelity of QTaskτi\\tau\_\{i\}on QNodenjn\_\{j\}by accumulating the success probabilities of all physical operations in the transpiled circuit, following\[[5](https://arxiv.org/html/2609.27446#bib.bib5)\]\. Formally, the execution fidelity is approximated as

ℱi,j=∏g∈Ci,j′\[1−Ej​\(g,q→g\)\],\\mathcal\{F\}\_\{i,j\}=\\prod\_\{g\\in C\_\{i,j\}^\{\\prime\}\}\[1\-E\_\{j\}\(g,\\vec\{q\}\_\{g\}\)\],\(10\)
whereEj​\(g,q→g\)E\_\{j\}\(g,\\vec\{q\}\_\{g\}\)is the physical error probability of operationg∈Ci,j′g\\in C\_\{i,j\}^\{\\prime\}on the target qubit setq→g\\vec\{q\}\_\{g\}\. Consequently,\[1−Ej​\(g,q→g\)\]\[1\-E\_\{j\}\(g,\\vec\{q\}\_\{g\}\)\]represents the successful execution probability of that operation\.

### III\-DProblem Formulation

We formally model quantum cloud task orchestration as an online sequential decision\-making process\. A sequence of QTasks𝒯=\{τ1,τ2,…,τN\}\\mathcal\{T\}=\\\{\\tau\_\{1\},\\tau\_\{2\},\\dots,\\tau\_\{N\}\\\}arrives at the system\. Given a cluster of heterogeneous QNodes𝒩=\{n1,n2,…,nM\}\\mathcal\{N\}=\\\{n\_\{1\},n\_\{2\},\\dots,n\_\{M\}\\\}, the orchestration problem is to determine an orchestration policyπ:𝒯→𝒩\\pi:\\mathcal\{T\}\\rightarrow\\mathcal\{N\}that assigns each QTask to a selected QNode\. For each task assignment, the orchestration performance score is defined as

Φi,π⁡\(τi\)=β​ℛi,π⁡\(τi\)e\+\(1−β\)​ℛi,π⁡\(τi\)w,\\Phi\_\{i,\\pi\(\\tau\_\{i\}\)\}=\\beta\\mathcal\{R\}^\{\\text\{e\}\}\_\{i,\\pi\(\\tau\_\{i\}\)\}\+\(1\-\\beta\)\\mathcal\{R\}^\{\\text\{w\}\}\_\{i,\\pi\(\\tau\_\{i\}\)\},\(11\)
whereβ∈\[0,1\]\\beta\\in\[0,1\]is the strategy adjustment parameter controlling the tradeoff between execution time and waiting time\. In this study,β\\betais fixed to0\.50\.5to assign equal priority to both optimisation objectives\.

The online orchestration objective is to maximise the cumulative scheduling performance, which can be formulated as

maxπ\\displaystyle\\max\_\{\\pi\}\\quad∑i=1NΦi,π⁡\(τi\),\\displaystyle\\sum\_\{i=1\}^\{N\}\\Phi\_\{i,\\pi\(\\tau\_\{i\}\)\},\(12a\)s\.t\.Wi≤\|Qπ⁡\(τi\)\|,∀τi∈𝒯,\\displaystyle W\_\{i\}\\leq\|Q\_\{\\pi\(\\tau\_\{i\}\)\}\|,\\qquad\\forall\\tau\_\{i\}\\in\\mathcal\{T\},\(12b\)\(g,q→g\)∈Bπ⁡\(τi\)×𝒜π⁡\(τi\),∀\(g,q→g\)∈Ci,π⁡\(τi\)′\.\\displaystyle\\begin\{aligned\} \(g,\\vec\{q\}\_\{g\}\)&\\in B\_\{\\pi\(\\tau\_\{i\}\)\}\\times\\mathcal\{A\}\_\{\\pi\(\\tau\_\{i\}\)\},\\forall\(g,\\vec\{q\}\_\{g\}\)\\in C\_\{i,\\pi\(\\tau\_\{i\}\)\}^\{\\prime\}\.\\end\{aligned\}\(12c\)
Constraint \([12b](https://arxiv.org/html/2609.27446#S3.E12.2)\) ensures that the selected QNode provides sufficient physical qubits for the assigned QTask\. Constraint \([12c](https://arxiv.org/html/2609.27446#S3.E12.3)\) ensures that the transpiled circuit is compatible with the native operations and valid qubit operands of the assigned QNode\.

### III\-EMarkov Decision Process \(MDP\) Formulation

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig04_qrlq_architecture.png)Fig\. 4:The proposed cost\-delay\-aware QRLQ framework for quantum cloud resource orchestration, with eight qubits used in our simulations\.The QRLQ framework models the resource scheduling process as an MDP\[[5](https://arxiv.org/html/2609.27446#bib.bib5),[15](https://arxiv.org/html/2609.27446#bib.bib15)\]\. This allows the system to use reinforcement learning methods to approximate a high\-performing action\-value function instead of using a fixed scheduling rule\. This MDP is represented by the tuple\(𝐒,𝐀,𝐏,𝐑,γ\)\(\\mathbf\{S\},\\mathbf\{A\},\\mathbf\{P\},\\mathbf\{R\},\\gamma\), with𝐒\\mathbf\{S\}denoting the continuous state space and𝐀\\mathbf\{A\}representing the discrete action space corresponding to hardware allocation decisions\. The transition function𝐏\\mathbf\{P\}defines the state transition of the environment, including the evolution from statests\_\{t\}tost\+1s\_\{t\+1\}by moving to the subsequent QTaskτi\+1\\tau\_\{i\+1\}in the submission sequence and updating the available timetjr′t\_\{j\}^\{r^\{\\prime\}\}of the selected QNode\. Finally,𝐑\\mathbf\{R\}is the reward function guiding the optimisation, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor that controls short\-term and long\-term returns throughout the scheduling decision process\.

At each decision steptt, triggered by the arrival of QTaskτi\\tau\_\{i\}, the QRLQ agent observes the statest∈𝐒s\_\{t\}\\in\\mathbf\{S\}, selects an actionata\_\{t\}according to the policyπ⁡\(at\|st\)\\pi\(a\_\{t\}\|s\_\{t\}\), receives an immediate scalar rewardrt∈ℝr\_\{t\}\\in\\mathbb\{R\}, and transitions to the next statest\+1s\_\{t\+1\}\.

State Space𝐒\\mathbf\{S\}:At decision stept∈𝕋t\\in\\mathbb\{T\}, the observed statest∈𝐒s\_\{t\}\\in\\mathbf\{S\}is constructed by concatenating two distinct feature vectors encoding the characteristics of the incoming QTaskτi\\tau\_\{i\}and the hardware configuration of the available QNodes\. Mathematically, the complete state vectorsts\_\{t\}is expressed as

st=\(𝐟tτi,𝐟t𝒩\)\.s\_\{t\}=\(\\mathbf\{f\}^\{\\tau\_\{i\}\}\_\{t\},\\mathbf\{f\}^\{\\mathcal\{N\}\}\_\{t\}\)\.\(13\)
Specifically, task features are represented by the vector𝐟tτi∈ℝp\\mathbf\{f\}^\{\\tau\_\{i\}\}\_\{t\}\\in\\mathbb\{R\}^\{p\}, whereppdenotes the number of extracted features\. These include the composite complexity, computed as the product of the number of shots and circuit layers, along with the quantum load, which encompasses the number of single\-qubit gates, two\-qubit gates, and measurement operations, as introduced in Eq\. \([1](https://arxiv.org/html/2609.27446#S3.E1)\)\. Meanwhile, the feature matrix𝐟t𝒩∈ℝM×o\\mathbf\{f\}^\{\\mathcal\{N\}\}\_\{t\}\\in\\mathbb\{R\}^\{M\\times o\}comprises the feature vectors of theMMavailable QNodes, where each node is characterised byoofeatures\. These hardware features include the relative ready time of the QNode and performance parameters such as the average execution duration of gates and corresponding measurement operations, derived from Eq\. \([2](https://arxiv.org/html/2609.27446#S3.E2)\)\. All features are scaled to\[0,1\]\[0,1\]and then mapped to rotation angles in\[0,π\]\[0,\\pi\]before being encoded into the parameterised quantum circuit\. Finally, the overall state space is defined as

𝐒=\{st∣st=\(𝐟tτi,𝐟t𝒩\),∀t∈𝕋\}\.\\mathbf\{S\}=\\\{s\_\{t\}\\mid s\_\{t\}=\(\\mathbf\{f\}^\{\\tau\_\{i\}\}\_\{t\},\\mathbf\{f\}^\{\\mathcal\{N\}\}\_\{t\}\),\\forall t\\in\\mathbb\{T\}\\\}\.\(14\)
Action Space𝐀\\mathbf\{A\}:This discrete space defines the choice of a candidate QNode for allocating the incoming QTask based on the learned policy\. At each decision steptt, the agent selects an actionat∈\{0,1,…,M−1\}a\_\{t\}\\in\\\{0,1,\\dots,M\-1\\\}, whereM=\|𝒩\|M=\|\\mathcal\{N\}\|denotes the total number of available QNodes\. Specifically,at=ja\_\{t\}=jrepresents assigning the task to thejj\-th QNodenj∈𝒩n\_\{j\}\\in\\mathcal\{N\}\. Consequently, the action space is mapped directly to the set of QNodes, expressed as𝐀=𝒩\\mathbf\{A\}=\\mathcal\{N\}\.

Reward Function𝐑\\mathbf\{R\}:The immediate rewardrtr\_\{t\}is formulated to directly align with the optimisation objective of the scheduling strategy\. Upon allocating the QTaskτi\\tau\_\{i\}to thejj\-th QNode at decision steptt, this reward integrates the performance scores of both execution time and waiting time, governed by a tradeoff weighting parameterβ∈\[0,1\]\\beta\\in\[0,1\]\. Formally, the calculation is expressed as

rt=β​ℛe\+\(1−β\)​ℛw\.r\_\{t\}=\\beta\\mathcal\{R\}^\{\\text\{e\}\}\+\(1\-\\beta\)\\mathcal\{R\}^\{\\text\{w\}\}\.\(15\)

## IVThe Proposed QRLQ Framework

### IV\-AOverall QRLQ Architecture

The proposed QRLQ framework introduces a hybrid architecture that integrates Double DQN and Duelling DQN and replaces the dense feature extractor with a PQC to address cost\-delay\-aware resource scheduling\. Double DQN separates action selection from action evaluation using two neural networks with the same structure, which reduces the overestimation bias in DQN and improves training stability\[[23](https://arxiv.org/html/2609.27446#bib.bib23)\]\. The Duelling DQN architecture\[[24](https://arxiv.org/html/2609.27446#bib.bib24)\]separates the state\-value functionV⁡\(s\)V\(s\)from the advantage functionA⁡\(s,a\)A\(s,a\)\. This separation allows the model to estimate both the general value of a scheduling state and the relative importance of each allocation action, thereby facilitating more effective action selection in environments with heterogeneous QTask characteristics\. The framework replaces the dense feature extractor of a classical D3QN with a parameterised quantum circuit\[[25](https://arxiv.org/html/2609.27446#bib.bib25),[14](https://arxiv.org/html/2609.27446#bib.bib14)\]\. As shown in Section[V\-B5](https://arxiv.org/html/2609.27446#S5.SS2.SSS5), this reduces the trainable parameter count from 758 to 214 while maintaining classical scheduling performance\.

Fig\.[4](https://arxiv.org/html/2609.27446#S3.F4)illustrates the high\-level architecture of the proposed QRLQ framework\. First, the environment state space is transformed into add\-dimensional vector with elements normalised to map onto the rotation angles of each qubit within annn\-qubit system\. Then, a trainable PQC processes the encoded state and applies entangling operations across adjacent qubits to transform the encoded feature vectors before measurement\. To support scalable circuit configurations, both the data encoding block and the PQC are formulated with a configurable number of layersll, allowing the architecture to adopt a data re\-uploading scheme\[[26](https://arxiv.org/html/2609.27446#bib.bib26)\]whenl\>1l\>1\. Quantum measurements are then performed to map the quantum state back to classical expectation values\. The output from these quantum measurements serves as the input feature vector for the classical post\-processing layer\. Within this layer, the data flow is separated into two independent computational streams\. One stream estimates the state value, whereas the other evaluates the advantage for each specific scheduling action\. The outputs from both streams are then combined to compute the final action\-value functionQ⁡\(s,a\)Q\(s,a\)for each candidate action\. Based on these estimated values, the policy selects the action with the highest estimated value to allocate the QTask to the selected QNodenjn\_\{j\}from theMMavailable options\.

### IV\-BParameterised Quantum Circuit \(PQC\) Design

Algorithm[1](https://arxiv.org/html/2609.27446#alg1)presents the detailed implementation of this hybrid architecture to compute the action\-value function\. To align with the information processing characteristics of the PQC, the featuressis\_\{i\}from the environment state𝐒\\mathbf\{S\}are normalised intos¯i\\bar\{s\}\_\{i\}to correspond with rotation angles within the interval\[0,π\]\[0,\\pi\], thereby mapping the information onto a hemisphere of the Bloch sphere\[[27](https://arxiv.org/html/2609.27446#bib.bib27)\]\. The normalised data is embedded into the quantum circuit via rotation gatesRxR\_\{x\},RyR\_\{y\}, andRzR\_\{z\}\. Following the encoding layer, the variational layer applies parameterised single\-qubit rotations along theyyandzzaxes to every qubit\[[14](https://arxiv.org/html/2609.27446#bib.bib14)\]\. The corresponding unitary operator is defined as

Uvar​\(𝜽\)=⨂i=0n−1Rz\(i\)​\(θi,z\)​Ry\(i\)​\(θi,y\),\\displaystyle U\_\{\\mathrm\{var\}\}\(\\boldsymbol\{\\theta\}\)=\\bigotimes\_\{i=0\}^\{n\-1\}R\_\{z\}^\{\(i\)\}\(\\theta\_\{i,z\}\)R\_\{y\}^\{\(i\)\}\(\\theta\_\{i,y\}\),\(16\)
where the rotation anglesθi,y\\theta\_\{i,y\}andθi,z\\theta\_\{i,z\}are trainable parameters that determine the local unitary transformations applied to theii\-th qubit\. Consequently, the overall quantum state of the circuit after the variational layer is expressed as

\|ψ′′⟩=Uvar​\(𝜽\)​\|ψ′⟩,\\displaystyle\|\\psi^\{\\prime\\prime\}\\rangle=U\_\{\\mathrm\{var\}\}\(\\boldsymbol\{\\theta\}\)\|\\psi^\{\\prime\}\\rangle,\(17\)
where\|ψ′⟩\|\\psi^\{\\prime\}\\rangleand\|ψ′′⟩\|\\psi^\{\\prime\\prime\}\\rangledenote the global quantum states of the register after the encoding layer and after passing through the variational layer, respectively\.

Although these variational operations provide value function approximation capabilities, they act locally on individual qubits\. To enable the model to capture complex correlations among the features of quantum cloud resources\[[15](https://arxiv.org/html/2609.27446#bib.bib15)\], quantum entanglement is implemented via CZ gates that connect adjacent qubits in a ring topology\. The output quantum state\|Ψo​u​t⟩\|\\Psi\_\{out\}\\rangleis formulated as

\|Ψo​u​t⟩=\(∏i=0n−1C​Z\(i,\(i\+1\)modn\)\)​\|ψ′′⟩,\\displaystyle\|\\Psi\_\{out\}\\rangle=\\left\(\\prod\_\{i=0\}^\{n\-1\}CZ^\{\(i,\(i\+1\)\\bmod n\)\}\\right\)\|\\psi^\{\\prime\\prime\}\\rangle,\(18\)
where the operatorC​Z\(i,\(i\+1\)modn\)CZ^\{\(i,\(i\+1\)\\bmod n\)\}entangles theii\-th control qubit with its adjacent target qubit, forming a ring topology that captures correlations among neighbouring resource features\.

Algorithm 1Proposed QRLQ implementation\.0:State

ss, number of qubits

nn, layers

ll, complete set of trainable parameters

θ=\{θi,y,θi,z,θV,θA\}\\theta=\\\{\\theta\_\{i,y\},\\theta\_\{i,z\},\\theta\_\{V\},\\theta\_\{A\}\\\}
1:Normalise input features

s¯i\\bar\{s\}\_\{i\}for all elements in

ss
2:Initialise quantum state

\|ψ⟩←\|0⟩⊗n\|\\psi\\rangle\\leftarrow\|0\\rangle^\{\\otimes n\}
3:for

k=1,…,lk=1,\\dots,ldo

4:\# data encoding

5:foreach qubit

i=0i=0to

n−1n\-1do

6:Apply

Rz​\(s¯2​n\+i\)​Ry​\(s¯n\+i\)​Rx​\(s¯i\)R\_\{z\}\(\\bar\{s\}\_\{2n\+i\}\)R\_\{y\}\(\\bar\{s\}\_\{n\+i\}\)R\_\{x\}\(\\bar\{s\}\_\{i\}\)to qubit

ii
7:endfor

8:\# parameterised quantum circuit

9:foreach qubit

i=0i=0to

n−1n\-1do

10:Apply

Ry​\(θi,yk\)R\_\{y\}\(\\theta^\{k\}\_\{i,y\}\)and

Rz​\(θi,zk\)R\_\{z\}\(\\theta^\{k\}\_\{i,z\}\)
11:endfor

12:foreach qubit

i=0i=0to

n−2n\-2do

13:Apply entanglement

C​Z​\(i,i\+1\)CZ\(i,i\+1\)
14:endfor

15:Apply

C​Z​\(n−1,0\)CZ\(n\-1,0\)
16:endfor

17:Obtain final quantum state

\|Ψo​u​t⟩\|\\Psi\_\{out\}\\rangle
18:\# measurement

19:

O←\[⟨Z0⟩,…,⟨Zn−1⟩\]O\\leftarrow\[\\langle Z\_\{0\}\\rangle,\\dots,\\langle Z\_\{n\-1\}\\rangle\]
20:\# duelling network

21:

V←fV​\(O,θV\)V\\leftarrow f\_\{V\}\(O,\\theta\_\{V\}\)
22:

A←fA​\(O,θA\)A\\leftarrow f\_\{A\}\(O,\\theta\_\{A\}\)
23:Compute

Q⁡\(s,a,θ\)Q\(s,a,\\theta\)using Eq\. \([21](https://arxiv.org/html/2609.27446#S4.E21)\)

23:

Q⁡\(s,a,θ\)Q\(s,a,\\theta\)for all actions

### IV\-CValue Observation and Duelling Output

To extract classical information from the quantum state transformed by the PQC layers, an expectation value measurement using the Pauli\-Z operator is applied across allnnqubits\. The expectation value on theii\-th qubit is defined as

⟨Oi⟩=⟨Ψo​u​t\|Zi\|Ψo​u​t⟩,\\displaystyle\\langle O\_\{i\}\\rangle=\\langle\\Psi\_\{out\}\|Z\_\{i\}\|\\Psi\_\{out\}\\rangle,\(19\)
whereZiZ\_\{i\}represents the Pauli\-Z operator acting on theii\-th qubit\. This process generates a feature vector expressed as

O=\[⟨O0⟩,⟨O1⟩,…,⟨On−1⟩\]T∈\[−1,1\]n\.\\displaystyle O=\[\\langle O\_\{0\}\\rangle,\\langle O\_\{1\}\\rangle,\\dots,\\langle O\_\{n\-1\}\\rangle\]^\{T\}\\in\[\-1,1\]^\{n\}\.\(20\)
Consequently, the vectorOOis processed through a classical post\-processing layer\. Specifically, the state\-value streamV⁡\(O,θV\)V\(O;\\theta\_\{V\}\)parameterised byθV\\theta\_\{V\}evaluates the overall value of the current state, whereas the advantage streamA⁡\(O,a,θA\)A\(O,a;\\theta\_\{A\}\)parameterised byθA\\theta\_\{A\}estimates the relative advantage of each action\. The action\-value function of the hybrid architecture is formally defined as

Q⁡\(s,a,θ\)=V⁡\(O,θV\)\+\(CLOSE\\displaystyle Q\(s,a;\\theta\)=V\(O;\\theta\_\{V\}\)\+\\Big\(A⁡\(O,a,θA\)\\displaystyle A\(O,a;\\theta\_\{A\}\)\(21\)−1\|𝒜\|∑a′A\(O,a′;θA\)\),\\displaystyle\-\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{a^\{\\prime\}\}A\(O,a^\{\\prime\};\\theta\_\{A\}\)\\Big\),
wherea′a^\{\\prime\}represents a feasible action and\|𝒜\|=\|𝒩\|\|\\mathcal\{A\}\|=\|\\mathcal\{N\}\|denotes the action space size equal to the number of available QNodes within the system\. The notationθ\\thetadenotes the complete set of trainable parameters for the hybrid model, including the classical weightsθV\\theta\_\{V\}andθA\\theta\_\{A\}alongside the quantum rotation anglesθi,y\\theta\_\{i,y\}andθi,z\\theta\_\{i,z\}within the PQC\.

TABLE II:Simulation hyperparameter settings across policies\[[14](https://arxiv.org/html/2609.27446#bib.bib14),[15](https://arxiv.org/html/2609.27446#bib.bib15)\]\.SettingsQRLQQDQND3QNArchitecture1 re\-upload layer16 variational gatesDuelling: 8 neurons10 re\-upload layers160 variational gates1 hidden layer \(LayerNorm\)16 neuronsDuelling: 8 neuronsLearning rateAngles: 0\.001Angles: 0\.001Scaling: 0\.0010\.001Training parameters214170758Shared training settings:Update: Every 10 steps,γ=0\.99\\gamma=0\.99, Batch size: 64, Exploration: 65% episodes,τ=0\.001\\tau=0\.001\.Algorithm 2Proposed QRLQ training workflow\.0:Replay buffer capacity

CC, minibatch size

BB, learning rate

α\\alpha, discount factor

γ\\gamma, soft update coefficient

τ\\tau, update freq

ff, total episodes

EE\.

1:Initialise replay buffer

𝒟←∅\\mathcal\{D\}\\leftarrow\\emptyset, global time step

t←0t\\leftarrow 0
2:Initialise

θo​n​l\\theta\_\{onl\}and

θt​a​r\\theta\_\{tar\}, set

θt​a​r←θo​n​l\\theta\_\{tar\}\\leftarrow\\theta\_\{onl\}
3:foreach episode

e=1,…,Ee=1,\\dots,Edo

4:Decay exploration rate

ϵ\\epsilon
5:

s←env\.reset\(\)s\\leftarrow\\text\{env\.reset\(\)\}
6:foreach time stepdo

7:Compute

Q⁡\(s,a,θo​n​l\)Q\(s,a,\\theta\_\{onl\}\)using Algorithm[1](https://arxiv.org/html/2609.27446#alg1)

8:Select action

a∼ϵ​\-greedy​\(Q\)a\\sim\\epsilon\\text\{\-greedy\}\(Q\)
9:Execute

\(r,s′,d\)←step​\(a\)\(r,s^\{\\prime\},d\)\\leftarrow\\text\{step\}\(a\)
10:Store transition

𝒟←𝒟∪\{\(s,a,r,s′,d\)\}\\mathcal\{D\}\\leftarrow\\mathcal\{D\}\\cup\\\{\(s,a,r,s^\{\\prime\},d\)\\\},

t←t\+1t\\leftarrow t\+1
11:if

tmodf=0t\\bmod f=0and

\|𝒟\|≥B\|\\mathcal\{D\}\|\\geq Bthen

12:Sample minibatch

\(si,ai,ri,si′,di\)\(s\_\{i\},a\_\{i\},r\_\{i\},s^\{\\prime\}\_\{i\},d\_\{i\}\)
13:

a∗←arg⁡maxa′⁡Q⁡\(s′,a′,θo​n​l\)a^\{\*\}\\leftarrow\\arg\\max\_\{a^\{\\prime\}\}Q\(s^\{\\prime\},a^\{\\prime\},\\theta\_\{onl\}\)
14:Compute target

YYusing Eq\. \([22](https://arxiv.org/html/2609.27446#S4.E22)\)

15:Optimise

θo​n​l\\theta\_\{onl\}via Eq\. \([23](https://arxiv.org/html/2609.27446#S4.E23)\)

16:Apply gradient clipping

‖∇‖≤1\\\|\\nabla\\\|\\leq 1
17:Soft update

θt​a​r←τ​θo​n​l\+\(1−τ\)​θt​a​r\\theta\_\{tar\}\\leftarrow\\tau\\theta\_\{onl\}\+\(1\-\\tau\)\\theta\_\{tar\}
18:endif

19:

s←s′s\\leftarrow s^\{\\prime\}
20:if

ddis Truethen

21:break

22:endif

23:endfor

24:endfor

### IV\-DLearning Stabilisation and Training Strategy

The main objective of the QRLQ framework is to approximate the optimal action\-value function to maximise the long\-term cumulative reward and support resource scheduling decisions under dynamic resource conditions\. Following the MDP formulation, the interaction process at each decision stepttgenerates a transition tuple\(st,at,rt,st\+1\)\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\}\), which is stored in an experience replay memory\. The training procedure is summarised in Algorithm[2](https://arxiv.org/html/2609.27446#alg2)\. To optimise the model, the system randomly samples mini\-batches from memory to break temporal correlation and stabilise training\. Based on these data samples, the QRLQ framework extends the double Q\-learning architecture\[[23](https://arxiv.org/html/2609.27446#bib.bib23)\]to reduce the overestimation bias in the DQN algorithm\. The framework uses two structurally identical neural networks comprising an online network parameterised byθo​n​l\\theta\_\{onl\}to select the optimal action in the subsequent state, and a target network parameterised byθt​a​r\\theta\_\{tar\}to evaluate the value of that action\. The temporal difference \(TD\) targetYtY\_\{t\}is formulated as

a∗\\displaystyle a^\{\*\}=arg⁡maxa′⁡Q⁡\(st\+1,a′,θo​n​l\),\\displaystyle=\\arg\\max\_\{a^\{\\prime\}\}Q\(s\_\{t\+1\},a^\{\\prime\};\\theta\_\{onl\}\),\(22\)Yt\\displaystyle Y\_\{t\}=rt\+γ⁡\(1−dt\)​Q​\(st\+1,a∗,θt​a​r\),\\displaystyle=r\_\{t\}\+\\gamma\(1\-d\_\{t\}\)Q\(s\_\{t\+1\},a^\{\*\};\\theta\_\{tar\}\),
wheredt∈\{0,1\}d\_\{t\}\\in\\\{0,1\\\}denotes the terminal flag, which equals 1 ifst\+1s\_\{t\+1\}is the final state of the scheduling sequence and 0 otherwise, thereby disabling the future value term\.

For a batch ofB\{B\}transitions sampled randomly from the experience replay memory, letδi=Q⁡\(si,ai,θo​n​l\)−Yi\\delta\_\{i\}=Q\(s\_\{i\},a\_\{i\};\\theta\_\{onl\}\)~\-~Y\_\{i\}denote the TD error of theii\-th sample\. The loss functionL⁡\(θo​n​l\)L\(\\theta\_\{onl\}\)\[[28](https://arxiv.org/html/2609.27446#bib.bib28)\]is given by

L⁡\(θo​n​l\)=1B​∑i=1B\{12​δi2if​\|δi\|<1\|δi\|−12if​\|δi\|≥1\.\\displaystyle L\(\\theta\_\{onl\}\)=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\begin\{cases\}\\frac\{1\}\{2\}\\delta\_\{i\}^\{2\}&\\text\{if \}\|\\delta\_\{i\}\|<1\\\\ \|\\delta\_\{i\}\|\-\\frac\{1\}\{2\}&\\text\{if \}\|\\delta\_\{i\}\|\\geq 1\.\\end\{cases\}\(23\)
To ensure stability during the optimisation process, gradient clipping is applied such that‖∇θo​n​lL‖≤1\.0\|\|\\nabla\_\{\\theta\_\{onl\}\}L\|\|\\leq 1\.0\. After gradient clipping, all classical and quantum parameters withinθo​n​l\\theta\_\{onl\}are updated using a gradient\-based optimisation method with a learning rateα\\alpha, which is expressed as

θo​n​l←θo​n​l−α​∇θo​n​lL\.\\displaystyle\\theta\_\{onl\}\\leftarrow\\theta\_\{onl\}\-\\alpha\\nabla\_\{\\theta\_\{onl\}\}L\.\(24\)
Finally, a soft\-target update strategy is implemented to incrementally update the target network’s weights\. This update rule is formulated as

θt​a​r←τ​θo​n​l\+\(1−τ\)​θt​a​r,\\displaystyle\\theta\_\{tar\}\\leftarrow\\tau\\theta\_\{onl\}\+\(1\-\\tau\)\\theta\_\{tar\},\(25\)
whereτ≪1\\tau\\ll 1denotes the soft update coefficient\. This update reduces unstable Q\-value estimates and helps stabilise the training process of the QRLQ framework\.

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig05_training_reward.png)Fig\. 5:Training curves of the mean reward\.![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig06a_execution_time_episodes.png)\(a\)Execution time across evaluation episodes\.
![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig06b_execution_time_average.png)\(b\)Average task execution time comparison\.
![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig06c_waiting_time_episodes.png)\(c\)Waiting time across evaluation episodes\.
![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig06d_waiting_time_average.png)\(d\)Average task waiting time comparison\.

Fig\. 6:Performance comparison of execution time and waiting time across policies\. Subfigures \(a\) and \(c\) show the variation across evaluation episodes, while \(b\) and \(d\) report the corresponding average values with standard deviation\.

## VPerformance Evaluation

### V\-ASimulation Setup

We set up a heterogeneous quantum cloud simulation environment using the QSimPy simulation tool\[[29](https://arxiv.org/html/2609.27446#bib.bib29)\]\. The quantum computing tasks are generated from the MQT Bench library\[[12](https://arxiv.org/html/2609.27446#bib.bib12)\], including more than 20 standard algorithms with abstract circuit depths ranging from 3 to 30 layers and requiring 2 to 27 measurement qubits\. The task arrival times follow a Poisson distribution to model stochastic job submissions observed in practical quantum cloud resource environments\. The simulation cluster is constructed using Qiskit fake backend instances that emulate representative IBM quantum devices\. The configuration includes two 27\-qubit processors representingibm\_sydneyandibm\_kolkataalongside three 127\-qubit processors representingibm\_sherbrooke,ibm\_washington, andibm\_brisbane\. The QRLQ framework is implemented using the TorchQuantum library\[[30](https://arxiv.org/html/2609.27446#bib.bib30)\]to enable the training of a PQC on classical hardware\. Simulating an idealised environment lets us isolate algorithmic feasibility from hardware\-induced errors\[[15](https://arxiv.org/html/2609.27446#bib.bib15),[20](https://arxiv.org/html/2609.27446#bib.bib20)\]\. All simulations were conducted on an Apple M1 Pro–based system with a 10\-core CPU, 16\-core GPU, 16\-core neural engine, and 16 GB unified memory\.

To evaluate the QRLQ framework, we establish comprehensive baselines that encompass classical scheduling heuristics\[[13](https://arxiv.org/html/2609.27446#bib.bib13),[5](https://arxiv.org/html/2609.27446#bib.bib5),[6](https://arxiv.org/html/2609.27446#bib.bib6)\]and representative classical and quantum DRL approaches, as detailed below\.

- •Greedy Earliest Availability \(GEA\)prioritises the earliest available resources\.
- •Greedy Fastest Duration \(GFD\)prioritises hardware with the fastest average gate execution duration\.
- •Cyclic Round Robin \(CRR\)employs a cyclic approach for allocation\.
- •Random Uniform Selection \(RUS\)performs random allocation of QTasks\.
- •Duelling Double DQN \(D3QN\)serves as a standard representative for classical DRL methods\.
- •Quantum DQN \(QDQN\)serves as a direct baseline to evaluate the proposed hybrid architecture\.

Table[II](https://arxiv.org/html/2609.27446#S4.T2)summarises the hyperparameter settings across the evaluated policies\. All three models use the same set of core reinforcement learning hyperparameters\. Although the learning rate is kept constant, the QDQN model requires an additional scaling parameter to map the raw expectation values produced by the quantum circuit from the interval\[−1,1\]\[\-1,1\]to approximate the target Q\-value range required for action selection\[[14](https://arxiv.org/html/2609.27446#bib.bib14)\]\.

### V\-BPerformance Study

#### V\-B1QRLQ Training Performance and Efficiency

We setβ=0\.5\\beta=0\.5for all trained policies to assign equal importance to execution time and waiting time in the reward function\. This configuration evaluates the proposed framework under a balanced objective\. Fig\.[5](https://arxiv.org/html/2609.27446#S4.F5)shows the mean reward over 1000 training episodes, averaged across five independent random seeds\. The shaded regions represent the corresponding 95% confidence intervals\. During the first 200 episodes, all three reinforcement learning methods rapidly improved their rewards, reflecting effective policy learning in the early training stage\. Among them, QRLQ consistently achieved higher mean rewards while maintaining relatively narrower confidence intervals than the baseline methods\. After this stage, the QDQN model continued to improve from a lower reward level and gradually approached the performance of D3QN toward the end of training, although it exhibited larger confidence intervals throughout most of the training process\. By comparison, the D3QN model continued to improve at a slower rate after its initial growth and eventually plateaued at a lower reward level than QRLQ\. In contrast, the QRLQ framework steadily increased its average reward throughout training and stabilised at approximately 0\.775 during the final 100 episodes, with only minor fluctuations during the final stage of training\. Overall, the QRLQ framework achieved the highest mean reward and exhibited narrower confidence intervals than the baseline methods during the final 100 training episodes\.

#### V\-B2Execution and Waiting Time Performance

We evaluated the QRLQ policy trained with random seed 21 on 21,000 independent QTasks over 100 evaluation episodes, similarly for the learning\-based baselines\. These QTasks were not used during training, allowing us to evaluate generalisation on unseen workloads\. Fig\.[6a](https://arxiv.org/html/2609.27446#S4.F6.sf1)shows the average execution time across evaluation episodes\. The execution times of QRLQ remained within a relatively narrow range across the evaluated episodes\. This narrow range indicates that QRLQ produced consistent execution\-time performance across different evaluation episodes\. QRLQ was second only to GFD, which mostly ranged from 1\.6 to 1\.9 seconds, and outperformed the remaining heuristic algorithms, including GEA, CRR, and RUS\. Fig\.[6b](https://arxiv.org/html/2609.27446#S4.F6.sf2)shows the overall average execution time, with error bars indicating the standard deviation\. The average results follow the same trend, with QRLQ outperforming all evaluated heuristic strategies except GFD\. Compared with the learning\-based baselines, the QRLQ framework achieved execution times comparable to those of the QDQN and D3QN models\. It reduced execution time by 1\.46% compared with QDQN and by 2\.70% compared with D3QN, while using substantially fewer trainable parameters\. Although GFD achieved the lowest execution time by selecting the fastest available QPU, this greedy selection can concentrate tasks on a small subset of devices, creating longer queues for subsequent tasks\.

Figs\.[6c](https://arxiv.org/html/2609.27446#S4.F6.sf3)and[6d](https://arxiv.org/html/2609.27446#S4.F6.sf4)present the variation in average waiting time per episode and the overall average across 100 evaluation episodes, respectively\. As shown in Fig\.[6c](https://arxiv.org/html/2609.27446#S4.F6.sf3), the local greedy objective of GFD led to queue congestion, resulting in high waiting times that mostly ranged from 148 to 210 seconds\. Although the GEA strategy initially reduced waiting times by prioritising early availability, its limited look\-ahead capability did not fully account for heterogeneous task complexities and varying shot requirements, leading to less effective global scheduling\. The proposed QRLQ framework reduced these bottlenecks by considering both execution time and waiting time\. As shown in Fig\.[6d](https://arxiv.org/html/2609.27446#S4.F6.sf4), QRLQ maintained a waiting time consistently under 35 seconds, outperforming all heuristic baselines, including GEA\. Compared with the learning\-based baselines, QRLQ yielded a lower average waiting time in the evaluated setting\. While the D3QN and QDQN models also avoided heavy congestion, QRLQ achieved a moderate reduction in waiting times with smaller variance, suggesting improved stability of its hybrid scheduling policy\.

TABLE III:Average cost and delay per QTask with standard error across policies\.PolicyAverage cost \($\)Average delay \(s\)GEA3\.90±\\pm0\.0336\.09±\\pm0\.15GFD2\.83±\\pm0\.03170\.81±\\pm0\.69CRR4\.14±\\pm0\.0440\.86±\\pm0\.22RUS4\.14±\\pm0\.0442\.34±\\pm0\.23QDQN3\.75±\\pm0\.0336\.92±\\pm0\.30D3QN3\.80±\\pm0\.0432\.94±\\pm0\.14QRLQ3\.70±\\pm0\.0329\.99±\\pm0\.14
#### V\-B3Cost–Delay Tradeoff Analysis

Based on the preceding analysis of execution time and waiting time, this section examines their impact on cost and delay\. To calculate cost, we applied the IBM Quantum on\-demand price of $96 per minute, which equates to $1\.60 per second of actual execution time on physical quantum hardware\. Table[III](https://arxiv.org/html/2609.27446#S5.T3)reports the average cost and delay per QTask\. Consistent with the execution time results, the GFD strategy recorded the lowest cost at $2\.83\. However, Table[III](https://arxiv.org/html/2609.27446#S5.T3)also shows a limitation of this strategy in terms of delay\. Because of the long waiting times discussed previously, GFD had the highest average delay of 170\.81 seconds, showing that a low execution cost does not necessarily lead to a low completion time\. In contrast, QRLQ achieved the lowest mean delay of 29\.99 seconds while incurring the second\-lowest mean cost of $3\.70 per QTask among the evaluated policies\. Its delay was lower than that of all heuristic approaches, including the GEA strategy that explicitly prioritises the earliest available resources\. In addition, QRLQ achieved both lower cost and shorter delay than the learning\-based baselines, including QDQN at $3\.75 and 36\.92 seconds and D3QN at $3\.80 and 32\.94 seconds\.

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig07_execution_fidelity_average.png)Fig\. 7:Average execution fidelity across evaluated policies\.
#### V\-B4Execution Fidelity Performance Analysis

Execution fidelity remains an important factor in the NISQ era\[[5](https://arxiv.org/html/2609.27446#bib.bib5)\]\. To further evaluate the proposed strategy, an additional baseline denoted as Greedy Lowest Error \(GLE\) was added to select the QNodes with the lowest average gate error rate within the system\. Fig\.[7](https://arxiv.org/html/2609.27446#S5.F7)shows that both the speed\-prioritised GFD strategy and the error\-prioritised GLE strategy achieved the highest execution fidelity, with an identical value of 0\.624\. This result suggests that in the evaluated quantum cloud setting, scheduling quantum computing tasks on higher\-quality QNodes can achieve high execution fidelity while also maintaining a short execution time\. In comparison, QRLQ achieved a mean execution fidelity of 0\.612, which was approximately 1\.9% below the highest observed value of 0\.624 and remained consistently higher than the traditional D3QN and baseline QDQN models, which recorded fidelities of 0\.607 and 0\.593, respectively\. Furthermore, the proposed approach outperformed heuristic strategies that do not explicitly consider hardware quality, such as GEA, CRR, and RUS\. Across the evaluated policies, QRLQ achieved the lowest mean delay, a lower mean cost than GEA, CRR, and RUS, and a mean execution fidelity close to the highest observed value\.

#### V\-B5Computational Complexity and Efficiency Analysis

We evaluated the computational efficiency of the proposed models by analysing the time complexity with respect to several key parameters\. Specifically,nndenotes the number of qubits, andllrepresents the number of quantum layers\. The parametersaaandbbindicate the number of variational gates and data embedding gates operating on each qubit within a single layer\[[15](https://arxiv.org/html/2609.27446#bib.bib15)\]\. Finally,PcP\_\{c\}represents the total number of trainable parameters of the classical neural network in the hybrid architecture\. In this work, we configurednn,aa,bb,ll, andPcP\_\{c\}to be 8, 2, 3, 1, and 198, respectively\.

When analysing the time complexity of execution on quantum hardware, the computational cost is determined by both quantum and classical components\. Within each quantum layer, data embedding gates and variational gates operate independently on each qubit, whereas entanglement gates require sequential operation\. This structure results in a complexity ofO⁡\(a\+b\+n\)O\(a\+b\+n\)per quantum layer andO⁡\(l×\(a\+b\+n\)\)O\(l\\times\(a\+b\+n\)\)acrossllquantum layers\. The output from the quantum circuit then passes through a classical neural network, so the overall time complexity is approximated asO⁡\(l×\(a\+b\+n\)\+Pc\)O\(l\\times\(a\+b\+n\)\+P\_\{c\}\)\. When simulating the algorithm on a classical computer, the computational cost increases exponentially because every quantum operation updates the state vector in a Hilbert space of size2n2^\{n\}\. The total number of quantum gates in the circuit is calculated asl×\(\(a\+b\)×n\+n\)l\\times\\left\(\(a\+b\)\\times n\+n\\right\)\. Therefore, the simulation time complexity for the quantum component alone is expressed asO⁡\(l×\(\(a\+b\)×n\+n\)×2n\)O\(l\\times\\left\(\(a\+b\)\\times n\+n\\right\)\\times 2^\{n\}\)\. The classical computation associated withPcP\_\{c\}is then added, yielding an overall simulation time complexity ofO⁡\(l×\(\(a\+b\)×n\+n\)×2n\+Pc\)O\(l\\times\\left\(\(a\+b\)\\times n\+n\\right\)\\times 2^\{n\}\+P\_\{c\}\)\.

Regarding memory requirements during execution on quantum hardware, the model requires memory allocation for the trainable quantum rotation angles, which yields a space complexity ofO⁡\(l×a×n\)O\(l\\times a\\times n\)\. When combining this requirement with the trainable weights of the classical network, the overall space complexity is given byO⁡\(l×a×n\+Pc\)O\(l\\times a\\times n\+P\_\{c\}\)\. In classical simulation, the storage space required on platforms such as TorchQuantum depends primarily on the need to maintain a complex state vector of size2n2^\{n\}alongside the parameter countPcP\_\{c\}of the hybrid architecture\. Consequently, the overall classical simulation space complexity isO⁡\(2n\+Pc\)O\(2^\{n\}\+P\_\{c\}\)\.

Based on the complexity analysis, the proposed QRLQ architecture requires 16 trainable quantum parameters\. When combined with the 198 parameters of the classical network, the total parameter count reaches 214\. Compared to the classical D3QN model comprising 758 parameters, the proposed hybrid framework uses fewer trainable parameters than the classical D3QN baseline\. Compared with the QDQN baseline using a 10\-layer, 8\-qubit circuit, QRLQ employs a shallower quantum circuit, resulting in fewer quantum operations than the QDQN baseline\. Finally, regarding classical simulation overhead, although maintaining the2n2^\{n\}state vector typically requires exponential resources, our implementation uses only 8 qubits, yielding a state vector with 256 amplitudes\. With the number of qubits fixed, the simulation cost depends mainly onll\. Therefore, compared with the QDQN baseline, the shallower quantum circuit in QRLQ reduces the number of quantum operations to be simulated, helping balance quantum feature extraction and training time on current quantum simulation platforms\. In our simulations, QRLQ required approximately 10\.5 hours of training on both CPU and GPU, QDQN required approximately 11\.25 hours on both CPU and GPU, and D3QN required approximately 10\.5 hours on the GPU\.

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig08_noise_waiting_time.png)

![Refer to caption](https://arxiv.org/html/2609.27446v1/figures/fig08_noise_execution_time.png)

Fig\. 8:Comparison of waiting time and execution time under different noise levels, with horizontal lines indicating the mean values of the classical baselines\.
#### V\-B6Impact of Noise Model and Discussion

To evaluate the scheduling performance of the trained policies under simulated noise conditions, we introduced parameter perturbations during the evaluation phase by applying random Gaussian perturbations to the variational parameters of the PQCs\[[31](https://arxiv.org/html/2609.27446#bib.bib31)\]\. Specifically, each trained rotation angleθ\\thetawas perturbed such thatθnoisy=θ\+δ​θ\\theta\_\{\\mathrm\{noisy\}\}=\\theta\+\\delta\\theta, whereδ​θ\\delta\\thetawas sampled from a Gaussian distribution𝒩⁡\(0,σ2\)\\mathcal\{N\}\(0,\\sigma^\{2\}\)withσ\\sigmadenoting the perturbation level\.

Fig\.[8](https://arxiv.org/html/2609.27446#S5.F8)shows that the scheduling performance of both models changed as the perturbation level increased\. QRLQ exhibited relatively small changes in waiting time across the evaluated perturbation levels, whereas QDQN showed a substantially larger variation in waiting time\. The execution time of both models remained within a relatively narrow range, although QRLQ exhibited a larger variation than QDQN\. In terms of waiting time, QRLQ remained below most of the classical baseline means across the evaluated perturbation levels, whereas QDQN exceeded some of the classical baseline means as the perturbation level increased\. For execution time, the results of both quantum models remained within the range of the classical baseline means across the evaluated perturbation levels, although QRLQ increased gradually with the perturbation level\. QDQN uses a larger PQC with 160 variational gates compared with 16 in QRLQ, which may contribute to the greater sensitivity in waiting time observed under parameter perturbations\. The results suggest that QRLQ maintained more consistent waiting\-time performance than QDQN as the perturbation level increased, and the shallower PQC used by QRLQ may be less sensitive to the evaluated parameter perturbations\.

The current simulations use five QNodes with an 8\-qubit shallow\-depth PQC\. For larger action spaces, the architecture can be extended by increasing the number of qubits, while the existing data re\-uploading mechanism allows additional input features to be encoded when needed\. The classical duelling heads can also be expanded to accommodate additional scheduling actions\. The appropriate quantum circuit size for larger action spaces remains to be evaluated\.

## VIConclusion and Future Work

We have developed QRLQ, a cost\-delay\-aware quantum cloud scheduling framework for heterogeneous QaaS environments operating under a uniform time\-based pricing model\. By integrating PQCs into a D3QN, QRLQ provides a parameter\-efficient reinforcement learning approach to jointly optimise execution cost and delay\. Simulation results demonstrated that QRLQ achieved a 5–11% lower mean cost relative to availability\-based and rotation\-based heuristics and reduced mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively, while maintaining execution fidelity within 2% of a fidelity\-greedy policy\. Compared with the classical DRL baseline, QRLQ achieved comparable scheduling performance using approximately 72% fewer trainable parameters, suggesting the feasibility of parameter\-efficient quantum\-classical models for quantum cloud resource orchestration\. Despite these encouraging results, several directions remain for future research\. First, the current framework assumes a single quantum task per QPU, and extending QRLQ to support quantum multiprogramming would enable concurrent workload execution\. Second, future work will investigate the scalability of QRLQ under larger action spaces by studying appropriate qubit counts and circuit depths\. Third, integrating circuit cutting and distributed quantum execution could improve scalability beyond individual device limitations\. Finally, extending the scheduling framework to iterative hybrid quantum\-classical workloads, such as VQE and QAOA, and validating the proposed approach on physical quantum hardware clusters would further demonstrate its practicality under realistic NISQ conditions\.

## References

- \[1\]H\. T\. Nguyen, P\. Krishnan, D\. Krishnaswamy, M\. Usman, and R\. Buyya, “Quantum cloud computing: a review, open problems, and future directions,” 2024, DOI: 10\.48550/arXiv\.2404\.11420\.
- \[2\]H\. T\. Nguyen, M\. Usman, and R\. Buyya, “DRLQ: A Deep Reinforcement Learning\-based Task Placement for Quantum Cloud Computing,” in*Proceedings of the IEEE 17th International Conference on Cloud Computing \(CLOUD\)*, Shenzhen, China, Jul\. 2024\.
- \[3\]G\. S\. Ravi, K\. N\. Smith, P\. Murali, and F\. T\. Chong, “Adaptive job and resource management for the growing quantum cloud,” in*Proceedings of the IEEE International Conference on Quantum Computing and Engineering \(QCE\)*, Broomfield, CO, USA, Oct\. 2021\.
- \[4\]A\. Wack, H\. Paik, A\. Javadi\-Abhari, P\. Jurcevic, I\. Faro, J\. M\. Gambetta, and B\. R\. Johnson, “Quality, speed, and scale: three key attributes to measure the performance of near\-term quantum computers,” 2024, DOI: 10\.48550/arXiv\.2110\.14108\.
- \[5\]H\. T\. Nguyen, M\. Usman, and R\. Buyya, “QFOR: A Fidelity\-aware Orchestrator for Quantum Computing Environments using Deep Reinforcement Learning,”*ACM Transactions on Quantum Computing*, Mar\. 2026\.
- \[6\]W\. Luo, J\. Zhao, T\. Zhan, and Q\. Guan, “Adaptive job scheduling in quantum clouds using reinforcement learning,” in*Proceedings of the International Conference on Parallel Processing \(ICPP\)*, New York, NY, USA, Sep\. 2025\.
- \[7\]J\. Preskill, “Quantum computing in the nisq era and beyond,”*Quantum*, vol\. 2, p\. 79, Aug\. 2018\.
- \[8\]F\. Bova, A\. Goldfarb, and R\. G\. Melko, “Commercial applications of quantum computing,”*EPJ Quantum Technology*, vol\. 8, no\. 1, p\. 2, Jan\. 2021\.
- \[9\]H\. T\. Nguyen, M\. Usman, and R\. Buyya, “QFaaS: A Serverless Function\-as\-a\-Service framework for Quantum computing,”*Future Generation Computer Systems*, vol\. 154, pp\. 281–300, May 2024\.
- \[10\]S\. Ghosh and P\. Kuila, “Cost\-efficient quantum cloud task offloading with quantum\-inspired particle swarm optimization,”*Future Generation Computer Systems*, vol\. 175, p\. 108095, Feb\. 2026\.
- \[11\]E\. Giortamis, F\. Romao, N\. Tornow, D\. Lugovoy, and P\. Bhatotia, “Qonductor: A cloud orchestrator for quantum computing,” in*Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis \(SC\)*, New York, NY, USA, Nov\. 2025\.
- \[12\]N\. Quetschlich, L\. Burgholzer, and R\. Wille, “MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing,”*Quantum*, vol\. 7, p\. 1062, Jul\. 2023\.
- \[13\]T\. Li and Z\. Zhao, “Moirai: Optimizing quantum serverless function orchestration via device allocation and circuit deployment,” in*Proceedings of the IEEE International Conference on Web Services \(ICWS\)*, Shenzhen, China, Jul\. 2024\.
- \[14\]A\. Skolik, S\. Jerbi, and V\. Dunjko, “Quantum agents in the gym: a variational quantum algorithm for deep q\-learning,”*Quantum*, vol\. 6, p\. 720, May 2022\.
- \[15\]S\. Dai, N\. Saurabh, Q\. Wang, J\. Nian, S\. Kan, Y\. Mao, and L\. Cheng, “Quantum reinforcement learning for qos\-aware real\-time job scheduling in cloud systems,”*IEEE Systems Journal*, vol\. 19, no\. 2, pp\. 471–482, Jun\. 2025\.
- \[16\]S\. C\. Prabhashana, D\. V\. Huynh, H\. Jung, B\. Canberk, S\. L\. Cotton, and T\. Q\. Duong, “Quantum deep reinforcement learning for urllc satellite\-air\-ground integrated networks with digital twin applications,”*IEEE Internet of Things Journal*, vol\. 13, no\. 3, pp\. 4230–4246, Feb\. 2026\.
- \[17\]J\. R\. McClean, S\. Boixo, V\. N\. Smelyanskiy, R\. Babbush, and H\. Neven, “Barren plateaus in quantum neural network training landscapes,”*Nature Communications*, vol\. 9, no\. 1, p\. 4812, Nov\. 2018\.
- \[18\]J\. Li, Y\. Song, Y\. Liu, J\. Pan, L\. Yang, T\. Humble, and W\. Jiang, “Qusplit: achieving both high fidelity and throughput via job splitting on noisy quantum computers,”*Quantum Machine Intelligence*, vol\. 7, no\. 2, p\. 99, Oct\. 2025\.
- \[19\]R\. Kaewpuang, M\. Xu, D\. Niyato, H\. Yu, Z\. Xiong, and J\. Kang, “Stochastic qubit resource allocation for quantum cloud computing,” in*Proceedings of the IEEE/IFIP Network Operations and Management Symposium \(NOMS\)*, Miami, FL, USA, May 2023\.
- \[20\]X\. Wei, X\. Gao, K\. Ye, C\.\-Z\. Xu, and Y\. Wang, “A quantum reinforcement learning approach for joint resource allocation and task offloading in mobile edge computing,”*IEEE Transactions on Mobile Computing*, vol\. 24, no\. 4, pp\. 2580–2593, Apr\. 2025\.
- \[21\]G\. S\. Ravi, K\. N\. Smith, P\. Gokhale, and F\. T\. Chong, “Quantum computing in the cloud: Analyzing job and machine characteristics,” in*Proceedings of the IEEE International Symposium on Workload Characterization \(IISWC\)*, Storrs, CT, USA, Nov\. 2021\.
- \[22\]Y\. Ohkura, T\. Satoh, and R\. Van Meter, “Simultaneous execution of quantum circuits on current and near\-future nisq systems,”*IEEE Transactions on Quantum Engineering*, vol\. 3, pp\. 1–10, Apr\. 2022\.
- \[23\]H\. v\. Hasselt, A\. Guez, and D\. Silver, “Deep reinforcement learning with double q\-learning,” in*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, Phoenix, Arizona, Feb\. 2016\.
- \[24\]Z\. Wang, T\. Schaul, M\. Hessel, H\. Hasselt, M\. Lanctot, and N\. Freitas, “Dueling network architectures for deep reinforcement learning,” in*Proceedings of the International Conference on Machine Learning \(ICML\)*, New York, NY, USA, Jun\. 2016\.
- \[25\]S\. Jerbi, C\. Gyurik, S\. C\. Marshall, H\. J\. Briegel, and V\. Dunjko, “Parametrized quantum policies for reinforcement learning,” in*Proceedings of the 35th International Conference on Neural Information Processing Systems*, Red Hook, NY, USA, Dec\. 2021\.
- \[26\]A\. Pérez\-Salinas, A\. Cervera\-Lierta, E\. Gil\-Fuster, and J\. I\. Latorre, “Data re\-uploading for a universal quantum classifier,”*Quantum*, vol\. 4, p\. 226, Feb\. 2020\.
- \[27\]J\. B\. Costello, S\. D\. O’Hara, Q\. Wu, D\. C\. Valovcin, L\. N\. Pfeiffer, K\. W\. West, and M\. S\. Sherwin, “Reconstruction of bloch wavefunctions of holes in a semiconductor,”*Nature*, vol\. 599, no\. 7883, pp\. 57–61, Nov\. 2021\.
- \[28\]V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski, S\. Petersen, C\. Beattie, A\. Sadik, I\. Antonoglou, H\. King, D\. Kumaran, D\. Wierstra, S\. Legg, and D\. Hassabis, “Human\-level control through deep reinforcement learning,”*Nature*, vol\. 518, no\. 7540, pp\. 529–533, Feb\. 2015\.
- \[29\]H\. T\. Nguyen, M\. Usman, and R\. Buyya, “QSimPy: A learning\-centric simulation framework for quantum cloud resource management,” in*Quantum Computing*, R\. Buyya and S\. S\. Gill, Eds\. Elsevier, 2025, pp\. 165–183\.
- \[30\]H\. Wang, Y\. Ding, J\. Gu, Y\. Lin, D\. Z\. Pan, F\. T\. Chong, and S\. Han, “Quantumnas: Noise\-adaptive search for robust quantum circuits,” in*Proceedings of the IEEE International Symposium on High\-Performance Computer Architecture \(HPCA\)*, Seoul, South Korea, Apr\. 2022\.
- \[31\]A\. Skolik, S\. Mangini, T\. Bäck, C\. Macchiavello, and V\. Dunjko, “Robustness of quantum reinforcement learning under hardware errors,”*EPJ Quantum Technology*, vol\. 10, no\. 1, p\. 8, Feb\. 2023\.