When does deep reinforcement learning pay off in HVAC supervisory control? Four small Modelica case studies trained directly on FMUs

arXiv cs.LG Papers

Summary

该论文通过四个基于Modelica的可复现办公楼案例研究,评估深度强化学习在HVAC监督控制中的实际收益,发现无储能场景下DRL相对良好规则基线的边际极小(0.18%-1.3%),而在储能场景中优势更明显但仍远未达到完美预见最优。

arXiv:2611.22144v1 Announce Type: new Abstract: Deep reinforcement learning (DRL) is often proposed for the supervisory control of heating, ventilation and air-conditioning (HVAC) systems. The reported gains are, however, difficult to interpret, because the baselines are often weak and the attainable optimum is seldom known. This paper examines four small, reproducible case studies based on an office building in Brisbane, Australia: pre-cooling under a time-of-use tariff; chiller staging with condenser water reset; chilled water supply temperature reset with a wet cooling coil; and chilled water storage under time-of-use prices and a demand charge. The plants were modelled with the Modelica Buildings library, and in three of the cases a soft actor-critic agent was trained directly on the exported functional mock-up unit (FMU). Each agent was compared with tuned rule-based controllers and, where practicable, with a near-optimal reference from hourly exhaustive search or dynamic programming with perfect foresight. In the two cases without storage the agent reached the near-optimal reference, but the margin above a well-chosen rule was small: 1.3% for condenser water reset and 0.18% for chilled water reset. In the latter case two common reset rules consumed more energy than a fixed set point. With a chilled water storage tank the margin was considerably larger. The agent outperformed both rules (-7.8%, compared with -3.9% and -6.3%) and adopted midday charging during the solar off-peak period, yet it obtained only about 45% of the saving achieved by the perfect-foresight schedule and did little to limit the demand peak. In the pre-cooling case a hand-tuned rule remained superior (-12.0% compared with -5.6%). These results suggest that studies of DRL for HVAC control should state the margin between the best rule and a near-optimal reference. Some practical difficulties encountered in training on co-simulation FMUs are also reported.
Original Article
View Cached Full Text

Cached at: 11/19/26, 03:37 AM

# When does deep reinforcement learning pay off in HVAC supervisory control? Four small Modelica case studies trained directly on FMUs
Source: [https://arxiv.org/html/2611.22144](https://arxiv.org/html/2611.22144)
Journal:arXivBingnan ZhaoEmail:[bingnan\.zhao@hdr\.qut\.edu\.au](mailto:[email protected])Corresponding author:Corresponding author\.Affiliation:Signal Processing, Artificial Intelligence and Vision Technologies \(SAIVT\), Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, AustraliaAffiliation:International Laboratory for Air Quality and Health, Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, AustraliaHarshala GammulleAffiliation:Signal Processing, Artificial Intelligence and Vision Technologies \(SAIVT\), Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, AustraliaTharindu FernandoAffiliation:Signal Processing, Artificial Intelligence and Vision Technologies \(SAIVT\), Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia

###### Abstract

Deep reinforcement learning \(DRL\) has frequently been proposed for the supervisory control of heating, ventilation and air\-conditioning \(HVAC\) systems\. The gains reported in the literature are, however, difficult to interpret, because the baselines are often weak and the attainable optimum is seldom known\. This paper examines four small and fully reproducible case studies based on an office building in Brisbane, Australia: pre\-cooling under a time\-of\-use tariff; chiller staging with condenser water reset; chilled water supply temperature reset with a wet cooling coil; and chilled water storage under time\-of\-use prices and a demand charge\. The plants were modelled with the Modelica Buildings library, and in three of the cases a soft actor\-critic agent was trained directly on the exported functional mock\-up unit \(FMU\)\. Each agent was compared with tuned rule\-based controllers and, where this was practicable, with a near\-optimal reference obtained either by hourly exhaustive search or by dynamic programming with perfect foresight\.

In the two cases without storage the agent reached the near\-optimal reference, but the margin above a well\-chosen rule was small, namely 1\.3 % for condenser water reset and 0\.18 % for chilled water reset\. In the latter case two common reset rules consumed more energy than a fixed set point\. With a chilled water storage tank the margin was considerably larger\. The agent outperformed both rules \(−7\.8%\-7\.8\\,\\%weekly cost, compared with−3\.9%\-3\.9\\,\\%and−6\.3%\-6\.3\\,\\%\) and adopted midday charging during the solar off\-peak period, yet it obtained only about 45 % of the saving achieved by the perfect\-foresight schedule and did little to limit the demand peak\. In the pre\-cooling case a hand\-tuned rule remained superior \(−12\.0%\-12\.0\\,\\%compared with−5\.6%\-5\.6\\,\\%\)\. These results suggest that studies of DRL for HVAC control should state the margin between the best rule and a near\-optimal reference\. Some practical difficulties encountered in training on co\-simulation FMUs are also reported\. The models, trained agents and scripts are openly available\.

###### Keywords:

Deep reinforcement learning , HVAC supervisory control , Chiller plant , Chilled water reset , Thermal energy storage , Modelica , Functional mock\-up unit

††highlights:Four open Modelica HVAC control problems with tuned rules and near\-optimal references Soft actor\-critic agents were trained directly on co\-simulation FMUs Without storage, a well\-chosen rule left a margin of only0\.2–1\.3 %With chilled water storage, DRL outperformed both rules but not the optimum Practical difficulties of training on co\-simulation FMUs are reported## Nomenclature

AbbreviationsAHUair handling unitHVACheating, ventilation and air\-conditioningCHWchilled waterMPCmodel predictive controlCOPcoefficient of performanceRCresistance–capacitance \(model\)DRLdeep reinforcement learningRLreinforcement learningFMIFunctional Mock\-up InterfaceSACsoft actor\-criticFMUfunctional mock\-up unitVAVvariable air volumeSymbolsata\_\{t\}action in hourttrtr\_\{t\}reward in hourttctc\_\{t\}cost incurred in hourtt\(AUD or kWh\)ssreward scaling constantctrefc\_\{t\}^\{\\mathrm\{ref\}\}cost of the reference controller in hourtt
## 1Introduction

Supervisory control of HVAC systems determines set points and operating modes at intervals of minutes to hours, while local feedback loops track these set points\. Reinforcement learning \(RL\) is attractive for this task because a controller can be learned from a simulator without an explicit optimisation model, and a considerable body of work has explored the idea\[[Wei et al\., 2017](https://arxiv.org/html/2611.22144#bib.bib6),[Zhang et al\., 2019](https://arxiv.org/html/2611.22144#bib.bib25),[Wang and Hong, 2020](https://arxiv.org/html/2611.22144#bib.bib5),[Yu et al\., 2021](https://arxiv.org/html/2611.22144#bib.bib12)\]\. Two features of this literature make the reported results difficult to compare\. First, the baseline is often a fixed set point rather than the kind of rule that an experienced engineer would apply, such as wet\-bulb tracking of the condenser water temperature, outdoor\-air reset, or the sequences of ASHRAE Guideline 36\[[ASHRAE, 2021](https://arxiv.org/html/2611.22144#bib.bib10)\]\. Second, the optimum is rarely known, so it is not clear whether a reported saving is large or small relative to what was achievable\. Benchmarking frameworks such as BOPTEST\[[Blum et al\., 2021](https://arxiv.org/html/2611.22144#bib.bib4)\]and its Gym interface\[[Arroyo and others, 2021](https://arxiv.org/html/2611.22144#bib.bib18)\]go a long way towards resolving the first difficulty for a defined set of test cases\.

The present paper approaches the problem from another direction\. It considers four deliberately small control problems, chosen so that a near\-optimal reference could be computed in most of them\. Each problem uses an openly available Modelica model, at least one tuned rule and, where practicable, an hourly exhaustive search or a dynamic programme with perfect foresight\. The aim is not to establish whether DRL is able to control HVAC plant, which earlier work has already shown, but to examine under what conditions it is worth applying\. This is expressed by two quantities: the margin that a problem leaves above a good rule, and the fraction of that margin which the agent obtains\.

## 2Related work

Reviews of RL for building control\[[Wang and Hong, 2020](https://arxiv.org/html/2611.22144#bib.bib5),[Yu et al\., 2021](https://arxiv.org/html/2611.22144#bib.bib12)\]and of field demonstrations\[[Khabbazi and others, 2025](https://arxiv.org/html/2611.22144#bib.bib14)\]discuss these questions at some length\.[Biemann et al\. \[2021\]](https://arxiv.org/html/2611.22144#bib.bib13)compared four actor\-critic algorithms on a simulated data centre, and several authors have compared RL with model predictive control \(MPC\) on a common emulator\[[Arroyo et al\., 2022](https://arxiv.org/html/2611.22144#bib.bib16),[Stoffel and others, 2023](https://arxiv.org/html/2611.22144#bib.bib17),[Wang and others, 2023](https://arxiv.org/html/2611.22144#bib.bib15)\], in which case MPC provides a practical upper reference\. Modelica models have been coupled to RL through FMUs, for instance in ModelicaGym\[[Lukianykhin and Bogodorova, 2019](https://arxiv.org/html/2611.22144#bib.bib19)\]and BOPTEST\-Gym\[[Arroyo and others, 2021](https://arxiv.org/html/2611.22144#bib.bib18)\]\. At the plant level, RL has been applied to cooling water systems\[[Qiu and others, 2020](https://arxiv.org/html/2611.22144#bib.bib20)\], to chiller plants\[[Wang and others, 2025](https://arxiv.org/html/2611.22144#bib.bib21)\], and to active and passive thermal storage\[[Henze and Schoenmann, 2003](https://arxiv.org/html/2611.22144#bib.bib7),[Liu and Henze, 2007](https://arxiv.org/html/2611.22144#bib.bib22)\]\. For chilled water systems without storage,[Braun et al\. \[1989\]](https://arxiv.org/html/2611.22144#bib.bib23)showed long ago that a simple control law can approach the optimum closely\. Chilled water reset based on outdoor temperature remains common practice and continues to be evaluated in office buildings\[[Lee et al\., 2022](https://arxiv.org/html/2611.22144#bib.bib24)\]\.

The contribution of the present paper is therefore not a new algorithm\. It consists of a small set of open, FMU\-based problems in which the margin above a tuned rule is measured, together with some practical observations on training RL agents directly on co\-simulation FMUs\.

## 3Methodology

### 3\.1Common framework

All plants were modelled in Modelica using the Buildings library\[[Wetter et al\., 2014](https://arxiv.org/html/2611.22144#bib.bib3)\], version 12\.0\.0, and exported as FMI 2\.0 co\-simulation units\[[Blochwitz and others, 2011](https://arxiv.org/html/2611.22144#bib.bib8)\]with the CVODE solver\. Each FMU was wrapped as a Gymnasium environment\[[Towers and others, 2024](https://arxiv.org/html/2611.22144#bib.bib9)\]and run through fmpy with a communication interval of 10 minutes; the supervisory decision interval was one hour\. The agents were trained with soft actor\-critic \(SAC\)\[[Haarnoja et al\., 2018](https://arxiv.org/html/2611.22144#bib.bib1)\]as implemented in Stable\-Baselines3\[[Raffin et al\., 2021](https://arxiv.org/html/2611.22144#bib.bib2)\]\. In each case two random seeds were trained in parallel on a laptop CPU, which took between 9 and 25 minutes\. The checkpoint that performed best on two validation periods was retained and evaluated on a separate test period\.

The weather data are those of the Brisbane Airport TMYx 2011–2025 file\. The building is an office occupied on weekdays from 07:00 to 18:00\. Cases 2 and 3 minimise energy use\. Cases 1 and 4 minimise electricity cost under the same time\-of\-use tariff, with an off\-peak period from 11:00 to 16:00 at 0\.08 AUD/kWh, a peak period from 16:00 to 21:00 at 0\.48 AUD/kWh, and a shoulder rate of 0\.26 AUD/kWh at all other times\.

### 3\.2Reward with an exogenous reference

The hourly cost is dominated by the building load, whereas the part that the controller can influence is typically one to three orders of magnitude smaller\. In every hour, therefore, the cost of a reference controller operating under the same load and weather was subtracted from the cost incurred by the agent:

rt=−\(ct\(at\)−ctref\)/s,r\_\{t\}=\-\\bigl\(c\_\{t\}\(a\_\{t\}\)\-c\_\{t\}^\{\\mathrm\{ref\}\}\\bigr\)/s,\(1\)wheressis a scaling constant\. Because the reference depends only on exogenous inputs, it adds a policy\-independent term to the return and leaves the optimal policy unchanged\. In Cases 2 and 4 the reference was evaluated with a quasi\-steady Python approximation of the plant, and in Case 3 with a second instance of the FMU running at the fixed set point\. In Case 2 this modification was necessary for learning to take place; it was retained in Cases 3 and 4\.

### 3\.3Near\-optimal references

Where the plant carries little state from one hour to the next \(Cases 2 and 3\), an hourly exhaustive search was performed: in every operating hour each candidate action was simulated and the least costly one retained\. In Case 3 the search was carried out on the FMU itself, using a second instance that replayed the week up to the hour in question\. For the storage problem \(Case 4\) a dynamic programme over the state of charge was solved with perfect foresight of load and weather\. It was formulated on the quasi\-steady approximation, and the demand charge was handled by sweeping over a cap on the plant power\. The resulting schedule was then applied to the FMU in the same way as any other controller\. Neither reference is a strict bound: the hourly search is greedy, and the agent equalled or marginally improved on it, while the dynamic programme optimises an approximation of the FMU rather than the FMU itself\. Both are accordingly described as near\-optimal references\.

## 4Case studies

The four cases are summarised in Table[1](https://arxiv.org/html/2611.22144#S4.T1), and the results on the test periods are given in Table[2](https://arxiv.org/html/2611.22144#S4.T2)\.

Table 1:The four case studies\.Table 2:Results on the test period \(Case 1: three hottest consecutive weekdays; Cases 2–4: one summer week\)\. Savings are relative to the first row of each case; a negative saving means more energy than that row\. The headroom is the difference between the best rule and the near\-optimal reference \(hourly optimum or perfect\-foresight schedule\), expressed as a percentage of the first row\.CaseControllerResultSavingRemark1Fixed 24 °C13\.31 AUD–Expert pre\-cooling rule11\.70 AUD12\.0 %hand\-tunedDRL \(SAC\)12\.56 AUD5\.6 %second seed 6\.4 %2Fixed 29\.4 °C12,22512,225kWh–Wet bulb \+ 4 K12,03012,030kWh1\.6 %DRL \(SAC\)11,86511,865kWh2\.9 %Hourly optimum11,87111,871kWh2\.9 %headroom 1\.3 %3Fixed 6\.67 °C5,6065,606kWh–Outdoor\-air reset \(tuned\)5,6095,609kWh−0\.05\-0\.05%Pump trim & respond5,6395,639kWh−0\.59\-0\.59%DRL \(SAC\)5,5965,596kWh0\.18 %Hourly optimum5,5965,596kWh0\.18 %headroom 0\.18 %4No storage3,7273,727AUD–Time\-of\-use rule3,5833,583AUD3\.9 %Demand\-limit rule \(tuned\)3,4923,492AUD6\.3 %DRL \(SAC\)3,4373,437AUD7\.8 %second seed 4\.0 %Perfect foresight3,0773,077AUD17\.4 %headroom 11\.1 %### 4\.1Case 1: pre\-cooling under a time\-of\-use tariff

A two\-node RC model of an office zone, served by an idealised chiller, is exposed to three consecutive hot weekdays\. The agent sets the zone temperature set point, and the comfort range during occupancy is 21–25 °C\. In this case the agent was trained on a Python model with the same equations as the Modelica model; when its test episode was replayed in Modelica, the zone temperature agreed to within 0\.04 K \(root\-mean\-square\)\.

Without being instructed to do so, the agent learned to store cooling in the building mass during the inexpensive midday period and to allow the zone temperature to rise during the peak period \(Fig\.[1](https://arxiv.org/html/2611.22144#S4.F1)\)\. A simple expert rule, which holds 22 °C during the off\-peak period and 25 °C during the peak, follows the same strategy more decisively and saves about twice as much\. Plausible explanations are that the stored cooling is only partly observable to the agent and that the price signal recurs only once a day\. No near\-optimal reference was computed for this case\.

![Refer to caption](https://arxiv.org/html/2611.22144v1/figures/case01_behaviour.png)Figure 1:Case 1: zone temperature, set point and chiller power over the three test days\. Green: off\-peak; red: peak\.
### 4\.2Case 2: chiller staging and condenser water reset

The agent selects the condenser water set point and the number of chillers in operation\. It arrived at essentially the rule used in practice, tracking the wet\-bulb temperature with an approach of about 4 K and starting the second chiller at a load of about 720 kW \(Fig\.[2](https://arxiv.org/html/2611.22144#S4.F2)\), and its energy use was practically the same as that of the hourly exhaustive search\. The entire margin above the tuned rule, however, was only 1\.3 %\. This is in keeping with the finding of[Braun et al\. \[1989\]](https://arxiv.org/html/2611.22144#bib.bib23)that simple control laws come close to the optimum for chilled water systems without storage\.

![Refer to caption](https://arxiv.org/html/2611.22144v1/figures/case02_policy.png)Figure 2:Case 2: condenser water set point against wet bulb \(left\) and staging against load \(right\) for the rule, the agent and the hourly optimum\.
### 4\.3Case 3: chilled water supply temperature reset

Raising the chilled water temperature improves the COP of the chiller by roughly 3 % per kelvin\. In this plant, however, it also requires a larger water flow through the coil, and the pump power rises with approximately the cube of the flow\. Above about 8 °C the pump reaches full speed, the supply air temperature can no longer be held, and the supply fan, whose power also varies roughly with the cube of the flow, accounts for the additional losses\. In consequence the best set point decreases as the load increases, from about 7\.75 °C below 250 kW to about 5\.6 °C above 400 kW\. The agent learned this relationship without any knowledge of the coil or pump characteristics \(Fig\.[3](https://arxiv.org/html/2611.22144#S4.F3)\)\. Both reset rules, which raise the set point, used slightly more energy than the fixed set point\. The benefit of chilled water reset therefore appears to depend on the size of the coil and on the share of pumping energy, and it would seem prudent to verify it for a given plant rather than to assume it\[[Lee et al\., 2022](https://arxiv.org/html/2611.22144#bib.bib24)\]\.

![Refer to caption](https://arxiv.org/html/2611.22144v1/figures/case03_policy.png)Figure 3:Case 3: chilled water set point against zone load \(left\) and outdoor temperature \(right\)\. The agent and the hourly optimum lower the set point as the load rises; the outdoor\-air reset does not\.
### 4\.4Case 4: chilled water storage with a demand charge

In this case the agent sets the rate at which the tank is charged or discharged in each hour\. A demand charge of 20 AUD/kW per month, pro\-rated to one week, is levied on the highest hourly plant power, and staging and condenser water reset follow the rule of Case 2\.

The agent achieved a lower cost than both rules \(Fig\.[4](https://arxiv.org/html/2611.22144#S4.F4)\), principally by charging the tank during the midday off\-peak period, when rooftop photovoltaic generation depresses the price, rather than at night\. Its weekly peak of 275 kW was, however, scarcely lower than the 279 kW of the plant without storage, whereas the demand\-limiting rule reached 209 kW and the perfect\-foresight schedule combined midday charging with a peak of 236 kW\. The demand charge is determined by a single hour of the week, and in100,000100,000training steps the agent, which acts at hourly intervals, did not learn to relate its earlier decisions to that hour\. This observation is consistent with earlier work on RL for thermal storage, which found that the learned controller compared favourably with conventional strategies but did not reach the performance of predictive optimal control\[[Henze and Schoenmann, 2003](https://arxiv.org/html/2611.22144#bib.bib7)\], and that learning became slow as the state and action spaces grew\[[Liu and Henze, 2007](https://arxiv.org/html/2611.22144#bib.bib22)\]\.

![Refer to caption](https://arxiv.org/html/2611.22144v1/figures/case04_week.png)Figure 4:Case 4: building and chiller load, tank heat flow, state of charge and plant power over the test week for the demand\-limit rule, the agent and the perfect\-foresight schedule\.

## 5Discussion

### 5\.1Margin above a good rule

Cases 2 and 3 indicate that, for a supervisory problem with little storage and a sensible existing rule, little remains to be gained\. The agent reached the near\-optimal reference, but that reference was only 0\.2–1\.3 % better than the rule\. An hourly exhaustive search of this kind requires only minutes of computation, and it seems advisable to carry it out before a DRL study of such a problem is undertaken\. Case 3 also illustrates the opposite situation, in which a widely used rule performed slightly worse than no reset at all; the reference made this apparent\.

### 5\.2Problems with storage

When decisions are coupled in time the margin was about ten times larger, and the agent found a strategy that the rules did not use\. Two features appear to have limited its performance: cost terms determined by a single extreme hour, such as the demand charge, which a tuned rule handled better; and storage that is only partly observed, as with the building mass in Case 1, where a hand\-tuned rule also performed better than the agent\. In such problems the fraction of the attainable saving obtained by the agent appears to be a more informative measure than the saving relative to a baseline\. In Case 4 the agent obtained about 45 % of the saving of the perfect\-foresight schedule, but only about 13 % of the margin above the best rule \(1\.5 of 11\.1 percentage points\)\.

### 5\.3Training on co\-simulation FMUs

Training directly on the FMU required only 0\.6–5 ms per simulated hour, so that no surrogate model was needed\. Several difficulties with the co\-simulation FMUs used here are worth recording\. A change of input at a communication point did not trigger an event, so that thresholds evaluated on internal variables, for example staging based on the chiller load, retained stale values; in one instance the chillers remained off and the stagnant water was heated beyond the validity range of the medium model\. Discrete decisions were therefore taken outside the FMU and passed to it as inputs, and the FMU was initialised with the inputs of the first action\. Furthermore, a state restored withfmi2SetFMUstatedid not respond correctly to subsequent input changes, so the exhaustive search replays the trajectory instead\. Any physically impossible source of free cooling in the model was quickly found and exploited by the agent; reporting the unserved cooling and the maximum supply temperature for every controller revealed such errors at once\. Finally, the formulation of the action proved important: an action for which zero means “no intervention” \(the charging rate\) was learned much more readily than a set\-point\-like action \(a target for the chiller load\)\.

### 5\.4Limitations

Each case uses one building, one climate, one tariff and a single test period, and only two training seeds\. In Cases 2 and 3 the differences between the agent, the reference and the best rule \(0\.2–1\.3 %\) are of the same order as the modelling error; in Case 3, for example, the FMU and the continuous\-time simulation of the same model differ by 0\.6 %, and the chiller curves are extrapolated above a leaving water temperature of 8\.9 °C\. The conclusion that the margin is small therefore appears robust, whereas the ranking of controllers with nearly equal energy use should not be relied upon\. The result of Case 3 also depends on design choices, notably the coil size, the pump characteristic and the supply air set point of 13 °C at full cooling\. The models are deliberately small, and some components are idealised, among them the cooling tower in Case 3 and the weekly pro\-rated demand charge in Case 4\. The forecasts available to the agents are perfect\. The findings should accordingly be read as statements about these particular plants rather than about DRL or reset strategies in general\.

## 6Conclusion

For four small Modelica plants, SAC agents learned sensible supervisory policies; in the two cases without storage they matched the near\-optimal reference\. Whether the effort was worthwhile depended more on the problem than on the algorithm\. Without storage the margin above a good rule was between 0\.2 and 1\.3 %; with storage it exceeded 10 %, of which the agent obtained only a part\. It is suggested that the margin be estimated before DRL is applied, and that the fraction of it obtained be reported\. The models, exported FMUs, trained agents and scripts are available online\[[Zhao, 2026](https://arxiv.org/html/2611.22144#bib.bib11)\]\. Should the FMUs not run in a particular environment, they can be regenerated from the Modelica sources\.

## Code and data

## References

- J\. Arroyo, C\. Manna, F\. Spiessens, and L\. HelsenReinforced model predictive control \(RL\-MPC\) for building energy management\.Applied Energy309,pp\. 118346\.External Links:[Document](https://dx.doi.org/10.1016/j.apenergy.2021.118346)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Arroyoet al\.\(2021\)J\. Arroyoet al\.An OpenAI\-Gym environment for the building optimization testing \(BOPTEST\) framework\.InProceedings of Building Simulation 2021,External Links:[Document](https://dx.doi.org/10.26868/25222708.2021.30380)Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1),[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- ASHRAE \(2021\)ASHRAEGuideline 36\-2021: high\-performance sequences of operation for HVAC systems\.ASHRAE,Atlanta, GA\.Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1)\.
- Biemannet al\.\(2021\)M\. Biemann, F\. Scheller, X\. Liu, and L\. HuangExperimental evaluation of model\-free reinforcement learning algorithms for continuous HVAC control\.Applied Energy298,pp\. 117164\.External Links:[Document](https://dx.doi.org/10.1016/j.apenergy.2021.117164)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Blochwitzet al\.\(2011\)T\. Blochwitzet al\.The functional mockup interface for tool independent exchange of simulation models\.InProceedings of the 8th International Modelica Conference,pp\. 105–114\.Cited by:[§3\.1](https://arxiv.org/html/2611.22144#S3.SS1.p1.1)\.
- Blumet al\.\(2021\)D\. Blum, J\. Arroyo, S\. Huang, J\. Drgoňa, F\. Jorissen, H\. T\. Walnum, Y\. Chen, K\. Benne, D\. Vrabie, M\. Wetter, and L\. HelsenBuilding optimization testing framework \(BOPTEST\) for simulation\-based benchmarking of control strategies in buildings\.Journal of Building Performance Simulation14\(5\),pp\. 586–610\.Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1)\.
- Braunet al\.\(1989\)J\. E\. Braun, S\. A\. Klein, W\. A\. Beckman, and J\. W\. MitchellMethodologies for optimal control of chilled water systems without storage\.ASHRAE Transactions95\(1\),pp\. 652–662\.Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1),[§4\.2](https://arxiv.org/html/2611.22144#S4.SS2.p1.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 1861–1870\.Cited by:[§3\.1](https://arxiv.org/html/2611.22144#S3.SS1.p1.1)\.
- Henze and Schoenmann \(2003\)G\. P\. Henze and J\. SchoenmannEvaluation of reinforcement learning control for thermal energy storage systems\.HVAC&R Research9\(3\),pp\. 259–275\.Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1),[§4\.4](https://arxiv.org/html/2611.22144#S4.SS4.p2.1)\.
- Khabbaziet al\.\(2025\)A\. Khabbaziet al\.Lessons learned from field demonstrations of model predictive control and reinforcement learning for residential and commercial HVAC: a review\.Applied Energy,pp\. 126459\.External Links:[Document](https://dx.doi.org/10.1016/j.apenergy.2025.126459)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Leeet al\.\(2022\)Y\. Lee, S\. Hong, J\. Lee, Y\. Yoon, J\. Choi, and K\. LeeChilled water temperature set\-point reset based on outdoor air temperature and its cooling energy performance in an office building\.Journal of Mechanical Science and Technology36,pp\. 1557–1568\.External Links:[Document](https://dx.doi.org/10.1007/s12206-022-0241-4)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1),[§4\.3](https://arxiv.org/html/2611.22144#S4.SS3.p1.1)\.
- Liu and Henze \(2007\)S\. Liu and G\. P\. HenzeEvaluation of reinforcement learning for optimal control of building active and passive thermal storage inventory\.Journal of Solar Energy Engineering129\(2\),pp\. 215–225\.External Links:[Document](https://dx.doi.org/10.1115/1.2710491)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1),[§4\.4](https://arxiv.org/html/2611.22144#S4.SS4.p2.1)\.
- Lukianykhin and Bogodorova \(2019\)O\. Lukianykhin and T\. BogodorovaModelicaGym: applying reinforcement learning to Modelica models\.InProceedings of the 9th International Workshop on Equation\-based Object\-oriented Modeling Languages and Tools,External Links:[Document](https://dx.doi.org/10.1145/3365984.3365985)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Qiuet al\.\(2020\)S\. Qiuet al\.Model\-free control method based on reinforcement learning for building cooling water systems: validation by measured data\-based simulation\.Energy and Buildings218,pp\. 110055\.External Links:[Document](https://dx.doi.org/10.1016/j.enbuild.2020.110055)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Raffinet al\.\(2021\)A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. DormannStable\-baselines3: reliable reinforcement learning implementations\.Journal of Machine Learning Research22\(268\),pp\. 1–8\.Cited by:[§3\.1](https://arxiv.org/html/2611.22144#S3.SS1.p1.1)\.
- Stoffelet al\.\(2023\)P\. Stoffelet al\.Evaluation of advanced control strategies for building energy systems\.Energy and Buildings,pp\. 112709\.External Links:[Document](https://dx.doi.org/10.1016/j.enbuild.2022.112709)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Towerset al\.\(2024\)M\. Towerset al\.Gymnasium: a standard interface for reinforcement learning environments\.External Links:2407\.17032Cited by:[§3\.1](https://arxiv.org/html/2611.22144#S3.SS1.p1.1)\.
- Wanget al\.\(2023\)D\. Wanget al\.Comparison of reinforcement learning and model predictive control for building energy system optimization\.Applied Thermal Engineering,pp\. 120430\.External Links:[Document](https://dx.doi.org/10.1016/j.applthermaleng.2023.120430)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Wanget al\.\(2025\)X\. Wanget al\.Leveraging reinforcement learning for optimal control of chiller plant with complex hydraulic and thermodynamic characteristics\.Energy and Buildings,pp\. 116037\.External Links:[Document](https://dx.doi.org/10.1016/j.enbuild.2025.116037)Cited by:[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Wang and Hong \(2020\)Z\. Wang and T\. HongReinforcement learning for building controls: the opportunities and challenges\.Applied Energy269,pp\. 115036\.Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1),[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Weiet al\.\(2017\)T\. Wei, Y\. Wang, and Q\. ZhuDeep reinforcement learning for building HVAC control\.InProceedings of the 54th Annual Design Automation Conference,pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1)\.
- Wetteret al\.\(2014\)M\. Wetter, W\. Zuo, T\. S\. Nouidui, and X\. PangModelica Buildings library\.Journal of Building Performance Simulation7\(4\),pp\. 253–270\.Cited by:[§3\.1](https://arxiv.org/html/2611.22144#S3.SS1.p1.1)\.
- Yuet al\.\(2021\)L\. Yu, S\. Qin, M\. Zhang, C\. Shen, T\. Jiang, and X\. GuanA review of deep reinforcement learning for smart building energy management\.IEEE Internet of Things Journal8\(15\),pp\. 12046–12063\.External Links:[Document](https://dx.doi.org/10.1109/JIOT.2021.3078462)Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1),[§2](https://arxiv.org/html/2611.22144#S2.p1.1)\.
- Zhanget al\.\(2019\)Z\. Zhang, A\. Chong, Y\. Pan, C\. Zhang, and K\. P\. LamWhole building energy model for HVAC optimal control: a practical framework based on deep reinforcement learning\.Energy and Buildings199,pp\. 472–490\.External Links:[Document](https://dx.doi.org/10.1016/j.enbuild.2019.07.029)Cited by:[§1](https://arxiv.org/html/2611.22144#S1.p1.1)\.
- Zhao \(2026\)B\. ZhaoHvac\-drl\-lab: small, reproducible HVAC case studies with Modelica and deep reinforcement learning\.Note:Version 0\.3\.0\.[https://github\.com/Bingnan\-Zhao/hvac\-drl\-lab](https://github.com/Bingnan-Zhao/hvac-drl-lab)External Links:[Document](https://dx.doi.org/10.5281/zenodo.23910085)Cited by:[§6](https://arxiv.org/html/2611.22144#S6.p1.1)\.

Similar Articles

Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation

arXiv cs.AI

这篇论文在统一实验协议下,用统计方法系统比较了六种深度强化学习处理(PPO、A2C、DQN,各含/不含动作掩码)与传统启发式算法及随机对照在HPC作业调度上的表现,发现MaskablePPO在异构负载上优于对照,但多数情况下DRL并未显著超越动作掩码带来的收益。