Green BOA: Determining the environmental break-even point for ML-based data compression

# Determining the environmental break-even point for ML-based data compression
Source: [https://arxiv.org/html/2608.19994](https://arxiv.org/html/2608.19994)
## Green BOA: Determining the environmental break\-even point for ML\-based data compressionConference:2nd International Workshop on Low Carbon Computing; Lancaster, UK;

Caterina Doglioni,Akshat Guptaemail:[akshat\.gupta\-4@postgrad\.manchester\.ac\.uk](mailto:[email protected])Affiliation:University of Manchester,Manchester,UK,Thomas Elliottemail:[thomas\.elliott@manchester\.ac\.uk](mailto:[email protected])Affiliation:University of Manchester,Manchester,UK,Hanzila Hussainemail:[hanzilahussain234@gmail\.com](mailto:[email protected])Affiliation:University of Manchester,Manchester,UK,Sanjiban Senguptaemail:[sanjiban\.sengupta@cern\.ch](mailto:[email protected])Affiliation:CERN/University of Manchester,Manchester,UKandZhengkai Sunemail:[zhengkai\.sun@student\.manchester\.ac\.uk](mailto:[email protected])Affiliation:University of Manchester,Manchester,UK

© none

###### Abstract\.

We summarise the outcome of two summer internship projects based at the University of Manchester, focused on the break\-even point in terms of environmental sustainability for ML\-based data compression algorithms\. Using the example of a ML\-based lossless compression algorithm, we compare estimates for the carbon\-equivalent of the infrastructure needed for ML training and inference with the carbon\-equivalent savings from reduced disk storage requirements, and discuss their break\-even point\.

Context\.Big Data experiments, such as those at the Large Hadron Collider\([2](https://arxiv.org/html/2608.19994#bib.bib2)\)and in other astroparticle physics experiments \(e\.g\.\([13](https://arxiv.org/html/2608.19994#bib.bib13)\)\), will be recording several Exabytes of data\. This has a significant cost in terms of both budget\([1](https://arxiv.org/html/2608.19994#bib.bib1)\)and environmental resources for data storage \(see e\.g\.\([15](https://arxiv.org/html/2608.19994#bib.bib5);[11](https://arxiv.org/html/2608.19994#bib.bib10)\)\)\. R&D on data compression is ongoing, and includes machine learning \(ML\-\) based data compression\. Since this kind of compression techniques are generally more computationally intensive than standard compression algorithms such as ZSTD\([4](https://arxiv.org/html/2608.19994#bib.bib11)\),and LZMA\([12](https://arxiv.org/html/2608.19994#bib.bib12)\), we investigate the break\-even point between theC​O2CO\_\{2\}\-equivalent cost of training and executing the ML\-based compression algorithm, and the embodied pllus operationalC​O2CO\_\{2\}\-equivalent of disk storage that would be displaced by storing data in its compressed form\.

Methods\.We consider the energy use of a ML\-based data compression algorithms developed at the University of Manchester\([7](https://arxiv.org/html/2608.19994#bib.bib3)\), and estimate its energy usage for ML training and inference \(compression/decompression round trip\) on an Nvidia T4 GPU\. We consider several country\-specific scenarios for where these steps will be executed, to convert energy consumption into aC​O2CO\_\{2\}\-equivalent based on average energy mix\.

We then consider the carbon\-equivalent cost of manufacturing and operating disk storage devices, in the shape of Hard Disk Drives \(HDD\) and tapes, for a 5 years lifetime\. As examples, we consider Seagate Exos X18 HDDs\([14](https://arxiv.org/html/2608.19994#bib.bib14)\)and tape cartridges\([10](https://arxiv.org/html/2608.19994#bib.bib15)\)\(following the methodologies in\([11](https://arxiv.org/html/2608.19994#bib.bib10);[16](https://arxiv.org/html/2608.19994#bib.bib16)\)\), hypothetically located in the UK\.

Inspired by\([3](https://arxiv.org/html/2608.19994#bib.bib6)\), we define the break\-even point for this project as thesize of the uncompressed dataset at which the carbon cost of training and running the compression algorithm equals the embodied carbon of the extra disk storage needed for the uncompressed data\. Further information on the methods can be found in the Appendix\.

Rather than a full analysis, this is a proof\-of\-principle calculation with several assumptions using a limited amount of data\. Only operational costs from CPU, GPU and RAM are considered for the ML\-based data compression, as we assume that the GPU will already be available and that its use for ML\-based data compression will be negligible compared to other uses within a LHC experiment\. Thescenarioconsidered for the ML\-based data compression comprises training on the full dataset, followed by one compression and one decompression round \(on an existing temporary disk\)\. A broader range of scenarios will be explored in future work\. Since training dominates the energy consumption, this can be considered an upper bound in terms of energy consumption, as in practice BOA’s generalisation capability outlined in\([7](https://arxiv.org/html/2608.19994#bib.bib3)\)allows for training only on a small dataset and then using this trained model for a much larger dataset\.

Results and conclusions\.

As it can be seen in Figure[1](https://arxiv.org/html/2608.19994#S0.F1), the location of the break\-even point is very sensitive to the carbon intensity of the country where the ML\-based data compression is performed\. As shown in previous literature, tape storage has a lowerC​O2CO\_\{2\}\-equivalent footprint than HDDs, but at the cost of slower data access\.

![Refer to caption](https://arxiv.org/html/2608.19994v1/break-even.png)Figure 1\.Results of the break\-even analysis in terms ofC​O2CO\_\{2\}\-equivalent for ML\-based data compression and subsequent displaced HDD/tape storage\.We also compared ML\-based data compression to standard algorithms, taking performance \(compression ratio\) into account: since BOA generally achieves a better compression ratio but at lower throughput with respect to standard algorithms\([7](https://arxiv.org/html/2608.19994#bib.bib3)\), it will be moreC​O2CO\_\{2\}\-intensive per unit of data processed than non\-ML algorithms\. While the team is working on improving the throughput, it will still be worth examining whether the compression gains of ML\-based approaches justify the additional environmental cost in deployment at scale\.

## Appendix: technical configuration

In the following, we detail the computing hardware and software configuration used to obtain these results, as well as the setup for the estimate of HDD and tapeC​O2CO\_\{2\}\-equivalent\.

### ML data compression

The BOA ML compression algorithm was trained and executed on a Google Colab notebook with a runtime Nvidia T4 GPU\. The model chosen was a two\-layer Mamba\-v1 backbone with a hidden dimension of 64, a byte vocabulary of 256 symbols, a sequence length of 10,000 bytes and a batch size of 5\. Training was performed for eight epochs in FP32 precision, using a learning rate of5×10−45\\times 10^\{\-4\}and a random seed of 42\. The training step included the eight training epochs, validation, final test evaluation and checkpoint writing\. The compression step included the execution of one compression and decompression round\-trip\. The file that was compressed was the 49\.92 MB "bundled CMS file" \(CMS\_DATA\_float32\.bin\) from\([7](https://arxiv.org/html/2608.19994#bib.bib3)\)that can be found within the BOA GitHub repository\([8](https://arxiv.org/html/2608.19994#bib.bib4)\); each data record contains 24float32features\. The results from this file were scaled to the maximum dataset size shown in Figure[1](https://arxiv.org/html/2608.19994#S0.F1)after checking linearity of training and inference carbon costs\.

Carbon tracking was performed using CodeCarbon\([6](https://arxiv.org/html/2608.19994#bib.bib7)\), following a tutorial in\([9](https://arxiv.org/html/2608.19994#bib.bib9)\)\. CodeCarbon was used in one\-second power\-sampling intervals usingmachinemode to measure the energy consumption of CPU, GPU and RAM used in the ML\-based compression training and inference\. The same setup was used for the compression\-decompression round trip using ZSTD and LZMA\. We are aware thatmachinemode measures the energy consumption of the hardware stack of GPU, CPU and RAM \(which may as well be shared when using a Google Colab notebook\), but it is the only way to obtain a GPU energy estimate using CodeCarbon\. Thermal Design Power \(TDP\) scaling for an Intel\(R\) Xeon\(R\) CPU @ 2\.00GHz was used for the CPU power consumption corresponding to 8W, while the GPU power consumption used thenvidia\-ml\-pypackage\.

The conversion between energy use and carbon equivalent was performed using CodeCarbon’s national energy mix averaged scenarios\([5](https://arxiv.org/html/2608.19994#bib.bib8)\)\.

### Storage servers

The carbon cost included both embodied and operational carbon\([11](https://arxiv.org/html/2608.19994#bib.bib10);[16](https://arxiv.org/html/2608.19994#bib.bib16)\)\.

For the storage of D GB over T years on a medium m, the total storage carbon cost is

\(1\)Cm​\(D,T\)=Cm,embodied​\(D\)\+Cm,operational​\(D,T\)\.C\_\{m\}\(D,T\)=C\_\{m,\\mathrm\{embodied\}\}\(D\)\+C\_\{m,\\mathrm\{operational\}\}\(D,T\)\.
For HDD storage, a drive with usable capacityKH​D​DK\_\{HDD\}is the basic hardware unit\. The number of powered drives required to store D GB is

\(2\)NHDD​\(D\)=max⁡\(NHDD,min,⌈DKHDD⌉\),N\_\{\\mathrm\{HDD\}\}\(D\)=\\max\\left\(N\_\{\\mathrm\{HDD,min\}\},\\left\\lceil\\frac\{D\}\{K\_\{\\mathrm\{HDD\}\}\}\\right\\rceil\\right\),where we chooseNHDD,min=1N\_\{\\mathrm\{HDD,min\}\}=1, meaning at least one drive is assumed to remain installed and operational\. Given the limited scope and total data volume of this study, the displaced carbon \(non\-use\) for the HDD is allocated proportionally to the storage capacity

\(3\)CHDD,non​\-​use​\(D\)=D​eHDD,non​\-​use,C\_\{\\mathrm\{HDD,non\\text\{\-\}use\}\}\(D\)=D\\,e\_\{\\mathrm\{HDD,non\\text\{\-\}use\}\},whereeHDD,non​\-​usee\_\{\\mathrm\{HDD,non\\text\{\-\}use\}\}is the capacity\-allocated displaced carbon ing​C​O2​e/k​W​hgCO\_\{2\}e/kWh\. The operational carbon was calculated using the integer number of powered drives

\(4\)CHDD,operational​\(D,T\)=NHDD​\(D\)​PHDD1000​\(24\)​\(365\.25\)​T​IHDD​PUE,C\_\{\\mathrm\{HDD,operational\}\}\(D,T\)=N\_\{\\mathrm\{HDD\}\}\(D\)\\frac\{P\_\{\\mathrm\{HDD\}\}\}\{1000\}\(24\)\(365\.25\)TI\_\{\\mathrm\{HDD\}\}\\mathrm\{PUE\},we neglect the Power Usage Effectiveness \(PUE\) throughout \(and set it equal to unity\)\. So the total HDD carbon cost is the sum of these two quantities\. The HDD scenario used the Seagate Exos X18 18 TB drive\. Its nominal capacity and average idle power were taken as 18 TB and 5\.3 W, respectively\([14](https://arxiv.org/html/2608.19994#bib.bib14)\)\. The displacedk​g​C​O2​ekgCO\_\{2\}evalue was taken as 27k​g​C​O2​ekgCO\_\{2\}eper drive, corresponding to a proportionality factor of 1\.50g​C​O2​e/G​BgCO\_\{2\}e/GB\.

For Tape storage, cartridge\-related non\-use carbon was calculated using an integer number of LTO\-8 cartridges, while operational carbon was calculated as proportional to the complete RAL Tape\-service factor in\([11](https://arxiv.org/html/2608.19994#bib.bib10)\)\. While inconsistent with the choice made for HDDs, this choice reflected existing data availability\.

Similarly to HDD, the number of cartridges is

\(5\)NTape​\(D\)=max⁡\(1,⌈DKTape⌉\)\.N\_\{\\mathrm\{Tape\}\}\(D\)=\\max\\left\(1,\\left\\lceil\\frac\{D\}\{K\_\{\\mathrm\{Tape\}\}\}\\right\\rceil\\right\)\.
The total carbon cost is calculated as

\(6\)CTape​\(D,T\)=NTape​\(D\)​ETape,non​\-​usecartridge\+D1000​T​eTape,RAL,C\_\{\\mathrm\{Tape\}\}\(D,T\)=N\_\{\\mathrm\{Tape\}\}\(D\)E\_\{\\mathrm\{Tape,non\\text\{\-\}use\}\}^\{\\mathrm\{cartridge\}\}\+\\frac\{D\}\{1000\}\\,T\\,e\_\{\\mathrm\{Tape,RAL\}\},whereETape,non​\-​usecartridgeE\_\{\\mathrm\{Tape,non\\text\{\-\}use\}\}^\{\\mathrm\{cartridge\}\}is the non\-use lifecycle proxy for one cartridge ing​C​O2​e/c​a​r​t​r​i​d​g​egCO\_\{2\}e/cartridge, andeTape,RALe\_\{\\mathrm\{Tape,RAL\}\}is the complete RAL Tape\-service operational factor ing​C​O2​e/G​B​y​e​a​rgCO\_\{2\}e/GByear\. The first term is cartridge\-related displaced \(non\-use\) carbon, while the second represents operational emissions allocated to the stored data volume\. The model used an LTO\-8 cartridge with a native capacity of 12 TB\. The corresponding lifecycle data report a total value of13\.72​k​g​C​O2​e13\.72kgCO\_\{2\}eper cartridge, of which 5\.70k​g​C​O2​ekgCO\_\{2\}eis assigned to the operational phase\([10](https://arxiv.org/html/2608.19994#bib.bib15);[11](https://arxiv.org/html/2608.19994#bib.bib10)\)\.

\(7\)ETape,non​\-​usecartridge=ETape,totalcartridge−ETape,usecartridge\.E\_\{\\mathrm\{Tape,non\\text\{\-\}use\}\}^\{\\mathrm\{cartridge\}\}=E\_\{\\mathrm\{Tape,total\}\}^\{\\mathrm\{cartridge\}\}\-E\_\{\\mathrm\{Tape,use\}\}^\{\\mathrm\{cartridge\}\}\.Here,ETape,totalcartridgeE\_\{\\mathrm\{Tape,total\}\}^\{\\mathrm\{cartridge\}\}is the reported total lifecycle carbon emission of one tape cartridge, andETape,usecartridgeE\_\{\\mathrm\{Tape,use\}\}^\{\\mathrm\{cartridge\}\}is the corresponding use\-phase contribution\. Using the literature values of13\.72​kgCO2​e/cartridge13\.72~\\mathrm\{kgCO\_\{2\}e/cartridge\}and5\.70​kgCO2​e/cartridge5\.70~\\mathrm\{kgCO\_\{2\}e/cartridge\}, respectively\([10](https://arxiv.org/html/2608.19994#bib.bib15);[11](https://arxiv.org/html/2608.19994#bib.bib10)\), gives

\(8\)ETape,non​\-​usecartridge=13\.72−5\.70=8\.02​kgCO2​e/cartridge\.E\_\{\\mathrm\{Tape,non\\text\{\-\}use\}\}^\{\\mathrm\{cartridge\}\}=13\.72\-5\.70=8\.02~\\mathrm\{kgCO\_\{2\}e/cartridge\}\.
###### Acknowledgements\.

This work was funded by EPSRC Studentship EP/W524347/1 \(Project 2932638\) via MADSIM CDT, the University of Manchester Dame Kathleen Ollerenshaw Fellowship, and is part of a project that has received funding from the European Research Council under the European Union’s Horizon 2020 research and innovation program \(grant agreement 101002463\)\. Summer student funding was also provided by the N8CIR summer internship scheme\.

## References

- \[1\]\(2022\)ATLAS Software and Computing HL\-LHC Roadmap\.Technical reportCERN,Geneva\.External Links:[Link](https://cds.cern.ch/record/2802918)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1)\.
- CERN \(2017\)CERNLarge Hadron Collider FAQ\.Technical reportCERN,Geneva\.External Links:[Link](https://cds.cern.ch/record/2255762)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1)\.
- Coignionet al\.\(2025\)T\. Coignion, C\. Quinton, and R\. RouvoyWhen faster isn’t greener: the hidden costs of llm\-based code optimization\.In2025 40th IEEE/ACM International Conference on Automated Software Engineering \(ASE\),pp\. 1655–1666\.External Links:[Link](https://doi.org/10.1109/ASE63991.2025.00139),[Document](https://dx.doi.org/10.1109/ASE63991.2025.00139)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p4.1)\.
- Collet and Kucherawy \(2021\)Y\. Collet and M\. KucherawyZstandard Compression and the ‘application/zstd’ Media Type\.RFCTechnical Report8878,IETF\.External Links:[Document](https://dx.doi.org/10.17487/RFC8878),[Link](https://www.rfc-editor.org/info/rfc8878)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1)\.
- Courtyet al\.\(2023\)B\. Courtyet al\.Global electricity carbon\-intensity data for private infrastructure\.Note:Data file in the CodeCarbon source repositoryCountry\-average factors used for regional scenarios; accessed 5 August 2026External Links:[Link](https://github.com/mlco2/codecarbon/blob/master/codecarbon/data/private_infra/global_energy_mix.json)Cited by:[ML data compression](https://arxiv.org/html/2608.19994#Sx1.SSx1.p3.1)\.
- Courtyet al\.\(2026\)Mlco2/codecarbon: v3\.3\.0External Links:[Document](https://dx.doi.org/10.5281/zenodo.21791294),[Link](https://doi.org/10.5281/zenodo.21791294)Cited by:[ML data compression](https://arxiv.org/html/2608.19994#Sx1.SSx1.p2.1)\.
- Guptaet al\.\(2026a\)A\. Gupta, C\. Doglioni, and T\. J\. ElliottBOA constrictor: a mamba\-based lossless compressor for scientific data\.Machine Learning: Science and Technology7\(3\),pp\. 035014\.External Links:[Document](https://dx.doi.org/10.1088/2632-2153/ae64a9),[Link](https://doi.org/10.1088/2632-2153/ae64a9)Cited by:[ML data compression](https://arxiv.org/html/2608.19994#Sx1.SSx1.p1.1),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p2.1),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p5.1),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p8.1)\.
- Guptaet al\.\(2026b\)A\. Gupta, C\. Doglioni, and T\. ElliottBoa constrictor: a mamba\-based lossless compressor for scientific data\.Technical reportZenodo,Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.18481928),[Link](https://doi.org/10.5281/zenodo.18481928)Cited by:[ML data compression](https://arxiv.org/html/2608.19994#Sx1.SSx1.p1.1)\.
- Hanna and Perez Murcia \(2026\)M\. Hanna and A\. F\. Perez MurciaReducing your climate impact when training ml models\.Tutorial,Climate Change AI\.Note:[https://github\.com/climatechange\-ai\-tutorials/tracking\-ml\-emissions](https://github.com/climatechange-ai-tutorials/tracking-ml-emissions)External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.5281/zenodo.21828628)Cited by:[ML data compression](https://arxiv.org/html/2608.19994#Sx1.SSx1.p2.1)\.
- Johns \(2021\)B\. JohnsImproving information technology sustainability with modern tape storage\.Technical reportBrad Johns Consulting and Fujifilm\.Note:Lifecycle assessment used for the LTO\-8 cartridge proxy; accessed 5 August 2026External Links:[Link](https://asset.fujifilm.com/www/de/files/2023-10/97ddc3473883421cef1fb820d236dfa2/Improving_IT_Sustainability_with_Tape_BJC_0.pdf)Cited by:[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p6.2),[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p6.3),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p3.1)\.
- Packer and Skipsey \(2025\)A\. Packer and S\. C\. SkipseyCarbon costs of storage: a U\.K\. perspective\.EPJ Web of Conferences337,pp\. 01157\.External Links:[Document](https://dx.doi.org/10.1051/epjconf/202533701157),[Link](https://doi.org/10.1051/epjconf/202533701157)Cited by:[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p1.1),[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p4.1),[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p6.2),[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p6.3),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p3.1)\.
- Pavlov \(2007\)I\. PavlovLZMA SDK \(software development kit\)\.External Links:[Link](https://7-zip.org/sdk.html)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1)\.
- Scaife \(2020\)A\. M\. M\. ScaifeBig telescope, big data: towards exascale with the Square Kilometre Array\.Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences378\(2166\),pp\. 20190060\.External Links:[Document](https://dx.doi.org/10.1098/rsta.2019.0060)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1)\.
- Seagate Technology \(2023\)Seagate TechnologyExos X18 18TB product sustainability report\.Technical reportSeagate Technology\.Note:Accessed 5 August 2026External Links:[Link](https://www.seagate.com/content/dam/seagate/assets/esg/planet/product-sustainability/images/exos-x18-sustainability-report/files/Exos-X18-18TB-Sustainability-Report-2023.pdf)Cited by:[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p3.4),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p3.1)\.
- Tannu and Nair \(2023\)S\. Tannu and P\. J\. NairThe dirty secret of ssds: embodied carbon\.ACM SIGEnergy Energy Informatics Review3\(3\),pp\. 4–9\.External Links:ISSN 2770\-5331,[Link](http://dx.doi.org/10.1145/3630614.3630616),[Document](https://dx.doi.org/10.1145/3630614.3630616)Cited by:[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p1.1)\.
- Wadenstein and Vanderbauwhede \(2025\)M\. Wadenstein and W\. VanderbauwhedeLife cycle analysis for emissions of scientific computing centres\.Eur\. Phys\. J\. C85\(8\),pp\. 913\.External Links:2506\.14365,[Document](https://dx.doi.org/10.1140/epjc/s10052-025-14650-8)Cited by:[Storage servers](https://arxiv.org/html/2608.19994#Sx1.SSx2.p1.1),[Green BOA: Determining the environmental break\-even point for ML\-based data compression](https://arxiv.org/html/2608.19994#p3.1)\.