FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment

# FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment
Source: [https://arxiv.org/html/2608.14550](https://arxiv.org/html/2608.14550)
\(30 April 2026\)

###### Abstract\.

AI efficiency has recently taken the spotlight in both academy and industry due to massive model scales, high energy demands, and environmental costs\. While reporting Floating Point Operations \(FLOPs\) is a traditional approach for assessing computational costs, the relationship between FLOPs and execution time is not straightforward, as layers with the same number of FLOPs may not have the same execution time because some operations are more easily parallelized than others\. This paper sets out to replicate the original experiments from a study that proposed theα−F​L​O​P​s\\alpha\-FLOPsestimation formula to verify whether the results remain applicable on newer, more powerful hardware\.

During the replication process, we identify limitations in the replication materials provided by the original study, including a lack of specific dependency details and transparency regarding regression data\. Our results validate the thesis that raw FLOPs alone are not an appropriate metric for execution time, as spatial dimensions remain more easily parallelized than kernel dimensions\. However, fine\-grained measurements reveal that the relationship is much less straightforward than previously shown, with newer hardware exhibiting instabilities and discontinuities in execution time, including jumps and oscillations, that theα−F​L​O​P​s\\alpha\-FLOPsformula generally underestimates\. Ultimately, this work validates the empirical findings from the original study but shows negative results when applying theα−F​L​O​P​s\\alpha\-FLOPsestimation\. We also highlight the critical need for complete and accurate replication packages for research on hardware\-dependent efficiency assessment and provide a complete replication package for our implementation to facilitate further study\.

Green AI, Efficiency Assessment, Convolutional Neural Networks, FLOPs, SE for AI

††copyright:acmlicensed††journalyear:2025††doi:XXXXXXX\.XXXXXXX††conference:The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom††ccs:Computer systems organization Neural networks††ccs:Software and its engineering Software performance††ccs:General and reference Measurement††ccs:General and reference Metrics## 1\.Introduction

AI efficiency has recently taken the spotlight in both academy and industry\(Shiet al\.,[2024](https://arxiv.org/html/2608.14550#bib.bib33); Schwartzet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib31); Tan and Le,[2019](https://arxiv.org/html/2608.14550#bib.bib5)\)\. AI capabilities come from massive scales: for example, Large Language Models \(LLMs\) have billions of parameters trained on terabytes of open\-source code from platforms such as GitHub\(Lozhkovet al\.,[2024](https://arxiv.org/html/2608.14550#bib.bib44)\)\. This scale brings significant computational challenges, including the need for dedicated hardware for parallel computation, such as GPUs\(Wuet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib30)\), along with high energy demands and environmental costs\(Schwartzet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib31)\)\. The International Energy Agency projects that AI energy consumption will double over the next 5 years\(International Energy Agency,[2025](https://arxiv.org/html/2608.14550#bib.bib45)\), further straining existing electrical infrastructure and increasing greenhouse gas emissions\. Given this scale, it is highly relevant to put focus on AI optimization to reduce inference time and energy consumption\.

To optimize AI models, developers need accurate metrics to assess relative costs and model efficiency\. While execution time or energy consumption can be accurate metrics, they can be difficult to measure during model development, since they require specialized software profilers and access to bare metal\(Cruzet al\.,[2025](https://arxiv.org/html/2608.14550#bib.bib19); Roque and Cruz,[2025](https://arxiv.org/html/2608.14550#bib.bib81)\)\. The traditional alternative to reporting the computational approach of an AI model is to report Floating Point Operations \(FLOPs\)\(Desislavovet al\.,[2023](https://arxiv.org/html/2608.14550#bib.bib48)\)\. This metric is easy to derive from the model’s architecture by counting the number of matrix multiplications and their dimensions, and gives a sense of computational costs: models with higher FLOPs need to perform more multiplications, which directly affects time and energy consumption\.

However, the relationship between FLOPs and execution time is not as straightforward\. The work by\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)examines this relationship for Convolutional Neural Networks\. Their work runs multiple experiments for different convolutional configurations that introduce FLOPs across different dimensions, such as input size, input and output channels or kernel size\. Their results show that layers with the same number of FLOPs do not necessarily have the same execution time, because some convolutional operations and dimensions are parallelized more easily than others\. To mitigate this, the paper proposes an estimation formula,α−F​L​O​P​s\\alpha\-FLOPs, derived from their empirical measurements\. Theα−F​L​O​P​s\\alpha\-FLOPsformula can provide reasonable execution\-time estimates by running preliminary measurements and performing a regression\.

The findings of their manuscript have valuable implications for Software Engineering for AI \(SE4AI\)\. Providing developers with accurate, hardware\-aware execution\-time estimates supports informed decision\-making during model design\. The proposed approach has the potential to be adapted and extended for different AI architectures to identify similar parallelization patterns in other AI models, and theα−F​L​O​P​s\\alpha\-FLOPsformula could be extended to account for these different architectures\. However, one of the limitations intrinsic to this approach is that it is highly hardware\-dependent\. The method requires preliminary measurements for each specific GPU, and the estimation formula was derived from empirical measurements in the hardware available to the authors\. Given the rapid evolution of AI\-specific hardware, some of the original findings may no longer be applicable due to new hardware optimizations or architectures\.

Therefore, we set out to replicate the original experiments from the study to verify whether the results remain applicable\. During the replication process, we identified limitations in the replication materials provided by the study\. The provided replication package contains only the data and code to plot it, but no code to run experiments\. Additionally, there are other limitations, such as the absence of specific dependency details \(e\.g\., AI libraries and GPU drivers\), a lack of transparency about the data used for regression, and inconsistencies between the paper details and practical implementations\.

Concretely, we define the following replication objectives:

- •RO1\.Replicate the finding that raw FLOPs are not a sufficient proxy for CNN execution time, even when configurations have equal theoretical FLOPs\.
- •RO2\.Replicate the finding that execution time depends on where FLOPs are introduced, with FLOPs in spatial dimensions being more parallelizable than FLOPs in kernel dimensions\.
- •RO3\.Replicate the finding that theα\\alpha\-FLOPs correction can provide accurate execution\-time estimates after hardware\-specific regression\.

The contributions of this paper can be summarized as follows: \(1\) replicate the experiments from the original study\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)in newer hardware and identify how efficiency patterns change with hardware evolution; \(2\) replicate theα−F​L​O​P​s\\alpha\-FLOPsestimation method and evaluate its validity for our hardware; \(3\) analyze the impact of poor replication packages when attempting to reproduce hardware\-dependent results; and \(4\) a complete replication package of the original experiments, available in Zenodo111[https://doi\.org/10\.5281/zenodo\.18842300](https://doi.org/10.5281/zenodo.18842300)\.

## 2\.Background

The intuition behind using FLOPs for comparing costs of neural network models is simple: the model needs to perform a certain number of operations for a forward pass, and each of these operations takes some amount of timetF​L​O​Pt\_\{FLOP\}\.

\(1\)ET​o​t​a​l=tF​L​O​P⋅F​L​O​P​sE\_\{Total\}=t\_\{FLOP\}\\cdot FLOPs
However, this approach is naive\. Mainly, it does not account for the effect of parallelization\. When running multiple operations in parallel, their costs are not additive\. Instead, depending on the nature of the operations and the capacity of the GPU, parallel multiplications will take as much time and energy as a single multiplication\. According to Gustafson’s law\(Gustafson,[1988](https://arxiv.org/html/2608.14550#bib.bib14)\), for a task with a sequential and a parallel component, the sequential component takes the same time, while the parallel component’s cost decreases with hardware capacity\. Therefore, different neural networks with the same theoretical FLOP count can have different energy consumption\.

The work by\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)shows this in practice for shows\. In their study, the authors measure the execution times of convolutional layers across different hyperparameter combinations\. For CNNs, these hyperparameters are the input dimensions, the number of channels, and the kernel size\. In their experiments, they measure forward time for different hyperparameter combinations, increasing some parameters while reducing others, to control and maintain the same number of FLOPs\.

Results show that layers with the same theoretical FLOPs exhibit vastly different execution times, depending on the hyperparameters and the dimension at which those FLOPs are applied\. They observe that FLOPs are easily parallelizable for spatial dimensions\. They find a significant reduction in execution time as the input size increases and the channel size decreases\. On the other hand, increasing the kernel dimension while reducing the size or the number of channels leads to higher execution time for layers with the same FLOPs\.

Based on their results, the study proposes a correction for FLOPs, where the estimated execution time for a layer is expressed as:

\(2\)T=αk​\(S\)⋅F​L​O​P​ST=\\alpha\_\{k\}\(S\)\\cdot FLOPS\(3\)αk​\(S\)=\(Sk\+βk​\(S−Sk\)S\)γk\\alpha\_\{k\}\(S\)=\\left\(\\frac\{S\_\{k\}\+\\beta\_\{k\}\(S\-S\_\{k\}\)\}\{S\}\\right\)^\{\\gamma\_\{k\}\}
In this formula,SkS\_\{k\}is defined as “a parameter that grows slowly with K, withS1=1S\_\{1\}=1”\. Parametersβk\\beta\_\{k\}andγk\\gamma\_\{k\}are determined via regression using empirical data from different convolutional configurations\. These parameters are hardware\-dependent, but the paper notes that, based on a preliminary study,βk\\beta\_\{k\}andγk\\gamma\_\{k\}show little variance across different hardware\. This formula provides reasonable estimates of CNN execution time for the study’s setup\. The paper demonstrates the validity of the estimation by plotting the FLOPs\-based estimate alongside the empirical results, showing that the estimate lies close to the original values\.

The contributions of the original paper can be summarized as follows: a\) the paper empirically demonstrates that FLOPs alone are not a good measure of CNN execution time\. b\) the relationship between execution time and FLOPs depends on the specific hyperparameter configuration, and how parallelizable those FLOPs are\. c\) provides an empirically obtained formula that can estimate the execution time of a convolutional layer based on FLOPs and hyperparameters\. In addition to the paper and its contributions, the authors provide a dataset with the data for the measurements presented in the manuscript222[https://github\.com/asperti/alpha\_flops\_dataset](https://github.com/asperti/alpha_flops_dataset)\. These contributions paint a clear picture of FLOPs’ limited validity as a metric for assessing the computational cost of an AI model\. The correction formula provided can also be adapted and applied to different AI architectures, which motivates our attempt to replicate the results\.

## 3\.Limitations to Replicability

The findings of the original study are valuable for the field of efficiency and energy assessment of AI and CNNs\. A similar empirical methodology could be extended and applied to different AI architectures to identify FLOP and execution\-time patterns across their hyperparameters\. However, since the approach is empirical and hardware\-dependant, we need to verify that it remains applicable to a newer GPU, motivating this replication\.

During this replication, we identified several limitations in the replication materials provided in the original paper\. The dataset provided partially covers the necessary information, but it is far from a replication package, as it contains only the data needed to recreate the plots\. The authors mention that they “ do not provide the obvious code for timing execution or for regression over data ”\. Additionally, the paper does not include other relevant details for replication, such as the library or driver versions used in the experiment, code or details for crucial steps in this methodology, the data used for regression, and vague definitions of certain parameters in the formula, among others\.

This section presents in detail the limitations encountered when attempting to replicate the results of this paper on newer hardware and how these limitations hinder replicability\. These limitations will inform the methodology for our own replication, which will include several assumptions to address the missing information\.

##### Lack of dependency specification

Neither the paper nor the dataset specifies which of the two major AI libraries \(TensorFlow or PyTorch\) is used to implement the convolutional layers, or the versions used\. Additionally, the authors do not mention GPU drivers or CUDA versions used\. All of these details introduce variability in the methodology\. Newer AI libraries might introduce optimizations on how layers and matrices are processed, reducing execution time for the same hardware\.

##### Missing dataset and configuration for regression

Before being able to compute time estimations, one needs to define the hardware\-specific parametersβk\\beta\_\{k\}andγk\\gamma\_\{k\}from theα−F​L​O​P​s\\alpha\-FLOPsfunction in[Equation 3](https://arxiv.org/html/2608.14550#S2.E3)\. To do this, the original study uses nonlinear regression on data collected from a sample of convolutional configurations\. The authors define the data as triplets\(S,C,K\)\(S,C,K\), along with the execution timeTT, whereSSis the input size,CCis the channel size, andKKis the kernel size\. In this regard, the paper lacks information about the sampling strategy used or the range of values tested for these parameters\. This definition also does not clarify the range for the 2D values\. For a 2D convolution, size S is defined by width and height,S=W×HS=W\\times H; the number of input and output channels can also be different,C=Ci​n×Co​u​tC=C\_\{in\}\\times C\_\{out\}, and the kernel size is also 2\-dimensional,K=K1×K2K=K\_\{1\}\\times K\_\{2\}\. Matrix multiplication efficiency in t a GPU depends not only on the total number of elements but also on the dimensionality\(Hendersonet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib2)\)\. Additionally, the paper does not provide information on the configuration used for the non\-linear regression, such as the library or functions used, the initial values forβk\\beta\_\{k\}andγk\\gamma\_\{k\}, and the lower and upper bounds for these parameters\.

##### Inconsistencies between paper and code

The dataset provided with the paper includes a function to compute execution\-time predictions using theα−F​L​O​P​s\\alpha\-FLOPsformula, with the values reported in the paper\. However, this computation contains two inconsistencies with respect to the formulas stated in the manuscript\. First, the number of FLOPs for an input of width and heightW,HW,H, and input channelsCi​nC\_\{in\}, and a convolutional layer with kernel sizeK1×K2K\_\{1\}\\times K\_\{2\}and output channelsCo​u​tC\_\{out\}is computed asF​L​O​P​s=𝟐⋅K1⋅K2⋅Ci​n⋅W⋅H⋅Co​u​tFLOPs=\\mathbf\{2\}\\cdot K\_\{1\}\\cdot K\_\{2\}\\cdot C\_\{in\}\\cdot W\\cdot H\\cdot C\_\{out\}\. However, the function provided in the artifact does not use the constant term22\. While this is just a scaling factor, it can lead to inconsistencies if it is used in the regression but not in the estimations\. Second, after the computationT=αk​\(S\)⋅F​L​O​P​ST=\\alpha\_\{k\}\(S\)\\cdot FLOPS, the code multiplies this value by a scaling factor called*final*,T=f​i​n​a​l⋅TT=final\\cdot T\. This value is set tof​i​n​a​l=0\.0375final=0\.0375in the artifact\. There is no justification for it\. We conjecture that this is an additional regression parameter or a manually tuned value to adjust the prediction to the real measurements as closely as possible\.

## 4\.Methodology

Our replication is based on the following three steps defined in the original study: \(1\) data collection of execution times for different convolutional configurations; \(2\) regression over the collected data to determine the parameters for theα−F​L​O​P​s\\alpha\-FLOPsformula in our hardware; \(3\) time measurements for the experiments reported in the original paper, with estimations from theα−F​L​O​P​s\\alpha\-FLOPsformula\.

This section introduces the parameter settings for the three steps and the experimental setup\. These settings follow the instructions in the paper and the provided dataset as closely as possible\. When details from available sources are lacking, we make assumptions when selecting parameters and report them in the text\.

### 4\.1\.Parameter setting

##### Data collection

Using our own GPU, we collect execution time data for a set of convolutional configurations based on the following sampling strategy:

- •Input and output sizesW,H∈\{1,2,4,8,16,32,64,96,128,256\}W,H\\in\\\{1,2,4,8,16,32,64,96,128,256\\\}\. We test every possible combination, not only square matrices\. Input and output have the same dimensionality, as per the original paper\.
- •Input and output channelsCi​n,Co​u​t∈\{10⋅x\},x∈\[1,9\]∪\{100\+50⋅x\},x∈\[0,20\]∪\{2x⋅100\},x∈\{4,5\}C\_\{in\},C\_\{out\}\\in\\\{10\\cdot x\\\},x\\in\\left\[1,9\\right\]\\cup\\\{100\+50\\cdot x\\\},x\\in\\left\[0,20\\right\]\\cup\\\{2^\{x\}\\cdot 100\\\},x\\in\\\{4,5\\\}\. We select these values because they fall within the range of channel sizes tested across the different experiments\.
- •Kernel sizesK∈\[1,14\]K\\in\\left\[1,14\\right\]\. We test only for square kernels, since all the original experiments did so, although this is not explicitly defined in the original paper\.

To obtain execution time, we run each configuration 50 times and take the average execution time\.

Assumption 1:The minimum and maximum values selected for the regression were selected based on the experiments shown in the original paper\. These ranges cover most of the configurations that will be later analyzed during the experiments\. The replication material does not provide any information on the measurement ranges used for their regression, neither in the paper nor in the dataset\.

##### Configuration for the regression

To perform the regression using theα−F​L​O​P​s\\alpha\-FLOPsformula, we use the least\-squares approach\. This approach minimizes the Mean Squared Error between measurements and an arbitrary function, in this case, theα−F​L​O​P​s\\alpha\-FLOPsformula\. We define the parameters to fitβk\\beta\_\{k\}andγk\\gamma\_\{k\}\. We also add an additional scaling factor,f​i​n​a​lkfinal\_\{k\}, as a regression parameter, which multiplies the original formula\. The updated formula used for regression looks like

αk​\(S\)=f​i​n​a​lk⋅\(Sk\+βk​\(S−Sk\)S\)γk\\alpha\_\{k\}\(S\)=final\_\{k\}\\cdot\\left\(\\frac\{S\_\{k\}\+\\beta\_\{k\}\(S\-S\_\{k\}\)\}\{S\}\\right\)^\{\\gamma\_\{k\}\}To perform the regression, the function used iscurve\_fitfrom thescipypackage\. It requires initial parameter values, as well as lower and upper bounds\. Based on the values from the original paper, we set up the initial values toβk=0\.01\\beta\_\{k\}=0\.01,γk=0\.8\\gamma\_\{k\}=0\.8, andf​i​n​a​lk=0\.3final\_\{k\}=0\.3\. We define lower and upper bounds for all three parameters to\[0,2\]\[0,2\]\. Similarly to the original paper, we perform two separate regressions: one for convolutional layers withK=1K=1and another forK\>1K\>1\.

Assumption 2: We choose the non\-linear least\-squares based on the regression definition given in the paper and email communication with the authors\. We believe this approach is the one used for the study, but it remains vague in the original material\. Thecurve\_fitfunction from thescipypackage was chosen based on data\-science standards, and the initial values and bounds were chosen based on the parameter values from the original paper\.

Assumption 3: Thef​i​n​a​lkfinal\_\{k\}parameter is added as a mirror to the one hardcoded in the provided dataset\. Since the original value does not appear in the paper and is not justified in the original code, it is more appropriate to treat it as an additional regression parameter rather than to reuse or manually adjust it\.

ExperimentHHWWKKCi​nC\_\{in\}Co​u​tC\_\{out\}Original Figure\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)A112211𝟐𝐱⋅𝟏𝟎𝟎,𝐱∈\[𝟎,𝟗\]\\mathbf\{2^\{x\}\\cdot 100,x\\in\\left\[0,9\\right\]\}𝟐𝐱⋅𝟓𝟎,𝐱∈\[𝟎,𝟗\]\\mathbf\{2^\{x\}\\cdot 50,x\\in\\left\[0,9\\right\]\}Figure 3b222211𝟐𝐱⋅𝟓𝟎,𝐱∈\[𝟎,𝟗\]\\mathbf\{2^\{x\}\\cdot 50,x\\in\\left\[0,9\\right\]\}𝟐𝐱⋅𝟓𝟎,𝐱∈\[𝟎,𝟗\]\\mathbf\{2^\{x\}\\cdot 50,x\\in\\left\[0,9\\right\]\}444411𝟐𝐱⋅𝟐𝟓,𝐱∈\[𝟎,𝟗\]\\mathbf\{2^\{x\}\\cdot 25,x\\in\\left\[0,9\\right\]\}𝟐𝐱⋅𝟐𝟓,𝐱∈\[𝟎,𝟗\]\\mathbf\{2^\{x\}\\cdot 25,x\\in\\left\[0,9\\right\]\}Bround⁡\(300/K\)\\operatorname\{round\}\(300/K\)round⁡\(300/K\)\\operatorname\{round\}\(300/K\)\[𝟏,𝟑𝟎\]\\mathbf\{\\left\[1,30\\right\]\}\{150,100,70,50\}\\\{150,100,70,50\\\}Co​u​t←Ci​nC\_\{out\}\\leftarrow C\_\{in\}Figure 4aC10101010\[𝟏,𝟑𝟎\]\\mathbf\{\\left\[1,30\\right\]\}round⁡\(R/K\),\\operatorname\{round\}\(R/K\),R∈\{4500,3000,2100,1500\}R\\in\\\{4500,3000,2100,1500\\\}Co​u​t←Ci​nC\_\{out\}\\leftarrow C\_\{in\}Figure 4bD100100\[𝟏,𝟔𝟓𝟎\]\\mathbf\{\\left\[1,650\\right\]\}33\{50,100,150\}\\\{50,100,150\\\}100100Figure 5aE10010010010033\{50,100,150\}\\\{50,100,150\\\}\[𝟏,𝟔𝟓𝟎\]\\mathbf\{\\left\[1,650\\right\]\}Figure 5bF\{100,200,300\}\\\{100,200,300\\\}100100115050\[𝟏,𝟔𝟓𝟎\]\\mathbf\{\\left\[1,650\\right\]\}Figure 6aG1010100100\{1,3,5\}\\\{1,3,5\\\}\[𝟏,𝟏𝟎𝟎𝟎\]\\mathbf\{\\left\[1,1000\\right\]\}100100Figure 6bTable 1\.Parameter setup for the different experiments\. The independent variable is marked in bold\. We also indicate to which figure in the original paper each experiment corresponds\.
##### Measurement set

Table[1](https://arxiv.org/html/2608.14550#S4.T1)shows the different experiments run in this replication\. Each experiment measures execution time for a forward pass across different convolutional configurations, following specific rules for the hyperparameters\. The experiments can be divided in 2 groups:

- •Constant FLOPs: For these experiments, the variables are configured such that some hyperparameters are increased while others are decreased, following a proportion that keeps the FLOPs constant\. This effectively changes the parallelization capabilities between different configurations\. The objective of these experiments is to determine whether configurations with the same FLOPs have the same execution time, regardless of parallelization\. These are experiments A, B, and C\.
- •Increasing FLOPs: In these experiments, two hyperparameters are increased, while the rest are kept constant\. These experiments examine the shape of execution time as certain dimensions are increased, to determine how each dimensions affect final execution times These are experiments D, E, F and G\.

For each experiment, we run each configuration 2000 times and report the average execution time to remove noise\. The configurations for each experiment are taken directly from the dataset provided with the original manuscript, but we extend some of them to make use of the increased capabilities of our hardware\.

Experiment A is extended to test one additional power of 2 in the input channels compared to the original study\.

For the Increasing FLOPs experiments, we measure the whole range of the variable\. This is, from 1 to 650 in D, E, F, and from 1 to 1000 in G\. In the original study, the authors limited their testing to a sample of this space, with equidistant data points, 32 for D, E, F, and 10 for G\.

### 4\.2\.Experimental setup

##### Hardware

Our replication is performed on an AMD Ryzen 7900X CPU and an Nvidia RTX 4090 GPU\. The 4090 was launched in October 2022\. It has a maximum clock speed of 2520 MHz, with 16,384 CUDA cores\. It also has a larger L2 cache of 72MB and 24GB of GDDR6X VRAM\.

Comparatively, the original experiments are run on an Intel Core i7\-9850H CPU and a NVIDIA Quadro T2000\. The Quadro T2000 is a mobile GPU for laptops and is older than the 4090, launched in May 2019\. It has 1,024 CUDA cores, with a maximum frequency of 1785 MHz\. It also has 1MB of L2 cache and 4GB of GDDR5 VRAM\.

##### Library versions

For our replication, we choose PyTorch version 2\.9, with CUDA library 12\.2\. The choice is based on PyTorch’s current popularity and the currently available versions for it and CUDA\. This provides a perspective on CNN efficiency in the current state of the art, both hardware\- and software\-wise, and closer to a real use scenario\.

Assumption 4: We could not identify the Deep Learning library used for the experiments, nor the version of the CUDA libraries, from the manuscript or the dataset\. These versions are likely far from the original versions in the paper\. Since there is no reference in the original library, we choose library versions that give a view of computational efficiency in current\-day scenarios\.

## 5\.Results

### 5\.1\.Regression Results

After running the regression on our collected data, we obtain the following values forβK\\beta\_\{K\},γK\\gamma\_\{K\}, andf​i​n​a​lKfinal\_\{K\}\. The values obtained for our hardware areβ1=0\.00972514\\beta\_\{1\}=0\.00972514,γ1=1\.1779239\\gamma\_\{1\}=1\.1779239andf​i​n​a​l1=0\.00549217final\_\{1\}=0\.00549217forK=1K=1, andβK=0\.00785036\\beta\_\{K\}=0\.00785036,γK=1\.0601711\\gamma\_\{K\}=1\.0601711andf​i​n​a​lK=0\.00401956final\_\{K\}=0\.00401956forK\>1K\>1\.

### 5\.2\.Experiments Results

In the following sections, we present the results for the replications of the original experiments\. We visualize results as plots, with average times on the y\-axis and the independent variable on the x\-axis\. We also show the Root Mean Square Error \(RMSE\) and the normalized RMSE by range for each configuration\. The datapoints do not include error bars, since the Inter Quartile Range of all time measurements is negligible, at least 3 orders of magnitude smaller than actual measurements\.

#### 5\.2\.1\.Constant FLOPs

![Refer to caption](https://arxiv.org/html/2608.14550v1/x1.png)Figure 1\.Experiment A: Execution time and estimations for increasing channels with different spatial dimensions\. Estimations are represented with the dashed line\.[Figure 1](https://arxiv.org/html/2608.14550#S5.F1)shows the forward pass execution time for increasing input and output channels, for configurations from Experiment A\. Each colored line represents a different set of configurations, and dashed lines show the time estimation using theα−F​L​O​P​s\\alpha\-FLOPsformula\. Each of the points in the same vertical line has the same number of theoretical FLOPs: the FLOPs introduced by a higher input size are compensated for by using a lower number of input and output channels\. If FLOPs alone were a good proxy for execution time, all of the points in the same vertical line should take the same amount of time to complete a forward pass\. Instead, the figure shows that FLOPs are more computationally intensive when introduced in the channel dimension, whereas they can be more easily parallelized across spatial dimensions\. The estimation provided by theα−F​L​O​P​s\\alpha\-FLOPsformula lies close to the actual measurements, within∼5%\\sim 5\\%difference, overestimating for smaller input sizes and underestimating for larger inputs\. RMSE is between0\.020\.02and0\.040\.04ms, or between2\.4%−13%2\.4\\%\-13\\%normalized for the measurement range for the three configurations which is acceptable given that measurements lie between0to1\.51\.5ms\.

These results are consistent with those of the original study\. For the original experiments, execution times range from∼0\.2\\sim 0\.2to88milliseconds due to significantly less powerful hardware, while our results show a∼10×\\sim 10\\timesspeedup\. However, the evolution of execution time with increasing size follows the same pattern and aligns well with the original estimation function\. In this scenario, our setup behaves as observed in the original study\.

![Refer to caption](https://arxiv.org/html/2608.14550v1/x2.png)Figure 2\.Experiment B: Execution time and estimations for increasing kernels and decreasing spatial dimensions\. Estimations are represented with the dashed line\.[Figure 2](https://arxiv.org/html/2608.14550#S5.F2)plots the results for Experiment B\. In this figure, the input size is fixed, and the kernel sizeKK\(x\-axis\) increases while the input size dimensions decrease at a fixed ratio, maintainingH=W=300/KH=W=300/K\. Each configuration set, represented by a color, uses a different number of input and output channels\. Thanks to the ratio rule, each point of a configuration set has the same theoretical FLOPs\. If FLOPs alone provided a good estimate of execution time, each colored line would be approximately horizontal, with FLOPs increasing only as the number of channels increases\. Instead, the execution time increases consistently asKKincreases and the number of dimensions decreases, following an approximately quadratic trend\. This again supports the original findings that convolutional operations are more readily parallelizable across spatial dimensions than across kernel dimensions\. One exception to this pattern occurs forCi​n=Co​u​t=150C\_\{in\}=C\_\{out\}=150, which exhibits a notable reduction in execution time for valuesK\>19K\>19, withH=W=round⁡\(300/19\)=16H=W=\\operatorname\{round\}\(300/19\)=16\. At this point, the execution time decreases by approximately47%47\\%, from0\.69​ms0\.69\\,\\text\{ms\}to0\.44​ms0\.44\\,\\text\{ms\}, after which the trend continues from this lower baseline, remaining below the curve forCi​n=Co​u​t=100C\_\{in\}=C\_\{out\}=100, despite the latter being theoretically less computationally demanding\. The estimates provided by theα\\alpha\-FLOPs formula reproduce the qualitative shape of the empirical data but fail to match its scale, remaining distant from the measured results\. RMSE is between0\.20\.2and0\.60\.6ms, corresponding to around47%47\\%deviation from the actual measurement range\.

Compared to the results from the original study, our empirical measurements follow similar trends, albeit with the same∼10×\\sim 10\\timesspeedup\. However, we did not achieve the same level of accuracy in our hardware for theα−F​L​O​P​s\\alpha\-FLOPsestimations\. Additionally, the decrease observed for theCi​n=Co​u​t=150C\_\{in\}=C\_\{out\}=150does not happen in the original study\.

![Refer to caption](https://arxiv.org/html/2608.14550v1/x3.png)Figure 3\.Experiment C: Execution time and estimations for increasing kernels and decreasing channels\. Estimations are represented with the dashed line\.[Figure 3](https://arxiv.org/html/2608.14550#S5.F3)shows the results for Experiment C\. Similarly to the previous experiment, the kernel size \(x\-axis\), but now the input size is fixed, while the number of input and output channels decreases at a fixed ratioRR, maintainingC⋅K=RC\\cdot K=R\. This way, each set of configurations, represented by a color, has the same theoretical FLOPs, which increase only as the ratio increases\. Sets with higher ratios have higher theoretical FLOPs, and their execution times should consistently exceed those of smaller configurations\. However, the results for these experiments are inconsistent\. While the most expensive set \(C⋅K=4500C\\cdot K=4500\) is the most expensive for lower values ofKK, it gets overtaken by smaller sets for larger values ofKK\. The estimation for these runs consistently underestimates execution time and does not adapt well to measurement evolution\.

The results of this experiment differ from those of the original study\. The behavior of these convolutional configurations observed in the previous study is much more stable\. The execution time for increasing kernel sizes and decreasing channels shows approximately constant behavior, close to the theoretical behavior, and the differences between sets with differentRRvalues are consistent, with lowerRRvalues remaining below largerRRvalues\.

#### 5\.2\.2\.Increasing FLOPs Experiments

![Refer to caption](https://arxiv.org/html/2608.14550v1/x4.png)\(a\)Limited sampling: 33 datapoints
![Refer to caption](https://arxiv.org/html/2608.14550v1/x5.png)\(b\)Full sampling: 649 datapoints

Figure 4\.Experiment D: Execution time and estimations for increasing input size and different input channels\. Estimations are represented with the dashed line\.[Figure 4](https://arxiv.org/html/2608.14550#S5.F4)presents the results for Experiment D, where one input dimensionWWacts as the independent variable, for three different values for input channels, and with the rest of the hyperparameters fixed\.[4\(a\)](https://arxiv.org/html/2608.14550#S5.F4.sf1)shows the results using the original resolution, withWWincreased in steps of 20 Considering only theoretical FLOPs as a proxy, execution time should increase linearly with FLOPs\. Instead, in these configurations, execution time seems to increase in discrete steps for specific values ofWW, with a small linear increase between steps\. The estimate from ourα−F​L​O​P​s\\alpha\-FLOPsimplementation underestimates the actual measurements by about50%50\\%, though it follows the same increasing trend\. RMSE is up to1\.51\.5ms for the worst case, and predictions deviate between39%−56%39\\%\-56\\%according to normalized RMSE\.

[4\(b\)](https://arxiv.org/html/2608.14550#S5.F4.sf2)shows the same setup with a higher resolution of samples, measuring for everyWWvalue between 1 and 650\. This figure shows that the increase in execution time is not as stable as it might seem when sampling only a few values\. This resolution helps identify where the step jumps occur, with the largest ones occurring for valuesW=\{164,246,328,492\}W=\\\{164,246,328,492\\\}\. The figure also reveals oscillations for large values ofCi​n=150C\_\{in\}=150, where convolutional configurations for certain values ofWWperform faster than others\.

Empirical measurements from the previous study for this experiment are closer to the theoretical model\. In the original experiment, execution time increases linearly with input sizeWW\. The two larger configurations show a similar step\-up forW=420W=420\. In comparison, our results are more erratic, both at the lower resolution used in the original paper and at our extended resolution\.

![Refer to caption](https://arxiv.org/html/2608.14550v1/x6.png)\(a\)Limited sampling: 33 datapoints
![Refer to caption](https://arxiv.org/html/2608.14550v1/x7.png)\(b\)Full sampling: 649 datapoints

Figure 5\.Experiment E: Execution time and estimations for increasing output channels and different input channels\. Estimations are represented with the dashed line\.[Figure 5](https://arxiv.org/html/2608.14550#S5.F5)shows the results of Experiment E\. For this experiment, the independent variable is the number of output channelsCo​u​tC\_\{out\}, for three different values of input channelsCi​nC\_\{in\}\.[5\(a\)](https://arxiv.org/html/2608.14550#S5.F5.sf1)shows a lower resolution with 33 datapoints\. The behavior is similar to the previous figure: execution time increases stepwise for certain values ofCo​u​tC\_\{out\}, with a marginal increase between steps\. Theα−F​L​O​P​s\\alpha\-FLOPsformula once again underestimates real execution time, showing larger errors for smaller layers but better predictions for larger configurations, with normalized RMSE ranging from19%19\\%to64%64\\%\.

[5\(b\)](https://arxiv.org/html/2608.14550#S5.F5.sf2)shows the maximum resolution with all possible values ofCo​u​tC\_\{out\}\. The configurations in this experiment are much more unstable than in the previous experiment and reveal behaviors that are hidden at lower resolution\. In this case, the values ofCo​u​tC\_\{out\}that trigger the larger step ups areCo​u​t=\{101,192,257,385,513\}C\_\{out\}=\\\{101,192,257,385,513\\\}\. Additionally, two bands of values are distinguishable for the three values ofCi​nC\_\{in\}at every step\. Zooming in on the figure shows that the lower band is formed by values ofCo​u​tC\_\{out\}that are multiples of 4\. These configurations are executing between17−20%17\-20\\%faster than the rest of the configurations\.

The previous study presents similar results for this setup, with execution time increasing stepwise at approximately the same rate\. Compared to our estimates, the original results overestimate the actual execution time of the convolutional configurations\.

![Refer to caption](https://arxiv.org/html/2608.14550v1/x8.png)\(a\)Limited sampling: 33 datapoints
![Refer to caption](https://arxiv.org/html/2608.14550v1/x9.png)\(b\)Full sampling: 649 datapoints

Figure 6\.Experiment F: Execution time and estimations for increasing output channels and different input sizes\. Estimations are represented with the dashed line\.[Figure 6](https://arxiv.org/html/2608.14550#S5.F6)displays the results of Experiment F\. In this experiment, the independent variable is again the number of output channelsCo​u​tC\_\{out\}, but tested for different input heightsHH, with a kernel sizeK=1K=1\.[6\(a\)](https://arxiv.org/html/2608.14550#S5.F6.sf1)presents the results for the limited sampling of 33 datapoints\. In this configuration, the forward pass time increases consistently with both the number of output channelsCo​u​tC\_\{out\}and the input heightHH\. The accuracy of theα−F​L​O​P​s\\alpha\-FLOPsestimation is much worse than seen in previous experiments\. Here, the formula underestimates execution time by a wide margin across all three sets of configurations, with values close to0\. Normalized error ranges from48%48\\%to as high as82%82\\%of the actual measurements

The full\-resolution results in[6\(b\)](https://arxiv.org/html/2608.14550#S5.F6.sf2)reveal less instability than those of previous experiments in this category\. ForH=300H=300, the execution time follows a linear trend, but shows a significant jump in latency afterC​o​u​tCoutexceeds 500\. In contrast, for H=100 and H=200, the execution time remains relatively linear across wide ranges ofCo​u​tC\_\{out\}, with some jumps in latency at certain values that return to the original trend\.

These results are slightly more stable than those from the original experiments\. The original study shows approximately linear behavior, with small step\-ups similar to those observed in the previous experiments, though smaller in magnitude\.

![Refer to caption](https://arxiv.org/html/2608.14550v1/x10.png)\(a\)Limited sampling: 9 datapoints
![Refer to caption](https://arxiv.org/html/2608.14550v1/x11.png)\(b\)Full sampling: 1000 datapoints

Figure 7\.Experiment G: Execution time and estimations for increasing input channels and different kernel sizes\. Estimations are represented with the dashed line\.[Figure 7](https://arxiv.org/html/2608.14550#S5.F7)displays the results of Experiment G\. In this scenario, the independent variable is the number of input channelsCi​nC\_\{in\}, tested across three different kernel sizesK∈\{1,3,5\}K\\in\\\{1,3,5\\\}, while maintaining a small spatial resolution ofH=W=10H=W=10\.

[7\(a\)](https://arxiv.org/html/2608.14550#S5.F7.sf1)illustrates the low\-resolution sampling with only 10 datapoints\. The execution time forK=1K=1is nearly negligible and remains constant regardless ofCi​nC\_\{in\}size\. For larger kernels, the execution time increases but shows unexpected non\-linearities\. For example, execution time suddenly improves atCi​n=500C\_\{in\}=500for bothK=3K=3andK=5K=5\. The estimation formula fails to capture these hardware\-specific variations and generally underestimates the computational cost for K=3 and K=5\.

[7\(b\)](https://arxiv.org/html/2608.14550#S5.F7.sf2)presents the full sampling with 1000 datapoints, and reveals vast instabilities that are not visible with the original resolution\. The instability is highly pronounced forK=5K=5\. For lower values ofCi​nC\_\{in\}, the execution time of these configurations divides roughly into two bands, with multiples of 4 executing slightly faster, as seen for experiment E\. However, for values aboveCi​n≈400C\_\{in\}\\approx 400, measurements are widely scattered across multiple efficiency tiers\. Another discontinuity in execution time is visible forK=3K=3atCi​n≈480C\_\{in\}\\approx 480, where the execution time abruptly drops, and the two previously mentioned efficiency bands become visible\.

## 6\.Discussion

### 6\.1\.Estimating time through FLOPs

Based on the results and our replication objectives, we achieved a partially successful replication\. With respect toRO1, we replicate the empirical results showing that, in our setup, FLOPs is not a reliable proxy for convolutional\-layer execution time\. The first three experiments \(Experiments A, B, and C\), in which FLOPs are held constant across configurations, show that execution times for configurations with the same FLOPs are not always equal\. Significant variance in execution time is introduced depending on which dimensions are increased and how effectively the GPU can parallelize these operations\. In general, as observed in our results and in those of the previous paper, spatial dimensions are more easily parallelized than kernel dimensions\. The results from the Increasing FLOPs experiments \(Experiments D through G\) also show that execution time does not always increase linearly with FLOPs, despite the theoretical equation for FLOPs suggesting the opposite\. The results of our replication experiments validate the thesis of previous work\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)that using raw FLOPs as a direct metric for execution time or the computational cost of a convolutional layer is not appropriate\.

Despite these findings being available for several years, FLOPs is still a widely used metric when reporting computational costs and model complexity of AI models, both in Computer Vision with CNNs\(Yanget al\.,[2017](https://arxiv.org/html/2608.14550#bib.bib64); Menget al\.,[2024](https://arxiv.org/html/2608.14550#bib.bib65); Yeomet al\.,[2021](https://arxiv.org/html/2608.14550#bib.bib66); Sainet al\.,[2025](https://arxiv.org/html/2608.14550#bib.bib67)\)and in newer fields such as LLM research\(Liuet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib69); Jiaoet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib70); Liuet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib71); Liet al\.,[2021](https://arxiv.org/html/2608.14550#bib.bib72); Feduset al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib73); Duet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib74); Huanget al\.,[2024](https://arxiv.org/html/2608.14550#bib.bib75); Zhanget al\.,[2024](https://arxiv.org/html/2608.14550#bib.bib76)\)\. This happens not only in research, but also for commercial products or platforms for model sharing, such as HuggingFace\(Jernite and Luccioni,[2026](https://arxiv.org/html/2608.14550#bib.bib9)\)\. The reason is simple: FLOPs is an easy\-to\-calculate metric that can be computed even without running the model, and intuitively, the more operations an AI model needs to perform, the more time, energy, and resources it takes\. As shown by these results, practitioners need to be careful when doing these approximations\.

Considering ourRO2, we partially replicate the overall trend from the original study: spatial FLOPs are generally easier to parallelize than kernel FLOPs\. However, the results of the experiments also reveal that the relationship between FLOPs and execution time is much less straightforward than shown in the previous study\. Fine\-grained measurements for different values of the independent variables show that efficiency can vary greatly between consecutive values\. Execution time, in some cases, evolves in jumps rather than uniformly and shows patterns that depend on input dimensions\. Many of these jumps or behaviors appear in multiples of certain numbers, such as 4, 32, 41, or 64, as shown in experiments D and E\.

The most likely explanation is that the patterns derive from both hardware and software characteristics\. GPUs perform multiple optimizations to improve parallelization\. Some examples are Spatial Tiling for caching\(van Werkhovenet al\.,[2014](https://arxiv.org/html/2608.14550#bib.bib8)\)or thread parallelization with streaming multiprocessors\(Tasoulas and Anagnostopoulos,[2019](https://arxiv.org/html/2608.14550#bib.bib7)\)\. The GPU’s specs, such as cache, memory, or bus sizes, also play a role\. The RTX 4090 has 72 MB of L2 cache and a 384\-bit bus\(Luoet al\.,[2024](https://arxiv.org/html/2608.14550#bib.bib6)\)\. A small increase in input size can cause the cache or bus to fill up, requiring additional memory reads and/or writes\.

Software causes for these patterns can be differences in the implementations of convolution algorithms\(Cheng and Parhi,[2020](https://arxiv.org/html/2608.14550#bib.bib10)\)\. The NVIDIA CUDA libraries include multiple algorithms for convolution operations\. Some of these implementations prioritize computational efficiency or memory efficiency\. Depending on input size, the CUDA libraries can select one of the multiple implementations, which can affect execution time\.

Finally, with respect toRO3, we do not manage to replicate accurate estimations using theα−F​L​O​P​s\\alpha\-FLOPsapproach\. Theα−F​L​O​P​s\\alpha\-FLOPsformula is an attempt to mathematically model the non\-linearity observed in the convolutional experiments\. The formula accounts for the greater parallelization of spatial dimensions and the additional costs associated with higher kernels\. Our results show limited success in this area\. While the estimations work well for a couple of experiments, most configurations are underestimated\.

The limitations of the formula are evident in its components\. The correction only considers the input sizeSSand kernel sizeKK, but does not account for the impact of input and output channels\. Our results show that these two factors play an important role in execution time for our setup, exceeding the effect of the single observer in the original study\. If the authors did not observe these effects in their hardware, they did not account for them, making the formula obsolete for newer GPUs\.

Potentially, theα−F​L​O​P​s\\alpha\-FLOPsformula could be extended and adapted, based on the novel behaviors observed in our results, including variables to account for the effect of input and output channels\. However, it would not be possible to design a formula that accounts for edge cases related to the software and hardware optimizations we have discussed, due to the number of dependencies introduced by different software libraries or hardware configurations\.

### 6\.2\.Replication Material

Ultimately, we partially reproduced the results of the previous study\. Some of the reasons for replication failures are likely due to differences in hardware\. However, the lack of replication materials in the original paper significantly undermines the results\. The multiple assumptions we had to make about some of the authors’ choices introduce confounding factors, preventing us from separating which differences arise from hardware or from setup discrepancies\.

For the empirical measurements, we obtain values that are consistently∼10×\\sim 10\\timesfaster than in the original study\. The main reason for this speedup is differences in GPU hardware, including available cores, memory, and compute units\. For example, the Ada Lovelace architecture of the RTX 4090 includes compute units to perform operations on TF32 floating\-point numbers, a format with a reduced mantissa designed for AI workloads\(Halbiniaket al\.,[2025](https://arxiv.org/html/2608.14550#bib.bib11)\), which is not available in the Turing architecture in the original paper\. This format can give up to3−4×3\-4\\timesspeedup for AI workloads\. Combined with other architectural improvements, the hardware factor is more likely to explain the overall speedup observed in the measurements\.

Aside from the time measurement magnitudes, the behavior observed in the experiments for the different convolutional configurations on our hardware only partially resembles the original results\. For Experiments A, E, and F, the trends observed for execution time are similar to the original results\. For the remaining experiments, while trends are mostly similar, we obtain more unstable results\. This might be explained again by differences in hardware\. This is likely the case for experiments B and C, where we used the exact same values as the original study\.

However, thanks to the higher resolution of the independent variable in the Increasing FLOPs experiments, we observe high execution\-time variance across very similar convolutional configurations\. Using the lower\-resolution version from the original paper, this variance is hidden, and the results are much closer to the original study\. Therefore, it is not possible to determine whether this variance was also present in the hardware used in the original paper or was introduced by our newer architecture\.

The main negative result in our replication is the usage ofα−F​L​O​P​s\\alpha\-FLOPsto estimate energy consumption\. Except for two experiments, the estimates fall short of the actual time values\. There are two possible reasons to explain this\. First, the formula itself might not adapt well to the significantly faster timings observed in our GPU\. The formula is not linear, as it includes polynomial terms due toγk\\gamma\_\{k\},KK, andSSin the denominator\. Therefore, the formula might lose resolution for lower timings\.

A second reason for the negative results in the estimation can be that we did not perform the regression as the original authors did\. In this scenario, the results could potentially be reproduced, but since the replicability package was missing, some of the assumptions we made could be incorrect, tainting the results\. The assumptions are threats to the validity of the replication and could have affected the results in the following way\.

Assumption 1: We define the range of hyperparameters for the regression dataset based on the experiments to replicate\. Since the search space for all parameter combinations is very large, we had to select a sampling strategy\. It is possible that the sampling strategy we chose differs too much from the original sampling\. Given the high variance observed during the measurements, it is possible that the distribution of time measurements in the regression dataset differs significantly from that used in the original paper\.

Assumption 2: The original study provided vague details regarding the non\-linear regression implementation\. While we set up the configuration as close as possible to the original paper, by using thescipy\.curve\_fitlibrary and setting specific initial values, our replication for the regression might have converged on a different local minimum\. This could have contributed to the consistent underestimation of execution times observed in the experiments\.

Assumption 3: In the original dataset, a hardcodedf​i​n​a​lfinalscaling factor was used without justification in theα−F​L​O​P​s\\alpha\-FLOPscomputation\. Treating this as an additional regression parameter in the replication was intended to allow the formula to adapt to the RTX 4090’s significantly higher computing power, rather than reusing values without justification\. However, this additional degree of freedom may have allowed the model to overfit to the average latency of the regression set, masking the formula’s inability to capture how individual hyperparameters—specifically kernel size and channel counts—interact on newer GPU architectures\.

Assumption 4: Our choice of library versions probably had an effect on time measurements for both the regression dataset and experiments, due to the different optimizations introduced since the writing of the original study\.

Ultimately, this paper highlights the need for complete and accurate replication packages for research on efficiency assessment, especially when this efficiency depends heavily on hardware\. A proper replication package would have removed the need for our assumptions, and all of the negative results observed could be directly attributed to hardware differences\. This would largely facilitate the reuse of techniques and results\. For replication packages that yield positive results, techniques like this could be easily reused and extended, and for negative results, it would be easier to identify outdated components due to hardware evolution and propose potential fixes\.

For these reasons, we provide a complete replication package for our implementation of theα−F​L​O​P​s\\alpha\-FLOPsmethodology\. The original approach has the potential to provide reliable estimations\. Our intention is to facilitate other researchers’ study of these empirical results and the estimation approach on their own hardware, so extensions and fixes for different kinds of hardware can be developed\.

## 7\.Related Work

FLOPs as proxy for AI model efficiency\. The validity of floating\-point operations \(FLOPs\) as a proxy for AI model training time and energy consumption remains an ongoing debate in the field\. FLOPs were first proposed as an energy proxy in the early stages of Green AI\(Schwartzet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib31); Lacosteet al\.,[2019](https://arxiv.org/html/2608.14550#bib.bib82)\)because they can be easily calculated solely from architectural details\.\(Desislavovet al\.,[2023](https://arxiv.org/html/2608.14550#bib.bib48)\)\(Desislavovet al\.,[2023](https://arxiv.org/html/2608.14550#bib.bib48)\)realizes a study on inference time and energy consumption for a set of AI models for Computer Vision and Natural Language Processing tasks\. There, the authors propose using FLOPs as a proxy for computational efficiency, relating the FLOPs a model uses during inference to hardware FLOPs and combining them with GPU hardware specifications, specifically TFLOP/s and power\. However, their study fails to address the gap between logical FLOPs and hardware FLOPs, as it does not account for potential hardware optimizations introduced by GPUs or TPUs\.

Besides the work of\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)that is the main focus of this replication, other works also address this mismatch\. The work of\(Chenet al\.,[2023](https://arxiv.org/html/2608.14550#bib.bib4)\)\(Chenet al\.,[2023](https://arxiv.org/html/2608.14550#bib.bib4)\)reports that reducing FLOPs does not necessarily yield a similar reduction in latency and proposes a novel family of neural networks that, instead of minimizing FLOPs, optimizes their parallelization to extract spatial features more efficiently\. The proposed neural networks achieve36%36\\%higher inference throughput on GPU, and are up to2\.8×2\.8\\timesfaster than comparable neural networks\. Compared to our work, this paper does not focus on execution\-time estimation or reporting, but rather on exploiting higher parallelization\.

\(Hendersonet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib2)\)\(Hendersonet al\.,[2020](https://arxiv.org/html/2608.14550#bib.bib2)\)also investigates this issue for the energy consumption of image classification algorithms\. In their paper, they repeatedly run inference on pre\-trained image classification models and measure FLOPs and energy usage, finding little correlation between the two when comparing different architectures but a significant correlation when comparing models within the same architecture\. They determine that FLOPs can be useful to compare relative ordering between architectural classes, but not as a full proxy for energy consumption\. This work relies on energy measurements using a profiler, which is cumbersome and requires access to bare\-metal hardware, and again does not propose estimation techniques that are more accessible\.

Replication of hardware\-bound benchmarks\. One of the main problems with unreliable replication packages is that, in cases of negative reproducibility, it is hard to determine their origin The work by\(Pearceet al\.,[2025](https://arxiv.org/html/2608.14550#bib.bib1)\)\(Pearceet al\.,[2025](https://arxiv.org/html/2608.14550#bib.bib1)\)describes this problem in HPC benchmarking, where poor replication scripts make it hard to track whether negative results stem from the replication software or from differences in hardware\. They propose an open\-source framework to address this challenge, providing a step\-by\-step methodology for creating replication packages for HPC benchmarks that are repeatable, replicable, and reproducible\. The same principles apply to AI research\. To properly compare the efficiency of AI models across different GPU architectures, replication packages need to be complete and highly reliable, so as to discard any software factors and focus only on hardware differences\.

## 8\.Conclusion

Our replication study confirms that FLOPs are an inadequate metric for directly assessing the execution time or computational cost of AI models, as efficiency varies significantly depending on how effectively operations can be parallelized across different dimensions\. While our experiments on modern hardware \(NVIDIA RTX 4090\) showed a consistent10×10\\timesspeedup compared to the original study’s results, the underlying efficiency trends remained highly hardware\-dependent and subject to non\-linear "jumps" caused by specific GPU architectures and software optimizations\.

With respect to our replication objectives, we provide the following answers:

- •RO1:We replicate previous findings\(Aspertiet al\.,[2022](https://arxiv.org/html/2608.14550#bib.bib49)\)that FLOPs are not a good proxy for CNN execution time\. Across the experiments, configurations with equal FLOPs do not exhibit equal execution time\.
- •RO2:The overall trend from the original study is replicated: spatial FLOPs are generally easier to parallelize than kernel FLOPs\. However, on newer hardware we also observe discontinuities and oscillations that were not clearly visible in the original results\.
- •RO3:In our setup, we do not replicate the result of accurate time estimations\. Theα\\alpha\-FLOPs model underestimates latency in most configurations and does not consistently provide accurate estimates\.

Theα−F​L​O​P​s\\alpha\-FLOPsestimation methodology showed limited success on newer hardware, consistently underestimating latency across most configurations\. This failure seems to stem largely from the formula’s inability to account for the impact of input and output channels, which played a much more significant role in our setup than in the original study\. Furthermore, our high\-resolution measurements revealed that execution time is far more unstable than previously reported\. However, the lack of original execution code and specific dependency details \(such as library and driver versions\) forced us to make several assumptions that introduced confounding factors into our replication\. This means that our conclusions about the methodology for newer hardware might be compromised by one of our assumptions\. Future work on this topic will focus on studying GPU optimization behavior more closely and applying possible fixes to the original approach\.

Ultimately, this work validates the empirical findings from the original study but shows negative results when applying theα−F​L​O​P​s\\alpha\-FLOPsestimation approach\. Our study also highlights the critical necessity for complete and accurate replication packages in AI efficiency research\. To foster transparency and enable the community to extend this estimation approach to newer architectures, we provide a fully documented replication package\. These findings serve as a call to action for researchers to adopt more rigorous replication standards for efficiency assessment of AI\.

## References

- A\. Asperti, D\. Evangelista, and M\. Marzolla \(2022\)Dissecting flops along input dimensions for greenai cost estimations\.InMachine Learning, Optimization, and Data Science,Cham,pp\. 86–100\.External Links:ISBN 978\-3\-030\-95470\-3Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p3.2),[§1](https://arxiv.org/html/2608.14550#S1.p7.1),[§2](https://arxiv.org/html/2608.14550#S2.p3.1),[Table 1](https://arxiv.org/html/2608.14550#S4.T1.5.5.7),[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p1.1),[§7](https://arxiv.org/html/2608.14550#S7.p2.2),[1st item](https://arxiv.org/html/2608.14550#S8.I1.i1.p1.1)\.
- J\. Chen, S\. Kao, H\. He, W\. Zhuo, S\. Wen, C\. Lee, and S\.\-H\. G\. Chan \(2023\)Run, don’t walk: chasing higher flops for faster neural networks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 12021–12031\.Cited by:[§7](https://arxiv.org/html/2608.14550#S7.p2.2)\.
- C\. Cheng and K\. K\. Parhi \(2020\)Fast 2d convolution algorithms for convolutional neural networks\.IEEE Transactions on Circuits and Systems I: Regular Papers67\(5\),pp\. 1678–1691\.External Links:[Document](https://dx.doi.org/10.1109/TCSI.2020.2964748)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p5.1)\.
- L\. Cruz, J\. P\. Fernandes, M\. H\. Kirkeby, S\. Martínez\-Fernández, J\. Sallou, H\. Anwar, E\. Barba Roque, J\. Bogner, J\. Castaño, F\. Castor, A\. Chasmawala, S\. Cunha, D\. Feitosa, A\. González, A\. Jedlitschka, P\. Lago, H\. Muccini, A\. Oprescu, P\. Rani, J\. Saraiva, F\. Sarro, R\. Selvan, K\. Vaidhyanathan, R\. Verdecchia, and I\. P\. Yamshchikov \(2025\)Greening ai\-enabled systems with software engineering: a research agenda for environmentally sustainable ai practices\.SIGSOFT Softw\. Eng\. Notes50\(3\),pp\. 14–23\.External Links:ISSN 0163\-5948,[Link](https://doi.org/10.1145/3743095.3743099),[Document](https://dx.doi.org/10.1145/3743095.3743099)Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p2.1)\.
- R\. Desislavov, F\. Martínez\-Plumed, and J\. Hernández\-Orallo \(2023\)Trends in ai inference energy consumption: beyond the performance\-vs\-parameter laws of deep learning\.Sustainable Computing: Informatics and Systems38,pp\. 100857\.External Links:ISSN 2210\-5379,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.suscom.2023.100857),[Link](https://www.sciencedirect.com/science/article/pii/S2210537923000124)Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p2.1),[§7](https://arxiv.org/html/2608.14550#S7.p1.1)\.
- N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong,et al\.\(2022\)GLaM: efficient scaling of language models with mixture\-of\-experts\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 5547–5569\.External Links:[Link](https://proceedings.mlr.press/v162/du22c.html)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.J\. Mach\. Learn\. Res\.23,pp\. 120:1–120:39\.External Links:[Link](https://jmlr.org/papers/v23/21-0998.html)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- J\. L\. Gustafson \(1988\)Reevaluating amdahl’s law\.Commun\. ACM31\(5\),pp\. 532–533\.External Links:ISSN 0001\-0782,[Link](https://doi.org/10.1145/42411.42415),[Document](https://dx.doi.org/10.1145/42411.42415)Cited by:[§2](https://arxiv.org/html/2608.14550#S2.p2.1)\.
- K\. Halbiniak, K\. Rojek, R\. Wyrzykowski, P\. Gepner, and N\. Meyer \(2025\)Influence of mixed precision on performance and accuracy of dnn training for ai\-accelerated cfd simulations on nvidia multi\-gpu system\.InComputational Science – ICCS 2025 Workshops,M\. Paszynski, A\. S\. Barnard, and Y\. J\. Zhang \(Eds\.\),Cham,pp\. 142–157\.External Links:ISBN 978\-3\-031\-97554\-7Cited by:[§6\.2](https://arxiv.org/html/2608.14550#S6.SS2.p2.2)\.
- P\. Henderson, J\. Hu, J\. Romoff, E\. Brunskill, D\. Jurafsky, and J\. Pineau \(2020\)Towards the systematic reporting of the energy and carbon footprints of machine learning\.J\. Mach\. Learn\. Res\.21\(1\)\.External Links:ISSN 1532\-4435Cited by:[§3](https://arxiv.org/html/2608.14550#S3.SS0.SSS0.Px2.p1.13),[§7](https://arxiv.org/html/2608.14550#S7.p3.1)\.
- K\. Huang, H\. Yin, H\. Huang, and W\. Gao \(2024\)Towards green AI in fine\-tuning large language models via adaptive backpropagation\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Vja3ecieXY)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- International Energy Agency \(2025\)Energy and ai\.Special ReportInternational Energy Agency \(IEA\),Paris, France\.Note:Published April 2025\. CC BY 4\.0 licenceExternal Links:[Link](https://www.iea.org/reports/energy-and-ai)Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p1.1)\.
- Y\. Jernite and S\. Luccioni \(2026\)AI’s never just one thing: different flops for different folks\.External Links:[Link](https://huggingface.co/spaces/)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang,et al\.\(2020\)TinyBERT: distilling BERT for natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Online,pp\. 4163–4174\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.372/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.372)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- A\. Lacoste, A\. Luccioni, V\. Schmidt, and T\. Dandres \(2019\)Quantifying the carbon emissions of machine learning\.CoRRabs/1910\.09700\.External Links:[Link](http://arxiv.org/abs/1910.09700),1910\.09700Cited by:[§7](https://arxiv.org/html/2608.14550#S7.p1.1)\.
- X\. Li, Y\. Shao, T\. Sun, H\. Yan,et al\.\(2021\)Accelerating BERT inference for sequence labeling via early\-exit\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),Online,pp\. 189–199\.External Links:[Link](https://aclanthology.org/2021.acl-long.16/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.16)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- W\. Liu, P\. Zhou, Z\. Wang, Z\. Zhao,et al\.\(2020\)FastBERT: a self\-distilling BERT with adaptive inference time\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 6035–6044\.External Links:[Link](https://aclanthology.org/2020.acl-main.537/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.537)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- X\. Liu, T\. Sun, J\. He, J\. Wu,et al\.\(2022\)Towards efficient NLP: A standard evaluation and A strong baseline\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10\-15, 2022,pp\. 3288–3303\.External Links:[Link](https://doi.org/10.18653/v1/2022.naacl-main.240),[Document](https://dx.doi.org/10.18653/V1/2022.NAACL-MAIN.240)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- A\. Lozhkov, R\. Li, L\. B\. Allal, F\. Cassano,et al\.\(2024\)StarCoder 2 and the stack v2: the next generation\.External Links:2402\.19173Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p1.1)\.
- W\. Luo, R\. Fan, Z\. Li, D\. Du, Q\. Wang, and X\. Chu \(2024\)Benchmarking and dissecting the nvidia hopper gpu architecture\.In2024 IEEE International Parallel and Distributed Processing Symposium \(IPDPS\),Vol\.,pp\. 656–667\.External Links:[Document](https://dx.doi.org/10.1109/IPDPS57955.2024.00064)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p4.1)\.
- X\. Meng, W\. Chen, R\. Benbaki, and R\. Mazumder \(2024\)FALCON: FLOP\-aware combinatorial optimization for neural network pruning\.InProceedings of The 27th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.238,pp\. 4384–4392\.External Links:[Link](https://proceedings.mlr.press/v238/meng24a.html)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- O\. Pearce, G\. Becker, S\. Brink, N\. Hanford, D\. Yokelson, A\. Knox, and B\. Rountree \(2025\)HPC benchmarking: repeat, replicate, reproduce\.InProceedings of the 3rd ACM Conference on Reproducibility and Replicability,ACM REP ’25,New York, NY, USA,pp\. 85–95\.External Links:ISBN 9798400719585,[Link](https://doi.org/10.1145/3736731.3746150),[Document](https://dx.doi.org/10.1145/3736731.3746150)Cited by:[§7](https://arxiv.org/html/2608.14550#S7.p4.1)\.
- E\. B\. Roque and L\. Cruz \(2025\)Energy aware development of neuromorphic implantables: from metrics to action\.CoRRabs/2506\.09599\.External Links:[Link](https://doi.org/10.48550/arXiv.2506.09599),[Document](https://dx.doi.org/10.48550/ARXIV.2506.09599),2506\.09599Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p2.1)\.
- A\. Sain, S\. Maity, P\. N\. Chowdhury, S\. Koley,et al\.\(2025\)Sketch down the flops: towards efficient networks for human sketch\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11\-15, 2025,pp\. 28383–28393\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2025/html/Sain%5C_Sketch%5C_Down%5C_the%5C_FLOPs%5C_Towards%5C_Efficient%5C_Networks%5C_for%5C_Human%5C_Sketch%5C_CVPR%5C_2025%5C_paper.html),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.02643)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- R\. Schwartz, J\. Dodge, N\. A\. Smith, and O\. Etzioni \(2020\)Green ai\.Commun\. ACM63\(12\),pp\. 54–63\.External Links:ISSN 0001\-0782,[Link](https://doi.org/10.1145/3381831),[Document](https://dx.doi.org/10.1145/3381831)Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p1.1),[§7](https://arxiv.org/html/2608.14550#S7.p1.1)\.
- J\. Shi, Z\. Yang, H\. J\. Kang, B\. Xu,et al\.\(2024\)Greening large language models of code\.InProceedings of the 46th International Conference on Software Engineering: Software Engineering in Society,ICSE\-SEIS’24,New York, NY, USA,pp\. 142–153\.External Links:ISBN 9798400704994,[Link](https://doi.org/10.1145/3639475.3640097),[Document](https://dx.doi.org/10.1145/3639475.3640097)Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p1.1)\.
- M\. Tan and Q\. Le \(2019\)Efficientnet: rethinking model scaling for convolutional neural networks\.InInternational conference on machine learning,pp\. 6105–6114\.Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p1.1)\.
- Z\. Tasoulas and I\. Anagnostopoulos \(2019\)Improving gpu performance with a power\-aware streaming multiprocessor allocation methodology\.Electronics8\(12\)\.External Links:[Link](https://www.mdpi.com/2079-9292/8/12/1451),ISSN 2079\-9292,[Document](https://dx.doi.org/10.3390/electronics8121451)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p4.1)\.
- B\. van Werkhoven, J\. Maassen, H\. E\. Bal, and F\. J\. Seinstra \(2014\)Optimizing convolution operations on gpus using adaptive tiling\.Future Generation Computer Systems30,pp\. 14–26\.Note:Special Issue on Extreme Scale Parallel Architectures and Systems, Cryptography in Cloud Computing and Recent Advances in Parallel and Distributed Systems, ICPADS 2012 Selected PapersExternal Links:ISSN 0167\-739X,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.future.2013.09.003),[Link](https://www.sciencedirect.com/science/article/pii/S0167739X13001829)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p4.1)\.
- C\. Wu, R\. Raghavendra, U\. Gupta, B\. Acun,et al\.\(2022\)Sustainable AI: environmental implications, challenges and opportunities\.InProceedings of the Fifth Conference on Machine Learning and Systems, MLSys 2022, Santa Clara, CA, USA, August 29 \- September 1, 2022,External Links:[Link](https://proceedings.mlsys.org/paper%5C_files/paper/2022/hash/462211f67c7d858f663355eff93b745e-Abstract.html)Cited by:[§1](https://arxiv.org/html/2608.14550#S1.p1.1)\.
- T\. Yang, Y\. Chen, and V\. Sze \(2017\)Designing energy\-efficient convolutional neural networks using energy\-aware pruning\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 6071–6079\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2017.643),ISSN 1063\-6919Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- S\. Yeom, K\. Shim, and J\. Hwang \(2021\)Toward compact deep neural networks via energy\-aware pruning\.CoRRabs/2103\.10858\.External Links:[Link](https://arxiv.org/abs/2103.10858),2103\.10858Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.
- C\. Zhang, B\. Wang, and D\. Song \(2024\)On elastic language models\.ACM Trans\. Inf\. Syst\.42\(6\)\.External Links:ISSN 1046\-8188,[Link](https://doi.org/10.1145/3677375),[Document](https://dx.doi.org/10.1145/3677375)Cited by:[§6\.1](https://arxiv.org/html/2608.14550#S6.SS1.p2.1)\.