用于电力负荷预测的极低状态量化 Transformer

# Extreme Low-State Quantized Transformer for Electricity-Load Forecasting
Source: [https://arxiv.org/html/2610.28784](https://arxiv.org/html/2610.28784)
Muhammad Sabbir AlamAffiliation:Department of Mechanical and Nuclear Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USAHarindra Sandun MavikumbureAffiliation:Department of Computer Science, Virginia Commonwealth University, Richmond, VA, 23284, USAVictor CobileanAffiliation:Department of Computer Science, Virginia Commonwealth University, Richmond, VA, 23284, USAMilos ManicAffiliation:Department of Computer Science, Virginia Commonwealth University, Richmond, VA, 23284, USAJayasimha AtulasimhaAffiliation:Department of Mechanical and Nuclear Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USAAffiliation:Department of Electrical and Computer Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USA\*Corresponding author: jatulasimha@vcu\.edu

###### Abstract

Aggressive low\-state weight quantization can substantially reduce the representational flexibility of neural networks, making it difficult to preserve predictive performance when only a few discrete weight states are available\. Here, we investigate this regime for Transformer\-based electricity\-load forecasting using 3\-, 5\-, 7\-, and 16\-state weight quantization across feed\-forward\-only and attention\-plus\-feed\-forward quantization\. Two forecasting tasks are considered, predicting either the next hour or the next 24 hours from the preceding 48 hours, across two co\-primary electricity\-consumption datasets\. Direct post\-training quantization \(PTQ\) increased mean normalized root\-mean\-square error \(NRMSE\) relative to full precision by 6\.25 and 7\.04 percentage points in Datasets A and B, respectively, whereas quantization\-aware fine\-tuning \(QAT\-FT\) recovered approximately 90\.2% and 94\.5% of these penalties\. Quantization\-aware training directly from random initialization \(Scratch\-QAT\) also produced competitive low\-state solutions without requiring a pretrained full\-precision starting point, with particularly favorable behavior in the three\-state regime\. We further introduce accumulated\-update thresholding, which accumulates optimizer\-proposed target\-weight changes before committing them to the continuous shadow weights\. Relative to exactly paired Scratch\-QAT controls, this reduced committed shadow\-weight updates by approximately 99\.7% and discrete quantized state transitions by approximately 77–80%, while mean NRMSE changed by only \+0\.042 and \+0\.197 percentage points in Datasets A and B\. These results demonstrate that useful Transformer forecasting can be retained under highly discrete weight representations while strongly suppressing committed and discrete target\-state activity, highlighting a promising route toward edge\-based load forecasting in which multilevel synaptic devices could natively encode low\-state weights and support compact, low\-activity, resource\-efficient learning and inference\.

## 1Introduction

Deploying neural networks under severe numerical constraints requires more than simply reducing parameter precision after training\. When weights are restricted to only a few discrete states, the representational flexibility available to the model becomes strongly limited, making it increasingly difficult to preserve full\-precision behavior\. This aggressive low\-state regime is particularly relevant when model parameters must ultimately be represented by a small number of accessible levels, but it also poses a fundamental learning question: how much predictive capability can be retained when the weight space itself is highly discrete? Addressing this question requires training strategies designed for low\-state operation rather than treating quantization only as a final compression step\([Hubara et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib13);[Esser et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib9);[Nagel et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib26);[Lin et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib21);[Shao et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib31)\)\. Recent work continues to show that preserving accuracy becomes particularly demanding in highly constrained low\-bit Transformer representations\([Lin et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib21);[Shao et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib31)\)\.

Post\-training quantization \(PTQ\) is attractive because it can reduce weight precision without modifying the original training procedure or requiring end\-to\-end retraining\. At moderate precision, this simplicity can make PTQ an efficient route from a pretrained full\-precision model to a lower\-precision representation, and improved rounding, reconstruction, and calibration methods have extended its reach toward lower bit widths\([Nagel et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib25);[Li et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib19);[Frantar et al\., 2023](https://arxiv.org/html/2610.28784#bib.bib10);[Xiao et al\., 2023](https://arxiv.org/html/2610.28784#bib.bib36);[Lin et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib21);[Shao et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib31)\)\. However, the difficulty becomes much greater when the available weight space collapses to only a few discrete states, where coarse projection can substantially disturb the learned solution\. Quantization\-aware training \(QAT\) addresses this limitation by incorporating the low\-state representation into training itself, allowing the model to adapt while quantization is active\([Krishnamoorthi, 2018](https://arxiv.org/html/2610.28784#bib.bib17);[Jacob et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib14);[Esser et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib9);[Nagel et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib26)\)\.

When such quantization is incorporated during training, a common pathway is quantization\-aware fine\-tuning \(QAT\-FT\), which begins from a pretrained full\-precision solution and adapts it to the discrete weight space\([Nagel et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib26)\)\. This raises a different question from whether QAT can recover PTQ\-induced degradation: must a useful extremely low\-state model originate from a full\-precision precursor at all? Prior studies show that neural networks can be optimized directly with very low\-bit weights and activations\([Hubara et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib13);[Esser et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib9)\)\. If competitive solutions can emerge directly from random initialization while quantization is active from the outset, low\-state learning need not be viewed only as adaptation of an already trained continuous model\. Direct random\-initialized QAT \(Scratch\-QAT\) therefore provides a complementary test of whether the constrained weight space itself can support effective learning without any pretraining\.

Low\-state learning also raises an issue not captured by forecasting accuracy alone: how often training actually requires a committed parameter change or a transition between discrete weight states\. This distinction is relevant to in\-memory and synaptic\-device computing, where discrete weights can be represented by programmable conductance states and physical reprogramming carries implementation costs\([Le Gallo et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib18);[Sebastian et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib29);[Azam et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib5);[Misba et al\., 2022](https://arxiv.org/html/2610.28784#bib.bib23);[Alam et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib3);[Dhull et al\., 2025](https://arxiv.org/html/2610.28784#bib.bib8);[Sehgal et al\., 2025](https://arxiv.org/html/2610.28784#bib.bib30);[Aguirre et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib1);[Jiang et al\., 2026](https://arxiv.org/html/2610.28784#bib.bib15)\)\. Mixed\-precision computational\-memory work has further shown that small numerical updates can be accumulated at higher precision and transferred to device states only when sufficiently large relative to programming granularity\([Nandakumar et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib27)\)\. We therefore separate optimizer\-proposed updates from committed weight updates and actual discrete state transitions, and ask whether small adjustments can be accumulated before commitment\. Such a strategy could reduce state\-changing activity while preserving gradual optimization, with prospective relevance to multilevel synaptic devices and resource\-constrained edge hardware\([Sebastian et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib29);[Alam et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib3);[Dhull et al\., 2025](https://arxiv.org/html/2610.28784#bib.bib8);[Bettayeb et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib6)\)\.

These questions are particularly relevant to electricity\-load forecasting, where models must capture recurring temporal structure and short\-term variability over near\-term and multi\-step horizons\([Hong and Fan, 2016](https://arxiv.org/html/2610.28784#bib.bib12);[Amarasinghe et al\., 2017](https://arxiv.org/html/2610.28784#bib.bib4);[Al Misba et al\., 2023](https://arxiv.org/html/2610.28784#bib.bib2)\)\. Transformer and attention\-based architectures have become prominent in time\-series forecasting, with recent developments targeting interpretable multi\-horizon prediction, long\-sequence efficiency, decomposition, patch\-based representations, and variable\-centric modeling\([Lim et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib20);[Zhou et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib37);[Wu et al\., 2021](https://arxiv.org/html/2610.28784#bib.bib35);[Zhou et al\., 2022](https://arxiv.org/html/2610.28784#bib.bib38);[Nie et al\., 2023](https://arxiv.org/html/2610.28784#bib.bib28);[Liu et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib22);[Su et al\., 2025](https://arxiv.org/html/2610.28784#bib.bib32)\)\. Transformer\-based methods have also been applied directly to electricity\-load forecasting\([Chan and Yeo, 2024](https://arxiv.org/html/2610.28784#bib.bib7)\)\. We therefore investigate aggressive low\-state quantization in this setting\. The Transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2610.28784#bib.bib34)\)is well suited to this study because its self\-attention mechanism can model temporal dependencies across the input sequence, while its distinct attention and feed\-forward weight blocks provide two major parameter groups whose sensitivity to severe quantization can be examined separately\. By evaluating both one\-hour and 24\-hour forecasting across two co\-primary electricity\-consumption datasets with contrasting load characteristics, this study examines whether extremely low\-state Transformer learning remains effective across different forecasting horizons, demand patterns, and quantization scopes\.

This study makes five principal contributions\. First, it establishes the behavior of Transformer forecasting under aggressive 3\-, 5\-, 7\-, and 16\-state weight quantization across feed\-forward\-only and attention\-plus\-feed\-forward quantization, showing that direct PTQ becomes particularly damaging at the lowest state counts, whereas performance progressively approaches full\-precision behavior as state resolution increases\. Second, it shows that quantization\-aware fine\-tuning recovers most of the severe PTQ penalty, recovering approximately 90\.2% in Dataset A and 94\.5% in Dataset B\. Third, it demonstrates that training directly from random initialization under quantized forward conditions—Scratch\-QAT—can also produce competitive low\-state solutions without requiring a pretrained full\-precision starting point, with particularly favorable behavior in the most restrictive three\-state regime\. Fourth, it introduces accumulated\-update thresholding, in which small optimizer\-proposed weight changes are accumulated before being committed to the target weights; only committed changes can subsequently alter the corresponding quantized weight from one discrete state to another\. This approach reduces committed weight updates by approximately 99\.7% and discrete state transitions by approximately 77–80% while producing only small mean changes in forecasting error, demonstrating that optimizer activity can be strongly decoupled from the weight changes that are actually committed and from those that change the discrete state representation\. Finally, the principal PTQ, QAT\-FT, and thresholding effects reproduce across the two co\-primary datasets and all four distinct load\-series analysis units, including transfer of the methodology to Dataset B without Dataset\-B\-specific retuning, supporting the robustness of the observed low\-state learning behavior across heterogeneous electricity\-demand settings\.

The remainder of this paper is organized as follows\. Section 2 describes the datasets, forecasting protocol, quantized Transformer framework, training strategies, and evaluation methods\. Section 3 presents the main forecasting, quantization, thresholding, and cross\-dataset results\. Section 4 discusses the implications, limitations, and prospective hardware relevance of the findings\. Section 5 concludes the paper\.

## 2Methods

### 2\.1Co\-primary datasets and forecasting tasks

Two publicly available electricity\-consumption datasets from different geographical and consumption settings were treated as co\-primary datasets and evaluated using the same forecasting formulation \(Table[1](https://arxiv.org/html/2610.28784#S2.T1)\)\.

Table 1:Co\-primary datasets and forecasting protocol\.Dataset A was the UCIElectricityLoadDiagrams20112014dataset\([Trindade, 2015](https://arxiv.org/html/2610.28784#bib.bib33)\), containing measurements from 370 clients in Portugal\. Three anonymous client series, MT\_004, MT\_022, and MT\_324, were analyzed from 1 January 2012 through 31 December 2014\. The original 15\-min measurements were averaged to hourly resolution\. The three series were selected to provide contrasting temporal and statistical load characteristics; detailed descriptive characterization is reported in the Supplementary Information\. No interpolation was applied, and forecasting samples intersecting the three identified daylight\-saving\-time artifact hours were excluded\.

Dataset B was the UCIIndividual Household Electric Power Consumptiondataset\([Hebrail and Berard, 2006](https://arxiv.org/html/2610.28784#bib.bib11)\)\. Global\_active\_power from a single household in Sceaux, France, was analyzed from 1 January 2007 through 31 December 2009\. Minute\-level measurements were averaged to hourly resolution, with an hour considered valid only when all 60 minute\-level observations were available\. Invalid hours were retained as missing values, no interpolation or imputation was performed, and forecasting samples were retained only when all required input and target observations were finite\.

Both datasets used the preceding 48 hourly observations as input\. For the one\-hour\-ahead task \(H1\),

\[xt−47,…,xt\]→xt\+1\.\[x\_\{t\-47\},\\ldots,x\_\{t\}\]\\rightarrow x\_\{t\+1\}\.\(1\)
whereas for the day\-ahead task \(H24\),

\[xt−47,…,xt\]→\[xt\+1,…,xt\+24\]\.\[x\_\{t\-47\},\\ldots,x\_\{t\}\]\\rightarrow\[x\_\{t\+1\},\\ldots,x\_\{t\+24\}\]\.\(2\)
H24 was therefore implemented as a direct multi\-output forecast, rather than by recursively feeding earlier predictions back into the model\. Chronological development, normalization, and final\-test procedures are described in Section 2\.2\.

### 2\.2Development protocol and final\-test locking

A strictly chronological development protocol separated model development from final performance evaluation\. Each co\-primary dataset used three expanding\-window validation folds, with every validation interval occurring later than its corresponding training interval \(Table[2](https://arxiv.org/html/2610.28784#S2.T2)\)\. Final test periods were reserved until completion of development and methodological locking\.

Table 2:Chronological development and final evaluation protocolNormalization was performed independently within each development fold using z\-score scaling fitted only to the corresponding training interval\. Validation observations were excluded from scaler estimation\. The same training\-only normalization principle was used for final model fitting; invalid or excluded observations were handled according to the dataset\-specific validity rules described in Section 2\.1\.

After Dataset\-A development, the principal experimental choices were locked before final\-test evaluation\. These included the forecasting tasks, horizon\-specific Transformer configurations, 3\-, 5\-, 7\-, and 16\-state representations, two quantization locations, QAT\-FT and Scratch\-QAT procedures, Scratch training endpoints, and the selected high\-α\\alphaschedule\. All selections were based on development\-stage evidence rather than final\-test performance\.

The methodology established using Dataset A was subsequently transferred to Dataset B without Dataset\-B\-specific retuning\. State counts, quantization locations, the high\-α\\alpharule, Transformer configurations, and QAT/Scratch\-QAT procedures were not reselected using Dataset\-B performance\. Following final\-test access in either dataset, no performance\-driven retraining or modification of these methodological choices was performed\.

### 2\.3Controlled Transformer architecture

A conventional encoder\-only Transformer was used as the common forecasting backbone for both co\-primary datasets\. The architecture was intentionally kept relatively compact so that the experiments could isolate the effects of low\-state weight quantization and training strategy, rather than introduce a specialized forecasting architecture as an additional source of variation\. The same frozen horizon\-specific Transformer configurations were used for Dataset A and Dataset B, consistent with the methodology\-transfer protocol described in Section 2\.2\.

Each input sample had shape48×148\\times 1, corresponding to the preceding 48 hourly load observations\. The scalar value at each time step was first projected through a dense layer to a 64\-dimensional representation \(dmodel=64d\_\{\\mathrm\{model\}\}=64\), after which a trainable positional embedding of length 48 and dimension 64 was added to preserve temporal order\. The resulting sequence was processed by pre\-normalization Transformer encoder blocks\. Each block contained four\-head self\-attention, with an attention key dimension of64/4=1664/4=16per head, followed by a position\-wise feed\-forward network of dimension64→128→6464\\rightarrow 128\\rightarrow 64with GELU activation\. Layer normalization preceded both the self\-attention and feed\-forward sublayers, and residual connections were applied after each sublayer\. Dropout was applied within the attention and feed\-forward paths\.

After the final encoder block, a final LayerNorm was applied and only the representation corresponding to the most recent input time step was retained for forecasting\. This 64\-dimensional last\-token representation was passed to a dense output layer containing either one unit for H1 or 24 units for H24, producing the direct forecasts defined in Section 2\.1\. The H1 configuration used three encoder blocks, dropout of 0\.05, and 103,809 trainable parameters, whereas the H24 configuration used one encoder block, dropout of 0\.10, and 38,360 trainable parameters\. Both configurations retaineddmodel=64d\_\{\\mathrm\{model\}\}=64, four attention heads, and an FFN dimension of 128\.

### 2\.4Low\-state weight quantization

Weight quantization used a uniform symmetric per\-weight\-tensor quantizer withLLdiscrete states\. For each targeted weight tensorWW,

wheressdenotes the largest absolute weight value within that tensor\. The spacing between adjacent states was

Δ=2​sL−1,\\Delta=\\frac\{2s\}\{L\-1\},\(4\)
giving

𝒬L=\{−s,−s\+Δ,…,−s\+\(L−1\)​Δ\}\.\\mathcal\{Q\}\_\{L\}=\\\{\-s,\\,\-s\+\\Delta,\\ldots,\-s\+\(L\-1\)\\Delta\\\}\.\(5\)
Each targeted weight was mapped deterministically to its nearest available state\. Because quantization was performed per weight tensor, each targeted kernel had its own symmetric range\[−s,s\]\[\-s,s\]\. The reference tensor used to establish this fixed range depended on the training pathway and is described in Section 2\.5\.

Development\-stage experiments evaluated 2, 3, 4, 5, 7, 8, 16, 32, 64, 128, and 256 states\. Forecasting performance progressively approached the full\-precision reference as state resolution increased, while differences among the higher\-state conditions became comparatively small\. The final 3\-, 5\-, 7\-, and 16\-state set was therefore selected to focus the main experiments on the strongly quantized regime while spanning progressively finer state resolution, with 16 states providing a higher\-resolution condition near the region where further increases produced diminishing differences\. The odd 3\-, 5\-, and 7\-state symmetric grids additionally contain an exact zero state\. Such low\-state representations are also relevant to prospective multilevel synaptic\-device implementations, where model weights could be encoded using a limited number of accessible physical states\.

Two quantization locations were evaluated\. In FFN\-Q, only the two feed\-forward weight kernels within each encoder block were quantized\. In Attention\+FFN\-Q, quantization additionally included the query, key, value, and attention\-output kernels\. Activations, biases, LayerNorm operations, softmax, and all non\-target parameters remained in FP32\.

For H1, FFN\-Q quantized 49,152 of 103,809 trainable parameters \(47\.35%\), whereas Attention\+FFN\-Q quantized 98,304 \(94\.70%\)\. For H24, the corresponding values were 16,384 of 38,360 parameters \(42\.71%\) and 32,768 \(85\.42%\), respectively\. Thus, Attention\+FFN\-Q imposed the low\-state constraint on the large majority of trainable parameters, making it substantially more restrictive than FFN\-Q\. The same state counts and quantization locations were used for both co\-primary datasets\.

### 2\.5Training strategies

Four training and quantization pathways were compared: full precision \(FP\), post\-training quantization \(PTQ\), quantization\-aware fine\-tuning \(QAT\-FT\), and random\-initialized quantization\-aware training \(Scratch\-QAT\)\. All neural\-network optimization used Adam with mean\-squared error \(MSE\) loss\. PTQ and QAT\-FT were independent branches originating from the same pretrained FP checkpoint, whereas Scratch\-QAT was initialized randomly and trained with quantization active from the beginning\.

For the quantization\-aware training pathways,α\\alphadenotes the dimensionless accumulated\-update threshold coefficient controlling when an optimizer\-proposed change to a targeted continuous weight is committed\. For a targeted tensor with fixed symmetric range\[−s,s\]\[\-s,s\], the threshold magnitude isα​s\\alpha s\. Thus,α=0\\alpha=0represents the immediate\-commit condition, whereasα\>0\\alpha\>0allows smaller proposed changes to accumulate before commitment\. The complete accumulated\-update procedure and selected high\-α\\alpharule are described in Section 2\.6\.

#### 2\.5\.1Full precision

The FP model provided the unquantized reference for each dataset and forecasting horizon\. All weights were trained in full precision using the horizon\-specific architecture described in Section 2\.3\. The resulting checkpoint served as both the performance reference and the common starting point for the PTQ and QAT\-FT branches\.

#### 2\.5\.2Post\-training quantization

For PTQ, the trained FP checkpoint was quantized directly using the state count and quantization location defined in Section 2\.4\. For each targeted tensor, the fixed symmetric range\[−s,s\]\[\-s,s\]was obtained from the corresponding trained FP weights, which were then mapped to their nearest available quantization states\. No retraining or weight adaptation was performed after quantization\.

#### 2\.5\.3Quantization\-aware fine\-tuning

QAT\-FT began independently from the same pretrained FP checkpoint used for PTQ; it was not initialized from the PTQ model\. Per\-tensor quantization ranges were calculated from the original FP checkpoint and held fixed during fine\-tuning\.

Following the straight\-through estimator \(STE\) principle used in low\-precision training\([Hubara et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib13)\), continuous shadow weights were maintained for each targeted kernel while their quantized counterparts were used for model computation\. This distinction is particularly important at very low state counts, where individual optimizer updates can be substantially smaller than the spacing between neighboring quantization states\. The continuous shadow representation preserves such fine\-grained updates even when the corresponding quantized weight does not immediately change state\.

During each minibatch, the targeted kernels were quantized for the forward pass and gradient calculation, after which the continuous shadow weights were restored before the Adam update\. Thus, the network was evaluated under the prescribed discrete\-state representation while optimization acted on continuous underlying parameters\. QAT\-FT was performed for 25 epochs with a learning rate of1×10−41\\times 10^\{\-4\}, batch size 128, seed 42, andα=0\\alpha=0\. No early stopping or learning\-rate scheduler was used\.

#### 2\.5\.4Random\-initialized quantization\-aware training \(Scratch\-QAT\)

Scratch\-QAT tested whether useful low\-state solutions could be learned without a pretrained FP starting point\. The Transformer was freshly initialized using Glorot\-uniform initialization with seed 42, and the resulting initialization was saved asW0W\_\{0\}\. For each targeted tensor, the fixed quantization range\[−s,s\]\[\-s,s\]was derived from this sameW0W\_\{0\}, rather than from a pretrained FP checkpoint\. Quantization was active from the first minibatch, using the same continuous\-shadow\-weight and STE\-style framework as QAT\-FT\.

Scratch\-QAT used Adam with a learning rate of2×10−32\\times 10^\{\-3\}, batch size 128, and seed 42\. The primary endpoint was 40 epochs \(E40\), while a 25\-epoch checkpoint \(E25\) was retained for a training\-budget\-matched comparison with QAT\-FT\.

Two exactly paired Scratch\-QAT conditions were evaluated: Scratchα=0\\alpha=0and Scratch high\-α\\alpha\. Each pair began from the identical savedW0W\_\{0\}and used the same optimizer initialization, minibatch ordering, learning rate, batch size, quantization range, state count, quantization location, and epoch budget; only the accumulated\-update threshold differed\. The thresholding procedure and high\-α\\alpharule are described in Section 2\.6\.

### 2\.6Accumulated\-update thresholding

Accumulated\-update thresholding was introduced within Scratch\-QAT to control how frequently optimizer\-proposed changes were committed to the targeted continuous shadow weights\. The rule operates after Adam has generated a proposed continuous target\-weight update but before that change is committed\. It therefore acts at the weight\-commit stage without removing or skipping conventional forward passes, backward passes, minibatches, or Adam optimizer steps\.

For each targeted weight elementii, a non\-trainable accumulatorAiA\_\{i\}was initialized to zero\. Letwi\(t\)w\_\{i\}^\{\(t\)\}denote the currently committed continuous shadow weight at minibatchtt, and letw~i\(t\+1\)\\widetilde\{w\}\_\{i\}^\{\(t\+1\)\}denote the optimizer\-proposed continuous value after clipping to the fixed tensor range\[−s,s\]\[\-s,s\]\. The proposed target\-weight change was

δ​wi\(t\)=w~i\(t\+1\)−wi\(t\),\\delta w\_\{i\}^\{\(t\)\}=\\widetilde\{w\}\_\{i\}^\{\(t\+1\)\}\-w\_\{i\}^\{\(t\)\},\(6\)
and was added to the existing accumulator,

A¯i\(t\)=Ai\(t\)\+δ​wi\(t\)\.\\bar\{A\}\_\{i\}^\{\(t\)\}=A\_\{i\}^\{\(t\)\}\+\\delta w\_\{i\}^\{\(t\)\}\.\(7\)
Thus, sub\-threshold optimizer proposals were retained rather than discarded and could combine with subsequent proposed changes\. The tensor\-specific commit threshold was

and a target element became eligible for commitment when

\|A¯i\(t\)\|≥α​s\.\\left\|\\bar\{A\}\_\{i\}^\{\(t\)\}\\right\|\\geq\\alpha s\.\(9\)
The threshold is therefore defined relative to the tensor rangess, not the quantization\-state spacingΔ\\Delta\. Forα=0\\alpha=0, every nonzero proposed target\-weight change was committed immediately; forα\>0\\alpha\>0, sub\-threshold changes remained stored until their accumulated magnitude reached the threshold\.

When the criterion was satisfied, the accumulated change was applied to the continuous shadow weight and the corresponding accumulator was reset\. Otherwise, the committed shadow weight remained unchanged and the accumulator was retained\. The next quantized value was always obtained from the currently committed shadow weight rather than from the uncommitted accumulator\. Consequently, a committed continuous shadow\-weight update and a discrete quantized state transition are distinct events: a committed update does not necessarily cross a quantization\-state boundary\.

The selected high\-α\\alphaschedule was related to the geometry of the uniform quantization grid\. Since

Δ=2​sL−1,\\Delta=\\frac\{2s\}\{L\-1\},
from Eq\. \([4](https://arxiv.org/html/2610.28784#S2.E4)\), choosingα=1/\(L−1\)\\alpha=1/\(L\-1\)givesα​s=Δ/2\\alpha s=\\Delta/2\. The final high\-α\\alpharule was therefore

αhigh​\(L\)=min⁡\(0\.25,1L−1\),\\alpha\_\{\\mathrm\{high\}\}\(L\)=\\min\\left\(0\.25,\\frac\{1\}\{L\-1\}\\right\),\(10\)
yieldingα=0\.25\\alpha=0\.25,0\.250\.25,1/61/6, and1/151/15for 3, 5, 7, and 16 states, respectively\. The cap at 0\.25 prevented the threshold from exceeding the largest initially considered value for the coarsest representations\. A moderate\-α\\alphaschedule was also evaluated during Dataset\-A development, but the high\-α\\alphaschedule provided stronger activity suppression while retaining comparable forecasting performance and was therefore selected as the primary thresholded condition before final\-test evaluation\. The same rule was transferred unchanged to Dataset B\.

The nonzero threshold exploration was motivated by prior work on low\-resolution synaptic and memory implementations and tolerance to weight perturbations\([Misba et al\., 2022](https://arxiv.org/html/2610.28784#bib.bib23);[Joshi et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib16)\)\. Here, however,α\\alphaspecifically denotes the accumulated\-update commit\-threshold coefficient rather than a directly measured physical\-device quantity\.

The complete operational sequence is summarized in Algorithm[1](https://arxiv.org/html/2610.28784#alg1)\.

Algorithm 1Accumulated\-update thresholding for Scratch\-QAT1:Training minibatches; targeted continuous shadow weights

ww; fixed tensor ranges

\[−s,s\]\[\-s,s\]; state count

LL; threshold coefficient

α\\alpha
2:Notation:

ki\(t\)k\_\{i\}^\{\(t\)\}denotes the discrete quantization\-state index associated with targeted weight element

iiat minibatch

tt
3:Initialize element\-wise accumulators

Ai←0A\_\{i\}\\leftarrow 0
4:Initialize discrete states

ki\(0\)←index⁡\[QL​\(wi\(0\)\)\]k\_\{i\}^\{\(0\)\}\\leftarrow\\operatorname\{index\}\\left\[Q\_\{L\}\\left\(w\_\{i\}^\{\(0\)\}\\right\)\\right\]
5:foreach minibatch

ttdo

6:Save the current continuous target shadow weights

7:Quantize targeted weights using

QL​\(⋅\)Q\_\{L\}\(\\cdot\)
8:Perform the forward pass using the quantized model and compute the MSE loss

9:Perform backpropagation under the quantized model

10:Restore the continuous shadow weights

11:Apply Adam to obtain proposed continuous target weights

12:foreach targeted weight element

iido

13:Clip the optimizer\-proposed value to the fixed range

\[−s,s\]\[\-s,s\]
14:

δ​wi\(t\)←w~i\(t\+1\)−wi\(t\)\\displaystyle\\delta w\_\{i\}^\{\(t\)\}\\leftarrow\\widetilde\{w\}\_\{i\}^\{\(t\+1\)\}\-w\_\{i\}^\{\(t\)\}
15:

A¯i\(t\)←Ai\(t\)\+δ​wi\(t\)\\displaystyle\\bar\{A\}\_\{i\}^\{\(t\)\}\\leftarrow A\_\{i\}^\{\(t\)\}\+\\delta w\_\{i\}^\{\(t\)\}
16:if

\(α=0∧A¯i\(t\)≠0\)∨\(α\>0∧\|A¯i\(t\)\|≥α​s\)\(\\alpha=0\\land\\bar\{A\}\_\{i\}^\{\(t\)\}\\neq 0\)\\lor\(\\alpha\>0\\land\|\\bar\{A\}\_\{i\}^\{\(t\)\}\|\\geq\\alpha s\)then

17:

wi\(t\+1\)←clip⁡\(wi\(t\)\+A¯i\(t\),−s,s\)\\displaystyle w\_\{i\}^\{\(t\+1\)\}\\leftarrow\\operatorname\{clip\}\\left\(w\_\{i\}^\{\(t\)\}\+\\bar\{A\}\_\{i\}^\{\(t\)\},\-s,s\\right\)
18:

Ai\(t\+1\)←0\\displaystyle A\_\{i\}^\{\(t\+1\)\}\\leftarrow 0
19:else

20:

wi\(t\+1\)←wi\(t\)\\displaystyle w\_\{i\}^\{\(t\+1\)\}\\leftarrow w\_\{i\}^\{\(t\)\}
21:

Ai\(t\+1\)←A¯i\(t\)\\displaystyle A\_\{i\}^\{\(t\+1\)\}\\leftarrow\\bar\{A\}\_\{i\}^\{\(t\)\}
22:endif

23:Recalculate the discrete quantized state:

ki\(t\+1\)←index⁡\[QL​\(wi\(t\+1\)\)\]k\_\{i\}^\{\(t\+1\)\}\\leftarrow\\operatorname\{index\}\\left\[Q\_\{L\}\\left\(w\_\{i\}^\{\(t\+1\)\}\\right\)\\right\]
24:Record proposed updates, committed updates, state transitions, and transition distance

25:endfor

26:endfor

Proposed updates, committed shadow\-weight updates, discrete state transitions, and transition distance were recorded separately; their formal definitions are given in Section 2\.7\.

### 2\.7Update and transition accounting

To quantify the effect of accumulated\-update thresholding independently of forecasting accuracy, training activity was recorded at four levels: proposed target\-weight updates, committed shadow\-weight updates, discrete quantized state transitions, and transition distance\. All quantities were counted for individual elements of the targeted weight tensors and accumulated across minibatches, targeted tensors, and epochs\. They therefore represent weight\-element activity rather than the number of optimizer steps\. Here,ttindexes minibatch update steps across all epochs andiiindexes targeted weight elements\.

A proposed target\-weight update was counted whenever Adam generated a nonzero change in a targeted continuous shadow weight after clipping the optimizer\-proposed value to its fixed tensor range\. Using the proposed change defined in Section 2\.6,

pi\(t\)=𝟙\[δwi\(t\)≠0\],p\_\{i\}^\{\(t\)\}=\\mathds\{1\}\\left\[\\delta w\_\{i\}^\{\(t\)\}\\neq 0\\right\],\(11\)
and

Nprop=∑t∑ipi\(t\)\.N\_\{\\mathrm\{prop\}\}=\\sum\_\{t\}\\sum\_\{i\}p\_\{i\}^\{\(t\)\}\.\(12\)
A committed shadow\-weight update was counted only when thresholding and range clipping produced an actual change in the committed continuous shadow weight:

ui\(t\)=𝟙\[wi\(t\+1\)≠wi\(t\)\],u\_\{i\}^\{\(t\)\}=\\mathds\{1\}\\left\[w\_\{i\}^\{\(t\+1\)\}\\neq w\_\{i\}^\{\(t\)\}\\right\],\(13\)
with

Ncommit=∑t∑iui\(t\)\.N\_\{\\mathrm\{commit\}\}=\\sum\_\{t\}\\sum\_\{i\}u\_\{i\}^\{\(t\)\}\.\(14\)
Thus, crossing the threshold alone was insufficient to count a commit if clipping left the shadow value unchanged\. For theα=0\\alpha=0control, every nonzero proposed update was committed, and the implementation verifiedNprop=NcommitN\_\{\\mathrm\{prop\}\}=N\_\{\\mathrm\{commit\}\}\.

A discrete quantized state transition was counted separately using the state indices defined in Section 2\.6:

qi\(t\)=𝟙\[ki\(t\+1\)≠ki\(t\)\],q\_\{i\}^\{\(t\)\}=\\mathds\{1\}\\left\[k\_\{i\}^\{\(t\+1\)\}\\neq k\_\{i\}^\{\(t\)\}\\right\],\(15\)
giving

Ntrans=∑t∑iqi\(t\)\.N\_\{\\mathrm\{trans\}\}=\\sum\_\{t\}\\sum\_\{i\}q\_\{i\}^\{\(t\)\}\.\(16\)
Because a committed continuous shadow\-weight change can remain within the same quantization interval,

Ntrans≤Ncommit\.N\_\{\\mathrm\{trans\}\}\\leq N\_\{\\mathrm\{commit\}\}\.\(17\)
Committed updates and discrete state transitions therefore represent distinct levels of target\-weight activity\.

Transition distance additionally captured the magnitude of movement through the discrete state space:

Dtrans=∑t∑i\|ki\(t\+1\)−ki\(t\)\|\.D\_\{\\mathrm\{trans\}\}=\\sum\_\{t\}\\sum\_\{i\}\\left\|k\_\{i\}^\{\(t\+1\)\}\-k\_\{i\}^\{\(t\)\}\\right\|\.\(18\)
An adjacent\-state transition contributes one unit, whereas a transition spanning multiple state intervals contributes correspondingly more\. Consequently,

Dtrans≥Ntrans\.D\_\{\\mathrm\{trans\}\}\\geq N\_\{\\mathrm\{trans\}\}\.\(19\)
The commit acceptance rate was defined as

Rcommit=100NcommitNprop\(%\),R\_\{\\mathrm\{commit\}\}=100\\frac\{N\_\{\\mathrm\{commit\}\}\}\{N\_\{\\mathrm\{prop\}\}\}\\quad\(\\%\),\(20\)
representing the percentage of nonzero optimizer\-proposed target\-weight changes that resulted in actual committed shadow\-weight changes\.

Because the number of targeted parameters differs across quantization locations and forecasting architectures, activity reductions were evaluated within exactly paired high\-α\\alphaandα=0\\alpha=0Scratch\-QAT runs rather than by comparing unrelated raw counts\. These paired runs used identical target tensors, initialization, minibatch sequence, and training budget, as described in Sections 2\.5–2\.6\.

These activity measures do not represent reductions in conventional optimization workload\. The pairedα=0\\alpha=0and high\-α\\alpharuns executed the same minibatches, forward/backward evaluations, and Adam optimizer steps\. Thresholding instead changes how frequently proposed target\-weight changes are committed and how frequently those commits produce discrete state transitions\. Such reductions may be relevant to prospective implementations in which physical state changes incur write costs; device write energy, endurance, lifetime, latency, and physical programming pulses were not measured here\.

### 2\.8Evaluation metrics, baselines, and evidence hierarchy

Forecasting performance was evaluated after inverse transformation to the original load scale using root\-mean\-square error \(RMSE\), mean absolute error \(MAE\), and normalized root\-mean\-square error \(NRMSE\)\. ForNNobservationsyny\_\{n\}and predictionsy^n\\hat\{y\}\_\{n\},

RMSE=1N​∑n=1N\(y^n−yn\)2,\\mathrm\{RMSE\}=\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\(\\hat\{y\}\_\{n\}\-y\_\{n\}\)^\{2\}\},\(21\)
MAE=1N​∑n=1N\|y^n−yn\|,\\mathrm\{MAE\}=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\|\\hat\{y\}\_\{n\}\-y\_\{n\}\|,\(22\)
and

NRMSE=100RMSEy¯\(%\),y¯=1N∑n=1Nyn\.\\mathrm\{NRMSE\}=100\\frac\{\\mathrm\{RMSE\}\}\{\\bar\{y\}\}\\quad\(\\%\),\\qquad\\bar\{y\}=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}y\_\{n\}\.\(23\)
NRMSE was the principal forecasting metric because it expresses error relative to the mean load level and facilitates comparison among series with different absolute magnitudes\. RMSE and MAE were retained as complementary measures\.

For H1, metrics were calculated over all valid next\-hour predictions\. For H24, the primary metrics were calculated jointly over all valid samples and all 24 simultaneously predicted forecast hours\. Metrics were also calculated separately for forecast hours 1–24 to examine lead\-time\-dependent behavior, with those trajectories reported in the results section\.

Three conventional baselines were used for context: persistence, daily seasonality, and weekly seasonality\. Persistence repeated the most recently observed load value across the required forecast horizon\. The daily and weekly seasonal baselines used

y^t=yt−24\\hat\{y\}\_\{t\}=y\_\{t\-24\}\(24\)
and

y^t=yt−168,\\hat\{y\}\_\{t\}=y\_\{t\-168\},\(25\)
respectively\. For H24, these references were constructed separately for each target timestamp\. The weekly baseline was treated as a strong seasonal benchmark rather than an information\-matched comparator because it accesses observations outside the Transformer’s fixed 48\-h input window\.

Differences in NRMSE are reported in percentage points \(pp\)\. For example, the PTQ penalty relative to FP was

ΔPTQ−FP=NRMSEPTQ−NRMSEFP\.\\Delta\_\{\\mathrm\{PTQ\-FP\}\}=\\mathrm\{NRMSE\}\_\{\\mathrm\{PTQ\}\}\-\\mathrm\{NRMSE\}\_\{\\mathrm\{FP\}\}\.\(26\)
The fraction of the PTQ\-associated penalty recovered by QAT\-FT was defined as

RQAT=100​NRMSEPTQ−NRMSEQAT​\-​FTNRMSEPTQ−NRMSEFP\.R\_\{\\mathrm\{QAT\}\}=100\\frac\{\\mathrm\{NRMSE\}\_\{\\mathrm\{PTQ\}\}\-\\mathrm\{NRMSE\}\_\{\\mathrm\{QAT\\text\{\-\}FT\}\}\}\{\\mathrm\{NRMSE\}\_\{\\mathrm\{PTQ\}\}\-\\mathrm\{NRMSE\}\_\{\\mathrm\{FP\}\}\}\.\(27\)
Thus,RQAT=100%R\_\{\\mathrm\{QAT\}\}=100\\%denotes recovery to the FP NRMSE level, whereas values below 100% indicate a residual gap relative to FP\.

The broad Dataset\-A development sweep comprised 11 state counts across three load series, two forecasting horizons, two quantization locations, and three expanding\-window folds, corresponding to 396 fold\-specific evaluations\. Based exclusively on development\-stage evidence, the final state set was restricted to 3, 5, 7, and 16 states\. The frozen final evaluation therefore comprised 48 Dataset\-A configurations and 16 Dataset\-B configurations, giving 64 structured configurations across the two co\-primary datasets\.

Primary conclusions were therefore based on the magnitude and direction of paired effects, agreement across both co\-primary datasets and the four series, configuration\-level support density, consistency across state\-count, horizon, and quantization\-location strata, and leave\-one\-series\-out sensitivity analysis\. Configuration\-level counts were interpreted descriptively rather than as independent statistical replicates\.

## 3Results

### 3\.1Aggressive low\-state PTQ produces a reproducible accuracy penalty that QAT\-FT largely avoids

Direct post\-training quantization \(PTQ\) substantially degraded forecasting performance across the tested 3\-, 5\-, 7\-, and 16\-state weight representations\. Across the 64\-configuration structured comparison set spanning these state counts, both quantization locations, both forecasting horizons, and the load series within each co\-primary dataset, mean NRMSE in Dataset A increased from 11\.3299% for the full\-precision \(FP\) reference to 17\.5770% under PTQ, corresponding to a 6\.2471\-percentage\-point penalty\. In Dataset B, mean NRMSE similarly increased from 57\.7067% for FP to 64\.7475% under PTQ, a 7\.0408\-percentage\-point penalty\. QAT\-FT was evaluated independently from the same pretrained FP starting point over the same 3\-, 5\-, 7\-, and 16\-state conditions and achieved substantially lower mean NRMSE values of 11\.9424% in Dataset A and 58\.0938% in Dataset B\. Using the dataset\-level mean NRMSE values, QAT\-FT closed 90\.20% of the FP–PTQ performance gap in Dataset A and 94\.50% in Dataset B \(Fig\.[1](https://arxiv.org/html/2610.28784#S3.F1); Table[3](https://arxiv.org/html/2610.28784#S3.T3)\)\.

Table 3:Headline forecasting and accumulated\-update results across the two co\-primary datasets\.Thus, across the full tested low\-state range, direct quantization of the pretrained FP solution incurred a substantial average accuracy penalty, whereas independently applied QAT\-FT produced models whose performance remained much closer to the FP reference\.

The magnitude of the PTQ penalty depended strongly on the number of available weight states\. At the most aggressive three\-state condition, PTQ increased mean NRMSE relative to FP by 17\.4513 percentage points in Dataset A and 17\.4046 percentage points in Dataset B\. The corresponding penalties decreased to 5\.1237 and 8\.1646 percentage points at five states, 2\.1185 and 2\.2876 percentage points at seven states, and only 0\.2951 and 0\.3063 percentage points at 16 states\. QAT\-FT remained substantially closer to FP at every tested state count\. At three states, its residual mean NRMSE difference from FP was 1\.7855 percentage points in Dataset A and 1\.0687 percentage points in Dataset B; at 16 states, these differences decreased to only 0\.0067 and 0\.0534 percentage points, respectively\. These results identify the three\-state condition as the strongest quantization stress regime and show that the separation between PTQ and QAT\-FT becomes particularly pronounced as the available weight\-state representation becomes more restrictive \(Fig\.[1](https://arxiv.org/html/2610.28784#S3.F1)\)\.

Figure 1:State\-dependent PTQ degradation and QAT\-FT residual gap relative to full precision\. Mean NRMSE differences relative to the full\-precision reference are shown for direct post\-training quantization \(PTQ\) and quantization\-aware fine\-tuning \(QAT\-FT\) at 3, 5, 7, and 16 weight states in Datasets A and B\. Dataset\-A state\-level means summarize 12 configurations per state across three load series, two forecasting horizons, and two quantization locations; Dataset\-B means summarize four configurations per state across one load series, two horizons, and two quantization locations\. PTQ degradation is largest under the three\-state constraint and decreases strongly as state resolution increases, whereas the residual QAT\-FT difference from full precision remains substantially smaller across the tested state counts\.The same overall pattern was broadly supported across the structured configuration comparisons\. PTQ produced higher NRMSE than FP in 61 of 64 configurations \(95\.31%\), while QAT\-FT produced lower NRMSE than PTQ in 62 of 64 configurations \(96\.88%\)\. Within Dataset A, QAT\-FT outperformed the corresponding PTQ condition in all 48/48 matched cases, whereas in Dataset B it did so in 14/16 cases\. The few non\-supportive comparisons occurred in H1 FFN\-Q conditions and are documented in the Supplementary Information\. Collectively, the results show that direct PTQ becomes increasingly detrimental as the available weight\-state representation becomes more restrictive, whereas independently evaluated QAT\-FT provides substantially greater robustness to low\-state quantization and closes most of the FP–PTQ performance gap in both co\-primary datasets\.

### 3\.2Competitive low\-state solutions can also be trained directly from scratch

We next examined whether competitive low\-state solutions could be obtained without initialization from a pretrained full\-precision model\. Both QAT\-FT and Scratch\-QAT use the same underlying quantization\-aware formulation, in which continuous shadow weights are maintained while quantized weights are used in the forward pass; QAT\-FT begins from a pretrained FP checkpoint, whereas Scratch\-QAT begins from random Glorot\-uniform initialization and follows its separately frozen training recipe\. For the primary Scratch\-QAT endpoint at 40 epochs \(E40\), averaged across the tested 3\-, 5\-, 7\-, and 16\-state conditions, both quantization locations, both forecasting horizons, and the load series within each dataset, Scratchα=0\\alpha=0achieved a mean NRMSE of 11\.8001% in Dataset A, compared with 11\.9424% for pretrained QAT\-FT\. In Dataset B, the corresponding values were 57\.6379% and 58\.0938%, respectively \(Fig\.[2](https://arxiv.org/html/2610.28784#S3.F2); Table[3](https://arxiv.org/html/2610.28784#S3.T3)\)\. Thus, the dataset\-level E40 averages slightly favored Scratch\-QAT in both datasets\. However, this advantage was not uniform across individual configurations: Scratchα=0\\alpha=0E40 outperformed QAT\-FT in 16/48 Dataset\-A configurations and 13/16 Dataset\-B configurations, corresponding to 29/64 \(45\.31%\) across the complete structured comparison set\. These results therefore support the competitiveness of direct Scratch\-QAT, rather than a general superiority over pretrained QAT\-FT\.

The strongest and most reproducible Scratch\-QAT advantage occurred under the most restrictive three\-state quantization\. At the E40 endpoint, Scratchα=0\\alpha=0achieved a mean NRMSE 1\.1717 percentage points lower than QAT\-FT in Dataset A and 0\.8942 percentage points lower in Dataset B\. At the configuration level, Scratch\-QAT was favorable in 9/12 three\-state cases in Dataset A and 3/4 cases in Dataset B, giving 12/16 favorable three\-state comparisons overall\. Moreover, at the load\-series summary level, three\-state Scratchα=0\\alpha=0was favorable relative to QAT\-FT across all four distinct load series nested within the two co\-primary datasets\. This concentration of support at three states is particularly notable because the same state count represented the strongest quantization\-stress condition identified in Section 3\.1 \(Fig\.[1](https://arxiv.org/html/2610.28784#S3.F1)\)\.

Figure 2:Direct Scratch\-QAT relative to pretrained QAT\-FT across low\-state weight representations\. State\-dependent forecasting performance is compared for QAT\-FT and the primary E40 Scratch\-QAT conditions across 3, 5, 7, and 16 states in Datasets A and B\. The clearest common advantage of direct Scratch\-QAT occurs under the most restrictive three\-state representation\. At higher state counts, the relative behavior is less uniform, supporting the interpretation that Scratch\-QAT is a competitive direct\-training pathway rather than a generally superior replacement for pretrained QAT\-FT\. QAT\-FT uses its frozen 25\-epoch fine\-tuning procedure, whereas E40 is the frozen primary Scratch\-QAT endpoint\.To determine whether the favorable low\-state Scratch\-QAT behavior was confined to the aggregate H24 metric or persisted across the forecast horizon, NRMSE was examined separately for forecast hours 1–24 \(Fig\.[3](https://arxiv.org/html/2610.28784#S3.F3)\)\. The three\-state Scratch\-QAT advantage extended across the full forecast horizon\. In Dataset A, mean three\-state Scratchα=0\\alpha=0NRMSE changed from 10\.4257% at hour 1 to 15\.6494% at hour 24, compared with 12\.0979% to 16\.7757% for QAT\-FT\. In Dataset B, the corresponding Scratch trajectory changed from 51\.9642% to 65\.4218%, compared with 53\.0955% to 66\.4997% for QAT\-FT\. Thus, the mean three\-state Scratchα=0\\alpha=0trajectory remained lower than QAT\-FT at all 24 forecast hours in both datasets\.

Figure 3:Forecast\-lead dependence of H24 forecasting performance across quantization states and training strategies\. NRMSE is shown separately for forecast hours 1–24\. Columns correspond to 3, 5, 7, and 16 quantization states, with Dataset A shown in the upper row and Dataset B in the lower row\. Curves show FP, PTQ, QAT\-FT, primary E40 Scratch\-QAT withα=0\\alpha=0, and E40 Scratch\-QAT with the frozen high\-α\\alphacondition\. For Dataset A, each quantized\-method trajectory is the descriptive mean across the three load series and two quantization locations at the indicated state count; for Dataset B, each trajectory is the mean across the two quantization locations\. FP is state independent and is shown as the corresponding dataset\-level reference in each column\.High\-α\\alphathresholding did not introduce a qualitatively different forecast\-lead pattern\. At three states, the high\-α\\alphatrajectory changed from 10\.4327% to 15\.5368% in Dataset A and from 51\.9798% to 64\.6495% in Dataset B, remaining close to the paired Scratchα=0\\alpha=0trajectory throughout the horizon\. Similar overlap was observed across the other state counts, consistent with the small aggregate accuracy changes accompanying the large activity reductions reported in Section 3\.3\. Overall, the per\-hour analysis shows that the central low\-state findings are not restricted to the aggregate H24 metric: PTQ degradation, QAT\-FT recovery, the particularly strong three\-state Scratch\-QAT behavior, and the broadly preserved high\-α\\alphaforecasting trajectory are all visible across the 24 simultaneously predicted forecast hours\.

Taken together, these results show that the quantization\-aware training formulation can produce competitive low\-state models from either a pretrained FP starting point or random initialization, although the relative advantage depends on the quantization regime\. Scratch\-QAT does not consistently outperform pretrained QAT\-FT across the full configuration space; instead, its clearest reproducible advantage occurs under the most restrictive three\-state condition\. Accordingly, Scratch\-QAT is best interpreted as a viable direct\-training pathway for low\-state models, rather than as a universally superior replacement for pretrained QAT\-FT\.

### 3\.3Accumulated\-update thresholding strongly suppresses committed and discrete target\-state activity

We next evaluated whether accumulated\-update thresholding could reduce target\-weight activity during Scratch\-QAT while preserving forecasting performance\. Scratchα=0\\alpha=0and Scratch high\-α\\alphawere evaluated as exactly paired conditions, with only the accumulated\-update threshold differing, as defined in Sections 2\.6–2\.7\. Results are reported separately for committed shadow\-weight updates and discrete quantized state transitions, which represent distinct levels of training activity\.

Figure 4:Forecasting accuracy and target\-state activity under accumulated\-update thresholding\. Scratch high\-α\\alphais compared with its exactly paired Scratchα=0\\alpha=0control across the 3\-, 5\-, 7\-, and 16\-state conditions in both co\-primary datasets\. High\-α\\alphareduced committed shadow\-weight updates by 99\.7271% in Dataset A and 99\.6741% in Dataset B, while discrete quantized state transitions were reduced by 79\.9747% and 77\.0203%, respectively\. These large activity reductions were accompanied by mean NRMSE changes of only \+0\.0416 and \+0\.1965 percentage points\. Committed\-update suppression remained above 99% at every tested state count in both datasets, whereas state\-transition suppression remained substantial but decreased as the number of available states increased\.Figure 5:Epoch\-wise evolution of accumulated\-update thresholding activity during final Scratch\-QAT training\. Curves show the mean within\-configuration cumulative quantities across the 48 Dataset\-A and 16 Dataset\-B pairedα=0\\alpha=0/high\-α\\alphafinal configurations over 40 epochs\. \(a\) Cumulative optimizer\-proposed target\-weight changes under high\-α\\alpha, expressed relative to the pairedα=0\\alpha=0control; values remain near 100%, showing that the threshold does not suppress proposal generation\. \(b\) Cumulative reduction in committed shadow\-weight updates relative toα=0\\alpha=0\. \(c\) Cumulative reduction in discrete quantized state\-transition events relative toα=0\\alpha=0\. \(d\) Cumulative commit\-acceptance percentage under high\-α\\alpha\. Reductions were calculated within matched configurations before averaging so that architectures and quantization locations with different numbers of targeted parameters were not differentially weighted\. These quantities describe algorithmic committed and discrete target\-state activity; they do not represent measured physical memory writes, write energy, endurance, or hardware lifetime\.Across the tested 3\-, 5\-, 7\-, and 16\-state conditions, Scratch high\-α\\alphareduced committed shadow\-weight updates by an average of 99\.7271% in Dataset A and 99\.6741% in Dataset B relative to the paired Scratchα=0\\alpha=0condition \(Fig\.[4](https://arxiv.org/html/2610.28784#S3.F4); Table[3](https://arxiv.org/html/2610.28784#S3.T3)\)\. Mean commit acceptance rates were correspondingly only 0\.2716% and 0\.3247%, respectively, indicating that fewer than one\-half of one percent of nonzero optimizer\-proposed target\-weight updates resulted in committed shadow\-weight updates on average\. These reductions in committed activity were accompanied by substantially fewer discrete quantized state transitions, with mean reductions of 79\.9747% in Dataset A and 77\.0203% in Dataset B\. The transition reduction remained substantial at every tested state count: 88\.07%/85\.24% in Dataset A/B at three states, 84\.54%/80\.95% at five states, 80\.69%/78\.90% at seven states, and 66\.61%/62\.98% at 16 states\. In contrast, committed shadow\-weight update reduction remained above 99% at every state count in both datasets, further demonstrating that committed shadow\-weight updates and discrete state transitions represent different levels of training activity\.

To examine whether this suppression persisted throughout training, the complete final Scratch\-QAT histories were analyzed over the 40 training epochs \(Fig\.[5](https://arxiv.org/html/2610.28784#S3.F5)\)\. Within each paired configuration, cumulative high\-α\\alphaactivity was compared with its matchedα=0\\alpha=0control before averaging across configurations\. Optimizer\-proposed target\-weight activity remained essentially unchanged, with the E40 ratio of cumulative proposed\-update activity under high\-α\\alphato that underα=0\\alpha=0reaching 100\.37% in Dataset A and 100\.30% in Dataset B\. In contrast, committed\-update suppression reached 99\.73%/99\.67% at E40, while discrete state\-transition suppression reached 79\.97%/77\.02%\. Mean cumulative commit acceptance remained below 1% throughout training and ended at 0\.272%/0\.325%\. These trajectories confirm that thresholding persistently suppresses committed and discrete target\-state activity rather than reducing optimizer\-proposed activity\.

These large activity reductions were accompanied by comparatively small changes in forecasting error\. In Dataset A, mean NRMSE changed from 11\.8001% for Scratchα=0\\alpha=0to 11\.8417% for Scratch high\-α\\alpha, a difference of only \+0\.0416 percentage points\. High\-α\\alphanevertheless achieved lower NRMSE than its pairedα=0\\alpha=0run in 22/48 configurations, while 39/48 remained within \+0\.25 percentage points of theα=0\\alpha=0result\. In Dataset B, mean NRMSE changed from 57\.6379% to 57\.8345%, corresponding to \+0\.1965 percentage points; high\-α\\alphaachieved lower NRMSE in 6/16 configurations and remained within \+0\.25 percentage points in 9/16\. Thus, orders\-of\-magnitude suppression of committed activity and large reductions in discrete state transitions were obtained without a comparably large change in mean forecasting performance\.

The most aggressive three\-state condition provides a particularly informative example of this trade\-off\. In Dataset A, Scratch high\-α\\alphareduced committed shadow\-weight updates by 99\.9036% and discrete state transitions by 88\.0657% relative to Scratchα=0\\alpha=0, while mean NRMSE was 0\.1602 percentage points lower\. Dataset B showed the same state\-specific direction: committed shadow\-weight updates fell by 99\.8892%, state transitions by 85\.2441%, and mean NRMSE was 0\.0700 percentage points lower than underα=0\\alpha=0\.

Overall, accumulated\-update thresholding reduced committed shadow\-weight updates by approximately 99\.7% and discrete state transitions by approximately 77–80% across the two datasets while maintaining comparable forecasting accuracy\. These reductions concern committed and discrete target\-state activity rather than fewer Adam steps or lower conventional training FLOPs, potentially making the approach relevant to low\-state implementations in which physical state changes incur write costs; however, any hardware implications remain prospective\.

### 3\.4Representative forecast traces under frozen evaluation settings

Representative forecast traces \(Figs\.[6](https://arxiv.org/html/2610.28784#S3.F6)and[7](https://arxiv.org/html/2610.28784#S3.F7)\) are provided to visually complement the aggregate final\-test analyses\. Dataset A is represented by MT\_022 and Dataset B by Household\_Power\. Quantized comparisons use the broader Attention\+FFN\-Q condition at 3, 5, 7, and 16 states and include PTQ, QAT\-FT, E40 Scratch\-QAT withα=0\\alpha=0, and E40 Scratch\-QAT with the frozen high\-α\\alphasetting\.

The displayed weeks were selected without reference to any quantized\-model predictions\. Complete Monday–Sunday weeks were first ranked by actual\-load variability, and only the 50% most variable weeks were retained; among these, the week whose FP H1 NRMSE was closest to the corresponding full final\-test FP H1 NRMSE was selected\. This yielded 27 October–2 November 2014 for MT\_022 and 2–8 November 2009 for Household\_Power\. The same calendar weeks were used for the H24 visualization, where seven non\-overlapping direct 24\-h forecasts initiated at 00:00 each day were concatenated chronologically\.

Figure 6:Representative H1 forecast traces across training strategies\.Figure 7:Representative H24 forecast traces across training strategies\.
### 3\.5Primary effects reproduce across datasets and load series

The principal findings reproduced across the two co\-primary datasets under the methodology transferred from Dataset A without Dataset\-B\-specific retuning\. At the series\-summary level, all four distinct load series retained the expected directions for PTQ degradation, QAT\-FT improvement over PTQ, committed\-update suppression, and discrete state\-transition suppression \(Table[4](https://arxiv.org/html/2610.28784#S3.T4)\)\.

Table 4:Series\-level mean effects across the four load series\.Directional consistency was also retained after stratification by each principal experimental factor\. In both datasets, all four primary effects retained their expected direction at every tested state count, both forecasting horizons, and both quantization locations \(Table[5](https://arxiv.org/html/2610.28784#S3.T5)\)\.

Table 5:Cross\-dataset directional concordance across the principal experimental factors\.Leave\-one\-series\-out sensitivity analysis further tested whether any principal conclusion depended disproportionately on a single load series \(Table[6](https://arxiv.org/html/2610.28784#S3.T6)\)\.

Table 6:Leave\-one\-series\-out sensitivity of the four primary effects\.Every primary conclusion retained its expected direction after removal of MT\_004, MT\_022, MT\_324, or Household\_Power individually\. The distinctive three\-state Scratch\-QAT result was also consistent across the four series: at the series\-summary level, Scratch\-QATα=0\\alpha=0achieved lower mean NRMSE than QAT\-FT for all four distinct load series\. Together, these results show that the principal low\-state effects were preserved across datasets, load series, state counts, forecasting horizons, and quantization locations and were not dependent on any single series\.

## 4Discussion

### 4\.1Why aggressive PTQ fails most severely at very low state counts

The strong state dependence of the PTQ penalty is consistent with the increasingly coarse representation imposed as the number of available weight states decreases\. Because the FP model is optimized in a continuous parameter space and only quantized after training, aggressive PTQ can displace learned weights from their optimized values without allowing the model to adapt to the discrete constraint\. As the state count increases, the quantization grid becomes finer and this mismatch is reduced, consistent with the progressive approach toward FP performance observed at higher state counts\. This interpretation agrees with prior studies showing that direct low\-precision post\-training quantization can degrade model accuracy, particularly under restrictive numerical representations\([Krishnamoorthi, 2018](https://arxiv.org/html/2610.28784#bib.bib17);[Nagel et al\., 2019](https://arxiv.org/html/2610.28784#bib.bib24)\)\.

### 4\.2Why QAT\-FT is more robust than direct PTQ

The greater robustness of QAT\-FT is consistent with its ability to optimize the model while the targeted weights are already subject to the low\-state representation during the forward pass\. Rather than projecting an FP solution onto a discrete grid only after optimization, QAT\-FT allows the continuous shadow weights to adapt under the errors introduced by quantization, enabling the learned solution to accommodate the available weight states\. This distinction becomes particularly important under aggressive quantization, where the mismatch between an unconstrained FP solution and its discrete representation is largest\. Similar benefits of incorporating quantization into the training process have been reported in prior low\-precision neural\-network studies\([Hubara et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib13);[Jacob et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib14)\)\.

### 4\.3What Scratch\-QAT adds

Scratch\-QAT addresses a different question from QAT\-FT: rather than adapting a pretrained FP solution to a low\-state representation, it tests whether a useful solution can emerge directly while quantization is active throughout training\. The fact that competitive models could be obtained from random initialization therefore indicates that full\-precision pretraining is not strictly required for learning under severe weight\-state constraints\. The particularly favorable behavior observed at three states is notable because this is also the regime in which direct PTQ was most disruptive, suggesting that learning directly within the constrained representation can be especially valuable when the available state space is extremely limited\. On average, Scratchα=0\\alpha=0achieved slightly lower NRMSE than QAT\-FT in both datasets, by 0\.1423 and 0\.4559 percentage points in Datasets A and B, respectively\. However, because the advantage was not consistent across all state counts and configurations, Scratch\-QAT should be interpreted as a viable alternative training pathway without requiring full\-precision pretraining, rather than a generally superior replacement for pretrained QAT\-FT\.

### 4\.4Accumulated\-update thresholding changes the relevant notion of training activity

A key implication of accumulated\-update thresholding is that optimizer activity and physically meaningful state\-changing activity need not be equivalent\. Conventional training treats each optimizer update as part of the computational workload, whereas a low\-state implementation may be more sensitive to how often those updates ultimately require a committed weight change or a transition between discrete states\. The strong suppression of such activity with little change in forecasting performance suggests that many small optimizer\-driven adjustments are not individually necessary for maintaining an effective discrete solution\. Accumulating these adjustments before committing them therefore provides a form of temporal filtering of parameter updates, potentially allowing the optimization process to retain gradual information while avoiding unnecessary state changes\. This behavior could be particularly advantageous for hardware implementations based on multi\-state synaptic devices, where changing a stored weight may require a physical programming operation\. The observed reduction in committed and discrete target\-state activity is therefore potentially relevant to implementations in which state changes incur physical programming costs; quantifying any resulting effects on write activity, latency, or endurance will require device\-specific validation beyond the present study\.

## 5Conclusion

This study shows that extremely low\-state Transformer learning remains viable for electricity\-load forecasting, but that the training pathway becomes critical as the available weight representation becomes increasingly discrete\. Direct PTQ produced substantial degradation at the lowest state counts, whereas QAT\-FT recovered most of the resulting accuracy loss across both co\-primary datasets\. Scratch\-QAT further demonstrated that competitive low\-state solutions can emerge directly from random initialization without requiring a pretrained full\-precision precursor, with particularly favorable behavior in the three\-state regime\. Accumulated\-update thresholding provided a complementary reduction in training activity by allowing small optimizer\-proposed changes to accumulate before being committed, reducing committed shadow\-weight updates by approximately 99\.7% and discrete state transitions by approximately 77–80% while producing only small mean changes in forecasting error\. The principal PTQ, QAT\-FT, and thresholding effects remained consistent under transfer of the frozen methodology to the second co\-primary dataset without dataset\-specific retuning\. Together, these findings show that aggressive weight\-state reduction, direct low\-state learning, and strong suppression of committed state\-changing activity can coexist within a Transformer forecasting framework\. Looking forward, AI hardware in which nanoscale multilevel nonvolatile devices directly encode highly quantized weights and are integrated with CMOS neuron and peripheral circuitry could provide a natural architecture for translating such low\-state learning into compact, high\-performance, and energy\-efficient computing systems\([Le Gallo et al\., 2018](https://arxiv.org/html/2610.28784#bib.bib18);[Sebastian et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib29);[Nandakumar et al\., 2020](https://arxiv.org/html/2610.28784#bib.bib27);[Misba et al\., 2022](https://arxiv.org/html/2610.28784#bib.bib23);[Alam et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib3);[Aguirre et al\., 2024](https://arxiv.org/html/2610.28784#bib.bib1);[Dhull et al\., 2025](https://arxiv.org/html/2610.28784#bib.bib8)\)\.

## 6Acknowledgments

This research was supported by the Commonwealth Cyber Initiative \(CCI\) under Grant No\. VV\-1Q23\-008\.

## References

- Aguirre et al\. \(2024\)F\. Aguirre, A\. Sebastian, M\. Le Gallo, et al\.Hardware implementation of memristor\-based artificial neural networks\.*Nature Communications*, 15:1974, 2024\.doi:10\.1038/s41467\-024\-45670\-9\.
- Al Misba et al\. \(2023\)Walid Al Misba, Harindra S\. Mavikumbure, Md Mahadi Rajib, Daniel L\. Marino, Victor Cobilean, Milos Manic, and Jayasimha Atulasimha\.Spintronic physical reservoir for autonomous prediction and long\-term household energy load forecasting\.*IEEE Access*, 11:124725–124737, 2023\.doi:10\.1109/ACCESS\.2023\.3326414\.
- Alam et al\. \(2024\)Muhammad Sabbir Alam, Walid Al Misba, and Jayasimha Atulasimha\.Quantized non\-volatile nanomagnetic domain wall synapse based autoencoder for efficient unsupervised network anomaly detection\.*Neuromorphic Computing and Engineering*, 4\(2\):024012, 2024\.doi:10\.1088/2634\-4386/ad49ce\.
- Amarasinghe et al\. \(2017\)Kasun Amarasinghe, Daniel L\. Marino, and Milos Manic\.Deep neural networks for energy load forecasting\.In*2017 IEEE 26th International Symposium on Industrial Electronics \(ISIE\)*, pages 1483–1488, 2017\.doi:10\.1109/ISIE\.2017\.8001465\.
- Azam et al\. \(2020\)Md Ali Azam, Dhritiman Bhattacharya, Damien Querlioz, Caroline A\. Ross, and Jayasimha Atulasimha\.Voltage control of domain walls in magnetic nanowires for energy\-efficient neuromorphic devices\.*Nanotechnology*, 31\(14\):145201, 2020\.doi:10\.1088/1361\-6528/ab6234\.
- Bettayeb et al\. \(2024\)Meriem Bettayeb, Yasmin Halawani, Muhammad Umair Khan, Hani Saleh, and Baker Mohammad\.Efficient memristor accelerator for transformer self\-attention functionality\.*Scientific Reports*, 14:24173, 2024\.doi:10\.1038/s41598\-024\-75021\-z\.
- Chan and Yeo \(2024\)Jun Wei Chan and Chai Kiat Yeo\.A transformer based approach to electricity load forecasting\.*The Electricity Journal*, 37\(2\):107370, 2024\.doi:10\.1016/j\.tej\.2024\.107370\.
- Dhull et al\. \(2025\)Seema Dhull, Walid Al Misba, Arshid Nisar, Jayasimha Atulasimha, and Brajesh Kumar Kaushik\.Quantized magnetic domain wall synapse for efficient deep neural networks\.*IEEE Transactions on Neural Networks and Learning Systems*, 36\(3\):4996–5005, 2025\.doi:10\.1109/TNNLS\.2024\.3369969\.
- Esser et al\. \(2020\)Steven K\. Esser, Jeffrey L\. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S\. Modha\.Learned step size quantization\.In*International Conference on Learning Representations*, 2020\.
- Frantar et al\. \(2023\)Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.GPTQ: Accurate post\-training quantization for generative pre\-trained transformers\.In*The Eleventh International Conference on Learning Representations*, 2023\.
- Hebrail and Berard \(2006\)Georges Hebrail and Alice Berard\.Individual household electric power consumption, 2006\.
- Hong and Fan \(2016\)Tao Hong and Shu Fan\.Probabilistic electric load forecasting: A tutorial review\.*International Journal of Forecasting*, 32\(3\):914–938, 2016\.doi:10\.1016/j\.ijforecast\.2015\.11\.011\.
- Hubara et al\. \(2018\)Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El\-Yaniv, and Yoshua Bengio\.Quantized neural networks: Training neural networks with low precision weights and activations\.*Journal of Machine Learning Research*, 18\(187\):1–30, 2018\.
- Jacob et al\. \(2018\)Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko\.Quantization and training of neural networks for efficient integer\-arithmetic\-only inference\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 2704–2713, 2018\.doi:10\.1109/CVPR\.2018\.00286\.
- Jiang et al\. \(2026\)Zhixing Jiang, Han Zhao, Jianshi Tang, Yuyao Lu, Qi Qin, Ze Wang, Ruofei Hu, Ruihua Yu, Yuan He, Junyang Zhang, Mingcheng Shi, Ning Deng, Bin Gao, He Qian, and Huaqiang Wu\.Strategies of high\-accuracy memristor\-based analogue computing in memory for artificial intelligence\.*Nature Materials*, 25:1110–1124, 2026\.doi:10\.1038/s41563\-026\-02600\-y\.
- Joshi et al\. \(2020\)V\. Joshi, M\. Le Gallo, S\. Haefeli, I\. Boybat, S\. R\. Nandakumar, C\. Piveteau, M\. Dazzi, B\. Rajendran, A\. Sebastian, and E\. Eleftheriou\.Accurate deep neural network inference using computational phase\-change memory\.*Nature Communications*, 11\(1\), 2020\.doi:10\.1038/s41467\-020\-16108\-9\.
- Krishnamoorthi \(2018\)Raghuraman Krishnamoorthi\.Quantizing deep convolutional networks for efficient inference: A whitepaper\.*arXiv preprint arXiv:1806\.08342*, 2018\.doi:10\.48550/arXiv\.1806\.08342\.
- Le Gallo et al\. \(2018\)Manuel Le Gallo, Abu Sebastian, Roland Mathis, Matteo Manica, Heiner Giefers, Tomas Tuma, Costas Bekas, Alessandro Curioni, and Evangelos Eleftheriou\.Mixed\-precision in\-memory computing\.*Nature Electronics*, 1:246–253, 2018\.doi:10\.1038/s41928\-018\-0054\-8\.
- Li et al\. \(2021\)Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu\.BRECQ: Pushing the limit of post\-training quantization by block reconstruction\.In*International Conference on Learning Representations*, 2021\.
- Lim et al\. \(2021\)Bryan Lim, Sercan Ö\. Arık, Nicolas Loeff, and Tomas Pfister\.Temporal fusion transformers for interpretable multi\-horizon time series forecasting\.*International Journal of Forecasting*, 37\(4\):1748–1764, 2021\.doi:10\.1016/j\.ijforecast\.2021\.03\.012\.
- Lin et al\. \(2024\)Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems*, volume 6, 2024\.
- Liu et al\. \(2024\)Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long\.iTransformer: Inverted transformers are effective for time series forecasting\.In*International Conference on Learning Representations*, 2024\.
- Misba et al\. \(2022\)W\. A\. Misba, M\. Lozano, D\. Querlioz, and J\. Atulasimha\.Energy efficient learning with low resolution stochastic domain wall synapse for deep neural networks\.*IEEE Access*, 10:84946–84959, 2022\.doi:10\.1109/ACCESS\.2022\.3196688\.
- Nagel et al\. \(2019\)Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling\.Data\-free quantization through weight equalization and bias correction\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 1325–1334, 2019\.
- Nagel et al\. \(2020\)Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort\.Up or down? adaptive rounding for post\-training quantization\.In*Proceedings of the 37th International Conference on Machine Learning*, volume 119 of*Proceedings of Machine Learning Research*, pages 7197–7206, 2020\.
- Nagel et al\. \(2021\)Markus Nagel, Mart van Baalen Fournarakis, et al\.A white paper on neural network quantization\.*arXiv preprint arXiv:2106\.08295*, 2021\.doi:10\.48550/arXiv\.2106\.08295\.
- Nandakumar et al\. \(2020\)S\. R\. Nandakumar, Manuel Le Gallo, Christophe Piveteau, Vinay Joshi, Giovanni Mariani, Irem Boybat, Geethan Karunaratne, Riduan Khaddam\-Aljameh, Urs Egger, Anastasios Petropoulos, Theodore Antonakopoulos, Bipin Rajendran, Abu Sebastian, and Evangelos Eleftheriou\.Mixed\-precision deep learning based on computational memory\.*Frontiers in Neuroscience*, 14:406, 2020\.doi:10\.3389/fnins\.2020\.00406\.
- Nie et al\. \(2023\)Yuqi Nie, Nam H\. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam\.A time series is worth 64 words: Long\-term forecasting with transformers\.In*International Conference on Learning Representations*, 2023\.
- Sebastian et al\. \(2020\)Abu Sebastian, Manuel Le Gallo, Riduan Khaddam\-Aljameh, and Evangelos Eleftheriou\.Memory devices and applications for in\-memory computing\.*Nature Nanotechnology*, 15:529–544, 2020\.doi:10\.1038/s41565\-020\-0655\-z\.
- Sehgal et al\. \(2025\)Anubha Sehgal, Muhammad Sabbir Alam, Jayasimha Atulasimha, and Sourajeet Roy\.Energy efficient on\-chip learning using quantized triple\-level sot\-mram based synapse\.TechRxiv preprint, 2025\.
- Shao et al\. \(2024\)Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo\.OmniQuant: Omnidirectionally calibrated quantization for large language models\.In*International Conference on Learning Representations*, 2024\.
- Su et al\. \(2025\)Liyilei Su, Xumin Zuo, Rui Li, Xin Wang, Heng Zhao, and Bingding Huang\.A systematic review for transformer\-based long\-term series forecasting\.*Artificial Intelligence Review*, 58:80, 2025\.doi:10\.1007/s10462\-024\-11044\-2\.
- Trindade \(2015\)Artur Trindade\.Electricityloaddiagrams20112014, 2015\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N\. Gomez, Lukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, volume 30, pages 5998–6008, 2017\.
- Wu et al\. \(2021\)Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long\.Autoformer: Decomposition transformers with auto\-correlation for long\-term series forecasting\.In*Advances in Neural Information Processing Systems*, volume 34, 2021\.
- Xiao et al\. \(2023\)Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han\.SmoothQuant: Accurate and efficient post\-training quantization for large language models\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 38087–38099, 2023\.
- Zhou et al\. \(2021\)Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang\.Informer: Beyond efficient transformer for long sequence time\-series forecasting\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 35, pages 11106–11115, 2021\.doi:10\.1609/aaai\.v35i12\.17325\.
- Zhou et al\. \(2022\)Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin\.FEDformer: Frequency enhanced decomposed transformer for long\-term series forecasting\.In*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pages 27268–27286, 2022\.