SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers

# SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers
Source: [https://arxiv.org/html/2608.13702](https://arxiv.org/html/2608.13702)
Kiran NairThanks:Corresponding author:kiran\.prasannannair@coyotes\.usd\.eduAffiliation:USD Artificial Intelligence Research LabAffiliation:Department of Computer ScienceAffiliation:University of South DakotaAffiliation:Vermillion, SD 57069, USARodrigue RizkAffiliation:USD Artificial Intelligence Research LabAffiliation:Department of Computer ScienceAffiliation:University of South DakotaAffiliation:Vermillion, SD 57069, USAKC SantoshAffiliation:USD Artificial Intelligence Research LabAffiliation:Department of Computer ScienceAffiliation:University of South DakotaAffiliation:Vermillion, SD 57069, USA

###### Abstract

Spiking neural networks \(SNNs\) offer an energy\-efficient alternative to conventional deep neural networks by exploiting sparse event\-driven computation, but their training remains challenging because the non\-differentiable spike function requires surrogate gradients whose fixed shape may be suboptimal across layers and training stages\. In this work, we introduce SAGE, an uncertainty\-modulated surrogate\-gradient mechanism for Transformer\-based SNNs\. SAGE estimates block\-level uncertainty from normalized self\-attention entropy and uses this signal to adapt the surrogate\-gradient slope during training while leaving the inference model unchanged\. By modulating only the training\-time surrogate parameter, the proposed method preserves the original architecture and deployment cost while improving optimization flexibility\. Experiments on CIFAR\-10/100 demonstrate that SAGE achieves improved accuracy over fixed\-surrogate baselines, with results up to 1\-2% consistent gains across multiple simulation time steps\. These results highlight the potential of attention\-derived uncertainty as a lightweight training signal for adaptive surrogate\-gradient learning in transformer\-based SNNs\.

## 1Introduction

Spiking Neural Networks \(SNNs\), recognized as the third generation of neural network models, have emerged as an energy\-efficient, event\-driven paradigm for deep learning[24](https://arxiv.org/html/2608.13702#bib.bib33);[32](https://arxiv.org/html/2608.13702#bib.bib35)\. By replacing continuous floating\-point activations with discrete binary spikes \(00or11\), SNNs running on neuromorphic hardware replace resource\-heavy Multiply\-Accumulate operations with sparse, addition\-only \(AC\) operations, dramatically reducing computational energy consumption[14](https://arxiv.org/html/2608.13702#bib.bib40);[19](https://arxiv.org/html/2608.13702#bib.bib34)\. Building upon these biologically plausible properties, recent architectural advances have successfully introduced self\-attention mechanisms into SNNs\. Most notably, the Spikformer architecture[52](https://arxiv.org/html/2608.13702#bib.bib10)established Spiking Self\-Attention \(SSA\), which eliminates softmax normalizations to compute sparse, AC attention maps across spike\-form Query, Key, and Value tensors\. This breakthrough demonstrated that Spiking Vision Transformers \(ViTs\) can achieve competitive accuracy on large\-scale visual benchmarks while maintaining ultra\-low hardware energy profiles\.

Despite their efficient forward computation, training deep Spiking ViTs remains a fundamental challenge\. Since the Heaviside spike\-generation function is non\-differentiable, Backpropagation Through Time \(BPTT\) cannot be applied directly\. Instead, modern SNN training relies on surrogate gradients that approximate the derivative during backward propagation[21](https://arxiv.org/html/2608.13702#bib.bib3);[27](https://arxiv.org/html/2608.13702#bib.bib5);[43](https://arxiv.org/html/2608.13702#bib.bib36);[50](https://arxiv.org/html/2608.13702#bib.bib41)\. However, conventional surrogate gradient functions, such as static arctangent, Fast Sigmoid, or piecewise linear derivatives, apply fixed or globally uniform gradient response windows across all spatial sequence dimensions[2](https://arxiv.org/html/2608.13702#bib.bib38);[51](https://arxiv.org/html/2608.13702#bib.bib39)\. While recent works have explored temporal\-wise learnable surrogates to track time\-step dynamics[51](https://arxiv.org/html/2608.13702#bib.bib39), existing paradigms treat all spatial tokens identically during backpropagation, ignoring the complex, token\-level feature dynamics inherent to vision transformer layers\.

This spatially uniform gradient approximation creates a critical failure mode in deep Spiking transformers\. Visual tokens exhibit vastly different levels of semantic uncertainty: high\-confidence tokens \(such as distinct object boundaries\) require narrow, precise gradient updates, whereas high\-uncertainty tokens \(such as ambiguous background clutter or complex textures\) demand broader, exploratory gradient flow to discover optimal representations\. By enforcing a static surrogate window across all tokens, standard backpropagation fails to provide contextual credit assignment\. In deep network layers, this token\-blind gradient estimation exacerbates severe gradient vanishing and causes attention heads to collapse into “dead heads", where membrane potentials remain subthreshold and weight updates stall completely[52](https://arxiv.org/html/2608.13702#bib.bib10);[51](https://arxiv.org/html/2608.13702#bib.bib39)\. Consequently, a major challenge in scaling Spiking ViTs lies in developing an adaptive surrogate mechanism that dynamically tunes gradient flow to local token uncertainty without compromising the deterministic, low\-power binary execution of the forward pass\.

Motivated by this challenge, we propose SAGE \(Surrogate\-gradient Adaptation via attention\-Guided Entropy\), an uncertainty\-aware surrogate\-gradient framework for Spiking ViTs\. Instead of using a fixed or globally learnable surrogate throughout training, SAGE estimates the uncertainty of each transformer block from the dispersion of attention entropy across self\-attention heads and adaptively modulates the surrogate\-gradient slope during backpropagation\. The proposed framework operates exclusively during training, requiring no architectural modifications, additional inference\-time parameters, or changes to the forward computation\. Extensive experiments on CIFAR\-10, CIFAR\-100, and ImageNet\-200 demonstrate that SAGE consistently improves training effectiveness while preserving the efficiency and deployment characteristics of the original Spikformer\. Precisely, our contributions are as follows:

1. 1\.We introduce the first uncertainty\-aware adaptive surrogate\-gradient framework for spiking transformers\.
2. 2\.We demonstrate that entropy dispersion across attention heads provides a reliable online uncertainty signal for surrogate adaptation and is substantially more discriminative than alternative metrics for characterizing transformer\-block uncertainty\.
3. 3\.We validate SAGE on competitive datasets, where it consistently outperforms fixed and learnable surrogate baselines by up to 1\-2% Top\-1 accuracy while adding only 0\.03 ms training overhead per mini\-batch\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2608.13702#S2)presents the motivation for the proposed framework, establishing the need for uncertainty\-aware surrogate adaptation through theoretical insights and supporting literature\. Section[3](https://arxiv.org/html/2608.13702#S3)reviews the existing literature on surrogate\-gradient learning and Spiking ViTs\. Section[4](https://arxiv.org/html/2608.13702#S4)introduces the proposed SAGE framework, detailing the uncertainty estimation strategy and its integration into Spikformer training\. Section[5](https://arxiv.org/html/2608.13702#S5)describes the experimental setup and presents comprehensive evaluations\. Section[6](https://arxiv.org/html/2608.13702#S6)provides a detailed analysis and discussion of the experimental findings, while Section[7](https://arxiv.org/html/2608.13702#S7)concludes the paper\.

## 2Motivation

Deep SNNs are trained using surrogate gradients, which approximate the derivative of the non\-differentiable spike function during backpropagation\. Over the past decade, numerous surrogate\-gradient formulations have been proposed, differing primarily in their functional forms[49](https://arxiv.org/html/2608.13702#bib.bib42), smoothness characteristics[35](https://arxiv.org/html/2608.13702#bib.bib43), and gradient scaling strategies[43](https://arxiv.org/html/2608.13702#bib.bib36)\. Despite these differences, existing methods employ a fixed surrogate response for every neuron, treating all spike events identically regardless of the confidence of the underlying representation\. This assumption contrasts with modern learning paradigms, where intermediate representations contribute unequally to downstream predictions[40](https://arxiv.org/html/2608.13702#bib.bib32), and uncertainty estimation is widely used to identify ambiguous or unreliable features[10](https://arxiv.org/html/2608.13702#bib.bib44)\. Consequently, applying identical surrogate gradients to neurons encoding both highly certain and highly uncertain information may lead to suboptimal credit assignment during optimization[27](https://arxiv.org/html/2608.13702#bib.bib5)\.

These observations naturally raise the following question:should surrogate gradients remain uniform when the underlying representations exhibit different levels of uncertainty?We hypothesize that surrogate gradient modulation should instead reflect the uncertainty of the encoded representation\. Intuitively, confident representations require less surrogate smoothing, whereas uncertain representations may benefit from broader surrogate support to maintain effective gradient propagation\. This intuition is consistent with the broader principles of adaptive computation and uncertainty\-aware learning in modern deep networks[40](https://arxiv.org/html/2608.13702#bib.bib32);[10](https://arxiv.org/html/2608.13702#bib.bib44)\. Motivated by this observation, we propose an uncertainty\-modulated surrogate gradient that dynamically adapts the backward surrogate response using attention\-derived uncertainty while preserving the deterministic forward dynamics of the Spiking ViTs\. The proposed formulation is analyzed theoretically in Proposition[1](https://arxiv.org/html/2608.13702#Thmproposition1), which establishes the monotonic relationship between uncertainty and the surrogate gradient magnitude\.

###### Proposition 1\.

Letzcz\_\{c\}denote the centered normalized entropy\-dispersion statistic for a transformer block\. The adaptive surrogate\-gradient slope is defined as

α⁡\(zc\)=\{α0,\|zc\|<δ,α0\+β​tanh⁡\(zc\),\|zc\|≥δ,\\alpha\(z\_\{c\}\)=\\begin\{cases\}\\alpha\_\{0\},&\|z\_\{c\}\|<\\delta,\\\\ \\alpha\_\{0\}\+\\beta\\tanh\(z\_\{c\}\),&\|z\_\{c\}\|\\geq\\delta,\\end\{cases\}\(1\)whereα0\>0\\alpha\_\{0\}\>0,β\>0\\beta\>0, andδ\>0\\delta\>0\. Then, outside the dead\-zone region,α⁡\(zc\)\\alpha\(z\_\{c\}\)is a monotonically increasing function ofzcz\_\{c\}\.

###### Proof\.

For\|zc\|≥δ\|z\_\{c\}\|\\geq\\delta,

d​αd​zc=β⁡\(1−tanh2⁡\(zc\)\)\.\\frac\{d\\alpha\}\{dz\_\{c\}\}=\\beta\\left\(1\-\\tanh^\{2\}\(z\_\{c\}\)\\right\)\.\(2\)Sinceβ\>0\\beta\>0and1−tanh2⁡\(zc\)\>01\-\\tanh^\{2\}\(z\_\{c\}\)\>0for all finitezcz\_\{c\},

d​αd​zc\>0\.\\frac\{d\\alpha\}\{dz\_\{c\}\}\>0\.\(3\)Therefore,α\\alphais strictly monotonic with respect to the centered uncertainty statistic outside the dead\-zone region\. Within the dead zone,α=α0\\alpha=\\alpha\_\{0\}is constant, intentionally suppressing small uncertainty fluctuations\. ∎

## 3Related Works

This section reviews the three research directions most closely related to the proposed SAGE framework\. We first summarize advances in direct training of Spiking Neural Networks, where surrogate gradients have become the standard approach for optimizing deep SNNs\. We then discuss the evolution of Spiking ViTs, highlighting recent architectural developments built upon self\-attention mechanisms\. Finally, we review existing surrogate\-gradient optimization strategies and their limitations, thereby motivating the need for an uncertainty\-guided adaptive surrogate framework for spiking transformers\.

##### Spiking Neural Networks & Direct Training

SNNs represent a biologically inspired, event\-driven paradigm that achieves significant energy efficiency by transmitting discrete binary spikes rather than continuous floating\-point activations[11](https://arxiv.org/html/2608.13702#bib.bib1);[38](https://arxiv.org/html/2608.13702#bib.bib2)\. Early efforts to scale SNNs relied on Artificial Neural Network \(ANN\)\-to\-SNN conversion, which typically incurred excessive latency and high inference time steps[13](https://arxiv.org/html/2608.13702#bib.bib37)\. To overcome these latency bottlenecks, direct training via Spatio\-Temporal Backpropagation \(STBP\) and BPTT emerged as the dominant framework for optimizing deep architectures[21](https://arxiv.org/html/2608.13702#bib.bib3);[43](https://arxiv.org/html/2608.13702#bib.bib36)\. Direct training algorithms substitute the non\-differentiable Heaviside step function with smooth Surrogate Gradients during backward propagation, enabling competitive low\-latency execution[27](https://arxiv.org/html/2608.13702#bib.bib5)\. Recent advances have expanded direct SNN training across diverse domains, including dynamic spiking graph networks[46](https://arxiv.org/html/2608.13702#bib.bib6), asynchronous event\-based vision processors[48](https://arxiv.org/html/2608.13702#bib.bib7), and attention\-guided spiking architectures[45](https://arxiv.org/html/2608.13702#bib.bib4)\. Nevertheless, a fundamental limitation persists: direct training algorithms rely heavily on static or globally uniform surrogate derivative functions[27](https://arxiv.org/html/2608.13702#bib.bib5);[51](https://arxiv.org/html/2608.13702#bib.bib39), which fail to accommodate localized spatial variations in complex feature representations\.

##### Spiking Vision Transformers

To combine the representation capability of self\-attention[40](https://arxiv.org/html/2608.13702#bib.bib32)with the low\-power characteristics of neuromorphic processing, recent studies have adapted ViTs to spiking domain operations\. The seminal Spikformer architecture[52](https://arxiv.org/html/2608.13702#bib.bib10)introduced SSA, which eliminates softmax normalizations to compute sparse, AC attention maps across spike\-form Query, Key, and Value tensors\. Building on this foundation, subsequent work introduced multiscale spiking ViTs[47](https://arxiv.org/html/2608.13702#bib.bib11), hybrid event\-based detection models[44](https://arxiv.org/html/2608.13702#bib.bib12), biological visual mechanisms like saccadic attention[41](https://arxiv.org/html/2608.13702#bib.bib9), and structural bridges between ResNets and ViTs[34](https://arxiv.org/html/2608.13702#bib.bib8)\. Additionally, gating mechanisms[26](https://arxiv.org/html/2608.13702#bib.bib14)and dynamic time\-step allocation frameworks[7](https://arxiv.org/html/2608.13702#bib.bib13)have been explored to enhance information flow control and reduce latency\. However, while these architectures optimize forward\-pass execution and structural feature routing, their backward optimization remains constrained by static surrogate functions[52](https://arxiv.org/html/2608.13702#bib.bib10), leaving deep spiking transformer layers susceptible to vanishing gradients and dead attention heads\.

##### Surrogate Gradient Optimization

Overcoming the non\-differentiability of spiking activations has motivated extensive research into surrogate gradient formulation\. Conventional direct training paradigms employ fixed analytical derivatives, such as Fast Sigmoid, arctangent, or piecewise linear functions[2](https://arxiv.org/html/2608.13702#bib.bib38)\. To mitigate gradient vanishing and mismatch, researchers have developed adaptive and learnable surrogate functions[28](https://arxiv.org/html/2608.13702#bib.bib22);[5](https://arxiv.org/html/2608.13702#bib.bib23);[12](https://arxiv.org/html/2608.13702#bib.bib25), including adaptive smoothing gradient learning[42](https://arxiv.org/html/2608.13702#bib.bib24), cross\-layer threshold adaptations[1](https://arxiv.org/html/2608.13702#bib.bib16), and dual\-stage threshold\-gradient optimization[17](https://arxiv.org/html/2608.13702#bib.bib20)\. Recent extensions have further explored lightweight adaptive surrogates[15](https://arxiv.org/html/2608.13702#bib.bib17), membrane\-potential\-driven gradient scaling[16](https://arxiv.org/html/2608.13702#bib.bib18);[37](https://arxiv.org/html/2608.13702#bib.bib19), sparse low\-activity firing constraints[36](https://arxiv.org/html/2608.13702#bib.bib21), and adaptive surrogates for sequential reinforcement learning[39](https://arxiv.org/html/2608.13702#bib.bib15)\. Although these approaches dynamically modulate gradient shapes, they primarily adapt across global temporal dimensions or layer\-wide statistics[16](https://arxiv.org/html/2608.13702#bib.bib18);[51](https://arxiv.org/html/2608.13702#bib.bib39)\. Inspired by uncertainty\-guided learning[29](https://arxiv.org/html/2608.13702#bib.bib28);[20](https://arxiv.org/html/2608.13702#bib.bib29), confidence\-aware optimization[25](https://arxiv.org/html/2608.13702#bib.bib30);[23](https://arxiv.org/html/2608.13702#bib.bib31), and causal entropy principles[3](https://arxiv.org/html/2608.13702#bib.bib26);[6](https://arxiv.org/html/2608.13702#bib.bib27), there remains an unmapped frontier: formulating an uncertainty\-modulated, token\-level surrogate gradient that dynamically tunes its derivative window based on spatial attention entropy during backpropagation\.

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/umsen_arch.png)Figure 1:Overview of the proposedSAGE\(Surrogate\-gradientAdaptation via attention\-GuidedEntropy\) framework\.Top:Standard Spikformer architecture consisting of a spiking patch splitting module, SSA, Multilayer Perceptron\(MLP\) blocks, and a classification head\.Bottom:Detailed view of the SSA module\. Multi\-head attention \(MHA\) maps are generated from the spike\-based query \(QQ\), key \(KK\), and value \(VV\) representations\. Per\-head attention entropy is computed from each attention map, and the standard deviation across attention heads is used as an uncertainty signal to guide adaptive surrogate\-gradient modulation during training\. The forward inference path remains unchanged, while the uncertainty\-guided controller is active only during backpropagation\.

## 4Method

### 4\.1Overview of SAGE

Figure[1](https://arxiv.org/html/2608.13702#S3.F1)presents an overview of the proposedSAGE\(Surrogate\-gradientAdaptation via attention\-GuidedEntropy\) framework for adaptive surrogate\-gradient optimization in spiking transformers\. Built upon the standard Spikformer architecture[52](https://arxiv.org/html/2608.13702#bib.bib10), SAGE preserves the original forward inference pipeline and introduces no modifications to the network architecture or inference procedure\. Instead, the proposed framework operates exclusively during training by adaptively modulating the surrogate gradient used for backpropagation\. The central idea is that the multi\-head SSA module naturally produces attention distributions whose variability reflects the model’s uncertainty during optimization\. Rather than applying a fixed surrogate gradient uniformly across all transformer blocks throughout training, SAGE derives an uncertainty signal from the attention maps of each block and uses it to dynamically adjust the surrogate gradient parameter, enabling different blocks to receive optimization behavior that is consistent with their current uncertainty\.

Importantly, SAGE is a training\-time optimization framework and does not alter the inference process\. During inference, both the uncertainty estimation module and the adaptive controller are removed, leaving the original Spikformer architecture unchanged\. Consequently, SAGE preserves the inference graph, computational complexity, and latency of the baseline model while remaining readily applicable to existing spiking transformer architectures without introducing additional inference overhead\.

### 4\.2Attention Uncertainty Estimation

To derive a training signal that reflects the confidence of the network, SAGE exploits the attention distributions naturally produced by the SSA module, as illustrated in Figure[1](https://arxiv.org/html/2608.13702#S3.F1)\. Given the query and key spike representations, SSA module computes the raw attention scores as

𝐒=s​𝐐𝐊⊤,\\mathbf\{S\}=s\\,\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\},\(4\)wheres=0\.125s=0\.125is the fixed scaling factor used in the Spikformer[52](https://arxiv.org/html/2608.13702#bib.bib10)implementation\. These raw attention scores are used directly in the standard SSA forward computation and are not normalized by a softmax operation\. For uncertainty estimation only, SAGE constructs an auxiliary probability distribution from the detached attention scores\. Specifically, the scores are temperature\-scaled and normalized as

𝐀=Softmax⁡\(detach⁡\(𝐒\)Tent\),\\mathbf\{A\}=\\mathrm\{Softmax\}\\left\(\\frac\{\\mathrm\{detach\}\(\\mathbf\{S\}\)\}\{T\_\{\\mathrm\{ent\}\}\}\\right\),\(5\)wheredetach⁡\(⋅\)\\mathrm\{detach\}\(\\cdot\)indicates that the attention scores are excluded from gradient computation, ensuring that the uncertainty estimation influences only the surrogate\-gradient controller and does not modify the forward attention computation\. The entropy temperature is fixed toTent=0\.25T\_\{\\mathrm\{ent\}\}=0\.25, selected based on a one\-factor\-at\-a\-time \(OFAT\) sensitivity analysis over multiple temperature values \(see Appendix figure[5](https://arxiv.org/html/2608.13702#A1.F5)\)\. The resulting auxiliary distributions𝐀\(1\),𝐀\(2\),…,𝐀\(H\)\\mathbf\{A\}^\{\(1\)\},\\mathbf\{A\}^\{\(2\)\},\\ldots,\\mathbf\{A\}^\{\(H\)\}correspond to theHHattention heads and are used exclusively to compute the uncertainty signal for SAGE\. For each attention head, we quantify the uncertainty of its attention distribution using the normalized Shannon entropy[33](https://arxiv.org/html/2608.13702#bib.bib51),

Ei=−1log⁡N∑j=1NA\(i\)jlogA\(i\)j,E\_\{i\}=\-\\frac\{1\}\{\\log N\}\\sum\_\{j=1\}^\{N\}A^\{\(i\)\}\_\{j\}\\log A^\{\(i\)\}\_\{j\},\(6\)whereNNis the number of attention elements within a head and the normalization confines the entropy to the range\[0,1\]\[0,1\]\. While the mean entropy reflects the overall uncertainty of the attention mechanism, our discussion \(Section[6](https://arxiv.org/html/2608.13702#S6)\) shows that the variation of entropy across attention heads is considerably more informative during training\. Therefore, SAGE estimates the uncertainty of each transformer block using the dispersion of the per\-head entropies,

D=Std⁡\(E1,E2,…,EH\),D=\\mathrm\{Std\}\\left\(E\_\{1\},E\_\{2\},\\ldots,E\_\{H\}\\right\),\(7\)whereStd⁡\(⋅\)\\mathrm\{Std\}\(\\cdot\)denotes the standard deviation across theHHattention heads\. A larger value ofDDindicates greater disagreement among attention heads, suggesting higher uncertainty in the current representation, whereas a smaller value implies more consistent attention patterns\. This dispersion statistic serves as the uncertainty signal that drives the adaptive surrogate\-gradient controller described in the following subsection\.

Table 1:Comparison of state\-of\-the\-art SNN and Spiking Transformer methods on CIFAR\-10 and CIFAR\-100\.
### 4\.3Uncertainty\-Guided Surrogate Modulation

Conventional SNNs employ a fixed surrogate\-gradient parameter throughout training, resulting in identical gradient characteristics regardless of the optimization state or the confidence of intermediate representations\. In contrast, SAGE adapts the surrogate gradient according to the uncertainty signal extracted from the attention mechanism\. LetDt\(l\)D\_\{t\}^\{\(l\)\}denote the entropy\-dispersion estimate of transformer blockllat training steptt, obtained from Eq\. \([7](https://arxiv.org/html/2608.13702#S4.E7)\)\. Rather than using a fixed surrogate parameter \(e\.g\.,α=4\\alpha=4\), it computes a block\-specific surrogate parameter that evolves during optimization\. To obtain a stable control signal, the instantaneous dispersion is first smoothed using an exponential moving average \(EMA\),

D¯t\(l\)=β​D¯t−1\(l\)\+\(1−β\)​Dt\(l\),\\bar\{D\}\_\{t\}^\{\(l\)\}=\\beta\\bar\{D\}\_\{t\-1\}^\{\(l\)\}\+\(1\-\\beta\)D\_\{t\}^\{\(l\)\},\(8\)whereβ\\betadenotes the EMA decay factor\. The running meanμt\(l\)\\mu\_\{t\}^\{\(l\)\}and standard deviationσt\(l\)\\sigma\_\{t\}^\{\(l\)\}of the smoothed dispersion are then maintained throughout training\. Using these statistics, the current uncertainty is normalized as

zt\(l\)=D¯t\(l\)−μt\(l\)σt\(l\)\+ε,z\_\{t\}^\{\(l\)\}=\\frac\{\\bar\{D\}\_\{t\}^\{\(l\)\}\-\\mu\_\{t\}^\{\(l\)\}\}\{\\sigma\_\{t\}^\{\(l\)\}\+\\varepsilon\},\(9\)whereε\\varepsilonis a small constant for numerical stability\. The normalized uncertainty is further centered across theLLtransformer blocks to obtain a relative block\-wise uncertainty score,

z~t\(l\)=zt\(l\)−1L​∑r=1Lzt\(r\),\\tilde\{z\}\_\{t\}^\{\(l\)\}=z\_\{t\}^\{\(l\)\}\-\\frac\{1\}\{L\}\\sum\_\{r=1\}^\{L\}z\_\{t\}^\{\(r\)\},\(10\)wherez~t\(l\)\\tilde\{z\}\_\{t\}^\{\(l\)\}denotes the centered normalized entropy\-dispersion statistic of blockllat training steptt\. To suppress minor fluctuations around the block\-wise mean, SAGE applies a dead\-zone controller before modulating the surrogate\-gradient slope,

α^t\(l\)=\{4\.0,\|z~t\(l\)\|<0\.25,4\.0\+0\.5​tanh⁡\(z~t\(l\)\),otherwise,\\hat\{\\alpha\}\_\{t\}^\{\(l\)\}=\\begin\{cases\}4\.0,&\|\\tilde\{z\}\_\{t\}^\{\(l\)\}\|<0\.25,\\\\\[4\.0pt\] 4\.0\+0\.5\\tanh\\\!\\left\(\\tilde\{z\}\_\{t\}^\{\(l\)\}\\right\),&\\text\{otherwise\},\\end\{cases\}\(11\)followed by

αt\(l\)=clip⁡\(α^t\(l\),3,5\)\.\\alpha\_\{t\}^\{\(l\)\}=\\mathrm\{clip\}\\left\(\\hat\{\\alpha\}\_\{t\}^\{\(l\)\},\\,3,\\,5\\right\)\.\(12\)The centering operation enables SAGE to adapt the surrogate gradient relative to the uncertainty distribution across transformer blocks, while the dead zone prevents small variations from triggering unnecessary updates\. Thetanh⁡\(⋅\)\\tanh\(\\cdot\)mapping provides a smooth bounded modulation around the baseline valueα=4\\alpha=4, and the final clipping operation constrains the surrogate slope to the interval\[3,5\]\[3,5\]\. Consequently, transformer blocks exhibiting higher uncertainty receive a different surrogate gradient than more confident blocks, enabling the optimization process to adapt to the evolving attention dynamics while maintaining stable training\. During the initial warm\-up stage, the surrogate parameter is fixed atα=4\\alpha=4while the controller accumulates reliable running statistics before adaptive modulation begins\.

### 4\.4Integration into Spikformer Training

SAGE is designed as a lightweight training\-time optimization framework that can be integrated into existing Spikformer implementations with minimal modification\. During the forward pass, the network architecture, feature extraction process, and inference computation remain identical to the original Spikformer\. The proposed framework simply accesses the MHA maps generated by each SSA module and computes the uncertainty estimate described in section[4\.2](https://arxiv.org/html/2608.13702#S4.SS2)\. The resulting uncertainty signal is then passed through the adaptive controller in Section[4\.3](https://arxiv.org/html/2608.13702#S4.SS3)to obtain a block\-specific surrogate parameterα\\alpha, which is used only during backpropagation\. Consequently, SAGE introduces no architectural changes, additional learnable parameters, or inference\-time overhead\.

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/surrogate_comparison_T1.png)\(a\)T=1T=1
![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/surrogate_comparison_T2.png)\(b\)T=2T=2
![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/surrogate_comparison_T4.png)\(c\)T=4T=4
![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/surrogate_comparison_T8.png)\(d\)T=8T=8
![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/surrogate_comparison_T12.png)\(e\)T=12T=12

Figure 2:Top\-1 accuracy of fixed, learnable, and SAGE surrogate gradients across datasets and simulation timesteps\.The additional computation introduced by SAGE is negligible compared with the cost of self\-attention\. For an input sequence of lengthNNandHHattention heads, the dominant complexity of the attention mechanism remains the scaled dot\-product computation with complexity𝒪⁡\(N2\)\\mathcal\{O\}\(N^\{2\}\)\. In comparison, the proposed uncertainty estimation computes the entropy of each attention head in𝒪⁡\(H​N\)\\mathcal\{O\}\(HN\), followed by a standard deviation across heads with complexity𝒪⁡\(H\)\\mathcal\{O\}\(H\)\. The subsequent exponential moving average, running statistics, and adaptive controller involve only constant\-time operations per transformer block\. Since𝒪⁡\(H​N\)≪𝒪⁡\(N2\)\\mathcal\{O\}\(HN\)\\ll\\mathcal\{O\}\(N^\{2\}\)for practical transformer configurations, the overall computational complexity of Spikformer remains unchanged\. Experimental profiling further confirms that the adaptive controller incurs only approximately0\.030\.03ms of additional computation per mini\-batch during training, while introducing zero overhead during inference\.

## 5Experimental Results

We conduct comprehensive experiments to evaluate the effectiveness of the proposed SAGE framework on image classification benchmarks\. SAGE is evaluated using the Spikformer backbone and compared with both the original Spikformer baseline and recent state\-of\-the\-art SNNs\. In addition, extensive ablation studies are performed to analyze the contribution of each component of the proposed adaptive surrogate\-gradient strategy\.

### 5\.1Experimental Setup

#### 5\.1\.1Datasets

Experiments are conducted on three widely used image classification benchmarks: CIFAR\-10, CIFAR\-100[18](https://arxiv.org/html/2608.13702#bib.bib49), and ImageNet\-200[8](https://arxiv.org/html/2608.13702#bib.bib50)\. CIFAR\-10 and CIFAR\-100 each contain 60,000 RGB images of resolution32×3232\\times 32, divided into 50,000 training and 10,000 testing samples\. CIFAR\-10 consists of 10 object categories, whereas CIFAR\-100 contains 100 fine\-grained classes, providing a more challenging classification benchmark due to increased inter\-class similarity\. ImageNet\-200 is a commonly used subset of the ImageNet benchmark containing 200 object categories with approximately 128k training images and 10k validation images, offering substantially greater visual diversity while maintaining manageable computational cost\.

#### 5\.1\.2Implementation Details

SAGE is implemented on top of the official Spikformer[52](https://arxiv.org/html/2608.13702#bib.bib10)framework without modifying the network architecture or inference pipeline\. Unless otherwise specified, experiments employ the Spikformer\-4\-384 backbone comprising four transformer blocks, an embedding dimension of 384, 12 attention heads, a patch size of 4, an MLP expansion ratio of 4, and four simulation time steps\. Models are trained from scratch following the official Spikformer training protocol, including RandAugment, MixUp, Random Erasing, label smoothing, cosine learning\-rate scheduling with warmup, and the AdamW optimizer\. All experiments are implemented in PyTorch using the SpikingJelly framework and are conducted on NVIDIA Tesla V100 GPUs\.

During training, SAGE computes a block\-level uncertainty signal from the normalized attention entropy obtained from temperature\-scaled attention maps \(Tent=0\.25T\_\{\\mathrm\{ent\}\}=0\.25\)\. The entropy dispersion across attention heads is smoothed using an exponential moving average, normalized using running statistics, and mapped to an adaptive surrogate\-gradient slope through the controller described in Section[4\.3](https://arxiv.org/html/2608.13702#S4.SS3)\. The surrogate parameter is initialized with the standard SpikingJelly value \(α=4\\alpha=4\) during the warmup stage before adaptive modulation is activated\. Importantly, SAGE modifies only the surrogate\-gradient computation during training, while the forward computation graph, network parameters, and inference procedure remain identical to the original Spikformer\.

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/per_head_entropy_std_block_0_T0p25.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/per_head_entropy_std_block_1_T0p25.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/per_head_entropy_std_block_2_T0p25.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/per_head_entropy_std_block_3_T0p25.png)

Figure 3:Per\-head attention entropy variability across the four SSA blocks during training\. Each heatmap visualizes the standard deviation of the normalized attention entropy for each attention head over training epochs, computed from temperature\-scaled attention maps \(T=0\.25T=0\.25\)\. Rows correspond to attention heads, columns represent training epochs\.Table 2:Comparison on the ImageNet\-200 benchmark\.

### 5\.2Our Results

Table[1](https://arxiv.org/html/2608.13702#S4.T1)and Table[2](https://arxiv.org/html/2608.13702#S5.T2)summarizes the performance of the proposed SAGE framework on CIFAR\-10, CIFAR\-100, and ImageNet\-200, together with representative state\-of\-the\-art SNN and Spiking Transformer methods\. Following common practice, the reported results for Hybrid Training[31](https://arxiv.org/html/2608.13702#bib.bib45), DIET\-SNN[30](https://arxiv.org/html/2608.13702#bib.bib46), STBP\-tdBN[50](https://arxiv.org/html/2608.13702#bib.bib41), TET[9](https://arxiv.org/html/2608.13702#bib.bib47), RMP\-SNN[13](https://arxiv.org/html/2608.13702#bib.bib37), QCFS[4](https://arxiv.org/html/2608.13702#bib.bib48), and Spikformer[52](https://arxiv.org/html/2608.13702#bib.bib10)are reproduced from the original publications and the recent benchmark study in[51](https://arxiv.org/html/2608.13702#bib.bib39)\. No re\-implementation or retraining of these methods was performed\. Our experiments were conducted only on the Spikformer baseline and the proposed SAGE framework using the same training protocol and hyperparameter settings described in Section[5\.1\.2](https://arxiv.org/html/2608.13702#S5.SS1.SSS2)\.

To evaluate the effect of adaptive surrogate\-gradient modulation, we additionally compare three surrogate\-gradient configurations: Fixed, Learnable, and the proposed SAGE\. The Fixed configuration employs the conventional sigmoid surrogate with a constant slopeα=4\\alpha=4throughout training\. The Learnable configuration replaces the fixed slope with a single trainable surrogate\-gradient parameter optimized jointly with the network parameters\. In contrast, SAGE adaptively modulates the surrogate\-gradient slope during training using the proposed uncertainty\-driven entropy estimation while leaving the inference model unchanged\. To investigate robustness across different temporal resolutions, all three surrogate\-gradient configurations were evaluated under simulation time stepsT=\{1,2,4,8,12\}T=\\\{1,2,4,8,12\\\}\. The corresponding classification accuracies are presented in Fig\.[2](https://arxiv.org/html/2608.13702#S4.F2), where each subplot reports the Top\-1 accuracy obtained on CIFAR\-10, CIFAR\-100, and ImageNet\-200 for a fixed simulation time step\.

## 6Discussion

##### Performance Analysis

Table[1](https://arxiv.org/html/2608.13702#S4.T1)and Fig\.[2](https://arxiv.org/html/2608.13702#S4.F2)summarize the performance of the proposed SAGE framework under different surrogate\-gradient formulations and simulation time steps\. Across the evaluated datasets, SAGE consistently achieves competitive or superior classification accuracy compared with both the conventional fixed surrogate \(α=4\\alpha=4\) and the learnable surrogate formulation\. Under the standard Spikformer setting \(T=4T=4\), SAGE attains the highest Top\-1 accuracy on both CIFAR\-10 and CIFAR\-100 while maintaining competitive performance on ImageNet\-200\. Furthermore, the temporal analysis across simulation time stepsT=1,2,4,8,12T=\{1,2,4,8,12\}shows that SAGE consistently achieves competitive or superior performance at lower simulation lengths \(T≤4T\\leq 4\), with the largest improvement observed under the standard Spikformer setting ofT=4T=4\. As the number of simulation time steps increases beyondT=4T=4, the performance gap gradually narrows, and SAGE becomes comparable to the learnable surrogate formulation atT=8T=8andT=12T=12, indicating that the benefits of uncertainty\-guided surrogate adaptation are most pronounced in the low\-latency regime\. These results suggest that uncertainty\-guided surrogate adaptation provides a robust optimization strategy that generalizes across multiple datasets and temporal settings while preserving the original Spikformer architecture\.

##### Understanding SAGE

The design of SAGE was motivated by the observation that different Spikformer blocks exhibit distinct attention dynamics throughout training\. To identify a suitable uncertainty signal, we first analyzed the evolution of per\-head attention statistics across training epochs\. As illustrated in Fig\.[3](https://arxiv.org/html/2608.13702#S5.F3), the standard deviation of the normalized attention entropy consistently revealed clear block\-wise and head\-wise variations, indicating that different transformer blocks experience varying levels of uncertainty during optimization\. We also investigated alternative impurity measures, including Gini impurity[22](https://arxiv.org/html/2608.13702#bib.bib52), under multiple temperature scaling values \(Tent=\{0\.25,0\.5,1\.0,2\.0\}T\_\{\\mathrm\{ent\}\}=\\\{0\.25,0\.5,1\.0,2\.0\\\}\), with the corresponding results provided in the Appendix Fig\.[4](https://arxiv.org/html/2608.13702#A1.F4)\. While Gini impurity produced nearly saturated and highly uniform responses across blocks and temperatures, exhibiting limited temporal variation, the entropy\-based formulation provided richer and more discriminative dynamics throughout training\. Furthermore, using the standard deviation of per\-head entropy, rather than the mean entropy, captures the dispersion among attention heads, directly reflecting the degree of disagreement within each transformer block\. Since SAGE aims to adapt the surrogate gradient according to the consistency of attention behavior rather than its average confidence, entropy dispersion offers a more informative block\-level uncertainty estimate for guiding surrogate\-gradient adaptation\.

##### Practical Implications

Beyond the observed accuracy improvements, SAGE offers several practical advantages for spiking transformer optimization\. Since the proposed framework operates exclusively during training, the forward inference graph remains identical to the original Spikformer architecture\. Consequently, SAGE introduces no additional learnable parameters, preserves the original model size, and incurs no increase in inference latency or computational cost after training\. The adaptive controller consists only of lightweight statistical operations, including entropy computation, exponential moving average updates, and a simple nonlinear mapping, contributing an average overhead of approximately0\.030\.03ms per iteration during training\. As a result, the proposed method can be seamlessly integrated into existing surrogate\-gradient training pipelines as a plug\-and\-play optimization strategy without requiring architectural modifications or changes to deployment, making it readily applicable to a broad range of SNNs and transformer\-based SNN models\.

##### Limitations and Future Work

Although SAGE demonstrates consistent improvements on Spikformer across multiple image classification benchmarks, the current study focuses on attention\-based spiking transformer architectures where uncertainty can be naturally estimated from self\-attention distributions\. Extending the proposed framework to convolutional SNNs or other non\-transformer architectures will require alternative uncertainty measures, such as spike\-rate or membrane\-potential statistics\. Furthermore, the present work adapts only the surrogate\-gradient slope while keeping all neuron dynamics unchanged\. Future work will investigate jointly adapting additional neuron parameters, including firing thresholds and membrane time constants, as well as learning more general uncertainty\-aware controllers that can automatically optimize the surrogate\-gradient behavior across different datasets, architectures, and simulation time steps\.

## 7Conclusion

We have presented SAGE, an uncertainty\-aware surrogate\-gradient adaptation framework for Spiking ViTs\. By leveraging the dispersion of attention entropy across self\-attention heads, SAGE dynamically modulates the surrogate\-gradient slope during training while preserving the original network architecture, model parameters, and inference pipeline\. Our experiments on CIFAR\-10, CIFAR\-100, and ImageNet\-200 demonstrated that the proposed approach consistently improves or maintains competitive classification performance over fixed and learnable surrogate\-gradient formulations, particularly in low\-latency settings\. Furthermore, SAGE introduces only negligible training\-time overhead and no additional inference cost, making it a practical plug\-and\-play optimization strategy for spiking transformers\. These results highlight the potential of uncertainty\-guided surrogate optimization as an effective direction for improving the training of deep SNNs\.

## References

- Q\. Ai, Y\. Yang, M\. Cai, K\. Chen, Q\. Liu, and L\. MaA cross\-layer residual spiking neural network with adaptive threshold leaky integrate\-and\-fire neuron and learnable surrogate gradient\.Knowledge\-Based Systems319,pp\. 113575\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Bellecet al\.\(2018\)G\. Bellec, D\. Salaj, A\. Subramoney, R\. Legenstein, and W\. MaassLong short\-term memory and learning\-to\-learn in networks of spiking neurons\.Advances in neural information processing systems31\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p2.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Branchiniet al\.\(2023\)N\. Branchini, V\. Aglietti, N\. Dhir, and T\. DamoulasCausal entropy optimization\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 8586–8605\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Buet al\.\(2023\)T\. Bu, W\. Fang, J\. Ding, P\. Dai, Z\. Yu, and T\. HuangOptimal ann\-snn conversion for high\-accuracy and ultra\-low\-latency spiking neural networks\.arXiv preprint arXiv:2303\.04347\.Cited by:[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.8.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.13702#S5.T2.5.5.1)\.
- Cheet al\.\(2022\)K\. Che, L\. Leng, K\. Zhang, J\. Zhang, Q\. Meng, J\. Cheng, Q\. Guo, and J\. LiaoDifferentiable hierarchical and surrogate gradient search for spiking neural networks\.Advances in Neural Information Processing Systems35,pp\. 24975–24990\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Chenget al\.\(2026\)G\. Cheng, C\. Lyu, S\. Gao, W\. Zhang, and K\. ChenGroup entropy\-controlled policy optimization\.arXiv preprint arXiv:2607\.16850\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Dattaet al\.\(2025\)G\. Datta, Z\. Liu, A\. Li, and P\. A\. BeerelDynamic spikformer: low\-latency & energy\-efficient spiking neural networks with dynamic time steps for vision transformers\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Denget al\.\(2009\)J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-FeiImagenet: a large\-scale hierarchical image database\.In2009 IEEE conference on computer vision and pattern recognition,pp\. 248–255\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.13702#S5.SS1.SSS1.p1.1)\.
- Denget al\.\(2022\)S\. Deng, Y\. Li, S\. Zhang, and S\. GuTemporal efficient training of spiking neural network via gradient re\-weighting\.arXiv preprint arXiv:2202\.11946\.Cited by:[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.5.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.13702#S5.T2.5.4.1)\.
- Gawlikowskiet al\.\(2023\)J\. Gawlikowski, C\. R\. N\. Tassi, M\. Ali, J\. Lee, M\. Humt, J\. Feng, A\. Kruspe, R\. Triebel, P\. Jung, R\. Roscher,et al\.A survey of uncertainty in deep neural networks: j\. gawlikowski et al\.\.Artificial intelligence review56\(Suppl 1\),pp\. 1513–1589\.Cited by:[§2](https://arxiv.org/html/2608.13702#S2.p1.1),[§2](https://arxiv.org/html/2608.13702#S2.p2.1)\.
- Ghosh\-Dastidar and Adeli \(2009\)S\. Ghosh\-Dastidar and H\. AdeliSpiking neural networks\.International journal of neural systems19\(04\),pp\. 295–308\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2024\)Y\. Guo, Y\. Chen, Z\. Hao, W\. Peng, Z\. Jie, Y\. Zhang, X\. Liu, and Z\. MaTake a shortcut back: mitigating the gradient vanishing for training spiking neural networks\.Advances in Neural Information Processing Systems37,pp\. 24849–24867\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Hanet al\.\(2020\)B\. Han, G\. Srinivasan, and K\. RoyRmp\-snn: residual membrane potential neuron for enabling deeper high\-accuracy and low\-latency spiking neural network\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 13558–13567\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.7.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1)\.
- Horowitz \(2014\)M\. Horowitz1\.1 computing’s energy problem \(and what we can do about it\)\.In2014 IEEE international solid\-state circuits conference digest of technical papers \(ISSCC\),pp\. 10–14\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p1.1)\.
- Houet al\.\(2026\)K\. Hou, K\. Wu, and Y\. ZhouAdaptive and lightweight surrogate gradients: enhancing training efficiency of spiking neural networks\.Frontiers in Neuroscience20,pp\. 1795946\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Jianget al\.\(2025\)J\. Jiang, L\. Wang, R\. Jiang, J\. Fan, and R\. YanAdaptive gradient learning for spiking neural networks by exploiting membrane potential dynamics\.arXiv preprint arXiv:2505\.11863\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Jianget al\.\(2026\)J\. Jiang, W\. Xu, J\. Fan, and R\. YanDs\-atgo: dual\-stage synergistic learning via forward adaptive threshold and backward gradient optimization for spiking neural networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 1855–1863\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Krizhevskyet al\.\(2009\)A\. Krizhevsky G\. Hintonet al\.Learning multiple layers of features from tiny images\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.13702#S5.SS1.SSS1.p1.1)\.
- Kunduet al\.\(2021\)S\. Kundu, M\. Pedram, and P\. A\. BeerelHire\-snn: harnessing the inherent robustness of energy\-efficient deep spiking neural networks by training with crafted input noise\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 5209–5218\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p1.1)\.
- Laiet al\.\(2024\)L\. Lai, E\. Ohn\-Bar, S\. Arora, and J\. S\. K\. YiUncertainty\-guided never\-ending learning to drive\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15088–15098\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Leeet al\.\(2016\)J\. H\. Lee, T\. Delbrück, and M\. PfeifferTraining deep spiking neural networks using backpropagation\.Frontiers in neuroscience10,pp\. 508\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p2.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Loh \(2011\)W\. LohClassification and regression trees\.Wiley interdisciplinary reviews: data mining and knowledge discovery1\(1\),pp\. 14–23\.Cited by:[§6](https://arxiv.org/html/2608.13702#S6.SS0.SSS0.Px2.p1.1)\.
- Lvet al\.\(2026\)W\. Lv, Y\. Chen, X\. Fang, X\. Zhu, J\. Wen, G\. Zhou, and S\. ChanConfidence\-aware with prototype alignment for partial multi\-label learning\.Advances in Neural Information Processing Systems38,pp\. 170397–170416\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Maass \(1997\)W\. MaassNetworks of spiking neurons: the third generation of neural network models\.Neural Networks10\(9\),pp\. 1659–1671\.External Links:ISSN 0893\-6080,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S0893-6080%2897%2900011-7),[Link](https://www.sciencedirect.com/science/article/pii/S0893608097000117)Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p1.1)\.
- Moonet al\.\(2020\)J\. Moon, J\. Kim, Y\. Shin, and S\. HwangConfidence\-aware learning for deep neural networks\.Ininternational conference on machine learning,pp\. 7034–7044\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Nairet al\.\(2026\)K\. Nair, R\. Rizk, and K\. SantoshA spike\-gated residual unit for information flow control in transformers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) Workshops,pp\. 3469–3478\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Neftciet al\.\(2019\)E\. O\. Neftci, H\. Mostafa, and F\. ZenkeSurrogate gradient learning in spiking neural networks\.IEEE Signal Processing Magazine36\(6\),pp\. 51–63\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p2.1),[§2](https://arxiv.org/html/2608.13702#S2.p1.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Perez\-Nieves and Goodman \(2021\)N\. Perez\-Nieves and D\. GoodmanSparse spiking gradient descent\.Advances in Neural Information Processing Systems34,pp\. 11795–11808\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Qiao and Peng \(2021\)F\. Qiao and X\. PengUncertainty\-guided model generalization to unseen domains\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 6790–6800\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Rathi and Roy \(2020\)N\. Rathi and K\. RoyDiet\-snn: direct input encoding with leakage and threshold optimization in deep spiking neural networks\.arXiv preprint arXiv:2008\.03658\.Cited by:[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.3.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1)\.
- Rathiet al\.\(2020\)N\. Rathi, G\. Srinivasan, P\. Panda, and K\. RoyEnabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation\.arXiv preprint arXiv:2005\.01807\.Cited by:[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.2.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.13702#S5.T2.5.2.1)\.
- Royet al\.\(2019\)K\. Roy, A\. Jaiswal, and P\. PandaTowards spike\-based machine intelligence with neuromorphic computing\.Nature575\(7784\),pp\. 607–617\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p1.1)\.
- Shannon \(1948\)C\. E\. ShannonA mathematical theory of communication\.The Bell system technical journal27\(3\),pp\. 379–423\.Cited by:[§4\.2](https://arxiv.org/html/2608.13702#S4.SS2.p1.3)\.
- Shiet al\.\(2024\)X\. Shi, Z\. Hao, and Z\. YuSpikingresformer: bridging resnet and vision transformer in spiking neural networks\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 5610–5619\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Shrestha and Orchard \(2018\)S\. B\. Shrestha and G\. OrchardSlayer: spike layer error reassignment in time\.Advances in neural information processing systems31\.Cited by:[§2](https://arxiv.org/html/2608.13702#S2.p1.1)\.
- Stanojevicet al\.\(2024\)A\. Stanojevic, S\. Woźniak, G\. Bellec, G\. Cherubini, A\. Pantazi, and W\. GerstnerHigh\-performance deep spiking neural networks with 0\.3 spikes per neuron\.Nature Communications15\(1\),pp\. 6793\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Sunet al\.\(2026\)Q\. Sun, Z\. Cao, K\. Geng, Z\. Zhang, and B\. HouOptimization method for surrogate function in spiking neural networks based on membrane potential distribution\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 25718–25726\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Tavanaeiet al\.\(2019\)A\. Tavanaei, M\. Ghodrati, S\. R\. Kheradpisheh, T\. Masquelier, and A\. MaidaDeep learning in spiking neural networks\.Neural networks111,pp\. 47–63\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Van den Bergheet al\.\(2026\)K\. Van den Berghe, S\. Stroobants, V\. Janapa Reddi, and G\. De CroonAdaptive surrogate gradients for sequential reinforcement learning in spiking neural networks\.Advances in Neural Information Processing Systems38,pp\. 147904–147926\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2608.13702#S2.p1.1),[§2](https://arxiv.org/html/2608.13702#S2.p2.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, D\. Zhang, A\. Belatreche, Y\. Xiao, Y\. Liang, Y\. Shan, Q\. Sun, E\. Zhang, and M\. ZhangSpiking vision transformer with saccadic attention\.InThe 13th International Conference on Learning Representations,pp\. 72872–72893\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2023\)Z\. Wang, R\. Jiang, S\. Lian, R\. Yan, and H\. TangAdaptive smoothing gradient learning for spiking neural networks\.InInternational conference on machine learning,pp\. 35798–35816\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2018\)Y\. Wu, L\. Deng, G\. Li, J\. Zhu, and L\. ShiSpatio\-temporal backpropagation for training high\-performance spiking neural networks\.Frontiers in neuroscience12,pp\. 331\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p2.1),[§2](https://arxiv.org/html/2608.13702#S2.p1.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2025\)Q\. Xu, J\. Deng, J\. Shen, B\. Chen, H\. Tang, and G\. PanHybrid spiking vision transformer for object detection with event cameras\.InInternational Conference on Machine Learning,pp\. 69147–69159\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2023\)M\. Yao, G\. Zhao, H\. Zhang, Y\. Hu, L\. Deng, Y\. Tian, B\. Xu, and G\. LiAttention spiking neural networks\.IEEE transactions on pattern analysis and machine intelligence45\(8\),pp\. 9393–9410\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Yinet al\.\(2024\)N\. Yin, M\. Wang, Z\. Chen, G\. De Masi, H\. Xiong, and B\. GuDynamic spiking graph neural networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 16495–16503\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2024\)L\. Yu, H\. Chen, Z\. Wang, S\. Zhan, J\. Shao, Q\. Liu, and S\. XuSpikingvit: a multiscale spiking vision transformer model for event\-based object detection\.IEEE Transactions on Cognitive and Developmental Systems17\(1\),pp\. 130–146\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1)\.
- Zenget al\.\(2025\)D\. Zeng, Y\. Wang, H\. Cao, W\. Liu, Y\. Xiao, W\. Chen, M\. Zhang, G\. Wang, Y\. Yang,et al\.Leveraging asynchronous spiking neural networks for ultra efficient event\-based visual processing\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 1620–1628\.Cited by:[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1)\.
- Zenke and Ganguli \(2018\)F\. Zenke and S\. GanguliSuperspike: supervised learning in multilayer spiking neural networks\.Neural computation30\(6\),pp\. 1514–1541\.Cited by:[§2](https://arxiv.org/html/2608.13702#S2.p1.1)\.
- Zhenget al\.\(2021\)H\. Zheng, Y\. Wu, L\. Deng, Y\. Hu, and G\. LiGoing deeper with directly\-trained larger spiking neural networks\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 11062–11070\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p2.1),[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.4.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.13702#S5.T2.5.3.1)\.
- Zhouet al\.\(2026\)F\. Zhou, X\. Wei, Y\. Liu, and Q\. YuAdvancing direct training for spiking neural networks with circulate\-firing neurons and learnable gradients\.arXiv preprint arXiv:2605\.27412\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p2.1),[§1](https://arxiv.org/html/2608.13702#S1.p3.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px3.p1.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1)\.
- Zhouet al\.\(2022\)Z\. Zhou, Y\. Zhu, C\. He, Y\. Wang, S\. Yan, Y\. Tian, and L\. YuanSpikformer: when spiking neural network meets transformer\.arXiv preprint arXiv:2209\.15425\.Cited by:[§1](https://arxiv.org/html/2608.13702#S1.p1.1),[§1](https://arxiv.org/html/2608.13702#S1.p3.1),[§3](https://arxiv.org/html/2608.13702#S3.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.13702#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.13702#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2608.13702#S4.T1.5.1.6.1),[§5\.1\.2](https://arxiv.org/html/2608.13702#S5.SS1.SSS2.p1.1),[§5\.2](https://arxiv.org/html/2608.13702#S5.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.13702#S5.T2.5.6.1)\.

## Appendix AAdditional Analysis of Uncertainty Measures

To investigate suitable uncertainty measures for surrogate\-gradient adaptation, we compared normalized attention entropy with Gini impurity computed from the attention distributions\. Figure[4](https://arxiv.org/html/2608.13702#A1.F4)illustrates the evolution of Gini impurity across the four Spikformer blocks under different temperature scaling values \(Tent=\{0\.25,0\.5,1\.0,2\.0\}T\_\{\\mathrm\{ent\}\}=\\\{0\.25,0\.5,1\.0,2\.0\\\}\)\. Compared with the entropy\-based analysis presented in the main paper, Gini impurity exhibits substantially smaller temporal variation and remains nearly saturated throughout training, providing limited discrimination between transformer blocks\. Consequently, SAGE adopts the standard deviation of normalized attention entropy as the uncertainty signal for adaptive surrogate\-gradient modulation\.

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/gini_impurity_block_0.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/gini_impurity_block_1.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/gini_impurity_block_2.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/gini_impurity_block_3.png)

Figure 4:Evolution of Gini impurity across the four Spikformer transformer blocks during training under different attention\-temperature values \(Tent=0\.25T\_\{\\mathrm\{ent\}\}=0\.25,0\.50\.5,1\.01\.0, and2\.02\.0\)\. Across all blocks, Gini impurity remains highly saturated with only minor temporal fluctuations, indicating limited sensitivity to changes in attention uncertainty\.To determine an appropriate entropy temperature for SAGE, we performed a one\-factor\-at\-a\-time \(OFAT\) sensitivity analysis by evaluating the evolution of the standard deviation of normalized attention entropy under four temperature values \(Tent=\{0\.25,0\.5,1\.0,2\.0\}T\_\{\\mathrm\{ent\}\}=\\\{0\.25,0\.5,1\.0,2\.0\\\}\)\. Figure[5](https://arxiv.org/html/2608.13702#A1.F5)summarizes the results across the four Spikformer transformer blocks\. Lower entropy temperatures preserve substantially greater entropy dispersion, providing a richer uncertainty signal for the adaptive controller, whereas larger temperatures progressively smooth the attention distribution and reduce the available variability\. Consequently,Tent=0\.25T\_\{\\mathrm\{ent\}\}=0\.25was selected for all experiments\.

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/entropy_std_block_0.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/entropy_std_block_1.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/entropy_std_block_2.png)

![Refer to caption](https://arxiv.org/html/2608.13702v1/figures/entropy_std_block_3.png)

Figure 5:Sensitivity analysis of the entropy temperature\. Evolution of the standard deviation of normalized attention entropy across the four Spikformer blocks during training under different entropy temperatures \(Tent=0\.25T\_\{\\mathrm\{ent\}\}=0\.25,0\.50\.5,1\.01\.0, and2\.02\.0\)\. Based on this analysis,Tent=0\.25T\_\{\\mathrm\{ent\}\}=0\.25was adopted throughout all experiments\.Algorithm 1SAGE Training for Spikformer0:Training set

𝒟\\mathcal\{D\}, Spikformer model

fθf\_\{\\theta\}with

BBSSA blocks, baseline surrogate slope

α0=4\.0\\alpha\_\{0\}=4\.0
0:Entropy temperature

τe=0\.25\\tau\_\{e\}=0\.25, EMA factor

β=0\.95\\beta=0\.95, dead\-zone threshold

δ=0\.25\\delta=0\.25
0:Slope bounds

αmin=3\.0\\alpha\_\{\\min\}=3\.0,

αmax=5\.0\\alpha\_\{\\max\}=5\.0
1:Initialize one controller per SSA block:

𝒞1,…,𝒞B\\mathcal\{C\}\_\{1\},\\ldots,\\mathcal\{C\}\_\{B\}
2:Initialize EMA dispersion

mbm\_\{b\}, running mean

μb\\mu\_\{b\}, running variance

σb2\\sigma\_\{b\}^\{2\}, and slope

αb=α0\\alpha\_\{b\}=\\alpha\_\{0\}for each block

bb
3:foreach epoch

e=1,…,Ee=1,\\ldots,Edo

4:foreach mini\-batch

\(x,y\)∈𝒟\(x,y\)\\in\\mathcal\{D\}do

5:if

e=1e=1or training step is in warm\-upthen

6:Set

αb←α0\\alpha\_\{b\}\\leftarrow\\alpha\_\{0\}for all blocks

7:endif

8:Apply current block\-wise surrogate slopes

\{αb\}b=1B\\\{\\alpha\_\{b\}\\\}\_\{b=1\}^\{B\}to all LIF nodes inside each transformer block

9:Forward pass through Spikformer:

y^=fθ​\(x\)\\hat\{y\}=f\_\{\\theta\}\(x\)
10:foreach SSA block

b=1,…,Bb=1,\\ldots,Bdo

11:Extract detached raw attention scores:

Ab=\(Qb​Kb⊤\)⋅sA\_\{b\}=\(Q\_\{b\}K\_\{b\}^\{\\top\}\)\\cdot s
12:Compute temperature\-scaled attention probabilities:

Pb=softmax⁡\(Abτe\)P\_\{b\}=\\operatorname\{softmax\}\\left\(\\frac\{A\_\{b\}\}\{\\tau\_\{e\}\}\\right\)
13:Compute normalized attention entropy:

Hb=−1log⁡N∑j=1NPb,jlog\(Pb,j\+ϵ\)H\_\{b\}=\-\\frac\{1\}\{\\log N\}\\sum\_\{j=1\}^\{N\}P\_\{b,j\}\\log\(P\_\{b,j\}\+\\epsilon\)
14:Average entropy over time, batch, and token dimensions to obtain per\-head entropy:

hb=meanT,B,N⁡\(Hb\)h\_\{b\}=\\operatorname\{mean\}\_\{T,B,N\}\(H\_\{b\}\)
15:Compute raw entropy dispersion across heads:

db=std⁡\(hb\)d\_\{b\}=\\operatorname\{std\}\(h\_\{b\}\)
16:Update EMA dispersion:

mb←β​mb\+\(1−β\)​dbm\_\{b\}\\leftarrow\\beta m\_\{b\}\+\(1\-\\beta\)d\_\{b\}
17:Update running mean and variance of

mbm\_\{b\}
18:Compute temporal normalized uncertainty:

zb=mb−μbσb\+ϵz\_\{b\}=\\frac\{m\_\{b\}\-\\mu\_\{b\}\}\{\\sigma\_\{b\}\+\\epsilon\}
19:endfor

20:Center normalized uncertainty across blocks:

z~b=zb−1B​∑k=1Bzk\\tilde\{z\}\_\{b\}=z\_\{b\}\-\\frac\{1\}\{B\}\\sum\_\{k=1\}^\{B\}z\_\{k\}
21:foreach block

b=1,…,Bb=1,\\ldots,Bdo

22:if

\|z~b\|<δ\|\\tilde\{z\}\_\{b\}\|<\\deltathen

23:

αb←α0\\alpha\_\{b\}\\leftarrow\\alpha\_\{0\}
24:else

25:

αb←α0\+0\.5​tanh⁡\(z~b\)\\alpha\_\{b\}\\leftarrow\\alpha\_\{0\}\+0\.5\\tanh\(\\tilde\{z\}\_\{b\}\)
26:endif

27:Clamp:

αb←clip⁡\(αb,αmin,αmax\)\\alpha\_\{b\}\\leftarrow\\operatorname\{clip\}\(\\alpha\_\{b\},\\alpha\_\{\\min\},\\alpha\_\{\\max\}\)
28:endfor

29:Compute loss

ℒ⁡\(y^,y\)\\mathcal\{L\}\(\\hat\{y\},y\)using the official training criterion

30:Backpropagate using surrogate gradients parameterized by the active

αb\\alpha\_\{b\}
31:Update model parameters

θ\\thetawith the optimizer

32:Reset spiking neuron states

33:endfor

34:endfor