DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

# DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Source: [https://arxiv.org/html/2606.05435](https://arxiv.org/html/2606.05435)
###### Abstract

Differentially private stochastic gradient descent \(DP\-SGD\) has become the standard framework for privacy\-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation\. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction\. The shift and scaling parameters are selected adaptively based on the empirical mean and variance\. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself\. On the other hand, DP\-Adam is an algorithm that exploits Adam\-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping\. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum \(DP\-MacAdam\), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum\. We perform an analysis showing that DP\-MacAdam estimates the gradient variances in a bias\-free manner\. In addition, we empirically evaluate the privacy and accuracy of DP\-MacAdam, demonstrating that it achieves improved model utility compared to DP\-SGD, AdaClip, and DP\-Adam baselines, without requiring manual tuning of the clipping threshold\.

## IIntroduction

The proliferation of machine learning models trained on sensitive data has made privacy\-preserving optimization a central concern in modern deep learning\. Differential privacy \(DP\)\[[1](https://arxiv.org/html/2606.05435#bib.bib1)\]has emerged as the gold standard for formal privacy guarantees, providing rigorous, mathematically provable protections against the leakage of individual training examples\. In the context of deep learning, DP is most commonly realized through DP\-SGD\[[2](https://arxiv.org/html/2606.05435#bib.bib2)\], which clips per\-sample gradients to bound their sensitivity and injects calibrated Gaussian noise before each parameter update\. While DP\-SGD offers strong theoretical guarantees, its empirical performance often lags behind modern adaptive optimizers, motivating the development of differentially private variants of Adam\[[3](https://arxiv.org/html/2606.05435#bib.bib3),[4](https://arxiv.org/html/2606.05435#bib.bib4)\]\.

The Adam optimizer maintains exponential moving averages \(EMA\) of the first and second moments of stochastic gradients, yielding coordinate\-wise adaptive learning rates\. For a variety of ML tasks, including image and natural language processing, Adam consistently outperforms SGD and is now the de facto iterative optimization algorithm for training large models\. It is therefore natural to apply Adam in the DP setting by feeding privatized gradients directly into its update rule, yielding the algorithm DP\-Adam\[[5](https://arxiv.org/html/2606.05435#bib.bib5)\]\.

However, a crucial limitation of both DP\-SGD and its Adam\-based variants is the dependence on a fixed gradient clipping thresholdCC\. In practice, the choice ofCCcritically affects the privacy\-utility tradeoff: a threshold that is too small discards signal through excessive clipping, while one that is too large increases sensitivity and forces larger noise additions\. Pichapati et al\.\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\]address this limitation through AdaClip, a theoretically\-motivated adaptive clipping strategy that maintains coordinate\-wise exponential moving estimates of the gradient mean and variance, using them to center, scale, and clip gradients dynamically at each iteration\. By adapting the clipping geometry to the local gradient distribution, AdaClip provably reduces the expected noise added per iteration compared to isotropicℓ2\\ell\_\{2\}clipping, yielding improved model utility under the same privacy budget\.

The two approaches, DP\-Adam and AdaClip, address key yet complementary limitations of DP\-SGD\. DP\-Adam maintains running estimates of the gradient mean and second moment, and uses them to compute adaptive, coordinate\-wise parameter updates\. However, DP\-Adam still relies on standard clipping, forgoing the noise reduction benefits of adaptive clipping\. AdaClip, on the other hand, maintains coordinate\-wise estimates of the gradient mean and variance to perform adaptive clipping and reduce the additive noise, but discards these statistics after preprocessing and falls back to a plain SGD update, leaving the adaptive information unused\. The combination of adaptive clipping and adaptive momentum updates under differential privacy therefore remains unexplored\.

We propose DP\-MacAdam \(DifferentiallyPrivateMechanism withAdaptiveClipping andAdaptiveMomentum\), which combines both adaptive clipping and adaptive momentum\. Crucially, this combination comes at no additional privacy cost, since all moment estimates are derived entirely from the privatized gradients, and thus, satisfy\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP by the post\-processing property of DP\.

Main Contributions\.Our main contributions are:

- •We propose DP\-MacAdam, the first algorithm to combine AdaClip\-style coordinate\-wise adaptive clipping with Adam\-style adaptive momentum under differential privacy guarantees\.
- •We derive a novel bias correction factor that yields an unbiased estimate of the gradient variance from the EMA of noise\-corrected instantaneous variance estimates, accounting for both DP noise inflation and EMA initialization bias\.
- •We empirically demonstrate that DP\-MacAdam outperforms state\-of\-the\-art DP optimizers on MNIST and CIFAR\-10 across a range of privacy budgets, without requiring manual tuning of the clipping threshold or dimensionality reduction as a preprocessing step\.

Related Work\.Differentially private stochastic gradient descent was introduced by Abadi et al\.\[[2](https://arxiv.org/html/2606.05435#bib.bib2)\], establishing the canonical recipe of per\-sample gradient clipping followed by Gaussian noise addition\. Convergence properties of the Adam optimizer were established by Kingma and Ba\[[3](https://arxiv.org/html/2606.05435#bib.bib3)\]and later refined by Bock et al\.\[[4](https://arxiv.org/html/2606.05435#bib.bib4)\]\. DP\-Adam was first introduced in TensorFlow Privacy\[[7](https://arxiv.org/html/2606.05435#bib.bib7)\]as a straightforward extension of Adam to the DP setting\. Tang et al\.\[[5](https://arxiv.org/html/2606.05435#bib.bib5)\]identified the DP noise bias in Adam’s second moment estimator and proposed DP\-AdamBC as a correction\. Choi et al\.\[[8](https://arxiv.org/html/2606.05435#bib.bib8)\]extended this line of work by incorporating decoupled weight decay, yielding DP\-AdamW and DP\-AdamW\-BC with improved empirical performance across image, text, and graph classification tasks\. Adaptive clipping strategies for DP\-SGD were proposed by Pichapati et al\.\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\]through AdaClip, which uses coordinate\-wise gradient statistics to minimize the expected noise norm added per iteration\. Gilani et al\.\[[9](https://arxiv.org/html/2606.05435#bib.bib9)\]generalized this idea through GeoClip, which clips and perturbs gradients in a transformed basis, provably reducing added noise compared to isotropic clipping\.

## IIProblem Setup

We briefly review some relevant definitions\. Let𝒟=\{\(x\(i\),y\(i\)\)\}i=1n\\mathcal\{D\}=\\left\\\{\(x^\{\(i\)\},y^\{\(i\)\}\)\\right\\\}\_\{i=1\}^\{n\}denote a dataset ofnni\.i\.d\. samples, wherex\(i\)∈𝒳x^\{\(i\)\}\\in\\mathcal\{X\}andy\(i\)∈𝒴y^\{\(i\)\}\\in\\mathcal\{Y\}\. Given a parametric modelθ∈ℝd\\theta\\in\\mathbb\{R\}^\{d\}and a loss functionℓ:ℝd×𝒳×𝒴↦ℝ\\ell:\\mathbb\{R\}^\{d\}\\times\\mathcal\{X\}\\times\\mathcal\{Y\}\\mapsto\\mathbb\{R\}, the goal is to find the model parametersθ\\thetathat minimize the empirical riskℒ​\(θ\)=1n​∑i=1nℓ​\(θ;x\(i\),y\(i\)\)\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(\\theta;x^\{\(i\)\},y^\{\(i\)\}\)\. For notational simplicity, we omit the superscript\(i\)\(i\)when referring to a generic samplexxand its associated quantities\.

Two datasets𝒟\\mathcal\{D\}and𝒟~\\tilde\{\\mathcal\{D\}\}are consideredneighbors, denoted𝒟∼𝒟~\\mathcal\{D\}\\sim\\tilde\{\\mathcal\{D\}\}, if they differ by at most one entry\. Differential privacy \(DP\) is defined with respect to all such neighboring datasets as follows\.

###### Definition 1\(Differential Privacy\[[1](https://arxiv.org/html/2606.05435#bib.bib1)\]\)

A randomized algorithm, or mechanism𝒜:\(𝒳×𝒴\)n→𝒮\\mathcal\{A\}:\(\\mathcal\{X\}\\times\\mathcal\{Y\}\)^\{n\}\\to\\mathcal\{S\}is considered \(ϵ,δ\\epsilon,\\delta\)\-differentially private \(\(ϵ,δ\\epsilon,\\delta\)\-DP\) if, for every pair of neighboring datasets𝒟∼𝒟~∈\(𝒳×𝒴\)n\\mathcal\{D\}\\sim\\tilde\{\\mathcal\{D\}\}\\in\(\\mathcal\{X\}\\times\\mathcal\{Y\}\)^\{n\}, and for allS⊆𝒮S\\subseteq\\mathcal\{S\},

Pr⁡\{𝒜​\(𝒟\)∈S\}≤eϵ​Pr⁡\{𝒜​\(𝒟~\)∈S\}\+δ\.\\displaystyle\\Pr\\\{\\mathcal\{A\}\(\\mathcal\{D\}\)\\in S\\\}\\leq e^\{\\epsilon\}\\Pr\\\{\\mathcal\{A\}\(\\tilde\{\\mathcal\{D\}\}\)\\in S\\\}\+\\delta\.\(1\)

### II\-APrivate Optimization and Gradient Clipping

The empirical risk can be minimized using a variety of first\-order optimization methods, which differ in how they make use of the gradientsg=∇θℓ​\(θ;x,y\)g=\\nabla\_\{\\theta\}\\ell\(\\theta;x,y\)to update the model parameters\. A standard approach to enforce DP in gradient\-based optimization is to bound the sensitivity of each gradient update\. Given a per\-sample gradientgtg\_\{t\}at time steptt, theℓ2\\ell\_\{2\}\-sensitivity is controlled by clipping:

g¯t=gtmax⁡\(1,‖gt‖2/C\),\\displaystyle\\bar\{g\}\_\{t\}=\\frac\{g\_\{t\}\}\{\\max\\\!\\left\(1,\\\|g\_\{t\}\\\|\_\{2\}/C\\right\)\},\(2\)which ensures‖g¯t‖2≤C\\\|\\bar\{g\}\_\{t\}\\\|\_\{2\}\\leq Cfor a fixed clipping thresholdC\>0C\>0\. Gaussian noisezt∼𝒩​\(0,σ2​C2​Id\)z\_\{t\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}C^\{2\}I\_\{d\}\)is then added to the clipped gradient, yielding a privatized estimateg~t=g¯t\+zt\\tilde\{g\}\_\{t\}=\\bar\{g\}\_\{t\}\+z\_\{t\}that satisfies\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP by the Gaussian mechanism\[[1](https://arxiv.org/html/2606.05435#bib.bib1)\]\.

Using the privatized gradientg~t\\tilde\{g\}\_\{t\}, DP\-SGD\[[2](https://arxiv.org/html/2606.05435#bib.bib2)\]updates the model parameters along the negative gradient direction:

θt=θt−1−η​g~t,\\displaystyle\\theta\_\{t\}=\\theta\_\{t\-1\}\-\\eta\\tilde\{g\}\_\{t\},\(3\)whereη\\etais the learning rate\. DP\-Adam\[[7](https://arxiv.org/html/2606.05435#bib.bib7)\]extends the Adam optimizer\[[3](https://arxiv.org/html/2606.05435#bib.bib3),[4](https://arxiv.org/html/2606.05435#bib.bib4)\]to the differentially private setting by maintaining exponential moving averages of the privatized gradients and their second moments:

mt\\displaystyle m\_\{t\}=β1​mt−1\+\(1−β1\)​g~t,vt=β2​vt−1\+\(1−β2\)​g~t2,\\displaystyle=\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)\\tilde\{g\}\_\{t\},\\quad v\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\tilde\{g\}\_\{t\}^\{2\},\(4\)whereβ1,β2\\beta\_\{1\},\\beta\_\{2\}are exponential decay rates\. After computing bias\-corrected estimatesm^t=mt/\(1−β1t\)\\hat\{m\}\_\{t\}=m\_\{t\}/\(1\-\\beta\_\{1\}^\{t\}\)andv^t=vt/\(1−β2t\)\\hat\{v\}\_\{t\}=v\_\{t\}/\(1\-\\beta\_\{2\}^\{t\}\), DP\-Adam updates the model parameters using

θt=θt−1−η​m^tv^t\+γ,\\displaystyle\\theta\_\{t\}=\\theta\_\{t\-1\}\-\\eta\\frac\{\\hat\{m\}\_\{t\}\}\{\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\gamma\},\(5\)whereγ\\gammais a stability constant\.

### II\-BAdaptive Clipping

Both DP\-SGD and DP\-Adam rely on a fixed, isotropic clipping thresholdCC, applied uniformly across all gradient coordinates to bound theℓ2\\ell\_\{2\}\-sensitivity\. As discussed in the introduction, choosingCCto avoid over\- or under\-clipping is a challenge\. Pichapati et al\.\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\]address this through AdaClip, a coordinate\-wise adaptive clipping strategy which first centers and scales the gradients using running estimates of the gradient meanmt∈ℝdm\_\{t\}\\in\\mathbb\{R\}^\{d\}and scaling factorbt∈ℝdb\_\{t\}\\in\\mathbb\{R\}^\{d\}:

wt=gt−mtbt,w¯t=wtmax⁡\(1,‖wt‖2\),\\displaystyle w\_\{t\}=\\frac\{g\_\{t\}\-m\_\{t\}\}\{b\_\{t\}\},\\qquad\\bar\{w\}\_\{t\}=\\frac\{w\_\{t\}\}\{\\max\(1,\\\|w\_\{t\}\\\|\_\{2\}\)\},\(6\)ensuring unitℓ2\\ell\_\{2\}\-norm sensitivity after the transform\. Gaussian noise is added to the clipped transformed gradients, and the result is mapped back to the original gradient space:

g~t=bt⊙\(w¯t\+zt\)\+mt\.\\displaystyle\\tilde\{g\}\_\{t\}=b\_\{t\}\\odot\\left\(\\bar\{w\}\_\{t\}\+z\_\{t\}\\right\)\+m\_\{t\}\.\(7\)The privatized gradientg~t\\tilde\{g\}\_\{t\}is then used in the gradient descent step\. By centering and scaling gradients coordinate\-wise before clipping, AdaClip concentrates the noise budget on dimensions with high variance—where it matters most—and adds less noise to dimensions that carry little information\. This is shown to reduce the total noise added per iteration compared to fixedℓ2\\ell\_\{2\}clipping, with the advantage growing with the parameter dimensiondd\.

## IIIDifferentially Private Mechanism for Adaptive Clipping with Adaptive Momentum \(DP\-MacAdam\)

We present DP\-MacAdam in Algorithm[1](https://arxiv.org/html/2606.05435#alg1), which integrates the coordinate\-wise adaptive clipping strategy of AdaClip\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\]with Adam’s adaptive momentum updates\[[3](https://arxiv.org/html/2606.05435#bib.bib3)\]under differential privacy\. DP\-MacAdam has several key properties\. First, instead of bounding the noise\-corrected instantaneous variance estimateutu\_\{t\}as done in AdaClip, we feed it into an exponential moving averagests\_\{t\}with the same decay rateβ1\\beta\_\{1\}as the mean estimatemtm\_\{t\}\. Second, to ensure thatsts\_\{t\}an unbiased estimate ofg~t\\tilde\{g\}\_\{t\}, we must account for the bias introduced by the weighted averaging of the past estimates\. We derive this bias\-correction factorκt=2​\(β1−β1t\)/\(1\+β1\)\\kappa\_\{t\}=2\(\\beta\_\{1\}\-\\beta\_\{1\}^\{t\}\)/\(1\+\\beta\_\{1\}\)in Appendix[A](https://arxiv.org/html/2606.05435#A1)\. To ensure thats^t\\hat\{s\}\_\{t\}is both lower and upper bounded, we clamp it between constantsh1h\_\{1\}andh2h\_\{2\}\. Finally, the bias\-corrected estimates^t\\hat\{s\}\_\{t\}is used to update the adaptive scaling vectorbtb\_\{t\}following the formula used in AdaClip\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\]\. We leave the derivation of the optimal choice ofbtb\_\{t\}under our EMA\-based variance estimate for future work\.

Algorithm 1DP\-MacAdam1:Dataset

𝒟\\mathcal\{D\}, learning rate

η\\eta, noise multiplier

σ\\sigma, batch size

BB, hyperparameters

β1,β2,h1,h2\\beta\_\{1\},\\beta\_\{2\},h\_\{1\},h\_\{2\}, stability constant

γ\\gamma, number of iterations

TT
2:

θT\\theta\_\{T\}
3:Initialize:

θ0\\theta\_\{0\};

m0=𝟎,m^0=𝟎,v0=𝟎,s02=𝟎,b0=\(1/d\)⋅𝟏m\_\{0\}=\\mathbf\{0\},\\ \\hat\{m\}\_\{0\}=\\mathbf\{0\},\\ v\_\{0\}=\\mathbf\{0\},\\ s\_\{0\}^\{2\}=\\mathbf\{0\},\\ b\_\{0\}=\(1/d\)\\cdot\\mathbf\{1\}
4:for

t=1t=1to

TTdo

5:Sample mini\-batch

ℬt=\{\(xt\(i\),yt\(i\)\)\}i=1B\\mathcal\{B\}\_\{t\}=\\left\\\{\(x\_\{t\}^\{\(i\)\},y\_\{t\}^\{\(i\)\}\)\\right\\\}\_\{i=1\}^\{B\}
6:for

i=1i=1to

BBdo

7:

gt\(i\)←∇θℓ​\(θt−1;xt\(i\),yt\(i\)\)g\_\{t\}^\{\(i\)\}\\leftarrow\\nabla\_\{\\theta\}\\ell\\left\(\\theta\_\{t\-1\};x\_\{t\}^\{\(i\)\},y\_\{t\}^\{\(i\)\}\\right\)
8:

wt\(i\)←gt\(i\)−m^t−1bt−1w\_\{t\}^\{\(i\)\}\\leftarrow\\dfrac\{g\_\{t\}^\{\(i\)\}\-\\hat\{m\}\_\{t\-1\}\}\{b\_\{t\-1\}\}
9:

w¯t\(i\)←wt\(i\)max⁡\(1,‖wt\(i\)‖2\)\\bar\{w\}\_\{t\}^\{\(i\)\}\\leftarrow\\dfrac\{w\_\{t\}^\{\(i\)\}\}\{\\max\(1,\\\|w\_\{t\}^\{\(i\)\}\\\|\_\{2\}\)\}
10:endfor

11:

zt∼𝒩​\(0,σ2B2​Id\)z\_\{t\}\\sim\\mathcal\{N\}\\\!\\left\(0,\\dfrac\{\\sigma^\{2\}\}\{B^\{2\}\}I\_\{d\}\\right\)
12:

w~t←1B​∑i=1Bw¯t\(i\)\+zt\\tilde\{w\}\_\{t\}\\leftarrow\\dfrac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\bar\{w\}\_\{t\}^\{\(i\)\}\+z\_\{t\}
13:

g~t←bt−1⊙w~t\+m^t−1\\tilde\{g\}\_\{t\}\\leftarrow b\_\{t\-1\}\\odot\\tilde\{w\}\_\{t\}\+\\hat\{m\}\_\{t\-1\}
14:

mt←β1​mt−1\+\(1−β1\)​g~tm\_\{t\}\\leftarrow\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)\\tilde\{g\}\_\{t\}
15:

vt←β2​vt−1\+\(1−β2\)​g~t2v\_\{t\}\\leftarrow\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\tilde\{g\}\_\{t\}^\{2\}
16:

m^t←mt1−β1t\\hat\{m\}\_\{t\}\\leftarrow\\dfrac\{m\_\{t\}\}\{1\-\\beta\_\{1\}^\{t\}\},

v^t←vt1−β2t\\hat\{v\}\_\{t\}\\leftarrow\\dfrac\{v\_\{t\}\}\{1\-\\beta\_\{2\}^\{t\}\}
17:

θt←θt−1−η​m^tv^t\+γ\\theta\_\{t\}\\leftarrow\\theta\_\{t\-1\}\-\\eta\\dfrac\{\\hat\{m\}\_\{t\}\}\{\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\gamma\}
18:

ut←\(g~t−m^t\)⊙\(g~t−m^t\)u\_\{t\}\\leftarrow\(\\tilde\{g\}\_\{t\}\-\\hat\{m\}\_\{t\}\)\\odot\(\\tilde\{g\}\_\{t\}\-\\hat\{m\}\_\{t\}\)
19:

st←β1​st−1\+\(1−β1\)​uts\_\{t\}\\leftarrow\\beta\_\{1\}s\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)u\_\{t\}
20:

κt←2​\(β1−β1t\)1\+β1\\kappa\_\{t\}\\leftarrow\\dfrac\{2\(\\beta\_\{1\}\-\\beta\_\{1\}^\{t\}\)\}\{1\+\\beta\_\{1\}\}
21:

s^t←min⁡\(max⁡\(stκt−bt−12⋅σ2B2,h1\),h2\)\\hat\{s\}\_\{t\}\\leftarrow\\min\\\!\\left\(\\max\\\!\\left\(\\dfrac\{s\_\{t\}\}\{\\kappa\_\{t\}\}\-b\_\{t\-1\}^\{2\}\\cdot\\dfrac\{\\sigma^\{2\}\}\{B^\{2\}\},\\ h\_\{1\}\\right\),h\_\{2\}\\right\)
22:

bt←s^t1/4⋅\(∑j=1ds^t,j1/2\)1/2b\_\{t\}\\leftarrow\\hat\{s\}\_\{t\}^\{1/4\}\\cdot\\left\(\\sum\_\{j=1\}^\{d\}\\hat\{s\}\_\{t,j\}^\{1/2\}\\right\)^\{1/2\}
23:endfor

24:return

θT\\theta\_\{T\}

### III\-APrivacy Analysis

The privacy of DP\-MacAdam follows that of DP\-SGD and DP\-Adam\. Since the only interaction with the private dataset𝒟\\mathcal\{D\}occurs through the per\-sample gradients, privacy is guaranteed at the point of noise injection in step 10 of Algorithm[1](https://arxiv.org/html/2606.05435#alg1)\. All downstream computations—the moment estimatesmtm\_\{t\},vtv\_\{t\}, the variance estimatests\_\{t\}, and the adaptive scaling vectorbtb\_\{t\}—are functions solely of the privatized gradientg~t\\tilde\{g\}\_\{t\}and public hyperparameters\. By the post\-processing property of differential privacy\[[1](https://arxiv.org/html/2606.05435#bib.bib1)\], DP\-MacAdam therefore inherits the same\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP guarantee as DP\-SGD, for any privacy accountant used to track the cumulative privacy loss acrossTTiterations\. We use the Connect\-the\-Dots\[[10](https://arxiv.org/html/2606.05435#bib.bib10)\]privacy accountant in our experiments\. We state the privacy guarantee of Algorithm[1](https://arxiv.org/html/2606.05435#alg1)formally in Proposition 1\. The proof follows directly from the privacy analysis of DP\-SGD\[[2](https://arxiv.org/html/2606.05435#bib.bib2)\]\.

###### Proposition 1\(Privacy guarantee of DP\-MacAdam\)

For any privacy accountantCompose​\(T,θ1,…,T\)\\mathrm\{Compose\}\(T,\\theta\_\{1,\\ldots,T\}\)under whichDP​\-​SGD​\(θ,C,𝒟,σ,B\)\\mathrm\{DP\\text\{\-\}SGD\}\(\\theta,C,\\mathcal\{D\},\\sigma,B\)\[[2](https://arxiv.org/html/2606.05435#bib.bib2)\]satisfies\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP,DP​\-​MacAdam​\(θ,𝒟,σ,B\)\\mathrm\{DP\\text\{\-\}MacAdam\}\(\\theta,\\mathcal\{D\},\\sigma,B\)satisfies the same\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP guarantee under the same accountant\.

### III\-BNoise Scaling

In DP\-MacAdam, the adaptive parametersm^t\\hat\{m\}\_\{t\}andbtb\_\{t\}naturally set the clipping threshold, so there is no need for a separate clipping hyperparameterCC, as in DP\-SGD and DP\-ADAM\. Thus, in step 6 of Algorithm[1](https://arxiv.org/html/2606.05435#alg1), centering and scaling each entry of the gradient vectorgtg\_\{t\}ensures that theℓ2\\ell\_\{2\}\-norm ofw¯t\\bar\{w\}\_\{t\}is at most11\. Gaussian noise is then added to the average of the clipped transformed gradientsw¯t\\bar\{w\}\_\{t\}rather than to each individual gradient\. For a batch sizeBB, the average ofBBclipped gradients hasℓ2\\ell\_\{2\}\-sensitivity1/B1/B\(eachw¯t\(i\)\\bar\{w\}\_\{t\}^\{\(i\)\}has at most unit norm, and changing one sample affects the average by at most1/B1/B\), so the noise is scaled accordingly aszt∼𝒩​\(0,σ2B2​Id\)z\_\{t\}\\sim\\mathcal\{N\}\(0,\\frac\{\\sigma^\{2\}\}\{B^\{2\}\}I\_\{d\}\)\. This is equivalent to adding noise𝒩​\(0,σ2​Id\)\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)to each individual gradient and averaging, but is more efficient in practice\. After the map\-back in step 11, the effective noise variance isbt−12​σ2B2\\frac\{b\_\{t\-1\}^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}; this is why in step 19, this term is subtracted to calculates^t\\hat\{s\}\_\{t\}, so as to estimate the variance of the un\-noised gradient\.111Clipping introduces a non\-linear function to the gradient prior to noise, so even removing the noise does not give a completely unbiased estimate of the pure gradients\. The adaptive clipping will mitigate this limitation over the course of the training process\.

### III\-CBias\-Corrected Variance Estimation

In Algorithm[1](https://arxiv.org/html/2606.05435#alg1), steps 5–11 follow the AdaClip approach to compute the centered and clipped gradient; steps 12–15 follow the Adam approach of computing the moving average and second moment of the gradients, and then taking the descent step\. A key aspect of DP\-MacAdam is in combining these statistics to update the center and scale factors for the next batch\. This is captured in steps 16–19 where we form an instantaneous estimate of the gradient variance \(step 16\), compute its moving average \(step 17\), correct for the bias introduced by the moving average \(step 18\), bound the estimated variance \(step 19\), and use AdaClip’s update rule for center and scaling parameters for the next batch \(step 20\)\.

In Appendix[A](https://arxiv.org/html/2606.05435#A1), we derive the bias correction factorκt\\kappa\_\{t\}used in step 19 of Algorithm[1](https://arxiv.org/html/2606.05435#alg1)\. The goal is to obtain an unbiased estimates^t\\hat\{s\}\_\{t\}of the true gradient varianceσg2\\sigma\_\{g\}^\{2\}from the exponential moving average \(EMA\)sts\_\{t\}\. The challenge here arises from the fact thatsts\_\{t\}is a weighted average of past noise\-corrected variance estimatesuiu\_\{i\}, each of which is itself centered around a noisy meanm^t\\hat\{m\}\_\{t\}—so the bias correction must account for both the EMA initialization bias and the correlation structure introduced by the weighted averaging\.

It is worth recalling that the empirical estimate of the variance fromnnsamples \(with equal weights\) requires scaling by1/\(n−1\)1/\(n\-1\)rather than1/n1/nto ensure an unbiased estimate\. This same phenomenon must be accounted for here for the exponential weighted average\.

### III\-DExtension to DP\-MacAdamBC

A related line of work addresses a different but complementary bias in DP\-Adam\. Tang et al\.\[[5](https://arxiv.org/html/2606.05435#bib.bib5)\]observe that the addition of DP noise introduces a constant upward shiftΦ=\(σ​C/B\)2\\Phi=\(\\sigma C/B\)^\{2\}in Adam’s second moment estimatev^t\\hat\{v\}\_\{t\}, since the noise is independent of the gradient and its variance adds directly to the EMA of squared gradients\. Under typical DP parameters, this bias dominatesv^t\\hat\{v\}\_\{t\}, effectively reducing DP\-Adam to DP\-SGD with momentum and a specific learning rate schedule\. They propose DP\-AdamBC, which corrects for this by subtracting the known biasΦ\\Phifrom the second moment estimate in the parameter update:

θt=θt−1−η​m^t/max⁡\(v^t−Φ,γ′\)\.\\displaystyle\\theta\_\{t\}=\\theta\_\{t\-1\}\-\\eta\\,\\hat\{m\}\_\{t\}/\\sqrt\{\\max\(\\hat\{v\}\_\{t\}\-\\Phi,\\ \\gamma^\{\\prime\}\)\}\.\(8\)SinceΦ\\Phiis computable from public hyperparametersσ\\sigma,CC, andBB, this correction comes at no additional privacy cost\. The adaptive clipping strategy of DP\-MacAdam is orthogonal to this correction and could be combined with DP\-AdamBC by replacing the standard parameter update in Algorithm[1](https://arxiv.org/html/2606.05435#alg1)with the bias\-corrected update above, where in place of the constantΦ\\Phiwe use\(σ/B\)2\(\\sigma/B\)^\{2\}\. We call this variant of the algorithm DP\-MacAdamBC\.

## IVExperimental Results

We compare the performance of our proposed DP\-MacAdam with that of DP\-SGD, AdaClip, DP\-Adam, and DP\-AdamBC\. We focus on image classification tasks with MNIST\[[11](https://arxiv.org/html/2606.05435#bib.bib11)\]and CIFAR\-10\[[12](https://arxiv.org/html/2606.05435#bib.bib12)\]datasets\. For MNIST, we train a two\-layer fully connected neural network from scratch\. The network takes flattened28×2828\\times 28pixel inputs and passes them through a hidden layer of10001000units with ReLU activation, followed by a linear output layer of1010units corresponding to the digit classes\. The model has a total ofd=795,010d=795\{,\}010trainable parameters\. For CIFAR\-10\[[12](https://arxiv.org/html/2606.05435#bib.bib12)\], we use a 5\-layer CNN similar to that of\[[5](https://arxiv.org/html/2606.05435#bib.bib5)\], trained from scratch\. The model takes32×3232\\times 32RGB images as input and has a total ofd=582,346d=582\{,\}346trainable parameters\.

Throughout our experiments, we use learning rateη=0\.1\\eta=0\.1for DP\-SGD andη=0\.001\\eta=0\.001for the rest\. We use the standard Adam momentum coefficientsβ1=0\.9,β2=0\.999,\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.999,andγ=10−8\\gamma=10^\{\-8\}\. All models are trained for 5 epochs with a batch sizeB=256B=256for MNIST andB=512B=512for CIFAR\-10, respectively\. We adopt the same hyperparameter settings reported by\[[5](https://arxiv.org/html/2606.05435#bib.bib5)\]and\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\]where applicable\. The clipping norm for DP\-SGD, DP\-Adam and DP\-AdamBC is set toC=1\.0C=1\.0\. For various noise multiplierσ\\sigma, the overall privacy budgetϵ\\epsilonis computed using the Connect\-the\-Dots accountant\[[10](https://arxiv.org/html/2606.05435#bib.bib10)\], with the privacy parameterδ=10−5\\delta=10^\{\-5\}\. All results are reported as mean±\\pmstandard deviation over 5 random seeds\.

TABLE I:Test accuracy \(%\) on MNIST dataset across noise multipliers \(σ\\sigma\);h1=10−9h\_\{1\}=10^\{\-9\}andh2=10−6h\_\{2\}=10^\{\-6\}\.TABLE II:Test accuracy \(%\) on CIFAR\-10 dataset across noise multipliers \(σ\\sigma\);h1=5×10−5h\_\{1\}=5\\times 10^\{\-5\}andh2=1\.0h\_\{2\}=1\.0\.We observe that DP\-MacAdam outperforms the state\-of\-the\-art algorithms on both MNIST and CIFAR\-10 across most evaluated privacy budgets, with the exception of the highest noise setting on CIFAR\-10 \(σ=1\.5\\sigma=1\.5\)\. DP\-MacAdam\-BC, however, underperforms relative to DP\-MacAdam on MNIST, and performs poorly on CIFAR\-10\. Following\[[5](https://arxiv.org/html/2606.05435#bib.bib5)\], we hypothesize that the second moment bias correction yields larger gains on tasks where Adam and sign descent outperform SGD in the non\-private case\. It is worth noting that unlike\[[6](https://arxiv.org/html/2606.05435#bib.bib6)\], DP\-MacAdam \(i\) operates directly on the raw inputs without spending privacy budget on dimensionality reduction via PCA, and \(ii\) applies clamping only after the bias\-corrected variance estimates^t\\hat\{s\}\_\{t\}is computed rather than on the instantaneous estimateutu\_\{t\}, yielding a more stablebtb\_\{t\}\.

## VConclusion

We have proposed DP\-MacAdam, a differentially private optimization algorithm that combines coordinate\-wise adaptive clipping from AdaClip with Adam momentum updates\. By maintaining running estimates of the gradient mean and variance from the privatized gradients, DP\-MacAdam adapts the clipping threshold coordinate\-wise at each iteration while simultaneously using the same statistics to drive adaptive parameter updates\. We have derived a novel bias correction factorκt\\kappa\_\{t\}that accounts for both the DP noise inflation and the initialization bias introduced by the exponential moving average of the variance estimate\. Empirical results on real\-world datasets show that DP\-MacAdam is consistently better than DP\-Adam and DP\-AdamBC for typical privacy values\. Future work includes a formal convergence analysis of DP\-MacAdam, which can lead to determining the optimal choice of the scaling vectorbtb\_\{t\}as well as empirical evaluation on more complex learning tasks \(e\.g\., NLP\) over large datasets\.

## Appendix ABias Correction Factorκt\\kappa\_\{t\}

In this analysis, we useβ\\betain place ofβ1\\beta\_\{1\}for simplicity\. We also perform our analysis in an element\-by\-element basis, so we treat each vector as as a scalar\. The EMA of the noisy gradients is

mt=β​μt−1\+\(1−β\)​g~t=∑i=1t\(1−β\)​βt−i​g~i,\\displaystyle m\_\{t\}=\\beta\\mu\_\{t\-1\}\+\(1\-\\beta\)\\tilde\{g\}\_\{t\}=\\sum\_\{i=1\}^\{t\}\(1\-\\beta\)\\beta^\{t\-i\}\\tilde\{g\}\_\{i\},\(9\)assumingm0=𝟎m\_\{0\}=\\mathbf\{0\}\. The bias\-corrected mean estimate is thus:

m^t\\displaystyle\\hat\{m\}\_\{t\}=mt1−βt=∑i=1t\(1−β\)​βt−i1−βt​g~i\.\\displaystyle=\\dfrac\{m\_\{t\}\}\{1\-\\beta^\{t\}\}=\\sum\_\{i=1\}^\{t\}\\dfrac\{\(1\-\\beta\)\\beta^\{t\-i\}\}\{1\-\\beta^\{t\}\}\\tilde\{g\}\_\{i\}\.\(10\)Letci=\(1−β\)​βt−i1−βtc\_\{i\}=\\dfrac\{\(1\-\\beta\)\\beta^\{t\-i\}\}\{1\-\\beta^\{t\}\}\. Note that∑ici=1\\sum\_\{i\}c\_\{i\}=1\. Now, the instantaneous variance ofg~i\\tilde\{g\}\_\{i\}is

ut=\(g~t−m^t\)2\.\\displaystyle u\_\{t\}=\\left\(\\tilde\{g\}\_\{t\}\-\\hat\{m\}\_\{t\}\\right\)^\{2\}\.\(11\)Applying EMA, we have

st\\displaystyle s\_\{t\}=β​st−1\+\(1−β\)​ut=\(1−βt\)​∑i=1tci​ui\.\\displaystyle=\\beta s\_\{t\-1\}\+\(1\-\\beta\)u\_\{t\}=\(1\-\\beta^\{t\}\)\\sum\_\{i=1\}^\{t\}c\_\{i\}u\_\{i\}\.\(12\)Taking the expectation, we obtain

𝔼​\[st\]=\(1−βt\)​∑i=1tci​𝔼​\[ui\]\.\\displaystyle\\mathbb\{E\}\[s\_\{t\}\]=\(1\-\\beta^\{t\}\)\\sum\_\{i=1\}^\{t\}c\_\{i\}\\mathbb\{E\}\[u\_\{i\}\]\.\(13\)The expectation at a given stepttis

𝔼​\[ut\]\\displaystyle\\mathbb\{E\}\[u\_\{t\}\]=𝔼​\[\(g~t−m^t\)2\]\\displaystyle=\\mathbb\{E\}\\left\[\(\\tilde\{g\}\_\{t\}\-\\hat\{m\}\_\{t\}\)^\{2\}\\right\]\(14\)=𝔼​\[\(g~t​∑i=1tci−∑i=1tci​g~i\)2\]\\displaystyle=\\mathbb\{E\}\\left\[\\left\(\\tilde\{g\}\_\{t\}\\sum\_\{i=1\}^\{t\}c\_\{i\}\-\\sum\_\{i=1\}^\{t\}c\_\{i\}\\tilde\{g\}\_\{i\}\\right\)^\{2\}\\right\]\(15\)=∑i=1t∑j=1tci​cj​𝔼​\[\(g~t−g~i\)​\(g~t−g~j\)\]\.\\displaystyle=\\sum\_\{i=1\}^\{t\}\\sum\_\{j=1\}^\{t\}c\_\{i\}c\_\{j\}\\,\\mathbb\{E\}\\left\[\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{i\}\)\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{j\}\)\\right\]\.\(16\)Letμg\\mu\_\{g\}andσg2\\sigma\_\{g\}^\{2\}be the true variance ofgtg\_\{t\}\. Sinceg~t\\tilde\{g\}\_\{t\}has noise added, its mean and variance areμg\\mu\_\{g\}andσg2\+bt−12​σ2B2\\sigma\_\{g\}^\{2\}\+\\frac\{b\_\{t\-1\}^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\. We assume that the clipping parameterbtb\_\{t\}changes slowly enough that it is a constant, sobt=bb\_\{t\}=b\. We also assume that across batches the gradientsgtg\_\{t\}are i\.i\.d\. Now, we consider different cases:

- •i=j=ti=j=t, ori≠j,i=ti\\neq j,i=t, ori≠j,j=ti\\neq j,j=t: 𝔼​\[\(g~t−g~i\)​\(g~t−g~j\)\]=0\.\\displaystyle\\mathbb\{E\}\\left\[\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{i\}\)\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{j\}\)\\right\]=0\.\(17\)
- •i=j,i≠t,j≠ti=j,\\ i\\neq t,\\ j\\neq t: 𝔼​\[\(g~t−g~i\)2\]\\displaystyle\\mathbb\{E\}\\left\[\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{i\}\)^\{2\}\\right\]=𝔼​\[g~t2\]−2​𝔼​\[g~t​g~i\]\+𝔼​\[g~i2\]\\displaystyle=\\mathbb\{E\}\[\\tilde\{g\}\_\{t\}^\{2\}\]\-2\\mathbb\{E\}\[\\tilde\{g\}\_\{t\}\\tilde\{g\}\_\{i\}\]\+\\mathbb\{E\}\[\\tilde\{g\}\_\{i\}^\{2\}\]=σg2\+m^t2−2​m^t2\+σg2\+m^t2\\displaystyle=\\sigma\_\{g\}^\{2\}\+\\hat\{m\}\_\{t\}^\{2\}\-2\\hat\{m\}\_\{t\}^\{2\}\+\\sigma\_\{g\}^\{2\}\+\\hat\{m\}\_\{t\}^\{2\}=2​\(σg2\+b2​σ2B2\)\.\\displaystyle=2\\left\(\\sigma\_\{g\}^\{2\}\+\\frac\{b^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\\right\)\.\(18\)
- •i≠j≠ti\\neq j\\neq t: 𝔼​\[\(g~t−g~i\)​\(g~t−g~j\)\]\\displaystyle\\hskip\-18\.06749pt\\mathbb\{E\}\\left\[\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{i\}\)\(\\tilde\{g\}\_\{t\}\-\\tilde\{g\}\_\{j\}\)\\right\]=𝔼​\[g~t2\]−𝔼​\[g~t​g~j\]−𝔼​\[g~i​g~t\]\+𝔼​\[g~i​g~j\]\\displaystyle=\\mathbb\{E\}\[\\tilde\{g\}\_\{t\}^\{2\}\]\-\\mathbb\{E\}\[\\tilde\{g\}\_\{t\}\\tilde\{g\}\_\{j\}\]\-\\mathbb\{E\}\[\\tilde\{g\}\_\{i\}\\tilde\{g\}\_\{t\}\]\+\\mathbb\{E\}\[\\tilde\{g\}\_\{i\}\\tilde\{g\}\_\{j\}\]=σg2\+b2​σ2B2\.\\displaystyle=\\sigma\_\{g\}^\{2\}\+\\frac\{b^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\.\(19\)

Combining everything, we have

𝔼​\[ut\]=\(2​∑i=1tci2\+∑i≠jci​cj\)​\(σg2\+b2​σ2B2\)\.\\displaystyle\\mathbb\{E\}\[u\_\{t\}\]=\\left\(2\\sum\_\{i=1\}^\{t\}c\_\{i\}^\{2\}\+\\sum\_\{i\\neq j\}c\_\{i\}c\_\{j\}\\right\)\\left\(\\sigma\_\{g\}^\{2\}\+\\frac\{b^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\\right\)\.\(20\)Now we take the weighted sum,

∑k=1tck​𝔼​\[uk\]\\displaystyle\\sum\_\{k=1\}^\{t\}c\_\{k\}\\mathbb\{E\}\[u\_\{k\}\]=∑k=1tck​\(2​∑i≠kci2\+∑i≠j≠kci​cj\)​\(σg2\+b2​σ2B2\)\\displaystyle=\\sum\_\{k=1\}^\{t\}c\_\{k\}\\left\(2\\sum\_\{i\\neq k\}c\_\{i\}^\{2\}\+\\sum\_\{i\\neq j\\neq k\}c\_\{i\}c\_\{j\}\\right\)\\left\(\\sigma\_\{g\}^\{2\}\+\\frac\{b^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\\right\)\(21\)=A​\(σg2\+b2​σ2B2\),\\displaystyle=A\\left\(\\sigma\_\{g\}^\{2\}\+\\frac\{b^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\\right\),\(22\)where

A\\displaystyle A=∑k=1tck​\(2​∑i≠kci2\+∑i≠j≠kci​cj\)\\displaystyle=\\sum\_\{k=1\}^\{t\}c\_\{k\}\\left\(2\\sum\_\{i\\neq k\}c\_\{i\}^\{2\}\+\\sum\_\{i\\neq j\\neq k\}c\_\{i\}c\_\{j\}\\right\)\(23\)=∑k=1tck​\(2​∑i≠kci2\+\(∑i≠kci\)2−∑i≠kci2\)\\displaystyle=\\sum\_\{k=1\}^\{t\}c\_\{k\}\\left\(2\\sum\_\{i\\neq k\}c\_\{i\}^\{2\}\+\\left\(\\sum\_\{i\\neq k\}c\_\{i\}\\right\)^\{2\}\-\\sum\_\{i\\neq k\}c\_\{i\}^\{2\}\\right\)\(24\)=∑k=1tck​\(∑i≠kci2\+\(1−ck\)2\)\\displaystyle=\\sum\_\{k=1\}^\{t\}c\_\{k\}\\left\(\\sum\_\{i\\neq k\}c\_\{i\}^\{2\}\+\(1\-c\_\{k\}\)^\{2\}\\right\)\(25\)=∑k=1tck​\(∑i=1kci2−ck2\+1−2​ck\+ck2\)\\displaystyle=\\sum\_\{k=1\}^\{t\}c\_\{k\}\\left\(\\sum\_\{i=1\}^\{k\}c\_\{i\}^\{2\}\-c\_\{k\}^\{2\}\+1\-2c\_\{k\}\+c\_\{k\}^\{2\}\\right\)\(26\)=∑k=1tck​∑i=1kci2\+∑k=1tck−2​∑k=1tck2\\displaystyle=\\sum\_\{k=1\}^\{t\}c\_\{k\}\\sum\_\{i=1\}^\{k\}c\_\{i\}^\{2\}\+\\sum\_\{k=1\}^\{t\}c\_\{k\}\-2\\sum\_\{k=1\}^\{t\}c\_\{k\}^\{2\}\(27\)=1−∑k=1tck2\\displaystyle=1\-\\sum\_\{k=1\}^\{t\}c\_\{k\}^\{2\}\(28\)Therefore,

𝔼​\[st\]\\displaystyle\\mathbb\{E\}\[s\_\{t\}\]=\(1−βt\)​∑k=1tck​𝔼​\[uk\]\\displaystyle=\(1\-\\beta^\{t\}\)\\sum\_\{k=1\}^\{t\}c\_\{k\}\\,\\mathbb\{E\}\[u\_\{k\}\]=\(1−βt\)​\(1−∑k=1tck2\)​\(σg2\+b2​σ2B2\)\.\\displaystyle=\(1\-\\beta^\{t\}\)\\left\(1\-\\sum\_\{k=1\}^\{t\}c\_\{k\}^\{2\}\\right\)\\left\(\\sigma\_\{g\}^\{2\}\+\\frac\{b^\{2\}\\sigma^\{2\}\}\{B^\{2\}\}\\right\)\.\(29\)The bias correction factor is thus:

κt\\displaystyle\\kappa\_\{t\}=\(1−βt\)​\(1−∑k=1tck2\),\\displaystyle=\(1\-\\beta^\{t\}\)\\left\(1\-\\sum\_\{k=1\}^\{t\}c\_\{k\}^\{2\}\\right\),\(30\)which can be simplified toκt=2​\(β−βt\)1\+β\\kappa\_\{t\}=\\dfrac\{2\(\\beta\-\\beta^\{t\}\)\}\{1\+\\beta\}\. We can also see thatstκt−bt−12​σ2B2\\frac\{s\_\{t\}\}\{\\kappa\_\{t\}\}\-b\_\{t\-1\}^\{2\}\\frac\{\\sigma^\{2\}\}\{B^\{2\}\}is an un\-biased estimate ofσg2\\sigma\_\{g\}^\{2\}; this explains the exact form of step 19 in Algorithm[1](https://arxiv.org/html/2606.05435#alg1)prior to clamping\.

## Acknowledgment

This work is supported in part by NSF grants CIF\-2312666 and SCH\-2205080\.

## References

- \[1\]C\. Dwork, F\. McSherry, K\. Nissim, and A\. Smith, “Calibrating Noise to Sensitivity in Private Data Analysis,” in*Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4\-7, 2006\. Proceedings 3*\. Springer, 2006, pp\. 265–284\.
- \[2\]M\. Abadi, A\. Chu, I\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. Zhang, “Deep Learning with Differential Privacy,” in*Proceedings of the 2016 ACM SIGSAC conference on computer and communications security*, 2016, pp\. 308–318\.
- \[3\]D\. P\. Kingma and J\. Ba, “Adam: A Method for Stochastic Optimization,”*arXiv preprint arXiv:1412\.6980*, 2014\.
- \[4\]S\. Bock, J\. Goppold, and M\. Weiß, “An Improvement of the Convergence Proof of the ADAM\-Optimizer,”*arXiv preprint arXiv:1804\.10587*, 2018\.
- \[5\]Q\. Tang, F\. Shpilevskiy, and M\. Lécuyer, “DP\-AdamBC: Your DP\-Adam Is Actually DP\-SGD \(Unless You Apply Bias Correction\),” in*Proceedings of the AAAI Conference on Artificial Intelligence*, vol\. 38, no\. 14, 2024, pp\. 15 276–15 283\.
- \[6\]V\. Pichapati, A\. T\. Suresh, F\. X\. Yu, S\. J\. Reddi, and S\. Kumar, “AdaCliP: Adaptive Clipping for Private SGD,”*arXiv preprint arXiv:1908\.07643*, 2019\.
- \[7\]TensorFlow Privacy Authors, “TensorFlow Privacy,”https://github\.com/tensorflow/privacy/tree/master/tensorflow\_privacy/privacy/optimizers, 2019, accessed: 2026, dp\_optimizer\.py, line 386\.
- \[8\]J\. Chooi, K\. Cong, R\. Li, and L\. Sun, “DP\-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning,”*arXiv preprint arXiv:2511\.07843*, 2025\.
- \[9\]A\. Gilani, N\. Tasnim, L\. Sankar, and O\. Kosut, “GeoClip: Geometry\-Aware Clipping for Differentially Private SGD,”*Advances in Neural Information Processing Systems*, vol\. 38, pp\. 6233–6257, 2026\.
- \[10\]V\. Doroshenko, B\. Ghazi, P\. Kamath, R\. Kumar, and P\. Manurangsi, “Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions,” in*Privacy Enhancing Technologies Symposium \(PETS\)*, 2022\.
- \[11\]Y\. LeCun, L\. Bottou, Y\. Bengio, and P\. Haffner, “Gradient\-Based Learning Applied to Document Recognition,”*Proceedings of the IEEE*, vol\. 86, no\. 11, pp\. 2278–2324, 1998\.
- \[12\]A\. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” University of Toronto, Tech\. Rep\., 2009\.