Differentially Private Natural Gradient Descent

# Differentially Private Natural Gradient Descent
Source: [https://arxiv.org/html/2607.05866](https://arxiv.org/html/2607.05866)
Li Pan,Chen KaiInstitute of Information Engineering, Chinese Academy of SciencesBeijingChina[chenkai@iie\.ac\.cn](https://arxiv.org/html/2607.05866v1/mailto:[email protected]),Chang ShuaiInstitute of Information Engineering, Chinese Academy of SciencesBeijingChina[changshuai751x@iie\.ac\.cn](https://arxiv.org/html/2607.05866v1/mailto:[email protected]),Zhang Sheng ZhiDepartment of Computer Science, Metropolitan College, Boston UniversityBostonUnited States[shengzhi@bu\.edu](https://arxiv.org/html/2607.05866v1/mailto:[email protected]),Lv Pei ZhuoNanyang Technological UniversitySingaporeSingapore[lvpeizhuo@gmail\.com](https://arxiv.org/html/2607.05866v1/mailto:[email protected])andHe Jin WenInstitute of Information Engineering, Chinese Academy of SciencesBeijingChina[hejinwen@iie\.ac\.cn](https://arxiv.org/html/2607.05866v1/mailto:[email protected])

###### Abstract\.

Under a fixed privacy budget, the utility of differentially private \(DP\) training is ultimately determined by its optimization efficiency\. Standard first\-order DP optimizers such as DP\-SGD rely solely on local gradients and ignore the underlying loss curvature\. This geometric blindness causes severe zigzagging in ill\-conditioned landscapes, squandering precious privacy budgets on inefficient iterations\. Practitioners are thus trapped in a bind: either stop training prematurely or inject massive per\-step noise, both of which critically compromise final model utility\. Natural Gradient Descent \(NGD\) resolves this by preconditioning gradients with curvature, aligning updates with the loss geometry and extracting more efficient signal from every noisy step, offering a principled pathway to break the privacy\-utility bottleneck\.

Despite its theoretical appeal, directly integrating NGD with DP introduces fundamental challenges: curvature estimation itself consumes prohibitive privacy budgets, isotropic DP operations conflict with the anisotropic scaling of NGD, and the inverse curvature catastrophically amplify parameter updates in flat directions, causing training instability\. We propose DP\-NGD, a practical framework that systematically addresses these obstacles by decoupling curvature estimation from private data, reconciling isotropic DP constraints with anisotropic second\-order optimization via a whitened\-space mechanism, and dynamically clamping the curvature to stabilize training\. Extensive experiments on standard benchmarks demonstrate that DP\-NGD achieves state\-of\-the\-art accuracy, breaking through the utility ceilings of first\-order baselines while delivering up to a10×10\\timesconvergence speedup under the same privacy budget\.

PVLDB Reference Format: PVLDB, 14\(1\): XXX\-XXX, 2026\. [doi:XX\.XX/XXX\.XX](https://doi.org/XX.XX/XXX.XX)††This work is licensed under the Creative Commons BY\-NC\-ND 4\.0 International License\. Visit[https://creativecommons\.org/licenses/by\-nc\-nd/4\.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/)to view a copy of this license\. For any use beyond those covered by this license, obtain permission by emailing[info@vldb\.org](https://arxiv.org/html/2607.05866v1/mailto:[email protected])\. Copyright is held by the owner/author\(s\)\. Publication rights licensed to the VLDB Endowment\. Proceedings of the VLDB Endowment, Vol\. 14, No\. 1 ISSN 2150\-8097\. [doi:XX\.XX/XXX\.XX](https://doi.org/XX.XX/XXX.XX)

PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at[https://github\.com/XXX](https://github.com/XXX)\.

## 1\.Introduction

Differential Privacy \(DP\)\(Dwork,[2006](https://arxiv.org/html/2607.05866#bib.bib15); Dwork and Roth,[2014](https://arxiv.org/html/2607.05866#bib.bib16)\)has emerged as the gold standard for privacy\-preserving machine learning\. Under a fixed privacy budget, the final utility of DP training is bottlenecked by its optimization efficiency\. Standard first\-order DP optimizers, such as DP\-SGD\(Abadi et al\.,[2016](https://arxiv.org/html/2607.05866#bib.bib2); Bu et al\.,[2022](https://arxiv.org/html/2607.05866#bib.bib9)\)and its adaptive variants, suffer from inherent “geometric blindness”: they rely solely on local gradients and ignore the loss curvature\. This leads to the severe zigzagging and slow convergence in highly ill\-conditioned loss landscapes typical of deep networks, as illustrated in Fig\.[1\(a\)](https://arxiv.org/html/2607.05866#S1.F1.sf1)\. In non\-private training, slow convergence is often harmless, since one can simply run more iterations\. However, in DP training, every iteration strictly consumes the privacy budget\. This inefficiency squanders the precious budget on uninformative steps and traps practitioners in a bind: either stop training prematurely or inject massive per\-step noise, both of which critically compromise final model utility\.

Natural Gradient Descent \(NGD\)\(Martens,[2020](https://arxiv.org/html/2607.05866#bib.bib30); Kunstner et al\.,[2019](https://arxiv.org/html/2607.05866#bib.bib26)\), a classic second\-order optimization method, fundamentally resolves this geometric blindness\. As depicted in Fig\.[1\(b\)](https://arxiv.org/html/2607.05866#S1.F1.sf2), by preconditioning the gradient with a curvature matrixFF\(e\.g\., the Fisher Information Matrix\), NGD corrects the update direction to point more directly toward the local optimum, significantly accelerating convergence across ill\-conditioned landscapes\. Crucially, this acceleration directly translates to higher utility under the same privacy guarantee\. Under standard DP accounting mechanisms \(e\.g\., Rényi DP\(Mironov,[2017](https://arxiv.org/html/2607.05866#bib.bib33); Mironov et al\.,[2019](https://arxiv.org/html/2607.05866#bib.bib34)\)or Moments Accountant\(Wang et al\.,[2018](https://arxiv.org/html/2607.05866#bib.bib41); Feng et al\.,[2024](https://arxiv.org/html/2607.05866#bib.bib17)\)\), the required per\-step noise multiplierσ\\sigmascales asσ∝T\\sigma\\propto\\sqrt\{T\}for a fixed total privacy budget\. By drastically reducing the required number of iterationsTT, NGD enables substantially lower per\-step noise injection, thereby elevating the signal\-to\-noise ratio \(SNR\) throughout training\. This provides a mathematically principled pathway to break the longstanding privacy\-utility bottleneck\.

![Refer to caption](https://arxiv.org/html/2607.05866v1/sgd.png)\(a\)SGD \(geometry\-blind\)
![Refer to caption](https://arxiv.org/html/2607.05866v1/ngd.png)\(b\)NGD \(curvature\-aware\)

Figure 1\.Why optimization efficiency dictates differential privacy utility\. Here,θt\\theta\_\{t\}represents the current model parameters andθ∗\\theta^\{\*\}is the local optimum\.\(a\) SGD \(geometry\-blind\)\.The gradientggsimply follows the orthogonal direction of the loss contours rather than pointing toward the local optimum\. This blindness induces severe zigzagging and slow convergence, rapidly depleting the fixed privacy budget\.\(b\) NGD \(curvature\-aware\)\.Second\-order methods precondition the gradient with the inverse curvatureF−1F^\{\-1\}\. The resulting natural gradientF−1​gF^\{\-1\}gpoints more directly towardθ∗\\theta^\{\*\}, significantly accelerating convergence and drastically reducing inefficient steps\. Because fewer iterations allow the fixed budget to be concentrated, it significantly lowers the per\-step noise \(σ∝T\\sigma\\propto\\sqrt\{T\}\) and raises the effective signal\-to\-noise ratio, ultimately unlocking superior model utility\.Despite its theoretical appeal, integrating NGD with DP is hindered by three fundamental challenges\. First, standard NGD requires accurate curvature estimation from private data, which would consume a prohibitive fraction of the privacy budget in DP settings\. Diverting the budgetϵ\\epsilonto this auxiliary computation starves the primary optimization process, largely negating any theoretical SNR gain\. Second, differential privacy mandates isotropic clipping and noise injection to uniformly bound sensitivity, whereas NGD relies on anisotropic, curvature\-aware scaling to achieve acceleration\. Directly imposing isotropic DP constraints on natural gradients corrupts the geometric properties essential for second\-order optimization\. Third, in flat directions of the loss landscape where the eigenvalues ofFFapproach zero \(λi→0\\lambda\_\{i\}\\to 0\), the inverse curvatureF−1F^\{\-1\}magnifies the noisy gradients by1/λi→∞1/\\lambda\_\{i\}\\to\\infty, triggering training instability and catastrophic divergence\.

To systematically resolve these bottlenecks, we propose DP\-NGD, a practical and robust framework\. First, we eliminate the privacy overhead of curvature estimation by computingFFsolely on a small public auxiliary dataset to extract structural priors, thereby preserving the entire privacy budget for gradient updates\. Second, we resolve the geometric incompatibility via anF−1/2F^\{\-1/2\}\-whitened space update mechanism\. We prove that performing isotropic DP operations in this whitened space mathematically translates to applying anisotropic, curvature\-aware updates in the original parameter space\. Crucially, this theoretical alignment reveals a profound connection—it unifies the DP sensitivity constraint with the KL trust region of NGD\. Finally, to prevent parameter explosion in flat directions, we derive a dynamic clamping floor from the expected Euclidean step size of DP\-SGD\. By clamping the curvature eigenvalues consistently above this threshold, we effectively suppress flat\-direction parameter explosion while preserving second\-order acceleration in steep directions\.

Contributions\.Our main contributions are summarized as follows:

∙\\bulletPrivacy\-Free Curvature Acceleration\.Our key insight is that DP noise disproportionately corrupts fine\-grained gradient components, making DP training inherently tolerant of curvature estimation errors in these directions\. Consequently, coarse\-grained structural priors suffice for effective second\-order acceleration\. We realize this through a decoupled DP\-NGD framework that estimates the curvature entirely on a small public auxiliary dataset, incurring zero additional privacy cost over standard DP\-SGD\.

∙\\bulletKL\-DP Duality\.We prove that applying isotropic DP operations within anF−1/2F^\{\-1/2\}\-whitened space mathematically translates to curvature\-aligned anisotropic updates in the original parameter space\. This establishes a rigorous KL\-DP Duality that unifies the DP sensitivity constraint with the KL trust region of NGD\. To ensure stable training, we further introduce a dynamic clamping mechanism that effectively prevents parameter explosion in flat directions\.

∙\\bulletSuperior Efficiency and Utility\.Across multiple benchmarks, DP\-NGD consistently achieves state\-of\-the\-art accuracy while dramatically accelerating convergence\. For instance, it breaks through the utility ceilings of baselines with up to a10×10\\timesconvergence speedup \(e\.g\., reaching the peak accuracy of DP\-SGD using only42\.8%\\text\{42\.8\}\\%of the iterations on CIFAR\-10 underϵ=1\.0\\epsilon=\\text\{1\.0\}\)\. DP\-NGD offers a highly practical and robust second\-order accelerator for privacy\-preserving deep learning\.

## 2\.Preliminaries

### 2\.1\.Differential Privacy & DP\-SGD

Differential Privacy\.Differential Privacy \(DP\)\(Dong et al\.,[2022](https://arxiv.org/html/2607.05866#bib.bib14)\)provides a rigorous mathematical framework for privacy\-preserving computation\. A randomized algorithmℳ\\mathcal\{M\}satisfies\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP if, for any two adjacent datasetsDDandD′D^\{\\prime\}differing by at most one example, and for any subset of outputsS⊆Range​\(ℳ\)S\\subseteq\\text\{Range\}\(\\mathcal\{M\}\), it holds that:

\(1\)Pr⁡\[ℳ​\(D\)∈S\]≤eϵ​Pr⁡\[ℳ​\(D′\)∈S\]\+δ\.\\Pr\[\\mathcal\{M\}\(D\)\\in S\]\\leq e^\{\\epsilon\}\\Pr\[\\mathcal\{M\}\(D^\{\\prime\}\)\\in S\]\+\\delta\.
DP\-SGD & Isotropic Operations\.The standard approach to training deep networks under DP is DP\-SGD\. At each iteration, it computes the per\-sample gradientgig\_\{i\}and clips it to a maximumL2L\_\{2\}\-norm boundCC, i\.e\.,g¯i=gi⋅min⁡\(1,C‖gi‖2\)\\bar\{g\}\_\{i\}=g\_\{i\}\\cdot\\min\(1,\\frac\{C\}\{\\\|g\_\{i\}\\\|\_\{2\}\}\)\. Then, it aggregates the clipped gradients and injects Gaussian noise to ensure privacy:

\(2\)g~=1\|ℬ\|​\(∑i∈ℬg¯i\+𝒩​\(0,σ2​C2​𝐈\)\),\\tilde\{g\}=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\left\(\\sum\_\{i\\in\\mathcal\{B\}\}\\bar\{g\}\_\{i\}\+\\mathcal\{N\}\(0,\\sigma^\{2\}C^\{2\}\\mathbf\{I\}\)\\right\),whereℬ\\mathcal\{B\}is the mini\-batch andσ\\sigmais the noise multiplier\. Crucially, both theL2L\_\{2\}\-norm clipping boundary and the spherical Gaussian noise covariance \(σ2​C2​𝐈\\sigma^\{2\}C^\{2\}\\mathbf\{I\}\) are strictly isotropic across the entire parameter space\.

Privacy Accounting\.To rigorously track the privacy loss overTTiterations, modern implementations typically rely on advanced privacy accounting methods\(Wang et al\.,[2022](https://arxiv.org/html/2607.05866#bib.bib40)\), such as the Moments Accountant\(Wang et al\.,[2018](https://arxiv.org/html/2607.05866#bib.bib41)\)or Rényi Differential Privacy \(RDP\)\(Mironov,[2017](https://arxiv.org/html/2607.05866#bib.bib33); Mironov et al\.,[2019](https://arxiv.org/html/2607.05866#bib.bib34)\)\. Under standard composition theorems, for a fixed overall privacy budget\(ϵ,δ\)\(\\epsilon,\\delta\)and sampling rateq=\|ℬ\|/Nq=\|\\mathcal\{B\}\|/N, the required per\-step noise multiplierσ\\sigmaasymptotically scales as:

\(3\)σ∝q​T​log⁡\(1/δ\)ϵ\.\\sigma\\propto\\frac\{q\\sqrt\{T\\log\(1/\\delta\)\}\}\{\\epsilon\}\.This establishes a fundamental property: when other privacy parameters are fixed, the injected noise scales with the square root of the total iterations \(σ∝T\\sigma\\propto\\sqrt\{T\}\)\. Consequently, accelerating convergence to reduceTTprovides a mathematically principled pathway to decrease per\-step noise and improve model utility\.

### 2\.2\.Natural Gradient Descent & K\-FAC

KL Trust Region\.Natural Gradient Descent \(NGD\) approaches optimization from an information\-geometric perspective\. Instead of updating parameters in the Euclidean space, NGD minimizes the objectiveℒ\\mathcal\{L\}within a local KL\-divergence trust region\(Xu et al\.,[2024](https://arxiv.org/html/2607.05866#bib.bib43); Pajarinen et al\.,[2019](https://arxiv.org/html/2607.05866#bib.bib36)\)to maintain distributional stability\. The KL\-divergence between the predictive distributions of the current parametersθ\\thetaand the updated parametersθ\+Δ​θ\\theta\+\\Delta\\thetacan be approximated via its second\-order Taylor expansion:

\(4\)KL​\(Pθ∥Pθ\+Δ​θ\)≈12​Δ​θT​F​Δ​θ,\\text\{KL\}\(P\_\{\\theta\}\\\|P\_\{\\theta\+\\Delta\\theta\}\)\\approx\\frac\{1\}\{2\}\\Delta\\theta^\{T\}F\\Delta\\theta,whereF=𝔼x,y∼Pθ​\[∇θlog⁡Pθ​\(y\|x\)​∇θlog⁡Pθ​\(y\|x\)T\]F=\\mathbb\{E\}\_\{x,y\\sim P\_\{\\theta\}\}\[\\nabla\_\{\\theta\}\\log P\_\{\\theta\}\(y\|x\)\\nabla\_\{\\theta\}\\log P\_\{\\theta\}\(y\|x\)^\{T\}\]is the Fisher Information Matrix \(FIM\), which captures the local loss curvature\. Solving the trust\-region optimization yields the NGD update rule:

\(5\)Δ​θ=−η​F−1​∇θℒ,\\Delta\\theta=\-\\eta F^\{\-1\}\\nabla\_\{\\theta\}\\mathcal\{L\},whereη\\etais the learning rate andg≜∇θℒg\\triangleq\\nabla\_\{\\theta\}\\mathcal\{L\}denotes the first\-order gradient of the loss\. In this context,F−1​gF^\{\-1\}gis formally defined as the natural gradient\. Unlike standard first\-order methods, preconditioning the gradients with inverse curvatureF−1F^\{\-1\}enforces an anisotropic scaling: it aggressively accelerates progress along flat directions while damping updates in steep directions\.

K\-FAC Approximation\.In deep neural networks, computing and inverting the exact FIM\(Kudo and Tajima,[2022](https://arxiv.org/html/2607.05866#bib.bib25); Rodriguez et al\.,[2013](https://arxiv.org/html/2607.05866#bib.bib37)\)is computationally intractable due to the massive parameter dimensionality\. The Kronecker\-Factored Approximate Curvature \(K\-FAC\)\(Grosse and Martens,[2016](https://arxiv.org/html/2607.05866#bib.bib21); Dangel et al\.,[2024](https://arxiv.org/html/2607.05866#bib.bib12)\)efficiently approximatesFFby assuming independence between the activationsala\_\{l\}and the pre\-activation gradientsglg\_\{l\}of each layerll\. This decouples the layer\-wise FIM blockFlF\_\{l\}into a Kronecker product of two smaller matrices:Fl≈Al⊗GlF\_\{l\}\\approx A\_\{l\}\\otimes G\_\{l\}, whereAl=𝔼​\[al​alT\]A\_\{l\}=\\mathbb\{E\}\[a\_\{l\}a\_\{l\}^\{T\}\]andGl=𝔼​\[gl​glT\]G\_\{l\}=\\mathbb\{E\}\[g\_\{l\}g\_\{l\}^\{T\}\]\. This structural factorization compresses the curvature degrees of freedom from𝒪​\(d2\)\\mathcal\{O\}\(d^\{2\}\)to𝒪​\(din2\+dout2\)\\mathcal\{O\}\(d\_\{\\text\{in\}\}^\{2\}\+d\_\{\\text\{out\}\}^\{2\}\), wheredin,dout≪dd\_\{\\text\{in\}\},d\_\{\\text\{out\}\}\\ll d, drastically reducing the requisite sample complexity\. This provides a mathematical justification for why a limited public dataset suffices to extract reliable coarse\-grained geometric priors\.

## 3\.The DP\-NGD Framework

To address the three fundamental challenges that hinder the integration of NGD and DP, we propose DP\-NGD, a practical and robust framework built upon three mutually reinforcing components\. We introduce each component in the following sections, and finally unify them into a complete training algorithm\.

### 3\.1\.Privacy\-Free Coarse\-Grained Curvature Estimation

Standard natural gradient descent \(NGD\) estimates the curvature matrix directly from the training data\. In a DP setting, doing so would consume a substantial portion of the privacy budget merely for preconditioning, leaving little budget for the actual gradient updates that drive learning\. We avoid this prohibitive overhead through strict computational decoupling: curvature estimation is performed exclusively on a public auxiliary dataset \(𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}\), while gradient clipping and DP noise injection are confined entirely to the private training set \(𝒟priv\\mathcal\{D\}\_\{\\text\{priv\}\}\)\. Because𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}isa prioripublic and disjoint from𝒟priv\\mathcal\{D\}\_\{\\text\{priv\}\}—lying strictly outside the adjacency relation defining DP—this computation incurs zero privacy cost\. Consequently, the entire privacy budgetϵ\\epsilonis preserved for gradient updates\.

A natural question arises: why does an out\-of\-distribution \(OOD\) public dataset suffice for effective curvature estimation? We identify a fundamental synergy between DP and NGD, termedCoarse\-Grained Curvature Transferability\. The cornerstone of this synergy is that DP noise disproportionately corrupts fine\-grained gradient components\. Specifically, in the flat directions of the loss landscape, the true gradient signals are typically small and are therefore dominated by the injected DP noise\. Preconditioning these fine\-grained directions with inaccurate curvature merely maps random noise into random noise, incurring minimal utility loss\. Consequently, DP training is inherently tolerant of curvature estimation errors in these fine\-grained dimensions: it demands structural fidelity exclusively in the dominant, coarse\-grained eigenspaces to achieve effective second\-order acceleration\.

Fortunately, this requirement for coarse\-grained fidelity is naturally satisfied in our optimization paradigm\. First, a shared model architecture imposes strong inductive priors\(Mehmeti\-Göpel and Wand,[2025](https://arxiv.org/html/2607.05866#bib.bib31); Boix\-Adserà,[2025](https://arxiv.org/html/2607.05866#bib.bib8)\), yielding covariance matrices that are structurally homomorphic across datasets\. Second, natural images share universal low\-level statistics\(Guth and M’enard,[2024](https://arxiv.org/html/2607.05866#bib.bib22); Venkatesan et al\.,[2016](https://arxiv.org/html/2607.05866#bib.bib39)\)—such as edges, textures, and color smoothness—that dominate the top eigenspaces of the curvature matrix, allowing even OOD data to reliably capture the macroscopically steep directions\. Third, the K\-FAC approximation acts as a low\-pass filter: its expectation over samples effectively averages out fine\-grained fluctuations, isolating precisely the statistically stable, coarse\-grained structures we rely on\. Together, these properties ensure that the coarse\-grained curvature priors essential for acceleration can be effectively extracted even from OOD public auxiliary dataset\(Block et al\.,[2024](https://arxiv.org/html/2607.05866#bib.bib7); Lowy et al\.,[2023](https://arxiv.org/html/2607.05866#bib.bib28)\)\.

Specifically, we precompute the curvature matrixFFon𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}via the K\-FAC approximation\. For each layerll, the diagonal blockFlF\_\{l\}is constructed as the Kronecker product of the activation covarianceAlA\_\{l\}and the pre\-activation gradient covarianceGlG\_\{l\}:

\(6\)Fl=Al⊗Gl,Al=𝔼𝒟pub​\[al​al⊤\],Gl=𝔼𝒟pub​\[gl​gl⊤\]\.F\_\{l\}=A\_\{l\}\\otimes G\_\{l\},\\quad A\_\{l\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\text\{pub\}\}\}\[a\_\{l\}a\_\{l\}^\{\\top\}\],\\quad G\_\{l\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\text\{pub\}\}\}\[g\_\{l\}g\_\{l\}^\{\\top\}\]\.The complete curvature is then assembled as a block\-diagonal matrix across all layers:F=diag⁡\(F1,F2,…,FL\)F=\\operatorname\{diag\}\(F\_\{1\},F\_\{2\},\\ldots,F\_\{L\}\)\. Notably, our method requires only a small public auxiliary set for effective curvature estimation \(e\.g\.,∼1%\\sim 1\\%of the private training data size; detailed in Section[4\.4](https://arxiv.org/html/2607.05866#S4.SS4)\)\. This extreme sample efficiency stems from two factors\. First, K\-FAC drastically reduces the requisite sample complexity from𝒪​\(d\)\\mathcal\{O\}\(d\)to the order of a single layer’s width\. Second, as analyzed above, our framework primarily relies on the dominant eigenspaces and naturally tolerates fine\-grained estimation inaccuracies, further relaxing the sample requirement\.

### 3\.2\.Reconciling Isotropic Privacy with Anisotropic Optimization

Directly imposing isotropic DP constraints on anisotropic natural gradients corrupts the geometric properties essential for second\-order optimization\. To resolve this conflict, we introduce a Whitened\-Space Update Mechanism\. By shifting all DP operations into anF−1/2F^\{\-1/2\}\-whitened space, we prove that the isotropic DP mechanism translates into anisotropic, curvature\-aligned parameter updates in the original space—thus fully preserving the acceleration of NGD under DP constraints\. Crucially, the whitening matrixF−1/2F^\{\-1/2\}is not chosen arbitrarily: it emerges directly from aligning the DPL2L\_\{2\}\-norm constraint with the KL trust region, a derivation we provide in Appendix[A](https://arxiv.org/html/2607.05866#A1)for completeness\.

Algorithm[1](https://arxiv.org/html/2607.05866#alg1)formalizes the Whitened\-Space Update Mechanism\. For each sample in the Poisson\-sampled batch, the raw gradient is first projected into the whitened space viaF−1/2F^\{\-1/2\}\(Line 2\)\. This geometric transformation creates an isotropic space with respect to the geometry induced byFF\. Within this whitened space, standardL2L\_\{2\}\-norm clipping can be safely applied without distorting the relative scale of curvature directions \(Line 3\)\. Following clipping, the gradients are aggregated and perturbed with isotropic Gaussian noise \(Lines 5–6\)\. This procedure is identical to standard DP\-SGD, except that it operates on the whitened gradientsg^i\\hat\{g\}\_\{i\}rather than the raw gradientsgig\_\{i\}\. Finally, the sanitized gradient is projected back to the original parameter space \(Line 7\)\. Since the whitening matrixF−1/2F^\{\-1/2\}is precomputed entirely on public data, it acts as a linear transformation strictly independent of the sensitive data\. By the post\-processing property of DP\(Zhu et al\.,[2022](https://arxiv.org/html/2607.05866#bib.bib45),[2020](https://arxiv.org/html/2607.05866#bib.bib46)\), this inverse projection does not incur any additional privacy cost\. Consequently, the privacy accounting remains identical to DP\-SGD\. A formal privacy analysis is provided in Section[3\.5](https://arxiv.org/html/2607.05866#S3.SS5)\.

Having established the privacy guarantees of the whitened\-space mechanism, we now formally prove that the resulting parameter update is anisotropic and curvature\-aligned, successfully recovering the natural gradient in expectation\.

0:Poisson\-sampled batch

ℬ\\mathcal\{B\}, per\-sample gradients

\{gi\}xi∈ℬ\\\{g\_\{i\}\\\}\_\{x\_\{i\}\\in\\mathcal\{B\}\}, whitening matrix

F−1/2F^\{\-1/2\}, clipping threshold

CC, noise multiplier

σ\\sigma, expected batchsize

BB, learning rate

η\\eta\.

0:Parameter update

Δ​θ\\Delta\\theta
1:foreach sample

xi∈ℬx\_\{i\}\\in\\mathcal\{B\}do

2:

g^i←F−1/2​gi\\hat\{g\}\_\{i\}\\leftarrow F^\{\-1/2\}g\_\{i\}\{Project to the whitened space\}

3:

g¯i←g^i⋅min⁡\(1,C‖g^i‖2\)\\bar\{g\}\_\{i\}\\leftarrow\\hat\{g\}\_\{i\}\\cdot\\min\\left\(1,\\frac\{C\}\{\\\|\\hat\{g\}\_\{i\}\\\|\_\{2\}\}\\right\)\{Isotropic

L2L\_\{2\}clipping\}

4:endfor

5:

ξ∼𝒩​\(0,σ2​C2​𝐈\)\\xi\\sim\\mathcal\{N\}\\left\(0,\\sigma^\{2\}C^\{2\}\\mathbf\{I\}\\right\)\{Sample isotropic Gaussian noise\}

6:

g~←1B​\(∑i∈ℬg¯i\+ξ\)\\tilde\{g\}\\leftarrow\\frac\{1\}\{B\}\\big\(\\sum\_\{i\\in\\mathcal\{B\}\}\\bar\{g\}\_\{i\}\+\\xi\\big\)\{Aggregate and inject noise\}

7:

Δ​θ←−η​F−1/2​g~\\Delta\\theta\\leftarrow\-\\eta F^\{\-1/2\}\\tilde\{g\}\{Project back to original space\}

8:return

Δ​θ\\Delta\\theta

Algorithm 1Whitened\-Space Update Mechanism###### Theorem 3\.1 \(Curvature Alignment of Whitened\-Space Updates\)\.

LetΔ​θ\\Delta\\thetabe the parameter update generated by Algorithm[1](https://arxiv.org/html/2607.05866#alg1), and letFFbe the estimated curvature matrix\. For any sampleii, define its adaptive clipping scalar asci=min⁡\(1,C‖F−1/2​gi‖2\)c\_\{i\}=\\min\\left\(1,\\frac\{C\}\{\\\|F^\{\-1/2\}g\_\{i\}\\\|\_\{2\}\}\\right\)\. In expectation over the isotropic Gaussian noiseξ∼𝒩​\(0,σ2​C2​𝐈\)\\xi\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}C^\{2\}\\mathbf\{I\}\), the parameter update recovers the adaptively clipped natural gradient:

𝔼ξ​\[Δ​θ\]=−ηB​∑i∈ℬci​F−1​gi\\mathbb\{E\}\_\{\\xi\}\[\\Delta\\theta\]=\-\\frac\{\\eta\}\{B\}\\sum\_\{i\\in\\mathcal\{B\}\}c\_\{i\}F^\{\-1\}g\_\{i\}Furthermore, the injected DP noise is transformed into an anisotropic, curvature\-aligned distribution in the original parameter space, with a covariance matrix proportional toF−1F^\{\-1\}:

Cov​\(Δ​θ\)=η2​σ2​C2B2​F−1\\text\{Cov\}\(\\Delta\\theta\)=\\frac\{\\eta^\{2\}\\sigma^\{2\}C^\{2\}\}\{B^\{2\}\}F^\{\-1\}

###### Proof\.

Based on Algorithm[1](https://arxiv.org/html/2607.05866#alg1), the final parameter updateΔ​θ\\Delta\\thetais a linear combination of the gradient signal and the injected noise\. Its complete algebraic expression is formulated as:

\(7\)Δ​θ=−η​F−1/2​g~=−ηB​∑i∈ℬF−1/2​g¯i−ηB​F−1/2​ξ\\Delta\\theta=\-\\eta F^\{\-1/2\}\\tilde\{g\}=\-\\frac\{\\eta\}\{B\}\\sum\_\{i\\in\\mathcal\{B\}\}F^\{\-1/2\}\\bar\{g\}\_\{i\}\-\\frac\{\\eta\}\{B\}F^\{\-1/2\}\\xiwhereg¯i=g^i​min⁡\(1,C‖g^i‖2\)\\bar\{g\}\_\{i\}=\\hat\{g\}\_\{i\}\\min\(1,\\frac\{C\}\{\\\|\\hat\{g\}\_\{i\}\\\|\_\{2\}\}\)is the clipped gradient in the whitened space, andξ∼𝒩​\(0,σ2​C2​𝐈\)\\xi\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}C^\{2\}\\mathbf\{I\}\)is the isotropic Gaussian noise\. We establish the curvature alignment of the final parameter update by analyzing its expectation and covariance, respectively\.

#### Part I: Parameter Update Expectation\.

For each sampleii, we denote its adaptive clipping factor asci=min⁡\(1,C‖g^i‖2\)c\_\{i\}=\\min\\left\(1,\\frac\{C\}\{\\\|\\hat\{g\}\_\{i\}\\\|\_\{2\}\}\\right\)\. Sincecic\_\{i\}is a scalar, it commutes with matrix multiplication\. Together with the fact that the injected DP noise is zero\-mean \(𝔼ξ​\[ξ\]=0\\mathbb\{E\}\_\{\\xi\}\[\\xi\]=0\), the stochastic noise term in Eq\. \([7](https://arxiv.org/html/2607.05866#S3.E7)\) vanishes\. Taking the expectation overξ\\xi, we obtain:

\(8\)𝔼ξ​\[Δ​θ\]=−ηB​∑i∈ℬF−1/2​\(ci​F−1/2​gi\)=−ηB​∑i∈ℬci​F−1​gi\\mathbb\{E\}\_\{\\xi\}\[\\Delta\\theta\]=\-\\frac\{\\eta\}\{B\}\\sum\_\{i\\in\\mathcal\{B\}\}F^\{\-1/2\}\\left\(c\_\{i\}F^\{\-1/2\}g\_\{i\}\\right\)=\-\\frac\{\\eta\}\{B\}\\sum\_\{i\\in\\mathcal\{B\}\}c\_\{i\}F^\{\-1\}g\_\{i\}Because the clipping factorcic\_\{i\}exclusively rescales the magnitude without altering the vector’s orientation, this expectation recovers the adaptively clipped natural gradientF−1​giF^\{\-1\}g\_\{i\}\.

#### Part II: Parameter Update Covariance\.

Next, we analyze the covariance matrix of the updateΔ​θ\\Delta\\theta\. Since the first term in Eq\. \([7](https://arxiv.org/html/2607.05866#S3.E7)\) is deterministic with respect to the DP mechanism, it contributes nothing to the variance\. The covariance is entirely determined by the linear transformation of the Gaussian noise termξ\\xi\. Using the property of covariance under affine transformations,Cov​\(A​X\)=A​Cov​\(X\)​AT\\text\{Cov\}\(AX\)=A\\text\{Cov\}\(X\)A^\{T\}, we derive:

Cov​\(Δ​θ\)\\displaystyle\\text\{Cov\}\(\\Delta\\theta\)=Cov​\(−ηB​F−1/2​ξ\)\\displaystyle=\\text\{Cov\}\\left\(\-\\frac\{\\eta\}\{B\}F^\{\-1/2\}\\xi\\right\)=\(−ηB​F−1/2\)​Cov​\(ξ\)​\(−ηB​F−1/2\)T\\displaystyle=\\left\(\-\\frac\{\\eta\}\{B\}F^\{\-1/2\}\\right\)\\text\{Cov\}\(\\xi\)\\left\(\-\\frac\{\\eta\}\{B\}F^\{\-1/2\}\\right\)^\{T\}=η2B2​F−1/2​\(σ2​C2​𝐈\)​F−1/2\\displaystyle=\\frac\{\\eta^\{2\}\}\{B^\{2\}\}F^\{\-1/2\}\(\\sigma^\{2\}C^\{2\}\\mathbf\{I\}\)F^\{\-1/2\}\(9\)=η2​σ2​C2B2​F−1\\displaystyle=\\frac\{\\eta^\{2\}\\sigma^\{2\}C^\{2\}\}\{B^\{2\}\}F^\{\-1\}Here we utilize the symmetry ofF−1/2F^\{\-1/2\}, a property intrinsically inherited from the Fisher information matrix\. This demonstrates that the originally isotropic noise is transformed into an anisotropic distribution whose covariance is precisely shaped by the inverse curvatureF−1F^\{\-1\}, thereby aligning with the natural gradient\.

Combining Part I and Part II, the whitened\-Space mechanism preserves the curvature\-aligned geometry for both the gradient signal and the DP perturbation, thereby completing the proof\. ∎

Together with the privacy guarantee established, Theorem[3\.1](https://arxiv.org/html/2607.05866#S3.Thmtheorem1)confirms that the Whitened\-Space Update Mechanism resolves the inherent geometric conflict between DP and NGD\. By achieving this theoretical alignment, our method strictly satisfies the differential privacy constraints while preserving the crucial acceleration properties of second\-order optimization\.

The KL\-DP Duality\.Theorem[3\.1](https://arxiv.org/html/2607.05866#S3.Thmtheorem1)reveals a deeper structural connection within our method: the DP sensitivity boundCCsimultaneously defines the radius of the KL trust region for NGD\.

To avoid destabilizing updates caused by pathological curvature, NGD controls the per\-step distributional shift by bounding the KL divergence\. Under the standard small\-step assumption of NGD, higher\-order terms are negligible and the KL divergence is locally governed by

\(10\)DKL​\(Pθ∥Pθ\+Δ​θ\)=12​Δ​θ⊤​F​Δ​θ\.D\_\{\\text\{KL\}\}\(P\_\{\\theta\}\\parallel P\_\{\\theta\+\\Delta\\theta\}\)=\\frac\{1\}\{2\}\\Delta\\theta^\{\\top\}F\\Delta\\theta\.Consider the parameter update contributed by a single clipped sample in our whitened\-space mechanism:Δ​θi=−η​F−1/2​g¯i\\Delta\\theta\_\{i\}=\-\\eta F^\{\-1/2\}\\bar\{g\}\_\{i\}\. Substituting this into the above KL divergence yields:

DKL\\displaystyle D\_\{\\text\{KL\}\}=12​\(−η​F−1/2​g¯i\)⊤​F​\(−η​F−1/2​g¯i\)\\displaystyle=\\frac\{1\}\{2\}\\left\(\-\\eta F^\{\-1/2\}\\bar\{g\}\_\{i\}\\right\)^\{\\top\}F\\left\(\-\\eta F^\{\-1/2\}\\bar\{g\}\_\{i\}\\right\)=η22​g¯i⊤​\(F−1/2​F​F−1/2\)​g¯i\\displaystyle=\\frac\{\\eta^\{2\}\}\{2\}\\bar\{g\}\_\{i\}^\{\\top\}\\left\(F^\{\-1/2\}FF^\{\-1/2\}\\right\)\\bar\{g\}\_\{i\}\(11\)=η22​‖g¯i‖22\.\\displaystyle=\\frac\{\\eta^\{2\}\}\{2\}\\\|\\bar\{g\}\_\{i\}\\\|\_\{2\}^\{2\}\.Since DP clipping in the whitened space enforces‖g¯i‖2≤C\\\|\\bar\{g\}\_\{i\}\\\|\_\{2\}\\leq C, we immediately obtain

\(12\)‖g¯i‖2≤C⟹DKL≤η22​C2\.\\\|\\bar\{g\}\_\{i\}\\\|\_\{2\}\\leq C\\quad\\Longrightarrow\\quad D\_\{\\text\{KL\}\}\\leq\\frac\{\\eta^\{2\}\}\{2\}C^\{2\}\.This implies that the scalarCCplays a dual role\. As a privacy safeguard, it bounds the maximal contribution of any individual sample, rigorously ensuring differential privacy\. As an optimization boundary, it explicitly limits the distributional shift that any single data point can induce, matching the trust\-region constraint required for stable NGD convergence\. This marks a fundamental departure from standard DP\-SGD, where the clipping threshold is an arbitrary, isotropic bound imposed without regard to the geometry of the loss landscape\. In our framework, the DP sensitivity constraint is instead defined in a principled, curvature\-aware manner derived directly from second\-order optimization theory\.

The underlying mechanism behind this duality is the geometric equivalence established by the whitening transformation:

\(13\)‖g^i‖2=‖F−1/2​gi‖2=gi⊤​F−1​gi=‖gi‖F−1\.\\\|\\hat\{g\}\_\{i\}\\\|\_\{2\}=\\\|F^\{\-1/2\}g\_\{i\}\\\|\_\{2\}=\\sqrt\{g\_\{i\}^\{\\top\}F^\{\-1\}g\_\{i\}\}=\\\|g\_\{i\}\\\|\_\{F^\{\-1\}\}\.Geometrically, this equivalence explains how our mechanism natively adapts to the local loss landscape\. Bounding theL2L\_\{2\}\-norm in the whitened space \(‖g^i‖2≤C\\\|\\hat\{g\}\_\{i\}\\\|\_\{2\}\\leq C\) is mathematically equivalent to bounding the curvature\-aware Mahalanobis norm in the parameter space \(‖gi‖F−1≤C\\\|g\_\{i\}\\\|\_\{F^\{\-1\}\}\\leq C\)\. Consequently, steep directions \(associated with large eigenvalues ofFF\) are inherently suppressed to prevent divergence, while informative components along flat directions \(small eigenvalues\) are faithfully preserved\.

### 3\.3\.Dynamic Curvature Clamping via Euclidean Step Equivalence

Although DP clipping bounds the gradient’sL2L\_\{2\}\-norm in the whitened space, the inverse projection viaF−1/2F^\{\-1/2\}leaves the Euclidean step size unbounded in the original parameter space\. In flat directions where the curvature eigenvalues approach zero \(λi→0\\lambda\_\{i\}\\to 0\), parameter updates are magnified by a factor of1/λi→∞1/\\sqrt\{\\lambda\_\{i\}\}\\to\\infty, causing severe training instability\. Crucially, this parameter explosion can occur without violating the KL trust region: flat directions permit large parameter moves with negligible distributional shift\(Chaudhari et al\.,[2016](https://arxiv.org/html/2607.05866#bib.bib11); Foret et al\.,[2020](https://arxiv.org/html/2607.05866#bib.bib18)\), making the KL constraint structurally blind to such instability\. An explicit Euclidean safeguard on the parameter update is therefore necessary\.

#### Deriving the Euclidean Safety Bound\.

To establish this safeguard, we adopt standard DP\-SGD as a trusted baseline\. DP\-SGD maintains well\-controlled expected Euclidean displacements throughout training\. By enforcing that the expected displacement of our DP\-NGD update does not exceed that provable upper bound, we obtain a rigorous safety floor for robust convergence\.

In standard DP\-SGD, the parameter update is formulated asΔ​θsgd=−ηsgd​\(g¯sgd\+1B​ξsgd\)\\Delta\\theta\_\{\\text\{sgd\}\}=\-\\eta\_\{\\text\{sgd\}\}\\big\(\\bar\{g\}\_\{\\text\{sgd\}\}\+\\frac\{1\}\{B\}\\xi\_\{\\text\{sgd\}\}\\big\), where the gradient is clipped to‖g¯sgd‖2≤Csgd\\\|\\bar\{g\}\_\{\\text\{sgd\}\}\\\|\_\{2\}\\leq C\_\{\\text\{sgd\}\}and the noise isξsgd∼𝒩​\(0,σ2​Csgd2​𝐈\)\\xi\_\{\\text\{sgd\}\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}C\_\{\\text\{sgd\}\}^\{2\}\\mathbf\{I\}\)\. Because the DP noise is zero\-mean and independent of the gradient, the cross term𝔼​\[g¯sgd⊤​ξsgd\]\\mathbb\{E\}\[\\bar\{g\}\_\{\\text\{sgd\}\}^\{\\top\}\\xi\_\{\\text\{sgd\}\}\]vanishes\. Substituting𝔼​\[‖ξsgd‖22\]=d​σ2​Csgd2\\mathbb\{E\}\[\\\|\\xi\_\{\\text\{sgd\}\}\\\|\_\{2\}^\{2\}\]=d\\sigma^\{2\}C\_\{\\text\{sgd\}\}^\{2\}for thedd\-dimensional Gaussian noise, the expected squared Euclidean step size simplifies to

𝔼​\[‖Δ​θsgd‖22\]\\displaystyle\\mathbb\{E\}\[\\\|\\Delta\\theta\_\{\\text\{sgd\}\}\\\|\_\{2\}^\{2\}\]=ηsgd2​\(‖g¯sgd‖22\+1B2​𝔼​\[‖ξsgd‖22\]\)\\displaystyle=\\eta\_\{\\text\{sgd\}\}^\{2\}\\Bigl\(\\\|\\bar\{g\}\_\{\\text\{sgd\}\}\\\|\_\{2\}^\{2\}\+\\frac\{1\}\{B^\{2\}\}\\mathbb\{E\}\[\\\|\\xi\_\{\\text\{sgd\}\}\\\|\_\{2\}^\{2\}\]\\Bigr\)\(14\)≤ηsgd2​Csgd2​\(1\+d​σ2B2\)\.\\displaystyle\\leq\\eta\_\{\\text\{sgd\}\}^\{2\}C\_\{\\text\{sgd\}\}^\{2\}\\Bigl\(1\+\\frac\{d\\sigma^\{2\}\}\{B^\{2\}\}\\Bigr\)\.
For our DP\-NGD framework, the parameter update is formulated asΔ​θngd=−ηngd​F−1/2​\(g¯ngd\+1B​ξngd\)\\Delta\\theta\_\{\\text\{ngd\}\}=\-\\eta\_\{\\text\{ngd\}\}F^\{\-1/2\}\\big\(\\bar\{g\}\_\{\\text\{ngd\}\}\+\\frac\{1\}\{B\}\\xi\_\{\\text\{ngd\}\}\\big\)\. Taking the expectation and applying the Rayleigh quotient bound for symmetric matrices \(v⊤​F−1​v≤1λmin​‖v‖22v^\{\\top\}F^\{\-1\}v\\leq\\frac\{1\}\{\\lambda\_\{\\text\{min\}\}\}\\\|v\\\|\_\{2\}^\{2\}\), we obtain the upper bound

𝔼​\[‖Δ​θngd‖22\]\\displaystyle\\mathbb\{E\}\[\\\|\\Delta\\theta\_\{\\text\{ngd\}\}\\\|\_\{2\}^\{2\}\]=ηngd2​\(g¯ngd⊤​F−1​g¯ngd\+1B2​𝔼​\[ξngd⊤​F−1​ξngd\]\)\\displaystyle=\\eta\_\{\\text\{ngd\}\}^\{2\}\\Bigl\(\\bar\{g\}\_\{\\text\{ngd\}\}^\{\\top\}F^\{\-1\}\\bar\{g\}\_\{\\text\{ngd\}\}\+\\frac\{1\}\{B^\{2\}\}\\mathbb\{E\}\\bigl\[\\xi\_\{\\text\{ngd\}\}^\{\\top\}F^\{\-1\}\\xi\_\{\\text\{ngd\}\}\\bigr\]\\Bigr\)\(15\)≤ηngd2λmin​Cngd2​\(1\+d​σ2B2\)\.\\displaystyle\\leq\\frac\{\\eta\_\{\\text\{ngd\}\}^\{2\}\}\{\\lambda\_\{\\text\{min\}\}\}C\_\{\\text\{ngd\}\}^\{2\}\\Bigl\(1\+\\frac\{d\\sigma^\{2\}\}\{B^\{2\}\}\\Bigr\)\.
Assuming matched privacy configurations \(identical batch sizeBBand noise multiplierσ\\sigma\), enforcing𝔼​\[‖Δ​θngd‖22\]≤𝔼​\[‖Δ​θsgd‖22\]\\mathbb\{E\}\[\\\|\\Delta\\theta\_\{\\text\{ngd\}\}\\\|\_\{2\}^\{2\}\]\\leq\\mathbb\{E\}\[\\\|\\Delta\\theta\_\{\\text\{sgd\}\}\\\|\_\{2\}^\{2\}\]allows the common factor\(1\+d​σ2/B2\)\(1\+d\\sigma^\{2\}/B^\{2\}\)to cancel out\. Solving for the minimum eigenvalue yields the safety boundλsafe\\lambda\_\{\\text\{safe\}\}:

\(16\)λmin≥\(ηngd​Cngdηsgd​Csgd\)2≜λsafe\.\\lambda\_\{\\text\{min\}\}\\geq\\left\(\\frac\{\\eta\_\{\\text\{ngd\}\}C\_\{\\text\{ngd\}\}\}\{\\eta\_\{\\text\{sgd\}\}C\_\{\\text\{sgd\}\}\}\\right\)^\{2\}\\triangleq\\lambda\_\{\\text\{safe\}\}\.
This inequality reveals that as long as every eigenvalue satisfiesλi≥λsafe\\lambda\_\{i\}\\geq\\lambda\_\{\\text\{safe\}\}, the expected Euclidean norm of the DP\-NGD update is bounded by that of the DP\-SGD baseline\. To enforce this guarantee throughout training, each time the curvature matrixFFis estimated, we dynamically clamp its eigenvalues to this safety floor:

\(17\)λ~i=max⁡\(λi,λsafe\)\.\\tilde\{\\lambda\}\_\{i\}=\\max\(\\lambda\_\{i\},\\lambda\_\{\\text\{safe\}\}\)\.Preconditioning the gradient with this clamped curvature ensures optimization stability in the Euclidean space\. Unlike standard NGD, which routinely adds a damping term\(F\+λ​I\)−1\(F\+\\lambda I\)^\{\-1\}that indiscriminately perturbs all eigen\-directions and sacrifices curvature accuracy in steep directions, our clamping operation is highly selective\. It leaves well\-conditioned, steep directions \(λi≥λsafe\\lambda\_\{i\}\\geq\\lambda\_\{\\text\{safe\}\}\) completely untouched, fully preserving their second\-order acceleration, while safely lifting dangerously flat directions \(λi<λsafe\\lambda\_\{i\}<\\lambda\_\{\\text\{safe\}\}\) to the safety floor\.

#### The Dynamic Clamping Schedule\.

Enforcingλsafe\\lambda\_\{\\text\{safe\}\}as a fixed clamping floor effectively suppresses parameter explosion, but is unnecessarily conservative during the high\-SNR mid\-training phase\. The theoretical bound assumes a worst\-case scenario where all eigenvalues are clamped toλsafe\\lambda\_\{\\text\{safe\}\}\. In practice, however, most eigen\-directions remain steep \(λi≫λsafe\\lambda\_\{i\}\\gg\\lambda\_\{\\text\{safe\}\}\) during this mid\-phase, and the actual Euclidean displacement is far below the safety limit\. A static floor ignores this available margin, needlessly sacrificing curvature accuracy and suppressing second\-order acceleration when Euclidean safety is already guaranteed\.

To recover this lost efficiency during mid\-training without compromising robust convergence in late\-stage flat basins, we introduce a dynamic clamping schedule over a total ofTTiterations:

\(18\)λt=\{λsafe−\(λsafe−λbase\)⋅tT1,if​0≤t<T1λbase\+\(λsafe−λbase\)⋅\(t−T1T−T1\)p,if​T1≤t≤T\\lambda\_\{t\}=\\begin\{cases\}\\lambda\_\{\\text\{safe\}\}\-\(\\lambda\_\{\\text\{safe\}\}\-\\lambda\_\{\\text\{base\}\}\)\\cdot\\frac\{t\}\{T\_\{1\}\},&\\text\{if \}0\\leq t<T\_\{1\}\\\\\[4\.0pt\] \\lambda\_\{\\text\{base\}\}\+\(\\lambda\_\{\\text\{safe\}\}\-\\lambda\_\{\\text\{base\}\}\)\\cdot\\left\(\\frac\{t\-T\_\{1\}\}\{T\-T\_\{1\}\}\\right\)^\{p\},&\\text\{if \}T\_\{1\}\\leq t\\leq T\\end\{cases\}whereT1T\_\{1\}is a brief warmup period,λbase<λsafe\\lambda\_\{\\text\{base\}\}<\\lambda\_\{\\text\{safe\}\}is a standard numerical stabilizer adopted from classical NGD, andp\>1p\>1controls the steepness of the polynomial transition\. BothT1T\_\{1\}\(as a fraction ofTT\) andppare insensitive within practical ranges; a detailed sensitivity analysis is provided in Section[4\.7](https://arxiv.org/html/2607.05866#S4.SS7)\.

This formulation naturally yields three distinct phases\. In the initial warmup \(0≤t<T10\\leq t<T\_\{1\}\), curvature estimates are highly unreliable due to random initialization; thus, the floor starts at the conservative boundλsafe\\lambda\_\{\\text\{safe\}\}and linearly decays toλbase\\lambda\_\{\\text\{base\}\}as training stabilizes\. During the prolonged mid\-phase \(T1≤t≪TT\_\{1\}\\leq t\\ll T\), the exponentp\>1p\>1creates an implicit acceleration plateau whereλt\\lambda\_\{t\}remains nearly flat aroundλbase\\lambda\_\{\\text\{base\}\}, fully unleashing second\-order acceleration while the signal\-to\-noise ratio remains high\. Finally, as the model enters flat basins in the late stage and the clamped fraction of eigenvalues inevitably rises, the polynomial tail drivesλt\\lambda\_\{t\}rapidly back toλsafe\\lambda\_\{\\text\{safe\}\}atTT, seamlessly restoring the theoretical Euclidean safety guarantee\.

0:Private dataset

𝒟\\mathcal\{D\}\(size

NN\), public dataset

𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}, learning rate

η\\eta, DP clipping bound

CC, noise multiplier

σ\\sigma, expected batch size

BB, curvature update interval

TcurT\_\{\\text\{cur\}\}, total iterations

TT\.

0:Differentially private model parameters

θT\\theta\_\{T\}\.

1:Initialize model parameters

θ0\\theta\_\{0\}\.

2:Compute theoretical safety floor

λsafe\\lambda\_\{\\text\{safe\}\}via Eq\. \([16](https://arxiv.org/html/2607.05866#S3.E16)\)\.

13:for

t=0t=0to

T−1T\-1do

4:\[Module 1\] Privacy\-Free Curvature Estimation

5:if

tmodTcur=0t\\bmod T\_\{\\text\{cur\}\}=0then

6:Compute empirical curvature

FFon

𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}via K\-FAC\.

7:Perform eigendecomposition:

F=Q​Λ​Q⊤F=Q\\Lambda Q^\{\\top\}\. \{Cache

QQ,

Λ\\Lambdafor

TcurT\_\{\\text\{cur\}\}steps\}

28:endif

9:\[Module 2\] Dynamic Curvature Clamping

10:Compute dynamic clamping floor

λt\\lambda\_\{t\}for step

ttvia Eq\. \([18](https://arxiv.org/html/2607.05866#S3.E18)\)\.

11:Clamp eigenvalues element\-wise:

Λ~=max⁡\(Λ,λt​𝐈\)\\tilde\{\\Lambda\}=\\max\(\\Lambda,\\lambda\_\{t\}\\mathbf\{I\}\)\.

312:Update whitening matrix:

F~−1/2=Q​Λ~−1/2​Q⊤\\tilde\{F\}^\{\-1/2\}=Q\\tilde\{\\Lambda\}^\{\-1/2\}Q^\{\\top\}\.

13:\[Module 3\] Whitened\-Space Update Mechanism

14:Sample batch

ℬt⊆𝒟\\mathcal\{B\}\_\{t\}\\subseteq\\mathcal\{D\}with probability

q=B/Nq=B/N\.

15:Compute per\-sample gradients:

gi\(t\)=∇θℒ​\(xi,θt\)g\_\{i\}^\{\(t\)\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\(x\_\{i\},\\theta\_\{t\}\)for each sample

xi∈ℬtx\_\{i\}\\in\\mathcal\{B\}\_\{t\}\.

416:

Δ​θt←WhitenedSpaceUpdate​\(\{gi\(t\)\}xi∈ℬt,F~−1/2,C,σ\)\\Delta\\theta\_\{t\}\\leftarrow\\text\{WhitenedSpaceUpdate\}\\big\(\\\{g\_\{i\}^\{\(t\)\}\\\}\_\{x\_\{i\}\\in\\mathcal\{B\}\_\{t\}\},\\tilde\{F\}^\{\-1/2\},C,\\sigma\\big\)\. \{Algorithm[1](https://arxiv.org/html/2607.05866#alg1): Project

→\\toClip

→\\toNoise

→\\toInverse project\}

17:Update parameters:

θt\+1=θt\+Δ​θt\\theta\_\{t\+1\}=\\theta\_\{t\}\+\\Delta\\theta\_\{t\}\.

18:endfor

19:Return

θT\\theta\_\{T\}\.

Algorithm 2Differentially Private Natural Gradient Descent \(DP\-NGD\)

### 3\.4\.The Complete Algorithm

We consolidate our three core components into a unified, end\-to\-end DP\-NGD framework\. Algorithm[2](https://arxiv.org/html/2607.05866#alg2)outlines the complete training procedure, which is structurally divided into three modules corresponding to the fundamental challenges\.

Module 1: Privacy\-Free Curvature Estimation\.To avoid incurring significant privacy cost on curvature estimation, we precompute the empirical Fisher matrixFFexclusively on a small public auxiliary dataset𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}using the K\-FAC approximation\. To amortize the computational overhead, curvature estimation and its subsequent eigendecomposition \(F=Q​Λ​Q⊤F=Q\\Lambda Q^\{\\top\}\) are performed once everyTcurT\_\{\\text\{cur\}\}iterations and cached for reuse\.

Module 2: Dynamic Curvature Clamping\.Before constructing the whitening matrixF~−1/2\\tilde\{F\}^\{\-1/2\}, we clamp the curvature eigenvalues to a dynamically scheduled floorλt\\lambda\_\{t\}:Λ~=max⁡\(Λ,λt​𝐈\)\\tilde\{\\Lambda\}=\\max\(\\Lambda,\\lambda\_\{t\}\\mathbf\{I\}\)\. Here,λt\\lambda\_\{t\}is generated solely based on the current training stepttaccording to Eq\. \([18](https://arxiv.org/html/2607.05866#S3.E18)\)\. This stage\-aware clamping serves a threefold purpose: it stabilizes randomly initialized parameters during the early phase, fully unleashes second\-order acceleration in the high\-SNR mid\-phase, and ensures robust convergence in late\-stage flat basins\.

Module 3: Whitened\-Space Update Mechanism\.A minibatchℬt\\mathcal\{B\}\_\{t\}is sampled from the private dataset via Poisson subsampling with probabilityqq\. We then compute the per\-sample gradients\{gi\(t\)\}xi∈ℬt\\\{g\_\{i\}^\{\(t\)\}\\\}\_\{x\_\{i\}\\in\\mathcal\{B\}\_\{t\}\}and execute the whitened\-space mechanism defined in Algorithm[1](https://arxiv.org/html/2607.05866#alg1)\. This subroutine follows a seamless pipeline: projecting gradients into the whitened space, applying isotropicL2L\_\{2\}\-norm clipping, injecting isotropic Gaussian noise, and finally mapping the aggregated update back to the parameter space viaF~−1/2\\tilde\{F\}^\{\-1/2\}\. By the mathematical equivalence established in Theorem[3\.1](https://arxiv.org/html/2607.05866#S3.Thmtheorem1), this mechanism preserves the anisotropic acceleration of NGD under rigorous DP guaranties\.

Remark: Computational Efficiency\.Despite the additional steps introduced by curvature estimation, the computational overhead of DP\-NGD is well\-amortized\. This efficiency stems from three structural properties: \(1\) curvature estimation operates on a small public batch and is lazily updated only once everyTcurT\_\{\\text\{cur\}\}iterations; \(2\) the K\-FAC approximation reduces the eigendecomposition complexity from𝒪​\(d3\)\\mathcal\{O\}\(d^\{3\}\)to𝒪​\(din3\+dout3\)\\mathcal\{O\}\(d\_\{\\text\{in\}\}^\{3\}\+d\_\{\\text\{out\}\}^\{3\}\), wheredin,dout≪dd\_\{\\text\{in\}\},d\_\{\\text\{out\}\}\\ll d; and \(3\) second\-order optimization substantially reduces the total number of iterations required to achieve baseline accuracy, effectively offsetting the additional per\-step cost\. A detailed empirical analysis of end\-to\-end wall\-clock runtime is provided in Section[4\.6](https://arxiv.org/html/2607.05866#S4.SS6)\.

### 3\.5\.Privacy Analysis

We now establish the privacy guarantee of the complete DP\-NGD framework\. Among the three modules of Algorithm[2](https://arxiv.org/html/2607.05866#alg2), Modules 1 and 2 incur zero privacy cost\. The curvature matrixFFis estimated exclusively on the public auxiliary dataset𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}, which is disjoint from the private training set and lies strictly outside the adjacency relation that defines differential privacy\. The dynamic clamping floorλt\\lambda\_\{t\}is computed solely from the current training stepttand the predefined hyperparametersη\\etaandCC, without accessing any private data\. Both modules are therefore privacy\-free by construction\.

Module 3, the whitened\-space update, is the only component that accesses private data\. It consists of four privacy\-relevant operations: projecting gradients through the precomputed linear mapF~−1/2\\tilde\{F\}^\{\-1/2\}; clipping them to a constantL2L\_\{2\}\-sensitivityCC; injecting isotropic Gaussian noise parameterized byσ\\sigmaandCC; and performing the inverse projection viaF~−1/2\\tilde\{F\}^\{\-1/2\}\. Because the whitening matrixF~−1/2\\tilde\{F\}^\{\-1/2\}is derived from public data and remains fixed for the current step, it acts as a linear transformation strictly independent of the private data\. By the post\-processing property of DP, both the projection and the inverse projection viaF~−1/2\\tilde\{F\}^\{\-1/2\}incur no additional privacy cost\. Consequently, the privacy loss of Module 3 stems exclusively from the intermediate clipping and noise injection steps\.

The clipping and noise injection steps in DP\-NGD are executed identically to those in standard DP\-SGD, while the only differences are the clipping space \(whitened vs\. Euclidean\) and the clipping threshold \(CngdC\_\{\\text\{ngd\}\}vs\.CsgdC\_\{\\text\{sgd\}\}\), as summarized in Table[1](https://arxiv.org/html/2607.05866#S3.T1)\. The clipping space depends solely on the whitening matrixF~−1/2\\tilde\{F\}^\{\-1/2\}, which has been shown to leave the privacy budget unchanged\. The clipping threshold acts symmetrically as a scaling factor for both the global sensitivity and the injected noise variance, mathematically canceling out in the privacy loss derivation\. Thus, the step\-wise privacy loss of DP\-NGD is identical to that of standard DP\-SGD\. Given this strict equivalence, the end\-to\-end privacy guarantee overTTiterations composes identically to that of standard DP\-SGD, and the final\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP bound can be tightly tracked by any standard privacy accountant \(e\.g\., Rényi DP or the PRV accountant\) without modification\.

Table 1\.Privacy cost comparison between DP\-SGD and DP\-NGD\.The only differences are the clipping space and the clipping threshold\. The former depends on a fixed linear mappingF~−1/2\\tilde\{F\}^\{\-1/2\}that does not alter the privacy cost\. The latter identically scales both global sensitivity and noise variance, canceling out in the privacy loss derivation\. Therefore, identical per\-step privacy loss is guaranteed, yielding the exact same total privacy budget\.ComponentStandard DP\-SGDDP\-NGD \(Ours\)Clipping spaceEuclidean \(∥⋅∥2\\\|\\cdot\\\|\_\{2\}\)Whitened \(∥⋅∥F~−1\\\|\\cdot\\\|\_\{\\tilde\{F\}^\{\-1\}\}\)Clipping thresholdCsgdC\_\{\\text\{sgd\}\}CngdC\_\{\\text\{ngd\}\}Noise mechanism𝒩​\(0,σ2​Csgd2​𝐈\)\\mathcal\{N\}\\big\(0,\\sigma^\{2\}C\_\{\\text\{sgd\}\}^\{2\}\\mathbf\{I\}\\big\)𝒩​\(0,σ2​Cngd2​𝐈\)\\mathcal\{N\}\\big\(0,\\sigma^\{2\}C\_\{\\text\{ngd\}\}^\{2\}\\mathbf\{I\}\\big\)Per\-step Privacy LossDetermined byσ,q\\sigma,qDetermined byσ,q\\sigma,qTotal Budget\(ϵ,δ\)\(\\epsilon,\\delta\)\(ϵ,δ\)\(\\epsilon,\\delta\)

## 4\.Evaluation

### 4\.1\.Experimental Setup

Datasets and Models\.We evaluate on three standard benchmarks: CIFAR\-10\(Abouelnaga et al\.,[2016](https://arxiv.org/html/2607.05866#bib.bib3)\), SVHN\(Madhulika and Sampath,[2022](https://arxiv.org/html/2607.05866#bib.bib29)\), and UTKFace\(Chandaliya et al\.,[2019](https://arxiv.org/html/2607.05866#bib.bib10)\), covering diverse image domains\. We adopt WRN\-16\-4\(Tang et al\.,[2023](https://arxiv.org/html/2607.05866#bib.bib38)\)and ResNet\-20\(Huo et al\.,[2020](https://arxiv.org/html/2607.05866#bib.bib23)\)as our model architectures, both widely used in DP research\. All Batch Normalization layers are replaced with Group Normalization \(GN\)\(Wu and He,[2018](https://arxiv.org/html/2607.05866#bib.bib42); Babu et al\.,[2024](https://arxiv.org/html/2607.05866#bib.bib5)\)to prevent privacy leakage through batch statistics\. A small public auxiliary set containing 500 images \(≈1%\\approx 1\\%of the training set size\) is constructed by random sampling from out\-of\-distribution \(OOD\) sources—CIFAR\-100 and ImageNet\. All images are resized to32×3232\\times 32pixels\.

Baselines\.The essence of NGD lies in utilizing curvature to precondition \(or project\) the first\-order gradients\. Consequently, the advantage of our method hinges on the quality of the curvature estimation\. To validate this, we construct a strictly controlled suite of baselines, which also leverage public data to construct their own preconditioners or projections\. To ensure fair comparison, every auxiliary\-data\-assisted baseline adopts exactly the same public dataset𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}\.

∙\\bulletDP\-SGD & DP\-SGD\-PT\. DP\-SGD is the canonical isotropic first\-order baseline, implemented following the state\-of\-the\-art recipe of DeepMind\(De et al\.,[2022](https://arxiv.org/html/2607.05866#bib.bib13)\)\. To verify that our gains stem from geometric curvature extraction rather than mere feature initialization, we additionally introduce DP\-SGD\-PT, which is pre\-trained on𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}before DP training\.

∙\\bulletAdaDPS\. AdaDPS\(Li et al\.,[2022](https://arxiv.org/html/2607.05866#bib.bib27)\)is a representative adaptive DP optimizer that uses𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}to estimate gradient second moments, approximating the diagonal of the Fisher matrixFFas per parameter learning‑rate scalers\. This baseline tests whether capturing only diagonal variance is sufficient under DP; the performance gap relative to DP\-NGD reveals the necessity of capturing parameter cross\-correlations through our block\-diagonal preconditioner\.

∙\\bulletGEP\. Gradient Embedding Perturbation \(GEP\)\(Yu et al\.,[2021](https://arxiv.org/html/2607.05866#bib.bib44)\)uses𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}to construct a low\-rank subspace via PCA\(Jolliffe and Cadima,[2016](https://arxiv.org/html/2607.05866#bib.bib24); Greenacre et al\.,[2003](https://arxiv.org/html/2607.05866#bib.bib20)\), and subsequently projects private gradients onto this subspace to reduce the noise dimension\. Comparing against GEP demonstrates that low\-rank geometric projections tend to discard critical tail features, whereas our block\-diagonal preconditioner operates in the full\-dimensional parameter space, preserving the complete manifold topology without information truncation\.

Unified Training Configuration\.To establish strong baselines, all methods share a unified modern DP training recipe:16×16\\timesaugmentation multiplicity\(Bao et al\.,[2023](https://arxiv.org/html/2607.05866#bib.bib6)\), exponential moving average \(EMA\)\(Morales\-Brotons et al\.,[2024](https://arxiv.org/html/2607.05866#bib.bib35); Ahn and Cutkosky,[2024](https://arxiv.org/html/2607.05866#bib.bib4)\)for evaluation, and a uniformly large batch size ofB=4096B=4096\. We evaluate the privacy\-utility trade\-off across privacy budgetsϵ∈\{1\.0,2\.0,4\.0,6\.0,8\.0\}\\epsilon\\in\\\{1\.0,2\.0,4\.0,6\.0,8\.0\\\}with a fixedδ=10−5\\delta=10^\{\-5\}\. The expected batch sizeB=4096B=4096implies the same subsampling rateq=B/Nq=B/Nfor all methods\. To guarantee absolute comparison fairness, the per\-step noise multiplierσ\\sigmais independently calibrated for each method via the PRV accountant\(Gopi et al\.,[2021](https://arxiv.org/html/2607.05866#bib.bib19); Meiser and Mohammadi,[2018](https://arxiv.org/html/2607.05866#bib.bib32)\)provided by Opacus, ensuring that the target privacy budget is exactly exhausted at the final iteration\.

Crucially, we emphasize that the varying choices of the clipping thresholdCCdo not compromise comparison fairness\. SinceCCsymmetrically scales both the global sensitivity and the injected noise variance, it cancels out mathematically in the privacy loss derivation, thus affecting solely the optimization dynamics rather than the privacy budget\. Notably, by the KL\-DP duality established in Section[3\.2](https://arxiv.org/html/2607.05866#S3.SS2), our clipping threshold simultaneously defines the KL trust\-region radius of NGD\. We therefore adopt a fixedC=10\.0C=10\.0, directly inherited from the standard NGD KL constraint\. For the baseline methods, we set the clipping thresholdCCto1\.01\.0for DP\-SGD and2\.02\.0for AdaDPS, following their respective optimal configurations\. For GEP, we adopt its recommended settings ofC=10\.0C=10\.0for the gradient embedding andC=2\.0C=2\.0for the residual gradient\.

Hyperparameter Tuning\.To report every method at its peak performance, we conduct a rigorous hyperparameter tuning protocol\. For the universal optimization parameters—specifically the learning rateη\\etaand total training epochsTT—we perform a comprehensive grid search to identify the optimal utility\-privacy trade\-off for each individual method\. For method\-specific structural parameters \(e\.g\., adaptive momentum factors\), we strictly adopt the recommended values from the original papers; when explicit recommendations are absent, we search within a narrow range around the original experimental settings\. This protocol ensures that any observed performance gaps reflect fundamental algorithmic advantages rather than suboptimal hyperparameter choices\. For our DP\-NGD, we set the auxiliary public dataset size to\|𝒟pub\|=500\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|=500\(approximately1%1\\%of the private training data\), the curvature update interval toTcur=8T\_\{\\text\{cur\}\}=8, and configure the dynamic clamping schedule withT1=0\.1​TT\_\{1\}=0\.1\\,Tandp=10p=10\. The robustness of these choices is validated by the sensitivity analysis in Section[4\.7](https://arxiv.org/html/2607.05866#S4.SS7)\.

### 4\.2\.End\-to\-End Utility Evaluation

Table 2\.Test accuracy \(%\) across three datasets and five privacy budgets\. Results report the mean and standard deviation over 5 independent runs\. Best results are in bold\. All public\-data\-assisted methods use exactly the same auxiliary dataset\.Dataset \(Model\)Methodϵ=1\.0\\epsilon=1\.0ϵ=2\.0\\epsilon=2\.0ϵ=4\.0\\epsilon=4\.0ϵ=6\.0\\epsilon=6\.0ϵ=8\.0\\epsilon=8\.0ACCACCACCACCACCCIFAR\-10DP\-SGD58\.16%±\\pm0\.3565\.71%±\\pm0\.4172\.73%±\\pm0\.4877\.49%±\\pm0\.6580\.42%±\\pm0\.52\(WRN\-16\-4\)DP\-SGD\-PT58\.68%±\\pm0\.3265\.83%±\\pm0\.4973\.06%±\\pm0\.3577\.94%±\\pm0\.6180\.83%±\\pm0\.39GEP58\.75%±\\pm0\.4166\.48%±\\pm0\.3673\.46%±\\pm0\.3078\.24%±\\pm0\.5880\.55%±\\pm0\.25AdaDPS59\.25%±\\pm0\.4566\.94%±\\pm0\.3973\.59%±\\pm0\.5478\.57%±\\pm0\.4980\.62%±\\pm0\.66DP\-NGD \(Ours\)60\.78%±\\pm0\.3867\.75%±\\pm0\.5674\.02%±\\pm0\.4478\.61%±\\pm0\.4981\.09%±\\pm0\.45SVHNDP\-SGD79\.34%±\\pm0\.5883\.96%±\\pm0\.3587\.01%±\\pm0\.4187\.49%±\\pm0\.2888\.60%±\\pm0\.55\(WRN\-16\-4\)DP\-SGD\-PT79\.54%±\\pm0\.4584\.11%±\\pm0\.5287\.37%±\\pm0\.3988\.59%±\\pm0\.4689\.18%±\\pm0\.54GEP79\.73%±\\pm0\.3186\.21%±\\pm0\.3788\.27%±\\pm0\.5488\.84%±\\pm0\.6089\.29%±\\pm0\.58AdaDPS79\.87%±\\pm0\.3586\.12%±\\pm0\.5188\.23%±\\pm0\.3889\.35%±\\pm0\.4489\.98%±\\pm0\.41DP\-NGD \(Ours\)84\.65%±\\pm0\.3788\.42%±\\pm0\.4590\.70%±\\pm0\.5292\.28%±\\pm0\.4892\.67%±\\pm0\.47UTKFaceDP\-SGD57\.53%±\\pm0\.4565\.02%±\\pm0\.4070\.15%±\\pm0\.3573\.10%±\\pm0\.3174\.17%±\\pm0\.58\(ResNet\-20\)DP\-SGD\-PT58\.20%±\\pm0\.4165\.46%±\\pm0\.3670\.32%±\\pm0\.3273\.29%±\\pm0\.5874\.55%±\\pm0\.55GEP58\.49%±\\pm0\.5267\.90%±\\pm0\.4571\.24%±\\pm0\.3874\.43%±\\pm0\.4475\.38%±\\pm0\.53AdaDPS58\.75%±\\pm0\.5568\.49%±\\pm0\.4871\.19%±\\pm0\.4174\.56%±\\pm0\.3775\.58%±\\pm0\.33DP\-NGD \(Ours\)62\.95%±\\pm0\.5570\.01%±\\pm0\.3873\.88%±\\pm0\.4876\.05%±\\pm0\.4576\.22%±\\pm0\.52Table[2](https://arxiv.org/html/2607.05866#S4.T2)summarizes the test accuracies across three standard benchmarks and five privacy budgets\. DP\-NGD consistently achieves the highest performance in every setting\. The advantage is more pronounced under strict privacy constraints: atϵ=1\.0\\epsilon=1\.0, DP\-NGD outperforms the most competitive baseline by4\.78%4\.78\\%on SVHN and4\.20%4\.20\\%on UTKFace\. This aligns with our core insight—when first\-order gradients are heavily corrupted by DP noise, coarse\-grained curvature priors deliver the strongest acceleration benefit\. Asϵ\\epsilonincreases and the signal\-to\-noise ratio improves, the relative gains from second\-order acceleration gradually moderate \(e\.g\., sub\-1%1\\%improvement on CIFAR\-10 atϵ=8\.0\\epsilon=8\.0\)\. This is because, when the noise magnitude is sufficiently low, the residual inaccuracies in curvature estimation from public data become a limiting factor for leveraging precise fine\-grained gradient components\.

Comparing DP\-NGD against AdaDPS and GEP validates the necessity of our block\-diagonal preconditioner\. AdaDPS captures only diagonal variance, while GEP projects gradients into a low\-rank subspace; both methods simplify the geometry and inevitably discard information—cross\-parameter correlations in AdaDPS, and fine\-grained tail features in GEP\. By employing a block\-diagonal preconditioner that operates in the full\-dimensional parameter space, DP\-NGD consistently surpasses both\. The performance of DP\-SGD\-PT further sharpens this picture: simple pre\-training on𝒟pub\\mathcal\{D\}\_\{\\text\{pub\}\}yields only marginal utility gains over standard DP\-SGD and is largely outperformed by methods that actively exploit the geometry of the gradient space\. This empirical gap confirms that the primary value of auxiliary public data lies not in mere feature initialization, but in extracting transferable geometric priors that reshape the optimization dynamics\. Ultimately, these comparisons establish that the structural fidelity of the geometric prior directly dictates the utility gains under differential privacy\.

### 4\.3\.Convergence Efficiency and Speedup

![Refer to caption](https://arxiv.org/html/2607.05866v1/x1.png)Figure 2\.Convergence efficiency vs\. total training steps\. We compare the test accuracy of DP\-NGD, DP\-SGD\-PT, and AdaDPS under a strict privacy budget \(CIFAR\-10,ϵ=1\.0\\epsilon=1\.0\) and a moderate privacy budget \(SVHN,ϵ=6\.0\\epsilon=6\.0\)\. Each point represents an independent full training run with a given total number of steps, where the per\-step noise multiplierσ\\sigmais calibrated to exactly exhaust the target privacy budget at the final step\. DP\-NGD consistently achieves higher accuracy at every step budget, demonstrating superior per\-step optimization efficiency\.Figure[2](https://arxiv.org/html/2607.05866#S4.F2)reports the test accuracy of DP\-NGD, DP\-SGD\-PT, and AdaDPS at varying total training steps under a low privacy budget \(CIFAR\-10,ϵ=1\.0\\epsilon=1\.0\) and a moderate privacy budget \(SVHN,ϵ=6\.0\\epsilon=6\.0\)\. Each point on the horizontal axis represents an independent full training run with a given total number of steps, where the per\-step noise multiplierσ\\sigmais calibrated to exactly exhaust the target privacy budget at the final step\. DP\-NGD consistently achieves higher accuracy than the baselines when evaluated at the same training step\. Moreover, it reaches the peak performance of the baselines with substantially fewer steps\. On CIFAR\-10, DP\-NGD attains59\.75%59\.75\\%at step 360, already exceeding the best accuracy of AdaDPS \(58\.28%58\.28\\%at step 840\)—a2\.3×2\.3\\timesreduction in the steps required to reach that utility level\. On SVHN, DP\-NGD reaches89\.38%89\.38\\%at step 340, surpassing the peak of both DP\-SGD\-PT and AdaDPS \(89\.25%89\.25\\%at step 3400\), corresponding to a10\.0×10\.0\\timesefficiency gain\. This advantage stems from the higher per\-step optimization efficiency of curvature\-aware updates\. By preconditioning gradients with the block\-diagonal inverse curvature, DP\-NGD aligns each update with the dominant geometric directions of the loss landscape\. This effectively suppresses wasteful oscillations in ill\-conditioned terrain and extracting more meaningful signal per noisy step compared to diagonal \(AdaDPS\) or isotropic \(DP\-SGD\-PT\) approaches\.

### 4\.4\.Sample Efficiency of Public Data

![Refer to caption](https://arxiv.org/html/2607.05866v1/x2.png)Figure 3\.Sample efficiency of public auxiliary data\. Test accuracy is reported on the UTKFace dataset \(ϵ=2\.0\\epsilon=2\.0\) across varying public dataset sizes\|𝒟pub\|\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|\. DP\-NGD achieves near\-peak accuracy with very few public samples and remains insensitive to\|𝒟pub\|\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|across the entire range\. In contrast, DP\-SGD\-PT, AdaDPS, and GEP all exhibit a clear dependence on the amount of public data, with their accuracy improving steadily with more data\.Figure[3](https://arxiv.org/html/2607.05866#S4.F3)evaluates the impact of the public dataset size\|𝒟pub\|\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|on model utility using the UTKFace dataset atϵ=2\.0\\epsilon=2\.0\. By varying the number of public samples from5050to50005000, we assess how effectively each method extracts geometric priors from limited auxiliary data\. DP\-NGD exhibits remarkable sample efficiency and a distinct insensitivity to\|𝒟pub\|\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|\. With as few as5050public samples, our method already achieves69\.29%69\.29\\%test accuracy, and this performance fluctuates by less than1%1\\%across the entire evaluated range\. In striking contrast, the baselines show a clear dependence on the volume of public data\. While the accuracy of DP\-SGD\-PT, AdaDPS, and GEP improves steadily as\|𝒟pub\|\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|increases, they consistently underperform DP\-NGD even when utilizing the maximum available50005000samples\.

This stark difference directly reinforces our core claim: coarse\-grained curvature priors are sufficient for effective second\-order acceleration under DP\. The K\-FAC approximation compresses curvature degrees of freedom from𝒪​\(d2\)\\mathcal\{O\}\(d^\{2\}\)to layer\-width dimensions, enabling reliable estimation of the dominant geometric directions from very few public samples\. Although additional public data might refine fine\-grained curvature components, these precise directions are ultimately overwhelmed by DP noise during training, resulting in diminishing returns\. Conversely, baselines relying on diagonal scaling \(AdaDPS\) or low\-rank subspaces \(GEP\) require significantly more samples to construct statistically stable surrogates of the loss geometry\. Furthermore, DP\-SGD\-PT, which exploits public data merely for weight initialization, yields the weakest per\-sample benefit\. Overall, these results confirm that extracting structural geometric priors is a fundamentally more data\-efficient strategy to leverage auxiliary information\.

### 4\.5\.Ablation Studies

Table 3\.Necessity of curvature decoupling on CIFAR\-10 atϵ=2\.0\\epsilon=2\.0and4\.04\.0\. We evaluate the impact of allocating a fraction of the privacy budget \(20%20\\%or50%50\\%\) to private curvature estimation\. Even a modest budget split severely degrades accuracy, dropping performance below the first\-order DP\-SGD\-PT baseline that uses no curvature at all\.Variant Configurationϵ=2\.0\\epsilon=2\.0ϵ=4\.0\\epsilon=4\.0DP\-SGD\-FT \(No Curvature\)65\.86%72\.75%Private DP\-NGD \(ϵ\\epsilonsplit 50/50\)54\.35%61\.23%Private DP\-NGD \(ϵ\\epsilonsplit 20/80\)56\.12%64\.50%Public DP\-NGD \(Decoupled\)67\.78%73\.88%The Necessity of Curvature Decoupling\.We verify the necessity of decoupling curvature estimation from private data by comparing three strategies on CIFAR‑10 atϵ=2\.0\\epsilon=2\.0and4\.04\.0: \(i\) our default DP‑NGD, which estimates the curvature matrix entirely on public data and preserves the full privacy budget for gradient updates; \(ii\) a variant that estimates curvature on private data, allocating a fraction of the privacy budget to the curvature computation and the remainder to gradient updates; and \(iii\) DP‑SGD‑PT, which merely pre‑trains on public data without any curvature estimation\. When20%20\\%of the privacy budget is diverted to private curvature estimation, the accuracy drops dramatically from67\.78%67\.78\\%to56\.12%56\.12\\%atϵ=2\.0\\epsilon=2\.0, and from73\.88%73\.88\\%to64\.50%64\.50\\%atϵ=4\.0\\epsilon=4\.0\. A50%50\\%split worsens the results further, yielding only54\.35%54\.35\\%and61\.23%61\.23\\%, respectively\. In sharp contrast, our public‑data‑driven curvature estimation incurs zero privacy cost and consistently outperforms even the DP‑SGD‑PT baseline \(65\.86%65\.86\\%and72\.75%72\.75\\%\)\. These results expose a fundamental tension: under a fixed privacy budget, splitting that budget between curvature and gradients severely degrades the signal‑to‑noise ratio of the updates that actually drive learning, leading to catastrophic utility collapse\. Decoupling curvature extraction from private data is therefore not merely a convenience but a necessary condition for making second‑order optimization viable under differential privacy\.

Table 4\.Ablation on whitened\-space clipping on CIFAR\-10 \(ϵ∈\{2\.0,4\.0\}\\epsilon\\in\\\{2\.0,4\.0\\\}\)\. We compare our whitened\-space DP operations against two alternatives: pre\-conditioning clipping \(clipping raw gradients before applying curvature\) and post\-conditioning clipping \(clipping the natural gradient directly\)\. Pre\-conditioning suffers from geometric incompatibility, while post\-conditioning distorts the anisotropic descent direction despite being privacy\-compliant\. Whitened\-space clipping uniquely satisfies both the strict DP sensitivity bound and the directional integrity of second\-order updates\.Clipping Strategyϵ=2\.0\\epsilon=2\.0ϵ=4\.0\\epsilon=4\.0Pre\-conditioning Clipping61\.25%66\.82%Post\-conditioning Clipping64\.54%68\.48%Whitened\-Space Clipping \(Ours\)67\.78%73\.88%Ablation on Whitened\-Space Clipping\.We validate the necessity of whitened\-space clipping on CIFAR\-10 \(ϵ∈\{2\.0,4\.0\}\\epsilon\\in\\\{2\.0,4\.0\\\}\) by comparing it against two alternatives: clipping raw gradients before applying curvature \(pre\-conditioning\), and clipping the resulting natural gradients directly \(post\-conditioning\)\. Our method achieves the highest accuracy \(67\.78%67\.78\\%and73\.88%73\.88\\%\), substantially outperforming both pre\-conditioning \(61\.25%61\.25\\%and66\.82%66\.82\\%\) and post\-conditioning \(64\.54%64\.54\\%and68\.48%68\.48\\%\)\. Pre\-conditioning performs worst: isotropicL2L\_\{2\}clipping in the original space treats all gradient directions uniformly, irreversibly corrupting the gradient structure before curvature information can be incorporated\. Post\-conditioning, although privacy\-compliant because the curvature matrix comes from public data, applies an isotropic clip to the already anisotropic natural gradient, thereby distorting its carefully aligned descent direction\. Whitened\-space clipping resolves this tension by performing DP operations in theF−1/2F^\{\-1/2\}\-whitened space, where an isotropicL2L\_\{2\}constraint is mathematically equivalent to the anisotropic Fisher\-norm constraint required by NGD\. This preserves both the strict DP sensitivity bound and the directional integrity of second\-order updates\.

![Refer to caption](https://arxiv.org/html/2607.05866v1/x3.png)Figure 4\.Ablation on the clamping floor using UTKFace \(ϵ=2\.0\\epsilon=2\.0\)\. Training loss trajectories for three strategies are shown\. Training diverges rapidly without clamping, while both clamped strategies converge stably\. Our dynamic schedule further relaxes the floor to avoid over\-suppressing well\-conditioned eigenvalues, yielding lower loss in high\-SNR mid\-phase and higher final accuracy\.Ablation on the Clamping Floor\.We evaluate the impact of the dynamic clamping schedule on training stability and final utility using UTKFace \(ϵ=2\.0\\epsilon=2\.0\)\. We compare three configurations: \(i\) our proposed dynamic schedule, \(ii\) a fixed clamping floor set to the theoretical safety boundλsafe\\lambda\_\{\\text\{safe\}\}, and \(iii\) no clamping floor at all\. Training without any clamping diverges rapidly within the first few steps, producing NaN values and confirming that bounding only theL2L\_\{2\}\-norm in the whitened space is insufficient to guarantee Euclidean stability of the resulting natural gradient\. Both the fixedλsafe\\lambda\_\{\\text\{safe\}\}and our dynamic schedule converge stably\. In the early phase \(<200<200steps\), when curvature estimates are unreliable, the fixedλsafe\\lambda\_\{\\text\{safe\}\}yields a slightly lower training loss\. However, as training stabilizes and enters the high\-SNR mid\-phase, the fixed floor becomes overly conservative\. The loss curve of our dynamic schedule consistently lies below that of the fixedλsafe\\lambda\_\{\\text\{safe\}\}from the mid\-phase onward, indicating that a static floor persistently lifts eigenvalues that are already well\-conditioned, needlessly sacrificing curvature accuracy and suppressing the second\-order acceleration\. Our dynamic schedule avoids this by relaxing the floor toward a low base value, achieving consistently lower training loss and a markedly higher final test accuracy \(70\.06%70\.06\\%vs\.68\.53%68\.53\\%\)\.

### 4\.6\.End\-to\-End System Overhead

Table 5\.End\-to\-end system overhead comparison between DP\-NGD and DP\-SGD\-PT\. Despite a higher per\-step cost, DP\-NGD achieves comparable or lower total runtime due to sharply fewer steps\. All runs use a single NVIDIA RTX 3090\.Dataset\(ϵ\\epsilon\)MethodPer\-step Time\(s\)Total StepsTotal Time\(Hrs\)CIFAR\-10\(ϵ=1\.0\\epsilon=1\.0\)DP\-SGD\-PT14\.28403\.31DP\-NGD38\.23603\.82SVHN\(ϵ=6\.0\\epsilon=6\.0\)DP\-SGD\-PT13\.6340012\.84DP\-NGD39\.83403\.76Table[5](https://arxiv.org/html/2607.05866#S4.T5)reports the per\-step wall\-clock time, the number of training steps required to reach the peak accuracy of DP\-SGD\-PT, and the resulting total wall\-clock time for both methods on two representative benchmarks\. All measurements are performed on a single NVIDIA GeForce RTX 3090 GPU under identical batch size, augmentation, and privacy configurations\. DP\-NGD incurs a higher per\-step cost than DP\-SGD\-PT \(3838–4040s vs\.1414s\), primarily due to the whitened\-space projection and the periodic curvature estimation\. However, this overhead is substantially offset by the sharply reduced number of steps resulting from curvature\-aware optimization\. On CIFAR\-10 \(ϵ=1\.0\\epsilon=1\.0\), DP\-NGD reaches the peak accuracy of DP\-SGD\-PT in only360360steps, compared to840840steps for the baseline\. The total runtime is3\.823\.82hours, comparable to that of DP\-SGD\-PT \(3\.313\.31hours\), while achieving higher final accuracy\. On SVHN \(ϵ=6\.0\\epsilon=6\.0\), the advantage is dramatic: DP\-NGD requires only340340steps to match the baseline accuracy that demands3,4003\{,\}400steps for DP\-SGD\-PT\. The total runtime drops from12\.8412\.84hours to3\.763\.76hours—a3\.4×3\.4\\timesspeedup\. This stark contrast shows that in ill\-conditioned optimization landscapes, the ability of curvature\-aware preconditioning to suppress wasteful oscillations far outweighs its per\-step computational cost\. Overall, these results confirm that DP\-NGD offers practical accelerator for differentially private training\.

### 4\.7\.Hyperparameter Robustness

Table 6\.Hyperparameter sensitivity on UTKFace \(ϵ=2\.0\\epsilon=2\.0\)\. We evaluate the robustness of DP\-NGD to the curvature update intervalTcurT\_\{\\text\{cur\}\}, the polynomial exponentpp, and the transition step ratioT1T\_\{1\}\. Test accuracy remains stable across all three parameters, with fluctuations within a narrow range\. Default values are underlined\.\(a\) Sensitivity to Curvature Update Interval\(Tc​u​rT\_\{cur\}\)Value1481632Accuracy70\.14%70\.02%70\.05%69\.87%67\.53%
\(b\) Sensitivity to Polynomial Exponent\(pp\)Value2351015Accuracy68\.96%69\.21%69\.08%70\.05%69\.63%
\(c\) Sensitivity to Transition Step\(T1T\_\{1\}ratio\)Value0\.050\.10\.150\.20\.25Accuracy69\.51%70\.05%69\.25%69\.46%69\.26%

Table[6](https://arxiv.org/html/2607.05866#S4.T6)evaluates the sensitivity of DP\-NGD to its three key hyperparameters on UTKFace \(ϵ=2\.0\\epsilon=2\.0\): the curvature update intervalTcurT\_\{\\text\{cur\}\}, the polynomial exponentpp, and the transition step ratioT1T\_\{1\}\. Across all three parameters, the test accuracy remains stable within a narrow range, confirming the robustness of our framework to these design choices\. The curvature update intervalTcurT\_\{\\text\{cur\}\}can be extended from11to1616steps with less than a0\.3%0\.3\\%accuracy drop; only at an extremely sparse update frequency of3232steps does accuracy decline noticeably to67\.53%67\.53\\%, still a competitive result\. The polynomial exponentppand the transition ratioT1T\_\{1\}exhibit similar insensitivity: varyingppfrom22to1515orT1T\_\{1\}from0\.050\.05to0\.250\.25induces fluctuations of less than1%1\\%\. This robustness reflects the underlying stability of coarse\-grained geometric priors: once the dominant eigen\-directions of the curvature are captured from public data, the precise details of the dynamic clamping schedule exert only a marginal influence on optimization dynamics\. The high tolerance to these hyperparameters significantly reduces the tuning burden in practical deployments\.

## 5\.Conclusion

We proposed DP\-NGD, a practical framework that enables effective second\-order optimization under differential privacy\. By decoupling curvature estimation from private data, reconciling isotropic DP operations with anisotropic natural gradient updates through a whitened\-space mechanism, and dynamically clamping curvature eigenvalues, DP\-NGD systematically resolves the three fundamental obstacles that have so far prevented the integration of NGD and DP\. Extensive experiments demonstrate that DP\-NGD consistently achieves state\-of\-the\-art accuracy across multiple benchmarks and privacy budgets, breaking through the utility ceilings of first\-order baselines with up to10×10\\timesconvergence speedup\. Notably, these gains are obtained with a trivially small public auxiliary dataset and without incurring additional privacy cost\. Our work opens up several promising directions, including extending coarse\-grained curvature priors to large\-scale models, relaxing the reliance on public data through fully private geometric estimation, and exploring curvature\-aware optimization under other privacy notions\.

## Appendix ADerivation of the Whitening MatrixW=F−1/2W=F^\{\-1/2\}

To reconcile this conflict between DP and NGD, we exploit two critical property: \(1\) the DP privacy guarantee holds strictly as long as the sensitivity is bounded under theL2L\_\{2\}norm inanycoordinate space; \(2\) The acceleration of NGD requires only the final parameter updates to align with the local loss geometry\. This insight motivates a natural strategy: transfer all DP operations into an intermediate projected space where the isotropicL2L\_\{2\}constraint can be safely applied, and then map the result back to the original parameter space so that the final update recovers the curvature\-aligned geometry of NGD\.

We now formalize the two geometric requirements\. First, NGD constrains the per\-step distributional shift via the KL divergence\. Under the standard second\-order Taylor expansion and the NGD update ruleΔ​θ=−η​F−1​g\\Delta\\theta=\-\\eta F^\{\-1\}g, the KL constraint reduces to a bound on theFisher normof the gradient:

\(19\)‖g‖F−12=g⊤​F−1​g\.\\\|g\\\|\_\{F^\{\-1\}\}^\{2\}=g^\{\\top\}F^\{\-1\}g\.
Second, DP requires that the update vector used for clipping and noising has a boundedL2L\_\{2\}norm\. In a projected space defined by a linear mapWW, theL2L\_\{2\}norm of the projected gradient is

\(20\)‖W​g‖22=g⊤​\(W⊤​W\)​g\.\\\|Wg\\\|\_\{2\}^\{2\}=g^\{\\top\}\(W^\{\\top\}W\)g\.
To guarantee that the isotropic DP operations in the projected space do not corrupt the curvature\-aware information, theL2L\_\{2\}geometry of the projected space must be intrinsically equivalent to the KL geometry of the original space\. Equating the quadratic forms in Eqs\. \([19](https://arxiv.org/html/2607.05866#A1.E19)\) and \([20](https://arxiv.org/html/2607.05866#A1.E20)\) yields the fundamental condition

\(21\)W⊤​W=F−1\.W^\{\\top\}W=F^\{\-1\}\.This equation demands a matrixWWwhose symmetric product recovers the inverse Fisher\. Since the Fisher matrixFFand its inverseF−1F^\{\-1\}are symmetric and positive semi\-definite, the unique symmetric positive semi\-definite solution to Eq\. \([21](https://arxiv.org/html/2607.05866#A1.E21)\) is the inverse square root ofFF:

The derivation reveals why theF−1/2F^\{\-1/2\}\-whitened space is the unique geometry that aligns DP and NGD\. In this space, bounding theL2L\_\{2\}norm of the projected gradient‖F−1/2​g‖2\\\|F^\{\-1/2\}g\\\|\_\{2\}is mathematically identical to bounding the Fisher norm‖g‖F−1\\\|g\\\|\_\{F^\{\-1\}\}, which in turn is equivalent to respecting the NGD trust region\. Consequently, performing standard isotropic DP clipping and noising in theF−1/2F^\{\-1/2\}\-whitened space, and then mapping back viaF1/2F^\{1/2\}, automatically yields an anisotropic, curvature\-aware update in the original parameter space\. This provides the theoretical foundation for the KL\-DP duality\.

#### Efficient computation via K\-FAC\.

In deep networks, constructing the fullF−1/2F^\{\-1/2\}would be prohibitive\. The Kronecker\-Factored Approximate Curvature \(K\-FAC\) resolves this by factorizing each layer’s Fisher block asFl≈Al⊗GlF\_\{l\}\\approx A\_\{l\}\\otimes G\_\{l\}\. The whitening matrix can then be efficiently computed via eigendecomposition of the small Kronecker factors:

Fl−1/2≈\(QA​ΛA−1/2​QA⊤\)⊗\(QG​ΛG−1/2​QG⊤\),F\_\{l\}^\{\-1/2\}\\approx\(Q\_\{A\}\\Lambda\_\{A\}^\{\-1/2\}Q\_\{A\}^\{\\top\}\)\\otimes\(Q\_\{G\}\\Lambda\_\{G\}^\{\-1/2\}Q\_\{G\}^\{\\top\}\),reducing the complexity from𝒪​\(d3\)\\mathcal\{O\}\(d^\{3\}\)to𝒪​\(din3\+dout3\)\\mathcal\{O\}\(d\_\{\\text\{in\}\}^\{3\}\+d\_\{\\text\{out\}\}^\{3\}\), wheredin,dout≪dd\_\{\\text\{in\}\},d\_\{\\text\{out\}\}\\ll d\. This makes the whitened\-space update computationally tractable\.

## Appendix BOptimal Hyperparameter Configurations

Table[7](https://arxiv.org/html/2607.05866#A2.T7)lists the best hyperparameter configurations identified via grid search for DP\-NGD across all datasets and privacy budgets reported in the main text\. These settings correspond to the results presented in Table[2](https://arxiv.org/html/2607.05866#S4.T2)\.

Table 7\.Optimal learning rates and total training steps for DP\-NGD\. All configurations use a fixed batch size ofB=4096B=4096, curvature update intervalTcur=8T\_\{\\text\{cur\}\}=8, warmup fractionT1=0\.1T\_\{1\}=0\.1, polynomial exponentp=10p=10, and auxiliary public dataset size\|𝒟pub\|=500\|\\mathcal\{D\}\_\{\\text\{pub\}\}\|=500\.Dataset \(Model\)ϵ\\epsilonTotal StepsTTLearning Rateη\\etaCIFAR\-10 \(WRN\-16\-4\)1\.08400\.012\.013200\.014\.019200\.016\.018000\.028\.026400\.02SVHN \(WRN\-16\-4\)1\.013600\.012\.018700\.014\.020400\.016\.023800\.028\.034000\.02UTKFace \(ResNet\-20\)1\.05500\.012\.08500\.014\.09000\.026\.011000\.028\.017000\.02
## References

- \(1\)
- Abadi et al\.\(2016\)Martín Abadi, Andy Chu, Ian J\. Goodfellow, H\. B\. McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang\. 2016\.Deep Learning with Differential Privacy\.*Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security*\(2016\)\.[https://api\.semanticscholar\.org/CorpusID:207241585](https://api.semanticscholar.org/CorpusID:207241585)
- Abouelnaga et al\.\(2016\)Yehya Abouelnaga, Ola S Ali, Hager Rady, and Mohamed Moustafa\. 2016\.Cifar\-10: Knn\-based ensemble of classifiers\. In*2016 International Conference on Computational Science and Computational Intelligence \(CSCI\)*\. IEEE, 1192–1195\.
- Ahn and Cutkosky \(2024\)Kwangjun Ahn and Ashok Cutkosky\. 2024\.Adam with model exponential moving average is effective for nonconvex optimization\.*ArXiv*abs/2405\.18199 \(2024\)\.[https://api\.semanticscholar\.org/CorpusID:270068108](https://api.semanticscholar.org/CorpusID:270068108)
- Babu et al\.\(2024\)Bachina Harish Babu, V\. R\. Vinothini, Gautam Solaimalai, R\. Vanitha, Ajay Singh Yadav, P\. Sukania, Vishal Vijayan, and Revanth Srinivasan\. 2024\.Theoretical optimization of group size in group normalization for enhanced deep neural network training\.*AIP Conference Proceedings*\(2024\)\.[https://api\.semanticscholar\.org/CorpusID:274008853](https://api.semanticscholar.org/CorpusID:274008853)
- Bao et al\.\(2023\)Wenxuan Bao, F\. Pittaluga, Vijay Kumar, and Vincent Bindschaedler\. 2023\.DP\-Mix: Mixup\-based Data Augmentation for Differentially Private Learning\.*ArXiv*abs/2311\.01295 \(2023\)\.[https://api\.semanticscholar\.org/CorpusID:264935065](https://api.semanticscholar.org/CorpusID:264935065)
- Block et al\.\(2024\)Adam Block, Mark Bun, Rathin Desai, Abhishek Shetty, and Zhiwei Steven Wu\. 2024\.Oracle\-Efficient Differentially Private Learning with Public Data\.*ArXiv*abs/2402\.09483 \(2024\)\.[https://api\.semanticscholar\.org/CorpusID:267682343](https://api.semanticscholar.org/CorpusID:267682343)
- Boix\-Adserà \(2025\)Enric Boix\-Adserà\. 2025\.On the inductive bias of infinite\-depth ResNets and the bottleneck rank\.*ArXiv*abs/2501\.19149 \(2025\)\.[https://api\.semanticscholar\.org/CorpusID:276079304](https://api.semanticscholar.org/CorpusID:276079304)
- Bu et al\.\(2022\)Zhiqi Bu, Yu\-Xiang Wang, Sheng Zha, and George Karypis\. 2022\.Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger\.*ArXiv*abs/2206\.07136 \(2022\)\.[https://api\.semanticscholar\.org/CorpusID:249674655](https://api.semanticscholar.org/CorpusID:249674655)
- Chandaliya et al\.\(2019\)Praveen Kumar Chandaliya, V\.Chandra Kumar, Mayank Harjani, and Neeta Nain\. 2019\.SCDAE: Ethnicity and Gender Alteration on CLF and UTKFace Dataset\. In*International Conference on Computer Vision and Image Processing*\.[https://api\.semanticscholar\.org/CorpusID:214730428](https://api.semanticscholar.org/CorpusID:214730428)
- Chaudhari et al\.\(2016\)Pratik Chaudhari, Anna Choromańska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Tour Chayes, Levent Sagun, and Riccardo Zecchina\. 2016\.Entropy\-SGD: biasing gradient descent into wide valleys\.*Journal of Statistical Mechanics: Theory and Experiment*2019 \(2016\)\.[https://api\.semanticscholar\.org/CorpusID:13807351](https://api.semanticscholar.org/CorpusID:13807351)
- Dangel et al\.\(2024\)Felix Dangel, Johannes Müller, and Marius Zeinhofer\. 2024\.Kronecker\-factored approximate curvature for physics\-informed neural networks\.*Advances in Neural Information Processing Systems*37 \(2024\), 34582–34636\.
- De et al\.\(2022\)Soham De, Leonard Berrada, Jamie Hayes, Samuel L\. Smith, and Borja Balle\. 2022\.Unlocking High\-Accuracy Differentially Private Image Classification through Scale\.*ArXiv*abs/2204\.13650 \(2022\)\.[https://api\.semanticscholar\.org/CorpusID:248427073](https://api.semanticscholar.org/CorpusID:248427073)
- Dong et al\.\(2022\)Wei Dong, Yuting Liang, and Ke Yi\. 2022\.Differentially Private Covariance Revisited\.*ArXiv*abs/2205\.14324 \(2022\)\.[https://api\.semanticscholar\.org/CorpusID:249191345](https://api.semanticscholar.org/CorpusID:249191345)
- Dwork \(2006\)Cynthia Dwork\. 2006\.Differential Privacy\. In*International Colloquium on Automata, Languages and Programming*\.[https://api\.semanticscholar\.org/CorpusID:2565493](https://api.semanticscholar.org/CorpusID:2565493)
- Dwork and Roth \(2014\)Cynthia Dwork and Aaron Roth\. 2014\.The Algorithmic Foundations of Differential Privacy\.*Found\. Trends Theor\. Comput\. Sci\.*9 \(2014\), 211–407\.[https://api\.semanticscholar\.org/CorpusID:207178262](https://api.semanticscholar.org/CorpusID:207178262)
- Feng et al\.\(2024\)Shuya Feng, Meisam Mohammady, Hanbin Hong, Shenao Yan, Ashish Kundu, Binghui Wang, and Yuan Hong\. 2024\.Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergence\.*Proceedings of the Fifteenth ACM Conference on Data and Application Security and Privacy*\(2024\)\.[https://api\.semanticscholar\.org/CorpusID:278238496](https://api.semanticscholar.org/CorpusID:278238496)
- Foret et al\.\(2020\)Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur\. 2020\.Sharpness\-Aware Minimization for Efficiently Improving Generalization\.*ArXiv*abs/2010\.01412 \(2020\)\.[https://api\.semanticscholar\.org/CorpusID:222134093](https://api.semanticscholar.org/CorpusID:222134093)
- Gopi et al\.\(2021\)Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz\. 2021\.Numerical Composition of Differential Privacy\. In*Neural Information Processing Systems*\.[https://api\.semanticscholar\.org/CorpusID:235358383](https://api.semanticscholar.org/CorpusID:235358383)
- Greenacre et al\.\(2003\)Michael Greenacre, Patrick J\. F\. Groenen, Trevor J\. Hastie, Alfonso Iodice D’Enza, Angelos I\. Markos, and Elena Tuzhilina\. 2003\.Principal Component Analysis\.*Technometrics*45 \(2003\), 276 – 276\.[https://api\.semanticscholar\.org/CorpusID:2534141](https://api.semanticscholar.org/CorpusID:2534141)
- Grosse and Martens \(2016\)Roger Grosse and James Martens\. 2016\.A kronecker\-factored approximate fisher matrix for convolution layers\. In*International Conference on Machine Learning*\. PMLR, 573–582\.
- Guth and M’enard \(2024\)Florentin Guth and Brice M’enard\. 2024\.On the universality of neural encodings in CNNs\.*ArXiv*abs/2409\.19460 \(2024\)\.[https://api\.semanticscholar\.org/CorpusID:272987466](https://api.semanticscholar.org/CorpusID:272987466)
- Huo et al\.\(2020\)Tianjiao Huo, Jiaqi Fan, Xin Li, Hong Chen, Bingzhao Gao, and Xuesong Li\. 2020\.Traffic Sign Recognition Based on ResNet\-20 and Deep Mutual Learning\.*2020 Chinese Automation Congress \(CAC\)*\(2020\), 4770–4774\.[https://api\.semanticscholar\.org/CorpusID:231737679](https://api.semanticscholar.org/CorpusID:231737679)
- Jolliffe and Cadima \(2016\)Ian T\. Jolliffe and Jorge Cadima\. 2016\.Principal component analysis: a review and recent developments\.*Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences*374 \(2016\)\.[https://api\.semanticscholar\.org/CorpusID:20101754](https://api.semanticscholar.org/CorpusID:20101754)
- Kudo and Tajima \(2022\)Daigo Kudo and Hiroyasu Tajima\. 2022\.Fisher information matrix as a resource measure in the resource theory of asymmetry with general connected\-Lie\-group symmetry\.*Physical Review A*\(2022\)\.[https://api\.semanticscholar\.org/CorpusID:248562504](https://api.semanticscholar.org/CorpusID:248562504)
- Kunstner et al\.\(2019\)Frederik Kunstner, Philipp Hennig, and Lukas Balles\. 2019\.Limitations of the empirical fisher approximation for natural gradient descent\.*Advances in neural information processing systems*32 \(2019\)\.
- Li et al\.\(2022\)Tian Li, Manzil Zaheer, Sashank J\. Reddi, and Virginia Smith\. 2022\.Private Adaptive Optimization with Side Information\. In*International Conference on Machine Learning*\.[https://api\.semanticscholar\.org/CorpusID:246823521](https://api.semanticscholar.org/CorpusID:246823521)
- Lowy et al\.\(2023\)Andrew Lowy, Zeman Li, Tianjian Huang, and Meisam Razaviyayn\. 2023\.Optimal Differentially Private Learning with Public Data\.*ArXiv*abs/2306\.15056 \(2023\)\.[https://api\.semanticscholar\.org/CorpusID:275750699](https://api.semanticscholar.org/CorpusID:275750699)
- Madhulika and Sampath \(2022\)PSS Madhulika and Nalini Sampath\. 2022\.An application of normalizer free neural networks on the SVHN dataset\. In*2022 International conference on applied artificial intelligence and computing \(ICAAIC\)*\. IEEE, 238–242\.
- Martens \(2020\)James Martens\. 2020\.New insights and perspectives on the natural gradient method\.*Journal of Machine Learning Research*21, 146 \(2020\), 1–76\.
- Mehmeti\-Göpel and Wand \(2025\)Christian H\. X\. Ali Mehmeti\-Göpel and Michael Wand\. 2025\.ResNets Are Deeper Than You Think\.*ArXiv*abs/2506\.14386 \(2025\)\.[https://api\.semanticscholar\.org/CorpusID:279410014](https://api.semanticscholar.org/CorpusID:279410014)
- Meiser and Mohammadi \(2018\)Sebastian Meiser and Esfandiar Mohammadi\. 2018\.Tight on Budget?: Tight Bounds for r\-Fold Approximate Differential Privacy\.*Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security*\(2018\)\.[https://api\.semanticscholar\.org/CorpusID:52504941](https://api.semanticscholar.org/CorpusID:52504941)
- Mironov \(2017\)Ilya Mironov\. 2017\.Rényi Differential Privacy\.*2017 IEEE 30th Computer Security Foundations Symposium \(CSF\)*\(2017\), 263–275\.[https://api\.semanticscholar\.org/CorpusID:9386213](https://api.semanticscholar.org/CorpusID:9386213)
- Mironov et al\.\(2019\)Ilya Mironov, Kunal Talwar, and Li Zhang\. 2019\.Rényi Differential Privacy of the Sampled Gaussian Mechanism\.*ArXiv*abs/1908\.10530 \(2019\)\.[https://api\.semanticscholar\.org/CorpusID:201660159](https://api.semanticscholar.org/CorpusID:201660159)
- Morales\-Brotons et al\.\(2024\)Daniel Morales\-Brotons, Thijs Vogels, and Hadrien Hendrikx\. 2024\.Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits\.*Trans\. Mach\. Learn\. Res\.*2024 \(2024\)\.[https://api\.semanticscholar\.org/CorpusID:270355881](https://api.semanticscholar.org/CorpusID:270355881)
- Pajarinen et al\.\(2019\)Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann\. 2019\.Compatible natural gradient policy search\.*Machine Learning*108 \(2019\), 1443 – 1466\.[https://api\.semanticscholar\.org/CorpusID:59842948](https://api.semanticscholar.org/CorpusID:59842948)
- Rodriguez et al\.\(2013\)Carl L Rodriguez, Benjamin Farr, Will M Farr, and Ilya Mandel\. 2013\.Inadequacies of the Fisher information matrix in gravitational\-wave parameter estimation\.*Physical Review D—Particles, Fields, Gravitation, and Cosmology*88, 8 \(2013\), 084013\.
- Tang et al\.\(2023\)Xinyu Tang, Ashwinee Panda, Vikash Sehwag, and Prateek Mittal\. 2023\.Differentially Private Image Classification by Learning Priors from Random Processes\.*J\. Priv\. Confidentiality*15 \(2023\)\.[https://api\.semanticscholar\.org/CorpusID:259129508](https://api.semanticscholar.org/CorpusID:259129508)
- Venkatesan et al\.\(2016\)Ragav Venkatesan, Vijetha Gattupalli, and Baoxin Li\. 2016\.On the generality of neural image features\.*2016 IEEE International Conference on Image Processing \(ICIP\)*\(2016\), 41–45\.[https://api\.semanticscholar\.org/CorpusID:9264808](https://api.semanticscholar.org/CorpusID:9264808)
- Wang et al\.\(2022\)Hua Wang, Sheng yang Gao, Huanyu Zhang, Milan Shen, and Weijie Su\. 2022\.Edgeworth Accountant: An Analytical Approach to Differential Privacy Composition\.*J\. Amer\. Statist\. Assoc\.*\(2022\)\.[https://api\.semanticscholar\.org/CorpusID:249538149](https://api.semanticscholar.org/CorpusID:249538149)
- Wang et al\.\(2018\)Yu\-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan\. 2018\.Subsampled Rényi Differential Privacy and Analytical Moments Accountant\. In*International Conference on Artificial Intelligence and Statistics*\.[https://api\.semanticscholar\.org/CorpusID:51893629](https://api.semanticscholar.org/CorpusID:51893629)
- Wu and He \(2018\)Yuxin Wu and Kaiming He\. 2018\.Group normalization\. In*Proceedings of the European conference on computer vision \(ECCV\)*\. 3–19\.
- Xu et al\.\(2024\)Haotian Xu, Junyu Xuan, Guangquan Zhang, and Jie Lu\. 2024\.Trust region policy optimization via entropy regularization for Kullback\-Leibler divergence constraint\.*Neurocomputing*589 \(2024\), 127716\.[https://api\.semanticscholar\.org/CorpusID:269195555](https://api.semanticscholar.org/CorpusID:269195555)
- Yu et al\.\(2021\)Da Yu, Huishuai Zhang, Wei Chen, and Tie\-Yan Liu\. 2021\.Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning\.*ArXiv*abs/2102\.12677 \(2021\)\.[https://api\.semanticscholar\.org/CorpusID:232046284](https://api.semanticscholar.org/CorpusID:232046284)
- Zhu et al\.\(2022\)Keyu Zhu, Ferdinando Fioretto, and Pascal Van Hentenryck\. 2022\.Post\-processing of Differentially Private Data: A Fairness Perspective\.*ArXiv*abs/2201\.09425 \(2022\)\.[https://api\.semanticscholar\.org/CorpusID:246240328](https://api.semanticscholar.org/CorpusID:246240328)
- Zhu et al\.\(2020\)Keyu Zhu, Pascal Van Hentenryck, and Ferdinando Fioretto\. 2020\.Bias and Variance of Post\-processing in Differential Privacy\.*ArXiv*abs/2010\.04327 \(2020\)\.[https://api\.semanticscholar\.org/CorpusID:222272017](https://api.semanticscholar.org/CorpusID:222272017)