The Fast Mixing Mechanism for Differential Privacy

# The Fast Mixing Mechanism for Differential Privacy
Source: [https://arxiv.org/html/2605.30600](https://arxiv.org/html/2605.30600)
###### Abstract

Randomized sketching is a central tool for compressing large\-scale optimization problems while preserving accuracy\. In particular, sketches that are based on structured matrices, such as the Hadamard matrix, can be applied efficiently and often yield solutions that approximate those of the original problem at much lower computational cost\. In differential privacy \(DP\), Gaussian sketching has been used to solve DP linear regression, beginning withSheffet \([2017](https://arxiv.org/html/2605.30600#bib.bib135),[2019](https://arxiv.org/html/2605.30600#bib.bib137)\)and later refined byLevet al\.\([2025](https://arxiv.org/html/2605.30600#bib.bib56),[2026](https://arxiv.org/html/2605.30600#bib.bib149)\)\. However, although these methods achieve strong utility guarantees, they usually do not improve runtime over classical DP approaches\. In this work, we introduce a new DP sketching mechanism based on fast transforms, which, in certain cases, matches the runtime of classical fast sketching methods\. We prove state\-of\-the\-art privacy guarantees for this mechanism and show that, in favorable regimes, they match those of the Gaussian sketch up to a constant factor\. As an application, we combine this mechanism with recent sketch\-based methods for DP linear regression to obtain a new algorithm with strong utility and improved runtime\. We establish privacy and accuracy guarantees for this algorithm, yielding, to the best of our knowledge, the first fast method for DP ordinary least squares\.

Omri Lev†Moshe Shenfeld⋆,‡Vishwak Srinivasan†,‡Katrina Ligett⋆Ashia C\. Wilson††Massachusetts Institute of Technology⋆The Hebrew University of Jerusalem
††footnotetext:Correspondence:omrilev@mit\.edu\. Code available at[https://github\.com/omrilev1/FastMix](https://github.com/omrilev1/FastMix)\.$\\ddagger$$\\ddagger$footnotetext:Equal contribution\.## 1Introduction

Modern technologies to collect and process data have led to the development of large\-scale datasets that require computationally efficient methods to analyze them\. In recent years,*randomized sketching*has proven to be an attractive approach for designing computationally efficient data analysis tools; seeWoodruff \([2014](https://arxiv.org/html/2605.30600#bib.bib140)\)for an overview\. Roughly speaking, these sketching methods transform the data via random projections, and the data analysis is conducted on this transformed data\. These random projections are developed with two key goals in mind: \(a\) reducing the size of the data, and consequently the computational cost of the data analysis, and \(b\) preserving essential structures of the original data, therefore ensuring that the conclusion of the data analysis does not differ vastly from the original data analysis\. This motivates the development of random projections that address both desiderata, and there has been significant progress made towards this recently\(Drineaset al\.,[2006](https://arxiv.org/html/2605.30600#bib.bib31); Sarlos,[2006](https://arxiv.org/html/2605.30600#bib.bib65); Ailon and Liberty,[2009](https://arxiv.org/html/2605.30600#bib.bib76),[2013](https://arxiv.org/html/2605.30600#bib.bib79); Drineaset al\.,[2011](https://arxiv.org/html/2605.30600#bib.bib9); Pilanci and Wainwright,[2015](https://arxiv.org/html/2605.30600#bib.bib132),[2016](https://arxiv.org/html/2605.30600#bib.bib133),[2017](https://arxiv.org/html/2605.30600#bib.bib59); Cohenet al\.,[2016](https://arxiv.org/html/2605.30600#bib.bib12); Avronet al\.,[2017](https://arxiv.org/html/2605.30600#bib.bib1); Derezinskiet al\.,[2021](https://arxiv.org/html/2605.30600#bib.bib151)\)\.

Simultaneously, with the advent of these large datasets — particularly those built using personal data, there has been increased attention towards preserving the privacy of the participating individuals\. Differential privacy \(DP\)\(Dworket al\.,[2006](https://arxiv.org/html/2605.30600#bib.bib45)\)provides a rigorous theoretical framework for the development of mechanisms that provably protect privacy, and DP mechanisms have been deployed in practice of late\(Research,[2017](https://arxiv.org/html/2605.30600#bib.bib127); Bureau,[2019](https://arxiv.org/html/2605.30600#bib.bib38); Research,[2020](https://arxiv.org/html/2605.30600#bib.bib123); Security,[2022](https://arxiv.org/html/2605.30600#bib.bib122)\)\. Here, the tradeoff to keep in mind is between preserving privacy and the quality of the data analysis as compared to the non\-private data analysis; for instance, a completely garbled output achieves complete privacy, but fundamentally disregards the data analysis problem\.

A relatively underexplored direction is to view the randomness of randomized sketching as a means of providing DP\. A notable example is the work ofBlockiet al\.\([2012](https://arxiv.org/html/2605.30600#bib.bib40)\), which proved that random projections can provide differential privacy, and the work ofSheffet \([2017](https://arxiv.org/html/2605.30600#bib.bib135),[2019](https://arxiv.org/html/2605.30600#bib.bib137)\), which builds on this work and demonstrates how Gaussian sketching can be applied to private linear regression\. More recent papers\(Liet al\.,[2025](https://arxiv.org/html/2605.30600#bib.bib24); Levet al\.,[2025](https://arxiv.org/html/2605.30600#bib.bib56),[2026](https://arxiv.org/html/2605.30600#bib.bib149)\)refine the privacy analysis for Gaussian sketching, develop improved algorithms for private linear regression, and demonstrate gains over standard baselines both theoretically and empirically\. Yet these methods do not provide the main computational benefit usually associated with sketching\. Specifically, if𝖲∈ℝk×n\\mathsf\{S\}\\in\\mathbb\{R\}^\{k\\times n\}is a dense Gaussian matrix andX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}, computing𝖲​X\\mathsf\{S\}XcostsO​\(k​n​d\)O\(knd\)operations, often comparable to solving the original problem\. In contrast, modern non\-private sketching methods are often built using sparse or structured transforms that are both computationally fast and preserve structures of the data, such as the subsampled randomized Hadamard transform \(SRHT\)\. Based on the above discussion, our work is motivated by the natural hypothesis:

*Can we design a fast DP mechanism that preserves properties essential for data analysis?*

This question was first studied byUpadhyay \([2014](https://arxiv.org/html/2605.30600#bib.bib57)\), who proposed a DP sketching mechanism that can be applied significantly faster than the Gaussian sketch\. However, the performance of their method is usually unsatisfactory; while their mechanism works in principle, we demonstrate that it is impractical for typical privacy regimes, incurring a substantial slack relative to the privacy guarantees of the Gaussian sketch\. Thus, from a practical standpoint, this question remains open\.

### 1\.1Our Contributions

We introduce a new family of fast DP sketches, which closes a gap between non\-private fast sketching methods and recently studied private sketches\. Our fast DP sketching mechanism \(presented in Section[3](https://arxiv.org/html/2605.30600#S3)\) applies two sketches, first a large fast sketch that compresses the dataset, and second a small Gaussian sketch which provides DP\. This approach is analogous to prior works on non\-private fast sketching for approximate matrix multiplication \(seeCohenet al\.\([2016](https://arxiv.org/html/2605.30600#bib.bib12), Appendix A\.3\)for example\)\. For an input matrixX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}and a target sketch sizekk, this scheme is faster than the dense Gaussian sketch by a factorO~​\(min⁡\{nk,k\}\)\\widetilde\{O\}\\left\(\\min\\left\\\{\\frac\{n\}\{k\},k\\right\\\}\\right\)\. To show that this is indeed private, we provide Rényi\-DP \(a stronger notion of DP\) guarantee for our proposed scheme\.

We then apply our mechanism to DP ordinary least squares \(DP\-OLS\)\. Instantiating it with SRHT sketches, we derive privacy and accuracy guarantees that explicitly quantify the price of fast, private sketching relative to non\-fast constructions\. For well\-conditioned design matrices, these guarantees match those of state\-of\-the\-art slow methods for DP\-OLS\. Experiments on large\-scale linear regression datasets show that our method achieves state\-of\-the\-art running times while maintaining accuracy close to non\-fast baselines\. In contrast,Upadhyay \([2014](https://arxiv.org/html/2605.30600#bib.bib57)\)do not apply their fast sketching mechanism to DP\-OLS, nor do they provide accuracy guarantees comparable to those of state\-of\-the\-art DP\-OLS methods\. Thus, to our knowledge, our work provides the first fast algorithm for DP\-OLS\.

## 2Problem Setup and Background

##### Notation\.

Random variables are in sans\-serif \(e\.g\.,𝖷,𝗒\\mathsf\{X\},\\mathsf\{y\}\), and their realizations in serif \(e\.g\.,X,yX,y\)\. The set\{1,…,n\}\\\{1,\\ldots,n\\\}is denoted by\[n\]\[n\]\. Theℓ2\\ell\_\{2\}norm ofv∈ℝdv\\in\\mathbb\{R\}^\{d\}is‖v‖\\\|v\\\|\. LetX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}\. We useλminX,λmaxX,‖X‖op\\lambda^\{X\}\_\{\\min\},\\lambda^\{X\}\_\{\\max\},\\\|X\\\|\_\{\\mathrm\{op\}\}to denoteλmin​\(X⊤​X\)\\lambda\_\{\\min\}\(X^\{\\top\}X\),λmax​\(X⊤​X\)\\lambda\_\{\\max\}\(X^\{\\top\}X\), andλmaxX\\sqrt\{\\lambda\_\{\\max\}^\{X\}\}respectively\. The all\-zeros column vector inℝd\\mathbb\{R\}^\{d\}is0→d\\vec\{0\}\_\{d\}and thek×kk\\times kidentity matrix is𝕀k\\mathbb\{I\}\_\{k\}\. The Gaussian distribution with meanμ\\muand \(co\)varianceΣ\\SigmaisN​\(μ,Σ\)\\pazocal\{N\}\(\\mu,\\Sigma\), andN​\(0,𝕀k1×k2\)\\pazocal\{N\}\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times k\_\{2\}\}\)denotes ak1×k2k\_\{1\}\\times k\_\{2\}random matrix with i\.i\.d\. standard normal entries\. Throughout,O~​\(⋅\)\\widetilde\{O\}\(\\cdot\)hides polylogarithmic factors only in quantities whose dependence in the displayed bound is at most polynomial\. The standard basis forℝd\\mathbb\{R\}^\{d\}is denoted by\{ei\}i=1d\\left\\\{e\_\{i\}\\right\\\}^\{d\}\_\{i=1\}, whereei∈ℝde\_\{i\}\\in\\mathbb\{R\}^\{d\}are the columns of𝕀d\\mathbb\{I\}\_\{d\}\. Additional notation is in Appendix[A](https://arxiv.org/html/2605.30600#A1)\.

##### Problem Setup\.

We are interested in designing differentially private mechanisms that operate over tabular datasets\. These are represented as a matrixX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}\. We make a bounded\-domain assumption, namely, that there exists a known constantCX\\textsc\{C\}\_\{X\}such that‖xi‖≤CX\\left\\\|x\_\{i\}\\right\\\|\\leq\\textsc\{C\}\_\{X\}for alli∈\[n\]i\\in\[n\], as in several prior analyses in both private and non\-private settings \(e\.g\.,Shamir \([2015](https://arxiv.org/html/2605.30600#bib.bib44)\); Wang \([2018](https://arxiv.org/html/2605.30600#bib.bib48)\)\)\. Specifically, from a privacy perspective, this condition can be enforced by “clipping” the data\. Throughout, we assume thatCX=1\\textsc\{C\}\_\{X\}=1without loss of generality\. SinceX′=X/CXX^\{\\prime\}=\\nicefrac\{\{X\}\}\{\{\\textsc\{C\}\_\{\\hskip\-0\.73616ptX\}\}\}andθ′=CX​θ\\theta^\{\\prime\}=\\textsc\{C\}\_\{X\}\\thetagiveX​θ=X′​θ′X\\theta=X^\{\\prime\}\\theta^\{\\prime\}, any bound forCX=1\\textsc\{C\}\_\{X\}=1extends to generalCX\\textsc\{C\}\_\{X\}by rescaling\.

### 2\.1Differential Privacy

Differential privacy\(Dworket al\.,[2006](https://arxiv.org/html/2605.30600#bib.bib45)\)is a property of a randomized mechanismM\\pazocal\{M\}that quantifies how different the output distributions ofM\\pazocal\{M\}are on “neighboring” datasets\. Here, datasetsX,X′X,X^\{\\prime\}are*neighbors*ifX′X^\{\\prime\}is formed by removing an element fromXX, or vice versa\.111For simplicity, we identify row removal with replacement by0→d\\vec\{0\}\_\{d\}, so the dimension remains constant\. This notion is sometimes called*zero\-out*neighboring\.We denote this relation byX≃X′X\\simeq X^\{\\prime\}\. Throughout, we use the following standard definitions\.

###### Definition 1\(\(ε,δ\)\(\\varepsilon,\\delta\)\-DP\)\.

A randomized mechanismM\\pazocal\{M\}is said to satisfy*\(ε,δ\)\(\\varepsilon,\\delta\)\-differential privacy*if for allX,X′X,X^\{\\prime\}such thatX′≃XX^\{\\prime\}\\simeq Xand measurable subsetsS⊆Range​\(M\)\\pazocal\{S\}\\subseteq\\mathrm\{Range\}\(\\pazocal\{M\}\),

Pr⁡\(M​\(X\)∈S\)≤eε⋅Pr⁡\(M​\(X′\)∈S\)\+δ\.\\Pr\(\\pazocal\{M\}\(X\)\\in\\pazocal\{S\}\)\\leq e^\{\\varepsilon\}\\cdot\\Pr\(\\pazocal\{M\}\(X^\{\\prime\}\)\\in\\pazocal\{S\}\)\+\\delta\.\(1\)

Our work leverages the Rényi \-DP\(Mironov,[2017](https://arxiv.org/html/2605.30600#bib.bib52)\)variant of differential privacy\.

###### Definition 2\(\(α,r​\(α\)\)\(\\alpha,r\(\\alpha\)\)\-RDP\)\.

A randomized mechanismM\\pazocal\{M\}is said to satisfy\(α,r​\(α\)\)\(\\alpha,r\(\\alpha\)\)\-RDP for someα\>1\\alpha\>1if for allX,X′X,X^\{\\prime\}such thatX≃X′X\\simeq X^\{\\prime\},

Dα\(M\(X\)∥M\(X′\)\)≤r\(α\),\{D\_\{\\alpha\}\\left\(\\pazocal\{M\}\(X\)\\middle\\\|\\pazocal\{M\}\(X^\{\\prime\}\)\\right\)\}\\leq r\(\\alpha\),\(2\)whereDα\(ℙ∥ℚ\)≔1α−1log\(𝔼𝗑∼ℚ\[ℙα\(𝗑\)ℚ−α\(𝗑\)\]\)\{D\_\{\\alpha\}\\left\(\\mathbb\{P\}\\middle\\\|\\mathbb\{Q\}\\right\)\}\\hskip\-1\.99997pt\\coloneqq\\hskip\-1\.99997pt\\frac\{1\}\{\\alpha\-1\}\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\big\(\\mathbb\{E\}\_\{\{\\mathsf\{x\}\}\\sim\\mathbb\{Q\}\}\\left\[\\mathbb\{P\}^\{\\alpha\}\(\{\\mathsf\{x\}\}\)\\mathbb\{Q\}^\{\-\\alpha\}\(\{\\mathsf\{x\}\}\)\\right\]\\hskip\-1\.49994pt\\big\)denotes theα\\alpha\-Rényi divergence\(Rényi,[1960](https://arxiv.org/html/2605.30600#bib.bib134)\)\.222Given random variables𝗉∼ℙ\\mathsf\{p\}\\sim\\mathbb\{P\}and𝗊∼ℚ\\mathsf\{q\}\\sim\\mathbb\{Q\},Dα\(𝗉∥𝗊\)\{D\_\{\\alpha\}\\left\(\\mathsf\{p\}\\middle\\\|\\mathsf\{q\}\\right\)\}denotes theα\\alpha\-Rényi divergenceDα\(ℙ∥ℚ\)\{D\_\{\\alpha\}\\left\(\\mathbb\{P\}\\middle\\\|\\mathbb\{Q\}\\right\)\}\.

One can translate\(α,r​\(α\)\)\(\\alpha,r\(\\alpha\)\)\-RDP to classical\(ε,δ\)\(\\varepsilon,\\delta\)\-DP as demonstrated below\.

###### Proposition 1\(Canonneet al\.\([2020](https://arxiv.org/html/2605.30600#bib.bib51), Proposition 12\)\)\.

IfM\\pazocal\{M\}satisfies\(α,r​\(α\)\)\(\\alpha,r\(\\alpha\)\)\-RDP, then it also satisfies\(ε,δ\)\(\\varepsilon,\\delta\)\-DPfor any0<δ<10<\\delta<1, whereε=r​\(α\)\+log⁡\(1−1α\)−log⁡\(α​δ\)\(α−1\)\\varepsilon=r\(\\alpha\)\+\\log\\left\(1\-\\frac\{1\}\{\\alpha\}\\right\)\-\\frac\{\\log\(\\alpha\\delta\)\}\{\(\\alpha\-1\)\}\.

Both\(α,r​\(α\)\)\(\\alpha,r\(\\alpha\)\)\-RDP and\(ε,δ\)\(\\varepsilon,\\delta\)\-DP are preserved under post\-processing and degrade gracefully under composition\. In particular, post\-processing ensures that if a mechanismffsatisfies either definition, then so doesg∘fg\\circ ffor any \(possibly randomized\)gg\(Dworket al\.,[2014](https://arxiv.org/html/2605.30600#bib.bib142); Mironov,[2017](https://arxiv.org/html/2605.30600#bib.bib52)\)\.

### 2\.2Subsampled Randomized Hadamard Transform

Our fast DP sketch is based on the Subsampled Randomized Hadamard Transform \(SRHT\), which we discuss in this subsection\. This is a standard tool for fast dimensionality\-reduction of large matrices via matrix multiplication\(Tropp,[2011](https://arxiv.org/html/2605.30600#bib.bib85); Ailon and Liberty,[2013](https://arxiv.org/html/2605.30600#bib.bib79)\)\.

###### Definition 3\(SRHT\)\.

Letnnbe a power of22\. An SRHT matrix𝖲SRHT∈ℝk×n\\mathsf\{S\}\_\{\\text\{SRHT\}\}\\in\\mathbb\{R\}^\{k\\times n\},k≤nk\\leq n, is defined as

𝖲SRHT:=nk​𝖯k×n​Hn​𝖡n\\mathsf\{S\}\_\{\\mathrm\{SRHT\}\}:=\\sqrt\{\\frac\{n\}\{k\}\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}\\mathsf\{B\}\_\{n\}\(3\)where𝖡n=diag​\(𝖻1,…,𝖻n\)\\mathsf\{B\}\_\{n\}=\\mathrm\{diag\}\(\\mathsf\{b\}\_\{1\},\\ldots,\\mathsf\{b\}\_\{n\}\)and𝖻i\\mathsf\{b\}\_\{i\}are drawn independently from a Rademacher distribution,Hn∈ℝn×nH\_\{n\}\\in\\mathbb\{R\}^\{n\\times n\}is the normalized Hadamard matrix\(Walsh,[1923](https://arxiv.org/html/2605.30600#bib.bib11)\)that satisfiesHn⊤​Hn=𝕀nH\_\{n\}^\{\\top\}H\_\{n\}=\\mathbb\{I\}\_\{n\}, and𝖯k×n\\mathsf\{P\}\_\{k\\times n\}is formed by randomly samplingkkrows of𝕀n\\mathbb\{I\}\_\{n\}uniformly without replacement\.

We provide background on Hadamard matrices and properties of SRHT matrices in Appendix[B](https://arxiv.org/html/2605.30600#A2)\. The SRHT is particularly appealing from a computational perspective; computingSSRHT​XS\_\{\\mathrm\{SRHT\}\}XforX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}takesO​\(n​d​log⁡\(k\)\)O\(nd\\log\(k\)\)operations\(Ailon and Liberty,[2009](https://arxiv.org/html/2605.30600#bib.bib76)\), which scales exponentially better inkkthan the cost of left multiplyingXXby an arbitrary,k×nk\\times ndense matrix\. Statistically, the SRHT also ensures that the minimal singular value and the norms of the rows are approximately preserved, while controlling the correlations between the columns of the output\. These properties are based on prior results ofPilanci and Wainwright \([2015](https://arxiv.org/html/2605.30600#bib.bib132)\)andCohenet al\.\([2016](https://arxiv.org/html/2605.30600#bib.bib12)\), and are used in the analysis of our fast DP sketching mechanism\. We discuss these in more detail in Appendix[E](https://arxiv.org/html/2605.30600#A5)\.

### 2\.3Differential Privacy via Gaussian Sketches

As a prelude to our fast DP sketching mechanism, we first discuss the “slow” version which is based on a dense sketching matrix, specifically defined by𝖲G∼N​\(0,𝕀k×n\)\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\sim\\pazocal\{N\}\(0,\\mathbb\{I\}\_\{k\\times n\}\)for a target sketch sizekkand input matrixX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}\. The use of such Gaussian sketches for information\-theoretic privacy dates back to the work ofZhouet al\.\([2007](https://arxiv.org/html/2605.30600#bib.bib25)\), andBlockiet al\.\([2012](https://arxiv.org/html/2605.30600#bib.bib40)\)adapted it for differentially\-private covariance estimation\. Later work bySheffet \([2017](https://arxiv.org/html/2605.30600#bib.bib135),[2019](https://arxiv.org/html/2605.30600#bib.bib137)\)studies the Gaussian sketching mechanism in isolation; this analysis was recently sharpened byLevet al\.\([2025](https://arxiv.org/html/2605.30600#bib.bib56)\), who provide a tighter Rényi differential privacy analysis of the mechanism\. In the following lemma, we present the mechanism and its privacy guarantee fromLevet al\.\([2025](https://arxiv.org/html/2605.30600#bib.bib56)\)\.

###### Proposition 2\(\(Levet al\.,[2025](https://arxiv.org/html/2605.30600#bib.bib56)\)\)\.

LetX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}such that‖xi‖≤1\\\|x\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]and let

M​\(X\):=𝗀⊤​X\+σ​ξ⊤,with𝗀∼N​\(0→n,𝕀n\),ξ∼N​\(0→d,𝕀d\)\.\\pazocal\{M\}\(X\):=\\mathsf\{g\}^\{\\top\}X\+\\sigma\\xi^\{\\top\}~,\\quad\\text\{with\}\\quad\\mathsf\{g\}\\sim\\pazocal\{N\}\(\\vec\{0\}\_\{n\},\\mathbb\{I\}\_\{n\}\),~\\xi\\sim\\pazocal\{N\}\(\\vec\{0\}\_\{d\},\\mathbb\{I\}\_\{d\}\)\.\(4\)Assume that we have knowledge ofλ¯min\\overline\{\\lambda\}\_\{\\min\}such thatλminX≥λ¯min\\lambda^\{X\}\_\{\\min\}\\geq\\overline\{\\lambda\}\_\{\\min\}, and thatγ:=σ2\+λ¯min\>1\\gamma:=\\sigma^\{2\}\+\\overline\{\\lambda\}\_\{\\min\}\>1\. Then, for everyα∈\(1,γ\)\\alpha\\in\(1,\\gamma\),M​\(X\)\\pazocal\{M\}\(X\)is\(α,φ​\(α;γ\)\)\(\\alpha,\\varphi\(\\alpha;\\gamma\)\)\-RDP, where

φ​\(α;γ\)≔α2​\(α−1\)​log⁡\(1−1γ\)−12​\(α−1\)​log⁡\(1−αγ\)\.\\varphi\(\\alpha;\\gamma\)\\coloneq\\frac\{\\alpha\}\{2\(\\alpha\-1\)\}\\log\\left\(1\-\\frac\{1\}\{\\gamma\}\\right\)\-\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(1\-\\frac\{\\alpha\}\{\\gamma\}\\right\)\.\(5\)

Proposition[2](https://arxiv.org/html/2605.30600#Thmprop2)bounds the divergence incurred by multiplyingXXby a single Gaussian vector\. For a Gaussian sketch𝖲G\\mathsf\{S\}\_\{\\mathrm\{G\}\}withkkindependent Gaussian rows, each row corresponds to one such release; hence, by composition in RDP, the divergence bound for𝖲G​X\\mathsf\{S\}\_\{\\mathrm\{G\}\}Xiskktimes the bound in \([5](https://arxiv.org/html/2605.30600#S2.E5)\)\. To interpret the role ofγ\\gamma, note that it can be viewed as a lower bound on the squared minimal singular value of the augmented matrixX~:=\[X⊤,σ​𝕀d\]⊤\.\\widetilde\{X\}:=\[X^\{\\top\},\\sigma\\mathbb\{I\}\_\{d\}\]^\{\\top\}\.Thus, given any lower boundλminX≥λ¯min\\lambda^\{X\}\_\{\\min\}\\geq\\overline\{\\lambda\}\_\{\\min\}, the termσ​ξ\\sigma\\xican be interpreted as lifting the minimal singular value to a desired levelγ\\gamma\. Gaussian sketches can yield state\-of\-the\-art accuracy guarantees for DP linear regression\(Levet al\.,[2026](https://arxiv.org/html/2605.30600#bib.bib149)\), improving over classical DP constructions such asWang \([2018](https://arxiv.org/html/2605.30600#bib.bib48)\); however, they do not improve the asymptotic runtime\. Indeed, forming𝖲G​X\\mathsf\{S\}\_\{\\mathrm\{G\}\}XcostsO​\(n​k​d\)O\(nkd\)in the worst case\. Since typicallyk=Ω​\(d\)k=\\Omega\(d\), this is at leastO​\(n​d2\)O\(nd^\{2\}\), which is often comparable to solving the underlying optimization problem\.

## 3The Fast Mixing Mechanism for Differential Privacy

![Refer to caption](https://arxiv.org/html/2605.30600v1/x1.png)
![Refer to caption](https://arxiv.org/html/2605.30600v1/x2.png)
Upadhyay ’14, Theorem 16​​∙\\bullet​FastMix \(ours\):CSf​X2=2​CX2\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}=2\\textsc\{C\}^\{2\}\_\{X\}​​FastMix \(ours\):CSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\infty​​★\\bigstar​GaussMix \[Lev et al\. ’25\]

Figure 1:The additive noise scaleσ\\sigmarequired to attain targetε\\varepsilonfor Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)compared withUpadhyay \([2014](https://arxiv.org/html/2605.30600#bib.bib57)\)and GaussMix\(Levet al\.,[2025](https://arxiv.org/html/2605.30600#bib.bib56), Lemma\. 1\)forn∈\{104,105\}n\\in\\\{10^\{4\},10^\{5\}\\\}\. We setλminX=n8​d​CX2\\lambda^\{X\}\_\{\\min\}=\\frac\{n\}\{8d\}\\textsc\{C\}^\{2\}\_\{X\}andλminSf​X=56​λminX\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}=\\frac\{5\}\{6\}\\lambda^\{X\}\_\{\\min\}\. We plot the noise required by[FastMix](https://arxiv.org/html/2605.30600#alg1)withCSf​X2=2​CX2\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}=2\\textsc\{C\}^\{2\}\_\{X\}, and in the extreme caseCSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\infty\. The curves use the exact characterization instead of \([8](https://arxiv.org/html/2605.30600#S3.E8)\) \(see Appendix[C](https://arxiv.org/html/2605.30600#A3)for details\)\.In this section, we present our key development: a DP mechanism with worst\-case runtime matching SRHT for sufficiently largenn\. It builds on the simpler mechanism

MSf​\(X\)=𝗀⊤​Sf​X\+σ​ξ⊤\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X\)=\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\sigma\\xi^\{\\top\}\(6\)whereSf∈ℝk×nS\_\{\\mathrm\{f\}\}\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathbb\{R\}^\{k\\hskip\-0\.40974pt\\times\\hskip\-0\.40974ptn\}is a prespecified matrix,𝗀∼N​\(0→k,𝕀k\)\\mathsf\{g\}\\hskip\-1\.49994pt\\sim\\hskip\-1\.49994pt\\pazocal\{N\}\(\\vec\{0\}\_\{k\}\\hskip\-0\.50003pt,\\hskip\-0\.50003pt\\mathbb\{I\}\_\{k\}\),ξ∼N​\(0→d,𝕀d\)\\xi\\hskip\-1\.49994pt\\sim\\hskip\-1\.49994pt\\pazocal\{N\}\(\\vec\{0\}\_\{d\}\\hskip\-0\.50003pt,\\hskip\-0\.50003pt\\mathbb\{I\}\_\{d\}\), andσ\>0\\sigma\\hskip\-1\.49994pt\>\\hskip\-1\.49994pt0\. The following lemma proves the RDP guarantees ofMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}, generalizing the result ofLevet al\.\([2025](https://arxiv.org/html/2605.30600#bib.bib56), Lemma 1\)\.

###### Lemma 1\.

LetXXbe a dataset such that‖xi‖≤1\\\|x\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]\. SupposeSf∈ℝk×nS\_\{\\mathrm\{f\}\}\\in\\mathbb\{R\}^\{k\\times n\}is a matrix such that‖Sf​ei‖≤1\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]\. DefineCSf​X2≔1\+2⋅maxi∈\[n\]​‖X⊤​Sf⊤​Sf​ei−xi‖\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}^\{2\}\\coloneq 1\+2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\\|X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\\|\. Assume that

γ≔CSf​X−2⋅\(λminSf​X\+σ2\)\>258\.\\gamma\\coloneq\\textsc\{C\}^\{\-2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\cdot\\left\(\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\>\\frac\{25\}\{8\}\.\(7\)Then

maxX′≃XDα\(MSf\(X\)∥MSf\(X′\)\)≤25​α32​γ2,forallα∈\(1,8​γ25\)\.\\underset\{X^\{\\prime\}\\simeq X\}\{\\max\}\\ \{D\_\{\\alpha\}\\left\(\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X\)\\middle\\\|\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X^\{\\prime\}\)\\right\)\}\\leq\\frac\{25\\alpha\}\{32\\gamma^\{2\}\},\\ \\ \\mathrm\{for\\ all\}\\ \\alpha\\in\\left\(1,\\frac\{8\\gamma\}\{25\}\\right\)\.\(8\)

Proof SketchThe proof follows by calculating theα\\alpha\-Rényi divergence betweenMSf​\(X\)\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X\)andMSf​\(X′\)\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X^\{\\prime\}\)\. This is possible since𝗀⊤​Sf​X\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}Xand𝗀⊤​Sf​X′\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}are both zero mean Gaussian vectors with covariancesX⊤​Sf⊤​Sf​XX^\{\\top\}S^\{\\top\}\_\{\\mathrm\{f\}\}S\_\{\\mathrm\{f\}\}XandX′⁣⊤​Sf⊤​Sf​X′X^\{\\prime\\top\}S^\{\\top\}\_\{\\mathrm\{f\}\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\. The proof then follows by using the closed\-form formula for the Rényi divergence between multivariate Gaussian vectors, and by crucially noting that sinceX′≃XX^\{\\prime\}\\simeq X, the covariances differ by a rank\-22update\. We then use the assumption made onγ\\gammain \([7](https://arxiv.org/html/2605.30600#S3.E7)\) to derive an upper bound that is independent ofXX\. The full proof is presented in Appendix[C\.1](https://arxiv.org/html/2605.30600#A3.SS1)\.

The quantityCSf​X\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}can be interpreted as the row bound forSf​XS\_\{\\mathrm\{f\}\}X\. Similarly to the Gaussian sketching mechanism, for everyα\>1\\alpha\>1and fixedCSf​X\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}, this upper bound decays to zero asγ\\gammaincreases\. Thus, \([8](https://arxiv.org/html/2605.30600#S3.E8)\) can be made arbitrarily small by increasingγ\\gamma, which in turn is done by increasing the noise levelσ2\\sigma^\{2\}\. Thus,γ\\gammaparallels a similar quantity inLevet al\.\([2025](https://arxiv.org/html/2605.30600#bib.bib56), Lemma\. 1\), calculated for the matrixSf​XS\_\{\\mathrm\{f\}\}Xinstead ofXX\. The mechanismMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}can be viewed as applying a premultiplication to the augmented matrix\[\(Sf​X\)⊤,σ​𝕀d\]⊤\\big\[\(S\_\{\\mathrm\{f\}\}X\)^\{\\top\},\\sigma\\mathbb\{I\}\_\{d\}\\big\]^\{\\top\}by a Gaussian random vector of lengthk\+dk\+d\. We also recover the result ofLevet al\.\([2025](https://arxiv.org/html/2605.30600#bib.bib56), Lemma\. 1\)whenSf⊤​Sf=𝕀nS^\{\\top\}\_\{\\mathrm\{f\}\}S\_\{\\mathrm\{f\}\}=\\mathbb\{I\}\_\{n\}, in which caseMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}coincides withM\\pazocal\{M\}in distribution\.

Algorithm 1[FastMix](https://arxiv.org/html/2605.30600#alg1)1:Dataset

X∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}, parameters

k1,γ,τ,ωk\_\{1\},\\gamma,\\tau,\\omega, sketch

Sf∈ℝk2×nS\_\{\\mathrm\{f\}\}\\in\\mathbb\{R\}^\{k\_\{2\}\\times n\}\.

2:

Z←Sf​X,m^←maxi∈\[n\]⁡‖Z⊤​Sf​ei−xi‖,Δ^←maxi,j∈\[n\]⁡\|ei⊤​\(Sf⊤​Sf−𝕀n\)​ej\|Z\\leftarrow S\_\{\\mathrm\{f\}\}X,\\widehat\{m\}\\leftarrow\\max\\limits\_\{i\\in\[n\]\}\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|,\\widehat\{\\Delta\}\\leftarrow\\max\\limits\_\{i,j\\in\[n\]\}\\left\|e^\{\\top\}\_\{i\}\\left\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{j\}\\right\|and

Λ^←2−mini∈\[n\]⁡‖Sf​ei‖2\\widehat\{\\Lambda\}\\leftarrow 2\-\\min\\limits\_\{i\\in\[n\]\}\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\.

3:

m~←max⁡\{0,m^\+ω​Δ^​\(τ−𝗓1\)\}\\widetilde\{m\}\\leftarrow\\max\\left\\\{0,\\widehat\{m\}\+\\omega\\widehat\{\\Delta\}\(\\tau\-\\mathsf\{z\}\_\{1\}\)\\right\\\}for

𝗓1∼Lap​\(0,1\)\\mathsf\{z\}\_\{1\}\\sim\\mathrm\{Lap\}\(0,1\)\.

4:

λ~←max⁡\{0,λminZ−ω​\(Λ^\+2​m~\)​\(τ−𝗓2\)\}\\widetilde\{\\lambda\}\\leftarrow\\max\\left\\\{0,\\lambda^\{Z\}\_\{\\min\}\-\\omega\\left\(\\widehat\{\\Lambda\}\+2\\widetilde\{m\}\\right\)\(\\tau\-\\mathsf\{z\}\_\{2\}\)\\right\\\}for

𝗓2∼Lap​\(0,1\)\\mathsf\{z\}\_\{2\}\\sim\\mathrm\{Lap\}\(0,1\)\.

5:

η~←max⁡\{0,γ​\(1\+2​m~\)−λ~\}\\widetilde\{\\eta\}\\leftarrow\\sqrt\{\\max\\left\\\{0,\\gamma\\left\(1\+2\\widetilde\{m\}\\right\)\-\\widetilde\{\\lambda\}\\right\\\}\}\.

6:Output:

𝖲G​Z\+η~​ξ\\mathsf\{S\}\_\{\\mathrm\{G\}\}Z\+\\widetilde\{\\eta\}\\xiwith

𝖲G∼N​\(0,𝕀k1×k2\),ξ∼N​\(0,𝕀k1×d\),\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\sim\\pazocal\{N\}\\left\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times k\_\{2\}\}\\right\),\\xi\\sim\\pazocal\{N\}\\left\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times d\}\\right\),and

η~\\widetilde\{\\eta\}\.

The simpler mechanismMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}is the basis for our proposed fast mixing mechanism[FastMix](https://arxiv.org/html/2605.30600#alg1)which we discuss next\. While \([7](https://arxiv.org/html/2605.30600#S3.E7)\) provides guidance for choosingσ\\sigma,γ\\gammais not a private quantity by definition\. A key step in[FastMix](https://arxiv.org/html/2605.30600#alg1)is in selectingσ\\sigmaforMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}privately, which is achieved by private estimation ofCSf​X2\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}^\{2\}andλminSf​X\\lambda\_\{\\min\}^\{S\_\{\\mathrm\{f\}\}X\}\. We do so by identifying the sensitivity \(i\.e\., the maximal change for neighboring datasets\) ofX↦CSf​X2X\\mapsto\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}^\{2\}andX↦λminSf​XX\\mapsto\\lambda\_\{\\min\}^\{S\_\{\\mathrm\{f\}\}X\}and apply the Laplace mechanism to ensures privacy \(Lines\.[3](https://arxiv.org/html/2605.30600#alg1.l3)and[4](https://arxiv.org/html/2605.30600#alg1.l4)\)\. After this, we calibrate the noise level with these private estimates and applyMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\.

We recapitulate our goals: we are interested in designing a mechanism that is both fast and has a comparable noise budget to the dense Gaussian sketch\. The latter can be satisfied if the transformX↦Sf​XX\\mapsto S\_\{\\mathrm\{f\}\}Xapproximately preservesλminX\\lambda\_\{\\min\}^\{X\}andCX2\\textsc\{C\}\_\{X\}^\{2\}\. Curiously, we find that SRHT provides a means to achieve both of these goals\. First, as discussed a priori, the SRHT is faster than premultiplyingXXby a dense Gaussian matrix\. Second, for anyχ∈\(0,1\]\\chi\\in\(0,1\], when the target sketch sizekksatisfiesk​χ2=Θ​\(rank​\(X\)⋅log4⁡\(n\)\)k\\chi^\{2\}=\\Theta\\left\(\\mathrm\{rank\}\(X\)\\cdot\\log^\{4\}\(n\)\\right\)implies that

\|λminSf​X−λminX\|≤χ⋅λminXandCSf​X2−CX2≤2​χ⋅CX​‖X‖op\.\\left\|\\lambda\_\{\\min\}^\{S\_\{\\mathrm\{f\}\}X\}\-\\lambda^\{X\}\_\{\\min\}\\right\|\\leq\\chi\\cdot\\lambda\_\{\\min\}^\{X\}\\qquad\\text\{and\}\\qquad\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}^\{2\}\-\\textsc\{C\}\_\{X\}^\{2\}\\leq 2\\chi\\cdot\\textsc\{C\}\_\{X\}\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\.See Appendix[E](https://arxiv.org/html/2605.30600#A5)for an extended discussion of this\. Our main theorem for[FastMix](https://arxiv.org/html/2605.30600#alg1)is stated next\.

###### Theorem 1\.

Letδ∈\(0,1\),k1≥1\\delta\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\(\\hskip\-0\.50003pt0,\\hskip\-0\.50003pt1\\hskip\-0\.50003pt\)\\hskip\-0\.50003pt,\\hskip\-0\.50003ptk\_\{1\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pt1andSfS\_\{\\mathrm\{f\}\}matrix satisfying‖Sf​ei‖≤1\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt1for alli∈\[n\]i\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\[n\]\. Then,

1. 1\.For everyγ\>25/8\\gamma\\hskip\-1\.99997pt\>\\hskip\-1\.99997pt\\nicefrac\{\{25\}\}\{\{8\}\}andω\>0\\omega\\hskip\-1\.99997pt\>\\hskip\-1\.99997pt0, the output of Algorithm[1](https://arxiv.org/html/2605.30600#alg1)withτ≥log⁡\(3/2​δ\)\\tau\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt\\log\(\\nicefrac\{\{3\}\}\{\{2\\delta\}\}\)is\(ε​\(ω,γ,k1,δ\),δ\)\(\\varepsilon\(\\omega\\hskip\-0\.50003pt,\\hskip\-0\.50003pt\\gamma\\hskip\-0\.50003pt,\\hskip\-0\.50003ptk\_\{1\}\\hskip\-0\.50003pt,\\hskip\-0\.50003pt\\delta\)\\hskip\-0\.50003pt,\\hskip\-0\.50003pt\\delta\)\-DPwith respect toXXwhere ε​\(ω,γ,k1,δ\)\\displaystyle\\varepsilon\(\\omega,\\gamma,k\_\{1\},\\delta\)≤2ω\+\{25​k132​γ2\+2​25​k1​log⁡\(3/δ\)32​γ2,log⁡\(3/δ\)≤\(8​γ25−1\)2⋅25​k132​γ2,k14​γ\+25​log⁡\(3/δ\)8​γ−25,otherwise\.\\displaystyle\\leq\\frac\{2\}\{\\omega\}\+\\begin\{cases\}\\frac\{25k\_\{1\}\}\{32\\gamma^\{2\}\}\+2\\sqrt\{\\frac\{25k\_\{1\}\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{32\\gamma^\{2\}\}\},&\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\\leq\\left\(\\frac\{8\\gamma\}\{25\}\-1\\right\)^\{2\}\\cdot\\frac\{25k\_\{1\}\}\{32\\gamma^\{2\}\},\\\\ \\frac\{k\_\{1\}\}\{4\\gamma\}\+\\frac\{25\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{8\\gamma\-25\},&\\mathrm\{otherwise\}\.\\end\{cases\}\(9\)
2. 2\.ForSf∈SRHT​\(k2,n\)S\_\{\\mathrm\{f\}\}\\in\\mathrm\{SRHT\}\(k\_\{2\},n\), Algorithm[1](https://arxiv.org/html/2605.30600#alg1)runs inO​\(n​d​log⁡\(n\)\+k1​k2​d\+k2​d2\+d3\)\.O\\left\(nd\\log\(n\)\+k\_\{1\}k\_\{2\}d\+k\_\{2\}d^\{2\}\+d^\{3\}\\right\)\.

Proof SketchWe note that[FastMix](https://arxiv.org/html/2605.30600#alg1)comprises three private\-mechanism calls: calculatingm~\\widetilde\{m\}, calculatingλ~\\widetilde\{\\lambda\}, and computing the output𝖲G​Z\+η~​ξ\\mathsf\{S\}\_\{\\mathrm\{G\}\}Z\+\\widetilde\{\\eta\}\\xi\. First, note thatm~≥m^\\widetilde\{m\}\\geq\\widehat\{m\}andλ~≤λminZ\\widetilde\{\\lambda\}\\leq\\lambda^\{Z\}\_\{\\min\}w\.p\. at least1−2​δ/31\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\nicefrac\{\{2\\delta\}\}\{\{3\}\}, which holds sinceτ≥log⁡\(3/2​δ\)\\tau\\geq\\log\(\\nicefrac\{\{3\}\}\{\{2\\delta\}\}\)and since𝗓1\\mathsf\{z\}\_\{1\}and𝗓2\\mathsf\{z\}\_\{2\}are Laplace variables\. Conditioned on this event, note that the sensitivity ofm^\\widehat\{m\}is upper bounded byΔ^\\widehat\{\\Delta\}\(Lemma[4](https://arxiv.org/html/2605.30600#Thmlem4)\) and the sensitivity ofλminZ\\lambda^\{Z\}\_\{\\min\}is upper bounded byΛ^\+2​m^≤Λ^\+2​m~\\widehat\{\\Lambda\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt2\\widehat\{m\}\\leq\\widehat\{\\Lambda\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt2\\widetilde\{m\}\(Lemma[5](https://arxiv.org/html/2605.30600#Thmlem5)\)\. Thus, the privacy properties of the Laplace mechanism apply to the release ofm^\\widehat\{m\}, while for the release ofλ~\\widetilde\{\\lambda\}, the privacy guarantees hold since we use a private upper bound on the local sensitivity\. Then, sinceSfS\_\{\\mathrm\{f\}\}has unit column\-norms, the conditions of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)hold\. Multiplying the guarantees of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)byk1k\_\{1\}, converting to\(ε,δ\)\(\\varepsilon,\\delta\)\-DP using Proposition[1](https://arxiv.org/html/2605.30600#Thmprop1), and combining with the guarantees of the steps that use the Laplace mechanism yields the overall privacy guarantees\. The runtime argument follows using the fact that the runtime complexity of sketching with an SRHT costsO​\(n​d​log⁡\(n\)\)O\(nd\\log\(n\)\), and using properties of the Hadamard matrix for computing the different quantities efficiently\. The full proof, along with the tightest bound onε\\varepsilonobtained by using closed\-form guarantees obtained for Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1), is in Appendix[C\.4](https://arxiv.org/html/2605.30600#A3.SS4)\.

##### Role ofSfS\_\{\\mathrm\{f\}\}\.

The matrixSfS\_\{\\mathrm\{f\}\}is treated as deterministic, and our privacy analysis does not rely on any randomness in its construction\. The quantityΔ^=maxi≠j⁡\|ei⊤​Sf⊤​Sf​ej\|\\widehat\{\\Delta\}=\\max\_\{i\\neq j\}\|e\_\{i\}^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{j\}\|is referred to as the*coherence*ofSfS\_\{\\mathrm\{f\}\}\. The application ofSfS\_\{\\mathrm\{f\}\}can be viewed as a computationally efficient pre\-processing step that also “accelerates” the application of the Gaussian sketch\. SinceSfS\_\{\\mathrm\{f\}\}is assumed to be public, any quantity that depends only onSfS\_\{\\mathrm\{f\}\}\(such asΔ^\\widehat\{\\Delta\}andΛ^\\widehat\{\\Lambda\}\) need not itself be privatized\. As discussed previously, the SRHT is viable from a computational and privacy standpoint, and additionally satisfiesΛ^=1\\widehat\{\\Lambda\}=1\. Additionally, for the SRHT, we can rely on classical sketching results to obtain high\-probability approximation guarantees for downstream applications, as shown in Section[4](https://arxiv.org/html/2605.30600#S4)\.

##### Analytical Bound\.

As shown in Appendix[D](https://arxiv.org/html/2605.30600#A4), Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)impliesε=O​\(CSf​X4​k1\(λminSf​X\+σ2\)2\+CSf​X2​k1​log⁡\(1/δ\)λminSf​X\+σ2\)\\varepsilon\\hskip\-1\.99997pt=\\hskip\-1\.99997ptO\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\tfrac\{\\textsc\{C\}^\{4\}\_\{S\_\{\\mathrm\{f\}\}X\}k\_\{1\}\}\{\(\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\+\\sigma^\{2\}\)^\{2\}\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\tfrac\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\sqrt\{k\_\{1\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\{\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\+\\sigma^\{2\}\}\\hskip\-1\.99997pt\\right\)wheneverλminSf​X\+σ2\>a0\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\+\\sigma^\{2\}\>a\_\{0\}andk1\>a1⋅log⁡\(1/δ\)k\_\{1\}\>a\_\{1\}\\cdot\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)for constantsa0a\_\{0\}anda1a\_\{1\}\. Moreover, for anyω≥3ε\\omega\\geq\\frac\{3\}\{\\varepsilon\}, the releasesλ~\\widetilde\{\\lambda\}andm~\\widetilde\{m\}contribute at most2​ε3\\frac\{2\\varepsilon\}\{3\}toε\\varepsilon, so[FastMix](https://arxiv.org/html/2605.30600#alg1)matches this bound up to constant degradation inε\\varepsilon\. In comparison, the pure Gaussian sketch withk1k\_\{1\}rows satisfiesε=O​\(k1\(λminX\+σ2\)2\+k1​log⁡\(1/δ\)λminX\+σ2\)\\varepsilon\\hskip\-1\.99997pt=\\hskip\-1\.99997ptO\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\tfrac\{k\_\{1\}\}\{\(\\lambda^\{X\}\_\{\\min\}\+\\sigma^\{2\}\)^\{2\}\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\tfrac\{\\sqrt\{k\_\{1\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\{\\lambda^\{X\}\_\{\\min\}\+\\sigma^\{2\}\}\\hskip\-1\.99997pt\\right\)\(Levet al\.,[2025](https://arxiv.org/html/2605.30600#bib.bib56), Section 4\)\. Thus,[FastMix](https://arxiv.org/html/2605.30600#alg1)incurs two degradations: each factork1λminX\+σ2\\tfrac\{\\sqrt\{k\_\{1\}\}\}\{\\lambda^\{X\}\_\{\\min\}\+\\sigma^\{2\}\}is multiplied byCSf​X2≥1\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\geq 1, increasingε\\varepsilon; andλminX\\lambda^\{X\}\_\{\\min\}is replaced byλminSf​X\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\. As discussed, the latter typically changesλminX\\lambda^\{X\}\_\{\\min\}only by a constant multiplicative factor, whereas the former can be as large asO​\(rank​\(X\)​log4⁡\(n\)k2⋅n\)O\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\sqrt\{\\frac\{\\mathrm\{rank\}\(X\)\\log^\{4\}\(n\)\}\{k\_\{2\}\}\\hskip\-1\.00006pt\\cdot\\hskip\-1\.00006ptn\}\\right\)\.

##### Comparison to Existing Literature\.

To our knowledge, little prior work has studied DP via fast sketches\. The closest related mechanism is due toUpadhyay \([2014](https://arxiv.org/html/2605.30600#bib.bib57)\), whose construction also first applies an SRHT, but then uses a sparse Gaussian outer sketch\. Their privacy bound scales asO​\(log⁡\(1/δ\)​k​log⁡\(1/δ\)λminSf​X\+σ2\)O\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\\sqrt\{\\tfrac\{k\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\{\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\+\\sigma^\{2\}\}\}\\hskip\-1\.49994pt\\right\), whereSfS\_\{\\mathrm\{f\}\}is the SRHT\. This is looser than[FastMix](https://arxiv.org/html/2605.30600#alg1)by a factorO​\(log⁡\(1/δ\)​\(λminSf​X\+σ2\)1/2CSf​X2\)O\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\frac\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\\left\(\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{\\nicefrac\{\{1\}\}\{\{2\}\}\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\hskip\-1\.99997pt\\right\)\. Their guarantee resembles our analysis in Appendix[C\.2](https://arxiv.org/html/2605.30600#A3.SS2), corresponding toCSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\infty, with further constant andlog⁡\(1/δ\)\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)loss from analyzing directly in\(ε,δ\)\(\\varepsilon,\\delta\)\-DP rather than via RDP \. By Figure[1](https://arxiv.org/html/2605.30600#S3.F1), our approach typically reducesσ2\\sigma^\{2\}by more than two orders of magnitude\.

##### Choosingk2k\_\{2\}\.

While Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1)does not depend explicitly onk2k\_\{2\}, it depends onk2k\_\{2\}implicitly throughλminZ,m^\\lambda^\{Z\}\_\{\\min\},\\widehat\{m\}andΔ^\\widehat\{\\Delta\}\. Note that, for matching the guarantee of the Gaussian sketch, our goal is to makeλminZ≈λminX\\lambda^\{Z\}\_\{\\min\}\\approx\\lambda^\{X\}\_\{\\min\}and to keepm^\\widehat\{m\}andΔ^\\widehat\{\\Delta\}small\. Without additional assumptions, choosingSf∼SRHT​\(k2,n\)S\_\{\\mathrm\{f\}\}\\hskip\-2\.5pt\\sim\\hskip\-2\.5pt\\mathrm\{SRHT\}\(k\_\{2\},n\)yields that for anyχ∈\(0,1\]\\chi\\hskip\-2\.5pt\\in\\hskip\-2\.5pt\(0,1\]s\.t\.k2​χ2log4⁡\(n\)≥max⁡\{c0⋅rank​\(X\),c2⋅log⁡\(c1/ϱ\)\}\\frac\{k\_\{2\}\\chi^\{2\}\}\{\\log^\{4\}\(n\)\}\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt\\max\\left\\\{c\_\{0\}\\hskip\-1\.99997pt\\cdot\\hskip\-1\.99997pt\\mathrm\{rank\}\(X\),c\_\{2\}\\hskip\-1\.99997pt\\cdot\\hskip\-1\.99997pt\\log\(\\nicefrac\{\{c\_\{1\}\}\}\{\{\\varrho\}\}\)\\right\\\},λminZ≥\(1−χ\)​λminX\\lambda^\{Z\}\_\{\\min\}\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt\(1\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\chi\)\\lambda^\{X\}\_\{\\min\}andm^≤χ2​‖X‖op\\widehat\{m\}\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt\\frac\{\\chi\}\{2\}\\hskip\-1\.99997pt\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}w\.p\. at least1−ϱ1\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\varrhofor constantsc0,c1,c2c\_\{0\},c\_\{1\},c\_\{2\}\(Lemma[10](https://arxiv.org/html/2605.30600#Thmlem10)\)\. Moreover, ifk2​χ2≥c~0⋅log⁡\(c~1​n2/χ​ϱ\)⋅log⁡\(c~2​n2/ϱ\)k\_\{2\}\\chi^\{2\}\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt\\widetilde\{c\}\_\{0\}\\hskip\-1\.99997pt\\cdot\\hskip\-1\.99997pt\\log\\left\(\\nicefrac\{\{\\widetilde\{c\}\_\{1\}n^\{2\}\}\}\{\{\\chi\\varrho\}\}\\right\)\\hskip\-1\.99997pt\\cdot\\hskip\-1\.99997pt\\log\\left\(\\nicefrac\{\{\\widetilde\{c\}\_\{2\}n^\{2\}\}\}\{\{\\varrho\}\}\\right\)thenΔ^≤χ\\widehat\{\\Delta\}\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt\\chiw\.p\. at least1−ϱ1\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\varrhofor another three constants\(c~0,c~1,c~2\)\(\\widetilde\{c\}\_\{0\},\\widetilde\{c\}\_\{1\},\\widetilde\{c\}\_\{2\}\)\(Lemma[9](https://arxiv.org/html/2605.30600#Thmlem9)\)\. Consequently, choosingk2k\_\{2\}above these values ensures thatλminZ\\lambda^\{Z\}\_\{\\min\}andCSf​X2\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}are close toλminX\\lambda^\{X\}\_\{\\min\}andCX2\\textsc\{C\}^\{2\}\_\{X\}up to factors1\+χ1\+\\chiand1\+χ2​CX​‖X‖op1\+\\frac\{\\chi\}\{2\\textsc\{C\}\_\{X\}\}\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}, respectively, making the degradation relative to Proposition[2](https://arxiv.org/html/2605.30600#Thmprop2)bounded and allowing control over the privacy guarantee of[FastMix](https://arxiv.org/html/2605.30600#alg1)\.

##### Complexity\.

Settingk1​χ2=O​\(max⁡\{d,log⁡\(1/ϱ\)\}\)k\_\{1\}\\chi^\{2\}\\hskip\-1\.49994pt=\\hskip\-1\.49994ptO\(\\max\\hskip\-1\.49994pt\\left\\\{d,\\log\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\right\\\}\)andk2​χ2=O~​\(max⁡\{rank​\(X\),log2⁡\(1/ϱ\)\}\)k\_\{2\}\\chi^\{2\}\\hskip\-1\.49994pt=\\hskip\-1\.49994pt\\widetilde\{O\}\\hskip\-1\.49994pt\\left\(\\max\\hskip\-1\.49994pt\\left\\\{\\mathrm\{rank\}\(X\),\\log^\{2\}\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\right\\\}\\right\)yields runtime ofO~​\(n​d\+d3\+dχ4⋅max⁡\{rank​\(X\),log2⁡\(1/ϱ\)\}⋅max⁡\{d,log⁡\(1/ϱ\)\}\)\\widetilde\{O\}\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994ptnd\\hskip\-1\.49994pt\+\\hskip\-1\.49994ptd^\{3\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\frac\{d\}\{\\chi^\{4\}\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\max\\hskip\-1\.49994pt\\left\\\{\\hskip\-1\.00006pt\\mathrm\{rank\}\(X\),\\log^\{2\}\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\hskip\-1\.00006pt\\right\\\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\max\\hskip\-1\.49994pt\\left\\\{\\hskip\-1\.00006ptd,\\log\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\hskip\-1\.00006pt\\right\\\}\\hskip\-1\.49994pt\\right\)\. As discussed, these values allow controlled degradation relative to Proposition[2](https://arxiv.org/html/2605.30600#Thmprop2), and thus represent typical choices ofk1k\_\{1\}andk2k\_\{2\}\. Whenever the dominating term isn​dnd—corresponding to the regime where sketching is most beneficial—this matches, up to logarithmic factors, the cost of sketching with SRHT\.

Algorithm 2Fast IHM1:Dataset

\(X,Y\)\(X,Y\); noise parameters

\(γ,τ,σ,ω\)\(\\gamma,\\tau,\\sigma,\\omega\); sketch sizes

\(k1,k2\)\(k\_\{1\},k\_\{2\}\); iterations

TT; clip

C\>0\\textsc\{C\}\>0\.

2:Initialize

θ^0←0→d\\widehat\{\\theta\}\_\{0\}\\leftarrow\\vec\{0\}\_\{d\}\.

3:Compute

η,\{X~t\}t=0T−1←[MultipleFastMixing](https://arxiv.org/html/2605.30600#alg3)​\(X,1,γ,τ,ω,k2,T\)\\eta,\\big\\\{\\widetilde\{X\}\_\{t\}\\big\\\}^\{T\-1\}\_\{t=0\}\\leftarrow\\hyperref@@ii\[alg:gauss\_fast\_mix\_eta\]\{\\texttt\{MultipleFastMixing\}\}\(X,1,\\gamma,\\tau,\\omega,k\_\{2\},T\)\.

4:for

t=0,…,T−1t=0,\\ldots,T\-1:do

5:Sample

𝖲G,t∼N​\(0,𝕀k1×k2\),ξt∼N​\(0,𝕀k1×d\),ζt∼N​\(0→d,𝕀d\)\\mathsf\{S\}\_\{\\mathrm\{G\},t\}\\sim\\pazocal\{N\}\\left\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times k\_\{2\}\}\\right\),\\xi\_\{t\}\\sim\\pazocal\{N\}\\left\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times d\}\\right\),\\zeta\_\{t\}\\sim\\pazocal\{N\}\(\\vec\{0\}\_\{d\},\\mathbb\{I\}\_\{d\}\)\.

6:Calculate

X^t=𝖲G,t​X~t\+η​ξt\\widehat\{X\}\_\{t\}=\\mathsf\{S\}\_\{\\mathrm\{G\},t\}\\widetilde\{X\}\_\{t\}\+\\eta\\xi\_\{t\}\.

7:Calculate

G~t=X⊤​clipC​\(Y−X​θ^t\)−η2​θ^t\+σ​ζt\\widetilde\{G\}\_\{t\}=X^\{\\top\}\\mathrm\{clip\}\_\{\\textsc\{C\}\}\(Y\-X\\widehat\{\\theta\}\_\{t\}\)\-\\eta^\{2\}\\widehat\{\\theta\}\_\{t\}\+\\sigma\\zeta\_\{t\}\.

8:Update

θ^t\+1=θ^t\+\(1k1​X^t⊤​X^t\)−1​G~t\\widehat\{\\theta\}\_\{t\+1\}=\\widehat\{\\theta\}\_\{t\}\+\(\\tfrac\{1\}\{k\_\{1\}\}\\widehat\{X\}\_\{t\}^\{\\top\}\\widehat\{X\}\_\{t\}\)^\{\-1\}\\widetilde\{G\}\_\{t\}\.

9:Output:

θ^T\\widehat\{\\theta\}\_\{T\}

## 4Application: Fast DP\-OLS

We now present an application of[FastMix](https://arxiv.org/html/2605.30600#alg1)to DP\-OLS\. Using[FastMix](https://arxiv.org/html/2605.30600#alg1), we obtain a new algorithm with state\-of\-the\-art runtimes\. We demonstrate empirical runtime improvements of factors up to≈3\\approx 3for large\-scale datasets\. As we show, in certain cases, these runtime improvements come without paying for accuracy\. Recall that the non\-private OLS baseline is given by

θ⋆:=argmin𝜃​‖Y−X​θ‖2\\theta^\{\\star\}:=\\underset\{\\theta\}\{\\mathrm\{argmin\}\}\\ \\left\\\|Y\-X\\theta\\right\\\|^\{2\}whereX∈ℝn×dX\\hskip\-1\.99997pt\\in\\hskip\-1\.99997pt\\mathbb\{R\}^\{n\\hskip\-0\.40974pt\\times\\hskip\-0\.40974ptd\}andY∈ℝnY\\hskip\-1\.99997pt\\in\\hskip\-1\.99997pt\\mathbb\{R\}^\{n\}\. Its DP version seeks a regressorθ^\\widehat\{\\theta\}that satisfies\(ε,δ\)\(\\varepsilon,\\delta\)\-DP \(Definition[1](https://arxiv.org/html/2605.30600#Thmdefn1)\) while minimizing‖Y−X​θ^‖2\\\|Y\\hskip\-2\.5pt\-\\hskip\-2\.5ptX\\widehat\{\\theta\}\\\|^\{2\}\. We focus on the regimen≥dn\\hskip\-2\.5pt\\geq\\hskip\-2\.5ptd\. To obtain a private and fast solution, we adapt the iterative Hessian mixing \(IHM\) algorithm of\(Levet al\.,[2026](https://arxiv.org/html/2605.30600#bib.bib149)\)to use[FastMix](https://arxiv.org/html/2605.30600#alg1)\. Our algorithm is presented in Algorithm[2](https://arxiv.org/html/2605.30600#alg2); the next theorem provides its privacy and accuracy guarantees\.

###### Theorem 2\.

Letθ^\\widehat\{\\theta\}be the output of Algorithm[2](https://arxiv.org/html/2605.30600#alg2)\. Then, there exist\(γ,τ,σ,ω\)\(\\gamma,\\tau,\\sigma,\\omega\)such that

1. 1\.The outputθ^\\widehat\{\\theta\}is\(ε,δ\)\(\\varepsilon,\\delta\)\-DPwith respect to\(X,Y\)\(X,Y\)\.
2. 2\.Letϱ∈\(0,1\]\\varrho\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\(0\\hskip\-0\.50003pt,\\hskip\-0\.50003pt1\]\. Then, for everyχ∈\(0,14\]\\chi\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\bigl\(0,\\hskip\-1\.00006pt\\frac\{1\}\{4\}\\bigr\]there existk1,k2,T,k\_\{1\}\\hskip\-0\.50003pt,\\hskip\-0\.50003ptk\_\{2\}\\hskip\-0\.50003pt,\\hskip\-0\.50003ptT,andCs\.t\. w\.p\. at least1−ϱ1\-\\varrho: ‖Y−X​θ^T‖2−‖Y−X​θ∗‖2≤\{O~​\(γf⋅Cf2​\(1\+m¯\)−λminX​Cf2\)if​λminX≤γfO~​\(γf2​Cf2λminX​\(1\+m¯\)2\)otherwise\\left\\\|Y\-X\\widehat\{\\theta\}\_\{T\}\\right\\\|^\{2\}\-\\left\\\|Y\-X\\theta^\{\*\}\\right\\\|^\{2\}\\leq\\begin\{cases\}\\widetilde\{O\}\\left\(\\gamma\_\{\\mathrm\{f\}\}\\cdot\\textsc\{C\}^\{2\}\_\{\\mathrm\{f\}\}\\left\(1\+\\overline\{m\}\\right\)\-\\lambda^\{X\}\_\{\\min\}\\textsc\{C\}^\{2\}\_\{\\mathrm\{f\}\}\\right\)&\\mathrm\{if\}\\ \\lambda^\{X\}\_\{\\min\}\\leq\\gamma\_\{\\mathrm\{f\}\}\\\\ \\widetilde\{O\}\\left\(\\frac\{\\gamma^\{2\}\_\{\\mathrm\{f\}\}\\textsc\{C\}^\{2\}\_\{\\mathrm\{f\}\}\}\{\\lambda^\{X\}\_\{\\min\}\\left\(1\+\\overline\{m\}\\right\)^\{2\}\}\\right\)&\\mathrm\{otherwise\}\\end\{cases\}\(10\)where gX≔\(ε​dlog⁡\(1/δ\)​\(λmaxX−λminX\)\)2/3,Cf2:=CY2\+‖θ∗‖2,m¯≔χ​\(λmaxX\+log⁡\(1/δ\)ε\)g^\{X\}\\coloneqq\\left\(\\frac\{\\varepsilon d\}\{\\sqrt\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\\left\(\\lambda^\{X\}\_\{\\max\}\-\\lambda^\{X\}\_\{\\min\}\\right\)\\right\)^\{\\nicefrac\{\{2\}\}\{\{3\}\}\},\\textsc\{C\}^\{2\}\_\{\\mathrm\{f\}\}:=\\textsc\{C\}^\{2\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\},\\overline\{m\}\\coloneqq\\chi\\left\(\\sqrt\{\\lambda^\{X\}\_\{\\max\}\}\+\\frac\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\{\\varepsilon\}\\right\)\(11\)andγf≔max⁡\{d,log⁡\(1/ϱ\),gX\}​log⁡\(1/δ\)​\(1\+m¯\)ε\\gamma\_\{\\mathrm\{f\}\}\\coloneqq\\frac\{\\sqrt\{\\max\\left\\\{d,\\log\\left\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\\right\),g^\{X\}\\right\\\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\\left\(1\+\\overline\{m\}\\right\)\}\{\\varepsilon\}\.
3. 3\.Under the previousχ,ϱ,k1,k2,T\\chi,\\varrho,k\_\{1\},k\_\{2\},TandC, the algorithm runs in O~​\(n​d\+dχ2⋅max⁡\{d,gX,log⁡\(1/ϱ\)\}⋅\(d\+1χ2⋅max⁡\{rank​\(X\),gX,log2⁡\(1/ϱ\)\}\)\)\.\\widetilde\{O\}\\left\(nd\+\\frac\{d\}\{\\chi^\{2\}\}\\cdot\\max\\left\\\{d,g^\{X\},\\log\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\right\\\}\\cdot\\left\(d\+\\frac\{1\}\{\\chi^\{2\}\}\\cdot\\max\\left\\\{\\mathrm\{rank\}\(X\),g^\{X\},\\log^\{2\}\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\right\\\}\\right\)\\right\)\.\(12\)

Proof SketchThe proof adapts the analysis ofLevet al\.\([2026](https://arxiv.org/html/2605.30600#bib.bib149)\)to[FastMix](https://arxiv.org/html/2605.30600#alg1), where the effective sketch concatenates two sketches\. Using properties of SRHT, this roughly incurs a loss of factor1−χ1\-\\chiinλminX\\lambda\_\{\\min\}^\{X\}and1\+m¯1\+\\overline\{m\}inγ\\gamma\. Substituting these in the asymptotic analysis ofLevet al\.\([2026](https://arxiv.org/html/2605.30600#bib.bib149)\)yields the stated argument\. The runtime complexity is dominated byTTsteps of[FastMix](https://arxiv.org/html/2605.30600#alg1)\(splitted betweenMultipleFastMixingand Line\.[6](https://arxiv.org/html/2605.30600#alg2.l6)\), andTTiterations of solving an OLS problem of sizek1×dk\_\{1\}\\times d\. Summing these and substituting the asymptotic arguments of the parametersk1,k2,Ck\_\{1\},k\_\{2\},\\textsc\{C\}andTTused for deriving \([11](https://arxiv.org/html/2605.30600#S4.E11)\) yields the stated argument\. The full proof is in Appendix[E](https://arxiv.org/html/2605.30600#A5)\.

The guarantee \([10](https://arxiv.org/html/2605.30600#S4.E10)\) parallelsLevet al\.\([2026](https://arxiv.org/html/2605.30600#bib.bib149), Theorem\. 1\)\. The main difference is that the bound is multiplied by1\+m¯1\+\\overline\{m\}wheneverλminX≤γf\\lambda^\{X\}\_\{\\min\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\gamma\_\{\\mathrm\{f\}\}, and that the thresholdγf\\gamma\_\{\\mathrm\{f\}\}is inflated by the same factor\. This is since the Gaussian sketch is applied to\{𝖲H,t​X\}t\\\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\\}\_\{t\}rather than directly toXX\. This degradation can be controlled by increasingk2k\_\{2\}at the cost of a runtime increase, as demonstrated in Section[4\.1](https://arxiv.org/html/2605.30600#S4.SS1)\. For runtime, wheneverd​χ2≥max⁡\{rank​\(X\),gX,log2⁡\(1/ϱ\)\}d\\chi^\{2\}\\geq\\max\\left\\\{\\mathrm\{rank\}\(X\),g^\{X\},\\log^\{2\}\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\right\\\}, the overall runtime evaluates toO~​\(n​d\+d3χ2\)\\widetilde\{O\}\\left\(nd\+\\frac\{d^\{3\}\}\{\\chi^\{2\}\}\\right\), incurring a slight increase relative to the runtime of[FastMix](https://arxiv.org/html/2605.30600#alg1)and shows the trade\-off inχ\\chi\. In particular, in many regimes, it is significantly better than the complexity of classical OLS solvers, given byO​\(n​d2\)O\\left\(nd^\{2\}\\right\)\. Comparing this to the complexity of solving OLS using sketching with SRHT, which isO~​\(n​d\+d2⋅rank​\(X\)χ2\)\\widetilde\{O\}\\left\(nd\+\\frac\{d^\{2\}\\cdot\\mathrm\{rank\}\(X\)\}\{\\chi^\{2\}\}\\right\)\(Pilanci and Wainwright,[2015](https://arxiv.org/html/2605.30600#bib.bib132), Section\. III\.A\), this complexity is worse wheneverrank​\(X\)≪d\\mathrm\{rank\}\(X\)\\ll d, and asymptotically matches whenevern=Ω​\(d2χ2\)n=\\Omega\\left\(\\frac\{d^\{2\}\}\{\\chi^\{2\}\}\\right\)\.

##### Remark\.

Another fast DP\-OLS algorithm can be obtained by combining[FastMix](https://arxiv.org/html/2605.30600#alg1)with the sketch\-and\-solve method ofSheffet \([2017](https://arxiv.org/html/2605.30600#bib.bib135)\)\. However, sketch\-and\-solve sketches\(X,Y\)\(X,Y\)rather thanXXalone\. Sinceλmin\(X,Y\)≤λminX\\lambda\_\{\\min\}^\{\(X\\hskip\-0\.81949pt,\\hskip\-0\.81949ptY\)\}\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt\\lambda\_\{\\min\}^\{X\}andλmax\(X,Y\)≥λmaxX\\lambda\_\{\\max\}^\{\(X\\hskip\-0\.81949pt,\\hskip\-0\.81949ptY\)\}\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt\\lambda\_\{\\max\}^\{X\},333This holds sinceλmax\(X,Y\)=max𝑣​v⊤​\(X,Y\)⊤​\(X,Y\)​v≥max𝑢​u⊤​X⊤​X​u=λmaxX\\lambda\_\{\\max\}^\{\(X\\hskip\-0\.81949pt,\\hskip\-0\.81949ptY\)\}\\hskip\-1\.99997pt=\\hskip\-1\.99997pt\\underset\{v\}\{\\max\}\\ v^\{\\top\}\\hskip\-1\.00006pt\(X\\hskip\-1\.00006pt,\\hskip\-1\.00006ptY\)^\{\\top\}\\hskip\-1\.00006pt\(X\\hskip\-1\.00006pt,\\hskip\-1\.00006ptY\)v\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt\\underset\{u\}\{\\max\}\\ u^\{\\top\}\\hskip\-1\.00006ptX^\{\\top\}\\hskip\-1\.00006ptXu\\hskip\-1\.99997pt=\\hskip\-1\.99997pt\\lambda^\{X\}\_\{\\max\}and similarly forλmin\(X,Y\)≤λminX\\lambda\_\{\\min\}^\{\(X\\hskip\-0\.81949pt,\\hskip\-0\.81949ptY\)\}\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt\\lambda\_\{\\min\}^\{X\}\.the resulting utility degradation is typically worse than in the IHM\-based construction and, for practicalε\\varepsilon, yields substantially lower accuracy\. Thus, the practical gain comes not from[FastMix](https://arxiv.org/html/2605.30600#alg1)alone, but from using it within IHM\(Levet al\.,[2026](https://arxiv.org/html/2605.30600#bib.bib149)\)\.

![Refer to caption](https://arxiv.org/html/2605.30600v1/x3.png)Black Friday
![Refer to caption](https://arxiv.org/html/2605.30600v1/x4.png)Beijing
![Refer to caption](https://arxiv.org/html/2605.30600v1/x5.png)YearsMSD
![Refer to caption](https://arxiv.org/html/2605.30600v1/x6.png)Synthetic: Correlated
![Refer to caption](https://arxiv.org/html/2605.30600v1/x7.png)Rossman
![Refer to caption](https://arxiv.org/html/2605.30600v1/x8.png)Synthetic:λmin=n/d\\lambda\_\{\\min\}=\\nicefrac\{\{n\}\}\{\{d\}\}
\+\\boldsymbol\{\+\}FastIHM:k2=r1⋅max⁡\{d,4​Tϱ\}\\mathrm\{FastIHM\}\\hskip\-1\.7pt:\\hskip\-1\.7ptk\_\{2\}=r\_\{1\}\\cdot\\max\\\{d,\\hskip\-0\.85005pt\\frac\{4T\}\{\\varrho\}\\hskip\-0\.85005pt\\\}×\\boldsymbol\{\\times\}FastIHM:k2=r2⋅max⁡\{d,4​Tϱ\}\\mathrm\{FastIHM\}\\hskip\-1\.7pt:\\hskip\-1\.7ptk\_\{2\}=r\_\{2\}\\cdot\\max\\\{d,\\hskip\-0\.85005pt\\frac\{4T\}\{\\varrho\}\\hskip\-0\.85005pt\\\}▲\\blacktriangleFastIHM:k2=r3⋅max⁡\{d,4​Tϱ\}\\mathrm\{FastIHM\}\\hskip\-1\.7pt:\\hskip\-1\.7ptk\_\{2\}=r\_\{3\}\\cdot\\max\\\{d,\\hskip\-0\.85005pt\\frac\{4T\}\{\\varrho\}\\hskip\-0\.85005pt\\\}■\\blacksquareIHM\\ \\mathrm\{IHM\}FastIHS:k=6⋅max⁡\{d,4​Tϱ\}\\ \\mathrm\{FastIHS\}:k=6\\cdot\\max\\\{d,\\frac\{4T\}\{\\varrho\}\\\}

Figure 2:Performance comparison of[FastIHM](https://arxiv.org/html/2605.30600#alg2)and IHM\. The upper plots are fornd≈1450\\frac\{n\}\{d\}\\approx 1450and the lower fornd≈16000\\frac\{n\}\{d\}\\approx 16000\. For each dataset, we use three values ofk2k\_\{2\}, represented byr1,r2r\_\{1\},r\_\{2\}, andr3r\_\{3\}\. The plots illustrate the dependence onλmaxX\\lambda^\{X\}\_\{\\max\}andλminX\\lambda^\{X\}\_\{\\min\}, where smallerλmaxX\\lambda^\{X\}\_\{\\max\}and largerλminX\\lambda^\{X\}\_\{\\min\}improve performance\. Moreover, in certain cases,[FastIHM](https://arxiv.org/html/2605.30600#alg2)matches the performance of IHM\.Dataset\(r1,r2,r3\)\(r\_\{1\},r\_\{2\},r\_\{3\}\)Fast IHSFast IHM:r1r\_\{1\}Fast IHM:r2r\_\{2\}Fast IHM:r3r\_\{3\}Black Friday\(4,100,200\)\(4,100,200\)4\.224\.222\.442\.441\.951\.951\.611\.61Beijing\(4,100,200\)\(4,100,200\)6\.896\.893\.123\.122\.382\.381\.941\.94YearsMSD\(4,100,200\)\(4,100,200\)7\.107\.102\.512\.511\.701\.701\.531\.53Synthetic Correlated\(4,400,800\)\(4,400,800\)4\.314\.312\.332\.332\.142\.141\.911\.91Rossman\(4,400,800\)\(4,400,800\)5\.135\.133\.013\.012\.632\.632\.402\.40Synthetic:λmin=nd\\lambda\_\{\\min\}=\\frac\{n\}\{d\}\(4,400,800\)\(4,400,800\)5\.05\.02\.522\.522\.262\.262\.062\.06

Figure 3:The ratio between the runtime of IHM and each of the different methods \(FastIHS and FastIHM\), for the three ratiosr1,r2r\_\{1\},r\_\{2\}andr3r\_\{3\}\. In most cases, FastIHM is22to33times faster than IHM\.
### 4\.1Simulations

Our experiments are designed to highlight the runtime advantage of[FastMix](https://arxiv.org/html/2605.30600#alg1)\. For accuracy comparisons, we use the IHM fromLevet al\.\([2026](https://arxiv.org/html/2605.30600#bib.bib149)\)\. This method was shown to attain state\-of\-the\-art performance over a large collection of datasets, and further fixes the underlying implementation, making the runtime differences only due to the replacement of the Gaussian sketch with[FastMix](https://arxiv.org/html/2605.30600#alg1)\. Our goal is to show that[FastMix](https://arxiv.org/html/2605.30600#alg1)attains state\-of\-the\-art runtime for a given approximation accuracy, rather than to improve on the best known accuracy guarantees, which were already established in\(Levet al\.,[2026](https://arxiv.org/html/2605.30600#bib.bib149)\)\. We use IHS with SRHT \(also termed*FastIHS*\) as our non\-private baseline\. We evaluate the methods on six large\-scale datasets chosen to illustrate the applicability of the utility guarantee in \([10](https://arxiv.org/html/2605.30600#S4.E10)\)\. Dataset statistics are reported in Appendix[J](https://arxiv.org/html/2605.30600#A10), with full experimental details in Appendix[H](https://arxiv.org/html/2605.30600#A8)\. In Appendix[I](https://arxiv.org/html/2605.30600#A9), we also include experiments with extremen/d\\nicefrac\{\{n\}\}\{\{d\}\}, a regime where sketching is especially favorable\. Each experiment shows the accuracy\-privacy trade\-off induced by varyingk2k\_\{2\}, and we further report the speedup factor attained relative to IHM\. As shown in Figure[3](https://arxiv.org/html/2605.30600#S4.F3),[FastIHM](https://arxiv.org/html/2605.30600#alg2)is substantially faster than IHM, with an accuracy loss that decreases ask2k\_\{2\}increases\. The first row, corresponding to Black Friday, Years, and Beijing, considers datasets withn/d≈1450\\nicefrac\{\{n\}\}\{\{d\}\}\\approx 1450,λminX≈0\\lambda\_\{\\min\}^\{X\}\\approx 0, and decreasingλmaxX\\lambda\_\{\\max\}^\{X\}\. WhenλmaxX\\lambda\_\{\\max\}^\{X\}is smaller, as is the case for the Beijing and the YearsMSD datasets,[FastIHM](https://arxiv.org/html/2605.30600#alg2)attains performance closer to IHM for the higher regime ofε\\varepsilon, while remaining faster\. For the Black Friday dataset, whereλmaxX≈4⋅104\\lambda^\{X\}\_\{\\max\}\\approx 4\\cdot 10^\{4\}, there is a performance loss between[FastIHM](https://arxiv.org/html/2605.30600#alg2)and IHM, even for largerε\\varepsilon’s\. The second row considers datasets withn/d≈214\\nicefrac\{\{n\}\}\{\{d\}\}\\approx 2^\{14\}and varying values ofλminX\\lambda\_\{\\min\}^\{X\}andλmaxX\\lambda\_\{\\max\}^\{X\}\. Synthetic Correlated and Rossman both have poorly conditioned design matrices; However, the final plot corresponds to a setting whereλminX\\lambda\_\{\\min\}^\{X\}nearly attains its maximal theoretical value,n/d\\nicefrac\{\{n\}\}\{\{d\}\}\. In this case,[FastIHM](https://arxiv.org/html/2605.30600#alg2)matches IHM’s accuracy, as predicted by \([10](https://arxiv.org/html/2605.30600#S4.E10)\)\. Surprisingly, in this setting, the performance of[FastIHM](https://arxiv.org/html/2605.30600#alg2)is not monotone inε\\varepsilon: for some values ofε\\varepsilon, it even outperforms IHM and matches the non\-private baseline\. We attribute this behavior to the implicit regularization induced by adding noise to the sketched Hessian\. In IHM, the added noise level is always zero for allε\\varepsilon\. In contrast, for[FastIHM](https://arxiv.org/html/2605.30600#alg2), the noise is zero only for the two largest values ofε\\varepsilonand for the choicesr2r\_\{2\}andr3r\_\{3\}\. For smaller values ofε\\varepsilon, the noise acts as a form of Hessian regularization\. This is consistent with prior work showing that IHS can benefit from regularization or debiasing of the sketched Hessian\(Zhang and Pilanci,[2023](https://arxiv.org/html/2605.30600#bib.bib2); Derezinskiet al\.,[2020](https://arxiv.org/html/2605.30600#bib.bib3)\)\. In our setting, this effect arises implicitly through the privacy noise added to the sketched Hessian\. Developing a theory for optimal Hessian debiasing in the private setting is an interesting direction for future work\.

## 5Discussion and Future Work

We introduced a fast DP mechanism based on sketching, paralleling fast sketching methods\. The mechanism attains state\-of\-the\-art DP guarantees and, to the best of our knowledge, is the first fast sketching\-based DP mechanism whose privacy degradation relative to a Gaussian sketch is only a constant factor in some regimes\. We applied it to DP\-OLS, obtaining an algorithm with state\-of\-the\-art runtime and provable accuracy guarantees\. Relative to prior DP sketching\-based approaches, our method is significantly faster and, in some regimes, incurs negligible accuracy degradation\. Large\-scale regression simulations confirm the gains, demonstrating speedups\.

Our results suggest several directions for future work\. The current instantiation of[FastMix](https://arxiv.org/html/2605.30600#alg1)concatenates an SRHT with a Gaussian sketch\. This choice has three drawbacks: it does not exploit sparsity in the input matrix; it does not minimize correlations between sketch columns, leavingΔ^\\widehat\{\\Delta\}in Line[2](https://arxiv.org/html/2605.30600#alg1.l2)relatively large \(see Appendix[F](https://arxiv.org/html/2605.30600#A6)\); and its cost depends onk1​k2​dk\_\{1\}k\_\{2\}d, rather thank​d2kd^\{2\}for a single sketch withkkrows, slightly increasing the complexity relative toPilanci and Wainwright \([2015](https://arxiv.org/html/2605.30600#bib.bib132), Section\. III\.A\), in particular wheneverXXhas a low\-rank structure\. It would therefore be valuable to study alternative sketches within[FastMix](https://arxiv.org/html/2605.30600#alg1), such as sparse sketches or sketches with nearly minimal coherence, and to examine whether the concatenated sketch can be replaced by a single sketch\. Such variants may improve both accuracy and runtime\. In particular, since modern sketching methods adapt the sketch to the matrix being sketched \(see, e\.g\.,\(Derezinskiet al\.,[2021](https://arxiv.org/html/2605.30600#bib.bib151)\)\), it is interesting to ask whether analogous constructions are possible under DP\. Beyond DP\-OLS,[FastMix](https://arxiv.org/html/2605.30600#alg1)may serve as a primitive for second\-order DP convex optimization algorithms in the spirit of the Newton sketch\(Pilanci and Wainwright,[2017](https://arxiv.org/html/2605.30600#bib.bib59)\), enabling computationally efficient DP second\-order methods\.

## 6Acknowledgemens

OL and AW were supported in part by the Simons Foundation Collaboration on Algorithmic Fairness under Award SFI\-MPS\-TAF\-0008529\-14, and VS and AW were supported in part by Assicurazioni Generali S\.p\.A\. through MIT Award 036189\-00006\. The work of MS and KL was supported in part by ERC grant 101125913, Simons Foundation Collaboration 733792, and Apple\. Views and opinions expressed are however those of the author\(s\) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency\. Neither the European Union nor the granting authority can be held responsible for them\.

## References

- Fast dimension reduction using Rademacher series on dual BCH codes\.Discrete & Computational Geometry42\(4\),pp\. 615–630\.Cited by:[§C\.4\.2](https://arxiv.org/html/2605.30600#A3.SS4.SSS2.Px1.p1.7),[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.30600#S2.SS2.p2.6)\.
- N\. Ailon and E\. Liberty \(2013\)An almost optimal unrestricted fast Johnson\-Lindenstrauss transform\.ACM Transactions on Algorithms \(TALG\)9\(3\),pp\. 1–12\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.30600#S2.SS2.p1.1)\.
- \[3\]\(2020\)Airlines DepDelay 10M\.Note:OpenML dataset, ID 42728External Links:[Link](https://www.openml.org/d/42728)Cited by:[Table 1](https://arxiv.org/html/2605.30600#A10.T1.12.12.14.1),[Appendix\. I](https://arxiv.org/html/2605.30600#A9.p1.6)\.
- H\. Avron, K\. L\. Clarkson, and D\. P\. Woodruff \(2017\)Sharper bounds for regularized data fitting\.Approximation, Randomization, and Combinatorial Optimization\. Algorithms and Techniques\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1)\.
- H\. Avron, P\. Maymounkov, and S\. Toledo \(2010\)Blendenpik: supercharging LAPACK’s least\-squares solver\.SIAM Journal on Scientific Computing32\(3\),pp\. 1217–1236\.Cited by:[Remark 1](https://arxiv.org/html/2605.30600#Thmremark1.p1.1)\.
- B\. Balle and Y\. Wang \(2018\)Improving the Gaussian mechanism for differential privacy: analytical calibration and optimal denoising\.InProceedings of the International Conference on Machine Learning,pp\. 394–403\.Cited by:[Appendix\. H](https://arxiv.org/html/2605.30600#A8.p3.15)\.
- T\. Bertin\-Mahieux \(2011\)Year Prediction MSD\.Note:UCI Machine Learning RepositoryCited by:[Table 1](https://arxiv.org/html/2605.30600#A10.T1.9.9.9.2)\.
- \[8\]\(2019\)Black friday\.Note:OpenML dataset, ID 41540External Links:[Link](https://www.openml.org/d/41540)Cited by:[Table 1](https://arxiv.org/html/2605.30600#A10.T1.10.10.10.2)\.
- J\. Blocki, A\. Blum, A\. Datta, and O\. Sheffet \(2012\)The Johnson\-Lindenstrauss transform itself preserves differential privacy\.InIEEE Symposium on Foundations of Computer Science,pp\. 410–419\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p3.4),[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p1.3)\.
- C\. Boutsidis and A\. Gittens \(2013\)Improved matrix algorithms via the subsampled randomized Hadamard transform\.SIAM Journal on Matrix Analysis and Applications34\(3\),pp\. 1301–1340\.Cited by:[Remark 1](https://arxiv.org/html/2605.30600#Thmremark1.p1.1)\.
- M\. Bun, C\. Dwork, G\. N\. Rothblum, and T\. Steinke \(2018\)Composable and versatile privacy via truncated CDP\.InProceedings of the 50th ACM SIGACT Symposium on Theory of Computing,pp\. 74–86\.Cited by:[§D\.3](https://arxiv.org/html/2605.30600#A4.SS3.p1.2)\.
- U\. S\. C\. Bureau \(2019\)A History of Census Privacy Protections\.Note:[https://www\.census\.gov/library/visualizations/2019/comm/history\-privacy\-protection\.html](https://www.census.gov/library/visualizations/2019/comm/history-privacy-protection.html)Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p2.1)\.
- C\. L\. Canonne, G\. Kamath, and T\. Steinke \(2020\)The discrete Gaussian for differential privacy\.Advances in Neural Information Processing Systems33,pp\. 15676–15688\.Cited by:[§E\.2\.1](https://arxiv.org/html/2605.30600#A5.SS2.SSS1.p8.7),[Proposition 1](https://arxiv.org/html/2605.30600#Thmprop1)\.
- M\. B\. Cohen, J\. Nelson, and D\. P\. Woodruff \(2016\)Optimal approximate matrix product in terms of stable rank\.In43rd International Colloquium on Automata, Languages, and Programming,Vol\.55,pp\. 11:1–11:14\.Cited by:[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.7.p2.1),[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.p2.2),[§1\.1](https://arxiv.org/html/2605.30600#S1.SS1.p1.3),[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.30600#S2.SS2.p2.6)\.
- M\. Derezinski, B\. Bartan, M\. Pilanci, and M\. W\. Mahoney \(2020\)Debiasing distributed second order optimization with surrogate sketching and scaled regularization\.Advances in Neural Information Processing Systems33,pp\. 6684–6695\.Cited by:[§4\.1](https://arxiv.org/html/2605.30600#S4.SS1.p1.22)\.
- M\. Derezinski, J\. Lacotte, M\. Pilanci, and M\. W\. Mahoney \(2021\)Newton\-LESS: Sparsification without trade\-offs for the sketched Newton update\.Advances in Neural Information Processing Systems34,pp\. 2835–2847\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[§5](https://arxiv.org/html/2605.30600#S5.p2.5)\.
- P\. Drineas, M\. W\. Mahoney, S\. Muthukrishnan, and T\. Sarlós \(2011\)Faster least squares approximation\.Numerische mathematik117\(2\),pp\. 219–249\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1)\.
- P\. Drineas, M\. W\. Mahoney, and S\. Muthukrishnan \(2006\)Sampling algorithms forℓ2\\ell\_\{2\}regression and applications\.InProceedings of the seventeenth annual ACM\-SIAM symposium on Discrete algorithm,pp\. 1127–1136\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1)\.
- C\. Dwork, F\. McSherry, K\. Nissim, and A\. Smith \(2006\)Calibrating noise to sensitivity in private data analysis\.InTheory of Cryptography Conference,pp\. 265–284\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.30600#S2.SS1.p1.6)\.
- C\. Dwork, A\. Roth,et al\.\(2014\)The algorithmic foundations of differential privacy\.Foundations and Trends® in Theoretical Computer Science9\(3–4\),pp\. 211–407\.Cited by:[§C\.4\.1](https://arxiv.org/html/2605.30600#A3.SS4.SSS1.Px1.p1.16),[§2\.1](https://arxiv.org/html/2605.30600#S2.SS1.p4.5)\.
- M\. Gil, F\. Alajaji, and T\. Linder \(2013\)Rényi divergence measures for commonly used univariate continuous distributions\.Information Sciences249,pp\. 124–131\.Cited by:[Appendix\. C](https://arxiv.org/html/2605.30600#A3.p1.1)\.
- M\. Haikin, M\. Gavish, D\. G\. Mixon, and R\. Zamir \(2021\)Asymptotic frame theory for analog coding\.Foundations and Trends in Communications and Information Theory18\(4\),pp\. 526–645\.Cited by:[Appendix\. F](https://arxiv.org/html/2605.30600#A6.p1.6)\.
- J\. Lacotte, S\. Liu, E\. Dobriban, and M\. Pilanci \(2020\)Optimal iterative sketching methods with the subsampled randomized Hadamard transform\.Advances in Neural Information Processing Systems33,pp\. 9725–9735\.Cited by:[Appendix\. H](https://arxiv.org/html/2605.30600#A8.p4.12)\.
- O\. Lev, M\. Shenfeld, V\. Srinivasan, K\. Ligett, and A\. C\. Wilson \(2026\)Near\-optimal private linear regression via iterative Hessian mixing\.Proceedings of the International Conference on Machine Learning \(to appear\)\.Cited by:[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.p2.2),[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.p5.4),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px2),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px2.1.p1.9),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px2.2.p1.4),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px2.3.p1.1),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px2.p2.6),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px3),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px3.p3.21),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px3.p3.23),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px3.p3.6),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px3.p4.3),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.p1.1),[Appendix\. E](https://arxiv.org/html/2605.30600#A5.p1.1),[Appendix\. H](https://arxiv.org/html/2605.30600#A8.p3.15),[§1](https://arxiv.org/html/2605.30600#S1.p3.4),[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p2.14),[§4](https://arxiv.org/html/2605.30600#S4.SS0.SSS0.Px1.p1.5),[§4\.1](https://arxiv.org/html/2605.30600#S4.SS1.p1.22),[§4](https://arxiv.org/html/2605.30600#S4.p1.7),[§4](https://arxiv.org/html/2605.30600#S4.p2.9),[§4](https://arxiv.org/html/2605.30600#S4.p3.13),[Lemma 12](https://arxiv.org/html/2605.30600#Thmlem12.p1.5.5),[Lemma 13](https://arxiv.org/html/2605.30600#Thmlem13.p1.5.5),[Lemma 14](https://arxiv.org/html/2605.30600#Thmlem14.p1.14.14)\.
- O\. Lev, V\. Srinivasan, M\. Shenfeld, K\. Ligett, A\. Sekhari, and A\. C\. Wilson \(2025\)The Gaussian mixing mechanism: Rényi differential privacy via Gaussian sketches\.Advances in Neural Information Processing Systems39\.Cited by:[§C\.1](https://arxiv.org/html/2605.30600#A3.SS1.SSS0.Px1.p1.11),[§D\.2](https://arxiv.org/html/2605.30600#A4.SS2.p1.12),[Appendix\. D](https://arxiv.org/html/2605.30600#A4.p2.1),[Appendix\. D](https://arxiv.org/html/2605.30600#A4.p2.6),[§1](https://arxiv.org/html/2605.30600#S1.p3.4),[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p1.3),[Figure 1](https://arxiv.org/html/2605.30600#S3.F1),[Figure 1](https://arxiv.org/html/2605.30600#S3.F1.21.7),[§3](https://arxiv.org/html/2605.30600#S3.SS0.SSS0.Px2.p1.20),[§3](https://arxiv.org/html/2605.30600#S3.p1.6),[§3](https://arxiv.org/html/2605.30600#S3.p3.16),[Proposition 2](https://arxiv.org/html/2605.30600#Thmprop2),[Remark 3](https://arxiv.org/html/2605.30600#Thmremark3.p1.2),[Remark 3](https://arxiv.org/html/2605.30600#Thmremark3.p1.6)\.
- Q\. Li, Z\. Chen, and A\. Banerjee \(2025\)Sketched Gaussian mechanism for private federated learning\.Advances in Neural Information Processing Systems39\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p3.4)\.
- I\. Mironov \(2017\)Rényi differential privacy\.InIEEE 30th Computer Security Foundations Symposium,pp\. 263–275\.Cited by:[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.p1.1),[§2\.1](https://arxiv.org/html/2605.30600#S2.SS1.p2.1),[§2\.1](https://arxiv.org/html/2605.30600#S2.SS1.p4.5),[Lemma 7](https://arxiv.org/html/2605.30600#Thmlem7)\.
- M\. Pilanci and M\. J\. Wainwright \(2015\)Randomized sketches of convex programs with sharp guarantees\.IEEE Transactions on Information Theory61\(9\),pp\. 5096–5115\.Cited by:[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.1.p1.7),[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.30600#S2.SS2.p2.6),[§4](https://arxiv.org/html/2605.30600#S4.p3.13),[§5](https://arxiv.org/html/2605.30600#S5.p2.5)\.
- M\. Pilanci and M\. J\. Wainwright \(2016\)Iterative Hessian sketch: Fast and accurate solution approximation for constrained least\-squares\.Journal of Machine Learning Research17\(53\),pp\. 1–38\.Cited by:[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.1.p1.7),[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.3.p3.8),[§E\.1](https://arxiv.org/html/2605.30600#A5.SS1.9.p1.2),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px1.p3.4),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px1.p6.6),[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.Px2.1.p1.9),[Appendix\. H](https://arxiv.org/html/2605.30600#A8.p4.12),[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[footnote 6](https://arxiv.org/html/2605.30600#footnote6)\.
- M\. Pilanci and M\. J\. Wainwright \(2017\)Newton sketch: a near linear\-time optimization algorithm with linear\-quadratic convergence\.SIAM Journal on Optimization27\(1\),pp\. 205–245\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1),[§5](https://arxiv.org/html/2605.30600#S5.p2.5)\.
- A\. Rényi \(1960\)On measures of entropy and information\.InProc\. 4th Berkeley Sympos\. Math\. Statist\. and Prob\., Vol\. I,pp\. 547–561\.Cited by:[Definition 2](https://arxiv.org/html/2605.30600#Thmdefn2.p1.7)\.
- A\. Research \(2017\)Learning with privacy at scale\.Apple Machine Learning Research\.External Links:[Link](https://docs-assets.developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf)Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p2.1)\.
- F\. Research \(2020\)Protecting privacy in facebook mobility data during the covid\-19 response\.External Links:[Link](https://research.facebook.com/blog/2020/6/protecting-privacy-in-facebook-mobility-data-during-the-covid-19-response/)Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p2.1)\.
- \[34\]\(2015\)Rossmann store sales\.Note:Kaggle competitionForecast sales using store, promotion, and competitor dataExternal Links:[Link](https://www.kaggle.com/c/rossmann-store-sales)Cited by:[Table 1](https://arxiv.org/html/2605.30600#A10.T1.12.12.12.2)\.
- ruiqurm \(2018\)Housing price in Beijing\.Note:[https://www\.kaggle\.com/datasets/ruiqurm/lianjia](https://www.kaggle.com/datasets/ruiqurm/lianjia)Accessed: 2022\-07\-28Cited by:[Table 1](https://arxiv.org/html/2605.30600#A10.T1.8.8.8.2)\.
- T\. Sarlos \(2006\)Improved approximation algorithms for large matrices via random projections\.In2006 47th annual IEEE symposium on foundations of computer science,pp\. 143–152\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1)\.
- S\. Security \(2022\)Differential privacy at Snapchat\.External Links:[Link](https://eng.snap.com/differential-privacy-at-snap)Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p2.1)\.
- O\. Shamir \(2015\)The sample complexity of learning linear predictors with the squared loss\.\.Journal of Machine Learning Research16,pp\. 3475–3486\.Cited by:[§2](https://arxiv.org/html/2605.30600#S2.SS0.SSS0.Px2.p1.10)\.
- O\. Sheffet \(2017\)Differentially private ordinary least squares\.InProceedings of the International Conference on Machine Learning,pp\. 3105–3114\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p3.4),[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p1.3),[§4](https://arxiv.org/html/2605.30600#S4.SS0.SSS0.Px1.p1.5)\.
- O\. Sheffet \(2019\)Old techniques in differentially private linear regression\.InAlgorithmic Learning Theory,pp\. 789–827\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p3.4),[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p1.3)\.
- G\. W\. Stewart \(1991\)Perturbation theory for the singular value decomposition\.InSVD and Signal Processing, II: Algorithms, Analysis and Applications,pp\. 99–109\.Cited by:[§C\.3](https://arxiv.org/html/2605.30600#A3.SS3.6.p1.5)\.
- J\. A\. Tropp \(2011\)Improved analysis of the subsampled randomized Hadamard transform\.Advances in Adaptive Data Analysis3\(01n02\),pp\. 115–126\.Cited by:[§2\.2](https://arxiv.org/html/2605.30600#S2.SS2.p1.1)\.
- J\. Upadhyay \(2014\)Randomness\-efficient fast Johnson–Lindenstrauss transform with applications in differential privacy and compressed sensing\.arXiv preprint arXiv:1410\.2470\.Cited by:[§1\.1](https://arxiv.org/html/2605.30600#S1.SS1.p2.1),[§1](https://arxiv.org/html/2605.30600#S1.p4.1),[Figure 1](https://arxiv.org/html/2605.30600#S3.F1),[Figure 1](https://arxiv.org/html/2605.30600#S3.F1.21.7),[§3](https://arxiv.org/html/2605.30600#S3.SS0.SSS0.Px3.p1.7)\.
- S\. Vadhan \(2017\)The complexity of differential privacy\.InTutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich,pp\. 347–450\.Cited by:[§C\.4\.1](https://arxiv.org/html/2605.30600#A3.SS4.SSS1.Px1.p1.16)\.
- J\. L\. Walsh \(1923\)A closed set of normal orthogonal functions\.American Journal of Mathematics45\(1\),pp\. 5–24\.Cited by:[Definition 3](https://arxiv.org/html/2605.30600#Thmdefn3.p1.11)\.
- Y\. Wang \(2018\)Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain\.InConference on Uncertainty in Artificial Intelligence,pp\. 93–103\.Cited by:[§E\.2\.2](https://arxiv.org/html/2605.30600#A5.SS2.SSS2.p1.1),[§2](https://arxiv.org/html/2605.30600#S2.SS0.SSS0.Px2.p1.10),[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p2.14)\.
- L\. Welch \(1974\)Lower bounds on the maximum cross correlation of signals \(corresp\.\)\.IEEE Transactions on Information Theory20\(3\),pp\. 397–399\.Cited by:[Appendix\. F](https://arxiv.org/html/2605.30600#A6.p1.6)\.
- D\. P\. Woodruff \(2014\)Sketching as a tool for numerical linear algebra\.Foundations and Trends® in Theoretical Computer Science10\(1–2\),pp\. 1–157\.Cited by:[§1](https://arxiv.org/html/2605.30600#S1.p1.1)\.
- F\. Zhang and M\. Pilanci \(2023\)Optimal shrinkage for distributed second\-order optimization\.InProceedings of the International Conference on Machine Learning,pp\. 41523–41549\.Cited by:[§4\.1](https://arxiv.org/html/2605.30600#S4.SS1.p1.22)\.
- S\. Zhou, L\. Wasserman, and J\. Lafferty \(2007\)Compressed regression\.Advances in Neural Information Processing Systems20\.Cited by:[§2\.3](https://arxiv.org/html/2605.30600#S2.SS3.p1.3)\.

## Appendix\. AAdditional notation

Random variables are denoted using sans\-serif fonts \(e\.g\.,𝖷,𝗒\\mathsf\{X\},\\mathsf\{y\}\), while their realizations are represented by regular italics \(e\.g\.,X,yX,y\)\. The natural logarithm \(with baseee\) is denoted bylog⁡\(⋅\)\\log\(\\cdot\)\. TheL2L\_\{2\}norm of a vectorv≔\(v1,…,vd\)v\\coloneqq\(v\_\{1\},\\ldots,v\_\{d\}\)is given by\(∑i=1dvi2\)12\\left\(\\sum^\{d\}\_\{i=1\}v^\{2\}\_\{i\}\\right\)^\{\\frac\{1\}\{2\}\}and is denoted by‖v‖\\left\\\|v\\right\\\|, itsL∞L\_\{\\infty\}norm is given bymaxi=1,…,d​\|vi\|\\underset\{i=1,\\ldots,d\}\{\\max\}\\ \\left\|v\_\{i\}\\right\|and is denoted by‖v‖∞\\left\\\|v\\right\\\|\_\{\\infty\}, and its Mahalanobis norm with respect to a PSD matrixBB, namely, the quantity\(v⊤​B​v\)12\\left\(v^\{\\top\}Bv\\right\)^\{\\frac\{1\}\{2\}\}, is denoted by‖v‖B\\left\\\|v\\right\\\|\_\{B\}\. In the case whereAAis a square matrix, its trace and determinant are denoted byTr​\(A\)\\mathrm\{Tr\}\(A\)anddet\(A\)\\det\(A\)respectively, and its minimal and the maximal eigenvalues areλmin​\(A\)\\lambda\_\{\\min\}\(A\)andλmax​\(A\)\\lambda\_\{\\max\}\(A\)respectively\. The minimal and maximal eigenvalues of its Gram matrix are denoted byλminA:=λmin​\(A⊤​A\)\\lambda^\{A\}\_\{\\min\}:=\\lambda\_\{\\min\}\(A^\{\\top\}A\)andλmaxA:=λmax​\(A⊤​A\)\\lambda^\{A\}\_\{\\max\}:=\\lambda\_\{\\max\}\(A^\{\\top\}A\)\. The operator norm of a matrixAAis denoted by‖A‖op\\left\\\|A\\right\\\|\_\{\\mathrm\{op\}\}and is defined bysupv:‖v‖=1​‖A​v‖=λmaxA\\underset\{v:\\left\\\|v\\right\\\|=1\}\{\\sup\}\\ \\left\\\|Av\\right\\\|=\\sqrt\{\\lambda^\{A\}\_\{\\max\}\}and the Frobenius norm by‖A‖fro:=\(∑i=1n∑j=1dai​j2\)1/2\\left\\\|A\\right\\\|\_\{\\mathrm\{fro\}\}\\hskip\-2\.5pt:=\\hskip\-2\.5pt\\left\(\\hskip\-1\.49994pt\\sum^\{n\}\_\{i=1\}\\hskip\-1\.00006pt\\sum^\{d\}\_\{j=1\}\\hskip\-1\.00006pta^\{2\}\_\{ij\}\\hskip\-1\.49994pt\\right\)^\{\\nicefrac\{\{1\}\}\{\{2\}\}\}\. We use the notationA⪰0A\\succeq 0andA≻0A\\succ 0for denoting situations in whichAAis Positive Semi\-Definite \(PSD\) and in whichAAis Positive Definite \(PD\)\. The Loewner order is defined in the usual way, whereB⪯AB\\preceq AdenotesA−B⪰0A\-B\\succeq 0\. We usually denote our dataset\{xi\}i=1n\\left\\\{x\_\{i\}\\right\\\}^\{n\}\_\{i=1\}where eachxi∈ℝdx\_\{i\}\\in\\mathbb\{R\}^\{d\}in the matrix formX=\(x1,x2,…,xn\)⊤X=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{n\}\)^\{\\top\}\. The set of integer numbers from11tonnis denoted by\[n\]\[n\]\. The all zeros column vector of sizeddis denoted by0→d≔\(0,0,…,0\)⊤\\vec\{0\}\_\{d\}\\coloneqq\(0,0,\\ldots,0\)^\{\\top\}, and a matrix of sizea×ba\\times bfull of zeros is denoted by0a×b0\_\{a\\times b\}\. We denote byN​\(0,𝕀k1×k2\)\\pazocal\{N\}\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times k\_\{2\}\}\)ak1×k2k\_\{1\}\\times k\_\{2\}matrix consisting of i\.i\.d\. Gaussian elements with zero mean and unit variance\. Thek×kk\\times kidentity matrix is denoted by𝕀k\\mathbb\{I\}\_\{k\}\. We writey​\(x\)=O​\(x\)y\(x\)=O\(x\)if∃c\>0,x0\>0\\exists c\>0,x\_\{0\}\>0such that\|y​\(x\)\|≤c​x\|y\(x\)\|\\leq cxfor allx≥x0x\\geq x\_\{0\}\. We writey​\(x\)=o​\(x\)y\(x\)=o\(x\)iflimx→∞​y​\(x\)x=0\\underset\{x\\to\\infty\}\{\\lim\}\\frac\{y\(x\)\}\{x\}=0\. We writey​\(x\)=Ω​\(x\)y\(x\)=\\Omega\(x\)if∃c\>0,x0\>0\\exists c\>0,x\_\{0\}\>0such thaty​\(x\)≥c​xy\(x\)\\geq cxfor allx≥x0x\\geq x\_\{0\}\. We writey​\(x\)=Θ​\(x\)y\(x\)=\\Theta\(x\)ify​\(x\)=O​\(x\)y\(x\)=O\(x\)andy​\(x\)=Ω​\(x\)y\(x\)=\\Omega\(x\)\. Given two probability distributionsℙ\\mathbb\{P\}andℚ\\mathbb\{Q\}and a scalar parameterλ∈\[0,1\]\\lambda\\in\[0,1\], the notationλ​ℙ\+\(1−λ\)​ℚ\\lambda\\mathbb\{P\}\+\(1\-\\lambda\)\\mathbb\{Q\}denotes the distribution of the random variable𝗑=𝖻𝗉\+\(1−𝖻\)​𝗊\\mathsf\{x\}=\\mathsf\{b\}\\mathsf\{p\}\+\(1\-\\mathsf\{b\}\)\\mathsf\{q\}for𝗉∼ℙ,𝗊∼ℚ,𝖻∼Ber​\(λ\)\\mathsf\{p\}\\sim\\mathbb\{P\},\\mathsf\{q\}\\sim\\mathbb\{Q\},\\mathsf\{b\}\\sim\\mathrm\{Ber\}\(\\lambda\)and𝗉⟂⟂𝗊,𝗊⟂⟂𝖻,𝗉⟂⟂𝖻\\mathsf\{p\}\\mathchoice\{\\mathrel\{\\hbox to0\.0pt\{$\\displaystyle\\perp$\\hss\}\\mkern 3\.0mu\{\\displaystyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\textstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\textstyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\scriptstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\scriptstyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\scriptscriptstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\scriptscriptstyle\\perp\}\}\}\\mathsf\{q\},\\mathsf\{q\}\\mathchoice\{\\mathrel\{\\hbox to0\.0pt\{$\\displaystyle\\perp$\\hss\}\\mkern 3\.0mu\{\\displaystyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\textstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\textstyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\scriptstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\scriptstyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\scriptscriptstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\scriptscriptstyle\\perp\}\}\}\\mathsf\{b\},\\mathsf\{p\}\\mathchoice\{\\mathrel\{\\hbox to0\.0pt\{$\\displaystyle\\perp$\\hss\}\\mkern 3\.0mu\{\\displaystyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\textstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\textstyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\scriptstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\scriptstyle\\perp\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{$\\scriptscriptstyle\\perp$\\hss\}\\mkern 3\.0mu\{\\scriptscriptstyle\\perp\}\}\}\\mathsf\{b\}\.

## Appendix\. BProperties of Subsampled Randomized Hadamard Transform

We first repeat the formal definition of a Hadamard matrix\. Then, we provide an auxiliary lemma regarding SRHT, which will be proved useful in our analysis\.

###### Definition 4\(Hadamard Matrix\)\.

An Hadamard matrix of size2​n2nis denoted byH2​nH\_\{2n\}and is defined via the recursive formula

H~2​n=\(H~nH~nH~n−H~n\),H~1=\(1\),Hn=1n​H~n\.\\displaystyle\\widetilde\{H\}\_\{2n\}=\\begin\{pmatrix\}\\widetilde\{H\}\_\{n\}&\\widetilde\{H\}\_\{n\}\\\\ \\widetilde\{H\}\_\{n\}&\-\\widetilde\{H\}\_\{n\}\\end\{pmatrix\},\\ \\ \\ \\ \\widetilde\{H\}\_\{1\}=\(1\),\\ \\ \\ \\ H\_\{n\}=\\frac\{1\}\{\\sqrt\{n\}\}\\widetilde\{H\}\_\{n\}\.\(13\)

###### Lemma 2\.

For anyn≥1n\\hskip\-1\.99997pt\\geq\\hskip\-1\.99997pt1andk∈\[n\]k\\hskip\-1\.99997pt\\in\\hskip\-1\.99997pt\[n\], anySf∼SRHT​\(k,n\)S\_\{\\mathrm\{f\}\}\\sim\\mathrm\{SRHT\}\(k,\\hskip\-1\.00006ptn\)satisfies the following properties\.

1. \(1\)For allj1,j2∈\[k\]j\_\{1\},j\_\{2\}\\in\[k\]such thatj1≠j2j\_\{1\}\\neq j\_\{2\}, thej1j\_\{1\}\-th row\(Sf\)j1,:\(S\_\{\\mathrm\{f\}\}\)\_\{j\_\{1\},:\}and thej2j\_\{2\}\-th row\(Sf\)j2,:\(S\_\{\\mathrm\{f\}\}\)\_\{j\_\{2\},:\}are orthogonal\.
2. \(2\)For alli∈\[n\]i\\in\[n\], theii\-th column\(Sf\):,i\(S\_\{\\mathrm\{f\}\}\)\_\{:,i\}satisfies‖\(Sf\):,i‖=1\\\|\(S\_\{\\mathrm\{f\}\}\)\_\{:,i\}\\\|=1\.

###### Proof\.

We start with establishing\(1\)\(1\)\. To that end, recall that anySf∈SRHT​\(k,n\)S\_\{\\mathrm\{f\}\}\\in\\mathrm\{SRHT\}\(k,n\)is given by

Sf=nk​Pk×n​Hn​BnS\_\{\\mathrm\{f\}\}=\\sqrt\{\\frac\{n\}\{k\}\}P\_\{k\\times n\}H\_\{n\}B\_\{n\}\(14\)whereBnB\_\{n\}is a diagonal matrix with entries\{±1\}\\left\\\{\\pm 1\\right\\\},HnH\_\{n\}is a normalized Hadamard matrix, andPk×nP\_\{k\\times n\}is a row subsampling matrix with a single distinct entry equal to11in each row\. Then, denoting the rows of the matrixHnH\_\{n\}by\{hi⊤\}i=1n\\left\\\{h^\{\\top\}\_\{i\}\\right\\\}^\{n\}\_\{i=1\}where eachhi∈ℝn×1h\_\{i\}\\in\\mathbb\{R\}^\{n\\times 1\}we get that the rows ofSfS\_\{\\mathrm\{f\}\}are given by\{nk​hji⊤​Bn\}i=1k\\left\\\{\\sqrt\{\\frac\{n\}\{k\}\}h^\{\\top\}\_\{j\_\{i\}\}B\_\{n\}\\right\\\}^\{k\}\_\{i=1\}where eachji∈\[n\]j\_\{i\}\\in\[n\]is a distinct index from\[n\]\[n\]\. Thus, we get that the inner product between two different arbitrary rows with indicesj1j\_\{1\}andj2j\_\{2\}such thatj1≠j2j\_\{1\}\\neq j\_\{2\}is given by

nk​hj1⊤​Bn​Bn⊤​hj2=nk​hj1⊤​hj2=0\\frac\{n\}\{k\}h^\{\\top\}\_\{j\_\{1\}\}B\_\{n\}B^\{\\top\}\_\{n\}h\_\{j\_\{2\}\}=\\frac\{n\}\{k\}h^\{\\top\}\_\{j\_\{1\}\}h\_\{j\_\{2\}\}=0\(15\)and where the first equality is sinceBn​Bn⊤=𝕀nB\_\{n\}B^\{\\top\}\_\{n\}=\\mathbb\{I\}\_\{n\}by the definition ofBnB\_\{n\}, and the last equality is since the rows of the Hadamard matrix are orthogonal\. This proves point \(1\)\. Moreover, forj1=j2j\_\{1\}=j\_\{2\}we note that this inner product is given bynk\\frac\{n\}\{k\}\.

To prove \(2\), sinceHnH\_\{n\}is a normalized Hadamard matrix, and since the entries of𝖡n\\mathsf\{B\}\_\{n\}and𝖯k×n\\mathsf\{P\}\_\{k\\times n\}are either0’s or either±1\\pm 1, the entries ofSfS\_\{\\mathrm\{f\}\}belong to\{±1k\}\\left\\\{\\pm\\frac\{1\}\{\\sqrt\{k\}\}\\right\\\}\. Since each column ofSfS\_\{\\mathrm\{f\}\}contains exactlykkelements, the norm of each column is11\. ∎

## Appendix\. CProof of Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1)

Our proof will use the next standard formula for the Rényi divergence between twodd\-dimensional multivariate Gaussian distributions\[Gilet al\.,[2013](https://arxiv.org/html/2605.30600#bib.bib113)\]

Dα\(N\(0→d,Σ1\)∥N\(0→d,Σ2\)\)=−12​\(α−1\)log\(det​\(Σ1\+α​\(Σ2−Σ1\)\)\(det​\(Σ1\)\)1−α​\(det​\(Σ2\)\)α\)\{D\_\{\\alpha\}\\left\(\\pazocal\{N\}\\left\(\\vec\{0\}\_\{d\},\\Sigma\_\{1\}\\right\)\\middle\\\|\\pazocal\{N\}\\left\(\\vec\{0\}\_\{d\},\\Sigma\_\{2\}\\right\)\\right\)\}=\-\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\mathrm\{det\}\\left\(\\Sigma\_\{1\}\+\\alpha\\left\(\\Sigma\_\{2\}\-\\Sigma\_\{1\}\\right\)\\right\)\}\{\\left\(\\mathrm\{det\}\\left\(\\Sigma\_\{1\}\\right\)\\right\)^\{1\-\\alpha\}\\left\(\\mathrm\{det\}\\left\(\\Sigma\_\{2\}\\right\)\\right\)^\{\\alpha\}\}\\right\)\\ \\ \\\(16\)which holds for allα\\alphasuch thatα​Σ1−1\+\(1−α\)​Σ2−1=Σ2−1\+α​\(Σ1−1−Σ2−1\)≻0\\alpha\\Sigma^\{\-1\}\_\{1\}\+\(1\-\\alpha\)\\Sigma^\{\-1\}\_\{2\}=\\Sigma^\{\-1\}\_\{2\}\+\\alpha\\left\(\\Sigma^\{\-1\}\_\{1\}\-\\Sigma^\{\-1\}\_\{2\}\\right\)\\succ 0\.

We will first provide a proof of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)\. To that end, we start with providing a tight characterization of the privacy guarantee ofMSf\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}, which we then simplify towards the form presented in Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)\. Then, we provide some additional auxiliary lemmas, which are necessary for establishing the overall privacy guarantees\. For proving the final claim, we will show that the assumptions of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)hold in the setting of[FastMix](https://arxiv.org/html/2605.30600#alg1), which, under the right choice of parameters, makes its output\(ε​\(ω,γ,k1,δ\),δ\)\(\\varepsilon\(\\omega,\\gamma,k\_\{1\},\\delta\),\\delta\)\-DP\.

### C\.1Proof of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)

Without loss of generality, we considerX−X′=e1​x1⊤X\-X^\{\\prime\}=e\_\{1\}x^\{\\top\}\_\{1\}; in simpler terms,X′X^\{\\prime\}is obtained by zeroing out the first row inXX\. We introduce some notation for convenience before delving into the proof\. Theit​hi^\{th\}column ofSfS\_\{\\mathrm\{f\}\}isui≔Sf​eiu\_\{i\}\\coloneq S\_\{\\mathrm\{f\}\}e\_\{i\}\. The matricesZ,Z′Z,Z^\{\\prime\}are the outputs ofX,X′X,X^\{\\prime\}transformed usingSfS\_\{\\mathrm\{f\}\}respectively as

Z≔Sf​XandZ′≔Sf​X′=Z−Sf​e1​x1⊤=Z−u1​x1⊤\.Z\\coloneq S\_\{\\mathrm\{f\}\}X\\qquad\\text\{and\}\\qquad Z^\{\\prime\}\\coloneq S\_\{\\mathrm\{f\}\}X^\{\\prime\}=Z\-S\_\{\\mathrm\{f\}\}e\_\{1\}x^\{\\top\}\_\{1\}=Z\-u\_\{1\}x^\{\\top\}\_\{1\}\.\(17\)Finally, we will use the notationM≔\(Z⊤​Z\+σ2​𝕀d\)−1M\\coloneq\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\)^\{\-1\}andδi≔Z⊤​Sf​ei−xi\\delta\_\{i\}\\coloneq Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}throughout\.

We begin by observing the difference betweenZ′⁣⊤​Z′Z^\{\\prime\\top\}Z^\{\\prime\}andZ⊤​ZZ^\{\\top\}Zas these play a key role in the Rényi divergence calculation that follows\. By definition,

Z′⁣⊤​Z′=\(Z−u1​x1⊤\)⊤​\(Z−u1​x1⊤\)=Z⊤​Z\+‖u1‖2​x1​x1⊤−Z⊤​u1​x1⊤−\(Z⊤​u1​x1⊤\)⊤\.\\displaystyle Z^\{\\prime\\top\}Z^\{\\prime\}=\\left\(Z\-u\_\{1\}x^\{\\top\}\_\{1\}\\right\)^\{\\top\}\\left\(Z\-u\_\{1\}x^\{\\top\}\_\{1\}\\right\)=Z^\{\\top\}Z\+\\left\\\|u\_\{1\}\\right\\\|^\{2\}x\_\{1\}x^\{\\top\}\_\{1\}\-Z^\{\\top\}u\_\{1\}x^\{\\top\}\_\{1\}\-\\left\(Z^\{\\top\}u\_\{1\}x^\{\\top\}\_\{1\}\\right\)^\{\\top\}\.\(18\)Definex~1=‖u1‖​x1\\widetilde\{x\}\_\{1\}=\\\|u\_\{1\}\\\|x\_\{1\}andu~1=u1‖u1‖\\widetilde\{u\}\_\{1\}=\\frac\{u\_\{1\}\}\{\\\|u\_\{1\}\\\|\}\. Then,

Z′⁣⊤​Z′=Z⊤​Z\+x~1​x~1⊤−Z⊤​u~1​x~1⊤−\(Z⊤​u~1​x~1⊤\)⊤\.Z^\{\\prime\\top\}Z^\{\\prime\}=Z^\{\\top\}Z\+\\widetilde\{x\}\_\{1\}\\widetilde\{x\}\_\{1\}^\{\\top\}\-Z^\{\\top\}\\widetilde\{u\}\_\{1\}\\widetilde\{x\}\_\{1\}^\{\\top\}\-\(Z^\{\\top\}\\widetilde\{u\}\_\{1\}\\widetilde\{x\}\_\{1\}^\{\\top\}\)^\{\\top\}\.Since‖u1‖≤1\\\|u\_\{1\}\\\|\\leq 1, we have‖x~1‖=‖u1‖​‖x1‖≤1\\\|\\widetilde\{x\}\_\{1\}\\\|=\\\|u\_\{1\}\\\|\\\|x\_\{1\}\\\|\\leq 1by the preconditions onSfS\_\{\\mathrm\{f\}\}andXX, and‖u~1‖=1\\\|\\widetilde\{u\}\_\{1\}\\\|=1\.444If‖u1‖=0\\left\\\|u\_\{1\}\\right\\\|=0, thenZ′=ZZ^\{\\prime\}=Zand the divergence is0\. Thus, we assume that‖u1‖\>0\\left\\\|u\_\{1\}\\right\\\|\>0\.The remainder of the proof only uses bounds on‖x~1‖\\\|\\widetilde\{x\}\_\{1\}\\\|and‖u~1‖\\\|\\widetilde\{u\}\_\{1\}\\\|, and therefore without loss of generality, we assume that‖u1‖=1,‖x1‖≤1\\\|u\_\{1\}\\\|=1,\\\|x\_\{1\}\\\|\\leq 1and write

Z′⁣⊤​Z′=Z⊤​Z\+x1​x1⊤−Z⊤​u1​x1⊤−\(Z⊤​u1​x1⊤\)⊤\.\\displaystyle Z^\{\\prime\\top\}Z^\{\\prime\}=Z^\{\\top\}Z\+x\_\{1\}x^\{\\top\}\_\{1\}\-Z^\{\\top\}u\_\{1\}x^\{\\top\}\_\{1\}\-\\left\(Z^\{\\top\}u\_\{1\}x^\{\\top\}\_\{1\}\\right\)^\{\\top\}\.\(19\)
Next, we note that the difference betweenZ′⁣⊤​Z′⁣⊤Z^\{\\prime\\top\}Z^\{\\prime\\top\}andZ⊤​ZZ^\{\\top\}Zcan be expressed as a rank\-2 update\. More precisely, defineU∈ℝd×2≔\[−Z⊤​u1\+x1,x1\]U\\in\\mathbb\{R\}^\{d\\times 2\}\\coloneq\[\-Z^\{\\top\}u\_\{1\}\+x\_\{1\},x\_\{1\}\]andV∈ℝd×2≔\[x1,−Z⊤​u1\]V\\in\\mathbb\{R\}^\{d\\times 2\}\\coloneq\[x\_\{1\},\-Z^\{\\top\}u\_\{1\}\]\. Then,

U​V⊤\\displaystyle UV^\{\\top\}=−Z⊤​u1​x1⊤\+x1​x1⊤−x1​u1⊤​Z=Z′⁣⊤​Z′⁣⊤−Z⊤​Z\.\\displaystyle=\-Z^\{\\top\}u\_\{1\}x\_\{1\}^\{\\top\}\+x\_\{1\}x\_\{1\}^\{\\top\}\-x\_\{1\}u\_\{1\}^\{\\top\}Z=Z^\{\\prime\\top\}Z^\{\\prime\\top\}\-Z^\{\\top\}Z~\.
Since𝗀\\mathsf\{g\}andξ\\xiare independent standard normal vectors this implies that

\(𝗀⊤​Sf​X\+σ​ξ⊤\)⊤\\displaystyle\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\sigma\\xi^\{\\top\}\)^\{\\top\}∼N​\(0,Z⊤​Z\+σ2​𝕀d\),\\displaystyle\\sim\\pazocal\{N\}\\left\(0,Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\),\(20\)\(𝗀⊤​Sf​X′\+σ​ξ⊤\)⊤\\displaystyle\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\+\\sigma\\xi^\{\\top\}\)^\{\\top\}∼N​\(0,Z′⁣⊤​Z′\+σ2​𝕀d\)\\displaystyle\\sim\\pazocal\{N\}\\left\(0,Z^\{\\prime\\top\}Z^\{\\prime\}\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\(21\)=\(d\)​N​\(0,Z⊤​Z\+σ2​𝕀d\+x1​x1⊤−Z⊤​u1​x1⊤−\(Z⊤​u1​x1⊤\)⊤\)\.\\displaystyle\\overset\{\(d\)\}\{=\}\\pazocal\{N\}\\left\(0,Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\+x\_\{1\}x^\{\\top\}\_\{1\}\-Z^\{\\top\}u\_\{1\}x^\{\\top\}\_\{1\}\-\\left\(Z^\{\\top\}u\_\{1\}x^\{\\top\}\_\{1\}\\right\)^\{\\top\}\\right\)\.\(22\)Using \([16](https://arxiv.org/html/2605.30600#A3.E16)\), we have

Dα\(𝗀⊤SfX\+σξ⊤∥𝗀⊤SfX′\+σξ⊤\)=−12​\(α−1\)​log⁡\(det​\(Z⊤​Z\+σ2​𝕀d\+α​\(Z′⁣⊤​Z′−Z⊤​Z\)\)\(det​\(Z⊤​Z\+σ2​𝕀d\)\)1−α​\(det​\(Z′⁣⊤​Z′\+σ2​𝕀d\)\)α\)\.\{D\_\{\\alpha\}\\left\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\sigma\\xi^\{\\top\}\\middle\\\|\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\+\\sigma\\xi^\{\\top\}\\right\)\}\\\\ =\-\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\+\\alpha\(Z^\{\\prime\\top\}Z^\{\\prime\}\-Z^\{\\top\}Z\)\\right\)\}\{\\left\(\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\right\)^\{1\-\\alpha\}\\left\(\\mathrm\{det\}\\left\(Z^\{\\prime\\top\}Z^\{\\prime\}\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\right\)^\{\\alpha\}\}\\right\)\.\(23\)
Now, using the matrix determinant lemma anddet​\(A​B\)=det​\(A\)​det​\(B\)\\mathrm\{det\}\(AB\)=\\mathrm\{det\}\(A\)\\mathrm\{det\}\(B\)we get for anyβ≥0\\beta\\geq 0

det​\(Z⊤​Z\+σ2​𝕀d\+β​U​V⊤\)=det​\(Z⊤​Z\+σ2​𝕀d\)​det​\(𝕀2\+β​V⊤​M​U\)\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\+\\beta UV^\{\\top\}\\right\)=\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\mathrm\{det\}\\left\(\\mathbb\{I\}\_\{2\}\+\\beta V^\{\\top\}MU\\right\)\(24\)
where the inner matrixV⊤​M​UV^\{\\top\}MUcan be rewritten explicitly as

V⊤​M​U=\[x1⊤​M​\(−Z⊤​u1\+x1\)x1⊤​M​x1\(u1\)⊤​Z​M​\(Z⊤​u1−x1\)−\(u1\)⊤​Z​M​x1\]\.V^\{\\top\}MU=\\begin\{bmatrix\}x^\{\\top\}\_\{1\}M\(\-Z^\{\\top\}u\_\{1\}\+x\_\{1\}\)&x^\{\\top\}\_\{1\}Mx\_\{1\}\\\\ \(u\_\{1\}\)^\{\\top\}ZM\(Z^\{\\top\}u\_\{1\}\-x\_\{1\}\)&\-\(u\_\{1\}\)^\{\\top\}ZMx\_\{1\}\\end\{bmatrix\}\.\(25\)
Define

s1:=x1⊤​M​x1,s2:=\(u1\)⊤​Z​M​Z⊤​u1,c≔−x1⊤​M​Z⊤​u1=−\(u1\)⊤​Z​M​x1\.s\_\{1\}:=x^\{\\top\}\_\{1\}Mx\_\{1\},\\qquad s\_\{2\}:=\(u\_\{1\}\)^\{\\top\}ZMZ^\{\\top\}u\_\{1\},\\qquad c\\coloneq\-x^\{\\top\}\_\{1\}MZ^\{\\top\}u\_\{1\}=\-\(u\_\{1\}\)^\{\\top\}ZMx\_\{1\}\.\(26\)
Then,

det​\(𝕀2\+β​V⊤​M​U\)=det​\(\(1\+β​s1\+β​cβ​s1β​s2\+β​c1\+β​c\)\)=1\+β​s1\+2​β​c\+β2​c2−β2​s1​s2\.\\mathrm\{det\}\\left\(\\mathbb\{I\}\_\{2\}\+\\beta V^\{\\top\}MU\\right\)=\\mathrm\{det\}\\left\(\\begin\{pmatrix\}1\+\\beta s\_\{1\}\+\\beta c&\\beta s\_\{1\}\\\\ \\beta s\_\{2\}\+\\beta c&1\+\\beta c\\end\{pmatrix\}\\right\)=1\+\\beta s\_\{1\}\+2\\beta c\+\\beta^\{2\}c^\{2\}\-\\beta^\{2\}s\_\{1\}s\_\{2\}~\.\(27\)Applying this todet​\(Z′⁣⊤​Z′\+σ2​𝕀d\)\\mathrm\{det\}\\left\(Z^\{\\prime\\top\}Z^\{\\prime\}\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)withβ=1\\beta=1, we get

det​\(Z′⁣⊤​Z′\+σ2​𝕀d\)\\displaystyle\\mathrm\{det\}\\left\(Z^\{\\prime\\top\}Z^\{\\prime\}\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)=det​\(Z⊤​Z\+σ2​𝕀d\)​\(1\+s1\+2​c\+c2−s1​s2\),\\displaystyle=\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\left\(1\+s\_\{1\}\+2c\+c^\{2\}\-s\_\{1\}s\_\{2\}\\right\)~,\(28\)and applying this withβ=α\\beta=\\alphafor the numerator in \([23](https://arxiv.org/html/2605.30600#A3.E23)\), we obtain

det​\(Z⊤​Z\+σ2​𝕀d\+α​\(Z′⁣⊤​Z′−Z⊤​Z\)\)=det​\(Z⊤​Z\+σ2​𝕀d\)​\(1\+α​s1\+2​α​c\+α2​c2−α2​s1​s2\)\.\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\+\\alpha\(Z^\{\\prime\\top\}Z^\{\\prime\}\-Z^\{\\top\}Z\)\\right\)=\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\left\(1\+\\alpha s\_\{1\}\+2\\alpha c\+\\alpha^\{2\}c^\{2\}\-\\alpha^\{2\}s\_\{1\}s\_\{2\}\\right\)~\.\(29\)Substituting these inside \([23](https://arxiv.org/html/2605.30600#A3.E23)\) yields

Dα\(𝗀⊤SfX\+σξ⊤\\displaystyle D\_\{\\alpha\}\\left\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\sigma\\xi^\{\\top\}\\right\.∥𝗀⊤SfX′\+σξ⊤\)\\displaystyle\\\|\\left\.\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\+\\sigma\\xi^\{\\top\}\\right\)\(30\)=−12​\(α−1\)​log⁡\(det​\(Z⊤​Z\+σ2​𝕀d\+α​\(Z′⁣⊤​Z′−Z⊤​Z\)\)\(det​\(Z⊤​Z\+σ2​𝕀d\)\)1−α​\(det​\(Z′⁣⊤​Z′\+σ2​𝕀d\)\)α\)\\displaystyle=\-\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\+\\alpha\(Z^\{\\prime\\top\}Z^\{\\prime\}\-Z^\{\\top\}Z\)\\right\)\}\{\\left\(\\mathrm\{det\}\\left\(Z^\{\\top\}Z\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\right\)^\{1\-\\alpha\}\\left\(\\mathrm\{det\}\\left\(Z^\{\\prime\\top\}Z^\{\\prime\}\+\\sigma^\{2\}\\mathbb\{I\}\_\{d\}\\right\)\\right\)^\{\\alpha\}\}\\right\)\(31\)=12​\(α−1\)​log⁡\(\(1−\(s1​s2−c2\)\+\(2​c\+s1\)\)α1−α2​\(s1​s2−c2\)\+α​\(2​c\+s1\)\)\.\\displaystyle=\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\left\(1\-\(s\_\{1\}s\_\{2\}\-c^\{2\}\)\+\\left\(2c\+s\_\{1\}\\right\)\\right\)^\{\\alpha\}\}\{1\-\\alpha^\{2\}\\left\(s\_\{1\}s\_\{2\}\-c^\{2\}\\right\)\+\\alpha\\left\(2c\+s\_\{1\}\\right\)\}\\right\)\.\(32\)
Our goal now is to obtain a closed\-form upper bound on \([32](https://arxiv.org/html/2605.30600#A3.E32)\), which depends only on global properties ofSf​XS\_\{\\mathrm\{f\}\}Xsuch as the row norm boundCSf​X2\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}and the minimal eigenvalueλminSf​X\\lambda^\{S\_\{\\mathrm\{f\}\}X\}\_\{\\min\}\. To do this, we obtain bounds ons1​s2−c2s\_\{1\}s\_\{2\}\-c^\{2\}and2​c\+s12c\+s\_\{1\}that solely depend on these quantities, and then obtain an upper bound for \([32](https://arxiv.org/html/2605.30600#A3.E32)\) by casting it as an optimization problem\.

Firstly,s1​s2−c2s\_\{1\}s\_\{2\}\-c^\{2\}is non\-negative due to the Cauchy\-Schwarz inequality\. Then, we further note that the next upper bound

s1​s2−c2\\displaystyle s\_\{1\}s\_\{2\}\-c^\{2\}=\(x1⊤​M​x1\)​\(u1⊤​Z​M​Z⊤​u1\)−\(x1⊤​M​Z⊤​u1\)2\\displaystyle=\\left\(x^\{\\top\}\_\{1\}Mx\_\{1\}\\right\)\\left\(u^\{\\top\}\_\{1\}ZMZ^\{\\top\}u\_\{1\}\\right\)\-\\left\(x^\{\\top\}\_\{1\}MZ^\{\\top\}u\_\{1\}\\right\)^\{2\}\(33\)=\(x1⊤​M​x1\)​\(\(x1\+δ1\)⊤​M​\(x1\+δ1\)\)−\(x1⊤​M​\(x1\+δ1\)\)2\\displaystyle=\\left\(x^\{\\top\}\_\{1\}Mx\_\{1\}\\right\)\\left\(\(x\_\{1\}\+\\delta\_\{1\}\)^\{\\top\}M\(x\_\{1\}\+\\delta\_\{1\}\)\\right\)\-\\left\(x^\{\\top\}\_\{1\}M\(x\_\{1\}\+\\delta\_\{1\}\)\\right\)^\{2\}\(34\)=\(x1⊤​M​x1\)​\(δ1⊤​M​δ1\)−\(x1⊤​M​δ1\)2\\displaystyle=\\left\(x^\{\\top\}\_\{1\}Mx\_\{1\}\\right\)\\left\(\\delta\_\{1\}^\{\\top\}M\\delta\_\{1\}\\right\)\-\\left\(x^\{\\top\}\_\{1\}M\\delta\_\{1\}\\right\)^\{2\}\(35\)≤\(x1⊤​M​x1\)​\(δ1⊤​M​δ1\)\\displaystyle\\leq\\left\(x^\{\\top\}\_\{1\}Mx\_\{1\}\\right\)\\left\(\\delta\_\{1\}^\{\\top\}M\\delta\_\{1\}\\right\)\(36\)≤‖x1‖2​‖δ1‖2​\(λmax​\(M\)\)2\\displaystyle\\leq\\left\\\|x\_\{1\}\\right\\\|^\{2\}\\left\\\|\\delta\_\{1\}\\right\\\|^\{2\}\\left\(\\lambda\_\{\\max\}\(M\)\\right\)^\{2\}\(37\)≤‖Z⊤​Sf​e1−x1‖2\(λminZ\+σ2\)2\\displaystyle\\leq\\frac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{1\}\-x\_\{1\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\(38\)≤maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2\\displaystyle\\leq\\underset\{i\\in\[n\]\}\{\\max\}\\ \\frac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\(39\)holds by the definition ofδ1\\delta\_\{1\}and the assumption‖x1‖2≤1\\left\\\|x\_\{1\}\\right\\\|^\{2\}\\leq 1\.

Next, we work with2​c\+s12c\+s\_\{1\}\. Note thatδ1=Z⊤​u1−x1\\delta\_\{1\}=Z^\{\\top\}u\_\{1\}\-x\_\{1\}and

2​c\+s1=−2​δ1⊤​M​x1−2​‖x1‖M2\+‖x1‖M2=−2​δ1⊤​M​x1−‖x1‖M2\.2c\+s\_\{1\}=\-2\\delta\_\{1\}^\{\\top\}Mx\_\{1\}\-2\\\|x\_\{1\}\\\|^\{2\}\_\{M\}\+\\\|x\_\{1\}\\\|^\{2\}\_\{M\}=\-2\\delta\_\{1\}^\{\\top\}Mx\_\{1\}\-\\\|x\_\{1\}\\\|^\{2\}\_\{M\}~\.Therefore,

−‖x1‖M2−2​‖x1‖M​‖δ1‖M≤2​c\+s1≤−‖x1‖M2\+2​‖x1‖M​‖δ1‖M,\-\\left\\\|x\_\{1\}\\right\\\|^\{2\}\_\{M\}\-2\\left\\\|x\_\{1\}\\right\\\|\_\{M\}\\left\\\|\\delta\_\{1\}\\right\\\|\_\{M\}\\leq 2c\+s\_\{1\}\\leq\-\\left\\\|x\_\{1\}\\right\\\|^\{2\}\_\{M\}\+2\\left\\\|x\_\{1\}\\right\\\|\_\{M\}\\left\\\|\\delta\_\{1\}\\right\\\|\_\{M\},\(40\)and which we can further bound as

2​c\+s1\\displaystyle 2c\+s\_\{1\}≤2​‖x1‖M​‖δ1‖M≤2​‖Z⊤​Sf​e1−x1‖λminZ\+σ2≤2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖λminZ\+σ2,\\displaystyle\\leq 2\\left\\\|x\_\{1\}\\right\\\|\_\{M\}\\left\\\|\\delta\_\{1\}\\right\\\|\_\{M\}\\leq\\frac\{2\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{1\}\-x\_\{1\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\leq 2\\cdot\\frac\{\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\},\(41\)2​c\+s1\\displaystyle 2c\+s\_\{1\}≥−1λminZ\+σ2−2​‖Z⊤​Sf​e1−x1‖λminZ\+σ2≥−1\+2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖λminZ\+σ2\.\\displaystyle\\geq\-\\frac\{1\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-2\\frac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{1\}\-x\_\{1\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\geq\-\\frac\{1\+2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\.\(42\)
From \([32](https://arxiv.org/html/2605.30600#A3.E32)\), we obtain natural constraints for the quantities involved:

1−\(s1​s2−c2\)\+s1\+2​c\>0,1−α2​\(s1​s2−c2\)\+α​\(2​c\+s1\)\>0\.1\-\(s\_\{1\}s\_\{2\}\-c^\{2\}\)\+s\_\{1\}\+2c\>0,\\quad 1\-\\alpha^\{2\}\\left\(s\_\{1\}s\_\{2\}\-c^\{2\}\\right\)\+\\alpha\\left\(2c\+s\_\{1\}\\right\)\>0\.\(43\)
With this, we obtain an upper bound for \([32](https://arxiv.org/html/2605.30600#A3.E32)\) by posing this as an optimization problem below; this can be viewed as the tightest upper bound possible with the given constraints:

sup\(s1,s2,c\)∈𝒟1\{log⁡\(\(1\+s1\+2​c\+c2−s1​s2\)α1\+α​\(s1\+2​c\)\+α2​\(c2−s1​s2\)\)\},\\sup\_\{\(s\_\{1\},s\_\{2\},c\)\\in\\mathcal\{D\}\_\{1\}\}\\;\\Bigg\\\{\\log\\left\(\\frac\{\\big\(1\+s\_\{1\}\+2c\+c^\{2\}\-s\_\{1\}s\_\{2\}\\big\)^\{\\alpha\}\}\{1\+\\alpha\\left\(s\_\{1\}\+2c\\right\)\+\\alpha^\{2\}\\left\(c^\{2\}\-s\_\{1\}s\_\{2\}\\right\)\}\\right\)\\Bigg\\\},\(44\)where

𝒟1\\displaystyle\\mathcal\{D\}\_\{1\}:=\{\(s1,s2,c\)∈ℝ3:−maxi∈\[n\]‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2≤c2−s1s2≤0,α\>1,\\displaystyle:=\\Bigg\\\{\(s\_\{1\},s\_\{2\},c\)\\in\\mathbb\{R\}^\{3\}:\-\\underset\{i\\in\[n\]\}\{\\max\}\\ \\frac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\\leq c^\{2\}\-s\_\{1\}s\_\{2\}\\leq 0,\\ \\alpha\>1,−1\+2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖λminZ\+σ2≤s1\+2​c≤2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖λminZ\+σ2,\\displaystyle\\qquad\\qquad\-\\frac\{1\+2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\leq s\_\{1\}\+2c\\leq\\frac\{2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\},1\+s1\+2c\+c2−s1s2\>0,1\+α\(s1\+2c\)\+α2\(c2−s1s2\)\>0\}\.\\displaystyle\\qquad\\qquad 1\+s\_\{1\}\+2c\+c^\{2\}\-s\_\{1\}s\_\{2\}\>0,\\ 1\+\\alpha\\left\(s\_\{1\}\+2c\\right\)\+\\alpha^\{2\}\\left\(c^\{2\}\-s\_\{1\}s\_\{2\}\\right\)\>0\\Bigg\\\}\.
From Claim\.[1](https://arxiv.org/html/2605.30600#Thmclaim1)applied with

a\\displaystyle a←s1,b←s2,c←c,ℓ←maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2,m←1\+2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖λminZ\+σ2\\displaystyle\\leftarrow s\_\{1\},b\\leftarrow s\_\{2\},c\\leftarrow c,\\ell\\leftarrow\\frac\{\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\},m\\leftarrow\\frac\{1\+2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\(45\)andM←2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖λminZ\+σ2M\\leftarrow\\frac\{2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}, we have the following bound for the objective in \([44](https://arxiv.org/html/2605.30600#A3.E44)\) as

Dα\(𝗀⊤SfX\+σξ⊤∥𝗀⊤SfX′\+σξ⊤\)≤12​\(α−1\)​log⁡\(\(1−maxi∈\[n\]​1\+2​‖Z⊤​Sf​ei−xi‖λminZ\+σ2−maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2\)α1−α⋅maxi∈\[n\]​1\+2​‖Z⊤​Sf​ei−xi‖λminZ\+σ2−α2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2\)\.\{D\_\{\\alpha\}\\left\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\sigma\\xi^\{\\top\}\\middle\\\|\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\+\\sigma\\xi^\{\\top\}\\right\)\}\\\\ \\leq\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\left\(1\-\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{1\+2\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\\right\)^\{\\alpha\}\}\{1\-\\alpha\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{1\+2\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-\\alpha^\{2\}\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\}\\right\)\.
Then, since

1\+2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖=CSf​X2,maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2=14​\(1−CSf​X2\)2,\\displaystyle 1\+2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|=\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\},\\ \\ \\ \\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}=\\frac\{1\}\{4\}\\left\(1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\right\)^\{2\},\(46\)and sinceλminZ\+σ2CSf​X2=γ\\frac\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}=\\gammawe obtain the upper bound

Dα\(𝗀⊤SfX\+σξ⊤∥𝗀⊤SfX′\+σξ⊤\)\\displaystyle\{D\_\{\\alpha\}\\left\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\sigma\\xi^\{\\top\}\\middle\\\|\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\+\\sigma\\xi^\{\\top\}\\right\)\}\(47\)≤12​\(α−1\)​log⁡\(\(1−maxi∈\[n\]​1\+2​‖Z⊤​Sf​ei−xi‖λminZ\+σ2−maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2\)α1−α⋅maxi∈\[n\]​1\+2​‖Z⊤​Sf​ei−xi‖λminZ\+σ2−α2⋅maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖2\(λminZ\+σ2\)2\)\\displaystyle\\qquad\\leq\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\left\(1\-\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{1\+2\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\\right\)^\{\\alpha\}\}\{1\-\\alpha\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{1\+2\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-\\alpha^\{2\}\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\tfrac\{\\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|^\{2\}\}\{\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\}\\right\)\(48\)=12​\(α−1\)​log⁡\(\(1−CSf​X2λminZ\+σ2−\(CSf​X22​\(λminZ\+σ2\)\)2​\(1−CSf​X2CSf​X2\)2\)α1−α​CSf​X2λminZ\+σ2−\(α​CSf​X22​\(λminZ\+σ2\)\)2​\(1−CSf​X2CSf​X2\)2\)\\displaystyle\\qquad=\\frac\{1\}\{2\(\\alpha\-1\)\}\\log\\left\(\\frac\{\\left\(1\-\\tfrac\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-\\left\(\\tfrac\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{2\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\}\\right\)^\{2\}\\left\(\\tfrac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\\right\)^\{\\alpha\}\}\{1\-\\tfrac\{\\alpha\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-\\left\(\\tfrac\{\\alpha\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{2\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\}\\right\)^\{2\}\\left\(\\tfrac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\}\\right\)\(49\)=12​\(α−1\)⋅\(α⋅log\(1−1γ−14​γ2\(1−CSf​X2CSf​X2\)2\)\\displaystyle\\qquad=\\frac\{1\}\{2\(\\alpha\-1\)\}\\cdot\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{1\}\{\\gamma\}\-\\frac\{1\}\{4\\gamma^\{2\}\}\\left\(\\frac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\\right\)\\right\.\(50\)−log\(1−αγ−α24​γ2\(1−CSf​X2CSf​X2\)2\)\)\\displaystyle\\qquad\\qquad\\qquad\\qquad\\qquad\\left\.\-\\log\\left\(1\-\\frac\{\\alpha\}\{\\gamma\}\-\\frac\{\\alpha^\{2\}\}\{4\\gamma^\{2\}\}\\left\(\\frac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\\right\)\\right\)\(51\)≔ϕ​\(α;γ,CSf​X2\)\.\\displaystyle\\qquad\\coloneq\\phi\\left\(\\alpha;\\gamma,\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\right\)\.\(52\)Finally, we place constraints onα,γ\\alpha,\\gammafor which the Rényi divergence above is well\-defined\. In particular, we require

1−1γ−14​γ2​\(1−CSf​X2CSf​X2\)2\>0and1−αγ−α24​γ2​\(1−CSf​X2CSf​X2\)2\>0\.1\-\\frac\{1\}\{\\gamma\}\-\\frac\{1\}\{4\\gamma^\{2\}\}\\left\(\\frac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\>0\\quad\\text\{and\}\\quad 1\-\\frac\{\\alpha\}\{\\gamma\}\-\\frac\{\\alpha^\{2\}\}\{4\\gamma^\{2\}\}\\left\(\\frac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\>0~\.It can be verified that when

γ\>γ¯,1<α<γ/γ¯\\displaystyle\\gamma\>\\overline\{\\gamma\},\\ \\ \\ 1<\\alpha<\\nicefrac\{\{\\gamma\}\}\{\{\\overline\{\\gamma\}\}\}\(53\)whereγ¯≔12​\(1\+1\+\(CSf​X−2−1\)2\)\\overline\{\\gamma\}\\coloneqq\\frac\{1\}\{2\}\\left\(1\+\\sqrt\{1\+\(\\textsc\{C\}\_\{S\_\{\\mathrm\{f\}\}X\}^\{\-2\}\-1\)^\{2\}\}\\right\), this holds\.

It remains to analyze the case where one row ofX′X^\{\\prime\}is zeroed out relative toXX\. In this case,Sf​X′=Sf​X\+Sf​ei​xi⊤=Sf​X\+ui​xi⊤S\_\{\\mathrm\{f\}\}X^\{\\prime\}=S\_\{\\mathrm\{f\}\}X\+S\_\{\\mathrm\{f\}\}e\_\{i\}x\_\{i\}^\{\\top\}=S\_\{\\mathrm\{f\}\}X\+u\_\{i\}x\_\{i\}^\{\\top\}\. The analysis is unchanged, except for a sign change inuiu\_\{i\}, which yields a sign change incc\. By symmetry, it leads to the same final result\. Since the bound is independent of the row removed fromXX, it applies whenever any pointi∈\[n\]i\\in\[n\]is removed\. This completes the proof of the closed\-form and tightest guarantee\.

##### Simplifying the Bound\.

Note that the previous proof established the upper bound

maxX′≃XDα\(MSf\(X\)∥MSf\(X′\)\)≤ϕ\(α;γ,CSf​X2\)\\underset\{X^\{\\prime\}\\simeq X\}\{\\max\}\\ \{D\_\{\\alpha\}\\left\(\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X\)\\middle\\\|\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X^\{\\prime\}\)\\right\)\}\\leq\\phi\\left\(\\alpha;\\gamma,\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\right\)\(54\)for allα∈\(1,γ/γ¯\)\\alpha\\in\(1,\\nicefrac\{\{\\gamma\}\}\{\{\\overline\{\\gamma\}\}\}\)provided thatγ\>γ¯\\gamma\>\\overline\{\\gamma\}\. First, sinceCSf​X2≥1\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\geq 1and\(1−CSf​X2CSf​X2\)2≤1\\left\(\\frac\{1\-\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\\right\)^\{2\}\\leq 1for allCSf​X2≥1\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\geq 1, the previous monotonicity arguments further yields the bound

max\(X,X′\):X≃X′\\displaystyle\\underset\{\(X,X^\{\\prime\}\):\\ X\\simeq X^\{\\prime\}\}\{\\max\}\\Dα\(MSf\(X\)∥MSf\(X′\)\)\\displaystyle\{D\_\{\\alpha\}\\left\(\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X\)\\middle\\\|\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X^\{\\prime\}\)\\right\)\}\(55\)≤12​\(α−1\)​\(α⋅log⁡\(1−1γ−14​γ2\)−log⁡\(1−αγ−α24​γ2\)\)\\displaystyle\\leq\\frac\{1\}\{2\(\\alpha\-1\)\}\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{1\}\{\\gamma\}\-\\frac\{1\}\{4\\gamma^\{2\}\}\\right\)\-\\log\\left\(1\-\\frac\{\\alpha\}\{\\gamma\}\-\\frac\{\\alpha^\{2\}\}\{4\\gamma^\{2\}\}\\right\)\\right\)\(56\)≔ϕ¯​\(α;γ\)\\displaystyle\\coloneq\\overline\{\\phi\}\(\\alpha;\\gamma\)\(57\)for allα∈\(1,4​γ/5\)\\alpha\\in\(1,\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\)provided thatγ\>5/4\\gamma\>\\nicefrac\{\{5\}\}\{\{4\}\}and where these constraints were obtained sinceγ¯≤5/4\\overline\{\\gamma\}\\leq\\nicefrac\{\{5\}\}\{\{4\}\}\. Now, note that by Calim\.[2](https://arxiv.org/html/2605.30600#Thmclaim2), over this range forα\\alphaandγ\\gammait further holds that

ϕ¯​\(α;γ\)≤12​\(α−1\)​\(α⋅log⁡\(1−54​γ\)−log⁡\(1−5​α4​γ\)\)\.\\displaystyle\\overline\{\\phi\}\(\\alpha;\\gamma\)\\leq\\frac\{1\}\{2\(\\alpha\-1\)\}\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{5\}\{4\\gamma\}\\right\)\-\\log\\left\(1\-\\frac\{5\\alpha\}\{4\\gamma\}\\right\)\\right\)\.\(58\)This upper bound matches the functionφ​\(α;4​γ/5\)\\varphi\(\\alpha;\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\)defined in Proposition[2](https://arxiv.org/html/2605.30600#Thmprop2)\. Thus, usingLevet al\.\[[2025](https://arxiv.org/html/2605.30600#bib.bib56), Corollary\. 1\], we know that this is upper bounded by

12​\(α−1\)​\(α⋅log⁡\(1−54​γ\)−log⁡\(1−5​α4​γ\)\)≤25​α32​γ2\\displaystyle\\frac\{1\}\{2\(\\alpha\-1\)\}\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{5\}\{4\\gamma\}\\right\)\-\\log\\left\(1\-\\frac\{5\\alpha\}\{4\\gamma\}\\right\)\\right\)\\leq\\frac\{25\\alpha\}\{32\\gamma^\{2\}\}\(59\)for allα∈\(1,8​γ/25\)\\alpha\\in\(1,\\nicefrac\{\{8\\gamma\}\}\{\{25\}\}\)and provided thatγ≥25/8\\gamma\\geq\\nicefrac\{\{25\}\}\{\{8\}\}\. This establishes the bound presented in Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)\. ∎

#### C\.1\.1Supplementary Algebraic Claims

Here, we state and prove algebraic claims that we have used in the proof of Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1)\.

###### Claim 1\.

Letα\>1\\alpha\>1\. Supposem,ℓ\>0m,\\ell\>0andMMis a general constant satisfying\|M\|≤m\\left\|M\\right\|\\leq m\. Consider the set𝒟1⊆ℝ3\\mathcal\{D\}\_\{1\}\\subseteq\\mathbb\{R\}^\{3\}

𝒟1\\displaystyle\\mathcal\{D\}\_\{1\}=\{\(a,b,c\)∈ℝ3:−ℓ≤c2−a​b≤0−m≤a\+2​c≤M1\+a\+2​c\+c2−a​b\>01\+α​\(a\+2​c\)\+α2​\(c2−a​b\)\>0\}\.\\displaystyle=\\left\\\{\(a,b,c\)\\in\\mathbb\{R\}^\{3\}:\\begin\{array\}\[\]\{l\}\-\\ell\\leq c^\{2\}\-ab\\leq 0\\\\ \-m\\leq a\+2c\\leq M\\\\ 1\+a\+2c\+c^\{2\}\-ab\>0\\\\ 1\+\\alpha\(a\+2c\)\+\\alpha^\{2\}\(c^\{2\}\-ab\)\>0\\end\{array\}\\right\\\}~\.Then,

sup\(a,b,c\)∈𝒟1log⁡\(\(1\+a\+2​c\+c2−a​b\)α1\+α​\(a\+2​c\)\+α2​\(c2−a​b\)\)≤log⁡\(\(1−m−ℓ\)α1−α​m−α2​ℓ\)\.\\sup\_\{\(a,b,c\)\\in\\mathcal\{D\}\_\{1\}\}\\log\\left\(\\frac\{\(1\+a\+2c\+c^\{2\}\-ab\)^\{\\alpha\}\}\{1\+\\alpha\(a\+2c\)\+\\alpha^\{2\}\(c^\{2\}\-ab\)\}\\right\)\\leq\\log\\left\(\\frac\{\(1\-m\-\\ell\)^\{\\alpha\}\}\{1\-\\alpha m\-\\alpha^\{2\}\\ell\}\\right\)\.

###### Proof\.

For convenience, we use the following notation\. Letx=c2−a​bx=c^\{2\}\-ab,y=2​c\+ay=2c\+a,A=1\+x\+yA=1\+x\+y, andB=1\+α​y\+α2​xB=1\+\\alpha y\+\\alpha^\{2\}x\. The objective can be expressed asα⋅log⁡\(A\)−log⁡\(B\)\\alpha\\cdot\\log\(A\)\-\\log\(B\)\. For anyyy,

∂∂x​\{α⋅log⁡\(A\)−log⁡\(B\)\}=αA−α2B=αA​B​\(B−α​A\)=α​\(α−1\)A​B​\(−1\+α​x\)\.\\frac\{\\partial\}\{\\partial x\}\\\{\\alpha\\cdot\\log\(A\)\-\\log\(B\)\\\}=\\frac\{\\alpha\}\{A\}\-\\frac\{\\alpha^\{2\}\}\{B\}=\\frac\{\\alpha\}\{AB\}\(B\-\\alpha A\)=\\frac\{\\alpha\(\\alpha\-1\)\}\{AB\}\(\-1\+\\alpha x\)~\.Sinceα\>1\\alpha\>1andx=c2−a​b≤0x=c^\{2\}\-ab\\leq 0according to the constraints, this partial derivative is non\-positive\. As a result, we can eliminatexxfrom the maximization to obtain the upper bound characterized by the problem

supy∈𝒟2log⁡\(\(1\+y−ℓ\)α1\+α​y−α2​ℓ\)\\sup\_\{y\\in\\mathcal\{D\}\_\{2\}\}\\log\\left\(\\frac\{\(1\+y\-\\ell\)^\{\\alpha\}\}\{1\+\\alpha y\-\\alpha^\{2\}\\ell\}\\right\)where𝒟2:=\{y∈ℝ:−m≤y≤M,1\+y−ℓ\>0,1\+α​y−α2​ℓ\>0\}\\mathcal\{D\}\_\{2\}:=\\\{y\\in\\mathbb\{R\}:\-m\\leq y\\leq M,1\+y\-\\ell\>0,1\+\\alpha y\-\\alpha^\{2\}\\ell\>0\\\}\.

We take the derivative of this objective inyyto obtain

∂∂y​log⁡\(\(1\+y−ℓ\)α1\+α​y−α2​ℓ\)=α​\(α−1\)​\(y−\(α\+1\)​ℓ\)\(1\+y−ℓ\)​\(1\+α​y−α2​ℓ\)\.\\frac\{\\partial\}\{\\partial y\}\\log\\left\(\\frac\{\(1\+y\-\\ell\)^\{\\alpha\}\}\{1\+\\alpha y\-\\alpha^\{2\}\\ell\}\\right\)=\\frac\{\\alpha\(\\alpha\-1\)\(y\-\(\\alpha\+1\)\\ell\)\}\{\(1\+y\-\\ell\)\(1\+\\alpha y\-\\alpha^\{2\}\\ell\)\}\.\(61\)We note that this derivative attains a single0aty⋆=\(α\+1\)​ℓy^\{\\star\}=\(\\alpha\+1\)\\ell, and moreover is negative for ally<y⋆y<y^\{\\star\}and positive for ally\>y⋆y\>y^\{\\star\}\. Therefore, since the objective is continuous and differentiable,y⋆y^\{\\star\}is the unique minimizer whenever it lies in the feasible domain; otherwise, the objective is monotone on the feasible interval\. In either case, the supremum over the bounded feasible interval is attained on its boundary\.

We note that the interval boundaries in our case are given by−m\-mandMM\. We demonstrate below that the maximum is attained at−m\-m\. To see this, let

Δ​\(y\)≔log⁡\(\(1−y−ℓ\)α1−α​y−α2​ℓ\)−log⁡\(\(1\+y−ℓ\)α1\+α​y−α2​ℓ\),y≥0\.\\Delta\(y\)\\coloneq\\log\\left\(\\frac\{\\left\(1\-y\-\\ell\\right\)^\{\\alpha\}\}\{1\-\\alpha y\-\\alpha^\{2\}\\ell\}\\right\)\-\\log\\left\(\\frac\{\\left\(1\+y\-\\ell\\right\)^\{\\alpha\}\}\{1\+\\alpha y\-\\alpha^\{2\}\\ell\}\\right\),\\ \\ \\ \\ \\ y\\geq 0\.\(62\)Then, using \([61](https://arxiv.org/html/2605.30600#A3.E61)\), we note that

∂Δ​\(y\)∂y\\displaystyle\\frac\{\\partial\\Delta\(y\)\}\{\\partial y\}=α​\(α−1\)​\(y\+\(α\+1\)​ℓ\)\(1−y−ℓ\)​\(1−α​y−α2​ℓ\)−α​\(α−1\)​\(y−\(α\+1\)​ℓ\)\(1\+y−ℓ\)​\(1\+α​y−α2​ℓ\)\\displaystyle=\\frac\{\\alpha\(\\alpha\-1\)\\left\(y\+\(\\alpha\+1\)\\ell\\right\)\}\{\(1\-y\-\\ell\)\(1\-\\alpha y\-\\alpha^\{2\}\\ell\)\}\-\\frac\{\\alpha\(\\alpha\-1\)\\left\(y\-\(\\alpha\+1\)\\ell\\right\)\}\{\(1\+y\-\\ell\)\(1\+\\alpha y\-\\alpha^\{2\}\\ell\)\}\(63\)=2​α​\(α−1\)​\(α\+1\)​\(y2\+ℓ​\(1−ℓ\)​\(1−α2​ℓ\)\)\(1−y−ℓ\)​\(1\+y−ℓ\)​\(1−α​y−α2​ℓ\)​\(1\+α​y−α2​ℓ\)\.\\displaystyle=\\frac\{2\\alpha\(\\alpha\-1\)\(\\alpha\+1\)\\left\(y^\{2\}\+\\ell\(1\-\\ell\)\(1\-\\alpha^\{2\}\\ell\)\\right\)\}\{\(1\-y\-\\ell\)\(1\+y\-\\ell\)\(1\-\\alpha y\-\\alpha^\{2\}\\ell\)\(1\+\\alpha y\-\\alpha^\{2\}\\ell\)\}\.\(64\)Now, since the denominator is positive for allyyin the interval \(since the positiveness of the different factors is required for the positiveness of the factors inside thelog\\log\), and sinceα\>1\\alpha\>1, we note that the sign of this derivative is determined by the sign ofy2\+ℓ​\(1−ℓ\)​\(1−α2​ℓ\)y^\{2\}\+\\ell\(1\-\\ell\)\(1\-\\alpha^\{2\}\\ell\)\. However, since the definiteness of thelog\\logaty=0y=0requires1−α2​ℓ≥01\-\\alpha^\{2\}\\ell\\geq 0and1−ℓ≥01\-\\ell\\geq 0and sinceℓ≥0\\ell\\geq 0we get thatℓ​\(1−ℓ\)​\(1−α2​ℓ\)≥0\\ell\(1\-\\ell\)\(1\-\\alpha^\{2\}\\ell\)\\geq 0, soy2\+ℓ​\(1−ℓ\)​\(1−α2​ℓ\)≥0y^\{2\}\+\\ell\(1\-\\ell\)\(1\-\\alpha^\{2\}\\ell\)\\geq 0, and this derivative is non\-negative\. SinceΔ​\(0\)=0\\Delta\(0\)=0, we get thatΔ≥0\\Delta\\geq 0, so the negative branch is larger\. Since\|M\|≤m\\left\|M\\right\|\\leq m, this together with the previous analysis of the derivative further says that the maximum of the overall objective is attained on−m\-m\. This completes the proof\. ∎

###### Claim 2\.

Letx\>5/4x\>\\nicefrac\{\{5\}\}\{\{4\}\}\. Then, for allα∈\(1,4​x/5\)\\alpha\\in\(1,\\nicefrac\{\{4x\}\}\{\{5\}\}\),

α⋅log⁡\(1−1x−14​x2\)−log⁡\(1−αx−α24​x2\)≤α⋅log⁡\(1−54​x\)−log⁡\(1−5​α4​x\)\.\\displaystyle\\alpha\\cdot\\log\\left\(1\-\\frac\{1\}\{x\}\-\\frac\{1\}\{4x^\{2\}\}\\right\)\-\\log\\left\(1\-\\frac\{\\alpha\}\{x\}\-\\frac\{\\alpha^\{2\}\}\{4x^\{2\}\}\\right\)\\leq\\alpha\\cdot\\log\\left\(1\-\\frac\{5\}\{4x\}\\right\)\-\\log\\left\(1\-\\frac\{5\\alpha\}\{4x\}\\right\)\.\(65\)

###### Proof\.

Defineg​\(z\):=log⁡\(1−z−z24\)−log⁡\(1−5​z4\)g\(z\):=\\log\\left\(1\-z\-\\frac\{z^\{2\}\}\{4\}\\right\)\-\\log\\left\(1\-\\frac\{5z\}\{4\}\\right\)forz∈\[0,4/5\)z\\in\[0,\\nicefrac\{\{4\}\}\{\{5\}\}\)\. We note that the desired inequality is equivalent to

α⋅g​\(1/x\)≤g​\(α/x\)\.\\displaystyle\\alpha\\cdot g\\left\(\\nicefrac\{\{1\}\}\{\{x\}\}\\right\)\\leq g\\left\(\\nicefrac\{\{\\alpha\}\}\{\{x\}\}\\right\)\.\(66\)Sinceg​\(0\)=0g\(0\)=0, the claim follows ifg​\(z\)g\(z\)is convex over the rangez∈\[0,4/5\)z\\in\[0,\\nicefrac\{\{4\}\}\{\{5\}\}\), since convexity then implies that for allα\>1\\alpha\>1

g​\(z\)=g​\(1α​\(α​z\)\+\(1−1α\)⋅0\)≥1α⋅g​\(α​z\)\+\(1−1α\)​g​\(0\)\\displaystyle g\(z\)=g\\left\(\\frac\{1\}\{\\alpha\}\(\\alpha z\)\+\\left\(1\-\\frac\{1\}\{\\alpha\}\\right\)\\cdot 0\\right\)\\geq\\frac\{1\}\{\\alpha\}\\cdot g\(\\alpha z\)\+\\left\(1\-\\frac\{1\}\{\\alpha\}\\right\)g\(0\)\(67\)soα⋅g​\(z\)≥g​\(α​z\)\\alpha\\cdot g\(z\)\\geq g\(\\alpha z\), and settingz=1/xz=\\nicefrac\{\{1\}\}\{\{x\}\}would prove the statement of the lemma\.

The remainder of the proof focuses on the convexity ofgg\. The second derivative ofggis

g′′​\(z\)\\displaystyle g^\{\\prime\\prime\}\(z\)=−2​\(4−4​z−z2\)\+\(4\+2​z\)2\(4−4​z−z2\)2\+25\(4−5​z\)2=16\+32​z−112​z2\+80​z3−25​z4\(4−5​z\)2​\(4−4​z−z2\)2\.\\displaystyle=\-\\frac\{2\\left\(4\-4z\-z^\{2\}\\right\)\+\(4\+2z\)^\{2\}\}\{\\left\(4\-4z\-z^\{2\}\\right\)^\{2\}\}\+\\frac\{25\}\{\\left\(4\-5z\\right\)^\{2\}\}=\\frac\{16\+32z\-112z^\{2\}\+80z^\{3\}\-25z^\{4\}\}\{\(4\-5z\)^\{2\}\(4\-4z\-z^\{2\}\)^\{2\}\}\.\(68\)The denominator is positive forz∈\[0,4/5\)z\\in\[0,\\nicefrac\{\{4\}\}\{\{5\}\}\)\. The numerator has two real roots overℝ\\mathbb\{R\}atz1≈0\.814z\_\{1\}\\approx 0\.814andz2≈−0\.247z\_\{2\}\\approx\-0\.247\. As a result, since onz=0z=0its positive, the quartic polynomial in the numerator is also non\-negative in the rangez∈\[0,4/5\)z\\in\[0,\\nicefrac\{\{4\}\}\{\{5\}\}\), which establishes thatg′′​\(z\)≥0g^\{\\prime\\prime\}\(z\)\\geq 0inz∈\[0,4/5\)z\\in\[0,\\nicefrac\{\{4\}\}\{\{5\}\}\)\. This asserts thatggis convex, thus proving the claim\. ∎

### C\.2Bound ForCSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\infty

We note that the previous bound requires that the quantitymaxi∈\[n\]​‖Z⊤​Sf​ei−xi‖\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|will be finite\. However, as we show now, a finite divergence is also possible in the case where we do not rely on such a bound\. This requires a particular analysis since takingCSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\inftyin the previous analysis yieldsγ→∞\\gamma\\to\\infty\. However, as we show now, a more fine\-grained analysis in this case leads to a finite divergence that depends onλminZ\+σ2\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\.

In particular, note that the next trivial bounds can be obtained ons1,s2s\_\{1\},s\_\{2\}, andcc:

s1≤‖x1‖2λminZ\+σ2≤1λminZ\+σ2,s2≤λmax​\(Z⊤​Z\)λmax​\(Z⊤​Z\)\+σ2​‖u1‖2≤‖u1‖2≤1,c≤s1​s2≤\(λminZ\+σ2\)−1/2\.s\_\{1\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\frac\{\\left\\\|x\_\{1\}\\right\\\|^\{2\}\}\{\\lambda^\{Z\}\_\{\\min\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\sigma^\{2\}\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\frac\{1\}\{\\lambda^\{Z\}\_\{\\min\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\sigma^\{2\}\},\\ \\ s\_\{2\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\frac\{\\lambda\_\{\\max\}\(Z^\{\\top\}Z\)\}\{\\lambda\_\{\\max\}\(Z^\{\\top\}Z\)\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\sigma^\{2\}\}\\left\\\|u\_\{1\}\\right\\\|^\{2\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\left\\\|u\_\{1\}\\right\\\|^\{2\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt1,\\ \\ c\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\sqrt\{s\_\{1\}s\_\{2\}\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\left\(\\lambda^\{Z\}\_\{\\min\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\sigma^\{2\}\\right\)^\{\-\\nicefrac\{\{1\}\}\{\{2\}\}\}\.\(69\)Then, using the notation

log⁡\(\(1\+s1\+2​c\+c2−s1​s2\)α1\+α​\(s1\+2​c\)\+α2​\(c2−s1​s2\)\)≔log⁡\(AαB\)=α⋅log⁡\(A\)−log⁡\(B\),\\log\\left\(\\frac\{\\big\(1\+s\_\{1\}\+2c\+c^\{2\}\-s\_\{1\}s\_\{2\}\\big\)^\{\\alpha\}\}\{1\+\\alpha\\left\(s\_\{1\}\+2c\\right\)\+\\alpha^\{2\}\\left\(c^\{2\}\-s\_\{1\}s\_\{2\}\\right\)\}\\right\)\\coloneq\\log\\left\(\\frac\{A^\{\\alpha\}\}\{B\}\\right\)=\\alpha\\cdot\\log\(A\)\-\\log\(B\),\(70\)we note that the divergence is a monotonically increasing function ofs2s\_\{2\}, since

∂∂s2​\{α⋅log⁡\(A\)−log⁡\(B\)\}=−α​s1A\+α2​s1B=α​s1A​B​\(α​A−B\)=α​\(α−1\)​s1A​B​\(1\+α​\(s1​s2−c2\)\)≥0\.\\frac\{\\partial\}\{\\partial s\_\{2\}\}\\left\\\{\\alpha\\cdot\\log\(A\)\-\\log\(B\)\\right\\\}=\-\\frac\{\\alpha s\_\{1\}\}\{A\}\+\\frac\{\\alpha^\{2\}s\_\{1\}\}\{B\}=\\frac\{\\alpha s\_\{1\}\}\{AB\}\\left\(\\alpha A\-B\\right\)=\\frac\{\\alpha\(\\alpha\-1\)s\_\{1\}\}\{AB\}\\left\(1\+\\alpha\(s\_\{1\}s\_\{2\}\-c^\{2\}\)\\right\)\\geq 0\.\(71\)Thus, we can substitutes2=1s\_\{2\}=1and to get the upper bound

log⁡\(\(1\+s1\+2​c\+c2−s1​s2\)α1\+α​\(s1\+2​c\)\+α2​\(c2−s1​s2\)\)≤2​α⋅log⁡\(1\+c\)−log⁡\(\(1\+α​c\)2\+α​\(1−α\)​s1\)\.\\log\\left\(\\frac\{\\big\(1\+s\_\{1\}\+2c\+c^\{2\}\-s\_\{1\}s\_\{2\}\\big\)^\{\\alpha\}\}\{1\+\\alpha\\left\(s\_\{1\}\+2c\\right\)\+\\alpha^\{2\}\\left\(c^\{2\}\-s\_\{1\}s\_\{2\}\\right\)\}\\right\)\\leq 2\\alpha\\cdot\\log\(1\+c\)\-\\log\\left\(\\left\(1\+\\alpha c\\right\)^\{2\}\+\\alpha\(1\-\\alpha\)s\_\{1\}\\right\)\.\(72\)Now, since1−α≤01\-\\alpha\\leq 0, this function is monotonically increasing ins1s\_\{1\}\. Thus, we substitutes1=\(λminZ\+σ2\)−1s\_\{1\}=\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{\-1\}and obtain the upper bound

log⁡\(\(1\+s1\+2​c\+c2−s1​s2\)α1\+α​\(s1\+2​c\)\+α2​\(c2−s1​s2\)\)≤2​α⋅log⁡\(1\+c\)−log⁡\(\(1\+α​c\)2\+α−α2λminZ\+σ2\)\.\\log\\left\(\\frac\{\\big\(1\+s\_\{1\}\+2c\+c^\{2\}\-s\_\{1\}s\_\{2\}\\big\)^\{\\alpha\}\}\{1\+\\alpha\\left\(s\_\{1\}\+2c\\right\)\+\\alpha^\{2\}\\left\(c^\{2\}\-s\_\{1\}s\_\{2\}\\right\)\}\\right\)\\leq 2\\alpha\\cdot\\log\(1\+c\)\-\\log\\left\(\\left\(1\+\\alpha c\\right\)^\{2\}\+\\tfrac\{\\alpha\-\\alpha^\{2\}\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\right\)\.\(73\)Now, this bound can be optimized overccunder the constraint in which\|c\|≤\(λminZ\+σ2\)−1/2\\left\|c\\right\|\\leq\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{\-\\nicefrac\{\{1\}\}\{\{2\}\}\}\. However, by similar arguments as used before, we can prove that the maximum is attained onc=−\(λminZ\+σ2\)−1/2c=\-\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{\-\\nicefrac\{\{1\}\}\{\{2\}\}\}, which yields the closed\-form bound

Dα\(𝗀⊤SfX\+σξ⊤∥𝗀⊤SfX′\+σξ⊤\)\\displaystyle\{D\_\{\\alpha\}\\left\(\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\sigma\\xi^\{\\top\}\\middle\\\|\\mathsf\{g\}^\{\\top\}S\_\{\\mathrm\{f\}\}X^\{\\prime\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\sigma\\xi^\{\\top\}\\right\)\}\(74\)≤12​\(α−1\)⋅\(2​α⋅log⁡\(1−1λminZ\+σ2\)−log⁡\(\(1−αλminZ\+σ2\)2\+α−α2λminZ\+σ2\)\)\\displaystyle\\qquad\\qquad\\qquad\\leq\\frac\{1\}\{2\(\\alpha\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt1\)\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt2\\alpha\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log\\left\(1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\tfrac\{1\}\{\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\\right\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\left\(1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\tfrac\{\\alpha\}\{\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\\hskip\-1\.49994pt\\right\)^\{2\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\tfrac\{\\alpha\-\\alpha^\{2\}\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\right\)\(75\)=12​\(α−1\)⋅\(2​α⋅log⁡\(1−1λminZ\+σ2\)−log⁡\(1−2​αλminZ\+σ2\+αλminZ\+σ2\)\)\\displaystyle\\qquad\\qquad\\qquad=\\frac\{1\}\{2\(\\alpha\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt1\)\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt2\\alpha\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log\\left\(1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\tfrac\{1\}\{\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\\right\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\log\\left\(1\-\\tfrac\{2\\alpha\}\{\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\+\\tfrac\{\\alpha\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\right\)\(76\)and which is defined for all1<α<λminZ\+σ22​λminZ\+σ2−11<\\alpha<\\tfrac\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\{2\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-1\}provided thatλminZ\+σ2\>1\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\>1\.

### C\.3Auxiliary Lemmas

We now provide three additional auxiliary lemmas, which will prove useful for making our overall privacy claim of Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1)\. Throughout, we use the notion ofℓ1\\ell\_\{1\}sensitivity, which, given a target functionf​\(X\)f\(X\), is defined by

max\(X,X′\):X′≃X​\|f​\(X\)−f​\(X′\)\|\.\\displaystyle\\underset\{\(X,X^\{\\prime\}\):X^\{\\prime\}\\simeq X\}\{\\max\}\\ \\left\|f\(X\)\-f\(X^\{\\prime\}\)\\right\|\.\(77\)
###### Lemma 3\.

Letu,v∈ℝdu,v\\in\\mathbb\{R\}^\{d\}\. Then,

‖u​v⊤‖op=‖u‖​‖v‖,‖u​v⊤\+v​u⊤‖op=‖u‖​‖v‖\+\|u⊤​v\|\.\\displaystyle\\left\\\|uv^\{\\top\}\\right\\\|\_\{\\mathrm\{op\}\}=\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|,\\qquad\\left\\\|uv^\{\\top\}\+vu^\{\\top\}\\right\\\|\_\{\\mathrm\{op\}\}=\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\+\\left\|u^\{\\top\}v\\right\|\.\(78\)

###### Proof\.

If eitheru=0→du=\\vec\{0\}\_\{d\}orv=0→dv=\\vec\{0\}\_\{d\}, both identities are immediate\. Hence, assumeu,v≠0→du,v\\neq\\vec\{0\}\_\{d\}, and define

u¯≔u‖u‖,v¯≔v‖v‖\.\\overline\{u\}\\coloneq\\frac\{u\}\{\\left\\\|u\\right\\\|\},\\qquad\\overline\{v\}\\coloneq\\frac\{v\}\{\\left\\\|v\\right\\\|\}\.
We first calculate the operator norm ofu​v⊤uv^\{\\top\}\. We have

u​v⊤=‖u‖​‖v‖​u¯​v¯⊤\.\\displaystyle uv^\{\\top\}=\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\\,\\overline\{u\}\\ \\overline\{v\}^\{\\top\}\.\(79\)Sinceu¯​v¯⊤\\overline\{u\}\\ \\overline\{v\}^\{\\top\}is a rank\-one matrix with its only nonzero singular value equal to one, it follows that

‖u​v⊤‖op=‖u‖​‖v‖\.\\displaystyle\\left\\\|uv^\{\\top\}\\right\\\|\_\{\\mathrm\{op\}\}=\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\.\(80\)
Next, consider

A≔u​v⊤\+v​u⊤=‖u‖​‖v‖​\(u¯​v¯⊤\+v¯​u¯⊤\)\.\\displaystyle A\\coloneq uv^\{\\top\}\+vu^\{\\top\}=\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\\left\(\\overline\{u\}\\ \\overline\{v\}^\{\\top\}\+\\overline\{v\}\\ \\overline\{u\}^\{\\top\}\\right\)\.\(81\)Then,

‖A‖op=max‖x‖=1⁡\|x⊤​A​x\|=2​‖u‖​‖v‖​max‖x‖=1⁡\|\(u¯⊤​x\)​\(v¯⊤​x\)\|\.\\displaystyle\\left\\\|A\\right\\\|\_\{\\mathrm\{op\}\}=\\max\_\{\\left\\\|x\\right\\\|=1\}\\left\|x^\{\\top\}Ax\\right\|=2\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\\max\_\{\\left\\\|x\\right\\\|=1\}\\left\|\(\\overline\{u\}^\{\\top\}x\)\(\\overline\{v\}^\{\\top\}x\)\\right\|\.\(82\)Thus, it remains to show that

max‖x‖=1⁡\|\(u¯⊤​x\)​\(v¯⊤​x\)\|=1\+\|u¯⊤​v¯\|2\.\\displaystyle\\max\_\{\\left\\\|x\\right\\\|=1\}\\left\|\(\\overline\{u\}^\{\\top\}x\)\(\\overline\{v\}^\{\\top\}x\)\\right\|=\\frac\{1\+\\left\|\\overline\{u\}^\{\\top\}\\overline\{v\}\\right\|\}\{2\}\.\(83\)
Letρ≔\|u¯⊤​v¯\|\\rho\\coloneq\\left\|\\overline\{u\}^\{\\top\}\\overline\{v\}\\right\|\. For any‖x‖=1\\left\\\|x\\right\\\|=1, by2​\|a​b\|≤a2\+b22\|ab\|\\leq a^\{2\}\+b^\{2\},

2​\|\(u¯⊤​x\)​\(v¯⊤​x\)\|\\displaystyle 2\\left\|\(\\overline\{u\}^\{\\top\}x\)\(\\overline\{v\}^\{\\top\}x\)\\right\|≤\(u¯⊤​x\)2\+\(v¯⊤​x\)2\\displaystyle\\leq\(\\overline\{u\}^\{\\top\}x\)^\{2\}\+\(\\overline\{v\}^\{\\top\}x\)^\{2\}\(84\)=x⊤​\(u¯​u¯⊤\+v¯​v¯⊤\)​x\\displaystyle=x^\{\\top\}\\left\(\\overline\{u\}\\ \\overline\{u\}^\{\\top\}\+\\overline\{v\}\\ \\overline\{v\}^\{\\top\}\\right\)x\(85\)≤λmax​\(u¯​u¯⊤\+v¯​v¯⊤\)\.\\displaystyle\\leq\\lambda\_\{\\max\}\\left\(\\overline\{u\}\\ \\overline\{u\}^\{\\top\}\+\\overline\{v\}\\ \\overline\{v\}^\{\\top\}\\right\)\.\(86\)The nonzero eigenvalues ofu¯​u¯⊤\+v¯​v¯⊤\\overline\{u\}\\ \\overline\{u\}^\{\\top\}\+\\overline\{v\}\\ \\overline\{v\}^\{\\top\}are1\+ρ1\+\\rhoand1−ρ1\-\\rho, which follows since the eigenvectors of this matrix areu¯\+v¯‖u¯\+v¯‖\\frac\{\\overline\{u\}\+\\overline\{v\}\}\{\\left\\\|\\overline\{u\}\+\\overline\{v\}\\right\\\|\}andu¯−v¯‖u¯−v¯‖\\frac\{\\overline\{u\}\-\\overline\{v\}\}\{\\left\\\|\\overline\{u\}\-\\overline\{v\}\\right\\\|\}\. Hence,

max‖x‖=1⁡\|\(u¯⊤​x\)​\(v¯⊤​x\)\|=1\+ρ2\.\\displaystyle\\max\_\{\\left\\\|x\\right\\\|=1\}\\left\|\(\\overline\{u\}^\{\\top\}x\)\(\\overline\{v\}^\{\\top\}x\)\\right\|=\\frac\{1\+\\rho\}\{2\}\.\(87\)Substituting this into the previous equation gives

‖u​v⊤\+v​u⊤‖op\\displaystyle\\left\\\|uv^\{\\top\}\+vu^\{\\top\}\\right\\\|\_\{\\mathrm\{op\}\}=2​‖u‖​‖v‖⋅1\+\|u¯⊤​v¯\|2\\displaystyle=2\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\\cdot\\frac\{1\+\\left\|\\overline\{u\}^\{\\top\}\\overline\{v\}\\right\|\}\{2\}\(88\)=‖u‖​‖v‖\+\|u⊤​v\|\.\\displaystyle=\\left\\\|u\\right\\\|\\left\\\|v\\right\\\|\+\\left\|u^\{\\top\}v\\right\|\.\(89\)This completes the proof\. ∎

###### Lemma 4\.

LetXXbe a dataset such that‖xi‖≤1\\\|x\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]\. SupposeSf∈ℝk×nS\_\{\\mathrm\{f\}\}\\in\\mathbb\{R\}^\{k\\times n\}is a matrix such that‖Sf​ei‖≤1\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]\. Then, theℓ1\\ell\_\{1\}sensitivity ofmaxi∈\[n\]​‖X⊤​Sf⊤​Sf​ei−xi‖\\ \\underset\{i\\in\[n\]\}\{\\max\}\\left\\\|X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|is bounded from above byΔ^≔maxi,j​\|ei⊤​\(Sf⊤​Sf−𝕀n\)​ej\|\\widehat\{\\Delta\}\\coloneq\\underset\{i,j\}\{\\max\}\\ \\left\|e^\{\\top\}\_\{i\}\\left\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{j\}\\right\|\.

###### Proof\.

Choose an arbitraryj∈\[n\]j\\in\[n\]and defineP:=Sf⊤​Sf−𝕀nP:=S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\. For eachi∈\[n\]i\\in\[n\], set

vi≔X⊤​P​ei,vi′≔\(X−ej​xj⊤\)⊤​P​ei=vi−xj​\(ej⊤​P​ei\)\.v\_\{i\}\\coloneq X^\{\\top\}Pe\_\{i\},\\qquad v\_\{i\}^\{\\prime\}\\coloneq\\left\(X\-e\_\{j\}x\_\{j\}^\{\\top\}\\right\)^\{\\top\}Pe\_\{i\}=v\_\{i\}\-x\_\{j\}\\left\(e\_\{j\}^\{\\top\}Pe\_\{i\}\\right\)\.\(90\)We note thatvi′v^\{\\prime\}\_\{i\}is the quantityfi​\(X\):=X⊤​\(Sf⊤​Sf−𝕀n\)​eif\_\{i\}\(X\):=X^\{\\top\}\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\)e\_\{i\}, evaluated on the neighboring datasetX′X^\{\\prime\}obtained by zeroing out thejt​hj^\{th\}row inXX\. By the triangle and Cauchy\-Schwarz inequalities, we get

‖vi′‖−‖vi‖≤\|‖vi′‖−‖vi‖\|≤‖vi′−vi‖=‖xj‖​\|ej⊤​P​ei\|\.\\left\\\|v\_\{i\}^\{\\prime\}\\right\\\|\-\\left\\\|v\_\{i\}\\right\\\|\\leq\\left\|\\left\\\|v\_\{i\}^\{\\prime\}\\right\\\|\-\\left\\\|v\_\{i\}\\right\\\|\\right\|\\leq\\left\\\|v\_\{i\}^\{\\prime\}\-v\_\{i\}\\right\\\|=\\left\\\|x\_\{j\}\\right\\\|\\left\|e\_\{j\}^\{\\top\}Pe\_\{i\}\\right\|~\.\(91\)Substituting the definition ofPPand the condition on‖xi‖\\\|x\_\{i\}\\\|,

\|‖vi′‖−‖vi‖\|≤maxi,j​\|ei⊤​\(Sf⊤​Sf−𝕀n\)​ej\|\\left\|\\left\\\|v\_\{i\}^\{\\prime\}\\right\\\|\-\\left\\\|v\_\{i\}\\right\\\|\\right\|\\leq\\underset\{i,j\}\{\\max\}\\ \\left\|e^\{\\top\}\_\{i\}\\left\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{j\}\\right\|\(92\)for everyi∈\[n\]i\\in\[n\]\. Now, letai≔‖vi′‖a\_\{i\}\\coloneq\\left\\\|v\_\{i\}^\{\\prime\}\\right\\\|andbi:=‖vi‖b\_\{i\}:=\\left\\\|v\_\{i\}\\right\\\|\. Then,

\|maxi∈\[n\]​ai−maxi∈\[n\]​bi\|≤maxi∈\[n\]​\|ai−bi\|≤maxi,j​\|ei⊤​\(Sf⊤​Sf−𝕀n\)​ej\|\.\\left\|\\underset\{i\\in\[n\]\}\{\\max\}\\ a\_\{i\}\-\\underset\{i\\in\[n\]\}\{\\max\}\\ b\_\{i\}\\right\|\\leq\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\|a\_\{i\}\-b\_\{i\}\\right\|\\leq\\underset\{i,j\}\{\\max\}\\ \\left\|e^\{\\top\}\_\{i\}\\left\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{j\}\\right\|\.\(93\)Sincejjis arbitrary, and since the bound is independent of the particular choice ofXX, it remains valid after taking the maximum over allj∈\[n\]j\\in\[n\]and all neighboring pairs\(X,X′\)\(X,X^\{\\prime\}\)\. Hence,

max\(X,X′\):X′≃X⁡\|maxi∈\[n\]⁡fi​\(X\)−maxi∈\[n\]⁡fi​\(X′\)\|≤maxi,j⁡\|ei⊤​\(Sf⊤​Sf−𝕀n\)​ej\|\.\\displaystyle\\max\_\{\(X,X^\{\\prime\}\):\\,X^\{\\prime\}\\simeq X\}\\left\|\\max\_\{i\\in\[n\]\}f\_\{i\}\(X\)\-\\max\_\{i\\in\[n\]\}f\_\{i\}\(X^\{\\prime\}\)\\right\|\\leq\\max\_\{i,j\}\\left\|e\_\{i\}^\{\\top\}\\\!\\left\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{j\}\\right\|\.\(94\)This is precisely the desiredℓ1\\ell\_\{1\}\-sensitivity\. ∎

###### Lemma 5\.

LetSf∈ℝk×nS\_\{\\mathrm\{f\}\}\\in\\mathbb\{R\}^\{k\\times n\}satisfying‖Sf​ei‖≤1\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]\. LetX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}satisfying‖xi‖≤1\\\|x\_\{i\}\\\|\\leq 1for alli∈\[n\]i\\in\[n\]and defineZ≔Sf​XZ\\coloneq S\_\{\\mathrm\{f\}\}X\. Setm≔maxi∈\[n\]​‖Z⊤​Sf​ei−xi‖m\\coloneq\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|andΛ≔2−mini∈\[n\]​‖Sf​ei‖2\\Lambda\\coloneq 2\-\\underset\{i\\in\[n\]\}\{\\min\}\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\. Then, theℓ1\\ell\_\{1\}sensitivity ofλminZ\\lambda^\{Z\}\_\{\\min\}is upper bounded byΛ\+2​m\\Lambda\+2m\.

###### Proof\.

DenotingX⊤​Sf⊤​Sf​ei=xi\+δiX^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}=x\_\{i\}\+\\delta\_\{i\}and recalling that‖Sf​ei‖2≤1\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\\leq 1for alli∈\[n\]i\\in\[n\]we note that

\(X−ei​xi⊤\)⊤​Sf⊤​Sf​\(X−ei​xi⊤\)\\displaystyle\\left\(X\-e\_\{i\}x^\{\\top\}\_\{i\}\\right\)^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\\left\(X\-e\_\{i\}x^\{\\top\}\_\{i\}\\right\)=X⊤​Sf⊤​Sf​X\+‖Sf​ei‖2​xi​xi⊤−X⊤​Sf⊤​Sf​ei​xi⊤−xi​ei⊤​Sf⊤​Sf​X\\displaystyle=X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}x\_\{i\}x^\{\\top\}\_\{i\}\-X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}x^\{\\top\}\_\{i\}\-x\_\{i\}e^\{\\top\}\_\{i\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\(95\)=X⊤​Sf⊤​Sf​X\+‖Sf​ei‖2​xi​xi⊤−\(xi\+δi\)​xi⊤−xi​\(xi\+δi\)⊤\\displaystyle=X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}x\_\{i\}x^\{\\top\}\_\{i\}\-\(x\_\{i\}\+\\delta\_\{i\}\)x^\{\\top\}\_\{i\}\-x\_\{i\}\(x\_\{i\}\+\\delta\_\{i\}\)^\{\\top\}\(96\)=X⊤​Sf⊤​Sf​X\+\(‖Sf​ei‖2−2\)​xi​xi⊤−δi​xi⊤−xi​δi⊤\.\\displaystyle=X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}X\+\\left\(\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\-2\\right\)x\_\{i\}x^\{\\top\}\_\{i\}\-\\delta\_\{i\}x^\{\\top\}\_\{i\}\-x\_\{i\}\\delta^\{\\top\}\_\{i\}\.\(97\)Now, by Weyl’s inequality\[Stewart,[1991](https://arxiv.org/html/2605.30600#bib.bib17), Theorem\. 1\]for matricesX⊤​XX^\{\\top\}XandY⊤​YY^\{\\top\}Yit holds

\|λminX−λminY\|≤‖X⊤​X−Y⊤​Y‖op\.\\left\|\\lambda^\{X\}\_\{\\min\}\-\\lambda^\{Y\}\_\{\\min\}\\right\|\\leq\\left\\\|X^\{\\top\}X\-Y^\{\\top\}Y\\right\\\|\_\{\\mathrm\{op\}\}\.\(98\)Thus,

\|λminZ−Sf​ei​xi⊤−λminZ\|\\displaystyle\\left\|\\lambda^\{Z\-S\_\{\\mathrm\{f\}\}e\_\{i\}x^\{\\top\}\_\{i\}\}\_\{\\min\}\-\\lambda^\{Z\}\_\{\\min\}\\right\|≤‖\(‖Sf​ei‖2−2\)​xi​xi⊤−δi​xi⊤−xi​δi⊤‖op\\displaystyle\\leq\\left\\\|\\left\(\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\-2\\right\)x\_\{i\}x^\{\\top\}\_\{i\}\-\\delta\_\{i\}x^\{\\top\}\_\{i\}\-x\_\{i\}\\delta^\{\\top\}\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\(99\)≤\|‖Sf​ei‖2−2\|​‖xi​xi⊤‖op\+‖δi​xi⊤\+xi​δi⊤‖op\\displaystyle\\leq\\left\|\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\-2\\right\|\\left\\\|x\_\{i\}x^\{\\top\}\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\+\\left\\\|\\delta\_\{i\}x^\{\\top\}\_\{i\}\+x\_\{i\}\\delta^\{\\top\}\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\(100\)=\|‖Sf​ei‖2−2\|​‖xi‖2\+‖δi‖​‖xi‖\+\|δi⊤​xi\|\\displaystyle=\\left\|\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\-2\\right\|\\left\\\|x\_\{i\}\\right\\\|^\{2\}\+\\left\\\|\\delta\_\{i\}\\right\\\|\\left\\\|x\_\{i\}\\right\\\|\+\\left\|\\delta^\{\\top\}\_\{i\}x\_\{i\}\\right\|\(101\)≤\|‖Sf​ei‖2−2\|​‖xi‖2\+2​‖δi‖​‖xi‖\\displaystyle\\leq\\left\|\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\-2\\right\|\\left\\\|x\_\{i\}\\right\\\|^\{2\}\+2\\left\\\|\\delta\_\{i\}\\right\\\|\\left\\\|x\_\{i\}\\right\\\|\(102\)≤maxi∈\[n\]​\|‖Sf​ei‖2−2\|\+2​m\\displaystyle\\leq\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\|\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\-2\\right\|\+2m\(103\)≤\(2−mini∈\[n\]​‖Sf​ei‖2\)\+2​m\\displaystyle\\leq\\left\(2\-\\underset\{i\\in\[n\]\}\{\\min\}\\ \\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}\\right\)\+2m\(104\)=Λ\+2​m\\displaystyle=\\Lambda\+2m\(105\)where we have used the triangle inequality, Lemma[3](https://arxiv.org/html/2605.30600#Thmlem3), the assumptions‖xi‖≤1\\left\\\|x\_\{i\}\\right\\\|\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt1and‖Sf​ei‖≤1\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt1for alli∈\[n\]i\\hskip\-1\.99997pt\\in\\hskip\-1\.99997pt\[n\], and the definitions ofmmandΛ\\Lambda\.

∎

###### Lemma 6\.

Given a datasetD\\pazocal\{D\}and a binary random variable𝖼∈\{0,1\}\\mathsf\{c\}\\in\\\{0,1\\\}, letM​\(D\)\\pazocal\{M\}\(\\pazocal\{D\}\)defined as

M​\(D\)≔\{Mgood​\(D\)if𝖼=0Mbad​\(D\)else\\displaystyle\\pazocal\{M\}\(\\pazocal\{D\}\)\\coloneq\\begin\{cases\}\\pazocal\{M\}\_\{\\mathrm\{good\}\}\(\\pazocal\{D\}\)\\ \\ &\\mathrm\{if\}\\ \\ \\mathsf\{c\}=0\\\\ \\pazocal\{M\}\_\{\\mathrm\{bad\}\}\(\\pazocal\{D\}\)\\ \\ &\\mathrm\{else\}\\end\{cases\}\(106\)whereℙ​\(𝖼=1\)=1−ℙ​\(𝖼=0\)=p\\mathbb\{P\}\(\\mathsf\{c\}=1\)=1\-\\mathbb\{P\}\(\\mathsf\{c\}=0\)=p\. If the mechanismMgood​\(D\)\|\{𝖼=0\}\\pazocal\{M\}\_\{\\mathrm\{good\}\}\(\\pazocal\{D\}\)\|\\left\\\{\\mathsf\{c\}=0\\right\\\}is\(ε,δ\)\(\\varepsilon,\\delta\)\-DPthenM​\(D\)\\pazocal\{M\}\(\\pazocal\{D\}\)is\(ε,δ\+p\)\(\\varepsilon,\\delta\+p\)\-DP\.

###### Proof\.

The proof holds following the definition of\(ε,δ\)\(\\varepsilon,\\delta\)\-DP\. In particular, for any neighborsD,D′\\pazocal\{D\},\\pazocal\{D\}^\{\\prime\}andS⊆range​\(M\)\\pazocal\{S\}\\subseteq\\mathrm\{range\}\(\\pazocal\{M\}\), sinceMgood\\pazocal\{M\}\_\{\\mathrm\{good\}\}is\(ε,δ\)\(\\varepsilon,\\delta\)\-DP, and by using the law of total probability, it holds

ℙ​\(M​\(D\)∈S\)\\displaystyle\\mathbb\{P\}\\left\(\\pazocal\{M\}\(\\pazocal\{D\}\)\\in\\pazocal\{S\}\\right\)=ℙ​\(𝖼=0\)​ℙ​\(M​\(D\)∈S\|𝖼=0\)\+ℙ​\(𝖼=1\)​ℙ​\(M​\(D\)∈S\|𝖼=1\)\\displaystyle=\\mathbb\{P\}\(\\mathsf\{c\}=0\)\\mathbb\{P\}\\left\(\\pazocal\{M\}\(\\pazocal\{D\}\)\\in\\pazocal\{S\}\|\\mathsf\{c\}=0\\right\)\+\\mathbb\{P\}\(\\mathsf\{c\}=1\)\\mathbb\{P\}\\left\(\\pazocal\{M\}\(\\pazocal\{D\}\)\\in\\pazocal\{S\}\|\\mathsf\{c\}=1\\right\)=\\displaystyle=\(107\)=\(1−p\)⋅ℙ​\(Mgood​\(D\)∈S\|𝖼=0\)\+p⋅ℙ​\(Mbad​\(D\)∈S\|𝖼=1\)\\displaystyle=\(1\-p\)\\cdot\\mathbb\{P\}\\left\(\\pazocal\{M\}\_\{\\mathrm\{good\}\}\(\\pazocal\{D\}\)\\in\\pazocal\{S\}\|\\mathsf\{c\}=0\\right\)\+p\\cdot\\mathbb\{P\}\\left\(\\pazocal\{M\}\_\{\\mathrm\{bad\}\}\(\\pazocal\{D\}\)\\in\\pazocal\{S\}\|\\mathsf\{c\}=1\\right\)\(108\)≤ℙ​\(Mgood​\(D\)∈S\|𝖼=0\)\+p\\displaystyle\\leq\\mathbb\{P\}\\left\(\\pazocal\{M\}\_\{\\mathrm\{good\}\}\(\\pazocal\{D\}\)\\in\\pazocal\{S\}\|\\mathsf\{c\}=0\\right\)\+p\(109\)≤eε⋅ℙ​\(Mgood​\(D′\)∈S\|𝖼=0\)\+δ\+p\\displaystyle\\leq e^\{\\varepsilon\}\\cdot\\mathbb\{P\}\\left\(\\pazocal\{M\}\_\{\\mathrm\{good\}\}\(\\pazocal\{D\}^\{\\prime\}\)\\in\\pazocal\{S\}\|\\mathsf\{c\}=0\\right\)\+\\delta\+p\(110\)which matches the\(ε,δ\+p\)\(\\varepsilon,\\delta\+p\)\-DP definition\. ∎

### C\.4Proof of Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1)

#### C\.4\.1Differential Privacy

We recall certain definitions here for the convenience of the reader\.

Λ^\\widehat\{\\Lambda\}2−mini∈\[n\]​‖Sf​ei‖22\-\\underset\{i\\in\[n\]\}\{\\min\}\\ \\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}Δ^\\widehat\{\\Delta\}maxi,j∈\[n\]​\|ei⊤​\(Sf⊤​Sf−𝕀n\)​ej\|\\underset\{i,j\\in\[n\]\}\{\\max\}\\ \\left\|e\_\{i\}^\{\\top\}\(S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}\-\\mathbb\{I\}\_\{n\}\)e\_\{j\}\\right\|m^\\widehat\{m\}maxi∈\[n\]​‖X⊤​Sf⊤​Sf​ei−xi‖\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|X^\{\\top\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\-x\_\{i\}\\right\\\|m~\\widetilde\{m\}max⁡\{1,m^\+ω​Δ^​\(τ−𝗓1\)\}\\max\\left\\\{1,\\widehat\{m\}\+\\omega\\widehat\{\\Delta\}\(\\tau\-\\mathsf\{z\}\_\{1\}\)\\right\\\}where𝗓1∼Lap​\(0,1\)\\mathsf\{z\}\_\{1\}\\sim\\mathrm\{Lap\}\(0,1\)λ~\\widetilde\{\\lambda\}max⁡\{0,λminSf​X−ω​\(Λ^\+2​m~\)​\(τ−𝗓2\)\}\\max\\left\\\{0,\\lambda\_\{\\min\}^\{S\_\{\\mathrm\{f\}\}X\}\-\\omega\(\\widehat\{\\Lambda\}\+2\\widetilde\{m\}\)\(\\tau\-\\mathsf\{z\}\_\{2\}\)\\right\\\}where𝗓2∼Lap​\(0,1\)\\mathsf\{z\}\_\{2\}\\sim\\mathrm\{Lap\}\(0,1\)Letℰ:=\{\(λ~≤λ^\)​⋂\(m~≥m^\)\}\\mathcal\{E\}\\hskip\-1\.99997pt:=\\hskip\-1\.99997pt\\left\\\{\\left\(\\widetilde\{\\lambda\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widehat\{\\lambda\}\\right\)\\hskip\-1\.49994pt\\bigcap\\hskip\-1\.49994pt\\left\(\\widetilde\{m\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pt\\widehat\{m\}\\right\)\\right\\\}\. First, note that for theτ\\taustated in the theorem and by using a union\-bound, it holds

ℙ​\(ℰc\)≤ℙ​\(λ^≤λ~\)\+ℙ​\(m~≤m^\)=ℙ​\(𝗓1≥τ\)\+ℙ​\(𝗓2≥τ\)≤2​δ3\.\\mathbb\{P\}\\left\(\\mathcal\{E\}^\{\\mathrm\{c\}\}\\right\)\\leq\\mathbb\{P\}\\left\(\\widehat\{\\lambda\}\\leq\\widetilde\{\\lambda\}\\right\)\+\\mathbb\{P\}\\left\(\\widetilde\{m\}\\leq\\widehat\{m\}\\right\)=\\mathbb\{P\}\\left\(\\mathsf\{z\}\_\{1\}\\geq\\tau\\right\)\+\\mathbb\{P\}\\left\(\\mathsf\{z\}\_\{2\}\\geq\\tau\\right\)\\leq\\frac\{2\\delta\}\{3\}\.\(112\)From now on, we prove the privacy guarantees of the mechanism conditioned on the eventℰ\\mathcal\{E\}, where by Lemma[6](https://arxiv.org/html/2605.30600#Thmlem6)this incurs a loss of2​δ/3\\nicefrac\{\{2\\delta\}\}\{\{3\}\}in the overallδ\\delta\.

##### Privact Guarantees for the Releasesm~\\widetilde\{m\}andλ~\.\\widetilde\{\\lambda\}\.

Since by Lemma[4](https://arxiv.org/html/2605.30600#Thmlem4)theℓ1\\ell\_\{1\}sensitivity ofm^\\widehat\{m\}is upper bounded byΔ^\\widehat\{\\Delta\},m~\\widetilde\{m\}is1ω\\frac\{1\}\{\\omega\}\-DP release ofm^\\widehat\{m\}by using classical properties of the Laplace mechanism\[Dworket al\.,[2014](https://arxiv.org/html/2605.30600#bib.bib142), Section\. 3\.3\]\. Moreover, since by Lemma[5](https://arxiv.org/html/2605.30600#Thmlem5)theℓ1\\ell\_\{1\}sensitivity ofλ^\\widehat\{\\lambda\}isΛ^\+2​m^\\widehat\{\\Lambda\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt2\\widehat\{m\}, which underℰ\\mathcal\{E\}is upper bounded byΛ^\+2​m~\\widehat\{\\Lambda\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt2\\widetilde\{m\}, using the same properties of the Laplace mechanism we get thatλ~\\widetilde\{\\lambda\}is1ω\\frac\{1\}\{\\omega\}\-DP release ofλ^\\widehat\{\\lambda\}\(seeVadhan \[[2017](https://arxiv.org/html/2605.30600#bib.bib42), Section\. 3\]why using a global upper bound on the local sensitivity provides DP\)\. Thus, these relevases contribute another2/ω\\nicefrac\{\{2\}\}\{\{\\omega\}\}to the overallε\\varepsilon\.

##### Guarantees of Sketching Steps\.

We note that theSfS\_\{\\mathrm\{f\}\}considered in the theorem satisfies the unit column\-norm condition required by Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1), and moreover \([7](https://arxiv.org/html/2605.30600#S3.E7)\) is satisfied underℰ\\mathcal\{E\}, since underℰ\\mathcal\{E\}it holdsη~2≥γ​\(1\+2​m^\)−λ^\\widetilde\{\\eta\}^\{2\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pt\\gamma\\left\(1\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt2\\widehat\{m\}\\right\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\widehat\{\\lambda\}\. Thus, the privacy guarantee of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)holds with respect to every row in the output𝖲G​Sf​X\+η~​ξ\\mathsf\{S\}\_\{\\mathrm\{G\}\}S\_\{\\mathrm\{f\}\}X\+\\widetilde\{\\eta\}\\xi\. Then, since the output comprised ofk1k\_\{1\}independnt rows of the Gaussian sketch applied overSf​XS\_\{\\mathrm\{f\}\}X, using composition for thek1k\_\{1\}different rows we get that the output satisfies\(α,k1⋅ϕ¯​\(α;γ\)\)\\left\(\\alpha,k\_\{1\}\\hskip\-1\.99997pt\\cdot\\hskip\-1\.99997pt\\overline\{\\phi\}\\left\(\\alpha;\\gamma\\right\)\\right\)\-RDP for1<α<4​γ/51\\hskip\-1\.99997pt<\\hskip\-1\.99997pt\\alpha\\hskip\-1\.99997pt<\\hskip\-1\.99997pt\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}, where we have used the upper boundϕ​\(α;γ,CSf​X2\)≤ϕ¯​\(α;γ\)\\phi\\left\(\\alpha;\\gamma,\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\right\)\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt\\overline\{\\phi\}\(\\alpha;\\gamma\)\. Using the conversion from RDP to\(ε,δ\)\(\\varepsilon,\\delta\)\-DP of Proposition[1](https://arxiv.org/html/2605.30600#Thmprop1)this contributes anotherδ/3\\nicefrac\{\{\\delta\}\}\{\{3\}\}to the overallδ\\deltaand

min1<α<4​γ/5​\{k1​ϕ¯​\(α;γ\)\+log⁡\(3/δ\)\+\(α−1\)​log⁡\(1−1/α\)−log⁡\(α\)α−1\}\\displaystyle\\underset\{1<\\alpha<\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\}\{\\min\}\\ \\left\\\{k\_\{1\}\\overline\{\\phi\}\(\\alpha;\\gamma\)\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\+\(\\alpha\-1\)\\log\(1\-\\nicefrac\{\{1\}\}\{\{\\alpha\}\}\)\-\\log\(\\alpha\)\}\{\\alpha\-1\}\\right\\\}\(113\)to the overallε\\varepsilon\.

##### Composition\.

Composing all the different components \(while by using Lemma[6](https://arxiv.org/html/2605.30600#Thmlem6)we addℙ​\(ℰc\)\\mathbb\{P\}\\left\(\\mathcal\{E\}^\{\\mathrm\{c\}\}\\right\)to the overallδ\\delta\) yields that the output satisfies\(ε^​\(ω,γ,k1,δ\),δ\)\(\\widehat\{\\varepsilon\}\(\\omega,\\gamma,k\_\{1\},\\delta\),\\delta\)\-DP, where

ε^​\(ω,γ,k1,δ\)\\displaystyle\\widehat\{\\varepsilon\}\(\\omega,\\gamma,k\_\{1\},\\delta\)≔2ω\+min1<α<4​γ/5​\{k1​ϕ¯​\(α;γ\)\+log⁡\(3/δ\)\+\(α−1\)​log⁡\(1−1/α\)−log⁡\(α\)α−1\},\\displaystyle\\coloneq\\frac\{2\}\{\\omega\}\+\\underset\{1<\\alpha<\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\}\{\\min\}\\ \\left\\\{k\_\{1\}\\overline\{\\phi\}\(\\alpha;\\gamma\)\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\+\(\\alpha\-1\)\\log\(1\-\\nicefrac\{\{1\}\}\{\{\\alpha\}\}\)\-\\log\(\\alpha\)\}\{\\alpha\-1\}\\right\\\},\(114\)ϕ¯​\(α;γ\)\\displaystyle\\overline\{\\phi\}\(\\alpha;\\gamma\)≔12​\(α−1\)⋅\(α⋅log⁡\(1−1γ−14​γ2\)−log⁡\(1−αγ−α24​γ2\)\)\\displaystyle\\coloneq\\frac\{1\}\{2\(\\alpha\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt1\)\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\alpha\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{1\}\{\\gamma\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{1\}\{4\\gamma^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\log\\left\(\\hskip\-1\.49994pt1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{\\alpha\}\{\\gamma\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{\\alpha^\{2\}\}\{4\\gamma^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\right\)\(115\)and which holds for allγ\>5/4\\gamma\>\\nicefrac\{\{5\}\}\{\{4\}\}\. Substituting the upper bound onϕ¯\\overline\{\\phi\}derived in Appendix[C\.1](https://arxiv.org/html/2605.30600#A3.SS1)yields

ε​\(ω,γ,k1,δ\)\\displaystyle\\varepsilon\(\\omega,\\gamma,k\_\{1\},\\delta\)≔2ω\+min1<α<8​γ/25​\{25​α​k132​γ2\+log⁡\(3/δ\)\+\(α−1\)​log⁡\(1−1/α\)−log⁡\(α\)α−1\}\\displaystyle\\coloneq\\frac\{2\}\{\\omega\}\+\\underset\{1<\\alpha<\\nicefrac\{\{8\\gamma\}\}\{\{25\}\}\}\{\\min\}\\ \\left\\\{\\frac\{25\\alpha k\_\{1\}\}\{32\\gamma^\{2\}\}\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\+\(\\alpha\-1\)\\log\(1\-\\nicefrac\{\{1\}\}\{\{\\alpha\}\}\)\-\\log\(\\alpha\)\}\{\\alpha\-1\}\\right\\\}\(116\)≤2ω\+min1<α<8​γ/25​\{25​α​k132​γ2\+log⁡\(3/δ\)α−1\}\.\\displaystyle\\leq\\frac\{2\}\{\\omega\}\+\\underset\{1<\\alpha<\\nicefrac\{\{8\\gamma\}\}\{\{25\}\}\}\{\\min\}\\ \\left\\\{\\frac\{25\\alpha k\_\{1\}\}\{32\\gamma^\{2\}\}\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{\\alpha\-1\}\\right\\\}\.\(117\)Substutitungα=min⁡\{8​γ25,1\+γ​32​log⁡\(3/δ\)25​k1\}\\alpha=\\min\\left\\\{\\frac\{8\\gamma\}\{25\},1\+\\gamma\\sqrt\{\\frac\{32\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{25k\_\{1\}\}\}\\right\\\}yields the upper bound of Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1)\.

#### C\.4\.2Runtime Complexity

We provide the proof by explicitly writing the complexity of the different steps inside[FastMix](https://arxiv.org/html/2605.30600#alg1)and their runtime complexity\.

##### ComputingZZ\.

Involves applying an SRHT of size\(k2,n\)\(k\_\{2\},n\)toX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}\. Following classical arguments \(see, for example,Ailon and Liberty \[[2009](https://arxiv.org/html/2605.30600#bib.bib76)\]\), the runtime complexity of this step isO​\(n​d​log⁡\(k2\)\)O\(nd\\log\(k\_\{2\}\)\)\. However, in our implementation, we first form the full matrixHn​𝖡n​XH\_\{n\}\\mathsf\{B\}\_\{n\}Xand then subsample its rows according to the non\-zero locations in the matrix𝖯k×n\\mathsf\{P\}\_\{k\\times n\}, since we use the values computed forHn​𝖡n​XH\_\{n\}\\mathsf\{B\}\_\{n\}Xin our subsequent steps\. Thus, the complexity of this step isO​\(n​d​log⁡\(n\)\)O\(nd\\log\(n\)\)\.

##### Computingλ~\\widetilde\{\\lambda\}\.

SinceZ∈ℝk2×dZ\\in\\mathbb\{R\}^\{k\_\{2\}\\times d\}, one can computeZ⊤​ZZ^\{\\top\}ZinO​\(k2​d2\)O\(k\_\{2\}d^\{2\}\)time and then compute its minimal eigenvalue inO​\(d3\)O\(d^\{3\}\)\(via standard eigensolvers\)\. Thus, the total complexity isO​\(k2​d2\+d3\)O\(k\_\{2\}d^\{2\}\+d^\{3\}\)\. In the regimek2≥dk\_\{2\}\\geq d\(\\big\(which is necessary for ensuring thatλminX≈λminZ\\lambda\_\{\\min\}^\{X\}\\approx\\lambda\_\{\\min\}^\{Z\}\)\\big\), this simplifies toO​\(k2​d2\)O\(k\_\{2\}d^\{2\}\)\. We note that in this regime, this can be achieved via a single SVD ofZZ, which costsO​\(k2​d2\)O\(k\_\{2\}d^\{2\}\)\.

##### Computingm^\\widehat\{m\}andΔ^\\widehat\{\\Delta\}\.

For computingm^\\widehat\{m\}, we first note that

Z⊤​Sf​ei−xi=X⊤​\(nk​𝖡n⊤​Hn⊤​𝖯k×n⊤​𝖯k×n​Hn​𝖡n−𝕀n\)​ei=\(Hn​𝖡n​X\)⊤​\(nk​𝖯k×n⊤​𝖯k×n−𝕀n\)​Hn​𝖡n​ei\.\\displaystyle Z^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{i\}\\hskip\-1\.99997pt\-\\hskip\-1\.99997ptx\_\{i\}\\hskip\-1\.99997pt=\\hskip\-1\.99997ptX^\{\\top\}\\hskip\-3\.00003pt\\left\(\\frac\{n\}\{k\}\\mathsf\{B\}^\{\\top\}\_\{n\}H^\{\\top\}\_\{n\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}\\mathsf\{B\}\_\{n\}\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\mathbb\{I\}\_\{n\}\\hskip\-1\.99997pt\\right\)\\hskip\-1\.99997pte\_\{i\}\\hskip\-1\.99997pt=\\hskip\-1\.99997pt\(H\_\{n\}\\mathsf\{B\}\_\{n\}X\)^\{\\top\}\\hskip\-3\.00003pt\\left\(\\frac\{n\}\{k\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\mathbb\{I\}\_\{n\}\\hskip\-1\.99997pt\\right\)\\hskip\-1\.99997ptH\_\{n\}\\mathsf\{B\}\_\{n\}e\_\{i\}\.\(118\)Then, since𝖯k×n\\mathsf\{P\}\_\{k\\times n\}selectskkrows, the matrixnk​𝖯k×n⊤​𝖯k×n−𝕀n\\frac\{n\}\{k\}\\mathsf\{P\}\_\{k\\times n\}^\{\\top\}\\mathsf\{P\}\_\{k\\times n\}\-\\mathbb\{I\}\_\{n\}is diagonal, with exactlykknonzero diagonal corrections relative to−𝕀n\-\\mathbb\{I\}\_\{n\}\. Hence, multiplying by this matrix amounts only to rescaling selected rows \(equivalently, columns after transposition\), so forming\(Hn​𝖡n​X\)⊤​\(nk​𝖯k×n⊤​𝖯k×n−𝕀n\)\(H\_\{n\}\\mathsf\{B\}\_\{n\}X\)^\{\\top\}\\\!\\left\(\\frac\{n\}\{k\}\\mathsf\{P\}\_\{k\\times n\}^\{\\top\}\\mathsf\{P\}\_\{k\\times n\}\-\\mathbb\{I\}\_\{n\}\\right\)costsO​\(n​d\)O\(nd\)\. Then, since we haveHn​𝖡n​XH\_\{n\}\\mathsf\{B\}\_\{n\}Xfrom the previous computations, the overall computations requires applying the Hadamard transformHn​BnH\_\{n\}B\_\{n\}on the matrix\(Hn​𝖡n​X\)⊤​\(nk​𝖯k×n⊤​𝖯k×n−𝕀n\)\(H\_\{n\}\\mathsf\{B\}\_\{n\}X\)^\{\\top\}\\left\(\\frac\{n\}\{k\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}\-\\mathbb\{I\}\_\{n\}\\right\), which costsO​\(n​d​log⁡\(n\)\)O\(nd\\log\(n\)\), and then computing the norms of the columns of this matrix which takes anotherO​\(n​d\)O\(nd\)\. Thus, the overall complexity of this step isO​\(n​d​log⁡\(n\)\)O\(nd\\log\(n\)\)\.

ComputingΔ^\\widehat\{\\Delta\}requires computing the coherenceΔ^=maxi≠j​\|ei⊤​Sf⊤​Sf​ej\|\.\\widehat\{\\Delta\}=\\underset\{i\\neq j\}\{\\max\}\\ \\left\|e^\{\\top\}\_\{i\}S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}e\_\{j\}\\right\|\.First, we note that ifSf=nk​𝖯k×n​Hn​𝖡nS\_\{\\mathrm\{f\}\}=\\sqrt\{\\frac\{n\}\{k\}\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}\\mathsf\{B\}\_\{n\}we haveSf⊤​Sf=nk​𝖡n​Hn⊤​𝖯k×n⊤​𝖯k×n​Hn​𝖡n\.S\_\{\\mathrm\{f\}\}^\{\\top\}S\_\{\\mathrm\{f\}\}=\\frac\{n\}\{k\}\\mathsf\{B\}\_\{n\}H\_\{n\}^\{\\top\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}\\mathsf\{B\}\_\{n\}\.Since𝖡n\\mathsf\{B\}\_\{n\}is diagonal with Rademacher entries, multiplying on the left and right by𝖡n\\mathsf\{B\}\_\{n\}only changes the signs of the entries, and therefore does not affect their absolute values\. Hence,

Δ^=nk​maxi≠j⁡\|ei⊤​Hn⊤​𝖯k×n⊤​𝖯k×n​Hn​ej\|\.\\widehat\{\\Delta\}=\\frac\{n\}\{k\}\\max\_\{i\\neq j\}\\left\|e^\{\\top\}\_\{i\}H\_\{n\}^\{\\top\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}e\_\{j\}\\right\|\.\(119\)
Now letz∈\{0,1\}nz\\in\\\{0,1\\\}^\{n\}denote the indicator vector of the sampled rows, namely

zr=1if row​r​is selected by​𝖯k×n,zr=0​otherwise\.z\_\{r\}=1\\quad\\text\{if row \}r\\text\{ is selected by \}\\mathsf\{P\}\_\{k\\times n\},\\qquad z\_\{r\}=0\\text\{ otherwise\.\}\(120\)Writinghih\_\{i\}for theii\-th column ofHnH\_\{n\}, we obtain

ei⊤​Hn⊤​𝖯k×n⊤​𝖯k×n​Hn​ej=∑r=1nzr​\(hi\)r​\(hj\)r\.e^\{\\top\}\_\{i\}H\_\{n\}^\{\\top\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}e\_\{j\}=\\sum\_\{r=1\}^\{n\}z\_\{r\}\(h\_\{i\}\)\_\{r\}\(h\_\{j\}\)\_\{r\}\.\(121\)Since𝖯k×n​Hn\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}is obtained by selectingkkrows of the Hadamard matrix, and since the entry\-wise product of two Hadamard columns is, up to the factor1/n1/\\sqrt\{n\}, another Hadamard column555this holds since, by the definition of the elements of the Hadamard matrix it holds that\(Hn\)i,j​\(Hn\)k,j=\(−1\)⟨i,j⟩2\+⟨k,j⟩2=\(−1\)⟨i⊕k,j⟩2\(H\_\{n\}\)\_\{i,j\}\(H\_\{n\}\)\_\{k,j\}=\(\-1\)^\{\\langle i,j\\rangle\_\{2\}\+\\langle k,j\\rangle\_\{2\}\}=\(\-1\)^\{\\langle i\\oplus k,j\\rangle\_\{2\}\}, corresponding to the element\(Hn\)i⊕k,j\(H\_\{n\}\)\_\{i\\oplus k,j\}and where⟨a,b⟩2\\langle a,b\\rangle\_\{2\}denotes the mod\-22inner product between the binary expansions ofaaandbb, for everyi≠ji\\neq jthere exists a non\-constant Hadamard columnhℓh\_\{\\ell\}such thathi⊙hj=1n​hℓh\_\{i\}\\odot h\_\{j\}=\\frac\{1\}\{\\sqrt\{n\}\}h\_\{\\ell\}wherea⊙ba\\odot bdenotes entry\-wise product between the vectorsaaandbb\. Therefore,

ei⊤​Hn⊤​𝖯k×n⊤​𝖯k×n​Hn​ej=1n​∑r=1nzr​\(hℓ\)r=1n​\(Hn​z\)ℓ\.e^\{\\top\}\_\{i\}H\_\{n\}^\{\\top\}\\mathsf\{P\}^\{\\top\}\_\{k\\times n\}\\mathsf\{P\}\_\{k\\times n\}H\_\{n\}e\_\{j\}=\\frac\{1\}\{\\sqrt\{n\}\}\\sum\_\{r=1\}^\{n\}z\_\{r\}\(h\_\{\\ell\}\)\_\{r\}=\\frac\{1\}\{\\sqrt\{n\}\}\(H\_\{n\}z\)\_\{\\ell\}\.\(122\)It follows that

Δ^=nk​maxℓ≠0⁡\|\(Hn​z\)ℓ\|,\\widehat\{\\Delta\}=\\frac\{\\sqrt\{n\}\}\{k\}\\max\_\{\\ell\\neq 0\}\\left\|\(H\_\{n\}z\)\_\{\\ell\}\\right\|,\(123\)where the maximum is taken over the values of the Hadamard transformed vectorHn​zH\_\{n\}z\.

Hence, computingΔ^\\widehat\{\\Delta\}reduces to: \(i\) forming the indicator vectorzzof the sampled rows, \(ii\) applying one Hadamard transform tozz, and \(iii\) taking the maximum absolute value over the resulting outputs\. The overall complexity is thereforeO​\(n​log⁡n\)O\(n\\log n\)time andO​\(n\)O\(n\)memory\.

##### Computing𝖲G​Z\\mathsf\{S\}\_\{\\mathrm\{G\}\}Z\.

Since the Gaussian sketch𝖲G\\mathsf\{S\}\_\{\\mathrm\{G\}\}is a dense matrix of sizek1×k2k\_\{1\}\\times k\_\{2\}, and since it is applied to a matrix of sizek2×dk\_\{2\}\\times d, this complexity isO​\(k1​k2​d\)O\(k\_\{1\}k\_\{2\}d\)\.

##### Overall Computation\.

Combining these terms leads to the next overall complexity

O​\(k1​k2​d\+n​d​log⁡\(n\)\+k2​d2\+d3\)\.O\\left\(k\_\{1\}k\_\{2\}d\+nd\\log\(n\)\+k\_\{2\}d^\{2\}\+d^\{3\}\\right\)\.\(124\)

## Appendix\. DSimplifyingϕ\\phiand Closed\-Form\(ε,δ\)\(\\varepsilon,\\delta\)\-DP Guarantees

Recall that we have established the upper bound

maxX′≃XDα\(MSf​\(X\)∥MSf\(X′\)\)≤12​\(α−1\)⋅\(α⋅log\(1−1γf−14​γf2\)−log\(1−αγf−α24​γf2\)\)≔ϕ¯\(α;γf\)\\displaystyle\\underset\{X^\{\\prime\}\\simeq X\}\{\\max\}\{D\_\{\\alpha\}\\left\(\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\(X\)\}\\middle\\\|\\pazocal\{M\}\_\{S\_\{\\mathrm\{f\}\}\}\(X^\{\\prime\}\)\\right\)\}\\hskip\-1\.99997pt\\leq\\hskip\-1\.99997pt\\frac\{1\}\{2\(\\alpha\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt1\)\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\left\(\\hskip\-1\.99997pt\\alpha\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log\\left\(1\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\frac\{1\}\{\\gamma\_\{\\mathrm\{f\}\}\}\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\frac\{1\}\{4\\gamma\_\{\\mathrm\{f\}\}^\{2\}\}\\right\)\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\log\\left\(1\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\frac\{\\alpha\}\{\\gamma\_\{\\mathrm\{f\}\}\}\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\frac\{\\alpha^\{2\}\}\{4\\gamma\_\{\\mathrm\{f\}\}^\{2\}\}\\hskip\-1\.99997pt\\right\)\\hskip\-1\.99997pt\\right\)\\hskip\-1\.99997pt\\coloneq\\hskip\-1\.99997pt\\overline\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\}\}\)\(125\)for all1<α<4​γf/51<\\alpha<\\nicefrac\{\{4\\gamma\_\{\\mathrm\{f\}\}\}\}\{\{5\}\}andγf\>5/4\\gamma\_\{\\mathrm\{f\}\}\>\\nicefrac\{\{5\}\}\{\{4\}\}and whereγf≔λminZ\+σ2CSf​X2\\gamma\_\{\\mathrm\{f\}\}\\coloneq\\frac\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}\.

Our goal now is to simplifyϕ¯​\(α;γf\)\\overline\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\}\}\)\. As we will show, it admits a simplified expression which allows us to assess its degradation relative to the guarantees of the Gaussian sketch \(given in Proposition[2](https://arxiv.org/html/2605.30600#Thmprop2)\)\. To that end, note that byLevet al\.\[[2025](https://arxiv.org/html/2605.30600#bib.bib56), Lemma\. 1\]

maxX′≃XDα\(M𝕀n\(X\)∥M𝕀n\(X′\)\)≤12​\(α−1\)⋅\(α⋅log\(1−1γ\)−log\(1−αγ\)\)\\displaystyle\\underset\{X^\{\\prime\}\\simeq X\}\{\\max\}\{D\_\{\\alpha\}\\left\(\\pazocal\{M\}\_\{\\mathbb\{I\}\_\{n\}\}\(X\)\\middle\\\|\\pazocal\{M\}\_\{\\mathbb\{I\}\_\{n\}\}\(X^\{\\prime\}\)\\right\)\}\\leq\\frac\{1\}\{2\(\\alpha\-1\)\}\\cdot\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{1\}\{\\gamma\}\\right\)\-\\log\\left\(1\-\\frac\{\\alpha\}\{\\gamma\}\\right\)\\right\)\(126\)forγ\>1\\gamma\>1and1<α<γ1<\\alpha<\\gammaand whereγ\\gammalower boundsλminX\+σ2\\lambda^\{X\}\_\{\\min\}\+\\sigma^\{2\}\. Additionally, byLevet al\.\[[2025](https://arxiv.org/html/2605.30600#bib.bib56), Corollary\. 1\], for all1<α<2​γ/51<\\alpha<\\nicefrac\{\{2\\gamma\}\}\{\{5\}\}it holds that

12​\(α−1\)⋅\(α⋅log⁡\(1−1γ\)−log⁡\(1−αγ\)\)≤α2​γ2\.\\frac\{1\}\{2\(\\alpha\-1\)\}\\cdot\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{1\}\{\\gamma\}\\right\)\-\\log\\left\(1\-\\frac\{\\alpha\}\{\\gamma\}\\right\)\\right\)\\leq\\frac\{\\alpha\}\{2\\gamma^\{2\}\}\.\(127\)
### D\.1Upper Bound onϕ¯\\overline\{\\phi\}

We note that in the range1<α<4​γf/51<\\alpha<\\nicefrac\{\{4\\gamma\_\{\\mathrm\{f\}\}\}\}\{\{5\}\}andγf\>5/4\\gamma\_\{\\mathrm\{f\}\}\>\\nicefrac\{\{5\}\}\{\{4\}\}it holds that

ϕ¯​\(α;γf\)≤12​\(α−1\)⋅\(α⋅log⁡\(1−54​γf\)−log⁡\(1−5​α4​γf\)\)\.\\overline\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\}\}\)\\leq\\frac\{1\}\{2\(\\alpha\-1\)\}\\cdot\\left\(\\alpha\\cdot\\log\\left\(1\-\\frac\{5\}\{4\\gamma\_\{\\mathrm\{f\}\}\}\\right\)\-\\log\\left\(1\-\\frac\{5\\alpha\}\{4\\gamma\_\{\\mathrm\{f\}\}\}\\right\)\\right\)\.\(128\)Then, using \([127](https://arxiv.org/html/2605.30600#A4.E127)\) we get that

ϕ¯​\(α;γf\)≤25​α32​γf2\\displaystyle\\overline\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\}\}\)\\leq\\frac\{25\\alpha\}\{32\\gamma^\{2\}\_\{\\mathrm\{f\}\}\}\(129\)for all1<α<8​γf/251<\\alpha<\\nicefrac\{\{8\\gamma\_\{\\mathrm\{f\}\}\}\}\{\{25\}\}\. Sinceγf\\gamma\_\{\\mathrm\{f\}\}lower boundsλminZ\+σ2CSf​X2\\frac\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\{\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\}, comparing \([126](https://arxiv.org/html/2605.30600#A4.E126)\) and \([128](https://arxiv.org/html/2605.30600#A4.E128)\) suggests that our new guarantees are degraded by the factor54⋅CSf​X2\\frac\{5\}\{4\}\\cdot\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}relative to Proposition[2](https://arxiv.org/html/2605.30600#Thmprop2), and moreover suffers from being dependent on the minimal eigenvalueλminZ\\lambda^\{Z\}\_\{\\min\}rather than onλminX\\lambda^\{X\}\_\{\\min\}\. Since for many choices ofSfS\_\{\\mathrm\{f\}\}\(\\big\(for example, wheneverSfS\_\{\\mathrm\{f\}\}is an SRHT withk​χ2=Ω​\(rank​\(X\)⋅log4⁡\(n\)\)k\\chi^\{2\}=\\Omega\\left\(\\mathrm\{rank\}\(X\)\\cdot\\log^\{4\}\(n\)\\right\)\)\\big\)it holds thatλminZ≥\(1−χ\)​λminX\\lambda^\{Z\}\_\{\\min\}\\geq\(1\-\\chi\)\\lambda^\{X\}\_\{\\min\}forχ∈\(0,1\]\\chi\\in\(0,1\], this adds at most multiplicative degradation of1−χ1\-\\chi\.

### D\.2Upper Bound onϕ¯\\overline\{\\phi\}WheneverCSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\infty

In the absence of any meaningful bound onCSf​X2\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}, \(namely, assuming thatCSf​X2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}X\}\\to\\infty\), by the analysis from Appendix[C\.2](https://arxiv.org/html/2605.30600#A3.SS2)the overall bound on the divergence becomesϕ^​\(α;γf,∞\)\\widehat\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\},\\infty\}\), where

ϕ^​\(α;γf,∞\)\\displaystyle\\widehat\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\},\\infty\}\)=12​\(α−1\)⋅\(2​α⋅log⁡\(1−1γf,∞\)−log⁡\(1−2​αγf,∞\+αγf,∞\)\)\\displaystyle=\\frac\{1\}\{2\(\\alpha\-1\)\}\\cdot\\left\(2\\alpha\\cdot\\log\\left\(1\-\\frac\{1\}\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\\right\)\-\\log\\left\(1\-\\frac\{2\\alpha\}\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\+\\frac\{\\alpha\}\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\\right\)\\right\)\(130\)=12​\(α−1\)⋅\(α⋅log⁡\(1−\(2γf,∞−1γf,∞\)\)−log⁡\(1−α​\(2γf,∞−1γf,∞\)\)\)\\displaystyle=\\frac\{1\}\{2\(\\alpha\-1\)\}\\cdot\\left\(\\alpha\\cdot\\log\\left\(1\-\\left\(\\frac\{2\}\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\-\\frac\{1\}\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\\right\)\\right\)\-\\log\\left\(1\-\\alpha\\left\(\\frac\{2\}\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\-\\frac\{1\}\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\\right\)\\right\)\\right\)\(131\)for all1<α<γf,∞21<\\alpha<\\frac\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\{2\}and provided thatγf,∞\>1\\gamma\_\{\\mathrm\{f\},\\infty\}\>1whereγf,∞\\gamma\_\{\\mathrm\{f\},\\infty\}lower boundsλminZ\+σ2\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\. Then, using \([127](https://arxiv.org/html/2605.30600#A4.E127)\) we get that this is upper bounded by

ϕ^​\(α;γf,∞\)≤α2⋅\(2γf,∞−1γf,∞\)2=2​αγf,∞⋅\(1−12​γf,∞\)2≤2​αγf,∞\\widehat\{\\phi\}\(\\alpha;\\gamma\_\{\\mathrm\{f\},\\infty\}\)\\leq\\frac\{\\alpha\}\{2\}\\cdot\\left\(\\frac\{2\}\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\-\\frac\{1\}\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\\right\)^\{2\}=\\frac\{2\\alpha\}\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\\cdot\\left\(1\-\\frac\{1\}\{2\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\\right\)^\{2\}\\leq\\frac\{2\\alpha\}\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\(132\)for all1<α<γf,∞51<\\alpha<\\frac\{\\sqrt\{\\gamma\_\{\\mathrm\{f\},\\infty\}\}\}\{5\}, suggesting that this bound decays like\(λminZ\+σ2\)−1\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{\-1\}instead of\(λminZ\+σ2\)−2\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{\-2\}as inLevet al\.\[[2025](https://arxiv.org/html/2605.30600#bib.bib56)\]\. Additionally, since it depends onλminZ\\lambda^\{Z\}\_\{\\min\}, it is similarly degraded by the multiplicative constant1−χ1\-\\chi\.

### D\.3Closed\-Form\(ε,δ\)\(\\varepsilon,\\delta\)\-DP Guarantees

Incorporating these inside the conversion from RDP to\(ε,δ\)\(\\varepsilon,\\delta\)\-DP\[Bunet al\.,[2018](https://arxiv.org/html/2605.30600#bib.bib143), Lemma\. 6\]and multiplying byk1k\_\{1\}, which amounts to applying a full Gaussian sketch \(rather than a single Gaussian vector as currently analyzed\), yields the closed\-form upper bounds on the privacy guarantees, which are given by

ε≤\{2​k1λminZ\+σ2\+8​k1​log⁡\(1/δ\)λminZ\+σ2if​log⁡\(1/δ\)<\(λminZ\+σ25−1\)2⋅2​k1λminZ\+σ22​k15​λminZ\+σ2\+5​log⁡\(1/δ\)λminZ\+σ2−5otherwise\\displaystyle\\varepsilon\\leq\\begin\{cases\}\\frac\{2k\_\{1\}\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\+\\sqrt\{\\frac\{8k\_\{1\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\\ &\\mathrm\{if\}\\ \\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)<\\left\(\\frac\{\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\{5\}\-1\\right\)^\{2\}\\cdot\\frac\{2k\_\{1\}\}\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\\\\ \\frac\{2k\_\{1\}\}\{5\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\}\+\\frac\{5\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\{\\sqrt\{\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\}\-5\}\\ &\\mathrm\{otherwise\}\\end\{cases\}\(133\)ifCSf2→∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}\}\\to\\inftyand by

ε≤\{25​k132​\(λminZ\+σ2\)2\+50​k1​log⁡\(1/δ\)4⋅\(λminZ\+σ2\)if​log⁡\(1/δ\)<\(8​\(λminZ\+σ2\)25−1\)2⋅25​k132​\(λminZ\+σ2\)2k14⋅\(λminZ\+σ2\)\+log⁡\(1/δ\)825⋅\(λminZ\+σ2\)−1otherwise\\displaystyle\\varepsilon\\leq\\begin\{cases\}\\frac\{25k\_\{1\}\}\{32\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\+\\frac\{\\sqrt\{50k\_\{1\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\{4\\cdot\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\}\\ &\\mathrm\{if\}\\ \\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)<\\left\(\\frac\{8\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\}\{25\}\-1\\right\)^\{2\}\\cdot\\frac\{25k\_\{1\}\}\{32\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)^\{2\}\}\\\\ \\frac\{k\_\{1\}\}\{4\\cdot\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\}\+\\frac\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\{\\frac\{8\}\{25\}\\cdot\\left\(\\lambda^\{Z\}\_\{\\min\}\+\\sigma^\{2\}\\right\)\-1\}\\ &\\mathrm\{otherwise\}\\end\{cases\}\(134\)ifCSf2<∞\\textsc\{C\}^\{2\}\_\{S\_\{\\mathrm\{f\}\}\}<\\infty\.

## Appendix\. EProof of Theorem[2](https://arxiv.org/html/2605.30600#Thmthm2)

Our proof follows similarly to the analysis of the iterative Hessian mixing, presented inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]\. We first provide auxiliary lemmas that will assist us in establishing the accuracy guarantees\.

### E\.1Auxiliary Definitions and Lemmas

We first provide the RDP guarantees of the Gaussian mechanism, as established byMironov \[[2017](https://arxiv.org/html/2605.30600#bib.bib52)\]\.

###### Lemma 7\(\[Mironov,[2017](https://arxiv.org/html/2605.30600#bib.bib52)\]\)\.

Letf:X→ℝmf:\\pazocal\{X\}\\to\\mathbb\{R\}^\{m\}be a map\. If there existsb\>0b\>0such for anyX,X′∈XX,X^\{\\prime\}\\in\\pazocal\{X\}satisfyingX≃X′X\\simeq X^\{\\prime\}it holds that‖f​\(X\)−f​\(X′\)‖≤b\\\|f\(X\)\-f\(X^\{\\prime\}\)\\\|\\leq b, then the Gaussian mechanismX↦f​\(X\)\+σ​ξX\\mapsto f\(X\)\+\\sigma\\xiforξ∼N​\(0→m,𝕀m\)\\xi\\sim\\pazocal\{N\}\(\\vec\{0\}\_\{m\},\\mathbb\{I\}\_\{m\}\)satisfies\(α,α​b22​σ2\)\(\\alpha,\\frac\{\\alpha b^\{2\}\}\{2\\sigma^\{2\}\}\)for allα\>1\\alpha\>1\.

Next, we prove auxiliary lemmas for the overall accuracy guarantee\. The first establishes guarantees for a sketch formed by concatenating Gaussian and fast sketches, as in[FastMix](https://arxiv.org/html/2605.30600#alg1)\. We later use it to generalizeLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]\. The second usesCohenet al\.\[[2016](https://arxiv.org/html/2605.30600#bib.bib12), Theorem\. 1\]for boundingΔ^\\widehat\{\\Delta\}\. The third shows that the SRHT sketch preservesλminX\\lambda^\{X\}\_\{\\min\}with high probability\.

###### Lemma 8\.

Let𝖲tot\\mathsf\{S\}\_\{\\mathrm\{tot\}\}defined via𝖲tot=𝖲G​𝖲^H\\mathsf\{S\}\_\{\\mathrm\{tot\}\}=\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}for𝖲G∼N​\(0,𝕀k1×\(k2\+d\)\)\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\sim\\pazocal\{N\}\(0,\\mathbb\{I\}\_\{k\_\{1\}\\times\(k\_\{2\}\+d\)\}\)and𝖲^H≔\(𝖲H0k2×d0d×n𝕀d\)\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}\\coloneq\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}&0\_\{k\_\{2\}\\times d\}\\\\ 0\_\{d\\times n\}&\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}where𝖲H∼SRHT​\(k2,n\)\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\sim\\mathrm\{SRHT\}\(k\_\{2\},n\)\. Then, there exists universal constants\(c0,c1,c2,c3\)\(c\_\{0\},c\_\{1\},c\_\{2\},c\_\{3\}\)such that for anyA∈ℝ\(n\+d\)×dA\\in\\mathbb\{R\}^\{\(n\+d\)\\times d\}of the formA=\[B⊤,b​𝕀d\]⊤A=\\Big\[B^\{\\top\},b\\mathbb\{I\}\_\{d\}\\Big\]^\{\\top\}for someB∈ℝn×dB\\in\\mathbb\{R\}^\{n\\times d\}andb∈ℝb\\in\\mathbb\{R\}, and for anyχ∈\(0,1\]\\chi\\in\(0,1\]andk1,k2∈ℕk\_\{1\},k\_\{2\}\\in\\mathbb\{N\}satisfyingk1​χ2≥c0​d,k2​χ2≥c0⋅rank​\(B\)⋅log4⁡\(n\)k\_\{1\}\\chi^\{2\}\\geq c\_\{0\}d,\\ k\_\{2\}\\chi^\{2\}\\geq c\_\{0\}\\cdot\\mathrm\{rank\}\\left\(B\\right\)\\cdot\\log^\{4\}\(n\)it holds

ℙ​\(𝒞​\(A,χ,k1\)\)≥1−c1⋅\(exp⁡\{−c2​k1​χ2\}\+exp⁡\{−c3​k2​χ2log4⁡\(n\)\}\)\\mathbb\{P\}\\left\(\\mathcal\{C\}\\left\(A,\\chi,k\_\{1\}\\right\)\\right\)\\geq 1\-c\_\{1\}\\cdot\\left\(\\exp\\left\\\{\-c\_\{2\}k\_\{1\}\\chi^\{2\}\\right\\\}\+\\exp\\left\\\{\-\\frac\{c\_\{3\}k\_\{2\}\\chi^\{2\}\}\{\\log^\{4\}\(n\)\}\\right\\\}\\right\)\(135\)where

𝒞​\(A,χ,k1\)≔\{supu∈ℝd,v∈ℝd\|u⊤​A⊤​\(1k1​𝖲tot⊤​𝖲tot−𝕀n\+d\)​A​v\|‖A​u‖⋅‖A​v‖≤χ2​⋂infv∈ℝd‖𝖲tot​A​v‖2k1⋅‖A​v‖2≥1−χ\}\.\\mathcal\{C\}\(A,\\chi,k\_\{1\}\)\\coloneq\\left\\\{\\sup\_\{u\\in\\mathbb\{R\}^\{d\},v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\|u^\{\\top\}A^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}\_\{\\mathrm\{tot\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{tot\}\}\-\\mathbb\{I\}\_\{n\+d\}\\right\)Av\\right\|\}\{\\\|Au\\\|\\cdot\\\|Av\\\|\}\\leq\\frac\{\\chi\}\{2\}\\ \\bigcap\\ \\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{tot\}\}Av\\right\\\|^\{2\}\}\{k\_\{1\}\\cdot\\\|Av\\\|^\{2\}\}\\geq 1\-\\chi\\right\\\}\.\(136\)

###### Proof\.

Let

EH​\(η\)≔\{supu∈ℝn,v∈ℝd\|u⊤​\(𝖲H⊤​𝖲H−𝕀n\)​B​v\|‖u‖⋅‖B​v‖≤η2,infv∈ℝd​‖𝖲H​B​v‖2‖B​v‖2≥1−η\}E\_\{\\mathrm\{H\}\}\(\\eta\)\\coloneq\\left\\\{\\sup\_\{u\\in\\mathbb\{R\}^\{n\},v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\|u^\{\\top\}\\left\(\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\}\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Bv\\right\|\}\{\\left\\\|u\\right\\\|\\cdot\\left\\\|Bv\\right\\\|\}\\leq\\frac\{\\eta\}\{2\},\\ \\ \\ \\ \\ \\underset\{v\\in\\mathbb\{R\}^\{d\}\}\{\\inf\}\\ \\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Bv\\right\\\|^\{2\}\}\{\\left\\\|Bv\\right\\\|^\{2\}\}\\geq 1\-\\eta\\right\\\}\(137\)forη∈\(0,1\]\\eta\\in\(0,1\]\. ByPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Lemma\. 1\]applied as inPilanci and Wainwright \[[2015](https://arxiv.org/html/2605.30600#bib.bib132), Corollary\. 2\],ℙ​\(EH​\(η\)\)≥1−c1,H⋅exp⁡\{−c2,H​k2​η2\}\\mathbb\{P\}\\left\(E\_\{\\mathrm\{H\}\}\(\\eta\)\\right\)\\geq 1\-c\_\{1,\\mathrm\{H\}\}\\cdot\\exp\\left\\\{\-c\_\{2,\\mathrm\{H\}\}k\_\{2\}\\eta^\{2\}\\right\\\}provided thatk2​η2≥c0,H⋅rank​\(B\)⋅log4⁡\(n\)k\_\{2\}\\eta^\{2\}\\geq c\_\{0,\\mathrm\{H\}\}\\cdot\\mathrm\{rank\}\\left\(B\\right\)\\cdot\\log^\{4\}\(n\)for some universal constantsc0,H,c1,H,c2,Hc\_\{0,\\mathrm\{H\}\},c\_\{1,\\mathrm\{H\}\},c\_\{2,\\mathrm\{H\}\}\.666By our convention, sketches in the setSRHT​\(k,n\)\\mathrm\{SRHT\}\(k,n\)contain additional normalization by factork\\sqrt\{k\}which was not presented inPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133)\]\. Thus, we avoid the factor1/k2\\nicefrac\{\{1\}\}\{\{k\_\{2\}\}\}while considering the events in \([137](https://arxiv.org/html/2605.30600#A5.E137)\)\.In particular, this holds for thek2k\_\{2\}from the lemma statement and withη←χ\\eta\\leftarrow\\chiwheneverc0≥c0,Hc\_\{0\}\\geq c\_\{0,\\mathrm\{H\}\}\.

Then, we note that conditioned onEH​\(η\)E\_\{\\mathrm\{H\}\}\(\\eta\), for everyv∈ℝdv\\in\\mathbb\{R\}^\{d\}it holds that

‖𝖲H​B​v‖2\+b2​‖v‖2≥\(1−χ\)​‖B​v‖2\+b2​‖v‖2≥\(1−χ\)​\(‖B​v‖2\+b2​‖v‖2\),\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Bv\\right\\\|^\{2\}\+b^\{2\}\\left\\\|v\\right\\\|^\{2\}\\geq\(1\-\\chi\)\\left\\\|Bv\\right\\\|^\{2\}\+b^\{2\}\\left\\\|v\\right\\\|^\{2\}\\geq\(1\-\\chi\)\\left\(\\left\\\|Bv\\right\\\|^\{2\}\+b^\{2\}\\left\\\|v\\right\\\|^\{2\}\\right\),\(138\)and which then implies

infv∈ℝd‖𝖲H​B​v‖2\+b2​‖v‖2‖B​v‖2\+b2​‖v‖2≥1−χ\.\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Bv\\right\\\|^\{2\}\+b^\{2\}\\left\\\|v\\right\\\|^\{2\}\}\{\\left\\\|Bv\\right\\\|^\{2\}\+b^\{2\}\\left\\\|v\\right\\\|^\{2\}\}\\geq 1\-\\chi\.\(139\)Additionally, we note that

supu1∈ℝn,u2∈ℝd,v∈ℝd\|\(u1⊤,u2⊤\)​\(\(𝖲H⊤​𝖲H0n×d0d×d𝕀d\)−𝕀n\+d\)​\(Bb​𝕀d\)​v\|‖u1‖2\+‖u2‖2⋅‖B​v‖2\+b2​‖v‖2≤supu1∈ℝn,v∈ℝd\|u1⊤​\(𝖲H⊤​𝖲H−𝕀n\)​B​v\|‖u1‖⋅‖B​v‖≤χ2\.\\sup\_\{u\_\{1\}\\in\\mathbb\{R\}^\{n\},u\_\{2\}\\in\\mathbb\{R\}^\{d\},v\\in\\mathbb\{R\}^\{d\}\}\\hskip\-1\.49994pt\\frac\{\\left\|\\left\(u^\{\\top\}\_\{1\},u^\{\\top\}\_\{2\}\\right\)\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}&0\_\{n\\times d\}\\\\ 0\_\{d\\times d\}&\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathbb\{I\}\_\{n\+d\}\\hskip\-1\.49994pt\\right\)\\begin\{pmatrix\}B\\\\ b\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}v\\right\|\}\{\\sqrt\{\\left\\\|u\_\{1\}\\right\\\|^\{2\}\+\\left\\\|u\_\{2\}\\right\\\|^\{2\}\}\\cdot\\sqrt\{\\left\\\|Bv\\right\\\|^\{2\}\+b^\{2\}\\left\\\|v\\right\\\|^\{2\}\}\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\sup\_\{u\_\{1\}\\in\\mathbb\{R\}^\{n\},v\\in\\mathbb\{R\}^\{d\}\}\\hskip\-1\.49994pt\\frac\{\\left\|u^\{\\top\}\_\{1\}\\hskip\-1\.49994pt\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathbb\{I\}\_\{n\}\\right\)\\hskip\-1\.49994ptBv\\right\|\}\{\\left\\\|u\_\{1\}\\right\\\|\\cdot\\left\\\|Bv\\right\\\|\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\frac\{\\chi\}\{2\}\.\(140\)
Now, we make the notation𝖢≔𝖲^H​A∈ℝk2×d\\mathsf\{C\}\\coloneq\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}A\\in\\mathbb\{R\}^\{k\_\{2\}\\times d\}and

EG​\(η;M\)≔\{supu∈ℝk2\+d,v∈ℝd\|u⊤​\(1k1​𝖲G⊤​𝖲G−𝕀k2\+d\)​M​v\|‖u‖⋅‖M​v‖≤η2,infv∈ℝd​‖𝖲G​M​v‖2k1⋅‖M​v‖2≥1−η\}\.E\_\{\\mathrm\{G\}\}\(\\eta;M\)\\coloneq\\left\\\{\\sup\_\{u\\in\\mathbb\{R\}^\{k\_\{2\}\+d\},v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\|u^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{G\}\}\\mathsf\{S\}\_\{\\mathrm\{G\}\}\-\\mathbb\{I\}\_\{k\_\{2\}\+d\}\\right\)Mv\\right\|\}\{\\left\\\|u\\right\\\|\\cdot\\left\\\|Mv\\right\\\|\}\\leq\\frac\{\\eta\}\{2\},\\ \\ \\underset\{v\\in\\mathbb\{R\}^\{d\}\}\{\\inf\}\\ \\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{G\}\}Mv\\right\\\|^\{2\}\}\{k\_\{1\}\\cdot\\left\\\|Mv\\right\\\|^\{2\}\}\\geq 1\-\\eta\\right\\\}\\hskip\-2\.5pt\.\(141\)UsingPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Lemma\. 1\], we know that for every realization𝖢=C\\mathsf\{C\}=Cit holds thatℙ​\(EG​\(η;C\)\)≥1−c1,G⋅exp⁡\{−c2,G​k1​η2\}\\mathbb\{P\}\\left\(E\_\{\\mathrm\{G\}\}\(\\eta;C\)\\right\)\\geq 1\-c\_\{1,\\mathrm\{G\}\}\\cdot\\exp\\left\\\{\-c\_\{2,\\mathrm\{G\}\}k\_\{1\}\\eta^\{2\}\\right\\\}provided thatk1​η2≥c0,G⋅rank​\(C\)k\_\{1\}\\eta^\{2\}\\geq c\_\{0,\\mathrm\{G\}\}\\cdot\\mathrm\{rank\}\(C\), and which sincerank​\(C\)≤d\\mathrm\{rank\}\(C\)\\leq dholds by the constrains we have onk1k\_\{1\}in the lemma statement forη←χ\\eta\\leftarrow\\chi\. Then, by using the law of total probability, we note that under the same constraint onk1k\_\{1\}it further holds that

ℙ​\(EG​\(η;𝖢\)\)≥1−c1,G⋅exp⁡\{−c2,G​k1​η2\}\\mathbb\{P\}\\left\(E\_\{\\mathrm\{G\}\}\(\\eta;\\mathsf\{C\}\)\\right\)\\geq 1\-c\_\{1,\\mathrm\{G\}\}\\cdot\\exp\\left\\\{\-c\_\{2,\\mathrm\{G\}\}k\_\{1\}\\eta^\{2\}\\right\\\}\(142\)
Thus, since

infv∈ℝd‖𝖲G​𝖢​v‖2k1⋅‖A​v‖2=infv∈ℝd\{‖𝖲G​𝖢​v‖2k1⋅‖𝖢​v‖2⋅‖𝖢​v‖2‖A​v‖2\},\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\mathsf\{C\}v\\right\\\|^\{2\}\}\{k\_\{1\}\\cdot\\left\\\|Av\\right\\\|^\{2\}\}=\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\left\\\{\\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\mathsf\{C\}v\\right\\\|^\{2\}\}\{k\_\{1\}\\cdot\\left\\\|\\mathsf\{C\}v\\right\\\|^\{2\}\}\\cdot\\frac\{\\left\\\|\\mathsf\{C\}v\\right\\\|^\{2\}\}\{\\left\\\|Av\\right\\\|^\{2\}\}\\right\\\},\(143\)we get that

infv∈ℝd‖𝖲G​𝖢​v‖2k1⋅‖A​v‖2≥\(1−χ\)⋅infv∈ℝd​‖𝖢​v‖2‖A​v‖2≥\(1−χ\)2\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\mathsf\{C\}v\\right\\\|^\{2\}\}\{k\_\{1\}\\cdot\\left\\\|Av\\right\\\|^\{2\}\}\\geq\(1\-\\chi\)\\cdot\\underset\{v\\in\\mathbb\{R\}^\{d\}\}\{\\inf\}\\ \\frac\{\\left\\\|\\mathsf\{C\}v\\right\\\|^\{2\}\}\{\\left\\\|Av\\right\\\|^\{2\}\}\\geq\(1\-\\chi\)^\{2\}\(144\)where the first inequality is by \([141](https://arxiv.org/html/2605.30600#A5.E141)\) and the second inequality is by \([139](https://arxiv.org/html/2605.30600#A5.E139)\), and both hold since they hold for every realization𝖢=C\\mathsf\{C\}=C, thus holds with probability11whenever we look at the event with the random matrix𝖢\\mathsf\{C\}\.

Note that by pickingM←𝖲^H​AM\\leftarrow\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Ain \([141](https://arxiv.org/html/2605.30600#A5.E141)\) and for thek1k\_\{1\}in the lemma statement we have

supw∈ℝk2\+d,v∈ℝd\|w⊤​\(1k1​𝖲G⊤​𝖲G−𝕀k2\+d\)​𝖲^H​A​v\|‖w‖⋅‖𝖲^H​A​v‖≤χ2\.\\sup\_\{w\\in\\mathbb\{R\}^\{k\_\{2\}\+d\},v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\|w^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{G\}\}\\mathsf\{S\}\_\{\\mathrm\{G\}\}\-\\mathbb\{I\}\_\{k\_\{2\}\+d\}\\right\)\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\\right\|\}\{\\left\\\|w\\right\\\|\\cdot\\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\\right\\\|\}\\leq\\frac\{\\chi\}\{2\}\.\(145\)Then, by writing

1k1​𝖲tot⊤​𝖲tot−𝕀n\+d=1k1​𝖲^H⊤​𝖲G⊤​𝖲G​𝖲^H−𝕀n\+d=𝖲^H⊤​\(1k1​𝖲G⊤​𝖲G−𝕀k2\+d\)​𝖲^H\+\(𝖲^H⊤​𝖲^H−𝕀n\+d\)\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}\_\{\\mathrm\{tot\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{tot\}\}\-\\mathbb\{I\}\_\{n\+d\}=\\frac\{1\}\{k\_\{1\}\}\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{G\}\}\\mathsf\{S\}\_\{\\mathrm\{G\}\}\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\+d\}=\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{G\}\}\\mathsf\{S\}\_\{\\mathrm\{G\}\}\-\\mathbb\{I\}\_\{k\_\{2\}\+d\}\\right\)\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}\+\\left\(\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}^\{\\top\}\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\+d\}\\right\)\(146\)and using the independence between𝖲H\\mathsf\{S\}\_\{\\mathrm\{H\}\}and𝖲G\\mathsf\{S\}\_\{\\mathrm\{G\}\}we note that for everyw,v∈ℝdw,v\\in\\mathbb\{R\}^\{d\}andu=A​wu=Awit holds that

\|u⊤​\(1k1​𝖲tot⊤​𝖲tot−𝕀n\+d\)​A​v\|‖u‖​‖A​v‖\\displaystyle\\frac\{\\left\|u^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}\_\{\\mathrm\{tot\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{tot\}\}\-\\mathbb\{I\}\_\{n\+d\}\\right\)Av\\right\|\}\{\\left\\\|u\\right\\\|\\left\\\|Av\\right\\\|\}=\|\(𝖲^H​u\)⊤​\(1k1​𝖲G⊤​𝖲G−𝕀k2\+d\)​𝖲^H​A​v\+u⊤​\(𝖲^H⊤​𝖲^H−𝕀n\+d\)​A​v\|‖u‖​‖A​v‖\\displaystyle=\\frac\{\\left\|\\left\(\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}u\\right\)^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{G\}\}\\mathsf\{S\}\_\{\\mathrm\{G\}\}\-\\mathbb\{I\}\_\{k\_\{2\}\+d\}\\right\)\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\+u^\{\\top\}\\left\(\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}^\{\\top\}\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\+d\}\\right\)Av\\right\|\}\{\\left\\\|u\\right\\\|\\left\\\|Av\\right\\\|\}\(147\)≤\|\(𝖲^H​u\)⊤​\(1k1​𝖲G⊤​𝖲G−𝕀k2\+d\)​𝖲^H​A​v\|\+\|u⊤​\(𝖲^H⊤​𝖲^H−𝕀n\+d\)​A​v\|‖u‖​‖A​v‖\\displaystyle\\leq\\frac\{\\left\|\\left\(\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}u\\right\)^\{\\top\}\\left\(\\frac\{1\}\{k\_\{1\}\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{G\}\}\\mathsf\{S\}\_\{\\mathrm\{G\}\}\-\\mathbb\{I\}\_\{k\_\{2\}\+d\}\\right\)\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\\right\|\+\\left\|u^\{\\top\}\\left\(\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}^\{\\top\}\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\+d\}\\right\)Av\\right\|\}\{\\left\\\|u\\right\\\|\\left\\\|Av\\right\\\|\}\(148\)≤χ2⋅‖𝖲^H​u‖⋅‖𝖲^H​A​v‖\+χ2⋅‖u‖⋅‖A​v‖‖u‖​‖A​v‖\\displaystyle\\leq\\frac\{\\frac\{\\chi\}\{2\}\\cdot\\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}u\\right\\\|\\cdot\\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\\right\\\|\+\\frac\{\\chi\}\{2\}\\cdot\\left\\\|u\\right\\\|\\cdot\\left\\\|Av\\right\\\|\}\{\\left\\\|u\\right\\\|\\left\\\|Av\\right\\\|\}\(149\)=χ2⋅‖𝖲^H​A​w‖⋅‖𝖲^H​A​v‖\+χ2⋅‖A​w‖⋅‖A​v‖‖A​w‖​‖A​v‖\\displaystyle=\\frac\{\\frac\{\\chi\}\{2\}\\cdot\\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Aw\\right\\\|\\cdot\\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\\right\\\|\+\\frac\{\\chi\}\{2\}\\cdot\\left\\\|Aw\\right\\\|\\cdot\\left\\\|Av\\right\\\|\}\{\\left\\\|Aw\\right\\\|\\left\\\|Av\\right\\\|\}\(150\)≤12​\(χ\+χ⋅\(1\+χ2\)\)\\displaystyle\\leq\\frac\{1\}\{2\}\\left\(\\chi\+\\chi\\cdot\\left\(1\+\\frac\{\\chi\}\{2\}\\right\)\\right\)\(151\)≤54​χ\\displaystyle\\leq\\frac\{5\}\{4\}\\chi\(152\)where we have used \([146](https://arxiv.org/html/2605.30600#A5.E146)\) and \([140](https://arxiv.org/html/2605.30600#A5.E140)\) and the inequalities

‖𝖲^H​A​v‖2≤\(1\+χ2\)​‖A​v‖2,‖𝖲^H​A​w‖2≤\(1\+χ2\)​‖A​w‖2\\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Av\\right\\\|^\{2\}\\leq\\left\(1\+\\frac\{\\chi\}\{2\}\\right\)\\left\\\|Av\\right\\\|^\{2\},\\ \\ \\ \\left\\\|\\widehat\{\\mathsf\{S\}\}\_\{\\mathrm\{H\}\}Aw\\right\\\|^\{2\}\\leq\\left\(1\+\\frac\{\\chi\}\{2\}\\right\)\\left\\\|Aw\\right\\\|^\{2\}\(153\)which due to \([140](https://arxiv.org/html/2605.30600#A5.E140)\)\. Since, as discussed, the conditional bound forEG​\(η;𝖢\)E\_\{\\mathrm\{G\}\}\(\\eta;\\mathsf\{C\}\)holds uniformly for every𝖢=C\\mathsf\{C\}=Cwheneverk1​η12≥c0,G​dk\_\{1\}\\eta^\{2\}\_\{1\}\\geq c\_\{0,\\mathrm\{G\}\}d, it holds that

ℙ​\(EG​\(η1;C\)∣𝖢=C\)≥1−c1,G⋅exp⁡\{−c2,G​k1​η12\},\\mathbb\{P\}\\left\(E\_\{\\mathrm\{G\}\}\(\\eta\_\{1\};C\)\\mid\\mathsf\{C\}=C\\right\)\\geq 1\-c\_\{1,\\mathrm\{G\}\}\\cdot\\exp\\left\\\{\-c\_\{2,\\mathrm\{G\}\}k\_\{1\}\\eta^\{2\}\_\{1\}\\right\\\},\(154\)and therefore,

ℙ​\(EH​\(η2\)∩EG​\(η1;𝖢\)\)\\displaystyle\\mathbb\{P\}\\left\(E\_\{\\mathrm\{H\}\}\(\\eta\_\{2\}\)\\cap E\_\{\\mathrm\{G\}\}\(\\eta\_\{1\};\\mathsf\{C\}\)\\right\)=𝔼​\[𝟙EH​\(η2\)⋅ℙ​\(EG​\(η1;𝖢\)∣𝖢\)\]\\displaystyle=\\mathbb\{E\}\\left\[\\mathbbm\{1\}\_\{E\_\{\\mathrm\{H\}\}\(\\eta\_\{2\}\)\}\\cdot\\mathbb\{P\}\\left\(E\_\{\\mathrm\{G\}\}\(\\eta\_\{1\};\\mathsf\{C\}\)\\mid\\mathsf\{C\}\\right\)\\right\]\(155\)≥\(1−c1,G⋅exp⁡\{−c2,G​k1​η12\}\)⋅ℙ​\(EH​\(η2\)\)\\displaystyle\\geq\\left\(1\-c\_\{1,\\mathrm\{G\}\}\\cdot\\exp\\left\\\{\-c\_\{2,\\mathrm\{G\}\}k\_\{1\}\\eta^\{2\}\_\{1\}\\right\\\}\\right\)\\cdot\\mathbb\{P\}\\left\(E\_\{\\mathrm\{H\}\}\(\\eta\_\{2\}\)\\right\)\(156\)≥1−c1,G⋅exp⁡\{−c2,G​k1​η12\}−c1,F⋅exp⁡\{−c2,F​k2​η22log4⁡\(n\)\}\\displaystyle\\geq 1\-c\_\{1,\\mathrm\{G\}\}\\cdot\\exp\\left\\\{\-c\_\{2,\\mathrm\{G\}\}k\_\{1\}\\eta^\{2\}\_\{1\}\\right\\\}\-c\_\{1,\\mathrm\{F\}\}\\cdot\\exp\\left\\\{\-\\frac\{c\_\{2,\\mathrm\{F\}\}k\_\{2\}\\eta^\{2\}\_\{2\}\}\{\\log^\{4\}\(n\)\}\\right\\\}\(157\)where𝟙EH​\(η\)\\mathbbm\{1\}\_\{E\_\{\\mathrm\{H\}\}\(\\eta\)\}denotes an indicator for the eventEH​\(η\)E\_\{\\mathrm\{H\}\}\(\\eta\)\. The proof is finalized by adjusting constants and noting that this bound holds withη1←χ,η2←χ\\eta\_\{1\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\chi,\\eta\_\{2\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\chifor thek1,k2k\_\{1\},k\_\{2\}used in the lemma statement\. ∎

The proof also implicitly justifies the choice of the parametersk1k\_\{1\}andk2k\_\{2\}in the lemma statement\. Since the overall approximation budgetχ\\chiis shared by the sketches𝖲H\\mathsf\{S\}\_\{\\mathrm\{H\}\}and𝖲G\\mathsf\{S\}\_\{\\mathrm\{G\}\}, a natural choice is to split it evenly, setting the approximation parameter of each sketch toχ/2\\nicefrac\{\{\\chi\}\}\{\{2\}\}, as in our calibration ofk1k\_\{1\}andk2k\_\{2\}\. Additionally, sincek2k\_\{2\}relates to an SRHT sketch, it should be inflated by a factorlog4⁡\(n\)\\log^\{4\}\(n\)to account for a similar factor in the constraints, and furthermore, since it is applied directly toBB, it can be scaled withrank​\(B\)\\mathrm\{rank\}\(B\)rather thandd\.

###### Lemma 9\.

Let𝖲H∼SRHT​\(k,n\)\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\sim\\mathrm\{SRHT\}\(k,n\)and letX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}withn≥dn\\geq d\. Then, there exist three universal constantsc0,c1,c2\>0c\_\{0\},c\_\{1\},c\_\{2\}\>0such that for anyχ,ϱ∈\(0,1\]\\chi,\\varrho\\in\(0,1\]satisfying

k​χ2≥c0⋅\(rank​\(X\)\+log⁡\(n​c1χ​ϱ\)​log⁡\(c2​n​\(rank​\(X\)\+1\)ϱ\)\),k\\chi^\{2\}\\geq c\_\{0\}\\cdot\\left\(\\mathrm\{rank\}\(X\)\+\\log\\left\(\\frac\{nc\_\{1\}\}\{\\chi\\varrho\}\\right\)\\log\\left\(\\frac\{c\_\{2\}n\\left\(\\mathrm\{rank\}\(X\)\+1\\right\)\}\{\\varrho\}\\right\)\\right\),\(158\)then w\.p\. at least1−ϱ1\-\\varrho,

maxi∈\[n\]⁡‖\(𝖲H​X\)⊤​𝖲H⊤​ei−xi‖≤χ​‖X‖op\.\\max\_\{i\\in\[n\]\}\\left\\\|\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\\right\)^\{\\top\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\}\}e\_\{i\}\-x\_\{i\}\\right\\\|\\leq\\chi\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\.\(159\)

###### Proof\.

LetX=U​Σ​V⊤X=U\\Sigma V^\{\\top\}be a compact SVD ofXX, whereU∈ℝn×rU\\in\\mathbb\{R\}^\{n\\times r\}has orthonormal columns withr≔rank​\(X\)r\\coloneq\\mathrm\{rank\}\(X\)and note that‖X‖op=‖V​Σ‖op\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}=\\left\\\|V\\Sigma\\right\\\|\_\{\\mathrm\{op\}\}\. Then, for everyi∈\[n\]i\\in\[n\]

‖\(𝖲H​X\)⊤​𝖲H⊤​ei−xi‖=‖X⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖=‖V​Σ​U⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖≤‖X‖op​‖U⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖\.\\left\\\|\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\\right\)^\{\\top\}\\hskip\-1\.49994pt\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\}\}e\_\{i\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994ptx\_\{i\}\\right\\\|\\hskip\-1\.49994pt=\\hskip\-1\.49994pt\\left\\\|X^\{\\top\}\\hskip\-1\.49994pt\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathbb\{I\}\_\{n\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pte\_\{i\}\\right\\\|\\hskip\-1\.49994pt=\\hskip\-1\.49994pt\\left\\\|V\\Sigma U^\{\\top\}\\hskip\-1\.49994pt\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathbb\{I\}\_\{n\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pte\_\{i\}\\right\\\|\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\\hskip\-1\.49994pt\\left\\\|U^\{\\top\}\\hskip\-1\.49994pt\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathbb\{I\}\_\{n\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pte\_\{i\}\\right\\\|\.\(160\)Thus, it is enough to bound‖U⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖\\left\\\|U^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|\. Fixi∈\[n\]i\\in\[n\]and letQi∈ℝn×tiQ\_\{i\}\\in\\mathbb\{R\}^\{n\\times t\_\{i\}\}have orthonormal columns spanningspan​\(U,ei\)\\mathrm\{span\}\(U,e\_\{i\}\)and whereti≤r\+1t\_\{i\}\\leq r\+1\. Since bothUUandeie\_\{i\}belong tospan​\(Qi\)\\mathrm\{span\}\(Q\_\{i\}\), there existRiR\_\{i\}andqiq\_\{i\}such that

U=Qi​Ri,ei=Qi​qi,‖Ri‖op≤1,‖qi‖2≤1\.U=Q\_\{i\}R\_\{i\},\\qquad e\_\{i\}=Q\_\{i\}q\_\{i\},\\qquad\\left\\\|R\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\\leq 1,\\qquad\\left\\\|q\_\{i\}\\right\\\|\_\{2\}\\leq 1\.\(161\)Therefore,

‖U⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖=‖Ri⊤​Qi⊤​\(𝖲H⊤​𝖲H−𝕀n\)​Qi​qi‖≤‖Qi⊤​\(𝖲H⊤​𝖲H−𝕀n\)​Qi‖op\.\\left\\\|U^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|=\\left\\\|R\_\{i\}^\{\\top\}Q\_\{i\}^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Q\_\{i\}q\_\{i\}\\right\\\|\\leq\\left\\\|Q\_\{i\}^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Q\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\.\(162\)Hence

‖X⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖≤‖X‖op​‖Qi⊤​\(𝖲H⊤​𝖲H−𝕀n\)​Qi‖op\.\\left\\\|X^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|\\leq\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\\,\\left\\\|Q\_\{i\}^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Q\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\.\(163\)
Then, for each fixedii, byCohenet al\.\[[2016](https://arxiv.org/html/2605.30600#bib.bib12), Theorem\. 9\]it holds that

𝔼​\[‖Qi⊤​\(𝖲H⊤​𝖲H−𝕀n\)​Qi‖op2\]≤χ2​δi,\\mathbb\{E\}\\left\[\\left\\\|Q\_\{i\}^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Q\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}^\{2\}\\right\]\\leq\\chi^\{2\}\\delta\_\{i\},\(164\)provided

k≥Cχ2​\(\(r\+1\)\+log⁡\(1χ​δi\)​log⁡\(r\+1δi\)\)k\\geq\\frac\{C\}\{\\chi^\{2\}\}\\left\(\(r\+1\)\+\\log\\left\(\\frac\{1\}\{\\chi\\delta\_\{i\}\}\\right\)\\log\\left\(\\frac\{r\+1\}\{\\delta\_\{i\}\}\\right\)\\right\)\(165\)for a universal constantC\>0C\>0\. By Markov’s inequality, this further implies that

ℙ​\(‖Qi⊤​\(𝖲H⊤​𝖲H−𝕀n\)​Qi‖op\>χ\)≤δi\.\\mathbb\{P\}\\left\(\\left\\\|Q\_\{i\}^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Q\_\{i\}\\right\\\|\_\{\\mathrm\{op\}\}\>\\chi\\right\)\\leq\\delta\_\{i\}\.\(166\)Consequently,

ℙ​\(‖X⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖\>χ​‖X‖op\)≤δi\.\\mathbb\{P\}\\left\(\\left\\\|X^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|\>\\chi\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\\right\)\\leq\\delta\_\{i\}\.\(167\)
Then, choosingδi≔ϱ/n\\delta\_\{i\}\\coloneq\\nicefrac\{\{\\varrho\}\}\{\{n\}\}and union bounding overi∈\[n\]i\\in\[n\]yields

ℙ​\(maxi∈\[n\]⁡‖X⊤​\(𝖲H⊤​𝖲H−𝕀n\)​ei‖\>χ​‖X‖op\)≤∑i=1nδi=ϱ\.\\mathbb\{P\}\\left\(\\max\_\{i\\in\[n\]\}\\left\\\|X^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\}\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|\>\\chi\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\\right\)\\leq\\sum\_\{i=1\}^\{n\}\\delta\_\{i\}=\\varrho\.\(168\)This completes the proof\. ∎

###### Lemma 10\.

LetX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}and let𝖲H∈ℝk×n\\mathsf\{S\}\_\{\\mathrm\{H\}\}\\in\\mathbb\{R\}^\{k\\times n\}beSRHT sketch \(Definition[3](https://arxiv.org/html/2605.30600#Thmdefn3)\)\. There exists universal constantsc0,c1,c2c\_\{0\},c\_\{1\},c\_\{2\}such that ifk≥c0⋅rank​\(X\)χ2⋅log4⁡\(n\)k\\geq\\frac\{c\_\{0\}\\cdot\\mathrm\{rank\}\(X\)\}\{\\chi^\{2\}\}\\cdot\\log^\{4\}\(n\)then w\.p\. at least1−c1⋅exp⁡\{−c2​k​χ2log4⁡\(n\)\}1\-c\_\{1\}\\cdot\\exp\\left\\\{\-\\frac\{c\_\{2\}k\\chi^\{2\}\}\{\\log^\{4\}\(n\)\}\\right\\\}

\(1−χ\)⋅λminX≤λmin𝖲H​X≤\(1\+χ\)⋅λminX,\(1−χ\)⋅λmaxX≤λmax𝖲H​X≤\(1\+χ\)⋅λmaxX\.\\left\(1\-\\chi\\right\)\\cdot\\lambda^\{X\}\_\{\\min\}\\leq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\}\_\{\\min\}\\leq\\left\(1\+\\chi\\right\)\\cdot\\lambda^\{X\}\_\{\\min\},\\ \\ \\ \\left\(1\-\\chi\\right\)\\cdot\\lambda^\{X\}\_\{\\max\}\\leq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\}\_\{\\max\}\\leq\\left\(1\+\\chi\\right\)\\cdot\\lambda^\{X\}\_\{\\max\}\.\(169\)

###### Proof\.

The proof follows byPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Lemma\. 1\], which implies that fork≥c0⋅rank​\(X\)χ2⋅log4⁡\(n\)k\\geq\\frac\{c\_\{0\}\\cdot\\mathrm\{rank\}\(X\)\}\{\\chi^\{2\}\}\\cdot\\log^\{4\}\(n\)w\.p\. at least1−c1⋅exp⁡\{−c2​k​χ2log4⁡\(n\)\}1\-c\_\{1\}\\cdot\\exp\\left\\\{\-\\frac\{c\_\{2\}k\\chi^\{2\}\}\{\\log^\{4\}\(n\)\}\\right\\\}

supv∈ℝd:‖v‖=1​\|X⊤​v⊤​\(𝖲H⊤​𝖲H−𝕀n\)​X​v\|≤χ​‖X​v‖2\.\\underset\{v\\in\\mathbb\{R\}^\{d\}:\\ \\left\\\|v\\right\\\|=1\}\{\\sup\}\\ \\left\|X^\{\\top\}v^\{\\top\}\\left\(\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\}\}\\mathsf\{S\}\_\{\\mathrm\{H\}\}\-\\mathbb\{I\}\_\{n\}\\right\)Xv\\right\|\\leq\\chi\\left\\\|Xv\\right\\\|^\{2\}\.\(170\)We note that \([170](https://arxiv.org/html/2605.30600#A5.E170)\) implies

\(1−χ\)​‖X​v‖2≤‖𝖲H​X​v‖2≤\(1\+χ\)​‖X​v‖2,∀v∈ℝd\.\\left\(1\-\\chi\\right\)\\left\\\|Xv\\right\\\|^\{2\}\\leq\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Xv\\right\\\|^\{2\}\\leq\\left\(1\+\\chi\\right\)\\left\\\|Xv\\right\\\|^\{2\},\\ \\ \\ \\forall v\\in\\mathbb\{R\}^\{d\}\.\(171\)We now restrict attention to vectorsvvthat have unit norm\. Then, since by definition‖X​v‖2≥λminX\\left\\\|Xv\\right\\\|^\{2\}\\geq\\lambda^\{X\}\_\{\\min\},

\(1−χ\)​λminX≤‖𝖲H​X​v‖2≤\(1\+χ\)​‖X​v‖2\.\\left\(1\-\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\\leq\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Xv\\right\\\|^\{2\}\\leq\\left\(1\+\\chi\\right\)\\left\\\|Xv\\right\\\|^\{2\}\.\(172\)Settingv←v⋆v\\leftarrow v^\{\\star\}, wherev⋆v^\{\\star\}corresponds to the minimal singular value ofXX, yields

\(1−χ\)​λminX≤‖𝖲H​X​v⋆‖2≤\(1\+χ\)​λminX\.\\left\(1\-\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\\leq\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Xv^\{\\star\}\\right\\\|^\{2\}\\leq\\left\(1\+\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\.\(173\)This implies

\(1−χ\)​λminX≤λmin𝖲H​X≤\(1\+χ\)​λminX\\left\(1\-\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\\leq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\}\_\{\\min\}\\leq\\left\(1\+\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\(174\)since‖𝖲H​X​v⋆‖2≥λmin𝖲H​X\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\}\}Xv^\{\\star\}\\right\\\|^\{2\}\\geq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\}\_\{\\min\}and since we could instead pickvvto be the singular vector corresponding to the minimum singular value of𝖲H​X\\mathsf\{S\}\_\{\\mathrm\{H\}\}X\. The corresponding derivation for the maximum eigenvalue proceeds in the same manner\. The proof then follows after adjustingχ\\chiand the relevant constants\. ∎

Finally, the next lemma provides high probability bounds for the norm of a Gaussian vector

###### Lemma 11\.

Letξ∼N​\(0→d,𝕀d\)\\xi\\sim\\pazocal\{N\}\(\\vec\{0\}\_\{d\},\\mathbb\{I\}\_\{d\}\)and letϱ∈\(0,1\]\\varrho\\in\(0,1\]\. Then,

ℙ​\(‖ξ‖2≤\(2​d\+4​log⁡\(1/ϱ\)\)2\)≥1−ϱ2\.\\mathbb\{P\}\\left\(\\left\\\|\\xi\\right\\\|^\{2\}\\leq\\left\(\\sqrt\{2d\}\+\\sqrt\{4\\log\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\}\\right\)^\{2\}\\right\)\\geq 1\-\\varrho^\{2\}\.\(175\)

The previous lemmas and definitions allow us to derive high\-probability accuracy guarantees for[FastIHM](https://arxiv.org/html/2605.30600#alg2)\. In particular, Lemma[8](https://arxiv.org/html/2605.30600#Thmlem8)parallels a similar lemma proved inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]and enables a recursive characterization\. In addition, Lemma[9](https://arxiv.org/html/2605.30600#Thmlem9)and Lemma[10](https://arxiv.org/html/2605.30600#Thmlem10)control the degradation induced by applying the Gaussian sketch to the compressed datasets\{𝖲H,t​X\}t\\left\\\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\right\\\}\_\{t\}, as done in[FastMix](https://arxiv.org/html/2605.30600#alg1)\. Using the notation of Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1), they controlλmin𝖲H,t​X\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}\_\{\\min\}andC𝖲H,t​X2\\textsc\{C\}^\{2\}\_\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}for allt=0,1,…,T−1t=0,1,\\ldots,T\-1\.

### E\.2Proof of Theorem[2](https://arxiv.org/html/2605.30600#Thmthm2)

We first prove the DP guarantee for the algorithm, and then the utility guarantee\. Throughout, given\(k1,T,C\)\(k\_\{1\},T,\\textsc\{C\}\)and for a target failure probabilityϱ\\varrhowe make the next choice of noise parameters

σ\\displaystyle\\sigma=3​T​log⁡\(3/δ\)​C2​ε​\(1\+1\+ε3​log⁡\(3/δ\)\),τ=max⁡\{log⁡\(3​Tδ\),log⁡\(16​Tϱ\)\},ω=6​Tε\\displaystyle=\\frac\{3\\sqrt\{T\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\\textsc\{C\}\}\{\\sqrt\{2\}\\varepsilon\}\\left\(1\+\\sqrt\{1\+\\frac\{\\varepsilon\}\{3\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\}\\right\),\\ \\tau=\\max\\left\\\{\\log\\left\(\\frac\{3T\}\{\\delta\}\\right\),\\log\\left\(\\frac\{16T\}\{\\varrho\}\\right\)\\right\\\},\\ \\omega=\\frac\{6T\}\{\\varepsilon\}\(176\)γ1\\displaystyle\\gamma\_\{1\}=15​2​k1​T​log⁡\(3/δ\)8​ε​\(1\+1\+ε3​log⁡\(3/δ\)\),γ2=2​k1​T\+25​log⁡\(3/δ\)8​ε\\displaystyle\\hskip\-1\.99997pt=\\hskip\-1\.99997pt\\frac\{15\\sqrt\{2k\_\{1\}T\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\}\{8\\varepsilon\}\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt1\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\sqrt\{1\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\frac\{\\varepsilon\}\{3\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\}\\hskip\-1\.49994pt\\right\),\\gamma\_\{2\}\\hskip\-1\.99997pt=\\hskip\-1\.99997pt\\frac\{2k\_\{1\}T\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt25\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{8\\varepsilon\}\(177\)and

γ=\{γ1if​log⁡\(3/δ\)≤\(8​γ125−1\)2⋅25​k1​T32​γ12γ2otherwise\.\\displaystyle\\gamma\\hskip\-1\.99997pt=\\hskip\-1\.99997pt\\begin\{cases\}\\gamma\_\{1\}\\ &\\mathrm\{if\}\\ \\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\\leq\\left\(\\frac\{8\\gamma\_\{1\}\}\{25\}\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt1\\right\)^\{2\}\\hskip\-1\.99997pt\\cdot\\hskip\-1\.99997pt\\frac\{25k\_\{1\}T\}\{32\\gamma^\{2\}\_\{1\}\}\\\\ \\gamma\_\{2\}\\ &\\mathrm\{otherwise\}\\end\{cases\}\.\(178\)
#### E\.2\.1Differential Privacy:

Algorithm[2](https://arxiv.org/html/2605.30600#alg2)involves2​T2Tcalls to private mechanisms:TTcalls to the Gaussian mechanism for constructingG~t\\widetilde\{G\}\_\{t\}and anotherTTcalls for forming\(X~t,η~t\)\(\\widetilde\{X\}\_\{t\},\\widetilde\{\\eta\}\_\{t\}\)\(splitted between the functionMultipleFastMixingand the fast sketching step in Line\.[6](https://arxiv.org/html/2605.30600#alg2.l6)\)\. Inside the functionMultipleFastMixing,m~t\\widetilde\{m\}\_\{t\}andλ~t\\widetilde\{\\lambda\}\_\{t\}are computed via the Laplace mechanism\. Moreover, given these private values, the RDP guarantees for the steps computingX~t\\widetilde\{X\}\_\{t\}are known due to Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)\. Throughout, quantities computed in thett’th call to[FastMix](https://arxiv.org/html/2605.30600#alg1)are indexed bytt\(\\Big\(for example,m~t,λ~t,𝖲H,t,𝖲G,t\\widetilde\{m\}\_\{t\},\\widetilde\{\\lambda\}\_\{t\},\\mathsf\{S\}\_\{\\mathrm\{H\},t\},\\mathsf\{S\}\_\{\\mathrm\{G\},t\}\)\\Big\)\.

Our proof uses Lemma[7](https://arxiv.org/html/2605.30600#Thmlem7)and Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1)to calculate the RDP guarantees for each step formingG~t\\widetilde\{G\}\_\{t\}andX~t\\widetilde\{X\}\_\{t\}, and then applies the RDP composition theorem to obtain their overall guarantees separately\. In particular, we separate the contribution of the Laplace mechanisms from the overall RDP guarantees of the output of[FastMix](https://arxiv.org/html/2605.30600#alg1)\. We then convert these RDP guarantees to\(ε,δ\)\(\\varepsilon,\\delta\)\-DP, and combine them with theε\\varepsilon\-DP guarantee of the steps involving the Laplace mechanism\.

Generating\{m~t,λ~t\}t=0T−1\\left\\\{\\widetilde\{m\}\_\{t\},\\widetilde\{\\lambda\}\_\{t\}\\right\\\}^\{T\-1\}\_\{t=0\}\.As proved in Lemma[4](https://arxiv.org/html/2605.30600#Thmlem4), theℓ1\\ell\_\{1\}sensitivity of eachm~t\\widetilde\{m\}\_\{t\}isΔ^t\\widehat\{\\Delta\}\_\{t\}, and as proved in Lemma[5](https://arxiv.org/html/2605.30600#Thmlem5), theℓ1\\ell\_\{1\}sensitivity ofλ~t\\widetilde\{\\lambda\}\_\{t\}isΛ^t\+2​m^t\\widehat\{\\Lambda\}\_\{t\}\+2\\widehat\{m\}\_\{t\}\. Thus, under the event ensuring thatm~t\\widetilde\{m\}\_\{t\}provides an upper bound onm^t\\widehat\{m\}\_\{t\}, and since we use the Laplace mechanism for the release of these quantities, for the value ofω\\omegawe use, the contribution of each of these steps isε6​T\\frac\{\\varepsilon\}\{6T\}, which after composing both steps overTTiterations yields overall contribution ofε3\\frac\{\\varepsilon\}\{3\}to the overallε\\varepsilon\.

Moreover, it holds thatℙ​\(m~t≥m^t\)=ℙ​\(λ~t≤λmin𝖲H,t​X\)=1−12⋅exp⁡\{−τ\}≥1−δ6​T\\mathbb\{P\}\\left\(\\widetilde\{m\}\_\{t\}\\geq\\widehat\{m\}\_\{t\}\\right\)=\\mathbb\{P\}\\left\(\\widetilde\{\\lambda\}\_\{t\}\\leq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}\_\{\\min\}\\right\)=1\-\\frac\{1\}\{2\}\\cdot\\exp\\left\\\{\-\\tau\\right\\\}\\geq 1\-\\frac\{\\delta\}\{6T\}\. Thus, definingℬ≔⋂t=0T−1\{\{m~t≥m^t\}∩\{λ~t≤λmin𝖲H,t​X\}\},\\mathcal\{B\}\\coloneq\\bigcap^\{T\-1\}\_\{t=0\}\\left\\\{\\left\\\{\\widetilde\{m\}\_\{t\}\\geq\\widehat\{m\}\_\{t\}\\right\\\}\\cap\\left\\\{\\widetilde\{\\lambda\}\_\{t\}\\leq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}\_\{\\min\}\\right\\\}\\right\\\},it holds thatℙ​\(ℬ\)≥\(1−δ6​T\)2​T≥1−δ3\.\\mathbb\{P\}\\left\(\\mathcal\{B\}\\right\)\\geq\\left\(1\-\\frac\{\\delta\}\{6T\}\\right\)^\{2T\}\\geq 1\-\\frac\{\\delta\}\{3\}\.

Gaussian Mechanism\.Let\(X′,Y′\)≃\(X,Y\)\(X^\{\\prime\},Y^\{\\prime\}\)\\simeq\(X,Y\)i\.e\.,\(X′,Y′\)\(X^\{\\prime\},Y^\{\\prime\}\)is a zero\-out neighbor of\(X,Y\)\(X,Y\)\. Then, since we setCX=1\\textsc\{C\}\_\{X\}=1, by the Cauchy\-Schwarz inequality we get that

‖X′⁣⊤​clipC​\(Y′−X′​θ^t\)−X⊤​clipC​\(Y−X​θ^t\)‖≤C\\left\\\|X^\{\\prime\\top\}\\mathrm\{clip\}\_\{\\textsc\{C\}\}\\left\(Y^\{\\prime\}\-X^\{\\prime\}\\widehat\{\\theta\}\_\{t\}\\right\)\-X^\{\\top\}\\mathrm\{clip\}\_\{\\textsc\{C\}\}\\left\(Y\-X\\widehat\{\\theta\}\_\{t\}\\right\)\\right\\\|\\leq\\textsc\{C\}\(179\)which holds since the clipping operation happens coordinate\-wise, thusX′⁣⊤​clipC​\(Y′−X′​θ^t\)−X⊤​clipC​\(Y−X​θ^t\)X^\{\\prime\\top\}\\mathrm\{clip\}\_\{\\textsc\{C\}\}\\left\(Y^\{\\prime\}\-X^\{\\prime\}\\widehat\{\\theta\}\_\{t\}\\right\)\-X^\{\\top\}\\mathrm\{clip\}\_\{\\textsc\{C\}\}\\left\(Y\-X\\widehat\{\\theta\}\_\{t\}\\right\)is of the formclipC​\(yi−xi⊤​θ^t\)⋅xi\\mathrm\{clip\}\_\{\\textsc\{C\}\}\\left\(y\_\{i\}\-x^\{\\top\}\_\{i\}\\widehat\{\\theta\}\_\{t\}\\right\)\\cdot x\_\{i\}for the coordinateiithat have been removed and which by the assumption‖xi‖≤1\\left\\\|x\_\{i\}\\right\\\|\\leq 1is upper bounded byC\. Thus, theℓ2\\ell\_\{2\}sensitivity of the quantityX⊤​clipC​\(Y−X​θ^t\)X^\{\\top\}\\mathrm\{clip\}\_\{\\textsc\{C\}\}\\left\(Y\-X\\widehat\{\\theta\}\_\{t\}\\right\)is at mostCirrespective ofθ^t\\widehat\{\\theta\}\_\{t\}\.

Then, Lemma[7](https://arxiv.org/html/2605.30600#Thmlem7)and \([179](https://arxiv.org/html/2605.30600#A5.E179)\) yield that the RDP guarantee of each step that formsG~t\\widetilde\{G\}\_\{t\}is\(α,α​C22​σ2\)\\left\(\\alpha,\\frac\{\\alpha\\textsc\{C\}^\{2\}\}\{2\\sigma^\{2\}\}\\right\)\.

Fast Sketching\.Under the eventℬ\\mathcal\{B\}it holds

λmin𝖲H,t​X\+η2≥γ⋅\(1\+2⋅maxi∈\[n\]​‖\(𝖲H,t​X\)⊤​𝖲H,t−xi‖\),for​allt=0,1,…,T−1\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}\_\{\\min\}\+\\eta^\{2\}\\geq\\gamma\\cdot\\left\(1\+2\\cdot\\underset\{i\\in\[n\]\}\{\\max\}\\ \\left\\\|\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\right\)^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}\-x\_\{i\}\\right\\\|\\right\),\\ \\ \\mathrm\{for\\ all\}\\ \\ t=0,1,\\ldots,T\-1\(180\)and for all the cases of the algorithm\. Thus, by Lemma[1](https://arxiv.org/html/2605.30600#Thmlem1), the RDP guarantee of each of these steps is upper bounded by\(α,k1⋅ϕ¯​\(α;γ\)\)\\left\(\\alpha,k\_\{1\}\\cdot\\overline\{\\phi\}\(\\alpha;\\gamma\)\\right\)whereϕ¯​\(α;γ\)\\overline\{\\phi\}\(\\alpha;\\gamma\)was defined in Section[3](https://arxiv.org/html/2605.30600#S3)and is given by

ϕ¯​\(α;γ\)≔12​\(α−1\)⋅\(α⋅log⁡\(1−1γ−14​γ2\)−log⁡\(1−αγ−α24​γ2\)\)for​γ\>5/4​and​α∈\(1,4​γ/5\)\.\\overline\{\\phi\}\(\\alpha;\\gamma\)\\hskip\-1\.49994pt\\coloneq\\hskip\-1\.49994pt\\frac\{1\}\{2\(\\alpha\-1\)\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\alpha\\cdot\\log\\left\(\\hskip\-1\.49994pt1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{1\}\{\\gamma\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{1\}\{4\\gamma^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\log\\left\(\\hskip\-1\.49994pt1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{\\alpha\}\{\\gamma\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\frac\{\\alpha^\{2\}\}\{4\\gamma^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\right\)\\ \\ \\mathrm\{for\}\\ \\gamma\>\\nicefrac\{\{5\}\}\{\{4\}\}\\ \\mathrm\{and\}\\ \\alpha\\in\\left\(1,\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\\right\)\.\(181\)
Composition\.We compose theTTdifferent steps of the Gaussian mechanism and theTTsketching steps in RDP\. Then, we convert this to\(ε,δ\)\(\\varepsilon,\\delta\)\-DP via the conversion fromCanonneet al\.\[[2020](https://arxiv.org/html/2605.30600#bib.bib51)\], and then compose with the steps that perform the calculations ofm~t\\widetilde\{m\}\_\{t\}andλ~t\\widetilde\{\\lambda\}\_\{t\}in\(ε,δ\)\(\\varepsilon,\\delta\)\-DP\. For theTTsteps of the Gaussian mechanism, we get

ε^Gauss\\displaystyle\\widehat\{\\varepsilon\}\_\{\\text\{\\tiny Gauss\}\}=minα\>1​\{α​T​C22​σ2\+log⁡\(3/δ\)\+\(α−1\)​log⁡\(1−1/α\)−log⁡\(α\)α−1\}\\displaystyle=\\underset\{\\alpha\>1\}\{\\min\}\\ \\left\\\{\\frac\{\\alpha T\\textsc\{C\}^\{2\}\}\{2\\sigma^\{2\}\}\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\+\(\\alpha\-1\)\\log\(1\-\\nicefrac\{\{1\}\}\{\{\\alpha\}\}\)\-\\log\(\\alpha\)\}\{\\alpha\-1\}\\right\\\}\(182\)≤minα\>1​\{α​T​C22​σ2\+log⁡\(3/δ\)α−1\}\\displaystyle\\leq\\underset\{\\alpha\>1\}\{\\min\}\\ \\left\\\{\\frac\{\\alpha T\\textsc\{C\}^\{2\}\}\{2\\sigma^\{2\}\}\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{\\alpha\-1\}\\right\\\}\(183\)=2​T​log⁡\(3/δ\)​Cσ\+T​C22​σ2\\displaystyle=\\frac\{\\sqrt\{2T\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\\textsc\{C\}\}\{\\sigma\}\+\\frac\{T\\textsc\{C\}^\{2\}\}\{2\\sigma^\{2\}\}\(184\)andδ^Gauss=δ/3\\widehat\{\\delta\}\_\{\\text\{\\tiny Gauss\}\}=\\nicefrac\{\{\\delta\}\}\{\{3\}\}\. For theTTsketching steps, we get

ε^FastMix=min1<α<4​γ/5​\{k1​T⋅ϕ¯​\(α;γ\)\+log⁡\(3/δ\)\+\(α−1\)​log⁡\(1−1/α\)−log⁡\(α\)α−1\}\\widehat\{\\varepsilon\}\_\{\\text\{\\tiny FastMix\}\}=\\underset\{1<\\alpha<\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\}\{\\min\}\\ \\left\\\{k\_\{1\}T\\cdot\\overline\{\\phi\}\(\\alpha;\\gamma\)\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\+\(\\alpha\-1\)\\log\(1\-\\nicefrac\{\{1\}\}\{\{\\alpha\}\}\)\-\\log\(\\alpha\)\}\{\\alpha\-1\}\\right\\\}\(185\)andδ^FastMix=δ/3\\widehat\{\\delta\}\_\{\\text\{\\tiny FastMix\}\}=\\nicefrac\{\{\\delta\}\}\{\{3\}\}\. Then, we get that the overall process is\(ε^,δ^\)\\left\(\\widehat\{\\varepsilon\},\\widehat\{\\delta\}\\right\)\-DP where

δ^=δ3\+δ^FastMix\+δ^Gauss=δ\\widehat\{\\delta\}=\\frac\{\\delta\}\{3\}\+\\widehat\{\\delta\}\_\{\\text\{\\tiny FastMix\}\}\+\\widehat\{\\delta\}\_\{\\text\{\\tiny Gauss\}\}=\\delta\(186\)where the firstδ/3\\nicefrac\{\{\\delta\}\}\{\{3\}\}is by the probability of the eventℬ\\mathcal\{B\}not holding \(which, according to Lemma[6](https://arxiv.org/html/2605.30600#Thmlem6)added as an additional term toδ\\delta\) and whereε^\\widehat\{\\varepsilon\}is

ε^\\displaystyle\\widehat\{\\varepsilon\}≤ε3\+ε^FastMix\+ε^Gauss\\displaystyle\\leq\\frac\{\\varepsilon\}\{3\}\+\\widehat\{\\varepsilon\}\_\{\\text\{\\tiny FastMix\}\}\+\\widehat\{\\varepsilon\}\_\{\\text\{\\tiny Gauss\}\}\(187\)≤ε3\+min1<α<4​γ/5​\{k1​T⋅ϕ¯​\(α;γ\)\+log⁡\(3/δ\)\+\(α−1\)​log⁡\(1−1/α\)−log⁡\(α\)α−1\}\\displaystyle\\leq\\hskip\-2\.5pt\\frac\{\\varepsilon\}\{3\}\\hskip\-2\.5pt\+\\hskip\-2\.5pt\\underset\{1<\\alpha<\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\}\{\\min\}\\hskip\-2\.5pt\\left\\\{k\_\{1\}T\\cdot\\overline\{\\phi\}\(\\alpha;\\gamma\)\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\+\(\\alpha\-1\)\\log\(1\-\\nicefrac\{\{1\}\}\{\{\\alpha\}\}\)\-\\log\(\\alpha\)\}\{\\alpha\-1\}\\right\\\}\(188\)\+2​T​log⁡\(3/δ\)​Cσ\+T​C22​σ2\\displaystyle\\qquad\\qquad\+\\frac\{\\sqrt\{2T\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\\textsc\{C\}\}\{\\sigma\}\+\\frac\{T\\textsc\{C\}^\{2\}\}\{2\\sigma^\{2\}\}\(189\)≤ε3\+min1<α<4​γ/5​\{k1​T⋅ϕ¯​\(α;γ\)\+log⁡\(3/δ\)α−1\}\\displaystyle\\leq\\hskip\-2\.5pt\\frac\{\\varepsilon\}\{3\}\\hskip\-2\.5pt\+\\hskip\-2\.5pt\\underset\{1<\\alpha<\\nicefrac\{\{4\\gamma\}\}\{\{5\}\}\}\{\\min\}\\hskip\-2\.5pt\\left\\\{k\_\{1\}T\\cdot\\overline\{\\phi\}\(\\alpha;\\gamma\)\+\\frac\{\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\{\\alpha\-1\}\\right\\\}\(190\)\+2​T​log⁡\(3/δ\)​Cσ\+T​C22​σ2\.\\displaystyle\\qquad\\qquad\+\\frac\{\\sqrt\{2T\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\}\}\)\}\\textsc\{C\}\}\{\\sigma\}\+\\frac\{T\\textsc\{C\}^\{2\}\}\{2\\sigma^\{2\}\}\.\(191\)Now, for our choice ofσ\\sigma, the sum of the third and fourth terms equalsε/3\\nicefrac\{\{\\varepsilon\}\}\{\{3\}\}\. Moreover, by \([134](https://arxiv.org/html/2605.30600#A4.E134)\) in Appendix[D](https://arxiv.org/html/2605.30600#A4), for our choice ofγ\\gammathe second term is upper bounded byε/3\\nicefrac\{\{\\varepsilon\}\}\{\{3\}\}\. Combining these bounds with Lemma[6](https://arxiv.org/html/2605.30600#Thmlem6)yieldsε^=ε\\widehat\{\\varepsilon\}=\\varepsilonand concludes the proof\. ∎

#### E\.2\.2Excess Empirical Risk:

We follow the proof suggested inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]and seek to obtain an high probability bound on‖θ^T−θ∗‖X⊤​X2\\\|\\widehat\{\\theta\}\_\{T\}\-\\theta^\{\*\}\\\|^\{2\}\_\{X^\{\\top\}X\}, which corresponds to the excess empirical risk\[Wang,[2018](https://arxiv.org/html/2605.30600#bib.bib48), Lemma\. 5\]\.

Recall that in terms of notation,m~t\\widetilde\{m\}\_\{t\}denote the private version ofm^t\\widehat\{m\}\_\{t\}andλ~t\\widetilde\{\\lambda\}\_\{t\}the private version ofλ^t\\widehat\{\\lambda\}\_\{t\}, whereλ^t≔λmin𝖲H,t​X\.\\widehat\{\\lambda\}\_\{t\}\\coloneq\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}\_\{\\min\}\.For anya≥0a\\geq 0, we defineX¯a≔\(X⊤,a​𝕀d\)⊤\\overline\{X\}\_\{a\}\\coloneq\(X^\{\\top\},\\,a\\mathbb\{I\}\_\{d\}\)^\{\\top\}andMa≔X⊤​X\+a2​𝕀d=X¯a⊤​X¯aM\_\{a\}\\coloneq X^\{\\top\}X\+a^\{2\}\\mathbb\{I\}\_\{d\}=\\overline\{X\}\_\{a\}^\{\\top\}\\overline\{X\}\_\{a\}\.

##### Step 1: Defining the High\-Probability Event\.

We perform the analysis on an explicit high\-probability event\. For eacht=0,…,T−1t=0,\\ldots,T\-1, define

ℬ1,t​\(χ\)\\displaystyle\\mathcal\{B\}\_\{1,t\}\(\\chi\)≔\{maxi≠j⁡\|ei⊤​𝖲H,t⊤​𝖲H,t​ej\|≤χ\},\\displaystyle\\coloneq\\left\\\{\\max\_\{i\\neq j\}\\left\|e\_\{i\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}e\_\{j\}\\right\|\\leq\\chi\\right\\\},\(192\)ℬ2,t​\(χ\)\\displaystyle\\mathcal\{B\}\_\{2,t\}\(\\chi\)≔\{supu∈ℝn,v∈ℝd\|u⊤​\(𝖲H,t⊤​𝖲H,t−𝕀n\)​X​v\|‖u‖​‖X​v‖≤χ2,infv∈ℝd‖𝖲H,t​X​v‖2‖X​v‖2≥1−χ\}\.\\displaystyle\\coloneq\\left\\\{\\sup\_\{u\\in\\mathbb\{R\}^\{n\},v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\|u^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{H\},t\}^\{\\top\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}\-\\mathbb\{I\}\_\{n\}\\right\)Xv\\right\|\}\{\\left\\\|u\\right\\\|\\left\\\|Xv\\right\\\|\}\\leq\\frac\{\\chi\}\{2\},\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|\\mathsf\{S\}\_\{\\mathrm\{H\},t\}Xv\\right\\\|^\{2\}\}\{\\left\\\|Xv\\right\\\|^\{2\}\}\\geq 1\-\\chi\\right\\\}\.\(193\)
ByPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Lemma\. 1\], there exist constantsCH,cH,c¯HC\_\{\\mathrm\{H\}\},c\_\{\\mathrm\{H\}\},\\bar\{c\}\_\{\\mathrm\{H\}\}such that for anyχ∈\(0,1\]\\chi\\in\(0,1\],ℙ​\(ℬ2,t​\(χ\)\)≥1−c¯H⋅exp⁡\{−cH​k2​χ2log4⁡\(n\)\}\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{2,t\}\(\\chi\)\\right\)\\geq 1\-\\bar\{c\}\_\{\\mathrm\{H\}\}\\cdot\\exp\\left\\\{\-\\frac\{c\_\{\\mathrm\{H\}\}k\_\{2\}\\chi^\{2\}\}\{\\log^\{4\}\(n\)\}\\right\\\}provided thatk2​χ2≥CH⋅rank​\(X\)⋅log4⁡\(n\)\.k\_\{2\}\\chi^\{2\}\\geq C\_\{\\mathrm\{H\}\}\\cdot\\mathrm\{rank\}\(X\)\\cdot\\log^\{4\}\(n\)\.

Moreover, usingX=ejX=e\_\{j\}for allj∈\[n\]j\\in\[n\]in Lemma[9](https://arxiv.org/html/2605.30600#Thmlem9), there existCcoh,ccoh,c¯coh\>0C\_\{\\mathrm\{coh\}\},c\_\{\\mathrm\{coh\}\},\\bar\{c\}\_\{\\mathrm\{coh\}\}\>0such thatℙ​\(ℬ1,t​\(χ\)\)≥1−ϱ4​T\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{1,t\}\(\\chi\)\\right\)\\geq 1\-\\frac\{\\varrho\}\{4T\}provided thatk2​χ2≥Ccoh​\(1\+log⁡\(c¯coh​T​n2χ​ϱ\)​log⁡\(ccoh​T​n2ϱ\)\)\.k\_\{2\}\\chi^\{2\}\\geq C\_\{\\mathrm\{coh\}\}\\left\(1\+\\log\\left\(\\frac\{\\bar\{c\}\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\chi\\varrho\}\\right\)\\log\\left\(\\frac\{c\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\varrho\}\\right\)\\right\)\.Therefore, if

k2​χ2≥max⁡\{CH⋅rank​\(X\)⋅log4⁡\(n\),log4⁡\(n\)cH​log⁡\(4​T​c¯Hϱ\),Ccoh​\(1\+log⁡\(c¯coh​T​n2χ​ϱ\)​log⁡\(ccoh​T​n2ϱ\)\)\},k\_\{2\}\\chi^\{2\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pt\\max\\Bigg\\\{\\hskip\-1\.49994ptC\_\{\\mathrm\{H\}\}\\cdot\\mathrm\{rank\}\(X\)\\cdot\\log^\{4\}\(n\),\\frac\{\\log^\{4\}\(n\)\}\{c\_\{\\mathrm\{H\}\}\}\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{4T\\bar\{c\}\_\{\\mathrm\{H\}\}\}\{\\varrho\}\\hskip\-1\.49994pt\\right\),C\_\{\\mathrm\{coh\}\}\\hskip\-1\.49994pt\\left\(1\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{\\bar\{c\}\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\chi\\varrho\}\\hskip\-1\.49994pt\\right\)\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{c\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\varrho\}\\right\)\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\Bigg\\\},\(194\)thenℙ​\(ℬ1,tc​\(χ\)\)≤ϱ4​T\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{1,t\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)\\leq\\frac\{\\varrho\}\{4T\}andℙ​\(ℬ2,tc​\(χ\)\)≤ϱ4​T\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{2,t\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)\\leq\\frac\{\\varrho\}\{4T\}for everyt=0,1,…,T−1t=0,1,\\ldots,T\-1\. Now, define

ℬtot​\(χ\)≔⋂t=0T−1\(ℬ1,t​\(χ\)∩ℬ2,t​\(χ\)\)\.\\displaystyle\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\)\\coloneq\\bigcap\_\{t=0\}^\{T\-1\}\\left\(\\mathcal\{B\}\_\{1,t\}\(\\chi\)\\cap\\mathcal\{B\}\_\{2,t\}\(\\chi\)\\right\)\.\(195\)By the union bound,

ℙ​\(ℬtotc​\(χ\)\)≤∑t=0T−1ℙ​\(ℬ1,tc​\(χ\)\)\+∑t=0T−1ℙ​\(ℬ2,tc​\(χ\)\)≤T⋅ϱ4​T\+T⋅ϱ4​T=ϱ2,\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{\\mathrm\{tot\}\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)\\leq\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{1,t\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)\+\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{2,t\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)\\leq T\\cdot\\frac\{\\varrho\}\{4T\}\+T\\cdot\\frac\{\\varrho\}\{4T\}=\\frac\{\\varrho\}\{2\},\(196\)and henceℙ​\(ℬtot​\(χ\)\)≥1−ϱ2\\mathbb\{P\}\(\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\)\)\\geq 1\-\\frac\{\\varrho\}\{2\}\. We note that the eventℬtot​\(χ\)\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\)implies

λmin𝖲H,t​X≥\(1−χ\)​λminXandmaxi∈\[n\]⁡‖X⊤​\(𝖲H,t⊤​𝖲H,t−𝕀n\)​ei‖≤χ2​‖X‖op\\lambda^\{\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\}\_\{\\min\}\\geq\(1\-\\chi\)\\lambda^\{X\}\_\{\\min\}\\ \\ \\ \\mathrm\{and\}\\ \\ \\ \\max\_\{i\\in\[n\]\}\\left\\\|X^\{\\top\}\\left\(\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\},t\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|\\leq\\frac\{\\chi\}\{2\}\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\(197\)for allt=0,…,T−1t=0,\\ldots,T\-1\. This holds since for everyi∈\[n\]i\\in\[n\], the eventℬ2,t​\(χ\)\\mathcal\{B\}\_\{2,t\}\(\\chi\)implies

‖X⊤​\(𝖲H,t⊤​𝖲H,t−𝕀n\)​ei‖=sup‖v‖=1\|ei⊤​\(𝖲H,t⊤​𝖲H,t−𝕀n\)​X​v\|≤χ2​sup‖v‖=1‖X​v‖=χ2​‖X‖op\.\\left\\\|X^\{\\top\}\\left\(\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\},t\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}\-\\mathbb\{I\}\_\{n\}\\right\)e\_\{i\}\\right\\\|=\\sup\_\{\\left\\\|v\\right\\\|=1\}\\left\|e\_\{i\}^\{\\top\}\\left\(\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\},t\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}\-\\mathbb\{I\}\_\{n\}\\right\)Xv\\right\|\\leq\\frac\{\\chi\}\{2\}\\sup\_\{\\left\\\|v\\right\\\|=1\}\\left\\\|Xv\\right\\\|=\\frac\{\\chi\}\{2\}\\left\\\|X\\right\\\|\_\{\\mathrm\{op\}\}\.
Conditioning onℬtot​\(χ\)\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\), we next define for eacht=0,…,T−1t=0,\\ldots,T\-1the events

𝒜1,t​\(χ\)≔\{supu∈ℝk2\+d,v∈ℝd\|u⊤​\(1k1​\(𝖲G,t,ξt\)⊤​\(𝖲G,t,ξt\)−𝕀k2\+d\)​\(𝖲H,t​Xη​𝕀d\)​v\|‖u‖​‖\(𝖲H,t​Xη​𝕀d\)​v‖≤χ2,infv∈ℝd‖\(𝖲G,t,ξt\)​\(𝖲H,t​Xη​𝕀d\)​v‖2k1​‖\(𝖲H,t​Xη​𝕀d\)​v‖2≥1−χ\},\\displaystyle\\mathcal\{A\}\_\{1,t\}\(\\chi\)\\hskip\-3\.00003pt\\coloneq\\hskip\-3\.00003pt\\left\\\{\\hskip\-5\.0pt\\begin\{aligned\} &\\sup\_\{u\\in\\mathbb\{R\}^\{k\_\{2\}\+d\},v\\in\\mathbb\{R\}^\{d\}\}\\hskip\-3\.50006pt\\frac\{\\left\|u^\{\\top\}\\hskip\-3\.00003pt\\left\(\\hskip\-1\.99997pt\\frac\{1\}\{k\_\{1\}\}\\hskip\-1\.99997pt\\left\(\\mathsf\{S\}\_\{\\mathrm\{G\},t\},\\xi\_\{t\}\\right\)^\{\\top\}\\hskip\-1\.99997pt\\left\(\\mathsf\{S\}\_\{\\mathrm\{G\},t\},\\xi\_\{t\}\\right\)\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\mathbb\{I\}\_\{k\_\{2\}\+d\}\\right\)\\hskip\-3\.00003pt\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\\\ \\eta\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}\\hskip\-1\.99997ptv\\right\|\}\{\\left\\\|u\\right\\\|\\left\\\|\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\\\ \\eta\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}v\\right\\\|\}\\hskip\-2\.5pt\\leq\\hskip\-2\.5pt\\frac\{\\chi\}\{2\},\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|\(\\mathsf\{S\}\_\{\\mathrm\{G\},t\},\\xi\_\{t\}\)\\hskip\-3\.00003pt\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\\\ \\eta\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}\\hskip\-1\.99997ptv\\right\\\|^\{2\}\}\{k\_\{1\}\\left\\\|\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\\\\ \\eta\\mathbb\{I\}\_\{d\}\\end\{pmatrix\}v\\right\\\|^\{2\}\}\\hskip\-3\.00003pt\\geq\\hskip\-3\.00003pt1\\hskip\-2\.5pt\-\\hskip\-2\.5pt\\chi\\end\{aligned\}\\hskip\-7\.5pt\\right\\\},\(198\)𝒜2,t≔\{m^t≤m~t≤m^t\+12​χ​T​max⁡\{log⁡\(3​T/δ\),log⁡\(16​T/ϱ\)\}ε\},𝒜3,t≔\{𝗓2,t≤τ\},\\displaystyle\\mathcal\{A\}\_\{2,t\}\\coloneq\\left\\\{\\widehat\{m\}\_\{t\}\\leq\\widetilde\{m\}\_\{t\}\\leq\\widehat\{m\}\_\{t\}\+\\frac\{12\\chi T\\max\\left\\\{\\log\\left\(\\nicefrac\{\{3T\}\}\{\{\\delta\}\}\\right\),\\log\\left\(\\nicefrac\{\{16T\}\}\{\{\\varrho\}\}\\right\)\\right\\\}\}\{\\varepsilon\}\\right\\\},\\ \\mathcal\{A\}\_\{3,t\}\\coloneq\\left\\\{\\mathsf\{z\}\_\{2,t\}\\leq\\tau\\right\\\},\(199\)𝒜4,t≔\{‖ζt‖2≤\(2​d\+4​log⁡\(8​T/ϱ\)\)2\}\.\\displaystyle\\mathcal\{A\}\_\{4,t\}\\coloneq\\left\\\{\\left\\\|\\zeta\_\{t\}\\right\\\|^\{2\}\\leq\\left\(\\sqrt\{2d\}\+\\sqrt\{4\\log\(\\nicefrac\{\{8T\}\}\{\{\\varrho\}\}\)\}\\right\)^\{2\}\\right\\\}\.\(200\)
ByPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Lemma\. 1\], there exist constantsCG,cG,c¯G\>0C\_\{\\mathrm\{G\}\},c\_\{\\mathrm\{G\}\},\\bar\{c\}\_\{\\mathrm\{G\}\}\>0such that𝒜1,t​\(χ\)\\mathcal\{A\}\_\{1,t\}\(\\chi\)holds w\.p\. at least1−ϱ8​T1\-\\frac\{\\varrho\}\{8T\}provided thatk1​χ2≥max⁡\{CG​d,1cG​log⁡\(8​T​c¯Gϱ\)\}\.k\_\{1\}\\chi^\{2\}\\geq\\max\\left\\\{C\_\{\\mathrm\{G\}\}d,\\,\\frac\{1\}\{c\_\{\\mathrm\{G\}\}\}\\log\\\!\\left\(\\frac\{8T\\bar\{c\}\_\{\\mathrm\{G\}\}\}\{\\varrho\}\\right\)\\right\\\}\.Moreover, by Lemma[11](https://arxiv.org/html/2605.30600#Thmlem11)applied withξ←ζt\\xi\\leftarrow\\zeta\_\{t\}andϱ←ϱ/8​T\\varrho\\leftarrow\\nicefrac\{\{\\varrho\}\}\{\{8T\}\}it holds that

ℙ​\(𝒜4,tc\)≤\(ϱ8​T\)2≤ϱ8​T,\\mathbb\{P\}\(\\mathcal\{A\}\_\{4,t\}^\{\\mathrm\{c\}\}\)\\leq\\left\(\\frac\{\\varrho\}\{8T\}\\right\)^\{2\}\\leq\\frac\{\\varrho\}\{8T\},\(201\)and by our choice ofτ\\tauand since𝗓2,t∼Lap​\(0,1\)\\mathsf\{z\}\_\{2,t\}\\sim\\mathrm\{Lap\}\(0,1\),

ℙ​\(𝒜3,tc\)≤ϱ8​T\.\\mathbb\{P\}\(\\mathcal\{A\}\_\{3,t\}^\{\\mathrm\{c\}\}\)\\leq\\frac\{\\varrho\}\{8T\}\.\(202\)
Finally, underℬtot\\mathcal\{B\}\_\{\\mathrm\{tot\}\}and for our choice ofτ\\tau,

ℙ​\(𝒜2,tc∣ℬtot​\(χ\)\)≤ϱ8​T\\mathbb\{P\}\\left\(\\mathcal\{A\}\_\{2,t\}^\{\\mathrm\{c\}\}\\mid\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\)\\right\)\\leq\\frac\{\\varrho\}\{8T\}\(203\)which holds since conditioned onℬtot​\(χ\)\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\),𝒜2,t\\mathcal\{A\}\_\{2,t\}is implied by the event\{𝗓1,t∈\[−τ,τ\]\}\\\{\\mathsf\{z\}\_\{1,t\}\\in\[\-\\tau,\\tau\]\\\}\. Now, define

𝒜tot​\(χ\)≔ℬtot​\(χ\)∩⋂t=0T−1\(𝒜1,t​\(χ\)∩𝒜2,t∩𝒜3,t∩𝒜4,t\)\.\\mathcal\{A\}\_\{\\mathrm\{tot\}\}\(\\chi\)\\coloneq\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\)\\cap\\bigcap\_\{t=0\}^\{T\-1\}\\left\(\\mathcal\{A\}\_\{1,t\}\(\\chi\)\\cap\\mathcal\{A\}\_\{2,t\}\\cap\\mathcal\{A\}\_\{3,t\}\\cap\\mathcal\{A\}\_\{4,t\}\\right\)\.\(204\)Then,

ℙ​\(𝒜totc​\(χ\)\)\\displaystyle\\mathbb\{P\}\\left\(\\mathcal\{A\}\_\{\\mathrm\{tot\}\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)≤ℙ\(ℬtotc\(χ\)\)\+ℙ\(⋃t=0T−1\(𝒜1,tc\(χ\)∪𝒜2,tc∪𝒜3,tc∪𝒜4,tc\)\|ℬtot\(χ\)\)\\displaystyle\\leq\\mathbb\{P\}\\left\(\\mathcal\{B\}\_\{\\mathrm\{tot\}\}^\{\\mathrm\{c\}\}\(\\chi\)\\right\)\+\\mathbb\{P\}\\left\(\\bigcup\_\{t=0\}^\{T\-1\}\\bigl\(\\mathcal\{A\}\_\{1,t\}^\{\\mathrm\{c\}\}\(\\chi\)\\cup\\mathcal\{A\}\_\{2,t\}^\{\\mathrm\{c\}\}\\cup\\mathcal\{A\}\_\{3,t\}^\{\\mathrm\{c\}\}\\cup\\mathcal\{A\}\_\{4,t\}^\{\\mathrm\{c\}\}\\bigr\)\\middle\|\\mathcal\{B\}\_\{\\mathrm\{tot\}\}\(\\chi\)\\right\)≤ϱ2\+∑t=0T−1\(ϱ8​T\+ϱ8​T\+ϱ8​T\+ϱ8​T\)\\displaystyle\\leq\\frac\{\\varrho\}\{2\}\+\\sum\_\{t=0\}^\{T\-1\}\\hskip\-1\.49994pt\\left\(\\frac\{\\varrho\}\{8T\}\+\\frac\{\\varrho\}\{8T\}\+\\frac\{\\varrho\}\{8T\}\+\\frac\{\\varrho\}\{8T\}\\right\)=ϱ\\displaystyle=\\varrho\(205\)and which impliesℙ​\(𝒜tot​\(χ\)\)≥1−ϱ,\\mathbb\{P\}\(\\mathcal\{A\}\_\{\\mathrm\{tot\}\}\(\\chi\)\)\\geq 1\-\\varrho,provided that

k1​χ2\\displaystyle k\_\{1\}\\chi^\{2\}≥max⁡\{CG​d,1cG​log⁡\(8​T​c¯Gϱ\)\},\\displaystyle\\geq\\max\\left\\\{C\_\{\\mathrm\{G\}\}d,\\frac\{1\}\{c\_\{\\mathrm\{G\}\}\}\\log\\left\(\\frac\{8T\\bar\{c\}\_\{\\mathrm\{G\}\}\}\{\\varrho\}\\right\)\\right\\\},\(206\)k2​χ2\\displaystyle k\_\{2\}\\chi^\{2\}\\hskip\-1\.49994pt≥max⁡\{CH⋅rank​\(X\)⋅log4⁡\(n\),log4⁡\(n\)cH​log⁡\(4​T​c¯Hϱ\),Ccoh​\(2\+log⁡\(c¯coh​T​n2χ​ϱ\)​log⁡\(ccoh​T​n2ϱ\)\)\},\\displaystyle\\geq\\hskip\-1\.49994pt\\max\\Bigg\\\{\\hskip\-1\.49994ptC\_\{\\mathrm\{H\}\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\mathrm\{rank\}\(X\)\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log^\{4\}\(n\),\\frac\{\\log^\{4\}\(n\)\}\{c\_\{\\mathrm\{H\}\}\}\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{4T\\bar\{c\}\_\{\\mathrm\{H\}\}\}\{\\varrho\}\\hskip\-1\.49994pt\\right\),C\_\{\\mathrm\{coh\}\}\\hskip\-1\.49994pt\\left\(2\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{\\bar\{c\}\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\chi\\varrho\}\\hskip\-1\.49994pt\\right\)\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{c\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\varrho\}\\right\)\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\Bigg\\\},\(207\)for constants\(CG,cG,c¯G,CH,cH,c¯H,Ccoh,ccoh,c¯coh\)\\left\(C\_\{\\mathrm\{G\}\},c\_\{\\mathrm\{G\}\},\\bar\{c\}\_\{\\mathrm\{G\}\},C\_\{\\mathrm\{H\}\},c\_\{\\mathrm\{H\}\},\\bar\{c\}\_\{\\mathrm\{H\}\},C\_\{\\mathrm\{coh\}\},c\_\{\\mathrm\{coh\}\},\\bar\{c\}\_\{\\mathrm\{coh\}\}\\right\)andχ,ϱ∈\(0,1\]\\chi,\\varrho\\in\(0,1\]\.

##### Step 2: Extending Auxiliary Lemmas FromLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]:

Recall that Lemma[8](https://arxiv.org/html/2605.30600#Thmlem8)extendsLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 8\]by showing that the concatenated sketch satisfies𝒞​\(A,χ,k\)\\mathcal\{C\}\(A,\\chi,k\)for a matrixAA, approximation parameterχ\\chi, and sketch dimensionkk\. It remains to extendLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemmas 10–12\]\. For completeness, we restate these lemmas in terms of𝒞​\(A,χ,k\)\\mathcal\{C\}\(A,\\chi,k\)and derive them fromLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]\. We begin by recalling the definition of𝒞​\(A,χ,k\)\\mathcal\{C\}\(A,\\chi,k\):

𝒞​\(A,χ,k\)≔\{S∈ℝk×n:supu∈ℝd,v∈ℝd\|u⊤​A⊤​\(1k​S⊤​S−𝕀n\)​A​v\|‖A​u‖⋅‖A​v‖≤χ2​⋂infv∈ℝd‖S​A​v‖2k⋅‖A​v‖2≥1−χ\}\.\\mathcal\{C\}\(A,\\chi,k\)\\hskip\-1\.49994pt\\coloneq\\hskip\-1\.49994pt\\left\\\{\\hskip\-1\.49994ptS\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathbb\{R\}^\{k\\times n\}:\\hskip\-1\.49994pt\\sup\_\{u\\in\\mathbb\{R\}^\{d\},v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\|u^\{\\top\}A^\{\\top\}\\hskip\-1\.49994pt\\left\(\\frac\{1\}\{k\}S^\{\\top\}S\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathbb\{I\}\_\{n\}\\right\)\\hskip\-1\.49994ptAv\\right\|\}\{\\\|Au\\\|\\cdot\\\|Av\\\|\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\frac\{\\chi\}\{2\}\\bigcap\\inf\_\{v\\in\\mathbb\{R\}^\{d\}\}\\frac\{\\left\\\|SAv\\right\\\|^\{2\}\}\{k\\cdot\\\|Av\\\|^\{2\}\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pt1\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\chi\\hskip\-1\.49994pt\\right\\\}\.\(208\)
###### Lemma 12\.

\(Extension ofLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 10\]\)\. Givenχ∈\(0,1/2\]\\chi\\in\\big\(0,\\nicefrac\{\{1\}\}\{\{2\}\}\\big\],A∈ℝn×d,k∈ℕA\\in\\mathbb\{R\}^\{n\\times d\},k\\in\\mathbb\{N\},B∈𝒞​\(A,χ,k\)B\\in\\mathcal\{C\}\(A,\\chi,k\),b∈ℝnb\\in\\mathbb\{R\}^\{n\}andθ0∈ℝd\\theta\_\{0\}\\in\\mathbb\{R\}^\{d\}we have

‖θ^​\(A,b,B,θ0,0→d\)−θ∗​\(A,b\)‖A⊤​A2≤χ2​‖θ0−θ∗​\(A,b\)‖A⊤​A2\.\\left\\\|\\widehat\{\\theta\}\(A,b,B,\\theta\_\{0\},\\vec\{0\}\_\{d\}\)\-\\theta^\{\*\}\(A,b\)\\right\\\|^\{2\}\_\{A^\{\\top\}A\}\\leq\\chi^\{2\}\\left\\\|\\theta\_\{0\}\-\\theta^\{\*\}\(A,b\)\\right\\\|^\{2\}\_\{A^\{\\top\}A\}\.\(209\)

###### Proof\.

As inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\], the result follows directly from the analysis inPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Appendix\. C\], withθ0\\theta\_\{0\}playing the role ofxtx^\{t\}inPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Equation\. 42\]andBBplaying the role of the sketch𝖲\\mathsf\{S\}\. The desired guarantee then follows from the assumptionB∈𝒞​\(A,χ,k\)B\\in\\mathcal\{C\}\(A,\\chi,k\), since the event inPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Equation\. 43\]corresponds to the setting of𝒞\\mathcal\{C\}, where the left and right vectors multiplying1k​B⊤​B−𝕀n\\frac\{1\}\{k\}B^\{\\top\}B\-\\mathbb\{I\}\_\{n\}are of the formA​vAv\. Hence,Pilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133), Equation\. 44a\]holds under events of the formB∈𝒞​\(A,χ,k\)B\\in\\mathcal\{C\}\(A,\\chi,k\)\. ∎

###### Lemma 13\.

\(Extension ofLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 11\]\)\. Givenχ∈\(0,1\]\\chi\\in\(0,1\],A∈ℝn×dA\\in\\mathbb\{R\}^\{n\\times d\}andB∈ℝk×nB\\in\\mathbb\{R\}^\{k\\times n\}such thatB∈𝒞​\(A,χ,k\)B\\in\\mathcal\{C\}\(A,\\chi,k\)andA⊤​A≻0A^\{\\top\}A\\succ 0it holds

11\+χ2​\(A⊤​A\)−1⪯\(1k​\(B​A\)⊤​\(B​A\)\)−1⪯11−χ2​\(A⊤​A\)−1\\frac\{1\}\{1\+\\frac\{\\chi\}\{2\}\}\\left\(A^\{\\top\}A\\right\)^\{\-1\}\\preceq\\left\(\\frac\{1\}\{k\}\(BA\)^\{\\top\}\(BA\)\\right\)^\{\-1\}\\preceq\\frac\{1\}\{1\-\\frac\{\\chi\}\{2\}\}\\left\(A^\{\\top\}A\\right\)^\{\-1\}\(210\)and furthermore for any vectorv∈ℝdv\\in\\mathbb\{R\}^\{d\}

k2​‖\(\(B​A\)⊤​\(B​A\)\)−1​v‖A⊤​A2≤1\(1−χ2\)2⋅‖v‖2λmin​\(A⊤​A\)\.k^\{2\}\\left\\\|\\left\(\(BA\)^\{\\top\}\(BA\)\\right\)^\{\-1\}v\\right\\\|^\{2\}\_\{A^\{\\top\}A\}\\leq\\frac\{1\}\{\\left\(1\-\\frac\{\\chi\}\{2\}\\right\)^\{2\}\}\\cdot\\frac\{\\left\\\|v\\right\\\|^\{2\}\}\{\\lambda\_\{\\min\}\\left\(A^\{\\top\}A\\right\)\}\.\(211\)

###### Proof\.

The proof followsLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 11\], noting thatLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Equation\. 23\]remains valid under the eventB∈𝒞​\(A,χ,k\)B\\in\\mathcal\{C\}\(A,\\chi,k\), since it only requires multiplying1k​B⊤​B\\frac\{1\}\{k\}B^\{\\top\}Bfrom both sides by a vector of the formA​vAv, thus covered by𝒞​\(A,χ,k\)\\mathcal\{C\}\(A,\\chi,k\)\. The rest of the proof is unchanged\. ∎

###### Lemma 14\.

\(Extension ofLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 12\]\)\. LetY∈ℝnY\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathbb\{R\}^\{n\}andX∈ℝn×dX\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathbb\{R\}^\{n\\times d\}satisfyingCX=1\\textsc\{C\}\_\{X\}\\hskip\-1\.49994pt=\\hskip\-1\.49994pt1, and letM≔X⊤​XM\\hskip\-1\.49994pt\\coloneq\\hskip\-1\.49994ptX^\{\\top\}\\hskip\-1\.49994ptX\. Suppose\{Bt\}t=0T−1\\\{B\_\{t\}\\\}\_\{t=0\}^\{T\-1\}and\{vt\}t=0T−1\\\{v\_\{t\}\\\}\_\{t=0\}^\{T\-1\}satisfyBt∈𝒞​\(X,χ,k\)B\_\{t\}\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathcal\{C\}\\left\(X,\\chi,k\\right\)andvt∈𝒜4​\(ϱ,d\)v\_\{t\}\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathcal\{A\}\_\{4\}\(\\varrho,d\)for allt∈\{0,…,T−1\}t\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\\{0,\\ldots,T\-1\\\}and for parametersχ,ϱ∈\(0,1\]\\chi,\\varrho\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\(0,1\]andk∈ℕk\\hskip\-1\.49994pt\\in\\hskip\-1\.49994pt\\mathbb\{N\}\. Assume thatχ​κ​\(M\)2−χ<1\\frac\{\\chi\\sqrt\{\\kappa\(M\)\}\}\{2\-\\chi\}\\hskip\-1\.49994pt<\\hskip\-1\.49994pt1whereκ​\(M\)≔λmax​\(M\)λmin​\(M\)\.\\kappa\(M\)\\hskip\-1\.49994pt\\coloneq\\hskip\-1\.49994pt\\frac\{\\lambda\_\{\\max\}\(M\)\}\{\\lambda\_\{\\min\}\(M\)\}\.If\{θ^t\}t=0T−1\\\{\\widehat\{\\theta\}\_\{t\}\\\}\_\{t=0\}^\{T\-1\}is defined according to the recursion

θ^t\+1​\(θ^t,σ​vt\)=θ^t\+\(1k​X⊤​Bt⊤​Bt​X\)−1​\(X⊤​\(Y−X​θ^t\)\+σ​vt\),θ^0=0→d,\\widehat\{\\theta\}\_\{t\+1\}\\left\(\\widehat\{\\theta\}\_\{t\},\\sigma v\_\{t\}\\right\)=\\widehat\{\\theta\}\_\{t\}\+\\left\(\\frac\{1\}\{k\}X^\{\\top\}B^\{\\top\}\_\{t\}B\_\{t\}X\\right\)^\{\-1\}\\left\(X^\{\\top\}\\left\(Y\-X\\widehat\{\\theta\}\_\{t\}\\right\)\+\\sigma v\_\{t\}\\right\)~,\\quad\\widehat\{\\theta\}\_\{0\}=\\vec\{0\}\_\{d\},\(212\)forσ≥0\\sigma\\geq 0, then

‖Y−X​θ^t‖∞≤CY\+\(1\+\(χ​κ​\(M\)2−χ\)t\)​‖θ∗‖\+2​σ2−χ\(1\+κ\(M\)\)⋅2​d\+4​log⁡\(8​T/ϱ\)λmin​\(M\)\.\\left\\\|Y\-X\\widehat\{\\theta\}\_\{t\}\\right\\\|\_\{\\infty\}\\leq\\textsc\{C\}\_\{Y\}\+\\left\(1\+\\left\(\\frac\{\\chi\\sqrt\{\\kappa\(M\)\}\}\{2\-\\chi\}\\right\)^\{t\}\\right\)\\left\\\|\\theta^\{\*\}\\right\\\|\+\\frac\{2\\sigma\}\{2\-\\chi\(1\+\\sqrt\{\\kappa\(M\)\)\}\}\\cdot\\frac\{\\sqrt\{2d\}\+\\sqrt\{4\\log\(\\nicefrac\{\{8T\}\}\{\{\\varrho\}\}\)\}\}\{\\lambda\_\{\\min\}\(M\)\}\.\(213\)

###### Proof\.

The proof follows since Lemma[12](https://arxiv.org/html/2605.30600#Thmlem12)and Lemma[13](https://arxiv.org/html/2605.30600#Thmlem13)implies thatLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 11\]andLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Lemma\. 10\]holds\. The remaining steps follow as inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]\. ∎

##### Step 3: Adjusting the Overall Guarantee FromLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]:

Throughout, we denoteX¯η≔\(X⊤,η​𝕀d\)⊤\\overline\{X\}\_\{\\eta\}\\coloneq\\left\(X^\{\\top\},\\eta\\mathbb\{I\}\_\{d\}\\right\)^\{\\top\}\. Before presenting the proof we recall that for anyχ∈\(0,1\]\\chi\\in\(0,1\], conditioned on𝒜tot​\(χ\)\\mathcal\{A\}\_\{\\mathrm\{tot\}\}\(\\chi\)it holds that

\(1−χ\)​λminX≤λ^t≤\(1\+χ\)​λminX,Δ^t≤χ,m^t≤χ​λmaxX\\left\(1\-\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\\leq\\widehat\{\\lambda\}\_\{t\}\\leq\\left\(1\+\\chi\\right\)\\lambda^\{X\}\_\{\\min\},\\ \\ \\widehat\{\\Delta\}\_\{t\}\\leq\\chi,\\ \\ \\widehat\{m\}\_\{t\}\\leq\\chi\\sqrt\{\\lambda^\{X\}\_\{\\max\}\}\(214\)andλ~t≤λ^t,m^t≤m~t≤m^t\+12​χ​T⋅max⁡\{log⁡\(3​T/δ\),log⁡\(16​T/ϱ\)\}ε\\widetilde\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widehat\{\\lambda\}\_\{t\},\\ \\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widetilde\{m\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\frac\{12\\chi T\\cdot\\max\\left\\\{\\log\\left\(\\nicefrac\{\{3T\}\}\{\{\\delta\}\}\\right\),\\log\\left\(\\nicefrac\{\{16T\}\}\{\{\\varrho\}\}\\right\)\\right\\\}\}\{\\varepsilon\}, all holds∀t=0,1,…,T−1\\forall t\\hskip\-1\.49994pt=\\hskip\-1\.49994pt0,1,\\ldots,T\-1\. Consequently,

m¯≔maxt=0,1,…,T−1​m~t≤χ​\(λmaxX\+12​T⋅max⁡\{log⁡\(3​T/δ\),log⁡\(16​T/ϱ\)\}ε\)\.\\overline\{m\}\\coloneq\\underset\{t=0,1,\\ldots,T\-1\}\{\\max\}\\ \{\\widetilde\{m\}\_\{t\}\}\\leq\\chi\\left\(\\sqrt\{\\lambda^\{X\}\_\{\\max\}\}\+\\frac\{12T\\cdot\\max\\left\\\{\\log\\left\(\\nicefrac\{\{3T\}\}\{\{\\delta\}\}\\right\),\\log\\left\(\\nicefrac\{\{16T\}\}\{\{\\varrho\}\}\\right\)\\right\\\}\}\{\\varepsilon\}\\right\)\.\(215\)Additionally,𝒜tot​\(χ\)\\mathcal\{A\}\_\{\\mathrm\{tot\}\}\(\\chi\)holds under the next choice of parameters

k1​χ2\\displaystyle k\_\{1\}\\chi^\{2\}≥max⁡\{CG​d,1cG​log⁡\(8​T​c¯Gϱ\)\},\\displaystyle\\geq\\max\\left\\\{C\_\{\\mathrm\{G\}\}d,\\frac\{1\}\{c\_\{\\mathrm\{G\}\}\}\\log\\left\(\\frac\{8T\\bar\{c\}\_\{\\mathrm\{G\}\}\}\{\\varrho\}\\right\)\\right\\\},\(216\)k2​χ2\\displaystyle k\_\{2\}\\chi^\{2\}\\hskip\-1\.49994pt≥max⁡\{CH⋅rank​\(X\)⋅log4⁡\(n\),log4⁡\(n\)cH​log⁡\(4​T​c¯Hϱ\),Ccoh​\(2\+log⁡\(c¯coh​T​n2χ​ϱ\)​log⁡\(ccoh​T​n2ϱ\)\)\}\.\\displaystyle\\geq\\hskip\-1\.49994pt\\max\\Bigg\\\{\\hskip\-1\.49994ptC\_\{\\mathrm\{H\}\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\mathrm\{rank\}\(X\)\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log^\{4\}\(n\),\\frac\{\\log^\{4\}\(n\)\}\{c\_\{\\mathrm\{H\}\}\}\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{4T\\bar\{c\}\_\{\\mathrm\{H\}\}\}\{\\varrho\}\\hskip\-1\.49994pt\\right\),C\_\{\\mathrm\{coh\}\}\\hskip\-1\.49994pt\\left\(2\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{\\bar\{c\}\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\chi\\varrho\}\\hskip\-1\.49994pt\\right\)\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{c\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\varrho\}\\right\)\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\Bigg\\\}\.\(217\)Lastly, we assume thatγ=γ1\\gamma=\\gamma\_\{1\}, which puts the additional constraintk1​T=Ω​\(log⁡\(1/δ\)\)k\_\{1\}T=\\Omega\\left\(\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\\right\)\.

Starting with the no\-clipping conditions \(parallelingLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Appendix \. C, Step\. 2\]\), we follow the notation fromLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Appendix\. C\]and defineκX​\(a\)≔λmaxX\+aλminX\+a,κ​\(A\)≔λmax​\(A\)λmin​\(A\)\\kappa^\{X\}\(a\)\\hskip\-1\.99997pt\\coloneq\\hskip\-1\.99997pt\\frac\{\\lambda^\{X\}\_\{\\max\}\\hskip\-1\.22911pt\+\\hskip\-1\.22911pta\}\{\\lambda^\{X\}\_\{\\min\}\\hskip\-1\.22911pt\+\\hskip\-1\.22911pta\},\\kappa\(A\)\\hskip\-1\.99997pt\\coloneq\\hskip\-1\.99997pt\\frac\{\\lambda\_\{\\max\}\(A\)\}\{\\lambda\_\{\\min\}\(A\)\}\. First, fort=0t=0no clipping occurs wheneverC≥CY\\textsc\{C\}\\geq\\textsc\{C\}\_\{Y\}\. Then, for everyt≥1t\\geq 1, givenχ∈\(0,1\]\\chi\\hskip\-1\.99997pt\\in\\hskip\-1\.99997pt\(0,1\]satisfyingχ​κX​\(η2\)2−χ<1\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\\hskip\-1\.99997pt<\\hskip\-1\.99997pt1, by Lemma[14](https://arxiv.org/html/2605.30600#Thmlem14)andLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Appendix\. C, Step 2\]

‖Y−X​θ^t‖∞\\displaystyle\\left\\\|Y\-X\\widehat\{\\theta\}\_\{t\}\\right\\\|\_\{\\infty\}≤CY\+‖θ∗‖\+11−χ​κX​\(η2\)2−χ⋅σ1−χ2⋅2​d\+4​log⁡\(8​T/ϱ\)λminX\+η2\+\(χ​κX​\(η2\)2−χ\)t​‖θ∗‖\\displaystyle\\leq\\textsc\{C\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|\+\\frac\{1\}\{1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\}\\cdot\\frac\{\\sigma\}\{1\-\\frac\{\\chi\}\{2\}\}\\cdot\\frac\{\\sqrt\{2d\}\+\\sqrt\{4\\log\\left\(\\nicefrac\{\{8T\}\}\{\{\\varrho\}\}\\right\)\}\}\{\\lambda^\{X\}\_\{\\min\}\+\\eta^\{2\}\}\+\\left\(\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\\right\)^\{t\}\\left\\\|\\theta^\{\*\}\\right\\\|\(218\)≤\(a\)​CY\+‖θ∗‖\+11−χ​κX​\(η2\)2−χ⋅σ1−χ2⋅2​d\+4​log⁡\(8​T/ϱ\)\(1−χ\)​γ\+χ​κX​\(η2\)2−χ​‖θ∗‖\\displaystyle\\overset\{\(a\)\}\{\\leq\}\\textsc\{C\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|\+\\frac\{1\}\{1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\}\\cdot\\frac\{\\sigma\}\{1\-\\frac\{\\chi\}\{2\}\}\\cdot\\frac\{\\sqrt\{2d\}\+\\sqrt\{4\\log\\left\(\\nicefrac\{\{8T\}\}\{\{\\varrho\}\}\\right\)\}\}\{\\left\(1\-\\chi\\right\)\\gamma\}\+\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\\left\\\|\\theta^\{\*\}\\right\\\|\(219\)≤\(b\)​CY\+‖θ∗‖\+11−χ​κX​\(η2\)2−χ⋅C\(1−χ\)2⋅2​d\+4​log⁡\(8​T/ϱ\)k1\+χ​κX​\(η2\)2−χ​‖θ∗‖\\displaystyle\\overset\{\(b\)\}\{\\leq\}\\textsc\{C\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|\+\\frac\{1\}\{1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\}\\cdot\\frac\{\\textsc\{C\}\}\{\\left\(1\-\\chi\\right\)^\{2\}\}\\cdot\\frac\{\\sqrt\{2d\}\+\\sqrt\{4\\log\\left\(\\nicefrac\{\{8T\}\}\{\{\\varrho\}\}\\right\)\}\}\{\\sqrt\{k\_\{1\}\}\}\+\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\\left\\\|\\theta^\{\*\}\\right\\\|\(220\)≤\(c\)​CY\+‖θ∗‖\+χ1−χ​κX​\(η2\)2−χ⋅C\(1−χ\)2\+χ​κX​\(η2\)2−χ​‖θ∗‖\\displaystyle\\overset\{\(c\)\}\{\\leq\}\\textsc\{C\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|\+\\frac\{\\chi\}\{1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\}\\cdot\\frac\{\\textsc\{C\}\}\{\\left\(1\-\\chi\\right\)^\{2\}\}\+\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)\}\}\{2\-\\chi\}\\left\\\|\\theta^\{\*\}\\right\\\|\(221\)where \(a\) is due to \([214](https://arxiv.org/html/2605.30600#A5.E214)\), since

\(1−χ\)​γ≤\(1−χ\)​\(λ^t\+η2\)≤\(1−χ\)​λ^t\+η2≤λ^t1\+χ\+η2≤λminX\+η2,\\left\(1\-\\chi\\right\)\\gamma\\leq\\left\(1\-\\chi\\right\)\\left\(\\widehat\{\\lambda\}\_\{t\}\+\\eta^\{2\}\\right\)\\leq\\left\(1\-\\chi\\right\)\\widehat\{\\lambda\}\_\{t\}\+\\eta^\{2\}\\leq\\frac\{\\widehat\{\\lambda\}\_\{t\}\}\{1\+\\chi\}\+\\eta^\{2\}\\leq\\lambda^\{X\}\_\{\\min\}\+\\eta^\{2\},\(222\)\(b\) is by substitutingγ\\gammaandσ\\sigma, since1\+1\+x1\+1\+x4≤2\\frac\{1\+\\sqrt\{1\+x\}\}\{1\+\\sqrt\{1\+\\frac\{x\}\{4\}\}\}\\leq 2, and \(c\) holds by \([216](https://arxiv.org/html/2605.30600#A5.E216)\), for sufficiently large \(yet still constants\)CG,1cGC\_\{\\mathrm\{G\}\},\\frac\{1\}\{c\_\{\\mathrm\{G\}\}\}andc¯G\\overline\{c\}\_\{\\mathrm\{G\}\}\. This is belowCwhenever

CY1−χ\(1−χ​κX​\(η2\)2−χ\)​\(1−χ\)2\+1\+χ​κX​\(η2\)2−χ1−χ\(1−χ​κX​\(η2\)2−χ\)​\(1−χ\)2​‖θ∗‖≤C\\frac\{\\textsc\{C\}\_\{Y\}\}\{1\-\\frac\{\\chi\}\{\\left\(1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\(\\eta^\{2\}\)\}\}\{2\-\\chi\}\\right\)\(1\-\\chi\)^\{2\}\}\}\+\\frac\{1\+\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\(\\eta^\{2\}\)\}\}\{2\-\\chi\}\}\{1\-\\frac\{\\chi\}\{\\left\(1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\(\\eta^\{2\}\)\}\}\{2\-\\chi\}\\right\)\(1\-\\chi\)^\{2\}\}\}\\left\\\|\\theta^\{\*\}\\right\\\|\\leq\\textsc\{C\}\(223\)provided thatχ\(1−χ​κX​\(η2\)2−χ\)​\(1−χ\)2<1\\frac\{\\chi\}\{\\left\(1\-\\frac\{\\chi\\sqrt\{\\kappa^\{X\}\(\\eta^\{2\}\)\}\}\{2\-\\chi\}\\right\)\(1\-\\chi\)^\{2\}\}<1\. As inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Appendix\. C\], forχ\\chibelow a sufficiently small constant less than14\\frac\{1\}\{4\}, attainable by increasingk1k\_\{1\}andk2k\_\{2\}by constant factors, this bound is less thanCwheneverC≥a0⋅max⁡\{CY,‖θ∗‖\}\\textsc\{C\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pta\_\{0\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\max\\hskip\-1\.49994pt\\left\\\{\\textsc\{C\}\_\{Y\},\\left\\\|\\theta^\{\*\}\\right\\\|\\right\\\}andk1≥a1⋅κX​\(η2\)⋅d,k2≥a1⋅κX​\(η2\)⋅d⋅log4⁡\(n\)k\_\{1\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pta\_\{1\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\kappa^\{X\}\(\\eta^\{2\}\)\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994ptd,\\ k\_\{2\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pta\_\{1\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\kappa^\{X\}\(\\eta^\{2\}\)\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994ptd\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\log^\{4\}\(n\)for constantsa0,a1a\_\{0\},a\_\{1\}\. Now, to further simplify this argument, we note that

κX​\(η2\)=λmaxX−λminXλminX\+η2\+1≤λmaxX−λminX\(1−χ\)​γ\+1\\kappa^\{X\}\\left\(\\eta^\{2\}\\right\)=\\frac\{\\lambda^\{X\}\_\{\\max\}\-\\lambda^\{X\}\_\{\\min\}\}\{\\lambda^\{X\}\_\{\\min\}\+\\eta^\{2\}\}\+1\\leq\\frac\{\\lambda^\{X\}\_\{\\max\}\-\\lambda^\{X\}\_\{\\min\}\}\{\\left\(1\-\\chi\\right\)\\gamma\}\+1\(224\)which holds by using \([222](https://arxiv.org/html/2605.30600#A5.E222)\)\. Then, provided thatχ∈\(0,1/4\]\\chi\\in\(0,\\nicefrac\{\{1\}\}\{\{4\}\}\], following analysis similar to that presented inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]we get that the constraints onk1,k2k\_\{1\},k\_\{2\}can be enforced by setting

k1≥a2⋅\(d⋅\(λmaxX−λminX\)​εlog⁡\(1/δ\)\)2/3,k2≥a2⋅\(d⋅\(λmaxX−λminX\)​εlog⁡\(1/δ\)⋅log4⁡\(n\)\)2/3\.k\_\{1\}\\geq a\_\{2\}\\cdot\\left\(d\\cdot\\frac\{\\left\(\\lambda^\{X\}\_\{\\max\}\-\\lambda^\{X\}\_\{\\min\}\\right\)\\varepsilon\}\{\\sqrt\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\\right\)^\{\\nicefrac\{\{2\}\}\{\{3\}\}\},\\ \\ \\ k\_\{2\}\\geq a\_\{2\}\\cdot\\left\(d\\cdot\\frac\{\\left\(\\lambda^\{X\}\_\{\\max\}\-\\lambda^\{X\}\_\{\\min\}\\right\)\\varepsilon\}\{\\sqrt\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\\cdot\\log^\{4\}\(n\)\\right\)^\{\\nicefrac\{\{2\}\}\{\{3\}\}\}\.\(225\)Overall, conditioned on𝒜tot\\mathcal\{A\}\_\{\\mathrm\{tot\}\}clipping does not occur provided thatC≥a0⋅max⁡\{CY,‖θ∗‖\}\\textsc\{C\}\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pta\_\{0\}\\cdot\\max\\hskip\-1\.49994pt\\left\\\{\\textsc\{C\}\_\{Y\},\\left\\\|\\theta^\{\*\}\\right\\\|\\right\\\}and

k1​χ2\\displaystyle k\_\{1\}\\chi^\{2\}≥a2⋅max⁡\{d,gX,log⁡\(8​T​c¯Gϱ\),\(2​d\+4​log⁡\(8​T/ϱ\)\)2\},\\displaystyle\\geq a\_\{2\}\\cdot\\max\\left\\\{d,g^\{X\},\\log\\left\(\\frac\{8T\\bar\{c\}\_\{\\mathrm\{G\}\}\}\{\\varrho\}\\right\),\\left\(\\sqrt\{2d\}\+\\sqrt\{4\\log\\left\(\\nicefrac\{\{8T\}\}\{\{\\varrho\}\}\\right\)\}\\right\)^\{2\}\\right\\\},\(226\)k2​χ2\\displaystyle k\_\{2\}\\chi^\{2\}≥a2⋅max⁡\{rank​\(X\)​log4⁡\(n\),log8/3⁡\(n\)​gX,log4⁡\(n\)​log⁡\(4​T​c¯Hϱ\),2\+log⁡\(c¯coh​T​n2χ​ϱ\)​log⁡\(ccoh​T​n2ϱ\)\}\\displaystyle\\hskip\-1\.49994pt\\geq\\hskip\-1\.49994pta\_\{2\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\max\\hskip\-1\.49994pt\\Bigg\\\{\\hskip\-1\.49994pt\\mathrm\{rank\}\(X\)\\log^\{4\}\(n\),\\log^\{\\nicefrac\{\{8\}\}\{\{3\}\}\}\(n\)g^\{X\},\\log^\{4\}\(n\)\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{4T\\bar\{c\}\_\{\\mathrm\{H\}\}\}\{\\varrho\}\\hskip\-1\.49994pt\\right\),2\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{\\bar\{c\}\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\chi\\varrho\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\log\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{c\_\{\\mathrm\{coh\}\}Tn^\{2\}\}\{\\varrho\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\Bigg\\\}\(227\)forχ≤1/4\\chi\\leq\\nicefrac\{\{1\}\}\{\{4\}\}and constantsa0,a2a\_\{0\},a\_\{2\}, and recall thatgXg^\{X\}was defined in the theorem statement and is given bygX≔\(ε​dlog⁡\(1/δ\)⋅\(λmaxX−λminX\)\)2/3g^\{X\}\\coloneq\\left\(\\frac\{\\varepsilon d\}\{\\sqrt\{\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\\cdot\\left\(\\lambda^\{X\}\_\{\\max\}\-\\lambda^\{X\}\_\{\\min\}\\right\)\\right\)^\{\\nicefrac\{\{2\}\}\{\{3\}\}\}\.

Now, given a vectorθ\\theta, let

θ^t\+1​\(η,σ;θ\)≔θ\+\(1k1​\(𝖲tot,t​X¯η\)⊤​\(𝖲tot,t​X¯η\)\)−1​\(X⊤​\(Y−X​θ\)−η2​θ\+σ​ζt\)\\widehat\{\\theta\}\_\{t\+1\}\\left\(\\eta,\\sigma;\\theta\\right\)\\coloneq\\theta\+\\left\(\\frac\{1\}\{k\_\{1\}\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{tot\},t\}\\overline\{X\}\_\{\\eta\}\\right\)^\{\\top\}\\left\(\\mathsf\{S\}\_\{\\mathrm\{tot\},t\}\\overline\{X\}\_\{\\eta\}\\right\)\\right\)^\{\-1\}\\left\(X^\{\\top\}\(Y\-X\\theta\)\-\{\\eta\}^\{2\}\\theta\+\\sigma\\zeta\_\{t\}\\right\)\\hskip\-1\.49994pt\(228\)where𝖲tot,t≔\(𝖲G,t,ξt\)​\(𝖲H,t0k2×d0d×n𝕀d\)\\mathsf\{S\}\_\{\\mathrm\{tot\},t\}\\coloneq\\left\(\\mathsf\{S\}\_\{\\mathrm\{G\},t\},\\xi\_\{t\}\\right\)\\begin\{pmatrix\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}&0\_\{k\_\{2\}\\times d\}\\\\ 0\_\{d\\times n\}&\\mathbb\{I\}\_\{d\}\\end\{pmatrix\},θ^t\+1​\(η,σ,θ^t\)\\widehat\{\\theta\}\_\{t\+1\}\\left\(\\eta,\\sigma,\\widehat\{\\theta\}\_\{t\}\\right\)is the update step from Line[8](https://arxiv.org/html/2605.30600#alg2.l8), and we omitted the clipping provided that the previous conditions are met\. Then, using Lemma[13](https://arxiv.org/html/2605.30600#Thmlem13)and Lemma[12](https://arxiv.org/html/2605.30600#Thmlem12)the analysis presented inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149), Appendix\. C, Step\. 3\]holds and yields

‖θ^T​\(η,σ\)−θ∗‖X⊤​X2≤O​\(σ2​max⁡\{d,log⁡\(T/ϱ\)\}λminX\+η2\)\+\(2​χ\)2​T​κX​\(η2\)​\(λminX\+η2\)​‖θ∗‖2\+η2​‖θ∗‖2\.\\left\\\|\\widehat\{\\theta\}\_\{T\}\(\\eta,\\sigma\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\theta^\{\*\}\\right\\\|^\{2\}\_\{X^\{\\top\}X\}\\hskip\-2\.5pt\\leq\\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\hskip\-1\.99997pt\\frac\{\\sigma^\{2\}\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\right\\\}\}\{\\lambda^\{X\}\_\{\\min\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\eta^\{2\}\}\\hskip\-1\.99997pt\\right\)\\hskip\-2\.5pt\+\\hskip\-2\.5pt\\left\(\\sqrt\{2\}\\chi\\right\)^\{2T\}\\hskip\-3\.00003pt\\kappa^\{X\}\\hskip\-1\.99997pt\\left\(\\eta^\{2\}\\right\)\\hskip\-1\.99997pt\\left\(\\lambda^\{X\}\_\{\\min\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\eta^\{2\}\\right\)\\hskip\-1\.49994pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\eta^\{2\}\\hskip\-1\.49994pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\.\(229\)Then, substituting

γ=γ1=Θ​\(k1​T​log⁡\(1/δ\)ε\),σ2=Θ​\(T​log⁡\(1/δ\)​C2ε2\),m^t≤χ​λmaxX,λ~t≤λ^t\\displaystyle\\gamma\\hskip\-1\.49994pt=\\hskip\-1\.49994pt\\gamma\_\{1\}\\hskip\-1\.49994pt=\\hskip\-1\.49994pt\\Theta\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{\\sqrt\{k\_\{1\}T\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\}\{\\varepsilon\}\\hskip\-1\.49994pt\\right\),\\ \\sigma^\{2\}\\hskip\-1\.49994pt=\\hskip\-1\.49994pt\\Theta\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{T\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\\textsc\{C\}^\{2\}\}\{\\varepsilon^\{2\}\}\\hskip\-1\.49994pt\\right\),\\ \\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\chi\\sqrt\{\\lambda^\{X\}\_\{\\max\}\},\\ \\widetilde\{\\lambda\}\_\{t\}\\leq\\widehat\{\\lambda\}\_\{t\}\(230\)\(1−χ\)​λminX≤λ^t≤\(1\+χ\)​λminX,m^t≤m~t≤m^t\+12​χ​T⋅max⁡\{log⁡\(3​T/δ\),log⁡\(16​T/ϱ\)\}ε,\\displaystyle\\left\(1\-\\chi\\right\)\\lambda^\{X\}\_\{\\min\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widehat\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\left\(1\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\chi\\right\)\\lambda^\{X\}\_\{\\min\},\\ \\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widetilde\{m\}\_\{t\}\\hskip\-1\.49994pt\\leq\\hskip\-1\.49994pt\\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\frac\{12\\chi T\\cdot\\max\\left\\\{\\log\\left\(\\nicefrac\{\{3T\}\}\{\{\\delta\}\}\\right\),\\log\\left\(\\nicefrac\{\{16T\}\}\{\{\\varrho\}\}\\right\)\\right\\\}\}\{\\varepsilon\},\(231\)λ^t\+η2=\{γ​\(1\+2​m~t\)if​η2\>0λ^totherwise,\\displaystyle\\qquad\\qquad\\qquad\\qquad\\qquad\\widehat\{\\lambda\}\_\{t\}\+\\eta^\{2\}=\\begin\{cases\}\\gamma\\left\(1\+2\\widetilde\{m\}\_\{t\}\\right\)\\ &\\mathrm\{if\}\\ \\eta^\{2\}\>0\\\\ \\widehat\{\\lambda\}\_\{t\}\\ &\\mathrm\{otherwise\}\\end\{cases\},\(232\)λminX\+η2≤λ^t1−χ\+η2≤λ^t\+η21−χ,λminX\+η2≥λ^t1\+χ\+η2≥λ^t\+η21\+χ\\displaystyle\\qquad\\qquad\\lambda^\{X\}\_\{\\min\}\+\\eta^\{2\}\\leq\\frac\{\\widehat\{\\lambda\}\_\{t\}\}\{1\-\\chi\}\+\\eta^\{2\}\\leq\\frac\{\\widehat\{\\lambda\}\_\{t\}\+\\eta^\{2\}\}\{1\-\\chi\},\\ \\ \\ \\lambda^\{X\}\_\{\\min\}\+\\eta^\{2\}\\geq\\frac\{\\widehat\{\\lambda\}\_\{t\}\}\{1\+\\chi\}\+\\eta^\{2\}\\geq\\frac\{\\widehat\{\\lambda\}\_\{t\}\+\\eta^\{2\}\}\{1\+\\chi\}\(233\)in \([229](https://arxiv.org/html/2605.30600#A5.E229)\) and using the notationm¯≔maxt=0,1,…,T−1​m~t\\overline\{m\}\\coloneq\\underset\{t=0,1,\\ldots,T\-1\}\{\\max\}\\widetilde\{m\}\_\{t\}yields

‖θ^T​\(η,σ\)−θ∗‖X⊤​X2\\displaystyle\\left\\\|\\widehat\{\\theta\}\_\{T\}\(\\eta,\\sigma\)\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\theta^\{\*\}\\right\\\|^\{2\}\_\{X^\{\\top\}X\}\\hskip\-1\.99997pt\(234\)≤O​\(σ2​max⁡\{d,log⁡\(T/ϱ\)\}λ^t\+η2\)\+\(2​χ\)2​T​κX​\(η2\)​\(λ^t\+η21−χ\)​‖θ∗‖2\+η2​‖θ∗‖2,for​all​t=0,1,…,T−1,\\displaystyle\\leq\\hskip\-1\.99997ptO\\hskip\-1\.49994pt\\left\(\\hskip\-1\.49994pt\\frac\{\\sigma^\{2\}\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\right\\\}\}\{\\widehat\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\eta^\{2\}\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\left\(\\sqrt\{2\}\\chi\\right\)^\{2T\}\\hskip\-1\.99997pt\\kappa^\{X\}\\hskip\-1\.99997pt\\left\(\\eta^\{2\}\\right\)\\hskip\-1\.99997pt\\left\(\\hskip\-1\.49994pt\\frac\{\\widehat\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\eta^\{2\}\}\{1\-\\chi\}\\hskip\-1\.49994pt\\right\)\\hskip\-1\.49994pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\eta^\{2\}\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\},\\ \\mathrm\{for\\ all\\ \}t=0,1,\\ldots,T\-1,\(235\)≤\{O​\(σ2​max⁡\{d,log⁡\(T/ϱ\)\}γ\)\+\(2​χ\)2​T​κX​\(η2\)​γ​\(1\+2​m¯\)1−χ​‖θ∗‖2\+η2​‖θ∗‖2if​λminX<γ​\(1\+2​m¯\)1−χO​\(σ2​max⁡\{d,log⁡\(T/ϱ\)\}λminX\)\+\(2​χ\)2​T​κX​\(0\)​λminX​1\+χ1−χ​‖θ∗‖2otherwise\\displaystyle\\leq\\hskip\-1\.99997pt\\begin\{cases\}\\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\frac\{\\sigma^\{2\}\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\right\\\}\}\{\\gamma\}\\right\)\+\\left\(\\sqrt\{2\}\\chi\\right\)^\{2T\}\\kappa^\{X\}\(\\eta^\{2\}\)\\frac\{\\gamma\\left\(1\+2\\overline\{m\}\\right\)\}\{1\-\\chi\}\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\+\\eta^\{2\}\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\qquad\\qquad\\qquad\\qquad\\qquad&\\mathrm\{if\}\\ \\lambda^\{X\}\_\{\\min\}\\hskip\-3\.50006pt<\\hskip\-3\.50006pt\\frac\{\\gamma\\left\(1\+2\\overline\{m\}\\right\)\}\{1\-\\chi\}\\\\ \\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\frac\{\\sigma^\{2\}\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\right\\\}\}\{\\lambda^\{X\}\_\{\\min\}\}\\right\)\+\\left\(\\sqrt\{2\}\\chi\\right\)^\{2T\}\\kappa^\{X\}\(0\)\\lambda^\{X\}\_\{\\min\}\\frac\{1\+\\chi\}\{1\-\\chi\}\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}&\\mathrm\{otherwise\}\\end\{cases\}\(236\)≤\{O​\(T​max⁡\{d,log⁡\(T/ϱ\),gX\}​log⁡\(1/δ\)​\(CY2\+‖θ∗‖2\)​\(1\+2​m¯\)ε​\(1\+\(2​χ\)2​T​κX​\(η2\)\+11\+2​m¯\)−λminX​\(CY2\+‖θ∗‖2\)\)if​λminX<γ​\(1\+2​m¯\)1−χO​\(T​max⁡\{d,log⁡\(T/ϱ\)\}​log⁡\(1/δ\)​\(CY2\+‖θ∗‖2\)ε2​λminX\+\(2​χ\)2​T​κX​\(0\)​λminX​‖θ∗‖2\)otherwise\\displaystyle\\leq\\hskip\-1\.99997pt\\begin\{cases\}\\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\hskip\-2\.5pt\\frac\{\\sqrt\{T\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\),g^\{X\}\\right\\\}\\hskip\-0\.81949pt\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\\left\(\\hskip\-0\.81949pt\\textsc\{C\}^\{2\}\_\{Y\}\\hskip\-0\.81949pt\+\\hskip\-0\.81949pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\hskip\-0\.81949pt\\right\)\\left\(\\hskip\-0\.81949pt1\\hskip\-0\.81949pt\+\\hskip\-0\.81949pt2\\overline\{m\}\\right\)\}\{\\varepsilon\}\\hskip\-3\.00003pt\\left\(\\hskip\-1\.99997pt1\\hskip\-2\.5pt\+\\hskip\-2\.5pt\(\\sqrt\{2\}\\chi\)^\{2T\}\\hskip\-1\.49994pt\\kappa^\{X\}\(\\eta^\{2\}\)\\hskip\-2\.5pt\+\\hskip\-2\.5pt\\frac\{1\}\{1\\hskip\-0\.81949pt\+\\hskip\-0\.81949pt2\\overline\{m\}\}\\right\)\\hskip\-2\.5pt\-\\hskip\-2\.5pt\\lambda^\{X\}\_\{\\min\}\\hskip\-2\.5pt\\left\(\\hskip\-1\.99997pt\\textsc\{C\}^\{2\}\_\{Y\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\hskip\-1\.99997pt\\right\)\\hskip\-2\.5pt\\right\)&\\mathrm\{if\}\\ \\lambda^\{X\}\_\{\\min\}\\hskip\-3\.50006pt<\\hskip\-3\.50006pt\\frac\{\\gamma\\left\(1\\hskip\-0\.81949pt\+\\hskip\-0\.81949pt2\\overline\{m\}\\right\)\}\{1\\hskip\-0\.81949pt\-\\hskip\-0\.81949pt\\chi\}\\\\ \\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\hskip\-1\.99997pt\\frac\{T\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\right\\\}\\hskip\-0\.81949pt\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\\left\(\\hskip\-0\.81949pt\\textsc\{C\}^\{2\}\_\{Y\}\\hskip\-0\.81949pt\+\\hskip\-0\.81949pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\hskip\-0\.81949pt\\right\)\}\{\\varepsilon^\{2\}\\lambda^\{X\}\_\{\\min\}\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\left\(\\sqrt\{2\}\\chi\\right\)^\{2T\}\\hskip\-1\.99997pt\\kappa^\{X\}\(0\)\\lambda^\{X\}\_\{\\min\}\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\hskip\-1\.99997pt\\right\)&\\mathrm\{otherwise\}\\end\{cases\}\(237\)≤\{O​\(T​max⁡\{d,log⁡\(T/ϱ\),gX\}​log⁡\(1/δ\)​\(CY2\+‖θ∗‖2\)​\(1\+2​m¯\)ε​\(1\+11\+2​m¯\)−λminX​\(CY2\+‖θ∗‖2\)\)if​λminX<γ​\(1\+2​m¯\)1−χO​\(T​max⁡\{d,log⁡\(T/ϱ\)\}​log⁡\(1/δ\)​\(CY2\+‖θ∗‖2\)ε2​λminX\+8−T​λminX​‖θ∗‖2\)otherwise\\displaystyle\\leq\\hskip\-1\.99997pt\\begin\{cases\}\\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\frac\{\\sqrt\{T\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\),g^\{X\}\\right\\\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\\left\(\\textsc\{C\}^\{2\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\right\)\\left\(1\+2\\overline\{m\}\\right\)\}\{\\varepsilon\}\\left\(1\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\frac\{1\}\{1\+2\\overline\{m\}\}\\right\)\\hskip\-1\.99997pt\-\\hskip\-1\.99997pt\\lambda^\{X\}\_\{\\min\}\\hskip\-1\.99997pt\\left\(\\textsc\{C\}^\{2\}\_\{Y\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\right\)\\right\)\\qquad\\qquad\\quad&\\mathrm\{if\}\\ \\lambda^\{X\}\_\{\\min\}\\hskip\-3\.50006pt<\\hskip\-3\.50006pt\\frac\{\\gamma\\left\(1\+2\\overline\{m\}\\right\)\}\{1\-\\chi\}\\\\ \\hskip\-2\.5ptO\\hskip\-2\.5pt\\left\(\\frac\{T\\max\\left\\\{d,\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\right\\\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\\left\(\\textsc\{C\}^\{2\}\_\{Y\}\+\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\right\)\}\{\\varepsilon^\{2\}\\lambda^\{X\}\_\{\\min\}\}\+8^\{\-T\}\\lambda^\{X\}\_\{\\min\}\\left\\\|\\theta^\{\*\}\\right\\\|^\{2\}\\right\)&\\mathrm\{otherwise\}\\end\{cases\}\(238\)where we have used the lower boundλ^t\+η2=γ​\(1\+2​m~t\)≥γ\\widehat\{\\lambda\}\_\{t\}\+\\eta^\{2\}=\\gamma\(1\+2\\widetilde\{m\}\_\{t\}\)\\geq\\gammafor the case whereη2\>0\\eta^\{2\}\>0and the last step is forχ≤14\\chi\\hskip\-2\.5pt\\leq\\hskip\-2\.5pt\\frac\{1\}\{4\}andχ​κX​\(η2\)2−χ<1\\frac\{\\chi\\sqrt\{\\hskip\-0\.81949pt\\kappa^\{X\}\\hskip\-0\.81949pt\(\\eta^\{2\}\)\}\}\{2\-\\chi\}\\hskip\-2\.5pt<\\hskip\-2\.5pt1\. In the first case, the optimalTTis constant; in the second,T⋆=O​\(log⁡\(\(ε​λminX\)2max⁡\{d,log⁡\(1/ϱ\)\}​log⁡\(1/δ\)\)\)\.T^\{\\star\}\\hskip\-2\.5pt=\\hskip\-2\.5ptO\\hskip\-3\.00003pt\\left\(\\hskip\-1\.99997pt\\log\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\frac\{\(\\varepsilon\\lambda\_\{\\min\}^\{X\}\)^\{2\}\}\{\\max\\left\\\{d,\\log\(\\nicefrac\{\{1\}\}\{\{\\varrho\}\}\)\\hskip\-0\.81949pt\\right\\\}\\hskip\-1\.63885pt\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\\hskip\-1\.99997pt\\right\)\\hskip\-2\.5pt\\right\)\.Substituting this,m¯≤O​\(χ​\(λmaxX\+T​max⁡\{log⁡\(T/δ\),log⁡\(T/ϱ\)\}ε\)\)\\overline\{m\}\\hskip\-2\.5pt\\leq\\hskip\-2\.5ptO\\hskip\-1\.99997pt\\left\(\\hskip\-1\.99997pt\\chi\\hskip\-1\.99997pt\\left\(\\hskip\-2\.5pt\\sqrt\{\\lambda\_\{\\max\}^\{X\}\}\\hskip\-1\.99997pt\+\\hskip\-1\.99997pt\\frac\{T\\max\\left\\\{\\log\(\\nicefrac\{\{T\}\}\{\{\\delta\}\}\),\\log\(\\nicefrac\{\{T\}\}\{\{\\varrho\}\}\)\\hskip\-0\.81949pt\\right\\\}\}\{\\varepsilon\}\\hskip\-1\.99997pt\\right\)\\hskip\-2\.5pt\\right\), and noting that the asymptotics are unchanged underλminX<γ​\(1\+2​m¯\)\\lambda\_\{\\min\}^\{X\}\\hskip\-2\.5pt<\\hskip\-2\.5pt\\gamma\(\\hskip\-1\.00006pt1\\hskip\-1\.00006pt\+\\hskip\-1\.00006pt2\\overline\{m\}\)completes the proof\. ∎

#### E\.2\.3Runtime Argument:

The algorithm runtime is dominated byTTsteps of[FastMix](https://arxiv.org/html/2605.30600#alg1)\(splitted betweenMultipleFastMixingand Line\.[6](https://arxiv.org/html/2605.30600#alg2.l6)\) andTTtimes of solving a linear regression problem of sizek1×dk\_\{1\}\\times d\(Line\.[8](https://arxiv.org/html/2605.30600#alg2.l8)\)\. The complexity of the former was established in Theorem[1](https://arxiv.org/html/2605.30600#Thmthm1), and the complexity of the latter is given byO​\(k1​d2\+d3\)O\\left\(k\_\{1\}d^\{2\}\+d^\{3\}\\right\)via standard arguments\. Summing these together yields

O​\(T⋅\(k1​k2​d\+n​d​log⁡\(n\)\+k1​d2\+k2​d2\+d3\)\)\.O\\left\(T\\cdot\\left\(k\_\{1\}k\_\{2\}d\+nd\\log\(n\)\+k\_\{1\}d^\{2\}\+k\_\{2\}d^\{2\}\+d^\{3\}\\right\)\\right\)\.\(239\)Substituting the previous values ofk1,k2k\_\{1\},k\_\{2\}andT⋆T^\{\\star\}yields the stated guarantee\. ∎

## Appendix\. FCoherence for SRHT

Recall that for SRHT‖Sf​ei‖2=1\\left\\\|S\_\{\\mathrm\{f\}\}e\_\{i\}\\right\\\|^\{2\}=1for alli∈\[n\]i\\in\[n\]\. Then,Δ^\\widehat\{\\Delta\}\(formally defined in Line[2](https://arxiv.org/html/2605.30600#alg1.l2)\) is the*coherence*ofSfS\_\{\\mathrm\{f\}\}\. By Cauchy\-Schwarz and the normalization of the columns ofSfS\_\{\\mathrm\{f\}\}, it can be shown to be upper\-bounded by11\. Moreover, by Welch’s lower bound\[Welch,[1974](https://arxiv.org/html/2605.30600#bib.bib15), Theorem 1\]\(see also the discussion inHaikinet al\.\[[2021](https://arxiv.org/html/2605.30600#bib.bib150), Section 4\.1\]\),

Δ^≥n−kk​\(n−1\)\.\\widehat\{\\Delta\}\\geq\\sqrt\{\\frac\{n\-k\}\{k\(n\-1\)\}\}\.\(240\)In particular, whennk≫1\\frac\{n\}\{k\}\\gg 1, the optimal coherence scale permitted by Welch’s bound isΘ​\(k−1/2\)\\Theta\\left\(k^\{\-\\nicefrac\{\{1\}\}\{\{2\}\}\}\\right\), while forn≈k\+1n\\approx k\+1it might get as small asΘ​\(k−1\)\\Theta\\left\(k^\{\-1\}\\right\)\. For the SRHT, as shown in Appendix[E](https://arxiv.org/html/2605.30600#A5), w\.p\. at least1−ϱ1\-\\varrhothere exists constants\(C,c,C¯\)\(C,c,\\bar\{C\}\)such thatΔ^≤χ\\widehat\{\\Delta\}\\leq\\chifor everyχ∈\(0,1\]\\chi\\in\(0,1\]satisfying

k​χ2≥C​\(2\+log⁡\(c​n2χ​ϱ\)​log⁡\(c¯​n2ϱ\)\)\.k\\chi^\{2\}\\geq C\\left\(2\+\\log\\left\(\\frac\{cn^\{2\}\}\{\\chi\\varrho\}\\right\)\\log\\left\(\\frac\{\\bar\{c\}n^\{2\}\}\{\\varrho\}\\right\)\\right\)\.\(241\)Hence, the resulting upper bound is of the formΔ^=O~​\(log⁡\(n/ϱ\)k\)\\widehat\{\\Delta\}=\\widetilde\{O\}\\left\(\\frac\{\\log\\left\(\\nicefrac\{\{n\}\}\{\{\\varrho\}\}\\right\)\}\{\\sqrt\{k\}\}\\right\); Relative to Welch’s lower bound, the SRHT coherence bound degrades by a factor of roughlylog⁡\(n/ϱ\)\\log\(\\nicefrac\{\{n\}\}\{\{\\varrho\}\}\)whenevern≫kn\\gg k\.

## Appendix\. GAlgorithms

Algorithm 3Multiple Fast Mixing1:Dataset

X∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}, row norm bound

CX\\textsc\{C\}\_\{X\}, parameters

γ,τ,ω,k,T\\gamma,\\tau,\\omega,k,T\.

2:Set

η=0\\eta=0\.

3:for

t=0,1,…,T−1t=0,1,\\ldots,T\-1do:

4:Sample

𝖲H,t​∼iid​SRHT​\(k,n\)\\mathsf\{S\}\_\{\\mathrm\{H\},t\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\mathrm\{SRHT\}\(k,n\)\.

5:Compute

X~t←𝖲H,t​X\\widetilde\{X\}\_\{t\}\\leftarrow\\mathsf\{S\}\_\{\\mathrm\{H\},t\}X\.

6:Compute

m^t←maxi∈\[n\]​‖X~t⊤​𝖲H,t​ei−xi‖,Δ^t←CX⋅maxi,j:i≠j​\|ei⊤​𝖲H,t⊤​𝖲H,t​ej\|,λ^t←λminX~t\\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\underset\{i\\in\[n\]\}\{\\max\}\\left\\\|\\widetilde\{X\}^\{\\top\}\_\{t\}\\hskip\-1\.49994pt\\mathsf\{S\}\_\{\\mathrm\{H\},t\}e\_\{i\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994ptx\_\{i\}\\right\\\|,\\widehat\{\\Delta\}\_\{t\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\textsc\{C\}\_\{X\}\\hskip\-1\.49994pt\\cdot\\hskip\-1\.49994pt\\underset\{i,j:\\ i\\neq j\}\{\\max\}\\left\|e^\{\\top\}\_\{i\}\\mathsf\{S\}^\{\\top\}\_\{\\mathrm\{H\},t\}\\mathsf\{S\}\_\{\\mathrm\{H\},t\}e\_\{j\}\\right\|,\\widehat\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\lambda^\{\\widetilde\{X\}\_\{t\}\}\_\{\\min\}\.

7:Compute

m~t←max⁡\{0,m^t\+ω​Δ^t​\(τ−𝗓1,t\)\}\\widetilde\{m\}\_\{t\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\max\\left\\\{0,\\widehat\{m\}\_\{t\}\\hskip\-1\.49994pt\+\\hskip\-1\.49994pt\\omega\\widehat\{\\Delta\}\_\{t\}\(\\tau\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathsf\{z\}\_\{1,t\}\)\\right\\\}for

𝗓1,t∼Lap​\(0,1\)\\mathsf\{z\}\_\{1,t\}\\hskip\-1\.49994pt\\sim\\hskip\-1\.49994pt\\mathrm\{Lap\}\(0,1\)\.

8:Compute

λ~t←max⁡\{0,λ^t−ω​CX​\(CX\+2​m~t\)​\(τ−𝗓2,t\)\}\\widetilde\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\\leftarrow\\hskip\-1\.49994pt\\max\\left\\\{0,\\widehat\{\\lambda\}\_\{t\}\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\omega\\textsc\{C\}\_\{X\}\\left\(\\textsc\{C\}\_\{X\}\+2\\widetilde\{m\}\_\{t\}\\right\)\\left\(\\tau\\hskip\-1\.49994pt\-\\hskip\-1\.49994pt\\mathsf\{z\}\_\{2,t\}\\right\)\\right\\\}for

𝗓2,t∼Lap​\(0,1\)\\mathsf\{z\}\_\{2,t\}\\hskip\-1\.49994pt\\sim\\hskip\-1\.49994pt\\mathrm\{Lap\}\(0,1\)\.

9:Set

η~t←max⁡\{0,γ​CX​\(CX\+2​m~t\)−λ~t\}\\widetilde\{\\eta\}\_\{t\}\\leftarrow\\sqrt\{\\max\\left\\\{0,\\gamma\\textsc\{C\}\_\{X\}\\left\(\\textsc\{C\}\_\{X\}\+2\\widetilde\{m\}\_\{t\}\\right\)\-\\widetilde\{\\lambda\}\_\{t\}\\right\\\}\}\.

10:If

η~t≥η\\widetilde\{\\eta\}\_\{t\}\\geq\\etaupdate

η←η~t\\eta\\leftarrow\\widetilde\{\\eta\}\_\{t\}\.

11:Output:

η,\{X~t\}t=0T−1\\eta,\\ \\\{\\widetilde\{X\}\_\{t\}\\\}^\{T\-1\}\_\{t=0\}\.

## Appendix\. HExperimental Details

The experiments were run on 12th Gen Intel\(R\) Core\(TM\) i7\-1255U and implemented in Python 3\.14\. We used five real datasets and two synthetic datasets\. The plots throughout the paper are for the normalized \(divided bynn\) excess empirical risks, as they match the performance metric used in our accuracy theorem\. In all cases, we normalized the training data so that the maximumℓ2\\ell\_\{2\}\-norm of any training sample was 1 \(namely,‖xi‖2≤1\\left\\\|x\_\{i\}\\right\\\|^\{2\}\\leq 1and\|yi\|≤1\\left\|y\_\{i\}\\right\|\\leq 1for alli=1,…,ni=1,\\ldots,n, soCX=CY=1\\textsc\{C\}\_\{X\}=\\textsc\{C\}\_\{Y\}=1\)\. Moreover, to allow compatibility with the Hadamard transform, we subsample \(randomly, without replacement\) the datasets to a number of samples which is a power of22\. The airline dataset was subsampled to41943044194304points, to allow for reasonable solution times\. For performing the Hadamard transform, we have used the optimized implementation from[https://github\.com/falconn\-lib/ffht](https://github.com/falconn-lib/ffht)\.

We report the excess empirical risk\. First, we calculate the normalized mean squared error \(MSE\) for the train set, computed as the squared error in predictingyiy\_\{i\}viaxi⊤​θ^x\_\{i\}^\{\\top\}\\widehat\{\\theta\}, averaged over the train set, and we call this quantity train MSE\. The excess empirical risk is defined as the difference between this quantity and the error attained by the optimal solutionθ∗\\theta^\{\*\}\. The results are averaged over3030independent trials, and we report both the empirical means and 95% confidence intervals, calculated via±1\.96⋅std\#​runs\\pm 1\.96\\cdot\\frac\{\\mathrm\{std\}\}\{\\sqrt\{\\\#\\mathrm\{runs\}\}\}\. For running times, we measure wall\-clock time via the functionperf\_counter\_nsfrom Python’stimelibrary, and report the average across the experiments as well as9595% confidence intervals\.

Throughout the experiments, we fixed the failure probability toϱ=δ/10\\varrho=\\nicefrac\{\{\\delta\}\}\{\{10\}\}, so the system failure probability is only slightly larger than the additive slack in the DP definition,δ\\delta\. Moreover, we setδ=1n2\\delta=\\frac\{1\}\{n^\{2\}\}\. For IHM, we usedk=6⋅max⁡\{d,log⁡\(4​T/ϱ\)\},T=4k=6\\cdot\\max\\left\\\{d,\\log\(\\nicefrac\{\{4T\}\}\{\{\\varrho\}\}\)\\right\\\},T=4andC=CY\\textsc\{C\}=\\textsc\{C\}\_\{Y\}, following similar choices inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\], where since the comparison for[FastIHM](https://arxiv.org/html/2605.30600#alg2)is done for relatively higher range ofε\\varepsilonwe have increasedTTfrom33as was simulated inLevet al\.\[[2026](https://arxiv.org/html/2605.30600#bib.bib149)\]toT=4T=4\. For FastIHM, we usedk1=6⋅max⁡\{d,log⁡\(4​T/ϱ\)\},T=4k\_\{1\}=6\\cdot\\max\\left\\\{d,\\log\(\\nicefrac\{\{4T\}\}\{\{\\varrho\}\}\)\\right\\\},T=4andC=CY\\textsc\{C\}=\\textsc\{C\}\_\{Y\}\. For both schemes, we omitted the term−η2​θ^t\-\\eta^\{2\}\\widehat\{\\theta\}\_\{t\}from the computation ofG~t\\widetilde\{G\}\_\{t\}, as it improves performance by making the iterates converge closer toθ∗\\theta^\{\*\}than toθ∗​\(η2\)\\theta^\{\*\}\(\\eta^\{2\}\)\. This does not affect the privacy analysis, so we omit this term in the empirical evaluation\. Whenever we used the Gaussian mechanism, we computed the noise level using the analytic, tight formula fromBalle and Wang \[[2018](https://arxiv.org/html/2605.30600#bib.bib7)\]\.

Our synthetic datasets aim to address two edge dataset settings\. The first case aims to examine the performance of our algorithm in a setting whereλminX≫1\\lambda^\{X\}\_\{\\min\}\\gg 1, and under which, as predicted by Theorem[2](https://arxiv.org/html/2605.30600#Thmthm2), the[FastIHM](https://arxiv.org/html/2605.30600#alg2)should perform similarly to the IHM\. To that end, we generate covariates that are distributed i\.i\.d\. and uniformly on thedd\-dimensional sphere\. In that case, we note that

X⊤​X=∑i=1nxi​xi⊤≈nd​𝕀dX^\{\\top\}X=\\sum^\{n\}\_\{i=1\}x\_\{i\}x^\{\\top\}\_\{i\}\\approx\\frac\{n\}\{d\}\\mathbb\{I\}\_\{d\}\(242\)soλminX≈nd\\lambda^\{X\}\_\{\\min\}\\approx\\frac\{n\}\{d\}, which corresponds to its maximal attainable value since for any matrixXXthe next bounds holdd⋅λminX≤Tr​\(X⊤​X\)≤nd\\cdot\\lambda^\{X\}\_\{\\min\}\\leq\\mathrm\{Tr\}\(X^\{\\top\}X\)\\leq n\. Then, we generate responses according to the linear model

yi=xi⊤​θ0\+σ​𝗓iy\_\{i\}=x^\{\\top\}\_\{i\}\\theta\_\{0\}\+\\sigma\\mathsf\{z\}\_\{i\}\(243\)whereθ0\\theta\_\{0\}is a randomly generated unit vector,𝗓i​∼iid​N​\(0,1\)\\mathsf\{z\}\_\{i\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\pazocal\{N\}\(0,1\)andσ2=0\.1\\sigma^\{2\}=0\.1\. The second dataset aims to test the performance of the algorithm in a poorly conditioned setting \(termed*synthetic correlated*\), where usually the computational advantage of the IHS becomes prominent \(see, for example, experiments done inPilanci and Wainwright \[[2016](https://arxiv.org/html/2605.30600#bib.bib133)\], Lacotteet al\.\[[2020](https://arxiv.org/html/2605.30600#bib.bib152)\]\)\. To that end, we generate a design matrix built from vectors sampled as𝗑i​∼iid​N​\(0→d,Σ\)\{\\mathsf\{x\}\}\_\{i\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\pazocal\{N\}\(\\vec\{0\}\_\{d\},\\Sigma\)and whereΣi​j=2⋅\(0\.99\)\|i−j\|\\Sigma\_\{ij\}=2\\cdot\(0\.99\)^\{\\left\|i\-j\\right\|\}and responses from the same linear model \([243](https://arxiv.org/html/2605.30600#A8.E243)\)\. For both cases, we have fixedn=219n=2^\{19\}andd=25d=2^\{5\}\.

## Appendix\. IAdditional Experiments

We now provide additional experiments for a setting where the ration/d\\nicefrac\{\{n\}\}\{\{d\}\}is extremely large, representing a setting where sketching is most beneficial in terms of runtime\. In particular, we use the airline dataset\[[3](https://arxiv.org/html/2605.30600#bib.bib28)\], subsampled ton=222n=2^\{22\}samples, leads ton/d=222/4=220\\nicefrac\{\{n\}\}\{\{d\}\}=\\nicefrac\{\{2^\{22\}\}\}\{\{4\}\}=2^\{20\}\. We sweep across values ofk2max⁡\{d,log⁡\(4​T/ϱ\)\}∈\[4,1200\]\\frac\{k\_\{2\}\}\{\\max\\left\\\{d,\\log\(\\nicefrac\{\{4T\}\}\{\{\\varrho\}\}\)\\right\\\}\}\\in\[4,1200\]and fixT=4,k1=6⋅max⁡\{d,log⁡\(4​T/ϱ\)\}T=4,k\_\{1\}=6\\cdot\\max\\left\\\{d,\\log\(\\nicefrac\{\{4T\}\}\{\{\\varrho\}\}\)\\right\\\}andk=6⋅max⁡\{d,log⁡\(4​T/ϱ\)\}k=6\\cdot\\max\\left\\\{d,\\log\(\\nicefrac\{\{4T\}\}\{\{\\varrho\}\}\)\\right\\\}\. As seen in Figure[4](https://arxiv.org/html/2605.30600#A9.F4), in this regime,[FastIHM](https://arxiv.org/html/2605.30600#alg2)allows for attaining close\-to\-optimal performance at a significantly lower runtime, demonstrating the most favorable regime for sketching\.

λmaxX≈356246,λminX≈0\.685\\lambda^\{X\}\_\{\\max\}\\approx 356246,\\ \\lambda^\{X\}\_\{\\min\}\\approx 0\.685![Refer to caption](https://arxiv.org/html/2605.30600v1/x9.png)

\+\\boldsymbol\{\+\}FastIHM:k2=4⋅max⁡\{d,4​Tϱ\}\\mathrm\{FastIHM\}\\hskip\-1\.7pt:\\hskip\-1\.7ptk\_\{2\}=4\\cdot\\max\\\{d,\\hskip\-0\.85005pt\\frac\{4T\}\{\\varrho\}\\hskip\-0\.85005pt\\\}×\\boldsymbol\{\\times\}FastIHM:k2=600⋅max⁡\{d,4​Tϱ\}\\mathrm\{FastIHM\}\\hskip\-1\.7pt:\\hskip\-1\.7ptk\_\{2\}=600\\cdot\\max\\\{d,\\hskip\-0\.85005pt\\frac\{4T\}\{\\varrho\}\\hskip\-0\.85005pt\\\}▲\\blacktriangleFastIHM:k2=1200⋅max⁡\{d,4​Tϱ\}\\mathrm\{FastIHM\}\\hskip\-1\.7pt:\\hskip\-1\.7ptk\_\{2\}=1200\\cdot\\max\\\{d,\\hskip\-0\.85005pt\\frac\{4T\}\{\\varrho\}\\hskip\-0\.85005pt\\\}■\\blacksquareIHM\\ \\mathrm\{IHM\}FastIHS:k=6⋅max⁡\{d,4​Tϱ\}\\ \\mathrm\{FastIHS\}:k=6\\cdot\\max\\\{d,\\frac\{4T\}\{\\varrho\}\\\}
\(r1,r2,r3\)\(r\_\{1\},r\_\{2\},r\_\{3\}\)Fast IHSFast IHM:r1r\_\{1\}Fast IHM:r2r\_\{2\}Fast IHM:r3r\_\{3\}Airline\(4,600,1200\)\(4,600,1200\)41\.9541\.9516\.7116\.7115\.2615\.2614\.1814\.18

Figure 4:Accuracy\-runtime trade\-off for the Airline dataset\.
## Appendix\. JDataset Statistics

The next table summarizes key statistics about the different datasets used for conducting the simulations in Section[4\.1](https://arxiv.org/html/2605.30600#S4.SS1)\.

DatasetnnddλminX\\lambda^\{X\}\_\{\\min\}λmaxX\\lambda^\{X\}\_\{\\max\}1n​‖Y−X​θ∗‖2\\frac\{1\}\{n\}\\hskip\-1\.7pt\\left\\\|Y\\hskip\-1\.7pt\-\\hskip\-1\.7ptX\\theta^\{\*\}\\right\\\|^\{2\}1n​‖Y‖2\\frac\{1\}\{n\}\\hskip\-1\.7pt\\left\\\|Y\\right\\\|^\{2\}nd\\frac\{n\}\{d\}Beijing\[ruiqurm,[2018](https://arxiv.org/html/2605.30600#bib.bib84)\]163849<10−3<10^\{\-3\}4435\.300\.00060\.0151310\.7YearMSD\[Bertin\-Mahieux,[2011](https://arxiv.org/html/2605.30600#bib.bib30)\]13107290<10−3<10^\{\-3\}464\.780\.0040\.9881456\.3Black Friday\[[8](https://arxiv.org/html/2605.30600#bib.bib29)\]13107281<10−3<10^\{\-3\}40022\.540\.0220\.2811618\.2Synthetic:λminX≈nd\\lambda^\{X\}\_\{\\min\}\\approx\\frac\{n\}\{d\}5242883216166\.4216637\.760\.0340\.04516384Synthetic: Correlated524288323\.2718723\.400\.00060\.03916384Rossman\[[34](https://arxiv.org/html/2605.30600#bib.bib27)\]52428828<10−3<10^\{\-3\}8390\.500\.02490\.26518724\.5Airline\[[3](https://arxiv.org/html/2605.30600#bib.bib28)\]419430440\.685356246\.30\.0050\.0061048576

Table 1:Key parameters from the datasets simulated in this work\.