Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

# Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations
Source: [https://arxiv.org/html/2606.05599](https://arxiv.org/html/2606.05599)
00footnotetext:*MSC2020 Subject Classifications:*62G08, 62G35\.00footnotetext:*Keywords and phrases:*Approximation theory, Huber regression, Nonparametric regression, Residual networks \(ResNets\), Quantile regression\.\(Department of Statistics, The Pennsylvania State University\)

###### Abstract

This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network \(DNN\) estimators\. While standard ReLU networks achieve minimax\-optimal rates in theL2​\(P\)L^\{2\}\(P\)norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least\-squares ReLU estimators can suffer from thecurse of dimensionalityin their uniform convergence behavior\. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst\-case reliability, we address this limitation by analyzing smoothly activated DNNs \(smooth DNNs\), encompassing both feedforward and residual structures\. We establish novel pseudo\-dimension bounds, non\-asymptotic approximation guarantees, and Hölder\-norm bounds for the approximators of these models\. Leveraging these results, we derive non\-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least\-squares, quantile, and logistic regression\. We prove that smooth DNNs can mitigate the curse of dimensionality in uniform convergence by adaptively exploiting the low\-dimensional hierarchical composition structure of the target function\. Supported by both simulation studies and a real\-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees\.

## 1Introduction

Feedforward neural networks \(FNNs\) with rectified linear unit \(ReLU\) activations have received much attention in nonparametric regression due to their ability to efficiently approximate functions with latent low\-dimensional structures\[[4](https://arxiv.org/html/2606.05599#bib.bib13),[37](https://arxiv.org/html/2606.05599#bib.bib8),[26](https://arxiv.org/html/2606.05599#bib.bib11)\]\. This structural flexibility has facilitated their applications to a broad range of statistical problems, including robust regression\[[12](https://arxiv.org/html/2606.05599#bib.bib7),[9](https://arxiv.org/html/2606.05599#bib.bib30)\], survival analysis\[[51](https://arxiv.org/html/2606.05599#bib.bib16)\], and factor and interaction models\[[13](https://arxiv.org/html/2606.05599#bib.bib17),[6](https://arxiv.org/html/2606.05599#bib.bib18)\], among others\.

While ReLU FNN estimators achieve minimax\-optimal convergence rates in theL2​\(P\)L^\{2\}\(P\)norm\[[12](https://arxiv.org/html/2606.05599#bib.bib7)\], their uniform convergence properties remain less understood\. Yet, uniform convergence is a fundamental prerequisite for many downstream statistical theoretical analyses and decision\-making applications, such as ensuring reliable individualized treatment recommendations\[[16](https://arxiv.org/html/2606.05599#bib.bib20)\], constructing valid confidence bands\[[47](https://arxiv.org/html/2606.05599#bib.bib19)\], and enabling transfer learning\[[36](https://arxiv.org/html/2606.05599#bib.bib21)\]\. Current theoretical guarantees for the uniform convergence of ReLU FNNs are notably limited: they are often restricted to shallow ReLU network estimators with univariate covariates\[[36](https://arxiv.org/html/2606.05599#bib.bib21)\]or rely on the availability of a uniformly consistent pilot estimator\[[21](https://arxiv.org/html/2606.05599#bib.bib46)\]\. This critical theoretical gap raises significant concerns regarding the reliability of ReLU FNN estimators in contexts where uniform guarantees are essential, which restricts their practical applicability\.

To investigate this open problem, we first characterize a fundamental bottleneck in the uniform convergence of ReLU FNNs\. Specifically, we establish the first theoretical lower bound in the literature demonstrating that least\-squares ReLU FNN estimators inherently suffer from thecurse of dimensionalityin their uniform convergence rates, even when they simultaneously achieve the minimax\-optimal convergence rate in theL2​\(P\)L^\{2\}\(P\)norm\. This underscores that the optimalL2​\(P\)L^\{2\}\(P\)convergence of ReLU FNNs does not translate to the uniform reliability required for downstream statistical theoretical analyses\.

Beyond these theoretical limitations, the focus on ReLU FNNs increasingly diverges from modern deep learning practice\. The piecewise linear ReLU activation has largely been replaced byC∞C^\{\\infty\}–smooth alternatives, such as the sigmoid linear unit \(SiLU\)\[[35](https://arxiv.org/html/2606.05599#bib.bib24),[11](https://arxiv.org/html/2606.05599#bib.bib25)\], and the Gaussian error linear unit \(GELU\)\[[20](https://arxiv.org/html/2606.05599#bib.bib26)\], driven by their superior empirical performance\. Furthermore, residual architectures \(ResNets\)\[[18](https://arxiv.org/html/2606.05599#bib.bib23),[19](https://arxiv.org/html/2606.05599#bib.bib31)\]have fundamentally reshaped neural network design\. By resolving the vanishing gradient and degradation problems inherent in training very deep networks, residual connections have become the indispensable backbone of modern deep learning\. Although the statistical theory underlying these components remains limited, the integration of residual architectures withC∞C^\{\\infty\}\-smooth activations forms the structural basis of current powerful foundation models, from Vision Transformers\[[10](https://arxiv.org/html/2606.05599#bib.bib29)\]to state\-of\-the\-art large language models like LLaMA\[[44](https://arxiv.org/html/2606.05599#bib.bib28)\]and DeepSeek\[[45](https://arxiv.org/html/2606.05599#bib.bib38)\]\.

To overcome this bottleneck, motivated by recent architectural developments in deep learning, we focus on deep neural networks with smooth activations \(smooth DNNs\), encompassing both feedforward and residual architectures, and develop a rigorous theoretical framework for their uniform convergence analysis\. Through a comprehensive theoretical analysis together with supporting simulation studies, we show that smooth DNNs can substantially mitigate thecurse of dimensionalityin uniform convergence and provide strictly stronger uniform guarantees than ReLU FNNs\. These results position smooth DNNs as a principled and theoretically grounded alternative for statistical learning tasks in which uniform convergence is essential\.

The primary methodological and theoretical contributions of this work are threefold\. Together, they bridge a critical gap between the empirical success of smooth DNNs and the current lack of a general statistical theory for their uniform convergence:

- •Characterizing the curse of dimensionality for ReLU FNNs in the uniform convergence\. We establish the first theoretical lower bound demonstrating that ReLU FNNs inherently suffer from thecurse of dimensionalityin the uniform norm\. While achieving minimax\-optimalL2​\(P\)L^\{2\}\(P\)convergence\[[12](https://arxiv.org/html/2606.05599#bib.bib7)\], Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)reveals that the uniform convergence rate of the least\-squares ReLU FNN estimator can be bounded below byn−1d\+1n^\{\-\\frac\{1\}\{d\+1\}\}and above byn−1d\+2n^\{\-\\frac\{1\}\{d\+2\}\}, wherennis the sample size andddis the feature dimension, regardless of the Hölder smoothness of the target function\. Furthermore, our analysis based on interpolation inequalities in Section[2\.2](https://arxiv.org/html/2606.05599#S2.SS2)suggests that this limitation stems fundamentally from the limited smoothness of the ReLU activation, which prevents ReLU FNNs from exploiting higher\-order regularity\.
- •A foundational theoretical framework for smooth DNNs\.The statistical theory for deep neural networks, particularly ResNets, with smooth activations, remains largely absent\. Existing literature predominantly focuses on either FNNs with less common activations, such as sigmoid, tanh, or rectified power unit \(RePU\) activations\[[8](https://arxiv.org/html/2606.05599#bib.bib35),[5](https://arxiv.org/html/2606.05599#bib.bib34),[41](https://arxiv.org/html/2606.05599#bib.bib33)\]or residual architectures with ReLU activations\[[34](https://arxiv.org/html/2606.05599#bib.bib48),[29](https://arxiv.org/html/2606.05599#bib.bib49),[28](https://arxiv.org/html/2606.05599#bib.bib50)\]\. To enable the uniform convergence analysis of smooth DNNs, Section[4](https://arxiv.org/html/2606.05599#S4)develops a comprehensive set of theoretical tools\. Specifically, we establish an upper bound on the pseudo\-dimension in Theorem[4\.4](https://arxiv.org/html/2606.05599#S4.Thmtheorem4)and derive approximation error bounds for both Sobolev functions and hierarchical composition models in Theorems[4\.6](https://arxiv.org/html/2606.05599#S4.Thmtheorem6)and[4\.8](https://arxiv.org/html/2606.05599#S4.Thmtheorem8), respectively\. To our knowledge, these results provide the first theoretical foundation for establishing uniform convergence guarantees for smooth DNN estimators\.
- •Uniform convergence and robustness guarantees\.Building on our theoretical framework, we establish uniform convergence guarantees for smooth DNN estimators in Huber, least\-squares, quantile, and logistic regression\. In particular, Theorem[5\.4](https://arxiv.org/html/2606.05599#S5.Thmtheorem4)shows that the smooth DNN Huber estimator is non\-asymptotically robust in the uniform norm, extending existing robustness results for ReLU FNN Huber estimators in theL2​\(P\)L^\{2\}\(P\)norm\[[12](https://arxiv.org/html/2606.05599#bib.bib7),[9](https://arxiv.org/html/2606.05599#bib.bib30)\]\. Theorem[5\.5](https://arxiv.org/html/2606.05599#S5.Thmtheorem5)then gives the corresponding uniform convergence rate for the smooth DNN least\-squares estimator as a special case of Huber regression, and shows that it can overcome thecurse of dimensionalityexhibited by ReLU least\-squares regression in Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)\. Analogous uniform convergence and robustness guarantees for quantile and logistic regression are established in Theorems[5\.7](https://arxiv.org/html/2606.05599#S5.Thmtheorem7)and[5\.8](https://arxiv.org/html/2606.05599#S5.Thmtheorem8), respectively\. Notably, the uniform convergence guarantee for logistic regression also provides rigorous theoretical support for downstream applications such as probability estimation for double Higgs boson production\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]; see Theorem[5\.10](https://arxiv.org/html/2606.05599#S5.Thmtheorem10)\. More broadly, across all these tasks, we prove that smooth DNN estimators can adapt to the low\-dimensional hierarchical composition structure of the target function, thereby enjoying a clear theoretical advantage over ReLU FNNs in uniform convergence\.

To support our methodological and theoretical results, Section[6](https://arxiv.org/html/2606.05599#S6)benchmarks the numericalL2​\(P\)L^\{2\}\(P\)and uniform estimation errors of smooth DNN estimators against those of ReLU FNN estimators in Huber regression\. The substantial performance gains achieved by smooth DNN estimators provide empirical support for our methods and theory\. Taken together, these results support smooth DNN estimators as effective alternatives to ReLU FNNs in practical applications where uniform convergence is required\.

The rest of this paper is organized as follows\. Section[2](https://arxiv.org/html/2606.05599#S2)establishes the lower and upper uniform convergence rates of the least squares ReLU FNN estimators\. Section[3](https://arxiv.org/html/2606.05599#S3)presents the new methodology for smooth DNNs\. In Section[4](https://arxiv.org/html/2606.05599#S4), we establish the key statistical properties of smooth DNNs\. Building on these foundational elements, in Section[5](https://arxiv.org/html/2606.05599#S5), we derive uniform convergence rates of smooth DNN estimators for Huber, least\-squares, quantile, and logistic regression\. In Section[6](https://arxiv.org/html/2606.05599#S6), we present simulation studies and a real application\. Section[7](https://arxiv.org/html/2606.05599#S7)includes a few concluding remarks\. The complete proofs and additional technical or numerical results are presented in the Supplementary Materials\.

Before proceeding, we introduce the following notation and terminology that will be used throughout this paper\. We use the notationa≲ba\\lesssim bto mean there is a constantC\>0C\>0independent ofaaandbb, such thata≤C​ba\\leq Cb\. We saya≍ba\\asymp bif botha≲ba\\lesssim bandb≲ab\\lesssim ahold\. We saya≲log⁡nba\\lesssim\_\{\\log n\}bif there exists a functionC​\(log⁡n\)C\(\\log n\)such thata≤C​\(log⁡n\)⋅ba\\leq C\(\\log n\)\\cdot b\. We may further use the notation≲log\\lesssim\_\{\\log\}to suppress any other logarithmic terms\.a∨b=max⁡\(a,b\)a\\lor b=\\max\(a,b\)anda∧b=min⁡\(a,b\)a\\land b=\\min\(a,b\)\. Forn∈ℕ\+n\\in\\mathbb\{N\}\_\{\+\}, denote\[n\]=\{1,2,⋯,n\}\[n\]=\\\{1,2,\\cdots,n\\\}\. Forx∈ℝx\\in\\mathbb\{R\}, denote⌈x⌉=min⁡\{n∈ℕ:n≥x\}\\lceil x\\rceil=\\min\\\{n\\in\\mathbb\{N\}:n\\geq x\\\}\. Forx∈ℝdx\\in\\mathbb\{R\}^\{d\}, denote itsℓ2\\ell\_\{2\}norm as∥⋅∥2\\\|\\cdot\\\|\_\{2\}andℓ∞\\ell\_\{\\infty\}norm as∥⋅∥∞\\\|\\cdot\\\|\_\{\\infty\}\. For anyM\>0M\>0, letL∞​\(M\)L^\{\\infty\}\(M\)denote the collection of real\-valued functions uniformly bounded byMM\. ForΩ⊂ℝd\\Omega\\subset\\mathbb\{R\}^\{d\}, we denote the uniform norm of a functionffonΩ\\Omegaby‖f‖L∞​\(Ω\)\\\|f\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\. For𝐱=\(x1,⋯,xd\)∈ℝd\\mathbf\{x\}=\(x\_\{1\},\\cdots,x\_\{d\}\)\\in\\mathbb\{R\}^\{d\}and𝜶=\(α1,⋯,αd\)∈ℕd\{\\bm\{\\alpha\}\}=\(\\alpha\_\{1\},\\cdots,\\alpha\_\{d\}\)\\in\\mathbb\{N\}^\{d\}, we denote𝐱𝜶=x1α1​x2α2​⋯​xdαd\\mathbf\{x\}^\{\\bm\{\\alpha\}\}=x\_\{1\}^\{\\alpha\_\{1\}\}x\_\{2\}^\{\\alpha\_\{2\}\}\\cdots x\_\{d\}^\{\\alpha\_\{d\}\}, and\|𝜶\|=\|α1\|\+\|α2\|\+⋯\+\|αd\|\|\{\\bm\{\\alpha\}\}\|=\|\\alpha\_\{1\}\|\+\|\\alpha\_\{2\}\|\+\\cdots\+\|\\alpha\_\{d\}\|\. LetDi\\mathrm\{D\}\_\{i\}be the derivative operator to theii\-th variable fori∈\[d\]i\\in\[d\], the multi\-index derivative of order𝜶∈ℕd\{\\bm\{\\alpha\}\}\\in\\mathbb\{N\}^\{d\}is defined asD𝜶:=D1α1​⋯​Ddαd\\mathrm\{D\}^\{\\bm\{\\alpha\}\}:=\\mathrm\{D\}\_\{1\}^\{\\alpha\_\{1\}\}\\cdots\\mathrm\{D\}\_\{d\}^\{\\alpha\_\{d\}\}\. Letβ=r\+s\\beta=r\+sfor some non\-negative integerrrand0<s≤10<s\\leq 1,d∈ℕ\+d\\in\\mathbb\{N\}\_\{\+\}, andC\>0C\>0\. Let∥⋅∥Cβ​\(Ω\)\\\|\\cdot\\\|\_\{C^\{\\beta\}\(\\Omega\)\}denote the Hölder norm of orderβ\\beta, and add\-variate functionffis called\(β,C\)\(\\beta,C\)\-smooth onΩ⊆ℝd\\Omega\\subseteq\\mathbb\{R\}^\{d\}, if

‖f‖Cβ​\(Ω\):=max0≤\|𝜶\|≤r⁡‖D𝜶​f‖L∞​\(Ω\)∨max𝜶:\|𝜶\|=r​sup𝐱,𝐲∈Ω,𝐱≠𝐲\|D𝜶​f​\(𝐱\)−D𝜶​f​\(𝐲\)\|‖𝐱−𝐲‖2s≤C\.\\\|f\\\|\_\{C^\{\\beta\}\(\\Omega\)\}:=\\max\_\{0\\leq\|\{\\bm\{\\alpha\}\}\|\\leq r\}\\\|\\mathrm\{D\}^\{\\bm\{\\alpha\}\}f\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\lor\\max\_\{\{\\bm\{\\alpha\}\}:\|\{\\bm\{\\alpha\}\}\|=r\}\\sup\_\{\\mathbf\{x\},\\mathbf\{y\}\\in\\Omega,\\mathbf\{x\}\\neq\\mathbf\{y\}\}\\frac\{\|\\mathrm\{D\}^\{\\bm\{\\alpha\}\}f\(\\mathbf\{x\}\)\-\\mathrm\{D\}^\{\\bm\{\\alpha\}\}f\(\\mathbf\{y\}\)\|\}\{\\\|\\mathbf\{x\}\-\\mathbf\{y\}\\\|\_\{2\}^\{s\}\}\\leq C\.

## 2Uniform Convergence of ReLU FNN Estimators

In this section, we investigate the fundamental limitations of ReLU FNN estimators in achieving uniform convergence\. Section[2\.1](https://arxiv.org/html/2606.05599#S2.SS1)establishes a theoretical lower bound on the uniform convergence rates of least\-squares ReLU estimators in Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1), demonstrating that they inherently suffer from the*curse of dimensionality*, even when theirL2​\(P\)L^\{2\}\(P\)convergence rate is minimax\-optimal\. To explain the mechanism behind this bottleneck, Section[2\.2](https://arxiv.org/html/2606.05599#S2.SS2)reveals how the limited smoothness of ReLU FNNs leads to slow uniform convergence rates\.

### 2\.1A Lower Bound on Uniform Convergence Rates

To establish that the*curse of dimensionality*in uniform convergence is an inherent limitation of ReLU FNN estimators, we focus on the foundational framework of nonparametric least\-squares regression\. Let\{\(Xi,Yi\)\}i=1n\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}be the*i\.i\.d\.*observations generated from

Yi=f0​\(Xi\)\+ξi,Y\_\{i\}=f\_\{0\}\(X\_\{i\}\)\+\\xi\_\{i\},\(1\)wheref0f\_\{0\}is the unknown regression function,ξi\\xi\_\{i\}denotes the random noise, andPPrepresents the distribution of\(Xi,Yi\)\(X\_\{i\},Y\_\{i\}\)\.

To estimatef0f\_\{0\}, we consider empirical risk minimization over a class of ReLU FNNs\. Specifically, given a network widthW∈ℕ\+W\\in\\mathbb\{N\}\_\{\+\}and depthD∈ℕ\+D\\in\\mathbb\{N\}\_\{\+\}, a ReLU FNN is defined as a functionf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}of the composition form

f​\(x\)=ℒD\+1∘σ∗∘ℒD∘σ∗∘⋯∘ℒ2∘σ∗∘ℒ1​\(x\),f\(x\)=\\mathcal\{L\}\_\{D\+1\}\\circ\\sigma\_\{\\ast\}\\circ\\mathcal\{L\}\_\{D\}\\circ\\sigma\_\{\\ast\}\\circ\\cdots\\circ\\mathcal\{L\}\_\{2\}\\circ\\sigma\_\{\\ast\}\\circ\\mathcal\{L\}\_\{1\}\(x\),\(2\)whereσ∗​\(x\)=x∨0\\sigma\_\{\\ast\}\(x\)=x\\lor 0is the ReLU activation function applied entrywise, andℒi:ℝdi−1→ℝdi\\mathcal\{L\}\_\{i\}:\\mathbb\{R\}^\{d\_\{i\-1\}\}\\to\\mathbb\{R\}^\{d\_\{i\}\}are affine transformation maps with the dimension vector\(d0,…,dD\+1\)=\(d,W,…,W,1\)\(d\_\{0\},\\ldots,d\_\{D\+1\}\)=\(d,W,\\ldots,W,1\)\.

To rule out pathological estimators that interpolate the noise through increasingly steep local oscillations, we restrict attention to uniformly bounded ReLU networks with uniformly bounded parameters\. Accordingly, for somer≥2r\\geq 2andM≥0M\\geq 0, we define the candidate class by

ℱReLU\\displaystyle\\mathcal\{F\}\_\{\\mathrm\{ReLU\}\}:=\{f​of the form \([2](https://arxiv.org/html/2606.05599#S2.E2)\):​maxi∈\[D\+1\]⁡‖Ai‖∞∨‖bi∥∞≤r\}∩L∞​\(M\)\.\\displaystyle=\\bigl\\\{f\\text\{ of the form~\\eqref\{eq: definition of feedforward network\}: \}\\max\_\{i\\in\[D\+1\]\}\\\|A\_\{i\}\\\|\_\{\\infty\}\\lor\\\|b\_\{i\}\\\|\_\{\\infty\}\\leq r\\bigr\\\}\\cap L^\{\\infty\}\(M\)\.Let the corresponding least\-squares ReLU estimatorf^n\\hat\{f\}\_\{n\}be defined by

f^n∈arg⁡minf∈ℱReLU∑i=1n\(f​\(Xi\)−Yi\)2,\\hat\{f\}\_\{n\}\\in\\mathop\{\\arg\\min\}\_\{f\\in\\mathcal\{F\}\_\{\\mathrm\{ReLU\}\}\}\\sum\_\{i=1\}^\{n\}\\bigl\(f\(X\_\{i\}\)\-Y\_\{i\}\\bigr\)^\{2\},\(3\)We further require that the estimator sequence is uniformly Lipschitz, in the sense thatsupn≥1‖f^n‖Lip<∞\\sup\_\{n\\geq 1\}\\\|\\hat\{f\}\_\{n\}\\\|\_\{\\text\{Lip\}\}<\\infty\. These restrictions are imposed to exclude the possibility that poor uniform convergence arises from pathological overfitting to the noise\.

Meanwhile, to isolate the architectural limitation of ReLU FNNs from pathological distributions, we also impose standard regularity conditions on the data\-generating process\. Specifically, it is assumed in Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)below that the covariatesXiX\_\{i\}are uniformly distributed on\[0,1\]d\[0,1\]^\{d\}, the noisesξi\\xi\_\{i\}are independent ofXiX\_\{i\}and uniformly bounded, and the target functionf0f\_\{0\}is Hölder\-smooth and uniformly bounded\. These assumptions describe a highly regular nonparametric estimation setting: the covariate distribution is benign, the regression function is smooth, and the noise is bounded\. Therefore, any slow uniform convergence rate established below cannot be attributed to irregular features of the data\-generating process\.

Taken together, the restrictions on the estimator and on the data\-generating distribution ensure that the lower bound in Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)reflects an intrinsic limitation of the ReLU architecture itself\. In particular, the slow uniform convergence rate is not caused by heavy\-tailed noise, pathological covariate design, lack of smoothness off0f\_\{0\}, or overfitting to the observations, but rather by the intrinsic limitations of ReLU FNN architectures\. Under these conditions, we obtain the following lower bound on the uniform convergence rate off^n\\hat\{f\}\_\{n\}\.

###### Theorem 2\.1\.

Let𝒫\\mathcal\{P\}denote the collection of distributions\(Xi,Yi\)∼P\(X\_\{i\},Y\_\{i\}\)\\sim Psuch that, in model \([1](https://arxiv.org/html/2606.05599#S2.E1)\), the covariatesXiX\_\{i\}follow uniform distribution on\[0,1\]d\[0,1\]^\{d\}, the noise satisfies\|ξi\|≤1\|\\xi\_\{i\}\|\\leq 1and𝔼​\[ξi\]=0\\mathbb\{E\}\[\\xi\_\{i\}\]=0, and the regression function satisfiesf0∈Cα​\(\[0,1\]d\)∩L∞​\(\[0,1\]d\)f\_\{0\}\\in C^\{\\alpha\}\(\[0,1\]^\{d\}\)\\cap L^\{\\infty\}\(\[0,1\]^\{d\}\)for someα≥2\\alpha\\geq 2\. Suppose the ReLU FNN with widthWWand depthDDsatisfies,n34​\(d\+1\)≲W≲D≲2n316​\(d\+1\)n^\{\\frac\{3\}\{4\(d\+1\)\}\}\\lesssim W\\lesssim D\\lesssim 2^\{\\,n^\{\\frac\{3\}\{16\(d\+1\)\}\}\}up to poly\-logarithmic factors\. Then for anyd≥2d\\geq 2,

supP∈𝒫𝔼​‖f^n−f0‖L∞​\(\[0,1\]d\)≳n−1d\+1\.\\sup\_\{P\\in\\mathcal\{P\}\}\\ \\mathbb\{E\}\\bigl\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\bigr\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\ \\gtrsim\\ n^\{\-\\frac\{1\}\{d\+1\}\}\.

Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)explicitly formalizes the classical*curse of dimensionality*for the least\-squares ReLU estimatorf^n\\hat\{f\}\_\{n\}, since the exponent of its uniform convergence rate depends poorly on dimensiondd\. In contrast,\[[12](https://arxiv.org/html/2606.05599#bib.bib7)\]proved that theL2​\(P\)L^\{2\}\(P\)convergence rates of ReLU FNN estimators do not have this issue\. Specifically, under these conditions, theL2​\(P\)L^\{2\}\(P\)estimation error of ReLU estimatorf^n\\hat\{f\}\_\{n\}satisfies

supP∈𝒫𝔼​‖f^n−f0‖L2​\(P\)≲log\(W​D\)2n\+\(W​D\)−2​α/d\.\\sup\_\{P\\in\\mathcal\{P\}\}\\mathbb\{E\}\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{2\}\(P\)\}\\ \\lesssim\_\{\\log\}\\ \\sqrt\{\\frac\{\(WD\)^\{2\}\}\{n\}\}\\;\+\\;\(WD\)^\{\-2\\alpha/d\}\.In particular, provided the target functionf0f\_\{0\}is not excessively smooth \(i\.e\.,α≤d​\(d−2\)/6\\alpha\\leq d\(d\-2\)/6\), one can choose a network architecture withW≍D≍nd4​\(d\+2​α\)≥n34​\(d\+1\)W\\asymp D\\asymp n^\{\\frac\{d\}\{4\(d\+2\\alpha\)\}\}\\geq n^\{\\frac\{3\}\{4\(d\+1\)\}\}\. Under this configuration,f^n\\hat\{f\}\_\{n\}simultaneously attains the minimax\-optimalL2​\(P\)L^\{2\}\(P\)raten−α2​α\+dn^\{\-\\frac\{\\alpha\}\{2\\alpha\+d\}\}while remaining trapped at the slow, dimension\-dependent rate ofn−1d\+1n^\{\-\\frac\{1\}\{d\+1\}\}in the uniform norm\. This stark discrepancy highlights a critical intrinsic limitation that least\-squares ReLU FNN estimators can be suboptimal for downstream applications requiring uniform convergence guarantees\. We provide a more detailed discussion of Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)and its implications in Section S1 of the Supplementary Material\.

### 2\.2An Interpolation Perspective on Uniform Convergence Rates

In Section[2\.1](https://arxiv.org/html/2606.05599#S2.SS1), ReLU least squares estimators are shown to suffer from thecurse of dimensionalityunder the uniform norm\. In contrast, many classical nonparametric estimators, such as splines, wavelets, and Fourier series, typically avoid this issue\. A key reason is their linear structure: these estimators lie in the linear span of basis functions and, conditioned on the design matrix, behave analogously to ordinary least squares \(OLS\) estimators\.

To illustrate this structural contrast, consider observations\{\(Xi,Yi\)\}i=1n\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}generated from the parametric linear modelYi=Xi⊤​β0\+ξiY\_\{i\}=X\_\{i\}^\{\\top\}\\beta\_\{0\}\+\\xi\_\{i\}, whereXiX\_\{i\}is supported on\[0,1\]d\[0,1\]^\{d\}, and letβ^n\\hat\{\\beta\}\_\{n\}denote the OLS estimator\. Define the true and estimated regression functions byf0​\(x\)=x⊤​β0f\_\{0\}\(x\)=x^\{\\top\}\\beta\_\{0\}andf^n​\(x\)=x⊤​β^n\\hat\{f\}\_\{n\}\(x\)=x^\{\\top\}\\hat\{\\beta\}\_\{n\}, respectively\. Assuming the population second\-moment matrixΣ:=𝔼​\[X​X⊤\]\\Sigma:=\\mathbb\{E\}\[XX^\{\\top\}\]is positive definite with the smallest eigenvalueλmin​\(Σ\)\>0\\lambda\_\{\\min\}\(\\Sigma\)\>0, theL2​\(P\)L^\{2\}\(P\)andL∞L^\{\\infty\}estimation errors are deterministically linked\. Specifically, because‖x‖2≤d\\\|x\\\|\_\{2\}\\leq\\sqrt\{d\}, Cauchy–Schwarz implies‖f^n−f0‖L∞​\(\[0,1\]d\)≤d​‖β^n−β0‖2\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\leq\\sqrt\{d\}\\\|\\hat\{\\beta\}\_\{n\}\-\\beta\_\{0\}\\\|\_\{2\}\. Concurrently,‖f^n−f0‖L2​\(P\)2=\(β^n−β0\)⊤​Σ​\(β^n−β0\)≥λmin​\(Σ\)​‖β^n−β0‖22\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{2\}\(P\)\}^\{2\}=\(\\hat\{\\beta\}\_\{n\}\-\\beta\_\{0\}\)^\{\\top\}\\Sigma\(\\hat\{\\beta\}\_\{n\}\-\\beta\_\{0\}\)\\geq\\lambda\_\{\\min\}\(\\Sigma\)\\\|\\hat\{\\beta\}\_\{n\}\-\\beta\_\{0\}\\\|\_\{2\}^\{2\}\. Combining these relations yields the inequality that implies that theL∞L^\{\\infty\}andL2​\(P\)L^\{2\}\(P\)estimation errors are equivalent for linear models:

λmin​\(Σ\)d​‖f^n−f0‖L∞​\(\[0,1\]d\)≤‖f^n−f0‖L2​\(P\)≤‖f^n−f0‖L∞​\(\[0,1\]d\)\.\\sqrt\{\\frac\{\\lambda\_\{\\min\}\(\\Sigma\)\}\{d\}\}\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\leq\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{2\}\(P\)\}\\leq\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\.
Beyond parametric linear models, analogous interpolation inequalities serve as powerful classical tools to establish uniform convergence rates for broader function classes by exploiting underlying smoothness properties\. For example, the Gagliardo–Nirenberg inequality\[[14](https://arxiv.org/html/2606.05599#bib.bib2),[33](https://arxiv.org/html/2606.05599#bib.bib3)\]applies to Hölder functions, while the Brezis–Mironescu inequality\[[7](https://arxiv.org/html/2606.05599#bib.bib6)\]applies to \(fractional\) Sobolev functions\.

This viewpoint provides an analytic explanation for the slow uniform convergence rates observed in ReLU FNN estimators\. To see this, consider the standard Gagliardo–Nirenberg inequality on\[0,1\]d\[0,1\]^\{d\}: for any functionf∈Cα​\(\[0,1\]d\)f\\in C^\{\\alpha\}\(\[0,1\]^\{d\}\), the uniform norm is bounded by

‖f‖L∞​\(\[0,1\]d\)≲‖f‖Cα​\(\[0,1\]d\)d2​α\+d⋅‖f‖L2​\(P\)2​α2​α\+d\.\\\|f\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\lesssim\\\|f\\\|\_\{C^\{\\alpha\}\(\[0,1\]^\{d\}\)\}^\{\\frac\{d\}\{2\\alpha\+d\}\}\\cdot\\\|f\\\|\_\{L^\{2\}\(P\)\}^\{\\frac\{2\\alpha\}\{2\\alpha\+d\}\}\.\(4\)This interpolation bound shows that when the candidate function class possesses substantial smoothness \(α≫d\\alpha\\gg d\), theL∞L^\{\\infty\}convergence rate closely tracks theL2​\(P\)L^\{2\}\(P\)rate\. However, because ReLU FNNs are generally only piecewise affine and need not belong toC1C^\{1\}, this interpolation argument can use at most Hölder exponentsα∈\(0,1\)\\alpha\\in\(0,1\)\. Applying \([4](https://arxiv.org/html/2606.05599#S2.E4)\) together with the standardL2​\(P\)L^\{2\}\(P\)rate‖f^n−f0‖L2​\(P\)≲n−1/2\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{2\}\(P\)\}\\lesssim n^\{\-1/2\}, and then lettingα→1\\alpha\\to 1, yields

‖f^n−f0‖L∞​\(\[0,1\]d\)≲n−1/\(d\+2\)\.\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\lesssim n^\{\-1/\(d\+2\)\}\.When the Gagliardo–Nirenberg inequality is tight, this bound is sharp\. This issue shows that the poor intrinsic smoothness of the ReLU activation function fundamentally prevents its network architecture from adapting to fast uniform convergence requirements, rendering it ill\-suited for downstream tasks requiring strong uniform convergence guarantees\.

## 3Smoothly Activated Deep Neural Networks

This section introduces smoothly activated deep neural networks \(smooth DNNs\), which encompass both deep feedforward and deep residual architectures, as a powerful and theoretically rigorous alternative\. Section[3\.1](https://arxiv.org/html/2606.05599#S3.SS1)reviews smooth activation functions widely adopted in modern deep learning and then summarizes the foundational framework of residual networks\. Leveraging these components, Section[3\.2](https://arxiv.org/html/2606.05599#S3.SS2)presents the mathematical formulation and characterization of smooth DNNs\.

### 3\.1Preliminaries

We first characterize the infinitely differentiable \(i\.e\.,C∞C^\{\\infty\}\-smooth\) activation functions and the structural configurations of residual networks\.

Smooth Activation Functions\.Nonlinear activation functions govern the expressiveness of neural networks\. While the ReLU activation serves as a baseline for deep compositional mapping, its non\-differentiability at the origin constrains its functional regularity\. Non\-smooth variants such as Leaky ReLU, PReLU, ELU, and SELU introduce non\-zero components on the negative real line but remain non\-differentiable, making them analytically ill\-suited for establishing fast uniform convergence rates under the interpolation perspective described in Section[2\.2](https://arxiv.org/html/2606.05599#S2.SS2)\.

![Refer to caption](https://arxiv.org/html/2606.05599v1/x1.png)Figure 1:ReLU and its non\-C∞C^\{\\infty\}variants \(left\); and itsC∞C^\{\\infty\}variants \(right\)\.To achieve higher\-order functional smoothness, we consider a class ofC∞C^\{\\infty\}\-smooth activations, including SiLU \(also known as Swish\)\[[35](https://arxiv.org/html/2606.05599#bib.bib24),[11](https://arxiv.org/html/2606.05599#bib.bib25)\], GELU\[[20](https://arxiv.org/html/2606.05599#bib.bib26)\], and Mish\[[32](https://arxiv.org/html/2606.05599#bib.bib27)\]\. As illustrated in Figure[1](https://arxiv.org/html/2606.05599#S3.F1), these activations exhibit smooth transitions near the origin\. Formally, theseC∞C^\{\\infty\}\-smooth activation functions admits a factorization of the formσ​\(x\)=x⋅ψ​\(x\)\\sigma\(x\)=x\\cdot\\psi\(x\), whereψ:ℝ→ℝ\\psi:\\mathbb\{R\}\\to\\mathbb\{R\}is a non\-decreasing, uniformly bounded, and infinitely differentiable function\. The explicit functional forms are given by:

SiLU​\(x\)=\\displaystyle\\text\{SiLU\}\(x\)=x⋅exp⁡\(x\)1\+exp⁡\(x\);\\displaystyle x\\cdot\\frac\{\\exp\(x\)\}\{1\+\\exp\(x\)\};\(5\)GELU​\(x\)=\\displaystyle\\text\{GELU\}\(x\)=x⋅12​\(1\+tanh⁡\(2/π​\(x\+0\.044715​x3\)\)\);\\displaystyle x\\cdot\\frac\{1\}\{2\}\\Bigl\(1\+\\tanh\\bigl\(\\sqrt\{2/\\pi\}\\,\(x\+0\.044715\\,x^\{3\}\)\\bigr\)\\Bigr\);\(6\)Mish​\(x\)=\\displaystyle\\text\{Mish\}\(x\)=x⋅tanh⁡\(log⁡\(1\+exp⁡\(x\)\)\)\.\\displaystyle x\\cdot\\tanh\\\!\\bigl\(\\log\(1\+\\exp\(x\)\)\\bigr\)\.\(7\)Here we use the hyperbolic tangent version of the GELU activation, and its associated functionψ\\psiapproximates the distribution function of the standard normal distribution\. WritingReLU​\(x\)=x⋅H​\(x\)\\mathrm\{ReLU\}\(x\)=x\\cdot H\(x\)withH​\(x\)=𝕀​\(x≥0\)H\(x\)=\\mathbb\{I\}\(x\\geq 0\), the corresponding functionsψ\\psiof SiLU, GELU, and Mish converge toH​\(x\)H\(x\)exponentially fast asx→±∞x\\to\\pm\\infty\. As shown in Section[4\.2](https://arxiv.org/html/2606.05599#S4.SS2), this property ensures that the resulting network classes enjoy strong approximation properties for functions in Sobolev spaces and hierarchical composition models\.

Residual Networks\.In the context of nonparametric function estimation, ReLU FNNs have been shown to possess strong approximation capabilities relative to classical nonparametric models, such as wavelets\[[4](https://arxiv.org/html/2606.05599#bib.bib13),[37](https://arxiv.org/html/2606.05599#bib.bib8),[26](https://arxiv.org/html/2606.05599#bib.bib11)\]\. However, as compositional depth increases, standard deep feedforward architectures often exhibit ill\-conditioned optimization landscapes, which can lead to higher empirical risk even under careful parameter initialization and normalization strategies\[[18](https://arxiv.org/html/2606.05599#bib.bib23)\]\.

To regularize the optimization landscape,*residual networks*\(ResNets\) incorporate identity skip connections around each affine–activation block\[[18](https://arxiv.org/html/2606.05599#bib.bib23),[19](https://arxiv.org/html/2606.05599#bib.bib31)\]\. Rather than applying an isolated nonlinear transformation at each layer, the residual architecture models a functional perturbation of the preceding state\. Formally, given the output of theℓ\\ell\-th layerhℓ​\(x\)h\_\{\\ell\}\(x\), the\(ℓ\+1\)\(\\ell\+1\)\-th layer is defined ashℓ\+1​\(x\)=hℓ​\(x\)\+ℱℓ​\(hℓ​\(x\)\)h\_\{\\ell\+1\}\(x\)\\;=\\;h\_\{\\ell\}\(x\)\\;\+\\;\\mathcal\{F\}\_\{\\ell\}\\bigl\(h\_\{\\ell\}\(x\)\\bigr\), whereℱℓ\\mathcal\{F\}\_\{\\ell\}is typically implemented via a small stack of affine layers, normalizations, and nonlinear activations\.

![Refer to caption](https://arxiv.org/html/2606.05599v1/x2.png)Figure 2:Architectures of an FNN \(left\) and residual blocks withℒ1=id\\mathcal\{L\}\_\{1\}=\\text\{id\}\(right\)\.As illustrated in Figure[2](https://arxiv.org/html/2606.05599#S3.F2), the identity skip connection creates a direct pathwayhℓ→hℓ\+1h\_\{\\ell\}\\\!\\to h\_\{\\ell\+1\}for more stable information and gradient propagation across layers\. By mitigating the structural degradation of gradients, residual architectures make DNNs easier to train\. We will incorporate residual architectures into our theoretical framework as an optimization\-oriented generalization of feedforward architectures\.

### 3\.2Mathematical Formulation of Smooth DNNs

We now define the function classes of smooth DNNs under both feedforward and residual architectures\. To parameterize these architectures simultaneously, we first introduce an activation operator that unifies coordinatewise mappings and Gated Linear Unit \(GLU\) formulations\.

###### Definition 3\.1 \(Activation operator\)\.

Given aC∞C^\{\\infty\}\-smooth activation functionσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}, the activation operator𝒜σ:ℝq→ℝq~\\mathcal\{A\}\_\{\\sigma\}:\\mathbb\{R\}^\{q\}\\to\\mathbb\{R\}^\{\\tilde\{q\}\}is defined by one of the following mappings:

- •Standard activation:q=q~q=\\tilde\{q\}, and𝒜σ​\(z\)=σ​\(z\)\\mathcal\{A\}\_\{\\sigma\}\(z\)=\\sigma\(z\), whereσ​\(z\)\\sigma\(z\)is evaluated coordinatewise\.
- •GLU\-type activation:q=2​q~q=2\\tilde\{q\}, and forz=\(z1⊤,z2⊤\)⊤∈ℝqz=\(z\_\{1\}^\{\\top\},z\_\{2\}^\{\\top\}\)^\{\\top\}\\in\\mathbb\{R\}^\{q\}withz1,z2∈ℝq~z\_\{1\},z\_\{2\}\\in\\mathbb\{R\}^\{\\tilde\{q\}\},𝒜σ​\(z\)=σ​\(z1\)⊙z2,\\mathcal\{A\}\_\{\\sigma\}\(z\)=\\sigma\(z\_\{1\}\)\\odot z\_\{2\},whereσ​\(z1\)\\sigma\(z\_\{1\}\)is evaluated coordinatewise and⊙\\odotdenotes the Hadamard product\.

###### Definition 3\.2 \(Residual block\)\.

Given the input and output dimensionsd1,d2∈ℕ\+d\_\{1\},d\_\{2\}\\in\\mathbb\{N\}\_\{\+\}and widthW1,W2∈ℕ\+W\_\{1\},W\_\{2\}\\in\\mathbb\{N\}\_\{\+\}, letq=W2q=W\_\{2\}for standard activation, andq=2​W2q=2W\_\{2\}for GLU\-type activation\. Denote matrices𝐀1∈ℝW1×d1\\mathbf\{A\}\_\{1\}\\in\\mathbb\{R\}^\{W\_\{1\}\\times d\_\{1\}\},𝐀2∈ℝq×W1\\mathbf\{A\}\_\{2\}\\in\\mathbb\{R\}^\{q\\times W\_\{1\}\}, and𝐀3∈ℝd2×W2\\mathbf\{A\}\_\{3\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times W\_\{2\}\}, vectors𝐛1∈ℝW1\\mathbf\{b\}\_\{1\}\\in\\mathbb\{R\}^\{W\_\{1\}\},𝐛2∈ℝq\\mathbf\{b\}\_\{2\}\\in\\mathbb\{R\}^\{q\}, and𝐛3∈ℝd2\\mathbf\{b\}\_\{3\}\\in\\mathbb\{R\}^\{d\_\{2\}\}, and aC∞C^\{\\infty\}\-smooth activation functionσ\\sigma, a residual blockϕ\\phiwith parameters\(d1,d2,W1,W2\)\(d\_\{1\},d\_\{2\},W\_\{1\},W\_\{2\}\)is defined as

ϕ​\(x\)=\(id\+ℒ3∘𝒜σ∘ℒ2\)∘ℒ1​\(x\),x∈ℝd1,\\phi\(x\)=\(\\text\{id\}\+\\mathcal\{L\}\_\{3\}\\circ\\mathcal\{A\}\_\{\\sigma\}\\circ\\mathcal\{L\}\_\{2\}\)\\circ\\mathcal\{L\}\_\{1\}\(x\),\\qquad x\\in\\mathbb\{R\}^\{d\_\{1\}\},whereiddenotes the identity mapping andℒi​\(z\):=𝐀i​z\+𝐛i\\mathcal\{L\}\_\{i\}\(z\):=\\mathbf\{A\}\_\{i\}z\+\\mathbf\{b\}\_\{i\}fori=1,2,3i=1,2,3\.

Next, we define smooth DNNs for both residual and feedforward architectures\.

###### Definition 3\.3 \(Smooth DNN\)\.

A*smooth ResNet*of depthDDand widthWWis a functionf:ℝd1→ℝdD\+2f:\\mathbb\{R\}^\{d\_\{1\}\}\\to\\mathbb\{R\}^\{d\_\{D\+2\}\}of the form

f​\(x\)=ℒD\+1∘ϕD∘ϕD−1∘⋯∘ϕ1​\(x\),f\(x\)=\\mathcal\{L\}\_\{D\+1\}\\circ\\phi\_\{D\}\\circ\\phi\_\{D\-1\}\\circ\\cdots\\circ\\phi\_\{1\}\(x\),\(8\)where eachϕi:ℝdi→ℝdi\+1\\phi\_\{i\}:\\mathbb\{R\}^\{d\_\{i\}\}\\to\\mathbb\{R\}^\{d\_\{i\+1\}\}is a residual block with parameters\(di,di\+1,W1​i,W2​i\)\(d\_\{i\},d\_\{i\+1\},W\_\{1i\},W\_\{2i\}\)as in Definition[3\.2](https://arxiv.org/html/2606.05599#S3.Thmtheorem2), andℒD\+1:ℝdD\+1→ℝdD\+2\\mathcal\{L\}\_\{D\+1\}:\\mathbb\{R\}^\{d\_\{D\+1\}\}\\to\\mathbb\{R\}^\{d\_\{D\+2\}\}is affine\. The hidden dimensions of residual blocks are required to satisfy\(max2≤i≤D\+1⁡di\)∨\(max1≤i≤D⁡\(W1​i∨W2​i\)\)≤W\.\\Bigl\(\\max\_\{2\\leq i\\leq D\+1\}d\_\{i\}\\Bigr\)\\lor\\Bigl\(\\max\_\{1\\leq i\\leq D\}\(W\_\{1i\}\\lor W\_\{2i\}\)\\Bigr\)\\leq W\.

A*smooth FNN*of depthDDand widthWWis a functionf:ℝd1→ℝdD\+2f:\\mathbb\{R\}^\{d\_\{1\}\}\\to\\mathbb\{R\}^\{d\_\{D\+2\}\}of the form

f​\(x\)=ℒD\+1∘𝒜σ∘ℒD∘⋯∘𝒜σ∘ℒ1​\(x\),f\(x\)=\\mathcal\{L\}\_\{D\+1\}\\circ\\mathcal\{A\}\_\{\\sigma\}\\circ\\mathcal\{L\}\_\{D\}\\circ\\cdots\\circ\\mathcal\{A\}\_\{\\sigma\}\\circ\\mathcal\{L\}\_\{1\}\(x\),\(9\)whereℒi:ℝdi→ℝd~i\\mathcal\{L\}\_\{i\}:\\mathbb\{R\}^\{d\_\{i\}\}\\to\\mathbb\{R\}^\{\\tilde\{d\}\_\{i\}\}are affine maps, with input dimensions given by\(d1,d2,…,dD\+1\)=\(d1,W,…,W\)\(d\_\{1\},d\_\{2\},\\ldots,d\_\{D\+1\}\)=\(d\_\{1\},W,\\ldots,W\)\. The output dimensions are given by\(d~1,…,d~D,d~D\+1\)=\(W,…,W,dD\+2\)\(\\tilde\{d\}\_\{1\},\\ldots,\\tilde\{d\}\_\{D\},\\tilde\{d\}\_\{D\+1\}\)=\(W,\\ldots,W,d\_\{D\+2\}\)in the standard activation case, and\(2​W,…,2​W,dD\+2\)\(2W,\\ldots,2W,d\_\{D\+2\}\)in the GLU\-type case\.

For a fixed activation functionσ∈C∞​\(ℝ\)\\sigma\\in C^\{\\infty\}\(\\mathbb\{R\}\), the functional class of smooth DNNs with depthDD, widthWW, and mapping fromℝd1\\mathbb\{R\}^\{d\_\{1\}\}toℝdD\+2\\mathbb\{R\}^\{d\_\{D\+2\}\}is given by:

ℱ​\(d1,dD\+2,D,W,σ\)=\{x↦f​\(x\):f​is of the form \([8](https://arxiv.org/html/2606.05599#S3.E8)\) or \([9](https://arxiv.org/html/2606.05599#S3.E9)\)\}\.\\mathcal\{F\}\(d\_\{1\},d\_\{D\+2\},D,W,\\sigma\)=\\bigl\\\{x\\mapsto f\(x\):f\\text\{ is of the form \\eqref\{eq: definition of smooth rn\} or \\eqref\{eq: definition of smooth fnn\}\}\\bigr\\\}\.Unlike ReLU networks, the compositional structure of smooth DNNs ensures that anyf∈ℱ​\(d1,dD\+2,D,W,σ\)f\\in\\mathcal\{F\}\(d\_\{1\},d\_\{D\+2\},D,W,\\sigma\)inherits theC∞C^\{\\infty\}regularity of its activation primitives, rendering the empirical process amenable to high\-order metric entropy bounds\. Apart from the architectural change, the estimation procedure remains the same as for ReLU networks\. Specifically, given a loss functionℓ:ℝdy×ℝdy→ℝ\+\\ell:\\mathbb\{R\}^\{d\_\{y\}\}\\times\\mathbb\{R\}^\{d\_\{y\}\}\\to\\mathbb\{R\}\_\{\+\}and*i\.i\.d\.*observations\{Xi,Yi\}∈ℝdx×ℝdy\\\{X\_\{i\},Y\_\{i\}\\\}\\in\\mathbb\{R\}^\{d\_\{x\}\}\\times\\mathbb\{R\}^\{d\_\{y\}\}fori∈\[n\]i\\in\[n\], we define the smooth DNN estimator by

f^n∈arg⁡minf∈ℱ​\(dx,dy,D,W,σ\)1n​∑i=1nℓ​\(f​\(Xi\),Yi\)\.\\hat\{f\}\_\{n\}\\;\\in\\;\\mathop\{\\arg\\min\}\_\{f\\in\\mathcal\{F\}\(d\_\{x\},d\_\{y\},D,W,\\sigma\)\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\\big\(f\(X\_\{i\}\),\\,Y\_\{i\}\\big\)\.\(10\)

## 4Theoretical Properties of Smooth DNNs

This section develops the core analytic properties of the smooth DNNs\. Specifically, the pseudo–dimension \(Pdim\) is quantified in Section[4\.1](https://arxiv.org/html/2606.05599#S4.SS1)\. The approximation error bounds for Sobolev functions and hierarchical composition models are established in Section[4\.2](https://arxiv.org/html/2606.05599#S4.SS2)\. Hölder norm upper bounds for approximators are provided in Section[4\.3](https://arxiv.org/html/2606.05599#S4.SS3)\.

### 4\.1Pseudo\-Dimension

In statistical learning theory, the statistical \(generalization\) error off^n\\hat\{f\}\_\{n\}is governed by the complexity of the candidate classℱ​\(d1,dD\+2,D,W,σ\)\\mathcal\{F\}\(d\_\{1\},d\_\{D\+2\},D,W,\\sigma\), with higher complexity typically yielding slower rates\. Complexity analysis for neural networks began with binary threshold activations and was later extended to real\-valued networks via the Pdim\[[2](https://arxiv.org/html/2606.05599#bib.bib14)\]\. More recently, nearly tight Pdim bounds have been established for networks with piecewise\-polynomial activations, most notably ReLU\[[3](https://arxiv.org/html/2606.05599#bib.bib12)\]\. While these results largely tie complexity control to specific activation classes,\[[15](https://arxiv.org/html/2606.05599#bib.bib32)\]bounded complexity in terms of the operation number needed to compute the network output\. Building on this idea,\[[24](https://arxiv.org/html/2606.05599#bib.bib22)\]showed that when the neural network is describable by Pfaffian functions, the Pdim admits polynomial upper bounds in the number of parameters\. This framework is particularly convenient for establishing Pdim bounds of smooth DNNs\. Hence, we first recall the definition of a Pfaffian function\.

###### Definition 4\.1 \(Pfaffian Function\)\.

APfaffian chainof orderℓ≥0\\ell\\geq 0and degreeα≥1\\alpha\\geq 1on a domainU⊆ℝdU\\subseteq\\mathbb\{R\}^\{d\}is a sequence of analytic functions\(f1,f2,⋯,fℓ\)\(f\_\{1\},f\_\{2\},\\cdots,f\_\{\\ell\}\)overUUsuch that,

d​fj​\(𝐱\)d​xi=gi​j​\(𝐱,f1​\(𝐱\),⋯,fj​\(𝐱\)\),i∈\[d\],j∈\[ℓ\],for any𝐱=\(x1,⋯,xd\)∈U,\\frac\{\\text\{d\}f\_\{j\}\(\\mathbf\{x\}\)\}\{\\text\{d\}x\_\{i\}\}=g\_\{ij\}\(\\mathbf\{x\},f\_\{1\}\(\\mathbf\{x\}\),\\cdots,f\_\{j\}\(\\mathbf\{x\}\)\),\\quad i\\in\[d\],\\,j\\in\[\\ell\],\\quad\\text\{for any $\\mathbf\{x\}=\(x\_\{1\},\\cdots,x\_\{d\}\)\\in U$,\}where eachgi​j​\(𝐱,y1,⋯,yj\)g\_\{ij\}\(\\mathbf\{x\},y\_\{1\},\\cdots,y\_\{j\}\)is a polynomial in\(𝐱,y1,⋯,yj\)∈ℝd\+j\(\\mathbf\{x\},y\_\{1\},\\cdots,y\_\{j\}\)\\in\\mathbb\{R\}^\{d\+j\}of degree at mostα\\alpha\. Given a Pfaffian chain\(f1,…,fℓ\)\(f\_\{1\},\\dots,f\_\{\\ell\}\)onUU, a functionf:U→ℝf:U\\to\\mathbb\{R\}is called a*Pfaffian function*with*Pfaffian format*\(d,ℓ,α,β\)\(d,\\ell,\\alpha,\\beta\), if there is a polynomialppof degreeβ\\betasuch that

f​\(𝐱\)=p​\(𝐱,f1​\(𝐱\),⋯,fℓ​\(𝐱\)\)\.f\(\\mathbf\{x\}\)=p\(\\mathbf\{x\},f\_\{1\}\(\\mathbf\{x\}\),\\cdots,f\_\{\\ell\}\(\\mathbf\{x\}\)\)\.

Indeed, the smooth activations of interest in modern deep learning practice are Pfaffian\.

###### Proposition 4\.2\.

All of theC∞C^\{\\infty\}activations in \([5](https://arxiv.org/html/2606.05599#S3.E5)\)–\([7](https://arxiv.org/html/2606.05599#S3.E7)\) are Pfaffian function onℝ\\mathbb\{R\}, and their Pfaffian formats are\(1,1,2,2\)\(1,1,2,2\)for SiLU,\(1,2,4,2\)\(1,2,4,2\)for GELU, and\(1,6,5,2\)\(1,6,5,2\)for Mish\.

We next recall the definition of Pdim for real\-valued function classes\.

###### Definition 4\.3 \(Pseudo\-dimension\[[2](https://arxiv.org/html/2606.05599#bib.bib14)\]\)\.

Letℱ⊂ℝ𝒳\\mathcal\{F\}\\subset\\mathbb\{R\}^\{\\mathcal\{X\}\}be a class of real\-valued functions\. We say thatx1,…,xn∈𝒳x\_\{1\},\\dots,x\_\{n\}\\in\\mathcal\{X\}together withy1,…,yn∈ℝy\_\{1\},\\dots,y\_\{n\}\\in\\mathbb\{R\}are*pseudo\-shattered*byℱ\\mathcal\{F\}if\{\(𝕀​\{f​\(xi\)\>yi\}\)i=1n:f∈ℱ\}=\{0,1\}n\.\\big\\\{\(\\,\\mathbb\{I\}\\\{f\(x\_\{i\}\)\>y\_\{i\}\\\}\)\_\{i=1\}^\{n\}:f\\in\\mathcal\{F\}\\big\\\}=\\\{0,1\\\}^\{n\}\.The*pseudo\-dimension*ofℱ\\mathcal\{F\}, denoted asPdim⁡\(ℱ\)\\operatorname\{Pdim\}\(\\mathcal\{F\}\), is the largestnnfor which there existx1,…,xnx\_\{1\},\\dots,x\_\{n\}andy1,…,yny\_\{1\},\\dots,y\_\{n\}that are pseudo\-shattered byℱ\\mathcal\{F\}\.

We now present an upper bound on the Pdim of smooth DNNs with Pfaffian activations\.

###### Theorem 4\.4\.

Letσ∈C∞​\(ℝ\)\\sigma\\in C^\{\\infty\}\(\\mathbb\{R\}\)be an analytic activation with Pfaffian format\(1,ℓ,α,β\)\(1,\\ell,\\alpha,\\beta\)\. Then, for some suppressed universal constant, the Pdim of the smooth DNN satisfies

Pdim⁡\(ℱ​\(d,1,D,W,σ\)\)≲logD4​W6,\\operatorname\{Pdim\}\(\\mathcal\{F\}\(d,1,D,W,\\sigma\)\)\\ \\lesssim\_\{\\log\}\\ D^\{4\}W^\{6\},where the suppressed constant depend ondd, Pfaffian format ofσ\\sigma, and poly\-logarithmic terms ofDDandWW\.

### 4\.2Approximation Error

As the smooth DNN becomes deeper or wider, its increased Pdim may incur larger statistical error, while the increased capacity can reduce approximation error by representing more complex targets\. Thus, non\-asymptotic approximation bounds for smooth DNNs are required to identify architectural scalings that balance these two errors\.

Building on approximation theory for ReLU FNNs\[[48](https://arxiv.org/html/2606.05599#bib.bib9),[30](https://arxiv.org/html/2606.05599#bib.bib10)\], recent studies have extended approximation guarantees to smooth FNNs\. In particular,\[[50](https://arxiv.org/html/2606.05599#bib.bib36)\]showed that any ReLU FNN can be approximated arbitrarily well by FNNs for a broad class of smooth activations\. More recently,\[[46](https://arxiv.org/html/2606.05599#bib.bib37)\]leveraged linear combinations of smooth activations to approximate monomials and derived non\-asymptotic approximation error bounds for such networks\. Following the strategy in\[[46](https://arxiv.org/html/2606.05599#bib.bib37)\], we establish non\-asymptotic approximation error bounds for smooth DNNs when approximating Sobolev functions and hierarchical composition models\. We begin by stating the analytic assumptions on the activationσ\\sigma\.

###### Assumption 4\.5\.

Suppose the activationσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}satisfies, for some𝔪≥3\\mathfrak\{m\}\\geq 3,

- •\(𝔪\\mathfrak\{m\}\-th order quasi\-decay\) LetH​\(x\):=𝕀​\(x\>0\)H\(x\):=\\mathbb\{I\}\(x\>0\)be the Heaviside step function, and defineψ​\(x\):=σ​\(x\)/x\\psi\(x\):=\\sigma\(x\)/x\. There exist constantsC,G\>0C,G\>0such that, for anyx≠0x\\neq 0and0≤k≤𝔪0\\leq k\\leq\\mathfrak\{m\}, \|ψ\(k\)​\(x\)−H\(k\)​\(x\)\|≤min⁡\{C\|x\|k\+1,G\}\.\|\\psi^\{\(k\)\}\(x\)\-H^\{\(k\)\}\(x\)\|\\leq\\min\\Bigl\\\{\\frac\{C\}\{\|x\|^\{k\+1\}\},\\,G\\Bigr\\\}\.
- •\(LocalC∞C^\{\\infty\}\) There exista∈ℝa\\in\\mathbb\{R\}andδ∗\>0\\delta\_\{\\ast\}\>0such that σ∈C∞​\(a−δ∗,a\+δ∗\),σ′​\(a\)≠0,andσ′′​\(a\)≠0\.\\sigma\\in C^\{\\infty\}\(a\-\\delta\_\{\\ast\},a\+\\delta\_\{\\ast\}\),\\quad\\sigma^\{\\prime\}\(a\)\\neq 0,\\quad\\text\{and\}\\quad\\sigma^\{\{\\prime\\prime\}\}\(a\)\\neq 0\.

This assumption ensures that smooth DNNs can approximate monomials and two specific functions that are central in neural network approximation theory\. For theC∞C^\{\\infty\}activations of interest in this work, including SiLU, GELU, and Mish, the localC∞C^\{\\infty\}assumption is satisfied\. Since these activations admit a uniformly bounded and increasing functionψ​\(x\)=σ​\(x\)/x\\psi\(x\)=\\sigma\(x\)/xwith exponentially decaying derivatives, the𝔪\\mathfrak\{m\}\-th order quasi\-decay assumption also holds\.

We now present the approximation error bounds of smooth DNNs for Sobolev functions\.

###### Theorem 4\.6\.

Suppose Assumption[4\.5](https://arxiv.org/html/2606.05599#S4.Thmtheorem5)holds\. Then, for any Sobolev functionf∈Wα,∞​\(Ω\)f\\in W^\{\\alpha,\\infty\}\(\\Omega\)and anyW,D∈ℕ\+W,D\\in\\mathbb\{N\}\_\{\+\}satisfyinglog2⁡W≤D\\log\_\{2\}W\\leq D, there exists a smooth DNNϕ\\phiwith width≲W​log⁡\(W\)\\lesssim W\\,\\log\(W\)and depth≲D​log⁡\(D\)\\lesssim D\\,\\log\(D\), where the suppressed constants depend only ondd,α\\alpha, and𝔪\\mathfrak\{m\}, such that, fors=0,1,⋯,𝔪∧⌊α⌋s=0,1,\\cdots,\\mathfrak\{m\}\\land\\lfloor\\alpha\\rfloor,

‖f−ϕ‖Ws,∞​\(Ω\)≤C16​\(α,d,𝔪\)​‖f‖Wα,∞​\(Ω\)​\(W​D\)−2​\(α−s\)d\.\\\|f\-\\phi\\\|\_\{W^\{s,\\infty\}\(\\Omega\)\}\\leq C\_\{16\}\(\\alpha,d,\\mathfrak\{m\}\)\\\|f\\\|\_\{W^\{\\alpha,\\infty\}\(\\Omega\)\}\\,\(WD\)^\{\-\\frac\{2\(\\alpha\-s\)\}\{d\}\}\.\(11\)

This result establishes approximation rates for smooth DNNs when approximating Sobolev functions\. In what follows, we establish non\-asymptotic approximation error bounds for smooth DNNs under hierarchical composition models\.

###### Definition 4\.7 \(Hierarchical composition model\)\.

Givend,ℓ∈ℕd,\\ell\\in\\mathbb\{N\},C\>0C\>0, and𝒫⊆ℕ\+×ℕ\+\\mathscr\{P\}\\subseteq\\mathbb\{N\}\_\{\+\}\\times\\mathbb\{N\}\_\{\+\}, such thatsup\(β,t\)∈𝒫β∨t<∞\\sup\_\{\(\\beta,t\)\\in\\mathscr\{P\}\}\\beta\\lor t<\\infty, the hierarchical composition modelℋ​\(d,ℓ,𝒫,C\)\\mathcal\{H\}\(d,\\ell,\\mathscr\{P\},C\)is defined recursively as follows\. Forℓ=1\\ell=1,

ℋ​\(d,1,𝒫,C\)=\\displaystyle\\mathcal\{H\}\(d,1,\\mathscr\{P\},C\)=\{h:ℝd→ℝ:h\(x\)=g\(xπ​\(1\),⋯,xπ​\(t\)\),whereπ:\[t\]→\[d\]and\\displaystyle\\\{h:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}:h\(x\)=g\(x\_\{\\pi\(1\)\},\\cdots,x\_\{\\pi\(t\)\}\),\\text\{ where \}\\pi:\[t\]\\to\[d\]\\text\{ and \}g:ℝt→ℝis\(β,C\)\-smooth for some\(β,t\)∈𝒫\};\\displaystyle\\qquad g:\\mathbb\{R\}^\{t\}\\to\\mathbb\{R\}\\text\{ is \}\(\\beta,C\)\\text\{\-smooth for some \}\(\\beta,t\)\\in\\mathscr\{P\}\\\};and forℓ\>1\\ell\>1,

ℋ​\(d,ℓ,𝒫,C\)=\\displaystyle\\mathcal\{H\}\(d,\\ell,\\mathscr\{P\},C\)=\{h:ℝd→ℝ:h\(x\)=g\(f1\(x\),⋯,ft\(x\)\),wherefi∈ℋ\(d,ℓ−1,𝒫\)and\\displaystyle\\\{h:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}:h\(x\)=g\(f\_\{1\}\(x\),\\cdots,f\_\{t\}\(x\)\),\\text\{ where \}f\_\{i\}\\in\\mathcal\{H\}\(d,\\ell\-1,\\mathscr\{P\}\)\\text\{ and \}g:ℝt→ℝis\(β,C\)\-smooth for some\(β,t\)∈𝒫\}\.\\displaystyle\\qquad g:\\mathbb\{R\}^\{t\}\\to\\mathbb\{R\}\\text\{ is \}\(\\beta,C\)\\text\{\-smooth for some \}\(\\beta,t\)\\in\\mathscr\{P\}\\\}\.The adjusted model complexity ofℋ​\(d,ℓ,𝒫,C\)\\mathcal\{H\}\(d,\\ell,\\mathscr\{P\},C\)is defined asγ∗=min\(β,t\)∈𝒫⁡β/t\\gamma^\{\\ast\}=\\min\_\{\(\\beta,t\)\\in\\mathscr\{P\}\}\\beta/t\.

Leveraging results in Theorem[4\.6](https://arxiv.org/html/2606.05599#S4.Thmtheorem6), we obtain the following approximation error results for smooth DNNs under hierarchical composition models\.

###### Theorem 4\.8\.

For anyf0∈ℋ​\(d,ℓ,𝒫,C\)f\_\{0\}\\in\\mathcal\{H\}\(d,\\ell,\\mathscr\{P\},C\),W,D∈ℕ\+W,D\\in\\mathbb\{N\}\_\{\+\}withlog2⁡W≤D\\log\_\{2\}W\\leq D, andW≥max⁡\{t:\(β,t\)∈𝒫\}W\\geq\\max\\\{t:\(\\beta,t\)\\in\\mathscr\{P\}\\\}, there is aσ\\sigma\-activated smooth DNNϕf0\\phi\_\{f\_\{0\}\}with widthC19​\(d,𝔪,ℓ,𝒫\)⋅W​log⁡\(W\),C\_\{19\}\(d,\\mathfrak\{m\},\\ell,\\mathscr\{P\}\)\\cdot W\\log\(W\),depthC19​\(d,𝔪,ℓ,𝒫\)⋅D​log⁡\(D\),C\_\{19\}\(d,\\mathfrak\{m\},\\ell,\\mathscr\{P\}\)\\cdot D\\log\(D\),and activationσ\\sigmasatisfying Assumption[4\.5](https://arxiv.org/html/2606.05599#S4.Thmtheorem5), such that

‖f0−ϕf0‖L∞​\(\[0,1\]d\)≤C18​\(d,𝔪,C,ℓ,𝒫\)​\(W​D\)−2​γ∗\.\\\|f\_\{0\}\-\\phi\_\{f\_\{0\}\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\leq C\_\{18\}\(d,\\mathfrak\{m\},C,\\ell,\\mathscr\{P\}\)\\,\(WD\)^\{\-2\\gamma^\{\\ast\}\}\.\(12\)

Theorem[4\.8](https://arxiv.org/html/2606.05599#S4.Thmtheorem8)shows that when the unknown targetf0f\_\{0\}admits a hierarchical composition of several\(β,C\)\(\\beta,C\)\-smooth functions, each with either high smoothness or low input dimension, smooth DNNs can adapt to this intrinsic low\-dimensional structure without explicit knowledge of the underlying composition\. In particular, smooth DNNs achieve the same approximation bounds as ReLU FNNs under the hierarchical composition model\[[4](https://arxiv.org/html/2606.05599#bib.bib13),[37](https://arxiv.org/html/2606.05599#bib.bib8),[26](https://arxiv.org/html/2606.05599#bib.bib11)\]\.

### 4\.3Upper Bounds of Hölder Norms

As discussed in Section[2\.2](https://arxiv.org/html/2606.05599#S2.SS2), uniform convergence rates are governed by the smoothness of the candidate function class\. Consequently, to establish uniform convergence rates for smooth DNN estimators, we need to characterize their Hölder norms across different orders\.

We first examine the analytic properties of the activations SiLU, GELU, and Mish\.

###### Proposition 4\.9\.

For the activation functionσ\\sigmacorresponding to SiLU, GELU, and Mish, there exists a constantCσ\>0C\_\{\\sigma\}\>0, depending only onσ\\sigma, such that

supz∈ℝ\|σ\(m\)​\(z\)\|≤\(Cσ​m\)m,for all​m≥1\.\\sup\_\{z\\in\\mathbb\{R\}\}\|\\sigma^\{\(m\)\}\(z\)\|\\leq\(C\_\{\\sigma\}\\,m\)^\{m\},\\quad\\text\{for all \}m\\geq 1\.

Proposition[4\.9](https://arxiv.org/html/2606.05599#S4.Thmtheorem9)shows that the derivatives of the smooth activations of interest grow at a factorial rate\. As these activations are the building blocks of smooth DNNs, this result provides a natural baseline for the order of the Hölder\-norm bounds that one can expect for smooth DNNs\. The following result further establishes factorial\-order Hölder norm bounds for smooth DNN approximators on a subset ofΩ\\Omegathat is arbitrarily close toΩ\\Omega\.

###### Theorem 4\.10\.

Under the conditions of Theorem[4\.6](https://arxiv.org/html/2606.05599#S4.Thmtheorem6), for anyδ∈\(0,1\)\\delta\\in\(0,1\), there exists a smooth DNNϕ\\phi, whose activationσ\\sigmasatisfies Assumption[4\.5](https://arxiv.org/html/2606.05599#S4.Thmtheorem5), with the same width and depth as in Theorem[4\.6](https://arxiv.org/html/2606.05599#S4.Thmtheorem6), such that \([11](https://arxiv.org/html/2606.05599#S4.E11)\) continues to hold, and there exists a measurable subsetΩδ⊆Ω\\Omega\_\{\\delta\}\\subseteq\\Omegawith the Lebesgue measureλd​\(Ωδ\)≥1−δ\\lambda\_\{d\}\(\\Omega\_\{\\delta\}\)\\geq 1\-\\deltasatisfying

‖ϕ‖C𝔪​\(Ωδ\)≤\(C19​\(α,d\)​𝔪\)𝔪​‖f‖Wα,∞​\(Ω\)\.\\\|\\phi\\\|\_\{C^\{\\mathfrak\{m\}\}\(\\Omega\_\{\\delta\}\)\}\\leq\\Bigl\(C\_\{19\}\(\\alpha,d\)\\,\\mathfrak\{m\}\\Bigr\)^\{\\mathfrak\{m\}\}\\,\\\|f\\\|\_\{W^\{\\alpha,\\infty\}\(\\Omega\)\}\.\(13\)

This result shows that the smooth DNN approximator admits a factorial\-order upper bound on the Hölder normC𝔪C^\{\\mathfrak\{m\}\}, as that for analytic activations in Proposition[4\.9](https://arxiv.org/html/2606.05599#S4.Thmtheorem9), even when𝔪\>α\\mathfrak\{m\}\>\\alphaexceeds the smoothness order of the target function\. Moreover, this result does not requireσ\\sigmato be analytic, indicating that the factorial\-order upper bound originates from the intrinsic structure of smooth DNNs rather than from the analyticity ofσ\\sigma\.

Rather than establishing a factorial\-order upper bound on the entire domainΩ=\[0,1\]d\\Omega=\[0,1\]^\{d\}, we obtain such a bound only on a subsetΩδ\\Omega\_\{\\delta\}, whose measure can be made arbitrarily close to that ofΩ\\Omega\. This restriction arises because the smooth DNN approximatorϕ\\phiinvolves certain sub\-networks whose Sobolev norm control is difficult to establish on small regions of the domain\. That said, we believe that the present result is sufficient to explain the well\-behaved Hölder norms of smooth DNN approximators\.

## 5Uniform Convergence of Smooth DNN Estimators

This section derives uniform convergence rates for smooth DNN estimators under several widely used models\. Specifically, in Section[5\.1](https://arxiv.org/html/2606.05599#S5.SS1), we study smooth DNN estimators under the Huber regression model, establish their robustness to heavy\-tailed noise, and consider least\-squares regression as a special case of Huber regression\. We then investigate quantile regression with smooth DNN estimators in Section[5\.2](https://arxiv.org/html/2606.05599#S5.SS2)\. Uniform convergence rates for smooth DNN estimators under logistic regression are established in Section[5\.3](https://arxiv.org/html/2606.05599#S5.SS3)\.

Before turning to these models, we assume the covariateXiX\_\{i\}is supported onΩ:=\[0,1\]d\\Omega:=\[0,1\]^\{d\}and admits a bounded density throughout this section\. In particular, there exist constants0<c<C<∞0<c<C<\\infty, such as the densitypXp\_\{X\}ofXiX\_\{i\}satisfies

0<c<pX​\(x\)≤C<∞,for all​x∈\[0,1\]d\.0<c<p\_\{X\}\(x\)\\leq C<\\infty,\\quad\\text\{for all \}x\\in\[0,1\]^\{d\}\.We also impose the following assumption on the activation functionσ\\sigmato invoke the theoretical properties of smooth DNNs established in Section[4](https://arxiv.org/html/2606.05599#S4)\. SiLU, GELU, and Mish are key examples satisfying these conditions\.

###### Assumption 5\.1\.

The activationσ\\sigmais Pfaffian and satisfies Assumption[4\.5](https://arxiv.org/html/2606.05599#S4.Thmtheorem5)for any𝔪∈ℕ\\mathfrak\{m\}\\in\\mathbb\{N\}\.

To leverage interpolation inequalities when deriving uniform convergence rates, we impose the following assumption on the Hölder norms of the candidate function class\. As discussed in Section[4\.3](https://arxiv.org/html/2606.05599#S4.SS3), the rationale is that both analytic activations and smooth DNN approximators to Sobolev functions satisfy factorial\-type upper bounds on their Hölder norms\. Thus, this condition can be interpreted as restricting attention to a bounded subset of smooth DNNs in a neighborhood of the approximator\.

###### Assumption 5\.2\.

For target functionf0∈ℋ​\(d,ℓ,𝒫,C\)f\_\{0\}\\in\\mathcal\{H\}\(d,\\ell,\\mathscr\{P\},C\), define the approximator

fn∗∈arg⁡minf∈ℱ​\(d,1,D,W,σ\)‖f−f0‖L∞​\(Ω\)\.f\_\{n\}^\{\\ast\}\\in\\mathop\{\\arg\\min\}\_\{f\\in\\mathcal\{F\}\(d,1,D,W,\\sigma\)\}\\\|f\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\.We assume thatfn∗∈ℱ​\(d,1,D,W,σ;A\)f\_\{n\}^\{\\ast\}\\in\\mathcal\{F\}\(d,1,D,W,\\sigma;A\)for someA\>0A\>0, where

ℱ​\(d,1,D,W,σ;A\)=\{f∈ℱ​\(d,1,D,W,σ\):‖f‖C𝔪​\(Ω\)≤\(A​𝔪\)𝔪​for all𝔪∈ℕ\}\.\\mathcal\{F\}\(d,1,D,W,\\sigma;A\)=\\\{f\\in\\mathcal\{F\}\(d,1,D,W,\\sigma\):\\\|f\\\|\_\{C^\{\\mathfrak\{m\}\}\(\\Omega\)\}\\leq\(A\\,\\mathfrak\{m\}\)^\{\\mathfrak\{m\}\}\\text\{ for all $\\mathfrak\{m\}\\in\\mathbb\{N\}$\}\\\}\.

### 5\.1Huber Regression and Least Squares

In this section, we study the uniform convergence of smooth DNN estimators in Huber regression and least squares\. Specifically, let\{\(Xi,Yi\)\}i=1n∼P\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}\\sim Pbe*i\.i\.d\.*observations satisfying

Yi=f0​\(Xi\)\+ξi,where𝔼​\[ξi∣Xi\]=0,Y\_\{i\}=f\_\{0\}\(X\_\{i\}\)\+\\xi\_\{i\},\\qquad\\text\{where\}\\quad\\mathbb\{E\}\[\\xi\_\{i\}\\mid X\_\{i\}\]=0,\(14\)wheref0f\_\{0\}denotes the target function of interest\.

Motivated by the prevalence of heavy\-tailed data in real\-world applications, a growing line of work has investigated theL2L^\{2\}estimation error for nonparametric function estimators under heavy\-tailed noise\[[17](https://arxiv.org/html/2606.05599#bib.bib5),[27](https://arxiv.org/html/2606.05599#bib.bib4),[12](https://arxiv.org/html/2606.05599#bib.bib7),[9](https://arxiv.org/html/2606.05599#bib.bib30)\]\. In contrast to light\-tailed noise, heavy\-tailed noise produces outliers inYiY\_\{i\}more frequently, making classical least\-squares estimators unreliable\. To address this issue, the Huber loss was introduced as a robust alternative, defined as, for the Huber parameterτ≥0\\tau\\geq 0,ℓτ​\(x\)=12​x2​𝕀​\(\|x\|≤τ\)\+\(τ​\|x\|−12​τ2\)​𝕀​\(\|x\|\>τ\)\.\\ell\_\{\\tau\}\(x\)=\\frac\{1\}\{2\}x^\{2\}\\mathbb\{I\}\(\|x\|\\leq\\tau\)\+\(\\tau\|x\|\-\\frac\{1\}\{2\}\\tau^\{2\}\)\\mathbb\{I\}\(\|x\|\>\\tau\)\.Then, for a function classℱn\\mathcal\{F\}\_\{n\}, the Huber estimator is defined as

f^n​\(τ\)∈arg⁡minf∈ℱn1n​∑i=1nℓτ​\(Yi−f​\(Xi\)\)\.\\hat\{f\}\_\{n\}\(\\tau\)\\in\\mathop\{\\arg\\min\}\_\{f\\in\\mathcal\{F\}\_\{n\}\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\_\{\\tau\}\\big\(Y\_\{i\}\-f\(X\_\{i\}\)\\big\)\.\(15\)
Recent advances have characterized the robustness of the Huber estimatorf^n​\(τ\)\\hat\{f\}\_\{n\}\(\\tau\)for different function classes with non\-asymptotic analysis\. Whenℱn\\mathcal\{F\}\_\{n\}consists of linear functions,\[[42](https://arxiv.org/html/2606.05599#bib.bib1)\]showed that the adaptive Huber estimator achieves a sub\-Gaussian concentration bound in the presence of heavy\-tailed noise\. More recently, analogous robustness guarantees for ReLU FNN estimators were established in prior works\[[12](https://arxiv.org/html/2606.05599#bib.bib7),[9](https://arxiv.org/html/2606.05599#bib.bib30)\]\. However, these results are formulated in theL2​\(P\)L^\{2\}\(P\)norm rather than inL∞L^\{\\infty\}, leaving open whether DNN estimators remain robust in the uniform norm under heavy\-tailed regimes\.

The assumption below is imposed to show that smooth DNN Huber estimators are robust in the uniform norm under heavy\-tailed noise\.

###### Assumption 5\.3\.

For the model in \([14](https://arxiv.org/html/2606.05599#S5.E14)\), assume:

- •Target function:For someM\>1M\>1,‖f0‖L∞​\(\[0,1\]d\)≤M\\\|f\_\{0\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\leq M\.
- •Noise:For somem\>1m\>1, there existsvm∈\(0,∞\)v\_\{m\}\\in\(0,\\infty\)such that∥𝔼\[\|ξi\|m∣Xi\]∥L∞≤vm\\\|\\mathbb\{E\}\[\|\\xi\_\{i\}\|^\{m\}\\mid X\_\{i\}\]\\\|\_\{L^\{\\infty\}\}\\leq v\_\{m\}\.

We now establish robustness guarantees for the estimatorf^n​\(τ\)\\hat\{f\}\_\{n\}\(\\tau\)in theL∞​\(Ω\)L^\{\\infty\}\(\\Omega\)norm\.

###### Theorem 5\.4\.

Suppose Assumptions[5\.1](https://arxiv.org/html/2606.05599#S5.Thmtheorem1),[5\.2](https://arxiv.org/html/2606.05599#S5.Thmtheorem2)and[5\.3](https://arxiv.org/html/2606.05599#S5.Thmtheorem3)\(takingv2=∞v\_\{2\}=\\inftyifm<2m<2\) hold, and assumeτ≥2​max⁡\{2​M,\(2​vm\)1/m\}\.\\tau\\geq 2\\max\\\{2M,\(2v\_\{m\}\)^\{1/m\}\\\}\.Letℱn=ℱ​\(d,1,D,W,σ;A\)∩L∞​\(M\)\\mathcal\{F\}\_\{n\}=\\mathcal\{F\}\(d,1,D,W,\\sigma;A\)\\cap L^\{\\infty\}\(M\)be the uniformly bounded smooth DNN class\. Denote the effective sample size asn~=n/\(D4​W6\)\\tilde\{n\}=n/\(D^\{4\}W^\{6\}\)and assumen~≥1\\tilde\{n\}\\geq 1\. Then, for anyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta,

‖f^n​\(τ\)−f0‖L∞​\(Ω\)≲log\(δ𝔰\+\(W​D\)−2​γ∗\+vmτm−1\+τ∧\(v2\+M\)​τ​log⁡\(10/δ\)n\)0\.99\.\\displaystyle\\\|\\hat\{f\}\_\{n\}\(\\tau\)\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\lesssim\_\{\\log\}\\Bigl\(\\delta\_\{\\mathfrak\{s\}\}\+\(WD\)^\{\-2\\gamma^\{\\ast\}\}\+\\frac\{v\_\{m\}\}\{\\tau^\{m\-1\}\}\+\\sqrt\{\\tau\\land\(\\sqrt\{v\_\{2\}\}\+M\)\}\\,\\sqrt\{\\frac\{\\tau\\,\\log\(10/\\delta\)\}\{n\}\}\\Bigr\)^\{0\.99\}\.where the exponent0\.990\.99can be replaced by any constant in\(0,1\)\(0,1\)\. The suppressed constant depends ondd,AA,CC,ℓ\\ell,𝒫\\mathscr\{P\},σ\\sigma,PP, and polylogarithmic terms innn,WW, andDD\. Moreover,

δ𝔰=\{\(M\+v2\)​\(M\+vm1/m\)⋅n~1/\(2​m\)−1/2ifτ≥n~1m​\(M\+vm1/m\)andm≥2,τ∧\(M\+v2\)⋅τ/n~otherwise\.\\delta\_\{\\mathfrak\{s\}\}=\\begin\{cases\}\\sqrt\{\(M\+\\sqrt\{v\_\{2\}\}\)\(M\+v\_\{m\}^\{1/m\}\)\}\\cdot\\tilde\{n\}^\{1/\(2m\)\-1/2\}&\\text\{if $\\tau\\geq\\tilde\{n\}^\{\\frac\{1\}\{m\}\}\\,\(M\+v\_\{m\}^\{1/m\}\)$ and $m\\geq 2$,\}\\\\ \\sqrt\{\\tau\\land\(M\+\\sqrt\{v\_\{2\}\}\)\}\\cdot\\sqrt\{\\tau/\\tilde\{n\}\}&\\text\{otherwise\.\}\\end\{cases\}\(16\)

In this result, the uniform estimation error of the smooth DNN Huber estimatorf^n​\(τ\)\\hat\{f\}\_\{n\}\(\\tau\)is decomposed into four components: \(i\) statistical errorδ𝔰\\delta\_\{\\mathfrak\{s\}\}, \(ii\) approximation error\(W​D\)−2​γ∗\(WD\)^\{\-2\\gamma^\{\\ast\}\}, \(iii\) Huberization biasvm/τm−1v\_\{m\}/\\tau^\{m\-1\}, and \(iv\) a sub\-Gaussian deviation term\. These components reveal two natural trade\-offs\. First, increasingτ\\taureduces the Huber bias but enlarges both the deviation term andδ𝔰\\delta\_\{\\mathfrak\{s\}\}, since the Huber loss approaches the least\-squares loss\. Second, enlarging the network architecture \(in terms ofDDandWW\) decreases the approximation error but reduces the effective sample sizen~\\tilde\{n\}, thereby increasing the statistical error\. Moreover, thisL∞L^\{\\infty\}estimation error decomposition parallels that of ReLU Huber estimators in theL2L^\{2\}norm\[[12](https://arxiv.org/html/2606.05599#bib.bib7),[9](https://arxiv.org/html/2606.05599#bib.bib30)\], with the only distinction arising from the pseudo\-dimensions of the two function classes\. This comparison highlights a concrete advantage of smooth DNN estimators over existing ReLU FNNs in terms of uniform convergence guarantees\.

We now turn to least\-squares regression as a special case of Huber regression\. As shown in Section[2](https://arxiv.org/html/2606.05599#S2), ReLU FNN least\-squares estimators suffer from thecurse of dimensionalityeven under idealized settings, in sharp contrast to their favorableL2L^\{2\}convergence behavior established in\[[12](https://arxiv.org/html/2606.05599#bib.bib7),[9](https://arxiv.org/html/2606.05599#bib.bib30)\]\. Since the least\-squares loss can be viewed as the limiting case of the Huber lossℓτ\\ell\_\{\\tau\}asτ→∞\\tau\\to\\infty, Theorem[5\.4](https://arxiv.org/html/2606.05599#S5.Thmtheorem4)suggests the corresponding uniform convergence behavior of smooth DNN least\-squares estimators under heavy\-tailed noise\. However, one cannot directly letτ→∞\\tau\\to\\inftyin Theorem[5\.4](https://arxiv.org/html/2606.05599#S5.Thmtheorem4), because the final term, namely the sub\-Gaussian deviation bound, diverges in this limit\. This is precisely because least\-squares regression is not robust to heavy\-tailed noise, and therefore cannot satisfy a non\-asymptotic heavy\-tailed robustness guarantee of sub\-Gaussian type\. For this reason, in Theorem[5\.5](https://arxiv.org/html/2606.05599#S5.Thmtheorem5)below, we formulate the uniform estimation error of smooth DNN least\-squares estimators in terms of convergence in expectation rather than a non\-asymptotic deviation bound\.

###### Theorem 5\.5\.

Under the conditions and notation of Theorem[5\.4](https://arxiv.org/html/2606.05599#S5.Thmtheorem4), whenm≥2m\\geq 2, the smooth DNN least\-squares estimatorf^n\\hat\{f\}\_\{n\}satisfies

𝔼​‖f^n−f0‖L∞​\(Ω\)≲log\(\(M\+vm1/m\)⋅n~1/\(2​m\)−1/2\+\(W​D\)−2​γ∗\)0\.99\.\\mathbb\{E\}\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\lesssim\_\{\\log\}\\Bigl\(\(M\+v\_\{m\}^\{1/m\}\)\\cdot\\tilde\{n\}^\{1/\(2m\)\-1/2\}\+\(WD\)^\{\-2\\gamma^\{\\ast\}\}\\Bigr\)^\{0\.99\}\.\\\\In particular, minimizing overDDandWWyields

𝔼​‖f^n−f0‖L∞​\(Ω\)≲logn−0\.99​γ∗​\(1−1/m\)2​γ∗\+2​\(1−1/m\)\.\\mathbb\{E\}\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\lesssim\_\{\\log\}n^\{\-\\frac\{0\.99\\,\\gamma^\{\\ast\}\(1\-1/m\)\}\{2\\gamma^\{\\ast\}\+2\(1\-1/m\)\}\}\.\\\\

While Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)shows that the convergence rate of ReLU FNN least\-squares estimators can be as slow asn−1d\+1n^\{\-\\frac\{1\}\{d\+1\}\}regardless of the smoothness off0f\_\{0\}, smooth DNN least\-squares estimators can adapt to the hierarchical composition structure off0f\_\{0\}, so that their convergence rates depend on the intrinsic dimensionality of its low\-dimensional representation\. When the noise is light\-tailed, as in Theorem[2\.1](https://arxiv.org/html/2606.05599#S2.Thmtheorem1)\(in the sense thatm→∞m\\to\\infty\), the resulting rate in Theorem[5\.5](https://arxiv.org/html/2606.05599#S5.Thmtheorem5)scales asn−0\.99​γ∗2​γ∗\+2n^\{\-\\frac\{0\.99\\,\\gamma^\{\\ast\}\}\{2\\gamma^\{\\ast\}\+2\}\}\. Thus, smooth DNN estimators can mitigate thecurse of dimensionalityfor least\-squares regression in the uniform norm by adapting to the low\-dimensional hierarchical composition structure off0f\_\{0\}\.

### 5\.2Quantile Regression

As another robust estimation task, quantile regression estimates conditional quantile functions\[[25](https://arxiv.org/html/2606.05599#bib.bib15)\]\. For a fixed quantile levelτ∈\(0,1\)\\tau\\in\(0,1\), suppose we observe*i\.i\.d\.*samples\{\(Xi,Yi\)\}i=1n\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}such that theτ\\tau\-th conditional quantile ofYiY\_\{i\}givenXi=xX\_\{i\}=xisf0​\(x\)f\_\{0\}\(x\)\. Thenf0f\_\{0\}minimizes the population quantile regression loss: for the check lossρτ​\(u\)=u​\(τ−𝕀​\(u<0\)\)\\rho\_\{\\tau\}\(u\)=u\(\\tau\-\\mathbb\{I\}\(u<0\)\),

f0∈arg⁡min𝔼​\[ρτ​\(Y1−f​\(X1\)\)\]\.f\_\{0\}\\in\\mathop\{\\arg\\min\}\\mathbb\{E\}\[\\rho\_\{\\tau\}\(Y\_\{1\}\-f\(X\_\{1\}\)\)\]\.Recent work has established non\-asymptoticL2​\(P\)L^\{2\}\(P\)convergence rates and non\-asymptotic robustness guarantees for ReLU FNN quantile regression estimators\[[9](https://arxiv.org/html/2606.05599#bib.bib30),[49](https://arxiv.org/html/2606.05599#bib.bib39)\]\. In the following, we establish robustness guarantees for smooth DNN quantile estimators in theL∞L^\{\\infty\}norm\. Specifically, we introduce the following regularity condition to ensure well\-behaved conditional densities in a neighborhood of the target quantile:

###### Assumption 5\.6\.

Assume for someM\>1M\>1,‖f0‖L∞​\(\[0,1\]d\)≤M\\\|f\_\{0\}\\\|\_\{L^\{\\infty\}\(\[0,1\]^\{d\}\)\}\\leq M\. DenotepY\|X=xp\_\{Y\|X=x\}as the condition density ofYYgivenX=xX=x\. Assume there exists someδ\>0\\delta\>0, such that0<inft∈\[f0​\(x\)−δ,f0​\(x\)\+δ\]pY\|X=x​\(t\)≤supt∈ℝpY\|X=x​\(t\)<∞,0<\\inf\_\{t\\in\[f\_\{0\}\(x\)\-\\delta,f\_\{0\}\(x\)\+\\delta\]\}p\_\{Y\|X=x\}\(t\)\\leq\\sup\_\{t\\in\\mathbb\{R\}\}p\_\{Y\|X=x\}\(t\)<\\infty,almost surely\.

Leveraging the theoretical results developed in Section[4](https://arxiv.org/html/2606.05599#S4), we establish robustness guarantees for the smooth DNN quantile regression estimator in theL∞L^\{\\infty\}norm below\.

###### Theorem 5\.7\.

Suppose Assumptions[5\.1](https://arxiv.org/html/2606.05599#S5.Thmtheorem1),[5\.2](https://arxiv.org/html/2606.05599#S5.Thmtheorem2)and[5\.6](https://arxiv.org/html/2606.05599#S5.Thmtheorem6)holds, and denote the smooth DNN classℱn=ℱ​\(d,1,D,W,σ;A\)∩L∞​\(M\)\.\\mathcal\{F\}\_\{n\}=\\mathcal\{F\}\(d,1,D,W,\\sigma;A\)\\cap L^\{\\infty\}\(M\)\.Define the smooth DNN quantile estimator as

f^n∈arg⁡minf∈ℱn∑i=1nρτ​\(Yi−f​\(Xi\)\)\.\\hat\{f\}\_\{n\}\\in\\mathop\{\\arg\\min\}\_\{f\\in\\mathcal\{F\}\_\{n\}\}\\sum\_\{i=1\}^\{n\}\\rho\_\{\\tau\}\(Y\_\{i\}\-f\(X\_\{i\}\)\)\.Denote the effective sample size asn~=n/\(D4​W6\)\\tilde\{n\}=n/\(D^\{4\}W^\{6\}\)\. Then, for anyδ∈\(0,1\)\\delta\\in\(0,1\),

ℙ​\(‖f^n−f0‖L∞​\(Ω\)≳log\(log⁡\(2/δ\)⋅n~−12\+\(W​D\)−2​γ∗\)0\.99\)≤δ\.\\mathbb\{P\}\\Bigl\(\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\gtrsim\_\{\\log\}\\bigl\(\\sqrt\{\\log\(2/\\delta\)\}\\cdot\\tilde\{n\}^\{\-\\frac\{1\}\{2\}\}\+\(WD\)^\{\-2\\gamma^\{\\ast\}\}\\bigr\)^\{0\.99\}\\Bigr\)\\leq\\delta\.

Similar to Theorem[5\.4](https://arxiv.org/html/2606.05599#S5.Thmtheorem4), the exponent0\.990\.99is flexible and can be replaced by any value strictly smaller than11\. Moreover, by optimizing over the depth and width of smooth DNNs in terms of\(D,W\)\(D,W\), the convergence rate of the smooth DNN quantile estimator scales as𝔼​‖f^n−f0‖L∞​\(Ω\)≲logn−0\.99​γ∗2\+2​γ∗\.\\mathbb\{E\}\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\lesssim\_\{\\log\}n^\{\-\\frac\{0\.99\\,\\gamma^\{\\ast\}\}\{2\+2\\gamma^\{\\ast\}\}\}\.Notably, this convergence exponent does not depend on the ambient dimensiondd, but only on the adjusted model\-complexity parameterγ∗\\gamma^\{\\ast\}\. This justifies that the smooth DNN estimator can mitigate thecurse of dimensionalityby adapting to the low\-dimensional hierarchical composition structure off0f\_\{0\}\.

### 5\.3Logistic Regression

In this subsection, we first establish uniform estimation error bounds for smooth DNN logistic regression estimators in Theorem[5\.8](https://arxiv.org/html/2606.05599#S5.Thmtheorem8)\. Building on this result, Theorem[5\.10](https://arxiv.org/html/2606.05599#S5.Thmtheorem10)further establishes uniform convergence guarantees for the density\-ratio\-based distribution estimator studied in\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]\. For*i\.i\.d\.*observations\{\(Xi,Yi\)\}i=1n\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, define the smooth DNN logistic regression estimator by

f^n∈arg⁡minf∈ℱn1n∑i=1n−Yi⋅f\(Xi\)\+log\(1\+exp\(f\(Xi\)\)\)=:1n∑i=1nℓ\(f;Xi,yi\),\\hat\{f\}\_\{n\}\\in\\mathop\{\\arg\\min\}\_\{f\\in\\mathcal\{F\}\_\{n\}\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\-Y\_\{i\}\\cdot f\(X\_\{i\}\)\+\\log\(1\+\\exp\(f\(X\_\{i\}\)\)\)=:\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(f;X\_\{i\},y\_\{i\}\),whereℱn=ℱ​\(d,1,D,W,σ;A\)∩L∞​\(M\)\\mathcal\{F\}\_\{n\}=\\mathcal\{F\}\(d,1,D,W,\\sigma;A\)\\cap L^\{\\infty\}\(M\)\. Let the target functionf0f\_\{0\}satisfy

𝔼​\[Yi\|Xi\]=exp⁡\(f0​\(Xi\)\)1\+exp⁡\(f0​\(Xi\)\)\.\\mathbb\{E\}\[Y\_\{i\}\|X\_\{i\}\]=\\frac\{\\exp\(f\_\{0\}\(X\_\{i\}\)\)\}\{1\+\\exp\(f\_\{0\}\(X\_\{i\}\)\)\}\.
As in the cases of Huber and quantile regression, convergence rates for the smooth DNN logistic regression estimator can be established below by leveraging the theoretical results developed in Section[4](https://arxiv.org/html/2606.05599#S4)\.

###### Theorem 5\.8\.

Suppose Assumptions[5\.1](https://arxiv.org/html/2606.05599#S5.Thmtheorem1),[5\.2](https://arxiv.org/html/2606.05599#S5.Thmtheorem2)hold\. Assume‖f0‖L∞​\(Ω\)≤M\\\|f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\leq M\. Denote the effective sample size asn~=n/\(D4​W6\)\\tilde\{n\}=n/\(D^\{4\}W^\{6\}\)\. Then, for anyδ∈\(0,1\)\\delta\\in\(0,1\),

ℙ​\(‖f^n−f0‖L∞​\(Ω\)≳log\(log⁡\(10/δ\)n~\+\(W​D\)−2​γ∗\)0\.99\)≤δ\.\\mathbb\{P\}\\Bigl\(\\\|\\hat\{f\}\_\{n\}\-f\_\{0\}\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\gtrsim\_\{\\log\}\\bigl\(\\sqrt\{\\frac\{\\log\(10/\\delta\)\}\{\\tilde\{n\}\}\}\+\(WD\)^\{\-2\\gamma^\{\\ast\}\}\\bigr\)^\{0\.99\}\\Bigr\)\\leq\\delta\.where the exponent0\.990\.99can be replaced by any constant in\(0,1\)\(0,1\)\.

Theorem[5\.8](https://arxiv.org/html/2606.05599#S5.Thmtheorem8)shows that the smooth DNN logistic regression estimator enjoys an estimation error bound analogous to that of the smooth DNN quantile regression estimator established in Theorem[5\.7](https://arxiv.org/html/2606.05599#S5.Thmtheorem7)\. This further supports the reliability of smooth DNN estimators across a range of regression and classification models\.

The uniform convergence guarantee established in Theorem[5\.8](https://arxiv.org/html/2606.05599#S5.Thmtheorem8)is also useful for supporting the practical application of smooth DNNs to estimating the probability of Higgs boson pair production decaying into four bottom quarks \(denoted byH​H→4​bHH\\to 4b\)\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]\. We briefly reformulate this problem within the nonparametric framework considered in\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]and then state the corresponding uniform convergence guarantee\.

In this application, the eventH​H→4​bHH\\to 4bis the signal of interest, while other physical processes producing four bottom quarks form the background\. Accurate estimation of both signal and background distributions is needed to distinguish signal from background and conduct subsequent hypothesis testing\. While the signal distribution can be approximated using physical simulation, simulating the background distribution is computationally prohibitive\. Consequently, the main problem in\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]is to estimate the background distribution from observed data contaminated by signal events\.

To this end,\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]introduced an auxiliary distribution together with density ratio estimation\. LetΩ⊆ℝd\\Omega\\subseteq\\mathbb\{R\}^\{d\}denote the state space of the observed physical quantities, and letPPandQQdenote the probability measures of the background and auxiliary distributions onΩ\\Omega, respectively\. In that work,QQcorresponds to events with three observed bottom quarks, which share similar physical characteristics with background events\. Density ratio estimation is reduced to a binary classification problem: letYYbe binary and letEEsatisfy

E∣Y=0∼Q,E∣Y=1∼P\.E\\mid Y=0\\sim Q,\\qquad E\\mid Y=1\\sim P\.Define

ψ​\(x\):=ℙ​\(Y=1∣E=x\),x∈Ω\.\\psi\(x\):=\\mathbb\{P\}\(Y=1\\mid E=x\),\\qquad x\\in\\Omega\.Then Bayes’ rule gives, whenℙ​\(Y=1\)=ℙ​\(Y=0\)=0\.5\\mathbb\{P\}\(Y=1\)=\\mathbb\{P\}\(Y=0\)=0\.5,

P​\(A\)=∫Aψ​\(y\)1−ψ​\(y\)​Q​\(d​y\),for any measurable​A⊆Ω\.P\(A\)=\\int\_\{A\}\\frac\{\\psi\(y\)\}\{1\-\\psi\(y\)\}\\,Q\(\\text\{d\}y\),\\qquad\\text\{for any measurable \}A\\subseteq\\Omega\.Hence, ifψ^\\hat\{\\psi\}is an estimator ofψ\\psiandℚN\\mathbb\{Q\}\_\{N\}is an empirical measure ofQQ, then a plug\-in estimator ofPPis

P^​\(A\)=∫Aψ^​\(y\)1−ψ^​\(y\)​ℚN​\(d​y\),for any measurable​A⊆Ω\.\\hat\{P\}\(A\)=\\int\_\{A\}\\frac\{\\hat\{\\psi\}\(y\)\}\{1\-\\hat\{\\psi\}\(y\)\}\\,\\mathbb\{Q\}\_\{N\}\(\\text\{d\}y\),\\qquad\\text\{for any measurable \}A\\subseteq\\Omega\.Similarly, ifppandqqare the densities ofPPandQQ, andq^\\hat\{q\}is an estimator ofqq, then

p^​\(x\)=ψ^​\(x\)1−ψ^​\(x\)​q^​\(x\),x∈Ω\.\\hat\{p\}\(x\)=\\frac\{\\hat\{\\psi\}\(x\)\}\{1\-\\hat\{\\psi\}\(x\)\}\\,\\hat\{q\}\(x\),\\qquad x\\in\\Omega\.In\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\], the density ratioψ​\(y\)1−ψ​\(y\)\\frac\{\\psi\(y\)\}\{1\-\\psi\(y\)\}is estimated using a SiLU ResNet tailored to the structure and symmetries of the collider data\.

A key difficulty is that the observed four\-bottom\-quark data are contaminated by signal events, so one cannot directly fit a logistic regression model on the full domain\. Following\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\], suppose

Ω=𝒳∪𝒴,𝒳∩𝒴=∅,\\Omega=\\mathcal\{X\}\\cup\\mathcal\{Y\},\\qquad\\mathcal\{X\}\\cap\\mathcal\{Y\}=\\emptyset,where signal events occur only on𝒳\\mathcal\{X\}\. Then contamination is present only on𝒳\\mathcal\{X\}, while the observed four\-bottom quark events on𝒴\\mathcal\{Y\}are purely background\. Therefore, one may estimate the classifier between background and three\-bottom quark events in𝒴\\mathcal\{Y\}, and then extrapolate it to the full domainΩ\\Omega\. Although this extrapolation assumption is statistically strong, it is reasonable in this physical setting and can be cross\-checked against an alternative optimal\-transport\-based approach\. To establish theoretical guarantees for\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\], we adopt this framework in a slightly relaxed form\.

###### Assumption 5\.9\.

LetP~\\widetilde\{P\}denote the probability measure induced by the combined distribution ofPPandQQ, and letP~𝒴\\widetilde\{P\}\_\{\\mathcal\{Y\}\}denote the restriction ofP~\\widetilde\{P\}to𝒴\\mathcal\{Y\}\. For the candidate smooth DNN classℱn\\mathcal\{F\}\_\{n\}, assume that there existsεn\>0\\varepsilon\_\{n\}\>0such thatsupf∈ℱn\|P~​ℓ​\(f\)−P~𝒴​ℓ​\(f\)\|≤εn\.\\sup\_\{f\\in\\mathcal\{F\}\_\{n\}\}\|\\widetilde\{P\}\\ell\(f\)\-\\widetilde\{P\}\_\{\\mathcal\{Y\}\}\\ell\(f\)\|\\leq\\varepsilon\_\{n\}\.

Compared with Assumption 1 in\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\], which requires the binary classifier on𝒴\\mathcal\{Y\}to coincide exactly with the binary classifier on the whole domainΩ\\Omega, the above assumption allows the two population risk minimizers to differ and quantifies the resulting discrepancy between the corresponding population loss functions throughεn\\varepsilon\_\{n\}\. In particular, whenP~\\widetilde\{P\}andP~𝒴\\widetilde\{P\}\_\{\\mathcal\{Y\}\}exhibit similar physical patterns for distinguishing four\-bottom\-quark events from three\-bottom\-quark events, one expectsεn\\varepsilon\_\{n\}to be small\.

Leveraging Assumption[5\.9](https://arxiv.org/html/2606.05599#S5.Thmtheorem9)together with the uniform convergence guarantee in Theorem[5\.8](https://arxiv.org/html/2606.05599#S5.Thmtheorem8), we establish the following result for the density\-ratio\-based distribution estimators\.

###### Theorem 5\.10\.

Under the conditions of Theorem[5\.8](https://arxiv.org/html/2606.05599#S5.Thmtheorem8), suppose in addition that Assumption[5\.9](https://arxiv.org/html/2606.05599#S5.Thmtheorem9)holds\. LetX1,…,XnX\_\{1\},\\ldots,X\_\{n\}be i\.i\.d\. observations fromP~𝒴\\widetilde\{P\}\_\{\\mathcal\{Y\}\}, and letYi=1Y\_\{i\}=1ifXiX\_\{i\}corresponds to a four\-bottom\-quark background event andYi=0Y\_\{i\}=0ifXiX\_\{i\}corresponds to a three\-bottom\-quark auxiliary event\. Letf^n\\hat\{f\}\_\{n\}be the smooth DNN logistic regression estimator based on\{\(Xi,Yi\)\}i=1n\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, and define the plug\-in estimator ofψ\\psiby

ψ^​\(x\)=exp⁡\(f^n​\(x\)\)1\+exp⁡\(f^n​\(x\)\)\.\\hat\{\\psi\}\(x\)=\\frac\{\\exp\(\\hat\{f\}\_\{n\}\(x\)\)\}\{1\+\\exp\(\\hat\{f\}\_\{n\}\(x\)\)\}\.Then, for anyδ∈\(0,1\)\\delta\\in\(0,1\), the following bounds hold with probability at least1−δ1\-\\delta:

\|P^​\(A\)−P​\(A\)\|≲log\(log⁡\(10/δ\)n~\+\(W​D\)−2​γ∗\+εn\)0\.99\+\|\(Q−ℚN\)​exp⁡\(f0​\(⋅\)\)​𝕀​\(A\)\|\|\\hat\{P\}\(A\)\-P\(A\)\|\\lesssim\_\{\\log\}\\Bigl\(\\sqrt\{\\frac\{\\log\(10/\\delta\)\}\{\\tilde\{n\}\}\}\+\(WD\)^\{\-2\\gamma^\{\\ast\}\}\+\\sqrt\{\\varepsilon\_\{n\}\}\\Bigr\)^\{0\.99\}\+\\Bigl\|\(Q\-\\mathbb\{Q\}\_\{N\}\)\\exp\(f\_\{0\}\(\\cdot\)\)\\mathbb\{I\}\(A\)\\Bigr\|for any measurableAA, and

‖p^−p‖L∞​\(Ω\)≲log\(log⁡\(10/δ\)n~\+\(W​D\)−2​γ∗\+εn\)0\.99\+‖q^−q‖L∞​\(Ω\)\.\\\|\\hat\{p\}\-p\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\lesssim\_\{\\log\}\\Bigl\(\\sqrt\{\\frac\{\\log\(10/\\delta\)\}\{\\tilde\{n\}\}\}\+\(WD\)^\{\-2\\gamma^\{\\ast\}\}\+\\sqrt\{\\varepsilon\_\{n\}\}\\Bigr\)^\{0\.99\}\+\\\|\\hat\{q\}\-q\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\.Here, the exponent0\.990\.99may be replaced by any constant in\(0,1\)\(0,1\)\.

Theorem[5\.10](https://arxiv.org/html/2606.05599#S5.Thmtheorem10)transfers the uniform convergence rate of the smooth DNN logistic regression estimator to uniform convergence guarantees for the density\-ratio\-based estimatorsP^\\hat\{P\}andp^\\hat\{p\}of the background distribution\. This provides a theoretical justification for the density\-ratio\-based method in\[[31](https://arxiv.org/html/2606.05599#bib.bib47)\]under the relaxed extrapolation condition in Assumption[5\.9](https://arxiv.org/html/2606.05599#S5.Thmtheorem9)\.

In particular, the theorem yields non\-asymptotic estimation error bounds for bothP^\\hat\{P\}andp^\\hat\{p\}\. Each bound consists of four components: the statistical error termlog⁡\(10/δ\)/n~\\sqrt\{\\log\(10/\\delta\)/\\tilde\{n\}\}, the approximation error term\(W​D\)−2​γ∗\(WD\)^\{\-2\\gamma^\{\\ast\}\}, the extrapolation errorεn\\varepsilon\_\{n\}, and the estimation error of the auxiliary distribution, represented by\|\(Q−ℚN\)​exp⁡\(f0​\(⋅\)\)​𝕀​\(A\)\|\|\(Q\-\\mathbb\{Q\}\_\{N\}\)\\exp\(f\_\{0\}\(\\cdot\)\)\\mathbb\{I\}\(A\)\|and‖q^−q‖L∞​\(Ω\)\\\|\\hat\{q\}\-q\\\|\_\{L^\{\\infty\}\(\\Omega\)\}, respectively\. The first two terms arise from smooth DNN logistic regression, while the termεn\\varepsilon\_\{n\}quantifies the discrepancy introduced by extrapolation, which is expected to be small in this physical setting\. Moreover, when the auxiliary distribution can be simulated, the last term is typically negligible in this setting\. Besides, if a collection of measurable sets𝒜\\mathcal\{A\}is sufficiently regular so that

supA∈𝒜\|\(Q−ℚN\)​exp⁡\(f0​\(⋅\)\)​𝕀​\(A\)\|→0as​N→∞,\\sup\_\{A\\in\\mathcal\{A\}\}\\Bigl\|\(Q\-\\mathbb\{Q\}\_\{N\}\)\\exp\(f\_\{0\}\(\\cdot\)\)\\mathbb\{I\}\(A\)\\Bigr\|\\to 0\\qquad\\text\{as \}N\\to\\infty,and if‖q^−q‖L∞​\(Ω\)→0\\\|\\hat\{q\}\-q\\\|\_\{L^\{\\infty\}\(\\Omega\)\}\\to 0\(e\.g\., permitted by kernel density estimation\[[22](https://arxiv.org/html/2606.05599#bib.bib51)\]\), the auxiliary\-distribution estimation error can be made asymptotically negligible for sufficiently largeNN\.

## 6Numerical Experiments

Section[6\.1](https://arxiv.org/html/2606.05599#S6.SS1)benchmarks SiLU FNN and ResNet Huber estimators against ReLU FNN estimators through simulation studies\. In Section[6\.2](https://arxiv.org/html/2606.05599#S6.SS2), we apply SiLU ResNet quantile estimators to analyze the temperature sensitivity of ozone concentrations across regions\. All experiments were conducted on a computer running Ubuntu 24\.04\.2 LTS with an AMD EPYC 7532 CPU, 512 GB RAM, and an NVIDIA RTX 3090 GPU\.111The Python code used to reproduce both the simulation and real\-data experiments is available in an anonymized GitHub repository at[https://anonymous\.4open\.science/r/Uniform\_convergence\-DF22/](https://anonymous.4open.science/r/Uniform_convergence-DF22/)\.

### 6\.1Simulation Studies

In this subsection, we compare the numerical performance of SiLU DNN and ReLU FNN estimators for Huber regression\. We consider*i\.i\.d\.*samples\{\(Xi,Yi\)\}i=1n\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}generated fromYi=f0​\(Xi\)\+g​\(Xi\)​εi,Y\_\{i\}=f\_\{0\}\(X\_\{i\}\)\+g\(X\_\{i\}\)\\,\\varepsilon\_\{i\},where the covariateXi∈ℝ8X\_\{i\}\\in\\mathbb\{R\}^\{8\}is sampled uniformly from\[0,1\]8\[0,1\]^\{8\}, andεi\\varepsilon\_\{i\}is drawn from a Student’sttdistribution with either 2 or 4 degrees of freedom \(t2t\_\{2\}ort4t\_\{4\}\)\. Following\[[49](https://arxiv.org/html/2606.05599#bib.bib39)\]to induce noise, the target and scale functions are defined as

f0​\(x\)=\\displaystyle f\_\{0\}\(x\)=cos⁡\(2​π​x1\)\+11\+exp⁡\(−x2−x3\)\+1\(1\+x4\+x5\)3\+1x6\+exp⁡\(x7​x8\),\\displaystyle\\cos\(2\\pi x\_\{1\}\)\+\\frac\{1\}\{1\+\\exp\(\-x\_\{2\}\-x\_\{3\}\)\}\+\\frac\{1\}\{\(1\+x\_\{4\}\+x\_\{5\}\)^\{3\}\}\+\\frac\{1\}\{x\_\{6\}\+\\exp\(x\_\{7\}x\_\{8\}\)\},g​\(x\)=\\displaystyle g\(x\)=sin⁡\(π​\(x1\+x2\)2\)\+log⁡\(1\+x32​x42​x52\)\+x81\+exp⁡\(−x6−x7\)\.\\displaystyle\\sin\\\!\\Big\(\\frac\{\\pi\(x\_\{1\}\+x\_\{2\}\)\}\{2\}\\Big\)\+\\log\\\!\\big\(1\+x\_\{3\}^\{2\}x\_\{4\}^\{2\}x\_\{5\}^\{2\}\\big\)\+\\frac\{x\_\{8\}\}\{1\+\\exp\(\-x\_\{6\}\-x\_\{7\}\)\}\.
For the Huber estimator in \([15](https://arxiv.org/html/2606.05599#S5.E15)\), we set the Huber parameter toτ∈\{0\.25,0\.5,1,2,4\}\\tau\\in\\\{0\.25,0\.5,1,2,4\\\}\. The depth and width of the ReLU FNNs, SiLU FNNs, and SiLU ResNets are all set to44\. For the SiLU ResNets, we takeℒ1\\mathcal\{L\}\_\{1\}andℒ3\\mathcal\{L\}\_\{3\}in Definition[3\.2](https://arxiv.org/html/2606.05599#S3.Thmtheorem2)to be identity maps so that all architectures have matched parameter sizes\.

Table 1:L2​\(P\)L^\{2\}\(P\)andL∞​\(\[0,1\]8\)L^\{\\infty\}\(\[0,1\]^\{8\}\)estimation errors of the Huber estimators undert2t\_\{2\}noise, with estimated convergence exponents and bootstrap standard deviations \(shown in parentheses\)\.n=512n=512n=1024n=1024n=2048n=2048Estimated Convergence Exponentsτ\\tauModelL2L^\{2\}L∞L^\{\\infty\}L2L^\{2\}L∞L^\{\\infty\}L2L^\{2\}L∞L^\{\\infty\}L2L^\{2\}L∞L^\{\\infty\}0\.25ReLU FNN0\.69382\.24630\.48811\.95250\.35151\.60190\.4905 \(0\.0473\)0\.2439 \(0\.0424\)SiLU FNN0\.72561\.95040\.56581\.83610\.29101\.32850\.6591 \(0\.0424\)0\.2770 \(0\.0400\)SiLU ResNet0\.62182\.18070\.42271\.60110\.24061\.08360\.6848 \(0\.0378\)0\.5045 \(0\.0333\)0\.5ReLU FNN0\.70012\.27300\.49201\.98120\.34651\.61720\.5073 \(0\.0472\)0\.2456 \(0\.0437\)SiLU FNN0\.72381\.94690\.56281\.81290\.29111\.30840\.6571 \(0\.0438\)0\.2867 \(0\.0353\)SiLU ResNet0\.61712\.16390\.41991\.58510\.23991\.07590\.6815 \(0\.0395\)0\.5040 \(0\.0344\)1\.0ReLU FNN0\.69842\.21950\.48521\.98040\.34591\.63150\.5070 \(0\.0465\)0\.2220 \(0\.0427\)SiLU FNN0\.72361\.92910\.56881\.83450\.29421\.30300\.6491 \(0\.0439\)0\.2830 \(0\.0367\)SiLU ResNet0\.61282\.15170\.41461\.57280\.24061\.10730\.6743 \(0\.0394\)0\.4792 \(0\.0349\)2\.0ReLU FNN0\.71702\.23580\.51812\.04490\.35091\.71480\.5154 \(0\.0423\)0\.1914 \(0\.0461\)SiLU FNN0\.73171\.93660\.59851\.88900\.32101\.38100\.5944 \(0\.0420\)0\.2439 \(0\.0346\)SiLU ResNet0\.63552\.20530\.44971\.63710\.25121\.15180\.6694 \(0\.0337\)0\.4686 \(0\.0352\)4\.0ReLU FNN0\.74232\.45420\.57442\.44120\.39511\.83250\.4549 \(0\.0366\)0\.2107 \(0\.0575\)SiLU FNN0\.74892\.09550\.63352\.01120\.34581\.53450\.5573 \(0\.0371\)0\.2248 \(0\.0367\)SiLU ResNet0\.67462\.41250\.50251\.79400\.30241\.33920\.5789 \(0\.0338\)0\.4246 \(0\.0377\)

Table 2:L2​\(P\)L^\{2\}\(P\)andL∞​\(\[0,1\]8\)L^\{\\infty\}\(\[0,1\]^\{8\}\)estimation errors of the Huber estimators undert4t\_\{4\}noise, with estimated convergence exponents and bootstrap standard deviations \(shown in parentheses\)\.n=512n=512n=1024n=1024n=2048n=2048Estimated Convergence Exponentsτ\\tauModelL2L^\{2\}L∞L^\{\\infty\}L2L^\{2\}L∞L^\{\\infty\}L2L^\{2\}L∞L^\{\\infty\}L2L^\{2\}L∞L^\{\\infty\}0\.25ReLU FNN0\.69902\.15020\.47971\.77000\.29421\.42060\.6242 \(0\.0477\)0\.2990 \(0\.0380\)SiLU FNN0\.72641\.97610\.54811\.71150\.27951\.18680\.6891 \(0\.0453\)0\.3678 \(0\.0293\)SiLU ResNet0\.62422\.02730\.39361\.47150\.22061\.03430\.7502 \(0\.0355\)0\.4854 \(0\.0271\)0\.5ReLU FNN0\.69222\.10360\.47191\.80290\.29301\.45620\.6202 \(0\.0497\)0\.2653 \(0\.0384\)SiLU FNN0\.72381\.96010\.54241\.68970\.27181\.21170\.7066 \(0\.0448\)0\.3469 \(0\.0362\)SiLU ResNet0\.62582\.01860\.38621\.45410\.21731\.01770\.7632 \(0\.0358\)0\.4941 \(0\.0279\)1\.0ReLU FNN0\.69212\.09360\.47301\.78850\.28881\.50780\.6304 \(0\.0486\)0\.2368 \(0\.0416\)SiLU FNN0\.72261\.92420\.54831\.61630\.25841\.18100\.7418 \(0\.0414\)0\.3521 \(0\.0352\)SiLU ResNet0\.62482\.00700\.38821\.43370\.21661\.00550\.7642 \(0\.0358\)0\.4985 \(0\.0290\)2\.0ReLU FNN0\.70742\.09760\.48101\.79260\.31821\.58940\.5763 \(0\.0487\)0\.2001 \(0\.0417\)SiLU FNN0\.72521\.89670\.56761\.63640\.26871\.19160\.7161 \(0\.0398\)0\.3353 \(0\.0289\)SiLU ResNet0\.63011\.96980\.41151\.47320\.23221\.03220\.7202 \(0\.0387\)0\.4662 \(0\.0252\)4\.0ReLU FNN0\.72362\.12930\.52361\.86360\.33411\.65460\.5575 \(0\.0455\)0\.1819 \(0\.0413\)SiLU FNN0\.73171\.91230\.58941\.72040\.29141\.25310\.6642 \(0\.0386\)0\.3049 \(0\.0250\)SiLU ResNet0\.63512\.02730\.43981\.60760\.26041\.13790\.6433 \(0\.0397\)0\.4166 \(0\.0256\)

![Refer to caption](https://arxiv.org/html/2606.05599v1/x3.png)Figure 3:Side\-by\-side boxplots of theL2​\(P\)L^\{2\}\(P\)estimation errors\.![Refer to caption](https://arxiv.org/html/2606.05599v1/x4.png)Figure 4:Side\-by\-side boxplots of theL∞L^\{\\infty\}estimation errors\.We consider sample sizesn∈\{512,1024,2048\}n\\in\\\{512,1024,2048\\\}and replicate each setting 100 times\. For evaluation, given an estimatorf^n\\hat\{f\}\_\{n\}, we independently draw\{Xj\}j=110000\\\{X\_\{j\}\\\}\_\{j=1\}^\{10000\}uniformly from\[0,1\]8\[0,1\]^\{8\}and compute the pointwise errors\{\|f^n​\(Xj\)−f0​\(Xj\)\|\}j=110000\\\{\|\\hat\{f\}\_\{n\}\(X\_\{j\}\)\-f\_\{0\}\(X\_\{j\}\)\|\\\}\_\{j=1\}^\{10000\}\. Based on these errors, we report the estimatedL2L^\{2\}andL∞L^\{\\infty\}errors in Tables[1](https://arxiv.org/html/2606.05599#S6.T1)and[2](https://arxiv.org/html/2606.05599#S6.T2)undert2t\_\{2\}andt4t\_\{4\}noise, respectively\. The corresponding side\-by\-side boxplots for theL2L^\{2\}andL∞L^\{\\infty\}metrics are presented in Figures[3](https://arxiv.org/html/2606.05599#S6.F3)and[4](https://arxiv.org/html/2606.05599#S6.F4), respectively\. As shown in Tables[1](https://arxiv.org/html/2606.05599#S6.T1)–[2](https://arxiv.org/html/2606.05599#S6.T2)and Figures[3](https://arxiv.org/html/2606.05599#S6.F3)–[4](https://arxiv.org/html/2606.05599#S6.F4), the SiLU FNN and ResNet estimators consistently outperform the ReLU FNN estimator across all settings\. Moreover, SiLU ResNet estimators exhibit smaller estimation error compared to SiLU FNN estimators, which is consistent with the optimization advantages of residual architectures\. To further assess the statistical estimation performance, we additionally report the estimated convergence exponents\. Specifically, under the scaling lawn−bn^\{\-b\}for the estimation error, we estimatebbvia log–log regression and report the results in Tables[1](https://arxiv.org/html/2606.05599#S6.T1)and[2](https://arxiv.org/html/2606.05599#S6.T2), together with bootstrap standard deviations based on 1000 resamples\. Overall, while all estimators yield comparable exponents for theL2L^\{2\}norm, ReLU FNNs have much smaller uniform convergence exponents\. This empirical pattern is consistent with our theory: ReLU FNNs are not well suited to tasks requiring uniform convergence\.

### 6\.2Real Data Analysis: Temperature Sensitivity of Ozone

Ozone is widely recognized as an air pollutant that affects human health\[[38](https://arxiv.org/html/2606.05599#bib.bib44)\]\. The IPCC Sixth Assessment Report assesses that climate\-driven changes in meteorology can influence ozone concentration, and that shifts in the geographical distribution of emissions have contributed to strong regional variations in ozone trends\[[43](https://arxiv.org/html/2606.05599#bib.bib41)\]\. These assessments motivate us to study the relationship between temperature and ozone concentrations over the contiguous United States \(CONUS\) using quantile regression\. From a methodological perspective, uniform convergence is important here because it supports reliable estimation uniformly over the spatial domain, rather than only in an average sense\.

Our response is the daily maximum 8\-hour average ozone concentration, measured in parts per billion \(ppb\), obtained from the U\.S\. EPA Air Quality System \(AQS\)\. The covariates include latitude, longitude, year, day\-of\-year, and daily maximum and minimum near\-surface temperatures \(tmmx, tmmn\) in∘C at AQS monitoring sites\. The temperature covariates are obtained from the gridMET dataset\[[1](https://arxiv.org/html/2606.05599#bib.bib42)\]and mapped to each monitoring site via nearest\-neighbor assignment to the closest gridMET grid cell\. We construct the dataset using daily observations from the warm season \(May–September\) over the years 2015–2024, when photochemical ozone formation is most active\. The resulting dataset contains approximately 1\.84 million observations from about 1,500 monitoring sites\.

For this quantile regression task, we consider a SiLU ResNet and a ReLU FNN, both with depth 6 and width 128, and train them in PyTorch using the Adam optimizer for 200 epochs\. As in Section[6\.1](https://arxiv.org/html/2606.05599#S6.SS1), we takeℒ1\\mathcal\{L\}\_\{1\}andℒ3\\mathcal\{L\}\_\{3\}for the SiLU RNs in Definition[3\.2](https://arxiv.org/html/2606.05599#S3.Thmtheorem2)to be identity maps so that the two architectures have comparable parameter sizes\. We fit conditional quantiles at levelsτ∈\{0\.10,0\.30,0\.50,0\.87\}\\tau\\in\\\{0\.10,0\.30,0\.50,0\.87\\\}, following the practice in ozone studies\[[39](https://arxiv.org/html/2606.05599#bib.bib40)\], since these quantiles admit meaningful practical interpretations\. Specifically, theτ=0\.10\\tau=0\.10quantile reflects relatively clean\-condition days, theτ=0\.30\\tau=0\.30quantile serves as a robust proxy for the photochemical baseline, theτ=0\.50\\tau=0\.50quantile summarizes the typical ozone level, and theτ=0\.87\\tau=0\.87quantile provides an upper\-tail summary of broadly elevated ozone days during the warm season\.

We then perform a counterfactual perturbation analysis using the trained models\. Specifically, using temperatures from the warm season of 2025, we construct perturbed covariates by adding a constant warming incrementΔ∈\{0\.5,1\.0,1\.5\}\\Delta\\in\\\{0\.5,1\.0,1\.5\\\}to both tmmx and tmmn, while holding the remaining covariates fixed\. Using the original and perturbed covariates, we compute predicted daily ozone concentrations and report their differences over the CONUS at 20km resolution\. Averaging the predictions over the warm season, Figure[5](https://arxiv.org/html/2606.05599#S6.F5)presents the counterfactual differences in ozone concentrations across quantiles predicted by the SiLU ResNets over the CONUS\.

![Refer to caption](https://arxiv.org/html/2606.05599v1/x5.png)Figure 5:Overall counterfactual difference by SiLU ResNets in ozone concentration\.Figure[5](https://arxiv.org/html/2606.05599#S6.F5)reveals a spatially heterogeneous temperature sensitivity of ozone across the CONUS\. Across all four quantile levels, warming induces broadly positive changes over the northern tier of the United States and California\. These patterns are consistent with prior studies based on classical models\[[39](https://arxiv.org/html/2606.05599#bib.bib40),[40](https://arxiv.org/html/2606.05599#bib.bib45)\]\. In contrast, ozone decreases over parts of the South and the Gulf Coast, consistent with prior evidence that, over tropical and subtropical oceans, increases in temperature and water vapor can reduce photochemical ozone production\[[23](https://arxiv.org/html/2606.05599#bib.bib43)\]\. Taken together, these counterfactual patterns suggest that the SiLU ResNet estimators capture climate\-driven and geographically heterogeneous temperature–ozone relationships\.

To further compare the counterfactual prediction performance of the ReLU FNN and SiLU ResNet estimators, we complement the visual inspection with two quantitative diagnostics\. The is*Spatial Roughness*\(SR\), measures the local irregularity of a rasterized counterfactual ozone\-change surface through first\-order finite differences; larger values indicate less spatial smoothness and more abrupt local variation\. The second is*Extreme Value Fraction*\(EVF\), defined as the fraction of grid cells with extremely large values\. This diagnostic measures the tendency of a fitted model to generate excessive local extremes\. For each model, Table[3](https://arxiv.org/html/2606.05599#S6.T3)reports the average diagnostic value over all combinations of quantile levelτ\\tauand warming incrementΔ\\Deltawithin each month, as well as over the full warm season for the overall summary\.

More precisely, letDτ,Δ,t​\(s\)D\_\{\\tau,\\Delta,t\}\(s\)denote the daily counterfactual ozone change at grid cellss, for quantile levelτ\\tau, warming incrementΔ\\Delta, and datett\. For each monthmm, and for the overall warm season, we first averageDτ,Δ,t​\(s\)D\_\{\\tau,\\Delta,t\}\(s\)over time to obtain a mean counterfactual fieldZ​\(s\)Z\(s\)\. LetZZbe a rasterized counterfactual ozone\-change field on a two\-dimensional grid, represented by the array\(Zi,j\)i,j\(Z\_\{i,j\}\)\_\{i,j\}\. Let𝒢⊆ℕ\+×ℕ\+\\mathcal\{G\}\\subseteq\\mathbb\{N\}\_\{\+\}\\times\\mathbb\{N\}\_\{\+\}denote the set of valid grid\-cell indices, and letℋ,𝒱⊆ℕ\+×ℕ\+\\mathcal\{H\},\\mathcal\{V\}\\subseteq\\mathbb\{N\}\_\{\+\}\\times\\mathbb\{N\}\_\{\+\}denote the sets of valid grid cells whose horizontal and vertical neighbors are also valid, respectively, namely,

\(i,j\)∈ℋ​if​\(i,j\)∈𝒢​and​\(i,j\+1\)∈𝒢;\(i,j\)∈𝒱​if​\(i,j\)∈𝒢​and​\(i\+1,j\)∈𝒢\.\(i,j\)\\in\\mathcal\{H\}\\text\{ if \}\(i,j\)\\in\\mathcal\{G\}\\text\{ and \}\(i,j\+1\)\\in\\mathcal\{G\};\\qquad\(i,j\)\\in\\mathcal\{V\}\\text\{ if \}\(i,j\)\\in\\mathcal\{G\}\\text\{ and \}\(i\+1,j\)\\in\\mathcal\{G\}\.We define the spatial roughness ofZZby

SR​\(Z\)=∑\(i,j\)∈ℋ\|Zi,j\+1−Zi,j\|\+∑\(i,j\)∈𝒱\|Zi\+1,j−Zi,j\|\|ℋ\|\+\|𝒱\|,\\mathrm\{SR\}\(Z\)=\\frac\{\\sum\_\{\(i,j\)\\in\\mathcal\{H\}\}\|Z\_\{i,j\+1\}\-Z\_\{i,j\}\|\+\\sum\_\{\(i,j\)\\in\\mathcal\{V\}\}\|Z\_\{i\+1,j\}\-Z\_\{i,j\}\|\}\{\|\\mathcal\{H\}\|\+\|\\mathcal\{V\}\|\},that is, the mean absolute difference over all valid horizontal and vertical neighboring pairs, and

EVF​\(Z;u\)=1\|𝒢\|​∑s∈𝒢𝟏​\{\|Z​\(s\)\|\>u\},\\mathrm\{EVF\}\(Z;u\)=\\frac\{1\}\{\|\\mathcal\{G\}\|\}\\sum\_\{s\\in\\mathcal\{G\}\}\\mathbf\{1\}\\\{\|Z\(s\)\|\>u\\\},with thresholdu=3\.0u=3\.0ppb\. For each model and each month, we average these diagnostics over all quantile levelsτ∈\{0\.10,0\.30,0\.50,0\.87\}\\tau\\in\\\{0\.10,0\.30,0\.50,0\.87\\\}and warming incrementsΔ∈\{0\.5,1\.0,1\.5\}\\Delta\\in\\\{0\.5,1\.0,1\.5\\\}\. Standard deviations are computed using a month\-stratified paired bootstrap over dates with 1000 resamples\.

Table 3:Diagnostics for the counterfactual ozone\-change predictions of the ReLU FNNs and SiLU ResNets, with bootstrap standard deviations in parentheses\.MetricSREVFModelReLU FNNSiLU RNReLU FNNSiLU RNMay0\.1174 \(0\.0156\)0\.0705 \(0\.0069\)0\.0011 \(0\.0022\)0\.0007 \(0\.0008\)June0\.1564 \(0\.0181\)0\.0768 \(0\.0063\)0\.0137 \(0\.0039\)0\.0020 \(0\.0029\)July0\.1589 \(0\.0165\)0\.0918 \(0\.0054\)0\.0227 \(0\.0060\)0\.0097 \(0\.0041\)August0\.1448 \(0\.0177\)0\.0804 \(0\.0070\)0\.0107 \(0\.0030\)0\.0108 \(0\.0018\)September0\.1409 \(0\.0194\)0\.0759 \(0\.0066\)0\.0085 \(0\.0043\)0\.0047 \(0\.0022\)Overall0\.0813 \(0\.0037\)0\.0555 \(0\.0017\)0\.0012 \(0\.0008\)0\.0004 \(0\.0006\)

As shown in Table[3](https://arxiv.org/html/2606.05599#S6.T3), the SiLU ResNet yields uniformly smaller SR values than the ReLU FNN, indicating more spatially coherent counterfactual predictions\. For EVF, the SiLU ResNet also attains smaller values in most months and in the overall summary\. In addition, the SiLU ResNet exhibits smaller bootstrap standard deviations for both diagnostics in all reported cases\. Taken together, these results suggest that the SiLU ResNet produces more stable and spatially coherent quantile predictions, while generating fewer potentially spurious extreme local responses in this application\.

## 7Conclusion

In this work, we investigate the slow uniform convergence of ReLU FNN estimators and develop a statistical theory for smooth DNNs, an alternative class that aligns with modern deep learning practice while enjoying uniform convergence guarantees in many statistical learning tasks\. We first show that, despite minimax\-optimalL2​\(P\)L^\{2\}\(P\)convergence, least\-squares ReLU FNN estimators can suffer from thecurse of dimensionalityin uniform convergence, owing to the low smoothness of the ReLU activation\. Motivated by the prevalence of residual architectures andC∞C^\{\\infty\}activations in state\-of\-the\-art deep learning models, we introduce smooth DNNs and establish their core statistical properties, including pseudo\-dimension bounds, approximation error bounds for Sobolev functions and hierarchical composition models, and factorial\-type Hölder norm bounds for approximators\. Building on these foundations, we derive non\-asymptoticL∞L^\{\\infty\}convergence rates for smooth DNN estimators in several representative tasks: Huber, least\-squares, quantile, and logistic regression, showing that smooth DNNs can provide reliable uniform convergence guarantees while adapting to low\-dimensional hierarchical structure\. Together, these results position smooth DNNs as a theoretically grounded alternative to ReLU FNN estimators in settings where uniform convergence is essential\.

## References

- \[1\]J\. T\. Abatzoglou\(2013\)Development of gridded surface meteorological data for ecological applications and modelling\.International Journal of Climatology33\(1\),pp\. 121–131\.Cited by:[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p2.1)\.
- \[2\]M\. Anthony and P\. L\. Bartlett\(2009\)Neural network learning: theoretical foundations\.Cambridge University Press\.Cited by:[§4\.1](https://arxiv.org/html/2606.05599#S4.SS1.p1.2),[Definition 4\.3](https://arxiv.org/html/2606.05599#S4.Thmtheorem3)\.
- \[3\]P\. L\. Bartlett, N\. Harvey, C\. Liaw, and A\. Mehrabian\(2019\)Nearly\-tight vc\-dimension and pseudodimension bounds for piecewise linear neural networks\.Journal of Machine Learning Research20\(63\),pp\. 1–17\.Cited by:[§4\.1](https://arxiv.org/html/2606.05599#S4.SS1.p1.2)\.
- \[4\]B\. Bauer and M\. Kohler\(2019\)On deep learning as a remedy for the curse of dimensionality in nonparametric regression\.The Annals of Statistics47\(4\),pp\. 2261 – 2285\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p4.1),[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p7.2)\.
- \[5\]D\. Belomestny, A\. Naumov, N\. Puchkin, and S\. Samsonov\(2023\)Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise\-polynomial activations\.Neural Networks161,pp\. 242–253\.Cited by:[2nd item](https://arxiv.org/html/2606.05599#S1.I1.i2.p1.1)\.
- \[6\]S\. Bhattacharya, J\. Fan, and D\. Mukherjee\(2024\)Deep neural networks for nonparametric interaction models with diverging dimension\.The Annals of Statistics52\(6\),pp\. 2738–2766\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p1.1)\.
- \[7\]H\. Brezis and P\. Mironescu\(2018\)Gagliardo–nirenberg inequalities and non\-inequalities: the full story\.Annales de l’Institut Henri Poincaré C, Analyse non linéaire35\(5\),pp\. 1355–1376\.Cited by:[§2\.2](https://arxiv.org/html/2606.05599#S2.SS2.p3.1)\.
- \[8\]T\. De Ryck, S\. Lanthaler, and S\. Mishra\(2021\)On the approximation of functions by tanh neural networks\.Neural Networks143,pp\. 732–750\.Cited by:[2nd item](https://arxiv.org/html/2606.05599#S1.I1.i2.p1.1)\.
- \[9\]Y\. Ding, R\. Li, and L\. Xue\(2025\)New empirical process tools and their applications to robust deep relu networks and phase transitions for nonparametric regression\.arXiv preprint arXiv:2511\.15841\.Cited by:[3rd item](https://arxiv.org/html/2606.05599#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2606.05599#S1.p1.1),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p2.5),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p3.4),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p6.11),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p7.4),[§5\.2](https://arxiv.org/html/2606.05599#S5.SS2.p1.10)\.
- \[10\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2)\.
- \[11\]S\. Elfwing, E\. Uchibe, and K\. Doya\(2018\)Sigmoid\-weighted linear units for neural network function approximation in reinforcement learning\.Neurocomputing275,pp\. 2166–2173\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p3.4)\.
- \[12\]J\. Fan, Y\. Gu, and W\. Zhou\(2024\)How do noise tails impact on deep relu networks?\.The Annals of Statistics52\(4\),pp\. 1845–1871\.Cited by:[1st item](https://arxiv.org/html/2606.05599#S1.I1.i1.p1.5),[3rd item](https://arxiv.org/html/2606.05599#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2606.05599#S1.p1.1),[§1](https://arxiv.org/html/2606.05599#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.05599#S2.SS1.p6.5),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p2.5),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p3.4),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p6.11),[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p7.4)\.
- \[13\]J\. Fan and Y\. Gu\(2024\)Factor augmented sparse throughput deep relu neural networks for high dimensional regression\.Journal of the American Statistical Association119\(548\),pp\. 2680–2694\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p1.1)\.
- \[14\]E\. Gagliardo\(1959\)Ulteriori proprietà di alcune classi di funzioni in più variabili\.Ricerche Mat\.8,pp\. 24–51\.Cited by:[§2\.2](https://arxiv.org/html/2606.05599#S2.SS2.p3.1)\.
- \[15\]P\. Goldberg and M\. Jerrum\(1993\)Bounding the vapnik\-chervonenkis dimension of concept classes parameterized by real numbers\.InProceedings of the sixth Annual Conference on Computational Learning Theory,pp\. 361–369\.Cited by:[§4\.1](https://arxiv.org/html/2606.05599#S4.SS1.p1.2)\.
- \[16\]W\. Guo, X\. Zhou, and S\. Ma\(2021\)Estimation of optimal individualized treatment rules using a covariate\-specific treatment effect curve with high\-dimensional covariates\.Journal of the American Statistical Association116\(533\),pp\. 309–321\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p2.1)\.
- \[17\]Q\. Han and J\. A\. Wellner\(2019\)Convergence rates of least squares regression estimators with heavy\-tailed errors\.The Annals of Statistics47\(4\),pp\. 2286 – 2319\.Cited by:[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p2.5)\.
- \[18\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p4.1),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p5.5)\.
- \[19\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Identity mappings in deep residual networks\.InEuropean Conference on Computer Vision \(ECCV\),pp\. 630–645\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p5.5)\.
- \[20\]D\. Hendrycks and K\. Gimpel\(2016\)Gaussian error linear units \(gelus\)\.arXiv preprint arXiv:1606\.08415\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p3.4)\.
- \[21\]M\. Imaizumi\(2023\)Sup\-norm convergence of deep neural network estimator for nonparametric regression by adversarial training\.arXiv preprint arXiv:2307\.04042\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p2.1)\.
- \[22\]H\. Jiang\(2017\)Uniform convergence rates for kernel density estimation\.InInternational Conference on Machine Learning,pp\. 1694–1703\.Cited by:[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p11.11)\.
- \[23\]C\. Johnson, W\. Collins, D\. Stevenson, and R\. Derwent\(1999\)Relative roles of climate and emissions changes on future tropospheric oxidant concentrations\.Journal of Geophysical Research: Atmospheres104\(D15\),pp\. 18631–18645\.Cited by:[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p5.1)\.
- \[24\]M\. Karpinski and A\. Macintyre\(1997\)Polynomial bounds for vc dimension of sigmoidal and general pfaffian neural networks\.Journal of Computer and System Sciences54\(1\),pp\. 169–176\.Cited by:[§4\.1](https://arxiv.org/html/2606.05599#S4.SS1.p1.2)\.
- \[25\]R\. Koenker\(2005\)Quantile regression\.Vol\.38,Cambridge University Press\.Cited by:[§5\.2](https://arxiv.org/html/2606.05599#S5.SS2.p1.8)\.
- \[26\]M\. Kohler and S\. Langer\(2021\)On the rate of convergence of fully connected deep neural network regression estimates\.The Annals of Statistics49\(4\),pp\. 2231–2249\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p4.1),[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p7.2)\.
- \[27\]A\. K\. Kuchibhotla and R\. K\. Patra\(2022\)On least squares estimation under heteroscedastic and heavy\-tailed errors\.The Annals of Statistics50\(1\),pp\. 277–302\.Cited by:[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p2.5)\.
- \[28\]H\. Liu, M\. Chen, S\. Er, W\. Liao, T\. Zhang, and T\. Zhao\(2022\)Benefits of overparameterized convolutional residual networks: function approximation under smoothness constraint\.InInternational Conference on Machine Learning,pp\. 13669–13703\.Cited by:[2nd item](https://arxiv.org/html/2606.05599#S1.I1.i2.p1.1)\.
- \[29\]H\. Liu, M\. Chen, T\. Zhao, and W\. Liao\(2021\)Besov function approximation and binary classification on low\-dimensional manifolds using convolutional residual networks\.InInternational Conference on Machine Learning,pp\. 6770–6780\.Cited by:[2nd item](https://arxiv.org/html/2606.05599#S1.I1.i2.p1.1)\.
- \[30\]J\. Lu, Z\. Shen, H\. Yang, and S\. Zhang\(2021\)Deep network approximation for smooth functions\.SIAM Journal on Mathematical Analysis53\(5\),pp\. 5465–5506\.Cited by:[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p2.1)\.
- \[31\]T\. Manole, P\. Bryant, J\. Alison, M\. Kuusela, and L\. Wasserman\(2024\)Background modeling for double higgs boson production: density ratios and optimal transport\.The Annals of Applied Statistics18\(4\),pp\. 2950–2978\.Cited by:[3rd item](https://arxiv.org/html/2606.05599#S1.I1.i3.p1.1),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p1.1),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p10.2),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p4.1),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p5.1),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p6.20),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p6.7),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p7.5),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p7.6),[§5\.3](https://arxiv.org/html/2606.05599#S5.SS3.p8.6)\.
- \[32\]D\. Misra\(2020\)Mish: a self regularized non\-monotonic neural activation function\.InProceedings of the British Machine Vision Conference 2020,Cited by:[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p3.4)\.
- \[33\]L\. Nirenberg\(1966\)An extended interpolation inequality\.Annali della Scuola Normale Superiore di Pisa\-Scienze Fisiche e Matematiche20\(4\),pp\. 733–737\.Cited by:[§2\.2](https://arxiv.org/html/2606.05599#S2.SS2.p3.1)\.
- \[34\]K\. Oono and T\. Suzuki\(2019\)Approximation and non\-parametric estimation of resnet\-type convolutional neural networks\.InInternational Conference on Machine Learning,pp\. 4922–4931\.Cited by:[2nd item](https://arxiv.org/html/2606.05599#S1.I1.i2.p1.1)\.
- \[35\]P\. Ramachandran, B\. Zoph, and Q\. V\. Le\(2017\)Searching for activation functions\.arXiv preprint arXiv:1710\.05941\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p3.4)\.
- \[36\]J\. Schmidt\-Hieber and P\. Zamolodtchikov\(2024\)Local convergence rates of the nonparametric least squares estimator with applications to transfer learning\.Bernoulli30\(3\),pp\. 1845–1877\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p2.1)\.
- \[37\]J\. Schmidt\-Hieber\(2020\)Nonparametric regression using deep neural networks with relu activation function\.The Annals of Statistics48\(4\),pp\. 1875 – 1897\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.05599#S3.SS1.p4.1),[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p7.2)\.
- \[38\]J\. L\. Schnell and M\. J\. Prather\(2017\)Co\-occurrence of extremes in surface ozone, particulate matter, and temperature over eastern north america\.Proceedings of the National Academy of Sciences114\(11\),pp\. 2854–2859\.Cited by:[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p1.1)\.
- \[39\]J\. L\. Schnell, M\. J\. Prather, B\. Josse, V\. Naik, L\. W\. Horowitz, G\. Zeng, D\. T\. Shindell, G\. Faluvegi, P\. Stegmann, G\. A\. Folberth,et al\.\(2016\)Effect of climate change on surface ozone over north america, europe, and east asia\.Geophysical Research Letters43\(7\),pp\. 3509–3518\.Cited by:[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p3.7),[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p5.1)\.
- \[40\]L\. Schwarz, K\. Hansen, A\. Alari, S\. D\. Ilango, N\. Bernal, R\. Basu, A\. Gershunov, and T\. Benmarhnia\(2021\)Spatial variation in the joint effect of extreme heat events and ozone on respiratory hospitalizations in california\.Proceedings of the National Academy of Sciences118\(22\),pp\. e2023078118\.Cited by:[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p5.1)\.
- \[41\]G\. Shen, Y\. Jiao, Y\. Lin, and J\. Huang\(2023\)Differentiable neural networks with repu activation: with applications to score estimation and isotonic regression\.arXiv preprint arXiv:2305\.00608\.Cited by:[2nd item](https://arxiv.org/html/2606.05599#S1.I1.i2.p1.1)\.
- \[42\]Q\. Sun, W\. Zhou, and J\. Fan\(2020\)Adaptive huber regression\.Journal of the American Statistical Association115\(529\),pp\. 254–265\.Cited by:[§5\.1](https://arxiv.org/html/2606.05599#S5.SS1.p3.4)\.
- \[43\]S\. Szopa, V\. Naik, B\. Adhikary, P\. Artaxo, T\. Berntsen, W\. D\. Collins, S\. Fuzzi, L\. Gallardo, A\. Kiendler\-Scharr, Z\. Klimont,et al\.\(2023\)Short\-lived climate forcers \(chapter 6\)\.Intergovernmental Panel on Climate Change \(IPCC\) 2021: Climate Change 2021: The Physical Science Basis\.,pp\. 817–922\.Cited by:[§6\.2](https://arxiv.org/html/2606.05599#S6.SS2.p1.1)\.
- \[44\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2)\.
- \[45\]Z\. Xie, Y\. Wei, H\. Cao, C\. Zhao, C\. Deng, J\. Li, D\. Dai, H\. Gao, J\. Chang, L\. Zhao,et al\.\(2025\)Mhc: manifold\-constrained hyper\-connections\.arXiv preprint arXiv:2512\.24880\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p4.2)\.
- \[46\]Y\. Yang and J\. He\(2025\)Deep neural networks with general activations: super\-convergence in sobolev norms\.arXiv preprint arXiv:2508\.05141\.Cited by:[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p2.1)\.
- \[47\]F\. Yao, H\. Müller, and J\. Wang\(2005\)Functional linear regression analysis for longitudinal data\.The Annals of Statistics33\(6\),pp\. 2873–2903\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p2.1)\.
- \[48\]D\. Yarotsky\(2017\)Error bounds for approximations with deep relu networks\.Neural Networks94,pp\. 103–114\.Cited by:[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p2.1)\.
- \[49\]M\. Yu, K\. M\. Tan, H\. J\. Wang, and W\. Zhou\(2025\)Deep neural expected shortfall regression with tail\-robustness\.arXiv preprint arXiv:2511\.08772\.Cited by:[§5\.2](https://arxiv.org/html/2606.05599#S5.SS2.p1.10),[§6\.1](https://arxiv.org/html/2606.05599#S6.SS1.p1.8)\.
- \[50\]S\. Zhang, J\. Lu, and H\. Zhao\(2024\)Deep network approximation: beyond relu to diverse activation functions\.Journal of Machine Learning Research25\(35\),pp\. 1–39\.Cited by:[§4\.2](https://arxiv.org/html/2606.05599#S4.SS2.p2.1)\.
- \[51\]Q\. Zhong, J\. Mueller, and J\. Wang\(2022\)Deep learning for the partially linear cox model\.The Annals of Statistics50\(3\),pp\. 1348–1375\.Cited by:[§1](https://arxiv.org/html/2606.05599#S1.p1.1)\.