Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation

# Sign Betting for Misspecified Regression:Variance Adaptivity and Optimal Approximation
Source: [https://arxiv.org/html/2610.20903](https://arxiv.org/html/2610.20903)
Yinan LiAffiliation:Department of Computer ScienceAffiliation:University of ArizonaEmail:[yinanli@arizona\.edu](mailto:)Han HanAffiliation:Department of Computer ScienceAffiliation:University of ArizonaEmail:[hanhan1@arizona\.edu](mailto:)

###### Abstract

We study bounded regression under model misspecification, evaluating a functionffby itsL1​\(PX\)L\_\{1\}\(P\_\{X\}\)distance to the conditional meanη⁡\(x\)=𝔼⁡\[Y∣X=x\]\\eta\(x\)=\\mathbb\{E\}\[Y\\mid X=x\], rather than the standardL2​\(PX\)L\_\{2\}\(P\_\{X\}\)\. ThisL1L\_\{1\}metric is natural for downstream decision problems such as bandits and reinforcement learning\. We analyze the min–max estimator of[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16), which we call*sign\-betting regression*\. Our main results show that sign\-betting regression achieves high\-probabilityL1L\_\{1\}oracle inequalities under misspecification\. Under realizability, our bound adapts to the average conditional variance and recovers the variance\-adaptive rate of[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)\. We further characterize the optimal universal approximation factor for proper learning by showing that sign betting achieves factor 3, while no proper learner can satisfy a distribution\-free, high\-probabilityL1L\_\{1\}oracle inequality with a universal approximation factor strictly below 3\. We extend the upper\-bound analysis beyond finite classes and complement the theory with experiments in synthetic and real\-world model\-selection tasks\.

### 1Introduction

Regression is a fundamental problem in statistics and machine learning and often an intermediate step in a downstream decision problem, with a quantity of interest being the underlying conditional mean\. We study bounded regression, where\(X,Y\)\(X,Y\)denotes a covariate–response pair withY∈\[0,1\]Y\\in\[0,1\], and evaluate a predictorffby its expected absolute discrepancy from the conditional mean,ε1​\(f\):=𝔼⁡\[\|f⁡\(X\)−η⁡\(X\)\|\]\\varepsilon\_\{1\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\|f\(X\)\-\\eta\(X\)\|\\right\], whereη⁡\(x\):=𝔼⁡\[Y∣X=x\]\\eta\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\[Y\\mid X=x\]\.

TheL1L\_\{1\}accuracy of a conditional\-mean predictor has several downstream interpretations\. For binary responses, it controls the excess risk of the plug\-in classifiergf\(x\):=𝟏\{f\(x\)≥1/2\}g\_\{f\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathbf\{1\}\\\{f\(x\)\\geq 1/2\\\}relative to the Bayes classifierg⋆\(x\):=𝟏\{η\(x\)≥1/2\}g^\{\\star\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathbf\{1\}\\\{\\eta\(x\)\\geq 1/2\\\}\. WritingRcls​\(g\):=ℙ⁡\(g⁡\(X\)≠Y\)R\_\{\\mathrm\{cls\}\}\(g\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\PP\(g\(X\)\\neq Y\)for the classification risk, the standard comparison inequality givesRcls​\(gf\)−Rcls​\(g⋆\)≤2​ε1​\(f\)R\_\{\\mathrm\{cls\}\}\(g\_\{f\}\)\-R\_\{\\mathrm\{cls\}\}\(g^\{\\star\}\)\\leq 2\\varepsilon\_\{1\}\(f\)\([Audibert and Tsybakov, 2007](https://arxiv.org/html/2610.20903#bib.bib12)\)\. TheL1L\_\{1\}metric is also natural for downstream decision problems such as bandits and reinforcement learning, where policy suboptimality is typically controlled through differences in expected rewards or costs rather than their squares\([Ayoub et al\., 2025](https://arxiv.org/html/2610.20903#bib.bib5)\)\. Finally, whenffis used as a reward model to select the highest\-scoring response amongNNcandidates, each marginally distributed according toPXP\_\{X\}, the expected reward loss relative to the best candidate in the same set is at mostN​ε1​\(f\)N\\varepsilon\_\{1\}\(f\)\. This connectsL1L\_\{1\}reward\-model accuracy to recent analyses of Best\-of\-NNand related inference\-time selection procedures\([Huang et al\., 2025](https://arxiv.org/html/2610.20903#bib.bib14)\)\. See derivations in Appendix[A](https://arxiv.org/html/2610.20903#A1)\.

##### Regression under misspecification\.

Letℱ\\mathcal\{F\}be a class of\[0,1\]\[0,1\]\-valued predictors and defineε1,∗:=inff∈ℱε1​\(f\)\\varepsilon\_\{1,\*\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\inf\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\)\. Under misspecification, we do not assume thatη∈ℱ\\eta\\in\\mathcal\{F\}\. We ask whether a learner can compete with the best achievableL1L\_\{1\}error within the class\. Ideally, an oracle inequality of the form

ε1​\(f^\)≤Caprx​ε1,∗\+rn,rn→0as​n→∞,\\varepsilon\_\{1\}\(\\widehat\{f\}\)\\leq C\_\{\\rm aprx\}\\,\\varepsilon\_\{1,\*\}\+r\_\{n\},\\qquad r\_\{n\}\\to 0\\quad\\text\{as \}n\\to\\infty,\(1\)whereCaprxC\_\{\\rm aprx\}is a universal constant\. We callCaprxC\_\{\\rm aprx\}the*approximation factor*andrnr\_\{n\}the*estimation term*\. Beyond requiringrnr\_\{n\}to vanish, we seek guarantees that adapt to the expected conditional variance𝔼⁡\[Var⁡\(Y∣X\)\]\\EE\[\\Var\(Y\\mid X\)\]and become sharper when this quantity is small\. This leads to two central questions:

> *What approximation and noise\-adaptive guarantees are achievable forL1L\_\{1\}conditional\-mean estimation under model misspecification? When the learner is required to return a member ofℱ\\mathcal\{F\}, what is the smallest universal approximation factor in a distribution\-free high\-probability oracle inequality?*

##### Why standard regression objectives do not resolve the problem\.

A loss that identifies the conditional mean under unrestricted optimization can select an approximation toη\\etathat differs from the bestL1L\_\{1\}approximation when the model class is misspecified\. Squared loss is the canonical example: minimizing its population risk overℱ\\mathcal\{F\}selects the bestL2L\_\{2\}approximation toη\\eta, rather than the bestL1L\_\{1\}approximation targeted by \([1](https://arxiv.org/html/2610.20903#S1.E1)\)\. More generally, standard losses such as squared and log loss induce their own notions of projection ontoℱ\\mathcal\{F\}under misspecification\([Mey and Loog, 2021](https://arxiv.org/html/2610.20903#bib.bib15)\)\. We show that this mismatch is fundamental: e\.g\., squared\-loss ERM admits anL1L\_\{1\}guarantee whose approximation term scales asε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}, and this dependence is unavoidable even in a noiseless problem with a function class of size 2\. Consequently, squared\-loss ERM cannot satisfy \([1](https://arxiv.org/html/2610.20903#S1.E1)\) withCaprxC\_\{\\rm aprx\}being a universal constant\. A second issue is statistical adaptation\. Although ann−1/2n^\{\-1/2\}estimation term is unavoidable in the worst case over distributions of\(X,Y\)\(X,Y\)withY∈\[0,1\]Y\\in\[0,1\]\([Tsybakov and Zaiats, 2009](https://arxiv.org/html/2610.20903#bib.bib4);[Wainwright, 2019](https://arxiv.org/html/2610.20903#bib.bib2)\), it can be overly pessimistic when the conditional variance of the labels is small\([Foster and Krishnamurthy, 2021](https://arxiv.org/html/2610.20903#bib.bib1);[Li et al\., 2026](https://arxiv.org/html/2610.20903#bib.bib6)\)\. Our goal is to achieve linear dependence on the bestL1L\_\{1\}approximation errorε1,∗\\varepsilon\_\{1,\*\}together with adaptation to low noise\. As we show in Section[3](https://arxiv.org/html/2610.20903#S3), the standard regression objectives considered therein do not, in general, provide either of these desiderata\.

##### Sign\-Betting Regression\.

We study a min\-max estimator introduced by[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16)for fixed\-design regression with possibly heteroscedastic and heavy\-tailed errors\. We refer to this procedure as*Sign\-Betting Regression \(SBR\)*, reflecting the directional betting interpretation developed in Section[4\.2](https://arxiv.org/html/2610.20903#S4.SS2)\.

##### Our results\.

Our first main result is a high\-probability generalization inequality for SBR\. In the general misspecified setting, it impliesε1​\(f^\)≤3​ε1,∗\+rn\\varepsilon\_\{1\}\(\\widehat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+r\_\{n\}, where the estimation termrnr\_\{n\}adapts to the residual second moment of anL1L\_\{1\}\-optimal comparator\. Under realizability, the approximation terms disappear and the leading statistical term adapts directly to the average conditional variance𝔼⁡\[Var⁡\(Y∣X\)\]\\EE\[\\Var\(Y\\mid X\)\], recovering the variance\-adaptive rate of[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)\. Our second main result identifies the optimal universal approximation factor for proper learning\. We prove that, for proper learning, \([1](https://arxiv.org/html/2610.20903#S1.E1)\) cannot hold with any universal constantCaprx<3C\_\{\\rm aprx\}<3and any distribution\-independentrnr\_\{n\}\. Since SBR achieves factor33, this characterizes the optimal universal proper approximation factor under our learning criterion\. Finally, we extend the upper\-bound analysis beyond finite classes and complement our theoretical results with experiments on synthetic data, real\-world datasets, and LLM model selection\.

Our setting and questions differ from those of[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16)in several respects\. They study fixed\-design regression and expectedL1L\_\{1\}error evaluated on the observed design points\. We instead consider i\.i\.d\. random design and study high\-probability oracle inequalities for the population error‖f−η‖L1​\(PX\)\\\|f\-\\eta\\\|\_\{L\_\{1\}\(P\_\{X\}\)\}\. A direct black\-box conversion—conditioning on the covariates, applying their fixed\-design result, and then using global uniform convergence—yields a population guarantee in expectation but incurs a variance\-independentn−1/2n^\{\-1/2\}generalization term, which can be substantially larger than our bound on low\-noise instances\. Our direct random\-design analysis achieves variance adaptation\. See Appendix[I](https://arxiv.org/html/2610.20903#A9)for details\.

##### Paper organization\.

Section[2](https://arxiv.org/html/2610.20903#S2)formalizes the learning problem, theL1L\_\{1\}oracle criterion, and the noise quantities used throughout the paper\. Section[3](https://arxiv.org/html/2610.20903#S3)shows how standard regression objectives can fail both the desired approximation and noise\-adaptation guarantees\. Section[4](https://arxiv.org/html/2610.20903#S4)develops the population geometry of sign betting and establishes our main variance\-adaptive oracle inequality\. Section[5](https://arxiv.org/html/2610.20903#S5)characterizes the optimal universal approximation factor for proper learning, and Section[6](https://arxiv.org/html/2610.20903#S6)extends the upper\-bound framework beyond finite classes\. Appendix[B](https://arxiv.org/html/2610.20903#A2)discusses related work and connections to prior results, and Appendix[M](https://arxiv.org/html/2610.20903#A13)presents the experiments\. The remaining appendices contain proofs and supporting technical material\.

### 2Problem setting and learning objective

##### Setting and notation\.

Let𝒳\\mathcal\{X\}be a measurable input space, and letPPbe an unknown distribution over𝒳×\[0,1\]\\mathcal\{X\}\\times\[0,1\]\. We observenni\.i\.d\. samples\(Xi,Yi\)i=1n∼Pn\(X\_\{i\},Y\_\{i\}\)\_\{i=1\}^\{n\}\\sim P^\{n\}\. We writePXP\_\{X\}for the marginal distribution ofXXandη⁡\(x\):=𝔼⁡\[Y∣X=x\]\\eta\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\[Y\\mid X=x\]for the regression function\. We consider a nonempty classℱ⊆\{f:𝒳→\[0,1\]\}\\mathcal\{F\}\\subseteq\\\{f\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\[0,1\]\\\}of measurable predictors, without assuming thatη∈ℱ\\eta\\in\\mathcal\{F\}\. For a measurableg:𝒳→ℝg\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\\mathbb\{R\}andp∈\[1,∞\)p\\in\[1,\\infty\), we write‖g‖p:=\(𝔼X∼PX⁡\|g⁡\(X\)\|p\)1/p\\mathinner\{\\\!\\left\\\|g\\right\\\|\}\_\{p\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(\\EE\_\{X\\sim P\_\{X\}\}\|g\(X\)\|^\{p\}\)^\{1/p\}\. For a measurable functionZ=Z⁡\(X,Y\)Z=Z\(X,Y\), we use the shorthandP​Z:=𝔼⁡\[Z⁡\(X,Y\)\]PZ\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\[Z\(X,Y\)\]andPn​Z:=n−1​∑i=1nZ⁡\(Xi,Yi\)P\_\{n\}Z\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=n^\{\-1\}\\sum\_\{i=1\}^\{n\}Z\(X\_\{i\},Y\_\{i\}\)\.

##### Approximation error\.

Forp∈\[1,∞\)p\\in\[1,\\infty\)andf∈ℱf\\in\\mathcal\{F\}, defineεp​\(f\):=‖f−η‖p\\varepsilon\_\{p\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{p\}andεp,∗:=inff∈ℱεp​\(f\)\\varepsilon\_\{p,\*\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\inf\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{p\}\(f\)\. Our main performance criterion is theL1L\_\{1\}conditional\-mean approximation errorε1​\(f\)=‖f−η‖1=𝔼⁡\|f⁡\(X\)−η⁡\(X\)\|\\varepsilon\_\{1\}\(f\)=\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{1\}=\\EE\|f\(X\)\-\\eta\(X\)\|, andε1,∗\\varepsilon\_\{1,\*\}is the smallest such error attainable withinℱ\\mathcal\{F\}\. When the infimum exists, we letf⋆∈arg~minf∈ℱε1​\(f\)f\_\{\\star\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\)denote a bestL1L\_\{1\}approximation toη\\etainℱ\\mathcal\{F\}\. In particular, such a minimizer always exists for the finite classes considered in our main results\. We say that\(P,ℱ\)\(P,\\mathcal\{F\}\)is*realizable*\(or*well\-specified*\) ifη∈ℱ\\eta\\in\\mathcal\{F\}, and*misspecified*otherwise\. For a finite class, realizability is equivalent toε1,∗=0\\varepsilon\_\{1,\*\}=0\. Our results do not assume realizability\. Under misspecification, our guarantees compare the learned predictor with arbitrary comparatorsf0∈ℱf\_\{0\}\\in\\mathcal\{F\}, allowing the bound to trade off their approximation error \(how wellf0f\_\{0\}approximatesη\\eta\) and comparator\-dependent estimation terms\. We distinguish this criterion in Appendix[E](https://arxiv.org/html/2610.20903#A5)from the absolute\-loss risk𝔼⁡\|Y−f⁡\(X\)\|\\EE\|Y\-f\(X\)\|, which is generally a different population objective\.

##### Noise and approximation decomposition\.

Letξ:=Y−η⁡\(X\)\\xi\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=Y\-\\eta\(X\),σ2​\(X\):=Var⁡\(Y∣X\)=𝔼⁡\[ξ2∣X\]\\sigma^\{2\}\(X\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\Var\(Y\\mid X\)=\\EE\[\\xi^\{2\}\\mid X\], andσ¯2:=𝔼⁡\[σ2​\(X\)\]\\bar\{\\sigma\}^\{2\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\[\\sigma^\{2\}\(X\)\]\. Then𝔼⁡\[ξ∣X\]=0\\EE\[\\xi\\mid X\]=0, and for every predictorff,𝔼⁡\[\(Y−f⁡\(X\)\)2∣X\]=σ2​\(X\)\+\(η⁡\(X\)−f⁡\(X\)\)2\\EE\[\(Y\-f\(X\)\)^\{2\}\\mid X\]=\\sigma^\{2\}\(X\)\+\(\\eta\(X\)\-f\(X\)\)^\{2\}\. The first term is the conditional label variance, while the second is the squared approximation error\. This decomposition will enter the variance\-dependent bounds under misspecification\.

##### Proper learning\.

A learner is*proper*with respect toℱ\\mathcal\{F\}if it always returns a predictorf^∈ℱ\\hat\{f\}\\in\\mathcal\{F\}; otherwise it is*improper*\. Our main results focus on proper learning: our main algorithm is proper, and our lower bound characterizes the best universal approximation factor achievable by proper learners\.

### 3Limitations of standard regression objectives

We first ask whether familiar regression objectives already provide theL1L\_\{1\}oracle guarantee in \([1](https://arxiv.org/html/2610.20903#S1.E1)\)\. We focus here on squared loss and log loss, while deferring absolute loss to Appendix[E](https://arxiv.org/html/2610.20903#A5)and betting loss of[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)to Appendix[F](https://arxiv.org/html/2610.20903#A6)\. These objectives fail for two conceptually different reasons\. Absolute loss is misaligned with the target itself: its population minimizer is generally a conditional median rather than the conditional meanη\\eta\. By contrast, squared loss, log loss, and betting loss are all aligned with conditional\-mean regression under realizability, with betting loss enjoying variance\-adaptive guarantees\([Li et al\., 2026](https://arxiv.org/html/2610.20903#bib.bib6)\)\. Nevertheless, under model misspecification, squared loss, log loss, and betting loss may not produce a predictor whoseL1L\_\{1\}error is within any universal constant factor ofε1,∗\\varepsilon\_\{1,\*\}\. These limitations motivate an objective that achieves anL1L\_\{1\}oracle guarantee with a universal constant approximation factor under misspecification while enjoying variance\-adaptive statistical guarantees\.

#### 3\.1Squared\-loss ERM

We begin with the familiar squared\-loss ERM,f^sq∈arg~minf∈ℱ1n​∑i=1n\(Yi−f⁡\(Xi\)\)2\\widehat\{f\}\_\{\\rm sq\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\bigl\(Y\_\{i\}\-f\(X\_\{i\}\)\\bigr\)^\{2\}\. Squared loss is calibrated to the conditional mean when optimization is unrestricted\. Under misspecification, however, minimizing squared loss overℱ\\mathcal\{F\}does not yield the bestL1L\_\{1\}approximation toη\\eta\. Indeed, for everyf∈ℱf\\in\\mathcal\{F\},

𝔼⁡\[\(Y−f⁡\(X\)\)2\]=𝔼⁡\[Var⁡\(Y∣X\)\]\+‖f−η‖22\.\\EE\\bigl\[\(Y\-f\(X\)\)^\{2\}\\bigr\]=\\EE\\\!\\left\[\\Var\(Y\\mid X\)\\right\]\+\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{2\}^\{2\}\.\(2\)Thus, at the population level, squared loss selects anL2L\_\{2\}\-optimal approximation toη\\eta, whereas our benchmarkε1,∗\\varepsilon\_\{1,\*\}is defined by the bestL1L\_\{1\}approximation\. The following proposition quantifies the resulting finite\-sample guarantee\. The proof is deferred to Appendix[C\.1](https://arxiv.org/html/2610.20903#A3.SS1)\.

###### Proposition 1\(L1L\_\{1\}guarantee for squared\-loss ERM\)\.

SupposeY∈\[0,1\]Y\\in\[0,1\]andℱ⊆\[0,1\]𝒳\\mathcal\{F\}\\subseteq\[0,1\]^\{\\mathcal\{X\}\}is finite\. There exists a universal constantC\>0C\>0such that, with probability at least1−δ1\-\\delta,

ε1​\(f^sq\)≤ε2,∗\+C​log⁡\(2​\|ℱ\|2/δ\)n≤ε1,∗\+C​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\rm sq\}\)\\leq\\varepsilon\_\{2,\*\}\+C\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\\leq\\sqrt\{\\varepsilon\_\{1,\*\}\}\+C\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.\(3\)

The first inequality in \([3](https://arxiv.org/html/2610.20903#S3.E3)\) reflects the natural geometry of squared loss: apart from estimation error, squared\-loss ERM competes withε2,∗\\varepsilon\_\{2,\*\}\. The second inequality follows fromε2,∗≤ε1,∗\\varepsilon\_\{2,\*\}\\leq\\sqrt\{\\varepsilon\_\{1,\*\}\}for\[0,1\]\[0,1\]\-valued hypothesis classℱ\\mathcal\{F\}\. Next, we show in Proposition[2](https://arxiv.org/html/2610.20903#Thmtheorem2)that this square\-root dependence onε1,∗\\varepsilon\_\{1,\*\}is not merely an artifact of converting anL2L\_\{2\}bound toL1L\_\{1\}\.

###### Proposition 2\(Squared loss has no constantL1L\_\{1\}approximation factor\)\.

For everyp∈\(0,1/4\)p\\in\(0,1/4\), there exists a noiseless regression problem and a two\-element classℱ⊆\[0,1\]𝒳\\mathcal\{F\}\\subseteq\[0,1\]^\{\\mathcal\{X\}\}such thatε1,∗=p\\varepsilon\_\{1,\*\}=p, while the unique population squared\-loss minimizer,fsq∗∈arg~minf∈ℱ​𝔼​\[\(Y−f⁡\(X\)\)2\],f\_\{\\rm sq\}^\{\*\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\EE\\bigl\[\(Y\-f\(X\)\)^\{2\}\\bigr\],satisfies

ε1​\(fsq∗\)=12​ε1,∗\.\\varepsilon\_\{1\}\(f\_\{\\rm sq\}^\{\*\}\)=\\frac\{1\}\{2\}\\sqrt\{\\varepsilon\_\{1,\*\}\}\.\(4\)Consequently, squared\-loss ERM cannot satisfy the basic goal \([1](https://arxiv.org/html/2610.20903#S1.E1)\) for any finite universal approximation factor\.

Proposition[2](https://arxiv.org/html/2610.20903#Thmtheorem2)shows that theε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}dependence in \([3](https://arxiv.org/html/2610.20903#S3.E3)\) is unavoidable up to a universal constant\. This is a population\-level obstruction: even with noiseless labels and arbitrarily many samples, squared loss can prefer anL2L\_\{2\}\-better predictor whoseL1L\_\{1\}error is of orderε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}rather thanε1,∗\\varepsilon\_\{1,\*\}\. Appendix[C\.2](https://arxiv.org/html/2610.20903#A3.SS2)gives a sharper result \(with proof\) that, for anyγ∈\(0,1\)\\gamma\\in\(0,1\), one can construct an instance satisfyingε1​\(fsq∗\)=\(1−γ\)​ε1,∗\\varepsilon\_\{1\}\(f\_\{\\rm sq\}^\{\*\}\)=\(1\-\\gamma\)\\sqrt\{\\varepsilon\_\{1,\*\}\}\. Proposition[2](https://arxiv.org/html/2610.20903#Thmtheorem2)is a specialization withγ=1/2\\gamma=1/2\.

Then−1/2n^\{\-1/2\}statistical term in \([3](https://arxiv.org/html/2610.20903#S3.E3)\) cannot be improved in general\. Standard minimax lower bounds show anΩ\(n−1/2\)\\Omega\(n^\{\-1/2\}\)error for arbitrary estimators in simple bounded mean\-estimation problems; see, e\.g\.,[Tsybakov and Zaiats \(2009\)](https://arxiv.org/html/2610.20903#bib.bib4);[Wainwright \(2019\)](https://arxiv.org/html/2610.20903#bib.bib2)\. The limitation of the estimation error from squared\-loss ERM is the failure to adapt to easier instances\. In realizable full\-information cost\-sensitive classification,[Foster and Krishnamurthy \(2021, Theorem 2\)](https://arxiv.org/html/2610.20903#bib.bib1)show that least\-squares plug\-in prediction can incurΩ\(n−1/2\)\\Omega\(n^\{\-1/2\}\)excess risk even when the first\-order rate isO⁡\(1/n\)O\(1/n\)\. Analogously, in realizable\[0,1\]\[0,1\]\-valued regression,[Li et al\. \(2026, Proposition 8\)](https://arxiv.org/html/2610.20903#bib.bib6)show that squared\-loss ERM can incurΩ\(n−1/2\)\\Omega\(n^\{\-1/2\}\)L1L\_\{1\}error even when the first\-order rate isO⁡\(1/n\)O\(1/n\)\.

Taken together, these results identify two distinct limitations of squared\-loss ERM for our objective\. Under misspecification, itsL2L\_\{2\}approximation can be arbitrarily worse than the bestL1L\_\{1\}approximation in multiplicative terms; even under realizability, squared\-loss ERM does not adapt to favorable conditional variance\.

#### 3\.2Log\-loss ERM

Another natural choice for conditional\-mean estimation is log loss, which is widely used for learning probabilistic predictions\. Recall the log\-loss ERM

f^log∈arg~minf∈ℱ1n​∑i=1nℓlog​\(Yi,f⁡\(Xi\)\),ℓlog​\(y,q\):=−y​log⁡q−\(1−y\)​log⁡\(1−q\)\.\\widehat\{f\}\_\{\\log\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\_\{\\log\}\\bigl\(Y\_\{i\},f\(X\_\{i\}\)\\bigr\),\\qquad\\ell\_\{\\log\}\(y,q\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\-y\\log q\-\(1\-y\)\\log\(1\-q\)\.\(5\)For anyY∈\[0,1\]Y\\in\[0,1\], conditioning onX=xX=xgives

𝔼⁡\[ℓlog​\(Y,q\)∣X=x\]=−η⁡\(x\)​log⁡q−\(1−η⁡\(x\)\)​log⁡\(1−q\),\\EE\[\\ell\_\{\\log\}\(Y,q\)\\mid X=x\]=\-\\eta\(x\)\\log q\-\(1\-\\eta\(x\)\)\\log\(1\-q\),which is uniquely minimized atq=η⁡\(x\)q=\\eta\(x\)\. Thus, like squared loss, log loss is calibrated to the conditional mean when optimization is unrestricted\.

Log loss is particularly relevant to our comparison because, under realizability, it can exhibit favorable first\-order behavior that squared loss fails to achieve\. In the cost\-sensitive classification setting,[Foster and Krishnamurthy \(2021\)](https://arxiv.org/html/2610.20903#bib.bib1)show that log\-loss regression yields a first\-order guarantee\. More directly for our regression setting,[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)establish a first\-order guarantee for log\-loss ERM measured in the sameL1L\_\{1\}prediction error considered in this paper\. This motivates asking whether it also avoids the other deficiency of squared loss under misspecification: the mismatch between its population objective and the bestL1L\_\{1\}approximation inℱ\\mathcal\{F\}\. We show that it does not\. Even when all predictions are uniformly bounded away from00and11, log\-loss ERM can incurL1L\_\{1\}error of orderε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}, and therefore cannot achieve a universal constant\-factorL1L\_\{1\}oracle guarantee\.

Forp∈\[0,1\]p\\in\[0,1\]andq∈\(0,1\)q\\in\(0,1\), letkl⁡\(p,q\):=p​log⁡pq\+\(1−p\)​log⁡1−p1−q,\{\\mathrm\{kl\}\}\(p,q\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=p\\log\\frac\{p\}\{q\}\+\(1\-p\)\\log\\frac\{1\-p\}\{1\-q\},with the convention0​log⁡0=00\\log 0=0\. At the population level \(see Appendix[D\.1](https://arxiv.org/html/2610.20903#A4.SS1)for the derivation\),

𝔼⁡\[ℓlog​\(Y,f⁡\(X\)\)−ℓlog​\(Y,η⁡\(X\)\)\]=𝔼⁡\[kl⁡\(η⁡\(X\),f⁡\(X\)\)\]\.\\EE\\\!\\left\[\\ell\_\{\\log\}\(Y,f\(X\)\)\-\\ell\_\{\\log\}\(Y,\\eta\(X\)\)\\right\]=\\EE\\\!\\left\[\{\\mathrm\{kl\}\}\\bigl\(\\eta\(X\),f\(X\)\\bigr\)\\right\]\.\(6\)Thus, under misspecification, population log\-loss minimization selects a KL\-optimal approximation toη\\eta, which may not coincide with the bestL1L\_\{1\}approximation\.

Because log loss becomes increasingly sensitive as predictions approach00or11, anL1L\_\{1\}guarantee in terms ofε1,∗\\varepsilon\_\{1,\*\}requires explicit control of the prediction range\. The following proposition gives the resulting guarantee under a restricted boundary condition\.

###### Proposition 3\(L1L\_\{1\}guarantee for log\-loss ERM\)\.

SupposeY∈\[0,1\]Y\\in\[0,1\],ℱ\\mathcal\{F\}is finite, and, for someγ∈\(0,1/2\)\\gamma\\in\(0,1/2\),ℱ⊆\[γ,1−γ\]𝒳\.\\mathcal\{F\}\\subseteq\[\\gamma,1\-\\gamma\]^\{\\mathcal\{X\}\}\.Then, with probability at least1−δ1\-\\delta,

ε1​\(f^log\)≤log⁡\(1/γ\)1−γ​ε1,∗\+\(2\+23​log⁡1γ\)​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\leq\\sqrt\{\\frac\{\\log\(1/\\gamma\)\}\{1\-\\gamma\}\\,\\varepsilon\_\{1,\*\}\}\+\\sqrt\{\\left\(2\+\\frac\{2\}\{3\}\\log\\frac\{1\}\{\\gamma\}\\right\)\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.\(7\)In particular, for every fixedγ\\gamma,ε1\(f^log\)≲γε1,∗\+log⁡\(\|ℱ\|/δ\)/n\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\lesssim\_\{\\gamma\}\\sqrt\{\\varepsilon\_\{1,\*\}\}\+\\sqrt\{\\log\(\|\\mathcal\{F\}\|/\\delta\)/n\}\.

The proof is deferred to Appendix[D\.2](https://arxiv.org/html/2610.20903#A4.SS2)\. Proposition[3](https://arxiv.org/html/2610.20903#Thmtheorem3)shows that, even under a restricted boundary condition, the naturalL1L\_\{1\}guarantee for log\-loss ERM scales asε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}rather than linearly withε1,∗\\varepsilon\_\{1,\*\}\. As with squared loss, one might wonder whether thisε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}dependence is merely an artifact of the analysis\. The next proposition shows that it is unavoidable: even for a two\-element class uniformly bounded away from00and11, the population log\-loss minimizer can haveL1L\_\{1\}error of orderε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}\.

###### Proposition 4\(Log loss has no constantL1L\_\{1\}approximation factor\)\.

For everyp∈\(0,1\)p\\in\(0,1\), there exists a noiseless regression problem and a two\-element classℱ⊆\[1/4,3/4\]𝒳\\mathcal\{F\}\\subseteq\[1/4,3/4\]^\{\\mathcal\{X\}\}such thatε1,∗=p/4\\varepsilon\_\{1,\*\}=p/4, while the unique population log\-loss minimizerflog∗∈arg~minf∈ℱ​𝔼​\[ℓlog​\(Y,f⁡\(X\)\)\]f\_\{\\log\}^\{\*\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\EE\[\\ell\_\{\\log\}\(Y,f\(X\)\)\]satisfies

ε1​\(flog∗\)=12​ε1,∗\.\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)=\\frac\{1\}\{2\}\\sqrt\{\\varepsilon\_\{1,\*\}\}\.\(8\)Consequently, log\-loss ERM cannot satisfy the basic goal \([1](https://arxiv.org/html/2610.20903#S1.E1)\) for any finite universal approximation factor, even when all predictions are uniformly bounded away from00and11\.

Proposition[4](https://arxiv.org/html/2610.20903#Thmtheorem4)shows that theε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}dependence in Proposition[3](https://arxiv.org/html/2610.20903#Thmtheorem3)is unavoidable up to a universal constant\. Even when the hypothesis class is uniformly bounded away from00and11, the geometry of KL divergence can make the population log\-loss minimizer haveL1L\_\{1\}error of orderε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}, even thoughℱ\\mathcal\{F\}contains another hypothesis withL1L\_\{1\}errorε1,∗\\varepsilon\_\{1,\*\}\. Thus, as with squared loss, log\-loss ERM cannot in general achieve linear dependence onε1,∗\\varepsilon\_\{1,\*\}or a finite universalL1L\_\{1\}approximation factor\.

The restrictionℱ⊆\[γ,1−γ\]𝒳\\mathcal\{F\}\\subseteq\[\\gamma,1\-\\gamma\]^\{\\mathcal\{X\}\}is essential for Proposition[3](https://arxiv.org/html/2610.20903#Thmtheorem3)\. Without requiring predictions to stay uniformly away from00and11, log\-loss ERM can fail much more severely: its population minimizer may haveL1L\_\{1\}error bounded away from zero even asε1,∗→0\\varepsilon\_\{1,\*\}\\to 0\. We give this stronger boundary\-sensitive obstruction in Appendix[D\.4](https://arxiv.org/html/2610.20903#A4.SS4)\. Thus, even in the more favorable setting where all hypotheses are bounded away from00and11, the worst\-case approximation dependence of log\-loss ERM remains of orderε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}, matching the qualitative limitation of squared\-loss ERM\.

### 4Sign betting loss and variance\-adaptive bounds

#### 4\.1Sign betting for misspecified regression

Motivated by the limitations of standard regression objectives established in Section[3](https://arxiv.org/html/2610.20903#S3), we turn to the min–max pairwise estimator introduced by[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16)\. We show that this procedure, which we call*Sign\-Betting Regression \(SBR\)*, aligns more with the bestL1L\_\{1\}approximation to the conditional mean withinℱ\\mathcal\{F\}and yields strong oracle guarantees under misspecification\. Givennni\.i\.d\. samples\(Xi,Yi\)i=1n\(X\_\{i\},Y\_\{i\}\)\_\{i=1\}^\{n\}, forf,h∈ℱf,h\\in\\mathcal\{F\}, define

T^​\(f,h\):=1n​∑i=1n\(Yi−f⁡\(Xi\)\)​sign⁡\(h⁡\(Xi\)−f⁡\(Xi\)\),\\widehat\{T\}\(f,h\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\bigl\(Y\_\{i\}\-f\(X\_\{i\}\)\\bigr\)\\sign\\bigl\(h\(X\_\{i\}\)\-f\(X\_\{i\}\)\\bigr\),\(9\)wheresign⁡\(0\)=0\\sign\(0\)=0\. Our*Sign\-Betting Regression \(SBR\)*algorithm returns

f^∈arg~minf∈ℱsuph∈ℱT^\(f,h\)\.\\hat\{f\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f,h\)\.\(10\)For the finite classes considered in our main results, the supremum above is a maximum\.

The terminology “sign betting” comes from viewinghhas a challenger that places a directional bet against the residual offf\. At an inputxx, the challenger bets\+1\+1whenh⁡\(x\)\>f⁡\(x\)h\(x\)\>f\(x\)and−1\-1whenh⁡\(x\)<f⁡\(x\)h\(x\)<f\(x\)\. Its payoff is\(Y−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\(Y\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\. Thus, ifhhlies aboveff, the challenger benefits when the observed response is also aboveff; ifhhlies belowff, it benefits when the observed response is belowff\. A large value ofT^​\(f,h\)\\widehat\{T\}\(f,h\)therefore indicates that the direction fromfftowardhhis systematically supported by the residuals\.

The inner supremum in \([10](https://arxiv.org/html/2610.20903#S4.E10)\) searches for the challengerh∈ℱh\\in\\mathcal\{F\}that achieves the largest empirical sign\-betting valueT^​\(f,h\)\\widehat\{T\}\(f,h\)against a given candidateff\. The outer minimization then selects a predictor whose residuals are difficult to exploit by anyh∈ℱh\\in\\mathcal\{F\}\. Thus, unlike ordinary empirical risk minimization with a pointwise lossℓ⁡\(Y,f⁡\(X\)\)\\ell\(Y,f\(X\)\), the objective associated withffdepends on howffcompares with the other predictors inℱ\\mathcal\{F\}\.

We next study the population geometry of sign betting\. The key identity shows that, for anyf,h∈ℱf,h\\in\\mathcal\{F\}, the two betting quantities associated with\(f,h\)\(f,h\)and\(h,f\)\(h,f\)sum exactly to‖f−h‖1\\mathinner\{\\\!\\left\\\|f\-h\\right\\\|\}\_\{1\}\. This identity will allow us to translate control of the sign\-betting objective into control ofL1L\_\{1\}error\.

#### 4\.2Population geometry of sign betting

We first study the population behavior of sign betting\. We will show that, for any pairf,h∈ℱf,h\\in\\mathcal\{F\}, the two population quantitiesTP​\(f,h\)T\_\{P\}\(f,h\)andTP​\(h,f\)T\_\{P\}\(h,f\)sum exactly to theL1​\(PX\)L\_\{1\}\(P\_\{X\}\)distance betweenffandhh\. This identity leads to a population performance guarantee for sign betting and provides the basic mechanism for the high\-probability guarantees developed in the next subsection\. Forf,h∈ℱf,h\\in\\mathcal\{F\}, define

TP​\(f,h\):=𝔼⁡\[\(Y−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\]=𝔼⁡\[\(η⁡\(X\)−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\]\.T\_\{P\}\(f,h\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\(Y\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\\right\]=\\EE\\\!\\left\[\(\\eta\(X\)\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\\right\]\.\(11\)The second equality follows by conditioning onXXand using𝔼⁡\[Y∣X\]=η⁡\(X\)\\EE\[Y\\mid X\]=\\eta\(X\)\. We also define the population sign\-betting objective

BP​\(f\):=suph∈ℱTP​\(f,h\)\.B\_\{P\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{h\\in\\mathcal\{F\}\}T\_\{P\}\(f,h\)\.\(12\)
The basic geometry ofTP​\(⋅,⋅\)T\_\{P\}\(\\cdot,\\cdot\)follows from a pairwise identity:

###### Lemma 5\(Pairwise identity for sign betting\)\.

For anyf,h∈ℱf,h\\in\\mathcal\{F\},

TP​\(f,h\)\+TP​\(h,f\)=‖f−h‖1\.T\_\{P\}\(f,h\)\+T\_\{P\}\(h,f\)=\\mathinner\{\\\!\\left\\\|f\-h\\right\\\|\}\_\{1\}\.\(13\)

###### Proof\.

Usingsign⁡\(f−h\)=−sign⁡\(h−f\)\\sign\(f\-h\)=\-\\sign\(h\-f\),

TP​\(f,h\)\+TP​\(h,f\)\\displaystyle T\_\{P\}\(f,h\)\+T\_\{P\}\(h,f\)=𝔼⁡\[\(η−f\)​sign⁡\(h−f\)\+\(η−h\)​sign⁡\(f−h\)\]\\displaystyle=\\EE\\\!\\left\[\(\\eta\-f\)\\sign\(h\-f\)\+\(\\eta\-h\)\\sign\(f\-h\)\\right\]=𝔼⁡\[\(h−f\)​sign⁡\(h−f\)\]\\displaystyle=\\EE\\\!\\left\[\(h\-f\)\\sign\(h\-f\)\\right\]=𝔼⁡\|h−f\|=‖f−h‖1\.\\displaystyle=\\EE\|h\-f\|=\\mathinner\{\\\!\\left\\\|f\-h\\right\\\|\}\_\{1\}\.∎

The population objectiveBP​\(f\)B\_\{P\}\(f\)is also directly controlled by theL1L\_\{1\}approximation error:

###### Lemma 6\(Population sign\-betting objective\)\.

For everyf∈ℱf\\in\\mathcal\{F\},

0≤BP​\(f\)≤ε1​\(f\)\.0\\leq B\_\{P\}\(f\)\\leq\\varepsilon\_\{1\}\(f\)\.\(14\)

###### Proof\.

Sinceh=fh=fis an admissible choice in the definition ofBP​\(f\)B\_\{P\}\(f\)in \([12](https://arxiv.org/html/2610.20903#S4.E12)\) andTP​\(f,f\)=0T\_\{P\}\(f,f\)=0, we haveBP​\(f\)≥0B\_\{P\}\(f\)\\geq 0\. Moreover, for everyh∈ℱh\\in\\mathcal\{F\},

TP​\(f,h\)=𝔼⁡\[\(η⁡\(X\)−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\]≤𝔼\|η⁡\(X\)−f⁡\(X\)\|=ε1​\(f\)\.T\_\{P\}\(f,h\)=\\EE\\\!\\left\[\(\\eta\(X\)\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\\right\]\\leq\\EE\|\\eta\(X\)\-f\(X\)\|=\\varepsilon\_\{1\}\(f\)\.Taking the supremum overh∈ℱh\\in\\mathcal\{F\}yieldsBP​\(f\)≤ε1​\(f\)B\_\{P\}\(f\)\\leq\\varepsilon\_\{1\}\(f\)\. ∎

Lemma[5](https://arxiv.org/html/2610.20903#Thmtheorem5)and Lemma[6](https://arxiv.org/html/2610.20903#Thmtheorem6)yield a population performance guarantee for sign betting:

###### Proposition 7\(Population guarantee for sign betting\)\.

Suppose the minimum ofBPB\_\{P\}overℱ\\mathcal\{F\}is attained, and letfB∈arg~minf∈ℱBP​\(f\)f\_\{B\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}B\_\{P\}\(f\)\. Then, for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

ε1​\(fB\)≤ε1​\(f0\)\+2​BP​\(f0\)\.\\varepsilon\_\{1\}\(f\_\{B\}\)\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+2B\_\{P\}\(f\_\{0\}\)\.\(15\)Consequently,ε1​\(fB\)≤3​ε1​\(f0\)\\varepsilon\_\{1\}\(f\_\{B\}\)\\leq 3\\varepsilon\_\{1\}\(f\_\{0\}\)\.

###### Proof\.

By Lemma[5](https://arxiv.org/html/2610.20903#Thmtheorem5),

‖fB−f0‖1\\displaystyle\\mathinner\{\\\!\\left\\\|f\_\{B\}\-f\_\{0\}\\right\\\|\}\_\{1\}=TP​\(fB,f0\)\+TP​\(f0,fB\)≤BP​\(fB\)\+BP​\(f0\)≤2​BP​\(f0\),\\displaystyle=T\_\{P\}\(f\_\{B\},f\_\{0\}\)\+T\_\{P\}\(f\_\{0\},f\_\{B\}\)\\leq B\_\{P\}\(f\_\{B\}\)\+B\_\{P\}\(f\_\{0\}\)\\leq 2B\_\{P\}\(f\_\{0\}\),where the last inequality follows from the optimality offBf\_\{B\}\. Therefore,

ε1​\(fB\)≤ε1​\(f0\)\+‖fB−f0‖1≤ε1​\(f0\)\+2​BP​\(f0\)\.\\varepsilon\_\{1\}\(f\_\{B\}\)\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+\\mathinner\{\\\!\\left\\\|f\_\{B\}\-f\_\{0\}\\right\\\|\}\_\{1\}\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+2B\_\{P\}\(f\_\{0\}\)\.The second claim follows fromBP​\(f0\)≤ε1​\(f0\)B\_\{P\}\(f\_\{0\}\)\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)by Lemma[6](https://arxiv.org/html/2610.20903#Thmtheorem6)\. ∎

Equation \([15](https://arxiv.org/html/2610.20903#S4.E15)\) is more informative than the worst\-case factor\-three bound: the additional approximation penalty is governed by the population sign\-betting valueBP​\(f0\)B\_\{P\}\(f\_\{0\}\), which can be much smaller thanε1​\(f0\)\\varepsilon\_\{1\}\(f\_\{0\}\)\. We next show that the empirical Sign\-Betting Regression algorithm inherits this population guarantee, up tof0f\_\{0\}\-dependent estimation terms that decay withnnand adapt to the label variance\.

#### 4\.3Main variance\-adaptive guarantee

We now turn to the finite\-sample performance of SBR\. As we will see, the leading estimation term is controlled by anf0f\_\{0\}\-dependent residual second moment\. Forf0∈ℱf\_\{0\}\\in\\mathcal\{F\}, define

v⁡\(f0\)\\displaystyle v\(f\_\{0\}\):=suph∈ℱ𝔼\[\(Y−f0\(X\)\)2𝟏\{h\(X\)≠f0\(X\)\}\]\\displaystyle\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{h\\in\\mathcal\{F\}\}\\EE\\\!\\left\[\(Y\-f\_\{0\}\(X\)\)^\{2\}\\mathbf\{1\}\\\{h\(X\)\\neq f\_\{0\}\(X\)\\\}\\right\]=suph∈ℱ𝔼\[\(σ2\(X\)\+\(η\(X\)−f0\(X\)\)2\)𝟏\{h\(X\)≠f0\(X\)\}\]\.\\displaystyle=\\sup\_\{h\\in\\mathcal\{F\}\}\\EE\\\!\\left\[\\left\(\\sigma^\{2\}\(X\)\+\(\\eta\(X\)\-f\_\{0\}\(X\)\)^\{2\}\\right\)\\mathbf\{1\}\\\{h\(X\)\\neq f\_\{0\}\(X\)\\\}\\right\]\.\(16\)
The second equality follows from the noise–approximation decomposition in Section[2](https://arxiv.org/html/2610.20903#S2)\. In particular,v⁡\(f0\)≤σ¯2\+ε2​\(f0\)2v\(f\_\{0\}\)\\leq\\bar\{\\sigma\}^\{2\}\+\\varepsilon\_\{2\}\(f\_\{0\}\)^\{2\}\. Thus, under misspecification,v⁡\(f0\)v\(f\_\{0\}\)captures both the conditional label variance and the squared approximation residual, restricted to regions wheref0f\_\{0\}disagrees with other predictors inℱ\\mathcal\{F\}\. This localization can makev⁡\(f0\)v\(f\_\{0\}\)smaller than the corresponding global residual second momentσ¯2\+ε2​\(f0\)2\\bar\{\\sigma\}^\{2\}\+\\varepsilon\_\{2\}\(f\_\{0\}\)^\{2\}\. Our main finite\-class guarantee is the following\.

###### Theorem 8\(Variance\-adaptive guarantee for SBR\)\.

Supposeℱ\\mathcal\{F\}is finite, and letf^\\hat\{f\}be the output of SBR in \([10](https://arxiv.org/html/2610.20903#S4.E10)\)\. Forρ\>0\\rho\>0, letCρ:=133\+1ρ\+2312​ρC\_\{\\rho\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{13\}\{3\}\+\\frac\{1\}\{\\rho\}\+\\frac\{23\}\{12\}\\rho\. Then, for everyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta, the following holds simultaneously for everyρ\>0\\rho\>0:

ε1​\(f^\)≤inff0∈ℱ\{ε1​\(f0\)\+\(2\+ρ\)​BP​\(f0\)\+\(2\+ρ\)​2​v​\(f0\)​log⁡\(2​\|ℱ\|2/δ\)n\}\+Cρ​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\\inf\_\{f\_\{0\}\\in\\mathcal\{F\}\}\\left\\\{\\varepsilon\_\{1\}\(f\_\{0\}\)\+\(2\+\\rho\)B\_\{P\}\(f\_\{0\}\)\+\(2\+\\rho\)\\sqrt\{\\frac\{2v\(f\_\{0\}\)\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\\right\\\}\+C\_\{\\rho\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\.\(17\)

The parameterρ\\rhois used only in the analysis and is not an input to SBR\. Since the event in Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)holds simultaneously for allρ\>0\\rho\>0,ρ\\rhocan be chosen as a function off0f\_\{0\}and the other quantities when interpreting the bound\. In particular, the theorem implies that, for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

ε1​\(f^\)≤ε1​\(f0\)\+2​BP​\(f0\)\+rn​\(f0,δ\),\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+2B\_\{P\}\(f\_\{0\}\)\+r\_\{n\}\(f\_\{0\},\\delta\),\(18\)wherern​\(f0,δ\)→0r\_\{n\}\(f\_\{0\},\\delta\)\\to 0asn→∞n\\to\\inftywithPP,f0f\_\{0\},ℱ\\mathcal\{F\}, andδ\\deltafixed\. Thus, the finite\-sample guarantee recovers the population bound in Proposition[7](https://arxiv.org/html/2610.20903#Thmtheorem7)up to a vanishing estimation term\. An explicit form ofrn​\(f0,δ\)r\_\{n\}\(f\_\{0\},\\delta\)is given in Corollary[18](https://arxiv.org/html/2610.20903#Thmtheorem18)in the appendix\.

The full proof of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)is given in Appendix[G](https://arxiv.org/html/2610.20903#A7)\. At a high level, its proof follows the ingredients developed in the previous subsection\. To elaborate, the pairwise identity in Lemma[5](https://arxiv.org/html/2610.20903#Thmtheorem5)continues to hold exactly on the sample:

T^​\(f,h\)\+T^​\(h,f\)=Pn​\|f−h\|\.\\widehat\{T\}\(f,h\)\+\\widehat\{T\}\(h,f\)=P\_\{n\}\|f\-h\|\.Hence, mirroring the proof of Proposition[7](https://arxiv.org/html/2610.20903#Thmtheorem7), for anyf0∈ℱf\_\{0\}\\in\\mathcal\{F\}, the empirical optimality off^\\hat\{f\}implies

Pn​\|f^−f0\|≤2​suph∈ℱT^​\(f0,h\)\.P\_\{n\}\|\\hat\{f\}\-f\_\{0\}\|\\leq 2\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f\_\{0\},h\)\.\(19\)Bernstein concentration then upper bounds the right\-hand side in terms ofBP​\(f0\)B\_\{P\}\(f\_\{0\}\)andv⁡\(f0\)v\(f\_\{0\}\), and a further concentration step converts the empirical distance on the left\-hand side in \([19](https://arxiv.org/html/2610.20903#S4.E19)\) into‖f^−f0‖1\\\|\\hat\{f\}\-f\_\{0\}\\\|\_\{1\}\.

#### 4\.4Consequences of the main guarantee

We now specialize Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)to three regimes: the general agnostic setting, realizability, and favorable misspecification\. Each specialization represents a different aspect of the guarantee\.

##### General agnostic guarantee\.

We first consider the general agnostic setting, where we impose no additional relationship betweenη\\etaandℱ\\mathcal\{F\}or any additional condition on theL1L\_\{1\}\-optimal predictorf⋆∈arg~minf∈ℱε1​\(f\)f\_\{\\star\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\), beyond the basic assumptions of Section[2](https://arxiv.org/html/2610.20903#S2)\. By Lemma[6](https://arxiv.org/html/2610.20903#Thmtheorem6),BP​\(f⋆\)≤ε1​\(f⋆\)=ε1,∗\.B\_\{P\}\(f\_\{\\star\}\)\\leq\\varepsilon\_\{1\}\(f\_\{\\star\}\)=\\varepsilon\_\{1,\*\}\.Combining this bound with \([18](https://arxiv.org/html/2610.20903#S4.E18)\) gives the following guarantee\.

###### Corollary 9\(General agnostic guarantee\)\.

Under the assumptions of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), with probability at least1−δ1\-\\delta,

ε1​\(f^\)≤3​ε1,∗\+rn​\(f⋆,δ\),\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+r\_\{n\}\(f\_\{\\star\},\\delta\),\(20\)wherern​\(f⋆,δ\)r\_\{n\}\(f\_\{\\star\},\\delta\)is given explicitly in Corollary[18](https://arxiv.org/html/2610.20903#Thmtheorem18)and satisfiesrn​\(f⋆,δ\)→0r\_\{n\}\(f\_\{\\star\},\\delta\)\\to 0asn→∞n\\to\\inftywithPP,ℱ\\mathcal\{F\}, andδ\\deltafixed\.

Thus, without assuming realizability or any additional property of theL1L\_\{1\}\-optimal predictor, SBR achieves the basic goal \([1](https://arxiv.org/html/2610.20903#S1.E1)\) with approximation factor33\. The estimation term isf⋆f\_\{\\star\}\-dependent and retains variance adaptation through the residual second momentv⁡\(f⋆\)v\(f\_\{\\star\}\)\. The factor33in \([20](https://arxiv.org/html/2610.20903#S4.E20)\) is obtained by optimizing the analysis parameterρ\\rhoin Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), as detailed in Corollary[18](https://arxiv.org/html/2610.20903#Thmtheorem18)\. We return to the optimality of the factor33in Section[5](https://arxiv.org/html/2610.20903#S5)\.

##### Realizable case\.

We next consider the realizable setting, whereη∈ℱ\\eta\\in\\mathcal\{F\}\. Takingf0=ηf\_\{0\}=\\etain Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)givesε1​\(η\)=0\\varepsilon\_\{1\}\(\\eta\)=0andBP​\(η\)=0B\_\{P\}\(\\eta\)=0\. Moreover, by \([16](https://arxiv.org/html/2610.20903#S4.E16)\),

v\(η\)=suph∈ℱ𝔼\[σ2\(X\)𝟏\{h\(X\)≠η\(X\)\}\]≤σ¯2,v\(\\eta\)=\\sup\_\{h\\in\\mathcal\{F\}\}\\EE\\\!\\left\[\\sigma^\{2\}\(X\)\\mathbf\{1\}\\\{h\(X\)\\neq\\eta\(X\)\\\}\\right\]\\leq\\bar\{\\sigma\}^\{2\},whereσ¯2=𝔼⁡\[σ2​\(X\)\]\\bar\{\\sigma\}^\{2\}=\\EE\[\\sigma^\{2\}\(X\)\]is the average conditional label variance\. Thus, under realizability, the approximation terms vanish and the estimation error adapts directly to the conditional label variance\.

###### Corollary 10\(Realizable guarantee\)\.

Under the assumptions of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), suppose that\(P,ℱ\)\(P,\\mathcal\{F\}\)is realizable\. Then, for everyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta, simultaneously for everyρ\>0\\rho\>0,

ε1​\(f^\)\\displaystyle\\varepsilon\_\{1\}\(\\hat\{f\}\)≤\(2\+ρ\)​2​v​\(η\)​log⁡\(2​\|ℱ\|2/δ\)n\+Cρ​log⁡\(2​\|ℱ\|2/δ\)n\\displaystyle\\leq\(2\+\\rho\)\\sqrt\{\\frac\{2v\(\\eta\)\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\+C\_\{\\rho\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\(21\)≤\(2\+ρ\)​2​σ¯2​log⁡\(2​\|ℱ\|2/δ\)n\+Cρ​log⁡\(2​\|ℱ\|2/δ\)n\.\\displaystyle\\leq\(2\+\\rho\)\\sqrt\{\\frac\{2\\bar\{\\sigma\}^\{2\}\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\+C\_\{\\rho\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\.

The second bound in \([21](https://arxiv.org/html/2610.20903#S4.E21)\) recovers, up to constants, the realizable variance\-adaptive rate established by[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6), where the estimation error for their betting loss scales asσ¯2​log⁡\(\|ℱ\|/δ\)/n\+log⁡\(\|ℱ\|/δ\)/n\\sqrt\{\\bar\{\\sigma\}^\{2\}\\log\(\|\\mathcal\{F\}\|/\\delta\)/n\}\+\\log\(\|\\mathcal\{F\}\|/\\delta\)/n\. Moreover, the localized quantityv⁡\(η\)v\(\\eta\)can be smaller than the global average conditional varianceσ¯2\\bar\{\\sigma\}^\{2\}, since it includes label variance only on regions where some predictor inℱ\\mathcal\{F\}disagrees withη\\eta\. Thus, our Sign\-Betting Regression retains the favorable noise adaptation available under realizability while extending the guarantee to misspecified classes\.

##### Favorable misspecification\.

Finally, because the approximation term in Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)depends onBP​\(f0\)B\_\{P\}\(f\_\{0\}\)rather than only onε1​\(f0\)\\varepsilon\_\{1\}\(f\_\{0\}\), the guarantee can be substantially sharper on favorable misspecified instances\. Suppose that anL1L\_\{1\}\-optimal predictorf⋆∈arg~minf∈ℱε1​\(f\)f\_\{\\star\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\)satisfiesBP​\(f⋆\)=0B\_\{P\}\(f\_\{\\star\}\)=0\. Takingf0=f⋆f\_\{0\}=f\_\{\\star\}gives the following result\.

###### Corollary 11\(Favorable misspecification\)\.

Under the assumptions of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), suppose thatf⋆∈arg~minf∈ℱε1​\(f\)f\_\{\\star\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\)satisfiesBP​\(f⋆\)=0B\_\{P\}\(f\_\{\\star\}\)=0\. Then, for everyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta, simultaneously for everyρ\>0\\rho\>0,

ε1​\(f^\)≤ε1,∗\+\(2\+ρ\)​2​v​\(f⋆\)​log⁡\(2​\|ℱ\|2/δ\)n\+Cρ​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\\varepsilon\_\{1,\*\}\+\(2\+\\rho\)\\sqrt\{\\frac\{2v\(f\_\{\\star\}\)\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\+C\_\{\\rho\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\.\(22\)

Thus, even whenε1,∗\>0\\varepsilon\_\{1,\*\}\>0, SBR achieves approximation factor11on such misspecified instances while retaining a variance\-adaptive estimation term\. We give a sufficient condition under which anL1L\_\{1\}\-optimal predictor satisfiesBP​\(f⋆\)=0B\_\{P\}\(f\_\{\\star\}\)=0under genuine misspecification in Appendix[J](https://arxiv.org/html/2610.20903#A10)\.

### 5Optimality of the universal proper approximation factor

Corollary[9](https://arxiv.org/html/2610.20903#Thmtheorem9)shows that SBR achieves approximation factor33while retaining the variance\-adaptive, comparator\-dependent estimation term inherited from Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)\. To characterize whether this factor can be improved by a different proper learner, it is useful to first record a coarser consequence in which the estimation term is uniform over the underlying distribution\. This places the upper bound in the same distribution\-free framework as the lower bound developed below: for any target accuracyε\>0\\varepsilon\>0, the sample size required to guaranteeε1​\(f^\)≤3​ε1,∗\+ε\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+\\varepsilondepends only on\|ℱ\|\|\\mathcal\{F\}\|,ε\\varepsilon, and the confidence levelδ\\delta, and not on the underlying distributionPP\. We then show that no proper learner can replace the factor33by any universal constant strictly smaller than33\.

###### Corollary 12\(Uniform achievability of approximation factor three\)\.

Under the assumptions of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), there exists a universal constantC\>0C\>0such that, for everyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta,

ε1​\(f^\)≤3​ε1,∗\+C​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+C\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.\(23\)

We defer the proof to Appendix[K\.1](https://arxiv.org/html/2610.20903#A11.SS1)\. Note that, unlike Corollary[9](https://arxiv.org/html/2610.20903#Thmtheorem9), the estimation term in \([23](https://arxiv.org/html/2610.20903#S5.E23)\) depends only onnn,\|ℱ\|\|\\mathcal\{F\}\|, andδ\\delta, and is therefore distribution\-independent\. For fixedℱ\\mathcal\{F\}andδ\\delta, it vanishes asn→∞n\\to\\inftyuniformly over all distributionsPP\. Corollary[12](https://arxiv.org/html/2610.20903#Thmtheorem12)shows that approximation factor33is achievable by a proper learner with a distribution\-independent estimation term\. We next show that this factor is sharp: no proper learning algorithm can achieve a universal approximation factor strictly smaller than33, even if the estimation term is allowed to vanish arbitrarily slowly with the sample size\.

###### Theorem 13\(Sharp lower bound for proper learning\)\.

Fix anyC∈\[1,3\)C\\in\[1,3\)\. There exist a measurable input space𝒳\\mathcal\{X\}, a fixed two\-element classℱC⊆\{f:𝒳→\[0,1\]\}\\mathcal\{F\}\_\{C\}\\subseteq\\\{f\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\[0,1\]\\\}, independent of the sample size, and a constantεC\>0\\varepsilon\_\{C\}\>0such that the following holds\. For everyn≥1n\\geq 1and every possibly randomized proper learnerf^n∈ℱC\\widehat\{f\}\_\{n\}\\in\\mathcal\{F\}\_\{C\}based onnni\.i\.d\. observations, there exists a distributionPPon𝒳×\{0,1\}\\mathcal\{X\}\\times\\\{0,1\\\}, with finitely supported marginalPXP\_\{X\}, such that

ℙ⁡\(ε1​\(f^n\)\>C​ε1,∗\+εC\)≥38\.\\PP\\left\(\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)\>C\\varepsilon\_\{1,\*\}\+\\varepsilon\_\{C\}\\right\)\\geq\\frac\{3\}\{8\}\.\(24\)The probability in \([24](https://arxiv.org/html/2610.20903#S5.E24)\) includes any internal randomization of the learner\.

Theorem[13](https://arxiv.org/html/2610.20903#Thmtheorem13)is a lower bound on proper learning itself, rather than on SBR or on any particular proper learning algorithm\. It rules out any uniform high\-probability guarantee of the form

ε1​\(f^n\)≤C​ε1,∗\+rn​\(ℱ,δ\),C∈\[0,3\),\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)\\leq C\\varepsilon\_\{1,\*\}\+r\_\{n\}\(\\mathcal\{F\},\\delta\),\\qquad C\\in\[0,3\),\(25\)where the estimation termrn​\(ℱ,δ\)r\_\{n\}\(\\mathcal\{F\},\\delta\)is independent ofPPand satisfiesrn​\(ℱ,δ\)→0r\_\{n\}\(\\mathcal\{F\},\\delta\)\\to 0asn→∞n\\to\\inftyfor every fixedℱ\\mathcal\{F\}andδ\\delta\. Notably, the impossibility holds regardless of how slowly the estimation termrn​\(ℱ,δ\)r\_\{n\}\(\\mathcal\{F\},\\delta\)vanishes withnn\. Together with Corollary[12](https://arxiv.org/html/2610.20903#Thmtheorem12), this shows that33is the smallest universal approximation factor achievable by proper learners under the distribution\-free, high\-probability criterion considered here\. We defer the proof of Theorem[13](https://arxiv.org/html/2610.20903#Thmtheorem13), together with the formal derivation of the impossibility of \([25](https://arxiv.org/html/2610.20903#S5.E25)\), to Appendix[K\.2](https://arxiv.org/html/2610.20903#A11.SS2)and Appendix[K\.3](https://arxiv.org/html/2610.20903#A11.SS3), respectively\.

### 6Beyond finite classes

The variance\-adaptive guarantee of SBR extends beyond finite hypothesis classes\. The extension is governed by the pseudo\-dimension of the hypothesis classℱ\\mathcal\{F\}and the VC dimension of its induced pairwise comparison class\. Define𝒞ℱ:=\{x↦𝟙\{h\(x\)\>f\(x\)\}:f,h∈ℱ\}\\mathcal\{C\}\_\{\\mathcal\{F\}\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{x\\mapsto\\one\\\{h\(x\)\>f\(x\)\\\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}f,h\\in\\mathcal\{F\}\\\}, and letdpred:=Pdim⁡\(ℱ\)d\_\{\\rm pred\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\operatorname\{Pdim\}\(\\mathcal\{F\}\)be the pseudo\-dimension ofℱ\\mathcal\{F\},dcmp:=VCdim⁡\(𝒞ℱ\)d\_\{\\rm cmp\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\operatorname\{VCdim\}\(\\mathcal\{C\}\_\{\\mathcal\{F\}\}\)be the VC dimension of𝒞ℱ\\mathcal\{C\}\_\{\\mathcal\{F\}\}\.

###### Theorem 14\(Variance\-adaptive guarantee beyond finite classes\)\.

Letℱ\\mathcal\{F\}be a class of measurable functions from𝒳\\mathcal\{X\}to\[0,1\]\[0,1\], supposedpred,dcmp<∞d\_\{\\rm pred\},d\_\{\\rm cmp\}<\\infty, and suppose the minimum in \([10](https://arxiv.org/html/2610.20903#S4.E10)\) is attained\. Writed:=1\+dpred\+dcmpd\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=1\+d\_\{\\rm pred\}\+d\_\{\\rm cmp\}\. Forρ\>0\\rho\>0, letCρC\_\{\\rho\}be as in Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)\. There exists a universal constantc\>0c\>0such that, for everyδ∈\(0,1\)\\delta\\in\(0,1\), lettingαδ:=c​d​log⁡\(e​n\)\+log⁡\(2/δ\)n\\alpha\_\{\\delta\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=c\\,\\frac\{d\\log\(en\)\+\\log\(2/\\delta\)\}\{n\}, with probability at least1−δ1\-\\delta, the following holds simultaneously for everyρ\>0\\rho\>0:

ε1​\(f^\)≤inff0∈ℱ\{ε1​\(f0\)\+\(2\+ρ\)​BP​\(f0\)\+\(2\+ρ\)​2​v​\(f0\)​αδ\}\+Cρ​αδ\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\\inf\_\{f\_\{0\}\\in\\mathcal\{F\}\}\\left\\\{\\varepsilon\_\{1\}\(f\_\{0\}\)\+\(2\+\\rho\)B\_\{P\}\(f\_\{0\}\)\+\(2\+\\rho\)\\sqrt\{2v\(f\_\{0\}\)\\alpha\_\{\\delta\}\}\\right\\\}\+C\_\{\\rho\}\\alpha\_\{\\delta\}\.\(26\)

Thus, the same approximation terms and comparator\-dependent variance adaptation of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)continue to hold for infinite classes satisfying these complexity conditions, with the finite\-class log\-cardinality dependence replaced by a complexity termdd, which captures both prediction and comparison complexity of the hypothesis classℱ\\mathcal\{F\}\. Appendix[L](https://arxiv.org/html/2610.20903#A12)provides the proof and verifies the required complexity bounds for bounded linear, logistic, and sparse prediction classes as examples\.

### 7Conclusion

We studied conditional\-mean regression under model misspecification with populationL1L\_\{1\}error as the target metric\. We showed that several standard regression objectives do not, in general, provide a constant\-factorL1L\_\{1\}oracle guarantee under misspecification\. In contrast, the min–max pairwise estimator of[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16), which we call*Sign\-Betting Regression*, admits high\-probabilityL1L\_\{1\}oracle inequalities with comparator\-dependent estimation terms\. Under realizability, these bounds recover the variance\-adaptive rates established by[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)\. We further showed that the resulting approximation factor33is optimal for proper learning: no proper learner can achieve any universal approximation factor strictly below33in a distribution\-free high\-probabilityL1L\_\{1\}oracle inequality with a vanishing estimation term\. We also extended the upper bound beyond finite classes using prediction and pairwise\-comparison complexity measures\. Experiments in Appendix[M](https://arxiv.org/html/2610.20903#A13)illustrate the behavior of Sign\-Betting Regression in synthetic and real\-data model\-selection settings\. Together, these results provide a sharp statistical characterization of properL1L\_\{1\}conditional\-mean learning under misspecification\.

### References

- Ahsen and Vidyasagar \(2019\)M\. E\. Ahsen and M\. VidyasagarAn approach to one\-bit compressed sensing based on probably approximately correct learning theory\.Journal of Machine Learning Research20\(11\),pp\. 1–23\.Cited by:[§L\.4](https://arxiv.org/html/2610.20903#A12.SS4.SSS0.Px3.p3.2)\.
- Anthony and Bartlett \(2009\)M\. Anthony and P\. L\. BartlettNeural network learning: theoretical foundations\.cambridge university press\.Cited by:[§L\.4](https://arxiv.org/html/2610.20903#A12.SS4.SSS0.Px3.p2.3),[§L\.4](https://arxiv.org/html/2610.20903#A12.SS4.p1.1)\.
- Audibert and Tsybakov \(2007\)J\. Audibert and A\. B\. TsybakovFast learning rates for plug\-in classifiers\.The Annals of Statistics35\(2\),pp\. 608 – 633\.External Links:[Document](https://dx.doi.org/10.1214/009053606000001217),[Link](https://doi.org/10.1214/009053606000001217)Cited by:[Appendix A](https://arxiv.org/html/2610.20903#A1.SS0.SSS0.Px1.p1.6),[§1](https://arxiv.org/html/2610.20903#S1.p2.1)\.
- Ayoubet al\.\(2025\)A\. Ayoub, D\. Szepesvári, A\. Bakhtiari, C\. Szepesvári, and D\. SchuurmansRectifying regression in reinforcement learning\.arXiv preprint arXiv:2510\.00885\.Cited by:[§1](https://arxiv.org/html/2610.20903#S1.p2.1)\.
- Baraudet al\.\(2025\)Y\. Baraud, H\. Halconruy, and G\. MaillardRobust density estimation with the l 1\-loss\. applications to the estimation of a density on the line satisfying a shape constraint\.InAnnales de l’Institut Henri Poincaré \(B\) Probabilités et Statistiques,Vol\.61,pp\. 2098–2148\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px4.p1.1)\.
- Baraud and Maillard \(2025\)Y\. Baraud and G\. MaillardEstimating a regression function under possible heteroscedastic and heavy\-tailed errors\. application to shape\-restricted regression\.arXiv preprint arXiv:2506\.00852\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px1.p1.1),[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px4.p1.1),[Appendix I](https://arxiv.org/html/2610.20903#A9.SS0.SSS0.Px1.p2.2),[Appendix I](https://arxiv.org/html/2610.20903#A9.p1.1),[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px4.p2.1),[§4\.1](https://arxiv.org/html/2610.20903#S4.SS1.p1.1),[§7](https://arxiv.org/html/2610.20903#S7.p1.1),[Abstract](https://arxiv.org/html/2610.20903#abstract1.1)\.
- Bousquetet al\.\(2019\)O\. Bousquet, D\. Kane, and S\. MoranThe optimal approximation factor in density estimation\.InConference on Learning Theory,pp\. 318–341\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px4.p2.1),[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px4.p3.1)\.
- Cuiet al\.\(2024\)G\. Cui, L\. Yuan, N\. Ding, G\. Yao, B\. He, W\. Zhu, Y\. Ni, G\. Xie, R\. Xie, Y\. Lin, Z\. Liu, and M\. SunULTRAFEEDBACK: boosting language models with scaled AI feedback\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 9722–9744\.External Links:[Link](https://proceedings.mlr.press/v235/cui24f.html)Cited by:[§M\.3](https://arxiv.org/html/2610.20903#A13.SS3.SSS0.Px1.p1.1)\.
- Foster and Krishnamurthy \(2021\)D\. J\. Foster and A\. KrishnamurthyEfficient first\-order contextual bandits: prediction, allocation, and triangular discrimination\.Advances in Neural Information Processing Systems \(NeurIPS\),pp\. 18907–18919\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.20903#S3.SS1.p4.1),[§3\.2](https://arxiv.org/html/2610.20903#S3.SS2.p2.1)\.
- Gneiting \(2011\)T\. GneitingMaking and evaluating point forecasts\.Journal of the American Statistical Association106\(494\),pp\. 746–762\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px2.p1.2)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§M\.3](https://arxiv.org/html/2610.20903#A13.SS3.SSS0.Px1.p2.1),[§M\.3](https://arxiv.org/html/2610.20903#A13.SS3.SSS0.Px2.p1.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.In2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 770–778\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.90)Cited by:[§M\.2](https://arxiv.org/html/2610.20903#A13.SS2.SSS0.Px2.p1.1)\.
- Hébert\-Johnsonet al\.\(2018\)U\. Hébert\-Johnson, M\. Kim, O\. Reingold, and G\. RothblumMulticalibration: calibration for the \(computationally\-identifiable\) masses\.InInternational conference on machine learning,pp\. 1939–1948\.Cited by:[Appendix A](https://arxiv.org/html/2610.20903#A1.SS0.SSS0.Px2.p2.3),[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px5.p1.1)\.
- Hosuet al\.\(2020\)V\. Hosu, H\. Lin, T\. Sziranyi, and D\. SaupeKonIQ\-10k: an ecologically valid database for deep learning of blind image quality assessment\.IEEE Transactions on Image Processing29\(\),pp\. 4041–4056\.External Links:[Document](https://dx.doi.org/10.1109/TIP.2020.2967829)Cited by:[§M\.2](https://arxiv.org/html/2610.20903#A13.SS2.SSS0.Px1.p1.1)\.
- Huanget al\.\(2025\)A\. Huang, A\. Block, Q\. Liu, N\. Jiang, A\. Krishnamurthy, and D\. J\. FosterIs best\-of\-n the best of them? coverage, scaling, and optimality in inference\-time alignment\.arXiv preprint arXiv:2503\.21878\.Cited by:[Appendix A](https://arxiv.org/html/2610.20903#A1.SS0.SSS0.Px3.p2.4),[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.20903#S1.p2.1)\.
- Kimet al\.\(2019\)M\. P\. Kim, A\. Ghorbani, and J\. ZouMultiaccuracy: black\-box post\-processing for fairness in classification\.InProceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 247–254\.Cited by:[Appendix A](https://arxiv.org/html/2610.20903#A1.SS0.SSS0.Px2.p2.3),[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px5.p1.1)\.
- Liet al\.\(2026\)Y\. Li, S\. Yoon, E\. Huang, and K\. JunSecond\-order bounds for\[0,1\]\[0,1\]\-valued regression via betting loss\.InProceedings of Thirty Ninth Conference on Learning Theory,S\. Hanneke and T\. Lattimore \(Eds\.\),Proceedings of Machine Learning Research, Vol\.336,pp\. 4674–4721\.External Links:[Link](https://proceedings.mlr.press/v336/li26a.html)Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px3.p1.1),[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px3.p2.1),[Appendix F](https://arxiv.org/html/2610.20903#A6.p1.1),[Appendix F](https://arxiv.org/html/2610.20903#A6.p2.1),[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2610.20903#S3.SS1.p4.1),[§3\.2](https://arxiv.org/html/2610.20903#S3.SS2.p2.1),[§3](https://arxiv.org/html/2610.20903#S3.p1.1),[§4\.4](https://arxiv.org/html/2610.20903#S4.SS4.SSS0.Px2.p2.1),[§7](https://arxiv.org/html/2610.20903#S7.p1.1),[Abstract](https://arxiv.org/html/2610.20903#abstract1.1)\.
- Mey and Loog \(2021\)A\. Mey and M\. LoogConsistency and finite sample behavior of binary class probability estimation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 8967–8974\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px2.p2.1),[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px2.p1.1)\.
- Russakovskyet al\.\(2015\)O\. Russakovsky, J\. Deng, H\. Su, J\. Krause, S\. Satheesh, S\. Ma, Z\. Huang, A\. Karpathy, A\. Khosla, M\. Bernstein, A\. C\. Berg, and L\. Fei\-FeiImageNet Large Scale Visual Recognition Challenge\.International Journal of Computer Vision \(IJCV\)115\(3\),pp\. 211–252\.External Links:[Document](https://dx.doi.org/10.1007/s11263-015-0816-y)Cited by:[§M\.2](https://arxiv.org/html/2610.20903#A13.SS2.SSS0.Px2.p1.1)\.
- Shalev\-Shwartz and Ben\-David \(2014\)S\. Shalev\-Shwartz and S\. Ben\-DavidUnderstanding machine learning: From theory to algorithms\.Cambridge university press\.Cited by:[§L\.4](https://arxiv.org/html/2610.20903#A12.SS4.SSS0.Px3.p2.3),[§L\.4](https://arxiv.org/html/2610.20903#A12.SS4.p1.1)\.
- Tsybakov and Zaiats \(2009\)A\. B\. Tsybakov and V\. ZaiatsIntroduction to nonparametric estimation\.Vol\.11,Springer\.Cited by:[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.20903#S3.SS1.p4.1)\.
- Vapnik \(1968\)V\. VapnikOn the uniform convergence of relative frequencies of events to their probabilities\.InDoklady Akademii Nauk USSR,Vol\.181,pp\. 781–787\.Cited by:[§L\.4](https://arxiv.org/html/2610.20903#A12.SS4.p1.1)\.
- Wainwright \(2019\)M\. J\. WainwrightHigh\-dimensional statistics: a non\-asymptotic viewpoint\.Vol\.48,Cambridge university press\.Cited by:[§1](https://arxiv.org/html/2610.20903#S1.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.20903#S3.SS1.p4.1)\.
- Wellneret al\.\(2013\)J\. Wellneret al\.Weak convergence and empirical processes: with applications to statistics\.Springer Science & Business Media\.Cited by:[§L\.1](https://arxiv.org/html/2610.20903#A12.SS1.p3.1.1),[§L\.1](https://arxiv.org/html/2610.20903#A12.SS1.p3.2.1),[§L\.2](https://arxiv.org/html/2610.20903#A12.SS2.SSS0.Px1.p3.1)\.
- Yatracos \(1985\)Y\. G\. YatracosRates of convergence of minimum distance estimators and kolmogorov’s entropy\.The Annals of Statistics13\(2\),pp\. 768–774\.Cited by:[Appendix B](https://arxiv.org/html/2610.20903#A2.SS0.SSS0.Px4.p1.1)\.

## Appendix

### Appendix ADownstream implications ofL1L\_\{1\}conditional\-mean accuracy

For completeness, we record several elementary consequences of controlling theL1L\_\{1\}error

ε1\(f\):=𝔼\[\|f\(X\)−η\(X\)\|\],η\(x\):=𝔼\[Y∣X=x\]\.\\varepsilon\_\{1\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\|f\(X\)\-\\eta\(X\)\|\\right\],\\qquad\\eta\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\[Y\\mid X=x\]\.These observations motivate our choice of performance criterion\.

##### Binary classification\.

SupposeY∈\{0,1\}Y\\in\\\{0,1\\\}, so thatη⁡\(x\)=ℙ⁡\(Y=1∣X=x\)\\eta\(x\)=\\PP\(Y=1\\mid X=x\)\. Let

gf\(x\):=𝟏\{f\(x\)≥1/2\},g⋆\(x\):=𝟏\{η\(x\)≥1/2\}\.g\_\{f\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathbf\{1\}\\\{f\(x\)\\geq 1/2\\\},\\qquad g^\{\\star\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathbf\{1\}\\\{\\eta\(x\)\\geq 1/2\\\}\.Then

Rcls​\(gf\)−Rcls​\(g⋆\)≤2​ε1​\(f\),R\_\{\\mathrm\{cls\}\}\(g\_\{f\}\)\-R\_\{\\mathrm\{cls\}\}\(g^\{\\star\}\)\\leq 2\\varepsilon\_\{1\}\(f\),\(27\)whereRcls​\(g\):=ℙ⁡\(g⁡\(X\)≠Y\)R\_\{\\mathrm\{cls\}\}\(g\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\PP\(g\(X\)\\neq Y\)\. Indeed, the standard excess\-risk identity gives

Rcls\(gf\)−Rcls\(g⋆\)=𝔼\[\|2η\(X\)−1\|𝟏\{gf\(X\)≠g⋆\(X\)\}\]\.R\_\{\\mathrm\{cls\}\}\(g\_\{f\}\)\-R\_\{\\mathrm\{cls\}\}\(g^\{\\star\}\)=\\EE\\\!\\left\[\|2\\eta\(X\)\-1\|\\mathbf\{1\}\\\{g\_\{f\}\(X\)\\neq g^\{\\star\}\(X\)\\\}\\right\]\.Whenevergf​\(X\)≠g⋆​\(X\)g\_\{f\}\(X\)\\neq g^\{\\star\}\(X\), the valuesf⁡\(X\)f\(X\)andη⁡\(X\)\\eta\(X\)lie on opposite sides of1/21/2, and hence

\|η⁡\(X\)−1/2\|≤\|η⁡\(X\)−f⁡\(X\)\|\.\|\\eta\(X\)\-1/2\|\\leq\|\\eta\(X\)\-f\(X\)\|\.Therefore,

\|2η\(X\)−1\|𝟏\{gf\(X\)≠g⋆\(X\)\}≤2\|η\(X\)−f\(X\)\|,\|2\\eta\(X\)\-1\|\\mathbf\{1\}\\\{g\_\{f\}\(X\)\\neq g^\{\\star\}\(X\)\\\}\\leq 2\|\\eta\(X\)\-f\(X\)\|,which proves \([27](https://arxiv.org/html/2610.20903#A1.E27)\); see also[Audibert and Tsybakov \(2007\)](https://arxiv.org/html/2610.20903#bib.bib12)\.

##### Calibration and subpopulation prediction errors\.

Consider the populationL1L\_\{1\}calibration error

CE1​\(f\):=𝔼⁡\[\|𝔼⁡\[Y∣f⁡\(X\)\]−f⁡\(X\)\|\]\.\\mathrm\{CE\}\_\{1\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\\left\|\\EE\[Y\\mid f\(X\)\]\-f\(X\)\\right\|\\right\]\.Since

𝔼⁡\[Y∣f⁡\(X\)\]=𝔼⁡\[η⁡\(X\)∣f⁡\(X\)\],\\EE\[Y\\mid f\(X\)\]=\\EE\[\\eta\(X\)\\mid f\(X\)\],conditional Jensen’s inequality yields

CE1​\(f\)\\displaystyle\\mathrm\{CE\}\_\{1\}\(f\)=𝔼⁡\[\|𝔼⁡\[η⁡\(X\)−f⁡\(X\)∣f⁡\(X\)\]\|\]\\displaystyle=\\EE\\\!\\left\[\\left\|\\EE\[\\eta\(X\)\-f\(X\)\\mid f\(X\)\]\\right\|\\right\]≤𝔼⁡\|η⁡\(X\)−f⁡\(X\)\|=ε1​\(f\)\.\\displaystyle\\leq\\EE\|\\eta\(X\)\-f\(X\)\|=\\varepsilon\_\{1\}\(f\)\.\(28\)
More generally, for every measurable subpopulationG⊆𝒳G\\subseteq\\mathcal\{X\},

\|𝔼\[\(Y−f\(X\)\)𝟏\{X∈G\}\]\|\\displaystyle\\left\|\\EE\\\!\\left\[\(Y\-f\(X\)\)\\mathbf\{1\}\\\{X\\in G\\\}\\right\]\\right\|=\|𝔼\[\(η\(X\)−f\(X\)\)𝟏\{X∈G\}\]\|\\displaystyle=\\left\|\\EE\\\!\\left\[\(\\eta\(X\)\-f\(X\)\)\\mathbf\{1\}\\\{X\\in G\\\}\\right\]\\right\|≤𝔼\[\|η\(X\)−f\(X\)\|𝟏\{X∈G\}\]\\displaystyle\\leq\\EE\\\!\\left\[\|\\eta\(X\)\-f\(X\)\|\\mathbf\{1\}\\\{X\\in G\\\}\\right\]≤ε1​\(f\)\.\\displaystyle\\leq\\varepsilon\_\{1\}\(f\)\.\(29\)Equivalently, wheneverℙ⁡\(X∈G\)\>0\\PP\(X\\in G\)\>0,

\|𝔼⁡\[Y−f⁡\(X\)∣X∈G\]\|≤ε1​\(f\)ℙ⁡\(X∈G\)\.\\left\|\\EE\[Y\-f\(X\)\\mid X\\in G\]\\right\|\\leq\\frac\{\\varepsilon\_\{1\}\(f\)\}\{\\PP\(X\\in G\)\}\.Thus, smallL1L\_\{1\}conditional\-mean error controls both global calibration error and aggregate residual errors over arbitrary subpopulations\. These properties are closely related to the objectives studied in multiaccuracy and multicalibration\([Hébert\-Johnson et al\., 2018](https://arxiv.org/html/2610.20903#bib.bib13);[Kim et al\., 2019](https://arxiv.org/html/2610.20903#bib.bib20)\)\.

##### Best\-of\-NNselection\.

We next consider a setting in whichffserves as a reward model for selecting among a collection of candidate responses\. LetX1,…,XNX\_\{1\},\\ldots,X\_\{N\}be candidate responses, and assume that eachXiX\_\{i\}has marginal distributionPXP\_\{X\}\. Independence among the candidates is not required\. Define

i⋆∈arg~maxi∈\[N\]η​\(Xi\),i^∈arg~maxi∈\[N\]f​\(Xi\),i^\{\\star\}\\in\\argmax\_\{i\\in\[N\]\}\\eta\(X\_\{i\}\),\\qquad\\widehat\{i\}\\in\\argmax\_\{i\\in\[N\]\}f\(X\_\{i\}\),with arbitrary tie\-breaking\. Then

𝔼⁡\[η⁡\(Xi⋆\)−η⁡\(Xi^\)\]≤N​ε1​\(f\)\.\\EE\\\!\\left\[\\eta\(X\_\{i^\{\\star\}\}\)\-\\eta\(X\_\{\\widehat\{i\}\}\)\\right\]\\leq N\\varepsilon\_\{1\}\(f\)\.\(30\)
To see this, by the definition ofi^\\widehat\{i\},

f⁡\(Xi⋆\)−f⁡\(Xi^\)≤0\.f\(X\_\{i^\{\\star\}\}\)\-f\(X\_\{\\widehat\{i\}\}\)\\leq 0\.Hence, samplewise,

η⁡\(Xi⋆\)−η⁡\(Xi^\)\\displaystyle\\eta\(X\_\{i^\{\\star\}\}\)\-\\eta\(X\_\{\\widehat\{i\}\}\)=η⁡\(Xi⋆\)−f⁡\(Xi⋆\)\+f⁡\(Xi⋆\)−f⁡\(Xi^\)\+f⁡\(Xi^\)−η⁡\(Xi^\)\\displaystyle=\\eta\(X\_\{i^\{\\star\}\}\)\-f\(X\_\{i^\{\\star\}\}\)\+f\(X\_\{i^\{\\star\}\}\)\-f\(X\_\{\\widehat\{i\}\}\)\+f\(X\_\{\\widehat\{i\}\}\)\-\\eta\(X\_\{\\widehat\{i\}\}\)≤\|η⁡\(Xi⋆\)−f⁡\(Xi⋆\)\|\+\|η⁡\(Xi^\)−f⁡\(Xi^\)\|\\displaystyle\\leq\|\\eta\(X\_\{i^\{\\star\}\}\)\-f\(X\_\{i^\{\\star\}\}\)\|\+\|\\eta\(X\_\{\\widehat\{i\}\}\)\-f\(X\_\{\\widehat\{i\}\}\)\|≤∑i=1N\|η⁡\(Xi\)−f⁡\(Xi\)\|\.\\displaystyle\\leq\\sum\_\{i=1\}^\{N\}\|\\eta\(X\_\{i\}\)\-f\(X\_\{i\}\)\|\.Taking expectations and using the common marginal distribution of theXiX\_\{i\}’s gives

𝔼⁡\[η⁡\(Xi⋆\)−η⁡\(Xi^\)\]≤∑i=1N𝔼\|η⁡\(Xi\)−f⁡\(Xi\)\|=N​ε1​\(f\)\.\\EE\\\!\\left\[\\eta\(X\_\{i^\{\\star\}\}\)\-\\eta\(X\_\{\\widehat\{i\}\}\)\\right\]\\leq\\sum\_\{i=1\}^\{N\}\\EE\|\\eta\(X\_\{i\}\)\-f\(X\_\{i\}\)\|=N\\varepsilon\_\{1\}\(f\)\.Thus,L1L\_\{1\}accuracy of a reward model yields a direct guarantee for Best\-of\-NNselection\. The bound requires only control of the marginal prediction error of each candidate and does not require independence among the candidates\. This provides one connection between the regression criterion studied here and recent analyses of inference\-time selection with imperfect reward models\([Huang et al\., 2025](https://arxiv.org/html/2610.20903#bib.bib14)\)\.

### Appendix BRelated work

##### Closest prior work: fixed\-design sign\-based regression\.

The closest work to ours is[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16), who introduce the same minimax signed\-residual estimator, up to normalization and sign conventions, in a fixed\-design regression setting\. Their design points are deterministic, while the errors are independent but may be non\-identically distributed, heteroscedastic, and heavy\-tailed; only a finite moment of orderp∈\[1,2\]p\\in\[1,2\]is required\. They establish nonasymptotic oracle inequalities for the expectedL1L\_\{1\}error evaluated on the observed design points\. Their bounds already accommodate model misspecification, incur an approximation factor three, and adapt to averaged noise moments and to comparator\-dependent complexity of the associated comparison sets\.

Our statistical target and questions are different\. We consider i\.i\.d\. random design and study the population error‖f−η‖L1​\(PX\)\\\|f\-\\eta\\\|\_\{L\_\{1\}\(P\_\{X\}\)\}rather than error restricted to the observed design points\. We establish distribution\-free, high\-probability population oracle inequalities whose second\-order terms retain both directional approximation geometry and disagreement\-localized residual moments\. In addition, we study the information\-theoretic approximation factor achievable by arbitrary proper learners and prove that no proper learner can universally improve on factor three under our high\-probability learning criterion\. Appendix[I](https://arxiv.org/html/2610.20903#A9)gives a more detailed comparison\. In particular, conditioning on the random covariates and applying the fixed\-design result as a black box, followed by ordinary uniform convergence, yields a population bound with an additional variance\-independentn−1/2n^\{\-1/2\}generalization term\. Our direct random\-design analysis avoids this crude global penalty through a localized treatment of the empirical\-to\-population transition\.

##### Conditional\-mean estimation and regression under misspecification\.

Our target,

ε1​\(f\)=𝔼⁡\|f⁡\(X\)−η⁡\(X\)\|,η⁡\(x\)=𝔼⁡\[Y∣X=x\],\\varepsilon\_\{1\}\(f\)=\\EE\|f\(X\)\-\\eta\(X\)\|,\\qquad\\eta\(x\)=\\EE\[Y\\mid X=x\],should be distinguished from minimizing an observable loss such as𝔼⁡\|Y−f⁡\(X\)\|\\EE\|Y\-f\(X\)\|\. The distinction between a statistical functional and the loss used to elicit it is classical: squared loss elicits the conditional mean, whereas absolute loss elicits a conditional median\([Gneiting, 2011](https://arxiv.org/html/2610.20903#bib.bib11)\)\. More generally, losses that correctly identify the conditional mean under unrestricted optimization can induce a different approximation geometry once optimization is restricted to a misspecified class\. For example, squared\-loss minimization projectsη\\etaonto the model class in squaredL2L\_\{2\}distance rather than inL1L\_\{1\}distance\.

Finite\-sampleL1L\_\{1\}estimation of binary class probabilities through empirical risk minimization with probability\-estimation losses was studied by[Mey and Loog \(2021\)](https://arxiv.org/html/2610.20903#bib.bib15), including a discussion of model misspecification\. Their perspective starts from a surrogate loss and studies the probability estimate induced by minimizing that loss\. Under misspecification, the resulting population optimizer is therefore determined by the geometry of the chosen surrogate\. Our benchmark instead fixes the target metric first and asks to compete directly with

ε1,∗=inff∈ℱ‖η−f‖L1​\(PX\)\.\\varepsilon\_\{1,\*\}=\\inf\_\{f\\in\\mathcal\{F\}\}\\\|\\eta\-f\\\|\_\{L\_\{1\}\(P\_\{X\}\)\}\.This distinction motivates our negative results for standard regression objectives: a loss may be well suited for estimating the conditional mean under realizability while failing to provide any universal constant\-factor approximation to the bestL1L\_\{1\}predictor in a misspecified class\.

##### First\- and second\-order regression guarantees\.

A related line of work studies statistical guarantees that improve when the underlying prediction or decision problem is favorable\. In cost\-sensitive classification and contextual bandits,[Foster and Krishnamurthy \(2021\)](https://arxiv.org/html/2610.20903#bib.bib1)show that log\-loss regression and triangular discrimination can yield first\-order guarantees whose leading term adapts to the performance of the optimal decision rule\. For bounded regression,[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)introduce a betting loss and establish variance\-adaptive, or second\-order, guarantees, improving on first\-order dependence by adapting directly to label variance without requiring that variance to be known in advance\. The present work asks how such favorable second\-order behavior interacts with*model misspecification*\. In particular, the relevant second\-order quantity under misspecification is no longer determined by label variance alone: our bound also reflects the approximation residual and the regions on which competing predictors disagree with a comparator\. At the same time, we require linear control of the bestL1L\_\{1\}approximation error\. As shown in Section[3](https://arxiv.org/html/2610.20903#S3), these two goals need not be inherited by regression objectives that have favorable guarantees in realizable settings\.

Although we use the term*sign betting*, SBR should also be distinguished from the betting loss of[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)\. The two objectives are different\. SBR compares a candidateffagainst competitorshhthrough signed residual statistics of the form

\(Y−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\),\(Y\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\),and minimizes the worst such comparison\. The resulting population identity directly exposes theL1L\_\{1\}distance between competing predictors\. This pairwise comparison structure is what permits the misspecified oracle analysis developed here\.

##### Minimum\-distance estimation, pairwise testing, and the factor three\.

The pairwise\-comparison structure of SBR has close antecedents in minimum\-distance and testing\-based estimation\. In density estimation,[Yatracos \(1985\)](https://arxiv.org/html/2610.20903#bib.bib17)developed minimum\-distance procedures based on sets that distinguish pairs of candidate densities, leading to constant\-factorL1L\_\{1\}or total\-variation oracle guarantees\. Related testing\-based ideas underlie more recent robustL1L\_\{1\}density estimators, including[Baraud et al\. \(2025\)](https://arxiv.org/html/2610.20903#bib.bib18), and form part of the methodological lineage of the regression estimator of[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16)\. These methods share the principle that, rather than minimizing the target loss directly, one constructs pairwise tests or signed statistics whose expectations reveal which candidate is closer to the target\.

The appearance of approximation factor three also has a particularly close parallel in density hypothesis selection\.[Bousquet et al\. \(2019\)](https://arxiv.org/html/2610.20903#bib.bib19)show that, when an estimator is required to output one of the candidate densities, factor three is the optimal universal approximation factor in total variation, even for a class containing only two candidates; if improper outputs are allowed, the optimal factor improves to two\. There is a useful metric correspondence between their problem and ours\. For a fixed covariate marginalPXP\_\{X\}, associate a predictorffwith the joint distribution

Qf​\(d​x,d​y\):=PX​\(d​x\)​Bernoulli​\(f⁡\(x\)\)​\(d​y\)\.Q\_\{f\}\(dx,dy\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=P\_\{X\}\(dx\)\\,\\mathrm\{Bernoulli\}\(f\(x\)\)\(dy\)\.Then

TV⁡\(Qf,Qη\)=𝔼PX\|f⁡\(X\)−η⁡\(X\)\|=ε1​\(f\)\.\\operatorname\{TV\}\(Q\_\{f\},Q\_\{\\eta\}\)=\\EE\_\{P\_\{X\}\}\|f\(X\)\-\\eta\(X\)\|=\\varepsilon\_\{1\}\(f\)\.Thus the same factor\-three geometry appears naturally in the two problems\.

The statistical models are nevertheless different\. Density hypothesis selection assumes access to fully specified candidate distributions, whereas our candidates specify conditional\-mean functions and the covariate marginalPXP\_\{X\}is itself unknown and sampled as part of the regression problem\. Moreover, our lower bound concerns a uniform, high\-probability regression oracle inequality with a distribution\-independent vanishing remainder\. Accordingly, our proper\-learning lower bound is proved directly rather than by invoking the density\-estimation result\. The connection to[Bousquet et al\. \(2019\)](https://arxiv.org/html/2610.20903#bib.bib19)should therefore be viewed as identifying a common proper\-versus\-improper approximation phenomenon rather than as an equivalence of the two learning problems\.

##### Calibration, subgroup accuracy, and downstream decision making\.

Finally, theL1L\_\{1\}conditional\-mean criterion has connections to several other prediction objectives\. Smallε1​\(f\)\\varepsilon\_\{1\}\(f\)controls aggregate residuals over every measurable subpopulation, relating it to multiaccuracy\([Kim et al\., 2019](https://arxiv.org/html/2610.20903#bib.bib20)\)and multicalibration\([Hébert\-Johnson et al\., 2018](https://arxiv.org/html/2610.20903#bib.bib13)\)\. For binary responses it also controls the excess error of the plug\-in classifier, while whenffserves as a reward model it controls the regret of selecting the largest predicted reward among a finite collection of candidates\. The latter connects to recent work on the statistical behavior of Best\-of\-NNand other inference\-time selection procedures under imperfect reward models\([Huang et al\., 2025](https://arxiv.org/html/2610.20903#bib.bib14)\)\. These connections motivate the choice ofL1​\(PX\)L\_\{1\}\(P\_\{X\}\)error as our primary statistical target; their elementary derivations are collected in Appendix[A](https://arxiv.org/html/2610.20903#A1)\.

### Appendix CProofs and Additional Results for Squared\-Loss ERM

#### C\.1Proof of Proposition[1](https://arxiv.org/html/2610.20903#Thmtheorem1)

We note that a generic uniform\-convergence analysis of squared loss followed by the conversionε1​\(f\)≤ε2​\(f\)\\varepsilon\_\{1\}\(f\)\\leq\\varepsilon\_\{2\}\(f\)yields only anO\(n−1/4\)O\(n^\{\-1/4\}\)statistical term\. To showO\(n−1/2\)O\(n^\{\-1/2\}\)rate in Proposition[1](https://arxiv.org/html/2610.20903#Thmtheorem1), our proof proceeds by comparing empirical squared risks pairwise and exploiting the fact that the variance of a squared\-loss difference is controlled by theL2L\_\{2\}distance between the corresponding predictors\.

###### Proof\.

Forf,g∈ℱf,g\\in\\mathcal\{F\}, define the pairwise loss differenceΔf,g​\(X,Y\):=\(Y−f⁡\(X\)\)2−\(Y−g⁡\(X\)\)2\\Delta\_\{f,g\}\(X,Y\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(Y\-f\(X\)\)^\{2\}\-\(Y\-g\(X\)\)^\{2\}\. SinceY,f⁡\(X\),g⁡\(X\)∈\[0,1\]Y,f\(X\),g\(X\)\\in\[0,1\], we have\|Δf,g\|≤1\|\\Delta\_\{f,g\}\|\\leq 1, and

Δf,g2=\(f−g\)2​\(f\+g−2​Y\)2≤4​\(f−g\)2\.\\Delta\_\{f,g\}^\{2\}=\(f\-g\)^\{2\}\(f\+g\-2Y\)^\{2\}\\leq 4\(f\-g\)^\{2\}\.HenceVar⁡\(Δf,g\)≤4​‖f−g‖22\\Var\(\\Delta\_\{f,g\}\)\\leq 4\\mathinner\{\\\!\\left\\\|f\-g\\right\\\|\}\_\{2\}^\{2\}\. A one\-sided Bernstein inequality and a union bound over all ordered pairs\(f,g\)∈ℱ2\(f,g\)\\in\\mathcal\{F\}^\{2\}imply that, with probability at least1−δ1\-\\delta, simultaneously for everyf,g∈ℱf,g\\in\\mathcal\{F\},

\(P−Pn\)​Δf,g≤2​2​‖f−g‖2​log⁡\(2​\|ℱ\|2/δ\)n\+43​log⁡\(2​\|ℱ\|2/δ\)n\.\(P\-P\_\{n\}\)\\Delta\_\{f,g\}\\leq 2\\sqrt\{2\}\\,\\mathinner\{\\\!\\left\\\|f\-g\\right\\\|\}\_\{2\}\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\+\\frac\{4\}\{3\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\.\(31\)
Letfsq∗∈arg~minf∈ℱ‖f−η‖2f\_\{\\rm sq\}^\{\*\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{2\}, so that‖fsq∗−η‖2=ε2,∗\\mathinner\{\\\!\\left\\\|f\_\{\\rm sq\}^\{\*\}\-\\eta\\right\\\|\}\_\{2\}=\\varepsilon\_\{2,\*\}\. By the conditional\-mean decomposition in \([2](https://arxiv.org/html/2610.20903#S3.E2)\), for everyf,g∈ℱf,g\\in\\mathcal\{F\},

P​Δf,g=‖f−η‖22−‖g−η‖22\.P\\Delta\_\{f,g\}=\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{2\}^\{2\}\-\\mathinner\{\\\!\\left\\\|g\-\\eta\\right\\\|\}\_\{2\}^\{2\}\.Moreover, empirical optimality off^sq\\widehat\{f\}\_\{\\rm sq\}givesPn​Δf^sq,fsq∗≤0P\_\{n\}\\Delta\_\{\\widehat\{f\}\_\{\\rm sq\},f\_\{\\rm sq\}^\{\*\}\}\\leq 0\. Therefore, on the event \([31](https://arxiv.org/html/2610.20903#A3.E31)\), writingr:=‖f^sq−η‖2r\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathinner\{\\\!\\left\\\|\\widehat\{f\}\_\{\\rm sq\}\-\\eta\\right\\\|\}\_\{2\}ande:=ε2,∗e\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\varepsilon\_\{2,\*\}, we obtain

r2−e2≤2​2​\(r\+e\)​log⁡\(2​\|ℱ\|2/δ\)n\+43​log⁡\(2​\|ℱ\|2/δ\)n,r^\{2\}\-e^\{2\}\\leq 2\\sqrt\{2\}\(r\+e\)\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\+\\frac\{4\}\{3\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\},where we used‖f^sq−fsq∗‖2≤r\+e\\mathinner\{\\\!\\left\\\|\\widehat\{f\}\_\{\\rm sq\}\-f\_\{\\rm sq\}^\{\*\}\\right\\\|\}\_\{2\}\\leq r\+e\.

Ifr≤er\\leq e, the desired conclusion is immediate\. Otherwise, letq:=r−e\>0q\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=r\-e\>0\. Sincer2−e2=q⁡\(r\+e\)r^\{2\}\-e^\{2\}=q\(r\+e\)andr\+e≥qr\+e\\geq q, the preceding inequality yields

q2≤2​2​q​log⁡\(2​\|ℱ\|2/δ\)n\+43​log⁡\(2​\|ℱ\|2/δ\)n\.q^\{2\}\\leq 2\\sqrt\{2\}\\,q\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\+\\frac\{4\}\{3\}\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\.Solving this quadratic inequality gives

q≤\(2\+103\)​log⁡\(2​\|ℱ\|2/δ\)n\.q\\leq\\left\(\\sqrt\{2\}\+\\sqrt\{\\frac\{10\}\{3\}\}\\right\)\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.Consequently,

‖f^sq−η‖2≤ε2,∗\+\(2\+103\)​log⁡\(2​\|ℱ\|2/δ\)n\.\\mathinner\{\\\!\\left\\\|\\widehat\{f\}\_\{\\rm sq\}\-\\eta\\right\\\|\}\_\{2\}\\leq\\varepsilon\_\{2,\*\}\+\\left\(\\sqrt\{2\}\+\\sqrt\{\\frac\{10\}\{3\}\}\\right\)\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.Sinceε1​\(f^sq\)≤‖f^sq−η‖2\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\rm sq\}\)\\leq\\mathinner\{\\\!\\left\\\|\\widehat\{f\}\_\{\\rm sq\}\-\\eta\\right\\\|\}\_\{2\}, this proves the first inequality in \([3](https://arxiv.org/html/2610.20903#S3.E3)\)\.

For the second inequality, letf1∗∈arg~minf∈ℱε1​\(f\)f\_\{1\}^\{\*\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\)\. Becausef1∗f\_\{1\}^\{\*\}andη\\etaare\[0,1\]\[0,1\]\-valued,\|f1∗−η\|≤1\|f\_\{1\}^\{\*\}\-\\eta\|\\leq 1, and hence‖f1∗−η‖22≤‖f1∗−η‖1=ε1,∗\\mathinner\{\\\!\\left\\\|f\_\{1\}^\{\*\}\-\\eta\\right\\\|\}\_\{2\}^\{2\}\\leq\\mathinner\{\\\!\\left\\\|f\_\{1\}^\{\*\}\-\\eta\\right\\\|\}\_\{1\}=\\varepsilon\_\{1,\*\}\. Thereforeε2,∗≤ε1,∗\\varepsilon\_\{2,\*\}\\leq\\sqrt\{\\varepsilon\_\{1,\*\}\}, which completes the proof\.

∎

#### C\.2A population lower bound for squared loss

The following generalizes Proposition[2](https://arxiv.org/html/2610.20903#Thmtheorem2)and implies that theε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}dependence in Eqn\. \([3](https://arxiv.org/html/2610.20903#S3.E3)\) is unavoidable\.

###### Proposition 15\(Sharpness of theε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}dependence\)\.

For everyγ∈\(0,1\)\\gamma\\in\(0,1\)and everyp∈\(0,\(1−γ\)2\)p\\in\(0,\(1\-\\gamma\)^\{2\}\), there exists a noiseless regression problem and a two\-element classℱ⊆\[0,1\]𝒳\\mathcal\{F\}\\subseteq\[0,1\]^\{\\mathcal\{X\}\}such thatε1,∗=p\\varepsilon\_\{1,\*\}=p, while the unique population squared\-loss minimizerfsq∗f\_\{\\rm sq\}^\{\*\}satisfies

ε1​\(fsq∗\)=\(1−γ\)​ε1,∗\.\\varepsilon\_\{1\}\(f\_\{\\rm sq\}^\{\*\}\)=\(1\-\\gamma\)\\sqrt\{\\varepsilon\_\{1,\*\}\}\.\(32\)Moreover, squared\-loss ERM selectsfsq∗f\_\{\\rm sq\}^\{\*\}eventually almost surely asn→∞n\\to\\infty\.

###### Proof\.

Let𝒳=\{a,b\}\\mathcal\{X\}=\\\{a,b\\\}withℙ⁡\(X=b\)=p\\PP\(X=b\)=pandℙ⁡\(X=a\)=1−p\\PP\(X=a\)=1\-p, and letY=0Y=0almost surely\. Thusη≡0\\eta\\equiv 0\. Consider the two predictorsfspike,fflat∈\[0,1\]𝒳f\_\{\\rm spike\},f\_\{\\rm flat\}\\in\[0,1\]^\{\\mathcal\{X\}\}defined byfspike​\(a\)=0f\_\{\\rm spike\}\(a\)=0,fspike​\(b\)=1f\_\{\\rm spike\}\(b\)=1, andfflat​\(x\)=\(1−γ\)​pf\_\{\\rm flat\}\(x\)=\(1\-\\gamma\)\\sqrt\{p\}for bothx∈𝒳x\\in\\mathcal\{X\}\. The latter is\[0,1\]\[0,1\]\-valued becausep<1p<1andγ∈\(0,1\)\\gamma\\in\(0,1\)\.

TheirL1L\_\{1\}approximation errors are

ε1​\(fspike\)=p,ε1​\(fflat\)=\(1−γ\)​p\.\\varepsilon\_\{1\}\(f\_\{\\rm spike\}\)=p,\\qquad\\varepsilon\_\{1\}\(f\_\{\\rm flat\}\)=\(1\-\\gamma\)\\sqrt\{p\}\.Sincep<\(1−γ\)2p<\(1\-\\gamma\)^\{2\}, we havep<\(1−γ\)​pp<\(1\-\\gamma\)\\sqrt\{p\}\. Hencefspikef\_\{\\rm spike\}is the uniqueL1L\_\{1\}\-optimal element ofℱ:=\{fspike,fflat\}\\mathcal\{F\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{f\_\{\\rm spike\},f\_\{\\rm flat\}\\\}andε1,∗=p\\varepsilon\_\{1,\*\}=p\.

On the other hand, because the regression problem is noiseless withη≡0\\eta\\equiv 0, population squared risk equals squaredL2L\_\{2\}error\. For the two predictors,

𝔼⁡\[\(Y−fspike​\(X\)\)2\]=p,𝔼⁡\[\(Y−fflat​\(X\)\)2\]=\(1−γ\)2​p<p\.\\EE\\bigl\[\(Y\-f\_\{\\rm spike\}\(X\)\)^\{2\}\\bigr\]=p,\\qquad\\EE\\bigl\[\(Y\-f\_\{\\rm flat\}\(X\)\)^\{2\}\\bigr\]=\(1\-\\gamma\)^\{2\}p<p\.Thereforefflatf\_\{\\rm flat\}is the unique population squared\-loss minimizer, sofsq∗=fflatf\_\{\\rm sq\}^\{\*\}=f\_\{\\rm flat\}\. Consequently,

ε1​\(fsq∗\)=\(1−γ\)​p=\(1−γ\)​ε1,∗,\\varepsilon\_\{1\}\(f\_\{\\rm sq\}^\{\*\}\)=\(1\-\\gamma\)\\sqrt\{p\}=\(1\-\\gamma\)\\sqrt\{\\varepsilon\_\{1,\*\}\},which proves \([32](https://arxiv.org/html/2610.20903#A3.E32)\)\.

It remains to verify the statement for empirical squared\-loss ERM\. The empirical squared loss offflatf\_\{\\rm flat\}is deterministic and equal to\(1−γ\)2​p\(1\-\\gamma\)^\{2\}p\. Forfspikef\_\{\\rm spike\}, it is

1n∑i=1n𝟏\{Xi=b\},\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{X\_\{i\}=b\\\},which converges almost surely toppby the strong law of large numbers\. Since\(1−γ\)2​p<p\(1\-\\gamma\)^\{2\}p<p, squared\-loss ERM therefore uniquely selectsfflat=fsq∗f\_\{\\rm flat\}=f\_\{\\rm sq\}^\{\*\}for all sufficiently largenn, almost surely\. ∎

In particular, takingγ=1/2\\gamma=1/2recovers Proposition[2](https://arxiv.org/html/2610.20903#Thmtheorem2)\. More generally, becauseγ\\gammacan be arbitrarily small, Proposition[15](https://arxiv.org/html/2610.20903#Thmtheorem15)shows that the coefficient in front ofε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}can be made arbitrarily close to one\. It also implies the absence of any finite universal approximation factor: for any fixedC<∞C<\\infty, choosing, for example,γ=1/2\\gamma=1/2and thenp\>0p\>0sufficiently small givesε1​\(fsq∗\)\>C​ε1,∗\\varepsilon\_\{1\}\(f\_\{\\rm sq\}^\{\*\}\)\>C\\varepsilon\_\{1,\*\}, and empirical squared\-loss ERM converges almost surely to this population minimizer\.

### Appendix DProofs and Additional Results for Log\-Loss ERM

#### D\.1Proof of Equation \([6](https://arxiv.org/html/2610.20903#S3.E6)\)

By conditioning onXXand using𝔼⁡\[Y∣X\]=η⁡\(X\)\\EE\[Y\\mid X\]=\\eta\(X\),

𝔼⁡\[ℓlog​\(Y,f⁡\(X\)\)\|X\]\\displaystyle\\EE\\\!\\left\[\\ell\_\{\\log\}\(Y,f\(X\)\)\\,\\middle\|\\,X\\right\]=−η⁡\(X\)​log⁡f⁡\(X\)−\(1−η⁡\(X\)\)​log⁡\(1−f⁡\(X\)\),\\displaystyle=\-\\eta\(X\)\\log f\(X\)\-\(1\-\\eta\(X\)\)\\log\(1\-f\(X\)\),𝔼⁡\[ℓlog​\(Y,η⁡\(X\)\)\|X\]\\displaystyle\\EE\\\!\\left\[\\ell\_\{\\log\}\(Y,\\eta\(X\)\)\\,\\middle\|\\,X\\right\]=−η⁡\(X\)​log⁡η⁡\(X\)−\(1−η⁡\(X\)\)​log⁡\(1−η⁡\(X\)\)\.\\displaystyle=\-\\eta\(X\)\\log\\eta\(X\)\-\(1\-\\eta\(X\)\)\\log\(1\-\\eta\(X\)\)\.Therefore,

𝔼⁡\[ℓlog​\(Y,f⁡\(X\)\)−ℓlog​\(Y,η⁡\(X\)\)\|X\]\\displaystyle\\EE\\\!\\left\[\\ell\_\{\\log\}\(Y,f\(X\)\)\-\\ell\_\{\\log\}\(Y,\\eta\(X\)\)\\,\\middle\|\\,X\\right\]=η⁡\(X\)​log​η⁡\(X\)f⁡\(X\)\+\(1−η⁡\(X\)\)​log​1−η⁡\(X\)1−f⁡\(X\)\\displaystyle\\qquad=\\eta\(X\)\\log\\frac\{\\eta\(X\)\}\{f\(X\)\}\+\(1\-\\eta\(X\)\)\\log\\frac\{1\-\\eta\(X\)\}\{1\-f\(X\)\}=kl⁡\(η⁡\(X\),f⁡\(X\)\)\.\\displaystyle\\qquad=\{\\mathrm\{kl\}\}\\bigl\(\\eta\(X\),f\(X\)\\bigr\)\.Taking expectation overXXproves \([6](https://arxiv.org/html/2610.20903#S3.E6)\)\.

The same calculation remains valid whenη⁡\(X\)∈\{0,1\}\\eta\(X\)\\in\\\{0,1\\\}, with the standard convention0​log⁡0=00\\log 0=0\.

#### D\.2Proof of Proposition[3](https://arxiv.org/html/2610.20903#Thmtheorem3)

###### Proof\.

LetM:=\|ℱ\|M\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\|\\mathcal\{F\}\|and define

α:=log⁡\(2​M2/δ\)n\.\\alpha\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{\\log\(2M^\{2\}/\\delta\)\}\{n\}\.Forf∈ℱf\\in\\mathcal\{F\}, write

K⁡\(f\):=𝔼⁡\[kl⁡\(η⁡\(X\),f⁡\(X\)\)\],K\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\{\\mathrm\{kl\}\}\\bigl\(\\eta\(X\),f\(X\)\\bigr\)\\right\],and define the squared Hellinger\-type distance

H2​\(f,η\):=𝔼⁡\[\(f⁡\(X\)−η⁡\(X\)\)2\+\(1−f⁡\(X\)−1−η⁡\(X\)\)2\]\.H^\{2\}\(f,\\eta\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\\bigl\(\\sqrt\{f\(X\)\}\-\\sqrt\{\\eta\(X\)\}\\bigr\)^\{2\}\+\\bigl\(\\sqrt\{1\-f\(X\)\}\-\\sqrt\{1\-\\eta\(X\)\}\\bigr\)^\{2\}\\right\]\.The elementary relation between total variation and Hellinger distance for Bernoulli distributions implies

ε1​\(f\)=𝔼⁡\|f⁡\(X\)−η⁡\(X\)\|≤H⁡\(f,η\)\.\\varepsilon\_\{1\}\(f\)=\\EE\|f\(X\)\-\\eta\(X\)\|\\leq H\(f,\\eta\)\.\(33\)
We first relate the Hellinger distance of the empirical minimizer to the empirical excess log loss\. Forf∈ℱf\\in\\mathcal\{F\}, let

Δf​\(X,Y\):=ℓlog​\(Y,f⁡\(X\)\)−ℓlog​\(Y,η⁡\(X\)\)\.\\Delta\_\{f\}\(X,Y\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\ell\_\{\\log\}\(Y,f\(X\)\)\-\\ell\_\{\\log\}\(Y,\\eta\(X\)\)\.FixX=xX=x, and abbreviatep=η⁡\(x\)p=\\eta\(x\)andq=f⁡\(x\)q=f\(x\)\. Whenp∈\(0,1\)p\\in\(0,1\),

exp⁡\(−12​Δf​\(x,y\)\)=\(qp\)y/2​\(1−q1−p\)\(1−y\)/2\.\\exp\\left\(\-\\frac\{1\}\{2\}\\Delta\_\{f\}\(x,y\)\\right\)=\\left\(\\frac\{q\}\{p\}\\right\)^\{y/2\}\\left\(\\frac\{1\-q\}\{1\-p\}\\right\)^\{\(1\-y\)/2\}\.The right\-hand side is a convex function ofy∈\[0,1\]y\\in\[0,1\]\. Since𝔼⁡\[Y∣X=x\]=p\\EE\[Y\\mid X=x\]=p, convexity and the chord joining its values at00and11give

𝔼\[e−Δf/2\|X=x\]\\displaystyle\\EE\\\!\\left\[e^\{\-\\Delta\_\{f\}/2\}\\,\\middle\|\\,X=x\\right\]≤p​qp\+\(1−p\)​1−q1−p\\displaystyle\\leq p\\sqrt\{\\frac\{q\}\{p\}\}\+\(1\-p\)\\sqrt\{\\frac\{1\-q\}\{1\-p\}\}=p​q\+\(1−p\)​\(1−q\)\.\\displaystyle=\\sqrt\{pq\}\+\\sqrt\{\(1\-p\)\(1\-q\)\}\.The same inequality holds directly whenp∈\{0,1\}p\\in\\\{0,1\\\}, since thenY=pY=palmost surely conditional onX=xX=x\. Therefore,

𝔼e−Δf/2≤1−12H2\(f,η\)≤exp\(−12H2\(f,η\)\)\.\\EE e^\{\-\\Delta\_\{f\}/2\}\\leq 1\-\\frac\{1\}\{2\}H^\{2\}\(f,\\eta\)\\leq\\exp\\left\(\-\\frac\{1\}\{2\}H^\{2\}\(f,\\eta\)\\right\)\.By independence,

𝔼⁡exp⁡\{n2​H2​\(f,η\)−12​∑i=1nΔf​\(Xi,Yi\)\}≤1\.\\EE\\exp\\left\\\{\\frac\{n\}\{2\}H^\{2\}\(f,\\eta\)\-\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}\\Delta\_\{f\}\(X\_\{i\},Y\_\{i\}\)\\right\\\}\\leq 1\.Hence, by Markov’s inequality,

ℙ⁡\(H2​\(f,η\)\>Pn​Δf\+2​α\)≤e−n​α\.\\PP\\left\(H^\{2\}\(f,\\eta\)\>P\_\{n\}\\Delta\_\{f\}\+2\\alpha\\right\)\\leq e^\{\-n\\alpha\}\.A union bound overf∈ℱf\\in\\mathcal\{F\}shows that, with probability at least1−δ/21\-\\delta/2,

H2​\(f,η\)≤Pn​Δf\+2​αfor every​f∈ℱ\.H^\{2\}\(f,\\eta\)\\leq P\_\{n\}\\Delta\_\{f\}\+2\\alpha\\qquad\\text\{for every \}f\\in\\mathcal\{F\}\.\(34\)Indeed,M​e−n​α=δ/\(2​M\)≤δ/2Me^\{\-n\\alpha\}=\\delta/\(2M\)\\leq\\delta/2\.

On this event, empirical optimality off^log\\widehat\{f\}\_\{\\log\}implies that, for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

H2​\(f^log,η\)≤Pn​Δf^log\+2​α≤Pn​Δf0\+2​α\.H^\{2\}\(\\widehat\{f\}\_\{\\log\},\\eta\)\\leq P\_\{n\}\\Delta\_\{\\widehat\{f\}\_\{\\log\}\}\+2\\alpha\\leq P\_\{n\}\\Delta\_\{f\_\{0\}\}\+2\\alpha\.\(35\)
It remains to control the empirical excess loss of the comparator\. Fixf0∈ℱf\_\{0\}\\in\\mathcal\{F\}and write

K0:=K\(f0\),B:=log1γ\.K\_\{0\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=K\(f\_\{0\}\),\\qquad B\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\log\\frac\{1\}\{\\gamma\}\.Sincef0​\(X\)∈\[γ,1−γ\]f\_\{0\}\(X\)\\in\[\\gamma,1\-\\gamma\], we have

Δf0≤BandP​Δf0=K0\.\\Delta\_\{f\_\{0\}\}\\leq B\\qquad\\text\{and\}\\qquad P\\Delta\_\{f\_\{0\}\}=K\_\{0\}\.\(36\)We next show that

P​Δf02≤\(B\+2\)​K0\.P\\Delta\_\{f\_\{0\}\}^\{2\}\\leq\(B\+2\)K\_\{0\}\.\(37\)
To prove \([37](https://arxiv.org/html/2610.20903#A4.E37)\), again condition onX=xX=xand writep=η⁡\(x\)p=\\eta\(x\)andq=f0​\(x\)q=f\_\{0\}\(x\)\. SinceΔf0​\(x,y\)\\Delta\_\{f\_\{0\}\}\(x,y\)is affine inyy, its square is convex, and therefore

𝔼⁡\[Δf02\|X=x\]≤\\displaystyle\\EE\\\!\\left\[\\Delta\_\{f\_\{0\}\}^\{2\}\\,\\middle\|\\,X=x\\right\]\\leq\{\}p​log2​pq\+\(1−p\)​log2​1−p1−q\.\\displaystyle p\\log^\{2\}\\frac\{p\}\{q\}\+\(1\-p\)\\log^\{2\}\\frac\{1\-p\}\{1\-q\}\.We use the elementary inequality

t​log2​t≤\(B\+2\)​\(t​log⁡t−t\+1\),0≤t≤eB\.t\\log^\{2\}t\\leq\(B\+2\)\\bigl\(t\\log t\-t\+1\\bigr\),\\qquad 0\\leq t\\leq e^\{B\}\.\(38\)For completeness, ift≤1t\\leq 1, writingu=−log⁡tu=\-\\log tand usingeu≥1\+u\+u2/2e^\{u\}\\geq 1\+u\+u^\{2\}/2gives

t​log2​t≤2​\(t​log⁡t−t\+1\)\.t\\log^\{2\}t\\leq 2\(t\\log t\-t\+1\)\.Ift≥1t\\geq 1, writeu=log⁡t∈\[0,B\]u=\\log t\\in\[0,B\]\. Since

u−2\+\(u\+2\)​e−u≥0,u\-2\+\(u\+2\)e^\{\-u\}\\geq 0,we have

u2≤\(u\+2\)​\(u−1\+e−u\)≤\(B\+2\)​\(u−1\+e−u\),u^\{2\}\\leq\(u\+2\)\(u\-1\+e^\{\-u\}\)\\leq\(B\+2\)\(u\-1\+e^\{\-u\}\),which is equivalent to \([38](https://arxiv.org/html/2610.20903#A4.E38)\)\.

Applying \([38](https://arxiv.org/html/2610.20903#A4.E38)\) tot=p/qt=p/qandt=\(1−p\)/\(1−q\)t=\(1\-p\)/\(1\-q\), which are both at most1/γ=eB1/\\gamma=e^\{B\}, yields

p​log2​pq\\displaystyle p\\log^\{2\}\\frac\{p\}\{q\}≤\(B\+2\)​\[p​log⁡pq−p\+q\],\\displaystyle\\leq\(B\+2\)\\left\[p\\log\\frac\{p\}\{q\}\-p\+q\\right\],\(1−p\)​log2​1−p1−q\\displaystyle\(1\-p\)\\log^\{2\}\\frac\{1\-p\}\{1\-q\}≤\(B\+2\)​\[\(1−p\)​log⁡1−p1−q−\(1−p\)\+\(1−q\)\]\.\\displaystyle\\leq\(B\+2\)\\left\[\(1\-p\)\\log\\frac\{1\-p\}\{1\-q\}\-\(1\-p\)\+\(1\-q\)\\right\]\.Summing the two inequalities gives

𝔼⁡\[Δf02\|X=x\]≤\(B\+2\)​kl​\(p,q\)\.\\EE\\\!\\left\[\\Delta\_\{f\_\{0\}\}^\{2\}\\,\\middle\|\\,X=x\\right\]\\leq\(B\+2\)\{\\mathrm\{kl\}\}\(p,q\)\.Taking expectation overXXproves \([37](https://arxiv.org/html/2610.20903#A4.E37)\)\.

A one\-sided Bernstein inequality, followed by a union bound overf0∈ℱf\_\{0\}\\in\\mathcal\{F\}, now implies that, with probability at least1−δ/21\-\\delta/2, simultaneously for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

Pn​Δf0≤K0\+2​\(B\+2\)​K0​α\+2​B3​α\.P\_\{n\}\\Delta\_\{f\_\{0\}\}\\leq K\_\{0\}\+\\sqrt\{2\(B\+2\)K\_\{0\}\\alpha\}\+\\frac\{2B\}\{3\}\\alpha\.\(39\)Here we usedVar⁡\(Δf0\)≤P​Δf02\\Var\(\\Delta\_\{f\_\{0\}\}\)\\leq P\\Delta\_\{f\_\{0\}\}^\{2\}together with \([37](https://arxiv.org/html/2610.20903#A4.E37)\)\.

Combining \([35](https://arxiv.org/html/2610.20903#A4.E35)\) and \([39](https://arxiv.org/html/2610.20903#A4.E39)\), we obtain, simultaneously for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

H2​\(f^log,η\)≤K0\+2​\(B\+2\)​K0​α\+\(2\+2​B3\)​α\.H^\{2\}\(\\widehat\{f\}\_\{\\log\},\\eta\)\\leq K\_\{0\}\+\\sqrt\{2\(B\+2\)K\_\{0\}\\alpha\}\+\\left\(2\+\\frac\{2B\}\{3\}\\right\)\\alpha\.\(40\)Since

2\+2​B3≥B\+22,2\+\\frac\{2B\}\{3\}\\geq\\frac\{B\+2\}\{2\},the right\-hand side of \([40](https://arxiv.org/html/2610.20903#A4.E40)\) is at most

\(K0\+\(2\+2​B3\)​α\)2\.\\left\(\\sqrt\{K\_\{0\}\}\+\\sqrt\{\\left\(2\+\\frac\{2B\}\{3\}\\right\)\\alpha\}\\right\)^\{2\}\.Together with \([33](https://arxiv.org/html/2610.20903#A4.E33)\), this yields

ε1​\(f^log\)≤K⁡\(f0\)\+\(2\+2​B3\)​αfor every​f0∈ℱ\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\leq\\sqrt\{K\(f\_\{0\}\)\}\+\\sqrt\{\\left\(2\+\\frac\{2B\}\{3\}\\right\)\\alpha\}\\qquad\\text\{for every \}f\_\{0\}\\in\\mathcal\{F\}\.\(41\)
Finally, we relate the KL approximation error to theL1L\_\{1\}approximation error\. For anyp∈\[0,1\]p\\in\[0,1\]andq∈\[γ,1−γ\]q\\in\[\\gamma,1\-\\gamma\], convexity ofp↦kl⁡\(p,q\)p\\mapsto\{\\mathrm\{kl\}\}\(p,q\), together withkl⁡\(q,q\)=0\{\\mathrm\{kl\}\}\(q,q\)=0, gives

kl⁡\(p,q\)≤\|p−q\|​max⁡\{−log⁡q1−q,−log⁡\(1−q\)q\}\.\{\\mathrm\{kl\}\}\(p,q\)\\leq\|p\-q\|\\max\\left\\\{\\frac\{\-\\log q\}\{1\-q\},\\frac\{\-\\log\(1\-q\)\}\{q\}\\right\\\}\.Becauseq,1−q≥γq,1\-q\\geq\\gamma,

max⁡\{−log⁡q1−q,−log⁡\(1−q\)q\}≤log⁡\(1/γ\)1−γ\.\\max\\left\\\{\\frac\{\-\\log q\}\{1\-q\},\\frac\{\-\\log\(1\-q\)\}\{q\}\\right\\\}\\leq\\frac\{\\log\(1/\\gamma\)\}\{1\-\\gamma\}\.Hence

K⁡\(f\)≤log⁡\(1/γ\)1−γ​ε1​\(f\)for every​f∈ℱ\.K\(f\)\\leq\\frac\{\\log\(1/\\gamma\)\}\{1\-\\gamma\}\\varepsilon\_\{1\}\(f\)\\qquad\\text\{for every \}f\\in\\mathcal\{F\}\.\(42\)
Letf∗∈arg~minf∈ℱε1​\(f\)f\_\{\*\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\), which exists becauseℱ\\mathcal\{F\}is finite\. Substitutingf0=f∗f\_\{0\}=f\_\{\*\}into \([41](https://arxiv.org/html/2610.20903#A4.E41)\) and applying \([42](https://arxiv.org/html/2610.20903#A4.E42)\) gives

ε1​\(f^log\)≤log⁡\(1/γ\)1−γ​ε1,∗\+\(2\+23​log⁡1γ\)​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\leq\\sqrt\{\\frac\{\\log\(1/\\gamma\)\}\{1\-\\gamma\}\\,\\varepsilon\_\{1,\*\}\}\+\\sqrt\{\\left\(2\+\\frac\{2\}\{3\}\\log\\frac\{1\}\{\\gamma\}\\right\)\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.This proves \([7](https://arxiv.org/html/2610.20903#S3.E7)\)\. For fixedγ\\gamma, the display immediately implies

ε1\(f^log\)≲γε1,∗\+log⁡\(\|ℱ\|/δ\)n\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\lesssim\_\{\\gamma\}\\sqrt\{\\varepsilon\_\{1,\*\}\}\+\\sqrt\{\\frac\{\\log\(\|\\mathcal\{F\}\|/\\delta\)\}\{n\}\}\.∎

#### D\.3Proof of Proposition[4](https://arxiv.org/html/2610.20903#Thmtheorem4)

###### Proof\.

Fixp∈\(0,1\)p\\in\(0,1\)\. Let𝒳=\{a,b\}\\mathcal\{X\}=\\\{a,b\\\}with

ℙ⁡\(X=b\)=p,ℙ⁡\(X=a\)=1−p,\\PP\(X=b\)=p,\\qquad\\PP\(X=a\)=1\-p,and let

Y=12almost surely\.Y=\\frac\{1\}\{2\}\\qquad\\text\{almost surely\}\.Thus the problem is noiseless and

η⁡\(a\)=η⁡\(b\)=12\.\\eta\(a\)=\\eta\(b\)=\\frac\{1\}\{2\}\.
Consider the two hypothesesfspikef\_\{\\rm spike\}andfflatf\_\{\\rm flat\}defined by

fspike​\(a\)=12,fspike​\(b\)=34,f\_\{\\rm spike\}\(a\)=\\frac\{1\}\{2\},\\qquad f\_\{\\rm spike\}\(b\)=\\frac\{3\}\{4\},and

fflat​\(x\)=12\+p4,x∈\{a,b\}\.f\_\{\\rm flat\}\(x\)=\\frac\{1\}\{2\}\+\\frac\{\\sqrt\{p\}\}\{4\},\\qquad x\\in\\\{a,b\\\}\.Sincep∈\(0,1\)p\\in\(0,1\), both hypotheses take values in\[1/4,3/4\]\[1/4,3/4\]\. Letℱ=\{fspike,fflat\}\\mathcal\{F\}=\\\{f\_\{\\rm spike\},f\_\{\\rm flat\}\\\}\.

TheirL1L\_\{1\}errors are

ε1​\(fspike\)=p​\|34−12\|=p4,\\varepsilon\_\{1\}\(f\_\{\\rm spike\}\)=p\\left\|\\frac\{3\}\{4\}\-\\frac\{1\}\{2\}\\right\|=\\frac\{p\}\{4\},and

ε1​\(fflat\)=p4\.\\varepsilon\_\{1\}\(f\_\{\\rm flat\}\)=\\frac\{\\sqrt\{p\}\}\{4\}\.Becausep<pp<\\sqrt\{p\}forp∈\(0,1\)p\\in\(0,1\),fspikef\_\{\\rm spike\}is the uniqueL1L\_\{1\}\-optimal hypothesis, and hence

ε1,∗=p4\.\\varepsilon\_\{1,\*\}=\\frac\{p\}\{4\}\.\(43\)
We next compare the population log risks\. Sinceη≡1/2\\eta\\equiv 1/2, the excess population log risk of anyffis𝔼⁡\[kl⁡\(1/2,f⁡\(X\)\)\]\\EE\[\{\\mathrm\{kl\}\}\(1/2,f\(X\)\)\]\. Forfspikef\_\{\\rm spike\},

𝔼⁡\[kl⁡\(12,fspike​\(X\)\)\]\\displaystyle\\EE\\\!\\left\[\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},f\_\{\\rm spike\}\(X\)\\right\)\\right\]=p​kl​\(12,34\)\\displaystyle=p\\,\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},\\frac\{3\}\{4\}\\right\)=−p2​log⁡34\.\\displaystyle=\-\\frac\{p\}\{2\}\\log\\frac\{3\}\{4\}\.\(44\)Forfflatf\_\{\\rm flat\}, using

4​fflat​\(1−fflat\)=1−p4,4f\_\{\\rm flat\}\(1\-f\_\{\\rm flat\}\)=1\-\\frac\{p\}\{4\},we obtain

𝔼⁡\[kl⁡\(12,fflat​\(X\)\)\]\\displaystyle\\EE\\\!\\left\[\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},f\_\{\\rm flat\}\(X\)\\right\)\\right\]=kl⁡\(12,12\+p4\)\\displaystyle=\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},\\frac\{1\}\{2\}\+\\frac\{\\sqrt\{p\}\}\{4\}\\right\)=−12​log⁡\(1−p4\)\.\\displaystyle=\-\\frac\{1\}\{2\}\\log\\left\(1\-\\frac\{p\}\{4\}\\right\)\.\(45\)
To compare the two quantities, considerg⁡\(u\):=−log⁡\(1−u/4\)g\(u\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\-\\log\(1\-u/4\)on\[0,1\]\[0,1\]\. The functionggis strictly convex andg⁡\(0\)=0g\(0\)=0\. Therefore, for everyp∈\(0,1\)p\\in\(0,1\),

−log⁡\(1−p4\)=g⁡\(p\)<p​g​\(1\)=−p​log⁡34\.\-\\log\\left\(1\-\\frac\{p\}\{4\}\\right\)=g\(p\)<pg\(1\)=\-p\\log\\frac\{3\}\{4\}\.Combining this inequality with \([44](https://arxiv.org/html/2610.20903#A4.E44)\)–\([45](https://arxiv.org/html/2610.20903#A4.E45)\) shows that

𝔼⁡\[ℓlog​\(Y,fflat​\(X\)\)\]<𝔼⁡\[ℓlog​\(Y,fspike​\(X\)\)\]\.\\EE\\\!\\left\[\\ell\_\{\\log\}\(Y,f\_\{\\rm flat\}\(X\)\)\\right\]<\\EE\\\!\\left\[\\ell\_\{\\log\}\(Y,f\_\{\\rm spike\}\(X\)\)\\right\]\.Hencefflatf\_\{\\rm flat\}is the unique population log\-loss minimizer:

flog∗=fflat\.f\_\{\\log\}^\{\*\}=f\_\{\\rm flat\}\.ItsL1L\_\{1\}error is therefore

ε1​\(flog∗\)=p4\.\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)=\\frac\{\\sqrt\{p\}\}\{4\}\.Using \([43](https://arxiv.org/html/2610.20903#A4.E43)\),

p4=12​p4=12​ε1,∗,\\frac\{\\sqrt\{p\}\}\{4\}=\\frac\{1\}\{2\}\\sqrt\{\\frac\{p\}\{4\}\}=\\frac\{1\}\{2\}\\sqrt\{\\varepsilon\_\{1,\*\}\},which proves \([8](https://arxiv.org/html/2610.20903#S3.E8)\)\.

For completeness, this population obstruction also applies to empirical log\-loss ERM\. Sinceℱ\\mathcal\{F\}contains only two hypotheses and both are uniformly bounded away from00and11, their log losses are bounded\. The strong law of large numbers therefore implies that their empirical risks converge almost surely to their respective population risks\. Because the population inequality above is strict,

f^log=fflat\\widehat\{f\}\_\{\\log\}=f\_\{\\rm flat\}eventually almost surely\. Consequently,

ε1​\(f^log\)⟶p4=12​ε1,∗almost surely\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\longrightarrow\\frac\{\\sqrt\{p\}\}\{4\}=\\frac\{1\}\{2\}\\sqrt\{\\varepsilon\_\{1,\*\}\}\\qquad\\text\{almost surely\}\.
Finally,

ε1​\(flog∗\)ε1,∗=1p,\\frac\{\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)\}\{\\varepsilon\_\{1,\*\}\}=\\frac\{1\}\{\\sqrt\{p\}\},which diverges asp↓0p\\downarrow 0\. Thus, for any finite constantCC, one can chooseppsufficiently small so thatε1​\(flog∗\)\>C​ε1,∗\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)\>C\\varepsilon\_\{1,\*\}\. Since empirical log\-loss ERM eventually selectsflog∗f\_\{\\log\}^\{\*\}almost surely, it cannot satisfy a universal constant\-factorL1L\_\{1\}oracle guarantee of the form \([1](https://arxiv.org/html/2610.20903#S1.E1)\)\. This failure occurs even thoughℱ⊆\[1/4,3/4\]𝒳\\mathcal\{F\}\\subseteq\[1/4,3/4\]^\{\\mathcal\{X\}\}and the labels are noiseless\. ∎

#### D\.4Failure without uniform boundary control

We next show that the restriction that all hypotheses stay uniformly away from00and11is essential for Proposition[3](https://arxiv.org/html/2610.20903#Thmtheorem3)\. Without such a restriction, the failure of log\-loss ERM can be substantially stronger than theε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}phenomenon in Proposition[4](https://arxiv.org/html/2610.20903#Thmtheorem4): the bestL1L\_\{1\}approximation can have arbitrarily small error, while the population log\-loss minimizer hasL1L\_\{1\}error bounded away from zero\.

###### Proposition 16\(Failure of log\-loss ERM without uniform boundary control\)\.

For everyp∈\(0,1/4\]p\\in\(0,1/4\], there exists a noiseless regression problem and a two\-element classℱ⊆\(0,1\)𝒳\\mathcal\{F\}\\subseteq\(0,1\)^\{\\mathcal\{X\}\}such that

ε1,∗=p\(12−e−1/p\)≤p2,\\varepsilon\_\{1,\*\}=p\\left\(\\frac\{1\}\{2\}\-e^\{\-1/p\}\\right\)\\leq\\frac\{p\}\{2\},while the unique population log\-loss minimizerflog∗f\_\{\\log\}^\{\*\}satisfies

ε1​\(flog∗\)=14\.\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)=\\frac\{1\}\{4\}\.Consequently, there is no functionΨ:\[0,1\]→\[0,∞\)\\Psi\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\[0,1\]\\to\[0,\\infty\)satisfyingΨ⁡\(ε\)→0\\Psi\(\\varepsilon\)\\to 0asε→0\\varepsilon\\to 0for which population log\-loss minimization obeys

ε1​\(flog∗\)≤Ψ⁡\(ε1,∗\)\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)\\leq\\Psi\(\\varepsilon\_\{1,\*\}\)uniformly over finite classesℱ⊆\(0,1\)𝒳\\mathcal\{F\}\\subseteq\(0,1\)^\{\\mathcal\{X\}\}\. Moreover, empirical log\-loss ERM eventually selectsflog∗f\_\{\\log\}^\{\*\}almost surely\.

###### Proof\.

Fixp∈\(0,1/4\]p\\in\(0,1/4\]\. Let𝒳=\{a,b\}\\mathcal\{X\}=\\\{a,b\\\}with

ℙ⁡\(X=b\)=p,ℙ⁡\(X=a\)=1−p,\\PP\(X=b\)=p,\\qquad\\PP\(X=a\)=1\-p,and let

Y=12almost surely\.Y=\\frac\{1\}\{2\}\\qquad\\text\{almost surely\}\.Thus

η⁡\(a\)=η⁡\(b\)=12,\\eta\(a\)=\\eta\(b\)=\\frac\{1\}\{2\},and the regression problem is noiseless\.

Consider the two hypothesesf0f\_\{0\}andgggiven by

f0\(a\)=12,f0\(b\)=e−1/p,f\_\{0\}\(a\)=\\frac\{1\}\{2\},\\qquad f\_\{0\}\(b\)=e^\{\-1/p\},and

g⁡\(x\)=34,x∈\{a,b\}\.g\(x\)=\\frac\{3\}\{4\},\\qquad x\\in\\\{a,b\\\}\.Both hypotheses take values strictly between00and11, so all population log risks are finite\.

TheirL1L\_\{1\}errors are

ε1\(f0\)=p\(12−e−1/p\)≤p2\\varepsilon\_\{1\}\(f\_\{0\}\)=p\\left\(\\frac\{1\}\{2\}\-e^\{\-1/p\}\\right\)\\leq\\frac\{p\}\{2\}and

ε1​\(g\)=14\.\\varepsilon\_\{1\}\(g\)=\\frac\{1\}\{4\}\.Sincep≤1/4p\\leq 1/4ande−1/p\>0e^\{\-1/p\}\>0,

ε1​\(f0\)<14=ε1​\(g\),\\varepsilon\_\{1\}\(f\_\{0\}\)<\\frac\{1\}\{4\}=\\varepsilon\_\{1\}\(g\),sof0f\_\{0\}is the uniqueL1L\_\{1\}\-optimal hypothesis and

ε1,∗=p\(12−e−1/p\)\.\\varepsilon\_\{1,\*\}=p\\left\(\\frac\{1\}\{2\}\-e^\{\-1/p\}\\right\)\.\(46\)In particular,ε1,∗→0\\varepsilon\_\{1,\*\}\\to 0asp↓0p\\downarrow 0\.

We now compare the population log risks\. Sinceη≡1/2\\eta\\equiv 1/2, their excess log risks are given by

K⁡\(f\):=𝔼⁡\[kl⁡\(12,f⁡\(X\)\)\]\.K\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\\\!\\left\[\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},f\(X\)\\right\)\\right\]\.Forf0f\_\{0\},

K⁡\(f0\)\\displaystyle K\(f\_\{0\}\)=pkl\(12,e−1/p\)\\displaystyle=p\\,\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},e^\{\-1/p\}\\right\)=p⁡\[12​log⁡1/2e−1/p\+12​log⁡1/21−e−1/p\]\\displaystyle=p\\left\[\\frac\{1\}\{2\}\\log\\frac\{1/2\}\{e^\{\-1/p\}\}\+\\frac\{1\}\{2\}\\log\\frac\{1/2\}\{1\-e^\{\-1/p\}\}\\right\]=12−plog2−p2log\(1−e−1/p\)\\displaystyle=\\frac\{1\}\{2\}\-p\\log 2\-\\frac\{p\}\{2\}\\log\\left\(1\-e^\{\-1/p\}\\right\)≥12−p​log⁡2\.\\displaystyle\\geq\\frac\{1\}\{2\}\-p\\log 2\.\(47\)Forgg,

K⁡\(g\)\\displaystyle K\(g\)=kl⁡\(12,34\)=12​log⁡43\.\\displaystyle=\{\\mathrm\{kl\}\}\\\!\\left\(\\frac\{1\}\{2\},\\frac\{3\}\{4\}\\right\)=\\frac\{1\}\{2\}\\log\\frac\{4\}\{3\}\.\(48\)Forp≤1/4p\\leq 1/4,

12−p​log⁡2≥12−14​log⁡2\>12​log⁡43\.\\frac\{1\}\{2\}\-p\\log 2\\geq\\frac\{1\}\{2\}\-\\frac\{1\}\{4\}\\log 2\>\\frac\{1\}\{2\}\\log\\frac\{4\}\{3\}\.Combining this with \([47](https://arxiv.org/html/2610.20903#A4.E47)\) and \([48](https://arxiv.org/html/2610.20903#A4.E48)\) yields

K⁡\(f0\)\>K⁡\(g\)\.K\(f\_\{0\}\)\>K\(g\)\.Henceggis the unique population log\-loss minimizer:

Therefore,

ε1​\(flog∗\)=14,\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)=\\frac\{1\}\{4\},\(49\)whereasε1,∗→0\\varepsilon\_\{1,\*\}\\to 0by \([46](https://arxiv.org/html/2610.20903#A4.E46)\)\.

This already rules out any uniform population guarantee of the formε1​\(flog∗\)≤Ψ⁡\(ε1,∗\)\\varepsilon\_\{1\}\(f\_\{\\log\}^\{\*\}\)\\leq\\Psi\(\\varepsilon\_\{1,\*\}\)withΨ⁡\(ε\)→0\\Psi\(\\varepsilon\)\\to 0asε→0\\varepsilon\\to 0\.

Finally, both hypotheses have bounded log loss on this fixed instance, and the population log\-risk gap is strictly positive\. By the strong law of large numbers, their empirical log risks converge almost surely to their respective population risks\. Thus empirical log\-loss ERM selectsggeventually almost surely, and consequently

ε1​\(f^log\)⟶14almost surely\.\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{\\log\}\)\\longrightarrow\\frac\{1\}\{4\}\\qquad\\text\{almost surely\}\.∎

Proposition[16](https://arxiv.org/html/2610.20903#Thmtheorem16)shows that some uniform control away from00and11is necessary for anL1L\_\{1\}oracle guarantee of the form in Proposition[3](https://arxiv.org/html/2610.20903#Thmtheorem3)\. The failure is not caused by allowing predictions to equal exactly00or11: for every fixed instance in the construction, all predictions lie strictly inside\(0,1\)\(0,1\)and all log risks are finite\. Rather, the problem is that theL1L\_\{1\}\-optimal hypothesis is allowed to approach the boundary asε1,∗→0\\varepsilon\_\{1,\*\}\\to 0\. In that regime, an arbitrarily smallL1L\_\{1\}approximation error can still incur a nonvanishing log\-loss penalty, causing log\-loss minimization to prefer a hypothesis whoseL1L\_\{1\}error stays bounded away from zero\.

### Appendix EWhy not absolute loss?

Since our target is theL1L\_\{1\}error‖f−η‖1=𝔼⁡\|f⁡\(X\)−η⁡\(X\)\|\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{1\}=\\EE\|f\(X\)\-\\eta\(X\)\|, it is natural to ask whether one can simply minimize the empirical absolute loss1n​∑i=1n\|Yi−f⁡\(Xi\)\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|Y\_\{i\}\-f\(X\_\{i\}\)\|\. Despite the superficial similarity inL1L\_\{1\}measure, this approach is fundamentally misaligned with our objective: its population counterpart𝔼⁡\|Y−f⁡\(X\)\|\\EE\|Y\-f\(X\)\|targets the conditional median ofY\|XY\\mid X, whereas our target is the conditional meanη⁡\(x\)=𝔼⁡\[Y∣X=x\]\\eta\(x\)=\\EE\[Y\\mid X=x\]\.

Indeed, for eachxx, the conditional absolute riskt↦𝔼⁡\[\|Y−t\|∣X=x\]t\\mapsto\\EE\[\|Y\-t\|\\mid X=x\]is minimized by a conditional median ofY\|X=xY\\mid X=x\. Thus, even with infinite data, absolute\-loss minimization may not recover the best approximation toη\\etainL1L\_\{1\}\. In fact, the failure can occur even under realizability\.

Consider the simplest example whereX≡x0X\\equiv x\_\{0\}andY∼Bernoulli⁡\(p\)Y\\sim\{\\mathrm\{Bernoulli\}\}\(p\)for some0<p<1/20<p<1/2\. Letℱ=\{0,p\}\\mathcal\{F\}=\\\{0,p\\\}consist of two constant predictors\. Sinceη⁡\(x0\)=p∈ℱ\\eta\(x\_\{0\}\)=p\\in\\mathcal\{F\}, we haveε1,∗:=inff∈ℱ‖f−η‖1=0\\varepsilon\_\{1,\*\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\inf\_\{f\\in\\mathcal\{F\}\}\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{1\}=0\. On the other hand, for anyt∈\[0,1\]t\\in\[0,1\], the population absolute risk is

𝔼⁡\|Y−t\|=p⁡\(1−t\)\+\(1−p\)​t=p\+\(1−2​p\)​t\.\\EE\|Y\-t\|=p\(1\-t\)\+\(1\-p\)t=p\+\(1\-2p\)t\.Sincep<1/2p<1/2, this risk is strictly increasing intt, and hence its unique minimizer overℱ\\mathcal\{F\}is00, rather than the true conditional meanpp\.

This example highlights the basic distinction between absolute loss and our sign\-betting objective\. Absolute loss compares a predictor directly with the observed responseYYand therefore inherits the median\-seeking geometry of absolute deviation\. In contrast, sign betting uses pairwise signed residual comparisons designed to recover the geometry of distance to the conditional mean\. We next make this distinction more explicit at the population level\.

##### The sign in absolute loss depends on the observed response\.

Recall that\|Y−f⁡\(X\)\|=\(Y−f⁡\(X\)\)​sign⁡\(Y−f⁡\(X\)\)\|Y\-f\(X\)\|=\(Y\-f\(X\)\)\\sign\(Y\-f\(X\)\)\. Thus absolute loss can equivalently be written as𝔼⁡\[sups∈\[−1,1\]s⁡\(Y−f⁡\(X\)\)\]\\EE\[\\sup\_\{s\\in\[\-1,1\]\}s\(Y\-f\(X\)\)\]\. Importantly, the maximizing direction is chosen separately for each realization of\(X,Y\)\(X,Y\)and therefore may depend on the observed responseYY\. Conditioning onX=xX=x, the resulting objective ist↦𝔼⁡\[\|Y−t\|∣X=x\]t\\mapsto\\EE\[\|Y\-t\|\\mid X=x\], whose minimizers are exactly the conditional medians ofY\|X=xY\\mid X=x\. Hence the full conditional distribution ofY\|XY\\mid Xmatters, rather than only its conditional mean\.

Our sign\-betting objective instead takes the form

L^SB​\(f\):=suph∈ℱ1n​∑i=1n\(Yi−f⁡\(Xi\)\)​sign⁡\(h⁡\(Xi\)−f⁡\(Xi\)\)\.\\widehat\{L\}\_\{\\rm SB\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{h\\in\\mathcal\{F\}\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(Y\_\{i\}\-f\(X\_\{i\}\)\)\\sign\(h\(X\_\{i\}\)\-f\(X\_\{i\}\)\)\.For every fixed pair\(f,h\)\(f,h\), the directionsign⁡\(h⁡\(X\)−f⁡\(X\)\)\\sign\(h\(X\)\-f\(X\)\)is a function ofXXalone and does not depend on the realized response\. Consequently,

𝔼⁡\[\(Y−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\]=𝔼⁡\[\(η⁡\(X\)−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\],\\EE\[\(Y\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\]=\\EE\[\(\\eta\(X\)\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\],where the equality follows from𝔼⁡\[Y−η⁡\(X\)∣X\]=0\\EE\[Y\-\\eta\(X\)\\mid X\]=0\. Thus the population sign\-betting objective

BP​\(f\):=suph∈ℱ𝔼⁡\[\(Y−f⁡\(X\)\)​sign⁡\(h⁡\(X\)−f⁡\(X\)\)\]B\_\{P\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{h\\in\\mathcal\{F\}\}\\EE\[\(Y\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\)\]depends on the conditional distribution ofY\|XY\\mid Xonly through its meanη⁡\(X\)\\eta\(X\)\.

This difference can also be viewed as a difference in the order of the supremum and expectation\. In the scalar case, absolute loss has the form

𝔼⁡\|Y−f\|=𝔼⁡\[sups∈\[−1,1\]s⁡\(Y−f\)\],\\EE\|Y\-f\|=\\EE\\left\[\\sup\_\{s\\in\[\-1,1\]\}s\(Y\-f\)\\right\],where the sign may adapt to each realization ofYY\. In contrast, if the direction must be chosen before observing the response, then

sups∈\[−1,1\]𝔼⁡\[s⁡\(Y−f\)\]=\|𝔼⁡\[Y\]−f\|\.\\sup\_\{s\\in\[\-1,1\]\}\\EE\[s\(Y\-f\)\]=\|\\EE\[Y\]\-f\|\.The former is minimized by a median ofYY, whereas the latter is minimized by its mean\. Sign betting implements the analogous distinction in regression by restricting the betting direction to beXX\-measurable\.

##### A direct comparison on the realizable Bernoulli example\.

Consider again the example at the beginning of this appendix section:X≡x0X\\equiv x\_\{0\},Y∼Bernoulli⁡\(p\)Y\\sim\{\\mathrm\{Bernoulli\}\}\(p\)with0<p<1/20<p<1/2, andℱ=\{0,p\}\\mathcal\{F\}=\\\{0,p\\\}\. Hereη=p\\eta=p\. As shown at the beginning of this section, absolute loss strictly prefers the predictor00, since𝔼⁡\|Y−t\|=p\+\(1−2​p\)​t\\EE\|Y\-t\|=p\+\(1\-2p\)tis increasing intt\.

The sign\-betting population objective makes the opposite choice\. Forf=pf=p, choosingh=ph=pgives the zero direction, while choosingh=0h=0gives

𝔼⁡\[\(Y−p\)​sign⁡\(0−p\)\]=−𝔼⁡\[Y−p\]=0\.\\EE\[\(Y\-p\)\\sign\(0\-p\)\]=\-\\EE\[Y\-p\]=0\.HenceBP​\(p\)=0B\_\{P\}\(p\)=0\. Forf=0f=0, choosingh=ph=pgives

𝔼⁡\[\(Y−0\)​sign⁡\(p−0\)\]=𝔼⁡\[Y\]=p,\\EE\[\(Y\-0\)\\sign\(p\-0\)\]=\\EE\[Y\]=p,and thereforeBP​\(0\)=pB\_\{P\}\(0\)=p\. Thus sign betting uniquely prefersp=ηp=\\eta, while absolute loss uniquely prefers the conditional median00\. In particular,

arg~minf∈ℱ​𝔼​\|Y−f⁡\(X\)\|=\{0\},arg~minf∈ℱBP​\(f\)=\{η\}\.\\argmin\_\{f\\in\\mathcal\{F\}\}\\EE\|Y\-f\(X\)\|=\\\{0\\\},\\qquad\\argmin\_\{f\\in\\mathcal\{F\}\}B\_\{P\}\(f\)=\\\{\\eta\\\}\.
More generally, wheneverη∈ℱ\\eta\\in\\mathcal\{F\}, we haveBP​\(η\)=0B\_\{P\}\(\\eta\)=0\. For anyf∈ℱf\\in\\mathcal\{F\}, usingh=ηh=\\etaas the challenger gives

BP​\(f\)≥𝔼⁡\[\(η⁡\(X\)−f⁡\(X\)\)​sign⁡\(η⁡\(X\)−f⁡\(X\)\)\]=‖f−η‖1\.B\_\{P\}\(f\)\\geq\\EE\[\(\\eta\(X\)\-f\(X\)\)\\sign\(\\eta\(X\)\-f\(X\)\)\]=\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{1\}\.Henceη\\etais the unique population minimizer whenever‖f−η‖1\>0\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{1\}\>0for everyf≠ηf\\neq\\eta\. This is fundamentally different from absolute loss, which can fail to recoverη\\etaeven under realizability\.

##### Connection to the dual representation of theL1L\_\{1\}error\.

The distinction above can also be understood through

‖f−η‖1=sups:𝒳→\[−1,1\]s​measurable𝔼\[\(η\(X\)−f\(X\)\)s\(X\)\]=sups:𝒳→\[−1,1\]s​measurable𝔼\[\(Y−f\(X\)\)s\(X\)\]\.\\mathinner\{\\\!\\left\\\|f\-\\eta\\right\\\|\}\_\{1\}=\\sup\_\{\\begin\{subarray\}\{c\}s\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\[\-1,1\]\\\\ s\\ \{\\rm measurable\}\\end\{subarray\}\}\\EE\[\(\\eta\(X\)\-f\(X\)\)s\(X\)\]=\\sup\_\{\\begin\{subarray\}\{c\}s\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\[\-1,1\]\\\\ s\\ \{\\rm measurable\}\\end\{subarray\}\}\\EE\[\(Y\-f\(X\)\)s\(X\)\]\.The optimal direction iss⁡\(X\)=sign⁡\(η⁡\(X\)−f⁡\(X\)\)s\(X\)=\\sign\(\\eta\(X\)\-f\(X\)\)\. Our sign\-betting objective is a restricted version of this dual formulation, using the comparison directions\{sign\(h−f\):h∈ℱ\}\\\{\\sign\(h\-f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}h\\in\\mathcal\{F\}\\\}available from the model class\. In contrast, absolute loss uses the response\-dependent directionsign⁡\(Y−f⁡\(X\)\)\\sign\(Y\-f\(X\)\), which lead to a target of conditional median\.

### Appendix FFailure of the original betting loss under misspecification

Betting loss was introduced in[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)to obtain variance\-adaptive guarantees in the realizable setting\. Under misspecification, however, it suffers from a similar approximation\-error obstruction as squared and log loss: itsL1L\_\{1\}error can scale asε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}, rather than linearly inε1,∗\\varepsilon\_\{1,\*\}\. This is not merely a limitation of the existing realizable analysis\. The following proposition shows that the original betting loss can have asymptoticL1L\_\{1\}error of orderε1,∗\\sqrt\{\\varepsilon\_\{1,\*\}\}, ruling out any universal constant\-factor agnosticL1L\_\{1\}oracle guarantee\.

Recall the betting objective of[Li et al\. \(2026\)](https://arxiv.org/html/2610.20903#bib.bib6)\. Forf,h∈ℱf,h\\in\\mathcal\{F\},ϕ∈\[0,n/4\]\\phi\\in\[0,n/4\], andc∈\[0,1/4\]c\\in\[0,1/4\], define

bh,fϕ,c​\(x\):=\(ϕ⁡\(h⁡\(x\)−f⁡\(x\)\)\)¯\[−c,c\],b\_\{h,f\}^\{\\phi,c\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\{\\overline\{\\mathinner\{\\left\(\\phi\(h\(x\)\-f\(x\)\)\\right\)\}\}\}\_\{\[\-c,c\]\},and

L^bet,n​\(f\):=maxh∈ℱ⁡maxϕ∈\[0,n/4\]​maxc∈\[0,1/4\]​1n​∑i=1nlog⁡\(1\+\(Yi−f⁡\(Xi\)\)​bh,fϕ,c​\(Xi\)\)\.\\widehat\{L\}\_\{\\rm bet,n\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\max\_\{h\\in\\mathcal\{F\}\}\\max\_\{\\phi\\in\[0,n/4\]\}\\max\_\{c\\in\[0,1/4\]\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\log\\left\(1\+\(Y\_\{i\}\-f\(X\_\{i\}\)\)b\_\{h,f\}^\{\\phi,c\}\(X\_\{i\}\)\\right\)\.The original betting estimator is any empirical minimizerf^bet∈arg~minf∈ℱL^bet,n​\(f\)\\hat\{f\}\_\{\\rm bet\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\widehat\{L\}\_\{\\rm bet,n\}\(f\)\. For the proof, it is useful to introduce the corresponding population objective

Lbet,n​\(f\):=maxh∈ℱ⁡maxϕ∈\[0,n/4\]​maxc∈\[0,1/4\]⁡𝔼⁡\[log⁡\(1\+\(Y−f⁡\(X\)\)​bh,fϕ,c​\(X\)\)\]\.L\_\{\\rm bet,n\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\max\_\{h\\in\\mathcal\{F\}\}\\max\_\{\\phi\\in\[0,n/4\]\}\\max\_\{c\\in\[0,1/4\]\}\\EE\\left\[\\log\\left\(1\+\(Y\-f\(X\)\)b\_\{h,f\}^\{\\phi,c\}\(X\)\\right\)\\right\]\.We keep the subscriptnnbecause the admissible range ofϕ\\phidepends on the sample size\.

###### Proposition 17\.

For everyp∈\(0,1/64\]p\\in\(0,1/64\], there exist a distribution on a two\-point input space and a two\-element classℱ\\mathcal\{F\}such thatε1,∗=p\\varepsilon\_\{1,\*\}=p, but every empirical minimizerf^bet\\hat\{f\}\_\{\\rm bet\}of the original betting objective satisfies

‖f^bet−η‖1⟶1−p4​palmost surely as​n→∞\.\\mathinner\{\\\!\\left\\\|\\hat\{f\}\_\{\\rm bet\}\-\\eta\\right\\\|\}\_\{1\}\\longrightarrow\\frac\{1\-p\}\{4\}\\sqrt\{p\}\\qquad\\text\{almost surely as \}n\\to\\infty\.Consequently, there is no universal constantC<∞C<\\inftysuch that the original betting estimator satisfies

‖f^bet−η‖1≤C​ε1,∗\+on​\(1\)\\mathinner\{\\\!\\left\\\|\\hat\{f\}\_\{\\rm bet\}\-\\eta\\right\\\|\}\_\{1\}\\leq C\\varepsilon\_\{1,\*\}\+o\_\{n\}\(1\)over arbitrary misspecified finite classes\.

###### Proof\.

Fixp∈\(0,1/64\]p\\in\(0,1/64\]and let𝒳=\{a,b\}\\mathcal\{X\}=\\\{a,b\\\}, withℙ⁡\(X=b\)=p\\PP\(X=b\)=pandℙ⁡\(X=a\)=1−p\\PP\(X=a\)=1\-p\. Let

Y∣X=a∼Bernoulli\(1/2\),Y∣X=b=0almost surely\.Y\\mid X=a\\sim\{\\mathrm\{Bernoulli\}\}\(1/2\),\\qquad Y\\mid X=b=0\\quad\\text\{almost surely\}\.Thusη⁡\(a\)=1/2\\eta\(a\)=1/2andη⁡\(b\)=0\\eta\(b\)=0\.

Sett:=p/4t\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sqrt\{p\}/4and consider the two\-element classℱ=\{f0,g\}\\mathcal\{F\}=\\\{f\_\{0\},g\\\}defined by

abη1/20f01/21g1/2\+t0\.\\begin\{array\}\[\]\{c\|cc\}&a&b\\\\ \\hline\\cr\\eta&1/2&0\\\\ f\_\{0\}&1/2&1\\\\ g&1/2\+t&0\.\\end\{array\}TheirL1L\_\{1\}errors are

‖f0−η‖1=p,‖g−η‖1=\(1−p\)​t=1−p4​p\.\\mathinner\{\\\!\\left\\\|f\_\{0\}\-\\eta\\right\\\|\}\_\{1\}=p,\\qquad\\mathinner\{\\\!\\left\\\|g\-\\eta\\right\\\|\}\_\{1\}=\(1\-p\)t=\\frac\{1\-p\}\{4\}\\sqrt\{p\}\.Sincep≤1/64p\\leq 1/64, we have4​p≤1/2<1−p4\\sqrt\{p\}\\leq 1/2<1\-p, and therefore\(1−p\)​t\>p\(1\-p\)t\>p\. Hencef0f\_\{0\}is the uniqueL1L\_\{1\}oracle and

ε1,∗=p\.\\varepsilon\_\{1,\*\}=p\.
We next show that the population betting objective strictly prefersgg\.

##### Step 1: theL1L\_\{1\}oracle has population betting loss larger thanp/5p/5\.

Forf=f0f=f\_\{0\}, choose the challengerh=gh=gand takeϕ=c=1/4\\phi=c=1/4, which is admissible for everyn≥1n\\geq 1\. AtX=aX=a, the bet ist/4t/4, whereas atX=bX=bthe bet is−1/4\-1/4\. Therefore,

Lbet,n​\(f0\)\\displaystyle L\_\{\\rm bet,n\}\(f\_\{0\}\)≥p​log⁡54\+1−p2​log⁡\(1−t264\)\\displaystyle\\geq p\\log\\frac\{5\}\{4\}\+\\frac\{1\-p\}\{2\}\\log\\left\(1\-\\frac\{t^\{2\}\}\{64\}\\right\)=p​log⁡54\+1−p2​log⁡\(1−p1024\)\.\\displaystyle=p\\log\\frac\{5\}\{4\}\+\\frac\{1\-p\}\{2\}\\log\\left\(1\-\\frac\{p\}\{1024\}\\right\)\.Usinglog⁡\(5/4\)≥7/32\\log\(5/4\)\\geq 7/32andlog⁡\(1−u\)≥−2​u\\log\(1\-u\)\\geq\-2uforu∈\[0,1/2\]u\\in\[0,1/2\],

Lbet,n​\(f0\)\\displaystyle L\_\{\\rm bet,n\}\(f\_\{0\}\)≥7​p32−\(1−p\)​p1024\\displaystyle\\geq\\frac\{7p\}\{32\}\-\\frac\{\(1\-p\)p\}\{1024\}≥\(732−11024\)​p\>p5\.\\displaystyle\\geq\\left\(\\frac\{7\}\{32\}\-\\frac\{1\}\{1024\}\\right\)p\>\\frac\{p\}\{5\}\.This bound holds for everyn≥1n\\geq 1\.

##### Step 2: the worseL1L\_\{1\}predictor has population betting loss at mostp/8p/8\.

Forf=gf=g, the challengerh=gh=ggives betting loss zero, so it remains to considerh=f0h=f\_\{0\}\. AtX=bX=b, we haveY−g⁡\(b\)=0Y\-g\(b\)=0, and hence this point contributes zero regardless of the bet\.

AtX=aX=a, writeu:=min⁡\{ϕ​t,c\}∈\[0,1/4\]u\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\min\\\{\\phi t,c\\\}\\in\[0,1/4\]\. Sincef0​\(a\)−g⁡\(a\)=−tf\_\{0\}\(a\)\-g\(a\)=\-t, the bet equals−u\-u\. Conditional onX=aX=a, the expected log payoff is

12​log⁡\(1\+u⁡\(12\+t\)\)\+12​log⁡\(1−u⁡\(12−t\)\)\\displaystyle\\frac\{1\}\{2\}\\log\\left\(1\+u\\left\(\\frac\{1\}\{2\}\+t\\right\)\\right\)\+\\frac\{1\}\{2\}\\log\\left\(1\-u\\left\(\\frac\{1\}\{2\}\-t\\right\)\\right\)=12​log⁡\(1\+2​t​u−\(14−t2\)​u2\)\.\\displaystyle=\\frac\{1\}\{2\}\\log\\left\(1\+2tu\-\\left\(\\frac\{1\}\{4\}\-t^\{2\}\\right\)u^\{2\}\\right\)\.Completing the square gives

1\+2​t​u−\(14−t2\)​u2≤11−4​t2\.1\+2tu\-\\left\(\\frac\{1\}\{4\}\-t^\{2\}\\right\)u^\{2\}\\leq\\frac\{1\}\{1\-4t^\{2\}\}\.Consequently,

Lbet,n​\(g\)\\displaystyle L\_\{\\rm bet,n\}\(g\)≤−1−p2​log⁡\(1−4​t2\)\\displaystyle\\leq\-\\frac\{1\-p\}\{2\}\\log\(1\-4t^\{2\}\)=−1−p2​log⁡\(1−p4\)\.\\displaystyle=\-\\frac\{1\-p\}\{2\}\\log\\left\(1\-\\frac\{p\}\{4\}\\right\)\.Using−log⁡\(1−u\)≤u/\(1−u\)\-\\log\(1\-u\)\\leq u/\(1\-u\),

Lbet,n​\(g\)≤p⁡\(1−p\)8​\(1−p/4\)<p8\.L\_\{\\rm bet,n\}\(g\)\\leq\\frac\{p\(1\-p\)\}\{8\(1\-p/4\)\}<\\frac\{p\}\{8\}\.Thus, uniformly overnn,

Lbet,n​\(f0\)−Lbet,n​\(g\)\>p5−p8=3​p40\.L\_\{\\rm bet,n\}\(f\_\{0\}\)\-L\_\{\\rm bet,n\}\(g\)\>\\frac\{p\}\{5\}\-\\frac\{p\}\{8\}=\\frac\{3p\}\{40\}\.In particular, the population betting objective strictly prefersgg, despitef0f\_\{0\}being the uniqueL1L\_\{1\}oracle\.

##### Step 3: the empirical objective eventually makes the same choice\.

The distribution of\(X,Y\)\(X,Y\)is supported on only three atoms,

\{\(a,0\),\(a,1\),\(b,0\)\}\.\\\{\(a,0\),\(a,1\),\(b,0\)\\\}\.For anyf,h∈ℱf,h\\in\\mathcal\{F\}and any admissible\(ϕ,c\)\(\\phi,c\), define

ℓf,h,ϕ,c​\(x,y\):=log⁡\(1\+\(y−f⁡\(x\)\)​bh,fϕ,c​\(x\)\)\.\\ell\_\{f,h,\\phi,c\}\(x,y\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\log\\left\(1\+\(y\-f\(x\)\)b\_\{h,f\}^\{\\phi,c\}\(x\)\\right\)\.Since\|y−f⁡\(x\)\|≤1\|y\-f\(x\)\|\\leq 1and\|bh,fϕ,c​\(x\)\|≤1/4\|b\_\{h,f\}^\{\\phi,c\}\(x\)\|\\leq 1/4, we have

34≤1\+\(y−f⁡\(x\)\)​bh,fϕ,c​\(x\)≤54,\\frac\{3\}\{4\}\\leq 1\+\(y\-f\(x\)\)b\_\{h,f\}^\{\\phi,c\}\(x\)\\leq\\frac\{5\}\{4\},and hence\|ℓf,h,ϕ,c​\(x,y\)\|≤log⁡\(4/3\)\|\\ell\_\{f,h,\\phi,c\}\(x,y\)\|\\leq\\log\(4/3\)uniformly over allf,h,ϕ,cf,h,\\phi,cand allnn\.

Letq^n​\(z\)\\widehat\{q\}\_\{n\}\(z\)andq⁡\(z\)q\(z\)denote the empirical and population probabilities of an atomzz\. Then

supf,h,ϕ,c\|\(Pn−P\)​ℓf,h,ϕ,c\|\\displaystyle\\sup\_\{f,h,\\phi,c\}\\left\|\(P\_\{n\}\-P\)\\ell\_\{f,h,\\phi,c\}\\right\|≤log\(4/3\)∑z∈\{\(a,0\),\(a,1\),\(b,0\)\}\|q^n\(z\)−q\(z\)\|⟶0almost surely\.\\displaystyle\\leq\\log\(4/3\)\\sum\_\{z\\in\\\{\(a,0\),\(a,1\),\(b,0\)\\\}\}\|\\widehat\{q\}\_\{n\}\(z\)\-q\(z\)\|\\longrightarrow 0\\qquad\\text\{almost surely\}\.Importantly, this bound is uniform over all admissible betting parameters, so the fact that the rangeϕ∈\[0,n/4\]\\phi\\in\[0,n/4\]grows withnncauses no difficulty\.

It follows that

maxf∈ℱ⁡\|L^bet,n​\(f\)−Lbet,n​\(f\)\|⟶0almost surely\.\\max\_\{f\\in\\mathcal\{F\}\}\\left\|\\widehat\{L\}\_\{\\rm bet,n\}\(f\)\-L\_\{\\rm bet,n\}\(f\)\\right\|\\longrightarrow 0\\qquad\\text\{almost surely\}\.Combining this with the uniform population gap above, we obtain

L^bet,n​\(g\)<L^bet,n​\(f0\)\\widehat\{L\}\_\{\\rm bet,n\}\(g\)<\\widehat\{L\}\_\{\\rm bet,n\}\(f\_\{0\}\)for all sufficiently largenn, almost surely\. Hence every empirical minimizer satisfiesf^bet=g\\hat\{f\}\_\{\\rm bet\}=geventually almost surely\. Therefore,

‖f^bet−η‖1⟶‖g−η‖1=1−p4​palmost surely\.\\mathinner\{\\\!\\left\\\|\\hat\{f\}\_\{\\rm bet\}\-\\eta\\right\\\|\}\_\{1\}\\longrightarrow\\mathinner\{\\\!\\left\\\|g\-\\eta\\right\\\|\}\_\{1\}=\\frac\{1\-p\}\{4\}\\sqrt\{p\}\\qquad\\text\{almost surely\}\.
Finally,

‖g−η‖1ε1,∗=1−p4​p⟶∞as​p↓0\.\\frac\{\\mathinner\{\\\!\\left\\\|g\-\\eta\\right\\\|\}\_\{1\}\}\{\\varepsilon\_\{1,\*\}\}=\\frac\{1\-p\}\{4\\sqrt\{p\}\}\\longrightarrow\\infty\\qquad\\text\{as \}p\\downarrow 0\.Thus, for any fixedC<∞C<\\infty, one can choosep\>0p\>0sufficiently small such that‖g−η‖1\>C​ε1,∗\\mathinner\{\\\!\\left\\\|g\-\\eta\\right\\\|\}\_\{1\}\>C\\varepsilon\_\{1,\*\}\. Since every empirical minimizer converges almost surely togg, no universal bound of the form

‖f^bet−η‖1≤C​ε1,∗\+on​\(1\)\\mathinner\{\\\!\\left\\\|\\hat\{f\}\_\{\\rm bet\}\-\\eta\\right\\\|\}\_\{1\}\\leq C\\varepsilon\_\{1,\*\}\+o\_\{n\}\(1\)can hold over arbitrary misspecified finite classes\. ∎

##### The obstruction is already present at the population level\.

The proof establishes more than asymptotic failure of the empirical estimator\. For everynn, the population objective satisfies

Lbet,n​\(g\)<Lbet,n​\(f0\),L\_\{\\rm bet,n\}\(g\)<L\_\{\\rm bet,n\}\(f\_\{0\}\),with a gap bounded below by a positive multiple ofpp\. Thus the failure is not caused by sampling fluctuations or by insufficient concentration of the empirical betting loss\. Even with exact knowledge of the population objective, minimizing the original betting loss selects the predictor with the largerL1L\_\{1\}error on this instance\.

Proposition[17](https://arxiv.org/html/2610.20903#Thmtheorem17)shows that the original betting loss ERM can asymptotically select a predictor with errorΘ⁡\(ε1,∗\)\\Theta\(\\sqrt\{\\varepsilon\_\{1,\*\}\}\), and this deficiency is not merely an artifact of that proof strategy\. This motivates the use of a different objective in the misspecified setting\.

### Appendix GProof of the finite\-class variance\-adaptive guarantee

###### Proof of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)\.

LetM:=\|ℱ\|M\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\|\\mathcal\{F\}\|and, for brevity in the proof, write

α:=log⁡\(2​M2/δ\)n\.\\alpha\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{\\log\(2M^\{2\}/\\delta\)\}\{n\}\.Forf,h∈ℱf,h\\in\\mathcal\{F\}, define

Zf,h\(X,Y\):=\(Y−f\(X\)\)sign\(h\(X\)−f\(X\)\),Df,h\(X\):=\|f\(X\)−h\(X\)\|\.Z\_\{f,h\}\(X,Y\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(Y\-f\(X\)\)\\sign\(h\(X\)\-f\(X\)\),\\qquad D\_\{f,h\}\(X\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\|f\(X\)\-h\(X\)\|\.ThusT^​\(f,h\)=Pn​Zf,h\\widehat\{T\}\(f,h\)=P\_\{n\}Z\_\{f,h\}andTP​\(f,h\)=P​Zf,hT\_\{P\}\(f,h\)=PZ\_\{f,h\}\.

We first establish a uniform concentration event\. SinceY,f⁡\(X\)∈\[0,1\]Y,f\(X\)\\in\[0,1\], we have\|Zf,h\|≤1\|Z\_\{f,h\}\|\\leq 1, and hence\|Zf,h−P​Zf,h\|≤2\|Z\_\{f,h\}\-PZ\_\{f,h\}\|\\leq 2andVar⁡\(Zf,h\)≤PZf,h2\\Var\(Z\_\{f,h\}\)\\leq PZ\_\{f,h\}^\{2\}\. A one\-sided Bernstein inequality therefore gives, for each fixed pair\(f,h\)\(f,h\),

\(Pn−P\)​Zf,h≤2​P​Zf,h2​α\+43​α\(P\_\{n\}\-P\)Z\_\{f,h\}\\leq\\sqrt\{2PZ\_\{f,h\}^\{2\}\\,\\alpha\}\+\\frac\{4\}\{3\}\\alpha\(50\)except on an event of probability at moste−n​αe^\{\-n\\alpha\}\.

Similarly, since0≤Df,h≤10\\leq D\_\{f,h\}\\leq 1,

Var⁡\(Df,h\)≤PDf,h2≤PDf,h\.\\Var\(D\_\{f,h\}\)\\leq PD\_\{f,h\}^\{2\}\\leq PD\_\{f,h\}\.The lower\-tail Bernstein inequality gives

\(P−Pn\)​Df,h≤2​P​Df,h​α\+23​α\(P\-P\_\{n\}\)D\_\{f,h\}\\leq\\sqrt\{2PD\_\{f,h\}\\,\\alpha\}\+\\frac\{2\}\{3\}\\alpha\(51\)except on an event of probability at moste−n​αe^\{\-n\\alpha\}\.

There are at mostM2M^\{2\}ordered pairs in each of \([50](https://arxiv.org/html/2610.20903#A7.E50)\) and \([51](https://arxiv.org/html/2610.20903#A7.E51)\)\. Therefore, by a union bound, both inequalities hold simultaneously for everyf,h∈ℱf,h\\in\\mathcal\{F\}with probability at least

1−2​M2​e−n​α=1−δ\.1\-2M^\{2\}e^\{\-n\\alpha\}=1\-\\delta\.We work on this event throughout the remainder of the proof\.

Fix an arbitrary comparatorf0∈ℱf\_\{0\}\\in\\mathcal\{F\}\. By the definition ofBP​\(f0\)B\_\{P\}\(f\_\{0\}\), for everyh∈ℱh\\in\\mathcal\{F\},

P​Zf0,h=TP​\(f0,h\)≤BP​\(f0\)\.PZ\_\{f\_\{0\},h\}=T\_\{P\}\(f\_\{0\},h\)\\leq B\_\{P\}\(f\_\{0\}\)\.Moreover,

PZf0,h2=𝔼\[\(Y−f0\(X\)\)2𝟏\{h\(X\)≠f0\(X\)\}\]≤v\(f0\)\.PZ\_\{f\_\{0\},h\}^\{2\}=\\EE\\\!\\left\[\(Y\-f\_\{0\}\(X\)\)^\{2\}\\mathbf\{1\}\\\{h\(X\)\\neq f\_\{0\}\(X\)\\\}\\right\]\\leq v\(f\_\{0\}\)\.Hence \([50](https://arxiv.org/html/2610.20903#A7.E50)\) implies

suph∈ℱT^​\(f0,h\)≤BP​\(f0\)\+2​v​\(f0\)​α\+43​α\.\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f\_\{0\},h\)\\leq B\_\{P\}\(f\_\{0\}\)\+\\sqrt\{2v\(f\_\{0\}\)\\alpha\}\+\\frac\{4\}\{3\}\\alpha\.\(52\)
The key deterministic step is the sample analogue of Lemma[5](https://arxiv.org/html/2610.20903#Thmtheorem5)\. Pointwise,

Zf,h\+Zh,f=\|f−h\|,Z\_\{f,h\}\+Z\_\{h,f\}=\|f\-h\|,and therefore

Pn​\|f^−f0\|\\displaystyle P\_\{n\}\|\\hat\{f\}\-f\_\{0\}\|=T^​\(f^,f0\)\+T^​\(f0,f^\)\\displaystyle=\\widehat\{T\}\(\\hat\{f\},f\_\{0\}\)\+\\widehat\{T\}\(f\_\{0\},\\hat\{f\}\)≤suph∈ℱT^​\(f^,h\)\+suph∈ℱT^​\(f0,h\)\\displaystyle\\leq\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(\\hat\{f\},h\)\+\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f\_\{0\},h\)≤2​suph∈ℱT^​\(f0,h\),\\displaystyle\\leq 2\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f\_\{0\},h\),\(53\)where the last inequality follows from the empirical optimality off^\\hat\{f\}in \([10](https://arxiv.org/html/2610.20903#S4.E10)\)\. Combining \([52](https://arxiv.org/html/2610.20903#A7.E52)\) and \([53](https://arxiv.org/html/2610.20903#A7.E53)\) gives

Pn​\|f^−f0\|≤2​BP​\(f0\)\+2​2​v​\(f0\)​α\+83​α\.P\_\{n\}\|\\hat\{f\}\-f\_\{0\}\|\\leq 2B\_\{P\}\(f\_\{0\}\)\+2\\sqrt\{2v\(f\_\{0\}\)\\alpha\}\+\\frac\{8\}\{3\}\\alpha\.\(54\)
It remains to convert the empirical distance to its population counterpart\. Let

d:=‖f^−f0‖1=P​Df^,f0\.d\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathinner\{\\\!\\left\\\|\\hat\{f\}\-f\_\{0\}\\right\\\|\}\_\{1\}=PD\_\{\\hat\{f\},f\_\{0\}\}\.Because \([51](https://arxiv.org/html/2610.20903#A7.E51)\) holds simultaneously for all pairs inℱ\\mathcal\{F\}, it may be applied to the data\-dependent pair\(f^,f0\)\(\\hat\{f\},f\_\{0\}\)\. Thus

d≤Pn​\|f^−f0\|\+2​d​α\+23​α\.d\\leq P\_\{n\}\|\\hat\{f\}\-f\_\{0\}\|\+\\sqrt\{2d\\alpha\}\+\\frac\{2\}\{3\}\\alpha\.Using \([54](https://arxiv.org/html/2610.20903#A7.E54)\),

d≤2​BP​\(f0\)\+2​2​v​\(f0\)​α\+2​d​α\+103​α\.d\\leq 2B\_\{P\}\(f\_\{0\}\)\+2\\sqrt\{2v\(f\_\{0\}\)\\alpha\}\+\\sqrt\{2d\\alpha\}\+\\frac\{10\}\{3\}\\alpha\.\(55\)
Fix anyρ\>0\\rho\>0and set

θ:=ρ2\+ρ∈\(0,1\)\.\\theta\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{\\rho\}\{2\+\\rho\}\\in\(0,1\)\.Young’s inequality yields

2​d​α≤θ​d\+α2​θ\.\\sqrt\{2d\\alpha\}\\leq\\theta d\+\\frac\{\\alpha\}\{2\\theta\}\.Substituting this into \([55](https://arxiv.org/html/2610.20903#A7.E55)\) and using1/\(1−θ\)=\(2\+ρ\)/21/\(1\-\\theta\)=\(2\+\\rho\)/2, we obtain

d\\displaystyle d≤\(2\+ρ\)​\(BP​\(f0\)\+2​v​\(f0\)​α\)\\displaystyle\\leq\(2\+\\rho\)\\left\(B\_\{P\}\(f\_\{0\}\)\+\\sqrt\{2v\(f\_\{0\}\)\\alpha\}\\right\)\+\(133\+1ρ\+2312​ρ\)​α\.\\displaystyle\\quad\+\\left\(\\frac\{13\}\{3\}\+\\frac\{1\}\{\\rho\}\+\\frac\{23\}\{12\}\\rho\\right\)\\alpha\.\(56\)Finally, by the triangle inequality,

ε1​\(f^\)=‖f^−η‖1≤ε1​\(f0\)\+‖f^−f0‖1\.\\varepsilon\_\{1\}\(\\hat\{f\}\)=\\mathinner\{\\\!\\left\\\|\\hat\{f\}\-\\eta\\right\\\|\}\_\{1\}\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+\\mathinner\{\\\!\\left\\\|\\hat\{f\}\-f\_\{0\}\\right\\\|\}\_\{1\}\.Combining this with \([56](https://arxiv.org/html/2610.20903#A7.E56)\) gives, for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

ε1​\(f^\)≤\\displaystyle\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\{\}ε1​\(f0\)\+\(2\+ρ\)​BP​\(f0\)\+\(2\+ρ\)​2​v​\(f0\)​α\\displaystyle\\varepsilon\_\{1\}\(f\_\{0\}\)\+\(2\+\\rho\)B\_\{P\}\(f\_\{0\}\)\+\(2\+\\rho\)\\sqrt\{2v\(f\_\{0\}\)\\alpha\}\+\(133\+1ρ\+2312​ρ\)​α\.\\displaystyle\+\\left\(\\frac\{13\}\{3\}\+\\frac\{1\}\{\\rho\}\+\\frac\{23\}\{12\}\\rho\\right\)\\alpha\.
The concentration event above is uniform over all pairs inℱ\\mathcal\{F\}and does not depend on eitherf0f\_\{0\}orρ\\rho\. Hence the bound holds simultaneously for every comparatorf0∈ℱf\_\{0\}\\in\\mathcal\{F\}and everyρ\>0\\rho\>0\. Taking the infimum overf0∈ℱf\_\{0\}\\in\\mathcal\{F\}and substitutingα=log⁡\(2​\|ℱ\|2/δ\)/n\\alpha=\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)/ncompletes the proof\. ∎

### Appendix HThe finite\-sample guarantee recovers the population guarantee

###### Corollary 18\(Optimized finite\-sample bound\)\.

Under the assumptions of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), with probability at least1−δ1\-\\delta, simultaneously for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\},

ε1​\(f^\)≤ε1​\(f0\)\+2​BP​\(f0\)\+rn​\(f0,δ\),\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+2B\_\{P\}\(f\_\{0\}\)\+r\_\{n\}\(f\_\{0\},\\delta\),\(57\)where, writing

α:=log⁡\(2​\|ℱ\|2/δ\)n,s\(f0\):=2​v​\(f0\)​α,\\alpha\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\},\\qquad s\(f\_\{0\}\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sqrt\{2v\(f\_\{0\}\)\\alpha\},we may take

rn​\(f0,δ\):=2​s​\(f0\)\+133​α\+2​α⁡\(BP​\(f0\)\+s⁡\(f0\)\+2312​α\)\.r\_\{n\}\(f\_\{0\},\\delta\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=2s\(f\_\{0\}\)\+\\frac\{13\}\{3\}\\alpha\+2\\sqrt\{\\alpha\\left\(B\_\{P\}\(f\_\{0\}\)\+s\(f\_\{0\}\)\+\\frac\{23\}\{12\}\\alpha\\right\)\}\.\(58\)Consequently, for every fixedf0∈ℱf\_\{0\}\\in\\mathcal\{F\}andδ∈\(0,1\)\\delta\\in\(0,1\),rn​\(f0,δ\)→0r\_\{n\}\(f\_\{0\},\\delta\)\\to 0asn→∞n\\to\\infty\.

###### Proof\.

Fixf0∈ℱf\_\{0\}\\in\\mathcal\{F\}\. On the event of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8), for everyρ\>0\\rho\>0,

ε1​\(f^\)≤\\displaystyle\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\{\}ε1​\(f0\)\+\(2\+ρ\)​BP​\(f0\)\+\(2\+ρ\)​s​\(f0\)\\displaystyle\\varepsilon\_\{1\}\(f\_\{0\}\)\+\(2\+\\rho\)B\_\{P\}\(f\_\{0\}\)\+\(2\+\\rho\)s\(f\_\{0\}\)\+\(133\+1ρ\+2312​ρ\)​α\.\\displaystyle\+\\left\(\\frac\{13\}\{3\}\+\\frac\{1\}\{\\rho\}\+\\frac\{23\}\{12\}\\rho\\right\)\\alpha\.Rearranging theρ\\rho\-dependent terms gives

ε1​\(f^\)≤\\displaystyle\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\{\}ε1​\(f0\)\+2​BP​\(f0\)\+2​s​\(f0\)\+133​α\\displaystyle\\varepsilon\_\{1\}\(f\_\{0\}\)\+2B\_\{P\}\(f\_\{0\}\)\+2s\(f\_\{0\}\)\+\\frac\{13\}\{3\}\\alpha\+ρ⁡\(BP​\(f0\)\+s⁡\(f0\)\+2312​α\)\+αρ\.\\displaystyle\+\\rho\\left\(B\_\{P\}\(f\_\{0\}\)\+s\(f\_\{0\}\)\+\\frac\{23\}\{12\}\\alpha\\right\)\+\\frac\{\\alpha\}\{\\rho\}\.Since the theorem holds simultaneously for everyρ\>0\\rho\>0, we may choose

ρ=αBP​\(f0\)\+s⁡\(f0\)\+2312​α\.\\rho=\\sqrt\{\\frac\{\\alpha\}\{B\_\{P\}\(f\_\{0\}\)\+s\(f\_\{0\}\)\+\\frac\{23\}\{12\}\\alpha\}\}\.Using

infρ\>0\{ρ​A\+αρ\}=2​α​A,A\>0,\\inf\_\{\\rho\>0\}\\left\\\{\\rho A\+\\frac\{\\alpha\}\{\\rho\}\\right\\\}=2\\sqrt\{\\alpha A\},\\qquad A\>0,yields \([57](https://arxiv.org/html/2610.20903#A8.E57)\)–\([58](https://arxiv.org/html/2610.20903#A8.E58)\)\. The same high\-probability event is uniform overf0∈ℱf\_\{0\}\\in\\mathcal\{F\}, so the result holds simultaneously for all comparatorsf0f\_\{0\}\.

Finally, for fixedf0f\_\{0\},ℱ\\mathcal\{F\}, andδ\\delta, we haveα→0\\alpha\\to 0ands⁡\(f0\)=2​v​\(f0\)​α→0s\(f\_\{0\}\)=\\sqrt\{2v\(f\_\{0\}\)\\alpha\}\\to 0\. Hence every term in \([58](https://arxiv.org/html/2610.20903#A8.E58)\) converges to zero\. ∎

Corollary[18](https://arxiv.org/html/2610.20903#Thmtheorem18)makes precise the connection between the finite\-sample result and the population guarantee in Proposition[7](https://arxiv.org/html/2610.20903#Thmtheorem7)\. For every fixed comparatorf0f\_\{0\}, SBR satisfies the same leading population boundε1​\(f0\)\+2​BP​\(f0\)\\varepsilon\_\{1\}\(f\_\{0\}\)\+2B\_\{P\}\(f\_\{0\}\), up to an estimation term that vanishes with the sample size\. The optimizing value ofρ\\rhois used only in the analysis and does not need to be known or supplied to SBR\.

### Appendix IRelation to the fixed\-design guarantee of Baraud and Maillard

We clarify here the relation between our random\-design guarantees and the fixed\-design result of[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16)\. Their estimator coincides with SBR up to normalization when the optimization is exact, but their statistical setting and risk criterion are different\. In particular, they consider deterministic design points and control the expectedL1L\_\{1\}error evaluated on those design points\. Our results instead concern high\-probability population error under i\.i\.d\. random design\.

We record the consequence obtained by applying their theorem conditionally on the random covariates and then using a standard uniform\-convergence argument\. This provides a natural black\-box baseline against which to compare our direct random\-design analysis\.

##### An application of the fixed\-design result\.

Suppose thatℱ\\mathcal\{F\}is finite, withM:=\|ℱ\|M\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\|\\mathcal\{F\}\|, and define

R\(f\):=ε1\(f\)=𝔼\|f\(X\)−η\(X\)\|,Rn\(f\):=1n∑i=1n\|f\(Xi\)−η\(Xi\)\|\.R\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\varepsilon\_\{1\}\(f\)=\\EE\|f\(X\)\-\\eta\(X\)\|,\\qquad R\_\{n\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|f\(X\_\{i\}\)\-\\eta\(X\_\{i\}\)\|\.Let

σ¯n2:=1n∑i=1nσ2\(Xi\),σ¯2:=𝔼\[σ2\(X\)\]\.\\bar\{\\sigma\}\_\{n\}^\{2\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sigma^\{2\}\(X\_\{i\}\),\\qquad\\bar\{\\sigma\}^\{2\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\[\\sigma^\{2\}\(X\)\]\.
Condition onX1:n:=\(X1,…,Xn\)X\_\{1\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}n\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(X\_\{1\},\\ldots,X\_\{n\}\)\. The responses then form a fixed\-design regression problem,

Yi=η\(Xi\)\+ξi,𝔼\[ξi∣X1:n\]=0,𝔼\[ξi2∣X1:n\]=σ2\(Xi\)\.Y\_\{i\}=\\eta\(X\_\{i\}\)\+\\xi\_\{i\},\\qquad\\EE\[\\xi\_\{i\}\\mid X\_\{1\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}n\}\]=0,\\qquad\\EE\[\\xi\_\{i\}^\{2\}\\mid X\_\{1\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}n\}\]=\\sigma^\{2\}\(X\_\{i\}\)\.Hence thep=2p=2specialization of the oracle inequality of[Baraud and Maillard \(2025\)](https://arxiv.org/html/2610.20903#bib.bib16)applies conditionally onX1:nX\_\{1\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}n\}\.

For a finite class, the comparison\-set families appearing in their complexity measure contain at mostMMsets, so their relevant VC complexity is at most of orderlog⁡M\\log M\. Consequently, their theorem implies, for a universal constantκ\>0\\kappa\>0,

𝔼\[Rn\(f^\)\|X1:n\]≤3inff∈ℱRn\(f\)\+κσ¯n2​log⁡Mn\.\\EE\\\!\\left\[R\_\{n\}\(\\hat\{f\}\)\\,\\middle\|\\,X\_\{1\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}n\}\\right\]\\leq 3\\inf\_\{f\\in\\mathcal\{F\}\}R\_\{n\}\(f\)\+\\kappa\\sqrt\{\\frac\{\\bar\{\\sigma\}\_\{n\}^\{2\}\\log M\}\{n\}\}\.\(59\)We suppress inessential modifications of the logarithmic complexity term\.

Taking expectations overX1:nX\_\{1\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}n\}, using

𝔼⁡\[inff∈ℱRn​\(f\)\]≤inff∈ℱ𝔼⁡Rn​\(f\)=ε1,∗,\\EE\\\!\\left\[\\inf\_\{f\\in\\mathcal\{F\}\}R\_\{n\}\(f\)\\right\]\\leq\\inf\_\{f\\in\\mathcal\{F\}\}\\EE R\_\{n\}\(f\)=\\varepsilon\_\{1,\*\},and Jensen’s inequality gives

𝔼⁡Rn​\(f^\)≤3​ε1,∗\+κ​σ¯2​log⁡Mn\.\\EE R\_\{n\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+\\kappa\\sqrt\{\\frac\{\\bar\{\\sigma\}^\{2\}\\log M\}\{n\}\}\.\(60\)Thus the fixed\-design result already yields a variance\-sensitive bound for the expected error of SBR evaluated on the observed covariates\.

##### A direct conversion to population risk\.

To pass from the empirical\-design error to population error, define

Δn:=supf∈ℱ\|Rn​\(f\)−R⁡\(f\)\|\.\\Delta\_\{n\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{f\\in\\mathcal\{F\}\}\|R\_\{n\}\(f\)\-R\(f\)\|\.Sincef^∈ℱ\\hat\{f\}\\in\\mathcal\{F\},

R⁡\(f^\)≤Rn​\(f^\)\+Δn\.R\(\\hat\{f\}\)\\leq R\_\{n\}\(\\hat\{f\}\)\+\\Delta\_\{n\}\.Moreover, because\|f⁡\(X\)−η⁡\(X\)\|∈\[0,1\]\|f\(X\)\-\\eta\(X\)\|\\in\[0,1\], standard finite\-class uniform convergence gives

𝔼⁡\[Δn\]≲log⁡Mn\.\\EE\[\\Delta\_\{n\}\]\\lesssim\\sqrt\{\\frac\{\\log M\}\{n\}\}\.Combining this with \([60](https://arxiv.org/html/2610.20903#A9.E60)\) yields the population bound

𝔼⁡ε1​\(f^\)≤3​ε1,∗\+κ​σ¯2​log⁡Mn\+C​log⁡Mn\\EE\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+\\kappa\\sqrt\{\\frac\{\\bar\{\\sigma\}^\{2\}\\log M\}\{n\}\}\+C\\sqrt\{\\frac\{\\log M\}\{n\}\}\(61\)for a universal constantCC\.

The last term in \([61](https://arxiv.org/html/2610.20903#A9.E61)\) is independent of the conditional label variance\. In particular, even in the realizable noiseless caseε1,∗=σ¯2=0\\varepsilon\_\{1,\*\}=\\bar\{\\sigma\}^\{2\}=0, this direct black\-box argument leaves anO⁡\(log⁡\(M\)/n\)O\(\\sqrt\{\\log\(M\)/n\}\)population\-generalization term\.

This calculation highlights the additional issue created by random design: a variance\-sensitive fixed\-design bound does not by itself provide a variance\-sensitive population guarantee\. Our analysis does not treat the fixed\-design theorem as a black box followed by global uniform convergence\. Instead, it analyzes the signed\-residual process and the empirical\-to\-population transition jointly, leading to the localized, high\-probability guarantees of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)\. In particular, our bound retains comparator\-dependent approximation and second\-order quantities and can achieve substantially smaller statistical terms on favorable instances\.

### Appendix JA sufficient condition for favorable misspecification

###### Proposition 19\(A sufficient condition for favorable misspecification\)\.

Let

Dℱ:=\{x∈𝒳:∃f,h∈ℱ,f\(x\)≠h\(x\)\}D\_\{\\mathcal\{F\}\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{x\\in\\mathcal\{X\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\exists f,h\\in\\mathcal\{F\},\\ f\(x\)\\neq h\(x\)\\\}be the disagreement region ofℱ\\mathcal\{F\}\. Suppose there existsf⋆∈ℱf\_\{\\star\}\\in\\mathcal\{F\}such that

f⋆​\(x\)=η⁡\(x\)for every​x∈Dℱ\.f\_\{\\star\}\(x\)=\\eta\(x\)\\qquad\\text\{for every \}x\\in D\_\{\\mathcal\{F\}\}\.Thenf⋆f\_\{\\star\}is anL1L\_\{1\}\-optimal predictor andBP​\(f⋆\)=0B\_\{P\}\(f\_\{\\star\}\)=0\.

###### Proof\.

OutsideDℱD\_\{\\mathcal\{F\}\}, all predictors inℱ\\mathcal\{F\}agree, and hence everyf∈ℱf\\in\\mathcal\{F\}satisfiesf=f⋆f=f\_\{\\star\}onDℱcD\_\{\\mathcal\{F\}\}^\{c\}\. OnDℱD\_\{\\mathcal\{F\}\}, we havef⋆=ηf\_\{\\star\}=\\eta\. Therefore, for everyf∈ℱf\\in\\mathcal\{F\},

ε1​\(f\)=ε1​\(f⋆\)\+‖f−f⋆‖1≥ε1​\(f⋆\),\\varepsilon\_\{1\}\(f\)=\\varepsilon\_\{1\}\(f\_\{\\star\}\)\+\\mathinner\{\\\!\\left\\\|f\-f\_\{\\star\}\\right\\\|\}\_\{1\}\\geq\\varepsilon\_\{1\}\(f\_\{\\star\}\),sof⋆f\_\{\\star\}isL1L\_\{1\}\-optimal\.

Moreover, for everyh∈ℱh\\in\\mathcal\{F\},sign⁡\(h−f⋆\)=0\\sign\(h\-f\_\{\\star\}\)=0outsideDℱD\_\{\\mathcal\{F\}\}, whereasη−f⋆=0\\eta\-f\_\{\\star\}=0onDℱD\_\{\\mathcal\{F\}\}\. Hence

TP​\(f⋆,h\)=𝔼⁡\[\(η−f⋆\)​sign⁡\(h−f⋆\)\]=0\.T\_\{P\}\(f\_\{\\star\},h\)=\\EE\[\(\\eta\-f\_\{\\star\}\)\\sign\(h\-f\_\{\\star\}\)\]=0\.Taking the supremum overh∈ℱh\\in\\mathcal\{F\}givesBP​\(f⋆\)=0B\_\{P\}\(f\_\{\\star\}\)=0\. ∎

### Appendix KProofs for the sharp factor\-three characterization

#### K\.1Proof of the uniform factor\-three guarantee

###### Proof of Corollary[12](https://arxiv.org/html/2610.20903#Thmtheorem12)\.

Let

αδ:=log⁡\(2​\|ℱ\|2/δ\)n,\\alpha\_\{\\delta\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\},and fix

f⋆∈arg~minf∈ℱε1​\(f\)\.f\_\{\\star\}\\in\\argmin\_\{f\\in\\mathcal\{F\}\}\\varepsilon\_\{1\}\(f\)\.SinceY∈\[0,1\]Y\\in\[0,1\]and everyf∈ℱf\\in\\mathcal\{F\}is\[0,1\]\[0,1\]\-valued,

ε1,∗≤1,BP​\(f⋆\)≤ε1,∗≤1,v⁡\(f⋆\)≤1\.\\varepsilon\_\{1,\*\}\\leq 1,\\qquad B\_\{P\}\(f\_\{\\star\}\)\\leq\\varepsilon\_\{1,\*\}\\leq 1,\\qquad v\(f\_\{\\star\}\)\\leq 1\.\(62\)
Suppose first thatαδ≤1\\alpha\_\{\\delta\}\\leq 1\. Since the event in Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)holds simultaneously for everyρ\>0\\rho\>0, we may choose

ρ=αδ\.\\rho=\\sqrt\{\\alpha\_\{\\delta\}\}\.Applying Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)withf0=f⋆f\_\{0\}=f\_\{\\star\}and using \([62](https://arxiv.org/html/2610.20903#A11.E62)\) gives

ε1​\(f^\)\\displaystyle\\varepsilon\_\{1\}\(\\hat\{f\}\)≤ε1,∗\+\(2\+ρ\)​BP​\(f⋆\)\+\(2\+ρ\)​2​v​\(f⋆\)​αδ\+Cρ​αδ\\displaystyle\\leq\\varepsilon\_\{1,\*\}\+\(2\+\\rho\)B\_\{P\}\(f\_\{\\star\}\)\+\(2\+\\rho\)\\sqrt\{2v\(f\_\{\\star\}\)\\alpha\_\{\\delta\}\}\+C\_\{\\rho\}\\alpha\_\{\\delta\}≤3​ε1,∗\+ρ​ε1,∗\+\(2\+ρ\)​2​αδ\+\(133\+1ρ\+2312​ρ\)​αδ\\displaystyle\\leq 3\\varepsilon\_\{1,\*\}\+\\rho\\varepsilon\_\{1,\*\}\+\(2\+\\rho\)\\sqrt\{2\\alpha\_\{\\delta\}\}\+\\left\(\\frac\{13\}\{3\}\+\\frac\{1\}\{\\rho\}\+\\frac\{23\}\{12\}\\rho\\right\)\\alpha\_\{\\delta\}≤3​ε1,∗\+αδ\+3​2​αδ\+133​αδ\+αδ\+2312​αδ3/2\.\\displaystyle\\leq 3\\varepsilon\_\{1,\*\}\+\\sqrt\{\\alpha\_\{\\delta\}\}\+3\\sqrt\{2\\alpha\_\{\\delta\}\}\+\\frac\{13\}\{3\}\\alpha\_\{\\delta\}\+\\sqrt\{\\alpha\_\{\\delta\}\}\+\\frac\{23\}\{12\}\\alpha\_\{\\delta\}^\{3/2\}\.Becauseαδ≤1\\alpha\_\{\\delta\}\\leq 1,

αδ≤αδ,αδ3/2≤αδ\.\\alpha\_\{\\delta\}\\leq\\sqrt\{\\alpha\_\{\\delta\}\},\\qquad\\alpha\_\{\\delta\}^\{3/2\}\\leq\\sqrt\{\\alpha\_\{\\delta\}\}\.Hence

ε1​\(f^\)≤3​ε1,∗\+\(334\+3​2\)​αδ\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+\\left\(\\frac\{33\}\{4\}\+3\\sqrt\{2\}\\right\)\\sqrt\{\\alpha\_\{\\delta\}\}\.\(63\)
If insteadαδ\>1\\alpha\_\{\\delta\}\>1, boundedness gives

ε1​\(f^\)≤1\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 1\.Since

334\+3​2\>1andαδ\>1,\\frac\{33\}\{4\}\+3\\sqrt\{2\}\>1\\qquad\\text\{and\}\\qquad\\sqrt\{\\alpha\_\{\\delta\}\}\>1,the bound in \([63](https://arxiv.org/html/2610.20903#A11.E63)\) also holds trivially in this case\.

Therefore, for alln≥1n\\geq 1andδ∈\(0,1\)\\delta\\in\(0,1\),

ε1​\(f^\)≤3​ε1,∗\+\(334\+3​2\)​log⁡\(2​\|ℱ\|2/δ\)n\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq 3\\varepsilon\_\{1,\*\}\+\\left\(\\frac\{33\}\{4\}\+3\\sqrt\{2\}\\right\)\\sqrt\{\\frac\{\\log\(2\|\\mathcal\{F\}\|^\{2\}/\\delta\)\}\{n\}\}\.\(64\)The estimation term in \([64](https://arxiv.org/html/2610.20903#A11.E64)\) depends only onnn,\|ℱ\|\|\\mathcal\{F\}\|, andδ\\delta, and hence is uniform over the underlying distributionPP\. ∎

#### K\.2Proof of Theorem[13](https://arxiv.org/html/2610.20903#Thmtheorem13)

###### Proof\.

FixC∈\[1,3\)C\\in\[1,3\)\. Take𝒳=ℕ\\mathcal\{X\}=\\mathbb\{N\}, seta:=\(C\+1\)/\(C\+5\)a\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(C\+1\)/\(C\+5\), and define

f−​\(x\)≡a,f\+​\(x\)≡1−a,ℱC:=\{f−,f\+\}\.f\_\{\-\}\(x\)\\equiv a,\\qquad f\_\{\+\}\(x\)\\equiv 1\-a,\\qquad\\mathcal\{F\}\_\{C\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{f\_\{\-\},f\_\{\+\}\\\}\.Also set

q:=12​\(1−a\)=C\+58,e:=12−a,r:=1\+a1−a=C\+32\.q\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{2\(1\-a\)\}=\\frac\{C\+5\}\{8\},\\qquad e\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{2\}\-a,\\qquad r\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\+a\}\{1\-a\}=\\frac\{C\+3\}\{2\}\.Thena∈\[1/3,1/2\)a\\in\[1/3,1/2\),q∈\(0,1\)q\\in\(0,1\),e\>0e\>0, andr\>Cr\>C\. Defineg:=\(r−C\)​e\>0g\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(r\-C\)e\>0and

εC:=g2=\(3−C\)28​\(C\+5\)\.\\varepsilon\_\{C\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{g\}\{2\}=\\frac\{\(3\-C\)^\{2\}\}\{8\(C\+5\)\}\.BothℱC\\mathcal\{F\}\_\{C\}andεC\\varepsilon\_\{C\}depend only onCC\.

Now fixn≥1n\\geq 1and a possibly randomized proper learner with outputf^n∈ℱC\\widehat\{f\}\_\{n\}\\in\\mathcal\{F\}\_\{C\}\. All probabilities involving its output include the learner’s internal randomization\.

For an integerNNto be chosen below, construct two priorsΠ−\\Pi\_\{\-\}andΠ\+\\Pi\_\{\+\}over vectorsθ=\(θ1,…,θN\)\\theta=\(\\theta\_\{1\},\\ldots,\\theta\_\{N\}\)of conditional means\. Under each prior, the coordinates are independent, with

Π−:θj\\displaystyle\\Pi\_\{\-\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\quad\\theta\_\{j\}=\{a,with probability​q,1,with probability​1−q,\\displaystyle=\\begin\{cases\}a,&\\text\{with probability \}q,\\\\ 1,&\\text\{with probability \}1\-q,\\end\{cases\}Π\+:θj\\displaystyle\\Pi\_\{\+\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\quad\\theta\_\{j\}=\{1−a,with probability​q,0,with probability​1−q\.\\displaystyle=\\begin\{cases\}1\-a,&\\text\{with probability \}q,\\\\ 0,&\\text\{with probability \}1\-q\.\\end\{cases\}For each fixedθ\\theta, letPθP\_\{\\theta\}be the distribution defined by

X∼Unif\(\[N\]\),Y∣X=j∼Bernoulli\(θj\)\.X\\sim\\operatorname\{Unif\}\(\[N\]\),\\qquad Y\\mid X=j\\sim\\operatorname\{Bernoulli\}\(\\theta\_\{j\}\)\.Its regression function satisfiesηθ​\(j\)=θj\\eta\_\{\\theta\}\(j\)=\\theta\_\{j\}forj∈\[N\]j\\in\[N\]\. Forf∈ℱCf\\in\\mathcal\{F\}\_\{C\}, write

dθ​\(f\):=𝔼Pθ\|f⁡\(X\)−ηθ​\(X\)\|=1N​∑j=1N\|f⁡\(j\)−θj\|\.d\_\{\\theta\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\EE\_\{P\_\{\\theta\}\}\|f\(X\)\-\\eta\_\{\\theta\}\(X\)\|=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\|f\(j\)\-\\theta\_\{j\}\|\.UnderΠ−\\Pi\_\{\-\}, direct calculation gives

𝔼Π−⁡dθ​\(f−\)\\displaystyle\\EE\_\{\\Pi\_\{\-\}\}d\_\{\\theta\}\(f\_\{\-\}\)=\(1−q\)​\(1−a\)=e,\\displaystyle=\(1\-q\)\(1\-a\)=e,𝔼Π−⁡dθ​\(f\+\)\\displaystyle\\EE\_\{\\Pi\_\{\-\}\}d\_\{\\theta\}\(f\_\{\+\}\)=q⁡\(1−2​a\)\+\(1−q\)​a=r​e\.\\displaystyle=q\(1\-2a\)\+\(1\-q\)a=re\.By symmetry,𝔼Π\+⁡dθ​\(f\+\)=e\\EE\_\{\\Pi\_\{\+\}\}d\_\{\\theta\}\(f\_\{\+\}\)=eand𝔼Π\+⁡dθ​\(f−\)=re\\EE\_\{\\Pi\_\{\+\}\}d\_\{\\theta\}\(f\_\{\-\}\)=re\.

Lett:=g/\[4​\(1\+C\)\]t\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=g/\[4\(1\+C\)\]\. Fors∈\{−,\+\}s\\in\\\{\-,\+\\\}, lets¯\\bar\{s\}denote the opposite sign and define

Gs:=\{\|dθ\(fs\)−e\|≤t,\|dθ\(fs¯\)−re\|≤t\}\.G\_\{s\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\left\\\{\|d\_\{\\theta\}\(f\_\{s\}\)\-e\|\\leq t,\\quad\|d\_\{\\theta\}\(f\_\{\\bar\{s\}\}\)\-re\|\\leq t\\right\\\}\.Each risk is an average ofNNindependent\[0,1\]\[0,1\]\-valued random variables under the corresponding prior\. Hoeffding’s inequality and a union bound therefore yield

Πs​\(Gsc\)≤4​exp⁡\(−2​N​t2\)\.\\Pi\_\{s\}\(G\_\{s\}^\{c\}\)\\leq 4\\exp\(\-2Nt^\{2\}\)\.OnGsG\_\{s\},

dθ​\(fs¯\)−C​dθ​\(fs\)\\displaystyle d\_\{\\theta\}\(f\_\{\\bar\{s\}\}\)\-Cd\_\{\\theta\}\(f\_\{s\}\)≥r​e−t−C⁡\(e\+t\)\\displaystyle\\geq re\-t\-C\(e\+t\)=g−\(1\+C\)​t=3​g4\>εC\.\\displaystyle=g\-\(1\+C\)t=\\frac\{3g\}\{4\}\>\\varepsilon\_\{C\}\.SinceC≥1C\\geq 1, this also impliesdθ​\(fs¯\)\>dθ​\(fs\)d\_\{\\theta\}\(f\_\{\\bar\{s\}\}\)\>d\_\{\\theta\}\(f\_\{s\}\)\. Thusfsf\_\{s\}is the uniqueL1L\_\{1\}\-optimal member ofℱC\\mathcal\{F\}\_\{C\}onGsG\_\{s\}, and

dθ​\(fs¯\)\>C​minf∈ℱC​dθ​\(f\)\+εC\.d\_\{\\theta\}\(f\_\{\\bar\{s\}\}\)\>C\\min\_\{f\\in\\mathcal\{F\}\_\{C\}\}d\_\{\\theta\}\(f\)\+\\varepsilon\_\{C\}\.\(65\)
We next show that the two priors induce nearly indistinguishable sample distributions\. Let

Qs:=∫Pθ⊗n​Πs​\(𝑑θ\),s∈\{−,\+\},Q\_\{s\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\int P\_\{\\theta\}^\{\\otimes n\}\\,\\Pi\_\{s\}\(d\\theta\),\\qquad s\\in\\\{\-,\+\\\},be the mixture law of the entire training sample\. In particular,QsQ\_\{s\}is a mixture of product distributions, not a product of single\-observation mixtures\. Under both priors,

𝔼Π−⁡θj=qa\+\(1−q\)=12,𝔼Π\+⁡θj=q⁡\(1−a\)=12\.\\EE\_\{\\Pi\_\{\-\}\}\\theta\_\{j\}=qa\+\(1\-q\)=\\frac\{1\}\{2\},\\qquad\\EE\_\{\\Pi\_\{\+\}\}\\theta\_\{j\}=q\(1\-a\)=\\frac\{1\}\{2\}\.LetD:=\{X1,…,Xn​are all distinct\}D\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{X\_\{1\},\\ldots,X\_\{n\}\\text\{ are all distinct\}\\\}\. Conditional on any distinct input sequence, the sampled coordinatesθX1,…,θXn\\theta\_\{X\_\{1\}\},\\ldots,\\theta\_\{X\_\{n\}\}are independent under either prior\. After integrating out these coordinates, the conditional label law isBernoulli⁡\(1/2\)⊗n\\operatorname\{Bernoulli\}\(1/2\)^\{\\otimes n\}under bothQ−Q\_\{\-\}andQ\+Q\_\{\+\}\. The input law is also identical under the two mixtures\. Hence their restrictions toDDcoincide\.

WritingTV⁡\(Q−,Q\+\):=supA\|Q−​\(A\)−Q\+​\(A\)\|\\operatorname\{TV\}\(Q\_\{\-\},Q\_\{\+\}\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{A\}\|Q\_\{\-\}\(A\)\-Q\_\{\+\}\(A\)\|, we obtain

TV⁡\(Q−,Q\+\)≤Q−​\(Dc\)≤∑1≤i<j≤nℙ⁡\(Xi=Xj\)=n⁡\(n−1\)2​N,\\operatorname\{TV\}\(Q\_\{\-\},Q\_\{\+\}\)\\leq Q\_\{\-\}\(D^\{c\}\)\\leq\\sum\_\{1\\leq i<j\\leq n\}\\PP\(X\_\{i\}=X\_\{j\}\)=\\frac\{n\(n\-1\)\}\{2N\},where the collision probabilities are computed under the common input law\. Choose an integer

N≥max⁡\{1,6​n​\(n−1\),log⁡482​t2\}\.N\\geq\\max\\left\\\{1,\\,6n\(n\-1\),\\,\\frac\{\\log 48\}\{2t^\{2\}\}\\right\\\}\.The preceding bounds then give

TV\(Q−,Q\+\)≤112,Πs\(Gsc\)≤112\(s∈\{−,\+\}\)\.\\operatorname\{TV\}\(Q\_\{\-\},Q\_\{\+\}\)\\leq\\frac\{1\}\{12\},\\qquad\\Pi\_\{s\}\(G\_\{s\}^\{c\}\)\\leq\\frac\{1\}\{12\}\\quad\(s\\in\\\{\-,\+\\\}\)\.\(66\)
Letℙs\\mathbb\{P\}\_\{s\}denote the joint experiment that first drawsθ∼Πs\\theta\\sim\\Pi\_\{s\}, then draws the sampleS∼Pθ⊗nS\\sim P\_\{\\theta\}^\{\\otimes n\}, and includes the learner’s internal randomization\. For each sampleSS, letϕ⁡\(S\)∈\[0,1\]\\phi\(S\)\\in\[0,1\]be the probability, over this internal randomization, that the learner returnsf\+f\_\{\+\}\. Properness implies that it returnsf−f\_\{\-\}with probability1−ϕ⁡\(S\)1\-\\phi\(S\)\. Consequently,

ℙ−​\(f^n=f\+\)\+ℙ\+​\(f^n=f−\)\\displaystyle\\mathbb\{P\}\_\{\-\}\(\\widehat\{f\}\_\{n\}=f\_\{\+\}\)\+\\mathbb\{P\}\_\{\+\}\(\\widehat\{f\}\_\{n\}=f\_\{\-\}\)=∫ϕ​d​Q−\+∫\(1−ϕ\)​d​Q\+\\displaystyle=\\int\\phi\\,dQ\_\{\-\}\+\\int\(1\-\\phi\)\\,dQ\_\{\+\}=1\+∫ϕ​d​\(Q−−Q\+\)\\displaystyle=1\+\\int\\phi\\,d\(Q\_\{\-\}\-Q\_\{\+\}\)≥1−TV⁡\(Q−,Q\+\)\\displaystyle\\geq 1\-\\operatorname\{TV\}\(Q\_\{\-\},Q\_\{\+\}\)≥1112\.\\displaystyle\\geq\\frac\{11\}\{12\}\.The first inequality follows from0≤ϕ≤10\\leq\\phi\\leq 1and the definition of total variation; for example, writeϕ\(S\)=∫01𝟏\{ϕ\(S\)\>u\}du\\phi\(S\)=\\int\_\{0\}^\{1\}\\mathbf\{1\}\\\{\\phi\(S\)\>u\\\}\\,du\. Removing the bad environments and using \([66](https://arxiv.org/html/2610.20903#A11.E66)\) yields

12​∑s∈\{−,\+\}ℙs​\(f^n=fs¯,Gs\)\\displaystyle\\frac\{1\}\{2\}\\sum\_\{s\\in\\\{\-,\+\\\}\}\\mathbb\{P\}\_\{s\}\\\!\\left\(\\widehat\{f\}\_\{n\}=f\_\{\\bar\{s\}\},\\,G\_\{s\}\\right\)≥1124−12​∑s∈\{−,\+\}Πs​\(Gsc\)\\displaystyle\\geq\\frac\{11\}\{24\}\-\\frac\{1\}\{2\}\\sum\_\{s\\in\\\{\-,\+\\\}\}\\Pi\_\{s\}\(G\_\{s\}^\{c\}\)\(67\)≥1124−112=38\.\\displaystyle\\geq\\frac\{11\}\{24\}\-\\frac\{1\}\{12\}=\\frac\{3\}\{8\}\.
The left\-hand side of \([67](https://arxiv.org/html/2610.20903#A11.E67)\) equals

12​∑s∈\{−,\+\}∫𝟏Gs​\(θ\)​ℙS∼Pθ⊗n⁡\(f^n=fs¯\)​Πs​\(𝑑θ\)\.\\frac\{1\}\{2\}\\sum\_\{s\\in\\\{\-,\+\\\}\}\\int\\mathbf\{1\}\_\{G\_\{s\}\}\(\\theta\)\\,\\PP\_\{S\\sim P\_\{\\theta\}^\{\\otimes n\}\}\\\!\\left\(\\widehat\{f\}\_\{n\}=f\_\{\\bar\{s\}\}\\right\)\\,\\Pi\_\{s\}\(d\\theta\)\.Since both priors have finite support, this is a finite weighted average\. It follows that there exists∈\{−,\+\}s\\in\\\{\-,\+\\\}and a fixedθ∈Gs\\theta\\in G\_\{s\}such that

ℙS∼Pθ⊗n⁡\(f^n=fs¯\)≥38\.\\PP\_\{S\\sim P\_\{\\theta\}^\{\\otimes n\}\}\\\!\\left\(\\widehat\{f\}\_\{n\}=f\_\{\\bar\{s\}\}\\right\)\\geq\\frac\{3\}\{8\}\.For thisθ\\theta, \([65](https://arxiv.org/html/2610.20903#A11.E65)\) implies

ℙS∼Pθ⊗n⁡\(dθ​\(f^n\)\>C​minf∈ℱC​dθ​\(f\)\+εC\)≥38\.\\PP\_\{S\\sim P\_\{\\theta\}^\{\\otimes n\}\}\\left\(d\_\{\\theta\}\(\\widehat\{f\}\_\{n\}\)\>C\\min\_\{f\\in\\mathcal\{F\}\_\{C\}\}d\_\{\\theta\}\(f\)\+\\varepsilon\_\{C\}\\right\)\\geq\\frac\{3\}\{8\}\.TakingP=PθP=P\_\{\\theta\}proves \([24](https://arxiv.org/html/2610.20903#S5.E24)\)\. The final counterexample is the fixed distributionPθP\_\{\\theta\}, not either sample mixtureQsQ\_\{s\}; its observations are i\.i\.d\. and its input marginal has finite support\. ∎

The construction above uses the same input marginalPX=Unif​\(\[N\]\)P\_\{X\}=\\text\{Unif\}\(\[N\]\)in the two hard experiments\. Hence the proof in fact continues to hold even if the learner is givenPXP\_\{X\}\.

#### K\.3Impossibility of sub\-three uniform guarantees

We now derive the impossibility of \([25](https://arxiv.org/html/2610.20903#S5.E25)\)\. FixC∈\[1,3\)C\\in\[1,3\), and letℱC\\mathcal\{F\}\_\{C\}andεC\>0\\varepsilon\_\{C\}\>0be supplied by Theorem[13](https://arxiv.org/html/2610.20903#Thmtheorem13)\. Suppose, toward a contradiction, that a possibly randomized proper learning rule satisfies

ℙS∼P⊗n⁡\(ε1​\(f^n\)≤C​ε1,∗\+rn​\(ℱC,δ\)\)≥1−δ\\PP\_\{S\\sim P^\{\\otimes n\}\}\\left\(\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)\\leq C\\varepsilon\_\{1,\*\}\+r\_\{n\}\(\\mathcal\{F\}\_\{C\},\\delta\)\\right\)\\geq 1\-\\delta\(68\)for everyn≥1n\\geq 1, everyδ∈\(0,1\)\\delta\\in\(0,1\), and every distributionPPon𝒳×\[0,1\]\\mathcal\{X\}\\times\[0,1\]\. Here the errors are evaluated underPPrelative toℱC\\mathcal\{F\}\_\{C\}, and the learner may depend onδ\\delta\. Assume that the estimation term is deterministic, independent ofPP, and satisfiesrn​\(ℱC,δ\)→0r\_\{n\}\(\\mathcal\{F\}\_\{C\},\\delta\)\\to 0for each fixedδ\\delta\.

Setδ=1/4\\delta=1/4\. SinceℱC\\mathcal\{F\}\_\{C\}andεC\\varepsilon\_\{C\}are fixed independently ofnn, there existsnnsufficiently large thatrn​\(ℱC,1/4\)≤εCr\_\{n\}\(\\mathcal\{F\}\_\{C\},1/4\)\\leq\\varepsilon\_\{C\}\. For thisnn, \([68](https://arxiv.org/html/2610.20903#A11.E68)\) implies, for everyPP,

ℙS∼P⊗n⁡\(ε1​\(f^n\)\>C​ε1,∗\+εC\)\\displaystyle\\PP\_\{S\\sim P^\{\\otimes n\}\}\\left\(\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)\>C\\varepsilon\_\{1,\*\}\+\\varepsilon\_\{C\}\\right\)≤ℙS∼P⊗n⁡\(ε1​\(f^n\)\>C​ε1,∗\+rn​\(ℱC,1/4\)\)≤14\.\\displaystyle\\qquad\\leq\\PP\_\{S\\sim P^\{\\otimes n\}\}\\left\(\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)\>C\\varepsilon\_\{1,\*\}\+r\_\{n\}\(\\mathcal\{F\}\_\{C\},1/4\)\\right\)\\leq\\frac\{1\}\{4\}\.This contradicts Theorem[13](https://arxiv.org/html/2610.20903#Thmtheorem13), which supplies a distributionPPfor which the first probability is at least3/83/8\. The same contradiction applies if the proposed guarantee is required only for all sufficiently largenn, provided the sample\-size threshold is independent ofPP\.

ForC<1C<1, the argument is trivial\. Take the singleton classℱ=\{f≡0\}\\mathcal\{F\}=\\\{f\\equiv 0\\\}and a distribution withX≡1X\\equiv 1andY≡1Y\\equiv 1\. Every proper learner then satisfiesε1​\(f^n\)=ε1,∗=1\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)=\\varepsilon\_\{1,\*\}=1deterministically\. Thus \([25](https://arxiv.org/html/2610.20903#S5.E25)\) fails with probability one wheneverrn​\(ℱ,1/4\)<1−Cr\_\{n\}\(\\mathcal\{F\},1/4\)<1\-C, which holds for all sufficiently largennifrn​\(ℱ,1/4\)→0r\_\{n\}\(\\mathcal\{F\},1/4\)\\to 0\.

Therefore, for everyC∈\[0,3\)C\\in\[0,3\), no proper learner can satisfy a distribution\-free high\-probability oracle inequality of the form

ε1​\(f^n\)≤C​ε1,∗\+rn​\(ℱ,δ\)\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{n\}\)\\leq C\\varepsilon\_\{1,\*\}\+r\_\{n\}\(\\mathcal\{F\},\\delta\)with a distribution\-independent estimation term satisfyingrn​\(ℱ,δ\)→0r\_\{n\}\(\\mathcal\{F\},\\delta\)\\to 0for each fixedℱ\\mathcal\{F\}andδ\\delta\. In particular, the conclusion is unaffected by how slowly the estimation term vanishes withnn\.

Together with Corollary[12](https://arxiv.org/html/2610.20903#Thmtheorem12), which establishes such a uniform guarantee with approximation factor33, this proves that33is the smallest universal approximation factor achievable by proper learners under the distribution\-free, high\-probability criterion considered in Section[5](https://arxiv.org/html/2610.20903#S5)\.

### Appendix LBeyond finite classes: proof and examples

This appendix proves Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)and verifies its complexity conditions for several standard infinite hypothesis classes\. The proof has two ingredients\. First, prediction and comparison complexity control the metric entropy of the empirical processes induced by SBR\. Second, a uniform Bernstein inequality converts these entropy bounds into variance\-sensitive concentration\.

Throughout this appendix, covering numbers are taken with centers in the class\. We suppress standard measurability issues throughout\.

#### L\.1Prediction and comparison complexity

Recall the pairwise comparison class

𝒞ℱ:=\{x↦𝟙\{h\(x\)\>f\(x\)\}:f,h∈ℱ\},\\mathcal\{C\}\_\{\\mathcal\{F\}\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\left\\\{x\\mapsto\\one\\\{h\(x\)\>f\(x\)\\\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}f,h\\in\\mathcal\{F\}\\right\\\},and letdpred:=Pdim⁡\(ℱ\)d\_\{\\rm pred\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\operatorname\{Pdim\}\(\\mathcal\{F\}\),dcmp:=VCdim⁡\(𝒞ℱ\)d\_\{\\rm cmp\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\operatorname\{VCdim\}\(\\mathcal\{C\}\_\{\\mathcal\{F\}\}\), andd:=1\+dpred\+dcmpd\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=1\+d\_\{\\rm pred\}\+d\_\{\\rm cmp\}\. Forf,h∈ℱf,h\\in\\mathcal\{F\}, define

Zf,h\(x,y\):=\(y−f\(x\)\)sign\(h\(x\)−f\(x\)\),Df,h\(x\):=\|f\(x\)−h\(x\)\|,Z\_\{f,h\}\(x,y\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\(y\-f\(x\)\)\\sign\(h\(x\)\-f\(x\)\),\\qquad D\_\{f,h\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\|f\(x\)\-h\(x\)\|,and introduce the corresponding function classes

𝒵:=\{Zf,h:f,h∈ℱ\},𝒟:=\{Df,h:f,h∈ℱ\}\.\\mathcal\{Z\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{Z\_\{f,h\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}f,h\\in\\mathcal\{F\}\\\},\\qquad\\mathcal\{D\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\\{D\_\{f,h\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}f,h\\in\\mathcal\{F\}\\\}\.
The next proposition shows that the two complexity measures in Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)imply uniform polynomial covering bounds for the function classes needed in the proof\.

###### Proposition 20\(Prediction and comparison complexity imply uniform entropy\)\.

Supposeℱ⊆\{f:𝒳→\[0,1\]\}\\mathcal\{F\}\\subseteq\\\{f\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\[0,1\]\\\}anddpred,dcmp<∞d\_\{\\rm pred\},d\_\{\\rm cmp\}<\\infty\. There exist universal constantsA0≥eA\_\{0\}\\geq eandc0\>0c\_\{0\}\>0such that, for every probability measureQQon𝒳×\[0,1\]\\mathcal\{X\}\\times\[0,1\]and everyu∈\(0,1\]u\\in\(0,1\],

log⁡N⁡\(u,𝒵,L2​\(Q\)\)≤c0​d​log⁡\(A0/u\),\\log N\\bigl\(u,\\mathcal\{Z\},L\_\{2\}\(Q\)\\bigr\)\\leq c\_\{0\}d\\log\(A\_\{0\}/u\),and

log⁡N⁡\(u,𝒟,L2​\(QX\)\)≤c0​d​log⁡\(A0/u\),\\log N\\bigl\(u,\\mathcal\{D\},L\_\{2\}\(Q\_\{X\}\)\\bigr\)\\leq c\_\{0\}d\\log\(A\_\{0\}/u\),whereQXQ\_\{X\}denotes the𝒳\\mathcal\{X\}\-marginal ofQQ\.

###### Proof\.

We first record the standard entropy bounds that will be used throughout\. SincePdim⁡\(ℱ\)=dpred\\operatorname\{Pdim\}\(\\mathcal\{F\}\)=d\_\{\\rm pred\}, the classℱ\\mathcal\{F\}is a VC\-subgraph class with VC index at mostdpred\+1d\_\{\\rm pred\}\+1\(the shift by one is only a matter of convention\)\. Because all functions inℱ\\mathcal\{F\}take values in\[0,1\]\[0,1\], Theorem 2\.6\.7 of[Wellner and others \(2013\)](https://arxiv.org/html/2610.20903#bib.bib7)therefore implies that there exist universal constantsa0≥ea\_\{0\}\\geq eandc1\>0c\_\{1\}\>0such that, for every probability measureRRon𝒳\\mathcal\{X\}and everyu∈\(0,1\]u\\in\(0,1\],

log⁡N⁡\(u,ℱ,L2​\(R\)\)≤c1​\(1\+dpred\)​log⁡\(a0/u\)\.\\log N\\bigl\(u,\\mathcal\{F\},L\_\{2\}\(R\)\\bigr\)\\leq c\_\{1\}\(1\+d\_\{\\rm pred\}\)\\log\(a\_\{0\}/u\)\.\(69\)Likewise, since𝒞ℱ\\mathcal\{C\}\_\{\\mathcal\{F\}\}is a\{0,1\}\\\{0,1\\\}\-valued class of VC dimensiondcmpd\_\{\\rm cmp\}, the uniform entropy bound for VC classes of sets\([Wellner and others, 2013](https://arxiv.org/html/2610.20903#bib.bib7), Theorem 2\.6\.4\)gives

log⁡N⁡\(u,𝒞ℱ,L2​\(R\)\)≤c1​\(1\+dcmp\)​log⁡\(a0/u\)\.\\log N\\bigl\(u,\\mathcal\{C\}\_\{\\mathcal\{F\}\},L\_\{2\}\(R\)\\bigr\)\\leq c\_\{1\}\(1\+d\_\{\\rm cmp\}\)\\log\(a\_\{0\}/u\)\.\(70\)The precise universal constants in \([69](https://arxiv.org/html/2610.20903#A12.E69)\)–\([70](https://arxiv.org/html/2610.20903#A12.E70)\) will not be important\.

We first bound the entropy of𝒵\\mathcal\{Z\}\. Forf,h∈ℱf,h\\in\\mathcal\{F\}, let

ch,f\(x\):=𝟙\{h\(x\)\>f\(x\)\}\.c\_\{h,f\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\one\\\{h\(x\)\>f\(x\)\\\}\.Under the conventionsign⁡\(0\)=0\\sign\(0\)=0,

sign⁡\(h−f\)=ch,f−cf,h,\\sign\(h\-f\)=c\_\{h,f\}\-c\_\{f,h\},and hence everyZf,h∈𝒵Z\_\{f,h\}\\in\\mathcal\{Z\}is contained in the larger product class

ℋ:=\{\(x,y\)↦\(y−f\(x\)\)\(c1\(x\)−c2\(x\)\):f∈ℱ,c1,c2∈𝒞ℱ\}\.\\mathcal\{H\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\left\\\{\(x,y\)\\mapsto\(y\-f\(x\)\)\(c\_\{1\}\(x\)\-c\_\{2\}\(x\)\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}f\\in\\mathcal\{F\},\\;c\_\{1\},c\_\{2\}\\in\\mathcal\{C\}\_\{\\mathcal\{F\}\}\\right\\\}\.
The map definingℋ\\mathcal\{H\}is Lipschitz in its three functional arguments\. Indeed, forf,f′∈ℱf,f^\{\\prime\}\\in\\mathcal\{F\}andc1,c2,c1′,c2′∈𝒞ℱc\_\{1\},c\_\{2\},c\_\{1\}^\{\\prime\},c\_\{2\}^\{\\prime\}\\in\\mathcal\{C\}\_\{\\mathcal\{F\}\},

\|\(y−f\)​\(c1−c2\)−\(y−f′\)​\(c1′−c2′\)\|\\displaystyle\\left\|\(y\-f\)\(c\_\{1\}\-c\_\{2\}\)\-\(y\-f^\{\\prime\}\)\(c\_\{1\}^\{\\prime\}\-c\_\{2\}^\{\\prime\}\)\\right\|≤\|f−f′\|\|c1−c2\|\+\|y−f′\|\(\|c1−c1′\|\+\|c2−c2′\|\)\\displaystyle\\quad\\leq\|f\-f^\{\\prime\}\|\\,\|c\_\{1\}\-c\_\{2\}\|\+\|y\-f^\{\\prime\}\|\\bigl\(\|c\_\{1\}\-c\_\{1\}^\{\\prime\}\|\+\|c\_\{2\}\-c\_\{2\}^\{\\prime\}\|\\bigr\)≤\|f−f′\|\+\|c1−c1′\|\+\|c2−c2′\|,\\displaystyle\\quad\\leq\|f\-f^\{\\prime\}\|\+\|c\_\{1\}\-c\_\{1\}^\{\\prime\}\|\+\|c\_\{2\}\-c\_\{2\}^\{\\prime\}\|,\(71\)where the last inequality usesf′,y∈\[0,1\]f^\{\\prime\},y\\in\[0,1\]andc1,c2∈\{0,1\}c\_\{1\},c\_\{2\}\\in\\\{0,1\\\}\.

Fix a probability measureQQon𝒳×\[0,1\]\\mathcal\{X\}\\times\[0,1\], and letQXQ\_\{X\}denote its𝒳\\mathcal\{X\}\-marginal\. Setδ:=u/6\\delta\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=u/6\. Chooseδ\\delta\-covers ofℱ\\mathcal\{F\}and𝒞ℱ\\mathcal\{C\}\_\{\\mathcal\{F\}\}inL2​\(QX\)L\_\{2\}\(Q\_\{X\}\)\. By \([71](https://arxiv.org/html/2610.20903#A12.E71)\) and Minkowski’s inequality, approximating each of the three components withinδ\\deltaproduces anL2​\(Q\)L\_\{2\}\(Q\)error of at most

3​δ=u2\.3\\delta=\\frac\{u\}\{2\}\.Thus the corresponding product functions form an externalu/2u/2\-cover of𝒵\\mathcal\{Z\}\. If covering centers are required to belong to𝒵\\mathcal\{Z\}, then for every covering ball that intersects𝒵\\mathcal\{Z\}choose one element of that intersection as its new center\. By the triangle inequality this produces an internaluu\-cover with no increase in cardinality\. Consequently,

N⁡\(u,𝒵,L2​\(Q\)\)≤N⁡\(u/6,ℱ,L2​\(QX\)\)​N​\(u/6,𝒞ℱ,L2​\(QX\)\)2\.N\\bigl\(u,\\mathcal\{Z\},L\_\{2\}\(Q\)\\bigr\)\\leq N\\bigl\(u/6,\\mathcal\{F\},L\_\{2\}\(Q\_\{X\}\)\\bigr\)N\\bigl\(u/6,\\mathcal\{C\}\_\{\\mathcal\{F\}\},L\_\{2\}\(Q\_\{X\}\)\\bigr\)^\{2\}\.\(72\)
Applying \([69](https://arxiv.org/html/2610.20903#A12.E69)\)–\([70](https://arxiv.org/html/2610.20903#A12.E70)\) to \([72](https://arxiv.org/html/2610.20903#A12.E72)\) gives

log⁡N⁡\(u,𝒵,L2​\(Q\)\)\\displaystyle\\log N\\bigl\(u,\\mathcal\{Z\},L\_\{2\}\(Q\)\\bigr\)≤c1​\[\(1\+dpred\)\+2​\(1\+dcmp\)\]​log⁡\(6​a0/u\)\\displaystyle\\leq c\_\{1\}\\Bigl\[\(1\+d\_\{\\rm pred\}\)\+2\(1\+d\_\{\\rm cmp\}\)\\Bigr\]\\log\(6a\_\{0\}/u\)≤3​c1​d​log⁡\(6​a0/u\),\\displaystyle\\leq 3c\_\{1\}d\\log\(6a\_\{0\}/u\),sinced=1\+dpred\+dcmpd=1\+d\_\{\\rm pred\}\+d\_\{\\rm cmp\}\.

It remains to control𝒟\\mathcal\{D\}\. The absolute\-value map satisfies

\|\|f−h\|−\|f′−h′\|\|≤\|f−f′\|\+\|h−h′\|\.\\bigl\|\|f\-h\|\-\|f^\{\\prime\}\-h^\{\\prime\}\|\\bigr\|\\leq\|f\-f^\{\\prime\}\|\+\|h\-h^\{\\prime\}\|\.Therefore, takingu/2u/2\-covers of the two copies ofℱ\\mathcal\{F\}and applying Minkowski’s inequality yields

N⁡\(u,𝒟,L2​\(QX\)\)≤N​\(u/2,ℱ,L2​\(QX\)\)2\.N\\bigl\(u,\\mathcal\{D\},L\_\{2\}\(Q\_\{X\}\)\\bigr\)\\leq N\\bigl\(u/2,\\mathcal\{F\},L\_\{2\}\(Q\_\{X\}\)\\bigr\)^\{2\}\.\(73\)Here the resulting centers are themselves of the form\|f′−h′\|\|f^\{\\prime\}\-h^\{\\prime\}\|and hence belong to𝒟\\mathcal\{D\}when the covers ofℱ\\mathcal\{F\}are chosen internally\.

Using \([69](https://arxiv.org/html/2610.20903#A12.E69)\) once more,

log⁡N⁡\(u,𝒟,L2​\(QX\)\)≤2​c1​\(1\+dpred\)​log⁡\(2​a0/u\)≤2​c1​d​log⁡\(2​a0/u\)\.\\log N\\bigl\(u,\\mathcal\{D\},L\_\{2\}\(Q\_\{X\}\)\\bigr\)\\leq 2c\_\{1\}\(1\+d\_\{\\rm pred\}\)\\log\(2a\_\{0\}/u\)\\leq 2c\_\{1\}d\\log\(2a\_\{0\}/u\)\.
Choosing universal constants, for exampleA0≥max⁡\{e,6​a0\}A\_\{0\}\\geq\\max\\\{e,6a\_\{0\}\\\}andc0≥3​c1c\_\{0\}\\geq 3c\_\{1\}, proves both asserted bounds\. ∎

#### L\.2A uniform Bernstein inequality

We next give the empirical\-process inequality used to replace the finite union bound in the proof of Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)\. The important feature is that the leading deviation term depends on the individual second momentP​g2Pg^\{2\}\.

###### Lemma 21\(Uniform Bernstein inequality\)\.

LetW1,…,WnW\_\{1\},\\ldots,W\_\{n\}be i\.i\.d\. from a probability measurePP, and let𝒢\\mathcal\{G\}be a class of measurable functionsg:𝒲→\[−1,1\]g\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{W\}\\to\[\-1,1\]\. Suppose there existA≥eA\\geq eandν≥1\\nu\\geq 1such that, for every probability measureQQon𝒲\\mathcal\{W\}and everyu∈\(0,1\]u\\in\(0,1\],

log⁡N⁡\(u,𝒢,L2​\(Q\)\)≤ν​log⁡\(A/u\)\.\\log N\\bigl\(u,\\mathcal\{G\},L\_\{2\}\(Q\)\\bigr\)\\leq\\nu\\log\(A/u\)\.\(74\)Forζ∈\(0,1\)\\zeta\\in\(0,1\), define

β:=ν​log⁡\(A​n\)\+log⁡\(2/ζ\)n\.\\beta\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{\\nu\\log\(An\)\+\\log\(2/\\zeta\)\}\{n\}\.Then, with probability at least1−ζ1\-\\zeta,

\|\(Pn−P\)​g\|≤33​\(P​g2​β\+β\)simultaneously for all​g∈𝒢\.\|\(P\_\{n\}\-P\)g\|\\leq 33\\left\(\\sqrt\{Pg^\{2\}\\,\\beta\}\+\\beta\\right\)\\qquad\\text\{simultaneously for all \}g\\in\\mathcal\{G\}\.\(75\)

###### Proof\.

The proof has two steps\. We first derive a uniform deviation bound for a fixed tuning parameterλ\\lambda, and then optimizeλ\\lambdaon a finite grid\.

##### Step 1: A uniform bound for fixedλ\\lambda\.

We begin with the following elementary inequality: for everya,b∈\[−1,1\]a,b\\in\[\-1,1\]and0<λ≤1/80<\\lambda\\leq 1/8,

12​eλ⁡\(a−b\)−8​λ2​b2\+12​eλ⁡\(b−a\)−8​λ2​a2≤1\.\\frac\{1\}\{2\}e^\{\\lambda\(a\-b\)\-8\\lambda^\{2\}b^\{2\}\}\+\\frac\{1\}\{2\}e^\{\\lambda\(b\-a\)\-8\\lambda^\{2\}a^\{2\}\}\\leq 1\.\(76\)Indeed, the left\-hand side equals

e−4​λ2​\(a2\+b2\)​cosh⁡\(λ⁡\(a−b\)\+4​λ2​\(a2−b2\)\)\.e^\{\-4\\lambda^\{2\}\(a^\{2\}\+b^\{2\}\)\}\\cosh\\\!\\left\(\\lambda\(a\-b\)\+4\\lambda^\{2\}\(a^\{2\}\-b^\{2\}\)\\right\)\.Sincecosh⁡\(t\)≤et2/2\\cosh\(t\)\\leq e^\{t^\{2\}/2\}and

\|λ⁡\(a−b\)\+4​λ2​\(a2−b2\)\|=λ​\|a−b\|​\|1\+4​λ​\(a\+b\)\|≤2​λ​\|a−b\|,\\left\|\\lambda\(a\-b\)\+4\\lambda^\{2\}\(a^\{2\}\-b^\{2\}\)\\right\|=\\lambda\|a\-b\|\\,\|1\+4\\lambda\(a\+b\)\|\\leq 2\\lambda\|a\-b\|,the preceding display is at most

exp⁡\(−4​λ2​\(a2\+b2\)\+2​λ2​\(a−b\)2\)=e−2​λ2​\(a\+b\)2≤1\.\\exp\\\!\\left\(\-4\\lambda^\{2\}\(a^\{2\}\+b^\{2\}\)\+2\\lambda^\{2\}\(a\-b\)^\{2\}\\right\)=e^\{\-2\\lambda^\{2\}\(a\+b\)^\{2\}\}\\leq 1\.
Now fix0<λ≤1/80<\\lambda\\leq 1/8andδ∈\(0,1\)\\delta\\in\(0,1\), and write

H:=ν​log⁡\(A​n\)\.H\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\nu\\log\(An\)\.We claim that, with probability at least1−δ1\-\\delta,

\|\(Pn−P\)​g\|≤8​λ​P​g2\+H\+1\+log⁡\(2/δ\)λ​nfor every​g∈𝒢\.\|\(P\_\{n\}\-P\)g\|\\leq 8\\lambda Pg^\{2\}\+\\frac\{H\+1\+\\log\(2/\\delta\)\}\{\\lambda n\}\\qquad\\text\{for every \}g\\in\\mathcal\{G\}\.\(77\)
LetW1′,…,Wn′W\_\{1\}^\{\\prime\},\\ldots,W\_\{n\}^\{\\prime\}be an independent copy of the sample\. This is the usual ghost\-sample symmetrization device; see, e\.g\.,[Wellner and others \(2013, Section 2\.3\.1\)](https://arxiv.org/html/2610.20903#bib.bib7)\. Define

Dλ​\(g\):=λ​∑i=1n\(g⁡\(Wi\)−g⁡\(Wi′\)\)−8​λ2​∑i=1ng​\(Wi′\)2D\_\{\\lambda\}\(g\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\lambda\\sum\_\{i=1\}^\{n\}\\bigl\(g\(W\_\{i\}\)\-g\(W\_\{i\}^\{\\prime\}\)\\bigr\)\-8\\lambda^\{2\}\\sum\_\{i=1\}^\{n\}g\(W\_\{i\}^\{\\prime\}\)^\{2\}and

Zλ:=supg∈𝒢\{λ​n​\(Pn−P\)​g−8​λ2​n​P​g2\}\.Z\_\{\\lambda\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{g\\in\\mathcal\{G\}\}\\left\\\{\\lambda n\(P\_\{n\}\-P\)g\-8\\lambda^\{2\}nPg^\{2\}\\right\\\}\.Conditional on the original sample,

𝔼W′⁡Dλ​\(g\)=λ​n​\(Pn−P\)​g−8​λ2​nPg2\.\\EE\_\{W^\{\\prime\}\}D\_\{\\lambda\}\(g\)=\\lambda n\(P\_\{n\}\-P\)g\-8\\lambda^\{2\}nPg^\{2\}\.Hence Jensen’s inequality gives

eZλ\\displaystyle e^\{Z\_\{\\lambda\}\}=supg∈𝒢exp⁡\{𝔼W′⁡Dλ​\(g\)\}\\displaystyle=\\sup\_\{g\\in\\mathcal\{G\}\}\\exp\\\!\\left\\\{\\EE\_\{W^\{\\prime\}\}D\_\{\\lambda\}\(g\)\\right\\\}≤supg∈𝒢𝔼W′⁡eDλ​\(g\)\\displaystyle\\leq\\sup\_\{g\\in\\mathcal\{G\}\}\\EE\_\{W^\{\\prime\}\}e^\{D\_\{\\lambda\}\(g\)\}≤𝔼⁡supg∈𝒢W′⁡eDλ​\(g\)\.\\displaystyle\\leq\\EE\_\{W^\{\\prime\}\}\\sup\_\{g\\in\\mathcal\{G\}\}e^\{D\_\{\\lambda\}\(g\)\}\.\(78\)Taking expectation over the original sample,

𝔼⁡eZλ≤𝔼W,W′⁡\[supg∈𝒢eDλ​\(g\)\]\.\\EE e^\{Z\_\{\\lambda\}\}\\leq\\EE\_\{W,W^\{\\prime\}\}\\left\[\\sup\_\{g\\in\\mathcal\{G\}\}e^\{D\_\{\\lambda\}\(g\)\}\\right\]\.\(79\)
Fix the two samples and let

Q:=12​n​∑i=1n\(δWi\+δWi′\)Q\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{2n\}\\sum\_\{i=1\}^\{n\}\\bigl\(\\delta\_\{W\_\{i\}\}\+\\delta\_\{W\_\{i\}^\{\\prime\}\}\\bigr\)be their pooled empirical measure\. By \([74](https://arxiv.org/html/2610.20903#A12.E74)\), there exists anL2​\(Q\)L\_\{2\}\(Q\)\-cover\{h1,…,hM\}⊆𝒢\\\{h\_\{1\},\\ldots,h\_\{M\}\\\}\\subseteq\\mathcal\{G\}of radius1/n1/nsatisfying

M≤\(A​n\)ν=eH\.M\\leq\(An\)^\{\\nu\}=e^\{H\}\.Thus, for everyg∈𝒢g\\in\\mathcal\{G\}, somehjh\_\{j\}satisfies

‖g−hj‖L2​\(Q\)≤1n,Q​\|g−hj\|≤1n,\\\|g\-h\_\{j\}\\\|\_\{L\_\{2\}\(Q\)\}\\leq\\frac\{1\}\{n\},\\qquad Q\|g\-h\_\{j\}\|\\leq\\frac\{1\}\{n\},\(80\)where the second inequality follows from Cauchy–Schwarz\.

Next, independently for eachii, swapWiW\_\{i\}andWi′W\_\{i\}^\{\\prime\}with probability1/21/2\. Denote the resulting ordered pair by\(W~i,W~i′\)\(\\widetilde\{W\}\_\{i\},\\widetilde\{W\}\_\{i\}^\{\\prime\}\), and define

Dλε​\(g\):=λ​∑i=1n\(g⁡\(W~i\)−g⁡\(W~i′\)\)−8​λ2​∑i=1ng​\(W~i′\)2\.D\_\{\\lambda\}^\{\\varepsilon\}\(g\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\lambda\\sum\_\{i=1\}^\{n\}\\bigl\(g\(\\widetilde\{W\}\_\{i\}\)\-g\(\\widetilde\{W\}\_\{i\}^\{\\prime\}\)\\bigr\)\-8\\lambda^\{2\}\\sum\_\{i=1\}^\{n\}g\(\\widetilde\{W\}\_\{i\}^\{\\prime\}\)^\{2\}\.The pooled empirical measureQQis unchanged by the swaps, so the same cover applies to every swap configuration\.

Lethjh\_\{j\}satisfy \([80](https://arxiv.org/html/2610.20903#A12.E80)\)\. Since\|g2−hj2\|≤2​\|g−hj\|\|g^\{2\}\-h\_\{j\}^\{2\}\|\\leq 2\|g\-h\_\{j\}\|, we have

\|Dλε​\(g\)−Dλε​\(hj\)\|\\displaystyle\\left\|D\_\{\\lambda\}^\{\\varepsilon\}\(g\)\-D\_\{\\lambda\}^\{\\varepsilon\}\(h\_\{j\}\)\\right\|≤λ​∑i=1n\(\|g−hj\|​\(W~i\)\+\|g−hj\|​\(W~i′\)\)\\displaystyle\\leq\\lambda\\sum\_\{i=1\}^\{n\}\\left\(\|g\-h\_\{j\}\|\(\\widetilde\{W\}\_\{i\}\)\+\|g\-h\_\{j\}\|\(\\widetilde\{W\}\_\{i\}^\{\\prime\}\)\\right\)\+16λ2∑i=1n\|g−hj\|\(W~i′\)\\displaystyle\\quad\+16\\lambda^\{2\}\\sum\_\{i=1\}^\{n\}\|g\-h\_\{j\}\|\(\\widetilde\{W\}\_\{i\}^\{\\prime\}\)≤\(2​λ\+32​λ2\)​n​Q​\|g−hj\|\\displaystyle\\leq\(2\\lambda\+32\\lambda^\{2\}\)nQ\|g\-h\_\{j\}\|≤2​λ\+32​λ2≤1\.\\displaystyle\\leq 2\\lambda\+32\\lambda^\{2\}\\leq 1\.\(81\)Consequently,

supg∈𝒢eDλε​\(g\)≤e​∑j=1MeDλε​\(hj\)\.\\sup\_\{g\\in\\mathcal\{G\}\}e^\{D\_\{\\lambda\}^\{\\varepsilon\}\(g\)\}\\leq e\\sum\_\{j=1\}^\{M\}e^\{D\_\{\\lambda\}^\{\\varepsilon\}\(h\_\{j\}\)\}\.\(82\)
Conditional on the two samples, fix one cover elementhjh\_\{j\}\. For theiith pair, averaging over the two possible swap configurations gives

12​eλ⁡\(a−b\)−8​λ2​b2\+12​eλ⁡\(b−a\)−8​λ2​a2,a=hj​\(Wi\),b=hj​\(Wi′\),\\frac\{1\}\{2\}e^\{\\lambda\(a\-b\)\-8\\lambda^\{2\}b^\{2\}\}\+\\frac\{1\}\{2\}e^\{\\lambda\(b\-a\)\-8\\lambda^\{2\}a^\{2\}\},\\qquad a=h\_\{j\}\(W\_\{i\}\),\\quad b=h\_\{j\}\(W\_\{i\}^\{\\prime\}\),which is at most11by \([76](https://arxiv.org/html/2610.20903#A12.E76)\)\. Since the swaps are independent acrossii,

𝔼ε⁡eDλε​\(hj\)≤1\.\\EE\_\{\\varepsilon\}e^\{D\_\{\\lambda\}^\{\\varepsilon\}\(h\_\{j\}\)\}\\leq 1\.\(83\)Taking expectation in \([82](https://arxiv.org/html/2610.20903#A12.E82)\) therefore yields

𝔼ε⁡\[supg∈𝒢eDλε​\(g\)\]≤eM≤eH\+1\.\\EE\_\{\\varepsilon\}\\left\[\\sup\_\{g\\in\\mathcal\{G\}\}e^\{D\_\{\\lambda\}^\{\\varepsilon\}\(g\)\}\\right\]\\leq eM\\leq e^\{H\+1\}\.\(84\)
Finally, since each pair\(Wi,Wi′\)\(W\_\{i\},W\_\{i\}^\{\\prime\}\)consists of two i\.i\.d\. observations, independently swapping its coordinates does not change the joint distribution\. Hence

𝔼W,W′⁡\[supg∈𝒢eDλ​\(g\)\]=𝔼W,W′,ε⁡\[supg∈𝒢eDλε​\(g\)\]≤eH\+1\.\\EE\_\{W,W^\{\\prime\}\}\\left\[\\sup\_\{g\\in\\mathcal\{G\}\}e^\{D\_\{\\lambda\}\(g\)\}\\right\]=\\EE\_\{W,W^\{\\prime\},\\varepsilon\}\\left\[\\sup\_\{g\\in\\mathcal\{G\}\}e^\{D\_\{\\lambda\}^\{\\varepsilon\}\(g\)\}\\right\]\\leq e^\{H\+1\}\.Together with \([79](https://arxiv.org/html/2610.20903#A12.E79)\), this gives

𝔼⁡eZλ≤eH\+1\.\\EE e^\{Z\_\{\\lambda\}\}\\leq e^\{H\+1\}\.\(85\)
By Markov’s inequality,

Pr⁡\(Zλ\>H\+1\+log⁡\(2/δ\)\)≤δ2\.\\Pr\\\!\\left\(Z\_\{\\lambda\}\>H\+1\+\\log\(2/\\delta\)\\right\)\\leq\\frac\{\\delta\}\{2\}\.Thus, except on an event of probability at mostδ/2\\delta/2,

\(Pn−P\)​g≤8​λ​P​g2\+H\+1\+log⁡\(2/δ\)λ​n∀g∈𝒢\.\(P\_\{n\}\-P\)g\\leq 8\\lambda Pg^\{2\}\+\\frac\{H\+1\+\\log\(2/\\delta\)\}\{\\lambda n\}\\qquad\\forall g\\in\\mathcal\{G\}\.Applying the same argument to−𝒢\-\\mathcal\{G\}and taking a union bound proves \([77](https://arxiv.org/html/2610.20903#A12.E77)\)\.

##### Step 2: Simultaneously optimizeλ\\lambda\.

Consider the finite grid

Λ:=\{j8​n:j=1,…,n\}\.\\Lambda\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\left\\\{\\frac\{j\}\{8n\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}j=1,\\ldots,n\\right\\\}\.Apply \([77](https://arxiv.org/html/2610.20903#A12.E77)\) to eachλ∈Λ\\lambda\\in\\Lambdawithδ=ζ/n\\delta=\\zeta/n, and take a union bound\. With probability at least1−ζ1\-\\zeta,

\|\(Pn−P\)​g\|≤8​λ​P​g2\+H\+1\+log⁡\(2​n/ζ\)λ​n∀g∈𝒢,∀λ∈Λ\.\|\(P\_\{n\}\-P\)g\|\\leq 8\\lambda Pg^\{2\}\+\\frac\{H\+1\+\\log\(2n/\\zeta\)\}\{\\lambda n\}\\qquad\\forall g\\in\\mathcal\{G\},\\quad\\forall\\lambda\\in\\Lambda\.\(86\)
By definition,

n​β=H\+log⁡\(2/ζ\)\.n\\beta=H\+\\log\(2/\\zeta\)\.Moreover, sinceA≥eA\\geq eandν≥1\\nu\\geq 1,

H=ν​log⁡\(A​n\)≥1\+log⁡n\.H=\\nu\\log\(An\)\\geq 1\+\\log n\.Therefore

H\+1\+log⁡\(2​n/ζ\)=n​β\+1\+log⁡n≤2​n​β\.H\+1\+\\log\(2n/\\zeta\)=n\\beta\+1\+\\log n\\leq 2n\\beta\.Hence, on the same event,

\|\(Pn−P\)​g\|≤8​λ​P​g2\+2​βλ∀g∈𝒢,∀λ∈Λ\.\|\(P\_\{n\}\-P\)g\|\\leq 8\\lambda Pg^\{2\}\+\\frac\{2\\beta\}\{\\lambda\}\\qquad\\forall g\\in\\mathcal\{G\},\\quad\\forall\\lambda\\in\\Lambda\.\(87\)
Fixg∈𝒢g\\in\\mathcal\{G\}and write

v:=P​g2∈\[0,1\]\.v\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=Pg^\{2\}\\in\[0,1\]\.
Ifv≤βv\\leq\\beta, chooseλ=1/8\\lambda=1/8in \([87](https://arxiv.org/html/2610.20903#A12.E87)\)\. Then

\|\(Pn−P\)​g\|≤v\+16​β≤17​β\.\|\(P\_\{n\}\-P\)g\|\\leq v\+16\\beta\\leq 17\\beta\.
Now supposev\>βv\>\\beta, and define the continuous optimizer

λ⋆:=18​βv\.\\lambda\_\{\\star\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\frac\{1\}\{8\}\\sqrt\{\\frac\{\\beta\}\{v\}\}\.Thenλ⋆<1/8\\lambda\_\{\\star\}<1/8\. Also,β≥1/n\\beta\\geq 1/nandv≤1v\\leq 1imply

λ⋆≥18​n≥18​n\.\\lambda\_\{\\star\}\\geq\\frac\{1\}\{8\\sqrt\{n\}\}\\geq\\frac\{1\}\{8n\}\.Letλ\\lambdabe the largest element ofΛ\\Lambdasatisfyingλ≤λ⋆\\lambda\\leq\\lambda\_\{\\star\}\. Since the grid spacing is1/\(8​n\)1/\(8n\),

λ⋆2≤λ≤λ⋆\.\\frac\{\\lambda\_\{\\star\}\}\{2\}\\leq\\lambda\\leq\\lambda\_\{\\star\}\.Using \([87](https://arxiv.org/html/2610.20903#A12.E87)\),

\|\(Pn−P\)​g\|\\displaystyle\|\(P\_\{n\}\-P\)g\|≤8​λ⋆​v\+4​βλ⋆\\displaystyle\\leq 8\\lambda\_\{\\star\}v\+\\frac\{4\\beta\}\{\\lambda\_\{\\star\}\}=v​β\+32​v​β\\displaystyle=\\sqrt\{v\\beta\}\+32\\sqrt\{v\\beta\}=33​v​β\.\\displaystyle=33\\sqrt\{v\\beta\}\.
Combining the two cases yields

\|\(Pn−P\)​g\|≤33​\(P​g2​β\+β\)∀g∈𝒢\.\|\(P\_\{n\}\-P\)g\|\\leq 33\\left\(\\sqrt\{Pg^\{2\}\\,\\beta\}\+\\beta\\right\)\\qquad\\forall g\\in\\mathcal\{G\}\.Since the event in \([87](https://arxiv.org/html/2610.20903#A12.E87)\) is common to allg∈𝒢g\\in\\mathcal\{G\}, this proves the result\. ∎

#### L\.3Proof of Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)

We now combine Proposition[20](https://arxiv.org/html/2610.20903#Thmtheorem20)and Lemma[21](https://arxiv.org/html/2610.20903#Thmtheorem21)with the same pairwise argument used for the finite\-class result\.

###### Proof of Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)\.

Let

L^​\(f\):=suph∈ℱT^​\(f,h\)\.\\widehat\{L\}\(f\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sup\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f,h\)\.By Proposition[20](https://arxiv.org/html/2610.20903#Thmtheorem20), the classes𝒵\\mathcal\{Z\}and𝒟\\mathcal\{D\}satisfy the entropy condition in Lemma[21](https://arxiv.org/html/2610.20903#Thmtheorem21)with parameters of orderdd\. Applying Lemma[21](https://arxiv.org/html/2610.20903#Thmtheorem21)separately to𝒵\\mathcal\{Z\}and𝒟\\mathcal\{D\}, each with failure probabilityδ/2\\delta/2, and increasing the universal constantccin the definition ofαδ\\alpha\_\{\\delta\}if necessary, we obtain an event of probability at least1−δ1\-\\deltaon which, simultaneously for everyf,h∈ℱf,h\\in\\mathcal\{F\},

\(Pn−P\)​Zf,h\\displaystyle\(P\_\{n\}\-P\)Z\_\{f,h\}≤2​P​Zf,h2​αδ\+43​αδ,\\displaystyle\\leq\\sqrt\{2PZ\_\{f,h\}^\{2\}\\,\\alpha\_\{\\delta\}\}\+\\frac\{4\}\{3\}\\alpha\_\{\\delta\},\(88\)\(P−Pn\)​Df,h\\displaystyle\(P\-P\_\{n\}\)D\_\{f,h\}≤2​P​Df,h​αδ\+23​αδ\.\\displaystyle\\leq\\sqrt\{2PD\_\{f,h\}\\,\\alpha\_\{\\delta\}\}\+\\frac\{2\}\{3\}\\alpha\_\{\\delta\}\.\(89\)For \([89](https://arxiv.org/html/2610.20903#A12.E89)\), we usedP​Df,h2≤P​Df,hPD\_\{f,h\}^\{2\}\\leq PD\_\{f,h\}, since0≤Df,h≤10\\leq D\_\{f,h\}\\leq 1\. We work on this event for the remainder of the proof\.

Fix any comparatorf0∈ℱf\_\{0\}\\in\\mathcal\{F\}\. By \([88](https://arxiv.org/html/2610.20903#A12.E88)\), for everyh∈ℱh\\in\\mathcal\{F\},

Pn​Zf0,h≤P​Zf0,h\+2​P​Zf0,h2​αδ\+43​αδ\.P\_\{n\}Z\_\{f\_\{0\},h\}\\leq PZ\_\{f\_\{0\},h\}\+\\sqrt\{2PZ\_\{f\_\{0\},h\}^\{2\}\\,\\alpha\_\{\\delta\}\}\+\\frac\{4\}\{3\}\\alpha\_\{\\delta\}\.SinceP​Zf0,h≤BP​\(f0\)PZ\_\{f\_\{0\},h\}\\leq B\_\{P\}\(f\_\{0\}\)andP​Zf0,h2≤v⁡\(f0\)PZ\_\{f\_\{0\},h\}^\{2\}\\leq v\(f\_\{0\}\), taking the supremum overhhyields

L^​\(f0\)≤BP​\(f0\)\+2​v​\(f0\)​αδ\+43​αδ\.\\widehat\{L\}\(f\_\{0\}\)\\leq B\_\{P\}\(f\_\{0\}\)\+\\sqrt\{2v\(f\_\{0\}\)\\alpha\_\{\\delta\}\}\+\\frac\{4\}\{3\}\\alpha\_\{\\delta\}\.\(90\)
The empirical analogue of Lemma[5](https://arxiv.org/html/2610.20903#Thmtheorem5)holds exactly:

T^​\(f,h\)\+T^​\(h,f\)=Pn​\|f−h\|\.\\widehat\{T\}\(f,h\)\+\\widehat\{T\}\(h,f\)=P\_\{n\}\|f\-h\|\.Hence, by the empirical optimality off^\\hat\{f\},

Pn​\|f^−f0\|\\displaystyle P\_\{n\}\|\\hat\{f\}\-f\_\{0\}\|=T^​\(f^,f0\)\+T^​\(f0,f^\)\\displaystyle=\\widehat\{T\}\(\\hat\{f\},f\_\{0\}\)\+\\widehat\{T\}\(f\_\{0\},\\hat\{f\}\)≤L^​\(f^\)\+L^​\(f0\)≤2​L^​\(f0\)\.\\displaystyle\\leq\\widehat\{L\}\(\\hat\{f\}\)\+\\widehat\{L\}\(f\_\{0\}\)\\leq 2\\widehat\{L\}\(f\_\{0\}\)\.\(91\)
Letr:=‖f^−f0‖1r\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathinner\{\\\!\\left\\\|\\hat\{f\}\-f\_\{0\}\\right\\\|\}\_\{1\}\. Since \([89](https://arxiv.org/html/2610.20903#A12.E89)\) holds uniformly over all pairs, it applies to the data\-dependent pair\(f^,f0\)\(\\hat\{f\},f\_\{0\}\):

r≤Pn​\|f^−f0\|\+2​r​αδ\+23​αδ\.r\\leq P\_\{n\}\|\\hat\{f\}\-f\_\{0\}\|\+\\sqrt\{2r\\alpha\_\{\\delta\}\}\+\\frac\{2\}\{3\}\\alpha\_\{\\delta\}\.Combining this inequality with \([90](https://arxiv.org/html/2610.20903#A12.E90)\) and \([91](https://arxiv.org/html/2610.20903#A12.E91)\) gives

r≤2​BP​\(f0\)\+2​2​v​\(f0\)​αδ\+2​r​αδ\+103​αδ\.r\\leq 2B\_\{P\}\(f\_\{0\}\)\+2\\sqrt\{2v\(f\_\{0\}\)\\alpha\_\{\\delta\}\}\+\\sqrt\{2r\\alpha\_\{\\delta\}\}\+\\frac\{10\}\{3\}\\alpha\_\{\\delta\}\.\(92\)
Fixρ\>0\\rho\>0and letθ:=ρ/\(2\+ρ\)\\theta\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\rho/\(2\+\\rho\)\. By Young’s inequality,

2​r​αδ≤θ​r\+αδ2​θ\.\\sqrt\{2r\\alpha\_\{\\delta\}\}\\leq\\theta r\+\\frac\{\\alpha\_\{\\delta\}\}\{2\\theta\}\.Substituting this into \([92](https://arxiv.org/html/2610.20903#A12.E92)\) and using1/\(1−θ\)=\(2\+ρ\)/21/\(1\-\\theta\)=\(2\+\\rho\)/2gives

r≤\(2\+ρ\)​\(BP​\(f0\)\+2​v​\(f0\)​αδ\)\+Cρ​αδ,r\\leq\(2\+\\rho\)\\left\(B\_\{P\}\(f\_\{0\}\)\+\\sqrt\{2v\(f\_\{0\}\)\\alpha\_\{\\delta\}\}\\right\)\+C\_\{\\rho\}\\alpha\_\{\\delta\},whereCρC\_\{\\rho\}is the same constant as in Theorem[8](https://arxiv.org/html/2610.20903#Thmtheorem8)\.

Finally, by the triangle inequality,

ε1​\(f^\)≤ε1​\(f0\)\+‖f^−f0‖1\.\\varepsilon\_\{1\}\(\\hat\{f\}\)\\leq\\varepsilon\_\{1\}\(f\_\{0\}\)\+\\mathinner\{\\\!\\left\\\|\\hat\{f\}\-f\_\{0\}\\right\\\|\}\_\{1\}\.The concentration event is independent off0f\_\{0\}andρ\\rho, so the resulting inequality holds simultaneously for everyf0∈ℱf\_\{0\}\\in\\mathcal\{F\}and everyρ\>0\\rho\>0\. Taking the infimum overf0f\_\{0\}proves \([26](https://arxiv.org/html/2610.20903#S6.E26)\)\. ∎

#### L\.4Examples of infinite hypothesis classes

We conclude by verifying the complexity conditions in Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)for several standard prediction classes\. We use standard results on the VC dimension of linear threshold classes and the pseudo\-dimension of real\-valued linear classes; see, e\.g\.,[Vapnik \(1968\)](https://arxiv.org/html/2610.20903#bib.bib8);[Anthony and Bartlett \(2009\)](https://arxiv.org/html/2610.20903#bib.bib9);[Shalev\-Shwartz and Ben\-David \(2014\)](https://arxiv.org/html/2610.20903#bib.bib3)\. Recall that the relevant complexity isd=1\+dpred\+dcmpd=1\+d\_\{\\rm pred\}\+d\_\{\\rm cmp\}\.

##### Bounded linear prediction classes\.

Letϕ:𝒳→ℝm\\phi\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\\mathbb\{R\}^\{m\}be a fixed measurable feature map anda:𝒳→ℝa\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\\mathbb\{R\}a fixed offset\. Consider

ℱlin:=\{fθ\(x\)=a\(x\)\+⟨θ,ϕ\(x\)⟩:θ∈Θ\}⊆\{f:𝒳→\[0,1\]\},\\mathcal\{F\}\_\{\\rm lin\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\left\\\{f\_\{\\theta\}\(x\)=a\(x\)\+\\langle\\theta,\\phi\(x\)\\rangle\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\theta\\in\\Theta\\right\\\}\\subseteq\\\{f\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\[0,1\]\\\},whereΘ⊆ℝm\\Theta\\subseteq\\mathbb\{R\}^\{m\}is any parameter set for which the predictions lie in\[0,1\]\[0,1\]\.

For a subgraph thresholdtt,

fθ\(x\)\>t⟺⟨\(θ,1\),\(ϕ\(x\),a\(x\)−t\)⟩\>0\.f\_\{\\theta\}\(x\)\>t\\quad\\Longleftrightarrow\\quad\\left\\langle\(\\theta,1\),\(\\phi\(x\),a\(x\)\-t\)\\right\\rangle\>0\.Thus the subgraph class is a subclass of the class of homogeneous halfspaces inℝm\+1\\mathbb\{R\}^\{m\+1\}, and hence

dpred≤m\+1\.d\_\{\\rm pred\}\\leq m\+1\.Moreover,

fγ\(x\)\>fθ\(x\)⟺⟨γ−θ,ϕ\(x\)⟩\>0\.f\_\{\\gamma\}\(x\)\>f\_\{\\theta\}\(x\)\\quad\\Longleftrightarrow\\quad\\langle\\gamma\-\\theta,\\phi\(x\)\\rangle\>0\.Therefore𝒞ℱlin\\mathcal\{C\}\_\{\\mathcal\{F\}\_\{\\rm lin\}\}is a subclass of the class of homogeneous halfspaces inℝm\\mathbb\{R\}^\{m\}, and hence

dcmp≤m\.d\_\{\\rm cmp\}\\leq m\.Consequently,d=O⁡\(m\)d=O\(m\), and Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)applies with

αδ≲m​log⁡\(e​n\)\+log⁡\(2/δ\)n\.\\alpha\_\{\\delta\}\\lesssim\\frac\{m\\log\(en\)\+\\log\(2/\\delta\)\}\{n\}\.

##### Logistic and other monotone\-link prediction classes\.

Letℓ:ℝ→\(0,1\)\\ell\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathbb\{R\}\\to\(0,1\)be a fixed strictly increasing bijection, and consider

ℱℓ:=\{fθ\(x\)=ℓ\(a\(x\)\+⟨θ,ϕ\(x\)⟩\):θ∈Θ\}\.\\mathcal\{F\}\_\{\\ell\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\left\\\{f\_\{\\theta\}\(x\)=\\ell\\\!\\left\(a\(x\)\+\\langle\\theta,\\phi\(x\)\\rangle\\right\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\theta\\in\\Theta\\right\\\}\.This includes logistic prediction withℓ⁡\(z\)=\(1\+e−z\)−1\\ell\(z\)=\(1\+e^\{\-z\}\)^\{\-1\}and probit prediction withℓ⁡\(z\)=Φ⁡\(z\)\\ell\(z\)=\\Phi\(z\)\. Sinceℓ\\ellis strictly increasing, for everyt∈\(0,1\)t\\in\(0,1\),

fθ\(x\)\>t⟺⟨\(θ,1\),\(ϕ\(x\),a\(x\)−ℓ−1\(t\)\)⟩\>0\.f\_\{\\theta\}\(x\)\>t\\quad\\Longleftrightarrow\\quad\\left\\langle\(\\theta,1\),\(\\phi\(x\),a\(x\)\-\\ell^\{\-1\}\(t\)\)\\right\\rangle\>0\.Thus the subgraph class is again a subclass of the class of homogeneous halfspaces inℝm\+1\\mathbb\{R\}^\{m\+1\}, yielding

dpred≤m\+1\.d\_\{\\rm pred\}\\leq m\+1\.Similarly,

fγ\(x\)\>fθ\(x\)⟺⟨γ−θ,ϕ\(x\)⟩\>0,f\_\{\\gamma\}\(x\)\>f\_\{\\theta\}\(x\)\\quad\\Longleftrightarrow\\quad\\langle\\gamma\-\\theta,\\phi\(x\)\\rangle\>0,so𝒞ℱℓ\\mathcal\{C\}\_\{\\mathcal\{F\}\_\{\\ell\}\}is a subclass of the class of homogeneous halfspaces inℝm\\mathbb\{R\}^\{m\}, and therefore

dcmp≤m\.d\_\{\\rm cmp\}\\leq m\.Consequently,d=O⁡\(m\)d=O\(m\)and

αδ≲m​log⁡\(e​n\)\+log⁡\(2/δ\)n\.\\alpha\_\{\\delta\}\\lesssim\\frac\{m\\log\(en\)\+\\log\(2/\\delta\)\}\{n\}\.

##### Sparse linear and sparse monotone\-link prediction classes\.

Letϕ:𝒳→ℝp\\phi\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}\\mathcal\{X\}\\to\\mathbb\{R\}^\{p\}, let1≤s≤p1\\leq s\\leq p, and restrict parameter vectors to satisfy‖θ‖0≤s\\mathinner\{\\\!\\left\\\|\\theta\\right\\\|\}\_\{0\}\\leq s\. Consider either the bounded linear class

fθ​\(x\)=a⁡\(x\)\+⟨θ,ϕ⁡\(x\)⟩f\_\{\\theta\}\(x\)=a\(x\)\+\\langle\\theta,\\phi\(x\)\\rangleor the monotone\-link class

fθ​\(x\)=ℓ⁡\(a⁡\(x\)\+⟨θ,ϕ⁡\(x\)⟩\),f\_\{\\theta\}\(x\)=\\ell\\\!\\left\(a\(x\)\+\\langle\\theta,\\phi\(x\)\\rangle\\right\),with the linear class restricted as needed so that its predictions lie in\[0,1\]\[0,1\]\.

There are at most

∑j=0s\(pj\)≤\(e​ps\)s\\sum\_\{j=0\}^\{s\}\\binom\{p\}\{j\}\\leq\\left\(\\frac\{ep\}\{s\}\\right\)^\{s\}possible supports of size at mostss, and conditional on a fixed support the prediction class has pseudo\-dimensionO⁡\(s\)O\(s\)\. Combining the corresponding growth\-function bound with a union bound over supports gives

dpred=O⁡\(s​log⁡e​ps\)\.d\_\{\\rm pred\}=O\\\!\\left\(s\\log\\frac\{ep\}\{s\}\\right\)\.See, e\.g\.,[Anthony and Bartlett \(2009\)](https://arxiv.org/html/2610.20903#bib.bib9);[Shalev\-Shwartz and Ben\-David \(2014\)](https://arxiv.org/html/2610.20903#bib.bib3)for the standard growth\-function and pseudo\-dimension machinery\.

For pairwise comparisons, ifθ\\thetaandγ\\gammaare bothss\-sparse, thenγ−θ\\gamma\-\\thetais at most2​s2s\-sparse\. Hence𝒞ℱ\\mathcal\{C\}\_\{\\mathcal\{F\}\}is a subclass of the class of2​s2s\-sparse homogeneous halfspaces inℝp\\mathbb\{R\}^\{p\}\. Applying the same support\-counting and growth\-function argument gives

dcmp=O⁡\(s​log⁡e​ps\)\.d\_\{\\rm cmp\}=O\\\!\\left\(s\\log\\frac\{ep\}\{s\}\\right\)\.Related explicit lower and upper bounds on the VC dimension of sparse halfspaces are given by[Ahsen and Vidyasagar \(2019\)](https://arxiv.org/html/2610.20903#bib.bib10)\. Therefore

d=O⁡\(s​log⁡e​ps\),d=O\\\!\\left\(s\\log\\frac\{ep\}\{s\}\\right\),and Theorem[14](https://arxiv.org/html/2610.20903#Thmtheorem14)applies with

αδ≲s​log⁡\(e​p/s\)​log⁡\(e​n\)\+log⁡\(2/δ\)n\.\\alpha\_\{\\delta\}\\lesssim\\frac\{s\\log\(ep/s\)\\log\(en\)\+\\log\(2/\\delta\)\}\{n\}\.

### Appendix MExperiments

#### M\.1Synthetic Experiment

In this section, we provide empirical results that validate our theoretical findings\. Our synthetic experiment is a controlled finite\-class sanity check\. It isolates the approximation behavior predicted by the theory by exactly enumerating a small candidate class\. It is not intended to demonstrate scalable optimization of the sign\-betting objective over large hypothesis classes\.

##### Function class\.

We consider add\-dimensional regression setting with target regression functionη\\eta\. We use the logistic prediction family

fθ​\(x\):=σ⁡\(x⊤​θ\),f\_\{\\theta\}\(x\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\sigma\(x^\{\\top\}\\theta\),whereσ⁡\(z\):=1/\(1\+exp⁡\(−z\)\)\\sigma\(z\)\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=1/\(1\+\\exp\(\-z\)\)is the sigmoid function\. We define the target regression function as

η⁡\(x\)=12,∀x∈𝒳\.\\eta\(x\)=\\frac\{1\}\{2\},\\qquad\\forall x\\in\\mathcal\{X\}\.
To study model misspecification in a controlled setting, we consider a finite function class that does not contain the target regression function\. Specifically, we choose two inputs

xc=e1,xr=e2,x\_\{c\}=e\_\{1\},\\qquad x\_\{r\}=e\_\{2\},wheree1,e2∈ℝde\_\{1\},e\_\{2\}\\in\\mathbb\{R\}^\{d\}are the first two standard basis vectors, and construct two groups of candidate functions\.

The first group consists of*local\-error*candidates that agree with the target function onxcx\_\{c\}but deviate from it onxrx\_\{r\}:

fjL​\(xc\)=12,fjL​\(xr\)=12\+Aj\.f\_\{j\}^\{L\}\(x\_\{c\}\)=\\frac\{1\}\{2\},\\qquad f\_\{j\}^\{L\}\(x\_\{r\}\)=\\frac\{1\}\{2\}\+A\_\{j\}\.Their corresponding parameters are

θjL=\(0,logit⁡\(12\+Aj\),0,…,0\)\.\\theta\_\{j\}^\{L\}=\\left\(0,\\,\\operatorname\{logit\}\\\!\\left\(\\frac\{1\}\{2\}\+A\_\{j\}\\right\),0,\\ldots,0\\right\)\.
The second group consists of*distributed\-error*candidates that deviate from the target function on both inputs:

fjD​\(xc\)=fjD​\(xr\)=12\+Bj\.f\_\{j\}^\{D\}\(x\_\{c\}\)=f\_\{j\}^\{D\}\(x\_\{r\}\)=\\frac\{1\}\{2\}\+B\_\{j\}\.Their corresponding parameters are

θjD=\(logit⁡\(12\+Bj\),logit⁡\(12\+Bj\),0,…,0\)\.\\theta\_\{j\}^\{D\}=\\left\(\\operatorname\{logit\}\\\!\\left\(\\frac\{1\}\{2\}\+B\_\{j\}\\right\),\\operatorname\{logit\}\\\!\\left\(\\frac\{1\}\{2\}\+B\_\{j\}\\right\),0,\\ldots,0\\right\)\.
LetℱL\\mathcal\{F\}\_\{L\}andℱD\\mathcal\{F\}\_\{D\}denote the two groups of candidate functions\. We finally construct the finite function class as

ℱ:=ℱL∪ℱD\.\\mathcal\{F\}\\mathrel\{\\mathop\{\\mathchar 58\\relax\}\}=\\mathcal\{F\}\_\{L\}\\cup\\mathcal\{F\}\_\{D\}\.The target regression functionη\\etais not contained inℱ\\mathcal\{F\}\.

##### Data distribution\.

We consider a two\-point input distribution with

xc=e1=\(1,0\)⊤,xr=e2=\(0,1\)⊤,x\_\{c\}=e\_\{1\}=\(1,0\)^\{\\top\},\\qquad x\_\{r\}=e\_\{2\}=\(0,1\)^\{\\top\},and sample

X=\{xc,with probability​1−p,xr,with probability​p\.X=\\begin\{cases\}x\_\{c\},&\\text\{with probability \}1\-p,\\\\ x\_\{r\},&\\text\{with probability \}p\.\\end\{cases\}The target regression function is fixed as

η⁡\(x\)=𝔼⁡\[Y∣X=x\]=12\.\\eta\(x\)=\\mathbb\{E\}\[Y\\mid X=x\]=\\frac\{1\}\{2\}\.
We use a parameterρ∈\[0,1\)\\rho\\in\[0,1\)to control the conditional label noise\. This experimental noise parameter is unrelated to the parameterρ\\rhoappearing in our theoretical bounds\. Whenρ=0\\rho=0, the label is deterministic:

Y\|X=x=12\.Y\\mid X=x=\\frac\{1\}\{2\}\.Forρ\>0\\rho\>0, we generate the label as

Y\|X=x∼Beta⁡\(η⁡\(x\)​1−ρρ,\(1−η⁡\(x\)\)​1−ρρ\)\.Y\\mid X=x\\sim\\operatorname\{Beta\}\\left\(\\eta\(x\)\\frac\{1\-\\rho\}\{\\rho\},\(1\-\\eta\(x\)\)\\frac\{1\-\\rho\}\{\\rho\}\\right\)\.Sinceη⁡\(x\)=1/2\\eta\(x\)=1/2, this reduces to

Y\|X=x∼Beta⁡\(1−ρ2​ρ,1−ρ2​ρ\)\.Y\\mid X=x\\sim\\operatorname\{Beta\}\\left\(\\frac\{1\-\\rho\}\{2\\rho\},\\frac\{1\-\\rho\}\{2\\rho\}\\right\)\.In both cases,

𝔼⁡\[Y∣X=x\]=12,Var⁡\(Y∣X=x\)=ρ4,\\mathbb\{E\}\[Y\\mid X=x\]=\\frac\{1\}\{2\},\\qquad\\operatorname\{Var\}\(Y\\mid X=x\)=\\frac\{\\rho\}\{4\},whereρ=0\\rho=0corresponds to the deterministic label setting\.

##### Training\.

Given the candidate setℱ\\mathcal\{F\}, we select a function by minimizing either log loss, squared loss, or sign betting loss on the validation dataDnvalD\_\{n\}^\{\\mathrm\{val\}\}\. The log loss offfis

Lnlog\(f\)=−1n∑\(x,y\)∈Dn\(ylogf\(x\)\+\(1−y\)log\(1−f\(x\)\)\),L\_\{n\}^\{\\log\}\(f\)=\-\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}\}\\left\(y\\log f\(x\)\+\(1\-y\)\\log\(1\-f\(x\)\)\\right\),and the selected function is

f^log=arg⁡minf∈ℱ​Lnlog​\(f\)\.\\widehat\{f\}^\{\\log\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}L\_\{n\}^\{\\log\}\(f\)\.
Similarly, the squared loss offfis

Lnsq​\(f\)=1n​∑\(x,y\)∈Dn\(y−f⁡\(x\)\)2,L\_\{n\}^\{\\mathrm\{sq\}\}\(f\)=\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}\}\\left\(y\-f\(x\)\\right\)^\{2\},and the selected function is

f^sq=arg⁡minf∈ℱ​Lnsq​\(f\)\.\\widehat\{f\}^\{\\mathrm\{sq\}\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}L\_\{n\}^\{\\mathrm\{sq\}\}\(f\)\.
For sign betting loss, for eachf,h∈ℱf,h\\in\\mathcal\{F\}, we define

T^​\(f,h\)=1n​∑\(x,y\)∈Dn\(y−f⁡\(x\)\)​sign⁡\(h⁡\(x\)−f⁡\(x\)\),\\widehat\{T\}\(f,h\)=\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}\}\\left\(y\-f\(x\)\\right\)\\operatorname\{sign\}\\\!\\left\(h\(x\)\-f\(x\)\\right\),wheresign⁡\(0\)=0\\operatorname\{sign\}\(0\)=0, and select

f^SB=arg⁡minf∈ℱ​maxh∈ℱ​T^​\(f,h\)\.\\widehat\{f\}^\{\\mathrm\{SB\}\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}\\max\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f,h\)\.

##### Evaluation\.

For each selected functionf^\\widehat\{f\}, we evaluate its population mean absolute error \(MAE\),

ε1​\(f^\)=𝔼X​\[\|f^​\(X\)−η⁡\(X\)\|\]\.\\varepsilon\_\{1\}\(\\widehat\{f\}\)=\\mathbb\{E\}\_\{X\}\\left\[\\left\|\\widehat\{f\}\(X\)\-\\eta\(X\)\\right\|\\right\]\.
Since the input distribution is supported on only two points, this expectation can be computed exactly as

ε1​\(f^\)=\(1−p\)​\|f^​\(xc\)−12\|\+p​\|f^​\(xr\)−12\|\.\\varepsilon\_\{1\}\(\\widehat\{f\}\)=\(1\-p\)\\left\|\\widehat\{f\}\(x\_\{c\}\)\-\\frac\{1\}\{2\}\\right\|\+p\\left\|\\widehat\{f\}\(x\_\{r\}\)\-\\frac\{1\}\{2\}\\right\|\.
We fix the feature dimension tod=2d=2and the rare\-input probability top=0\.1p=0\.1, vary the training sample\-to\-dimension ration/d∈\{2,4,8,16,32,64\},n/d\\in\\\{2,4,8,16,32,64\\\},and consider different noise levelsρ∈\{0,0\.02,0\.04\}\.\\rho\\in\\\{0,0\.02,0\.04\\\}\.For each configuration\(n/d,ρ\)\(n/d,\\rho\), we repeat the experimentB=1000B=1000times\. Iff^b\\widehat\{f\}\_\{b\}denotes the selected function in repetitionbb, we report the average MAE

ε¯1=1B​∑b=1Bε1​\(f^b\),\\overline\{\\varepsilon\}\_\{1\}=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{b\}\),with standard error

SE=SD⁡\(ε1​\(f^1\),…,ε1​\(f^B\)\)B\.\\mathrm\{SE\}=\\frac\{\\operatorname\{SD\}\\left\(\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{1\}\),\\ldots,\\varepsilon\_\{1\}\(\\widehat\{f\}\_\{B\}\)\\right\)\}\{\\sqrt\{B\}\}\.
Table 1:Comparison of average mean absolute error \(MAE\) under log loss, squared loss, and sign betting loss across varyingn/dn/d\. Reported values are the mean MAE over repeated trials, with±\\pmdenoting the standard error\. Best results are shown inbold\. Sign betting loss achieves the lowest MAE for all tested values ofn/dn/d, tying with the other methods atn/d=2n/d=2\.
##### Results\.

Table[1](https://arxiv.org/html/2610.20903#A13.T1)reports the average MAE with standard error for the noiseless setting \(ρ=0\\rho=0\), while Figure[1](https://arxiv.org/html/2610.20903#A13.F1)summarizes the corresponding results across different noise levelsρ∈\{0,0\.02,0\.04\}\\rho\\in\\\{0,0\.02,0\.04\\\}and varying values of sample\-to\-dimension ration/dn/d\. In the noiseless setting, all three methods achieve the same average MAE at the smallest sample size\. For all larger values ofn/dn/d, sign betting consistently achieves the lowest MAE, and its MAE decreases toward the best population MAE attainable within the function class as the sample size increases\. Figure[1](https://arxiv.org/html/2610.20903#A13.F1)shows that this advantage persists in the presence of label noise\. Across both noisy settings, sign betting consistently achieves the lowest MAE over all tested values ofn/dn/d, and its MAE decreases as the sample size increases\.

Figure 1:Comparison of average mean absolute error \(MAE\) obtained under log loss, squared loss, and sign betting loss in the synthetic experiment across varyingn/dn/dand noise levelsρ∈\{0,0\.02,0\.04\}\\rho\\in\\\{0,0\.02,0\.04\\\}\. Error bars denote the standard error\.

#### M\.2Real\-World Experiment

##### Dataset\.

We use the KonIQ\-10k[Hosu et al\. \(2020\)](https://arxiv.org/html/2610.20903#bib.bib21)image\-quality dataset, in which each image is associated with multiple human quality ratings on a five\-point scale\. We rescale each ratingr∈\{1,2,3,4,5\}r\\in\\\{1,2,3,4,5\\\}toY=r−14∈\{0,14,12,34,1\},Y=\\frac\{r\-1\}\{4\}\\in\\left\\\{0,\\frac\{1\}\{4\},\\frac\{1\}\{2\},\\frac\{3\}\{4\},1\\right\\\},so that the responses lie in\[0,1\]\[0,1\], consistent with our regression setting\.

For an imagexx, letck​\(x\)c\_\{k\}\(x\)denote the number of observed ratings at levelk∈\{1,…,5\}k\\in\\\{1,\\ldots,5\\\}\. We define the empirical rating distribution by

q^k​\(x\)=ck​\(x\)∑ℓ=15cℓ​\(x\),\\widehat\{q\}\_\{k\}\(x\)=\\frac\{c\_\{k\}\(x\)\}\{\\sum\_\{\\ell=1\}^\{5\}c\_\{\\ell\}\(x\)\},and its empirical mean by

η¯​\(x\)=∑k=15q^k​\(x\)​k−14\.\\bar\{\\eta\}\(x\)=\\sum\_\{k=1\}^\{5\}\\widehat\{q\}\_\{k\}\(x\)\\frac\{k\-1\}\{4\}\.Hereη⁡\(x\)=𝔼⁡\[Y∣X=x\]\\eta\(x\)=\\mathbb\{E\}\[Y\\mid X=x\]denotes the population conditional mean from our theoretical formulation, whileη¯​\(x\)\\bar\{\\eta\}\(x\)is its empirical counterpart computed from all available human ratings for imagexx\.

##### Function class\.

We extract image features using a ResNet\-18[He et al\. \(2016\)](https://arxiv.org/html/2610.20903#bib.bib22)backbone pretrained on ImageNet[Russakovsky et al\. \(2015\)](https://arxiv.org/html/2610.20903#bib.bib23), with all backbone parameters frozen\. Based on these features, we train five families of regression models on the training set using different model configurations and random seeds\.Together, these models form a finite hypothesis class containing 38 predictors,

ℱ=\{f1,…,f38\}\.\\mathcal\{F\}=\\\{f\_\{1\},\\ldots,f\_\{38\}\\\}\.Once the candidate models are trained, all model parameters are frozen during model selection\.

##### Training\.

Given the hypothesis classℱ\\mathcal\{F\}, we select a function using log loss, squared loss, or sign betting loss on the validation set\. For each repetition, we use the samenndistinct validation images for all three methods and sample one human rating for each selected image, forming

Dnval=\{\(Xi,Yi\)\}i=1n\.D\_\{n\}^\{\\mathrm\{val\}\}=\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}\.
The log loss offfis

Lnlog\(f\)=−1n∑\(x,y\)∈Dnval\(ylogf\(x\)\+\(1−y\)log\(1−f\(x\)\)\),L\_\{n\}^\{\\log\}\(f\)=\-\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}^\{\\mathrm\{val\}\}\}\\left\(y\\log f\(x\)\+\(1\-y\)\\log\(1\-f\(x\)\)\\right\),and the selected function is

f^log=arg⁡minf∈ℱ​Lnlog​\(f\)\.\\widehat\{f\}^\{\\log\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}L\_\{n\}^\{\\log\}\(f\)\.
Similarly, the squared loss offfis

Lnsq​\(f\)=1n​∑\(x,y\)∈Dnval\(y−f⁡\(x\)\)2,L\_\{n\}^\{\\mathrm\{sq\}\}\(f\)=\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}^\{\\mathrm\{val\}\}\}\\left\(y\-f\(x\)\\right\)^\{2\},and the selected function is

f^sq=arg⁡minf∈ℱ​Lnsq​\(f\)\.\\widehat\{f\}^\{\\mathrm\{sq\}\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}L\_\{n\}^\{\\mathrm\{sq\}\}\(f\)\.
For sign betting, we use

T^​\(f,h\)=1n​∑\(x,y\)∈Dnval\(y−f⁡\(x\)\)​sign⁡\(h⁡\(x\)−f⁡\(x\)\),\\widehat\{T\}\(f,h\)=\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}^\{\\mathrm\{val\}\}\}\\left\(y\-f\(x\)\\right\)\\operatorname\{sign\}\\\!\\left\(h\(x\)\-f\(x\)\\right\),and select

f^SB=arg⁡minf∈ℱ​maxh∈ℱ​T^​\(f,h\),\\widehat\{f\}^\{\\mathrm\{SB\}\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}\\max\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f,h\),wheresign⁡\(0\)=0\\operatorname\{sign\}\(0\)=0\.

##### Evaluation\.

We evaluate each selected functionf^\\widehat\{f\}on the test set using mean absolute error \(MAE\) with respect to the empirical mean human scoreη¯​\(x\)\\bar\{\\eta\}\(x\):

R^test​\(f^\)=1m​∑i=1m\|f^​\(Xitest\)−η¯​\(Xitest\)\|,\\widehat\{R\}\_\{\\mathrm\{test\}\}\(\\widehat\{f\}\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\left\|\\widehat\{f\}\(X\_\{i\}^\{\\mathrm\{test\}\}\)\-\\bar\{\\eta\}\(X\_\{i\}^\{\\mathrm\{test\}\}\)\\right\|,wherem=2015m=2015is the number of test images\.

We also report the model\-selection regret relative to the best model inℱ\\mathcal\{F\}\. Specifically, we define

Rbest=minf∈ℱ⁡R^test​\(f\),R^\{\\mathrm\{best\}\}=\\min\_\{f\\in\\mathcal\{F\}\}\\widehat\{R\}\_\{\\mathrm\{test\}\}\(f\),and the regret of a selected functionf^\\widehat\{f\}as

Regret⁡\(f^\)=R^test​\(f^\)−Rbest\.\\operatorname\{Regret\}\(\\widehat\{f\}\)=\\widehat\{R\}\_\{\\mathrm\{test\}\}\(\\widehat\{f\}\)\-R^\{\\mathrm\{best\}\}\.Here,RbestR^\{\\mathrm\{best\}\}denotes the test MAE of the best model among the 38 candidates inℱ\\mathcal\{F\}\.

##### Results\.

Figure[2](https://arxiv.org/html/2610.20903#A13.F2)\(a\) summarizes the average test MAE with standard error from function selection with log loss, squared loss, and sign betting loss across different validation sizesnn, together with the test MAE of the best model in each hypothesis class\. The results show that the test MAE decreases as the validation size increases for all three selection methods, with sign betting consistently achieving the lowest MAE and approaching the best model as validation size increases\. Similarly, the model selection regret decreases with increasingnn, with sign betting consistently attaining the lowest regret among the three methods\.

Figure 2:Comparison of average test MAE obtained under log loss, squared loss, and sign betting loss in the real\-world and LLM experiments across varying validation sizesnn\. Error bars denote the standard error, and dashed horizontal lines indicate the MAE of the best model in each function class\.

#### M\.3LLM Experiment

##### Dataset\.

We use the UltraFeedback[Cui et al\. \(2024\)](https://arxiv.org/html/2610.20903#bib.bib24)dataset, which contains prompts paired with multiple responses\. In our experiments, each prompt is associated with44responses, and we treat each prompt\-response pair as one regression sample\. Specifically, we represent an input as

whereqqdenotes the prompt andrrdenotes one of its responses\.

To obtain reference rewards, we use Llama\-3\.3\-70B\-Instruct[Grattafiori and others \(2024\)](https://arxiv.org/html/2610.20903#bib.bib25)as the teacher model\. Given a prompt\-response pairxx, the teacher assigns a reward

steacher​\(x\)∈\{0,1,2,3,4\}\.s\_\{\\mathrm\{teacher\}\}\(x\)\\in\\\{0,1,2,3,4\\\}\.We normalize the score to\[0,1\]\[0,1\]and define

Y=g⁡\(x\)=steacher​\(x\)4\.Y=g\(x\)=\\frac\{s\_\{\\mathrm\{teacher\}\}\(x\)\}\{4\}\.Thus, each regression example consists of a prompt\-response pairxxand its normalized teacher rewardYY\. We useg⁡\(x\)g\(x\)as the teacher reference reward for both model selection and evaluation\.

##### Function class\.

We construct a finite class of candidate reward models using Llama\-3\.2\-3B\-Instruct[Grattafiori and others \(2024\)](https://arxiv.org/html/2610.20903#bib.bib25)as the backbone\. We attach a five\-class scoring head that predicts

pθ​\(k∣x\),k∈\{0,1,2,3,4\}\.p\_\{\\theta\}\(k\\mid x\),\\qquad k\\in\\\{0,1,2,3,4\\\}\.
The models are trained on the training set using the teacher reward as labels\. We use seven different random seeds and retain five checkpoints along each training trajectory, resulting in7×5=357\\times 5=35reward models\. Although the models are trained as five\-class classifiers, each model outputs a prediction reward in\[0,1\]\[0,1\]using the normalized expected score

fj​\(x\)=∑k=04pj​\(k∣x\)​k4\.f\_\{j\}\(x\)=\\sum\_\{k=0\}^\{4\}p\_\{j\}\(k\\mid x\)\\frac\{k\}\{4\}\.We therefore define the finite hypothesis class as

ℱ=\{f1,…,f35\}\.\\mathcal\{F\}=\\\{f\_\{1\},\\ldots,f\_\{35\}\\\}\.All reward models are frozen before the model selection stage, and the teacher model is not included inℱ\\mathcal\{F\}\.

##### Training\.

Given the candidate setℱ\\mathcal\{F\}, we select a function by minimizing either log loss, squared loss, or sign betting loss on the validation data\. For each repetition, we samplennprompt\-response pairs from the validation set, forming

Dnval=\{\(Xi,Yi\)\}i=1n\.D\_\{n\}^\{\\mathrm\{val\}\}=\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{n\}\.
The log loss offfis

Lnlog\(f\)=−1n∑\(x,y\)∈Dnval\[ylogf~\(x\)\+\(1−y\)log\(1−f~\(x\)\)\],L\_\{n\}^\{\\log\}\(f\)=\-\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}^\{\\mathrm\{val\}\}\}\\left\[y\\log\\widetilde\{f\}\(x\)\+\(1\-y\)\\log\(1\-\\widetilde\{f\}\(x\)\)\\right\],where

f~​\(x\)=clip⁡\(f⁡\(x\),10−6,1−10−6\),\\widetilde\{f\}\(x\)=\\operatorname\{clip\}\\\!\\left\(f\(x\),10^\{\-6\},1\-10^\{\-6\}\\right\),and the selected function is

f^log=arg⁡minf∈ℱ​Lnlog​\(f\)\.\\widehat\{f\}^\{\\log\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}L\_\{n\}^\{\\log\}\(f\)\.
Similarly, the squared loss offfis

Lnsq​\(f\)=1n​∑\(x,y\)∈Dnval\(y−f⁡\(x\)\)2,L\_\{n\}^\{\\mathrm\{sq\}\}\(f\)=\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}^\{\\mathrm\{val\}\}\}\\left\(y\-f\(x\)\\right\)^\{2\},and the selected function is

f^sq=arg⁡minf∈ℱ​Lnsq​\(f\)\.\\widehat\{f\}^\{\\mathrm\{sq\}\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}L\_\{n\}^\{\\mathrm\{sq\}\}\(f\)\.
For sign betting, we use

T^​\(f,h\)=1n​∑\(x,y\)∈Dnval\(y−f⁡\(x\)\)​sign⁡\(h⁡\(x\)−f⁡\(x\)\),\\widehat\{T\}\(f,h\)=\\frac\{1\}\{n\}\\sum\_\{\(x,y\)\\in D\_\{n\}^\{\\mathrm\{val\}\}\}\\left\(y\-f\(x\)\\right\)\\operatorname\{sign\}\\\!\\left\(h\(x\)\-f\(x\)\\right\),and select

f^SB=arg⁡minf∈ℱ​maxh∈ℱ​T^​\(f,h\),\\widehat\{f\}^\{\\mathrm\{SB\}\}=\\arg\\min\_\{f\\in\\mathcal\{F\}\}\\max\_\{h\\in\\mathcal\{F\}\}\\widehat\{T\}\(f,h\),wheresign⁡\(0\)=0\\operatorname\{sign\}\(0\)=0\.

##### Evaluation\.

We evaluate each selected reward modelf^\\widehat\{f\}on the test set using mean absolute error \(MAE\) with respect to the teacher reference reward:

Rtest​\(f^\)=1m​∑i=1m\|f^​\(xitest\)−yitest\|,R\_\{\\mathrm\{test\}\}\(\\widehat\{f\}\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\left\|\\widehat\{f\}\(x\_\{i\}^\{\\mathrm\{test\}\}\)\-y\_\{i\}^\{\\mathrm\{test\}\}\\right\|,where the test set containsm=4000m=4000prompt\-response pairs andyitesty\_\{i\}^\{\\mathrm\{test\}\}is the normalized reward assigned by the teacher model\. Lower MAE indicates that the selected reward model more closely matches the teacher scoring function on the test set\.

We also report the model selection regret relative to the best model in the function class\. Specifically, we define

Rbest=minf∈ℱ⁡Rtest​\(f\),R^\{\\mathrm\{best\}\}=\\min\_\{f\\in\\mathcal\{F\}\}R\_\{\\mathrm\{test\}\}\(f\),and

Regret⁡\(f^\)=Rtest​\(f^\)−Rbest\.\\operatorname\{Regret\}\(\\widehat\{f\}\)=R\_\{\\mathrm\{test\}\}\(\\widehat\{f\}\)\-R^\{\\mathrm\{best\}\}\.Here,RbestR^\{\\mathrm\{best\}\}is the test MAE of the best model among the 35 candidates inℱ\\mathcal\{F\}\.

##### Results\.

Figure[2](https://arxiv.org/html/2610.20903#A13.F2)\(b\) summarizes the average test MAE and model selection regret, with standard error, for function selection with log loss, squared loss, and sign betting loss across different validation sizesnn\. The test MAE of the best model in the function class is also shown as a reference\. Across all validation sizes, sign betting consistently achieves lower test MAE than both log loss and squared loss\. As the validation size increases, the test MAE of sign betting decreases and moves closer to the best model\. Similarly, the model selection regret of sign betting decreases with increasingnnand remains the lowest among the three methods across all validation sizes\.