Estimating Rare Events in Language Models with Proper Evaluation

# Estimating Rare Events in Language Models with Proper Evaluation
Source: [https://arxiv.org/html/2607.18454](https://arxiv.org/html/2607.18454)
Nikita Y\. Parulekarnparule2@jhu\.edu Johns Hopkins UniversityAnqi Liualiu\.cs@jhu\.edu Johns Hopkins University

###### Abstract

Quantifying the risk of rare failures in language models, such as those triggered by adversarial distribution shifts or very large\-scale deployments, requires estimating probabilities far too small for random sampling\. While recent work has formalized Low Probability Estimation, existing pipelines remain fragile in the rarest regimes: estimators can suffer zero\-estimate collapse or systematic bias, and standard evaluation losses can become unstable or poorly matched to asymmetric safety costs\. In this work, we introduce Gradient Activation Adaptive Multi\-Level Splitting \(GA\-AMLS\), which adapts rare\-event Monte Carlo methods to the continuous activation space of language models\. Specifically, GA\-AMLS uses a gradient\-based MCMC kernel to navigate activation space, eliminating the zero\-estimate collapse of input\-space search and replacing the independence assumptions of prior activation\-space estimators with conditional sampling under an explicit, heavier\-tailed activation prior\. We also propose the Shifted\-Power Bregman \(SPB\) Loss, a proper scoring rule that remains finite for zero\-estimates and offers tunable asymmetry between underestimation and overestimation penalties\. Experiments on small transformer models reveal a bias\-variance tradeoff: GA\-AMLS achieves the lowest loss under symmetric evaluation, reducing average log\-space squared error relative to the strongest baseline across model sizes, while methods with overestimation bias prevail under asymmetric penalties\. Our findings highlight that estimator choice should be matched to deployment context\. More broadly, our work establishes activation space as a tractable domain for rare\-event estimation in language models, circumventing the brittleness of discrete input\-space search\.

## 1Introduction

Modern large language models \(LLMs\) are trained to generalize to new inputs by minimizing expected loss during training\. However, this averaging objective leaves them susceptible to producing highly undesirable outputs on a small subset of rare inputs\. While these events may appear statistically negligible during training, the rarity of a failure is often relative\. In deployment, distribution shifts, whether manufactured through adversarial jailbreaks or incidental due to goal misgeneralization\(Shahet al\.,[2022](https://arxiv.org/html/2607.18454#bib.bib31)\), can make these catastrophic failures significantly more common\. Furthermore, in safety\-critical applications such as autonomous vehicles or medical infrastructure, even a single failure can be unacceptable\. Additionally, the large\-scale use of language models can make low\-probability events eventual certainties\. Therefore, there is a growing need for accurate low probability estimation \(LPE\) methods: the ability to quantify the likelihood of a model producing a specific harmful output, even when that probability is extremely small, such as between10−510^\{\-5\}and10−910^\{\-9\}, ranges where naive sampling is computationally infeasible\. Following the problem setting introduced byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\), we study LPE in the context of argmax sampling, where our goal is to estimate the probability that a specific target token will have the largest output logit within a computationally constrained budget as shown in Figure[1\(a\)](https://arxiv.org/html/2607.18454#S1.F1.sf1)\.

The status quo established byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)comprises two classes of methods\. Input\-space Importance Sampling searches the discrete token space; for the rarest targets it often fails to find any triggering input, collapsing estimates to zero, and it additionally requires expensive full backward passes and knowledge of exact input\-sequence probabilities\. The activation\-space alternative, Quadratic Logit Decomposition \(QLD\), avoids zero estimates by recombining whitened activation components as if they were independent, producingn2n^\{2\}synthetic candidates fromnnempirical activations\. But whitening guarantees only uncorrelatedness, not independence, and language\-model activations carry heavy\-tailed, outlier\-dominated structure\(Sunet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib20); Dettmerset al\.,[2022](https://arxiv.org/html/2607.18454#bib.bib26)\)that can survive whitening; empirically, the resulting distortion surfaces mainly as overestimation on rare events\. Finally, the metrics used to compare these methods, Itakura\-Saito \(IS\) loss and squared error loss in log space, are undefined at zero estimates, whichWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)patch with an affine calibration that itself requires known ground\-truth probabilities\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/x1.png)\(a\)Low\-probability estimation setup\.
![Refer to caption](https://arxiv.org/html/2607.18454v1/1b.png)\(b\)GA\-AMLS and QLD estimates on the colon distribution and 1\-layer model\. Each point represents a different target token\.

Figure 1:Overview of the low\-probability estimation setup and GA\-AMLS estimates\.In this paper, we address both the algorithmic and evaluative limitations of current LPE approaches through two coupled contributions: an activation\-space estimation algorithm, and a numerically stable evaluation metric that makes comparing all estimators, including those whose estimates collapse to zero, well\-posed\.

First, we proposeGradient Activation Adaptive Multi\-Level Splitting\(GA\-AMLS\), which adapts the Adaptive Multi\-Level Splitting \(AMLS\) algorithm\(Guyaderet al\.,[2011](https://arxiv.org/html/2607.18454#bib.bib33)\)to operate entirely within the continuous activation space of language models\. WhileWebbet al\.\([2019](https://arxiv.org/html/2607.18454#bib.bib122)\)previously applied AMLS to computer vision models in the input space, ours is, to our knowledge, the first method to apply it in the activation space of language models\. Figure[2](https://arxiv.org/html/2607.18454#S1.F2)gives a schematic overview of GA\-AMLS, while Figure[1\(b\)](https://arxiv.org/html/2607.18454#S1.F1.sf2)previews its comparison with the strongest activation\-space baseline, QLD, on a representative distribution\. By working in activation space, we can estimate tail probabilities without the expensive full backward passes required by Importance Sampling\. We also avoid needing to compute the input sequence probability under the original distribution, which Importance Sampling methods require for importance weights\. Instead, we decompose the search for activations that trigger the target token, into a sequence of intermediate levels, using a gradient\-based Metropolis\-adjusted Langevin algorithm \(MALA\) kernel\(Roberts and Tweedie,[1996](https://arxiv.org/html/2607.18454#bib.bib36); Srinivasanet al\.,[2025](https://arxiv.org/html/2607.18454#bib.bib138)\)to guide the sampler within each level\. The MALA kernel’s drift term allows the evolved activations to stay “close” to the typical activations of the input distribution, while the adaptive levels progressively steer towards the failure region\. This replaces QLD’s independence\-based recombination with conditional sampling under an explicit prior\. Idealized AMLS is an unbiased estimator of the tail probability of its sampling distribution\(Cérouet al\.,[2019](https://arxiv.org/html/2607.18454#bib.bib125)\); because our MCMC kernel targets a fitted activation prior rather than the true activation distribution, GA\-AMLS inherits this guarantee only up to prior mismatch \(Section[7](https://arxiv.org/html/2607.18454#S7)\)\.

Second, we introduce*Shifted Power Bregman*\(SPB\) Loss, a family of proper scoring\-rule divergences designed for evaluating rare\-probability estimates\. SPB remains finite when an estimator returns exactly zero, supports tunable asymmetry between underestimation and overestimation through an interpretable asymmetry parameter, and approximately preserves scale invariance across several orders of magnitude of the ground\-truth probability\. The unshifted power family also recovers standard losses as special cases, including squared error and the Itakura–Saito loss used byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\(Appendix[B\.8](https://arxiv.org/html/2607.18454#A2.SS8)\)\. Thus SPB Loss is a strict generalization of existing evaluation losses, with an explicit finiteness correction and a tunable asymmetry knob\. A comparison with standard losses is presented in Table[1](https://arxiv.org/html/2607.18454#S1.T1)\.

Empirically, the estimator ranking depends on the evaluation cost model\. Under approximately symmetric penalties, including log\-space squared error and corresponding SPB settings, GA\-AMLS has lower bias and outperforms the baselines\. Under strongly asymmetric penalties that heavily punish underestimation, including Itakura–Saito loss and corresponding SPB settings, QLD can outperform GA\-AMLS because its positive bias acts as a conservative hedge against false negatives\. This rank inversion is visible under pre\-existing metrics as well as under SPB; SPB makes the tradeoff explicit while avoiding the numerical instability caused by zero estimates\. Consequently, the appropriate estimator depends on the deployment context: conservative catastrophic\-risk audits may prefer overestimation, whereas applications sensitive to excessive false alarms may prefer the lower\-bias GA\-AMLS diagnostic\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/Fig2._lessblur.png)Figure 2:GA\-AMLS in activation space\.The rare\-event probability is decomposed into a product of conditional probabilities over adaptive score levels\. At each level, activations below the threshold are discarded and survivors are resampled\. MALA rejuvenation uses the fitted activation\-prior gradient to improve mixing within the constrained level\.Table 1:SPB Loss is the only loss satisfying all four desiderata for LPE evaluation: scale invariance, finiteness at zero estimates, tunable asymmetry between underestimation and overestimation penalties viaα\\alpha, and strict propriety\.
## 2Background and Problem Setup

### 2\.1Problem Setup

Low\-probability estimation\.We adopt the low\-probability estimation \(LPE\) setting ofWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\. LetMMbe a language model with vocabulary𝒱\\mathcal\{V\}, and letx∼𝒟x\\sim\\mathcal\{D\}be an input sequence drawn from a specified input distribution\. We consider deterministic argmax decoding:M​\(x\)M\(x\)is the token with maximal output logit\. For a target tokent∈𝒱t\\in\\mathcal\{V\}, the rare\-event probability of interest isqt=Prx∼𝒟⁡\[M​\(x\)=t\]\.q\_\{t\}=\\Pr\_\{x\\sim\\mathcal\{D\}\}\[M\(x\)=t\]\.In our experiments,qtq\_\{t\}lies in the extreme\-tail regime10−9≤qt≤10−510^\{\-9\}\\leq q\_\{t\}\\leq 10^\{\-5\}, where naive Monte Carlo sampling is infeasible\.

Activation\-space formulation\.GA\-AMLS operates on internal activations rather than discrete text\. Letf:𝒱∗→ℝdf:\\mathcal\{V\}^\{\*\}\\to\\mathbb\{R\}^\{d\}map an input sequence to its final\-layer pre\-LayerNorm, pre\-unembedding activationa=f​\(x\)a=f\(x\)\. Given unembedding matrixWUW\_\{U\}, the logits arez​\(a\)=LayerNorm​\(a\)​WU\.z\(a\)=\\mathrm\{LayerNorm\}\(a\)W\_\{U\}\.The eventM​\(x\)=tM\(x\)=tcan therefore be written as an activation\-space constraint:qt=Prx∼𝒟⁡\[zt​\(f​\(x\)\)\>maxi≠t⁡zi​\(f​\(x\)\)\]\.q\_\{t\}=\\Pr\_\{x\\sim\\mathcal\{D\}\}\\left\[z\_\{t\}\(f\(x\)\)\>\\max\_\{i\\neq t\}z\_\{i\}\(f\(x\)\)\\right\]\.

Existing estimators\.We compare against the methods introduced byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\. Input\-space importance sampling methods such as ITGIS and MHIS search over discrete token sequences and reweight samples to estimateqtq\_\{t\}\. These estimators can be unbiased in principle, but empirically in the extreme\-tail regime they often fail to find triggering inputs, producing zero estimates\. QLD avoids this failure mode by operating in activation space\. It whitens pre\-unembedding activations, decomposes each activation asu=a\+bu=a\+binto a target\-direction component and an orthogonal residual, and estimatesqtq\_\{t\}by recombining componentsa\(i\)\+b\(j\)a^\{\(i\)\}\+b^\{\(j\)\}\. This convertsnnactivations inton2n^\{2\}candidate activations, but relies on treating the whitened components as independent\. Since whitening guarantees uncorrelatedness rather than independence except under stronger assumptions such as joint Gaussianity, non\-Gaussian activation structure can introduce bias\.

### 2\.2Sampling Methods

The Adaptive Multilevel Splitting \(AMLS\) Algorithm\.Adaptive Multi\-Level Splitting \(AMLS\) is a rare\-event Monte Carlo method that estimates a small probability by decomposing it into a product of larger conditional probabilities\(Guyaderet al\.,[2011](https://arxiv.org/html/2607.18454#bib.bib33); Cérouet al\.,[2019](https://arxiv.org/html/2607.18454#bib.bib125); Webbet al\.,[2019](https://arxiv.org/html/2607.18454#bib.bib122)\)\. LetV∼GV\\sim Gbe a random variable, lets:ℝd→ℝs:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be a score function, and let the rare event be\{s​\(V\)≥τ\}\\\{s\(V\)\\geq\\tau\\\}\. For levels−∞=L0<L1<⋯<LK=τ\-\\infty=L\_\{0\}<L\_\{1\}<\\cdots<L\_\{K\}=\\tau,Pr⁡\(s​\(V\)≥τ\)=∏k=1KPr⁡\(s​\(V\)≥Lk∣s​\(V\)≥Lk−1\)\.\\Pr\(s\(V\)\\geq\\tau\)=\\prod\_\{k=1\}^\{K\}\\Pr\(s\(V\)\\geq L\_\{k\}\\mid s\(V\)\\geq L\_\{k\-1\}\)\.

AMLS estimates these factors using a population ofNNparticles\. At each level, the thresholdLkL\_\{k\}is chosen adaptively from the empirical score distribution; particles belowLkL\_\{k\}are discarded, survivors are resampled, and the resulting duplicates are rejuvenated with an MCMC kernel targetingG\(⋅∣s\(V\)≥Lk\)G\(\\cdot\\mid s\(V\)\\geq L\_\{k\}\)\. The final estimate isq^AMLS=∏k=1Kp^k,p^k=1N​∑i=1N𝟏​\{s​\(Vi\)≥Lk\}\.\\hat\{q\}\_\{\\mathrm\{AMLS\}\}=\\prod\_\{k=1\}^\{K\}\\hat\{p\}\_\{k\},\\qquad\\hat\{p\}\_\{k\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{s\(V\_\{i\}\)\\geq L\_\{k\}\\\}\.Thus, AMLS replaces one extremely rare event with a sequence of more frequent conditional events\.

MALA Proposal Kernel\.The efficiency of AMLS depends on the MCMC kernel used to rejuvenate particles\. We use the Metropolis\-adjusted Langevin algorithm \(MALA\), which proposes moves using local gradient information\. For a target densityπ\\pi, MALA proposes\(Roberts and Tweedie,[1996](https://arxiv.org/html/2607.18454#bib.bib36); Srinivasanet al\.,[2025](https://arxiv.org/html/2607.18454#bib.bib138)\)x′=x\+h​∇xlog⁡π​\(x\)\+2​h​ξ,ξ∼𝒩​\(0,I\),x^\{\\prime\}=x\+h\\nabla\_\{x\}\\log\\pi\(x\)\+\\sqrt\{2h\}\\,\\xi,\\qquad\\xi\\sim\\mathcal\{N\}\(0,I\),and accepts or rejects the proposal using the standard Metropolis\-Hastings correction\.

## 3Gradient Activation AMLS \(GA\-AMLS\) for Estimating Rare Events

We adapt the general AMLS algorithm to work in the activation space of language models, and use a MALA kernel to guide the search \(Section[2\.2](https://arxiv.org/html/2607.18454#S2.SS2)\)\. We call this method Gradient Activation AMLS \(GA\-AMLS\), presented in Appendix[C](https://arxiv.org/html/2607.18454#A3)as Algorithm[1](https://arxiv.org/html/2607.18454#alg1)and outlined below\.

Notation\. Leta∈ℝda\\in\\mathbb\{R\}^\{d\}denote the raw pre\-LayerNorm pre\-unembedding activation, and letuudenote its whitened coordinate\. Given meanμ\\muand covarianceΣ\\Sigma, choose a de\-whitening matrixAAsuch thatA​A⊤=ΣAA^\{\\top\}=\\Sigma, and defineu=\(a−μ\)​A−⊤,a​\(u\)=u​A⊤\+μ\.u=\(a\-\\mu\)A^\{\-\\top\},a\(u\)=uA^\{\\top\}\+\\mu\.GA\-AMLS and the MALA kernel operate overuu, while model logits are evaluated by inserting the corresponding unwhitened activationa​\(u\)a\(u\)back into the model\.

Prepare the Activation Space: We use activations from the pre\-LayerNorm pre\-unembedding layer, and whiten them\. Raw transformer activationsaaare characteristically anisotropic, with a few outlier dimensions whose scale, in magnitude and in across\-distribution variance, is orders of magnitude larger than the rest\. This outlier structure is documented in large language models\(Sunet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib20); Dettmerset al\.,[2022](https://arxiv.org/html/2607.18454#bib.bib26)\); we verify empirically that it also holds at the smaller models we study in the Appendix in Figure[12](https://arxiv.org/html/2607.18454#A4.F12)\. To improve mixing of MCMC kernels in this high\-dimensional space, we operate in whitened coordinatesu=\(a−μ\)​A−⊤u=\(a\-\\mu\)A^\{\-\\top\}\. This global preconditioning makes isotropic Euclidean steps inuu\-space correspond to Mahalanobis\-scaled steps in the original activation space\. We sample pre\-LayerNorm activations so that the evolved activations do not need to satisfy the LayerNorm constraint directly\.

Define the Score function: We define the score functions\{s\}that directly describes how “close” an activation is to an activation producing the target token\. Given whitened pre\-LayerNorm pre\-unembedding activationuas input,AAas the de\-whitening matrix,μ\\muas the mean,ttas the target token index, the score function computes:

s\(u\)=\(LayerNorm\(uA⊤\+μ\)WU\)t−maxj≠t\(LayerNorm\(uA⊤\+μ\)WU\)j\.s\(\\textbf\{u\}\)=\\Bigl\(\\mathrm\{LayerNorm\}\(\\textbf\{u\}A^\{\\top\}\+\\mu\)\\,W\_\{U\}\\Bigr\)\_\{t\}\-\\max\_\{j\\neq t\}\\Bigl\(\\mathrm\{LayerNorm\}\(\\textbf\{u\}A^\{\\top\}\+\\mu\)\\,W\_\{U\}\\Bigr\)\_\{j\}\.
That is,s​\(u\)s\(\\textbf\{u\}\)is the margin by which the target token’s logit exceeds the maximum logit among all other tokens\. Using the general AMLS algorithm from Section[2\.2](https://arxiv.org/html/2607.18454#S2.SS2), we define the rare event asP​\(s​\(u\)≥τ\)P\\bigl\(s\(\\textbf\{u\}\)\\geq\\tau\\bigr\)\. In our settingτ=0\\tau=0, since this is when the logit for the target tokenttis the largest, and thusttwill be the next token predicted under our assumption of argmax sampling\.

Determine the prior distribution:The priorπ\\piis defined over whitened activationsuuand is used to guide the MCMC kernel\. In whitened space, we use a diagonal product Student\-ttprior\. This captures heavier\-than\-Gaussian marginal tails\. This choice is supported by the tuning ablations in Appendix[D\.1](https://arxiv.org/html/2607.18454#A4.SS1), where the Student\-ttprior outperforms a Gaussian prior\.

Choose the MCMC kernelKK: To efficiently evolve the clones’ activations within the high\-dimensional activation space, we use the Metropolis\-adjusted Langevin algorithm \(MALA\) as outlined in Section[2\.2](https://arxiv.org/html/2607.18454#S2.SS2), with the below correction to take into account the constraint induced by the score mechanism of AMLS\.

We must strictly enforce the level constraintss​\(u\)≥Lis\(\\textbf\{u\}\)\\geq L\_\{i\}\. This is addressed through a Metropolis\-Hastings filter\.

The proposal densityq​\(u′∣u\)q\(\\textbf\{u\}^\{\\prime\}\\mid\\textbf\{u\}\)is Gaussian, while the prior distribution is Student’sttdistribution\.

The acceptance probabilityrris computed in two stages to handle the hard constraint:

1. 1\.Hard score check: Ifs​\(u′\)<Lis\(\\textbf\{u\}^\{\\prime\}\)<L\_\{i\}, the proposal is invalid and immediately rejected \(r=0r=0\);
2. 2\.Ratio Computation: If valid, we compute the standard Metropolis Hastings \(MH\) acceptance ratiorr: r=min⁡\(1,π​\(u′\)​q​\(u∣u′\)π​\(u\)​q​\(u′∣u\)\)\.r=\\min\\left\(1,\\frac\{\\pi\(\\textbf\{u\}^\{\\prime\}\)\\,q\(\\textbf\{u\}\\mid\\textbf\{u\}^\{\\prime\}\)\}\{\\pi\(\\textbf\{u\}\)\\,q\(\\textbf\{u\}^\{\\prime\}\\mid\\textbf\{u\}\)\}\\right\)\.

We accept the proposalu′\\textbf\{u\}^\{\\prime\}with probabilityrr; otherwise, the activation remains atu\.

Standard Random Walk Metropolis\-Hastings \(RWMH\) proposes isotropic steps blind to the geometry of the distribution, leading to slow mixing in this high dimensional activation space\. MALA improves upon RWMH by using gradient information from the underlying prior distribution to guide the proposal towards the activation space’s typical set for the specific input domain𝒟\\mathcal\{D\}\. In GA\-AMLS this MALA kernel is applied to the restricted targetπk​\(u\)∝π​\(u\)​1​\{s​\(u\)≥Lk\},\\pi\_\{k\}\(u\)\\propto\\pi\(u\)\\,\\mathbf\{1\}\\\{s\(u\)\\geq L\_\{k\}\\\},so proposals below the current levelLkL\_\{k\}are rejected by the MH filter\. Thus the Langevin drift improves exploration within the current constrained level set, while the adaptive thresholdsLkL\_\{k\}move the population toward the rare target\-token regions\. This choice is also supported by the tuning ablations in Appendix[D\.1](https://arxiv.org/html/2607.18454#A4.SS1), where MALA outperforms RWMH despite using half as many MH proposal steps\.

To ensure optimal mixing, we adapt the step sizehh\(from MALA under[2\.2](https://arxiv.org/html/2607.18454#S2.SS2)\) during a burn\-in period ofTburnT\_\{\\text\{burn\}\}steps\. We employ a multiplicative update rule to tunehhtoward a target acceptance rate ofr∗=0\.57r^\{\*\}=0\.57, which is theoretically optimal for MALA in high dimensions\(Roberts and Rosenthal,[1998](https://arxiv.org/html/2607.18454#bib.bib133)\):ht\+1=ht⋅exp⁡\(ω​\(r¯t−r∗\)\)h\_\{t\+1\}=h\_\{t\}\\cdot\\exp\\left\(\\omega\(\\bar\{r\}\_\{t\}\-r^\{\*\}\)\\right\), whereω=0\.1\\omega=0\.1is the adaptation rate andr¯t\\bar\{r\}\_\{t\}is the empirical acceptance rate at steptt\. AfterTburnT\_\{\\text\{burn\}\}steps,hhis frozen to ensure the Markov chain satisfies detailed balance condition during the sampling\.

## 4Shifted Power Bregman \(SPB\) Loss for Evaluation

We introduce Shifted\-Power Bregman \(SPB\) Loss, a proper scoring rule for evaluating rare\-probability estimates\. SPB remains finite when an estimator returns zero and provides tunable asymmetry between underestimation and overestimation\. We define both the pointwise loss and a dataset\-level aggregation that controls scale dependence across rare\-event probabilities\.

Desiderata for an ideal loss for LPE\. To evaluate an estimator for LPE, an ideal loss would satisfy the following requirements:

Scale Invariance:As mentioned inWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\), it should penalize relative errors similarly across the scale of ground truth probabilities, as we care about performance across a wide range of rarities \(e\.g\.,10−510^\{\-5\}to10−910^\{\-9\}\)\.

Asymmetric Sensitivity:It should support tunable asymmetry between underestimation and overestimation penalties, as different deployment contexts may prioritize avoiding underestimation over overestimation, or vice versa\.

Numerical Stability:It must be defined when the estimate is exactly zero, avoiding the need for calibration steps that require ground truth knowledge\.

Statistical Rigor:It should be the divergence of a proper scoring rule, so that an estimator minimizing expected loss is incentivized to report its unbiased best estimate rather than a systematically distorted one\(Gneiting and Raftery,[2007](https://arxiv.org/html/2607.18454#bib.bib146)\)\.

The Itakura–Saito \(IS\) loss used inWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)satisfies most criteria but fails on numerical stability: it is undefined \(infinite\) when the estimate is0\.Wu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)overcome this by fitting an affine transformationx↦a​xc\+bx\\mapsto ax^\{c\}\+bto the outputs, where parameters are chosen to minimize IS loss via leave\-one\-out\-cross\-validation \(LOOCV\) for each method, input, distribution pair\. A significant drawback is that this transformation relies on knowing the ground truth probabilities to calculate the IS loss which is to be minimized during this calibration\. In a realistic deployment setting, establishing this ground truth for rare tokens is computationally infeasible using naive sampling, as estimating these probabilities is the very problem we are trying to solve\. SPB uses ground truth only as the reference value in the final score, as any evaluation metric does\. By contrast, the affine calibration used in prior evaluation procedures assumes that ground\-truth probabilities are available to*transform each estimator’s outputs*before scoring\. This is a stronger requirement than ordinary evaluation against ground truth, since such calibration information would generally not be available in deployment\.

Pointwise SPB Loss\. We propose pointwise Shifted\-Power Bregman \(SPB\) Loss, derived from the framework of proper scoring rules\. It overcomes this limitation by accepting a trade\-off: if the estimate is0, the loss will not be infinite, but rather capped by a large finite constant\. We also introduce parametersα\\alpha: to control the degree of asymmetric sensitivity, andγ\\gamma: a dataset level rarity premium\.

FollowingBujaet al\.\([2005](https://arxiv.org/html/2607.18454#bib.bib5)\), every smooth proper scoring rule for binary probability estimation induces a Bregman divergence determined by a nonnegative weight functionω​\(t\)≥0\\omega\(t\)\\geq 0on\(0,1\)\(0,1\)\. Ifqqdenotes the true probability andppthe forecast, the associated divergence can be written asB​\(q∣p\)=∫qp\(t−q\)​ω​\(t\)​𝑑t\.B\(q\\mid p\)=\\int\_\{q\}^\{p\}\(t\-q\)\\omega\(t\)\\,dt\.

We define the shifted\-power weight function as:ωε​\(t\)=\(t\+ε\)−α\\omega\_\{\\varepsilon\}\(t\)=\(t\+\\varepsilon\)^\{\-\\alpha\}, whereα\>0,ε\>0\\alpha\>0,\\varepsilon\>0\. The shift parameterε\\varepsilonensures the penalty remains finite atp=0p=0\. Geometrically, this integral accumulates cost along the one\-dimensional probability axis between the truthqqand the predictionpp\. At an intermediate probability leveltt, the local contribution is the distance from the truth, scaled by the weightωε​\(t\)=\(t\+ε\)−α\\omega\_\{\\varepsilon\}\(t\)=\(t\+\\varepsilon\)^\{\-\\alpha\}\. Since this weight is largest near zero, errors that push the prediction toward smaller probabilities are penalized more strongly\. For an overestimatep=m​qp=mq, the loss integrates overt∈\[q,m​q\]t\\in\[q,mq\], where the weight decreases asttincreases\. For a reciprocal underestimatep=q/mp=q/m, the loss integrates overt∈\[q/m,q\]t\\in\[q/m,q\], which includes smaller probabilities whereωε​\(t\)\\omega\_\{\\varepsilon\}\(t\)is larger\. Increasingα\\alphasteepens this growth near zero and therefore increases the relative penalty for underestimation compared with overestimation\. Thus,α\\alphaprovides a tunable asymmetry parameter, as formalized in Theorem[2](https://arxiv.org/html/2607.18454#Thmtheorem2)\.

Dataset\-Level SPB Loss and the Rarity Premiumγ\\gamma\.To evaluate an estimator across a diverse set of rare events, we aggregate the pointwise divergences into a single dataset\-level loss\. As derived in Appendix[A](https://arxiv.org/html/2607.18454#A1), the*unshifted*pointwise divergence \(ε=0\\varepsilon=0\) scales exactly proportionally toq2−αq^\{2\-\\alpha\}for a fixed relative errorm=p/qm=p/q\. While the shift parameterε\>0\\varepsilon\>0slightly perturbs this geometry, the shifted divergenceBε​\(q∣p\)B\_\{\\varepsilon\}\(q\\mid p\)remains dominated by this leading\-orderq2−αq^\{2\-\\alpha\}scaling\. Left uncorrected, an aggregate loss would be dominated by the absolute scale of the ground\-truth probabilities rather than the relative accuracy of the estimators\.

To neutralize this and establish controlled scale invariance, we define the dataset\-level SPB Loss as the arithmetic mean111Crucially, we divide by the sample sizennrather than the sum of the weights∑iw​\(qi\)\\sum\_\{i\}w\(q\_\{i\}\)\. Since the weightw​\(q\)w\(q\)is specifically designed to perform a pointwise cancellation of theq2−αq^\{2\-\\alpha\}scale factor, applying a normalized weighted average would reintroduce a dependence on the empirical distribution of the ground truth probabilitiesqq, undoing the cancellation\.of the*weighted*pointwise divergences acrossnnsamples:ℒSPB=1n​∑i=1nw​\(qi\)​Bε​\(qi∣pi\)\.\\mathcal\{L\}\_\{\\text\{SPB\}\}\\;=\\;\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}w\(q\_\{i\}\)\\,B\_\{\\varepsilon\}\(q\_\{i\}\\mid p\_\{i\}\)\.We define the weight function as:w​\(q\)=qα−2−γw\(q\)\\;=\\;q^\{\\,\\alpha\-2\-\\gamma\}, whereγ≥0\\gamma\\geq 0\. The parameterγ\\gammaallows us to toggle between two distinct evaluation modes:

Scale Neutrality \(γ=0\\gamma=0\):By settingγ=0\\gamma=0, the weight becomesqα−2q^\{\\alpha\-2\}\. This term cancels the leadingq2−αq^\{2\-\\alpha\}scaling of the divergence\. Consequently, the final metric approximates scale invariance, depending almost entirely on the relative multiplicative error \(m=p/qm=p/q\) rather than the absolute rarity of the event\. The residual distortion due toε\\varepsilonis formalized and bounded in the Appendix in Theorem[1](https://arxiv.org/html/2607.18454#Thmtheorem1)\.

The Rarity Premium \(γ\>0\\gamma\>0\):In specific safety\-critical deployments, evaluating performance symmetrically across scales is less important than severely punishing errors on the deepest tail events\. Settingγ\>0\\gamma\>0intentionally breaks scale neutrality by introducing an extraq−γq^\{\-\\gamma\}factor\. This acts as a tunable*rarity premium*, disproportionately upweighting the smallest\-qqevents in the aggregate score\.

##### Relation to other Losses

The unshifted power family \(ε=0\\varepsilon=0\) unifies standard losses:α=0\\alpha=0recovers squared error andα=2\\alpha=2is*exactly*the Itakura–Saito loss \(Appendix[B\.8](https://arxiv.org/html/2607.18454#A2.SS8)\)\. Therefore SPB comes from a strict generalization of the metric used byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\), adding finiteness at zero estimates \(withε\>0\\varepsilon\>0\) and a tunable asymmetry knob\.

### 4\.1Practical parameter selection \(α\\alpha,γ\\gamma,ε\\varepsilon\)

The metric has three user\-facing parameters: 1\)α\\alphasets the FN\-vs\-FP asymmetry:α\>1\.5\\alpha\>1\.5penalizes underestimation more than overestimation andα<1\.5\\alpha<1\.5the reverse \(Theorem[2](https://arxiv.org/html/2607.18454#Thmtheorem2), Eq\.[20](https://arxiv.org/html/2607.18454#A2.E20)in Appendix; fine\-grained control ofα\\alphais described in Appendix[B\.6\.1](https://arxiv.org/html/2607.18454#A2.SS6.SSS1)\)\. 2\)γ≥0\\gamma\\geq 0is an optional rarity premium:γ=0\\gamma=0gives scale invariance up to a toleranceη\\eta\(Theorem[1](https://arxiv.org/html/2607.18454#Thmtheorem1)in Appendix\), whileγ\>0\\gamma\>0multiplies each example’s contribution to the aggregate loss byq−γq^\{\-\\gamma\}, so at a fixed relative error a10×10\\timesrarer event carries10γ×10^\{\\gamma\}\\timesthe weight\. 3\)ε\\varepsilonis the floor that makesp=0p=0finite at the cost of controlled distortion; a heuristic default isε=0\.01​qmin/mmax\\varepsilon=0\.01\\,q\_\{\\min\}/m\_\{\\max\}, and a guaranteed choice is any fraction of the bound given in Appendix by Corollary[1](https://arxiv.org/html/2607.18454#Thmcorollary1)\(interpretation in Appendix[B\.5](https://arxiv.org/html/2607.18454#A2.SS5)\)\. Table[2](https://arxiv.org/html/2607.18454#A2.T2)\(Appendix[B\.6](https://arxiv.org/html/2607.18454#A2.SS6)\) suggests settings by evaluation use case\.

## 5Experimental Results

This section empirically evaluates both components of our proposal\. Section[5\.1](https://arxiv.org/html/2607.18454#S5.SS1)tests GA\-AMLS on the real\-data LPE benchmark ofWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\), comparing it against input\-space Importance Sampling methods and QLD using both point\-estimate plots and aggregate SPB Loss\. Section[5\.2](https://arxiv.org/html/2607.18454#S5.SS2)uses synthetic experiments to validate the behavior predicted by the SPB Loss theory, including approximate scale invariance, bounded distortion from theε\\varepsilon\-shift, and tunable asymmetry between underestimation and overestimation\.

### 5\.1GA\-AMLS: Real Data Experiments

Dataset:We use 8 distributions and the ground\-truth probabilities as used inWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\. They generated the ground\-truth probabilities for each of the 8 distributions by running forward passes on2322^\{32\}random samples\. We select a random set of 256 tokens among those with ground\-truth probabilities between10−910^\{\-9\}and10−510^\{\-5\}, and test the methods on these tokens\.

Methodology:We follow the experimental setup ofWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\. To prevent overfitting, the method was only run on the first four distributions during development, and finalized before testing on the last four distributions\. We test the methods on 3 models: a 1\-layer, a 2\-layer, and a 4\-layer transformer fromNanda and Bloom \([2022](https://arxiv.org/html/2607.18454#bib.bib3)\)\. All models have a hidden dimension ofd=512d=512, a vocabulary size of\|V\|=48262\|V\|=48262, GELU non\-linearitiesHendrycks and Gimpel \([2023](https://arxiv.org/html/2607.18454#bib.bib15)\), and were trained on the C4 datasetRaffelet al\.\([2023](https://arxiv.org/html/2607.18454#bib.bib35)\)and CodeParrotTunstallet al\.\([2022](https://arxiv.org/html/2607.18454#bib.bib34)\)\. We use the Shifted\-Power Bregman \(SPB\) Loss as our evaluation loss\. By varying the asymmetry parameterα\\alphaand the dataset\-level rarity premiumγ\\gamma, we demonstrate how different penalization profiles \(detailed in Appendix in Table[2](https://arxiv.org/html/2607.18454#A2.T2)\) alter the apparent ranking of the methods\. For hyperparameter tuning details please refer to Appendix[D\.1](https://arxiv.org/html/2607.18454#A4.SS1)\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/multi_metric.png)Figure 3:Multi\-metric evaluation:Plotted side\-by\-side, we observe a rank inversion depending on the metric’s sensitivity to underestimation\. Under symmetric penalties \(SPBα=1\.5\\alpha=1\.5and Squared Error in log space\), GA\-AMLS’s lower bias leads to outperformance\. Under severe asymmetric penalties \(SPBα=2\\alpha=2and IS Loss\), QLD’s strategy of systematic overestimation guards it against false\-negative penalties, resulting in lower loss\. SPB Loss safely evaluates input\-space methods that collapse to zero, overcoming the numerical instability of uncalibrated IS loss and squared error loss in log space\. \(All plotted with log transformed y axis\.\)Baselines:We compare against QLD, ITGIS and MHIS as described in Section[2\.1](https://arxiv.org/html/2607.18454#S2.SS1)\.

Performance under Symmetric Penalties:[Figure 3](https://arxiv.org/html/2607.18454#S5.F3)compares the overall evaluation losses of all methods across different model sizes grouped by symmetric and asymmetric metrics, while Figure[16](https://arxiv.org/html/2607.18454#A6.F16)\(Appendix[F](https://arxiv.org/html/2607.18454#A6)\) details the SPB loss for each of the 8 individual data distributions\. Under evaluation metrics that penalize overestimation and underestimation symmetrically, such as log\-space Squared Error and SPB Loss withα=1\.5\\alpha=1\.5, we observe that GA\-AMLS significantly outperforms all baseline methods \([Figure 3](https://arxiv.org/html/2607.18454#S5.F3), first two panels\)\. This outperformance is consistent across 7 of the 8 individual distributions \([Figure 16](https://arxiv.org/html/2607.18454#A6.F16)\)\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/t.png)Figure 4:Examples of probability estimate vs ground truth on two input distributions and models\.Input Space Methods \(ITGIS/MHIS\) frequently fail to find valid triggers, causing their estimates to collapse to zero \(placed at the bottom of each graph for visibility\)\. While QLD shows a systematic overestimation bias, GA\-AMLS centers closest to the ground truth, at the cost of higher variance\.Performance under Asymmetric Penalties:Focusing on the latter half of[Figure 3](https://arxiv.org/html/2607.18454#S5.F3), the figure illustrates the methods’ performance under asymmetric metrics that severely penalize underestimation \(SPB Loss withα=2\\alpha=2and IS Loss\)\. When evaluating with these metrics, we observe a clear inversion in method rankings, where QLD outperforms GA\-AMLS under these conditions\. This occurs because QLD is biasing its estimates toward overestimation\. While less accurate overall, this systematic positive bias acts as a natural safeguard against the severe false\-negative penalties imposed by asymmetric losses\.

Deep\-Tail Overestimation and Rarity Premiumγ\\gamma:[Figure 4](https://arxiv.org/html/2607.18454#S5.F4)plots the methods’ individual point estimates against the true probabilities to visualize estimator bias, and Figure[17](https://arxiv.org/html/2607.18454#A6.F17)\(Appendix[F](https://arxiv.org/html/2607.18454#A6)\) shows how the aggregate SPB loss shifts across methods as the rarity premiumγ\\gammaincreases\. The scatter plots \([Figure 4](https://arxiv.org/html/2607.18454#S5.F4)and Appendix[E](https://arxiv.org/html/2607.18454#A5)\) reveal that QLD’s overestimation bias is not uniform; it becomes significantly more pronounced at the deepest tail \(the rarest events\)\. Consequently, as seen in[Figure 17](https://arxiv.org/html/2607.18454#A6.F17), as we increase the dataset\-level rarity premium \(γ\>0\\gamma\>0\) to disproportionately upweight the rarest events, the performance gap between QLD and GA\-AMLS narrows, demonstrating the cost of QLD’s deep\-tail overbias\.

The Zero\-Estimate Collapse and Bias\-Variance Tradeoff:The scatter plots of method estimates \([Figure 4](https://arxiv.org/html/2607.18454#S5.F4)and Appendix[E](https://arxiv.org/html/2607.18454#A5)\) also directly capture the distribution and structural failure modes of each estimator’s predictions\. The distinct performance profiles of these methods stem from a fundamental bias\-variance tradeoff, visually observed in these plots\. The input\-space Importance Sampling methods \(MHIS/ITGIS\) suffer from frequently collapsing to zero for the rarest events\. QLD avoids zero\-estimates but shifts the entire distribution of predictions upward\. GA\-AMLS shows lower bias and is more centred around the ground truth line, but exhibits higher variance across tokens than QLD \(as seen visually in scatterplots in Appendix[E](https://arxiv.org/html/2607.18454#A5)\)\.

### 5\.2SPB Loss: Synthetic Data Experiments

We now isolate the behavior of SPB Loss itself, independently of any estimator\. The goal of these synthetic experiments is to illustrate the loss geometry discussed in Section[4\.1](https://arxiv.org/html/2607.18454#S4.SS1)and formalized in Appendix[B](https://arxiv.org/html/2607.18454#A2): approximate scale invariance, controlled distortion from the finite shiftε\\varepsilon, and tunable asymmetry between underestimation and overestimation\.

Setup\.We evaluate rare probabilitiesq∈\[10−9,10−5\]q\\in\[10^\{\-9\},10^\{\-5\}\]on a log\-spaced grid and we perturb each probability by fixed multiplicative factors\. For scale\-invariance experiments, predictions range from0\.1​q0\.1qto10​q10q\. For asymmetry experiments, we compare paired reciprocal errors: an overestimate/False\-Positive \(FP\)p=m​qp=mqand an underestimate/False\-Negative \(FN\)p=q/mp=q/m, withmmranging up to10001000\. We use the scale\-neutral settingγ=0\\gamma=0\.

For finite\-shiftε\>0\\varepsilon\>0plots, we compare two choices ofε\\varepsilon: one chosen according to the sufficient bound in Corollary[1](https://arxiv.org/html/2607.18454#Thmcorollary1), and a simple heuristic choiceε=0\.01​\(qmin/mmax\)\\varepsilon=0\.01\(q\_\{\\min\}/m\_\{\\max\}\)\. The tolerance for the scale\-distortion plots isη=0\.05\\eta=0\.05\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/paper_metrics_exp_unshifted_scale_invariance.png)\(a\)Unshifted scale invariance\.
![Refer to caption](https://arxiv.org/html/2607.18454v1/paper_metrics_exp_shift_distortion_fp.png)\(b\)FP distortion\.
![Refer to caption](https://arxiv.org/html/2607.18454v1/paper_metrics_exp_shift_distortion_fn.png)\(c\)FN distortion\.

Figure 5:Scale invariance and finite\-shift distortion of SPB Loss\.\(a\)Withε=0\\varepsilon=0, the weighted loss is flat acrossqqfor each fixed multiplicative error, showing exact scale neutrality\.\(b\)–\(c\)Introducingε\>0\\varepsilon\>0makes the loss finite at zero estimates but slightly perturbs the unshifted geometry\. The observed distortion remains below the target toleranceη=0\.05\\eta=0\.05for both overestimation \(FP\) and underestimation \(FN\)\.
![Refer to caption](https://arxiv.org/html/2607.18454v1/assymetry_corr_corrected.png)\(a\)Corollaryε\\varepsilon\.
![Refer to caption](https://arxiv.org/html/2607.18454v1/paper_metrics_exp_asymmetry_heuristic.png)\(b\)Heuristicε\\varepsilon\.
![Refer to caption](https://arxiv.org/html/2607.18454v1/paper_metrics_exp_alpha_bar_chart.png)\(c\)Comparison with standard losses\.

Figure 6:Asymmetry control viaα\\alpha\.\(a\)–\(b\)FN/FP loss ratios for reciprocal multiplicative errors under the Corollary\-based and heuristic choices ofε\\varepsilon\. Ratios above one indicate that underestimation is penalized more strongly than overestimation\.\(c\)Atm=10m=10, SPB Loss provides a tunable continuum between approximately symmetric behavior and strongly FN\-averse behavior, compared with log\-space squared error and IS Loss\.Scale invariance and shift distortion\.Figure[5](https://arxiv.org/html/2607.18454#S5.F5)examines whether SPB Loss treats the same relative error similarly across different rarity levels\. In Panel[5\(a\)](https://arxiv.org/html/2607.18454#S5.F5.sf1), the unshifted weighted loss is flat as a function ofqq: for a fixed multiplicative error, the loss is essentially independent of whether the true probability is10−510^\{\-5\}or10−910^\{\-9\}\. This is the desired scale\-neutral behavior\.

Panels[5\(b\)](https://arxiv.org/html/2607.18454#S5.F5.sf2)and[5\(c\)](https://arxiv.org/html/2607.18454#S5.F5.sf3)then show what changes when the finite shiftε\>0\\varepsilon\>0is introduced\. The shift is needed to keep the loss finite when an estimator outputs zero, but it slightly perturbs the ideal unshifted geometry\. The plots report this perturbation separately for overestimation and underestimation\. For both the Corollary\-based shift and the heuristic shift, the observed distortion remains below the target toleranceη=0\.05\\eta=0\.05over the displayed range\. Thus, in this regime, the finite correction preserves the practical scale\-neutral behavior of the loss\.

Asymmetry control viaα\\alpha\.Figure[6](https://arxiv.org/html/2607.18454#S5.F6)studies the relative penalty for FN versus FP\. Panels[6\(a\)](https://arxiv.org/html/2607.18454#S5.F6.sf1)and[6\(b\)](https://arxiv.org/html/2607.18454#S5.F6.sf2)plot the FN/FP loss ratio for reciprocal errors: a value above one means that underestimatingqqby a factormmis penalized more than overestimatingqqby the same factor\. The curves show thatα\\alphacontrols this asymmetry\. Values above the thresholdα=1\.5\\alpha=1\.5, such asα=1\.6\\alpha=1\.6andα=2\.5\\alpha=2\.5, produce FN/FP ratios above one across the plotted range, consistent with Theorem[2](https://arxiv.org/html/2607.18454#Thmtheorem2)\. Theα=0\.5\\alpha=0\.5curve is included as a contrasting setting outside this FN\-averse regime\.

Panel[6\(c\)](https://arxiv.org/html/2607.18454#S5.F6.sf3)compares this behavior with standard losses at a fixed multiplicative errorm=10m=10\. Log\-space squared error is approximately symmetric between underestimation and overestimation, while IS Loss is FN\-averse\. SPB Loss interpolates between these behaviors throughα\\alpha:α≈1\.5\\alpha\\approx 1\.5is close to symmetric, while largerα\\alphaincreasingly penalizes underestimation\. This gives SPB Loss an explicit asymmetry knob while retaining finiteness at zero estimates\.

## 6Related Work

Low Probability Estimation in LLMs\.Low probability estimation for language models was formalized byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\), who introduced input\-space Importance Sampling methods and the activation\-space QLD baseline used in our experiments\. We adopt their benchmark setting, but replace discrete input\-space search with gradient\-guided sampling in activation space and introduce SPB Loss to handle zero estimates and asymmetric safety costs\. In a complementary line of work,Joneset al\.\([2025](https://arxiv.org/html/2607.18454#bib.bib30)\)address rare behavior estimation from a forecasting perspective rather than direct probability estimation\. Their approach leverages elicitation probabilities defined as the probability that a given query produces a target behavior under repeated sampling, and uses extreme value theory to forecast how deployment\-scale risks grow from smaller evaluation sets\. Their method is particularly suited for anticipating when rare failures might emerge at scale, whereas our approach provides precise probability estimates for individual rare events\. The two approaches are complementary: GA\-AMLS can provide accurate point estimates of elicitation probabilities that could serve as inputs to the forecasting framework ofJoneset al\.\([2025](https://arxiv.org/html/2607.18454#bib.bib30)\), potentially improving the precision of deployment\-scale risk predictions\. Recent work byDormanet al\.\([2026](https://arxiv.org/html/2607.18454#bib.bib16)\)develops an end\-to\-end framework for rare event analysis over full text completions in the discrete token space, using MCMC\-based trajectory sampling from tilted completion distributions\. This requires repeated full\-sequence generation and mixing in sequence space, whereas GA\-AMLS operates in continuous activation space and avoids generating or scoring full completions\. Another complementary line of work estimates rare harmful behaviors in stochastic language\-model output distributions\.Angellet al\.\([2026](https://arxiv.org/html/2607.18454#bib.bib2)\)estimate, for a fixed query, the probability that repeated sampling from a target model produces a harmful completion\. They construct unsafe proposal models using activation steering and use importance sampling to reweight generated completions back to the target model, yielding substantial sample savings relative to naive Monte Carlo\. This setting differs from ours in both the source of randomness and the event definition: they estimate behavior\-level probabilities over full stochastic completions for fixed prompts, whereas we estimate single\-token argmax probabilities over random inputs under deterministic decoding\. Beyond trained transformers,Wuet al\.\([2026](https://arxiv.org/html/2607.18454#bib.bib1)\)study sample\-free mechanistic estimation for wide random MLPs with Gaussian inputs\. Their cumulant\-propagation algorithms approximate activation distributions through fixed randomly initialized networks using cumulants and Hermite expansions, and can outperform Monte Carlo sampling, including on low\-probability threshold events\. This is complementary to our approach: their guarantees and experiments apply to analytically tractable random MLPs and closed\-form input distributions, whereas we study trained transformer language models and use adaptive rare\-event sampling in activation space\.

Activation\-Space Methods\.A growing interpretability literature supports treating language\-model activations as a structured domain for analysis and intervention\. The linear representation hypothesis\(Parket al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib6)\), representation engineering\(Zouet al\.,[2023a](https://arxiv.org/html/2607.18454#bib.bib7)\), and activation steering methods\(Turneret al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib8); Panicksseryet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib9)\)show that model behaviors can often be manipulated through directions in residual\-stream space\. Recent work further suggests that such behavioral structure need not be captured by a single global direction: heterogeneity\-aware steering methods model activations as clustered distributions and use optimal transport to construct input\-dependent interventions\(Abdullaevet al\.,[2026](https://arxiv.org/html/2607.18454#bib.bib4)\)\. Related work finds low\-dimensional structure for refusal behavior\(Arditiet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib10)\), while sparse autoencoders reveal richer monosemantic feature structure in activations\(Brickenet al\.,[2023](https://arxiv.org/html/2607.18454#bib.bib11); Templetonet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib12); Gaoet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib13)\)\. The Tuned Lens\(Belroseet al\.,[2025](https://arxiv.org/html/2607.18454#bib.bib14)\)further motivates operating near the pre\-unembedding layer, where activations are predictive of output tokens\. GA\-AMLS builds on this view of activation space, but treats it as a probability space: instead of identifying causal directions, interventions, or features, we estimate the probability mass of rare\-event regions\.

Rare\-Event Sampling\.Our method adapts Adaptive Multi\-Level Splitting \(AMLS\), a rare\-event Monte Carlo technique from statistical physics and particle simulation\(Cérouet al\.,[2019](https://arxiv.org/html/2607.18454#bib.bib125)\)\.Webbet al\.\([2019](https://arxiv.org/html/2607.18454#bib.bib122)\)applied AMLS to neural\-network robustness verification in computer vision using a random\-walk Metropolis\-Hastings kernel in input space\. Directly applying this approach to language models is difficult because text is discrete\. GA\-AMLS instead moves the estimator to continuous activation space, where gradient\-based MCMC kernels such as MALA can be used efficiently\.

Adversarial Search and Defenses\.Red\-teaming and jailbreaking methods search for individual inputs that trigger undesired behavior, using methods ranging from discrete gradient\-based attacks such as GCG\(Zouet al\.,[2023b](https://arxiv.org/html/2607.18454#bib.bib19)\)to activation\-space attacks such as Subspace Rerouting\(Winningeret al\.,[2025](https://arxiv.org/html/2607.18454#bib.bib17)\)\. These methods are useful for finding failures and generating adversarial training data\(Goodfellowet al\.,[2015](https://arxiv.org/html/2607.18454#bib.bib25); Madryet al\.,[2019](https://arxiv.org/html/2607.18454#bib.bib24)\), including in latent adversarial training settings\(Casperet al\.,[2024](https://arxiv.org/html/2607.18454#bib.bib99); Sheshadriet al\.,[2025](https://arxiv.org/html/2607.18454#bib.bib128)\)\. However, finding one failure is not the same as estimating its probability, and transfer across search methods can be unreliable\(Kanget al\.,[2019](https://arxiv.org/html/2607.18454#bib.bib22); Weiet al\.,[2023](https://arxiv.org/html/2607.18454#bib.bib21)\)\. GA\-AMLS is therefore diagnostic rather than defensive: it estimates how much probability mass lies in the rare\-event region\.

## 7Discussion and Limitations

While GA\-AMLS provides a principled activation\-space estimator for rare model behaviours, several limitations remain\.

Residual bias and activation\-space approximation\.GA\-AMLS is designed as an unbiased rare\-event estimator under the assumed activation prior and exact MCMC sampling\. In practice, residual bias can arise from three sources\. First, we approximate the true activation distribution with a fitted Student’sttdistribution, which introduces distributional mismatch\. Second, finite MCMC budgets may prevent full mixing between AMLS levels, leaving residual correlations among particles\. Third, because GA\-AMLS operates in continuous activation space, extreme activations found by the sampler may not correspond to any valid discrete token sequence\. We partially mitigate this activation pre\-image issue by fitting the prior to activations induced by samples from the input distribution𝒟\\mathcal\{D\}, encouraging samples to remain near the empirical activation manifold\. However, the gap between continuous activations and discrete inputs remains a fundamental limitation of activation\-space methods, including QLD\.

Computational cost\.GA\-AMLS has a different computational profile from input\-space importance sampling\. It avoids full\-model backward passes and operates from cached activations, but it requires many sequential activation\-space scorer evaluations across AMLS levels\. GA\-AMLS is computationally heavier than QLD: QLD reduces estimation to fast closed\-form matrix operations, whereas GA\-AMLS requires sequential Markov\-chain evolution across adaptive levels\. This additional cost is the price of replacing QLD’s independence\-based recombination approximation with conditional sampling under a fitted activation prior\. It may therefore limit very large sweeps over many target tokens, layers, or models\.

Evaluation scope\.For comparability with prior low\-probability estimation work, our experiments follow the benchmark setting ofWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\): the input distribution𝒟\\mathcal\{D\}consists of independent tokens, and rare events are defined by a single target token under deterministic argmax conditions\. This controlled setting enables direct comparison with ITGIS, MHIS, and QLD, but it does not capture correlated natural prompts, instruction\-following contexts, or multi\-step autoregressive failures\. Extending activation\-space rare\-event estimation to realistic prompt distributions and trajectory\-level events is an important direction for future work\. Finally, all of our evidence comes from the11–44layer,d=512d=512transformers of this benchmark; whether the bias–variance profile of GA\-AMLS, and the adequacy of a diagonal Student\-ttprior in whitened coordinates, persist at modern scale is untested\.

## 8Conclusion

We introduced GA\-AMLS, an activation\-space adaptive multilevel splitting method for estimating rare LLM behaviors that are infeasible to measure accurately with naive sampling\. Empirically, GA\-AMLS reduces zero\-estimate failures and under symmetric penalties improves rare\-probability estimation relative to baselines\. We also proposed SPB Loss to evaluate rare\-event estimates under asymmetric costs for underestimation and overestimation\. While the method depends on the fidelity of the activation\-space model, the results support activation\-space rare\-event simulation as a useful tool for measuring low\-probability LLM behavior\.

## Acknowledgement

This project is partially supported by Coefficient Giving\.

## References

- Concept heterogeneity\-aware representation steering\.External Links:2603\.02237,[Link](https://arxiv.org/abs/2603.02237)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- R\. Angell, R\. Singhal, Z\. Horvitz, Z\. Yu, R\. Ranganath, K\. McKeown, and H\. He \(2026\)Estimating tail risks in language model output distributions\.External Links:2604\.22167,[Link](https://arxiv.org/abs/2604.22167)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p1.1)\.
- A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. Nanda \(2024\)Refusal in language models is mediated by a single direction\.External Links:2406\.11717,[Link](https://arxiv.org/abs/2406.11717)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- N\. Belrose, I\. Ostrovsky, L\. McKinney, Z\. Furman, L\. Smith, D\. Halawi, S\. Biderman, and J\. Steinhardt \(2025\)Eliciting latent predictions from transformers with the tuned lens\.External Links:2303\.08112,[Link](https://arxiv.org/abs/2303.08112)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. L\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Note:Transformer Circuits ThreadExternal Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- A\. Buja, W\. Stuetzle, and Y\. Shen \(2005\)Loss functions for binary class probability estimation and classification: structure and applications\.Working draftUniversity of Pennsylvania, The Wharton School\.Cited by:[§4](https://arxiv.org/html/2607.18454#S4.p9.5)\.
- S\. Casper, L\. Schulze, O\. Patel, and D\. Hadfield\-Menell \(2024\)Defending Against Unforeseen Failure Modes with Latent Adversarial Training\.arXiv\.Note:arXiv:2403\.05030 \[cs\]External Links:[Link](http://arxiv.org/abs/2403.05030),[Document](https://dx.doi.org/10.48550/arXiv.2403.05030)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- F\. Cérou, A\. Guyader, and M\. Rousset \(2019\)Adaptive multilevel splitting: Historical perspective and recent results\.Chaos: An Interdisciplinary Journal of Nonlinear Science29\(4\),pp\. 043108\(en\)\.External Links:ISSN 1054\-1500, 1089\-7682,[Link](https://pubs.aip.org/cha/article/29/4/043108/1027309/Adaptive-multilevel-splitting-Historical),[Document](https://dx.doi.org/10.1063/1.5082247)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.18454#S2.SS2.p1.5),[§6](https://arxiv.org/html/2607.18454#S6.p3.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.External Links:2208\.07339,[Link](https://arxiv.org/abs/2208.07339)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p2.2),[§3](https://arxiv.org/html/2607.18454#S3.p3.3)\.
- J\. M\. Dorman, E\. Gillman, D\. C\. Rose, J\. F\. Mair, and J\. P\. Garrahan \(2026\)Rare event analysis of large language models\.External Links:2602\.06791,[Link](https://arxiv.org/abs/2602.06791)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p1.1)\.
- L\. Gao, T\. D\. la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. Wu \(2024\)Scaling and evaluating sparse autoencoders\.External Links:2406\.04093,[Link](https://arxiv.org/abs/2406.04093)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- T\. Gneiting and A\. E\. Raftery \(2007\)Strictly Proper Scoring Rules, Prediction, and Estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\(en\)\.External Links:ISSN 0162\-1459, 1537\-274X,[Link](http://www.tandfonline.com/doi/abs/10.1198/016214506000001437),[Document](https://dx.doi.org/10.1198/016214506000001437)Cited by:[§4](https://arxiv.org/html/2607.18454#S4.p6.1)\.
- I\. J\. Goodfellow, J\. Shlens, and C\. Szegedy \(2015\)Explaining and harnessing adversarial examples\.External Links:1412\.6572,[Link](https://arxiv.org/abs/1412.6572)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- A\. Guyader, N\. Hengartner, and E\. Matzner\-Løber \(2011\)Simulation and estimation of extreme quantiles and extreme probabilities\.Applied Mathematics & Optimization64,pp\. 171–196\.External Links:[Document](https://dx.doi.org/10.1007/s00245-011-9135-z)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.18454#S2.SS2.p1.5)\.
- D\. Hendrycks and K\. Gimpel \(2023\)Gaussian error linear units \(gelus\)\.External Links:1606\.08415,[Link](https://arxiv.org/abs/1606.08415)Cited by:[§5\.1](https://arxiv.org/html/2607.18454#S5.SS1.p2.4)\.
- E\. Jones, M\. Tong, J\. Mu, M\. Mahfoud, J\. Leike, R\. Grosse, J\. Kaplan, W\. Fithian, E\. Perez, and M\. Sharma \(2025\)Forecasting rare language model behaviors\.External Links:2502\.16797,[Link](https://arxiv.org/abs/2502.16797)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p1.1)\.
- D\. Kang, Y\. Sun, T\. Brown, D\. Hendrycks, and J\. Steinhardt \(2019\)Transfer of adversarial robustness between perturbation types\.External Links:1905\.01034,[Link](https://arxiv.org/abs/1905.01034)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- A\. Madry, A\. Makelov, L\. Schmidt, D\. Tsipras, and A\. Vladu \(2019\)Towards deep learning models resistant to adversarial attacks\.External Links:1706\.06083,[Link](https://arxiv.org/abs/1706.06083)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- N\. Nanda and J\. Bloom \(2022\)TransformerLens\.Note:[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)Cited by:[§5\.1](https://arxiv.org/html/2607.18454#S5.SS1.p2.4)\.
- N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.External Links:2312\.06681,[Link](https://arxiv.org/abs/2312.06681)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- K\. Park, Y\. J\. Choe, and V\. Veitch \(2024\)The linear representation hypothesis and the geometry of large language models\.External Links:2311\.03658,[Link](https://arxiv.org/abs/2311.03658)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2023\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.External Links:1910\.10683,[Link](https://arxiv.org/abs/1910.10683)Cited by:[§5\.1](https://arxiv.org/html/2607.18454#S5.SS1.p2.4)\.
- G\. O\. Roberts and J\. S\. Rosenthal \(1998\)Optimal Scaling of Discrete Approximations to Langevin Diffusions\.Journal of the Royal Statistical Society Series B: Statistical Methodology60\(1\),pp\. 255–268\(en\)\.External Links:ISSN 1369\-7412, 1467\-9868,[Link](https://academic.oup.com/jrsssb/article/60/1/255/7083121),[Document](https://dx.doi.org/10.1111/1467-9868.00123)Cited by:[§3](https://arxiv.org/html/2607.18454#S3.p13.10)\.
- G\. O\. Roberts and R\. L\. Tweedie \(1996\)Exponential convergence of langevin distributions and their discrete approximations\.Bernoulli2\(4\),pp\. 341–363\.External Links:ISSN 13507265,[Link](http://www.jstor.org/stable/3318418)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.18454#S2.SS2.p3.2)\.
- R\. Shah, V\. Varma, R\. Kumar, M\. Phuong, V\. Krakovna, J\. Uesato, and Z\. Kenton \(2022\)Goal misgeneralization: why correct specifications aren’t enough for correct goals\.External Links:2210\.01790,[Link](https://arxiv.org/abs/2210.01790)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p1.2)\.
- A\. Sheshadri, A\. Ewart, P\. Guo, A\. Lynch, C\. Wu, V\. Hebbar, H\. Sleight, A\. C\. Stickland, E\. Perez, D\. Hadfield\-Menell, and S\. Casper \(2025\)Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs\.arXiv\.Note:arXiv:2407\.15549 \[cs\]External Links:[Link](http://arxiv.org/abs/2407.15549),[Document](https://dx.doi.org/10.48550/arXiv.2407.15549)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- V\. Srinivasan, A\. Wibisono, and A\. Wilson \(2025\)High\-accuracy sampling from constrained spaces with the Metropolis\-adjusted Preconditioned Langevin Algorithm\.arXiv\.Note:arXiv:2412\.18701 \[stat\]External Links:[Link](http://arxiv.org/abs/2412.18701),[Document](https://dx.doi.org/10.48550/arXiv.2412.18701)Cited by:[§D\.1](https://arxiv.org/html/2607.18454#A4.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.18454#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.18454#S2.SS2.p3.2)\.
- M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu \(2024\)Massive activations in large language models\.External Links:2402\.17762,[Link](https://arxiv.org/abs/2402.17762)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p2.2),[§3](https://arxiv.org/html/2607.18454#S3.p3.3)\.
- A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, B\. Chen, A\. Pearce, C\. Citro, E\. Ameisen, A\. Jones, H\. Cunningham, N\. L\. Turner, C\. McDougall, M\. MacDiarmid, A\. Tamkin, E\. Durmus, T\. Hume, F\. Mosconi, C\. D\. Freeman, T\. R\. Sumers, E\. Rees, J\. Batson, A\. Jermyn, S\. Carter, C\. Olah, and T\. Henighan \(2024\)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet\.Note:Transformer Circuits ThreadExternal Links:[Link](https://transformer-circuits.pub/2024/scaling-monosemanticity)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- L\. Tunstall, L\. von Werra, and T\. Wolf \(2022\)Natural language processing with transformers: building language applications with hugging face\.O’Reilly Media\.External Links:ISBN 9781098103248,LCCN 2023275986,[Link](https://books.google.ch/books?id=7hhyzgEACAAJ)Cited by:[§5\.1](https://arxiv.org/html/2607.18454#S5.SS1.p2.4)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2024\)Steering language models with activation engineering\.External Links:2308\.10248,[Link](https://arxiv.org/abs/2308.10248)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- S\. Webb, T\. Rainforth, Y\. W\. Teh, and M\. P\. Kumar \(2019\)A Statistical Approach to Assessing Neural Network Robustness\.arXiv\.Note:arXiv:1811\.07209 \[stat\]External Links:[Link](http://arxiv.org/abs/1811.07209),[Document](https://dx.doi.org/10.48550/arXiv.1811.07209)Cited by:[§1](https://arxiv.org/html/2607.18454#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.18454#S2.SS2.p1.5),[§6](https://arxiv.org/html/2607.18454#S6.p3.1)\.
- A\. Wei, N\. Haghtalab, and J\. Steinhardt \(2023\)Jailbroken: how does llm safety training fail?\.External Links:2307\.02483,[Link](https://arxiv.org/abs/2307.02483)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- T\. Winninger, B\. Addad, and K\. Kapusta \(2025\)Using mechanistic interpretability to craft adversarial attacks against large language models\.External Links:2503\.06269,[Link](https://arxiv.org/abs/2503.06269)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.
- G\. Wu and J\. Hilton \(2025\)Estimating the probabilities of rare outputs in language models\.External Links:2410\.13211,[Link](https://arxiv.org/abs/2410.13211)Cited by:[§D\.1](https://arxiv.org/html/2607.18454#A4.SS1.SSS0.Px1.p2.3),[§D\.1](https://arxiv.org/html/2607.18454#A4.SS1.SSS0.Px1.p4.2),[§1](https://arxiv.org/html/2607.18454#S1.p1.2),[§1](https://arxiv.org/html/2607.18454#S1.p2.2),[§1](https://arxiv.org/html/2607.18454#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.18454#S2.SS1.p1.8),[§2\.1](https://arxiv.org/html/2607.18454#S2.SS1.p3.6),[§4](https://arxiv.org/html/2607.18454#S4.SS0.SSS0.Px1.p1.4),[§4](https://arxiv.org/html/2607.18454#S4.p3.2),[§4](https://arxiv.org/html/2607.18454#S4.p7.2),[§5\.1](https://arxiv.org/html/2607.18454#S5.SS1.p1.3),[§5\.1](https://arxiv.org/html/2607.18454#S5.SS1.p2.4),[§5](https://arxiv.org/html/2607.18454#S5.p1.1),[§6](https://arxiv.org/html/2607.18454#S6.p1.1),[§7](https://arxiv.org/html/2607.18454#S7.p4.5)\.
- W\. Wu, V\. Lecomte, M\. Winer, G\. Robinson, J\. Hilton, and P\. Christiano \(2026\)Estimating the expected output of wide random mlps more efficiently than sampling\.External Links:2605\.05179,[Link](https://arxiv.org/abs/2605.05179)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. Hendrycks \(2023a\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.External Links:2310\.01405,[Link](https://arxiv.org/abs/2310.01405)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p2.1)\.
- A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. Fredrikson \(2023b\)Universal and transferable adversarial attacks on aligned language models\.External Links:2307\.15043,[Link](https://arxiv.org/abs/2307.15043)Cited by:[§6](https://arxiv.org/html/2607.18454#S6.p4.1)\.

##### Appendix outline\.

Appendix[A](https://arxiv.org/html/2607.18454#A1)develops the SPB loss theory from its Bregman\-divergence representation, derives the unshifted power family, and analyzes the resulting false\-negative/false\-positive asymmetry\. Appendix[B](https://arxiv.org/html/2607.18454#A2)gives the shifted\-power parameterization used in the paper, proves bounded scale distortion, explains how to chooseε\\varepsilon,α\\alpha, andγ\\gamma, provides a closed\-form computation, connects SPB to standard losses, and empirically verifies the intended scaling behavior\. Appendix C gives the full GA\-AMLS procedure in Algorithm[1](https://arxiv.org/html/2607.18454#alg1)\. Appendix D reports experimental and implementation details, including hyperparameters, baseline implementation notes, hyperparameter sweeps and ablations, and the activation\-whitening diagnostic\. Appendix[E](https://arxiv.org/html/2607.18454#A5)provides the full set of estimate versus ground\-truth scatterplots across model sizes and input distributions\. Appendix[F](https://arxiv.org/html/2607.18454#A6)breaks down SPB losses by distribution, model size, method, and evaluation profile\.

##### Code Availability\.

The code is available upon request currently and will be made public soon\.

## Appendix ASPB Loss Theory

### A\.1Bregman integral representation

Letq∈\(0,1\)q\\in\(0,1\)denote the true probability andp∈\[0,1\]p\\in\[0,1\]the predicted probability\. A one\-dimensional Bregman divergence generated by a twice differentiable convex potential with second derivativeω​\(t\)\>0\\omega\(t\)\>0can be written as

Bω​\(q∣p\)=∫qp\(t−q\)​ω​\(t\)​𝑑t\.B\_\{\\omega\}\(q\\mid p\)=\\int\_\{q\}^\{p\}\(t\-q\)\\omega\(t\)\\,dt\.\(1\)This is an oriented integral\. Equivalently,

Bω​\(q∣p\)=\{∫qp\(t−q\)​ω​\(t\)​𝑑t,p≥q,∫pq\(q−t\)​ω​\(t\)​𝑑t,p<q\.B\_\{\\omega\}\(q\\mid p\)=\\begin\{cases\}\\displaystyle\\int\_\{q\}^\{p\}\(t\-q\)\\omega\(t\)\\,dt,&p\\geq q,\\\\\[8\.0pt\] \\displaystyle\\int\_\{p\}^\{q\}\(q\-t\)\\omega\(t\)\\,dt,&p<q\.\\end\{cases\}Thus the divergence is nonnegative, and ifω​\(t\)\>0\\omega\(t\)\>0almost everywhere, thenBω​\(q∣p\)=0B\_\{\\omega\}\(q\\mid p\)=0if and only ifp=qp=q, whenever the integral is finite\.

### A\.2The unshifted power family

Consider the unshifted power weight

ω0​\(t\)=t−α,α\>0\.\\omega\_\{0\}\(t\)=t^\{\-\\alpha\},\\qquad\\alpha\>0\.\(2\)For a multiplicative overestimatep=m​qp=mq, withm\>1m\>1, the change of variablest=q​ut=qugives

B0​\(q∣m​q\)\\displaystyle B\_\{0\}\(q\\mid mq\)=∫qm​q\(t−q\)​t−α​𝑑t\\displaystyle=\\int\_\{q\}^\{mq\}\(t\-q\)t^\{\-\\alpha\}\\,dt=q2−α​∫1m\(u−1\)​u−α​𝑑u\\displaystyle=q^\{2\-\\alpha\}\\int\_\{1\}^\{m\}\(u\-1\)u^\{\-\\alpha\}\\,du=q2−α​IF​P​\(α,m\),\\displaystyle=q^\{2\-\\alpha\}I\_\{FP\}\(\\alpha,m\),\(3\)where

IF​P​\(α,m\):=∫1m\(u−1\)​u−α​𝑑u\.I\_\{FP\}\(\\alpha,m\):=\\int\_\{1\}^\{m\}\(u\-1\)u^\{\-\\alpha\}\\,du\.\(4\)Similarly, for a multiplicative underestimatep=q/mp=q/m,

B0​\(q∣q/m\)\\displaystyle B\_\{0\}\(q\\mid q/m\)=∫q/mq\(q−t\)​t−α​𝑑t\\displaystyle=\\int\_\{q/m\}^\{q\}\(q\-t\)t^\{\-\\alpha\}\\,dt=q2−α​∫1/m1\(1−u\)​u−α​𝑑u\\displaystyle=q^\{2\-\\alpha\}\\int\_\{1/m\}^\{1\}\(1\-u\)u^\{\-\\alpha\}\\,du=q2−α​IF​N​\(α,m\),\\displaystyle=q^\{2\-\\alpha\}I\_\{FN\}\(\\alpha,m\),\(5\)where

IF​N​\(α,m\):=∫1/m1\(1−u\)​u−α​𝑑u\.I\_\{FN\}\(\\alpha,m\):=\\int\_\{1/m\}^\{1\}\(1\-u\)u^\{\-\\alpha\}\\,du\.\(6\)
Therefore both false\-positive and false\-negative multiplicative errors factor into a ground\-truth scale termq2−αq^\{2\-\\alpha\}and a relative\-error term\. The dataset\-level weight

w​\(q\)=qα−2w\(q\)=q^\{\\alpha\-2\}\(7\)cancels theq2−αq^\{2\-\\alpha\}prefactor exactly:

w​\(q\)​B0​\(q∣m​q\)=IF​P​\(α,m\),w​\(q\)​B0​\(q∣q/m\)=IF​N​\(α,m\)\.w\(q\)B\_\{0\}\(q\\mid mq\)=I\_\{FP\}\(\\alpha,m\),\\qquad w\(q\)B\_\{0\}\(q\\mid q/m\)=I\_\{FN\}\(\\alpha,m\)\.Thus the weighted unshifted power family is exactly multiplicatively scale\-invariant: for fixed relative errormm, the weighted loss is independent of the absolute scaleqq\.

More generally, if one wants to add an explicit rarity premiumγ≥0\\gamma\\geq 0, one can use

wγ​\(q\)=qα−2−γ,w\_\{\\gamma\}\(q\)=q^\{\\alpha\-2\-\\gamma\},in which case the weighted unshifted loss scales asq−γq^\{\-\\gamma\}times the relative\-error term\. The formal scale\-invariance results below correspond to the scale\-neutral caseγ=0\\gamma=0\.

### A\.3Unshifted FN/FP asymmetry

Define the unshifted FN\-to\-FP ratio

Rα​\(m\):=IF​N​\(α,m\)IF​P​\(α,m\)\.R\_\{\\alpha\}\(m\):=\\frac\{I\_\{FN\}\(\\alpha,m\)\}\{I\_\{FP\}\(\\alpha,m\)\}\.\(8\)The following lemma shows that forα\>3/2\\alpha\>3/2, underestimation is penalized more than overestimation, and the asymmetry increases with the multiplicative error size\.

###### Lemma 1\.

Forα\>3/2\\alpha\>3/2,Rα​\(m\)R\_\{\\alpha\}\(m\)is strictly increasing on\(1,∞\)\(1,\\infty\)\. Moreover,Rα​\(m\)\>1R\_\{\\alpha\}\(m\)\>1for allm\>1m\>1\.

###### Proof\.

Starting from the definition ofIF​NI\_\{FN\}, apply the substitutionu=1/vu=1/v\. Thend​u=−v−2​d​vdu=\-v^\{\-2\}\\,dv, and asuugoes from1/m1/mto11,vvgoes frommmto11\. Hence

IF​N​\(α,m\)\\displaystyle I\_\{FN\}\(\\alpha,m\)=∫1/m1\(1−u\)​u−α​𝑑u\\displaystyle=\\int\_\{1/m\}^\{1\}\(1\-u\)u^\{\-\\alpha\}\\,du=∫m1\(1−1v\)​vα​\(−v−2\)​𝑑v\\displaystyle=\\int\_\{m\}^\{1\}\\left\(1\-\\frac\{1\}\{v\}\\right\)v^\{\\alpha\}\(\-v^\{\-2\}\)\\,dv=∫1m\(1−1v\)​vα−2​𝑑v\\displaystyle=\\int\_\{1\}^\{m\}\\left\(1\-\\frac\{1\}\{v\}\\right\)v^\{\\alpha\-2\}\\,dv=∫1m\(v−1\)​vα−3​𝑑v\.\\displaystyle=\\int\_\{1\}^\{m\}\(v\-1\)v^\{\\alpha\-3\}\\,dv\.Renamingvvasuu,

IF​N​\(α,m\)=∫1m\(u−1\)​uα−3​𝑑u=∫1m\(u−1\)​u−α​u2​α−3​𝑑u\.I\_\{FN\}\(\\alpha,m\)=\\int\_\{1\}^\{m\}\(u\-1\)u^\{\\alpha\-3\}\\,du=\\int\_\{1\}^\{m\}\(u\-1\)u^\{\-\\alpha\}u^\{2\\alpha\-3\}\\,du\.Let

f​\(u\):=\(u−1\)​u−α,g​\(u\):=u2​α−3\.f\(u\):=\(u\-1\)u^\{\-\\alpha\},\\qquad g\(u\):=u^\{2\\alpha\-3\}\.Thenf​\(u\)\>0f\(u\)\>0foru\>1u\>1, and becauseα\>3/2\\alpha\>3/2,g​\(u\)g\(u\)is strictly increasing on\(1,∞\)\(1,\\infty\)\. Moreover,

IF​P​\(α,m\)=∫1mf​\(u\)​𝑑u,IF​N​\(α,m\)=∫1mf​\(u\)​g​\(u\)​𝑑u\.I\_\{FP\}\(\\alpha,m\)=\\int\_\{1\}^\{m\}f\(u\)\\,du,\\qquad I\_\{FN\}\(\\alpha,m\)=\\int\_\{1\}^\{m\}f\(u\)g\(u\)\\,du\.Thus

Rα​\(m\)=∫1mf​\(u\)​g​\(u\)​𝑑u∫1mf​\(u\)​𝑑u,R\_\{\\alpha\}\(m\)=\\frac\{\\int\_\{1\}^\{m\}f\(u\)g\(u\)\\,du\}\{\\int\_\{1\}^\{m\}f\(u\)\\,du\},soRα​\(m\)R\_\{\\alpha\}\(m\)is theff\-weighted average of the increasing functionggover\[1,m\]\[1,m\]\.

To prove strict monotonicity, define

A​\(m\):=∫1mf​\(u\)​g​\(u\)​𝑑u,C​\(m\):=∫1mf​\(u\)​𝑑u\.A\(m\):=\\int\_\{1\}^\{m\}f\(u\)g\(u\)\\,du,\\qquad C\(m\):=\\int\_\{1\}^\{m\}f\(u\)\\,du\.ThenRα​\(m\)=A​\(m\)/C​\(m\)R\_\{\\alpha\}\(m\)=A\(m\)/C\(m\), and

A′​\(m\)=f​\(m\)​g​\(m\),C′​\(m\)=f​\(m\)\.A^\{\\prime\}\(m\)=f\(m\)g\(m\),\\qquad C^\{\\prime\}\(m\)=f\(m\)\.Therefore

Rα′​\(m\)\\displaystyle R\_\{\\alpha\}^\{\\prime\}\(m\)=A′​\(m\)​C​\(m\)−A​\(m\)​C′​\(m\)C​\(m\)2\\displaystyle=\\frac\{A^\{\\prime\}\(m\)C\(m\)\-A\(m\)C^\{\\prime\}\(m\)\}\{C\(m\)^\{2\}\}=f​\(m\)C​\(m\)​\[g​\(m\)−A​\(m\)C​\(m\)\]\\displaystyle=\\frac\{f\(m\)\}\{C\(m\)\}\\left\[g\(m\)\-\\frac\{A\(m\)\}\{C\(m\)\}\\right\]=f​\(m\)C​\(m\)​\[g​\(m\)−Rα​\(m\)\]\.\\displaystyle=\\frac\{f\(m\)\}\{C\(m\)\}\\left\[g\(m\)\-R\_\{\\alpha\}\(m\)\\right\]\.Sinceggis strictly increasing, its weighted average over\[1,m\]\[1,m\]is strictly smaller than its endpoint valueg​\(m\)g\(m\)\. Hence

g​\(m\)−Rα​\(m\)\>0\.g\(m\)\-R\_\{\\alpha\}\(m\)\>0\.Alsof​\(m\)\>0f\(m\)\>0andC​\(m\)\>0C\(m\)\>0form\>1m\>1\. Therefore

Rα′​\(m\)\>0\.R\_\{\\alpha\}^\{\\prime\}\(m\)\>0\.SoRαR\_\{\\alpha\}is strictly increasing on\(1,∞\)\(1,\\infty\)\.

Finally, becauseg​\(u\)=u2​α−3\>1g\(u\)=u^\{2\\alpha\-3\}\>1for allu\>1u\>1, its weighted average is also greater than11\. HenceRα​\(m\)\>1R\_\{\\alpha\}\(m\)\>1for allm\>1m\>1\. ∎

At the boundaryα=3/2\\alpha=3/2, we haveg​\(u\)≡1g\(u\)\\equiv 1, soR3/2​\(m\)=1R\_\{3/2\}\(m\)=1for allm\>1m\>1\. Thusα=3/2\\alpha=3/2is the symmetric point, whileα\>3/2\\alpha\>3/2makes false negatives more costly than false positives; the mirrored argument, withggstrictly decreasing, shows thatα<3/2\\alpha<3/2makes false positives more costly\.

## Appendix BSPB Loss Guarantees and Parameterization

### B\.1The shifted power family

The unshifted power weightt−αt^\{\-\\alpha\}gives exact scale invariance after dataset weighting, but it is singular att=0t=0\. In particular, forα≥1\\alpha\\geq 1, predictingp=0p=0can produce an infinite penalty\. To make the loss finite at zero, define the shifted power weight

ωε​\(t\):=\(t\+ε\)−α,α\>0,ε\>0\.\\omega\_\{\\varepsilon\}\(t\):=\(t\+\\varepsilon\)^\{\-\\alpha\},\\qquad\\alpha\>0,\\quad\\varepsilon\>0\.\(9\)The corresponding shifted divergence is

Bε​\(q∣p\)=∫qp\(t−q\)​\(t\+ε\)−α​𝑑t\.B\_\{\\varepsilon\}\(q\\mid p\)=\\int\_\{q\}^\{p\}\(t\-q\)\(t\+\\varepsilon\)^\{\-\\alpha\}\\,dt\.\(10\)Because\(t\+ε\)−α≤ε−α<∞\(t\+\\varepsilon\)^\{\-\\alpha\}\\leq\\varepsilon^\{\-\\alpha\}<\\inftyon\[0,1\]\[0,1\], the shifted divergence is finite even whenp=0p=0\.

The price of this shift is that exact multiplicative scale invariance is lost: after the change of variablest=q​ut=qu, the ratio

x:=εqx:=\\frac\{\\varepsilon\}\{q\}remains inside the integral\. The following results show that the desired properties are preserved on a finite evaluation box

q∈\[qmin,qmax\],m∈\(1,mmax\],q\\in\[q\_\{\\min\},q\_\{\\max\}\],\\qquad m\\in\(1,m\_\{\\max\}\],providedε\\varepsilonis sufficiently small relative toqminq\_\{\\min\}\.

### B\.2Bounded scale distortion

###### Theorem 1\(Bounded Scale Distortion\)\.

Fixα\>0\\alpha\>0,η∈\(0,1\)\\eta\\in\(0,1\), andmmax\>1m\_\{\\max\}\>1\. Letw​\(q\)=qα−2w\(q\)=q^\{\\alpha\-2\}\. If

ε≤qminmmax​\(\(1−η\)−1/α−1\),\\varepsilon\\leq\\frac\{q\_\{\\min\}\}\{m\_\{\\max\}\}\\left\(\(1\-\\eta\)^\{\-1/\\alpha\}\-1\\right\),\(11\)then, uniformly overq∈\[qmin,qmax\]q\\in\[q\_\{\\min\},q\_\{\\max\}\]andm∈\(1,mmax\]m\\in\(1,m\_\{\\max\}\],

0≤IF​P​\(α,m\)−w​\(q\)​Bε​\(q∣m​q\)IF​P​\(α,m\)\\displaystyle 0\\leq\\frac\{I\_\{FP\}\(\\alpha,m\)\-w\(q\)B\_\{\\varepsilon\}\(q\\mid mq\)\}\{I\_\{FP\}\(\\alpha,m\)\}≤η,\\displaystyle\\leq\\eta,\(12\)0≤IF​N​\(α,m\)−w​\(q\)​Bε​\(q∣q/m\)IF​N​\(α,m\)\\displaystyle 0\\leq\\frac\{I\_\{FN\}\(\\alpha,m\)\-w\(q\)B\_\{\\varepsilon\}\(q\\mid q/m\)\}\{I\_\{FN\}\(\\alpha,m\)\}≤η\.\\displaystyle\\leq\\eta\.\(13\)

###### Proof\.

Fixq∈\[qmin,qmax\]q\\in\[q\_\{\\min\},q\_\{\\max\}\]andm∈\(1,mmax\]m\\in\(1,m\_\{\\max\}\], and define

x:=εq\.x:=\\frac\{\\varepsilon\}\{q\}\.
First consider the overestimatep=m​qp=mq\. Usingt=q​ut=qu,

Bε​\(q∣m​q\)\\displaystyle B\_\{\\varepsilon\}\(q\\mid mq\)=∫qm​q\(t−q\)​\(t\+ε\)−α​𝑑t\\displaystyle=\\int\_\{q\}^\{mq\}\(t\-q\)\(t\+\\varepsilon\)^\{\-\\alpha\}\\,dt=q2−α​∫1m\(u−1\)​\(u\+x\)−α​𝑑u\\displaystyle=q^\{2\-\\alpha\}\\int\_\{1\}^\{m\}\(u\-1\)\(u\+x\)^\{\-\\alpha\}\\,du=q2−α​IF​P,ε​\(α,m;x\),\\displaystyle=q^\{2\-\\alpha\}I\_\{FP,\\varepsilon\}\(\\alpha,m;x\),\(14\)where

IF​P,ε​\(α,m;x\):=∫1m\(u−1\)​\(u\+x\)−α​𝑑u\.I\_\{FP,\\varepsilon\}\(\\alpha,m;x\):=\\int\_\{1\}^\{m\}\(u\-1\)\(u\+x\)^\{\-\\alpha\}\\,du\.Foru∈\[1,m\]u\\in\[1,m\],

u≤u\+x≤u​\(1\+x\)\.u\\leq u\+x\\leq u\(1\+x\)\.Sinceα\>0\\alpha\>0, raising to the power−α\-\\alphareverses the inequalities:

u−α≥\(u\+x\)−α≥u−α​\(1\+x\)−α\.u^\{\-\\alpha\}\\geq\(u\+x\)^\{\-\\alpha\}\\geq u^\{\-\\alpha\}\(1\+x\)^\{\-\\alpha\}\.Multiplying by\(u−1\)≥0\(u\-1\)\\geq 0and integrating gives

\(1\+x\)−α​IF​P​\(α,m\)≤IF​P,ε​\(α,m;x\)≤IF​P​\(α,m\)\.\(1\+x\)^\{\-\\alpha\}I\_\{FP\}\(\\alpha,m\)\\leq I\_\{FP,\\varepsilon\}\(\\alpha,m;x\)\\leq I\_\{FP\}\(\\alpha,m\)\.\(15\)Therefore

0≤IF​P​\(α,m\)−IF​P,ε​\(α,m;x\)IF​P​\(α,m\)≤1−\(1\+x\)−α\.0\\leq\\frac\{I\_\{FP\}\(\\alpha,m\)\-I\_\{FP,\\varepsilon\}\(\\alpha,m;x\)\}\{I\_\{FP\}\(\\alpha,m\)\}\\leq 1\-\(1\+x\)^\{\-\\alpha\}\.\(16\)
Now consider the underestimatep=q/mp=q/m\. Again usingt=q​ut=qu,

Bε​\(q∣q/m\)\\displaystyle B\_\{\\varepsilon\}\(q\\mid q/m\)=∫q/mq\(q−t\)​\(t\+ε\)−α​𝑑t\\displaystyle=\\int\_\{q/m\}^\{q\}\(q\-t\)\(t\+\\varepsilon\)^\{\-\\alpha\}\\,dt=q2−α​∫1/m1\(1−u\)​\(u\+x\)−α​𝑑u\\displaystyle=q^\{2\-\\alpha\}\\int\_\{1/m\}^\{1\}\(1\-u\)\(u\+x\)^\{\-\\alpha\}\\,du=q2−α​IF​N,ε​\(α,m;x\),\\displaystyle=q^\{2\-\\alpha\}I\_\{FN,\\varepsilon\}\(\\alpha,m;x\),\(17\)where

IF​N,ε​\(α,m;x\):=∫1/m1\(1−u\)​\(u\+x\)−α​𝑑u\.I\_\{FN,\\varepsilon\}\(\\alpha,m;x\):=\\int\_\{1/m\}^\{1\}\(1\-u\)\(u\+x\)^\{\-\\alpha\}\\,du\.Foru∈\[1/m,1\]u\\in\[1/m,1\], we have1/u≤m1/u\\leq m\. Hence

u\+x=u​\(1\+xu\)≤u​\(1\+m​x\),u\+x=u\\left\(1\+\\frac\{x\}\{u\}\\right\)\\leq u\(1\+mx\),and alsou\+x≥uu\+x\\geq u\. Therefore

u≤u\+x≤u​\(1\+m​x\)\.u\\leq u\+x\\leq u\(1\+mx\)\.Raising to the power−α\-\\alphagives

u−α≥\(u\+x\)−α≥u−α​\(1\+m​x\)−α\.u^\{\-\\alpha\}\\geq\(u\+x\)^\{\-\\alpha\}\\geq u^\{\-\\alpha\}\(1\+mx\)^\{\-\\alpha\}\.Multiplying by\(1−u\)≥0\(1\-u\)\\geq 0and integrating gives

\(1\+m​x\)−α​IF​N​\(α,m\)≤IF​N,ε​\(α,m;x\)≤IF​N​\(α,m\)\.\(1\+mx\)^\{\-\\alpha\}I\_\{FN\}\(\\alpha,m\)\\leq I\_\{FN,\\varepsilon\}\(\\alpha,m;x\)\\leq I\_\{FN\}\(\\alpha,m\)\.\(18\)Therefore

0≤IF​N​\(α,m\)−IF​N,ε​\(α,m;x\)IF​N​\(α,m\)≤1−\(1\+m​x\)−α\.0\\leq\\frac\{I\_\{FN\}\(\\alpha,m\)\-I\_\{FN,\\varepsilon\}\(\\alpha,m;x\)\}\{I\_\{FN\}\(\\alpha,m\)\}\\leq 1\-\(1\+mx\)^\{\-\\alpha\}\.\(19\)
Now assume \([11](https://arxiv.org/html/2607.18454#A2.E11)\)\. Sinceq≥qminq\\geq q\_\{\\min\}andm≤mmaxm\\leq m\_\{\\max\},

m​x=m​εq≤mmax​εqmin≤\(1−η\)−1/α−1\.mx=\\frac\{m\\varepsilon\}\{q\}\\leq\\frac\{m\_\{\\max\}\\varepsilon\}\{q\_\{\\min\}\}\\leq\(1\-\\eta\)^\{\-1/\\alpha\}\-1\.Thus

1\+m​x≤\(1−η\)−1/α\.1\+mx\\leq\(1\-\\eta\)^\{\-1/\\alpha\}\.Equivalently,

\(1\+m​x\)−α≥1−η,\(1\+mx\)^\{\-\\alpha\}\\geq 1\-\\eta,so

1−\(1\+m​x\)−α≤η\.1\-\(1\+mx\)^\{\-\\alpha\}\\leq\\eta\.This proves the FN distortion bound\. Sincex≤m​xx\\leq mx, the same condition also implies

1−\(1\+x\)−α≤η,1\-\(1\+x\)^\{\-\\alpha\}\\leq\\eta,and therefore proves the FP distortion bound\.

Finally, becausew​\(q\)=qα−2w\(q\)=q^\{\\alpha\-2\},

w​\(q\)​Bε​\(q∣m​q\)=IF​P,ε​\(α,m;x\),w\(q\)B\_\{\\varepsilon\}\(q\\mid mq\)=I\_\{FP,\\varepsilon\}\(\\alpha,m;x\),and

w​\(q\)​Bε​\(q∣q/m\)=IF​N,ε​\(α,m;x\)\.w\(q\)B\_\{\\varepsilon\}\(q\\mid q/m\)=I\_\{FN,\\varepsilon\}\(\\alpha,m;x\)\.Substituting these identities into the relative distortion bounds proves the claim\. ∎

### B\.3Asymmetric sensitivity

###### Theorem 2\(Asymmetric Sensitivity\)\.

Fixα\>3/2\\alpha\>3/2and1<m0≤mmax1<m\_\{0\}\\leq m\_\{\\max\}\. Let

Rα​\(m\):=IF​N​\(α,m\)IF​P​\(α,m\)\.R\_\{\\alpha\}\(m\):=\\frac\{I\_\{FN\}\(\\alpha,m\)\}\{I\_\{FP\}\(\\alpha,m\)\}\.If

ε<qminmmax​\[Rα​\(m0\)1/α−1\],\\varepsilon<\\frac\{q\_\{\\min\}\}\{m\_\{\\max\}\}\\left\[R\_\{\\alpha\}\(m\_\{0\}\)^\{1/\\alpha\}\-1\\right\],\(20\)then

Bε​\(q∣q/m\)\>Bε​\(q∣m​q\)B\_\{\\varepsilon\}\(q\\mid q/m\)\>B\_\{\\varepsilon\}\(q\\mid mq\)\(21\)for allq∈\[qmin,qmax\]q\\in\[q\_\{\\min\},q\_\{\\max\}\]and allm∈\[m0,mmax\]m\\in\[m\_\{0\},m\_\{\\max\}\]\.

###### Proof\.

Fixq∈\[qmin,qmax\]q\\in\[q\_\{\\min\},q\_\{\\max\}\]andm∈\[m0,mmax\]m\\in\[m\_\{0\},m\_\{\\max\}\], and define

x:=εq\.x:=\\frac\{\\varepsilon\}\{q\}\.
For the overestimatep=m​qp=mq,

Bε​\(q∣m​q\)=q2−α​IF​P,ε​\(α,m;x\),B\_\{\\varepsilon\}\(q\\mid mq\)=q^\{2\-\\alpha\}I\_\{FP,\\varepsilon\}\(\\alpha,m;x\),where

IF​P,ε​\(α,m;x\)=∫1m\(u−1\)​\(u\+x\)−α​𝑑u\.I\_\{FP,\\varepsilon\}\(\\alpha,m;x\)=\\int\_\{1\}^\{m\}\(u\-1\)\(u\+x\)^\{\-\\alpha\}\\,du\.Sincex≥0x\\geq 0andα\>0\\alpha\>0,

\(u\+x\)−α≤u−α\.\(u\+x\)^\{\-\\alpha\}\\leq u^\{\-\\alpha\}\.Therefore

IF​P,ε​\(α,m;x\)≤IF​P​\(α,m\)\.I\_\{FP,\\varepsilon\}\(\\alpha,m;x\)\\leq I\_\{FP\}\(\\alpha,m\)\.\(22\)
For the underestimatep=q/mp=q/m,

Bε​\(q∣q/m\)=q2−α​IF​N,ε​\(α,m;x\),B\_\{\\varepsilon\}\(q\\mid q/m\)=q^\{2\-\\alpha\}I\_\{FN,\\varepsilon\}\(\\alpha,m;x\),where

IF​N,ε​\(α,m;x\)=∫1/m1\(1−u\)​\(u\+x\)−α​𝑑u\.I\_\{FN,\\varepsilon\}\(\\alpha,m;x\)=\\int\_\{1/m\}^\{1\}\(1\-u\)\(u\+x\)^\{\-\\alpha\}\\,du\.Foru∈\[1/m,1\]u\\in\[1/m,1\], we have

u\+x=u​\(1\+xu\)≤u​\(1\+m​x\)\.u\+x=u\\left\(1\+\\frac\{x\}\{u\}\\right\)\\leq u\(1\+mx\)\.Hence

\(u\+x\)−α≥u−α​\(1\+m​x\)−α\.\(u\+x\)^\{\-\\alpha\}\\geq u^\{\-\\alpha\}\(1\+mx\)^\{\-\\alpha\}\.Multiplying by\(1−u\)≥0\(1\-u\)\\geq 0and integrating gives

IF​N,ε​\(α,m;x\)≥\(1\+m​x\)−α​IF​N​\(α,m\)\.I\_\{FN,\\varepsilon\}\(\\alpha,m;x\)\\geq\(1\+mx\)^\{\-\\alpha\}I\_\{FN\}\(\\alpha,m\)\.\(23\)
It is therefore enough to ensure

\(1\+m​x\)−α​IF​N​\(α,m\)\>IF​P​\(α,m\)\.\(1\+mx\)^\{\-\\alpha\}I\_\{FN\}\(\\alpha,m\)\>I\_\{FP\}\(\\alpha,m\)\.Equivalently, using

Rα​\(m\)=IF​N​\(α,m\)IF​P​\(α,m\),R\_\{\\alpha\}\(m\)=\\frac\{I\_\{FN\}\(\\alpha,m\)\}\{I\_\{FP\}\(\\alpha,m\)\},it is enough to have

\(1\+m​x\)α<Rα​\(m\),\(1\+mx\)^\{\\alpha\}<R\_\{\\alpha\}\(m\),or

1\+m​x<Rα​\(m\)1/α\.1\+mx<R\_\{\\alpha\}\(m\)^\{1/\\alpha\}\.
By Lemma[1](https://arxiv.org/html/2607.18454#Thmlemma1),Rα​\(m\)R\_\{\\alpha\}\(m\)is strictly increasing inmmforα\>3/2\\alpha\>3/2\. Sincem≥m0m\\geq m\_\{0\},

Rα​\(m\)≥Rα​\(m0\)\.R\_\{\\alpha\}\(m\)\\geq R\_\{\\alpha\}\(m\_\{0\}\)\.Also, sincem≤mmaxm\\leq m\_\{\\max\}andq≥qminq\\geq q\_\{\\min\},

m​x=m​εq≤mmax​εqmin\.mx=\\frac\{m\\varepsilon\}\{q\}\\leq\\frac\{m\_\{\\max\}\\varepsilon\}\{q\_\{\\min\}\}\.The assumed condition \([20](https://arxiv.org/html/2607.18454#A2.E20)\) implies

1\+mmax​εqmin<Rα​\(m0\)1/α\.1\+\\frac\{m\_\{\\max\}\\varepsilon\}\{q\_\{\\min\}\}<R\_\{\\alpha\}\(m\_\{0\}\)^\{1/\\alpha\}\.Therefore

1\+m​x<Rα​\(m0\)1/α≤Rα​\(m\)1/α\.1\+mx<R\_\{\\alpha\}\(m\_\{0\}\)^\{1/\\alpha\}\\leq R\_\{\\alpha\}\(m\)^\{1/\\alpha\}\.Hence

\(1\+m​x\)−α​IF​N​\(α,m\)\>IF​P​\(α,m\)\.\(1\+mx\)^\{\-\\alpha\}I\_\{FN\}\(\\alpha,m\)\>I\_\{FP\}\(\\alpha,m\)\.Combining this strict inequality with \([22](https://arxiv.org/html/2607.18454#A2.E22)\) and \([23](https://arxiv.org/html/2607.18454#A2.E23)\), we obtain

IF​N,ε​\(α,m;x\)\>IF​P,ε​\(α,m;x\)\.I\_\{FN,\\varepsilon\}\(\\alpha,m;x\)\>I\_\{FP,\\varepsilon\}\(\\alpha,m;x\)\.Multiplying by the positive factorq2−αq^\{2\-\\alpha\}gives

Bε​\(q∣q/m\)\>Bε​\(q∣m​q\)\.B\_\{\\varepsilon\}\(q\\mid q/m\)\>B\_\{\\varepsilon\}\(q\\mid mq\)\.This proves the theorem\. ∎

### B\.4Simultaneous finite\-error guarantee

###### Corollary 1\(Simultaneous finite\-error guarantee\)\.

Fixα\>3/2\\alpha\>3/2,η∈\(0,1\)\\eta\\in\(0,1\), and

1<m0≤m≤mmax\.1<m\_\{0\}\\leq m\\leq m\_\{\\max\}\.Let

Rα​\(m\):=IF​N​\(α,m\)IF​P​\(α,m\)\.R\_\{\\alpha\}\(m\):=\\frac\{I\_\{FN\}\(\\alpha,m\)\}\{I\_\{FP\}\(\\alpha,m\)\}\.If

0≤ε<min⁡\{qminmmax​\(\(1−η\)−1/α−1\),qminmmax​\[Rα​\(m0\)1/α−1\]\},0\\leq\\varepsilon<\\min\\left\\\{\\frac\{q\_\{\\min\}\}\{m\_\{\\max\}\}\\left\(\(1\-\\eta\)^\{\-1/\\alpha\}\-1\\right\),\\;\\frac\{q\_\{\\min\}\}\{m\_\{\\max\}\}\\left\[R\_\{\\alpha\}\(m\_\{0\}\)^\{1/\\alpha\}\-1\\right\]\\right\\\},\(24\)then the shifted weighted loss simultaneously satisfies:

1. 1\.approximate scale invariance within relative toleranceη\\eta: 0≤IF​P​\(α,m\)−w​\(q\)​Bε​\(q∣m​q\)IF​P​\(α,m\)≤η,0\\leq\\frac\{I\_\{FP\}\(\\alpha,m\)\-w\(q\)B\_\{\\varepsilon\}\(q\\mid mq\)\}\{I\_\{FP\}\(\\alpha,m\)\}\\leq\\eta,and 0≤IF​N​\(α,m\)−w​\(q\)​Bε​\(q∣q/m\)IF​N​\(α,m\)≤η;0\\leq\\frac\{I\_\{FN\}\(\\alpha,m\)\-w\(q\)B\_\{\\varepsilon\}\(q\\mid q/m\)\}\{I\_\{FN\}\(\\alpha,m\)\}\\leq\\eta;
2. 2\.finite\-error asymmetric sensitivity: Bε​\(q∣q/m\)\>Bε​\(q∣m​q\)\.B\_\{\\varepsilon\}\(q\\mid q/m\)\>B\_\{\\varepsilon\}\(q\\mid mq\)\.

Both guarantees hold uniformly forq∈\[qmin,qmax\]q\\in\[q\_\{\\min\},q\_\{\\max\}\]andm∈\[m0,mmax\]m\\in\[m\_\{0\},m\_\{\\max\}\]\.

###### Proof\.

The first term in the minimum in \([24](https://arxiv.org/html/2607.18454#A2.E24)\) is exactly the sufficient condition of Theorem[1](https://arxiv.org/html/2607.18454#Thmtheorem1)\. The second term is exactly the sufficient condition of Theorem[2](https://arxiv.org/html/2607.18454#Thmtheorem2)\. Therefore both conclusions hold simultaneously\. ∎

### B\.5Interpretation of the shift bound

Settingε=0\\varepsilon=0gives exact multiplicative scale invariance and, forα\>3/2\\alpha\>3/2, strict FN\-over\-FP asymmetry\. However, the unshifted loss can be infinite when a model predictsp=0p=0\. Introducingε\>0\\varepsilon\>0removes this singularity, but it also introduces an additive floor into the local weight\(t\+ε\)−α\(t\+\\varepsilon\)^\{\-\\alpha\}\. Ifε\\varepsilonis too large relative toqminq\_\{\\min\}, this floor overwhelms the relevant probability scale and destroys both approximate scale invariance and asymmetric sensitivity\.

The bounds above quantify the permissible size ofε\\varepsilon\. They are sufficient rather than necessary, and are intentionally conservative\. A simple practical choice is to set

ε=c​qminmmax,c≪1,\\varepsilon=c\\frac\{q\_\{\\min\}\}\{m\_\{\\max\}\},\\qquad c\\ll 1,and then verify that this value lies below the theorem\-specific upper bounds\.

### B\.6Parameter Guidance

Table[2](https://arxiv.org/html/2607.18454#A2.T2)summarizes suggested\(α,γ\)\(\\alpha,\\gamma\)settings for the evaluation use cases referenced in Section[4\.1](https://arxiv.org/html/2607.18454#S4.SS1)\. The parameterγ\\gammacontrols any additional dataset\-level rarity premium, whileα\\alphacontrols the asymmetry between reciprocal overestimation and underestimation\. We now describe a more fine\-grained way to chooseα\\alphaby specifying a desired worst\-case FN/FP ratio\.

#### B\.6\.1Choosingα\\alphavia a worst\-case FN/FP ratio

A natural way to calibrate the asymmetry parameterα\\alphais to specify a desired lower boundr⋆\>1r\_\{\\star\}\>1on the cost of reciprocal underestimation relative to reciprocal overestimation\. In the unshifted geometry,

B0​\(q∣q/m\)B0​\(q∣m​q\)=IF​N​\(α,m\)IF​P​\(α,m\)=Rα​\(m\),\\frac\{B\_\{0\}\(q\\mid q/m\)\}\{B\_\{0\}\(q\\mid mq\)\}=\\frac\{I\_\{FN\}\(\\alpha,m\)\}\{I\_\{FP\}\(\\alpha,m\)\}=R\_\{\\alpha\}\(m\),so this ratio is independent of the base probabilityqq\. The same is true after multiplying both losses by any dataset\-level weight depending only onqq, since the weight cancels in the ratio\.

Such a guarantee cannot hold uniformly asm↓1m\\downarrow 1\. Indeed, for every fixedα\\alpha,

limm↓1Rα​\(m\)=1\.\\lim\_\{m\\downarrow 1\}R\_\{\\alpha\}\(m\)=1\.Thus no finiteα\\alphacan guaranteeRα​\(m\)≥r⋆\>1R\_\{\\alpha\}\(m\)\\geq r\_\{\\star\}\>1for arbitrarily small multiplicative errors\. A meaningful worst\-case calibration must therefore specify a minimum multiplicative errorm0\>1m\_\{0\}\>1\.

Fixm0\>1m\_\{0\}\>1andmmax≥m0m\_\{\\max\}\\geq m\_\{0\}\. We require

Rα​\(m\)≥r⋆∀m∈\[m0,mmax\]\.R\_\{\\alpha\}\(m\)\\geq r\_\{\\star\}\\qquad\\forall m\\in\[m\_\{0\},m\_\{\\max\}\]\.Forr⋆\>1r\_\{\\star\}\>1, any feasible choice must haveα\>3/2\\alpha\>3/2\. By Lemma[1](https://arxiv.org/html/2607.18454#Thmlemma1),Rα​\(m\)R\_\{\\alpha\}\(m\)is then increasing inmm, so the worst case occurs atm=m0m=m\_\{0\}\. Therefore, in the unshifted geometry, the uniform calibration condition overm∈\[m0,mmax\]m\\in\[m\_\{0\},m\_\{\\max\}\]is equivalent to

Rα​\(m0\)≥r⋆\.R\_\{\\alpha\}\(m\_\{0\}\)\\geq r\_\{\\star\}\.
Using the identityIF​N​\(α,m\)=IF​P​\(3−α,m\)I\_\{FN\}\(\\alpha,m\)=I\_\{FP\}\(3\-\\alpha,m\), we can write

Rα​\(m\)=IF​P​\(3−α,m\)IF​P​\(α,m\)\.R\_\{\\alpha\}\(m\)=\\frac\{I\_\{FP\}\(3\-\\alpha,m\)\}\{I\_\{FP\}\(\\alpha,m\)\}\.Here

IF​P​\(β,m\)=∫1m\(u−1\)​u−β​𝑑u=\{m2−β−12−β−m1−β−11−β,β∉\{1,2\},m−1−log⁡m,β=1,log⁡m\+1m−1,β=2\.I\_\{FP\}\(\\beta,m\)=\\int\_\{1\}^\{m\}\(u\-1\)u^\{\-\\beta\}\\,du=\\begin\{cases\}\\displaystyle\\frac\{m^\{2\-\\beta\}\-1\}\{2\-\\beta\}\-\\frac\{m^\{1\-\\beta\}\-1\}\{1\-\\beta\},&\\beta\\notin\\\{1,2\\\},\\\\\[12\.50002pt\] m\-1\-\\log m,&\\beta=1,\\\\\[5\.0pt\] \\log m\+\\frac\{1\}\{m\}\-1,&\\beta=2\.\\end\{cases\}
For fixedm0\>1m\_\{0\}\>1, the mapα↦Rα​\(m0\)\\alpha\\mapsto R\_\{\\alpha\}\(m\_\{0\}\)is strictly increasing\. Indeed, defining

F​\(β,m\)=IF​P​\(β,m\),F\(\\beta,m\)=I\_\{FP\}\(\\beta,m\),we have

∂βF​\(β,m\)=−∫1m\(u−1\)​u−β​log⁡u​d​u<0\.\\partial\_\{\\beta\}F\(\\beta,m\)=\-\\int\_\{1\}^\{m\}\(u\-1\)u^\{\-\\beta\}\\log u\\,du<0\.ThusF​\(β,m\)F\(\\beta,m\)is strictly decreasing inβ\\beta, and

Rα​\(m0\)=F​\(3−α,m0\)F​\(α,m0\)R\_\{\\alpha\}\(m\_\{0\}\)=\\frac\{F\(3\-\\alpha,m\_\{0\}\)\}\{F\(\\alpha,m\_\{0\}\)\}is strictly increasing inα\\alpha\. Moreover,

R3/2​\(m0\)=1,limα→∞Rα​\(m0\)=∞\.R\_\{3/2\}\(m\_\{0\}\)=1,\\qquad\\lim\_\{\\alpha\\to\\infty\}R\_\{\\alpha\}\(m\_\{0\}\)=\\infty\.Hence for everyr⋆\>1r\_\{\\star\}\>1there is a unique threshold

α⋆=α⋆​\(m0,r⋆\)\>3/2\\alpha\_\{\\star\}=\\alpha\_\{\\star\}\(m\_\{0\},r\_\{\\star\}\)\>3/2satisfying

Rα⋆​\(m0\)=r⋆\.R\_\{\\alpha\_\{\\star\}\}\(m\_\{0\}\)=r\_\{\\star\}\.Choosingα≥α⋆\\alpha\\geq\\alpha\_\{\\star\}guarantees

minm∈\[m0,mmax\]⁡Rα​\(m\)≥r⋆\\min\_\{m\\in\[m\_\{0\},m\_\{\\max\}\]\}R\_\{\\alpha\}\(m\)\\geq r\_\{\\star\}for the unshifted loss\.

##### Computingα⋆\\alpha\_\{\\star\}\.

The thresholdα⋆\\alpha\_\{\\star\}generally has no closed\-form inverse, but it can be computed by bisection becauseα↦Rα​\(m0\)\\alpha\\mapsto R\_\{\\alpha\}\(m\_\{0\}\)is continuous and strictly increasing\. A practical procedure is:

1. 1\.Choose the minimum relevant multiplicative errorm0\>1m\_\{0\}\>1, the desired worst\-case ratior⋆\>1r\_\{\\star\}\>1, and a numerical toleranceτ\>0\\tau\>0\.
2. 2\.Setαlo=32\.\\alpha\_\{\\mathrm\{lo\}\}=\\frac\{3\}\{2\}\.ThenRαlo​\(m0\)=1<r⋆\.R\_\{\\alpha\_\{\\mathrm\{lo\}\}\}\(m\_\{0\}\)=1<r\_\{\\star\}\.
3. 3\.Choose an initial upper bracketαhi\>3/2\\alpha\_\{\\mathrm\{hi\}\}\>3/2\. Increase it, for example by repeatedly doubling,αhi←2​αhi,\\alpha\_\{\\mathrm\{hi\}\}\\leftarrow 2\\alpha\_\{\\mathrm\{hi\}\},untilRαhi​\(m0\)≥r⋆\.R\_\{\\alpha\_\{\\mathrm\{hi\}\}\}\(m\_\{0\}\)\\geq r\_\{\\star\}\.
4. 4\.Whileαhi−αlo\>τ,\\alpha\_\{\\mathrm\{hi\}\}\-\\alpha\_\{\\mathrm\{lo\}\}\>\\tau,setαmid=αlo\+αhi2\.\\alpha\_\{\\mathrm\{mid\}\}=\\frac\{\\alpha\_\{\\mathrm\{lo\}\}\+\\alpha\_\{\\mathrm\{hi\}\}\}\{2\}\.IfRαmid​\(m0\)<r⋆,R\_\{\\alpha\_\{\\mathrm\{mid\}\}\}\(m\_\{0\}\)<r\_\{\\star\},setαlo←αmid\.\\alpha\_\{\\mathrm\{lo\}\}\\leftarrow\\alpha\_\{\\mathrm\{mid\}\}\.Otherwise setαhi←αmid\.\\alpha\_\{\\mathrm\{hi\}\}\\leftarrow\\alpha\_\{\\mathrm\{mid\}\}\.
5. 5\.Returnαhi\\alpha\_\{\\mathrm\{hi\}\}\. This gives a conservative numerical choice satisfyingRαhi​\(m0\)≥r⋆\.R\_\{\\alpha\_\{\\mathrm\{hi\}\}\}\(m\_\{0\}\)\\geq r\_\{\\star\}\.

##### Effect of the shift\.

For the shifted loss, the exact unshifted ratio no longer holds\. However, the proof of Theorem[2](https://arxiv.org/html/2607.18454#Thmtheorem2)gives the conservative lower bound

Bε​\(q∣q/m\)Bε​\(q∣m​q\)≥\(1\+m​ε/q\)−α​Rα​\(m\)\.\\frac\{B\_\{\\varepsilon\}\(q\\mid q/m\)\}\{B\_\{\\varepsilon\}\(q\\mid mq\)\}\\geq\(1\+m\\varepsilon/q\)^\{\-\\alpha\}R\_\{\\alpha\}\(m\)\.Therefore, uniformly overq∈\[qmin,qmax\]q\\in\[q\_\{\\min\},q\_\{\\max\}\]andm∈\[m0,mmax\]m\\in\[m\_\{0\},m\_\{\\max\}\],

Bε​\(q∣q/m\)Bε​\(q∣m​q\)≥\(1\+mmax​εqmin\)−α​Rα​\(m0\)\.\\frac\{B\_\{\\varepsilon\}\(q\\mid q/m\)\}\{B\_\{\\varepsilon\}\(q\\mid mq\)\}\\geq\\left\(1\+\\frac\{m\_\{\\max\}\\varepsilon\}\{q\_\{\\min\}\}\\right\)^\{\-\\alpha\}R\_\{\\alpha\}\(m\_\{0\}\)\.Thus a sufficient condition for the shifted loss to achieve worst\-case ratio at leastr⋆r\_\{\\star\}is

\(1\+mmax​εqmin\)−α​Rα​\(m0\)≥r⋆\.\\left\(1\+\\frac\{m\_\{\\max\}\\varepsilon\}\{q\_\{\\min\}\}\\right\)^\{\-\\alpha\}R\_\{\\alpha\}\(m\_\{0\}\)\\geq r\_\{\\star\}\.Equivalently, after choosingα\\alpha, it is sufficient to choose

ε≤qminmmax​\[\(Rα​\(m0\)r⋆\)1/α−1\]\.\\varepsilon\\leq\\frac\{q\_\{\\min\}\}\{m\_\{\\max\}\}\\left\[\\left\(\\frac\{R\_\{\\alpha\}\(m\_\{0\}\)\}\{r\_\{\\star\}\}\\right\)^\{1/\\alpha\}\-1\\right\]\.If a strict ratio\>r⋆\>r\_\{\\star\}is desired, use a strict inequality\. This bound is meaningful for positiveε\\varepsilononly when

Rα​\(m0\)\>r⋆\.R\_\{\\alpha\}\(m\_\{0\}\)\>r\_\{\\star\}\.Consequently, if one intends to use a positive shiftε\>0\\varepsilon\>0, one should chooseα\\alphawith some margin above the unshifted thresholdα⋆\\alpha\_\{\\star\}, rather than choosing exactlyα=α⋆\\alpha=\\alpha\_\{\\star\}\. Theorem[2](https://arxiv.org/html/2607.18454#Thmtheorem2)corresponds to the special caser⋆=1r\_\{\\star\}=1\.

Table 2:Suggested shifted\-power metric parameters by evaluation use case\. Hereα\\alphacontrols FN\-vs\-FP asymmetry \(higherα\\alphapenalizes underestimation more\), andγ\\gammacontrols dataset\-level rarity premium \(higherγ\\gammaupweights smaller ground\-truth probabilitiesqq\)\.

### B\.7Computation: closed form

Closed\-formBε​\(q∣p\)B\_\{\\varepsilon\}\(q\\mid p\)\.Forα≠1,2\\alpha\\neq 1,2, the shifted power family admits the closed form

Bε​\(q∣p\)=\(p\+ε\)2−α−\(q\+ε\)2−α2−α−\(ε\+q\)​\[\(p\+ε\)1−α−\(q\+ε\)1−α\]1−α\.B\_\{\\varepsilon\}\(q\\mid p\)=\\frac\{\(p\+\\varepsilon\)^\{2\-\\alpha\}\-\(q\+\\varepsilon\)^\{2\-\\alpha\}\}\{2\-\\alpha\}\-\\frac\{\(\\varepsilon\+q\)\\left\[\(p\+\\varepsilon\)^\{1\-\\alpha\}\-\(q\+\\varepsilon\)^\{1\-\\alpha\}\\right\]\}\{1\-\\alpha\}\.\(25\)Forα=1\\alpha=1:

Bε​\(q∣p\)=\(p−q\)−\(ε\+q\)​log⁡\(p\+εq\+ε\)\.B\_\{\\varepsilon\}\(q\\mid p\)=\(p\-q\)\-\(\\varepsilon\+q\)\\log\\\!\\left\(\\frac\{p\+\\varepsilon\}\{q\+\\varepsilon\}\\right\)\.\(26\)Forα=2\\alpha=2:

Bε​\(q∣p\)=log⁡\(p\+εq\+ε\)\+\(ε\+q\)​\(1p\+ε−1q\+ε\)\.B\_\{\\varepsilon\}\(q\\mid p\)=\\log\\\!\\left\(\\frac\{p\+\\varepsilon\}\{q\+\\varepsilon\}\\right\)\+\(\\varepsilon\+q\)\\left\(\\frac\{1\}\{p\+\\varepsilon\}\-\\frac\{1\}\{q\+\\varepsilon\}\\right\)\.\(27\)

### B\.8Connections to standard losses

For the unshifted familyω0​\(t\)=t−α\\omega\_\{0\}\(t\)=t^\{\-\\alpha\}, the divergence is

B0​\(q∣p\)=∫qp\(t−q\)​t−α​𝑑t\.B\_\{0\}\(q\\mid p\)=\\int\_\{q\}^\{p\}\(t\-q\)t^\{\-\\alpha\}\\,dt\.
Forα=0\\alpha=0,

B0​\(q∣p\)=∫qp\(t−q\)​𝑑t=\(p−q\)22,B\_\{0\}\(q\\mid p\)=\\int\_\{q\}^\{p\}\(t\-q\)\\,dt=\\frac\{\(p\-q\)^\{2\}\}\{2\},so the family recovers squared error up to a constant factor1/21/2\.

Forα=2\\alpha=2,

B0​\(q∣p\)=∫qp\(1t−qt2\)​𝑑t=log⁡pq\+qp−1=qp−log⁡qp−1,B\_\{0\}\(q\\mid p\)=\\int\_\{q\}^\{p\}\\left\(\\frac\{1\}\{t\}\-\\frac\{q\}\{t^\{2\}\}\\right\)\\,dt=\\log\\frac\{p\}\{q\}\+\\frac\{q\}\{p\}\-1=\\frac\{q\}\{p\}\-\\log\\frac\{q\}\{p\}\-1,which is exactly the Itakura–Saito divergenceDIS​\(q,p\)D\_\{\\mathrm\{IS\}\}\(q,p\)\.

### B\.9Empirical verification of SPB scaling behavior

Figure[7](https://arxiv.org/html/2607.18454#A2.F7)contrasts the scale behavior of the metrics for a fixed relative error across the ground\-truth range\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/Scale_dependence_across_metrics.png)Figure 7:Scale dependence of each metric for a fixed relative error \(overestimatep=10​qp=10q, left; underestimatep=q/10p=q/10, right\) across ground\-truth probabilitiesq∈\[10−9,10−5\]q\\in\[10^\{\-9\},10^\{\-5\}\]\. Squared error varies by orders of magnitude withqq, while IS loss, log\-space squared error, and the weighted shifted\-power loss are flat \(scale invariant\)\.

## Appendix CPseudocode

Algorithm[1](https://arxiv.org/html/2607.18454#alg1)gives the full GA\-AMLS procedure described in Section[3](https://arxiv.org/html/2607.18454#S3)\.

Algorithm 1GA\-AMLS with MALA moves and a Student\-ttprior1:Input distribution

𝒟\\mathcal\{D\}, model

MM, target token

tt, population size

NN, quantile level

ρ\\rho, target score

τ\\tau, MALA step size

hh, MCMC steps

TT
2:Probability estimate

P^\\widehat\{P\}
3:Draw calibration inputs from

DDand compute raw activations

\{aj\}j=1ncal\\\{a\_\{j\}\\\}\_\{j=1\}^\{n\_\{\\mathrm\{cal\}\}\}at the chosen site\.

4:Estimate

μ,Σ\\mu,\\Sigma, choose

AAsuch that

A​A⊤=ΣAA^\{\\top\}=\\Sigma, and transform activations by

uj←\(aj−μ\)​A−⊤\.u\_\{j\}\\leftarrow\(a\_\{j\}\-\\mu\)A^\{\-\\top\}\.
5:Fit the prior in whitened coordinates as an independent Student\-

ttdistribution

π​\(u\)=∏d=1Dtν=5​\(ud;0,1\)\.\\pi\(u\)=\\prod\_\{d=1\}^\{D\}t\_\{\\nu=5\}\(u\_\{d\};0,1\)\.
6:Define the score

s​\(u\)s\(u\)by inserting the unwhitened activation

a​\(u\)=u​A⊤\+μa\(u\)=uA^\{\\top\}\+\\muinto the model\.

7:Initialize particles

\{ui\}i=1N\\\{u\_\{i\}\\\}\_\{i=1\}^\{N\}by sampling from the whitened calibration activations\.

8:

k←0k\\leftarrow 0,

L0←−∞L\_\{0\}\\leftarrow\-\\infty,

P^←1\\widehat\{P\}\\leftarrow 1
9:while

Lk<τL\_\{k\}<\\taudo

10:Compute scores

\{s​\(ui\)\}i=1N\\\{s\(u\_\{i\}\)\\\}\_\{i=1\}^\{N\}\.

11:Set the next level

Lk\+1←min⁡\{τ,Quantileρ⁡\(\{s​\(ui\)\}i=1N\)\}\.L\_\{k\+1\}\\leftarrow\\min\\left\\\{\\tau,\\operatorname\{Quantile\}\_\{\\rho\}\\left\(\\\{s\(u\_\{i\}\)\\\}\_\{i=1\}^\{N\}\\right\)\\right\\\}\.
12:Estimate the conditional survival probability

p^k←1N​∑i=1N𝟏​\{s​\(ui\)≥Lk\+1\},P^←P^​p^k\.\\widehat\{p\}\_\{k\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{s\(u\_\{i\}\)\\geq L\_\{k\+1\}\\\},\\qquad\\widehat\{P\}\\leftarrow\\widehat\{P\}\\widehat\{p\}\_\{k\}\.
13:Keep the survivors

𝒮←\{ui:s​\(ui\)≥Lk\+1\}\\mathcal\{S\}\\leftarrow\\\{u\_\{i\}:s\(u\_\{i\}\)\\geq L\_\{k\+1\}\\\}and resample with replacement from

𝒮\\mathcal\{S\}until the population again has size

NN\.

14:for

r=1,…,Tr=1,\\dots,Tdo

15:foreach particle

uiu\_\{i\}do

16:Compute the prior score gradient

gi←∇ulog⁡π​\(ui\)\.g\_\{i\}\\leftarrow\\nabla\_\{u\}\\log\\pi\(u\_\{i\}\)\.
17:Propose a MALA move

ui′←ui\+h​gi\+2​h​ξi,ξi∼𝒩​\(0,I\)\.u\_\{i\}^\{\\prime\}\\leftarrow u\_\{i\}\+hg\_\{i\}\+\\sqrt\{2h\}\\,\\xi\_\{i\},\\qquad\\xi\_\{i\}\\sim\\mathcal\{N\}\(0,I\)\.
18:if

s​\(ui′\)<Lk\+1s\(u\_\{i\}^\{\\prime\}\)<L\_\{k\+1\}then

19:Reject the proposal\.

20:else

21:Compute

gi′←∇ulog⁡π​\(ui′\)g\_\{i\}^\{\\prime\}\\leftarrow\\nabla\_\{u\}\\log\\pi\(u\_\{i\}^\{\\prime\}\)and proposal densities

q​\(ui′∣ui\)=𝒩​\(ui′;ui\+h​gi,2​h​I\),q​\(ui∣ui′\)=𝒩​\(ui;ui′\+h​gi′,2​h​I\)\.q\(u\_\{i\}^\{\\prime\}\\mid u\_\{i\}\)=\\mathcal\{N\}\(u\_\{i\}^\{\\prime\};\\,u\_\{i\}\+hg\_\{i\},\\,2hI\),\\qquad q\(u\_\{i\}\\mid u\_\{i\}^\{\\prime\}\)=\\mathcal\{N\}\(u\_\{i\};\\,u\_\{i\}^\{\\prime\}\+hg\_\{i\}^\{\\prime\},\\,2hI\)\.
22:Accept

ui′u\_\{i\}^\{\\prime\}with probability

ai=min⁡\{1,π​\(ui′\)​q​\(ui∣ui′\)π​\(ui\)​q​\(ui′∣ui\)\}\.a\_\{i\}=\\min\\left\\\{1,\\,\\frac\{\\pi\(u\_\{i\}^\{\\prime\}\)q\(u\_\{i\}\\mid u\_\{i\}^\{\\prime\}\)\}\{\\pi\(u\_\{i\}\)q\(u\_\{i\}^\{\\prime\}\\mid u\_\{i\}\)\}\\right\\\}\.
23:endif

24:endfor

25:Adapt

hhduring burn\-in toward the MALA target acceptance rate\.

26:endfor

27:

k←k\+1k\\leftarrow k\+1
28:endwhile

29:return

P^\\widehat\{P\}

## Appendix DExperimental and Implementation Details

### D\.1Hyperparameters

##### MAPLA tuning candidate\.

In the tuning sweep we also considered MAPLA, the Metropolis\-adjusted Preconditioned Langevin Algorithm\(Srinivasanet al\.,[2025](https://arxiv.org/html/2607.18454#bib.bib138)\)\. MAPLA generalizes MALA by replacing the isotropic proposal covariance with a position\-dependent metric\. For a target densityπ\\pi, its proposal has the form

u′∼𝒩​\(u\+h​G​\(u\)−1​∇ulog⁡π​\(u\),2​h​G​\(u\)−1\),u^\{\\prime\}\\sim\\mathcal\{N\}\\\!\\left\(u\+hG\(u\)^\{\-1\}\\nabla\_\{u\}\\log\\pi\(u\),2hG\(u\)^\{\-1\}\\right\),followed by the usual Metropolis–Hastings correction\. In our AMLS setting this kernel is applied to the constrained target

πk​\(u\)∝π​\(u\)​𝟏​\{s​\(u\)≥Lk\}\.\\pi\_\{k\}\(u\)\\propto\\pi\(u\)\\mathbf\{1\}\\\{s\(u\)\\geq L\_\{k\}\\\}\.We used MAPLA only as a hyperparameter\-tuning candidate: although it can reduce rejections near level\-set boundaries by adapting proposal geometry, its extra metric computations did not yield a sufficient improvement over MALA under a matched compute budget\. Therefore all main GA\-AMLS results use MALA\.

Tuning protocol\.LikeWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\), to prevent overfitting our method was only run on the first four distributions during development, and architectural choices were finalized before testing on the last four distributions\. We chose the MH kernel and the prior as those which minimized squared error loss and IS loss on 100 randomly chosen tokens with ground\-truth probabilities in the range\[10−5,10−3\]\[10^\{\-5\},10^\{\-3\}\]; probabilities in this range are directly estimable by naive Monte Carlo with a modest budget, so tuning does not require privileged knowledge of the extreme tail\. The selected prior and kernel were consistent across metrics, distributions𝒟\\mathcal\{D\}, and model sizes\. Both MALA and MAPLA outperformed the random\-walk kernel \(RWMH\), and the Student\-ttprior outperformed a Gaussian prior; since MAPLA’s higher computational cost did not yield a commensurate performance gain over MALA, we chose the MALA kernel\. To ensure a fair comparison across MH kernels, we equalized the total computational budget per rejuvenation step: since MALA and MAPLA require gradient computations, we allocate 1500 MH steps to RWMH and 750 steps to the gradient\-based kernels\.

GA\-AMLS configurationAll GA\-AMLS estimates are from a single run per \(token, distribution, model\)\. We use a population ofN=2000N=2000particles, threshold quantileρ=0\.6\\rho=0\.6,T=900T=900MALA steps per level with a burn\-in ofTburn=126T\_\{\\mathrm\{burn\}\}=126steps and initial step sizeh0=10−3h\_\{0\}=10^\{\-3\}, and a diagonal Student\-ttprior withν=5\\nu=5degrees of freedom fit onncal=216=65,536n\_\{\\mathrm\{cal\}\}=2^\{16\}=65\{,\}536calibration activations drawn from𝒟\\mathcal\{D\}\. The calibration activations are generated once per \(model,D\) pair and reused across target tokens\.

Baselines\.For the importance\-sampling baselines ITGIS and MHIS we used the temperatures tuned byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\.We identified and fixed a numerical\-tolerance bug in the public QLD implementation: in the randomized projection routine for finding the shortest accepting vector, the tolerance was meant to ignore only slightly negative constraint violationsP​x\+b≥−tolPx\+b\\geq\-\\texttt\{tol\}, but the original code usedP​x\+b<tolPx\+b<\\texttt\{tol\}, incorrectly marking feasible near\-boundary constraints as violated\.; all QLD results in this paper use the corrected implementation, which improves QLD’s performance relative to the numbers reported byWu and Hilton \([2025](https://arxiv.org/html/2607.18454#bib.bib37)\)\.

Ablation studies\.Figures[8](https://arxiv.org/html/2607.18454#A4.F8)–[11](https://arxiv.org/html/2607.18454#A4.F11)report the per\-distribution SPB loss for every kernel–prior–hyperparameter configuration explored during tuning, under each of the four evaluation profiles of Table[2](https://arxiv.org/html/2607.18454#A2.T2)\. Two patterns hold consistently across distributions and model sizes: \(i\) the MALA kernel outperforms RWMH despite using half as many MH proposal steps, and \(ii\) the Student\-ttprior outperforms a Gaussian prior\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/shifted_power_bregman_product_monitoring_logx.png)Figure 8:Hyperparameter sweep under the product\-monitoring evaluation profile \(α=2\\alpha=2,γ=0\\gamma=0,ε=2\.073×10−12\\varepsilon=2\.073\\times 10^\{\-12\}\)\. Bars show average SPB loss \(±1\.28\\pm 1\.28SEM, log scale\) for each kernel–prior–hyperparameter configuration, per input distribution \(rows\) and model size \(columns\)\. Configuration names encode kernel, prior, activation site, and tuning parameters\.![Refer to caption](https://arxiv.org/html/2607.18454v1/shifted_power_bregman_deep_tail_stress_testing_logx.png)Figure 9:Hyperparameter sweep under the deep\-tail stress\-testing profile \(α=2\\alpha=2,γ=1\\gamma=1\); format as in Figure[8](https://arxiv.org/html/2607.18454#A4.F8)\.![Refer to caption](https://arxiv.org/html/2607.18454v1/shifted_power_bregman_unbiased_method_benchmarking_logx.png)Figure 10:Hyperparameter sweep under the unbiased method benchmarking profile \(α=1\.5\\alpha=1\.5,γ=0\\gamma=0\); format as in Figure[8](https://arxiv.org/html/2607.18454#A4.F8)\.![Refer to caption](https://arxiv.org/html/2607.18454v1/shifted_power_bregman_catastrophic_risk_audit_logx.png)Figure 11:Hyperparameter sweep under the catastrophic\-risk audit profile \(α=4\\alpha=4,γ=2\\gamma=2\); format as in Figure[8](https://arxiv.org/html/2607.18454#A4.F8)\.

### D\.2Activation Whitening Diagnostic

Figure[12](https://arxiv.org/html/2607.18454#A4.F12)plots the held\-out marginal variance of each pre\-LayerNorm, pre\-unembedding activation dimension, sorted by variance, before and after whitening; the whitening transform used by GA\-AMLS substantially flattens this marginal variance profile\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/anisotropy_fixed.png)Figure 12:Marginal activation variances before and after whitening\.Letai∈ℝda\_\{i\}\\in\\mathbb\{R\}^\{d\}denote held\-out pre\-LayerNorm, pre\-unembedding activations\. For each model size and input distribution, we compute the empirical marginal varianceVari​\(ai​j\)\\mathrm\{Var\}\_\{i\}\(a\_\{ij\}\)of every activation dimensionjj, and repeat the calculation after applying the whitening transform used by GA\-AMLS\. For visualization, dimensions are sorted from largest to smallest marginal variance within each distribution; the x\-axis is therefore the sorted dimension rank\. Each panel reports the median sorted curve across the eight input distributions, with shaded bands showing the interquartile range\. Raw activations exhibit strong marginal anisotropy, while whitening produces a much flatter variance profile, making isotropic MALA proposals in whitened coordinates more reasonable\.

## Appendix EScatterplots

Figures[13](https://arxiv.org/html/2607.18454#A5.F13)–[15](https://arxiv.org/html/2607.18454#A5.F15)show estimates against ground truth for all four methods on all 8 input distributions, for each model size, complementing the two examples in Figure[4](https://arxiv.org/html/2607.18454#S5.F4)\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/Scatter1L.png)Figure 13:Estimates vs\. ground truth for all methods across all 8 input distributions \(1\-layer model\)\. MHIS/ITGIS zero estimates are plotted as crosses at the bottom of each panel for visibility\.![Refer to caption](https://arxiv.org/html/2607.18454v1/Scatter2L.png)Figure 14:As Figure[13](https://arxiv.org/html/2607.18454#A5.F13), for the 2\-layer model\.![Refer to caption](https://arxiv.org/html/2607.18454v1/Scatter4L.png)Figure 15:As Figure[13](https://arxiv.org/html/2607.18454#A5.F13), for the 4\-layer model\.
## Appendix FAll Methods Loss by Distributions

Figure[16](https://arxiv.org/html/2607.18454#A6.F16)breaks the aggregate SPB results down by input distribution, Figure[17](https://arxiv.org/html/2607.18454#A6.F17)reports the aggregate SPB loss across model sizes for all four\(α,γ\)\(\\alpha,\\gamma\)evaluation profiles, and Tables[3](https://arxiv.org/html/2607.18454#A6.T3)and[4](https://arxiv.org/html/2607.18454#A6.T4)report per\-distribution SPB loss\.

![Refer to caption](https://arxiv.org/html/2607.18454v1/corrected_dist.png)Figure 16:SPB loss across individual distributions and parameter configurations\. GA\-AMLS \(blue\) consistently outperforms baseline methods across 7 of the 8 text distributions under a symmetric penalty \(α=1\.5\\alpha=1\.5\), demonstrating that aggregate improvements are not skewed by outliers\. As the penalty for underestimation increases \(α\>1\.5\\alpha\>1\.5\), QLD \(orange\) gains an advantage due to its systematic overestimation bias\. Lower is better; error bars denote standard error across the three model sizes\.![Refer to caption](https://arxiv.org/html/2607.18454v1/corrected_layers.png)Figure 17:The SPB loss of all methods across different model sizes and parameter combinations\.The solid lines indicate the loss of each method averaged over all 8 distributions, with bands showing standard error\. The colored points indicate the loss on individual distributions, with horizontal jitter added for visibility\. Lower is better\. We see that GA\-AMLS \(blue\) outperforms when the penalty for FN is approximately the same as FP, and when the weight on the rarest events increases\.Table 3:SPB loss comparison across model sizes \(α=1\.5\\alpha=1\.5,γ=0\\gamma=0,ε=εcorr\\varepsilon=\\varepsilon\_\{\\mathrm\{corr\}\}\)\. Lower is better\.\(a\)1\-layer model
\(b\)2\-layer model
\(c\)4\-layer model

Table 4:SPB loss comparison across model sizes \(α=2\\alpha=2,γ=0\\gamma=0,ε=εcorr\\varepsilon=\\varepsilon\_\{\\mathrm\{corr\}\}\)\. Lower is better\.\(a\)1\-layer model
\(b\)2\-layer model
\(c\)4\-layer model