单例稀缺性的重要性及补充材料

# Unseen-Species and Missing-Mass Estimation in Small-Sample Regimes
Source: [https://arxiv.org/html/2610.22604](https://arxiv.org/html/2610.22604)
## Rarity of Singletons Matters: Unseen\-Species and Missing\-Mass Estimation in Small\-Sample Regimes Thanks:This work was supported by the French/US joint project LUCAS between the Agence Nationale de la Recherche \(ANR\) \(grant \#ANR\-25\-CE23\-2189\) and the U\.S\. National Science Foundation \(NSF\) \(grant \#s 2530255 and 2530256\)\.

Pragatheeswaran VipulanandanAffiliation:Dept\. of Elect\. and Comp\. Engineering University of Miami Coral Gables, FL 33146, USA pxv245@miami\.eduKamal PremaratneAffiliation:Dept\. of Elect\. and Comp\. Engineering University of Miami Coral Gables, FL 33146, USA kamal@miami\.eduDilip SarkarAffiliation:Dept\. of Comp\. Science University of Miami Coral Gables, FL 33146, USA sarkar@miami\.edu

###### Abstract

Estimating missing\-mass and unseen\-species counts from finite samples relies critically on rare observations, particularly singletons\. Yet in small\-sample regime, these estimators can be highly*fragile*in the sense that they may undergo large changes with even modest sampling fluctuations\. We introduce a framework that explicitly accounts for this fragility and uses it to adaptively regularize standard missing\-mass and unseen\-species estimators\. The resulting fragility\-aware formulation provides a principled correction that becomes stronger precisely when the underlying rare\-event statistics are least reliable\. The entropy\-KL optimization is strictly concave, admits a closed\-form solution, requires no Poissonization, and can be applied as a lightweight correction to classical estimators such as Good\-Turing and Good\-Toulmin\. Across 360 experiments spanning synthetic distributions and language corpora, our fragility\-aware estimators remain competitive in stable regimes while consistently improving stability in long\-tailed, singleton\-dominated, small\-sample regimes, with the largest gains occurring precisely where classical estimators are most fragile\.

###### Index Terms:

Unseen events; missing\-mass; fragility\.

## IIntroduction

A central question in statistical learning is how well a finite sample represents an underlying discrete distribution\. information about the unobserved plays a critical role in addressing this question\. In particular, the*unseen\-species*count \(i\.e\., the number of species yet undiscovered\) and the*missing\-mass*\(i\.e\., the probability mass associated with unseen\-species\) characterize sample coverage, generalization risk, and the likelihood of encountering new outcomes; in turn, they inform decisions regarding whether available data are sufficient or whether further sampling is warranted\[[1](https://arxiv.org/html/2610.22604#as1_bib.bib1),[2](https://arxiv.org/html/2610.22604#as1_bib.bib2)\]\.

### I\-APrevious Work

Estimation of unseen\-species and missing\-mass dates to\[[3](https://arxiv.org/html/2610.22604#as1_bib.bib3),[4](https://arxiv.org/html/2610.22604#as1_bib.bib4)\], with later work addressing development of estimators and performance guarantees under a variety of sampling models\[[1](https://arxiv.org/html/2610.22604#as1_bib.bib1),[5](https://arxiv.org/html/2610.22604#as1_bib.bib5),[6](https://arxiv.org/html/2610.22604#as1_bib.bib6),[7](https://arxiv.org/html/2610.22604#as1_bib.bib7)\]\. Much of this work exploits occupancy statistics \(particularly the frequencies of rare occurrences\) to infer unseen\-species and missing\-mass\. More recent results have looked at the information that these occupancy statistics carry and the fundamental limits of extrapolation from finite sample regimes\[[5](https://arxiv.org/html/2610.22604#as1_bib.bib5),[6](https://arxiv.org/html/2610.22604#as1_bib.bib6),[8](https://arxiv.org/html/2610.22604#as1_bib.bib8)\]\.

A practical issue however remains: in small or highly fragmented samples, the occupancy statistic of rare\-events \(i\.e\., singletons and doubletons\) these estimators rely on can themselves be highly variable or*fragile*in the sense that modest sampling fluctuations can produce substantial changes in missing\-mass and unseen\-species estimates\[[9](https://arxiv.org/html/2610.22604#as1_bib.bib9)\]\. This issue is particularly relevant in applications where only limited observations are available and sampling variability can dominate asymptotic behavior\[[10](https://arxiv.org/html/2610.22604#as1_bib.bib10)\]\. Thus, beyond estimating the unseen\-species and missing\-mass, there is a need to quantify their*fragility,*i\.e\., their sensitivity to sampling variability\.

Contributions\.We introduce a fragility\-aware \(FA\) framework for missing\-mass and unseen\-species estimation\. While only the main ideas and results are presented below, more detailed analysis, derivations, proofs, and additional experiments appear in\[[11](https://arxiv.org/html/2610.22604#as1_bib.bib11)\]\. The principal contributions are:

1. 1\.a singleton\-based diagnostic that we call the*fragility measure*to quantify when small\-sample estimators are sensitive to limited or fragmented data or observations;
2. 2\.a concave*entropy\-KL correction*of a base estimator and closed\-form estimators for both unseen\-species count and their missing\-mass;
3. 3\.*theoretical analysis*to characterize the expected fragility under multinomial sampling using a delta\-method approximation; and
4. 4\.extensive*empirical validation*of our proposed framework across 360 experiments spanning synthetic distributions and language corpora, demonstrating improved stability in highly fragmented sampling regimes\.

## IIMethodology

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/pipeline2.jpg)Fig\. 1:Overview of our fragility\-aware estimation pipeline\.### II\-APreliminaries

Take a set of species𝒮=\{sℓ:1≤ℓ≤S\}\\mathcal\{S\}=\\\{s\_\{\\ell\}:1\\leq\\ell\\leq S\\\}having the population frequencies\{pℓ\}ℓ=1S\\\{p\_\{\\ell\}\\\}\_\{\\ell=1\}^\{S\}\. Given annn\-length samplex¯\(n\)=\{x1,…,xn\}\\underline\{x\}^\{\(n\)\}=\\\{x\_\{1\},\\ldots,x\_\{n\}\\\}, the*occupancy*of speciessℓs\_\{\\ell\}\(number of times speciessℓs\_\{\\ell\}occurs\) is

Nℓ\(n\)=∑i=1n𝟙​\(Xi=sℓ\)\.N^\{\(n\)\}\_\{\\ell\}=\\sum\_\{i=1\}^\{n\}\\mathbb\{1\}\(X\_\{i\}=s\_\{\\ell\}\)\.\(1\)The number of*kk\-occupancies*\(number of distinct species occurring exactlykktimes\) and the number of*observed species*\(distinct species observed at least once\) are, respectively,

Φk\(n\)=∑sℓ∈𝒮\(Nℓ\(n\)=k\);Φobs\(n\)=∑ℓ𝟙​\(Nℓ\(n\)\>0\)\.\\Phi^\{\(n\)\}\_\{k\}=\\sum\_\{s\_\{\\ell\}\\in\\mathcal\{S\}\}\\mathbb\{1\}\\\!\\bigl\(N^\{\(n\)\}\_\{\\ell\}=k\\bigr\);\\;\\;\\Phi\_\{\\mathrm\{obs\}\}^\{\(n\)\}=\\sum\_\{\\ell\}\\mathbb\{1\}\(N\_\{\\ell\}^\{\(n\)\}\>0\)\.\(2\)Then,Φ0\(n\)\\Phi\_\{0\}^\{\(n\)\}is the number of missing species andΦ1\(n\)\\Phi\_\{1\}^\{\(n\)\}the number of singleton species\.

Henceforth, unless needed, we will drop the superscript\(n\)\(n\)\.

Missing\-Mass\.*Probability mass ofkk\-occupancy*and*probability mass of observed species*are\[[3](https://arxiv.org/html/2610.22604#as1_bib.bib3)\], respectively,

Mk=∑sℓ∈𝒮pℓ​1​\(Nℓ=k\);Mobs=∑sℓ∈𝒮pℓ​1​\(Nℓ\>0\)\.M\_\{k\}\\\!=\\\!\\sum\_\{s\_\{\\ell\}\\in\\mathcal\{S\}\}p\_\{\\ell\}\\,\\mathbb\{1\}\(N\_\{\\ell\}=k\);\\;\\;M\_\{\\mathrm\{obs\}\}\\\!=\\\!\\sum\_\{s\_\{\\ell\}\\in\\mathcal\{S\}\}p\_\{\\ell\}\\,\\mathbb\{1\}\(N\_\{\\ell\}\>0\)\.\(3\)Then,M0M\_\{0\}is the*missing\-mass,*M1M\_\{1\}the*singleton mass,*and1−M01\-M\_\{0\}the*sample coverage*\[[1](https://arxiv.org/html/2610.22604#as1_bib.bib1)\]\. As Good establishes\[[3](https://arxiv.org/html/2610.22604#as1_bib.bib3)\],M0M\_\{0\}is identical to the probability that the next generation expresses an unseen species\.

The*Good\-Turin \(GT\) estimate*ofMkM\_\{k\}\[[3](https://arxiv.org/html/2610.22604#as1_bib.bib3)\]is

M^kGT=\(k\+1\)/n⋅Φk\+1\.\\widehat\{M\}^\{\\mathrm\{GT\}\}\_\{k\}=\(k\+1\)/n\\cdot\\Phi\_\{k\+1\}\.\(4\)Then, the GT estimate of missing\-mass isM^0GT=Φ1/n\\widehat\{M\}^\{\\mathrm\{GT\}\}\_\{0\}=\\Phi\_\{1\}/n\.

Unseen\-Species\.Unseen\-species estimation refers to the number of new species that may appear in additional draws\. For this purpose, Chao estimator and abundance\-based coverage estimator \(ACE\) exploit singleton/doubleton ratios\[[1](https://arxiv.org/html/2610.22604#as1_bib.bib1),[12](https://arxiv.org/html/2610.22604#as1_bib.bib12)\]and Good\-Toulmin estimator extrapolates expected discoveries\[[4](https://arxiv.org/html/2610.22604#as1_bib.bib4)\]\. Jackknife methods are used to reduce bias\[[13](https://arxiv.org/html/2610.22604#as1_bib.bib13)\]\.

SupposeUnewU\_\{\\mathrm\{new\}\}denotes the number of yet unseen new species that one sees innnewn\_\{\\mathrm\{new\}\}additional sample draws \(so that we end up with a sample of sizen\+nnewn\+n\_\{\\mathrm\{new\}\}\)\. Then, the*Good\-Toulmin \(GTo\) estimate*ofUnewU\_\{\\mathrm\{new\}\}\[[4](https://arxiv.org/html/2610.22604#as1_bib.bib4)\]is

U^new\(GTo\)=∑k=1n\(−1\)k\+1​tk​Φk,where​nnew=t​n\.\\widehat\{U\}\_\{\\mathrm\{new\}\}^\{\(\\mathrm\{GTo\}\)\}=\\sum\_\{k=1\}^\{n\}\(\-1\)^\{k\+1\}\\,t^\{k\}\\,\\Phi\_\{k\},\\text\{ where \}n\_\{\\mathrm\{new\}\}=t\\,n\.\(5\)This estimator of the number of new speciesUnewU\_\{\\mathrm\{new\}\}one may see innnewn\_\{\\mathrm\{new\}\}additional samples is unbiased and performs well for0≤t≤10\\leq t\\leq 1, but can become unstable fort\>1t\>1\[[4](https://arxiv.org/html/2610.22604#as1_bib.bib4)\]\.

### II\-BMotivation

Existing missing\-mass and unseen\-species estimators depend strongly on low\-frequencykk\-occupancies \(i\.e\., lowkkor rare species\)\. As mentioned earlier, in small\-sampling environments, these counts can be highly sensitive to sampling variability and hence fluctuate substantially\. This gives rise to high fragility of the missing\-mass and unseen\-species estimates\. For instance, the GT estimator\[[3](https://arxiv.org/html/2610.22604#as1_bib.bib3)\]and its variants are singleton\-driven and hence highly fragile at smallnnvalues\. So is the missing\-mass estimator in\[[8](https://arxiv.org/html/2610.22604#as1_bib.bib8)\]because it is largely determined by the observed expected occupancy values\. Poissonization is often used to simplify theoretical analysis to yield tractable error bounds\[[14](https://arxiv.org/html/2610.22604#as1_bib.bib14),[15](https://arxiv.org/html/2610.22604#as1_bib.bib15),[6](https://arxiv.org/html/2610.22604#as1_bib.bib6)\], but it ignores any multinomial dependence in the original sampling problem\. Moreover, the bounds extracted apply under asymptotic conditions\[[6](https://arxiv.org/html/2610.22604#as1_bib.bib6)\]\. We therefore seek estimators that explicitly account for fragility in small\-sample regimes with no recourse to Poissonization in order to retain the multinomial setting\.

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/synthetic_results/family_grid_winrate_ucr_mass.png)

\(a\) Missing\-mass estimation

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/synthetic_results/family_grid_winrate_ucr_count.png)

\(b\) Unseen\-species prediction

Fig\. 2:Per\-family UCR win rates for missing\-mass \(top\): GT\[[3](https://arxiv.org/html/2610.22604#as1_bib.bib3)\], SGT\[[16](https://arxiv.org/html/2610.22604#as1_bib.bib16)\], BD\[[17](https://arxiv.org/html/2610.22604#as1_bib.bib17)\], PY\[[18](https://arxiv.org/html/2610.22604#as1_bib.bib18)\]; and unseen\-species \(bottom\): GTo\[[4](https://arxiv.org/html/2610.22604#as1_bib.bib4)\], Chao\[[1](https://arxiv.org/html/2610.22604#as1_bib.bib1)\], ACE\[[19](https://arxiv.org/html/2610.22604#as1_bib.bib19)\], Orl\-SGT\[[5](https://arxiv.org/html/2610.22604#as1_bib.bib5)\], JK1/2\[[13](https://arxiv.org/html/2610.22604#as1_bib.bib13)\]\. Both include FA \(ours\)\. Entries indicate the fraction of experiments where the row method achieves lower UCR than the column method\.
### II\-CFragility\-Aware Missing\-Mass

Using the probability mass expressionsMkM\_\{k\}andMobsM\_\{\\mathrm\{obs\}\}in \([3](https://arxiv.org/html/2610.22604#S2.E3)\), we define fragility and regularization strength as

Fmass=M1/Mobs;α⁡\(Fmass\)=α0​eγ​Fmass,F\_\{\\mathrm\{mass\}\}=M\_\{1\}/M\_\{\\mathrm\{obs\}\};\\;\\;\\alpha\\,\(F\_\{\\mathrm\{mass\}\}\)=\\alpha\_\{0\}\\,e^\{\\gamma\\,F\_\{\\mathrm\{mass\}\}\},\(6\)whereα0\>0\\alpha\_\{0\}\>0andγ<0\\gamma<0\.

Computation of fragilityFmass=M1/MobsF\_\{\\mathrm\{mass\}\}=M\_\{1\}/M\_\{\\mathrm\{obs\}\}requires the population frequencies\{pℓ\}\\\{p\_\{\\ell\}\\\}, which we approximate via a Zipf\-regularized empirical histogram\[[20](https://arxiv.org/html/2610.22604#as1_bib.bib20)\]\. We emphasize that this is solely to estimateFmassF\_\{\\mathrm\{mass\}\}; the underlying GT and GTo estimators, andFcountF\_\{\\mathrm\{count\}\}introduced in \([9](https://arxiv.org/html/2610.22604#S2.E9)\), are all count\-based\.

The fragility\-aware missing\-mass estimate is computed via

M^0†=argmaxM∈\(0,1\)\{h\(M\)−α\(Fmass\)DKL\(M∥M^\)\},\\widehat\{M\}\_\{0\}^\{\{\\dagger\}\}=\\arg\\max\_\{M\\in\(0,1\)\}\\left\\\{h\(M\)\-\\alpha\(F\_\{\\mathrm\{mass\}\}\)\\,D\_\{\\mathrm\{KL\}\}\(M\\\|\\widehat\{M\}\)\\right\\\},\(7\)withM^\\widehat\{M\}a base estimator \(we use the GT estimateM^0GT=Φ1/n\\widehat\{M\}^\{\\mathrm\{GT\}\}\_\{0\}=\\Phi\_\{1\}/nin \([4](https://arxiv.org/html/2610.22604#S2.E4)\)\),hhthe Bernoulli entropy, andDKL\(M∥M^\)D\_\{\\mathrm\{KL\}\}\(M\\\|\\widehat\{M\}\)the KL divergence between Bernoulli parameters\.

The KL term preserves fidelity toM^\\widehat\{M\}\. Note that, sinceγ<0\\gamma<0, increasing fragility reducesα⁡\(Fmass\)\\alpha\(F\_\{\\mathrm\{mass\}\}\), thereby weakening the KL penalty and giving greater weight to the entropy term\. In turn, as this entropy term increases, the estimateM^0†\\widehat\{M\}\_\{0\}^\{\{\\dagger\}\}moves toward the maximally noncommittal value0\.50\.5\. In essence, under high fragility, the correction moves the estimate toward a less committed state instead of placing excessive confidence in an unstable base estimate\.

Since the objective in \([7](https://arxiv.org/html/2610.22604#S2.E7)\) is differentiable and strictly concave inMM, its unique maximizer can be obtained by setting the derivative w\.r\.t\.MMto zero\. The first\-order condition yields the closed\-form solution \(see\[[11](https://arxiv.org/html/2610.22604#as1_bib.bib11)\]for details\)

logit⁡\(M^0†\)=α⁡\(Fmass\)1\+α⁡\(Fmass\)​logit​\(M^\),\\mathrm\{logit\}\(\\widehat\{M\}\_\{0\}^\{\{\\dagger\}\}\)=\\frac\{\\alpha\(F\_\{\\mathrm\{mass\}\}\)\}\{1\+\\alpha\(F\_\{\\mathrm\{mass\}\}\)\}\\,\\mathrm\{logit\}\(\\widehat\{M\}\),\(8\)wherelogit⁡\(x\)=log⁡\(x/\(1−x\)\)\\mathrm\{logit\}\(x\)=\\log\\bigl\(x/\(1\-x\)\\bigr\)\.

### II\-DFragility\-Aware Unseen\-Species

Whereas missing\-mass quantifies the total probability mass assigned tokk\-occupancy species, unseen\-species quantifies the number of distinct species that remain unobserved\. To account for sampling fragility in this setting, consider thekk\-occupancy and total observed species countsΦk\\Phi\_\{k\}andΦobs\\Phi\_\{\\mathrm\{obs\}\}in \([2](https://arxiv.org/html/2610.22604#S2.E2)\)\. Then, analogous to \([6](https://arxiv.org/html/2610.22604#S2.E6)\), we define count\-based fragility and associated regularization strength as

Fcount=Φ1/Φobs;α⁡\(Fcount\)=α0​eγ​Fcount,F\_\{\\mathrm\{count\}\}=\\Phi\_\{1\}/\\Phi\_\{\\mathrm\{obs\}\};\\;\\;\\alpha\(F\_\{\\mathrm\{count\}\}\)=\\alpha\_\{0\}\\,e^\{\\gamma\\,F\_\{\\mathrm\{count\}\}\},\(9\)withα0\>0\\alpha\_\{0\}\>0andγ<0\\gamma<0\.

The fragility\-aware unseen\-species count is then computed via

u^new†=argmaxu∈\(0,1\)\{h\(u\)−α\(Fcount\)DKL\(u∥u^new\)\},\\widehat\{u\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}=\\arg\\max\_\{u\\in\(0,1\)\}\\left\\\{h\(u\)\-\\alpha\\,\(F\_\{\\mathrm\{count\}\}\)\\,D\_\{\\mathrm\{KL\}\}\(u\\\|\\widehat\{u\}\_\{\\mathrm\{new\}\}\)\\right\\\},\(10\)where, as we recall,UnewU\_\{\\mathrm\{new\}\}is the number of previously unobserved species discovered innnewn\_\{\\mathrm\{new\}\}additional draws following an initial sample of sizenn;unew=Unew/Umaxu\_\{\\mathrm\{new\}\}=U\_\{\\mathrm\{new\}\}/U\_\{\\max\}isUnewU\_\{\\mathrm\{new\}\}normalized byUmaxU\_\{\\max\}, an upper bound forUnewU\_\{\\mathrm\{new\}\}so that0≤Unew≤Umax0\\leq U\_\{\\mathrm\{new\}\}\\leq U\_\{\\max\}\(e\.g\., one could use, as we do,Umax=nnewU\_\{\\max\}=n\_\{\\mathrm\{new\}\}for this\)\. Similarly,u^new=U^new/Umax\\widehat\{u\}\_\{\\mathrm\{new\}\}=\\widehat\{U\}\_\{\\mathrm\{new\}\}/U\_\{\\max\}is the normalized value ofU^new\\widehat\{U\}\_\{\\mathrm\{new\}\}, a base estimator ofUnewU\_\{\\mathrm\{new\}\}\(we use the GTo estimatorU^new\(GTo\)\\widehat\{U\}\_\{\\mathrm\{new\}\}^\{\(\\mathrm\{GTo\}\)\}in \([5](https://arxiv.org/html/2610.22604#S2.E5)\) for this\)\.

As before, the KL term preserves fidelity tou^new\\widehat\{u\}\_\{\\mathrm\{new\}\}\. Sinceγ<0\\gamma<0, increasing fragility reducesα⁡\(Fcount\)\\alpha\(F\_\{\\mathrm\{count\}\}\), thereby weakening the KL penalty and emphasizing the entropy term\. As this entropy term increases, the estimateu^new†\\widehat\{u\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}moves toward the maximally noncommittal value0\.50\.5; in turn,U^new†=Umax​u^new†\\widehat\{U\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}=U\_\{\\max\}\\,\\widehat\{u\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}, the corresponding fragility\-aware estimate of the unseen species count, moves towards0\.5​Umax0\.5\\,U\_\{\\max\}\. So, under high fragility, the correction moves the unseen\-species estimate toward the least committed count, viz\., the midpoint of the interval\[0,Umax\]\[0,\\,U\_\{\\max\}\]\.

With an argument similar to that in Section[II\-C](https://arxiv.org/html/2610.22604#S2.SS3), we can get the closed\-form solution \(see supplementary\[[11](https://arxiv.org/html/2610.22604#as1_bib.bib11)\]for details\)\.

logit⁡\(u^new†\)=α⁡\(Fcount\)1\+α⁡\(Fcount\)​logit​\(u^new\)\.\\mathrm\{logit\}\(\\widehat\{u\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}\)=\\frac\{\\alpha\(F\_\{\\mathrm\{count\}\}\)\}\{1\+\\alpha\(F\_\{\\mathrm\{count\}\}\)\}\\,\\mathrm\{logit\}\(\\widehat\{u\}\_\{\\mathrm\{new\}\}\)\.\(11\)

## IIIExperiments

We evaluate our estimates on synthetic distributions \(with known\{pℓ\}\\\{p\_\{\\ell\}\\\}\) and language corpora\. Each trial draws an i\.i\.d\. sample of sizenn, computes estimators from observed occupancies, then measures error on a disjoint future block of sizennewn\_\{\\mathrm\{new\}\}from the same population \(Fig\.[1](https://arxiv.org/html/2610.22604#S2.F1)\)\. Ground truth is empirical:M0M\_\{0\}is the fraction of holdout probability on species absent from the sample;UnewU\_\{\\mathrm\{new\}\}is the number of species in the holdout that were unseen in the sample\.

### III\-ASynthetic Distributions

Synthetic populations span stable to fragmented regimes:Uniform\(S∈\{200,500,1000\}S\\in\\\{200,500,1000\\\}\) as an equal\-probability reference;Zipf\(s∈\{0\.5,1\.0,1\.5\}s\\in\\\{0\.5,1\.0,1\.5\\\}\) has varying tail heaviness;Dirichlet\(α∈\{0\.3,1\.0,3\.0\}\\alpha\\in\\\{0\.3,1\.0,3\.0\\\}\) has random sparse/smooth populations; andHead\-tail mixtures\(ρ∈\{0\.7,0\.85,0\.95\}\\rho\\in\\\{0\.7,0\.85,0\.95\\\}\) show dominant head plus diffused tail\. Sample sizesn∈\{25,50,100,200,500,1000\}n\\in\\\{25,50,100,200,500,1000\\\}cover small to moderate regimes\.

### III\-BReal\-World Datasets

Shakespeare\[[21](https://arxiv.org/html/2610.22604#as1_bib.bib21)\]: after lowercasing, punctuation normalization, and tokenization, uni\-grams are species and corpus frequencies define the population; random length\-nnsubsamples simulate partial observation\.Google 1\-gram\[[22](https://arxiv.org/html/2610.22604#as1_bib.bib22)\]: token corpus frequencies define the population with sharper tail sparsity than Shakespeare; subsamples at varyingnnsimulate incomplete coverage\. Forecast horizonsrfuture=nnew/n∈\[0\.03,0\.10\]r\_\{\\mathrm\{future\}\}=n\_\{\\mathrm\{new\}\}/n\\in\[0\.03,0\.10\]set holdout size andUmax=nnewU\_\{\\max\}=n\_\{\\mathrm\{new\}\}; they define evaluation targets, not estimator inputs\.

### III\-CBaselines

Missing\-mass:GT; smoothed GT \(SGT\); symmetric Dirichlet Bayes \(BD\); Pitman\-Yor \(PY\) Bayes\.

Unseen counts:Chao; ACE; GTo; Orl\-SGT; first\- and second\-order jackknife \(JK1, JK2\)\.

Experiments111Code at https://github\.com/pragasv/Rarity\-of\-Singletons\-Matters\.gitran on a workstation \(64 GB RAM, NVIDIA A6000\)\. The fragility mapα⁡\(F\)=α0​eγ​F\\alpha\(F\)=\\alpha\_\{0\}e^\{\\gamma F\}usesα0\>0\\alpha\_\{0\}\>0as baseline trust in GT/GTo andγ<0\\gamma<0to weaken adjustment as fragility falls\. We fix\(α0,γ\)=\(100,−1\.0\)\(\\alpha\_\{0\},\\gamma\)=\(100,\-1\.0\)globally across all configurations\.

### III\-DEvaluation Metrics

We report absolute error\|M^0−M0\|\|\\widehat\{M\}\_\{0\}\-M\_\{0\}\|and\|U^new−Unew\|\|\\widehat\{U\}\_\{\\mathrm\{new\}\}\-U\_\{\\mathrm\{new\}\}\|over 3000 trials per configuration\.Upper confidence risk \(UCR\)bins fragility, with bin weightswiw\_\{i\}, mean errorμi\\mu\_\{i\}, and std\.σi\\sigma\_\{i\}:

UCR=∑iwi​\(μi\+z​σi\),\\mathrm\{UCR\}=\\sum\_\{i\}w\_\{i\}\\,\(\\mu\_\{i\}\+z\\,\\sigma\_\{i\}\),\(12\)wherez=1\.96z=1\.96\. Lower UCR indicates accuracy and stability\. To capture effect size beyond win rates, we report normalized pairwise gain \(baseline headroom gain\):

gi​j=tanh⁡\(γ⋅U​C​Rj−U​C​Ri\|U​C​Ri\|\+\|U​C​Rj\|\),g\_\{ij\}=\\tanh\\\!\\left\(\\gamma\\cdot\\frac\{UCR\_\{j\}\-UCR\_\{i\}\}\{\|UCR\_\{i\}\|\+\|UCR\_\{j\}\|\}\\right\),\(13\)withgi​j\>0g\_\{ij\}\>0when row methodiilowers UCR versus column baselinejj\.

## IVResults

### IV\-AWorked Example

TABLE I:Worked example on the semantic\-clustering query \(n=15n=15, six clusters,nnew=5n\_\{\\mathrm\{new\}\}=5,\{Φ1,Φ2,Φobs\}=\{3,2,6\}\\\{\\Phi\_\{1\},\\,\\Phi\_\{2\},\\,\\Phi\_\{\\mathrm\{obs\}\}\\\}=\\\{3,\\,2,\\,6\\\}, andM0=0\.210M\_\{0\}=0\.210\)\. The upper panel compares missing\-mass estimators \(GT, SGT, BD, PY, and FA\), while the lower panel compares unseen\-species count estimators \(GTo, Chao, ACE, Orl\-SGT, and FA\)\. Our FA method achieves the lowest absolute error in both tasks\.
Table[I](https://arxiv.org/html/2610.22604#S4.T1)shows the application of our proposed technique of missing\-mass and unseen\-count estimation to the same example in\[[23](https://arxiv.org/html/2610.22604#as1_bib.bib23)\]\. The input query*“Which oil producer is a close ally of the United States?”*was presented, and repeated fifteen times, to an open\-weight LLM\. We choose an LLM setting because it embodies the small\-sample, open\-vocabulary regime that motivates this work: per\-query generations are few, the number of possible semantic clusters is unknown, and rare singletons can appear freely, with no fixed support size or tail assumptions\.

The 15 responses form six semantic clusters \(Φobs=6\\Phi\_\{\\mathrm\{obs\}\}=6\) with occupancy countsNℓ=\{1,8,2,2,1,1\}N\_\{\\ell\}=\\\{1,8,2,2,1,1\\\}and occupancy statisticsΦk=\{3,2,⋯,1\}\\Phi\_\{k\}=\\\{3,\\,2,\\,\\cdots,1\\\}yielding three singletons \(Φ1\\Phi\_\{1\}\), two doubletons \(Φ2\\Phi\_\{2\}\) and one cluster observed eight times\. The observed clusters account for∑ℓpc\(ℓ\)=0\.79\\sum\_\{\\ell\}p\_\{c\}^\{\(\\ell\)\}=0\.79of the population probability, leaving a true missing\-mass ofM0=0\.21M\_\{0\}=0\.21\. We model this remaining mass as 21 unseen meanings, each with probability0\.010\.01\. Fornnew=5n\_\{\\mathrm\{new\}\}=5additional generations, the expected number of new semantics is𝔼⁡\[Unew\]=21​\(1−0\.995\)=1\.029\\mathbb\{E\}\[U\_\{\\mathrm\{new\}\}\]=21\(1\-0\.99^\{5\}\)=1\.029\.

For each missing\-mass estimator\-GT, SGT, BD, PY, and our FA estimator, the estimated missing\-massM^0\\widehat\{M\}\_\{0\}yields adjusted*observed coverage shares*c^\(ℓ\)=\(1−M^0\)​Pobs\(ℓ\)/Pobs\\widehat\{c\}^\{\(\\ell\)\}=\(1\-\\widehat\{M\}\_\{0\}\)\\,P\_\{\\mathrm\{obs\}\}^\{\(\\ell\)\}/P\_\{\\mathrm\{obs\}\}, with the residualM^0\\widehat\{M\}\_\{0\}reserved for unseen meanings \(*Unseen*row in Table[I](https://arxiv.org/html/2610.22604#S4.T1)\)\. The lower panel in Table[I](https://arxiv.org/html/2610.22604#S4.T1)separately compares unseen\-species estimates from GTo, Chao, ACE, Orl\-SGT, and our FA estimator\. FA estimates use \([8](https://arxiv.org/html/2610.22604#S2.E8)\)\-\([10](https://arxiv.org/html/2610.22604#S2.E10)\) and attains the*lowest absolute error of all methods*on both tasks\. The richness estimators Chao/ACE target total support rather than thennewn\_\{\\mathrm\{new\}\}\-horizon and overshoot\. FA thus uses the extra fragility information \(pc\(ℓ\)p\_\{c\}^\{\(\\ell\)\}viaFmassF\_\{\\mathrm\{mass\}\}\) to recalibrate GT when it is unreliable, the per\-query face of the variance reduction quantified by our UCR analysis across repeated trials \(see Sec\.[IV](https://arxiv.org/html/2610.22604#S4)\)\.

### IV\-BSynthetic Distributions

For the synthetic families of uniform, Zipf, Dirichlet, head\-tail span smooth, skewed, and fragmented landscapes, Fig\.[2](https://arxiv.org/html/2610.22604#S2.F2)presents the per\-family UCR win rates for missing\-mass and unseen\-species estimation, while Fig\.[3](https://arxiv.org/html/2610.22604#S4.F3)summarizes the overall win rates and normalized gainsgi​jg\_\{ij\}\.

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/synthetic_results/overall_winrate_ucr_mass.png)

\(a\) Missing\-mass win rate

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/synthetic_results/overall_winrate_ucr_count.png)

\(b\) Unseen\-species win rate

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/improvements/overall_bhg_ucr_mass.png)

\(c\) Missing\-mass gain

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/improvements/overall_bhg_ucr_count.png)

\(d\) Unseen\-species gain

Fig\. 3:Synthetic overview \(144 experiments\)\. Top: pairwise UCR win rates; bottom: normalized pairwise gaingi​jg\_\{ij\}\(see \([13](https://arxiv.org/html/2610.22604#S3.E13)\)\)\. Blue indicates that the row method lowers UCR relative to the column baseline\.Missing\-mass\.Win rates \(Fig\.[3](https://arxiv.org/html/2610.22604#S4.F3)a\) and gain \(Fig\.[3](https://arxiv.org/html/2610.22604#S4.F3)c\) agree: our estimator matches GT \(g≈0g\\approx 0\) while substantially lowering UCR versus SGT, PY, and Bayes\-Dir \(Table\)\. Per\-family gains are largest on Zipf and Dirichlet; on head\-tail mixtures, GT can edge ahead when singletons are stable \(Supp\.\)\.

Unseen\-counts\.Gains versus jackknife, ACE, and Chao are large \(g\>0\.4g\>0\.4in Fig\.[3](https://arxiv.org/html/2610.22604#S4.F3)d\), while differences among GT, Orlitsky\-SGT, and our method are small\. Win rates understate how poorly unstable baselines perform \(Fig\.[3](https://arxiv.org/html/2610.22604#S4.F3)b\)\. Absolute UCR tables and head\-tail analysis are in Supp\.\.

### IV\-CReal\-World Datasets

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/shakespear/overall_winrate_mass_pooled.png)

\(a\) Shakespeare: missing\-mass

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/shakespear/overall_winrate_count_pooled.png)

\(b\) Shakespeare: unseen\-species

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/google/overall_winrate_mass_pooled.png)

\(c\) Google: missing\-mass

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/google/overall_winrate_count_pooled.png)

\(d\) Google: unseen\-species

Fig\. 4:Pairwise UCR win rates across Shakespeare \(top\) and Google 1\-gram \(bottom\), each with 108 experiments, pooled overrfuture∈\[0\.03,0\.10\]r\_\{\\mathrm\{future\}\}\\in\[0\.03,0\.10\]\. Each entry\(i,j\)\(i,j\)denotes the fraction of experiments where row methodiioutperforms column methodjj; values above0\.50\.5indicate more frequent wins\. Fragility\-aware estimation performs best on unseen counts and the heavier\-tailed Google corpus\.Shakespeare Corpus \(Project Gutenberg\)\.Fig\.[4](https://arxiv.org/html/2610.22604#S4.F4)\(a\) shows competitiveness against classical methods and above GT on missing\-mass\. Fig\.[4](https://arxiv.org/html/2610.22604#S4.F4)\(b\) showcases conservative adjustments beat aggressive extrapolation under long\-tail noise\. Additional details across forecast horizons are in Supp\., Supp\.\.

Google 1\-Gram Corpus\.Google uni\-grams have larger vocabulary and sharper head\-tail separation\. Fig\.[4](https://arxiv.org/html/2610.22604#S4.F4)\(c\) showcases entropy\-regularized adjustment outperforms GT\-type estimators when frequent tokens dominate mass\. Fig\.[4](https://arxiv.org/html/2610.22604#S4.F4)\(d\) shows stability correction moderates over\-discovery from aggressive smoothers\. Additional details are across forecast horizons are provided in Supp\.and Supp\.\.

## VConclusion

We introduce fragility\-aware missing\-mass and unseen\-species estimators that account for sampling instability through singleton\-based fragility measures and entropy\-KL regularization\. The strictly concave objective admits a unique closed\-form solution without Poissonization, providing a lightweight correction to classical estimators such as GT and GTo\. Across 360 experiments on synthetic distributions and language corpora, our approach maintains competitive accuracy while improving stability \(lower UCR\), particularly in long\-tailed, singleton\-dominated, small\-sample regimes\. These findings highlight the importance of rare\-event stability for reliable inference from incomplete data, with applications to coverage estimation, biodiversity, and uncertainty quantification in language models\.

## VIAI Statement

OpenAI’s ChatGPT was used solely for language refinement and assistance with code development\. The authors take full responsibility for the content of this work\.

## References

- \[1\]A\. Chao, “Nonparametric estimation of the number of classes in a population,”*Scandinavian Journal of statistics*, pp\. 265–270, 1984\.
- \[2\]P\. Vipulanandan, K\. Premaratne, and D\. Sarkar, “Sensitivity and uncertainty quantification in regression neural networks: A quantum tensor network based approach,” in*Proceedings of the International Symposium on Artificial Intelligence and Mathematics \(ISAIM\)*, Fort Lauderdale, FL, USA, 2026\.
- \[3\]I\. J\. Good, “The population frequencies of species and the estimation of population parameters,”*Biometrika*, vol\. 40, no\. 3\-4, pp\. 237–264, 1953\.
- \[4\]I\. J\. Good and G\. H\. Toulmin, “The number of new species, and the increase in population coverage, when a sample is increased,”*Biometrika*, vol\. 43, no\. 1\-2, pp\. 45–63, 1956\.
- \[5\]A\. Orlitsky, A\. T\. Suresh, and Y\. Wu, “Optimal prediction of the number of unseen species,”*Proceedings of the National Academy of Sciences*, vol\. 113, no\. 47, pp\. 13 283–13 288, 2016\.
- \[6\]Y\. Hao and P\. Li, “Optimal prediction of the number of unseen species with multiplicity,”*Advances in Neural Information Processing Systems*, vol\. 33, pp\. 8553–8564, 2020\.
- \[7\]M\. Kestemont, F\. Karsdorp, E\. de Bruijn, M\. Driscoll, K\. A\. Kapitan, P\. Ó Macháin, D\. Sawyer, R\. Sleiderink, and A\. Chao, “Forgotten books: The application of unseen species models to the survival of culture,”*Science*, vol\. 375, no\. 6582, pp\. 765–769, 2022\.
- \[8\]S\. Lee and M\. Boehme, “How much is unseen depends chiefly on information about the seen,” in*The Thirteenth International Conference on Learning Representations*, 2025\. \[Online\]\. Available:[https://openreview\.net/forum?id=uqWM9hBDAE](https://openreview.net/forum?id=uqWM9hBDAE)
- \[9\]J\. Weltz, E\. Laber, and A\. Volfovsky, “Hidden population estimation with indirect inference and auxiliary information,”*Proceedings of machine learning research*, vol\. 244, p\. 3730, 2024\.
- \[10\]J\. E\. Schmitz and S\. Rahmann, “A comprehensive review and evaluation of species richness estimation,”*Briefings in Bioinformatics*, vol\. 26, no\. 2, p\. bbaf158, 2025\.
- \[11\]P\. Vipulanandan, K\. Premaratne, and D\. Sarkar, “Supplementary materials: Rarity of singletons matters,” 2026, manuscript submitted to arXiv\.
- \[12\]A\. Chao, W\.\-H\. Hwang, Y\. Chen, and C\. Kuo, “Estimating the number of shared species in two communities,”*Statistica Sinica*, pp\. 227–246, 2000\.
- \[13\]S\. Zahl, “Jackknifing an index of diversity,”*Ecology*, vol\. 58, no\. 4, pp\. 907–913, 1977\.
- \[14\]D\. A\. McAllester and R\. E\. Schapire, “On the convergence rate of Good\-Turing estimators\.” in*COLT*, 2000, pp\. 1–6\.
- \[15\]E\. Drukh and Y\. Mansour, “Concentration bounds for unigram language models\.”*Journal of Machine Learning Research*, vol\. 6, no\. 8, 2005\.
- \[16\]W\. A\. Gale and G\. Sampson, “Good\-turing frequency estimation without tears,”*Journal of quantitative linguistics*, vol\. 2, no\. 3, pp\. 217–237, 1995\.
- \[17\]G\. Casella and R\. Berger,*Statistical Inference*\. Duxbury, 2002\.
- \[18\]J\. Pitman and M\. Yor, “The two\-parameter poisson\-dirichlet distribution derived from a stable subordinator,”*The Annals of Probability*, pp\. 855–900, 1997\.
- \[19\]A\. Chao and S\.\-M\. Lee, “Estimating the number of classes via sample coverage,”*Journal of the American statistical Association*, vol\. 87, no\. 417, pp\. 210–217, 1992\.
- \[20\]H\. Baayen,*Word Frequency Distributions*\. Springer, 2001\.
- \[21\]W\. Shakespeare,*The Complete Works of William Shakespeare*\. Project Gutenberg, 2004\. \[Online\]\. Available:[https://www\.gutenberg\.org/ebooks/100](https://www.gutenberg.org/ebooks/100)
- \[22\]J\.\-B\. Michel, Y\. K\. Shen, A\. P\. Aiden, A\. Veres, M\. K\. Gray, G\. B\. Team, J\. P\. Pickett, D\. Hoiberg, D\. Clancy, P\. Norvig*et al\.*, “Quantitative analysis of culture using millions of digitized books,”*science*, vol\. 331, no\. 6014, pp\. 176–182, 2011\.
- \[23\]P\. Vipulanandan, K\. Premaratne, and D\. Sarkar, “Semantic uncertainty quantification of hallucinations in LLMs: A quantum tensor network based method,” in*International Conference on Learning Representations \(ICLR\)*, 2026\. \[Online\]\. Available:[https://openreview\.net/forum?id=11kPIEkj75](https://openreview.net/forum?id=11kPIEkj75)

## Supplementary Materials: Rarity of Singletons Matters

This document provides supplementary details, extended derivations, and additional experimental evaluations to accompany the main manuscript\.

## S\-IProperties of the Fragility Adjustment

LetA∈\(0,1\)A\\in\(0,1\)denote a base estimate andF∈\[0,1\]F\\in\[0,1\]the corresponding fragility measure, whereF=FmassF=F\_\{\\mathrm\{mass\}\}for missing\-mass andF=FcountF=F\_\{\\mathrm\{count\}\}for unseen\-species count estimation\.

Both fragility\-aware estimators maximize the generic entropy–KL objective

J\(q\)=h\(q\)−α\(F\)DKL\(q∥A\),q∈\(0,1\),J\(q\)=h\(q\)\-\\alpha\(F\)D\_\{\\mathrm\{KL\}\}\(q\\\|A\),\\qquad q\\in\(0,1\),\(S1\)whereqqrepresents either the missing\-mass parameterMMor the normalized unseen\-species parameteruu\.

The corresponding fragility\-aware estimateq^†\\widehat\{q\}^\{\{\\dagger\}\}is the unique maximizer ofJ⁡\(q\)J\(q\)\. Thus, all structural properties derived below therefore apply to both the fragility\-aware missing\-mass estimator and the fragility\-aware unseen\-species estimator\.

### S\-I\-AConvexity of the objective

###### Lemma 1\.

LetA∈\(0,1\)A\\in\(0,1\)andα⁡\(F\)\>0\\alpha\(F\)\>0\. The objective

J\(q\)=h\(q\)−α\(F\)DKL\(q∥A\)J\(q\)=h\(q\)\-\\alpha\(F\)\\,D\_\{\\mathrm\{KL\}\}\(q\\\|A\)is strictly concave inq∈\(0,1\)q\\in\(0,1\)\.

###### Proof\.

The Bernoulli entropy satisfies

h⁡\(q\)=−q​log⁡q−\(1−q\)​log⁡\(1−q\),h\(q\)=\-q\\log q\-\(1\-q\)\\log\(1\-q\),with derivatives

h′​\(q\)=log⁡1−qq,h′′​\(q\)=−1q−11−q\.h^\{\\prime\}\(q\)=\\log\\\!\\frac\{1\-q\}\{q\},\\qquad h^\{\\prime\\prime\}\(q\)=\-\\frac\{1\}\{q\}\-\\frac\{1\}\{1\-q\}\.
The KL divergence in its first argument is

DKL\(q∥A\)=qlogqA\+\(1−q\)log1−q1−A,D\_\{\\mathrm\{KL\}\}\(q\\\|A\)=q\\log\\\!\\frac\{q\}\{A\}\+\(1\-q\)\\log\\\!\\frac\{1\-q\}\{1\-A\},whose derivatives are

∂∂q​DKL=log⁡q⁡\(1−A\)A⁡\(1−q\),∂2∂q2​DKL=1q\+11−q\.\\frac\{\\partial\}\{\\partial q\}D\_\{\\mathrm\{KL\}\}=\\log\\\!\\frac\{q\(1\-A\)\}\{A\(1\-q\)\},\\qquad\\frac\{\\partial^\{2\}\}\{\\partial q^\{2\}\}D\_\{\\mathrm\{KL\}\}=\\frac\{1\}\{q\}\+\\frac\{1\}\{1\-q\}\.
Therefore the second derivative ofJJis

J′′​\(q\)=h′′​\(q\)−α⁡\(F\)​∂2∂q2​DKL=−\(1\+α⁡\(F\)\)​\(1q\+11−q\)\.J^\{\\prime\\prime\}\(q\)=h^\{\\prime\\prime\}\(q\)\-\\alpha\(F\)\\frac\{\\partial^\{2\}\}\{\\partial q^\{2\}\}D\_\{\\mathrm\{KL\}\}=\-\\bigl\(1\+\\alpha\(F\)\\bigr\)\\left\(\\frac\{1\}\{q\}\+\\frac\{1\}\{1\-q\}\\right\)\.
Sinceα⁡\(F\)\>0\\alpha\(F\)\>0and1q\+11−q\>0\\frac\{1\}\{q\}\+\\frac\{1\}\{1\-q\}\>0for allq∈\(0,1\)q\\in\(0,1\), it follows thatJ′′​\(q\)<0J^\{\\prime\\prime\}\(q\)<0everywhere on\(0,1\)\(0,1\)\. HenceJJis strictly concave and admits a unique maximizer\. ∎

### S\-I\-BLimit Analysis

The following result characterizes the limiting behavior of the generic fragility\-aware estimatorq^†\\widehat\{q\}^\{\{\\dagger\}\}introduced in Section[S\-I](https://arxiv.org/html/2610.22604#as1_S1)\. Here,q=Mq=MwithA=M^A=\\widehat\{M\}for missing\-mass estimation, andq=uq=uwithA=u^newA=\\widehat\{u\}\_\{\\mathrm\{new\}\}for normalized unseen\-species estimation\.

Lemma\.LetA∈\(0,1\)A\\in\(0,1\)and letq^†​\(α\)\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)denote the unique maximizer of the generic objectiveJ⁡\(q\)J\(q\)for a givenα\>0\\alpha\>0\. Then

limα→∞q^†​\(α\)=A\.\\lim\_\{\\alpha\\to\\infty\}\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)=A\.
*Proof: Write the objective as*

Jα\(q\)=h\(q\)−αDKL\(q∥A\),J\_\{\\alpha\}\(q\)=h\(q\)\-\\alpha\\,D\_\{\\mathrm\{KL\}\}\(q\\\|A\),*where*

h\(q\)=−qlog\(q\)−\(1−q\)log\(1−q\);DKL\(q∥A\)=qlogqA\+\(1−q\)log1−q1−A\.h\(q\)=\-q\\log\\,\(q\)\-\(1\-q\)\\,\\log\\,\(1\-q\);\\;\\;D\_\{\\mathrm\{KL\}\}\(q\\\|A\)=q\\,\\log\\,\\frac\{q\}\{A\}\+\(1\-q\)\\,\\log\\frac\{1\-q\}\{1\-A\}\.*ButJαJ\_\{\\alpha\}is strictly concave on\(0,1\)\(0,1\)\(see Lemma[1](https://arxiv.org/html/2610.22604#as1_Thmlemma1)\)\. So, the maximizerq^†​\(α\)\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)is unique and satisfies the first\-order conditionJα′​\(q^†​\(α\)\)=0J\_\{\\alpha\}^\{\\prime\}\(\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\)=0\. A direct calculation of the derivativesh′​\(q\)h^\{\\prime\}\(q\)and∂DKL\(q∥A\)/∂q\\partial D\_\{\\mathrm\{KL\}\}\(q\\\|A\)/\\partial qyields thatq^†​\(α\)\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)satisfies*

log⁡1−q^†​\(α\)q^†​\(α\)=α​log⁡q^†​\(α\)​\(1−A\)\(1−q^†​\(α\)\)​A\.\\log\\frac\{1\-\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\}\{\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\}=\\alpha\\,\\log\\frac\{\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\\,\(1\-A\)\}\{\(1\-\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\)\\,A\}\.*We can express this as*

−logit⁡\(q^†​\(α\)\)=α⁡\(logit⁡\(q^†​\(α\)\)−logit⁡\(A\)\),\-\\mathrm\{logit\}\\,\(\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\)=\\alpha\\,\(\\mathrm\{logit\}\\,\(\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\)\-\\mathrm\{logit\}\\,\(A\)\),*wherelogit⁡\(∙\)\\mathrm\{logit\}\(\\centerdot\)is the logit function \(e\.g\.,logit⁡\(A\)=log⁡\(A/\(1−A\)\)\\mathrm\{logit\}\\,\(A\)=\\log\\,\(A/\(1\-A\)\)\)\. This gives us*

logit⁡\(q^†​\(α\)\)=αα\+1​logit​\(A\)\.\\mathrm\{logit\}\\,\(\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\)=\\frac\{\\alpha\}\{\\alpha\+1\}\\,\\mathrm\{logit\}\\,\(A\)\.*So, asα→∞\\alpha\\to\\infty,logit⁡\(q^†​\(α\)\)→logit⁡\(A\)\\mathrm\{logit\}\\,\(\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\)\\to\\mathrm\{logit\}\\,\(A\)and henceq^†​\(α\)→A\\widehat\{q\}^\{\{\\dagger\}\}\(\\alpha\)\\to A\. Q\.E\.D\.*

This result applies to both estimation tasks\. For missing\-mass estimation, the fragility\-aware estimateM^0†\\widehat\{M\}\_\{0\}^\{\{\\dagger\}\}converges to its base estimateM^\\widehat\{M\}asα→∞\\alpha\\to\\infty\. Similarly, for unseen\-species estimation, the normalized estimateu^new†\\widehat\{u\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}converges tou^new\\widehat\{u\}\_\{\\mathrm\{new\}\}, implying that the corresponding unnormalized estimateU^new†\\widehat\{U\}\_\{\\mathrm\{new\}\}^\{\{\\dagger\}\}converges toU^new\\widehat\{U\}\_\{\\mathrm\{new\}\}\. Conversely, asα→0\+\\alpha\\to 0^\{\+\}, the closed\-form solution shows that both normalized estimators approach1/21/2, corresponding to a missing\-mass estimate of1/21/2and an unseen\-species count estimate ofUmax/2U\_\{\\max\}/2, respectively\.

### S\-I\-CExpectation of Fragility

For missing\-mass fragilityF^mass=P1/Pobs\\widehat\{F\}\_\{\\mathrm\{mass\}\}=P\_\{1\}/P\_\{\\mathrm\{obs\}\}, the ratio is nonlinear in correlated sums and𝔼⁡\[F^mass\]\\mathbb\{E\}\[\\widehat\{F\}\_\{\\mathrm\{mass\}\}\]has no exact closed form\. A second\-order delta approximation\[Cochran1977Sampling\]gives

𝔼⁡\[F^mass\]\\displaystyle\\mathbb\{E\}\[\\widehat\{F\}\_\{\\mathrm\{mass\}\}\]≈μAμB−Cov⁡\(P1,Pobs\)μB2\+μA​Var⁡\(Pobs\)μB3,\\displaystyle\\approx\\frac\{\\mu\_\{A\}\}\{\\mu\_\{B\}\}\-\\frac\{\\Cov\(P\_\{1\},P\_\{\\mathrm\{obs\}\}\)\}\{\\mu\_\{B\}^\{2\}\}\+\\frac\{\\mu\_\{A\}\\,\\Var\(P\_\{\\mathrm\{obs\}\}\)\}\{\\mu\_\{B\}^\{3\}\},\(S2\)withμA=𝔼⁡\[P1\]\\mu\_\{A\}=\\mathbb\{E\}\[P\_\{1\}\],μB=𝔼⁡\[Pobs\]\\mu\_\{B\}=\\mathbb\{E\}\[P\_\{\\mathrm\{obs\}\}\]\. Undernnmultinomial draws andNℓ∼Binomial⁡\(n,pℓ\)N\_\{\\ell\}\\sim\\mathrm\{Binomial\}\(n,p\_\{\\ell\}\),

μA\\displaystyle\\mu\_\{A\}=n​∑ℓ=1Spℓ2​\(1−pℓ\)n−1,\\displaystyle=n\\sum\_\{\\ell=1\}^\{S\}p\_\{\\ell\}^\{2\}\(1\-p\_\{\\ell\}\)^\{n\-1\},μB\\displaystyle\\mu\_\{B\}=1−∑ℓ=1Spℓ​\(1−pℓ\)n\.\\displaystyle=1\-\\sum\_\{\\ell=1\}^\{S\}p\_\{\\ell\}\(1\-p\_\{\\ell\}\)^\{n\}\.\(S3\)Variances and covariances follow from binomial indicator moments \(e\.g\., forℓ≠m\\ell\\neq m,𝔼⁡\[𝟙​\(Nℓ\>0\)​𝟙​\(Nm\>0\)\]=1−\(1−pℓ\)n−\(1−pm\)n\+\(1−pℓ−pm\)n\\mathbb\{E\}\[\\mathbb\{1\}\(N\_\{\\ell\}\>0\)\\mathbb\{1\}\(N\_\{m\}\>0\)\]=1\-\(1\-p\_\{\\ell\}\)^\{n\}\-\(1\-p\_\{m\}\)^\{n\}\+\(1\-p\_\{\\ell\}\-p\_\{m\}\)^\{n\}\)\. The same structure applies toF^count=Φ1/Φobs\\widehat\{F\}\_\{\\mathrm\{count\}\}=\\Phi\_\{1\}/\\Phi\_\{\\mathrm\{obs\}\}with count\-based indicators\.

When many species carry smallpℓp\_\{\\ell\}at fixednn, singleton mass dominates andμA/μB\\mu\_\{A\}/\\mu\_\{B\}rises—predicting high fragility and unstable GT/GTo corrections\. Asnnincreases,μB→1\\mu\_\{B\}\\to 1and expected fragility typically falls unless the tail remains heavy \(Zipf/Dirichlet\)\. Structured head–tail splits can instead stabilize singleton statistics, explaining when GT remains competitive \(App\.[S\-II\-B](https://arxiv.org/html/2610.22604#as1_S2.SS2)\)\.

## S\-IIExtended Synthetic Results

### S\-II\-AZipf distributions

Table[S1](https://arxiv.org/html/2610.22604#as1_S2.T1)reports UCR for missing\-mass estimation on Zipf populations \(pi∝i−sp\_\{i\}\\propto i^\{\-s\},s∈\{0\.5,1\.0,1\.5\}s\\in\\\{0\.5,1\.0,1\.5\\\}\) across sample sizesNN\. Ours matches or beats GT at most\(s,N\)\(s,N\)while substantially outperforming SGT and Bayesian baselines at smallnn; absolute gaps are modest versus GT but large versus PY/Bayes–Dir\.

TABLE S1:UCR \(lower is better\) for missing\-mass estimation on Zipf populations across tail exponentssand sample sizeNN\.
### S\-II\-BHead–Tail mixtures

Head–tail populations assign fractionρ\\rhoof mass to a head ofS/2S/2species and1−ρ1\-\\rhoto a uniform tail\. Largerρ\\rhoconcentrates probability on frequent types, which can stabilize singleton counts relative to Zipf/Dirichlet tails—the main synthetic regime where GT occasionally matches or beats our adjustment on missing mass\. Table[S2](https://arxiv.org/html/2610.22604#as1_S2.T2)reports UCR for missing\-mass estimation on Head–Tail populations\.

TABLE S2:Representative UCR values for missing\-mass estimation on head–tail mixtures \(ρ\\rho= head\-mass fraction\)\. GT is within 0\.001–0\.002 of Ours at most settings; both dominate SGT and Bayesian baselines at smallNN\.
### S\-II\-CNormalized pairwise gain

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/improvements/family_bhg_ucr_mass.png)

\(a\) Missing mass

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/improvements/family_bhg_ucr_count.png)

\(b\) Unseen\-species prediction

Fig\. S1:Per\-family normalized pairwise gain for \(a\) missing\-mass and \(b\) unseen\-species estimation\.Fig\.[S1](https://arxiv.org/html/2610.22604#as1_S2.F1)presents the per\-family normalized pairwise gains for missing\-mass and unseen\-species estimation\. For missing\-mass \(Fig\.[S1](https://arxiv.org/html/2610.22604#as1_S2.F1)a\), larger gains are observed in Zipf and Dirichlet distributions, where rare\-event statistics are more fragile, while head–tail mixtures exhibit smaller improvements as GT remains competitive\. For unseen\-species estimation \(Fig\.[S1](https://arxiv.org/html/2610.22604#as1_S2.F1)b\), the fragility\-aware method achieves substantial gains over classical estimators such as Chao, ACE, and jackknife methods, highlighting its effectiveness in reducing estimation error under fragmented, small\-sample regimes\.

## S\-IIIExtended Evaluation of Fragility\-Aware Unseen\-Species Estimation

### S\-III\-AShakespeare Corpus: Missing Mass Estimation

Fig\.[S2](https://arxiv.org/html/2610.22604#as1_S3.F2)shows missing\-mass win rates across horizonsrfuturer\_\{\\mathrm\{future\}\}\. The fragility\-aware estimator stays competitive with classical methods as the horizon grows, while singleton\-driven estimators vary more\.

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/shakespear/winrate_by_mass_mratio_grid.png)Fig\. S2:Missing\-mass win\-rate heatmaps on Shakespeare acrossrfuture∈\[0\.03,0\.10\]r\_\{\\mathrm\{future\}\}\\in\[0\.03,0\.10\]\(fraction of experiments where the row beats the column on UCR\)\.
### S\-III\-BShakespeare Corpus: Missing Count Estimation

Fig\.[S3](https://arxiv.org/html/2610.22604#as1_S3.F3)shows unseen\-count win rates across horizons\. The fragility\-aware estimator remains robust under long\-tailed extrapolation where rare\-count baselines degrade\.

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/shakespear/winrate_by_count_mratio_grid.png)Fig\. S3:Unseen\-count win\-rate heatmaps on Shakespeare acrossrfuture∈\[0\.03,0\.10\]r\_\{\\mathrm\{future\}\}\\in\[0\.03,0\.10\]\.
### S\-III\-CGoogle n\-gram Corpus: Missing Mass Estimation

Fig\.[S4](https://arxiv.org/html/2610.22604#as1_S3.F4)shows missing\-mass win rates on Google 1\-gram\. The fragility\-aware estimator attains the highest win rates across horizons, while frequency\-driven estimators are more horizon\-sensitive in this heavy\-tailed corpus\.

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/google/winrate_by_mass_mratio_grid.png)Fig\. S4:Missing\-mass win\-rate heatmaps on Google 1\-gram acrossrfuture∈\[0\.03,0\.10\]r\_\{\\mathrm\{future\}\}\\in\[0\.03,0\.10\]\.
### S\-III\-DGoogle n\-gram Corpus: Missing Count Estimation

Fig\.[S5](https://arxiv.org/html/2610.22604#as1_S3.F5)shows unseen\-count win rates on Google 1\-gram\. Explicit fragility modeling improves robustness as the extrapolation horizon grows\.

![Refer to caption](https://arxiv.org/html/2610.22604v1/img/google/winrate_by_count_mratio_grid.png)Fig\. S5:Unseen\-count win\-rate heatmaps on Google 1\-gram acrossrfuture∈\[0\.03,0\.10\]r\_\{\\mathrm\{future\}\}\\in\[0\.03,0\.10\]\.

## References

- \[1\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1),[§I](https://arxiv.org/html/2610.22604#S1.p1.1),[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p3.2),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p5.1)\.
- \[2\]Cited by:[§I](https://arxiv.org/html/2610.22604#S1.p1.1)\.
- \[3\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1),[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p3.1),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p3.2),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p4.1),[§II\-B](https://arxiv.org/html/2610.22604#S2.SS2.p1.1)\.
- \[4\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1),[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p5.1),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p6.1),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p6.2)\.
- \[5\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1),[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2)\.
- \[6\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1),[§II\-B](https://arxiv.org/html/2610.22604#S2.SS2.p1.1)\.
- \[7\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1)\.
- \[8\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p1.1),[§II\-B](https://arxiv.org/html/2610.22604#S2.SS2.p1.1)\.
- \[9\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p2.1)\.
- \[10\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p2.1)\.
- \[11\]Cited by:[§I\-A](https://arxiv.org/html/2610.22604#S1.SS1.p3.1),[§II\-C](https://arxiv.org/html/2610.22604#S2.SS3.p5.1),[§II\-D](https://arxiv.org/html/2610.22604#S2.SS4.p4.1)\.
- \[12\]Cited by:[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p5.1)\.
- \[13\]Cited by:[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2),[§II\-A](https://arxiv.org/html/2610.22604#S2.SS1.p5.1)\.
- \[14\]Cited by:[§II\-B](https://arxiv.org/html/2610.22604#S2.SS2.p1.1)\.
- \[15\]Cited by:[§II\-B](https://arxiv.org/html/2610.22604#S2.SS2.p1.1)\.
- \[16\]Cited by:[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2)\.
- \[17\]Cited by:[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2)\.
- \[18\]Cited by:[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2)\.
- \[19\]Cited by:[Fig\. 2](https://arxiv.org/html/2610.22604#S2.F2)\.
- \[20\]Cited by:[§II\-C](https://arxiv.org/html/2610.22604#S2.SS3.p2.1)\.
- \[21\]Cited by:[§III\-B](https://arxiv.org/html/2610.22604#S3.SS2.p1.1)\.
- \[22\]Cited by:[§III\-B](https://arxiv.org/html/2610.22604#S3.SS2.p1.1)\.
- \[23\]Cited by:[§IV\-A](https://arxiv.org/html/2610.22604#S4.SS1.p1.1)\.