A Calculus-Based Framework for Determining Vocabulary Size in End-to-End ASR

# A Calculus-Based Framework for Determining Vocabulary Size in End-to-End ASR
Source: [https://arxiv.org/html/2605.14427](https://arxiv.org/html/2605.14427)
###### Abstract

In hybrid automatic speech recognition \(ASR\) systems, the vocabulary size is unambiguous, typically determined by the number of phones, bi\-phones, or tri\-phones present in the language\. In contrast, end\-to\-end ASR systems derive their vocabulary, often referred to as tokens from the text corpus used for training\. The choice and, more importantly, the size of this vocabulary is a critical hyper\-parameter in training end\-to\-end ASR systems\. Tokenization algorithms such as Byte Pair Encoding \(BPE\), WordPiece, and Unigram Language Model \(ULM\) use the vocabulary size as an input hyper\-parameter to generate the sub\-words employed during ASR training\. Popular toolkits like ESPNet provide a fixed vocabulary size in their training recipes, but there is little documentation or discussion in the literature regarding how these values are determined\. Recent work\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\]has formalized an approach to identify the vocabulary size best suited for end\-to\-end ASR, introducing a cost function framework that treats the tokenization process as a black box\. In this paper, we build upon that foundation by curve fitting the training data and using the principle of first and second derivative tests in calculus to formally estimate the vocabulary size hyper\-parameter\. We demonstrate the utility and usefulness of our approach by applying it on a standard Librispeech corpus and show that the optimal choice of vocabulary size hyper\-parameter improves the performance of the ASR\. The main contribution of this paper in formalizing an approach to identify the vocabulary size best suited for training an end\-to\-end ASR system\.

## IIntroduction

End\-to\-end automatic speech recognition \(ASR\) systems have become the dominant paradigm for speech\-to\-text modeling by directly mapping acoustic feature sequences to discrete sub\-word or character sequences\. While this approach eliminates explicit phonetic and lexicon modeling, it introduces several representation\-level design choices that significantly affect model performance\. One such choice is the vocabulary size used in sub\-word tokenization, which determines the resolution of the discrete output space employed during training and decoding\.

In contrast to hybrid ASR systems, where the vocabulary is implicitly fixed by the phonetic inventory of the language, end\-to\-end ASR systems rely on data\-driven tokenization algorithms such as Byte Pair Encoding \(BPE\), WordPiece, and the Unigram Language Model\. These tokenization methods require the vocabulary size to be specified*a priori*\. Popular ASR tool\-kits, including ESPNet, typically adopt fixed heuristic values for this parameter; however, the rationale behind these choices is often undocumented, and the relationship between corpus statistics and the optimal vocabulary size remains poorly understood\.

Recent work\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\]addressed this issue by introducing a cost\-minimization framework that models the effect of vocabulary size through corpus\-derived statistics while treating the tokenizer as a black box\. Although effective, the proposed approach relies on empirical grid search over candidate vocabulary sizes and does not explicitly characterize the analytical structure of the resulting cost function\.

In this paper, we extend this line of work by developing a calculus\-based framework for estimating the vocabulary size in end\-to\-end ASR systems\. We view vocabulary size as a*representation resolution parameter*, analogous to dictionary size in vector quantization or model order in statistical signal processing\. By explicitly modeling corpus\-dependent cost components as smooth, differentiable functions of the vocabulary size, we derive first\- and second\-order optimality conditions that enable principled estimation of the optimal vocabulary size without exhaustive experimentation\.

Specifically, we incorporate normalization of cost components based on corpus statistics to ensure numerical stability and interpretability across datasets\. We further employ improved curve\-fitting techniques using second\-order polynomials and polynomial–exponential models to capture the observed behavior of token imbalance and sequence length growth as functions of vocabulary size\. The resulting formulation allows the optimal vocabulary size to be obtained by solving a constrained stationary\-point problem, providing analytical insight into how corpus characteristics govern tokenizer design\.

Experimental validation on the LibriSpeech\-100 corpus using a state\-of\-the\-art conformer\-based ASR model demonstrates that vocabulary sizes estimated by the proposed framework achieve competitive or improved word error rates compared to commonly used heuristic choices\. These results highlight the utility of analytically guided representation design in end\-to\-end ASR systems\. The main contributions of this paper are summarized as follows:

- •We formulate vocabulary size selection in end\-to\-end ASR as a continuous optimization problem by modeling corpus\-derived cost components as differentiable functions of vocabulary size\.
- •We derive first\- and second\-order optimality conditions that characterize the existence of an optimal vocabulary size, enabling analytical or numerical estimation without exhaustive grid search\.
- •We introduce normalization of cost components based on corpus statistics, allowing stable optimization and meaningful interpretation of cost weights across datasets\.
- •We empirically validate the proposed framework on the LibriSpeech\-100 corpus using a conformer\-based ASR system and demonstrate competitive or improved word error rates relative to widely adopted heuristic vocabulary sizes\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2605.14427#S2)introduces the extended problem formulation and analytical framework\. Section[III](https://arxiv.org/html/2605.14427#S3)presents the experimental setup and evaluates the proposed approach on a standard ASR benchmark\. Section[IV](https://arxiv.org/html/2605.14427#S4)concludes the paper and discusses limitations and future directions\.

## IIProblem Setup

We adopt the problem formulation and notation from\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\], where the vocabulary size is determined by minimizing a cost function that jointly accounts for \(i\) the number of tokens, \(ii\) class imbalance, and \(iii\) computational cost\. Specifically, the cost function comprises three terms: the vocabulary sizenn, the ratio of frequent to infrequent token occurrencesΔ​\(n\)\\Delta\(n\), and the total number of tokens required to cover the corpusΘ​\(n\)\\Theta\(n\)\. For completeness, we restate the cost function𝒞​\(n\)\{\\cal\{C\}\}\(n\)from\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\]as

𝒞​\(n\)=\{α1​n⏞t1\+α2​Δ​\(n\)⏞t2\+α3​Θ​\(n\)⏞t3\}\{\\cal\{C\}\}\(n\)=\\left\\\{\\alpha\_\{1\}\\overbrace\{n\}^\{t\_\{1\}\}\+\\alpha\_\{2\}\\overbrace\{\\Delta\(n\)\}^\{t\_\{2\}\}\+\\alpha\_\{3\}\\overbrace\{\\Theta\(n\)\}^\{t\_\{3\}\}\\right\\\}\(1\)whereα1,2,3\\alpha\_\{1,2,3\}denote the weights associated with the respective componentst1,2,3t\_\{1,2,3\}\. The optimal vocabulary size corresponds to

n∗=arg⁡minn⁡\{𝒞​\(n\)\}n^\{\*\}=\\arg\\min\_\{n\}\\left\\\{\{\\cal\{C\}\}\(n\)\\right\\\}\(2\)
In this work, we extend the formulation by normalizing the cost components to ensure stability and comparability across corpora\. The cost terms are modeled using second\-order polynomials and polynomial–exponential forms to accurately capture their empirical behavior\. The optimal vocabulary size is obtained analytically by solving the first\- and second\-order optimality conditions\. This provides a principled mechanism for estimating the vocabulary\-size hyper\-parameter in end\-to\-end ASR systems, which constitutes the main contribution of this work\.

### II\-AFinding optimal number of tokens \(nn\)

We first compute the first derivative of𝒞​\(n\)\{\\cal\{C\}\}\(n\)with respect tonn:

d​𝒞​\(n\)d​n=α1\+α2​Δ′​\(n\)\+α3​Θ′​\(n\),\\frac\{d\{\\cal\{C\}\}\(n\)\}\{dn\}=\\alpha\_\{1\}\+\\alpha\_\{2\}\\Delta^\{\\prime\}\(n\)\+\\alpha\_\{3\}\\Theta^\{\\prime\}\(n\),\(3\)and set it to zero, yielding

α1\+α2​Δ′​\(n\)\+α3​Θ′​\(n\)=0\.\\alpha\_\{1\}\+\\alpha\_\{2\}\\Delta^\{\\prime\}\(n\)\+\\alpha\_\{3\}\\Theta^\{\\prime\}\(n\)=0\.\(4\)We assume that the first derivativesΔ′​\(n\)\\Delta^\{\\prime\}\(n\)andΘ′​\(n\)\\Theta^\{\\prime\}\(n\)exist\. Solving \([4](https://arxiv.org/html/2605.14427#S2.E4)\) yields a candidate solution fornn, whose value depends on the weighting coefficientsα1,2,3\\alpha\_\{1,2,3\}\.

To verify that this solution corresponds to a minimum, we compute the second derivative

d2​𝒞​\(n\)d​n2=α2​Δ′′​\(n\)\+α3​Θ′′​\(n\),\\frac\{d^\{2\}\{\\cal\{C\}\}\(n\)\}\{dn^\{2\}\}=\\alpha\_\{2\}\\Delta^\{\\prime\\prime\}\(n\)\+\\alpha\_\{3\}\\Theta^\{\\prime\\prime\}\(n\),\(5\)assuming the existence ofΔ′′​\(n\)\\Delta^\{\\prime\\prime\}\(n\)andΘ′′​\(n\)\\Theta^\{\\prime\\prime\}\(n\)\. The solutionnnobtained from \([4](https://arxiv.org/html/2605.14427#S2.E4)\) minimizes𝒞​\(n\)\{\\cal\{C\}\}\(n\)if \([5](https://arxiv.org/html/2605.14427#S2.E5)\) is strictly positive\.

### II\-BPolynomial\-Based Functional Modeling

The functionsΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)are modeled using \(i\) a second\-order polynomial and \(ii\) a second\-order polynomial augmented with an exponential term, respectively\. This choice ensures the existence of well\-defined first\- and second\-order derivatives while providing sufficient flexibility to capture empirical trends\.

#### II\-B1Second\-Order Polynomial Fit

We first modelΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)using second\-order polynomials:

Δ​\(n\)≜d2​n2\+d1​n\+d0,\\Delta\(n\)\\triangleq d\_\{2\}n^\{2\}\+d\_\{1\}n\+d\_\{0\},\(6\)Θ​\(n\)≜f2​n2\+f1​n\+f0\.\\Theta\(n\)\\triangleq f\_\{2\}n^\{2\}\+f\_\{1\}n\+f\_\{0\}\.\(7\)The corresponding first\- and second\-order derivatives follow directly:

Δ′​\(n\)=2​d2​n\+d1,Θ′​\(n\)=2​f2​n\+f1,\\Delta^\{\\prime\}\(n\)=2d\_\{2\}n\+d\_\{1\},\\quad\\Theta^\{\\prime\}\(n\)=2f\_\{2\}n\+f\_\{1\},\(8\)Δ′′​\(n\)=2​d2,Θ′′​\(n\)=2​f2\.\\Delta^\{\\prime\\prime\}\(n\)=2d\_\{2\},\\quad\\Theta^\{\\prime\\prime\}\(n\)=2f\_\{2\}\.\(9\)Substituting these into \([4](https://arxiv.org/html/2605.14427#S2.E4)\) yields

n=−\(α1\+α2​d1\+α3​f1\)2​\(α2​d2\+α3​f2\)\.n=\\frac\{\-\(\\alpha\_\{1\}\+\\alpha\_\{2\}d\_\{1\}\+\\alpha\_\{3\}f\_\{1\}\)\}\{2\(\\alpha\_\{2\}d\_\{2\}\+\\alpha\_\{3\}f\_\{2\}\)\}\.\(10\)The solution corresponds to a minimum ifd2​𝒞d​n2\>0\\frac\{d^\{2\}\{\\cal\{C\}\}\}\{dn^\{2\}\}\>0, namely

2​\(α2​d2\+α3​f2\)\>0,2\(\\alpha\_\{2\}d\_\{2\}\+\\alpha\_\{3\}f\_\{2\}\)\>0,\(11\)and yields a positivennprovided \(numerator in \([10](https://arxiv.org/html/2605.14427#S2.E10)\) is negative\)

α1\+α2​d1\+α3​f1<0\.\\alpha\_\{1\}\+\\alpha\_\{2\}d\_\{1\}\+\\alpha\_\{3\}f\_\{1\}<0\.\(12\)Note thatnnis dependent on the values ofα1,2,3\\alpha\_\{1,2,3\}which need to be chosen heuristically\.

#### II\-B2Polynomial–Exponential Model

Empirical observations \(see Section[III\-A](https://arxiv.org/html/2605.14427#S3.SS1)\) indicate that a pure second\-order polynomial does not adequately model the behavior ofΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\), both of which exhibit exponential\-like trends\. To address this, we augment the polynomial model with an exponential term:

Δexp​\(n\)≜g3​n2\+g2​n\+g1​e1/n\+g0,\\Delta\_\{\\mathrm\{exp\}\}\(n\)\\triangleq g\_\{3\}n^\{2\}\+g\_\{2\}n\+g\_\{1\}e^\{1/n\}\+g\_\{0\},\(13\)Θexp​\(n\)≜h3​n2\+h2​n\+h1​e1/n\+h0\.\\Theta\_\{\\mathrm\{exp\}\}\(n\)\\triangleq h\_\{3\}n^\{2\}\+h\_\{2\}n\+h\_\{1\}e^\{1/n\}\+h\_\{0\}\.\(14\)Solving the resulting nonlinear equation numerically under the constraint

d2​𝒞​\(n\)d​n2\\displaystyle\\frac\{d^\{2\}\{\\cal\{C\}\}\(n\)\}\{dn^\{2\}\}=α2​\(2​g3\+3​g1​e1/nn4\)\\displaystyle=\\alpha\_\{2\}\\\!\\left\(2g\_\{3\}\+3g\_\{1\}\\frac\{e^\{1/n\}\}\{n^\{4\}\}\\right\)\+α3​\(2​h3\+3​h1​e1/nn4\)\>0\\displaystyle\+\\alpha\_\{3\}\\\!\\left\(2h\_\{3\}\+3h\_\{1\}\\frac\{e^\{1/n\}\}\{n^\{4\}\}\\right\)\>0\(15\)yields the optimal vocabulary size \(see Appendix[A](https://arxiv.org/html/2605.14427#A1)\)\.

## IIIExperimental Analysis

To estimate the optimal vocabulary sizennfollowing the methodology described in the previous section, we first modelΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)andselectthe weighting coefficientsα1,2,3\\alpha\_\{1,2,3\}to construct the cost function in \([1](https://arxiv.org/html/2605.14427#S2.E1)\)\. All experiments are conducted on the LibriSpeech\-100 corpus\[[8](https://arxiv.org/html/2605.14427#bib.bib23)\]\. LibriSpeech\-100 consists of 100 hours of read English speech along with corresponding text transcriptions\. The training set containsk=28,538\{k\}=28\{,\}538sentences comprisingw=990,093\{w\}=990\{,\}093word tokens, of whichwu=33,798\{w\}\_\{u\}=33\{,\}798are unique\. The corpus contains a total ofc=5,298,301c=5\{,\}298\{,\}301characters, withcu=28c\_\{u\}=28unique characters\.

The LibriSpeech\-100 dataset is chosen for two reasons: \(i\) it is one of the most widely used benchmarks for training low\-resource end\-to\-end ASR systems, and \(ii\) it is supported by a well\-established training recipe in the ESPNet toolkit, which we employ in our experiments\.

We computeΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)for values ofnnin the rangecu≤n≤5000c\_\{u\}\\leq n\\leq 5000using the LibriSpeech\-100 corpus\. Figure[1\(a\)](https://arxiv.org/html/2605.14427#S3.F1.sf1)illustrates the behavior ofΔ​\(n\)\\Delta\(n\), while Figure[1\(b\)](https://arxiv.org/html/2605.14427#S3.F1.sf2)shows the correspondingΘ​\(n\)\\Theta\(n\)curve, both derived directly from the corpus statistics\.

### III\-ASecond Order Polynomial Fit

We now fit a second\-order polynomial \(as described in Section[II\-B1](https://arxiv.org/html/2605.14427#S2.SS2.SSS1)\) using thecurve\_fitfunction from thescipy\.optimizemodule in theSciPylibrary\. The estimated coefficients forΔ​\(n\)\\Delta\(n\)ared2=6\.89∗10−5d\_\{2\}=6\.89\*10^\{\-5\},d1=0\.24d\_\{1\}=0\.24, andd0=21\.23d\_\{0\}=21\.23, with anR2R^\{2\}value of1\.001\.00\. Similarly, forΘ​\(n\)\\Theta\(n\), we obtainf2=0\.101f\_\{2\}=0\.101,f1=−716\.87f\_\{1\}=\-716\.87, andf0=2\.47∗106f\_\{0\}=2\.47\*10^\{6\}, with anR2R^\{2\}value of0\.730\.73\. The fitted models are given by

Δ​\(n\)\\displaystyle\\Delta\(n\)=\\displaystyle=6\.89∗10−5​n2\+0\.24​n\+21\.23\\displaystyle 6\.89\*10^\{\-5\}n^\{2\}\+0\.24n\+21\.23Θ​\(n\)\\displaystyle\\Theta\(n\)=\\displaystyle=0\.101n2\+−716\.87n\+2\.47∗106\\displaystyle 0\.101n^\{2\}\+\-716\.87n\+2\.47\*10^\{6\}\(16\)
TheR2R^\{2\}metric \(coefficient of determination\) indicates the goodness of fit, with values closer to11implying a better fit\[[5](https://arxiv.org/html/2605.14427#bib.bib26)\]\. The fittedΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)curves are shown in red in Figures[1\(a\)](https://arxiv.org/html/2605.14427#S3.F1.sf1)and[1\(b\)](https://arxiv.org/html/2605.14427#S3.F1.sf2), respectively, while the empirical curves derived from the LibriSpeech\-100 corpus are shown in blue\. As observed earlier, the second\-order polynomial fit does not accurately captureΔ​\(n\)\\Delta\(n\)forn\>2500n\>2500, and similarly fails to modelΘ​\(n\)\\Theta\(n\)well forn<2500n<2500\.

![Refer to caption](https://arxiv.org/html/2605.14427v1/deltan-polyfit.png)\(a\)Δ​\(n\)\\Delta\(n\)\.
![Refer to caption](https://arxiv.org/html/2605.14427v1/thetan-polyfit.png)\(b\)Θ​\(n\)\\Theta\(n\)\.

Figure 1:Second\-order polynomial fit \(Eq\. \([16](https://arxiv.org/html/2605.14427#S3.E16)\)\) of \(a\)Δ​\(n\)\\Delta\(n\)and \(b\)Θ​\(n\)\\Theta\(n\)\. The fitted curves are shown in red, while the empirical curves derived from the LibriSpeech\-100 corpus are shown in blue\.Using Eq\. \([10](https://arxiv.org/html/2605.14427#S2.E10)\), the minimizing value ofnnis given by

n=716\.87​α3−α1−0\.24​α22​\(6\.89∗10−5​α2\+0\.101​α3\)n=\\frac\{716\.87\\alpha\_\{3\}\-\\alpha\_\{1\}\-0\.24\\alpha\_\{2\}\}\{2\(6\.89\*10^\{\-5\}\\alpha\_\{2\}\+0\.101\\alpha\_\{3\}\)\}\(17\)subject to the conditions that

2​\(6\.89∗10−5​α2\+0\.101​α3\)\>02\(6\.89\*10^\{\-5\}\\alpha\_\{2\}\+0\.101\\alpha\_\{3\}\)\>0\(18\)and

716\.87​α3−α1−0\.24​α2<0\.716\.87\\alpha\_\{3\}\-\\alpha\_\{1\}\-0\.24\\alpha\_\{2\}<0\.\(19\)As noted earlier, the identified value ofnndepends on the choice ofα1,2,3\\alpha\_\{1,2,3\}and can be computed only once these weights are specifieda priori\.

#### III\-A1Findingnn

State\-of\-the\-art automatic speech recognition \(ASR\) systems commonly employ the Conformer encoder\-decoder architecture\. The Conformer model implemented in the ESPNet toolkit\[[11](https://arxiv.org/html/2605.14427#bib.bib10)\], using the LibriSpeech\-100 \(low\-resource\) recipe, recommends an SentencePiece\-Unigram language model withn=300n=300\. We verified that multiple configurations ofα1,2,3\\alpha\_\{1,2,3\}map to the same heuristic value ofnn, which motivates the need for normalization \(see Appendix[B](https://arxiv.org/html/2605.14427#A2)\)\.

To determine an optimal value ofnn, greater control over the admissible range ofα1,2,3\\alpha\_\{1,2,3\}is required so that an optimalnncan be obtained via \([17](https://arxiv.org/html/2605.14427#S3.E17)\)\. We address this next\.

Observe that the ranges of the three termst1,2,3t\_\{1,2,3\}vary significantly \(Table[I](https://arxiv.org/html/2605.14427#S3.T1)\)\.

TABLE I:Range of the termst1,2,3t\_\{1,2,3\}\. Here,cuc\_\{u\}denotes the number of unique characters,wu\{w\}\_\{u\}the number of unique words,fc\+f\_\{c\}^\{\+\}the frequency of the most frequent character, andccthe total number of characters\.From the corpus, we can directly compute the total number of words \(w\{w\}\), the number of unique words \(wu\{w\}\_\{u\}\), the total number of characters \(cc\), the number of unique characters \(cuc\_\{u\}\), and the frequency of the most frequent character \(fc\+f\_\{c\}^\{\+\}\)\. Hence, the minimum and maximum values oft1,2,3t\_\{1,2,3\}in Table[I](https://arxiv.org/html/2605.14427#S3.T1)are knowna priori\. For example, the termt1=nt\_\{1\}=ncan take values in the range\[cu,wu\]\[c\_\{u\},\{w\}\_\{u\}\]\.

Normalizingt1,2,3t\_\{1,2,3\}using these bounds allows us to define a normalized cost function

𝒞n​o​r​m=α1′​t1n​o​r​m\+α2′​t2n​o​r​m\+α3′​t3n​o​r​m,\{\\cal\{C\}\}^\{norm\}=\\alpha^\{\\prime\}\_\{1\}t^\{norm\}\_\{1\}\+\\alpha^\{\\prime\}\_\{2\}t^\{norm\}\_\{2\}\+\\alpha^\{\\prime\}\_\{3\}t^\{norm\}\_\{3\},\(20\)wheret1n​o​r​m=\(t1−cu\)/\(wu−cu\)t^\{norm\}\_\{1\}=\(t\_\{1\}\-c\_\{u\}\)/\(\{w\}\_\{u\}\-c\_\{u\}\),t2n​o​r​m=t2/fc\+t^\{norm\}\_\{2\}=t\_\{2\}/f\_\{c\}^\{\+\}, andt3n​o​r​m=t3/ct^\{norm\}\_\{3\}=t\_\{3\}/c\. The weightsα1,2,3′\\alpha^\{\\prime\}\_\{1,2,3\}now reflect the relative importance assigned to the normalized components\. Substituting fort1,2,3n​o​r​mt^\{norm\}\_\{1,2,3\}, we obtain

𝒞n​o​r​m=α1′​\(n−cuwu−cu\)\+α2′​Δn​o​r​m​\(n\)\+α3′​Θn​o​r​m​\(n\),\{\\cal\{C\}\}^\{norm\}=\\alpha^\{\\prime\}\_\{1\}\\left\(\\frac\{n\-c\_\{u\}\}\{\{w\}\_\{u\}\-c\_\{u\}\}\\right\)\+\\alpha^\{\\prime\}\_\{2\}\\Delta^\{norm\}\(n\)\+\\alpha^\{\\prime\}\_\{3\}\\Theta^\{norm\}\(n\),\(21\)whereΔn​o​r​m​\(n\)=Δ​\(n\)/fc\+\\Delta^\{norm\}\(n\)=\\Delta\(n\)/f\_\{c\}^\{\+\}andΘn​o​r​m​\(n\)=Θ​\(n\)/c\\Theta^\{norm\}\(n\)=\\Theta\(n\)/c\.

As before, we fit second\-order polynomials toΔn​o​r​m​\(n\)\\Delta^\{norm\}\(n\)andΘn​o​r​m​\(n\)\\Theta^\{norm\}\(n\), yielding \(see Fig\.[2](https://arxiv.org/html/2605.14427#S3.F2)\)

Δn​o​r​m​\(n\)\\displaystyle\\Delta^\{norm\}\(n\)=\\displaystyle=2\.48∗10−8​n2−1\.76∗10−4​n\+3\.06∗10−3\\displaystyle 2\.48\*10^\{\-8\}n^\{2\}\-1\.76\*10^\{\-4\}n\+3\.06\*10^\{\-3\}Θn​o​r​m​\(n\)\\displaystyle\\Theta^\{norm\}\(n\)=\\displaystyle=2\.37∗10−8​n2\+8\.37∗10−5​n−3\.40∗10−3\.\\displaystyle 2\.37\*10^\{\-8\}n^\{2\}\+8\.37\*10^\{\-5\}n\-3\.40\*10^\{\-3\}\.
The minimizing value ofnnis then given by

n=−\(8\.37∗10−5​α3′\+α1′−1\.76∗10−4​α2′\)2​\(2\.48∗10−8​α2′\+2\.37∗10−8​α3′\),n=\\frac\{\-\(8\.37\*10^\{\-5\}\\alpha^\{\\prime\}\_\{3\}\+\\alpha^\{\\prime\}\_\{1\}\-1\.76\*10^\{\-4\}\\alpha^\{\\prime\}\_\{2\}\)\}\{2\(2\.48\*10^\{\-8\}\\alpha^\{\\prime\}\_\{2\}\+2\.37\*10^\{\-8\}\\alpha^\{\\prime\}\_\{3\}\)\},\(23\)subject to

2​\(2\.48∗10−8​α2′\+2\.37∗10−8​α3′\)\>02\(2\.48\*10^\{\-8\}\\alpha^\{\\prime\}\_\{2\}\+2\.37\*10^\{\-8\}\\alpha^\{\\prime\}\_\{3\}\)\>0\(24\)and

\(α1′−1\.76∗10−4​α2′\+8\.37∗10−5​α3′\)<0,\(\\alpha^\{\\prime\}\_\{1\}\-1\.76\*10^\{\-4\}\\alpha^\{\\prime\}\_\{2\}\+8\.37\*10^\{\-5\}\\alpha^\{\\prime\}\_\{3\}\)<0,\(25\)to ensuren\>0n\>0\.

![Refer to caption](https://arxiv.org/html/2605.14427v1/td2td3_5k_polyfit.png)Figure 2:Δn​o​r​m​\(n\)\\Delta^\{norm\}\(n\)\(right\),Θn​o​r​m​\(n\)\\Theta^\{norm\}\(n\)\(left\) shown in red are the result of second order polynomial fit \(Eq \(LABEL:eq:polyfit\_norm\)\) whileΔn​o​r​m​\(n\)\\Delta^\{norm\}\(n\)andΘn​o​r​m​\(n\)\\Theta^\{norm\}\(n\)derived from LibriSpeech\-100 corpus is shown as the blue curve\. The y\-axis is between0and11because of the normalization as seen in \(LABEL:eq:polyfit\_norm\)\.Algorithm 1Findnn\(using second order polynomial\)1:

d2=2\.48∗10−8d\_\{2\}=2\.48\*10^\{\-8\},

d1=−1\.76∗10−4d\_\{1\}=\-1\.76\*10^\{\-4\},

d0=3\.06∗10−3d\_\{0\}=3\.06\*10^\{\-3\}
2:

f2=2\.37∗10−8f\_\{2\}=2\.37\*10^\{\-8\},

f1=8\.37∗10−5f\_\{1\}=8\.37\*10^\{\-5\},

f0=−3\.40∗10−3f\_\{0\}=\-3\.40\*10^\{\-3\}\{As mentioned in \(LABEL:eq:polyfit\_norm\)\}

3:

4:Define the objective function, constraints, initialization:

5:

𝚘𝚋𝚓​\(α1′,α2′,α3′,f1,f2,d1,d2\)\{\\tt obj\}\(\\alpha^\{\\prime\}\_\{1\},\\alpha^\{\\prime\}\_\{2\},\\alpha^\{\\prime\}\_\{3\},f\_\{1\},f\_\{2\},d\_\{1\},d\_\{2\}\)
6:

n=−\(f1∗α3′\+α1′\+d1∗α2′\)\(2∗\(d2∗α2′\+f2∗α3′\)\)n=\\frac\{\-\(f\_\{1\}\*\\alpha^\{\\prime\}\_\{3\}\+\\alpha^\{\\prime\}\_\{1\}\+d\_\{1\}\*\\alpha^\{\\prime\}\_\{2\}\)\}\{\(2\*\(d\_\{2\}\*\\alpha^\{\\prime\}\_\{2\}\+f\_\{2\}\*\\alpha^\{\\prime\}\_\{3\}\)\)\}\{Equation \([23](https://arxiv.org/html/2605.14427#S3.E23)\)\}

7:return

nn
8:

𝚌𝚘𝚗𝚜←2​\(d2​α2′\+f2​α3′\)\>0,−\(f1​α3′\+α1′\+d1​α2′\)<0\{\\tt cons\}\\leftarrow 2\(d\_\{2\}\\alpha^\{\\prime\}\_\{2\}\+f\_\{2\}\\alpha^\{\\prime\}\_\{3\}\)\>0,\-\(f\_\{1\}\\alpha^\{\\prime\}\_\{3\}\+\\alpha^\{\\prime\}\_\{1\}\+d\_\{1\}\\alpha^\{\\prime\}\_\{2\}\)<0,

0≤α1,2,3′≤10\\leq\\alpha^\{\\prime\}\_\{1,2,3\}\\leq 1and

∑i=13αi′=1\\sum\_\{i=1\}^\{3\}\\alpha^\{\\prime\}\_\{i\}=1\{Equations \([24](https://arxiv.org/html/2605.14427#S3.E24)\) and \([25](https://arxiv.org/html/2605.14427#S3.E25)\)\}

9:

𝚒𝚗𝚒𝚝←𝚗𝚙\.𝚛𝚊𝚗𝚍𝚘𝚖\.𝚛𝚊𝚗𝚍​\(𝟹\)\{\\tt init\}\\leftarrow\{\\tt np\.random\.rand\(3\)\}
10:

11:Optimize:

12:

result←𝚖𝚒𝚗𝚒𝚖𝚒𝚣𝚎\(𝚘𝚋𝚓,𝚒𝚗𝚒𝚝,constraints=𝚌𝚘𝚗𝚜,method=𝚂𝙻𝚂𝚀𝙿′′\)result\\leftarrow\{\\tt minimize\}\(\{\\tt obj\},\{\\tt init\},constraints=\{\\tt cons\},method=\{\\tt\{\}^\{\\prime\}SLSQP^\{\\prime\}\}\)
13:if

r​e​s​u​l​t\.s​u​c​c​e​s​sresult\.successthen

14:

α1′,α2′,α3′←r​e​s​u​l​t\.x\\alpha^\{\\prime\}\_\{1\},\\alpha^\{\\prime\}\_\{2\},\\alpha^\{\\prime\}\_\{3\}\\leftarrow result\.x
15:else

16:choose another

𝚒𝚗𝚒𝚝\{\\tt init\}\{not all initial values ofα\\alpha’s result in a solution\}

17:Go to Line \#[9](https://arxiv.org/html/2605.14427#alg1.l9)

18:endif

TABLE II:WERs \(in %;↓\\downarrowbetter\) on LibriSpeech\-100\. “avg”: average over “clean” and “other”\.
#### III\-A2Performance on Speech Recognition

As described earlier, our experiments use the LibriSpeech\-100 corpus, consisting of 100 hours of read English speech with corresponding text transcriptions\[[8](https://arxiv.org/html/2605.14427#bib.bib23)\]\. The text transcripts are used to estimate the optimal number of tokens \(nn\), and the effectiveness of the estimated vocabulary size is then validated by training ASR systems using the same tokens\. We report recognition performance on the “test\-clean”\[[6](https://arxiv.org/html/2605.14427#bib.bib24)\]and “test\-other”\[[7](https://arxiv.org/html/2605.14427#bib.bib25)\]evaluation sets \(sentences:2,9392\{,\}939; words:52,34352\{,\}343\)\.

We employ the state\-of\-the\-art Conformer encoder–decoder architecture\[[1](https://arxiv.org/html/2605.14427#bib.bib49)\]for ASR\. The Conformer model is implemented using the ESPNet toolkit\[[11](https://arxiv.org/html/2605.14427#bib.bib10)\]with the LibriSpeech\-100 \(low\-resource\) training recipe\. To ensure a fair comparison, we modify*only*the number of sub\-word units hyper\-parameter in the original recipe while keeping all other model hyper\-parameters unchanged\.

The encoder consists of 12 layers and the decoder consists of 6 layers\. The model dimension is 256 with 4 attention heads\. Training is performed using the Adam optimizer\[[2](https://arxiv.org/html/2605.14427#bib.bib11)\]withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.98\\beta\_\{2\}=0\.98, andϵ=10−9\\epsilon=10^\{\-9\}, following the optimization strategy in\[[10](https://arxiv.org/html/2605.14427#bib.bib12)\]\. A warm\-up schedule of 25 000 steps is used, and all models are trained for 100 epochs with a batch size of 64, consistent with the ESPNet recipe\. All experiments are conducted using a single NVIDIA RTX 3090 GPU\.

No language model is employed during decoding via shallow fusion\. The sub\-word units derived from the training text serve as the output labels for the ASR model\. The input features consist of 80\-dimensional log Mel spectrograms augmented with pitch, resulting in a total of 81\-dimensional feature vectors\. Additionally, standard data augmentation techniques are applied, including three\-way speed perturbation\[[3](https://arxiv.org/html/2605.14427#bib.bib2)\]with factors 0\.9, 1\.0, and 1\.1, along with SpecAugment\[[9](https://arxiv.org/html/2605.14427#bib.bib6)\]\.

Table[II](https://arxiv.org/html/2605.14427#S3.T2)summarizes the ASR performance in terms of word error rate \(WER; lower is better\) for the default ESPNet setting \(n=300n=300\) and for the estimated optimal vocabulary size \(n∗=382n^\{\*\}=382\)\. While the overall performance forn=300n=300andn∗=382n^\{\*\}=382is comparable, a modest improvement is observed forn∗n^\{\*\}on the “test\-avg” set \(14\.35 versus 14\.55\)\. For “dev\-avg”, the performance withn∗=382n^\{\*\}=382\(13\.95\) is close to that obtained withn=300n=300\(13\.85\)\.

We emphasize that the commonly used value ofn=300n=300, to the best of our knowledge, lacks a principled justification in the literature\. In contrast, this work proposes a formal, analytically grounded framework for estimating the vocabulary\-size hyper\-parameter, yieldingn∗=382n^\{\*\}=382for LibriSpeech\-100\.

However, we note that the second\-order polynomial does not adequately fitΘn​o​r​m​\(n\)\\Theta^\{norm\}\(n\), as discussed earlier and illustrated in Fig\.[2](https://arxiv.org/html/2605.14427#S3.F2)\. This observation motivates the exploration of a more expressive model by incorporating an exponential term alongside the second\-order polynomial\. We hypothesize that improved curve fitting of the empirical data can lead to a more accurate estimate of the optimal vocabulary sizen∗n^\{\*\}\.

### III\-BExponential and Second\-Order Polynomial Fit

As discussed in Section[II\-B2](https://arxiv.org/html/2605.14427#S2.SS2.SSS2), we augment the second\-order polynomial with an exponential term to modelΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)\. The parameters of the resulting functions are estimated using thecurve\_fitroutine from thescipy\.optimizemodule in theSciPylibrary\. The fitted models are given by

Δe​x​p​\(n\)\\displaystyle\\Delta\_\{exp\}\(n\)=6\.8​e−05∗n2\+2\.47​e−01∗n\\displaystyle=6\.8e\-05\*n^\{2\}\+2\.47e\-01\*n\+1\.15​e\+03∗e\(1n\)−1\.14​e\+03\\displaystyle\+1\.15e\+03\*e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\-1\.14e\+03Θe​x​p​\(n\)\\displaystyle\\Theta\_\{exp\}\(n\)=3\.8​e−02∗n2−3\.12​e\+02∗n\\displaystyle=3\.8e\-02\*n^\{2\}\-3\.12e\+02\*n\+1\.12​e\+08∗e\(1n\)−1\.11​e\+08\\displaystyle\+1\.12e\+08\*e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\-1\.11e\+08\(26\)where the fitted parameters forΔexp​\(n\)\\Delta\_\{\\mathrm\{exp\}\}\(n\),\{g3,g2,g1,g0\}=\{6\.8​e−05,2\.47​e−01,1\.15​e\+03,−1\.14​e\+03\}\\\{g\_\{3\},g\_\{2\},g\_\{1\},g\_\{0\}\\\}=\\\{6\.8e\-05,2\.47e\-01,1\.15e\+03,\-1\.14e\+03\\\}, yield anR2R^\{2\}value of1\.001\.00, and those forΘexp​\(n\)\\Theta\_\{\\mathrm\{exp\}\}\(n\),\{h3,h2,h1,h0\}=\{3\.8​e−02,−3\.12​e\+02,1\.12​e\+08,−1\.11​e\+08\}\\\{h\_\{3\},h\_\{2\},h\_\{1\},h\_\{0\}\\\}=\\\{3\.8e\-02,\-3\.12e\+02,1\.12e\+08,\-1\.11e\+08\\\}, yield anR2R^\{2\}value of0\.990\.99\.

The improved goodness of fit is evident from the red curves in Fig\.[3\(a\)](https://arxiv.org/html/2605.14427#S3.F3.sf1)forΔexp​\(n\)\\Delta\_\{\\mathrm\{exp\}\}\(n\)and Fig\.[3\(b\)](https://arxiv.org/html/2605.14427#S3.F3.sf2)forΘexp​\(n\)\\Theta\_\{\\mathrm\{exp\}\}\(n\)\. We hypothesize that this improved functional representation enables a more accurate estimation of the optimal vocabulary sizenn\.

As before, solving the first\-order optimality condition yields

2​n3​\(6\.8​e−05∗α2\+3\.8​e−02∗α3\)\\displaystyle 2n^\{3\}\\left\(6\.8e\-05\*\\alpha\_\{2\}\+3\.8e\-02\*\\alpha\_\{3\}\\right\)\+\\displaystyle\+\(27\)n2\(α1\+2\.47e−01∗α2\+−3\.12e\+02∗α3\)\\displaystyle n^\{2\}\\left\(\\alpha\_\{1\}\+2\.47e\-01\*\\alpha\_\{2\}\+\-3\.12e\+02\*\\alpha\_\{3\}\\right\)−\\displaystyle\-e\(1n\)​\(1\.15​e\+03∗α2\+1\.12​e\+08∗α3\)\\displaystyle e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\\left\(1\.15e\+03\*\\alpha\_\{2\}\+1\.12e\+08\*\\alpha\_\{3\}\\right\)=\\displaystyle=0\\displaystyle 0subject to the second\-order condition

α2​\(2∗6\.8​e−05\+3∗1\.15​e\+03∗e\(1n\)n4\)\\displaystyle\\alpha\_\{2\}\\left\(2\*6\.8e\-05\+3\*1\.15e\+03\*\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{4\}\}\\right\)\+\\displaystyle\+\(28\)α3​\(2∗3\.8​e−02\+3∗1\.12​e\+08∗e\(1n\)n4\)\\displaystyle\\alpha\_\{3\}\\left\(2\*3\.8e\-02\+3\*1\.12e\+08\*\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{4\}\}\\right\)\>\\displaystyle\>0\\displaystyle 0which ensures that the obtained solution corresponds to a minimum of the cost function\.

Equations \([III\-B](https://arxiv.org/html/2605.14427#S3.Ex9)\) and \([III\-B](https://arxiv.org/html/2605.14427#S3.Ex10)\) are obtained by substituting the fitted coefficients from \([26](https://arxiv.org/html/2605.14427#S3.E26)\) into the general expressions in \([30](https://arxiv.org/html/2605.14427#A1.E30)\) and \([15](https://arxiv.org/html/2605.14427#S2.E15)\), respectively\. We use thefsolvefunction from thescipy\.optimizemodule to numerically solve \([III\-B](https://arxiv.org/html/2605.14427#S3.Ex9)\) under the constraint \([III\-B](https://arxiv.org/html/2605.14427#S3.Ex10)\) \(see Algorithm[2](https://arxiv.org/html/2605.14427#alg2)\)\.

The solver is executed 25,000 times with different initializations ofα1,2,3\\alpha\_\{1,2,3\}, while enforcing an additional constraintn<500n<500\(Algorithm[2](https://arxiv.org/html/2605.14427#alg2), Line[18](https://arxiv.org/html/2605.14427#alg2.l18)\)\. In approximately 98% of the runs, the solver converges to values of

n∗∈\[\\displaystyle n^\{\*\}\\in\[57\.49,58\.27,58\.46,58\.65,58\.84,\\displaystyle 57\.49,\\;58\.27,\\;58\.46,\\;58\.65,\\;58\.84,\\;59\.23,59\.42,59\.81,60\.00,60\.58\],\\displaystyle 59\.23,\\;59\.42,\\;59\.81,\\;60\.00,\\;60\.58\],\(29\)which lie in a narrow range aroundn≈60n\\approx 60\.

Notably, it was shown in\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\]that the best\-performing end\-to\-end ASR system achieved its lowest error rates when the vocabulary size was set ton∗=61n^\{\*\}=61\. The values ofn∗∈\(57,61\)n^\{\*\}\\in\(57,61\)obtained here are therefore consistent with, and closely match, that empirically optimal setting\.

We emphasize that in\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\], the weightsα1,2,3\\alpha\_\{1,2,3\}were selected heuristically via grid search\. In contrast, the present work demonstrates that the optimal vocabulary size can be estimated analytically by identifying a double\-differentiable function that accurately fitsΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)derived from the training corpus\. As shown in Table[III](https://arxiv.org/html/2605.14427#S3.T3), the ASR system trained withn∗=61n^\{\*\}=61outperforms the commonly adopted ESPNet configuration withn=300n=300on both “dev\-avg” \(13\.20 versus 13\.85\) and “test\-avg” \(13\.60 versus 14\.55\) subsets of LibriSpeech\-100\.

The proposed framework relies on identifying a suitable twice\-differentiable functional form that provides a high\-quality curve fit \(i\.e\., highR2R^\{2\}values\) to the empiricalΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)statistics\. The dependence on the quality of this fit constitutes the primary limitation of the approach\.

![Refer to caption](https://arxiv.org/html/2605.14427v1/data_delta.png)\(a\)Δe​x​p​\(n\)\\Delta\_\{exp\}\(n\)\.
![Refer to caption](https://arxiv.org/html/2605.14427v1/data_theta.png)\(b\)Θe​x​p​\(n\)\\Theta\_\{exp\}\(n\)\.

Figure 3:Exponential and second order polynomial to representΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)\. Shown in Eq \([16](https://arxiv.org/html/2605.14427#S3.E16)\)\.Algorithm 2Findnn\(modeled using second order polynomial and exponential term\)1:Step 1: Generate Randomα1,α2,α3\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}

2:Generate 3 random values

r1,r2,r3r\_\{1\},r\_\{2\},r\_\{3\}between 0 and 1

3:Normalize:

α1=r1r1\+r2\+r3,α2=r2r1\+r2\+r3,α3=r3r1\+r2\+r3\\alpha\_\{1\}=\\frac\{r\_\{1\}\}\{r\_\{1\}\+r\_\{2\}\+r\_\{3\}\},\\alpha\_\{2\}=\\frac\{r\_\{2\}\}\{r\_\{1\}\+r\_\{2\}\+r\_\{3\}\},\\alpha\_\{3\}=\\frac\{r\_\{3\}\}\{r\_\{1\}\+r\_\{2\}\+r\_\{3\}\}
4:

5:Step 2: Define Equation \([III\-B](https://arxiv.org/html/2605.14427#S3.Ex9)\)

6:Define

f​\(n,α1,α2,α3\)=2​n3​\(6\.8​e−05​α2\+3\.8​e−02​α3\)\+n2​\(α1\+2\.47​e−01​α2−3\.12​e\+02​α3\)−e​\(1/n\)​\(1\.15​e\+03​α2\+1\.12​e\+08​α3\)f\(n,\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}\)=2n^\{3\}\(6\.8e\-05\\alpha\_\{2\}\+3\.8e\-02\\alpha\_\{3\}\)\+n^\{2\}\(\\alpha\_\{1\}\+2\.47e\-01\\alpha\_\{2\}\-3\.12e\+02\\alpha\_\{3\}\)\-e\(1/n\)\(1\.15e\+03\\alpha\_\{2\}\+1\.12e\+08\\alpha\_\{3\}\)
7:

8:Step 3: Define Constraint Equation \([III\-B](https://arxiv.org/html/2605.14427#S3.Ex10)\)

9:Define

g​\(n,α2,α3\)=α2​\(2×6\.8​e−05\+3×1\.15​e\+03×\(e​\(1/n\)/n4\)\)\+α3​\(2×3\.8​e−02\+3×1\.12​e\+08×\(e​\(1/n\)/n4\)\)g\(n,\\alpha\_\{2\},\\alpha\_\{3\}\)=\\alpha\_\{2\}\(2\\times 6\.8e\-05\+3\\times 1\.15e\+03\\times\(e\(1/n\)/n^\{4\}\)\)\+\\alpha\_\{3\}\(2\\times 3\.8e\-02\+3\\times 1\.12e\+08\\times\(e\(1/n\)/n^\{4\}\)\)
10:

11:Step 4: Set Initial Guess fornn

12:Set

n​\_​i​n​i​t​i​a​l=1\.0n\\\_initial=1\.0
13:

14:Step 5: Solve the Equation

15:ifusingfsolvethen

16:Solve

f​\(n,α1,α2,α3\)=0f\(n,\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}\)=0with constraint

g​\(n,α2,α3\)\>0g\(n,\\alpha\_\{2\},\\alpha\_\{3\}\)\>0
17:else

18:Minimize

\|f​\(n,α1,α2,α3\)\|\|f\(n,\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}\)\|with bounds on

n<500n<500
19:endif

20:

21:Step 6: Print the Results

22:Print values of

α1\\alpha\_\{1\},

α2\\alpha\_\{2\},

α3\\alpha\_\{3\}, and

nn

TABLE III:WERs \(in %;↓\\downarrowbetter\) on LibriSpeech\-100\. “avg”: average over “clean” and “other”\.

## IVConclusions

End\-to\-end deep learning architectures have become the dominant paradigm across several application domains, including automatic speech recognition \(ASR\)\. These models rely on a set of hyper\-parameters whose careful selection is critical for achieving optimal performance\. In practice, however, hyper\-parameters are often chosen heuristically or through extensive empirical searches over large configuration spaces\. This reliance on heuristics remains a common limitation of most end\-to\-end systems, including ASR\.

Unlike hybrid ASR systems, where the vocabulary is fixed and derived from the phonetic structure of the target language, end\-to\-end ASR systems introduce vocabulary size as an explicit and influential hyper\-parameter\. Popular frameworks such as ESPNet provide predefined vocabulary sizes as part of their training recipes\. However, these choices are typically not accompanied by an explanation of the underlying rationale or a principled method for their derivation\. To the best of our knowledge, systematic studies examining the impact of vocabulary size on end\-to\-end ASR performance have been limited until recent work demonstrated its measurable influence\[[4](https://arxiv.org/html/2605.14427#bib.bib39)\]\.

In this paper, we introduce a mathematical formalism that brings interpretability and analytical rigor to the process of selecting the vocabulary size hyper\-parameter, moving beyond heuristic choices prevalent in existing literature\. Leveraging the first\- and second\-derivative tests from classical calculus, we model corpus\-dependent statistics using \(i\) a second\-order polynomial and \(ii\) a second\-order polynomial augmented with an exponential term\. These functional forms ensure the existence of well\-defined first\- and second\-order derivatives, enabling analytical characterization of optimality conditions\.

The resulting equations are solved numerically to estimate the optimal vocabulary size for a given training corpus\. Using the LibriSpeech\-100 dataset, we observe that incorporating an exponential term significantly improves the modeling of corpus statistics compared to a pure second\-order polynomial\. This improved fit leads to a more accurate estimate of the optimal vocabulary size, which is further validated through end\-to\-end ASR experiments\. Our results demonstrate that the proposed framework can effectively identify a principled vocabulary size that yields improved recognition performance relative to commonly adopted heuristic choices\.

Overall, this work provides a formal and explainable methodology for estimating the vocabulary size hyper\-parameter in end\-to\-end ASR systems, addressing an important gap in current practice\.

As a limitation, the applicability of the proposed framework depends on the ability to accurately fit twice\-differentiable functions to the corpus\-derived statisticsΔ​\(n\)\\Delta\(n\)andΘ​\(n\)\\Theta\(n\)\. The quality of the estimated vocabulary size is therefore contingent on the fidelity of this curve\-fitting process, which represents a known constraint of the proposed approach\.

## Acknowledgments

Would like to acknowledge the experiments conducted by Dr Ashish Panda which have been reported in Table[II](https://arxiv.org/html/2605.14427#S3.T2)and Table[III](https://arxiv.org/html/2605.14427#S3.T3)\.

## Statement on GenAI Usage

The formulation of the problem, the mathematical modeling, and the analytical derivation of the optimal vocabulary size are entirely the work of the author\. Generative AI \(Copilot\) was used solely as an assistive tool to improve the clarity, organization, and readability of the manuscript\. In particular, Copilot provided suggestions related to phrasing, flow, and structural organization—for example, recommending that material initially included in the main text be moved to the Appendix to improve the overall presentation\. No scientific content, analysis, results, or conclusions were generated by the AI\.

## References

- \[1\]A\. Gulati, J\. Qin, C\. Chiu, N\. Parmar, Y\. Zhang, J\. Yu, W\. Han, S\. Wang, Z\. Zhang, Y\. Wu, and R\. Pang\(2020\)Conformer: convolution\-augmented transformer for speech recognition\.InInterspeech 2020,pp\. 5036–5040\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2020-3015),ISSN 2958\-1796Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p2.1)\.
- \[2\]D\. P\. Kingma and J\. Ba\(2015\)Adam: A method for stochastic optimization\.In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\-9, 2015, Conference Track Proceedings,Y\. Bengio and Y\. LeCun \(Eds\.\),Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p3.3)\.
- \[3\]T\. Ko, V\. Peddinti, D\. Povey, and S\. Khudanpur\(2015\)Audio augmentation for speech recognition\.InProc\. Interspeech 2015,pp\. 3586–3589\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2015-711)Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p4.1)\.
- \[4\]S\. K\. Kopparapu and A\. Panda\(2024\-December 1–5\)A cost minimization approach to fix the vocabulary size in a tokenizer for an end\-to\-end ASR system\.InProceedings of the 2024 International Conference on Pattern Recognition \(ICPR\),Kolkata, India\.Cited by:[§I](https://arxiv.org/html/2605.14427#S1.p3.1),[§II](https://arxiv.org/html/2605.14427#S2.p1.4),[§III\-B](https://arxiv.org/html/2605.14427#S3.SS2.p6.2),[§III\-B](https://arxiv.org/html/2605.14427#S3.SS2.p7.5),[TABLE III](https://arxiv.org/html/2605.14427#S3.T3.3.1.5.3.1),[§IV](https://arxiv.org/html/2605.14427#S4.p2.1)\.
- \[5\]D\. C\. Montgomery, E\. A\. Peck, and G\. G\. Vining\(2021\)Introduction to linear regression analysis\.5th edition,John Wiley & Sons,Hoboken, NJ\.External Links:ISBN 978\-1119578727,[Link](https://www.wiley.com/en-us/Introduction+to+Linear+Regression+Analysis,+5th+Edition-p-9781119578727)Cited by:[§III\-A](https://arxiv.org/html/2605.14427#S3.SS1.p2.8)\.
- \[6\]V\. Panayotov, G\. Chen, D\. Povey, and S\. Khudanpur\(2015\)LibriSpeech ASR corpus: test\-clean\-100\.Note:[https://www\.openslr\.org/resources/12/test\-clean\.tar\.gz](https://www.openslr.org/resources/12/test-clean.tar.gz)Accessed: 2024\-06\-26Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p1.3)\.
- \[7\]V\. Panayotov, G\. Chen, D\. Povey, and S\. Khudanpur\(2015\)LibriSpeech ASR corpus: test\-other\-100\.Note:[https://www\.openslr\.org/resources/12/test\-other\.tar\.gz](https://www.openslr.org/resources/12/test-other.tar.gz)Accessed: 2024\-06\-26Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p1.3)\.
- \[8\]V\. Panayotov, G\. Chen, D\. Povey, and S\. Khudanpur\(2015\)LibriSpeech ASR corpus: train\-clean\-100\.Note:[https://www\.openslr\.org/resources/12/train\-clean\-100\.tar\.gz](https://www.openslr.org/resources/12/train-clean-100.tar.gz)Accessed: 2024\-06\-26Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p1.3),[§III](https://arxiv.org/html/2605.14427#S3.p1.9)\.
- \[9\]D\. S\. Park, W\. Chan, Y\. Zhang, C\. Chiu, B\. Zoph, E\. D\. Cubuk, and Q\. V\. Le\(2019\)SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition\.InProc\. Interspeech 2019,pp\. 2613–2617\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2019-2680)Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p4.1)\.
- \[10\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.Cited by:[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p3.3)\.
- \[11\]S\. Watanabe, T\. Hori, S\. Karita, T\. Hayashi, J\. Nishitoba, Y\. Unno, N\. Enrique Yalta Soplin, J\. Heymann, M\. Wiesner, N\. Chen, A\. Renduchintala, and T\. Ochiai\(2018\)ESPnet: End\-to\-End Speech Processing Toolkit\.InProc\. Interspeech 2018,pp\. 2207–2211\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2018-1456)Cited by:[§III\-A1](https://arxiv.org/html/2605.14427#S3.SS1.SSS1.p1.3),[§III\-A2](https://arxiv.org/html/2605.14427#S3.SS1.SSS2.p2.1)\.

## Appendix AExponential and Second order Polynomial Fit

As mentioned,Δe​x​p​\(n\)\\Delta\_\{exp\}\(n\)\([13](https://arxiv.org/html/2605.14427#S2.E13)\) andΘe​x​p​\(n\)\\Theta\_\{exp\}\(n\)\([14](https://arxiv.org/html/2605.14427#S2.E14)\) represent the second order polynomial with an exponential term\.

We can compute the first and second derivative as

Δe​x​p′​\(n\)\\displaystyle\\Delta\_\{exp\}^\{\\prime\}\(n\)=\\displaystyle=2​g3​n\+g2−g1​e\(1n\)n2,\\displaystyle 2g\_\{3\}n\+g\_\{2\}\-g\_\{1\}\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{2\}\},Θe​x​p′​\(n\)\\displaystyle\\Theta\_\{exp\}^\{\\prime\}\(n\)=\\displaystyle=2​h3​n\+h2−h1​e\(1n\)n2\\displaystyle 2h\_\{3\}n\+h\_\{2\}\-h\_\{1\}\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{2\}\}Δe​x​p′′​\(n\)\\displaystyle\\Delta\_\{exp\}^\{\\prime\\prime\}\(n\)=\\displaystyle=2​g3\+3​g1​e\(1n\)n4,\\displaystyle 2g\_\{3\}\+3g\_\{1\}\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{4\}\},Θe​x​p′′​\(n\)\\displaystyle\\Theta\_\{exp\}^\{\\prime\\prime\}\(n\)=\\displaystyle=2​h3\+3​h1​e\(1n\)n4\.\\displaystyle 2h\_\{3\}\+3h\_\{1\}\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{4\}\}\.Substituting in \([4](https://arxiv.org/html/2605.14427#S2.E4)\) we get

α1\+α2​\(2​g3​n\+g2−g1​e\(1n\)n2\)\\displaystyle\\alpha\_\{1\}\+\\alpha\_\{2\}\\left\(2g\_\{3\}n\+g\_\{2\}\-g\_\{1\}\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{2\}\}\\right\)\+α3​\(2​h3​n\+h2−h1​e\(1n\)n2\)=0\\displaystyle\+\\alpha\_\{3\}\\left\(2h\_\{3\}n\+h\_\{2\}\-h\_\{1\}\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{2\}\}\\right\)=0⟹α1\+α2​g2\+α3​h2\+2​n​\(α2​g3\+α3​h3\)\\displaystyle\\implies\\;\\alpha\_\{1\}\+\\alpha\_\{2\}g\_\{2\}\+\\alpha\_\{3\}h\_\{2\}\+2n\\left\(\\alpha\_\{2\}g\_\{3\}\+\\alpha\_\{3\}h\_\{3\}\\right\)−e\(1n\)n2​\(α2​g1\+α3​h1\)=0\\displaystyle\-\\frac\{e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\}\{n^\{2\}\}\\left\(\\alpha\_\{2\}g\_\{1\}\+\\alpha\_\{3\}h\_\{1\}\\right\)=0⟹2​n3​\(α2​g3\+α3​h3\)\+n2​\(α1\+α2​g2\+α3​h2\)\\displaystyle\\implies\\;2n^\{3\}\\left\(\\alpha\_\{2\}g\_\{3\}\+\\alpha\_\{3\}h\_\{3\}\\right\)\+n^\{2\}\\left\(\\alpha\_\{1\}\+\\alpha\_\{2\}g\_\{2\}\+\\alpha\_\{3\}h\_\{2\}\\right\)−e\(1n\)​\(α2​g1\+α3​h1\)=0\\displaystyle\-e^\{\\left\(\\frac\{1\}\{n\}\\right\)\}\\left\(\\alpha\_\{2\}g\_\{1\}\+\\alpha\_\{3\}h\_\{1\}\\right\)=0\(30\)a cubic equation\. To find the roots of \([30](https://arxiv.org/html/2605.14427#A1.E30)\), we can use numerical method like Newton\-Raphson or other root\-finding algorithms, under the constraint \([15](https://arxiv.org/html/2605.14427#S2.E15)\)\.

## Appendix BReverse Solving forα\\alphaatn=300n=300

As in Algorithm[3](https://arxiv.org/html/2605.14427#alg3)we use the Sequential Least Squares Quadratic Programming \(SLSQP\) method to find the value ofα1,2,3\\alpha\_\{1,2,3\}which satisfiesn=300n=300, a valueheuristicallychosen in ESPNet recipe\. Specifically, we use nonlinear solvers from thescipy\.optimizelibrary in Python with different initial guesses ofα1,2,3\\alpha\_\{1,2,3\}\(column 1, Table[IV](https://arxiv.org/html/2605.14427#A2.T4)\)\. Table[IV](https://arxiv.org/html/2605.14427#A2.T4)shows a sample set ofα1,2,3\\alpha\_\{1,2,3\}’s which results inn≈300n\\approx 300when \([17](https://arxiv.org/html/2605.14427#S3.E17)\) is solved\. As can be seen, different initial values ofα1,2,3\\alpha\_\{1,2,3\}’s \(Line[7](https://arxiv.org/html/2605.14427#alg3.l7), Algorithm[3](https://arxiv.org/html/2605.14427#alg3)\) result in values ofα1,2,3\\alpha\_\{1,2,3\}’s \(Line[12](https://arxiv.org/html/2605.14427#alg3.l12), Algorithm[3](https://arxiv.org/html/2605.14427#alg3)\) which satisfyn≈300n\\approx 300\(Line[3](https://arxiv.org/html/2605.14427#alg3.l3), Algorithm[3](https://arxiv.org/html/2605.14427#alg3)\)\. Table[IV](https://arxiv.org/html/2605.14427#A2.T4)shows a list ofα1,2,3\\alpha\_\{1,2,3\}’s which result inn=300n=300\. While different values ofα1,2,3\\alpha\_\{1,2,3\}’s \(example\(23\.01,−31\.56,0\.02\)\(23\.01,\-31\.56,0\.02\)and\(263571\.00,−361185\.85,246\.75\)\(263571\.00,\-361185\.85,246\.75\)\) are valid it can be seen that the ratios\(α1α3\)\\left\(\\frac\{\\alpha\_\{1\}\}\{\\alpha\_\{3\}\}\\right\),\(α2α3\)\\left\(\\frac\{\\alpha\_\{2\}\}\{\\alpha\_\{3\}\}\\right\), and\(α3α3\)\\left\(\\frac\{\\alpha\_\{3\}\}\{\\alpha\_\{3\}\}\\right\)are quite close to each other \(see Table[IV](https://arxiv.org/html/2605.14427#A2.T4)\)\.

TABLE IV:Values ofα1,2,3\\alpha\_\{1,2,3\}which result inn=300n=300\([4](https://arxiv.org/html/2605.14427#S2.E4)\)\. The left column is the initial values to Algorithm[3](https://arxiv.org/html/2605.14427#alg3)\.Algorithm 3Solving forα1,2,3\\alpha\_\{1,2,3\}forn=300n=300\(value heuristically chosen in ESPNet recipe for LibriSpeech\-100 corpus\)1:Define the objective function, constraints, initialization:

2:

𝚘𝚋𝚓​\(α1,α2,α3\)\{\\tt obj\}\(\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}\)
3:

n←716\.87×α3−α1−0\.24×α21\.38×10−4×α2\+0\.202×α3n\\leftarrow\\frac\{716\.87\\times\\alpha\_\{3\}\-\\alpha\_\{1\}\-0\.24\\times\\alpha\_\{2\}\}\{1\.38\\times 10^\{\-4\}\\times\\alpha\_\{2\}\+0\.202\\times\\alpha\_\{3\}\}\{Equation \([17](https://arxiv.org/html/2605.14427#S3.E17)\)\}

4:return

\(n−300\)2\(n\-300\)^\{2\}\{Returnsnnclose to300300\.\}

5:

𝚌𝚘𝚗𝚜←2​\(6\.89∗10−5​α2\+0\.101​α3\)\>0,716\.87​α3−α1−0\.24​α2<0\{\\tt cons\}\\leftarrow 2\(6\.89\*10^\{\-5\}\\alpha\_\{2\}\+0\.101\\alpha\_\{3\}\)\>0,716\.87\\alpha\_\{3\}\-\\alpha\_\{1\}\-0\.24\\alpha\_\{2\}<0
6:\{Equations \([18](https://arxiv.org/html/2605.14427#S3.E18)\), \([19](https://arxiv.org/html/2605.14427#S3.E19)\)\}

7:

𝚒𝚗𝚒𝚝←𝚗𝚙\.𝚛𝚊𝚗𝚍𝚘𝚖\.𝚛𝚊𝚗𝚍​\(𝟹\)\{\\tt init\}\\leftarrow\{\\tt np\.random\.rand\(3\)\}\{Initializingα\\alpha’s\}

8:

9:Optimize:

10:

result←𝚖𝚒𝚗𝚒𝚖𝚒𝚣𝚎\(𝚘𝚋𝚓,𝚒𝚗𝚒𝚝,constraints=𝚌𝚘𝚗𝚜,method=𝚂𝙻𝚂𝚀𝙿′′\)result\\leftarrow\{\\tt minimize\}\(\{\\tt obj\},\{\\tt init\},constraints=\{\\tt cons\},method=\{\\tt\{\}^\{\\prime\}SLSQP^\{\\prime\}\}\)
11:if

r​e​s​u​l​t\.s​u​c​c​e​s​sresult\.successthen

12:

α1,α2,α3←r​e​s​u​l​t\.x\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}\\leftarrow result\.x
13:else

14:choose another

𝚒𝚗𝚒𝚝\{\\tt init\}\{not all initial values ofα\\alpha’s result in a solution\}

15:Go to Line \#[10](https://arxiv.org/html/2605.14427#alg3.l10)

16:endif