Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment

# Best-of-𝑁 TTS Evaluation is Confounded by ASR Family Alignment
Source: [https://arxiv.org/html/2607.08256](https://arxiv.org/html/2607.08256)
###### Abstract

Best\-of\-NN\(BoN\) inference improves content consistency in zero\-shot text\-to\-speech by selecting fromNNcandidates with an automatic speech recognition \(ASR\) verifier\. We identify an underexplored evaluation confound: a verifier’s apparent quality depends strongly on which ASR family judges it\. On LibriSpeech\-PC test\-clean\(Meisteret al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib15)\)with F5\-TTS\(Chenet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib1)\), verifier rankings reverse across Whisper, wav2vec 2\.0, and HuBERT evaluators, and same\-family verifier–evaluator pairs recover 2–3×\\timesmore oracle headroom than cross\-family pairs despite near\-identical representations \(linear CKA0\.9780\.978\)—a pattern consistent with identity\- or lineage\-level coupling rather than representational overlap\. We propose twocross\-family rank ensembles\(rank\-averaging and conjunctive max\-rank\) that attain the lowest mean WER across three independent evaluators—1\.61%1\.61\\%atN=10N\{=\}10\(−12%\-12\\%relative to F5\-TTS\)—with no measurable degradation under automatic SIM\-o/UTMOS metrics; the best single verifier drives WER from2\.06%2\.06\\%to1\.72%1\.72\\%\(−16\.5%\-16\.5\\%\) under the official F5\-TTS evaluator\. We recommend cross\-evaluator triangulation as default reporting practice\.

text\-to\-speech, best\-of\-N, verifier, evaluation, flow matching

## 1Introduction

Recent flow\-matching zero\-shot TTS systems—F5\-TTS\(Chenet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib1)\), E2 TTS\(Eskimezet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib2)\), CosyVoice 2\(Duet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib3)\), MaskGCT\(Wanget al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib4)\), Seed\-TTS\(Anastassiouet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib19)\), NaturalSpeech 3\(Juet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib20)\)—produce speech that is indistinguishable from human recordings in naturalness and speaker similarity, yet still produce word\-level content errors on a non\-trivial fraction of utterances\. Best\-of\-NN\(BoN\) inference is the standard inference\-time remedy: synthesizeNNcandidates with different random initializations and select the one whose ASR\-decoded transcript matches the target text most closely\.

BoN is reported to reduce WER by1010–30%30\\%relative across recent flow\-matching systems, but the literature is inconsistent on which*verifier*ASR to use—reported choices include wav2vec 2\.0\(Baevskiet al\.,[2020](https://arxiv.org/html/2607.08256#bib.bib11)\), Whisper\(Radfordet al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib16)\), and its distillations\(Gandhiet al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib10)\)—and the evaluation ASR is usually a single fixed model\. The choice of evaluator is a design decision that has so far received no systematic attention\.

In this paper we run a four\-way evaluator ablation spanning the Whisper\(Radfordet al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib16)\), wav2vec 2\.0\(Baevskiet al\.,[2020](https://arxiv.org/html/2607.08256#bib.bib11)\), and HuBERT\(Hsuet al\.,[2021](https://arxiv.org/html/2607.08256#bib.bib12)\)families \(§[3](https://arxiv.org/html/2607.08256#S3)\) over a shared set of BoN candidates\. Our contributions are the following\.

- •We document that BoN verifier rankings are systematically evaluator\-dependent: the same generated outputs are ranked in opposing directions by evaluators from different ASR families, so the verifier preferred under one family can lose under another \(§[4](https://arxiv.org/html/2607.08256#S4)\)\.
- •We quantify the magnitude of this confound on LibriSpeech\-PC test\-clean and show that the \(verifier, evaluator\) family pairing is a much larger lever on reported WER than the choice between common verifier checkpoints, while a large oracle headroom remains unexploited \(§[4](https://arxiv.org/html/2607.08256#S4)\)\.
- •We rule out audio\-encoder representation similarity \(linear CKA\) as the dominant explanation; the pattern is instead consistent with identity\- or lineage\-level coupling between verifier and evaluator, a speech analog of LLM\-as\-a\-judge self\-bias \(§[5](https://arxiv.org/html/2607.08256#S5)\)\.
- •We propose*cross\-family rank ensembles*that select candidates by aggregating verifiers across ASR families, and recommend*cross\-evaluator triangulation*—reporting WER under at least two ASR families with disjoint training lineages—as a default reporting practice \(§[4\.4](https://arxiv.org/html/2607.08256#S4.SS4)\)\.

## 2Related Work

Inference\-time remedies for flow\-matching TTS—classifier\-free guidance \(CFG\)\(Ho and Salimans,[2021](https://arxiv.org/html/2607.08256#bib.bib18)\), BoN reranking\(Duet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib3); Yanget al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib5)\), reverse inference\(Huet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib6)\)—report no cross\-evaluator validation; recent strong systems\(Anastassiouet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib19); Juet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib20)\)similarly tune verifiers against a single fixed ASR\. Speech preference optimization\(Zhanget al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib21)\)shares the same single\-judge risk\. BoN scaling theory in LMs\(Yuet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib7); Wanget al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib8)\)motivates verifier ensembling but has no TTS analog\. The structural precedent is LLM\-as\-judge self\-bias\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib9)\); we document the speech analog\. Evaluator\-driven shifts in spoken\-language conclusions are established for ASR fairness\(Koeneckeet al\.,[2020](https://arxiv.org/html/2607.08256#bib.bib22); Fenget al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib23)\)and for MOS\-predictor cross\-domain generalization\(Cooperet al\.,[2022](https://arxiv.org/html/2607.08256#bib.bib24); Mittaget al\.,[2021](https://arxiv.org/html/2607.08256#bib.bib25); Babaet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib26)\)\.

## 3Setup

Synthesis\.We use the publicly released F5\-TTS base model\(Chenet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib1)\)with its default inference recipe:3232ODE solver steps, CFG\(Ho and Salimans,[2021](https://arxiv.org/html/2607.08256#bib.bib18)\)scale2\.02\.0, and sway sampling\. We generate1010BoN candidates per utterance; for eachN∈\{3,5,10\}N\\in\\\{3,5,10\\\}we take the leadingNNcandidates and select the one with the lowest joint WER\+CER score against the reference transcript\.

Verifiers\.We compare three single\-model verifiers and one three\-way ensemble\. The single verifiers are a wav2vec 2\.0 base model \(95M parameters;*w2v2\-base*\), a small Distil\-Whisper \(166M;*distil\-sm*\), and a large Distil\-Whisper \(756M;*distil\-v3*\)\.*ens3*denotes the three\-way rank\-average of these three verifiers\. In the scaling study \(§[4\.4](https://arxiv.org/html/2607.08256#S4.SS4)\) we additionally consider two cross\-family rank ensembles that aggregate only the cross\-family pair \{w2v2\-base, distil\-v3\}\.

Evaluators\.For the main results \(§[4](https://arxiv.org/html/2607.08256#S4)\) we score every candidate with four ASR evaluators spanning three families: a Whisper\-large\-v3 evaluator served via the faster\-whisper runtime \(*fwhisper\-lgv3*, the official F5\-TTS evaluator\(Chenet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib1)\)\); a large wav2vec 2\.0 model \(*w2v2\-lv60*\); the same Distil\-Whisper used as a verifier \(*distil\-sm*\); and a large HuBERT model \(*hubert\-lg*\)\. For the CKA–WER analysis \(§[5](https://arxiv.org/html/2607.08256#S5)\) we additionally include Whisper\-medium \(*whisper\-med*\) as a fourth evaluator to expand the pool of evaluator pairs\.

Data & normalization\.We evaluate on the LibriSpeech\-PC test\-clean cross\-sentence subset\(Meisteret al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib15)\)\(11271127samples,44–1010s reference\) and run the F5\-TTS official evaluation pipeline verbatim, applying its default text normalization uniformly to all four ASR outputs\. Whisper’s English\-specific normalizer is not applied separately, to keep the cross\-ASR WER comparison on the same footing\.

Quality\.Speaker similarity SIM\-o is the cosine of WavLM\(Chenet al\.,[2022](https://arxiv.org/html/2607.08256#bib.bib17)\)speaker embeddings; reference and synthesized audio are resampled symmetrically from2424kHz to1616kHz before scoring\. Naturalness is the UTMOS\(Saekiet al\.,[2022](https://arxiv.org/html/2607.08256#bib.bib14)\)score from the standard2222\-strong checkpoint\. Real\-time factors are measured isolated on a single RTX 4090 GPU\.

Reproducibility\.All models are used at their default public revisions; code, decoding settings, and evaluation scripts are available at[https://github\.com/yu1012/BoN\-TTS](https://github.com/yu1012/BoN-TTS)\.

## 4Main Results

### 4\.1Results under the F5\-TTS Evaluator

Table 1:LibriSpeech\-PC test\-clean under the official fwhisper\-lgv3 evaluator\. “rel” is WER reduction vs\. F5\-TTS; “pp” is a two\-sided paired permutation test against F5\-TTS \(10,000 permutations\)\.MethodWERCERRTFrelppF5\-TTS \(baseline\)2\.060\.710\.190——BoN w2v2\-base1\.990\.620\.383−3\.5%\-3\.5\\%0\.31BoN distil\-sm2\.040\.590\.388−1\.0%\-1\.0\\%0\.93BoN distil\-v31\.880\.530\.379−8\.7%\\mathbf\{\-8\.7\\%\}0\.030BoN ens31\.910\.540\.429−7\.2%\-7\.2\\%0\.057Our single\-shot F5\-TTS baseline reaches2\.06%2\.06\\%WER, within0\.360\.36pp of the2\.42%2\.42\\%multi\-seed mean reported in the F5\-TTS paper\(Chenet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib1)\)—a single\-shot vs\. multi\-seed\-mean gap, not a pipeline discrepancy\. BoN with distil\-v3 drives WER to1\.88%1\.88\\%\(−8\.7%\-8\.7\\%,p=0\.030p\{=\}0\.030\) at RTF0\.3790\.379, the only significant single\-verifier reduction; ens3 is a close runner\-up \(−7\.2%\-7\.2\\%,p=0\.057p\{=\}0\.057\)\. The w2v2\-base verifier improves the point estimate \(−3\.5%\-3\.5\\%\) but is not significant \(p=0\.31p\{=\}0\.31\), and distil\-sm is flat \(p=0\.93p\{=\}0\.93\)\. Speaker similarity \(SIM\-o\) stays within±0\.0006\\pm 0\.0006and UTMOS within±0\.005\\pm 0\.005of F5\-TTS across all configurations \(§[5\.1](https://arxiv.org/html/2607.08256#S5.SS1)\), so the WER gain comes at no measurable quality cost\.

### 4\.2Cross\-Evaluator Triangulation

Table 2:Cross\-evaluator WER \(%\) on LibriSpeech\-PC test\-clean\. The same generated audio is evaluated by ASR checkpoints from different families \(aliases defined in §[3](https://arxiv.org/html/2607.08256#S3)\); the preferred verifier \(bold\) changes with the evaluator\.Methodfwhisper\-lgv3w2v2\-lv60hubert\-lgF5\-TTS \(baseline\)2\.061\.521\.92BoN w2v2\-base1\.991\.411\.74BoN distil\-sm2\.041\.501\.77BoN distil\-v31\.881\.451\.74BoN ens31\.911\.451\.76Same\-family preference\.Under fwhisper\-lgv3 \(Table[2](https://arxiv.org/html/2607.08256#S4.T2)\), the Distil\-Whisper verifier \(distil\-v3\) wins; under w2v2\-lv60, the wav2vec 2\.0 verifier \(w2v2\-base\) wins; under hubert\-lg, w2v2\-base again ranks at the top alongside distil\-v3\.

Magnitude of the swing\.The distil\-v3 BoN reduction is0\.180\.18pp under fwhisper\-lgv3 but only0\.070\.07pp under w2v2\-lv60, reversing the verifier ranking between Whisper and wav2vec 2\.0 evaluators\. A reader given only the F5\-TTS official evaluator would prefer distil\-v3 BoN; one given only w2v2\-lv60 would prefer w2v2\-base BoN\. Both are correct conditional on their evaluator, and incompatible\.

### 4\.3Oracle Headroom and Recovery

Table 3:Oracle headroom on LibriSpeech\-PC test\-clean withN=3N\{=\}3candidates per sample\.*Single\-shot*==first candidate;*Oracle*==per\-sample best of33\. The bottom block reports recovery\(Single−BoN\)/\(Single−Oracle\)\(\\mathrm\{Single\}\{\-\}\\mathrm\{BoN\}\)/\(\\mathrm\{Single\}\{\-\}\\mathrm\{Oracle\}\)in %; same\-family BoN recovers22–3×3\{\\times\}more than cross\-family BoN on the same audio\. The0\.020\.02pp gap between Single\-shot=2\.04=2\.04here and F5\-TTS=2\.06=2\.06in Tables[1](https://arxiv.org/html/2607.08256#S4.T1)/[4](https://arxiv.org/html/2607.08256#S4.T4)is Whisper batching non\-determinism \(decoded in anN=3N\{=\}3batch vs\. a single utterance\); both rows describe identical synthesized audio\.fwhisper\-lgv3w2v2\-lv60hubert\-lgSingle\-shot2\.041\.521\.94Oracle \(N=3N\{=\}3\)1\.421\.091\.18Headroom \(pp\)0\.630\.430\.76*Recovery \(%\) of oracle headroom:*BoN w2v2\-base7\.926\.127\.1BoN distil\-sm0\.06\.822\.6BoN distil\-v326\.018\.227\.1BoN ens320\.517\.024\.5To bound how far BoN can go atN=3N\{=\}3, we synthesize all three candidates per sample and transcribe each with every evaluator \(Table[3](https://arxiv.org/html/2607.08256#S4.T3), top block\)\. On the official evaluator, an oracle verifier would drive WER to1\.42%1\.42\\%, below F5\-TTS’s reported∼\\sim1\.51\.5–1\.7%1\.7\\%\(Chenet al\.,[2025](https://arxiv.org/html/2607.08256#bib.bib1)\): headroom for verifier improvement is substantial\. The bottom block makes the22–3×3\{\\times\}family swing precise: under fwhisper\-lgv3, distil\-v3 recovers26\.0%of headroom vs\. only7\.9%7\.9\\%for cross\-family w2v2\-base \(3\.3×3\.3\\times\); under w2v2\-lv60, w2v2\-base recovers26\.1%vs\.18\.2%18\.2\\%for cross\-family distil\-v3 \(1\.4×1\.4\\times\)\. Recovery is a property of the \(verifier, evaluator\) pair, not the verifier alone\.

### 4\.4Scaling and Bias\-Corrected Ensembles

To test whether the confound persists at higherNNand whether cross\-family aggregation mitigates it, we synthesizeN=10N\{=\}10candidates per sample and add six BoN configurations: single wav2vec 2\.0 and Distil\-Whisper verifiers atN∈\{5,10\}N\\in\\\{5,10\\\}, plus two cross\-family rank ensembles over the \(w2v2\-base, distil\-v3\) pair\.

- •rank\-avg\.Each verifier independently ranks theNNcandidates by WER\+CER score\. We then pick the candidate with the lowest average rank across the two families\.
- •max\-rank\.We pick the candidate with the lowest worst\-case rank, requiring it to rank well in both families\. This conjunctive form explicitly penalizes single\-family inflation\.

335510101\.61\.61\.71\.71\.81\.8NN\(BoN candidates\)Mean WER \(%\)w2v2\-basedistil\-v3rank\-avgmax\-rankF5\-TTSFigure 1:Mean WER \(averaged over fwhisper\-lgv3, w2v2\-lv60, hubert\-lg\) vs\.NN, from Table[4](https://arxiv.org/html/2607.08256#S4.T4)\. Both cross\-family rank ensembles \(rank\-avg, max\-rank\) atN=5N\{=\}5reduce WER under all three evaluators simultaneously, while the single w2v2\-base verifier regresses on fwhisper\-lgv3 atN=5N\{=\}5\. Cross\-family ensembles scale monotonically withNN\.Table 4:ScalingNNon LibriSpeech\-PC test\-clean\. Per\-evaluator WER \(%\) under three evaluators \(aliases as defined in §[3](https://arxiv.org/html/2607.08256#S3)\)\. Thepp\-value column reports a two\-sided paired permutation test against F5\-TTS under*fwhisper\-lgv3*\(10,000 permutations\)\. “↑\\uparrow” marks a regression vs\. the same row at the precedingNN\.Method \(NN\)fwhisper\-lgv3w2v2\-lv60hubert\-lgmeanppF5\-TTS \(baseline\)2\.061\.521\.921\.83—w2v2\-base \(33\)1\.991\.411\.741\.710\.05w2v2\-base \(55\)2\.20↑\\uparrow1\.411\.751\.790\.97w2v2\-base \(1010\)1\.981\.401\.721\.700\.08distil\-v3 \(33\)1\.881\.451\.741\.69<\.001<\\\!\.001distil\-v3 \(55\)1\.801\.481\.751\.68<\.001<\\\!\.001distil\-v3 \(1010\)1\.721\.441\.681\.61<\.001<\\\!\.001rank\-avg \(33\)2\.011\.431\.741\.730\.17rank\-avg \(55\)1\.901\.401\.661\.65\.001rank\-avg \(1010\)1\.811\.411\.601\.61<\.001<\\\!\.001max\-rank \(33\)1\.991\.421\.731\.710\.12max\-rank \(55\)1\.801\.431\.671\.63<\.001<\\\!\.001max\-rank \(1010\)1\.801\.401\.621\.61<\.001<\\\!\.001

Findings\(Figure[1](https://arxiv.org/html/2607.08256#S4.F1)\)\. \(i\) The single cross\-family verifier \(w2v2\-base\) under fwhisper\-lgv3 regresses from1\.99%1\.99\\%atN=3N\{=\}3to2\.20%2\.20\\%atN=5N\{=\}5\(p=0\.97p\{=\}0\.97\): more candidates expose a same\-family\-evaluator penalty\. \(ii\) The same\-family verifier \(distil\-v3\) scales monotonically on its own family \(1\.88%→1\.72%1\.88\\%\\rightarrow 1\.72\\%forN=3→10N\{=\}3\{\\to\}10,p<\.001p\{<\}\.001\)\. \(iii\) Cross\-family rank ensembles match the single best on mean WER and are the most cross\-evaluator\-robust\. The rank\-avg configuration atN=5N\{=\}5is the only one reachingp<0\.05p\{<\}0\.05under all three evaluators at once \(\.001/\.022/\.0002\.001/\.022/\.0002\); atN=10N\{=\}10, rank\-avg ties distil\-v3 at mean1\.61%1\.61\\%while winning the hubert\-lg column \(1\.60%1\.60\\%\)\.

Practical rule\.With a known same\-family evaluator, use distil\-v3 atN=10N\{=\}10\. Otherwise, use rank\-avg or max\-rank over the cross\-family pair atN=5N\{=\}5for the best compute–robustness frontier\.

## 5Analysis: Toward a Mechanism for the Confound

Table 5:Linear CKA \(mean\-pooled last hidden state\) between audio encoders of six ASR checkpoints on500500F5\-TTS\-synthesized waveforms \(a random subset of LibriSpeech test\-clean\)\. Aliases as in §[3](https://arxiv.org/html/2607.08256#S3): wav2vec 2\.0 family \(*w2v2\-base*,*w2v2\-lv60*\), Whisper family \(*distil\-sm*,*distil\-v3*,*whisper\-med*; the latter three are Distil\-Whisper or Whisper checkpoints\), HuBERT family \(*hubert\-lg*\)\.w2v2\-base w2v2\-lv60 distil\-sm distil\-v3 whisper\-med hubert\-lg w2v2\-base1\.000\.650\.140\.140\.150\.67w2v2\-lv600\.651\.000\.060\.050\.060\.55distil\-sm0\.140\.061\.000\.980\.980\.39distil\-v30\.140\.050\.981\.000\.980\.40whisper\-med0\.150\.060\.980\.981\.000\.40hubert\-lg0\.670\.550\.390\.400\.401\.00

To test whether the confound reduces to representation similarity, we measure linear CKA\(Kornblithet al\.,[2019](https://arxiv.org/html/2607.08256#bib.bib13)\)between audio encoders of six ASR checkpoints on500500F5\-TTS\-synthesized waveforms \(a random subset of LibriSpeech test\-clean\)\. Table[5](https://arxiv.org/html/2607.08256#S5.T5)confirms three intuitive clusters: Whisper–Distil\-Whisper \(CKA0\.970\.97–0\.980\.98internally\), wav2vec 2\.0 / HuBERT \(0\.550\.55–0\.670\.67\), and near\-zero cross\-cluster\.

The naive prediction \(high CKA implies similar WER rankings\) is not supported \(Figure[2](https://arxiv.org/html/2607.08256#S5.F2)\)\. The Whisper\-family pair \(distil\-sm↔\\leftrightarrowwhisper\-med, CKA0\.9780\.978\) is anti\-correlated \(r=−0\.52r\{=\}\{\-\}0\.52\), while the wav2vec 2\.0/HuBERT pair \(CKA0\.550\.55\) agrees nearly perfectly \(r=\+0\.94r\{=\}\{\+\}0\.94\)\. Pearson\(CKA,rr\) is−0\.36\-0\.36over the six pairs but reverses to\+0\.32\+0\.32once the same\-family outlier is removed: the trend is single\-point\-driven\.

The pattern is consistent not with representation similarity but with anidentity\- or lineage\-level coupling effect: a verifier’s selections are disproportionately favored by evaluators sharing its ASR lineage, while an independent same\-family member returns an uninflated score—structurally analogous to LLM\-as\-a\-judge self\-bias\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08256#bib.bib9)\)\. Disentangling identity\- from family\-level coupling is left to future work\.

### 5\.1Quality and cross\-evaluator robustness

SIM\-o and UTMOS atN=3N\{=\}3stay within±0\.001\\pm 0\.001and±0\.005\\pm 0\.005of F5\-TTS \(0\.94260\.9426/3\.8793\.879\), with no measurable degradation under either predictor\. Both may be near their resolution limit \(UTMOS saturates around4\.04\.0\), so human\-MOS / NISQA / UTMOSv2 triangulation\(Mittaget al\.,[2021](https://arxiv.org/html/2607.08256#bib.bib25); Babaet al\.,[2024](https://arxiv.org/html/2607.08256#bib.bib26)\)is deferred\. For cross\-evaluator robustness, rank\-avg and max\-rank atN=5N\{=\}5reduce WER under all three evaluators simultaneously, while distil\-v3 atN=10N\{=\}10stays flat on w2v2\-lv60\. The \(w2v2\-base, distil\-v3\) pair was fixed in advance as the largest publicly available verifier in each family, prior to any cross\-evaluator analysis\.

![Refer to caption](https://arxiv.org/html/2607.08256v1/x1.png)Figure 2:CKA does not predict WER agreement across the\(42\)=6\\binom\{4\}\{2\}\{=\}6evaluator pairs from\{\\\{distil\-sm, w2v2\-lv60, whisper\-med, hubert\-lg\}\\\}\. Pearsonrris computed over1313BoN configurations on a500500\-sample pilot\. Pairs:A, distil\-sm↔\\leftrightarroww2v2\-lv60;B, distil\-sm↔\\leftrightarrowhubert\-lg;C, w2v2\-lv60↔\\leftrightarrowwhisper\-med;D, w2v2\-lv60↔\\leftrightarrowhubert\-lg;E, whisper\-med↔\\leftrightarrowhubert\-lg;F, distil\-sm↔\\leftrightarrowwhisper\-med \(same\-family, red triangle\)\. The all\-6 trend \(r=−0\.36r\{=\}\{\-\}0\.36, solid\) is driven by the same\-family outlier F \(rWER=−0\.52r\_\{\\mathrm\{WER\}\}\{=\}\{\-\}0\.52at CKA0\.9780\.978\)\. Excluding F, the trend reverses tor=\+0\.32r\{=\}\{\+\}0\.32\(dashed\), confirming that CKA does not monotonically predict WER agreement\.

## 6Discussion and Conclusion

Any BoN comparison reported under a single ASR evaluator is potentially confounded; we recommend reporting WER under at least two ASR families with disjoint training lineages\. Ourcross\-family rank ensemblesrealize this principle on the selection side: they yield the most cross\-evaluator\-robust BoN configurations we test, reaching1\.61%1\.61\\%mean WER atN=10N\{=\}10\(−12%\-12\\%vs\. F5\-TTS\); the largest reduction under the official F5\-TTS evaluator alone,2\.06%→1\.72%2\.06\\%\\rightarrow 1\.72\\%\(−16\.5%\-16\.5\\%\), comes from the best same\-family verifier\. TheN=3N\{=\}3oracle WER of1\.42%1\.42\\%leaves substantial headroom for verifier\-design work \(e\.g\., adversarial reweighting against in\-family inflation\)\.

Limitations\.Our study uses one TTS backbone \(F5\-TTS\) on LibriSpeech\-PC test\-clean; generalization to other backbones \(CosyVoice 2, MaskGCT\) and a fourth ASR family \(Parakeet, Canary\), plus human\-MOS triangulation, is left to future work\.

## References

- P\. Anastassiou, J\. Chen, J\. Chen, Y\. Chen, Z\. Chen,et al\.\(2024\)Seed\-TTS: a family of high\-quality versatile speech generation models\.arXiv:2406\.02430\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p1.2),[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- K\. Baba, W\. Nakata, Y\. Saito, and H\. Saruwatari \(2024\)The T05 system for the VoiceMOS challenge 2024: transfer learning from deep image classifier to naturalness MOS prediction of high\-quality synthetic speech\.InIEEE Spoken Language Technology Workshop,pp\. 818–824\.External Links:[Document](https://dx.doi.org/10.1109/SLT61566.2024.10832315)Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.08256#S5.SS1.p1.8)\.
- A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli \(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p2.2),[§1](https://arxiv.org/html/2607.08256#S1.p3.1)\.
- S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao,et al\.\(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing\.Cited by:[§3](https://arxiv.org/html/2607.08256#S3.p5.3)\.
- Y\. Chen, Z\. Niu, Z\. Ma, K\. Deng, C\. Wang, J\. Zhao, K\. Yu, and X\. Chen \(2025\)F5\-TTS: a fairytaler that fakes fluent and faithful speech with flow matching\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6255–6271\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p1.2),[§3](https://arxiv.org/html/2607.08256#S3.p1.5),[§3](https://arxiv.org/html/2607.08256#S3.p3.1),[§4\.1](https://arxiv.org/html/2607.08256#S4.SS1.p1.14),[§4\.3](https://arxiv.org/html/2607.08256#S4.SS3.p1.11)\.
- E\. Cooper, W\. Huang, T\. Toda, and J\. Yamagishi \(2022\)Generalization ability of MOS prediction networks\.InICASSP,pp\. 8442–8446\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- Z\. Du, Y\. Wang, Q\. Chen, X\. Shi, X\. Lv, T\. Zhao, Z\. Gao, Y\. Yang, C\. Gao, H\. Wang,et al\.\(2024\)CosyVoice 2: scalable streaming speech synthesis with large language models\.arXiv:2412\.10117\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p1.2),[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- S\. E\. Eskimez, X\. Wang, M\. Thakker, C\. Li, C\. Tsai, Z\. Xiao, H\. Yang, Z\. Zhu, M\. Tang, X\. Tan, Y\. Liu, S\. Zhao, and N\. Kanda \(2024\)E2 tts: embarrassingly easy fully non\-autoregressive zero\-shot TTS\.arXiv:2406\.18009\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p1.2)\.
- S\. Feng, B\. M\. Halpern, O\. Kudina, and O\. Scharenborg \(2024\)Towards inclusive automatic speech recognition\.Computer Speech & Language84,pp\. 101567\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- S\. Gandhi, P\. von Platen, and A\. M\. Rush \(2023\)Distil\-Whisper: robust knowledge distillation via large\-scale pseudo labelling\.arXiv:2311\.00430\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p2.2)\.
- J\. Ho and T\. Salimans \(2021\)Classifier\-free diffusion guidance\.NeurIPS Workshop on Deep Generative Models and Downstream Applications \(arXiv:2207\.12598\)\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1),[§3](https://arxiv.org/html/2607.08256#S3.p1.5)\.
- W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed \(2021\)HuBERT: self\-supervised speech representation learning by masked prediction of hidden units\.IEEE/ACM TASLP\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p3.1)\.
- Y\. Hu, C\. Chen, S\. Wang, E\. S\. Chng, and C\. Zhang \(2024\)Robust zero\-shot text\-to\-speech synthesis with reverse inference optimization\.arXiv:2407\.02243\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- Z\. Ju, Y\. Wang, K\. Shen, X\. Tan, D\. Xin, D\. Yang, Y\. Liu, Y\. Leng, K\. Song, S\. Tang,et al\.\(2024\)NaturalSpeech 3: zero\-shot speech synthesis with factorized codec and diffusion models\.ICML\.Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p1.2),[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- A\. Koenecke, A\. Nam, E\. Lake, J\. Nudell, M\. Quartey, Z\. Mengesha, C\. Toups, J\. R\. Rickford, D\. Jurafsky, and S\. Goel \(2020\)Racial disparities in automated speech recognition\.Proceedings of the National Academy of Sciences\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton \(2019\)Similarity of neural network representations revisited\.InICML,Cited by:[§5](https://arxiv.org/html/2607.08256#S5.p1.5)\.
- A\. Meister, M\. Novikov, N\. Karpov, E\. Bakhturina, V\. Lavrukhin, and B\. Ginsburg \(2023\)LibriSpeech\-PC: benchmark for evaluation of punctuation and capitalization capabilities of end\-to\-end ASR models\.InIEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–7\.Cited by:[§3](https://arxiv.org/html/2607.08256#S3.p4.3)\.
- G\. Mittag, B\. Naderi, A\. Chehadi, and S\. Möller \(2021\)NISQA: a deep CNN\-self\-attention model for multidimensional speech quality prediction with crowdsourced datasets\.InInterspeech,pp\. 2127–2131\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.08256#S5.SS1.p1.8)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2023\)Robust speech recognition via large\-scale weak supervision\.InICML,Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p2.2),[§1](https://arxiv.org/html/2607.08256#S1.p3.1)\.
- T\. Saeki, D\. Xin, W\. Nakata, T\. Koriyama, S\. Takamichi, and H\. Saruwatari \(2022\)UTMOS: UTokyo\-SaruLab system for VoiceMOS challenge 2022\.InInterspeech,pp\. 4521–4525\.Cited by:[§3](https://arxiv.org/html/2607.08256#S3.p5.3)\.
- T\. Wang, J\. Chen, Q\. Jia, S\. Wang, R\. Fang, H\. Wang, Z\. Gao, C\. Xie, C\. Xu, J\. Dai, Y\. Liu, J\. Wu, S\. Ding, L\. Li, Z\. Huang, X\. Deng, T\. Yu, G\. Ma, H\. Xiao, Z\. Chen, D\. Xiang, Y\. Wang, Y\. Zhu, Y\. Xiao, J\. Wang, Y\. Wang, S\. Ding, J\. Huang, J\. Xu, Y\. Tayier, Z\. Hu, Y\. Gao, C\. Zheng, Y\. Ye, Y\. Li, L\. Wan, X\. Jiang, Y\. Wang, S\. Cheng, Z\. Song, X\. Tang, X\. Xu, N\. Zhang, H\. Chen, Y\. E\. Jiang, and W\. Zhou \(2024\)Weaver: foundation models for creative writing\.arXiv:2401\.17268\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- Y\. Wang, H\. Zhan, L\. Liu, R\. Zeng, H\. Guo, J\. Zheng, Q\. Zhang, X\. Zhang, S\. Zhang, and Z\. Wu \(2025\)MaskGCT: zero\-shot text\-to\-speech with masked generative codec transformer\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.08256#S1.p1.2)\.
- Y\. Yang, S\. Liu, J\. Li, Y\. Hu, H\. Wu, H\. Wang, J\. Yu, L\. Meng, H\. Sun, Y\. Liu, Y\. Lu, K\. Yu, and X\. Chen \(2025\)Pseudo\-autoregressive neural codec language models for efficient zero\-shot text\-to\-speech synthesis\.arXiv:2504\.10352\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- F\. Yu, Y\. Li, and B\. Wang \(2025\)Scaling flaws of verifier\-guided search in mathematical reasoning\.arXiv:2502\.00271\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- D\. Zhang, Z\. Li, S\. Li, X\. Zhang, P\. Wang, Y\. Zhou, and X\. Qiu \(2024\)SpeechAlign: aligning speech generation to human preferences\.NeurIPS \(arXiv:2404\.05600\)\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing,et al\.\(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.NeurIPS\.Cited by:[§2](https://arxiv.org/html/2607.08256#S2.p1.1),[§5](https://arxiv.org/html/2607.08256#S5.p3.1)\.