Towards Quantifying Benchmark Optimization in ASR Models
Summary
This paper quantifies how high-performing ASR models optimize for benchmarks in ways that inflate scores without improving real-world transcription, using behavioral probes to reveal benchmark-conditioned behaviors.
View Cached Full Text
Cached at: 08/21/26, 08:08 AM
Paper page - Towards Quantifying Benchmark Optimization in ASR Models
Source: https://huggingface.co/papers/2608.19936
Abstract
High-performing speech recognition models reproduce benchmark transcripts despite contradictory audio, revealing benchmark-optimized behaviors that inflate scores without improving real-world transcription.
Public benchmarks are important measures ofAutomatic Speech Recognition(ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifyingbenchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models’ capabilities of reproducing benchmark reference spans despite underdetermined audio:reference disagreement,masked-number recovery, andorthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety ofmechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated vialow-rank linear steeringor simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2608\.19936
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.19936 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.19936 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.19936 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Measuring benchmark optimization in speech recognition
This article discusses research on measuring benchmark optimization in speech recognition, where some ASR models may optimize for test benchmarks rather than real-world performance, and introduces tests to quantify this phenomenon.
What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR
This paper proposes a dual-reference benchmarking approach for atypical ASR, using both verbatim and intended transcriptions to evaluate 11 ASR models on stuttered speech, highlighting the importance of selecting the appropriate reference depending on the use case.
we benchmark models nobody actually runs
The article critiques the lack of systematic benchmarking for AI models in different quantization formats, highlighting discrepancies between benchmark results and real-world usage, and calls for more thorough evaluation.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
A systematic study defining and analyzing benchmark saturation across 60 language model benchmarks, finding nearly half exhibit saturation and that expert-curated benchmarks are more resilient, suggesting design choices for durable evaluation.
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
This paper introduces Benchmarking-Cultures-25, a dataset analyzing how AI model builders selectively highlight benchmarks in press releases. It finds a fragmented evaluation landscape with limited cross-model comparability, arguing that benchmarks are used as narrative devices for market positioning rather than standardized scientific measurement.