StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognition

# StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognition
Source: [https://arxiv.org/html/2608.13717](https://arxiv.org/html/2608.13717)
###### Abstract

Streaming automatic speech recognition \(ASR\) underperforms on domain\-shifted target audio, where labeled in\-domain data is costly to prepare while unlabeled audio is abundant\. We present StreamHear, a semi\-supervised pipeline that adapts a pretrained streaming student by fine\-tuning an offline transducer teacher on the labeled training set, generating pseudo\-labels on the unlabeled portion, and fine\-tuning the student on the mixture\. We further introduce a prior\-regularized dynamic\-programming realignment step that fixes chunk\-level word placement using an ASR\-hypothesis anchor\. Across four datasets spanning financial calls, prepared read speech, and phone\-quality dialogue, StreamHear consistently outperforms supervised student fine\-tuning and narrows the gap to the offline teacher\.

###### Index Terms:

automatic speech recognition, semi\-supervised learning, pseudo\-labeling, domain adaptation, cache\-aware streaming\.

††address:Capital One, USA
zefang\.liu@capitalone\.com## 1Introduction

Pretrained streaming automatic speech recognition \(ASR\) models, such as cache\-aware FastConformer\-RNN\-T\[[17](https://arxiv.org/html/2608.13717#bib.bib1)\], enable low\-latency transcription for real\-time applications\. Yet on domain\-shifted target audio such as financial earnings calls, international English, and phone\-quality customer\-service conversation, they still underperform, struggling with entity\-dense vocabulary, accented pronunciations, and narrowband acoustics\. In\-domain fine\-tuning is the natural remedy, but labeled in\-domain audio is costly to prepare at chunk granularity while unlabeled in\-domain audio is abundant\.

Semi\-supervised pseudo\-labeling addresses this gap and has been widely studied in ASR, from foundational iterative self\-training\[[9](https://arxiv.org/html/2608.13717#bib.bib8),[24](https://arxiv.org/html/2608.13717#bib.bib9),[14](https://arxiv.org/html/2608.13717#bib.bib10)\]to teacher\-student momentum methods\[[6](https://arxiv.org/html/2608.13717#bib.bib11)\], dynamic pseudo\-label caches with checkpoint\-averaged teachers\[[22](https://arxiv.org/html/2608.13717#bib.bib12)\], and correctors based on audio\-aware large language models \(LLMs\)\[[19](https://arxiv.org/html/2608.13717#bib.bib18),[16](https://arxiv.org/html/2608.13717#bib.bib2)\]\. Despite varied designs, these methods share a common axis: iterative refinement, exponential moving average \(EMA\) teachers, or auxiliary neural machinery to raise pseudo\-label quality\. For cache\-aware streaming ASR, however, a much simpler recipe using a fixed, domain\-adapted offline teacher has not been reported\.

In this work, we presentStreamHear, a semi\-supervised pipeline for adapting cache\-aware streaming ASR to a target domain, with three contributions\. First, theStreamHear pipelinefine\-tunes an offline transducer teacher on the labeled in\-domain training set, generates pseudo\-labels on the unlabeled portion, and fine\-tunes the streaming student on the mixture, all in a single pass without iteration or auxiliary machinery\. Second, we introduce a prior\-regularizeddynamic\-programming realignmentstep that redistributes ground\-truth words across chunk boundaries in the training data using an ASR\-hypothesis anchor, correcting residual placement errors left by Connectionist Temporal Classification \(CTC\) Segmentation\[[13](https://arxiv.org/html/2608.13717#bib.bib13)\]\. Third, ourempirical study on four datasetsshows that StreamHear consistently outperforms supervised student fine\-tuning and narrows the gap to the offline teacher, with ablations confirming robustness across latency and context configurations\.

## 2Related Work

Pseudo\-labeling for ASR originated with early self\-training work\[[9](https://arxiv.org/html/2608.13717#bib.bib8),[3](https://arxiv.org/html/2608.13717#bib.bib14)\]and was formalized into iterative pipelines by IPL\[[24](https://arxiv.org/html/2608.13717#bib.bib9)\]and slimIPL\[[14](https://arxiv.org/html/2608.13717#bib.bib10)\]\. Subsequent methods have raised label quality with EMA teachers that continuously trail the student \(MPL\[[6](https://arxiv.org/html/2608.13717#bib.bib11)\]\), dynamic pseudo\-label caches with checkpoint\-averaged teachers\[[22](https://arxiv.org/html/2608.13717#bib.bib12)\], confidence\- and uncertainty\-based filtering\[[8](https://arxiv.org/html/2608.13717#bib.bib21),[10](https://arxiv.org/html/2608.13717#bib.bib15),[11](https://arxiv.org/html/2608.13717#bib.bib16)\], incremental retraining with data filtering\[[2](https://arxiv.org/html/2608.13717#bib.bib17)\], and audio\-aware LLM correctors that condition on source audio\[[19](https://arxiv.org/html/2608.13717#bib.bib18),[16](https://arxiv.org/html/2608.13717#bib.bib2)\]\. The closest prior work applies pseudo\-labeling to streaming students: iterative noisy\-student training with a large offline teacher paired with a small streaming student\[[7](https://arxiv.org/html/2608.13717#bib.bib19)\], and knowledge distillation from Whisper into streaming Transformer\-Transducers trained from scratch\[[23](https://arxiv.org/html/2608.13717#bib.bib20)\]\. Neither targets cache\-aware streaming\[[17](https://arxiv.org/html/2608.13717#bib.bib1)\], an architecture that passes activation caches between chunks and eliminates the training\-inference gap\. Across these methods, the teacher is either updated during training or augmented with auxiliary neural machinery, and the streaming setting is served by iterative noisy\-student training or from\-scratch distillation\. Departing from these paradigms, StreamHear uses a fixed, domain\-adapted offline transducer as its teacher, mixing a single round of its pseudo\-labels with the labeled training set for one student fine\-tuning pass, thereby adapting a pretrained cache\-aware streaming transducer without iteration or auxiliary machinery\.

## 3Methodology

Our proposedStreamHearframework consists of three sequential learning stages preceded by a chunk\-level data preparation step\. We denote the offline teacher ASR model asMTM\_\{T\}and the cache\-aware streaming student ASR model asMSM\_\{S\}\. Given a labeled in\-domain training setDL=\{\(xl,yl\)\}D\_\{L\}=\\\{\(x\_\{l\},y\_\{l\}\)\\\}and an unlabeled in\-domain setDU=\{xu\}D\_\{U\}=\\\{x\_\{u\}\\\}, the pipeline executes teacher fine\-tuning, pseudo\-label generation, and student fine\-tuning in order, as summarized in Algorithm[1](https://arxiv.org/html/2608.13717#alg1)\. Unlike iterative pseudo\-labeling frameworks, StreamHear runs each stage exactly once and yields a single deployable streaming checkpoint\.

Algorithm 1StreamHear: Pseudo\-Labeling for Streaming ASR1:Models:Offline teacher

MTM\_\{T\}, Streaming student

MSM\_\{S\}
2:Data:Labeled

DL=\{\(xl,yl\)\}D\_\{L\}=\\\{\(x\_\{l\},y\_\{l\}\)\\\}, Unlabeled

DU=\{xu\}D\_\{U\}=\\\{x\_\{u\}\\\}
3:// Teacher fine\-tuning

4:Fine\-tune

MTM\_\{T\}on

DLD\_\{L\}
5:// Pseudo\-label generation

6:

DU′←\{\(xu,MT​\(xu\)\)∣xu∈DU\}D^\{\\prime\}\_\{U\}\\leftarrow\\\{\(x\_\{u\},M\_\{T\}\(x\_\{u\}\)\)\\mid x\_\{u\}\\in D\_\{U\}\\\}
7:// Optional confidence filter

8:Retain the top\-

KK% of

DU′D^\{\\prime\}\_\{U\}ranked by teacher log\-likelihood

9:// Student fine\-tuning

10:Fine\-tune

MSM\_\{S\}on

DL∪DU′D\_\{L\}\\cup D^\{\\prime\}\_\{U\}
11:return

MSM\_\{S\}

Teacher fine\-tuning\.The offline teacherMTM\_\{T\}, a full\-context transducer, is fine\-tuned on the labeled training setDLD\_\{L\}under the standard transducer loss\. This stage domain\-adapts the teacher to the target audio distribution, raising the quality of the pseudo\-labels it subsequently generates\.Pseudo\-label generation\.The fine\-tuned teacher transcribes the unlabeled portionDUD\_\{U\}once with greedy decoding, yielding pseudo\-labeled chunksDU′=\{\(xu,yu′\)\}D^\{\\prime\}\_\{U\}=\\\{\(x\_\{u\},y^\{\\prime\}\_\{u\}\)\\\}whereyu′=MT​\(xu\)y^\{\\prime\}\_\{u\}=M\_\{T\}\(x\_\{u\}\)\. Optionally, pseudo\-labels can be filtered to the top\-KK% ranked by sequence\-average teacher log\-likelihood\.Student fine\-tuning\.The cache\-aware streaming studentMSM\_\{S\}is fine\-tuned on the mixed manifestDL∪DU′D\_\{L\}\\cup D^\{\\prime\}\_\{U\}under the standard RNN\-T loss\.

Chunk\-level data preparation\.Cache\-aware streaming training requires each chunk to carry only the words spoken within its time interval\. We segment full\-length recordings using voice\-activity detection \(VAD\) and compute word\-level timestamps by applying CTC\-Segmentation\[[13](https://arxiv.org/html/2608.13717#bib.bib13)\]on top of a self\-supervised acoustic model\. This baseline leaves residual placement drift near non\-speech tags and VAD boundaries\. To repair these errors, we introduce a prior\-regularized dynamic\-programming \(DP\) realignment step, summarized in Algorithm[2](https://arxiv.org/html/2608.13717#alg2)\. We flatten the ground\-truth transcript and per\-chunk ASR hypothesis into two word streams, each word tagged with its host chunk index\. We then run Needleman\-Wunsch alignment scored by word match rewards, skip costs, and a prior penalty on the chunk\-index displacement between paired words\. Matched ground\-truth words inherit their hypothesis partner’s chunk; unmatched words keep their CTC\-Segmentation chunk, and a two\-pass sweep clamps them into monotonic order\.

Algorithm 2Dynamic\-Programming Realignment1:Input:Per\-chunk ground\-truth text and ASR hypothesis, match reward

rr, mismatch cost

cmc\_\{m\}, skip costs

cg,chc\_\{g\},c\_\{h\}, prior weight

λ\\lambda
2:Output:Updated word\-to\-chunk assignment

3:Concatenate ground\-truth words into

\(w1,…,wN\)\(w\_\{1\},\\dots,w\_\{N\}\)and hypothesis words into

\(h1,…,hM\)\(h\_\{1\},\\dots,h\_\{M\}\), tagging each with its host chunk index

cigtc^\{\\text\{gt\}\}\_\{i\},

cjhypc^\{\\text\{hyp\}\}\_\{j\}
4:Initialize

d​p​\[0,0\]=0dp\[0,0\]=0,

dp\[i,0\]=−i⋅cgdp\[i,0\]=\-i\\cdot c\_\{g\}for

i=1\.\.Ni=1\.\.N,

dp\[0,j\]=−j⋅chdp\[0,j\]=\-j\\cdot c\_\{h\}for

j=1\.\.Mj=1\.\.M
5:For

i=1\.\.Ni=1\.\.Nand

j=1\.\.Mj=1\.\.M, set

d​p​\[i,j\]dp\[i,j\]to the best of: diagonal

d​p​\[i−1,j−1\]\+si​j−λ​\|cjhyp−cigt\|dp\[i\{\-\}1,j\{\-\}1\]\+s\_\{ij\}\-\\lambda\|c^\{\\text\{hyp\}\}\_\{j\}\-c^\{\\text\{gt\}\}\_\{i\}\|with

si​j=rs\_\{ij\}\{=\}rif

wi≈hjw\_\{i\}\{\\approx\}h\_\{j\}else

−cm\-c\_\{m\}; vertical

d​p​\[i−1,j\]−cgdp\[i\{\-\}1,j\]\-c\_\{g\}; horizontal

d​p​\[i,j−1\]−chdp\[i,j\{\-\}1\]\-c\_\{h\}
6:Traceback from

\(N,M\)\(N,M\): matched

wiw\_\{i\}inherits

cjhypc^\{\\text\{hyp\}\}\_\{j\}; unmatched

wiw\_\{i\}keeps

cigtc^\{\\text\{gt\}\}\_\{i\}
7:Two\-pass sweep clamps unmatched words into monotonic chunk order

## 4Experiments

This section describes the datasets and their chunk\-level preparation, the training and inference setup, the main WER comparison against pretrained and fine\-tuned baselines, and five ablations probing alignment correction, pseudo\-label pool scaling, context sensitivity, per\-latency retraining, and the streaming student architecture\.

### 4\.1Datasets

We evaluate on three public English corpora plus one proprietary call\-center dataset\.Earnings\-21\[[4](https://arxiv.org/html/2608.13717#bib.bib4)\]contains 44 quarterly earnings calls across nine financial sectors with entity\-dense transcripts\.Earnings\-22\[[5](https://arxiv.org/html/2608.13717#bib.bib5)\]contains 125 English earnings calls sourced from global companies spanning seven world\-English regions; four files with predominantly non\-English audio are excluded at prepare time, leaving 121 effective files\.SPGISpeech\[[18](https://arxiv.org/html/2608.13717#bib.bib22)\]is a large\-scale corpus of professionally\-transcribed financial teleconference audio \(mix of narrated presentations and spontaneous Q&A\); we use its publicly\-released small subset, further downsampled per\[[16](https://arxiv.org/html/2608.13717#bib.bib2)\]ratios \(10% labeled train, 30% unlabeled, 10% validation, 10% test\) at the session level\.BankCallis a proprietary human\-annotated set of 155 stereo banking customer\-service calls \(channel 1 customer, channel 2 agent\); the unlabeled pool is 275 additional calls drawn as a 30% subsample of separately\-collected raw audio from the same domain\. Table[1](https://arxiv.org/html/2608.13717#S4.T1)lists per\-split hour totals\.

Table 1:Detailed statistics of the datasets utilized in experiments, reporting the total duration in hours for each partition split and the average utterance length in seconds\.Table 2:Word Error Rate \(WER, %\) comparison across all datasets\. We report the mean and standard deviation \(subscript\) over55seeds on the labeled test and unlabeled splits\. BankCall reports labeled test only \(unlabeled pool has no transcripts\)\.Boldindicates the best streaming\-student result per column\.For long\-form Earnings\-21 and Earnings\-22, we standardize raw recordings to 16 kHz, normalize non\-speech annotations to unified tags, and produce chunk\-level training data via VAD segmentation with pyannote\-3\.0\[[1](https://arxiv.org/html/2608.13717#bib.bib23)\], CTC\-Segmentation forced alignment using Parakeet\-CTC\-0\.6B\[[21](https://arxiv.org/html/2608.13717#bib.bib24)\], and the DP realignment step of Algorithm[2](https://arxiv.org/html/2608.13717#alg2)with a Whisper\-Large\-v3\[[20](https://arxiv.org/html/2608.13717#bib.bib3)\]hypothesis anchor\. VAD keeps each chunk within Whisper’s 30 s context so the anchor covers it in full\. Longer audio can be anchored with an audio\-LLM such as Voxtral\-Mini\-3B\[[15](https://arxiv.org/html/2608.13717#bib.bib6)\]that batches multi\-window inputs across a longer context\. SPGISpeech utterances arrive pre\-chunked from the source parquets, so preparation reduces to session\-level partitioning at ReHear\[[16](https://arxiv.org/html/2608.13717#bib.bib2)\]ratios\. For BankCall, human annotations already provide chunk\-level segments with speaker channel and timestamps, so no VAD or forced alignment is applied on the labeled portion; digit\-mask redactions are filled by a strict\-majority vote among three off\-the\-shelf ASR models \(Whisper\-Large\-v3, Voxtral\-Mini\-3B, Parakeet\-TDT\-0\.6B\-v3\) plus a human review pass, and the labeled split is stratified by call reason\. The unlabeled BankCall portion is segmented via VAD only\. Across all datasets, chunks with empty transcripts are dropped, and partitioning is enforced at the source file or session level\.

### 4\.2Experimental Setup

Models\.The offline teacher is Parakeet\-TDT\-0\.6B\-v3\[[21](https://arxiv.org/html/2608.13717#bib.bib24)\], a 0\.6B\-parameter FastConformer transducer with token\-and\-duration decoding\. The primary streaming student is Nemotron\-Speech\-Streaming\-EN\-0\.6B\[[17](https://arxiv.org/html/2608.13717#bib.bib1)\], a 0\.6B cache\-aware FastConformer\-RNN\-T trained with multi\-latency right\-context \(RC\) sampling; unless noted, we run inference atRC=1\\text\{RC\}\{=\}1\(8080ms chunk plus8080ms right\-context lookahead, for160160ms of algorithmic latency\)\. The student\-choice ablation additionally evaluates Nemotron\-3\.5\-ASR\-Streaming\-0\.6B, a multilingual variant withLC=56\\text\{LC\}\{=\}56and English language\-ID conditioning\.

Training\.All fine\-tuning runs share the same recipe: AdamW optimizer with learning rate2×10−42\\times 10^\{\-4\}, betas\[0\.9,0\.98\]\[0\.9,0\.98\], weight decay10−310^\{\-3\}; cosine schedule with10%10\\%warmup and minimum learning rate10−610^\{\-6\};1010epochs; bf16 precision; SpecAugment \(22freq masks×\\times1010time masks\)\. Effective batch size6464\(44A100\-SXM\-40GB GPUs, per\-GPU batch44, gradient accumulation44\) for the streaming student; teacher fine\-tuning uses per\-GPU batch22with gradient accumulation88to fit the RNN\-T joint on Earnings\-length chunks; Nemotron\-3\.5 drops to per\-GPU batch11with gradient accumulation1616\. Each experimental cell is repeated for55seeds with matched teacher\-student pairings for the pseudo\-labeling runs; we report mean and standard deviation\. All training and inference use the NVIDIA NeMo Speech framework111[https://github\.com/NVIDIA\-NeMo/Speech](https://github.com/NVIDIA-NeMo/Speech)\[[12](https://arxiv.org/html/2608.13717#bib.bib7)\]\.

Inference and metrics\.Inference uses greedy decoding under the cache\-aware streaming attention mask\. Word error rate \(WER\) is computed after a text normalization pipeline: bracket removal, number and typography normalization, lowercasing, contraction expansion, spelling and compound normalization, diacritic removal, filler\-word removal, and punctuation removal\. Latency is reported as the algorithmic latency \(\(1\+RC\)×80\(1\{\+\}\\text\{RC\}\)\\times 80ms\) imposed by the right\-context lookahead\.

### 4\.3Experimental Results

As presented in Table[2](https://arxiv.org/html/2608.13717#S4.T2), StreamHear consistently outperforms the fine\-tuned student on all four datasets: by0\.180\.18to0\.880\.88percentage points \(pp\) of word error rate on the labeled test split, and by0\.440\.44to1\.851\.85pp on the held\-out unlabeled split of the three public benchmarks\. Against the offline fine\-tuned teacher, StreamHear narrows the labeled\-test gap to at most0\.950\.95pp and the unlabeled gap to at most0\.110\.11pp; on Earnings\-22 unlabeled, the streaming student essentially matches the offline teacher \(9\.56%9\.56\\%vs9\.55%9\.55\\%\)\. These results confirm that pseudo\-labels from a domain\-adapted offline teacher transfer to a streaming student in a single fine\-tuning pass, without iterative refinement or auxiliary neural machinery\. On BankCall’s stereo audio, StreamHear’s0\.880\.88pp gain is essentially symmetric across the customer and agent channels, bringing streaming WER to12\.55%12\.55\\%and7\.00%7\.00\\%respectively \(−0\.90\-0\.90and−0\.85\-0\.85pp vs the fine\-tuned student\) and preserving the intrinsic∼5\.5\{\\sim\}5\.5pp customer/agent WER gap that reflects acoustic quality rather than a labeling asymmetry\.

### 4\.4Ablation Studies

We report five ablations\. All experiments use Nemotron\-Speech\-Streaming\-EN\-0\.6B on Earnings\-21 unless noted; all numbers are mean WER \(%\) over55seeds under the same normalization pipeline as Table[2](https://arxiv.org/html/2608.13717#S4.T2)\.

#### 4\.4\.1Alignment Correction

We measure alignment quality by running the pretrained offline teacher \(Parakeet\-TDT\-0\.6B\-v3\) on each dataset’s chunk text before and after DP realignment, computing WER against the teacher’s own audio hypothesis on the same chunks\. Since the DP step uses Whisper\-Large\-v3 as its position prior \(not Parakeet\), the teacher is an independent anchor\. Realignment cuts alignment\-quality WER by roughly99pp on both datasets \(Table[3](https://arxiv.org/html/2608.13717#S4.T3)\) while changing chunk count by at most0\.4%0\.4\\%, confirming that words are reassigned to the correct chunks rather than merely dropped\.

Table 3:Alignment\-quality WER \(%\) on chunk text against a pretrained Parakeet\-TDT\-0\.6B\-v3 audio hypothesis, before and after DP realignment\.
#### 4\.4\.2Confidence Filtering

We sweep a top\-KK% confidence filter on pseudo\-labels ranked by sequence\-average teacher log\-likelihood, forK∈\{25,50,75,100\}K\{\\in\}\\\{25,50,75,100\\\}, on Earnings\-21 and Earnings\-22 \(Table[4](https://arxiv.org/html/2608.13717#S4.T4)\)\. LargerKKmonotonically improves WER on both datasets and both splits; discarding low\-confidence pseudo\-labels never helps\. EvenK=25%K\{=\}25\\%already outperforms the fine\-tuned student, indicating that a domain\-adapted teacher produces uniformly usable pseudo\-labels and that quantity outweighs top\-KKquality\.

Table 4:Pseudo\-label pool scaling with a top\-KK% confidence filter\. WER \(%, mean±std\{\}\_\{\\pm\\text\{std\}\}\) as labeled test / unlabeled\.
#### 4\.4\.3Context Sweeps

We decode the base checkpoint, the fine\-tuned student, and StreamHear across right\-contextRC∈\{0,1,3,6,13\}\\text\{RC\}\{\\in\}\\\{0,1,3,6,13\\\}at fixedLC=70\\text\{LC\}\{=\}70, giving algorithmic latencies from8080ms to1\.121\.12s\. The streaming student is pretrained at RCs\{0,1,6,13\}\\\{0,1,6,13\\\};RC=3\\text\{RC\}\{=\}3is an interior point reached only through multi\-latency generalization\. The fine\-tuned student and StreamHear are each trained once at the default configuration\[LC=70,RC=1\]\[\\text\{LC\}\{=\}70,\\text\{RC\}\{=\}1\]and then decoded at every RC in the sweep\. Figure[1](https://arxiv.org/html/2608.13717#S4.F1)plots labeled test WER \(%\) against the algorithmic latency\(1\+RC\)×80\(1\{\+\}\\text\{RC\}\)\\times 80ms for the three model conditions\. Under all three, WER decreases with RC and saturates byRC=6\\text\{RC\}\{=\}6\. StreamHear preserves the qualitative shape of the base and fine\-tuned curves, so switching fromRC=1\\text\{RC\}\{=\}1toRC=13\\text\{RC\}\{=\}13yields only0\.200\.20pp at the cost of0\.960\.96s of extra algorithmic latency\. We similarly sweep left\-contextLC∈\{18,35,70,140,280\}\\text\{LC\}\{\\in\}\\\{18,35,70,140,280\\\}at fixedRC=1\\text\{RC\}\{=\}1\(Figure[2](https://arxiv.org/html/2608.13717#S4.F2)\), decoding the same checkpoints at each LC; all three curves reach their minima near the trained valueLC=70\\text\{LC\}\{=\}70and rise on either side, showing a much weaker effect than RC \(spread≤0\.65\{\\leq\}0\.65pp across the whole curve\)\.

Figure 1:Right\-context \(RC\) sweep on Earnings\-21 labeled test WER \(%\) at fixedLC=70\\text\{LC\}\{=\}70\. Algorithmic latency is\(1\+RC\)×80\(1\{\+\}\\text\{RC\}\)\\times 80ms\. Error bars are±1\{\\pm\}1std; student inference is deterministic, and StreamHear std \(≤0\.06\{\\leq\}0\.06pp\) is smaller than the marker\.Figure 2:Left\-context \(LC\) sweep on Earnings\-21 labeled test WER \(%\) at fixedRC=1\\text\{RC\}\{=\}1\. Left context isLC×80\\text\{LC\}\\times 80ms\. Error bars are±1\{\\pm\}1std; StreamHear std \(≤0\.05\{\\leq\}0\.05pp\) is smaller than the marker\.
#### 4\.4\.4Per\-Latency Retraining

We fine\-tune a separate checkpoint at eachRC∈\{0,3,6,13\}\\text\{RC\}\{\\in\}\\\{0,3,6,13\\\}\(both supervised and StreamHear\) and evaluate at the matching RC\. Compared against decoding the single checkpoint trained atRC=1\\text\{RC\}\{=\}1at those RCs \(Figure[1](https://arxiv.org/html/2608.13717#S4.F1)\), supervised per\-RC training outperforms decode\-sweep by0\.020\.02to0\.060\.06pp on labeled test across the four RCs, while StreamHear per\-RC training matches decode\-sweep to within0\.010\.01pp at every RC\. A single StreamHear checkpoint trained atRC=1\\text\{RC\}\{=\}1therefore suffices for the whole latency curve, obviating per\-RC retraining\.

#### 4\.4\.5Student Choice

We replace the primary student, hereafter Nemotron\-EN \(Nemotron\-Speech\-Streaming\-EN\-0\.6B\), with the multilingual Nemotron\-ML \(Nemotron\-3\.5\-ASR\-Streaming\-0\.6B, trained atLC=56\\text\{LC\}\{=\}56, prompt\-conditioned decoder\) and re\-run the full StreamHear pipeline on Earnings\-21 \(Table[5](https://arxiv.org/html/2608.13717#S4.T5)\)\. StreamHear outperforms supervised FT by0\.440\.44pp on labeled test and2\.282\.28pp on unlabeled, mirroring Nemotron\-EN’s0\.560\.56and1\.851\.85pp deltas: both students absorb a similar amount of extra signal from pseudo\-labels\. Absolute WER for Nemotron\-ML stays0\.890\.89\-2\.822\.82pp higher than Nemotron\-EN at every stage \(Student, FT Student, StreamHear\), so the choice of pretrained backbone sets a ceiling that StreamHear does not fully lift, but the recipe generalizes across streaming student architectures\.

Table 5:Student\-choice ablation on Earnings\-21 WER \(%, mean±std\{\}\_\{\\pm\\text\{std\}\}\) as labeled test / unlabeled\.

## 5Conclusion

We presented StreamHear, a semi\-supervised recipe that adapts a cache\-aware streaming ASR student to a target domain by fine\-tuning on pseudo\-labels from a domain\-adapted offline teacher\. A prior\-regularized DP realignment step redistributes ground\-truth words across chunk boundaries using an ASR hypothesis anchor\. Across Earnings\-21, Earnings\-22, SPGISpeech, and a proprietary call\-center corpus, BankCall, StreamHear consistently outperforms the supervised streaming student on both labeled test and unlabeled, and narrows the gap to the offline teacher\. Ablations show these gains are robust across latency operating points, pseudo\-label pool sizes, context settings, and student architectures\.

## References

- \[1\]H\. Bredin, R\. Yin, J\. M\. Coria, G\. Gelly, P\. Korshunov, M\. Lavechin, D\. Fustes, H\. Titeux, W\. Bouaziz, and M\. Gill\(2020\)Pyannote\.audio: neural building blocks for speaker diarization\.InICASSP 2020 \- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 7124–7128\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p2.1)\.
- \[2\]A\. Carofilis, P\. Rangappa, S\. Madikeri, S\. Kumar, S\. Burdisso, J\. Prakash, E\. Villatoro\-Tello, P\. Motlicek, B\. Sharma, K\. Hacioglu,et al\.\(2025\)Better semi\-supervised learning for multi\-domain ASR through incremental retraining and data filtering\.InProc\. Interspeech 2025,pp\. 3618–3622\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[3\]Y\. Chen, W\. Wang, and C\. Wang\(2020\)Semi\-supervised ASR by end\-to\-end self\-training\.InProc\. Interspeech 2020,pp\. 2787–2791\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[4\]M\. Del Rio, N\. Delworth, R\. Westerman, M\. Huang, N\. Bhandari, J\. Palakapilly, Q\. McNamara, J\. Dong, P\. Żelasko, and M\. Jetté\(2021\)Earnings\-21: a practical benchmark for ASR in the wild\.InProc\. Interspeech 2021,pp\. 3465–3469\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p1.1)\.
- \[5\]M\. Del Rio, P\. Ha, Q\. McNamara, C\. Miller, and S\. Chandra\(2022\)Earnings\-22: a practical benchmark for accents in the wild\.arXiv preprint arXiv:2203\.15591\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p1.1)\.
- \[6\]Y\. Higuchi, N\. Moritz, J\. Le Roux, and T\. Hori\(2021\)Momentum pseudo\-labeling for semi\-supervised speech recognition\.InProc\. Interspeech 2021,pp\. 726–730\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[7\]D\. Hwang, K\. C\. Sim, Z\. Huo, and T\. Strohman\(2022\)Pseudo label is better than human label\.InProc\. Interspeech 2022,pp\. 1421–1425\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[8\]Z\. Jin, D\. Zhong, X\. Song, Z\. Liu, N\. Ye, and Q\. Zeng\(2022\)Filter and evolve: progressive pseudo label refining for semi\-supervised automatic speech recognition\.arXiv preprint arXiv:2210\.16318\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[9\]J\. Kahn, A\. Lee, and A\. Hannun\(2020\)Self\-training for end\-to\-end speech recognition\.InICASSP 2020 \- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 7084–7088\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[10\]S\. Khurana, N\. Moritz, T\. Hori, and J\. Le Roux\(2021\)Unsupervised domain adaptation for speech recognition via uncertainty driven self\-training\.InICASSP 2021 \- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 6553–6557\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[11\]E\. Kim and K\. Lee\(2025\)Uncertainty\-aware self\-training for CTC\-based automatic speech recognition\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),Vol\.39,pp\. 24330–24338\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[12\]O\. Kuchaiev, J\. Li, H\. Nguyen, O\. Hrinchuk, R\. Leary, B\. Ginsburg, S\. Kriman, S\. Beliaev, V\. Lavrukhin, J\. Cook,et al\.\(2019\)NeMo: a toolkit for building AI applications using Neural Modules\.arXiv preprint arXiv:1909\.09577\.Cited by:[§4\.2](https://arxiv.org/html/2608.13717#S4.SS2.p2.1)\.
- \[13\]L\. Kürzinger, D\. Winkelbauer, L\. Li, T\. Watzel, and G\. Rigoll\(2020\)CTC\-segmentation of large corpora for German end\-to\-end speech recognition\.InInternational Conference on Speech and Computer \(SPECOM\),pp\. 267–278\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p3.1),[§3](https://arxiv.org/html/2608.13717#S3.p3.1)\.
- \[14\]T\. Likhomanenko, Q\. Xu, J\. Kahn, G\. Synnaeve, and R\. Collobert\(2021\)slimIPL: language\-model\-free iterative pseudo\-labeling\.InProc\. Interspeech 2021,pp\. 741–745\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[15\]A\. H\. Liu, A\. Ehrenberg, A\. Lo, G\. Lample, J\. Delignon, K\. R\. Chandu, P\. von Platen, P\. R\. Muddireddy,et al\.\(2025\)Voxtral\.arXiv preprint arXiv:2507\.13264\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p2.1)\.
- \[16\]Z\. Liu, C\. Zhu, S\. Cho, and S\. Zhang\(2026\)ReHear: iterative pseudo\-label refinement for semi\-supervised speech recognition via audio large language models\.arXiv preprint arXiv:2602\.18721\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1),[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p2.1)\.
- \[17\]V\. Noroozi, S\. Majumdar, A\. Kumar, J\. Balam, and B\. Ginsburg\(2024\)Stateful Conformer with cache\-based inference for streaming automatic speech recognition\.InICASSP 2024 \- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 12041–12045\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p1.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1),[§4\.2](https://arxiv.org/html/2608.13717#S4.SS2.p1.1)\.
- \[18\]P\. K\. O’Neill, V\. Lavrukhin, S\. Majumdar, V\. Noroozi, Y\. Zhang, O\. Kuchaiev, J\. Balam, Y\. Dovzhenko, K\. Freyberg, M\. D\. Shulman,et al\.\(2021\)SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end\-to\-end speech recognition\.InProc\. Interspeech 2021,pp\. 1434–1438\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p1.1)\.
- \[19\]J\. Prakash, B\. Kumar, K\. Hacioglu, B\. Sharma, S\. Gopalan, M\. Chetlur, S\. Venkatesan, and A\. Stolcke\(2025\)Better pseudo\-labeling with multi\-ASR fusion and error correction by SpeechLLM\.InProc\. Interspeech 2025,pp\. 579–583\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[20\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InProceedings of the International Conference on Machine Learning \(ICML\),pp\. 28492–28518\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p2.1)\.
- \[21\]D\. Rekesh, N\. R\. Koluguri, S\. Kriman, S\. Majumdar, V\. Noroozi, H\. Huang, O\. Hrinchuk, K\. Puvvada, A\. Kumar, J\. Balam, and B\. Ginsburg\(2023\)Fast conformer with linearly scalable attention for efficient speech recognition\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§4\.1](https://arxiv.org/html/2608.13717#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.13717#S4.SS2.p1.1)\.
- \[22\]N\. Tadevosyan, N\. Karpov, A\. Andrusenko, V\. Lavrukhin, and A\. Jukic\(2025\)Unified semi\-supervised pipeline for automatic speech recognition\.InProc\. Interspeech 2025,pp\. 3184–3188\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[23\]I\. Thorbecke, J\. Zuluaga\-Gomez, E\. Villatoro\-Tello, S\. Kumar, P\. Rangappa, S\. Burdisso, P\. Motlicek, S\. K\. Pandia, and A\. Ganapathiraju\(2024\)Fast streaming transducer ASR prototyping via knowledge distillation with whisper\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 16747–16762\.Cited by:[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.
- \[24\]Q\. Xu, T\. Likhomanenko, J\. Kahn, A\. Hannun, G\. Synnaeve, and R\. Collobert\(2020\)Iterative pseudo\-labeling for speech recognition\.InProc\. Interspeech 2020,pp\. 1006–1010\.Cited by:[§1](https://arxiv.org/html/2608.13717#S1.p2.1),[§2](https://arxiv.org/html/2608.13717#S2.p1.1)\.