Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming

# Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming
Source: [https://arxiv.org/html/2606.10675](https://arxiv.org/html/2606.10675)
Weber Zehavi Rousso Keshet

MeidanRotemJosephFaculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, 3200003, Israel[jkeshet@technion\.ac\.il](https://arxiv.org/html/2606.10675v1/mailto:[email protected])

###### Abstract

We present a method for accurate multilingual word\-level forced alignment, consisting of an alignment encoder and a learned alignment decoder\. The encoder integrates two representations: one from the Massively Multilingual Speech \(MMS\) model and another from a self\-supervised phoneme boundary detector \(UnSupSeg\)\. It learns to fuse them and to estimate word\-boundary probabilities over long temporal contexts\. The alignment decoder is a learned dynamic programming that combines encoder outputs with segmental features over the MMS and UnSupSeg representations to infer final word boundaries\. Trained iteratively on TIMIT and Buckeye, the proposed approach outperforms Montreal Forced Aligner \(MFA\) and MMS\-based alignment on both datasets\. On unseen languages \(Dutch, German, and Hebrew\), the proposed model achieves performance consistently better than or on par with existing alignment approaches, indicating its potential to scale to 1100\+ languages supported by MMS without further training\.

###### keywords:

forced alignment, word alignment, self\-supervised representation, multilingual

## 1Introduction

Accurate word\-level forced alignment is a fundamental component in speech and language processing\. Precise temporal alignment between audio signals and textual transcriptions enables fine\-grained analysis in linguistics, including the study of phonetics, phonology, prosody, and dialectal variation across languages\. Beyond linguistic research, reliable alignment is critical for the development and evaluation of speech technologies such as automatic speech recognition \(ASR\), speech synthesis, audio indexing, and segmentation systems\.

Recent advances in sequence modeling have led to substantial improvements in speech modeling and ASR, exemplified by systems such as wav2vec 2\.0\[wav2vec2\], HuBERT\[Hsu21\-HUB\], and Whisper\[Radford22\-RSR\]\. These approaches have markedly improved recognition accuracy and robustness across varied acoustic environments and languages\. However, for forced alignment tasks, traditional HMM\-GMM frameworks remain widely used and highly competitive\[Rousso24\-FACOMP\]\. In particular, the Montreal Forced Aligner \(MFA\)\[McAuliffe17\-MFA\]has established itself as one of the leading toolkits for word\- and phoneme\-level alignment, consistently ranking among the top\-performing systems in recent evaluations\[Rousso24\-FACOMP\]\.

The paper concludes with a comprehensive empirical evaluation\. We first detail the hyperparameter tuning and model selection procedure, and then report results on multiple manually annotated speech corpora with word\-level timestamps\. These include TIMIT\[Garofolo93\-TIM\], Buckeye\[Pitt05\-BUC\], a Hebrew dataset\[benshalom14\], the IFA Corpus for Dutch\[VanSon01\-IFA\], and PHONDAT for German\[tillmann1993theoretical\]\. The results demonstrate that our method outperforms competing approaches, including MFA and CTC\-based MMS forced alignment, on the English datasets\. On unseen languages, the proposed approach achieves superior performance at alignment tolerances of 50 msec and above\. Code and pretrained checkpoints will be released upon acceptance111[https://github\.com/MLSpeech/Multilingual\-Word\-Aligner](https://github.com/MLSpeech/Multilingual-Word-Aligner)\.

## 2Method

We assume that a waveform consisting ofTTsamples is transformed into a sequence ofLLframes, with the frame duration of 10 msec\. The speech utterance is represented as𝐗=\(𝐱1,…,𝐱L\)\\mathbf\{X\}=\(\\mathbf\{x\}\_\{1\},\\dots,\\mathbf\{x\}\_\{L\}\), where each frame𝐱l∈ℝd\\mathbf\{x\}\_\{l\}\\in\\mathbb\{R\}^\{d\}for1≤l≤L1\\leq l\\leq Lis add\-dimensional feature vector, and thus𝐗∈ℝL×d\\mathbf\{X\}\\in\\mathbb\{R\}^\{L\\times d\}\.

Let𝐰=\(w1,…,wK\)\\mathbf\{w\}=\(w\_\{1\},\\dots,w\_\{K\}\)denote the sequence of words in the utterance, whereKKis the number of words in the utterance, and each wordwk∈𝒱w\_\{k\}\\in\\mathcal\{V\}belongs to the vocabulary𝒱\\mathcal\{V\}\. Our objective is to determine the start time of each word given the input speech and its corresponding word sequence\. The sequence of these start times, referred to as the*alignment sequence*, is denoted by𝐚=\(a1,…,aK\)\\mathbf\{a\}=\(a\_\{1\},\\dots,a\_\{K\}\), where eachak∈ℒ=\{1,…,L\}a\_\{k\}\\in\\mathcal\{L\}=\\\{1,\\dots,L\\\}represents the frame index at which thekk\-th word end\.

Our approach builds uponMMpre\-trained models for speech alignment representation, and aims to refine them to produce a more accurate alignment sequence than any single model alone\. Each model takes as input the speech utterance𝐗\\mathbf\{X\}and the corresponding word sequence𝐰\\mathbf\{w\}, and outputs a sequence ofLLvectors, each indicating the confidence that the associated frame index corresponds to an alignment point\. The word sequence is optional for the pre\-trained models, as some of them generate the alignment sequence directly from the speech utterance alone\. Specifically, the dimensionality of the vectors produced by themm\-th model isDmD\_\{m\}\. Themm\-th model is then represented as𝐒m=fm​\(𝐗,𝐰\)\\mathbf\{S\}\_\{m\}=f\_\{m\}\(\\mathbf\{X\},\\mathbf\{w\}\)where𝐒m=\[𝐬m,1,…,𝐬m,L\]∈ℝDm×L\\mathbf\{S\}\_\{m\}=\[\\mathbf\{s\}\_\{m,1\},\\ldots,\\mathbf\{s\}\_\{m,L\}\]\\in\\mathbb\{R\}^\{D\_\{m\}\\times L\},𝐬m,l∈ℝDm\\mathbf\{s\}\_\{m,l\}\\in\\mathbb\{R\}^\{D\_\{m\}\}for the pre\-trained models1≤m≤M1\\leq m\\leq M\.

The output vectors from the pre\-trained models are normalized and concatenated to form the input to the alignment representation encoder, denotedgθg\_\{\\theta\}, whereθ\\thetarepresents the encoder parameters\. This encoder outputs𝐳=gθ​\(𝐒\)\\mathbf\{z\}=g\_\{\\theta\}\(\\mathbf\{S\}\), where𝐳∈\[0,1\]L\\mathbf\{z\}\\in\[0,1\]^\{L\}is a refined word boundaries given as a probability distribution over theLLframes, indicating the likelihood of a word boundary at each frame\.

The final part of our proposed model is an alignment decoderhψh\_\{\\psi\}, whereψ\\psirepresents the decoder’s parameters\. This decoder is a learned dynamic programming module that takes as input the encoder probability distribution𝐳\\mathbf\{z\}, the representations𝐒\\mathbf\{S\}, and the word sequence𝐰\\mathbf\{w\}and outputs an alignment sequence corresponding to the end times of the input words,𝐚^∈ℒK\\hat\{\\mathbf\{a\}\}\\in\\mathcal\{L\}^\{K\}\.

### 2\.1Alignment Representations

We used two representation models \(M=2M=2\)\. The first representation modelf1f\_\{1\}is based on an unsupervised phoneme segmentation framework, referred to as*UnSupSeg*, which is trained using self\-supervised contrastive learning\[Kreuk20\-SCL\]\. This model learns to identify phoneme boundaries directly from raw speech signals without relying on annotated data, enabling it to capture fine\-grained acoustic transitions that are informative for downstream word segmentation\. This model takes speech samples𝐗\\mathbf\{X\}as input and produces a sequence of vectors𝐒1=\[𝐬1,1,…,𝐬1,L\]\\mathbf\{S\}\_\{1\}=\[\\mathbf\{s\}\_\{1,1\},\\ldots,\\mathbf\{s\}\_\{1,L\}\], where each vector𝐬1,l∈ℝD1\\mathbf\{s\}\_\{1,l\}\\in\\mathbb\{R\}^\{D\_\{1\}\}corresponds to a 10 msec audio frame computed with an analysis window of 30 msec\.

The second representationf2f\_\{2\}is derived from the MMS self\-supervised speech model\[pratap2024scaling\]\. For a given speech utterance𝐗\\mathbf\{X\}and a sequence of words𝐰\\mathbf\{w\}, CTC alignment is applied\. The representation is defined as the likelihood of thekk\-th word,wkw\_\{k\}, start at frameaka\_\{k\}, and it is zero otherwise\. Formally,

s2,l=\{P​\(wk∣𝐗\),l=ak0,otherwise\.s\_\{2,l\}=\\begin\{cases\}P\(w\_\{k\}\\mid\\mathbf\{X\}\),&l=a\_\{k\}\\\\ 0,&\\text\{otherwise\}\\end\{cases\}~\.\(1\)This representation is extracted every 20 msec and each element having dimensionalityD2=1D\_\{2\}=1\. To maintain a temporal resolution of 10 ms, we upsampled the representation through simple interpolation\. While alternative strategies for exploiting confidence measures derived from the CTC alignment are possible, we adopt this simple formulation as a straightforward and interpretable representation of word\-level alignment confidence\.

### 2\.2Alignment Encoder

The representations from the pre\-trained representation models are normalized and concatenated to serve as input to the alignment encoder, whose objective is to produce a refined and improved version of these representations,𝐳=gθ​\(𝐒\)\\mathbf\{z\}=g\_\{\\theta\}\(\\mathbf\{S\}\)where𝐒∈ℝL×D\\mathbf\{S\}\\in\\mathbb\{R\}^\{L\\times D\}and𝐳∈\[0,1\]L\\mathbf\{z\}\\in\[0,1\]^\{L\}\. We evaluated several encoder architectures for implementinggθg\_\{\\theta\}, including VGG\[Simonyan15\-VGG\], Transformer encoder\[Hsu21\-HUB\], and Conformer\[Gulati20\-CON\]\. Across all architectures, the final layer was implemented as a softmax layer\.

The encoder parametersθ\\thetaare trained independently of the alignment decoder\. Specifically, the encoder is optimized for a binary classification task that predicts whether a given frame in the input sequence corresponds to a word boundary\. As boundary frames are substantially outnumbered by non\-boundary frames, the task exhibits a pronounced class imbalance\. To address this issue, we employ the*focal loss*\[lin2017focal\]instead of the standard cross\-entropy loss, as it is explicitly designed to mitigate the effects of class imbalance\. The focal loss hyperparameters \(α,γ\\alpha,\\gamma\) were selected via grid search on the validation set\.

### 2\.3Alignment Decoder

The final component of our approach is the alignment decoder, that is a learnable dynamic programming \(DP\) module that refines the raw boundary probabilities produced by the encoder\. Specifically, the objective of the decoder is to predict a sequence ofKKalignments \(start times\)𝐚\\mathbf\{a\}corresponding to theKKinput words𝐰\\mathbf\{w\}\. The decoder takes as input the predicted frame\-level boundary probabilities𝐳=g​\(𝐒,𝐰\)\\mathbf\{z\}=g\(\\mathbf\{S\},\\mathbf\{w\}\), the alignment representations𝐒\\mathbf\{S\}, and the input word sequence𝐰\\mathbf\{w\}\. Formally, the decoder is defined as

𝐚^=arg​max𝐚⁡hψ​\(𝐒,𝐳,𝐰,𝐚\)\.\\hat\{\\mathbf\{a\}\}=\\operatorname\*\{arg\\,max\}\_\{\\mathbf\{a\}\}~h\_\{\\psi\}\(\\mathbf\{S\},\\mathbf\{z\},\\mathbf\{w\},\\mathbf\{a\}\)\.\(2\)Following Keshet et al\.\[keshet2007large\], we model the decoder as a linear combination ofNNfeature functionsϕn\{\\phi\_\{n\}\}where each feature function evaluates a proposed alignment by assigning higher scores to well\-placed alignments and lower scores to poorly placed ones\. Formally, the decoder is formulated as follows

hψ​\(𝐒,𝐳,𝐰,𝐚\)=∑n=1N∑k=1Kψn​ϕn​\(𝐒,𝐳,wk,ak−1,ak\)h\_\{\\psi\}\(\\mathbf\{S\},\\mathbf\{z\},\\mathbf\{w\},\\mathbf\{a\}\)=\\sum\_\{n=1\}^\{N\}\\sum\_\{k=1\}^\{K\}\\psi\_\{n\}\\phi\_\{n\}\(\\mathbf\{S\},\\mathbf\{z\},w\_\{k\},a\_\{k\-1\},a\_\{k\}\)\(3\)Each feature function captures a distinct structural aspect of alignment quality, such as temporal consistency, boundary likelihood, or agreement with the input word sequence\. The feature functions used in the decoder are described next\. Finding the alignment sequence that maximizeshψh\_\{\\psi\}is done using a dynamic programming with constraints on the minimal word duration,

𝐚^=arg​max𝐚:ak−ak−1\>Lmin​∑n=1N∑k=1Kψn​ϕn​\(𝐒,𝐳,wk,ak−1,ak\)\\hat\{\\mathbf\{a\}\}=\\operatorname\*\{arg\\,max\}\_\{\\mathbf\{a\}:a\_\{k\}\-a\_\{k\-1\}\>L\_\{\\text\{min\}\}\}~~\\sum\_\{n=1\}^\{N\}\\sum\_\{k=1\}^\{K\}\\psi\_\{n\}\\phi\_\{n\}\(\\mathbf\{S\},\\mathbf\{z\},w\_\{k\},a\_\{k\-1\},a\_\{k\}\)\(4\)The parameters\{ψn\}\\\{\\psi\_\{n\}\\\}are optimized by iterative optimization process\[keshet2007large\]\.

The first feature function,ϕ1\\phi\_\{1\}, measures the Euclidean distance between representations around a candidate boundaryaka\_\{k\}\. This feature is computed using the UnSupSeg model representations, under the assumption that when the boundaryaka\_\{k\}is correctly placed, the representations immediately before and after the boundary exhibit a large distance\. Formally,

ϕ1​\(𝐒,𝐳,wk,ak−1,ak\)=∥𝐬1,ak−1−𝐬1,ak\+1∥22\.\\phi\_\{1\}\(\\mathbf\{S\},\\mathbf\{z\},w\_\{k\},a\_\{k\-1\},a\_\{k\}\)=\\lVert\\mathbf\{s\}\_\{1,a\_\{k\}\-1\}\-\\mathbf\{s\}\_\{1,a\_\{k\}\+1\}\\rVert^\{2\}\_\{2\}~\.\(5\)
The second feature function incorporates the word transition scores derived from the encoder's output\. The word transition score is calculated as follows:

ϕ2​\(𝐒,𝐳,wk,ak−1,ak\)=zak\.\\phi\_\{2\}\(\\mathbf\{S\},\\mathbf\{z\},w\_\{k\},a\_\{k\-1\},a\_\{k\}\)=z\_\{a\_\{k\}\}~\.\(6\)
The third feature function is also based on the encoder's output\. For each wordwkw\_\{k\}, this feature function is the normalized sum of the encoder's output from the beginning of the word till the end\. Namely,

ϕ3​\(𝐒,𝐳,wk,ak−1,ak\)=−1ak−ak−1−1​∑l=ak−1\+1ak−1zl\.\\phi\_\{3\}\(\\mathbf\{S\},\\mathbf\{z\},w\_\{k\},a\_\{k\-1\},a\_\{k\}\)=\-\\frac\{1\}\{a\_\{k\}\{\-\}a\_\{k\-1\}\{\-\}1\}\\sum\_\{l=a\_\{k\-1\}\+1\}^\{a\_\{k\}\-1\}z\_\{l\}~\.\(7\)Note that this feature function is assigned a negative sign, since it yields alowvalue when the normalized sum of the encoder’s output corresponds to a single word, and a high value otherwise\.

The fourth feature function is derived from the MMS representation\[pratap2024scaling\]\. The MMS emissions matrix, denoted as𝐔MMS\\mathbf\{U\}^\{\\text\{MMS\}\}, represents the probability distribution of each letter, given the speech utterance𝐗\\mathbf\{X\}\. This matrix has dimensions equal to the number of frames and the number of letter in the language\. The fourth feature function is defined as the likelihood of the emission of the characters of the wordwkw\_\{k\}\. We sum the emissions in this time interval for the letters that make up the word:

ϕ4​\(𝐒,𝐳,wk,ak−1,ak\)=∑l=ak−1ak∑c∈wkUl,cMMS\.\\phi\_\{4\}\(\\mathbf\{S\},\\mathbf\{z\},w\_\{k\},a\_\{k\-1\},a\_\{k\}\)=\\sum\_\{l=a\_\{k\-1\}\}^\{a\_\{k\}\}\\sum\_\{c\\in w\_\{k\}\}U^\{\\text\{MMS\}\}\_\{l,c\}~\.\(8\)This feature is high when the alignment sequence accurately represents the word sequence\.

## 3Experimental Results

### 3\.1Datasets

We trained and evaluated the proposed method on the TIMIT\[Garofolo93\-TIM\]and Buckeye\[Pitt05\-BUC\]speech corpora\. These datasets provide manually aligned phonetic and orthographic transcriptions for read speech \(5\.1 hours\) and conversational speech \(40 hours\), respectively\. For each corpus, the data were partitioned at the speaker level into training, validation, and test sets using an 80/10/10 split\.

We evaluate the model on TIMIT and Buckeye, and additionally on unseen languages: Hebrew, German, and Dutch\. For Hebrew, we use a broadcast news dataset comprising 10 minutes of speech annotated at the phoneme level by professional linguists\[benshalom14\]\. For Dutch, we used the IFA Corpus\[VanSon01\-IFA\], a database of approximately five hours of hand\-segmented speech from eight speakers, covering a range of speaking styles\. For German, we evaluate on the full PHONDAT German corpus, which includes 201 speakers and 21,587 utterances\[tillmann1993theoretical\]\.

### 3\.2Architectures, model selection, and hyperparameters

The alignment representations are described in Sec\.[2\.1](https://arxiv.org/html/2606.10675#S2.SS1)\. For the alignment encoder, we considered three backbone architectures: VGG\[Simonyan15\-VGG\], a Transformer encoder\[Hsu21\-HUB\], and Conformer\[Gulati20\-CON\]\. Model selection was based on the F1\-score on the validation set, reflecting the fact that the encoder predicts boundary likelihoods, which might not be the same number of input word\.

Hyperparameters were tuned on the TIMIT validation set\. For VGG, we evaluated depths of 13, 16, and 19 layers, with input context windows of 23, 27, 31, 35, and 41 frames\. For the Transformer encoder, we considered 4, 8, and 16 layers, and context sizes of 50, 80, 100, and 200 frames\. For the Conformer, we explored configurations with 12, 16, and 20 blocks, 10, 12, or 16 attention heads per block, and context windows of 250, 300, and 350 frames\. The best\-performing configurations were VGG19 with a 31\-frame context window, a Transformer with 8 layers and a 100\-frame context, and a Conformer with 16 blocks, 12 attention heads per block, a convolutional kernel size of 7, and a 300\-frame context window\.

Table 1:Word boundary prediction performance of the alignment encoder on the TIMIT and Buckeye validation sets\. All metrics are computed at a frame resolution of 10 msec\.Table[1](https://arxiv.org/html/2606.10675#S3.T1)reports the performance of the evaluated architectures on the TIMIT and Buckeye validation sets\. The results correspond to the binary classification of word boundaries and are normalized by the relative number of input frames for each method\. Performance is measured in terms of accuracy, precision, recall, and F1\-score\. Overall, the Conformer and VGG architectures achieved the strongest results across datasets, whereas the Transformer exhibited the lowest performance\. These findings suggest that, although attention mechanisms provide a global contextual representation, the localized feature extraction enabled by convolutional components is more effective for the boundary detection task\.

As the DP procedure includes a non\-differentiable operation, the encoder and decoder could not be optimized jointly and were therefore trained separately\. To alleviate this limitation, we fine\-tuned the encoder for an additional 10 epochs, evaluating the decoder on the validation set at each epoch and applying early stopping based on validation performance\. The decoder was then fine\-tuned in a subsequent stage\. This training procedure was applied consistently across all architectures described above\. We evaluated alignment accuracy using the fine\-tuned decoder for each model across multiple tolerance thresholds, consistent with the evaluation method in Rousso et al\.\[Rousso24\-FACOMP\]\. As shown in Table[2](https://arxiv.org/html/2606.10675#S3.T2), both the Conformer and VGG architectures outperformed the Transformer, with comparable performance between them\. However, the Conformer achieved superior results on several key metrics\.

Table 2:Word alignment accuracy of encoder and decoder on the TIMIT and Buckeye validation set\. Thresholds are in msec\.Given its computational efficiency, robust performance across evaluation metrics, and suitability for real\-time deployment, the Conformer was selected as the final model\. Following this selection, the Conformer encoder was trained for 30 epochs, with early stopping based on the F1\-score computed on the validation set\.

### 3\.3Comparison with Other Models

Table[3](https://arxiv.org/html/2606.10675#S3.T3)compares our approach, denotedMWA, with established alignment systems on the TIMIT and Buckeye test sets\. We evaluate against MMS CTC\-based alignment\[pratap2024scaling\], MFA\[McAuliffe17\-MFA\], WhisperX\[Bain23\-WhisperX\], as reported in\[Rousso24\-FACOMP\], and Nvidia\-Canary\-1B\[nvidia\-canary\]\.

Our model consistently outperforms all baselines across datasets and evaluation thresholds\. Notably, it surpasses MFA, a widely adopted state\-of\-the\-art forced alignment system, as well as MMS, which serves as one of the alignment representations used in our framework\. These results demonstrate that the proposed method not only improves upon existing alignment pipelines but also yields gains over the underlying representation model on which it partially builds\.

Table 3:Word alignment accuracy of our model, MMS, MFA and WhisperX, on the TIMIT and Buckeye test set\.
### 3\.4Unseen Languages

Our alignment model was trained exclusively on English\. However, since our model is based on MMS, which was pretrained on more than 1,100 languages, and on UnSupSeg, which is language\-independent, we can assess the performance of the proposed system on unseen languages\. Specifically, we evaluate the model's performance on Hebrew, German, and Dutch without further training\. The corresponding results are reported in Table[4](https://arxiv.org/html/2606.10675#S3.T4)\.

We evaluate two variants of the model: one trained on TIMIT and another trained on Buckeye\. Across all tested languages, the TIMIT\-trained model consistently outperforms the Buckeye\-trained version, suggesting that cleaner read speech provides more transferable supervision for boundary detection than conversational speech\. For Hebrew, where MFA is unavailable, we compare only against MMS\-based alignment\. Our model outperforms MMS under stricter tolerance thresholds, while MMS performs better at more relaxed tolerances\. Performance on Dutch is overall lower\. On the German PHONDAT corpus\[tillmann1993theoretical\], the TIMIT\-trained model again surpasses the Buckeye\-trained model\. Its performance is comparable to MFA and exceeds it at certain tolerance thresholds\. Overall, these findings demonstrate that the proposed method generalizes well across languages, despite being trained exclusively on English data\.

Table 4:Word alignment accuracy of our model against MMS\[pratap2024scaling\]and MFA\[McAuliffe17\-MFA\]on the Hebrew, German \- PHONDAT, and Dutch \- IFA Corpus datasets

## 4Discussion

We proposed a method for accurate word alignment based on the MMS and an accurate self\-supervised phoneme boundary representation \(UnSupSeg\)\. It does not rely on phonemes and therefore eliminates the need for G2P conversions\. It proposes a potential replacement for MFA that is based on an HMM\-GMM constraint model with G2P\. We demonstrate the effectiveness of our approach on hand\-annotated datasets in American English, Hebrew, Dutch, and German, achieving competitive and often superior performance to MFA\. Since our framework relies on robust multilingual representations such as MMS, it has strong potential to scale to the full set of languages supported by MMS, enabling accurate and resource\-efficient word alignment beyond English\.

## 5Acknowledgments

This work was supported by NSF DRL Grant No\. 2219843 and BSF Grant No\. 2022618\. We also thank Rob van Son for his guidance and support with the IFA Corpus\.

## 6Generative AI Use Disclosure

Generative AI tools were used solely for language editing and manuscript polishing\. They did not contribute to the scientific content, analysis, or core writing of the paper, and no AI system is listed as a co\-author\.

## References