VTaMo: Video-Text Alignment Model for Sign Language Translation

# VTaMo: Video-Text Alignment Model for Sign Language Translation
Source: [https://arxiv.org/html/2607.09126](https://arxiv.org/html/2607.09126)
11institutetext:New York University Abu Dhabi, UAE22institutetext:ChatSign Technology
22email:jh10472@nyu\.edu, yf23@nyu\.edu###### Abstract

Sign language translation \(SLT\) converts continuous sign videos into spoken language text\. Gloss\-free approaches leverage pre\-trained visual encoders and language models but rely on implicit cross\-modal alignment from translation supervision alone\. We present VTaMo, a framework that introduces explicit multi\-granularity alignment at three levels: \(1\) local alignment via entropy\-regularized optimal transport with a learnable null token for fine\-grained frame\-to\-token correspondences; \(2\) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover’s Distance; and \(3\) position\-aligned contrastive learning for discriminative token\-level representations\. Experiments on Phoenix\-2014T, CSL\-Daily, How2Sign, and OpenASL demonstrate consistent state\-of\-the\-art performance, with ablations confirming the complementary contributions of each component\. Code is available at[https://github\.com/junyi2005/vtamo](https://github.com/junyi2005/vtamo)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.09126v1/x1.png)Figure 1:Motivation and performance of VTaMo\.\(a\)Prior gloss\-free sign language translation methods typically rely on*implicit*cross\-modal alignment learned inside the decoder attention, which can yield diffuse or mismatched cross\-attention and translation errors\.\(b\)VTaMo introduces*explicit*multi\-granularity vision–text alignment—including local OT\-based token\-to\-frame matching, global distribution alignment, and position\-aligned contrastive learning—to sharpen correspondences and improve decoding\.\(c\)VTaMo achieves state\-of\-the\-art sign language translation quality across four benchmarks \(Phoenix\-2014T, CSL\-Daily, How2Sign, and OpenASL\), compared against SpaMo\[hwang2025efficient\], Uni\-Sign\[li2025unisign\], SHuBERT\[gueuwou2025shubert\], and SSVP\-SLT\[rust2024towards\]\. Video frames in \(a\) and \(b\) are from the How2Sign dataset\[duarte2021how2sign\]\.Sign language translation \(SLT\) aims to convert continuous sign language videos into spoken language sentences, enabling more accessible communication between Deaf and hard\-of\-hearing signers and hearing communities\[camgoz2018neural,zhou2021improving\]\. Recent gloss\-free systems decode text directly from visual features using large pre\-trained sequence\-to\-sequence language models\[wong2024sign2gpt,gong2024llms,chen2024factorized\], avoiding the costly gloss annotation required by gloss\-based pipelines\. However, most gloss\-free benchmarks provide only natural language sentences as supervision, without gloss annotations or temporal segmentation\[duarte2021how2sign,shi2022open\]\. The core difficulty is therefore twofold: the semantic gap between vision and text, and the unknown correspondence between the temporal order of visual signs and the token order of the target text\.

Sign languages often do not follow the word order of the corresponding spoken language: a signer may express key content words first and add grammatical relations later, so the temporal progression of gestures can differ from the target word order\. As a result, visual features extracted along the video timeline are often misaligned with the text tokens that an autoregressive decoder is trained to predict\. Left unresolved, the decoder must simultaneously learn translation and implicitly discover a latent cross\-modal permutation, which increases optimization difficulty and degrades accuracy, especially on large\-scale benchmarks such as How2Sign and OpenASL\. Fig\.[1](https://arxiv.org/html/2607.09126#S1.F1)illustrates this gap: when alignment is left implicit, the cross\-attention map can be noisy and the decoder may produce incorrect word ordering, whereas our approach makes alignment explicit\.

Existing methods largely treat the visual stream as an ordered sequence and rely on decoder attention to bridge the modality gap\. ShuBERT\[gueuwou2025shubert\]and Uni\-Sign\[li2025unisign\]strengthen representations through large\-scale pretraining but do not explicitly model frame\-to\-token correspondence within a sentence, while SpaMo\[hwang2025efficient\]uses batch\-level contrastive learning yet leaves fine\-grained, sentence\-internal alignment unresolved\. These limitations motivate a model that learns correspondence between visual segments and text tokens and uses it to present the decoder with a semantically ordered visual sequence\.

We proposeVTaMo, a vision–text alignment model for gloss\-free SLT that explicitly aligns and reorders visual features before text generation\. Because many spoken words such as articles and prepositions have no sign\-level counterpart, we align against and decode a content\-word*pseudo\-gloss*of the target rather than the raw sentence\. Given a sign video, we build token\-level pseudo\-gloss embeddings from a frozen language model embedding layer and temporal visual embeddings from a video encoder, then estimate a soft alignment between visual positions and pseudo\-gloss tokens by solving an entropy\-regularized optimal transport problem\[cuturi2013sinkhorn\]with a learnable null token\. The null token absorbs transitional gestures and co\-articulation that correspond to no explicit word, stabilizing alignment on continuous signing\. Using the resulting transport plan, we reorder the visual features into a sequence that follows the target token order and feed it to the language model decoder\. At inference the target token order is unknown, so the visual features are not reordered; the decoder generates pseudo\-gloss directly from the signing\-order features, and a lightweight text\-only recovery model restores spoken word order and the omitted function words\.

VTaMo is trained end to end with three complementary alignment objectives beyond the standard translation loss\. First, a*local*optimal transport loss encourages fine\-grained correspondence between temporal visual segments and text tokens while allowing unaligned frames to map to the null token\. Second, a*global*loss calibrates the sentence\-level geometry of the visual and textual embedding spaces through a learnable orthogonal transformation, supported by a memory queue that diversifies the sentence pairs used for alignment\. Third, a*position\-aligned contrastive*loss\[oord2018infonce\]sharpens token\-level discriminability between reordered visual features and their text token embeddings without altering the language model embedding space\. Together, these losses make the reordered visual features semantically coherent, simplifying decoder learning and improving robustness across datasets\.

We evaluate on four widely used benchmarks spanning three languages and diverse data regimes: Phoenix\-2014T \(German\), CSL\-Daily \(Chinese\), How2Sign and OpenASL \(English\)\. Across all benchmarks, VTaMo achieves state\-of\-the\-art performance under the gloss\-free setting and yields consistent gains over strong baselines that decode without explicit alignment\. Ablation studies verify that each alignment component contributes complementary improvements, and qualitative analysis of the learned transport plans reveals interpretable correspondences between signing segments and target tokens\.

## 2Related Work

### 2\.1Gloss\-free SLT, Alignment, and Large\-Scale Pretraining

Gloss\-free sign language translation \(SLT\) translates continuous sign videos into spoken\-language sentences without gloss supervision, but it typically underperforms gloss\-based pipelines due to the absence of explicit intermediate structure\[camgoz2020sign,zhou2021improving,zhou2021spatial,yin2021simul,chen2022simple,chen2022twostream,zhang2023sltunet,jing2024vk\]\. Prior work improves gloss\-free SLT by strengthening temporal modeling\[li2020tspnet\], introducing cross\-modal alignment objectives\[zhao2021conditional,lin2023gloss,fu2023token,jiao2024vap\], incorporating large language models\[wong2024sign2gpt,gong2024llms,chen2024factorized\], scaling training data\[uthus2024youtube,rust2024towards\], and large\-scale sign pretraining such as ShuBERT\[gueuwou2025shubert\]and Uni\-Sign\[li2025unisign\]; SpaMo\[hwang2025efficient\]further emphasizes the non\-trivial sign–text correspondence via contrastive objectives\. These approaches—including VAP\[jiao2024vap\], which applies text\-derived constraints only as visual pre\-training—keep sentence\-internal correspondence implicit and rely on coarse decoder attention to resolve word order\. Our method instead estimates token\-level alignment and reorders visual features before decoding via Sinkhorn OT with a learnable null assignment and complementary local and global objectives\.

### 2\.2Cross\-Modal and Video–Text Alignment

Our objectives build on general cross\-modal alignment tools but adapt them to the structure of sign language\. Entropy\-regularized optimal transport with the Sinkhorn algorithm\[cuturi2013sinkhorn\]is a standard device for matching features across modalities, e\.g\., aligning visual features with text prompts in vision–language models\[chen2023plot\]; learnable orthogonal transformations relate two embedding spaces without distorting their geometry, as in cross\-lingual word\-embedding alignment\[lample2018word\]; and contrastive video–text frameworks such as CLIP4Clip\[luo2022clip4clip\], VideoCLIP\[xu2021videoclip\], and X\-CLIP\[ma2022xclip\]learn joint embeddings for retrieval\. Closer in spirit, procedure\-learning and step\-grounding methods align video frames with an ordered sequence of procedure steps, e\.g\., grounding instructional\-article steps via narrations\[mavroudi2023learning\]and optimal\-transport\-guided procedure learning\[chowdhury2024opel\]\. These methods, however, target global or clip\-level correspondence, or assume a largely fixed, monotonic step order\. Sign language translation differs: the mapping is*non\-monotonic*at the lexical level and*partial*, since many frames are transitional and words have no manual counterpart\. VTaMo addresses both by coupling OT with a null assignment, sentence\-level orthogonal calibration, and position\-aligned contrastive learning to yield a reordered visual sequence\.

## 3Method

![Refer to caption](https://arxiv.org/html/2607.09126v1/x2.png)Figure 2:VTaMo pipeline\.A sign\-language video is encoded into visual tokens using a frozen CLIP\-ViT backbone followed by a lightweight temporal encoder and fusion projection \(A,B\)\. Given the ground\-truth text embeddings \(D\), VTaMo performslocal alignmentwith an entropy\-regularized OT solver \(Sinkhorn\) to obtain soft token–frame matches, andglobal alignmentby mapping sign features into the text space with an orthogonal transform and a memory queue \(C1,C2\)\. The resulting correspondence is used forwindow reordering\(C3\) andposition\-aligned contrastive learning\(C4\)\. A LoRA\-adapted Flan\-T5 decoder generates the translation \(E\)\. Video frames are from the Phoenix\-2014T dataset\[camgoz2018neural\]\.We presentVTaMo, a framework for gloss\-free sign language translation that explicitly aligns and reorders visual features before text generation\. As illustrated in Fig\.[2](https://arxiv.org/html/2607.09126#S3.F2), VTaMo introduces three complementary alignment objectives: \(1\)*local alignment*via entropy\-regularized optimal transport for fine\-grained frame\-to\-token correspondences; \(2\)*global alignment*via a learnable orthogonal transformation to calibrate the embedding spaces at the sentence level; and \(3\)*position\-aligned contrastive learning*for discriminative representations\.

### 3\.1Problem Formulation

Given a sign language video, we extract frame\-level features𝐕=\{𝐯1,…,𝐯L\}\\mathbf\{V\}=\\\{\\mathbf\{v\}\_\{1\},\\dots,\\mathbf\{v\}\_\{L\}\\\}using a pre\-trained CLIP\-ViT encoder, where𝐯i∈ℝdv\\mathbf\{v\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{v\}\}\. The goal is to produce the spoken language sentence corresponding to the video\.

Spoken sentences contain many function words that have no direct sign\-level realization, which makes a strict frame\-to\-word alignment ill\-posed\. We therefore align against, and decode, a*pseudo\-gloss*sequence𝐘=\{y1,…,yU\}\\mathbf\{Y\}=\\\{y\_\{1\},\\dots,y\_\{U\}\\\}ofUUtokens derived from the target sentence by a fixed offline filter\. Concretely, we apply a frozen spaCy part\-of\-speech tagger that retains sign\-relevant content tokens \(NOUN,VERB,ADJ,ADV,NUM,PRON,PROPN\) and removes tokens that usually lack a direct sign counterpart \(DET,ADP,AUX,PART,PUNCT,CCONJ\); the same filtering rule is applied across all datasets using language\-specific spaCy models\. Both the text tokens used for alignment and the decoder targets are pseudo\-gloss, and a lightweight text\-only recovery model restores fluent sentences at inference\.

The visual features are projected toℝdh\\mathbb\{R\}^\{d\_\{h\}\}via a linear layer, temporally downsampled by an attention\-based encoder to produce𝐇=\{𝐡1,…,𝐡M\}\\mathbf\{H\}=\\\{\\mathbf\{h\}\_\{1\},\\dots,\\mathbf\{h\}\_\{M\}\\\}withM≪LM\\ll L, and mapped to the language model dimensiondtd\_\{t\}by a fusion projector\. A LoRA\-adapted Flan\-T5 model then autoregressively generates the pseudo\-gloss sequence\. The central challenge is that sign language does not follow the word order of spoken language, and the temporal granularity of visual frames far exceeds that of linguistic tokens\.

### 3\.2Attention\-Based Temporal Encoding

We use an attention\-based temporal convolution module withNℓN\_\{\\ell\}cascaded layers, each performing2×2\\timesdownsampling via learnable attention\-weighted aggregation over local windows of sizeWW\. For each output positionii, a local window𝒲i\\mathcal\{W\}\_\{i\}centered at position2​i2iis extracted, and a query is formed from the mean\-pooled local context plus a learnable positional embedding𝐩\\mathbf\{p\}:

𝐪i=1\|𝒲i\|​∑j∈𝒲i𝐱j\+𝐩\.\\mathbf\{q\}\_\{i\}=\\frac\{1\}\{\|\\mathcal\{W\}\_\{i\}\|\}\\sum\_\{j\\in\\mathcal\{W\}\_\{i\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{p\}\.\(1\)Multi\-head attention with relative position bias then aggregates the local context:

αi,j=softmax​\(𝐪i⊤​𝐤jdh/Nhead\+rj−ci\),𝐨i=∑j∈𝒲iαi,j​𝐯j,\\alpha\_\{i,j\}=\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{q\}\_\{i\}^\{\\top\}\\mathbf\{k\}\_\{j\}\}\{\\sqrt\{d\_\{h\}/N\_\{\\text\{head\}\}\}\}\+r\_\{j\-c\_\{i\}\}\\right\),\\qquad\\mathbf\{o\}\_\{i\}=\\sum\_\{j\\in\\mathcal\{W\}\_\{i\}\}\\alpha\_\{i,j\}\\,\\mathbf\{v\}\_\{j\},\(2\)wherecic\_\{i\}is the window center,rj−cir\_\{j\-c\_\{i\}\}is a learnable relative position bias, andNheadN\_\{\\text\{head\}\}is the number of attention heads\. This ensures gradient flow to all frames, yielding more expressive temporal representations than max\-pooling\.

### 3\.3Local Alignment via Optimal Transport

Local alignment is the core component of VTaMo, establishing fine\-grained correspondences between temporal visual segments and text tokens, formulated as an entropy\-regularized optimal transport problem\. Let𝐒=\{𝐬1,…,𝐬M\}∈ℝM×dt\\mathbf\{S\}=\\\{\\mathbf\{s\}\_\{1\},\\dots,\\mathbf\{s\}\_\{M\}\\\}\\in\\mathbb\{R\}^\{M\\times d\_\{t\}\}denote the visual features after fusion projection, and𝐄=\{𝐞1,…,𝐞U\}∈ℝU×dt\\mathbf\{E\}=\\\{\\mathbf\{e\}\_\{1\},\\dots,\\mathbf\{e\}\_\{U\}\\\}\\in\\mathbb\{R\}^\{U\\times d\_\{t\}\}the pseudo\-gloss token embeddings from the frozen encoder embedding layer\. Since𝐒\\mathbf\{S\}is optimized against the fixed text embeddings𝐄\\mathbf\{E\}under the alignment and translation losses, their cosine similarity reflects sign\-level semantic correspondence, making it a meaningful transport cost\. We introduce a single learnable null token𝐞∅∈ℝdt\\mathbf\{e\}\_\{\\varnothing\}\\in\\mathbb\{R\}^\{d\_\{t\}\}, prepended at position0of the text sequence, forming an augmented sequence𝐄~∈ℝK×dt\\tilde\{\\mathbf\{E\}\}\\in\\mathbb\{R\}^\{K\\times d\_\{t\}\}withK=U\+1K=U\+1\. The assignment is deliberately not one\-to\-one: frames that carry no explicit sign\-language semantics—such as background, transition, or reset frames—are all allowed to align to this same null token, so the model is never forced to map every frame onto a content token\. The pairwise cost matrix uses cosine distance, with a learnable biasb∅b\_\{\\varnothing\}controlling null affinity:

Cm,k=\{1−𝐬¯m⊤​𝐞¯k−b∅,if​k=0​\(null\),1−𝐬¯m⊤​𝐞¯k,otherwise,C\_\{m,k\}=\\begin\{cases\}1\-\\bar\{\\mathbf\{s\}\}\_\{m\}^\{\\top\}\\bar\{\\mathbf\{e\}\}\_\{k\}\-b\_\{\\varnothing\},&\\text\{if \}k=0\\text\{ \(null\)\},\\\\ 1\-\\bar\{\\mathbf\{s\}\}\_\{m\}^\{\\top\}\\bar\{\\mathbf\{e\}\}\_\{k\},&\\text\{otherwise\},\\end\{cases\}\(3\)where𝐬¯m\\bar\{\\mathbf\{s\}\}\_\{m\}and𝐞¯k\\bar\{\\mathbf\{e\}\}\_\{k\}areℓ2\\ell\_\{2\}\-normalized vectors\. We solve the entropy\-regularized OT problem via the Sinkhorn algorithm in log\-domain:

𝐀∗=arg⁡min𝐀∈Π​\(𝐚,𝐛\)⁡⟨𝐀,𝐂⟩−ε​H​\(𝐀\),\\mathbf\{A\}^\{\*\}=\\arg\\min\_\{\\mathbf\{A\}\\in\\Pi\(\\mathbf\{a\},\\mathbf\{b\}\)\}\\;\\langle\\mathbf\{A\},\\mathbf\{C\}\\rangle\-\\varepsilon\\,H\(\\mathbf\{A\}\),\(4\)whereΠ​\(𝐚,𝐛\)\\Pi\(\\mathbf\{a\},\\mathbf\{b\}\)is the set of transport plans with uniform marginals over valid frame and token positions,H​\(𝐀\)=−∑m,kAm,k​log⁡Am,kH\(\\mathbf\{A\}\)=\-\\sum\_\{m,k\}A\_\{m,k\}\\log A\_\{m,k\}is the entropy term, andε\\varepsiloncontrols the sharpness of assignments\. For numerical stability we iterate the dual potentials in the log domain forNiterN\_\{\\text\{iter\}\}steps,

log⁡um\\displaystyle\\log u\_\{m\}←log⁡am−log​∑kexp⁡\(−Cm,k/ε\+log⁡vk\),\\displaystyle\\leftarrow\\log a\_\{m\}\-\\log\\textstyle\\sum\_\{k\}\\exp\\\!\\left\(\-C\_\{m,k\}/\\varepsilon\+\\log v\_\{k\}\\right\),log⁡vk\\displaystyle\\log v\_\{k\}←log⁡bk−log​∑mexp⁡\(−Cm,k/ε\+log⁡um\),\\displaystyle\\leftarrow\\log b\_\{k\}\-\\log\\textstyle\\sum\_\{m\}\\exp\\\!\\left\(\-C\_\{m,k\}/\\varepsilon\+\\log u\_\{m\}\\right\),\(5\)and recover the plan asAm,k=exp⁡\(log⁡um−Cm,k/ε\+log⁡vk\)A\_\{m,k\}=\\exp\\\!\\left\(\\log u\_\{m\}\-C\_\{m,k\}/\\varepsilon\+\\log v\_\{k\}\\right\)\. We adopt a multi\-phase annealing schedule that progressively decreasesε\\varepsilonfrom a high initial value to encourage exploration before converging to sharp, discrete assignments\.

The local alignment loss combines three terms\. The first is the transport costℒtrans=1B​∑i∑m,kAm,k\(i\)​Cm,k\(i\)\\mathcal\{L\}\_\{\\text\{trans\}\}=\\frac\{1\}\{B\}\\sum\_\{i\}\\sum\_\{m,k\}A\_\{m,k\}^\{\(i\)\}C\_\{m,k\}^\{\(i\)\}\. The second is a temporal variation regularization that penalizes adjacent frames switching tokens, computed on the row\-normalized plan𝐀^\\hat\{\\mathbf\{A\}\}restricted to real text tokens:

ℒtv=1\(M−1\)​\(K−1\)​∑m=1M−1∑k=1K−1\|A^m\+1,k−A^m,k\|\.\\mathcal\{L\}\_\{\\text\{tv\}\}=\\frac\{1\}\{\(M\-1\)\(K\-1\)\}\\sum\_\{m=1\}^\{M\-1\}\\sum\_\{k=1\}^\{K\-1\}\\big\|\\hat\{A\}\_\{m\+1,k\}\-\\hat\{A\}\_\{m,k\}\\big\|\.\(6\)The third is a null cost regularization that keeps the mean cost\-based null affinityp¯∅\\bar\{p\}\_\{\\varnothing\}—computed via a row\-wise softmax on the cost matrix independently of the transport plan—below a target ratioρtarget\\rho\_\{\\text\{target\}\}, preventing the learnable biasb∅b\_\{\\varnothing\}from making the null column uniformly cheaper than all real tokens:

ℒnull=\[p¯∅−ρtarget\]\+2,p¯∅=1M​∑m=1Mexp⁡\(−Cm,0/τ\)∑k=0K−1exp⁡\(−Cm,k/τ\)\.\\mathcal\{L\}\_\{\\text\{null\}\}=\\big\[\\bar\{p\}\_\{\\varnothing\}\-\\rho\_\{\\text\{target\}\}\\big\]\_\{\+\}^\{2\},\\qquad\\bar\{p\}\_\{\\varnothing\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\frac\{\\exp\(\-C\_\{m,0\}/\\tau\)\}\{\\sum\_\{k=0\}^\{K\-1\}\\exp\(\-C\_\{m,k\}/\\tau\)\}\.\(7\)The complete local objective is their weighted sum:

ℒlocal=βlocal​ℒtrans\+βtv​ℒtv\+βnull​ℒnull\.\\mathcal\{L\}\_\{\\text\{local\}\}=\\beta\_\{\\text\{local\}\}\\,\\mathcal\{L\}\_\{\\text\{trans\}\}\+\\beta\_\{\\text\{tv\}\}\\,\\mathcal\{L\}\_\{\\text\{tv\}\}\+\\beta\_\{\\text\{null\}\}\\,\\mathcal\{L\}\_\{\\text\{null\}\}\.\(8\)

### 3\.4Global Alignment via Orthogonal Transformation

Because the visual and textual encoders are pre\-trained on different modalities, paired sign and text sentence embeddings can suffer a coordinate \(orientation\) mismatch even when they are semantically equivalent\. We therefore apply a learnable*orthogonal transformation*—a rotation of the feature space that preserves vector lengths and angular relationships—to the visual sentence embeddings before comparing the two modalities at the sentence level\. We constrain the transformation to be orthogonal precisely because the goal is to correct orientation without reshaping the space: this preserves norms and angular structure for the cosine\-based transport cost and avoids the spurious low\-cost matches that an unconstrained projection could introduce\.

We aggregate frame\-level features into one sentence\-level vector per video and per sentence via attention pooling, using a mean query𝐪\\mathbf\{q\}over valid positions and the normalized weighted sum

𝐳=∑lαl​𝐱l‖∑lαl​𝐱l‖2,αl=softmaxl​\(𝐱l⊤​𝐪/dt\),\\mathbf\{z\}=\\frac\{\\sum\_\{l\}\\alpha\_\{l\}\\mathbf\{x\}\_\{l\}\}\{\\big\\\|\\sum\_\{l\}\\alpha\_\{l\}\\mathbf\{x\}\_\{l\}\\big\\\|\_\{2\}\},\\qquad\\alpha\_\{l\}=\\mathrm\{softmax\}\_\{l\}\\\!\\big\(\\mathbf\{x\}\_\{l\}^\{\\top\}\\mathbf\{q\}/\\sqrt\{d\_\{t\}\}\\big\),\(9\)yielding normalized sentence\-level vectors𝐳sign,𝐳text∈ℝdt\\mathbf\{z\}^\{\\text\{sign\}\},\\mathbf\{z\}^\{\\text\{text\}\}\\in\\mathbb\{R\}^\{d\_\{t\}\}\. A FIFO memory queue stores recent sentence\-level vector pairs so that the sentence\-level transport is computed over a sufficiently diverse batch rather than a few in\-batch pairs\. A learnable matrix𝐓∈ℝdt×dt\\mathbf\{T\}\\in\\mathbb\{R\}^\{d\_\{t\}\\times d\_\{t\}\}, initialized near identity, transforms the visual sentence vectors:

𝐳~nsign=𝐳nsign​𝐓‖𝐳nsign​𝐓‖2\.\\tilde\{\\mathbf\{z\}\}\_\{n\}^\{\\text\{sign\}\}=\\frac\{\\mathbf\{z\}\_\{n\}^\{\\text\{sign\}\}\\,\\mathbf\{T\}\}\{\\\|\\mathbf\{z\}\_\{n\}^\{\\text\{sign\}\}\\,\\mathbf\{T\}\\\|\_\{2\}\}\.\(10\)We then form the pairwise cosine costCn,n′g=1−\(𝐳~nsign\)⊤​𝐳n′textC^\{g\}\_\{n,n^\{\\prime\}\}=1\-\(\\tilde\{\\mathbf\{z\}\}\_\{n\}^\{\\text\{sign\}\}\)^\{\\top\}\\mathbf\{z\}\_\{n^\{\\prime\}\}^\{\\text\{text\}\}between the transformed visual vectors and the textual vectors and solve a Sinkhorn OT problem with uniform marginals to obtain a sentence\-level transport plan𝐏\\mathbf\{P\}; the resulting Earth Mover’s Distance loss is

ℒEMD=∑n,n′Pn,n′​\(1−\(𝐳~nsign\)⊤​𝐳n′text\)\.\\mathcal\{L\}\_\{\\text\{EMD\}\}=\\sum\_\{n,n^\{\\prime\}\}P\_\{n,n^\{\\prime\}\}\\,\(1\-\(\\tilde\{\\mathbf\{z\}\}\_\{n\}^\{\\text\{sign\}\}\)^\{\\top\}\\mathbf\{z\}\_\{n^\{\\prime\}\}^\{\\text\{text\}\}\)\.\(11\)An orthogonality constraintℒorth=‖𝐓⊤​𝐓−𝐈‖F2\\mathcal\{L\}\_\{\\text\{orth\}\}=\\\|\\mathbf\{T\}^\{\\top\}\\mathbf\{T\}\-\\mathbf\{I\}\\\|\_\{F\}^\{2\}keeps𝐓\\mathbf\{T\}close to a pure rotation and preserves pairwise distances\. The global loss is activated only after local alignment stabilizes, using a scheduled ramp\-up:

ℒglobal=λg​\(t\)⋅ℒEMD\+βorth​ℒorth\.\\mathcal\{L\}\_\{\\text\{global\}\}=\\lambda\_\{g\}\(t\)\\cdot\\mathcal\{L\}\_\{\\text\{EMD\}\}\+\\beta\_\{\\text\{orth\}\}\\,\\mathcal\{L\}\_\{\\text\{orth\}\}\.\(12\)

### 3\.5Window\-Based Feature Reordering

Once local alignment has produced a reliable transport plan𝐀\\mathbf\{A\}, we reorder the visual features to match the target token order\. This is necessary because the decoder’s autoregressive cross\-entropy loss expects the input to follow the target text order; otherwise the visual features stay in temporal order and the cross\-entropy gradient counteracts the alignment objectives\.

Concretely, we partition theMMvisual frames intoNw=U\+2N\_\{w\}=U\+2overlapping temporal windows, where the two extra windows absorb boundary effects; theww\-th window spans\[sw,ew\)\[s\_\{w\},e\_\{w\}\)withsw=⌊w​\(M−1\)/Nw\+0\.5⌋s\_\{w\}=\\lfloor w\\,\(M\-1\)/N\_\{w\}\+0\.5\\rfloorandew=⌊\(w\+1\)​\(M−1\)/Nw\+0\.5⌋\+1e\_\{w\}=\\lfloor\(w\+1\)\(M\-1\)/N\_\{w\}\+0\.5\\rfloor\+1\. For each window we accumulate the alignment mass𝝅w=∑m=swew−1𝐀m,:∈ℝK\\boldsymbol\{\\pi\}\_\{w\}=\\sum\_\{m=s\_\{w\}\}^\{e\_\{w\}\-1\}\\mathbf\{A\}\_\{m,:\}\\in\\mathbb\{R\}^\{K\}and assign it to the real text token with the largest aggregated mass,ϕ​\(w\)=arg⁡maxu∈\{1,…,U\}⁡πw,u\\phi\(w\)=\\arg\\max\_\{u\\in\\\{1,\\dots,U\\\}\}\\pi\_\{w,u\}; windows dominated by the null token are excluded\. Collecting the windows assigned to each tokenuuintoΩu=\{w∣ϕ​\(w\)=u\}\\Omega\_\{u\}=\\\{w\\mid\\phi\(w\)=u\\\}and traversing the token order left to right yields the reordered sequence

𝐑=Concat​\(\{𝐒sw:ew−1∣w∈Ω1↑\},…,\{𝐒sw:ew−1∣w∈ΩU↑\}\)∈ℝU′×dt,\\mathbf\{R\}=\\mathrm\{Concat\}\\big\(\\\{\\mathbf\{S\}\_\{s\_\{w\}:e\_\{w\}\-1\}\\mid w\\in\\Omega\_\{1\}^\{\\uparrow\}\\\},\\dots,\\\{\\mathbf\{S\}\_\{s\_\{w\}:e\_\{w\}\-1\}\\mid w\\in\\Omega\_\{U\}^\{\\uparrow\}\\\}\\big\)\\in\\mathbb\{R\}^\{U^\{\\prime\}\\times d\_\{t\}\},\(13\)whereΩu↑\\Omega\_\{u\}^\{\\uparrow\}orders the windows inΩu\\Omega\_\{u\}by their original temporal index\. This preserves local temporal continuity, and the reordered sequence is fed to the decoder\. For the contrastive objective, frames assigned to each tokenuuare mean\-pooled to produce a single token\-level representation𝐫u\\mathbf\{r\}\_\{u\}, yielding𝐑=\{𝐫1,…,𝐫U\}\\mathbf\{R\}=\\\{\\mathbf\{r\}\_\{1\},\\dots,\\mathbf\{r\}\_\{U\}\\\}aligned one\-to\-one with𝐄\\mathbf\{E\}\.

### 3\.6Position\-Aligned Contrastive Learning

Because reordering is used only during training, the model must also function at inference when the target order is unknown\. We therefore impose a token\-level contrastive objective that binds each reordered visual feature to its text token embedding, so that even when tokens arrive out of order each is paired with its correct counterpart and the decoder can recover its lexical item; restoring fluent word order is then left to the recovery model\.

We apply learnable projection heads to both modalities:

𝐫^u=LN​\(𝐖2v​σ​\(𝐖1v​𝐫u\)\),𝐞^u=LN​\(𝐖2t​σ​\(𝐖1t​𝐞u\)\),\\hat\{\\mathbf\{r\}\}\_\{u\}=\\text\{LN\}\\\!\\big\(\\mathbf\{W\}\_\{2\}^\{\\text\{v\}\}\\,\\sigma\(\\mathbf\{W\}\_\{1\}^\{\\text\{v\}\}\\,\\mathbf\{r\}\_\{u\}\)\\big\),\\quad\\hat\{\\mathbf\{e\}\}\_\{u\}=\\text\{LN\}\\\!\\big\(\\mathbf\{W\}\_\{2\}^\{\\text\{t\}\}\\,\\sigma\(\\mathbf\{W\}\_\{1\}^\{\\text\{t\}\}\\,\\mathbf\{e\}\_\{u\}\)\\big\),\(14\)whereσ\\sigmais GELU activation andLNis layer normalization\. Afterℓ2\\ell\_\{2\}\-normalization, the InfoNCE loss is:

ℒcontra=−1\|𝒱\|​∑i∈𝒱log⁡exp⁡\(sim​\(𝐫^i,𝐞^i\)/τc\)∑j∈𝒱exp⁡\(sim​\(𝐫^i,𝐞^j\)/τc\),\\mathcal\{L\}\_\{\\text\{contra\}\}=\-\\frac\{1\}\{\|\\mathcal\{V\}\|\}\\sum\_\{i\\in\\mathcal\{V\}\}\\log\\frac\{\\exp\\\!\\big\(\\text\{sim\}\(\\hat\{\\mathbf\{r\}\}\_\{i\},\\hat\{\\mathbf\{e\}\}\_\{i\}\)/\\tau\_\{c\}\\big\)\}\{\\displaystyle\\sum\_\{j\\in\\mathcal\{V\}\}\\exp\\\!\\big\(\\text\{sim\}\(\\hat\{\\mathbf\{r\}\}\_\{i\},\\hat\{\\mathbf\{e\}\}\_\{j\}\)/\\tau\_\{c\}\\big\)\},\(15\)where𝒱\\mathcal\{V\}denotes valid token positions andτc\\tau\_\{c\}is a temperature hyperparameter\. Gradients flow through the visual branch but are blocked from the text branch to preserve the pre\-trained language model representations\.

##### Inference and pseudo\-gloss recovery\.

At test time the target order is unknown, so no reordering is applied: the decoder reads the visual tokens in their original temporal order and emits a pseudo\-gloss sequence in visual \(signing\) order rather than spoken\-language order\. The token\-level grounding above makes this feasible, since each visual token is decoded into its correct lexical item even when the tokens arrive out of order\. To recover a fluent translation, we pass this video\-order pseudo\-gloss through a lightweight recovery model that restores spoken word order and re\-inserts the function words dropped by the pseudo\-gloss filter \(Sec\.[3\.1](https://arxiv.org/html/2607.09126#S3.SS1)\)\. Crucially, this recovery model is trained purely on text and never sees sign videos: we take plain sentences, apply the same pseudo\-gloss extractor, permute the content words, and train the model to reconstruct the original sentence from this shuffled input\.

The overall training objective combines the translation loss with the three alignment losses:

ℒ=ℒT5\+λlocal​ℒlocal\+λg​\(t\)​ℒEMD\+βorth​ℒorth\+βcontra​ℒcontra,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{T5\}\}\+\\lambda\_\{\\text\{local\}\}\\,\\mathcal\{L\}\_\{\\text\{local\}\}\+\\lambda\_\{g\}\(t\)\\,\\mathcal\{L\}\_\{\\text\{EMD\}\}\+\\beta\_\{\\text\{orth\}\}\\,\\mathcal\{L\}\_\{\\text\{orth\}\}\+\\beta\_\{\\text\{contra\}\}\\,\\mathcal\{L\}\_\{\\text\{contra\}\},\(16\)whereℒT5\\mathcal\{L\}\_\{\\text\{T5\}\}is the cross\-entropy loss for autoregressive generation\. Training proceeds in two phases: in the first phase, only alignment losses are active while the language model is frozen, allowing the visual encoder to establish meaningful correspondences; in the second phase, the full objective is optimized jointly with the LoRA\-adapted language model\.

## 4Experiments

### 4\.1Datasets and Metrics

We evaluate our method on four publicly available sign language translation benchmarks that span different languages, vocabulary sizes, and recording conditions: Phoenix\-2014T\[camgoz2018neural\]\(German\), CSL\-Daily\[zhou2021improving\]\(Chinese\), How2Sign\[duarte2021how2sign\]and OpenASL\[shi2022open\]\(English\)\. We report BLEU\-nnscores\[papineni2002bleu\]forn∈\{1,2,3,4\}n\\in\\\{1,2,3,4\\\}and ROUGE\-L F1\[lin2004rouge\]as the primary evaluation metrics\. For English SLT datasets \(How2Sign and OpenASL\), we additionally report BLEURT\[sellam2020bleurt\]scores following established protocol\. For every benchmark, each model trains only on its official split without additional sign\-language data or task\-specific pre\-training\.

### 4\.2Implementation Details

##### Visual features\.

We extract frame\-level spatial features using a pre\-trained CLIP\-ViT\-Large model\[radford2021clip\]with S2\-Wrapper\[shi2024when\]for multi\-scale feature extraction at scales 1 and 2, yielding 2048\-dimensional feature vectors per frame\.

##### Language model\.

We employ Flan\-T5\-XL\[chung2024scaling\]as the language model backbone and adapt it using LoRA\[hu2022lora\]with rankr=16r=16, scaling factorα=32\\alpha=32, and dropout rate 0\.1\. The LoRA adapters are applied to the query and value projection matrices across all transformer layers\.

##### Architecture\.

The visual features are projected fromdv=2048d\_\{v\}=2048to an intermediate dimensiondh=768d\_\{h\}=768through a linear layer\. The attention\-based temporal encoder consists ofNℓ=2N\_\{\\ell\}=2cascaded layers with 4 attention heads and a window size of 8, yielding a4×4\\timestemporal downsampling\. The fusion projector maps the features fromdhd\_\{h\}to the T5 embedding dimensiondtd\_\{t\}\. The contrastive projection heads are two\-layer MLPs with GELU activation and layer normalization, projecting to 768 dimensions\.

##### Training\.

We train all models using AdamW\[loshchilov2019decoupled\]with a peak learning rate of6×10−46\\times 10^\{\-4\}, cosine learning rate schedule, and linear warm\-up over the first 2,000 steps\. The batch size is 6 with gradient accumulation over 2 steps, and gradient clipping is applied with a maximum norm of 1\.0\. All experiments were conducted on a single NVIDIA A100 GPU with mixed\-precision \(bf16\) training\.

##### Alignment hyperparameters\.

For the local alignment, the Sinkhorn algorithm runs forNiter=10N\_\{\\text\{iter\}\}=10iterations\. The epsilon annealing schedule proceeds fromεhigh=0\.12\\varepsilon\_\{\\text\{high\}\}=0\.12toεmid=0\.10\\varepsilon\_\{\\text\{mid\}\}=0\.10over the first 10 epochs and further toεlow=0\.03\\varepsilon\_\{\\text\{low\}\}=0\.03over the following 80 epochs\. The null ratio target isρtarget=0\.2\\rho\_\{\\text\{target\}\}=0\.2, and the loss weights areβlocal=2\.0\\beta\_\{\\text\{local\}\}=2\.0,βtv=0\.1\\beta\_\{\\text\{tv\}\}=0\.1,βnull=0\.1\\beta\_\{\\text\{null\}\}=0\.1, with the overall local\-alignment weightλlocal=1\.0\\lambda\_\{\\text\{local\}\}=1\.0\. For the global alignment, the EMD solver usesεg=0\.05\\varepsilon\_\{g\}=0\.05with 20 Sinkhorn iterations, the memory queue capacity is 256, and the weight ramps from 0 toλgmax=0\.1\\lambda\_\{g\}^\{\\text\{max\}\}=0\.1over 4,000 steps after warm\-up\. The orthogonality penalty coefficient isβorth=0\.05\\beta\_\{\\text\{orth\}\}=0\.05\. The contrastive learning temperature isτc=0\.1\\tau\_\{c\}=0\.1, and the contrastive loss weight isβcontra=1\.0\\beta\_\{\\text\{contra\}\}=1\.0\.

##### Training cost\.

On a single A100, end\-to\-end training takes 14 h on Phoenix\-2014T, 32 h on CSL\-Daily, and 50 h on How2Sign\. The optimal transport solvers add negligible overhead: the local Sinkhorn step takes 54 ms \(<<3\.2% of the per\-step time\), and the global OT over the256×256256\\times 256memory queue takes<<38 ms\.

### 4\.3Comparison with State\-of\-the\-Art

Results on Phoenix\-2014T and CSL\-Daily\.As shown in Table[1](https://arxiv.org/html/2607.09126#S4.T1), most prior methods rely on visual\-encoder fine\-tuning or gloss supervision, yet VTaMo reaches state\-of\-the\-art on both benchmarks without either\. On Phoenix\-2014T it attains 28\.86 BLEU\-4, surpassing the strongest gloss\-free method SpaMo\[hwang2025efficient\]by 4\.54 points and coming within 0\.09 of the best gloss\-based result \(TS\-SLT\[chen2022twostream\], 28\.95\)\. On CSL\-Daily it reaches 27\.16 BLEU\-4, a 6\.61\-point gain over SpaMo and 0\.80 over Uni\-Sign, showing that explicit multi\-granularity alignment especially helps on topically diverse data\.

Table 1:Performance comparison on the Phoenix\-2014T\[camgoz2018neural\]and CSL\-Daily datasets\[zhou2021improving\]; all numbers are reported on the official test sets\. “Vis\.Ft\.” denotes visual fine\-tuning on sign language datasets\. Bold marks the best result\.Results on How2Sign\.How2Sign is more challenging, with open\-domain topics and a much larger vocabulary\. As shown in Table[3](https://arxiv.org/html/2607.09126#S4.T3), VTaMo achieves the best BLEU\-1, BLEU\-4, and BLEURT on the official test set \(BLEU\-4 18\.47, BLEURT 50\.26\) using only RGB and no large\-scale pre\-training—an 8\.36 BLEU\-4 gain over SpaMo\[hwang2025efficient\]—and ranks second on ROUGE\-L behind SSVP\-SLT\[rust2024towards\]\(38\.40 vs\. 37\.14\)\. It also surpasses SHuBERT\[gueuwou2025shubert\]\(16\.2 / 49\.9\) and Uni\-Sign\[li2025unisign\]\(14\.9 / 49\.4\), both using pose\+RGB and large\-scale pre\-training\.

Results on OpenASL\.OpenASL is the hardest setting, with a vocabulary exceeding 10,000 words\. As shown in Table[3](https://arxiv.org/html/2607.09126#S4.T3), VTaMo attains 25\.94 BLEU\-4 and 62\.48 BLEURT on the official test set, outperforming Uni\-Sign\[li2025unisign\]and SHuBERT\[gueuwou2025shubert\]by 2\.80 and 2\.74 BLEU\-4 despite their pose\+RGB inputs and large\-scale pre\-training\. Gains over SpaMo grow on the larger\-vocabulary How2Sign and OpenASL, consistent with alignment helping more as complexity increases\.

Table 2:Results on How2Sign\[duarte2021how2sign\]\. Bold marks the best result\.
Table 3:Results on OpenASL\[shi2022open\]\. Bold marks the best result\.

### 4\.4Ablation Study

All ablations are conducted on the Phoenix\-2014T test set\.

##### Alignment components\.

Removing each alignment loss in turn \(Table[5](https://arxiv.org/html/2607.09126#S4.T5)\) shows that all three are necessary and non\-substitutable\. The position\-aligned contrastive lossℒcontra\\mathcal\{L\}\_\{\\text\{contra\}\}matters most \(−\-6\.30 BLEU\-4\), providing token\-level discriminative grounding; the local OT lossℒlocal\\mathcal\{L\}\_\{\\text\{local\}\}is second \(−\-5\.09\), as it underpins the window\-based reordering that feeds ordered inputs to the decoder; and the global lossℒEMD\\mathcal\{L\}\_\{\\text\{EMD\}\}\(−\-2\.65\) adds complementary embedding\-space calibration\.

##### Language\-model backbone\.

To test whether the gains stem from a favorable language model rather than our alignment design, we vary the backbone following SpaMo’s protocol \(Table[5](https://arxiv.org/html/2607.09126#S4.T5)\)\. The ranking matches SpaMo—Flan\-T5\-XL is best, mT0\-XL is close behind, and Llama\-2 underperforms despite its larger size—confirming that scaling the LM alone does not help under the limited\-data SLT regime\. Critically, under*every*fixed backbone our method beats SpaMo by\+2\.49\+2\.49to\+4\.54\+4\.54BLEU\-4, so the improvement comes from the alignment block\.

##### Local alignment quality\.

Three transport\-plan metrics \(Table[7](https://arxiv.org/html/2607.09126#S4.T7)\)—peak assignment probability, normalized entropy, and segment change rate—characterize alignment sharpness\. The full model is the sharpest and most confident; removing epsilon annealing yields diffuse assignments, removing the TV loss fragments them, and removing the null token forces transitional frames onto real tokens\.

##### Global alignment components\.

Table[7](https://arxiv.org/html/2607.09126#S4.T7)isolates the global module and reports the epoch of peak development BLEU\-4 as an efficiency proxy\. The orthogonality constraint prevents distance distortion, scheduled activation avoids premature gradients before local plans stabilize, and the memory queue is the largest contributor to faster convergence \(36 vs\. 68 epochs without it\)\.

Table 4:Alignment\-component ablation on the Phoenix\-2014T\[camgoz2018neural\]test set\. Each row removes one loss from the full model\.
Table 5:Language\-model backbone ablation on Phoenix\-2014T\[camgoz2018neural\]\(BLEU\-4\), following SpaMo\[hwang2025efficient\]’s protocol\.Δ\\Delta: our gain over SpaMo at the same backbone\.

Table 6:Alignment quality metrics on the Phoenix\-2014T\[camgoz2018neural\]development set\.
Table 7:Ablation of global alignment components on Phoenix\-2014T\[camgoz2018neural\]\. Ep\.↓\\downarrow: best epoch\.

### 4\.5Noise Sensitivity Analysis

Since VTaMo operates on RGB input without visual\-encoder fine\-tuning, we assess its robustness to appearance shifts\. Using Stable Diffusion\[rombach2022ldm\]with ControlNet\[zhang2023controlnet\]for pose\-conditioned synthesis, we build four controlled conditions on the entire How2Sign test set while preserving each signer’s hand and body kinematics \(Fig\.[3](https://arxiv.org/html/2607.09126#S4.F3)\): two replace the background \(BG\-1, outdoor park; BG\-2, indoor kitchen\) and two alter signer appearance \(AP\-1, full body shape and identity transfer to a synthetic signer; AP\-2, clothing\)\. All appearance modifications are entirely synthetic—the transferred identity in AP\-1 does not represent any real individual—and are used solely for robustness evaluation\. Table[8](https://arxiv.org/html/2607.09126#S4.T8)reports single\-run performance across all conditions: background replacement causes only marginal degradation \(≤\\leq0\.42 BLEU\-4\) and appearance edits even smaller drops \(≤\\leq0\.33\), showing that VTaMo attends to gesture semantics rather than scene context or signer identity\. Additional examples are in the supplementary material\.

![Refer to caption](https://arxiv.org/html/2607.09126v1/x3.png)Figure 3:Representative augmented samples used in the robustness analysis on How2Sign\[duarte2021how2sign\]\. \(a\) and \(c\) are background\-replacement conditions generated via text\-prompt editing; \(b\) is a full body shape and identity transfer to a synthetic signer generated via text prompt\. Hand and body kinematics are preserved via ControlNet pose conditioning\. The synthetic signer in \(b\) does not represent any real individual\. All modifications are used solely for research evaluation\.Table 8:Noise sensitivity analysis on the entire How2Sign\[duarte2021how2sign\]test set\. BG: background replacement, AP: appearance modification\. All modifications generated with Stable Diffusion and ControlNet pose conditioning\.
### 4\.6Qualitative Analysis

![Refer to caption](https://arxiv.org/html/2607.09126v1/x4.png)Figure 4:Qualitative analysis on How2Sign\[duarte2021how2sign\]test samples\. \(a\) OT transport plans from VTaMo’s Sinkhorn solver: rows are visual frames and columns are pseudo\-gloss tokens, with darker cells indicating higher transport mass; the block\-diagonal structure reflects temporally coherent frame\-to\-token correspondences while the null token absorbs transitional frames\. \(b\) t\-SNE of visual features, colored by semantic category, forming distinct clusters\. \(c\) Explicit frame\-to\-pseudo\-gloss alignment on a test sentence: each colored interval marks the contiguous video segment assigned to a predicted content token, with representative frames shown above the timeline\.Fig\.[4](https://arxiv.org/html/2607.09126#S4.F4)\(a\) visualizes the learned transport plans on How2Sign: they exhibit clear block\-diagonal structure—contiguous frame groups aligning to individual tokens—with the null token absorbing transitional frames, and the ordering is largely monotonic with local reorderings where signing order diverges from the target\. Fig\.[4](https://arxiv.org/html/2607.09126#S4.F4)\(c\) marks the video segment assigned to each predicted content token along the timeline, and Fig\.[4](https://arxiv.org/html/2607.09126#S4.F4)\(b\) shows a t\-SNE with distinct semantic clusters, indicating alignment that is both temporally coherent at the frame level and semantically organized at the sentence level\.

## 5Conclusion

VTaMo achieves state\-of\-the\-art gloss\-free SLT through explicit multi\-granularity cross\-modal alignment: local OT with a null token, global orthogonal calibration, and position\-aligned contrastive learning\. Experiments on four benchmarks confirm consistent gains, and ablations verify each component’s contribution, showing that explicit alignment is an effective, transferable direction for gloss\-free SLT with modest overhead\.

## 6Limitations

Our evaluation targets offline accuracy rather than systems\-level concerns such as real\-time or on\-device deployment; the spoken\-language order is also recovered by a text\-only model rather than observed directly\.

## Acknowledgements

This work was partially supported by ChatSign Technology, Ltd\.; and the NYUAD Center for AI and Robotics \(CAIR\), funded by Tamkeen under the NYUAD Research Institute Award CG010\. The generous computational support was provided by the HPC resources at NYU Abu Dhabi and NYU New York\.

## References

## Appendix

This appendix provides expanded details and additional results complementing the main paper\. Section[S\.1](https://arxiv.org/html/2607.09126#S1a)presents the full formulations for each component of VTaMo, including the attention\-based temporal encoder, the complete local alignment derivation \(null\-token augmentation, cost matrix, Sinkhorn iterations, epsilon annealing, and loss terms\), the global alignment module \(attention pooling, memory queue, orthogonal mapping, and EMD loss\), the window\-based reordering procedure, and the position\-aligned contrastive objective\. Section[S\.2](https://arxiv.org/html/2607.09126#S2a)contains supplementary experimental results: a comparison of temporal\-pooling strategies \(attention pooling vs\. max\-pooling\), a comparison against generic video–text alignment objectives \(CLIP4Clip, VideoCLIP, X\-CLIP\), qualitative visualizations of the learned transport plans and t\-SNE feature geometry, and representative translation examples on How2Sign comparing VTaMo against SpaMo\.

## S\.1Additional Method Details

### S\.1\.1Attention\-Based Temporal Encoding

The main paper briefly introduces the attention\-based temporal encoder\. Here we provide the full formulation\. Prior work commonly uses fixed temporal convolution with max\-pooling for downsampling sign language features\. Such pooling discards gradient information from non\-selected frames and limits the expressiveness of the learned temporal representation\. We instead use an attention\-based temporal convolution module that performs learnable weighted aggregation inside each local temporal region\. The module containsNℓN\_\{\\ell\}cascaded attention\-pooling layers\. Each layer performs2×2\\timestemporal downsampling, yielding an overall downsampling factor of2Nℓ2^\{N\_\{\\ell\}\}\. Given an input sequence𝐗∈ℝT×dh\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times d\_\{h\}\}, we first apply a one\-dimensional convolution with kernel size55and batch normalization\. For each output positionii, a local window𝒲i\\mathcal\{W\}\_\{i\}centered around position2​i2iis extracted\. A learnable query is then constructed from the mean\-pooled local context and a positional embedding:

𝐪i=1\|𝒲i\|​∑j∈𝒲i𝐱j\+𝐩,\\mathbf\{q\}\_\{i\}=\\frac\{1\}\{\|\\mathcal\{W\}\_\{i\}\|\}\\sum\_\{j\\in\\mathcal\{W\}\_\{i\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{p\},\(17\)where𝐩∈ℝdh\\mathbf\{p\}\\in\\mathbb\{R\}^\{d\_\{h\}\}denotes the positional embedding\.

Multi\-head attention with relative position bias is then applied within the window:

αi,j=softmax​\(𝐪i⊤​𝐤jdh/Nhead\+rj−ci\),𝐨i=∑j∈𝒲iαi,j​𝐯j,\\alpha\_\{i,j\}=\\mathrm\{softmax\}\\left\(\\frac\{\\mathbf\{q\}\_\{i\}^\{\\top\}\\mathbf\{k\}\_\{j\}\}\{\\sqrt\{d\_\{h\}/N\_\{\\mathrm\{head\}\}\}\}\+r\_\{j\-c\_\{i\}\}\\right\),\\qquad\\mathbf\{o\}\_\{i\}=\\sum\_\{j\\in\\mathcal\{W\}\_\{i\}\}\\alpha\_\{i,j\}\\mathbf\{v\}\_\{j\},\(18\)wherecic\_\{i\}denotes the center of the local window andNheadN\_\{\\mathrm\{head\}\}is the number of attention heads\. The output is passed through a linear projection, layer normalization, and dropout\. This design allows gradients to flow to all frames in the local region and produces more informative temporal representations than max\-pooling\.

### S\.1\.2Expanded Local Alignment Formulation

The main paper presents the core optimal transport formulation\. This subsection gives the complete construction of the local alignment module\. Let𝐒=\{𝐬1,…,𝐬M\}∈ℝM×dt\\mathbf\{S\}=\\\{\\mathbf\{s\}\_\{1\},\\dots,\\mathbf\{s\}\_\{M\}\\\}\\in\\mathbb\{R\}^\{M\\times d\_\{t\}\}denote the visual features after fusion projection, and let𝐄=\{𝐞1,…,𝐞U\}∈ℝU×dt\\mathbf\{E\}=\\\{\\mathbf\{e\}\_\{1\},\\dots,\\mathbf\{e\}\_\{U\}\\\}\\in\\mathbb\{R\}^\{U\\times d\_\{t\}\}denote the text token embeddings from the frozen encoder embedding layer of the language model\.

#### S\.1\.2\.1Null token augmentation

Not every visual frame corresponds to a valid lexical unit\. Many frames represent transitional gestures, co\-articulation, or short pauses\. To model this explicitly, we introduce a learnable null token𝐞∅∈ℝdt\\mathbf\{e\}\_\{\\varnothing\}\\in\\mathbb\{R\}^\{d\_\{t\}\}and prepend it to the text sequence:

𝐄~=\{𝐞∅,𝐞1,…,𝐞U\}∈ℝK×dt,K=U\+1\.\\tilde\{\\mathbf\{E\}\}=\\\{\\mathbf\{e\}\_\{\\varnothing\},\\mathbf\{e\}\_\{1\},\\dots,\\mathbf\{e\}\_\{U\}\\\}\\in\\mathbb\{R\}^\{K\\times d\_\{t\}\},\\qquad K=U\+1\.\(19\)A learnable scalar biasb∅b\_\{\\varnothing\}controls the affinity of visual frames to the null token\.

#### S\.1\.2\.2Cost matrix construction

We compute the pairwise transport cost using cosine distance\. Both modalities are firstℓ2\\ell\_\{2\}normalized:

𝐬¯m=𝐬m‖𝐬m‖2,𝐞¯k=𝐞~k‖𝐞~k‖2\.\\bar\{\\mathbf\{s\}\}\_\{m\}=\\frac\{\\mathbf\{s\}\_\{m\}\}\{\\\|\\mathbf\{s\}\_\{m\}\\\|\_\{2\}\},\\qquad\\bar\{\\mathbf\{e\}\}\_\{k\}=\\frac\{\\tilde\{\\mathbf\{e\}\}\_\{k\}\}\{\\\|\\tilde\{\\mathbf\{e\}\}\_\{k\}\\\|\_\{2\}\}\.\(20\)The local cost matrix𝐂∈ℝM×K\\mathbf\{C\}\\in\\mathbb\{R\}^\{M\\times K\}is defined as

Cm,k=\{1−𝐬¯m⊤​𝐞¯k−b∅,if​k=0,1−𝐬¯m⊤​𝐞¯k,otherwise\.C\_\{m,k\}=\\begin\{cases\}1\-\\bar\{\\mathbf\{s\}\}\_\{m\}^\{\\top\}\\bar\{\\mathbf\{e\}\}\_\{k\}\-b\_\{\\varnothing\},&\\text\{if \}k=0,\\\\ 1\-\\bar\{\\mathbf\{s\}\}\_\{m\}^\{\\top\}\\bar\{\\mathbf\{e\}\}\_\{k\},&\\text\{otherwise\}\.\\end\{cases\}\(21\)Columns corresponding to padding tokens are assigned a very large cost to avoid spurious assignments\.

#### S\.1\.2\.3Sinkhorn optimal transport

We solve the entropy\-regularized transport problem

𝐀∗=arg⁡min𝐀∈Π​\(𝐚,𝐛\)⁡⟨𝐀,𝐂⟩−ε​H​\(𝐀\),\\mathbf\{A\}^\{\*\}=\\arg\\min\_\{\\mathbf\{A\}\\in\\Pi\(\\mathbf\{a\},\\mathbf\{b\}\)\}\\langle\\mathbf\{A\},\\mathbf\{C\}\\rangle\-\\varepsilon H\(\\mathbf\{A\}\),\(22\)whereΠ​\(𝐚,𝐛\)\\Pi\(\\mathbf\{a\},\\mathbf\{b\}\)denotes the set of transport plans with prescribed marginals, andH​\(𝐀\)=−∑m,kAm,k​log⁡Am,kH\(\\mathbf\{A\}\)=\-\\sum\_\{m,k\}A\_\{m,k\}\\log A\_\{m,k\}is the entropy term\. The marginals are initialized as uniform distributions over valid visual positions and valid token positions\.

For numerical stability, we use the Sinkhorn algorithm in the log domain\. Let𝐊=exp⁡\(−𝐂/ε\)\\mathbf\{K\}=\\exp\(\-\\mathbf\{C\}/\\varepsilon\)denote the Gibbs kernel\. The dual variables are updated as

log⁡um\\displaystyle\\log u\_\{m\}←log⁡am−log​∑kexp⁡\(−Cm,k/ε\+log⁡vk\),\\displaystyle\\leftarrow\\log a\_\{m\}\-\\log\\sum\_\{k\}\\exp\\left\(\-C\_\{m,k\}/\\varepsilon\+\\log v\_\{k\}\\right\),\(23\)log⁡vk\\displaystyle\\log v\_\{k\}←log⁡bk−log​∑mexp⁡\(−Cm,k/ε\+log⁡um\)\.\\displaystyle\\leftarrow\\log b\_\{k\}\-\\log\\sum\_\{m\}\\exp\\left\(\-C\_\{m,k\}/\\varepsilon\+\\log u\_\{m\}\\right\)\.\(24\)AfterNiterN\_\{\\mathrm\{iter\}\}iterations, the transport plan is recovered as

Am,k=exp⁡\(log⁡um−Cm,k/ε\+log⁡vk\)\.A\_\{m,k\}=\\exp\\left\(\\log u\_\{m\}\-C\_\{m,k\}/\\varepsilon\+\\log v\_\{k\}\\right\)\.\(25\)

#### S\.1\.2\.4Epsilon annealing

To balance exploration and assignment sharpness, we use a multi\-stage annealing schedule forε\\varepsilon:

ε​\(t\)=\{εhigh,t≤twarm,εhigh−εhigh−εmidt2−twarm​\(t−twarm\),twarm<t≤t2,εmid−εmid−εlowt3−t2​\(t−t2\),t2<t≤t3,εlow,t\>t3\.\\varepsilon\(t\)=\\begin\{cases\}\\varepsilon\_\{\\mathrm\{high\}\},&t\\leq t\_\{\\mathrm\{warm\}\},\\\\ \\varepsilon\_\{\\mathrm\{high\}\}\-\\frac\{\\varepsilon\_\{\\mathrm\{high\}\}\-\\varepsilon\_\{\\mathrm\{mid\}\}\}\{t\_\{2\}\-t\_\{\\mathrm\{warm\}\}\}\\left\(t\-t\_\{\\mathrm\{warm\}\}\\right\),&t\_\{\\mathrm\{warm\}\}<t\\leq t\_\{2\},\\\\ \\varepsilon\_\{\\mathrm\{mid\}\}\-\\frac\{\\varepsilon\_\{\\mathrm\{mid\}\}\-\\varepsilon\_\{\\mathrm\{low\}\}\}\{t\_\{3\}\-t\_\{2\}\}\\left\(t\-t\_\{2\}\\right\),&t\_\{2\}<t\\leq t\_\{3\},\\\\ \\varepsilon\_\{\\mathrm\{low\}\},&t\>t\_\{3\}\.\\end\{cases\}\(26\)This schedule keeps the transport plan soft during early training and gradually sharpens the assignments after the alignment becomes stable\.

#### S\.1\.2\.5Local alignment loss terms

The local alignment objective contains three terms\. The transport cost term is

ℒtrans=1B​∑i=1B∑m,kAm,k\(i\)​Cm,k\(i\)\.\\mathcal\{L\}\_\{\\mathrm\{trans\}\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\sum\_\{m,k\}A\_\{m,k\}^\{\(i\)\}C\_\{m,k\}^\{\(i\)\}\.\(27\)To encourage temporal smoothness, we compute a total variation penalty on the row\-normalized transport plan restricted to real text tokens\. Let

A^m,k=Am,k∑k′=1K−1Am,k′,k=1,…,K−1\.\\hat\{A\}\_\{m,k\}=\\frac\{A\_\{m,k\}\}\{\\sum\_\{k^\{\\prime\}=1\}^\{K\-1\}A\_\{m,k^\{\\prime\}\}\},\\qquad k=1,\\dots,K\-1\.\(28\)The temporal variation term is

ℒtv=1\(M−1\)​\(K−1\)​∑m=1M−1∑k=1K−1\|A^m\+1,k−A^m,k\|\.\\mathcal\{L\}\_\{\\mathrm\{tv\}\}=\\frac\{1\}\{\(M\-1\)\(K\-1\)\}\\sum\_\{m=1\}^\{M\-1\}\\sum\_\{k=1\}^\{K\-1\}\\left\|\\hat\{A\}\_\{m\+1,k\}\-\\hat\{A\}\_\{m,k\}\\right\|\.\(29\)To prevent the learnable null biasb∅b\_\{\\varnothing\}from making the null token uniformly cheaper than real tokens, we regulate the cost\-based null affinity \(computed via row\-wise softmax on the cost matrix, independently of the transport plan\):

p∅\(m\)=exp⁡\(−Cm,0/τ\)∑k=0K−1exp⁡\(−Cm,k/τ\),p\_\{\\varnothing\}^\{\(m\)\}=\\frac\{\\exp\(\-C\_\{m,0\}/\\tau\)\}\{\\sum\_\{k=0\}^\{K\-1\}\\exp\(\-C\_\{m,k\}/\\tau\)\},\(30\)ℒnull=\[p¯∅−ρtarget\]\+2,p¯∅=1M​∑m=1Mp∅\(m\)\.\\mathcal\{L\}\_\{\\mathrm\{null\}\}=\\left\[\\bar\{p\}\_\{\\varnothing\}\-\\rho\_\{\\mathrm\{target\}\}\\right\]\_\{\+\}^\{2\},\\qquad\\bar\{p\}\_\{\\varnothing\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}p\_\{\\varnothing\}^\{\(m\)\}\.\(31\)The complete local objective is

ℒlocal=βlocal​ℒtrans\+βtv​ℒtv\+βnull​ℒnull\.\\mathcal\{L\}\_\{\\mathrm\{local\}\}=\\beta\_\{\\mathrm\{local\}\}\\mathcal\{L\}\_\{\\mathrm\{trans\}\}\+\\beta\_\{\\mathrm\{tv\}\}\\mathcal\{L\}\_\{\\mathrm\{tv\}\}\+\\beta\_\{\\mathrm\{null\}\}\\mathcal\{L\}\_\{\\mathrm\{null\}\}\.\(32\)

### S\.1\.3Expanded Global Alignment Formulation

The main paper introduces the sentence\-level global alignment mechanism\. This subsection provides the full formulation\.

#### S\.1\.3\.1Sentence\-level attention pooling

We compute sentence\-level visual and textual representations using parameter\-free attention pooling\. Given a sequence𝐗=\{𝐱1,…,𝐱L\}∈ℝL×dt\\mathbf\{X\}=\\\{\\mathbf\{x\}\_\{1\},\\dots,\\mathbf\{x\}\_\{L\}\\\}\\in\\mathbb\{R\}^\{L\\times d\_\{t\}\}with validity mask𝐦∈\{0,1\}L\\mathbf\{m\}\\in\\\{0,1\\\}^\{L\}, we first define a global query

𝐪=∑lml​𝐱l∑lml\.\\mathbf\{q\}=\\frac\{\\sum\_\{l\}m\_\{l\}\\mathbf\{x\}\_\{l\}\}\{\\sum\_\{l\}m\_\{l\}\}\.\(33\)The attention weights are then

αl=exp⁡\(𝐱l⊤​𝐪/dt\)∑l′exp⁡\(𝐱l′⊤​𝐪/dt\)\.\\alpha\_\{l\}=\\frac\{\\exp\\left\(\\mathbf\{x\}\_\{l\}^\{\\top\}\\mathbf\{q\}/\\sqrt\{d\_\{t\}\}\\right\)\}\{\\sum\_\{l^\{\\prime\}\}\\exp\\left\(\\mathbf\{x\}\_\{l^\{\\prime\}\}^\{\\top\}\\mathbf\{q\}/\\sqrt\{d\_\{t\}\}\\right\)\}\.\(34\)The pooled representation is

𝐳=∑lαl​𝐱l‖∑lαl​𝐱l‖2\.\\mathbf\{z\}=\\frac\{\\sum\_\{l\}\\alpha\_\{l\}\\mathbf\{x\}\_\{l\}\}\{\\left\\\|\\sum\_\{l\}\\alpha\_\{l\}\\mathbf\{x\}\_\{l\}\\right\\\|\_\{2\}\}\.\(35\)This yields normalized sentence\-level vectors𝐳sign\\mathbf\{z\}^\{\\mathrm\{sign\}\}and𝐳text\\mathbf\{z\}^\{\\mathrm\{text\}\}for the visual and textual modalities\.

#### S\.1\.3\.2Memory queue

The global Earth Mover’s Distance objective is most effective when computed over a sufficiently diverse set of sentence pairs\. Since mini\-batches are small, we maintain a first\-in first\-out memory queue that stores recent sentence\-level vector pairs\. At each iteration, the current detached sentence vectors are appended to the queue\. The accumulated vectors\{𝐳nsign\}n=1N\\\{\\mathbf\{z\}^\{\\mathrm\{sign\}\}\_\{n\}\\\}\_\{n=1\}^\{N\}and\{𝐳ntext\}n=1N\\\{\\mathbf\{z\}^\{\\mathrm\{text\}\}\_\{n\}\\\}\_\{n=1\}^\{N\}are then used to compute the global alignment loss\.

#### S\.1\.3\.3Learnable orthogonal mapping

We introduce a learnable transformation matrix𝐓∈ℝdt×dt\\mathbf\{T\}\\in\\mathbb\{R\}^\{d\_\{t\}\\times d\_\{t\}\}, initialized near the identity matrix, to map the visual sentence vectors into the textual embedding space:

𝐳~nsign=𝐳nsign​𝐓‖𝐳nsign​𝐓‖2\.\\tilde\{\\mathbf\{z\}\}^\{\\mathrm\{sign\}\}\_\{n\}=\\frac\{\\mathbf\{z\}^\{\\mathrm\{sign\}\}\_\{n\}\\mathbf\{T\}\}\{\\\|\\mathbf\{z\}^\{\\mathrm\{sign\}\}\_\{n\}\\mathbf\{T\}\\\|\_\{2\}\}\.\(36\)

#### S\.1\.3\.4Global Earth Mover’s Distance loss

Using the transformed visual vectors and the original textual vectors, we define the sentence\-level cost matrix

Cn,n′g=1−\(𝐳~nsign\)⊤​𝐳n′text\.C^\{g\}\_\{n,n^\{\\prime\}\}=1\-\\left\(\\tilde\{\\mathbf\{z\}\}^\{\\mathrm\{sign\}\}\_\{n\}\\right\)^\{\\top\}\\mathbf\{z\}^\{\\mathrm\{text\}\}\_\{n^\{\\prime\}\}\.\(37\)A sentence\-level transport plan𝐏\\mathbf\{P\}is then computed with uniform marginals, and the global alignment loss is

ℒEMD=∑n,n′Pn,n′​Cn,n′g\.\\mathcal\{L\}\_\{\\mathrm\{EMD\}\}=\\sum\_\{n,n^\{\\prime\}\}P\_\{n,n^\{\\prime\}\}C^\{g\}\_\{n,n^\{\\prime\}\}\.\(38\)

#### S\.1\.3\.5Orthogonality constraint and scheduled activation

To preserve pairwise geometry, we regularize the mapping matrix with

ℒorth=‖𝐓⊤​𝐓−𝐈‖F2\.\\mathcal\{L\}\_\{\\mathrm\{orth\}\}=\\\|\\mathbf\{T\}^\{\\top\}\\mathbf\{T\}\-\\mathbf\{I\}\\\|\_\{F\}^\{2\}\.\(39\)
The global loss is activated only after local alignment becomes reliable\. We use the schedule

λg​\(t\)=\{0,t≤twarm,λgmax⋅t−twarmtramp,twarm<t≤twarm\+tramp,λgmax,t\>twarm\+tramp\.\\lambda\_\{g\}\(t\)=\\begin\{cases\}0,&t\\leq t\_\{\\mathrm\{warm\}\},\\\\ \\lambda\_\{g\}^\{\\max\}\\cdot\\frac\{t\-t\_\{\\mathrm\{warm\}\}\}\{t\_\{\\mathrm\{ramp\}\}\},&t\_\{\\mathrm\{warm\}\}<t\\leq t\_\{\\mathrm\{warm\}\}\+t\_\{\\mathrm\{ramp\}\},\\\\ \\lambda\_\{g\}^\{\\max\},&t\>t\_\{\\mathrm\{warm\}\}\+t\_\{\\mathrm\{ramp\}\}\.\\end\{cases\}\(40\)The total global objective is

ℒglobal=λg​\(t\)​ℒEMD\+βorth​ℒorth\.\\mathcal\{L\}\_\{\\mathrm\{global\}\}=\\lambda\_\{g\}\(t\)\\mathcal\{L\}\_\{\\mathrm\{EMD\}\}\+\\beta\_\{\\mathrm\{orth\}\}\\mathcal\{L\}\_\{\\mathrm\{orth\}\}\.\(41\)

### S\.1\.4Window\-Based Reordering Guided by Vision–Text Alignment

This section details the window\-based reordering module introduced in Sec\.[3\.5](https://arxiv.org/html/2607.09126#S3.SS5)\. The motivation for reordering is to resolve a fundamental mismatch between the alignment objective and the sequence generation objective\. Specifically, the cross\-entropy loss supervises the decoder to generate text in the ground\-truth word order, whereas the temporal order of visual features does not necessarily follow that same order\. This mismatch is particularly pronounced in sign language, where the semantic order expressed in visual signing may differ from the order of the target spoken\-language sentence\. As a result, if the visual sequence is kept in its original temporal order, the optimization target imposed by cross\-entropy can become inconsistent with the correspondence learned by the vision–text alignment module\.

To reduce this inconsistency, we reorder the visual feature sequence according to the alignment result before passing it to the decoder\. In this way, the reordered visual sequence is made more compatible with the target text order, so that the alignment\-induced correspondence and the autoregressive generation objective become mutually consistent during training\.

Let𝐒=\{𝐬1,…,𝐬M\}\\mathbf\{S\}=\\\{\\mathbf\{s\}\_\{1\},\\dots,\\mathbf\{s\}\_\{M\}\\\}denote the frame\-level visual feature sequence after temporal encoding, and let𝐀∈ℝM×K\\mathbf\{A\}\\in\\mathbb\{R\}^\{M\\times K\}denote the local transport plan, whereK=U\+1K=U\+1includes the null token andUUis the number of text tokens\.

Rather than performing reordering at the individual\-frame level, we operate on short temporal windows\. The reason is that, during inference, the target text order is unknown, and a practically useful alignment pattern should therefore associate a continuous local visual segment with the same text token\. If isolated frames were reordered independently, the resulting correspondence could become temporally fragmented, making it difficult for the decoder to recover stable lexical units from the visual stream\. Window\-based reordering instead encourages locally coherent visual evidence to move together, which better matches the requirement of translation at test time\.

We partition theMMframe positions intoNw=U\+2N\_\{w\}=U\+2overlapping temporal windows\. The additional two windows absorb boundary effects\. The boundaries of theww\-th window are defined as

sw=⌊w⋅M−1Nw\+0\.5⌋,ew=⌊\(w\+1\)⋅M−1Nw\+0\.5⌋\+1\.s\_\{w\}=\\left\\lfloor w\\cdot\\frac\{M\-1\}\{N\_\{w\}\}\+0\.5\\right\\rfloor,\\qquad e\_\{w\}=\\left\\lfloor\(w\+1\)\\cdot\\frac\{M\-1\}\{N\_\{w\}\}\+0\.5\\right\\rfloor\+1\.\(42\)
For each windowww, we accumulate the alignment mass over the corresponding temporal region:

𝝅w=∑m=swew−1𝐀m,:∈ℝK\.\\boldsymbol\{\\pi\}\_\{w\}=\\sum\_\{m=s\_\{w\}\}^\{e\_\{w\}\-1\}\\mathbf\{A\}\_\{m,:\}\\in\\mathbb\{R\}^\{K\}\.\(43\)The window is then assigned to the real text token with the largest aggregated alignment mass:

ϕ​\(w\)=arg⁡maxu∈\{1,…,U\}⁡πw,u\.\\phi\(w\)=\\arg\\max\_\{u\\in\\\{1,\\dots,U\\\}\}\\pi\_\{w,u\}\.\(44\)If the dominant mass of a window lies on the null token, that window is excluded from reordering\.

For each text tokenuu, we collect the set of windows assigned to that token:

Ωu=\{w∣ϕ​\(w\)=u\}\.\\Omega\_\{u\}=\\\{\\,w\\mid\\phi\(w\)=u\\,\\\}\.\(45\)The reordered visual sequence is then constructed by traversing the text token order from left to right and appending the corresponding windowed visual segments:

𝐑=Concat​\(\{𝐒sw:ew−1∣w∈Ω1↑\},\{𝐒sw:ew−1∣w∈Ω2↑\},…,\{𝐒sw:ew−1∣w∈ΩU↑\}\),\\mathbf\{R\}=\\mathrm\{Concat\}\\Big\(\\\{\\mathbf\{S\}\_\{s\_\{w\}:e\_\{w\}\-1\}\\mid w\\in\\Omega\_\{1\}^\{\\uparrow\}\\\},\\\{\\mathbf\{S\}\_\{s\_\{w\}:e\_\{w\}\-1\}\\mid w\\in\\Omega\_\{2\}^\{\\uparrow\}\\\},\\dots,\\\{\\mathbf\{S\}\_\{s\_\{w\}:e\_\{w\}\-1\}\\mid w\\in\\Omega\_\{U\}^\{\\uparrow\}\\\}\\Big\),\(46\)whereΩu↑\\Omega\_\{u\}^\{\\uparrow\}denotes the windows inΩu\\Omega\_\{u\}sorted by their original temporal order, and𝐒sw:ew−1\\mathbf\{S\}\_\{s\_\{w\}:e\_\{w\}\-1\}denotes the original feature subsequence within windowww\.

This design preserves local temporal continuity while rearranging visual segments into an order that is more compatible with the aligned text sequence\. Consequently, the reordered representation serves as an intermediate form that better reconciles alignment learning with sequence generation\.

### S\.1\.5Expanded Position\-Aligned Contrastive Learning

After reordering, the visual sequence𝐑=\{𝐫1,…,𝐫U\}\\mathbf\{R\}=\\\{\\mathbf\{r\}\_\{1\},\\dots,\\mathbf\{r\}\_\{U\}\\\}is positionally aligned with the text token sequence𝐄=\{𝐞1,…,𝐞U\}\\mathbf\{E\}=\\\{\\mathbf\{e\}\_\{1\},\\dots,\\mathbf\{e\}\_\{U\}\\\}\. We use this structure to impose a token\-level contrastive objective\. Learnable projection heads are applied to both modalities:

𝐫^u=LN​\(𝐖2v​σ​\(𝐖1v​𝐫u\)\),𝐞^u=LN​\(𝐖2t​σ​\(𝐖1t​𝐞u\)\),\\hat\{\\mathbf\{r\}\}\_\{u\}=\\mathrm\{LN\}\\big\(\\mathbf\{W\}^\{v\}\_\{2\}\\sigma\(\\mathbf\{W\}^\{v\}\_\{1\}\\mathbf\{r\}\_\{u\}\)\\big\),\\qquad\\hat\{\\mathbf\{e\}\}\_\{u\}=\\mathrm\{LN\}\\big\(\\mathbf\{W\}^\{t\}\_\{2\}\\sigma\(\\mathbf\{W\}^\{t\}\_\{1\}\\mathbf\{e\}\_\{u\}\)\\big\),\(47\)whereσ\\sigmadenotes GELU andLN\\mathrm\{LN\}denotes layer normalization\. After flattening over the batch and valid positions, the InfoNCE loss is

ℒcontra=−1\|𝒱\|​∑i∈𝒱log⁡exp⁡\(sim​\(𝐫^i,𝐞^i\)/τc\)∑j∈𝒱exp⁡\(sim​\(𝐫^i,𝐞^j\)/τc\),\\mathcal\{L\}\_\{\\mathrm\{contra\}\}=\-\\frac\{1\}\{\|\\mathcal\{V\}\|\}\\sum\_\{i\\in\\mathcal\{V\}\}\\log\\frac\{\\exp\\left\(\\mathrm\{sim\}\(\\hat\{\\mathbf\{r\}\}\_\{i\},\\hat\{\\mathbf\{e\}\}\_\{i\}\)/\\tau\_\{c\}\\right\)\}\{\\sum\_\{j\\in\\mathcal\{V\}\}\\exp\\left\(\\mathrm\{sim\}\(\\hat\{\\mathbf\{r\}\}\_\{i\},\\hat\{\\mathbf\{e\}\}\_\{j\}\)/\\tau\_\{c\}\\right\)\},\(48\)where𝒱\\mathcal\{V\}denotes the set of valid token positions\. Padding positions are masked out of the denominator\. Gradients from this loss are applied only to the visual branch, while the text branch is detached, so that the pre\-trained language model embedding space remains stable\.

## S\.2Additional Qualitative Experiments and Results

### S\.2\.1Attention Pooling versus Max\-Pooling

We compare the proposed attention\-based temporal convolution module \(Sec\.[S\.1\.1](https://arxiv.org/html/2607.09126#S1.SS1)\) with conventional max\-pooling on the Phoenix\-2014T\[camgoz2018neural\]test set \(Table[S1](https://arxiv.org/html/2607.09126#S2.T1)\)\. Max\-pooling keeps only the maximum activation at each position and discards gradient information from all other frames in the pooling window, whereas the attention variant computes a weighted aggregation over all frames, letting gradients reach every frame and yielding more expressive temporal representations\. The attention module improves BLEU\-4 from 26\.57 to 28\.86 and ROUGE\-L from 55\.23 to 60\.24\.

Table S1:Comparison of temporal downsampling strategies on the Phoenix\-2014T\[camgoz2018neural\]test set\.
### S\.2\.2Comparison with Generic Video–Text Alignment

To test whether the structured correspondence learned by VTaMo could be obtained with off\-the\-shelf objectives, we replace our three alignment losses with established generic video–text alignment recipes inside the same pipeline \(Table[S2](https://arxiv.org/html/2607.09126#S2.T2)\)\. A CLIP\-ViT\-only variant with no alignment reaches only 10\.45 BLEU\-4 on the Phoenix\-2014T\[camgoz2018neural\]test set, and CLIP4Clip\[luo2022clip4clip\], VideoCLIP\[xu2021videoclip\], and X\-CLIP\[ma2022xclip\]stay at 9\.47–11\.13, all far below our 28\.86\. These recipes align a whole clip to a caption and assume a near\-monotonic relation, so they do not resolve the non\-monotonic, partial, token\-level correspondence that sign language requires\.

Table S2:Plugging generic video–text alignment objectives into our pipeline \(Phoenix\-2014T\[camgoz2018neural\]test set, BLEU\-4\)\. Each variant replaces our three alignment losses while keeping all other components fixed\.![Refer to caption](https://arxiv.org/html/2607.09126v1/x5.png)Figure S1:Qualitative analysis of the learned alignment and feature geometry\. The left panel shows representative transport plans between visual positions and text tokens\. The right panel shows a t\-SNE visualization of visual features on OpenASL\[shi2022open\]\.
### S\.2\.3Qualitative Analysis of Alignment and Feature Geometry

We further provide qualitative analysis to examine the behavior of the proposed alignment framework beyond the robustness study\. Fig\.[S1](https://arxiv.org/html/2607.09126#S2.F1)visualizes the learned transport plans and the resulting feature organization\.

The left panel shows representative transport plans between temporal visual positions and text tokens\. The alignment maps exhibit clear structured correspondences, indicating that contiguous temporal regions are consistently associated with semantically relevant target tokens\. The right panel shows a t\-SNE visualization of the resulting visual features\. Semantically related samples form compact groups, which suggests that the alignment objectives induce a more organized cross\-modal representation space\.

### S\.2\.4Representative Translation Examples

Table[S3](https://arxiv.org/html/2607.09126#S2.T3)presents representative translation examples on the How2Sign\[duarte2021how2sign\]test set\. Compared with SpaMo\[hwang2025efficient\], our method more faithfully preserves key content words and overall sentence meaning\. The advantage is especially clear in examples that require precise lexical grounding, attribute preservation, and procedural understanding\.

Table S3:Representative translation examples on the How2Sign\[duarte2021how2sign\]test set, comparing SpaMo\[hwang2025efficient\]and Ours on five examples\.