SignRAG：统一的无词汇手语翻译检索增强框架

# Unified Retrieval-Augmented Gloss-Free Sign Language Translation
Source: [https://arxiv.org/html/2610.11371](https://arxiv.org/html/2610.11371)
Zhi RaoAffiliation:Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, ChinaEmail:[zhir@student\.must\.edu\.mo](mailto:)Yucheng ZhouQianran SunAffiliation:MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaYiqing HuangAffiliation:Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, ChinaLongcan YuanAffiliation:Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, ChinaJiayi HouAffiliation:Yale University, New Haven, CT, USAChengwen YaoAffiliation:VIVO AI Lab, ChinaLin ChengAffiliation:VIVO AI Lab, ChinaDonghui SunAffiliation:VIVO AI Lab, ChinaXiaoxin ChenAffiliation:VIVO AI Lab, ChinaJun Wan††thanks:Corresponding author\.Affiliation:Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, ChinaAffiliation:MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China

###### Abstract

Contemporary decoder\-only large language models \(LLMs\) have demonstrated strong capabilities across a wide range of domains\. However, existing pretraining paradigms for gloss\-free sign language translation \(SLT\) are largely designed around conventional encoder\-decoder pretrained language models, which limits their direct applicability to decoder\-only LLMs\. To address this limitation, we propose SignRAG, a unified framework combining hierarchical pretraining, target\-domain retrieval augmentation, and retrieval\-aware reinforcement fine\-tuning\. Hierarchical pretraining first learns linguistically grounded sign representations and then jointly aligns the sign encoder with an LLM, mitigating cross\-modal optimization imbalance\. For downstream adaptation, SignRAG complements parameter\-based fine\-tuning with a target\-domain retrieval gallery that provides instance\-specific translation cues\. To ensure that retrieved contexts are used appropriately, we further introduce Retrieval Utility\-Guided Reinforcement Fine\-Tuning \(RUG\-RFT\), which combines translation\-quality and retrieval\-utility rewards to encourage beneficial retrieval use while suppressing harmful reliance\. Experiments on multiple SLT benchmarks establish new state\-of\-the\-art performance\. In particular, to the best of our knowledge, SignRAG is the first gloss\-free approach to outperform gloss\-supervised methods across all reported metrics on CSL\-Daily\. Our code has been released at[GitHub](https://github.com/shahelaojieraozhi/SignRAG), together with models of different sizes to support future academic research\.

## 1Introduction

Sign language translation \(SLT\) aims to translate visual sign sequences into natural language\. Despite recent progress, existing methods still struggle to generalize across datasets with different signers, signing styles, and vocabulary distributions\. A natural solution is to pretrain on large\-scale sign language corpora and subsequently adapt the model to downstream SLT benchmarks\.

Existing pretraining paradigms for gloss\-free SLT can be broadly divided into two categories\.*Encoder\-only pretraining*independently pretrains the sign encoder before transferring it to downstream translation models\([Zhou et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib11);[Chen et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib14);[Wong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib13);[Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16);[Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17)\), whereas*end\-to\-end pretraining*jointly optimizes visual and language components on large\-scale sign\-language corpora\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21);[Zhang et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib55);[Tanzer and Zhang, 2025](https://arxiv.org/html/2610.11371#bib.bib47);[Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19);[Fish and Bowden, 2026](https://arxiv.org/html/2610.11371#bib.bib54)\)\. However, most existing methods still rely on T5\-based encoder–decoder models, whose language modeling capacity remains limited compared with contemporary LLMs\.

Replacing these decoders with a substantially stronger LLM is not a drop\-in upgrade\. Because the language branch is already strongly pretrained, it can dominate optimization and leave the comparatively weaker visual pathway under\-trained\. We refer to this phenomenon asCross\-Modal Optimization Imbalance\. Such imbalance has been widely observed in multimodal learning\([Wang et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib51);[Huang et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib52);[Peng et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib49)\)and can further induce shortcut learning driven by strong language priors\([Goyal et al\., 2017](https://arxiv.org/html/2610.11371#bib.bib50)\)\. It raises a key question:*how should large\-scale sign pretraining be organized so that visual pathway still learns semantically grounded representations?*

Figure 1:Motivation\.Hierarchical pretraining and retrieval augmentation improve sign\-to\-language learning\.\(a\),\(b\)Comparison of different pretraining strategies\. The solid and dashed lines denote the previous state\-of\-the\-art performance under the gloss\-free and gloss\-based settings, respectively, corresponding to Geo\-Sign\([Fish and Bowden, 2026](https://arxiv.org/html/2610.11371#bib.bib54)\)and CV\-SLT\([Zhao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib7)\)\.\(c\)Comparison of the semantic distance between generated and ground\-truth \(GT\) texts with and without RAG\. Semantic distance is measured using the independent sentence encoderparaphrase\-multilingual\-MiniLM\-L12\-v2\. Bars report the mean cosine distance over all samples, with error bars denoting±\\pmSEM\. Samples are drawn from the CSL\-Daily dev set\.We therefore propose aHierarchical Pretrainingstrategy for LLM\-based SLT\. Inspired by the staged\-alignment paradigm of general\-purpose multimodal large language models \(MLLMs\)\([Liu et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib32);[Ye et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib48);[Bai et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib33);[McKinzie et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib53)\), we organize large\-scale pretraining into two stages\. We first use fine\-grained sign–text supervision to learn linguistically grounded sign representations, and then jointly pretrain the sign encoder, multimodal projector, and LLM on the same corpus\. In this way, the model first learns*what the sign means*and then learns*how to express its semantics in natural language*\. As shown in Fig\.[1](https://arxiv.org/html/2610.11371#S1.F1)\(a\) and \(b\), hierarchical pretraining clearly outperforms alternative pretraining schemes, while thePretrain→\\rightarrowSFTvariant alone already surpasses previous state\-of\-the\-art methods under both gloss\-free and gloss\-based settings\.

After large\-scale pretraining, downstream adaptation introduces a second challenge, which we term theTarget\-Domain Adaptation Dilemma\. Sign\-language datasets exhibit substantial distribution shifts across languages, recording conditions, and signers, while downstream training sets are typically much smaller than the pretraining corpus\. Aggressive fine\-tuning may therefore erode transferable knowledge acquired during pretraining, whereas conservative fine\-tuning may leave the target domain insufficiently adapted\. To address this dilemma, we complement parametric SFT with anon\-parametric target\-domain semantic memory\. Specifically, we construct a retrieval gallery exclusively from the target\-domain training split and retrieve semantically related sign instances together with their translations as instance\-specific linguistic cues\. As shown in Fig\.[1](https://arxiv.org/html/2610.11371#S1.F1)\(c\), retrieval augmentation reduces the semantic distance between generated and GT texts from0\.2340\.234to0\.2060\.206\.

Retrieval, however, is not always beneficial, leading to a third challenge that we term theRetrieval Reliance Dilemma\. Standard SFT optimizes token\-level likelihood but provides no explicit supervision on whether retrieved contexts actually benefit generation, making both underutilization and harmful reliance difficult to regulate\. We first introduce retrieval dropout during retrieval\-augmented SFT to improve robustness when retrieved information is absent\. We then proposeRetrieval Utility\-Guided Reinforcement Fine\-Tuning \(RUG\-RFT\)based on Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib42)\)\. RUG\-RFT combines translation\-quality and retrieval\-utility rewards, encouraging the model to exploit informative retrieval while suppressing harmful reliance on misleading contexts\.

We conduct extensive experiments on four downstream SLT benchmarks: CSL\-Daily, PHOENIX\-2014T, How2Sign, and OpenASL\. SignRAG consistently outperforms existing gloss\-free state\-of\-the\-art methods\. On CSL\-Daily, it further surpasses gloss\-supervised methods across all metrics; to the best of our knowledge, this is the first gloss\-free SLT approach to achieve such performance\. Our main contributions are summarized as follows:

- •We propose a hierarchical pretraining paradigm that first learns linguistically grounded sign representations through fine\-grained sign–text alignment and then performs large\-scale joint multimodal pretraining with a contemporary LLM\.
- •We introduce a target\-domain retrieval\-augmented adaptation mechanism that constructs a non\-parametric semantic memory to complement parameter\-based SFT with instance\-specific target\-domain knowledge\.
- •We propose Retrieval Utility\-Guided Reinforcement Fine\-Tuning \(RUG\-RFT\), which jointly optimizes translation quality and retrieval contribution to enable more effective and robust use of retrieved contexts\.
- •Extensive experiments across multiple SLT benchmarks validate the effectiveness of the proposed framework and establish new state\-of\-the\-art performance\.

## 2Related Work

Sign language translation\.Early SLT relies on gloss supervision via gloss recognition or cascaded translation\([Camgoz et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib3);[Zhou et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib4);[Chen et al\., 2022c](https://arxiv.org/html/2610.11371#bib.bib5);[Chen et al\., 2022b](https://arxiv.org/html/2610.11371#bib.bib6)\), whereas gloss\-free SLT maps sign videos directly to text using vision–language learning, LLM decoding, or cross\-modal alignment\([Camgoz et al\., 2018](https://arxiv.org/html/2610.11371#bib.bib2);[Li et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib8);[Zhao et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib9);[Yin et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib10);[Zhou et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib11);[Wong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib13);[Chen et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib14);[Gong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib15);[Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17);[Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16)\), and external large\-scale pretraining further improves transferability\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21);[Rust et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib22);[Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)\.Retrieval\-augmented generation and reinforcement learning\.RAG conditions generation on retrieved non\-parametric contexts\([Lewis et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib40);[Chen et al\., 2022a](https://arxiv.org/html/2610.11371#bib.bib41)\), retrieval\-based SLT\([Yao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib1)\)mainly targets gloss\-based settings, and GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib42);[Guo et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib43)\)has been applied to retrieval\-enhanced reasoning and generation\([Huang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib44);[Li et al\., 2025a](https://arxiv.org/html/2610.11371#bib.bib45);[Wang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib46)\)\. SignRAG studies*how*large\-scale sign pretraining should be organized under a modern LLM decoder, and complements parametric adaptation with a sign\-to\-text retrieval gallery and retrieval Utility\-Guided reinforcement fine\-tuning, without gloss annotations\. A more detailed discussion is provided in Appendix[A](https://arxiv.org/html/2610.11371#A1)\.

## 3Method

![Refer to caption](https://arxiv.org/html/2610.11371v1/SignRAG.png)Figure 2:Overview of SignRAG\.\(1\)Hierarchical pretraining on large\-scale sign corpora: Stage I learns language\-aware sign representations with a contrastive objectiveℒcon\\mathcal\{L\}\_\{\\mathrm\{con\}\}and a skeleton\-to\-text objectiveℒslt\\mathcal\{L\}\_\{\\mathrm\{slt\}\}; Stage II couples the pretrained sign encoder with the LLM \(Qwen3\) via a multimodal projector and performs large\-scale joint pretraining\.\(2\)Target\-domain supervised fine\-tuning: a*sign retrieval gallery*provides semantically related reference translations as auxiliary cues, and the model is fine\-tuned with sign features and retrieved texts, with random retrieval dropout\.\(3\)Retrieval utility\-guided reinforcement fine\-tuning: a quality reward and a retrieval\-utility reward are combined and optimized with GRPO\.As illustrated in Fig\.[2](https://arxiv.org/html/2610.11371#S3.F2),SignRAGconsists of three stages for gloss\-free sign language translation \(SLT\)\.Stage 1performs hierarchical pretraining, where Stage I learns linguistically grounded sign representations and Stage II jointly pretrains the sign encoder, multimodal projector, and LLM\.Stage 2performs retrieval\-augmented SFT using a target\-domain*Sign Retrieval Gallery*, which retrieves semantically related sign instances and provides their paired translations as auxiliary cues; retrieval dropout is further applied during training\.Stage 3appliesRetrieval Utility\-Guided Reinforcement Fine\-Tuning \(RUG\-RFT\)with GRPO, combining a Quality Reward and a Retrieval\-Utility Reward to improve translation quality and encourage appropriate use of retrieved contexts\. We next describe these three components in detail\.

### 3\.1Hierarchical Pretraining

##### Stage I: Sign–text alignment\.

We first align skeletal sign representations with natural\-language semantics\. Given a sign videoVV, MMPose\([MMPose Contributors, 2020](https://arxiv.org/html/2610.11371#bib.bib29)\)first extracts 2D keypoints, which are subsequently encoded by a CoSign\-based skeleton encoder\([Jiao et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib28)\)into frame\-level skeletal representations\. The paired translationTTis encoded by an mBART\-based text encoder\([Liu et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib36)\)\. The two encoders are jointly optimized with a bidirectional InfoNCE contrastive objectiveℒcon\\mathcal\{L\}\_\{\\mathrm\{con\}\}\([Cheng et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib27)\)and a CoCa\-style skeleton\-to\-text translation objectiveℒslt\\mathcal\{L\}\_\{\\mathrm\{slt\}\}\([Yu et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib31)\):

ℒpt=ℒcon\+ℒslt\.\\mathcal\{L\}\_\{\\mathrm\{pt\}\}=\\mathcal\{L\}\_\{\\mathrm\{con\}\}\+\\mathcal\{L\}\_\{\\mathrm\{slt\}\}\.\(1\)Detailed formulations are provided in Appendix[B\.1](https://arxiv.org/html/2610.11371#A2.SS1)\. This stage endows the sign encoder with linguistically grounded representations before it is coupled with the LLM\.

##### Stage II: Large\-scale sign MLLM pretraining\.

We then couple the aligned sign encoder with an LLM\. A GELU\-activated MLP projector maps the skeletal features into the embedding space of Qwen3\([Yang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib34)\), and LoRA adapters are attached to the query and value projections of the LLM self\-attention layers\. The sign encoder, the projector, and the LoRA\-adapted LLM are jointly optimized on large\-scale sign–text corpora, CSL\-News\([Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)for Chinese sign language and YouTube\-ASL\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21)\)for American sign language, with the autoregressive objective

ℒlm=−∑t=1Tlogpθ\(yt∣y<t,𝐈,𝐙\),\\mathcal\{L\}\_\{\\mathrm\{lm\}\}=\-\\sum\_\{t=1\}^\{T\}\\log p\_\{\\theta\}\\\!\\left\(y\_\{t\}\\mid y\_\{<t\},\\mathbf\{I\},\\mathbf\{Z\}\\right\),\(2\)where𝐙\\mathbf\{Z\}denotes the projected sign features and𝐈\\mathbf\{I\}the instruction\. Because the sign encoder enters joint training already grounded in language semantics, it no longer acts as the weak branch when coupled with the LLM, which mitigates the Cross\-Modal Optimization Imbalance discussed in the Introduction\.

### 3\.2Retrieval\-Augmented SFT

##### Sign retrieval gallery\.

For each downstream dataset, we fine\-tune the sign encoder on its training set and build a*sign\-to\-text*retrieval gallery from that split only; validation and test samples serve exclusively as queries, so that retrieved knowledge always comes from the target domain\. For a gallery videoVgV^\{g\}, its frame\-level representation is

Zg=SkeletonEnc​\(Vg\)∈ℝLg×C,Z^\{g\}=\\texttt\{SkeletonEnc\}\(V^\{g\}\)\\in\\mathbb\{R\}^\{L\_\{g\}\\times C\},\(3\)whereLgL\_\{g\}denotes the temporal length andCCis the feature dimension\. Each gallery entry stores the encoded sign feature sequence and its paired ground\-truth translation,

𝒢n=\(Zng,Tng\)\.\\mathcal\{G\}\_\{n\}=\\left\(Z^\{g\}\_\{n\},T^\{g\}\_\{n\}\\right\)\.\(4\)Query videos are encoded by the same skeleton encoder for sign\-level retrieval\.

##### Attention\-weighted sign–sign similarity\.

Given a query sign sequenceZq∈ℝLq×CZ^\{q\}\\in\\mathbb\{R\}^\{L\_\{q\}\\times C\}and a gallery sequenceZg∈ℝLg×CZ^\{g\}\\in\\mathbb\{R\}^\{L\_\{g\}\\times C\}, we first project their frame\-level features into a sharedDD\-dimensional embedding space:

Q=Norm⁡\(ϕq​\(Zq\)\),G=Norm⁡\(ϕg​\(Zg\)\),Q=\\operatorname\{Norm\}\\\!\\left\(\\phi\_\{q\}\(Z^\{q\}\)\\right\),\\qquad G=\\operatorname\{Norm\}\\\!\\left\(\\phi\_\{g\}\(Z^\{g\}\)\\right\),\(5\)whereQ∈ℝLq×DQ\\in\\mathbb\{R\}^\{L\_\{q\}\\times D\}andG∈ℝLg×DG\\in\\mathbb\{R\}^\{L\_\{g\}\\times D\}\. Here,ϕq​\(⋅\)\\phi\_\{q\}\(\\cdot\)andϕg​\(⋅\)\\phi\_\{g\}\(\\cdot\)denote the projection heads, andNorm⁡\(⋅\)\\operatorname\{Norm\}\(\\cdot\)performs row\-wiseℓ2\\ell\_\{2\}normalization\.

We then compute the frame\-level cosine similarity between the query and gallery sequences:

Ei,j=Qi⊤​Gj,E\_\{i,j\}=Q\_\{i\}^\{\\top\}G\_\{j\},\(6\)whereE∈ℝLq×LgE\\in\\mathbb\{R\}^\{L\_\{q\}\\times L\_\{g\}\}\. For each query frame, we normalize its similarities over the gallery sequence to obtain attention weights:

Pi,j=exp⁡\(Ei,j\)∑j′=1Lgexp⁡\(Ei,j′\)\.P\_\{i,j\}=\\frac\{\\exp\(E\_\{i,j\}\)\}\{\\sum\_\{j^\{\\prime\}=1\}^\{L\_\{g\}\}\\exp\(E\_\{i,j^\{\\prime\}\}\)\}\.\(7\)The global sign–sign similarity is obtained by aggregating the attention\-weighted frame\-level similarities:

S⁡\(Q,G\)=1Lq​∑i=1Lq∑j=1LgPi,j​Ei,j\.S\(Q,G\)=\\frac\{1\}\{L\_\{q\}\}\\sum\_\{i=1\}^\{L\_\{q\}\}\\sum\_\{j=1\}^\{L\_\{g\}\}P\_\{i,j\}E\_\{i,j\}\.\(8\)
Compared with pooling an entire sign video into a single global representation, our formulation preserves fine\-grained frame\-level correspondences between sign sequences, which are crucial for retrieving semantically relevant samples\. For clarity, the padding masks used to handle variable\-length sequences are omitted from the above formulations\. To improve memory efficiency, we shard the gallery across distributed workers and compute sign–sign similarities in query–gallery chunks\. The exact global top\-KKresults are then obtained by merging the local top\-KKcandidates returned by all workers\. Further implementation details are provided in Appendix[B\.4](https://arxiv.org/html/2610.11371#A2.SS4)\.

##### Retrieval\-augmented fine\-tuning\.

We use skeleton embeddingsZsZ\_\{s\}as the visual representation of sign language and project them into the LLM latent space:

Z=MLP⁡\(Zs\),Z=\\mathrm\{MLP\}\(Z\_\{s\}\),\(9\)where the projector is a GELU\-activated MLP\. We use Qwen3\([Yang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib34)\)as the base LLM and form its input by concatenating the instruction𝐈\\mathbf\{I\}, sign features𝐙\\mathbf\{Z\}, and retrieved contexts𝐑\\mathbf\{R\}\.

To support the subsequent Retrieval\-Utility Reward, we apply retrieval dropout during SFT:

𝐑~=\{𝐑,with probability​1−p,∅,with probability​p,\\tilde\{\\mathbf\{R\}\}=\\begin\{cases\}\\mathbf\{R\},&\\text\{with probability \}1\-p,\\\\ \\varnothing,&\\text\{with probability \}p,\\end\{cases\}\(10\)where∅\\varnothingdenotes an empty token sequence without placeholder tokens\. The model is fine\-tuned with LoRA by minimizing

ℒsft=−∑t=1Tlogpθ\(yt∣y<t,𝐈,𝐙,𝐑~\),\\mathcal\{L\}\_\{\\mathrm\{sft\}\}=\-\\sum\_\{t=1\}^\{T\}\\log p\_\{\\theta\}\\\!\\left\(y\_\{t\}\\mid y\_\{<t\},\\mathbf\{I\},\\mathbf\{Z\},\\tilde\{\\mathbf\{R\}\}\\right\),\(11\)whereTTis the target sequence length andθ\\thetadenotes the trainable LoRA parameters\. This encourages coherent generation under both retrieval\-available and retrieval\-absent conditions\.

### 3\.3Retrieval Utility\-Guided Reinforcement Fine\-Tuning

SFT optimizes token\-level likelihood and provides no explicit signal for whether retrieved contexts are useful\. We therefore introduce RUG\-RFT to optimize sentence\-level quality while regulating retrieval reliance\. RUG\-RFT combines a*Quality Reward*with a*Retrieval\-Utility Reward*under Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib42)\), which estimates advantages from intra\-group reward statistics without a separate value model\.

##### GRPO with group sampling\.

Given an input\(𝐈,𝐙,𝐑\)\(\\mathbf\{I\},\\mathbf\{Z\},\\mathbf\{R\}\), the current policyπθ\\pi\_\{\\theta\}samplesNNcandidate outputs\{O1,…,ON\}\\\{O\_\{1\},\\ldots,O\_\{N\}\\\}, each scored byℛ⁡\(Oi\)\\mathcal\{R\}\(O\_\{i\}\)to obtainrir\_\{i\}\. GRPO standardizes rewards within the group to compute

Ai=ri−mean​\{r1,…,rN\}std​\{r1,…,rN\}\.A\_\{i\}=\\frac\{r\_\{i\}\-\\text\{mean\}\\\{r\_\{1\},\\ldots,r\_\{N\}\\\}\}\{\\text\{std\}\\\{r\_\{1\},\\ldots,r\_\{N\}\\\}\}\.\(12\)Letc1=πθ​\(Oi∣𝐈,𝐙,𝐑\)/πold​\(Oi∣𝐈,𝐙,𝐑\)c\_\{1\}=\\pi\_\{\\theta\}\(O\_\{i\}\\mid\\mathbf\{I\},\\mathbf\{Z\},\\mathbf\{R\}\)/\\pi\_\{\\text\{old\}\}\(O\_\{i\}\\mid\\mathbf\{I\},\\mathbf\{Z\},\\mathbf\{R\}\)andc2=clip​\(c1,1−ε,1\+ε\)c\_\{2\}=\\text\{clip\}\(c\_\{1\},1\-\\varepsilon,1\+\\varepsilon\)\. The policy is updated by maximizing

𝒥GRPO\(θ\)=𝔼\[1N∑i=1Nmin\(c1Ai,c2Ai\)−βDKL\(πθ∥πref\)\],\\mathcal\{J\}\_\{\\text\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\min\(c\_\{1\}A\_\{i\},\\,c\_\{2\}A\_\{i\}\)\-\\beta\\,D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\right\],\(13\)where the KL term regularizes deviation from the reference policyπref\\pi\_\{\\text\{ref\}\}\.

#### 3\.3\.1Retrieval\-Utility Reward

We quantify the contribution of retrieval for a*fixed output*using teacher\-forcing conditional likelihoods\. Given\(𝐈,𝐙,𝐑\)\(\\mathbf\{I\},\\mathbf\{Z\},\\mathbf\{R\}\), the current policyπθ\\pi\_\{\\theta\}generatesY=\(y1,…,yT\)Y=\(y\_\{1\},\\ldots,y\_\{T\}\)under the retrieval\-augmented condition\. We then score the sameYYwith retrieval usingπθ\\pi\_\{\\theta\}and without retrieval using a frozen reference policyπref\\pi\_\{\\text\{ref\}\}:

ℓwith​\(Y\)\\displaystyle\\ell^\{\\text\{with\}\}\(Y\)=1T​∑t=1Tlog⁡pθ​\(yt∣y<t,𝐈,𝐙,𝐑\),\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\log p\_\{\\theta\}\\\!\\left\(y\_\{t\}\\mid y\_\{<t\},\\mathbf\{I\},\\mathbf\{Z\},\\mathbf\{R\}\\right\),\(14\)ℓwo​\(Y\)\\displaystyle\\ell^\{\\text\{wo\}\}\(Y\)=1T​∑t=1Tlog⁡pref​\(yt∣y<t,𝐈,𝐙\)\.\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\log p\_\{\\text\{ref\}\}\\\!\\left\(y\_\{t\}\\mid y\_\{<t\},\\mathbf\{I\},\\mathbf\{Z\}\\right\)\.\(15\)Here,πref\\pi\_\{\\text\{ref\}\}is the frozen SFT checkpoint and is identical to the KL anchor in Eq\.[13](https://arxiv.org/html/2610.11371#S3.E13)\. Usingπref\\pi\_\{\\text\{ref\}\}forℓwo​\(Y\)\\ell^\{\\text\{wo\}\}\(Y\)prevents the policy from increasing the reward by artificially lowering its retrieval\-free likelihood\. We define the retrieval\-induced likelihood gain as

Δ​ℓret​\(Y\)=ℓwith​\(Y\)−ℓwo​\(Y\),\\Delta\\ell\_\{\\mathrm\{ret\}\}\(Y\)=\\ell^\{\\text\{with\}\}\(Y\)\-\\ell^\{\\text\{wo\}\}\(Y\),\(16\)where a positive value indicates that retrieval increases the model’s confidence inYY\.

#### 3\.3\.2Quality Reward and Reward Composition

To align with task metrics while regulating retrieval reliance, we combine a sentence\-level Quality Reward with the Retrieval\-Utility Reward\. Letℛqual​\(Y\)\\mathcal\{R\}\_\{\\text\{qual\}\}\(Y\)denote a translation\-quality score, such as BLEU or ROUGE, computed against the ground truth\. We retain only positive retrieval utility:

ℛutil\+​\(Y\)=max⁡\(0,Δ​ℓret​\(Y\)\)\.\\mathcal\{R\}\_\{\\text\{util\}\}^\{\+\}\(Y\)=\\max\\\!\\left\(0,\\,\\Delta\\ell\_\{\\mathrm\{ret\}\}\(Y\)\\right\)\.\(17\)To balance the magnitudes of the two rewards, we introduce

γ=𝔼^​\[\|ℛqual\|\]max⁡\(𝔼^​\[ℛutil\+\],τ\),\\gamma=\\frac\{\\widehat\{\\mathbb\{E\}\}\\big\[\|\\mathcal\{R\}\_\{\\text\{qual\}\}\|\\big\]\}\{\\max\\\!\\left\(\\widehat\{\\mathbb\{E\}\}\\big\[\\mathcal\{R\}\_\{\\text\{util\}\}^\{\+\}\\big\],\\,\\tau\\right\)\},\(18\)where𝔼^​\[⋅\]\\widehat\{\\mathbb\{E\}\}\[\\cdot\]is an exponential moving average over recent batches,τ\\tauprevents early instability, andγ\\gammais treated as stop\-gradient\. The total reward used in Eq\.[12](https://arxiv.org/html/2610.11371#S3.E12)is

ℛtotal​\(Y\)=ℛqual​\(Y\)\+α⋅γ⋅ℛutil\+​\(Y\),\\mathcal\{R\}\_\{\\text\{total\}\}\(Y\)=\\mathcal\{R\}\_\{\\text\{qual\}\}\(Y\)\+\\alpha\\cdot\\gamma\\cdot\\mathcal\{R\}\_\{\\text\{util\}\}^\{\+\}\(Y\),\(19\)whereα\\alphacontrols the strength of the Retrieval\-Utility Reward\.

## 4Experiment

### 4\.1Experimental Setup

##### Datasets and metrics\.

We use CSL\-News\([Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)and YouTube\-ASL\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21)\)as large\-scale pretraining datasets for Chinese Sign Language and American Sign Language, respectively\. We then evaluate SignRAG on four downstream SLT benchmarks: CSL\-Daily\([Zhou et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib4)\), PHOENIX\-2014T\([Camgoz et al\., 2018](https://arxiv.org/html/2610.11371#bib.bib2)\), How2Sign\([Duarte et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib24)\), and OpenASL\([Shi et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib25)\)\. Their train/validation/test splits are 18,401/1,077/1,176, 7,096/519/642, 31,085/1,739/2,348, and 96,476/966/975, respectively\. We report BLEU\-1/2/3/4\([Papineni et al\., 2002](https://arxiv.org/html/2610.11371#bib.bib37)\)and ROUGE\-L\([Lin, 2004](https://arxiv.org/html/2610.11371#bib.bib38)\)following standard SLT evaluation protocols\.

##### Retrieval gallery protocol\.

For each dataset, we build a separate train\-only retrieval gallery from its training split, where each entry stores the encoded sign representation and paired translation\. Validation and test samples are used only as queries and are never included in the gallery; during training, the current query instance is excluded to prevent self\-retrieval\. The details are shown in Appendix[C\.7](https://arxiv.org/html/2610.11371#A3.SS7)\. A detailed discussion of the efficiency, computational cost, and retrieval scalability of SignRAG is provided in Appendix[E](https://arxiv.org/html/2610.11371#A5)\.

Table 1:Main results on four SLT benchmarks\. “w/ gloss” and “w/o gloss” denote methods with and without gloss supervision, respectively\. “Ext\.” indicates whether external sign\-language datasets are used for pre\-training\. For the “w/o gloss” setting, methods with and without external pre\-training data are separated and compared independently\. Bold scores indicate the best results within each dataset, gloss setting, and external\-data setting\. “Venue” denotes the publication venue and year\.DatasetGlossMethodVenueExt\.R\-L↑\\uparrowB@1↑\\uparrowB@2↑\\uparrowB@3↑\\uparrowB@4↑\\uparrowCSL\-Dailyw/ glossSignBT\([Zhou et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib4)\)CVPR’21✗49\.3151\.4237\.2627\.7621\.34SLTUNET\([Zhang et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib18)\)ICLR’23✗54\.0854\.9841\.4431\.8425\.01TS\-SLT\([Chen et al\., 2022c](https://arxiv.org/html/2610.11371#bib.bib5)\)NeurIPS’22✗55\.7255\.4442\.5932\.8725\.79CV\-SLT\([Zhao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib7)\)AAAI’24✗57\.0658\.2945\.1535\.7728\.94w/o glossGFSLT\-VLP\([Zhou et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib11)\)ICCV’23✗36\.4439\.3724\.9316\.2611\.00Sign2GPT\([Wong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib13)\)ICLR’24✗42\.3641\.7528\.7320\.6015\.40VAP\([Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17)\)ECCV’24✗48\.5649\.99––20\.85C2RL\([Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16)\)TCSVT’25✗48\.2149\.3236\.2827\.5421\.61RVLF\([Rao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib26)\)Findings CVPR’26✗55\.9254\.9642\.4533\.3326\.71SignRAG\-SFT–✗55\.4354\.2241\.0732\.2126\.06SignRAG\-RFT–✗56\.3656\.6043\.8734\.4327\.73Uni\-Sign\([Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)ICLR’25✓56\.5155\.0842\.1432\.9826\.36Geo\-Sign\([Fish and Bowden, 2026](https://arxiv.org/html/2610.11371#bib.bib54)\)NeurIPS’25✓57\.9755\.89––27\.42SignRAG\-SFT–✓60\.2059\.3946\.9437\.9531\.22SignRAG\-RFT–✓62\.3461\.1248\.7439\.7332\.91PHOENIX\-2014Tw/ glossSignBT\([Zhou et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib4)\)CVPR’21✗49\.5450\.8037\.7529\.7224\.32TS\-SLT\([Chen et al\., 2022c](https://arxiv.org/html/2610.11371#bib.bib5)\)NeurIPS’22✗53\.4854\.9042\.4334\.4628\.95CV\-SLT\([Zhao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib7)\)AAAI’24✗54\.3354\.8842\.6834\.7929\.27w/o glossGFSLT\-VLP\([Zhou et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib11)\)ICCV’23✗42\.4943\.7133\.1826\.1121\.44Sign2GPT\([Wong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib13)\)ICLR’24✗48\.9049\.5435\.9628\.8322\.52FLa\-LLM\([Chen et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib14)\)LREC\-COLING’24✗45\.2746\.2935\.3328\.0323\.09VAP\([Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17)\)ECCV’24✗51\.2853\.07––26\.16C2RL\([Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16)\)TCSVT’25✗50\.9652\.8140\.2032\.2026\.75RVLF\([Rao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib26)\)Findings CVPR’26✗52\.5653\.6041\.2033\.2427\.86SignRAG\-SFT–✗52\.1453\.0540\.8832\.6827\.30SignRAG\-RFT–✗53\.0754\.0942\.6234\.3128\.63How2Signw/o glossGloFE\-VN\([Lin et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib12)\)ACL’23✗12\.614\.97\.33\.92\.2OpenSLT\([Tarrés et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib20)\)CVPRW/WiCV’23✗–34\.019\.312\.28\.0C2RL\([Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16)\)TCSVT’25✗27\.029\.118\.612\.99\.4FLa\-LLM\([Chen et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib14)\)LREC\-COLING’24✗27\.829\.819\.013\.39\.7VAP\([Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17)\)ECCV’24✗27\.839\.2––12\.9RVLF\([Rao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib26)\)Findings CVPR’26✗34\.739\.425\.318\.614\.3SignRAG\-SFT–✗34\.740\.225\.918\.413\.7SignRAG\-RFT–✗36\.242\.928\.020\.315\.2YouTube\-ASL\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21)\)NeurIPS D&B’23✓–37\.824\.116\.912\.4SignMusketeers\([Gueuwou et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib23)\)Findings ACL’25✓–41\.527\.219\.314\.3Uni\-Sign\([Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)ICLR’25✓36\.040\.227\.119\.714\.9SSVP\-SLT\([Rust et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib22)\)ACL’24✓38\.443\.228\.820\.815\.5SignRAG\-SFT–✓36\.743\.328\.620\.715\.6SignRAG\-RFT–✓38\.544\.730\.322\.116\.7OpenASLw/o glossOpenASL\([Shi et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib25)\)EMNLP’22✗21\.0220\.9212\.088\.596\.72GloFE\-VN\([Lin et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib12)\)ACL’23✗21\.7521\.5612\.749\.057\.06C2RL\([Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16)\)TCSVT’25✗31\.3631\.4621\.8516\.5813\.21VAP\([Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17)\)ECCV’24✗41\.3845\.92––21\.23RVLF\([Rao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib26)\)Findings CVPR’26✗43\.9248\.7335\.4027\.9922\.84SignRAG\-SFT–✗42\.9646\.2434\.2527\.1222\.24SignRAG\-RFT–✗43\.8749\.1436\.5829\.0123\.61Uni\-Sign\([Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)ICLR’25✓43\.2249\.3536\.2228\.5523\.14SignRAG\-SFT–✓45\.3350\.3138\.4530\.2825\.12SignRAG\-RFT–✓46\.9151\.5639\.5232\.0626\.62
##### Implementation details\.

We adopt CoSign\([Jiao et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib28)\)with a three\-layer Transformer\-based skeleton encoder and use Qwen3\-8B as the decoder\-only LLM across all datasets\. All experiments are conducted on four NVIDIA RTX 4090D GPUs, each equipped with 48 GB of memory, using bfloat16 precision\. The first and second pre\-training stages are run for 200 and 50 epochs, respectively, followed by 10 epochs of SFT and 2 epochs of RFT\. We use SGD with a learning rate of0\.010\.01in the first pre\-training stage and AdamW in all subsequent stages\. The learning rates for SFT and RFT are set to1×10−41\\times 10^\{\-4\}and1×10−51\\times 10^\{\-5\}, respectively, with a retrieval dropout probability ofp=0\.5p=0\.5\. Checkpoints are selected based on validation BLEU\-4 scores\. For the number of retrieved items, we setK=3K=3\. Full implementation details are provided in Appendix[C](https://arxiv.org/html/2610.11371#A3)\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2610.11371#S4.T1)summarizes the main results\. SignRAG outperforms existing state\-of\-the\-art methods across all evaluated datasets\. In particular, on CSL\-Daily, SignRAG further surpasses gloss\-based approaches, suggesting that large\-scale pretraining on external sign language data, together with the capabilities of modern LLMs, can partially compensate for the absence of costly gloss annotations\. Even without pretraining on additional data, SignRAG achieves state\-of\-the\-art performance under the same setting\. The training protocol follows RVLF\([Rao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib26)\)\. It is worth noting that, due to the lack of a large\-scale German sign language pretraining dataset, all results on PHOENIX\-2014T are obtained without pretraining on additional data\. Compared with the Chinese datasets, the gains of SignRAG over existing state\-of\-the\-art methods are more moderate on the English datasets\. This discrepancy may be attributed to differences in dataset difficulty and evaluation protocols between Chinese and English benchmarks\. In addition, the YouTube\-ASL data used for pretraining are incomplete, which may limit the effectiveness of large\-scale pretraining on English sign language translation\. Further details are provided in Appendix[D\.4](https://arxiv.org/html/2610.11371#A4.SS4)\. For a fair comparison, we do not include methods that use substantially larger pretraining datasets or introduce modules specifically designed for English SLT datasets\([Zhang et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib55);[Jang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib56)\)\.

### 4\.3Ablation Study

Table[2](https://arxiv.org/html/2610.11371#S4.T2)\(a\) evaluates the contributions of pretraining, retrieval augmentation, and RUG\-RFT on CSL\-Daily\. Pretraining improves ROUGE\-L from 55\.03 to 58\.78 and BLEU\-4 from 23\.46 to 29\.72\. Adding retrieval further raises them to 60\.20 and 31\.22, respectively\. With RUG\-RFT, performance reaches 62\.34 ROUGE\-L and 32\.91 BLEU\-4, with consistent gains across all metrics\. These results demonstrate the effectiveness of progressively introducing pretraining, retrieval, and retrieval utility\-guided reinforcement fine\-tuning\. Table[2](https://arxiv.org/html/2610.11371#S4.T2)\(b\) analyzes the reward design\. Starting from the same retrieval\-augmented SFT model, both the Quality Reward and Retrieval\-Utility Reward improve performance independently\. Their combination achieves the best results across all metrics, reaching 62\.34 ROUGE\-L and 32\.91 BLEU\-4\. This confirms that translation quality and retrieval utility provide complementary optimization signals for RUG\-RFT\.

Table 2:Ablation studies on CSL\-Daily\. Components in \(a\) are added cumulatively\. Reward variants in \(b\) start from the same retrieval\-augmented SFT model\.
### 4\.4Analysis

##### Retrieval quality analysis\.

We investigate the relationship between the number of retrieved contexts, retrieval quality, and final translation performance\. For each query, we vary the number of retrieved contextsKKand measure the token\-level overlap between the retrieved texts and the ground\-truth translation\. The resulting scores are then averaged over all queries to reflect the overall retrieval quality\. Chinese sentences are segmented using Jieba, with punctuation, empty tokens, and symbol\-only tokens removed\. Let𝒯GT\\mathcal\{T\}\_\{\\mathrm\{GT\}\}denote the token set of the ground\-truth translation, and let𝒯ret\(K\)\\mathcal\{T\}\_\{\\mathrm\{ret\}\}^\{\(K\)\}denote the token set aggregated from the top\-KKretrieved translations\. We define*Retrieval Recall*,*Retrieval Precision*, and*Retrieval F1*as

Recret​\(K\)=\|𝒯GT∩𝒯ret\(K\)\|\|𝒯GT\|,Precret​\(K\)=\|𝒯GT∩𝒯ret\(K\)\|\|𝒯ret\(K\)\|,F1ret​\(K\)=2​Recret​\(K\)​Precret​\(K\)Recret​\(K\)\+Precret​\(K\)\.\\mathrm\{Rec\}\_\{\\mathrm\{ret\}\}\(K\)=\\tfrac\{\|\\mathcal\{T\}\_\{\\mathrm\{GT\}\}\\cap\\mathcal\{T\}\_\{\\mathrm\{ret\}\}^\{\(K\)\}\|\}\{\|\\mathcal\{T\}\_\{\\mathrm\{GT\}\}\|\},~~\\mathrm\{Prec\}\_\{\\mathrm\{ret\}\}\(K\)=\\tfrac\{\|\\mathcal\{T\}\_\{\\mathrm\{GT\}\}\\cap\\mathcal\{T\}\_\{\\mathrm\{ret\}\}^\{\(K\)\}\|\}\{\|\\mathcal\{T\}\_\{\\mathrm\{ret\}\}^\{\(K\)\}\|\},~~~\\mathrm\{F1\}\_\{\\mathrm\{ret\}\}\(K\)=\\tfrac\{2\\,\\mathrm\{Rec\}\_\{\\mathrm\{ret\}\}\(K\)\\,\\mathrm\{Prec\}\_\{\\mathrm\{ret\}\}\(K\)\}\{\\mathrm\{Rec\}\_\{\\mathrm\{ret\}\}\(K\)\+\\mathrm\{Prec\}\_\{\\mathrm\{ret\}\}\(K\)\}\.\(20\)
Here,Recret\\mathrm\{Rec\}\_\{\\mathrm\{ret\}\}measures how much ground\-truth lexical information is covered by the retrieved contexts, whilePrecret\\mathrm\{Prec\}\_\{\\mathrm\{ret\}\}measures how much of the retrieved lexical content is relevant to the ground truth\. Their harmonic mean,F1ret\\mathrm\{F1\}\_\{\\mathrm\{ret\}\}, captures the trade\-off between retrieval coverage and retrieval noise\. As shown in Fig\.[3\(a\)](https://arxiv.org/html/2610.11371#S4.F3.sf1), asKKincreases,F1ret\\mathrm\{F1\}\_\{\\mathrm\{ret\}\}first improves and then continuously decreases, and the final translation performance exhibits the same trend\. This is because introducing additional less\-relevant contexts can instead interfere with SLT performance\. All BLEU\-4 scores in Fig\.[3\(a\)](https://arxiv.org/html/2610.11371#S4.F3.sf1)are obtained from SFT models\.

\(a\)Retrieval quality vs\. retrieval number\(b\)BLEU\-4 vs\. LLM size
Figure 3:Main analysis\.\(a\)Relationship between retrieval quality and final translation performance under different numbers of retrieved contexts\.\(b\)BLEU\-4 scaling with LLM size\. We evaluate Qwen3 backbones with different parameter scales \(0\.6B, 4B, and 8B\) under the same training and evaluation protocol\.
##### Model scaling analysis\.

We further investigate how the scale of the language model affects translation performance\. As shown in Fig\.[3\(b\)](https://arxiv.org/html/2610.11371#S4.F3.sf2), under the same training and evaluation protocol, BLEU\-4 increases from 22\.72 to 28\.58 and 32\.91 when scaling the Qwen3 backbone from 0\.6B to 4B and 8B, respectively\. The consistent improvement shows that contemporary LLMs also exhibit stronger capabilities in sign language translation\. These results further indicate that SignRAG can effectively benefit from scaling the underlying LLM\.

##### Case study\.

We provide a qualitative comparison with the current SOTA model, Geo\-Sign\([Fish and Bowden, 2026](https://arxiv.org/html/2610.11371#bib.bib54)\), to analyze the effect of retrieved contexts\. As shown in Table[3](https://arxiv.org/html/2610.11371#S4.T3), the reference sentence contains fine\-grained semantic cues, including “Traditionally” and “lucky money\.” Although Geo\-Sign produces a partially related translation, it fails to preserve these cues\. In contrast, SignRAG retrieves semantically aligned contexts that provide relevant topical and lexical cues\. This further supports our finding that retrieval can improve translation performance\. Additional qualitative examples are provided in Table[15](https://arxiv.org/html/2610.11371#A6.T15)in the Appendix\. Meanwhile, the bad case presented in Appendix[D\.7](https://arxiv.org/html/2610.11371#A4.SS7)shows that noisy retrieval can mislead the model, further motivating RUG\-RFT\.

Table 3:Qualitative comparison on CSL\-Daily\.

## 5Conclusion

We propose SignRAG, a unified framework for gloss\-free sign language translation\. SignRAG strengthens sign representation learning through staged pretraining and leverages target\-domain semantic memory to provide instance\-specific linguistic cues for downstream translation\. Furthermore, RUG\-RFT jointly considers translation quality and retrieval utility to encourage effective use of beneficial retrieved contexts\. Experiments across multiple SLT benchmarks show that SignRAG achieves new state\-of\-the\-art performance and surpasses existing gloss\-supervised methods across all reported metrics on CSL\-Daily\. Future work will explore more efficient and robust retrieval mechanisms, as well as knowledge transfer across datasets and languages\.

### AI Use Statement

We used generative AI tools primarily to assist with writing and language polishing, including improvements to grammar, wording, clarity, and readability\. We did not use generative AI tools for research design, experimental design, data analysis, or result interpretation\. All AI\-assisted text was carefully reviewed and revised by the authors, who take full responsibility for the final content of the manuscript\.

### Reproducibility Statement

Although we have provided the experimental settings, training protocols, and implementation details as comprehensively as possible in the main paper and Appendix, we recognize that reproducibility is essential for impactful scientific research\. To facilitate further verification and future research, upon acceptance, we will release the source code, pretrained checkpoints from different training stages, and models of different sizes\. We hope these resources will support reproducible evaluation and further academic research\.

## References

- Baiet al\.\(2025\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang,et al\.Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§1](https://arxiv.org/html/2610.11371#S1.p4.1)\.
- Camgozet al\.\(2018\)N\. C\. Camgoz, S\. Hadfield, O\. Koller, H\. Ney, and R\. BowdenNeural sign language translation\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 7784–7793\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[Table 7](https://arxiv.org/html/2610.11371#A4.T7.2.8.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1)\.
- Camgozet al\.\(2020\)N\. C\. Camgoz, O\. Koller, S\. Hadfield, and R\. BowdenSign language transformers: joint end\-to\-end sign language recognition and translation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10023–10033\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Chenet al\.\(2022a\)W\. Chen, H\. Hu, X\. Chen, P\. Verga, and W\. W\. CohenMuRAG: multimodal retrieval\-augmented generator for open question answering over images and text\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 5558–5570\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Chenet al\.\(2022b\)Y\. Chen, F\. Wei, X\. Sun, Z\. Wu, and S\. LinA simple multi\-modality transfer learning baseline for sign language translation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 5120–5130\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Chenet al\.\(2022c\)Y\. Chen, R\. Zuo, F\. Wei, Y\. Wu, S\. Liu, and B\. MakTwo\-stream network for sign language recognition and translation\.Advances in Neural Information Processing Systems35,pp\. 17043–17056\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.18.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.4.1)\.
- Chenet al\.\(2025\)Z\. Chen, B\. Zhou, Y\. Huang, J\. Wan, Y\. Hu, H\. Shi, Y\. Liang, Z\. Lei, and D\. ZhangC2\{\}^\{2\}rl: content and context representation learning for gloss\-free sign language translation and retrieval\.IEEE Transactions on Circuits and Systems for Video Technology\.External Links:[Document](https://dx.doi.org/10.1109/TCSVT.2025.3553052)Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.24.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.30.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.44.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.9.1)\.
- Chenet al\.\(2024\)Z\. Chen, B\. Zhou, J\. Li, J\. Wan, Z\. Lei, N\. Jiang, Q\. Lu, and G\. ZhaoFactorized learning assisted with large language model for gloss\-free sign language translation\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation,pp\. 7071–7081\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.22.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.31.1)\.
- Chenget al\.\(2023\)Y\. Cheng, F\. Wei, J\. Bao, D\. Chen, and W\. ZhangCiCo: domain\-aware sign language retrieval via cross\-lingual contrastive learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19016–19026\.Cited by:[§B\.2](https://arxiv.org/html/2610.11371#A2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px1.p1.1)\.
- Chenget al\.\(2024\)Z\. Cheng, S\. Leng, H\. Zhang, Y\. Xin, X\. Li, G\. Chen, Y\. Zhu, W\. Zhang, Z\. Luo, D\. Zhao, and L\. BingVideoLLaMA 2: advancing spatial\-temporal modeling and audio understanding in video\-LLMs\.arXiv preprint arXiv:2406\.07476\.Cited by:[§D\.2](https://arxiv.org/html/2610.11371#A4.SS2.p2.1)\.
- Duarteet al\.\(2021\)A\. Duarte, S\. Palaskar, L\. Ventura, D\. Ghadiyaram, K\. DeHaan, F\. Metze, J\. Torres, and X\. Giró\-i\-NietoHow2Sign: a large\-scale multimodal dataset for continuous american sign language\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 2735–2744\.Cited by:[Table 7](https://arxiv.org/html/2610.11371#A4.T7.2.5.2),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1)\.
- Fish and Bowden \(2026\)E\. Fish and R\. BowdenGeo\-sign: hyperbolic contrastive regularisation for geometrically aware sign language translation\.Advances in Neural Information Processing Systems38,pp\. 99293–99330\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[Figure 1](https://arxiv.org/html/2610.11371#S1.F1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§4\.4](https://arxiv.org/html/2610.11371#S4.SS4.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.14.1)\.
- Gonget al\.\(2024\)J\. Gong, L\. G\. Foo, Y\. He, H\. Rahmani, and J\. LiuLLMs are good sign language translators\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 18362–18372\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Goyalet al\.\(2017\)Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. ParikhMaking the v in vqa matter: elevating the role of image understanding in visual question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2610.11371#S1.p3.1)\.
- Gueuwouet al\.\(2025\)S\. Gueuwou, X\. Du, G\. Shakhnarovich, and K\. LivescuSignMusketeers: an efficient multi\-stream approach for sign language translation at scale\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 22506–22521\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1157)Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.37.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Huanget al\.\(2025\)J\. Huang, S\. Madala, R\. Sidhu, C\. Niu, H\. Peng, J\. Hockenmaier, and T\. ZhangRAG\-RL: advancing retrieval\-augmented generation via reinforcement learning and curriculum learning\.arXiv preprint arXiv:2503\.12759\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Huanget al\.\(2022\)Y\. Huang, J\. Lin, C\. Zhou, H\. Yang, and L\. HuangModality competition: what makes joint training of multi\-modal network fail in deep learning? \(provably\)\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),Cited by:[§D\.3](https://arxiv.org/html/2610.11371#A4.SS3.p2.1),[§1](https://arxiv.org/html/2610.11371#S1.p3.1)\.
- Janget al\.\(2025\)Y\. Jang, L\. Momeni, Z\. Jiang, J\. S\. Chung, G\. Varol, and A\. ZissermanLost in translation, found in embeddings: sign language translation and alignment\.arXiv preprint arXiv:2512\.08040\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.11371#S4.SS2.p1.1)\.
- Jiaoet al\.\(2024\)P\. Jiao, Y\. Min, and X\. ChenVisual alignment pre\-training for sign language translation\.InComputer Vision – ECCV 2024,pp\. 349–367\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-72946-1%5F20)Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.23.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.32.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.45.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.8.1)\.
- Jiaoet al\.\(2023\)P\. Jiao, Y\. Min, Y\. Li, X\. Wang, L\. Lei, and X\. ChenCoSign: exploring co\-occurrence signals in skeleton\-based continuous sign language recognition\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20676–20686\.Cited by:[§C\.2](https://arxiv.org/html/2610.11371#A3.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px3.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Liet al\.\(2020\)D\. Li, C\. Xu, X\. Yu, K\. Zhang, B\. Swift, H\. Suominen, and H\. LiTSPNet: hierarchical feature learning via temporal semantic pyramid for sign language translation\.Advances in Neural Information Processing Systems33,pp\. 12034–12045\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Liet al\.\(2025a\)Y\. Li, Q\. Luo, X\. Li, B\. Li, Q\. Cheng, B\. Wang, Y\. Zheng, Y\. Wang, Z\. Yin, and X\. QiuR3\-RAG: learning step\-by\-step reasoning and retrieval for LLMs via reinforcement learning\.arXiv preprint arXiv:2505\.23794\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Liet al\.\(2025b\)Z\. Li, W\. Zhou, W\. Zhao, K\. Wu, H\. Hu, and H\. LiUni\-sign: toward unified sign language understanding at scale\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=0Xt7uT04cQ)Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[Table 7](https://arxiv.org/html/2610.11371#A4.T7.2.2.2),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.13.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.38.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.49.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1)\.
- Linet al\.\(2023\)K\. Lin, X\. Wang, L\. Zhu, K\. Sun, B\. Zhang, and Y\. YangGloss\-free end\-to\-end sign language translation\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12904–12916\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.722)Cited by:[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.28.3),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.43.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual instruction tuning\.Advances in Neural Information Processing Systems36,pp\. 34892–34916\.Cited by:[§C\.4](https://arxiv.org/html/2610.11371#A3.SS4.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p4.1)\.
- Liuet al\.\(2020\)Y\. Liu, J\. Gu, N\. Goyal, X\. Li, S\. Edunov, M\. Ghazvininejad, M\. Lewis, and L\. ZettlemoyerMultilingual denoising pre\-training for neural machine translation\.Transactions of the Association for Computational Linguistics8,pp\. 726–742\.Cited by:[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px1.p1.1)\.
- McKinzieet al\.\(2024\)B\. McKinzie, Z\. Gan, J\. Fatahalian, A\. Toshev, P\. Dufter, A\. Shalit, and Y\. YangMM1: methods, analysis & insights from multimodal LLM pre\-training\.InProceedings of the European Conference on Computer Vision \(ECCV\),Cited by:[§1](https://arxiv.org/html/2610.11371#S1.p4.1)\.
- MMPose Contributors \(2020\)MMPose ContributorsOpenMMLab pose estimation toolbox and benchmark\.Note:[https://github\.com/open\-mmlab/mmpose](https://github.com/open-mmlab/mmpose)Cited by:[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px1.p1.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1)\.
- Penget al\.\(2022\)X\. Peng, Y\. Wei, A\. Deng, D\. Wang, and D\. HuBalanced multimodal learning via on\-the\-fly gradient modulation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 8238–8247\.Cited by:[§D\.3](https://arxiv.org/html/2610.11371#A4.SS3.p2.1),[§1](https://arxiv.org/html/2610.11371#S1.p3.1)\.
- Popović \(2015\)M\. PopovićchrF: character n\-gram f\-score for automatic mt evaluation\.InProceedings of the Tenth Workshop on Statistical Machine Translation,pp\. 392–395\.Cited by:[§D\.5](https://arxiv.org/html/2610.11371#A4.SS5.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,pp\. 8748–8763\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1)\.
- Raoet al\.\(2026\)Z\. Rao, Y\. Zhou, B\. Zhou, Y\. Huang, S\. Escalera, and J\. WanRVLF: a reinforcing vision\-language framework for gloss\-free sign language translation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) Findings,pp\. 9237–9247\.Cited by:[§4\.2](https://arxiv.org/html/2610.11371#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.10.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.25.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.33.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.46.1)\.
- Rustet al\.\(2024\)P\. Rust, B\. Shi, S\. Wang, N\. C\. Camgöz, and J\. MaillardTowards privacy\-aware sign language translation at scale\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8624–8641\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.467)Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.39.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu,et al\.DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p2.1),[§1](https://arxiv.org/html/2610.11371#S1.p6.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[§3\.3](https://arxiv.org/html/2610.11371#S3.SS3.p1.1)\.
- Shiet al\.\(2022\)B\. Shi, D\. Brentari, G\. Shakhnarovich, and K\. LivescuOpen\-domain sign language translation learned from online video\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 6365–6379\.Cited by:[Table 7](https://arxiv.org/html/2610.11371#A4.T7.2.6.1),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.42.3)\.
- Tanzer and Zhang \(2025\)G\. Tanzer and B\. ZhangYoutube\-sl\-25: a large\-scale, open\-domain multilingual sign language parallel corpus\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 81921–81934\.Cited by:[§1](https://arxiv.org/html/2610.11371#S1.p2.1)\.
- Tarréset al\.\(2023\)L\. Tarrés, G\. I\. Gállego, A\. Duarte, J\. Torres, and X\. Giró\-i\-NietoSign language translation from instructional videos\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,pp\. 5625–5635\.Cited by:[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.29.1)\.
- Uthuset al\.\(2023\)D\. Uthus, G\. Tanzer, and M\. GeorgYouTube\-ASL: a large\-scale, open\-domain american sign language\-english parallel corpus\.Advances in Neural Information Processing Systems36,pp\. 29029–29047\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[Table 7](https://arxiv.org/html/2610.11371#A4.T7.2.3.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.36.1)\.
- Wanget al\.\(2025\)Q\. Wang, R\. Ding, Y\. Zeng, Z\. Chen, L\. Chen, S\. Wang, P\. Xie, F\. Huang, and F\. ZhaoVRAG\-RL: empower vision\-perception\-based RAG for visually rich information understanding via iterative reasoning with reinforcement learning\.arXiv preprint arXiv:2505\.22019\.Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Wanget al\.\(2020\)W\. Wang, D\. Tran, and M\. FeiszliWhat makes training multi\-modal classification networks hard?\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§D\.3](https://arxiv.org/html/2610.11371#A4.SS3.p2.1),[§1](https://arxiv.org/html/2610.11371#S1.p3.1)\.
- Wonget al\.\(2024\)R\. Wong, N\. C\. Camgoz, and R\. BowdenSign2GPT: leveraging large language models for gloss\-free sign language translation\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=LqaEEs3UxU)Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.21.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.7.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.11371#S3.SS2.SSS0.Px3.p1.2)\.
- Yaoet al\.\(2026\)H\. Yao, W\. Zhou, H\. Zhou, H\. Hu, and H\. LiRetrieval\-augmented sign language translation\.ACM Transactions on Multimedia Computing, Communications and Applications22\(2\)\.External Links:[Document](https://dx.doi.org/10.1145/3771277)Cited by:[§A\.2](https://arxiv.org/html/2610.11371#A1.SS2.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Yeet al\.\(2024\)Q\. Ye, H\. Xu, J\. Ye, M\. Yan, A\. Hu, H\. Liu, Q\. Qian, J\. Zhang, and F\. HuangMPLUG\-owi2: revolutionizing multi\-modal large language model with modality collaboration\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 13040–13051\.Cited by:[§1](https://arxiv.org/html/2610.11371#S1.p4.1)\.
- Yinet al\.\(2023\)A\. Yin, T\. Zhong, L\. Tang, W\. Jin, T\. Jin, and Z\. ZhaoGloss attention for gloss\-free sign language translation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 2551–2562\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Yuet al\.\(2022\)J\. Yu, Z\. Wang, V\. Vasudevan, L\. Yeung, M\. Seyedhosseini, and Y\. WuCoCa: contrastive captioners are image\-text foundation models\.arXiv preprint arXiv:2205\.01917\.Cited by:[§3\.1](https://arxiv.org/html/2610.11371#S3.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2023\)B\. Zhang, M\. Müller, and R\. SennrichSLTUNET: a simple unified model for sign language translation\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EBS4C77p_5S)Cited by:[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.3.1)\.
- Zhanget al\.\(2024\)B\. Zhang, G\. Tanzer, and O\. FiratScaling sign language translation\.Advances in neural information processing systems37,pp\. 114018–114047\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.11371#S4.SS2.p1.1)\.
- Zhaoet al\.\(2021\)J\. Zhao, W\. Qi, W\. Zhou, N\. Duan, M\. Zhou, and H\. LiConditional sentence generation and cross\-modal reranking for sign language translation\.IEEE Transactions on Multimedia24,pp\. 2662–2672\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1)\.
- Zhaoet al\.\(2024\)R\. Zhao, L\. Zhang, B\. Fu, C\. Hu, J\. Su, and Y\. ChenConditional variational autoencoder for sign language translation with cross\-modal alignment\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19643–19651\.Cited by:[Figure 1](https://arxiv.org/html/2610.11371#S1.F1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.19.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.5.1)\.
- Zhouet al\.\(2023\)B\. Zhou, Z\. Chen, A\. Clapés, J\. Wan, Y\. Liang, S\. Escalera, Z\. Lei, and D\. ZhangGloss\-free sign language translation: improving from visual\-language pretraining\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20871–20881\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[§C\.3](https://arxiv.org/html/2610.11371#A3.SS3.p1.1),[§1](https://arxiv.org/html/2610.11371#S1.p2.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.20.2),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.6.2)\.
- Zhouet al\.\(2021\)H\. Zhou, W\. Zhou, W\. Qi, J\. Pu, and H\. LiImproving sign language translation with monolingual data by sign back\-translation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 1316–1325\.Cited by:[§A\.1](https://arxiv.org/html/2610.11371#A1.SS1.p1.1),[Table 7](https://arxiv.org/html/2610.11371#A4.T7.2.7.1),[§2](https://arxiv.org/html/2610.11371#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.11371#S4.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.17.3),[Table 1](https://arxiv.org/html/2610.11371#S4.T1.2.2.3)\.
- Zhuet al\.\(2024\)Q\. Zhu, J\. Li, F\. Yuan, J\. Fan, and Q\. GanA chinese continuous sign language dataset based on complex environments\.arXiv preprint arXiv:2409\.11960\.Cited by:[§D\.6](https://arxiv.org/html/2610.11371#A4.SS6.p1.1)\.

## APPENDIX

## Appendix AExtended Related Work

This section provides the full version of the related\-work discussion that is condensed into a single paragraph in the main text\.

### A\.1Sign Language Translation

Sign language translation \(SLT\) aims to recover the semantic content conveyed by a signer from a sequence of performed signs\. Unlike standard video understanding tasks, SLT must account for the unique linguistic structure of sign language\. Specifically, sign sequences are more naturally aligned with gloss units, whereas spoken language sentences often follow substantially different grammatical patterns\. This mismatch makes direct translation challenging\. As a result, early SLT methods\([Camgoz et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib3);[Zhou et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib4);[Chen et al\., 2022c](https://arxiv.org/html/2610.11371#bib.bib5);[Chen et al\., 2022b](https://arxiv.org/html/2610.11371#bib.bib6)\)commonly relied on gloss supervision, typically by introducing auxiliary Sign Language Recognition modules or explicitly modeling gloss generation before sentence translation\. Despite their strong performance, such methods depend heavily on costly gloss annotations, which limits scalability\. To address this issue, recent work has increasingly focused on gloss\-free SLT, which learns a direct mapping from sign videos to spoken language sentences\. Early studies\([Camgoz et al\., 2018](https://arxiv.org/html/2610.11371#bib.bib2);[Camgoz et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib3);[Li et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib8);[Zhao et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib9);[Yin et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib10)\)demonstrated the feasibility of this setting, although a clear performance gap from gloss\-based methods remains\. More recent efforts attempt to close this gap by improving visual\-text representation learning\. For example, GFSLT\([Zhou et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib11)\)adapts CLIP\([Radford et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib30)\)to sign language, inspiring a series of vision\-language SLT models\([Wong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib13);[Chen et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib14);[Gong et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib15);[Jiao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib17);[Chen et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib16)\)\. Another line of work improves SLT from the data perspective\. Recent studies\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21);[Rust et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib22);[Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19);[Zhang et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib55);[Gueuwou et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib23);[Jang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib56);[Fish and Bowden, 2026](https://arxiv.org/html/2610.11371#bib.bib54)\)show that pre\-training on large\-scale sign corpora can significantly benefit downstream translation\. However, most existing efforts focus on pre\-training T5\-based encoder–decoder architectures, whose language priors, general knowledge, and generative capabilities remain limited compared with contemporary LLMs\. Therefore, how to design effective pre\-training strategies for adapting sign representations to contemporary LLM\-based SLT models remains largely unexplored\.

### A\.2RL for Retrieval\-Enhanced Reasoning

Retrieval\-augmented generation \(RAG\) enhances parametric language models by incorporating non\-parametric external contexts, where retrieved evidence is used as additional information during generation\([Lewis et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib40);[Chen et al\., 2022a](https://arxiv.org/html/2610.11371#bib.bib41)\)\. By providing explicit external references, RAG improves knowledge utilization and reduces the dependence on purely parametric memory\. Beyond general text generation, retrieval\-based approaches have also been explored in sign language translation\. However, existing retrieval\-based SLT methods\([Yao et al\., 2026](https://arxiv.org/html/2610.11371#bib.bib1)\)mainly focus on gloss\-based settings, where retrieval is performed under the guidance of gloss representations\.

In parallel, reinforcement learning \(RL\) has become an effective post\-training strategy for improving the reasoning and generation abilities of large language models\. Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib42);[Guo et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib43)\)provides an efficient RL framework by estimating relative advantages among grouped samples without requiring an additional value model\. Recent studies have explored combining RL with RAG to optimize retrieval\-enhanced reasoning and generation\. RAG\-RL\([Huang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib44)\)improves answer generation in retrieval contexts through reinforcement learning and curriculum strategies\. R3\-RAG\([Li et al\., 2025a](https://arxiv.org/html/2610.11371#bib.bib45)\)further enables joint reasoning and retrieval optimization with outcome and process rewards\. VRAG\-RL\([Wang et al\., 2025](https://arxiv.org/html/2610.11371#bib.bib46)\)extends this paradigm to visual\-rich scenarios by jointly optimizing retrieval, reasoning, and generation\.

These studies demonstrate the potential of RL for dynamically optimizing the interaction between retrieval and generation\. However, existing RL\-based RAG methods mainly target knowledge\-intensive reasoning tasks, rather than multimodal translation with sign language inputs\. In contrast, SignRAG investigates how large\-scale sign pretraining should be organized under a modern LLM decoder, and complements parametric adaptation with a sign\-to\-text retrieval gallery and retrieval utility\-guided reinforcement fine\-tuning\. Unlike previous retrieval\-based SLT approaches, SignRAG operates without gloss annotations and explicitly optimizes the usefulness of retrieved translation contexts during generation\.

## Appendix BDetailed Formulations and Derivations

### B\.1Vision\-Language Pre\-training

For completeness, we provide additional details of the vision\-language pre\-training procedure\.

### B\.2Attention\-weighted Skeleton–Text Similarity

Skeleton and text features are projected into a sharedDD\-dimensional space andℓ2\\ell\_\{2\}\-normalized\. Given skeleton features\{fi\}i=1M\\\{f\_\{i\}\\\}\_\{i=1\}^\{M\}and text features\{tj\}j=1L\\\{t\_\{j\}\\\}\_\{j=1\}^\{L\}, we compute token\-level similarities

Ei,j=fi⊤​tj\.E\_\{i,j\}=f\_\{i\}^\{\\top\}t\_\{j\}\.\(21\)
To calculate the global skeleton\-to\-text similarityzS,Tz\_\{S,T\}, a row\-wise Softmax\([Cheng et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib27)\)is applied to𝑬\\bm\{E\}to obtain the attention weights𝑷\\bm\{P\}:

Pi,j=exp⁡\(Ei,j/τ\)∑k=1Lexp⁡\(Ei,k/τ\),\\displaystyle P\_\{i,j\}=\\frac\{\\exp\(E\_\{i,j\}/\\tau\)\}\{\\sum\_\{k=1\}^\{L\}\\exp\(E\_\{i,k\}/\\tau\)\},\(22\)whereτ\\taudenotes a trainable temperature parameter\. The global skeleton–text similarity is defined as

zS,T=1M​∑i=1M∑j=1LPi,j​Ei,j\.z\_\{S,T\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\sum\_\{j=1\}^\{L\}P\_\{i,j\}\\,E\_\{i,j\}\.\(23\)The text\-to\-skeleton similarity is computed analogously\.

### B\.3Bidirectional InfoNCE Loss

Given a mini\-batch ofBBpaired samples, we construct batch\-level similarity matrices and apply a bidirectional InfoNCE objective with temperatureτ′\\tau^\{\\prime\}\.

### B\.4Memory\-bounded distributed retrieval\.

This section describes the implementation of memory\-efficient retrieval in SignRAG\. Let𝒢\\mathcal\{G\}denote a training gallery containingNNsign sequences, each represented by frame\-level features with channel dimensionCC\. Loading the entire gallery into GPU memory requires storage proportional to𝒪⁡\(N​L​C\)\\mathcal\{O\}\(NLC\), whereLLdenotes the padded temporal length\. Comparing a query batch against the full gallery further introduces memory costs for pairwise similarities and intermediate computations\. To support large galleries, we combine CPU\-resident gallery storage, distributed sharding, and block\-wise similarity computation, followed by exact distributed top\-KKaggregation\.

##### CPU\-resident gallery sharding\.

WithRRdistributed workers, we partition the gallery into disjoint shards:

𝒢=⋃r=1R𝒢\(r\),𝒢\(r\)∩𝒢\(r′\)=∅,r≠r′\.\\mathcal\{G\}=\\bigcup\_\{r=1\}^\{R\}\\mathcal\{G\}^\{\(r\)\},\\qquad\\mathcal\{G\}^\{\(r\)\}\\cap\\mathcal\{G\}^\{\(r^\{\\prime\}\)\}=\\varnothing,\\quad r\\neq r^\{\\prime\}\.\(24\)Each worker is responsible for approximatelyN/RN/Rgallery samples\. The encoded features of its local shard are retained in CPU memory and transferred to GPU only for the current similarity computation\. Thus, no worker needs to keep its entire gallery shard, or the complete gallery, in GPU memory\.

##### Block\-wise similarity computation\.

We divide the queries into chunks𝒬b\\mathcal\{Q\}\_\{b\}of at mostBqB\_\{q\}sequences and each local gallery shard into chunks𝒢c\(r\)\\mathcal\{G\}^\{\(r\)\}\_\{c\}of at mostBgB\_\{g\}sequences:

\|𝒬b\|≤Bq,\|𝒢c\(r\)\|≤Bg\.\|\\mathcal\{Q\}\_\{b\}\|\\leq B\_\{q\},\\qquad\|\\mathcal\{G\}^\{\(r\)\}\_\{c\}\|\\leq B\_\{g\}\.\(25\)Each query chunk is broadcast to all workers\. Workerrrthen compares it against the chunks of its local gallery shard in turn\. For the current query–gallery block, the aggregated similarities are

Ab,c\(r\)​\[u,v\]=S⁡\(Qub,Gv\(r,c\)\),Ab,c\(r\)∈ℝ\|𝒬b\|×\|𝒢c\(r\)\|,A^\{\(r\)\}\_\{b,c\}\[u,v\]=S\\left\(Q^\{b\}\_\{u\},G^\{\(r,c\)\}\_\{v\}\\right\),\\qquad A^\{\(r\)\}\_\{b,c\}\\in\\mathbb\{R\}^\{\|\\mathcal\{Q\}\_\{b\}\|\\times\|\\mathcal\{G\}^\{\(r\)\}\_\{c\}\|\},\(26\)whereS⁡\(⋅,⋅\)S\(\\cdot,\\cdot\)is the attention\-weighted sign–sign similarity defined in Eq\.[8](https://arxiv.org/html/2610.11371#S3.E8)\. After evaluating a block, its similarity scores are transferred to CPU memory, and the GPU storage occupied by the current gallery chunk can be released before loading the next one\. The query chunk remains available while the worker processes its local gallery\. Scores from successive gallery chunks are collected on CPU for local candidate selection\.

##### GPU memory usage\.

LetLLbound the padded temporal lengths of the active query and gallery chunks\. The resident retrieval tensors on each GPU require

ℳresident=𝒪⁡\(\(Bq\+Bg\)​L​C\+Bq​Bg\)\.\\mathcal\{M\}\_\{\\mathrm\{resident\}\}=\\mathcal\{O\}\\left\(\(B\_\{q\}\+B\_\{g\}\)LC\+B\_\{q\}B\_\{g\}\\right\)\.\(27\)The first term accounts for the active query and gallery features, and the second accounts for their aggregated similarity scores\. This bound excludes model parameters, temporary intermediate activations, and kernel workspace; the actual peak GPU memory also depends on the implementation ofS⁡\(⋅,⋅\)S\(\\cdot,\\cdot\)\. For fixed chunk sizes and sequence lengths, the resident GPU storage is independent of the total gallery sizeNN\. The chunk sizes can therefore be adjusted to the available device memory\. CPU storage still grows with the local gallery size and the similarity scores retained for candidate selection, and retrieval still evaluates every eligible query–gallery pair\.

##### Candidate filtering\.

During training\-set retrieval, we exclude the query itself, identified by its sample identity, and gallery samples whose reference translations are identical to that of the query\. These exclusions prevent trivial retrieval of the query or entries with identical supervision\. Letyxy\_\{x\}denote the reference translation of samplexx\. The eligible local gallery for a training queryqqis

ℰ\(r\)\(q\)=\{g∈𝒢\(r\)\|g≠q,yg≠yq\}\.\\mathcal\{E\}^\{\(r\)\}\(q\)=\\left\\\{g\\in\\mathcal\{G\}^\{\(r\)\}\\;\\middle\|\\;g\\neq q,\\;y\_\{g\}\\neq y\_\{q\}\\right\\\}\.\(28\)For development and test queries, retrieval is performed against the training gallery without using the query reference translation, soℰ\(r\)​\(q\)=𝒢\(r\)\\mathcal\{E\}^\{\(r\)\}\(q\)=\\mathcal\{G\}^\{\(r\)\}\. All exclusions are applied before local top\-KKselection to ensure that eligible candidates are not displaced by entries that would subsequently be removed\.

##### Distributed top\-KKaggregation\.

After evaluating its local gallery, workerrrselects

𝒞K\(r\)​\(q\)=TopKg∈ℰ\(r\)​\(q\)⁡S⁡\(q,g\),\\mathcal\{C\}^\{\(r\)\}\_\{K\}\(q\)=\\operatorname\{TopK\}\_\{g\\in\\mathcal\{E\}^\{\(r\)\}\(q\)\}S\(q,g\),\(29\)returning all eligible entries if fewer thanKKare available\. Only the selected sample identifiers and their similarity scores are communicated for final aggregation\. The global retrieved set is

ℛK​\(q\)=TopKg∈⋃r=1R𝒞K\(r\)​\(q\)⁡S⁡\(q,g\)\.\\mathcal\{R\}\_\{K\}\(q\)=\\operatorname\{TopK\}\_\{g\\in\\bigcup\_\{r=1\}^\{R\}\\mathcal\{C\}^\{\(r\)\}\_\{K\}\(q\)\}S\(q,g\)\.\(30\)This aggregation preserves exact exhaustive retrieval\. Under a consistent ordering for tied scores, any candidate outside its shard’s local top\-KKis preceded by at leastKKeligible candidates from that shard and therefore cannot belong to the global top\-KK\. Consequently, selecting from the union of local candidates yields the same result as selecting directly from the full eligible gallery\. Final aggregation requires at mostR​KRKcandidates per query, without gathering the gallery features or the full similarity matrix across workers\.

##### Communication and CPU fallback\.

In our implementation, variable\-length query payloads and retrieved candidate metadata are communicated through a CPU\-side Gloo process group\. This avoids the additional GPU\-memory allocation associated with object collectives over NCCL\. If a query–gallery block still exceeds the available GPU memory, the implementation supports a CPU fallback that evaluates the same similarity functionS⁡\(⋅,⋅\)S\(\\cdot,\\cdot\)\. The fallback changes the execution device while preserving the scoring definition, eligibility rules, and exhaustive retrieval procedure\.

## Appendix CMore Implementation Details

To enable a thorough understanding of SignRAG, this section presents more implementation details\.

### C\.1Computing Infrastructure

The entire experimental process was conducted on a computational infrastructure comprising an AMD EPYC 7502 32\-Core Processor, 4 NVIDIA RTX 4090D GPUs with 48GB of VRAM, running the Ubuntu 20\.04 operating system, with Python 3\.10, CUDA 11\.8, transformers 4\.51\.3 and PyTorch 2\.1\.2\.

### C\.2Keypoint Extraction\.

Following the preprocessing protocol adopted in CoSign\([Jiao et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib28)\), we extract frame\-level human keypoints from sign\-language videos using the off\-the\-shelf MMPose estimator\. To focus on motion and appearance cues that are most relevant to sign understanding, we retain a compact set of keypoints covering the upper body, hands, mouth, and face\. Specifically, the selected keypoints include 9 body joints, 21 joints for each hand, 8 mouth landmarks, and 18 facial landmarks, resulting in 77 keypoints per frame\. Additional implementation details follow CoSign\([Jiao et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib28)\)\.

Table 4:Implementation details of the four training stages\. LR denotes learning rate, FFN denotes feed\-forward network, and RFT denotes reinforcement fine\-tuning\.
### C\.3Pretraining\(Stage I\): Sign\-text pre\-training\.

In the first stage, we train a compact sign\-text model to align skeleton\-based sign representations with textual semantics\. Following GFSLT\-VLP\([Zhou et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib11)\), the text encoder and decoder are initialized by reusing the first three layers of mBART, while the skeleton encoder adopts the same three\-layer Transformer structure\. Each Transformer block uses 8 attention heads, a hidden size of 512, and a feed\-forward dimension of 2048\. The model is optimized for 200 epochs using SGD with a learning rate of 0\.01 and a weight decay of 0\.001\. A cosine learning rate schedule is applied with 20 warmup epochs\. The batch size is set to 8\. For CSL\-News, we follow prior work and reduce the vocabulary to a task\-specific subset of 11,540 tokens\. The contrastive feature dimension for both visual and textual representations is set to 1024\.

### C\.4Pretraining\(Stage II\): Large\-scale sign MLLM pretraining\.

We use Qwen3\-8B as the base language model\. Following the design of LLaVA\([Liu et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib32)\), an MLP projector with GELU activation maps the 512\-dimensional visual features into the 4096\-dimensional embedding space of the LLM\. To enable parameter\-efficient adaptation, we apply LoRA to the query and value projection matrices in the self\-attention layers of the LLM\. The LoRA rank, scaling parameter, and dropout rate are set to 16, 32, and 0\.3, respectively\. The model is trained for 50 epochs using the AdamW optimizer with a learning rate of1×10−41\\times 10^\{\-4\}and a batch size of 8\. Cross\-entropy loss is used as the training objective\.

### C\.5SFT: Retrieval\-Augmented Supervised Fine\-Tuning\.

After large\-scale sign MLLM pretraining, we merge the learned LoRA parameters into the base LLM to obtain the final pretrained LLM checkpoint\. We then attach a new set of LoRA modules using the same LoRA configuration, including the rank, scaling factor, and dropout rate\. The main difference in this stage is that the training data are switched from large\-scale pretraining corpora, such as CSL\-News and YouTube\-ASL, to downstream SLT datasets\. Retrieved texts are additionally incorporated and combined with the sign features to construct the multimodal input\. The system prompt and instruction template used in this stage are shown in Fig\.[4](https://arxiv.org/html/2610.11371#A3.F4)\. During SFT, the model is trained for 10 epochs using AdamW with a learning rate of1×10−41\\times 10^\{\-4\}\. The batch size is set to 4, and cross\-entropy loss is used as the training objective\.

Figure 4:System prompt and instruction template\. Here,with RAGdenotes the setting where retrieved contexts are provided to the LLM, whilewithout RAGdenotes the setting where no retrieved contexts are provided\.
### C\.6RFT: Retrieval Utility\-Guided Reinforcement Fine\-Tuning\.

After supervised fine\-tuning, we merge the learned LoRA parameters with our pretrained LLM weights to obtain the SignRAG\-SFT checkpoint\. We then attach a new set of LoRA modules for reinforcement fine\-tuning\. In this stage, the LoRA rank is increased to 64, the scaling factor is set to 128, and the dropout rate is reduced to 0\.05\. The model is trained for 2 epochs using AdamW with a learning rate of1×10−51\\times 10^\{\-5\}and a batch size of 8\. Following GRPO, we sample 8 candidate responses for each input and compute group\-relative advantages based on both the sentence\-level quality reward and the retrieval\-utility reward\. The same prompt template used in the SFT stage is adopted for RFT\. For datasets containing very short sentences, we append sentence\-final punctuation before reward computation to reduce the instability of BLEU\-4\-based training signals\.

### C\.7Identical\-text filtering in training retrieval\.

We use CSL\-Daily as an example to illustrate the retrieval procedure\. The retrieval gallery𝒢\\mathcal\{G\}consists of all training samples, where each gallery entry is represented as𝒢i=\(gi,ti,i​di\)\\mathcal\{G\}\_\{i\}=\(g\_\{i\},t\_\{i\},id\_\{i\}\), containing the sign sample, its translation, and sample ID\. Given a queryqq, all gallery samples are first ranked according to their visual similarity, followed by sample\-level and text\-level filtering\. The complete procedure is summarized in Algorithm[1](https://arxiv.org/html/2610.11371#alg1)\.

Algorithm 1Retrieval Context Construction1:Query

qqwith

i​dqid\_\{q\}and

tqt\_\{q\}; gallery

𝒢=\{\(gi,ti,i​di\)\}i=1N\\mathcal\{G\}=\\\{\(g\_\{i\},t\_\{i\},id\_\{i\}\)\\\}\_\{i=1\}^\{N\}; similarity function

S⁡\(⋅,⋅\)S\(\\cdot,\\cdot\); retrieval number

KK
2:Retrieved contexts

ℛK​\(q\)\\mathcal\{R\}\_\{K\}\(q\)
3:Compute visual similarities:

4:

si=S\(q,gi\),i=1,…,Ns\_\{i\}=S\(q,g\_\{i\}\),\\qquad i=1,\\ldots,N
5:Rank gallery samples in descending order:

6:

π=Argsorti⁡\(si\)\\pi=\\operatorname\{Argsort\}\_\{i\}\(s\_\{i\}\)
7:Initialize

ℛK​\(q\)←∅\\mathcal\{R\}\_\{K\}\(q\)\\leftarrow\\varnothing
8:for

i∈πi\\in\\pido

9:if

i​di=i​dqid\_\{i\}=id\_\{q\}then

10:continue

11:endif

12:ifTrainRetrieval

∧\\landti=tqt\_\{i\}=t\_\{q\}then

13:continue

14:endif

15:

ℛK​\(q\)←ℛK​\(q\)∪\{ti\}\\mathcal\{R\}\_\{K\}\(q\)\\leftarrow\\mathcal\{R\}\_\{K\}\(q\)\\cup\\\{t\_\{i\}\\\}
16:if

\|ℛK​\(q\)\|=K\|\\mathcal\{R\}\_\{K\}\(q\)\|=Kthen

17:break

18:endif

19:endfor

20:return

ℛK​\(q\)\\mathcal\{R\}\_\{K\}\(q\)

During training\-set self\-retrieval, identical\-text filtering is introduced to prevent the target sentence of the query from being directly provided as retrieval context\. Without this filtering, samples sharing the same target annotation may dominate the retrieved candidates, allowing the translation model to exploit a trivial shortcut by copying the retrieved target sentence\. More importantly, this would introduce a mismatch between training and dev/test inference, since the exact target translation of a query is generally unavailable in the retrieval gallery at inference time\.The effectiveness of identical\-text filtering on CSL\-Daily is evaluated in Table[5](https://arxiv.org/html/2610.11371#A3.T5)\. Only 2 test samples share identical translations with the training set, suggesting that the improvements of SignRAG are not driven by a trivial retrieval shortcut that directly matches test queries to training samples with the same target translation\.

Table 5:Effect of identical\-text filtering during training\-set self\-retrieval on CSL\-Daily\.

## Appendix DFurther Analyses

### D\.1Analysis of different pretraining strategies\.

We compare three pretraining strategies\.*Encoder\-only pretraining*initializes only the sign encoder from Stage I before downstream SFT\.*End\-to\-end pretraining*directly pretrains a randomly initialized sign encoder jointly with the LLM\.*Hierarchical pretraining*first learns a language\-aware sign encoder and then jointly pretrains the sign encoder, multimodal projector, and LLM\. As shown in Table[6](https://arxiv.org/html/2610.11371#A4.T6), hierarchical pretraining substantially outperforms direct end\-to\-end pretraining on CSL\-Daily, achieving BLEU\-4 and ROUGE\-L scores of 29\.72 and 58\.78, compared with 9\.53 and 38\.70, respectively\. These results highlight the importance of learning semantically grounded sign representations before integrating them with an LLM\.

Table 6:Comparison of pretraining strategies\. Results on CSL\-News and CSL\-Daily correspond to pretraining and downstream SFT evaluation, respectively\.
### D\.2Directly Fine\-Tuning a Video LLM\.

Constructing large\-scale sign language datasets remains challenging due to the high cost of data collection and annotation, as well as the limited availability of annotators with professional sign language expertise\. Consequently, a substantial cross\-modal semantic gap persists between sign videos and natural language text\. Video LLMs have demonstrated strong capabilities in temporal modeling and natural language generation\. Adapting such models with a limited number of sign–text pairs therefore provides a potentially effective way to leverage their existing video understanding and language modeling capabilities for cross\-modal alignment, while reducing the dependence on large\-scale sign language annotations and costly training\. From this perspective, adapting general\-purpose Video LLMs to sign language translation represents a promising research direction\.

To ensure a fair comparison and isolate the effect of the model architecture as much as possible, we strictly follow the official training configuration of VideoLLaMA2, keeping all training settings and hyperparameters unchanged while replacing only the original video data with sign language data\. As shown in Fig\.[5](https://arxiv.org/html/2610.11371#A4.F5), our preliminary experiments reveal a substantial performance gap between directly fine\-tuning VideoLLaMA2 \(7B\)\([Cheng et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib35)\)on sign–text pairs and our SFT model\. We attribute this gap in part to the distinctive nature of sign language, whose semantics are substantially more structured and abstract than those of general video content\. The visual encoder of a general\-purpose Video LLM is not specifically designed to model the fine\-grained visual patterns underlying sign language and therefore struggles to learn sufficiently discriminative and linguistically relevant sign representations\. Moreover, compared with generic video understanding, SLT requires visual semantics at a much finer granularity, demanding accurate modeling of handshape, motion, spatial location, non\-manual cues, and their temporal compositions\. Capturing such subtle linguistic information remains challenging for general\-purpose video encoders\.

These observations also motivate a central goal of our work: to move beyond the direct adaptation of existing general\-purpose Video LLMs and toward a multimodal foundation model with stronger sign representation learning and language modeling capabilities for SLT\. Although the current analysis is based on a relatively limited experimental setting, further exploring Video LLMs with stronger visual backbones, larger model scales, and more extensive sign\-language\-oriented pre\-training remains an important direction for future work\.

Figure 5:Performance comparison between directly fine\-tuning VideoLLaMA2 and the SignRAG\-SFT model\.
### D\.3Why Does Hierarchical Pretraining Benefit Sign MLLMs?

We further analyze why hierarchical pretraining is particularly important when replacing conventional pretrained language models with stronger LLMs\. Empirically, we observe an interesting phenomenon: directly coupling a sign encoder with a powerful LLM can perform substantially worse than using a T5\-based architecture, whereas pretraining the sign encoder before multimodal joint training significantly improves performance\.

We attribute this phenomenon toCross\-Modal Optimization Imbalance\([Wang et al\., 2020](https://arxiv.org/html/2610.11371#bib.bib51);[Huang et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib52);[Peng et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib49)\)\. At the beginning of end\-to\-end multimodal training, the sign encoder is either randomly initialized or only weakly aligned with linguistic semantics, while the LLM already possesses strong language modeling capability and mature linguistic priors\. Therefore, the sign representation

Zs=Esign​\(X\)Z\_\{s\}=E\_\{\\mathrm\{sign\}\}\(X\)\(31\)may still mainly encode low\-level motion patterns, whereas the LLM can already partially fit the target sentence through its pretrained language knowledge\. This imbalance can lead to

Weak Sign Features⏟visual pathway\+Strong Language Prior⏟LLM→Language\-Dominated Optimization\.\\underbrace\{\\text\{Weak Sign Features\}\}\_\{\\text\{visual pathway\}\}\+\\underbrace\{\\text\{Strong Language Prior\}\}\_\{\\text\{LLM\}\}\\rightarrow\\text\{Language\-Dominated Optimization\}\.\(32\)As a result, the model may reduce the generation loss without sufficiently improving the semantic quality of the sign representation\.

There are two main reasons for this behavior\. First, the sign encoder only receives indirect supervision from the final generation objective:

ℒgen→LLM→Projector→Sign​Encoder\.\\mathcal\{L\}\_\{\\mathrm\{gen\}\}\\rightarrow\\mathrm\{LLM\}\\rightarrow\\mathrm\{Projector\}\\rightarrow\\mathrm\{Sign~Encoder\}\.\(33\)Hence, the gradient signal reaching the sign encoder is less direct than explicit sign–text semantic supervision\. Second, the multimodal projector may partially compensate for immature sign representations by mapping them into the LLM latent space, rather than forcing the sign encoder itself to develop linguistically meaningful features\. Consequently, end\-to\-end optimization may successfully reduce the generation loss while still producing weak visual grounding\. In other words,

Lower Generation Loss⇏Better Sign Representation\.\\text\{Lower Generation Loss\}\\not\\Rightarrow\\text\{Better Sign Representation\}\.\(34\)
This issue is less pronounced for T5\-based SLT models\. Compared with modern LLMs, T5\-style language models generally provide weaker linguistic priors and therefore cannot rely as heavily on language\-side shortcuts\. To minimize the translation objective, they must depend more strongly on visual evidence from the sign encoder, resulting in tighter visual\-to\-language coupling\. This explains why a T5\-based model can remain relatively stable even when the sign encoder and language model are trained jointly\.

In contrast, stronger LLMs require more semantically mature modality representations before multimodal optimization\. Our hierarchical pretraining explicitly addresses this requirement\. In the first stage, we perform sign–text alignment:

Esign→Sign–Text AlignmentZssem,E\_\{\\mathrm\{sign\}\}\\xrightarrow\{\\text\{Sign\-\-Text Alignment\}\}Z\_\{s\}^\{\\mathrm\{sem\}\},\(35\)whereZssemZ\_\{s\}^\{\\mathrm\{sem\}\}is encouraged to encode linguistically structured sign semantics rather than only motion patterns\. In the second stage, the aligned sign encoder is integrated with the multimodal projector and LLM:

Zssem→Projector→LLM\.Z\_\{s\}^\{\\mathrm\{sem\}\}\\rightarrow\\mathrm\{Projector\}\\rightarrow\\mathrm\{LLM\}\.\(36\)This decomposition transforms a difficult joint optimization problem into two progressively learned objectives:

Sign Representation Learning→Cross\-Modal Generation\\boxed\{\\text\{Sign Representation Learning\}\\rightarrow\\text\{Cross\-Modal Generation\}\}\(37\)The first stage focuses on*what the sign means*, while the second stage focuses on*how to express the acquired sign semantics in natural language*\.

Overall, these results suggest that the advantage of a stronger LLM can only be fully realized when it is paired with a sufficiently strong modality interface\. More generally, effective Sign MLLM pretraining depends not only on language\-model capacity, but also on the quality of modality representations, cross\-modal alignment, and optimization order:

MLLM Performance∝Representation Quality\+Alignment Quality\+Optimization Strategy\.\\boxed\{\\text\{MLLM Performance\}\\propto\\text\{Representation Quality\}\+\\text\{Alignment Quality\}\+\\text\{Optimization Strategy\}\.\}\(38\)This observation motivates our hierarchical design and explains why directly scaling the language backbone does not necessarily lead to better sign language translation\.

### D\.4Dataset statistics\.

Table 7:Statistics of the pretraining and fine\-tuning datasets\. For YouTube\-ASL, we report both the original dataset size and the number of samples available to us\.StageDatasetSign language\# SamplesPretrainingCSL\-News\([Li et al\., 2025b](https://arxiv.org/html/2610.11371#bib.bib19)\)Chinese722,715YouTube\-ASL\([Uthus et al\., 2023](https://arxiv.org/html/2610.11371#bib.bib21)\)\(original\)American530,161YouTube\-ASL \(available\)American434,953Fine\-tuningHow2Sign\([Duarte et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib24)\)American35,172OpenASL\([Shi et al\., 2022](https://arxiv.org/html/2610.11371#bib.bib25)\)American98,417CSL\-Daily\([Zhou et al\., 2021](https://arxiv.org/html/2610.11371#bib.bib4)\)Chinese20,654PHOENIX\-2014T\([Camgoz et al\., 2018](https://arxiv.org/html/2610.11371#bib.bib2)\)German8,257Table[7](https://arxiv.org/html/2610.11371#A4.T7)summarizes the datasets used for pretraining and fine\-tuning, all of which consist of sentence\-level sign–text pairs\. For YouTube\-ASL, we successfully obtained 434,953 of the 530,161 reported samples, corresponding to 82\.04% of the original dataset\. Some of the provided YouTube links have become invalid over time, preventing access to the corresponding source videos\. We therefore use the available subset for pretraining\.

### D\.5Metric overfitting analysis\.

Table 8:Additional automatic metrics on CSL\-Daily for checking metric overfitting\. SFT and RFT denote SignRAG\-SFT and SignRAG\-RFT; R\-L, B@4, and BSim denote ROUGE\-L, BLEU\-4, and BERTSim\.Since RUG\-RFT optimizes sentence\-level BLEU and ROUGE rewards, we further evaluate SignRAG with chrF\([Popović, 2015](https://arxiv.org/html/2610.11371#bib.bib39)\)and BERTSim, a BERT\-based semantic similarity score computed withbert\-base\-chinese\.111https://huggingface\.co/google\-bert/bert\-base\-chineseAs shown in Table[8](https://arxiv.org/html/2610.11371#A4.T8), SignRAG\-RFT improves over SignRAG\-SFT on ROUGE\-L, BLEU\-4, chrF, and BERTSim\. These complementary metrics suggest that the gains are not limited to direct BLEU/ROUGE\-oriented optimization\.

### D\.6Transferability across Chinese SLT benchmarks\.

To further evaluate whether our pretraining strategy generalizes beyond a specific Chinese SLT dataset, we transfer the pretrained model to CE\-CSL\([Zhu et al\., 2024](https://arxiv.org/html/2610.11371#bib.bib57)\), a continuous sign language benchmark collected under complex environments\. As shown in Table[9](https://arxiv.org/html/2610.11371#A4.T9), ourPretrain→\\rightarrowSFTconsistently outperforms Uni\-Sign and Geo\-Sign across all metrics\. In particular, it achieves 24\.44 BLEU\-4 and 54\.55 ROUGE\-L, surpassing the strongest baselines by 2\.37 and 2\.54 points, respectively\. These results demonstrate that the learned representations can effectively transfer across different Chinese SLT benchmarks, supporting the generalizability of our pretraining strategy\.

Table 9:Comparison on the CE\-CSL benchmark\.We further evaluate the adaptability of different pretrained SLT models on CE\-CSL\. Uni\-Sign and Geo\-Sign are included as representative baselines with publicly available pretrained weights\. The results demonstrate the capability of ourPretrain→\\rightarrowSFTto transfer and adapt to different downstream SLT benchmarks\.
### D\.7Failure Case \(Harmful Reliance on Retrieved Contexts\)\.

Table[10](https://arxiv.org/html/2610.11371#A4.T10)presents a representative failure case illustrating harmful reliance on retrieved contexts\. Although thePretrain→\\rightarrowSFTvariant, which is fine\-tuned using only visual input without retrieval, correctly translates the input asThe weather is nice today, SignRAG\-SFT is misled by the incorrect retrieved context and generatesThe supermarket is holding an event\. After RFT, SignRAG\-RFT recovers the correct translation, indicating that retrieval utility\-guided reinforcement fine\-tuning can reduce harmful reliance on misleading retrieved information\.

Table 10:Qualitative example illustrating harmful reliance on retrieved contexts and the effect of RFT\. Red text highlights misleading or incorrectly generated content\.

## Appendix EEfficiency, Computational Cost, and Retrieval Scalability

In this section, we provide a systematic analysis of the computational overhead, retrieval scalability, memory consumption, and training cost of SignRAG\. Unless otherwise specified, all efficiency experiments are conducted using Qwen3\-8B on four NVIDIA RTX 4090D GPUs\. For a fair comparison, the Non\-RAG baseline \(Pretrain→\\rightarrowSFT\) and SignRAG\-SFT are evaluated using the same hardware, batch size, numerical precision, and decoding strategy\. Since inference latency reported in prior work is often measured under different hardware and implementation settings, we focus on the incremental cost of SignRAG relative to this hardware\-controlled baseline\.

### E\.1Online SLT Inference Efficiency

A key property of SignRAG is that retrieval is performed*offline*\. Specifically, gallery search is conducted once before training or evaluation, and the retrieved results are stored in the data manifests\. Therefore, no full\-gallery search is performed during the deployedmodel\.generatepath\. The additional online inference cost mainly arises from the increased input length introduced by the retrieved textual contexts\.

Table[11](https://arxiv.org/html/2610.11371#A5.T11)reports the online generation efficiency on CSL\-Daily\. Compared with the Non\-RAG baseline \(Pretrain→\\rightarrowSFT\), SignRAG\-SFT increases the average latency from 42\.69 ms/sample to 61\.98 ms/sample, corresponding to an additional 19\.29 ms/sample\. The throughput decreases from 23\.42 to 16\.13 samples/s, while the peak GPU memory increases from 23\.28 to 29\.03 GiB/GPU\. This corresponds to a throughput reduction of approximately 31\.13% and an additional memory consumption of 5\.75 GiB/GPU\. Despite this moderate overhead, SignRAG\-SFT improves BLEU\-4 on CSL\-Daily from 29\.72 to 31\.22\.

Table 11:Online inference efficiency on CSL\-Daily\.Both methods use Qwen3\-8B and four NVIDIA RTX 4090D GPUs under the same inference configuration\. Retrieval itself is performed offline and is therefore excluded from the deployed generation path\.
### E\.2Offline Retrieval Efficiency on CSL\-Daily

We next evaluate the efficiency of the offline retrieval stage\. The retrieval gallery consists of the target\-domain training samples, while each evaluation sample serves as a query\. Table[12](https://arxiv.org/html/2610.11371#A5.T12)reports the retrieval latency and throughput when progressively increasing the gallery size while keeping the same set of 256 queries\.

As the gallery grows from 1K to 18\.4K samples, the average retrieval latency increases from 3\.49 to 23\.88 ms/query\. At the same time, retrieval throughput decreases from 286\.67 to 42\.02 queries/s\. These results show that the retrieval cost increases with gallery size, but remains practical at the scale of CSL\-Daily\.

Table 12:Offline retrieval efficiency on CSL\-Daily with varying gallery sizes\.All settings use the same 256 queries\.For the complete CSL\-Daily test set, retrieving 1,176 queries from the full gallery of 18,401 training samples requires 64\.70 s in total, corresponding to 55\.01 ms/query\. The difference between this end\-to\-end retrieval latency and the 256\-query micro\-benchmark in Table[12](https://arxiv.org/html/2610.11371#A5.T12)is caused by the complete retrieval pipeline, which additionally includes data loading, batching, result aggregation, and manifest construction\. The peak active retrieval memory is 1\.17 GiB per GPU, while the corresponding CPU memory usage is 10\.42 GiB per rank\.

Combining the one\-time offline retrieval cost with SignRAG generation, the complete retrieval\-and\-generation evaluation on CSL\-Daily requires 137\.59 s for 1,176 test samples\. This corresponds to approximately 117\.00 ms/sample and 0\.153 GPU\-hours when accounting for all four GPUs\. Since retrieval results can be pre\-computed and reused, the deployed online inference latency remains the 61\.98 ms/sample reported in Table[11](https://arxiv.org/html/2610.11371#A5.T11)\.

### E\.3Retrieval Scalability on OpenASL

To further evaluate scalability beyond CSL\-Daily, we conduct the same retrieval analysis on OpenASL, whose training gallery contains approximately 96K samples\. Table[13](https://arxiv.org/html/2610.11371#A5.T13)reports the retrieval efficiency under progressively increasing gallery sizes\.

With a gallery of 95,888 samples, the retrieval latency reaches 705\.85 ms/query for the 256\-query benchmark, corresponding to a throughput of 1\.42 queries/s\. For the complete OpenASL test set, retrieving 971 queries from the full 95,888\-sample gallery requires 715\.90 s in total, corresponding to 737\.29 ms/query\. These results demonstrate that token\-level retrieval remains feasible at the approximately 100K\-gallery scale, although its cost grows substantially with gallery size\.

Table 13:Offline retrieval efficiency at different gallery sizes on OpenASL\.The same 256 queries are used for all gallery\-size experiments\.
### E\.4Training Cost

Table[14](https://arxiv.org/html/2610.11371#A5.T14)summarizes the training cost of the complete SignRAG pipeline, with CSL\-News serving as the pre\-training dataset and CSL\-Daily as the downstream fine\-tuning dataset\. The two\-stage pre\-training procedure requires 106\.67 GPU\-hours for Stage I and 277\.50 GPU\-hours for Stage II\. Downstream SignRAG\-SFT requires 24\.04 GPU\-hours, while SignRAG\-RFT requires an additional 5\.78 GPU\-hours\. Overall, the complete training pipeline requires 413\.99 GPU\-hours\.

Table 14:Training cost of SignRAG\.
### E\.5Discussion on Million\-Scale Retrieval

The above experiments demonstrate the practical feasibility of SignRAG on CSL\-Daily and OpenASL, with gallery sizes of approximately 18K and 96K, respectively\. However, token\-level matching becomes increasingly expensive as the gallery grows\. For substantially larger corpora such as BOBSL, which contains approximately 1\.2 million sentences, we do not currently report direct retrieval results\. At present, BOBSL is not available to us, and a reliable evaluation would additionally require data preparation, model pretraining, feature extraction, index construction, and retrieval\-quality validation\. Therefore, without direct experimental evidence, we avoid extrapolating the current measurements to the million\-scale setting\.

A potential direction for improving scalability is*coarse\-to\-fine retrieval*\. For example, global video representations or domain information could first be used to shortlist a candidate pool through approximate nearest\-neighbor search, after which the proposed fine\-grained token\-level matching would be applied only to the shortlisted candidates\. Gallery sharding and batched retrieval could further reduce wall\-clock latency\.

However, coarse\-to\-fine retrieval introduces a non\-trivial accuracy–efficiency trade\-off\. A coarse retrieval stage based on a single global representation may discard samples that would otherwise receive high scores under full token\-level matching\. Therefore, we do not claim that coarse\-to\-fine retrieval can fully preserve the accuracy of exhaustive SignRAG retrieval\. A more comprehensive evaluation at the million\-scale should jointly consider \(i\) retrieval latency, \(ii\) Top\-KKcandidate preservation, \(iii\) retrieval quality, and \(iv\) downstream SLT performance under different candidate\-pool sizes\.

Finally, we emphasize that the primary contribution of SignRAG is not to claim unrestricted scalability to arbitrary million\-scale open\-domain corpora, but rather to extend retrieval\-augmented SLT from settings that rely on gloss annotations to a gloss\-free formulation and to demonstrate its effectiveness on multiple downstream benchmarks, including relatively large\-scale datasets such as How2Sign and OpenASL\. The OpenASL results in Table[13](https://arxiv.org/html/2610.11371#A5.T13)directly characterize the computational behavior of the current implementation at the largest scale evaluated in this work\.

For very large datasets, the limited vocabulary of sign language may also reduce the need for global Top\-KKretrieval\. A fixed subset of the gallery may already provide useful retrieval candidates\. However, this hypothesis still requires empirical validation\. Million\-scale retrieval therefore remains a limitation of the current work and an important direction for future study\.

## Appendix FMore Cases

Due to space limitations in the main paper, we provide additional qualitative examples in Table[15](https://arxiv.org/html/2610.11371#A6.T15)\. These cases offer a more comprehensive view of the translation performance of our method across diverse examples\.

Table 15:Additional qualitative comparisons on CSL\-Daily\. Each example presents the reference translation, top\-3 retrieved texts, and outputs of Geo\-Sign and SignRAG\. English translations are shown in parentheses below the original Chinese sentences\.