RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Hugging Face Daily Papers Papers
rna-model foundation-model long-context bioinformatics deep-learning generative-model preprint

Summary

RIBOSPAN is a large bidirectional RNA foundation model pretrained on up to 10,240 nucleotides, enabling high-resolution full-transcript modeling and mRNA generation through discrete diffusion.

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlled long-context representation benchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.
Original Article
View Cached Full Text

Cached at: 08/25/26, 08:34 AM

Paper page - RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Source: https://huggingface.co/papers/2608.22849

Abstract

RIBOSPAN is a large bidirectional RNA foundation model pretrained on up to 10,240 nucleotides that enables high-resolution full-transcript modeling, strong long-context representations, and discrete-diffusion-based mRNA generation and redesign.

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existingRNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectionalRNA foundation modelnatively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines densebidirectional self-attention,single-nucleotide tokenization, and attention-isolatedsequence packingto enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlledlong-context representationbenchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-timeYaRN scalingrecovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioneddiscrete-diffusionframework for full-length mRNA generation and redesign, includingsynonymous-codon diffusionfor protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2608\.22849

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper4

#### SII-GAIR-NLP/RIBOSPAN-10K-15 Fill-Mask• Updatedabout 4 hours ago • 3 #### SII-GAIR-NLP/RIBOSPAN-FM Fill-Mask• Updatedabout 5 hours ago • 2 #### SII-GAIR-NLP/RIBOSPAN-1K-15 Fill-Mask• Updatedabout 5 hours ago #### SII-GAIR-NLP/RIBOSPAN-1K-40 Fill-Mask• Updatedabout 5 hours ago

Datasets citing this paper1

#### SII-GAIR-NLP/RIBOSPAN-FM-Corpus Viewer• Updatedabout 3 hours ago • 180k • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.22849 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

Training mRNA Language Models Across 25 Species for $165

Hugging Face Blog

This article details the development of CodonRoBERTa, a language model trained across 25 species for mRNA codon optimization, highlighting a cost-effective pipeline that includes protein folding and sequence design.

LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling

arXiv cs.CL

LDARNet is a 120M-parameter hierarchical genomic foundation model that introduces learnable adaptive tokenization (inspired by H-Net's dynamic chunking) for masked language modeling on DNA sequences. It achieves state-of-the-art results on 5 histone modification tasks and outperforms models up to 20× larger on several genomic benchmarks, with learned token boundaries aligning with biological features like promoter motifs and splice junctions.

ProtSent: Protein Sentence Transformers

arXiv cs.LG

This article introduces ProtSent, a contrastive fine-tuning framework for protein language models that improves embedding quality for downstream tasks like remote homology detection and structural retrieval.