Carbon: Decoding the Language of Life

Reddit r/LocalLLaMA Models
dna-foundation-model open-source bioinformatics genomic-model hugging-face efficiency

Summary

Hugging Face released Carbon, a family of open DNA foundation models that matches state-of-the-art performance of Evo2-7B while being 275x faster, using 6-mer tokenization, factorized loss, and curated genomic data.

https://preview.redd.it/rajj11v7j42h1.png?width=1744&format=png&auto=webp&s=72381de22a9bac4b30a59498d549bb09df075df3 Hey, it's loubna from Hugging Face. Very happy to share our latest release: Carbon 🧬, a family of open DNA foundation models. Carbon-3B matches the current SOTA (Evo2-7B) while being 275x faster. We borrowed a lot from how modern LLMs are trained and from our SmolLM work, but DNA isn't language. Genomes are noisy, redundant, and shaped by evolution rather than communication. So we adjusted the recipe: **Tokenizer.** Most genomic models tokenize at the nucleotide level, which blows up sequence length. BPE is the obvious LLM-style fix, but it doesn't behave well on DNA. We use deterministic 6-mer tokens (one token = 6 nucleotides): 6× shorter sequences and cheaper attention. **Training loss.** With 6-mer tokens, cross-entropy scores a prediction that gets 5 of 6 nucleotides right the same as one that's completely wrong. This gets brittle late in training and produces loss spikes. We switch mid-training to a more flexible factorized loss (FNS). **Data.** Genomes are mostly sparse, repetitive background. We curate down to a staged functional DNA + mRNA mixture, with every ratio chosen by ablation. Like mixing a web corpus, but for biology. \- Technical report: [https://github.com/huggingface/carbon/blob/main/tech-report.pdf](https://github.com/huggingface/carbon/blob/main/tech-report.pdf) \- Demo (with a biology primer for our ML friends): [https://huggingface.co/spaces/HuggingFaceBio/carbon-demo](https://huggingface.co/spaces/HuggingFaceBio/carbon-demo) Happy to answer questions in the comments 🤗
Original Article

Similar Articles

@aisolopreneur: Hugging Face just released Carbon-X, an open model you can ask questions about DNA in plain English, and on its own tes…

X AI KOLs Timeline

Hugging Face released Carbon-X, an open DNA language model that answers questions about gene sequences in plain English, scoring 66.7% on DNA benchmarks near GPT-6.1 Sol at roughly 1/1,800 the cost per problem. It runs on a single 80GB GPU, potentially making large-scale variant classification dramatically cheaper for hospitals and researchers, though it is not yet clinically validated.

Carbon-X is an open 14.8B model trained jointly on DNA and text [R]

Reddit r/MachineLearning

Carbon-X is an open 14.8B decoder-only model that jointly processes raw DNA and text via a shared 6-mer token stream, achieving 66.7% average on five DNA benchmarks — nearly matching GPT-6.1 Sol — with Apache 2.0 weights that fit on a single 80GB GPU.