Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

arXiv cs.CL Papers
language-encoders icelandic nlp multilingual transfer-learning pretraining benchmarking

Summary

Researchers train ModernBERT-based Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, showing that a 1B model trained from scratch beats IceBERT on all eleven tasks, and that warm-starting from an English Ettin encoder yields the best scores across every benchmark while highlighting the importance of vocabulary transplantation.

arXiv:2610.36230v1 Announce Type: new Abstract: IceBERT, the reference encoder for Icelandic, found that a cleaned web crawl is enough to train a strong model and that starting from a multilingual model can help on some tasks. We revisit both findings with the ModernBERT architecture. We train Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, and rerun IceBERT's benchmarks and five newer tasks for 30 encoders under one protocol with confidence intervals. Our 1B model trained from scratch, to our knowledge the first monolingual Icelandic encoder of that size, is significantly better than IceBERT on all eleven tasks; warm-started from the English Ettin encoder of the same shape, it has the best score of all encoders on every task we rank. Because our model shapes match the English Ettin encoders and the multilingual mmBERT, we can also test the warm start under controlled conditions, at six model sizes and four token budgets. With our vocabulary transplanted, a warm start was never significantly worse than scratch training on the same Icelandic text, and from base size up it was not significantly worse on four tasks than scratch training on 4 times the text (8 times at base and 1B). As models grow, most of the transfer comes from the source model's transformer layers, so even an English source helps. Keeping the source vocabulary instead cost 5 F1 on question answering and nearly three times the compute.
Original Article

Similar Articles

m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder

arXiv cs.CL

This paper introduces m3BERT, a multilingual bidirectional encoder with a novel pretraining strategy that jointly optimizes representations across transformer layers and multiple embedding dimensions, enabling a single model to be adapted to varied resource constraints. It significantly outperforms state-of-the-art models on the Bing-Click industrial retrieval dataset.

UltraBERT: Beyond the Limit of BERT Pre-Training

Hugging Face Daily Papers

UltraBERT revisits bidirectional encoder pre-training design choices, showing performance saturation beyond a few trillion tokens, masking-ratio trade-offs between classification and retrieval, and proposes a dual-LM-head distillation approach achieving state-of-the-art on GLUE (91.0) and BEIR (47.7).

Introducing the Ettin Reranker Family

Reddit r/LocalLLaMA

Introducing the Ettin Reranker family: six new state-of-the-art CrossEncoder rerankers at various sizes, built on ModernBERT encoders, with open-source data and training recipe.