Tag
Aleph Alpha describes how they scaled pre-training of their 30B-A3B MoE model (revealed to be Kolibri Origin) from 16 to 512 NVIDIA B200 GPUs using a hierarchical hyperparameter search approach, achieving 35.3% MFU and near-linear scaling just 6% below ideal, trained with a custom fork of torchtitan over InfiniBand/NVLink.
Aleph Alpha's Kolibri 78B model is now available to try for free online through a chat interface hosted by Tesseracted, highlighted via the r/euainews community.
Aleph Alpha has released a technical report PDF for Kolibri, covering the model's architecture, training methodology, and evaluation results.
Aleph Alpha released Kolibri, an open-weight sovereign German/English Mixture-of-Experts LLM with 78B total parameters (~3.5B active per token), trained from scratch on German and Finnish infrastructure under Apache 2.0, with weights on Hugging Face and a 262k-token native context.
Aleph Alpha describes Savanna, their model factory that turns the entire training pipeline into code, enabling hermetic, one-click training runs and better team coordination.