inclusionAI/Ling-3.0-flash-base · Hugging Face

Source: https://huggingface.co/inclusionAI/Ling-3.0-flash-base

🤗Hugging Face| 🤖ModelScope| 🐙OpenRouter

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#introductionIntroduction

We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasinga collection of checkpoints during the training processas following:

These checkpoints correspond to different stages of the training process:

• Pretrained checkpointhave completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training.
• Mid-trained checkpointhave completed mid-training but have not undergone WSM merging (or learning-rate decay) or post-training.
• Merged checkpointshave undergoneWSMmerging (or learning-rate decay) based on the mid-training checkpoints but have not undergone post-training.

These checkpoints are released to support continued pretraining, fine-tuning, and further research. For the post-trained model, please see and seeLing-3.0-tinyandLing-3.0-flash.

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#model-overviewModel Overview

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#key-featuresKey features

• **Highly sparse (1/64) MoE architecture:**512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
• **Native hybrid linear attention:**Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
• **Warmup-Stable and Merge:**We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
• **Scale Seamlessly:**Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.

Model TypeBase (final checkpoint of WSM merging)ArchitectureHybrid-linear MoEParameter ScaleTotoal 124B, Activated 5.1B (Non-emb)Transformer Layers35 KDA + 7 Gated MLA (5:1)Number of Dense Layers2Number of Routed Experts512Number of Shared Experts1Number of Activated Experts8Attention Heads32Hidden Size2560Expert Intermediate Size768Dense Intermediate Size6144Vocabulary Size157,184

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#base-model-evaluationBase Model Evaluation

To systematically assess the capabilities of the base model, we use a self-built comprehensive benchmark suite covering several key domains, including mathematics, coding, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e.,Ling\-3\.0\-flash\-base, is compared below:

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#intended-useIntended Use

Recommended use cases:

• Continued pre-training
• Mid-training
• Supervised fine-tuning for domain adaptation
• Preference optimization and RL post-training Distillation research
• Long-context and MoE systems research

Not recommended as-is for:

• Direct end-user chat deployment
• Safety-critical applications without additional alignment and evaluation
• Production use without post-training and task-specific validation

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#usageUsage

For fine-tuning examples, please refer to ourling-cookbook.

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#faqFAQ

If you have any question, please feel free to add a discussion.

https://huggingface.co/inclusionAI/Ling-3.0-flash-base#licenseLicense

This model is released under theMIT License.