Aurora2-400M-A20M - a tiny, sparse MoE model.
Summary
Aurora2-400M-A20M is a tiny decoder-only sparse MoE language model with ~400M total and 19M active parameters, pretrained from scratch on 100B tokens in about 36 hours on a single RTX Pro 6000 GPU. It features top-1 routing with 32 routed plus 1 shared expert per layer, grouped-query attention, hashed n-gram embeddings, value residuals, and other architectural extras, released as a base (non-instruction-tuned) model on Hugging Face.
View Cached Full Text
Cached at: 11/19/26, 03:43 AM
AuroraAI-Research/Aurora2-400M-A20M · Hugging Face
Source: https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#aurora2-400m-a20mAurora2-400M-A20M
Aurora2-400M-A20M is a decoder-only language model with approximately 400M total parameters and 19M active parameters per token. It uses a top-1 Mixture-of-Experts (MoE) architecture with 32 routed experts and one shared expert per layer.
The model was pretrained from scratch on 100B tokens using a single GPU. It is a base model, not an instruction-tuned assistant: there was no supervised instruction-tuning or RLHF stage.
PropertyValueTotal parameters399,597,454Active parameters per token19,050,382Experts32 routed + 1 shared per layerLayers12Hidden size192Context length2,048 tokensVocabulary16,384Training tokens100BTraining precisionBF16 compute, FP32 master weightsHardwareSingle RTX Pro 6000, approximately 36 hours
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#architectureArchitecture
Aurora2 uses a relatively narrow attention stack paired with large, sparse feed-forward layers. Each token is routed to one of 32 experts, alongside the always-active shared expert.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#core-configurationCore configuration
ComponentConfigurationHidden size192Layers12Attention heads6 query / 2 KVAttention typeGrouped-query attention (3:1)Head dimension32Positional encodingRoPE, base 10,000NormalizationPre-RMSNorm, FP32 QK normalizationAttentionFlexAttention, causal and per-document maskingRouted experts32 per layer, SwiGLU, FFN dimension 1,776Shared expert1 per layer, SwiGLU, FFN dimension 256RoutingFP32 softmax, top-1, droplessOutput headTied token embeddingsLogit soft-capping30 The router scales each selected expert’s output by its routing probability. There is no expert capacity factor or token dropping.
Router training includes a Switch-style load-balancing loss (coefficient 0.01) and router z-loss (coefficient 0.001).
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#additional-componentsAdditional components
- **Hashed n-gram embeddings:**Causal hashes of the last 2, 3 and 4 token IDs index three separate 16,384-bucket embedding tables. Their outputs are concatenated, projected to the hidden dimension and added to the token embeddings through a learned per-channel gate.
- **Embedding skip connections:**Every block combines its input with the original embedding using learned coefficients, initialized to
\[1, 0\]. - **Value residuals:**Layers after the first mix their value tensors with the first layer’s values, using learned coefficients initialized to
\[0\.5, 0\.5\]. - **Per-head attention gates:**Each attention head has a learned output gate derived from the first 12 channels of the normalized block input. It is initialized to produce a multiplier of 1.
- **Learned router bias:**Each expert has a trainable bias added to its routing logits.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#tokenizerTokenizer
Aurora2 uses a byte-level BPE tokenizer trained from scratch.
- Vocabulary size: 16,384.
- Every digit is an individual token (
"123"becomes three tokens). - Special tokens:-
<\|bos\|\>: ID 0 -<\|eos\|\>: ID 1 -<\|pad\|\>: ID 2 - The tokenizer automatically prepends
<\|bos\|\>to encoded text.
During training, documents were packed together, with each document starting with a BOS token. No EOS was inserted between documents. BOS tokens also served as document boundaries for attention masking.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#training-dataTraining data
Training data was streamed from the Hugging Face Hub, tokenized on the fly and packed into sequences of 2,048 tokens. The tokenizer was trained on approximately 1B characters from the same data mixture.
Only training splits were used. ClimbMix shard 6542 was reserved for validation; benchmark evaluation splits were not used for training.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#staged-data-mixtureStaged data mixture
The mixture was adjusted throughout training. Math and QA/chat sources received greater weight during the final decay phase, while ClimbMix remained the majority source.
Training stageClimbMixFineMathWikipediaSmolTalkOpenMathNumina0–60B tokens89.3%5.4%2.7%1.25%0.75%0.6%60–80B tokens85.8%6.0%3.0%2.5%1.5%1.2%80–100B tokens75.3%7.8%3.9%6.25%3.75%3.0%
Instruction-style datasets were converted into plain-text formats, includingQuestion: \.\.\. Answer: \.\.\.,Problem: \.\.\. Solution: \.\.\.andUser: \.\.\. Assistant: \.\.\.. These examples were part of pretraining; they do not constitute an instruction-tuning stage.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#training-procedureTraining procedure
HyperparameterValueTotal training tokens100BSequences per step128Sequence length2,048Tokens per step262,144Approximate optimizer steps381,000Learning-rate scheduleWarmup-Stable-Decay (WSD)Warmup1B tokens, linearStable phase1B–80B tokensDecayCosine decay over the final 20B tokensPeak learning rate1.5e-3Router learning-rate multiplier0.5OptimizerMuon + AdamWGradient clipping1.0Random seed1337
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#optimizer-configurationOptimizer configuration
Muon was used for the attention projections and shared-expert matrices. It used momentum 0.95, five Newton-Schulz iterations and weight decay 0.1.
AdamW handled the token and n-gram embeddings, router parameters, normalization parameters, scalar/gating parameters and routed experts.
AdamW used betas\(0\.9, 0\.95\), epsilon1e\-8and weight decay0\.1. Embeddings used weight decay0\.01, while norms, gates and scalar parameters had no weight decay.
Muon was deliberately not used on the routed experts because it negatively affected routing balance and training speed in experiments.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#evaluationEvaluation
The following benchmark results were obtained for the 100B-token checkpoint.
Scores are shown as percentages.accis standard accuracy, whileacc\_normis normalized accuracy.
Benchmarkaccacc_normHellaSwag32.8238.36ARC-Easy51.1848.11ARC-Challenge22.9526.11ARC37.0737.11PIQA67.4166.92ArithMark337.3037.40 Intelligence Index (acc_norm): 21.52
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#quick-startQuick start
The repository includes a native Transformers implementation with custom model code. You needtrust\_remote\_code=Trueto load it.
Install the dependencies:
pip install -U transformers torch accelerate safetensors
Load the FP16 model:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AuroraAI-Research/Aurora2-400M-A20M"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
subfolder="fp16",
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
).eval()
# <|bos|> (id 0) is prepended automatically
inputs = tokenizer("The Capital of France is", return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.5,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The repository contains three precision variants:
DirectoryWeightsRootFP32bf16/BF16fp16/FP16
The tokenizer is stored in the repository root, so it does not need a subfolder argument.
**Hardware note:**The inference implementation uses BF16 activations on CUDA, even when the stored weights are FP16. A CUDA GPU with BF16 support is therefore recommended.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#inference-notesInference notes
- Use batch size 1 or unpadded batches. The inference implementation does not reproduce the training-time per-document attention mask.
- Context length is 2,048 tokens.
- Beam search is not supported.
- Generation stops when the model produces BOS or EOS.
- The model is intended for causal text generation, not instruction-following.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#intended-use-and-limitationsIntended use and limitations
Aurora2 is primarily intended for research into small language models, sparse Mixture-of-Experts architectures, routing strategies, optimization and training efficiency. It can also serve as a starting point for further fine-tuning.
A few limitations are worth keeping in mind:
- **Base model:**It has not undergone instruction tuning or preference optimization. It is not designed to behave like a conversational assistant.
- **Limited active capacity:**Only about 19M parameters are active per token. Its factual knowledge, reasoning and long-range coherence are limited compared with substantially larger models.
- **English-focused:**Both the tokenizer and training mixture are primarily English-oriented.
- **Short context:**The model was trained with a 2,048-token context length.
- **Training data:**Web-scale datasets can contain inaccuracies, biases and offensive material. Outputs may reflect these issues.
- **Downstream use:**The model should not be used for decisions with real-world consequences. Check the licenses of individual training datasets before downstream use.
https://huggingface.co/AuroraAI-Research/Aurora2-400M-A20M#citationCitation
@misc{aurora2_400m_a20m,
title = {Aurora2-400M-A20M: A Small Sparse Mixture-of-Experts Language Model},
author = {AuroraAI-Research},
year = {2026},
note = {Pretrained on 100B tokens on a single GPU}
}
Similar Articles
Aurora-80K releases! A modern tiny language model.
Aurora-80K is a modern tiny language model with 80 thousand parameters, trained on 40M tokens and released on Hugging Face with benchmark results.
AuroraIMG2-50M-Preview Releases!
AuroraAI-Research 发布了 AuroraIMG2-50M-Preview,一个基于 5040 万参数流匹配扩散 Transformer 的 256×256 文生图模型,搭配 1260 万参数文本编码器,全部从零训练仅需单卡 RTX PRO 6000 上 10 小时完成。
I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens
YOON1v released Apex-2, a from-scratch decoder-only Mixture-of-Experts LLM with 3.87B total / 1.45B active parameters, pretrained on 86.5B tokens and SFT-tuned, with full code, architecture writeup, and benchmark results (HumanEval 43.9) released on Hugging Face and GitHub.
Multilingual Tiny (3.7B) Reasoning MoE pretrained from scratch on a consumer-grade GPU
A multilingual 3.7B parameter reasoning MoE model has been pretrained from scratch on a consumer-grade GPU over several months, with support for 13 languages and available on Hugging Face.
@_akhaliq: A.X K2 just dropped on Hugging Face Large-Scale Sparse MoE (688B / 33B Active) https://huggingface.co/skt/A.X-K2
SKT released A.X K2, a 688B-parameter sparse MoE language model with 33B active parameters, natively trained in FP8 and featuring Think/Non-Think reasoning modes, on Hugging Face.