I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens

Reddit r/LocalLLaMA Models

Summary

YOON1v released Apex-2, a from-scratch decoder-only Mixture-of-Experts LLM with 3.87B total / 1.45B active parameters, pretrained on 86.5B tokens and SFT-tuned, with full code, architecture writeup, and benchmark results (HumanEval 43.9) released on Hugging Face and GitHub.

No content available
Original Article
View Cached Full Text

Cached at: 10/04/26, 05:20 PM

YOON1v/Apex-2 · Hugging Face

Source: https://huggingface.co/YOON1v/Apex-2

https://huggingface.co/YOON1v/Apex-2#apex-2APEX-2

A from-scratch Mixture-of-Experts LLM: 3.87B total / 1.45B active parameters, pretrained on 86.5B tokens and instruction-tuned with SFT.

This checkpoint is the SFT release (pretrain → 2-stage SFT). A DPO stage was tried and dropped because it lowered code, math and instruction-following scores (see below).

https://huggingface.co/YOON1v/Apex-2#model-detailsModel details

ArchitectureDecoder-only MoE Transformer, every layer MoE. GQA (16Q/4KV, head_dim 128) + QK-RMSNorm + RoPE (θ=10K) + SwiGLU experts + RMSNorm, weight tyingParameters3,869.1M total · 1,453.2M active per token(1,142.0M non-embedding active), measuredLayers · d_model32 · 2048Experts16 per layer, top-4 routing with renormalized probabilities, expert width 1024Router lossesload-balancing aux loss 0.01 (global-batch statistics) + router z-loss 1e-3Context length4096Vocab151,936 (Qwen3 tokenizer), tied embeddingsWeightsstored asbfloat16(~7.7GB), cast from the float32 training exportPretrain54,250 steps · 86.5B tokens (web · code in 11 languages · math · curated), 1M-token batches; the last 20.4B tokens are an LR decay phase (web 30 / code 35 / math 15 / curated 20); parts trained with DiLoCo on 2× GH200Post-trainingSFT stage 1: 2.21B tokens (general 29 / code 44 / math 27). SFT stage 2: 0.28B tokens × 2 epochs (execution-verified code, competition math, precise instruction following)HF formatWeights map 1:1 ontoQwen3MoeForCausalLM(the training model uses fused expert tensors; the export splits them per expert). Export check: logits relative difference 1.0e-6 and 100% next-token agreement against the training checkpoint

https://huggingface.co/YOON1v/Apex-2#usageUsage

The model uses the ChatML template (<\|im\_start\|\>role\\n\.\.\.<\|im\_end\|\>) shipped intokenizer\_config\.json; generation ends at<\|im\_end\|\>.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("YOON1v/Apex-2", dtype=torch.bfloat16, device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("YOON1v/Apex-2")

messages = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

vLLM loads it directly as well:LLM\(model="YOON1v/Apex\-2", dtype="bfloat16", max\_model\_len=4096\).

https://huggingface.co/YOON1v/Apex-2#benchmarksBenchmarks

Measured by us with vLLM greedy decoding and the chat template (prompt + answer ≤ 4096 tokens). Model-written code was executed in a sandbox.

AreaBenchmarkApex-2 (SFT)BaseCodeHumanEval43.936.6CodeHumanEval+41.532.9CodeMBPP56.354.8CodeMBPP+48.946.3CodeMultiPL-E HumanEval C++36.0—CodeMultiPL-E MBPP C++41.6—CodeLiveCodeBench v5–v63.2—Code└ LiveCodeBench easy (84)11.9—Code└ LiveCodeBench medium (104)1.0—Code└ LiveCodeBench hard (154)0.0—CodeCRUXEval-O8.5—CodeCRUXEval-I3.2—MathGSM8K32.4(0-shot CoT)15.1 (8-shot)MathMATH-500 (0-shot CoT)21.0—Instruction followingIFEval prompt strict44.7—Instruction followingIFEval instruction strict56.6—KnowledgeMMLU (5-shot)28.628.2CommonsenseHellaSwag (acc_norm)62.260.8CommonsenseARC-e (acc_norm)64.166.7CommonsenseARC-c (acc_norm)38.439.7CommonsensePIQA (acc_norm)74.873.9CommonsenseWinoGrande (acc)61.860.1CommonsenseLAMBADA (acc)52.854.5 Compared with similar-size models (official numbers; protocols differ, so this is a rough comparison):

ModelPretraining tokensHumanEvalHumanEval+MBPPMBPP+GSM8KIFEvalMMLUApex-2 (3.87B · 1.45B active)0.087T43.941.556.348.932.444.728.6Apex-1 DPO (1.1B dense)0.02T8.5—5.2—1.9—24.9Qwen2.5-1.5B-Instruct18T61.6—63.2—73.242.550.7Qwen2.5-Coder-1.5B-Instruct5.5T70.766.569.259.4———Qwen3-1.7B (non-thinking)36T—————68.264.4Llama-3.2-1B-Instruct9T————44.459.5*49.3Gemma-3-1B-it2T41.5—35.2—62.880.2*38.8OLMoE-1B-7B (1.3B active)5.1T62.354.4——72.466.4*55.1DeepSeek-Coder-1.3B2T65.960.465.354.8——— DeepSeek-Coder numbers are the Qwen2.5-Coder report’s re-evaluation. * Different IFEval metric: Apex-2 and Qwen report prompt-level strict; the others report an average of metrics or do not specify.

With 1/23 to 1/400 of the pretraining data of these models, Apex-2’s base model matches Qwen2.5-1.5B (18T tokens) on HumanEval+ (32.9 vs 32.9). The big gaps are knowledge (MMLU) and math, which mainly track pretraining scale.

SFT data was 13-gram decontaminated against HumanEval(+), MBPP(+), GSM8K test, MATH, MMLU, ARC, HellaSwag, PIQA, WinoGrande, IFEval and LAMBADA. For LiveCodeBench only the newer v5–v6 problems (after 2024-08) were used, to reduce contamination risk.

**Why no DPO:**DPO on allenai/Dolci-Instruct-DPO (220K pairs) made answers 2.3× longer (320 → 733 tokens on average). Chosen-answer likelihood also fell during training. In evaluation HumanEval+ went 41.5 → 32.3, MBPP+ 48.9 → 37.3, GSM8K 32.4 → 14.3 and IFEval 44.7 → 35.7, so this SFT checkpoint is the release.

https://huggingface.co/YOON1v/Apex-2#known-limitationsKnown limitations

  • English-centric. Other languages, including Korean, barely work (multilingual data was excluded from SFT).
  • Limited world knowledge (MMLU ~29%), so it often states wrong facts with confidence.
  • Weak at competitive programming (LiveCodeBench medium ~1%) and at predicting code execution (CRUXEval).
  • 4096-token context. No tool use, no thinking mode.

https://huggingface.co/YOON1v/Apex-2#training-dataTraining data

  • Pretrain: 40 deduplicated sources, including:- web: FineWeb-Edu, DCLM, FinePDFs - code: StarCoder-family and The Stack-derived code in 11 languages, OpenCoder corpora, synthetic code - math: FineMath, InfiWebMath, Nemotron-CC-Math - curated: Wikipedia, StackExchange, peS2o, Cosmopedia, Gutenberg
  • SFT:- general: allenai/Dolci-Instruct-SFT (tool-use, multilingual and identity subsets removed) - code: OpenCoder opc-sft-stage1/2, nvidia/OpenCodeInstruct (unit-test pass rate ≥ 0.9), bigcode self-oss-instruct, m-a-p/Code-Feedback - math: nvidia/OpenMathInstruct-2, AI-MO/NuminaMath-CoT

Some SFT sources contain synthetic data generated by third-party models (e.g. GPT-4-class, Llama-3.1-405B, Qwen2.5-Coder); check each dataset’s terms for your use case.

https://huggingface.co/YOON1v/Apex-2#licenseLicense

Apache 2.0.

Similar Articles

I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)

Reddit r/LocalLLaMA

An independent researcher trained Apex-3, a 21B-parameter MoE (3.36B active) from scratch on 70B tokens, releasing a stable checkpoint and sharing training-log evidence of emergent behaviors. The clearest finding is a sharp phase transition in in-context learning around 1.3B-1.5B tokens, while arithmetic gains appear continuous in probability despite looking abrupt in exact match.

Aurora2-400M-A20M - a tiny, sparse MoE model.

Reddit r/LocalLLaMA

Aurora2-400M-A20M is a tiny decoder-only sparse MoE language model with ~400M total and 19M active parameters, pretrained from scratch on 100B tokens in about 36 hours on a single RTX Pro 6000 GPU. It features top-1 routing with 32 routed plus 1 shared expert per layer, grouped-query attention, hashed n-gram embeddings, value residuals, and other architectural extras, released as a base (non-instruction-tuned) model on Hugging Face.

Qwen3.5 2B Dense converted to Mixture of Experts

Reddit r/LocalLLaMA

A developer shares their latest dense-to-MoE conversion, turning Qwen3.5-2B into a 1.3B-active MoE (bayliner1980/Qwen3.5-2B-MoE-A1.3B) with improved training data variety that fixed dead experts, plus benchmark and fact-recall evaluations showing modest quality loss after conversion.

Lessons learned while building Apex-2

Reddit r/LocalLLaMA

A developer shares lessons learned while building Apex-2, a small open LLM trained on ~80B tokens for about $2,000, covering GPU bottlenecks, DiLoCo-style distributed training with GH200 instances, and aggressive MinHash deduplication of FineWeb-Edu and DCLM.