@nrehiew_: For the visual learners
Summary
A tweet describes a large mixture-of-experts model with 975B total parameters (41B active) trained on 45T tokens of multimodal data, featuring 6 routed experts and 2 shared experts, with comparisons to DeepSeek-V3.
View Cached Full Text
Cached at: 07/16/26, 04:05 AM
For the visual learners https://t.co/SNNv7jukLt
wh (@nrehiew_): On the MoE side
975B 41B active with 45T tokens of multimodal data About 10% more active params than Dsv3 and 3x the number of tokens.
Dsv3 but with 6 routed experts, 256 but 2 shared experts. 2 shared experts is weird. Most ppl do 1 or 0. This also means increased
Similar Articles
DeepSeek-V3 Technical Report
DeepSeek-V3 is a parameter-efficient Mixture-of-Experts language model with 671B total parameters, achieving strong performance comparable to leading closed-source models while requiring only 2.788M H800 GPU hours for training.
deepseek-ai/DeepSeek-V4-Flash-DSpark
DeepSeek releases V4 series of Mixture-of-Experts language models (Pro 1.6T/49B activated, Flash 284B/13B activated) supporting one-million-token context with hybrid attention and speculative decoding, claiming best open-source model performance.
@modal: DeepSeek-V4-Flash has 284B total parameters with 13B active per token. Combined with a hybrid compressed attention mech…
DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.
deepseek-ai/DeepSeek-V4-Pro
DeepSeek releases V4-Pro and V4-Flash, Mixture-of-Experts models supporting million-token context with hybrid attention and Muon optimizer.
tencent/Hy3
Tencent released Hy3, a 295B-parameter Mixture-of-Experts model with 21B active parameters, under Apache 2.0 license, outperforming similar-size models and rivaling larger open-source models with 2-5x parameters.