@nrehiew_: For the visual learners
Summary
A tweet describes a large mixture-of-experts model with 975B total parameters (41B active) trained on 45T tokens of multimodal data, featuring 6 routed experts and 2 shared experts, with comparisons to DeepSeek-V3.
View Cached Full Text
Cached at: 07/16/26, 04:05 AM
For the visual learners https://t.co/SNNv7jukLt
wh (@nrehiew_): On the MoE side
975B 41B active with 45T tokens of multimodal data About 10% more active params than Dsv3 and 3x the number of tokens.
Dsv3 but with 6 routed experts, 256 but 2 shared experts. 2 shared experts is weird. Most ppl do 1 or 0. This also means increased
Similar Articles
@ClementDelangue: Totally agree it deserves more upvotes! Here is a link to it if you and others want to upvote it (I don’t think you upv…
DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters and 1M-token context that achieves major KV cache compression gains via CSA2 and FP4 caching, cutting HBM footprint to ~890 bytes per token while boosting agentic performance.
DeepSeek-V3 Technical Report
DeepSeek-V3 is a parameter-efficient Mixture-of-Experts language model with 671B total parameters, achieving strong performance comparable to leading closed-source models while requiring only 2.788M H800 GPU hours for training.
deepseek-ai/DeepSeek-V4-Flash-DSpark
DeepSeek releases V4 series of Mixture-of-Experts language models (Pro 1.6T/49B activated, Flash 284B/13B activated) supporting one-million-token context with hybrid attention and speculative decoding, claiming best open-source model performance.
@modal: DeepSeek-V4-Flash has 284B total parameters with 13B active per token. Combined with a hybrid compressed attention mech…
DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.
@MaximeRivest: 4B at a time, we could get to 10T-parameter models: a true mixture of experts. Jev showed us we can route pretty decent…
Maxime Rivest argues that 4B-parameter expert models could be combined into a true mixture-of-experts reaching 10T total parameters, enabling frontier-level performance at negligible cost on edge hardware, as small models fine-tuned on few synthetic examples already rival frontier systems.