@eliebakouch: here is an overview of the architecture of all the major american open weight models lots of similarities: sliding wind…

X AI KOLs Timeline News
open-weight-models model-architecture mixture-of-experts sliding-window mamba scaling

Summary

An overview comparing the architectures of major American open-weight models, highlighting shared design patterns like sliding windows, latent MoE variants, and abundant normalization layers, with a focus on Muse Spark Open's distinctive approach. The analysis notes how differing constraints between American and Chinese open-weight labs lead to divergent architecture and scaling choices.

here is an overview of the architecture of all the major american open weight models lots of similarities: sliding window (except mamba for nvidia), latent moe or a variant from muse, and a LOT of norms everywhere (muse has 9 norms per layer if i counted right!!) let's focus on muse, it's the only model in this list without shared experts. they are also doing different residuals with some form of gate (still single stream), and are more conservative in the hybrid ratio (3:1) and larger sliding window (2k). there are not a lot of details on the training procedure but you can notice there are a bunch of "muP like" multipliers including one for the router with a 1/sqrt(nb of experts) factor 👀 super interesting to see this quite different approach to scaling between american and chinese open weight labs, different constraints lead to different architectures
Original Article
View Cached Full Text

Cached at: 10/17/26, 08:53 AM

here is an overview of the architecture of all the major american open weight models

lots of similarities: sliding window (except mamba for nvidia), latent moe or a variant from muse, and a LOT of norms everywhere (muse has 9 norms per layer if i counted right!!)

let’s focus on muse, it’s the only model in this list without shared experts. they are also doing different residuals with some form of gate (still single stream), and are more conservative in the hybrid ratio (3:1) and larger sliding window (2k). there are not a lot of details on the training procedure but you can notice there are a bunch of “muP like” multipliers including one for the router with a 1/sqrt(nb of experts) factor 👀

super interesting to see this quite different approach to scaling between american and chinese open weight labs, different constraints lead to different architectures

Alexandr Wang (@alexandr_wang): 1/ muse spark open is the most capable US open-weight model to date.

816B total params, 42B active. 1M-token context. natively multimodal. built for coding and agentic work like long-horizon tasks, hard software engineering, multi-step reasoning.

america! 🇺🇲

Similar Articles

@garrytan: Open weight models rising

X AI KOLs Timeline

Garry Tan shares Alexandr Wang's announcement that Muse Spark Open is the most capable US open-weight model to date, featuring 816B total parameters, 42B active, 1M-token context, and native multimodality built for coding and agentic tasks.

@levie: They did it. They made a Muse open weights model, which is a huge step forward for OSS in the US. This is going to furt…

X AI KOLs Timeline

Aaron Levie celebrates Meta's release of the Muse open-weights model, calling it a major step forward for open-source AI in the US that will let enterprises mix frontier and custom post-trained models, boosting token consumption across the stack. The quoted post details Muse Spark Open as the most capable US open-weight model, featuring 816B total/42B active params, 1M-token context, native multimodality, and agentic coding capabilities.