Qwen 4 architecture: What do we know?
Summary
The author speculates on the possible architecture of Qwen 4, discussing potential designs like embedding-offloaded linear attention or multi-head latent attention hybrids based on reverse engineering Deepseek.
Similar Articles
Qwen4's architecture is here early, firing 6B parameters out of 125B (3 minute read)
Alibaba's Qwen team released Qwen3.8-Flash-Next, an open-weight preview of the Qwen4 architecture that activates only 6B parameters out of 125B to reduce inference costs and address hardware limitations.
Qwen/Qwen3.8-Flash-Next
Release of Qwen3.8-Flash-Next, an open-weight AI model introducing architectural innovations like Hybrid Attention with QSA and Gated Residual for improved efficiency and scalability, previewing the future Qwen4 architecture.
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
The paper presents Qwen3.8-Flash-Next, a sparse mixture-of-experts AI model that enhances efficiency, capability, and training stability through architectural innovations like hybrid attention and n-gram embeddings.
Qwen4!
Qwen4 has been released, with hints of a 120B parameter model surfacing via a GitHub PR in the qwen-code repository.
I study how AI organizes meaning internally. Here's what Qwen 2.5 looks like before it starts thinking.
The author visualizes Qwen 2.5 7B's input embedding space as an 'Embedding Sea' topographic map and discusses interpretability tools like logit lens and Jacobian lens. They highlight a trend of AI models optimizing for code over conversation and propose building a creativity-focused AI model with introspection capability.