Paper page - On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Source: https://huggingface.co/papers/2608.30320 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Qwen3.8-Flash-Next is a sparse mixture-of-experts architecture combining hybrid gated delta-net and sparse attention layers, gated residual branches, and off-accelerator n-gram embeddings to improve efficiency, capability, and training stability.

We describe the architecture and ablations of Qwen3.8-Flash-Next, asparse mixture-of-expertsmodel with 125B parameters, 6B activated per token, and additional 51B parameters ofn-gram embeddingtables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid ofGated DeltaNet(GDN) andglobal attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced byQwen Sparse Attention(QSA), which scores context atmicro-block granularitywith a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call theGated Residual(GR). Capacity is added outside the backbone by a singlen-gram embeddinglayer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and theMuon optimizertogether shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.30320 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.30320 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.30320 in a Space README.md to link it from this page.

Collections including this paper3