The WeLM team from WeChat has revealed at least two model scales in this paper: 80B and 617B. Each scale includes a standard version and an HD4 version, namely WeLM-80B, WeLM-HD4-80B, WeLM-617B, and WeLM-HD4-617B.

Among these, WeLM-617B is the flagship base model with a total of 617B parameters, employing a MoE architecture where only part of the parameters are activated during inference. WeLM-HD4-617B introduces Hidden Decoding on the same 617B Transformer backbone and is currently the most noteworthy version. The paper shows that the HD4 version outperforms its corresponding standard version across nine common benchmarks.

The WeChat model team explored a scaling path distinct from simply stacking parameter counts: instead of increasing the number of Transformer layers or their width, each token is expanded into four hidden streams, allocating more internal computation per token. This allows the model’s capability to continue improving while keeping the backbone network largely unchanged. This is the core idea of Hidden Decoding.

“Good enough is fine, but it must be useful.” Note that this is not a benchmark score card for WeLM.

https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/…

Hidden Decoding at Scale: Scaling Latent Space Computation for Large Frontier Models | WeLM Blog

Source: https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/ Table of Contents- Frontier-scale MoE Results (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#frontier-scale-moe-results)

• Method Review and New Designs (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#method-review-and-new-designs)- Multi-stream Expansion and Training Objective (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#multi-stream-expansion-and-training-objective) - Stream-Factorized Attention (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#stream-factorized-attention) - KV Retention and KV-Mirror Acceleration (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#kv-retention-and-kv-mirror-acceleration)
• Computational Cost (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#computational-cost)- Training Cost Near-Linear (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#training-cost-near-linear) - Inference Throughput (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#inference-throughput)
• Ablation and Interpretability (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#ablation-and-interpretability)- Attention Combination Ablation (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#attention-combination-ablation) - KV Retention Ablation (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#kv-retention-ablation) - Stream and LM-head Probing (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#stream-and-lm-head-probing)
• Conclusion and Outlook (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#conclusion-and-outlook)

In the previous blog post (https://welm.weixin.qq.com/posts/hidden_decoding/), we introduced the core idea of Hidden Decoding: without increasing the main parameters of the Transformer, the sequence length is expanded by n times by replicating multiple copies of the Vocab Embedding, giving each token n times the effective computation in a single forward pass. We validated the loss reduction and benchmark improvements at multiple scales including 6B, 8B, and 80B. This article is the advanced sequel: we push Hidden Decoding to frontier scale (100B+ MoE), train WeLM-HD4-80B and WeLM-HD4-617B (n=4), comprehensively surpass their respective autoregressive baselines in fully controlled experiments, and provide the key engineering designs that make this method truly trainable and serviceable on ultra-large models. The formal paper for this work can be found at Tencent/Sequential-Hidden-Decoding (https://github.com/Tencent/Sequential-Hidden-Decoding/blob/main/paper/hidden_decoding_at_scale.pdf).

Here are two core conclusions from this article:

1. Frontier-scale training expansion: To the best of our knowledge, this is the first instance of sequence-length scaling achieved at the 100B+ MoE scale via continued pre-training (CPT) – a scale previously difficult to reach for deep looped or length-expansion methods.
2. Stream-Factorized Attention makes expansion trainable on large models: Most layers only attend within the same stream, and only a few layers perform cross-stream mixing. This reduces the attention overhead from expansion from O(n^2) to nearly linear, enabling Hidden Decoding to be deployed at the 100B+ MoE scale with acceptable training and serving costs.

Hidden Decoding illustration and improvements on frontier modelsFigure 1: Left: Hidden Decoding expands each token into multiple streams along the sequence (expansion factor shown is 2). A single forward pass processes all streams; loss is computed only on the last stream (E_2/h_2) of each token, with preceding streams (E_1/h_1) serving as latent space intermediate computation without loss. Right: Improvement (\Delta) of WeLM-HD4-80B and WeLM-HD4-617B over their matched autoregressive baselines (WeLM-80B / WeLM-617B) under the same early SFT-only post-training.

Frontier-scale MoE Results

To test whether Hidden Decoding can strengthen our largest deployed models, we compare two variants for each WeLM MoE checkpoint – everything identical except for Hidden Decoding: WeLM-80B (3B active) vs. WeLM-HD4-80B, WeLM-617B (23B active) vs. WeLM-HD4-617B. Both are then adapted using the same early SFT-only recipe (short SFT, no RL), making this a single-variable ablation of the Hidden Decoding method.

Table 1: Early SFT-only results. Each WeLM-HD4 model (highlighted) is compared with its matched autoregressive counterpart under the same short SFT recipe (no RL); HD4 denotes Hidden Decoding with n=4, keeping active Transformer parameters per token unchanged (3B for 80B, 23B for 617B). Kimi K2.6 is listed as an absolute score reference for a mainstream frontier model. All numbers in this table are self-tested. *FrontierMath uses public Tiers 1–4 example problems (epoch.ai/frontiermath/tiers-1-4/benchmark-problems (https://epoch.ai/frontiermath/tiers-1-4/benchmark-problems)). †HLE is no-tools, text-only setting. ‡HMMT is the weighted average of three test sets from February 2025, November 2025, and February 2026.

Hidden Decoding improves all nine shared benchmarks at both scales. The largest gains appear on difficult math and science tasks: WeLM-HD4-80B achieves +4.2 on SciCode, +4.0 on PHYBench, +3.2 on FrontierMath; WeLM-HD4-617B achieves +2.1 on GPQA Diamond, +1.8 on HLE. We also list test results for Kimi K2.6 (1T total parameters, 32B active) under the same test setting as an absolute score reference for a mainstream frontier model (Moonshot AI 2026 (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#ref-kimi2026k26)): WeLM-HD4-617B scores higher on GPQA Diamond, PHYBench, MathArena Apex, HMMT, and SciCode. These results indicate that Hidden Decoding can scale to frontier-scale MoE and improve it without adding backbone parameters – a scale regime where looped models struggle.

We place the WeLM-HD4-617B-Base checkpoint in the context of external base model references. We mainly focus on the “autoregressive vs. HD” comparison under the same backbone, data, and training configuration.

Table 2: Base model reference. The WeLM columns use our base model evaluation protocol before instruction tuning or RL; the matched WeLM-617B-Base vs. WeLM-HD4-617B-Base gives a controlled “autoregressive vs. HD” comparison. External numbers are taken from the Qwen3.5 report’s base model table (Qwen Team 2026 (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#ref-qwen2026qwen35)), provided as absolute score references.

Even before post-training, Hidden Decoding brings gains at both scales: full-suite average of +0.99 on 80B and +1.03 on 617B, with larger improvements on harder reasoning and knowledge tasks like SuperGPQA and MMLU-Pro, and these gains are maintained after the same light post-training.

Method Review and New Designs

Multi-stream Expansion and Training Objective

Given an input sequence X=(x_1,\dots,x_L) and n embedding tables E_1,\dots,E_n, we place the k-th stream representation of the i-th token at physical position t=(i-1)n+k, forming an expanded sequence of length nL: S_t = E_k(x_i), \quad 1 \le i \le L,\ 1 \le k \le n. This expanded sequence is fed into the same Transformer under standard causal attention and continuous RoPE positions. During training, only the last stream E_n(x_i) of each token is supervised: from its hidden state, the shared LM Head predicts x_{i+1}; the first n-1 streams receive no direct loss, acting as “latent space intermediate computation” that progressively refines representations. For details on the advantages of “supervising only the last stream” over all-token loss / sum, and the cyclic replication initialization’s progressive expansion 1\to2\to4\to8, see the previous blog post (https://welm.weixin.qq.com/posts/hidden_decoding/).

Stream-Factorized Attention

Retaining full cross-stream attention in all layers would cause attention cost to grow as O(n^2), making training infeasible for large models with long sequences. To address this, we propose Stream-Factorized Attention: the majority of layers attend only within the same stream, with only a few layers performing cross-stream mixing.

For a position t in the expanded sequence, define its token index and stream index: i(t)=\lceil t/n \rceil, \quad s(t)=(t-1)\bmod n + 1. Accordingly, intra-stream layers only attend to earlier positions within the same stream, costing O(nL^2); cross-stream layers additionally attend across streams, following the original attention pattern of the base model (sliding window, or full attention if the base model uses it).

Three attention masks for Stream-Factorized AttentionFigure 2: Three attention masks for Stream-Factorized Attention. (a) Full cross-stream layers use the standard causal (lower triangular) mask; (b) local cross-stream layers use a causal sliding window mask; (c) intra-stream layers are causal within each stream and mutually invisible between streams. Most layers are (c), with only a few layers enabling (a) or (b) for cross-stream mixing.

By design, the vast majority of layers are intra-stream, and only a small fraction enable cross-stream attention. Importantly, no new attention layers are added: we enable cross-stream attention on all local attention layers and a small subset of full attention layers. Since dense attention over the full nL sequence is avoided, the additional attention cost drops from n^2 to nearly linear in n – this is the key to making Hidden Decoding trainable at the 100B+ MoE scale. Furthermore, this layout where most layers maintain a single-stream causal path with only a few layers introducing cross-stream perturbations makes CPT closer to the starting autoregressive checkpoint, allowing for a smoother start.

KV Retention and KV-Mirror Acceleration

Retaining intermediate stream KV: Hidden Decoding by default retains a separate KV cache for each stream, allowing later positions to leverage computations from earlier intermediate streams. We ablate this in the Ablation and Interpretability section. Additionally, we introduce stream parallelism to shard intra-stream KV across different ranks, reducing KV storage and memory access pressure.

KV-mirror acceleration for training: Our WeLM backbone employs a KV-mirror design (WeChat AI 2026 (https://welm.weixin.qq.com/posts/hidden_decoding_at_scale/#ref-welm2026moe)) – each layer in the later part no longer computes K/V from its own layer input but instead reuses hidden states from a layer in the earlier part (layers are paired in a U-shape, with the last 1/3 layers mirroring the first 1/3). Since the K/V of mirrored layers depends only on hidden states from earlier layers, and only the last stream is supervised (intermediate streams contribute only through their K/V), in mirrored layers this context is already fixed by the earlier part, so intermediate streams no longer need to be processed. Therefore, we only pass the last stream through the mirrored layers, while the earlier layers still process all n streams. Under the 80B, 32k training setting, this optimization reduces the single batch time from 15s to 12s (about 20% improvement, 1.25×). This saving is specific to WeLM and stacks on top of the near-linear cost already provided by Stream-Factorized Attention.

Computational Cost

Training Cost Near-Linear

Expanding each token into n streams turns a sequence of length L into nL: non-attention computation scales roughly linearly with the number of positions, while dense attention would balloon by O(n^2). Stream-Factorized Attention, by keeping most full attention layers intra-stream, avoids this expansion, making attention cost close to linear in n.

Training costFigure 3: Single batch training time for HD relative to the unexpanded baseline when n=4: 5.1× for 80B (256k→1M), 4.4× for 617B (32k→128k), close to the linear reference (n=4\times), far below n^2=16\times for dense attention.

Empirical measurements confirm the analysis: the 4\times effective sequence costs 5.1× on 80B and 4.4× on 617B, close to the 4\times linear reference and far below 16\times for dense attention. This near-linear cost is precisely what makes WeLM-HD4-617B trainable.

More importantly, HD4 is not trained from scratch but uses incremental continued training on existing autoregressive base checkpoints: it only continues training after the HD starting point, without needing to rerun the full 8k pre-training phase. Therefore, counting training tokens after the HD starting point, the total training cost of HD4 is far lower than training the full autoregressive base from scratch – 1.07T vs. 20.39T for 80B, 0.90T vs. 17.06T for 617B.

Table 5: Training token budget for the autoregressive base path and HD4 continued training window. Autoregressive rows count the full base model training path; HD4 rows count only training after the HD starting point.

Inference Throughput

One major advantage of Hidden Decoding over looped models is that its additional computation is parallel: looped models repeatedly run the backbone, each step waiting for the previous one, so the extra cost cannot be hidden; Hidden Decoding distributes the computation across n streams, processing them in parallel in a single forward pass. Under large batch sizes, decoding is compute-bound, and extra computation reduces throughput (as with any method that adds computation); but under small batch sizes, decoding is bandwidth-bound with substantial idle compute, and the parallel streams can utilize this idle capacity.

WeLM-HD4-80B serving throughput matrixFigure 4: Serving throughput matrix for WeLM-HD4-80B. Each cell shows the decoding completion throughput (%) of Hidden Decoding relative to the matched WeLM-80B baseline (with stream expansion disabled), under the same input length bucket and batch size, averaged over 5 runs (small text shows sample standard deviation in pp); both run on the same 8×H20, excluding prefill.

Ablation and Interpretability

Hidden Decoding relies on an unusual choice: only the last stream is trained to predict the next token, with preceding streams receiving no direct loss. The main results show this choice improves accuracy, but accuracy alone doesn’t explain what the intermediate streams actually do – the gain could simply come from a longer sequence, more prediction targets, or explicit aggregation of stream outputs. We use ablations and probes to distinguish these possibilities.

Attention Combination Ablation

Stream-Factorized Attention reduces cost by limiting expensive full cross-stream mixing. Therefore, we need to test how much full mixing is actually needed for good accuracy. On a 21B MoE (27 layers), we compare three variants of Hidden Decoding that differ only in “how many full cross-stream layers are used” (all 27 layers, 4 layers, 1 layer), with the remaining layers distributed between local and intra-stream, and a no-expansion autoregressive baseline as reference.

Table 3: Attention combination ablation (21B MoE). All Hidden Decoding variants substantially exceed the no-expansion autoregressive baseline, with the Stream-Factorized (SF) variants using only 1 or 4 full attention layers approaching the all-full configuration. Numbers in parentheses indicate the number of full attention layers (out of 27 total); overall average is the mean of the 13 listed benchmarks. Best per row in bold.

Every Hidden Decoding model substantially exceeds the autoregressive baseline (overall improvement of +1.55 to +2.62). All-full is best, but the cheaper 4-full and 1-full variants recover most of the gain (+2.16 and +1.55). A few full layers suffice: the rest of the cross-stream mixing can be local or even skipped, allowing Stream-Factorized Attention to retain most accuracy gains while staying cheap.

KV Retention Ablation

On KV retention, we perform a qualitative comparison on a 21B MoE with n=2: replacing per-stream KV with shared KV (retaining the last-stream prediction target and individual stream trajectories, but using a single shared KV in intra-stream layers). Results show that shared KV lowers the average score in both layouts (SF-1full: 69.00 → 68.59; SF-4full: 69.46 → 69.06), indicating that maintaining independent KV states for different streams is helpful.

Table 4: Small-scale KV retention ablation (21B MoE, HD n=2). Qualitative test of whether stream-independent KV matters. Accuracy columns are few-shot accuracy (%, higher is better); Avg is the average of all nine few-shot tasks. Per-stream KV yields higher averages in both layouts.

Stream and LM-head Probing

To connect the accuracy effects above with internal computation, we conduct probing on a trained Qwen3-8B-Base + Hidden Decoding (n=8, E7 is the final prediction stream, E0–E6 are intermediate streams), yielding two consistent findings.

First, intermediate streams form distinct internal states that are read by the final stream. The different streams of the same token diverge significantly in the middle Transformer layers and partially converge near the output layers (average cosine similarity 0.987 → 0.637 → 0.783), and are always higher than the baseline of different tokens; concurrently, the final stream E7 allocates a non-trivial amount of attention to other streams (especially E0, peaking at 0.52), forming a readout pathway from intermediate to prediction streams (Figure 5).

Hidden state similarity of streams for the same token and attention proportion of the final streamFigure 5: Probing on Qwen3-8B-Base + Hidden Decoding (n=8). (a) Cosine similarity of hidden states for streams of the same token across Transformer layers (dashed line is baseline for different tokens); (b) Attention proportion of the final stream E7 to each stream.

Second, in the vocabulary space, intermediate streams retain a broader candidate distribution than the final stream. When applying the shared LM Head to each stream, the top-1 token of intermediate streams often differs from E7 (max difference rate ~63%), and their probe entropy is generally higher than that of the final stream, indicating that a wider token-level uncertainty is preserved before the final prediction converges (Figure 6).

Top-1 difference rate of intermediate streams and probe entropy of each streamFigure 6: Next-token distribution obtained by applying the shared LM Head to the hidden state of each stream. (a) Proportion of cases where the top-1 token of each intermediate stream differs from E7; (b) Average probe entropy of each stream (n=2,4,8).

Overall, ablations and probes point to the same explanation: the training objective ablation shows that “supervising only the last stream” outperforms imposing loss on each stream or summing outputs; KV and attention analyses show that per-stream KV is beneficial, streams diverge within the Transformer, and the final stream reads other streams; LM-head probing provides an interpretability view from the vocabulary space – intermediate streams retain a broader set of candidates before the final stream converges to a prediction. This is precisely what Hidden Decoding means by using preceding streams as “latent space computation states.”

Conclusion and Outlook

This paper advances Hidden Decoding from method validation to frontier scale. By expanding along the sequence dimension and combining with Stream-Factorized Attention, we trained WeLM-HD4-80B and WeLM-HD4-617B with n=4, comprehensively surpassing their matched autoregressive baselines under the same early SFT-only post-training; cost measurements explain why this scale is practically feasible. Together, these results demonstrate that sequence-length scaling is a viable, parameter-free scaling path for large frontier models.

Looking ahead, we will continue to push Hidden Decoding on stronger baselines. After post-training with higher-quality, larger-scale data, the 80B and 617B MoE autoregressive bases have already shown significantly higher benchmark scores and more stable user experiences (see Table 6); next, we will further verify the gains of Hidden Decoding on these stronger baselines and fully integrate it with RL to continuously expand the model’s capability frontier.

Table 6: Performance of new autoregressive MoE bases after post-training with higher quality and larger scale data on various benchmarks; Kimi K2.6 as a reference mainstream frontier model. All scores are percentages. The Kimi K2.6 column shows self-tested values within the same evaluation framework as WeLM, with official reported values in parentheses where available.

• *FrontierMath uses public Tiers 1–4 sample problems.
• †HLE is no-tools, text-only setting.
• ‡HMMT is the weighted average of test sets from February 2025, November 2025, and February 2026.
• §Chinese Writing and Multilingual-Open are internal datasets.
• Terminal-Bench 2.1: evaluated using the Terminus-2 framework and its provided JSON parser; decoding parameters: temperature=1.0, top_p=0.95, context window 256K; results are the average of two independent runs.

References