Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

# Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Source: [https://arxiv.org/html/2607.08186](https://arxiv.org/html/2607.08186)
###### Abstract

Scaling Large Language Models \(LLMs\) has been driven mainly by enlarging the Transformer backbone, but for an already\-strong model this requires another round of costly pretraining\. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed\. Depth\-recurrent \(looped\) Transformers pursue this goal but are hard to scale, because looped computation does not fit naturally with the pipeline parallelism used to train the largest models\. We add computation along the sequence\-length dimension, where the extra computation is simply a longer input and stays compatible with standard large\-model training\. We proposeHidden Decoding, a sequence\-length scaling method applied during continued pretraining \(CPT\)\. It expands each token intonnstreams with independent embedding tables and keeps the intermediate streams’ key–value cache as context, so each token performs more internal computation without adding or widening Transformer layers\. To keep this affordable at scale, we introduceStream\-Factorized Attention, in which most layers attend only within each stream and only a few layers mix across streams, reducing the attention cost from quadratic to roughly linear innn\. Experiments support two scaling results\. At frontier scale, we train WeLM\-HD4\-80B and WeLM\-HD4\-617B atn=4n\{=\}4and improve their matched non\-HD baselines, making Hidden Decoding the first demonstrated sequence\-length scaling method at the 100B\+ MoE scale\. Across expansion factors, the gains grow asnnincreases, showing that sequence\-length expansion is a practical fixed\-backbone scaling path for frontier\-scale LLMs\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x1.png)Figure 1:Hidden Decoding scales and improves frontier models\.\(a\)On the dense Qwen3\-8B\-Base, average accuracy grows steadily with the expansion factornn\.\(b\)ImprovementΔ\\Delta\(in points\) of WeLM\-HD4\-80B and WeLM\-HD4\-617B over their matched non\-HD counterparts, WeLM\-80B and WeLM\-617B, after the same early SFT\-only post\-training\.## 1Introduction

Scaling has been a primary source of progress in Large Language Models \(LLMs\)\. Increasing model size, training data, and training compute has consistently led to stronger models\[kaplan2020scaling,hoffmann2022training\]\. However, for already strong foundation models, further scaling the Transformer backbone is costly: it often requires another round of large\-scale pretraining and increases both training and serving costs\. This motivates the setting of this paper: improving an existing backbone by giving each token more internal computation while keeping the Transformer backbone fixed\.

Recent reasoning models and test\-time scaling studies show that, even with fixed parameters, spending more computation per problem can improve accuracy\[openai2024reasoning,deepseek2025r1,muennighoff2025s1,snell2024scaling\]\. This motivates a goal that recent latent\-reasoning work has started to pursue: moving part of the extra thinking computation from visible tokens into the model’s internal computation\.

A prominent direction toward this goal is recurrent\-depth or looped Transformers, which add computation by reusing the same Transformer blocks several times on each token\[geiping2025latent\_recurrent,zhu2025ouro,saunshi2025reasoninglatentthoughtspower,park2026loopus\]\. This is a direct way to increase per\-token computation with a fixed set of Transformer weights\. However, it is hard to scale\. Training the largest MoE models relies on pipeline parallelism\[huang2019gpipe,narayanan2021efficient,deepseekai2025deepseekv3technicalreport,kimiteam2026kimik2openagentic\], which assumes each input passes through the model’s stages only once\. A looped model breaks this assumption, because it feeds the same hidden states back through the same stages on every iteration, which stalls the pipeline and leaves GPUs idle\. Consistent with this, looped models have stayed small: the largest is the 40B dense LoopCoder, trained without pipeline parallelism\[yang2026iquestcoderv1technicalreport\], while other looped language models remain at a few billion parameters\[geiping2025latent\_recurrent,zhu2025ouro\]\. Looping therefore still lacks a practical way to scale to very large models while keeping GPUs efficiently utilized\.

A key observation is that sequence\-dimension computation fits large\-scale training better than depth reuse\. Along the sequence dimension, the expansion simply amounts to feeding a longer input, which is naturally compatible with the standard optimizations used for large\-model training\. The general paradigm is to expand each input token into several streams along the sequence, process the expanded sequence in a single forward pass, and apply the next\-token loss only to the final stream, so the intermediate streams add per\-token computation without adding new Transformer layers or widening existing ones\. One instantiation is the Parallel Hidden Decoding Transformer \(PHD\)\[wu2025efficientpretraininglengthscaling\], which repeats each token but, to keep inference cheap, makes the repeated\-token KV transient: intermediate streams can help within the current token, but they are not retained as independent KV context for later tokens\. The capability\-scaling setting therefore needs persistent intermediate\-stream states that remain available across positions during CPT\.

In this work, we propose a sequence\-length scaling method namedHidden Decoding, which improves the capability of frontier\-scale LLMs through continued pretraining \(CPT\)\. Two design choices make the expanded streams useful for capability scaling\. To give different streams distinct initial states, we replicate the vocabulary embedding table intonnper\-stream tables, expanding each tokenxix\_\{i\}into streams\(E1​\(xi\),E2​\(xi\),…,En​\(xi\)\)\(E\_\{1\}\(x\_\{i\}\),E\_\{2\}\(x\_\{i\}\),\\ldots,E\_\{n\}\(x\_\{i\}\)\); through CPT, these tables learn diverse initial representations of the same token\. To let intermediate computation persist across positions, we keep the KV of the intermediate streams, so the computation accumulated across streams stays available as context for later tokens\.

Retaining the KV of all streams makes intermediate computation available to later tokens, but it also makes the attention cost grow asO​\(n2​L2\)O\(n^\{2\}L^\{2\}\), which becomes infeasible to train for large models with long sequences\. We therefore introduceStream\-Factorized Attention\. The key idea is to limit cross\-stream attention to a subset of layers: most layers attend only within each stream, and the streams exchange information only at the remaining layers, which follow the base model’s attention pattern \(sliding\-window, or full when the base model uses full attention\)\. Because dense attention over the fulln​LnLsequence is thus avoided at most layers, the added attention cost stays roughly linear rather than quadratic innn\. This efficiency is what makes Hidden Decoding practical to train at the scale of MoE models with over 100B parameters, a regime not reached by prior looped or length\-scaling methods\.

Experiments directly test the two scaling claims\. To test whether Hidden Decoding scales to the 100B\+ MoE regime, we train WeLM\-HD4\-80B and WeLM\-HD4\-617B atn=4n\{=\}4\. WeLM\-HD4\-80B improves all nine shared benchmarks over WeLM\-80B, with large gains on SciCode \(45\.8→50\.045\.8\\to 50\.0\) and PHYBench \(69\.8→73\.869\.8\\to 73\.8\)\. WeLM\-HD4\-617B also improves all nine shared benchmarks over WeLM\-617B, including GPQA Diamond \(89\.1→91\.289\.1\\to 91\.2\), HLE \(33\.6→35\.433\.6\\to 35\.4\), and FrontierMath \(49\.0→51\.049\.0\\to 51\.0\)\. These results use an early SFT\-only post\-training recipe: each matched pair of non\-HD and HD models uses the same SFT training recipe, with a short supervised fine\-tuning schedule and no reinforcement learning\. We use them as controlled comparisons; mature WeLM release scores are outside the scope of this paper\. To test whether the expansion factor itself provides scaling, we increasennfrom22to88in the 80B progressive\-expansion study\. MMLU improves monotonically \(85\.0→86\.7→87\.585\.0\\to 86\.7\\to 87\.5\), and Pile\-test BPB falls from0\.3860\.386to0\.3780\.378\. Together, these results establish sequence\-length scaling as a practical fixed\-backbone scaling path for frontier\-scale LLMs\.

Main ContributionsFrontier\-Scale Fixed\-Backbone Scaling\.To our knowledge, we are the first to demonstrate sequence\-length scaling at the 100B\+ MoE scale through continued pretraining \(CPT\)\. This yields two key models, WeLM\-HD4\-80B and WeLM\-HD4\-617B, in a regime not reached by prior depth\-recurrent \(looped\) or length\-scaling approaches\.Efficient Sequence\-Length Expansion\.To make the expanded sequence trainable at frontier scale, we introduceStream\-Factorized Attention: most layers attend only within each stream, while a small subset mixes streams\. This keeps the attention cost near\-linear innn; on WeLM\-HD4\-80B and WeLM\-HD4\-617B, the4×4\\timesexpanded sequence costs only5\.1×5\.1\\timesand4\.4×4\.4\\timesper batch, respectively, far below the dense\-attention16×16\\timesbaseline\.Expansion\-Factor Scaling\.We show that increasing the expansion factor improves language\-modeling loss and downstream accuracy, validatingnnas a practical scaling knob for a fixed Transformer backbone\.

## 2Method

Hidden Decodingis a sequence\-length scaling method that increases the computation each token receives without enlarging the Transformer backbone\. Each input token is represented innnparallel streams inside the Transformer, so a length\-LLsequence is processed as a length\-n​LnLsequence in a single forward pass, giving each tokennninternal computation steps before its prediction\. Figure[2](https://arxiv.org/html/2607.08186#S2.F2)contrasts Hidden Decoding with depth\-recurrent \(looped\) computation: Hidden Decoding places the extra computation along the sequence dimension, while looped computation reuses the backbone along the depth dimension\. This section formalizes the multi\-stream expansion and training objective \(§[2\.1](https://arxiv.org/html/2607.08186#S2.SS1)\), introducesStream\-Factorized Attentionto keep the expansion affordable at scale \(§[2\.2](https://arxiv.org/html/2607.08186#S2.SS2)\), and describes how to grow the expansion factor progressively from a converged checkpoint \(§[2\.3](https://arxiv.org/html/2607.08186#S2.SS3)\)\.

#### Setup and notation\.

A standard Transformer language model maps each tokenxix\_\{i\}through a single vocabulary embedding tableEEto form the input sequence\(E​\(x1\),…,E​\(xL\)\)\(E\(x\_\{1\}\),\\dots,E\(x\_\{L\}\)\); it adds positions with Rotary Positional Embeddings \(RoPE\)\[su2024roformer\], applies causal attention, and predicts the next token from each position’s hidden state through a shared headgθ​\(⋅\)g\_\{\\theta\}\(\\cdot\)\. Hidden Decoding leaves this path unchanged and changes only how each token is embedded into the sequence\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/figures/method.png)Figure 2:Looped latent computation vs\. Hidden Decoding\.\(a\)Depth\-recurrent \(looped\) computation reuses the same backbonefθf\_\{\\theta\}forKKsteps on each token’s hidden state before predicting the next token\.\(b\)Hidden Decoding expands each token intonnstreams with independent embedding tablesE1,…,EnE\_\{1\},\\dots,E\_\{n\}, interleaving them into a length\-n​LnLsequence that is processed in a single pass through the same backbone\. The next\-token cross\-entropy is applied only at the final\-stream positions \(En/hnE\_\{n\}/h\_\{n\}\); the earlier streams \(E1,…,En−1E\_\{1\},\\dots,E\_\{n\-1\}\) receive no loss and act as latent computation states\. The panel shows a two\-stream example\.
### 2\.1Multi\-Stream Token Expansion

The core construction of Hidden Decoding is to replace the single embedding table of a standard model withnnindependent embedding tables, which we call*streams*, and to interleave their outputs into one longer input sequence\.

#### Expanded sequence\.

Given an input sequenceX=\(x1,…,xL\)X=\(x\_\{1\},\\dots,x\_\{L\}\)andnnembedding tablesE1,…,EnE\_\{1\},\\dots,E\_\{n\}, we form an expanded sequenceSSof lengthn​LnLby placing the stream\-kkrepresentation of tokenxix\_\{i\}at physical positiont=\(i−1\)​n\+kt=\(i\-1\)n\+k:

St=Ek​\(xi\),1≤i≤L,1≤k≤n\.S\_\{t\}=E\_\{k\}\(x\_\{i\}\),\\qquad 1\\leq i\\leq L,\\quad 1\\leq k\\leq n\.\(1\)Forn=2n=2, this yieldsS=\(E1​\(x1\),E2​\(x1\),E1​\(x2\),E2​\(x2\),…\)S=\\big\(E\_\{1\}\(x\_\{1\}\),E\_\{2\}\(x\_\{1\}\),E\_\{1\}\(x\_\{2\}\),E\_\{2\}\(x\_\{2\}\),\\dots\\big\)\. The expanded sequence is fed to the same Transformer under standard causal attention, with contiguous RoPE positions0,1,…,n​L−10,1,\\dots,nL\-1\. The expansion factornndirectly controls how much extra per\-token computation is added\.

#### Training objective\.

Only the final stream of each token,En​\(xi\)E\_\{n\}\(x\_\{i\}\), is supervised: from its hidden state we predict the next tokenxi\+1x\_\{i\+1\}through the shared language\-modeling headgθg\_\{\\theta\}, while the firstn−1n\-1streams receive no direct loss\. Lettinghth\_\{t\}denote the hidden state at physical positiontt, the objective is the next\-token cross\-entropy applied only at the final\-stream positionst=i​nt=in:

ℒ​\(θ\)=−∑i=1Llog⁡gθ​\(xi\+1∣hi​n\)\.\\mathcal\{L\}\(\\theta\)=\-\\sum\_\{i=1\}^\{L\}\\log g\_\{\\theta\}\\\!\\big\(x\_\{i\+1\}\\mid h\_\{in\}\\big\)\.\(2\)Because attention is causal, the final streamEn​\(xi\)E\_\{n\}\(x\_\{i\}\)attends to all earlier streams of the same token and of all preceding tokens\. The firstn−1n\-1streams therefore act as intermediate computation states that progressively refine the representation before the final, prediction\-bearing stream\. Supervising only the final stream and giving each stream its own embedding table give the intermediate streams this latent\-computation role; supervising all streams, or summing their outputs before prediction, degrades performance in the supervision\-design ablation in §[5\.1](https://arxiv.org/html/2607.08186#S5.SS1)\.

### 2\.2Stream\-Factorized Attention

Running dense attention over then​LnLexpanded positions costsO​\(n2​L2\)O\(n^\{2\}L^\{2\}\), so the training cost grows sharply for large LLMs with long sequences\.Stream\-Factorized Attentionreduces this by making most layers attend only*within*a stream and confining*across*\-stream mixing to a subset of layers\.

#### Stream and token index\.

For a positiont∈\{1,…,n​L\}t\\in\\\{1,\\dots,nL\\\}in the expanded sequence, let its token and stream indices be

i​\(t\)=⌈t/n⌉,s​\(t\)=\(\(t−1\)modn\)\+1,i\(t\)=\\left\\lceil t/n\\right\\rceil,\\qquad s\(t\)=\\big\(\(t\-1\)\\bmod n\\big\)\+1,\(3\)so thatSt=Es​\(t\)​\(xi​\(t\)\)S\_\{t\}=E\_\{s\(t\)\}\(x\_\{i\(t\)\}\)\. Consecutive blocks ofnnpositions correspond to consecutive tokens, with thennstreams ordered inside each block\.

#### Intra\-stream and cross\-stream layers\.

Using the indices in Eq\.[3](https://arxiv.org/html/2607.08186#S2.E3), an*intra\-stream*layer restricts attention to earlier positions of the same stream,

Mt,t′intra=𝟏​\[t′≤t∧s​\(t′\)=s​\(t\)\],M^\{\\text\{intra\}\}\_\{t,t^\{\\prime\}\}=\\mathbf\{1\}\\\!\\left\[\\,t^\{\\prime\}\\leq t\\ \\wedge\\ s\(t^\{\\prime\}\)=s\(t\)\\,\\right\],\(4\)which costsO​\(n​L2\)O\(nL^\{2\}\)instead of theO​\(n2​L2\)O\(n^\{2\}L^\{2\}\)of a dense layer; a*cross\-stream*layer also attends across streams, following the model’s usual attention pattern\. Figure[3](https://arxiv.org/html/2607.08186#S2.F3)illustrates the three resulting mask types\.

#### Design\.

Most layers are intra\-stream, and cross\-stream attention is enabled at only a subset of layers\. We add no new attention layers for this: we reuse the base model’s own attention layers as the cross\-stream layers\. When the base model contains sliding\-window attention, those layers serve as the cross\-stream layers, so cross\-stream attention is local; otherwise the cross\-stream layers are full\-attention layers\. Because most layers are intra\-stream, the added attention cost stays close to linear innnrather than quadratic \(§[3](https://arxiv.org/html/2607.08186#S3)\); the number and placement of cross\-stream layers is a design choice we study in §[4](https://arxiv.org/html/2607.08186#S4)\. This layout also keeps CPT closer to the checkpoint before HD expansion: most layers preserve the base model’s single\-stream causal path, while only a few layers introduce cross\-stream perturbations; Appendix[H](https://arxiv.org/html/2607.08186#A8)gives a separate 617B\-scale startup\-loss check for this effect\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/figures/attention-mask.png)Figure 3:Attention masks in Stream\-Factorized Attention\.Each panel shows which positions a query \(row\) may attend to \(colored\) in the expanded sequence\.\(a\)A full cross\-stream layer uses the standard causal \(lower\-triangular\) mask\.\(b\)A local cross\-stream layer uses a causal sliding\-window mask\.\(c\)An intra\-stream layer is causal within each stream, so the streams are invisible to one another\. Stream\-Factorized Attention makes most layers intra\-stream \(c\) and enables cross\-stream mixing \(a or b\) at only a few layers\.

### 2\.3Progressive Expansion

A large expansion factornncan be trained directly, but progressively growing it provides a better initialization for the many embedding streams and saves compute\. We therefore grow the factor progressively—1→2→4→81\\\!\\to\\\!2\\\!\\to\\\!4\\\!\\to\\\!8—initializing each factor from the converged checkpoint of the previous one\. Concretely, we use*Cyclic Replication Initialization*: when doubling fromnnto2​n2n, the new streams copy the existing tables,

En\+k←Ek,1≤k≤n,E\_\{n\+k\}\\leftarrow E\_\{k\},\\qquad 1\\leq k\\leq n,\(5\)so that the expanded model initially reproduces the behavior of the smaller one and then adapts smoothly\. For example,n=2→4n\{=\}2\\\!\\to\\\!4initializes the tables as\(E1,E2,E1,E2\)\(E\_\{1\},E\_\{2\},E\_\{1\},E\_\{2\}\), andn=4→8n\{=\}4\\\!\\to\\\!8as\(E1,E2,E3,E4,E1,E2,E3,E4\)\(E\_\{1\},E\_\{2\},E\_\{3\},E\_\{4\},E\_\{1\},E\_\{2\},E\_\{3\},E\_\{4\}\)\. In practice, we introduce successive expansions at scheduled token counts during continued pretraining, letting each factor converge before the next expansion\.

## 3Computational Cost

Hidden Decoding adds computation to every token, so its training and serving costs must be measured\. We separate two kinds of cost: the cost inherent to the method, which holds on any backbone, and an extra saving specific to our WeLM backbone\. We report the training measurements in §[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)and the serving measurements in §[4\.5](https://arxiv.org/html/2607.08186#S4.SS5)\.

### 3\.1Training Cost

Expanding each token intonnstreams turns a length\-LLsequence into lengthn​LnL\. Non\-attention computation scales roughly with the number of positions, but dense attention over the expanded sequence would scale asO​\(n2​L2\)O\(n^\{2\}L^\{2\}\), orn2×n^\{2\}\\timesthe attention cost of the unexpanded sequence\. Stream\-Factorized Attention \(§[2\.2](https://arxiv.org/html/2607.08186#S2.SS2)\) avoids this dense\-attention blow\-up by keeping most layers intra\-stream, so the attention cost stays close to linear innn\. This near\-linear attention design is inherent to the method and holds on any backbone\.

Our WeLM backbones allow one further, architecture\-specific saving through their*KV\-mirror*design\[welmblog\]\. In a standard layer, the keys and values come from that layer’s own input hidden states; under KV\-mirror, each later layer instead computes them from an earlier layer’s hidden states, using its own projection weights\. The layers are paired in a U\-shape, with the last third mirroring the first third\. Because a mirrored layer’s keys and values depend only on the early layer’s hidden states, our training framework computes them at the early layers, and the mirrored layers reuse them directly\.

KV\-mirror lets Hidden Decoding skip work in mirrored layers\. Only the final stream is supervised \(§[2\.1](https://arxiv.org/html/2607.08186#S2.SS1)\), so the intermediate streams matter only through the keys and values they add as context\. In the mirrored layers this context is already fixed by the early layers, and the intermediate streams carry no loss, so these layers need not process the intermediate streams at all\.We therefore run only the final stream through the mirrored layers, while the early layers still process allnnstreams\.In our 80B and 617B models the last third of the layers are mirrored, so the intermediate streams are skipped across that final third\. This saving is specific to WeLM; it comes on top of the near\-linear cost that Stream\-Factorized Attention already provides on any backbone\. As an implementation check, this KV\-mirror optimization reduces per\-batch time from15​s15\\,\\mathrm\{s\}to12​s12\\,\\mathrm\{s\}in an 80B 32k training setting, a20%20\\%reduction \(1\.25×1\.25\\timesspeedup\)\. We use this number to quantify the WeLM\-specific training\-system saving used by WeLM\-HD4\-80B and WeLM\-HD4\-617B; the main cost comparison in §[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)reports end\-to\-end HD training time under the long\-context settings\. The net effect is that training cost grows near\-linearly with the number of streamsnn\. We measure this at both scales in §[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)\(Figure[4](https://arxiv.org/html/2607.08186#S4.F4)a\)\.

### 3\.2Inference Cost

A key advantage of Hidden Decoding over looped models is that its extra computation runs in parallel\. A looped model adds computation by repeating the backbone step by step, and each step waits for the previous one, so the added cost cannot be hidden\. Hidden Decoding spreads its computation across thennstreams, which are processed together in a single forward pass\. At large batch sizes, decoding is compute\-bound, so this extra computation lowers throughput\. At small batch sizes, decoding is bound by memory bandwidth and leaves much of the compute idle; the parallel streams can use that idle compute, so the added cost stays small\. The one remaining overhead in this regime is the larger KV cache, which Stream\-Factorized Attention keeps modest by reading the full expanded KV at only a few layers\.The throughput penalty is therefore smallest in the latency\-sensitive, short\-input/small\-batch regime and grows with larger batches or longer inputs, as we measure in §[4\.5](https://arxiv.org/html/2607.08186#S4.SS5)\.

For WeLM models, KV\-mirror can also reduce decoding\-side computation by skipping mirrored\-layer work for intermediate streams\. The serving benchmark in §[4\.5](https://arxiv.org/html/2607.08186#S4.SS5)uses the current benchmark implementation without this additional KV\-mirror decoding optimization, because integrating it into the decoding kernel requires separate engineering\. The reported throughput therefore does not include this WeLM\-specific speedup\.

## 4Experiments

We use the experiments to test three parts of Hidden Decoding \(HD\)\. The frontier\-scale MoE study checks whether HD improves the largest models we deploy, producing WeLM\-HD4\-80B and WeLM\-HD4\-617B at acceptable training cost \(§[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)\)\. The expansion\-factor study checks whether adding more streams behaves like a scaling axis \(§[4\.3](https://arxiv.org/html/2607.08186#S4.SS3)\)\. The ablations and throughput measurement check which attention choices and serving costs matter in practice \(§[4\.4](https://arxiv.org/html/2607.08186#S4.SS4), §[4\.5](https://arxiv.org/html/2607.08186#S4.SS5)\)\.

### 4\.1Setup

#### Models\.

We apply HD on top of strong pre\-trained checkpoints through continued pretraining \(CPT\), without enlarging the Transformer backbone\. Our main HD models are WeLM\-HD4\-80B and WeLM\-HD4\-617B, obtained by expanding WeLM MoE checkpoints at 80B \(3B activated\) and 617B \(23B activated\) total parameters ton=4n\{=\}4\. Their matched non\-HD counterparts are WeLM\-80B and WeLM\-617B\. When reporting base\-model evaluations before post\-training, we append\-Base, giving WeLM\-80B\-Base vs\. WeLM\-HD4\-80B\-Base and WeLM\-617B\-Base vs\. WeLM\-HD4\-617B\-Base\. We use smaller 21B \(0\.7B activated\) and 6B \(0\.6B activated\) MoE models for ablations, and the dense Qwen3\-8B\-Base\[qwen3\]\{NoHyper\}111[https://github\.com/Tencent/Sequential\-Hidden\-Decoding](https://github.com/Tencent/Sequential-Hidden-Decoding)\.222[https://huggingface\.co/collections/tencent/sequential\-hidden\-decoding](https://huggingface.co/collections/tencent/sequential-hidden-decoding)\.for progressive expansion and probes\.

Only the embedding tables grow with the expansion factornn, while the active Transformer parameters per token stay unchanged\. For example, expanding ton=4n\{=\}4increases the stored embedding parameters from26\.926\.9B to107\.4107\.4B for WeLM\-HD4\-617B and from1\.21\.2B to3\.13\.1B on Qwen3\-8B\-Base \(reaching5\.65\.6B atn=8n\{=\}8\)\. These tables are sparse lookups: each position reads a single embedding row, and the tables do not enter the attention or feed\-forward matrix multiplications that set the compute cost\. We therefore report the embedding growth separately and compare matched non\-HD and HD models as a fixed\-Transformer\-backbone, fixed\-active\-parameter test\.

#### Configurations\.

The large MoE models are expanded ton=4n\{=\}4\. The dense model is expanded progressively ton∈\{2,4,8\}n\\in\\\{2,4,8\\\}for the expansion\-factor study\. Following §[2\.2](https://arxiv.org/html/2607.08186#S2.SS2), most layers are intra\-stream and cross\-stream mixing is placed on a subset of layers\. For the 80B model \(49 layers\), we use 6 full, 20 intra\-stream, and 23 sliding\-window layers\. For the 617B model \(94 layers\), we use 25 full and 69 intra\-stream layers\. The base models’ native context length is 256k for 80B and 32k for 617B\. RoPE and context\-extension settings are held fixed within each matched comparison between non\-HD and HD models, including the shared early SFT recipe in Appendix[D](https://arxiv.org/html/2607.08186#A4)\. Appendix[B](https://arxiv.org/html/2607.08186#A2)gives the backbone and HD configuration details for the two main WeLM models\.

#### Continued pretraining\.

We introduce Hidden Decoding during continued pretraining\. We start from a baseline checkpoint and continue training with the expansion enabled, using the same data and schedule as the baseline except for the 617B long\-context\-stage difference in Table[1](https://arxiv.org/html/2607.08186#S4.T1)\. The expansion stays on from that point onward: it carries from the 32k continuation stage into a later 256k long\-context stage, and then through the early SFT\-only post\-training recipe\. The non\-HD baseline is trained in the same way with the expansion turned off, so the two runs differ only in Hidden Decoding\. For the 80B comparison, the two runs see identical data at every stage\. For the 617B comparison, the HD long\-context stage uses fewer tokens than the non\-HD model, so its gains are a conservative lower bound\.

Table[1](https://arxiv.org/html/2607.08186#S4.T1)reports the training\-token budget behind this comparison\.

Table 1:Continued\-pretraining token budgets for non\-HD base paths and HD4 continuation windows\.Non\-HD rows count the full base\-model path\. HD4 rows count only the training after the HD start point\.ScaleRunHD starttokens8kpretrain32kcontinuation256kcontinuationTotal80BWeLM\-80B\-Base–17\.81T2\.01T0\.57T20\.39T80BWeLM\-HD4\-80B\-Base19\.32T00\.50T0\.57T1\.07T617BWeLM\-617B\-Base–14\.25T2\.20T0\.61T17\.06T617BWeLM\-HD4\-617B\-Base15\.86T00\.59T0\.30T0\.90TFor 80B, Hidden Decoding starts after19\.3219\.32T tokens on the non\-HD training path, and WeLM\-HD4\-80B\-Base then trains on the same remaining0\.500\.50T 32k tokens and0\.570\.57T 256k tokens as the non\-HD continuation\. For 617B, Hidden Decoding starts after15\.8615\.86T tokens on the non\-HD training path\. The HD4 run matches the remaining0\.590\.59T 32k continuation, but uses a shorter 256k continuation \(0\.300\.30T vs\.0\.610\.61T\)\. Relative to the full non\-HD base paths, the HD4\-only token budgets are about5\.3%5\.3\\%at both scales, so the higher per\-token cost is concentrated late in training\.

#### Post\-training scope\.

The post\-training used for the early post\-training tables is intentionally lightweight: it is an early supervised fine\-tuning run and does not include reinforcement learning\. We use this stage to test whether the gains from Hidden Decoding persist under the same downstream adaptation recipe for the non\-HD and HD models\. Appendix[D](https://arxiv.org/html/2607.08186#A4)gives the SFT recipe\. We report these scores as controlled early\-SFT comparisons, with mature WeLM release scores left outside the scope of this paper\.

#### Evaluation\.

We use two evaluation suites\. For early post\-training results \(Tables[2](https://arxiv.org/html/2607.08186#S4.T2)and[11](https://arxiv.org/html/2607.08186#A5.T11)\), we evaluate on recent hard benchmarks covering mathematics and science \(HMMT and MathArena Apex\[balunovic2025matharena\], IMO\-AnswerBench\[luong2025imobench\], FrontierMath\[glazer2024frontiermath\], PolyMath\[wang2025polymath\], GPQA\[rein2024gpqa\], PHYBench\[qiu2025phybench\], CritPt\[zhu2025critpt\]\), knowledge and reasoning \(HLE\[phan2025hle\], MMMLU\[openaimmmlu\], AA\-OmniScience\[jackson2025aaomniscience\], AA\-LCR\[aalcr\], ARC\-AGI\-2\[chollet2025arcagi2\]\), code \(SciCode\[tian2024scicode\], LiveCodeBench\[jain2024livecodebench\]\), instruction following \(IFBench\[pyatkin2025ifbench\]\), and agentic tasks \(Terminal\-Bench 2\[terminalbench\],τ2\\tau^\{2\}\-Bench\[barres2025tau2\], GDPval\[patwardhan2025gdpval\]\)\.

For scaling and ablation studies \(§[4\.3](https://arxiv.org/html/2607.08186#S4.SS3), §[4\.4](https://arxiv.org/html/2607.08186#S4.SS4)\) and pretraining\-stage results \(Appendix[I](https://arxiv.org/html/2607.08186#A9)\), we use a standard suite: MMLU\[hendrycks2020measuring\], MMLU\-Pro\[wang2024mmlupro\], CMMLU\[li2023cmmlu\], C\-Eval\[huang2023ceval\], SuperGPQA\[du2025supergpqa\], ARC\-C\[clark2018think\], HellaSwag\[zellers2019hellaswag\], BBH\[suzgun2022bbh\], GSM8K\[cobbe2021training\], MATH\[hendrycks2021measuring\], SimpleQA\[wei2024simpleqa\], Chinese SimpleQA\[he2024chinesesimpleqa\], HumanEval\+/MBPP\+\[liu2023code\], MultiPL\-E\[cassano2022multiple\], and CRUXEval\[gu2024cruxeval\]\. Tables[8](https://arxiv.org/html/2607.08186#A3.T8)and[9](https://arxiv.org/html/2607.08186#A3.T9)give the per\-benchmark settings\.

### 4\.2Frontier\-Scale MoE Results

To test whether Hidden Decoding strengthens the largest models we deploy, we compare two forms of each WeLM MoE checkpoint that share everything except Hidden Decoding: WeLM\-80B vs\. WeLM\-HD4\-80B, and WeLM\-617B vs\. WeLM\-HD4\-617B\. The HD models add Hidden Decoding during continued pretraining \(Table[1](https://arxiv.org/html/2607.08186#S4.T1)\); both forms are then adapted with the same early SFT\-only post\-training recipe, so the comparison isolates our method\. Table[2](https://arxiv.org/html/2607.08186#S4.T2)evaluates both scales on the same nine hard benchmarks\.

Table 2:Early SFT♣results for WeLM\-HD4\-80B and WeLM\-HD4\-617B\.We compare each HD4 model with its matched non\-HD counterpart: WeLM\-80B vs\. WeLM\-HD4\-80B and WeLM\-617B vs\. WeLM\-HD4\-617B\. HD4 denotes Hidden Decoding withn=4n\{=\}4; active Transformer parameters per token are unchanged \(33B for 80B,2323B for 617B\)\. Both forms use the same early SFT\-only post\-training recipe, with a short SFT schedule and no RL\. Kimi K2\.6\[moonshotai2026kimik26\]is included as a mainstream frontier\-model reference for the absolute score scale; its model card reports11T total parameters and3232B activated parameters\.‡HMMT is the weighted average over the Feb\. 2025, Nov\. 2025, and Feb\. 2026 sets\.80B MoE617B MoEExternalBenchmarkWeLM80BWeLM\-HD480BWeLM617BWeLM\-HD4617BKimi K2\.61T\-A32BGPQA Diamond87\.688\.889\.191\.290\.4HLE27\.428\.433\.635\.436\.9MMMLU84\.485\.686\.487\.588\.0FrontierMath⋆45\.849\.049\.051\.053\.2PHYBench69\.873\.875\.376\.374\.0MathArena Apex16\.420\.124\.224\.723\.8HMMT‡93\.394\.196\.096\.296\.0IMO\-AnswerBench85\.085\.387\.588\.591\.5SciCode45\.850\.051\.452\.150\.7
Hidden Decoding improves every benchmark at both scales\. The largest gains appear on hard math and science tasks: WeLM\-HD4\-80B improves SciCode by\+4\.2\+4\.2, PHYBench by\+4\.0\+4\.0, and FrontierMath by\+3\.2\+3\.2; WeLM\-HD4\-617B improves GPQA by\+2\.1\+2\.1and HLE by\+1\.8\+1\.8\. We also include Kimi K2\.6 \(1T total, 32B activated\) as a mainstream frontier\-model reference for absolute score scale: WeLM\-HD4\-617B is higher on GPQA Diamond, PHYBench, MathArena Apex, HMMT, and SciCode, while Kimi K2\.6 is higher on HLE, MMMLU, FrontierMath, and IMO\-AnswerBench\. On the broader 80B suite in Appendix[E](https://arxiv.org/html/2607.08186#A5), the largest gains shift toward agentic and long\-horizon tasks, including Terminal\-Bench 2 \(44\.9→58\.444\.9\\\!\\to\\\!58\.4\) and ARC\-AGI\-2 \(6\.9→11\.66\.9\\\!\\to\\\!11\.6\)\.These results show that Hidden Decoding can be scaled to frontier\-size MoE models and improve them without enlarging the Transformer backbone\.This is the scaling regime where looped models have been hard to apply, because repeated depth passes fit poorly with the pipeline parallelism used to train large MoEs\.

#### Training cost measurement\.

To verify the training\-cost analysis in §[3](https://arxiv.org/html/2607.08186#S3), we measure per\-batch training time on the WeLM backbones relative to the unexpanded baseline\. Figure[4](https://arxiv.org/html/2607.08186#S4.F4)\(a\) shows that a4×4\\timeslonger effective sequence costs5\.1×5\.1\\timeson 80B \(256​k→1​M256\\mathrm\{k\}\\\!\\rightarrow\\\!1\\mathrm\{M\}\) and4\.4×4\.4\\timeson 617B \(32​k→128​k32\\mathrm\{k\}\\\!\\rightarrow\\\!128\\mathrm\{k\}\)\. These costs stay close to the4×4\\timeslinear reference and far below the16×16\\timesdense\-attention cost\.This near\-linear cost makes WeLM\-HD4\-617B trainable, while dense attention over the expanded sequence would be infeasible\.Using full attention on only a few layers is a deliberate trade\-off; we measure its effect on accuracy with an ablation on a smaller model in §[4\.4](https://arxiv.org/html/2607.08186#S4.SS4)\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x2.png)Figure 4:\(a\) Training cost of HD \(n=4n\{=\}4\):per\-batch time relative to the unexpanded baseline, staying near the linear \(n=4×n\{=\}4\\times\) reference and far below dense attention \(n2=16×n^\{2\}\{=\}16\\times\)\.\(b\) Progressive expansion on the dense Qwen3\-8B\-Base:the mean improvement over the unexpanded baseline grows steadily with the expansion factornn; the shaded band shows the range across the seven benchmarks\.
#### External\-model reference\.

To place the base\-model strength of WeLM\-HD4\-617B\-Base in context, we compare it with the base\-model table reported in the Qwen3\.5 report\[qwen35blog\]\. Table[3](https://arxiv.org/html/2607.08186#S4.T3)keeps the overlapping benchmarks from that report and our base\-model evaluation\. We include both WeLM\-617B\-Base and WeLM\-HD4\-617B\-Base, evaluated with our base\-model protocol before instruction tuning or reinforcement learning\. Because model families and evaluation harnesses still differ, the controlled evidence remains the matched comparison between non\-HD and HD models under the same backbone, data, and training configuration\. The WeLM baseline scores are obtained by evaluating the matched non\-HD Base checkpoint with the same harness as the HD Base checkpoint\.

Table 3:Base\-model reference\.The WeLM columns use our base\-model protocol before instruction tuning or RL\. External numbers are copied from the Qwen3\.5 report’s base\-model table and provide absolute\-score context; the matched WeLM columns give the controlled comparison between non\-HD and HD models\.BenchmarkWeLM617B\-BaseWeLM\-HD4617B\-BaseDS\-V3\.2671B\-A37BK21T\-A32BQwen3\.5397B\-A17BMMLU89\.8590\.2288\.1187\.3888\.61MMLU\-Pro72\.2673\.9262\.8267\.6476\.01MMLU\-Redux89\.8190\.9487\.2986\.6589\.09SuperGPQA55\.3357\.0543\.4644\.8657\.96C\-Eval91\.9092\.5090\.4891\.8291\.82BBH92\.4193\.0786\.0389\.1190\.98KoRBench52\.0854\.8854\.0053\.8454\.08CRUX\-Input86\.0091\.7563\.2570\.5071\.13CRUX\-Output89\.3891\.1373\.8877\.1382\.38

### 4\.3Expansion\-Factor Scaling

To test whether the expansion factor is a usable scaling knob, we expand the dense Qwen3\-8B\-Base step by step ton∈\{2,4,8\}n\\in\\\{2,4,8\\\}while keeping the Transformer backbone fixed\. Figure[4](https://arxiv.org/html/2607.08186#S4.F4)\(b\) reports improvement over the unexpanded baseline\. The mean improvement rises almost steadily withnn, and every benchmark gains; the largest improvements reach\+5\.6\+5\.6on HellaSwag and\+5\.1\+5\.1on BBH and MATH\.

The same trend appears at MoE scale\. Expanding the 80B MoE raises MMLU from85\.185\.1atn=1n\{=\}1to87\.587\.5atn=8n\{=\}8and lowers Pile\-test BPB from0\.3860\.386to0\.3780\.378\(full per\-factor results in Appendix[G](https://arxiv.org/html/2607.08186#A7)\)\.Within a fixed Transformer backbone, the expansion factor acts as a reliable scaling knob: increasing it improves accuracy and language\-modeling loss\.

Table 4:Attention\-composition ablation \(21B MoE\)\.We compare a no\-expansion baseline with three HD variants that differ only in the number of full cross\-stream layers\. Header parentheses give the number of full\-attention layers \(out of2727\)\. SF with44full layers nearly matches all\-full, while SF with11full layer still improves the baseline\.Hidden DecodingBenchmarkBaselineSF \(11\)SF \(44\)all\-full \(2727\)MMLU74\.175\.776\.476\.3MMLU\-Pro47\.550\.449\.850\.7CMMLU77\.979\.279\.879\.6C\-Eval78\.678\.979\.279\.7ARC\-C89\.589\.890\.790\.9SuperGPQA34\.135\.435\.736\.0BBH67\.071\.272\.072\.8GSM8K83\.486\.286\.786\.2MATH45\.249\.649\.350\.3SimpleQA3\.74\.03\.64\.1AA\-OmniScience12\.712\.414\.214\.1HumanEval\+37\.437\.639\.240\.5MBPP\+55\.256\.157\.859\.1Average54\.3355\.8856\.4956\.95
### 4\.4Attention Composition Ablation

To test how much full cross\-stream mixing is required for accuracy, we ablate the attention composition inside Stream\-Factorized Attention\. Each layer mixes streams in one of three ways:*full*\(across all streams over the whole sequence\),*local*\(across streams within a sliding window\), or*intra\-stream*\(no cross\-stream mixing\)\. Full mixing is the expensive case\.

We run this ablation on the 21B MoE, which has2727layers and whose base model already interleaves full\-attention and sliding\-window layers\. The reference is a standard no\-expansion baseline\. The three Hidden Decoding variants differ only in how many layers use full cross\-stream mixing:11,44, or all2727, with the remaining layers split between local and intra\-stream mixing \(exact layouts in Appendix[F](https://arxiv.org/html/2607.08186#A6)\)\. Table[4](https://arxiv.org/html/2607.08186#S4.T4)reports the evaluated benchmarks and their average\.

Every Hidden Decoding model beats the no\-expansion baseline on average, with gains from\+1\.55\+1\.55to\+2\.62\+2\.62\. The all\-full model is best, but the cheaper44\-full variant is close \(\+2\.16\+2\.16\), and the11\-full variant still improves the baseline \(\+1\.55\+1\.55\)\.A few full layers are enough: the remaining cross\-stream mixing can be local or skipped, which keeps Stream\-Factorized Attention cheap while preserving most of the accuracy gain\.

### 4\.5Inference Throughput

To measure serving cost under a matched setting, we vary the two factors that most directly affect decoding throughput: input length and batch size\. For WeLM\-HD4\-80B, we serve each request bucket twice on the same88H20 GPUs: once with Hidden Decoding enabled, and once on the matched WeLM\-80B baseline with stream expansion disabled\. The benchmark reports decoding\-only completion TPS after excluding prefill latency, with maximum output length81928192, a3030\-second warmup, and a4545\-second measurement window\. Each cell in Figure[5](https://arxiv.org/html/2607.08186#S4.F5)reports the mean Hidden Decoding completion throughput over five runs as a percentage of the matched baseline under the same input\-length bucket and batch size; the small label gives the run\-to\-run standard deviation in percentage points\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x3.png)Figure 5:Serving throughput matrix for WeLM\-HD4\-80B\.Each cell reports mean decoding\-only completion TPS for Hidden Decoding over five runs as a percentage of the matched WeLM\-80B baseline with stream expansion disabled under the same input\-length bucket and batch size; prefill latency is excluded\. Small text shows the five\-run sample standard deviation in percentage points\. Both models run on the same88H20 GPUs\. The benchmark uses a3030\-second warmup, a4545\-second measurement window, and maximum output length81928192\.The measurements identify a practical serving window\. At batch size11, Hidden Decoding keeps8383–88%88\\%of baseline throughput across all measured input buckets\. For the short, 2k, and 8k buckets, it keeps6969–88%88\\%through batch size44, so latency\-oriented small\-batch serving preserves most baseline decode throughput\. The cost becomes substantial in long\-context, high\-batch decoding: at batch size1616, the ratio falls to60%60\\%for short inputs,54%54\\%for the 2k bucket,49%49\\%for the 8k bucket, and44%44\\%for the 32k bucket; the largest measured setting, 32k input at batch size3232, keeps27%27\\%\. These matched measurements support the cost model in §[3\.2](https://arxiv.org/html/2607.08186#S3.SS2): Hidden Decoding is practical for interactive or moderate\-batch workloads, while long inputs and large batches expose the extra per\-token computation more directly\.

## 5Ablations and Probes of Intermediate Streams

Hidden Decoding relies on an unusual choice: only the final stream is trained to predict the next token, while the earlier streams receive no direct loss \(§[2\.1](https://arxiv.org/html/2607.08186#S2.SS1)\)\. The matched results in §[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)show that Hidden Decoding improves accuracy\. The accuracy gain alone leaves open what role the earlier streams play: a gain over a no\-expansion baseline could come from simpler effects such as a longer expanded sequence, more prediction targets, or explicit aggregation of stream outputs\. To test these possibilities, we use ablations and probes\. The training\-objective ablation compares final\-stream supervision with all\-token loss and output summation \(§[5\.1](https://arxiv.org/html/2607.08186#S5.SS1)\)\. The KV\-retention analysis tests whether per\-stream KV is useful, and stream probes inspect hidden\-state separation and final\-stream attention \(§[5\.2](https://arxiv.org/html/2607.08186#S5.SS2)\)\. The LM\-head probes provide a vocabulary\-space view of the intermediate stream states \(§[5\.3](https://arxiv.org/html/2607.08186#S5.SS3)\)\.

### 5\.1Training Objective Ablation

To separate the effect of final\-stream supervision from simpler alternatives, we train objective variants on the same 6B MoE backbone and training setup\. The comparison asks whether the gain comes from leaving intermediate streams without a direct loss, or whether similar gains appear when every stream receives a prediction target or when stream outputs are explicitly aggregated\. The baseline has no stream expansion\. The expanded variants include*all\-token loss*, which applies the next\-token loss to every stream;*sum*, which adds the stream outputs together before prediction; and Hidden Decoding, which supervises only the final stream\. The main comparison uses the same expansion factorn=2n\{=\}2for all expanded variants; we also report HD atn=3n\{=\}3to show whether the preferred objective continues to improve with more streams\.

Table[5](https://arxiv.org/html/2607.08186#S5.T5)reports the result\. On language modeling, all expanded variants improve over the no\-expansion baseline, but HD gives the lowest loss at the same expansion factor:1\.8741\.874atn=2n\{=\}2, compared with1\.8801\.880for all\-token loss and1\.8771\.877for sum\. The downstream results follow the same pattern\. HD atn=2n\{=\}2is best on all four reported benchmarks, with the clearest separation on ARC\-C \(74\.374\.3vs\.71\.971\.9for sum and65\.865\.8for all\-token loss\)\. Increasing HD ton=3n\{=\}3further lowers loss to1\.8571\.857and improves all four scores\. These results favor final\-stream supervision over both extra direct supervision on intermediate streams and explicit summation of stream outputs\.

Table 5:Supervision\-design ablation \(6B MoE\)\.All variants use the same backbone and training setup\. At the same expansion factorn=2n\{=\}2, applying loss to every stream \(all\-token loss\) or summing stream outputs \(sum\) underperforms supervising only the final stream \(HD\)\. Loss: lower is better\.MethodLoss↓\\downarrowMMLUARC\-CC\-EvalCMMLUBaseline1\.90853\.468\.758\.461\.2All\-token loss \(n=2n\{=\}2\)1\.88055\.565\.859\.163\.2Sum \(n=2n\{=\}2\)1\.87755\.671\.961\.063\.3HD \(n=2n\{=\}2\)1\.87456\.574\.361\.263\.7HD \(n=3n\{=\}3\)1\.85758\.575\.762\.565\.9
### 5\.2KV Retention Ablation and Stream Probes

Hidden Decoding normally retains a separate KV cache for each stream, so later positions can use the computation produced by earlier intermediate streams\. To test whether this retained per\-stream KV is useful, we compare two Hidden Decoding variants that keep the same backbone, training setup, expansion factor, and Stream\-Factorized Attention layout\. The reference setting is the normal*per\-stream KV*design\. The controlled variant is*shared KV*, a PHD\-like control\[wu2025efficientpretraininglengthscaling\]: it keeps the final\-stream prediction objective and separate stream trajectories, but removes independently retained stream KV by using one shared KV cache across streams in intra\-stream layers\. If the intermediate\-stream KV were redundant, replacing per\-stream KV with this PHD\-like control should have little systematic effect\.

We run this as a small supporting ablation on the 21B MoE atn=2n\{=\}2\. This setting is intentionally limited, so we use it as a qualitative check of direction\. We report two Stream\-Factorized Attention layouts, SF\(1 full\) and SF\(4 full\), which keep one or four full cross\-stream layers\. In each layout, the only intended difference between the two rows is whether KV is retained separately per stream or shared across streams\.

Table 6:Small KV\-retention ablation \(21B MoE, HDn=2n\{=\}2\)\.This is a qualitative check of whether stream\-specific KV matters\. Accuracy columns are few\-shotloss\_accin percent \(higher is better\)\.FulllayersKVtypeFew\-shot accuracyAvgARCEEGPQAMMLUMMLU\-ProQA\-MMLUSuperGPQA1Per\-stream91\.5054\.1774\.6248\.8380\.9035\.3664\.23Shared91\.0153\.1974\.2446\.6081\.0734\.6363\.464Per\-stream92\.3254\.9575\.2447\.8282\.1135\.5064\.66Shared91\.2853\.3274\.4147\.9081\.1535\.3563\.90Because this is a smalln=2n\{=\}2comparison, Table[6](https://arxiv.org/html/2607.08186#S5.T6)should be read qualitatively\. Shared KV lowers the average in both layouts:64\.23→63\.4664\.23\\\!\\to\\\!63\.46with one full cross\-stream layer and64\.66→63\.9064\.66\\\!\\to\\\!63\.90with four full cross\-stream layers\. The per\-task columns are noisy, as expected in this small comparison, but the average moves in the same direction in both layouts\. These consistent average drops indicate that preserving separate KV states for different streams improves accuracy in this setting\.

The KV ablation establishes that retaining per\-stream KV affects accuracy\. To connect this accuracy effect to internal computation, we inspect what the streams contain and whether the final stream reads them\. We analyze the trained dense Qwen3\-8B\-Base Hidden Decoding model atn=8n\{=\}8, where E7 is the final prediction stream and E0–E6 are intermediate streams\. Figure[6](https://arxiv.org/html/2607.08186#S5.F6)gives two probes: hidden\-state similarity measures whether streams form different internal states, and attention affinity measures whether E7 reads other streams\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x4.png)Figure 6:Intermediate streams form states that the final stream can read \(Qwen3\-8B\-Base \+ Hidden Decoding,n=8n\{=\}8\)\.\(a\)Same\-token streams separate inside the Transformer and partially move closer near the final layer; the dashed curve is a different\-token baseline\.\(b\)The final stream E7 assigns substantial attention to other streams, especially E0, showing a read path from intermediate streams to the prediction stream\.Panel \(a\) shows that the streams form different internal states\. Same\-token streams start highly aligned, separate through the middle layers, and move closer again near the output: the mean cosine drops from0\.9870\.987at the input to0\.6370\.637in the middle layers, then rises to0\.7830\.783at the final layer\. The curve remains above the different\-token baseline, which separates stream\-specific structure from generic hidden\-state anisotropy\. Panel \(b\) shows that these states are connected to the prediction stream\. E7 assigns substantial attention to other streams, especially E0, whose affinity peaks at0\.520\.52and is comparable to or larger than self\-attention in several sampled layers\.Together, the KV ablation and the probes support the same interpretation: intermediate streams carry distinct states, and retaining their KV gives later computation access to those states\.

### 5\.3LM\-Head Probes in Vocabulary Space

To see how intermediate stream states appear at the output vocabulary level, we apply the shared LM head to each stream hidden state and inspect the resulting next\-token distribution\. This probe shows what token candidates are visible at different stream positions\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x5.png)Figure 7:LM\-head probes show broader intermediate token distributions \(Qwen3\-8B\-Base \+ Hidden Decoding\)\.We apply the shared LM head to each stream hidden state and inspect the resulting probed next\-token distribution\.\(a\)In then=8n\{=\}8model, intermediate probes often have a top\-1 token different from the final stream E7\.\(b\)Intermediate probes have higher entropy than the final stream, indicating broader token\-level uncertainty before the final prediction\.Figure[7](https://arxiv.org/html/2607.08186#S5.F7)gives two aggregate views\. Panel \(a\) uses the dense Qwen3\-8B\-Base Hidden Decoding model atn=8n\{=\}8and measures how often each stream’s probed top\-1 token differs from the final stream E7\. Intermediate probes often differ from E7, with the largest difference rate around63%63\\%\. Panel \(b\) compares mean probe entropy across HD models withn=2,4,8n\{=\}2,4,8\. The final stream has the lowest entropy in each setting; atn=8n\{=\}8, E7 has entropy2\.092\.09bits, while several intermediate streams remain above33bits\.Together, the top\-1 differences and higher entropy suggest that intermediate streams keep a broader token\-level candidate space, while the final stream consolidates it into the prediction\.

Figure[8](https://arxiv.org/html/2607.08186#S5.F8)provides qualitative examples of the same pattern\. Their role is illustrative: in these prompts, early or middle probes may favor template, structural, or associated tokens, such as “a”, “located”, or “Apollo”, while later probes move closer to the final stream’s token\. These examples show the broader\-candidate\-space interpretation at the prompt level\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x6.png)Figure 8:Illustrative stream\-probe predictions \(Qwen3\-8B\-Base \+ Hidden Decoding,n=8n\{=\}8\)\.For prompts with a clear answer, the figure shows each stream probe’s top\-1 token and its probability\. Green marks probes whose top\-1 matches the final stream; grey marks probes that differ\. These examples serve as qualitative illustrations of the aggregate differences in Figure[7](https://arxiv.org/html/2607.08186#S5.F7)\.Overall, the ablations and probes support the same interpretation\. The supervision\-design comparison shows that final\-stream supervision works better than training the intermediate streams to predict or summing their outputs\. The KV and attention analyses show that stream\-specific KV is useful, the streams separate inside the Transformer, and the final stream reads other streams\. The LM\-head probes provide an interpretability view: intermediate streams preserve a broader set of token alternatives before the final stream settles on a prediction\. Together, these results support the role of earlier streams as latent computation states\.

## 6Conclusion

We introducedHidden Decoding, a sequence\-length scaling method for improving a fixed Transformer backbone through continued pretraining\. The core idea is to give each token more internal computation by expanding it intonnstreams, while keeping the Transformer layers unchanged\. Hidden Decoding supervises only the final stream and retains the intermediate streams’ KV so their computation remains available to later tokens\. Stream\-Factorized Attention makes this expansion trainable by keeping most layers intra\-stream and limiting cross\-stream mixing to a subset of layers\. The ablations support these design choices: final\-stream supervision outperforms direct losses on every stream or explicit summation of stream outputs, and retained per\-stream KV improves accuracy in the controlled small\-model checks\.

The experiments show that this scaling route is useful and trainable at frontier scale\. Atn=4n\{=\}4, Hidden Decoding produces WeLM\-HD4\-80B and WeLM\-HD4\-617B and improves their matched non\-HD counterparts under the same early SFT\-only post\-training setup\. The training\-time measurements explain why this scale is practical: the4×4\\timesexpanded sequence costs5\.1×5\.1\\timeson 80B and4\.4×4\.4\\timeson 617B, close to linear and far below dense attention over the expanded sequence\. The expansion\-factor result shows that increasingnnimproves both the dense 8B model and the 80B MoE, with 80B MMLU rising from85\.185\.1to87\.587\.5and Pile\-test BPB falling from0\.3860\.386to0\.3780\.378\.

Together, these results establish sequence\-length expansion as a practical fixed\-backbone scaling path for frontier\-scale LLMs\. It does not require training a larger Transformer backbone, and it avoids the looped execution pattern that conflicts with pipeline\-parallel large\-model training\. Instead, the extra computation appears as a longer sequence, which fits the same engineering stack used to train very large MoE models\. Hidden Decoding therefore provides an effective and engineering\-realistic way to keep improving strong LLMs when further backbone scaling is costly\.

## References

## Appendix AContributors

Contributors are listed alphabetically by English name\.

Aiwei LiuCheng ShiChuhan WuCi LeiDi LuDonald HeFan ZhangFanhao KongFeifei ZhangGuan WangHaicheng WangHaoyu LiuHoujin YuJiachen DingJiayi FengJie ZhouJijun ChiJindi ShiJing LeiJunjie ZhangLaiyi LiLe TianLinhao ZhangMiao FanSijun ZhangWei JiaWeiwei ShiWenhan LiWentao ZhaoWenteng LiangXiao ZhouXiaojin ZhouXihuai WangXinyu GaoXuanliang WangXuyang AoYang YuYangxiu YouYinuo ZhaoYufei KuangYufei WangYuan Liu \(刘\)Yuan Liu \(柳\)Yuwen ChenZhencong TianZhongyin ZhaoZilin YuZitao Wang

## Appendix BMain WeLM Model Configurations

To make the main WeLM setting reproducible, Table[7](https://arxiv.org/html/2607.08186#A2.T7)lists the backbone and Hidden Decoding configuration used for the 80B and 617B comparisons in §[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)\. The non\-HD and HD forms share the same Transformer backbone\. Hidden Decoding changes the stream expansion and Stream\-Factorized Attention layout, while leaving the Transformer widths and layer counts unchanged\.

Table 7:Main WeLM backbone and HD configuration\.Activated parameters are per token\. “Q/KV heads” reports query heads and grouped key–value heads\. “Experts/top\-kk” includes one shared expert\. “KV mirror” is the number of mirrored late layers; “SFA” reports intra/local/full layers\.BackboneMoEparamsActivatedparamsLayersHiddensizeQ/KVheadsHeaddim\.Experts/ top\-kkKVmirrorSFAintra/local/fullWeLM\-80B80B3B49204824/2256512\+1512\{\+\}1/ 101620/23/6WeLM\-617B617B23B94409696/8128512\+1512\{\+\}1/ 103069/0/25
## Appendix CEvaluation Configurations

To make the reported scores reproducible, we separate the evaluation settings into two suites\. Table[8](https://arxiv.org/html/2607.08186#A3.T8)gives the standard suite used for scaling studies, ablations, and base\-model evaluations\. Table[9](https://arxiv.org/html/2607.08186#A3.T9)gives the early post\-training suite used for the main WeLM results in Table[2](https://arxiv.org/html/2607.08186#S4.T2)and the additional 80B results in Table[11](https://arxiv.org/html/2607.08186#A5.T11)\.

Table 8:Per\-benchmark evaluation configurations\.Execution\-based code benchmarks are scored at pass@1 with greedy decoding\.BenchmarkShotsCoTScoringKnowledge & multiple\-choiceMMLU5NoAccuracyMMLU\-Pro5NoAccuracyCMMLU5NoAccuracyC\-Eval5NoAccuracyARC\-Challenge25NoAccuracyHellaSwag10NoAccuracyEE\-GPQA5NoCustom evaluatorSuperGPQA5NoAccuracy \(A–K\)Reasoning & mathBBH3YesEM / AccuracyGSM8K4YesNumeric matchMATH4YesMATHEvaluator v2MATH Modern CoT4YesMATHEvaluator v2MATH LLM Judge4YesLLM\-as\-JudgeFactual QASimpleQA5NoSubstring matchChinese SimpleQA10NoLLM JudgeAA\-OmniScience10NoLLM JudgeCodeCRUXEval1 / 2YesExecution \(pass@1\)EvalPlus1NoExecution \(pass@1\)MultiPL\-E1NoExecution \(pass@1\)Table 9:Evaluation settings for the early post\-training suite \(Tables[2](https://arxiv.org/html/2607.08186#S4.T2)and[11](https://arxiv.org/html/2607.08186#A5.T11)\)\.NNis the number of independent runs per question; scores are averaged over theNNruns\. LLM\-judge benchmarks use a fixed judge model and temperature\.‡HMMT is the weighted average over the Feb\. 2025, Nov\. 2025, and Feb\. 2026 sets\.†External evaluations\.BenchmarkNNScoringGPQA Diamond4Accuracy \(multiple choice\)HMMT‡4Weighted average; LLM judge \(answer equiv\.\)IMO\-AnswerBench1LLM judge \(answer equiv\.\)MathArena Apex32LLM judge \(HLE\-style\)FrontierMath32LLM judge \(HLE\-style\)PolyMath1LLM judge \(HLE\-style\)PHYBench4LLM judge \(HLE\-style\)CritPt8Two\-call, CritPt APIHLE1LLM judge \(HLE\-style\)MMMLU1LLM judge \(HLE\-style\)AA\-OmniScience2LLM judge \(4\-way\)AA\-LCR3LLM judge \(equality\)SciCode1Execution \(sub\-step rate\)LiveCodeBench v62Execution \(pass@1\)IFBench2Programmatic checks \(pass@1\)ARC\-AGI\-21Exact grid match \(pass@2\)Terminal\-Bench 23Agent \+ verifier \(resolved rate\)GDPval†–Win rateτ2\\tau^\{2\}\-Bench†–RewardTable[8](https://arxiv.org/html/2607.08186#A3.T8)reports the few\-shot count, chain\-of\-thought \(CoT\) setting, and scoring method for each benchmark in the standard suite\. For SuperGPQA, EE\-GPQA, Chinese SimpleQA, and AA\-OmniScience, the few\-shot exemplars are pre\-formatted as a fixed prefix in the data rather than retrieved dynamically; the same exemplars are therefore shared across all examples of those benchmarks\. The three MATH variants share the same dataset and differ only in scoring: rule\-based equivalence \(v2\), the same scorer with extra Chinese stopping tokens \(modern CoT\), and an LLM judge\. This lets us separate scoring effects from model effects\.

The standard suite spans four families\. Knowledge and multiple\-choice benchmarks are scored by exact\-match accuracy on the predicted option letter, without chain\-of\-thought\. Reasoning and math benchmarks use chain\-of\-thought prompting and then extract the final answer\. Factual\-QA benchmarks ask for short answers, scored either by normalized substring matching \(SimpleQA\) or by an LLM judge \(Chinese SimpleQA, AA\-OmniScience\)\. Code benchmarks execute the generated programs against test cases and report pass@1 under greedy decoding\. The LLM\-judge benchmarks use a fixed judge model and temperature throughout, so scores stay comparable across our runs\.

Table[9](https://arxiv.org/html/2607.08186#A3.T9)lists the settings for the early post\-training suite\. Each question is runNNtimes and the per\-question scores are averaged;NNis as large as3232for the small, high\-variance math sets \(MathArena Apex, FrontierMath\)\. Generation is free\-form with a generous token budget \(up to160160k tokens per response\), so the models can use long chains of thought\. The scoring styles are as follows\.

Most math, science, and knowledge benchmarks are open\-ended: the model writes a full solution, and an LLM judge compares its final answer to the gold answer for equivalence, following the protocol of Humanity’s Last Exam\[phan2025hle\]\(we refer to this as HLE\-style judging\)\. This covers FrontierMath, MathArena Apex, PHYBench, PolyMath, MMMLU, and HLE itself; the math benchmarks use Qwen3\.5\-397B\-A17B as a fixed judge model\. GPQA Diamond is the exception—it is multiple\-choice, scored by whether the selected option matches the correct label\. AA\-OmniScience uses a four\-way judge \(correct / incorrect / partial / not\-attempted\) and reports accuracy, and AA\-LCR uses a general equality judge over long\-document questions\.

The remaining benchmarks are scored by execution or by task\-specific verifiers rather than a judge\. SciCode generates code for each sub\-step of a problem and runs it against reference assertions, scoring the fraction of sub\-steps that pass; LiveCodeBench v6 and IFBench run the generated programs \(or instruction\-checking code\) and report pass@1\. ARC\-AGI\-2 parses the predicted grid from the response and requires an exact match, reporting task\-level pass@2\. Terminal\-Bench 2 is agentic: for each task the model drives an agent inside a containerized environment, and an environment verifier returns a reward, which we report as the resolved rate\. CritPt makes two model calls per question—first solving the physics problem, then filling in a code template—and submits the result to the CritPt scoring API\. Across all LLM\-judge benchmarks the judge model and temperature are fixed throughout, so scores stay comparable across our runs\.

## Appendix DEarly SFT Hyperparameters

To make the early post\-training comparison in Table[2](https://arxiv.org/html/2607.08186#S4.T2)reproducible, we use the same SFT hyperparameters for each matched pair of non\-HD and HD models\. This supervised\-only stage tests whether the gains from continued pretraining persist after the same lightweight adaptation step\.

Table 10:Early SFT hyperparameters for the matched WeLM comparisons\.The same recipe is used for each matched pair of non\-HD and HD models in Table[2](https://arxiv.org/html/2607.08186#S4.T2); no reinforcement learning stage is applied\.ItemSettingOptimizerAdam \+ MuonAdam parametersβ1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.95\\beta\_\{2\}\{=\}0\.95,ε=10−8\\varepsilon\{=\}10^\{\-8\}Muon parametersMomentum0\.950\.95, Nesterov enabled,55Newton–Schulz stepsLearning rate2\.0×10−52\.0\{\\times\}10^\{\-5\}ScheduleCosine decay to0, with5656warmup iterations and11251125decay iterationsRegularizationWeight decay0\.10\.1; gradient clipping1\.01\.0Precisionbfloat16BatchingGlobal token batch4,194,3044\{,\}194\{,\}304; micro\-batch token budget262,144262\{,\}144Context extensionYaRN scaling factor88from a3232k base context; rotary base500,000500\{,\}000Adaptation scopeLoRA disabled; supervised fine\-tuning only, with no RL
## Appendix EAdditional Benchmarks for WeLM\-HD4\-80B

To check whether the 80B gains extend beyond the nine benchmarks shared across scales in Table[2](https://arxiv.org/html/2607.08186#S4.T2), we evaluate WeLM\-HD4\-80B on a broader suite covering agentic, code, instruction\-following, and long\-context tasks\. Table[11](https://arxiv.org/html/2607.08186#A5.T11)compares WeLM\-80B and WeLM\-HD4\-80B under the same early SFT\-only post\-training setup as §[4\.2](https://arxiv.org/html/2607.08186#S4.SS2)\. WeLM\-HD4\-80B improves eight of the ten benchmarks, with the largest gains on Terminal\-Bench 2 \(44\.9→58\.444\.9\\\!\\to\\\!58\.4\) and ARC\-AGI\-2 \(6\.9→11\.66\.9\\\!\\to\\\!11\.6\)\.

Table 11:Additional benchmarks for WeLM\-HD4\-80B\.The highlighted column is WeLM\-HD4\-80B, compared against the matched non\-HD WeLM\-80B under the same early SFT\-only post\-training recipe as Table[2](https://arxiv.org/html/2607.08186#S4.T2)\.†External evaluations outside our harness\.BenchmarkWeLM80BWeLM\-HD480BTerminal\-Bench 244\.958\.4ARC\-AGI\-26\.911\.6τ2\\tau^\{2\}\-Bench†75\.478\.9GDPval†39\.642\.9PolyMath59\.462\.3IFBench70\.773\.0LiveCodeBench v683\.785\.1CritPt3\.74\.9AA\-OmniScience27\.727\.6AA\-LCR69\.369\.3
## Appendix FStream\-Factorized Attention Layouts

To make the attention\-cost analysis in §[3](https://arxiv.org/html/2607.08186#S3)and the ablation in Table[4](https://arxiv.org/html/2607.08186#S4.T4)reproducible, Table[12](https://arxiv.org/html/2607.08186#A6.T12)lists the per\-model layer composition used for Stream\-Factorized Attention\. Intra\-stream layers attend only within a stream\. Cross\-stream layers mix information across streams and follow the base model’s attention pattern: “local” denotes sliding\-window cross\-stream attention, and “full” denotes full cross\-stream attention\.

The three layer types differ sharply in cost over the expanded length\-n​LnLsequence: an intra\-stream layer costsO​\(n​L2\)O\(nL^\{2\}\), a local cross\-stream layer costsO​\(n​L​w\)O\(nLw\)for window sizeww, and a full cross\-stream layer costsO​\(n2​L2\)O\(n^\{2\}L^\{2\}\)\. Full cross\-stream layers are the only layers with quadratic growth innn, so the layouts keep their number small while still letting every token mix across streams\. For base models that already interleave sliding\-window and full attention \(21B and 80B\), the local cross\-stream layers reuse the base model’s sliding\-window layers; the 617B base uses full attention throughout, so its cross\-stream layers are all full \(2525of9494\), with the rest converted to intra\-stream\.

Table 12:Per\-model Stream\-Factorized Attention layouts\.“Local” denotes sliding\-window cross\-stream layers \(used when the base model has SWA\); “Full” denotes full\-attention cross\-stream layers\. The three 21B rows correspond to the configurations ablated in Table[4](https://arxiv.org/html/2607.08186#S4.T4)\.ModelLayersIntraLocalFull21B, all\-full27002721B, SF \(44full\)271013421B, SF \(11full\)271313180B4920236617B9469025
## Appendix GProgressive Expansion: Full Results

To show the full expansion\-factor trajectory behind §[4\.3](https://arxiv.org/html/2607.08186#S4.SS3), Tables[13](https://arxiv.org/html/2607.08186#A7.T13)and[14](https://arxiv.org/html/2607.08186#A7.T14)report results on the 80B MoE that yields WeLM\-HD4\-80B atn=4n\{=\}4and on the dense Qwen3\-8B\-Base\. Only the embedding tables grow withnn; the backbone and the active computation per token are held fixed\. The best value in each row is in bold\.

We introduce the higher factors during continued pretraining at scheduled token counts, letting each factor converge before the next\. On the 80B MoE, we expand ton=2n\{=\}2,44, and88after0,503503B, and906906B tokens of the 32k continuation phase\. The tables report two complementary signals: a steady reduction in language\-modeling loss \(Pile\-test BPB on the 80B model, from0\.3860\.386to0\.3780\.378\) and consistent downstream gains, withn=8n\{=\}8best on almost every benchmark\. The few exceptions are non\-monotonic only at intermediate factors \(e\.g\., MBPP\+ on 80B dips atn=2n\{=\}2before recovering\), while then=8n\{=\}8model remains the strongest overall\.

Table 13:Progressive expansion on the 80B MoE toward WeLM\-HD4\-80B\.Active parameters per token are fixed at33B; only embedding parameters grow withnn\. BPB: lower is better\.BenchmarkShotsBasen=2n\{=\}2n=4n\{=\}4n=8n\{=\}8Embedding params–6\.1B12\.1B24\.2B48\.4BTraining tokens–1\.37T503B906B1\.01TPile\-test \(BPB\)↓\\downarrow–0\.3860\.3870\.3820\.378BBH \(EM\)387\.588\.390\.090\.6MMLU \(EM\)585\.185\.086\.787\.5C\-Eval \(EM\)588\.888\.989\.989\.5SimpleQA516\.715\.117\.218\.4Chinese SimpleQA560\.762\.263\.864\.7HumanEval\+161\.061\.059\.162\.2MBPP\+167\.764\.470\.271\.2MATH460\.458\.471\.471\.7Table 14:Progressive expansion on the dense Qwen3\-8B\-Base\.Total parameters are fixed at88B; only embedding parameters grow withnn\.BenchmarkShotsBasen=2n\{=\}2n=4n\{=\}4n=8n\{=\}8Embedding params–1\.2B1\.9B3\.1B5\.6BTraining tokens–180B75B150B187BBBH \(EM\)378\.881\.383\.083\.9MMLU \(EM\)579\.880\.981\.982\.2ARC\-C2593\.994\.394\.494\.7HellaSwag1079\.783\.185\.085\.3GSM8K492\.593\.393\.994\.6MATH456\.058\.260\.061\.1MBPP\+166\.769\.468\.769\.4
## Appendix HCPT Startup Loss under Attention Compositions

To test whether using more intra\-stream layers gives a smoother starting point for continued pretraining, we compare threen=4n\{=\}4attention compositions on a separate 617B\-scale checkpoint\. All runs start from the same step\-16000 checkpoint before HD expansion and enable the expanded streams at the next step\. The model, data, training schedule, and expansion factor are matched; the variants differ only in attention composition: no intra\-stream layers, a 1:1 cross\-/intra\-stream layout, and a 1:3 cross\-/intra\-stream layout\.

Figure[9](https://arxiv.org/html/2607.08186#A8.F9)shows the loss immediately after the expansion is enabled\. At the first expanded step, the losses are1\.591\.59for no intra\-stream layers,1\.451\.45for the 1:1 layout, and1\.141\.14for the 1:3 layout\. The layout with no intra\-stream layers is higher than the 1:3 layout by\+0\.45\+0\.45at step 1 and still by\+0\.33\+0\.33at step 6\. The ordering no intra\-stream\>\>1:1\>\>1:3 holds throughout the shared six\-step window\.

The run without intra\-stream layers and the 1:1 run stop after 6 and 8 steps, so this check is limited to startup behavior rather than final convergence\. The 1:3 layout continues training for 475 steps\. These results support the design role of intra\-stream layers: they reduce attention cost and also reduce the startup loss increase when the expanded model starts CPT from a checkpoint before HD expansion\.

![Refer to caption](https://arxiv.org/html/2607.08186v1/x7.png)Figure 9:CPT startup loss under different attention compositions\.All runs enablen=4n\{=\}4expansion from the same 617B\-scale step\-16000 checkpoint\.\(a\)The startup window compares no intra\-stream layers, a 1:1 cross\-/intra\-stream layout, and a 1:3 cross\-/intra\-stream layout\. The run without intra\-stream layers and the 1:1 run stop after 6 and 8 steps, so the comparison is restricted to early startup behavior\.\(b\)The 1:3 layout continues training for 475 steps\.
## Appendix IBase\-Model Results with and without Hidden Decoding

To check whether the gains appear before post\-training, Table[15](https://arxiv.org/html/2607.08186#A9.T15)compares each non\-HD Base model against itsn=4n\{=\}4Hidden Decoding Base model\.

Table 15:WeLM Base models with and without Hidden Decoding\.80B MoE617B MoEBenchmarkWeLM80B\-BaseWeLM\-HD480B\-BaseWeLM617B\-BaseWeLM\-HD4617B\-BaseMMLU88\.088\.989\.8590\.22MMLU\-Pro68\.471\.372\.2673\.92C\-Eval91\.391\.491\.992\.5SuperGPQA50\.552\.655\.3357\.05BBH90\.791\.992\.4193\.07MATH63\.662\.670\.9270\.82SimpleQA14\.915\.341\.4741\.75Chinese SimpleQA56\.057\.478\.1378\.8AA\-OmniScience22\.224\.936\.7837\.12HumanEval\+70\.172\.076\.276\.2MBPP\+70\.971\.471\.470\.9Average62\.4263\.6170\.6071\.12Hidden Decoding improves both scales before post\-training\. The average rises from62\.4262\.42to63\.6163\.61on 80B and from70\.6070\.60to71\.1271\.12on 617B\. The larger gains appear on harder reasoning and knowledge tasks such as SuperGPQA and MMLU\-Pro, and Table[2](https://arxiv.org/html/2607.08186#S4.T2)shows that they persist after the same lightweight post\-training recipe\.