TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

# TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing
Source: [https://arxiv.org/html/2608.07851](https://arxiv.org/html/2608.07851)
###### Abstract

Residual connections rely on a static residual pathway, and are essential for training deep neural networks\. Hyper\-connections \(HC\) increase the expressivity of residual routing by incorporating multiple residual streams and learning dynamic information flow, while manifold\-constrained \(mHC\) variants stabilize training through doubly stochastic residual mixing\. However, a generator\-level bottleneck remains in existing methods: they use dense, unstructured generators for pre\-branch aggregation, residual mixing, and post\-branch redistribution, which results in parameter count growing rapidly with the number of streams\. To address this issue, we proposeTensorizedEfficientManifold\-constrainedParameterization forExpressive ResidualRouting \(TEMPER\), which represents these generators as multi\-way tensors over the input\-stream, feature, and output\-stream modes, and parameterizes them using tensor networks\. Such a structured low\-rank formulation is shown to preserve token\-dependent manifold\-constrained routing interface while substantially reducing parameter growth\. It also promotes interpretability and intuition, as: i\) tensor ranks control the dimensionality of the learned routing subspace, with full ranks recovering dense routing; while ii\) the generator approximation errors bound differences in routing logits and, consequently, in the routed\-block outputs\. Comprehensive experiments show that TEMPER matches or outperforms existing methods across language modeling and commonsense reasoning tasks, while requiring substantially fewer additional parameters\. At eight residual streams, TEMPER achieves the best CORE score while using about84%84\\%fewer additional parameters than mHC, thus showing a stronger performance\-parameter efficiency trade\-off\.

## 1Introduction

Residual connections\(Heet al\.[2016](https://arxiv.org/html/2608.07851#bib.bib1)\)are central to the stability and scalability of modern Transformers\(Vaswaniet al\.[2017](https://arxiv.org/html/2608.07851#bib.bib2); Brownet al\.[2020](https://arxiv.org/html/2608.07851#bib.bib20)\)\. In a standard residual block, a module computes an update which is added back to the token’s current representation\. This design is simple and effective, but it routes every token via the same fixed residual path\.

![Refer to caption](https://arxiv.org/html/2608.07851v1/x1.png)Figure 1:Comparison of CORE score and additional parameter cost atn=8n=8\. Our TEMPER method provides the strongest trade\-off among all hyper\-connection variants\.Recently, hyper\-connections \(HC\)\(Zhuet al\.[2025](https://arxiv.org/html/2608.07851#bib.bib3)\)have enriched this design by maintaining multiple residual streams and learning token\-dependent routing among them, thus enabling more complex residual topologies and information flow patterns\. Instead of applying an identical and fixed residual connection to all tokens, these methods allow each token to adaptively aggregate, mix, and redistribute information across several streams\. Manifold\-constrained hyper\-connections \(mHC\)\(Xieet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib4)\)further improve the training stability of HC by using the iterative Sinkhorn\-Knopp \(SK\) optimization\(Sinkhorn[1964](https://arxiv.org/html/2608.07851#bib.bib9)\)to constrain the residual mixing matrix,𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}, to be doubly stochastic\. Subsequent variants have reparameterized this residual mixing step\. Among these, the mHC\-lite method\(Yang and Gao[2026](https://arxiv.org/html/2608.07851#bib.bib5)\)achieves exact doubly stochastic residual mixing by constructing𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}as a convex combination of permutation matrices\(Birkhoff[1946](https://arxiv.org/html/2608.07851#bib.bib19)\), while KromHC\(Zhouet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib6)\)uses Kronecker products of smaller doubly stochastic matrices\(Taranenko[2023](https://arxiv.org/html/2608.07851#bib.bib25)\), in order to avoid factorial growth of parameters with the number of streams incurred by mHC\-lite\. Figure[1](https://arxiv.org/html/2608.07851#S1.F1)summarizes the performance\-parameter trade\-offs of the existing methods and the proposed TEMPER, and highlights the remaining parameter efficiency gap\.

Although these HC variants improve the parameterization of the residual mixing matrix, the generator\-level bottleneck still remains an open issue\. For example, mHC uses unstructured dense generators to produce all three token\-dependent routing projectors from stream\-feature representations\. As the number of streams increases, these generators become increasingly expensive in terms of both parameter count and computation, limiting the scalability of the expressive token\-dependent routing\. This raises the following question:

![Refer to caption](https://arxiv.org/html/2608.07851v1/x2.png)Figure 2:Overview of TEMPER architecture\. TEMPER replaces unstructured dense hyper\-connections routing generators with tensorized TEMPER\-CP or TEMPER\-Tucker generators, while preserving the same token\-dependent routing interfaces\.- •Can we replace the dense unstructured routing generators in mHC with structured, parameter\-efficient alternatives, while achieving on par or even better performance?

To answer this question, we proposeTensorizedEfficientManifold\-constrainedParameterization forExpressive ResidualRouting \(TEMPER\), which replaces the dense routing generators with parameter\-efficient tensor\-network parameterizations\. By exploiting the inherently multi\-way structure of mHC routing, TEMPER represents the generators for pre\-branch aggregation, residual mixing, and post\-branch redistribution as higher\-order tensors over their input\-stream, feature, and output\-stream modes, and factorizes each independently using a low\-rank tensor network\.

TEMPER retains the original token\-dependent routing interface, including doubly stochastic residual mixing, while inducing structured parameter sharing within each generator across stream–feature–output interactions\. Its tensor ranks provide an explicit capacity control\. Compact ranks restrict routing to low\-dimensional learned subspaces, whereas sufficiently large ranks recover the dense parameterization\. TEMPER thus reduces routing\-generator parameter growth without altering the surrounding hyper\-connection architecture\.

Empirically, we instantiate TEMPER with Canonical Polyadic \(CP\)\(Hitchcock[1927](https://arxiv.org/html/2608.07851#bib.bib27)\)and Tucker\(Tucker[1966](https://arxiv.org/html/2608.07851#bib.bib28)\)parameterizations\. Both outperform prior hyper\-connection baselines in downstream performance while using substantially fewer additional parameters\. At eight residual streams, TEMPER\-Tucker obtains the best CORE score \(0\.2060\.206versus0\.1950\.195for both mHC and KromHC\) with1\.931\.93M additional parameters, which is84%84\\%fewer than11\.9511\.95M in mHC\.

In summary, our contributions are as follows:

- •We introduceTEMPER, a tensorized manifold\-constrained hyper\-connections method that replaces dense token\-dependent routing generators with structured efficient tensor\-network parameterizations, while preserving the original mHC routing interface\.
- •We analyze TEMPER’s scaling and expressivity\. At fixed rank, routing\-generator parameter grows linearly with the number of streams\. Full\-rank TEMPER\-Tucker recovers dense routing, while generator approximation errors bound routing\-logit and block\-output deviations\.
- •We instantiate TEMPER with CP and Tucker parameterizations and evaluate them on language modeling and commonsense reasoning\. TEMPER achieves a stronger performance–parameter efficiency trade\-off than the existing residual and hyper\-connection baselines\.

## 2Related Work

In a Transformer blockf​\(⋅\)f\(\\cdot\), standard residual connections\(Heet al\.[2016](https://arxiv.org/html/2608.07851#bib.bib1)\)update every token’s latent representation using the same pre\-defined residual pathway

𝐡′=𝐡\+f​\(𝐡\),\\mathbf\{h\}^\{\\prime\}=\\mathbf\{h\}\+f\(\\mathbf\{h\}\),\(1\)where𝐡\\mathbf\{h\}is the hidden representation\. While this design is computationally efficient and stable in optimization, it imposes the identical information flow pattern across all tokens\.

### 2\.1Hyper\-Connections Variants

#### Hyper\-connections \(HC\)

\(Zhuet al\.[2025](https://arxiv.org/html/2608.07851#bib.bib3)\)address this limitation by expanding the residual path tonnstreams and introducing token\-specific routing over these streams\. This increases the topological and representational complexity of the residual path, enabling richer and more diverse feature interactions across streams\.

Specifically, at a blockll, each token’s representation is repeatednntimes to obtain the expanded state matrix111The superscript\(l\)\(l\)is omitted for clarity from here onwards\.𝐇\(l\)=\[𝐡1\(l\)⊤,…,𝐡n\(l\)⊤\]⊤∈ℝn×d\\mathbf\{H\}^\{\(l\)\}=\\begin\{bmatrix\}\\mathbf\{h\}\_\{1\}^\{\(l\)\\top\},\\ldots,\\mathbf\{h\}\_\{n\}^\{\(l\)\\top\}\\\\ \\end\{bmatrix\}^\{\\top\}\\in\\mathbb\{R\}^\{n\\times d\}\. A routing module produces three token\-dependent projection matrices𝐚pre∈ℝ1×n\\mathbf\{a\}^\{\\mathrm\{pre\}\}\\in\\mathbb\{R\}^\{1\\times n\},𝐀res∈ℝn×n\\mathbf\{A\}^\{\\mathrm\{res\}\}\\in\\mathbb\{R\}^\{n\\times n\}, and𝐚post∈ℝ1×n\\mathbf\{a\}^\{\\mathrm\{post\}\}\\in\\mathbb\{R\}^\{1\\times n\}, with the resulting update

𝐇′=𝐀res​𝐇\+\(𝐚post\)T​f​\(𝐚pre​𝐇\)\.\\mathbf\{H\}^\{\\prime\}=\\mathbf\{A\}^\{\\mathrm\{res\}\}\\mathbf\{H\}\+\(\\mathbf\{a\}^\{\\mathrm\{post\}\}\)^\{T\}f\\\!\\left\(\\mathbf\{a\}^\{\\mathrm\{pre\}\}\\mathbf\{H\}\\right\)\.\(2\)Thus, the residual mixing is no longer globally fixed\. Instead, it is selected dynamically for each token\.

#### Manifold\-Constrained Hyper\-Connections \(mHC\)

\(Xieet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib4)\)constrains the residual mixing matrix,𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}, to be doubly stochastic, since an unconstrained matrix can disrupt the identity\-mapping behavior which is the key to training stability in standard residual networks\. Let𝐡^=vec​\(RMSNorm​\(𝐇\)\)∈ℝ1×n​d\\hat\{\\mathbf\{h\}\}=\\mathrm\{vec\}\(\\mathrm\{RMSNorm\}\(\\mathbf\{H\}\)\)\\in\\mathbb\{R\}^\{1\\times nd\}be the normalized flattened state\. Then, mHC generates the routing parameters through

𝐚pre\\displaystyle\\mathbf\{a\}^\{\\mathrm\{pre\}\}=σ​\(αpre​𝐡^​𝐖pre\+𝐛pre\),\\displaystyle=\\sigma\\\!\\left\(\\alpha^\{\\mathrm\{pre\}\}\\hat\{\\mathbf\{h\}\}\\mathbf\{W\}^\{\\mathrm\{pre\}\}\+\\mathbf\{b\}^\{\\mathrm\{pre\}\}\\right\),\(3\)𝐀res\\displaystyle\\mathbf\{A\}^\{\\mathrm\{res\}\}=SK​\(exp⁡\(mat​\(αres​𝐡^​𝐖res\+𝐛res\)\)\)\.\\displaystyle=\\mathrm\{SK\}\\\!\\left\(\\exp\\\!\\left\(\\mathrm\{mat\}\\\!\\left\(\\alpha^\{\\mathrm\{res\}\}\\hat\{\\mathbf\{h\}\}\\mathbf\{W\}^\{\\mathrm\{res\}\}\+\\mathbf\{b\}^\{\\mathrm\{res\}\}\\right\)\\right\)\\right\)\.𝐚post\\displaystyle\\mathbf\{a\}^\{\\mathrm\{post\}\}=2​σ​\(αpost​𝐡^​𝐖post\+𝐛post\),\\displaystyle=2\\sigma\\\!\\left\(\\alpha^\{\\mathrm\{post\}\}\\hat\{\\mathbf\{h\}\}\\mathbf\{W\}^\{\\mathrm\{post\}\}\+\\mathbf\{b\}^\{\\mathrm\{post\}\}\\right\),Here,𝐖pre,𝐖post∈ℝn​d×n,𝐛pre,𝐛post∈ℝ1×n,𝐖res∈ℝn​d×n2,𝐛res∈ℝ1×n2,\\mathbf\{W\}^\{\\mathrm\{pre\}\},\\mathbf\{W\}^\{\\mathrm\{post\}\}\\in\\mathbb\{R\}^\{nd\\times n\},\\mathbf\{b\}^\{\\mathrm\{pre\}\},\\mathbf\{b\}^\{\\mathrm\{post\}\}\\in\\mathbb\{R\}^\{1\\times n\},\\mathbf\{W\}^\{\\mathrm\{res\}\}\\in\\mathbb\{R\}^\{nd\\times n^\{2\}\},\\mathbf\{b\}^\{\\mathrm\{res\}\}\\in\\mathbb\{R\}^\{1\\times n^\{2\}\},αpre,αpost,αres∈ℝ\\alpha^\{\\mathrm\{pre\}\},\\alpha^\{\\mathrm\{post\}\},\\alpha^\{\\mathrm\{res\}\}\\in\\mathbb\{R\}, andσ​\(⋅\)\\sigma\(\\cdot\)denotes theSigmoid​\(⋅\)\\mathrm\{Sigmoid\}\(\\cdot\)function\. Moreover,SK​\(⋅\)\\mathrm\{SK\}\(\\cdot\)denotes the Sinkhorn–Knopp operator\(Sinkhorn[1964](https://arxiv.org/html/2608.07851#bib.bib9)\), which iteratively normalizes𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}towards a doubly stochastic matrix\. As a result, residual mixing becomes a non\-expansive convex combination over streams, which improves training stability\.

#### mHC\-lite and KromHC\.

The mHC\-lite method\(Yang and Gao[2026](https://arxiv.org/html/2608.07851#bib.bib5)\)replaces iterative SK normalization with a Birkhoff–von Neumann mixture, while KromHC\(Zhouet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib6)\)factorizes that mixture using smaller Kronecker factors\. Both leave the pre\- and post\-routing generators unchanged\. Their residual mixers can be summarized as

𝐀literes\\displaystyle\\mathbf\{A\}^\{\\mathrm\{res\}\}\_\{\\mathrm\{lite\}\}=∑m=1n\!πm​𝐏m,\\displaystyle=\\sum\_\{m=1\}^\{n\!\}\\pi\_\{m\}\\mathbf\{P\}\_\{m\},\(4\)𝐀Kromres\\displaystyle\\mathbf\{A\}^\{\\mathrm\{res\}\}\_\{\\mathrm\{Krom\}\}=⨂k=K1𝐔k,𝐔k=∑m=1ik\!πmk​𝐏mk,\\displaystyle=\\bigotimes\_\{k=K\}^\{1\}\\mathbf\{U\}^\{k\},\\qquad\\mathbf\{U\}^\{k\}=\\sum\_\{m=1\}^\{i\_\{k\}\!\}\\pi\_\{m\}^\{k\}\\mathbf\{P\}\_\{m\}^\{k\},wheren=∏k=1Kikn=\\prod\_\{k=1\}^\{K\}i\_\{k\}, and𝝅\\boldsymbol\{\\pi\}and𝝅k\\boldsymbol\{\\pi\}^\{k\}are token\-dependent SoftMax weights over the corresponding permutation matrices\. Both constructions are exactly doubly stochastic, but mHC\-lite requires𝒪​\(n\!\)\\mathcal\{O\}\(n\!\)mixture components, whereas KromHC reduces this cost via smaller Kronecker factors\.

### 2\.2Tensor Methods in Large Language Models \(LLMs\)

Tensor decompositions, by their nature, improve parameter efficiency across LLM components\. The Tensor Train decomposition\(Oseledets[2011](https://arxiv.org/html/2608.07851#bib.bib24)\)compresses dense layers\(Novikovet al\.[2015](https://arxiv.org/html/2608.07851#bib.bib10)\), while TensorGPT\(Xuet al\.[2023](https://arxiv.org/html/2608.07851#bib.bib23)\)extends this approach to token embeddings\. TensorLLM\(Guet al\.[2025](https://arxiv.org/html/2608.07851#bib.bib11)\)and Tucker Attention\(Kleinet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib12)\)apply Tucker\-style factorizations to multi\-head attention\. In parameter\-efficient fine\-tuning, LoRETTA\(Yanget al\.[2024](https://arxiv.org/html/2608.07851#bib.bib13)\)uses Tensor Train adapters, whereas TeRA\(Guet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib14)\)enables high\-rank adaptation with a Tucker\-like network\.

The performance of tensorized models depends critically on their ranks and network structures\.Zhouet al\.\([2025](https://arxiv.org/html/2608.07851#bib.bib15)\)study tensor\-network ranks as design parameters, whileZenget al\.\([2024](https://arxiv.org/html/2608.07851#bib.bib21)\)andIacovideset al\.\([2025](https://arxiv.org/html/2608.07851#bib.bib16)\)explore domain\-aware tensor\-network structure search with LLM assistance\.

## 3Preliminaries

#### Notation\.

Scalars, vectors, matrices, and tensors are denoted byxx,𝐱\\mathbf\{x\},𝐗\\mathbf\{X\}, and𝒳\\mathcal\{X\}, respectively\(Kolda and Bader[2009](https://arxiv.org/html/2608.07851#bib.bib17); Cichockiet al\.[2015](https://arxiv.org/html/2608.07851#bib.bib18),[2016](https://arxiv.org/html/2608.07851#bib.bib22)\)\. We use\(⋅\)⊤\(\\cdot\)^\{\\top\}for matrix transpose,𝒜​\(i1,…,iN\)\\mathcal\{A\}\(i\_\{1\},\\ldots,i\_\{N\}\)for the\(i1,…,iN\)\(i\_\{1\},\\ldots,i\_\{N\}\)\-th tensor entry,×n\\times\_\{n\}for the mode\-nnproduct,∘\\circfor the outer product, and⊗\\otimesfor the Kronecker product\. The operatorsvec⁡\(⋅\)\\operatorname\{vec\}\(\\cdot\)andmat⁡\(⋅\)\\operatorname\{mat\}\(\\cdot\)denote the vectorization and matricization, respectively\.

#### Tensor Contraction\.

Contractions are used to sum over shared modes of compatible tensors\. For𝒜∈ℝI1×I2×⋯×IN\\mathcal\{A\}\\in\\mathbb\{R\}^\{I\_\{1\}\\times I\_\{2\}\\times\\cdots\\times I\_\{N\}\}and𝐗∈ℝI1×I2\\mathbf\{X\}\\in\\mathbb\{R\}^\{I\_\{1\}\\times I\_\{2\}\}, the contraction over the first two modes is

\(𝒜×1,2𝐗\)i3,…,iN=∑u=1I1∑v=1I2𝒜u,v,i3,…,iN​𝐗u,v\.\(\\mathcal\{A\}\\times\_\{1,2\}\\mathbf\{X\}\)\_\{i\_\{3\},\\ldots,i\_\{N\}\}=\\sum\_\{u=1\}^\{I\_\{1\}\}\\sum\_\{v=1\}^\{I\_\{2\}\}\\mathcal\{A\}\_\{u,v,i\_\{3\},\\ldots,i\_\{N\}\}\\mathbf\{X\}\_\{u,v\}\.\(5\)For𝒜\\mathcal\{A\}andℬ\\mathcal\{B\}with modeaaof𝒜\\mathcal\{A\}and modebbofℬ\\mathcal\{B\}both of sizeRR, tensor–tensor contraction

\(𝒜×abℬ\)i1,…,iN−1,j1,…,jM−1=∑t=1R𝒜i1,…,iN−1,t​ℬt,j1,…,jM−1,\(\\mathcal\{A\}\\times\_\{a\}^\{b\}\\mathcal\{B\}\)\_\{i\_\{1\},\\ldots,i\_\{N\-1\},\\,j\_\{1\},\\ldots,j\_\{M\-1\}\}=\\sum\_\{t=1\}^\{R\}\\mathcal\{A\}\_\{i\_\{1\},\\ldots,i\_\{N\-1\},t\}\\mathcal\{B\}\_\{t,j\_\{1\},\\ldots,j\_\{M\-1\}\},\(6\)with all other modes remaining uncontracted\.

#### Unfolding\.

Unfolding, or matricization, reshapes a tensor into a matrix by mapping the indices of selected modes to the matrix rows and the indices of the remaining modes to the matrix columns\. For𝒜∈ℝI1×I2×⋯×IN\\mathcal\{A\}\\in\\mathbb\{R\}^\{I\_\{1\}\\times I\_\{2\}\\times\\cdots\\times I\_\{N\}\}, the unfolding that groups the first two modes against the remaining modes is given by

mat​\(𝒜\)∈ℝI1​I2×∏m=3NIm,\\mathrm\{mat\}\(\\mathcal\{A\}\)\\in\\mathbb\{R\}^\{I\_\{1\}I\_\{2\}\\times\\prod\_\{m=3\}^\{N\}I\_\{m\}\},\(7\)while the entries of the unfolded matrix are given by

mat​\(𝒜\)\(i1−1\)​I2\+i2,1\+∑m=3N\(im−1\)​∏ℓ=m\+1NIℓ=𝒜i1,i2,…,iN\.\\mathrm\{mat\}\(\\mathcal\{A\}\)\_\{\(i\_\{1\}\-1\)I\_\{2\}\+i\_\{2\},\\;1\+\\sum\_\{m=3\}^\{N\}\(i\_\{m\}\-1\)\\prod\_\{\\ell=m\+1\}^\{N\}I\_\{\\ell\}\}=\\mathcal\{A\}\_\{i\_\{1\},i\_\{2\},\\ldots,i\_\{N\}\}\.\(8\)

## 4Methodology

Existing hyper\-connection variants mainly differ in how they constrain or parameterize the residual mixing matrix,𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}\. Our focus is on re\-designing the dense routing generators through the lens of tensor factorization\. The overall architecture is illustrated in Figure[2](https://arxiv.org/html/2608.07851#S1.F2)\.

### 4\.1A Tensor View of Dense Routing Generators

We now focus on the generators that produce the routing projectors\. While prior methods differ in their construction of𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}, they all produce𝐚pre\\mathbf\{a\}^\{\\mathrm\{pre\}\},𝐀res\\mathbf\{A\}^\{\\mathrm\{res\}\}, and𝐚post\\mathbf\{a\}^\{\\mathrm\{post\}\}using dense maps\. Our TEMPER replaces these dense generators with tensorized maps, while preserving the same routing interface\.

Let𝐇^=RMSNorm​\(𝐇\)∈ℝn×d\\hat\{\\mathbf\{H\}\}=\\mathrm\{RMSNorm\}\(\\mathbf\{H\}\)\\in\\mathbb\{R\}^\{n\\times d\}denote the normalized stream state\. Instead of flattening𝐇^\\hat\{\\mathbf\{H\}\}into a length\-n​dndvector, we retain its stream and feature modes\. The dense generators can then be viewed as the unfolding of tensors𝒲pre∈ℝn×d×n\\mathcal\{W\}^\{\\mathrm\{pre\}\}\\in\\mathbb\{R\}^\{n\\times d\\times n\},𝒲post∈ℝn×d×n\\mathcal\{W\}^\{\\mathrm\{post\}\}\\in\\mathbb\{R\}^\{n\\times d\\times n\}, and𝒲res∈ℝn×d×n×n\\mathcal\{W\}^\{\\mathrm\{res\}\}\\in\\mathbb\{R\}^\{n\\times d\\times n\\times n\}, where the first two modes correspond to the input\-streams and features, and the remaining modes index the routing output\-streams dimensions\. With this tensor view, dense mHC generators can be re\-written as

𝐚pre\\displaystyle\\mathbf\{a\}^\{\\mathrm\{pre\}\}=σ​\(αpre​𝒵pre\+𝐛pre\),\\displaystyle=\\sigma\\\!\\Bigl\(\\alpha^\{\\mathrm\{pre\}\}\\mathcal\{Z\}^\{\\mathrm\{pre\}\}\+\\mathbf\{b\}^\{\\mathrm\{pre\}\}\\Bigr\),\(9\)𝐀res\\displaystyle\\mathbf\{A\}^\{\\mathrm\{res\}\}=SK​\(exp⁡\(αres​𝒵res\+mat​\(𝐛res\)\)\),\\displaystyle=\\mathrm\{SK\}\\\!\\Bigl\(\\exp\\\!\\Bigl\(\\alpha^\{\\mathrm\{res\}\}\\mathcal\{Z\}^\{\\mathrm\{res\}\}\+\\mathrm\{mat\}\(\\mathbf\{b\}^\{\\mathrm\{res\}\}\)\\Bigr\)\\Bigr\),𝐚post\\displaystyle\\mathbf\{a\}^\{\\mathrm\{post\}\}=2​σ​\(αpost​𝒵post\+𝐛post\),\\displaystyle=2\\sigma\\\!\\Bigl\(\\alpha^\{\\mathrm\{post\}\}\\mathcal\{Z\}^\{\\mathrm\{post\}\}\+\\mathbf\{b\}^\{\\mathrm\{post\}\}\\Bigr\),where the three routing logits𝒵pre,𝒵res,𝒵post\\mathcal\{Z\}^\{\\mathrm\{pre\}\},\\mathcal\{Z\}^\{\\mathrm\{res\}\},\\mathcal\{Z\}^\{\\mathrm\{post\}\}are obtained by contracting the input\-stream and feature modes as

𝒵pre\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{pre\}\}=𝒲pre×1,2𝐇^,\\displaystyle=\\mathcal\{W\}^\{\\mathrm\{pre\}\}\\times\_\{1,2\}\\hat\{\\mathbf\{H\}\},𝒵ipre\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{pre\}\}\_\{i\}=∑s=1n∑c=1dH^s,c​𝒲s,c,ipre,\\displaystyle=\\sum\_\{s=1\}^\{n\}\\sum\_\{c=1\}^\{d\}\\hat\{H\}\_\{s,c\}\\mathcal\{W\}^\{\\mathrm\{pre\}\}\_\{s,c,i\},\(10\)𝒵res\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{res\}\}=𝒲res×1,2𝐇^,\\displaystyle=\\mathcal\{W\}^\{\\mathrm\{res\}\}\\times\_\{1,2\}\\hat\{\\mathbf\{H\}\},𝒵i,jres\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{res\}\}\_\{i,j\}=∑s=1n∑c=1dH^s,c​𝒲s,c,i,jres,\\displaystyle=\\sum\_\{s=1\}^\{n\}\\sum\_\{c=1\}^\{d\}\\hat\{H\}\_\{s,c\}\\mathcal\{W\}^\{\\mathrm\{res\}\}\_\{s,c,i,j\},𝒵post\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{post\}\}=𝒲post×1,2𝐇^,\\displaystyle=\\mathcal\{W\}^\{\\mathrm\{post\}\}\\times\_\{1,2\}\\hat\{\\mathbf\{H\}\},𝒵ipost\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{post\}\}\_\{i\}=∑s=1n∑c=1dH^s,c​𝒲s,c,ipost\.\\displaystyle=\\sum\_\{s=1\}^\{n\}\\sum\_\{c=1\}^\{d\}\\hat\{H\}\_\{s,c\}\\mathcal\{W\}^\{\\mathrm\{post\}\}\_\{s,c,i\}\.
Thus, dense routing learns independent parameters for each stream\-feature\-output interaction, with no explicit sharing across modes\. This can be inefficient when routing decisions depend on reusable shared stream\-feature\-output patterns\. TEMPER introduces such parameter sharing by factorizing the generator tensors with structured tensor networks\.

### 4\.2Tensorizing the Routing Generators

The tensorization is achieved by keeping the contractions in Eq\. \([10](https://arxiv.org/html/2608.07851#S4.E10)\) fixed and by replacing each dense generator tensor with a structured CP or Tucker tensor factorization\.

#### TEMPER\-CP\.

TEMPER\-CP uses CP decomposition to factorize each generator tensor into rank\-one outer products

𝒲pre\\displaystyle\\mathcal\{W\}^\{\\mathrm\{pre\}\}=∑q=1r𝐮qpre∘𝐜qpre∘𝐯qpre,\\displaystyle=\\sum\_\{q=1\}^\{r\}\\mathbf\{u\}^\{\\mathrm\{pre\}\}\_\{q\}\\circ\\mathbf\{c\}^\{\\mathrm\{pre\}\}\_\{q\}\\circ\\mathbf\{v\}^\{\\mathrm\{pre\}\}\_\{q\},\(11\)𝒲res\\displaystyle\\mathcal\{W\}^\{\\mathrm\{res\}\}=∑q=1r𝐮qres∘𝐜qres∘𝐯1,qres∘𝐯2,qres,\\displaystyle=\\sum\_\{q=1\}^\{r\}\\mathbf\{u\}^\{\\mathrm\{res\}\}\_\{q\}\\circ\\mathbf\{c\}^\{\\mathrm\{res\}\}\_\{q\}\\circ\\mathbf\{v\}^\{\\mathrm\{res\}\}\_\{1,q\}\\circ\\mathbf\{v\}^\{\\mathrm\{res\}\}\_\{2,q\},𝒲post\\displaystyle\\mathcal\{W\}^\{\\mathrm\{post\}\}=∑q=1r𝐮qpost∘𝐜qpost∘𝐯qpost\.\\displaystyle=\\sum\_\{q=1\}^\{r\}\\mathbf\{u\}^\{\\mathrm\{post\}\}\_\{q\}\\circ\\mathbf\{c\}^\{\\mathrm\{post\}\}\_\{q\}\\circ\\mathbf\{v\}^\{\\mathrm\{post\}\}\_\{q\}\.
Here,𝐮q∈ℝn\\mathbf\{u\}\_\{q\}\\in\\mathbb\{R\}^\{n\}and𝐜q∈ℝd\\mathbf\{c\}\_\{q\}\\in\\mathbb\{R\}^\{d\}are factors for the input\-stream and feature mode, respectively, while𝐯q∈ℝn\\mathbf\{v\}\_\{q\}\\in\\mathbb\{R\}^\{n\}is the output\-stream mode factor\. For𝒲res\\mathcal\{W\}^\{\\mathrm\{res\}\},𝐯1,q,𝐯2,q∈ℝn\\mathbf\{v\}\_\{1,q\},\\mathbf\{v\}\_\{2,q\}\\in\\mathbb\{R\}^\{n\}denote factors for its two output\-stream modes\. Within each generator, the component index,qq, couples the mode\-specific factors in each rank\-one term, inducing structured parameter sharing across stream\-feature\-output interactions\.

#### TEMPER\-Tucker\.

TEMPER\-Tucker employs Tucker decomposition to factorize each generator tensor using factor matrices and a small tensor core as

𝒲pre\\displaystyle\\mathcal\{W\}^\{\\mathrm\{pre\}\}=𝒢pre×1𝐔pre×2𝐂pre×3𝐕pre,\\displaystyle=\\mathcal\{G\}^\{\\mathrm\{pre\}\}\\times\_\{1\}\\mathbf\{U\}^\{\\mathrm\{pre\}\}\\times\_\{2\}\\mathbf\{C\}^\{\\mathrm\{pre\}\}\\times\_\{3\}\\mathbf\{V\}^\{\\mathrm\{pre\}\},\(12\)𝒲res\\displaystyle\\mathcal\{W\}^\{\\mathrm\{res\}\}=𝒢res×1𝐔res×2𝐂res×3𝐕1res×4𝐕2res,\\displaystyle=\\mathcal\{G\}^\{\\mathrm\{res\}\}\\times\_\{1\}\\mathbf\{U\}^\{\\mathrm\{res\}\}\\times\_\{2\}\\mathbf\{C\}^\{\\mathrm\{res\}\}\\times\_\{3\}\\mathbf\{V\}^\{\\mathrm\{res\}\}\_\{1\}\\times\_\{4\}\\mathbf\{V\}^\{\\mathrm\{res\}\}\_\{2\},𝒲post\\displaystyle\\mathcal\{W\}^\{\\mathrm\{post\}\}=𝒢post×1𝐔post×2𝐂post×3𝐕post\.\\displaystyle=\\mathcal\{G\}^\{\\mathrm\{post\}\}\\times\_\{1\}\\mathbf\{U\}^\{\\mathrm\{post\}\}\\times\_\{2\}\\mathbf\{C\}^\{\\mathrm\{post\}\}\\times\_\{3\}\\mathbf\{V\}^\{\\mathrm\{post\}\}\.We use Tucker ranks\(rn,rd,rn\)\(r\_\{n\},r\_\{d\},r\_\{n\}\)for the pre\- and post\-branch generators and\(rn,rd,rn,rn\)\(r\_\{n\},r\_\{d\},r\_\{n\},r\_\{n\}\)for the residual generator, with input\- and output\-stream factors𝐔,𝐕∈ℝn×rn\\mathbf\{U\},\\mathbf\{V\}\\in\\mathbb\{R\}^\{n\\times r\_\{n\}\}and feature factors𝐂∈ℝd×rd\\mathbf\{C\}\\in\\mathbb\{R\}^\{d\\times r\_\{d\}\}\. Given𝐇^∈ℝn×d\\hat\{\\mathbf\{H\}\}\\in\\mathbb\{R\}^\{n\\times d\}, the input is first projected into a compressed stream–feature representation:

𝐓pre\\displaystyle\\mathbf\{T\}^\{\\mathrm\{pre\}\}=\(𝐔pre\)⊤​𝐇^​𝐂pre,\\displaystyle=\(\\mathbf\{U\}^\{\\mathrm\{pre\}\}\)^\{\\top\}\\hat\{\\mathbf\{H\}\}\\mathbf\{C\}^\{\\mathrm\{pre\}\},\(13\)𝐓res\\displaystyle\\mathbf\{T\}^\{\\mathrm\{res\}\}=\(𝐔res\)⊤​𝐇^​𝐂res,\\displaystyle=\(\\mathbf\{U\}^\{\\mathrm\{res\}\}\)^\{\\top\}\\hat\{\\mathbf\{H\}\}\\mathbf\{C\}^\{\\mathrm\{res\}\},𝐓post\\displaystyle\\mathbf\{T\}^\{\\mathrm\{post\}\}=\(𝐔post\)⊤​𝐇^​𝐂post\.\\displaystyle=\(\\mathbf\{U\}^\{\\mathrm\{post\}\}\)^\{\\top\}\\hat\{\\mathbf\{H\}\}\\mathbf\{C\}^\{\\mathrm\{post\}\}\.The routing logits are then obtained by contracting this representation with the core and the output modes:

𝒵pre\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{pre\}\}=\(𝒢pre×1,2𝐓pre\)×1𝐕pre,\\displaystyle=\(\\mathcal\{G\}^\{\\mathrm\{pre\}\}\\times\_\{1,2\}\\mathbf\{T\}^\{\\mathrm\{pre\}\}\)\\times\_\{1\}\\mathbf\{V\}^\{\\mathrm\{pre\}\},\(14\)𝒵res\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{res\}\}=\(𝒢res×1,2𝐓res\)×1𝐕1res×2𝐕2res,\\displaystyle=\(\\mathcal\{G\}^\{\\mathrm\{res\}\}\\times\_\{1,2\}\\mathbf\{T\}^\{\\mathrm\{res\}\}\)\\times\_\{1\}\\mathbf\{V\}^\{\\mathrm\{res\}\}\_\{1\}\\times\_\{2\}\\mathbf\{V\}^\{\\mathrm\{res\}\}\_\{2\},𝒵post\\displaystyle\\mathcal\{Z\}^\{\\mathrm\{post\}\}=\(𝒢post×1,2𝐓post\)×1𝐕post\.\\displaystyle=\(\\mathcal\{G\}^\{\\mathrm\{post\}\}\\times\_\{1,2\}\\mathbf\{T\}^\{\\mathrm\{post\}\}\)\\times\_\{1\}\\mathbf\{V\}^\{\\mathrm\{post\}\}\.TEMPER\-Tucker therefore constrains routing to depend on a compressed stream\-feature representation rather than on an unrestricted dense map\.

#### Parameter Efficiency\.

Factorizing the routing generators substantially reduces the parameter cost of token\-dependent routing\. The pre\-, residual\-, and post\-routing outputs lie inℝn\\mathbb\{R\}^\{n\},ℝn×n\\mathbb\{R\}^\{n\\times n\}, andℝn\\mathbb\{R\}^\{n\}, respectively\. Counting generator weights only, the parameter count for each parameterization is

PmHC\\displaystyle P\_\{\\mathrm\{mHC\}\}=d​\(n3\+2​n2\),\\displaystyle=d\(n^\{3\}\+2n^\{2\}\),\(15\)PCP\\displaystyle P\_\{\\mathrm\{CP\}\}=3​d​r\+7​n​r,\\displaystyle=3dr\+7nr,PTucker\\displaystyle P\_\{\\mathrm\{Tucker\}\}=7​n​rn\+3​d​rd\+2​rn2​rd\+rn3​rd\.\\displaystyle=7nr\_\{n\}\+3dr\_\{d\}\+2r\_\{n\}^\{2\}r\_\{d\}\+r\_\{n\}^\{3\}r\_\{d\}\.Here, the rankrris shared across the three TEMPER\-CP generators, while TEMPER\-Tucker uses input\- and output\-stream rank,rnr\_\{n\}, and feature rank,rdr\_\{d\}\. Thus, for fixedddand tensor ranks, dense mHC scales as𝒪​\(n3\)\\mathcal\{O\}\(n^\{3\}\)in the number of streams, whereas both TEMPER parameterizations scale as𝒪​\(n\)\\mathcal\{O\}\(n\)\. Figure[3](https://arxiv.org/html/2608.07851#S4.F3)reports the complete learnable parameter counts in hyper\-connections, including biases, scalar gates, and RMSNorm scales\.

![Refer to caption](https://arxiv.org/html/2608.07851v1/x3.png)Figure 3:Trainable parameters per hyper\-connection, including routing generators, biases, scalar gates, and RMSNorm\. TEMPER scales more favorably with the number of streams\.Table 1:Additional parameters, training loss, validation BPB, and CORE score for hyper\-connections variants withD=12D=12blocks and two residual connections per block\. Then=4n=4andn=8n=8settings are respectively trained for 6,000 and 7,000 steps\.

### 4\.3Inductive Bias and Approximation Guarantees

As TEMPER\-Tucker routes through the three compressed representations in Eq\. \([13](https://arxiv.org/html/2608.07851#S4.E13)\), its ranks restrict which stream\-feature directions affect routing and control how closely the tensorized generators approximate their dense counterparts\. We formalize both properties below\. TEMPER\-CP is a special case of TEMPER\-Tucker with a superdiagonal core\.

###### Theorem 1\.

\(Rank and Null\-space Dimension of the Tucker Compression\) For any of the three generators in Eq\. \([12](https://arxiv.org/html/2608.07851#S4.E12)\), assume that𝐔∈ℝn×rn\\mathbf\{U\}\\in\\mathbb\{R\}^\{n\\times r\_\{n\}\}and𝐂∈ℝd×rd\\mathbf\{C\}\\in\\mathbb\{R\}^\{d\\times r\_\{d\}\}have full column rank\. Given the normalized input𝐇^∈ℝn×d\\hat\{\\mathbf\{H\}\}\\in\\mathbb\{R\}^\{n\\times d\}, define

L:ℝn×d→ℝrn×rd,L​\(𝐇^\)=𝐔⊤​𝐇^​𝐂\.L:\\mathbb\{R\}^\{n\\times d\}\\to\\mathbb\{R\}^\{r\_\{n\}\\times r\_\{d\}\},\\quad L\(\\hat\{\\mathbf\{H\}\}\)=\\mathbf\{U\}^\{\\top\}\\hat\{\\mathbf\{H\}\}\\mathbf\{C\}\.\(16\)Then,rank⁡\(L\)=rn​rd\\operatorname\{rank\}\(L\)=r\_\{n\}r\_\{d\}anddim𝒩​\(L\)=n​d−rn​rd\\dim\\mathcal\{N\}\(L\)=nd\-r\_\{n\}r\_\{d\}\.

Theorem[1](https://arxiv.org/html/2608.07851#Thmtheorem1)shows that routing depends only on anrn​rdr\_\{n\}r\_\{d\}\-dimensional learned subspace and is invariant to the remaining directions\. Increasing ranks enlarges this subspace, whereas low ranks impose a stronger bottleneck that may suppress noisy or redundant directions\. At full multilinear rank, TEMPER\-Tucker recovers the dense mHC generator, so Tucker ranks interpolate between compact and dense routing\.

###### Lemma 1\.

\(Generator\-to\-logit Error Bound\)\.Let𝒲\\mathcal\{W\}and𝒲~\\widetilde\{\\mathcal\{W\}\}denote any corresponding dense and tensorized routing generators\. For any normalized stream state𝐇^\\hat\{\\mathbf\{H\}\}, define𝒵𝒲​\(𝐇^\)=𝒲×1,2𝐇^\\mathcal\{Z\}\_\{\\mathcal\{W\}\}\(\\hat\{\\mathbf\{H\}\}\)=\\mathcal\{W\}\\times\_\{1,2\}\\hat\{\\mathbf\{H\}\}, and similarly for𝒲~\\widetilde\{\\mathcal\{W\}\}\. Then, the corresponding routing logits satisfy

‖𝒵𝒲​\(𝐇^\)−𝒵𝒲~​\(𝐇^\)‖F≤‖𝐇^‖F​‖𝒲−𝒲~‖F\.\\left\\\|\\mathcal\{Z\}\_\{\\mathcal\{W\}\}\(\\hat\{\\mathbf\{H\}\}\)\-\\mathcal\{Z\}\_\{\\widetilde\{\\mathcal\{W\}\}\}\(\\hat\{\\mathbf\{H\}\}\)\\right\\\|\_\{F\}\\leq\\\|\\hat\{\\mathbf\{H\}\}\\\|\_\{F\}\\left\\\|\\mathcal\{W\}\-\\widetilde\{\\mathcal\{W\}\}\\right\\\|\_\{F\}\.\(17\)

###### Theorem 2\.

\(End\-to\-End Routed\-Block Approximation Bound\) Let𝐇^=RMSNorm​\(𝐇\)\\hat\{\\mathbf\{H\}\}=\\mathrm\{RMSNorm\}\(\\mathbf\{H\}\)be the normalized stream state, and𝐇′\\mathbf\{H\}^\{\\prime\}and𝐇~′\\widetilde\{\\mathbf\{H\}\}^\{\\prime\}the outputs of the dense and tensorized routed blocks\. Letffbe the shared residual transformation\. Suppose thatffisLfL\_\{f\}\-Lipschitz and satisfies‖f​\(𝐱\)‖2≤Mf\\\|f\(\\mathbf\{x\}\)\\\|\_\{2\}\\leq M\_\{f\}for all relevant inputs, and thatΦ​\(𝐙\)=SK⁡\(exp⁡\(𝐙\)\)\\Phi\(\\mathbf\{Z\}\)=\\operatorname\{SK\}\\\!\\left\(\\exp\(\\mathbf\{Z\}\)\\right\)isLSKL\_\{\\mathrm\{SK\}\}\-Lipschitz on the residual\-logit domain\. Then,

‖𝐇′−𝐇~′‖F\\displaystyle\\left\\\|\\mathbf\{H\}^\{\\prime\}\-\\widetilde\{\\mathbf\{H\}\}^\{\\prime\}\\right\\\|\_\{F\}≤\|αres\|​LSK​‖𝐇^‖F​‖𝐇‖F​‖𝒲res−𝒲~res‖F\\displaystyle\\leq\|\\alpha^\{\\mathrm\{res\}\}\|L\_\{\\mathrm\{SK\}\}\\\|\\hat\{\\mathbf\{H\}\}\\\|\_\{F\}\\\|\\mathbf\{H\}\\\|\_\{F\}\\left\\\|\\mathcal\{W\}^\{\\mathrm\{res\}\}\-\\widetilde\{\\mathcal\{W\}\}^\{\\mathrm\{res\}\}\\right\\\|\_\{F\}\(18\)\+\|αpost\|2​Mf​‖𝐇^‖F​‖𝒲post−𝒲~post‖F\\displaystyle\+\\frac\{\|\\alpha^\{\\mathrm\{post\}\}\|\}\{2\}M\_\{f\}\\\|\\hat\{\\mathbf\{H\}\}\\\|\_\{F\}\\left\\\|\\mathcal\{W\}^\{\\mathrm\{post\}\}\-\\widetilde\{\\mathcal\{W\}\}^\{\\mathrm\{post\}\}\\right\\\|\_\{F\}\+\|αpre\|2​n​Lf​‖𝐇^‖F​‖𝐇‖F​‖𝒲pre−𝒲~pre‖F\.\\displaystyle\+\\frac\{\|\\alpha^\{\\mathrm\{pre\}\}\|\}\{2\}\\sqrt\{n\}\\,L\_\{f\}\\\|\\hat\{\\mathbf\{H\}\}\\\|\_\{F\}\\\|\\mathbf\{H\}\\\|\_\{F\}\\left\\\|\\mathcal\{W\}^\{\\mathrm\{pre\}\}\-\\widetilde\{\\mathcal\{W\}\}^\{\\mathrm\{pre\}\}\\right\\\|\_\{F\}\.

Lemma[1](https://arxiv.org/html/2608.07851#Thmlemma1)quantifies how tensor approximation error propagates to the routing logits\. It shows that the discrepancy between dense and tensorized routing logits is bounded by the generator error‖𝒲−𝒲~‖F\\left\\\|\\mathcal\{W\}\-\\widetilde\{\\mathcal\{W\}\}\\right\\\|\_\{F\}scaled by the normalized input magnitude‖𝐇^‖F\\\|\\hat\{\\mathbf\{H\}\}\\\|\_\{F\}\. Since RMS normalization keeps this magnitude controlled, the resulting logit discrepancy is governed primarily by the quality of tensor approximation\.

Theorem[2](https://arxiv.org/html/2608.07851#Thmtheorem2)extends this result to the complete routed block\. Under the regularity assumptions, the block\-output difference is bounded by the approximation errors of the pre\-, residual\-, and post\-routing generators\. Together, these results show how approximation errors are controlled as the routing parameterization moves from compact low\-rank structure towards the dense unrestricted model\. These guarantees consider expressivity and approximation, rather than generalization, and Section[5\.4](https://arxiv.org/html/2608.07851#S5.SS4)empirically examines why increasing the rank might not necessarily improve performance\. The proofs of theorems and lemma are given in the Supplement Section 1\.

## 5Experiments

To evaluate the performance of TEMPER, we used the nanochat\(Karpathy[2025](https://arxiv.org/html/2608.07851#bib.bib7)\)backbone and apply different hyper\-connections modules in different methods\. The base model hadD=12D=12Transformer blocks, hidden dimensiond=768d=768,66attention heads, context length20482048\. We replaced residual branches in both attention and feed\-forward network in every block, yielding2424residual modules per model\. The pretraining used ClimbMix\-400B dataset\(Diaoet al\.[2025](https://arxiv.org/html/2608.07851#bib.bib8)\)with a32,76832\{,\}768\-token BPE tokenizer\.4×4\\timesNVIDIA A100 \(80GB\) GPUs are used for the experiments\. All hyperparameters are provided in Supplement Section 4\.

### 5\.1Initialization

We followed KromHC\(Zhouet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib6)\)for routing\-bias and scalar\-gate initialization, yielding stream\-biased pre\- and post\-routing and near\-identity residual mixing\. Beyond this, all TEMPER\-CP factor entries were initialized independently from𝒩​\(0,r−1/2\)\\mathcal\{N\}\(0,r^\{\-1/2\}\), whererris the CP rank\. For TEMPER\-Tucker, all factor and core entries are initialized independently from𝒩​\(0,rmax−1/2\)\\mathcal\{N\}\(0,r\_\{\\max\}^\{\-1/2\}\), wherermaxr\_\{\\max\}is the maximum mode rank\. This rank\-scaled initialization keeps factor magnitudes controlled and improves optimization stability\.

Table 2:Commonsense reasoning results for two stream counts\. Best results are bolded\. Second best results are underlined\.Table 3:Language modeling results for the two stream counts\. Best results are bolded\. Second best results are underlined\.
### 5\.2Evaluation Metrics

Following KromHC\(Zhouet al\.[2026](https://arxiv.org/html/2608.07851#bib.bib6)\), to measure efficiency and language\-modeling quality, we report additional learnable parameters, training cross\-entropy \(CE\) loss, and validation bits\-per\-byte \(BPB\)\. Additionally, we report the CORE score\(Liet al\.[2024](https://arxiv.org/html/2608.07851#bib.bib26)\), the mean\-centered accuracy over 22 tasks \(20 shown here\), as a summary of downstream performance \(See Supplement Section 3\)\. For each task, we report task accuracy \(%\)\. Language\-modeling tasks were scored by exact continuation match, while multiple\-choice tasks select the option with the lowest continuation loss\.

### 5\.3Downstream Performance

Tables[1](https://arxiv.org/html/2608.07851#S4.T1),[2](https://arxiv.org/html/2608.07851#S5.T2), and[3](https://arxiv.org/html/2608.07851#S5.T3)report downstream performance forn=4n=4andn=8n=8hyper\-connection streams\. Across both settings, TEMPER provides the strongest performance–parameter efficiency trade\-off among the hyper\-connection baselines, matching or improving the downstream performance with substantially fewer additional parameters\. mHC\-lite results atn=8n=8are unavailable because its𝒪​\(n\!\)\\mathcal\{O\}\(n\!\)residual\-routing parameter growth led to an out\-of\-memory \(OOM\) error\.

As shown in Table[1](https://arxiv.org/html/2608.07851#S4.T1), TEMPER\-Tucker achieves the highest CORE score at bothn=4n=4andn=8n=8, improving over KromHC from0\.1860\.186to0\.1920\.192and from0\.1950\.195to0\.2060\.206, respectively\. Atn=8n=8, it uses1\.931\.93M additional parameters, compared with3\.393\.39M for KromHC and11\.9511\.95M for mHC\.

TEMPER\-Tucker also achieves the highest average accuracy on the commonsense and reasoning benchmark in Table[2](https://arxiv.org/html/2608.07851#S5.T2), reaching50\.0%50\.0\\%and51\.3%51\.3\\%, respectively\. Atn=8n=8, it outperformed the strongest existing baseline by1\.5%1\.5\\%\. For the language\-modeling and BBH\-oriented suite in Table[3](https://arxiv.org/html/2608.07851#S5.T3), TEMPER\-CP achieves the highest average accuracy in both stream counts, with27\.1%27\.1\\%forn=4n=4and27\.3%27\.3\\%forn=8n=8\.

These results suggest an expressivity–regularization trade\-off\. TEMPER\-Tucker’s learned core captures richer cross\-mode interactions, which may benefit tasks requiring diverse routing patterns, such as commonsense reasoning\. In comparison, TEMPER\-CP imposes a stronger separable prior with greater parameter sharing, which may improve generalization on language\-modeling and BBH\-oriented tasks\.

### 5\.4Ablation Studies

#### Frozen Tucker Core

![Refer to caption](https://arxiv.org/html/2608.07851v1/x4.png)Figure 4:Smoothed validation BPB gaps relative to TEMPER\-Tucker for the frozen\-core and TT variant atn=4n=4andn=8n=8\. Positive values indicate worse BPB\.Eq\. \([14](https://arxiv.org/html/2608.07851#S4.E14)\) separates Tucker routing into two stages\. First, factor matrices construct a compressed input\-stream\-feature and output\-stream representation, and then the core tensor models interactions among the compressed coordinates\. The frozen\-core variant isolates these roles by keeping the factor matrices trainable but fixing the core at initialization\. As shown in Figure[4](https://arxiv.org/html/2608.07851#S5.F4), the frozen\-core model has consistently worse validation BPB than the fully trainable Tucker model\. Thus, learning the projection subspaces alone is insufficient\. Adapting the interactions among those subspaces also contributes to the routing quality\.

#### Effect of Tensor Network Topology

Beyond the CP and Tucker decompositions, we evaluated a Tensor Train \(TT\) variant \(See Supplement Section 2\), given by

𝒲=𝒢1×31𝒢2×41𝒢3×51⋯×m\+11𝒢m,\\mathcal\{W\}=\\mathcal\{G\}\_\{1\}\\times\_\{3\}^\{1\}\\mathcal\{G\}\_\{2\}\\times\_\{4\}^\{1\}\\mathcal\{G\}\_\{3\}\\times\_\{5\}^\{1\}\\cdots\\times\_\{m\+1\}^\{1\}\\mathcal\{G\}\_\{m\},\(19\)wherem=3m=3for the pre\- and post\-branch generators andm=4m=4for the residual generator\. TT represents each generator as a chain of low\-rank cores, rather than Tucker’s single global core\. Figure[4](https://arxiv.org/html/2608.07851#S5.F4)shows that TT has worse validation BPB than Tucker at bothn=4n=4andn=8n=8, suggesting that a global core better suits this routing setting\.

#### Tensorizing only𝐖res\\mathbf\{W\}^\{\\text\{res\}\}

![Refer to caption](https://arxiv.org/html/2608.07851v1/x5.png)Figure 5:Ablations on TEMPER routing design\.Left: CORE score comparison at the final step for full TEMPER\-Tucker against a residual\-only variant that tensorizes𝐖res\\mathbf\{W\}^\{\\mathrm\{res\}\}only\.Right: CORE score at final step for TEMPER\-CP with ranksr∈\{2,4,8,16,32\}r\\in\\\{2,4,8,16,32\\\}atn=4n=4\.Theorem[2](https://arxiv.org/html/2608.07851#Thmtheorem2)decomposes the block\-output discrepancy into contributions from the three routing generators\. To isolate the effect of the residual generator, we tensorized only𝐖res\\mathbf\{W\}^\{\\text\{res\}\}, while keeping𝐖pre\\mathbf\{W\}^\{\\text\{pre\}\}and𝐖post\\mathbf\{W\}^\{\\text\{post\}\}dense\. As shown in Figure[5](https://arxiv.org/html/2608.07851#S5.F5)\(left\), this variant underperformed full TEMPER\-Tucker, whose CORE score increased from0\.1760\.176to0\.1920\.192atn=4n=4, and from0\.1890\.189to0\.2060\.206atn=8n=8\. These results show that tensorizing the residual generator,𝐖res\\mathbf\{W\}^\{\\text\{res\}\}, is beneficial, while jointly tensorizing all three generators yields the best performance\.

#### Tensor ranks

Theorem[1](https://arxiv.org/html/2608.07851#Thmtheorem1)shows that tensor rank controls the dimension of the learned stream\-feature subspace available for routing\. Figure[5](https://arxiv.org/html/2608.07851#S5.F5)\(right\) studies the effect of CP rank atn=4n=4under a fixed 6000\-step budget\. The final CORE score varied only from0\.1840\.184to0\.1890\.189, with the best results atr=2r=2\. This weak performance sensitivity to rank suggests that the dense generator is over\-parameterized for routing in this setting\. A small number of shared low\-rank components provides sufficient expressivity, while the resulting structural constraint may also act as an implicit regularizer\.

## 6Conclusion

We have introduced TEMPER, which jointly tensorizes pre\-aggregation, residual\-mixing, and post\-redistribution routing generators in hyper\-connections, reducing parameter growth while retaining effective routing capacity\. Our theoretical analysis characterizes routing capacity through tensor ranks and bounds how generator approximation errors propagate to routing logits and block outputs\. Across language modeling and reasoning benchmarks, TEMPER has matched or outperformed prior hyper\-connection methods, while requiring substantially fewer additional parameters, providing a scalable approach to expressive hyper\-connections in LLMs\.

## References

- G\. Birkhoff \(1946\)Tres Observaciones Sobre el Algebra Lineal\.Rev\. Univ\. Nac\. Tucumán Ser\. A5,pp\. 147–151\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language Models are Few\-Shot Learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p1.1)\.
- A\. Cichocki, N\. Lee, I\. Oseledets, A\. Phan, Q\. Zhao, and D\. P\. Mandic \(2016\)Tensor Networks for Dimensionality Reduction and Large\-scale Optimization Part 1 Low\-rank Tensor Decompositions\.Foundations and Trends in Machine Learning9\(4\-5\),pp\. 249–429\.Cited by:[§3](https://arxiv.org/html/2608.07851#S3.SS0.SSS0.Px1.p1.13)\.
- A\. Cichocki, D\. Mandic, L\. De Lathauwer, G\. Zhou, Q\. Zhao, C\. Caiafa, and H\. A\. Phan \(2015\)Tensor Decompositions for Signal Processing Applications: From Two\-way to Multiway Component Analysis\.IEEE signal processing magazine32\(2\),pp\. 145–163\.Cited by:[§3](https://arxiv.org/html/2608.07851#S3.SS0.SSS0.Px1.p1.13)\.
- S\. Diao, Y\. Yang, Y\. Fu, X\. Dong, D\. Su, M\. Kliegl, Z\. Chen, P\. Belcak, Y\. Suhara, H\. Yin, M\. Patwary, C\. Lin, J\. Kautz, and P\. Molchanov \(2025\)CLIMB: CLustering\-based Iterative Data Mixture Bootstrapping for Language Model Pre\-training\.arXiv preprint\.External Links:[Link](https://arxiv.org/abs/2504.13161)Cited by:[§5](https://arxiv.org/html/2608.07851#S5.p1.7)\.
- Y\. Gu, W\. Zhou, G\. Iacovides, and D\. Mandic \(2025\)TensorLLM: Tensorising Multi\-Head Attention for Enhanced Reasoning and Compression in LLMs\.In2025 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- Y\. Gu, W\. Zhou, G\. Iacovides, and D\. Mandic \(2026\)TeRA: Vector\-based Random Tensor Network for High\-rank Adaptation of Large Language Models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2314–2329\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep Residual Learning for Image Recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p1.1),[§2](https://arxiv.org/html/2608.07851#S2.p1.1)\.
- F\. L\. Hitchcock \(1927\)The Expression of a Tensor or a Polyadic as a Sum of Products\.Journal of Mathematics and Physics6\(1\-4\),pp\. 164–189\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p7.5)\.
- G\. Iacovides, W\. Zhou, C\. Li, Q\. Zhao, and D\. Mandic \(2025\)Domain\-aware Tensor Network Structure Search\.arXiv preprint arXiv:2505\.23537\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p2.1)\.
- A\. Karpathy \(2025\)nanochat: The best ChatGPT that $100 can buy\.GitHub\.External Links:[Link](https://github.com/karpathy/nanochat)Cited by:[§5](https://arxiv.org/html/2608.07851#S5.p1.7)\.
- T\. Klein, J\. Kusch, S\. Sager, S\. Schnake, and S\. Schotthöfer \(2026\)Tucker Attention: A Generalization of Approximate Attention Mechanisms\.arXiv preprint arXiv:2603\.30033\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- T\. G\. Kolda and B\. W\. Bader \(2009\)Tensor Decompositions and Applications\.SIAM Review51\(3\),pp\. 455–500\.External Links:[Document](https://dx.doi.org/10.1137/07070111X)Cited by:[§3](https://arxiv.org/html/2608.07851#S3.SS0.SSS0.Px1.p1.13)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak, Y\. Bitton, M\. Nezhurina, A\. Abbas, C\. Hsieh, D\. Ghosh, J\. Gardner, M\. Kilian, H\. Zhang, R\. Shao, S\. Pratt, S\. Sanyal, G\. Ilharco, G\. Daras, K\. Marathe, A\. Gokaslan, J\. Zhang, K\. Chandu, T\. Nguyen, I\. Vasiljevic, S\. Kakade, S\. Song, S\. Sanghavi, F\. Faghri, S\. Oh, L\. Zettlemoyer, K\. Lo, A\. El\-Nouby, H\. Pouransari, A\. Toshev, S\. Wang, D\. Groeneveld, L\. Soldaini, P\. W\. Koh, J\. Jitsev, T\. Kollar, A\. G\. Dimakis, Y\. Carmon, A\. Dave, L\. Schmidt, and V\. Shankar \(2024\)DataComp\-LM: In Search of the Next Generation of Training Sets for Language Models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 14200–14282\.External Links:[Document](https://dx.doi.org/10.52202/079017-0455),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/19e4ea30dded58259665db375885e412-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§5\.2](https://arxiv.org/html/2608.07851#S5.SS2.p1.1)\.
- A\. Novikov, D\. Podoprikhin, A\. Osokin, and D\. P\. Vetrov \(2015\)Tensorizing Neural Networks\.Advances in neural information processing systems28\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- I\. V\. Oseledets \(2011\)Tensor\-Train Decomposition\.SIAM Journal on Scientific Computing33\(5\),pp\. 2295–2317\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- R\. Sinkhorn \(1964\)A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices\.The Annals of Mathematical Statistics35\(2\),pp\. 876–879\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2),[§2\.1](https://arxiv.org/html/2608.07851#S2.SS1.SSS0.Px2.p1.8)\.
- A\. A\. Taranenko \(2023\)Products of Multidimensional Matrices, Stochastic Matrices, and Permanents\.arXiv preprint arXiv:2303\.17278\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2)\.
- L\. R\. Tucker \(1966\)Some Mathematical Notes on Three\-mode Factor Analysis\.Psychometrika31\(3\),pp\. 279–311\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p7.5)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is All You Need\.Advances in Neural Information Processing Systems30\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p1.1)\.
- Z\. Xie, Y\. Wei, H\. Cao, C\. Zhao, C\. Deng, J\. Li, D\. Dai, H\. Gao, M\. Xu, K\. Yu, L\. Zhao, S\. Zhou, Z\. Xu, Z\. Zhang, W\. Zeng, S\. Hu, Y\. Wang, J\. Yuan, L\. Wang, and W\. Liang \(2026\)mHC: Manifold\-Constrained Hyper\-Connections\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=mDhyxu8WRb)Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2),[§2\.1](https://arxiv.org/html/2608.07851#S2.SS1.SSS0.Px2.p1.2)\.
- M\. Xu, Y\. L\. Xu, and D\. P\. Mandic \(2023\)Tensorgpt: Efficient Compression of Large Language Models based on Tensor\-Train Decomposition\.arXiv preprint arXiv:2307\.00526\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- Y\. Yang, J\. Zhou, N\. Wong, and Z\. Zhang \(2024\)LoRETTA: Low\-Rank Economic Tensor\-Train Adaptation for Ultra\-Low\-Parameter Fine\-Tuning of Large Language Models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico\.External Links:[Link](https://aclanthology.org/2024.naacl-long.174/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.174)Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p1.1)\.
- Y\. Yang and J\. Gao \(2026\)mHC\-lite: You Don’t Need 20 Sinkhorn\-Knopp Iterations\.arXiv preprint arXiv:2601\.05732\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2),[§2\.1](https://arxiv.org/html/2608.07851#S2.SS1.SSS0.Px3.p1.5)\.
- J\. Zeng, C\. Li, Z\. Sun, Q\. Zhao, and G\. Zhou \(2024\)TNGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models \(LLMs\)\.arXiv preprint arXiv:2402\.02456\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p2.1)\.
- W\. Zhou, Y\. Gu, G\. Iacovides, and D\. Mandic \(2026\)KromHC: Manifold\-Constrained Hyper\-Connections with Kronecker\-Product Residual Matrices\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=TI7Q2o6EIa)Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2),[§2\.1](https://arxiv.org/html/2608.07851#S2.SS1.SSS0.Px3.p1.5),[§5\.1](https://arxiv.org/html/2608.07851#S5.SS1.p1.4),[§5\.2](https://arxiv.org/html/2608.07851#S5.SS2.p1.1)\.
- W\. Zhou, G\. Iacovides, K\. Konstantinidis, I\. Kisil, and D\. Mandic \(2025\)Understanding the Rank of Tensor Networks via an Intuitive Example\-Driven Approach\.arXiv preprint arXiv:2507\.10170\.Cited by:[§2\.2](https://arxiv.org/html/2608.07851#S2.SS2.p2.1)\.
- D\. Zhu, H\. Huang, Z\. Huang, Y\. Zeng, Y\. Mao, B\. Wu, Q\. Min, and X\. Zhou \(2025\)Hyper\-Connections\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 97183–97219\.Cited by:[§1](https://arxiv.org/html/2608.07851#S1.p2.2),[§2\.1](https://arxiv.org/html/2608.07851#S2.SS1.SSS0.Px1.p1.1)\.