Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

# Exponential-Linear Weight Reparameterization for Improved Optimization
Source: [https://arxiv.org/html/2607.09967](https://arxiv.org/html/2607.09967)
###### Abstract

Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps\. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations\. We introduceSymExpLin\(SEL\), a weight reparameterization for neural networks that combines a sign\-aware symmetric\-exponential pathway with an identity\-like linear pathway\. The symmetric\-exponential pathway is near\-linear for small raw weights but increasingly curved at larger magnitudes\. Additive updates in logarithmic space map to magnitude\-proportional changes in effective weight space\. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway\. These components create a curved parameter\-space geometry that empirically improves speed of loss descent over standard linear parameterization\. We also identify a useful*mismatched initialization*: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small\-model ablations, this improves early optimization and may act as a form of symmetry breaking\. We train transformers on OpenWebText over nine width×\\timesdepth configurations, SEL reaches matched validation loss in 1\.32–1\.49×\\timesfewer training steps, with the largest widths seeing the biggest gains\. SEL is practical: in our largest profiled configuration, a 1\.44×\\timesstep reduction and 5\.5% per\-step overhead correspond to an estimated 1\.37×\\timeswall\-clock speedup, and after training the parameterization is folded into standard linear weights with no inference cost\.

## 1Introduction

Standard neural network training optimizes parameters in an additive fashion with linear update steps\. Yet many computations within a network are naturally relative\. Doubling or halving a normalization gain, a feature amplitude, or the singular value of a matrix reflects analogous changes in logarithmic space\. However, they have unequal optimization distances in linear weight space\. More specifically, moving from11to0\.50\.5has half the linear distance of moving from11to22\. This is not to say we should necessarily think of all aspects of the network in a multiplicative fashion\. Activation functions alongside biases consider absolute scale to be important\. Nevertheless, the optimization employed in training has mostly played into the linear/absolute scale nature of networks\. When relative changes weights are to be achieved through additive updates, we should expect training to push a subset of parameters far from initialization whenever the task demands large multiplicative factors, while leaving many others near their starting scale\.

As shown in Figure[1](https://arxiv.org/html/2607.09967#S3.F1), we may observe this effect in pretrained language models, which develop heavy\-tailed weight distributions\. Many parameters remain near initialization scale, while a small fraction move far into the tails\. In our Qwen3\-4B\-Instruct analysis \(Figure[1](https://arxiv.org/html/2607.09967#S3.F1)\), the maximum weight is roughly 30×\\timesthe Xavier initialization scale even though the standard deviation grows only 1\.7×\\times\. This creates a heterogeneous optimization problem in which many coordinates may travel little from their original positions, while some coordinates must traverse much larger distances in weight space\.

Adaptive optimizers such as Adam\[Kingma and Ba,[2014](https://arxiv.org/html/2607.09967#bib.bib21)\]normalize update magnitudes per coordinate inducing similar magnitude across dimensions, rendering the updates invariant to parameter magnitude\. Preconditioning decouples step size from gradient scale to keep optimization well\-conditioned, but a parameter at0\.0040\.004and one at0\.40\.4can receive similar absolute updates while resulting in very different relative perturbations: the same step may be significant for a small parameter yet barely move a large one in relative terms\. Global learning rates may be chosen to respect the most sensitive small\-scale coordinates, potentially underserving larger magnitude directions such as feature gains, matrix modes, or other coordinates that may benefit from relative movement\.

One solution may be to try to correct this at the optimizer level through parameter\-magnitude scaling into the update rule, but this may fight against the preconditioning benefit\. We then propose to change the parameterization \(and thus geometry\) of the weights themselves\. The natural method for incorporating multiplicative relativity is to make use of an exponential mapping, which converts additive additive updates in the raw space into multiplicative updates in the output space\. However, the standard exponential function only outputs positive values when our weights are roughly centered and symmetrical around 0\. To address this issue, we utilize the symexp function, which is sign\-preserving, odd, monotonic, and offers the exponential behavior\. As to not overcommit to the exponential structure, we introduce a linear pathway along with learnable weights allowing the network to scale the effect of expoential and linear components\. We hypothesize as well that the linear pathway may provide a direct gradient route which may aid optimization\.

Each effective weight is decomposed into two complementary pathways: asymmetric\-exponential pathwayand alinear pathway\. In our recommended configuration, the effective weight is:

weff=sign⁡\(w\)​\[Eθ​\(\|w\|\)\+Lθ​\(w\)\]\+Δϕw\_\{\\mathrm\{eff\}\}=\\operatorname\{sign\}\(w\)\\,\[E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)\+L\_\{\\theta\}\(w\)\]\+\\Delta\_\{\\phi\}\(1\)whereEθ​\(\|w\|\)E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)is the learnably controlled symexp amplitude,Lθ​\(w\)L\_\{\\theta\}\(w\)is an identity\-like linear pathway, andΔϕ\\Delta\_\{\\phi\}is an optional additive residual, typically implemented as a low\-rank LoRA\-style correction\. The sign wrap applies the curved geometry to\|w\|\\left\\lvert w\\right\\rvertwhile keeping sign as a separable degree of freedom\. We further find that a deliberate*asymmetry*between how weights are initialized and how the forward pass combines these pathways yields additional gains, which we analyze in Section[4](https://arxiv.org/html/2607.09967#S4)and attribute to early symmetry\-breaking effects\.

Contributions:

- •We propose SymExpLin \(SEL\), a nonlinear weight reparameterization applicable to any learnable weight matrix or bias vector, decomposing effective weights into symmetric\-exponential and linear components with independently learnable structured scales\.
- •We identify*mismatched initialization*, using a different combination rule for weight inversion than for the forward pass, as a source of beneficial asymmetric bias in our small\-scale ablations\.
- •We introduce structured scale parameters with various factorization patterns and a targeted learning rate annealing schedule for managing redundant exponential controls\.
- •SEL is implementation\-friendly: the training transform is elementwise and incurs modest overhead in compiled implementations, while effective weights are precomputed after training, reducing to standard linear layers withzero inference cost\.
- •We note that SEL can be folded into standard weights after training and, conversely, standard weights can be inverted back into the SEL parameter space for future continued\-training or finetuning experiments\.
- •We evaluate across a grid of model scales \(\{1024,2048,3072\}×\{12,24,36\}\\\{1024,2048,3072\\\}\\times\\\{12,24,36\\\}\) on autoregressive language modeling, with targeted ablations for initialization asymmetry, the linear pathway, and scale learning\-rate annealing\.

## 2Related Work

##### Weight Reparameterization\.

Weight normalization\[Salimans and Kingma,[2016](https://arxiv.org/html/2607.09967#bib.bib3)\]separates weight magnitude from direction, improving optimization conditioning\. Spectral normalization\[Miyatoet al\.,[2018](https://arxiv.org/html/2607.09967#bib.bib4)\]constrains the spectral norm for training stability\. Both apply relatively simple transformations\. SEL introduces a richer nonlinear reparameterization that mixes symmetric\-exponential and linear components with learnable mixture coefficients, operating on the weight matrix itself rather than its norms\.

##### Symlog Transforms\.

The symmetric logarithmsign⁡\(x\)​ln⁡\(\|x\|\+1\)\\operatorname\{sign\}\(x\)\\ln\(\\left\\lvert x\\right\\rvert\+1\)and its inverse have been used for prediction targets in reinforcement learning\[Hafneret al\.,[2023](https://arxiv.org/html/2607.09967#bib.bib5)\], handling values spanning many orders of magnitude\. SEL applies a similar operation, a parameterized symmetric exponential, to*weight space*rather than prediction targets, with learnable scale parameters controlling the symmetric\-exponential and linear components\.

##### Initialization Schemes\.

Xavier\[Glorot and Bengio,[2010](https://arxiv.org/html/2607.09967#bib.bib6)\]and Kaiming\[Heet al\.,[2015](https://arxiv.org/html/2607.09967#bib.bib7)\]initialization maintain variance across layers\. Data\-dependent initialization\[Mishkin and Matas,[2016](https://arxiv.org/html/2607.09967#bib.bib10)\]matches target statistics layer by layer\. Our initialization via Newton’s method inversion ensures that*despite*the nonlinear reparameterization, effective weights at initialization match standard Xavier statistics\. Our mismatched initialization is different from these schemes: it preserves the intended Xavier scale under one transform rule, then deliberately evaluates the same raw weights under a slightly asymmetric rule\.

##### Learning Rate Parameterization\.

Maximal Update Parameterization \(μ\\muP\)\[Yanget al\.,[2022](https://arxiv.org/html/2607.09967#bib.bib11)\]provides principled per\-tensor learning rate scaling across model widths, addressing one axis of the init/optimizer mismatch we discuss in Section[3](https://arxiv.org/html/2607.09967#S3)\. SEL targets a complementary axis, adapting effective step size per\-parameter over the course of training rather than per\-tensor at initialization; the two are not mutually exclusive\.

##### Multiplicative Interactions\.

Jayakumaret al\.\[[2020](https://arxiv.org/html/2607.09967#bib.bib13)\]study where multiplicative interactions arise in neural networks and their benefits\. Our structured scale parameters withrow\_col\_multmode implement rank\-1 multiplicative interactions on the weight scaling, a lightweight form of structured multiplicative modulation\.

The Multiplicative Weights Update \(MWU\) method\[Aroraet al\.,[2012](https://arxiv.org/html/2607.09967#bib.bib32)\]makes relative\-scale updates explicit: weights are scaled by\(1±ϵ\)payoff\(1\\pm\\epsilon\)^\{\\text\{payoff\}\}rather than shifted additively, so adjustments are naturally proportional to current magnitude\. This principle underlies algorithms from boosting to online convex optimization, but has seen limited adoption in neural network training, where additive SGD\-style updates remain standard\. SEL does not update multiplicatively\. The optimizer still takes additive steps in raw parameter space, but the symmetric\-exponential pathway induces a related effect at the effective\-weight level: raw\-space sensitivity scales asexp⁡\(β​\|w\|\)\\exp\(\\beta\\left\\lvert w\\right\\rvert\), so identical raw steps produce larger effective\-weight displacements at larger magnitudes\. The linear pathway preserves additive behavior near the origin, yielding a hybrid regime rather than a fully multiplicative one\. This positions SEL as a way to obtain some of MWU’s relative\-scale behavior while remaining compatible with standard gradient\-based optimizers and preconditioning\.

## 3Motivation

Training moves parameters from a sampled simple starting distribution to a much broader final one\. At initialization, variance\-preserving schemes initalize parameters near a typical magnitude determined by its matrix’s width\. By the end of training, we observe that the distribution of weights moves towards a heavy tailed shape in which a small fraction of parameters live far outside the initial typical range \(Figure[1](https://arxiv.org/html/2607.09967#S3.F1)\)\. Different families of methods respond to different points on this trajectory:μ\\muP and related learning\-rate parameterizations set step size from the init\-side typical magnitude, a per\-tensor summary statistic downstream of model dimension, while SEL adapts effective step size dynamically from each parameter’s*current*magnitude, so as the distribution spreads during training, parameters that have travelled further receive proportionally larger effective steps\. This section discusses preconditioning, learning rate tuning, and update magnitude with respect to parameter magnitude\. We pose SEL as a complementary counterpart toμ\\muP along a second, dynamic axis\.

##### Adam normalizes away scale\.

Consider Adam’s update rule\. The first and second moment estimates aremt=β1​mt−1\+\(1−β1\)​gtm\_\{t\}=\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)g\_\{t\}andvt=β2​vt−1\+\(1−β2\)​gt2v\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}^\{2\}, and the update isΔ​w∝mt/vt\\Delta w\\propto m\_\{t\}/\\sqrt\{v\_\{t\}\}\. Under steady\-state conditions with approximately constant gradient magnitudes,vt≈𝔼​\[g2\]v\_\{t\}\\approx\\mathbb\{E\}\[g^\{2\}\], so the normalized updatemt/vtm\_\{t\}/\\sqrt\{v\_\{t\}\}has expected absolute value near 1 given the division by the estimated gradient’s standard deviation\. In the special caseβ1=β2\\beta\_\{1\}=\\beta\_\{2\}where the two moving averages track on identical timescales, Cauchy\-Schwarz tightens this to a hard bound:\|mt/vt\|≤1\\left\\lvert m\_\{t\}/\\sqrt\{v\_\{t\}\}\\right\\rvert\\leq 1at steady state\. Beyond steady state conditions, momentum suffers destructive interference under gradient sign flips, decreasing the numerator\. Meanwhilevtv\_\{t\}is sensitive to larger values potentially increasing denominator\. The essential point is invariance: uniformly scaling every gradient by a constant leaves\|mt/vt\|\\left\\lvert m\_\{t\}/\\sqrt\{v\_\{t\}\}\\right\\rvertunchanged, the normalization byvt\\sqrt\{v\_\{t\}\}cancels it, and no term in the update depends on\|w\|\\left\\lvert w\\right\\rvert\. Per\-dimension variation reflects the*shape*of the gradient distribution \(noise pattern, outlier frequency\) rather than its overall scale, so updates concentrate around a common magnitude set by the learning rate, invariant to both expected gradient magnitude and parameter magnitude\. A parameter at0\.0040\.004and one at0\.40\.4therefore receive updates of similar absolute size, corresponding to a100×100\\timesdifference in relative perturbation; global learning rate tuning must accommodate the most sensitive small\-scale coordinates, potentially under\-serving larger\-magnitude ones\.

##### Variance\-Preserving Initialization

Variance\-preserving initialization offer a typical parameter magnitude at the start of training, and may also guide common magnitudes at the end state of training\. Xavier initialization usesσ∝1/\(fan\_in\+fan\_out\)/2\\sigma\\propto 1/\\sqrt\{\(\\text\{fan\\\_in\}\+\\text\{fan\\\_out\}\)/2\}, so smaller matrices receive larger initialization values and larger matrices smaller ones\. The effect is pronounced for low\-rank adapters\[Huet al\.,[2021](https://arxiv.org/html/2607.09967#bib.bib12)\], where the rankr≪dr\\ll dyields much larger initialization scales than full weight matrices\. This is consistent with the observation that LoRA requires higher learning rates than full model weight finetuning\. Initialization thus writes scale information into the parameters that Adam then normalizes away\.

##### μ\\muP: per\-tensor, width\-driven, static\.

Maximal Update Parameterization\[Yanget al\.,[2022](https://arxiv.org/html/2607.09967#bib.bib11)\]formalizes one axis of this mismatch: typical init magnitude varies systematically with matrix width, and per\-tensor learning rates must be rescaled with width to preserve feature\-learning behavior across scales\.μ\\muP’s rescaling is theoretically derived, applied once at the tensor level, and driven by static properties of the architecture\. In effect, it utilizes either the standard deviation or uniform bound implied by the width and adjusts step size accordingly\.

##### A second, dynamic axis\.

μ\\muP addresses variation*across*tensors of different widths but not variation*within*a tensor over training\. Xavier init specifies a typical starting scale for each parameter, but pretrained networks develop heavy\-tailed weight distributions in which a small fraction of parameters end up an order of magnitude beyond that typical scale\. We verified this on Qwen3\-4B\-Instruct\[Qwen Team,[2025](https://arxiv.org/html/2607.09967#bib.bib26)\]: while Xavier initialization produces weights withmax≈0\.04\\max\\approx 0\.04andσ≈0\.014\\sigma\\approx 0\.014, the pretrained weights havemax≈1\.23\\max\\approx 1\.23\(a30×30\\timesincrease\) whileσ\\sigmaincreases only1\.7×1\.7\\times\. The distribution is*heavy\-tailed*: only0\.05%0\.05\\%of weights exceed0\.10\.1in magnitude, yet these outliers are functionally critical\[Dettmerset al\.,[2022](https://arxiv.org/html/2607.09967#bib.bib23)\]\(Figure[1](https://arxiv.org/html/2607.09967#S3.F1)\)\. Under Adam updates, or other optimizer with normalization, some parameters fundamentally have greater distance to travel requiring more optimization steps\.

![Refer to caption](https://arxiv.org/html/2607.09967v1/figures/weights.png)Figure 1:Weight distribution gap between initialization and pretrained models\.*Left:*Histogram comparison \(log scale\) showing pretrained weights have heavier tails than Xavier initialization\.*Middle:*Tail analysis\. The fraction of pretrained weights exceeding various thresholds\. The vertical line marks the Xavier max\.*Right:*Per\-layer growth ratio; most layers have weights that grew 5–15×\\timesbeyond their initialization range\.
##### SEL as a per\-parameter, adaptive analog\.

SEL takes a similar underlying observation asμ\\muP and pushes it along this second axis\. Instead of adjusting learning rates at a tensor or model\-level from static init statistics, we change the parameterization sucht that the local sensitivity∂weff/∂wraw\\partial w\_\{\\mathrm\{eff\}\}/\\partial w\_\{\\mathrm\{raw\}\}grows with\|w\|\\left\\lvert w\\right\\rvert\. Identical raw\-space steps produce larger effective\-weight movements for larger parameters\. In this sense SEL is a per\-parameter, dynamic, empirically motivated counterpart toμ\\muP’s per\-tensor, static, theoretically motivated rescaling\.μ\\muP derives scale from typical parameter value obtained from width, SEL obtain scale from individual dimensions throughout training\. The two operate on distinct axes, are not mutually exclusive, and can be composed where both apply\.

## 4Method

### 4\.1Background: SymLog and SymExp

The symmetric logarithm \(symlog\) and its inverse, the symmetric exponential \(symexp\), are odd functions that handle signed values spanning many orders of magnitude\[Hafneret al\.,[2023](https://arxiv.org/html/2607.09967#bib.bib5)\]:

symlog⁡\(x\)\\displaystyle\\operatorname\{symlog\}\(x\)=sign⁡\(x\)​ln⁡\(\|x\|\+1\)\\displaystyle=\\operatorname\{sign\}\(x\)\\ln\(\\left\\lvert x\\right\\rvert\+1\)\(2\)symexp⁡\(x\)\\displaystyle\\operatorname\{symexp\}\(x\)=sign⁡\(x\)​\(exp⁡\(\|x\|\)−1\)\\displaystyle=\\operatorname\{sign\}\(x\)\\bigl\(\\exp\(\\left\\lvert x\\right\\rvert\)\-1\\bigr\)\(3\)The symexp function is particularly relevant\. For small\|x\|\\left\\lvert x\\right\\rvert, it behaves approximately linearly \(symexp⁡\(x\)≈x\\operatorname\{symexp\}\(x\)\\approx x\), while for large\|x\|\\left\\lvert x\\right\\rvert, it grows exponentially\. It is sign\-preserving and differentiable everywhere except at the origin \(Figure[2](https://arxiv.org/html/2607.09967#S4.F2), left\)\.

A natural idea is to use symexp as a weight reparameterization\. Store raw parameterswwand usesymexp⁡\(w\)\\operatorname\{symexp\}\(w\)as the effective weight\. This creates a curved optimization landscape where large weights are amplified\. However, bare symexp has a fixed curvature and no mechanism to independently control the balance between its near\-linear small\-weight regime and its exponential large\-weight regime\.

SymExpLin generalizes this idea by \(1\) introducing a curvature parameterβ\\betathat controls the transition between linear and exponential regimes, \(2\) separating the exponential and linear contributions into independently scalable pathways, and \(3\) adding learnable modulation parameters that control curvature, thresholding, and pathway balance\.

### 4\.2The SymExpLin Transform

Given a raw weight parameterw∈ℝw\\in\\mathbb\{R\}, SymExpLin generalizes the symexp function in two key ways: \(1\) introducing a curvature parameterβ\\betathat controls the transition between linear and exponential regimes, and \(2\) adding a linear pathway that we hypothesize may aid in optimization\. In its simplest form, the transform is:

weff=sign⁡\(w\)⋅\(exp⁡\(β​\|w\|\)−1\)β\+wβw\_\{\\mathrm\{eff\}\}=\\frac\{\\operatorname\{sign\}\(w\)\\cdot\\bigl\(\\exp\(\\beta\\left\\lvert w\\right\\rvert\)\-1\\bigr\)\}\{\\beta\}\+\\frac\{w\}\{\\beta\}\(4\)The first term is a scaled symexp\. Placingβ\\betainside the exponent sets the exponential growth rate, while the outer1/β1/\\betanormalizes the leading\-order Taylor expansion so the symexp itself contributes unit slope at the origin regardless ofβ\\beta\(sinceexp⁡\(β​\|w\|\)−1≈β​\|w\|\\exp\(\\beta\\left\\lvert w\\right\\rvert\)\-1\\approx\\beta\\left\\lvert w\\right\\rvertnear zero\)\. The linear pathway is likewise divided byβ\\beta, so its contribution is proportional to1/β1/\\beta: it adds a small1/β1/\\betaslope near the origin that fades asβ\\betagrows and the exponential regime dominates\. The net effect:β\\betacontrols the transform’s curvature and growth rate rather than its overall scale\.

The choice ofβ\\betashould be calibrated to the typical initialization scale\. Recall that Xavier initialization usesσ∝1/d\\sigma\\propto 1/\\sqrt\{d\}, so larger matrices have smaller initial weight magnitudes\. To achieve meaningful curvature within this range,β\\betamust be correspondingly larger, otherwise the transform remains extensively in its near\-linear regime\. Figure[2](https://arxiv.org/html/2607.09967#S4.F2)\(right\) illustrates this relationship\.

![Refer to caption](https://arxiv.org/html/2607.09967v1/figures/symexp.png)Figure 2:Symmetric exponential transform and controls\.*Left:*The standard symexp function is approximately linear near zero and diverges exponentially from the identity at larger magnitudes while preserving sign\.*Middle:*The offsetnnsets the slope at the origin viaexp⁡\(−n\)\\exp\(\-n\):n=0n=0gives unit slope, positivennshrinks the slope so the transform starts flatter and reaches the exponential regime later, and negativennraises the initial slope so amplification begins earlier\.*Right:*The curvatureβ\\betais the exponential growth rate: largerβ\\betaproduces a sharper transition into the exponential regime and faster growth within it\. Since Xavier initialization magnitudes shrink with model width, larger models generally need largerβ\\betato reach meaningful curvature at typical weight magnitudes\.##### Adding learnable control\.

SymExpLin augments the base form with learnable parameters that balance the effect of the linear and exponential pathways, as well as aspects of the exponential pathway such as its growth rate and an offset for the slope at the origin\. Thesymmetric\-exponential pathwayis:

Eθ​\(\|w\|\)\\displaystyle E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)=ew​\(exp⁡\(β​\|w\|⋅m−n\)−exp⁡\(−n\)\)\\displaystyle=e\_\{w\}\\Bigl\(\\exp\\bigl\(\\beta\\left\\lvert w\\right\\rvert\\cdot m\-n\\bigr\)\-\\exp\(\-n\)\\Bigr\)\(5\)Sθ​\(w\)\\displaystyle S\_\{\\theta\}\(w\)=sign⁡\(w\)⋅Eθ​\(\|w\|\)\\displaystyle=\\operatorname\{sign\}\(w\)\\cdot E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)whereEθ​\(\|w\|\)E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)is the exponential amplitude,Sθ​\(w\)S\_\{\\theta\}\(w\)is the signed symmetric\-exponential pathway,mmis a learnable inner multiplier,nnis a learnable offset, andewe\_\{w\}is a learnable output scale\. We initializeew≈1/βe\_\{w\}\\approx 1/\\beta, so the base transform recovers the normalized symexp form in Eq\. \([4](https://arxiv.org/html/2607.09967#S4.E4)\)\.

The offsetnnsets the slope ofEθE\_\{\\theta\}at the origin: differentiating givesEθ′​\(0\)=ew⋅β⋅m⋅exp⁡\(−n\)E^\{\\prime\}\_\{\\theta\}\(0\)=e\_\{w\}\\cdot\\beta\\cdot m\\cdot\\exp\(\-n\), so with defaultsew=1/βe\_\{w\}=1/\\betaandm=1m=1the initial slope is exactlyexp⁡\(−n\)\\exp\(\-n\)\. The defaultn=0n=0recovers unit slope; positivennshrinks the initial slope, making the transform flatter near zero and delaying when the exponential regime takes over; negativennraises the initial slope so amplification begins earlier\. The two appearances ofnnin Equation \([5](https://arxiv.org/html/2607.09967#S4.E5)\), as−n\-ninside the exponent and as−exp⁡\(−n\)\-\\exp\(\-n\)outside, are what preserveEθ​\(0\)=0E\_\{\\theta\}\(0\)=0while allowingnnto modulate this slope; the effect is the single\-parameter reshaping just described\. Figure[2](https://arxiv.org/html/2607.09967#S4.F2)\(middle\) visualizes this effect\.

Thelinear pathwayis:

Lθ​\(w\)=w⋅lwL\_\{\\theta\}\(w\)=w\\cdot l\_\{w\}\(6\)wherelwl\_\{w\}is a learnable scale initialized near1/β1/\\beta\.

A learnable residualΔϕ\\Delta\_\{\\phi\}optionally provides an additive correction\. In our experiments, this residual is typically implemented as a low\-rank LoRA\-style update and can be omitted without changing the core SEL parameterization\. Thefull transformin*mismatch mode*\(our recommended configuration\) is:

weff=sign⁡\(w\)⋅\(Eθ​\(\|w\|\)\+Lθ​\(w\)\)\+Δϕw\_\{\\mathrm\{eff\}\}=\\operatorname\{sign\}\(w\)\\cdot\\bigl\(E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)\+L\_\{\\theta\}\(w\)\\bigr\)\+\\Delta\_\{\\phi\}\(7\)

##### Role of each component\.

The static curvatureβ\\betacontrols the transition between linear and exponential regimes while normalizing output magnitude\. The inner multipliermmmodifies the growth rate as a multiplier of the static choice ofβ\\beta\. The offsetnncontrols where exponential amplification begins\. The scalesewe\_\{w\}andlwl\_\{w\}independently weight each pathway’s contribution\. When enabled, the residualΔϕ\\Delta\_\{\\phi\}adds a sign\-independent correction; its learning rate is scaled by1/β1/\\betato keep its effective step size commensurate with the other terms\.

##### Comparison of combination modes\.

We define two modes for combining the pathways:

Congruent:weff\\displaystyle\\text\{Congruent:\}\\quad w\_\{\\mathrm\{eff\}\}=Sθ​\(w\)\+Lθ​\(w\)\\displaystyle=S\_\{\\theta\}\(w\)\+L\_\{\\theta\}\(w\)\(8\)Mismatch:weff\\displaystyle\\text\{Mismatch:\}\\quad w\_\{\\mathrm\{eff\}\}=sign⁡\(w\)⋅\(Eθ​\(\|w\|\)\+Lθ​\(w\)\)\\displaystyle=\\operatorname\{sign\}\(w\)\\cdot\\bigl\(E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)\+L\_\{\\theta\}\(w\)\\bigr\)\(9\)In congruent mode \([8](https://arxiv.org/html/2607.09967#S4.E8)\), the linear part is added*outside*the sign wrapping and thus independently preservesww’s sign\. In mismatch mode \([9](https://arxiv.org/html/2607.09967#S4.E9)\), both parts are inside the sign wrapping: for negativeww, the linear term \(which is negative\) reduces the magnitude\. Figure[3](https://arxiv.org/html/2607.09967#S4.F3)illustrates this difference: the two modes produce identical outputs for positive weights, but for negative weights, mismatch mode produces*smaller*effective magnitudes than congruent mode\. This asymmetry is one mechanism we study for improving early optimization\. In our width\-1024 ablations, mismatch mode outperforms congruent, particularly when combined with congruent initialization \(§[4\.3](https://arxiv.org/html/2607.09967#S4.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2607.09967v1/x1.png)Figure 3:Congruent vs mismatch combination modes\.*Left:*Both modes overlaid\. For positiveww, the curves coincide\. For negativeww, mismatch mode \(red\) produces smaller effective magnitudes than congruent mode \(blue\)\.*Right:*The difference between modes\. The asymmetry arises because in mismatch mode, the linear termLθ​\(w\)L\_\{\\theta\}\(w\)\(which is negative whenw<0w<0\) is wrapped inside the sign operation, reducing the magnitude rather than reinforcing it\.

### 4\.3Mismatched Initialization

To ensure that effective weights at initialization match standard Xavier statistics\[Glorot and Bengio,[2010](https://arxiv.org/html/2607.09967#bib.bib6)\], we initialize the raw weights by*inverting*the transform: given target weightsWtargetW\_\{\\mathrm\{target\}\}from Xavier uniform initialization, we solvef​\(w\)=Wtargetf\(w\)=W\_\{\\mathrm\{target\}\}via Newton’s method\.

The key design choice is*which*combination rule to use for this inversion\. In our width\-1024 ablations, using thecongruent formula\([8](https://arxiv.org/html/2607.09967#S4.E8)\) for inversion while using themismatch formula\([9](https://arxiv.org/html/2607.09967#S4.E9)\) for the forward pass yields the strongest results\. We call this*mismatched initialization*\.

Under congruent inversion, Newton’s method finds raw weightswwsuch thatSθ​\(w\)\+Lθ​\(w\)=WtargetS\_\{\\theta\}\(w\)\+L\_\{\\theta\}\(w\)=W\_\{\\mathrm\{target\}\}\. When these weights are then evaluated with the mismatch formulasign⁡\(w\)​\(Eθ​\(\|w\|\)\+Lθ​\(w\)\)\\operatorname\{sign\}\(w\)\(E\_\{\\theta\}\(\\left\\lvert w\\right\\rvert\)\+L\_\{\\theta\}\(w\)\), the behavior diverges by sign:

- •Positive weights\(w\>0w\>0\): Both formulas give the same result, sinceSθ\+Lθ=sign⁡\(w\)​\(Eθ\+Lθ\)S\_\{\\theta\}\+L\_\{\\theta\}=\\operatorname\{sign\}\(w\)\(E\_\{\\theta\}\+L\_\{\\theta\}\)whenLθ\>0L\_\{\\theta\}\>0\. The initialization is correct\.
- •Negative weights\(w<0w<0\): The congruent formula gives−Eθ\+Lθ\-E\_\{\\theta\}\+L\_\{\\theta\}\(symmetric\-exponential opposes, linear reinforces\), while the mismatch formula gives−\(Eθ\+Lθ\)=−\(Eθ−\|Lθ\|\)\-\(E\_\{\\theta\}\+L\_\{\\theta\}\)=\-\(E\_\{\\theta\}\-\\left\\lvert L\_\{\\theta\}\\right\\rvert\)\(both wrapped inside the negation, with the linear term’s magnitude reducing the total\)\. The effective weight magnitude is reduced by2​\|Lθ\|2\\left\\lvert L\_\{\\theta\}\\right\\rvertcompared to the congruent target\.

For our default parameters \(β=7\.5\\beta=7\.5\), the linear pathway contributes roughly 10–15% of the total effective weight magnitude for typical Xavier\-scale weights\. As beta increases, this further drops\. This means negative weights are under\-initialized by approximately 20–25% in magnitude, while positive weights are initialized correctly\. The result is anasymmetric initializationwhere positive weights start at full strength and negative weights start weaker\.

We hypothesize the benefit comes primarily from symmetry breaking: the positive/negative asymmetry gives early gradients a preferred direction to act on, rather than requiring the network to break its own initial symmetry from a fully symmetric Xavier distribution\. This may be reinforced by our scheduled scale learning rates \(§[4\.5](https://arxiv.org/html/2607.09967#S4.SS5)\), which accelerate the learning rate of the scale parameters early on and may let the scales commit to a direction prior to the weights themselves beginning to converge, greater coupling their updates\.

##### Xavier uniform over normal\.

We find that Xavier*uniform*initialization consistently outperforms Xavier normal for the target weights before inversion\. We attribute this to the bounded support of the uniform distribution: extreme values from the tails of the normal distribution can produce very large raw parameters after inversion through the symmetric\-exponential pathway, destabilizing early training\.

### 4\.4Structured Scale Parameters

Each learnable scale \(ewe\_\{w\},lwl\_\{w\},mm,ss,nn\) is parameterized with a chosen factorization pattern\. The optional residualΔϕ\\Delta\_\{\\phi\}is an additive correction, and is usually parameterized as a LoRA\-style update\. For a weight matrixW∈ℝdout×dinW\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}, available patterns include:

- •Global: a single shared scalar\.
- •Row / Column: a vector along one axis \(doutd\_\{\\mathrm\{out\}\}ordind\_\{\\mathrm\{in\}\}\)\.
- •Row×\\timesCol multiplicative: two rank\-1 vectorsr∈ℝdoutr\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\},c∈ℝdinc\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\}whose outer productr⋅c⊤r\\cdot c^\{\\top\}gives the scale matrix\. This is our default and recommended mode\.
- •LoRA: a low\-rank additive perturbationbase\+A​B\\text\{base\}\+ABwhereA∈ℝdout×rA\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\},B∈ℝr×dinB\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}, initialized withB=0B=0\[Huet al\.,[2021](https://arxiv.org/html/2607.09967#bib.bib12)\]\.

Each scale has a*learning rate multiplier*kkthat scales its learning rate relative to the base rate, allowing different components of the reparameterization to learn at different speeds\. One of the main reasons for this is that the scale parameters are initialized around 1\.0 and have to travel a significant distance in weight space for a meaningful effect, as per the prior explanation of 0\.5 as a halving operation and 2\.0 as a doubling operation\.

In our main runs, the output pathway scalesewe\_\{w\}andlwl\_\{w\}userow\_col\_multas an efficient trade\-off: it is more expressive than a global scalar while remaining much cheaper than full elementwise scale parameters\. The inner multipliermmalso usesrow\_col\_multin our experiments, though simplifying it to a row\-only or column\-only pattern is a natural future ablation\. For the offsetnn, we typically use a column\-wise scale, since offset effects are more sensitive to input\-axis structure\.

##### Weight decay on scale parameters\.

We disable weight decay for scale parameters that primarily control the*shape*of the reparameterization, such as the curvature multipliermmand offsetnn\. However, we keep weight decay on the output pathway scalesewe\_\{w\}andlwl\_\{w\}\. These scales directly weight the symmetric\-exponential and linear contributions, so they can increase the overall effective weight magnitude even if the raw weightwwitself is decayed\. Applying weight decay toewe\_\{w\}andlwl\_\{w\}prevents the model from growing the effective weight unboundedly by growing pathway scales\.

### 4\.5Exponential Pathway Learning Rate Annealing

The symmetric\-exponential pathway has two controls that affect its strength: the outer scaleewe\_\{w\}and the inner multipliermm\. While both increase the exponential contribution, they serve different roles:ewe\_\{w\}can go negative \(flipping the sign of the exponential part\), whilemmcontrols the curvature \(how sharply the exponential grows with\|w\|\\left\\lvert w\\right\\rvert\)\.

These controls are partially redundant, making joint optimization difficult\. We address this with a targeted learning rate annealing schedule:ewe\_\{w\}starts with a highkkvalue \(fast learning\) so it can quickly commit to a sign, then its learning rate decays over a cosine\-in\-log\-space schedule, whilemmstarts with a lowkkvalue and anneals upward to refine the curvature onceewe\_\{w\}has stabilized\.

Specifically, for a parameter with initial multiplierkstartk\_\{\\text\{start\}\}and final multiplierkendk\_\{\\text\{end\}\}, the effective multiplier at stepttis:

k​\(t\)=exp⁡\(ln⁡kstart\+ln⁡\(kend/kstart\)2​\(1−cos⁡\(π⋅min⁡\(t/T,1\)\)\)\)k\(t\)=\\exp\\\!\\Bigl\(\\ln k\_\{\\text\{start\}\}\+\\frac\{\\ln\(k\_\{\\text\{end\}\}/k\_\{\\text\{start\}\}\)\}\{2\}\\bigl\(1\-\\cos\(\\pi\\cdot\\min\(t/T,1\)\)\\bigr\)\\Bigr\)\(11\)whereTTis the total annealing steps\. This interpolates betweenkstartk\_\{\\text\{start\}\}andkendk\_\{\\text\{end\}\}in log space along a cosine curve, providing smooth transitions\. After stepTT, the multiplier remains atkendk\_\{\\text\{end\}\}\.

## 5Experiments

We evaluate SEL on autoregressive language modeling\. All experiments use the same codebase and training infrastructure\.

### 5\.1Experimental Setup

##### Architecture\.

We use a modern transformer with GeGLU feedforward blocks\[Shazeer,[2020](https://arxiv.org/html/2607.09967#bib.bib14)\]\(hidden dimension83​d\\frac\{8\}\{3\}drounded to the nearest multiple of 64\), RoPE positional encodings\[Suet al\.,[2024](https://arxiv.org/html/2607.09967#bib.bib15)\], RMSNorm\[Zhang and Sennrich,[2019](https://arxiv.org/html/2607.09967#bib.bib16)\], and QK\-normalization\. SEL is applied to all linear projections: Q, K, V, output projection, and both FFN layers\. Biases are enabled and also reparameterized\.

##### Scale grid\.

We evaluate the full Cartesian product of widths \{1024, 2048, 3072\} and depths \{12, 24, 36\}, for nine total configurations\.

##### Training\.

All models are trained on OpenWebText\[Gokaslan and Cohen,[2019](https://arxiv.org/html/2607.09967#bib.bib19)\]with sequence length 1024\. We use AdamW\[Loshchilov and Hutter,[2017](https://arxiv.org/html/2607.09967#bib.bib20)\]withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, weight decay 0\.01, gradient clipping at 1\.0, and linear learning rate warmup for 250 steps followed by linear decay\. Net batch size varies across runs; width\-1024 configurations are trained for 250k steps, while width\-2048 and width\-3072 configurations are trained for 150k steps\. Models are compiled withtorch\.compile\(reduce\-overhead mode, full graph\) and trained with bf16 mixed precision and gradient checkpointing\. Compiling is important for minimizing the overhead of our method\. Given that our method is composed entirely of elementwise operations, compiling offers a significant benefit in reducing the memory\-bound parts of the computation\.

##### SEL configuration\.

Unless otherwise noted, SEL experiments use: theβ\\betavalue selected by the initialization\-scale heuristic described below, mismatch forward mode with congruent initialization,row\_col\_multpattern for all scale parameters,kew=50k\_\{e\_\{w\}\}=50\(annealed tokew,end=8\.0k\_\{e\_\{w\},\\text\{end\}\}=8\.0\),km=0\.01k\_\{m\}=0\.01\(annealed tokm,end=0\.5k\_\{m,\\text\{end\}\}=0\.5\),klw=50k\_\{l\_\{w\}\}=50,\(annealed toklw,end=8\.0k\_\{l\_\{w\},\\text\{end\}\}=8\.0\), optional LoRA\-style residualΔϕ\\Delta\_\{\\phi\}withkΔ=0\.5k\_\{\\Delta\}=0\.5when enabled, weight decay enabled forewe\_\{w\}andlwl\_\{w\}but disabled for the shape\-controlling scale parameters, Xavier uniform initialization,beta\_as\_multenabled \(fixedβ\\betaappears in the exponent, while the old1/β1/\\betanormalizers are folded into the initialewe\_\{w\}andlwl\_\{w\}scales\)\.

### 5\.2Main Results

![Refer to caption](https://arxiv.org/html/2607.09967v1/)Figure 4:Evaluation loss during pretraining on OpenWebTextacross all nine width×\\timesdepth configurations\.*Top:*width 1024 \(250k steps,β=7\.5\\beta=7\.5\)\.*Middle:*width 2048 \(150k steps\)\.*Bottom:*width 3072 \(150k steps\)\.β\\betais tuned per configuration at the wider scales\. In each panel, the best SEL variant \(solid\) consistently reaches lower loss than the baseline \(dashed\)\. The annotation shows the final loss gapΔ\\Delta\.Table 1:Main results on OpenWebText\.Best SEL variant vs\. baseline at each scale\.Δ\\DeltaLoss is SEL minus baseline \(negative = better\)\. Speedup is baseline steps divided by SEL steps to reach baseline’s final loss e\.g\., 1\.25×\\timesmeans SEL reaches the same loss in 25% fewer steps\. Width 1024 trains for 250k steps \(β=7\.5\\beta=7\.5\); widths 2048 and 3072 train for 150k steps with per\-depthβ\\beta\.†Best variant is mismatch*without*offset \(all others use mismatch \+ offset\)\.Table[2](https://arxiv.org/html/2607.09967#S5.T2)expands the main comparison to show the two SEL variants run at each scale: mismatch without the optional residual/offset, and mismatch with the optional LoRA\-style residual plus offset\. At width 1024, the offset\+LoRA variant is consistently strongest; at larger widths, the no\-offset mismatch variant often matches or exceeds it\. We therefore report the better of the two in Table[1](https://arxiv.org/html/2607.09967#S5.T1)\.

Table 2:Variant comparison on the main grid\.Final eval loss for baseline, mismatch, and mismatch with optional LoRA\-style residual plus offset\.Figure[4](https://arxiv.org/html/2607.09967#S5.F4)shows evaluation loss curves across all nine configurations spanning three widths and three depths\. SEL consistently outperforms the baseline throughout training at every scale \(Table[1](https://arxiv.org/html/2607.09967#S5.T1)\)\. At width 1024 \(250k steps\), improvements grow with depth:Δ=−0\.038\\Delta=\-0\.038at depth 12,−0\.040\-0\.040at depth 24, and−0\.044\-0\.044at depth 36\. At width 2048 \(150k steps\), the gains are larger:Δ=−0\.047\\Delta=\-0\.047,−0\.042\-0\.042, and−0\.048\-0\.048for depths 12, 24, and 36\. At width 3072 \(150k steps\), gains remain strong:Δ=−0\.045\\Delta=\-0\.045,−0\.052\-0\.052, and−0\.043\-0\.043\. The largest single improvement isΔ=−0\.052\\Delta=\-0\.052at 3072×\\times24\. Across all nine configurations, the baseline never reaches SEL’s final loss within the same training budget\.

##### Choosingβ\\beta\.

We setβ\\betausing a simple initialization\-scale heuristic rather than treating it as a broad sweep\. Because Xavier uniform initialization has bounded support that shrinks with matrix width, larger matrices require largerβ\\betavalues for the symmetric\-exponential curvature to become active near typical initial weight magnitudes\. In practice, we chooseβ\\betaso that the exponential curve begins to noticeably depart from the linear regime around 1\.5–2 initialization standard deviations, while remaining within the support of the Xavier uniform draw\. This givesβ=7\.5\\beta=7\.5at width 1024,β∈\{12\.0,12\.5,15\.0\}\\beta\\in\\\{12\.0,12\.5,15\.0\\\}at width 2048, andβ∈\{17\.5,20\.0\}\\beta\\in\\\{17\.5,20\.0\\\}at width 3072\.

### 5\.3Ablation Studies

We explore a few ablations over specific design choices: initialization asymmetry, the necessity of the linear pathway, and the learning\-rate schedule for scale parameters\.

#### 5\.3\.1Initialization Asymmetry

At width 1024, we run a comparison varying the combination rule \(congruent vs mismatch\) along with the optional offset/LoRA components \(Table[3](https://arxiv.org/html/2607.09967#S5.T3)\)\. The mismatch forward mode improves over congruent mode at every depth, and adding offset/LoRA further improves the mismatch runs at this scale\. The asymmetric sign\-wrapping and initialization appears to be useful at all scales, though the main\-grid results in Table[2](https://arxiv.org/html/2607.09967#S5.T2)benefit from offset/LoRA changes at larger widths\.

Table 3:Width\-1024 initialization/combination comparison\.Final eval loss for the five runs available at width 1024\. All SEL runs use congruent inversion; mismatch rows therefore include the initialization/forward asymmetry described in §[4\.3](https://arxiv.org/html/2607.09967#S4.SS3)\.The same pattern holds in the newly run 3072×\\times12 comparison atβ=20\\beta=20\(Table[4](https://arxiv.org/html/2607.09967#S5.T4)\)\. Mismatch improves over congruent mode both with and without the optional offset/LoRA branch, though in this wider setting the no\-offset mismatch run is strongest\.

Table 4:Width\-3072 initialization/combination comparison\.Final eval loss at depth 12 withβ=20\\beta=20\.
#### 5\.3\.2Linear Pathway Necessity

A natural question is whether the linear pathwayLθ​\(w\)L\_\{\\theta\}\(w\)is truly necessary, or whether the symmetric\-exponential pathway alone is sufficient\. To test this, we compare the full method against anexponential\-onlyvariant that removes the linear pathway entirely:

weff=Sθ​\(w\)\+Δϕw\_\{\\mathrm\{eff\}\}=S\_\{\\theta\}\(w\)\+\\Delta\_\{\\phi\}\(12\)omitting theLθ​\(w\)L\_\{\\theta\}\(w\)term from Eq\. \([7](https://arxiv.org/html/2607.09967#S4.E7)\)\.

Figure[5](https://arxiv.org/html/2607.09967#S5.F5)compares the full method against the exponential\-only variant over their shared training range\. The exponential\-only run was stopped early after failing to approach the full model’s learning curve: at 31k steps it reached eval loss 3\.870, while the full model was already at 3\.142 at the same step and continued to 2\.925 by 99k steps\. This supports the view that the linear pathway provides a necessary effect\.

![Refer to caption](https://arxiv.org/html/2607.09967v1/x3.png)Figure 5:Exponential\-only ablation at 1024×\\times12\.Training step loss for the full SEL run and the exponential\-only variant, cropped to the shared 31k\-step range\. Removing the linear pathway leaves only the symmetric\-exponential branch; the run remains far above the full SEL curve and was interrupted, making it a clear failed\-to\-learn case rather than a competitive variant\.We hupothesize that the linear termLθ​\(w\)=w⋅lwL\_\{\\theta\}\(w\)=w\\cdot l\_\{w\}may provide one of two \(or both\) plausible stabilizing mechanisms: \(1\)local additive route: even when the symmetric\-exponential pathway’s contribution is small \(e\.g\., for near\-zero weights or when the offsetnnsuppresses activation\), the linear pathway preserves a direct dependence of effective weights on raw weights; \(2\)pathway balance: becauseewe\_\{w\}andlwl\_\{w\}are learned, the model can adjust how much it relies on the curved symmetric\-exponential branch versus the direct linear branch\.

#### 5\.3\.3LR Annealing

The scale parameters may need to move quickly enough early in training to select a useful regime, then slow down once the transform has settled\. We therefore test whether the schedule in §[4\.5](https://arxiv.org/html/2607.09967#S4.SS5)matters by comparing the default schedule against fixed learning\-rate multipliers\. The primary comparison isdefault annealingvsfixed multipliers\.

Table[5](https://arxiv.org/html/2607.09967#S5.T5)and Figure[6](https://arxiv.org/html/2607.09967#S5.F6)compare the default annealed schedule against fixed multipliers\. At the width\-1024 curvature setting, the annealed run is slightly better at the end of training \(2\.894 vs\. 2\.899\) with essentially identical runtime\. The scale trajectories show that most of the pathway rebalance occurs early:ewe\_\{w\}rises above its initialization,lwl\_\{w\}falls quickly, andmmremains close to one\.

Table 5:LR annealing ablation at 1024×\\times12\.Both runs use mismatch mode,β=7\\beta=7, AdamW, and 100k training steps\.![Refer to caption](https://arxiv.org/html/2607.09967v1/x4.png)
![Refer to caption](https://arxiv.org/html/2607.09967v1/x5.png)

Figure 6:LR annealing ablation and scale evolution\.*Left:*eval loss for annealed vs\. fixed scale multipliers, with the first 40% of training omitted for readability\.*Right:*mean effective scales forewe\_\{w\},lwl\_\{w\}, andmmover training in the annealed run\.
#### 5\.3\.4Curvature Parameterization

The inner multipliermmcontrols the effective curvatureβ​m\\beta min the exponential pathway\. This raises two parameterization questions\. First, should the base curvature be treated as a fixed scalar, withmminitialized near one, or should the learned inner multiplier absorb the baseβ\\betavalue directly? The latter is more naturally a multiplicative quantity: moving fromβ=7\\beta=7toβ=14\\beta=14and fromβ=7\\beta=7toβ=3\.5\\beta=3\.5are symmetric changes in log space, but not in linear parameter space\. We therefore parameterize the absorbed\-β\\betavariant with a positive exponential scale, so additive raw updates map to multiplicative changes in the effective curvature of the exponential pathway\.

Second, whenmmchanges the exponentβ​m​\|w\|\\beta m\\left\\lvert w\\right\\rvert, should it also change the near\-zero gain of the transform? By default, increasingmmincreases both curvature and the local derivative at the origin\. Thenormalize\_by\_inner\_multvariant instead divides the exponential and linear pathways bymm, sommmore directly controls curvature/growth rate while leaving the small\-signal normalization closer to fixed\. We evaluate the resulting2×22\\times 2design in a 100k\-step ablation \(Table[6](https://arxiv.org/html/2607.09967#S5.T6)\)\.

Table 6:Curvature parameterization ablation\.Actual 1024×\\times12, 100k\-step comparison of whetherβ\\betais fixed or absorbed into the learned inner multiplier, and whether the pathway outputs are normalized by that multiplier\. Lower eval loss is better\.The combined variant performs best in this sweep, suggesting that log\-space curvature updates and inner\-multiplier normalization are complementary rather than redundant\. For future work, it may be worth repeating the main experiments with the combined variant\.

### 5\.4Analysis

#### 5\.4\.1Weight Distribution Under the Transform

Figure[7](https://arxiv.org/html/2607.09967#S5.F7)compares standard baseline weights against mapped SEL weights at initialization and after training for matched 1024×\\times12 runs\. At initialization, the baseline raw weights and SEL effective weights are close by construction\. After training, the SEL transform produces a visibly different, more heavy\-tailed effective\-weight distribution than the baseline parameterization\.

![Refer to caption](https://arxiv.org/html/2607.09967v1/x6.png)Figure 7:Baseline vs\. SEL weight distributions\.Distribution of weights in a regular baseline parameterization training run \(blue\) vs\. and SEL training run at dim=1024 and depth=12\. Left shows the weight distribution at initialization and right shows the weight distribution at the end of 100k training steps\.
#### 5\.4\.2Scale Parameter Evolution

The scale trajectories in Figure[6](https://arxiv.org/html/2607.09967#S5.F6)show the change inewe\_\{w\},lwl\_\{w\}, andmmthroughout training\. In the annealed run, averaged across tracked effective\-scale tensors,ewe\_\{w\}rises from 1\.0 to about 1\.38, whilelwl\_\{w\}falls from 1\.0 to about 0\.54 by the final evaluation\. The inner multipliermmchanges much less, ending near 0\.98\. Most of the visible pathway rebalance occurs early in training, especially the initial rise inewe\_\{w\}and drop inlwl\_\{w\}\.

#### 5\.4\.3Transform Shape Visualization

Figure[8](https://arxiv.org/html/2607.09967#S5.F8)plots the learned transform by module role for theβ=7\.5\\beta=7\.5analysis run\. We observe in many cases the median example has a tilt, induced by modification of scale parameters, in slope at origin, pointing slightly upwards\. The visualization supports our hypothesis that the symmetric\-exponential shape is utilized, or even made more extreme, evidenced by the downweighting of the linear path while upweighting of the exponential path\.

![Refer to caption](https://arxiv.org/html/2607.09967v1/x7.png)Figure 8:Learned transform shapes by module role\.For sampled weight positions in the 1024×\\times12 annealed run withβ=7\.5\\beta=7\.5, we evaluateweffw\_\{\\mathrm\{eff\}\}as a function ofwraww\_\{\\mathrm\{raw\}\}using the learned per\-position scales\. Shaded bands show the 5–95% range across sampled positions; dashed lines show the identity\. The learned transforms preserve small\-weight control near zero while amplifying larger raw magnitudes\.![Refer to caption](https://arxiv.org/html/2607.09967v1/x8.png)Figure 9:Outlier\-sensitive transform shapes by module role\.The same learned transforms as Figure[8](https://arxiv.org/html/2607.09967#S5.F8), but with wider 1–99% and 0\.1–99\.9% bands plus individual curves from the largest sampled raw/effective weight positions\. Rare positions show much stronger tail amplification than the median, matching the heavy\-tail motivation for the parameterization\.

## 6Discussion

##### Why symmetric\-exponential \+ linear?

The symmetric\-exponential pathway provides the magnitude\-dependent sensitivity and multiplicative updates: small weights remain in a near\-linear regime, while larger raw weights enter a curved regime where linear raw\-parameter update steps can produce a larger effective\-weight change\. The linear pathway gives the model a direct additive route from raw to effective weights, and the learned pathway scales allow training to choose how much to rely on each branch\. This is loosely analogous to a residual path in the limited sense that gradients need not pass only through the curved branch, though both pathways are functions of the same underlying weight\. The exponential\-only ablation proposed in §[5\.3\.2](https://arxiv.org/html/2607.09967#S5.SS3.SSS2)is designed to test whether both components are essential\.

Viewed through the lens of the optimizer\-initialization mismatch described in Section[3](https://arxiv.org/html/2607.09967#S3), SEL can be understood as making output space parameter updates proportional to weight magnitude\. Under standard parameterization with Adam, a weight at magnitude0\.010\.01and one at magnitude1\.01\.0receive updates of similar absolute size, but the small weight is perturbed by100×100\\timesmore in relative terms\. SEL’s symmetric\-exponential pathway counteracts this by increasing∂weff/∂wraw\\partial w\_\{\\mathrm\{eff\}\}/\\partial w\_\{\\mathrm\{raw\}\}with\|w\|\\left\\lvert w\\right\\rvert: a comparable raw\-space step corresponds to a larger movement in effective weight space once the parameter has entered the curved regime\. Thus the parameterization offers the relative scaling that a standard parameterization with Adam would not have, without requiring per\-parameter learning rate tuning\.μ\\muP\[Yanget al\.,[2022](https://arxiv.org/html/2607.09967#bib.bib11)\]addresses a related but separate axis of the same mismatch: it rescales per\-tensor learning rates from static, width\-dependent init statistics/typical dimension magnitudes, whereas SEL adapts effective step size per\-parameter on the fly as weights grow\. The two are complementary and freely composable\.

##### The mismatch insight\.

A surprising finding in our ablations is that mismatched initialization outperforms the congruent versions\. The specific asymmetry introduced by the mismatch \(weaker negative weights\) may provide a useful inductive bias or early symmetry\-breaking effect\.

##### Structured scales as lightweight adaptation\.

The structured scale parameters add a small number of auxiliary parameters \(two vectors of sizedoutd\_\{\\mathrm\{out\}\}anddind\_\{\\mathrm\{in\}\}per scale, per layer\) that modulate the reparameterization\. This is significantly cheaper than full element\-wise scales and provides enough expressivity to adapt the transform per\-row and per\-column\. Therow\_col\_multpattern is balanced between expressivity and cost for the output pathway scales\. Each output neuron and each input feature has its own multiplicative adjustment to the exponential/linear balance\. We also experimented with additive row\+column composition, which is appealing for quantities that behave like log\-space shifts, such as the inner multiplier or offset\. However, additive row\+column scales introduce a gauge symmetry: adding a constant to the row vector and subtracting it from the column vector leaves the combined scale unchanged\. In practice this underperformed the multiplicative row×\\timescolumn form, so we userow\_col\_multby default\.

##### Future work: pretrained conversion and finetuning\.

Because SEL changes only the parameterization of weights, conversion is possible in both directions\. After training, effective weights can be materialized and deployed as ordinary linear layers\. Conversely, a standard pretrained weight matrix can be inverted into the SEL raw parameter space by solvingf​\(Wraw\)=Wpref\(W\_\{\\mathrm\{raw\}\}\)=W\_\{\\mathrm\{pre\}\}, yielding an equivalent model at the moment of conversion\. This makes continued pretraining and finetuning in SEL space a natural direction for future work\.

##### Training and inference cost\.

A key practical advantage of SEL over architectural modifications \(e\.g\., mixture\-of\-experts, nonlinear branches\) is that it changes the parameterization without changing the deployed architecture\. During training, the transform consists of element\-wise operations on weight tensors, so compiled implementations usingtorch\.compile, Triton kernels, and fused AdamW keep the overhead modest\. Table[7](https://arxiv.org/html/2607.09967#S6.T7)shows preliminary H200 profiling for representative small and large configurations\. Forward/backward overhead is small, while the fused AdamW step shows the most visible increase\. Combining per\-step overhead with the step reductions in Table[1](https://arxiv.org/html/2607.09967#S5.T1), SEL still yields estimated wall\-clock speedups to matched validation loss\.

Table 7:Preliminary H200 profiling and estimated wall\-clock speedup\.Timings are milliseconds per training step component, reported as baseline/SEL\. Step overhead is computed from the summed forward, backward, and optimizer times\. Estimated wall\-clock speedup divides the step reduction from Table[1](https://arxiv.org/html/2607.09967#S5.T1)by the measured per\-step overhead\.These measurements indicate that the transform’s training cost is small relative to the reduction in steps\. After training, one computesWeff=f​\(Wraw\)W\_\{\\mathrm\{eff\}\}=f\(W\_\{\\mathrm\{raw\}\}\)once and converts to standard linear layers\. All auxiliary parameters \(structured scales, the curvatureβ\\beta, etc\.\) are folded into the effective weights and discarded, givingzero additional inference cost\.

##### Alternative formulations: sinh\.

The hyperbolic sine functionsinh⁡\(x\)=\(ex−e−x\)/2\\sinh\(x\)=\(e^\{x\}\-e^\{\-x\}\)/2shares many properties with symexp: it is an odd function that behaves approximately linearly for small\|x\|\\left\\lvert x\\right\\rvert\(sincesinh⁡\(x\)≈x\\sinh\(x\)\\approx xfor\|x\|≪1\\left\\lvert x\\right\\rvert\\ll 1\) and grows exponentially for large\|x\|\\left\\lvert x\\right\\rvert\. A key advantage of sinh over symexp is that it is*fully smooth*everywhere, including at the origin, whereas symexp \(and by extension SEL\) has a non\-differentiable point atw=0w=0due to the\|w\|\\left\\lvert w\\right\\rvertterm\. A sinh\-based reparameterizationweff=sinh⁡\(β​w\)/βw\_\{\\mathrm\{eff\}\}=\\sinh\(\\beta w\)/\\betawould provide similar magnitude\-dependent curvature while guaranteeing continuous gradients\. We leave systematic comparison of sinh\-based transforms to future work, noting that preliminary experiments suggest comparable performance with potentially improved optimization smoothness\.

##### Interaction with weight decay\.

Weight decay is itself magnitude\-aware: applyingw←w−λ​w=w​\(1−λ\)w\\leftarrow w\-\\lambda w=w\(1\-\\lambda\)shrinks parameters in proportion to their current value, so larger parameters receive larger absolute shrinkage\. SEL introduces a different kind of magnitude dependence through the raw\-to\-effective map: as\|wraw\|\\left\\lvert w\_\{\\mathrm\{raw\}\}\\right\\rvertgrows, the local sensitivity∂weff/∂wraw\\partial w\_\{\\mathrm\{eff\}\}/\\partial w\_\{\\mathrm\{raw\}\}increases, so comparable raw\-space steps can produce larger movements in effective weight space\. Thus both mechanisms act in a magnitude\-aware manner\.

##### Weight decay on auxiliary scales\.

Decaying the raw weights alone does not constrain every route by which effective weights can grow: the output pathway scalesewe\_\{w\}andlwl\_\{w\}directly multiply the symmetric\-exponential and linear contributions\. Thus, they can increase effective magnitudes even whenwraww\_\{\\mathrm\{raw\}\}is decayed\. We therefore apply weight decay toewe\_\{w\}andlwl\_\{w\}, while disabling it for geometry\-shaping parameters such as the inner multipliermmand offsetnn\.

##### Outliers at scale and their functional importance\.

Recent work has shown that larger language models develop increasingly pronounced weight and activation outliers\. These outliers are not demonstrated to be critical to model performance and degrade performance if removed\.Dettmerset al\.\[[2022](https://arxiv.org/html/2607.09967#bib.bib23)\]documented that outlier features emerge at scale\. Models under∼\\sim6B parameters have few outliers, but larger models develop specific dimensions with extreme values that dominate attention\.Sunet al\.\[[2024](https://arxiv.org/html/2607.09967#bib.bib27)\]found “massive activations” in LLMs that can be 100,000×\\timeslarger than typical values; these activations remain constant regardless of input and function as indispensable bias terms\. Most strikingly,Yuet al\.\[[2024](https://arxiv.org/html/2607.09967#bib.bib25)\]showed that pruning a*single*“super weight” can destroy an LLM’s ability to generate coherent text, increasing perplexity by three orders of magnitude\. These super weights, typically found inmlp\.down\_projof early layers, create persistent super activations that suppress stopword likelihoods in final logits\.

This body of work suggests that during training, some parameters must grow to extreme values and that these are the parameters most disadvantaged by linear additive updates\. SEL’s exponential amplification directly addresses this: as weights grow, their effective gradients are amplified, accelerating progress toward large\-magnitude targets\. At scale, where initialization values shrink further \(due to larger matrix dimensions\) while the need for extreme outliers persists or intensifies, largerβ\\betavalues may be needed to provide greater curvature and accelerated update speed to reach final values\.

##### Multiplicative computation, multiplicative parameterization\.

A useful intuition for SEL comes from considering how parameters affect computation\. LayerNorm scale parameters provide a clean example: initialized atγ=1\.0\\gamma=1\.0, moving toγ=0\.5\\gamma=0\.5halves the output scale while moving toγ=2\.0\\gamma=2\.0doubles it\. These are symmetric operations in terms of their*computational effect*\(÷2\\div 2vs×2\\times 2\), yet asymmetric in optimization distance \(0\.50\.5vs1\.01\.0\)\. Under Adam’s near\-constant update magnitudes, “doubling” takes twice as many steps as “halving\.” The natural parameterization for such multiplicative quantities is logarithmic: withγeff=exp⁡\(γraw\)\\gamma\_\{\\mathrm\{eff\}\}=\\exp\(\\gamma\_\{\\mathrm\{raw\}\}\), both operations require\|Δ​γraw\|=log⁡2\\left\\lvert\\Delta\\gamma\_\{\\mathrm\{raw\}\}\\right\\rvert=\\log 2\.

We suggest this intuition extends, at least partially, to matrix weights\. Iny=W​xy=Wx, each weightwi​jw\_\{ij\}multiplicatively scalesxjx\_\{j\}’s contribution toyiy\_\{i\}\. A weight growing from0\.01→0\.10\.01\\to 0\.1and one growing from0\.1→1\.00\.1\\to 1\.0both represent10×10\\timesmultiplicative changes, but the latter requires covering10×10\\timesmore optimization distance\. Not all aspects of the function implemented by the composed neural network may act in a relative manner: biases are additive, activation functions have thresholds at absolute values\. However, SEL attempts to respect both regimes\. The offsetnnprovides explicit control over this transition\. This “soft log\-space” parameterization may explain why the symmetric\-exponential pathway helps weights reach large\-magnitude targets that the network’s computation requires\.

## 7Limitations

- •Implementation dependence: The training transform is element\-wise and is applied on every forward pass, so its overhead depends on how well these operations are fused bytorch\.compile/Triton or similar compilers\. The additional scale parameters also increase the number of optimizer tensors; foreach or fused optimizer implementations reduce the overhead compared with Python loops over many small tensors\.
- •Hyperparameter sensitivity: The method introduces several hyperparameters \(β\\beta,kkvalues for each scale, annealing schedule\)\. We userow\_col\_multas a robust structured\-scale default and chooseβ\\betawith the width\-dependent initialization heuristic above, but further work is needed to understand how these choices should scale beyond our tested model sizes\.
- •Scale: Our largest configuration \(3072×\\times36\) is relatively small by modern standards\. Validation at larger scales would strengthen the results\.
- •Task diversity: We evaluate only autoregressive language modeling\. The method may behave differently on other tasks \(classification, generation with different loss functions, multimodal models\)\.
- •Theoretical understanding: We provide intuitive explanations for why mismatched initialization helps but lack formal analysis\. A rigorous understanding of the optimization landscape under this reparameterization remains open\.

## 8Conclusion

We introduced SymExpLin \(SEL\), a nonlinear weight reparameterization that decomposes transformer linear layer weights into symmetric\-exponential and linear components\. One useful insight is that a deliberate asymmetry between initialization and forward\-pass combination rules can improve optimization in small\-scale ablations, possibly by acting as early symmetry breaking\. Learnable scale parameters allow modifying the the weight overall transform, and a targeted annealing schedule manages the transition between partially redundant exponential controls\. SEL has modest training overhead in compiled implementations because its transform is element\-wise, and adds no inference cost because it requires only a one\-time weight materialization after training\.

Overall, the remaining ablations support the same picture as the main grid: mismatch remains helpful at 3072×\\times12, the annealed schedule gives a small but consistent improvement over fixed multipliers, and removing the linear pathway produces a clear failed\-to\-learn case\.

## References

- S\. Arora, E\. Hazan, and S\. Kale \(2012\)The multiplicative weights update method: a meta\-algorithm and applications\.Theory of Computing8\(1\),pp\. 121–164\.Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px5.p2.2)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30318–30332\.Cited by:[§3](https://arxiv.org/html/2607.09967#S3.SS0.SSS0.Px4.p1.9),[§6](https://arxiv.org/html/2607.09967#S6.SS0.SSS0.Px9.p1.2)\.
- X\. Glorot and Y\. Bengio \(2010\)Understanding the difficulty of training deep feedforward neural networks\.InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,pp\. 249–256\.Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2607.09967#S4.SS3.p1.2)\.
- A\. Gokaslan and V\. Cohen \(2019\)OpenWebText corpus\.External Links:[Link](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§5\.1](https://arxiv.org/html/2607.09967#S5.SS1.SSS0.Px3.p1.2)\.
- D\. Hafner, J\. Pasukonis, J\. Ba, and T\. Lillicrap \(2023\)Mastering diverse domains through world models\.arXiv preprint arXiv:2301\.04104\.Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.09967#S4.SS1.p1.4)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2015\)Delving deep into rectifiers: surpassing human\-level performance on imagenet classification\.InProceedings of the IEEE International Conference on Computer Vision,pp\. 1026–1034\.Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§3](https://arxiv.org/html/2607.09967#S3.SS0.SSS0.Px2.p1.2),[4th item](https://arxiv.org/html/2607.09967#S4.I2.i4.p1.4)\.
- S\. M\. Jayakumar, W\. M\. Czarnecki, J\. Menick, J\. Schwarz, J\. Rae, S\. Osindero, Y\. W\. Teh, T\. Harley, and R\. Pascanu \(2020\)Multiplicative interactions and where to find them\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px5.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§1](https://arxiv.org/html/2607.09967#S1.p3.2)\.
- I\. Loshchilov and F\. Hutter \(2017\)Decoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§5\.1](https://arxiv.org/html/2607.09967#S5.SS1.SSS0.Px3.p1.2)\.
- D\. Mishkin and J\. Matas \(2016\)All you need is a good init\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Miyato, T\. Kataoka, M\. Koyama, and Y\. Yoshida \(2018\)Spectral normalization for generative adversarial networks\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3](https://arxiv.org/html/2607.09967#S3.SS0.SSS0.Px4.p1.9)\.
- T\. Salimans and D\. P\. Kingma \(2016\)Weight normalization: a simple reparameterization to accelerate training of deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Shazeer \(2020\)GLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§5\.1](https://arxiv.org/html/2607.09967#S5.SS1.SSS0.Px1.p1.1)\.
- J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu \(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§5\.1](https://arxiv.org/html/2607.09967#S5.SS1.SSS0.Px1.p1.1)\.
- M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu \(2024\)Massive activations in large language models\.arXiv preprint arXiv:2402\.17762\.Cited by:[§6](https://arxiv.org/html/2607.09967#S6.SS0.SSS0.Px9.p1.2)\.
- G\. Yang, E\. J\. Hu, I\. Babuschkin, S\. Sidor, X\. Liu, D\. Farhi, N\. Ryder, J\. Pachocki, W\. Chen, and J\. Gao \(2022\)Tensor programs V: tuning large neural networks via zero\-shot hyperparameter transfer\.arXiv preprint arXiv:2203\.03466\.Cited by:[§2](https://arxiv.org/html/2607.09967#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2607.09967#S3.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.09967#S6.SS0.SSS0.Px1.p2.6)\.
- M\. Yu, D\. Wang, Q\. Shan, C\. J\. Reed, and A\. Wan \(2024\)The super weight in large language models\.arXiv preprint arXiv:2411\.07191\.Cited by:[§6](https://arxiv.org/html/2607.09967#S6.SS0.SSS0.Px9.p1.2)\.
- B\. Zhang and R\. Sennrich \(2019\)Root mean square layer normalization\.Advances in Neural Information Processing Systems32\.Cited by:[§5\.1](https://arxiv.org/html/2607.09967#S5.SS1.SSS0.Px1.p1.1)\.

## Appendix AImplementation Details

##### SymExpLin Transform \(Full\)\.

The complete forward pass for a weight matrixW∈ℝdout×dinW\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}is:

1. 1\.Compute scaled tensor:z=β⋅w⋅m​\(\)z=\\beta\\cdot w\\cdot m\(\)wherem​\(\)m\(\)is the inner multiplier structured scale\.
2. 2\.Compute shifted absolute value:a=\|z\|\+s​\(\)a=\\left\\lvert z\\right\\rvert\+s\(\)wheres​\(\)s\(\)is the shift structured scale\.
3. 3\.Compute exponential part:E=\(exp⁡\(a−n​\(\)\)−exp⁡\(−n​\(\)\)\)⋅ew​\(\)E=\(\\exp\(a\-n\(\)\)\-\\exp\(\-n\(\)\)\)\\cdot e\_\{w\}\(\)wheren​\(\)n\(\)is the offset andew​\(\)e\_\{w\}\(\)is the exponential scale\.
4. 4\.Compute linear part:L=w⋅lw​\(\)L=w\\cdot l\_\{w\}\(\)wherelw​\(\)l\_\{w\}\(\)is the linear scale\.
5. 5\.Combine \(mismatch mode\):weff=sign⁡\(w\)​\(E\+L\)\+Δϕw\_\{\\mathrm\{eff\}\}=\\operatorname\{sign\}\(w\)\(E\+L\)\+\\Delta\_\{\\phi\}\.

##### Newton’s Method Initialization\.

Given target weightsWtargetW\_\{\\mathrm\{target\}\}from Xavier uniform initialization, we solve for raw weights vianiter=10n\_\{\\mathrm\{iter\}\}=10iterations of Newton’s method under the congruent formula:

f​\(x\)\\displaystyle f\(x\)=sign⁡\(x\)⋅ew,0​\(exp⁡\(β​\|x\|\)−1\)\+x⋅lw,0\\displaystyle=\\operatorname\{sign\}\(x\)\\cdot e\_\{w,0\}\\bigl\(\\exp\(\\beta\\left\\lvert x\\right\\rvert\)\-1\\bigr\)\+x\\cdot l\_\{w,0\}\(13\)f′​\(x\)\\displaystyle f^\{\\prime\}\(x\)=ew,0​β⋅exp⁡\(β​\|x\|\)\+lw,0\\displaystyle=e\_\{w,0\}\\beta\\cdot\\exp\(\\beta\\left\\lvert x\\right\\rvert\)\+l\_\{w,0\}\(14\)xk\+1\\displaystyle x\_\{k\+1\}=xk−f​\(xk\)−Wtargetf′​\(xk\)\\displaystyle=x\_\{k\}\-\\frac\{f\(x\_\{k\}\)\-W\_\{\\mathrm\{target\}\}\}\{f^\{\\prime\}\(x\_\{k\}\)\}\(15\)Hereew,0e\_\{w,0\}andlw,0l\_\{w,0\}are the initial exponential and linear pathway scales, set to1/β1/\\betain the default normalized parameterization\. The iteration converges rapidly \(typically within 5–6 iterations\) sinceffis monotonically increasing and smooth except atx=0x=0\.

##### Structured Scale Modes\.

Therow\_col\_multmode stores two vectorsr∈ℝdoutr\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\}andc∈ℝdinc\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\}, initialized so thatri⋅cj=vr\_\{i\}\\cdot c\_\{j\}=vfor alli,ji,jwherevvis the target value\. The effective scale isr⋅c⊤r\\cdot c^\{\\top\}, broadcast to match the weight matrix shape\. For multiplicative composition, each component is initialized tov\\sqrt\{v\}\(orv1/2v^\{1/2\}from the inverse transform if a nonlinear transform is applied\)\.

## Appendix BAlternative Formulations

During development, we explored several alternative formulations before arriving at the version presented in the main paper\. Here we document these variants for completeness\.

### B\.1Combination Modes

The main paper focuses on thecongruentandmismatchmodes for combining the symmetric\-exponential and linear pathways\. Two additional modes were explored:

##### Oppose mode\.

In this variant, the linear pathway*subtracts*from the exponential:

weff=Sθ​\(w\)−Lθ​\(w\)w\_\{\\mathrm\{eff\}\}=S\_\{\\theta\}\(w\)\-L\_\{\\theta\}\(w\)\(16\)whereSθ​\(w\)S\_\{\\theta\}\(w\)is the symmetric\-exponential pathway andLθ​\(w\)=w⋅lwL\_\{\\theta\}\(w\)=w\\cdot l\_\{w\}is the linear pathway\. This creates an asymmetry where the symmetric\-exponential pathway must “overcome” the linear opposition\. We observed reduced performance in this configuration, possibly because the opposing forces create optimization challenges\.

##### Exp\-only mode\.

This variant removes the linear pathway entirely:weff=Sθ​\(w\)w\_\{\\mathrm\{eff\}\}=S\_\{\\theta\}\(w\)\. Without the linear pathway, the transform behaves purely exponentially for all weight magnitudes\. This showed faster initial progress but sometimes exhibited instability; the linear pathway appears to serve as a stabilizing influence\.

### B\.2Damped Exponential Formulation

An alternative formulation introduces explicit damping to prevent exponential blowup for large weight magnitudes:

weff=sign⁡\(w\)⋅exp⁡\(β​\|w\|−n\)−exp⁡\(−n\)β⋅exp⁡\(\|w\|\)\+w⋅\(1−exp⁡\(−n⋅exp⁡\(\|w\|\)\)\)w\_\{\\mathrm\{eff\}\}=\\operatorname\{sign\}\(w\)\\cdot\\frac\{\\exp\(\\beta\\left\\lvert w\\right\\rvert\-n\)\-\\exp\(\-n\)\}\{\\beta\\cdot\\exp\(\\left\\lvert w\\right\\rvert\)\}\+w\\cdot\\left\(1\-\\exp\(\-n\\cdot\\exp\(\\left\\lvert w\\right\\rvert\)\)\\right\)\(17\)
This formulation has two key modifications:

##### Damping denominator\.

The symmetric\-exponential pathway is divided byβ⋅exp⁡\(\|w\|\)\\beta\\cdot\\exp\(\\left\\lvert w\\right\\rvert\)rather than justβ\\beta\. For large\|w\|\\left\\lvert w\\right\\rvert, this grows asexp⁡\(\|w\|\)\\exp\(\\left\\lvert w\\right\\rvert\), which partially cancels theexp⁡\(β​\|w\|\)\\exp\(\\beta\\left\\lvert w\\right\\rvert\)in the numerator\. The net effect is that the symmetric\-exponential pathway grows asexp⁡\(\(β−1\)​\|w\|\)\\exp\(\(\\beta\-1\)\\left\\lvert w\\right\\rvert\)for large weights, taming the otherwise unbounded exponential growth whenβ\>1\\beta\>1\.

##### Input\-dependent linear coefficient\.

Instead of a constant linear pathwayw/βw/\\beta, the coefficient\(1−exp⁡\(−n⋅exp⁡\(\|w\|\)\)\)\(1\-\\exp\(\-n\\cdot\\exp\(\\left\\lvert w\\right\\rvert\)\)\)depends on the weight magnitude\. For small\|w\|\\left\\lvert w\\right\\rvert, this coefficient is small \(suppressing the linear term\), while for large\|w\|\\left\\lvert w\\right\\rvertit rapidly saturates to 1\. This gives the linear pathway a naturally high contribution for weights that have already grown large, providing stability in the large\-weight regime\.

The damped formulation is more conservative than the version in the main paper: it provides magnitude\-dependent curvature while explicitly preventing runaway growth\. However, in practice we found that the simpler undamped formulation \(with appropriateβ\\betatuning\) achieved comparable or better results with fewer hyperparameters to tune\.

## Appendix CSupplementary Figures

### C\.1Transform Visualizations

![Refer to caption](https://arxiv.org/html/2607.09967v1/x9.png)Figure 10:Combined effect ofβ\\betaand offsetnn\.A 2×\\times2 grid showing how both parameters jointly affect the transform shape\. Higherβ\\betaincreases curvature; highernnsuppresses the exponential near zero\.![Refer to caption](https://arxiv.org/html/2607.09967v1/x10.png)Figure 11:Standard symexp vs\. SEL symmetric\-exponential pathway\.*Left:*The standard symexp function has fixed curvature\.*Right:*SEL withβ=15\\beta=15provides controllable curvature while maintaining similar qualitative behavior\.
### C\.2Mismatched Initialization Analysis

![Refer to caption](https://arxiv.org/html/2607.09967v1/x11.png)Figure 12:Detailed analysis of mismatched initialization\.*Left:*Scatter plot of target \(Xavier\) vs\. actual effective weights after congruent inversion \+ mismatch forward\. Positive weights \(blue\) lie on the identity while negative weights \(red\) are systematically under\-initialized\.*Middle:*Distribution shift between target and actual\.*Right:*Magnitude ratio by sign: positive weights achieve∼\\sim100% of target, negative weights only∼\\sim75–80%\.
### C\.3Weight Distribution Analysis

![Refer to caption](https://arxiv.org/html/2607.09967v1/figures/weights2.png)Figure 13:Optimization distance analysis\.*Left:*Distribution of optimization distances\|wpretrained−winit\|\\left\\lvert w\_\{\\mathrm\{pretrained\}\}\-w\_\{\\mathrm\{init\}\}\\right\\rvertshowing a heavy tail\.*Right:*Rank plot \(log\-log\) of theoretical minimum steps needed to reach target values, demonstrating that outlier weights require 10–20×\\timesmore steps than average\.![Refer to caption](https://arxiv.org/html/2607.09967v1/figures/weights4.png)Figure 14:Heavy\-tail characterization of pretrained weights\.*Top left:*Per\-layer max weight growth ratio\.*Top right:*Correlation between init max and pretrained max per layer\.*Bottom left:*Std comparison showing modest overall growth\.*Bottom right:*Percentile plot showing the heavy tail\. Pretrained weights \(red\) diverge sharply from Xavier \(blue\) in the upper percentiles\.