This paper proposes Native Factorized Weights for transformers, where every linear layer is trained as a product of two low-rank matrices from initialization. Experiments show a corpus-determined optimal rank that minimizes validation loss and a generalization band, outperforming dense baselines with fewer parameters.
I've been running an experiment I'm calling Native Factorized Weights (NFW): instead of training a standard transformer and compressing it afterward, every linear layer is replaced by W = V·Uᵀ at initialization and trained that way from scratch. No post-hoc SVD. No LoRA adapter on top of a pretrained model. The factorization is the basis for the model and earns the name "Sliver layers" as each layer is comprised of two low-rank matrices. A comparison of a dense W matrix using n^2 parameters, vs. U and V comprising 2nr parameters It works. At the right rank, NFW models outperform a dense baseline with the same layer count and a fraction of the parameter count. Because parameter count is dramatically reduced, the budget can be spent on doubling the hidden dimension instead. As such, most of the comparisons made in the paper are Config D (dense; n x n weights) vs. Config S{r} (sliver-of-rank-r; 2nr + 2nr weights). The Config S allocation achieves structural compression for r << n. That alone would be interesting. But the more surprising finding is why, and where the boundary of an appropriate rank is. The core finding There is a corpus-determined optimal rank r* that minimizes validation loss. It is not a function of model size - it is a function of the training corpus. Above a second threshold r' (the memorization onset), the model acquires enough capacity to start memorizing within the training budget, and validation loss begins rising. This creates a generalization band [r*, r') that is structurally safe to train in. It is not obvious from the loss curves alone that you are in or out of the band until you map the full rank sweep. Results on WikiText-2 (n=2048, L=4 transformer, 20k steps) The curve of NFW training outcomes by rank, compared to a dense W baseline The graphic above shows val @ 20,000 steps and the clear evidence that an optimal rank is bounded on both sides. This is matched by assessment of the lowest val observed at any step during training: Dense baseline: val perplexity 6.219, train/val gap 1.504 NFW r=8: val 7.423 (underfitting regime) NFW r=16: val 6.228 (approaching r*) NFW r=32: val 5.617, gap 1.302 - beats dense with fewer parameters NFW r=64: val 5.849, gap 1.841 (gap widening - approaching r') NFW r=128: val 6.083, gap 2.603 (memorization pressure visible) With production training conditions (dropout + warmup) Dense + dropout/warmup: val min 5.759 but diverged after step 9600 - gap exploded to 3.9 NFW r=32 + dropout/warmup: val 5.545, gap 1.148 - stable throughout The rank constraint and stochastic regularization act on orthogonal failure modes. Dropout combats noise; the rank ceiling combats memorization. A dense model with dropout can still memorize - it just memorizes noisily. The rank-constrained model structurally cannot. The geometric interpretation The weight matrices in a trained transformer live near a low-rank manifold. NFW just makes that manifold the training space rather than the ambient space. r* is where the manifold dimension matches the information content of the corpus. r' is where the manifold is wide enough to fit the training set. In current W matrices, the outcome of training resists clear decomposition into a U and V because the open field allows noise to accumulate and confounds distillation. By training the model using U and V as the only allowable storage, such noise is never allowed. Computational Advantage This structure has no native kernel for inference at this time. To perform inference, the W matrix must be materialized so that standard transformer inference can be performed. If this research is productive and the model type deserves adoption, a native kernel designed for it would achieve a comparable compression rate for calculation due to the reduced number of parameters. Code and results GitHub: [https://github.com/Malkom1366/native-factorized-weights](vscode-file://vscode-app/c:/Users/malko/AppData/Local/Programs/Microsoft%20VS%20Code/fcf604774b/resources/app/out/vs/code/electron-browser/workbench/workbench.html) Full results JSON, training script, and the Sliver layer implementation are all there. The ask I'm an independent researcher. I have the full (much longer) paper ready but arXiv requires endorsement for cs.LG first-time submitters. If you're eligible and willing to endorse after looking at the repo, I'd be very grateful. I am very eager for others to replicate these results and confirm that they hold up on more corpora.
This paper introduces an automated LLM pipeline to generate and verify human-readable descriptions of when individual weights in transformers matter, finding that 12-31% of weights in weight-sparse transformers are interpretable globally, outperforming dense transformers.
This paper systematically studies layer-wise contribution in RL post-training for LLMs, finding that training a single middle transformer layer can recover or even surpass full-parameter RL gains, with consistent patterns across models and tasks.
A study finds that reinforcement learning gains in transformers are concentrated in a single middle layer, and training just that layer can match or surpass full-parameter RL training across multiple models and tasks.
This paper introduces FLITE (Federated Low-rank Iterative Training Engine), a method for federated fine-tuning that reduces per-client communication to 1,280 floats per round (about 5KB) — an 8718× reduction over full-weight FedAvg — by using a frozen affine mapping network that generates weights from a small trainable latent and a low-rank seed-regenerable factorization, achieving accuracy within 0.5 percentage points of full-weight FedAvg on CIFAR-100 with ResNet-18.
Researchers from Adelaide University introduce LR-LoRA (Learnable Rank LoRA), a parameter-efficient fine-tuning method that dynamically learns the adapter rank for each transformer layer during training rather than using a fixed global rank. LR-LoRA achieves state-of-the-art performance on language understanding and commonsense reasoning benchmarks, outperforming fixed-rank LoRA baselines.